Chapter 10: State and Persistence: Sessions and Conversation Transcripts

Kinds of State

Before writing storage code, classify the data. "Memory" is not one thing: there is fast, disposable, right-now context — the session — and the slow, permanent, complete record — the transcript. Conflating them yields systems that are either too slow (durable storage on the hot path) or too forgetful (everything expires). Keeping them separate is the discipline at the heart of the chapter.

Session state is short-term data scoped to an active conversation: recent message history, the active workflow step, in-progress slot filling (for example, a booking where we have the date but not the time). It is small, hot, and disposable — losing it is inconvenient but not catastrophic because it can be rebuilt from the durable record. A conversation transcript is the opposite: the persistent, append-only record that survives sessions, restarts, and channels, kept for context reconstruction, analytics, retention, and audit.

Analogy: A barista's session is the sticky note on the cup — scribbled fast, thrown away once the drink is handed over. The transcript is the point-of-sale system — every order ever placed, queryable, retained for the tax year. You never run the espresso line off the accounting database, and you never file taxes from discarded sticky notes.

Figure 10.1: Ephemeral session state versus durable conversation history

Once state is a session, we decide what it is a session of. In a single-channel product "the user" suffices; in a multi-channel gateway it does not. The robust convention scopes by the tuple of platform, user, and channel — session:slack:U12345:C0777 versus session:sms:+15551234567 — so a user's Slack context does not bleed into their SMS context. Cross-channel memory, when desired, is achieved not by sharing one session but by having both channels' sessions project from the same durable history keyed to the underlying human.

Figure 10.3: The two layers of state — ephemeral session versus durable history

Workers are interchangeable because they are stateless — they hold no conversation memory between messages. The resolution is a clean division of labor: the worker is stateless; the store is stateful. On each turn a worker loads the session at the start, does its work, and writes it back at the end. Because the session lives in an external store every worker can reach, it does not matter which worker handles which message. Memory is externalized, so the things that process messages stay memoryless.

Figure 10.4: Deriving a scoping key from the (platform, user, channel) tuple

Session Storage

Redis stores are flat key-value spaces, but a naming convention gives them structure. The established pattern uses hierarchical, colon-delimited keys that co-locate everything about one session — a metadata key and a history key sharing a common prefix, e.g. agent:session:{id}:meta and agent:session:{id}:history. A hash (HSET) holds metadata (user id, channel, created_at, message_count, active-workflow state); a list (RPUSH/LRANGE) holds ordered recent turns as JSON, giving atomic append and fast ordered reads. To keep the hot store bounded, LTRIM history -N -1 keeps only the last N turns (50 is common), maintaining a sliding window.

The worked async store below derives a deterministic scoping key and appends inside a pipeline with transaction=True: the push, trim, counter increment, and both EXPIRE calls execute atomically, so history and metadata never fall out of sync and no key is left without a TTL.

The single most important operational rule for a session cache is that every key should carry a TTL. Redis deletes expired keys automatically, so TTL is both memory reclamation and the natural "forget an idle conversation" behavior. Keys without a TTL accumulate until out-of-memory — a classic production failure. Typical chatbot TTLs run 60 minutes to 2 hours (7200 seconds is common). A fixed TTL expires a set time after creation and can cut a user off mid-conversation; a sliding TTL re-arms expiry on every access, so active sessions persist and abandoned ones decay. Apply the same TTL to both the metadata hash and the history list so they expire together.

When Redis is unavailable, or session state must survive a restart, a database can back the store: a sessions table keyed by (channel, user_id) with a JSON/JSONB state column and an expires_at timestamp, expiry enforced by a WHERE expires_at > now() filter plus a background purge job. Many gateways use both — Redis as the hot cache, checkpointed periodically to a database for durability.

Because workers are interchangeable and messages flow through a queue, two messages from the same user can be processed nearly simultaneously by two workers. Both load the session, both mutate it, both write it back; whichever writes last wins, silently discarding the other's changes — the last-write-wins hazard. For ordering-tolerant conversational state this is acceptable (and the transcript still captures both messages). Where it bites is structured read-modify-write state — slot filling. Two mitigations keep this safe: prefer atomic field-level operations (HSET one field, RPUSH one turn, HINCRBY a counter) so Redis serializes updates; and where several fields must change together based on their current values, use optimistic concurrency with WATCH/MULTI/EXEC, which aborts and lets you retry if the key changed underneath you.

Conversation Transcripts

The session answers the next turn; the transcript is the truth. A practical transcript model captures, per message: a stable message_id, the owning conversation_id, the user_id and channel, a role (user / assistant / system / tool), the content, a created_at timestamp, and metadata such as token counts, model and version, tool invocations, and safety scores. A conversations header row holds conversation-level facts; the messages rows hold the ordered turns that reference it. The cardinal rule: transcripts are append-only — a correction is a new event, never an in-place mutation. A frozen=True dataclass makes a TranscriptTurn immutable, so the only way to change history is to append another turn.

Figure 10.5: Transcript data model — a conversation header and its ordered messages

The full transcript routinely exceeds an LLM's context window — the bounded set of tokens the model accepts per call. The gateway cannot concatenate everything; it assembles a budgeted view fresh each turn. Three strategies, often combined:

1. Recency windowing — include the last N turns verbatim, newest-first, stopping when the budget is reached.
2. Hierarchical summarization — compress earlier context into a running summary before appending new turns.
3. Retrieval-augmented context — embed each turn and, at inference time, retrieve the most relevant past turns rather than the most recent.

Turns dropped from the view are not lost — they remain in the durable transcript, and in a fuller implementation are exactly the ones fed to a summarization pass that updates the running summary. The principle bears repeating: the transcript is the complete truth on disk, and what reaches the model is a carefully budgeted view of it, assembled anew each turn.

Figure 10.6: Building a bounded context window from the durable transcript

Durable transcripts trigger obligations ephemeral sessions never do. Under GDPR there is no single correct retention policy period — it depends on the purpose of processing — but regulators expect you to justify your choice and honor deletion requests. The practical mechanisms are declarative retention rules and tiered storage: keep recent transcripts hot, move them to cold storage after roughly 30 days, and purge after the retention window closes, all automatically. A gateway stamps each conversation with a retention class that drives this lifecycle. Because transcripts drive decisions, they must support audit: log every access and every retention action (deletions, exports) in comprehensive, immutable logs kept separately from the transcript. The append-only discipline makes the record tamper-evident by construction; the strongest research systems add Merkle-root hash chains, but most gateways simply treat transcripts as append-only and log all reads, exports, and deletions.

Matching Storage to Need

We close by turning the two-kinds-of-state framework into a backend-selection discipline. The consensus architecture is a two-tier pattern: a fast cache for active session data plus a persistent store for long-term history. The Microsoft Agent Framework encodes exactly this split — local session state in memory for services that need no server-side persistence, versus service-managed or custom durable storage, with the session merely holding a pointer (a database key) into the durable record.

Figure 10.2: Storage-backend selection by state kind

One guardrail deserves emphasis for a stateless gateway: a shared history-provider instance must hold no per-session state; session-specific values like the database key live in the session object, not the provider. This is the same statelessness discipline from the first section, applied to the storage layer itself.

Each data type carries different consistency and durability expectations, and the backend must match them. Ephemeral session writes tolerate last-write-wins and can be lost on eviction, because the session is regenerable from the transcript. The durable transcript must never lose a write and is append-only. To resume a conversation cleanly after a worker restart, persist the full serialized session — not just the message text — then rehydrate it; a separate audit/eval history provider captures enriched I/O without affecting the primary context-building path, acknowledging that the audit copy and the context copy are different concerns.

Finally, serialization and schema evolution matter because both stores hold structured data that changes shape over time. JSON (or JSONB in Postgres) is the pragmatic default — human-readable, widely supported, forgiving of added fields. Plan for evolution: include a version field on serialized payloads, make new fields optional so old records still deserialize, and treat a format change as a migration rather than a silent break. The transcript, being the long-lived system of record, is where schema drift hurts most and where forward-compatible serialization pays off.