Key Points
- LLMs are inherently stateless; short-term memory exists to append prior conversation history to each new call.
- Short-term memory holds three distinct things: turn-by-turn history, working state / open tasks, and a spectrum of ephemeral-to-durable session data.
- Working memory is the agent's scratchpad — intermediate results the next step must consult (e.g., flight results feeding a hotel search).
- Ephemeral data (current page, half-composed draft) should expire; durable-within-session data (history, confirmed results) must survive turns and reconnects.
- The defining trait is that short-term memory resets when the session ends — a feature that keeps sessions clean and bounded.
Short-term memory is not one undifferentiated blob of text; it is a small set of distinct things, each with its own lifetime and purpose. The most obvious content is the conversation history — the ordered record of what the user said and what the assistant replied. Because the LLM is stateless, this transcript is the only reason the assistant can resolve "the same time tomorrow" or "the second one." In production, history is not merely logged; it is reconstituted and prepended to the next model call.
Short-term memory is also working memory: the immediate context needed right now to complete the current task. When an agent processes "Find flights to Paris, then recommend hotels near the Louvre," the flight results become working state that the hotel step must consult. This is more than a transcript — it is a scratchpad of intermediate results, open tasks, and partial plans.
Ephemeral vs. Durable Session Data
| Type | Examples | Treatment |
|---|
| Ephemeral | Current results page, half-composed draft, intermediate reasoning for the current step | Expire quietly when the session ends |
| Durable-within-session | Conversation history, confirmed intermediate results | Survive across turns and reconnects, but still belong to this session |
The defining property of all short-term memory is impermanence: working memory resets when the conversation ends, making it insufficient for learning from past sessions on its own. That reset is a feature — it keeps sessions bounded — but it means anything worth keeping forever must be deliberately promoted out before the reset. Production systems layer memory types: thread-scoped short-term memory for the current session, and cross-session long-term memory for persistent knowledge.
[Animation slot: three labeled bins — "Turn-by-turn history," "Working state / open tasks," "Ephemeral scratch" — with items flowing in during a session, then the ephemeral bin emptying at session end while durable items get flagged for promotion.]
Key Points
- Session state lives on a durability spectrum: process RAM (dev) → local file → production DB → fast external store like Redis.
- Redis is a common production choice for session-scoped memory: fast, supports list structures for history, configurable persistence, and microsecond-latency lookups.
- Every session needs a key that encodes who and which conversation — often a composite
thread_id like customer_{id}_session_{id} (keep under 255 chars for Postgres).
- A TTL expires abandoned sessions automatically — tune it: too short strands the user, too long piles up stale state.
- State reconstruction loads the latest checkpoint for the thread and rebuilds history — not a replay from scratch.
If short-term memory resets at session end but must survive within it, it needs a home outside the stateless model: the session store. The first decision is where state physically lives, a trade-off along a durability spectrum.
| Backend | Where state lives | Durability | Best for |
|---|
| InMemorySaver | Process RAM | Lost when the program stops | Development, prototyping |
| SqliteSaver | Local file | Survives restarts on one machine | Local apps, single-node tools |
| PostgresSaver | Production database | Survives restarts and long periods | Production, multi-node |
| Redis / Couchbase | Fast external store | Configurable persistence + TTL | High-throughput sessions |
Latency compounds across an agent's reasoning steps: if a query fans out into eight tool calls and each fetches and updates session state, a slow store multiplies that slowness eight times over. (A real gotcha: langgraph dev can silently ignore checkpointer config and force in-memory storage — always verify the durability you configured is the durability you get.)
Session Keys and Expiry
A session store is key-value, so every session needs a key encoding who and which conversation — e.g., Redis hashes keyed mysession:UserName:entry_id, or a composite thread_id such as customer_{customer_id}_session_{session_id}. The second essential property is expiry via a TTL. Think of the TTL as the parking meter of memory: while it runs, the space stays reserved; when it expires, the space is freed. Anything you want to keep, you take with you before the meter runs out.
Reconstructing Context on Reconnect
The dominant production pattern is checkpointers plus thread IDs. The checkpointer saves a snapshot of state at every super-step. On reconnect, the system loads the most recent checkpoint for that thread_id and reconstructs the full state — including accumulated message history — so the agent picks up where it left off. Crucially, state is rebuilt from the last checkpoint rather than replayed from scratch.
sequenceDiagram
participant User
participant App as "Assistant App"
participant Store as "Session Store (Checkpointer)"
participant LLM
User->>App: Turn 1 message
App->>LLM: Prompt with history
LLM-->>App: Response
App->>Store: "Write checkpoint (thread_id)"
Note over User,App: Connection drops mid-conversation
User->>App: Reconnect and resume
App->>Store: "Load latest checkpoint for thread_id"
Store-->>App: "Restored state + message history"
User->>App: "book the second hotel"
App->>LLM: "Prompt with rebuilt history"
LLM-->>App: Resolves reference correctly
App-->>User: Continues where it left off
Worked example. Maria books a Boston→Paris trip: the agent writes a checkpoint for customer_maria_session_0708 after finding flights, another after finding hotels. Her train enters a tunnel and the connection drops. Ten minutes later she reopens the app, the system loads the latest checkpoint, rebuilds the full flight-and-hotel context, and Maria simply types "book the second hotel." Without datastore-backed state, the tunnel would have cost her the whole booking.
[Animation slot: durability-spectrum slider from "RAM (volatile)" to "Redis / Postgres (durable)"; then a reconnect timeline showing checkpoints being written per turn, a dropped connection, and the latest checkpoint being loaded to restore state.]
Key Points
- Session memory stores the conversation; the context window constrains how much is consumed each turn. Storage and presentation are separate concerns.
- The sliding-window (buffer) approach keeps the last N turns; its failure mode is losing old-but-relevant turns entirely.
- The central trade-off: keep recent turns verbatim, compress older turns into a running summary — faithful but expensive vs. cheap but lossy.
- Semantic selection (e.g., RedisVL
SemanticSessionManager) surfaces the N most relevant turns via vector similarity, not just the newest.
- Because each request is stateless, the system carries state: load → assemble the window → call the model → write updated state back.
This section connects directly to Chapter 3. Session memory and context management are two sides of one coin: session memory stores the conversation, but the context window constrains how much of it the model can consume on any turn. Because LLMs have token limits, the memory layer stores the last N interactions and "slides" that window into the prompt. The distinction to internalize: the store may hold the complete history (subject to TTL), while the window receives only a curated slice.
A pure sliding window has a blunt failure mode: once a turn slides out of the last N, its information is gone from the prompt, even if the user refers back to it twenty turns later. The remedy is summarization. The practical recipe is a hybrid:
- Recent turns → verbatim. The last several exchanges are kept word-for-word, because exact phrasing matters for references and nuance.
- Older turns → summarized. Aging turns fold into a compact running summary — "Maria is booking a Boston-Paris trip; chose the 9 a.m. flight; wants a hotel near the Louvre under $300/night."
Three Presentation Strategies
| Strategy | What it presents | Strength | Weakness |
|---|
| Verbatim buffer (last N) | Most recent N turns, word-for-word | Perfect fidelity for recent context | Old-but-relevant turns lost entirely |
| Running summary | Compressed digest of older turns + recent verbatim | Large token savings; long horizon | Lossy; can drop specifics |
| Semantic selection | Turns most similar to the current query | Surfaces relevant old turns; token-efficient | Depends on embedding quality; adds a retrieval step |
Whichever strategy you choose, the obligation is the same: because each request is independent, the system carries state itself. On every turn the pipeline (1) loads session state by key, (2) assembles the window — verbatim recent turns plus a summary or semantically selected turns plus working state, (3) calls the model, and (4) writes updated state back as a new checkpoint. The window is a keyhole; the session store is the room behind the door.
[Animation slot: a growing transcript in the "store" panel; a fixed-size "window" keyhole showing recent turns verbatim while older turns collapse into a one-line running summary; toggle to "semantic selection" mode that highlights and pulls forward the turn most similar to the current query.]
Key Points
- The boundary is a real architectural split: within-thread checkpointers for session state, cross-thread Stores for long-term memory.
- Memory promotion moves durable, reusable facts (preferences, identity, recurring goals) from working memory into long-term memory — deduplicated first.
- Promotion happens at session end (as summaries) or continuously as salient facts appear; pathways are automatic extraction or manual creation.
- Two philosophies: Mem0 (passive extraction — predictable, cheap) vs. Letta/MemGPT (self-editing tiers: Core / Recall / Archival — intelligent, adaptive).
- Teardown rule: promote durable facts before letting transient state expire with its TTL; prune old checkpoints to keep reconstruction fast.
In the checkpointer model, checkpointers persist within-thread (session) state; for data shared across threads/sessions — long-term memory — the system provides a separate mechanism: application-defined key-value Stores. This split maps directly onto short-term versus long-term memory.
| Dimension | Short-Term (Session) | Long-Term |
|---|
| Scope | A single conversation / thread | Across all of a user's sessions |
| Lifetime | Resets when session ends (TTL) | Survives restarts; weeks-to-months |
| Persistence mechanism | Checkpointer (within-thread) | Store (cross-thread) |
| Contents | Turn-by-turn history, working state | Preferences, stable facts, past episodes |
| Failure if lost | This conversation resets | The assistant "forgets" the user entirely |
Promoting Facts to Long-Term Memory
Memory promotion moves durable, reusable facts out of ephemeral working memory into persistent long-term memory. It happens two ways in time — at session end (turns become "observations" — factual statements with references back to the dialog turns that produced them) and continuously when a salient fact surfaces. Pathways are automatic (extract-and-batch, e.g. ENABLE_DISCRETE_MEMORY_EXTRACTION=true) or manual (direct API / LLM tool). Extracted memory falls into semantic, episodic, and message types. Promotion must not create duplicates: hash-based plus semantic-similarity deduplication, with related memories merged by an LLM.
flowchart TD
A["Working memory (session)"] --> B{"Durable, reusable fact?"}
B -->|"No: transient detail"| C["Keep in session; expire with TTL"]
B -->|"Yes"| D{"Promotion pathway"}
D -->|Automatic| E["Extract and batch-process from working memory"]
D -->|Manual| F["Create directly via API or LLM tool"]
E --> G{"Duplicate check"}
F --> G
G -->|"Identical or similar exists"| H["Merge and consolidate via LLM"]
G -->|New| I["Write to long-term store"]
H --> I
I --> J["Persistent long-term memory"]
Two philosophies of promotion. Mem0 extracts passively: add() triggers an extraction pipeline that reconciles facts with a vector store via add/update/delete/skip — running outside the agent's reasoning loop, so inference costs stay low and identical inputs yield identical memories (predictable). Letta / MemGPT agents self-edit memory during reasoning, managing three OS-inspired tiers: Core (in-context, like RAM), Recall (searchable history, like a disk cache), and Archival (cold storage via tool calls) — intelligent, but quality depends on the model and it consumes inference tokens. The rule of thumb: promote stable preferences, identity facts, and recurring goals (deduped); keep transient details like "what page am I on" in working memory to expire with the TTL.
Session Teardown
A well-designed teardown does two jobs in sequence: (1) promote before you purge — because working memory resets at session end, the last chance to save durable facts is before teardown; (2) then let session state expire via TTL. Even within a long session, checkpoints accumulate and can raise latency, so implement retention policies or periodic pruning. Like a hotel front desk at checkout: they don't shred your loyalty profile — they update it (you liked the top floor, you order decaf) and then clear the room. Promotion updates the profile; teardown clears the room.
stateDiagram-v2
[*] --> Created: "First message / new thread_id"
Created --> Active: "Checkpoint written"
Active --> Active: "Turn taken; state updated"
Active --> Idle: "User goes quiet"
Idle --> Active: "Reconnect; load latest checkpoint"
Idle --> Expiring: "TTL elapses"
Active --> Expiring: "Session closed"
Expiring --> Teardown: "Promote durable facts first"
Teardown --> [*]: "Purge transient state"
[Animation slot: a fact traveling from the session box across the short-term/long-term boundary — passing through a dedup gate that merges a near-duplicate — landing in the long-term store; meanwhile the session box's transient items fade out as the TTL meter reaches zero.]