Study Guide: Managing Long Histories: Truncation, Summarization, and Compaction

Pre-Reading Check — Truncation Strategies

1. What is the defining behavior of a sliding-window truncation strategy?

It compresses older turns into a running summary before dropping them. It keeps a fixed-size buffer of the last k turns, with new turns pushing the oldest ones out of reach. It restricts the model's internal attention to a local window of tokens. It always deletes the system prompt first to save the most tokens.

2. Why does every serious implementation treat truncation as selective rather than blindly dropping the oldest messages?

Because deleting the oldest message would delete the system prompt, making the assistant forget who it is. Because selective truncation is cheaper than blind truncation. Because models cannot process messages that are deleted out of order. Because blind truncation always exceeds the hard context limit.

3. What is the difference between a hard limit and a soft limit in history management?

The hard limit is measured in messages; the soft limit is measured in tokens. The hard limit is the model's absolute context-window size that causes overflow if exceeded; the soft limit is a self-imposed threshold below it where you begin managing history. The hard limit applies to system prompts; the soft limit applies to user turns. They are two names for the same context-window boundary.

Truncation Strategies

Key Points

Truncation removes older content to make room for new content — the bluntest, cheapest tool, with no LLM call required.
The sliding window keeps a fixed-size buffer of the last k turns, giving predictable token usage and cost, but loses distant history entirely.
Truncation must be selective: always protect the system prompt (front) and recent turns (back); eat only the middle.
The "lost-in-the-middle" effect means models attend best to content at the start and end — so protecting both ends also positions content well.
Measure by token count, not message count, and act at a soft limit before hitting the hard context-overflow boundary.

Truncation is the bluntest instrument in the toolkit and, precisely because it is blunt, the one every engineer reaches for first. It means removing older content from the conversation history to make room for new content. There is no cleverness involved — you simply delete messages until history fits the budget. The subtlety lies entirely in what you delete and how you measure when to stop.

The simplest strategy, drop-oldest, sends only the last N messages and discards everything before them. A more disciplined version, the sliding window, maintains a fixed-size buffer that advances as the conversation progresses. A useful analogy is the conveyor-belt sushi bar: plates ride past on a belt and you can only reach the few in front of you; as new plates arrive, older ones roll out of reach and eventually leave the room. This gives two great virtues — predictable token usage and predictable cost — but its fatal flaw is that it loses early context entirely. A dietary restriction mentioned in turn 2 is gone by turn 15.

Animation slot: a conveyor belt of message "plates" — new turns slide into the fixed window on the right while the oldest turn falls off the left edge into a "Discarded" bin.

Figure 3.1: Sliding-window truncation — new turns enter the fixed-size buffer while the oldest turns fall off the end.

flowchart LR New["New turn arrives"] --> Window subgraph Window["Fixed-size window (last k turns)"] direction LR T3["Turn n-2"] --> T2["Turn n-1"] --> T1["Turn n (newest)"] end Old["Turn n-k (oldest)"] -.->|"pushed out of reach"| Dropped["Discarded — gone forever"] Window --> Old

Naive drop-oldest has an obvious bug: the very first message you delete is the system prompt — the instructions that define your assistant's personality, rules, and role. The standard pattern therefore separates must-have content (current user message, core instructions, system prompt) from optional content (prior history). Must-haves are always included; optional history is appended only if space remains. This also plays to the "lost-in-the-middle" finding: models perform worse on information buried in the middle than at the very beginning or end, so protecting both ends positions your important content where the model attends best.

Finally, the more robust approach truncates by token count, not message count. A "keep the last 20 messages" rule is fragile because one turn might be a two-word "yes" and the next a 4,000-token stack trace. Two thresholds matter: the hard limit is the model's absolute context size (exceed it and you get context overflow — a failed request), while the soft limit is a self-imposed threshold below it where you begin managing history. It is the fuel gauge's warning light versus running the tank dry on the highway.

Key Takeaway

Truncation is cheap and predictable but silently loses early instructions, preferences, and decisions — inadequate for multi-session, personalized conversations. Always protect the system prompt and recent turns, measure by tokens, and manage at a soft limit below the hard overflow boundary.

Review Check — Truncation Strategies

1. What is the defining behavior of a sliding-window truncation strategy?

2. Why does every serious implementation treat truncation as selective rather than blindly dropping the oldest messages?

3. What is the difference between a hard limit and a soft limit in history management?

Pre-Reading Check — Summarization of History

4. What is the Markovian property that makes recursive summarization efficient?

The memory of each session depends only on the current session plus the previous memory, never on the full raw history. Each summary must be re-derived from the entire raw transcript every turn. Older messages are always kept verbatim while recent ones are summarized. The summary length grows linearly with the number of sessions.

5. In salience-driven memory formation, which content is most important to preserve for a hyper-personalized assistant?

The most recent small talk, since it reflects the current mood. Durable user facts, open commitments, and decisions already made. Every message, compressed uniformly to the same ratio. Only the system prompt, since it defines behavior.

6. What is "summary drift," and why is it a consequence of the Markovian property?

It is when a summary omits fields inconsistently because it uses free text instead of a schema. It is when small errors compound across iterations because each new memory is generated from the previous summary rather than the raw source. It is when the summary grows too long and exceeds the token budget. It is when the model refuses to summarize recent verbatim turns.

Summarization of History

Key Points

Summarization compresses older messages into a compact digest instead of deleting them, and empirically outperforms sliding-window baselines for continuity.
A rolling summary keeps recent turns verbatim and folds older ones into a running digest, triggered on a token threshold (e.g., "summarize older than 20 messages, keep the last 10 verbatim").
Recursive summarization updates a single memory from (previous memory + current session), giving it a Markovian property — it never re-reads raw history.
Good summarization is salience-driven: keep durable facts, compress resolved small talk, using working / episodic / semantic memory tiers.
The two failure modes are fact loss (details dropped in compression) and summary drift (errors compounding like a game of telephone).

Truncation's core problem is that deleted context is gone forever. Summarization attacks that directly: instead of throwing older messages away, you compress them into a digest the model can still reference. You trade a little cost (an extra LLM call) and accept some lossy compression in exchange for keeping the gist.

A rolling summary maintains a continuously updated digest of older messages, keeping the most recent turns verbatim and compressing the rest when token counts exceed a limit. The canonical implementation is LangChain's ConversationSummaryBufferMemory: recent messages stay verbatim in a buffer, and when the buffer exceeds max_token_limit, the oldest messages are folded into a running summary. Recursive summarization goes further — it iteratively updates a single memory from (previous memory + current session), so if earlier memory captured "enjoys walking" and a new session mentions the gym, the integration yields "enjoys walking; recently joined a gym." Because memory depends only on the current session and the previous memory, it has a Markovian property that keeps it cheap. Think of it as a running journal: each night you read yesterday's summary, add today's events, and write a new consolidated entry — growing in coverage while staying constant in length.

Figure 3.2: The recursive-summarization loop — memory is updated from the previous memory plus the current session, never from raw history.

flowchart TD Init["Initialize memory: empty"] --> Session["New completed session content"] Session --> Update{"Update memory: previous memory + current session"} Update --> Memory["Updated memory (compact)"] Memory --> Respond["Generate response: recent memory + current dialogue"] Memory -.->|"feeds back as previous memory (Markovian)"| Update Respond --> Session

Animation slot: a "telephone game" demo — a fact passes through successive summaries, each step subtly distorting it, to visualize summary drift versus a verbatim-preserved copy.

The heart of good summarization is deciding what deserves verbatim treatment. This is salience — the property of being important enough to remember. Rather than compressing everything uniformly, salience-driven systems identify the specific facts, preferences, and patterns worth keeping. For a personalized assistant the salient content is the personalization payload: durable facts ("vegetarian," "prefers metric units"), open commitments, and decisions already made. Production systems formalize this with a multi-level hierarchy — immediate working memory (current session), episodic memory (important past moments), and semantic memory (general facts).

Summarization has two characteristic failure modes. Fact loss is when details vanish in compression — "discussed travel plans" has silently discarded the date, destination, and budget. Summary drift is subtler: because each new memory is generated from the previous summary, small errors compound over iterations like a game of telephone, leaving the assistant confidently wrong. The defenses are twofold: keep recent turns verbatim, and extract truly critical facts into a stable semantic-memory store rather than trusting them to survive repeated summarization.

Key Takeaway

Summarization preserves continuity that truncation destroys and beats sliding windows — but is lossy. Design summaries to be salience-driven: keep recent turns verbatim, aggressively compress resolved small talk, and pull durable facts into a stable store to guard against fact loss and summary drift.

Review Check — Summarization of History

4. What is the Markovian property that makes recursive summarization efficient?

5. In salience-driven memory formation, which content is most important to preserve for a hyper-personalized assistant?

6. What is "summary drift," and why is it a consequence of the Markovian property?

Pre-Reading Check — Context Compaction

7. What makes context compaction different from ordinary chat summarization?

It is triggered manually and never automatically. It is automatic, threshold-triggered summarization whose defining job is to preserve task state across a compression event. It deletes the system prompt to free the most space. It only ever applies to short, naturally bounded chatbots.

8. What does setting pause_after_compaction: true allow you to do?

It disables compaction entirely so no summary is generated. It halts execution after the summary is generated so you can manually preserve (pin) specific recent messages before continuing. It raises the token threshold above the model's hard limit. It converts a free-text summary into a structured one automatically.

9. Why is a structured summary generally more reliable than a free-text summary for compaction?

Because a schema reminds the summarizer what categories of information must be captured every time. Because free-text summaries cannot be generated by an LLM. Because structured summaries never lose any tokens at all. Because free-text summaries always exceed the token threshold.

Context Compaction

Key Points

Context compaction is summarization applied at scale to agentic workflows — automatic, token-threshold-triggered, and designed to preserve task state.
Claude's server-side compaction activates at a configurable threshold (default 150,000 tokens, minimum 50,000), replacing pre-compaction blocks with a generated summary in a compaction block.
The defining requirement is preserving task state and tool-related decisions needed to continue the work.
Pinning (via pause_after_compaction) and on-disk checkpoints (like CLAUDE.md, TODO.md) protect state a compression event would otherwise erase.
A structured summary (schema of open tasks, files modified, decisions) survives more reliably than a free-text paragraph.

Context compaction is summarization applied at scale to agentic workflows. Agentic tools like Claude Code accumulate an enormous amount into a single continuous prompt: files read, commands executed, grep results, diffs, and error messages all pile up. Without intervention the assistant hits context overflow — crashing or going "off the rails" as its instructions get buried. Compaction is the automatic release valve.

Compaction fires on a token threshold. Claude's server-side compaction activates when input tokens reach a configured threshold — default 150,000 tokens, minimum 50,000 — then generates a summary, creates a compaction block containing it, and removes all content blocks before that block. The Claude Code CLI exposes the user-facing /compact command, which fires automatically at roughly 95% of the window limit. One billing subtlety: compaction adds a sampling iteration to your bill (visible in the usage.iterations array), and pairing it with prompt caching recovers much of that cost.

Figure 3.3: The compaction-trigger decision flow — the API compacts only when input tokens cross the configured threshold.

flowchart TD Start["Incoming request"] --> Check{"Input tokens >= threshold? (default 150K, min 50K)"} Check -->|"No"| Continue["Continue normally — no compaction"] Check -->|"Yes"| Summarize["Generate summary of older conversation"] Summarize --> Block["Create compaction block with summary"] Block --> Remove["Remove all content blocks before compaction block"] Remove --> Resume["Continue response with compacted context"]

The defining requirement of compaction — the thing that separates it from ordinary chat summarization — is that it must preserve task state. If an agent is halfway through refactoring a module and compaction wipes out which files it already changed and why, it will flail afterward. Two mechanisms guard against this. Pinning via pause_after_compaction halts execution after the summary is generated, letting you manually preserve specific recent messages (e.g., the last ~3) before continuing. On-disk checkpoints persist critical state outside the conversation: Claude Code's CLAUDE.md survives every compaction because it is reloaded from disk as part of the system prompt. The lesson is concrete — move critical rules into a file that reloads after compaction, and create checkpoints like TODO.md so the agent can re-read its own progress. Think of compaction as a mountaineer's base camp: you record your altitude, route, and next objective on a card, then leave the rest behind.

Animation slot: a token gauge filling toward the 150K threshold; when it crosses, pre-compaction blocks collapse into a single "compaction block" while a pinned recent-messages tail and an on-disk CLAUDE.md icon stay intact.

Compaction summaries come in two flavors. A free-text summary is a natural-language paragraph — flexible but prone to omitting fields inconsistently. A structured summary imposes a schema: distinct slots for open tasks, files modified, decisions made, and user preferences. Structure is what makes state survive, because a schema reminds the summarizer what to capture every time. You steer this via the instructions field (or Claude Code's compactPrompt); structured-distillation approaches report roughly 11x token reduction while preserving retrievable information.

Key Takeaway

Compaction is automatic, threshold-triggered summarization for long agentic sessions; its whole job is to preserve task state. Trigger it at a soft token threshold, pin the most recent messages, prefer structured over free-text summaries, and persist critical rules to disk so they survive every compaction.

Review Check — Context Compaction

7. What makes context compaction different from ordinary chat summarization?

8. What does setting pause_after_compaction: true allow you to do?

9. Why is a structured summary generally more reliable than a free-text summary for compaction?

Pre-Reading Check — Hybrid Strategies in Practice

10. In the "verbatim recent + summarized distant" hybrid, how do the memory tiers map to the conversation?

Recent turns are summarized, while distant turns are kept verbatim. Recent turns stay verbatim (working memory), distant turns become compressed summaries (episodic), and durable facts are extracted to long-term storage (semantic). All turns are compressed equally and stored in a single flat buffer. Only the system prompt is kept; everything else is discarded.

11. In a personalized assistant, which content are the natural candidates for pinning?

Resolved sub-tasks and routine small talk. Allergies, accessibility needs, hard constraints, and explicit user instructions. The most token-heavy messages, regardless of content. Only messages from the current turn.

12. Why should you track dropped-segments metadata alongside per-call token counts?

Because token counts alone reveal what content was dropped and when, making loss visible rather than silent. Because recording what you dropped and when reveals why quality issues emerge — e.g., a spike in user re-clarifications correlating with a compaction event. Because dropped-segments metadata reduces the total token cost of a request. Because it lets you exceed the hard context limit safely.

Hybrid Strategies in Practice

Key Points

No single strategy wins outright, so production systems almost always combine several.
The workhorse hybrid is verbatim recent + summarized distant, mirroring the tiered working / episodic / semantic memory model.
Pinning marks specific messages as protected so they survive both truncation and compaction — the mechanism behind salience-based selection.
Natural pinning candidates are personalization primitives: allergies, accessibility needs, hard constraints, and explicit instructions. When in doubt, pin it.
Because every strategy discards information, measure the loss: track dropped-segments metadata and verify effective tokens with counting tooling.

No single strategy wins outright, which is why production systems almost always combine several. Before assembling a hybrid, it helps to see the four core strategies side by side.

Strategy	How it works	Cost	Fidelity	Best for	Key weakness
Truncation (drop-oldest)	Delete oldest messages until history fits	Cheapest (no LLM call)	Lowest — early context lost	Simple, short interactions	Silently loses early instructions and preferences
Sliding window	Fixed-size buffer of last k turns advances forward	Cheap, very predictable	Low — distant history gone	Naturally bounded conversations	No memory of distant turns
Summarization	Compress older turns into a running digest; keep recent verbatim	Moderate (extra LLM call)	Medium-high — gist preserved	Long personalized chat	Fact loss and summary drift
Compaction	Threshold-triggered structured summary that preserves task state	Moderate (adds an iteration)	Medium-high, task-state focused	Long agentic sessions with tools	Can drop reasoning/snippets unless pinned

The workhorse hybrid — and the pattern to reach for by default — is verbatim recent plus summarized distant. Recent exchanges stay raw; older content is folded into summary form; and the very oldest, most durable facts may be extracted into a long-term store. This mirrors the tiered memory model directly: short-term verbatim history (working), medium-term compressed summaries (episodic), and long-term extracted facts (semantic). The tiers are complementary — verbatim recency keeps the immediate conversation coherent, while summaries and extracted facts give access to older information without paying full token cost.

Figure 3.4: The hybrid tiered-memory layout — verbatim recent turns, compressed distant history, and extracted durable facts assembled into a single prompt.

flowchart TD History["Full conversation history"] --> Recent["Recent turns: kept verbatim (working memory)"] History --> Distant["Distant turns: compressed summary (episodic memory)"] History --> Facts["Durable facts extracted to long-term store (semantic memory)"] Recent --> Prompt["Assembled prompt within token budget"] Distant --> Prompt Facts --> Prompt

Animation slot: three colored memory tiers (verbatim / summarized / extracted facts) flowing into one assembled prompt bar, with a "pin" icon locking an allergy fact so it never enters the compressible stream.

Hybrids need a way to say "this message is too important to compress." That is pinning — marking specific messages as protected so they always survive truncation and compaction. It separates must-have content (current message, core rules, pinned facts) from optional history, always including the former. More sophisticated systems perform dynamic context selection, using keyword matching or semantic-similarity scoring to surface relevant history. In a personalized assistant the natural candidates for pinning are the personalization primitives: allergies, accessibility needs, hard constraints ("never email my boss"), and explicit instructions. When in doubt whether a fact is safe to compress, pin it.

Finally, you cannot manage what you do not measure, and every strategy here deliberately throws information away. Two practices make that loss visible: track dropped-segments metadata alongside token counts (a spike in re-clarifications like "I already told you..." correlating with a compaction event signals lost salience), and use token-counting tooling (e.g., /messages/count_tokens) to confirm your soft-limit logic behaves as intended. Choosing among strategies is ultimately a decision tree: bounded conversation → sliding window; long personalized chat → rolling summarization; long agentic task → compaction with pinning and checkpoints; and when relevant history grows huge, reach for retrieval (RAG).

Key Takeaway

Real systems are hybrids: keep recent turns verbatim, summarize the distant past, extract durable facts into long-term memory, and pin the critical personalization facts you cannot afford to lose. Instrument the pipeline with dropped-segments metadata and token counts so information loss becomes a metric you watch, not a surprise you discover from angry users.

Review Check — Hybrid Strategies in Practice

10. In the "verbatim recent + summarized distant" hybrid, how do the memory tiers map to the conversation?

11. In a personalized assistant, which content are the natural candidates for pinning?

12. Why should you track dropped-segments metadata alongside per-call token counts?

Chapter 3: Managing Long Histories: Truncation, Summarization, and Compaction

Learning Objectives

Truncation Strategies

Key Points

Key Takeaway

Summarization of History

Key Points

Key Takeaway

Context Compaction

Key Points

Key Takeaway

Hybrid Strategies in Practice

Key Points

Key Takeaway

Your Progress

Answer Explanations