Chapter 2: Tokens and the Context Window

1. Tokenization Fundamentals

A language model does not read letters or words — it reads sequences of integers. The tokenizer converts human text into those integers (and back), and the integer-mapped pieces it produces are tokens. Tokenization is the very first step in every request: text is split into tokens (whole words, word fragments, single characters, or bytes) and each is mapped to an ID in a fixed vocabulary.

Whole-word vocabularies would be enormous and still fail on typos, brand names, code, and emoji; raw characters make sequences painfully long. Modern models take a middle path — subword tokenization — and the dominant algorithm is byte-pair encoding. BPE starts from the smallest units and repeats one move: find the most frequent adjacent pair and merge it into a new unit, adding one vocabulary entry each time, until the vocabulary hits a target size. The effect is compression.

OpenAI's tiktoken implements byte-level BPE: because it operates on raw bytes, it is universal — any text produces a valid encoding, with no "unknown character." The catch: non-ASCII characters often span multiple bytes and therefore multiple tokens, so non-English or symbol-dense text tends to cost more tokens than the same meaning in plain English.

Tokens vs. Words vs. Characters

Because a token is neither a word nor a character, engineers estimate with rough ratios. The exact ratio depends on content: conversational text runs lower, while technical writing and especially source code push counts higher (punctuation, rare identifiers, and symbols fragment into many tokens). Vocabulary size also shifts the math: GPT-4/GPT-3.5-Turbo use cl100k_base (~100K vocab), while GPT-4o uses o200k_base (~200K vocab), so the same sentence may tokenize to fewer tokens under the larger vocabulary.

Counting Tokens Before You Send

Estimating is fine for a napkin sketch; production systems count exactly. Counting matters because tokens determine two things you cannot guess about: billing (you pay per token) and the context window (a hard maximum per request). The essential move is to select the encoding that matches your target model — counting with the wrong encoding gives the wrong answer.

2. The Context Window

If a token is the unit, the context window is the container — the total number of tokens a model can consider in a single request. It covers everything: system prompt, few-shot examples, tool definitions, user input, conversation history, retrieved documents, and the model's response. It is one shared token budget.

The single most overlooked fact: the window is shared between input and output. If your model has a 128K window and your prompt uses 120K, the model can generate only ~8K in response. The parameter max output tokens (often max_tokens) is how you explicitly reserve part of the window for the reply. Fail to reserve enough and generation is simply cut off when the window fills — the assistant stops mid-sentence, mid-list, or mid-JSON.

Model Context Limits Across Providers

For a hyper-personalized assistant, the 128K tier is the sweet spot: it holds a rich system prompt, a page of user memory, tool definitions, and a healthy stretch of history, with room reserved for a substantial reply. But the headline number is not what you build against — models advertising 200K windows show measurable quality degradation around 130K tokens. There are two costs, neither of which crashes: money (every token is billed, and agents make many calls carrying full context) and quality (examined next). The advertised window is a ceiling, not a target.

3. Building a Token Budget

Knowing the window is shared, the engineer's job is allocation — deciding in advance how many tokens each part may consume. Budgeting starts from the end: reserve the response first. If your assistant sometimes produces a 1,500-token summary, reserve at least that plus a margin as max output tokens, and everything else fits in what remains.

With output reserved, distribute the rest across the input zones. Because history is the zone that grows, it is the one you actively manage — via summarization (compress older turns), priority-based eviction (drop the oldest/least relevant first), and staged loading (bring detailed guidance into context only when a matching task appears). A budget summing to exactly 100% will overflow, so leave headroom and set your effective window below the maximum.

Worked Example: A Budget on a 128K Model

Advertised window: 128,000 tokens. Effective operating budget: 110,000 tokens (~18K top-level headroom against quality degradation). Allocate within 110K:

Note the two-layer safety design: the ~18K gap between the 128K window and the 110K effective budget guards against quality degradation, while the 3,500-token margin inside the budget guards against input variance. The response is reserved before anything else; the growing zone is the one you manage; you never spend to the ceiling.

4. Context Quality Effects

A larger window does not guarantee better quality; how you order and place content matters just as much as whether it fits. The best-documented failure is lost in the middle: models support million-token windows yet struggle to use information located in the middle of long contexts — a direct problem for RAG systems that depend on the model reading retrieved documents.

The foundational study is Liu et al. (2024), "Lost in the Middle," in TACL. Testing multi-document QA and key-value retrieval, they repositioned the answer-containing document among distractors and measured a 30%+ accuracy drop when it moved from position 1 to position 10 in a 20-document context — showing models do not robustly use information across all positions.

The pattern is a U-shaped performance curve: highest accuracy when relevant info is at the beginning (primacy) or end (recency), sagging in the center. The mechanism — shown by MIT and Google Cloud AI researchers — is a U-shaped attention bias: start and end tokens receive disproportionately strong attention, the middle receives less.

Signal-to-Noise in a Full Window

Because performance is best at the edges and worst in the middle, the most effective approach places the highest-ranked content at the beginning and end, leaving lower-ranked material in the middle. With ten ranked documents, you do not concatenate top-to-bottom — you interleave so your two best sit at the two edges. Every irrelevant token is noise: it dilutes the signal and pushes important content into the low-attention middle. For a personalized assistant, retrieving more memories can make answers worse if it buries the single most relevant fact. Curate ruthlessly, rank carefully, place your best at the edges.