Chapter 2: LLM Fundamentals for Claude Developers

1. How LLMs Work

Next-token generation and autoregressive decoding

The single most important idea for a developer is how Claude produces text. Its underlying model is autoregressive: during pretraining it learns "to predict the next word, given the previous context of text." It repeatedly predicts the most likely next unit of text, appends that prediction to the running context, and predicts again — one unit at a time, looping until it decides to stop. "Autoregressive" means regressing on itself: each new piece of output becomes part of the input for the next step.

Analogy — the world's most well-read autocomplete. Like predictive text on your phone, but at vastly larger scale: instead of one suggested word, Claude strings its own predictions together continuously, feeding each prediction back in to predict the next unit.

To avoid nonsense, Claude uses top-p sampling (nucleus sampling): only tokens whose cumulative probability reaches a threshold (~0.99–0.999) are candidates, and it samples from that "nucleus," discarding the long tail. A raw pretrained model "is not inherently good at answering questions or following instructions," so Claude is fine-tuned with RLHF to behave as a helpful assistant aligned to the HHH triad.

Figure 2.1: The autoregressive next-token generation loop

Tokens and tokenization

Tokens are "the smallest individual units of a language model" and "can correspond to words, subwords, characters, or even bytes." Text is encoded into tokens before processing; the model reasons entirely in tokens; output is decoded back to text. For Claude, a token approximately represents 3.5 English characters, though this varies by language.

Analogy — LEGO bricks for language. Common words like "the" are single large bricks; a rare word is built from several smaller subword or character bricks. This is why token counts don't map cleanly to word counts.

Figure 2.2: The tokenization pipeline — text in, text out

A cost-relevant change: Opus 4.7+ Opus models, Fable 5, Mythos 5, Mythos Preview, and Sonnet 5 use a newer tokenizer that produces ~30% more tokens for the same text. Do not reuse token counts from older models — recount against the specific model you plan to use. The free token-counting API (POST /v1/messages/count_tokens) returns input tokens before you send, e.g. { "input_tokens": 14 }.

Non-determinism and why outputs vary

Because generation samples, running the same prompt twice can produce different text. Even at temperature 0, "the results will not be fully deterministic and identical inputs may produce different outputs across API calls" — on both first-party and third-party inference. Adaptive thinking adds further variation, since thinking-token usage is decided per request. The takeaway is a design principle: do not assume byte-identical outputs; validate structure and meaning, not exact strings.

2. Context Windows and Sampling

Context window size and limits

The context window is "all the text a language model can reference when generating a response, including the response itself" — a working memory distinct from the training corpus. Everything competes for space: system prompt, every message (tool results, images, documents), tool definitions, and the generated output including extended thinking. Prompt caching changes what you pay, not whether tokens occupy the window.

Analogy — a whiteboard, not a library. Training is a career's worth of knowledge in your head; the context window is the fixed-size whiteboard in front of you during one meeting. When it fills, something must give.

For every 1M-window model, 1M is the default — no beta header required, billed at standard pricing. Fable 5 and Mythos 5 can generate up to 128K output tokens. A single request can include up to 600 images or PDF pages (100 for 200K-window models).

A larger window is not automatically better. Anthropic warns of context rot: as token count grows, "accuracy and recall degrade." A focused 20K-token prompt often outperforms a bloated 500K-token one.

Modern Sonnet/Haiku models track their remaining budget automatically, injecting tags like <budget:token_budget>200000</budget:token_budget> and post-tool updates like <system_warning>Token usage: 35000/200000; 165000 remaining</system_warning>. Developers never send these. Server-side compaction (beta) and context editing help conversations run past the limit.

Temperature, top-p, and sampling parameters

Temperature controls the randomness of predictions. Higher → more creative/diverse; lower → more conservative/deterministic. top_p (nucleus) and top_k limit which candidates are eligible, before temperature reshapes the odds among them.

Analogy — the creativity dial. Temperature is a dial on how much weight the model gives lower-ranked candidates. Down toward 0 → safe and repetitive; up → creative but prone to wandering.

The critical shift: Fable 5, Mythos 5, Mythos Preview, Opus 4.8, Opus 4.7, and Sonnet 5 reject non-default temperature, top_p, and top_k with a 400 error — on every request. Omit them and steer via prompting and the effort parameter. Porting code that hard-codes temperature: 0.7 onto Sonnet 5 will fail.

Deterministic vs. probabilistic behavior

"Temperature 0" is not "deterministic." Sampling parameters shape the shape of the distribution (peaked vs. spread) but don't eliminate underlying non-determinism. Lower temperature is more consistent, not guaranteed identical. On the newest models, the lever for consistency is a clear, well-structured prompt with examples.

3. Model Thinking Options

Fast mode and extended thinking

By default Claude answers directly. For hard problems, extended thinking generates internal reasoning — a private scratchpad — before committing to an answer. Historically it was manual: thinking: {type:"enabled", budget_tokens:N}, where budget_tokens is a subset of max_tokens, billed as output, and counts toward rate limits and the context window. Manual thinking is deprecated on Opus 4.6 and Sonnet 4.6 and rejected with a 400 on Fable 5, Mythos 5, Sonnet 5, Opus 4.8, and Opus 4.7.

Adaptive thinking and effort levels

The modern replacement is adaptive thinking (thinking: {type:"adaptive"}, no beta header). Claude evaluates each request's complexity and "determines whether and how much to use extended thinking." At the default effort (high) it almost always thinks; at lower effort it may skip thinking. It is recommended on Opus 4.8/4.7/4.6 and Sonnet 5/4.6, and the only mode on Fable 5 and Mythos 5.

Analogy — an experienced triage nurse. Manual budget_tokens gives every patient exactly ten minutes; adaptive thinking sizes up each case and routes the paper cut through fast while giving the broken leg its time. Adaptive also auto-enables interleaved thinking — Claude can think between tool calls.

Figure 2.3: Adaptive thinking — how Claude decides whether to reason

Triggering is promptable ("When in doubt, respond directly" reduces thinking; "Think carefully before responding" increases it). The display field controls visibility only: "summarized" returns a readable summary, "omitted" returns empty thinking blocks. On Fable 5, Mythos 5, Sonnet 5, Opus 4.8/4.7, and Mythos Preview the default is "omitted"; on Opus 4.6/Sonnet 4.6 it is "summarized". Display affects visibility, not billing — thinking is billed identically either way.

The effort parameter (output_config: {effort:"..."}) controls "how eager Claude is about spending tokens." It affects all tokens (text, tool calls/arguments, thinking), needs no thinking enabled, and influences the number of tool calls. effort: "high" is the default and equals omitting the parameter.

When thinking improves reasoning quality

Thinking is a tool, not a default virtue — it adds latency and cost. Use adaptive thinking for agentic behavior (multi-step tool use, complex coding, long-horizon loops); use higher effort for complex reasoning where quality outweighs speed; use lower effort for simple, high-volume, latency-sensitive tasks. Think of max_tokens as the hard cap and effort as the soft dial. Debugging rule: if you see stop_reason: "max_tokens", either raise max_tokens or lower effort.

4. Fundamental Prompting Techniques

Zero-shot prompting

In zero-shot prompting you provide no examples — only role, task description, and expected format. Success depends entirely on clear, direct, detailed instructions.

You are a support-ticket classifier. Classify each ticket into exactly one
category: BILLING, TECHNICAL, or ACCOUNT. Respond with only the category name.

Ticket: "My invoice shows a charge I don't recognize from last month."

Fast to write and cheap in tokens; for unambiguous tasks it is often all you need.

Single-shot and multi-shot (few-shot) prompting

Single-shot adds exactly one example; multi-shot / few-shot provides multiple. "A few well-crafted examples can dramatically improve accuracy and consistency."

Analogy — showing versus telling. Zero-shot hands a new hire a written policy and hopes; few-shot shows three completed examples and says "do it like these." Most people — and models — generalize better from concrete demonstrations.

You are a support-ticket classifier. Classify each ticket into exactly one
category: BILLING, TECHNICAL, or ACCOUNT. Respond with only the category name.

<examples>
<example>
Ticket: "I was double-charged for my subscription this month."
Category: BILLING
</example>
<example>
Ticket: "The app crashes every time I upload a photo."
Category: TECHNICAL
</example>
<example>
Ticket: "I need to change the email address on my profile."
Category: ACCOUNT
</example>
</examples>

Ticket: "My invoice shows a charge I don't recognize from last month."

Sanity check: "Show your prompt to a colleague with minimal context. If they'd be confused, Claude will be too."

Choosing the right technique

Sound default workflow: start zero-shot; if output is inconsistent or misreads edge cases, add examples — single-shot, then a diverse 3–5 multi-shot set. Because sampling parameters are unavailable on the newest models, few-shot is often the first tool to reach for when tightening reliability.