Chapter 13: Cost Control: Caching, Rate Limits, and Production Economics

Caching

Caching is the single highest-leverage cost control available to an assistant builder, because personalized prompts are mostly repetition. The system prompt, the tool definitions, the formatting instructions, the retrieved knowledge-base article — these bytes are identical across thousands of requests. Caching lets you pay full price to process them once and a steep discount to reuse them thereafter.

Prompt / Context Caching Mechanics

Prompt caching (also called context caching) stores the model's internal computation for a portion of a prompt so that subsequent requests reusing that portion skip the recomputation. The model loads the already-processed representation and continues from there rather than re-reading cached tokens from scratch.

Anthropic (Claude) uses explicit caching: you mark cacheable content with cache_control breakpoints. Cache writes cost more than a normal input token (1.25x for a 5-minute TTL, or 2.0x for a 1-hour extended TTL), while cache reads cost only ~10% of the base input price. OpenAI uses automatic caching, routing each request to a server that recently processed the same prefix — no code changes, no extra fee — enabled automatically for any prompt of 1,024 tokens or more, reducing latency up to 80% and input cost up to 90%.

The performance numbers are dramatic: Anthropic reports latency reductions up to 85% for long prompts — in a 100,000-token "book" example, response time dropped from 11.5 s to 2.4 s once caching was enabled. There is also a subtle rate-limit interaction: on Anthropic's current models, cache_read_input_tokens are excluded from the input-tokens-per-minute (ITPM) limit, so caching buys throughput headroom as well as cost savings.

Analogy: Prompt caching is like a barista who keeps your regular order on a sticky note. The first time, they take your full order slowly (the cache write, which even costs a little extra to write down). Every visit after, they glance at the note and start pulling shots immediately (the cheap, fast cache read). But the note only helps if your order starts the same way every time.

Cache-Friendly Prompt Ordering

Cache hits require an exact prefix match. The provider can only reuse computation up to the first byte that differs between two prompts. OpenAI caches the longest previously-computed prefix, starting at 1,024 tokens and growing in 128-token increments.

Figure 13.1: Prompt-cache hit/miss decision flow — an exact prefix match reads cheaply, otherwise the prefix is written to cache.

The hard ordering rule: put static content first, variable content last. This is exactly the static-first prompt assembly pattern from Chapter 4 — instructions and tools before per-user context before the live turn — which turns out to also be the cache-optimal layout.

A single misplaced variable — a timestamp, a session ID, a greeting with the user's first name at the top — can silently collapse your cache hit rate from 90% to near zero. Anthropic's multiple cache_control breakpoints do not increase cost; OpenAI's optional prompt_cache_key steers requests sharing a long prefix to the same warm server.

Semantic and Response Caching

Prompt caching reuses computation for identical prefixes, but two users often ask the same question in different words — "What's your refund policy?" vs. "How do I get my money back?" A semantic cache closes that gap by comparing meaning: each query is embedded, compared against cached query vectors (usually cosine similarity), and if close enough in meaning, the cached response is served directly — no model call at all. This is a response cache.

Savings run roughly 30–70%. The open-source GPTCache (sentence-transformer embeddings + a vector DB such as Milvus, Weaviate, or FAISS) reduced API calls up to 68.8% with hit rates of 61.6–68.8%. The catch is the similarity threshold: too conservative and you miss safe reuse; too aggressive and you serve a wrong answer to a merely similar-looking query — a correctness failure worse than a cost failure. The remedy is category-aware caching: vary threshold and TTL by query category.

For a personalized assistant this is critical: semantic caching is safest for shared, non-personalized knowledge and dangerous for anything depending on a user's state. "What is my current balance?" and "What is my account balance?" are near-identical semantically but must never share a cached response. Scope semantic caches by tenant and exclude any query whose answer depends on per-user data.

Rate Limits and Throughput

Caching lowers how much you spend and how long each call takes. Rate limits govern how many calls you can make. Cross a limit and the provider rejects your request with an HTTP 429 — and in a personalized assistant under load, hitting limits is not an edge case, it is Tuesday.

Token and Request Limits

A rate limit is a cap on your throughput, and the first surprise is that it is multi-dimensional: RPM (requests/min), TPM (tokens/min), RPD (requests/day), and TPD (tokens/day). Exceeding any one returns a 429. Anthropic further splits tokens into ITPM and OTPM, so a blended "TPM" can mislead you. Throughput — sustained useful work — is constrained by whichever dimension you saturate first: many tiny requests hit RPM; few huge personalized prompts hit TPM long before RPM.

Two facts sharpen this. First, a 429 failure often still consumes quota (for OpenAI, a rejected request still burned a slot and its tokens), so naive retry-immediately logic digs the hole deeper. Second, limits may be finer than "per minute": an RPM of 600 enforced as ~10 requests/second means a short burst can 429 you while technically under the per-minute ceiling.

Recall the caching payoff: because Anthropic excludes cache reads from ITPM, aggressive prompt caching effectively raises your usable throughput on the input dimension without a tier upgrade.

Exponential Backoff and Retries

When a 429 arrives, wait and retry — but how long you wait determines whether you recover gracefully or trigger a cascade. Exponential backoff waits on failure and doubles the wait after each successive failure (1s, 2s, 4s, 8s...). Its flaw: if a thousand clients all fail at once and all wait exactly 1 second, they all retry at the same instant — the thundering herd. The fix is jitter: randomize each wait so retries spread out.

Two refinements separate correct from naive: (1) Honor the retry-after header (both providers return it on 429s; prefer it, fall back to computed backoff only when absent). (2) Cap the backoff near 60 seconds, because providers use rolling 60-second windows and a drained bucket refills over roughly a minute, so waiting longer buys no headroom.

Figure 13.2: Exponential-backoff-with-jitter retry loop — a 429 waits then loops back, honoring retry-after first and capping the computed backoff.

Worked example — exponential backoff with jitter. Trace the no-retry-after path with full jitter, where backoff = min(60, base * 2**attempt) and wait = random(0, backoff):

Because each client draws an independent random wait, two clients that failed on the same millisecond almost certainly retry at different times — the herd disperses — and the 60 s cap ensures even a late attempt never sleeps past the point the bucket has already refilled.

Analogy: Exponential backoff without jitter is like a room full of people who all fall silent, then all start talking again at exactly the same second — instant chaos. Jitter is everyone silently counting to a slightly different number before speaking. And retry-after is the host announcing "give it 8 seconds" — at which point you should just listen instead of guessing.

Queuing and Concurrency Control

Retrying reactively after a 429 is damage control. The better posture is to never emit the request if it would exceed a limit. The core technique is a token bucket that refills at your permitted rate; each request removes a token, and when empty, requests wait. The essential LLM subtlety: one bucket is not enough. You need two buckets in parallel — one metering RPM, one metering TPM — and you must estimate a request's token count before sending it (via anthropic.count_tokens() or tiktoken). This pre-flight measurement prevents the worst failure mode: admitting 500 requests that each satisfy RPM but collectively blow past TPM, so none can proceed and the queue deadlocks.

FIFO queues such as the open-source LLMRateLimiter coordinate TPM and RPM and prevent burst problems, including across distributed workers — because in a multi-process deployment, each worker enforcing its own local limit will collectively overshoot the shared account limit unless they coordinate through a shared store (commonly Redis).

Figure 13.3: Client-side request queue with parallel request and token buckets — pre-flight token estimation gates admission so RPM and TPM must both clear before a request is sent.

Cost Modeling

You cannot control what you cannot count, and you cannot count what you cannot attribute. This section turns raw token counts into an accounting system that tells you not just how much you spent but who and what spent it.

Input vs. Output Token Pricing

The atomic unit of LLM cost is the token, and input and output are priced differently — output is almost always the expensive one, commonly several times the input rate. The base per-request formula is:

cost = (input_tokens × input_price) + (output_tokens × output_price)

Once caching is in play, the input side splits into three distinct rates:

cost = (fresh_input_tokens × input_price) + (cache_write_tokens × input_price × write_multiplier) + (cache_read_tokens × input_price × 0.10) + (output_tokens × output_price)

This decomposition explains several counterintuitive optimizations: a long static prefix is cheap once warm (its tokens shift to the 0.10x read rate), so bloating the static prefix costs far less than bloating the variable suffix; trimming output pays more than trimming input because output is the priciest term; and the cache write is a small up-front tax that breaks even almost immediately for content reused more than once or twice.

Per-User and Per-Feature Cost Accounting

The most expensive mistake in LLM operations: monitoring aggregate spend without attribution answers no actionable question. "We spent $4,200 today" tells you nothing about which user, feature, or tenant to fix. Cost accounting is fundamentally an attribution problem. The solution: pass metadata with every single API request — user_id, and ideally tenant_id, feature, and environment. The critical word is every: cron jobs, internal tools, and background refreshes all cost money. The reliable enforcement is a thin client wrapper at the call site the whole team must use, so no raw untagged call escapes.

Why this matters for a personalized assistant: pricing-tier calibration (price a "Pro" tier only if you know what a Pro user costs), per-user quota enforcement (you cannot cap what you do not measure), and abuse detection (a user whose cost spikes becomes immediately visible).

Worked example — per-user cost accounting. Model priced at $3 / M input ($0.000003/token) and $15 / M output ($0.000015/token; output is 5x input). Cached reads cost 10% of input = $0.0000003/token. Power user Dana runs 200 turns/day; each turn = 1,500-token static prefix (cached after turn 1) + 500 fresh input + 300 output.

Turn 1 (cold cache)

Turns 2–200 (warm cache, ×199)

Warm turns: 199 × $0.006450 = $1.283550. Add Turn 1: Dana's daily cost ≈ $1.295. The same workload without caching bills all 1,500 prefix tokens at full rate every turn: $0.0105/turn × 200 = $2.10/day. Caching cut Dana's cost roughly 38% — and because you attributed cost per user, you can prove it, project it, and detect the day it jumps to $12 because a bad deploy disabled the cache. Note that output (300 tokens, $0.0045/warm turn) is more than input and cache combined — so a concision instruction trimming 100 output tokens saves more than heroic prompt shrinking.

Budgets and Guardrails

Attribution tells you what happened; budgets stop the bad thing from happening. A quota is a hard cap that blocks or throttles once a user, tenant, or feature crosses a threshold. Two placement rules are non-negotiable: (1) Enforce in the proxy/gateway layer, not at the LLM call — by the call the input tokens are already sent and billed, so the guardrail must sit in middleware upstream. (2) Make token budgets primary, request-rate limits secondary — tokens reflect real cost (one long RAG query equals a thousand short chats), while request-rate limits guard against burst/concurrency abuse. You need both.

Hard caps matter most in free tiers, trials, and usage-based products. For multi-tenant assistants, enforce per-tenant quotas at the AI gateway and allocate upstream headroom explicitly: each tenant gets a reserved slice of TPM plus access to a shared burst pool, so no single tenant drains the ceiling and starves others.

Observability and Optimization

Caching, rate handling, and budgets are the controls. Observability is the feedback loop that tells you whether they are working, where money is leaking, and when something just broke.

Metrics, Dashboards, and Alerts

LLM cost monitoring is the continuous measurement, attribution, and optimization of costs from production LLM calls. "Continuous" is the operative word — end-of-month billing reconciliation is autopsy, not medicine. A production setup captures, per request: input/output/cache token counts, computed cost, latency, model version, and attribution metadata (user_id, tenant_id, feature). From these you build dashboards broken down by user, session, feature, geography, and model version.

The single most valuable alert is spike detection: compare the current period's cost to a rolling baseline (e.g., the same hour of day averaged over the past 7 days) and fire when the ratio exceeds roughly 2x, checked about every 5 minutes. This is the primary signal for catching deployment-correlated cost regressions — the ship that accidentally disabled caching, doubled the system prompt, or removed a max_tokens cap shows up within minutes rather than as a shocking invoice weeks later.

Note the recurring gateway pattern: the same proxy that enforces budgets and rate limits is the natural place to observe cost, because every request already flows through it.

Finding Cost Hot Spots

With attributed data flowing, hunting hot spots — the small number of users, features, or prompts responsible for a disproportionate share of spend — becomes a query rather than a guess. LLM cost distributions are typically heavily skewed. A systematic hunt asks, in order:

1. Which users cost the most? A cost-per-user histogram reveals the heavy tail; extreme outliers are candidates for a pricing-tier change or abuse investigation.
2. Which features cost the most per invocation? Rank features by cost — a rarely used feature with a huge per-call cost may dwarf a popular cheap one.
3. Where is the token bloat — input or output? Output-dominated features are a concision/max_tokens opportunity; input-dominated ones are a caching or retrieval-trimming opportunity.
4. What is the cache hit rate? A low hit rate on a supposedly static prefix points straight to a prompt-ordering bug — a variable leaked into the prefix.

Analogy: Finding cost hot spots is triage in an emergency room, not a routine checkup. You find the one bleeding out — the 2% of users or the one feature generating 60% of spend — and fix that first. Attribution lets you take the pulse of each patient instead of the room's average.

Continuous Cost Tuning

Cost control is not a project you finish; it is a loop you run. Every deployment can shift the numbers, so the optimization cadence must match the deployment cadence. The loop pulls together every lever in this chapter: Measure (attributed cost into dashboards) → Detect (spike alerts and hot-spot queries) → Optimize (apply the right lever: cacheable static prefix, semantic cache, output trimming, right-size the model, tighten budgets/quotas) → Verify (confirm the metric moved and quality did not regress).

Figure 13.4: The continuous measure–detect–optimize–verify cost-tuning loop, pairing each optimization lever with a quality check before looping back.

Two guardrails keep the loop honest. First, every optimization is a quality bet: an aggressive semantic-cache threshold, a smaller model, or a harsh max_tokens cap can each degrade answers, so pair cost metrics with quality evaluations. Second, let attribution set priority: because cost is skewed, always tune the biggest hot spot first — a 20% cut on the feature that is 60% of your bill beats a 90% cut on one that is 2%.