Design Patterns and Production: Auth, Idempotency, Retries, Observability, Testing, Cost

1. Auth and Safety for Tool Servers

An agent is a deputy: it acts on behalf of a user, wielding tools that touch real systems. Four defensive layers are needed — authenticating callers, authorizing them with least privilege, defending against injection and confused-deputy risks, and sandboxing dangerous actions behind human review.

Authentication establishes who a caller is, and the transport dictates the mechanism. MCP follows OAuth 2.1 conventions where they apply but does not mandate a single identity system.

Authentication mechanisms by transport

For direct Claude API integrations the caller is your own application, which authenticates to Anthropic with an API key or OAuth profile — a trust domain entirely separate from how end users authenticate to your tools.

The OAuth 2.1 + PKCE handshake for remote MCP servers

An unauthenticated client gets a 401 with a WWW-Authenticate header pointing to a Protected Resource Metadata document; the client discovers the authorization server, registers (statically or via DCR), completes an authorization-code grant with PKCE, and thereafter sends Authorization: Bearer <token> on every request. Two hard rules make this secure: PKCE applied universally (defeats code interception) and audience validation — the server must confirm the token's aud claim was minted for this server and reject generic audiences like api.

Authorization, injection, and confused deputy

Authorization answers what an authenticated caller may do, governed by least privilege and enforced with per-tool scopes checked on each route. Key hygiene: never hand-roll token validation, use short-lived tokens, enforce HTTPS, never log credentials, control Dynamic Client Registration, pin to a single issuer, and treat Mcp-Session-Id as untrusted input.

Even a correctly authenticated agent can be tricked. Prompt injection is the tool-server analog of SQL injection: untrusted content can smuggle instructions to hijack the agent. The confused deputy problem is the token-passthrough attack the audience check defends against — a component with more privilege than the requester blindly forwarding a token issued for another service.

Sandboxing isolates tool execution with containers and executable allowlists (never blocklists). Human-in-the-loop (HITL) gates hard-to-reverse actions behind explicit approval, with reversibility as the criterion. Promoting a dangerous action from an opaque bash string to a dedicated typed tool (send_email(to, subject, body)) makes it trivially gateable — which is why you reach for the manual agentic loop to intercept each tool_use block.

2. Idempotency and Retries

A tool call can succeed on the server but time out before the client sees the result; a rate limit can bounce a good request; and the model can decide to call the same tool a second time. Both infrastructure retries and model duplication converge on one requirement: the tool's side effect must be idempotent — executing it twice has the same effect as executing it once.

Idempotency keys and deduplication

An idempotency key names a logical operation so the server can recognize a repeat and skip re-executing the side effect. Three techniques deliver it together: (1) generate a stable client-side key derived deterministically from the operation's inputs (e.g. a hash of user_id + action + target); (2) make side-effecting tools idempotent via upsert semantics so a replay is a no-op; (3) record the Anthropic request-id to correlate retried calls. The critical detail: the dedup-check-and-store must be atomic (one INSERT ... ON CONFLICT DO NOTHING on a UNIQUE key column), or two concurrent retries both pass the check and duplicate the effect.

For work that must not duplicate and may run long, reach for the Message Batches API (submit and poll, sidestepping the mid-request-drop failure mode) and use streaming for requests over ~10 minutes.

Retry, exponential backoff, and jitter

Exponential backoff increases the wait geometrically (1s, 2s, 4s, 8s…) to give an overloaded system time to recover; jitter adds a random offset to prevent the "thundering herd" of synchronized retries. The single most important fact: the official SDKs already implement backoff and jitter correctly — tune max_retries and timeout rather than reimplementing. When you must roll a custom loop for a third-party API, catch the most specific retryable classes first (429, ≥500, connection errors) and never retry other 4xx.

Handling rate limits: 429, 529, and Retry-After

On a 429, check retry-after first and wait exactly that duration. Better still, steer client-side throttling before you hit a 429 by watching the anthropic-ratelimit-*-remaining headers. A 529 is a system-wide overload — back off, but do not count it against your own limit accounting. Limits are enforced with a token-bucket algorithm, so a burst of 60 simultaneous requests can trip a "60/minute" limit; pace requests and ramp new traffic gradually.

3. Observability and Testing

Observability is the property that a system's internal state can be inferred from its external outputs. For a Claude agent it rests on three OpenTelemetry signals, each with its own enable switch.

With the enhanced-telemetry beta on, each step of the loop becomes a span: claude_code.interaction (one turn), claude_code.llm_request (each API call, carrying model name, latency, token counts), claude_code.tool (each invocation, with child spans for permission waits and execution), and claude_code.hook. These nest naturally, and the SDK auto-propagates W3C trace context into the CLI subprocess, so an agent run appears inside your application's trace.

Privacy is the default: only durations, model/tool names, and token counts are recorded; content is opt-in via OTEL_LOG_USER_PROMPTS, OTEL_LOG_TOOL_DETAILS, etc. Two gotchas: never use the console exporter (stdout is the SDK's message channel) and lower export intervals for short-lived calls so data reaches the collector before the process exits.

Evals: testing agent behavior

Traces tell you what happened; evals tell you whether it was correct. An eval is a systematic test of agent behavior — the agent analog of a unit-test suite. Build a test set (representative inputs paired with expected actions: which tools fire, in what order, with what arguments), run and capture the transcript, then score with programmatic assertions and/or LLM-as-judge. The most powerful technique is replaying transcripts and asserting on tool_use: because the API is stateless, a captured transcript is a complete, replayable artifact, so a tool-use assertion is deterministic, cheap, and side-effect-free.

Monitoring drift, cost spikes, and failure rates

Evals catch regressions before deploy; monitoring catches them after. Three metrics deserve standing alerts: failure rates (errored vs successful calls, from log events), cost spikes (sudden token-spend increases, from metrics — watch the cache rate), and behavioral drift (the same eval suite run continuously against production; scores sliding even though nothing errored). Drift is the failure mode with no exception to catch, which is precisely why continuous evals matter. Observability and testing are one system: the traces you capture for debugging are the transcripts you replay for evals.

4. Cost and Latency Engineering

Every token has a price and a latency. In an agentic loop that re-sends a growing transcript each turn, both compound into the bill and the wait.

Token accounting

Usage splits into distinct categories, each priced differently: input tokens (base rate, reflecting only the uncached remainder after your last cache breakpoint), cache-write tokens (a premium over base), cache-read tokens (~10% of base), and output tokens (higher base output rate; thinking counts as output). The key identity: total prompt size = input_tokens + cache_creation_input_tokens + cache_read_input_tokens — sum all three or you will misjudge cost and cache effectiveness.

Prompt caching: the biggest single lever

Prompt caching reuses previously processed prompt prefixes. Claude renders a prompt in a fixed order (tools → system → messages), and caching is a prefix match: mark a breakpoint with cache_control: {"type": "ephemeral"} (up to four per request), and a later request with the same prefix skips reprocessing it.

The TTL clock resets on each read, so a chatty session stays warm. Cache stable parts (system instructions, context documents, tool definitions, history) and put volatile content after the last breakpoint, because any byte change in the prefix invalidates everything after it. A worked 10-turn example with a 50,000-token fixed prefix drops from $2.525 to $0.5625 — a 78% reduction — and every cached turn is also faster. Verify it works by reading cache_read_input_tokens; if it's zero, a silent invalidator (a datetime.now(), an unsorted json.dumps(), a per-request tool set) is shifting the prefix bytes.

Caching also raises effective throughput: for most models only uncached input counts toward the input-tokens-per-minute (ITPM) limit, so a 2M-ITPM limit at an 80% hit rate effectively processes ~10M input tokens/minute.

Other cost and latency levers

Model selection: route each task to the cheapest model that meets the quality bar; spawning a cheaper subagent preserves the main loop's model-scoped cache. Batching: the Message Batches API runs async at 50% of standard prices. max_tokens has no rate-limit downside — set it generously to avoid truncation.

Latency is dominated by round-trips. Two structural fixes: parallelize independent calls (one assistant message can contain multiple tool_use blocks; return all tool_result blocks in a single user message — splitting them trains the model to stop parallelizing), and compose calls into a script (programmatic tool calling keeps intermediate results in the code container, so only the final output re-enters context). Finally, a latency budget allocates target time per stage, and streaming wins perceived speed and is a correctness requirement for large outputs (the SDKs refuse non-streaming requests estimated to exceed ~10 minutes).