Chapter 14: Putting It Together: Architecture, Evaluation, and Next Steps

Part 1: A Reference Architecture

A reference architecture is a template solution: a named, reusable arrangement of components and their relationships that a team can adapt rather than reinvent. For a personalized assistant, it is the blueprint that says which subsystem owns which decision, and in what order they run. Everything from the earlier chapters becomes a labeled box in this blueprint.

Think of the assistant as a restaurant kitchen. The context window is the countertop; prompt assembly is the head chef writing the ticket; tools and the agent loop are the line cooks; MCP is the standardized rail system that lets any station snap into the line; memory and RAG are the walk-in cooler and recipe book; and routing and cost control are the expediter watching the budget. No single station makes the meal—the composition does.

The subsystem → chapter map

The table below is the load-bearing artifact of the chapter—the map from each subsystem to the chapter that taught it, the production layer it occupies, and the decision it owns.

Surrounding these six subsystems, production adds three cross-cutting layers: a model gateway (routing, failover, caching), a guardrails layer (input/output inspection), and an observability layer (recording everything for debugging and compliance). The six subsystems are the engine; these three are the chassis and dashboard that make the engine safe to drive on public roads.

Figure 14.1: The full reference architecture—six engine subsystems wrapped by the gateway, guardrails, and observability chassis.

Memory is four tiers, not one store

The memory built in Chapters 9–11 is really four tiers: working memory (task-scoped constraints and partial plans), conversation memory (last N turns plus rolling summaries, ephemeral), task/artifact memory (files, decisions, PR links as structured logs), and long-term memory (stable preferences and facts in a database with consent). Vector search is one tier among these—right for docs and long recall, but the wrong home for critical facts (use a database), permissions (use config), or financial data (never).

Figure 14.2: The four tiers of memory—"memory ≠ vector DB," each tier mapped to its home and lifetime.

Where personalization and cost live

Personalization is decided in three places: (1) retrieval—long-term memory and RAG select what the assistant knows about this user; (2) prompt assembly—those preferences become instructions the model must honor; and (3) memory write-back—new preferences are persisted with consent so the next session starts smarter. Personalization is therefore a loop, not a lookup.

Cost is decided at the gateway: routing chooses cheap triage vs. expensive reasoning, caching avoids paying twice, and per-request budgets (time, tokens, dollars) cap runaway loops. The two concerns interact—richer personalization means retrieving more context, which costs more—so the architecture exposes sampling rates, retrieval depth, and routing thresholds as dials, not hard-coded constants.

Part 2: Request Flow & Evaluation and Quality

Composition becomes concrete when you trace one request. Priya, a returning user, has told the assistant she prefers TypeScript and terse answers. She types: "Refactor the auth module in my project to use async/await, and open a PR." The request travels through all six subsystems plus the cross-cutting layers, in a predictable order.

Tracing Priya's request end to end

The request hits the gateway; a pre-LLM guardrail scans for PII and injection and flags "open a PR" for later approval (~200–300 ms budget). Routing triages it as genuine multi-step reasoning and sends it to the expensive reasoning model. Memory + RAG loads long-term preferences ("uses TypeScript," "terse"), pulls conversation memory, and retrieves the current auth source. Prompt assembly frames the code as untrusted reference data, never as commands. Context management measures the prompt against the window and summarizes older turns if needed. The agent loop over MCP calls read_file, write_file, run_tests, with a state reducer applying each deterministic result. At open_pr, post-LLM guardrails pause for human approval. Finally, memory write-back + observability log the decision, PR link, and OpenTelemetry traces.

Personalization threaded silently through steps 3, 4, and 8: the assistant produced TypeScript and stayed terse because long-term memory shaped prompt assembly, and it remembered the task because artifact memory captured it.

Figure 14.3: Priya's request traced end to end through the gateway, engine subsystems, and chassis.

Offline and online evaluation

Evaluation is the discipline of scoring an assistant's behavior against defined criteria; an eval harness runs those scorers over test cases or live traffic. It runs in two modes. Offline evaluation runs metrics on a curated golden set (typically 20–50 realistic cases with deterministic tool mocks, often pinned with a fixed seed) before deploy—a flight simulator. Online evaluation scores sampled live traffic (1–10% for high-volume apps) asynchronously to detect drift—the aircraft's live telemetry. You need both.

Many scorers use LLM-as-judge—an LLM scoring another LLM's output. The G-Eval method formalizes this with a chain-of-thought rubric. Best practice: require structured JSON output and an explicit rubric, sample and audit judge scores, and treat scores as flaky until proven stable.

Metrics mapped to subsystems

What you measure depends on which part you are grading. Retrieval metrics separate the RAG half: context precision (retrieved context is relevant), context recall (retrieval captured everything needed), and context relevance (topical relatedness). For assistants you must evaluate full trajectories, not just the final answer. Personalization metrics remain immature, so teams assemble custom golden sets.

Regression testing prompts and tools

A regression test confirms that a change—new model, edited prompt, swapped retriever, altered tool—did not break something that used to work. When factuality drops from 85% to 72%, you know something broke. The workflow is a virtuous cycle: detect regressions after any change; sample low-scoring production traces into datasets that become permanent regression tests; and integrate eval runs into CI/CD so metric deltas surface directly in pull requests.

Figure 14.4: The offline/online evaluation loop—production failures feed back into regression sets wired into CI.

Part 3: Production Readiness

An assistant that scores well in evaluation is still not production-ready. Production adds three obligations that never appeared in the demo: it must be safe around real users and money, it must scale reliably, and it must be monitored for cost and quality drift. This is the chassis and dashboard around the engine.

Safety, privacy, and guardrails

Guardrails are runtime controls that inspect and, when necessary, block or transform inputs and outputs—actively intervening rather than just observing. They use separate models for generation and evaluation and run in two positions: pre-LLM checks protect what goes in (PII, sensitive data, prompt injection), and post-LLM checks control what comes out (filtering unsafe content, enforcing approvals). Both operate within a tight ~200–300 ms budget—like airport security: a screening lane before boarding and a customs check on arrival, both fast enough not to strand travelers.

A concrete privacy-and-safety checklist from production practice:

• Policy as code gates capabilities per environment—allowlisting tools and requiring approval for irreversible actions like sendEmail, deleteRecord, or chargeCard.
• Treat retrieved content as untrusted input; separate data from instructions.
• Never embed raw secrets in model context—use short-lived tokens.
• Sandbox risky tools (shell, browser) with full audit logging.
• Add human-in-the-loop approvals for irreversible actions.

For personalized assistants, privacy has a memory dimension: long-term preferences are stored with consent tracking. Remembering a user "uses TypeScript" is convenient; remembering something they never consented to store is a liability.

Scaling and reliability

Reliability in a nondeterministic system does not come from a bigger model—it comes from architecture. Three moves carry most of the weight: (1) a state reducer for deterministic transitions (the model proposes; the reducer disposes)—the biggest reliability jump; (2) tool contracts as APIs—typed and validated (JSON Schema, Zod, OpenAPI), returning a result envelope { ok, data, error, meta }, using idempotency keys for side effects, and carrying budgets (e.g., a 15,000 ms timeout, max-retries, cost ceilings); and (3) failover at the gateway—LLM proxies plus checkpointing so a paused or crashed trajectory can resume rather than restart.

Rollout discipline matters as much as any single component. The recommended path: scope a focused pilot, instrument it with evaluations and observability, enforce clear guardrails, and expand in measured steps. A practitioner's 1–2 week build order:

1. Tool contracts + I/O validation.
2. State reducer for deterministic transitions.
3. Trace-level observability.
4. Small eval dataset with tool mocks.
5. Policy gating + approval UX.
6. Memory layers (summaries + artifacts first, vector search deferred).

The order deliberately defers vector search—reinforcing "memory ≠ vector DB." Most production wins come from structured state, summaries, and artifacts; teams add vector retrieval later for large doc sets or long-term recall.

Cost/quality monitoring in production

Once live, the assistant must be watched on two axes at once: is it good, and is it affordable? Observability—seeing, understanding, and explaining what agents do—answers "why did the agent do that?" via OpenTelemetry traces capturing step id, tool name, args hash, duration, result, and token usage. Those same token-usage and duration fields are the raw material for cost monitoring. Quality monitoring is online evaluation: score sampled traces (1–10%), compare trends against offline benchmarks to detect drift.

The two axes couple through the gateway. If cost per response rises, observability traces may reveal the agent retrying tools too often; you tighten the retry budget and add a cache, then quality monitoring confirms task success held steady while cost fell. This closed loop—observe, adjust the dial, re-measure—is production operations for a personalized assistant.

Part 4: Where to Go Next

You now have a complete, evaluated, production-hardened assistant. The field is not standing still, and this final section points at where a practitioner's attention should turn next.

Emerging patterns and standards

The dominant 2025 message crystallized: "Models are strong; reliability comes from architecture plus guardrails." The competitive differentiator is no longer "does it respond intelligently" but "does it choose the right action, stay within policy, measure regressions, debug trajectories, and stay trustworthy around real users and money." Several standards are consolidating: OpenTelemetry as the common backbone for agent observability, typed tool contracts (JSON Schema, Zod, OpenAPI)—the same standardization impulse behind MCP—and evaluation-in-CI, shifting from nice-to-have to table stakes. The through-line: the LLM ecosystem is professionalizing, adopting the contracts, telemetry, and test gates that mature software engineering has always relied on.

Multi-agent and long-horizon systems

The headline shift of 2025 was "from assistants to agents." A multi-agent system is one in which several specialized agents collaborate—each with its own tools, memory, and role—rather than a single agent doing everything. Everything in this chapter scales up, but the demands intensify: when five agents hand work to each other over a long horizon, "why did the agent do that?" becomes "which agent, at which step, and why?" The state reducer, tool contracts, and layered memory are not replaced—they are replicated per agent and coordinated between them. Long-horizon tasks lean especially hard on checkpointing and the four-tier memory model, since the system must survive interruptions and resume with context intact.

A practitioner's continued-learning path

The pragmatic path forward mirrors the rollout discipline: start small, instrument everything, expand in measured steps, and treat governance and measurement as ongoing disciplines. Concretely, a practitioner should:

1. Build the walking skeleton in the recommended order—tool contracts, state reducer, observability, a small eval set, policy gating, then memory layers. Resist starting with the vector DB.
2. Learn the tooling ecosystem hands-on—Ragas, DeepEval, TruLens, LangSmith, Arize Phoenix for evaluation; Braintrust, Langfuse, Evidently AI for unified scoring and LLM-as-judge.
3. Study the case-study literature—large corpora like the 419-case LLMOps study distill what actually works versus what merely demos well.
4. Treat evaluation as a living practice—keep feeding production failures back into regression sets, and keep the CI gate honest.

The enduring lesson of the whole book: models are strong; reliability comes from architecture plus guardrails, proven by evaluation.