Chapter 11: Retrieval, Embeddings, and RAG for Personalization

Part 1 · Embeddings, Vector Search & the RAG Pipeline

What an Embedding Represents

An embedding is a dense vector representation of data that compresses high-dimensional information into a lower-dimensional space while preserving semantic meaning. Instead of treating a word as a discrete, opaque symbol, an embedding model maps it into a continuous vector space — often several hundred to a couple thousand numbers — where geometric proximity reflects conceptual similarity. The classic illustration is vector arithmetic: king − man + woman ≈ queen. Relationships between concepts are encoded as directions and distances, a fundamental improvement over sparse one-hot encoding, which cannot capture that "physician" and "doctor" mean nearly the same thing.

Picture a vast map where every document, sentence, or memory is a pin, and pins near each other share meaning. A query is a new pin dropped onto the map; retrieval means grabbing the nearest neighbors. A query about "pasta for a meat-free guest" lands near vegetarian Italian recipes even though it never uses those exact words. That is semantic search. For personalization, when a user says "book me the usual table," the embedding lands near stored memories about their favorite restaurant even though the memory text never contained the word "usual."

Similarity and Vector Databases

If documents and queries are points, we measure how "close" two points are. The most common metric in RAG is cosine similarity, which measures the angle between two vectors: 1 means identical direction, 0 orthogonal, −1 opposite. It is favored because it is insensitive to magnitude — it focuses purely on direction, i.e., on meaning.

Storing millions of these vectors and querying them by similarity is the job of a vector database. Around 2020, specialized databases emerged — Pinecone, Weaviate, Milvus, Qdrant, and Chroma — that combine fast similarity search with full database features: CRUD, metadata filtering, horizontal scaling, and persistence. A useful distinction: FAISS is a library for fast nearest-neighbor search (great for local and research use), whereas Pinecone is a fully managed cloud database with namespaces and scaling handled for you.

Indexing and Approximate Nearest Neighbors

Comparing a query against every stored vector is prohibitive at scale — scoring 50 million memories on every message is a non-starter. ANN algorithms trade a small amount of accuracy for very large speedups (often 100x or more) by first reducing the search space and comparing only against a small, promising subset.

Key takeaway: An embedding is a point on a "meaning map"; cosine similarity measures how close two points are; ANN indexes like HNSW and IVF let a vector database search millions of points in milliseconds by trading a sliver of accuracy for a 100x+ speedup.

The RAG Pipeline

RAG (Retrieval-Augmented Generation) grounds an LLM's output in retrieved external knowledge rather than relying on parametric memory alone. It has two phases: an ingestion phase (ahead of time) and a query phase (every message).

Figure 11.1: The end-to-end RAG pipeline, from ingestion through grounded, cited generation.

Chunking and Ingestion

At ingestion, source documents are split into chunks, each is embedded, and the embeddings plus chunk text and metadata are stored in a vector database. Chunking exists because documents are too large to retrieve as single units — you want the relevant paragraph, not the whole 80-page manual. Recursive character splitting at 256–512 tokens is the most reliable default; overlap (a sliding window) preserves context across boundaries; semantic chunking can help dense recall but is not universally better.

Think of tearing a cookbook into index cards: cut too finely and "bake for 20 minutes" is meaningless; cut too coarsely and searching for "vegetarian lasagna" hands you the entire "Pasta" chapter. The sweet spot is one coherent recipe per card with a little overlap — exactly what 256–512-token recursive chunking achieves.

Retrieve, Augment, Generate

The query is embedded with the same model used at ingestion (essential — query and documents must live on the same map), the system retrieves the top-K most similar chunks, inserts them into the prompt as context, and the model generates an answer grounded in that evidence.

Figure 11.2: The per-query R-A-G phase, where retrieved chunks ground and augment the prompt.

Here is the full pipeline as a stage-by-stage table, with the personalization hook at each stage:

Grounding and Citation

The word "augmented" is doing quiet but critical work. By inserting retrieved chunks into the prompt, RAG grounds the model — it answers from evidence you can point to. This is the single most important reason RAG reduces hallucination: the correct answer is literally in the context window. Grounding also enables citation: because each chunk carries source metadata, the system can attach references ("according to your uploaded lease agreement, section 4…") — a trust feature for users and a debugging feature for engineers.

Key takeaway: RAG is a two-phase pipeline — ingest (chunk → embed → index), then per-query retrieve → augment → generate → cite. Grounding the model in retrieved, source-tagged chunks is what lets it answer from evidence and cite where each claim came from, instead of hallucinating.

Part 2 · Personalized Retrieval & Improving Retrieval Quality

Retrieving from User Memory and Documents

Personalized assistants build persistent memory using RAG plus vector search. Memories are stored as embeddings with rich metadata — user, timestamp, topic, confidence — and retrieved when relevant to the current turn, giving continuity across sessions. Each memory is a chunk, embedded and indexed exactly like a document, and surfaced by the same top-K similarity search. Rather than storing everything a user ever said, preference-aligned memory construction filters and shapes what gets remembered — moving "from volume to value" so the memory is a curated, value-dense set of facts.

Filtering by User and Metadata

Semantic similarity alone is dangerous in a multi-user system: the most similar chunk to your query might belong to a different user. The primary defense is metadata filtering. Every chunk is tagged with user_id, timestamps, topics, source, and confidence, and at retrieval time you filter on this metadata. Metadata filtering is described as being as important as semantic similarity for production-grade retrieval.

Metadata-filtering example (semantic search + hard isolation)

# Personalized retrieval: semantic search + hard metadata filter
results = vector_db.query(
    query_vector = embed("something upbeat for tonight"),  # semantic map lookup
    top_k = 5,
    filter = {
        "user_id": "user_1",                 # HARD isolation: only this user's data
        "mood_preferences": {"$in": ["upbeat"]},
        "confidence": {"$gte": 0.7},         # skip low-trust stale facts
        "timestamp": {"$gte": "2026-01-01"}  # recency: recent memories only
    }
)
# Reads as: "Find preferences that FEEL like this new request,
#            but only for user_1, only upbeat, only high-confidence, only recent."

Figure 11.3: Per-user retrieval combining semantic vector search with hard metadata filters for isolation.

A stronger form of isolation is the namespace: a hard partition at the index level — effectively a separate index per user or tenant. A search of that namespace only ever touches those vectors, preventing cross-user leaks by design rather than relying solely on filtering. Namespaces plus strict user_id filtering are the primary defense against cross-user leakage.

There is a performance decision hidden in filtering:

Blending Profile Memory with RAG

The richest assistants blend two retrieval sources: profile memory (durable facts and preferences) and document RAG (the user's uploaded files). Both live in the vector store as chunks, both are retrieved by the same top-K search, both are filtered to the user. The augmentation step interleaves them — a few high-confidence profile memories to steer how the assistant responds, plus the most relevant document chunks to ground what it says. Preference-aligned memory keeps the profile side compact so it does not crowd out document context in a limited prompt budget.

Key takeaway: Personalized retrieval adds retrieving from the user's own memory/documents and isolating that data with metadata filters and namespaces. Namespaces plus strict user_id filtering are the primary defense against cross-user leakage; pre-filter vs. post-filter trades speed against recall.

Re-ranking and Hybrid Search

Hybrid search combines sparse keyword retrieval (BM25) with dense vector search. BM25 excels at exact-match queries — product SKUs, codes, terminology, numbers — while dense retrieval handles semantic paraphrasing. A user asking about "error code E-4471" needs exact matching; a user asking "why won't it turn on" needs semantic matching. The two ranked lists are merged with Reciprocal Rank Fusion (RRF), score(d) = Σ 1/(k + rank(d)) with k=60, which operates on rank positions — elegantly solving the score-incompatibility problem (BM25 scores are unbounded; cosine ranges −1 to 1).

Semantic search does not universally dominate: on financial and tabular documents (T2-RAGBench), BM25 can outperform state-of-the-art dense retrieval — a reason to keep the keyword arm of hybrid search. The single highest-impact addition is a re-ranking stage: a cross-encoder rescores the top ~50–200 candidates. The distinction is architectural — the bi-encoder used for the index encodes query and document separately (fast, coarse), whereas a cross-encoder jointly encodes the query and each candidate together (slow, far more accurate). Adding a cross-encoder yielded the largest single improvement in one study: +17.2 pp MRR@3 and +12.1 pp Recall@5 over unreranked hybrid, at ~100–300 ms cost (total pipeline stays under ~500 ms).

Figure 11.4: The three-stage production retrieval pipeline — parallel hybrid search, RRF fusion, and cross-encoder re-ranking.

Query Rewriting

The retrieval quality of a RAG system is capped by the quality of the query you search with. Real user messages are terse or full of pronouns — "what about the other one?" is unsearchable on its own. Query rewriting transforms the raw message into a better retrieval query before embedding: resolving references, expanding abbreviations, and injecting profile context. For a personalized assistant, this folds in profile memory — rewriting "book the usual" into "book a table at Luigi's, party of two, 7pm" so retrieval and generation both have concrete terms.

Evaluating Retrieval and Reducing Hallucination

You cannot tune what you cannot measure. Retrieval quality is evaluated with ranking metrics:

The end goal of all this tuning is reducing hallucination. RAG's core mechanism is grounding: if the correct evidence is in the retrieved context, the model answers from it rather than inventing. But that only works if retrieval actually surfaces the right chunk. Every improvement here — hybrid search's better coverage, re-ranking's better precision, query rewriting's better queries — is a hallucination-reduction technique because it raises the probability that grounding evidence is present and prominent. Citation closes the loop by tying each claim to a source chunk.

Key takeaway: The production-grade pipeline is three stages — parallel BM25 + dense retrieval, RRF fusion (k=60), and cross-encoder re-ranking — beating naive dense retrieval by wide margins under a ~500 ms budget. Query rewriting improves the query; Recall@K / MRR@K / NDCG@K measure quality; every upgrade reduces hallucination by ensuring grounding evidence is retrieved and ranked to the top.