Retrieval-Augmented Generation in Practice

The Gap Between RAG Demo and RAG Production

A RAG demo is straightforward: embed some documents, embed a query, return the top-K chunks, stuff them into a prompt, call an LLM. This works in demos because the documents are clean, the queries are predictable, and nobody is measuring quality systematically.

RAG in production is different. Documents are messy (scanned PDFs, HTML soup, tables). Queries are diverse and ambiguous. The failure modes are non-obvious: good retrieval + bad generation, bad retrieval + surprisingly good generation, silent failures where the system confidently returns wrong answers.

This post covers the design decisions and evaluation practices that determine whether a RAG system is worth deploying.

Retrieval Quality: The Foundation of Everything

The LLM can only be as good as what you give it. Retrieval quality gates generation quality. A 90th percentile LLM cannot compensate for poor retrieval - it will confidently hallucinate from irrelevant context.

Measuring Retrieval Quality

You need a labeled evaluation set: query → relevant document(s). For each query in the set, measure:

Recall@K: Of all relevant documents, what fraction appear in the top K results?

Precision@K: Of the top K results, what fraction are actually relevant?

MRR (Mean Reciprocal Rank): Average of 1/rank of the first relevant result. Penalizes systems that rank the right document lower.

A retrieval system with Recall@5 = 0.90 will miss the relevant document 10% of the time, causing downstream generation failures that are impossible to recover from.

Building a Retrieval Eval Set

For 100–200 representative queries:

• Have domain experts write the relevant documents or document IDs
• Or use a production query sample with human annotation
• Or synthetically generate queries from documents using an LLM (generate 5 questions per document, use the document as ground truth)

Synthetic generation is fast and cheap; validate a sample with humans to check quality.

Chunking Strategy: The Most Underrated Decision

Chunking determines the unit of retrieval. A chunk that is too small lacks context; a chunk that is too large dilutes relevance and wastes context window tokens.

Fixed-Size Chunking

Split every N tokens with a K-token overlap. Simple and reproducible. Works well for uniform text (articles, documentation).

pythonCopy
from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=64,
    separators=["\n\n", "\n", ". ", " "],
)
chunks = splitter.split_text(document_text)

The separators list controls where splits happen - preferring paragraph boundaries over sentence boundaries over arbitrary positions.

Semantic Chunking

Split on topic shifts detected by semantic dissimilarity between adjacent sentences. Produces chunks that are more coherent but harder to reproduce and control in size.

Use when documents have distinct sections that should not be split (technical manuals, legal contracts, scientific papers).

Document Structure-Based Chunking

Use document metadata to guide chunking: split on headers, sections, pages. Preserves logical structure that aids relevance.

For PDFs: extract by page first; for HTML: split on <h2> and <h3> tags; for code: split on function/class boundaries.

Parent-Child Chunking

Index small chunks (for precise retrieval) but return large chunks (for context). The small child chunk is matched; the parent chunk is returned to the LLM.

This gets the precision benefits of small chunks and the context benefits of large ones. Requires storing parent-child relationships in the document store.

Embedding Model Selection

Not all embedding models are equivalent. Dimensions:

Domain fit: General-purpose models (OpenAI text-embedding-3-large, Cohere embed-english-v3) work well for most text. Domain-specific models (legal, medical, code) outperform on domain-specific tasks.

Dimension size: More dimensions = richer representation, higher storage cost, slower ANN search. 768 to 1536 dimensions is the common range.

Max token length: Chunks that exceed the embedding model's context are truncated. Check that your chunk size is under the model's limit.

Multilingual: If your documents span languages, use a multilingual embedding model (multilingual-e5-large, Cohere multilingual).

Test embedding models on your retrieval eval set, not on academic benchmarks. Domain performance varies significantly from general benchmarks.

Reranking: Getting Precision Right

ANN vector search is optimized for recall - it returns K candidates that are probably relevant. It is not optimized for precision - ranking the most relevant candidate first.

A reranker (cross-encoder) takes query-document pairs and computes a fine-grained relevance score by attending to both simultaneously. This is expensive per pair but highly accurate.

Two-stage pattern:

1. Vector search: retrieve top-50 candidates fast (< 20ms)
2. Reranker: score and re-sort top-50, return top-5 to the LLM (50–150ms)

Good rerankers: bge-reranker-v2-m3, Cohere Rerank, Jina Reranker. Evaluate on your own eval set - off-the-shelf rankings vary significantly by domain.

Failure Mode Taxonomy

Systematically classifying RAG failures tells you where to invest improvement effort.

Failure Type	Symptom	Root Cause	Fix
Retrieval miss	No relevant context retrieved	Poor embedding / wrong chunking	Better embedding model, different chunk size
Retrieval noise	Relevant context retrieved but buried	Low ranking of correct document	Add reranker
Context overflow	Correct context in prompt but ignored	Too much context, right document is diluted	Reduce K, improve reranker precision
Hallucination	Answer contradicts retrieved context	LLM not grounded to context	Grounding instructions, citation enforcement
Out-of-scope	User asks about something not in the knowledge base	No relevant documents	Detect and respond with "I don't know"
Temporal confusion	Outdated context returned	Stale index	Regular re-indexing, date-based filtering

Build a representative set of examples for each failure type. Track the rate of each in production. Prioritize fixes by frequency × severity.

Citation and Groundedness

A production RAG system should not just produce an answer - it should cite the sources. This enables users to verify claims and exposes the retrieval quality to users.

pythonCopy
system_prompt = """
Answer the question using only the provided context.
For every factual claim, add a citation in the format [Source: document_id].
If the context does not contain enough information to answer, say "I don't have enough information to answer this question" rather than guessing.
"""

Groundedness evaluation: does the answer contain only information present in the retrieved context? Use an LLM evaluator:

pythonCopy
eval_prompt = """
Given the context and the answer, identify any claims in the answer that are NOT supported by the context.
Return a JSON list of unsupported claims. Return an empty list if all claims are supported.
Context: {context}
Answer: {answer}
"""

High rates of unsupported claims indicate the LLM is hallucinating beyond the context. Strengthen grounding instructions or add explicit post-processing validation.

Advanced Patterns

Query Rewriting

User queries are often ambiguous or use vocabulary different from the documents. Rewrite the query before retrieval:

• Hypothetical document embedding (HyDE): Ask an LLM to generate a hypothetical document that would answer the query, embed that, and retrieve against it. Often improves recall for complex questions.
• Query expansion: Rewrite the query in multiple ways (synonyms, different phrasings) and combine results.
• Sub-question decomposition: Break a complex question into simpler sub-questions, retrieve for each, then synthesize.

Multi-Index Retrieval

For knowledge bases that span multiple domains, maintain separate indexes per domain and route queries to the appropriate index. Reduces noise from cross-domain retrieval.

Common Mistakes and Bad Instincts

Tuning prompts when the problem is retrieval. If the LLM is producing wrong answers, measure retrieval quality first. Adding more prompt instructions cannot fix what is not in the context.

Using the same embedding model for indexing and live queries without checking. Embedding models are updated. Using different versions for indexing and retrieval breaks semantic alignment.

Not building an eval set before optimizing. You cannot improve what you cannot measure. Build the eval set first, then optimize.

Retrieving too many chunks. More chunks = more noise = worse generation. K=50 is almost always too many. Measure the optimal K on your eval set.

Assuming reranking always helps. Rerankers improve precision but add latency. Measure the precision lift and latency cost on your specific system before making it the default.

Where to Go Next

RAG architecture is covered in Module 22 (Embeddings, Retrieval, and RAG Systems) of the College Student path and Module 16 (Retrieval Systems, Vector Databases, and RAG) of the SWE path. Both modules require building a complete RAG pipeline with a retrieval eval set, chunking strategy comparison, and groundedness evaluation.

Closing Thought

The practical standard is not memorization. It is whether you can use the idea to make a better engineering decision, explain that decision to someone else, and notice when reality disagrees with your assumptions.

Team Review Prompts

Before treating this work as complete, ask a teammate to review it using three prompts:

1. What assumption is most likely to break in production?
2. What evidence would make you trust the result?
3. What simpler approach should we compare against?

These questions are deliberately plain. They work because they force the discussion away from tool enthusiasm and back toward judgment, evidence, and maintainability.

RAG Is Two Systems

Retrieval-augmented generation has an offline system and an online system.

Offline indexing:

1. Collect documents
2. Parse them
3. Split them into chunks
4. Attach metadata
5. Create embeddings
6. Store vectors and source text
7. Re-index when documents change

Online answering:

1. Receive user question
2. Rewrite or classify query if needed
3. Retrieve candidate chunks
4. Rerank or filter candidates
5. Build grounded prompt
6. Generate answer
7. Validate citations and safety
8. Log query, retrieval, answer, and feedback

Debug these systems separately. If the answer is bad, first ask whether the right evidence was retrieved.

Chunking Decisions

Chunking controls what the retriever can find. Chunks that are too small lose meaning. Chunks that are too large dilute relevance.

Good chunking respects document structure:

• Headings
• Sections
• Paragraphs
• Tables
• Code blocks
• FAQs
• Product page boundaries

Metadata often matters as much as the embedding. Store document type, source, date, owner, permissions, and section title. Metadata filters can prevent irrelevant or unauthorized context from entering the prompt.

Retrieval Quality Metrics

Track retrieval before generation:

• Recall at K: did the right evidence appear in the top results?
• Mean reciprocal rank: how high did the first useful result appear?
• Context precision: how much retrieved context was actually relevant?
• No-answer detection: did the system avoid answering when evidence was absent?

Generation quality is bounded by retrieval quality. If evidence is missing, the generator is forced to guess or abstain.

Grounded Generation Pattern

Use prompts that separate evidence from instruction:

textCopy
Answer using only the provided sources.
If the sources do not contain the answer, say you do not know.
Include citations for each factual claim.

Sources:
...

Question:
...

This does not guarantee truth, but it creates a contract you can evaluate.

Common RAG Failures

• Stale index
• Bad document parsing
• Chunk boundaries that split concepts
• Embedding model mismatch
• Missing metadata filters
• Top-K too small or too large
• No reranking for ambiguous queries
• Generator ignores evidence
• Citations point to retrieved but irrelevant chunks
• Permission leaks through shared indexes

RAG reduces some hallucination risk, but it does not remove the need for evaluation.

Production Checklist

Before shipping RAG, verify:

• The index refresh process is documented
• Retrieval metrics exist for a labeled query set
• Answers cite source chunks
• Permissions are enforced before retrieval or before context assembly
• The system can say "I do not know"
• Users can report bad answers
• Logs include query, retrieved chunk IDs, prompt version, model version, and answer

That evidence makes RAG maintainable instead of magical.

Final Rule

RAG quality is limited by the weakest link: source quality, parsing, chunking, embeddings, retrieval, reranking, prompting, generation, citation, or permissions. Improve the layer that failed. Do not blindly swap models and hope the system becomes trustworthy.

Answer Synthesis and Citation Quality

Good RAG answers do more than paste retrieved text. They synthesize evidence while preserving traceability. A strong answer:

• Directly answers the question
• Cites the source for factual claims
• Mentions uncertainty when sources conflict
• Refuses when sources are insufficient
• Avoids adding unsupported background knowledge

Citation quality should be evaluated. A citation is useful only if the linked source actually supports the sentence. Many poor RAG systems retrieve relevant documents but attach citations loosely. Users then get false confidence.

Query Rewriting

Users rarely ask questions in the same language as your documents. Query rewriting can expand acronyms, resolve references, add product context, or split a complex question into sub-questions.

But query rewriting can also change meaning. Log original and rewritten queries. Evaluate whether rewriting improves retrieval on real examples. Never assume that a more verbose query is automatically better.

When Not to Use RAG

RAG is not always the answer. Avoid it when:

• The task does not need external knowledge
• The knowledge base is tiny and can fit safely in context
• The documents are too low quality to retrieve from
• Permissions are too complex for the current system
• The answer requires computation, not lookup
• A deterministic workflow would be simpler and safer

The best engineering choice is sometimes a form, a search UI, or a rules engine.

Hybrid Search and Reranking

Dense vector search is powerful, but it is not always enough. Keyword search is often better for exact names, IDs, error codes, and rare terms. Hybrid retrieval combines dense search with sparse keyword search so the system can handle both semantic similarity and exact matches.

Reranking adds a second stage. First retrieve a broad candidate set quickly. Then use a stronger reranker to sort the top candidates by relevance. This often improves answer quality more than switching to a larger generation model.

Permissions and Multi-Tenant Data

Enterprise RAG must enforce permissions. If two customers share one index without careful filtering, retrieval can leak data. Permission checks should happen before context reaches the model.

Store access-control metadata with each chunk:

• Tenant ID
• Document owner
• Access group
• Visibility level
• Effective date
• Expiration date

Then test permission boundaries with adversarial queries. Security bugs in retrieval are product-critical.

Freshness and Index Maintenance

A RAG system is only as current as its index. Document updates, deletions, and permission changes must propagate predictably.

Track:

• Last indexed timestamp
• Failed documents
• Deleted document tombstones
• Embedding model version
• Chunking strategy version
• Source parse errors

If a user asks about a new policy but the index is three weeks old, the model may answer confidently from stale evidence.

Building a RAG Eval Set

Create queries from real user behavior:

• Direct factual questions
• Ambiguous questions
• Questions with no answer in the corpus
• Multi-hop questions requiring several chunks
• Acronyms and synonyms
• Permission-bound questions

For each query, store expected source documents and answer criteria. This gives you a repeatable way to test chunking, embedding models, rerankers, and prompt changes.