RAG Evaluation and Failure Analysis

Most engineers know their RAG system is not working when users complain. The ones who ship reliable systems know it is not working before users see it, because they built an eval loop. This article is about how to build that loop and what to do when it tells you something is broken.

Building an Eval Set

A RAG eval set needs three things: questions, the chunks that should be retrieved, and reference answers.

pythonCopy
from dataclasses import dataclass

@dataclass
class RAGEvalCase:
    question: str
    relevant_chunk_ids: list[str]  # ground-truth chunks
    reference_answer: str          # human-written or GPT-4 generated

def generate_eval_set_from_docs(docs: list[dict], n: int = 100) -> list[RAGEvalCase]:
    """Use GPT-4o to generate eval cases from your actual documents."""
    client = OpenAI()
    eval_cases = []
    for doc in docs[:n]:
        resp = client.chat.completions.create(
            model="gpt-4o",
            messages=[{
                "role": "user",
                "content": (
                    f"Given this document chunk:\n\n{doc['text']}\n\n"
                    "Generate a question that can only be answered using this chunk. "
                    "Return JSON: {\"question\": \"...\", \"answer\": \"...\"}"
                ),
            }],
            response_format={"type": "json_object"},
        )
        import json
        data = json.loads(resp.choices[0].message.content)
        eval_cases.append(RAGEvalCase(
            question=data["question"],
            relevant_chunk_ids=[doc["id"]],
            reference_answer=data["answer"],
        ))
    return eval_cases

Fifty cases is enough to detect major problems. One hundred gives you statistical confidence. More than 200 is overkill for iteration.

Measuring Recall@k

pythonCopy
def evaluate_retrieval_pipeline(eval_cases: list[RAGEvalCase],
                                  retriever, k: int = 5) -> dict:
    recall_scores = []
    misses = []
    for case in eval_cases:
        results = retriever.query(case.question, top_k=k)
        retrieved_ids = {r["id"] for r in results}
        relevant = set(case.relevant_chunk_ids)
        recall = len(retrieved_ids & relevant) / len(relevant)
        recall_scores.append(recall)
        if recall < 1.0:
            misses.append({
                "question": case.question,
                "recall": recall,
                "missed_ids": list(relevant - retrieved_ids),
            })
    avg_recall = sum(recall_scores) / len(recall_scores)
    return {
        "recall@k": avg_recall,
        "k": k,
        "n": len(eval_cases),
        "misses": sorted(misses, key=lambda x: x["recall"])[:10],
    }

Target: recall@5 >= 0.80 before worrying about anything else. If you are below 0.60, the answer quality is almost certainly poor regardless of prompt engineering.

Faithfulness Scoring

Faithfulness measures whether the generated answer is supported by the retrieved context - or whether the model is hallucinating.

pythonCopy
def score_faithfulness(answer: str, context: str) -> float:
    """Ask an LLM to check if every claim in the answer is supported by context."""
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": (
                f"Context:\n{context}\n\nAnswer:\n{answer}\n\n"
                "For each factual claim in the answer, check if it is directly supported "
                "by the context. Return JSON: "
                "{\"score\": 0.0-1.0, \"unsupported_claims\": [list of strings]}"
            ),
        }],
        response_format={"type": "json_object"},
    )
    import json
    return json.loads(resp.choices[0].message.content)

# Run over your eval set
faithfulness_scores = []
for case in eval_cases:
    context = get_context(case.question)
    answer = generate_answer(case.question, context)
    result = score_faithfulness(answer, context)
    faithfulness_scores.append(result["score"])

print(f"Mean faithfulness: {sum(faithfulness_scores)/len(faithfulness_scores):.2f}")

A faithfulness score below 0.75 means more than 25% of your answers contain claims not supported by the retrieved context. That is your hallucination rate.

Common Failure Modes and Fixes

1. Bad Chunking

Symptom: recall@k is low; retrieved chunks are semantically adjacent to but not containing the answer.

Diagnosis: look at the retrieved chunks for your low-recall eval cases. If the answer is split across a chunk boundary, your chunks are too small or splitting at wrong boundaries.

Fix: increase chunk size, increase overlap, or use semantic/heading-aware splitting.

2. Wrong Embedding Model

Symptom: recall@k is mediocre across all query types; no clear pattern in misses.

Fix: benchmark two or three embedding models on your eval set. This takes 10 minutes and frequently finds a 10–20% recall improvement.

3. Irrelevant Retrieval

Symptom: retrieved chunks have the right keywords but wrong semantics. "Python performance" retrieves chunks about Python snakes.

Fix: add a reranker. Cross-encoders (like cross-encoder/ms-marco-MiniLM-L-6-v2) dramatically improve precision at the cost of ~50ms latency. Worth it for most applications.

4. Hallucinated Citations

Symptom: faithfulness score is low; the model cites specific numbers or names not present in any retrieved chunk.

Diagnosis: compare the answer against each retrieved chunk individually. Look for specific values - dates, percentages, names - that do not appear verbatim in context.

Fix: add explicit instructions: "Only state facts that appear in the provided context. If a specific number or name is not in the context, do not state it." Also check whether recall@k is the real root cause - if the relevant chunk was not retrieved, the model is filling from prior.

5. Metadata Filtering Not Applied

Symptom: old document versions or irrelevant departments appear in retrieval.

Fix: apply metadata filters before vector search. pgvector, Pinecone, and Qdrant all support pre-filter on metadata fields. A document dated 2022 should never appear in a query scoped to 2024 content.

Continuous Evaluation

Build eval into your CI/CD pipeline. Every time you change chunking, embedding model, or retrieval parameters, run the eval suite.

pythonCopy
# In your CI pipeline
def regression_check(eval_cases, retriever, baseline_recall=0.80):
    result = evaluate_retrieval_pipeline(eval_cases, retriever, k=5)
    assert result["recall@k"] >= baseline_recall, (
        f"Retrieval regression: recall@5 = {result['recall@k']:.2f}, "
        f"expected >= {baseline_recall}"
    )
    print(f"Recall@5: {result['recall@k']:.2f} ✓")

This makes retrieval quality visible and prevents silent regressions when you update your embedding model or refactor your chunker.

Common Mistakes

Evaluating only the final answer quality. A model can give a correct answer despite bad retrieval, if the answer happens to be in its training data. Always measure retrieval independently.

Using only 5 eval cases. Five cases do not give you statistical confidence. With 5 cases, a single retrieval failure moves your recall@5 by 20 percentage points. Use at least 50.

Not tracking failure mode distributions. The fix for "bad chunking" is different from the fix for "wrong embedding model." Log which failure mode each miss falls into, and fix the most common one first.

Where to Go Next

See also: [rag-foundations-retrieval-quality], [rag-vector-search-systems], [milestone-gate-2-applied-ai-engineering]