RAG Foundations and Retrieval Quality

Most RAG failures are retrieval failures, not generation failures. The LLM is fine - it just never received the relevant information. Engineers who spend weeks prompt-engineering while leaving their chunking strategy untouched are optimizing the wrong stage.

This article is about what actually determines retrieval quality: chunking, embedding choice, and the one metric that tells you whether your retrieval pipeline is working.

The Primary Metric: Recall@k

Recall@k answers: out of all the chunks that are relevant to this query, how many did we retrieve in the top-k?

pythonCopy
def recall_at_k(retrieved_ids: list[str], relevant_ids: set[str], k: int) -> float:
    top_k = set(retrieved_ids[:k])
    if not relevant_ids:
        return 1.0
    return len(top_k & relevant_ids) / len(relevant_ids)

# Evaluate across a test set
def evaluate_retrieval(test_cases, retriever, k=5):
    scores = []
    for case in test_cases:
        retrieved = retriever.query(case["question"], top_k=k)
        retrieved_ids = [r["chunk_id"] for r in retrieved]
        scores.append(recall_at_k(retrieved_ids, set(case["relevant_chunk_ids"]), k))
    return {"recall@k": sum(scores) / len(scores), "k": k, "n": len(scores)}

A recall@5 below 0.6 means you are handing the LLM incomplete information on 40% of queries. Improving recall from 0.6 to 0.85 will improve answer quality more than any prompt change.

Chunking Strategy

The chunk is the unit of retrieval. If your chunk is too large, it contains irrelevant sentences that dilute the embedding. If it is too small, you lose context that makes the chunk interpretable.

pythonCopy
from langchain.text_splitter import RecursiveCharacterTextSplitter

# Baseline: fixed-size with overlap
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,       # ~350-400 tokens for most models
    chunk_overlap=75,     # overlap preserves sentence context at boundaries
    separators=["\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_text(document)

# Better for structured docs: semantic splitting
# Split on section headers first, then recursively split sections
import re

def heading_aware_split(text: str, max_chunk: int = 500) -> list[str]:
    sections = re.split(r'\n#{1,3} ', text)
    result = []
    for section in sections:
        if len(section) <= max_chunk:
            result.append(section.strip())
        else:
            result.extend(splitter.split_text(section))
    return [c for c in result if c.strip()]

For technical documentation, heading-aware splitting consistently beats fixed-size. For dense prose (legal, medical), fixed-size with overlap is usually fine.

Embedding Model Choice

Not all embedding models are equal on your domain. The MTEB leaderboard is a useful starting point, but benchmark on your own data before committing.

pythonCopy
from sentence_transformers import SentenceTransformer
import numpy as np

def compare_embeddings(queries, corpus, relevant_map, model_names):
    results = {}
    for name in model_names:
        model = SentenceTransformer(name)
        corpus_embs = model.encode(corpus, normalize_embeddings=True, batch_size=64)
        query_embs = model.encode(queries, normalize_embeddings=True, batch_size=64)
        scores = query_embs @ corpus_embs.T  # cosine similarity (normalized)
        recalls = []
        for i, query in enumerate(queries):
            top5 = scores[i].argsort()[-5:][::-1].tolist()
            relevant = set(relevant_map[query])
            recalls.append(len(set(top5) & relevant) / max(len(relevant), 1))
        results[name] = sum(recalls) / len(recalls)
    return results

# Typical comparison
model_names = [
    "BAAI/bge-small-en-v1.5",    # fast, good baseline
    "BAAI/bge-large-en-v1.5",    # slower, usually better recall
    "text-embedding-3-small",     # OpenAI, strong on mixed domains
]

For code-heavy content, voyage-code-2 or text-embedding-3-large consistently outperform sentence-transformers. For English prose, bge-large is competitive at a fraction of the cost.

Why Bad Retrieval Causes Hallucination

When recall@k is low, the LLM receives partial context. It has two options: say "I don't know" (which it rarely does without explicit instructions) or fill the gap from its training distribution. That gap-filling is hallucination.

The pattern is predictable: if the answer to a question involves a specific number, name, or date, and the chunk containing that fact is not in the top-k, the model will often produce a plausible-looking but wrong value.

The fix is always retrieval-first: improve recall@k, then worry about prompt structure.

Retrieval Quality Checklist

Before touching your LLM prompts, verify:

• Recall@5 >= 0.75 on a labeled test set of 50+ queries
• Chunks average 300–600 tokens (not 50, not 2,000)
• Embedding model benchmarked on domain-similar data
• Queries and chunks in the same language (obvious, but often missed)
• Metadata filters applied before vector search to narrow candidate set

Common Mistakes

Using the default 1,000-token chunk size. Most default configs are too large. Smaller chunks with overlap usually improve recall.

Evaluating only the final answer quality. Measure retrieval independently. An 80% retrieval recall with a weak prompt beats 40% retrieval recall with a perfect prompt.

Ignoring the query-document gap. Some embedding models are trained for symmetric similarity (query and document are the same type). For asymmetric tasks (short question, long document), use an asymmetric model like BGE or use HyDE (generate a hypothetical answer, embed that).

Where to Go Next

See also: [rag-vector-search-systems], [rag-evaluation-failure-analysis], [llm-app-engineering-production]