Retrieval Systems, Vector Databases, and RAG

Retrieval-Augmented Generation (RAG) is the dominant architecture for grounding LLMs in external knowledge. The core problem it solves: LLMs have a knowledge cutoff, cannot access private data, and hallucinate when asked about content not in their training. RAG addresses all three by retrieving relevant documents at query time and providing them as context.

In production, "RAG" is rarely a single pattern. It is a retrieval pipeline with multiple retrieval strategies, ranking, and careful context assembly. This module covers the full system from document ingestion to answer generation.

The Basic RAG Architecture

Documents → Chunk → Embed → Vector Store
                                  ↓
User Query → Embed → Retrieve (top-k) → Rerank → Assemble Prompt → LLM → Answer

Each step has meaningful design choices. Let's build it bottom-up.

Document Chunking

How you chunk determines retrieval quality more than any other single decision:

pythonCopy
from langchain.text_splitter import RecursiveCharacterTextSplitter
import re

def chunk_document(text: str, chunk_size: int = 512, chunk_overlap: int = 64):
    """
    Recursive splitter: tries paragraph → sentence → word boundaries.
    Preserves semantic units better than fixed-size splits.
    """
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=chunk_overlap,
        separators=["\n\n", "\n", ". ", " "],
    )
    return splitter.split_text(text)

# For structured docs (markdown, code), split on semantic boundaries
def chunk_markdown(text: str):
    sections = re.split(r'\n#{1,3} ', text)
    return [s.strip() for s in sections if len(s.strip()) > 50]

Chunking rules of thumb:

• Chunk size 256–512 tokens works for most question-answering tasks
• Larger chunks (1024+) work better for synthesis tasks that need context across sections
• Always overlap chunks to avoid splitting a sentence at a boundary
• For code: chunk by function/class, not by line count

Embedding and Storing Documents

pythonCopy
from sentence_transformers import SentenceTransformer
import numpy as np
import psycopg2

model = SentenceTransformer("BAAI/bge-base-en-v1.5")

def ingest_documents(docs: list[dict], conn):
    """docs: list of {id, text, source, metadata}"""
    cursor = conn.cursor()
    for doc in docs:
        chunks = chunk_document(doc["text"])
        embeddings = model.encode(chunks, normalize_embeddings=True, batch_size=64)
        for i, (chunk, emb) in enumerate(zip(chunks, embeddings)):
            cursor.execute(
                """
                INSERT INTO document_chunks (document_id, chunk_index, content, embedding, source)
                VALUES (%s, %s, %s, %s::vector, %s)
                """,
                (doc["id"], i, chunk, emb.tolist(), doc["source"])
            )
    conn.commit()

pgvector schema:

sqlCopy
CREATE TABLE document_chunks (
    id          BIGSERIAL PRIMARY KEY,
    document_id TEXT NOT NULL,
    chunk_index INT NOT NULL,
    content     TEXT NOT NULL,
    embedding   vector(768) NOT NULL,
    source      TEXT,
    created_at  TIMESTAMPTZ DEFAULT now()
);

CREATE INDEX ON document_chunks USING hnsw (embedding vector_cosine_ops)
    WITH (m = 16, ef_construction = 64);

Retrieval: Vector Search + BM25 Hybrid

Pure vector search has a known weakness: it finds semantically similar content but can miss exact keyword matches. Hybrid search combines dense (vector) and sparse (BM25/keyword) retrieval:

pythonCopy
def hybrid_search(query: str, conn, k: int = 20, alpha: float = 0.5):
    """
    alpha=1.0: pure vector search
    alpha=0.0: pure BM25 keyword search
    alpha=0.5: balanced hybrid
    """
    query_emb = model.encode([query], normalize_embeddings=True)[0]
    cursor = conn.cursor()

    # Dense retrieval via pgvector
    cursor.execute("""
        SELECT id, content, source,
               1 - (embedding <=> %s::vector) AS vector_score
        FROM document_chunks
        ORDER BY embedding <=> %s::vector
        LIMIT %s
    """, (query_emb.tolist(), query_emb.tolist(), k * 2))
    dense_results = {row[0]: dict(id=row[0], content=row[1], source=row[2], vector_score=row[3])
                     for row in cursor.fetchall()}

    # Sparse retrieval via PostgreSQL full-text search
    cursor.execute("""
        SELECT id, content, source,
               ts_rank(to_tsvector('english', content), plainto_tsquery('english', %s)) AS bm25_score
        FROM document_chunks
        WHERE to_tsvector('english', content) @@ plainto_tsquery('english', %s)
        ORDER BY bm25_score DESC
        LIMIT %s
    """, (query, query, k * 2))
    sparse_results = {row[0]: dict(id=row[0], content=row[1], source=row[2], bm25_score=row[3])
                      for row in cursor.fetchall()}

    # Reciprocal Rank Fusion
    all_ids = set(dense_results) | set(sparse_results)
    scores = {}
    for rank, chunk_id in enumerate(sorted(dense_results, key=lambda x: dense_results[x]["vector_score"], reverse=True)):
        scores[chunk_id] = scores.get(chunk_id, 0) + alpha * (1 / (rank + 60))
    for rank, chunk_id in enumerate(sorted(sparse_results, key=lambda x: sparse_results[x]["bm25_score"], reverse=True)):
        scores[chunk_id] = scores.get(chunk_id, 0) + (1 - alpha) * (1 / (rank + 60))

    top_ids = sorted(scores, key=scores.get, reverse=True)[:k]
    return [dense_results.get(i) or sparse_results[i] for i in top_ids]

Reranking Retrieved Chunks

The top-k retrieved chunks may contain noise. A cross-encoder reranker reads the query and each candidate together and produces a relevance score:

pythonCopy
from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def rerank(query: str, candidates: list[dict], top_n: int = 5) -> list[dict]:
    pairs = [(query, c["content"]) for c in candidates]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [item for item, _ in ranked[:top_n]]

Cross-encoders are more accurate than bi-encoders for relevance ranking but too slow for full-corpus search. Use them as a second-stage reranker over the top-20 or top-50 bi-encoder results.

Prompt Assembly and Answer Generation

pythonCopy
from openai import OpenAI

client = OpenAI()

def answer_with_rag(query: str, conn) -> dict:
    # Retrieve and rerank
    candidates = hybrid_search(query, conn, k=20)
    top_chunks = rerank(query, candidates, top_n=5)

    # Assemble context
    context = "\n\n---\n\n".join([
        f"Source: {c['source']}\n{c['content']}"
        for c in top_chunks
    ])

    # Generate answer
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": (
                    "Answer the question using only the provided context. "
                    "If the answer is not in the context, say so explicitly. "
                    "Cite the source for each claim."
                ),
            },
            {
                "role": "user",
                "content": f"Context:\n{context}\n\nQuestion: {query}",
            },
        ],
        temperature=0.0,
    )

    return {
        "answer": response.choices[0].message.content,
        "sources": list({c["source"] for c in top_chunks}),
        "chunks_used": len(top_chunks),
    }

Evaluating RAG Systems

Three metrics to track:

Metric	What it measures	Tool
Retrieval recall@k	Are the relevant docs in the top-k?	Ground truth QA pairs
Context precision	What fraction of retrieved chunks are actually used?	LLM judge
Answer faithfulness	Does the answer match the retrieved context?	Ragas, TruLens

pythonCopy
# Build an eval dataset: 50-100 question-answer pairs with known source documents
# Measure: did the correct source appear in top-5? Did the answer match?

Common Mistakes and Bad Instincts

Using the same chunk size for all document types. FAQ entries need small chunks (64–128 tokens). Long-form documents need larger ones (512+). Technical documentation with code blocks needs section-level chunking. One-size-fits-all chunking produces mediocre retrieval across document types.

Not filtering by metadata before vector search. If your corpus contains documents for multiple customers or knowledge domains, pre-filter by tenant/category using SQL WHERE clauses before the vector search. Retrieving across irrelevant segments wastes context window and confuses the model.

Ignoring retrieval quality and only measuring answer quality. Bad answers often trace to retrieval failures (wrong or irrelevant chunks) rather than LLM failures. Always measure retrieval recall separately.

Where to Go Next

• agents-tools-and-workflow-graphs: add tool use and multi-step reasoning on top of RAG
• fine-tuning-adaptation-and-when-not-to-fine-tune: when to fine-tune the embedding model vs. improving the retrieval pipeline
• observability-drift-feedback-loops-and-llm-evals: build monitoring for RAG system quality in production

What to Practice Next

• Stand up a local Chroma or Qdrant instance, index 100+ documents, and run 10 semantic queries - inspect the metadata and distance scores returned, then tune the similarity threshold to reduce false positives.
• Implement hybrid search: combine BM25 keyword scores with dense vector scores using reciprocal rank fusion, and compare recall@5 against dense-only retrieval on a labeled test set.
• Benchmark three vector index types available in your chosen vector DB (flat, HNSW, IVF) on the same dataset for query latency and recall - produce a comparison table that you could use to justify an architecture choice.