Embeddings, Retrieval, and RAG Systems

Dense embeddings and retrieval-augmented generation (RAG) are the two architectural pillars of modern AI product work. Embeddings encode meaning as points in geometric space - enabling semantic search, deduplication, and recommendation. RAG uses embeddings to retrieve relevant documents at query time and inject them into an LLM's context, solving the knowledge-freshness and hallucination problems that limit standalone LLMs. Together they power enterprise search, document Q&A, code assistants, and customer support systems.

What Embeddings Are

An embedding is a dense vector representation of an object - text, image, user, product - such that semantically similar objects are close together in the vector space.

pythonCopy
from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer('all-MiniLM-L6-v2')

sentences = [
    "gradient descent minimizes the loss function",
    "backpropagation computes parameter gradients",
    "the weather is sunny today",
]
embeddings = model.encode(sentences, normalize_embeddings=True)
print(embeddings.shape)  # [3, 384]

# Semantic similarity via dot product (cosine sim for normalized vectors)
sim_matrix = embeddings @ embeddings.T
print(sim_matrix.round(3))
# [[1.    0.72  0.11]
#  [0.72  1.    0.09]
#  [0.11  0.09  1.  ]]
# The first two sentences are 0.72 similar; unrelated to the third (0.11)

Embeddings from a pretrained model encode the semantic content learned from billions of examples. A dot product between two normalized embeddings measures cosine similarity - the angle between them. Similar meaning → small angle → large cosine similarity.

Embedding Model Selection

Model	Dimensions	Speed	Use Case
all-MiniLM-L6-v2	384	Fast	Prototyping, low-resource inference
text-embedding-3-small (OpenAI)	1536	API call	Production, balanced cost/quality
voyage-3 (Voyage AI)	1024	API call	Best retrieval quality, 2025 benchmark
nomic-embed-text	768	Local GPU	Good open-source option

For production RAG: use an API-based embedding model so you don't manage GPU infrastructure for the embedding step. For local/offline applications, nomic-embed-text with Ollama is a practical choice.

Vector Databases: Storing and Searching Embeddings

A vector database stores embeddings and answers the query: "which of these N vectors is closest to this query vector?" using approximate nearest neighbor (ANN) search.

pythonCopy
import faiss
import numpy as np

d = 384  # embedding dimension
n = 100_000  # number of vectors

# Build index
# IndexFlatIP: exact cosine similarity (for normalized vectors), good up to ~1M
index = faiss.IndexFlatIP(d)

# For larger datasets, use IVF for approximate but much faster search
# index = faiss.index_factory(d, "IVF100,Flat")
# index.train(embeddings_matrix)

embeddings_matrix = np.random.randn(n, d).astype('float32')
faiss.normalize_L2(embeddings_matrix)  # normalize for cosine similarity
index.add(embeddings_matrix)

# Query
query_vec = np.random.randn(1, d).astype('float32')
faiss.normalize_L2(query_vec)
scores, indices = index.search(query_vec, k=5)
print(f"Top-5 most similar: {indices[0]}, scores: {scores[0].round(3)}")

For this project's PostgreSQL stack, pgvector is the right choice - it runs inside your existing database:

sqlCopy
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    content TEXT,
    source TEXT,
    embedding vector(384)
);
CREATE INDEX ON documents USING ivfflat (embedding vector_cosine_ops)
  WITH (lists = 100);

-- Similarity search
SELECT id, content, 1 - (embedding <=> $1::vector) AS similarity
FROM documents
ORDER BY embedding <=> $1::vector
LIMIT 5;

The RAG Architecture

RAG combines a retriever (embedding model + vector database) with a generator (LLM) into a pipeline that answers questions grounded in your documents:

Query → Embed Query → Vector Search → Top-K Chunks → Build Prompt → LLM → Answer

Document Ingestion and Chunking

pythonCopy
from pathlib import Path
import re

def chunk_document(text: str, chunk_size: int = 400, overlap: int = 50) -> list[str]:
    paragraphs = [p.strip() for p in re.split(r'
{2,}', text) if p.strip()]
    chunks, current, current_len = [], [], 0
    for para in paragraphs:
        para_len = len(para.split())
        if current_len + para_len > chunk_size and current:
            chunks.append(' '.join(current))
            overlap_words = ' '.join(current).split()[-overlap:]
            current = [' '.join(overlap_words)]
            current_len = len(overlap_words)
        current.append(para)
        current_len += para_len
    if current:
        chunks.append(' '.join(current))
    return chunks

Overlap ensures that context spanning a chunk boundary is captured. 50-word overlap on 400-word chunks is a reasonable starting point.

The Retriever

pythonCopy
embedding_model = SentenceTransformer('all-MiniLM-L6-v2')

def retrieve(query: str, index, records: list[dict], k: int = 5) -> list[dict]:
    q_emb = embedding_model.encode([query], normalize_embeddings=True).astype('float32')
    scores, indices = index.search(q_emb, k)
    return [
        {**records[idx], 'similarity': float(scores[0][i])}
        for i, idx in enumerate(indices[0]) if idx != -1
        and scores[0][i] >= 0.4  # filter low-relevance chunks
    ]

Hybrid search - combining dense (semantic) retrieval with sparse (BM25) retrieval - consistently outperforms either alone. Use it when your queries include exact model names, error codes, or other specific terms that semantic search struggles with.

The Generator

pythonCopy
import anthropic, json

client = anthropic.Anthropic()

def generate_answer(query: str, index, records: list[dict]) -> dict:
    chunks = retrieve(query, index, records, k=5)
    if not chunks:
        return {'answer': 'No relevant information found.', 'sources': []}

    context = '

---

'.join(
        f"[Source: {c['source']}]
{c['content']}" for c in chunks
    )

    response = client.messages.create(
        model='claude-opus-4-7',
        max_tokens=1024,
        temperature=0,
        system='Answer using ONLY the provided context. Cite sources. Say "I don't know" if the context is insufficient.',
        messages=[{'role': 'user', 'content': f"Context:
{context}

Question: {query}"}]
    )
    return {
        'answer': response.content[0].text,
        'sources': list({c['source'] for c in chunks}),
    }

Evaluating Retrieval Quality

Never tune the generator before the retrieval is working well. Retrieval recall@k tells you whether the right information is reaching the LLM at all:

pythonCopy
def recall_at_k(eval_pairs: list[tuple[str, str]], index, records, k=5) -> float:
    """Fraction of queries where expected_doc is in top-k retrieved sources."""
    hits = sum(
        any(r['source'] == expected for r in retrieve(query, index, records, k=k))
        for query, expected in eval_pairs
    )
    return hits / len(eval_pairs)

# Build a small eval set of (query, expected_source) pairs manually
eval_set = [
    ("how does gradient descent work", "gradient-descent-from-scratch.md"),
    ("what is attention mechanism", "attention-mechanism-explained.md"),
]
print(f"Recall@5: {recall_at_k(eval_set, index, records):.2f}")

Aim for Recall@5 > 0.85 before moving to generation quality.

Common Mistakes and Bad Instincts

Not splitting at semantic boundaries. Chunking at fixed character counts splits sentences and concepts in half. Always split at paragraph or sentence boundaries first, then enforce size.

Using different embedding models for indexing and querying. The embedding space is model-specific. If you switch models, re-embed the entire knowledge base.

No similarity threshold. A RAG system with no threshold will always return k chunks, even when all k are irrelevant. Add a minimum similarity cutoff (typically 0.4–0.6) and return a "no information" response rather than hallucinating from noise.

Evaluating only generation quality, not retrieval quality. Bad answers can come from bad retrieval (wrong chunks) or bad generation (correct chunks, wrong answer). Measure them separately.

Context stuffing. More chunks is not always better. The LLM's ability to extract the correct answer degrades when irrelevant chunks dilute the context. Top-5 with good filtering beats top-20 without.

Where to Go Next

• Module 23 (Agents, Tool Use, and Workflow Orchestration) builds on RAG by adding tool-calling and multi-step reasoning to LLM systems.
• Phase 4 (MLOps) covers operating RAG in production: embedding model updates, index refresh pipelines, latency monitoring.
• The standalone post rag-system-design-and-production-patterns covers advanced patterns: query rewriting, cross-encoder reranking, and multi-hop retrieval.

What to Practice Next

• Embed a small corpus (50-100 documents) with sentence-transformers and build a FAISS index; run 10 queries, inspect the top-3 retrieved chunks, and note where the results feel wrong - this surfaces chunking and embedding quality issues quickly.
• Measure recall@5 on a hand-labeled Q&A set before and after swapping embedding models (e.g., all-MiniLM-L6-v2 vs. text-embedding-3-small) - quantify the difference rather than guessing.
• Add a re-ranking step using a cross-encoder and measure whether it improves recall@5 on your labeled set; document the latency cost of the additional inference pass.