RAG and Vector Search Systems

Vector search is the retrieval engine inside every RAG system. Most engineers plug in a library and move on. That works until you hit a recall problem at scale - and then you need to understand what is actually happening inside the index.

This article covers how vector search works, the major index algorithms, when to use each, and how to benchmark your choice.

What Vector Search Is Actually Doing

Approximate nearest neighbor (ANN) search finds the k vectors closest to a query vector in high-dimensional space. "Approximate" is the key word - all production ANN algorithms trade some recall for speed.

pythonCopy
import numpy as np

def exact_nearest_neighbors(query: np.ndarray, corpus: np.ndarray, k: int) -> list[int]:
    """Exact kNN - O(n) per query, baseline for recall measurement."""
    similarities = corpus @ query  # assumes normalized vectors
    return np.argsort(similarities)[-k:][::-1].tolist()

def recall_at_k_vs_exact(ann_results: list[int], exact_results: list[int]) -> float:
    return len(set(ann_results) & set(exact_results)) / len(exact_results)

Your ANN index's recall should be measured against exact kNN on a representative sample. A recall@10 of 0.95 means for every 10 true nearest neighbors, your ANN index returns 9.5 of them on average.

HNSW vs IVF

These are the two dominant index families.

HNSW (Hierarchical Navigable Small World) builds a multi-layer graph. Search starts at the top layer and greedily navigates toward the query, then refines at each layer below.

pythonCopy
import hnswlib

dim = 768
index = hnswlib.Index(space='cosine', dim=dim)
index.init_index(max_elements=100_000, ef_construction=200, M=16)
# M: number of bidirectional links per node (higher = better recall, more RAM)
# ef_construction: search width during build (higher = better recall, slower build)

index.add_items(corpus_embeddings)
index.set_ef(50)  # ef at query time: higher = better recall, slower query

labels, distances = index.knn_query(query_embedding, k=10)

HNSW is the default choice for datasets under ~10M vectors. It has high recall, fast queries, but uses significant RAM (roughly 4 bytes * dim * M * n_vectors).

IVF (Inverted File Index with FAISS) clusters vectors into nlist Voronoi cells. Search only examines the nprobe nearest clusters.

pythonCopy
import faiss

dim = 768
nlist = 1000  # number of clusters
nprobe = 50   # clusters to search at query time (recall/speed tradeoff)

quantizer = faiss.IndexFlatL2(dim)
index = faiss.IndexIVFFlat(quantizer, dim, nlist, faiss.METRIC_INNER_PRODUCT)
index.train(corpus_embeddings.astype('float32'))
index.add(corpus_embeddings.astype('float32'))
index.nprobe = nprobe

distances, indices = index.search(query_embedding.reshape(1, -1).astype('float32'), k=10)

IVF uses much less RAM than HNSW. It is the choice for >10M vectors or memory-constrained environments. Set nprobe = nlist * 0.05 as a starting point and tune up for recall.

Cosine vs Dot Product

Cosine similarity normalizes both vectors; dot product does not. For most embedding models, vectors are normalized at output, making them identical. Verify:

pythonCopy
embeddings = model.encode(texts, normalize_embeddings=True)
norms = np.linalg.norm(embeddings, axis=1)
print(f"Norm range: {norms.min():.4f} – {norms.max():.4f}")  # should be ~1.0
# If normalized: cosine == dot product, use dot product (faster)
# If not normalized: use cosine

Using dot product on unnormalized vectors will give wrong results. When in doubt, normalize and use cosine.

pgvector vs FAISS vs Dedicated Vector DBs

Use case                   → Choice
───────────────────────────────────────────────────────
< 500K vectors, SQL joins  → pgvector (HNSW or IVFFlat)
< 50M vectors, standalone  → FAISS (in-memory or on-disk)
> 50M vectors, managed     → Pinecone, Weaviate, Qdrant
Multi-modal + metadata     → Weaviate or Qdrant
Already on Postgres        → pgvector first, migrate later

pgvector is dramatically underrated. For most RAG applications (hundreds of thousands of documents), it delivers production-quality recall with zero additional infrastructure.

sqlCopy
-- pgvector setup
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    content TEXT,
    embedding vector(768)
);
CREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops)
    WITH (m = 16, ef_construction = 64);

-- Query
SELECT id, content, 1 - (embedding <=> $1::vector) AS similarity
FROM documents
ORDER BY embedding <=> $1::vector
LIMIT 10;

Common Mistakes

Not benchmarking recall on your own data. MTEB benchmarks are on English text. If your corpus is code, multilingual, or domain-specific, rebuild the benchmark on a sample of your own data.

Treating nprobe/ef as a fixed parameter. These are runtime parameters. Tune them based on your latency budget. A query-time ef=200 gives higher recall than ef=50 with ~2x the latency.

Using L2 distance when your model outputs cosine embeddings. Many models are trained with cosine similarity. Using L2 on their outputs degrades recall significantly. Match the distance metric to the model.

Not batching index construction. Building an index with add_items one vector at a time is 10–100x slower than batching. Always add in chunks of at least 1,000.

Where to Go Next

See also: [rag-foundations-retrieval-quality], [rag-evaluation-failure-analysis], [inference-optimization-performance]