Designing a Search and Ranking System

Search and ranking is where ML delivers some of its highest business value. Whether it is e-commerce product search, content recommendation, or job matching, the architecture has the same core structure. Understanding it well translates to both system design interviews and real product work.

What Makes Search a Two-Stage Problem

Naive ranking: score every document in your corpus against every query. For a corpus of 100 million products and 10,000 queries per second, this is not feasible. The standard architecture uses two stages:

Query
  │
  ▼
┌─────────────────────────────────────────────────────────────┐
│  Stage 1: Retrieval (Candidate Generation)                  │
│  Goal: Get from 100M → ~1000 candidates quickly             │
│  Methods: BM25 keyword index, ANN vector search,            │
│           collaborative filtering lookup                    │
│  Latency budget: 10–50ms                                    │
└─────────────────────────────────────────────────────────────┘
  │ ~1000 candidates
  ▼
┌─────────────────────────────────────────────────────────────┐
│  Stage 2: Ranking (Scoring)                                 │
│  Goal: Score 1000 candidates, return top 10-20              │
│  Methods: LambdaRank, LightGBM ranker, neural ranker        │
│  Features: query-doc relevance, user context,               │
│            popularity, freshness, personalization           │
│  Latency budget: 50–150ms                                   │
└─────────────────────────────────────────────────────────────┘
  │ top 10-20 results
  ▼
Display to user

Stage 1: Retrieval

Lexical retrieval (BM25): Fast, interpretable, good for exact keyword matches.

pythonCopy
# Using Elasticsearch for BM25 retrieval
from elasticsearch import Elasticsearch

es = Elasticsearch()

def lexical_retrieve(query: str, index: str, k: int = 500) -> list[dict]:
    response = es.search(
        index=index,
        body={
            "query": {
                "multi_match": {
                    "query": query,
                    "fields": ["title^3", "description^2", "tags"],
                    "type": "best_fields"
                }
            },
            "size": k
        }
    )
    return [hit["_source"] | {"lexical_score": hit["_score"]}
            for hit in response["hits"]["hits"]]

Semantic retrieval (vector search): Finds conceptually similar items even without keyword overlap.

pythonCopy
import anthropic
import numpy as np

client = anthropic.Anthropic()

def embed_query(text: str) -> list[float]:
    response = client.embeddings.create(
        model="voyage-3",
        input=[text]
    )
    return response.embeddings[0].embedding

def vector_retrieve(query: str, vector_db, k: int = 500) -> list[dict]:
    query_embedding = embed_query(query)
    # ANN search in pgvector, Pinecone, Weaviate, etc.
    results = vector_db.search(
        vector=query_embedding,
        top_k=k,
        include_metadata=True
    )
    return results

Hybrid retrieval: Merge both, using Reciprocal Rank Fusion (RRF):

pythonCopy
def reciprocal_rank_fusion(
    lexical_results: list[dict],
    semantic_results: list[dict],
    k: int = 60
) -> list[dict]:
    """
    RRF combines multiple ranked lists without needing to normalize scores.
    Score(doc) = Σ 1 / (k + rank_in_list_i)
    """
    rrf_scores = {}

    for rank, doc in enumerate(lexical_results, 1):
        doc_id = doc['id']
        rrf_scores[doc_id] = rrf_scores.get(doc_id, 0) + 1 / (k + rank)
        if doc_id not in [d['id'] for d in semantic_results]:
            rrf_scores[doc_id] = doc  # store doc data

    for rank, doc in enumerate(semantic_results, 1):
        doc_id = doc['id']
        rrf_scores[doc_id] = rrf_scores.get(doc_id, 0) + 1 / (k + rank)

    # Sort by RRF score
    sorted_ids = sorted(rrf_scores, key=lambda x: rrf_scores[x]
                        if isinstance(rrf_scores[x], float) else 0, reverse=True)
    return sorted_ids[:500]

Stage 2: Ranking

The ranking model scores each candidate given the query and user context. It uses learning-to-rank (LTR) approaches.

Feature engineering for ranking:

pythonCopy
def extract_ranking_features(query: str, doc: dict, user: dict) -> np.ndarray:
    return np.array([
        # Query-document relevance
        bm25_score(query, doc['text']),
        cosine_similarity(query_embedding, doc['embedding']),
        exact_title_match(query, doc['title']),
        query_term_coverage(query, doc),

        # Document quality
        doc['click_through_rate'],
        doc['conversion_rate'],
        doc['avg_rating'],
        doc['review_count'],
        np.log1p(doc['sales_count']),

        # Freshness
        days_since_published(doc['published_at']),

        # Personalization
        user_item_affinity(user['id'], doc['id']),  # from collaborative filtering
        category_preference_match(user['preferred_categories'], doc['category']),
        price_range_match(user['price_sensitivity'], doc['price']),
    ])

Training a LambdaRank model with LightGBM:

pythonCopy
import lightgbm as lgb
import pandas as pd

# Training data: query, doc, user features + relevance label
# Relevance: 0=irrelevant, 1=clicked, 2=purchased (graded relevance)

train_data = lgb.Dataset(
    X_train,
    label=y_train,
    group=query_group_sizes_train  # how many docs per query
)

params = {
    "objective": "lambdarank",
    "metric": "ndcg",
    "ndcg_eval_at": [5, 10],
    "learning_rate": 0.05,
    "num_leaves": 127,
    "min_data_in_leaf": 50,
    "lambdarank_truncation_level": 30
}

ranker = lgb.train(
    params,
    train_data,
    num_boost_round=500,
    valid_sets=[val_data],
    callbacks=[lgb.early_stopping(50), lgb.log_evaluation(100)]
)

Offline Metrics for Ranking

Metric	What it measures	Formula
NDCG@K	Quality of top-K results, weighted by position	Σ (2^rel_i - 1) / log2(i+1)
MRR	How high is the first relevant result?	1/rank_of_first_relevant
Recall@K	Does the correct answer appear in top K?	hits_in_top_K / total_relevant
MAP	Mean precision across multiple queries	mean of avg precision per query

NDCG@10 is the most commonly used metric for ranked search results.

Handling Query Understanding

The query "red running shoes size 10" needs to be decomposed:

• Intent: product search
• Category: shoes → running shoes
• Attributes: color=red, size=10

pythonCopy
def parse_query(raw_query: str) -> dict:
    """Extract structured attributes from freetext query."""
    # Simple rule-based (scale to ML for complex cases)
    return {
        "raw": raw_query,
        "normalized": raw_query.lower().strip(),
        "detected_size": extract_size(raw_query),    # "size 10" → {"US": 10}
        "detected_color": extract_color(raw_query),  # "red" → "red"
        "category_hint": classify_category(raw_query)  # "running shoes" → "footwear/running"
    }

Logging and Iteration

The training data for your ranker comes from user behavior. Log everything:

pythonCopy
# Log every impression, click, and conversion
def log_search_event(event_type: str, query: str, doc_id: str,
                     rank: int, user_id: str):
    event = {
        "event_type": event_type,  # impression, click, purchase
        "query": query,
        "doc_id": doc_id,
        "rank": rank,
        "user_id": user_id,
        "timestamp": time.time(),
        "session_id": get_session_id()
    }
    analytics_log.append(event)

Build your training labels from this log: impressions shown without clicks are negative signals; clicks and purchases are positive signals. This is implicit feedback - noisier than explicit ratings but available at massive scale.

The Full System Architecture

┌────────────────────────────────────────────────────────────────────┐
│  Offline (batch):                                                  │
│  Raw data → Feature pipeline → Training → Model registry          │
│  Document store → Embedding pipeline → Vector index               │
│  User events → Label extraction → Training set refresh            │
└────────────────────────────────────────────────────────────────────┘
                          ↕ periodic update
┌────────────────────────────────────────────────────────────────────┐
│  Online (request):                                                 │
│  Query → [Query understanding] → [Retrieval: BM25 + ANN]          │
│        → [Feature extraction] → [Ranking model]                   │
│        → [Post-processing: dedup, filters, diversity]              │
│        → Results → Log impression                                  │
└────────────────────────────────────────────────────────────────────┘

The offline pipeline refreshes the ranker (weekly to daily) and the embedding index (hourly to daily for new content).

Common Design Pitfalls

Position bias: Users click higher-ranked results more. Your training data conflates "good result" with "high-ranked result." Mitigate with Inverse Propensity Scoring (IPS) or A/B testing with randomized position.

Popularity bias: Your ranker learns to rank popular items higher regardless of relevance. Add diversity constraints or re-rank to ensure long-tail items get exposure.

Cold start: New items have no click data. Use content-based features exclusively until enough interaction data accumulates.

Query volume distribution: 80% of queries occur once. Your offline NDCG is dominated by high-volume head queries. Monitor tail query quality separately.