Search, Ranking, Recommendation, and Personalization Systems

Search, ranking, and recommendation systems are the highest-value AI applications at most internet companies. Every time a user sees a feed, search result, product recommendation, or content suggestion, a ranking or recommendation system made that decision. These systems are mature, well-studied, and increasingly powered by the same neural architectures used in LLMs. Understanding them is a high-leverage specialization for ML engineers.

The Recommendation Problem

Given a user and a catalog of items, recommend items the user would engage with. The challenge: catalogs are millions of items; inference must be fast (milliseconds); user preferences are implicit (clicks, purchases, dwell time) rather than explicit (explicit ratings are rare and biased).

The standard architecture is a two-stage pipeline:

Candidate Generation (millions → hundreds)
         ↓
Scoring / Ranking (hundreds → tens)
         ↓
Business Logic Filtering (dedup, diversity, promo)
         ↓
Final Ranked List

Stage 1: Candidate Generation with Embeddings

pythonCopy
import torch
import torch.nn as nn
import torch.nn.functional as F

class TwoTowerModel(nn.Module):
    """
    Learns separate embedding towers for users and items.
    At training time: push interacted (user, item) pairs together.
    At serving time: pre-compute item embeddings and use ANN search.
    """
    def __init__(self, n_users: int, n_items: int, embed_dim: int = 128):
        super().__init__()
        self.user_tower = nn.Sequential(
            nn.Embedding(n_users, 256),
            nn.Linear(256, embed_dim),
            nn.ReLU(),
        )
        self.item_tower = nn.Sequential(
            nn.Embedding(n_items, 256),
            nn.Linear(256, embed_dim),
            nn.ReLU(),
        )

    def forward(self, user_ids, item_ids):
        user_emb = F.normalize(self.user_tower(user_ids), dim=-1)
        item_emb = F.normalize(self.item_tower(item_ids), dim=-1)
        return (user_emb * item_emb).sum(dim=-1)  # Cosine similarity

# Training: in-batch negatives (treat other items in the batch as negatives)
def in_batch_loss(user_embs, item_embs, temperature: float = 0.07):
    logits = user_embs @ item_embs.T / temperature
    labels = torch.arange(len(user_embs))
    return F.cross_entropy(logits, labels)

At serving time: pre-compute all item embeddings → build a FAISS index → compute user embedding at query time → ANN search. Scales to millions of items with millisecond retrieval.

Stage 2: Learning to Rank

The ranking model takes (user, item, context) features and predicts a utility score. Features are richer than in the embedding model - they can include cross-features, time-based signals, and content embeddings.

pythonCopy
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.preprocessing import StandardScaler
import pandas as pd

def build_ranking_features(user_df, item_df, interaction_df) -> pd.DataFrame:
    """Join user, item, and interaction features into ranking training data."""
    df = interaction_df.merge(user_df, on="user_id").merge(item_df, on="item_id")

    # Cross features
    df["user_item_category_match"] = (df["user_fav_category"] == df["item_category"]).astype(int)
    df["recency_score"] = 1 / (1 + df["item_age_days"])

    # Label: did the user convert (purchase/watch/click-through) after impression?
    # Build from interaction_df: impression → conversion events within window
    return df

# Pointwise ranking: predict P(click) per item, rank by score
X_train = ranking_df[feature_cols]
y_train = ranking_df["clicked"]

ranking_model = GradientBoostingClassifier(
    n_estimators=500, max_depth=5, learning_rate=0.05,
    subsample=0.8, min_samples_leaf=20,
)
ranking_model.fit(X_train, y_train)

# Feature importance analysis
importances = pd.Series(ranking_model.feature_importances_, index=feature_cols)
print(importances.sort_values(ascending=False).head(10))

Pairwise ranking (LambdaMART, RankNet) - optimizes that a positive item ranks above a negative item - produces better-ordered lists than pointwise P(click) ranking. Use when list quality matters more than individual score calibration.

Offline Evaluation Metrics

Metric	Measures	When to use
AUC-ROC	Classification quality (click prediction)	Pointwise models
NDCG@k	Normalized ranked order quality	Ranking models
Recall@k	Are relevant items in top-k?	Candidate generation
MRR	Mean Reciprocal Rank - how early is the first relevant item?	Search

pythonCopy
import numpy as np

def ndcg_at_k(recommended: list, relevant: set, k: int = 10) -> float:
    """
    Normalized Discounted Cumulative Gain.
    Rewards relevant items appearing higher in the ranked list.
    """
    dcg = sum(
        1 / np.log2(i + 2)  # position i (0-indexed), log base 2, position penalty
        for i, item in enumerate(recommended[:k])
        if item in relevant
    )
    ideal_hits = min(len(relevant), k)
    idcg = sum(1 / np.log2(i + 2) for i in range(ideal_hits))
    return dcg / idcg if idcg > 0 else 0.0

def recall_at_k(recommended: list, relevant: set, k: int = 10) -> float:
    return len(set(recommended[:k]) & relevant) / len(relevant) if relevant else 0.0

Online Evaluation: A/B Testing Ranking Changes

Offline metrics do not fully predict business impact. Always validate ranking changes in an A/B test:

pythonCopy
from scipy import stats

def analyze_ranking_ab_test(
    control_clicks: list[int],
    treatment_clicks: list[int],
) -> dict:
    """
    Test whether treatment ranking produces significantly more clicks.
    control_clicks/treatment_clicks: list of click counts per session.
    """
    stat, p_value = stats.mannwhitneyu(treatment_clicks, control_clicks, alternative="greater")
    control_mean = np.mean(control_clicks)
    treatment_mean = np.mean(treatment_clicks)
    lift = (treatment_mean - control_mean) / control_mean

    return {
        "control_mean": control_mean,
        "treatment_mean": treatment_mean,
        "relative_lift": lift,
        "p_value": p_value,
        "significant": p_value < 0.05,
    }

Common pitfalls in recommendation A/B tests:

• Novelty effect: users click on new content initially regardless of quality; run for ≥ 2 weeks
• Spillover: users in control may see items recommended to treatment group in social feeds
• Engagement vs. long-term value: optimizing CTR may degrade user satisfaction over time

Cold Start: Handling New Users and Items

pythonCopy
def get_candidates_cold_start(user: dict, catalog_df: pd.DataFrame, k: int = 50) -> list:
    """
    For users with no interaction history, fall back to:
    1. Popular items in user's demographic segment
    2. Content-based similarity to explicitly stated preferences
    3. Global trending items
    """
    if user.get("age_group") and user.get("category_preference"):
        # Segment-based popularity
        segment_popular = (
            catalog_df[
                (catalog_df["target_age_group"] == user["age_group"]) &
                (catalog_df["category"] == user["category_preference"])
            ]
            .sort_values("popularity_score", ascending=False)
            .head(k)
        )
        if len(segment_popular) >= k // 2:
            return segment_popular["item_id"].tolist()

    # Global popular fallback
    return catalog_df.sort_values("popularity_score", ascending=False).head(k)["item_id"].tolist()

Cold start is an unsolved problem in recommendation. Production systems combine multiple strategies and transition from cold-start to personalized recommendations as interaction data accumulates (typically 3-10 interactions).

Common Mistakes and Bad Instincts

Optimizing purely for engagement. CTR and watch time are easy to measure but may not correlate with user value or business goals. A recommendation system optimized for CTR may surface clickbait. Define the business metric carefully before choosing the optimization target.

Not evaluating cold start separately. Cold start performance is often much worse than the headline NDCG metric, which is dominated by warm users. Always slice eval metrics by user recency/activity level.

Ignoring position bias in training data. Items shown at position 1 get clicked more than identical items at position 10 because of position, not quality. Naive training on click data encodes this bias. Use inverse propensity scoring (IPS) or unbiased learning-to-rank techniques.

Where to Go Next

• capstone-build-and-operate-a-production-style-ai-system: apply the full ML engineering toolkit to build a recommendation or search feature as your capstone
• portfolio-conversion-turning-engineering-work-into-ml-evidence: frame recommendation system work as evidence for ML engineering roles