Recommendation Systems Engineering: From Collaborative Filtering to Two-Tower Models

Recommendation systems power Netflix, Spotify, Amazon, TikTok, and LinkedIn. They account for a significant portion of revenue at every major consumer platform. Understanding how they work - end to end - is one of the most valuable skills in applied ML.

This post covers the full engineering stack: retrieval, ranking, re-ranking, and the models at each stage.

The Core Problem

A recommendation system answers: "Given a user in a context, which items should we surface?"

The challenge is scale. A platform might have 100M users and 10M items. Computing scores for all (user, item) pairs at query time is impossible. Production systems solve this with a multi-stage pipeline:

All items (10M)
   ↓ Retrieval / Candidate generation (~1K candidates)
   ↓ Ranking (score and sort 1K candidates)
   ↓ Re-ranking (business rules, diversity, freshness)
   ↓ Final recommendations (10–50 items shown)

Each stage trades recall for speed. The retrieval stage must cast a wide net (high recall). The ranking stage can afford expensive models on a small candidate set.

Stage 1: Retrieval

Collaborative Filtering (Memory-Based)

The oldest and most intuitive approach: users who agreed in the past will agree in the future.

User-based CF: to recommend items for user A, find users most similar to A, return items they liked that A hasn't seen.

pythonCopy
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

# user_item_matrix: rows = users, cols = items, values = ratings (0 if unseen)
similarity = cosine_similarity(user_item_matrix)

def recommend_user_based(user_id, user_item_matrix, similarity, top_k=10):
    similar_users = np.argsort(-similarity[user_id])[1:50]  # top 50 similar
    candidate_items = user_item_matrix[similar_users].sum(axis=0)
    # Remove items the user has already seen
    already_seen = user_item_matrix[user_id] > 0
    candidate_items[already_seen] = -1
    return np.argsort(-candidate_items)[:top_k]

Item-based CF: instead of similar users, find items similar to what the user has interacted with.

Limitations: memory-based CF doesn't scale to millions of users/items and suffers from cold start (new users or items have no history).

Matrix Factorization

Decompose the user-item interaction matrix into low-dimensional user and item embeddings.

R ≈ U · Vᵀ

R: user-item matrix (n_users × n_items)
U: user embeddings (n_users × d)
V: item embeddings (n_items × d)

ALS (Alternating Least Squares): alternately fix V and optimize U, then fix U and optimize V. Efficient for sparse matrices.

pythonCopy
from implicit import als

model = als.AlternatingLeastSquares(factors=64, iterations=20, regularization=0.01)
model.fit(user_item_matrix.T)  # implicit expects item-user matrix

user_id = 42
recommendations = model.recommend(user_id, user_item_matrix[user_id])

The resulting user and item embeddings are dense representations that capture latent preferences - users with similar embeddings like similar things, items with similar embeddings attract similar users.

Two-Tower Model

The modern production standard for retrieval. Trains user and item encoders separately, then retrieves by approximate nearest neighbor search over item embeddings.

Architecture:
User features → [User Tower MLP] → user embedding (d-dim)
Item features → [Item Tower MLP] → item embedding (d-dim)

Training objective: maximize dot product for positive (user, item) pairs,
                    minimize for negative pairs

pythonCopy
import torch
import torch.nn as nn

class TwoTowerModel(nn.Module):
    def __init__(self, user_dim, item_dim, embed_dim=64):
        super().__init__()
        self.user_tower = nn.Sequential(
            nn.Linear(user_dim, 128),
            nn.ReLU(),
            nn.Linear(128, embed_dim)
        )
        self.item_tower = nn.Sequential(
            nn.Linear(item_dim, 128),
            nn.ReLU(),
            nn.Linear(128, embed_dim)
        )

    def forward(self, user_features, item_features):
        user_emb = self.user_tower(user_features)
        item_emb = self.item_tower(item_features)
        # Normalize for cosine similarity retrieval
        user_emb = nn.functional.normalize(user_emb, dim=-1)
        item_emb = nn.functional.normalize(item_emb, dim=-1)
        return user_emb, item_emb

    def score(self, user_emb, item_emb):
        return (user_emb * item_emb).sum(dim=-1)   # dot product

Training:

pythonCopy
model = TwoTowerModel(user_dim=100, item_dim=50)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

for user_features, pos_item_features, neg_item_features in dataloader:
    user_emb, pos_emb = model(user_features, pos_item_features)
    _, neg_emb = model(user_features, neg_item_features)

    pos_score = model.score(user_emb, pos_emb)
    neg_score = model.score(user_emb, neg_emb)

    # Pairwise loss: push positive score above negative
    loss = torch.clamp(1.0 - pos_score + neg_score, min=0).mean()
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

Serving: pre-compute all item embeddings. At query time, compute user embedding, then run approximate nearest neighbor (ANN) search over the item embedding index.

pythonCopy
import faiss

# Offline: build index
item_embeddings = compute_item_embeddings(all_items)   # (n_items, d)
index = faiss.IndexFlatIP(embed_dim)   # inner product (for normalized = cosine)
index.add(item_embeddings)

# Online: retrieve candidates
user_embedding = model.user_tower(user_features).detach().numpy()
distances, indices = index.search(user_embedding.reshape(1, -1), k=1000)
candidate_items = [all_items[i] for i in indices[0]]

Stage 2: Ranking

With ~1K candidates from retrieval, the ranking stage scores each (user, item) pair with a more expensive model.

Features for Ranking

Good ranking models use a rich feature set:

pythonCopy
features = {
    # User features
    'user_age_bucket': ...,
    'user_country': ...,
    'user_device': ...,
    'user_30d_category_clicks': ...,    # historical behavior

    # Item features
    'item_category': ...,
    'item_popularity_7d': ...,
    'item_age_days': ...,
    'item_avg_rating': ...,

    # Cross features (user × item)
    'user_item_category_match': ...,
    'user_has_interacted_with_author': ...,

    # Context
    'hour_of_day': ...,
    'day_of_week': ...,
    'request_surface': ...             # homepage vs. search vs. notification
}

Gradient Boosted Trees for Ranking

pythonCopy
import lightgbm as lgb

train_data = lgb.Dataset(X_train, label=y_train, group=train_groups)
val_data = lgb.Dataset(X_val, label=y_val, group=val_groups)

params = {
    'objective': 'lambdarank',    # LambdaMART
    'metric': 'ndcg',
    'ndcg_eval_at': [10],
    'num_leaves': 63,
    'learning_rate': 0.05,
}

model = lgb.train(params, train_data, num_boost_round=200,
                  valid_sets=[val_data], callbacks=[lgb.early_stopping(20)])

Neural Ranking Models

For complex feature interactions, a wide & deep model or transformer-based ranking model learns better representations than trees:

pythonCopy
class RankingModel(nn.Module):
    def __init__(self, feature_dim, user_embed_dim, item_embed_dim):
        super().__init__()
        # Wide part: linear on sparse features (fast, memorization)
        self.wide = nn.Linear(feature_dim, 1)
        # Deep part: MLP on dense features (generalization)
        self.deep = nn.Sequential(
            nn.Linear(user_embed_dim + item_embed_dim, 256),
            nn.ReLU(), nn.Dropout(0.1),
            nn.Linear(256, 128),
            nn.ReLU(),
            nn.Linear(128, 1)
        )

    def forward(self, sparse_features, user_emb, item_emb):
        wide_out = self.wide(sparse_features)
        deep_out = self.deep(torch.cat([user_emb, item_emb], dim=-1))
        return torch.sigmoid(wide_out + deep_out)

Stage 3: Re-ranking

After ranking, business logic adjusts the final slate:

• Diversity: avoid showing 10 similar items. Force category diversity.
• Freshness: inject recent items even if their score is lower.
• Business rules: suppress out-of-stock items, boost promoted items.
• Position bias correction: users click top results regardless of quality - debias training labels using inverse propensity weighting.

Evaluation

Offline Metrics

Offline evaluation uses holdout data - typically the last N days of interactions.

pythonCopy
# Evaluate retrieval recall
recall_at_k = retrieved_relevant_items / total_relevant_items

# Evaluate ranking quality
from sklearn.metrics import ndcg_score
ndcg = ndcg_score([relevance_scores], [predicted_scores], k=10)

Caveat: offline metrics don't fully predict online performance. A model with 5% better offline NDCG may have no improvement in click-through rate online.

Online Metrics

The only ground truth is an A/B test:

• CTR (click-through rate): easy to game, optimize for
• Engagement rate: time spent, likes, saves
• Long-term retention: are users returning next week?
• Conversion: purchases, subscriptions

Always instrument your experiment to detect novelty effect - a new algorithm looks good for 2 weeks because it's different, not because it's better.

Cold Start Problem

Every RecSys has a cold start problem: new users and new items have no interaction history.

New users: use content-based features (demographics, device, context) and ask for explicit preferences onboarding. Explore aggressively until you have interaction history.

New items: inject new items into retrieval explicitly (bypass embedding-based retrieval which can't represent new items). Use content features (title, category, description) instead of learned embeddings.

Solutions:

• Content-based filtering: recommend based on item features rather than interaction history
• Hybrid: blend collaborative and content-based signals
• Warm-up: give new items guaranteed impressions to build initial interaction data

Key Engineering Decisions

Decision	Options	Tradeoff
Retrieval approach	ANN + two-tower vs. BM25 vs. hybrid	Semantic quality vs. latency vs. freshness
Candidate set size	100 vs. 1000 vs. 10000	Ranking quality vs. ranking latency
Ranker complexity	GBT vs. Wide&Deep vs. Transformer	Quality vs. inference cost
Training data	All interactions vs. high-quality clicks	Bias vs. coverage
Re-ranking diversity	MMR vs. rule-based	Relevance vs. user experience

RecSys is where ML meets product design most directly. The best practitioners think simultaneously about user experience, business objectives, and model quality - not just offline metrics.

Common Mistakes

Optimizing for CTR when the business cares about LTV. Click-through rate is easy to measure and optimize, but it is a weak proxy for the business metric that actually matters - whether the recommendation led to a purchase, a subscription renewal, or long-term engagement. A recommendation model that maximizes CTR can actively harm LTV by surfacing clickbait. Align your training objective with the business metric from the start.

Ignoring position bias in click data. Items displayed in the top position receive more clicks regardless of their quality because users are more likely to look at and click on items in prominent positions. If you train a model on biased click data without correcting for position, the model learns to recommend whatever was already shown prominently - a self-reinforcing feedback loop. Use inverse propensity scoring or a dedicated position-debiasing layer.

Not evaluating cold-start users separately from warm users. A model trained on interaction-rich warm users will look excellent in aggregate offline evaluation, even if it completely fails new users who have no history. Cold-start performance is a distinct problem that requires a distinct evaluation split. Always report metrics broken down by user activity quintile so cold-start failure is visible.

What to Practice Next

• Compute NDCG@10 on a held-out test set for a collaborative filtering model; then split the test set into cold users (fewer than 5 interactions) and warm users (20+ interactions) and compare NDCG for each group.
• Identify a proxy metric (CTR, dwell time, save rate) that is easier to measure than your true business metric; write a one-paragraph argument for and against using it as your training signal.
• Implement a simple position-debiasing correction on a click dataset by down-weighting clicks on position 1 relative to position 5; remeasure offline metrics and explain the change.