ML System Design Interview Framework

ML system design interviews are different from regular software design interviews. You are not just designing services and APIs - you are making decisions about data, models, evaluation, and operational requirements. Without a framework, it is easy to jump to models before understanding the problem.

The Framework: 6 Steps

1. Clarify requirements and constraints
2. Frame the ML problem
3. Design the data layer
4. Select and design the model
5. Design serving and infrastructure
6. Plan evaluation and monitoring

Work through these in order. Do not skip to the model. Interviewers reward systematic thinking.

Step 1: Clarify Requirements

Spend the first 5 minutes asking clarifying questions. The problem as stated is almost never complete.

For any ML system, ask:

• What is the primary business metric? (revenue, retention, engagement, safety)
• What is the scale? (DAUs, queries per second, dataset size)
• What is the latency requirement? (real-time vs. batch)
• Are there regulatory or safety constraints? (GDPR, bias, content moderation)
• What data is currently available? (logs, labels, third-party signals)
• What does a false positive vs. false negative cost?

Example - "Design a video recommendation system":

• "How many daily active users are we designing for? 100M?"
• "Is this for the homepage or for 'watch next' after a video finishes? Different requirements."
• "What signals do we have? View counts, watch time, likes, subscriptions?"
• "Is there a freshness requirement - should we promote new videos?"
• "Are there policy concerns - avoid recommending harmful content?"

Step 2: Frame the ML Problem

Convert the business problem into an ML problem type. Be explicit about this.

Business problem	ML framing	Output
Which videos to recommend	Ranking: score items for user	Ranked list
Will this user churn	Binary classification	P(churn)
Is this review helpful	Regression or classification	Score 0–1
What is the sentiment	Multi-class classification	Label + confidence
Where is the defect	Object detection	Bounding boxes
What is the most relevant result	Retrieval + ranking	Ranked documents

State the objective function explicitly: "We are optimizing for expected watch time, subject to the constraint that harmful content is not surfaced."

Step 3: Data Layer Design

This is where most candidates underinvest. Data is often 80% of the work.

Address:

1. What data exists? Raw event logs, user profiles, item catalogs, labels
2. How is it labeled? Explicit (thumbs up/down) vs. implicit (watch time, clicks)
3. What is the class balance? If predicting fraud, expect 0.1–1% positive rate
4. How does training data map to production? Is there temporal leakage?
5. Feature types and storage:

Training data pipeline:
  Raw events (Kafka/S3)
    → Feature computation (Spark)
    → Feature store (offline: Parquet on S3; online: Redis)
    → Training set construction (point-in-time joins)
    → Model training

Serving pipeline:
  Request → Feature extraction (from feature store, request context)
           → Model inference
           → Post-processing
           → Response

Step 4: Model Selection

Do not open with a transformer. Start simple, justify complexity.

The model ladder:

Heuristic/rules → Linear model → GBT → Neural net → Pretrained LLM

Ask: Does the next step up justify its added complexity?

For each choice, state:

• Why this model for this problem
• What it cannot handle (its limitations)
• At what scale or accuracy requirement would you upgrade

Example - video recommendation:

• Retrieval: two-tower model (fast ANN search, trained on user-video interactions)
• Ranking: gradient boosted trees (interpretable, fast, handles tabular features well)
• Reranking (optional): neural net with attention over session context

State the training objective explicitly:

• "For ranking, we use LambdaRank with NDCG as the optimization target, with watch time as the relevance signal."

Step 5: Serving and Infrastructure

Cover the online serving path:

pythonCopy
# The structure to describe verbally:

def serve_recommendation(user_id: int, context: dict) -> list[dict]:
    # 1. Retrieve candidates (fast, approximate)
    candidates = retrieval_layer.get_candidates(user_id, k=500)

    # 2. Filter (already seen, policy violations, out of stock)
    filtered = filter_layer.apply(candidates, user_id, context)

    # 3. Extract features (from feature store + request context)
    features = feature_extractor.extract(user_id, filtered, context)

    # 4. Score with ranker
    scores = ranking_model.predict(features)

    # 5. Post-process (dedup, diversity, business rules)
    results = post_processor.apply(filtered, scores, k=20)

    return results

Address:

• Latency: what is the budget per stage?
• Scale: what is the QPS, what infrastructure do you need?
• Fallback: what happens if the model is down? (return popular items, cached results)
• Rollout: how do you safely deploy a new model version? (shadow mode → canary → full)

Step 6: Evaluation and Monitoring

This is where candidates demonstrate production maturity.

Offline evaluation:

• What metric measures model quality? (NDCG@10, AUC-PR, MAE)
• How is the holdout set constructed? (temporal split for time-series data)
• What are your minimum performance thresholds to deploy?

Online evaluation:

• A/B test setup: primary metric (watch time, revenue), guardrail metrics (not harmful content surfaced)
• How long to run the experiment? Calculate based on expected effect size and power
• What constitutes a win? Relative improvement threshold

Production monitoring:

• Feature drift: PSI on key features weekly
• Prediction distribution: alert if output mean shifts >2σ
• Business metric: rolling 7-day metric vs. rolling 30-day baseline
• Latency: p99 alert if >budget

Putting It Together: Sample Walk-Through

Question: "Design a spam detection system for a social platform."

Clarify:

• Scale: 10M posts per day, need real-time decision at post time
• Latency: <100ms (users expect immediate feedback)
• Regulatory: GDPR applies, need explainability for appeals
• Metrics: minimize false negatives (missed spam) while keeping false positive rate <0.5%

Frame:

• Binary classification: spam vs. not-spam
• Imbalanced (~2% spam)
• Real-time inference required

Data:

• Post content, user history, posting velocity, link domains, image metadata
• Labels: human moderation queue (expensive) + user reports (noisy)
• Temporal split for holdout (spam patterns evolve)

Model:

• Fast path: rule engine for known spam patterns (<5ms)
• ML path: gradient boosted tree on tabular features + TF-IDF text features (<30ms)
• Deep path (async): fine-tuned text classifier for borderline cases

Serving:

• Sync path returns fast model decision immediately
• Async path queues borderline posts for deep model review
• Human review queue for high-uncertainty predictions above threshold

Evaluation:

• Offline: AUC-PR, precision@recall=0.95
• Online: A/B test measuring spam rate in control vs. treatment feeds
• Monitor: daily spam rate, false positive rate from appeal resolutions

In 30 minutes, this structured approach covers the full problem space without getting stuck on any one component.

Common Mistakes

Jumping to model architecture before clarifying the problem. The most common fatal mistake in ML system design interviews is reaching for "I would use a transformer" before establishing what the system needs to optimize, what data is available, and what the latency and scale requirements are. Interviewers who hear architecture-first answers conclude the candidate cannot scope ambiguous problems. Always spend the first 3-5 minutes clarifying requirements before touching model choices.

Not discussing the serving latency SLA. A model that achieves state-of-the-art offline metrics is worthless if it takes 2 seconds to serve a recommendation that needs to respond in 50ms. Latency requirements fundamentally constrain architecture choices (model size, quantization, caching, approximate vs. exact retrieval). Bring up latency requirements early and let them guide your design decisions.

Giving a technically correct answer that ignores operational feasibility at scale. Saying "I would retrain the model daily on all historical data" may be technically correct but signals that the candidate has not thought about the compute cost, the orchestration complexity, or the monitoring required to catch a bad retrain before it ships. Design for the operational reality of a production system, not for a research setting.

What to Practice Next

• Take a classic ML system design question ("design a news feed ranking system") and walk through the full framework - problem scoping, data, features, model, serving, monitoring - in under 25 minutes without looking at notes; record yourself and review for gaps.
• For any ML design you produce, explicitly list the top three operational risks (data pipeline failure, model regression, feature drift) and describe how you would detect and respond to each.
• Practice stating your latency SLA requirement within the first five minutes of a design session; work backwards from that SLA to constrain which model architectures are feasible.