AI System Design and Product Tradeoffs

System design interviews at product companies test whether you can reason about building AI systems under realistic constraints: latency budgets, cost limits, reliability requirements, and safety considerations. This module teaches the framework for approaching these problems - not memorizing specific architectures, but reasoning through tradeoffs.

The System Design Framework

Every AI system design problem has the same structure. Work through it in order:

1. Clarify the problem: what is the ML objective? What defines success? Who uses this and how?
2. Define the data: where does it come from? How much labeled data exists? What is the freshness requirement?
3. Choose the ML approach: supervised/unsupervised/generative? What features? What model family?
4. Design the pipeline: how does data flow from source to prediction?
5. Design serving: batch or online? What are the latency and throughput requirements?
6. Define evaluation: offline metrics → online metrics. How do you detect degradation?
7. Address failure modes: what happens when the model is wrong? How do you prevent harm?

Worked Example: Personalized Content Ranking

Problem: Build a system that ranks documents for each user so the most relevant ones appear first.

Clarify:

• What is "relevant"? User clicks, time-on-page, explicit ratings?
• How many users? Documents? Queries per second?
• Cold start: what happens for new users with no history?

Data:

• User behavior logs: clicks, dwell time, shares
• Document features: topic, freshness, author, engagement history
• User features: demographics, historical topic preferences, device type
• Label definition: click = positive, skip after ≥ 3 seconds = negative

ML approach:

• Two-stage: candidate generation (fast, recall-focused) → ranking (slower, precision-focused)
• Stage 1: approximate nearest neighbor search in embedding space (user embedding vs. document embeddings)
• Stage 2: LightGBM ranker trained on pairwise click data
• Cold start: use content-based features (document topics) for new users

Pipeline design:

Batch (daily):
  User behavior logs → feature computation → user embeddings (retrain weekly)
  Document ingest → document embeddings (update on new content)

Online (per request):
  User ID → feature store lookup → candidate retrieval (ANN) → ranker → top-K documents

Serving design:

• p99 latency requirement: 150ms
• ANN search: ~10ms, ranker (top-100 candidates → re-rank with LightGBM): ~20ms
• Feature store lookup (Redis): ~5ms
• Budget: ~50ms buffer for network and serialization

Evaluation:

• Offline: NDCG@10 on held-out user sessions
• Online: CTR, avg session length, time-to-first-click

Key Tradeoff Axes

Latency vs. quality: a neural reranker may improve NDCG by 5% but add 80ms. Is that worth it? Depends on the SLO and user sensitivity to latency.

Freshness vs. stability: more frequent retraining produces a fresher model but increases infrastructure cost and regression risk. For most product ML, weekly retraining is a good default.

Build vs. buy: a managed vector database (Pinecone, Weaviate) costs more per query but eliminates operational burden. For early-stage products, buy. For scale, evaluate whether the cost savings justify building internally.

Precision vs. recall in safety-critical decisions: in a content moderation system, a false negative (allowing harmful content) is worse than a false positive (blocking legitimate content). Set the decision threshold to favor the less harmful error type.

Human-in-the-Loop Boundaries

Define where the model can act autonomously and where it requires human review:

Autonomous (no human needed):
- Personalized feed ranking
- Spam filtering with high confidence (score > 0.98)
- Product recommendations

Human review required:
- Account suspension (medium confidence)
- Medical or legal content flagging
- Content moderation for borderline cases

Human always required:
- Account termination
- Any action with significant financial consequence
- Outputs used in legal proceedings

Document these boundaries explicitly. Systems that start autonomous and expand their autonomy over time without review often cause unexpected harm.

Privacy, Compliance, and Data Governance

Every AI system that handles user data must address:

Data minimization: only collect and use features that are actually needed for the ML objective. Review every feature for necessity.

Retention policies: how long is training data kept? When does it need to be purged? Systems trained on user data may need to support right-to-deletion - which means identifying and retraining without that user's data.

Differential privacy for sensitive attributes: if your model is trained on sensitive demographic data (age, gender, health status), consider adding noise to training to prevent membership inference attacks.

Documentation: maintain a model card for every deployed model - what it does, what data it was trained on, known failure modes, and who is responsible for it.

Presenting a System Design

In an interview or design review, structure your presentation:

1. Requirements (2 min): ML objective, scale, latency, evaluation
2. High-level architecture (3 min): pipeline diagram, data flow
3. Key components (5 min): data, features, model, serving
4. Tradeoffs (3 min): 2-3 explicit tradeoffs you considered and why you chose this approach
5. Failure modes (2 min): what can go wrong, how you'd detect and handle it

Interviewers reward explicit tradeoff reasoning more than any specific architecture choice. "I chose X over Y because of [constraint] and [data property]" is stronger than "I would use X because it's popular."

Common Mistakes and Bad Instincts

Jumping to model selection before defining the objective. You cannot choose a model until you know what you're optimizing. Define the label, the metric, and the evaluation strategy first.

Not accounting for latency in the design. A design that looks clean on a whiteboard but requires 5 sequential network calls will fail the latency SLO. Estimate component latencies early and sum them.

Treating cold start as a minor edge case. New users and new items are often the majority of traffic for growth-stage products. Design your system for cold start from the beginning.

No failure mode analysis. Every design should include: what happens when the model is wrong (and how often that is), what happens when a dependency fails, and how you detect production degradation.

Where to Go Next

• Module 29 (Portfolio and Storytelling) covers how to present these design decisions as portfolio evidence.
• Module 30 (Interview Readiness) covers how to practice and structure your answers to system design questions under interview conditions.