Reading an ML System Design Document

Why Non-Technical People Need to Read ML Design Docs

ML system design documents describe what will be built, why those design decisions were made, and what the tradeoffs are. Decision-makers who cannot read them are forced to either rubber-stamp technical decisions or ask engineers to translate everything - neither is ideal.

Being able to engage with a design document at a meaningful level - asking the right questions, identifying the key risks, understanding the main tradeoffs - makes you a more effective leader and a stronger collaborator.

The Standard Structure of an ML Design Document

While formats vary, most ML design documents cover:

Problem statement: What is being built and why. Usually includes: the user problem, the proposed ML solution, success metrics, and how this connects to business objectives.

Data section: What data will be used, where it comes from, how it will be labeled, and known quality issues.

Features: What inputs the model will use. Why those features were chosen. What is not included and why.

Model approach: What type of model (classification, regression, ranking, generation). What architecture. What baseline to compare against.

Evaluation plan: What metrics will be used. What threshold must be met. How online performance will be measured.

Serving plan: How the model will be deployed. What the latency requirements are. How the system will be monitored.

Risks and unknowns: What could go wrong. What assumptions are being made. What the team does not know yet.

The Questions That Matter Most

When reviewing an ML design document, focus on:

On data: Is the training data representative of the population where the model will be deployed? Who labeled it? How consistent is the labeling?

On evaluation: Does the evaluation metric match what the product actually needs? Is there a plan for online evaluation (not just offline)?

On serving: What is the latency requirement and is the proposed architecture capable of meeting it? What happens when the model is wrong?

On risks: What is the model assuming about the future that might not hold? What has been explicitly left out of scope?

On success criteria: Is "good enough" defined clearly? Who decides when the model is ready to ship?

You do not need to evaluate the mathematical details. You need to evaluate whether the document has answered the right questions.

A Reading Walkthrough

When you first open an ML design document:

1. Read the problem statement first - does the problem make sense and is the proposed solution logically connected to it?
2. Skip to the evaluation plan - is success defined measurably?
3. Read the risks section - does the team know what they do not know?
4. Then read the data section - does the data support the claims in the problem statement?
5. Finally, review the serving plan for latency and monitoring.

This order prioritizes the questions most often left vague in ML design documents.

Where to Go Next

The next module gives you the framework for the product decisions that follow from reading design documents - how to make and defend tradeoffs around quality, cost, latency, and risk.

Common Mistakes

Skimming for the model and ignoring the data pipeline. Most ML system failures happen in data ingestion, feature computation, and serving infrastructure - not in model architecture. A design document that glosses over how training data is collected, labeled, and kept fresh is describing a system that will likely fail in production.

Treating documented design decisions as universal best practices. A choice that made engineering sense at Twitter's scale in 2018 - sharding strategy, approximate nearest neighbor index, feature freshness window - may be entirely wrong for a different scale, team size, or latency budget. Read design docs to understand the reasoning given the constraints, not to extract reusable templates.

Not noting what the document omits. Missing latency SLAs, missing descriptions of failure modes, and missing monitoring or alerting sections are red flags, not neutral gaps. What a design document does not say often reveals the risks the authors did not think through, and those are the areas most likely to cause incidents.