Product-Grade End-to-End AI Product (Capstone)

The end-to-end AI system capstone is the highest-fidelity demonstration of production ML ownership available outside of a job. The brief is deliberately open: build a production AI system that solves a real problem, with enough engineering rigor that a senior ML engineer would not be embarrassed to maintain it. This document defines the scope, evaluation criteria, technical expectations, and what "production-grade" actually means in practice.

Choosing a Problem Scope

The biggest mistake in a capstone is choosing a problem that is either too toy (no real data challenges, trivial evaluation) or too ambitious (can't ship in 4–8 weeks with real quality). Good capstone problem characteristics:

• Public or accessible data: you can get it reliably and it has enough volume to expose real data quality issues
• Clear business framing: you can explain why someone would pay for this
• Non-trivial evaluation: accuracy alone is not sufficient - you need a precision/recall tradeoff decision or a human-eval component
• Production path is real: you can actually serve predictions to real or simulated users

Strong examples: real-time fraud flag on public transaction data, job description classifier for career platform, RAG-based document Q&A for a specific domain, personalized content ranker for public RSS data.

Weak examples: MNIST classifier, sentiment analysis on IMDB reviews (benchmark saturation), model trained on your local photos.

System Architecture: Data → Model → Serving → Monitoring

Every production AI system has the same four-layer architecture:

[ Data layer ]     raw sources → validation → feature pipeline → versioned snapshots
[ Model layer ]    training script → experiment tracking → quality gate → registry
[ Serving layer ]  containerized API → load balancer → prediction logging
[ Monitoring ]     drift detection → alerting → incident response → retraining trigger

Designing each layer explicitly - before writing code - prevents the pattern where the model works but the system is unmaintainable.

Data Layer Design Decisions

• What is the source of truth for labels? If labels are delayed (e.g., conversion happens 14 days after the event), how does your training pipeline handle it?
• How do you prevent training/serving skew? The features computed at training time must be computed identically at serving time.
• What is the data versioning strategy? Each training run must reference an immutable, reproducible snapshot.

Serving Layer Requirements

A production-grade serving layer must satisfy:

pythonCopy
# These are the non-negotiables for a capstone submission
assert response.status_code == 200             # Service responds
assert latency_p99 < 200                       # Under load
assert "/health" endpoint returns 200          # Health check exists
assert prediction_log_written(request)         # Observability
assert model_version in response.headers       # Versioning

What "Production-Grade" Actually Means

"Production-grade" does not mean running on enterprise infrastructure. It means the system has the properties that make it safe to operate:

Reproducible: given a commit hash and a data snapshot, you can reproduce the training run exactly.

Observable: there is enough logging and alerting that a failure is detected before users report it.

Recoverable: when something goes wrong, you can roll back to the last known good state in under five minutes.

Documented: a new engineer can understand the system architecture, run it locally, and deploy a new version by reading the documentation.

A $5/month VPS with these four properties is more "production-grade" than a Kubernetes cluster that violates them.

Evaluation Criteria

Capstone submissions are evaluated on five dimensions, each scored 1–4:

Data pipeline (1–4): Does data flow from source to training snapshot automatically, with validation? Is it versioned and reproducible?

Model quality (1–4): Is there a quality gate that prevents regressions from shipping? Is the eval suite meaningful for the problem, or just accuracy on a balanced holdout?

Serving (1–4): Does the API handle load? Is it containerized? Is there versioning? Is latency measured?

Monitoring (1–4): Are there prediction logs? Are there alerts? Have you investigated a real or simulated incident and documented the response?

Narrative (1–4): Can you explain every significant technical decision and its tradeoff? Can you describe what you would do differently? Can you quantify the value of the system?

Minimum pass: 3.0 average, no dimension below 2.

Demonstrating Your System

The demo format matters. A 20-minute capstone demo should cover:

1. (3 min) Problem and why it matters - business framing, not model framing
2. (5 min) System walkthrough - show the architecture diagram, then walk the data → model → serve → monitor path
3. (5 min) Live demo - trigger a prediction, show the log entry, show the Grafana dashboard or monitoring output
4. (4 min) Hard problem - the most technically interesting challenge you solved
5. (3 min) What you would do next - specific, prioritized, not generic

Reviewers will stop you at any point to ask questions. The questions that trip people up: "why did you choose X over Y?", "walk me through what happens if this component fails", "how do you know the model is performing well in production right now?"

Common Mistakes

Building without a demo user. Systems that only work in localhost are not production-grade. Deploy to a public URL. It can be a free tier. The act of deploying forces you to solve real problems you would otherwise skip.

No data validation. Skipping input validation is the single most common cause of silent production failures. Add it to the data pipeline before anything else.

Conflating model complexity with system quality. A fine-tuned LLM in a fragile, unmonitored system is a weaker capstone than a logistic regression in a well-engineered platform. Evaluators know the difference.

Where to Go Next

• portfolio-interview-narrative - turn this capstone into a 6-part interview story that showcases production ownership
• capstone-product-grade-ai-platform - if your system includes a shared feature store and multi-model serving, see the platform capstone for additional evaluation criteria
• milestone-gate-2-production-readiness - run the gate 2 checklist against your capstone before you submit it