Product-Grade End-to-End AI Platform (Capstone)

The AI Platform Capstone is the culminating project for the SWE-to-ML-engineer path. It is not a tutorial to follow. It is an open brief: build a production-grade ML platform that demonstrates end-to-end ownership of the full ML engineering stack. This document describes the objectives, required components, evaluation rubric, and stretch goals.

Project Objective

Design and implement a production-grade ML platform for a problem of your choice. The platform must take data in, produce a trained model, serve predictions, and monitor model health - with each layer automated and observable.

The problem domain is flexible: recommendation, classification, regression, anomaly detection, or an LLM-augmented system. What matters is the platform quality, not the model complexity. A production-grade platform with a logistic regression model outscores a notebook with a transformer fine-tune.

Required Components

1. Data Pipeline

• Ingests data from at least one source (database, API, flat files)
• Performs schema validation before any downstream use
• Produces a versioned, immutable training snapshot (DVC or equivalent)
• Is orchestrated - does not require manual steps to run

raw_data/ → validate → feature_engineering → training_data/v{hash}.parquet

2. Training Pipeline

• Parameterized via config file (no hardcoded hyperparameters in the training script)
• Logs all experiments to an experiment tracker (MLflow, W&B, or Weights & Biases)
• Stores the trained artifact with a version tag
• Can be triggered from CI without manual intervention

3. Serving Layer

• Model exposed as an HTTP API (FastAPI, Flask, or equivalent)
• Containerized with Docker
• Includes /health and /metrics endpoints
• Latency p99 < 200 ms under a 50 RPS load test
• Tagged with the model version that is running

4. Monitoring

• Prediction logs written with input features, output score, and timestamp
• At least one automated alert (latency, error rate, or score distribution shift)
• A simple dashboard (Grafana, or a script that generates a weekly health report)
• Evidence that you have investigated at least one simulated or real alert

5. Deployment Automation

• A CI/CD pipeline that: validates data, trains, evaluates against a quality gate, and deploys to staging
• Blue-green or canary deployment for the production step
• A written and tested rollback runbook

Evaluation Rubric

Each component is scored 1–4:

Score	Meaning
1	Exists but is broken, manual, or non-reproducible
2	Works but requires manual steps; no error handling
3	Automated, observable, and documented
4	Automated, observable, tested, with demonstrated incident response

Minimum pass: average score ≥ 3 across all five components, with no component scoring 1.

A score of 4 on serving without a functioning data pipeline is still a fail. The point is end-to-end ownership.

Technical Stack Guidance

You are not required to use a specific stack, but this combination is well-tested and employable:

Layer	Recommended tooling
Data pipeline	Prefect or Airflow + Great Expectations
Experiment tracking	MLflow (self-hosted on a $5/mo VPS)
Data versioning	DVC + S3 or GCS
Serving	FastAPI + Docker + Kubernetes (or Render/Railway for simplicity)
Monitoring	Prometheus + Grafana, or Evidently AI
CI/CD	GitHub Actions

If you are building an LLM-augmented system, replace the training pipeline component with a fine-tuning or RAG pipeline, and replace the model quality gate with an LLM eval harness.

Stretch Goals

These are not required for a pass, but they differentiate a strong capstone from an adequate one:

• Online learning: the model updates on new data without a full retrain
• Feature store: a shared feature computation layer used by both training and serving
• A/B testing framework: route traffic between two model versions and compute lift
• Model cards: a complete model card for each deployed version
• Multi-model orchestration: a router that selects among multiple models based on request context

What Reviewers Look For

When presenting your capstone, reviewers will ask:

1. Walk me through what happens when new training data arrives. How does it become a production prediction?
2. How would you know if the model started performing badly at 2 AM?
3. Show me how to roll back to the previous version.
4. What is the hardest thing you had to debug in this system?

Prepare honest, specific answers backed by your system's logs and code.

Common Mistakes

Over-investing in the model, under-investing in the platform. A complex model in a fragile platform does not pass. Spend at least 60% of your time on the pipeline, serving, and monitoring layers.

Skipping the rollback test. You must demonstrate rollback, not just write the runbook. Run a deliberate rollback in staging and include the evidence in your submission.

Using hosted AutoML as your serving layer. AutoML platforms are legitimate tools in production, but for this capstone the goal is to demonstrate you understand and own the serving infrastructure. Build it yourself.

Where to Go Next

• milestone-gate-2-production-readiness - ensure you pass gate 2 before starting the capstone build
• portfolio-interview-narrative - after building the capstone, use it as the centerpiece of your interview narrative
• top-product-company-interview-readiness - the capstone is the primary evidence artifact for your ML system design and production ownership claims

What to Practice Next

• Draft a one-page design document for your capstone: define the problem, the ML task, the data source, the serving architecture, and one key non-functional requirement (latency, cost, or throughput).
• Stand up a minimal end-to-end pipeline - ingest raw data, train a baseline model, log it to MLflow, and serve it behind a FastAPI endpoint - before adding any complexity.
• Run a load test against your serving endpoint using Locust or k6, identify the bottleneck, and document what you would change to hit a 200 ms p99 latency target.