Milestone Gate 2: Production Readiness

Milestone Gate 2 is a self-assessment checkpoint. Before moving into advanced specialization and hiring prep, you should be able to demonstrate - not just describe - a set of concrete production ML skills. This gate is not a quiz. It is a rubric you apply to your own work.

What This Gate Covers

Gate 2 tests the intersection of ML engineering and software engineering: can you take a model from a notebook to a production system that is observable, maintainable, and recoverable?

The four areas:

1. Training pipeline - reproducible, versioned, configurable
2. Model serving - containerized, latency-tested, health-checked
3. Monitoring - drift detection, alerting, incident response
4. Rollback - the ability to revert a bad deployment safely

Pass/No-Pass Criteria

Training Pipeline

Pass: You have a training script (not a notebook) that accepts config parameters, logs metrics to an experiment tracker (MLflow, W&B, or equivalent), stores the trained artifact with a versioned name, and can be triggered by CI.

No-Pass: Training lives in a notebook. Re-running it produces different results depending on what state the notebook is in. Artifacts are overwritten rather than versioned.

Validation question: can a teammate reproduce your last training run by checking out the commit and running one command?

Model Serving

Pass: Your model runs as an HTTP service inside a Docker container. The container starts from a cold image in under 60 seconds. There is a /health endpoint. Latency p99 under your target load is measured and documented. The image is tagged with the model version.

No-Pass: The model is served from a script that runs on someone's laptop. There is no health check. Latency has never been measured.

Validation question: can you deploy your model to a new environment by pulling the image and running docker run?

Monitoring

Pass: You emit prediction logs (input features + output score + timestamp) to a data store. You have at least one alert: error rate or prediction distribution shift triggers a notification. You have investigated at least one real or simulated alert.

No-Pass: There are no prediction logs. You rely on user reports to learn that the model has degraded.

Validation question: if your model started returning garbage predictions tonight, how long before you would know?

Rollback

Pass: You have a written rollback runbook. You have tested it (run it in staging or against a previous version deliberately). A rollback takes under five minutes.

No-Pass: There is no rollback procedure. Fixing a bad deployment means redeploy-and-pray.

Validation question: walk me through what you would do if a deployment caused prediction quality to drop 15% at 11 PM.

Skills Checklist

Use this as a pre-interview and pre-capstone checklist. Mark each item honestly.

Training

• Training runs are parameterized via config file or CLI args
• Experiments logged to MLflow or equivalent
• Model artifacts stored with version tag in S3/GCS/registry
• Training triggered from a CI job (not just local)

Serving

• Dockerfile builds without warnings
• /health endpoint returns 200 and latency < 10 ms
• p99 latency measured under realistic load
• Image tagged with model version, not latest

Monitoring

• Prediction logs written (input, output, timestamp)
• At least one alert defined (error rate or distribution shift)
• Alert threshold documented and justified
• At least one simulated incident investigated end-to-end

Rollback

• Rollback runbook exists as a written document
• Rollback tested in a non-production environment
• Time-to-rollback measured (target: < 5 minutes)

If You Do Not Pass

No-pass does not mean stop. It means you have a specific list of things to build. Pick the weakest area and work through the checklist items one by one. Each item is a half-day of focused engineering work.

The most common pattern: strong training pipeline, weak monitoring, no rollback procedure. Address monitoring first because it is what tells you when the deployment goes wrong.

Common Mistakes

Conflating "it works on my machine" with production readiness. A model that produces correct predictions is necessary but not sufficient. Production readiness is about the surrounding system: observability, recovery, reproducibility.

Skipping the rollback test. Most engineers write a rollback plan and never test it. Test it. The first time you need a rollback is not the time to discover it does not work.

Treating this gate as a formality. The items on this checklist are exactly what senior engineers evaluate in system design interviews and what hiring managers look for in portfolio projects.

Where to Go Next

• mlops-cicd-ml-systems - automate everything on this checklist into a CI/CD pipeline
• observability-evalops-governance - go deeper on monitoring and governance after you pass this gate
• capstone-product-grade-ai-platform - the capstone assumes everything on this checklist is in place

What to Practice Next

• Work through the checklist item-by-item for a project you have in progress: for each criterion you cannot yet satisfy, write a one-sentence description of what is blocking you and how long it would take to fix.
• Add monitoring to a serving endpoint you control: instrument at minimum request count, p99 latency, and an application-level quality signal, and set an alert threshold for each.
• Write a one-page runbook for your model endpoint covering: how to roll back to the previous version, how to disable the endpoint under load, and who to contact if the quality signal degrades overnight.