MLOps and CI/CD for ML Teams

Software CI/CD is mature: commit, build, test, deploy. ML CI/CD is harder. Models are not just code - they are code plus data plus learned weights, and their quality is measured not by unit tests but by metrics on held-out data that may drift over time. A model that "passes tests" may still underperform in production if its training data is stale or if the production distribution has shifted.

This module covers the engineering practices that make ML development reproducible, testable, and safely deployable.

What MLOps Adds to Standard DevOps

Standard DevOps manages code artifacts. MLOps must additionally manage:

Artifact	Standard DevOps	MLOps Addition
Source code	Git + CI/CD	Same
Dependencies	requirements.txt / Docker	Same
Data	Not tracked	DVC / data versioning
Model weights	Not tracked	MLflow / model registry
Metrics	Not tracked	Experiment tracking
Deployment	Push image → Kubernetes	+ shadow mode + canary + rollback logic
Testing	Unit + integration	+ data validation + model quality gates

Data Version Control with DVC

DVC (Data Version Control) extends Git to version large files (data, models) stored in remote storage (S3, GCS):

bashCopy
# Initialize
dvc init
git add .dvc/

# Track a dataset
dvc add data/training_v3.parquet
git add data/training_v3.parquet.dvc
git commit -m "Add training dataset v3"

# Push data to remote storage
dvc remote add -d s3_storage s3://ml-data-bucket/dvc
dvc push

# To reproduce on another machine:
git pull
dvc pull    # Downloads data from S3, checksums verified

DVC .dvc files are small text files that live in Git, pointing to the actual data in remote storage. A Git commit hash + a .dvc file hash uniquely identifies both the code and the data used to produce a model.

Model Registry with MLflow

pythonCopy
import mlflow
from mlflow.tracking import MlflowClient

# Training: log model to MLflow
with mlflow.start_run() as run:
    # ... training code ...
    mlflow.log_metrics({"val_auc": 0.91, "val_f1": 0.87})
    mlflow.sklearn.log_model(
        sk_model=model,
        artifact_path="model",
        registered_model_name="churn-predictor",
    )

# Promote a model version to staging
client = MlflowClient()
client.transition_model_version_stage(
    name="churn-predictor",
    version=5,
    stage="Staging",
    archive_existing_versions=False,
)

# After validation, promote to production
client.transition_model_version_stage(
    name="churn-predictor",
    version=5,
    stage="Production",
    archive_existing_versions=True,  # Archive previous production version
)

# Load the current production model for serving
prod_model = mlflow.pyfunc.load_model("models:/churn-predictor/Production")

The model registry gives you: stage management (Staging → Production), version history, rollback by version number, and lineage linking model versions to training runs and datasets.

Automated Data Validation

Never deploy a model trained on unvalidated data. Use Great Expectations or Pandera to assert invariants:

pythonCopy
import pandera as pa
from pandera import Column, DataFrameSchema, Check

training_schema = DataFrameSchema(
    columns={
        "user_id": Column(int, Check.greater_than(0), nullable=False),
        "n_events_7d": Column(float, Check.greater_than_or_equal_to(0), nullable=False),
        "days_active_30d": Column(int, Check.in_range(0, 30), nullable=False),
        "label": Column(int, Check.isin([0, 1]), nullable=False),
    },
    checks=[
        Check(lambda df: df["label"].mean() > 0.01, error="Label rate < 1%: likely data issue"),
        Check(lambda df: df["label"].mean() < 0.5, error="Label rate > 50%: check sampling logic"),
        Check(lambda df: len(df) >= 10_000, error="Training set < 10K rows: insufficient"),
    ],
)

def validate_training_data(df):
    try:
        training_schema.validate(df, lazy=True)
        print("Data validation passed")
        return True
    except pa.errors.SchemaErrors as e:
        print(f"Validation failures:\n{e.failure_cases}")
        return False

Run data validation as the first step in your training pipeline. If validation fails, abort training and alert - training on bad data is worse than not training at all.

CI Pipeline for ML

yamlCopy
# .github/workflows/ml-ci.yml
name: ML CI

on:
  push:
    branches: [main]
  pull_request:
    paths:
      - "src/**"
      - "configs/**"
      - "data/*.dvc"

jobs:
  validate-and-train:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v4
        with: {python-version: "3.11"}

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Pull data
        run: dvc pull
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

      - name: Validate data
        run: python scripts/validate_data.py

      - name: Run unit tests
        run: pytest tests/unit/ -v

      - name: Run model quality gate
        run: python scripts/quality_gate.py --min-auc 0.85 --min-f1 0.80
        env:
          MLFLOW_TRACKING_URI: ${{ secrets.MLFLOW_URI }}

      - name: Build and push serving image
        if: github.ref == 'refs/heads/main'
        run: |
          docker build -t ml-serving:${{ github.sha }} .
          docker push $ECR_REGISTRY/ml-serving:${{ github.sha }}

The quality gate (quality_gate.py) runs inference on a held-out test set and fails the pipeline if metrics fall below thresholds. This prevents regressions from reaching production.

Canary and Shadow Deployment

pythonCopy
import random

def route_request(request, canary_fraction: float = 0.05):
    """
    Shadow mode: run both models, log both predictions, serve only current.
    Canary mode: serve new model to canary_fraction of traffic.
    """
    current_prediction = current_model.predict(request.features)

    if random.random() < canary_fraction:
        # Canary: serve new model to 5% of traffic
        new_prediction = new_model.predict(request.features)
        log_prediction(request, new_prediction, model_version="new")
        return new_prediction
    else:
        # Current model for 95% of traffic
        log_prediction(request, current_prediction, model_version="current")
        return current_prediction

def shadow_mode(request):
    """Shadow: always serve current, but also call new model for comparison."""
    current_prediction = current_model.predict(request.features)
    new_prediction = new_model.predict(request.features)
    log_prediction_comparison(request, current_prediction, new_prediction)
    return current_prediction  # User always gets current model

Shadow mode is safest for validating a new model - users never see new model output, but you collect enough data to compare quality. Move to canary (5% → 20% → 50% → 100%) once shadow metrics look good.

Common Mistakes and Bad Instincts

Skipping the model registry and deploying by copying files. Without a registry, you cannot tell which model version is running in production, cannot roll back to a specific version, and lose the lineage linking production models to training runs.

Treating the quality gate as a static threshold. A threshold of AUC > 0.85 that was set 18 months ago may be too low or too high for current business conditions. Review quality gate thresholds quarterly.

Not testing the serving code, only the model. The serving layer (API, preprocessing, postprocessing) has bugs too. Integration tests that send real HTTP requests to a test deployment catch a class of bugs that unit tests miss.

Where to Go Next

• observability-drift-feedback-loops-and-llm-evals: the deployment pipeline gets you to production; observability tells you whether the model is working
• ai-system-design-quality-cost-latency-and-safety-tradeoffs: design systems that make safe ML deployments operationally sustainable