MLOps and CI/CD for ML Systems

Continuous integration and continuous deployment are standard practice in software engineering. In ML, the same principles apply - but the pipeline is more complex because there are two additional artifacts to version and validate: data and trained models. A real ML CI/CD pipeline does not just test code; it validates data, trains a model, evaluates its quality, registers the artifact, and deploys it.

What a Real ML CI Pipeline Does

A typical ML CI pipeline runs in this order:

1. Data validation - schema check, null rate check, distribution snapshot
2. Training - reproducible training run with logged hyperparameters and metrics
3. Quality gate - compare new model against current production model on holdout data
4. Artifact registration - push model to registry if quality gate passes
5. Deploy to staging - run integration tests and latency benchmarks
6. Promote to production - canary rollout or blue-green cutover

yamlCopy
# .github/workflows/ml-pipeline.yml
name: ML Training and Deployment

on:
  push:
    paths:
      - 'src/training/**'
      - 'configs/**'

jobs:
  validate-data:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python scripts/validate_data.py --config configs/data_validation.yml

  train:
    needs: validate-data
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python scripts/train.py --config configs/training_config.yml
      - uses: actions/upload-artifact@v4
        with:
          name: model-artifact
          path: outputs/model/

  quality-gate:
    needs: train
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/download-artifact@v4
        with:
          name: model-artifact
          path: outputs/model/
      - run: python scripts/quality_gate.py --challenger outputs/model/ --champion-alias production

  deploy-staging:
    needs: quality-gate
    runs-on: ubuntu-latest
    steps:
      - run: ./scripts/deploy.sh staging
      - run: python scripts/integration_tests.py --env staging

DVC for Data Versioning

DVC (Data Version Control) versions data files alongside code using Git-tracked pointer files. Each training run references a specific, immutable data snapshot.

bashCopy
# Initialize DVC in your repo
dvc init
git add .dvc .dvcignore
git commit -m "Initialize DVC"

# Track a data file
dvc add data/training_set.parquet
git add data/training_set.parquet.dvc .gitignore
git commit -m "Track training data with DVC"

# Push data to remote storage
dvc remote add -d myremote s3://my-bucket/dvc-cache
dvc push

The .dvc file is a tiny YAML file containing a hash and the remote path. Check it into git. Now git checkout + dvc pull reproduces any historical training data exactly.

yamlCopy
# data/training_set.parquet.dvc
outs:
  - md5: e3b0c44298fc1c149afb
    size: 142857600
    path: training_set.parquet

MLflow for Experiment Tracking

MLflow logs hyperparameters, metrics, and artifacts for each training run. Every training job writes to MLflow before pushing its artifact anywhere.

pythonCopy
import mlflow

def train_and_log(config: dict):
    with mlflow.start_run():
        mlflow.log_params(config)

        model, metrics = train_model(config)

        mlflow.log_metrics(metrics)  # {"auc": 0.91, "f1": 0.78}
        mlflow.sklearn.log_model(model, artifact_path="model", registered_model_name="churn-prediction")

        # Tag the run for promotion filtering
        mlflow.set_tags({"data_version": config["data_hash"], "env": "ci"})

The quality gate script queries MLflow to compare the new run against the champion:

pythonCopy
def quality_gate(challenger_run_id: str, metric: str = "auc", min_delta: float = -0.005):
    client = mlflow.tracking.MlflowClient()
    challenger_metric = client.get_run(challenger_run_id).data.metrics[metric]

    champion = client.get_model_version_by_alias("churn-prediction", "production")
    champion_metric = float(champion.tags.get(f"metric.{metric}", 0))

    delta = challenger_metric - champion_metric
    passed = delta >= min_delta

    print(f"Challenger {metric}: {challenger_metric:.4f}, Champion: {champion_metric:.4f}, delta: {delta:+.4f}")
    if not passed:
        raise SystemExit(f"Quality gate failed: {metric} delta {delta:+.4f} below threshold {min_delta}")

Canary and Shadow Deployments

Shadow deployment sends a copy of live traffic to the new model without affecting the response returned to the user. You collect real predictions and compare them to the production model offline before any user-facing impact.

Canary deployment routes a small fraction of live traffic (1–5%) to the new model with real user impact. Monitor error rate and prediction distribution for 24 hours before scaling to 100%.

pythonCopy
# Feature flag - based canary in your serving layer
import random

def route_request(request, canary_fraction: float = 0.05):
    if random.random() < canary_fraction:
        return new_model.predict(request)
    return production_model.predict(request)

In Kubernetes, canary routing can be implemented with two Deployments and traffic splitting at the ingress level using weights, without requiring application-layer changes.

Common Mistakes

Training in CI without data pinning. If your CI training job pulls "the latest data," two runs on the same commit produce different models. Always pin the data version in your training config and verify the hash at the start of each run.

Skipping the quality gate. Deploying every artifact that builds without comparing it to the current production model means you will eventually ship a regression silently. The quality gate is the ML equivalent of a passing test suite.

One-shot promotion. Going straight from training to 100% traffic with no canary phase eliminates your ability to catch issues before they affect all users. Always route a small percentage first.

Where to Go Next

• mlops-deployment-serving - the deployment step in this pipeline - container building, serving patterns, and rollback
• monitoring-drift-llm-evaluation - the monitoring step that feeds back into the retraining trigger
• milestone-gate-2-production-readiness - use this pipeline as the reference for your gate 2 self-assessment