MLOps Deployment and Serving

Deploying a trained model is not the finish line - it is where the engineering work begins. A model that lives only in a Jupyter notebook or an S3 bucket delivers zero value. Getting it in front of real traffic, keeping it healthy, and being able to roll back when something goes wrong is the job of ML deployment engineering.

The Full Deployment Lifecycle

Every production deployment follows the same arc regardless of model type:

1. Package - containerize the model and its runtime dependencies
2. Register - push the image and model artifact to a registry
3. Serve - expose an HTTP endpoint that loads the model and handles requests
4. Route - send a fraction of traffic to the new version
5. Monitor - watch latency, error rate, and prediction distribution
6. Promote or rollback - cut over fully or revert based on observed signal

Building a Serving Container

A minimal FastAPI serving container looks like this:

pythonCopy
# app/main.py
import mlflow
import pandas as pd
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

app = FastAPI()
model = mlflow.pyfunc.load_model("/models/churn_v3")

class PredictRequest(BaseModel):
    user_id: str
    features: dict[str, float]

class PredictResponse(BaseModel):
    user_id: str
    score: float
    label: str

@app.post("/predict", response_model=PredictResponse)
def predict(req: PredictRequest):
    df = pd.DataFrame([req.features])
    score = float(model.predict(df)[0])
    return PredictResponse(user_id=req.user_id, score=score, label="churn" if score > 0.5 else "retain")

@app.get("/health")
def health():
    return {"status": "ok"}

The Dockerfile pins the Python version, installs only what the model needs, and copies the frozen artifact:

dockerfileCopy
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app/ ./app/
COPY models/ ./models/
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080"]

Batching for Throughput

A single-sample endpoint works for low-traffic scenarios. For high-throughput inference, accept a list of requests and run a single model forward pass:

pythonCopy
class BatchPredictRequest(BaseModel):
    requests: list[PredictRequest]

@app.post("/predict/batch")
def predict_batch(req: BatchPredictRequest):
    df = pd.DataFrame([r.features for r in req.requests])
    scores = model.predict(df).tolist()
    return [
        {"user_id": r.user_id, "score": s, "label": "churn" if s > 0.5 else "retain"}
        for r, s in zip(req.requests, scores)
    ]

Batch inference reduces per-sample overhead dramatically. A model that takes 2 ms per sample in a single-request loop may take 0.3 ms per sample in a batch of 64.

GPU vs CPU Serving

For large neural networks, GPU serving cuts latency by 10–100x. For gradient boosting or small sklearn models, CPU is cheaper and sufficient.

Factor	CPU	GPU
Cost per hour	$0.05–0.20	$0.50–3.00
Latency (small model)	1–5 ms	5–15 ms (CUDA init)
Latency (transformer)	200–800 ms	10–50 ms
Scaling unit	vCPU	GPU card

For GPU serving, use NVIDIA Triton Inference Server or TorchServe for batching and model management rather than rolling your own.

Blue-Green Deployment

Blue-green keeps two identical environments. Blue is live; green is the next version. To cut over, route 100% of traffic from blue to green in one switch. Rollback is a single routing change back to blue.

yamlCopy
# kubernetes/service.yaml - patch the selector to switch environments
apiVersion: v1
kind: Service
metadata:
  name: churn-model
spec:
  selector:
    version: green   # was: blue
  ports:
    - port: 80
      targetPort: 8080

Keep the blue deployment running for 30 minutes after the cutover. If alerts fire, patch the selector back to blue.

Rollback Runbook

A rollback should take under two minutes. Document it before you need it:

1. Identify the last stable image tag from your registry (mlflow-model:v12)
2. Update the Kubernetes deployment: kubectl set image deployment/churn-model app=registry/churn-model:v12
3. Wait for rollout: kubectl rollout status deployment/churn-model
4. Confirm health: curl https://api.internal/churn/health
5. Post incident note in Slack with timestamp and rollback reason

Production Deployment Checklist

• Model artifact stored in a versioned registry (MLflow, S3 + DVC)
• Serving container builds and passes health check in CI
• Latency p50 / p99 measured under load before promotion
• Blue-green or canary routing configured - not direct cutover
• Rollback runbook written and tested in staging
• Prediction logging enabled for drift detection
• Alert on error rate > 1% and latency p99 > SLA threshold

Common Mistakes

Baking the model into the image. Pulling a 2 GB model file at image build time makes rebuilds slow and images large. Instead, load the model from a registry or object store at container startup.

Skipping load testing. A model that handles 10 RPS in development may fall over at 200 RPS in production. Run locust or k6 against staging before any promotion.

No rollback plan. Teams that skip the rollback runbook always regret it at 2 AM. Write it before you deploy.

Where to Go Next

• mlops-cicd-ml-systems - wire this deployment into a full CI/CD pipeline with automated quality gates
• monitoring-drift-llm-evaluation - instrument your serving endpoint for drift detection and alerting
• observability-evalops-governance - extend observability beyond latency to data quality and model quality metrics