Model Serving, APIs, Inference, and Performance Tradeoffs

A trained model in a file is not a product. A production inference system handles concurrent requests, meets latency SLOs, degrades gracefully under load, and can be updated without downtime. This module covers the engineering behind putting models into production and the performance trade-offs that shape every deployment decision.

Batch vs. Online Inference

Batch inference: run the model on a large dataset on a schedule (daily, hourly). Results are precomputed and stored.

pythonCopy
# Daily batch scoring job
import pandas as pd
import lightgbm as lgb
from datetime import datetime

def batch_score(model_path: str, features_path: str, output_path: str):
    model = lgb.Booster(model_file=model_path)
    features = pd.read_parquet(features_path)

    scores = model.predict(features.drop(columns=['user_id']))
    output = features[['user_id']].copy()
    output['churn_probability'] = scores
    output['scored_at'] = datetime.utcnow().isoformat()
    output.to_parquet(output_path, index=False)
    print(f"Scored {len(output)} users → {output_path}")

Use batch inference when: latency > minutes is acceptable, you can precompute scores for all entities, and the feature computation cost is significant.

Online inference: serve the model as an API that responds to individual requests in real time.

Use online inference when: the request context is not known in advance (e.g., depends on the specific user action), freshness matters, or you need < 100ms latency.

Building a Serving API with FastAPI

pythonCopy
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import lightgbm as lgb
import numpy as np
import time

app = FastAPI()
model = lgb.Booster(model_file='model.lgb')
feature_names = model.feature_name()  # ordered list of features the model expects

class PredictionRequest(BaseModel):
    user_id: str
    features: dict[str, float]

class PredictionResponse(BaseModel):
    user_id: str
    churn_probability: float
    latency_ms: float

@app.post("/predict", response_model=PredictionResponse)
async def predict(req: PredictionRequest):
    t0 = time.time()

    # Validate feature completeness
    missing = set(feature_names) - set(req.features.keys())
    if missing:
        raise HTTPException(status_code=422, detail=f"Missing features: {missing}")

    # Assemble feature vector in model-expected order
    feature_vector = np.array([[req.features[f] for f in feature_names]])
    score = float(model.predict(feature_vector)[0])

    return PredictionResponse(
        user_id=req.user_id,
        churn_probability=score,
        latency_ms=(time.time() - t0) * 1000,
    )

@app.get("/health")
async def health():
    return {"status": "ok", "model_loaded": model is not None}

The /health endpoint is required by load balancers and container orchestrators - without it, traffic is never routed to your instance.

Latency Sources and Optimization

Understanding where latency comes from determines which optimization to apply:

Component	Typical Cost	Optimization
Model inference (tree)	1–20ms	Batch multiple requests together
Model inference (neural net, CPU)	50–200ms	Use GPU or quantize
Feature retrieval (DB query)	5–50ms	Use in-memory cache or feature store
Network overhead	1–10ms	Colocate model with feature store
Python overhead	1–5ms	Use numpy batching, avoid Python loops

Request batching: if you receive 100 requests/sec, batching 10 requests together for a single model.predict() call often gives 5–10x throughput improvement with minimal latency increase.

pythonCopy
import asyncio
from collections import defaultdict

class BatchPredictor:
    def __init__(self, model, batch_size=16, max_wait_ms=10):
        self.model = model
        self.batch_size = batch_size
        self.max_wait_ms = max_wait_ms
        self.queue = asyncio.Queue()

    async def predict_single(self, features: np.ndarray) -> float:
        future = asyncio.get_event_loop().create_future()
        await self.queue.put((features, future))
        return await future

    async def process_batches(self):
        while True:
            batch_items = []
            deadline = asyncio.get_event_loop().time() + self.max_wait_ms / 1000
            while len(batch_items) < self.batch_size:
                timeout = deadline - asyncio.get_event_loop().time()
                if timeout <= 0:
                    break
                try:
                    item = await asyncio.wait_for(self.queue.get(), timeout)
                    batch_items.append(item)
                except asyncio.TimeoutError:
                    break
            if batch_items:
                features_batch = np.vstack([f for f, _ in batch_items])
                scores = self.model.predict(features_batch)
                for (_, future), score in zip(batch_items, scores):
                    future.set_result(float(score))

Model Quantization: Smaller, Faster

Quantization reduces model precision (float32 → int8 or float16) to reduce memory and increase inference speed with minimal accuracy loss.

pythonCopy
import torch

# Dynamic quantization: apply at load time, no calibration data needed
# Good for transformer models
quantized_model = torch.quantization.quantize_dynamic(
    model, {torch.nn.Linear}, dtype=torch.qint8
)

# Before: model size 400MB, inference 120ms
# After: model size ~100MB, inference ~45ms on CPU (typical result)

# For production GPU serving, use float16 (bfloat16 on newer hardware)
model = model.half()  # float16
model = model.to('cuda')

Deployment Patterns

Blue-green deployment: run two identical environments (blue = current, green = new version). Switch traffic atomically when green is validated. Instant rollback by switching back to blue.

Canary release: send 5% of traffic to the new model version, monitor metrics for 1–24 hours, then gradually increase to 100%. Detects production degradation before it affects all users.

Shadow mode: route production traffic to the new model but ignore its output. Compare predictions against the current model to validate behavior without risk.

pythonCopy
# Canary traffic split (in nginx upstream or application layer)
import random

def route_request(user_id: str, canary_fraction: float = 0.05) -> str:
    # Deterministic routing by user_id ensures the same user always gets the same version
    hash_val = int(hashlib.md5(user_id.encode()).hexdigest(), 16) / (16**32)
    return "canary" if hash_val < canary_fraction else "stable"

Common Mistakes and Bad Instincts

No health check endpoint. Load balancers need /health to route traffic correctly. A missing health check means zero traffic after a restart.

Not validating feature names and order. The serving API and the training code must agree on the exact order of features. Mismatches produce wrong predictions silently - no error, just bad scores.

Serving raw model.predict() output without calibration. Gradient boosting outputs are not well-calibrated probabilities out of the box. If your product uses the score for decision-making (charge a different price, send a retention offer), calibrate first.

Loading the model from disk on every request. Load the model once at startup, keep it in memory. Model file loading can take 1–10 seconds and must not happen in the request path.

No latency SLO. "Make it fast" is not a SLO. Define p99 latency requirement (e.g., "99% of requests under 100ms") and instrument your serving to measure and alert on it.

Where to Go Next

• Module 26 (MLOps CI/CD) covers how to automate testing and validation of serving code before deploying.
• Module 27 (Observability) covers monitoring the latency, accuracy, and drift of live models.
• The standalone post mlops-fundamentals-for-production covers deployment patterns in more depth.

What to Practice Next

• Wrap a scikit-learn or PyTorch model in a FastAPI endpoint, add Pydantic request/response validation, and run a load test with Locust at 50 concurrent users - record p50, p95, and p99 latency.
• Compare synchronous vs. batched inference for the same model: measure throughput (requests/second) and p99 latency under identical load for batch sizes 1, 8, and 32.
• Deploy the same model using both a REST endpoint and Triton Inference Server (or TorchServe), and benchmark the two for throughput - note the operational complexity trade-off alongside the performance numbers.