Serving Models and LLM Systems in Production

Training a model is 20% of the work. Serving it reliably, efficiently, and observably at production latency is the other 80%. An ML engineer who can train but cannot serve is only half useful. This module covers the infrastructure decisions that determine whether your model helps users or becomes a maintenance burden.

Serving Options

Approach	Latency	Throughput	Cost	When to use
FastAPI + single model	10–50ms	Low (~100 RPS)	Low	Internal tools, low-traffic
FastAPI + batching	20–100ms	Medium	Medium	Moderate traffic
TorchServe / TF Serving	5–30ms	High	Medium	Dedicated model servers
vLLM	200ms–2s	High for LLMs	GPU-required	LLM serving
Triton Inference Server	1–10ms	Very high	GPU-required	High-throughput vision/NLP
Serverless (SageMaker EP)	100ms–2s cold	Auto-scale	Usage-based	Variable traffic

FastAPI Model Service: Production-Ready Pattern

pythonCopy
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import torch
import numpy as np
from contextlib import asynccontextmanager
import asyncio

# Shared model state
class ModelState:
    model = None
    lock = asyncio.Lock()

state = ModelState()

@asynccontextmanager
async def lifespan(app: FastAPI):
    # Load model once at startup, not per-request
    state.model = torch.load("model_weights.pt", map_location="cpu")
    state.model.eval()
    yield
    state.model = None  # Cleanup

app = FastAPI(lifespan=lifespan)

class PredictRequest(BaseModel):
    features: list[float]

class PredictResponse(BaseModel):
    prediction: int
    confidence: float
    latency_ms: float

@app.post("/predict", response_model=PredictResponse)
async def predict(req: PredictRequest):
    import time
    start = time.perf_counter()

    async with state.lock:
        x = torch.tensor([req.features], dtype=torch.float32)
        with torch.no_grad():
            logits = state.model(x)
            probs = torch.softmax(logits, dim=-1)
            pred = probs.argmax(dim=-1).item()
            conf = probs.max().item()

    latency_ms = (time.perf_counter() - start) * 1000
    return PredictResponse(prediction=pred, confidence=conf, latency_ms=latency_ms)

@app.get("/health")
async def health():
    if state.model is None:
        raise HTTPException(status_code=503, detail="Model not loaded")
    return {"status": "ok"}

Request Batching for Throughput

Individual requests processed one at a time underutilize GPU parallelism. Batching collects requests over a short window and processes them together:

pythonCopy
import asyncio
from collections import deque
import time

class BatchProcessor:
    def __init__(self, model, batch_size: int = 32, max_wait_ms: float = 10.0):
        self.model = model
        self.batch_size = batch_size
        self.max_wait_ms = max_wait_ms
        self.queue = deque()
        self.lock = asyncio.Lock()

    async def predict(self, features: list[float]) -> dict:
        future = asyncio.get_event_loop().create_future()
        async with self.lock:
            self.queue.append((features, future))
            if len(self.queue) >= self.batch_size:
                await self._flush()

        # Wait for result (may be set by another request's flush)
        return await future

    async def _flush(self):
        if not self.queue:
            return
        batch = list(self.queue)
        self.queue.clear()

        features_list = [item[0] for item in batch]
        futures = [item[1] for item in batch]

        x = torch.tensor(features_list, dtype=torch.float32)
        with torch.no_grad():
            logits = self.model(x)
            probs = torch.softmax(logits, dim=-1)

        for i, future in enumerate(futures):
            if not future.done():
                future.set_result({
                    "prediction": probs[i].argmax().item(),
                    "confidence": probs[i].max().item(),
                })

Dynamic batching is the standard GPU-serving pattern. vLLM and Triton implement sophisticated versions of this automatically for LLMs and other deep learning models.

vLLM for LLM Serving

For serving quantized or fine-tuned LLMs in production, vLLM provides dramatically better throughput than HuggingFace generate:

bashCopy
# Install and launch vLLM server
pip install vllm
python -m vllm.entrypoints.openai.api_server \
    --model mistralai/Mistral-7B-Instruct-v0.2 \
    --tensor-parallel-size 1 \
    --max-model-len 4096 \
    --port 8000

vLLM's PagedAttention manages KV cache memory like an OS manages virtual memory, dramatically reducing fragmentation and enabling higher batch sizes. Benchmark: HuggingFace generate at ~10 tokens/sec → vLLM at ~80–200 tokens/sec on the same GPU.

Call the vLLM server via the OpenAI-compatible API:

pythonCopy
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")

response = client.chat.completions.create(
    model="mistralai/Mistral-7B-Instruct-v0.2",
    messages=[{"role": "user", "content": "Explain gradient descent."}],
    max_tokens=512,
    temperature=0.0,
)
print(response.choices[0].message.content)

Model Quantization

Full-precision (float32) models are large and slow. Quantization reduces weights to int8 or int4, dramatically shrinking memory usage and improving inference speed:

pythonCopy
# bitsandbytes 4-bit quantization for HuggingFace models
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",     # Normal float 4-bit
    bnb_4bit_compute_dtype="bfloat16",
    bnb_4bit_use_double_quant=True,  # Double quantization saves another ~0.4 bits/param
)

model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.2",
    quantization_config=bnb_config,
    device_map="auto",
)
# 7B model: ~28GB in float32 → ~4GB in 4-bit

Quality vs. compression:

• float16/bfloat16: ~5-10% memory reduction, negligible quality loss
• int8: ~50% reduction, ~1-2% quality degradation on benchmarks
• 4-bit (NF4): ~75% reduction, ~3-5% quality degradation - acceptable for most tasks

Caching Strategies

pythonCopy
import hashlib
import json
import redis

r = redis.Redis(host="localhost", port=6379)

def cache_inference(model_name: str, features: list[float], ttl: int = 3600) -> dict | None:
    key = hashlib.sha256(
        json.dumps({"model": model_name, "features": features}).encode()
    ).hexdigest()

    cached = r.get(f"inference:{key}")
    if cached:
        return json.loads(cached)
    return None

def cache_store(model_name: str, features: list[float], result: dict, ttl: int = 3600):
    key = hashlib.sha256(
        json.dumps({"model": model_name, "features": features}).encode()
    ).hexdigest()
    r.setex(f"inference:{key}", ttl, json.dumps(result))

Cache deterministic model predictions (temperature=0, no randomness). Cache hit rates of 30-60% are common for recommendation and classification models - the cost savings are significant.

Model Routing, Cascades, and the Open-Weight Decision

Serving one model behind one endpoint is the 2023 architecture. Production LLM systems in 2026 almost always serve several, and the interesting engineering is in deciding which request goes where.

Routing sends each request to the cheapest model that can handle it. The router can be a rule ("classification requests go to the small model"), a classifier trained on your own eval results, or the small model itself deciding whether to escalate. Cascades try the cheap model first and escalate only when a verifier (a schema check, a confidence score, a judge) rejects the answer.

pythonCopy
def answer(request):
    small = SMALL.generate(request)
    if verifier.accepts(request, small):
        return small, "small"
    if request.needs_reasoning:
        return REASONING.generate(request), "reasoning"
    return LARGE.generate(request), "large"

Log the route taken on every request. Route distribution is one of the most useful cost and quality signals you will have, and a shift in it (suddenly 40% escalations instead of 10%) is an early warning that inputs changed.

Open-weight vs API is now a real choice rather than a compromise. Open-weight model families (Llama, Qwen, DeepSeek, Gemma, Mistral, among others) are competitive on many tasks, and a 7B-class model at 4-bit runs on a single consumer GPU or a modern phone. Self-host when at least one of these is true: data cannot leave your environment, per-token cost at your volume beats the engineering cost of running inference, you need to fine-tune, or you need deterministic latency. Use an API when you need the frontier, when volume is low, or when you do not have someone to own the serving stack. Most teams end up hybrid: an open-weight small model for the high-volume path and an API model for escalation.

Prompt Caching

Every major API now caches the prefix of a prompt across requests, and it changes how you should structure prompts. The cache matches on an exact prefix, so stable content (system prompt, tool definitions, reference documents) goes first and variable content (the user's message) goes last. A cache hit is typically billed at a fraction of the input price and skips the prefill compute, so on a long system prompt or a large tool list the difference is not marginal.

pythonCopy
messages = [
    {"role": "system", "content": [
        {"type": "text", "text": STABLE_SYSTEM_PROMPT},
        {"type": "text", "text": TOOL_DOCS, "cache_control": {"type": "ephemeral"}},
    ]},
    {"role": "user", "content": user_message},   # varies; after the cached prefix
]

Two operational rules. First, treat cache hit rate as a metric you monitor, next to latency and cost per request; a drop usually means someone changed the "stable" part of the prompt in a way that busted the cache. Second, do not put anything that changes per request (timestamps, request IDs, the user's name) in the cached prefix. It is the most common way teams pay full price for a prompt they think is cached.

Common Mistakes and Bad Instincts

Loading the model inside the request handler. Model loading takes seconds. Loading on every request adds seconds to every request latency. Load once at startup.

Not implementing health checks and readiness probes. Kubernetes and load balancers need to know when a serving pod is ready (model loaded, warm) vs. healthy (not crashed). Implement /health and /ready endpoints from day one.

Ignoring tail latency. Mean latency of 50ms is fine. p99 of 2 seconds means 1% of users wait 40x longer. Profile under load - tail latency often comes from garbage collection, cache misses, or queue backup.

Deploying without a rollback plan. If a new model version degrades key metrics, you need to roll back within minutes. Keep the previous model artifact and use a deployment pattern (blue/green, canary) that makes rollback a config change, not a redeployment.

Where to Go Next

• mlops-and-cicd-for-ml-teams: automate model deployment and integrate serving into your CI/CD pipeline
• observability-drift-feedback-loops-and-llm-evals: monitor serving latency, error rates, and model output quality in production
• ai-system-design-quality-cost-latency-and-safety-tradeoffs: make architectural choices about when to use API models vs. self-hosted serving

What to Practice Next

• Containerize a model server with Docker, add a /healthz liveness probe and a /readyz readiness probe, and verify that a Kubernetes (or local k3s) deployment restarts the pod correctly on a simulated failure.
• Implement a canary deployment for a model update: route 10% of traffic to the new version, monitor the quality signal and error rate for 30 minutes, and write the rollback command before you deploy.
• Set up autoscaling based on a custom metric (e.g., GPU utilization or request queue depth) using KEDA or HPA, and document the minimum and maximum replica settings and the reasoning behind them.