Latency vs. Quality Tradeoffs in ML Systems

Every production ML system faces the same fundamental tension: better models are slower, and faster models are less accurate. You need a concrete way to reason about this tradeoff, especially in system design interviews.

The Latency Budget Model

Start with what the user experience demands:

User-facing feature	Acceptable total latency	ML latency budget
Search autocomplete	50ms	10–20ms
Product search results	200ms	100ms
Recommendation feed (page load)	500ms	200ms
Content recommendation (background)	2s	1s
LLM chat (first token)	1s	800ms
Batch report	10 minutes	8 minutes

Your ML system must fit within the allocated budget. Work backwards from this constraint, not forward from model complexity.

The Cascade Architecture

Rather than using one model for everything, use cheaper models first and expensive models only when needed:

pythonCopy
async def predict_cascade(features: dict) -> dict:
    """
    Stage 1: Fast heuristic or tiny model  -  handles 80% of cases
    Stage 2: Medium model  -  handles 15% that Stage 1 is uncertain on
    Stage 3: Expensive model  -  handles the 5% that need it
    """

    # Stage 1: Rule-based or fast linear model (< 5ms)
    stage1_score = fast_linear_model.predict(features)
    stage1_confidence = abs(stage1_score - 0.5) * 2  # 0 = uncertain, 1 = confident

    if stage1_confidence > 0.8:
        return {
            "prediction": stage1_score,
            "model_used": "stage1",
            "latency_saved_ms": 150  # we skipped stages 2 and 3
        }

    # Stage 2: GBT or medium neural net (< 30ms)
    stage2_score = medium_model.predict(features)
    stage2_confidence = abs(stage2_score - 0.5) * 2

    if stage2_confidence > 0.7:
        return {
            "prediction": stage2_score,
            "model_used": "stage2",
            "latency_saved_ms": 120
        }

    # Stage 3: Full model (< 150ms)
    stage3_score = full_model.predict(features)
    return {
        "prediction": stage3_score,
        "model_used": "stage3"
    }

This gives you near-full-model quality at a fraction of the average latency because most cases are easy.

Model Quantization: The Free Lunch

Quantizing from 32-bit floats to 8-bit integers typically gives:

• 4× memory reduction
• 2–4× inference speedup
• <1% accuracy loss for most tasks

pythonCopy
import torch
from torch.quantization import quantize_dynamic

# Post-training dynamic quantization (simplest approach)
model_fp32 = load_model()
model_int8 = quantize_dynamic(
    model_fp32,
    {torch.nn.Linear},  # quantize all linear layers
    dtype=torch.qint8
)

# Measure the speedup
import time

def benchmark(model, inputs, n=100):
    # Warmup
    for _ in range(10):
        model(inputs)
    start = time.perf_counter()
    for _ in range(n):
        model(inputs)
    return (time.perf_counter() - start) / n * 1000  # ms

fp32_latency = benchmark(model_fp32, test_inputs)
int8_latency = benchmark(model_int8, test_inputs)
print(f"FP32: {fp32_latency:.1f}ms | INT8: {int8_latency:.1f}ms | Speedup: {fp32_latency/int8_latency:.1f}x")

For production deployment, also consider:

• ONNX Runtime: often 2–5× faster than native PyTorch for inference
• TensorRT (NVIDIA GPUs): 5–10× speedup through kernel fusion
• Distillation: train a smaller model to mimic a larger one

Caching Strategies

The fastest inference is the one you do not have to run:

pythonCopy
import hashlib
import redis
import json

r = redis.Redis()

def cached_predict(features: dict, model, ttl_seconds: int = 300):
    """
    Cache predictions for identical feature vectors.
    Useful when many users see the same content (popular items, trending topics).
    """
    cache_key = hashlib.md5(json.dumps(features, sort_keys=True).encode()).hexdigest()
    cached = r.get(cache_key)

    if cached:
        result = json.loads(cached)
        result['from_cache'] = True
        return result

    prediction = model.predict(features)
    result = {"prediction": prediction, "from_cache": False}
    r.setex(cache_key, ttl_seconds, json.dumps(result))
    return result

When caching is effective:

• Item-level features dominate (high cache hit rate)
• Predictions do not need to be real-time (5-minute TTL is fine)
• Corpus is not too large (millions, not billions)

When caching breaks down:

• User-specific personalization means every (user, item) pair is unique
• Real-time features change constantly (current inventory, live pricing)

Approximate Nearest Neighbor: The Retrieval Tradeoff

For embedding-based retrieval (search, recommendations), exact nearest neighbor search is O(n). At scale, you use approximate nearest neighbor (ANN) algorithms that trade recall for speed:

pythonCopy
import faiss
import numpy as np

d = 128  # embedding dimension
n_items = 1_000_000

# Exact search (slow but accurate  -  good for benchmarking)
index_flat = faiss.IndexFlatIP(d)
index_flat.add(item_embeddings)

# IVF index: partition space into clusters, only search nearby clusters
# nlist = number of partitions; nprobe = how many to search
nlist = 1000
index_ivf = faiss.IndexIVFFlat(faiss.IndexFlatIP(d), d, nlist)
index_ivf.train(item_embeddings)
index_ivf.add(item_embeddings)
index_ivf.nprobe = 10  # search 10 of 1000 clusters  -  fast but may miss some results

# IVF-PQ: also compress vectors (HNSW for even higher throughput)
# Benchmark the recall-latency tradeoff
def benchmark_ann(index, queries, k=10):
    start = time.perf_counter()
    _, indices = index.search(queries, k)
    latency = (time.perf_counter() - start) / len(queries) * 1000
    return latency, indices

# Tune nprobe to find your operating point on the recall-latency curve
for nprobe in [1, 5, 10, 50, 100]:
    index_ivf.nprobe = nprobe
    latency, approx_indices = benchmark_ann(index_ivf, test_queries)
    _, exact_indices = benchmark_ann(index_flat, test_queries)
    recall = np.mean([
        len(set(approx_indices[i]) & set(exact_indices[i])) / k
        for i in range(len(test_queries))
    ])
    print(f"nprobe={nprobe:3d}: {latency:.1f}ms, recall@10={recall:.3f}")

Asynchronous and Precomputed Predictions

Not all predictions need to happen at request time:

pythonCopy
# Pattern: precompute for the next session
# When user logs out or session ends, precompute their next-session recommendations
# in a background job so they are ready instantly on next visit

import celery

app = celery.Celery()

@app.task
def precompute_user_recommendations(user_id: int):
    user = get_user(user_id)
    recommendations = get_recommendations(user, k=50)
    # Store in Redis with 24-hour TTL
    r.setex(
        f"recs:{user_id}",
        86400,
        json.dumps([r['item_id'] for r in recommendations])
    )

# Trigger after session end
def on_session_end(user_id: int):
    precompute_user_recommendations.delay(user_id)

# Serve precomputed recs instantly
def get_homepage_recs(user_id: int) -> list:
    cached = r.get(f"recs:{user_id}")
    if cached:
        return json.loads(cached)
    # Fallback: compute synchronously (cold start)
    return get_recommendations(get_user(user_id), k=20)

Thinking About It in System Design Interviews

When asked about latency-quality tradeoffs in an interview, structure your answer:

1. State the constraint - what is the acceptable latency for this user-facing feature?
2. Identify the bottleneck - is it retrieval, model inference, or feature computation?
3. Apply the right technique - cascade models, quantization, caching, precomputation, ANN search
4. Quantify the tradeoff - "this approach reduces latency by 3× at a cost of <2% accuracy loss"
5. Design for monitoring - how will you know if the approximation is hurting quality?

The most common interview mistake is proposing a complex model without addressing whether it can meet latency requirements. Always ask about the latency budget before designing the system.

What to Practice Next

• Profile a model inference endpoint end-to-end with py-spy or torch.profiler - break down time spent in preprocessing, model forward pass, and postprocessing, then identify where the largest gains are available.
• Run a quantization experiment: compare FP32 vs. INT8 (using torch.quantization or bitsandbytes) on the same model for throughput, p50/p99 latency, and quality metric - produce a two-row comparison table.
• Define SLOs for a hypothetical production system (e.g., p99 latency < 300 ms, quality metric > 0.85) and determine which model configuration in your experiment satisfies both constraints simultaneously.