AI System Design: Quality, Cost, Latency, and Safety Tradeoffs

AI system design is not about picking the most impressive model - it is about making the right engineering tradeoffs for the problem's constraints. A 70B parameter model that answers correctly 95% of the time is not always better than a 7B model at 88% accuracy if the 7B model runs 10x faster and costs 20x less. The best AI engineers navigate quality, cost, latency, and safety tradeoffs explicitly rather than defaulting to "use the best available model."

The Tradeoff Space

Every AI system design decision lives in a four-dimensional space:

Dimension	What matters	How to measure
Quality	Task accuracy, user satisfaction	Eval harness, human ratings, business metrics
Cost	Compute cost per request	$/1K tokens (API), GPU-hours (self-hosted)
Latency	p50, p99 response time	ms from request to response complete
Safety	Harmful output rate, reliability, privacy	Adversarial eval, audit, incident rate

These dimensions trade off. Better quality usually means larger models (higher cost, higher latency). Stronger safety usually means more filtering and validation (higher latency). Understanding the constraints lets you make defensible design decisions.

Architecture Patterns and When to Use Each

Pattern 1: Direct Prompting

Use when: general-purpose tasks, low-volume features, prototyping.

User → Prompt → LLM API → Response

Cost: high per-request (frontier models). Latency: 500ms–2s. Quality: high out-of-the-box.

Suitable for: internal tools, low-frequency features, tasks with high variance that need general reasoning.

Pattern 2: RAG (Retrieval-Augmented Generation)

Use when: knowledge is domain-specific, frequently updated, or too large for context.

User → Retrieve(knowledge base) → Prompt + Context → LLM → Response

Cost: retrieval is cheap; LLM call is reduced by using smaller model with better context. Latency: +50–200ms for retrieval.

Suitable for: Q&A over private docs, customer support, knowledge bases. Default choice over fine-tuning for factual grounding.

Pattern 3: Fine-Tuned Smaller Model

Use when: high-volume, narrow task, consistent format.

User → Prompt → Fine-tuned 7B model → Response

Cost: low per-request after upfront training cost. Latency: 50–200ms on GPU.

Suitable for: classification, extraction, structured generation at scale. Break-even vs. API typically at ~1M requests.

Pattern 4: Classifier + LLM Routing

Use when: most requests are simple but some require complex reasoning.

User → Fast Classifier → Simple: small model
                       → Complex: large model

pythonCopy
def route_request(query: str) -> str:
    complexity = classify_complexity(query)  # Fast, cheap classifier
    if complexity == "simple":
        return call_llm("gpt-4o-mini", query, max_tokens=256)
    else:
        return call_llm("gpt-4o", query, max_tokens=1024)

# Typical result: 70% of queries go to cheap model
# Cost reduction: 60-80% with < 5% quality degradation on aggregate

Pattern 5: Cached Responses

Use when: many users ask semantically similar questions.

pythonCopy
from sentence_transformers import SentenceTransformer
import numpy as np

embedding_model = SentenceTransformer("bge-small-en-v1.5")
cache = {}  # In production: Redis + FAISS index over cached queries

def semantic_cache_lookup(query: str, threshold: float = 0.95) -> str | None:
    query_emb = embedding_model.encode([query], normalize_embeddings=True)[0]
    for cached_query, (cached_emb, cached_response) in cache.items():
        similarity = np.dot(query_emb, cached_emb)
        if similarity >= threshold:
            return cached_response
    return None

Semantic caching (not just exact-match) can achieve 30-50% cache hit rates on FAQ-style use cases, reducing LLM calls proportionally.

Safety and Reliability Design

Input Validation

pythonCopy
def validate_input(user_message: str) -> tuple[bool, str]:
    # Length limit
    if len(user_message) > 10_000:
        return False, "Message too long. Please be more concise."

    # Basic injection patterns
    injection_patterns = [
        "ignore all previous instructions",
        "you are now",
        "system: ",
    ]
    lower = user_message.lower()
    for pattern in injection_patterns:
        if pattern in lower:
            return False, "Input contains disallowed patterns."

    return True, ""

Output Guardrails

pythonCopy
from pydantic import BaseModel, validator
from typing import Optional
import re

class SafeResponse(BaseModel):
    content: str
    blocked: bool = False
    block_reason: Optional[str] = None

    @validator("content")
    def check_pii(cls, v):
        # Detect and redact common PII patterns
        v = re.sub(r'\b\d{3}-\d{2}-\d{4}\b', '[SSN REDACTED]', v)
        v = re.sub(r'\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b', '[CARD REDACTED]', v)
        return v

def apply_guardrails(response: str) -> SafeResponse:
    # Content moderation API call
    # In production: OpenAI Moderation API, Azure Content Safety, or custom classifier
    is_safe = check_content_safety(response)
    if not is_safe:
        return SafeResponse(content="", blocked=True, block_reason="Content policy violation")
    return SafeResponse(content=response)

Graceful Degradation

pythonCopy
def answer_with_fallback(query: str) -> dict:
    try:
        # Tier 1: full RAG + large model
        return {"answer": rag_answer(query), "tier": "full"}
    except Exception as e:
        log_error("rag_failed", e)
        try:
            # Tier 2: direct LLM without retrieval
            return {"answer": direct_llm(query), "tier": "no_rag"}
        except Exception as e2:
            log_error("llm_failed", e2)
            # Tier 3: static fallback
            return {
                "answer": "I am unable to process your request right now. Please try again later.",
                "tier": "fallback"
            }

Cost Modeling

Before committing to an architecture, model the cost at your expected usage:

pythonCopy
def estimate_monthly_cost(
    monthly_requests: int,
    avg_input_tokens: int,
    avg_output_tokens: int,
    model: str = "gpt-4o-mini",
) -> dict:
    # Example rates - always check current provider pricing
    pricing = {
        "gpt-4o-mini": {"input": 0.15, "output": 0.60},     # per 1M tokens
        "gpt-4o":      {"input": 2.50, "output": 10.00},
        "claude-3-5-haiku": {"input": 0.80, "output": 4.00},
    }
    rates = pricing.get(model, {"input": 2.0, "output": 8.0})

    input_cost = (monthly_requests * avg_input_tokens / 1_000_000) * rates["input"]
    output_cost = (monthly_requests * avg_output_tokens / 1_000_000) * rates["output"]
    total = input_cost + output_cost

    return {
        "model": model,
        "monthly_requests": monthly_requests,
        "monthly_cost_usd": round(total, 2),
        "cost_per_request_cents": round(total / monthly_requests * 100, 4),
    }

# Compare architectures
for req_count in [10_000, 100_000, 1_000_000]:
    estimate = estimate_monthly_cost(req_count, avg_input_tokens=500, avg_output_tokens=200)
    print(estimate)

At 1M requests/month with gpt-4o-mini: ~$225/month. At 1M requests with gpt-4o: ~$3,250/month. The routing pattern (70% mini, 30% 4o) costs ~$1,130. Cost modeling informs whether fine-tuning a self-hosted model is worth the engineering investment.

Common Mistakes and Bad Instincts

Defaulting to the most capable model for everything. GPT-4o is not always better than GPT-4o-mini for structured extraction tasks where the format is well-defined. Benchmark both before assuming the larger model is worth 10x the cost.

Not defining a fallback path before launch. "We'll add a fallback if it breaks" becomes "we're down because we have no fallback." Design the degradation path as part of the feature, not as a future ticket.

Treating safety as a post-launch concern. Output safety issues are much harder to fix in production than in design. Run red-team testing and apply guardrails before launch, not after the first incident.

Building a complex multi-model pipeline when a single model with better prompting would work. Complexity compounds failure rates. Each additional LLM call is a new failure point and a new cost. Start with the simplest architecture that achieves your quality target.

Where to Go Next

• portfolio-conversion-turning-engineering-work-into-ml-evidence: articulate these system design decisions as portfolio evidence for ML engineering roles
• interview-readiness-for-ml-ai-engineering-roles: practice system design interview questions using this framework