AI System Design for Product Constraints

In a demo, you control everything: the prompt, the latency, the cost, the data. In production, you operate under a set of constraints that were decided before you wrote a line of code. The engineers who ship AI features sustainably are the ones who treat these constraints as first-class design inputs, not obstacles to work around.

This article covers the five constraints that most commonly get skipped in demos but always matter in production.

1. Latency SLAs

Your product has a response time budget. Find out what it is before you design your AI feature.

pythonCopy
import time
import statistics
from typing import Callable

def measure_latency(fn: Callable, inputs: list, percentiles=(50, 95, 99)) -> dict:
    latencies = []
    for inp in inputs:
        start = time.perf_counter()
        fn(inp)
        latencies.append((time.perf_counter() - start) * 1000)
    latencies.sort()
    return {
        f"p{p}": latencies[int(len(latencies) * p / 100)]
        for p in percentiles
    }

# Typical web SLAs:
# Interactive UI feature: p95 < 500ms
# Background enrichment: p95 < 5s
# Async report: p95 < 30s

# If your LLM call is 2s and your SLA is 500ms, you have four options:
# 1. Stream the response (perceived latency improvement, not real)
# 2. Pre-compute and cache for predictable inputs
# 3. Move the feature to async / background
# 4. Use a faster (smaller) model

Streaming is the most common technique. It shifts the user experience from "wait 2 seconds, then see everything" to "see the first word in 200ms, full response in 2 seconds." The actual latency is the same.

2. Cost Budgets

LLM costs scale with usage in a way that database costs do not. Establish a cost budget per feature before launch.

pythonCopy
class CostBudget:
    def __init__(self, monthly_budget_usd: float, feature: str):
        self.budget = monthly_budget_usd
        self.feature = feature
        self.spent = self._load_current_spend()

    def check(self, estimated_cost: float) -> bool:
        """Returns True if this call is within budget."""
        if self.spent + estimated_cost > self.budget:
            log.warning(f"Feature {self.feature} approaching budget limit "
                        f"(${self.spent:.2f} / ${self.budget:.2f})")
            return False
        return True

    def record(self, actual_cost: float):
        self.spent += actual_cost
        self._persist(actual_cost)

Rule of thumb: prototype with GPT-4o, ship with GPT-4o-mini or equivalent. The 10x cost difference matters at scale. Benchmark quality on your specific task before switching - for many classification and extraction tasks, the difference is negligible.

3. Privacy and Data Residency

Many AI APIs send your data to third-party servers. This is a compliance issue, not just a preference.

pythonCopy
def redact_pii(text: str) -> tuple[str, dict]:
    """Redact PII before sending to external LLM API."""
    import re
    replacements = {}
    counter = 0

    patterns = {
        "email": r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',
        "phone": r'\b\d{3}[-.]?\d{3}[-.]?\d{4}\b',
        "ssn":   r'\b\d{3}-\d{2}-\d{4}\b',
        "card":  r'\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b',
    }
    for pii_type, pattern in patterns.items():
        for match in re.finditer(pattern, text):
            redacted_token = f"[{pii_type.upper()}_{counter}]"
            replacements[redacted_token] = match.group()
            text = text.replace(match.group(), redacted_token)
            counter += 1

    return text, replacements  # restore replacements in post-processing

For heavily regulated environments (healthcare, finance), consider self-hosted models. The operational overhead is real, but it eliminates the data residency problem entirely.

4. Audit Trails

For any AI feature that affects a user outcome (a decision, a recommendation, a content moderation action), you need a complete audit trail.

pythonCopy
from dataclasses import dataclass, asdict
from datetime import datetime, timezone
import uuid

@dataclass
class LLMAuditRecord:
    record_id: str
    feature: str
    user_id: str
    timestamp: str
    model: str
    prompt_hash: str       # hash of prompt, not raw prompt (may contain PII)
    response_summary: str  # truncated, sanitized
    input_tokens: int
    output_tokens: int
    latency_ms: float
    decision: str | None   # if this was a decision-making call

def audit_llm_call(feature, user_id, prompt, response, usage, latency_ms,
                   decision=None):
    import hashlib
    record = LLMAuditRecord(
        record_id=str(uuid.uuid4()),
        feature=feature,
        user_id=user_id,
        timestamp=datetime.now(timezone.utc).isoformat(),
        model=response.model,
        prompt_hash=hashlib.sha256(prompt.encode()).hexdigest()[:16],
        response_summary=response.choices[0].message.content[:200],
        input_tokens=usage.prompt_tokens,
        output_tokens=usage.completion_tokens,
        latency_ms=latency_ms,
        decision=decision,
    )
    audit_store.write(asdict(record))

5. Rate Limits

Your LLM provider has rate limits. Your own API probably has rate limits for the AI features. Model both.

pythonCopy
from asyncio import Semaphore

class RateLimitedLLMClient:
    def __init__(self, requests_per_minute: int = 60):
        self.semaphore = Semaphore(requests_per_minute // 10)
        self.rpm = requests_per_minute

    async def call(self, prompt: str) -> str:
        async with self.semaphore:
            return await self._call_with_retry(prompt)

Common Mistakes

Designing for average load. Your latency budget must hold at p95 under production traffic, not average traffic. Test with concurrent load.

Ignoring egress costs. Many cloud providers charge for data leaving a region. If your self-hosted model is in us-east-1 and your application is in eu-west-1, you are paying egress on every model call.

No cost alerting. Set a billing alert at 80% of your monthly LLM budget. You will hit it.

Where to Go Next

See also: [ai-system-design-reliability], [llm-app-engineering-production], [tool-using-agents-guardrails]