AI System Design and Reliability

Traditional software has well-understood reliability patterns: retry, circuit breaker, bulkhead, timeout. LLM-based systems need all of those, plus a set of patterns specific to the probabilistic, slow, and expensive nature of language models. This article covers what a reliable AI system looks like in production.

The Reliability Baseline

Before adding AI-specific patterns, apply the standard ones. LLM API calls are HTTP calls.

pythonCopy
import httpx
import asyncio
from functools import wraps

# Timeout: every LLM call must have one
# For streaming: timeout applies to first token, not completion
TIMEOUT = httpx.Timeout(connect=5.0, read=60.0, write=10.0, pool=5.0)

# Retry with jitter
async def retry_with_backoff(fn, max_retries=3, base_delay=1.0):
    for attempt in range(max_retries):
        try:
            return await fn()
        except (httpx.TimeoutException, httpx.ConnectError) as e:
            if attempt == max_retries - 1:
                raise
            delay = base_delay * (2 ** attempt) + (0.1 * attempt)
            await asyncio.sleep(delay)

Circuit Breaker for LLM Calls

A circuit breaker prevents cascading failures. When the LLM API is unavailable, stop hammering it.

pythonCopy
from enum import Enum
import time
import threading

class CircuitState(Enum):
    CLOSED = "closed"      # normal operation
    OPEN = "open"          # failing, reject calls fast
    HALF_OPEN = "half_open"  # testing if service recovered

class LLMCircuitBreaker:
    def __init__(self, failure_threshold=5, recovery_timeout=60):
        self.state = CircuitState.CLOSED
        self.failures = 0
        self.last_failure_time = None
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self._lock = threading.Lock()

    def call(self, fn, *args, **kwargs):
        with self._lock:
            if self.state == CircuitState.OPEN:
                if time.time() - self.last_failure_time > self.recovery_timeout:
                    self.state = CircuitState.HALF_OPEN
                else:
                    raise RuntimeError("Circuit is OPEN - LLM service unavailable")

        try:
            result = fn(*args, **kwargs)
            with self._lock:
                self.failures = 0
                self.state = CircuitState.CLOSED
            return result
        except Exception as e:
            with self._lock:
                self.failures += 1
                self.last_failure_time = time.time()
                if self.failures >= self.failure_threshold:
                    self.state = CircuitState.OPEN
            raise

llm_breaker = LLMCircuitBreaker(failure_threshold=5, recovery_timeout=60)

Graceful Degradation

Design every AI-powered feature so it has a non-AI fallback. The fallback does not need to be equivalent - it needs to keep the feature functional.

pythonCopy
def get_product_summary(product_id: str) -> dict:
    try:
        # Primary: LLM-generated dynamic summary
        return llm_breaker.call(generate_summary, product_id)
    except RuntimeError:
        # Degraded: static summary from DB
        product = db.get_product(product_id)
        return {
            "summary": product.description[:200],
            "source": "static",
            "degraded": True,
        }
    except Exception as e:
        log.error(f"Summary generation failed for {product_id}: {e}")
        return {
            "summary": "Summary temporarily unavailable.",
            "source": "error",
            "degraded": True,
        }

The degraded: True flag lets the frontend show a visual indicator and lets you track degradation rates in your metrics.

Health Checks

Your load balancer needs to know whether your service is healthy. An LLM-powered service is only healthy if it can reach the LLM API.

pythonCopy
from fastapi import FastAPI, status
from fastapi.responses import JSONResponse
import asyncio

app = FastAPI()

@app.get("/health")
async def health_check():
    checks = {}
    # Check LLM API reachability with a minimal call
    try:
        await asyncio.wait_for(ping_llm_api(), timeout=5.0)
        checks["llm_api"] = "ok"
    except asyncio.TimeoutError:
        checks["llm_api"] = "timeout"
    except Exception as e:
        checks["llm_api"] = f"error: {type(e).__name__}"

    checks["database"] = "ok" if db.ping() else "error"
    checks["vector_store"] = "ok" if vector_store.ping() else "error"

    is_healthy = all(v == "ok" for v in checks.values())
    return JSONResponse(
        content={"status": "healthy" if is_healthy else "degraded", "checks": checks},
        status_code=status.HTTP_200_OK if is_healthy else status.HTTP_503_SERVICE_UNAVAILABLE,
    )

What to Do When the Model Is Unavailable

Have a runbook. When the LLM API goes down:

1. Circuit breaker flips to OPEN: downstream calls fail fast, no queuing
2. Degraded paths activate: features that can fall back do so
3. Features that cannot fall back show a user-facing "temporarily unavailable" message
4. Alert fires: your on-call engineer sees a dashboard tile go red
5. Monitor recovery: circuit breaker moves to HALF_OPEN every 60 seconds, testing

The runbook should specify which features are critical (show degraded) versus non-critical (disable entirely). This is a product decision, not an engineering one - make it ahead of time.

Common Mistakes

No circuit breaker. Without one, a slow LLM API causes thread pool exhaustion in your application server. Every request waits 60 seconds before timing out.

Health check that lies. A health check that returns 200 OK even when the LLM API is unreachable means your load balancer routes traffic to broken instances.

Retry storms. Retrying immediately on a 429 (rate limit) with all clients simultaneously makes the problem worse. Use exponential backoff with jitter.

Treating degraded mode as temporary. LLM APIs have non-trivial downtime windows. Your degraded fallbacks must be robust enough to run for hours, not seconds.

Where to Go Next

See also: [llm-app-engineering-production], [tool-using-agents-guardrails], [ai-system-design-product-constraints]