LLM App Engineering

Getting an LLM feature to work in a demo takes an afternoon. Getting it to work reliably in production - without blowing your cost budget, silently failing, or randomly generating garbage - takes a different mindset. Most tutorials skip the boring parts. This article is the boring parts.

Here is an opinionated engineering checklist for shipping LLM features, drawn from what consistently breaks in real systems.

Structured Output First

Unstructured LLM output is hard to consume. Use Pydantic + instructor (or the OpenAI response_format) to force structured responses from day one.

pythonCopy
from openai import OpenAI
from pydantic import BaseModel, Field
import instructor

client = instructor.from_openai(OpenAI())

class ExtractedEntities(BaseModel):
    company: str = Field(description="Company name mentioned")
    sentiment: str = Field(description="positive, negative, or neutral")
    confidence: float = Field(ge=0.0, le=1.0, description="Confidence score")

def extract_entities(text: str) -> ExtractedEntities:
    return client.chat.completions.create(
        model="gpt-4o-mini",
        response_model=ExtractedEntities,
        messages=[{"role": "user", "content": f"Extract entities from: {text}"}],
    )

Instructor handles retries for validation failures automatically. Use max_retries=2 for production; beyond that, surface the error rather than burning tokens.

Retry Logic with Exponential Backoff

Rate limits and transient errors are normal. Do not let them cascade.

pythonCopy
import time
import logging
from openai import RateLimitError, APITimeoutError

def llm_call_with_retry(prompt: str, model: str = "gpt-4o-mini",
                         max_retries: int = 3) -> str:
    for attempt in range(max_retries):
        try:
            response = client.chat.completions.create(
                model=model,
                messages=[{"role": "user", "content": prompt}],
                timeout=30,
            )
            return response.choices[0].message.content
        except RateLimitError:
            wait = 2 ** attempt + 0.5
            logging.warning(f"Rate limited, waiting {wait}s (attempt {attempt + 1})")
            time.sleep(wait)
        except APITimeoutError:
            logging.warning(f"Timeout on attempt {attempt + 1}")
            if attempt == max_retries - 1:
                raise
    raise RuntimeError("LLM call failed after retries")

Caching for Repeated Prompts

Semantic caching cuts costs significantly for read-heavy features. Use exact-match caching for identical prompts (e.g., classification of the same input text), and semantic caching for query patterns.

pythonCopy
import hashlib
import json
import redis

cache = redis.Redis(host="localhost", port=6379, decode_responses=True)
CACHE_TTL = 3600  # 1 hour

def cached_llm_call(prompt: str, model: str = "gpt-4o-mini") -> str:
    key = f"llm:{hashlib.sha256((model + prompt).encode()).hexdigest()}"
    cached = cache.get(key)
    if cached:
        return json.loads(cached)
    result = llm_call_with_retry(prompt, model)
    cache.setex(key, CACHE_TTL, json.dumps(result))
    return result

In classification workloads, exact-match caching typically hits 30–60% of requests. Do not skip it.

Cost Tracking Per Feature

Without per-feature cost attribution, you will not know which feature is burning money.

pythonCopy
from dataclasses import dataclass, field
from collections import defaultdict

@dataclass
class LLMUsageTracker:
    costs: dict = field(default_factory=lambda: defaultdict(float))
    PRICES = {
        "gpt-4o-mini": {"input": 0.00015, "output": 0.0006},   # per 1K tokens
        "gpt-4o":      {"input": 0.0025,  "output": 0.01},
    }

    def record(self, feature: str, model: str, usage):
        price = self.PRICES.get(model, {"input": 0.01, "output": 0.03})
        cost = (usage.prompt_tokens / 1000 * price["input"]
                + usage.completion_tokens / 1000 * price["output"])
        self.costs[feature] += cost

    def report(self):
        for feature, cost in sorted(self.costs.items(), key=lambda x: -x[1]):
            print(f"  {feature}: ${cost:.4f}")

tracker = LLMUsageTracker()

Tag every LLM call with the product feature name. Log to your observability stack weekly. You will find one feature consuming 70% of your budget.

Fallback Strategy

Design every LLM feature with a fallback for when the model is unavailable or over budget.

pythonCopy
def classify_intent(text: str) -> str:
    try:
        return cached_llm_call(f"Classify intent: {text}", model="gpt-4o-mini")
    except Exception as e:
        logging.error(f"LLM classification failed: {e}")
        # Fallback: keyword-based classifier
        if any(w in text.lower() for w in ["buy", "purchase", "price"]):
            return "commercial_intent"
        return "informational"

The fallback does not need to be good. It needs to keep the feature available.

Common Mistakes

Treating LLM calls like database queries. Database queries are deterministic and fast. LLM calls are slow, probabilistic, and expensive. Wrap them in retries, caches, and budgets from the start.

Skipping structured output. Free-form strings break downstream code. Use Pydantic models with instructor or the native response_format parameter. The schema is also better prompt engineering.

Not setting timeouts. The default OpenAI client has a 10-minute timeout. In a web request context, this hangs your server. Always set timeout=30 or less.

Ignoring the token budget. A prompt that works in development balloons in production when users paste 10,000-word documents. Measure p95 prompt length. Truncate or chunk inputs explicitly.

Where to Go Next

See also: [rag-foundations-retrieval-quality], [tool-using-agents-guardrails], [ai-system-design-reliability]