LLM Product Engineering

Most engineers' first contact with LLMs is a demo: paste a prompt, get a response, it works. Production is different. Production means the model is called thousands of times per day by users with unpredictable inputs, your system has latency SLAs, and every failure is a user experience problem or a cost spike. The gap between a working demo and a reliable product is almost entirely in the engineering layer around the model, not the model itself.

This module covers the engineering patterns that teams actually use to ship LLM features that are reliable, observable, and cost-controlled.

Structured Output: Making LLMs Reliable

LLMs produce text. Your application needs data. The failure mode of free-text prompting is non-deterministic - sometimes the model adds a preamble, sometimes it changes field names, sometimes it explains instead of answering. Structured output eliminates this class of failure.

pythonCopy
from openai import OpenAI
from pydantic import BaseModel, Field
from typing import Literal

client = OpenAI()

class SentimentResult(BaseModel):
    sentiment: Literal["positive", "negative", "neutral"]
    confidence: float = Field(ge=0.0, le=1.0)
    key_phrase: str = Field(description="The phrase most responsible for this sentiment")

def analyze_sentiment(text: str) -> SentimentResult:
    response = client.beta.chat.completions.parse(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Analyze the sentiment of the given text."},
            {"role": "user", "content": text},
        ],
        response_format=SentimentResult,
    )
    return response.choices[0].message.parsed

result = analyze_sentiment("The onboarding was smooth but the pricing page confused me.")
print(result)
# sentiment='neutral' confidence=0.71 key_phrase='pricing page confused me'

When the API does not natively support structured output (Claude, Mistral), use instructor-style prompting + Pydantic validation with retry on parse failure.

Retry Logic and Exponential Backoff

LLM APIs return rate limit errors (429) and server errors (5xx). These are transient. Always wrap API calls in retry logic:

pythonCopy
import time
import random
from functools import wraps

def retry_with_backoff(max_retries: int = 3, base_delay: float = 1.0):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(max_retries):
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    if attempt == max_retries - 1:
                        raise
                    delay = base_delay * (2 ** attempt) + random.uniform(0, 0.5)
                    print(f"Attempt {attempt + 1} failed: {e}. Retrying in {delay:.1f}s")
                    time.sleep(delay)
        return wrapper
    return decorator

@retry_with_backoff(max_retries=3)
def call_llm(prompt: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    )
    return response.choices[0].message.content

In production, use tenacity or a dedicated library rather than hand-rolling this. The principle: exponential backoff with jitter prevents thundering-herd when a rate limit hits concurrent requests.

Prompt Management

Hard-coding prompts in application code is the equivalent of hard-coding SQL queries as string literals. When prompts change, you want to:

• Version them alongside code
• A/B test variants
• Roll back to a known-good version

pythonCopy
# prompts.py - version prompts as code, not scattered strings
from dataclasses import dataclass

@dataclass
class PromptTemplate:
    system: str
    user_template: str
    model: str
    temperature: float = 0.0
    max_tokens: int = 1024

    def render_user(self, **kwargs) -> str:
        return self.user_template.format(**kwargs)

SENTIMENT_PROMPT = PromptTemplate(
    system="You are a precise sentiment analyzer. Return JSON only.",
    user_template="Analyze: {text}",
    model="gpt-4o-mini",
    temperature=0.0,
)

SUMMARY_PROMPT = PromptTemplate(
    system="Summarize in 3 bullet points. Be concise.",
    user_template="Document: {document}\n\nFocus on: {focus_area}",
    model="gpt-4o",
    temperature=0.1,
)

For teams with many prompts, use LangSmith, PromptLayer, or a simple Git-versioned YAML file. The key discipline is treating prompts as artifacts, not magic strings.

Cost Tracking

LLM costs are real and variable. At scale, a single feature can cost thousands per month. Track it:

pythonCopy
from dataclasses import dataclass, field
import threading

# Token pricing varies - always check your provider's current pricing page.
# Example rates (check provider docs for current values):
# gpt-4o-mini: ~$0.15/1M input, ~$0.60/1M output
# gpt-4o: ~$2.50/1M input, ~$10.00/1M output

@dataclass
class UsageTracker:
    _lock: threading.Lock = field(default_factory=threading.Lock)
    prompt_tokens: int = 0
    completion_tokens: int = 0
    requests: int = 0

    def record(self, usage):
        with self._lock:
            self.prompt_tokens += usage.prompt_tokens
            self.completion_tokens += usage.completion_tokens
            self.requests += 1

tracker = UsageTracker()

def tracked_call(prompt: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    )
    tracker.record(response.usage)
    return response.choices[0].message.content

# Report periodically to your metrics system
print(f"Requests: {tracker.requests}, "
      f"Prompt tokens: {tracker.prompt_tokens:,}, "
      f"Completion tokens: {tracker.completion_tokens:,}")

Emit these metrics to Datadog/Grafana. Set alerts at cost thresholds. Cost spikes often indicate prompt injection, runaway retries, or an accidental loop.

Caching

LLM calls are expensive and often deterministic for the same input. Cache aggressively:

pythonCopy
import hashlib
import json
import redis

r = redis.Redis(host="localhost", port=6379, decode_responses=True)

def cache_key(model: str, messages: list[dict]) -> str:
    payload = json.dumps({"model": model, "messages": messages}, sort_keys=True)
    return "llm:" + hashlib.sha256(payload.encode()).hexdigest()

def cached_llm_call(model: str, messages: list[dict], ttl: int = 3600) -> str:
    key = cache_key(model, messages)
    cached = r.get(key)
    if cached:
        return cached

    response = client.chat.completions.create(model=model, messages=messages)
    result = response.choices[0].message.content
    r.setex(key, ttl, result)
    return result

Cache TTL strategy: classification/extraction = 24h (deterministic); summarization = 1h; creative generation = do not cache. Track cache hit rate - a high miss rate suggests prompts are unnecessarily varied.

LLM Evaluation in CI

You need automated tests that catch regressions when you change a prompt or model:

pythonCopy
import pytest

@pytest.mark.parametrize("text,expected_sentiment", [
    ("I love this product!", "positive"),
    ("Terrible customer support", "negative"),
    ("The package arrived yesterday", "neutral"),
])
def test_sentiment_analysis(text, expected_sentiment):
    result = analyze_sentiment(text)
    assert result.sentiment == expected_sentiment, (
        f"Expected {expected_sentiment} for '{text}', got {result.sentiment}"
    )

# For non-deterministic cases: LLM-as-judge
def llm_judge(response: str, criteria: str) -> bool:
    verdict = client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": f"Does this response satisfy: {criteria}?\n\nResponse: {response}\n\nAnswer YES or NO only."
        }]
    )
    return verdict.choices[0].message.content.strip().upper() == "YES"

Run eval suites on every prompt change in CI. Use a deterministic model (temperature=0) for test cases where you can define exact expected output. Use LLM-as-judge for quality checks that are hard to enumerate.

Common Mistakes and Bad Instincts

Shipping without a fallback. If your LLM call fails or returns unparseable output, your application should degrade gracefully, not crash. Define a fallback: return a default response, surface a "try again" message, or route to a non-LLM code path.

Not logging inputs and outputs in production. Without logs, you cannot debug user-reported failures, audit outputs, or build an eval dataset. Log every request and response (or a sampled fraction) from day one.

Over-engineering the prompt. Teams write 50-line prompts with elaborate XML tags, chain-of-thought instructions, and conditional branches. Simpler prompts are easier to debug, cheaper, and often equally accurate. Write the shortest prompt that passes your eval suite.

Assuming the model always returns valid structured output. Even with strict JSON mode, models occasionally truncate mid-response at the max_token limit, producing invalid JSON. Always wrap parsing in a try/except.

Where to Go Next

• retrieval-systems-vector-databases-and-rag: extend LLM features with external knowledge via RAG
• agents-tools-and-workflow-graphs: compose multiple LLM calls and tools into autonomous workflows
• observability-drift-feedback-loops-and-llm-evals: build systematic evaluation and monitoring for production LLM systems