LLM Application Engineering

LLM Engineering vs. LLM Research

Most public discussion of LLMs focuses on what they can do. LLM application engineering focuses on building reliable systems on top of what they can do - which is a fundamentally different problem.

An LLM that can summarize a document 95% of the time is not a reliable summarization service. A reliable service requires: detecting and handling the 5% failure cases, a consistent interface, latency within acceptable bounds, cost management, and a way to know when quality degrades. This post is about building that system.

Prompt Engineering: More Engineering Than Art

Prompt engineering is often presented as a creative skill. It is actually a structured engineering discipline.

The Anatomy of a Useful Prompt

[System context]
You are a technical support assistant for a software company.
Your responses should be concise, technically precise, and focused on actionable steps.

[Task definition with format]
The user will provide an error message. Your task is to:
1. Identify the most likely cause
2. Provide 2-3 specific remediation steps
3. Note if additional diagnostic information is needed

Format your response as:
LIKELY CAUSE: [one sentence]
STEPS: [numbered list]
NEEDS MORE INFO: [yes/no and what]

[Input]
User error: {error_message}

This structure is not arbitrary. Each section exists for a reason:

• System context sets the model's role and tone
• Task definition removes ambiguity about what "good" looks like
• Format specification makes the output parseable
• Input is clearly separated from instructions

Few-Shot Prompting

Providing examples in the prompt dramatically improves output consistency:

pythonCopy
prompt = """
Extract the key entities from the following text and return them as JSON.

Example:
Text: "John Smith from Acme Corp signed a contract with Beta Inc on March 15, 2024"
Output: {"people": ["John Smith"], "organizations": ["Acme Corp", "Beta Inc"], "dates": ["March 15, 2024"]}

Text: {input_text}
Output:
"""

Few-shot examples set the output format implicitly, handle edge cases the model might not handle by default, and align the model's behavior with your specific requirements.

Chain-of-Thought for Complex Reasoning

For tasks requiring multi-step reasoning, asking the model to "think step by step" before producing the final answer significantly improves accuracy:

pythonCopy
system_prompt = """
When solving this problem, first reason through the steps explicitly,
then provide your final answer in a clearly labeled section.

Format:
REASONING: [your step-by-step analysis]
ANSWER: [your final answer]
"""

This works because the model's generation of the reasoning steps conditions the final answer token prediction on better intermediate context.

Structured Output: Making LLMs Parse-Safe

Applications almost always need structured output (JSON, specific formats) rather than free text. Three approaches:

1. Prompt-Based JSON

Ask for JSON in the prompt and parse it. Works 80–90% of the time. Fails silently when the model produces invalid JSON, adds explanation text outside the JSON block, or changes key names.

Always wrap in error handling:

pythonCopy
import json

def parse_llm_json(response: str) -> dict | None:
    import re
    # 1. Try direct parse (handles JSON-only responses and tool-call outputs)
    try:
        return json.loads(response.strip())
    except json.JSONDecodeError:
        pass
    # 2. Extract from a ```json ... ``` code block
    fence = re.search(r'```(?:json)?\s*([\s\S]*?)\s*```', response)
    if fence:
        try:
            return json.loads(fence.group(1))
        except json.JSONDecodeError:
            pass
    # 3. Last resort: find the outermost { ... } span (handles JSON in prose)
    start, end = response.find('{'), response.rfind(')}}')
    if start != -1 and end > start:
        try:
            return json.loads(response[start:end + 1])
        except json.JSONDecodeError:
            pass
    return None

2. Tool/Function Calling

Modern LLM APIs (Anthropic, OpenAI) support structured tool definitions. The model is forced to call the tool with the specified schema:

pythonCopy
tools = [{
    "name": "extract_entities",
    "description": "Extract people, organizations, and dates from text",
    "input_schema": {
        "type": "object",
        "properties": {
            "people": {"type": "array", "items": {"type": "string"}},
            "organizations": {"type": "array", "items": {"type": "string"}},
            "dates": {"type": "array", "items": {"type": "string"}},
        },
        "required": ["people", "organizations", "dates"]
    }
}]

This is the most reliable structured output approach - the model cannot deviate from the schema.

3. Constrained Decoding (Outlines, Guidance)

Libraries like outlines constrain token sampling to only produce tokens consistent with a JSON schema. Zero parsing failures, but requires running your own model (not compatible with API calls).

Evaluation: The Part Everyone Skips

Deploying an LLM application without evaluation is deploying blind. You will not know if it is working, improving, or regressing.

Types of Evaluation

Reference-based: Compare model output to a gold standard (human-written ideal response). Metrics: ROUGE (recall-based), BLEU (precision-based), BERTScore (semantic similarity).

Model-based (LLM-as-judge): Use a capable LLM to rate outputs on specified dimensions (correctness, helpfulness, tone, groundedness). Requires calibration and testing the judge itself.

Human evaluation: The most expensive and most reliable. Use it to calibrate automatic metrics.

Task-specific: For extraction, measure precision/recall on entity extraction. For summarization, measure factual consistency against the source. For code generation, measure whether generated code passes a test suite.

A Minimal Eval Framework

pythonCopy
from anthropic import Anthropic
import json

client = Anthropic()

def evaluate_response(question: str, response: str, ideal: str) -> dict:
    """Use Claude to evaluate a response against an ideal answer."""
    eval_prompt = f"""
    Evaluate this AI response on a scale of 1-5 for each dimension.

    Question: {question}
    Ideal Answer: {ideal}
    Actual Response: {response}

    Score each dimension (1=poor, 5=excellent):
    - Correctness: Is the factual content accurate?
    - Completeness: Does it cover the key points?
    - Conciseness: Is it appropriately brief?

    Return JSON: {{"correctness": N, "completeness": N, "conciseness": N, "rationale": "..."}}
    """
    result = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=256,
        messages=[{"role": "user", "content": eval_prompt}]
    )
    return json.loads(result.content[0].text)

Building an Eval Dataset

Your eval dataset should contain:

• Hard cases that represent failure modes you care about
• Representative cases from normal usage
• Edge cases: very long inputs, empty inputs, adversarial inputs
• Cases where the correct answer is "I don't know" or "I can't help with that"

Aim for 50–200 examples for early-stage evals. Grow it as you learn more about failure modes.

Reliability Engineering for LLM Applications

LLMs are probabilistic. They occasionally produce wrong, inconsistent, or nonsensical outputs. Your application architecture must assume this and handle it.

Retry With Exponential Backoff

pythonCopy
import time
import anthropic

def call_with_retry(client, max_retries=3, **kwargs):
    for attempt in range(max_retries):
        try:
            return client.messages.create(**kwargs)
        except anthropic.RateLimitError:
            wait = 2 ** attempt
            time.sleep(wait)
        except anthropic.APIError as e:
            if e.status_code >= 500:
                time.sleep(2 ** attempt)
            else:
                raise
    raise RuntimeError(f"Failed after {max_retries} retries")

Output Validation

Define what a valid output looks like and reject anything that does not meet the criteria:

pythonCopy
def is_valid_extraction(output: dict) -> bool:
    required_keys = {"people", "organizations", "dates"}
    if not required_keys.issubset(output.keys()):
        return False
    if not all(isinstance(v, list) for v in output.values()):
        return False
    return True

Fallback Strategies

For every LLM call, define what happens when the model fails or produces invalid output:

• Retry with a modified prompt
• Fall back to a simpler rule-based approach
• Return a default safe response
• Route to human review

Cost and Latency Management

LLM API costs are proportional to token count. Latency increases with output length.

Count tokens before sending: Use the tokenizer to estimate cost. Reject inputs that would exceed budget.

Cache repeated calls: Identical inputs produce deterministic (or near-deterministic) outputs at temperature=0. Cache the results.

Use prompt caching: Anthropic's prompt caching feature caches the static portions of your prompt (system prompt, examples) so repeated calls with the same prefix cost less.

Choose the right model: Use a smaller, faster model for classification/routing tasks. Reserve the large model for complex generation tasks.

Common Mistakes and Bad Instincts

Not evaluating before deploying. "It works in my testing" is not evaluation. Build a dataset of representative cases and measure systematically.

Treating prompts as stable. Model provider updates change model behavior. Pin model versions and re-evaluate when upgrading.

Not handling rate limits. Production LLM applications hit rate limits. Build retry logic before you need it.

Trusting model-graded evals blindly. LLM-as-judge has biases - preferring longer responses, agreeing with the framing in the question, and favoring responses from the same model family. Calibrate against human ratings.

Prompting for JSON and not handling failures. Even with explicit JSON prompting, models occasionally produce invalid JSON. Always validate and handle failures.

Where to Go Next

LLM Application Engineering is Module 21 in the College Student path and Module 15 in the SWE path. Both modules require building a production-grade LLM application with evaluation harness, structured output, retry logic, and a cost analysis. The RAG module follows, covering retrieval-augmented approaches for grounding LLM outputs in specific knowledge bases.

The LLM Application Stack

An LLM feature is a system, not a prompt. A production stack usually includes:

• Input collection and validation
• Prompt construction
• Retrieval or context assembly
• Model call
• Structured output parsing
• Validation and policy checks
• Fallback handling
• Logging and evaluation
• Cost and latency monitoring

Any weak layer can make the whole feature unreliable.

Prompt Interfaces

A prompt should be treated like an API contract. It defines:

• Role and task
• Context
• Constraints
• Output format
• Examples
• Refusal or uncertainty behavior
• Quality rubric

Version prompts. Review prompt changes like code changes. A one-line prompt edit can change product behavior more than a backend refactor.

Structured Output

Free-form text is hard to integrate. For many applications, ask for structured output:

jsonCopy
{
  "category": "billing",
  "urgency": "high",
  "summary": "Customer was charged twice",
  "needs_human": true
}

Then validate it. If required fields are missing, retry with a repair prompt or route to a fallback. Never let unchecked model output directly drive irreversible actions.

Evaluation Harness

LLM evaluation needs examples, criteria, and repeatability. Build a small golden dataset:

• Common requests
• Edge cases
• Adversarial inputs
• Ambiguous inputs
• Known failure cases
• Domain-specific examples

For each example, store expected qualities rather than only one exact answer. A good answer may be phrased many ways. Grade factuality, completeness, tone, citation quality, schema validity, and safety separately.

Cost and Latency Design

LLM systems can become expensive quickly. Track:

• Input tokens
• Output tokens
• Model choice
• Cache hit rate
• Retrieval latency
• Retry rate
• Cost per successful task

Use cheaper models for classification, routing, rewriting, and extraction when they perform well enough. Reserve stronger models for tasks where quality gains justify cost.

Failure Modes

Common LLM application failures include:

• Hallucinated facts
• Ignored instructions
• Invalid JSON
• Prompt injection
• Retrieval misses
• Overlong context
• Hidden bias in generated language
• Cost spikes from retries or long outputs
• User trust damage from confident wrong answers

The engineering answer is not "make the prompt better." It is layered defense: validation, grounding, evaluation, monitoring, and graceful degradation.

A Production Readiness Review

Before launch, answer:

1. What should the model do?
2. What should it refuse or escalate?
3. How do we measure quality?
4. What is the fallback when quality is low?
5. What user data is sent to the provider?
6. What logs are retained?
7. What cost per task is acceptable?
8. Who owns incidents?

If these questions feel heavy, the feature is probably still a demo.

Closing Thought

The practical standard is not memorization. It is whether you can use the idea to make a better engineering decision, explain that decision to someone else, and notice when reality disagrees with your assumptions.

What to Do Next

Turn this article into a small artifact. Write a checklist, run a tiny experiment, sketch the architecture, or review an old project using the concepts above. Learning becomes durable when it changes what you inspect before you trust a result.

For a portfolio or team setting, save that artifact next to the code or decision memo. Future reviewers should be able to see not only what you built, but how you reasoned about correctness, risk, and tradeoffs.

Team Review Prompts

Before treating this work as complete, ask a teammate to review it using three prompts:

1. What assumption is most likely to break in production?
2. What evidence would make you trust the result?
3. What simpler approach should we compare against?

These questions are deliberately plain. They work because they force the discussion away from tool enthusiasm and back toward judgment, evidence, and maintainability.

Final Rule

Reliable LLM applications are built from small controlled pieces. Use the model for language and reasoning where it helps. Use deterministic code for validation, permissions, accounting, and irreversible actions. The boundary between the two is the heart of LLM engineering.

Caching and Idempotency

LLM applications should cache when the same request or same context appears repeatedly. Caching reduces cost and latency, but it must respect privacy and freshness. Do not cache sensitive personal data casually. Do not serve stale answers for fast-changing policies.

Idempotency matters for tool use. If a model retries an action, the system should not accidentally send two emails, create two tickets, or charge twice. Tool calls need request IDs, deduplication, and clear side-effect boundaries.

Observability Fields

Log enough to debug without storing unnecessary sensitive data:

• Prompt version
• Model name and version
• Token counts
• Latency
• Retrieval chunk IDs
• Parser result
• Validation errors
• Fallback path
• User feedback
• Cost estimate

This lets engineers answer: did quality fail because of retrieval, prompting, model behavior, parsing, or product assumptions?

Security and Prompt Injection

LLM applications that use tools or retrieval must treat user input as untrusted. A prompt injection attack tries to make the model ignore developer instructions, reveal hidden context, call tools incorrectly, or exfiltrate data.

Defenses include:

• Keep secrets out of prompts.
• Limit tools by role and context.
• Validate tool arguments with code.
• Separate retrieved content from system instructions.
• Use allowlists for actions.
• Require human approval for destructive operations.
• Log suspicious attempts for review.

The model should never be the only security boundary.

Human-in-the-Loop Design

Some tasks should not be fully automated. A useful LLM system can draft, classify, summarize, or recommend while leaving approval to a person. This is especially important for legal, medical, financial, hiring, safety, and customer-impacting workflows.

Design the handoff clearly. Show evidence, uncertainty, and recommended next actions. A human reviewer should not have to reverse-engineer why the model said something.

Versioning the Whole Behavior

Version prompts, retrieval indexes, model choices, parsers, eval datasets, and policy rules together. When behavior changes, you need to know which component changed. Otherwise every regression becomes a guessing game.