Structured Output From LLMs: Techniques and Tradeoffs

Why Structured Output Is Hard

LLMs generate text. Your application needs JSON. The problem: text generation is probabilistic, and the model will occasionally produce valid-sounding but unparseable text - a trailing comma, an explanation before the JSON block, an inconsistent key name.

The four techniques below represent increasing levels of reliability at increasing cost.

Technique 1: Prompt-Based JSON (80-90% Reliability)

Ask the model to output JSON in the prompt. Parse with a regex fallback.

pythonCopy
import json
import re

def extract_json(text: str) -> dict | None:
    # Try direct parse
    try:
        return json.loads(text)
    except json.JSONDecodeError:
        pass

    # Try to extract JSON from surrounding text
    patterns = [
        r'```json\s*([\s\S]*?)\s*```',  # Fenced code block
        r'\{[\s\S]*\}',                   # Any JSON object
    ]
    for pattern in patterns:
        match = re.search(pattern, text)
        if match:
            try:
                return json.loads(match.group(1) if '```' in pattern else match.group())
            except json.JSONDecodeError:
                continue
    return None

def extract_entities(text: str) -> dict | None:
    prompt = f"""Extract entities from this text. Return ONLY valid JSON, no other text.
Format: {{"people": [], "organizations": [], "locations": []}}

Text: {text}"""

    response = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=512,
        messages=[{"role": "user", "content": prompt}]
    )
    return extract_json(response.content[0].text)

When to use: Prototyping, low-stakes applications, simple schemas. Failure rate: ~5-15%. Must have fallback logic.

Technique 2: Tool/Function Calling (95-99% Reliability)

Modern LLM APIs support function calling: the model is forced to call a specified tool with a JSON schema. The model cannot deviate from the schema.

pythonCopy
tools = [{
    "name": "extract_entities",
    "description": "Extract named entities from text",
    "input_schema": {
        "type": "object",
        "properties": {
            "people": {
                "type": "array",
                "items": {"type": "string"},
                "description": "Names of people mentioned"
            },
            "organizations": {
                "type": "array",
                "items": {"type": "string"},
                "description": "Names of organizations mentioned"
            },
            "locations": {
                "type": "array",
                "items": {"type": "string"},
                "description": "Place names mentioned"
            }
        },
        "required": ["people", "organizations", "locations"]
    }
}]

response = client.messages.create(
    model="claude-haiku-4-5-20251001",
    max_tokens=512,
    tools=tools,
    tool_choice={"type": "tool", "name": "extract_entities"},  # Force tool use
    messages=[{"role": "user", "content": f"Extract entities: {text}"}]
)

# Response is guaranteed to be a tool call
tool_use = next(b for b in response.content if b.type == "tool_use")
entities = tool_use.input  # Already a parsed dict matching the schema

When to use: Any production extraction or classification pipeline. Failure rate: ~1-3% (schema validation failures, not JSON parse failures).

Technique 3: Pydantic Validation (Adds Type Safety)

After parsing JSON or tool output, validate against a Pydantic model to catch semantic errors:

pythonCopy
from pydantic import BaseModel, field_validator
from typing import Optional

class EntityExtraction(BaseModel):
    people: list[str]
    organizations: list[str]
    locations: list[str]

    @field_validator('people', 'organizations', 'locations')
    @classmethod
    def no_empty_strings(cls, v):
        return [item for item in v if item.strip()]

def extract_and_validate(text: str) -> EntityExtraction | None:
    raw = extract_via_tool_calling(text)
    try:
        return EntityExtraction(**raw)
    except Exception as e:
        # Log validation error, decide whether to retry or return None
        return None

Technique 4: Constrained Decoding (Near-100% Reliability)

Libraries like outlines constrain the token sampling to only produce tokens that are valid given a JSON schema. The model literally cannot generate invalid JSON.

pythonCopy
import outlines

model = outlines.models.transformers("mistralai/Mistral-7B-Instruct-v0.2")

schema = {
    "type": "object",
    "properties": {
        "sentiment": {"type": "string", "enum": ["positive", "negative", "neutral"]},
        "confidence": {"type": "number", "minimum": 0, "maximum": 1}
    }
}

generator = outlines.generate.json(model, schema)
result = generator("Classify sentiment: 'I loved this product!'")
# result is guaranteed to match the schema

When to use: When you need guaranteed valid output and are running your own model (not compatible with API calls to Anthropic/OpenAI).

Choosing a Technique

Scenario	Technique
Prototype / development	Prompt-based with regex fallback
Production extraction pipeline	Tool calling
Complex nested schemas	Tool calling + Pydantic validation
Self-hosted model, zero tolerance for format errors	Constrained decoding

What to Practice Next

• Use the OpenAI response_format={"type": "json_object"} parameter (or an equivalent structured-output API) for a task you care about - then deliberately craft a prompt that violates your schema and observe how the model handles it.
• Implement Pydantic validation on top of a raw LLM JSON response: define a schema, parse the output, catch ValidationError, and write a retry loop that re-prompts the model with the error message included.
• Compare two structured output approaches - constrained decoding (e.g., Outlines or Instructor) vs. plain prompting with a JSON schema in the system prompt - on 20 test cases; measure parse success rate and output quality for each.