Prompt Caching: The New Optimization Target

For most LLM applications, the largest cost lever available today is not a cheaper model. It is prompt caching: reusing the computation for the part of your prompt that does not change between requests. Every major API offers it, serving frameworks do it automatically for self-hosted models, and most teams are leaving most of the benefit on the table because their prompts are laid out wrong.

How It Works

When a model processes a prompt, it computes internal state (the key-value cache) for every token, in order. That computation, called prefill, is a large fraction of the cost of a request with a long prompt. If two requests share an identical prefix, the state for that prefix is identical, so it can be computed once and reused.

Providers expose this as prompt caching: mark (or automatically detect) a prefix, and subsequent requests with the same prefix within a time window skip the prefill for it. Cached input tokens are billed at a fraction of the normal input price, and latency drops because the prefill is skipped.

The key word is prefix. The cache matches from the start of the prompt up to the first difference. Everything after the first changed token is computed fresh.

Layout for Caching

Order the prompt from most stable to least:

pythonCopy
messages = [
    {"role": "system", "content": [
        {"type": "text", "text": SYSTEM_PROMPT},                 # stable: cached
        {"type": "text", "text": TOOL_DEFINITIONS},              # stable per task: cached
        {"type": "text", "text": REFERENCE_DOCS,
         "cache_control": {"type": "ephemeral"}},                # explicit breakpoint on some APIs
    ]},
    {"role": "user", "content": user_profile_block},             # stable per user
    *conversation_history,                                       # grows; prefix still hits
    {"role": "user", "content": latest_message},                 # varies
]

On APIs with explicit cache breakpoints, place one after the last stable block. On APIs with automatic prefix caching, the same layout works without annotation.

For agents, the tool definitions are usually the largest stable block and the biggest win. For RAG features with a fixed corpus, the reference documents are. For multi-turn chat, the conversation history is a growing prefix, so each turn hits on everything before the newest message.

What Busts the Cache

All of these put a changing token before stable content, and each one silently costs full price:

• A timestamp or date in the system prompt.
• The user's name interpolated into the system prompt.
• A request ID, session ID, or trace ID at the top.
• Tool definitions in a different order per request (some frameworks do this; sort them).
• Retrieved documents placed before the tool definitions, so the tools are recomputed every request.
• Rebuilding the system prompt from a template with any per-request variable.
• Whitespace or formatting differences from a non-deterministic template.

The fix is always the same: move the variable content after the stable content, or out of the prompt entirely (does the model really need the request ID?).

Measure It

Cache hit rate belongs on the dashboard next to latency and cost per request. Providers return cached and uncached token counts in the usage metadata; log both.

• A hit rate near zero means the layout is wrong or something per-request is at the top.
• A hit rate that drops after a deploy means someone edited the stable part in a way that changed it (even a typo fix changes the prefix; that is expected once, not continuously).
• A hit rate that drops without a deploy means traffic patterns changed (more new users, more cold sessions) or the cache window expired between requests.

Also log cost per request with and without caching so the savings are visible. Teams that make this number visible tend to protect it.

Self-Hosted

Serving frameworks (vLLM, SGLang, and others) implement prefix caching automatically: identical prefixes across requests share KV cache blocks in GPU memory. The same layout rules apply, and the win is throughput rather than a billing discount, because prefill compute is what limits how many requests a GPU can serve.

What to Practice Next

Pull the last thousand requests from one feature and compute the cached fraction. If it is under 50%, find the first varying token in the prompt and move it. Re-measure. Then add cache hit rate and cost per request to the dashboard and set an alert on a drop. The module context-engineering-designing-what-the-model-sees treats caching as one constraint among several on context layout.