Prompt Caching Isn't a Setting. It's a Prompt Architecture Decision.

How prefix caching actually works, why most teams get the ordering wrong, and the exact restructuring that turns a cache-miss prompt into a cache-hit one.

I once reviewed a system where every single call to the model was a cache miss. Not most calls. Every call. The team had read the docs, added the caching parameter where the provider asked for it, and moved on assuming it was handled. It wasn't. Somewhere in their prompt construction, a request ID had been added to the top of the payload "for logging," ahead of a system prompt and tool schema that were otherwise completely static. That one field, sitting a few hundred characters before the part that never changed, was enough to break the match on every call.

Nobody had touched the caching config. The bug was in the ordering.

That's the part people miss about prompt caching. It gets talked about like a checkbox, something you enable in the API call and forget. It isn't. Whether you get the discount and the latency improvement is decided entirely by how you construct the prompt, not by which provider you're using or what flag you passed.

What's actually being cached

Most providers can reuse computation across requests when a prompt's prefix, the leading portion of the input, matches a prefix they've processed recently. Instead of running the full attention computation over that content again, the provider serves the already-computed state for that segment. That's cheaper, often a significant discount on those tokens, and faster, because time-to-first-token drops when a chunk of the input doesn't need to be processed fresh.

The condition for that reuse is strict. The prefix has to match exactly: same tokens, same order, same whitespace, up to the point where content starts to differ. The instant something dynamic shows up before the end of what would otherwise be a stable block, the match breaks from that point forward. Everything after it in that request gets treated as new, even the parts that are genuinely unchanged.

So cache-hit rate isn't a runtime property you tune after the fact. It's a design decision you make once, when you decide what order things go into the prompt.

The one rule that matters: static first, variable last

Split any prompt you send to a model into two zones.

Zone A is static. System instructions, tool and function schemas, few-shot examples, reference documents that don't change within a session, a saved persona or config profile. This is the part that should look identical across hundreds or thousands of calls.

Zone B is variable by design. The current user message, a specific record being looked up, anything that's supposed to be different on every call.

The rule is simple to state and easy to violate in practice: Zone A goes first, completely, before a single token of Zone B appears. Never interleave them. A timestamp in a header, a session token added "just for tracing," a reordered field in a serialized dict, any of these sitting early in the prompt invalidates the cache for every static token that follows.

Here's what that actually looks like when you get it wrong versus right. This is a simplified version of the ticket-triage prompt I mentioned above, run locally so you can see exactly where the prefix breaks.

import hashlib, datetime

SYSTEM_INSTRUCTIONS = (
    "You are a support ticket triage assistant. Classify each ticket into "
    "one of: billing, technical, account, other. Respond with the category "
    "and a one-line reason."
)
TOOL_SCHEMA = (
    '{"name":"lookup_account","parameters":{"account_id":{"type":"string"}}}'
)
FEW_SHOT = (
    "Example - Ticket: 'I was charged twice this month.' -> billing, duplicate charge.\n"
    "Example - Ticket: 'App crashes on login.' -> technical, crash on auth.\n"
)

def build_prompt_BEFORE(user_message: str) -> str:
    # request_id and timestamp added at the top "for logging"
    header = f"request_id={hashlib.sha1(user_message.encode()).hexdigest()[:8]} "
    header += f"ts={datetime.datetime.now(datetime.timezone.utc).isoformat()}\n"
    return header + SYSTEM_INSTRUCTIONS + "\n" + TOOL_SCHEMA + "\n" + FEW_SHOT + user_message

def build_prompt_AFTER(user_message: str) -> str:
    # static block first, byte-identical every call; dynamic content last
    static_block = SYSTEM_INSTRUCTIONS + "\n" + TOOL_SCHEMA + "\n" + FEW_SHOT
    return static_block + user_message

def shared_prefix_len(a: str, b: str) -> int:
    n = 0
    for x, y in zip(a, b):
        if x != y:
            break
        n += 1
    return n

msg1 = "My subscription renewed but I wanted to cancel it."
msg2 = "The mobile app keeps freezing when I try to upload a photo."

before_1, before_2 = build_prompt_BEFORE(msg1), build_prompt_BEFORE(msg2)
after_1, after_2 = build_prompt_AFTER(msg1), build_prompt_AFTER(msg2)

print("BEFORE - shared prefix between call 1 and call 2:", shared_prefix_len(before_1, before_2))
print("Static block length that should have been cacheable:", len(SYSTEM_INSTRUCTIONS + TOOL_SCHEMA + FEW_SHOT))
print("AFTER - shared prefix between call 1 and call 2:", shared_prefix_len(after_1, after_2))

Output:

BEFORE - shared prefix between call 1 and call 2: 11
Static block length that should have been cacheable: 385
AFTER - shared prefix between call 1 and call 2: 387

Eleven characters. That's how much of the "before" prompt actually matched between two consecutive calls, because the timestamp and request ID sit before everything else and change every time. A 385-character static block that should be reused on every single request never gets the chance. Move the same dynamic content to the end and the shared prefix jumps to 387 characters, essentially the entire static block, on every call.

Nothing about the model changed between those two versions. Only the order.

Why this matters more in multi-turn and agentic systems

The same logic extends across turns. In a multi-turn conversation, the growing history is itself a prefix. As long as earlier turns aren't rewritten or reordered, each new turn's prompt (all prior turns plus the new message) shares an ever-longer stable prefix with the previous call, and a provider that supports prefix caching picks that up automatically.

This is where I see agentic systems lose the benefit without realizing it. A common pattern is to summarize or compress earlier turns once the context gets long, to save tokens. That's a reasonable thing to do for context management, but it rewrites the middle of the prefix, and rewriting anything in a prefix that a later call depends on breaks the cache chain from that point forward for every subsequent call. If you need to trim context, trim from the middle or drop the oldest turns wholesale. Both only cost you a few early cache checkpoints. Rewriting content that sits in the middle of an active conversation breaks the match at every point you touched.

The same thing applies to retrieved context in RAG pipelines. If the same documents are likely to come up again in a session, put them ahead of the user's question and keep the formatting deterministic, same field order, same whitespace, across calls that retrieve the same documents.

The bug that hides in plain sight: non-deterministic serialization

The timestamp-at-the-top mistake is the obvious one. The harder one to catch is when your static content isn't actually static; it just looks static in the source code.

The most common version of this: a tool schema or config object gets serialized fresh on every call instead of stored as a literal string. If the dict is built from a database row, a merged config, or anything where field order isn't guaranteed, two calls that are semantically identical can produce byte-different JSON.

import json

def build_schema_v1():
    return {
        "name": "get_order_status",
        "parameters": {
            "order_id": {"type": "string"},
            "region": {"type": "string"},
            "include_history": {"type": "boolean"},
        },
        "description": "Fetch the current status of a customer order.",
    }

def build_schema_v2():
    # same fields, same values, different insertion order
    return {
        "description": "Fetch the current status of a customer order.",
        "name": "get_order_status",
        "parameters": {
            "include_history": {"type": "boolean"},
            "order_id": {"type": "string"},
            "region": {"type": "string"},
        },
    }

print(json.dumps(build_schema_v1()) == json.dumps(build_schema_v2()))
# False

print(json.dumps(build_schema_v1(), sort_keys=True) == json.dumps(build_schema_v2(), sort_keys=True))
# True

Both dicts describe the same tool, same fields, same types, same description. Withoutsort_keys=True, they serialize to different strings, and a caching layer sees two different prefixes. This is the kind of bug that survives code review, because it "looks" static. Nobody reads a json.dumps(schema) line and thinks to ask whether the dict's field order is guaranteed. It's usually not, unless you pin it explicitly.

The fix isn't complicated once you know to look for it: assemble static blocks once, store the exact string, and reuse that literal on every call instead of re-templating or re-serializing it. Pin serialization settings. Sort keys explicitly. Watch for anything that iterates over an unordered collection to build content you expect to be identical.

How this differs by provider

Caching isn't implemented the same way everywhere, and assuming one provider's behavior applies to another is a common source of wasted engineering time.

Anthropic requires an explicit cache_control breakpoint marking where the cacheable prefix ends. You place it after your static block, system instructions, tools, and few-shot examples, and it needs periodic reuse to stay warm within its TTL. OpenAI and DeepSeek apply caching automatically once a prompt clears a minimum length; no explicit opt-in, but the ordering discipline still applies exactly the same way. Google's Gemini has both an automatic mode and an explicit context-caching mode for large stable content, where you create a cache object once and reference it by ID instead of resending the raw content every call.

The mechanism differs. The rule that makes it work doesn't. Static first, byte-identical, variable last. Check current docs before assuming a specific discount or TTL for whichever provider you're on; those numbers change, but the architecture principle is stable across all of them.

The audit I run on an existing codebase

When I'm brought in to look at why a system's cache-hit rate is low, or why nobody's sure if caching is even working, this is roughly the sequence:

First, find every place a prompt gets constructed. Every SDK call, every place a string gets built or a template gets rendered into a system, prompt, or messages parameter.

Second, classify every component of that prompt as static, semi-static (a user's saved profile, a session's retrieved documents), or dynamic (today's actual query, a live timestamp used in the task itself).

Third, check the ordering. Is anything dynamic or semi-static sitting before the end of the static block? This is where most of the real findings show up, timestamps near the top, config objects re-serialized without guaranteed key order, template variables that can produce different whitespace between calls even when the underlying data hasn't changed.

Fourth, confirm the provider-specific piece is actually in place. A cache_control breakpoint at the right boundary for Anthropic. Explicit context caching for large stable content on Gemini. No leftover explicit-caching code on a provider that doesn't need it.

Fifth, and this is the step teams skip most often: check whether the cache-hit and cache-miss token counts are actually being logged anywhere. If they're not, you have no way to confirm any fix worked. You're optimizing blind.

Proving it's actually working

Don't take "the code looks static" as confirmation. Log the provider's cache usage fields on every response and track the hit rate over time, not once in a test run. A hit rate that looks great in testing and degrades in production almost always means something environment-specific is leaking into the static block, a hostname, a process ID, a piece of config that differs between environments but not within one.

Set up a dashboard metric or alert for cache-hit ratio if the system runs at any real volume. A sudden drop is a regression signal the same way a latency spike would be, someone added a field to the static block without checking whether it's deterministic.

And when you first make this change, run a before-and-after comparison on comparable traffic. Confirm the token cost and latency actually moved, not just that the code compiles and the breakpoint is present. I've seen teams ship the cache_control marker, assume it's done, and never check whether the hit rate actually improved.

The takeaway

Prompt caching gets sold as a cost-optimization feature you flip on. In practice it's closer to a constraint on how you're allowed to write your prompt construction code. Static content first, assembled once, stored as a literal, byte-identical across calls. Dynamic content last, and only there. Everything about determinism, dict ordering, serialization, and template rendering matters more here than it does in most other parts of an LLM system, because a single unstable byte early in the prompt costs you every token that comes after it.

Get the ordering right once, and it stays right. Get it wrong, and you'll keep paying full price for tokens you were told would be discounted, and nobody will notice until someone actually logs the hit rate.

This connects to a problem I wrote about earlier: once your prompts are versioned and evaluated like code, caching is the next place that discipline pays off, because a cache-breaking change is exactly the kind of "small text diff, big cost impact" bug that versioning alone won't catch. Prompts Are Code. Treat Them Like It.

Found this useful? I do consultations on small, medium, and large-scale enterprise solutions. Contact me or send a note to discuss in detail.