The Boring Infrastructure Detail Nobody Talks About Enough
Everyone wants to talk about model architecture, context windows, agentic loops, whatever the hype cycle is pushing this month. Almost nobody wants to talk about prompt caching. Which is funny, because if you're running any kind of production LLM workload, caching is probably saving you more money right now than your last three prompt engineering "breakthroughs" combined.
I say this as someone who ignored it for way too long. I had a RAG pipeline last year burning through tokens like it was free money, and it took me embarrassingly long to realize I was re-sending the same 4,000-token system prompt on every single call without structuring it so the cache could actually catch it. Once I fixed the ordering, the bill dropped by something like 60%. Not a typo.
What Prompt Caching Actually Does
Most of the major providers — Anthropic, OpenAI, Google — now offer some flavor of prompt caching. The idea is simple. If you send the model the same chunk of text repeatedly (a system prompt, a big document, a set of tool definitions), the provider can cache the internal computation for that chunk instead of reprocessing it from scratch every time.
The catch, and this is where sequence comes in, is that caching works on a prefix basis. The model checks whether the beginning of your prompt matches something it's seen before. If it matches, great, you get a cache hit and pay a fraction of the cost for that portion. If even one token near the start is different, the whole thing invalidates and you're paying full price again.
This is not intuitive if you're used to thinking about prompts as bags of information rather than ordered sequences.
Why Order Is the Whole Game
Here's the part people get wrong constantly. They'll put a static system prompt at the top, which is good, but then they'll shove a timestamp, a session ID, or some dynamic user metadata right after it — still near the beginning. That single dynamic field breaks the cache for everything that follows it.
The fix is almost embarrassingly simple: structure your prompt so everything static comes first, and everything dynamic comes last.
- System instructions, tool schemas, few-shot examples — top of the prompt, unchanging
- Retrieved documents or long context — next, ideally stable across a session
- User's actual question, conversation turn, or dynamic variables — dead last
Do this and the cacheable prefix stays intact call after call. Get lazy about it and you're essentially paying for a cache you're not using, which is worse than not having one, because now you're paying the caching overhead fee on top of full-price tokens on some providers.
How Much Does It Actually Matter?
A lot, if your prompts are long and repetitive. Barely at all if they're short and mostly unique.
If you're running a customer support bot with a 3,000-token system prompt and short user messages, caching that system prompt properly can cut your input token costs dramatically since that prefix is the majority of every request. Anthropic's cached input pricing, for example, runs at roughly a tenth of the standard input rate once cached. OpenAI's caching (automatic, no special formatting required, though order still matters for the prefix matching) works similarly.
If your prompts are dynamic from token one — think a search engine where the query is the first thing in the prompt — caching does almost nothing for you. There's no stable prefix to hit.
Latency improves too, which people underweight. A cache hit means less computation, which means faster time-to-first-token. If you're building anything conversational, that matters more than the invoice.
The Mistakes I See Constantly
Putting a random UUID or current date at the top of a system prompt "for logging purposes." Just don't. Log it separately.
Reordering few-shot examples between calls because someone thought randomizing them would reduce bias. It might, marginally. It also nukes your cache every time.
Treating cache TTLs as infinite. Most providers cache for five minutes, sometimes up to an hour with extended options. If your traffic pattern has gaps longer than that, you're not benefiting nearly as much as you think you are — check your actual hit rate, don't assume it.
What to Actually Do About It
Restructure your prompts static-to-dynamic, every time, no exceptions. Measure your cache hit rate directly if your provider exposes it (Anthropic does, in the usage response). If you're below 70% on a workload with a large repeated prefix, something in your ordering is off and it's worth the twenty minutes to find it.
It's not a glamorous fix. Nobody's writing a conference talk titled "I Moved My Timestamp to the End of the Prompt." But the cost curve doesn't care about glamour, and this one's cheap to fix and expensive to ignore.