Prompt Caching Was On. It Was Barely Working.

Savant conversations are agentic: the model calls tools (web research, image generation, document creation) in a loop, and every iteration of that loop is a fresh API call that re-sends the entire conversation so far. When I finally instrumented a realistic conversation end to end, the headline number was hard to ignore:

219,785 input tokens billed at the full rate to produce about 34,000 tokens of output.

Seven user turns, 17 API calls, more than six input tokens billed for every output token received. In dollar terms, input was 57% of the bill. I believed prompt caching was on, and it was, in the sense that the code set the right fields. It was barely doing anything, and finding out why is most of this story. The rest is about the optimizations I tried afterward that measured out to nothing, which I'm including because that part usually gets left out of posts like this.

Measuring before optimizing

Guessing at token costs is easy and usually wrong, so the first step was a measurement harness: an integration test that drives a real conversation through the full backend. Real model, real MCP tool servers, real artifacts.

The scenario is deliberately mundane: an 8th-grade science teacher preparing class notes on the water cycle. Two research calls, two image generations, compiling a document, editing in an answer key, and a final summary. The conversation archiver writes every raw API request and the provider's own per-call token counts to a log, so the numbers below come from the API's own metering, not estimates. (They have one blind spot, owned up to under the table.)

The baseline, on Claude Haiku 4.5:

The anatomy of each request is system prompt, then tool definitions, then message history. The system-plus-tools prefix is large (roughly 9 to 10K tokens, with 43 MCP tools declared) but stable. The bill tracked the growing message history, especially the large tool results (research output, generated document content) that sit in history forever and get re-sent, at full price, on every single one of those 17 calls.

One note for anyone who reproduces the arithmetic: the table's figures are the API's input_tokens, which excludes cache reads and writes. The prefix flowed through those cache fields on every call, and the accounting dropped them entirely; that blindness is bug #1 below. So read the table as a floor. The real baseline bill was higher than it shows, which makes the savings later in this post, if anything, conservative.

The fix: making the cache real

Anthropic's prompt caching charges cache writes at 1.25 times the normal input rate and cache reads at 0.1 times. (Those are the 5-minute-lifetime figures; the 1-hour tier writes at 2 times.) If most of your input is served from cache reads, input cost mostly disappears. The mechanism is exact prefix matching on request content, from the front: change anything early, say the order of your tool definitions, and everything after it misses. That detail is the whole story below.

Making it effective took four changes.

1. Account for cache tokens. The client parsed the cache_creation_input_tokens and cache_read_input_tokens fields from API responses, then dropped them before they reached usage accounting. A broken cache and a working cache looked identical in my books. First change: carry cache writes and reads through the whole accounting pipeline. Everything else in this post was only visible after this existed.

2. Cache the message history, not just the prefix. Only system and tools carried a cache breakpoint. I added a rolling breakpoint on the last content block of the last message, so the whole conversation-so-far becomes cacheable, not just the static prefix.

3. Freeze the prompt per turn. History trimming was recomputed on every tool-loop iteration against a still-growing history, which meant the "same" conversation could serialize slightly differently between iterations. Slightly different means cache miss. Now the history prefix is computed once per user turn and stays byte-stable across the whole tool loop.

4. Deterministic tool ordering. This was the one that mattered most, and it hid in the most innocent-looking code. The tool list sent to the API was assembled by iterating over a Go map of MCP servers. Go randomizes map iteration order by design, so the tools block arrived in a different order on every turn. Since the cacheable prefix is system, then tools, then messages, a reshuffled tools block invalidated the cache for everything after it, including the entire message history. The fix, lightly simplified from Savant's MCP collection code:

ids := make([]string, 0, len(c.servers))
for id := range c.servers {
    ids = append(ids, id)
}
sort.Strings(ids)

for _, id := range ids {
    tools = append(tools, c.servers[id].Tools()...)
}

Measured as the drop in input cost attributable to caching, this fix took that number from roughly 11% to roughly 60%. A sort and a loop.

There was a fifth change with the same shape: stop trimming history below budget. The old policy aggressively collapsed all but the most recent tool results into placeholder markers. Collapsing a result on a later turn than it was first sent rewrites the middle of the history, which busts the cross-turn cache all over again. The new policy only trims when the conversation actually approaches the model's context window. Below that, full tool results stay in history: re-sending them costs 0.1 times as cache reads, the prefix stays append-only, and nothing is ever lost.

Results

Comparing full 7-turn runs, price-weighted (input at 1.0, cache writes at 1.25, cache reads at 0.1):

Two of these numbers are solid, and one is softer. The solid ones: with all five changes, about 86% of input tokens were served as cache reads, and input cost fell about 74% measured within-run, meaning the cached run against what those same requests would have cost uncached. (Deriving an input number from the table instead compares two different runs under two different trim policies, which is the noisy way.) These are deterministic properties of what got sent and how it was billed.

The softer one is the −21% total, and I'd rather flag it myself than have you find it. It is one run against one run, and agentic runs are noisy: on the identical scenario, output volume swung between 24K and 43K tokens purely from model nondeterminism, a swing worth roughly as much as the entire saving. Add the baseline caveat from the table above (a floor, because cache-field tokens were invisible to the accounting) and the honest claim is this: the input side of the bill collapsed, measurably and reproducibly; the total moved in the right direction in the runs I have. And below the context budget the whole thing is lossless: no summarization, no dropped context. The trim still exists as a last resort near the budget, and when it fires it busts the cache once; no measured workload came anywhere near it.

Yes, this is "just" prompt caching

If you build on these APIs you may be composing the comment already: the pricing is documented, the documentation tells you to keep the prefix stable, and this is a bug report dressed up as an optimization story.

Mostly agreed. Nothing here is a novel technique. The point is narrower and, I think, more useful: a silently broken cache is invisible. The fields were set, requests succeeded, responses streamed, every test passed. The only symptom was money, and the accounting that would have surfaced it was the piece that was missing. If your tool definitions are assembled from a hash map in any language with unstable iteration order, you may be paying full price right now with no way to know. Audit your request builder for byte-stability. It's a cheaper afternoon than mine was.

What the harness doesn't capture

Two caveats about real usage, both worse than the benchmark, both worth stating plainly.

First, the cache expires after five minutes. The harness fires turns back to back, well inside that window. A person who reads the reply and thinks for ten minutes comes back to a cold cache, and the next call re-writes the whole prefix at 1.25 times instead of reading it at 0.1. Cross-turn savings depend on conversation rhythm.

Second, mid-conversation tool changes. Savant lets the model request additional tool servers mid-conversation, and approving one changes the tools block. By the logic above, that busts the cache for everything after it, once per approval.

What survives both effects is the saving inside a single turn's tool loop, where calls come seconds apart with a stable tool set. In the measured runs that loop is where most of the calls are: 10 of the 17 were within-turn iterations.

What didn't work

I tried three more things. The honest result for all three was no.

Shrinking tool payloads. The plan was to reduce the huge tool results sitting in history: incremental document edits instead of regenerating whole documents, deduplicating repeated research results. The arithmetic was against it from the start. Once re-sent history is billed at 0.1 times, the most a payload diet can recover is that remaining sliver. The measured run agreed: $0.348 with the payload work, against $0.308 without it, a gap inside the noise band. The bill had flipped to output (43% of cost before this work, 62% after), and output does not care how small your history is.

The investigation still paid off, just not in dollars. What looked like a flaky document tool was an output ceiling: the model's document write was truncated at max_tokens, the document was never created, and the model then invented a document ID and kept going. Raising the ceiling fixed a real correctness bug. It just didn't make anything cheaper.

Switching models. The theory that a smarter model takes fewer tool-loop iterations and comes out cheaper was testable, so I tested it. On the same two turns, in one run, Claude Sonnet 4.6 cost $0.724 where Haiku 4.5 cost $0.165, ran more iterations rather than fewer, and was slow enough to hit the test timeout. Output tokens cost about five times input tokens on every Claude model in the lineup, so a bigger model mostly scales the bill. I'll hedge this one myself: it's one run on one workload, and a model that actually followed the incremental-edit instructions (Haiku ignored them) might win on cost per completed task. On what I measured, Haiku stays.

Deduplication. I had built a tidy optimization that replaced byte-identical repeated tool results with a reference marker. Post-work review found two things. It had never fired: across every measured run, no tool result was ever repeated byte for byte. And it was quietly dangerous: it mutated the in-memory conversation, and the persistence layer then saved the lossy marker to disk with no way to recover the original content. An optimization that never triggers and can silently destroy data has negative value. I deleted it.

What I took away

1. Anything nondeterministic in your prompt prefix is a cache-buster, and the failure is silent. Map iteration order, timestamps, re-serialization quirks, mid-history rewrites. The request succeeds either way; you just pay full price.
2. Instrument cache tokens before optimizing anything. The jump from 11% to 60% was only visible because the accounting existed by then.
3. Once input is cached, output dominates, and output costs about five times input on every Claude model. The next lever is getting the model to do less redundant work per task, not shrinking prompts further. I've deliberately paused there until volume justifies it.
4. Delete clever code that never fires. Dead optimizations aren't neutral; they're risk with no return.

I went into this believing I had a solved caching setup and an unsolved optimization problem. It was the other way around. If you spot a hole in the arithmetic, or a cache-buster I missed, write to me at support@usesavant.com. Corrections are cheaper than another instrumented run.

Savant is a desktop AI workspace: one app for Claude, GPT, Gemini, Grok, and local models, with built-in tools for research, documents, images, email, and more. It's currently in an invite-only rollout. You can request access here.