Musical Chairs, Measured: Does a Prompt Cache Survive Its Own Promise?

An experiment testing The Token Subsidy Is Ending. The design and prediction below were published before any data was collected; the results section reports how that prediction fared.

The Claim I Need to Check

In part 3 of the plateau series I wrote that KV caching doesn’t rescue agent economics, because caches live in GPU memory shared across thousands of concurrent users:

Let your agent idle for five minutes while a parallel subtask finishes, and the provider evicts your cache to seat someone else. When you resume: cache miss, full re-prefill, full price. It’s musical chairs with GPU memory.

That sentence is load-bearing for the post’s economic argument, and I sourced it from industry commentary rather than from anything I measured. It is also one of the few claims in the series that a person with an API key can test in an afternoon. So I’m testing it.

The Thing That Changed Nine Days Ago

I started designing this experiment expecting a frustrating result — that “the cache died early” would collapse into an argument about what was ever promised, since providers describe cache lifetimes in deliberately soft language.

Then GPT-5.6 went generally available on July 9th, and three things changed at once. Cache writes, which had always been free, began billing at 1.25× the uncached input rate. Developers got explicit cache breakpoints instead of automatic placement. And the lifetime language hardened from “generally 5 to 10 minutes of inactivity, up to a maximum of one hour” into “at least 30 minutes.”

That last one is a floor, not a hedge. A floor is falsifiable.

It also means there are now two different products to measure, and the contrast between them is the experiment. Models before GPT-5.6 have exactly the best-effort, pressure-evicted cache that post 006 describes — measuring their eviction curve tests my claim directly. GPT-5.6 and later have a guarantee — measuring theirs tests whether the guarantee is real. Plotting both on the same axes shows whether the industry actually moved.

And notice what the pricing change is, on its own, before any data comes in. A subsidy was withdrawn and simultaneously replaced with a priced, contractual product, in a single release. Free cache writes were the Installation-phase giveaway; 1.25× writes with a guaranteed TTL is what the same capability looks like once it has to pay for itself. That is the Turning Point happening on a pricing page, and it is arguably stronger evidence for this series’ thesis than anything the experiment itself will produce.

The Prediction

Committing to this before running anything, so the result can embarrass me:

On the legacy arm, I predict hit rate meaningfully below 100% at gaps of three to five minutes — the 60–90% range rather than the ceiling. That’s what musical chairs implies.

On the GPT-5.6 arm, I predict the 30-minute floor mostly holds — above 95% for any gap under 1,800 seconds. I expect the guarantee to be real, because it is now something OpenAI charges for.

If both predictions land, the post was right about the old world and needs a paragraph saying that world ended. If the legacy arm shows near-perfect retention, the musical-chairs mechanism was overstated and I’ll correct it. If the GPT-5.6 floor doesn’t hold, that is the most interesting outcome available here and the one worth writing up carefully.

Either way, the paradox at the center of part 3 survives — cheap tokens don’t help when token count per task grows faster than price falls. That argument never depended on eviction. Only the mechanism does.

Design

One trial is a write, a controlled wait, and a read: send a large prompt with a cache breakpoint, confirm it wrote, wait exactly g seconds, send the byte-identical prompt again, and record whether any of it came back from cache. OpenAI reports both sides directly — cache_write_tokens on the way in, cached_tokens on the way back — so each trial’s outcome is observed rather than inferred from latency or billing lag.

The gap ladder runs from zero to one hour, deliberately dense around 1,800 seconds where the GPT-5.6 floor sits, and covering the 300–900 second range where legacy models are documented to start evicting. Six replicates per cell, on gpt-5.6-luna against gpt-4o.

Three choices carry most of the validity.

Every trial is independent. A unique nonce leads each prompt, so each trial writes its own cache entry. Trials can’t contaminate each other, and they can all run concurrently — which is what keeps a matrix full of hour-long waits down to about two hours of wall clock instead of days.

Routing is held constant within a trial. OpenAI’s prompt_cache_key influences which machine serves a request, so both requests in a trial share one key while different trials use different keys. Without this, a “miss” could just mean the read landed on a different node than the write, and I’d be measuring routing while claiming to measure eviction.

The prompt is byte-stable. Filler comes from a seeded pseudo-random generator over a fixed word list. No timestamps, no UUIDs, no dictionary ordering — any of which would silently invalidate the prefix and produce a beautiful, meaningless curve of 100% misses.

The Rate Limit Ate the Experiment (Twice)

I had to rebuild this experiment five times, and the reason is worth reporting rather than hiding, because it turned out to be a smaller version of the thing the post is about.

My account is capped at 30,000 tokens per minute. The original design — a 9,600-token prefix, ten replicates, gaps out to two hours — needed 5.8 million tokens, which is 193 minutes of pure throughput before a single second of deliberate idling. It did not fit, and it failed in two opposite directions before I understood the shape of the constraint.

Sent too fast, the write legs exhausted the minute-budget in seconds and 159 of 198 trials died with a rate-limit error before their timers ever started. Those are recorded as errors and excluded — a write that never landed has measured nothing, and quietly scoring it as a cache miss would have manufactured precisely the eviction result I predicted. That is the failure mode I would least like to publish.

Then, having added a token budget, I created the opposite problem: the write legs monopolized it and the reads queued behind them. A trial with a nominal zero-second gap observed 931 seconds. Nothing was corrupted; the experiment had simply stopped measuring the variable it named.

The fix isn’t fewer requests or smaller ones. It’s that writes have to arrive more slowly than the budget can absorb them, so there is always headroom for a read to fire on time. Deliberately under-subscribing brought observed drift down to under a second. The instrumentation now checks that ratio before spending anything and refuses to start quietly when it’s wrong.

What survived is a smaller experiment: a 1,483-token prefix, six replicates instead of ten, and nothing past one hour. The prefix reduction is harmless, since the measurement is only whether the cached-token count comes back above zero. The replicate reduction is not — the confidence intervals below are wider than I wanted, and a small difference between the two arms should be read as unresolved rather than as a result.

And the thing I keep turning over: a rate limit is rationing. I could not run this experiment at the size I designed, on an account in good standing, because inference capacity is scarce enough to be metered by the minute. That is not a bug I worked around; it is the subsidy argument of part 3 showing up in the methodology section.

What Would Make This Wrong

This is one API key, one organization, one week. Eviction is driven by fleet-wide memory pressure from other tenants, which means I’m measuring one tenant’s weather, not a provider’s climate. Runs are labeled by time of day and kept separable rather than pooled.

The design measures survival from a single write, while a real agent reads repeatedly and each read extends the entry’s life. That’s the harder case, so eviction showing up here is meaningful — but production agents probably do better than these numbers, and I should not present them as what an agent experiences.

There’s a specific trap on the new arm worth naming: explicit cache breakpoints can be accepted without actually writing cache. A trial whose write leg never wrote has measured nothing, and silently counting it as a miss would manufacture precisely the result I predicted. The runner checks every write leg and the analysis excludes and reports the failures. If that count is large, the experiment is void rather than positive.

And an irony I can’t fully engineer away: running many concurrent trials adds a little memory pressure of exactly the kind I’m trying to detect. The fifteen-second stagger helps — the original design held every trial’s cache entry simultaneously, which was worse.

Results

120 trials, both arms, zero errors, observed gaps within three seconds of nominal throughout.

The floor is also a ceiling

GPT-5.6 hit 100% on every trial at every gap up to 1,680 seconds — thirty for thirty — and 0% on every trial from 1,800 seconds onward. Not a decay curve. A step function, landing exactly on the documented boundary.

The documentation says cached prefixes remain eligible “for at least 30 minutes” and that OpenAI “may retain it longer.” On this evidence it does not retain it longer, ever. The minimum is enforced precisely and the entry is gone the moment the obligation ends. Whatever is happening here, it is not memory pressure — pressure produces variance, and there is none.

My prediction was wrong about the old world

I predicted the legacy arm would show hit rates of 60–90% at three to five minutes, because that is what musical chairs implies. It didn’t. GPT-4o held 27 of 30 trials at gaps up to 28 minutes, and its documented window is “generally 5 to 10 minutes of inactivity.”

The single 83% cells are tempting to read as early eviction, but they can’t bear that weight: the 1,500-second cell came back 6 of 6, which cannot happen if eviction is progressive. What the data shows is a roughly flat ~90% out to 28 minutes, then a genuine decline — 67%, 50%, 33% — reaching zero by 45 minutes.

The claim in part 3 is not supported. Caches did not get evicted after five minutes of idling to seat another tenant. They survived roughly half an hour on the old best-effort product and about the same on the new guaranteed one. I’ll correct that post rather than leave the sentence standing.

And the 90%-versus-100% difference between arms, tempting as it is, does not survive contact with the sample size: Fisher’s exact test gives p = 0.24 at n=30 per arm. I cannot distinguish a genuine 10% miss rate from noise, and won’t pretend otherwise.

What the guarantee actually bought

Here is the part I did not anticipate. Both caches die at roughly the same age. GPT-4o’s median survival lands somewhere between 1,800 and 1,920 seconds; GPT-5.6’s is exactly 1,800. The guarantee did not buy more time. It bought the elimination of variance.

That reframes what the 1.25× cache-write charge is for. You are not paying for a longer-lived cache; you are paying to know precisely when it dies, so you can build a system that plans around the number instead of hoping. The best-effort cache was probably cheaper to operate and, on this sample, roughly as durable — it just couldn’t be designed against.

Which is the Turning Point in miniature, and not the version I expected to find. The thing that gets sold when the subsidy ends isn’t more capability. It’s a number you can put in a contract.

Limitations

One account, one evening, one region — a quiet Saturday night is plausibly the friendliest fleet conditions available, and eviction driven by other tenants’ pressure would show up worst at peak hours. Six replicates per cell means every interval here is wide; the step function is stark enough to survive that, the legacy decay curve’s shape is not. Each trial writes once and reads once, while a real agent reads repeatedly and refreshes the entry, so these are lower bounds on what production workloads see. And the experiment could only be run at this reduced size because of the rate limit described above.

The obvious follow-up is a peak-hours repeat at higher replication. If the GPT-5.6 step function holds at 6pm on a Tuesday, that’s a much stronger claim than one quiet evening can support.

Sources

• Prompt caching, OpenAI API documentation — breakpoints, TTL, usage fields, write pricing.
• Prompt Caching in the API, OpenAI.
• Model guidance, OpenAI API documentation — the GPT-5.6 variants.