Context pruning is a bet on the future

When an agent’s context window fills up, the obvious move is to prune. Drop the oldest, biggest tool results and carry on. It looks like good hygiene. It’s a bet, and on some workloads a losing one.

The trade-offs get much clearer if you hold it this way. The window is a cache, pruning is eviction, and eviction is one of the few problems in computing where we can prove you can’t do it optimally without seeing the future. Once you hold it that way, “just trim the old stuff” stops being obvious housekeeping and turns into a wager about what you’ll need again. This is the management half of the problem that context engineering only sets up. Composing the window is one job. Keeping it useful as it grows is another.

Cost is the wrong axis

The instinct to prune is mostly about money, and money turns out to be the wrong axis. Anthropic’s prompt cache charges a cache read at roughly 10% of the input price, so a warm window is already a tenth of full freight and carrying it is cheap. Shrinking a warm window aggressively trades that cheap read for a full-price rewrite of everything after the cut. Pruning to save money can cost you money. Inference is a real marginal cost on every turn, and the cache is what flattens it, and pruning fights the cache.

If cost isn’t the axis, two other things are. The first is attention quality. A window stuffed with stale output buries the signal the model needs, and a model reasoning over its own noise gets worse in ways that never show up on an invoice. The second is the ceiling: every model has a hard token limit and a long session walks towards it whatever the economics say. Those are the honest reasons to prune, and naming them changes what a good policy looks like, because a policy tuned for cost and a policy tuned for attention aren’t the same policy.

Every edit to the prefix has to earn its rewrite

The cache has one rule. The prefix you send has to match the prefix already in the cache byte for byte, up to the point you mark. Appending a new message keeps the prefix intact and stays cheap. Editing history doesn’t. Change a byte and everything after it is invalidated and re-billed at full price.

A prune is an edit. It deletes bytes in the middle of the cached region, which forces a rewrite of the whole suffix. Sometimes that’s worth paying for. The thing to hold onto is that it’s never free, so every pruning policy spends real money each time it fires, and one that fires on a bad guess spends it for nothing.

Why blind pruning thrashes

The standard policy is age plus size: evict anything older than N turns and bigger than M tokens. That rule bets that old and big means never needed again. When the bet is wrong, and a large result gets referenced just outside the retention window, you get a loop. You evict it, the model asks for it, you re-inflate it, it ages back out, you evict it again. Each cycle pays for a rewrite and leaves dead stubs behind.

Tuning doesn’t fix it, and there’s a proof of why. Belady’s result from the page-replacement literature says the provably optimal eviction policy requires knowing the future sequence of accesses, which nobody has. Every real policy approximates the future from the past, and age-and-size is a crude approximation. On the wrong workload it’s worse than doing nothing.

So a pruning policy has two honest forms. Drop only what’s provably dead, or learn from what gets recalled. Everything in between is guessing.

The two safe moves, and the one to refuse

The first honest form is provably-dead hygiene. Some results are dead with certainty rather than by guess. A file read whose contents got overwritten by a later write in the same session, say, or the same query run twice with identical parameters. Those bytes can never be correct or useful again, so dropping them can’t thrash, because there’s nothing to recall. It’s the only kind of prune that’s unconditionally safe and the one to reach for first.

The second is offload rather than delete. Instead of removing a large ageing result, swap it in the live window for a one-line stub pointing at the verbatim original in your own storage. If the model needs it, it pulls the exact bytes back from your store rather than re-running the tool against the external source. One rule makes this work: once a result has been recalled, exempt it from eviction. A recall is proof the result is in the working set, so the policy learns from its own mistakes instead of re-evicting the same bytes every few turns. That single exemption is the difference between an offload tier that helps and one that thrashes.

The workload decides

None of this has a universal answer. The right policy follows from your workload and your cost structure, and those vary a lot.

A coding agent lives on read-edit-read churn. It reads a file, edits it, reads it again, and that first read is now genuinely dead. This workload manufactures provably-dead results by the dozen, so hygiene alone reclaims a lot of window at no risk, and an offload tier earns its keep on top of that.

A data-analysis agent looks nothing like that. A schema it pulled an hour ago, a statistical result from turn three, those stay live and get referenced again when the agent writes its conclusion. The age-and-size rule that’s roughly safe on the coding agent misfires here, because on this workload “old” doesn’t predict “dead” at all. The results age without dying, and a policy that confuses the two throws away the working set.

When the problem is simply that the session got long, pruning is often the wrong instrument entirely. Compaction, meaning summarising the transcript into working memory and dropping the raw turns, holds onto the hard-won conclusions better than any rule guessing which raw bytes to discard. It’s lossy on purpose, and the skill is in choosing what to lose. A summary that keeps the chatter and drops the conclusions is worse than no summary.

Measure before you wire

The order that keeps you honest is to instrument first, read the distributions, then decide which policy earns its complexity. How long do sessions actually get? How often does a pruned result get recalled? How much of your window is provably dead on a real trace rather than a hypothetical one? Those numbers tell you whether you need hygiene, offload, compaction or nothing at all, and they routinely say less than you assumed.

The instinct is to reach for the cleverest policy. Use the one your workload justifies, which is usually the simplest and sometimes none. You’re betting on data you don’t have yet, so make the smallest bet that works and go and check the numbers.