An unlimited context window (without an unlimited model)

Every agent that runs long enough hits the same wall — usually twice.

The first wall is obvious: the model has a finite context window, and once the rendered input exceeds it, the call hard-fails. The second is quieter and more expensive. Most frontier models have a pricing cliff — below some token count your input is billed at the base rate, and above it the whole request (sometimes the whole session) is billed at a multiple. The model I run in production has a window of about a million tokens, but the cliff sits at 272K, where input jumps to 2× and output to 1.5×. So the number that actually governs my architecture isn’t a million. It’s 272K.

A long, tool-heavy conversation blows through both. Tool outputs are fat, reasoning accumulates, and every turn carries the last turn’s baggage forward. The job is to keep the model’s working context bounded and under the cliff — while never losing a word of history.

The one idea that makes the rest work

The most important decision is to stop treating “the conversation” as a single thing. It’s two.

There’s what the user sees — the chat transcript, one bubble per message, the thing your UI renders. And there’s what the model sees — the working set the LLM actually reasons over: every message, every reasoning step, every tool call and its result. These do not have to be the same object, and the moment you let them diverge, the problem becomes tractable.

I keep them in two separate stores. The display store is append-only and never touched — the user sees every message, in full, forever. The working store is the one I’m allowed to shrink. Compaction, masking, every trick below — they only ever touch the working store. The user’s history is never at risk.

Once you have that split, the job is narrow: keep the working store small and under the cliff, without losing anything. I do it with four cooperating layers.

Mask, don’t summarize

The first and cheapest layer runs on every model call. Before I send the history to the model, I walk it and replace any tool output over ~500 characters with a short, deterministic placeholder:

[output omitted from history: 14,213 chars from search]

No LLM. No summary. Just local string truncation. The insight is that the agent almost never needs to re-read a fat tool result on the next step — it needs to remember that the tool ran, roughly how big the result was, and whether it errored. That fits in a sentence. Replaying the full 14K payload on every subsequent call is pure waste.

Two details matter more than they look. First, the threshold: below ~500 characters the placeholder is longer than the thing it replaces, so masking would be net-negative — I only mask when it pays. Second, and this one costs people hours: I mask the output text but never remove the item, and I keep its identifiers intact. Modern tool-calling APIs reject a history where a tool result appears without its matching call, or where a reasoning step is orphaned. If you “compact” by deleting items, you’ll spend an afternoon debugging 400s. Mask in place; keep every item paired. (Reasoning items stay verbatim even though they’re bulky — you’ll need them intact for the compaction step below.)

Compact between turns — and put the threshold at the money

Masking keeps each call lean, but across a long conversation the working set still grows. The primary control is compaction between turns.

After each turn I read the real input-token count from the last model call — not an estimate, the number the API actually charged me for. If it crossed my threshold, I fold the accumulated history into a single opaque summary item plus a tail of the most recent items kept verbatim, and swap that in as the new working set.

The interesting part is where the threshold goes. It goes at the money, not at capacity. I set it to 200K — comfortably under the 272K cliff — so that after every compaction, the next message starts back in the cheap tier. It has nothing to do with running out of room; I still have hundreds of thousands of tokens to spare. It’s a cost boundary, not a capacity one.

Two subtleties from production. I read the token count for the top-level agent only, filtered by name — sub-agents share the same cost tracker, and a token-heavy sub-task shouldn’t trigger a conversation-level compaction. And on the very first turn, before there’s a real token count to read, I fall back to a cheap character estimate and lean conservative: better to under-compact than to compact a conversation that’s barely started.

Two backstops

Between-turn compaction handles the common case. Two more layers catch the tail.

A single turn can balloon on its own — the agent loops, calling tools several times, and each call carries a bigger input than the last. That can cross the cliff inside one turn, before the between-turn layer ever runs. So I set a second compaction threshold much higher — around 900K, ~86% of the window — that fires mid-stream if a single turn runs away. It’s disruptive (the model loses fidelity on the tool outputs it’s actively using), so it’s deliberately a rare backstop, not the primary control. And notice why there’s nothing between the cliff and this guard: a threshold there would be the worst of both worlds — it would still cross the cliff and disrupt the turn. So the thresholds are intentionally either “under the cliff” or “near the ceiling,” never in the dead zone.

The last resort is an error handler. If a call still fails with a context-window error despite everything above, I force one compaction and retry the call exactly once. If that retry fails, I surface a clean “this conversation grew too large — resend your last message” instead of a stack trace.

Never delete — one flag, two views

Compaction is lossy. The summary it produces is opaque; you can’t reconstruct the original turns from it. Which means I can’t actually delete the originals — I’d be destroying history, and the whole promise was that the user never loses anything.

So compaction doesn’t delete. It flips a boolean. Every item in the working store carries an is_summarized flag. Compaction sets it to true on the live items and inserts the compacted window as fresh rows. Now one table serves two readers: the model reads WHERE is_summarized = false and sees only the bounded, post-compaction window; an export or audit reads every row and sees the complete, lossless trace. Same data, two views, one flag.

Walk it forward through two compaction rounds and the shape is clear — the model’s view keeps shrinking back down; the archive only ever grows:

Stage	What the model sees is_summarized = false	Archive, lossless is_summarized = true
round 0 · normal turns	12345678	— empty —
after compaction #1	C1r7r8	12345678
… more turns …	C1r7r8910	1 … 8
after compaction #2	C2r10	1 … 8C1r7r8910

Why this is “unlimited”

Put it together and you’ve decoupled two things that looked like one. The conversation grows without bound — every message preserved in the display store, every raw item preserved in the archive. The model’s view stays bounded — masked per call, compacted under the cliff between turns, guarded mid-flight, caught on overflow.

It’s not an infinite model. It’s an unbounded conversation over a bounded, self-pruning working set. The user can talk to the agent effectively forever, the model never sees more than it can afford, and nothing is ever lost.

What transfers

Strip away the specifics and four principles carry to any long-running agent:

1. Separate what the user sees from what the model sees. They’re different data with different lifecycles. Conflating them is the root mistake.
2. The cheapest compaction is a placeholder, not a summary. Summarizing costs a model call and throws away the “what ran.” Mask fat observations in place and keep the memory of them.
3. Make compaction lossless at the storage layer even when it’s lossy at the model layer. Soft-delete, don’t delete. One flag buys you a bounded model view and a complete audit trail from the same table.
4. Put your thresholds at the economic boundaries, not at round numbers. The token count that matters is the one that changes your bill, and the one that changes whether the call succeeds at all. Compact just below the first; guard just below the second.

The context window is a budget, not a memory. Treat it that way and “unlimited” stops being a paradox.