pi's backpack, and the helper that empties it

The backpack keeps getting heavier.

Every time pi reads a file or runs a command, the answer goes into its backpack (the context). It never takes anything out. Three tasks in a row on a small Rust project, and the backpack goes from 40k to 90k tokens.

Heavy backpack = pi forgets what matters, gets slower, and eventually pi has to squash everything (compaction). We want to delay that.

The helper asks a tiny brain: “will pi need this again?”

Right before pi thinks, the helper looks at old tool outputs and asks Jev (a small, fast model) how likely pi is to need each one again. If Jev says “nah”, the output is swapped for a sticky note. pi can always ask for the real thing back with recall.

What exactly do we send Jev? A tiny card for each old output, plus what pi is doing right now.

Jev never sees the backpack. It sees a summary card for each candidate — the first 300 characters, the last 100, how old it is, and the first thing pi said after reading it — plus the task (the user's message) and now (pi's latest words). One question per card, all in one request. This is a real request from the main run, rebuilt with the same code.

Notice: now names get.rs and output.rs as the files pi is about to edit → Jev keeps them (0.57, 0.62) even though they're 51 steps old. That's the task-awareness we wanted.

Also notice two weaknesses of the card: the nine parallel reads all carry the same agentReaction (it's the text after the whole step, not per file), and the bash outputs have none at all (pi went straight to the next tool call). Jev is judging those mostly from head and arg. And targets.rs at 0.62 is an over-keep — nothing in now points at it.

Here is a real backpack, with and without the helper.

Same three-task session, pruning on, snapshot at the end of task 2. Top row: everything pi had collected. Bottom row: what was actually sent — green slivers are sticky notes. Hover a block to see what it was.

The bottom row is what pi actually carries. The gap on the right is the room we made.

Did it help? Yes, a little — and more the longer pi works.

Task 1 is the same in every mode (nothing is old enough to prune yet). By task 3, the helper has been cleaning for a while, so pi starts the task with a lighter backpack.

Whole-session peak: . Task 3 was the only task that was also cheaper with pruning.

Did we ever throw away something pi needed? No.

If pi needed a pruned thing, it would call recall. Across every live run, it never did. And every file, function and struct name pi wrote into its reports actually exists in the code — it did not make things up to fill the gaps.

Test files read “for style”, rg/grep exploration output, and files pi re-read after editing them. All reasonable.

How sure is Jev, really? Very sure at the bottom, guessing in the middle.

We replayed 14 old sessions and checked, for every output Jev scored: did pi later use something that only that output contained? (That's our stand-in for “needed it”.) Blue = Jev's score. Orange = the ones pi did use later.

Below p = 0.15, 1 in 28 was used later. From 0.15 to 0.30, about 1 in 4 — the same as picking at random ( of everything gets used later). Jev is only reliable at the very bottom. That is why the cutoff is low, and why the savings are modest.

At 0.35 (the original default) Jev is exactly as wrong as random. At 0.25 it's half as wrong and still saves ~half of what's possible.

Rough #1: on a small project, Jev thinks everything might be needed.

In the old sessions (long, messy real work) Jev's average score was 0.28. In the live runs (“read this whole small repo, then fix things”) it was ~0.5. When every file is a possible target, Jev — correctly — refuses to bet. So at cutoff 0.25, almost nothing gets pruned.

Scores are stable and drift down slowly — then jump back up when task 3 (“trace how mansk get works”) makes github.rs and get.rs relevant again. Jev is consistent and task-aware; it just isn't confident on a small repo. The helper will matter more on long, wandering sessions than on tidy small ones.

Rough #2: every swap means repacking the whole backpack.

The model provider remembers (“caches”) the start of the backpack so pi doesn't pay full price for it each time. Swap one thing in the middle and everything after it must be re-sent and re-remembered — 1.25× the normal price for those tokens. Saving a small thing this way costs more than it saves.

Pruning one item at a time (every-call) repacks constantly. Our fix: collect prunes until they add up to ≥5% of the budget, then swap them all in one repack. Same lighter backpack, half the repacks.

Rough #3: we asked Jev the same question hundreds of times.

Once the backpack is nearly full, the helper checks at every step. The first version re-asked about every old output every step — and Jev kept saying “keep”. That's wasted calls and wasted latency.

Fixed partly (re-ask only every 3 steps once over budget), but the ratio is still ~14 questions per item. A back-off (3, 6, 12 steps…) would cut it further; the scores barely move between asks anyway.

Rough #4: a rule with no brain did much of the useful work.

“pi read main.rs, then edited main.rs, then read it again → the first read is dead.” No model needed, never wrong. In several runs this rule pruned as much as Jev did.

If a dumb rule gets you most of the way, the expensive part has to earn its keep. Jev does — in the low tail — but the gap is smaller than we hoped.

Rough #5: it's hard to know if we were wrong.

The obvious check — “did pi read this file again later?” — is useless: pi never re-reads what is still in its backpack ( re-reads out of ~190). So we invented a stand-in: did pi later type a name that only this output contained? It works, but it over-counts and under-counts. Here are the “mistakes” it found:

Some are real misses (a name pi needed). Some are the proxy being picky (pi had already copied the name into its own notes). Only the live recall count is honest — and we have 13 runs of it, all zero. Need many more.

So… is it good?

As a tool: 6/10. Safe, never lost anything, makes the backpack ~15% lighter on long sessions, costs about the same. As a study: 8/10. We now know where Jev's signal is, what pruning costs, and that a dumb rule gets you far.