Apple Machine Learning Research has published Self-Compacting Language Model Agents, a NeurIPS-tagged paper written with Johns Hopkins University that answers a question every agent builder eventually hits: when should an agent summarize its own history? The method, called SelfCompact, hands the model a compaction tool and a short rubric and lets it decide. According to the Apple page, it improves accuracy by up to 18.1 points on competitive math, adds 5 to 9 points on agentic search, and cuts per-question cost by 30 to 70 percent, all without fine-tuning.
If you run long-horizon agents, this is directly relevant. Context handling is where most harnesses quietly lose quality, and we have covered the practical side before in our Claude Code token efficiency guide and the fast-jev compaction plugin post.
TL;DR: the questions people ask first
| Question | Answer |
|---|---|
| Who wrote it? | Tianjian Li, Jingyu Zhang, William Jurayj, Xi Wang, Chuanyang Jin, Mehrdad Farajtabar, Eric Nalisnick and Daniel Khashabi (Johns Hopkins plus Apple). |
| What is new? | The model itself chooses when to compact, guided by a rubric, instead of a fixed token trigger. |
| Does it need training? | No. Inference-time only. |
| What did it beat? | A no-summarization baseline and fixed-interval summarization. |
| Headline numbers | Up to +18.1 points math, +5 to 9 points agentic search, 30 to 70 percent lower per-question cost. |
| Scale of evaluation | Six benchmarks (competitive math and agentic search), seven models. |
| Is code out? | Not linked on the Apple page; the paper is on arXiv. |
The problem: long traces rot
An agent that reasons and calls tools produces a growing trace of chain-of-thought and tool outputs. The paper points to two failure modes. First, stale content stays in the window and can anchor later generations: an early wrong guess keeps pulling the model back. Second, the trace eventually exceeds the context window, forcing a hard cutoff.
Most scaffolds respond with fixed-interval compaction: when the context reaches some token threshold, summarize and continue. That is simple, and it is what many coding agents do today. But the authors argue the trigger is blind to structure. It fires whether the agent just finished a sub-problem or is halfway through an algebraic derivation, and in the second case the summary discards exactly the partial results the model needs.
Our own agent harness tier list discussion shows how much builders argue about this layer. SelfCompact is one of the cleaner attempts to give the argument some data.
How SelfCompact works
SelfCompact folds a broad agent trace into a compact summary, shown as strips folding into an accordion
The method has two inference-time pieces, and the authors insist both are required.
- A compaction tool. A function the model can call to summarize everything accumulated so far and continue from that summary.
- A lightweight rubric. A short instruction telling the model when compaction is a good idea (a sub-task has resolved, the trajectory is converging) and when to hold off (mid-derivation, or when the model is stuck).
The ablation story is the interesting part. With the tool alone, open-weight models use it unevenly: often at unhelpful moments, sometimes not at all. With the rubric alone, the model has no way to act on the advice. Put together, they produce adaptive compaction with no supervision signal and no weight updates.
The "meta-cognitive gap"
The authors report what they call a meta-cognitive gap: unprompted models cannot reliably tell when their own context has degraded. A model deep in a trace does not naturally notice that it is anchored on stale reasoning. But a rubric that names the situations closes much of that gap. The practical reading is that the capability is already latent in the model and the scaffold only needs to point at it, which is a recurring theme in modern harness design.
What the results say
The evaluation covers six benchmarks spanning competitive math and agentic search, run on seven models. According to the Apple summary:
- Math: up to 18.1 points above a no-summarization baseline.
- Agentic search: 5 to 9 points above baseline.
- Cost: 30 to 70 percent lower per-question cost.
- Versus fixed-interval summarization: matches or exceeds it at a fraction of the token cost.
Two caveats are worth stating plainly. The page we read gives ranges rather than a per-model table, so the exact gain for any one model is in the paper, not in this post. And "up to 18.1 points" is a best case; the agentic-search gains are smaller and more representative of general-purpose work.
Why it saves money
A long ribbon of agent context passing through a ring, illustrating how self-compacting agents keep long traces affordable
Cost falls for a straightforward reason. Every token in the context is re-read on each step, so a long trace is paid for again and again. Compacting at a natural boundary shrinks all later steps. Compacting at a bad moment, by contrast, can force the agent to redo work it just lost, which burns tokens in the other direction. By timing the compaction well, SelfCompact avoids both the bloat of never compacting and the rework of compacting blindly.
This lines up with what practitioners see with caching: summarizing changes the prefix, which can break prompt-cache hits, so you want to compact rarely and at good moments. Our Gemini context caching and harness cost breakdown covers that trade-off in detail.
How to apply the idea in your own harness
You do not need the paper's code to try the core idea. A minimal version looks like this:
- Expose a
compact_contexttool that takes a summary string written by the model and replaces the trace so far. - Add a short rubric to the system prompt. A reasonable draft: compact when you have finished a sub-task and no intermediate derivation is still open; do not compact while mid-calculation, mid-proof, or when you are stuck and need the exact earlier text.
- Log every compaction call with the token count before and after, so you can see whether the model uses the tool at sensible moments.
- Compare against your current fixed-threshold trigger on a handful of long tasks, tracking both accuracy and total tokens.
Because the paper found tool-only setups behave erratically on open-weight models, do not skip step 2. If your model keeps ignoring the tool or firing it constantly, tighten the rubric before changing anything else.
This also pairs with memory-style approaches: our post on why agents need documentation, not just memory argues for durable notes outside the window, and a compaction summary is one such note.
How it differs from other context-management tricks
Developers have tried several other ways to keep long traces manageable. Sliding windows simply drop the oldest turns, which is cheap but forgets early instructions. Retrieval-style memory stores notes outside the window and fetches them on demand, which adds infrastructure. Fine-tuned compaction trains the model to summarize, which needs data and compute. SelfCompact sits at the lightest end of that spectrum: a tool definition and a few sentences of prompt. That makes it an easy first experiment before anything heavier, and an easy baseline for any team that later builds something more elaborate.
Limits and open questions
- Domain coverage. Math and agentic search are the tested domains. Long coding sessions, where exact earlier file contents matter, may behave differently, and summaries can drop details a coding agent later needs.
- Rubric sensitivity. The paper shows the rubric matters, which also means wording could matter; how robust the gains are to rubric variants is something to test yourself.
- No code link yet. Reproduction currently means re-implementing from the paper.
- Summary quality. A model that compacts at the right time can still write a poor summary. The method addresses when, not how well.
- Vendor harnesses. Closed products such as coding assistants already ship their own compaction logic. Whether a model-chosen trigger beats those in practice is not tested in what we read.
Where this sits in Apple's research stream
Apple's machine learning group has been publishing steadily on efficiency and generation, such as the Normalizing Trajectory Models work on four-step image generation. SelfCompact is a different flavor: it is not about a new model but about squeezing more out of existing ones at inference time, which is exactly the kind of result open-weight users can use immediately. Most of the listed authors are marked as Johns Hopkins affiliated on the Apple page, with Mehrdad Farajtabar the one author without that mark.
What this means for what you build or pay
If you pay per token for long agent runs, the headline is the 30 to 70 percent cost reduction paired with accuracy gains: those two rarely arrive together. If you build harnesses, the lesson is to treat compaction as a decision the model can make, with guidance, rather than a timer. And if you evaluate agents, add a "bad compaction" test: tasks where a mid-derivation summary would be fatal.
Further reading and sources: the Apple Machine Learning Research page and the arXiv paper.
Related reading
- Claude Code token efficiency and prompt cache guide
- fast-jev compaction plugin for Claude Code
- Agent harness tier list reaction
- Gemini context caching and agent harness costs
- Agents need documentation, not just memory
- Apple Normalizing Trajectory Models
- ColGrep agent and Claude Code token savings
Details reflect the Apple Machine Learning Research page as of October 12, 2026; check the paper for full tables.