Memory Tax: Many Agent Memory Systems Lose to Plain RAG, Study Finds

Many dedicated agent memory systems score lower than a plain retrieval baseline over the raw chat history, even though they cost far more to run. That is the headline of "Memory Tax: Dissecting the Net Value of Agent Memory" (paper page on Hugging Face, arXiv 2610.34073), a University of Illinois Urbana-Champaign study that trended on the daily papers list on October 28, 2026. The authors tested 11 popular memory systems on three benchmarks and three model backbones, and give the shortfall a name: the memory tax.

If you build agents that remember users, projects or past runs, this matters before you add another memory layer. The paper does not say memory is useless. It says you should measure it against naive RAG first, and it shows where the losses come from.

TL;DR: what the paper found

What is "memory tax"?

Most memory papers compare a new design with other memory designs. The authors surveyed prior work and found that naive RAG, which keeps the raw history and retrieves chunks of it, is often missing from the comparison or only loosely specified. Without it, the value of building memory is tangled up with the value of simply retrieving evidence.

So the paper uses naive RAG as a reference and compares each memory system question by question:

• Memory benefit: questions only the memory system answers correctly. Extraction, aggregation and preference inference can help here.
• Memory tax: questions the memory system gets wrong that RAG gets right, because evidence was lost, buried or rewritten.
• Net value: benefit minus tax. When the tax is larger, memory ends up below RAG overall.

The project page makes the point that the same design choice can win one question and lose the next. Extracting atomic facts, linking sessions, anchoring time or rewriting content all deviate from the raw history. That is useful when the deviation matches the question and harmful when it does not.

This is a measurement framework, not a new product. Its value is that you can run the same comparison on your own memory layer.

How the experiment was run

The authors group the 11 open-source systems by how they store information:

Every system ran on all three backbones with the same embedding model, Qwen3-Embedding-0.6B. Answers were scored by an LLM judge backed by Qwen3.6-27B. Because memory stores differ so much in granularity, the authors aligned a "retrieval scope" so each system sees roughly as much of its store as the RAG baseline does, tuning top-k upward from the default in most cases. That guards against the obvious objection that memory systems lose only because they were given a smaller context.

They also report token cost, wall-clock time, LLM calls and store size. The project page's leaderboard summarizes the pattern: memory systems "usually pay far more cost than RAG, but often without a matching gain in accuracy." The exact per-system scores are in Table 1 of the paper and on the project page; we are not reproducing every cell here because the numbers vary by benchmark and model.

Agent memory pruning shown as shears trimming one branch of a small tree, a picture of how memory tax loses evidence

Where the tax comes from

The paper traces the tax along three dimensions.

1. Faithfulness: was the information kept?

Compression and rewriting can drop or distort evidence before retrieval even starts, and that loss cannot be recovered later. The authors located the raw turns that hold each answer (gold raw turns) and the memory units built from them (gold memory units). Answering with only the gold memory units removes retrieval from the picture, so any remaining tax comes from construction. They also sampled atomic facts and annotated how much of each survived: no loss, partial loss, major loss, or no memory unit at all. Loss exists in many systems, with varying severity.

2. Accessibility: can it be found?

Even when evidence survives, it has to be retrieved. Replaying retrieval round by round, the authors found memory systems recover fewer questions in the first round than raw chunks do, so the evidence sits deeper. Case studies show three patterns: evidence fragmented across units, diluted by unrelated content, or rephrased so it no longer matches the query. Systems such as MAGMA and LightMem score close to their gold-raw-turn results when handed the right memory units, but far below that in normal use, which points at access rather than storage.

3. Robustness: does it survive a model swap?

A long-lived agent may build memory with one model and read it with another. The authors built memory with each backbone and swapped the answering model. Changing the builder shifted scores, as expected. More interestingly, with the store fixed, switching the answering model hurt many memory systems more than it hurt RAG. Model-dependent operations add variability on top of raw history.

Benchmarks may be too easy for retrieval

The paper also looks at the evaluation side. Simple retrieval baselines already score high on several widely used long-term memory benchmarks: an IDF-keyword matcher that uses no embeddings and no LLM passes 50% on some, and a hybrid RAG reaches about 70%. When naive retrieval solves that much, there is little headroom for heavy memory to add value, and even modest losses show up.

The authors do not conclude that retrieval-friendly queries are unimportant. Accurate recall is a real job, and many real-world questions are exactly that. They suggest future benchmarks add harder settings and report memory systems alongside naive RAG so the remaining room is visible.

This lines up with other recent work we have covered. Agents don't need memory, they need documentation makes the practitioner version of the argument, and the lightweight approaches the paper cites as "inheriting" RAG's performance fit the same pattern.

Does it generalize beyond chat memory?

Two extensions suggest the benefit-tax view is not limited to conversation logs.

• Experiential memory (agents reusing past task traces, tested with Qwen3.5-4B across ALFWorld, WebShop, Mind2Web, BigCodeBench, AppWorld and tau2-Bench): distilling traces into workflows beats naive trajectory RAG on web tasks, while several memory methods fall behind it on tool use.
• World-model memory (keeping scenes consistent in long video generation): on the Context-as-Memory and MemoBench evaluations, reinjecting past raw frames stays strong. In the table on the project page, naive frames scored SSIM .560 on C-as-M-eval, ahead of the latent and KV-cache methods listed, which sit between .217 and .388.

The pattern repeats: the raw source is a hard baseline to beat.

What to change if you run an agent memory layer

The authors offer three design directions, tested only as proofs of concept.

A notebook with a bookmark as an agent memory store that still needs a path back to the raw source

Make deviations recoverable. Since harmful rewrites are hard to remove, keep a lightweight path back to the source. In their src-aug test, they retrieve from the raw history alongside memory and rerank both jointly. With Qwen3.6-27B:

Source: the project page's design-implications table. Tax is in percent of questions.

Let memory learn how it is used. A "recursive" variant feeds past queries and retrieved evidence back into memory construction, so similar needs later reach the right unit directly. The gains were mixed in the paper's PersonaMem-v2 test: SimpleMem was flat (46.7 to 46.8), MemoryOS moved from 48.2 to 49.1 and Mem-T from 42.1 to 45.7.

Model future demand. Memory built ahead of time cannot know what will be asked; memory built at query time adds latency. The authors suggest estimating which past information is likely to matter, treating memory as a kind of implicit recommender.

A practical checklist

1. Add a naive RAG baseline over raw history to your own evals, with the same context budget as your memory layer.
2. Compare per question, not just in aggregate: count what memory wins and what it loses.
3. Keep raw turns retrievable next to any summaries or extracted facts.
4. Re-test after changing the model that reads memory, not only the one that writes it.
5. Check whether your benchmark is solvable by keyword search; if it is, it says little about memory.

Caveats

• The study is a preprint. It has not been peer reviewed, and the authors' own LLM judge is Qwen3.6-27B.
• Cost limits meant only 20% of PersonaMem-v2 and 10% of MINT-Eval were evaluated, with stratified sampling. One GPT-5.6 Luna run used a 100-question LongMemEval subset, per the repository README.
• Results are specific to the configurations, top-k settings and open-source versions the authors pinned. Vendors may have changed their systems since.
• The README lists releasing reproduction code for each baseline and the ablation results as still pending, so full reproduction may not be possible yet. Results are already on Hugging Face as separate datasets.
• Memory and RAG are not exclusive. Many production systems use both, and the paper's own fix is a hybrid.

How this fits the memory wave

The last month has been busy for agent memory. We covered Databricks MemCurrent's claimed 3x pass-rate, Cognition's Devin dreaming memory, Tencent's open memory-writer model and Asymptote Labs' cross-harness Beacon. Each reports gains against its own setup. The Memory Tax paper is useful as the counterweight: before trusting any of them, ask what plain retrieval over the same history scores.

For background, our guides on what MEMORY.md does for agent persistence and how embeddings and vector search work explain the pieces that naive RAG relies on.

Related reading

• Agents don't need memory, they need documentation
• Databricks MemCurrent: agent memory claims
• Cognition Devin dreaming memory
• Tencent Hy-Memory-30B-A3B
• Beacon cross-harness agent memory
• What is MEMORY.md
• Embeddings and vector search guide

Sources: Memory Tax on Hugging Face Papers, arXiv HTML version, project page and GitHub repository.

Details are accurate as of October 28, 2026. The paper is a preprint and its repository is still being updated.