WikiTrace: Evidence-Linked Agent Memory When Lessons Hurt Performance

KAIST AI researchers released WikiTrace, an evidence-linked memory layer for LLM agents, in arXiv paper 2611.14320 on November 10, 2026 — with Apache-2.0 code and Dockerized benchmarks the same week. The headline claim is uncomfortable but familiar: memory can make an agent fail a task it would solve without memory, because distilled lessons are often wrong, over-general, or misread.

WikiTrace’s fix is procedural. Each insight stays linked to the compilation and raw trajectory of the run that produced it. On a new task, the agent starts with all insights, then descends into source runs with terminal tools when an insight lacks detail, conflicts with another, or came from a failed episode.

TL;DR

Memory-induced degradation in plain language

Many agent stacks today do three steps:

1. Run tasks and log trajectories
2. Compress logs into short lessons or insights
3. Prepend those lessons on the next task

Step 2 loses information. Lessons inherit labels only from success or failure, not from why a run worked. The model on step 3 cannot audit a lesson against its origin, so it may follow a confident but wrong rule — classic memory-induced degradation.

The WikiTrace authors frame outcomes with Gain and Regression against a No Memory rerun:

Regression is the degradation signal. A memory system with high Gain and high Regression is a bad trade for production — you win easy cases and break ones you used to pass.

explainx.ai covered adjacent ideas when Raindrop tested reasoning-inclusive memory on telecom tickets and when Anthropic described dreaming and memory components. WikiTrace is complementary: less about what text to store, more about provenance and verification hooks in the harness.

The README’s “eight suites” headline counts held-out evaluation splits across the five benchmark families — for example ALFWorld valid-seen and valid-unseen, PlanCraft seen and unseen, and AppWorld normal versus challenge — not five unrelated datasets. When you compare the +8.5 README figure with the +3.2 Hugging Face author post, check whether you are averaging every suite or only the five benchmark names tied to the paper abstract.

How WikiTrace works

The project README describes three layers:

1. Build — run the agent without memory on training tasks; a compiler writes a compilation per episode and up to three insights linked to that run
2. Run — evaluate a method on test tasks with the built corpus for that trial
3. Analyze — compute Gain, Regression, and plots versus No Memory

Methods compared via --method:

The wikitrace branch adds terminal tools so the model can open the linked trajectory when an insight is incomplete or suspicious. That is the evidence link the title promises — not vector retrieval alone, but replay authority.

Fingerprinting matters: the README says memory built under a different config, prompt, or git commit is refused, so evaluation cannot accidentally read stale corpora.

What the compiler writes (memory layers)

The repository documents memory in src/wikitrace/memory/README.md (referenced from the README). In practical terms, each training episode produces:

• A compilation — a compressed narrative of what happened, including why the run succeeded or failed
• Up to three insights — short lessons derived from that compilation
• Pointers back to the raw trajectory — tool calls, observations, and turns the harness logged

At inference time, kinj dumps insights plus a few similar runs into the user message. wikitrace keeps the same prompt injection but also exposes the corpus directory read-only. The system prompt nudges the model to descend when insights disagree or when a failed run spawned the lesson.

That descent pattern is why the authors emphasize terminal tools and a pinned Pi agent build in scripts/setup/pi.sh. WikiTrace is not a drop-in vector database; it is an opinionated harness where memory and tooling are co-designed.

The authors also host a project page with teaser figures for memory-induced degradation and the build–run–analyze loop. Those diagrams mirror the README narrative; use them when explaining the design to teammates who will not clone Docker images on day one.

Cost and resume behavior

Each test run writes metrics/summary.json with suite score and API cost, plus metrics/samples.jsonl for per-task rows. If a long benchmark job is killed, rerunning the same command resumes and skips finished samples — useful when ALFWorld or AppWorld suites run overnight.

The --corpus PATH flag lets you point kinj or wikitrace at another trial’s built memory, which is how ablation papers typically test whether insights alone or insights plus traces drive gains. Always pair corpus path with the same --trial metadata or analyze will disagree with your filenames.

Reported results (what the authors claim)

Numbers below come from the GitHub README and the Hugging Face paper page. Treat them as author-reported until you rerun wikitrace analyze.

• README headline: best on eight suites, +8.5 points average over the best prior memory method, 4.5% Regression
• Hugging Face community post from author Yumin Choi: +3.2 points average vs using the same insights without trace links across five benchmarks
• Hugging Face commenter orinmemory highlighted that rerunning without memory already flips 9.7% of solved tasks — a sobering baseline for any memory paper

Disagreement between +8.5 and +3.2 is mostly suite count and baselines, not a contradiction in the primary sources. Your reproduction should fix a trial ID and compare methods on the same corpus.

Try WikiTrace locally (commands from README)

Prerequisites: Python 3.11+, uv, Node 20, Docker.

git clone https://github.com/yukyeongleee/wikitrace.git
cd wikitrace
uv sync --extra dev
cp .env.example .env
# set OPENROUTER_API_KEY in .env
set -a ; . ./.env ; set +a
scripts/setup/pi.sh
scripts/setup/sandbox_image.sh

Install a benchmark (example ALFWorld):

DATA_ROOT=/path/to/data scripts/setup/alfworld.sh

Build memory for trial 0:

uv run wikitrace build \
  --config configs/alfworld.yaml \
  --benchmark alfworld \
  --suite valid-seen \
  --trial 0

Evaluate WikiTrace vs knowledge injection:

uv run wikitrace run \
  --config configs/alfworld.yaml \
  --benchmark alfworld \
  --suite valid-seen \
  --trial 0 \
  --method wikitrace

Analyze:

uv sync --extra analysis
uv run wikitrace analyze all --runs outputs/ --out analysis/

Outputs land under outputs/corpus/... for corpora and outputs/<benchmark>/<suite>/<method>/trial-NN/ for per-sample JSONL — suitable for comparing dollar cost per suite via metrics/summary.json.

Long-term agent memory shown as a green thread looping around labeled spools

Benchmarks and suites

Suites of one benchmark share training tasks, so they share one built memory per trial — important when you interpret unseen splits.

τ³-bench here is the retail domain in the WikiTrace harness, not the τ² telecom suite Raindrop used. Compare carefully when reading cross-post headlines.

Design lessons for builders

Small figure beside a notebook with a bookmark, as a metaphor for an agent checking a stored lesson before acting

Store provenance with every lesson. If you only embed insights in the system prompt, agents cannot appeal to source logs when two lessons disagree. A read-only mount plus terminal tools is one implementation; production might use structured citations to ticket IDs or commit SHAs instead.

Measure Regression, not just average score. Leaderboards that only show mean success hide memory that steals easy wins and breaks hard ones. WikiTrace’s analyze step exports both metrics with confidence intervals — copy that discipline in internal evals.

Separate compilation from inference. The build phase runs without memory, then compiles — similar in spirit to offline passes in loop engineering for coding agents, but specialized for embodied and tool benchmarks.

Redact before you link trajectories. WikiTrace sandboxes use --network none per setup scripts. Production trajectories often contain secrets; linking evidence without scrubbing is worse than no memory.

Do not trust single-number memory hype. The Raindrop 67% headline showed how quickly social summaries detach from primary sources. WikiTrace’s HF thread already asks how insight compilation scales when memory no longer fits context — a real deployment constraint.

How WikiTrace compares to nearby research memes

Limits and open questions

• Paper status: arXiv v1, submitted to ACL Rolling Review (October 2026 cycle); under review
• Stars were minimal at launch (4 on GitHub when we filed); expect setup friction on large benchmark downloads
• Insights still land in the first message today — scaling may need retrieval, per HF discussion
• Results depend on OpenRouter models in config — swap models and re-fingerprint corpora
• Not a hosted product; no Anthropic or OpenAI shipping integration

Porting the idea without the full harness

You do not need ALFWorld installed to apply the design pattern:

1. Version lessons with the git SHA and model name used when they were distilled
2. Store a pointer to the raw log object, not only the summary sentence
3. On conflict, fetch the log slice before editing code or calling tools
4. Log Regression on a fixed eval set whenever you change the memory writer prompt

Products like Raindrop focus on lesson quality from reviewer CoT; WikiTrace focuses on agent-side appeal to primary evidence. A mature stack might do both — reviewer-rich writes plus trace-linked reads.

For teams already using semantic caching or Bedrock AgentCore, treat WikiTrace as a research reference for eval hygiene, not a vendor integration.

Related reading

• Raindrop reasoning memory and 67% headline
• Anthropic components of agent memory
• Karpathy LLM wiki agent memory guide
• Loop engineering for coding agents
• WorkflowArena multi-agent benchmark
• What are agent skills
• Agent harness comparison landscape

Install paths, benchmark names, and reported metrics match the WikiTrace GitHub README and arXiv:2611.14320 as accessed November 11, 2026.