Google DeepMind's Recirculation paper has an unusually attractive premise: improve an off-the-shelf language model at inference time without changing its weights. On Gemma 3, the authors report lower perplexity, better state tracking, and a headline 21% GSM8K gain.
The practical story is more interesting than the headline. The August 18 preprint describes two variants, only one of which is fully training-free. Its detailed GSM8K section reports relative error reduction, not a 21-point accuracy jump. And the architecture moves compute into the least convenient part of many agent workloads: prompt prefill.
For builders already comparing recursive reasoning with chain-of-thought, Recirculation is a useful third option. It spends compute on maintaining internal state rather than printing more reasoning tokens.
TL;DR: what people are asking
| Question | Direct answer |
|---|---|
| What changes? | A small mixture of a deep-layer activation is fed into a shallower layer while processing the next input step |
| Are Gemma's weights changed? | No. Both variants freeze the base model |
| Is all training avoided? | Only in basic Recirculation. Adaptive Recirculation trains a small token-conditioned MLP |
| What is the 21% GSM8K result? | 20.9% relative error-rate reduction at pass@128; pass@1 error falls 8.8% |
| What improved most broadly? | Adaptive Recirculation cut mean perplexity 23.0% across nine datasets on Gemma 3 1B |
| What does it cost? | Almost no reported decode-latency increase, but prompt prefill becomes sequential and can be slow |
| Can I enable it in vLLM today? | Not as a standard flag; the paper presents research code paths, not a production serving integration |
| Should I ship it? | Benchmark it first on short-context, state-sensitive work; do not assume the Gemma result transfers to every model |
What Recirculation actually does
A normal feed-forward transformer lets each token move from shallow layers to deep layers once. Deep layers can resolve ambiguity using the full context, but that resolved representation is not available to the shallow layers processing what comes next.
The paper uses the word bank as its motivating example. Early layers initially represent both "river bank" and "financial bank." Deeper layers use context such as a fishing pole to settle on the river meaning. If later processing begins from the original ambiguous representation, the model can still answer an ATM question incorrectly.
Recirculation creates a controlled feedback path:
- Process token step
tthrough the transformer. - Take the residual-stream activation from a selected deep source layer.
- Normalize it and mix a small amount into a selected shallower destination layer.
- Use that enriched state while processing step
t + 1.
Conceptually, the update is:
next_state_at_destination =
alpha * normalized_deep_state + beta * current_destination_state
The paper's experiments use one additional iteration per stack. The key is not merely repeating compute; it is letting a contextualized state persist across input steps. That is why the authors describe the transformer as behaving more like a dynamical system.
Is Recirculation really training-free?
There are two answers because the paper evaluates two methods.
| Variant | Base weights | Extra learned parameters | What is tuned |
|---|---|---|---|
| Basic Recirculation | Frozen | None | Source layer, destination layer, fixed alpha and beta selected by a sweep |
| Adaptive Recirculation | Frozen | Small MLP | Token-conditioned vector-valued mixing coefficients |
| Full fine-tuning baseline | Changed | Whole model | Gemma 3 1B with recurrent connections |
Basic Recirculation is genuinely a training-free inference-time intervention. Adaptive Recirculation is more accurately described as base-weight preserving: it trains a small MLP that reads the source and destination activations and decides how strongly each feature should be mixed for each token.
For the perplexity experiment, that MLP was trained on 250 documents from each of arXiv, C4, and PG19. It then delivered a mean 23.0% perplexity reduction across nine evaluation datasets, compared with 8.5% for the fixed-coefficient method and 21.6% for full Gemma 3 1B fine-tuning.
That is still a meaningful efficiency result. It just is not the same as getting the adaptive gains with zero optimization work.
The 21% GSM8K headline needs a denominator
The paper's abstract says adaptive Recirculation produced a "21% increase in accuracy on GSM8K." The detailed results are more precise: on Gemma 3 4B, adaptive Recirculation produced an 8.8% reduction in error rate at pass@1 and a 20.9% reduction in error rate at pass@128.
Those are relative reductions in the remaining mistakes, not percentage-point gains in accuracy.
| Metric | What it asks | Reported adaptive result |
|---|---|---|
| pass@1 | Does the greedy first answer solve the problem? | 8.8% relative error reduction |
| pass@128 | Is any of 128 sampled answers correct? | 20.9% relative error reduction |
The distinction matters. Pass@128 measures whether the correct answer enters a broad sample set, so it says more about capability expansion than ordinary single-response product quality. A chat or tutoring interface usually serves one answer, not 128 and an oracle that selects the correct one.
This is also why we would not read the result in isolation from known GSM8K contamination and measurement problems. The paper does offer broader evidence—perplexity, instruction following, contextualization, and multiple-choice tasks—but gains on the eight single-token benchmarks were modest and not uniformly robust.
How is this different from chain-of-thought and looped transformers?
All three spend extra compute, but they put it in different places.
| Method | Where iteration happens | What persists | Main cost |
|---|---|---|---|
| Chain-of-thought | Generated token sequence | Text or hidden states behind emitted tokens | More decode tokens, latency, and token charges |
| Looped transformer | Repeated layers within depth | Activation refined through another layer pass | More layer computation per step |
| Recirculation | Across both depth and input steps | Deep contextual activation mixed into a shallow state | Serial context processing during prefill |
The chain-of-thought prompting guide explains why verbal intermediate steps can scaffold hard inference. Recirculation targets a more basic problem: keeping the model's interpretation stable while the context evolves. The authors argue those jobs are complementary, so a model could use Recirculation for state tracking and reserve visible reasoning tokens for difficult deductions.
It is also not ordinary activation steering. Steering usually adds a static direction derived outside the current example. Recirculation lets the model steer itself using its own token-specific deep activation.
Why prefill is the real product constraint
During autoregressive decode, every new token already depends on the prior step. The authors run two transformer stacks in parallel and report almost no additional generation latency on modern accelerator hardware.
That does not mean the technique is compute-free. It means the extra work may fit into parallel hardware capacity without extending the critical path in their setup. Builders still need to measure accelerator utilization, memory pressure, throughput, and cost per request.
Prefill is the sharper trade-off. Standard transformers process prompt tokens in parallel; Recirculation's state update depends on the previous input step, so the prompt must be processed serially. The authors explicitly warn that this may be infeasible for long contexts.
That splits likely workloads into two groups:
| Better first candidates | Poor first candidates |
|---|---|
| Short prompts with many generated tokens | Long RAG contexts with short answers |
| Multi-turn state tracking with compact history | Agents injecting huge tool schemas on every turn |
| Structured narratives where entities change over time | Document summarization dominated by prefill |
| Research serving where quality matters more than TTFT | Latency-sensitive search and autocomplete |
The trade-off is almost the inverse of Flash Attention improvements that accelerate Gemma prefill. It also matters for teams using context compression to protect agent latency: Recirculation makes every retained input token more expensive on the serial path.
What should builders test before taking this seriously?
The authors call Recirculation a methodological contribution, not a shovel-ready technique. There was no official reference implementation linked from the paper as of publication, so the sensible next step is a bounded research spike rather than an infrastructure rewrite.
Use this evaluation contract:
{
"model": "gemma-3-size-and-checkpoint-you-serve",
"workload": "your-state-sensitive-eval-set",
"variants": ["baseline", "fixed-recirculation", "adaptive-recirculation"],
"quality": ["task-success", "perplexity", "state-consistency"],
"latency": ["prefill-ms", "time-to-first-token", "decode-tokens-per-second"],
"cost": ["accelerator-seconds-per-request", "peak-memory", "batch-throughput"],
"decision": "ship only if quality gain beats prefill and throughput cost"
}
Then follow five rules:
- Measure absolute accuracy and relative error reduction. Report both so a favorable denominator cannot hide a small product gain.
- Separate pass@1 from pass@k. Your product probably cannot use an oracle to choose from 128 samples.
- Sweep prompt lengths. Test short, median, p95, and maximum contexts; a single average TTFT number will hide the serial-prefill cliff.
- Re-select layers per checkpoint. The paper found model- and task-dependent source/destination settings; do not copy Gemma 3 1B values into another architecture.
- Use fresh task data. The adaptive multiple-choice results depended heavily on the tuning dataset, and one MMLU setup included overlap between tuning examples and the evaluated test distribution.
For the wider evaluation stack, use the AI benchmarks guide to pair academic scores with workload-specific tests. If cost is the deciding factor, compare the quality delta against Gemma's existing price-quality advantage on RAG, not just against an unmodified local baseline.
What the paper establishes—and what it does not
The strongest contribution is a new design lens: a trained model's residual stream may reveal where recurrence naturally fits, allowing researchers to prototype architectural changes before paying to retrain the whole network.
The paper also establishes limits:
- The largest gains are concentrated in Gemma; under limited tuning, Ministral 3, Pythia, Qwen 3, and Phi-2 showed less than 0.5% perplexity reduction versus roughly 5% for comparable Gemma experiments.
- Optimal layers, mixing coefficients, and normalization can depend on the model and domain.
- Single-token benchmark gains are mixed.
- Long-context prefill can become impractically slow.
- Results come from a version-one preprint, not a peer-reviewed production study.
So the honest takeaway is not "free intelligence." It is that frozen weights do not imply a frozen architecture. The inference graph itself may be another scaling surface—alongside quantization, faster hardware such as Gemma 4 on Cerebras, and output-token reasoning.
Related on explainx.ai
- Recursive reasoning in 2026: HRM, TRM, and inference-time scaling
- Gemma 4 July update: faster prefill, tool calling, and vision
- Gemma 4 31B cost-quality benchmark for RAG
- Gemma 4 31B on Cerebras at 1,800+ tokens per second
- Chain-of-thought prompting: zero-shot vs few-shot
- Why benchmark contamination breaks AI model comparisons
- Headroom AI context compression for agents
Primary source: Mozer, Siddiqui, Sawyer, Sanyal, and Liu, "Recirculation," arXiv:2608.17981 · Experimental HTML version
This analysis reflects version 1 of the Recirculation preprint submitted August 18, 2026. The reported results are research measurements, not production guarantees; code availability and serving integrations may change after August 24, 2026.