Every additional token an LLM generates costs more compute than the one before it. That is the blunt cost curve of quadratic attention — every existing token has to be compared against every new one, so a sequence twice as long costs roughly four times as much to attend over, and the keys and values for every token have to stay in memory for as long as the conversation lasts. It is the single biggest reason long-context inference is expensive, and it is why so much research effort over the last three years has gone into finding a cheaper substitute for standard attention.
On August 31, 2026, Microsoft Principal Researcher Alexia Jolicoeur-Martineau — 2025 ARC Prize winner — published a paper with collaborators Rhea Sanjay Sukthanker, Pashmina Cameron, and Emy Gervais that undercuts one of the most hyped substitutes: retrofitting an already-trained model to use linear attention. Their result, titled "Sliding-window beats linear attention" (arXiv:2608.28444), is that a much simpler technique — sliding-window attention with attention sinks, applied with zero retraining — beats it.
TL;DR
| Question | Direct answer |
|---|---|
| What's the finding? | Sliding-window attention (SWA) with sinks, applied at inference with no post-training, beats models retrofitted to linear attention via post-training |
| Does this apply to models trained with linear attention from scratch? | No — see the scope section below. This is a post-training/retrofit comparison only |
| How much better is it? | 2-10x higher on long-context reasoning benchmarks (Needle-in-a-Haystack, BABILong) vs. post-trained linear-attention baselines |
| Does it cost anything to try? | No — SWA requires no retraining, no specialized kernels, and lower inference memory than either dense or linearized attention |
| What's the practical takeaway? | Don't reach for a linear-attention conversion of an existing model expecting a free lunch on memory — SWA with sinks is the more reliable lever |
Why quadratic attention is expensive, concretely
In standard (dense) self-attention, every token computes a relevance score against every other token in the sequence. For a sequence of length n, that is n² comparisons per layer. Two consequences follow directly:
- Compute scales quadratically. Double the context length and the attention computation roughly quadruples, not doubles.
- Memory scales linearly but indefinitely. Every token's key and value vectors must be cached (the "KV cache") for as long as it might be attended to later — which, in a standard causal transformer, is forever, for the rest of the conversation.
That second point is what the paper's abstract means by "every new token costs more than the previous one": token 10,000 has to be checked against, and remembered alongside, all 9,999 tokens that came before it, while token 100 only had 99. This is also why context-window pricing on frontier APIs is not a flat per-token rate — providers pay this cost curve directly, and it shows up in your bill as cumulative input tokens and long-context premiums.
Two broad families of fixes have emerged: cap what each token can see (sliding-window / sparse attention), or change the attention math itself so memory doesn't grow with sequence length (linear attention). This paper puts the two head-to-head — specifically, when applied to a model that already exists.
What "sliding-window attention with attention sinks" actually means
Sliding-window attention (SWA) is not new — it powers models like Mistral and Longformer. Instead of letting every token attend to the entire history, each token only attends to a fixed-size local window of the most recent tokens (the paper primarily tests small windows like 64 tokens, with a 4-token sink, and separately evaluates larger windows — 128, 256, 512 — for long-context tasks). That caps both compute and memory per layer at a constant size, regardless of how long the sequence gets.
The obvious problem: if a token can only see the last 64 or 256 tokens, how does it ever use information from earlier in a long document? Two things rescue this:
- Stacking layers compounds the effective range. If each layer lets a token see back n positions, after k layers a token has effectively accumulated context from k × n positions back — similar to how stacking convolutional layers expands a receptive field. Jolicoeur-Martineau described this directly: "SWA has a vertex-length. After k layers, each token has seen k×n_layers context-length... I expect SWA post-training to help for long-context by leveraging the older context accumulated after multiple layers."
- Attention sinks stop the mechanism from destabilizing. A small number of early tokens — the "sinks" — stay permanently visible no matter how far the window slides forward. This traces back to the StreamingLLM observation: softmax attention has to route its scores somewhere, and models learn to dump disproportionate attention weight onto the first few tokens as a kind of no-op. Drop those tokens from a sliding cache and attention quality collapses; keep them pinned in place and the window stays stable indefinitely. See the attention sink dictionary entry for the mechanism in more depth.
The paper's core move is combining both — a small local window plus a handful of persistent sink tokens — and applying it to an already-trained dense-attention model, at inference time, with no fine-tuning at all.
Why the linear-attention retrofit (LoLCATs-style) falls short here
Linear attention takes a different approach: instead of capping what a token can see, it changes the attention computation so it never needs to store an ever-growing KV cache in the first place — it compresses the running history into a fixed-size state, updated recurrently as each new token arrives. In theory this gives constant memory regardless of sequence length, which is why it has attracted so much attention (no pun intended) as a fix for the quadratic-cost problem.
The catch, per this paper: linear attention's fixed-size state means it must choose what to remember and what to discard as new tokens arrive. That's a real compression decision, not a free simplification. Methods like LoLCATs (Zhang et al., 2025) try to retrofit a pretrained quadratic-attention model into this scheme cheaply — combining a linear-attention approximation (Hedgehog) with sliding-window attention, using as little as 40M post-training tokens.
Jolicoeur-Martineau's team found this doesn't hold up:
"Retrofitting an LLM to use linear attention sounds great in theory, but it does not deliver its promises. It adds cost and complexity while degrading performance on long context massively."
"Linear attention has limited memory, so it must choose which tokens to remember and which to forget. Trying to learn this during post-training at a small cost is misguided."
The team's own investigation into why is arguably the most useful finding for practitioners: while trying to make LoLCATs-style retrofitting work well, they discovered the linear-attention component wasn't contributing much of anything. "I discovered that linear-attention was not doing anything since removing it and just keeping the SWA gave similar results," Jolicoeur-Martineau said. In other words, the sliding-window half of the LoLCATs recipe was doing the real work — the linear-attention half was mostly along for the ride.
The reported numbers back this up. Per the paper and the announcement thread, the authors evaluated up to 4K context length with a window size of 256 — well past the window itself, meaning SWA had to rely entirely on the multi-layer accumulation effect, not on directly seeing distant tokens. On the BABILong long-context reasoning benchmark at that setting, SWA recovered roughly 25% of the full baseline model's performance while the LoLCATs linear-attention retrofit reached only about 5% — close to collapse. On short-context general knowledge benchmarks like MMLU, the paper reports SWA recovering around 99% of average baseline performance, versus roughly 83% for LoLCATs and 62% for a related linearization method, Liger-GLA.
The scope limit: this is a post-training result, not a general claim
The original announcement title — "Sliding-window beats linear attention" — drew a pointed reply from well-known ML researcher Lucas Beyer: "I'm a bit disappointed you made the title so misleading. Just add in post training to it."
That critique is fair, and it matters for anyone skimming the headline. The paper is explicitly a post-training / retrofit comparison: it takes models that were already pretrained with standard quadratic attention, and asks which cheap conversion technique — sliding-window masking or linear-attention approximation — works better after the fact, without full retraining. The paper's own framing backs this: the authors describe their approach as "training-free SWA," explicitly contrasted with linearization approaches that require post-training, and note that linear-attention models "may have shown some promise, but they likely require to be trained from scratch or extensive post-training" to actually work well.
That means this result says nothing about linear-attention or state-space architectures that are pretrained from scratch to use that mechanism — Mamba, RWKV, and similar architectures aren't what's being compared here, and nothing in this paper argues they underperform dense attention when trained natively. The claim is narrower and, for builders doing inference-cost optimization on an existing model, arguably more useful: if you're taking a model you already have and trying to cheapen it after the fact, sliding-window-with-sinks is the more reliable lever than a linear-attention conversion.
What this means if you're optimizing inference cost today
The practical takeaway lines up with the paper's own recommendation, stated plainly in the abstract: "To reduce inference memory cost, we strongly recommend switching to SWA instead of post-training linear models."
If your goal is cutting memory and compute on a model you already have — not training something new from scratch — the ranked order this research suggests is:
- Sliding-window attention with sinks first. No retraining required, fast, low memory, and per this paper it holds up far better than a linear-attention retrofit on both short- and long-context tasks.
- Be skeptical of "cheap linearization" claims for existing models. A retrofit that only spends a small post-training budget to teach a model a new memory-compression policy is asking a lot in a short amount of time — the paper's authors call this "misguided," and their own ablation (removing the linear-attention component from LoLCATs and getting similar results) suggests the underlying mechanism may not have been doing much of the work analysts credited it with.
- Reserve judgment on natively-trained linear/state-space architectures. This paper doesn't touch that question, and Beyer's correction is the right one to keep in mind before overselling the headline in either direction.
For anyone building agent infrastructure or long-context RAG pipelines where inference memory is the binding constraint, this is a useful, low-cost thing to test before reaching for a bigger architectural change.
Related reading
- Update — September 1, 2026: Tsinghua reported the first deterministic SSSP improvement since 1984 — another "forever algorithm" assumption shifting. Sorting barrier breakthrough →
- What Is a Transformer? The Architecture Behind Every Modern LLM
- Context Window Pricing, Decoded
- LLM Context Window Explained
- SubQ: SSA sparse attention, 12M context, and long-context evals
- Gemma 4 Updates: Flash Attention and Tool Calling
- RAM Prices, AI Demand, and the Local Inference Cost Problem
Official paper: "Sliding-window beats linear attention", Jolicoeur-Martineau, Sukthanker, Cameron, and Gervais, arXiv:2608.28444 (August 2026).
Numbers, model names, and benchmark figures in this post reflect the arXiv paper and its authors' public announcement thread as of publication date; check the paper directly for any later revisions.