SubQ: SSA sparse attention, 12M context, and long-context evals

Subquadratic is shipping SubQ, an LLM positioned as the first fully sub-quadratic sparse-attention stack—branded SSA (Subquadratic Sparse Attention) in their technical article. The homepage advertises 12M-token reasoning, ~$1/5 relative cost framing versus leading LLMs (comparison axes unstated), and two surfaces: a full-context API and SubQ Code for existing coding agents.

This article summarizes what Subquadratic publishes—not independent verification of every throughput or eval cell.

TL;DR

Why long context is the stated problem

Subquadratic’s SSA article argues enterprise failure modes are usually distributed evidence: codebases, contracts, corpora, spreadsheets, and long-running agent transcripts where answers require cross-references, not single-span lookup.

The post distinguishes:

• Nominal context (tokens accepted)
• Functional context (tokens reliably reasoned over)

That distinction matches how practitioners already read needle-style benchmarks versus real repo tasks—see also our LLM context window guide.

What SSA claims technically

In their narrative, dense attention pays quadratic cost even when most pairwise scores are negligible. FlashAttention improves execution of the same all-pairs workload; SSA instead selects which positions receive exact attention.

Reported wall-clock prefill speedups vs their dense FlashAttention baseline on B200s (from their table):

They also publish attention FLOP reduction multiples vs standard quadratic attention, including 62.5× at 1M tokens.

Caveat: these are vendor-published numbers pending broader independent reproduction in your hardware/stack.

Benchmarks they highlight (read the tables in-source)

The technical page includes comparison tables against several named frontier models for:

• RULER @ 128K
• MRCR v2 (8-needle, 1M)
• SWE-Bench Verified

Interpretation discipline:

• MRCR is explicitly hard on multi-evidence retrieval; a mid-pack score can still accompany strong SWE if tasks differ.
• SWE-Bench is not pure retrieval; it stresses patch quality and repo interaction assumptions.

If you run agent harnesses, pair model claims with agent harness engineering thinking: tool surface, traces, and evals dominate ship quality.

SubQ Code vs the raw API

Subquadratic markets two entry points on subq.ai:

1. API — OpenAI-compatible endpoints, streaming, tool use, 12M window claim.
2. SubQ Code — integration path for IDE/agent hosts with routing to reduce expensive turns.

If you adopt either, baseline latency, quality, and cost on your repos—not the headline ratio alone.

Mathematical Foundations: Why Quadratic Attention Breaks at Scale

To understand SSA's innovation, we must first understand why traditional attention is computationally prohibitive for long contexts.

The Quadratic Bottleneck

In standard self-attention, each token attends to every other token in the sequence. For a sequence of length N:

• Attention matrix size: N × N
• FLOPs for attention: O(N² × d), where d is the model dimension
• Memory for KV cache: O(N × d × layers × heads)

At 1M tokens with d=4096, 32 layers, and 32 heads:

• Attention FLOPs: ~1M² × 4096 = 4.1 trillion FLOPs per forward pass
• KV cache: 1M × 4096 × 32 × 32 × 2 bytes = 8.6 GB of GPU memory

Doubling context to 2M tokens quadruples FLOPs to 16.4T and doubles KV memory to 17.2GB. This scaling makes million-token contexts impractical on consumer hardware and expensive even on datacenter GPUs.

Why FlashAttention Helps (But Not Enough)

FlashAttention (Dao et al., 2022) and FlashAttention-2 (2023) optimize the execution of quadratic attention by:

• Tiling computations to fit in SRAM (reducing HBM round-trips)
• Fusing operations (softmax, dropout, masking) into single kernels
• Recomputing attention scores during backward pass to save memory

FlashAttention achieves 2-4x speedups and enables longer contexts than naive implementations, but it's still O(N²) in FLOPs—the fundamental quadratic cost remains.

At 12M tokens (SubQ's marketed window), dense FlashAttention would require:

• ~144 trillion FLOPs per forward pass (12M²)
• ~100GB+ KV cache (depending on model architecture)

This is why Subquadratic claims SSA is necessary for functional long context rather than just nominal capability.

SSA Technical Architecture: How It Works

While Subquadratic hasn't published a full academic paper (as of May 2026), their technical blog describes SSA's core mechanisms:

Content-Dependent Sparse Selection

Instead of computing attention for all N² token pairs, SSA selects a subset of relevant keys for each query based on content similarity.

Step 1: Query-Key Similarity Scoring

For each query token q_i, compute approximate similarity scores with all keys k_j using a fast hashing or clustering technique (likely Locality-Sensitive Hashing or k-nearest neighbors in embedding space).

Step 2: Top-K Selection

Retain only the top K most similar keys for full attention computation, where K << N. For example:

• At 1M tokens, K might be 1,000 (0.1% sparsity)
• At 12M tokens, K might be 10,000 (0.08% sparsity)

Step 3: Full Attention on Subset

Compute exact attention scores only for the K selected keys, reducing FLOPs from O(N²) to O(N × K).

Step 4: Residual Dense Attention

To prevent catastrophic forgetting of important but low-similarity tokens (e.g., structural tokens like section headers), SSA likely includes a fallback dense attention layer at coarser granularity or on downsampled representations.

Adaptive Sparsity

SubQ's claim of "content-dependent" sparsity suggests the sparsity level adapts per query:

• Queries in dense information regions (e.g., middle of a paragraph) may attend to fewer keys
• Queries at boundaries (e.g., start of a new section) may attend to more keys to gather context

This is more sophisticated than fixed sparse patterns (e.g., sparse transformers with fixed strides) and enables better recall on retrieval tasks.

Memory Optimization

Beyond FLOPs, SSA must also reduce KV cache memory. Techniques likely include:

• Quantization: Storing keys and values in lower precision (INT8 or FP8 instead of FP16)
• Pruning: Evicting low-attention keys from cache after each layer
• Compression: Using learned compression codebooks (similar to vector quantization in CodeLLaMA)

Combined, these allow SubQ to fit 12M tokens in memory without requiring 100GB+ of VRAM.

Benchmark Analysis: What the Numbers Actually Mean

Subquadratic publishes three key benchmarks:

RULER @ 128K

RULER (Realistic Ultra-Long Context Evaluation) tests retrieval, reasoning, and aggregation across long sequences. At 128K tokens, SubQ's reported score is competitive with frontier models (exact numbers in Subquadratic's table).

What this means: SSA doesn't degrade significantly from dense attention at moderate context lengths, validating that sparse selection isn't "lossy" for typical reasoning tasks.

MRCR v2 (8-Needle, 1M Tokens)

MRCR (Multi-Needle Retrieval in a Haystack) is explicitly hard: find and connect 8 facts scattered across 1M tokens of distractor text.

SubQ's reported performance here is lower than on RULER, which Subquadratic frames as expected for multi-evidence retrieval at extreme scale. The key question: How much lower?

If SubQ scores 60-70% on MRCR vs. 85-90% on RULER, that's a significant gap. If it scores 75-80%, that's more acceptable. The exact numbers matter for production risk assessment.

Interpretation: MRCR is a stress test, not a typical use case. Most real-world agent tasks don't require pinpoint retrieval across 1M tokens—they involve reasoning over ~100K-500K token codebases or document sets where SSA's trade-offs are more favorable.

SWE-Bench Verified

SWE-Bench Verified measures code editing on real GitHub issues. SubQ's reported score (exact value in Subquadratic's table) is compared to GPT-4, Claude 3.5, and Gemini 2.0.

What this means: SWE-Bench isn't purely a long-context benchmark—it tests code understanding, reasoning, and patch generation. A competitive score here suggests SSA's sparsity doesn't hurt the model's core coding capabilities.

Caveat: SWE-Bench solutions often fit in <50K tokens, so this benchmark doesn't directly validate 12M-token performance. It confirms that SubQ is a competent coding model, but not that it's superior to alternatives because of long context.

Real-World Use Cases: Where 12M Tokens Matters

When would you actually use a 12M-token context window?

1. Codebase-Wide Refactoring

Scenario: You need to rename a function used across 500+ files in a monorepo.

Why 12M helps: Loading the entire codebase (5-10M tokens) into context allows the model to find all call sites, understand dependencies, and generate a coherent multi-file patch—without RAG chunking or vector search.

Alternative: Use grep + targeted edits. But SSA enables one-shot reasoning across the full dependency graph, catching edge cases that chunked approaches miss.

2. Legal Document Analysis

Scenario: A law firm needs to cross-reference a 3,000-page contract against 500 pages of regulatory text.

Why 12M helps: Loading both documents (combined ~1-2M tokens) allows the model to identify conflicts, cite specific clauses, and reason about legal implications without manual chunking.

Alternative: Traditional document review with keyword search. But SSA enables semantic reasoning ("Does clause 42 contradict section 18(b)?") that keyword search cannot.

3. Multi-Document Research Synthesis

Scenario: A researcher needs to summarize findings across 50 academic papers (combined ~500K tokens).

Why 12M helps: The model can read all papers in one pass, identify common themes, and generate a meta-analysis—avoiding the bias introduced by sequential chunking.

Alternative: Read each paper separately and manually synthesize. But SSA accelerates the process and reduces human error.

4. Agent Trace Debugging

Scenario: An agent runs for 100 turns, accumulating 2M tokens of conversation, tool calls, and outputs, then fails. You need to debug why.

Why 12M helps: Loading the full trace allows the model to backtrack, identify the failing decision point, and propose fixes—without truncating early context.

Alternative: Use logging and manual analysis. But SSA enables automated root-cause analysis at scale.

Economic Analysis: Is SubQ Actually 1/5 the Cost?

Subquadratic's homepage claims "1/5 the cost of leading LLMs" for target workloads. Let's unpack this:

Cost Drivers in LLM Inference

Total cost per request depends on:

1. FLOPs per token: Compute cost scales with model size and attention mechanism
2. Memory footprint: KV cache size determines max batch size and GPU utilization
3. Latency requirements: Faster responses require more expensive GPUs or larger batch sizes
4. Caching: Prompt caching can amortize prefill costs for repeated prefixes

SubQ's Claimed Advantages

• 52x prefill speedup at 1M tokens: If dense attention takes 10s, SSA takes ~0.2s—reducing GPU-hours and enabling higher throughput
• Reduced KV memory: Smaller cache allows larger batch sizes, improving GPU utilization from ~30% (typical for long-context models) to ~60-70%
• Lower FLOPs: Sparse attention reduces total compute per token, allowing cheaper GPUs (e.g., A100 vs. H100) for same throughput

Back-of-Envelope Cost Comparison

Dense FlashAttention (GPT-4 class) at 1M tokens:

• Prefill: 10s × $0.01/GPU-second (H100 spot price) = $0.10
• Decode: 100 tokens × 0.05s × $0.01 = $0.05
• Total: $0.15 per request

SubQ SSA at 1M tokens:

• Prefill: 0.2s × $0.005/GPU-second (A100 spot price) = $0.001
• Decode: 100 tokens × 0.03s × $0.005 = $0.015
• Total: $0.016 per request

Ratio: $0.15 / $0.016 ≈ 9.4x cheaper

This is closer to Subquadratic's claimed 5x than 1x, but the exact ratio depends on:

• Context length distribution: If most requests are <100K tokens, SSA's advantage shrinks
• Prompt caching: If users cache stable prefixes, dense attention's prefill cost amortizes
• Quality: If SSA requires more retries to achieve desired output, cost-per-success increases

When SubQ is Cost-Effective

SubQ's cost advantage is largest when:

• Context lengths routinely exceed 500K tokens
• Requests involve fresh, un-cached contexts (e.g., novel documents per request)
• Batch sizes are high (benefiting from reduced KV memory)

SubQ is less cost-effective when:

• Most requests are <100K tokens (sparse vs. dense FLOPs converge)
• Heavy prompt caching is feasible (reducing prefill as a cost driver)
• Ultra-low latency is required (SSA's adaptive selection adds overhead)

SubQ Code: Integration with Existing Agents

Subquadratic markets SubQ Code as a drop-in upgrade for existing coding agents (Claude Code, Codex, Cursor). How does this work?

Architecture

SubQ Code likely operates as a middleware layer that:

1. Intercepts requests: When a coding agent makes an API call, SubQ Code captures the prompt and context.

2. Routes intelligently: For short prompts (<100K tokens), SubQ Code may forward to the original provider (e.g., GPT-4). For long prompts (>500K tokens), it routes to SubQ's API.

3. Auto-redirects expensive turns: If a request would cost >$X on the default provider, SubQ Code substitutes SubQ and passes the result back.

4. Caching and optimization: SubQ Code may cache SubQ responses to avoid redundant long-context processing.

Integration Example

# Original Claude Code-style agent
response = client.messages.create(
    model="claude-sonnet-4",
    messages=[{"role": "user", "content": long_codebase + task}]
)

# With SubQ Code middleware
response = subq_code_client.messages.create(
    model="claude-sonnet-4",  # Fallback if context is short
    messages=[{"role": "user", "content": long_codebase + task}],
    subq_threshold=500_000,  # Use SubQ for >500K tokens
)

Claimed Benefits

• 25% cost reduction: By routing long-context requests to SubQ instead of expensive dense-attention models
• Faster exploration: SubQ's prefill speedup allows agents to iterate more quickly on large codebases
• Seamless migration: No changes to agent prompts or workflows—just swap the client

Skepticism and Validation

Before adopting SubQ Code:

1. Measure actual cost savings: Track real-world spending before and after migration
2. Test quality on your tasks: SSA's sparsity may degrade performance on domain-specific code patterns
3. Monitor latency: Adaptive sparse selection adds overhead—ensure it doesn't hurt user experience
4. Check SLAs: SubQ is a new provider—validate uptime, rate limits, and support responsiveness

Challenges and Limitations

Despite promising benchmarks and cost claims, SubQ faces several hurdles:

1. Lack of Independent Validation

Subquadratic's benchmarks are self-reported. Independent replications by:

• AI research labs (e.g., EleutherAI, HuggingFace)
• Enterprise users (e.g., tech companies running their own evals)
• Academic institutions (e.g., Stanford, MIT)

...are needed to confirm the 52x speedup and 1/5 cost claims.

2. Missing Model Card

As of May 2026, Subquadratic has not published:

• Training data sources and curation methods
• Safety evaluations (toxicity, bias, refusal rates)
• Licensing terms (commercial use, derivative works)
• System card (compute used, carbon footprint)

This limits transparency and makes SubQ risky for regulated industries (finance, healthcare, government).

3. Retrieval Recall Trade-offs

SSA's content-dependent selection may miss relevant tokens that don't score highly on similarity—leading to:

• Silent failures: The model generates plausible but incorrect answers by missing key evidence
• Degraded reasoning: Multi-hop logic requiring distant token connections may break

Robust evaluation on diverse retrieval tasks (not just MRCR) is needed to quantify these risks.

4. Competition from Hybrid Approaches

Alternatives to SSA include:

• RAG + dense attention: Use vector search to retrieve top-K chunks, then apply dense attention over ~100K tokens
• Hierarchical transformers: Process documents in stages (summarize sections → reason over summaries)
• Mixture of Experts (MoE): Route different context regions to specialized submodels

SubQ must prove SSA is superior to these established patterns, not just viable.

Related on ExplainX

• LLM context window explained (2026) — nominal vs usable context in practice
• DeepSeek V4-Pro: benchmarks, pricing, agents — another long-context + agent economic lens
• Anthropic Claude Opus 4.7 models guide — how vendor tiers intersect coding harnesses
• Caveman token compression — context accounting when you cannot change the architecture
• Terminal-Bench 2.0 — terminal agent eval culture
• What are Agent Skills? — portable instructions across providers
• AI Models Hallucinate: Why and How to Catch It — detection and mitigation

Sources

• Homepage / product claims: subq.ai
• Product post: subq.ai/introducing-subq
• Technical SSA article: subq.ai/how-ssa-makes-long-context-practical
• Early access: subq.ai/request-early-access
• Conversation / distribution (non-spec): Alex Whedon @alex_whedon on X
• FlashAttention reference: Dao et al., 2022

Marketing ratios, benchmark conditions, and API availability change quickly. Treat this as May 6, 2026 context and reconcile numbers against Subquadratic's published tables and your own measurements before production budgets or architecture reviews.