FrontierCode 1.1 shows that the best AI coding agents still miss more than 40% of the time when the bar is "would a maintainer merge this?" On October 12, 2026, Artificial Analysis published its first full results for the benchmark, which Cognition built with maintainers of real open-source projects. Claude Opus 5.5 leads at 52%, GPT-6.1 Sol and Claude Fable 5.1 tie at 49%, and nine models from six labs sit between 43% and 52%.
That is a tight cluster at the top, and a very different picture from the saturated pass-rate numbers on older coding tests. The more useful story is in the details: what the tasks look like, how strictly they are graded, and how wildly cost per task varies for nearly identical scores.
FrontierCode 1.1 at a glance
| Question | Answer |
|---|---|
| Who built it? | Cognition, with maintainers of 36 open-source repositories |
| What does it test? | Whether an agent's change to a real repo is something a maintainer would merge |
| How big is it? | 150 tasks in total; Artificial Analysis runs the 100-task Main set |
| Who leads? | Claude Opus 5.5 (max with fallback): 52% score, 58% pass rate |
| Cheapest strong result? | GPT-6.1 Sol (max): 49% at $0.81 per task |
| Best open weights? | MiMo-V2.6-Pro at 45% |
| Unsolved tasks? | 9 of 100 tasks passed by no model on any attempt |
| Public tasks? | No, private to avoid contamination |
What FrontierCode actually measures
Most coding benchmarks ask a narrow question: do the hidden tests pass? Cognition's launch post argues that functional correctness is now the baseline and the next question is whether a model writes code maintainers would merge. It also points to METR research suggesting that high-scoring models often produce patches human maintainers would reject.
To get at that, Cognition worked with maintainers of 36 flagship open-source repositories. According to the post, maintainers each spent more than 40 hours per task defining what "mergeable" means for their codebase. Every task then went through an adversarial "hack report", calibration against four sample solutions, pod-level review, and a final check by a Cognition researcher.
The prompts are shorter and less prescriptive than those in SWE-Bench Pro, so the agent has to work out scope and approach the way a contributor would. One maintainer, Tomer Nosrati, is quoted in the post: "Where others grade like a CI, FrontierCode grades like a tech lead."
How grading works
Cognition describes three techniques that go beyond a plain test run:
- Reverse-classical tests: tests the agent writes must fail on the broken base commit, so they actually exercise the bug.
- Scope constraints: limits on files touched and diff size, so a model cannot "fix" a bug by rewriting half the repo.
- Adaptive classical grading: an LLM aligns reference tests with a valid but different implementation, so a correct alternative design is not punished.
Rubric items are either blockers or non-blockers. Fail any blocker and the task scores zero. Pass all of them and the score lands between 70% and 100%, with non-blocking criteria (style, conventions, code quality) moving it within that band.
How Artificial Analysis ran it
Artificial Analysis evaluated FrontierCode 1.1 Main, the 100-task set, in its open-source Stirrup harness with a cap of 250 turns per attempt. Each model gets three attempts per task. Task scores are averaged across attempts and then across tasks with equal weight.
The judge for the LLM review portion is GPT-6 Luna (medium). Attempts that Cognition's trajectory scanner flags as looking up the solution score zero. The version 1.1 update, released in July, added rules for fair internet use for exactly this reason.
One caveat matters when comparing with other leaderboards: results depend on the harness. A model scored in Stirrup is not necessarily scored the way it would be inside its vendor's own agent product, and the numbers here are not interchangeable with Cognition's own leaderboard or with third-party boards. Artificial Analysis also notes the live leaderboard has newer entries than the launch article.
The results: a tight top, a long tail
Per Artificial Analysis:
| Model | Score | Cost per task |
|---|---|---|
| Claude Opus 5.5 (max with fallback) | 52% | $10.15 |
| GPT-6.1 Sol (max) | 49% | $0.81 |
| Claude Fable 5.1 (max with fallback) | 49% | $12.00 |
| MiMo-V2.6-Pro | 45% | $0.18 |
| DeepSeek V4.1 Flash (max) | 44% | n/a |
| Kimi K3 (max) | 44% | n/a |
| GPT-6 Luna (max) | 42% | $0.09 |
Pass rates range from 58% for Opus 5.5 down to 16% for Muse Glimmer (high). The headline sentence from Artificial Analysis is blunt: "Leading models still fail more than 40% of the time."
Two other facts stand out. Nine of the 100 tasks are passed by no model on any attempt, and no task is passed by every model on every attempt. That means the benchmark still has headroom at the hard end, and that model differences are real rather than noise around a shared ceiling.
Cost per task: where the real decision is
The cost spread is the part most useful to anyone choosing a model for a coding agent. Artificial Analysis measured $0.09 per task for GPT-6 Luna at max effort and $12.00 for Fable 5.1, more than a 100x gap.
On the Pareto frontier of score versus cost, four models stand out:
- GPT-6 Luna (max): 42% at $0.09
- MiMo-V2.6-Pro: 45% at $0.18
- GPT-6.1 Sol (max): 49% at $0.81
- Claude Opus 5.5 (max with fallback): 52% at $10.15
Moving from Sol to Opus 5.5 buys three points for more than 12x the cost. Sol also scores two points above GPT-6 Astra (max), which costs $4.22 per task. This lines up with our earlier look at GPT-6.1 Sol's cost efficiency versus Astra, where per-task cost, not list price, was the number that mattered.
Two branching lines merging at a green dot, representing a coding agent change being merged into a repository in the FrontierCode benchmark
Why tokens, not price, drive the bill
Output tokens explain much of the gap. Artificial Analysis reports roughly 27k output tokens per task for GPT-6 Astra (max) and about 33k for GPT-6.1 Sol (max), against about 157k for Opus 5.5 and about 236k for Claude Sonnet 5.5, the most of any model tested.
Sonnet 5.5 (max with fallback) costs $8.26 per task. It uses about 50% more output tokens than Opus 5.5 yet costs less per task, so token counts alone do not predict the bill. If you are budgeting an agent fleet, per-token pricing alone will mislead you. See also how a cache-read price change reshapes agentic cost math for Sonnet 5.5.
Time per task varies too: about 3 minutes for Inkling (xhigh) up to about 46 minutes for Qwen3.8 Max (0902). For interactive use, that is a bigger factor than a point or two of score.
What this means for people building with coding agents
1. A one-point lead is not a recommendation. The top nine models span 43% to 52%. With 100 tasks and three attempts each, differences of a point or two should be read as ties. Choose on cost, latency, and fit with your stack.
2. Cheap models are closer than the sticker suggests. A 42% score at $0.09 per task, or 45% from an open-weight model at $0.18, is a strong default for high-volume work such as triage, small fixes, and first-draft patches. Reserve the $10 tier for the hard tasks. For a broader comparison of budget models, see our small models comparison.
3. Merge-readiness is a different skill from passing tests. Scope discipline, matching project conventions, and not touching unrelated files are what separate a mergeable patch from a correct one. If your agent workflow does not enforce diff size and scope checks, you are likely seeing the gap this benchmark was built to expose.
4. Keep humans in review. More generated code means more to review. Our write-up of the Harvard and Jellyfish study on AI coding agents found review capacity, not generation, is the bottleneck. FrontierCode's 40%-plus failure rate is a reminder that review is not optional.
A pair of reviewers marking a sheet with a green check, illustrating maintainer review of AI-written patches on FrontierCode
Limits and open questions
The benchmark has real strengths, but readers should hold a few caveats in mind.
- Private tasks. Keeping tasks secret limits contamination, but it also means outsiders cannot inspect individual tasks or reproduce results independently.
- Self-reported quality claims. Cognition claims 81% fewer misclassification errors than other leading benchmarks, based on its own analysis. That has not been independently validated in anything we found.
- Rubric subjectivity. Maintainer judgment encodes taste. Different maintainers could weigh style or scope differently.
- A vendor-built benchmark. Cognition makes coding agents. The structure, with outside maintainers and a third-party runner like Artificial Analysis, is a mitigation, not a guarantee of neutrality.
- Harness effects. Scores depend on the Stirrup harness, turn caps, and the LLM judge, so they do not transfer one-to-one to your own agent setup.
- Small sample. One hundred tasks across 36 repositories means task selection can shift rankings.
Note also that the first launch results Cognition published in June covered older models, with Claude Opus 4.8 at 51.8% on the full 150-task Extended set and far lower on the hardest Diamond subset. Subsets, model versions and harnesses all differ across sources, so compare scores only within one run.
How this fits the broader coding-benchmark picture
Older benchmarks have trended toward saturation, which is why labs and evaluators keep building harder ones. We covered a similar move with SWE-Game, which grades coding agents on Godot games, where even Opus 5 stayed below 60%. And on general capability, Claude Opus 5.5 topped the Epoch Capabilities Index. FrontierCode adds a different axis: not "can it solve it" but "would I accept it".
Cognition is also pushing on agent infrastructure itself; see our look at Devin's open memory spec for how the company thinks about long-running agents.
What to watch next
Artificial Analysis says it plans to fold FrontierCode into its Coding Agents leaderboard. Expect more entries as new models ship, and watch whether the nine unsolved tasks fall. The first model to crack several of them would be a more meaningful signal than another point at the top.
If you run coding agents in production, the practical step is to build a small private set of your own repo's real issues, with your own merge criteria, and score candidates the way FrontierCode does: hidden tests, scope limits, and a short rubric of what your reviewers would block.
Related reading
- GPT-6.1 Sol vs Astra cost per task
- Claude Opus 5.5 tops the Epoch Capabilities Index
- SWE-Game: coding agents on Godot games
- AI coding agents: more code, not more software
- Small models compared: Haiku 5.5, GPT-6 Luna, DeepSeek V4.1 Flash
- Claude Sonnet 5.5 cache-read price cut
- Cognition Devin memory spec
Sources: Artificial Analysis FrontierCode article and Cognition's Introducing FrontierCode.
Scores and details are as of October 12-13, 2026. Artificial Analysis's live leaderboard may show newer results.