explainx / blog
Senior SWE-Bench from Snorkel AI, Princeton, and UW–Madison evaluates coding agents like senior engineers — realistic prompts, runtime bug investigation, and taste scoring. Opus 4.8 leads at 24% tasteful solve; frontier models fail over 75% of the time.
May 27, 2026
Datacurve's DeepSWE benchmark separates frontier coding agents more sharply than SWE-Bench Pro, with GPT-5.5 leading at 70%. The bigger story is benchmark quality: verifier errors, public-history contamination, and a git-history loophole that may have inflated some Claude Opus results.
Jul 1, 2026
GeneBench-Pro measures whether AI agents can do judgment-heavy computational biology — not just run a pipeline on clean data. GPT-5.6 Sol leads at ~29–32% pass rate on problems that cost human experts thousands of dollars each. Ten questions open-sourced on Hugging Face; 50-question subset heading to Artificial Analysis.
Jun 29, 2026
A new site called Commit History went viral on X in late June 2026, ranking developers by lifetime GitHub commits the way star-history.com ranks repo stars. Peter Steinberger leads combined totals at 268,000. Pieter Levels tops exposed private commits at 161,515. The leaderboard is part brag sheet, part Rorschach test for what "shipping" means when agents write half your diffs.
SWE-Bench Pro tells an agent exactly which function to add, which file to edit, and which test must pass. Real senior engineers get a Slack message: "Google Books should be a fallback metadata source for BookWorm."
Senior SWE-Bench — from Snorkel AI, Princeton, and UW–Madison — closes that gap. Its tagline: "We treat agents like senior engineers, so why evaluate them like junior engineers?"
The first public leaderboard is sobering: Claude Opus 4.8 leads at 24.0% tasteful solve rate. Even the best frontier model fails more than three out of four senior-level tasks when correctness and code taste both count.
| Site | senior-swe-bench.snorkel.ai |
| Tasks | 50 public + 50 private from real PRs |
| Repos | PostHog (8), Electric (6), Gitea (6), Better Auth (4), Harbor (4), +7 |
| Types | Feature · bug · perf · migration |
| Stacks | Python services, Elixir, Go, SQL, TS libs, Rust, TS frontend, +4 |
| Agent harness | Mini-SWE-Agent via Harbor |
| Leader | Claude Opus 4.8 max — 24.0% tasteful pass@1 |
| Key insight | Top models miss senior-level correctness + taste on over 75% of tasks |
Senior SWE-Bench feature tasks read like natural messages — median 639 characters, zero explicit code symbols in the prompt.
Compare to a typical SWE-Bench Pro instruction: ~6,008 characters, ~39 named code symbols, function signatures, file paths, and interface tables spelled out line by line.
Senior SWE-Bench median instruction length is 31% of SWE-Bench Pro — closer to how a staff engineer actually delegates.
Because solutions vary, Snorkel introduces a validation agent: expert-designed recipes generate behavioral tests that adapt to each submitted patch — not a single golden diff.
Bug and performance tasks come from PRs that required real debugging — starting services, reading logs, profiling, reproducing flaky behavior from user reports.
This is the opposite of "apply this one-line fix from the issue description." Agents must investigate before they patch.
A tasteful solve requires all of:
| Gate | Threshold |
|---|---|
| Verifiers | Pass |
| Validation agent | Pass |
| Rubric | above 0.5 |
| Bloat | under 2× reference size |
| Practice alignment | above 2/5 |
| Relative taste | above 2/5 |
Verifiers can enforce load-bearing codebase practices never stated in the prompt — naming conventions, error-handling patterns, how sibling files structure imports.
The benchmark explicitly punishes agents that over-build, ignore local style, or ship 2× bloated diffs that technically pass tests.
Public results use Mini-SWE-Agent with model-specific effort settings. Tasteful solve rate (pass@1):
| Rank | Model | Effort | Tasteful solve |
|---|---|---|---|
| 1 | Claude Opus 4.8 | max | 24.0% |
| — | Claude Sonnet 5 | max | 19.4% |
| 2 | GPT-5.5 | xhigh | 16.0% |
| 3 | Claude Opus 4.7 | max | 14.1% |
| 4 | GPT-5.4 | xhigh | 14.0% |
| 5 | GLM-5.2 | max | 12.5% |
| 6 | Kimi K2.6 | default | 8.2% |
| 7 | Claude Sonnet 4.6 | high | 8.2% |
| 8 | Gemini 3.1 Pro | high | 6.1% |
| 9 | Gemini 3.5 Flash | medium | 3.0% |
Takeaways:
The site states plainly: the top-performing frontier models fail to complete tasks with senior-level correctness and taste over 75% of the time.
Senior SWE-Bench compares itself to SWE-Bench Pro, DeepSWE, and internal FrontierCode metrics:
| Dimension | Senior SWE-Bench | SWE-Bench Pro | DeepSWE |
|---|---|---|---|
| Instruction style | Slack-style, under-specified | Detailed spec with symbols | Concise, original tasks |
| Median prompt length | ~639 chars (features) | ~6,008 chars | ~half of SWE-Bench Pro |
| Files per feature task | ~11 avg | ~5 avg | ~5 avg |
| Horizon | Hundreds of agent steps | Shorter patch scope | Long-horizon (668 LOC ref) |
| Scoring | Correctness + taste + validation agent | Verifier pass | Behavior verifiers |
| Opus 4.8 tokens/task | 117.1K output (features) | — | 97.0K (self-reported) |
Tasks span libraries to multi-service applications, authored by engineers with hundreds of commits in each repo — not synthetic toy repos.
50 public tasks are released for community eval; 50 private tasks hold out contamination. Dataset lives on GitHub; runs go through Harbor (same framework as Terminal-Bench 2.0).
The Senior SWE-Bench site contrasts prompt styles side by side.
SWE-Bench Pro (excerpt — Open Library / BookWorm Google Books task):
STAGED_SOURCES), exact URL templates, function signatures (stage_from_google_books, fetch_google_book, BaseLookupWorker)Senior SWE-Bench equivalent:
#eng-platform
Engineer 10:42 AM
Google Books should be a fallback metadata source for BookWorm
for fallback/staging imports.
Same underlying feature. One prompt is a contract; the other is a message. Senior SWE-Bench argues agents that ace the contract may still fail the message — and production work looks like the message.
Static golden tests break when valid implementations differ. Senior SWE-Bench's validation agent:
This mirrors how a senior reviewer would QA a PR — not just "CI green," but "would I merge this?"
Combined with verifiers (runtime tests) and taste metrics derived from observed codebase practices, the benchmark separates lucky passes from shippable work.
Do not read 24% as "Opus is bad." Read it as "senior work is hard and our old benchmarks were measuring something easier."
Practical guidance:
For agent loop design — how to structure multi-step coding sessions — see loop engineering for Claude Code.
Sources: Senior SWE-Bench, Snorkel AI, Harbor framework. Leaderboard and task stats accurate as of July 2, 2026.