Opus 5 on SlopCodeBench: 24% Strict Pass, Still Can't Run Lights-Off

Most coding benchmarks show a model the entire problem on turn one. SlopCodeBench — a March 2026 benchmark from Gabe Orlanski's lab at UW Madison — does the opposite: it reveals requirements one checkpoint at a time, the way real feature work actually arrives. That distinction is what let humanlayer's dhorthy run Claude Opus 5, Opus 4.8, and Sonnet 5 through 17 checkpoints on July 25, 2026, and produce the first real evidence for a claim explainx.ai has tracked across the Opus 5 launch cycle: today's frontier models still cannot be trusted to maintain a codebase lights-off.

The short version: Opus 5 won on strict pass rate (24% vs 6% for the other two) but also wrote roughly 5x more functions than Opus 4.8 over the same three problems. Whether that's "better engineering" or "more slop that happens to compile" is exactly what the numbers below let you judge for yourself.

TL;DR — what the run actually found

What SlopCodeBench actually changes about benchmarking

Benchmarks like SWE-bench and Frontier Code test whether a model can solve a well-specified problem once. SlopCodeBench tests something closer to how software actually gets built: a spec that evolves. Each challenge ships as a sequence of checkpoints — circuit_eval starts with a CLI that validates single-bit circuit files, then checkpoint 3 changes every signal into a multi-bit vector with new operators and width checks, and later checkpoints add three-valued logic, new file formats, and an optimizer pass. The model never sees checkpoint 3 while writing checkpoint 1.

That structure is the point. A model that hardcodes assumptions from checkpoint 1 — say, that every signal is a single bit — pays for it three checkpoints later when vectors show up and break code it already "finished." This is the same failure mode explainx.ai flagged when comparing Claude Code's model-vs-effort tradeoff: a model that tries harder on the current task without anticipating how the codebase needs to bend later still loses to a model that reasons about the shape of future change.

Scoring uses a strict pass: a checkpoint only counts if every new black-box test passes and every regression test inherited from earlier checkpoints still passes. Introduce a defect at checkpoint 4, and checkpoints 5 through 8 fail automatically for that problem — even if the model's code quality improves afterward. In humanlayer's 9 runs, that recovery never happened.

The numbers — strict pass and where each model broke

humanlayer picked three problems spanning difficulty tiers: circuit_eval (easy, 8 checkpoints), database_migration (medium, 5 checkpoints), and dynamic_config_service_api (hard, 4 checkpoints) — 17 checkpoints total, run against all three models in parallel with a fresh context window per checkpoint.

For context, the original SlopCodeBench paper reported GPT-5.4 at 11% and Opus 4.6 at 17% strict pass rate on its full problem set — numbers that made the benchmark's authors call it unsaturated at the time. Opus 5's 24% on this smaller subset is a real improvement over that baseline, and it's consistent with the incremental gains explainx.ai covered in Opus 5's ARC-AGI-3 result — better, not transformative.

The more interesting detail is where Opus 5 broke: three clean checkpoints in a row on circuit_eval, then a defect at every single checkpoint afterward. That pattern — strong early, then compounding failures — is exactly what strict-pass scoring is designed to expose, and it's why "24%" undersells how close Opus 5 came to a genuinely clean run on the easy problem.

Is "more functions" good engineering or just more slop?

This is where HN's discussion of the source post got interesting, and where the numbers cut against a simple "Opus 5 won" narrative.

Opus 5 wrote roughly 5x more functions than Opus 4.8 across the same challenges. Total source volume was 29,065 lines for Opus 5 against roughly 9,000 each for Opus 4.8 and Sonnet 5. But a large share of that gap is tests, not production code:

Strip out tests and Opus 5's production-code volume is closer to 1.8x Opus 4.8's, not 5x. Whether that's warranted depends on whether you think the extra tests bought correctness — and the strict-pass numbers suggest they did, partially: Opus 5's early checkpoints held up better precisely where the other two didn't have coverage to catch regressions.

The function-count story flips again on how those functions get reused. Opus 5's functions were single-use only 14.9% of the time — meaning most of its ~2,000 functions get called from multiple places, a shape consistent with decomposing logic rather than writing one giant function per feature. Opus 4.8 sat at 49.1% single-use, and Sonnet 5 at 71.5% — the highest of the three, meaning Sonnet wrote the most one-off, throwaway-looking functions relative to its total.

None of these numbers settle "good engineering" on their own. Lots of small, reused functions can mean clean decomposition or it can mean fragmented indirection that's harder to trace. That ambiguity is the honest state of the art here — deterministic metrics tell you what changed, not whether it's good.

Complexity and duplication — where the models actually diverged

The clearest quality signal in the run isn't function count — it's how each model handled growing requirements over the 8 checkpoints of circuit_eval.

Opus 4.8's duplication jump has a clear trigger: checkpoint 3 is where circuit_eval turns every signal into a vector (data[7:0] instead of data), adding slicing, indexing, and width checks. That's the point where a design that assumed single-bit signals everywhere starts fighting the new requirements, and Opus 4.8's response was apparently to copy-paste rather than refactor — its duplication inflection lines up exactly with that checkpoint.

Sonnet 5 and Opus 5 both avoided that trap, but by different means: Sonnet leaned on far more single-use functions (a "write a new small thing" strategy), while Opus 5 kept complexity low despite writing roughly 2,000 functions — a tradeoff between many small pieces and a few large ones that shows up clearly when you plot function count against mean complexity per model.

The honest read: every model's code got measurably worse — more verbose, more rule-violating — as the checkpoints piled up. The percentage of lines flagged as "too verbose" by SlopCodeBench's rule set rose from roughly 65% to 80% between checkpoint 1 and checkpoint 8, across all three models, Opus 5 included. That's the closest thing in this run to hard evidence for the year-old claim that AI-generated codebases degrade in quality as they age — not vibes, a repeatable measurement.

Every dollar bought correctness. Nobody bought enough of it.

humanlayer's own framing of the cost-vs-defects chart is worth repeating verbatim, because it's the single sentence that captures what this run actually shows: spend correlated with fewer defects across all nine runs, but no amount of spend got any model to a clean finish on any problem — not even the one labeled "easy."

That matters for anyone weighing effort vs model choice in a coding agent harness: throwing more inference budget at a task is not a substitute for a model that reliably avoids compounding early mistakes. Sonnet's own cost curve reinforces this — its first checkpoint ran more expensive than the other models', but once the initial build settled into maintenance mode, Sonnet became the cheapest of the three, suggesting cost efficiency shows up after the foundational work is done, not during it.

The TypeScript slop-density experiment — read with caveats

humanlayer went a step further and had Opus 5 port roughly 76 of SlopCodeBench's 200+ Python quality-detector rules to TypeScript, then ran them against humanlayer's own production monorepo as a sanity check. The result: their hand-reviewed TypeScript codebase scored 15.06 findings per thousand lines, versus 174.88 (pooled) to 178.88 (final snapshots) for Opus 5's lights-off SlopCodeBench output — roughly an 11.6x to 11.9x difference in slop-rule density.

Treat this one carefully. It's 76 ported rules against a 200+ rule Python original, applied to a different language and a different codebase with different review processes — the rule parity was never verified, and a production monorepo that's been through months of human review is not a fair apples-to-apples baseline against a single lights-off benchmark run. It's directionally interesting, not a controlled experiment.

What HN actually argued about

The comment threads on the source post split roughly three ways, and none of them fully resolve:

Is Opus 5 a real step up or an incremental release? Some commenters reported replacing Opus 4.8 at high effort with Opus 5 at medium effort and getting comparable or better results with fewer tokens — consistent with the pricing story in explainx.ai's Opus 5 launch coverage. Others said it felt "not revolutionary" compared to how Fable 5 felt at release, and a few said they preferred earlier Opus checkpoints entirely. Both readings are consistent with a 24% vs 17% baseline improvement — real, but not the kind of jump that ends an argument.

Is 200 rules flagging 89-98% of all lines a sign of bad code, or bad rules? When a rule set flags the overwhelming majority of lines across every model tested, that's either strong evidence the models write systematically low-quality code, or evidence the rules are calibrated too aggressively to be discriminating. Both can be true at once, and the run doesn't distinguish between them — which is a fair critique of slop-detection tooling broadly, not just this benchmark.

Is strict pass too harsh a scoring rule? Requiring zero regressions across every future checkpoint means one early mistake caps a model's entire score for a problem, regardless of how it performs afterward. That's a defensible design choice for measuring "can this ship unattended," but it does mean the headline percentages understate how much of the later work each model actually got right in isolation.

What's next — the handoff experiment

The most interesting unresolved idea from the source post is a handoff test: have a frontier model like Fable 5, GPT-5.6 Sol, or Opus 5 build checkpoints 1 through 7 of a challenge, then hand the resulting codebase to a smaller model — Sonnet 5 or GPT-5.6 Terra — to implement checkpoint 8. If the smaller model can't extend the frontier model's codebase cleanly, that's a stronger signal that the "smart" model didn't actually leave behind a maintainable structure, independent of whether the frontier model would have nailed checkpoint 8 itself.

That framing amplifies exactly the gap explainx.ai has been tracking since Claude Code's model-vs-effort piece: a model's raw capability and the maintainability of what it leaves behind are different axes, and most benchmarks only measure the first one.

Summary

humanlayer's July 25, 2026 SlopCodeBench run gave Opus 5 a 24% strict pass rate against 6% for both Opus 4.8 and Sonnet 5 — a real improvement, and consistent with gains explainx.ai has tracked on ARC-AGI-3. But no model finished any of the three challenges without at least one defect, Opus 5 wrote roughly 5x more functions than Opus 4.8 (about 1.8x more production code once tests are excluded), and every model's code quality measurably degraded across checkpoints. Treat this as one team's 17-checkpoint subset, not a definitive verdict — the deterministic quality metrics are repeatable, but the link between any single metric and real-world maintainability isn't established yet.

Related on explainx.ai

• Claude Opus 5 launch — near Fable 5 at half the price
• Opus 5 hits 30.2% on ARC-AGI-3 — what the jump means
• Claude Code model vs effort — knowing more vs trying harder
• Claude Opus 5 top 10 use cases
• Developers migrate to Opus 5 fast mode
• AI benchmarks — complete guide
• Senior SWE-bench — Snorkel AI coding agent benchmark
• Loop engineering for coding agents

Official sources: SlopCodeBench paper · SlopCodeBench runner (SprocketLab) · SlopCodeBench problems (gabeorlanski)

Benchmark numbers and methodology accurate as of the source run on July 25, 2026, covering 3 of SlopCodeBench's problems and 17 checkpoints total across three Claude models. This is a single team's subset run, not the full benchmark suite — verify against the original paper and independent replications before treating these percentages as a general verdict on any model's coding ability.