Fuzzilli vs BigSleep: Project Zero Finds AI Agents and Fuzzers Tie on V8 Bugs

Google Project Zero has published the first large side-by-side of a classic fuzzer and an AI bug-hunting agent on the same target. In a post dated October 27, 2026, Project Zero compares Fuzzilli, Google's generative JavaScript engine fuzzer, with BigSleep, an LLM-based agent that reviews new commits and writes proof-of-concept testcases, both pointed at Chrome's V8 engine. The verdict is roughly a tie, with surprisingly little overlap, and the authors conclude that "fuzzing and LLMs are still complementary approaches."

That matters beyond Chrome. AI vendors are selling vulnerability agents (see our coverage of OpenAI's Daybreak red and blue tiers and Cloudflare's Daybreak rebuild), and security teams keep asking whether an agent replaces the fuzzing they already run. Project Zero now has data.

TL;DR

Agents that act on your behalf need rules. Agentbeam stops the dangerous actions →

What was compared

Fuzzilli runs continuously across many instances and usually picks up new V8 changes within hours. BigSleep runs once a day, reviews new commits and writes testcases; its automated pipeline went live in April 2026. Both report through ClusterFuzz into Chrome's issue tracker, which gave Project Zero a common ledger.

The raw numbers hide the interesting part. Fuzzilli produces more vulnerability-flagged reports, but after removing duplicates and non-issues, the agent lands slightly ahead. The authors also say cost per bug cannot be compared, because neither system is tuned for cost.

How much do they overlap?

Two jars of captured bugs beside a small bar chart, showing Fuzzilli vs BigSleep V8 bug overlap

In production, only 20 of the 81 actionable reports, about 25 percent, were found by both. A slight majority were found first by Fuzzilli. The authors note this probably undercounts overlap, because of how duplicates were matched through ClusterFuzz and fix commits.

So they ran an experiment. Starting from about 1,150 historic Fuzzilli bugs (early 2023 to mid-2026), they narrowed to 132 reproducible on a standard debug build with only a few flags, and 112 after deduplication. For each, BigSleep reviewed the commit that likely introduced the bug, 25 attempts per commit, using the production setup with Gemini 3.6 Flash. A testcase counted only if it crashed the pre-fix build but not the fixed one.

• About 62 percent of bugs were rediscovered.
• About 12 percent of commits yielded a different, already fixed bug.
• About 26 percent yielded nothing.

Follow-up runs with 5 attempts per bug and varied prompts found 10 more, bringing the total to about 70 percent. The authors caution that tweaking the fuzzer could raise its numbers too.

Why is production overlap (about 25 percent) so much lower than the experiment (about 60 to 70 percent)? The post suggests a limited rediscovery window after a fix, different builds and flags, pipeline outages that skipped commits, and several model upgrades over the period.

Is it memorizing?

The obvious objection is that an LLM trained on public bug trackers might simply recall old bugs. Project Zero tested for it. With a model knowledge cutoff of March 2026 and an effective cutoff around January 2026 for the Fuzzilli bugs, they found no statistically significant memorization for pre-2026 bugs. They recommend re-checking with future models, since cutoffs will move past more of the bug history.

They also found that bugs slow for Fuzzilli to find were slightly more likely to be hard for BigSleep. The effect was statistically significant but small, which the authors read as a reason to keep fuzzers running rather than a sign that the two fail in the same places.

Where each approach wins

The post walks through three bugs. Two came from BigSleep:

1. Chromium bug 449760249 (October 2025). A hole-check elision issue for temporal-dead-zone variables. A labeled break inside a do-while, aimed at an outer switch, stops the loop builder from being notified. The original bug was found by Fuzzilli two years earlier; the agent found a variant by reasoning through the patch and testing hypotheses. A fuzzer would struggle because coverage feedback does not reward combining those features.
2. Chromium bug 443765373 (September 2025). A handler offset stored in a 28-bit field covers 256MB, but bytecode can reach 512MB. The proof of concept generates a function with about 500,000 yield* statements so the offset truncates. Fuzzers rarely produce such huge, slow testcases.

The third came from Fuzzilli: Chromium bug 40076141 (November 2023) in the Maglev compiler, where two individually correct optimizations interact, one inserting an unchecked integer untag and another removing a checked one. When a value leaves the small-integer range, memory corruption follows. It was never rediscovered in any experiment.

Overlap was also noticeably smaller for JIT compiler bugs than for the rest of the engine. The authors hypothesize, without testing it, that bugs which are both non-local and need complex testcases are hard for both approaches.

What this means for security teams

For defenders, the practical message is additive: an agent finds a different set of bugs than your fuzzers, so adding one does not make the other redundant. Two small corollaries follow.

• If you already have fuzz targets, an agent that reads commits is a cheap second net, especially for logic bugs and new language features.
• If you have neither, the result suggests starting with whichever is easier to run in your pipeline, then adding the other. Free options exist: see Theorem OSS Verify and Anthropic's open-source scanner, plus the open-source security audit skill from Cloudflare.

The pressure behind this is real: we covered Anthropic's warning that a vulnerability tsunami will get worse before it gets better. Attackers can run agents too, so defenders benefit from catching bugs in the same commit window, which is exactly what a daily commit-review agent does.

How to run both in practice

The post does not give a deployment recipe, but its design suggests one. Treat the two systems as stages that feed the same triage queue.

1. Continuous fuzzing on every build. Fuzzilli-style generation catches shallow and non-local bugs within hours of a change landing. Keep it cheap and always on.
2. A daily commit-review agent. BigSleep reads the day's commits, forms a hypothesis about what the change might break, and tries to prove it with a crashing testcase. A crash on the pre-fix build that vanishes after the fix is the filter Project Zero used for credit, and it is a good rule for any team: no reproducible crash, no report.
3. One deduplicating tracker. Both tools filed through ClusterFuzz, so duplicates and fixed-by-the-same-change reports collapsed automatically. Without that step, the 441 raw agent reports would have buried reviewers.
4. Rediscovery as a regression test. The 112-bug set is a ready benchmark. Replaying old bugs against a new model or prompt shows whether your agent got better, without waiting for fresh bugs.

What people will ask next

Does this prove AI agents are as good as fuzzers? No. It shows that on one target, with one agent and fuzzer, the actionable counts are comparable and the sets are different. The authors themselves note equivalent fuzzer tweaks could raise discovery rates.

Why do raw reports differ so much from actionable ones? Many agent reports were duplicates, intended behavior, or lacked a fixing commit. Project Zero's raw counts were 316 and 441, and only 37 and 45 ended up as actionable vulnerabilities, which is a reminder that triage cost is part of any tool's real price.

Will the gap close? The post notes that the BigSleep model was upgraded several times in the window, that earlier models likely performed worse, and that future models are expected to perform better. It also finds that bugs hard for a fuzzer are likely hard for an agent too, so the authors suggest keeping fuzzers running for now.

Caveats worth keeping

• The numbers cover V8 only, and one fuzzer and one agent. Other targets and tools may differ.
• BigSleep's model was upgraded several times in the window, so the 45 is not from a single fixed system.
• The rediscovery set was limited to bugs reproducible with a simple build, which favors bugs an agent can trigger from d8.
• Only one commit per bug was given to the agent in the experiment, and the commit identification was sometimes imperfect, which the authors say skews the comparison against BigSleep.

Related reading

• Anthropic's Claude exploit chains, including a four-bug Chrome chain

• OpenAI GPT-5.6-Cyber and Daybreak tiers

• Cloudflare and OpenAI Daybreak rebuild

• Anthropic open-source vulnerability scanner

• Theorem OSS Verify

• Anthropic on the vulnerability tsunami

• Sources: Project Zero post, Fuzzilli on GitHub. Figures are as published by Project Zero on October 27, 2026.