Build-A-Chip-Bench: Best AI Agent Scores 16.6% on End-to-End Chip Design

ChipAgents and NVIDIA have released Build-A-Chip-Bench, a benchmark that asks an AI agent to build a digital chip from a plain specification all the way to a verified GDS layout. The best result among five frontier models was 16.6%, from Claude Fable 5.1. GPT-6 Astra, which had the highest functional pass rate, scored 8.7%. Per the ChipAgents research page, the lesson is less "AI can't design chips" than "logic that works is far from a chip you can build." For the related industrial result, see our post on NVIDIA NeMo Gym and Siemens.

Quick reference

How the benchmark works

Each task gives the agent a workload specification and constraints. The page's example is an inference accelerator for Qwen3-0.6B at INT4 precision, built on the open Nangate45 library at 400 MHz, with a die area of at most 6 mm2, power under 10 W and decode throughput of at least 15 tokens per second. Tools and the process design kit are open source.

Verification is hidden. Each task has a verification IP and a grader running in separate containers that the agent cannot inspect, which prevents an agent from tuning to the tests.

Scoring uses two hard gates and a quality term:

1. Gate 1, functional: the design passes held-out tests against the hidden verification IP.
2. Gate 2, GDS: the design completes the physical flow to a routable, hold-clean, DRC-clean layout.
3. PPA: power, performance and area are scored from 0 to 100% only for designs that pass both gates.

The overall score averages, across tasks, the product of the two pass flags and the PPA score. A failing design scores zero. That is why a model can look strong on one column and weak overall.

The results table

These numbers are from ChipAgents' own page, with NVIDIA contributing to the suite. They have not been independently reproduced.

The page cautions that PPA is "not meaningful on its own" because it is averaged only over designs that pass. Read it with the pass columns. The press release summarizes the same pattern: tested agents passed functional verification on about 40% of tasks on average, while about 10% met the required PPA targets.

A code frame with a row of passing green dots, standing for the pass gates in Build-A-Chip-Bench

Why GPT-6 Astra loses despite 85% functional

The most interesting row is GPT-6 Astra. It has the best functional and tied-best physical pass rates, yet its PPA score is 10.2% against Fable 5.1's 25.5%. In plain terms, Astra gets more designs to work and to route, while Fable 5.1 produces designs that come closer to the area, power and throughput targets. Multiply it out and the better optimizer wins. If you are choosing a model for hardware work, that is a different axis from a general coding leaderboard; our GPT-6 Astra vs Claude Fable 5.1 comparison covers the general picture.

The authors also note a failure mode in their own words: "Agents rarely treat keeping existing behavior intact as a constraint while they optimize." In other words, an agent that shrinks a design may quietly break what already worked. That is a classic hazard in hardware, where a late change can invalidate verification.

What a chip flow involves, and where agents stumble

The announcement lists the stages the benchmark spans: architecture, microarchitecture, RTL, functional verification, logic synthesis, floor planning, placement, clock-tree synthesis, routing, timing and power signoff, and physical verification. For readers outside hardware, a quick translation helps.

• RTL is the code that describes what the chip does cycle by cycle. Language models are comparatively good at this part, which is why older benchmarks stopped here.
• Synthesis and floor planning turn that description into gates and decide roughly where blocks sit on the die. Choices here set the area and power budget.
• Placement, clock-tree synthesis and routing put the gates in exact positions and wire them together while keeping signals on time. Timing violations show up here and often send the design back to an earlier step.
• Signoff and physical verification check timing, power and manufacturing rules (DRC). A single violation means the layout cannot be built as is.

Each stage depends on the last, and an error early on can surface hours of compute later. That long chain is what the benchmark tries to capture, and it is also why Wang describes the challenge as carrying "engineering intent" across many connected decisions.

Questions readers ask

Could this become a standard like SWE-bench? It might, if the tasks and grader are released. For now the leaderboard is promised, not live.

Does a low score mean chip companies are safe from automation? No. The benchmark measures fully autonomous work. Assisted workflows, where engineers steer an agent through parts of the flow, are a different and more common use.

Why do closed models lead? The page says closed frontier models generally work through the whole flow. Plausible reasons include longer tool-use horizons and better planning, but the authors do not test why.

What the quotes say

ChipAgents founder and CEO William Wang said in the announcement: "The real challenge is carrying engineering intent through a long sequence of interconnected decisions." NVIDIA's Tim Costa, vice president and general manager for industrial and computational engineering, said chip design is "a defining test of AI's ability to make decisions that hold up" in the physical world.

Is 16.6% good or bad?

It depends on what you compare it to. Some context:

• Versus human engineers: unknown. The page gives no human baseline, and one would need a team and months.
• Versus other agent benchmarks: the scoring is deliberately unforgiving because failing a gate zeroes the task. A model that is 65% functional and 75% physical still lands at 16.6% after PPA.
• Versus assisted design: narrower tasks fare much better. The Siemens and NVIDIA agent result we covered moved from 55% to 95% on a defined task with training in a gym environment.
• Versus model tiers: the spread from 16.6% to 0.0% across five models shows this benchmark separates systems clearly, which is what a new benchmark wants. Smaller or open models (Gemini 3.8 Flash, Kimi K3, GLM-5.3) mostly fail the physical gate.

One caveat on the closed versus open gap: the page says "closed frontier models are generally able to work through the whole silicon design flow," while the weaker rows show how far open models trail here. Our earlier coverage of Kimi K3 shows strength elsewhere, so this is a domain-specific gap, not a general verdict.

Caveats before you cite this

1. The benchmark's creator ran the tests. ChipAgents sells chip-design agents, so it has a commercial interest in the field's attention. NVIDIA's contribution to the rubric helps, but the results are not independent.
2. No task count. Without it you cannot judge noise. A handful of tasks would make the percentages coarse.
3. One harness. All models ran under OpenCode. A different agent scaffold could change the ranking. See our OpenCode upgrade guide for how that harness works.
4. Not public yet. No code or data release is stated, so nobody can reproduce it today.
5. One report names the top model differently. A secondary summary attributed the top score to a model name; the research page itself lists Claude Fable 5.1, which is what we use.

What this means for people who build

• If you work in EDA or hardware, use this as a map of failure points. The jump from "works" to "meets PPA" is where agents lose points, so put your evaluation effort there.
• If you build agent products, note how a multiplicative score exposes weak links. Evaluate each stage and the product, not only the final output.
• If you track model choice, the chip-design ranking differs from coding rankings, so test on your own workload. Our Fable 5.1 vs Opus 5.5 comparison is a useful companion for general tasks.
• If you follow policy and supply chains, autonomous design is part of why labs and chip makers are tying up; see how OpenAI and Synopsys are approaching the same problem.

What to watch

The leaderboard ChipAgents says is coming, any release of tasks or grader code, results from additional models, and whether independent groups replicate the ranking. We will update this post if the data opens up.

Related reading

• NVIDIA NeMo Gym and Siemens chip-design agent
• OpenAI and Synopsys chip design
• GPT-6 Astra vs Claude Fable 5.1
• Claude Fable 5.1 and Mythos 5.1 launch
• Kimi K3 open weights
• OpenCode 2.0 guide
• Fable 5.1 vs Opus 5.5

Numbers are from ChipAgents' research page and press release as of October 14, 2026 and have not been independently reproduced.