EEBench: The Benchmark That Grades Whether AI Can Design Circuits

When OpenAI showcased GPT-6 Astra working inside KiCad on a printed circuit board, the demo racked up attention as evidence that AI could now do hardware engineering. It also raised a question the atopile team had already been quietly working on: how do you actually measure whether an AI-generated circuit is any good, rather than just watching it click around a CAD tool convincingly?

Their answer, published September 4, 2026 as "Can AI design circuit boards yet?" on eebench.org, is EEBench — a benchmark that grades AI circuit designs by simulating them in SPICE against real manufacturer component tolerances, not by checking whether a schematic looks plausible. It hit roughly #1 on Hacker News with 152 points and 94 comments, and the leaderboard it published contains a genuine surprise: OpenAI's models, strong on general coding benchmarks, land near the bottom.

TL;DR

Why a GUI CAD tool was the wrong benchmark

The atopile team's starting observation is worth sitting with: models already know more electronics than their CAD-tool output shows. They've read the textbooks, the datasheets, the application notes, the reference code. The bottleneck isn't knowledge — it's operating a graphical tool. A huge share of an agent's context in a GUI benchmark goes to tracking coordinates, menu state, and screen position, none of which measures electrical understanding.

EEBench sidesteps that entirely by using atopile instead of a GUI. Circuits are defined in a declarative language called ato — components, connections, and electrical constraints as code. An agent works directly on the circuit description, builds it, runs simulation, and inspects failures without ever leaving the code. That reallocates the benchmark's signal: less time testing computer-use skill, more time testing whether the electronics are actually correct.

This is the same underlying pattern explainx.ai has covered in math: Paul Graham's observation that LLMs excel at math because answers are clearly right or wrong, and Anthropic's Lean-formalized proof of Fermat's Last Theorem, both work because the domain has a cheap, automatic, deterministic right/wrong signal. EEBench applies that exact logic to hardware: a circuit graph, a SPICE simulation, and a measurement against a named limit is the electronics equivalent of a compiler plus a test suite.

The capacitor example: why "it compiles" isn't "it works"

The clearest illustration in the published article is a task built around a residential energy meter. Its 5V supply disappears, and the circuit must keep the processor alive for 20 milliseconds so it can save its reading — meaning the "protected rail" has to stay above the processor's 3.0V brownout threshold during that window.

Most models correctly reach for the obvious fix: add a capacitor. But real capacitors are messier than the nominal value printed on the datasheet. A ceramic capacitor can deliver far less than its rated capacitance once it's biased with voltage, every part carries manufacturing tolerance, and oversizing a capacitor costs money, board space, and recovery time. EEBench cuts power inside a SPICE simulation and measures the actual rail voltage against real component behavior — not the textbook nominal number.

One real failed submission used a capacitor rated at 22µF nominal. At the circuit's actual 4.7V bias, that capacitor delivered only 11.4µF effective capacitance — nowhere close to the 545µF the circuit genuinely required. The rail dropped below 3V after just 0.85ms, ten times faster than the 20ms requirement.

The design compiled successfully. The circuit still failed.

That gap — a design that builds cleanly but doesn't survive contact with real-world physics — is the single most useful thing for builders to take from EEBench, and it generalizes well past electronics. It's the same failure mode as code that passes a shallow test suite but breaks under real load, or a proof that type-checks but encodes the wrong specification. "It ran without errors" is not the same bar as "it works," and the more automated your verification, the more that gap matters.

Harder tasks: tolerance corners and real parts

Beyond the capacitor problem, EEBench includes tougher analog design tasks — for example, synthesizing a multiple-feedback low-pass filter around an op-amp, where the agent has to solve resistor and capacitor ratios that hit a required pole frequency while keeping gain, cutoff, and Q factor within spec across worst-case component-tolerance corners. The harness rebuilds the SPICE deck at those corners and runs AC and transient simulations against named measurement probes, each with an upper and lower spec limit.

Crucially, EEBench uses real manufacturer parts — specs extracted from actual datasheets and fed into the SPICE models, not idealized textbook components. That means an agent has to find a design that survives every tolerance corner and uses parts that actually exist, can be ordered, and are reasonably priced. Cost is only scored once the circuit is verified to work — a submission is graded against a reference bill of materials for cost efficiency, but only after functional correctness clears the bar.

Grading is fully deterministic end to end: build the design, construct the circuit graph and BOM, run SPICE simulations and checks, and produce a per-requirement measurement against a limit. The atopile team draws the direct analogy: it's the same shape as giving a coding agent a compiler plus a test suite, except the tests measure voltages and component behavior instead of function outputs. That same grading infrastructure doubles as a reinforcement-learning reward signal for post-training — a failed run tells you exactly which voltage or corner failed, or whether a design was needlessly expensive, which is a far more useful training signal than a model just asserting a design "looks plausible."

EEBench V1 covers analog and digital design through simulation only. It does not yet test PCB layout, routing, manufacturing, or physical bring-up — those are explicitly planned for later versions.

The leaderboard (September 1, 2026, 13 tasks)

This leaderboard is the solid, citable data — published in the main article on September 1, 2026, with each model run multiple times per task and averaged, variance reported rather than single-run scores. That methodology choice directly addressed a fair question raised in the Hacker News thread by commenter "embedding-shape" about whether the scores were noisy single runs; EEBench team member "iopapa" confirmed averaging-with-variance was already the design.

A separate, thinner data point: in a Hacker News comment reply posted after the article shipped, "iopapa" said GPT-6 Astra results came in at 69.3%, a new #1, with Gemini 3.8 Flash at 55.4% (5th place). That's a genuine claim from an EEBench team member, but it's a single comment reply, not a published, methodology-documented leaderboard entry — treat it as a preliminary signal, not a confirmed result, until it appears in the main article.

Two things stand out in the solid data. First, OpenAI's models underperform here relative to their standing on general coding and reasoning benchmarks — GPT-5.5 and GPT-5.6 Sol both land well below the Claude and Grok models, with real spread across runs. Second, xAI publishing its own EEBench-adjacent result directly in the Grok 4.6 model card — under an "engineering acceleration" section alongside 3D modeling and parametric CAD evals — is notable on its own. It's a sign this benchmark category is starting to get cited directly by frontier labs, not just third-party evaluators. xAI's model card also connects Grok 4.6's EEBench result to engineering-domain reinforcement-learning training that reportedly included CAD environments specifically, which fits the story its score tells.

Elon Musk has said Grok 4.7 is coming "within weeks," trained further on SpaceX engineering data specifically to make it good at engineering work — the EEBench team says they're eager to test it on release. That timeline is unconfirmed and should be read as a rumor, not a commitment.

An important nuance beyond the raw average: Hacker News commenter "chmod775" pointed out that Grok and Opus 5 are more consistent — they rarely fail completely on a task — while models like Fable 5.1 have higher variance, sometimes excelling and sometimes failing outright. A single average score hides that difference, and it matters if you're picking a model for a workflow where a total failure is costlier than a mediocre-but-safe result.

What builders are actually reporting

The Hacker News discussion under the launch is unusually substantive for a benchmark thread, and several comments add real-world texture the leaderboard alone doesn't:

• Commenter "CyLith" had Claude Opus 4.8 design a textbook 74-series-logic circuit that outputs monochrome VGA video from an EEPROM, routed the board themselves, and fabbed it via JLCPCB for $6. One uncaught error needed a manual blue-wire fix; otherwise it worked — a concrete, real-money data point for current capability.
• Commenter "nickff" pushed back on a framing several other commenters used, pointing out that PCB auto-routing is a decades-old, largely solved sub-problem — the actually hard part is component placement, not routing itself.
• Commenter "akiselev" argued current models are only good at schematic capture and netlists — a textual, graph-shaped representation — while PCB routing is an NP-complete geometric problem where an LLM's "geometric intuition is nonexistent." Countering that, commenter "mapontosevenths" reported using Sol to fully route a simple RP2040 board successfully, via generated Python routing scripts — suggesting the limitation is capability- and task-complexity-dependent, not a hard ceiling.
• Commenter "rpcope1" gave a detailed, credible timeline: pre-2025 models couldn't generate anything complex, schematics for 8-bit PIC-based circuits became genuinely usable "around Opus 4.7," but PCB layout has "basically been terrible every time," and complex analog/RF work still "falls down quickly."
• One skeptical top-level comment, from "oxqbldpxo", argued flatly: "It is 3 years later and these AIs have not produced anything meaningful." It's worth weighing honestly against the leaderboard data and CyLith's working $6 board rather than dismissing either side.

Is EEBench neutral? The team says not entirely

To its credit, the atopile team didn't bury this: EEBench team member "iopapa" said directly in the Hacker News comments that the project is advertisement-adjacent for atopile's own agent harness — they built it because they needed a rigorous way to pick which model to use in their own product. That's a disclosed conflict of interest, not a hidden one, and it doesn't invalidate the methodology — the SPICE grading is deterministic and independently reproducible regardless of who runs it — but it's worth knowing going in.

The practical verdict for builders

Read across the leaderboard, the capacitor example, and the Hacker News anecdotes, and a fair current-capability picture looks like this:

• Good today: schematic capture, datasheet-grounded component selection, and reasoning about circuit topology under simulated real-world tolerances — the tasks EEBench V1 actually tests.
• Weak today: PCB layout and placement (as distinct from routing, which is largely solved tooling), and analog/RF design requiring the kind of tacit, hard-won expert judgment that doesn't show up cleanly in training data.
• The one lesson that generalizes: "the design compiled" and "the design works" are different bars, and the gap between them is exactly where AI-generated artifacts — code, proofs, or circuits — quietly fail. Verification that only checks the first bar is verification that misses the failure that actually matters.

Related on explainx.ai

• Paul Graham on why LLMs crush math but lag at writing — the same "verifiable domains concentrate AI progress" pattern, applied to writing instead of hardware
• Claude wrote the first machine-checked Lean 4 proof of Fermat's Last Theorem — deterministic, machine-checkable grading in mathematics
• The 11 best GPT-6 Astra demos from launch week, verified — including the KiCad and BenchCAD demo that's the direct hook for this story
• GPT-6 Astra launch: benchmarks and pricing
• How to read AI benchmarks without getting fooled
• AI benchmarks: the complete guide
• Terminal-Bench 2.0: grading AI agents on real terminal tasks

Primary source: "Can AI design circuit boards yet?" — atopile / EEBench, published September 4, 2026 · Hacker News discussion (152 points, 94 comments)

Accurate as of September 5, 2026. The main EEBench leaderboard reflects September 1, 2026 published data. The GPT-6 Astra (69.3%) and Gemini 3.8 Flash (55.4%) figures come from a single Hacker News comment reply posted by an EEBench team member after the article's publication, not the main published leaderboard — treat them as preliminary until formally published. Follow @explainx_ai for updates.