How well it works: measured results

You should not have to take our word for anything on this site. This page shows what the lab's tools achieved in measured runs on the lab's own server, how each number was taken, and what it does not show. Every figure is generated from the raw result files; the generation date is printed with it, and the page is regenerated after each release.

AI repair: does a fix really fix the design?#

When Nelix Agent proposes a repair and calls it Verified, is it right? We measure that with bug-fix tasks where a hidden test the AI never sees decides. A fix counts as a true fix only if that hidden test passes; a fix shown to the student as verified that fails it counts against the lab. Three runs, 13–16 October 2026:

• 500 single-line bugs in 28 designs (29 kinds of bug), the first version of the lab's checked repair (every proposed fix must pass the lab's own checks before it is shown) against the single AI answer it replaced;
• a hard set of 150 tasks built to break the old flow (traps where the student's own test is too thin, spec misreads, timing/reset/CDC bugs, multi-line and two-place bugs);
• a fresh standard set of 300 tasks neither version had seen.

The checked repair that is live today fixed everything the single AI answer fixed on these sets and nine more hard tasks. More important than the rate is the label: in 435 answers labelled Verified, none was wrong. When it was not sure it said so: 13 answers were shown as Suggested fix: not verified (all 13 were in fact right) and 2 as Passes your tests, check it yourself (1 right, 1 wrong). The single AI answer, which called every answer verified, was wrong 9 times on the hard set.

What it costs and how long it takes, measured in the same runs:

The first checked repair was about half the cost and 2.5 times faster than the single AI answer. The live version adds a second opinion and more checks, so it costs about 40 % more per fix than the single answer and takes about the same time; that is the price of the label being right.

Chip builds on the four processes#

Every chip build the lab kept a report for between 1 and 27 October 2026: real people's designs, built in the lab's queue with the lab's standard flows (OpenLane 2.3.10 for SKY130 and GF180MCU, LibreLane 3.0.14 for IHP SG13G2, OpenROAD-flow-scripts for ASAP7). These are counts and spreads of different designs at different clocks, not a comparison of the processes; the benchmark further down compares them like for like.

A build "passes" when the flow finishes with a layout and its reports. A clean DRC/LVS/antenna result means the checks the lab runs for that process passed; the chip report lists what each process does not check (for example GF180MCU's own KLayout density and antenna deck, which is a separate button), and the ASAP7 flow runs no sign-off checks at all.

Analog simulations#

249 of 258 ngspice runs on the analog worker finished (SKY130 160, GF180MCU 60, IHP SG13G2 38); the failures were netlist errors in people's own circuits and one convergence failure. A run takes seconds:

Autopilot: from a description to a verified chip#

36 Autopilot runs by 6 people so far. A run counts as a verified chip only when its own tests pass, DRC, LVS and antenna are clean and timing is met, all measured by the tools, never by the AI.

The controlled benchmark: the same designs on every process#

The benchmark built the lab's own starter designs on each process, one after another through the lab's normal queue on a test account, with the clock and die size the lab gives that starter on that process (27 October 2026, 06:10–06:40 UTC). mac8, uart-tx and fifo ran on all four processes; the guided-learning starters (g1-counter, g3-pwm) only on SKY130, because the lab cannot copy a guided project to another process yet. Every build finished with a layout and clean DRC, LVS and antenna checks where the process runs them.

Autopilot on the lab's example specs#

Autopilot was started on the same example descriptions (8-bit counter, UART transmitter) on each process in Engineer mode, with the proposed answers to its questions accepted and no other input. 7 runs reached a result: on SKY130, GF180MCU and IHP SG13G2 every one ended in a verified chip (tests pass, DRC/LVS/antenna clean, timing met); on ASAP7 the chip built and the tests passed but the lab never calls an ASAP7 result verified, because that learning flow runs no sign-off checks. 9 further runs (FIFO and PWM specs) could not start: the AI provider refused every request at that moment because the lab's account had run out of prepaid credit, and Autopilot stopped at once saying so rather than faking a step.

Verification and the release gate (29 October 2026)#

Since 29 October 2026, every project has a V&V page. Before a release, the lab rebuilds a set of reference designs that each carry a known fault, and checks that its tools catch every fault. This is the release gate.

It was run again on 30 October 2026 with a seventh reference (a planted antenna fault): 25 PASS, 3 not applicable for ASAP7, 0 FAIL (record 20261030-155141), identical to the day before where both ran. It now reruns every week. On 29 October 2026 all four processes passed it. Each fault was caught by the checks listed:

The lint and simulation references were flagged as expected on every process. The gate's automated tests passed: 66 cases, 0 failed (tests-20261029-061347). ASAP7 passed on its second run, after its clean reference stopped expecting checks that the learning flow does not have.

The DFT check (stuck-at ATPG, teaching level) measured these coverages:

• SKY130: 91.2 % (VER-20261029-061514);
• GF180MCU: 98.0 % (VER-20261029-091549-619c78);
• IHP SG13G2: 98.8 % (VER-20261029-091550-d554e0);
• the same SKY130 design with scan switched off: 0 % (VER-20261029-094621-4d8041).

ASAP7 is not supported. The largest design built clean on each process is on Qualification envelope. Details of the gate are on Reference checks and release gate.

How the numbers were taken, and what they do not show#

• AI repair is scored by hidden tests the AI never sees. The task sets (1,005 standard tasks from 28 designs, 150 hard tasks), the random seeds and the raw per-task result files are kept on the lab server and can be shown to a reviewer.
• Every chip number is a post-layout tool result from the build's own reports (OpenLane, LibreLane or OpenROAD-flow-scripts). Nothing on this page is a silicon measurement; no Nelix design has been fabricated yet.
• Build times are wall-clock seconds inside the lab's build worker on the production server, where student jobs take priority. Queue time is not included.
• The build statistics above mix different designs and clocks; only the benchmark compares processes like for like, and even there a small starter design says little about a large one.
• The AI never decides pass or fail. Tests, synthesis, DRC, LVS, antenna and timing come from the tools; the AI's job is to write, explain and propose. See How agents are configured and Source provenance and limitations.
• Tool versions: Icarus 12.0, Verilator 5.020 and Yosys 0.33 for simulation and synthesis; OpenLane 2.3.10 (Yosys 0.46, Magic 8.3.489, KLayout 0.29.4, Netgen 1.5.278) for SKY130 and GF180MCU; LibreLane 3.0.14 for IHP SG13G2; OpenROAD-flow-scripts 26Q3 for ASAP7; ngspice on the analog worker.

Real designs. The lab also takes designs it did not write (the set LiveHD publishes results for: DINO, CVA6, CORE-ET, plus PicoRV32) through its normal import, build and V&V rows and publishes every measured result and every lab limit it meets on Benchmarks: real designs through the lab.