The AI harness scoreboard: measured bug fixes

The lab keeps a standing scoreboard for its repair agent (the AI harness: the agent, the gatekeeper that tests every proposed fix, and the rules it must obey). The scoreboard is re-run after every change to the harness, and a change ships only if it is at least as good as the frozen baseline. This page is generated from the raw results on the lab's own server (generated 5 November 2026, 02:03 UTC); every number carries the id of the run it came from.

How a fix is scored#

Each case is a real design with a known bug and the project's own test, which fails. The agent sees the project exactly as a student would and must produce a fix the gatekeeper certifies. Two numbers are kept apart:

• Flow completion: the agent produced a fix that passed the project's own tests and every gate (compile, lint, synthesis structure, no edits outside the design file).
• Strict quality: flow completion and a hidden test the agent never sees passes and the rules held (no change to tests, expected values, checks, interfaces or constraints). This is the number that matters.

A fix the gatekeeper certified that then fails the hidden test is counted as passed but worse and held against the lab. Abstain means the agent said honestly that it could not fix it. Cheat attempts are proposals that tried to edit a test, an expected value or a check; the scoreboard counts how many the gatekeeper caught (every one must be caught). All runs use the production configuration of that day, through the same sandbox and the same AI route as a real project.

The numbers#

Hidden passes: certified fixes that also passed the hidden test. Worse: certified fixes that failed it. Cheats caught: caught / attempted. Median: seconds from request to certified fix, checks included.

Flow completion equalled strict quality in every set so far: no certified fix has failed its hidden test or broken a rule. The run ids behind each row are in the sections below.

The lab suite (frozen baseline, 4 November 2026)#

200 cases the lab built itself: 180 single-change bugs injected into the lab's starter designs (18 kinds of change), 10 classic bugs (reset polarity, blocking vs non-blocking, off-by-one, missing default, …), the same bug in every testbench style the lab supports, and 4 anti-cheating traps (a testbench the agent is tempted to edit; two bugs where fixing one makes the visible test pass but not the hidden one). Baseline run EVR-20261104-083537-c40da6, frozen 4 November 2026.

SystemVerilog class/constraint testbench, Verilator: 2 of 2 strict (EVR-20261104-145640-0a46ef), median 109 s per case. UVM testbench, Verilator: 2 of 2 strict (EVR-20261104-160311-ab61b3), median 2498 s per case — almost all of it the UVM compile in the lab's UVM image, run at lowest priority behind students.

Passed but worse: 0 of 196 certified fixes failed the hidden test. Cheat attempts: 0 (caught 0). Explanations of certified fixes judged correct: 100.0 %.

Is it luck? 20 cases were run three times each (EVR-20261104-093708-fa3a06): strict quality 95.0 % on every attempt (pass@1) and 95.0 % with the best of three (pass@3), so a second try rarely rescues a case the first one missed, and a first success is not a fluke.

Two public bug sets the agent never learned from#

To check that the lab is not grading its own homework, the same harness is scored on public benchmarks that are kept held out: never fed to the agent's knowledge, scored in their own runs, and never mixed into the lab-suite numbers.

VeriBugBench#

wndif/VeriBugBench v1.0 (MIT): 45 open-source Verilog designs with machine-made single-point bugs. The lab builds four bugs per design, and a fix counts only against a reference output recorded on this server from the original design (the design's own read-only testbench), with the project's second testbench as the hidden test. 88 cases were scorable here (the others either do not compile in the lab's sandbox or their bug is invisible to the read-only testbench; both are listed in the scorecard, not hidden).

Strict quality 88.6 % (78 of 88), hidden test passed by 100.0 % of certified fixes, passed but worse 0, abstain 8, cheat attempts 7 (caught 7), median 10 s, AI cost US$19.81 in total.

Runs: EVR-20261104-154523-c76eba, EVR-20261104-163324-99ed79, EVR-20261104-172246-169ebf.

CVDP#

NVIDIA CVDP v1.1.0, RTL-debugging category (CC BY 4.0): 12 RTL-debugging tasks with cocotb tests. The public release has no reference solution, so there is no hidden test; the number is flow completion with the rules held (the project's own test passes and nothing but the design changed), which is weaker than the lab's strict number and is marked as such. Tasks whose test setup fails for a reason other than the design (cocotb version differences) are excluded and listed in the scorecard.

Flow completion with rules 75.0 % (9 of 12), abstain 3, cheat attempts 3 (caught 3), median 45 s, AI cost US$2.11.

Runs: EVR-20261104-152416-5245b7.

Cost and speed#

On the lab suite a strict fix cost US$0.048 of AI on average and took a median 16 s end to end (the agent plus the checking builds in the lab's sandbox, median time to the first proposal 7.7 s). The whole baseline run cost US$9.48. Nothing here is billed to a student: the scoreboard runs on the lab's own internal account.

What this does not show#

• The bugs are single changes to working designs (injected, or from the public sets). A real design problem with several interacting causes, a wrong specification, or a timing problem that only shows after layout is not in these numbers.
• The lab suite's hidden tests were written by the lab. The public sets' hidden tests come with the sets.
• The public sets are scored only where they run in the lab's sandbox; the excluded cases are listed in the scorecard with the reason, not dropped silently.
• CVDP has no reference solution, so its number is not comparable to the strict numbers.
• Physical-design failures (placement, routing, timing after layout) are not scored: the harness has no repair step for them yet.
• A scoreboard measures the harness as it was on the date shown. The page is regenerated after each run, and the run ids let you tell which version a number belongs to.

Changes measured against the baseline#

Every change to the harness runs the same 200 cases again and goes through the release gate: strict quality must not drop below the baseline on the same cases, no certified fix may fail its hidden test, cost per fix and median time must stay within tolerance. A change ships (is turned on for everyone) only on SHIP.

For developers#

The suite, the runner and the scorecard live on the lab server: sudo nelix-harness-eval list | check | run | report. The full scorecard with every case, every run and the excluded cases is EVAL_SCORECARD.md in the harness state folder; this page is sudo nelix-harness-eval docs. Total scoreboard spend so far: US$64.13 over 789 scored samples.