betaFeedback

WorksComputer Science

RTLReviewBench: LLM Hardware Reviewers Rarely Miss Bugs but Routinely Reject Correct RTL

Abstract

Hardware teams increasingly ask large language models (LLMs) to review register-transfer level (RTL) code, yet it is unclear whether a model can tell a correct module from a buggy one without simulating it. We introduce RTLReviewBench, a benchmark of 114 matched pairs of buggy and correct Verilog modules. Each pair shares one specification, most buggy modules are real failures of LLM code generators, and every label is confirmed by simulation. Evaluating 12 commercial and open-weight models, we find a consistent asymmetry: reviewers rarely accept buggy code but often reject correct code, producing more than five false rejections for every missed bug. Synthetic mutants hide part of the problem: on the same designs, models miss far fewer injected bugs than real ones. Asking a model to reason before it commits to a verdict sharply reduces false rejections for weaker reviewers, and a controlled ablation shows that where the reasoning appears in the output matters more than the instruction to reason, whereas sampling and majority voting do not help. A blinded audit further shows that some correct "buggy" verdicts rest on false technical claims. We release the benchmark, a harder split for future models, all model responses, and the evaluation code at https://github.com/yasirmahmud/RTLReviewBench

Introduction

LLMs now write a large share of first-draft code, and hardware design is no exception: generation benchmarks for Verilog report steadily rising pass rates [9, 10, 13, 14]. A natural next step is to use the same models as reviewers, checking a frozen RTL module against its specification before it enters an expensive simulation or formal sign-off loop. Review is attractive because it is cheap and needs no testbench, but it is only useful if the reviewer is reliable in both directions. A reviewer that accepts buggy RTL lets defects drift toward silicon; a reviewer that rejects correct RTL floods engineers with false alarms until its warnings are ignored.

Existing benchmarks do not measure this skill. RTL generation suites score the code a model writes, not its judgment of code it is given. Debugging and repair suites [12, 17, 21, 25] reward a passing patch, which a model can often produce by regenerating the module from the specification without understanding the original fault [25]. Software code-review and LLM-as-a-judge benchmarks [6, 7, 22] do separate the two error directions, and Jin and Chen [7] show that LLMs over-reject correct Python, but these benchmarks contain none of the semantics that make hardware review hard: clocked state, blocking versus non-blocking assignment, reset style, and context-dependent bit widths.

We introduce RTLReviewBench, a benchmark for execution-free RTL review (Figure 1). Each item pairs a specification with two modules that compile and share the same ports: a reference that passes the upstream testbench, and a buggy module that fails it. Most buggy modules (97 of

The RTLReviewBench task: specification and sibling RTL modules are reviewed by an LLM, producing a verdict and error-rate definitions

Figure 1. The RTLReviewBench task. A reviewer sees the specification and one of two sibling modules, never both, and must judge whether the module conforms. The example is a real false rejection caused by a misapplied Verilog width rule.

  1. are real functional failures of LLM code generators from RTLRepairBench [25]; the rest are hand-written faults from ChipBench [21]. None are produced by automatic mutation. A reviewer sees one module at a time and must return correct or buggy. Because the two siblings share a specification, we can measure missed bugs (false acceptance) and false alarms (false rejection) on the same designs.

Using RTLReviewBench, we evaluate 12 models from four providers and ask four questions:

  • RQ1: How reliably can LLMs review RTL, and which way do their errors go?

  • RQ2: Do synthetic mutants, the standard way to build bug benchmarks, measure the same thing as real bugs?

  • RQ3: Can prompting, sampling, or side-by-side comparison reduce review errors?

  • RQ4: When a reviewer flags a buggy module, is its explanation correct?

Our main findings are that (i) errors are strongly asymmetric: 11 of the 12 models miss at most 3.5% of bugs, while false rejection ranges from 2% to 59% of correct modules; (ii) synthetic mutants make reviewers look safer than they are on the same designs; (iii) putting the reasoning before the verdict in the output schema removes much of the over-rejection of weaker models, while majority voting over samples does not; and (iv) a fraction of correct “buggy” verdicts cite defects that do not exist.

Contributions.

  1. A benchmark of simulation-verified RTL bugs in matched pairs, mostly real LLM failures (Section 3), with an audit that removes six upstream items whose testbench failures stem from simulator artifacts rather than logic errors, a matched synthetic-mutant set, and a held-out Hard split on which the best current reviewer gets both siblings right on only 77% of pairs.

  1. An evaluation of 12 reviewers (§ 5.1 and § 5.2) showing that false rejection, not missed bugs, is the dominant failure, and that synthetic mutants understate how often real bugs are missed.

  1. Controlled studies of what helps (§ 5.3 and § 5.4): a 2×22 \times2 ablation that separates output ordering from the instruction to reason, comparisons with self-consistency and pairwise review, and a blinded audit of reviewer explanations.

All data, 17,623 model and judge responses, and code are released.1

Related Work

RTL generation. VerilogEval [9, 13], RTLLM [10], and VeriGen [16] established simulation-based pass@kk for single modules; CVDP [14], RealBench [8], TuRTLe [4], and RTL-BenchLS [18] extend coverage to verification tasks, formal equivalence, and larger designs. All of them grade code the model writes. RTLReviewBench instead fixes the code and grades the model’s judgment of it.

RTL debugging and repair. RTLFixer [17] repairs syntax errors; ChipBench [21], VeriBugBench [12], HWE-Bench [3], and Phoenix-bench [26] evaluate functional repair from single modules up to whole repositories. RTLRepairBench [25] shows that repair scores can be earned by regenerating code from the specification and that simulation oracles carry reset and time-zero confounds. We reuse its real LLM failures and adopt its concern with oracle validity, but ask for a verdict rather than a patch, which removes the regeneration shortcut.

Code review and LLM-as-a-judge. CodeJudgeBench [6] finds that judges are sensitive to response order; SWR-Bench [22] and MCR-Bench [23] evaluate review comments on pull requests; and Jin and Chen [7] report that LLMs systematically reject correct code and that richer prompts can make this worse. Related work documents position bias [24], the effect of output-format constraints on reasoning [15], and gains from self-consistency [19]. We test whether these findings carry over to hardware, where correctness depends on timing and bit-level semantics. Table 1 summarizes the differences.

BenchmarkDomainTaskMatched pairsFAR & FRR
VerilogEval v2 [13]RTLGenerate RTL from a specification——
ChipBench [21]RTLGenerate, debug, and write reference models——
VeriBugBench [12]RTLLocalize and repair injected faults——
RTLRepairBench [25]RTLRepair real and synthetic faultspartial—
CodeJudgeBench [6]SoftwareRank two candidate responses✓—
SWR-Bench [22]SoftwareComment on pull requests✓✓
Jin & Chen [7]SoftwareJudge one implementation✓✓
RTLReviewBench (ours)RTLJudge one frozen module, no repair✓✓

Table 1. RTLReviewBench compared with closely related benchmarks.

The RTLReviewBench Benchmark

Task

An item is a design kk with specification SkS_k and two modules CkcorC_k^{\mathrm{cor}} and CkbugC_k^{\mathrm{bug}} that compile and have identical interfaces. CkcorC_k^{\mathrm{cor}} passes the upstream reference testbench and CkbugC_k^{\mathrm{bug}} fails it. The reviewer receives SkS_k and a single module, never both, together with the instruction “Judge only the supplied RTL implementation. Do not write or repair RTL,” and returns a JSON object with a verdict (correct, buggy, or abstain), a confidence, and a short reason. File names, comments that hint at the answer, and sibling information are removed. Exact prompts are in Section A.

Construction

Figure 2 summarizes the pipeline; full accounting is in Section B.

Construction pipeline for the primary split

Figure 2. Construction of the primary split. The same filters produce the synthetic-mutant set (SynMut-50).

Sources. We draw real bugs from two pinned corpora. ChipBench [21] provides sequential debugging tasks in which engineers injected faults by hand into golden modules. RTLRepairBench [25] provides RealBugs: functional failures of Qwen2.5-Coder models on VerilogEval v2 tasks, each paired with the upstream reference.

Filtering and verification. We remove tasks whose specification is incomplete or contradicts the reference, and items whose specification or code leaks the answer (for example, a comment marking the correct line). Every surviving pair is compiled and simulated with Icarus Verilog 11.0 [20]; we keep a pair only if the reference passes its testbench and the buggy module fails it. To avoid counting the same design twice, we keep one pair per upstream task, giving 120 designs (Full-120).

Oracle audit. A failing testbench does not always mean a logic bug. Inspecting low-mismatch failures, we found six pairs in which the “buggy” module is functionally equivalent to the reference and fails only because of simulator semantics: an explicit #delay, a missing time-zero event, or a reference that relies on a non-synthesizable initial block (Section B.1). These six pairs (Artifact-6) account for half of all missed bugs on Full-120, so we report results on the remaining Clean-114 split and treat Full-120 as a sensitivity check.

Additional splits. Table 2 lists all splits. SynMut-50 contains single-operator mutants of VerilogEval references from RTLRepairBench, passed through the same filters. On 42 of its designs, the same reference also appears in Clean-114 with a real LLM bug (Matched-42), which lets us compare real and synthetic bugs on identical designs. Hard contains the 22 Clean-114 pairs on which at least half of our reviewers err on a sibling; because it is selected with the evaluated models, we intend it as a target for future models rather than an unbiased estimate for these ones.

SplitDesignsPurpose
Clean-114114Primary evaluation: 97 real LLM failures (RTLRepairBench) and 17 hand-written faults (ChipBench); 80 sequential designs
Full-120120Sensitivity check: Clean-114 plus the six Artifact-6 pairs
SynMut-5050Single-operator synthetic mutants (six operator types)
Matched-4242Designs present in both Clean-114 and SynMut-50
Hard22Pairs that at least 6 of 12 reviewers get wrong; headroom for future models

Table 2. Benchmark splits. All splits contain one buggy and one correct module per design.

Experimental Setup

Models. We evaluate 12 models from four providers (Table 7): Claude Haiku 5.5 (with and without extended thinking) and Claude Sonnet 4.6 from Anthropic; Gemini 3.1 Pro, Gemini 3.8 Flash, and the open-weight Gemma 4 31B from Google; GPT-5.4, GPT-4.1, GPT-4.1 mini, o4-mini, and the open-weight gpt-oss-120b from OpenAI; and Llama 3.3 70B from Meta. Five of them spend hidden reasoning tokens before answering. All runs use greedy decoding (T=0T = 0), JSON output mode, no examples, and no tools; queries were issued through an OpenAI-compatible endpoint in a single two-day window (Section E).

Model IDDeployment ParadigmReasoning Tokens{sup}†`Core 228 [?]SynMut (100)Reason-First (228)Pairwise / Audit (1,309)SC@5 (1,140)
claude-haiku-5.5Budget Commercial (Anthropic)None (0)✓✓✓✓ / ✓✓
gemini-3.8-flash{sup}†`Budget Commercial (Google)Adaptive (59k–766k)✓✓✓✓ / ✓✓
gpt-4.1-miniBudget Commercial (OpenAI)None (0)✓✓✓✓ / ✓—
claude-4.6-sonnetFlagship Commercial (Anthropic)Prose / Ext. (0)✓✓✓✓ / ✓—
gemini-3.1-pro{sup}†`Flagship Commercial (Google)Native (345k Clean)✓✓✓✓ / ✓—
gpt-5.4Flagship Commercial (OpenAI)None (0)✓✓✓✓ / ✓✓
gpt-4.1Flagship Commercial (OpenAI)None (0)✓✓✓✓ / ✓—
claude-haiku-5.5-thinking{sup}†`Test-Time Reasoning (Anthropic)Native (156k Clean)✓✓✓✓ / ✓—
o4-mini{sup}†`Test-Time Reasoning (OpenAI)Native (325k Clean)✓✓✓✓ / ✓—
gpt-oss-120b{sup}†`Open-Weight On-PremisesNative (180k Clean)✓✓✓✓ / ✓—
gemma-4-31b-itOpen-Weight On-PremisesNone (0)✓✓✓✓ / ✓—
llama-3.3-70b-instructOpen-Weight On-PremisesNone (0)✓✓✓✓ / ✓—

Table 7. Evaluated models (12 across 4 paradigms) and complete empirical study matrix across RTLReviewBench (17,623 API queries total; †native/adaptive test-time reasoning_tokens; counts shown as Clean-114 [Full-120]).

Metrics. We treat buggy as the positive class. The false acceptance rate (FAR) is the fraction of buggy modules judged correct, and the false rejection rate (FRR) is the fraction of correct modules judged buggy. Accuracy counts abstentions and invalid outputs as wrong. Because siblings share a specification, we also report pair accuracy, the fraction of designs for which both siblings are classified correctly. We give exact 95% Clopper–Pearson intervals [1], compare conditions on the same items with exact McNemar tests [11], and correct each family of tests with the Holm–Bonferroni procedure [5].

Results

RQ1: How Reliable Is LLM Review of RTL?

Table 3 and Figure 3 summarize single-module review on Clean-114.

Figure 3. Error direction on Clean-114. Bars to the left show the share of buggy modules accepted (FAR); bars to the right show the share of correct modules rejected (FRR). Across all models, correct modules are rejected 325 times and bugs are missed 59 times.

ReviewerAccuracyFARFRRPair
Claude Haiku 5.5 (thinking)97.4 [94.4, 99.0]2.61.894.7
Claude Haiku 5.597.4 [94.4, 99.0]3.51.894.7
Gemini 3.1 Pro96.9 [93.8, 98.8]0.95.393.9
Gemini 3.8 Flash93.4 [89.4, 96.3]0.013.286.8
Claude Sonnet 4.693.0 [88.9, 95.9]2.611.487.7
gpt-oss-120b91.7 [87.3, 94.9]1.814.984.2
o4-mini90.8 [86.3, 94.2]2.615.882.5
GPT-5.483.8 [78.3, 88.3]3.528.969.3
Gemma 4 31B78.9 [73.1, 84.1]3.538.657.9
GPT-4.1 mini71.9 [65.6, 77.7]2.653.545.6
GPT-4.169.7 [63.3, 75.6]1.858.840.4
Llama 3.3 70B66.2 [59.7, 72.3]26.341.236.0
Majority of three (cross-validated)97.4–––

Table 3. Clean-114 results (228 modules per model, sorted by accuracy). Brackets give 95% confidence intervals. Pair: both siblings correct. The full table, with Full-120 results, calibration, and latency, is Table 8.

The best reviewers are strong. Claude Haiku 5.5 and Gemini 3.1 Pro classify about 97% of modules correctly and get both siblings right on 94% of designs. Gemini 3.8 Flash misses no bug at all, but pays for it by rejecting 13% of correct modules. Larger or reasoning-enabled models do not reliably win: Claude Sonnet 4.6 rejects more correct modules than Claude Haiku 5.5, and turning on extended thinking for Haiku leaves its accuracy unchanged.

Errors are lopsided. Eleven of the twelve models miss at most 3.5% of bugs. What separates them is false rejection, which ranges from 2% to 59%. Pooled over all models, correct modules are rejected 5.5 times as often as bugs are missed (Figure 3). The weakest reviewers reject more than half of all correct modules, so in practice they would flag almost everything. Llama 3.3 70B is the only model that also misses many bugs.

Why correct code gets rejected. Reading the rejections (Section D), we find three recurring causes: misapplied Verilog semantics, such as the width rule in Figure 1; invented requirements about latency, reset style, or coding conventions that the specification does not state; and dialect purism, such as rejecting legal SystemVerilog constructs. Longer, multi-block sequential designs from ChipBench trigger the most rejections. Only one module defeats all 12 reviewers, and it reveals a mismatch between an upstream specification and its reference rather than a reviewer failure (Section D).

Combining reviewers. A majority vote over three models from different providers matches the best single model (97.4%) when the triple is chosen by five-fold cross-validation; the best triple chosen in hindsight reaches 98.2% (Section C). At this level, ensembles offer at most a small gain.

Headroom. On the Hard split, the best reviewer gets both siblings right on 77% of pairs and mean accuracy drops from 86% to 67% (Table 9); false rejection still dominates.

ModelClean-114 Acc.Acc.FARFRRPairReason-First Acc.
claude-haiku-5.597.4%88.6%18.2%4.5%77.3%90.9%
claude-haiku-5.5-thinking97.4%88.6%13.6%9.1%77.3%90.9%
gemini-3.1-pro96.9%84.1%4.5%27.3%68.2%90.9%
gpt-oss-120b91.7%75.0%10.9%40.9%54.5%72.7%
gemini-3.8-flash93.4%72.7%0.0%54.5%45.5%77.3%
claude-4.6-sonnet93.0%70.5%13.6%45.5%50.0%77.3%
o4-mini90.8%70.5%13.6%45.5%45.5%81.8%
gemma-4-31b-it78.9%52.3%13.6%81.8%4.5%63.6%
gpt-4.169.7%50.0%9.1%90.9%4.5%65.9%
gpt-5.483.8%50.0%13.6%86.4%9.1%56.8%
llama-3.3-70b-instruct66.2%50.0%22.7%77.3%9.1%52.3%
gpt-4.1-mini71.9%47.7%9.1%95.5%4.5%63.6%

Table 9. RTLReviewBench-Hard (22 Clean pairs / 44 cases; selection: ≥ 6/12 reviewers misclassify a sibling under Verdict-First). Scores for these 12 models are in-sample by construction.

Answer to RQ1 The best LLMs review RTL accurately, but errors across models are strongly asymmetric: reviewers rarely accept a buggy module and often reject a correct one. For most models, false rejection, not missed bugs, is the main obstacle to deployment, so FAR and FRR should always be reported separately.

RQ2: Do Synthetic Mutants Reflect Real Bugs?

Bug benchmarks are usually built by injecting single-operator mutations into correct code [12, 25]. Matched-42 lets us ask whether such mutants are as hard to catch as real bugs, holding the design and the correct reference fixed.

Mutants are easier to catch. Pooled over all twelve models, reviewers miss 28 of 504 real bugs but only 9 of 504 synthetic mutants on the same designs (exact McNemar p<0.001p < 0.001; Figure 4). The direction holds for ten of the twelve models. One plausible explanation is that a mutant such as a swapped operator or a flipped comparison changes a single visible token, whereas an LLM-written bug is embedded in code that was generated to look right.

Figure 4. Real LLM bugs versus synthetic mutants on the same 42 designs (Matched-42). Each row is a model; the top row pools all twelve. Synthetic mutants are missed less often (left) while false rejection of the shared correct reference barely changes (right).

False rejection is unchanged. The correct reference is identical in both conditions, and its rejection rate barely moves (22% versus 21%). Synthetic benchmarks therefore understate the risk of missed bugs while saying nothing new about false alarms.

Oracle artifacts distort the picture further. Before the oracle audit, bugs produced by the fine-tuned generator appeared to be a blind spot, with most models missing 30–50% of them. All of these apparent misses came from the three #delay artifacts; on the remaining clean items, nine of twelve models miss none.

Answer to RQ2 No. On identical designs, synthetic mutants are missed far less often than real LLM bugs, and they leave the dominant failure, false rejection, untouched. Benchmarks for RTL review should use real bugs and audit their simulation oracles.

RQ3: Can Prompting, Sampling, or Comparison Help?

We test three common ways to improve a judge: changing the prompt, sampling several answers, and comparing candidates side by side.

Reason-First prompting. Our default schema asks for the verdict first and the reason afterwards. Reason-First prompting makes two changes: the reason field comes before verdict, and the reviewer is told to trace signals, operators, and timing step by step before deciding. As Figure 5(a) shows, this improves three short-answer models by 11 to 16 points, mostly by cutting false rejection (GPT-4.1 goes from 70% to 86%). Models that already reason at length change little, although Gemini 3.1 Pro reaches 98.2%, the best single result in the study. Reason-First also removes a telling failure of the default schema: in 15 Clean responses, a model emitted a verdict, worked through the logic, and concluded the opposite inside its own explanation. For example, Gemma 4 answered "verdict":"correct" for an XNOR module and then wrote “This is XOR. The spec asks for XNOR. The implementation is buggy.” With the reason first, all 15 are resolved correctly.

Figure 5. (a) Accuracy with the default Verdict-First schema (grey) and with Reason-First prompting (blue); * marks Holm-significant changes. (b) A 2×22 \times2 ablation on the three models marked * separates the order of fields in the output from the instruction to trace the logic. Thick lines are means; thin lines are individual models.

Order matters more than the instruction. To separate the two changes, we ran the two missing cells of a 2×22 \times2 design on the three models that benefited (Figure 5(b)). Moving the reason before the verdict, without any tracing instruction, raises accuracy by 8 points on average and is significant for all three models. Adding the tracing instruction while the verdict still comes first triples output length but does not help (a 1-point drop on average). The full gain requires both: tracing helps only when it happens before the decision, a positive interaction of about 7 points (95% bootstrap interval 3.5 to 9.6). The output schema thus acts as a commitment device. Once a model has emitted a verdict, later reasoning rarely changes its decision.

Sampling and voting do not help. For three commercial models, we drew five samples at T=0.7T = 0.7 and took the majority. Accuracy changed by between −1.3 and 0 points, because 93–94% of items received five identical votes: the errors are systematic, not random. Voting across different models (RQ1) helps more than voting within one.

Side-by-side comparison helps but adds position bias. When shown both siblings and asked which one conforms, most models pick the correct module more reliably than they classify the two modules separately; Gemini 3.1 Pro picks the correct module under both orderings on 98% of pairs. Weaker models, however, often change their answer when the two candidates are swapped, mostly in favour of whichever module is shown first; Llama 3.3 70B does so on 20 of 114 pairs, echoing position bias in software judges [6, 24]. Comparison is also not available in real review, where no correct sibling exists. Full results for all three interventions are in Section C.

Answer to RQ3 Prompting helps when it changes when the model reasons: placing the reason before the verdict recovers most of the over-rejection of weaker reviewers, while an instruction to reason after committing does not help. Self-consistency sampling does not fix systematic errors, and side-by-side comparison introduces position bias in weaker models.

RQ4: Are Correct Verdicts Correctly Explained?

A verdict of buggy on a buggy module counts as correct even if the stated reason is wrong. A reviewer that rejects most code will also “catch” most bugs, often for the wrong reason. We therefore audited all 1,309 correct detections on Clean-114. Two judges from different providers (GPT-5.4 and Claude Sonnet 4.6) saw the specification, both siblings, and the failing simulation output, but not the reviewer’s identity, and labelled each explanation grounded (names the real defect), spurious (asserts a defect that does not exist), or vague. A third model (Gemini 3.1 Pro) resolved disagreements, and a human checked every item labelled spurious.

Most detections are well founded, but not all. Pooled across models, 94% of detections cite the real defect, and the strongest reviewers are almost always right for the right reason (Figure 6). The over-rejecting models lose the most: GPT-4.1 and GPT-4.1 mini lose eight to nine detections each to false technical claims. Llama 3.3 70B correctly explains only 41% of buggy modules, although it labels 74% of them buggy. In one case, GPT-4.1 and GPT-5.4 both flagged a module with an incorrect asynchronous reset, but cited a legal SystemVerilog construct as the defect and never mentioned the reset.

Stacked bar chart showing explanation audit results for buggy Clean-114 modules

Figure 6. Explanation audit of every buggy module flagged on Clean-114. A detection counts as “correct reason” only if the explanation identifies the actual defect.

Judges need checking too. The two judges agree on 93% of items (Cohen’s κ=0.54\kappa= 0.54 [2]). Both rate their own family’s explanations more favourably, by up to 9 points for Claude Sonnet 4.6, which is why we use cross-provider judges, a tie-breaker, and human review. The human check confirmed 48 of the 50 spurious labels.

Confidence tracks reasoning effort. Models that answer in under 100 tokens are also badly calibrated: they report near-certain confidence while being wrong a sixth to a third of the time (Gemma 4 reports full confidence on 224 of 228 modules). Models that reason at length are well calibrated (Table 8).

Model / GateClean Acc. (X/228, 95% CI)FAR (Miss) (F N/114)FRR (Over-Rej.) (F P/114)Pair: Both / 1 / 0 (114 Pairs)Brier ScoreCalib. GapMed. Lat. (ms)Med. Tok. (Ok / Err)Full-120 Sens. Acc.Full-120 Sens. FAR
Single Models (T=0.0T = 0.0, Sorted by Clean-114 Strict Accuracy then Brier Score)
claude-haiku-5.5-thk‡222/228 (97.4%) [94.4, 99.0]3/114 (2.6%)2/114 (1.8%‡)108 / 6 / 0 (94.7%)0.0280−7.4 pp8998555.0 / 1403.595.0%7.5%
claude-haiku-5.5222/228 (97.4%) [94.4, 99.0]4/114 (3.5%)2/114 (1.8%)108 / 6 / 0 (94.7%)0.0310−7.2 pp3617476.5 / 1595.095.0%8.3%
gemini-3.1-pro221/228 (96.9%) [93.8, 98.8]1/114 (0.9%)6/114 (5.3%)107 / 7 / 0 (93.9%)0.0299+3.0 pp12237903.0 / 1878.095.0%5.0%
gemini-3.8-flash213/228 (93.4%) [89.4, 96.3]0/114 (0.0%)15/114 (13.2%)99 / 15 / 0 (86.8%)0.0590+5.3 pp5505431.0 / 1218.092.5%2.5%
claude-4.6-sonnet§212/228 (93.0%) [88.9, 95.9]3/114 (2.6%)13/114 (11.4%)100 / 12 / 2 (87.7%)0.0600+2.1 pp8530383.5 / 625.591.2%6.7%
gpt-oss-120b209/228 (91.7%) [87.3, 94.9]2/114 (1.8%)17/114 (14.9%)96 / 17 / 1 (84.2%)0.0727+2.8 pp8815620.0 / 953.090.0%5.8%
o4-mini207/228 (90.8%) [86.3, 94.2]3/114 (2.6%)18/114 (15.8%)94 / 19 / 1 (82.5%)0.1048−8.8 pp8959874.0 / 2302.089.2%6.7%
gpt-5.4191/228 (83.8%) [78.3, 88.3]4/114 (3.5%)33/114 (28.9%)79 / 33 / 2 (69.3%)0.1558+14.5 pp407670.0 / 82.082.1%7.5%
gemma-4-31b-it180/228 (78.9%) [73.1, 84.1]4/114 (3.5%)44/114 (38.6%)66 / 48 / 0 (57.9%)0.2100+20.8 pp302984.0 / 212.077.5%7.5%
gpt-4.1-mini164/228 (71.9%) [65.6, 77.7]3/114 (2.6%)61/114 (53.5%)52 / 60 / 2 (45.6%)0.2348+19.6 pp215892.5 / 126.071.2%5.8%
gpt-4.1159/228 (69.7%) [63.3, 75.6]2/114 (1.8%)67/114 (58.8%)46 / 67 / 1 (40.4%)0.2716+25.5 pp624779.0 / 99.068.8%5.8%
llama-3.3-70b-instruct151/228 (66.2%) [59.7, 72.3]30/114 (26.3%)47/114 (41.2%)41 / 69 / 4 (36.0%)0.3278+32.3 pp370729.0 / 27.065.8%29.2%
Multi-Model Boolean Gates, 5-Fold Cross-Validation, & Deferral Regimes
Top-3 Diverse Maj. (H, P31, Oss)224/228 (98.2%) [95.6, 99.5]1/114 (0.9%)3/114 (2.6%)110 / 4 / 0 (96.5%)————96.2%5.0%
5-Fold Pair CV (Top-3 Fam. / Any)224 (98.2%) / 222 (97.4%)1 (0.9%) / 1 …
Top-3 Comm. Maj. (H, P31, Son)221/228 (96.9%) [93.8, 98.8]2/114 (1.8%)5/114 (4.4%)107 / 7 / 0 (93.9%)————95.0%5.8%
Low-FF PND Gate (H & P31)223/228 (97.8%) [94.9, 99.3]4/114 (3.5%)1/114 (0.9%)109 / 5 / 0 (95.6%)————95.4%8.3%
Strict OR Gate (H ∣ Fl)212/228 (93.0%) [88.9, 95.9]0/114 (0.0%)16/114 (14.0%)98 / 16 / 0 (86.0%)————92.1%2.5%
Consensus Subset (H ⊢ P31)¶217/219 (99.1%) [96.7, 99.9]1/111 (0.9%)1/108 (0.9%)—————97.0%5.2%
Consensus Subset (H == Fl)¶208/209 (99.5%) [97.4, 100.0]0/110 (0.0%)1/99 (1.0%)—————98.2%2.7%

Table 8. Zero-shot functional conformance performance on the primary Clean-114 track (114 designs / 228 balanced cases) with exact 95% Clopper-Pearson intervals [1], alongside Full-120 (240c) sensitivity columns and cross-validated ensembles. Bold highlights top single-model and ensemble operating points.

Answer to RQ4 Mostly, for strong reviewers. For weaker reviewers, a correct verdict is often a by-product of rejecting everything, and some cite defects that do not exist. Evaluations of LLM reviewers should check explanations, not only verdicts.

Discussion

Implications for practice. Our results suggest four practices for teams that use LLM reviewers. First, measure false rejection explicitly: it is the failure most likely to make a reviewer unusable. Second, request the reasoning before the verdict; this simple change to the output schema was the largest improvement we observed. Third, prefer combining different models, or deferring disagreements to simulation, over resampling one model. Fourth, treat explanations as claims to verify, especially from reviewers that reject often.

Implications for benchmark design. Synthetic mutants and unaudited simulation oracles both distorted our measurements: the former hid missed bugs, and the latter manufactured an apparent blind spot. RTL benchmarks should use real defects where possible, report how oracle artifacts were handled, and evaluate both error directions on matched items.

Threats to validity. Internal. Labels come from upstream testbenches; we audited low-mismatch failures by hand, but a formal equivalence check would cover inputs that the testbenches do not. The rationale audit relies on LLM judges, which we mitigated with cross-provider judges and human review. Construct. The task presents modules in isolation, which is stricter than review with access to a simulator or a diff. External. Designs are single modules of 6–134 lines, and 97 of the 114 come from VerilogEval, whose references may appear in training data; models score higher on these than on the newer ChipBench items, although the two sources also differ in design size. Results may change as hosted models are updated; all responses are released so the numbers can be rechecked.

Conclusion

We introduced RTLReviewBench, a benchmark of simulation-verified RTL bugs in matched pairs, most of them real LLM failures. Across twelve models, LLM reviewers rarely miss bugs but routinely reject correct RTL; synthetic mutants make them look safer than they are; reasoning before the verdict, rather than after it, removes much of the over-rejection; and some correct verdicts rest on false claims. We hope RTLReviewBench and its Hard split help make LLM review a dependable step in hardware verification rather than a source of false alarms.

Prompts

Listing 1: Frozen single-candidate (Verdict-First Baseline), Reason-First prompting, 2×2 ablation cells (vf_trace, rf_order_only), Pairwise comparison, and blinded Rationale Audit Judge templates.

````text === 1. SINGLE-CANDIDATE VERDICT-FIRST (Core & SynMut-50 Baseline) === SYSTEM: You are reviewing RTL for functional conformance to the given specification. Judge only the supplied RTL implementation. Do not write or repair RTL. If evidence is insufficient, you may abstain. Return only valid JSON matching {"verdict":"correct|buggy|abstain","confidence":0.0,"reason":"short explanation"}. Use one of the three literal verdict strings. Confidence must be between 0 and 1. Do not assume that syntactically valid code is functionally correct.

USER: SPECIFICATION (verbatim from the existing dataset): {specification}

RTL IMPLEMENTATION (candidate under review):

{candidate_rtl}

Is this RTL functionally consistent with the specification? Output JSON only, with verdict, confidence, and a short reason.

=== 2. REASON-FIRST PROMPTING (Schema Order + Tracing Instruction) === SYSTEM: You are reviewing RTL for functional conformance to the given specification. Judge only the supplied RTL implementation. Do not write or repair RTL. If evidence is insufficient, you may abstain. Return only valid JSON matching {"reason":"step-by-step functional check of signals, operators, and cycle timing against the spec","verdict":"correct| buggy|abstain","confidence":0.0}. First trace the RTL behavior inside "reason", then commit to one of the three literal verdict strings. Confidence must be between 0 and 1. Do not assume that syntactically valid code is functionally correct.

USER: [Same SPECIFICATION & RTL IMPLEMENTATION block] Is this RTL functionally consistent with the specification? Output JSON only, reasoning first in "reason", then giving "verdict" and "confidence".

=== 2a. ABLATION CELL A: VF_TRACE (Verdict-First Order + Tracing Instruction) === SYSTEM: You are reviewing RTL for functional conformance to the given specification. Judge only the supplied RTL implementation. Do not write or repair RTL. If evidence is insufficient, you may abstain. Return only valid JSON matching {"verdict":"correct|buggy|abstain","confidence":0.0,"reason":"step-by-step functional check of signals, operators, and cycle timing against the spec"}. In "reason", trace the RTL behavior step by step against the specification. Use one of the three literal verdict strings. Confidence must be between 0 and 1. Do not assume that syntactically valid code is functionally correct.

USER: [Same SPECIFICATION & RTL IMPLEMENTATION block] Is this RTL functionally consistent with the specification? Output JSON only, with verdict, confidence, and a step-by-step reason.

=== 2b. ABLATION CELL B: RF_ORDER_ONLY (Reason-First Order Only) === SYSTEM: You are reviewing RTL for functional conformance to the given specification. Judge only the supplied RTL implementation. Do not write or repair RTL. If evidence is insufficient, you may abstain. Return only valid JSON matching {"reason":"short explanation","verdict":"correct|buggy|abstain","confidence":0.0}. Use one of the three literal verdict strings. Confidence must be between 0 and 1. Do not assume that syntactically valid code is functionally correct.

USER: [Same SPECIFICATION & RTL IMPLEMENTATION block] Is this RTL functionally consistent with the specification? Output JSON only, with reason, verdict, and confidence, in that key order.

=== 3. BIDIRECTIONAL CONTRASTIVE PAIRWISE (FWD & REV) === SYSTEM: You are reviewing two candidate RTL implementations (Candidate A and Candidate B) for functional conformance to the given specification. Judge only the supplied RTL implementations. Do not write or repair RTL. ````

Exactly one implementation may conform cleanly, or both may fail/tie if evidence is ambiguous.
Return only valid JSON matching {"choice":"A|B|tie|neither","confidence":0.0,"reason":"short explanation"}.
Use one of the four literal choice strings ("A", "B", "tie", "neither"). Confidence must be between 0 and 1.

USER:
SPECIFICATION (verbatim from the existing dataset):
{specification}

CANDIDATE A:
```verilog
{candidate_a_rtl}

CANDIDATE B:

{candidate_b_rtl}

Which candidate RTL is functionally consistent with the specification? Output JSON only, with choice ("A", "B", "tie", or "neither"), confidence, and a short reason.

=== 4. BLINDED MULTI-JUDGE RATIONALE FIDELITY AUDIT === SYSTEM: You are an impartial senior RTL verification engineer auditing an anonymous code reviewer’s explanation on a KNOWN-BUGGY Verilog module. Classify the reviewer’s "reason" into EXACTLY ONE of:

  • "grounded": Correctly identifies (partially or fully) the real functional defect or mismatched signal/timing behavior responsible for failing the spec/simulation. Minor wording imprecision is acceptable if the identified mechanism matches the real bug.

  • "spurious_claim": Asserts a specific technical flaw in buggy.sv that is factually false, targets an unrelated/correct construct (e.g., falsely blaming non-blocking <=, nonexistent syntax/reset/latency error, or flipped diagnosis), while missing the actual defect.

  • "vague": Contains only generic boilerplate ("violates specification", "incorrect logic", "syntax/timing errors") with no concrete defect localization. Return ONLY valid JSON: {"label":"grounded|spurious_claim|vague","actual_bug":"one sentence describing real bug","reviewer_claim_accurate":true ,"confidence":0.0}

USER: SPECIFICATION: {specification} GOLDEN REFERENCE: {ref_rtl} BUGGY IMPLEMENTATION: {buggy_rtl} ICARUS VERILOG DIFF OUTPUT: {sim_log} ANONYMOUS REVIEWER REASON TO AUDIT: {reviewer_reason}


# Benchmark Construction Details

{xref}`Table 4` gives the full curation funnel and {xref}`Table 5` the size statistics of each split.

## The Artifact-6 Oracle Audit

**Oracle Artifact Audit (Artifact-6 vs. Clean-114).** Post-run audit of low-mismatch cases, upstream initial statements, and explicit simulation delays identified 6/120 upstream pairs (Artifact-6) where testbench failure arises from simulator event semantics or inertial delays rather than synthesizable functional logic defects: (a) three time-0 / initialization artifacts where candidate RTL is synthesizably valid—P020/C039 (always `@(*)` out = in omits the $t = 0$ continuous-assignment event in Icarus, causing 2/100 mismatches at $t = 5$; all 12 LLMs independently judge it correct), P047/C094, and P084/C167 (upstream RefModules use non-synthesizable initial out=0/Q=0 before the first clock edge, 1/100 and 1/199 mismatches despite no reset in the spec); and (b) three realbug_tuned inertial-delay cases (P018/C035 #10, P022/C044 #10, P028/C055 #5) whose Boolean logic is identical to RefModule but whose explicit Verilog #delay lags the 5 ps testbench clock half-period. Across all 12 reviewers, Artifact-6 alone accounts for 50.0% (59/118) of all Full-120 False Acceptances plus 4 FPs. Rather than altering the frozen evaluation manifests, we retain Full-120 as a sensitivity boundary and establish the filtered Clean-114 corpus

(114 pairs / 228 cases: 80 sequential, 34 combinational; 17 ChipBench + 90 realbug_base + 7 realbug_tuned; 68/114 buggy-first) as our primary evaluation slice throughout § 5.

Upstream Source Provenance and Exclusion Ledger

All candidate exclusions in Table 4 and Table 6 execute deterministically:

Pipeline Gate / DispositionCountExclusion / Selection Basis (Section B.2)
Core Track — Source 1: ChipBench (MIT; Sequential Multi-Always Pool)
Upstream debugging rows scanned89state_machine (6), timing (29), arithmetic (24), assignment (30)
– Excluded (incomplete spec / spec–ref conflict / non-seq)−36Missing standalone specs (−8), conflicts (−17), comb. slice (−11)
– Excluded by Icarus 11.0 compile / TB gate−4Buggy compile fails (−2), golden reference TB mismatches (−2)
Eligible → Clean ChipBench Retained49 → 1717 seq / 0 comb (arithmetic: 11, assignment: 6; 0 artifacts)
Core Track — Source 2: RTLRepairBench RealBugs (Apache-2.0 / MIT; VerilogEval v2 Pool)
Upstream RealBugs rows scanned274112 VerilogEval v2 tasks (252 base + 22 fine-tuned failures)
– Leakage exclusions (spec hints / RTL comments)−8Hint leakage in spec (−6), golden RTL correctness comments (−2)
– Excluded by Icarus 11.0 compile / TB gate−15Reference TB fails (−9), ref compile (−5), buggy SV cast (−1)
Unique Core Designs Retained (Full-120 Share)251 → 103realbug_base: 93, realbug_tuned: 10
– Audited Simulation-Only / #delay Artifacts (Artifact-6)−63 time-0/initial (P020, P047, P084) + 3 #delay (P018, P022, P028)
Clean RTLRepairBench Retained9763 seq / 34 comb (realbug_base: 90, realbug_tuned: 7)
Contrast Track — SynMut-50 (RTLRepairBench Sem85 Synthetic Semantic Mutations)
Upstream single-operator mutant rows scanned85VerilogEval v2 RefModule + single-operator semantic mutations
– Static leakage filter & Icarus 11.0 differential gate85 → 841 static check exclusion; all 84 pairs (168 logs) verified by Icarus 11.0
Balanced Rarest-Operator Deduplication84 → 5036 seq / 14 comb; 6 operators (42 Clean RB overlap + 2 Art.-6 + 6 Syn-only)
Primary Clean / Sensitivity Corpus114 + 50 (+6)Clean-114 (228c) + SynMut-50 (100c) [Full-120 (240c) w/ Artifact-6]

Table 4. Quantitative curation funnel across 448 upstream rows (363 Core + 85 Synthetic) yielding Clean-114 (228 cases), Artifact-6 (12 sensitivity cases), and SynMut-50 (100 cases) across 126 distinct designs.

Source AssetOrigin / RoleCommit SHALicenseRetained Pairs
ChipBench [21]Debug candidates + self-contained specsb4ba3a067598...MIT17 Clean (34c)
RTLRepairBench RealBugs [25]LLM fails + VerilogEval v2 refs17709198a893...Apache-2.0 / MIT97 Clean (+6 Art.)
RTLRepairBench Sem85 [25]Single-op mutants (SynMut-50)17709198a893...Apache-2.0 / MIT50 (100c)
Clean-114 + SynMut-5042 Clean Matched (+2 Art. overlap, 6 Syn-only)——164 (+6 Art.)

Table 6. Locked upstream repositories, commits, licenses, and curated RTLReviewBench pair counts across Clean-114, Artifact-6, and SynMut-50.

  • ChipBench (89 → 49 → 17): −8 missing self-contained specs (Prob018, Prob019, Prob021, Prob022); −17 tasks exhibiting upstream spec–ref conflicts (Prob005, Prob006, Prob007, Prob015, Prob023, Prob030); −11 non-sequential tasks lacking posedge clk (Prob011, Prob016, Prob017, Prob024, Prob025); −2 buggy compile failures; −2 reference simulation mismatches. Unique-design deduplication yields 17 Clean sequential pairs (11 arithmetic, 6 assignment).

  • RTLRepairBench RealBugs (274 → 251 → 103 → 97 Clean): −6 specification hint leakage rows (Prob062_bugs_mux2, Prob132_always_if2); −2 reference comment leakage rows (Prob114_bugs_case); −9 reference testbench failures (Prob034, Prob092, Prob094); −5 reference compile port mismatches; −1 buggy SV cast error. Unique-design deduplication produces 103 pairs, of which 6 (Artifact-6: P018, P020, P022, P028, P047, P084) are documented as simulation-only/#delay sensitivity cases, leaving 97 Clean pairs (90 realbug_base, 7 realbug_tuned; 63 sequential, 34 combinational).

  • RTLRepairBench Sem85 → SynMut-50 (85 → 84 → 50): 1 static exclusion followed by Icarus 11.0 regression across 84 pairs (168 logs). Balanced rarest-operator deduplication retains 50 unique designs (36 seq, 14 comb; 14 op_swap, 11 swap_logical, 10 swap_ternary, 10 flip_equality, 3 off_by_one_literal, 2 flip_reset_polarity), sharing 42 Clean overlapping designs (Matched-42) plus 2 Artifact-6 designs (Prob010, Prob104) with RealBugs.

Full Results

This appendix reports every metric behind the main-text summaries. Table 7 lists the models and studies; Table 8 gives the complete Clean-114 results with Full-120 sensitivity, calibration, and ensembles; Table 9 the Hard split; Table 10 SynMut-50 and Matched-42; Table 11 Reason-First and pairwise review; Table 12 the 2×22 \times2 ablation; Table 13 self-consistency; and Table 14 the explanation audit.

ModelSynMut-50 Full Track (N=100N = 100) Acc. (95% CI)FAR (/50)FRR (/50)Clean Matched-42 Base Designs (84 Cases Each) Real → Syn Acc.prawp_{\mathrm{raw}} / pHolmp_{\mathrm{Holm}}Real → Syn FARReal → Syn FRR
claude-haiku-5.5-thinking99.0% [94.6,100]0.0% (0)2.0% (1)95.2% → 98.8%0.250 / 1.002.4% → 0.0%7.1% → 2.4%
gemini-3.1-pro99.0% [94.6,100]0.0% (0)2.0% (1)96.4% → 98.8%0.500 / 1.002.4% → 0.0%4.8% → 2.4%
claude-haiku-5.597.0% [91.5, 99.4]2.0% (1)4.0% (2)96.4% → 97.6%1.000 / 1.004.8% → 0.0%2.4% → 4.8%
gemini-3.8-flash96.0% [90.1, 98.9]2.0% (1)6.0% (3)95.2% → 95.2%1.000 / 1.000.0% → 2.4%9.5% → 7.1%
o4-mini96.0% [90.1, 98.9]0.0% (0)8.0% (4)89.3% → 96.4%0.031 / 0.3757.1% → 0.0%14.3% → 7.1%
claude-4.6-sonnet94.0% [87.4, 97.8]2.0% (1)10.0% (5)91.7% → 94.0%0.625 / 1.004.8% → 2.4%11.9% → 9.5%
gpt-oss-120b87.0% [78.8, 92.9]0.0% (0)26.0% (13)88.1% → 88.1%1.000 / 1.004.8% → 0.0%19.0% → 23.8%
gpt-5.485.0% [76.5, 91.4]2.0% (1)28.0% (14)82.1% → 85.7%0.375 / 1.007.1% → 2.4%28.6% → 26.2%
gemma-4-31b-it82.0% [73.1, 89.0]0.0% (0)36.0% (18)78.6% → 83.3%0.125 / 1.004.8% → 0.0%38.1% → 33.3%
llama-3.3-70b-instruct81.0% [71.9, 88.2]8.0% (4)30.0% (15)73.8% → 81.0%0.109 / 1.0021.4% → 9.5%31.0% → 28.6%
gpt-4.176.0% [66.4, 84.0]2.0% (1)46.0% (23)77.4% → 75.0%0.625 / 1.002.4% → 2.4%42.9% → 47.6%
gpt-4.1-mini73.0% [63.2, 81.4]2.0% (1)52.0% (26)70.2% → 71.4%1.000 / 1.004.8% → 2.4%54.8% → 54.8%
Pooled Across All 12 Models (Matched-42: 504 Buggy / 504 Valid)86.2% → 88.8%—5.6% → 1.8%22.0% → 20.6%

Table 10. SynMut-50 (100 cases / 50 designs) vs. Clean Matched-42 real-vs-synthetic performance across all 12 models (95% Clopper-Pearson CIs), plus pooled SynMut-50 miss breakdown across six operators (600 buggy evaluations). Pooled SynMut-50 Miss Breakdown (10/600 = 1.7% [0.8, 3.0]): swap_ternary 6/120 (5.0% [1.9, 10.6]); flip_reset_polarity 1/24 (4.2% [0.1, 21.1]); off_by_one_literal 1/36 (2.8% [0.1, 14.5]); flip_equality 1/120 (0.8% [0.0, 4.6]); op_swap 1/168 (0.6% [0.0, 3.3]); swap_logical 0/132 (0.0% [0.0, 2.8]). Matched-42 pooled FAR: 28/504 [3.7, 7.9] → 9/504 [0.8, 3.4].

ModelAcc. (VF→RF)ΔAccprawp_{\mathrm{raw}} / pHolmp_{\mathrm{Holm}}FAR (VF→RF)FRR (VF→RF)BrierTok.PoolSwapΔIsoFlips (A/B)BiasNeith.
gpt-4.169.7 → 86.0%+16.2 pp4.3e−7 / 5.2e−61.8 → 8.8%58.8 → 19.3%0.272 → 0.13579 → 26072.8%62.3%+21.9 pp***6 (5/1)+10.5 pp40
gpt-4.1-mini71.9 → 83.8%+11.8 pp5.8e−4 / 0.000582.6 → 14.0%53.5 → 18.4%0.235 → 0.14893 → 29666.7%52.6%+7.0 pp***11 (9/2)+7.5 pp46
gemma-4-31b-it78.9 → 90.4%+11.4 pp1.3e−5 / 1.4e−43.5 → 3.5%38.6 → 15.8%0.210 → 0.09484 → 30082.9%76.3%+18.4 pp***3 (3/0)+1.8 pp34
gpt-5.483.8 → 88.6%+4.8 pp0.043 / 0.4303.5 → 3.5%28.9 → 19.3%0.160 → 0.10770 → 32078.5%72.8%+3.5 pp1 (1/0)−0.9 pp43
o4-mini90.8 → 93.9%+3.1 pp0.065 / 0.5232.6 → 1.8%15.8 → 10.5%0.105 → 0.096874 → 113092.1%89.5%+7.0 pp1 (1/0)+2.2 pp14
gemini-3.1-pro96.9 → 98.2%+1.3 pp0.250 / 1.000.9 → 0.0%5.3 → 3.5%0.030 → 0.018903 → 120498.7%98.2%+4.4 pp0 (0/0)−0.4 pp3
llama-3.3-70b66.2 → 67.1%+0.9 pp0.864 / 1.0026.3 → 36.0%41.2 → 28.9%0.328 → 0.29029 → 25564.9%52.6%+16.7 pp***20 (16/4)+11.4 pp16
claude-4.6-sonnet†93.0 → 93.4%+0.4 pp1.00 / 1.002.6 → 2.6%11.4 → 10.5%0.060 → 0.056384 → 58896.5%95.6%+7.9 pp0 (0/0)−0.9 pp6
gemini-3.8-flash§93.4 → 93.9%+0.4 pp1.00 / 1.000.0 → 0.0%13.2 → 12.3%0.059 → 0.056431 → 67998.7%97.4%+10.5 pp0 (0/0)+0.4 pp3
claude-haiku-5.597.4 → 97.4%0.0 pp1.00 / 1.003.5 → 1.8%18.8 → 2.6%0.031 → 0.032477 → 66895.2%93.9%+0.0 pp0 (0/0)+0.9 pp5
claude-haiku-5.5-think†97.1 → 90.4%−1.3 pp0.549 / 1.001.8 → 1.8%14.9 → 17.5%0.073 → 0.081620 → 75685.1%78.1%−6.1 pp0 (0/0)+1.3 pp29

Table 11. Complete 12-model Reason-First vs. Verdict-First prompting comparison (N=228N = 228 Clean cases each) and Bidirectional Contrastive Pairwise results (114 Clean pairs ×2 directions) with Holm-Bonferroni adjusted pp-values. Note: Clean Reason-First FRR drops cleanly under Holm correction for gpt-4.1 (pHolm=8.2×10−12p_{\mathrm{Holm}} = 8.2 \times10^{-12}), gpt-4.1-mini (pHolm=1.2×10−9p_{\mathrm{Holm}} = 1.2 \times10^{-9}), gemma-4 (pHolm=6.2×10−5p_{\mathrm{Holm}} = 6.2 \times10^{-5}), and llama-3.3 (pHolm=0.040p_{\mathrm{Holm}} = 0.040), while gpt-4-1-mini FAR rises (pHolm=0.012p_{\mathrm{Holm}} = 0.012); llama-3.3 emits 1 valid abstain (c187). †haiku-5.5-thinking had 2 invalids under RF (99.1% cov.) and Pairwise. ‡sonnet had 40.4% Clean RF / 17.5% PW strict JSON. §gem-3.8-flash logged 207.4k/137.8k Clean adaptive reasoning tokens on RF/PW. On Pairwise, * denotes Holm-adjusted significance vs. Isolated Joint-Pair Acc (*pHolm<0.05p_{\mathrm{Holm}} < 0.05, **pHolm<0.01p_{\mathrm{Holm}} < 0.01, ***pHolm<0.001p_{\mathrm{Holm}} < 0.001): gemma-4 pHolm=2.3×10−4p_{\mathrm{Holm}} = 2.3 \times10^{-4}, gpt-4.1 pHolm=4.5×10−4p_{\mathrm{Holm}} = 4.5 \times10^{-4}, gem-3.8-flash pHolm=0.0049p_{\mathrm{Holm}} = 0.0049, llama-3.3 pHolm=0.012p_{\mathrm{Holm}} = 0.012 (sonnet praw=0.022/pHolm=0.18p_{\mathrm{raw}} = 0.022/p_{\mathrm{Holm}} = 0.18, o4-mini praw=0.039/pHolm=0.27p_{\mathrm{raw}} = 0.039/p_{\mathrm{Holm}} = 0.27, gem-3.1-pro praw=0.063/pHolm=0.38p_{\mathrm{raw}} = 0.063/p_{\mathrm{Holm}} = 0.38).

ModelCell (Order, Trace)Clean Acc. (95% CI)FARFRRPair-BothBrierTok.§∆VF (pHolm)
gpt-4.1VF (V→C→R, no)159/228 (69.7%) [63.3, 75.6]1.8%58.8%40.4%0.271683.0—
A: vf_trace (V→C→R, yes)153/228 (67.1%) [60.6, 73.2]3.5%61.4%36.8%0.2994238.0−2.6 pp (0.438)
B: rf_order_only (R→V→C, no)177/228 (77.6%) [71.7, 82.9]10.5%33.3%58.8%0.2017110.0+7.9 pp (0.048∗)
RF (R→V→C, yes)196/228 (86.0%) [80.8, 90.2]8.8%19.3%73.7%0.1346270.5+16.2 pp (5.2e-6†)
gpt-4.1-miniVF (V→C→R, no)164/228 (71.9%) [65.6, 77.7]2.6%53.5%45.6%0.2348100.0—
A: vf_trace (V→C→R, yes)157/228 (68.9%) [62.4, 74.8]3.5%58.8%40.4%0.2618315.0−3.1 pp (0.438)
B: rf_order_only (R→V→C, no)186/228 (81.6%) [75.9, 86.4]13.2%23.7%67.5%0.1611105.0+9.6 pp (0.017∗)
RF (R→V→C, yes)191/228 (83.8%) [78.3, 88.3]14.0%18.4%70.2%0.1477302.0+11.8 pp (0.0058†)
gemma-4-31b-itVF (V→C→R, no)180/228 (78.9%) [73.1, 84.1]3.5%38.6%57.9%0.210095.0—
A: vf_trace‡ (V→C→R, yes)184/228 (80.7%) [75.0, 85.6]1.8%34.2%62.3%0.1823304.5+1.8 pp (0.688)
B: rf_order_only (R→V→C, no)194/228 (85.1%) [79.8, 89.4]2.6%27.2%71.9%0.1444138.5+6.1 pp (0.046∗)
RF (R→V→C, yes)206/228 (90.4%) [85.8, 93.9]3.5%15.8%82.5%0.0940315.0+11.4 pp (1.4e-4†)
Accuracy Decomposition (pp)Total (RF − VF)Order (B − VF)Instr. (A − VF)Inter.pH(RF vs. A)pH(RF vs. B)Full-120 (O / I / Int)
gpt-4.1+16.2 pp+7.9 pp−2.6 pp+11.0 pp5.1e-8***0.012∗+16.2 (+7.9/−1.7/+10.0)
gpt-4.1-mini+11.8 pp+9.6 pp−3.1 pp+5.3 pp1.9e-4***0.688+11.2 (+9.2/−2.9/+5.0)
gemma-4-31b-it+11.4 pp+6.1 pp+1.8 pp+3.5 pp0.0031∗∗0.046∗+11.2 (+6.2/+2.1/+2.9)
Mean (3 Models)+13.2 pp+7.9 pp−1.3 pp+6.6 pp——+12.9 (+7.8/−0.8/+6.0)

Table 12. Study 3c 2×2 factorial Reason-First ablation (vf_trace and rf_order_only; 1,440 new greedy T=0.0T = 0.0 queries) separating JSON schema key order from step-by-step tracing on the three largest Reason-First beneficiaries (Clean-114 primary, N=228N = 228 cases/cell). Note: Holm pp-values for A vs. VF, B vs. VF, RF vs. A, and RF vs. B adjust over the 12-test Study 3c family (∗pHolm<0.05p_{\mathrm{Holm}} < 0.05, ∗∗pHolm<0.01p_{\mathrm{Holm}} < 0.01, ∗∗∗pHolm<0.001p_{\mathrm{Holm}} < 0.001); †RF vs. VF pHolmp_{\mathrm{Holm}} comes from the 12-model Reason-First family (Table 11). §Median completion tokens across all 228 Clean cases (vs. ok-only medians in Tables 8 and 11). ‡All 1,436 valid Study 3c JSON outputs obey requested key ordering (0 violations); only 4/720 vf_trace calls hit 16,384-token truncation inside trailing "reason" (3 Clean on gemma-4: C009, C098, C116; 1 Artifact-6 on gpt-4.1-mini: C167), plus 1 Clean abstain each on gpt-4.1 (A: C051; B: C186). Decomposition components round independently to 0.1 pp.

Model / Regime5-Draw Mean±SDmaj@5 Acc. (95% CI)maj@5 FARmaj@5 FRRPair BothVote Brier5:0 Ok / 5:0 Err / Splitvs. T=0T=0 / RFvs. 3x Div.
claude-haiku-5.5 (5x)96.32 ± 0.50%96.1% [92.6, 98.2]4.4% [1.4, 9.9]3.5% [1.0, 8.7]92.1%0.025213 / 2 / 13 (5.7%)praw=0.375/0.375p_{\mathrm{raw}} = 0.375 / 0.375−2.2 pp (pHM=0.180p_{\mathrm{HM}} = 0.180)
gemini-3.8-flash† (5x)93.25 ± 0.91%93.4% [89.4, 96.3]0.0% [0.0, 1.2]13.2% [7.6, 20.8]86.8%0.055205 / 9 / 14 (6.1%)1.00 / 1.00−14.8 pp (pHM=0.0068∗∗∗p_{\mathrm{HM}} = 0.0068^{***})
gpt-5.4 (5x)84.04 ± 0.96%83.3% [78.4, 87.9]3.5% [1.0, 5.2]29.8% [21.6, 39.3]67.5%0.146184 / 28 / 16 (7.0%)1.00 / 0.029 (p=0.087p = 0.087)−14.9 pp (pHM=3.2e−9p_{\mathrm{HM}} = 3.2e-9)
Top-3 Diverse Maj. (3x)— (T=0.0T = 0.0)98.2% [95.6, 99.5]0.9% [0.0, 4.8]2.6% [0.5, 7.5]96.5%———Reference

Table 13. Study 6 stochastic self-consistency (majority@5 at T=0.7T = 0.7, 1,140 Clean draws/model) vs. deterministic single-pass and cross-family voting across three commercial providers (95% Clopper-Pearson CIs in brackets). Note: Individual Clean T=0.7T = 0.7 draws span [95.6, 96.9]% (haiku-5.5), [92.1, 94.3]% (gem-3.8-flash), and [82.9, 85.5]% (gpt-5.4). Clean 1× T=0.7T = 0.7 Reason-First baselines are 97.4%/97.4% (haiku-5.5), 93.4%/93.9% (gem-3.8-flash), and 83.8%/88.6% (gpt-5.4). †gemini-3.8-flash used 765.9k Clean (786.4k Full) adaptive reasoning tokens. Holm-adjusted significance across the 3 models per baseline: ∗∗pHolm<0.01p_{\mathrm{Holm}} < 0.01, ∗∗∗pHolm<0.001p_{\mathrm{Holm}} < 0.001.

Reviewer ModelTPs Audited (TP/114TP/114)GroundedSpurious ClaimVagueDisp.Grounded Prec./TPNominal TPREff. Grounded Sens. (∆Gap)
gemini-3.1-pro113113000100.0% [96.8, 100.0]99.1%99.1% [95.2, 100.0] (0.0 pp)
gemini-3.8-flash†11411130097.4% [92.5, 99.5]100.0%97.4% [92.5, 99.5] (−2.6 pp)
claude-haiku-5.5110110000100.0% [96.7, 100.0]96.5%96.5% [91.3, 99.0] (0.0 pp)
claude-4.6-sonnet11111010099.1% [95.1, 100.0]97.4%96.5% [91.3, 99.0] (−0.9 pp)
claude-haiku-5.5-thinking†11111010099.1% [95.1, 100.0]97.4%96.5% [91.3, 99.0] (−0.9 pp)
o4-mini11110830097.3% [92.3, 99.4]97.4%94.7% [88.9, 98.3] (−2.6 pp)
gemma-4-31b-it11010730097.3% [92.2, 99.4]96.5%93.9% [87.8, 97.5] (−2.6 pp)
gpt-oss-120b11210750095.5% [89.9, 98.5]98.2%93.9% [87.8, 97.5] (−4.4 pp)
gpt-5.411010550095.5% [89.7, 98.5]96.5%92.1% [85.5, 96.3] (−4.4 pp)
gpt-4.1-mini11110380092.8% [86.3, 96.8]97.4%90.4% [83.4, 95.1] (−7.0 pp)
gpt-4.111210390092.0% [85.3, 96.3]98.2%90.4% [83.4, 95.1] (−7.9 pp)
llama-3.3-70b-instruct84471223256.0% [44.7, 66.8]73.7%41.2% [32.1, 50.8] (−32.5 pp)
Pooled Clean Total (12 Models)1,3091,2349023294.3%95.7%90.2% (−5.5 pp)

Table 14. Blinded multi-judge rationale fidelity audit across all 1,309 Clean-114 True Positive buggy verdicts from all 12 models (N=114N = 114 Clean buggy cases each; primary panel gpt-5.4 + claude-4.6-sonnet: 92.6% raw agreement, κ=0.536\kappa= 0.536, 97 tiebroken by gemini-3.1-pro; 95% Clopper-Pearson CIs in brackets). †Manual verification of all 50 Clean (60 Full) consensus spurious_claim tags confirms 48/50 (96.0% [86.3, 99.5]; 58/60 = 96.7% Full) genuine false defect claims; only two Clean items reflect judge misses—gemini-3.8-flash C047 (valid De Morgan identity; human-verified flash Prec. = 98.2% [93.8, 99.8], EffSens = 98.2% [93.8, 99.8]), 112/114—and claude-haiku-5.5-thinking C011 (correctly flagged frozen q <= {q[5], q[4:0]}; human-verified haiku-5.5-thk Prec. = 100.0% [96.7, 100.0], EffSens = 97.4% [92.5, 99.5], 111/114). Stratified audit of 96 Full-120 consensus grounded rationales (8/model) finds 0/96 (0.0% [0.0, 3.8]) fabricated mechanisms and 94/96 (97.9%) strict unanimous matches.

Dataset SlicePairs (NN)Cases (2N2N)Seq / CombRTL Lines of Code MeanRTL Lines of Code Med [Min–Max]Specification Words MeanSpecification Words Med [Min–Max]Bug-1st Order
ChipBench (Sequential, Clean)173417 / 048.643.5 [11–107]249.9254.0 [144–374]6 / 17
RTLRepairBench Base (Clean)9018061 / 2926.220.0 [6–134]148.2117.5 [45–496]57 / 90
RTLRepairBench Tuned (Clean)7142 / 512.811.5 [6–27]85.765.0 [41–170]5 / 7
Clean Sequential / Combinational80 / 34160 / 6880 / 3433.5 / 17.527.0 / 13.0175.2 / 122.6144.5 / 110.047/80 ; 21/34
Clean-114 Primary Track11422880 / 3428.722.0 [6–134]159.5132.0 [41–496]68 / 114
Excluded Artifact-6 (3 sim + 3 #delay)6122 / 49.88.5 [5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18]71.560.0 [38–145]4 / 6
Full-120 Sensitivity Core12024082 / 3827.821.0 [5–134]155.1124.5 [38–496]72 / 120
SynMut-50 Contrast Track5010036 / 1422.620.0 [9–58]151.5117.5 [42–496]23 / 50

Table 5. Descriptive structural statistics across Clean-114, excluded Artifact-6, Full-120, and SynMut-50 over 126 unique designs.

Note: Seven models report strictly 0 API reasoning_tokens across all extension studies; gemini-3.8-flash allocates adaptive hidden reasoning (59.3k SynMut-50, 207.4k [212.7k Full] Reason-First, 137.8k [141.0k] Pairwise, 765.9k [786.4k] SC@5), folded into completion_tokens during Core (Section E). Study 3c (2×2 Reason-First ablation: vf_trace and rf_order_only) evaluates gpt-4.1, gpt-4.1-mini, and gemma-4-31b-it (480 queries each; 1,440 total; 0 reasoning_tokens).

¶99.6% coverage (227/228; valid case C206 failed JSON quotes → 2 FP + 1 TN−1 = 3/114 valid-case errors; selective acc 97.8% [94.9, 99.3]). §33.3% Clean (36.2% Full) strict JSON / 100% recovery (11/13 FPs in prose-prefixed CoT vs. 2/76 in strict JSON). ¶Accuracy on agreement slice; remaining disagreements (9 for H==P31, 19 for H==Fl) routed to EDA simulation. Abbreviations: H=claude-haiku-5.5, P31=gemini-3.1-pro, Fl=gemini-3.8-flash, Oss=gpt-oss-120b, Son=claude-4.6-sonnet.

Error Taxonomy and Case Studies

ClassHardware Failure Mechanism (Representative Case IDs)Haiku 5.5Hk-ThkGem 3.1PGem 3.8FGPT-4.1m
Clean-114 False Acceptances (FN: Buggy RTL Accepted as correct)
FN-1Clean simulation-only / init artifacts (excluded via Artifact-6)00000
FN-2Sequential control, shift slice, pipeline, & async reset / FSM (C003, C020, C209)33102
FN-3Combinational priority order & bit indexing (C184)10001
Clean Sub. FN43103
Clean-114 False Rejections (FP) & Schema Errors (I)
FP-1X-prop (1’bx), SV dialect (output reg+assign), latch purism (C139, C187, C240)00159
FP-2Latency, cycle threshold, reset convention, & waveforms (C004, C019, C130, C158)112422
FP-3Shift-width context, FSM encoding / K-map, & port bounds (C017, C025, C151, C230)113627
FP-4Autoregressive verdict–rationale self-contradiction (C068, C091, C150)00003
I-1Unescaped literal quotes breaking JSON string syntax (C206)01000
Clean Sub. FP+I2361561
Clean-114 TotalPer-Model Total (228 Clean evaluations/model)6671564
Artifact-6 Excl.FN (C035, C039, C044, C055, C094, C167) / FP (C168 q_in port)6 / 06 / 05 / 03 / 04 / 1
Full-120 Sens.Per-Model Total (240 evaluations/model)1212121869

Table 15. Mutually exclusive Clean-114 RTL error taxonomy across five commercial models (228 Clean evals/model), with Artifact-6 reconciliation to Full-120 (240 evals). Under Reason–First, FP-4 self-contradictions drop to 0 across all 12 models.

Case Study 1 — Frozen Shift Slice (C003, FN-2) & Parameter Width Truncation (C024). In Prob002_extraneous_items_sequence_detect (Listing 2), bits a_tem[7:0] are reassigned back to themselves instead of shifting (a_tem <= {a_tem[7:0], a}), freezing the lower eight bits while leaving assign match = match_b && match_f combinational as in the golden reference (C004). Yet claude-haiku-5.5 (c=0.60c = 0.60) claimed “The 9-bit window a_tem holds the last nine inputs...”, missing the self-looping slice (conversely, gemma-4 rejected C003 on the false claim that assign match omits a pipeline stage, earning a spurious_claim TP). On C024 (Prob003_ non-overlapping_sequence_detect), parameter sf6 = 4’d20 truncates 20 mod 16 = 4, aliasing failure state sf6 with s4=4’d4 and causing 352/1,215 mismatches—a real Verilog bit-width trap missed by gpt-5.4 and gpt-4.1-mini.

Listing 2: Frozen shift-register slice missed by claude-haiku-5.5 (C003, FN-2); assign match matches golden reference C004.

always @(posedge clk or negedge rst_n)
    if (!rst_n) a_tem <= 9’b0;
    else        a_tem <= {a, a_tem[7:0]}; // BUG: Bits [7:0] freeze instead of {a_tem[7:0], a}
assign match = match_b && match_f;          // Matches golden reference C004 structure

Case Study 2 — Verilog Bit-Width Context Rules (C017, FP-3) vs. Async-Reset Miss Masked by SV Purism (C209, Spurious TP / FN-2). Hardware semantic traps cut both ways (Listing 3): on valid C017, gemini-3.8-flash (c=0.98c = 0.98) misapplied self-determined width rules, claiming mul_a << i truncates at 4 bits even though IEEE 1800 context-determined assignment propagates the 8-bit LHS width of temp[i]. Conversely, on buggy Clean-114 C209 (Prob136_m2014_q6), the spec states “all sequential logic is triggered on the positive edge of the clock,” yet the candidate uses an asynchronous reset (always @(posedge clk or posedge reset)). Both gpt-4.1 and gpt-5.4 overlooked the reset defect entirely and earned nominal True Positives for the wrong reason: claiming output reg z driven by assign z = (currentState == E || currentState == F) is invalid RTL, even though SystemVerilog (-g2012) compiles it cleanly; inversely, gemini-3.1-pro correctly accepted the SV idiom yet missed the asynchronous reset (FN-2, its sole Clean-114 miss). The same SV dialect purism drives FP-1 rejection of valid C139.

Listing 3: IEEE 1800 context-determined width propagation misdiagnosed as truncation on C017 (gemini-3.8-flash, FP-3).

wire [N - 1 : 0] temp [0 : 3]; // N = 2 * size = 8 bits
genvar i;
generate
    for (i = 0; i < 4; i = i + 1) begin : loop
        assign temp[i] = mul_b[i] ? mul_a << i : ’d0; // Context-determined 8b via LHS temp[i]!
    end
endgenerate

Case Study 3 — Verdict-Before-Reason Contradictions Resolved Under Reason-First (C045, C049, C091, C150, C204). Under Verdict-First, shallow-budget models commit a verdict before completing truth-table verification: on C045 (~a ^ ~b for XNOR), gemma-4-31b-it output "verdict":"correct" (c=1.00c = 1.00), derived the truth table mid-field, and concluded “This is XOR. The spec asks for XNOR. The implementation is buggy.” On valid C150, gpt-4.1-mini wrote: “The initial assessment of buggy is incorrect. The code is correct.” Switching to Reason-First resolved all 15/15 Clean closing contradictions (16/16 Full) cleanly to ground truth, with zero remaining across 2,736 Clean responses.

Case Study 4 — Benchmark-Validity Finding: Simulation-Only Oracles (Artifact-6) vs. Spec Discrepancy (C151). Separating upstream simulation-oracle artifacts from true model deficits is essential for hardware benchmarks. In Artifact-6, Icarus 11.0 flags always @(*) out = in (C039) over a time-0 scheduling delta (2/100 mismatches at t=5t = 5 that all 12 LLMs independently classify as correct), penalizes synthesizable flip-flops lacking non-synthesizable initial statements (C094, C167), and rejects Boolean-equivalent modules containing explicit #5/#10 inertial delays (C035, C044, C055)—accounting for half of all naive Full-120 FNs (59/118). Conversely, on Clean114’s sole 0/12 case C151 (Prob093_ece241_2014_q3), every reviewer faithfully reads cd=11, ab=10 -> 1 (c|d) from the ASCII Karnaugh map and rejects canonical RefModule (assign mux_in[2] = d), which evaluates to 0 at that cell while still passing its upstream testbench—highlighting how subtle specification-to-reference discrepancies can masquerade as universal model failure.

Endpoint Reproducibility, Telemetry, and Compute Ledger

All 17,623 evaluated inference and audit calls executed live against an OpenAI-compatible chat-completions API gateway during a single frozen campaign window (November 9–10, 2026; local timestamps 2026-11-09T23:34 through 2026-11-10T22:21) with JSON mode (response_format={"type":"json_object"}), temperature=0.0 for Studies 1–5 (temperature=0.7 for Study 6), and max_tokens=16384 in all extension/audit runners (provider default in Core). Across all gateway responses, system_fingerprint returned unset (None), and response-level served_model fields matched the requested model_id across all 8 models with echo telemetry in Studies 2–5 and all 3 models in Study 3c (10 distinct models: claude-haiku-5.5, claude-haiku-5.5-thinking, gemini-3.1-pro, gemini-3.8-flash, gemma-4-31b-it, gpt-4.1, gpt-4.1-mini, gpt-5.4, gpt-oss-120b, and o4-mini).

As noted in § 4, four reviewers (haiku-5.5-thinking, gemini-3.1-pro, gpt-oss-120b, and o4-mini) allocate hidden reasoning tokens across Core and extension runs, whereas gemini-3.8-flash logs adaptive reasoning_tokens (59.3k SynMut-50, 207.4k [212.7k Full] Reason-First, 137.8k [141.0k Full] Pairwise, 765.9k [786.4k Full] SC@5) and folds them directly into completion_tokens during Core evaluation. Only claude-4.6-sonnet exhibited frequent prefixed chain-of-thought prose (sometimes with markdown backticks) before its JSON object, yielding 33.3% Clean (36.2% Full) strict-JSON compliance on Core, 43.0% on SynMut-50, 40.4% Clean (42.5% Full) on Reason-First, and 17.5% Clean (20.8% Full) on Pairwise, with 100.0% valid balanced-brace JSON recovery. Across Study 3c (1,440 calls), all 1,436 valid JSON responses obey the requested key order (0 violations), while the 4 invalid outputs (vf_trace only: gemma-4-31b-it C009, C098, C116; gpt-4.1-mini C167 in Artifact-6) emit "verdict" first and truncate at the 16,384-token cap inside trailing "reason". Table 16 reconciles exact query and token totals across the complete executed API matrix (Full-120 + extensions + audits).

Study / Experimental MatrixQueriesPrompt Tok.Compl. Tok.Reas. Tok.†Total Tok.Decoding
Study 1: Core Verdict-First (12 × 240)2,8801,871,0931,637,1561,019,8653,508,249Greedy (T=0.0T = 0.0)
Study 2: SynMut-50 Synthetic (12 × 100)1,200722,676605,158426,8481,327,834Greedy (T=0.0T = 0.0)
Study 3a: Reason-First Prompting (12 × 240)2,8801,967,3332,368,8051,363,1414,336,138Greedy (T=0.0T = 0.0)
Study 3b: Self-Consistency maj@5 (3 × 1,200)3,6002,445,8001,732,738786,4194,178,538Sample (T=0.7T = 0.7)
Study 3c: 2×2 Reason-First Ablation (3 × 2 × 240)1,440893,836413,70801,307,544Greedy (T=0.0T = 0.0)
Study 4: Bidirectional Pairwise (12 × 240)2,8802,576,2781,731,2291,211,1114,307,507Greedy (T=0.0T = 0.0)
Study 5: Blinded Rationale Audit‡2,743———3,863,696Greedy (T=0.0T = 0.0)
Grand Total (All 7 Studies)17,62310,477,0168,488,7944,807,38422,829,506—

Table 16. Complete executed API query and token ledger across all empirical studies (22,829,506 total tokens across 17,623 live gateway calls on Full-120 + extensions; 0 simulated/fallback entries).

References

  1. [1]Charles J. Clopper and Egon S. Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4):404–413, 1934.DOI
  2. [2]Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46, 1960.DOI
  3. [3]Fan Cui, Hongyuan Hou, Zizhang Luo, Chenyun Yin, and Yun Liang. HWE-Bench: Benchmarking LLM agents on real-world hardware bug repair tasks. arXiv preprint arXiv:2604.14709, 2026.arxiv.org/abs/2604.14709
  4. [4]Dario Garcia-Gasulla, Gokcen Kestor, Emanuele Parisi, Miquel Alberti-Binimelis, Cristian Gutierrez, Razine Moundir Ghorab, Orlando Montenegro, Bernat Homs, and Miquel Moreto. TuRTL: A unified evaluation of LLMs for RTL generation. arXiv preprint arXiv:2504.01986, 2025.
  5. [5]Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70, 1979.DOI
  6. [6]Hongchao Jiang, Yiming Chen, Yushi Cao, Hung-yi Lee, and Robby T. Tan. CodeJudgeBench: Benchmarking LLM-as-a-judge for coding tasks. arXiv preprint arXiv:2507.10535, 2025.arxiv.org/abs/2507.10535
  7. [7]Haolin Jin and Huaming Chen. Are LLMs reliable code reviewers? systematic overcorrection in requirement conformance judgement. arXiv preprint arXiv:2603.00539, 2026.
  8. [8]Pengwei Jin, Di Huang, Chongxiao Li, Shuyao Cheng, Yang Zhao, Xinyao Zheng, Jiaguo Zhu, Shuiyi Xing, Bohan Dou, Rui Zhang, Zidong Du, Qi Guo, and Xing Hu. RealBench: Benchmarking Verilog generation models with real-world IP designs. arXiv preprint arXiv:2507.16200, 2025.arxiv.org/abs/2507.16200
  9. [9]Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, and Haoxing Ren. VerilogEval: Evaluating large language models for Verilog code generation. arXiv preprint arXiv:2309.07544, 2023.arxiv.org/abs/2309.07544
  10. [10]Yao Lu, Shang Liu, Qijun Zhang, and Zhiyao Xie. RTLLM: An open-source benchmark for design RTL generation with large language model. In 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC), pages 722–727, 2024.arxiv.org/abs/2308.05345
  11. [11]Quinn McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157, 1947.
  12. [12]Xiankai Meng, Kejian Feng, Xinlin Zhao, Zhuo Zhang, Yan Lei, Xiaoguang Mao, and Jiang Wu. VeriBugBench: An empirically grounded framework for constructing Verilog RTL debugging benchmarks. arXiv preprint arXiv:2609.18022, 2026.arxiv.org/abs/2609.18022
  13. [13]Nathaniel Pinckney, Christopher Batten, Mingjie Liu, Haoxing Ren, and Brucek Khailany. Revisiting VerilogEval: A year of improvements in large-language models for hardware code generation. ACM Transactions on Design Automation of Electronic Systems, 30(6):91:1–91:20, 2025. arXiv:2408.11053.arxiv.org/abs/2408.11053
  14. [14]Nathaniel Pinckney, Chenhui Deng, Chia-Tung Ho, Yun-Da Tsai, Mingjie Liu, Wenfei Zhou, Brucek Khailany, and Haoxing Ren. Comprehensive Verilog design problems: A next-generation benchmark dataset for evaluating large language models and agents on RTL design and verification. arXiv preprint arXiv:2506.14074, 2025.arxiv.org/abs/2506.14074
  15. [15]Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. Let me speak freely? a study on the impact of format restrictions on large language model performance. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1218–1236, 2024.
  16. [16]Shailja Thakur, Baleegh Ahmad, Hammond Pearce, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri, and Siddharth Garg. VeriGen: A large language model for Verilog code generation. ACM Transactions on Design Automation of Electronic Systems, 29(3):1–31, 2024.arxiv.org/abs/2308.00708
  17. [17]Yun-Da Tsai, Mingjie Liu, and Haoxing Ren. RTLFixer: Automatically fixing RTL syntax errors with large language models. In Proceedings of the 61st ACM/IEEE Design Automation Conference (DAC), 2024. arXiv:2311.16543.arxiv.org/abs/2311.16543
  18. [18]Jing Wang, Shang Liu, Wenji Fang, Yuchao Wu, Yugao Zhu, and Zhiyao Xie. RTL-BenchLS: A large-scale benchmark for RTL reasoning and generation with large language models. arXiv preprint arXiv:2606.08976, 2026.arxiv.org/abs/2606.08976
  19. [19]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR), 2023. arXiv:2203.11171.arxiv.org/abs/2203.11171
  20. [20]Stephen Williams. Icarus verilog: Open-source verilog simulation and synthesis tool. https://steveicarus.github.io/iverilog/, 2020.
  21. [21]Zhongkai Yu, Chenyang Zhou, Yichen Lin, Hejia Zhang, Haotian Ye, Junxia Cui, Zaifeng Pan, Jishen Zhao, and Yufei Ding. ChipBench: A next-step benchmark for evaluating LLM performance in AI-aided chip design. arXiv preprint arXiv:2601.21448, 2026.arxiv.org/abs/2601.21448
  22. [22]Zhengran Zeng, Ruikai Shi, Keke Han, Yixin Li, Kaicheng Sun, Yidong Wang, Zhuohao Yu, Rui Xie, Wei Ye, and Shikun Zhang. SWR-Bench: Assessing LLM performance in real-world code review comment generation. Proceedings of the ACM on Software Engineering, 3(FSE):FSE137, 2026. arXiv:2509.01494.arxiv.org/abs/2509.01494
  23. [23]Dewu Zheng, Yanlin Wang, Xiwen Wang, Kefeng Duan, Hongyu Zhang, Xilin Liu, Yuchi Ma, and Zibin Zheng. From static to dynamic: Benchmarking real-world code review with MCR-Bench. Proceedings of the ACM on Software Engineering, 3(ISSTA):ISSTA128, 2026. arXiv:2608.27442.arxiv.org/abs/2608.27442
  24. [24]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. arXiv:2306.05685.arxiv.org/abs/2306.05685
  25. [25]Yeyin Zhu and Rishabh Ranawalt. What do RTL repair benchmarks actually measure? In NeurIPS Workshop on AI for Chip Design, 2026. arXiv:2610.28595.arxiv.org/abs/2610.28595
  26. [26]Qingyun Zou, Feng Yu, Hongshi Tan, Jiahao Cui, Bingsheng He, and Weng-Fai Wong. Is agentic AI ready for real-world hardware engineering? a deep dive with Phoenix-bench. arXiv preprint arXiv:2605.15226, 2026.arxiv.org/abs/2605.15226

Paper details

vlnc:W_469s2kdm
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents