betaFeedback

WorksComputer Science

Selecting LLM-Generated Hypotheses Raises Truth Beyond What an LLM Panel Expects: A Pre-registered Study on NAS-Bench-201 and HPOBench

Abstract

LLM research-ideation systems aim for hypotheses that are both true and non-obvious, yet neither property can normally be checked. We build a testbed in which both can. Generators propose conditional interaction claims about two fully tabulated benchmarks, NAS-Bench-201 and HPOBench; every claim is verified exactly against the tables, and its non-obviousness is scored against a calibrated blind prior elicited from other LLMs. Excess truth measures how much more often a run's claims hold than this panel expects. Across a series of pre-registered stages with Claude and GPT generators, our main positive result is a simple two-stage step found by a harness optimizer: draft a large pool of claims, then select a subset in a separate call. For Sonnet on NAS-Bench-201, this step raised excess truth in every pre-registered test, including against a same-pool ablation that keeps only likely-true claims, without lowering precision. The gain comes both from credit for true claims and from avoiding near-miss false ones. In post hoc checks it held under every calibration, panel and truth rule we tried, and it exceeded random and type-filter selection from the same pools. Because kept claims recur across runs, the evidence weakens when runs and claims are resampled jointly, and it is not significant against Sonnet's own prior. Two secondary findings are also pre-registered. GPT makes real use of the results of cheap probes, beating shuffled and withheld controls, though not more efficiently than asking for more claims. Optimizing the harness for an LLM judge raised judge scores while sharply cutting precision for every generator.

Introduction

Automated research systems now propose ideas, run experiments and write papers ([32, 57]). A growing line of work improves them by optimizing the scaffold around the model: prompts ([3]), agent workflows ([18]), and whole harnesses ([56]). With an executable score this works. Recent systems improve their own research agents by accepting only rewrites that raise a private held-out score ([48]), or turn papers into executable environments for repeated improvement ([52]). Research ideation has no such score. Whether an idea is true is known only after someone tests it, and whether it is surprising depends on what the field already believes. Ideation systems are therefore selected with an LLM judge’s novelty and quality ratings, or with similarity to later papers ([31]), and there are reasons to distrust both. Judge-rated novelty is negatively correlated with how ideas fare in hindsight ([25]), judges reward style over substance ([12, 24]), and under optimization proxy rewards are exploited ([15, 58]).

We built a testbed in which both halves of the objective can be measured. Hypotheses are conditional claims about two fully tabulated benchmarks, so every claim is checked exactly against the table. Surprise is scored against blind ratings of every possible claim by a panel of other LLMs, calibrated to truth over the whole claim space: a run scores R>0R > 0 when its claims hold more often than the calibrated panel predicted. In a pre-registered study with Claude and GPT generators we compared direct generation, which asks for ten hypotheses in one call, with evolved harnesses, with different optimizer fitness functions, and with measurement, a budget of about 100 cheap experiments read before proposing. This paper reports what raised verified surprise and how robust each gain is.

Findings.

  • A two-stage selection step raises excess truth relative to a calibrated LLM panel without losing precision. § 4.1 reports this finding for one generator on one benchmark. An optimizer-evolved harness, singled out post hoc among 19 harness evaluations and then confirmed in fresh runs, drafts 30 candidate claims and keeps 10 in a second call. Its rule targets claims the generator believes at ≥55%\ge55\% but expects other models to doubt at <50%< 50\%. For Sonnet on NAS-Bench-201, this step raised RR by 1.77 per run over direct generation, by 1.95 over a new batch of direct runs, and by 1.96 over a likely-true selection ablation. Each of these three pre-registered tests gave p<.001p < .001, although all three share the same 24 harness runs. Precision rose from .79 to .83. Post hoc, the step also beat random and claim-type selection from the identical pools, by 1.55 and 0.95 per run. Separately elicited blind priors do not match the rule’s literal thresholds, so the rule’s wording describes its design prompt rather than a verified internal mechanism.

  • GPT uses informative experimental values, though not more cheaply than pooling. In pre-registered tests reported in § 4.2, GPT did better after reading 100 cheap probes than after reading the same probes with shuffled or withheld values, on both benchmarks. A cheaper single-call baseline of 40 claims matched it on HPOBench, however, and on NAS-Bench-201 measurement pushed precision below the pre-registered bound of 0.8 times direct generation.

  • Optimizing a harness for an LLM judge inflated judge ratings while collapsing precision. § 4.4 shows that the harness evolved for the judge’s novelty-plus-quality score raised judge ratings on both benchmarks and for all three generators, while cutting precision by 46–68%. This result was pre-registered. Post hoc, the drop holds under every truth definition we tested and, on HPOBench, within claim type.

What the gain is. Two qualifications apply to the first finding throughout. The first is uncertainty. Resampling runs alone gives a 90% interval of [1.30, 2.25] against direct generation, but the kept claims recur across runs, and resampling propositions as well widens it to [0.38, 3.30]. Under this joint resampling the gain is above zero against the Claude and Claude+GPT panels, with p<.024p < .024, but not against Sonnet’s own blind prior, where pxp_x ranges from .06 to .19. The second is composition. RR is credit for true claims, ∑true(1−π)\sum_{\mathrm{true}}(1-\pi), minus cost for false ones, ∑falseπ\sum_{\mathrm{false}}\pi. At the main truth threshold of 2 points, credit accounts for +1.04 of the +1.77 gain and is only marginal with propositions resampled, at px=.054p_x = .054; the remaining +0.73 is avoided cost on false claims, mostly near misses. At thresholds of 1 and 1.5 points credit is nearly the whole gain, and at 3 points credit and avoided cost are about equal. The defensible reading is that the selected claims are true more often than a calibrated Claude panel expects, partly by avoiding claims that miss the threshold narrowly.

Table 1 gives the status of each claim. The study also produced negative results, which we report in § 5 as lessons for evaluating ideation: our original surprise metric was biased by direction and manufactured a confirmed measurement gain for Claude generators, and other harness variants did not beat direct generation.

Related Work

Ideation systems and their evaluation. Systems generate ideas with retrieval, role pipelines, evolutionary search or role-wise RL ([4, 29, 31]), and some rank hypotheses in tournaments judged by LLMs ([16]). They are evaluated by human experts ([44]), by executing the ideas ([43]), against datasets that can verify a hypothesis (Majumder et al., 2024; Liu et al., 2025), or by LLM judges, whose reliability has been studied statically [25, 24, 46]; domain experts disagree with judges that call model-written research questions highly novel [46]. Simulated environments test the full discovery cycle [23]. Audits of AI-scientist systems find failures that final outputs hide [34]. In science itself, surprising combinations of contents and contexts are associated with impact [42]. Outcome predictors have been proposed as rewards [53]. We check ideas against measured truth inside an optimization loop and score surprise against a calibrated panel.

ClaimEvidenceStatus
Stage-3 selection step raises RR at equal or higher precision (Sonnet, NB201)Stages 13–15Pre-registered; supported in all three tests
robust to calibration, panel and truth definitionApp. DPost hoc; holds under all 14 calibrations, 8 panels and 6 truth definitions (τ\tau 1–3 pp)
robust to resampling the recurring propositionsApp. FPost hoc; yes under the Claude and Claude+GPT panels (p×≤.024p_{\times} \leq.024); not under Sonnet’s own prior (p×=.06p_{\times} = .06–.19)
beats random and claim-type selection from its own poolsApp. CPost hoc; yes (+1.55, +0.95); gain concentrated in about 10 propositions
gain is credit for true claims, not only avoided errorsApp. FPost hoc; credit +1.04 of +1.77; with propositions resampled p×=.028p_{\times} = .028–.054
a selector given the scoring panel’s priors does betterApp. CPost hoc; yes (+3.42 against the harness’s +2.87); the harness has no access to them
kept claims follow the generator’s own beliefs (the rule’s wording)App. CPost hoc; not supported (27% of kept vs 45% of unkept rated ≥55\geq55 by Sonnet’s blind prior)
GPT: measurement uses informative values (beats shuffled and withheld)Stage 15Pre-registered; supported in 4 of 4
GPT: measurement beats same-cost direct baselines on RRStages 13–15Pre-registered on pp alone; supported in 5 of 6 tests (refuted against the 40-claim call on HPO). On NB201 precision is .61–.72 times every same-cost arm’s, below the 0.8 bound (post hoc reading); per distinct claim, not significant against pooled + rule on HPO (post hoc). No budget-efficient gain shown
GPT: measurement keeps precision ≥0.8×\geq0.8\times directStage 10Pre-registered; met on HPO, missed on NB201 (.69→.50)
Opus: measurement gains with GPT in the panel (NB201)Stage 12bPre-registered as a robustness test of a predicted null, which failed (Holm p<.05p < .05); panel- and calibration-dependent
Opus and GPT gain under informed panels and own priorSection 4.3Post hoc; hypothesis-generating (panel quality and frontier-model membership confounded)
The one judge-optimized harness: judge score up, precision downStages 5, 9, 10Pre-registered; supported in every pair (one harness, one optimizer run)
precision lower within claim typeApp. HPost hoc; yes on HPO; cannot be separated from type shift on NB201
Measurement raises svs_v for Claude generatorsStages 5, 6, 9Supported under the original metric; refuted by the pre-registered stage-12 rescoring (Section 5.1)

Table 1. Status of the paper’s claims. Table 11 lists every pre-registered criterion and its verdict. RR was introduced post hoc in stage 12 and is pre-registered from stage 12b on.

Eliciting probabilities and surprise. Forecasting the outcomes of experiments has been studied for people and models [6, 33, 17, 53]. Judged probabilities depend on how the outcome space is described [50] and are pulled toward a uniform split over whatever partition is offered [13]. Language models give inconsistent answers to paraphrases of the same question [11], their forecasts on logically related questions can be checked for consistency before outcomes are known [39], and their probability judgments violate probability theory in ways that resemble human errors [59]. Measurement in machine learning needs explicit validity checks [22]. Verbalized

Arms. Direct: one call for ten hypotheses. Measure-first: design 100 probes, read their results, then propose. Judge-optimized: the harness evolved for the judge’s score (stage 2). Stage-3 harness: the harness from the rerun sv optimizer. Scores. Precision: share of a run’s distinct claims that are true. RR: true claims beyond what the calibrated panel expected, per run. sv3: true claims with corrected prior below 50; sv: the same under the original prior. Models. H/S/O/G: Haiku, Sonnet, Opus, GPT.

confidences of language models can be better calibrated than their token probabilities ([49]), but elicitation struggles on questions that need expert knowledge ([55]). Our score credits a true claim by its improbability under a panel’s calibrated prior, not by the belief change of Bayesian surprise ([21]), which AutoDiscovery uses as the reward for open-ended discovery ([2]). Panels from several model families can replace a single judge ([51]); for surprise, we find that the panel’s knowledge relative to the generator matters.

Reward overoptimization. Proxy–gold divergence is well documented for reward models and judges ([15, 5, 58, 37, 47, 27]). Optimizing for human approval can make models more convincing without making them more correct ([54]). Common-mode judge error is not identifiable from judge scores alone ([1]). Our gold signal is a measured scientific table.

Harness optimization and self-improvement. Optimizers search prompts, workflows and harnesses against a validation score ([3, 18, 56]), falling back on the model’s own judgment when no verifier exists ([40]). Self-improving research agents rely on executable scores ([48, 52]) or on reviewer agents that check experimental evidence ([8]). Self-refinement and debate do not correct errors without external feedback ([19, 35, 9, 28]). Agents that run experiments optimize task performance or verify given hypotheses ([26, 45, 14, 20]). Our confirmed harness differs from these in that it uses no external feedback when it runs; its selection call is instructed to keep claims the generator believes but expects other models to doubt (whether it does so is examined in Appendix § 11).

Testbed

Figure 1 summarizes the testbed. Table 2 lists the stages, numbered in the order they were pre-registered, and the notation; Table 11 gives every pre-registered criterion and its verdict.

StageWhat it tested
1Screen of structural harness knobs (NB201, Sonnet, 72 runs)
2, 3Harness optimization with three fitness functions (judge score, number of true claims, sv); 3 reran the sv optimizer after a bug fix
4, 4bMeasurement pilots (8 and 16 runs per arm)
524-run tests on NB201: judge-optimized harness and measurement (Sonnet, Opus)
6–8Measurement controls for Opus on NB201: simulated noise and a cheap proxy (6), shuffled or withheld values (7), designing without results (8)
9Replication on HPOBench (Sonnet, Opus)
10GPT as generator on both benchmarks
11Prompt follow-up on how Sonnet uses evidence
12, 12bThree-way prior and corrected metric; GPT added to the panel
13Judge without explanations; 24-run confirmation of the stage-3 harness; token-matched direct baseline for GPT
14Selection-rule variants (on GPT’s direct pools; ablation of the stage-3 harness); judge retest; GPT as second judge
15New direct runs for Sonnet; shuffled and withheld values for GPT; a single-call 40-claim direct baseline for GPT

Table 2. Stages and notation. NB201: NAS-Bench-201; HPO: HPOBench.

Testbed workflow diagram

Figure 1. Every hypothesis is checked against a fully tabulated benchmark. Surprise is scored against a blind panel of other models that rated every possible claim: RR counts true hypotheses beyond what the calibrated panel expected, and sv3\mathrm{sv}_{3} counts true hypotheses the panel rated below 50. The panel was first asked only for P(higher)P(\mathrm{higher}) (original metric, § 5.1) and from stage 12 on for all three outcomes.

StagePre-registered predictionVerdict
1Structural knobs (retrieval, roles, challenge, selection) shift sv by ≥1\geq1/runInconclusive
1Judge novelty does not track sv (ρ≤.2\rho\leq.2)Supported (ρ=−.24\rho= -.24; novelty vs prior ρ=−.58\rho= -.58)
2Judge fitness raises the judge score without raising svSupported (p=.018p = .018; precision .78→\to.29)
2Validity fitness plays safe (prior ↑≥3\uparrow\geq3, fewer distinct claims)Supported (prior +6.2)
2sv fitness beats the other twoRefuted (final harness = direct generation)
3No harness beats direct generation on svSupported (the sv rerun after a bug fix regressed on fresh runs); post hoc, sv3 agrees, but RR favors that rerun harness (8 runs; confirmed in stage 13)
3Prior and precision trade off across harnessesInconclusive
4, 4bMeasurement raises sv; more probes raise it moreInconclusive (8 and 16 runs/arm); the 8 new Opus runs alone: refuted (+0.12)
5Judge harness: precision ↓≥.2\downarrow\geq.2, judge ↑\uparrow, sv ↓\downarrowPrecision and judge supported; sv refuted / inconclusive
5Measure-first raises sv (24 runs/arm)Supported (Opus), refuted (Sonnet); reversed in stage 12
6Gain survives simulated single-seed noise / doubled noise / 12-epoch proxySupported / inconclusive / inconclusive
7Shuffled and withheld values: is the gain driven by the values or by the design?Both inconclusive
8A design-only harness beats directInconclusive (Opus), refuted (Sonnet)
9HPO: judge harness precision ↓\downarrow, judge ↑\uparrowSupported (both models)
9HPO: judge harness sv ↓\downarrowSupported (Opus), refuted (Sonnet)
9HPO: measure-first raises svSupported (both); stage 12: refuted (Sonnet), inconclusive (Opus)
10GPT, judge harness: precision ↓≥.2\downarrow\geq.2, judge ↑\uparrow, sv ↓\downarrowPrecision and judge supported, sv refuted (both benchmarks)
10GPT, measure-first raises svInconclusive on NB201 (precision .69→\to.50); supported on HPO (+1.08)
11Weak-evidence note raises NB201 precision by ≥.15\geq.15 (Sonnet)Inconclusive (+.04)
11Knowledge filter raises NB201 precision by ≥.15∣Supported(+.15,Holm\geq.15 | Supported (+.15, Holmp = .001$)
11Removing the seed wording lowers HPO precision by ≥.10\geq.10Inconclusive (−.06-.06, Holm p=.007p = .007)
12≥20%\geq20\% of true lower claims are surprising under a three-way priorSupported (92%, 95%)
12Measure-first raises sv3Refuted in three pairs, inconclusive for Opus on HPO
12bStage 12 holds with GPT in the panelNot robust: Opus on NB201 gains (+1.42 sv3, +0.90 RR)
13The judge gain survives removing each hypothesis’s “why” (all six pairs)Partial: kept in four (+0.42 to +1.46), weakened in two (HPO, Sonnet and GPT)
13The stage-3 sv rerun harness raises RR (Sonnet, NB201, 24 runs), precision ≥0.8×\geq0.8\times directSupported (+1.77, p<.001p < .001; precision .79→\to.83)
13GPT: measure-first beats a token-matched direct baseline on RRSupported on both benchmarks: +2.24 (NB201), +0.70 (HPO), Holm p<.001p < .001; pooling alone: −0.64-0.64, +0.24
14GPT: measure-first also beats the baseline with the stage-3 harness’s selection ruleSupported on both benchmarks: +1.16 (NB201), +0.94 (HPO), Holm p≤.002p \leq.002
14The stage-3 harness’s gain comes from its selection rule (likely-true ablation, Sonnet, NB201)Supported (+1.96, p<.001p < .001; ablation vs direct −0.19-0.19)
14Explanations carry part of the judge gain (retest with them, per pair)In two pairs (HPO, Sonnet and Opus: +1.21, Holm p<.001p < .001); none on NB201, where GPT went the other way (−0.79-0.79)
14The judge gain transfers to a second judge, GPT (all six pairs)Partial: kept in three (+0.29 to +0.33), weakened in two, vanished in one (NB201, Opus); novelty up and quality down in all six
15The stage-3 harness beats a current batch of direct runs (Sonnet, NB201)Supported (+1.95, p<.001p < .001; drift between direct batches −0.18-0.18)
15GPT: measurement beats shuffled and withheld values (each benchmark)Supported on both: +1.40, +0.89 (NB201), +1.42, +0.85 (HPO), Holm p≤.005p \leq.005
15GPT: measurement beats a single-call 40-claim direct baseline (each benchmark)NB201 supported (+0.88, Holm p=.010p = .010); HPO refuted (+0.09, p=.36p = .36)

Table 11. Every pre-registered decision criterion and its verdict. Each pre-registration was written before any call of its stage; stage 12’s was written after the external audit and rescored runs already analyzed under the original metric. Differences are computed before rounding.

Checkable hypothesis spaces. A claim names an edit (set parameter jj to value bb), a condition (parameter kk at level cc) and a direction. It says that the edit’s effect on the metric is higher (or lower) when the condition holds than when it does not, on every dataset. The conditional difference is the edit’s mean effect over the parents that satisfy the condition minus its mean effect over those that do not; Appendix § 13 gives the formula, the derivation of the counts below and a worked example. A run’s precision is the share of its distinct, parseable claims that are true.

  • NAS-Bench-201 (NB201; Dong & Yang, [7]). The searched cell’s six edges and five operations give 870 (edit, condition) cells and 1,740 directional claims. A claim is true when the conditional difference has the same sign on CIFAR-10, CIFAR-100 and ImageNet16-120, with a minimum magnitude τ\tau of 2 percentage points (pp). Of the claims, 198 are true (random precision .114).

  • HPOBench (HPO; Eggensperger et al., [10]). The tabular MLP benchmark has five ordinal hyperparameters and 30,000 configurations, each trained with five seeds; we use three OpenML tasks. Edits move one grid step up or down, and conditions are low/mid/high bands. There are 148 cells and 296 claims, of which 41 are true with τ=0.5\tau= 0.5 pp (random precision .139).

The thresholds and the all-datasets rule were fixed before any generation; Appendix § 12 varies both.

Blind prior panel. Haiku, Sonnet and Opus rate every cell blind, giving P(higher)P(\mathrm{higher}), P(lower)P(\mathrm{lower}) and P(no consistent, meaningful difference)P(\text{no consistent, meaningful difference}), summing to 100 (stage 12). A claim’s prior is the panel’s mean probability of its stated direction, over the panel members other than the generator. This leave-one-out panel has two members for a Claude generator and all three Claude models for GPT; stage 12b adds GPT as a fourth member. A panel’s AUC is the area under the ROC curve of its prior for the stated direction against truth, over every claim in the space. On HPOBench, when k≠jk \ne j, “moving jj up has a higher effect” and “moving jj down has a lower effect” are the same proposition, so these wordings are merged before scoring.

Verified surprise. We score a run by R=∑h(Th−πh)R = \sum_h (T_h - \pi_h) over its distinct claims (duplicates and, on HPOBench, equivalent wordings merged; Appendix § 16 also reports RR per distinct claim). Here ThT_h is truth, and πh\pi_h is the panel’s prior calibrated to truth over the whole space, separately for each generator’s panel. The calibration is a lightly ridge-regularized logistic model (λ=10−3\lambda= 10^{-3}) on the prior’s logit (prior clipped to [0.5,99.5][0.5, 99.5]), the claim’s order (whether the condition is on the edited parameter itself, k=jk = j), their interaction, and the claim’s direction. The order and direction terms absorb most base-rate differences between claim types: self-conditions hold about three times as often as all other claims on NAS-Bench-201 (.275 against .088) and twice as often on HPOBench (.23 against .12), so a run gains little by choosing types alone. RR is counted in claims per run, and R>0R > 0 means the run’s claims hold more often than the panel would have predicted. Because RR can rise when a run trades many likely claims for a few unlikely true ones, we read a gain in RR as an improvement only when precision stays at least 0.8 times the comparison. arm’s § 5.2. This bound is part of the pre-registered criteria of the measurement tests (stages 2, 4–6 and 8–10), the stage-12 rescoring and the stage-13 confirmation of the stage-3 harness. The same-cost and control comparisons of stages 13c, 14 and 15 were pre-registered on pp alone, so for them the bound is a post hoc reading rule; Appendix § 16 quotes the criteria. Post hoc, Appendix § 14 splits RR into credit for true claims, ∑true(1−πh)\sum_{\mathrm{true}}(1-\pi_h), and the cost of false ones, ∑falseπh\sum_{\mathrm{false}}\pi_h, and gives every verdict at bounds from 0.6 to 1.0; the verdicts on the stage-3 and judge-optimized harnesses do not depend on the bound. We also report SV3\mathrm{SV}_3, the number of true claims with prior below 50. RR was introduced post hoc in stage 12 and is pre-registered from stage 12b on; SV3\mathrm{SV}_3 was pre-registered in stage 12. Both were computed on cached runs. Appendix § 12 varies the ridge penalty, clipping, covariates and fitting set.

What RR does and does not measure. RR is a measure of excess truth relative to a calibrated LLM panel, and we use “surprise” only in that sense. It is not surprise to experts or to the literature, which we did not measure. Four properties limit what a gain in RR shows. It depends on the panel, so the same claims can score differently against another panel (§ 4.3). It depends on the truth threshold: a claim that misses the threshold by a little costs its full π\pi, so part of a gain can come from avoiding near misses rather than from finding unexpected truths, which is why we report the credit and cost terms separately at each threshold. It gives partial credit, 1−π1-\pi, even for true claims the panel expected, so a run of many likely true claims can score like a run of a few unlikely ones; the precision bound guards against the latter trade. And it inherits any misfit of the calibration, which Appendix § 12 varies. The credit term is the part closest to the intended construct, and we state which findings rest on it.

Harnesses, optimizer and measurement. A harness sets retrieval (none, in-domain, cross-domain), roles (single, or gap finder, innovator and critic), self-challenge, pool size and selection rule, and a free-text strategy of at most 150 words. Direct generation, the baseline, uses none of these. An Opus optimizer reads scored traces and proposes edits over 3 rounds of 3 candidates, each scored on 6 runs. Its fitness is the judge’s score (Opus rating novelty and quality, each 1–10), the number of true hypotheses, or the original sv (§ 5.1). In measure-first arms the generator first designs K=100K=100 probes, each a parent configuration plus one edit, and reads the change in the metric. On NAS-Bench-201 this is a table lookup on the test split. Truth uses validation, but the two splits agree closely (edit effects correlate at r≥.997r \ge.997, and 99.8% of claims get the same truth label), so these probes are near-exact samples of the quantity truth averages (Appendix § 16). On HPOBench it is the test-accuracy difference of a single random training seed. On the generators’ own probes, its sign agrees with the 5-seed truth 70% of the time when the true difference is at least 0.5 pp.

Budgets. Direct generation used 3.5–5.9k tokens per run, the measure-first arm 17.4–36.5k, the stage-3 harness 17.5k and the judge-optimized harness 24.2–35.1k (Table 26 in the appendix). During optimization, candidate harnesses whose mean cost exceeded 40k tokens per run were disqualified. Claude counts include the CLI’s own system prompt, about 4k tokens per call, and exclude failed retries. Probes are table lookups and cost no compute. Where the budget could matter we compare with arms of similar cost: for the stage-3 harness, an ablation with the same prompt and pool (16.2k); for GPT, pooled and 40-claim direct baselines (stages 13–15); for Opus, arms that see shuffled or no values. Every run proposes ten hypotheses, so precision and the surprise scores compare what a fixed-size output contains. After duplicates and, on HPOBench, equivalent wordings are merged, a run holds 9–10 distinct claims on NAS-Bench-201 and 6–10 on HPOBench.

Bench.Gen.DirectMeasure-firstJudge-opt.Pooled direct
NB201Sonnet5.2k30.9k33.3k–
Opus5.9k32.6k35.1k–
GPT3.5k21.0k25.2k21.2k
HPOSonnet5.2k27.3k32.2k–
Opus5.4k36.5k34.4k–
GPT3.6k17.4k24.2k20.4k

Table 26. Mean tokens per run, 24 runs per arm. Claude counts are the CLI-reported input, output and cache tokens, including the CLI’s own system prompt; GPT counts are codex-cli’s “tokens used”. The two are not comparable with each other. Probes are table lookups and use no tokens or compute themselves; the measure-first cost is the design call plus the longer proposal prompt that lists the 100 results. The optimizer capped harnesses at 40k tokens per run. Pooled direct (stage 13, GPT only) runs direct generation four times and keeps ten of the pooled claims.

Protocol. The study has 17 pre-registered stages, each with written decision criteria (16 documents; stage 4b is an addendum to stage 4). We use one-sided permutation tests on runs, Holm correction within a family, and 90% bootstrap intervals. Key arms get 24 fresh runs. The models are claude-haiku-5-5, claude-sonnet-5-5 and claude-opus-5-5, called through the Claude Code CLI with tools disabled between 2026-10-21 and 2026-10-24 (UTC); the CLI’s session logs report exactly these identifiers for every logged call on every date (Table 22). The CLI logs also hold 1,504 earlier sessions, all of those on 2026-10-20 and 114 on 2026-10-21, that predate the first pre-registration (PREREQ_s1.md, 2026-10-21 00:56 UTC). They are pilots on other claim spaces and unrelated work, and are not used as evidence. Apart from two requests to review an unrelated manuscript that mentions NAS-Bench-201, a keyword search finds neither benchmark in them; the first session that poses a claim about a benchmark starts at 01:05 UTC, nine minutes after that document

Date (UTC)claude-haiku-5-5claude-sonnet-5-5claude-opus-5-5
2026-10-20766384240
2026-10-214551,6361,150
2026-10-223534110
2026-10-23–96576
2026-10-24–24–

Table 22. Claude Code CLI sessions by date and the model identifier each session’s log reports (temporary-directory sessions; includes calls from other work, among them all 2026-10-20 sessions, which predate this study’s first pre-registration and are pilots on other claim spaces). No session reports a different identifier.

ComparisonStagePrec.SV3SV_3ΔR\Delta R [90% CI]Δ(R/n)\Delta(R/n)CalibrationsPanelsTruth rules
Stage-3 harness–.835.17
vs direct generation13.793.54+1.77 [+1.30, +2.25]+0.17714/148/86/6
vs new direct runs15.74–+1.95 [+1.45, +2.45]+0.19514/14–6/6
vs likely-true ablation14.823.58+1.96 [+1.55, +2.38]+0.19614/14–6/6

caption: The stage-3 harness, Sonnet on NB201, 24 runs per arm, leave-one-out Claude panel (Haiku and Opus). All three tests pre-registered; p<.001p < .001 each. Prec.: precision of the comparison arm (first row: the harness). The ablation keeps the harness’s strategy and pool of 30 but selects the 10 claims most likely to be true. Every arm holds 10 distinct claims per run (ablation 9.96); Δ(R/n)\Delta(R/n): difference per distinct claim (post hoc; p<.001p < .001 each; Table 24). Post hoc robustness, counting p<.05p < .05: Calibrations, 14 calibration variants (Table 14); Panels, the 8 distinct panels, namely the seven drawn from Haiku, Opus and GPT (the three-member one is the leave-one-out panel with GPT added) and Sonnet’s own blind ratings (Table 17); Truth rules, six definitions of truth (Table 16). Table 20 adds intervals that also resample propositions.

Bench.ComparisonDef.λ=10\lambda=10λ=100\lambda=100Prior+TypeCVProp.RawSig./14
NB201Stage-3 harness vs direct+1.77*+1.57*+1.01*+1.81*+1.73*+1.79*+0.79*+0.98*14
vs new direct+1.95*+1.77*+1.36*+2.00*+1.89*+1.97*+0.97*+1.36*14
vs likely-true ablation+1.96*+1.65*+0.93*+1.97*+1.86*+1.94*+0.37*+0.88*14
GPT measure vs direct+1.60*+0.78*−0.46+1.58*+1.69*+1.57*−0.67−0.4512
vs pooled direct+2.24*+1.50*−0.15+2.34*+2.34*+2.20*−0.58−0.1612
vs pooled + stage-3 rule+1.16*+0.80*−0.74+1.28*+1.23*+1.11*−0.21−0.8612
vs 40-claim call+0.88*+0.58*−1.29+1.20*+0.60*+0.84*−0.24−1.3412
vs shuffled values+1.40*+1.85*+2.31*+1.44*+1.45*+1.40*+1.85*+2.23*14
vs withheld values+0.89*+0.10−0.77+0.83*+1.05*+0.88*−1.21−0.7211
HPOGPT measure vs direct+0.94*+0.63*+0.50*+0.94*+0.48*+0.97*+0.62*+0.56*14
vs pooled direct+0.70*+0.47*+0.28+0.73*+0.38*+0.66*+0.05+0.38*12
vs pooled + stage-3 rule+0.94*+0.83*+1.52*+0.20+0.21+0.98*+0.52*+1.13*12
vs 40-claim call+0.09+0.12+0.11+0.16+0.05+0.04−0.16+0.110
vs shuffled values+1.42*+1.79*+2.29*+1.30*+0.64*+1.51*+1.55*+2.01*14
vs withheld values+0.85*+0.02−0.09+0.49*+0.34*+0.88*+0.49*0.0012
NB201Opus measure vs direct+0.19−0.14−0.42+0.18−0.02+0.17−0.40−0.370
vs withheld values+0.92*+0.69*+0.24+0.90*+0.71*+0.89*−0.06+0.3012
vs shuffled values+1.84*+1.85*+1.74*+1.67*+1.63*+1.85*+1.02*+1.65*14
HPOOpus measure vs direct+0.42+0.21+0.10+0.42+0.47*+0.38+0.02+0.201
NB201Sonnet measure vs direct−0.57−1.07−2.07−0.65−0.74−0.57−0.81−2.110
HPOSonnet measure vs direct+0.11−0.68−0.33−0.62−0.19+0.08+0.18−0.490

Table 14. RR under alternative calibrations, scored against the leave-one-out Claude panel. Def.: the calibration of the main text (ridge λ=10−3\lambda= 10^{-3}, prior clipped to [0.5,99.5][0.5, 99.5], covariates logit prior, order, direction and logit×\timesorder). λ\lambda: ridge penalty. Prior: logit prior only. +Type: adds indicators for the edited value and the condition (NB201: operations; HPO: band, edit direction and edited parameter). CV: 10-fold cross-fitting by cell. Prop.: calibration fitted only on the claims the two compared arms proposed, not on the whole space; with an intercept this makes the two arms’ summed RR zero, so it asks only whether one arm’s claims beat the other’s at equal prior. Raw: uncalibrated ∑(T−p/100)\sum(T-p/100). Sig./14: number of the 14 calibrations (the columns shown plus λ∈{0,0.1,1}\lambda\in\{0,0.1,1\} and clipping at 0.1, 1, 2.5 and 5) with p<.05p<.05; Raw is not among them. Clipping never changes a difference by more than 0.02, cross-fitting by more than 0.09.

Bench.Truth ruleTrue (rate)Stage-3 harness (Sonnet) ΔR\Delta R vs direct / new directPrec.GPT measure-first ΔR\Delta R LOO / ownPrec.Judge-opt. bound met
NB201all, τ=1\tau= 1419 (.241)+1.08* / +1.44*.90→.88+1.06* / +1.30*.85→.62×0/3
all, τ=1.5\tau= 1.5288 (.166)+1.14* / +1.51*.89→.86+1.56* / +2.19*.79→.58×0/3
all, τ=2\tau= 2 (main)198 (.114)+1.77* / +1.95*.79→.83+1.60* / +2.69*.69→.50×0/3
all, τ=3\tau= 3‡108 (.062)+1.59* / +1.84*.59→.63+1.34* / +2.58*.49→.32×0/3
2 of 3, τ=2\tau= 2339 (.195)+1.63* / +1.78*.86→.90+1.98* / +2.51*.76→.620/3
HPOmean, τ=2\tau= 2314 (.180)+1.36* / +1.68*.91→.91+1.56* / +2.14*.78→.58×0/3
all, τ=0.25\tau= 0.2552 (.176)––+0.86* / +0.48*.94→.980/3
all, τ=0.5\tau= 0.5 (main)41 (.139)––+0.94* / +0.61*.90→.950/3
all, τ=1\tau= 1‡22 (.074)––+0.75* / +0.51*.71→.750/3
2 of 3, τ=0.5\tau= 0.584 (.284)––+0.61* / +0.20*.98→.990/3
mean, τ=0.5\tau= 0.588 (.297)––+0.56* / +0.20*.98→1.000/3

Table 16. Results under alternative truth definitions. A claim is true when the conditional difference exceeds τ\tau pp with the stated sign on all three datasets (main text), on at least two of three, or in the mean over datasets; RR is recalibrated under each definition. Stage-3 harness: Sonnet on NB201 against the stage-5 direct runs and the new stage-15 direct runs (leave-one-out Claude panel). Not shown as a column: the harness’s advantage over the likely-true ablation in the six NB201 rows is +0.69, +1.12, +1.96, +1.91, +1.73 and +1.35 (all p≤.002p \le.002), with the precision bound met in all six. GPT measure-first against direct generation under the leave-one-out Claude panel and GPT’s own blind ratings; × precision below 0.8 times direct generation’s. Judge-opt.: number of the three generators for which the judge-optimized harness keeps precision at least 0.8 times direct generation’s. ‡At these two thresholds the calibration’s undamped Newton fit diverges (every π\pi goes to 1); these rows use the same model fitted by Newton’s method with step halving on the penalized likelihood (script post_pos_round6d.py), which gives the same fit as the undamped one for every other truth definition here and for all 1,657 fits of the calibration and panel analyses (post_pos_round6e.py).

Bench.ComparisonOne memberTwo membersThree members
NB201Stage-3 harness (Sonnet)H +1.26*, O +1.51*, G +1.82*H+O +1.77*, H+G +1.74*, O+G +1.71*H+O+G +1.78*
GPT measureH −0.56, S −0.12, O +1.53*H+S +0.30, H+O +1.43*, S+O +1.71*H+S+O +1.60*
Opus measureH −0.31, S −0.21, G +1.25*H+S +0.19, H+G +0.91*, S+G +0.84*H+S+G +0.90*
Sonnet measureH −1.97, O −0.90, G −0.17H+O −0.57, H+G −0.30, O+G −0.23H+O+G −0.10
HPOGPT measureH +0.42, S +0.86*, O +0.78*H+S +0.74*, H+O +0.72*, S+O +1.01*H+S+O +0.94*
Opus measureH +0.04, S +0.54*, G +0.40*H+S +0.42, H+G +0.34, S+G +0.60*H+S+G +0.52*
Sonnet measureH −0.52, O 0.00, G −0.15H+O +0.11, H+G −0.07, O+G +0.07H+O+G +0.05

Table 17. ΔR\Delta R against direct generation under every panel drawn from the models other than the generator, grouped by panel size (H/S/O/G: Haiku/Sonnet/Opus/GPT; the panel is the mean of its members’ three-way priors, recalibrated per panel). The leave-one-out Claude panel of the main text is H+O for Sonnet, H+S for Opus and H+S+O for GPT.

TruthPanelComparisonΔR\Delta Rruns onlycrossed p×p_{\times}credit, crossed (p×)(p_{\times})cost avoided, crossed (p×)(p_{\times})
τ=2\tau= 2, all (main)Claude LOOvs direct+1.77[+1.31, +2.25][+0.38, +3.30].015 +1.04 [−0.03, +2.13] (.054)+0.73 [−0.09, +1.79] (.086)
vs new direct+1.95[+1.45, +2.46][+0.67, +3.38].004 +1.13 [+0.18, +2.15] (.028)+0.82 [+0.03, +1.86] (.039)
vs ablation+1.96[+1.56, +2.37][+0.36, +3.73].024 +1.22 [+0.04, +2.37] (.045)+0.74 [−0.23, +2.04] (.137)
Claude+GPT LOOvs direct+1.78[+1.33, +2.23][+0.55, +3.18].009 +1.05 [+0.22, +1.93] (.020)+0.72 [−0.15, +1.88] (.102)
vs new direct+1.85[+1.39, +2.33][+0.72, +3.22].002 +1.04 [+0.28, +1.87] (.013)+0.81 [−0.00, +1.86] (.051)
vs ablation+2.15[+1.75, +2.55][+0.68, +3.76].004 +1.36 [+0.45, +2.33] (.005)+0.79 [−0.29, +2.19] (.145)
Sonnet ownvs direct+1.25[+0.72, +1.78][−0.49, +2.93].116 +0.80 [−0.79, +2.33] (.189)+0.45 [−0.11, +1.20] (.117)
vs new direct+1.55[+1.05, +2.06][−0.10, +3.17].062 +1.09 [−0.35, +2.50] (.107)+0.46 [−0.08, +1.22] (.102)
vs ablation+1.16[+0.76, +1.57][−1.07, +3.35].192 +0.66 [−1.51, +2.64] (.304)+0.50 [−0.21, +1.38] (.165)
τ=1\tau= 1, allClaude LOOvs direct+1.08[+0.64, +1.53][+0.22, +2.01].018 +1.05 [+0.35, +1.81] (.006)+0.03 [−0.44, +0.57] (.478)
vs new direct+1.44[+0.91, +1.97][+0.53, +2.40].004 +1.02 [+0.37, +1.74] (.003)+0.42 [−0.15, +1.07] (.115)
vs ablation+0.69[+0.30, +1.09][−0.22, +1.61].102 +1.12 [+0.35, +1.93] (.007)−0.42 [−0.84, −0.08] (.978)
τ=1.5\tau= 1.5, allClaude LOOvs direct+1.14[+0.68, +1.59][+0.23, +2.13].017 +1.09 [+0.25, +1.96] (.013)+0.05 [−0.39, +0.58] (.433)
vs new direct+1.51[+1.00, +2.07][+0.56, +2.53].003 +1.15 [+0.37, +1.97] (.004)+0.36 [−0.12, +0.92] (.127)
vs ablation+1.12[+0.69, +1.52][+0.08, +2.15].039 +1.29 [+0.41, +2.27] (.008)−0.17 [−0.58, +0.29] (.734)
τ=3\tau= 3, allClaude LOOvs direct+1.59[+1.02, +2.18][+0.01, +3.24].049 +0.78 [−0.44, +2.04] (.143)+0.81 [−0.12, +1.91] (.080)
vs new direct+1.84[+1.26, +2.44][+0.29, +3.42].024 +0.94 [−0.19, +2.14] (.087)+0.91 [−0.01, +2.00] (.052)
vs ablation+1.91[+1.31, +2.50][+0.03, +3.79].047 +0.95 [−0.49, +2.32] (.134)+0.95 [−0.25, +2.34] (.101)
Sonnet ownvs direct+1.15[+0.58, +1.72][−0.46, +2.79].113 +0.76 [−0.73, +2.23] (.189)+0.39 [−0.12, +1.07] (.135)
vs new direct+1.50[+0.94, +2.07][−0.02, +3.05].053 +1.10 [−0.28, +2.53] (.098)+0.40 [−0.12, +1.14] (.145)
vs ablation+1.16[+0.63, +1.70][−0.77, +3.06].161 +0.65 [−1.26, +2.43] (.285)+0.52 [−0.16, +1.39] (.136)
τ=2\tau= 2, 2 of 3Claude LOOvs direct+1.63[+1.11, +2.13][+0.54, +2.79].005 +1.19 [−0.36, +2.07] (.008)+0.44 [−0.12, +1.22] (.117)
vs new direct+1.78[+1.25, +2.33][+0.80, +2.84].002 +1.21 [+0.41, +2.10] (.006)+0.58 [+0.09, +1.18] (.025)
vs ablation+1.73[+1.28, +2.17][+0.55, +3.01].007 +1.39 [+0.45, +2.37] (.006)+0.34 [−0.30, +1.24] (.260)
τ=2\tau= 2, meanClaude LOOvs direct+1.36[+0.86, +1.84][+0.36, +2.39].011 +1.20 [+0.37, +2.09] (.009)+0.16 [−0.24, +0.64] (.288)
vs new direct+1.68[+1.13, +2.23][+0.70, +2.71].003 +1.21 [−0.41, +2.08] (.005)−0.46 [−0.03, +0.96] (.036)
vs ablation+1.35[+0.93, +1.77][+0.27, +2.48].017 +1.40 [+0.47, +2.39] (.008)−0.05 [−0.40, +0.35] (.612)

Table 20. Uncertainty of the stage-3 harness’s gain when the recurring propositions are treated as a sample too (post hoc; Sonnet, NB201, 24 runs per arm). Runs only: 90% interval from resampling runs within each arm, computed from this table’s own draws, so it can differ from Table 3 in the second decimal. Crossed: runs are resampled within each arm and, jointly for both arms, the distinct propositions of the two arms are resampled with replacement (each run’s RR is recomputed with every proposition weighted by its resampled multiplicity); 4,000 draws. p×p_{\times}: share of crossed draws ≤0\le0. Credit: ∑true(1−π)\sum_{\mathrm{true}}(1-\pi); cost avoided: the comparison arm’s ∑falseπ\sum_{\mathrm{false}}\pi minus the harness’s. The calibration is refitted under each truth definition and panel; at τ=3\tau= 3 with a damped fit (Appendix § 12).

Bench.ComparisonDistinct claims per run: First armDistinct claims per run: Second armLeave-one-out Claude panel: ΔR\Delta RLeave-one-out Claude panel: Δ(R/n)\Delta(R/n)Leave-one-out Claude panel: (p)(p)Generator’s own estimates: ΔR\Delta RGenerator’s own estimates: Δ(R/n)\Delta(R/n)Generator’s own estimates: (p)(p)
NB201Stage-3 harness vs direct10.0010.00+1.77+0.177(< .001)+1.25+0.125(< .001)
vs new direct10.0010.00+1.95+0.195(< .001)+1.55+0.155(< .001)
vs likely-true ablation10.009.96+1.96+0.196(< .001)+1.16+0.114(< .001)
GPT measure-first vs direct10.0010.00+1.60+0.160(< .001)+2.69+0.269(< .001)
vs pooled direct10.0010.00+2.24+0.224(< .001)+2.97+0.297(< .001)
vs pooled + stage-3 rule10.0010.00+1.16+0.116(< .001)+2.02+0.202(< .001)
vs 40-claim call10.009.79+0.88+0.086(.005)+1.28+0.128(< .001)
vs shuffled values10.0010.00+1.40+0.140(< .001)+0.98+0.098(< .001)
vs withheld values10.0010.00+0.89+0.089(.004)+1.89+0.189(< .001)
Opus measure-first vs direct10.009.88+0.19+0.013(.366)+0.91+0.088(.009)
HPOGPT measure-first vs direct7.797.96+0.94+0.119(< .001)+0.61+0.076(.002)
vs pooled direct7.797.46+0.70+0.081(< .001)+0.72+0.087(< .001)
vs pooled + stage-3 rule7.795.50+0.94+0.055(.078)+0.53+0.015(.335)
vs 40-claim call7.797.33+0.09−0.005(.444)+0.42+0.045(.032)
vs shuffled values7.798.54+1.42+0.181(< .001)+1.01+0.128(< .001)
vs withheld values7.798.96+0.85+0.127(< .001)+0.51+0.078(.003)
Opus measure-first vs direct9.679.54+0.42+0.041(.056)+0.48+0.048(.004)
NB201Judge-opt. vs direct, Sonnet10.0010.00+0.88+0.088(.006)−1.05−0.105(.003)
Judge-opt. vs direct, Opus10.009.88−2.24−0.230(< .001)−1.07−0.109(< .001)
Judge-opt. vs direct, GPT10.0010.00+1.04+0.104(.003)+1.91+0.191(< .001)
HPOJudge-opt. vs direct, Sonnet10.009.38−0.89−0.096(< .001)−1.63−0.179(< .001)
Judge-opt. vs direct, Opus10.009.54−1.69−0.182(< .001)−1.24−0.127(< .001)
Judge-opt. vs direct, GPT10.007.96−0.87−0.121(< .001)−0.58−0.082(< .001)

Table 24. RR per distinct claim (post hoc). RR sums over a run’s distinct parseable claims, after duplicates and, on HPOBench, equivalent wordings are merged; nn is that number. Δ(R/n)\Delta(R/n): difference in the per-run mean of R/nR/n, with a one-sided permutation pp in the direction shown. Paper-default calibration.

:::

(Appendix § 16). Stages 10, 12b, 13, 14 and 15 used gpt-6-astra, requested by explicit identifier, through codex-cli 0.162.0 (medium reasoning effort, read-only sandbox, empty working directory, 2026-10-21 to 23), as a generator, as a fourth panel member and, in stage 14, as a second judge. The study used over 4,500 successful model calls, all cached; the sensitivity analyses of Appendix § 12 use no new calls.

Results

A selection step that raises verified surprise

The harness accepted when the SV optimizer was rerun in stage 3 proposes a pool of 30 claims. Its strategy asks for the generator’s probability and a note on how an intuitive reader would judge each claim, but the output format has no field for a probability and none of the 709 pooled claims states one. A separate call then keeps the 10 that best satisfy a rule the optimizer wrote, including both thresholds: those the generator rates at least 55% likely but expects other models to rate below 50% (Appendix § 9; Appendix § 13 gives the two calls, what the optimizer saw, and the stage-2 bug fix that preceded this run). Under the original metric it had regressed on fresh runs (1.67 in training, 1.25 fresh), but a post hoc rescoring of all 19 harness evaluations singled it out as the one harness that raised RR (§ 5.4; +1.98, Holm p=.02p = .02, 8 runs). Three pre-registered tests followed, each with 24 fresh runs and the criterion p<.05p < .05; stage 13 also required precision at least 0.8 times direct generation’s (Table 3). The harness’s precision was 1.05, 1.12 and 1.00 times the comparison arm’s in the three tests. The three tests share the same 24 harness runs from stage 13 and differ only in the comparison arm, so they are one confirmation against three comparators, not three independent replications.

Confirmation. Against direct generation (stage 13), RR rose by 1.77 per run, SV3SV_3 from 3.54 to 5.17 (p<.001p < .001), and precision from .79 to .83; the 8-run estimate (+1.98) shrank only to +1.77. Because that direct control had been run two days earlier, stage 15 ran 24 new direct runs: the harness beat them by 1.95, and the two direct batches differed by −0.18 ([−0.64, +0.31]). The harness used 17.5k tokens per run, against 5.2k for direct generation.

The selection rule carries the gain. A pre-registered ablation in stage 14 kept the strategy and the pool of 30 but selected the 10 claims most likely to be true. The gain vanished: RR fell 1.96 below the confirmed harness and 0.19 below direct generation, at the same precision (.82), with SV3SV_3 at 3.58 and 16.2k tokens per run. The ablation also matched the new stage-15 direct batch (−0.01). Generating 30 claims with the same strategy at the same cost thus does not produce the gain; the selection rule does. This ablation favors the harness by construction, however; claims selected for being likely true have high priors and earn little credit under RR, and the ablation did worse than random selection from its own pools. The random-selection contrast below is therefore the cleaner evidence that the selection step carries the gain.

What the rule selects. Post hoc, we compared the rule with other ways of keeping 10 claims from the same 24 cached pools of 30 (Appendix § 11, 5,000 draws each). Random selection from the pools gives R=+1.32R = +1.32 per run, close to direct generation (+1.10); the rule’s selections give +2.87, above every random draw. Keeping the “Other” type first (interactions between edges that both carry an operation other than none; Table 6) gives +1.92, and the rule beats this type filter in every draw as well (+0.95). The rule doubles the Other type’s share (20% of kept hypotheses against 10% of the pools and 3% for direct generation), and these hold 77% of the time against a base rate of 7%; but it also raises precision within each claim type (self-conditions .93 kept against .78 not kept, Other .77 against .64). The ablation’s likely-true selection did worse than random selection from its own pools (−0.36), mainly because it kept no Other-type claim, although its pools held 76, 35 of them true. With indicators for the edited operation and the condition’s operation added to the calibration (the +Type column of Table 14; these are operation indicators, not the three claim types), the harness’s gain over direct generation is +1.73. With the three claim types themselves in the calibration (an indicator for claims whose condition removes an edge, and its interaction with the prior), the three gains are +1.73, +1.90 and +1.89 (post hoc, all p<.001p < .001; Appendix § 16). The judge-optimized harness moved to the same claim type (74% of its hypotheses) but at 43% overall precision (§ 4.4). The rule’s design intent can also be imitated with cached priors (post hoc; Appendix § 11). Keeping, from each pool, the claims the scoring panel rates least likely gives +3.42, and keeping first those Sonnet’s blind prior rates at least 55 and the panel below 50 gives +3.47; both beat the harness (+2.87), but both read the very priors that RR subtracts, which the harness never sees. Keeping the claims with the largest gap between Sonnet’s blind prior and the panel’s gives +1.88, and keeping those Sonnet’s blind prior rates most likely gives +0.35. The harness thus recovers about three quarters of the gain from random selection to a selector given the panel’s priors, without access to them.

Self-cond.Removes edgeOther
Base rate of truth (all claims).275.173.067
Share, Sonnet direct / judge-opt..68 / .03.28 / .23.03 / .74
Share, Opus direct / judge-opt..31 / .00.67 / .17.03 / .82

Table 6. Claim types on NB201, as shares of all hypotheses of 24 runs per arm (about 240; a claim proposed in several runs counts each time). The judge-optimized strategy forbids self-conditions and conditions that make the edited component unused. “Removes edge” counts any condition that sets another edge to none, which is broader than the ban, so such claims remain at .17–.23. The stage-3 harness puts 20% of its hypotheses in “Other”, and 77% of those are true.

What the rule does not do. Two checks qualify the account in the rule’s own words. First, compliance: measured with Sonnet’s separately elicited blind prior, the kept claims are not the ones Sonnet rated at least 55% likely (27% of kept claims against 45% of those not kept), and that blind prior does not discriminate truth within the pools (AUC .51). They are, however, enriched for claims the leave-one-out panel rated below 50 (67% against 54%). Only 5 of 24 selection replies mention other models at all. Whatever the generator does in context, its selections favor claims the panel doubts and are more often true, but we cannot show that they follow its own stated probabilities. Second, concentration: the harness’s 240 hypotheses contain 93 distinct propositions. The largest single contribution comes from a false claim that direct generation proposes often and the panel expects (setting edge 2 to a 3×3 convolution helps less when input edge 0 is none; π=.97\pi= .97), which the harness avoids. That claim holds in the stated direction on all three datasets but misses τ\tau on CIFAR-10 by 0.07 pp. Dropping the 10 propositions that contribute most from both arms removes the gain against the stage-5 direct runs (+0.07) and the ablation (−0.02), and leaves +0.48 (p=.02p = .02) against the new stage-15 direct runs; counting each distinct claim once per arm leaves +0.60 per run against direct generation. To put this concentration into the uncertainty, we resampled propositions jointly with runs (Table 20). The 90% intervals widen from [+1.30, +2.25] to [+0.38, +3.30] against direct generation, and the gain stays above zero against all three arms under the main panel (px=.015,.004,.024p_x = .015, .004, .024) and the panel with GPT added (px≤.009p_x \leq.009). Scored against Sonnet’s own blind prior it does not (px=.116,.062,.192p_x = .116, .062, .192). The gain is therefore real on fresh runs, and under the main panel it generalizes beyond the particular propositions sampled, but it rests on a few recurring ones, as one would expect from a 1,740-claim space that a generator samples repeatedly.

Surprising truths or avoided errors? Post hoc, we split the gain into credit for true claims and cost avoided on false ones (Appendix § 14; summarized in § 1). Against direct generation, +1.04 of the +1.77 is credit. Over runs, credit alone is significant against all three comparison arms (+1.04, +1.13 and +1.22; +0.66 to +1.09 under Sonnet’s own ratings; all p≤.001p \leq.001). With propositions resampled as well it is only marginal against direct generation (px=.054,.028p_x = .054, .028 and .045 against the other two arms) and not significant under Sonnet’s own ratings (Table 20). The harness’s true claims have a lower mean calibrated prior than direct generation’s (.58 against .70). Most of the cost avoided comes from near misses, claims whose stated direction holds on every dataset but by less than τ\tau (0.44 per run against 0.97 for direct generation), including the top contributor above. At τ=1\tau= 1 and 1.5 pp, where most near misses become true, the gain is almost all credit and is significant with propositions resampled (px≤.013p_x \leq.013); at τ=3\tau= 3 it splits evenly.

Robustness. Post hoc, the gain held under all seven panels drawn from the other three models (+1.26 to +1.82, including each single member and the three-member panel, which is the leave-one-out panel with GPT added, +1.78) and under Sonnet’s own blind ratings (+1.25; all p<.001p < .001 over runs, but p×=.12p_{\times} = .12 against Sonnet’s ratings with propositions resampled). It held under all 14 calibration variants we tried, including a ridge penalty of 100 (+1.01) and a calibration fitted only on the two arms’ own claims (+0.79), and on the uncalibrated excess-truth score (+0.98; Table 14). It held at every truth threshold from 1 to 3 pp and when truth requires agreement on two of three datasets or in the mean (+1.08 to +1.84 against the two direct batches, all p<.001p < .001; Table 16); at τ=3\tau= 3 pp, where only 108 claims (6.2%) are true, it is +1.59 and +1.84, and the ablation contrast +1.91, with precision above direct generation’s (.63 against .59). The harness was found in one of two optimizer runs (the first returned direct generation) and tested only with Sonnet on NAS-Bench-201. Applied to GPT’s pooled direct claims, the same rule raised RR over likely-true selection on NAS-Bench-201 (+1.08) but not on HPOBench (−0.23; pre-registered as descriptive in stage 14).

GPT uses informative evidence, but no budget-efficient gain was shown

For GPT, the measure-first arm designs 100 probes, reads their values and then proposes. On the same designed probes, it beat shuffled and withheld values on both benchmarks (stage 15, pre-registered; Table 4). The probes and the proposal call are identical across these arms; only what the values say differs, so GPT’s proposals use what the probes say, not only the act of designing experiments or the longer prompt. This does not show how GPT combines the probes with its own knowledge, which our conflict analysis (§ 5.3) examined only for Sonnet and Opus.

Bench.Comparison (stage)TokensnnΔR\Delta R [90% CI]Δ(R/n)\Delta(R/n)Prec. ratioVerdictCalibrations
NB201Pooled direct (13)21.2k10.0+2.24 [+1.79, +2.74]+0.224.71Supported12/14
Pooled + stage-3 rule (14)–10.0+1.16 [+0.57, +1.76]+0.116.68Supported12/14
40-claim call (15)11k9.8+0.88 [+0.38, +1.40]+0.086.61Supported12/14
Shuffled values (15)–10.0+1.40 [+0.91, +1.92]+0.1402.64Supported14/14
Withheld values (15)–10.0+0.89 [+0.39, +1.41]+0.089.72Supported11/14
HPOPooled direct (13)20.4k7.5+0.70 [+0.39, +1.02]+0.081.98Supported12/14
Pooled + stage-3 rule (14)–5.5+0.94 [+0.53, +1.35]+0.055 (p=.08p = .08)1.15Supported12/14
40-claim call (15)11.0k7.3+0.09 [−0.31, +0.49]−0.005 (p=.44p = .44).97Refuted0/14
Shuffled values (15)–8.5+1.42 [+0.96, +1.89]+0.1811.63Supported14/14
Withheld values (15)–9.0+0.85 [+0.45, +1.26]+0.1271.05Supported12/14

Table 4. GPT, measure-first minus each pre-registered comparison arm on RR, 24 runs per arm, leave-one-out Claude panel. Measure-first costs 21.0k tokens per run on NB201 and 17.4k on HPO. Supported: Holm p≤.010p \le.010 within the stage; refuted: Holm p=.36p = .36 with the 90% CI upper bound below 0.5. Calibrations: number of the 14 calibration variants of Table 14 under which the difference has p<.05p < .05 (post hoc). Measure-first precision: .50 on NB201 (direct .69) and .95 on HPO (direct .90). Prec. ratio: measure-first precision over the comparison arm’s; the 0.8 bound was not part of these stages’ pre-registered criteria, so the verdicts follow pp alone, but under our reading rule the NAS-Bench-201 gains over every same-cost arm (ratio below 0.8) do not count as improvements. nn: distinct claims per run in the comparison arm (measure-first: 10.0 on NB201, 7.8 on HPO). Δ(R/n)\Delta(R/n): difference per distinct claim (post hoc; p≤.005p \le.005 unless shown; Table 24).

Against direct generation and same-cost baselines. Stage 10 compared measurement with direct generation, pre-registered on the original metric: on HPOBench it raised SV by 1.08 (Holm p<.001p < .001) without losing precision (.89 to .94); on NAS-Bench-201 the gain (+0.79, Holm p=.003p = .003) failed the precision condition (.69 to .50). Post hoc, RR rises by 0.94 and 1.60 per run. Because measurement costs about five times the tokens of direct generation, stages 13–15 added baselines of similar cost (Appendix § 16). Measurement beat four pooled direct runs, with or without the stage-3 rule, on both benchmarks, and a single 40-claim call on NAS-Bench-201; on HPOBench the 40-claim call matched it at about two thirds of its tokens (11.0k against 17.4k). Per distinct claim (post hoc; Table 24), one more HPOBench gain weakens: against pooled direct generation with the stage-3 rule, which keeps only 5.5 distinct claims per run, it is +0.055 (p=.078p = .078).

Scope. On HPOBench the gain over direct generation holds under all 14 calibrations, the uncalibrated score and every Claude panel, and it keeps precision, but a cheaper call matches it. On NAS-Bench-201 it costs precision beyond the bound and is calibration-dependent: with a ridge penalty of 100, a calibration fitted on the two arms’ own claims, or the uncalibrated score, measurement no longer beats direct generation (−0.46, −0.67, −0.45), although it still beats shuffled values. There, moreover, a probe reads almost exactly the quantity that truth averages (§ 3), so measurement is close to sampling the answer. On neither benchmark have we shown that measurement is a budget-efficient way to raise RR.

A measurement gain for Opus depends on the panel (post hoc, hypothesis-generating)

A non-Claude panel member. Stage 12b asked GPT the same three-way question (34 calls). GPT puts less mass on “no effect” (Table 8), and its prior predicts truth at least as well as Opus’s (AUC .91 and .95; Opus .88 and .93, Sonnet .71 and .72, Haiku .67 and .69; 90% intervals within ±.04 on NAS-Bench-201 and ±.09 on HPOBench). With GPT added to the leave-one-out panel (pre-registered), measurement raised both corrected scores for Opus on NAS-Bench-201 (sv3 +1.42, RR +0.90, Holm p<.05p < .05 over the four Claude pairs), with precision .87 to .75, within the bound. This gain does not come from the original artifact (§ 5.1). GPT rates most of Opus’s directly generated true claims as likely, mainly claims that setting another edge to none changes an edit’s effect (93 of the 109 true-claim instances over 24 runs that leave sv3), and measurement moves Opus away from such claims (from about two thirds to .47 of hypotheses). Under a GPT-only panel, RR favors measurement for Opus on NAS-Bench-201 by +1.25 (Holm p=.002p = .002). On the other Claude pairs the verdicts were inconclusive (Sonnet on NAS-Bench-201, Opus on HPOBench) or refuted (Sonnet on HPOBench; Table 9).

NB201HPO
Mean PP(higher), H/S/O/G21/25/46/5422/28/40/49
Lower share of true claims.57.54
Lower share of sv-eligible true claims.00.00
Equivalent wordings, opposite labels–41–60/60
Three-way PP(none), H/S/O/G70/68/47/2468/69/53/38
True lower claims, prior < 50 (three-way).92.95

Table 8. The direction bias of the complement-coded surprise metric. H/S/O/G: Haiku/Sonnet/Opus/GPT; probabilities are means over cells. GPT’s two-way ratings (stage 10) are descriptive and enter no panel. sv-eligible: true and leave-one-out prior < 50 (averaged over both generators’ panels).

Bench.Gen.Orig. ΔSV\Delta\mathrm{SV}Mix partCorrected, Claude panel ΔSV3\Delta\mathrm{SV}_3 [90% CI]Corrected, Claude panel ΔR\Delta RGPT added to panel ΔSV3\Delta\mathrm{SV}_3GPT added to panel ΔR\Delta R
NB201Sonnet+0.04+0.14−0.92 [−1.42, −0.42]−0.57+0.29−0.10
Opus+1.04{sup}*`+1.02−0.75 [−1.42, −0.04]+0.19+1.42{sup}‡`+0.90{sup}‡`
GPT+0.79+0.11+0.08 [−0.58, +0.75]{sup}§`+1.60{sup}†`
HPOSonnet+0.88{sup}*`+0.68−1.92 [−2.33, −1.50]+0.11−1.33+0.05
Opus+0.50{sup}*`+0.15+0.04 [−0.50, +0.62]+0.42+0.08+0.52
GPT+1.08{sup}*`+0.25+0.67 [+0.25, +1.12]{sup}§`+0.94{sup}†`−−

Table 9. Measure-first minus direct generation, 24 fresh runs per arm. Precision as pre-registered: NB201 .79→.45 (Sonnet), .87→.75 (Opus), .69→.50 (GPT); HPO .82→.83, .90→.89, .89→.94 (with merged wordings, as used for SV3\mathrm{SV}_3 and RR: .81→.82, .90→.89, .90→.95). {sup}*Pre-registered support under the original metric; GPT on NB201 was inconclusive because precision fell below 0.8 times direct generation’s. Mix part: the part of the original difference explained by the change in the number of higher-direction claims, holding SV per higher claim at the two arms’ mean (post hoc). Corrected, Claude panel: the leave-one-out Claude panel (two members for Claude generators, three for GPT). Under $\mathrm{SV}_3$ (pre-registered rescoring, stage 12) the four Claude pairs are refuted, refuted, refuted, inconclusive; {sup}§post hoc. For Claude generators no Claude-panel ΔR\Delta R is significant in the positive direction (smallest pp = .06); {sup}†$p$ < .001, one-sided, post hoc. GPT added to panel: the Claude leave-one-out panel plus GPT (stage 12b), pre-registered; {sup}‡Holm pp < .05 over the four Claude pairs. A GPT generator’s leave-one-out panel cannot include GPT.

Panel dependence. Post hoc, we rescored measurement against direct generation under every panel drawn from the models other than the generator (Figure 2; details in Appendix § 15). On NAS-Bench-201, measurement raises RR for Opus and GPT under all 8 panels that include the other frontier model (AUC ≥ .87; +0.84 to +1.71, p≤.015p \le.015, uncorrected) and under none of the 6 that do not (AUC ≤ .74; −0.56 to +0.30). Sonnet gains under no panel, including its own blind ratings. On HPOBench, GPT gains under all seven panels and Opus under four. Against Opus’s own blind ratings, a strong panel without the other frontier model, Opus gains too (+0.91 on NAS-Bench-201, +0.48 on HPOBench), and the same probes with shuffled or withheld values do not produce its gain (Table 21). We read this as a hypothesis, not a finding. No panel has an AUC between .74 and .87, so panel quality and the presence of the other frontier model are confounded. The boundary was read off the data, and the panels share members, so the split is one contrast per generator. Under the main Claude panel, Opus’s gain is +0.19 and significant in none of the 14 calibrations, and with GPT in the panel it disappears under heavy shrinkage (−0.03 with a ridge penalty of 100). A pre-registered test with a panel member of intermediate AUC would separate the explanations.

Figure 2. Measure-first minus direct generation on RR against the panel’s AUC, for every panel drawn from the other three models (post hoc, 24 runs per arm). Filled: p<.05p < .05, one-sided, uncorrected. Circled: the Claude leave-one-out panel of Table 9. Ticks along the bottom: each generator’s own AUC.

PanelExactNoise2×2 \times noise12-ep. proxyWithheldShuffledDesign only
Claude leave-one-out (H+S)+0.19−0.14-0.14−0.27-0.27−0.71-0.71−0.73-0.73−1.65-1.65−1.23-1.23
H+S+G+0.90*+0.50+0.45+0.24−0.46-0.46−0.76-0.76−0.95-0.95
G only+1.25*+0.76*+0.72*+0.65*−0.30-0.30−0.63-0.63−0.77-0.77
Opus’s own ratings+0.91*+0.48+0.26−0.27-0.27−0.30-0.30−0.86-0.86−1.02-1.02
Precision (direct .87).75.71.69.65.79.58.69

Table 21. Evidence or tokens? Opus on NB201, measurement variants and controls minus direct generation on RR (post hoc; 24 runs per arm; *p<.05p < .05, one-sided, uncorrected). All arms but design-only reuse the same 100 designed probes and differ only in what the proposal prompt shows: exact values, values with simulated single-seed noise or doubled noise, values from a 12-epoch proxy, no values, or values shuffled across probes. Design-only writes 100 experiments it knows will not be run. H/S/G: Haiku/Sonnet/GPT.

A precision check shows the cost of optimizing for judged novelty

We evolved one harness for the judge’s score, in one optimizer run, so this section is a case study of that harness. Table 5 shows that it raises that score and cuts precision by 46–68%, also when transferred unchanged to HPOBench and when GPT is the generator (pre-registered in stages 5, 9 and 10). The evolved strategy forbids two claim types “because experts find them obvious”: conditioning on the current value of the edited parameter, and conditions that make the edited component unused. These are the claim types most often true (Table 6). Before any optimization, a screen of 72 runs had shown what the judge rewards: its novelty rating correlates with the mean prior of a run’s hypotheses at ρ=−.58\rho= -.58 (−.74 under the corrected prior, post hoc), and its quality rating with the number of true hypotheses at ρ=+.53\rho= +.53. Post hoc, the precision loss holds under every truth definition of Appendix § 12: for all three generators and all eleven definitions, the judge-optimized harness falls below 0.8 times direct generation’s precision (Table 16).

Bench.Gen.HarnessPrec.JudgeSV₃ΔR\Delta R
NB201SonnetDirect.79–3.54
Judge-opt..43–4.25+0.88†
OpusDirect.878.256.96
Judge-opt..288.882.75−2.24†
GPTDirect.698.044.08
Judge-opt..268.632.58+1.04†
HPOSonnetDirect.828.125.33
Judge-opt..439.174.33−0.89†
OpusDirect.908.175.38
Judge-opt..429.964.25−1.69†
GPTDirect.898.333.33
Judge-opt..358.633.50−0.87†

Table 5. Judge fitness, 24 fresh runs per arm (stage 5 for NB201, stage 9 for HPO, stage 10 for GPT). The harness was evolved with Sonnet on NB201 (stage 2) and applied unchanged to the other generators and to HPO. Judge: novelty + quality (2–20); Sonnet’s NB201 runs in stage 5 were not judged (stage 13 judged them without explanations). Pre-registered: precision drop ≥ .2 with p<.05p < .05 in every judge-optimized row; judge increase p<.05p < .05 in every row with judge scores; both held. HPO precision is unmerged; merging equivalent wordings changes it by at most .01. ΔR\Delta R: change in claims true beyond the calibrated leave-one-out Claude panel’s expectation (post hoc); †p<.01p < .01, one-sided in the direction shown, uncorrected.

Type shift or worse claims? The judge prompt asks whether experts would find the hypotheses surprising, and the evolved strategy bans the two claim types most often true, so part of the precision loss follows from base rates. Post hoc, we compared precision within claim type (Table 25). On HPOBench every judge-optimized claim is of the Other type, and these hold far less often than direct generation’s Other-type claims (.35–.43 against .80–.96; the 90% intervals of the difference all lie below −0.3). The type mix alone would predict a precision of .80–.96, so on HPOBench the harness’s claims are worse within type, not only of a different type. On NAS-Bench-201 direct generation proposes almost no Other-type claims (8, 6 and 0 over 24 runs), so the two explanations cannot be separated there; within the one type both arms use often, the judge-optimized harness is less precise for Sonnet and Opus (.49 against .65, .52 against .89) and similar for GPT (.62 against .66). All of this concerns one judge-optimized harness from one optimizer run; we have not shown that judge optimization in general behaves this way.

Bench.Gen.Precision: Judge-opt.Precision: DirectPrecision: Type-adjustedTypeSharePrecisionnn
NB201Sonnet.43.79.36Self-condition.03 / .68.57 / .877 / 164
Removes an edge.23 / .28.49 / .6555 / 68
Other.74 / .03.40 / .25178 / 8
NB201Opus.28.86.71Self-condition.00 / .31– / .840 / 73
Removes an edge.17 / .67.52 / .8942 / 158
Other.82 / .03.22 / .67198 / 6
NB201GPT.26.69–Self-condition.00 / .23– / .790 / 56
Removes an edge.11 / .77.62 / .6626 / 184
Other.89 / .00.22 / –214 / 0
HPOSonnet.43.81.80Self-condition.00 / .44– / .820 / 98
Other1.00 / .56.43 / .80240 / 127
HPOOpus.42.90.96Self-condition.00 / .55– / .850 / 127
Other1.00 / .45.42 / .96240 / 102
HPOGPT.35.89.82Self-condition.00 / .62– / .930 / 118
Other1.00 / .38.35 / .82240 / 73

Table 25. Precision of the judge-optimized harness within claim type (post hoc; claims pooled over 24 runs per arm, so precision can differ by .01 from the per-run means of Table 5). Type-adjusted: the precision implied by the judge-optimized harness’s mix of claim types alone, its type shares times direct generation’s within-type precision; “–” where direct generation proposed no claim of a type the harness used. On NB201, direct generation proposed only 8, 6 and 0 Other claims, so the type-adjusted values there rest on very few claims. Removes an edge: the condition sets another edge to none (NB201 only).

Style or content? Three pre-registered rejudgings of all 288 direct and judge-optimized runs ask whether the judge’s gain is a matter of style (Table 7; details in Appendix § 10). Without each hypothesis’s one-sentence “why”, the judge still preferred the judge-optimized sets in four of six pairs, always through novelty (+0.62 to +1.42), not quality. A retest with the original prompt reproduced the gain (+0.46 to +1.92), but single-run totals barely correlate between judgings (r=−.22r = -.22 to .62), so only arm means reproduce. A second judge, GPT, rated the judge-optimized sets higher in novelty and lower in quality in all six pairs, so its total rose only slightly (0 to +0.33; pre-registered verdict partial). Both judges thus read the harness’s claims as more novel; precision, which needs no judge, shows what they cost.

Bench.Gen.OriginalNo “why” (13)Retest (14)“Why” effectΔ\Delta totalΔ\Delta noveltyΔ\Delta quality
NB201Sonnet–+1.17+0.67−0.50+0.04+0.83−0.79
Opus+0.62+0.42+0.62+0.210.00+1.00−1.00
GPT+0.58+1.46+0.67−0.79†+0.33*+1.08−0.75
HPOSonnet+1.04+0.21+1.42+1.21†+0.29*+0.88−0.58
Opus+1.79+0.71+1.92+1.21†+0.33*+0.88−0.54
GPT+0.29+0.38+0.46+0.08+0.33+0.88−0.54

Table 7. Judge-optimized minus direct generation, judge totals (novelty + quality, 2–20; components 1–10), 24 runs per arm. Original: the stage 5, 9 and 10 judgments. No “why”: explanations removed (stage 13; kept in four pairs, p≤.022p \le.022). Retest: the original prompt, judged again (stage 14). “Why” effect: retest gain minus no-“why” gain, two-sided, Holm over six pairs; †Holm p<.05p < .05. GPT judge: the original prompt with GPT as judge (stage 14); *one-sided p<.05p < .05 (pre-registered “kept”). Every GPT novelty and quality difference has p≤.001p \le.001 (post hoc).

RR alone would not have caught it. Under the corrected prior, the judge harness lowers RR by 0.9–2.2 per run in three of four Claude pairs and for GPT on HPOBench (p≤.001p \le.001; with a prior-only calibration, significantly only in the two Opus pairs). For Sonnet and GPT on NAS-Bench-201, however, it raises RR (+0.88, +1.04) while precision falls to .43 and .26. Half or more of the rise is cost avoided, because its many false claims are ones the panel also doubted (credit +0.43 and −0.38, cost avoided +0.45 and +1.41; Appendix § 14); for GPT this rise turns into a drop under a weaker Haiku and Sonnet panel (−0.86). This is why we pair RR with the precision bound (§ 5.2).

Sensitivity of the positive results

Appendix § 12 reports three post hoc sensitivity analyses on cached runs, added after the main study. Calibration (Table 14 and Table 15): 14 variants of the ridge penalty (0 to 100), prior clipping (0.1 to 5), covariates (prior only; claim-type indicators), 10-fold cross-fitting by cell, and a fit on the two compared arms’ claims only, plus the uncalibrated ∑(T−p/100)\sum(T-p/100). Clipping changes no difference by more than 0.02 and cross-fitting by no more than 0.09. The stage-3 harness and GPT’s and Opus’s advantages over shuffled values are significant in all 14 variants under both the leave-one-out panel and the generator’s own ratings; GPT’s other NAS-Bench-201 gains and Opus’s other gains are not robust to λ=100\lambda= 100 or the arms-only fit (details in Appendix § 12). Truth definition (Table 16): τ∈{1,1.5,2,3}\tau\in\{1, 1.5, 2, 3\} pp on NAS-Bench-201 and {0.25,0.5,1}\{0.25, 0.5, 1\} pp on HPOBench, and agreement on two of three datasets or in the mean instead of on all three. The stage-3 harness’s gain and GPT’s measurement gain over direct generation hold under every definition (at τ=3\tau= 3 on NAS-Bench-201, +1.59 and +1.34; at τ=1\tau= 1 on HPOBench, GPT +0.75), although GPT’s NAS-Bench-201 precision stays below the bound throughout. At these two strictest thresholds the default Newton fit of the calibration diverges, so we refit them with step halving, which reproduces the default fit wherever that converges (all 1,657 fits of the calibration and panel analyses). The judge harness fails the precision bound everywhere. Panel size (Table 17): see § 4.3. The stage-3 harness is the result that survives every analysis in the appendix, although once propositions are resampled it does not survive scoring against Sonnet’s own prior (Table 20); the measurement results for GPT on NAS-Bench-201 and for Opus depend on the calibration as well as the panel.

Bench.ComparisonDef.λ=10\lambda=10λ=100\lambda=100Prior+TypeCVProp.RawSig./14
NB201Stage-3 harness vs direct+1.25*+1.19*+1.00*+1.29*+1.14*+1.26*+0.84*+1.16*14
vs new direct+1.55*+1.50*+1.35*+1.60*+1.44*+1.57*+1.11*+1.48*14
vs likely-true ablation+1.16*+1.08*+0.85*+1.19*+1.02*+1.17*+0.40*+1.05*14
GPT measure vs direct+2.69*+2.28*+1.04*+2.46*+2.61*+2.69*+0.49+0.2913
vs pooled direct+2.97*+2.66*+1.34*+2.96*+3.06*+2.96*+0.39+0.4913
vs pooled + stage-3 rule+2.02*+1.92*+0.81*+2.00*+2.11*+1.98*+0.06−0.0913
vs 40-claim call+1.28*+1.27*+0.09+1.61*+1.20*+1.25*−0.07−0.7912
vs shuffled values+0.98*+1.10*+1.36*+1.03*+0.95*+0.98*+0.89*+0.69*14
vs withheld values+1.89*+1.47*+0.46+1.62*+1.87*+1.93*+0.03−0.0712
HPOGPT measure vs direct+0.61*+0.50*+0.41*+0.61*+0.24+0.64*+0.39*+0.45*13
vs pooled direct+0.72*+0.51*+0.28+0.72*+0.60*+0.71*+0.02+0.29*12
vs pooled + stage-3 rule+0.53*+0.47*+1.06*+0.39*+0.23+0.54*+0.48*+0.70*13
vs 40-claim call+0.42*+0.27+0.14+0.45*+0.56*+0.40*−0.20+0.1211
vs shuffled values+1.01*+1.32*+1.98*+0.94*+0.58*+1.04*+1.02*+1.69*14
vs withheld values+0.51*+0.14−0.01+0.44*+0.14+0.53*+0.37*+0.2410
NB201Opus measure vs direct+0.91*+0.58+0.04+0.67*+0.72*+0.88*+0.02−0.1211
vs withheld values+1.20*+1.07*+0.56*+1.24*+1.03*+1.19*+0.24+0.5113
vs shuffled values+1.77*+1.81*+1.74*+1.79*+1.65*+1.76*+1.24*+1.64*14
HPOOpus measure vs direct+0.48*+0.29+0.10+0.47*+0.57*+0.49*−0.02+0.1811
NB201Sonnet measure vs direct−2.05−2.12−2.33−1.90−2.19−2.06−2.05−2.030
HPOSonnet measure vs direct+0.25−0.35−0.23−0.30−0.14+0.31+0.50*−0.321

Table 15. RR under alternative calibrations, scored against each generator’s own blind three-way ratings.

Diagnostics: what did not work and why

A complement-coded surprise metric manufactured a gain

Before stage 12, each panel member gave only the probability that the edit’s effect is higher under the condition, and a lower claim’s prior was 100 minus it. A hypothesis was surprising-valid (SVSV) if true with prior below 50. If the panel’s beliefs over the three outcomes are P(higher)P(\text{higher}), P(lower)P(\text{lower}) and P(none)P(\text{none}), the complement rule gives a lower claim the prior

100−P(higher)=P(lower)+P(none)≥P(none).100 - P(\text{higher}) = P(\text{lower}) + P(\text{none}) \ge P(\text{none}).

When the panel expects no effect for most cells, a lower claim can count as surprising only if the panel confidently expected the opposite direction. The same holds whenever one direction’s probability is derived from a one-sided answer, for example by taking 100 minus the rated plausibility of “X improves Y” as that of “X harms Y”.

Under this metric, stages 5, 6 and 9 confirmed that measurement raises sv: Opus on NAS-Bench-201 from 1.08 to 2.12 per run (Holm p=.003p = .003), Sonnet on HPOBench from 1.83 to 2.71 and Opus from 4.17 to 4.67. An external audit then pointed to the elicitation; its numbers reproduced exactly from our caches (Table 8, Figure 3). The panel’s mean PP(higher) was 21–46, so no true lower claim was sv-eligible in any of the four model–benchmark pairs, while 57% (NAS-Bench-201) and 54% (HPOBench) of true claims are lower-direction. On HPOBench the two wordings of a proposition received opposite labels for 41–60 of 60 propositions. Measurement shifted proposals toward higher claims, the only direction the metric credited (Opus on NAS-Bench-201 from .27 to .53 of hypotheses, Sonnet on HPOBench from .41 to .55), and this shift accounts for +1.02 of Opus’s +1.04 and +0.68 of Sonnet’s +0.88 (Table 9).

Figure 3. The direction bias, Claude generators and Claude panel. (a) Share of higher-direction hypotheses, the only direction the original metric could credit; Opus on HPO barely changes. (b) Measure-first minus direct generation under the original metric, the corrected metric (SV3\mathrm{SV}_3, pre-registered rescoring) and the threshold-free calibrated score RR (post hoc), 24 runs per arm.

Stage 12 re-elicited a three-way prior (102 calls). Its pre-registration was written before any stage-12 call but after the audit, and the runs it rescored had already been analyzed, so it constrains the analysis but not the data. All three models put the largest share on “no effect” (47–70 on average); the two HPOBench wordings now differ by a median of 3 points; and 92% and 95% of true lower claims have a prior below 50. Under sv3_3 the measurement effect for Claude generators is refuted in three pairs and inconclusive in the fourth, and the pre-registered surprisal-weighted credit agrees. Keeping the original ratings for higher claims and using the new ones only for lower claims (pre-registered) gives the same picture (−0.42, −0.88, −1.12, −0.04), so the reversal comes from the missing lower-direction prior, not from the new question format. The stage-3 harness’s gain was measured on the corrected metrics and does not depend on this artifact.

Why the two corrected metrics disagree

For Claude generators with the Claude panel, ΔSV3\Delta\mathrm{SV}_3 is negative in three pairs while ΔR\Delta R is positive, though not significant, in three. With three outcomes most claims get a prior below 50, so SV3\mathrm{SV}_3 is close to a count of true, non-obvious claims and tends to fall when precision falls. RR credits each true claim by how unlikely the calibrated panel found it, so a shift toward lower-precision claims can offset a precision loss, and RR alone can reward a loss of truth: for GPT on NAS-Bench-201 it rewarded the judge-optimized harness, at 26% precision, by +1.04 under the Claude panel and +1.91 under GPT’s own prior, not far below measurement (+1.60, +2.69). Hence the pre-registered precision bound. Measurement meets it for Opus on both benchmarks and for GPT on HPOBench, and misses it for GPT and Sonnet on NAS-Bench-201 (.69 to .50, .79 to .45). The stage-3 harness meets it with room to spare (.79 to .83).

Sonnet does not discount unrepresentative evidence

Sonnet gained from measurement under no panel. For Sonnet on NAS-Bench-201, measurement lowers precision from .79 to .45. Post hoc, we split measure-first hypotheses by whether the model’s own probes point in the direction that its own blind three-way rating considered less likely of the two (Table 10). On NAS-Bench-201 such conflict claims rarely concern real effects (25–30% of the time), because single-parent contrasts often reflect one parent architecture’s idiosyncrasy. Of Sonnet’s claims, 37% are conflict claims, against 5% of Opus’s. A pre-registered prompt follow-up (stage 11; Appendix § 10) suggests that Sonnet discounts evidence described as noisy but not evidence described as unrepresentative: removing the “single training seed” wording on HPOBench raised its siding with conflicting evidence, while a truthful note on NAS-Bench-201 that each result comes from a single parent cell did not lower it. Asking Sonnet to propose only claims that its own knowledge also supports raised precision from .44 to .60 (Holm pp = .001), still below direct generation. Two of the stage’s four predictions missed their thresholds.

Bench.Gen.Conflict claims (share)True-cell sharePrecision: conflict / agree
NB201Sonnet86 (.37).30.24 / .56
Opus12 (.05).25.17 / .78
HPOSonnet21 (.09).57.48 / .88
Opus17 (.07).76.76 / .89

Table 10. Measure-first hypotheses split by whether the model’s own probes agree with the direction its own blind three-way rating prefers (post hoc, 24 runs per arm; hypotheses without probe evidence omitted). True-cell share: the share of conflict claims whose cell has a real effect in either direction. With the original one-directional rating the conflict shares are similar (Sonnet 42% on NB201, Opus 15%), but siding rates are not comparable across ratings (Sonnet on HPO: .61 against .10), because the two define the preferred direction differently.

Other harnesses did not beat direct generation

We rescored, post hoc, all 19 harness evaluations on fresh NAS-Bench-201 runs, 13 for Sonnet and 6 for Opus (the judge harness was evaluated twice per model). They include an L9 orthogonal-array screen of the structural knobs, the final harnesses of the three fitness arms, prior-avoiding selection, cross-domain retrieval, and a harness that designs 100 experiments it knows will not be run. None beats direct generation on sv3 after Holm correction (Table 27 in the appendix). On RR, two Sonnet harnesses do: the judge-optimized harness (+0.88, Holm pp = .04, at precision .43) and the stage-3 harness (+1.98, Holm pp = .02), which § 4.1 confirms. No Opus harness significantly exceeds direct generation on either score. The validity fitness, also grounded in truth, produced a harness that played safe (Table 11). With 8 runs per arm for 15 of the 19 evaluations, the smallest sv3 difference detectable with 80% power was about two claims per run (about three after Holm correction), so these are absence-of-evidence results. Cross-domain retrieval, the largest single gain reported for a recent ideation system ([31]), did not help on any metric.

Gen.StageHarnessnnΔprec.\Delta\mathrm{prec.}ΔSV3\Delta\mathrm{SV}_{3}HolmΔR\Delta RHolm
Sonnet1L9 cell (0,1,1,1)8−.09−1.501−2.021
1L9 cell (0,2,2,2)8−.45−1.121−1.151
1L9 cell (1,0,1,2)8−.03+2.00.343+1.08.405
1L9 cell (1,1,2,0)8−.51−2.121−1.431
1L9 cell (1,2,0,1)8−.01−0.751−1.251
1L9 cell (2,0,2,1)8+.03−0.121−1.091
1L9 cell (2,1,0,2)8−.56−1.751−1.401
1L9 cell (2,2,1,0)8−.14−0.501−0.711
2judge-fitness final8−.49−0.621−0.091
2validity-fitness final8+.01−0.751−0.541
3sv-fitness rerun final8+.14+2.00.172+1.98.023
5judge-optimized (24 runs)24−.36+0.71.385+0.88.038
8design-only, 100 probes24−.09−0.041−0.291
13sv-fitness rerun, confirmation24+.04+1.63–+1.77–
Opus3judge-fitness final8−.54−4.001−1.561
3validity-fitness final8−.06−0.751−0.581
3prior-avoiding selection8−.39−3.001−1.271
3cross-domain retrieval8−.14−0.621+0.041
5judge-optimized (24 runs)24−.59−4.211−2.241
8design-only, 100 probes24−.18−1.751−1.231

Table 27. Every internal-reasoning harness evaluated on fresh NB201 runs, minus direct generation from the same stage, under the leave-one-out Claude panel (post hoc rescoring). L9 cells give the levels of retrieval, roles, challenge and selection; level 0 is direct generation. Holm correction is within each generator. Stage 13 is a pre-registered confirmation on 24 new seeds, outside the Holm family (one-sided p<.001p < .001 for ΔSV3\Delta\mathrm{SV}_{3} and ΔR\Delta R).

Discussion

Why might the selection rule work? The confirmed harness’s rule, written by the optimizer, asks the generator in a separate selection call to keep the claims it believes but expects other models to doubt. Direct generation and likely-true selection keep the claims the generator is most confident about, which are also the claims every model expects. The intent is to spend the generator’s own knowledge where it differs from the consensus, which is what RR rewards. In our data the rule does more than filter claim type: from the same pools it beat random selection by 1.55 and an Other-type filter by 0.95 per run, and it raised precision within each type. But the kept claims are not those Sonnet’s separately elicited blind prior rates likely, so the mechanism in the rule’s wording, belief against expected doubt, remains a hypothesis; what we can show is that the selection step favors claims the panel doubts and that hold. The rule needs no external feedback at run time, so it is cheap to apply. Its 50% threshold echoes the score definition the optimizer was shown. We have tested it with only one generator on one benchmark, and on GPT’s pooled direct claims it helped on one benchmark and not the other.

When does evidence help? For GPT, measurement raised RR on both benchmarks only when the probe values were informative, so its proposals use the evidence. We have not shown that measurement is a good use of budget: on HPOBench a cheaper 40-claim call matched it, and on NAS-Bench-201 it cost precision. Nor have we tested how GPT weighs evidence against its own knowledge. Measurement never helped Sonnet, which followed unrepresentative probes and lost precision. For Opus a gain appeared only under panels at least as informed as Opus. One hypothesis, for a pre-registered test with a panel member of intermediate quality, is that a gain from evidence can register as surprise only relative to a panel that knows at least as much as the generator.

Surprise relative to whom, and contamination. RR measures surprise relative to the blind priors of other LLMs, not relative to experts or the literature. Both benchmarks are public [7, 10], and the strongest priors predict truth with AUC .88 to .95, so the models may have seen the tables or papers about them. Contamination that the panel shares is discounted by construction, because the calibrated prior already predicts those claims. A generator that recalls more than its panel, however, earns RR for recall rather than reasoning, which is one reading of the panel dependence in § 4.3. For the stage-3 harness this reading fits less well: Sonnet’s own blind prior predicts truth less well than Opus’s, with an AUC of .71 against .88; the gain is positive when scored against Sonnet’s own prior, at +1.25 per run, although not above zero with propositions resampled; and the kept claims are not those that prior rates likely. We cannot rule contamination out. A private or post-cutoff table would, but the one private-table transfer we ran (Appendix § 10) tested only the judge-optimized harness.

Checks for evaluating ideation systems. Each failure we met has a cheap check that would have caught it.

  • Report precision on a checkable subset next to any judge or surprise score. Judge-scored benchmarks would rank the judge-optimized harness higher, and RR alone rewarded it in two pairs.

  • Elicit priors over every outcome, not one direction, and audit the direction balance of the true claims a surprise metric can credit against that of all true claims. Ours was 0% against 54–57%, a table that needs no calls beyond the priors themselves.

  • Make the panel at least as informed as the generator. Report each member’s and the generator’s own AUC against truth. A leave-one-out panel from one model family is weakest for that family’s strongest model, which then earns credit for what it already knows. The generator’s own blind ratings are a cheap strong panel, although they capture its knowledge only in part.

  • Merge logically equivalent claims before scoring, and check that a proposition’s label does not depend on its wording.

  • Report a calibrated excess-truth score with its sensitivity to the calibration. Our GPT and Opus measurement gains on NAS-Bench-201 depended on how much the calibration trusted the prior; the stage-3 harness’s did not.

What pre-registration did and did not protect. Pre-registration [38] kept us from tuning analyses after seeing results, and it made the confirmation of the stage-3 harness and the reversal of the original measurement result equally easy to report. It did not protect the construct: the flawed metric was locked in before any data were collected and passed three confirmation stages. Construct checks such as a direction-balance audit need no calls beyond the priors and belong in the pre-registration, as a gate before any confirmatory stage.

Relation to recursive self-improvement. Harness evolution succeeds when selection uses executable truth [48, 52], and reviewer agents keep experimental claims honest when scores exist [8]. Without an executable score, evolution here failed with a judge. With a sparse fitness that counted only true, low-prior claims, the original svsv that we later found to be biased by direction, one of two optimizer runs produced a harness that we singled out post hoc among 19 evaluations. For one generator on one benchmark, its gain held on corrected scores, fresh seeds and every panel we tried. It rests partly on a few recurring claims, however, and with those resampled it is not distinguishable from zero against the generator’s own prior. The harness encodes the objective in its selection step, and replacing only that step removed the gain. Whether harnesses found this way transfer to other generators and benchmarks is the most important open question this work leaves.

Limitations

  • The stage-3 harness was found in one of two optimizer runs and confirmed with one generator (Sonnet) on one benchmark (NAS-Bench-201). Its gain rests on a few recurring propositions: without the 10 that contribute most it is not significant against two of three comparison arms. The harness and its ablation were run a day apart; their pools are similar, with pool RR differing by +0.19 per run at p=.76p = .76, so we attribute little of their difference to the batch. With propositions resampled along with runs, its gain is above zero under the main panel but not under Sonnet’s own blind prior, and its credit term is only marginal against direct generation (p×=.054p_{\times} = .054).

  • Conditional-interaction claims on tabular benchmarks are a narrow proxy for research ideas. Both benchmarks are public, and the strongest models’ priors predict truth with AUC .88–.95, so they may partly know the tables (§ 6).

  • RR before stage 12b and all sensitivity analyses are post hoc, and RR depends on a calibration fitted to truth over the claim space. The measurement gains for GPT on NAS-Bench-201 and for Opus do not survive heavy shrinkage of that calibration, and per distinct claim GPT’s HPOBench gain over pooled direct generation with the stage-3 rule is not significant. On NAS-Bench-201 the probes’ test-split values almost equal the validation values that define truth. The three-way prior may itself be pulled toward an even split over its three outcomes.

  • The judge is Opus except in the stage-14 transfer test with GPT, and the prior panel is Claude except in stage 12b and the post hoc panel analyses. GPT ran through codex-cli, an agent shell whose behavior may differ from the API, and neither interface controls sampling temperature. Opus is the judge, the optimizer’s proposer and a panel member, so self-preference may favor Opus outputs [41]; the leave-one-out panel excludes the generator but not its model family.

  • Stage 12 rescored runs that had already been analyzed, so its pre-registration constrains the analysis but not the data.

  • Stages 13 to 15 compare new arms with direct and measure-first runs made up to three days earlier. For Sonnet, 24 direct runs made alongside stage 15 differed from the stage-5 batch by −0.18 per run, and the stage-3 harness’s gain held against them. For GPT, direct runs repeated one to two days later changed RR by at most 0.3 per run. Opus’s direct runs varied more across batches (2.98 to 3.71), but no comparison in stages 13 to 15 uses an Opus generator.

  • In one of 24 Sonnet measure-first runs on NAS-Bench-201, all 100 probes failed to parse because the model abbreviated operation names. Excluding it leaves svsv at 1.17 and precision at .44.

  • The optimizer budget is small, and most harness comparisons have 8 runs per arm, so negative results for harnesses may partly reflect search limits and low power. The judge result rests on one judge-optimized harness from one optimizer run, evolved with Sonnet and transferred unchanged; it shows that judge optimization can cost precision, not how often it does.

The 16 pre-registration documents are listed with their SHA-256 hashes in Table 23 and are included in the supplement and at https://osf.io/c2hkg/?view_only=d05d8b1acd504e6ab87a242522b769df; every model call on the two public benchmarks, the analysis scripts and a command-line tool for checkable hypothesis generation will be released.

DocumentCreated (UTC)Last modified (UTC)SHA-256
PREREG_s1.md2026-10-21 00:562026-10-21 00:569d7125b6b956db218b5252ec4a8eda318ef03de5d72a94c941e89ec0aaa57112
PREREG_s2.md2026-10-21 03:562026-10-21 03:56ff825cd04b8ccb9a6c0d06804584b5ff21e384014d4e64d7af32010abe1de50c
PREREG_s3.md2026-10-21 04:532026-10-21 04:5307f52c56b97a846b20fa78be1e0fd31159aae5197448627d0e4fd3cd32aa005a
PREREG_s4.md2026-10-21 05:372026-10-21 05:522a9c42cdb788d5c2c19c21657929bc4277b1065e97d3d1431db16b10c7551ee5
PREREG_s5.md2026-10-21 06:042026-10-21 06:0452d567d9091c092c341e06b02dacbb3bf97bb55ab83936566b3fa88b2a448840
PREREG_s6.md2026-10-21 07:292026-10-21 07:292c2f7d9899d4210bbef90f91694a6d10ff27716c047862f458398dbfbde0f493
PREREG_s7.md2026-10-21 08:042026-10-21 08:047013979422c869a0dc159786fdec1fed31ad0cea00c3e50edbf09b8e3f32a569
PREREG_s8.md2026-10-21 08:152026-10-21 08:158df7d217a437e8a56b7c1a735b5da29f65f7697258e844fd735bc0279d939643
PREREG_s9.md2026-10-21 08:442026-10-21 08:4499ab055f56cecab5fec9c2c4346579897e1b5491f05f393608b05dc1495feec4
PREREG_s10.md2026-10-21 18:582026-10-21 18:589cf1f532e70bd80fb001f9fcd80a415d2f29f9827608c3c9dcb5c73f1916665c
PREREG_s11.md2026-10-21 23:312026-10-21 23:317bad9c7e24822fdbf6190eacacf4cae49a24f5522c1b4d532e30f39ee31b4bb2
PREREG_s12.md2026-10-22 03:432026-10-22 03:43eeed4897748232b2eb411e81b552e21694c5ea022b574b63a79ea75db4a02671
PREREG_s12b.md2026-10-22 05:372026-10-22 05:37158e59b535e715ee245d12b6c95c81162e8208d4114f91912eaee107312075dd
PREREG_s13.md2026-10-23 05:212026-10-23 05:211db098e8f4aba34a33daf086a80157d3529a5c8e8e13f015671b896db7c79aa7
PREREG_s14.md2026-10-23 18:412026-10-23 18:41c37ca970ea6c17e170f25fc9c99c0d635f156858a597c888a3a5821f167e2857
PREREG_s15.md2026-10-24 01:422026-10-24 01:429a4baae40153af6de60f80eb0064aa0488b5c2def43946b845cd4bafa86a5d05

Table 23. The 16 stage pre-registration documents (stage 4b is an addendum inside PREREG_s4.md). Times are the local filesystem’s creation and last-modification times, converted to UTC; they are not third-party timestamps. The documents are in the supplementary material and in an anonymized OSF deposit, https://osf.io/c2hkg/?view_only=d05d8b1acd504e6ab87a242522b769df.

Broader Impact Statement

This work evaluates automated research-ideation systems. A reader could treat a gain on a narrow checkable subset as evidence that a system produces true and valuable ideas in general; our scores certify neither outside a checkable setting. Systems tuned to an LLM judge can produce plausible but false claims, as the judge-optimized harness did. An objective that rewards surprise relative to a panel rewards claims that contradict consensus, which is useful only when truth is checked; the selection rule we confirm should be paired with such checks. The study depends on closed models whose behavior may change. The public benchmarks contain no personal data.

Prompts

All prompts are shown for NAS-Bench-201. The HPOBench versions use their own benchmark description, claim wording and JSON fields (edited parameter and step direction, conditioning parameter and band), and describe each probe as two configurations trained once with a single training seed. Text in angle brackets is filled in per call. Claude models were called through the Claude Code CLI with tools disabled, and GPT through codex-cli in a read-only sandbox in an empty working directory. Prompts for the harness roles, the stage-1 screen, the stage-11 variants and the optimizer are in the released code.

Benchmark description (prefix of every NAS-Bench-201 prompt).

Benchmark: NAS-Bench-201 cell search space. A cell is a DAG with nodes 0 (input) to 3 (output) and six edges
    ; each edge carries one operation.
Edges: edge 0 = node 0 -> node 1; edge 1 = node 0 -> node 2; edge 2 = node 1 -> node 2; edge 3 = node 0 ->
    node 3; edge 4 = node 1 -> node 3; edge 5 = node 2 -> node 3.
Operations: none, skip_connect, nor_conv_1x1, nor_conv_3x3, avg_pool_3x3 (’none’ removes the edge).
Datasets: CIFAR-10, CIFAR-100, ImageNet16-120; fitness is validation accuracy of the full network built by
    stacking the cell.
An edit sets one edge j to operation b. A condition is ’parent edge k currently has operation c’ (k may
    equal j, in which case it says what op b replaces).
A hypothesis states that the edit’s effect on validation accuracy is HIGHER (or LOWER) when the parent
    satisfies the condition than when it does not, consistently across the three datasets and by a
    practically meaningful margin. It is about the interaction, not the edit’s average effect.

Direct generation.

<benchmark description>
(Independent attempt #<n>.)

Propose 10 conditional hypotheses about this benchmark that you believe are true.
End with JSON only: {"hypotheses": [{"edit_edge": <0-5>, "edit_op": "<op>", "cond_edge": <0-5>, "cond_op":
"<op>", "direction": "higher" or "lower", "why": "<one sentence>"}, ...]} with exactly 10 distinct
hypotheses.

Measure-first: design call, then proposal call.

<benchmark description>
(Independent attempt #<n>.)

You may run 100 experiments before proposing hypotheses. An experiment applies one edit to one parent cell
you choose and returns the change in test accuracy (percentage points) on the three datasets. Design
the experiments that will best reveal conditional effects; you will see the results and then propose 10
hypotheses.
End with JSON only: {"experiments": [{"parent": ["<op edge 0>", ..., "<op edge 5>"], "edit_edge": <0-5>, "
edit_op": "<op>"}, ...]} with exactly 100 experiments.
<benchmark description>
(Independent attempt #<n>.)

Before proposing, you ran 100 real experiments on this benchmark. Each experiment applied one edit to one
    parent cell and measured the change in test accuracy (percentage points) on the three datasets; these
    are real measurements and can serve as evidence:
1. parent (e0=<op>, ..., e5=<op>); edit: edge <j> -> <op>; change C10 <+x.xx>, C100 <+x.xx>, IN16 <+x.xx>
...

Propose 10 conditional hypotheses about this benchmark that you believe are true.
<same JSON format as direct generation>

Original prior elicitation (one direction, complement-coded). Each call rates 30 cells.

<benchmark description>

For each numbered claim, give your probability (0-100) that it holds (consistent across the three datasets, practically meaningful margin). If you think the effect is instead LOWER under the condition, give a low number.

  1. setting edge <j> to <b> has a HIGHER effect when parent edge <k> is <c> than when it is not ... End with JSON only: {"p": {"0": <0-100>, ...}} covering every claim.

Corrected prior elicitation (three outcomes, stage 12).

Each call rates 30 cells.

<benchmark description>

For each numbered (edit, condition) pair, compare the edit’s effect on validation accuracy when the condition holds with its effect when it does not. Give three probabilities (0-100, summing to 100): H = the effect is HIGHER under the condition, L = the effect is LOWER under the condition, N = neither (no difference that is consistent across the three datasets and of practically meaningful size).
0. edit: set edge <j> to <b>; condition: parent edge <k> is <c>
...
End with JSON only: {"p": {"0": [H, L, N], ...}} covering every pair.

Judge. The judge sees each hypothesis as JSON, including its one-sentence “why”; stage 13a repeats the judge with that field removed, and stage 14 repeats it unchanged with Opus and with GPT as the judge.

<benchmark description>

A research assistant proposed these hypotheses:
{"hypotheses": [{"edit_edge": 4, "edit_op": "nor_conv_3x3", "cond_edge": 4, "cond_op": "none", "direction": "higher", "why": "<the generator’s one-sentence reason>"}, ...]}

Rate this set as a reviewer would, without running anything: novelty (1-10: would an expert find these surprising and non-obvious?) and quality (1-10: soundness, specificity, plausibility). End with JSON only: {"score": {"novelty": <1-10>, "quality": <1-10>}}

The evolved judge-optimized harness.

Roles: gap finder, innovator and critic; challenge: negate assumptions; pool of 20, reduced to 10 by the rule below; no retrieval.

Selection rule: Discard any hypothesis whose condition is the edited field’s own current value, or whose condition merely disables or disconnects the edited component. From the rest, keep 10 with distinct mechanisms and field pairs, favouring those where an informed expert could plausibly bet on either direction.

Strategy: Two patterns are banned because experts find them obvious: (1) conditioning on the current value of the same field being edited, which just restates that upgrading a weak setting helps more than upgrading a strong one; (2) conditions that make the edited component unused, unreachable or inert. Instead, each hypothesis must link two DIFFERENT fields that interact through a non-trivial mechanism, such as substitution between parallel routes, a capacity already supplied elsewhere, one part acting as a bottleneck for another, or interference that turns a usually helpful edit harmful. No condition setting may appear in more than two hypotheses. Use the gaps and negated assumptions as the source of each idea. In each ’why’, name the mechanism and explain why the opposite direction is also a priori plausible.

The harness evolved for verified surprise (stage 3, confirmed in stage 13).

Roles: single; no challenge; pool of 30, reduced to 10 by the rule below; no retrieval.

Selection rule: Keep hypotheses you rate at least 55% likely true but expect other models to rate below 50%, and name the specific naive reasoning that would mislead them. Drop anything you rate below 50% yourself, and anything obvious from structure alone. Keep fields and directions diverse.

Strategy: Propose a broad, varied set of conditional hypotheses covering many different edited fields, conditioning fields and both directions. For each, briefly note your own probability that it holds and how an intuitive reader would judge it.

Additional results

Judge harness on a private table. A descriptive transfer to a private, unreleased table of model results from another domain, pre-registered in stage 2, showed the same pattern as Table 5 (judge 6.63 to 8.75, precision .40 to .03; 8 runs per arm). About 3% of that table’s directional claims are true, so the judge-optimized harness fell to chance precision there. We report it for completeness and do not rely on it; its runs are not part of the release. For GPT as generator, the original judge gains were +0.58 on NAS-Bench-201 and +0.29 on HPOBench (p=.005p = .005 and .04).

Judge rejudgings in detail. The judge sees each hypothesis with its one-sentence “why”, and the evolved strategy prescribes how to write it (“name the mechanism and explain why the opposite direction is also a priori plausible”). Stage 13 removed the “why” and left the prompt otherwise unchanged. The judge still preferred the judge-optimized sets in four of six pairs (verdict partial), and in every pair the gain was in novelty (+0.62 to +1.42), not in quality. Stage 14 repeated the original prompt. The retest reproduced the gain (+0.46 to +1.92). Only for Sonnet and Opus on HPOBench did the explanations add to it, by 1.21 points each (Holm p<.001p < .001). For Sonnet and Opus on NAS-Bench-201 we found no significant effect, and for GPT on NAS-Bench-201 removing the explanations raised the gain by 0.79 (Holm p=.012p = .012), which we did not predict. Within an arm, a run’s original and retest totals barely correlate (r=−.22r = -.22 to .62), so single-run judge scores are unreliable and only arm means reproduce. Stage 14 also gave the original prompt to GPT, a transfer test, since the harness was evolved against Opus. GPT rated the judge-optimized sets higher in novelty and lower in quality in all six pairs (post hoc, p≤.001p \le.001 for each), so its total rose only slightly (0 to +0.33). The rise was significant in three pairs, so the pre-registered verdict is partial. GPT’s ratings were nearly binary: novelty was 2 for 134 of 144 direct sets and 3 for 137 of 144 judge-optimized sets. When GPT judged GPT’s runs, both arms were its own, so self-preference would affect both.

Measurement variants and threshold sweep. With the Claude panel, every 24-run measurement variant for Opus on NAS-Bench-201 scores below direct generation on SV3_3 (6.96): exact values 6.21, simulated single-seed noise 5.92, doubled noise 5.88, values withheld 5.88, a 12-epoch proxy 5.25, and values shuffled across probes 4.58. A threshold sweep counts only true claims at the panel rated by 33 (pre-registered as descriptive) or 20 (post hoc). For Opus, measurement gains +0.92 at 20 and +0.79 at 33 on HPOBench, and +0.12 and +1.00 on NAS-Bench-201 (p≤.01p \le.01 except +0.12, p=.40p = .40). With GPT in the panel (post hoc), the gains at 33 are +0.88 and +0.75 (p<.01p < .01), but at 20 they are +0.00 and +0.04, because that panel rates almost no true claim below 20.

How the models use conflicting evidence. When the models make conflict claims (Table 10), they follow the probes on NAS-Bench-201 (92% for Sonnet, 83% for Opus, pooled over claims) but less often on HPOBench (10%, 24%), where probes are described as single training runs. A pre-registered follow-up (stage 11) changed only the propose prompt and reused Sonnet’s own probes (23–24 paired runs per arm); its conflict measures used the original one-directional rating. Removing the “single training seed” wording on HPOBench raised Sonnet’s siding with conflicting evidence from .61 to .80 and lowered precision by .06 (Holm p=.007p = .007; short of the pre-registered .10). A truthful note on NAS-Bench-201 that each result comes from a single parent cell did not lower siding (.91 to .95; predicted below .75) and changed precision by only +.04. Asking Sonnet to propose only claims that its own knowledge also supports raised precision from .44 to .60 (Holm $p = .001), still below direct generation, but cut conflict claims only from 40% to 30% (predicted below 25%). Recomputed post hoc with the three-way rating of Table 10, which defines the preferred direction differently, the shifts are .11 to .42, .91 to .89, and 37% to 26%; under neither rating does the note lower siding toward the predicted .75. The contrast is confounded with the benchmark and with adding versus removing text, and neither the two prompt-wording precision predictions nor the two mechanism predictions reached their thresholds.

Selection analyses of the stage-3 harness

All analyses in this section are post hoc and use only the cached generation pools, selections, runs and priors of the stage-3 harness (stage 13) and its likely-true ablation (stage 14), Sonnet on NB201, 24 runs each. (script post_pos_selection.py). RR uses the calibration of the main text (leave-one-out Claude panel). A pool holds 29.5 distinct valid claims on average, and every kept claim is in its pool.

Selection baselines. Each baseline keeps as many claims from each pool as the arm kept (10), and is repeated 5,000 times (Table 12). “Other first” keeps claims of the Other type (Table 6) before any other, in random order, and fills up at random; “non-self first” does the same for every claim that does not condition on the edited edge. The draw distribution holds the pools fixed, so it measures the selection step, not the sampling of pools; the paired test compares each run’s kept RR with that run’s expected baseline RR (500 draws) and flips signs over runs.

PoolsSelectionRR/run [5%, 95% of draws]Prec.Kept − baselinePaired: runs kept >, pp
Stage-3 harnessRule (kept)+2.87.825
Random+1.32 [+0.92, +1.72].745+1.5524/24, < .001
Other first+1.92 [+1.57, +2.27].739+0.9522/24, < .001
Non-self first+1.45 [+1.18, +1.73].646+1.42
Other, then removes-edge+1.62 [+1.35, +1.89].648+1.25
AblationLikely-true (kept)+0.91.824
Random+1.26 [+0.89, +1.66].749−0.367/24, .96
Other first+1.55 [+1.20, +1.88].683−0.648/24, .99

Table 12. Selection from the same pools of 30. No baseline draw reached the rule’s RR. Direct generation (stage 5) has RR = +1.10 per run.

Proxies for “believed but doubted”. The rule asks for claims the generator believes and expects others to doubt. Post hoc, we imitated that intent with the cached blind priors, keeping from each pool as many claims as the harness kept, in a fixed order with random tie-breaking (200 tie-breaks; script post_pos_round6b.py). Keeping the claims the leave-one-out panel rates least likely gives RR = +3.42 per run (precision .66). Keeping first the claims that meet the rule’s thresholds under blind priors (Sonnet’s own ≥55\ge55, the panel’s < 50; 3.5 per pool, 12% of pool claims), then the rest by panel prior, gives +3.47 (.70); keeping the lowest calibrated priors gives +3.38 (.65). Each beats the harness’s own selections (+2.87, precision .83; paired sign-flip p≥.989p \ge.989 for the harness being better). These selectors, however, read the panel’s priors, which are exactly what RR subtracts, and the harness never sees them. Selectors that use Sonnet’s blind prior do worse: the largest gap between Sonnet’s blind prior and the panel’s gives +1.88 (harness better by +0.99, pp = .003), and Sonnet’s most confident claims give +0.35 (+2.52, pp < .001). Measured against random selection (+1.32) and the panel-informed ceiling (+3.42), the harness recovers about three quarters of what is available, at higher precision than any panel-informed selector. Finally, we scored each pooled claim from generator-side signals only, without reading the scoring panel’s priors: in-context selection support combined with the generator’s own blind probability Pown(c)P_{\mathrm{own}}(c), minus a ridge-fitted structural prior P^struct(c)\hat{P}_{\mathrm{struct}}(c) (script post_pos_excess_truth.py). This selector is post hoc and its weights were set by hand after the results above were known; it also selects from pools of 30 or more claims, whereas a direct run proposes 10. It raised RR over direct generation in all six benchmark–generator pairs, but significantly in only two: Sonnet on NAS-Bench-201 (+0.68, pp = .008) and GPT on HPOBench (+1.18 → +2.06, pp = .0003, precision .898 → .917), the pair where the stage-3 prompt rule gave no gain over its comparison arm. The other four differences are +0.14 to +0.43 (pp = .07–.20). We read this as a hypothesis for why the prompt rule did not transfer, not as evidence.

Claim types. Shares of claim instances (precision): harness pools self-condition .59 (.83), removes-edge .31 (.59), Other .10 (.72); harness kept .57 (.93), .23 (.62), .20 (.77); ablation pools .62 (.82), .27 (.70), .11 (.46); ablation kept .68 (.89), .32 (.68), .00; direct generation, stage 5, .68 (.87), .28 (.65), .03 (.25); stage 15, .63 (.87), .29 (.59), .08 (.21). Within each type, the claims the rule kept were more often true than those it left (self .93 against .78, removes-edge .62 against .59, Other .77 against .64) and had higher mean T−πT-\pi (+0.22 against +0.07, +0.32 against −0.01, +0.46 against +0.35).

Compliance. We measure the rule’s two thresholds with priors elicited separately from the runs: Sonnet’s own blind three-way prior (≥55\ge55) and the leave-one-out Claude panel’s prior (<50< 50). Of the kept claims, 27% have an own prior of at least 55 (pool claims not kept: 45%; likely-true ablation kept: 60%; direct generation: 54%), 67% a panel prior below 50 (not kept: 54%; ablation: 40%; direct: 45%), and 6% both (not kept: 15%). Within the pools, Sonnet’s blind prior does not discriminate truth (AUC .51; panel prior .57; being kept .57). Five of 24 selection replies mention other models, and one says it did not estimate their ratings. The generator’s in-context probabilities are not recorded in structured form, so we cannot test compliance with them directly.

Concentration. The harness’s 240 hypotheses contain 93 distinct propositions (68 true); the stage-5 direct runs contain 62 (41), the stage-15 direct runs 76 (49) and the ablation 41 (33). Table 13 attributes the gain to propositions: a proposition’s contribution is its summed T−πT-\pi in the harness arm minus that in the comparison arm, per run. The largest contribution in all three comparisons comes from a false claim with π=.97\pi=.97 that the comparison arms propose often and the harness avoids. Counting each distinct claim once per arm, the gain over direct generation is +0.60 per run (stage 5), +0.57 (stage 15) and +0.71 over the ablation, with 90% run-bootstrap intervals above +0.3 in each case (the bootstrap understates the number of distinct claims, so these intervals are only indicative).

ComparisonΔR\Delta R510k=3k=3k=5k=5k=10k=10
Direct (stage 5)+1.77+1.10+1.70+0.94 (< .001)+0.67 (.006)+0.07 (.38)
New direct (stage 15)+1.95+0.98+1.47+1.20 (< .001)+0.97 (.001)+0.48 (.02)
Likely-true ablation+1.96+1.35+1.98+0.90 (< .001)+0.62 (.005)−0.02 (.53)

Table 13. Concentration of the stage-3 harness’s gain over its three pre-registered comparison arms. Parentheses: one-sided permutation pp over runs. Dropping propositions chosen for their contribution biases these differences toward zero.

Batch. The harness (stage 13) and the ablation (stage 14) generated their pools with the same prompt a day apart. The pools are similar: 29.5 and 29.3 distinct claims per run, precision .746 and .749, Other share .10 and .11, pool RR +3.90 and +3.71 per run (difference +0.19, two-sided p=.76p=.76). Their Other-type claims differ in precision (.72 against .46, on 69 and 76 claims). Decomposing the +1.96 difference between the arms: the rule beats random selection from its own pools by +1.55; random selections from the two batches differ by +0.06 (two-sided $p=.81); and random selection beats likely-true selection in the ablation’s pools by +0.36. Most of the difference therefore comes from the two selection steps, not from the batch, although the harness itself was not rerun alongside the ablation.

Sensitivity analyses

All analyses in this section are post hoc and use only cached runs and priors (script post_pos_sensitivity.py, 4,000 permutations, seed 28). Differences are in claims per run, arm minus comparison arm, 24 runs per arm; * one-sided permutation p<.05p<.05, uncorrected.

Truth definition and harness procedure

Truth. Let X\mathcal{X} be the benchmark’s configurations and acc⁡D(x)\operatorname{acc}_{D}(x) the validation accuracy of xx on dataset DD, in percentage points (NB201: CIFAR-10, CIFAR-100 and ImageNet16-120; HPOBench: the mean over five training seeds on each of three OpenML tasks). An edit ee maps a parent xx to a child e(x)e(x) that differs from xx in one parameter jj: on NAS-Bench-201, edge jj is set to operation bb; on HPOBench, parameter jj moves one grid step up or down. The edit’s parents PeP_e are the configurations for which it changes something: on NAS-Bench-201 those with xj≠bx_j\ne b, so no parent is a no-op; on HPOBench those not at the end of the grid in the direction of the step. Its effect on a parent is ΔD(x)=acc⁡D(e(x))−acc⁡D(x)\Delta_D(x)=\operatorname{acc}_D(e(x))-\operatorname{acc}_D(x). A condition (k,c)(k,c) splits the parents into S={x∈Pe:xk=c}S=\{x\in P_e:x_k=c\} (on HPOBench, xkx_k lies in band cc) and Pe∖SP_e\setminus S, and the conditional difference is the unweighted difference of means

dD(e,k,c)=1∣S∣∑x∈SΔD(x)−1∣Pe∖S∣∑x∈Pe∖SΔD(x).d_D(e,k,c)=\frac{1}{|S|}\sum_{x\in S}\Delta_D(x)-\frac{1}{|P_e\setminus S|}\sum_{x\in P_e\setminus S}\Delta_D(x).

A claim (e,k,c,s)(e,k,c,s) with direction s=+1s=+1 (higher) or −1-1 (lower) is true when sdD(e,k,c)≥τs d_D(e,k,c)\ge\tau on every dataset DD; the variants of Appendix Appendix § 12 replace “every dataset” with “at least two of three” or with the mean of dDd_D over datasets. The condition may be on the edited parameter itself (k=jk=j), where it states which value the edit replaces. A cell is kept only when both groups have at least 100 parents (NAS-Bench-201) or 500 (HPOBench).

Counts. NAS-Bench-201 has 56=15,6255^6=15,625 architectures. Its 30 edits (six edges, five operations) each have 4⋅55=12,5004\cdot5^5=12,500 parents, and 29 conditions (six edges times five operations, less k=jk=j, c=bc=b, which no parent satisfies). All 30×29=87030\times29=870 cells meet the minimum, giving 1,740 directional claims. HPOBench has five parameters with ten grid levels each, except depth with three. Depth’s levels are its bands; for the others, levels 1–3 are low, 4–7 mid and 8–10 high. Ten edits (five parameters, up or down) times 15 conditions (five parameters, three bands) give 150 cells, of which 148 meet the minimum (296 claims).

Equivalent wordings. On HPOBench, for k≠jk\ne j, the pairs (parent, child) of “move jj up” are the pairs (child, parent) of “move jj down”, and the edit does not change xkx_k. So “moving jj up has a higher effect when xkx_k is in band cc” and “moving jj down has a lower effect when xkx_k is in band $c” have the same dDd_D up to sign and the same truth. They are merged into one claim before scoring, and their priors are averaged.

Worked example. Take the claim “setting edge 2 to a 3×33\times3 convolution has a lower effect when edge 0 is none”. Of the edit’s 12,500 parents, 2,500 have edge 0 set to none. The conditional differences are −1.93, −2.73 and −2.57 pp on CIFAR-10, CIFAR-100 and ImageNet16-120, so the stated direction holds on all three datasets. The CIFAR-10 magnitude is below τ=2\tau=2, so the claim is false in the main analysis, though true at τ=1\tau=1 and 1.5. Direct generation proposes this claim often, the calibrated panel gives it π=.97\pi=.97, and

Def.: the calibration of the main text (ridge λ=10−3\lambda= 10^{-3}, prior clipped to [0.5, 99.5], covariates logit prior, order, direction and logit×order). λ\lambda: ridge penalty. Prior: logit prior only. +Type: adds indicators for the edited value and the condition (NB201: operations; HPO: band, edit direction and edited parameter). CV: 10-fold cross-fitting by cell. Prop.: calibration fitted only on the claims the two compared arms proposed, not on the whole space; with an intercept this makes the two arms’ summed RR zero, so it asks only whether one arm’s claims beat the other’s at equal prior. Raw: uncalibrated ∑(T−p/100)\sum(T - p/100). Sig./14: number of the 14 calibrations (the columns shown plus λ∈{0,0.1,1}\lambda\in\{0,0.1,1\} and clipping at 0.1, 1, 2.5 and 5) with p<.05p < .05; Raw is not among them. Clipping never changes a difference by more than 0.02, cross-fitting by more than 0.09.

it is the largest single contributor to the stage-3 harness’s gain (Section 4.1; Appendix F counts such near misses).

Optimizer. The optimizer is a single Opus prompt per round. It sees the harness fields and their allowed values, the token cap, and the definition of the score. For the SV arms the score is “the number of the 10 hypotheses that turn out TRUE when tested against measured results AND that other strong language models rate as less than 50% likely to hold”. It also sees the full history of harnesses tried, with mean score, mean tokens and status, the current best harness, and one unlabeled output of it. It does not see per-hypothesis truth or priors. It proposes 3 candidates per round for 3 rounds. Each candidate is run 6 times, and the round’s best is accepted if it beats the incumbent. Text fields (a selection rule of at most 50 words, a strategy of at most 150) must be domain-general: a regular expression rejects benchmark-specific terms.

The stage-2 to stage-3 fix. In stage 2 that expression’s pattern conv\w* also matched “conventional”. It wrongly rejected two of the three round-1 candidates of the SV arm, and one round-2 candidate of the true-count arm. Stage 3 narrowed the pattern to conv, convolution… and conv followed by a digit, and reran the SV arm with fresh optimizer calls. It also re-estimated the starting harness (direct generation) on 12 runs instead of 6: the 6-run estimate, 1.5, had been above stage-1 direct generation’s 1.25, which made candidates harder to accept. From the 12-run start of 1.42, one of the nine stage-3 candidates was accepted: the stage-3 harness, in round 3, with a training score of 1.67.

Where the thresholds come from. The selection rule, including its 55% and 50% thresholds, was written by the optimizer in its round-3 proposal, as free text in the selection-rule field. We did not choose or tune either value. The 50% echoes the score definition the optimizer was shown. Its stated rationale was: “Selects directly on the gap between the generator’s calibrated belief and naive expectation, which is exactly what the scoring rule rewards, without pushing for risky sign reversals.”

How the harness runs. A run makes two stateless calls to the generator.

Generation call. The prompt contains the benchmark description, an attempt number, “Approach to follow:” followed by the strategy (Appendix § 9), “Propose 30 conditional hypotheses about this benchmark that you believe are true”, and the output format. The format has fields for the edit, the condition, the direction and a one-sentence “why”. It has no field for a probability. None of the 709 hypotheses in the 24 confirmation pools states a probability, although the strategy asks for one.

Selection call. This is a separate call. It contains the benchmark description, the 30 candidates as JSON (with their “why”), and “Select the 10 that best satisfy this criterion:” followed by the rule, “Copy them exactly”, and the format. The ablation’s selection call reads “Select the 10 most likely to be true” instead. Any probabilities the rule refers to therefore exist only inside the selection call and are not recorded.

One selection reply put this directly: “The probabilities come from my own reasoning about NAS-Bench-201, and I did not estimate how other models would rate them.” This is why compliance (Appendix § 11) can only be checked against separately elicited ratings.

Surprising truths or avoided errors, and the precision bound

All analyses in this section are post hoc and use cached runs only. The script is post_pos_decomp.py.

Splitting RR. Because Th∈{0,1}T_h \in\{0,1\}, a run’s score splits as

R=∑h true(1−πh)⏟credit−∑h falseπh⏟costR = \underbrace{\sum_{h\ \mathrm{true}}(1-\pi_h)}_{\text{credit}} - \underbrace{\sum_{h\ \mathrm{false}}\pi_h}_{\text{cost}}

A difference in RR can therefore come from true claims, each credited by how little the panel expected it (credit), or from fewer false claims the panel expected (cost avoided). Only the first is surprise in the intended sense, although it also gives partial credit to true claims the panel expected. The second rewards avoiding consensus errors, as a precision gain does. Table 18 splits every headline comparison.

Bench.ComparisonΔR\Delta RCreditCost avoidedπˉtrue\bar{\pi}_{\mathrm{true}}ΔR\Delta RCreditCost avoidedPrec. ratio
NB201Stage-3 harness vs direct+1.77+1.04*+0.73*0.58/0.70+1.25+0.80*+0.45*1.05
vs new direct+1.95+1.13*+0.82*0.58/0.69+1.55+1.09*+0.46*1.12
vs likely-true ablation+1.96+1.22*+0.74*0.58/0.73+1.16+0.66*+0.50*1.00
HPOGPT measure-first vs direct+0.94+0.69*+0.25*0.69/0.77+0.61+0.40*+0.211.05
vs pooled direct+0.70+0.76*−0.060.69/0.78+0.72+0.80*−0.090.98
vs pooled + stage-3 rule+0.94+0.69*+0.25*0.69/0.65+0.53+0.30*+0.24*1.15
vs 40-claim call+0.09+0.18−0.090.69/0.71+0.42+0.57*−0.150.97
vs shuffled values+1.42+0.93*+0.49*0.69/0.70+1.01+0.56*+0.45*1.63
vs withheld values+0.85+0.53*+0.32*0.69/0.78+0.51+0.20+0.31*1.05
NB201GPT measure-first vs direct+1.60+0.46*+1.14*0.44/0.68+2.69+1.41*+1.28*0.72
vs pooled direct+2.24+0.97*+1.28*0.44/0.75+2.97+1.50*+1.47*0.71
vs shuffled values+1.40+1.66*−0.26*0.44/0.44+0.98+1.48*−0.50*2.64
vs withheld values+0.89−0.12+1.01*0.44/0.59+1.89+0.84*+1.05*0.72
Judge-opt. vs direct, Sonnet+0.88+0.43*+0.45*0.35/0.70−1.05−1.13*+0.080.54
Opus−2.24−2.37*+0.120.25/0.48−1.07−0.74*−0.32*0.32
GPT+1.04−0.38+1.41*0.30/0.68+1.91+0.57*+1.35*0.38
HPOJudge-opt. vs direct, Sonnet−0.89+0.11−1.00*0.60/0.79−1.63−0.64*−0.99*0.53
Opus−1.69−0.74*−0.95*0.36/0.60−1.24+0.29*−1.53*0.47
GPT−0.87−0.07−0.80*0.59/0.77−0.58−0.02−0.56*0.39

Table 18. RR split into the credit for true claims, ∑true(1−π)\sum_{\mathrm{true}}(1-\pi), and the cost of false claims, ∑falseπ\sum_{\mathrm{false}}\pi, per run; ΔR\Delta R is the difference in credit plus the difference in cost avoided (post hoc, paper-default calibration). *one-sided permutation p<.05p < .05 in the direction shown. πˉtrue\bar{\pi}_{\mathrm{true}}: mean calibrated prior of the true claims in the first and the second arm. Prec. ratio: precision of the first arm over the second; the pre-registered bound is 0.8. For the stage-3 harness, near misses (claims whose stated direction holds on all three datasets but by less than τ=2\tau= 2 pp) account for 0.44 of its false-claim cost per run against 0.97 for direct generation, 0.94 for the new stage-15 direct runs and 1.22 for the ablation; its other false claims cost 0.10, against 0.31, 0.42 and 0.07.

Stage-3 harness. Against each of its three comparison arms, the credit term alone is significant:

  • +1.04 of +1.77 against direct generation;

  • +1.13 of +1.95 against the new stage-15 direct runs;

  • +1.22 of +1.96 against the ablation;

  • +0.66 to +1.09 under Sonnet’s own ratings.

All are p≤.001p \le.001. Its true claims have a lower mean calibrated prior than direct generation’s (.58 against .70), while its precision is higher.

The rest of the gain, +0.73 to +0.82, is cost avoided. Most of that is near misses: claims whose stated direction holds on all three datasets but by less than τ=2\tau= 2 pp. Direct generation pays 0.97 per run for near misses and the harness 0.44. The largest is the worked example of Appendix § 13. Against the ablation, all of the cost avoided is near misses (+0.78 for near misses, −0.04 for other false claims). The cost term is thus partly an artifact of the threshold: at τ=1\tau= 1 and 1.5 pp, where most near misses become true, almost all of the gain is credit, and at τ=3\tau= 3 credit and cost avoided are about equal (Table 20). The credit term is the part of the gain that is surprise in the intended sense.

GPT measure-first. On HPOBench the gain is mostly credit:

  • +0.69 of +0.94 against direct generation;

  • +0.76 against pooled direct generation, where cost avoided is −0.06 and the total +0.70.

On NAS-Bench-201 it is mostly cost avoided (+1.14 of +1.60 against direct generation), and against withheld values the credit term is −0.12. This is consistent with the NB201 gain failing the precision bound: its false claims are many (5.0 per run against 3.1) but have low priors.

Judge-optimized harness. Where it raised RR, half or more of the rise is cost avoided:

  • Sonnet on NB201: +0.43 credit, +0.45 cost avoided;

  • GPT on NB201: −0.38 credit, +1.41 cost avoided.

Its many false claims are ones the panel also doubted, so they cost little. This is the case the precision bound exists for.

The precision bound. The bound of 0.8 times the comparison arm’s precision was first pre-registered in stage 2, before that stage’s runs, and is part of the measurement criteria of stages 4–6 and 8–10, the stage-12 rescoring and the stage-13 harness confirmation; stages 13c, 14 and 15 did not include it (Appendix § 16). It allows a run to give up about one in five true claims in exchange for surprise. Without such a bound, RR can rise when precision collapses (above). Table 19 gives each comparison’s precision ratio and the verdict at bounds from 0.6 to 1.0.

Bench.ComparisonPrec.Prec. (comparison)Ratio0.60.70.80.91.0
NB201Stage-3 harness vs direct0.830.791.05yesyesyesyesyes
vs new direct0.830.741.12yesyesyesyesyes
vs likely-true ablation0.830.821.00yesyesyesyesyes
GPT measure-first vs direct0.500.690.72yes————
vs pooled direct0.500.700.71yesyes———
vs pooled + stage-3 rule0.500.730.68yes————
vs 40-claim call0.500.820.61yes————
vs withheld values0.500.690.72yesyes———
Opus measure-first vs direct0.750.870.86yesyesyes——
HPOGPT measure-first vs direct0.950.901.05yesyesyesyesyes
vs pooled direct0.950.960.98yesyesyesyes—
vs pooled + stage-3 rule0.950.821.15yesyesyesyesyes
vs 40-claim call0.950.970.97yesyesyesyes—
vs withheld values0.950.901.05yesyesyesyesyes
NB201Judge-opt. vs direct, Sonnet0.430.790.54—————
Judge-opt. vs direct, Opus0.280.870.32—————
Judge-opt. vs direct, GPT0.260.690.38—————
HPOJudge-opt. vs direct, Sonnet0.430.810.53—————
Judge-opt. vs direct, Opus0.420.900.47—————
Judge-opt. vs direct, GPT0.350.900.39—————

Table 19. The precision bound at other values (post hoc). Ratio: precision of the first arm over the comparison arm (mean of per-run precision); a column shows whether the bound is met at that multiple. The pre-registered bound is 0.8.

  • Stage-3 harness. It meets every bound up to 1.0 (ratios 1.00 to 1.12).

  • Judge-optimized harness. It fails every bound from 0.6 up (ratios .32 to .54).

  • GPT measure-first on NB201. The ratios are .61 to .72. It would pass against direct, pooled and withheld-value generation at a bound of 0.7, and fails at 0.8.

  • GPT measure-first on HPOBench. The ratios are .97 to 1.15. It passes every bound up to 0.9, and at 1.0 fails against pooled direct generation and the 40-claim call.

The verdicts on the stage-3 harness and the judge-optimized harness therefore do not depend on the bound. The measurement verdicts do: GPT’s NB201 gain would pass at 0.7, Opus’s NB201 comparison with direct generation (ratio .86) would fail at 0.9, and two of GPT’s HPOBench comparisons would fail at 1.0.

Resampling propositions as well as runs (script post_pos_round6a.py, with post_pos_round6c.py and post_pos_round6f.py). The harness’s 240 hypotheses contain 93 distinct propositions, and a few recurring ones carry much of the gain (Appendix § 11). The run-level intervals of the main text treat these propositions as fixed. Table 20 also treats them as a sample. In each of 4,000 draws, the runs of each arm are resampled, the union of the two arms’ distinct propositions (106 to 125) is resampled with replacement, and each run’s RR, credit and cost are recomputed with every proposition weighted by its resampled count. Under the main panel, the gain stays above zero against all three arms at the main threshold (p×=.015p_{\times} = .015, .004, .024) and at every other truth definition except τ=1\tau= 1 against the ablation (p×=.102);atp_{\times} = .102); at\tau = 3$ it is just above zero (p×=.024p_{\times} = .024 to .049). With GPT added to the panel (main threshold) it is p×≤.009p_{\times} \le.009. Against Sonnet’s own blind prior it is not (p×=.062p_{\times} = .062 to .192). The credit term is more fragile than the total: at the main threshold p×=.054p_{\times} = .054 against direct generation and .028 and .045 against the other arms, and it is not significant against Sonnet’s own prior or at τ=3\tau= 3; at τ=1\tau= 1, 1.5, and the two- and mean-agreement rules it is (p×≤.013p_{\times} \le.013). The positive result that survives this analysis is that the selection step raises RR relative to the Claude panels on propositions beyond those sampled; how much of that is credit for true low-prior claims, rather than fewer expected errors, depends on the threshold and the panel.

Panel dependence of the measurement results in detail (post hoc)

This appendix gives the post hoc panel analyses summarized in § 4.3. All of it is hypothesis-generating: the panels share members, the AUC boundary below was read off the data, and panel quality is confounded with the presence of the other frontier model.

Panel strength and panel membership. Leaving the generator out of its own panel gave the two Claude generators panels of very different quality. Sonnet was scored by Haiku and Opus, whose prior predicts truth on NAS-Bench-201 with AUC .87, well above Sonnet’s own .71; Opus was scored by Haiku and Sonnet (.74), well below its own .88. This matters because, relative to a panel, a true claim can be surprising for two reasons: the generator knew something the panel did not, or it learned from evidence. Direct generation can earn RR only in the first way.

Post hoc, we therefore rescored each generator under every panel drawn from the other three models, seven per generator and benchmark (Figure 2; Table 28). Two properties of RR shape how these results should be read. First, as a panel approaches truth, every arm’s RR shrinks toward zero by construction. Direct generation’s RR does fall as the panel’s AUC rises, in all six generator–benchmark combinations (Spearman ρ\rho from −.68 to −.89); for Opus on NAS-Bench-201 it falls from +5.15 per run under Haiku to +1.03 under GPT.

Gen.PanelAUCRR directΔR\Delta RppGen.PanelAUCRR directΔR\Delta Rpp
\multicolumn{6}{c}{NB201}
SonnetH.67+3.74−1.971OpusH+S+G.90+1.36+0.90.008
SonnetH+O.87+1.10−0.57.954OpusG.91+1.03+1.25<.001
SonnetO.88+1.61−0.90.995OpusS+G.91+1.46+0.84.015
SonnetH+G.90+0.72−0.30.823GPTH.67+3.55−0.56.909
SonnetG.91+0.57−0.17.693GPTS.71+3.25−0.12.613
SonnetH+O+G.92+0.40−0.10.630GPTH+S.74+2.30+0.30.222
SonnetO+G.92+0.61−0.23.762GPTH+S+O.87+0.03+1.60<.001
OpusH.67+5.15−0.31.769GPTH+O.87−0.04+1.43<.001
OpusS.71+4.59−0.21.696GPTO.88−0.02+1.53<.001
OpusH+S.74+3.71+0.19.332GPTS+O.88+0.10+1.71<.001
OpusH+G.90+1.50+0.91.007
\multicolumn{6}{c}{HPO}
SonnetH.69+4.51−0.52.958OpusS+G.94+1.00+0.60.006
SonnetH+O.91+0.93+0.11.307OpusH+G.94+1.76+0.34.089
SonnetO.93+0.58−0.00.508OpusG.95+1.42+0.40.038
SonnetH+G.94+1.15−0.07.625GPTH.69+4.29+0.42.040
SonnetG.95+1.00−0.15.774GPTS.72+2.79+0.86<.001
SonnetH+O+G.95+0.38+0.05.399GPTH+S.74+2.69+0.74.001
SonnetO+G.96+0.23+0.07.341GPTH+S+O.91+1.18+0.94<.001
OpusH.69+5.29+0.04.439GPTS+O.91+0.82+1.01<.001
OpusS.72+2.89+0.54.015GPTH+O.91+1.28+0.72<.001
OpusH+S.74+2.97+0.42.065GPTO.93+0.59+0.78<.001
OpusH+S+G.92+1.28+0.52.017

Table 28. Measure-first minus direct generation on RR under every panel drawn from the other three models (post hoc; 24 runs per arm; one-sided permutation pp, uncorrected). H/S/O/G: Haiku/Sonnet/Opus/GPT.

Second, differences between arms shrink with it, so their correlation with AUC carries little information: Sonnet’s gain on NAS-Bench-201 is negative under all seven panels, yet p=+.86p = +.86. What matters is the sign of the difference. Below, a gain means one-sided p<.05p < .05, uncorrected.

On NAS-Bench-201, measurement raises RR significantly for Opus and GPT under all 8 panels that include the other frontier model (AUC ≥.87\geq.87; +0.84+0.84 to +1.71+1.71, p≤.015p \leq.015) and under none of the 6 that do not (AUC ≤.74\leq.74; −0.56-0.56 to +0.30+0.30). No panel falls between these values. Among these panels, therefore, panel quality cannot be separated from the presence of the other frontier model, and the split is one contrast per generator, not 14 independent ones.

A generator’s own blind ratings partly separate the two explanations: for Opus and GPT they form a strong panel (AUC .88–.95) that contains no other frontier model. They capture the generator’s knowledge only in part, since direct generation still earns positive RR against them in five of six generator–benchmark pairs (Sonnet +3.69 and +2.31, Opus +1.93 and +0.85, GPT +0.80 on HPObench; GPT on NAS-Bench-201 −1.34). Against these ratings (Table 29), measurement raises RR for Opus (+0.91 on NAS-Bench-201, +0.48 on HPObench) and for GPT (+2.69, +0.61; Holm p<.025p < .025 over six pairs). This favors panel quality over panel membership as the explanation. With a prior-only calibration, the Opus gain on NAS-Bench-201 weakens to +0.67 (p=.03p = .03, Holm .09). Sonnet gains under no panel, including its own (AUC .71–.72). On HPObench the panel matters little: GPT gains under all seven panels (+0.42 to +1.01), Opus under four and Sonnet under none. Because the panels share members and the boundary was read off the data, this analysis explains the pre-registered results; it does not test a new prediction.

Bench.Gen.Full calibrationPrior-only calibration
ΔR\Delta RppHolmΔR\Delta RppHolm
NB201Sonnet−2.0511−1.9011
Opus+0.91.007.021+0.67.029.088
GPT+2.69<.001<.001+2.46<.001<.001
HPOSonnet+0.25.115.230−0.30.8971
Opus+0.48.004.017+0.47.009.038
GPT+0.61.002.008+0.61.001.007

Table 29. Measure-first minus direct generation on RR when each generator is scored against its own blind three-way prior (post hoc; 24 runs per arm; one-sided permutation pp; Holm over the six pairs).

Panel size or panel composition? Leave-one-out panels have two members for Claude generators and three for GPT, and averaging more members changes the spread of the mean prior. Grouping every panel by size (post hoc; Table 17) shows that size does not drive the measurement results. Among two-member Claude panels scoring GPT on NAS-Bench-201, Haiku and Sonnet give +0.30 (p=.23p = .23), while Haiku and Opus give +1.43 and Sonnet and Opus +1.71 (p<.001p < .001); Opus alone gives +1.53. For Opus, Haiku and Sonnet gain +0.19, while GPT alone gives +1.25 and either Claude model with GPT +0.84 to +0.91. The stage-3 harness gains under every panel of every size (+1.26 to +1.82). For Opus on NAS-Bench-201, the measurement gain is also calibration-dependent: under the panel with GPT and under Opus’s own ratings it is significant in 11 of 14 calibrations but not with a ridge penalty of 10 or 100 or a calibration fitted on the arms’ own claims (+0.56, −0.03 and +0.03 with GPT in the panel, from the released sensitivity log; +0.58, +0.04 and +0.02 under Opus’s own ratings, Table 15).

Evidence or tokens? Measurement costs several times the tokens of direct generation (Table 26), so a gain could reflect the extra tokens rather than the evidence. For Opus on NAS-Bench-201, the arms of stages 6–8 separate the two. They reuse the same 100 designed probes and repeat only the proposal call. That call costs about as many tokens with shuffled values (18.2k) as with exact values (17.6k), and fewer with values withheld (14.8k); the design call adds 15.1k to every arm. The pre-registered stage-7 test of these controls, under the original metric, was inconclusive. Post hoc, under RR, exact values beat withheld values by 0.92 per run and shuffled values by 1.84 (p=.005p = .005 and p<.001p < .001). This holds even under the Claude panel, where no variant beats direct generation (largest difference +0.19), and the margins are larger under the other panels. Under every panel that predicts truth about as well as Opus, RR falls as the evidence gets worse (Table 21): exact values gain most over direct generation (+0.90 to +1.25, p≤.009p \le.009), simulated noise and the 12-epoch proxy gain less, and withheld values, shuffled values and the design-only harness lose (−0.30-0.30 to −1.02-1.02). For Opus on NAS-Bench-201, then, the same probes with uninformative values, at similar or lower cost, do not produce the gain.

Further checks (post hoc)

The checks in this appendix use cached runs, priors and tables only (no model calls). They are post hoc.

What RR sums over, and RR per distinct claim. RR, precision and sv3 are computed over a run’s distinct parseable claims: duplicates within a run are counted once, and on HPOBench the two wordings of the same proposition (§ 3) are merged first. On NAS-Bench-201 nearly every run holds 10 distinct claims, so RR and RR per claim tell the same story. On HPOBench runs hold fewer, and the number differs between arms, which can move a sum such as RR toward zero for an arm with more duplicates. Table 24 therefore repeats the main comparisons with R/nR/n, where nn is the run’s number of distinct claims. The stage-3 harness’s gains are unchanged (all three p<.001p < .001, under the leave-one-out panel and under Sonnet’s own ratings). GPT’s measurement gains on NAS-Bench-201 are unchanged as well. On HPOBench two contrasts weaken. Against the pooled direct baseline with the stage-3 rule, the gain in RR (+0.94, Table 4) is not significant per claim (+0.055, p=.078p = .078; under GPT’s own ratings +0.015, p=.34p = .34), because that arm holds only 5.5 distinct claims per run against 7.8 for measurement. Against the 40-claim call, the pre-registered harness’s null is also null per claim (−0.005-0.005, p=.44p = .44). The judge-optimized harness’s per-claim results have the same signs as its RR results.

The judge-optimized harness within claim type. The judge-optimized harness avoids the two claim types that are most often true, so part of its precision loss follows from base rates. Table 25 separates the two. On HPOBench, every claim it proposes is of the Other type, and its precision within that type (.35–.43) is far below direct generation’s Other-type precision (.80–.96; 90% bootstrap intervals of the difference [−0.42-0.42, −0.32-0.32], [−0.58-0.58, −0.49-0.49] and [−0.55-0.55, −0.39-0.39] for Sonnet, Opus and GPT). The type mix alone would predict a precision of .80–.96, so on HPOBench the drop is not a type-shift artifact: the harness’s claims are worse within type. On NAS-Bench-201 the two cannot be separated. There direct generation proposes almost no Other-type claims (8, 6 and 0 over 24 runs), so its within-type precision for the type that makes up 74–89% of the judge-optimized harness’s claims is not estimated. Within the one type both arms use often, claims that set another edge to none, the judge-optimized harness is less precise for Sonnet and Opus (.49 against .65; .52 against .89) and similar for GPT (.62 against .66).

Probe values and truth on NAS-Bench-201. On NAS-Bench-201 the probes report test accuracy and truth uses validation accuracy. The two splits give almost the same numbers. Over all 375,000 (parent, edit) pairs, the change in test accuracy correlates with the change in validation accuracy at r=.999r = .999, .999.999 and .997.997 (CIFAR-10, CIFAR-100, ImageNet16-120). Their signs agree in 98.5–98.9% of pairs whose validation change is at least 0.5 pp, and they differ by 0.33–0.45 pp on average. On the probes the generators designed, rr is .999.999 (Sonnet), 1.000 (Opus) and .997.997 (GPT). Truth computed on the test split would label 99.8% of the 1,740 claims the same way (198 true claims under each split, 196 in common). The separate split therefore gives almost no independence: on NAS-Bench-201, a probe is close to an exact sample of the quantity that truth averages, and a measure-first arm can in principle read truth off 100 well-chosen probes. The HPOBench probes are noisier (a single training seed, sign agreement 70% with the 5-seed truth when the true difference is at least 0.5 pp; § 3).

Model versions. Claude models were called as claude-haiku-5-5, claude-sonnet-5-5 and claude-opus-5-5 through the Claude Code CLI, each call in its own temporary directory. The CLI’s session logs record the model that answered each call. Every logged session on every study date reports exactly the requested identifier and no other (Table 22). The counts are sessions logged from temporary directories on each date. They include calls from other work (all of 2026-10-20, see the pre-registration paragraph below) and exclude failed calls without a reply, so they do not match the paper’s call counts exactly. GPT was called with the explicit model identifier gpt-6-astra (codex-cli 0.162.0, codex exec –ephemeral); ephemeral sessions keep no log, so we cannot audit the GPT version per call beyond the identifier we requested.

Which pre-registered criteria include the precision bound. The pre-registration documents are written in Chinese; we translate the decision criteria. Stage 10 (GPT, measurement against direct generation, each benchmark): “M: supported: A-100’s sv is above BASE, one-sided permutation test, Holm-corrected together with A-SV, p<.05p < .05, and precision ≥0.8×\geq0.8 \times BASE; refuted: the upper bound of the difference’s 90% CI <0.5< 0.5; otherwise inconclusive.” Stages 2, 4, 5, 6, 8, 9 and 12 state the same precision condition for their measurement or harness tests. Stage 3 has near-against direct generation: “supported: p<.05p < .05, and precision ≥0.8×\geq0.8 \times BASE; refuted: the upper bound of the 90% CI of ΔR<0.5\Delta R < 0.5; otherwise inconclusive.” Stage 13c (GPT, measurement against pooled direct generation): “supported (the gain from measurement is not only more tokens): Holm p<.05p < .05; refuted (tokens suffice to explain it): the upper bound of the 90% CI of A−D4<0.5A - D4 < 0.5; otherwise inconclusive.” Stage 14a (pooled with the stage-3 rule) and stage 15c (the 40-claim call) use criteria “of the same form as 13c”; stage 14b (the likely-true ablation), stage 15a (new direct runs) and stage 15b (shuffled and withheld values) state p<.05p < .05 (Holm-corrected where there are two tests) and the same refutation rule. None of stages 13c, 14 or 15 includes a precision condition, so their verdicts in Table 11 follow from pp alone and are as pre-registered. Applying the bound to them is our own post hoc reading: for GPT on NAS-Bench-201, measure-first precision (.50) is .61 to .72 times that of every same-cost arm (pooled direct .70, pooled with the stage-3 rule .73, 40-claim call .82, withheld values .69), so those gains are supported as pre-registered but do not count as improvements under the bound (Table 4). On HPOBench the ratios are .97 to 1.63. The stage-3 harness’s precision is 1.05, 1.12 and 1.00 times that of its three comparison arms.

A calibration with all three claim types. The default calibration separates self-conditions from all other claims but not the two other types, claims whose condition removes an edge and the remaining Other claims, whose base rates differ (.173 and .067 on NAS-Bench-201). The +Type variant of Table 14 adds operation indicators, which are not these types. We therefore added an indicator for removes-an-edge claims and its interaction with the prior’s logit to the default covariates (post hoc; NAS-Bench-201 only, since HPOBench has no such type). The stage-3 harness’s gains barely move: +1.73, +1.90 and +1.89 against direct generation, the new stage-15 direct runs and the ablation (default +1.77, +1.95, +1.96; all p<.001p < .001), and +1.24, +1.54 and +1.16 under Sonnet’s own ratings. GPT’s NAS-Bench-201 measurement gains change by at most 0.22 and stay significant (against direct generation +1.39; pooled +2.05; pooled with the rule +1.13; 40-claim call +0.86; shuffled +1.45; withheld +0.67, p=.021p = .021). Opus’s gain under the Claude panel stays null (+0.18). The judge-optimized harness’s rise in RR for GPT on NAS-Bench-201 shrinks from +1.04 to +0.79.

GPT: same-cost baselines in detail. A pooled direct baseline ran direct generation four times and asked GPT to keep the 10 pooled claims most likely to be true, at about the cost of measurement. Measurement beat it on both benchmarks. Pooling and selecting changed RR by −0.64 and +0.24 relative to a single direct run, at or above direct precision (.70 against .69, .96 against .90). Applying the stage-3 harness’s selection rule to the same pools did not close the gap either. Four pooled direct runs are redundant, however: their 40 hypotheses hold only 21.3 distinct claims on NAS-Bench-201 and 14.0 on HPOBench (post hoc). A single call that writes 40 claims and keeps the 10 most likely to be true is less redundant (36.8 and 25.3 distinct claims) and raises RR over a single direct run (+0.73, p=.014p = .014; +0.85, p<.001p < .001) at about 11k tokens per run. On NAS-Bench-201 measurement still beat it. On HPOBench it did not: there a single 40-claim call matched measurement at about two thirds of its tokens (11.0k against 17.4k), with precision .97 against .95. The same-budget advantage of measurement for GPT therefore holds on NAS-Bench-201 but not on HPOBench. Post hoc, against GPT’s own blind ratings, the pooled baseline gained nothing over direct generation (−0.28, −0.10) while measurement did (+2.69, +0.61). On NAS-Bench-201 the 96 pooled direct runs contain no claims of the non-structural “Other” type (0 of 954), which make up 60% of GPT’s measure-first hypotheses, so no selection from these pools can reach them. On NAS-Bench-201, GPT’s direct claims hold as often as the calibrated Claude panel expects (.69 against .69), while its measure-first claims were expected to hold at .33 and hold at .50, mostly non-structural interactions that the panel rates unlikely. Per distinct claim (Table 24), every NAS-Bench-201 contrast of Table 4 keeps p≤.005p \le.005, and so do the HPOBench contrasts with direct generation, pooled direct generation, and shuffled and withheld values; the refuted contrast with the 40-claim call stays null (−0.005).

Stages and pre-registration documents. The study has 17 pre-registered stages in 16 documents: stages 1 to 15 and stage 12b each have their own document, and stage 4b (the Opus confirmation) is an addendum to the stage-4 document, written before its calls. Table 11 lists each stage’s criteria and verdicts. Table 23 lists the documents with their SHA-256 hashes and file times. The documents themselves are in the supplementary material and at https://osf.io/c2hkg/?view_only=d05d8b1acd504e6ab87c242522b769df, so the hashes can be checked against the text we cite. The times come from the local filesystem and are not attested by a third party; a deposit made now cannot prove when a document was written, and the table is offered as a record, not as proof. Only one document changed after it was created: PREREG_s4.md, 15 minutes later, when the stage-4b addendum was added. In the study’s directory the only files older than the first document are the claim-space code and one literature-search cache entry. In the CLI session logs, the first session whose prompt poses a claim about either benchmark starts at 01:05 UTC on 2026-10-21, nine minutes after PREREG_s1.md; all 1,504 earlier sessions in Table 22 (every 2026-10-20 session and 114 on 2026-10-21) predate that document. A keyword search finds neither benchmark in them, except two sessions that ask for a review of an unrelated manuscript whose text mentions NAS-Bench-201. They are exploratory pilots on other claim spaces (with their own pre-registrations) and unrelated work, and none is used as evidence here. Dates written inside the documents are in local time (UTC−7), up to a day behind the UTC times used here; for example, PREREG_s15.md dates the stage-5 direct runs, made after 06:04 UTC on 2026-10-21, to “10-20”.

Additional tables

Table 26–Table 29 are generated from the cached analysis outputs by a script released with the paper.

References

  1. [1]Fariya Afrin and Ibne Farabi Shihab. Evaluator Ensembles Under Reward Hacking: Covariance Geometry and Finite-Search Guarantees, 2026. URL http://arxiv.org/abs/2608.08002v1.
  2. [2]Dhruv Agarwal, Bodhisattwa Prasad Majumder, Reece Adamson, Megha Chakravorty, Satvika Reddy Gavired d y, Aditya Parashar, Harshit Surana, Bhavana Dalvi Mishra, Andrew McCallum, Ashish Sabharwal, and Peter Clark. AutoDiscovery: Open-ended Scientific Discovery via Bayesian Surprise. In Advances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2507.00310.arxiv.org/abs/2507.00310
  3. [3]Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning, 2025. URL http://arxiv.org/abs/2507.19457v2.
  4. [4]Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models, 2024. URL http://arxiv.org/abs/2404.07738v2.
  5. [5]Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward Model Ensembles Help Mitigate Overoptimization. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.02743
  6. [6]Stefano DellaVigna, Devin Pope, and Eva Vivalt. Predict Science to Improve Science. Science, 366(6464): 428–429, 2019.DOI
  7. [7]Xuanyi Dong and Yi Yang. NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search. In International Conference on Learning Representations (ICLR), 2020.arxiv.org/abs/2001.00326
  8. [8]Yaxin Du, Xiyuan Yang, Zhifan Zhou, Yujie Ge, Cheng Wang, Jiajun Wang, Sijie Chen, Zehui Liu, Yuxin Zhang, Weicheng Gu, Julian Zhang, Zixing Lei, and Siheng Chen. RSI-Master: Structuring Experiments to Guide Autonomous Model Improvement, 2026. URL http://arxiv.org/abs/2609.35561.
  9. [9]Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving Factuality and Reasoning in Language Models through Multiagent Debate. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024.arxiv.org/abs/2305.14325
  10. [10]Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik, Matthias Feurer, René Sass, Aaron Klein, Noor Awad, Marius Lindauer, and Frank Hutter. HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2021.arxiv.org/abs/2109.06716
  11. [11]Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and Improving Consistency in Pretrained Language Models. Transactions of the Association for Computational Linguistics, 9:1012–1031, 2021.
  12. [12]Benjamin Feuer, Micah Goldblum, Teresa Datta, Sanjana Nambiar, Raz Besalel, Samuel Dooley, Max Cembalest, and John P. Dickerson. Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking. In International Conference on Learning Representations, 2025.arxiv.org/abs/2409.15268
  13. [13]Craig R. Fox and Robert T. Clemen. Subjective Probability Assessment in Decision Analysis: Partition Dependence and Bias toward the Ignorance Prior. Management Science, 51(9):1417–1432, 2005.DOI
  14. [14]Kanishk Gandhi, Michael Y. Li, Lyle Goodyear, Agam Bhatia, Louise Li, Aditi Bhaskar, Mohammed Zaman, and Noah D. Goodman. BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery, 2025. URL http://arxiv.org/abs/2501.01540v2.
  15. [15]Leo Gao, John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimization. In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023.arxiv.org/abs/2210.10760
  16. [16]Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Petar Sirkovic, Artiom Myaskovsky, Gregorz Glowaty, Felix Weissenberger, Alessio Orlandi, Dan Popovici, Anil Palepu, Keran Rong, Ryutaro Tanno, Khaled Saab, Fan Zhang, Jacob Blum, Andrew Carroll, Kavita Kulkarni, Nenad Tomasev, Dina Zverinski, Ivor Rendulic, Elahe Vedadi, Florian Hasler, Luka Rimanic, Marina Boia, Ivan Budiselic, Ben Feinstein, Mathias Bellaiche, Tom Sheffer, Jan Freyberg, Jeremy Ratcliff, Ottavia Bertolli, Katherine Chou, Avinatan Hassidim, Burak Gokturk, Amin Vahdat, Yuan Guan, Vikram Dhillon, Eeshit Dhaval Vaishnav, Byron Lee, Tiago R D Costa, José R Penadés, Gary Peltz, Yossi Matias, James Manyika, Demis Hassabis, Yunhan Xu, Pushmeet Kohli, Annalisa Pawlosky, Alan Karthikesalingam, and Vivek Natarajan. Accelerating scientific discovery with Co-Scientist, 2025. URL http://arxiv.org/abs/2502.18864v2.
  17. [17]L. Hewitt, A. Ashokkumar, I. Ghezae, and R. Willer. Predicting Results of Social Science Experiments Using Large Language Models, 2024. Working paper, Stanford University.
  18. [18]Shengran Hu, Cong Lu, and Jeff Clune. Automated Design of Agentic Systems. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2408.08435
  19. [19]Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large Language Models Cannot Self-Correct Reasoning Yet. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.01798
  20. [20]Kexin Huang, Ying Jin, Ryan Li, Michael Y. Li, Emmanuel Candès, and Jure Leskovec. Automated Hypothesis Validation with Agentic Sequential Falsifications, 2025. URL http://arxiv.org/abs/2502.09858v1.
  21. [21]Laurent Itti and Pierre Baldi. Bayesian Surprise Attracts Human Attention. Vision Research, 49(10): 1295–1306, 2009. doi: 10.1016/j.visres.2008.09.007.DOI
  22. [22]Abigail Z. Jacobs and Hanna Wallach. Measurement and Fairness. In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT), 2021.DOI
  23. [23]Peter Jansen, Marc-Alexandre Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Oyvind Tafjord, and Peter Clark. DiscoveryWorld: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), 2024.
  24. [24]Fengxian Ji, Yuke Li, Jingpu Yang, Juanfan Wu, Fan Zhang, Zhexuan Cui, Yu Xie, Min Peng, Qianqian Xie, Xiuying Chen, and Zhuohan Xie. Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation, 2026. URL http://arxiv.org/abs/2608.01666v3.
  25. [25]Bo Jiang. HindSight: Evaluating LLM-Generated Research Ideas via Future Impact, 2026. URL http://arxiv.org/abs/2603.15164v2.
  26. [26]Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixin g Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu. AIIDE: AI-Driven Exploration in the Space of Code, 2025. URL http://arxiv.org/abs/2502.13138v1.
  27. [27]Jacek Karwowski, Oliver Hayman, Xingjian Bai, Klaus Kiendlhofer, Charlie Griffin, and Joar Skalse. Goodhart’s Law in Reinforcement Learning, 2023. URL http://arxiv.org/abs/2310.09114v1.
  28. [28]Philippe Laban, Lidiya Murakhovsk’a, Caiming Xiong, and Chien-Sheng Wu. Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experiment, 2023. URL http://arxiv.org/abs/2311.08596v2.
  29. [29]Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, Deli Zhao, Yu Rong, Tian Feng, and Lidong Bing. Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents, 2024. URL http://arxiv.org/abs/2410.13185v5.
  30. [31]Jiarui Liu, Renjie Tao, Yiwei Liao, Chuanyang Jin, Kai Sun, Xiao Yang, Xinyuan Zhang, Xilun Chen, Zhuangqun Huang, Lechen Zhang, Yongjin Yang, Yinghui He, Weihao Xuan, Rakesh Wanga, Anuj Kumar, Mona T. Diab, Wen-tau Yih, and Xin Luna Dong. IdeaScientist: Orchestrating Agents for Grounded Scientific Ideation, 2026. URL http://arxiv.org/abs/2610.04074v2.
  31. [32]Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery, 2024. URL http://arxiv.org/abs/2408.06292v3.
  32. [33]Xiaoliang Luo et al. Large Language Models Surpass Human Experts in Predicting Neuroscience Results. Nature Human Behaviour, 9(2):305–315, 2025a. doi: 10.1038/s41562-024-02046-9.DOI
  33. [34]Ziming Luo, Atoosa Kasirzadeh, and Nihar B. Shah. The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systems, 2025b. URL http://arxiv.org/abs/2509.08713v2.
  34. [35]Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-Refine: Iterative Refinement with Self-Feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2303.17651
  35. [37]David Manheim and Scott Garrabrant. Categorizing Variants of Goodhart’s Law, 2018. URL http://arxiv.org/abs/1803.04585v4.
  36. [38]Brian A. Nosek, Charles R. Ebersole, Alexander C. DeHaven, and David T. Mellor. The Preregistration Revolution. Proceedings of the National Academy of Sciences, 115(11):2600–2606, 2018.
  37. [39]Daniel Paleka, Abhimanyu Pallavi Sudhir, Alejandro Alvarez, Vineeth Bhat, Adam Shen, Evan Wang, and Florian Tramèr. Consistency Checks for Language Model Forecasters. In International Conference on Learning Representations, 2025.arxiv.org/abs/2412.18544
  38. [40]Wenbo Pan, Shujie Liu, Chin-Yew Lin, Jingying Zeng, Xianfeng Tang, Xiangyang Zhou, Yan Lu, and Xiaohua Jia. Evolving Agents in the Dark: Retrospective Harness Optimization via Self-Preference, 2026. URL http://arxiv.org/abs/2606.05922v3.
  39. [41]Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM Evaluators Recognize and Favor Their Own Generations. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
  40. [42]Feng Shi and James Evans. Surprising combinations of research contents and contexts are related to impact and emerge with scientific outsiders from distant disciplines. Nature Communications, 14:1641, 2023.DOI
  41. [43]Chenglei Si, Tatsunori Hashimoto, and Diyi Yang. The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas, 2025a. URL http://arxiv.org/abs/2506.20803v1.
  42. [44]Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers. In International Conference on Learning Representations (ICLR), 2025b.
  43. [45]Chenglei Si, Zitong Yang, Yejin Choi, Emmanuel Candès, Diyi Yang, and Tatsunori Hashimoto. Towards Execution-Grounded Automated AI Research, 2026. URL http://arxiv.org/abs/2601.14525v1.
  44. [46]Soumitra Sinhahajari, Navonil Majumder, and Soujanya Poria. On the Limits of LLM-as-Judge for Scientific Novelty Assessment, 2026. URL http://arxiv.org/abs/2606.12071v1.
  45. [47]Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and Characterizing Reward Hacking, 2022. URL http://arxiv.org/abs/2209.13085v2.
  46. [48]Dhruv Srikanth, Bingchen Zhao, Dixing Xu, Yuxiang Wu, and Zhengyao Jiang. Recursive self-improvement of AI research agents, 2026. URL http://arxiv.org/abs/2609.26457v1.
  47. [49]Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5433–5442, 2023.
  48. [50]Amos Tversky and Derek J. Koehler. Support Theory: A Nonextensional Representation of Subjective Probability. Psychological Review, 101(4):547–567, 1994.
  49. [51]Pat Verga, Sebastian Hofstätter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models, 2024. URL http://arxiv.org/abs/2404.18796.
  50. [52]Renxiong Wang, Darwin Yi, Abril Herrlein, Anas Mahmoud, Advait Gosai, Lisiman Hua, MohammadHossein Rezaei, Xingang Guo, Anisha Gunjal, Utkarsh Tyagi, David J. Lee, Minglai Yang, Haris Riaz, Chenguang Wang, Huaxiu Yao, Daniel Yue Zhang, Aakash Sabharwal, Tong Zhao, and Yunzhong He. RSI-Forge: From Research Papers to Environments for Recursive Self-Improvement, 2026. URL http://arxiv.org/abs/2610.09426.
  51. [53]Jiaxin Wen, Chenglei Si, Yueh-han Chen, He He, and Shi Feng. Predicting Empirical AI Research Outcomes with Language Models, 2025a. URL http://arxiv.org/abs/2506.00794v1.
  52. [54]Jiaxin Wen, Ruiqi Zhong, Akbir Khan, Ethan Perez, Jacob Steinhardt, Minlie Huang, Samuel R. Bowman, He He, and Shi Feng. Language Models Learn to Mislead Humans via RLHF. In International Conference on Learning Representations, 2025b.arxiv.org/abs/2409.12822
  53. [55]Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hoo. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. In International Conference on Learning Representations, 2024.arxiv.org/abs/2306.13063
  54. [56]Ruiyao Xu, Yanfei Chen, Zhongying CuiZhu, Bhavana Dalvi Mishra, Yifei Ming, Han Yu, Rujun Han, Chen-Yu Lee, and Tomas Pfister. Harness²: Recursive Agent Harnessing for an Open World, 2026. URL http://arxiv.org/abs/2610.14224v1.
  55. [57]Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search, 2025. URL http://arxiv.org/abs/2504.08066v1.
  56. [58]Chenyu Zhou. More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges, 2026. URL http://arxiv.org/abs/2607.05904v1.
  57. [59]Jian-Qiao Zhu and Thomas L. Griffiths. Incoherent Probability Judgments in Large Language Models. In Proceedings of the Annual Meeting of the Cognitive Science Society (CogSci), 2024. arXiv:2401.16646.arxiv.org/abs/2401.16646

Paper details

vlnc:W_r3m8nxfp
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents