betaFeedback

WorksComputer Science

Fidelity Regret: Measuring Simulator Misspecification in Verifiable Design

Abstract

RLVF assumes the verifier evaluates the quantity of interest. In genetic-circuit design, a topological verifier can establish that a program is a valid circuit computing the target function, whereas ranking its physical performance requires a simulator. We measure fidelity regret, the reference simple regret incurred when a cheap simulator selects the design, over a 21,362-design combination subspace spanning 5 measured gate libraries and 3 organisms. Exact ordered-assignment checks retain positive regret in every tested cell while shifting magnitudes in both directions. Mean normalized regret is 0.369 over 72 cells and only 6 have zero regret. The discrepancy depends strongly on the reference axis: the yeast library SC1C1G1T1 has mean regret 0.113 for cytometry quality and 0.801 for growth burden, and every evaluated library–circuit-size pair has positive burden regret. Topological verification ties all valid gate assignments, while an oracle whitelist subject to retaining at least 25% of the design space leaves positive regret in 30 of 52 cells. Design-space optimization and feedback-guided search with frozen language models reach the high-regret regions; on a matched zero-regret control, additional optimization instead reduces regret. At a matched modeled-cost budget, mitigation is budget- and library-dependent. An in-sample fixed rule spending α=0.34α{=}0.34 of the budget re-scoring top cheap-simulator designs achieves mean excess regret 0.016, compared with 0.305 for cheap-only and 0.034 for reference-only selection, improving on cheap-only in 330 of 360 cells. More broadly, any workflow optimized through a simulator or digital twin can satisfy its verifier yet perform poorly on the intended physical objective. Fidelity regret provides a transferable template for detecting this gap before optimization and comparing budget-aware remedies across scientific domains.

Introduction

Reinforcement learning with verifiable rewards (RLVR) replaces a learned reward with a checker that directly evaluates task completion [13]. This connection between reward and task objective holds in settings such as theorem proving and unit-tested code. In scientific design, a checker may certify only a necessary condition, leaving a simulator to evaluate the physical quantity being optimized.

Genetic-circuit design exposes this separation. Using pysbol3 [18], a verifier can validate source syntax, SBOL structure, requested topology, and target truth table. [4] trains language models with a multiplicative five-level verifier; its topology tasks use symbolic truth-table propagation, while its separate T8 assignment evaluation adds Hill-response simulation. Here, topology-only verification gives every valid measured-gate assignment the same score, despite cell-dependent physical performance. Simulators must therefore rank valid designs, at the cost of repeated evaluations during optimization.

Before optimization, we measure fixed reward misspecification exactly over enumerable cells. Fidelity regret is the reference simple regret of the cheap simulator’s argmax; enumerating each (library, topology, cheap/reference-simulator pair) links it to multi-fidelity screening regret [21, 14]. Across 5 Cello UCF libraries, regret varies by axis; an oracle bound shows the evaluated per-gate-admissibility verifier class cannot eliminate it across all cells. Design-space and frozen-model optimizers reach high-regret regions; 1,280 matched-budget runs show mitigation depends on reference axis, cost, and library, motivating fixed-budget re-scoring.

Fidelity ladder and fidelity regret

The ladder. Five simulator levels score the same design using parameters from the measured Cello UCF tables [20, 9]: L0, topology only; L1, mean-field Hill steady state; L2, single-cell cytometry quality (with L2 d′d' for separation); L3, growth burden; and L4, response kinetics. The in-optimization cost model used for every budget comparison prices one L2 evaluation at 67.4 L1 evaluations and one L3 evaluation at 2.2. Appendix D reports the separate standalone timings and reconciles the two measurements.

The levels score distinct physical properties, so a gap combines a different objective with a more faithful model of the same objective. The falsification control in Section 4 isolates the latter. L2 d′d' measures ON/OFF separation irrespective of polarity (Appendix § 16).

The quantity. For a design space D\mathcal{D}, let dcheap∈arg⁡max⁡d∈Dscheap(d)d_{\mathrm{cheap}} \in\arg\max_{d \in\mathcal{D}} s_{\mathrm{cheap}}(d) denote the cheap simulator’s selected optimum. Fidelity regret is

Reg⁡=max⁡d∈Dsref(d)−sref(dcheap),(1)\operatorname{Reg} = \max_{d \in\mathcal{D}} s_{\mathrm{ref}}(d) - s_{\mathrm{ref}}(d_{\mathrm{cheap}}), \tag*{(1)}

which measures the reference quality forfeited by the cheap selection. The reported value uses the deterministic argmax returned by the enumeration implementation, and the tables retain the number of tied optima in every cell. Reported regret is conditional on this recorded selection when L1 has multiple optima; the fully tied L0 case is treated separately below. We normalize regret by the reference range over D\mathcal{D}, so 0 identifies a reference optimum and 1 the worst enumerated design. We also report the percentile and absolute value because normalization can magnify narrow reference ranges: Bth1C1G1T1’s worst L4 cell is 0.870 normalized and 0.085 decades absolute. The primary enumeration covers a combination subspace: an unordered choice of kk distinct gates evaluated once in the implementation’s fixed slot order. It contains 21,362 designs; the corresponding ordered-assignment space has k!k! times as many designs in each kk-specific cell and totals 432,372 designs. Exact ordered-assignment checks retain positive regret in all eight tested cells, but shift magnitudes in both directions (mean absolute change 0.273). The reported statistics are exact for the combination subspace and indicative, not conservative, for the ordered space (Appendix § 25). The reference composes per-gate wet-lab measurements through the fidelity ladder; it is not a whole-circuit wet-lab measurement (§7).

Axis-dependent fidelity regret

Across the 72 enumerated single-axis cells with L1 as the reward, mean normalized regret is 0.369. Twenty-five cells (34.7%) have regret at least 0.5, and 6 have zero regret. Median normalized regret is 0.195 for cytometry quality, 0.241 for cytometry separation, and 0.467 for growth burden (Figure 1a). The corresponding median percentiles of the L1 selection are 0.926, 0.938, and 0.549 (Figure 1b); a uniform selection has median percentile 0.5. The worst cell, SC1C1T1 on k3 against L3, has combination-space normalized regret 0.9994 (0.45th percentile) and is also the most ordering-sensitive, falling to 0.123 under exact ordered assignment; we treat it as an existence proof of severe regret rather than a transferable magnitude (Appendix § 25).

Normalized fidelity regret and percentile of chosen design across cytometry quality, cytometry separation, response kinetics, and growth burden

Figure 1. Exact fidelity regret with L1 as the reward. (a) Normalized regret for each (library, kk, topology) cell; the dashed line is the L0 uniform-tie-break expectation, 0.541. (b) Percentile of the L1-selected design under each reference. Panels c–d appear in Fig. 3.

Regret depends on the validation axis. Averaged over kk, SC1C1G1T1 has mean normalized regret 0.113 against cytometry quality and 0.801 against growth burden. The pattern extends across libraries: all ten (library, kk) burden cells have positive regret. Under the pre-specified operational rule in Appendix § 17, seven are UNSAFE, with worst regret above 0.35, and three are CAUTION. The mean-field simulator does not represent metabolic load and selects gates that the burden reference penalizes. (1) therefore provides a pre-run screen for the specific library and physical axis to be optimized.

Optimization and mitigation results

Figure 2. Optimization and mitigation results. Panels (a,b) show regret incurred; panels (c,d) show regret recovered on an axis unbounded below. (a) Design-space search, 30 seeds per arm. (b) Frozen-model search at maximum pressure; (c,d) Eight rewards at a matched modeled-cost 2.0 s pulse, 40 runs per cell. L2 discrepancy cells are marked not affordable.

Topological verification ties the design space. In all 120 supported L0 rows, the number of designs tied at the optimum equals the number of valid designs. Uniform tie breaking can therefore select any reference rank, giving a regret bracket [0,1][0,1] and an expected normalized regret of 0.541. These are properties of the tie distribution rather than achieved optimization results.

The multiplicative verifier hierarchy of Flynn [ [4] requires a valid document and target truth table. Its topology-based functional check cannot rank gate assignments within a fixed topology; its separate T8 evaluation instead introduces a Hill-response simulator. An oracle whitelist gives an exact, non-implementable upper bound on the evaluated per-gate-admissibility verifier class, subject to retaining at least 25% of the design space. Positive regret remains in 30 of 52 cells, including 17 of 20 burden cells, with a maximum of 0.511 (Appendix L). Thus this verifier class cannot eliminate fidelity regret across the evaluated cells.

Optimization pressure reaches the regret region

Exact regret characterizes the reward surface before optimization. We next test whether two empirical optimization settings select the high-regret regions.

Design-space search. We apply REINFORCE with a baseline [31] and (μ+λ)(\mu+\lambda) evolution [2] to gate assignments in 16 arms with 30 seeds per arm, evaluated against L3. The returned designs incur 0.789–3.252 of the achievable regret. All 16 arms have a 95% CI lower bound of at least 0.70 (Appendix J), and 7 have a lower bound of at least 1.0. Continued improvement of the cheap objective can therefore return designs with lower reference scores than the enumerated cheap optimum.

Frozen-model search. Held-fixed public language models propose designs through the same environment, without parameter updates (Appendix I). Under methods that return the cheap score to the model, regret rises from 0.602 [0.485, 0.718] at R=1R=1 to 0.742 [0.647, 0.836] at R=5R=5 for iterative refinement, and from 0.605 at G=1G=1 to 0.711 [0.613, 0.810] at G=4G=4 for evolutionary search. Pure best-of-nn shows no corresponding increase: Haiku 4.5 falls from 0.901 to 0.782, while Opus 4.5 changes from 0.880 to 0.885. The increase is observed only when the cheap score is returned to the model.

Model-tier comparison. Paired over 44 cells at maximum pressure, the Opus 4.5 minus Haiku 4.5 regret difference is +0.053 [+0.001, +0.105]. Opus also achieves a higher cheap reward, with a difference of +0.060 [+0.024, +0.096], and reaches the exact cheap optimum in 86.4% of runs compared with 60.0% for Haiku. On the two cells already near the regret ceiling, the difference is +0.012 [−0.032, +0.056] (n=30n=30), so the aggregate effect is carried by cells with headroom. This direction matches Pan et al. [ [21]'s finding that capable agents attain a misspecified reward more effectively and incur greater true-reward loss [21]. Sonnet 4.5 is a parsing artifact, not an intermediate capability tier: 79.8% of responses reach the output cap before completing a parseable program.

Matched falsification control. On SC1C1G1T1/and_k3, identical proposals and cheap-reward selections are scored against two references. Against L3, where enumerated regret is 0.771, observed regret rises from 0.558 to 0.674 with pressure. Against L2 d′d^\prime, where enumerated regret is zero, it falls from 0.313 to 0.199. The opposite trends under identical selection isolate reference misspecification from a generic effect of optimization pressure (Appendix F).

Mitigations at matched budget

Discrepancy prediction transfers unevenly. A discrepancy model predicts the reference-minus-cheap score. For L2, gradient boosting attains R2R^2 values of 0.520–0.972 on random splits and −1.334–0.699 under shifts toward optimizer-preferred designs. Its median R2R^2 on returned designs is 0.212, and on SC1C1G1T1 its shifted RMSE is 34% higher than using no correction. The L3 burden discrepancy remains more predictable, with median R2=0.740R^2=0.740 on returned designs (Appendix E).

The matched-budget study contains 1,280 runs at a budget of 2.0 modeled cost-equivalent seconds, with 20 seeds and 2 optimizers per cell. The in-loop cost model sets the exchange rates (Appendix D). Reference calls reduce the number of optimization steps available within this budget. We report the fraction of achievable regret recovered (0 matches the cheap optimum, 1 reaches the reference optimum, and negative values score below the cheap optimum; Appendix H).

Discrepancy-reward results. Against L3 on SC1C1G1T1, a passively fitted discrepancy reward (B=64B=64) recovers 0.770 [0.716, 0.816], compared with 0.007 for the unmitigated cheap reward (p<10−4p < 10^{-4}). On Eco1C2G2T2, its recovery remains negative at −0.479 [−0.587, −0.371], although this is an improvement of +0.245 over cheap-only (p=0.023p = 0.023). The active variant is indistinguishable from cheap-only in that cell (p=0.41p = 0.41), and active acquisition does not outperform passive acquisition at equal spend. For L2, the 64-evaluation acquisition costs 4.05 s and exceeds the 2.0 s budget; the 160 affected runs are recorded as budget_infeasible and have no recovery value.

Reference-only results. On SC1C1G1T1 against L2, spending the 2.0 s budget on L2 recovers −2.168 [−2.997, −1.418], compared with −0.720 for the cheap reward (p<10−4p < 10^{-4}). The budget buys about 32 evaluations, and neither optimizer leaves initialization. On Eco1C2G2T2, reference-only recovery is 0.433 compared with 0.155 for cheap-only (p<10−4p < 10^{-4}). Aggregation would obscure this sign reversal, so the results are reported by library.

Risk-averse rewards. Conservative objective models for offline model-based optimization address the model misspecification targeted by these risk-averse arms [30]. These rewards underperform the naive cheap reward in 3 of 4 cells. The largest loss occurs for SC1C1G1T1 against L2, where an 8-member CVaR0.05_{0.05} reward recovers −3.499, compared with −0.720 for the naive reward. Across 608 enumerated CVaR, mean–variance, and Wasserstein-DRO selections, 61 improve regret, 137 worsen it, and 410 are unchanged. Ensemble spread has median absolute Spearman correlation 0.094 with reference shortfall (with nominal L1 cost only 0.008 decades on SC1C1G1T1), and the positive training cell is explained primarily by replacing the point estimate with an ensemble mean (Appendix § 29).

Fidelity curricula. On SC1C1G1T1 against L3, each mixed schedule improves on cheap-only recovery: the 50/50 and 0.25 switch schedules recover 0.783 and 0.890, respectively, compared with 0.007. Reference-only reaches 0.944 and exceeds the mixed schedules by +0.053 to +0.775. Against L2, the mixed schedules do not separate from cheap-only. Reference-only recovery is −2.256; its paired difference against cheap-only is −1.536 (Appendix § 22).

Fixed-budget re-scoring. Rule-T allocates α=0.34\alpha=0.34 of budget to re-score and rerank top cheap designs; the fraction is selected and evaluated in-sample (no library held out). Mean excess regret is 0.016 (cheap-only: 0.305; reference-only: 0.034), within 0.002 of the leave-one-library-out oracle (0.014) and 0.006 of the leave-one-axis-out oracle (0.010). It beats cheap-only in 330/360 cells and matches or improves on reference-only in 198/360 (161 wins, 37 ties; Appendix § 28). Four adaptive methods track the best simple policy within 0.106 at budgets of at least 128 reference calls; inferred fidelity-bias estimates have max ∣ρs∣=0.293|\rho_s|=0.293 with enumerated regret (all p>0.27p>0.27; Appendix § 27).

Prior work studies proxy failures via reward misspecification, hacking, and large-scale RLHF reward-model overoptimization [21, 1, 28, 6]; our capability ladder reproduces proxy gains with lower true reward. Conservative offline model-based objectives motivate our risk-averse arms [30]; RLVR checkers reduce learned-model error but leave physical performance simulator-ranked [13]. Multi-fidelity optimization and discrepancy correction, unreliable sources, and multi-fidelity best-arm identification frame our pre-optimization regret measurement and tests of correction under optimizer-induced distribution shift [12, 5, 10, 11, 17, 24]. Cello/Cello 2.0 supply measured UCF libraries and expanded format; GenCircuit-RL supplies code generation [20, 9, 4]. Appendix § 30 compares multi-fidelity Bayesian optimization, bandits, distributionally robust optimization, and Goodhart-style proxy optimization.

Limitations

Reference scores compose per-gate wet-lab measurements into circuit-level predictions. Without whole-circuit wet-lab validation, composition error is unmeasured, and greater fidelity does not establish ground truth. Appendix § 17 details range dependence and fit-residual-based Hill-parameter uncertainty.

Five published Cello UCF libraries across three organisms are too few for statistical attribution of regret to library properties (Appendix § 19). The published Eco1C2G2T2 cytometry table has a decimal-point error, and Bth1C1G1T1 lacks single-cell cytometry; Appendix § 21 reports correction sensitivity and unsupported-axis handling.

Matched-budget results cover 2/4 cytometry-bearing libraries at a 2.0 s modeled-cost-equivalent budget. Enumeration is exact for combinations, not ordered assignments; the headline allocation fraction is selected in-sample (Appendices § 25 and § 28). Our LM experiments use frozen-weight inference-time search, not weight-space RL fine-tuning (e.g., GRPO [26]).

Impact statement

Fidelity regret measures cheap simulators’ reference shortfall before optimization and may reduce selection of circuits with poor predicted reference performance. Standard published gate characterizations support our intended-function study; verdicts apply only to evaluated models and design space. SAFE means zero worst-case normalized regret there (Appendix § 17), not biological safety or experimentally validated reliability. Misplaced confidence is the risk: syntax and topology may be verified while unmeasured simulator error misranks physically valid designs.

References

NeurIPS Paper Checklist

Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction report the exact measurement over a 21,362-design combination subspace (§2–§3, with ordered-assignment checks in Appendix K), the L2-versus-L3 asymmetry on SC1C1G1T1 (0.113 versus 0.801), the L0 tie result and 25%-floor oracle-whitelist bound, evidence from design-space and frozen-model optimization (§4), and matched-budget mitigations (§5). They report the fixed re-scoring rule as in-sample (mean excess regret 0.016 versus 0.305 for always-cheap and 0.034 for reference-only), and §7 states that weight-space RL fine-tuning was not evaluated.

Guidelines:

  • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.

  • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.

  • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

  • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: §7 is a dedicated limitations section. It states that the reference composes per-gate measurements without whole-circuit wet-lab validation; discusses range dependence and fit-residual Hill-parameter uncertainty (Appendix C); notes the five-library scope (Appendix E), the Eco1C2G2T2 decimal defect, and Bth1C1G1T1 missing cytometry (Appendix G); states the two-library 2.0 s modeled-budget scope (Appendix H); notes that enumeration is exact for the combination subspace rather than the ordered space and that the headline allocation fraction is selected in-sample (Appendices K and N); and notes that weight-space RL fine-tuning was not evaluated.

Guidelines:

  • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.

  • The authors are encouraged to create a separate “Limitations” section in their paper.

  • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

  • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

  • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

  • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

  • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

  • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [N/A]

Justification: The paper proves no theorems. The structural claim that L0L0 ties the design space is an empirical property of the enumerated cells (all 120 supported rows tie), reported as a measurement with its bracket [0, 1], not as a theorem.

Guidelines:

  • The answer [N/A] means that the paper does not include theoretical results.

  • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

  • All assumptions should be clearly stated or referenced in the statement of any theorems.

  • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

  • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

  • Theorems and Lemmas that the proof relies upon should be properly referenced.

Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [No]

Justification: The paper specifies the ladder equations (Appendix § 16), five public Cello UCF libraries with SHA-256 verification (Appendix § 15), the combination subspace, date-pinned public model ids (Appendix § 23), optimizer settings, and statistical procedures (Appendix § 24), but this submission omits the implementation, prompt templates, random seeds, and derived per-design tables needed to regenerate the reported results independently.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

  • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

  • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

  • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: This submission includes the manuscript source and figures, while the analysis code, derived data tables, and frozen-model response logs are not distributed with the submission; the underlying Cello UCF libraries are publicly available (Appendix A).

Guidelines:

  • The answer [N/A] means that paper does not include experiments requiring code.

  • Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

  • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

  • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

  • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

  • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?

Answer: [Yes]

Justification: No model parameters were trained in this work; the frozen-model track changes no weights. Proposal counts, optimization-pressure levels for the three inference-time optimizers, cheap/reference level assignments per arm, matched-budget mitigation settings (Appendix H), and the four discrepancy-model evaluation splits with their train/test construction (Appendix E) are specified.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

  • The full details can be provided either with the code, in appendix, or as supplemental material.

Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The stochastic optimizer results report 95% percentile-bootstrap intervals over seeds or paired runs (B=10,000B=10{,}000 and B=2,000B=2{,}000, Appendix § 24), with 30 seeds per design-space arm and 40 runs per matched-budget cell. The model-tier comparison is paired over 44 cells. Enumerated regret has no sampling interval because every design in each reported cell is evaluated.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

  • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

  • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

  • The assumptions made should be given (e.g., Normally distributed errors).

  • It should be clear whether the error bar is the standard deviation or the standard error of the mean.

  • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

  • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).

  • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Local enumeration and optimization ran CPU-only on an 8-core machine (7 parallel workers for enumeration at a 6.62×6.62\times aggregate speedup, Appendix § 15). Standalone and in-loop per-evaluation timings appear in Appendix § 18, and enumeration construction takes 11.911.9–21.7 μs21.7\,\mu\mathrm{s} per design. The frozen-model proposals use date-pinned public API models with call counts, token counts (2.00M total tokens), and list-price spend ($8.07) reported in Appendix § 23.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

  • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

  • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).

Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: The work involves no human subjects, personal data, or crowdsourcing. It uses published gate-characterization tables for laboratory chassis organisms and public API models. The paper reports the best-of-nn null result, the library-dependent discrepancy correction, budget-infeasible cells, and the high truncation rate of the intermediate model tier.

Guidelines:

  • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.

  • If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.

  • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: §8 is an impact statement. The positive case is that pre-optimization screening should reduce the selection of circuits whose predicted physical performance degrades under higher-fidelity models. Screening verdicts concern selection under the evaluated models, where SAFE means zero worst-case normalized regret in that space rather than biological safety. The stated risk is misplaced confidence: treating a topologically verified reward as validated when it retains unmeasured simulator error. The dual-use surface is discussed and assessed as limited because the work concerns whether intended circuits function using standard published gate characterizations.

Guidelines:

  • The answer [N/A] means that there is no societal impact of the work performed.

  • If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.

  • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

  • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

  • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

  • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [N/A]

Justification: No pretrained model, generative model, dataset, or software package is released with the submission.

Guidelines:

  • The answer [N/A] means that the paper poses no such risks.

  • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

  • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

  • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [No]

Justification: The paper cites and credits the originating publications for all five public Cello UCF v2 gate libraries (Appendix § 15), the prior code-generation setting and topological verifier (§1 and §3), and the date-pinned public API models (Appendix § 23), but it does not explicitly state the applicable license terms or terms of service for those assets.

Guidelines:

  • The answer [N/A] means that the paper does not use existing assets.

  • The authors should cite the original paper that produced the code package or dataset.

  • The authors should state which version of the asset is used and, if possible, include a URL.

  • The name of the license (e.g., CC-BY 4.0) should be included for each asset.

  • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

  • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

  • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

  • If this information is not available online, the authors are encouraged to reach out to the asset’s creators.

New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [N/A]

Justification: No new dataset, model, or software package is released with this submission.

Guidelines:

  • The answer [N/A] means that the paper does not release new assets.

  • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

  • The paper should discuss whether and how consent was obtained from people whose asset is used.

  • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [N/A]

Justification: No crowdsourcing and no research with human subjects.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

  • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [N/A]

Justification: No study participants.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

  • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

  • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: Large language models are a studied object of this work, not an incidental writing aid, and their role is reported in full. §4 reports the frozen-model track: three date-pinned public models proposing designs through the environment with no weight access and no parameter updates. Appendix § 23 gives exact model ids, per-model call and token counts, list-price cost ($8.07 total) and per-model proposal validity rates, including the 79.8% truncation rate that makes one model’s level a parsing artifact rather than a capability result.

Guidelines:

  • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

  • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.

Enumeration scope and per-library coverage

The primary enumeration covers 21,362 designs across 5 measured Cello UCF libraries and 3 organisms. Each library was fetched from the CIDAR Cello-UCF repository and verified by SHA-256. Table 1 gives the available fidelity levels. Unsupported levels are recorded as n.d. with a reason and excluded from summary statistics.

The measurements originate from the initial Eco1C1G1T1 Cello library [20], the Eco1C2G2T2 plasmid library [27], the Eco2C1G3T1 genome library [22], the yeast SC1C1G1T1 library [3], and the Bth1C1G1T1 B. thetaiotaomicron library [29]. Cello 2.0 documents their common UCF representation [9].

Combination subspace. In the primary enumeration, a design is a kk-element combination of distinct gates evaluated once under the implementation’s fixed slot-order convention. The count is therefore

2∑n∈{20,18,6,12,7}[(n3)+(n4)]=21,362,2\sum_{n\in\{20,18,6,12,7\}}\left[\binom{n}{3}+\binom{n}{4}\right]=21{,}362,

where the factor of two is the two topologies. It is not the count of ordered injective assignments, which would be

2∑n∈{20,18,6,12,7}[P(n,3)+P(n,4)]=432,372,P(n,k)=n!(n−k)!.2\sum_{n\in\{20,18,6,12,7\}}\left[P(n,3)+P(n,4)\right]=432{,}372,\qquad P(n,k)=\frac{n!}{(n-k)!}.

Slot order can change circuit scores, particularly in cascades, so no permutation invariance is assumed. The reported enumeration is exact for this combination subspace, not for the full ordered-assignment space. At the measured 112.0 ms aggregate CPU cost per L2 evaluation, the ordered Phase-1 L2 column would require approximately 13.5 CPU-hours; one 1,162,800-design Phase-2 library sweep would require approximately 1.5 CPU-days. Appendix K reports exact and sampled ordered-assignment checks.

L4 kinetics provenance. Only Eco1C2G2T2 supplies measured τon/τoff\tau_{\mathrm{on}}/\tau_{\mathrm{off}} values, for all 18 gates. The other four libraries use default kinetics. Among the 20 enumerated L4 cells, the 4 measured-kinetics cells have mean normalized regret 0.314 (maximum 0.452), while the 16 default-kinetics cells have mean 0.465 (maximum 0.938). L4 is therefore a common modeling axis but not a uniformly measured one. L2 and L3 are composed from per-gate tables supplied by every library that supports the corresponding level.

Construction and scoring costs. Constructing a design means enumerating an assignment and building its circuit object without evaluating a ladder level. This costs 11.9–21.7 μ\mus per design; the largest single cell, Eco1C1G1T1/nor_cascade_k4 with ordered assignment, constructs 116,280 circuits in 2.33 s. Reference scoring is measured separately. During parallel enumeration with 7 workers, L2 uses 112.0 ms of aggregate CPU time per design (averaged over the 16 cytometry-bearing cells). Memory contention from propagating the 250-bin by 12-level quantile table contributes to this parallel-enumeration cost.

The 112.0 ms parallel-enumeration measurement, the 86.4 ms standalone timing, and the 63.3 ms optimization-loop cost model measure different execution contexts. Budget comparisons use the optimization-loop model; construction timings are excluded. At the enumerated problem sizes, 7-worker parallelism provides a 6.62× aggregate speedup.

Simulator ladder definitions

Let P={(0,0),(0,1),(1,0),(1,1)}\mathcal{P} = \{(0,0),(0,1),(1,0),(1,1)\} be the evaluated input patterns, t(p)∈{0,1}t(p) \in\{0,1\} the target output, and P+={p:t(p)=1}\mathcal{P}^{+} = \{p : t(p) = 1\} and P−={p:t(p)=0}\mathcal{P}^{-} = \{p : t(p) = 0\}. Every ladder score is maximized. No score uses an inverse-trigonometric function or requires a branch convention.

L0: topology. Boolean values are propagated through the regulatory graph, with a NOR node emitting ¬(⋁iai)\neg\left(\bigvee_i a_i\right). The dimensionless score is

sL0=1∣P∣∑p∈P1[bout(p)=t(p)]∈[0,1].s_{\mathrm{L0}} = \frac{1}{|\mathcal{P}|} \sum_{p \in\mathcal{P}} \mathbf{1}[b_{\mathrm{out}}(p) = t(p)] \in[0,1].

It equals one when the topology implements the target and is independent of gate identity.

L1: mean-field steady state. Gate inputs are combined and propagated through the fitted repressive Hill functions to obtain the output yout(p)y_{\mathrm{out}}(p) in relative promoter units (RPU). Where specified by the library, the input combination includes the fitted tandem-interference factor. The score is

sL1=log⁡10(min⁡p∈P+yout(p)max⁡p∈P−yout(p)),s_{\mathrm{L1}} = \log_{10}\left(\frac{\min_{p \in\mathcal{P}^{+}} y_{\mathrm{out}}(p)}{\max_{p \in\mathcal{P}^{-}} y_{\mathrm{out}}(p)}\right),

measured in decades of RPU separation. Positive values place every ON output above every OFF output.

L2: cytometry quality and separation. A gate’s single-cell measurements provide P(y∣xℓ)P(y \mid x_\ell) at 12 input levels. For an input distribution QQ, propagation uses the equal-weight quantile approximation

P(y)=∫P(y∣x)Q(x) dx≈1nquad∑j=1nquadP(y∣xj),xj=Q−1(uj).P(y) = \int P(y \mid x)Q(x)\,\mathrm{d}x \approx\frac{1}{n_{\mathrm{quad}}}\sum_{j=1}^{n_{\mathrm{quad}}}P(y \mid x_j), \qquad x_j = Q^{-1}(u_j).

This composition assumes that the measured conditional spread is intrinsic to each gate and independent between gates in a cascade. The ON and OFF pattern distributions are pooled separately. If μ\mu and σ\sigma denote their means and standard deviations in log⁡10\log_{10} RPU, then

sL2=(μon−μoff)−(σon+σoff),sL2 d′=∣μon−μoff∣(σon2+σoff2)/2.s_{\mathrm{L2}} = (\mu_{\mathrm{on}}-\mu_{\mathrm{off}})-(\sigma_{\mathrm{on}}+\sigma_{\mathrm{off}}), \qquad s_{\mathrm{L2}}\,d' = \frac{|\mu_{\mathrm{on}}-\mu_{\mathrm{off}}|}{\sqrt{(\sigma_{\mathrm{on}}^2+\sigma_{\mathrm{off}}^2)/2}}.

The first score is in decades of RPU; d′d' is dimensionless. The absolute value in the d′d' numerator makes it invariant to reversing the ON/OFF mean ordering, so it certifies separation but not analog polarity. The signed L1 and L2 scores retain polarity and become negative when that ordering inverts. We use d′d' only as a separation reference, never as the sole certificate of correct direction.

L3: growth burden. The toxicity table gives each gate’s relative growth rate gi(x)g_i(x) at input level xx. Independent relative-growth effects are multiplied, and the worst truth-table state is scored:

sL3=min⁡p∈P∏i∈gatesgi(xi(p))∈(0,1],penaltylog⁡10=−log⁡10sL3.s_{\mathrm{L3}} = \min_{p \in\mathcal{P}} \prod_{i \in\mathrm{gates}} g_i(x_i(p)) \in(0,1], \qquad\mathrm{penalty}_{\log_{10}} = -\log_{10}s_{\mathrm{L3}}.

Each xi(p)x_i(p) is the L1 mean-field input to gate ii.

L4: response kinetics. Each logic node relaxes toward its Hill steady state as

dyidt=Hill⁡i(xi(t))−yiτi.\frac{\mathrm{d}y_i}{\mathrm{d}t} = \frac{\operatorname{Hill}_i(x_i(t))-y_i}{\tau_i}.

The implementation integrates this system with LSODA [7, 23] and measures the output’s 10–90% response time trespt_{\mathrm{resp}} for the configured OFF-to-ON input step. The maximize score is

sL4=−log⁡10(max⁡(tresp,10−6)),s_{\mathrm{L4}} = -\log_{10}\left(\max(t_{\mathrm{resp}},10^{-6})\right),

the negative logarithm of response time in simulator time units. Direction-dependent τi\tau_i values come from the UCF where available and from a library default otherwise.

Joint L2+L3 references. The three joint references use

sjoint=sL2−w penaltylog⁡10,w∈{0.5,1,2}.s_{\mathrm{joint}} = s_{\mathrm{L2}} - w\,\mathrm{penalty}_{\log_{10}}, \qquad w \in\{0.5,1,2\}.

Both terms are measured in decades, so ww is dimensionless.

Missing data. An unsupported level is neither imputed nor scored as zero. Bth1C1G1T1 lacks cytometry, so L2, L2 d′d', and the three joint references are recorded as n.d. and excluded from summaries. Default kinetics at L4 are retained but identified separately in Table 1.

LibraryOrganismGatesHillCytometryToxicityτ\tauLevels
Eco1C1G1T1E. coli NEB 10-beta202020200L0–L4†
Eco1C2G2T2E. coli NEB 10-beta1818181818L0–L4
Eco2C1G3T1E. coli MG1655 K-1266660L0–L4†
SC1C1G1T1S. cerevisiae BY4741121212120L0–L4†
Bth1C1G1T1B. thetaiotaomicron77070L0,L1,L3,L4†

Table 1. Per-library coverage. Bth1C1G1T1 ships growth-burden tables for all seven gates but no single-cell cytometry, so the cytometry references (L2, L2′ d and the joint L2+L3 references) are unsupported there; L3 and L4 are supported. The τ\tau column counts gates with measured τon/τoff\tau_{\mathrm{on}}/\tau_{\mathrm{off}} values; † denotes library-default kinetics.

Full combination-subspace exact-regret tables

Full four-panel version of Fig. 1

Figure 3. Full four-panel version of Fig. 1. (c) Worst normalized regret per library × reference with the group minimum beneath, bold where it is exactly zero; Bth1C1G1T1 cytometry cells are n.d. because cytometry is unsupported. (d) Fraction of achievable growth-burden quality kept on each library’s worst cell, annotated with the gate the cheap score recruits and the oracle avoids.

c: colour and the large number are the WORST normalized regret over the k=3k=3 and k=4k=4 enumerations in that library × reference group; the small number is the group minimum, in bold where it is exactly 0. Six of the 72 supported L1 cells are exact zeros (Eco2C1G3T1 chain_k3 and and_k3 on L2 and on L2 d′, Eco2C1G3T1 xnor_k4 and SC1C1G1T1 and_k3 on L2 d′). Bth1C1G1T1 has no cytometry, so L2 and L2 d′ are n.d., never zero.

  • Eco1C2G2T2 carries a UCF decimal defect in its shipped table (row 7 reads 0.069681, should be 0.69681); see Table T1 notes.

The full combination-subspace enumeration spans 280 (library, topology, cheap simulator, reference) rows, recording the selected cheap optimum, oracle design, number of cheap-score ties, reference distribution, and rank agreement. The 72 main-text cells are its evaluable single-axis L1-reward subset. Table 2 reports the worst normalized regret per (library, reference) over k∈{3,4}k \in\{3, 4\}, as visualized in Figure 1c.

LibraryL2 (cyt. quality)L2 d′d' (cyt. sep.)L3 (burden)L4 (kinetics)
Eco1C1G1T10.5360.6870.9080.863
Eco1C2G2T2{sup}*0.5140.5160.2900.452
Eco2C1G3T10.4700.4310.8520.926
SC1C1G1T10.1330.2300.9990.938
Bth1C1G1T1n.d.n.d.0.6300.870

Table 2. Worst normalized regret per library and reference axis, L1 paying the reward. n.d. = the reference is not evaluable for that library. {sup}* carries the Eco1C2G2T2 UCF defect flag (§G).

Regret metrics. Three quantities use different denominators. Let smax⁡=max⁡dsref(d)s^{\max} = \max_d s_{\mathrm{ref}}(d), smin⁡=min⁡dsref(d)s^{\min} = \min_d s_{\mathrm{ref}}(d), and let dcheap∗d^{*}_{\mathrm{cheap}} be the cheap-score maximizer. For a returned design dd, define

Rrange(d)=smax⁡−sref(d)smax⁡−smin⁡,Rfrac(d)=smax⁡−sref(d)smax⁡−sref(dcheap∗),Rec⁡(d)=1−Rfrac(d).R_{\mathrm{range}}(d) = \frac{s^{\max} - s_{\mathrm{ref}}(d)}{s^{\max} - s^{\min}}, \qquad R_{\mathrm{frac}}(d) = \frac{s^{\max} - s_{\mathrm{ref}}(d)}{s^{\max} - s_{\mathrm{ref}}(d^{*}_{\mathrm{cheap}})}, \qquad\operatorname{Rec}(d) = 1 - R_{\mathrm{frac}}(d).

RrangeR_{\mathrm{range}} is the range-normalized enumeration statistic and lies in [0,1][0, 1]. RfracR_{\mathrm{frac}} is the regret fraction incurred relative to the cheap optimum: it is one at dcheap∗d^{*}_{\mathrm{cheap}} and can exceed one for a design that the reference scores worse. Rec is the recovered fraction in Appendix § 22; it is zero at the cheap optimum, one at the reference optimum, and unbounded below. The enumeration fixes both anchors. When the cheap optimum is also reference-optimal, as in 6 of the 72 main-text cells, the denominator of RfracR_{\mathrm{frac}} is zero; RfracR_{\mathrm{frac}} and Rec are recorded as nan and excluded from means with the excluded count reported.

The matched zero-regret control in Appendix F instead reports RrangeR_{\mathrm{range}} for optimizer-returned designs, which can be positive when finite optimization does not return the exact cheap optimum.

Cheap-score ties. If the cheap argmax contains several designs with different reference scores, Eq. (1) requires an explicit tie rule. Across the enumeration, all 120 L1 cells have a unique cheap argmax, so tie-breaking does not affect any reported L1 point estimate. All 120 L0 cells are fully tied, with 15–4,845 optima, and are reported with the interval from best-tie to worst-tie regret and the uniform-tie-break expectation.

We also evaluate sensitivity to near-ties within a tolerance of 10−310^{-3} decades of L1 score. Forty-nine of 120 L1 cells contain a near-tied competitor; in 41, choosing among near-tied designs changes absolute regret by more than 0.01. The median normalized spread is 0.126 and the maximum is 0.697 for Eco1C1G1T1/and_k3 against L3.

Reference provenance. The UCF provides cytometry, Hill fits, and toxicity for individual gates; the ladder composes these measurements into circuit-level predictions. The study contains no whole-circuit wet-lab validation, and composition error is therefore unmeasured. A higher rung is a more detailed model rather than ground truth.

Hill-parameter uncertainty is derived from cytometry fit residuals and represents uncertainty in the fitted parameters. The UCF provides one median per input level, so replicate-level measurement scatter cannot be recovered. The optimizer study uses the AND target at k=4k=4, whereas the enumeration includes other targets. Results are kept separate across targets and, in the mitigation study, across libraries because the observed effects can reverse sign (Appendix § 22).

Range dependence. Normalized regret divides by the reference range over the enumerated space, so a narrow range inflates it. Bth1C1G1T1/and_k3 against L4 is 0.870 normalized but only 0.085 decades in absolute terms. Each normalized figure in this paper is paired with either the percentile of the pick or the absolute value.

Operational verdict rule. The prespecified screening convention labels a (library, kk, reference) cell SAFE when its worst normalized regret is 0, CAUTION when it is in (0, 0.35], UNSAFE when it exceeds 0.35, and n.d. when the reference is unsupported. The 0.35 boundary is an operational triage threshold and carries no significance-test interpretation. Among 70 rows, 41 are UNSAFE, 14 CAUTION, 5 SAFE, and 10 n.d.; the L3 burden axis contains 7 UNSAFE and 3 CAUTION rows.

Level disagreement and the cost ladder

The fidelity levels induce different gate orderings. L1-versus-L3 Spearman correlations are −0.243 for Eco1C1G1T1, −0.024 for Eco1C2G2T2, −0.257 for Eco2C1G3T1, −0.071 for Bth1C1G1T1, and +0.260 for SC1C1G1T1. Four are negative, and all are close to zero. Thus L1 provides little information about the measured burden ordering, which accounts for the burden regret in §3. L1-versus-cytometry agreement is higher, ranging from 0.50 to 0.89.

Standalone and optimization-loop costs. The standalone benchmark times isolated evaluations across L0–L4. The optimization-loop model measures L0–L3 calls in the training execution context and determines every budget comparison.

The standalone benchmark on 12 Eco1C1G1T1 circuits gives L0 0.000357 s, L1 0.000916 s, L3 0.003585 s, L2 0.086403 s, L4 0.475348 s, and cached L4 0.000033 s. It is the only measurement covering L4. The optimization-loop model gives L0 0.000418 s, L1 0.000939 s, L2 0.063297 s, and L3 0.002053 s. The measurements agree within 2.5% on L1 and 15% on L0; execution-context overhead and amortization produce differences of 1.37 × on L2 and 1.75 × on L3. All optimization budgets use the corresponding exchange rates of 67.4 L1 calls per L2 call and 2.2 L1 calls per L3 call.

Figure 4. (a) Spearman rank correlation between fidelity levels on each library’s gate set. (b) Cost per evaluation relative to L0, log axis. Measured on 12 Eco1C1G1T1 circuits, uncached. L4 with a warm cache falls to 0.091× L0. Bth1C1G1T1 ships no cytometry, so its L2/L4 pairs are n.d. * Eco1C2G2T2 carries a UCF decimal defect in its shipped table (row 7 reads 0.069681, should be 0.69681); see Table T1 notes.

Timing basisMeasuresL2 (s)L2/L1L3/L1
Standalone benchmarkisolated per-evaluation, all 5 levels0.08640394.33.9
In-loop cost modeloptimization-loop execution, L0–L30.06329767.42.2

Table 3. The two measured cost bases. Every budget comparison in this paper uses the in-loop exchange rates; the standalone benchmark supplies the relative-to-L0 ladder in Fig. 4b.

Ensemble cost. Under the optimization-loop model, one CPU-hour supports 59,903 evaluations of a 64-member L1 ensemble or 56,874 direct L2 evaluations. The break-even ensemble size is 67 members, equal to the L2/L1 exchange rate. An 8-member ensemble provides 479,228 evaluations per CPU-hour. The standalone measurement would place break-even at 94 members, demonstrating why budget comparisons use a single execution context. In §5, all variants select from identical proposal sets and match the high-fidelity budget by construction.

Discrepancy models under distribution shift

In Figure 5a, the two E. coli NEB 10-beta libraries have worst burden regrets of 0.908 and 0.290, while libraries from three organisms interleave in the observed ordering. Across these five libraries, regret varies more between gate libraries than by host organism, although five libraries are too few for formal cross-library attribution.

Three-panel plots showing worst normalized fidelity regret and discrepancy-model $R^2$ across libraries and splits

Figure 5. (a) Worst normalized regret per library on cytometry and burden; two E. coli libraries land at opposite ends. (b) Discrepancy-model R2R^2 across four splits: L2 includes negative values under shift, whereas L3 remains positive. (c) R2R^2 on optimizer-returned designs: L2 median 0.212, L3 median 0.740.

Four splits were evaluated per (library, reference, learner): random, topology_holdout, shift_q90 (train below the 90th cheap-score percentile, test above), and shift_q99. A fifth evaluation scores the model on the optimizer-returned designs from the design-space search.

On SC1C1G1T1 under shift_q90, the L2 correction’s RMSE reduction relative to no correction is −0.338, corresponding to a 34% increase in RMSE. On the designs returned by the optimizer, it reaches R2=−0.571R^2 = -0.571 for the same library. The L3 correction has positive R2R^2 for every library and split. Feature-importance analysis attributes the L3 result to two directly available features, reducing its dependence on extrapolated distributional properties.

Sample sizes. Eco2C1G3T1 contains 150 designs and has no shift_q99 split. The three remaining cytometry-bearing libraries carry all four splits, and Eco2C1G3T1 is excluded from means that require all four. Test sets of optimizer-returned designs contain 4–31 unique designs, so these estimates have substantial sampling uncertainty.

SplitL2 cytometry qualityL3 growth burden
random0.520 – 0.9720.898 – 0.988
topology_holdout−0.036 – 0.8870.780 – 0.976
shift_q90−1.334 – 0.6990.379 – 0.991
shift_q990.409 – 0.8570.856 – 0.996
optimizer-returned (median)0.2120.740

Table 4. Gradient-boosted discrepancy-model R2R^2, range across libraries. L2 includes negative values under distribution shift; L3 remains positive.

Pressure–regret trajectories

The values in this section are RrangeR_{\mathrm{range}}, not RfracR_{\mathrm{frac}}. Under best-of-nn, Haiku 4.5 regret falls from 0.901 to 0.782, Opus 4.5 changes from 0.880 to 0.885, and Sonnet 4.5 is constant. Pooled across models, iterative-refinement regret is 0.602, 0.691, 0.714, 0.727, and 0.742 for R=1…5R = 1 \ldots5; evolutionary-search regret is 0.605, 0.665, 0.718, and 0.711 for G=1…4G = 1 \ldots4. Regret increases when the optimizer feeds the cheap score back to the model and shows no corresponding increase under best-of-nn.

Matched falsification control. On the matched cell SC1C1G1T1/and_k3, identical proposals under identical cheap-reward selection scored against two references: against L3 (enumerated regret 0.771) regret rises 0.558, 0.575, 0.623, 0.628, 0.674 with pressure; against L2 d′d' (enumerated regret exactly 0.000) it falls 0.313, 0.291, 0.248, 0.233, 0.199. The opposite trends for the same proposals and selection rule identify reference misspecification as the source of the effect.

Library data and coverage sensitivity

Eco1C2G2T2 decimal point. Row 7 of the published cytometry table reads 0.069681; 0.69681 restores monotonicity of the transfer curve. Both variants were enumerated over all 30,600 designs. The cheap argmax is unchanged (index 6350), as is the number of cheap ties (2). This 30,600-design sensitivity enumeration is reported separately from the 21,362-design primary enumeration summarized in the main text.

Bth1C1G1T1 cytometry coverage. This library provides no single-cell cytometry collection. L2, L2 d′d', and the three joint L2+L3 references are unsupported and appear as n.d. in figures and tables. L3 and L4 remain supported for the enumeration; its worst L4 cell is the 0.870-normalized, 0.085-decade example above. A value of zero would be semantically incorrect: the L1/L2 score

Range-normalized regret against optimization pressure for the frozen models

Figure 6. Range-normalized regret against optimization pressure for the frozen models. Regret rises under optimizers that feed the cheap score back into the prompt and does not rise under pure best-of-nn selection. Range-normalized regret Rrange∈[0,1]R_{\mathrm{range}} \in[0,1] (§C) remains well defined on the zero-enumerated-regret cell of the falsification control, where a cheap-optimum-relative fraction has a zero denominator.

VariantReferenceAbsolute regretPercentile of pickSpearman
as publishedL21.17664.80.531
correctedL20.87984.30.593
as publishedL30.16697.10.063
correctedL30.16697.10.063

Table 5. Eco1C2G2T2 sensitivity to the decimal defect. The L3 result is unaffected; the L2 result changes materially, so both are reported.

log⁡10(min⁡ON/max⁡OFF)\log_{10}(\min_{\mathrm{ON}}/\max_{\mathrm{OFF}}) equals zero when ON and OFF are equal, which is higher than most of this design space.

Matched-budget mitigation study

Design. The study contains 1,280 runs: 2 libraries, 2 reference axes, 8 reward variants, and 40 runs per cell (20 seeds for each of REINFORCE and (μ+λ)(\mu+\lambda) evolution). Every run receives a modeled budget equivalent to 2.0 s under the optimization-loop cost model, so reference calls reduce the available training steps. The reported quantity is the fraction of achievable regret recovered, 1−1- regret fraction. Zero corresponds to the cheap simulator’s selected optimum, one to the enumerated reference optimum, and negative values to designs below the cheap optimum under the reference. The scale is unbounded below.

Budget infeasibility. The discrepancy variants allocate 64 high-fidelity evaluations from the same budget. For L2, this acquisition costs 4.05 s and exceeds the 2.0 s budget. The 160 affected runs are recorded as budget_infeasible with NaN recovery because they execute no training calls. Smaller L2 acquisition budgets were not evaluated.

Discrepancy reward. On SC1C1G1T1 against L3, passive discrepancy correction recovers 0.770, an improvement of +0.763 over cheap-only (p<10−4p < 10^{-4}). On Eco1C2G2T2 against L3, its recovery is −0.479 [−0.587, −0.371], an improvement of +0.245 over cheap-only (p=0.023p = 0.023). The active variant differs from cheap-only by +0.100 (p=0.41p = 0.41). These library-specific results are reported separately.

Reference-only reward. On SC1C1G1T1 against L2, the L2 reference-only arm recovers −2.168 [−2.997, −1.418], worse than the naive cheap reward’s −0.720 (Δ=−1.448\Delta= -1.448, p<10−4p < 10^{-4} paired), because 2 s buys roughly 32 L2 evaluations, not enough for either optimizer to leave its initialization. On Eco1C2G2T2, recovery is 0.433 [0.361, 0.502] against cheap-only’s 0.155 (Δ=+0.277\Delta= +0.277, p<10−4p < 10^{-4}). Against L3, where evaluations are far cheaper, it recovers 0.944 on SC1C1G1T1 and 0.555 on Eco1C2G2T2. The L2 sign reversal shows that the value of reference-only optimization at fixed budget depends on the library.

Risk aversion. Cheap-ensemble risk measures lose to the naive cheap reward in 3 of the 4 cells; the exception is SC1C1G1T1 × L3, where CVaR⁡0.05\operatorname{CVaR}_{0.05} gives a small gain (0.152 versus 0.007). The nominal-performance premium these rewards pay is small: 0.008 decades of L1 score on SC1C1G1T1 across κ=0.5\kappa= 0.5–2 and 0.340 on Eco1C2G2T2. Here κ\kappa is the risk-aversion multiplier in the mean−κσ\kappa\sigma statistic and is unrelated to circuit size k∈{3,4}k \in\{3, 4\}. The losses arise because the predictive spread has little alignment with reference shortfall (Appendix § 29).

Table 7: Curriculum schedules on SC1C1G1T1 at the matched budget. The reference_only row provides the direct-reference comparator.

ScheduleswitchL3 recovered [95% CI]vs cheap_only (pp)
cheap_only–0.007 [−0.036, 0.050]–
reference_then_cheap0.500.169 [0.093, 0.250]+0.162 (< 10−410^{-4})
interleaved0.500.641 [0.540, 0.739]+0.634 (< 10−410^{-4})
cheap_then_reference0.750.618 [0.506, 0.733]+0.611 (< 10−410^{-4})
cheap_then_reference0.500.783 [0.697, 0.861]+0.776 (< 10−410^{-4})
cheap_then_reference0.250.890 [0.853, 0.926]+0.883 (< 10−410^{-4})
reference_only (ceiling)–0.944 [0.926, 0.961]+0.937 (< 10−410^{-4})
Same schedules against L2: no mixed schedule separates from cheap-only (best +0.261, p=0.13p=0.13); reference_only recovery is −2.256; its paired difference from cheap-only is −1.536 (p=0.0002p=0.0002).

Table 7.

Curriculum. The sweep covers SC1C1G1T1. Against L3, cheap-then-reference at a 50/50 split recovers 0.783 compared with 0.007 for cheap-only; the reference-then-cheap control recovers 0.169, and a 0.25 switch recovers 0.890. Reference-only recovers 0.944 and exceeds the mixed schedules by +0.053 to +0.775. Against L2, no mixed schedule separates from cheap-only: the largest gain is +0.261 [−0.071, 0.621] at a 0.25 switch (p=0.13p = 0.13), and the matched 50/50 difference is +0.025 (p=0.88p = 0.88). Reference-only recovery is −2.256 [−3.075, −1.493]; its paired difference from cheap-only is −1.536 [−2.327, −0.789] (p=0.0002p = 0.0002).

Reference-only comparators. The −1.536 value is a paired difference within the curriculum sweep, whereas the −2.168 in Table 6 is an absolute recovery level from the mitigation study. The comparable curriculum-sweep level is −2.256. The remaining difference of approximately 0.09 between the two levels reflects budget accounting rather than different seeds: both use the same 20 seeds and two optimizers, and 32 of 40 paired runs are identical. The curriculum harness stops after 31 reference evaluations at 1.962 s charged cost, while the mitigation harness admits a 32nd evaluation and records 2.026 s. The values are therefore reported separately.

Reward variantSC1C1G1T1: L3 burdenSC1C1G1T1: L2 cytometryEco1C2G2T2: L3 burdenEco1C2G2T2: L2 cytometry
cheap L1 (naive)0.007−0.720−0.7240.155
reference simulator0.944−2.1680.5550.433
ensemble mean, M=8M=80.065−1.011−1.9250.149
ensemble mean −1σ0.123−2.475−2.1240.111
ensemble CVaR0.05_{0.05}0.152−3.499−1.9780.063
moment matched0.040−0.285−0.7820.176
discrepancy, passive B=64B=640.770infeasible−0.479infeasible
discrepancy, active B=64B=640.716infeasible−0.624infeasible

Table 6. Fraction of achievable regret recovered at a matched modeled budget equivalent to 2.0 s, mean [95% CI] over 40 runs. infeasible indicates that the declared acquisition cost exceeds the full budget and permits no training calls.

Scope. The study covers 2 of the 4 cytometry-bearing libraries (SC1C1G1T1 and Eco1C2G2T2) at a matched modeled budget equivalent to 2.0 s. Absolute recovery values, especially for the L2 reference-only arm, are specific to this budget.

Fidelity curricula on SC1C1G1T1

Figure 7. Fidelity curricula on SC1C1G1T1 (the only library swept), 40 runs per point at the matched modeled budget equivalent to 2.0 s. (a) Against growth burden every mixed schedule beats cheap-only; against cytometry quality none does, and reference_only has negative recovery. (b) On growth burden (L3), every cheap-first schedule lies below the reference_only line. On cytometry quality (L2), the ordering inverts because reference_only is harmful there. The recovery axis is unbounded below.

Controls. A reference-blinding check is run for every variant that does not read the reference: multiplying reference scores by −100-100 leaves the optimization trajectory unchanged. The discrepancy reward reads reference scores only on the designs in its paid acquisition set, and perturbing reference scores outside that set leaves the returned design unchanged.

Models, tokens and reproduction cost

The frozen-model experiment used three date-pinned public model ids: claude-haiku-4-5-20251001, claude-sonnet-4-5-20250929 and claude-opus-4-5-20251101.

ModelFresh callsInput tokensOutput tokensList cost (USD)
Haiku 4.5182840,62695,8441.32
Opus 4.5164760,51556,4535.21
Sonnet 4.5165182,68366,0001.54
Total5111,783,824218,2978.07

Table 8. Frozen-model spend at public list prices. Totals are 511 fresh calls, 2.00M tokens, $8.07.

The frozen-model experiment costs $8.07 at public list prices. Proposal validity differed sharply by model: invalid rates 2.4% (Haiku), 5.1% (Opus) and 79.8% (Sonnet), the last because only 3.6% of its responses closed without truncation before the output cap. Sonnet is therefore treated as a parsing artifact, leaving a two-tier capability comparison.

Statistical procedures

All empirical confidence intervals and arm comparisons in this paper use nonparametric bootstrap procedures.

Intervals. Reported intervals for a mean are percentile-bootstrap intervals. The resampling unit is a seeded optimizer run, indexed by (optimizer, seed) within a (library, reference, variant) cell. The point estimate is the ordinary sample mean, and the interval uses the 2.5th and 97.5th percentiles of bootstrap means. Non-finite values are removed before resampling and the retained count is reported. Mitigation, curriculum, and recovery tables use B=10,000B = 10{,}000 resamples; the 792-cell allocation sweep uses B=2,000B = 2{,}000. Paired comparisons. Comparisons with a baseline pair runs by optimizer and seed, which preserves the shared initialization. The statistic is the mean paired difference. The reported two-sided bootstrap value is

p=2min⁡{Pr⁡(dˉ∗≤0),Pr⁡(dˉ∗≥0)},p = 2\min\{\Pr(\bar{d}^{*} \le0),\Pr(\bar{d}^{*} \ge0)\},

capped at one. With B=10,000B = 10{,}000, the smallest nonzero two-sided value is 2×10−42 \times10^{-4}; a reported p<10−4p < 10^{-4} means that no resampled mean crossed zero and should be interpreted as a resolution statement rather than an exact tail probability.

Multiplicity and cell-level inference. No familywise-error or false-discovery-rate correction is applied, so all reported pp-values are unadjusted. Raw runs are not pooled across libraries for inference; comparisons are made within their stated cells.

Ordered-assignment sensitivity

The primary results use the combination subspace defined in Appendix § 15. Ordered-assignment checks quantify how much the reported values can change when gate position is also enumerated.

Exactly enumerable ordered spaces. Two ordered spaces were evaluated completely: Eco2C1G3T1/chain_k4, with P(6,4)=360P(6, 4) = 360 designs, and SC1C1G1T1/chain_k3, with P(12,3)=1,320P(12, 3) = 1{,}320. Table 9 compares range-normalized regret on the combination and ordered spaces.

LibraryTopologyOrdered designsReferenceCombinationOrdered
Eco2C1G3T1chain_k4360L20.4700.118
Eco2C1G3T1chain_k4360L2 d′d^{\prime}0.4310.229
Eco2C1G3T1chain_k4360L30.5470.436
Eco2C1G3T1chain_k4360L40.9260.506
SC1C1G1T1chain_k31,320L20.1330.139
SC1C1G1T1chain_k31,320L2 d′d^{\prime}0.2300.302
SC1C1G1T1chain_k31,320L30.9990.123
SC1C1G1T1chain_k31,320L40.0960.243

Table 9. Range-normalized regret on the combination subspace and the fully enumerated ordered-assignment space.

Regret remains positive in all eight ordered cells, but the magnitude changes in both directions: it is higher in three and lower in five, with mean absolute change 0.273 and maximum change 0.876. No ordered cell becomes zero-regret. The combination-space values are therefore neither uniformly conservative nor uniformly anti-conservative.

Large ordered spaces. For the two P(20,4)=116,280P(20, 4) = 116{,}280 spaces in Table 10, the L1 score was evaluated on every ordered design, so the cheap argmax is exact. The reference maximum was estimated from that design and a uniform sample of 4,000 additional designs. A sampled maximum cannot exceed the true maximum, so the reported absolute regret is a lower bound on the full ordered-space absolute regret. The normalized column uses the sampled reference range and is descriptive rather than a formal lower bound.

LibraryTopologyReferenceAbsolute lower bound [bootstrap]Normalized
Eco1C1G1T1chain_k4L20.164 [0.091, 0.164]0.059
Eco1C1G1T1chain_k4L2 d′d^{\prime}0.502 [0.258, 0.502]0.107
Eco1C1G1T1chain_k4L30.327 [0.299, 0.327]0.257
Eco1C1G1T1chain_k4L40.100 [0.091, 0.100]0.264
Eco1C1G1T1xnor_k4L20.171 [0.142, 0.171]0.066
Eco1C1G1T1xnor_k4L2 d′d^{\prime}0.392 [0.303, 0.392]0.087
Eco1C1G1T1xnor_k4L30.331 [0.312, 0.331]0.271
Eco1C1G1T1xnor_k4L41.417 [1.417, 1.417]0.974

Table 10. Ordered-assignment checks for Eco1C1G1T1, with 116,280 designs per cell. The cheap argmax is exact; the reference statistics use a sample of 4,001 designs.

All eight absolute lower bounds are positive. Validation against six exactly evaluated ordered cells and 200 replicates per sample size found mean normalized bias of −0.103 at m=10m = 10, −0.055 at m=50m = 50, −0.020 at m=200m = 200, and −0.008 at m=800m = 800; no replicate overestimated the exact value. Together, the exact and sampled checks preserve the qualitative positive-regret finding, but no specific 21,362-design value should be transferred to the full ordered space.

Subsampling in non-enumerable spaces. Exhaustive enumeration of the combination subspace is tractable at k≤4k \le4 and becomes prohibitive in larger circuit spaces. Validating subsampled estimators of Eq. (1) against exact values on enumerable cells shows that uniform-random subsampling of mm designs converges toward exact regret with modest finite-sample bias, whereas restricting a subsample to the top-mm designs ranked by cheap score is biased low throughout the evaluated range. A pre-run screen restricted to cheap-simulator favorites therefore underestimates exact regret.

Bounds for stronger per-gate verifiers

Method. We evaluate five verifier classes on every (library, topology, reference) cell. V0 is gate-blind. V1 blacklists the worst designs by cheap score. V2 imposes a growth floor, and V3 a quality floor, using cheap-simulator features. V4 is an oracle whitelist allowed to admit any subset using full reference scores. V4 is non-implementable and provides an exact upper bound for the per-gate-admissibility class through subset enumeration.

The regret numerator remains the full combination-subspace optimum, including when the verifier restricts the admitted set. Each verifier is evaluated at its best parameter subject to admitting at least 25% of the design space. This constraint prevents a low regret value obtained by removing nearly all candidates.

LibraryV0 gate-blindV1 blacklistV2 growth floorV3 quality floorV4 oracle
Bth1C1G1T10.3460.1500.1500.2730.035
Eco1C1G1T10.3830.1720.1740.3810.036
Eco1C2G2T20.2080.1200.1750.1570.089
Eco2C1G3T10.6360.3140.3140.6360.314
SC1C1G1T10.8010.6550.6550.7410.066

Table 11. Normalized regret under each verifier class against growth burden (L3), best parameter subject to admitting ≥25%\ge25\% of the space, averaged over topologies. V4 is the non-implementable oracle whitelist and therefore an upper bound on the whole class.

Results. Over all 52 (library, topology, reference) cells at ≥25%\ge25\% permissiveness, the gate-blind verifier leaves mean normalized regret 0.344 (max 0.999). The oracle whitelist reduces this to mean 0.058 and improves on V0 in 45 of 52 cells, with a median reduction of 0.239. Positive regret remains in 30 cells, with a maximum of 0.511. On the growth-burden axis, 17 of 20 cells retain nonzero regret under V4 (mean 0.108); the three exceptions are Bth1C1G1T1/chain_k3, Bth1C1G1T1/chain_k4 and Eco2C1G3T1/chain_k3. On the cytometry axes the bound is tighter: max 0.158 (L2) and 0.203 (L2 d′d').

The V4 column bounds the residual for verifiers that admit or reject designs through per-gate admissibility. This class cannot eliminate fidelity regret across the evaluated cells. The bound does not apply to a verifier that measures the reference axis or reshapes the design space.

Cheap features as constraints. Two cheap features predict simulator discrepancy in some libraries. We evaluate three uses: a hard constraint, a lexicographic preference, and a soft penalty against the naive cheap-argmax baseline and against two paid baselines (Figure 9). At their best parameter per cell, the free constraints help in 30/52 (hard), 32/52 (lexicographic) and 26/52 (soft) cells, with median gains over all cells of 0.054, 0.061 and 0.003. The paid comparators do better: a fitted discrepancy model helps in 37/46 cells (median 0.233) and the reference oracle in 46/52 (median 0.293). Medians are over every cell, including those the method does not help; over helping cells only they are 0.158, 0.181, 0.152, 0.295 and 0.343.

Figure 9. Cheap discrepancy-predicting features used as free constraints, against paid baselines. The best free method beats the naive baseline in 11 of 13 library-by-axis cells, at a median of 4.0 reference calls to match.

Tuning dependence. The gains above use the best parameter per cell, selected with reference evaluations. Two checks quantify this dependence. First, at one fixed parameter across all cells, the free constraints improve regret in 22 of 39 (library, reference, method) rows and worsen it in the other 17; per method the split is 9/4 (hard constraint at 0.90), 6/7 (lexicographic at 0.05) and 7/6 (soft penalty at 16). The fixed parameters improve 22 rows and worsen 17. Second, a threshold tuned on one reference axis and applied to another carries a penalty in 7 of 12 transfers, up to 0.270 normalized regret; the worst case is Eco2C1G3T1 tuned on L3 and applied to L2 d′d^\prime.

Figure 8. Fidelity regret under each verifier class as a function of how much of the design space the verifier admits. The oracle whitelist (V4) is a non-implementable upper bound on the per-gate-admissibility class; it flattens well above zero in most cells.

Matched against paid alternatives at equal budget, the best free method beats the naive baseline in 11 of 13 cells, and the reference budget needed to match the best free method has median 4.0 reference calls (range 1.0–64.0). The observed improvement is comparable to a small number of reference calls and depends on reference-based tuning.

Limitations. V4 is an upper bound over subsets of an enumerated space; it does not bound verifiers that reshape the design space itself. The 25% permissiveness floor is a choice: at lower floors V4’s numbers fall further, but the admitted set stops being a usable design space. The “two features predict discrepancy” premise is library-specific: it was established on the cytometry-bearing libraries and does not hold uniformly across the five. Finally, the free-versus-paid comparison equalises reference calls, not wall-clock; a fitted discrepancy model also costs fitting time that we do not charge it.

Adaptive allocation and an in-flight sentinel

Method. We compare 16 allocation policies at matched budget on every (library, topology, reference, budget) cell. The adaptive methods are MF-GP-UCB [10], BOCA [11], an approximate implementation of rMFBO [17], and multi-fidelity best-arm identification [24]. The comparators include successive halving [8], Hyperband [Li et al., 2018], three ϵ\epsilon-interleaving controls, the discrepancy-learning arms evaluated in Appendix § 22, and the two degenerate anchors cheap_only and reference_only.

Results. cheap_only incurs mean regret fraction 1.163, while the other policies range from 0.030 to 0.557. The assumption-free successive-halving floor (both elimination rates) is significantly better than cheap-only in 990 of 1152 cells (85.9%); the four adaptive multi-fidelity methods are better in 1847 of 2304 cells (80.2%). At the budgets where the comparison is meaningful (≥128\ge128 reference calls), the best adaptive policy differs from the best non-adaptive control by at most 0.106 normalized regret across all nine (library, reference) pairs, and by 0.000 in five. At these budgets, the best simple policy is one of reference_only, cheap_then_reference or a discrepancy arm.

Regret against budget for each allocation policy

Figure 10. Regret against budget for each allocation policy. The adaptive multi-fidelity methods and the best simple policies have similar frontiers.

Inferred bias and enumerated regret. Four policies estimate the cheap simulator’s fidelity bias during optimization. Across two reference axes, the largest correlation between this estimate and enumerated fidelity regret is ∣ρs∣=0.293|\rho_s| = 0.293, with minimum p=0.271p = 0.271; all four quality-axis correlations are negative.

With 16 cells on the quality axis and 20 on the burden axis, the design has 80% power only for ∣r∣≥0.651|r| \ge0.651 and ∣r∣≥0.591|r| \ge0.591 respectively; the largest correlation observed anywhere is 0.293. The strongest single case (MF-GP-UCB, quality axis, ρs=−0.293\rho_s = -0.293, n=16n = 16) has a 95% Fisher interval of [−0.689,+0.237][-0.689, +0.237], which includes moderate correlations of either sign. The data exclude a strong observed association at this scale but do not establish independence. Consequently, the inferred bias estimate does not provide an empirically validated replacement for the pre-run screen.

In-flight sentinel. We evaluate a sentinel that spends a small fraction of the budget re-scoring a sample of the optimizer’s current top-mm designs at the reference level and raises an alarm when rank agreement falls below τ\tau (Figure 11). At the best operating point of each of the nine (library, reference) cells, detection rate ranges from 0.143 to 0.683 at false-alarm rates at or below 0.200, for a cost between 0.09% and 11.57% of a full reference sweep.

Sentinel detection rate against cost across library, topology, reference, and budget cells

Figure 11. Sentinel detection rate against cost, as a fraction of a full reference sweep.

Converting detection into recovered regret gives a positive net gain in 7 of 9 cells, up to +0.407, and a negative gain in 2: −0.056 for Bth1C1G1T1/L3 and −0.022 for SC1C1G1T1/L2. In these two cells, allocating the sentinel calls to optimization would yield lower regret.

Cost-model sensitivity. Sweeping the price of a reference call gives a median rank Spearman correlation of 0.996 (minimum 0.732) with the baseline ordering. The top policy changes in 19 of 272 sweep points (7.0%), so broad rankings are stable while the optimum remains price-dependent.

Limitations. The four adaptive methods are re-implementations against their published descriptions rather than the authors’ code, and the rMFBO comparator is approximate. Negative results for these implementations are therefore weaker evidence than results obtained with the authors’ implementations. The design space here is the enumerated space, which is smaller and more structured than the spaces these methods target. Budgets are in reference-call equivalents under a modelled cost, not measured wall-clock. The sentinel’s alarm threshold is tuned per cell on the same data it is evaluated on, so its detection rates are optimistic.

A decision procedure for practitioners

Method. We define two decision rules for a cheap simulator, an available reference assay, and a fixed budget, and validate them on all 360 (library, topology, reference, budget) cells. Rule-T spends a fixed fraction α=0.34\alpha= 0.34 of the budget on reference evaluations of the cheap simulator’s top designs, then re-ranks. Rule-S first spends a diagnostic sample to estimate the cheap–reference rank correlation ρ\rho on the target reference axis, then either re-ranks, uses the reference alone, or accepts the cheap pick. Both are compared against always-cheap, always-reference, and two leave-one-out oracles that bound what any selection rule could achieve.

Results. Mean excess regret over the per-cell best achievable is 0.016 for Rule-T and 0.017 for Rule-S, against 0.305 for always-cheap and 0.034 for always-reference. The leave-one-out selection rules reach 0.014 (leave-one-library-out) and 0.010 (leave-one-axis-out), placing Rule-T within 0.002–0.006 of these reference-informed comparators. Rule-T beats always-cheap in 330 of 360 cells; against always-reference it wins in 161 and ties in 37, for 198 cells with equal or lower regret.

Performance varies by axis. Mean excess regret for Rule-T is 0.004 (L2), 0.002 (L2 d′), 0.014 (L3), and 0.038 (L4), compared with 0.170, 0.189, 0.447, and 0.363 for always-cheap. The largest absolute reduction occurs on the burden axis.

Axis transfer. Rule-S must estimate correlation on the target reference axis. Running the diagnostic on quality and acting on burden costs up to 0.355 normalized regret and is worse than diagnosing correctly in 35 of 80 cells. The worst cases are exactly where the two axes disagree: Eco2C1G3T1/and_k3 has ρL2=+0.696\rho_{L2} = +0.696 against ρL3=−0.386\rho_{L3} = -0.386, and SC1C1G1T1/and_k3 has

Figure 12. The decision procedure and its validation against the enumeration.

ρL2=+0.742\rho_{L2}=+0.742 against ρL3=−0.000\rho_{L3}=-0.000. Rule-T requires no diagnostic and beats naive axis transfer in 45 of 80 rows; it is worse in the remaining 35 by up to 0.297. Rule-T is therefore the preferred alternative when a diagnostic can only be collected on a different axis.

Diagnostic cost and convergence. The diagnostic sample costs a median 0.78% of the advised budget and up to 62.5×62.5\times it in the smallest-budget cells (Figure 13). More seriously, the diagnostic is asymptotically wrong, converging to the wrong action as the sample grows, in 63 of 360 cells, and its accuracy degrades with more samples in 59. These results support Rule-T as the default; Rule-S requires an affordable diagnostic on the target axis.

Figure 13. Cost of the Rule-S diagnostic as a fraction of the advised budget, and the cells where it converges on the wrong action.

Limitations. Both rules are validated on the enumeration used to define regret, and no additional library provides an external test. The leave-one-out comparators measure transfer within the available libraries and axes. The value α=0.34\alpha=0.34 was selected on these data. Of Rule-T’s 50 cells above 0.05 excess regret, 26 are on L4 and 14 on L3. Both rules also require access to a reference assay.

Risk-averse objectives for the cheap simulator

Method. We test whether risk-averse functionals of a cheap-simulator ensemble recover the reference optimum. The objectives are CVaR [25], mean–variance, and a distributionally robust objective over a Wasserstein ball [19], at a sweep of risk parameters, on every cell. The Wasserstein inner problem is solved exactly through its dual, and the solver’s limiting cases are verified against the point estimate and the worst case.

Results. Across the 608 rows in which a genuine risk objective is applied (excluding the point estimate and ensemble-mean baselines), 61 improve regret, 137 worsen it, and 410 leave it unchanged. The best change is +0.470 and the worst is −0.732. In the largest library, Eco1C1G1T1, all 152 objective–parameter rows produce a change of 0.000 on both axes because the ensemble members agree on the selected optimum.

Ensemble and risk components. Cheap-ensemble risk measures underperform the naive reward in 3 of 4 training cells. In SC1C1G1T1 × L3, an 8-member CVaR0.05\mathrm{CVaR}_{0.05} acquisition improves recovery from 0.007 to 0.152. Decomposing each selection into two steps (step 1, replacing the point estimate by the ensemble mean; step 2, moving from the ensemble mean to the risk-averse pick) shows the change is almost entirely step 1. At the (library, reference) aggregation, the largest mean step-2 magnitude is 0.043, compared with a largest mean step-1 magnitude of 0.482. Thus the ensemble substitution accounts for the gain in this training cell.

At finer granularity, step 1 is larger in 153 of the 178 rows with ∣Δ∣>0.05|\Delta| > 0.05 (86.0%); mean absolute changes are 0.300 for step 1 and 0.073 for step 2. Across all 608 risk rows, step 2 reaches an absolute change of 0.691, and 86 rows exceed 0.05 from step 2 alone. Of the three (library, reference) cells with a material mean change, Eco1C2G2T2/L2 is step-2 dominated (−0.041 compared with −0.019). The ensemble substitution accounts for most of the aggregate effect, while risk attitude can remain material in individual cells.

Figure 14. Change in normalized regret from each robust objective, relative to the naive point-estimate pick.

Mechanism. Modelled spread has weak association with reference shortfall: the median ∣Spearman⁡(ensemble SD,reference shortfall)∣|\operatorname{Spearman}(\text{ensemble SD},\text{reference shortfall})| over 28 cells is 0.094, and only 4 of 28 exceed 0.3. This weak association limits the ability of risk-averse objectives to identify reference failures.

Limitations. The ensemble is built by perturbing the cheap simulator’s parameters, so it represents parametric uncertainty. Structural misspecification is absent from this uncertainty model, which may explain the uninformative spread. Ensembles that represent structural uncertainty may behave differently. The risk parameter sweep is finite. Bth1C1G1T1 is excluded from this analysis because it lacks the cytometry axes the ensemble is built on.

Relationship to multi-fidelity optimization and reward misspecification

Multi-fidelity optimization uses correlated low- and high-fidelity evaluations to reduce the cost of finding a high-fidelity optimum [12, 5, 10, 11]. Fidelity regret evaluates a prerequisite for that strategy: whether optimizing the low-fidelity source selects a high-quality design under the reference. Exact combination-subspace enumeration measures this relationship before optimizer behavior is introduced. Appendix § 27 then compares adaptive multi-fidelity methods with assumption-light allocation rules on the same spaces.

Lee et al. study screening regret by ranking candidates with a low-fidelity score, evaluating the top candidates at high fidelity, and reporting their shortfall as a diagnostic for multi-fidelity Bayesian optimization on molecular and materials benchmarks [14]. Their paper presents this diagnostic without a formal definition or range normalization. Fidelity regret belongs to the same family, and range normalization is our reporting convention. Rule-T is a budgeted form of their surrogate-free screening baseline; our contribution is its exact validation against the enumeration and the axis-dependence finding.

Bandit and best-arm-identification methods [8, 24] (Li et al. 2018) motivate the successive-halving and direct reference-allocation comparators. They treat high-fidelity calls as a limited sampling resource and require fewer structural assumptions about the discrepancy. The decision rule in Appendix § 28 follows this perspective by reserving a fixed budget fraction for reference evaluation of high-ranked cheap candidates.

Distributionally robust optimization motivates the CVaR, mean–variance, and Wasserstein-ball objectives [25, 19] in Appendix § 29. These objectives address uncertainty represented by the cheap-simulator ensemble. Their effectiveness depends on ensemble spread predicting reference shortfall, which is weak in the evaluated cells.

Reward hacking and Goodhart-style proxy optimization describe failures caused by optimizing a proxy that diverges from the intended objective [1, 28, 16]. This paper measures the established proxy–reference shortfall exactly on an enumerable scientific-design space. Its distinction from learned reward-model overoptimization [6] is that the misspecification is fixed by the simulator pair and can be measured before policy training.

References

  1. [1]D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in AI safety. arXiv:1606.06565, 2016.arxiv.org/abs/1606.06565
  2. [2]H.-G. Beyer and H.-P. Schwefel. Evolution strategies: A comprehensive introduction. Natural Computing, 1(1):3–52, 2002.DOI
  3. [3]Y. Chen, S. Zhang, E. M. Young, T. S. Jones, D. Densmore, and C. A. Voigt. Genetic circuit design automation for yeast. Nature Microbiology, 5:1349–1360, 2020.DOI
  4. [4]N. Flynn. GenCircuit-RLI: Reinforcement learning from hierarchical verification for genetic circuit design. arXiv:2605.14215, 2026.arxiv.org/abs/2605.14215
  5. [5]A. I. J. Forrester, A. Sóbester, and A. J. Keane. Multi-fidelity optimization via surrogate modelling. Proceedings of the Royal Society A, 463(2088):3251–3269, 2007.DOI
  6. [6]L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10835–10866, 2023.arxiv.org/abs/2210.10760
  7. [7]A. C. Hindmarsh. ODEPACK, a systematized collection of ODE solvers. In R. S. Stepleman et al., editors, Scientific Computing, volume 1 of IMACS Transactions on Scientific Computation, pages 55–64. North-Holland, 1983.
  8. [8]K. Jamieson and A. Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pages 240–248, 2016.
  9. [9]T. S. Jones, S. M. D. Oliveira, C. J. Myers, C. A. Voigt, and D. Densmore. Genetic circuit design automation with Cello 2.0. Nature Protocols, 17(4):1097–1113, 2022.DOI
  10. [10]K. Kandasamy, G. Dasarathy, J. B. Oliva, J. Schneider, and B. Póczos. Gaussian process bandit optimisation with multi-fidelity evaluations. In Advances in Neural Information Processing Systems 29, pages 992–1000, 2016.
  11. [11]K. Kandasamy, G. Dasarathy, J. Schneider, and B. Póczos. Multi-fidelity Bayesian optimisation with continuous approximations. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1799–1808, 2017.arxiv.org/abs/1703.06240
  12. [12]M. C. Kennedy and A. O’Hagan. Bayesian calibration of computer models. Journal of the Royal Statistical Society: Series B, 63(3):425–464, 2001.DOI
  13. [13]N. Lambert et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv:2411.15124, 2024.arxiv.org/abs/2411.15124
  14. [14]J. Lee, E. Errington, C. D. Lorenz, and M. Guo. Transfer learning architectures for scalable multi-fidelity Bayesian optimization. arXiv:2607.23404, 2026.arxiv.org/abs/2607.23404
  15. [16]D. Manheim and S. Garrabrant. Categorizing variants of Goodhart’s law. arXiv:1803.04585, 2018.arxiv.org/abs/1803.04585
  16. [17]P. Mikkola, J. Martinelli, L. Filstroff, and S. Kaski. Multi-fidelity Bayesian optimization with unreliable information sources. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 7425–7454, 2023.arxiv.org/abs/2210.13937
  17. [18]T. Mitchell, J. Beal, and B. Bartley. pySBOL3: SBOL3 for Python programmers. ACS Synthetic Biology, 11(7):2523–2526, 2022.DOI
  18. [19]P. Mohajerin Esfahani and D. Kuhn. Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations. Mathematical Programming, 171:115–166, 2018.arxiv.org/abs/1505.05116
  19. [20]A. A. K. Nielsen et al. Genetic circuit design automation. Science, 352(6281):aac7341, 2016.DOI
  20. [21]A. Pan, K. Bhatia, and J. Steinhardt. The effects of reward misspecification: mapping and mitigating misaligned models. International Conference on Learning Representations, 2022. arXiv:2201.03544.arxiv.org/abs/2201.03544
  21. [22]Y. Park, A. Espah Borujeni, T. E. Gorochowski, J. Shin, and C. A. Voigt. Precision design of stable genetic circuits carried in highly-insulated E. coli genomic landing pads. Molecular Systems Biology, 16(8):e9584, 2020.DOI
  22. [23]L. R. Petzold. Automatic selection of methods for solving stiff and nonstiff systems of ordinary differential equations. SIAM Journal on Scientific and Statistical Computing, 4(1):136–148, 1983.DOI
  23. [24]R. Poiani, A. M. Metelli, and M. Restelli. Multi-fidelity best-arm identification. In Advances in Neural Information Processing Systems 35, pages 17857–17870, 2022.DOI
  24. [25]R. T. Rockafellar and S. Uryasev. Conditional value-at-risk for general loss distributions. Journal of Banking & Finance, 26(7):1443–1471, 2002.DOI
  25. [26]Z. Shao et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
  26. [27]J. Shin, S. Zhang, B. S. Der, A. A. K. Nielsen, and C. A. Voigt. Programming Escherichia coli to function as a digital display. Molecular Systems Biology, 16(3):e9401, 2020.DOI
  27. [28]J. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger. Defining and characterizing reward gaming. In Advances in Neural Information Processing Systems 35, pages 9460–9471, 2022.arxiv.org/abs/2209.13085
  28. [29]M. Taketani, J. Zhang, S. Zhang, A. J. Triassi, Y.-J. Huang, L. G. Griffith, and C. A. Voigt. Genetic circuit design automation for the gut resident species Bacteroides thetaiotaomicron. Nature Biotechnology, 38:962–969, 2020.DOI
  29. [30]B. Trabucco, A. Kumar, X. Geng, and S. Levine. Conservative objective models for effective offline model-based optimization. International Conference on Machine Learning, 2021. arXiv:2107.06882.arxiv.org/abs/2107.06882
  30. [31]R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229–256, 1992.

Paper details

vlnc:W_jncahv6d
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents