betaFeedback

WorksComputer Science

Does reflective LLM feedback survive a wet-lab-scale label budget?

Abstract

Can reflective language-model feedback improve a biological predictor when each cycle buys only one plate of labels? We separate three possible sources of benefit: proposing model edits, choosing measurements, and revising decisions through repeated feedback. We replay 3636 ProteinGym assays, starting with 9696 labels and acquiring 9696 per round for six rounds, with a sealed test set and a shared edit schema. The pre-specified comparison of full LLM control against random edits detects no advantage in test rank correlation (Δz=−0.008Δz=-0.008, 95%95\% CI [−0.034,+0.018][-0.034,+0.018]); equivalence is also unestablished. Exploratory ablations nevertheless identify useful model editing. With random acquisition fixed, LLM edits improve Fisher-transformed test Spearman by 0.0430.043 over no edits; with acquisition fixed to upper confidence bound, they improve normalized top-variant recovery by 0.0230.023. LLM acquisition alone does not detectably outperform fixed adaptive rules, and repeated feedback does not detectably outperform one initial recommendation under full control. Thus, the observed gains support constrained specialist editing, while the incremental value of recursive reflection remains unestablished at this label budget.

Introduction

An agent for biological discovery must decide both how to improve its predictor and what to measure next. A general-purpose large language model (LLM) could coordinate these decisions by reading a specialist’s diagnostics, proposing changes, and revising them after new measurements. In a small experimental campaign, however, the same scarce labels must support training and evaluation. As acquisition changes which variants are labeled, it also changes the evidence used to judge the next model edit. A better final predictor therefore need not imply that repeated reflection was responsible for the improvement.

We examine this distinction in protein-fitness prediction with batches of 96 labels, matching the nominal capacity of one assay plate. The LLM can edit a supervised predictor’s data processing, features, training recipe, and acquisition rule. Only the specialist is refitted: the generalist and protein encoder remain frozen. This setting tests a component of feedback-driven specialist improvement, rather than an agent that updates its own weights. We ask whether LLM proposals help, whether control of acquisition adds value, and whether repeated feedback improves on a single recommendation.

Across 36 assays, full control shows no detected advantage over random edits on the pre-specified ranking endpoint. Exploratory comparisons locate positive results in a narrower role: model editing under a prescribed acquisition rule. The benefit depends on the objective—global ranking or recovery of high-fitness variants—and does not establish an incremental benefit from repeated feedback. This decomposition provides a practical way to evaluate claims of biological recursive self-improvement without conflating proposal quality, acquisition, and reflection.

Preprint.

Related Work

Biological prediction and experiment selection. Combining evolutionary information with measured fitness supports learning from limited assay data [2]. EVOLVEpro couples protein-language-model features with iterative experimental selection [4], while BioDiscoveryAgent uses LLM reasoning to propose genetic perturbations [10]. Outside biology, label-efficient acquisition has been studied for optical-flow training [15] and industrial multimodal content understanding, where kNN-based Latent Space Broadening guides sample selection [11]. We evaluate the added value of generalist-guided specialist editing against fixed acquisition rules and non-LLM editing controls.

Model-based diagnosis has also been studied in biological workflows: Li uses experimental outcomes and records of reagents and technicians to identify sources of failure [5]. Our study instead examines whether diagnostic feedback improves protein-fitness prediction under a restricted assay-label budget.

Feedback-driven improvement. OPRO uses previous candidates and scores to guide optimization [14]; GEPA uses textual feedback to evolve prompts [1]. The Darwin Gödel Machine extends self-improvement to an agent’s code [16]. In a different domain, constrained integration embeds foundation-model predictions in discrete-choice models while preserving economic structure [12]. Here, the editable object is a supervised biological predictor with a bounded configuration space. Its evaluation signal comes from the same purchased labels used for training, and adaptive acquisition changes their distribution. We therefore evaluate repeated feedback separately from the ability to propose a useful initial configuration.

Resource-constrained agents and biomedical data processing. CE-Agent uses dynamic cloud–edge offloading and trajectory distillation to balance agent capability with latency and communication costs [3]. In clinical machine learning, Liang et al. [2025] use an LLM to extract treatment adherence from clinical notes and study how non-adherence affects causal estimates and predictive models. These applications motivate separating useful LLM-assisted operations from the added value of repeated reflection. Our study focuses on the cost of acquiring assay labels that both train the predictor and evaluate proposed edits.

Experimental Setup

Assays, features, and label budget. We replay existing single-substitution measurements from ProteinGym [9]. A filter fixed before the main results selects 36 assays and 102 variants: 1,500–6,000 single mutants, protein length ≤400\le400, and one assay per UniProt accession. Features comprise frozen ESM-2 150M mean embeddings [7], a masked-marginal zero-shot score [8], and optional substitution one-hot encodings. Each run starts with 96 labels and acquires 96 in each of six rounds, ending at 672. The primary split seals 25% of variants for testing; a harder split instead holds out 25% of residue positions. Test labels and scores never enter controller feedback. This is an offline replay, with no new wet-lab measurements.

Controller and acceptance rule. Five-fold cross-validation (CV) on purchased labels supplies the only selection signal. The full controller receives CV Spearman and a bootstrap interval, stratified errors, uncertainty calibration, the current configuration, and edit history. It proposes changes to 15 fields covering caption, pseudo-labeling, features, training, and acquisition. A shared validator rejects unknown fields and illegal categories. Non-acquisition edits are kept only if CV Spearman improves; acquisition edits bypass this gate for every arm because CV on existing labels cannot evaluate an unmeasured batch. The main controller uses the configured Opus 5 identifier without external tools; identifiers and prompts are detailed in Appendix A.

Controls and ablations. No edits refits the default ridge predictor with random acquisition. Random edits matches a frozen pilot’s distribution of LLM edit sizes; local search uses inner CV with nominal 1×1 \times or 5×5 \times LLM-call wall time, the latter capped at 24 evaluations. A static augmented ridge recipe supplies another non-LLM baseline. To separate model editing from measurement selection, we fix acquisition to random sampling or upper confidence bound (UCB), or let the LLM edit acquisition alone. Fixed greedy, UCB, and diversity-greedy rules provide acquisition controls. To test feedback, we compare the full controller with scalar-only diagnostics and with one recommendation in round 1, followed by retraining under its gated configuration. Additional controls appear in the appendix.

Endpoints and inference. The primary endpoint averages sealed-test Fisher z=arctanh⁡(ρ)z = \operatorname{arctanh}(\rho), where ρ\rho is Spearman correlation, over rounds 3–6. The secondary endpoint averages normalized top-variant recovery, R96=∣P96∩Tm∣/min⁡(96,m)R_{96} = \lvert P_{96} \cap T_m\rvert/ \min(96,m), over the same rounds: P96P_{96} contains the 96 highest predictions and TmT_m the true top m=round⁡(0.1ntest)m = \operatorname{round}(0.1n_{\mathrm{test}}) variants. This equals recall when m≤96m \le96 and precision when m>96m > 96. Both endpoints score unseen test variants, not the acquired batch. We average seeds 0–2 within each assay, then form paired tt intervals across 36 assays. Splits and random streams are shared across arms; adaptive policies can still acquire different labels. Full LLM versus random edits on the random split is the pre-specified primary contrast, with a 0.02 equivalence margin in zz. Other comparisons are exploratory, with Benjamini–Hochberg (BH) correction within each split, seed set, and endpoint family. The random split was selected after a four-assay pilot, search was redesigned after partial results, and an earlier grid with incorrectly keyed random streams was discarded.

Results

Full control shows no detected gain on the primary ranking endpoint. Full LLM minus random edits is −0.008-0.008 Fisher zz (95% CI [−0.034,+0.018][-0.034,+0.018], p=0.55p = 0.55; 20/36 assay wins). Equivalence is also unestablished at the pre-specified margin (two one-sided tests, p=0.17p = 0.17). Thus, neither superiority nor equivalence is established. The secondary endpoint differs: R96R_{96} increases by 0.032 ([0.014,0.049][0.014,0.049], q=0.003q = 0.003). This divergence motivates examining which decisions help each objective (Figure 1).

Figure 1. How model editing, acquisition, and feedback affect the two objectives. (a,b) Improvement over no edits as labels accumulate; shading gives pointwise 95% intervals across 36 assays, with seeds 0–2 averaged within assay. The gray window marks rounds 3–6 used for endpoint analysis. (c,d) Paired endpoint differences and 95% tt intervals; positive values favor the first arm. Only the marked ranking comparison is confirmatory. LLM + random/UCB fixes acquisition while allowing model edits.

ControllerAcquisitionMean zzR96R_{96}
No editsRandom0.6030.349
Random editsEditable0.5970.359
Local search (5×)Editable0.6230.354
Full LLMEditable0.5900.391
LLM, one callChosen once0.6000.390
LLM model editsRandom0.6470.358
No editsUCB0.5890.383
LLM model editsUCB0.5760.406
LLM acquisition onlyEditable0.5910.377

Table 1. Random split, 108 runs per arm, averaged over rounds 3–6. Acquisition is prescribed unless marked editable or chosen once. R96R_{96} is normalized top-variant recovery; Appendix C reports all 20 complete arms.

Model edits help when acquisition is prescribed. With random acquisition fixed, LLM model edits gain 0.043 in zz over no edits ([0.027, 0.059], q<10−3q < 10^{-3}; 31/36 wins) and 0.024 over nominal 5× search ([0.007, 0.040], q=0.019q = 0.019). They also exceed full control by 0.057 ([0.033, 0.081]). This ranking gain does not translate into a detected recovery gain over no edits: ΔR96=0.009\Delta R_{96} = 0.009 ([−0.005, +0.024]). Under fixed UCB, the pattern reverses: model editing improves R96R_{96} by 0.023 over UCB alone ([0.009, 0.037], q=0.005q = 0.005), while its ranking difference is inconclusive (Δz=−0.013\Delta z = -0.013, [−0.037, +0.010]). Prescribed acquisition therefore enables useful editing, but the benefit is specific to the evaluation objective.

LLM acquisition alone adds no detected benefit over fixed rules. Fixed adaptive policies achieve R96=0.380R_{96} = 0.380–0.3840.384, compared with 0.349 for random acquisition without edits. Acquisition-only LLM control reaches 0.377, with no detected advantage over fixed UCB (ΔR96=−0.006\Delta R_{96} = -0.006, [−0.013, +0.001], q=0.114q = 0.114). Moreover, switching the model-editing arm from random acquisition to UCB increases recovery by 0.047 ([0.032, 0.062]) while reducing ranking by 0.071 ([0.046, 0.095]). The full loop’s recovery improvement therefore cannot be taken as evidence of better global ranking, or attributed to a uniquely effective LLM acquisition policy.

The separation persists across acquisition rounds. Fixed-random editing improves mean ranking at every post-initial round, reaching Δz=0.056\Delta z = 0.056 at 672 labels (Figure 1a,Figure 1b). Fixed-UCB editing instead reaches ΔR96=0.064\Delta R_{96} = 0.064 versus no edits. These trajectories combine editing with additional measurements; they do not isolate reflection.

Repeated feedback has no detected advantage over one recommendation. Full LLM minus one-call control is −0.011-0.011 in zz ([−0.028, +0.007]) and +0.0005 in R96R_{96} ([−0.012, +0.013]). Replacing the diagnostic report with a scalar score likewise changes mean ranking by only +0.0006. These results do not establish equivalence, but they provide no evidence that repeated rich feedback explains the full controller’s performance. They also leave open whether feedback helps the successful fixed-acquisition arms, for which a matched one-call ablation was not run.

The feedback signal depends on which labels are acquired. A post-hoc configuration screen at 672 random labels finds a CV-selected gain of 0.072 in test zz over the default (32/36 assays), showing that the main null result does not reflect an already-optimal starting recipe. Yet the CV–test relationship varies with acquisition: the mean CV-minus-test gap is −0.032 for no edits and −0.184 for fixed diversity-greedy acquisition. These negative gaps indicate lower CV than test scores, not optimistic CV bias. They motivate investigating distribution-aware feedback, but do not establish that distribution shift causes the full controller’s ranking loss. The screen and diagnostics are descriptive rather than matched causal tests (Appendix B).

Controller and split sensitivity qualify the findings. With the legal-value menu omitted, Opus 5 exceeds Sonnet 5 by 0.035 in zz ([0.026, 0.044], q=7.1×10−8q = 7.1 \times10^{-8}; 35/36 wins) and Haiku 4.5 by 0.030 ([0.019, 0.040], q<10−3q < 10^{-3}). These are deployed-controller comparisons: schema rejections and failed calls also affect behavior (Appendix B). On the position-disjoint split, full LLM versus random edits remains inconclusive (Δz=0.023\Delta z = 0.023, [−0.003,+0.048][-0.003, +0.048]); random-fold CV does not match this test distribution. Five-seed sensitivity retains an inconclusive primary ranking contrast and a positive fixed-UCB editing effect on recovery (Appendix § 9).

Conclusion and limitation

At 96 labels per round, LLM-guided specialist editing can help under a prescribed acquisition rule, but the observed benefit depends on whether the goal is global ranking or top-variant recovery. The full controller shows no detected ranking advantage over random edits, and repeated feedback shows no detected incremental value over one recommendation. These findings support evaluating model proposals, acquisition, and reflection as separate contributions. Metric-dependent trade-offs also arise in credit-card fraud detection, where stronger amount weighting can improve dollar recall while reducing ranking quality and dollar precision [13].

The positive ablations are exploratory. Our evidence is limited to offline single-substitution assays, one frozen encoder, a bounded edit schema, and six acquisition rounds; it does not establish gains in prospective wet-lab discovery. Only one label budget is tested, so a causal budget threshold cannot be inferred. Repeated selection on purchased-label CV and changing acquisition distributions may affect the gate, but the present comparisons do not isolate that mechanism. Matched one-call controls for the fixed-acquisition arms and evaluation under distribution-aware selection are the next tests needed to establish the value of reflection.

Protocol and reproducibility

Available evidence and reconstruction. All reported statistics come from a frozen results snapshot containing per-round and per-run endpoints, supporting diagnostic summaries, and a provenance manifest. The snapshot cutoff is 27 September 2026, 05:04:52 UTC. An analysis script reconstructs the paired comparisons, figures, and tables from the exported endpoint summaries without modifying the snapshot, and a separate validation script checks input hashes, label budgets, round completeness, common initial endpoints, endpoint aggregation, and agreement with the supplied comparison statistics. Raw assay labels, embeddings, complete run records, and controller-call logs are not part of this summary snapshot; the original server audit cannot be rerun from these summaries alone.

An arm is eligible only if every one of the 36 assays has all requested seeds and all seven recorded states (initialization and rounds 1–6). Eligibility does not depend on effect size, significance, or direction. The snapshot contains 4,211 unique completed run summaries and 29,477 round records. The three-seed analyses include 20 random-split arms and 11 position-split arms. The five-seed analysis includes 11 complete random-split arms; nominal 5× search is excluded there because one of its 180 required runs is absent. All eligible arms are reported below, including controls omitted from the main table.

The original analysis code specifies the primary contrast and equivalence margin. We retain this three-seed test as the primary inference; subsequent ablations and expanded seed analyses remain exploratory. BH correction uses the original declared comparison families: 84 contrasts per endpoint for the three-seed random split, 18 for the position split, and 21 for the five-seed random split. The primary contrast retains its unadjusted inference; all reported qq values use the complete family, including comparisons absent from the main text. Intervals are pointwise 95% paired tt intervals, not simultaneous confidence bands.

Arm definitions. The principal LLM arms share a preamble describing the loop and gate. Full feedback includes CV Spearman with a bootstrap interval, errors by residue position, amino-acid class and fitness quintile, uncertainty calibration, configuration, edit history, and legal action values. Scalar feedback retains the menu but replaces the diagnostic report with CV Spearman. No-menu feedback retains diagnostics and current field names but omits enumerated legal values.

One-call control makes its only proposal in round 1. After validation and the ordinary CV gate, its resulting configuration remains fixed while the predictor is retrained as labels accumulate. No-curation control permits pseudo-labeling, feature, training, and acquisition edits only. Fixed-random and fixed-UCB model-editing arms permit all non-acquisition edits while prescribing the acquisition rule each round. Acquisition-only LLM control changes the acquisition strategy and exploration parameter while retaining the other defaults. Non-LLM acquisition controls prescribe greedy, UCB, or diversity-greedy selection and leave the remaining fields at default. Static ridge uses an augmented recipe inspired by Hsu et al. [2022] and re-tunes regularization.

The configured model names are Opus 5, Sonnet 5, and Haiku 4.5. Their CLI identifiers are claude-opus-5, claude-sonnet-5, and claude-haiku-4-5-20251001, respectively. These names describe the experiment configurations, not independently verified model sizes. The prompt templates and selected response examples are reproduced verbatim in Appendix F. Controller substitutions use the same feedback variants, label budgets, and edit schema.

Endpoint definition. The exported topk field is computed by the evaluation code as follows:

R96=∣P96∩Tm∣min⁡(96,m),m=max⁡(1,round⁡(0.1ntest)).R_{96} = \frac{\lvert P_{96} \cap T_m \rvert}{\min(96,m)}, \qquad m = \max(1,\operatorname{round}(0.1n_{\mathrm{test}})).

This denominator is the maximum possible overlap, so R96R_{96} is bounded by one and equals recall only for m≤96m \le96. Both endpoints use sealed-test predictions; R96R_{96} is not the acquired batch’s measured hit rate. We average rounds 3–6 (384–672 labels), then seeds within assays, giving each assay equal weight.

Diagnostic evidence and interpretation

Available headroom. A post-hoc configuration screen holds a random set of 672 labels fixed, samples 120 configurations per assay, and selects by three-fold CV. Candidate heads are ridge, small MLP, and a ridge ensemble; gradient boosting is excluded for computational cost. Fourteen configurations with nonfinite correlations from constant predictions on CCDB are omitted from the summaries. The CV-selected configuration gains a mean 0.072 test Fisher zz over the default and improves 32/36 assays. Selection uses CV, not test performance. This shows that the default is not already optimal, but different compute, CV folds, and labeled-data trajectories prevent treating the screen as a matched online baseline or a formal upper bound.

Figure 2. Information flow in the budgeted loop. Purchased-label CV informs model edits and the controller. Non-acquisition edits pass an acceptance gate; acquisition follows a separate path and reveals the next 96 labels. The sealed test set is used only for external scoring.

Acquisition and the feedback distribution. In the supplied diagnostics, the mean CV-minus-test Fisher-zz gap over rounds 3–6 is −0.032 for no edits, −0.068 for fixed UCB, −0.184 for diversity-greedy, and −0.152 for full LLM control. A negative gap means CV is lower than sealed-test performance; it is not evidence of optimistically biased CV. The variation is consistent with acquisition-dependent evaluation distributions, but does not identify why full control loses ranking performance. Changing acquisition also changes subsequent training sets and proposals. Testing a distribution-aware gate would require a new controlled comparison.

Schema behavior and controller reliability. The Opus no-menu arm reaches mean z=0.638z = 0.638, versus 0.603 for no edits. However, its validator rejects 640 categorical entries across 509/648 calls, and the arm never changes the default head or feature set. Its apparent benefit therefore cannot isolate the value of removing the menu: the intervention also changes the effective set of edits.

All four Sonnet and Haiku substitution arms cover 36 assays and three seeds. Under full feedback, Opus exceeds each substitute by approximately 0.022 in zz (q=0.019q = 0.019 for Sonnet, q=0.008q = 0.008 for Haiku). Under no-menu feedback, mean zz is 0.638 for Sonnet, 0.602 for Sonnet, and 0.608 for Haiku. Across the two Haiku arms, 190/1,296 scheduled calls fail after wrapper retries; the two Sonnet arms have no failed calls. Among the Opus ablations, there are 26/648 failed calls under fixed random acquisition, 33/648 without curation, and 5/108 for one-call control. Failed calls preserve the current configuration. These comparisons are deliberately deployed systems, combining model behavior, schema compatibility, and execution reliability.

All completed three-seed controls

The tables report the mean Fisher-transformed rank correlation, its difference from no edits, and normalized recovery. Each mean gives equal weight to the 36 assays after averaging seeds 0–2 and rounds 3–6. On the position split, random-fold CV operates on purchased variants while the test set consists of unseen residue positions; the two evaluation distributions consequently differ.

ControllerMean zzΔz\Delta z vs. no editsR96R_{96}
Fixed div.-greedy0.597−0.007-0.0070.384
Fixed greedy0.566−0.038-0.0380.380
Fixed UCB0.589−0.014-0.0140.383
No edits0.603+0.000+0.0000.349
LLM acq. only0.591−0.012-0.0120.377
LLM full0.590−0.014-0.0140.391
Full: Haiku 4.50.568−0.036-0.0360.375
Full: Sonnet 50.568−0.036-0.0360.384
LLM + fixed UCB0.576−0.028-0.0280.406
LLM + random acq.0.647+0.043+0.0430.358
LLM no curation0.598−0.006-0.0060.390
LLM no menu0.638+0.034+0.0340.375
No menu: Haiku 4.50.608+0.005+0.0050.350
No menu: Sonnet 50.602−0.001-0.0010.364
LLM one call0.600−0.003-0.0030.390
LLM scalar0.590−0.013-0.0130.403
Random edits0.597−0.006-0.0060.359
Search 5x0.623+0.020+0.0200.354
Search 1x0.619+0.016+0.0160.352
Static ridge0.599−0.004-0.0040.333

Table 2. Random split: all 20 complete arms, 108 runs per arm. Unqualified LLM arms use the configured Opus 5 controller.

ControllerMean zzΔz\Delta z vs. no editsR96R_{96}
Fixed div.-greedy0.402+0.0000.285
Fixed greedy0.404+0.0030.288
Fixed UCB0.402+0.0010.285
No edits0.401+0.0000.276
LLM full0.461+0.0600.319
LLM no menu0.453+0.0520.293
LLM scalar0.446+0.0450.316
Random edits0.439+0.0370.290
Search 5x0.506+0.1050.313
Search 1x0.499+0.0980.302
Static ridge0.492+0.0910.296

Table 3. Position-disjoint split: all 11 complete arms, 108 runs per arm.

Five-seed sensitivity

Expanding to seeds 0–4 leaves full LLM versus random edits inconclusive for ranking: Δz=−0.012\Delta z = -0.012 (95% CI [−0.034, +0.010]). The fixed-UCB model-editing arm retains a recovery gain over UCB alone, ΔR96=0.020\Delta R_{96} = 0.020 ([0.007, 0.033], q=0.005q = 0.005). This is a post-hoc sensitivity analysis, not a replacement for the three-seed primary test. The fixed-random editing and one-call arms lack complete five-seed grids, so this analysis does not establish their robustness to additional seeds.

ControllerMean zzΔz\Delta z vs. no editsR96R_{96}
Fixed div.-greedy0.591−0.0110.383
Fixed greedy0.564−0.0380.382
Fixed UCB0.584−0.0180.385
No edits0.602+0.0000.354
LLM full0.587−0.0150.392
LLM + fixed UCB0.573−0.0290.405
LLM no menu0.635+0.0320.375
LLM scalar0.584−0.0190.402
Random edits0.599−0.0030.360
Search 1x0.614+0.0120.352
Static ridge0.596−0.0060.334

Table 4. All 11 complete random-split arms with seeds 0–4, 180 runs per arm, averaged over rounds 3–6. Eligibility follows Appendix § 6.

Action space and assay list

The schema rejects unknown fields and illegal categories and clamps out-of-range numbers. Acquisition comprises the selection rule and its exploration parameter; the remaining groups pass through the shared CV gate. The schema and defaults below reproduce the original settings.

FieldAllowed valuesDefault
Curation
drop_outliers.methodnone, iqr, zscorenone
drop_outliers.k[1.0, 4.0] (real)3.0
dedup_by_positionFalse, TrueFalse
reweightnone, inverse_zs_confidence, fitness_decile_balancednone
clip_targetsnone, winsorize_1, winsorize_5none
Pseudo-labeling and features
pseudo_labelnone, zero_shot_topk, self_training_confidentnone
n[0, 2000] (integer)0
feature_setemb, emb+zs, emb+zs+onehot, zs+onehotemb+zs
Acquisition
strategyrandom, greedy, ucb, ensemble_disagreement, diversity_greedy, zero_shot_priorrandom
beta[0.0, 3.0] (real)1.0
Training recipe
headridge, mlp_small, ensemble_ridge, gbrridge
lossmse, huber, rankmse
alpha[0.001, 1000.0] (real)1.0
epochs[20, 200] (integer)200
standardizeFalse, TrueTrue

Table 5. Allowed edits and default values. Numeric ranges are inclusive.

Assays

The pre-main-result filter selects the following 36 assays, totaling 101,201 single-substitution variants. Virus names are abbreviated for readability; the full taxonomic strings are given in the ProteinGym reference table. Fitness denotes the OrganismalFitness readout category.

AssayLengthVariantsReadoutOrganism
A0A247D711_LISMN_Stadelmann_2021871,653ActivityListeria monocytogenes
A4GRB6_PSEAI_Chen_20202665,004FitnessPseudomonas aeruginosa
AACC1_PSEAI_Dandage_20181771,801FitnessPseudomonas aeruginosa
BLAT_ECOLX_Firnberg_20142864,783FitnessEscherichia coli
CALM1_HUMAN_Weile_20171491,813FitnessHomo sapiens
CASP3_HUMAN_Roychowdhury_20202581,567ActivityHomo sapiens
CASP7_HUMAN_Roychowdhury_20202811,680ActivityHomo sapiens
CCDB_ECOLI_Tripathi_20161011,663FitnessEscherichia coli
DYR_ECOLI_Thompson_20191592,363FitnessEscherichia coli
ESTA_BACSU_Nutschel_20202122,172StabilityBacillus subtilis
HEM3_HUMAN_Loggerenberg_20233615,689ActivityHomo sapiens
KCNE1_HUMAN_Muhammad_2023_function1292,315ActivityHomo sapiens
KKA2_KLEPN_Melnikov_20142644,960FitnessKlebsiella pneumoniae
MET_HUMAN_Estevam_20232875,393ActivityHomo sapiens
MLAC_ECOLI_MacRae_20232114,007FitnessEscherichia coli
MTH3_HAEAE_RockahShmuel_20153301,777FitnessHaemophilus aegyptius
NUD15_HUMAN_Suiter_20201642,844ExpressionHomo sapiens
OTC_HUMAN_Lo_20233541,570ActivityHomo sapiens
P84126_THETH_Chan_20172541,519FitnessThermus thermophilus
PHOT_CHLRE_Chen_20231182,122ActivityChlamydomonas reinhardtii
Q53Z42_HUMAN_McShan_2019_binding-TAPBPR3653,344BindingHomo sapiens
R1AB_SARS2_Flynn_20223065,725FitnessSARS-CoV-2
RASH_HUMAN_Bandaru_20171893,134ActivityHomo sapiens
RASK_HUMAN_Weng_2022_abundance1883,066ExpressionHomo sapiens
REV_HV1H2_Fernandes_20161162,147FitnessHIV-1 (HXB2)
RNC_ECOLI_Weeks_20232264,277ActivityEscherichia coli
SERC_HUMAN_Xie_20233701,914FitnessHomo sapiens
SUMO1_HUMAN_Weile_20171011,700FitnessHomo sapiens
SYUA_HUMAN_Newberry_20201402,497FitnessHomo sapiens
TAT_HV1BR_Fernandes_2016861,577FitnessHIV-1 (BRU/LAI)
TPK1_HUMAN_Weile_20172433,181FitnessHomo sapiens
TPMT_HUMAN_Matreyek_20182453,648ExpressionHomo sapiens
TRPC_SACS2_Chan_20172481,519FitnessThermus thermophilus
TRPC_THEMA_Chan_20172521,519FitnessThermus thermophilus
UBC9_HUMAN_Weile_20171592,563FitnessHomo sapiens
VKOR1_HUMAN_Chiasson_2020_abundance1632,695ExpressionHomo sapiens

Table 6. Assay metadata. Length counts residues; variants are single substitutions.

Controller prompts and example responses

Prompt templates

Verbatim templates. Each call concatenates the shared preamble, the diagnostic report (or, under scalar feedback, CV Spearman alone), the current configuration, the action menu (full and scalar feedback only) and the instruction for its variant.

Shared preamble (all principal LLM arms).

You are improving a small supervised model that predicts a measured fitness value for
single amino-acid substitutions in one protein, from a public deep mutational scanning dataset.

The loop works like this. A fixed set of variants has hidden measurements. Each round you may change
the data curation, the pseudo-label generation, the training recipe, and the rule that picks which 96
still-unmeasured variants are measured next. The model is then retrained on all measured labels and
re-evaluated by 5-fold cross-validation over the labels purchased so far. Your edit is kept only if
that cross-validated Spearman improves; otherwise it is reverted. Changes to the acquisition rule are
always applied, because a batch of measurements cannot be un-ordered.

You never see the held-out test set and you never see test metrics. Optimizing the validation number
by exploiting its noise will not help: the reported outcome is measured on data you cannot see.

Action menu (full and scalar feedback; withheld under no-menu feedback).

You may change any subset of the following configuration fields. These are the ONLY legal
fields and the ONLY legal values. Anything else is rejected.

curation.drop_outliers.method : one of "none", "iqr", "zscore"
curation.drop_outliers.k      : float in [1.0, 4.0]        (threshold for the chosen method)
curation.dedup_by_position    : true or false              (keep at most one labeled variant per residue
                                                           position)
curation.reweight            : one of "none", "inverse_zs_confidence", "fitness_decile_balanced"
curation.clip_targets        : one of "none", "winsorize_1", "winsorize_5"
generation.pseudo_label      : one of "none", "zero_shot_topk", "self_training_confident"
generation.n                 : int in [0, 2000]            (number of pseudo-labeled variants to add)
generation.feature_set       : one of "emb", "emb+zs", "emb+zs+onehot", "zs+onehot"
acquisition.strategy         : one of "random", "greedy", "ucb", "ensemble_disagreement",
                               "diversity_greedy", "zero_shot_prior"
acquisition.beta             : float in [0.0, 3.0]         (exploration weight for ucb)
recipe.head                  : one of "ridge", "mlp_small", "ensemble_ridge", "gbr"
recipe.loss                  : one of "mse", "huber", "rank"  ("huber" applies to the linear head;
                                                               "rank" fits on rank-transformed targets and applies to any head)
recipe.alpha                 : float in [0.001, 1000]      (regularization strength)
recipe.epochs                : int in [20, 100]            (affects only the iterative heads,
                                                               mlp_small and gbr)
recipe.standardize           : true or false

Field meanings: "emb" is a mean-pooled frozen protein language model embedding of the full mutant
sequence; "zs" is that model's zero-shot log-likelihood ratio for the substitution; "onehot" is a
one-hot encoding of the substitution. Acquisition chooses which 96 unlabeled variants get measured
next round.

Instruction, full and scalar feedback.

Decide what to change for the next round. Think about what the report actually shows: whether the
model is overfitting, whether any stratum is systematically worse, whether the uncertainty estimate
is trustworthy enough to drive an uncertainty-based acquisition rule, and whether the remaining label
budget is better spent exploring or exploiting.

Reply with a short justification (at most 5 sentences) and then a single JSON object containing ONLY
the fields you want to change. To change nothing, return {}.

Instruction, no-menu feedback.

Decide what to change for the next round. Think about what the report actually shows: whether the
model is overfitting, whether any stratum is systematically worse, whether the uncertainty estimate
is trustworthy enough to drive an uncertainty-based acquisition rule, and whether the remaining label
budget is better spent exploring or exploiting.

Reply with a short justification (at most 5 sentences) and then a single JSON object containing ONLY
the configuration fields you want to change, using the same structure as the configuration shown
above. To change nothing, return {}.

Representative controller justifications

Verbatim free-text justifications from the Opus 5 controller, three per feedback variant, each the text the model produced before its JSON edit (the JSON is omitted). Headers give assay, seed and round.

Full feedback (full LLM control). AACC1_PSEAI_Dandage_2018, seed 0, round 1

With 96 labels and a high-dimensional PLM embedding, ridge at alpha=1.0 is badly under-regularized—the huge MAEs (1.2–2.4) and near-zero within-stratum correlations look like variance, not a missing feature. Since the scored metric is Spearman and the label distribution is clearly heavy-tailed at the low-fitness end (quintiles 1–2 are near-random), I’m switching to rank-transformed targets with an averaged ridge ensemble and much stronger regularization. The uncertainty estimate barely tracks error (ρ=0.18\rho=0.18), so UCB/disagreement acquisition would be unreliable; instead I’ll use diversity_greedy to spread the next 96 labels across embedding space, which is the right spend this early (1255 left in pool, top-10% recall already saturated). No pseudo-labels yet—self-training off a model this noisy would just amplify its own bias.

A0A247D711_LISMN_Stadelmann_2021, seed 1, round 1

With 96 labels against a high-dimensional PLM embedding, alpha=1.0 is almost certainly under-regularized—the pooled CV Spearman of 0.54 with wide strata CIs looks like variance, not a data problem, so I’d raise regularization sharply and average over an ensemble of ridges for stability. Since the metric is rank-based and the errors concentrate in the compressed top quintiles (MAE ∼0.6 with flat rank correlation there), fitting on rank-transformed targets should align the objective with evaluation better than MSE on a skewed fitness distribution. I’m leaving curation alone: dropping outliers or deduplicating by position would discard scarce labels, and no stratum is significantly worse than pooled, so there’s no systematic defect to patch yet. The uncertainty estimate barely tracks error (Spearman 0.278), so UCB or ensemble-disagreement acquisition would be chasing noise; with 1144 variants left and only 96 measured, this round is still about covering feature space, so I’ll use diversity_greedy rather than exploiting a model I don’t trust. Exploitation can wait until the CV correlation is high enough to justify it.

A4GRB6_PSEAI_Chen_2020, seed 2, round 1

Justification. With 96 labels and high-dimensional embeddings, alpha=1.0 is almost certainly under-regularized—the huge MAE (∼1.5) and the flat/negative Spearman in the middle-upper fitness quintiles look like variance, not bias, so I’m raising alpha sharply and switching to ensemble_ridge for a more stable fit. Since the metric is Spearman, fitting on rank-transformed targets aligns the loss with evaluation and also blunts the influence of the extreme low-fitness tail without discarding data. The uncertainty estimate is useless (Spearman(std, |error|) = 0.009), so UCB and ensemble_disagreement acquisition are off the table this round; with 3657 variants left and only 96 measured, coverage is worth more than exploitation, so diversity_greedy spreads the next batch across sequence/feature space—which should also help the weak position-tercile-0 stratum and the poorly-sampled low-fitness region. I’m holding off on pseudo-labels and outlier dropping until the base model is stable enough to trust.

No-menu feedback. A0A247D711_LISMN_Stadelmann_2021, seed 0, round 1

With 96 labels against high-dimensional embeddings, ridge at alpha=1.0 is almost certainly under-regularized—that fits the modest pooled Spearman plus the near-flat per-stratum profile (no stratum is genuinely worse; all CIs overlap the pool), so the fix is global capacity, not targeted curation. The ensemble uncertainty is uncalibrated (Spearman(std, |error|) = −0.029, and the highest-std bin is not the highest-error bin), so any UCB/max-variance acquisition would be driven by noise and I will not use it this round. The purchased labels are badly skewed toward the top of the fitness range (quintile 4 holds 54 of 96), and that is exactly the worst-resolved stratum at 0.115, so the batch is better spent on coverage than on exploitation—recall@top-10% is already 1.0, meaning there is nothing to gain by chasing the top now. I am therefore raising alpha to 20 and switching acquisition to diversity-based exploration over the 1144 remaining variants to spread labels across positions and mutant AA classes. Everything else stays fixed so the alpha change is cleanly attributable.

A0A247D711_LISMN_Stadelmann_2021, seed 1, round 1

Justification. With 96 labels and high-dimensional embedding features, the model is almost certainly variance-limited, and the flat within-quintile Spearmans (0.16, −0.04 in the top two bins) look like an over-fit head that only recovers the coarse dead-vs-functional split; raising ridge regularization is the highest-expected-value single edit. The ensemble uncertainty tracks error only weakly (rho=0.28, and bin 3’s mean std overshoots its mean error), so an uncertainty/UCB acquisition rule would be steering on noise. With 1144 variants left and many rounds to go, round 2 should still be exploration-weighted, but structured: the random batch left coverage uneven (position tercile 1 and charged/aromatic mutants are the weak strata), so a diversity/coverage-based batch over feature space and position should improve the model more than either random or greedy exploitation. I’m holding off on pseudo-labels and outlier dropping until the head is properly regularized, so the effect of this edit is interpretable.

A4GRB6_PSEAI_Chen_2020, seed 0, round 1

Justification. With 96 labels against a high-dimensional embedding+ZS feature vector, ridge at α=1.0\alpha=1.0 is badly under-regularized—the tell is large MAE (∼1.1\sim1.1–1.61.6) spread evenly across every stratum, and near-zero within-quintile Spearman, i.e. the model only separates dead from functional and fits noise elsewhere. No stratum is reliably worse than the pooled value (22% of CIs excluding it is about what noise alone gives), so there is no case for stratum-targeted curation or reweighting yet. The ensemble uncertainty is uninformative (Spearman(std, |error|) = 0.08), so any UCB/max-variance acquisition would be effectively random with extra bias—I am not using it. With only 96 of 3753 variants labeled and the model still coarse, this round should explore: diversity/coverage acquisition over positions and substitution types gives a better-conditioned design matrix than random draws, and it costs nothing in expectation versus random even if the model is weak. I am therefore raising alpha substantially and switching acquisition to diversity, leaving curation and pseudo-labels untouched so the regularization effect is measurable in isolation.

Scalar feedback (CV Spearman only, with menu). A0A247D711_LISMN_Stadelmann_2021, seed 2, round 1

With only 96 labels against high-dimensional embeddings, the dominant failure mode is variance, not bias—ridge at α=1.0\alpha=1.0 is almost certainly under-regularized, so I’m raising alpha sharply and using an ensemble of ridges for stability. Since the metric is rank correlation and DMS assays have compressed dead-vs-variant tails plus heavy upper noise, fitting on rank-transformed targets with light winsorization should align the objective with evaluation without discarding real signal. I’m leaving the feature set at emb+ZS: one-hot would add hundreds of sparse columns that 96 rows cannot support. For acquisition, model-derived uncertainty from 96 points is not yet trustworthy enough to drive UCB or disagreement sampling, and only ∼8%\sim8\% of the pool is labeled, so this round is better spent on coverage—diversity_greedy to spread across sequence/position space before exploiting. I’ll revisit uncertainty-driven acquisition once ∼200+\sim200+ labels make the ensemble spread meaningful.

A0A247D711_LISMN_Stadelmann_2021, seed 0, round 1

With 96 labels against a high-dimensional PLM embedding, ridge at α=1.0\alpha=1.0 is badly under-regularized—that’s the most likely source of the mediocre 0.378, not data quality, so the biggest single win is a much stronger penalty. Since the metric is Spearman, fitting rank-transformed targets aligns the loss with evaluation and also absorbs the heavy lower tail typical of DMS assays, which makes outlier dropping and winsorizing redundant for now. I’m switching to ensemble_ridge for variance reduction and to start building a usable disagreement signal for later rounds, while keeping emb+zs (one-hot would add parameters we can’t afford yet). For acquisition, 96 labels are far too few for a trustworthy uncertainty estimate, and zero_shot_prior/greedy would concentrate the batch on predicted-high variants and leave the fitness range poorly covered; diversity_greedy buys broad embedding coverage now, and I’ll move to ucb/ensemble_disagreement once the ensemble is calibrated. No pseudo-labels yet—adding 100s of zero-shot labels on top of 96 real ones would let the ZS prior dominate before we know the head is sound.

A4GRB6_PSEAI_Chen_2020, seed 1, round 1 With only 96 labels against a high-dimensional embedding, the dominant risk is variance, not bias: ridge at α=1.0\alpha=1.0 is almost certainly under-regularized, so I’m raising alpha sharply and switching to an averaged ridge ensemble. Since the metric is Spearman and DMS distributions have a heavy dead-variant tail, fitting on rank-transformed targets aligns the loss with the objective and absorbs outliers without discarding data. Uncertainty from a 96-point linear model is not trustworthy enough to drive UCB or disagreement acquisition yet, and the model is still coverage-limited rather than boundary-limited, so this batch should explore. diversity_greedy spreads the next 96 measurements across embedding space, which improves the design matrix for every later round; I’ll revisit uncertainty-driven acquisition once ∼300\sim300 labels make the variance estimates meaningful. No pseudo-labels yet—self-training off a 0.57-Spearman model would mostly inject its own errors.

References

  1. [1]Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alex Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. GEPA: Reflective prompt evolution can outperform reinforcement learning. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=RQm2KQTM5r.arxiv.org/abs/2507.19457
  2. [2]Chloe Hsu, Hunter Nisonoff, Clara Fannjiang, and Jennifer Listgarten. Learning protein fitness models from evolutionary and assay-labeled data. Nature Biotechnology, 40(7):1114–1122, 2022. doi: 10.1038/s41587-021-01146-5.DOI
  3. [3]Yan Huang, Meimei Zhang, Hong Lei, Chenyang Ji, Jiayi Wang, and Huidong Zhu. CE-Agent: a cloud-edge collaborative agent framework with dynamic offloading and synergistic distillation. Journal of Cloud Computing, 2026. doi: 10.1186/s13677-026-00984-5. URL https://doi.org/10.1186/s13677-026-00984-5.
  4. [4]Kaiyi Jiang, Zhaoqing Yan, Matteo Di Bernardo, Samantha R. Sgrizzi, Lukas Villiger, Alisan Kayabolen, B. J. Kim, Josephine K. Carscadden, Masahiro Hiraizumi, Hiroshi Nishimasu, Jonathan S. Gootenberg, and Omar O. Abudayyeh. Rapid in silico directed evolution by a protein language model with evolutionary prompting. Science, 387(6732):eadr6006, 2025. doi: 10.1126/science.adr6006. URL https://www.science.org/doi/abs/10.1126/science.adr6006.DOI
  5. [5]Shuowei Li. Root Cause Analysis of Failure in Molecular Biology Workflows. PhD thesis, University of Washington, 2022. URL https://hdl.handle.net/1773/49344.
  6. [7]Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabele, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1123–1130, 2023. doi: 10.1126/science.ade2574. URL https://www.science.org/doi/abs/10.1126/science.ade2574.DOI
  7. [8]Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 29287–29303. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/f51338d736f95dd4247296047067694-Paper.pdf.DOI
  8. [9]Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood van Niekerk, Steffanie Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, Jonathan Frazer, Mafalda Dias, Dinko Franceschi, Yarin Gal, and Debora Marks. ProteinGym: Large-scale benchmarks for protein fitness prediction and design. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 64331–64379. Curran Associates, Inc., 2023. doi: 10.52202/075280-2810. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/cac723e5ff29f65e3fcb b0739ae91bee-Paper-Datasets_and_Benchmarks.pdf.DOI
  9. [10]Yusuf H Roohani, Andrew H. Lee, Qian Huang, Jian Vora, Zachary Steinhart, Kexin Huang, Alexander Marson, Percy Liang, and Jure Leskovec. Biodiscoveryagent: An AI agent for designing genetic perturbation experiments. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=HAwZGLcye3.arxiv.org/abs/2405.17631
  10. [11]Yu Sun, Yin Li, Ruixiao Sun, Chunhui Liu, Fangming Zhou, Ze Jin, Linjie Wang, Xiang Shen, Zhuolin Hao, and Hongyu Xiong. Audio-enhanced vision-language modeling with latent space broadening for high quality data expansion. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, pages 4872–4881. Association for Computing Machinery, 2025. doi: 10.1145/3711896.3737195. URL https://doi.org/10.1145/3711896.3737195.
  11. [12]Yingshuo Wang, Xian Sun, Yanhang Li, Zhichao Fan, and Zexin Zhuang. Embedding foundation model predictions in discrete-choice models with structural guarantees, 2026. URL https://arxiv.org/abs/2606.26432.
  12. [13]Chenyu Wu. Class weighting versus amount conditioning in credit-card fraud detection: A dollar-metric study with a temporal explanation audit. arXiv preprint arXiv:2607.14686, 2026. doi: 10.48550/arXiv.2607.14686. URL https://arxiv.org/abs/2607.14686.
  13. [14]Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Bb4VGWQELI.arxiv.org/abs/2309.03409
  14. [15]Shuai Yuan, Xian Sun, Hannah Kim, Shuzhi Yu, and Carlo Tomasi. Optical flow training under limited label budget via active learning. In Computer Vision – ECCV 2022, pages 410–427. Springer, 2022. doi: 10.1007/978-3-031-20047-2_24. URL https://www.ecva.net/papers/eccv_2022/papers_ECCV/html/4388_ECCV_2022_paper.php.DOI
  15. [16]Jenny Zhang, Shengran Hu, Cong Lu, Robert Tjarko Lange, and Jeff Clune. Darwin gödel machine: Open-ended evolution of self-improving agents. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=pUpzQZTvGY.arxiv.org/abs/2505.22954

Paper details

vlnc:W_9v3daxs4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents