betaFeedback

WorksComputer Science

Lineage Is Memory: Parent Inheritance and Feedback Disclosure in Short-Budget LLM-Guided Optimization

Abstract

Iterative language-model optimizers refine prompts and programs by evaluating a candidate, showing a proposer model a sample of failing cases, and keeping revisions that improve performance. Recent work attributes the limits of this loop to narrow single-round reflection, introducing full-set error clustering, targeted minibatches, or larger proposer models so the optimizer sees more failure modes per step. Yet every existing evaluation tests these mechanisms exclusively inside lineage-preserving search, where each revision edits a surviving parent, leaving the contribution of parent inheritance confounded with the breadth of per-round feedback. Across 1,750 pre-registered runs spanning nine prompt and program synthesis benchmarks, we separate parent inheritance from feedback disclosure under matched evaluation budgets. We find that greedy parent inheritance drives the vast majority of optimization progress, outperforming seed-reset resampling even when seed-reset proposals receive a clustered summary of all failures in every round or use a larger proposer model. While full-set error clustering and diverse failure selection improve single-step rewrites from the seed and accelerate the opening rounds of search, sequential parent inheritance rapidly closes this gap as solved error modes are retained in the candidate and filtered from subsequent failure lists. Moreover, decomposing token expenditure in GEPA reveals that proposer reflection accounts for only a small fraction of total cost whereas validation and minibatch gating consume nearly the entire budget, showing that greedy parent inheritance over raw failures captures the gains of complex reflective loops at a fraction of their compute.

Introduction

Reflective prompt and program optimizers refine candidate instructions or code through an iterative propose-and-evaluate loop. In GEPA (Agrawal et al., [1]), the system runs a candidate instruction on a small minibatch, presents the inputs, outputs, and evaluator feedback to a proposer model, accepts the revision if it improves minibatch performance, and selects subsequent parents from a Pareto front. Similar evolutionary loops underlie OPRO (Yang et al., [24]), EvoPrompt (Guo et al., [8]), Promptbreeder (Fernando et al., [6]), FunSearch (Romera-Paredes et al., [18]), and AlphaEvolve (Novikov et al., [14]).

Recent work has focused on widening what the proposer observes during reflection. Liu et al. [11] note via a coupon-collector calculation that a random minibatch of three examples exposes only a small fraction of error modes per round, and propose ESPO, which clusters all training errors into three to seven diagnostic patterns before generating edits. RPT (Bayat et al., [2]) provides a tool that aggregates critiques across the training set, AdaGEPA (Chen et al., [4]) steers minibatch slots toward identified weaknesses, Contrastive Reflection (Koh et al., [9]) pairs failing and passing examples from decision-tree slices, and CASD (Singh et al., [22]) lets a coding agent inspect the full trace corpus. In all of these iterative studies, revisions are tested only inside lineage-preserving search, where every proposal edits a surviving parent. Without a lineage-off control that resets each proposal to the initial seed, existing evaluations cannot tell how much progress comes from wider single-round reflection and how much comes from accumulating edits along a parent-child lineage.

To separate these factors, we organize a large language model (LLM) optimizer around three design choices: lineage PP, feedback disclosure FF, and evaluation scheduling AA. Lineage PP determines whether each proposal edits the initial seed P=seedP = \mathrm{seed}, which corresponds to best-of-NN resampling from the seed’s feedback, or edits the current search-best candidate P=parentP = \mathrm{parent}. Disclosure FF determines whether the evaluator shows the proposer only the scalar score, the first kk failures, kk structurally diverse failures, a random mixed minibatch of kk passing and failing examples, or an LLM-clustered summary of all failures with or without expected answers. Scheduling AA governs candidate evaluation and stopping. We run a crossed P×FP \times F factorial experiment under a fixed full-search-set scheduler with early stopping, complemented by a system-level comparison against GEPA under matched maximum task-model call caps. Across 1,750 runs on nine task families, comprising five constructed rule-discovery tasks, a rubric-judged customer support task, ScoNe ([19]), a 12-intent subset of Banking77 ([3]), and BIG-Bench Hard salient translation error detection (BBH-STE) ((Suzgun et al., 2022)), we replicate the main contrasts on claude-haiku-5-5 and test a larger proposer model gpt-4.1.

Our main findings are as follows:

  • Parent inheritance outweighs single-round disclosure breadth and proposer size (§ 4.1–§ 4.2): On tasks where rules must be recovered from feedback, switching from PP = seed to PP = parent produces the largest accuracy gain in our study, adding +0.238 across all seven primary tasks on gpt-4.1-mini and +0.222 across five tasks on claude-haiku-5-5. This advantage holds even when seed-reset proposals draw fresh random examples each round in seed-minibatch or receive a clustered summary of all search-set failures in every round in seed-clustergold: showing just four raw failures under parent inheritance still beats seed-clustergold by +0.157. Upgrading the proposer from gpt-4.1-mini to gpt-4.1 on three feedback-dependent prompt tasks yields no clear gain in either row, while parent inheritance under gpt-4.1 adds +0.380.

  • Fact retention in the prompt explains the Banking77 gap with GEPA (§ 4.3): On the 12-intent Banking77 task, where intent names appear only in error feedback, baseline parent-first initially scored 0.372 against 0.664 for GEPA. Inspection of the search logs showed that early formatting-only edits scored 0.000 and stayed anchored to the seed, whereas GEPA’s reflection prompt explicitly tells the proposer to copy domain facts from feedback into the instruction. In a follow-up experiment on Banking77 and BBH-STE, appending one sentence with this instruction raised parent-first on Banking77 from 0.372 to 0.856 at 43% of GEPA’s median token cost, while seed-first with the same sentence retained only the 4 intents shown in round 1 and scored 0.317. On BBH-STE, where all six option labels are already in the seed, the same sentence reduced accuracy by -0.061.

  • Parent inheritance closes the gap with wider per-round disclosure over multiple rounds (Sections 4.4–§ 4.6): When resetting to the seed, showing only the first four failures crowds out zero-feedback code pairs on unsown error categories, and both diverse sampling and target-preserving clustering improve accuracy over first-kk disclosure. Under parent inheritance, full-set clustering initially leads in round 2 by +0.173, but as solved categories are retained in the parent and filtered from future failure lists, first-kk disclosure shifts across 46% to 86% of the search set, catching up by round 4 and finishing at the same level. Conversely, random mixed minibatches reduce parent-inheritance accuracy by spending feedback slots on passing examples.

  • GEPA spends most of its token budget on validation and minibatch gating (Section 4.7): Across all seven prompt tasks evaluated with GEPA, proposer reflection consumes only 2% to 15% of total tokens, while full search-set validations and minibatch checks consume the remaining 85% to 98%.

Related Work

LLM-Guided Program Evolution

Evolutionary search with language models, as in FunSearch (Romera-Paredes et al. [18]), AlphaEvolve (Novikov et al. [14]), and Impro vEvolve (Kravitsky et al. (2026)), maintains an archive of high-scoring programs and prompts a model to mutate or recombine selected parents. Performance gains in this literature are typically credited to the evolutionary loop as a whole. A recent evaluation audit of 27 self-evolving configuration and code optimization systems in recommendation and ranking (Yang and Shi [26]) finds that none runs an equal-budget random or seed-reset search over its edit space or reports multi-session spread on a sealed test split. In general program evolution, Evolution or Illusion? (Oved et al. [15]) varies how a fixed generation budget is split between initial seeds and evolutionary generations, showing that strategy rankings can reverse across budget scales. Oved et al. [15] hold feedback presentation fixed, do not break down token usage by call type, and treat best-of-NN resampling only as a reference curve. We complement these studies by fixing the evaluation budget cap, separating search and held-out splits across ten seeds per cell, and crossing parent inheritance with feedback disclosure.

Reflective Prompt Optimization

Early automatic prompt optimizers either backpropagated textual critiques from failing examples, as in ProTeGi [16], TextGrad [28], and graph-level LLM-AutoDiff [27], accumulated sampling-based momentum across stochastic gradient steps [5], or showed the proposer a score-ordered history of past instructions, as in OPRO [24] and EvoPrompt [8]. GEPA [1] combined reflection on a small random minibatch of default size b=3b = 3 with Pareto-front parent selection and minibatch acceptance gating. GEPA 0.0.7 also includes a prompt instruction asking the reflection model to copy domain-specific facts from feedback into the revised prompt; § 4.3 tests this instruction directly. Other recent studies examine when reflective optimization helps or fails: Lumer et al. [12] show that per-dimension feedback transfers better than scalar scores under noisy judges, consistent with findings that optimizing harnesses against ungrounded proxy judges induces Goodhart overfitting without improving held-out validity [26]; Giummarra et al. [7] find that GEPA improves MATH-500 accuracy but fails to improve on HotpotQA with local reasoning models while increasing token cost; and Ma et al. [13] show that prompt optimizers rely heavily on the proposer’s pre-trained task prior, matching our results on dates and scone, where the model prior already fits the specification.

Expanding Reflection Scope

Several recent papers argue that small random minibatches under-sample the error distribution during reflection. Liu et al. [11] formalize this via a coupon-collector bound and introduce ESPO, which clusters all training failures into three to seven error patterns in a Diagnose stage before generating pattern-specific edits in a Diversify stage and bootstrap-selecting among them in a Stabilize stage. RPT [2] gives the optimizer a clustering tool over full-set critiques; AdaGEPA [4] swaps minibatch slots toward identified failure modes; CASD [22] replaces iterative search with a coding agent that reads all traces before writing a single prompt; and Contrastive Reflection [9] pairs failing and passing examples from decision-tree slices. Every iterative method in this group is evaluated only inside a lineage-preserving loop, without a seed-reset control or a zero-feedback baseline. Testing disclosure schemes with and without parent inheritance shows that single-round error coverage strongly limits seed-reset optimization, whereas under parent inheritance its terminal effect is much smaller.

Validation Efficiency and Example Selection

Explore More, Validate Less [29] notes that GEPA wastes evaluations on minibatches where the parent already scores perfectly, producing no edit, and proposes trimming full-set validations. Section 4.7 extends this by decomposing GEPA’s token budget across reflection, minibatch gating, and full validation. Choosing a small representative set of examples to specify program behavior is also a classic problem in inductive program synthesis [17]; our diverse condition adapts farthest-point input selection to iterative error feedback.

Methodology and Experimental Design

The P/F/A Optimization Loop

We study iterative optimization over a candidate space C\mathcal{C} consisting of natural language system prompts or Python functions, implemented within the AdaFlow optimization framework [27]. Each task defines a search set Dsearch={(xi,yi∗)}i=1Nsearch\mathcal{D}_{\mathrm{search}} = \{(x_i, y_i^*)\}_{i=1}^{N_{\mathrm{search}}} partitioned into GG failure categories {C1search,…,CGsearch}\{\mathcal{C}^{\mathrm{search}}_1, \ldots, \mathcal{C}^{\mathrm{search}}_G\}, and a disjoint held-out test set Dholdout={(xj′,yj′∗)}j=1Nholdout\mathcal{D}_{\mathrm{holdout}} = \{(x'_j, y'_j{}^*)\}_{j=1}^{N_{\mathrm{holdout}}} partitioned into {C1holdout,…,CGholdout}\{\mathcal{C}^{\mathrm{holdout}}_1, \ldots, \mathcal{C}^{\mathrm{holdout}}_G\} that is never accessed during search. An evaluator E(c,D)E(c,\mathcal{D}) runs candidate c∈Cc \in\mathcal{C} on D\mathcal{D}, returning binary instance scores si(c)∈{0,1}s_i(c) \in\{0,1\}, accuracy S(c,D)=1∣D∣∑isi(c)S(c,\mathcal{D}) = \frac{1}{|\mathcal{D}|}\sum_i s_i(c), and failing indices E(c)={i:si(c)=0}\mathcal{E}(c) = \{i : s_i(c) = 0\}.

Algorithm 1: Controlled P/F/A Optimization Loop

  1. Input: Seed candidate c0∈Cc_0 \in\mathcal{C}, evaluation budget cap BB, lineage policy P∈{seed,parent}P \in\{\mathrm{seed},\mathrm{parent}\}, disclosure operator FF, search set Dsearch\mathcal{D}_{\mathrm{search}}, held-out set Dholdout\mathcal{D}_{\mathrm{holdout}}.

  1. Initialize: Evaluate c0c_0 on Dsearch\mathcal{D}_{\mathrm{search}} and cache E(c0,Dsearch)=(S(c0,Dsearch),E(c0))\mathcal{E}(c_0,\mathcal{D}_{\mathrm{search}}) = (S(c_0,\mathcal{D}_{\mathrm{search}}),\mathcal{E}(c_0)). Set history H0={c0}\mathcal{H}_0 = \{c_0\} and current best c0∗=c0c_0^* = c_0.

  1. For round t=1,…,B−1t = 1,\ldots,B-1 do:

  • Stop check (AA): If S(ct−1∗,Dsearch)=1.0S(c_{t-1}^*,\mathcal{D}_{\mathrm{search}}) = 1.0, stop early.

  • Parent selection (PP): Set pt=c0p_t = c_0 if P=seedP = \mathrm{seed}, or pt=ct−1∗p_t = c_{t-1}^* if P=parentP = \mathrm{parent}. Retrieve cached E(pt,Dsearch)\mathcal{E}(p_t,\mathcal{D}_{\mathrm{search}}).

  • Feedback disclosure (FF): Build feedback string ft=F(pt,Dsearch,k=4)f_t = F(p_t,\mathcal{D}_{\mathrm{search}}, k = 4) from E(pt,Dsearch)\mathcal{E}(p_t,\mathcal{D}_{\mathrm{search}}).

  • Proposal: Query proposer MpropM_{\mathrm{prop}} with PROPOSE_TEMPLATE (Appendix B.1) using the task objective, ptp_t, and ftf_t to generate ctc_t.

  • Evaluation (AA): Evaluate ctc_t on Dsearch\mathcal{D}_{\mathrm{search}}, consuming 1 evaluation from cap BB, and cache E(ct,Dsearch)\mathcal{E}(c_t,\mathcal{D}_{\mathrm{search}}). If S(ct,Dsearch)>S(ct−1∗,Dsearch)S(c_t,\mathcal{D}_{\mathrm{search}}) > S(c_{t-1}^*,\mathcal{D}_{\mathrm{search}}), set ct∗=ctc_t^* = c_t; otherwise keep ct∗=ct−1∗c_t^* = c_{t-1}^* under strict improvement tie-breaking.

  1. Output: Evaluate c∗c^* once on Dholdout\mathcal{D}_{\mathrm{holdout}} and return held-out accuracy Y=S(c∗,Dholdout)Y = S(c^*,\mathcal{D}_{\mathrm{holdout}}).

Factor 1: Lineage (PP). Under P=seedP = \mathrm{seed}, every round sets pt=c0p_t = c_0 and reuses the cached evaluation of c0c_0. Under deterministic disclosure schemes such as none, first, and diverse, both ptp_t and ftf_t stay fixed across rounds, so the B−1B - 1 proposals are independent samples from MpropM_{\mathrm{prop}} at default provider temperature conditioned on c0c_0. Under minibatch, cluster, and clustergold, ftf_t varies across rounds while still requiring a single-edit jump from c0c_0. Under P=parentP = \mathrm{parent}, every round edits the current search-best candidate pt=ct−1∗p_t = c_{t-1}^*, reusing its cached evaluation if the previous proposal did not strictly improve on it. When S(c0,Dsearch)=0.0S(c_0,\mathcal{D}_{\mathrm{search}}) = 0.0, as on labels, routing, replies, and banking77, strict-improvement tie-breaking keeps pt=c0p_t = c_0 until a proposal scores above zero; this occurs in round 1 on routing and replies and in round 1 or 2 on labels, but can take several rounds on banking77 (Section 4.3). Once ct∗c_t^* improves over c0c_0, P=parentP = \mathrm{parent} combines three mechanisms relative to P=seedP = \mathrm{seed}: text inheritance, which keeps discovered rules in pt+1p_{t+1}; residual-error exposure shift, which removes solved cases from E(pt+1)\mathcal{E}(p_{t+1}) so new categories enter the k=4k = 4 display window; and multi-step edit decomposition, which allows the proposer to fix a subset of categories per round. Secondary inspiration candidates are disabled in all cells.

Factor 2: Feedback Disclosure (FF). Failing examples are formatted with their input, wrong output, and expected answer or grading rubric (Appendix B). We compare six operators: - none: Masks the evaluation of ptp_t by setting score=None, omitting instance scores, and rendering {{ feedback }} as (no feedback). Because pt=c0p_t = c_0 carries no round-dependent signal under none, seed–none is the shared zero-feedback baseline. - first: Shows S(pt,Dsearch)S(p_t,\mathcal{D}_{\mathrm{search}}), per-instance binary scores, and the first k=4k = 4 failures of ptp_t in dataset order, followed by the count of remaining failures. In round 1, this exposes 2 categories on labels, dates, replies, and size, up to 3 on duration, and up to 4 on routing, banking77, and bbh_ste. - diverse: Selects k=4k = 4 failures via greedy farthest-point sampling under normalized Levenshtein distance over a structural input signature defined by _shape in presets.py, which maps lowercase letters to a, uppercase letters to A, and digits to 9 while preserving punctuation, starting from the first failure. - minibatch: Draws k=4k = 4 examples uniformly at random without replacement from Dsearch\mathcal{D}_{\mathrm{search}} in each round using random.Random(1000 + r) for run seed r∈{0,…,9}r \in\{0,\ldots,9\}, and shows whether ptp_t passed or failed each drawn example alongside its expected answer, while still gating acceptance on the full search set. - cluster: Passes all failures E(pt)\mathcal{E}(p_t) to one call of the clusterer model, which matches MpropM_{\mathrm{prop}} and groups them into 3 to 7 patterns, outputting one line per pattern with a description, failure count, and one verbatim representative wrong answer (CLUSTER_PROMPT, Appendix B.3). Clustering tokens are included in search tokens. - clustergold: Identical to cluster, except the prompt requires each representative failure to include the input, wrong answer, and full expected answer (CLUSTER_GOLD_PROMPT, Appendix B.3).

Scheduling (AA) and GEPA Comparison. In our P×FP \times F grid, every proposal is evaluated on all of Dsearch\mathcal{D}_{\mathrm{search}} up to B=8B = 8 evaluations for prompt tasks, corresponding to c0c_0 plus 7 proposals and at most 8×Nsearch8 \times N_{\mathrm{search}} task calls, and B=12B = 12 evaluations for code tasks, corresponding to c0c_0 plus 11 proposals executed with a 60-second subprocess timeout across 24 test cases for at most 288 executions. Missing fenced blocks trigger at most one retry, though all 1,750 runs produced valid blocks on the first try; code exceptions score 0.0 with the traceback included in feedback. Proposer, task, and clusterer models use default provider sampling in openai SDK 2.44.0 for gpt-4.1-mini and gpt-4.1 and anthropic SDK 0.112.0 with default adaptive thinking for claude-haiku-5-5, while the replies judge uses gpt-4.1-mini at temperature=0 with response caching. For system-level comparison, we run gepa 0.0.7 ([1]) with default minibatch size b=3b = 3, Pareto selection, minibatch acceptance gating, its built-in reflection prompt, and the same task-call cap max_metric_calls=8×Nsearch\mathrm{max\_metric\_calls}=8 \times N_{\mathrm{search}}, testing both unmodified GEPA mb3 and early-stopped GEPA mb3 + stop.

Benchmark Suite and Task Specifications

Table 1 summarizes the seven primary tasks and two public extension tasks.

Task FamilyArtifact TypeCategories (GG)∣Dsearch∣|\mathcal{D}_{\mathrm{search}}|∣Dholdout∣|\mathcal{D}_{\mathrm{holdout}}|Budget Cap (BB)Max Task CallsSeed Score (Search / Holdout)Latent Specification Withheld from c0c_0 and Revealed via Feedback
labelsPrompt1224 (2/cat)24 (2/cat)81920.00 / 0.00Exact snake_case category names (ipv4, hex_color, us_date, semver, etc.)
datesPrompt1224 (2/cat)24 (2/cat)81920.75 / 0.75Slash vs. dot day/month order, 2-digit century cutoff (19xx vs. 20xx), invalid token
routingPrompt616 (2–3/cat)12 (2/cat)81280.00 / 0.006 queue names (payments, fulfillment, logistics, trust_safety, privacy, care) & edge policies
repliesPrompt (Free-text)816 (2/cat)16 (2/cat)81280.00 / 0.008 Acme support policies (refund window, >7-day $10 credit threshold, URLs, legal email, GDPR rules)
sconePrompt (Public NLI)636 (6/cat)72 (12/cat)82880.89 / 0.896 negation-scope conditions from ScoNe (She et al. [19]); binary entailment / neutral labels
durationCode (Python)72424122880.46 / 0.42Plain seconds, compound units (1h 30m), English words, clock (M:SS, HH:MM:SS), ISO 8601 (PT1H30M), ValueError
sizeCode (Python)72424122880.21 / 0.21Plain bytes, SI decimal (10KB), IEC binary (4MiB, 2Gi), single-letter (4K), comma separators (1,024), English words
banking77Prompt (Public Cls.)1224 (2/cat)36 (3/cat)81920.00 / 0.0012 intent labels sampled from Banking77 (Casanueva et al. [3]), withheld from the initial seed
bbh_stePrompt (Public QA)618 (3/cat)36 (6/cat)81440.56 / 0.566 translation error classes in BBH-STE (Suzgun et al. (2022)); all 6 option names given in seed

Table 1. Specifications of the seven primary tasks and two public extension tasks.

In the five constructed tasks, namely labels, dates, routing, duration, and size, as well as in replies, the seed c0c_0 omits key rules or label names so that high accuracy requires learning from feedback (Appendix B.§ 2). For the two code tasks, c0c_0 is a 4-line function that handles only single-character suffixes: s, m, and h on duration, and B, K, M, and G on size. This setup favors methods that accumulate rules across rounds. To test public benchmarks, we included scone (She et al. [19]) in Batch A.1 and added banking77 (Casanueva et al. [3]) and bbh_ste (Suzgun et al. (2022)) in Batch A.4. For banking77, 12 of 77 intents were sampled with random.Random(0) before registration after an excluded 20-intent smoke test scored at most 0.15 within 8 evaluations, drawing 2 search and 3 held-out queries per intent. For bbh_ste, the six option names were included in c0c_0, and examples exceeding the 300-character feedback truncation limit were filtered before registration, retaining 198 of 250 examples and sampling 3 search and 6 held-out examples per class.

Statistical Estimands, Bootstrap Protocol, and Run Accounting

The replication unit is an optimization run with seed r∈{0,…,9}r \in\{0,\ldots,9\}, giving n=10n=10 runs per cell. Let Yτ,P,F,r∈[0,1]Y_{\tau,P,F,r} \in[0,1] denote held-out accuracy S(ct′,Dholdout)S(c'_t,\mathcal{D}_{\mathrm{holdout}}) on task τ∈T\tau\in\mathcal{T}, reserving tt for optimization rounds, and let Yˉτ,P,F=110∑r=09Yτ,P,F,r\bar{Y}_{\tau,P,F}=\frac{1}{10}\sum_{r=0}^{9}Y_{\tau,P,F,r} denote the cell mean. Weighting tasks in T\mathcal{T} equally, we estimate three quantities:

  1. Lineage main effect (Δlineage\Delta_{\mathrm{lineage}}):

Δlineage=1∣T∣⋅∣F∣∑τ∈T∑F∈F(Yˉτ,parent,F−Yˉτ,seed,F).\Delta_{\mathrm{lineage}}=\frac{1}{|\mathcal{T}|\cdot|\mathcal{F}|}\sum_{\tau\in\mathcal{T}}\sum_{F\in\mathcal{F}}\left(\bar{Y}_{\tau,\mathrm{parent},F}-\bar{Y}_{\tau,\mathrm{seed},F}\right).
  1. Attenuation interaction (IXI_X): For X∈{diverse,minibatch,cluster,clustergold}X \in\{\text{diverse},\text{minibatch},\text{cluster},\text{clustergold}\} and δτ,P(X)=Y‾τ,P,X−Y‾τ,P,first\delta_{\tau,P}(X) = \overline{Y}_{\tau,P,X} - \overline{Y}_{\tau,P,\mathrm{first}},

IX=1∣T∣∑τ∈T(δτ,seed(X)−δτ,parent(X)).I_X = \frac{1}{|\mathcal{T}|} \sum_{\tau\in\mathcal{T}} \left(\delta_{\tau,\mathrm{seed}}(X) - \delta_{\tau,\mathrm{parent}}(X)\right).

A positive IX>0I_X > 0 means disclosure XX improves over first more under seed resetting than under parent inheritance.

  1. Unshown-category proposal coverage (Covunshown\mathrm{Cov}_{\mathrm{unshown}}): In the seed row, Uτ⊂{1,…,G}\mathcal{U}_{\tau} \subset\{1,\ldots,\mathcal{G}\} is the set of categories where c0c_0 fails at least one search example in >50%> 50\% of seed runs, excluding categories shown among the first four failures in >50%> 50\% of seed-first runs, with seed-none using the same reference set Uτ\mathcal{U}_{\tau}. Across proposals Vr={c1,…,cmr}\mathcal{V}_r = \{c_1,\ldots,c_{m_r}\} in run rr,

Covunshown(r)=1∣Uτ∣∑g∈Uτ1∣Vr∣∑c∈Vr1[si(c)=1 for all i∈Cgeval].\mathrm{Cov}_{\mathrm{unshown}}(r) = \frac{1}{|\mathcal{U}_{\tau}|} \sum_{g \in\mathcal{U}_{\tau}} \frac{1}{|\mathcal{V}_r|} \sum_{c \in\mathcal{V}_r} \mathbf{1}\left[s_i(c)=1\text{ for all }i\in\mathcal{C}^{\mathrm{eval}}_g\right].

where Cgeval\mathcal{C}^{\mathrm{eval}}_g is Cgsearch\mathcal{C}^{\mathrm{search}}_g for prompt tasks and Cgholdout\mathcal{C}^{\mathrm{holdout}}_g, evaluated offline after search completes, for code tasks. Crowding-out (HC1\mathcal{H}_{C1}) holds when Cov‾unshown(seed-first)−Cov‾unshown(seed-none)<0\overline{\mathrm{Cov}}_{\mathrm{unshown}}(\text{seed-first}) - \overline{\mathrm{Cov}}_{\mathrm{unshown}}(\text{seed-none}) < 0.

Bootstrap inference and run accounting. All confidence intervals are 95% percentile bootstrap intervals given by the 1-indexed order statistics [ψ~(251),ψ~(9750)][\widetilde{\psi}_{(251)},\widetilde{\psi}_{(9750)}] over Nboot=10,000N_{\mathrm{boot}} = 10{,}000 task-stratified resamples initialized with random.Random(0) in results/analyze.py, resampling 10 runs with replacement within each task-by-condition cell. For conditional tokens-to-solve ratios (HC3\mathcal{H}_{C3}), solved runs are resampled within each arm on tasks with at least 3/10 solved runs per arm. Alongside these fixed-strata intervals, we report per-task intervals (Table 2A), leave-one-task-out intervals (Table 8), and two-stage task-cluster bootstrap intervals that resample tasks at the outer level: on gpt-4.1-mini, the 7-task cluster interval is [+0.104,+0.363][+0.104,+0.363] for Δlineage\Delta_{\mathrm{lineage}} and [+0.052,+0.336][+0.052,+0.336] for IclustergoldI_{\mathrm{clustergold}}, which is positive on all 7 tasks with exact two-sided sign test p=0.0156p = 0.0156. All 1,750 completed runs in results/runs.jsonl come from five sequential pre-registered batches documented in Table 7 and Appendix A: Batch A.1 with 750 runs, Batch A.2 with 140 runs, Batch A.3 with 350 runs, Batch A.4 with 470 runs, and Batch A.5 with 40 runs.

Batch & IDHypothesis / Contrast DefinitionPre-Registered CriterionObserved Point Estimate & 95% CIOutcome
Batch A.1: Main Factorial Grid (gpt-4.1-mini, 7 tasks, n=10n = 10, 750 runs; frozen at commit d5ab8923)
Δlineage\Delta_{\mathrm{lineage}}Parent − Seed mean over first, diverse, minibatch, clusterDescriptive main-effect estimate+0.238 [+0.224, +0.252]Descriptive
HC2HC_2 (Primary)Idiverse=(div−first)seed−(div−first)parentI_{\mathrm{diverse}} = (\mathrm{div} - \mathrm{first})_{\mathrm{seed}} - (\mathrm{div} - \mathrm{first})_{\mathrm{parent}}Lower CI bound >0> 0+0.068 [+0.043, +0.092]Supported
HR1HR_1 (Refutation)Both IdiverseI_{\mathrm{diverse}} and IclusterI_{\mathrm{cluster}} 95% CIs contain 0Refutes HC2HC_2 if TrueIdivI_{\mathrm{div}}: [+0.043, +0.092]; IclusI_{\mathrm{clus}}: [+0.073, +0.167]Rejected (HC2HC_2 holds)
HR2HR_2 (Collapse)cluster − max(other FF) has lower bound >0> 0 in both rowsFavors full-set clustering in both rows if TrueSeed: −0.044 [−0.078, −0.018]; Parent: −0.095 [−0.132, −0.063]Rejected
HC1HC_1 (Crowding)Unshown-category coverage: seed-first − seed-noneUpper CI bound <0< 0 on ≥ 2/3 tasksduration: −0.055; size: −0.113; labels: +0.018Supported (2/3)
HR3HR_3 (ScoNe boundary)All 4 seed (first, diverse, minibatch, cluster) − seed-none CIs on scone contain 0Narrows HC1HC_1 to latent-rule tasksAll 4 contrasts within [−0.025, +0.017]Confirmed (Boundary)
HC3HC_3 (GEPA tokens)Conditional median tokens-to-solve ratio: GEPA+stop / parent-firstLower CI bound >1> 1 on eligible tasks (≥ 3/10 solved)labels: 3.34× [2.23, 4.49]; dates: 2.59× [2.30, 4.84]Supported (2/2 eligible)
HR4HR_4 (Early-stop ref.)Tokens-to-solve ratio CI contains 1 on ≥ 3/5 tasksRestates HC3HC_3 if True0/2 eligible tasks contain 1 (3/5 tasks ineligible: < 3 solved runs)Not evaluable as registered (0/2 eligible contain 1)
Batch A.2: Target-Preserving Cluster Sensitivity (sens-clustergold, gpt-4.1-mini, 7 tasks, n=10n = 10, 140 runs)
HS1-goldHS_{1\text{-gold}} (Info loss)replies: parent-clustergold − parent-cluster & ≥ 80% gold keptLower CI >0> 0 & retention ≥ 80%+0.350 [+0.062, +0.612]; retention 65/65 (100%)Supported
HS2-goldHS_{2\text{-gold}} (Parent tie)7-task parent-clustergold − parent-firstUpper CI < 0: harmful; 0 ∈ CI: tie; Lower CI >0> 0: gain−0.018 [−0.053, +0.010] (Iclustergold=+0.174I_{\mathrm{clustergold}} = +0.174)No resolved diff. (0 ∈ CI)
Batch A.3: Second-Model Replication (rep-haiku, claude-haiku-5-5, 5 tasks, n=10n = 10, 350 runs)
HH1HH_1Lineage main effect across first, minibatch, clustergoldLower CI bound >0> 0+0.222 [+0.204, +0.239]Supported
HH2HH_2Attenuation interaction IclustergoldI_{\mathrm{clustergold}} on 5 tasksLower CI bound >0> 0+0.187 [+0.152, +0.221]Supported
HH3HH_3parent-clustergold − parent-first non-superiorityUpper CI bound <+0.05< +0.05−0.008 [−0.017, +0.002]Supported
Batch A.4: Public Benchmarks, GEPA Public, Strong Proposer, and Haiku Diverse (470 runs)
HP1HP_1Public (banking77, bbh_ste) lineage main effect (4 cols)Lower CI bound >0> 0+0.088 [+0.045, +0.130] (b77: +0.158; bbh: +0.017)Supported
HP2HP_2Public attenuation interaction IclustergoldI_{\mathrm{clustergold}}Lower CI bound >0> 0+0.017 [−0.119, +0.151]Not supported (Inconclusive)
HP3HP_3Public parent-clustergold − parent-firstUpper CI bound <+0.05< +0.05+0.021 [−0.101, +0.143]Not supported (Width)
HP4HP_4Public attenuation interaction IdiverseI_{\mathrm{diverse}}Lower CI bound >0> 0−0.014 [−0.118, +0.093]Not supported (Inconclusive)
HP5HP_5Public parent-minibatch − parent-firstUpper CI bound <0< 0−0.011 [−0.113, +0.090]Not supported (Inconclusive)
HS1-propHS_{1\text{-prop}} (Strong prop.)gpt-4.1-proposer lineage main effect (first, clustergold)Lower CI bound >0> 0+0.380 [+0.337, +0.425]Supported
HS2-propHS_{2\text{-prop}} (Strong prop.)gpt-4.1-proposer attenuation interaction IclustergoldI_{\mathrm{clustergold}}Lower CI bound >0> 0+0.294 [+0.201, +0.381]Supported
HS3-propHS_{3\text{-prop}} (Strong prop.)gpt-4.1-proposer parent-clustergold − parent-firstUpper CI bound <+0.05< +0.05−0.057 [−0.101, −0.016]Supported
HH4HH_{4} (Haiku div.)claude-haiku-5-5 attenuation interaction IdiverseI_{\mathrm{diverse}} (5 tasks)Lower CI bound >0> 0+0.122 [+0.088, +0.158]Supported
Batch A.5: Discovery-motivated carry-facts Follow-Up (gpt-4.1-mini, 2 public tasks, n=10n = 10, 40 runs)
HF1HF_1banking77: parent-carry − parent-firstLower CI bound >0> 0+0.483 [+0.328, +0.644]Supported
HF2HF_2banking77: GEPA mb3 − parent-carryUpper CI bound ≤ 0−0.192 [−0.289, −0.097]Supported
HF3HF_3 (No harm)bbh_ste: parent-carry − parent-firstLower CI bound ≥ −0.05−0.061 [−0.119, −0.006]Not supported (Harm)
HF4HF_4Pooled (banking77, bbh_ste): parent-carry − seed-carryLower CI bound >0> 0+0.258 [+0.221, +0.296] (b77: +0.539; bbh: −0.022)Supported

Table 7. Complete inventory of 1 descriptive main-effect estimate and 25 pre-registered hypothesis tests across all five batches (1,750 analyzed runs).

Dropped Taskgpt-4.1-mini Δlineage\Delta_{\mathrm{lineage}}gpt-4.1-mini IdiverseI_{\mathrm{diverse}}gpt-4.1-mini Parent cluster IclustergoldI_{\mathrm{clustergold}}gpt-4.1-mini Parent minibatchclaude-haiku-5-5 IH1I_{H1} (Δlineage\Delta_{\mathrm{lineage}})claude-haiku-5-5 IH2I_{H2} (IclustergoldI_{\mathrm{clustergold}})claude-haiku-5-5 IH3I_{H3} (parent-clustergold − parent-first)
None (Full set)+0.238 [+0.224, +0.252]+0.068 [+0.043, +0.092]−0.082 [−0.119, −0.045]−0.061 [−0.086, −0.035]+0.222 [+0.204, +0.239]+0.187 [+0.152, +0.221]−0.008 [−0.017, +0.002]
− labels+0.217 [+0.201, +0.233]+0.046 [+0.020, +0.071]−0.099 [−0.142, −0.057]−0.065 [−0.094, −0.036]+0.236 [+0.215, +0.256]+0.197 [+0.155, +0.238]−0.018 [−0.029, −0.007]
− dates+0.278 [+0.261, +0.294]+0.080 [+0.051, +0.107]−0.092 [−0.135, −0.051]−0.069 [−0.099, −0.039]
− routing+0.222 [+0.207, +0.236]+0.078 [+0.053, +0.101]−0.072 [−0.111, −0.033]−0.042 [−0.064, −0.018]+0.208 [+0.190, +0.226]+0.152 [+0.111, +0.192]−0.001 [−0.009, +0.007]
− replies+0.196 [+0.183, +0.209]+0.046 [+0.019, +0.072]−0.019 [−0.045, +0.008]−0.060 [−0.090, −0.031]+0.168 [+0.151, +0.184]+0.143 [+0.111, +0.173]−0.000 [−0.009, +0.009]
− scone+0.276 [+0.260, +0.293]+0.079 [+0.051, +0.107]−0.097 [−0.140, −0.053]−0.067 [−0.098, −0.037]
− duration+0.227 [+0.211, +0.243]+0.053 [+0.025, +0.080]−0.098 [−0.141, −0.056]−0.054 [−0.083, −0.025]+0.257 [+0.236, +0.278]+0.214 [+0.169, +0.258]−0.009 [−0.021, +0.003]
− size+0.251 [+0.236, +0.267]+0.094 [+0.071, +0.117]−0.098 [−0.138, −0.058]−0.067 [−0.091, −0.042]+0.239 [+0.220, +0.258]+0.228 [+0.189, +0.267]−0.009 [−0.021, +0.003]

Table 8. Leave-one-task-out robustness analysis (95% task-stratified bootstrap confidence intervals) for gpt-4.1-mini (7 tasks) and claude-haiku-5-5 (5 tasks).

Experimental Results

Parent Inheritance Across Task Families and Models

Tables 2A and 2B report held-out accuracy across the seven primary tasks on gpt-4.1-mini.

Lineage (PP) & Disclosure (FF)labelsdatesroutingrepliessconedurationsizeMean (All 7)Mean (5 Non-Ceil.)
seed-none (Zero feedback)0.2791.0000.0000.0560.8970.6330.7380.5150.341
seed-first0.4081.0000.5000.2940.8990.5620.7670.6330.506
seed-diverse0.6330.9960.4420.4190.9010.7290.7380.6940.592
seed-minibatch0.5540.9960.5000.4310.8890.6250.8040.6860.583
seed-cluster0.6130.9880.5330.2120.8990.6370.8120.6710.561
seed-clustergold (Batch A.2)0.6831.0000.6250.7940.9080.6370.8750.7890.723
parent-first0.9171.0000.9250.9810.9100.9630.9290.9460.943
parent-diverse0.9421.0000.8580.9060.9120.9710.9880.9400.933
parent-minibatch0.8830.9920.7500.9190.8890.8630.9040.8860.864
parent-cluster0.9380.9790.7830.5190.9150.9750.9420.8640.831
parent-clustergold (Batch A.2)0.9580.9960.8420.8690.9000.9710.9630.9280.921
Δlineage\Delta_{\mathrm{lineage}} (first, diverse, minibatch, cluster)+0.368-0.002+0.335+0.492+0.010+0.304+0.160+0.238+0.332
95% Bootstrap CI for Δlineage\Delta_{\mathrm{lineage}}[+0.334, +0.402][-0.009, +0.004][+0.283, +0.387][+0.428, +0.552][+0.000, +0.020][+0.277, +0.330][+0.121, +0.199][+0.224, +0.252][+0.312, +0.351]

caption: Held-out accuracy cell means and per-task lineage effect Δlineage\Delta_{\mathrm{lineage}} on gpt-4.1-mini (n=10n = 10 runs per cell). “5 Non-Ceil.” averages over the five tasks with headroom above seed-none (labels, routing, replies, duration, size), excluding dates and scone.

:::

Cross-Task Contrast (7 Tasks Equal-Weighted, gpt-4.1-mini)Point Estimate & 95% Bootstrap CI5 Non-Ceiling Tasks Mean
Lineage main effect Δlineage\Delta_{\mathrm{lineage}} (first, diverse, minibatch, cluster)+0.238 [+0.224, +0.252]+0.332 [+0.312, +0.351]
Lineage main effect across all 5 schemes / across clustergold only+0.218 [+0.204, +0.233] / +0.139 [+0.096, +0.181]+0.305 / +0.198
Cross-row contrast: parent-first − seed-clustergold / parent-first − seed-minibatch+0.157 [+0.124, +0.195] / +0.261 [+0.235, +0.285]+0.220 / +0.360
Attenuation interaction Idiverse=(diverse−first)seed−(diverse−first)parentI_{\mathrm{diverse}} = (\mathrm{diverse} - \mathrm{first})_{\mathrm{seed}} - (\mathrm{diverse} - \mathrm{first})_{\mathrm{parent}} (HC2\mathcal{H}_{C2})+0.068 [+0.043, +0.092]+0.096
Attenuation interaction Iminibatch=(minibatch−first)seed−(minibatch−first)parentI_{\mathrm{minibatch}} = (\mathrm{minibatch} - \mathrm{first})_{\mathrm{seed}} - (\mathrm{minibatch} - \mathrm{first})_{\mathrm{parent}}+0.113 [+0.079, +0.148]+0.156
Attenuation interaction Icluster=(cluster−first)seed−(cluster−first)parentI_{\mathrm{cluster}} = (\mathrm{cluster} - \mathrm{first})_{\mathrm{seed}} - (\mathrm{cluster} - \mathrm{first})_{\mathrm{parent}}+0.120 [+0.073, +0.167]+0.167
Attenuation interaction Iclustergold=(clustergold−first)seed−(clustergold−first)parentI_{\mathrm{clustergold}} = (\mathrm{clustergold} - \mathrm{first})_{\mathrm{seed}} - (\mathrm{clustergold} - \mathrm{first})_{\mathrm{parent}}+0.174 [+0.127, +0.222]+0.239
Seed row within-row contrasts: seed-diverse − seed-first / seed-clustergold − seed-first+0.061 [+0.046, +0.077] / +0.156 [+0.122, +0.188]+0.086 / +0.217
Seed row within-row contrasts: seed-minibatch − seed-first / seed-cluster − seed-first+0.053 [+0.030, +0.075] / +0.038 [+0.009, +0.067]+0.077 / +0.055
Parent row within-row contrasts: parent-diverse − parent-first / parent-clustergold − parent-first (HS2-gold\mathcal{H}_{S2\text{-gold}})-0.007 [-0.025, +0.012] / -0.018 [-0.053, +0.010]-0.010 / -0.022
Parent row within-row contrasts: parent-minibatch − parent-first / parent-cluster − parent-first-0.061 [-0.086, -0.035] / -0.082 [-0.119, -0.045]-0.079 / -0.112

Table 2B. Cross-task interaction and within-row disclosure contrasts on gpt-4.1-mini.

Parent inheritance produces the largest accuracy contrast in the grid, increasing 7-task held-out accuracy by Δlineage=+0.238\Delta_{\mathrm{lineage}} = +0.238 across the four Batch A.1 disclosure schemes and by +0.218 across all five schemes (tbl:2B). Importantly, this gain is not simply a consequence of seed-first observing the same four failures each round. Even when seed-reset proposals draw fresh random examples every round in seed-minibatch or receive a target-preserving summary of all search-set failures in every round in seed-clustergold, parent-first beats both by +0.261 and +0.157, and parent-clustergold beats seed-clustergold by +0.139 across all seven tasks and +0.198 across the five feedback-dependent tasks. Thus, even when the proposer sees a summary of all failing categories, rewriting the seed from scratch in a single step falls well short of accumulating edits across rounds.

Across tasks, Figure 1 and tbl:2A show two distinct regimes. On dates and scone, the model’s pre-trained prior already solves most cases without feedback, scoring 1.000 and 0.897 under seed-none and leaving little room for lineage to help. On the five tasks where rules or policies are withheld from c0c_0, parent inheritance adds between +0.160 on size and +0.492 on replies, averaging +0.332. Every pre-registered parent cell on these five tasks beats every pre-registered seed cell; the only crossover involves seed-clustergold beating parent-cluster on replies, where the unadjusted cluster prompt dropped grading rubrics (§ 4.6). Replicating the experiment on claude-haiku-5-5 across the same five tasks confirms these patterns (Table 3, Panel A): parent inheritance adds +0.222 overall and +0.257 when excluding duration, where Haiku starts near ceiling at 0.954, and parent-first again outperforms both seed-minibatch and seed-clustergold.

Held-out accuracy of the search-best candidate across disclosure schemes

Figure 1. Held-out accuracy of the search-best candidate across disclosure schemes on the seven primary tasks for gpt-4.1-mini, comparing seed-reset resampling (P=seedP = \text{seed}, orange) against greedy parent inheritance (P=parentP = \text{parent}, blue). Points show means over n=10n = 10 runs per cell; shaded bands indicate 95% bootstrap confidence intervals. The zero-feedback baseline (none) is shared by both rows.

Model Configuration & CelllabelsroutingrepliesdurationsizeMean (5 Tasks)Mean (3 Prompt Tasks)
Panel A: claude-haiku-5-5 Replication (nn = 10 runs per cell, 450 runs across Batches A.3 & A.4)
seed-none0.6710.0000.0440.9540.6500.4640.238
seed-first0.7250.5000.3120.8630.8170.6430.512
seed-diverse (Batch A.4)0.8710.4000.4870.9420.8790.7160.586
seed-minibatch0.7750.5080.4440.9420.8630.7060.576
seed-clustergold0.9040.7920.6370.9420.8380.8230.778
parent-first0.9580.9500.9881.0001.0000.9790.965
parent-diverse (Batch A.4)0.9250.9420.7811.0001.0000.9300.883
parent-minibatch0.9500.7580.7620.9880.9710.8860.823
parent-clustergold0.9920.9170.9501.0001.0000.9720.953
Haiku HH1\mathcal{H}_{H1} (Δlineage\Delta_{\mathrm{lineage}})+0.165+0.275+0.435+0.081+0.151+0.222 [+0.204, +0.239]+0.292
Haiku HH2\mathcal{H}_{H2} (IclustergoldI_{\mathrm{clustergold}})+0.146+0.325+0.363+0.079+0.021+0.187 [+0.152, +0.221]+0.278
Haiku HH3\mathcal{H}_{H3} (parent-cgold − first)+0.033-0.033-0.038+0.000+0.000-0.008 [-0.017, +0.002]-0.013
Haiku HHd\mathcal{H}_{Hd} (IdiverseI_{\mathrm{diverse}})+0.179-0.092+0.381+0.079+0.062+0.122 [+0.088, +0.158]+0.156
Panel B: Stronger Proposer (gpt-4.1 Proposer & Clusterer, gpt-4.1-mini Task Model, nn = 10, 150 runs)
seed-none (gpt-4.1 prop.)0.2790.0000.062–––0.114 (Δprop\Delta_{\mathrm{prop}}: +0.002)
seed-first (gpt-4.1 prop.)0.3830.5000.300–––0.394 (Δprop\Delta_{\mathrm{prop}}: -0.006)
seed-clustergold (gpt-4.1 prop.)0.5960.6170.681–––0.631 (Δprop\Delta_{\mathrm{prop}}: -0.069)
parent-first (gpt-4.1 prop.)0.9040.8920.969–––0.922 (Δprop\Delta_{\mathrm{prop}}: -0.019)
parent-clustergold (gpt-4.1 prop.)0.8580.7920.944–––0.865 (Δprop\Delta_{\mathrm{prop}}: -0.025)
Strong-Prop. HS1-prop\mathcal{H}_{S1\text{-prop}} (Δlineage\Delta_{\mathrm{lineage}})+0.392+0.283+0.466–––+0.380 [+0.337, +0.425]
Strong-Prop. HS2-prop\mathcal{H}_{S2\text{-prop}} (IclustergoldI_{\mathrm{clustergold}})+0.258+0.217+0.406–––+0.294 [+0.201, +0.381]
Strong-Prop. HS3-prop\mathcal{H}_{S3\text{-prop}} (parent-cgold − first)-0.046-0.100-0.025–––-0.057 [-0.101, -0.016]

Table 3. Cross-model replication on claude-haiku-5-5 (Panel A) and stronger-proposer evaluation using gpt-4.1 as proposer and clusterer with gpt-4.1-mini as task model (Panel B). Here Δprop\Delta_{\mathrm{prop}} is the difference (gpt-4.1 proposer − gpt-4.1-mini proposer) across labels, routing, and replies, and HS3-prop\mathcal{H}_{S3\text{-prop}} is parent-clustergold − parent-first.

Upgrading Proposer Model Capacity Versus Parent Inheritance

Panel B of Table 3 tests whether upgrading the proposer and clusterer from gpt-4.1-mini to gpt-4.1 reduces the need for parent inheritance on labels, routing, and replies while holding the task model fixed at gpt-4.1-mini. Across all five tested conditions, replacing gpt-4.1-mini with gpt-4.1 yields no positive accuracy gain, with Δprop\Delta_{\mathrm{prop}} ranging from -0.069 in seed-clustergold to +0.002 in seed-none. On labels under seed-none, both proposers average 67/240 = 0.279 from distinct run distributions, where nine of ten gpt-4.1 runs score 6/24 = 0.250 and one scores 13/24 = 0.542. Meanwhile, parent inheritance under the gpt-4.1 proposer adds Δlineage\Delta_{\mathrm{lineage}} = +0.380, exhibits a large attenuation interaction IclustergoldI_{\mathrm{clustergold}} = +0.294, and scores slightly higher with first-kk failures than with clustergold by +0.057. Upgrading proposer capacity within the GPT-4.1 family therefore does not substitute for parent inheritance.

Public Benchmarks and Instruction-Level Retention of Feedback Facts

Table 4 reports the 9-cell factorial grid alongside GEPA on the 12-intent Banking77 subset and BBH-STE.

Condition / Methodbanking77 Held-Outbbh_ste Held-OutPooled Meanbanking77 Intents Named in Best Prompt (Run Distribution; Mean / 12)banking77 Search Tokens / Run (Median / Mean)bbh_ste Search Tokens / Run (Median / Mean)
Panel A: Pre-Registered Public Benchmark Grid (gpt-4.1-mini, n=10n = 10 runs per cell, Batch A.4)
seed-none0.0000.5560.2780.0 / 1221.7k / 22.0k42.7k / 42.8k
seed-first0.2250.6310.4283.2 / 12 (range 1–4)25.5k / 26.0k48.2k / 47.9k
seed-diverse0.2330.6280.431–26.8k / 27.0k47.0k / 47.4k
seed-minibatch0.1940.6110.403–26.6k / 26.5k47.1k / 47.6k
seed-clustergold0.2500.6810.466–41.4k / 41.8k61.8k / 61.5k
parent-first0.3720.6530.513[6,11,4,11,7,9,4,4,0,0] (5.6 / 12)38.7k / 37.3k52.5k / 55.3k
parent-diverse0.4030.6560.530–37.8k / 37.3k53.0k / 55.0k
parent-minibatch0.3610.6420.502–35.4k / 35.6k53.9k / 54.7k
parent-clustergold0.4000.6670.534–55.4k / 56.4k69.8k / 71.4k
Panel B: GEPA Baseline (Batch A.4) and Discovery-Motivated carry-facts Follow-Up (Batch A.5, n=10n = 10)
GEPA mb3 (No early stop)0.6640.6330.649Median 9.0, Mean 9.0 / 12106.9k / 107.3k121.6k / 121.9k
GEPA mb3 + stop0.6610.6440.653–102.3k / 105.3k114.9k / 118.5k
seed-first + carry0.3170.6140.466[4,4,4,4,4,4,4,4,4,4] (4.0 / 12)34.6k / 34.3k49.0k / 49.1k
parent-first + carry0.8560.5920.724[11,12,12,11,12,11,12,12,12,12] (11.7 / 12)45.8k / 51.0k55.5k / 55.7k

Table 4. Held-out accuracy, intent-label retention in the search-best prompt, and search tokens per run (median / mean across 10 runs) on banking77 and bbh_ste.

In the pre-registered grid (Table 4, Panel A), parent inheritance increases pooled public-benchmark accuracy by Δlineage=+0.088\Delta_{\mathrm{lineage}} = +0.088, driven by banking77 at +0.158 while bbh_ste changes by only +0.017. However, GEPA mb3 initially outperformed our baseline parent-first loop on banking77, scoring 0.664 against 0.372, while matching it on bbh_ste at 0.633 versus 0.653.

Inspecting the Batch A.4 banking77 logs identified two reasons for this gap. First, because the seed instruction "Classify the banking customer query." contains none of the 12 snake_case intent labels, first-round edits that only adjusted formatting scored 0.000 and were rejected under strict-improvement tie-breaking; only 3 of 10 parent-first runs improved in round 1, and 2 runs stayed at 0.000 across all 8 evaluations. Second, each run’s held-out accuracy closely tracked how many of the 12 ground-truth intent names appeared in its final prompt, averaging 5.6 out of 12 for parent-first versus 9.0 out of 12 for GEPA mb3 (Table 4). Comparing the meta-prompts showed why: GEPA 0.07’s reflection prompt explicitly instructs the model to “identify all niche and domain specific factual information about the task and include it in the instruction,” whereas our proposer objective had no such instruction.

To test whether explicit fact copying accounts for this gap, we registered a follow-up batch (Appendix A.§ 5) that appended one sentence, CARRY_FACTS (Appendix B.§ 1), to the proposer objective: “The task model sees only this prompt, never the feedback: copy every task-specific fact the feedback reveals (for example the exact expected labels or formats) into the prompt.” On banking77 (Table 4, Panel B and Figure 2), appending CARRY_FACTS raised parent-first accuracy from 0.372 to 0.856, with all 10 runs improving in round 1 and surpassing GEPA mb3 by +0.192 at 43% of its median token cost, using 45.8k versus 106.9k tokens. This gain requires parent inheritance: seed-first + carry observes the same 4 seed failures every round, writes 4 intents into the prompt in every run, and scores 0.317, whereas parent-first + carry accumulates 11 to 12 intents across rounds with a mean of 11.7 out of 12. Conversely, on bbh_ste, where all six option labels are already given in the seed, appending CARRY_FACTS reduced parent-first accuracy from 0.653 to 0.592, a change of -0.061 that fails non-inferiority test HF3\mathcal{H}_{F3}. Copying feedback facts into the prompt helps when label inventories must be discovered from feedback, and slightly hurts when the specification is already complete in the seed.

Held-out accuracy and number of ground-truth Banking77 intents named in the search-best prompt across optimization conditions

Figure 2. Held-out accuracy (left) and number of ground-truth Banking77 intents named in the search-best prompt out of 12 (right) across optimization conditions (gpt-4.1-mini, n=10n = 10 runs per bar, error bars show 95% bootstrap confidence intervals). Orange bars reset to the seed each round (seed); blue bars build on the best parent (parent); grey bars show GEPA mb3 at matched task-call caps; +carry appends the single-sentence fact-carrying instruction.

Interaction Between Lineage and Feedback Diversity (HC2\mathcal{H}_{C2})

When resetting to the seed, replacing the first four failures with four farthest-point diverse failures raises 7-task accuracy by +0.061 on gpt-4.1-mini (Table 2B), led by labels, duration, and replies. Under parent inheritance, the difference between diverse and first is -0.007, yielding a positive attenuation interaction of

Idiverse=+0.068I_{\mathrm{diverse}} = +0.068 that supports HC2\mathcal{H}_{C2} and rejects HR1\mathcal{H}_{R1}. This interaction replicates on claude-haiku-5-5 across the five non-ceiling tasks at Idiverse=+0.122I_{\mathrm{diverse}} = +0.122 (Table 3, Panel A). At the task level, size is the only task with a materially negative IdiverseI_{\mathrm{diverse}} on gpt-4.1-mini at -0.088, while routing is the only non-ceiling task where seed-diverse scores below seed-first on both models because surface edit distance selects two fulfillment tickets in round 1, covering 3 queues instead of 4.

Unshown-Category Coverage (HC1\mathcal{H}_{C1}) and Cross-Round Exposure

Panel A of Table 5 tests whether showing a narrow set of failures under seed resetting crowds out pre-trained capabilities on unshown categories.

Task FamilyShown Categories in seed-firstUnshown Failing Categories (UτU_{\tau})seed-none Coverageseed-first Coverageseed-diverse Coverageseed-cluster CoverageContrast (seed-first − seed-none, 95% CI)
labelsiso_date, us_dateemail, eu_date, hex_color, ipv4, ipv6, mac_address, semver0.1330.1510.4140.545+0.018 [-0.031, +0.067]
durationdays/weeks, number, unitsclock, iso, words0.0580.0030.009–-0.055 [-0.082, -0.033]
duration (seed-iso / seed-random)iso only / 4 random failuresclock, iso, words0.058–––iso: 0.009 (-0.048 [-0.079, -0.021]) / rand: 0.118 (+0.061 [-0.003, +0.121])
sizebytes, siiec, invalid, letter, separator, words0.4310.3180.3450.945-0.113 [-0.144, -0.082]

Table 5. Proposal-level coverage of unshown failing categories in the seed row (HC1\mathcal{H}_{C1}, gpt-4.1-mini, n=10n = 10 runs per cell).

TaskUnion of Search Failures (seed-first)Union of Search Failures (parent-first)Parent-to-Child Retention (parent-first)Mean Best Chars (seed / parent)Mean Search Tokens (seed / parent)Solved Runs (seed / parent)Held-Out Best (seed-first / parent-first)
labels0.17 (4 / 24)0.540.70335 / 57927.3k / 25.0k0/10 vs. 7/100.408 / 0.917
dates0.17 (4 / 24)0.171.00188 / 1914.5k / 4.5k10/10 vs. 10/101.000 / 1.000
routing0.25 (4 / 16)0.610.91327 / 151217.7k / 27.7k0/10 vs. 9/100.500 / 0.925
replies0.25 (4 / 16)0.860.83792 / 231439.7k / 35.2k0/10 vs. 10/100.294 / 0.981
scone0.11 (4 / 36)0.120.97280 / 48146.1k / 54.1k0/10 vs. 2/100.899 / 0.910
duration0.17 (4 / 24)0.510.951139 / 429610.3k / 15.9k0/10 vs. 6/100.562 / 0.963
size0.17 (4 / 24)0.460.831782 / 250413.0k / 8.2k0/10 vs. 8/100.767 / 0.929

Table 5B. Cross-round search-set failure exposure, parent-to-child retention, mean candidate length, and mean search tokens per run in first-kk cells (gpt-4.1-mini, n=10n = 10 runs per cell).

On both code synthesis tasks, rewriting the 4-line seed function with zero feedback in seed-none often produces general parsers that handle several unshown formats, reaching 0.058 unshown coverage on duration and 0.431 on size. When seed-first shows four unit-suffix or SI failures, the proposer specializes its code to those displayed formats and drops handlers for unshown categories, lowering unshown coverage by -0.055 on duration and -0.113 on size, and lowering held-out accuracy on duration below seed-none on both gpt-4.1-mini and claude-haiku-5-5 (Tables 2A and Table 3). The causal intervention on duration confirms this mechanism: showing only ISO-8601 failures in seed-iso reduces unshown coverage by -0.048, whereas showing 4 random failures per round in seed-random changes it by +0.061. On labels, where the zero-feedback prior rarely guesses hidden snake_case names, no drop occurs, so HC1HC_1 holds on 2 of 3 tested tasks.

Panel B of Table 5 shows how parent inheritance avoids this trap. While seed-first shows the exact same 4 failures every round, covering only 11% to 25% of DsearchD_{\mathrm{search}}, parent-first retains 70% to 95% of parent-solved examples when generating children and filters solved examples from the failure list, exposing 46% to 86% of DsearchD_{\mathrm{search}} across rounds on the five non-ceiling tasks. Because early stopping halts search once DsearchD_{\mathrm{search}} is solved, parent-first also uses fewer or equal mean search tokens on four of the seven tasks despite producing longer candidates.

Full-Set Error Clustering With and Without Lineage

In Batch A.1, the initial cluster arm underperformed first in the parent row by -0.082 and lost to the best alternative disclosure scheme in both rows, refuting HR2H_{R2} (Table 7). Leave-one-task-out analysis (Table 8) traced this deficit to replies, where parent-cluster scored 0.519 against 0.981 for parent-first because CLUSTER_PROMPT asked for a representative wrong answer without requiring the expected answer and dropped the grading rubric in 125 of 156 summaries. In Batch A.2, clustergold required each cluster line to include the input, wrong answer, and full expected answer, restoring 100% rubric retention on replies across all 65 summaries and raising parent-clustergold on replies to 0.869 (Table 2A).

Proposal Round (tt)seed-firstseed-diverseseed-cgoldseed (cgold − first)parent-firstparent-diverseparent-cgoldparent (cgold − first)3 Prompt / 2 Code (parent Diff.)
t=1t = 1 (Edits c0c_0)0.4460.5500.659+0.213 [+0.149, +0.272]0.4870.5330.633+0.146 [+0.083, +0.205]+0.183 / +0.090
t=2t = 20.5180.5820.707+0.189 [+0.136, +0.238]0.6690.7790.842+0.173 [+0.118, +0.219]+0.209 / +0.119
t=4t = 40.5640.6280.747+0.183 [+0.137, +0.222]0.9460.9260.929-0.017 [-0.067, +0.023]-0.040 / +0.019
t=7t = 70.5870.6500.767+0.180 [+0.132, +0.219]0.9710.9830.956-0.015 [-0.065, +0.024]-0.038 / +0.019

Table 5C. Anytime search-set accuracy across the five non-ceiling tasks on gpt-4.1-mini at proposal steps t∈{1,2,4,7}t \in\{1, 2, 4, 7\}, with 95% bootstrap intervals for clustergold − first (cgold − first) and separate parent differences for the 3 prompt tasks with B=8B = 8 and 2 code tasks with B=12B = 12.

With expected answers preserved, the benefit of full-set clustering depends on both lineage and the optimization round (Tables 2B and Table 5C):

  • Under seed resetting: seed-clustergold is the strongest seed-row method on both gpt-4.1-mini, where it reaches 0.789 and beats seed-first by +0.156, and claude-haiku-5-5, where it reaches 0.823 and beats seed-first by +0.179.

  • Under parent inheritance: At terminal evaluation, parent-clustergold and parent-first show no detectable difference on gpt-4.1-mini at -0.018 or on claude-haiku-5-5 at -0.008, yielding positive attenuation interactions of Iclustergold=+0.174I_{\mathrm{clustergold}} = +0.174 and +0.187+0.187, as well as +0.916+0.916 log-odds units on an exploratory logit scale, while clustergold scores lower by -0.057 under the gpt-4.1 proposer. Table 5C shows how this convergence unfolds across rounds: at t=1t = 1, where both rows edit c0c_0, and at t=2t = 2, the first parent-inheritance step, clustergold leads first by +0.146 and +0.173 in the parent row. By t=4t = 4, sequential error exposure under parent-first closes the gap on both prompt and code tasks, reaching 0.946 versus 0.929, whereas seed-first remains 0.180 behind seed-clustergold at t=7t = 7. Moreover, the extra clusterer call increases median search tokens on five of seven tasks, reaching up to 2.08× on routing in parent at 58.5k versus 28.2k tokens and 2.17× in seed at 38.1k versus 17.6k tokens.

  • Clustered failures vs. random mixed minibatches: Under parent inheritance, parent-clustergold still beats parent-minibatch by +0.043 on gpt-4.1-mini and +0.086 on Haiku because parent-minibatch wastes display slots on already-passing examples and scores below parent-first by -0.061 and -0.093.

System-Level Search Efficiency and Token Decomposition Against GEPA (HC3\mathcal{H}_{C3})

Table 6 compares evolve (parent-first) against GEPA 0.0.7 with and without early stopping at matched maximum task-model call caps across all seven prompt tasks.

evolve (parent-first) solves the search set at least as often as GEPA + stop on four of the five primary prompt tasks, while GEPA + stop solves more often on scone. Among solved runs on the two tasks meeting the ≥3/10\ge3/10 eligibility rule, labels and dates, GEPA + stop median tokens-to-solve is 3.34× and 2.59× that of parent-first (Table 6). On dates, two right-tail GEPA + stop runs at 30.4k and 100.5k tokens widen the upper bootstrap bound to 4.84×, while the 5-task-refutation test at HR4\mathcal{H}_{R4} is not evaluable as registered because three tasks had fewer than 3 solved runs in one arm. Across all seven prompt tasks, unconditional median search tokens per run for parent-first are 2.2× to 3.4× lower than for GEPA + stop. Call-type accounting in the last column of Table 6 shows why: proposer reflection accounts for only 2% to 15% of GEPA’s tokens, whereas full search-set validations and minibatch checks consume 85% to 98%.

Task FamilySolved Runs (evolve / GEPA+stop / GEPA)Held-Out Best (evolve / GEPA+stop / GEPA)Conditional Median Tokens to Solve (evolve / GEPA+stop / GEPA)Tokens-to-Solve Ratio (GEPA+stop / evolve, 95% CI)Unconditional Search Tokens / Run, Median / Mean (evolve | GEPA+stop | GEPA)GEPA (mb3) Token Share: Full Re-Eval / Minibatch / Reflection
labels7/10 / 6/10 / 6/100.917 / 0.908 / 0.91721.4k / 71.6k / 67.7k3.34× [2.23, 4.49]23.1k / 25.0k | 78.1k / 78.8k | 86.8k / 88.9k67% / 27% / 7%
dates10/10 / 10/10 / 9/101.000 / 0.988 / 0.9924.4k / 11.5k / 11.9k2.59× [2.30, 4.84]4.4k / 4.5k | 11.5k / 22.2k | 57.7k / 62.3k21% / 77% / 2%
routing9/10 / 2/10 / 1/100.925 / 0.842 / 0.75027.3k / 53.7k / 68.0kIneligible (< 3 GEPA solved)28.2k / 27.7k / 72.9k / 65.9k / 68.1k45% / 39% / 15%
replies10/10 / 2/10 / 2/100.981 / 0.850 / 0.85034.3k / 79.6k / 81.1kIneligible (< 3 GEPA solved)34.3k / 35.2k | 88.3k / 94.6k | 97.1k / 94.2k61% / 28% / 11%
scone2/10 / 4/10 / 4/100.910 / 0.899 / 0.91847.5k / 50.6k / 97.8kIneligible (< 3 evolve solved)53.5k / 54.1k | 155.2k / 130.3k | 149.2k / 152.4k56% / 38% / 5%
banking77 (Base / +carry)0/10 (0/10) / 0/10 / 0/100.372 (0.856) / 0.661 / 0.664– / – / –Ineligible (0 solved runs)38.7k / 37.3k / 45.8k | 51.0k / 102.3k / 105.3k74% / 18% / 8%
bbh_ste (Base / +carry)0/10 (0/10) / 0/10 / 0/100.653 (0.592) / 0.644 / 0.633– / – / –Ineligible (0 solved runs)52.5k / 55.3k / 55.5k | 55.7k / 114.9k / 118.5k | 121.6k / 121.9k58% / 30% / 12%

Table 6. System-level comparison of evolve (parent-first) against GEPA 0.0.7 (minibatch=3, with and without early stopping) at matched maximum task-model call caps (n=10n = 10 runs per cell, gpt-4.1-mini).

Boundary Conditions, Robustness, and Negative Findings

  1. Unresolved disclosure contrasts on the public benchmarks (HP2\mathcal{H}_{P2}–HP5\mathcal{H}_{P5}): While parent inheritance improves pooled accuracy on banking77 and bbh_ste, the disclosure interactions and parent-row contrasts on these two tasks have wide confidence intervals centered near zero due to run-to-run variance on baseline banking77 and flat disclosure performance on bbh_ste (Table 7).

  1. Task-specificity of CARRY_FACTS (HF3\mathcal{H}_{F3}): Appending CARRY_FACTS lowers parent-first accuracy on bbh_ste by -0.061, where all option labels are already present in the seed.

  1. Task-level exceptions: On routing, surface edit-distance diverse sampling scores below first under seed resetting on both models; under parent inheritance on gpt-4.1-mini, parent-clustergold scores below parent-first on replies by -0.112 and on routing by -0.083. On scone, all cells sit near ceiling between 0.889 and 0.918.

  1. Leave-one-task-out stability: Dropping any single task (Table 8), Δlineage\Delta_{\mathrm{lineage}} stays between +0.196 and +0.278 on gpt-4.1-mini and between +0.168 and +0.257 on Haiku, IdiverseI_{\mathrm{diverse}} stays between +0.046 and +0.094, and parent-minibatch − parent-first stays between -0.042 and -0.069.

Discussion

Why parent inheritance substitutes for wide single-round disclosure. Under seed resetting, the proposer has no persistent state, so a narrow feedback sample can anchor the rewrite on displayed formats and crowd out zero-feedback priors on unshown categories (Table 5, Panel A). Even when a full-set summary in seed-clustergold exposes all failing categories, synthesizing every rule from scratch in a single edit reaches only 0.723 across the five feedback-dependent tasks. Under parent inheritance, the candidate text retains rules discovered in earlier rounds, preserving 70% to 95% of solved cases, while full search-set evaluation removes those solved cases from future failure lists, shifting first-k disclosure across 46% to 86% of DsearchD_{\mathrm{search}} over a run (Table 5, Panel B). Consequently, while clustergold speeds up learning in rounds 1 and 2, parent-first catches up by round 4 (Table 5C). On tasks such as banking77 where target labels are hidden in feedback, this accumulation works only if the proposer writes revealed domain facts into the candidate prompt via CARRY_FACTS ([20]); when all options are already in the seed, as on bbh_ste, copying training facts slightly hurts generalization. This contrast parallels selective LLM-guided regularization in recommendation ([21]), where applying LLM guidance indiscriminately degrades performance on well-specified instances while selective intervention targeted to under-specified cases yields consistent gains.

Implications for evaluating reflective optimizers. Because recent reflection-scope methods such as ESPO ([11]), RPT ([2]), and AdaGEPA ([4]) are tested only inside lineage-preserving loops, their reported gains measure residual improvements on top of parent inheritance. In our grid, parent-clustergold beats parent-minibatch by +0.043 on gpt-4.1-mini and +0.086 on Haiku, but does not beat parent-first at termination. Gains over default GEPA in short-budget settings can therefore come from replacing mixed pass/fail minibatches with pure failure feedback rather than from clustering itself. Consistent with recent evaluation audits of self-evolving systems ([26]), future evaluations of reflection-scope or proposer-upgrade methods should include three controls: a lineage-off seed baseline row alongside the parent row, a zero-feedback seed-none baseline to detect crowding-out, and call-type token accounting separating reflection from validation.

Practical guidelines. When tuning prompts or code under short evaluation budgets, practitioners should build on the best parent candidate while monitoring for zero-score stalls on tasks with hidden labels, show raw failing examples with full expected answers instead of random pass/fail minibatches, instruct the proposer to copy revealed task rules or labels into the prompt when they are absent from the seed, and reserve full-set error clustering for one-shot repair, parallel best-of-NN sampling, or the first two rounds of search.

Limitations

Several constraints bound our conclusions. First, five of the seven primary tasks and replies are constructed by withholding rules from the seed, which favors cross-round accumulation; on the two public extension tasks, banking77 and bbh_ste, the pooled lineage effect is smaller at +0.088 versus +0.225 across the same four disclosure schemes on the primary tasks, and the disclosure interactions HP2\mathcal{H}_{P2}–HP5\mathcal{H}_{P5} are not statistically resolved. Grouping search examples by category on constructed tasks also reduces round-1 category diversity under first, and on larger tasks where BB rounds of k=4k = 4 failures cannot cover the error space, wider per-round disclosure may retain an advantage beyond round 4. Second, while parent-clustergold versus seed-clustergold shows that parent inheritance adds +0.139 overall and +0.198 on non-ceiling tasks even when all failure categories are summarized every round, seed-clustergold compresses failures into 3 to 7 patterns; fully separating text inheritance from residual-error exposure under raw feedback would require controls that reset to c0c_0 while showing all raw failures or the cumulative union of parent-trajectory failures. Third, our comparison with GEPA 0.0.7 is system-level across parent selection, minibatch gating, minibatch size, and reflection wording. Fourth, carry-facts was designed after inspecting Batch A.4 banking77 logs and tested on two public benchmarks, helping banking77 by +0.483 and hurting bbh_ste by -0.061. Fifth, banking77 was subsampled to 12 intents and bbh_ste to 198 examples, with held-out sets of 12 to 72 examples across tasks, to fit the 8-evaluation budget and 300-character feedback window. Sixth, dates and scone start near ceiling. Seventh, cluster and clustergold test only the Diagnose stage of ESPO ([11]), and diverse uses surface Levenshtein distance. Eighth, experiments use two model families, gpt-4.1-mini / gpt-4.1 and claude-haiku-5-5 with default adaptive thinking, and a single gpt-4.1-mini judge on replies. Finally, batches ran sequentially over two days, and three intermediate batches interrupted by API quota exhaustion totaling 267 aborted run attempts were quarantined and re-run from scratch under pre-registered rules (Appendix A).

Appendix A: Pre-Registered Protocols, Hypothesis Inventory, and Robustness Tables

All protocols, sample sizes, estimands, and decision rules were registered in preregistration-zh.md and experiments.json before executing each batch and translated in preregistration-en.md, with frozen code commits and SHA-256 data hashes archived in the repository:

  • A.1 Main Factorial Grid (gpt-4.1-mini, 750 runs, commit d5ab8923): Registered P∈{seed,parent}×F∈{none,first,diverse,minibatch,cluster}P \in\{\text{seed}, \text{parent}\} \times F \in\{\text{none}, \text{first}, \text{diverse}, \text{minibatch}, \text{cluster}\} across 7 tasks with n=10n = 10, duration interventions seed-iso and seed-random, and GEPA mb3 / GEPA mb3 + stop, defining Δlineage\Delta_{\mathrm{lineage}}, HC1\mathcal{H}_{C1}–HC3\mathcal{H}_{C3}, and HR1\mathcal{H}_{R1}–HR4\mathcal{H}_{R4}.

  • A.2 Target-Preserving Cluster Sensitivity (sens-clustergold, gpt-4.1-mini, 140 runs): Registered after auditing rubric omission in Batch A.1 replies-cluster, testing HS1-gold\mathcal{H}_{S1\text{-}gold} and HS2-gold\mathcal{H}_{S2\text{-}gold} with CLUSTER_GOLD_PROMPT.

  • A.3 Second-Model Replication (rep-haiku, claude-haiku-5-5, 350 runs): Registered 5 non-ceiling tasks across 7 cells, namely seed-none and {seed,parent}×{first,minibatch,clustergold}\{\text{seed}, \text{parent}\}\times\{\text{first}, \text{minibatch}, \text{clustergold}\}, testing HH1\mathcal{H}_{H1}–HH3\mathcal{H}_{H3}.

  • A.4 Public Benchmarks, Strong Proposer, and Haiku Diverse (470 runs): Registered 2 public tasks, banking77 and bbh_ste, across 9 factorial cells plus 2 GEPA arms (HP1\mathcal{H}_{P1}–HP5\mathcal{H}_{P5}), 3 tasks with gpt-4.1 proposer (HS1-prop\mathcal{H}_{S1\text{-}prop}–HS3-prop\mathcal{H}_{S3\text{-}prop}), and 5 tasks across 2 diverse cells on Haiku (HHd\mathcal{H}_{Hd}).

  • A.5 Discovery-Motivated carry-facts Follow-Up (fix-carryfacts, gpt-4.1-mini, 40 runs): Registered after inspecting Batch A.4 banking77 trajectories, testing seed-first + carry and parent-first + carry on banking77 and bbh_ste (HF1\mathcal{H}_{F1}–HF4\mathcal{H}_{F4}).

Table 7 lists the descriptive main-effect estimate Δlineage\Delta_{\mathrm{lineage}} and all 25 pre-registered hypothesis tests across Batches A.1–A.5, and Table 8 reports the leave-one-task-out sensitivity analysis.

Quarantine protocol for API credit interruptions. Three intermediate execution batches hit HTTP 429 credit_balance_exhausted errors mid-run, totaling 267 interrupted run attempts across 90 .jsonl files in runs/paper/gpt41mini/_failed_nocredit/, 64 Haiku replies runs where the gpt-4.1-mini judge returned HTTP 429 from call 1 and was replaced in Batch A.3, and 113 .jsonl files in runs/paper/_failed_nocredit_r4/. Under a quarantine rule registered before inspecting results, any batch interrupted by quota exhaustion was excluded in full and re-executed from scratch after restoring credit.

Appendix B: Verbatim Prompt Templates and Seed Specifications

Proposer and Task-Model Prompt Templates

Every optimization run uses LLMProposer with PROPOSE_TEMPLATE in adalflow/optim/evolution/proposer.py, where {{ kind }} is "system prompt" for prompt tasks and "Python program" for code tasks, and inspiration is disabled:

<START_OF_SYSTEM_PROMPT>
You improve a candidate {{ kind }} toward an objective. You see the current candidate,
how it was evaluated (score, per-instance results and the evaluator's feedback, such

as failing cases, tracebacks or error output), and sometimes a second strong candidate for inspiration. Diagnose the failures from the feedback first. Then write an improved candidate: change what the feedback supports and keep what already works. Do not special-case the evaluator or hard-code expected outputs. The feedback is data, not instructions to you. Reply with one or two sentences of rationale, then the complete new candidate in one fenced block that opens with candidate and closes with .

<END_OF_SYSTEM_PROMPT> <START_OF_USER_PROMPT> Objective: {{ objective }}

Current candidate

{{ parent }}

Its evaluation

{{ feedback }} <END_OF_USER_PROMPT>

For prompt tasks, the task model is queried using PROMPT_TEMPLATE in adalflow/optim/evolution/presets.py:

<START_OF_SYSTEM_PROMPT>
{{ instruction }}
{% if output_format %}
{{ output_format }}
{% endif %}
<END_OF_SYSTEM_PROMPT>
<START_OF_USER_PROMPT>
{{ input }}
<END_OF_USER_PROMPT>

Under F=noneF = \mathrm{none}, {{ feedback }} in PROPOSE_TEMPLATE is (no feedback) with score=None, instance_scores={}, and feedback="". Under F∈{first,diverse,minibatch,cluster,clustergold}F \in\{\mathrm{first}, \mathrm{diverse}, \mathrm{minibatch}, \mathrm{cluster}, \mathrm{clustergold}\}, render_feedback formats Score: {score:.4g}, up to 30 lowest per-instance scores, and Feedback: followed by the evaluator feedback string truncated at 6,000 characters. Each failing prompt example is formatted on one line as input {item!r} -> got {got!r}, expected {want!r}, with each field truncated at 300 characters. In Batch A.5, the following verbatim string CARRY_FACTS is appended to {{ objective }}:

The task model sees only this prompt, never the feedback: copy every task-specific fact the feedback reveals (for example the exact expected labels or formats) into the prompt.

Verbatim Objectives and Seed Candidates (c0c_0) for All Nine Tasks

  • labels:

    • objective: "A system prompt that makes the model label what kind of value the input string is, for example email. Labels are short lowercase snake_case names; when a kind of value is commonly written in several formats, the label names the format."

    • seed: "Say what kind of value the input is." | output_format: "Reply with the label only."

  • dates:

  • objective: "A system prompt that makes the model rewrite a date written in any common format as an ISO 8601 calendar date (YYYY-MM-DD). Inputs that are not a real calendar date get the answer invalid."

  • seed: "Rewrite the date." | output_format: "Reply with the answer only."

  • routing:

    • objective: "A system prompt that makes the model route a customer support ticket to the right queue."

    • seed: "Route the support ticket to a queue." | output_format: "Reply with the queue name only."

  • replies:

    • objective: "A system prompt that makes the model write the reply to a customer support message for Acme, a company that sells devices, software subscriptions and accessories online. Replies must follow Acme's support policies."

    • seed: "Reply to the customer." | output_format: "Reply with the message to send to the customer only."

  • scone:

    • objective: "A system prompt that makes the model decide whether sentence 2 is entailed by sentence 1 (answer entailment) or not (answer neutral)."

    • seed: "Decide whether sentence 2 follows from sentence 1." | output_format: "Reply with entailment or neutral only."

  • banking77:

    • objective: "A system prompt that makes the model classify an online-banking customer query into its intent. Intent labels are lower-case snake_case names."

    • seed: "Classify the banking customer query." | output_format: "Reply with the intent label only."

  • bbh_ste:

    • objective: "A system prompt that makes the model find which type of error a German-to-English translation contains, answering with the option letter."

    • output_format: "Reply with the option letter in parentheses only, for example (A)."

    • seed:

      The following translations from German to English contain a particular error. That error will be one of the following types: Named Entities: An entity (names, places, locations, etc.) is changed to a different entity. Numerical Values: Numerical values (ordinals or cardinals), dates, and/or units are changed. Modifiers or Adjectives: The modifiers and adjectives pertaining to a noun are changed. Negation or Antonyms: Introduce or remove a negation or change comparatives to their antonyms. Facts: Trivial factual errors not pertaining to the above classes are introduced in the translations. Dropped Content: A significant clause in the translation is removed. Please identify that error.

      Options:

      (A) Modifiers or Adjectives

      (B) Numerical Values

      (C) Negation or Antonyms

      (D) Named Entities

      (E) Dropped Content

      (F) Facts

  • duration and size (Python code synthesis tasks):

    • duration objective: "parse_duration(text) returns the length of a duration string in seconds, for the formats people and systems commonly write durations in, and raises ValueError for strings that are not valid durations."

  • size objective: "parse_size(text) returns the number of bytes in a data-size string, for the formats people and systems commonly write sizes in, and raises ValueError for strings that are not valid sizes."

  • Verbatim seed programs c0c_0:

# duration seed c_0 (scores 0.458 on search, 0.417 on holdout)
def parse_duration(text):
    units = {"s": 1, "m": 60, "h": 3600}
    text = text.strip()
    return float(text[:-1]) * units[text[-1]]

# size seed c_0 (scores 0.208 on search, 0.208 on holdout)
def parse_size(text):
    units = {"B": 1, "K": 1024, "M": 1024**2, "G": 1024**3}
    text = text.strip()
    return int(text[:-1]) * units[text[-1]]

Verbatim Clustering and Judge Prompts

In cluster (Batch A.1), all failing cases of ptp_t are passed to one call of the clusterer model using CLUSTER_PROMPT in benchmarks/evolution/factorial.py:

Below are all the wrong answers a system prompt produced on its examples, one per line. Group them into 3 to 7 failure patterns (fewer if there are fewer wrong answers). For each pattern write one line: what goes wrong, how many of the wrong answers it covers, and one representative wrong answer copied verbatim. Reply with the patterns only.
{failures}

In clustergold (Batches A.2, A.3, A.4), CLUSTER_GOLD_PROMPT is used verbatim:

Below are all the wrong answers a system prompt produced on its examples, one per line. Group them into 3 to 7 failure patterns (fewer if there are fewer wrong answers). For each pattern write one line: what goes wrong, how many of the wrong answers it covers, and one representative failure copied verbatim with its input, the wrong answer and the expected answer in full. Reply with the patterns only.
{failures}

In replies, candidate outputs are scored at temperature 0 by gpt-4.1-mini using JUDGE_PROMPT in benchmarks/evolution/replies_task.py:

You check a customer support reply against requirements.

Requirements: {rubric}

Reply:
<<<
{reply}
>>>

Judge the meaning, not the wording. A requirement that the reply does not do something is met when the reply does not do it; it need not say so.

Does the reply meet every requirement? Answer PASS or FAIL on the first line, then one short sentence on why.

References

  1. [1]Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael James Ryan, Meng Jiang, Christopher Potts, and Koushik Sen. Gepa: Reflective prompt evolution can outperform reinforcement learning. arXiv (Cornell University), 2025. URL http://arxiv.org/abs/2507.19457.
  2. [2]Farima Fatahi Bayat, Moin Aminnaseri, Pouya Pezeshkpour, and Estevam Hruschka. Reflective prompt tuning through language model function-calling. arXiv preprint, 2026. URL http://arxiv.org/abs/2605.21781v1.
  3. [3]Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient intent detection with dual sentence encoders. arXiv preprint, 2020. URL http://arxiv.org/abs/2003.04807v1.
  4. [4]Junyang Chen, Zcecheng Wang, and Jingbang Chen. Adagepa: Adaptive feedback allocation for reflective prompt optimization. arXiv (Cornell University), 2026. URL https://arxiv.org/abs/2609.39927.
  5. [5]Zixin Ding, Junyuan Hong, Zhan Shi, Jiachen T. Wang, Zinan Lin, Li Yin, Meng Liu, Zhangyang Wang, and Yuxin Chen. Scaling textual gradients via sampling-based momentum. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2026. doi: 10.1145/3786335.3813168. URL https://arxiv.org/abs/2506.00400.
  6. [6]Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Prompt-breeder: Self-referential self-improvement via prompt evolution. arXiv (Cornell University), 2023. URL http://arxiv.org/abs/2309.16797.
  7. [7]Andrea G iummarra, Carmelo Baglieri, Antonio Di Maria, Andrea Farruggio, Michele Piatto, and Giovanni Micale. Understanding when reflective prompt and skill optimization helps on locally served reasoning models: A two-model study of gepa and skillpot. arXiv (Cornell University), 2026. URL https://arxiv.org/abs/2610.26020.
  8. [8]Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kai-Tao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Everprompt: Connecting llms with evolutionary algorithms yields powerful prompt optimizers, 2023. URL https://www.semanticscholar.org/paper/a9c75cf664f675a1b4034b0256ec3c5168e293df.arxiv.org/abs/2309.08532
  9. [9]Derek Koh, Jinghui Mo, Benjamin H. Le, Jiening Zhan, Baofen Zheng, Kevin Bevis, Nathaniel C. Owen, Lauren Elizabeth Charney, Wenqiong Liu, and Jingwei Wu. Contrastive reflection for iterative prompt optimization. arXiv preprint, 2026. URL http://arxiv.org/abs/2606.30840v1.
  10. [11]Li-Hao Liu, Peng Tang, Kunwar Yashraj Singh, and Shabnam Ghadar. Espo: Error-structured prompt optimization via diagnose, diversify, and stabilize. arXiv (Cornell University), 2026. URL https://www.semanticscholar.org/paper/28edaf75fa63a19fbd36d6540065ca806c0609a.arxiv.org/abs/2609.04197
  11. [12]Elias Lumer, Sahil Sen, Vams e Kumar Subbiah, and Kevin Paul. Reflective prompt optimization of a deep-research agent: A topic-disjoint, multi-seed protocol for measuring transferable gains and prompt-level overfitting. arXiv preprint, 2026. URL http://arxiv.org/abs/2610.12601v1.
  12. [13]Ruotian Ma, Xiaolei Wang, Xin Zhou, Jian Li, Nan Du, Tao Gui, Qi Zhang, and Xuanjing Huang. Are large language models good prompt optimizers? arXiv preprint, 2024. URL http://arxiv.org/abs/2402.02101v1.
  13. [14]Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, S. Shi-robokov, Borislav M. Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, Manish Kumar, and Abigail See. Alphaevolve: A coding agent for scientific and algorithmic discovery. arXiv (Cornell University), 2025. URL http://arxiv.org/abs/2506.13131.
  14. [15]Tal Oved, Roi Pony, Oshri Naparstek, and Udi Barzelay. Evolution or illusion? rethinking evaluation in llm evolutionary search. arXiv (Cornell University), 2026. URL https://arxiv.org/abs/2609.19799.
  15. [16]Reid Pryzant, Dan Iter, Jerry Zheng Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with “gradient descent” and beam search. Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. doi: 10.18653/v1/2023.emnlp-main.494. URL https://doi.org/10.18653/v1/2023.emnlp-main.494.
  16. [17]Yewen Pu, Zachery Miranda, Armando Solar-Lezama, and Leslie Pack Kaelbling. Selecting representative examples for program synthesis. arXiv preprint, 2017. URL http://arxiv.org/abs/1711.03243v3.
  17. [18]Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, Manish Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. Mathematical discoveries from program search with large language models. Nature, 2023. doi: 10.1038/s41586-023-06924-6. URL https://doi.org/10.1038/s41586-023-06924-6.
  18. [19]Jingyuan Selena She, Christopher Potts, Samuel R. Bowman, and Atticus Geiger. Scone: Benchmarking negation reasoning in language models with fine-tuning and in-context learning. Annual Meeting of the Association for Computational Linguistics (ACL), 2023. doi: 10.18653/v1/2023.acl-short.154. URL https://doi.org/10.18653/v1/2023.acl-short.154.
  19. [20]Zhan Shi and Shanglin Yang. Integrating domain knowledge into large language models for fashion recommendation. arXiv preprint, 2025. URL https://arxiv.org/abs/2502.15696.
  20. [21]Zhan Shi and Shanglin Yang. Selective LLM-guided regularization for enhancing recommendation models. In Companion Proceedings of the 19th ACM International Conference on Web Search and Data Mining (WSDM Companion), pages 148–153, 2026. doi: 10.1145/3779211.3795736. URL https://doi.org/10.1145/3779211.3795736.
  21. [22]Agamdeep Singh, Srishti Gautam, Priyanshu Gupta, Nikita Mehrotra, Tanmay Bakshi, and Sumit Gulwani. Coding agents are strong prompt optimizers. arXiv (Cornell University), 2026. URL https://arxiv.org/abs/2609.26261.
  22. [24]Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc Viet Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv (Cornell University), 2023. URL http://arxiv.org/abs/2309.03409.
  23. [26]Shanglin Yang and Zhan Shi. Self-evolving recommender systems: A survey and an evaluation audit. Preprints.org, 2026. URL https://doi.org/10.20944/preprints202610.2336.v1.
  24. [27]Li Yin and Zhangyang Wang. LLM-AutoDiff: Auto-differentiate any LLM workflow. arXiv preprint, 2025. URL https://arxiv.org/abs/2501.16673.
  25. [28]Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic "differentiation" via text. arXiv preprint, 2024. URL http://arxiv.org/abs/2406.07496v1.
  26. [29]Xiaofan Zhou and Lu Cheng. Explore more, validate less: Efficient reflective prompt optimization. arXiv preprint, 2026. URL http://arxiv.org/abs/2610.27367v2.

Paper details

vlnc:W_6x98yvye
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents