betaFeedback

WorksComputer Science

Generated Evaluation Suites as Diversity-Driven Search for Real Agent Regressions

Abstract

Generating an evaluation suite from an agent's specification is diversity-driven search: a generator fills a descriptor space (requirements ×\times personas ×\times scenario kinds), and the suite it emits is an archive, then scored like a benchmark by its mean pass rate, which throws away the search's product. We inject 7 controlled defects into a τ2τ^2-bench agent across two domains. Coverage yields a ceiling: if a defect's traced footprint holds bb scenarios that pass at baseline, it can itself contribute at most b/Nb/N to a suite of NN. For 4 of 5 traceable defects that ceiling lies at or below the detection threshold θ=0.10θ= 0.10, so no within-footprint effect can move the suite past it; in both domains the tool-removal defect destroys every covered scenario it can (exactly its ceiling) while the suite-level pass rate moves the other way. Defects split into two regimes: trace-aligned ones, where per-cell scoring recovers the tool-removal defect in full and reveals a second as a significant improvement, and trace-orthogonal ones, whose footprint is behavioural: forced escalation is detected in retail at a traced coverage of exactly zero. Diversity along the generator's own axes helps only where it aligns: at fixed suite size a happy-path-only suite detects nothing, while a kind-diverse one detects the orchestration defect in 5 of 20 draws. Evaluation archives should be scored per cell and should publish coverage as a first-class property.

Introduction

An agent’s specification (policy, tool schemas, routing rules, prompts) changes weekly while its evaluation suite is written once. A widely adopted response is to generate the suite from it: extract requirements, generate personas, cross them into scenarios, grade against predicates [8]. Read as search this is a familiar object: the generator fills a descriptor space, the suite is viewed as a descriptor-indexed archive [11, 13], and the argument for it is the argument for divergent search: coverage beats optimisation of a single objective [9].

The archive is then scored like a benchmark: run every scenario, average, compare. The mean over a diverse archive is dominated by the cells where nothing interesting happened. We ask what that mismatch costs when the question is practical (did this suite notice that the agent regressed?), answering it with mutation testing [4, 5] over controlled defects injected into a τ2\tau^2-bench agent [16, 2]. The answer is not that the search is bad, but that coverage and score are misaligned in a way computable from baseline results alone, before the defect is introduced: for most defects the ceiling on the defect-attributable signal already sits below the detection threshold.

Contributions. We formulate generated suites as diversity-driven search and give a coverage measure over the generator’s own descriptors, with a ceiling bounding what suite-level scoring can detect (§ 3.3); show over 7 defects and two domains that the ceiling binds for 4 of 5 traceable defects, and that a defect can saturate its ceiling while the suite-level pass rate moves the opposite way (§ 4.1); separate trace-aligned from trace-orthogonal defects, each demanding a different unit of scoring (§ 4.2); and report two diversity ablations that locate where diversity does and does not buy detection (§ 4.3). We will release the pipeline, mutation set, suites and transcripts.

Suites as archives

Diversity-driven search. The archive formulation above comes from novelty search, MAP-Elites and quality-diversity [9, 11, 13]. The idea reaches language models mainly through adversarial generation: Rainbow Teaming casts red-teaming as quality-diversity over a descriptor grid [14, 12]. Those methods search for inputs that break a fixed model; we study the dual, a fixed suite asked to notice when the system changes underneath it.

Agents, benchmarks, mutation analysis. τ\tau-bench, τ2\tau^{2}-bench [16, 2] and AgentBench [10] score agents on fixed task sets that are easily overfit [7]; MAST [3] and issue-tracker mining [19, 1, 17] catalogue the failures that dominate in deployment, and our mutation classes come from them. Specification-grounded generation [8] composes LLM test generation [6], simulated users [15] and LLM judges [18]. Mutation analysis scores a suite by the seeded faults it kills [4, 5]; here the mutated object is an agent configuration and the tests are stochastic conversations, so a kill must be defined statistically, and that definition, we show, can constrain detection before mutant evaluation.

Method

The archive

The generator consumes a specification SS (for τ2\tau^{2}-bench v1.0.1 (fc0055d), the domain policy document) plus tool schemas and emits executable scenarios with grading predicates over three descriptors: the requirement exercised, the persona driving the conversation, and the scenario kind (happy path, edge case, adversarial-benign). In both domains it extracts 30 requirements, writes 8 personas, and emits 3 scenarios per requirement, one per kind: a fully covered 30×330 \times3 grid of 90 scenarios. All 90 run on airline; retail evaluates a seeded 50 of them, and its S3 deltas are over those 50. Simulation runs in the benchmark’s unmodified harness. Models: gpt-5.6-luna as agent, user simulator and judge, gpt-5.6-terra as generator; temperature is left at the provider default because the agent model rejects temperature 0, and reasoning is disabled (zero reasoning tokens in the usage records). All runs took place on 29 August 2026 (Appendix § 13).

The defect space

Each mutation is a minimal, realistic edit to the agent configuration that leaves the harness untouched, modelling a failure class documented in the mining studies above. We instantiate 7 in the airline domain, one per operator: R1 deletes a policy clause (the 24-hour cancellation rule), R2 inverts a policy condition, T1 removes a tool (cancel_reservation), T2 renames a tool-schema parameter, O1 inverts the escalation rule so the agent always transfers to a human, and C1 and C2 truncate the policy to its first 40% and history to the last 4 turns (Appendix § 7). This is a pilot-scale set: each result is a statement about one instance, not its operator class.

Scoring, coverage, and the ceiling

Let p(Σ,A)p(\Sigma, A) be the pass rate of agent AA on suite Σ\Sigma and AmA_m be AA with mutation mm. A generated scenario passes on its programmatic end-state check where one exists (61 of 90 on airline) and on the judge’s verdict otherwise; hand-authored tasks pass on the benchmark’s own checker. Score, specified before running experiments. Σ\Sigma detects mm if Δm=p(Σ,A)−p(Σ,Am)\Delta_m = p(\Sigma, A) - p(\Sigma, A_m) exceeds θ=0.10\theta= 0.10 and a 2,000-sample paired bootstrap over scenarios yields a 95% interval excluding zero. Why θ=0.10\theta= 0.10. We fixed θ\theta in the experiment plan, together with the sensitivity values 0.05 and 0.15, as a practical floor on a regression worth reporting: a ten-point drop in pass rate, 5 of 50 hand-authored tasks or 9 of 90 generated scenarios. It is a convention, not a quantity derived from the data, so §4.1 reports every conclusion that depends on it as a function of θ\theta. Suites: S1, the hand-authored tasks; S2, the generated suite subsampled to ∣S1∣|S1| over 20 draws (detected if a majority detect); S3, the full generated suite. Each scenario runs once per configuration.

Coverage. For each mutation we compute mechanically, from the archive’s own descriptors, the set CmC_m of scenarios exercising the mutated element: for S3, those whose recorded requirement is the one mutated or whose grading names the affected tool; for S1, tasks whose criteria or instruction involve it. CmC_m is knowable before any conversation runs.

The ceiling. A defect can only change a scenario it reaches, so if bb of the ∣Cm∣\lvert C_m\rvert covered scenarios pass at baseline, the most it can itself contribute to the suite-level drop over a suite of NN is every one of those bb flipping: ceil⁡(m,Σ)=b/N\operatorname{ceil}(m,\Sigma)=b/N. Since bb is read off the baseline pass outcomes, the ceiling is computable from baseline results alone, before the defect is introduced. When ceil⁡(m,Σ)≤θ\operatorname{ceil}(m,\Sigma)\le\theta, the within-footprint contribution to Δm\Delta_m cannot exceed θ\theta; an observed Δm\Delta_m can still exceed the ceiling, through run-to-run noise or through behavioural effects the trace does not capture. The bound assumes the trace captures the affected set; for T2 it does not, since the renamed read tool is named in no grading criteria. Post-hoc scores, defined after those results and labelled as such throughout: scoped detection applies the same test to CmC_m alone; grading views recompute Δm\Delta_m under programmatic-only and judge-only grading.

Results

The ceiling is binding

Under the specified score the generated suite detects 1 of 7 airline defects and the hand-authored suite 0 of 7, as does the size-matched S2; baseline pass rates 0.50 (S1) and 0.88 (S3). Table 1 shows most of that outcome was bounded before the defects were introduced: of the 5 airline defects with a traceable footprint, 4 have a ceiling at or below θ\theta on both S1 and S3. R1’s S3 coverage is zero scenarios, so its ceiling is exactly zero: the archive has no cell for the deleted clause, and a requirement-traced suite can report that about itself; a hand-authored suite cannot.

∣Cm∣/N\lvert C_m\rvert/Nceilwhole-suite Δ\Delta [95% CI]scoped Δ\Delta [95% CI]
airline
R1S113/500.04-0.06 [-0.22, +0.10]-0.31 [-0.54, -0.08]
R1S30/900.00+0.04 [-0.03, +0.13]n/a
T1S17/500.06-0.04 [-0.18, +0.10]+0.43 [+0.14, +0.86]†
T1S312/900.09+0.08 [0.00, +0.17]+0.67 [+0.42, +0.92]†
O1S39/900.10+0.14 [+0.04, +0.26]†+0.11 [0.00, +0.33]
C1S351/900.49+0.02 [-0.07, +0.10]+0.08 [-0.04, +0.20]
retail
O1S10/480.00+0.17 [+0.04, +0.31]†n/a
T1S114/480.13-0.02 [-0.21, +0.17]+0.43 [+0.21, +0.71]†

Table 1. Coverage, ceiling, and what each score sees. ∣Cm∣/N\lvert C_m\rvert/N is the traced footprint over suite size, ceil is b/Nb/N, positive Δ\Delta a drop in pass rate, † a detection, “n/a” a coverage hole; remaining rows in Appendices C and D.

Sensitivity to θ\theta. Both halves of this outcome move with θ\theta, and we report them over its whole range (Appendix F). Since the interval condition does not depend on θ\theta, each action holds for every θ\theta below its Δ\Delta: airline O1 on S3 below 0.14, retail O1 on S1 below 0.17, and T2 on airline S3 joins below 0.10; no airline S1 or S2 detection appears at 0.05, 0.10 or 0.15. The ceiling count is not robust to lowering θ\theta: 4 of 5 traceable defects are ceiling-bound for every θ\theta from 0.10 up to C1’s ceiling of 0.49, but 3 at 0.09, 2 at 0.08 and only R1 at 0.05. Lowering θ\theta frees R2 and T1 from the ceiling yet detects neither, because their whole-suite intervals do not exclude zero; the one added detection is T2, whose affected set the trace does not capture.

A defect can saturate its ceiling and still read as an improvement. Removing cancel_reservation (T1) fails every hand-authored task whose checks require it (3 of 3 passing at baseline on airline, 6 of 6 on retail): exactly the ceiling. Yet the airline pass rate moves from 25 to 27 of 50 tasks and retail’s from 20 to 21 of 48: off-target drift exceeds what the defect destroyed. In both domains the score reports a total, verified tool breakage as a small improvement: averaging over the archive does not merely dilute the signal, it inverts it. C1 is the complement: the study’s highest coverage (51 of 90, ceiling 0.49) and still no detection, so a generous ceiling is necessary, not sufficient.

Two regimes

Trace-aligned defects (R1, R2, T1) have footprints the generator’s descriptors express: a requirement, a tool. Coverage is small and the ceiling binds; scoring at the covered cells recovers T1 in full: scoped Δ=+0.43\Delta= +0.43 (airline S1), +0.67+0.67 (airline S3) and +0.43+0.43 (retail S1), every interval excluding zero. Three reruns of each covered airline S1 task, run after the main analysis as a diagnostic, qualify the size but not the breakage: the mutant passes none of 21 trials, while the baseline passes only 3 of 21, all on one task, so two of the three single-run baseline passes did not recur and the rerun scoped Δ\Delta is 0.14. Recovery is not automatic: R2’s scoped Δ\Delta reaches θ\theta on neither suite (Appendix § 9), and R1’s is a statistically significant scoped improvement (Δ=−0.31\Delta= -0.31, interval excluding zero): the mutant is better on the 13 tasks it touches, because the baseline agent was over-refusing and the defect removed the clause it over-applied. A suite-level difference sums regressions and accidental fixes; per-cell scoring separates them.

Trace-orthogonal defects are the other regime; O1 is the clean case. Forcing escalation raises the share of airline hand-authored conversations containing a transfer from 36% to 84%, but that footprint is behavioural, not requirement-shaped, and the archive has no descriptor for it. Only 9 of 90 airline generated scenarios name escalation; on retail the generator extracted no escalation requirement and none of the five transfer-naming tasks fell in the seeded subsample, so traced coverage is zero on both suites. Coverage predicts nothing here: at a ceiling of exactly zero, retail O1 is detected on S1 at Δ=+0.17\Delta= +0.17 [+0.04, +0.31], and airline O1 on S3 is the study’s only specified-score detection, and a threshold-fragile one, absent at θ=0.15\theta= 0.15 (Appendix § 9). For these defects the whole suite is already the right unit, and scoping does not help: airline O1’s scoped interval touches zero on both suites. The regime also decides which grading view can see the defect: under programmatic end-state grading O1 scores no worse (airline S3 Δ=+0.07\Delta= +0.07, retail S3 Δ=0.00\Delta= 0.00), because where the expected end state is reachable by inaction an end-state check cannot tell doing nothing from doing the right thing; the transcript judge sees it in both domains (Δ=+0.22\Delta= +0.22 [+0.10, +0.36] airline, +0.20+0.20 [+0.06, +0.34] retail).

Where diversity buys detection

Both ablations reuse the conversations already run (Appendix § 11). Scenario kind, at fixed suite size, is the one size-matched diversity contrast here, and it is positive, though exploratory, since all-kind draws also shift requirement composition, so kind is not varied alone. Holding the airline suite at 30 scenarios, a happy-path-only suite detects none of the 7 defects, while kind-diverse suites of the same size detect 6 times across 140 draw-by-mutation trials, 5 of them O1, detected in 5 of 20 draws. The gain falls where the two-regime analysis predicts: edge and adversarial-benign scenarios provoke the escalation path, supplying the behavioural coverage the requirement descriptor cannot express. Persona count is inconclusive. Detection rises with persona count then falls at the maximum in both domains, but sub-suite size grows with persona count, confounding size with diversity; we see no curve, only that turning the persona dial up does not monotonically buy detection.

Implications

Score the archive per cell, not by its mean. A generated suite already records which requirement each scenario exercises; a suite-level pass rate discards that and, as § 4.1 shows, can invert the sign of a total breakage. Publish coverage as a first-class suite property. The ceiling needs only baseline results, so a suite can state before a defect is introduced where its own scoring is weakest. Match the descriptor space to the defect space. Requirements and personas do not span the behaviours a defect can change; a behavioural descriptor (does this conversation escalate, does it call this tool) would have given O1 a cell in both domains, making quality-diversity search over the defect space the natural next step.

Designing and learning behavioural descriptors. A behavioural descriptor is a function of the trajectory, not of the specification, so it is computed from transcripts rather than read off the generator. Designed descriptors come from the agent’s action vocabulary, which its tool schemas and policy already enumerate: each tool call, each terminal action (write, refuse, escalate) and the read/write split gives a binary cell that a deterministic trace parser evaluates on baseline transcripts. Coverage and the ceiling carry over unchanged, with CmC_m the scenarios whose baseline trajectory occupies the affected cell, still computable before the defect is introduced. O1’s footprint is such a cell: the share of airline hand-authored conversations that escalate moves from 36% to 84%. Learned descriptors serve where no single action names the failure: cluster baseline trajectories by tool-call sequence and use cluster membership as the cell, so a defect appears as mass moving between cells, testable with the same paired bootstrap on occupancy instead of pass rate and without a grading predicate. An occupancy shift flags changed behaviour, not by itself a regression; it says where to grade. Two cautions follow from our data. A cell the baseline rarely visits has a small ceiling of its own, so behavioural cells must be filled by search that targets them; edge and adversarial-benign scenarios already do part of this (§ 4.3). And the transcript judge partly sees behaviour already: its strictness concentrates on O1 in both domains (Appendix § 8), so judge-check disagreement is a cheap pointer to a missing behavioural cell. We have not evaluated either construction here.

Limitations and conclusion

We study two domains of one benchmark family and one agent model; the user simulator and judge are that same model, so all three may share blind spots, and judge-programmatic agreement is modest (Appendix § 8). Pass rates carry run-to-run noise the bootstrap only partly captures; on S1 intervals are about ±0.15 wide, as large as θ\theta, limiting power independently of the ceiling. Every operator is instantiated once per domain; T1 and O1 have a second instance on retail, with a different tool and policy, and each falls in the same regime as on airline, but no result generalises to its operator class; two to three instances per operator, with repeated rollouts, are the validation this pilot most needs. The retail replication covers two of the seven operators and was run after the airline results were known. Scoped scores, grading views and both ablations are post hoc on small subsets, and the ablation counts are draws, not independent samples.

Reading suite generation as diversity-driven search makes a practical prediction, and we confirm it: for defects whose affected set the trace captures, baseline coverage bounds what the within-footprint effect can contribute to a suite-level score, that bound is computable from baseline results alone, and at the specified threshold it already binds for most such defects. The fix is not more scenarios; it is scoring the archive the way archives are meant to be scored.

Mutation operators

Table 2: Mutation operators, one instance each; field grounding names the failure class each models.

IdClassOperatorField grounding
R1RequirementDelete one policy clause (24-hour cancellation)specification violation [3]
R2RequirementInvert one policy condition (“cannot” → “can”)incorrect functionality [19]
T1ToolRemove one tool (cancel_reservation)tool-contract breakage [1]
T2ToolRename one parameter in a tool schemaAPI/serialization mismatch [19]
O1OrchestrationInvert the escalation rule (always transfer to a human)orchestration and execution control [1]
C1ContextTruncate the policy document to its first 40%context loss [1, 3]
C2ContextTruncate conversation history to the last 4 turnsloss of history [3]

Table 2. Mutation operators, one instance each; field grounding names the failure class each models.

Grader validity

On the 488 dual-graded airline transcripts (the 61 generated scenarios carrying both a programmatic check and a judged predicate, over baseline and seven mutants) the two signals agree on 75% of verdicts (κ=0.21\kappa= 0.21); the judge is lenient on 10% and strict on 16%. Its strictness concentrates on O1: it fails 26% of O1 conversations the programmatic check passes, against 13% at baseline and 14% for the other six mutations, which is the same trace-orthogonality of § 4.2 reappearing in the grader. Retail shows the same concentration on its 33 dual-graded S3 scenarios: the judge fails 30% of O1 conversations the programmatic check passes, against 12% at baseline and 12% for T1.

Whole-suite results for every airline mutation

Table 3: Whole-suite score as specified before running experiments, airline, all seven mutations: Δm\Delta_m with paired-bootstrap 95% interval. S2 values are means over the 20 size-matched draws. Detection requires Δm>θ=0.10\Delta_m > \theta= 0.10 and an interval excluding zero; the only detection is marked †. S2 is detected only if a majority of draws detect, so its mean interval can exclude zero without a detection. T2 on S3 equals θ\theta exactly (9 of 90 scenarios) and so does not exceed it. Baseline pass rates 0.50 (S1), 0.88 (S2 mean), 0.88 (S3). The single detection is threshold-fragile: at θ=0.05\theta= 0.05 T2 is also detected on S3, and at θ=0.15\theta= 0.15 nothing is detected by any suite.

MutationS1S2S3
R1-0.06 [-0.22, +0.10]+0.05 [-0.05, +0.16]+0.04 [-0.03, +0.13]
R2+0.06 [-0.10, +0.22]0.00 [-0.10, +0.10]0.00 [-0.08, +0.08]
T1-0.04 [-0.18, +0.10]+0.08 [-0.02, +0.19]+0.08 [0.00, +0.17]
T2+0.12 [-0.02, +0.26]+0.10 [-0.01, +0.22]+0.10 [+0.01, +0.19]
O1-0.02 [-0.16, +0.12]+0.15 [+0.02, +0.28]+0.14 [+0.04, +0.26]†
C1+0.14 [0.00, +0.28]+0.02 [-0.10, +0.13]+0.02 [-0.07, +0.10]
C2-0.08 [-0.22, +0.06]-0.04 [-0.14, +0.05]-0.04 [-0.12, +0.03]

Table 3. Whole-suite score as specified before running experiments, airline, all seven mutations: Δm\Delta_m with paired-bootstrap 95% interval.

R2’s traced coverage on S3 is 3 of 90 scenarios (ceiling 0.02) and on S1 15 of 50 tasks (ceiling 0.08); both lie below θ\theta, which with R1, T1 and O1 gives the four ceiling-bound defects of §4.1; O1 qualifies on a ceiling of exactly 0.08 on both suites, and its footprint is in any case only incompletely traced (§4.2). R2’s scoped Δ\Delta is +0.07 [-0.13, +0.27] on S1 and 0.00 [0.00, 0.00] on its 3 S3 scenarios; T2’s is +0.12 [-0.08, +0.32] on its 25 S1 tasks, coinciding with its whole-suite value. O1’s S1 row, omitted from Table 1 for space, has traced coverage 8 of 50 (ceiling 0.10), whole-suite Δ\Delta = -0.02 and scoped Δ\Delta = -0.12 [-0.38, 0.00], the scoped interval touching zero, as §4.2 notes. C2 has no principled subset (history truncation touches every conversation long enough to lose a turn), so no ceiling is computed for it.

Retail replication

The airline protocol was rerun unchanged on the retail domain of τ2\tau^2-bench for the baseline and two mutations, O1 (invert the escalation rule) and T1 (remove the tool `exchange_delivered_order_ items), after the airline results were known; it is a post-hoc replication, not a study specified in advance over the operator set. Suites: S1, 48 hand-authored tasks drawn as a seeded subsample of the domain’s 114 (two conversations were unrecoverable); S2, 20 size-matched draws; S3, a seeded 50-scenario subsample of the 90 generated, 33 of which carry a programmatic end-state check. Baseline pass rates 0.42 (S1), 0.86 (S2 mean), 0.86 (S3). All but three of retail’s hand-authored tasks carry no judged predicate, so we report no S1 judge-only view. No S3 scenario names the removed tool in its grading criteria, so T1 has no traced S3 subset: the same trace limit as T2 on airline. Judge-programmatic agreement on the 99 dual-graded S3 transcripts is 74% (κ=0.09\kappa= 0.09), lower than on airline and on a smaller overlap.

Table 4: Retail replication, airline protocol unchanged: Δm\Delta_m with paired-bootstrap 95% interval, positive is a drop in pass rate; combined grading unless a view is named; S2 is the mean over 20 draws. Detection as in Table 3; scoped rows are post hoc, nn is the number of scenarios exercising the mutated element, flips are pass→\tofail / fail→\topass.

MutationSuitennΔ\Delta [95% CI]detected
O1S148+0.17 [+0.04, +0.31]yes
O1S248+0.10 [-0.04, +0.24]no
O1S350+0.10 [-0.04, +0.24]no
O1S3, programmatic only330.00 [-0.12, +0.12]no
O1S3, judge only50+0.20 [+0.06, +0.34]yes
T1S148-0.02 [-0.21, +0.17]no
T1S1, scoped (flips 6/0)14+0.43 [+0.21, +0.71]yes
T1S248+0.02 [-0.11, +0.15]no
T1S350+0.02 [-0.10, +0.16]no

Table 4.

Ablation protocol and results

Both ablations reuse the conversations already run: no additional simulation was performed, and each sub-suite is scored with the same θ\theta, the same 2,000-sample paired bootstrap and the same seed as the main analysis.

Table 5: Diversity ablations on the generated suite, counted as detections over (draw ×\times mutation) trials at θ=0.10\theta= 0.10. Airline instantiates 7 mutations, retail 2. Scenario kind is size-matched: the happy-only arm is the single deterministic sub-suite of all happy scenarios, the all-kinds arm is 20 draws of the same size from all three kinds. Personas is not size-matched (Appendix § 11). Retail’s kind ablation is degenerate and reported for completeness.

airlineretail
Scenario kind (size-matched)size 30size 16
happy path only0 / 70 / 2
all three kinds6 / 140 (O1: 5 of 20 draws)0 / 40
Personas(not size-matched)
P=1P = 12 / 140 (size 11)2 / 40 (size 6)
P=2P = 26 / 140 (size 23)4 / 40 (size 12)
P=4P = 422 / 140 (size 45)4 / 40 (size 25)
P=8P = 811 / 140 (size 50)0 / 40 (size 48)

Table 5.

Scenario kind. The happy-only arm is the full set of happy-path generated scenarios: a single deterministic sub-suite, hence 1 trial per mutation. The all-kinds arm draws 20 sub-suites of that same size uniformly from all three kinds. Suite size is therefore matched between arms, but kind composition is not the only thing that varies with it: drawing uniformly from all three kinds also changes which requirements the sub-suite covers.

Personas. For each level PP we draw 20 times: sample PP distinct personas, then sample min⁡(∣S1∣,scenarios available)\min(|S1|,\text{scenarios available}) scenarios from the union of their scenarios. Because a persona carries roughly 90/8 scenarios, the available pool (and hence sub-suite size) grows with PP until it saturates at ∣S1∣|S1|. Size and diversity are therefore confounded, and at P=4P = 4 the sample is very nearly the whole union, so consecutive draws differ only in persona choice. We report the counts and draw no curve from them.

Threshold sensitivity

All values are recomputed from the frozen results with no additional runs (script released with the pipeline). For S1 and S3, detection at any θ\theta follows from the stored Δ\Delta and interval; the recomputation reproduces the stored detections at all three specified values exactly. S2 detection is a majority over 20 draws whose per-draw Δ\Delta is not stored, so S2 is reported at the three specified values only. A traceable defect is ceiling-bound if its ceiling is at or below θ\theta on every suite where it has a traced subset; airline ceilings are R1 0.04 (S1) and 0.00 (S3), R2 0.08 and 0.02, T1 0.06 and 0.09, O1 0.10 and 0.10, C1 0.49 (S3 only).

θ\thetaairline detectionsretail detectionsceiling-bound
S1S2S3S1S2, S3(airline, of 5)
0.05nonenoneO1, T2O1none1 (R1)
0.10nonenoneO1O1none4 (R1, R2, T1, O1)
0.15nonenonenoneO1none4 (R1, R2, T1, O1)

Table 6. Specified-score detections and ceiling-bound airline defects at the three thresholds fixed before the runs. “none” means no mutation is detected on that suite.

Over a continuous grid the ceiling-bound count is 0 below 0.04, 1 from 0.04, 2 from 0.08, 3 from 0.09 (T1’s S3 ceiling is 0.089), 4 from 0.10 and 5 from C1’s 0.49.

Run configuration

Both domains ran on 29 August 2026 in the unmodified τ2\tau^{2}-bench harness, v1.0.1 (commit fc0055d): airline from 09:04 to 12:59 UTC, retail from 13:35 to 16:11 UTC. Agent, user simulator and judge are gpt-5.6-luna; the generator is gpt-5.6-terra. Temperature is omitted, so the provider default applies, because the agent model rejects temperature 0; reasoning effort is set to none, and the usage records show zero reasoning tokens per conversation. Each scenario runs once per configuration, at most two conversations concurrently. Analysis uses a 2,000-sample paired bootstrap over scenarios with a fixed seed and 20 S2 draws. C1 keeps the first 3,070 of the airline policy’s 7,676 characters; C2 keeps the last 4 turns. Airline simulation and judging cost $3.17 over 1,120 conversations, retail $1.54 over 300, and generating both suites $2.31.

References

  1. [1]Ali Asgari, Annibale Panichella, Pouria Derakhshanfar, and Mitchell Olsthoorn. What challenges do developers face in AI agent systems? An empirical study on Stack Overflow & GitHub issues. arXiv preprint arXiv:2510.25423, 2025.arxiv.org/abs/2510.25423
  2. [2]Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. χ²-bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982, 2025.arxiv.org/abs/2506.07982
  3. [3]Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Why do multi-agent LLM systems fail? In Advances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2503.13657.arxiv.org/abs/2503.13657
  4. [4]Richard A. DeMillo, Richard J. Lipton, and Frederick G. Sayward. Hints on test data selection: Help for the practicing programmer. Computer, 11(4):34–41, 1978.DOI
  5. [5]Yue Jia and Mark Harman. An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering, 37(5):649–678, 2011.DOI
  6. [6]Sungmin Kang, Juyeon Yoon, and Shin Yoo. Large language models are few-shot testers: Exploring LLM-based general bug reproduction. In Proceedings of the 45th International Conference on Software Engineering (ICSE), 2023.arxiv.org/abs/2209.11515
  7. [7]Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. AI agents that matter. arXiv preprint arXiv:2407.01502, 2024.arxiv.org/abs/2407.01502
  8. [8]Vishwanath Katharki and Sainyam Galhotra. Goal-oriented reliability and self-improvement for multi-agent systems. In Proceedings of the ACM Conference on AI and Agentic Systems (CAIS ’26), 2026. doi: 10.1145/3786335.3813230.DOI
  9. [9]Joel Lehman and Kenneth O. Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary Computation, 19(2):189–223, 2011.DOI
  10. [10]Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as agents. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2308.03688
  11. [11]Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909, 2015.DOI
  12. [12]Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03286, 2022.arxiv.org/abs/2202.03286
  13. [13]Justin K. Pugh, Lisa B. Soros, and Kenneth O. Stanley. Quality diversity: A new frontier for evolutionary computation. Frontiers in Robotics and AI, 3:40, 2016. doi: 10.3389/frobt.2016.00040.DOI
  14. [14]Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H. Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, Tim Rocktäschel, and Roberta Raileanu. Rainbow teaming: Open-ended generation of diverse adversarial prompts. arXiv preprint arXiv:2402.16822, 2024.arxiv.org/abs/2402.16822
  15. [15]Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. Agenda-based user simulation for bootstrapping a POMDP dialogue system. In Human Language Technologies 2007: NAACL, Companion Volume, Short Papers, 2007.DOI
  16. [16]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024.arxiv.org/abs/2406.12045
  17. [17]Xiaowen Zhang, Hannuo Zhang, and Shin Hwei Tan. Understanding bugs in modern agentic frameworks: A study of symptoms, root causes, and triggering conditions. arXiv preprint arXiv:2604.08906, 2026.DOI
  18. [18]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2023.arxiv.org/abs/2306.05685
  19. [19]Xinxue Zhu, Yanzhou Mu, Xiaoyu Zhang, Tianlin Li, Chao Shen, Chunrong Fang, Yang Liu, and Juan Zhai. Where agent frameworks fall short: Examining functional challenges and usability concerns. arXiv preprint arXiv:2602.21806, 2026.arxiv.org/abs/2602.21806

Paper details

vlnc:W_2f6g2rz5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents