betaFeedback

WorksComputer Science

Attributing Agent Failures to Their History Needs an Executed Counterfactual: Toward a Reporting Standard

Abstract

Published explanations of why long-horizon agents fail are commonly taxonomies labeled over completed trajectories: a judge reads a transcript and names the failure. We argue that a claim attributing an agent's behavior to its history should carry intervention-verified evidence, per instance or via a validated estimator: freeze an executable mid-trajectory state, alter only the history the agent sees, and continue - executed rather than inferred from the transcript, at a state whose restoration has been measured, with a replay certificate matched to the estimand. Each of the four manipulation classes we name has a precedent, and several methods already execute interventions from checkpoints, re-executed prefixes or recorded conversation states (for example DoVer, REFLECT, Min et al., Who&When Pro, Causal Agent Replay), but none reports a measured replay certificate. Two cautionary measurements, neither applying the standard, follow: a carrier taxonomy whose "concentration" could not be separated from benchmark construction (adjusted odds ratio 1.15, interval crossing 1), and hash-identical snapshots that, restored into identically seeded processes, agreed with one another in 24 of 24 states while differing from the live continuation in 9 and 20 of 24. Restoring the captured position of a random stream made all 24 agree with the live run.

The correlational trap

A common way of explaining agent failure is to collect finished trajectories, have a human or LLM judge read them, and assign each a taxonomy label [10, 25].

HORIZON [28] attributes failures across 3,100+ trajectories via a 7-category taxonomy applied by an LLM judge and never resumes from an intermediate state, so its “History Error Accumulation” label is post-hoc. MAST [5] derives 14 failure modes from 150 traces and reports that verification failures “appear frequently even in successful runs.” Both intervene only at task or system level (horizon extension; design changes worth +9.4% and +15.6%); the failure labels remain post-hoc.

AgentLocate [29] formalizes the decisive step counterfactually, as the earliest step whose single correction flips the outcome (Who&When’s definition [36]), and then states that its judge “does not execute” the counterfactual but “approximates this criterion by reasoning over the logged trajectory,” as RAFFLES [38] and DuoTrace [35] also do offline. Who&When Pro [17] does execute 12,326 golden labels from what it describes as exact prefix replay with one injected error. Who&When Pro argues that regenerating the whole trajectory, as AgentTracer [34] and the Aegis attribution pipeline [14] do, lets the pre-injection portion drift (no rate is reported). The field already defines causes counterfactually; AgentLocate and the Who&When evaluators still attribute from transcripts, and the executing methods of § 2 lack a measured replay certificate. In a two-coder LLM census of the 41 failure-analysis papers in our reading list (Appendix § 12), the coders disagreed on how many manipulate history or state mid-run (coder 1: 22; coder 2: 13, under a stricter boundary). Both found few that restore an executable intermediate state and continue the same agent: one and three respectively, and zero and two once Min et al. is treated as prefix re-execution. Two to three papers’ reading notes were too thin to code; secondhand notes give no upper bound.

A label assigned by reading a completed transcript, blinded or not, cannot distinguish a cause of failure from something that co-occurs with it.

Position. A claim that an agent behaved as it did because of something in its history should carry a history-manipulation control at a frozen executable state: the same agent, continued from the same restored state, under a history that differs in exactly the respect the explanation names. Causal interpretability already argues that a representational claim needs an intervention on a fixed forward pass [11, 27]; the frozen executable state plays that role for claims about history and state. We name four manipulation classes and their precedents (§ 2), report two cautionary measurements (§ 3), and propose an agenda (§ 5).

History manipulation at frozen executable states

Setup. Let a trajectory reach step tt with environment state sts_t, process state ptp_t (random streams, open handles; caches derived from the history are recomputed from the manipulated history ht′h'_t defined below), executable state xt=(st,pt)x_t = (s_t,p_t), and agent-visible history hth_t. Three properties are distinct: (i) the fork restores xtx_t (now cheap with sandbox checkpointing [9]; certified by a replay certificate: a measured restored-vs-reference agreement rate under a fixed probe policy); (ii) exact replay: continuations under a deterministic probe policy and coupled exogenous randomness are bitwise identical; (iii) continuations under independent randomness are distributionally equal. A frozen executable state is one where (i) has been measured, not assumed, along the probe run and meets a rate or margin fixed in advance for the estimand. Pathwise estimands additionally require (ii); distributional ones require replicated continuations against an estimand-matched control; the certificate is probe-specific, and § 3.2 measures (i) and (ii). A history manipulation replaces hth_t by ht′h'_t while holding the frozen state fixed, then continues under the same policy and compares arms. Four classes organize the history-based explanations in the precedents below:

  1. Healed history: errors repaired in place or an oracle history substituted.

  2. Provenance swap: byte-identical content re-attributed to a different source role (self vs. prior agent, tool, user).

  3. Placebo / length-matched synthetic history: same token and turn count, inert or idealized content.

  4. Counterfactual injection: a controlled error, stale value, or compaction summary inserted where none occurred.

Holding sts_t fixed while editing hth_t targets a controlled direct effect of what the agent is shown, given the realized world; if the error also changed the world, a healed history paired with the damaged state is off-support; the total effect needs a fork before the event or a joint history-and-state arm.

Precedents. Two precedents re-execute a recorded prefix to rebuild the state (Min et al., recipe hash; Who&When Pro, cached static observations plus an abort-on-exception guard); Causal Agent Replay (CAR) holds the recorded actions before a step fixed and re-decides from it; REFLECT resumes from a recorded conversation state; AgentDebug [39] re-rolls out from the critical step with feedback, mechanism unstated; DoVer [19] edits plans or messages at a checkpointed step and resumes; CausalFlow [4] replaces a step and re-executes downstream where an executor exists (predicting it otherwise); TraceElephant [7] replays to the identified step, alters its input and observes three further steps.

None reports the measured replay-agreement certificate its estimand requires, pathwise or distributional. Forking both arms from one restored instance validates their contrast; the live-vs-restored control ties that instance to the observed trajectory; separately restored arms need a between-arm identity check.

  • Healed history and controlled-error injection (no executable environment). Sinha et al. [24] replay the turn-100 step under injected histories with controlled error rates; in memory-augmented chat, Xiao et al. [30] repair the question, the memory, or both and continue.

  • Length-matched synthetic history (repeated social dilemmas). Liu et al. [18] replace 80 − XX of 80 rounds with synthetic cooperative records: restoring cooperation from 6.9% to 97.4% at X=2X = 2 for one model (less for weaker ones).

  • Provenance swap (single claims in math/logic traces). Chen et al. [6] move byte-identical wrong claims between the model’s own thought and user/tool/memory roles; explicit-correction rates rise by 23–93 points.

  • State-side counterfactual (environment, not history). ChainSWE [12] contrasts oracle-reset against inherited repositories: 58.9% → 36.5% per bug, 48% of downstream failures attributable to inherited state.

  • Step-level counterfactuals by prefix hold or conversation-state resume (attribution). Causal Agent Replay [23] applies do-operations at a step (resample, action, observation, history or policy), holds the recorded actions before the step fixed and re-runs forward to an outcome distribution with intervals, but validates only on two synthetic causal models and reports no numeric action-match rate or live-vs-restored baseline. REFLECT [16] injects a repair message into a recorded conversation state and re-runs the agent at temperature 0 (placebo and wrong-step controls at temperature 0.2), but reports no replay-fidelity measurement and does not state how the rollback state is restored.

  • Compaction summary vs. raw history by recipe-hash-checked prefix re-execution (class 4; closest executable-surface precedent; no measured replay certificate). Min et al. [20] re-create what their protocol treats as the same AppWorld [26] state at each of 590 compaction boundaries by re-executing the recorded prefix under a hash of the recipe, not of the reached state (whether the states matched was not measured; Appendix § 10), and continue under two arms, a less-compacted control and the new summary, three to five samples each, released as TRACE [21].

We propose transplanting interventionist evidence and replay validation, established in causal inference and record-and-replay, to agent-history attribution. Among the empirical LLM-agent papers we examined, Causal Agent Replay comes closest to the reporting norm; the increment here is narrow: a reported numeric replay certificate, an unedited live-vs-restored control, and an estimand-matched certificate per regime, the first of which CAR names but does not quantify. Resampling over reasoning traces [3] shares the estimand without an environment to restore; explainable-RL methods intervene, when they do, on actions and policies, not history [13, 37]. Our census excluded position papers without coded failure data, so it cannot show the norm is unproposed.

Two cautionary measurements

Neither applies the standard; amendments are disclosed (§ 4, Appendices § 8–§ 10).

A carrier taxonomy confounded with benchmark construction

We asked an observational question of ActBench [32], an agent-safety corpus (300 task pairs, 24,000 trajectories): which carrier delivers the payloads that convert into unsafe actions? Deterministic carrier labels covered 266/300 task pairs, a same-agent LLM consistency check over raw bundles (rule labels visible) agreed on 50/50 sampled cases, and a degeneracy guard, declared after the audit tallies but before any outcome statistic, held that if fewer than three classes survived the pre-specified collapse, the concentration and ranking layers were not evaluable. The guard fired: service_response (225 task pairs) vs. local_file_state (41) remained. The headline statistic, 84.9% [80.1, 89.4] of 9,199 attack-positive trajectories (ActBench’s attack-pass label, AGS ≥ 0.8) carried by service responses, tracks task-pair allocation by construction (225/266 = 84.6% of labeled task pairs are service-carried); the behavior- and attack-method-adjusted odds ratio was 1.148 [0.627, 2.016], crossing 1; and first-exposure recovery was 96.65% for service responses against 69.31% for the collapsed local-file/memory/tool-metadata class, so even exposure-conditional conversion rates are confounded by measurement recoverability (Appendix § 8). The measurement failed its pre-outcome amended per-class recovery floor (69.31% < 70%). The share is a valid descriptive statistic for the 266 labeled task pairs; these data cannot separate benchmark construction from agent response and identify no service-versus-local carrier effect in either direction.

Hash-identical snapshots need not replay exactly

We tested whether hash-identical states replay exactly with our own snapshot layer over AgentCE-Bench [31], with a fixed ground-truth continuation policy and no model in the loop, on 24 factorialcell states. A five-clone state-hash certificate passed every state tried with the stochastic tool-failure channel off (144 synthetic verdicts over 108 distinct states, plus 17 of 17 model-generated), and an in-snapshot sensitivity check flagged 45/45 single-field edits (blind to omitted state by construction). With the channel on, five unseeded naive restorations of each hash-identical state disagreed among themselves in 17/24 states at failure rate 0.05 and 22/24 at 0.20. The tool-failure draw consumes the process-global random stream, which the upstream runner seeds at start-up and re-seeds while building its task cache, and no snapshot field records the stream’s position.

In a seeded replication, two fresh single-threaded re-executions agreed in 24/24 states, as did a state rebuilt by replaying the recorded prefix. Under one seed per state, byte-identical snapshots restored into processes seeded with that state’s seed diverged from the live run in 9/24 and 20/24 states (0.38 and 0.83; independent-sequence reference 0.45 and 0.84, Appendix § 9); under the runner’s single fixed seed, in 18/24 and 24/24. In both regimes the five clones agreed with one another in 24/24. A clone-vs-clone gate whose clones share a seed position passes and is still wrong; a separately executed reference continuation, live or prefix-rebuilt, detects it. Capturing the stream state alongside the snapshot and restoring it before each continuation removed every divergence in the restoration arms: live-vs-restored 0/24 at both rates in the unseeded experiment and in the seeded replication, and five-clone 0/24 at both rates in the unseeded experiment. The snapshot omitted the stream position that § 2 counts as process state. Capturing it yields exact replay for this probe by coupling the exogenous randomness across clones (common random numbers [15]; record-and-replay [22]); it does not show the executable state is complete.

Two consequences follow. First, final-grader divergence under the oracle retry policy was 0/24 in every condition: an outcome-only metric would have hidden the problem, and whether realization-level infidelity moves model-driven behavior is unmeasured. Second, a replay-based interpretation should state whether it needs exact replay or distributional equality and show the matching measured agreement rate against an unedited-history live-vs-restored control.

Discussion and limitations

Claims. The four classes have precedents (§ 2); outcome-labeled taxonomies cannot by themselves establish causal attribution to history. Neither measurement establishes a behavioral result: Measurement 3.1 was declared non-evaluable (§ 3.1) and Measurement 3.2 is model-free.

Not measured. A third measurement, a prefix taxonomy over the TRACE boundaries, is not reported here: on artifact review its naive has-written label proved a trajectory-phase proxy, so it does not measure its construct (Appendix § 10).

Amendments. In reproducing Min et al.’s released scan, a byte-identity criterion failed on two bootstrap-bearing tables under an unsupported interpreter and, on a rationale later found mistaken, was amended to identical point estimates plus overlapping intervals; a supported interpreter re-produced all six tables byte-identically (Appendix § 10). The seeded replication’s pre-registration was frozen before execution; two arms were added by dated amendment after the fixed-seed results were seen, and the per-state arms were designated primary after both result sets were known (Appendix § 9).

Agenda

  1. The counterfactual norm. A claim that “the agent failed because of XX in its history” should carry per-instance intervention evidence (continuations from a frozen state with XX removed, replaced, or re-attributed; replicated unless replay is pathwise-certified) or an estimator validated on a representative, intervention-verified test set with reported calibration and uncertainty; otherwise the attribution is not intervention-verified under this standard.

  1. Replay certificates before interpretation. Report the certificate the estimand needs, on the surface and provider used: coupled-randomness agreement (§ 3.2) against an unedited-history control (pathwise), or restored-to-live equivalence within a prespecified margin on a named metric, with replicated controls (distributional), plus a detector-power control. For hosted models the pathwise route may be unavailable [1, 23, 33], as on the one path we tried (Appendix § 9); a paired-contrast design is in Appendix § 11 (proposed, not run).

Acknowledgments and Disclosure of Funding

This work was funded by Salesforce.

References

Measurement 3.1: the carrier taxonomy

Corpus and carrier labels

The corpus is a released set of paired benign and adversarial agent trajectories [32]: 300 task pairs and 24,000 trajectories over 20 released model–harness configurations (15 models, 6 harnesses; not a full cross), all parsed without error.

Carrier labels are deterministic: the rule reads the benchmark’s primary-carrier metadata (a mock-fixture field maps to service_response, a workspace file to the local classes), gated on the SHA-256-changed sources of the benign and adversarial bundles (215 and 36 pairs); where that metadata was absent, a deterministic path-prefix rule over the changed sources labeled 15 further pairs. The rule fired on 266/300 task pairs (88.67%), above its pre-specified floor of 85%; the 34 unlabeled pairs, which lacked usable metadata and had both fixture and workspace sources changed, and their 2,040 attack trajectories were excluded rather than assigned non-deterministically. Fine-class counts before collapse were service_response 225, memory_state 15, tool_metadata 14 and workspace_file 12. The protocol required 30 task pairs per analyzed class and pre-specified that classes short of it would be collapsed into a coarser taxonomy; the specific service-versus-local mapping was chosen at collapse time, after the fine-class counts were known. All three local classes fell short and were merged into local_file_state—task-reachable workspace documents, persistent memory state, and tool or skill metadata—leaving service_response (225 pairs) and local_file_state (41). The merged class pools carriers with different recovery rates, so the collapsed contrast supports no inference about memory, tool-metadata or workspace carriers separately.

Audit protocol

Fifty task pairs were relabeled by the same LLM agent that ran the gate, not blind to the rule (each packet displayed the rule’s carrier label and the benchmark’s primary-carrier field), with raw-bundle access: it read each case’s benign/adversarial source diff and task metadata and recorded a coarse class, a fine class and a note. Agreement with the deterministic rule was 50/50, against a pre-specified floor of 45/50; the sample split 44 service to 6 local, and 44/2/2/2 fine.

Two pre-results additions to the protocol

Both were logged after the 50-case tallies had been seen and before any concentration, odds-ratio, exposure or replication number existed; the skew of the audit sample was the sole trigger. Paraphrased: first, if fewer than three classes clear the cell-size floor after the collapse, the concentration and ranking layers are not evaluable—a top-two share over two exhaustive classes is 1.0 by construction and a rank correlation over two items can only be ±1—leaving the exposure-conditional layer, which conditions on exposures rather than on the carrier marginal. Second, first-exposure recovery must be reported per analyzed class, each class clearing 70%: a class below that floor estimates its conversion rate on a differently biased subsample, so the cross-class comparison would be confounded by recoverability rather than by behavior.

Estimates

Table 1 reports the analyzed estimates, all computed on the 15,960 labeled attack trajectories from the 266 labeled pairs under the benchmark’s own unsafe-action label.

Quantityservice_responselocal_file_state
Task pairs (of 266 labeled)225 (84.6%)41 (15.4%)
Labeled attack trajectories13,5002,460
Attack-positive trajectories (of 9,199)7,8101,389
Share of unsafe actions0.8490 [0.8008, 0.8940]0.1510 [0.1060, 0.1992]
Adjusted odds ratio1.1484 [0.6275, 2.0160]1 (reference)
First-exposure recovery13,048/13,500 = 96.65%1,705/2,460 = 69.31%
Exposure-conditional conversion7,629/13,048 = 58.47%1,122/1,705 = 65.81%

Table 1. Measurement 3.1, analyzed carrier estimates.

Intervals are 95% task-clustered bootstrap intervals at seed 20260901 (within-scenario dependence between tasks is not represented), with 5,000 replicates for shares and concentration indices and 2,000 for the adjusted model. Attack method is nearly collinear with carrier (only context injection contains both classes: 195 service, 12 local pairs), so the adjusted odds ratio is estimated under limited common support. The adjusted odds ratio is a logistic carrier contrast with fixed effects for 15 risk-behavior classes and the benchmark’s attack method, fit on all 15,960 labeled rows, with 0 failed bootstrap fits.

First exposure is the earliest transcript message containing an injected candidate or a fragment of it; for service fixtures the candidate set holds the entire attacked field value, so a fragment match can fall on unchanged benign text, whereas local-file candidates are the changed spans only. Pooled: 14,753/15,960 first exposures located (92.44%), payload candidates recoverable for 15,960/15,960; HHI 0.7436 [0.6809, 0.8104], Gini 0.3490 [0.3008, 0.3940]; top-one share half-width 0.0466; top-two share 1.0000, half-width 0.0000—mechanical at two exhaustive classes.

Replication and sensitivity

Table 2 records the folds for completeness, not as evidence. In the clean-twin control, the benign halves of the same pairs already differ by carrier class, on an outcome that no payload touches.

AnalysisResult
Leave-one-model-out, harness fixed15/15 computable folds; τ\tau-b median 1.000, minimum −1.000-1.000 (14 folds at +1+1, one at −1-1)
Leave-one-harness-out, model fixed6/6 computable folds; τ\tau-b median 1.000, minimum 1.000
Per-configuration share matrix20 configurations × 2 classes, 40 cells
Log-evidence-only proxy7,150 events; service share 0.8236; adjusted OR 0.8973 [0.4715, 1.7043]
Clean-twin negative controlbenign-task failure 13.49% (607/4,500) for service twins vs. 8.78% (72/820) for local twins; 5,320 rows, 266 pairs
Drop configurations below 200 usableall 20 had 798 usable labeled attack trajectories; 0 dropped, estimates unchanged

Table 2. Replication folds and sensitivity analyses; same seed, 2,000 replicates for the proxy model. Leave-one-out replication ran on behavior- and attack-method-adjusted carrier log-odds rankings along both released axes; with two surviving classes τ\tau-b can take only ±1.

Disposition under the frozen rule

The measurement was declared non-evaluable under the pre-outcome amended criterion. Two classes survived the collapse, so the concentration and ranking layers were not evaluable; and local_file_state recovery of 69.31% missed the per-class floor of 70% by 17 trajectories, leaving one evaluable class where the protocol required two, so the exposure-conditional layer failed too. The failure is substantive rather than an artifact of the threshold. Of the trajectories whose first exposure was not located, 181/452 (40.0%) for service_response and 267/755 (35.4%) for local_file_state were nonetheless attack-positive under the benchmark label, despite having no located first exposure. The located rate is therefore the generous reading of recovery, and the 27.3-point differential is the confound the per-class floor was written to catch.

Measurement 3.2: replay fidelity

Census and controls

The environment is AgentCE-Bench [31] at repository revision 22ae16b7, under our own snapshot layer (the benchmark ships no fork or restore API); a certificate should state its population of attempted restorations and count aborted or unreachable ones; here a snapshot is certified by restoring five clones and comparing SHA-256 digests. Census (seed 424242, tool-failure channel off): 144 verdicts over 108 distinct states (see below) = 6 domains × 3 grids (5 × 5, 5 × 7, 5 × 10) × 2 horizons (lowest and highest hidden-slot count, 1–25) × 2 branch budgets (0, 8) × 2 anchors (1/3, 2/3); 144/144 pass, failure fraction 0; the 144 verdicts cover 108 distinct states (the 72 one-slot rows are measured at both anchors, which coincide). A further 17 model-generated states passed the same gate. Two controls bound what the certificate can see (an in-snapshot sensitivity check cannot, by construction, detect omitted state): three single-field mutations (query-counter increment, grid-cell change, dropped last message) injected into 15 states were detected 45/45 (seed 20260901), and 20 states restored by five concurrent separate processes passed 20/20 (seed 20260902).

The uncaptured stream

With a positive tool-failure rate, the environment’s tool-dispatch layer draws every failure from the process-global random stream. The benchmark’s entry point seeds that stream at start-up and re-seeds it while constructing the task cache; a harness that constructs tasks directly, as ours does, does neither. Neither restores the stream’s position, and no snapshot field records it, so digest-identical clones resume at different points of one stream. Our repair is harness-side: capture the stream state at snapshot construction, restore it before each continuation; no upstream file or snapshot schema was modified.

Divergence measurements

Design: 24 states = 6 domains × hidden-slot count ∈{5,21}\in\{5, 21\} × anchors 1/3, 2/3, on the 5 × 7 grid at branch budget 8; deterministic prefix at rate 0, treatment rate from the anchor onward, fixed ground-truth continuation policy, at most 10 attempts per slot. Observables: per-call success/failure sequence, final serialized-snapshot hash (it includes the transcript, so it is not independent of the failure pattern), final grader result. All 120 clones in each naive five-clone condition restored to the candidate digest; 51 tool calls failed at rate 0.05, 229 at 0.20. Table 3 gives every measured row.

Table 3: Replay divergence in the unseeded experiment, all measured conditions (B1: five-clone experiment; B2: live-vs-restored and live-vs-live experiment; the seeded replication is in Table 4). “Naive” restores the serialized state only; “captured” additionally restores the random-stream position; live-vs-live rows restore nothing and are the unseeded re-execution baseline of the run named in the Machinery column. 10,000 state resamples per interval, seed 20260901; every fraction and interval endpoint is an integer multiple of 1/24 and is shown to two decimals. Intervals are the released state-resampling bootstrap intervals over the 24 factorial cells and are descriptive: the cells are not a random sample of the census, so no population coverage is claimed; at a zero count the bootstrap is degenerate (a binomial Clopper–Pearson 95% upper limit would be 0.14 [8], under a sampling model the design does not supply).

Exp.RateMachineryComparison / metricDiv./24Fraction [95% CI]
B10.05naivefive-clone / observable170.71 [0.54, 0.88]
B10.20naivefive-clone / observable220.92 [0.79, 1.00]
B20.05naivelive vs. restored / observable110.46 [0.25, 0.67]
B20.05naivelive vs. restored / failure pattern110.46 [0.25, 0.67]
B20.05naivelive vs. restored / final hash110.46 [0.25, 0.67]
B20.05naivelive vs. restored / final grader00.00 [0.00, 0.00]
B20.05naivelive vs. live / observable130.54 [0.33, 0.75]
B20.05naivelive vs. live / failure pattern130.54 [0.33, 0.75]
B20.05naivelive vs. live / final hash130.54 [0.33, 0.75]
B20.05naivelive vs. live / final grader00.00 [0.00, 0.00]
B20.20naivelive vs. restored / observable190.79 [0.63, 0.96]
B20.20naivelive vs. restored / failure pattern190.79 [0.63, 0.96]
B20.20naivelive vs. restored / final hash190.79 [0.63, 0.96]
B20.20naivelive vs. restored / final grader00.00 [0.00, 0.00]
B20.20naivelive vs. live / observable180.75 [0.58, 0.92]
B20.20naivelive vs. live / failure pattern180.75 [0.58, 0.92]
B20.20naivelive vs. live / final hash180.75 [0.58, 0.92]
B20.20naivelive vs. live / final grader00.00 [0.00, 0.00]
B10.05capturedfive-clone / observable00.00 [0.00, 0.00]
B10.20capturedfive-clone / observable00.00 [0.00, 0.00]
B20.05capturedlive vs. restored / observable00.00 [0.00, 0.00]
B20.05capturedlive vs. restored / failure pattern00.00 [0.00, 0.00]
B20.05capturedlive vs. restored / final hash00.00 [0.00, 0.00]
B20.05capturedlive vs. restored / final grader00.00 [0.00, 0.00]
B20.05capturedlive vs. live / observable100.42 [0.21, 0.63]
B20.05capturedlive vs. live / failure pattern100.42 [0.21, 0.63]
B20.05capturedlive vs. live / final hash100.42 [0.21, 0.63]
B20.05capturedlive vs. live / final grader00.00 [0.00, 0.00]
B20.20capturedlive vs. restored / observable00.00 [0.00, 0.00]
B20.20capturedlive vs. restored / failure pattern00.00 [0.00, 0.00]
B20.20capturedlive vs. restored / final hash00.00 [0.00, 0.00]
B20.20capturedlive vs. restored / final grader00.00 [0.00, 0.00]
B20.20capturedlive vs. live / observable180.75 [0.58, 0.92]
B20.20capturedlive vs. live / failure pattern180.75 [0.58, 0.92]
B20.20capturedlive vs. live / final hash180.75 [0.58, 0.92]
B20.20capturedlive vs. live / final grader00.00 [0.00, 0.00]

Table 3.

The unseeded five-clone figures (17/24, 22/24) are in the direction and magnitude given by the analytic expectation (0.73 at rate 0.05, 0.98 at 0.20; 17.4 and 23.6 of 24 expected against 17 and 22 observed) for five independent draw sequences computed from that experiment’s own mean clone lengths, a post hoc plug-in check from the run’s own observed lengths, not a prediction; the result is that the hash certificate was blind to them. The live-vs-live arm is an unseeded re-execution baseline, not a treatment; it moved between runs at rate 0.05 (13/24, then 10/24) and sat at 18/24 at 0.20 under both machineries. The seeded replication below removes that baseline’s noise: two fresh single-threaded seeded re-executions agree in 24/24 states (the upstream’s shipped runner executes tasks in a thread pool over one shared stream, a further uncaptured channel). The unseeded live-vs-restored figures in Table 3 (11/24, 19/24) are from this run; the replication’s unseeded arm is a separate run (14/24, 21/24).

Grader agreement

No five-clone condition held a state with more than one distinct final grader result: 0 of 24 at each rate, under both machineries. Under this retry-until-success oracle policy, realization-level divergence never flipped the outcome; an outcome-only metric would have reported perfect fidelity.

A second environment

AppWorld 0.1.3.post1 [26]. Reset probe: 10 tasks ×\times 5 fresh initializations compared on an independent full-database digest, the benchmark’s row-hash digest, a frozen clock, and untouched-state evaluator output—0/10 mismatched, 50/50 identical. Replay probe: 5 tasks, scripted prefix with one state-changing write, full per-application SQLite snapshot, 5 restored copies, two-read suffix—0/5 mismatched, 25/25 identical. The checkpoint/replay probe required first resetting cached named in-memory database engines; otherwise a file-copy restore was silently ignored, with no error raised, and was caught only by the probe’s restore-identity digest. AppWorld ships save_state/load_state, which we did not evaluate: the lesson is scoped to harness authors who build their own file-level snapshots.

Seeded replication

A second run under a pre-registration frozen before execution, and amended as disclosed below, re-executed everything seed-once: trajectories run at the treatment rate from the start, compared with the live continuation, on the same 24 states. Two fresh runs with the same seed agree exactly, and so does a state rebuilt by replaying the recorded prefix; a byte-identical snapshot restored into a freshly seeded process does not, under either seed regime, and five such clones agree with one another in 24/24 states while differing from the live run in 9 and 20 of 24 with per-state seeds (18 and 24 of 24 under the runner’s fixed seed, which yields only four distinct live realizations across the 24 states) (Table 4, Figure 1). A clone-vs-clone gate whose clones share a seed position therefore passes and is still wrong; a comparison against a separately executed reference continuation, live or prefix-rebuilt, detects it. The independent-sequence references quoted in the main text (0.45, 0.84) were recomputed with the pre-registered formula from the per-state run’s live continuation lengths (mean 7.04 and 8.13 calls); the released expectation file, computed from the fixed-seed run’s lengths, gives 0.47 and 0.83. They are plug-in references, not the exact null for this design: the live and restored continuations read shifted portions of one per-state seeded stream, and the realized live lengths depend on the retry rule.

Arm0.050.20
Two fresh runs, same seed (re-execution baseline)0/240/24
State rebuilt by replaying the recorded prefix (seeded)0/240/24
Byte-identical snapshot restored, seeded process, runner’s fixed seed (four distinct realizations)18/2424/24
Byte-identical snapshot restored, seeded process, one seed per state9/2420/24
Byte-identical snapshot restored, unseeded process (separate run)14/2421/24
Snapshot with captured random-stream state restored0/240/24
Five restored clones, seeded process: disagreement among the clones0/240/24
Final-grader divergence, any arm0/240/24

Table 4. Seeded replication: states (of 24) whose continuation diverged from the live continuation, by arm and tool-failure rate. Counts only; the 24 states are factorial cells. The last two rows report clone-vs-clone disagreement (both seed regimes) and final-grader divergence.

Figure 1. Seeded replication, one seed per state: divergence from the live continuation (fraction of 24 states) for the arms of Table 4 at tool-failure rate 0.05 (left) and 0.20 (right); error bars are exact binomial 95% intervals drawn for orientation only, since the states are factorial cells; the unseeded bar is the separate unseeded run of Table 4.

Two arms were added by dated amendment after the fixed-seed results had been seen: the five-clone seeded arm, to test the clone-agreement consequence that result suggested, and the per-state-seed repeat, because one process seed makes the six domains of a cell share a single live realization (four distinct realizations across the 24 states). No acceptance criterion changed; the per-state arms were designated primary after both result sets were known; the pre-registration file’s own dated correction records that its amendment timestamps had been misstated by five to eight minutes.

A provider-side observation

On τ−2\tau^{-2}-bench [2], with a third-party-served model behind a proxy at temperature 1.0 and a best-effort seed, three request payloads were each re-sent byte-identically with the same seed (four comparable call pairs across the three), and none reproduced its completion (0 of 4), while the serving path we controlled reproduced 34/34 calls and 2/2 episodes bitwise. Temperature-zero decoding is not reproducible across runs in general [1], and greedy decoding diverges across batch sizes and hardware under 16-bit precision [33], so a hosted pathwise certificate must be measured on the deployed configuration. 52 of 56 regeneration calls had left the original key path, so live re-execution yields about one comparable call per episode. No response carried a system_fingerprint, so silent downstream discard of the seed is not excluded; environment-hash identity there carried approximately zero discriminating information and is not offered as replay evidence. The sample is small, and non-reproduction is localized to the hosted path; we cannot localize it further within that path.

TRACE reproduction and a measurement not reported here

What was reproduced

The TRACE release [20, 21] is a frozen AppWorld cohort [26]: 590 compaction boundaries, 4,641 paired PRE/POST rollouts (4,640 scored), 20,133 steps, 147 tasks (140 with boundaries). Its shipped paired burden (TRACE’s per-boundary paired union delta dUd_U between its two arms) rises from +0.131 [+0.101, +0.160] at one action after a boundary to +0.509 [+0.432, +0.588] at five. We re-ran the released scan read-only at revision 9be7d669, modifying no data: unit tests 24/24 and 35/35, reproduction suite 4/6, a fresh scan over the same 590 boundaries and 20,133 steps, four of six shipped tables byte-identical.

The byte-identity failure and the amended criterion

Our frozen criterion required every shipped table to match by SHA-256. It failed on the two tables carrying bootstrap intervals, and on no others. An amendment was logged after that failure was observed, on the recorded but mistaken ground that the bootstrap was unseeded and byte identity across platforms therefore over-strict; the bootstrap is seeded, and the discrepancy later disappeared on a supported interpreter, consistent with an interpreter-version explanation (below). The amended criterion requires deterministic tables to be byte-identical and, on the stochastic tables, identical point estimates with row-by-row interval overlap and unchanged classification of whether an interval excludes zero (Table 5); differences below 10−1210^{-12} counted as negligible, three orders of magnitude above the largest observed difference. Downstream work used the fresh scan as its basis.

curves.csvmarginal.csv
Rows matched285/28545/45
Point-estimate values compared1,71045
Maxshipped − fresh, point estimates1.33×10−151.33 \times10^{-15}1.53×10−161.53 \times10^{-16}
Interval endpoints compared57090
Maxshipped − fresh, endpoints7.22×10−167.22 \times10^{-16}2.78×10−162.78 \times10^{-16}
Non-overlapping intervals00
Qualitative-classification changes00

Table 5. Shipped versus fresh bootstrap-bearing tables. All 330 intervals overlap, and no interval changes its qualitative classification.

Why the differences are not a result

The released bootstrap is seeded—a fixed constant, 20260803, in the verifier module, used by the resamplers—so these are not resampling differences. Our re-run used Python 3.10.8 while the package declares Python ≥3.11\ge3.11, and differences at 10−1510^{-15} are consistent with floating-point summation order across interpreter versions. We did not isolate the summation path ourselves. A subsequent re-run under Python 3.12.13, an interpreter the package supports, reproduced all shipped tables byte-identically and passed the release’s 6/6 reproduction checks, so the original criterion is met on a supported interpreter, consistent with the interpreter-version explanation.

Prefix re-execution as a restored state: what the recipe hash covers

TRACE does not snapshot: each rollout opens a fresh environment, re-executes the recorded code prefix with no model in the loop, installs one context (PRE or POST), and free-runs at most five primary actions. The stated reason is that the environment’s own save_state/load_state does not round-trip API-server state. Restoration is checked instead by the recipe hash replay_identity == SHA256(task, split, boundary_step, prefix_codes), which both arms of a boundary must carry identically.

Measurement 3.2 shows such a hash is not sufficient in general. The protocol’s own rationale is that a nondeterministic prefix would surface as a divergence; an input hash cannot detect that (identical hashes, different worlds). Two weaker grounds remain for treating the paired arms as comparable: no stochastic tool-failure channel of the kind that drives Measurement 3.2 is exercised in this protocol, and we found the surface deterministic under reset (ten tasks) and under mid-trajectory checkpoint/replay on five short scripted prefixes, not TRACE’s own (0/10 and 0/5 tasks with any mismatch, Appendix § 9), where a naive file-copy restore was nonetheless found to reuse cached in-process engines silently. Whether the 590 reached states were identical was measured neither by the authors nor by us: TRACE is the closest executable-surface precedent, and it reports no measured replay certificate, so its pairing is unverified.

A prefix taxonomy not reported here

A third measurement was planned: an observational taxonomy of boundary prefixes, contrasting continuation burden across classes. It is not reported here, on our own artifact review, for two reasons. First, the label did not measure its construct—prefixes classified as containing a state-changing write predominantly contained authentication endpoints as their only non-read call, so the label separated boundaries by whether the agent had yet logged into any application (a trajectory-phase proxy). Second, a first agreement check re-applied the labeling rule to the same endpoint-method map and was discarded as circular; a later map-blind re-labeling by two labelers of different model families agreed with the map on all 119 endpoint functions, which validates the map but not the construct. No estimate, interval, or cell count from the taxonomy appears anywhere in this paper.

History arms and paired contrasts (design only)

The design proposed in the second agenda item has never been run: every quantity below is a design parameter, not a result. Two local open-weight checkpoint pairs supply the powered lane (288 states on the main surface) and a half-grid replication of it; two smaller surfaces (36–48 states) replicate for sign; two hosted models form a frontier lane.

Arms

The design has eight arms (Table 6): two length controls, four selected content-by-role cells from the crossing of error content (healed, natural own, matched foreign, synthetic) with source role (self, prior-agent wrapper) at a frozen executable state, and two fixed-packet provenance arms outside the four content levels.

#ArmError contentSource roleIsolates
1Short verified statenone—fresh-context baseline
2Clean-longinert, token/turn-matchedselflength tax LL (with 1)
3Healed ownerrors corrected in placeselferror-content counterfactual
4Untouched ownnatural errorsselfΔerr\Delta_{\mathrm{err}} (with 3)
5Natural foreignmatched real foreign errorsprior-agentprovenance ×\times natural content
6Synthetic errorinjected matched errorsselfreproduction of the synthetic-injection result [24]
7Byte-identical selffixed error packetselfΔprov\Delta_{\mathrm{prov}} (with 8)
8Byte-identical foreignsame bytesprior-agentprovenance with content held fixed [6]

Table 6. History arms. Only assistant-side text differs; the frozen executable state (environment and process state), the user turns and all tool outputs are identical across arms. Arms 3–6 instantiate four of the eight crossed content-by-role cells (healed and synthetic content appear only under the self wrapper); arms 7–8 carry the provenance contrast on a fixed packet outside the four content levels; arm 5 is the natural-content, prior-agent cell and is not a clean provenance contrast.

Matching rules

  • Long-history arms are matched on tokenizer-specific token count and on assistant/tool-turn count; only the short-state control differs in length.

  • Assistant-side text only: transformations touching user or tool content are rejected automatically. Arms 7 and 8 are hash-verified and differ in the attributing wrapper alone.

  • A state is admitted only if five restored clones give identical state hashes and grader outcomes and an unedited-history restored continuation matches a separately executed live or prefix-rebuilt reference under the experiment’s policy, seed, and provider regime, with the pathway rate or distributional margin fixed in advance. Clone agreement alone is insufficient, since Measurement 3.2 found grader agreement across clones whose trajectories had diverged from the live reference; the snapshot must also carry process state (Appendix § 9).

  • A frozen judge builds and verifies the arms but never scores an outcome; it must first reach ≥0.90\ge0.90 precision and ≥0.85\ge0.85 recall on 400 synthetic corruptions plus 200 blinded, human-labeled real spans.

  • Primary success is exact and programmatic; on a pre-specified quarter of states, handoffs (a continuation started by one model from a checkpoint written by its partner model) are run in both directions between the two local open-weight checkpoint partners and between the two hosted frontier models.

Three non-additive probes

Three paired contrasts probe where early→\tolate decay comes from, a length tax, an error-content effect and a provenance effect, each identified on its own content distribution (Table 7); they do not share a target distribution and are not asserted to sum to the total decay, so no residual term is identified. Arm 5 changes content and role together, and arms 7–8 use a fixed packet outside the four content levels, so the provenance contrast is identified on its own distribution. Table 8 states what each probe’s sign would imply. If none of the three probes moves success, the design has not located the decay; it does not bound untested interventions.

SymbolContrastArms
LLpshort−pclean−longp_{\mathrm{short}} - p_{\mathrm{clean-long}}1 vs. 2
Δerr(d)\Delta_{\mathrm{err}}(d)phealed(d)−pown(d), d∈{early,late}p_{\mathrm{healed}}(d) - p_{\mathrm{own}}(d),\ d \in\{\mathrm{early},\mathrm{late}\}3 vs. 4
Δforeign,Δsynth\Delta_{\mathrm{foreign}}, \Delta_{\mathrm{synth}}phealed−pforeign, phealed−psynthp_{\mathrm{healed}} - p_{\mathrm{foreign}},\ p_{\mathrm{healed}} - p_{\mathrm{synth}}3 vs. 5, 3 vs. 6
Δprov\Delta_{\mathrm{prov}}pbyte−self−pbyte−foreignp_{\mathrm{byte-self}} - p_{\mathrm{byte-foreign}}7 vs. 8
SS[Δerr(late)−Δerr(early)]/[pown(early)−pown(late)][\Delta_{\mathrm{err}}(\mathrm{late}) - \Delta_{\mathrm{err}}(\mathrm{early})] / [p_{\mathrm{own}}(\mathrm{early}) - p_{\mathrm{own}}(\mathrm{late})]3, 4

Table 7. Estimands. All are risk differences in percentage points of exact continuation success, estimated pairwise within forked states and cluster-bootstrapped over tasks, except SS, which is dimensionless and reported unclipped. The reproduction tests compare Δforeign\Delta_{\mathrm{foreign}} against Δsynth\Delta_{\mathrm{synth}} as Δerr\Delta_{\mathrm{err}} as ±\pm5-point equivalence tests (both interval bounds inside the margin), not as failures to reject. SS is secondary, reported only when its denominator is at least 15 points (the floor assumed in the power sketch below) and its numerator sign-stable; the claim that contaminated history explains less than half of decay requires the 95% upper bound below 0.5.

ProbeSignatureRepair implied
Error contentΔerr\Delta_{\mathrm{err}}(late) large; foreign and synthetic arms reproduce it inside the ±\pm5-point margin; Δprov\Delta_{\mathrm{prov}} smallhistory editing
ProvenanceΔprov\Delta_{\mathrm{prov}} large with the bytes held fixed; the reproduction tests failrole-wrapper handling
Length taxLL large; contaminated arms barely move once token and turn counts are matchedneither; the effect was context length
None of the threeno contrast estimated precisely enough to locate the decay (the design is not powered for a 5-point equivalence margin)not located by this design

Table 8. What each probe’s sign would imply, and the repair it points to. Because each sign implies a different repair, the intended result is the set of contrasts, not the sign of any single one.

Power sketch

Primary contrasts are within-state paired differences. On the powered lane, a configurable-horizon and deterministically graded tool environment [31], 288 paired states clustered in 144 tasks run at three repetitions per arm. With p≈0.4p \approx0.4–0.60.6 at the anchors and within-task ICC ≈0.2\approx0.2, the standard error of an arm difference under a zero-within-state-correlation planning approximation is ≈2p(1−p)/neff\approx\sqrt{2p(1-p)/n_{\mathrm{eff}}} with neff≈400n_{\mathrm{eff}} \approx400–550550, i.e. 3.0–3.5 points, so the minimum detectable effect at 80% power, α=.05\alpha= .05 two-sided, is (1.96+0.84) SE≈8(1.96+0.84)\ \mathrm{SE} \approx8–10 points. The 5-point equivalence margins therefore require either neff≈1,500n_{\mathrm{eff}} \approx1{,}500 effective pairs or a pilot-estimated within-state covariance, which may shrink or widen the paired standard error; an earlier draft of this design quoted 4–5 points, which does not follow from the formula above; as sized, the design is powered for the sign of the primary contrasts, not for the margins. The two replication lanes (the two smaller surfaces, 36–48 states each) and the frontier lane (the two hosted models) are powered only for large effects on the same arithmetic (minimum detectable effects of roughly 20 points or more, revised from the 8–12 and 8–10 points an earlier version of this design quoted under the correlation assumption set aside above) and serve as sign checks. With a denominator of at least 15 points and Δerr\Delta_{\mathrm{err}} standard errors near 3 points, the standard error of SS is roughly 0.3 and a 95% interval roughly ±0.55\pm0.55: too wide to place SS against 0.5 except at the extremes, which is why SS is secondary. Variance components are pilot-estimated.

Cells already occupied

Three cells are partly occupied. Sinha et al. run healed and controlled-error histories, but on a synthetic running-sum task with the errors always in the assistant role: content varies only under the assistant role, so no provenance effect or content-by-provenance interaction is identified, and no tokenizer-exact length match is reported [24]. Chen et al. run the byte-identical role swap, but on single claims in short math and logic traces, scoring correction behavior rather than end-task success [6]. Min et al. occupy the compaction cell at matched decision points on an executable surface [20]. What is missing is a matched comparison of content and source role at a frozen executable state, with a length control and scoring on task success. The design has not been run.

Literature census

Question. Of the papers in our reading list whose primary contribution is failure taxonomy, attribution, localization or empirical diagnosis for LLM agents, how many manipulate the agent’s history or state mid-run and continue the agent from there?

Population. The frame is the 200 index cards read for this project: a convenience sample, not a systematic search. Two independent LLM agents coded it, the second blind to the first. Both read only our own secondhand summaries, no PDFs, so the codes describe the cards, not the papers, and can err in both directions (sparse cards create false negatives, misreadings false positives); silence was coded as silence.

Rubric and Stage 1. Table 9 states the frozen rubric. Of the 200 cards, 41 passed Stage 1. Both Stage 2 tables cover the same 41: the 41st (2603.24755) was promoted out of the excluded set mid-census, and coder 2—given 40 cards first—had independently screened it in.

Code set
Stage 1screen from title, TL;DR and tags; include if the paper (a) studies LLM agents or multi-turn LLM systems and (b) contributes primarily a taxonomy of failure modes, a failure-attribution or failure-localization method or benchmark, or an empirical diagnosis of why agents fail. Exclude capability benchmarks without failure analysis, methods papers whose failure discussion is incidental, surveys that do not themselves categorize failures, and position papers with no coded failure data
Q1 label sourcehuman annotation; LLM judge; rule-or-automatic; mixed; none; not stated in card
Q2 interventionnone; system-or-task change scored end-to-end; history-or-state manipulation at an intermediate point with continuation; other
Q3 classhealed history; provenance swap; placebo or length-matched synthetic history; counterfactual injection; environment-or-state reset; replay from checkpoint without content change; other
Q4 frozen statedoes the paper restore an executable intermediate state (environment plus process) and continue the same agent from it? yes; prefix re-execution only; no; not determinable from card
Q5 evidenceone verbatim quote (≤30\le30 words) from the card supporting Q2 and Q4
Q6 confidencehigh; medium; low (low whenever the card is silent on a coded dimension)

Table 9. The rubric as frozen before coding.

Table 9: The rubric as frozen before coding. Q3 is asked only when Q2 is a history-or-state manipulation, and admits multiple codes.

Table 10: Stage 2 counts, the two coders side by side (N=41N = 41 each). Q2 and Q4 each partition the 41 cards.

Coder 1Coder 2
Q2 = history-or-state manipulation with continuation2213
Q2 = system-or-task change scored end-to-end1322
Q2 = none66
Q4 = yes (frozen executable state, same agent continued)13
Q4 = prefix re-execution only11
Q4 = not determinable from card32
Q4 = no3635

Table 10. Stage 2 counts, the two coders side by side.

Agreement. Table 10 gives both coders’ counts and Tables 11 and 12 the confusion matrices. On Q2 the coders agreed on 32 of 41 cards—78.0%, Cohen’s κ=0.66\kappa= 0.66. That is low, and one-sided: all nine disagreements are coder 1 coding a history-or-state manipulation where coder 2 coded a system-or-task change scored end-to-end. Only coder 2 wrote that boundary down beforehand, treating a run scored from its start under a fixed continuation as end-to-end. On Q4 they agreed on 38 of 41—92.7%, κ=0.70\kappa= 0.70. Re-screening every fifth card of the excluded list alphabetically (32 of a 160-line list drawn before the 41st card was promoted out of it), coder 2 agreed with coder 1’s original screen on 31 of 32 and with the final revision on all 32; the one card that moved, 2603.24755, is the promotion described above.

Frozen state, and the three Q4 disagreements. Both coders coded 2608.06503 [20] yes and Who&When Pro, 2607.09996 [17], prefix re-execution only; coder 2 coded two more yes. Q4 counts throughout are the coders’ assigned codes, not verified restorations. By our own reading of its protocol (Appendix § 10), 2608.06503 re-executes a recorded prefix under a recipe hash, which coder 2’s written convention would code as prefix re-execution only; we leave both coders’ codes as recorded; harmonizing that one known inconsistency gives frozen-state counts of 0 and 2 and prefix-re-execution counts of 2 and 2, and the body reports the positively identified counts under both readings. On ChainSWE, 2607.02606 [12], both quoted the card’s line that oracle mode resets the container to the base commit and applies earlier gold patches before each stage, which coder 2’s convention makes a yes. On 2608.13417 coder 2 called its own yes “the least secure of the three,” the card never saying the running environment and process are restored. Coder 1 quoted the same lines in both cases and recorded no reason for its no. The third disagreement, 2607.24167, is not determinable for coder 1—local resume or safe-state backtracking, environment and process unstated—and no for coder 2.

Coder 2
Coder 1man.e2enonetot.
manipulation139022
end-to-end013013
none0066
total1322641

Table 11. Q2 confusion (rows: coder 1).

Coder 2
Coder 1yespre.non.d.tot.
yes10001
prefix only01001
no2034036
not determ.00123
total3135241

Table 12. Q4 confusion (rows: coder 1).

Not determinable. Only two cards are not determinable for both coders: 2509.25370, which re-rolls out from the critical step, and 2608.02464, which rolls a run back and re-runs it live. Neither says whether state is restored or the prefix re-executed; coder 1 also marked 2607.24167. They are the likeliest true frozen-state replays, so the coders positively identified one and three such papers (zero and two after harmonizing Min et al.), plus one or two prefix re-executions; because the cards are secondhand, no upper bound on the literature follows.

Limitations. Two executing methods published outside the frozen frame, DoVer [19] (checkpointed multi-agent debugging) and CausalFlow [4] (step replacement with downstream re-execution where an executor exists), would code as history-or-state manipulations with continuation; neither reports a replay-fidelity measurement, so neither changes the frozen-state counts. The second coder recommended resolving the two shared not-determinable cards from the papers before publication. One (AgentDebug, arXiv 2509.25370) was read from the primary PDF after the census closed: it re-rolls out from the identified critical step with feedback, up to five attempts, but does not state how that step’s state is reached and reports no fidelity measure, so its frozen-state code stays not determinable and no count changes; the other was not read, and the counts above are card-level. One code per coder, no adjudication, no third rater: each κ\kappa rests on one 41-card pairing, and the Q4 value sits on a table whose 34 no–no cells dominate. The inclusion check covers a fifth of the excluded cards. The paper reports the range.

#arXivShort titleQ2Q4Coder 2 differs
222505.00212Which Agent Causes Task Failures?noneno—
232606.06324From Failed Trajectories to Reliable Agentse2eno—
242505.23840Measuring Sycophancy in Multi-turn Dialoguese2eno—
252606.22528Governance Decayman.no—
262607.09510Failure as a Processnoneno—
272603.24631Coherence Collapsee2eno—
282603.06870LEAD: the no-recovery bottleneckman.noQ2 e2e
292503.13657Why Do Multi-Agent LLM Systems Fail?e2eno—
302608.18066On the Fragility of Self-Improving Agentse2eno—
312310.01798LLMs Cannot Self-Correct Reasoning Yetman.noQ2 e2e
322606.07889Strained Coherencenoneno—
332605.08060The Memory Curseman.no—
342509.25370Where LLM Agents Fail and How They Learnman.n.d.—
352608.02464Real-Time Detection and Repair of Failuresman.n.d.—
362505.06120LLMs Get Lost in Multi-Turn Conversatione2eno—
372604.11978The Long-Horizon Task Mirage?e2eno—
382505.02709Evaluating Goal Drift in LM Agentsman.no—
392502.15840Vending-Benche2eno—
402502.19559Stay Focused: problem drift in debateman.no—
412603.24755SlopCodeBenchman.noQ2 e2e

Table 14. The 41 included cards, part 2 of 2 (cards 22–41). Columns as in Table 13.

#arXivShort titleQ2Q4Coder 2 differs
12608.06503Toward Reliable Context Compressionman.yes—
22607.24167Falsifiable Commitment Planningman.n.d.Q2 e2e; Q4 no
32601.23045The Hot Mess of AIe2eno—
42509.09677The Illusion of Diminishing Returnsman.no—
52608.11772Diagnosis Before Recoverye2eno—
62607.02606ChainSWE: multi-bug maintenanceman.noQ4 yes
72601.00828Decomposing LLM Self-Correctionman.noQ2 e2e
82607.22585The Scaffold Effect in Coding Agentse2eno—
92608.15242LongRCA Benchnoneno—
102606.05806When Tools Failman.no—
112608.19652Can Agent Memory Systems Track Evolving State?man.noQ2 e2e
122608.13417Beyond Final Scoresman.noQ4 yes
132604.02547Beyond Resolution Ratesnoneno—
142606.29718Diagnosing and Mitigating Context Rotman.noQ2 e2e
152607.07989Who Broke the System?noneno—
162605.24279ContextEchoman.no—
172605.27898A Unified Framework for Agentic Evaluatione2eno—
182607.09996Who&When Proman.pre.—
192603.04474From Spark to Fireman.noQ2 e2e
202605.24197A Sober Look at Agentic Misalignmente2eno—
212602.07338Intent Mismatch Causes LLMs to Get Lostman.noQ2 e2e

Table 13. The 41 included cards, part 1 of 2 (cards 1–21). Q2: man. = history-or-state manipulation with continuation, e2e = system-or-task change scored end-to-end. Q4: pre. = prefix re-execution only, n.d. = not determinable from card. The last column shows coder 2’s code wherever it differs. Short titles are abridged from each card’s frontmatter.

Use of large language models

LLM agents were used as instruments in two reported measurements, as stated where they occur: the same-agent re-labeling pass of Measurement 3.1 (Appendix § 8) and the two-coder literature census (Appendix § 12). LLM assistance was additionally used in the engineering of the system and benchmark code and in manuscript preparation, under the authors’ direction and review; the authors take full responsibility for all content, claims, and measurements.

References

  1. [1]Berk Atil, Sarp Aykent, Alexa Chittams, Lisheng Fu, Rebecca J. Passonneau, Evan Radcliffe, Guru Rajan Rajagopal, Adam Sloan, Tomasz Tudrej, Ferhan Ture, Zhe Wu, Lixinyu Xu, and Breck Baldwin. Non-Determinism of "Deterministic" LLM Settings. arXiv preprint arXiv:2408.04667, 2024.arxiv.org/abs/2408.04667
  2. [2]Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment. arXiv preprint arXiv:2506.07982, 2025.arxiv.org/abs/2506.07982
  3. [3]Paul C. Bogdan, Uzay Macar, Neel Nanda, and Arthur Conmy. Thought Anchors: Which LLM Reasoning Steps Matter? arXiv preprint arXiv:2506.19143, 2025.arxiv.org/abs/2506.19143
  4. [4]Akash Bonagiri, Devang Borkar, Gerard Janno Andérias, Setareh Rafatirad, and Houman Homayoun. CausalFlow: Causal Attribution and Counterfactual Repair for LLM Agent Failures. arXiv preprint arXiv:2605.25338, 2026.
  5. [5]Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Why Do Multi-Agent LLM Systems Fail? arXiv preprint arXiv:2503.13657, 2025.arxiv.org/abs/2503.13657
  6. [6]Kuan-Yen Chen, Fang-Yi Su, Shih-Yen Lin, Bao Li, and Jung-Hsien Chiang. The Self-Correction Illusion: Role Labeling Gates Explicit Error Flagging in Large Language Models. arXiv preprint arXiv:2606.05976, 2026.arxiv.org/abs/2606.05976
  7. [7]Mengzhuo Chen, Junjie Wang, Fangwen Mu, Yawen Wang, Zhe Liu, Huanxiang Feng, and Qing Wang. Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems. arXiv preprint arXiv:2604.22708, 2026.arxiv.org/abs/2604.22708
  8. [8]C. J. Clopper and E. S. Pearson. The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial. Biometrika, 26(4):404–413, 1934.DOI
  9. [9]Yunpeng Dong, Jingkai He, Shiqi Liu, Yuze Hou, Dong Du, Zhonghu Xu, Si Yu, Baochuan Yang, Yubin Xia, and Haibo Chen. DeltaBox: Scaling Stateful AI Agents with Millisecond-Level Sandbox Checkpoint/Rollback. arXiv preprint arXiv:2605.22781, 2026.
  10. [10]Jie Gao, Kaiser Sun, Jen-Tse Huang, Katherine Van Koevering, Sijie Ji, Heyuan Huang, Weiyan Shi, Zhuoran Lu, Ziang Xiao, Daniel Khashabi, and Mark Dredze. How to Interpret Agent Behavior. arXiv preprint arXiv:2605.13625, 2026.arxiv.org/abs/2605.13625
  11. [11]Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal Abstractions of Neural Networks. Advances in Neural Information Processing Systems 34 (NeurIPS 2021); arXiv:2106.02997, 2021.arxiv.org/abs/2106.02997
  12. [12]Qirui Jin, Lingching Tung, Kenan Li, Qiyang Shi, Yushi She, Huanzhong Jia, Harrison Zhao, Kejing Xia, Zhenbang Du, Jiaxin Pei, Zhenyu Zhang, Zhen Qi, Yuyuan Duan, Wenke Lee, and Zijian Jin. ChainSWE: Benchmarking Coding Agents on Multi-Bug Software Maintenance. arXiv preprint arXiv:2607.02606, 2026.arxiv.org/abs/2607.02606
  13. [13]Haechang Kim, Hao Chen, Can Li, and Jong Min Lee. TalkToAgent: A Human-centric Explanation of Reinforcement Learning Agents with Large Language Models. arXiv preprint arXiv:2509.04809, 2025.arxiv.org/abs/2509.04809
  14. [14]Fanqi Kong, Ruijie Zhang, Huaxiao Yin, Guibin Zhang, Xiaofei Zhang, Ziang Chen, Zhaowei Zhang, Xiaoyuan Zhang, Song-Chun Zhu, and Xue Feng. Aegis: Automated Error Generation and Attribution for Multi-Agent Systems. arXiv preprint arXiv:2509.14295, 2025.arxiv.org/abs/2509.14295
  15. [15]Averill M. Law. Simulation Modeling and Analysis. McGraw-Hill, New York, 5th edition, 2015.
  16. [16]Xiaofeng Lin, Yingxu Wang, Tung Sum Thomas Kwok, Daniel Guo, Sahil Arun Nale, Charles Fleming, and Guang Cheng. REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Traces. arXiv preprint arXiv:2606.09071, 2026.arxiv.org/abs/2606.09071
  17. [17]Jiale Liu, Huajun Xi, Shaokun Zhang, Yifan Zeng, Tianwei Yue, Chi Wang, Jian Kang, Qingyun Wu, and Huazheng Wang. Who&When Pro: Can LLMs Really Attribute Failures in AI Agents? arXiv preprint arXiv:2607.09996, 2026.arxiv.org/abs/2607.09996
  18. [18]Jiayuan Liu, Tianqin Li, Shiyi Du, Xin Luo, Haoxuan Zeng, Emanuel Tewolde, Tai Sing Lee, Tonghan Wang, Carl Kingsford, and Vincent Conitzer. The Memory Curse: How Expanded Recall Erodes Cooperative Intent in LLM Agents. arXiv preprint arXiv:2605.08060, 2026.arxiv.org/abs/2605.08060
  19. [19]Ming Ma, Jue Zhang, Fangkai Yang, Yu Kang, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems. arXiv preprint arXiv:2512.06749, 2025.arxiv.org/abs/2512.06749
  20. [20]Guanghui Min, Liang Wu, Mayank Darbari, Chen Chen, and Liangjie Hong. Toward Reliable Context Compression for Long-Horizon Agents: An Empirical Study of Execution Instability. arXiv preprint arXiv:2608.06503, 2026.arxiv.org/abs/2608.06503
  21. [21]Guanghui Min, Liang Wu, Mayank Darbari, Chen Chen, and Liangjie Hong. TRACE: paired-continuation corpus and verifier. https://github.com/nokia-applied-research/Trace, 2026. Revision 9be7d669.
  22. [22]Robert O’Callahan, Chris Jones, Nathan Froyd, Kyle Huey, Albert Noll, and Nimrod Partush. Engineering Record and Replay for Deployability. In 2017 USENIX Annual Technical Conference (USENIX ATC 17), pages 377–389, 2017. Extended technical report: arXiv:1705.05937.arxiv.org/abs/1705.05937
  23. [23]Jaineet Shah. Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures. arXiv preprint arXiv:2606.08275, 2026.arxiv.org/abs/2606.08275
  24. [24]Akshit Sinha, Arvindh Arun, Shashwat Goel, Steffen Staab, and Jonas Geiping. The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs. In International Conference on Learning Representations (ICLR), 2026. arXiv:2509.09677.arxiv.org/abs/2509.09677
  25. [25]Kevin Song, Anand Jayarajan, Yaoyao Ding, Qidong Su, Zhanda Zhu, Sihang Liu, and Gennady Pekhimenko. Aegis: Taxonomy and Optimizations for Overcoming Agent-Environment Failures in LLM Agents. arXiv preprint arXiv:2508.19504, 2025.arxiv.org/abs/2508.19504
  26. [26]Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents. Proceedings of ACL 2024; arXiv:2407.18901, 2024.arxiv.org/abs/2407.18901
  27. [27]Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, and Stuart Shieber. Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias. Advances in Neural Information Processing Systems 33 (NeurIPS 2020); arXiv:2004.12265, 2020.arxiv.org/abs/2004.12265
  28. [28]Xinyu Jessica Wang, Haoyue Bai, Yiyou Sun, Haorui Wang, Shubai Zhang, Wenjie Hu, Mya Schroder, Bilge Mutlu, Dawn Song, and Robert D. Nowak. The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break. arXiv preprint arXiv:2604.11978, 2026.arxiv.org/abs/2604.11978
  29. [29]Yufei Xia, Anjun Gao, Yueyang Quan, Zhuqing Liu, and Minghong Fang. Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems. arXiv preprint arXiv:2607.07989, 2026.arxiv.org/abs/2607.07989
  30. [30]Shuyao Xiao, Shengling Wang, Xuan Chen, Ke Chao, Ming Cui, Feifei Qian, Fanlin Meng, Chaoyang Mei, Chaoyong Jiang, Qi Ouyang, and Junxi Yi. When Errors Become Memories: Causal Pathway Tracing in Multi-Turn Memory-Augmented LLMs. arXiv preprint arXiv:2608.30198, 2026.arxiv.org/abs/2608.30198
  31. [31]Wang Yang, Chaoda Song, Xinpeng Li, Debargha Ganguly, Chuang Ma, Shouren Wang, Zhihao Dou, Yuli Zhou, Vipin Chaudhary, and Xiaotian Han. AgentCE-Bench: Agent Configurable Evaluation with Scalable Horizons and Controllable Difficulty under Lightweight Environments. arXiv preprint arXiv:2604.06111, 2026.arxiv.org/abs/2604.06111
  32. [32]Hongwei Yao, Yiming Liu, Meihui Chen, Jieling Chen, Zikun Chen, Yiling He, Wangze Ni, Cong Wang, and Kui Ren. ActBench: Self-Evolving Benchmark of Behavioral Safety in Cowork Agents. arXiv preprint arXiv:2608.09476, 2026.arxiv.org/abs/2608.09476
  33. [33]Jiayi Yuan, Hao Li, Xinheng Ding, Wენya Xie, Yu-Jhe Li, Wenti an Zhao, Kun Wan, Jing Shi, Xia Hu, and Zirui Liu. Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference. arXiv preprint arXiv:2506.09501, 2025.arxiv.org/abs/2506.09501
  34. [34]Guibin Zhang, Junhao Wang, Junjie Chen, Wangchunshu Zhou, Kun Wang, and Shuicheng Yan. AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems? arXiv preprint arXiv:2509.03312, 2025.arxiv.org/abs/2509.03312
  35. [35]Jiayi Zhang, Zexin Wang, Degang Sun, Changhua Pei, Fei Sun, Gaogang Xie, and Jingjing Li. Detect Before You Attribute: Cascade Failure Attribution for Multi-Agent Systems. arXiv preprint arXiv:2608.29646, 2026.arxiv.org/abs/2608.29646
  36. [36]Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiqiang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, and Qingyun Wu. Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. arXiv preprint arXiv:2505.00212, 2025.arxiv.org/abs/2505.00212
  37. [37]Xijia Zhang, Yue Guo, Simon Stepputis, Katia Sycara, and Joseph Campbell. Understanding Your Agent: Leveraging Large Language Models for Behavior Explanation. arXiv preprint arXiv:2311.18062, 2023.arxiv.org/abs/2311.18062
  38. [38]Chenyang Zhu, Spencer Hong, Jingyu Wu, Kushal Chawla, Charlotte Tang, Youbing Yin, Nathan Wolfe, Erin Babinsky, and Daben Liu. RAFFLES: Reasoning-based Attribution of Faults for LLM Systems. arXiv preprint arXiv:2509.06822, 2025.arxiv.org/abs/2509.06822
  39. [39]Kunlun Zhu, Zijia Liu, Bingxuan Li, Muxin Tian, Yingxuan Yang, Jiaxun Zhang, Pengrui Han, Qipeng Xie, Fuyang Cui, Weijia Zhang, Xiaoteng Ma, Xiaodong Yu, Gowtham Ramesh, Jialian Wu, Zicheng Liu, Pan Lu, James Zou, and Jiaxuan You. Where LLM Agents Fail and How They can Learn From Failures. arXiv preprint arXiv:2509.25370, 2025.arxiv.org/abs/2509.25370

Paper details

vlnc:W_gsg2qrz2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

LLM assistance

View statement
Looking for related papers…
Contents