betaFeedback

WorksComputer Science

Knowing When to Intervene: Causal Evaluation of AI Agents

Abstract

Current evaluations of agentic AI systems primarily rely on benchmarks. However, the static nature of benchmarks makes them uninformative about the value of auditing interventions at deployment time. For reliable deployment, we often need to determine within an agent's trajectory whether or not to intervene, e.g., by verifying the agent's current output or steering it toward different behavior, using either human review or more costly models or time-consuming agentic supervision. The value of such interventions is fundamentally a causal inference question, asking whether the causal effect of such an intervention is beneficial for the final outcome. Estimating this causal effect is particularly challenging due to the dynamic nature of the problem: the impact of an intervention depends on the current context and environment, as well as the agent's future behavior. We develop the first methodologically grounded framework for evaluating the causal effects of randomized interventions within agent trajectories. By leveraging a relatively small dataset of agent trajectories collected under randomized interventions such as verification or screening, we can estimate the dynamic causal effect of a rich class of (contextual) intervention policies. The structural parameters that we estimate can also be used to construct safety leaderboards that capture the risk of various agentic actions, which can be used by practitioners for audit prioritization. Applying our methods to tau-bench Retail, ToolSandbox, and ImpossibleBench, we find that intervention effects vary substantially across agent actions and states. More broadly, our work suggests a rich research agenda at the intersection of causal inference and agentic evaluation, which we believe is essential for the reliable and safe deployment of AI agents in the real world.

Introduction

Agentic AI systems are designed to solve complex, long-horizon tasks through a sequence of decisions and tool calls in interactive environments. Current evaluations of these systems typically rely on benchmarks, each of which fixes a set of tasks, an environment with its tools and instructions, and criteria for scoring the outcome [4, 15, 19, 34, 37]. For example, in τ\tau-bench Retail [34], a customer-support agent interacts with simulated customers, retrieves account information, and manages orders through external tools. The task outcome then depends on the entire trajectory of interactions between the agent and the environment. While a valuable source of data for characterizing agent capabilities and safety, benchmarks have significant limitations in guiding how we intervene during real-world use.

In real-world deployments, humans need to determine within an agent’s trajectory whether or not to intervene, e.g., by verifying a proposed tool call, steering the agent toward a different action, or screening potentially adversarial content. This review process can correspond to either human review in high-stakes domains, or, in medium-stakes domains, to more costly (and more powerful) model supervision, or supervision via a more time-consuming agentic harness. For instance, would reviewing a proposed database write prevent a task failure, or would withholding an adversarial message prevent an agent from violating its instructions? Such decisions are fundamentally causal inference questions: the user must determine whether applying an intervention would improve the final outcome.

Estimating this causal effect is particularly challenging because it is a dynamic treatment effect problem. The impact of an intervention during task execution depends on the current context and environment, as well as the agent’s future behavior under its policy. In particular, an intervention can change subsequent states, actions, and intervention opportunities, so its effect also depends on how later interventions are assigned. Neither observed benchmark trajectories nor data from trajectory-level randomized experiments (see, e.g., [1, 9, 21]) fully resolve these questions. While analyses of observed benchmark trajectories can reveal potential intervention points, they cannot by themselves identify the causal effects of intervening at those points.

In this paper, we develop a methodologically grounded framework for the causal evaluation of interventions in agentic AI. We show that conducting random interventions in a single trajectory can support the estimation of multiple causal estimands at different points during task execution. Building on the framework of micro-randomized trials [16], we randomize intervention assignments at each eligible decision within a trajectory. At each eligible decision point, the experimental harness records the agent’s proposed action and observed history, randomly assigns intervention or control, and allows the agent to continue interacting with the environment (Figure 1).

Sequential randomization in $\tau$-bench Retail

Figure 1. Sequential randomization in τ\tau-bench Retail. Panel (a) assigns each eligible tool proposal to direct execution (At=0A_t = 0) or review (At=1A_t = 1); panel (b) shows the corresponding causal diagram, where XX is the task, LtL_t is the current observed information, YY is terminal success, and UU is unobserved agent state.

Despite its simplicity, this experimental design generates data that support causal evaluation of intervention policies beyond the randomized policy. Under standard causal identification assumptions (discussed in § 2.4), the data identify the causal effects of a rich class of intervention policies that depend on observable history. The same data also support comparisons across groups of intervention points defined by pre-assignment information, such as tool-call types, task stages, or patterns of prior failures. This analysis does not require access to the agent’s internal state or a Markovian model of its interactions with the environment. It only requires black-box input-output access to the AI agent.

We introduce three causal estimands to address different causal questions about interventions during agent execution, each with a corresponding estimation method:

  • Blip with randomized continuation: The causal effect of intervening at a particular decision when future interventions continue to be randomized. We estimate this effect using cross-fitted augmented inverse probability weighting (AIPW) with known assignment probabilities.

  • Blip with no-treatment continuation: The causal effect of intervening at a particular decision when no further interventions are applied for the remainder of the trajectory. We estimate this effect using structural nested mean models (SNMMs) and g-estimation [17, 26, 31].

  • Policy gain (off-policy evaluation): The causal effect of applying a target policy that intervenes based on the agent’s observed history, relative to never intervening throughout the trajectory. We estimate this gain using continuation-specific SNMMs and g-estimation.

These estimands are formally defined in § 2.3, with the first two measuring blip effects under different continuation policies and the third evaluating the gain of a target intervention policy through off-policy evaluation. Estimating the latter two effects using our SNMM approach requires additional assumptions on the specification of the blip-effect models beyond experimental randomization.

These blip effect estimands can also be useful as attribution tools, attributing risk or success to specific decisions in a trajectory. For example, if an agent fails to complete a task, the blip effect of intervening at a particular decision can indicate whether that decision was a key contributor to the failure. Conversely, if an agent succeeds, the blip effect can reveal whether intervening at a certain point would have improved the outcome even further. This attribution perspective can help users understand which decisions are most critical for task success or failure, guiding future interventions and policy design. These attribution effects can be estimated for individual decisions or averaged over groups of similar decisions, providing insights into the causal impact of specific actions within the agent’s trajectory. At various levels of granularity, these blip effects can be reported to users in the form of a “decision impact score,” leaderboard, which quantifies the expected change in terminal outcome if an intervention were applied at that group of agentic decision. This score can help users prioritize interventions and understand the causal dynamics of the agent’s behavior.

We apply our framework to tool-call review in τ\tau-bench Retail [4, 34] and ToolSandbox [20], and to adversarial feedback in ImpossibleBench [36]. In τ\tau-bench Retail, we estimate positive effects of tool-call review on task success for several order-handling groups. In ToolSandbox, the estimated effects of tool-call review vary across capabilities and agents. In ImpossibleBench, we estimate an increased probability of prohibited test edits following adversarial feedback in particular agent states, including near-completion states with contradictory tests. Across all three benchmarks, our experiments show how randomized interventions can identify decisions where tool-call reviews may improve or harm task performance, as well as states where adversarial feedback may increase safety violations.

More broadly, our work shifts the focus of agent evaluation beyond static benchmark assessments toward the decisions users face during deployment: not only whether an agent succeeds or fails, but also whether intervening at a particular point would change the outcome. This perspective opens up a broader research agenda at the intersection of causal inference and agentic evaluation, with the long-term goal of enabling the reliable and safe deployment of agentic AI systems in the real world.

Related work. Agent evaluation and intervention-based analysis. Existing benchmarks evaluate agent capabilities and safety across various interactive tasks [12, 20, 34, 36]. Recent work extends these evaluations through trajectory-level failure analysis [3, 10, 35] and intervention-based diagnosis using counterfactual replay or targeted re-execution [7, 22, 27, 33]. Other work develops runtime checks to steer agents or enforce constraints during execution [6, 28, 32]. Our paper complements these approaches by estimating the causal effects of interventions across different agent actions and states using randomized trajectories, without requiring counterfactual replay.

Randomized evaluation of AI agents. Recent studies have used randomized controlled trials (RCTs) to evaluate AI systems by comparing different agent configurations, assistance systems, and interactive environments [1, 9, 21]. These experiments primarily evaluate the effects of treatments assigned at the task or trajectory level. In contrast, we randomize interventions at intermediate decision points, enabling causal evaluation of individual interventions and history-dependent intervention policies from the same experimental data.

Sequential causal inference and policy evaluation. Our framework builds on micro-randomized trials and distal causal excursion effects, which study the effects of repeated interventions as the observed context evolves [8, 16, 25]. Our work is also closely related to dynamic treatment regimes and off-policy evaluation, which study the causal effects of sequential intervention policies [14, 23, 30]. We use augmented inverse probability weighting and structural nested mean models with g-estimation [17, 26, 31] to estimate intervention effects and evaluate intervention policies.

Experimental Design for AI Agent Evaluation

We formalize the randomized intervention design and the three causal estimands introduced above: the blip with randomized continuation, the blip with no-treatment continuation, and the policy gain. We first introduce the agent–environment model in § 2.1 and describe the randomized intervention design in § 2.2. We then define the three estimands in § 2.3 and discuss their identification and estimation in § 2.4, distinguishing the assumptions required for causal identification from the additional modeling assumptions used by our estimators.

Agent–Environment Model

Consider an AI agent interacting with an environment to complete a task. Each trajectory begins with a task input XX (e.g., a prompt) and consists of a sequence of agent actions and environment observations until termination. Let Y∈RY \in\mathbb{R} denote the terminal outcome, e.g., task success in τ-bench Retail, a similarity score in ToolSandbox, or a safety violation in ImpossibleBench.

We index the eligible intervention points within a trajectory by t=1,…,Tt = 1, \ldots, T, where TT is the total number of intervention points reached before termination. The eligibility criteria are specified in advance, but TT is random and may depend on previous interventions. For example, reviewing a proposed tool call may change the agent’s subsequent actions, including the number of tool calls it makes before termination. Let LtL_t denote the information observed at intervention point tt, including the agent’s proposed action, the current environment context, and all the observations since the previous intervention point.

Let At∈{0,1}A_t \in\{0, 1\} denote the assignment, with At=1A_t = 1 for intervention and At=0A_t = 0 for control. For example, an intervention may involve reviewing a proposed tool call before execution, as in τ-bench Retail and ToolSandbox, or providing adversarial feedback after an unsuccessful coding submission, as in ImpossibleBench. In the latter case the intervention’s goal is not to help reduce violations, but rather to measure at which point in the execution an adversarial prompt injection attack is most harmful; which can be valuable in understanding the most vulnerable points of the agentic execution.

The pre-assignment observed history at time tt is then

Ht=(X,L1,A1,…,At−1,Lt).(1)H_t = (X, L_1, A_1, \ldots, A_{t-1}, L_t). \tag*{(1)}

We note that tt indexes intervention points rather than agent actions, though the two may coincide.

Agent dynamics. An intervention can change the agent’s subsequent actions and the environment it encounters, thereby affecting both the terminal outcome and future intervention opportunities. We allow these agent–environment dynamics to depend on the entire interaction history, without assuming Markovianity or stationarity. We treat the agent as a black box: our framework only requires recorded interaction histories, intervention assignments, and terminal outcomes, without access to the agent’s internal state or model parameters.

Randomized Intervention Design

We consider an experimental design that randomly assigns interventions at eligible points within agent trajectories. At each intervention point, we specify an intervention and a control condition. For example, a proposed tool call may be reviewed or executed directly, while an unsuccessful coding submission may receive adversarial or neutral feedback. After each assignment, the agent continues interacting with the environment.

We collect agent trajectories under a randomized intervention policy gg. At each eligible intervention point tt, intervention is assigned conditional on the observed history HtH_t:

At∣Ht=h∼Bernoulli⁡(e(h)),e(h):=g(1∣h),(2)A_t \mid H_t = h \sim\operatorname{Bernoulli}(e(h)), \qquad e(h) := g(1 \mid h), \tag*{(2)}

where e(h)∈(0,1)e(h) \in(0,1) is the specified probability of intervention. In our experiments, we use uniform Bernoulli randomization, setting e(h)=1/2e(h) = 1/2 at every eligible intervention point.

Unlike trajectory-level randomization, our design repeatedly randomizes interventions within each trajectory. Earlier interventions may affect the agent’s subsequent behavior and future intervention opportunities, but each new assignment remains randomized conditional on the observed history. This design allows us to estimate causal effects across different agent actions and states, as well as evaluate history-dependent intervention policies, using the same collection of trajectories without requiring counterfactual replay. Figure 1 illustrates our design in τ\tau-bench Retail.

Causal Estimands

We consider three causal estimands to evaluate the effects of interventions during agent execution (see Figure 2). As a running example, consider a customer-support agent that manages customer orders through tool calls. At each proposed tool call, we may review the call before execution or execute it directly. In this example, we study the causal effects of reviewing proposed order-cancellation calls.

Figure 2. Three causal estimands for interventions during agent execution. Panels (a)–(c) show the blip with randomized continuation, the blip with no-treatment continuation, and the policy gain over never intervening, respectively.

Recall that gg denotes the randomized intervention policy used to generate the experimental data. In contrast, we use π\pi to denote a policy of interest that assigns interventions based on observed histories. For blip effects, π\pi governs interventions after the current decision, whereas for policy gain, it governs interventions throughout the trajectory.

Let Yt(a;π)Y_t(a; \pi) be the potential terminal outcome if we assign At=aA_t = a at the current point and follow π\pi thereafter, keeping the pre-assignment history hh fixed. Following the terminology of structural nested mean models (SNMMs) [31], we define the conditional blip effect as

b(h;π):=E[Yt(1;π)−Yt(0;π)∣Ht=h].(3)b(h; \pi) := \mathbb{E}[Y_t(1; \pi) - Y_t(0; \pi) \mid H_t = h]. \tag*{(3)}

This effect measures how intervening at the current decision changes the expected terminal outcome, accounting for its downstream effects on subsequent agent actions, environment states, and future intervention opportunities.

To evaluate intervention effects across different agent actions and states, let GG denote a group of intervention points defined by their pre-assignment histories. For example, GG may correspond to a class of tool calls, a specific API, or an agent state characterized by repeated failures. In our customer-support example, GG could represent all intervention points at which the agent proposes an order-cancellation call. These groups may overlap or form a hierarchy, allowing us to evaluate interventions at different levels of granularity. We define the group-averaged intervention effect under continuation policy π\pi as

τ(G;π):=Eg[∑t:Ht∈Gb(Ht;π)]Eg[∑t:Ht∈G1],(4)\tau(G;\pi) := \frac{\mathbb{E}_{g}\left[\sum_{t:H_t\in G} b(H_t;\pi)\right]}{\mathbb{E}_{g}\left[\sum_{t:H_t\in G}1\right]}, \tag*{(4)}

where Eg\mathbb{E}_{g} denotes expectation over trajectories generated under the randomized collection policy gg, and the denominator is assumed positive. Thus, the averaging distribution is determined by gg, even when the continuation policy π\pi differs from gg.

We now define the three estimands of interest.

Blip with randomized continuation. We first consider the effect of an intervention when future assignments continue to follow the randomized collection policy. Setting π=g\pi=g, the estimand τ(G;g)\tau(G;g) measures this effect over intervention points in GG. In our example, this measures the average effect of reviewing a proposed order-cancellation call on terminal task success, with subsequent reviews remaining randomized. This group-averaged blip is closely related to distal causal excursion effects in micro-randomized trials [8, 25].

Blip with no-treatment continuation. We next consider the effect of intervening at the current decision with no subsequent interventions. Let π0(h)≡0\pi_0(h)\equiv0 denote the never-intervene policy. We write τ(G;0):=τ(G;π0)\tau(G;0):=\tau(G;\pi_0) as shorthand for the group-averaged blip with no-treatment continuation. In our example, this measures the average effect of reviewing a proposed order-cancellation call when all subsequent tool calls are executed without review. Both blip estimands average over intervention points reached under the collection policy gg, but differ in how future interventions are assigned.

Policy gain. Beyond the effect of intervening at a single decision, we also consider the effect of applying an intervention policy throughout a trajectory. Let Y(π)Y(\pi) denote the potential outcome when π\pi is followed throughout a trajectory. We define the value of a policy as V(π):=E[Y(π)]V(\pi):=\mathbb{E}[Y(\pi)].

For a reporting group GG (e.g., a class of tool calls), consider a selective policy πG\pi_G that intervenes whenever the observed history belongs to GG, and use π0\pi_0 as the reference policy that never intervenes. We define the policy gain as

πG(h)=1{h∈G},π0(h)=0,Δ(G):=V(πG)−V(π0).(5)\pi_G(h)=1\{h\in G\},\qquad\pi_0(h)=0,\qquad\Delta(G):=V(\pi_G)-V(\pi_0). \tag*{(5)}

This estimand measures the difference in expected terminal outcomes between following the selective intervention policy and never intervening. In our example, this compares a policy that reviews every proposed order-cancellation call with a policy that never reviews any tool calls throughout the trajectory.

The policy gain compares expected terminal outcomes under different intervention policies, as in off-policy evaluation (see, e.g., [14, 29]). Since interventions can affect subsequent agent behavior and future intervention opportunities, different policies may generate different trajectories. Thus, the policy gain cannot generally be obtained by simply aggregating individual blip effects. Figure 2 illustrates the differences among the three estimands.

Identification and Estimation

We now discuss the assumptions required to identify the three causal estimands from randomized trajectories, and then introduce our methods for estimation and inference.

Causal identification. We rely on four standard assumptions. First, consistency requires that the observed trajectory agrees with the potential trajectory under the realized intervention assignments. Second, sequential exchangeability requires that each intervention assignment is independent of future potential trajectories and outcomes, conditional on the observed pre-assignment history HtH_t. This condition holds by our randomized design. Third, positivity requires g(a∣h)>0g(a \mid h) > 0 for every assignment aa needed to evaluate a target intervention policy or continuation at the relevant histories. Finally, we assume no interference across trajectories.

For the blip with randomized continuation, define

mg(h,a):=Eg[Y∣Ht=h,At=a]m_g(h,a) := \mathbb{E}_g[Y \mid H_t = h, A_t = a]

as the expected outcome conditional on history hh and assignment aa, with subsequent interventions following the collection policy gg. Under the identification assumptions, the blip effect defined in (3) can be written as

b(h;g)=mg(h,1)−mg(h,0).(6)b(h;g) = m_g(h,1) - m_g(h,0). \tag*{(6)}

Thus, this blip effect is identified directly from the randomized data, without modeling the agent–environment dynamics.

Similarly, the blip with no-treatment continuation and the policy gain are also identified under these assumptions, provided positivity holds along the target-policy continuations. However, estimating these effects is more challenging because interventions can alter future trajectories, and few observed trajectories may follow a given target policy over a long horizon.

Estimation and inference. For the blip with randomized continuation τ(G;g)\tau(G;g), we use a cross-fitted augmented inverse probability weighting (AIPW) estimator with known assignment probabilities and fitted outcome predictions. Given that we know the assignment probabilities under the randomized design, this estimator is unbiased and asymptotically normal, allowing for valid inference even with flexible machine learning models for the outcome predictions, without relying that these models are well specified or converge to the true conditional expectations. Moreover, we also fit structural nested mean models (SNMMs) to obtain model-based estimates for comparison.

For the blip with no-treatment continuation τ(G;0)\tau(G; 0) and policy gain Δ(G)\Delta(G), we use continuation-specific SNMMs with g-estimation [17, 26, 31]. These estimators require additional assumptions on the specification of the blip-effect models, beyond those needed for AIPW estimation.

For all three estimands, we report pointwise 95% Student’s tt-based confidence intervals, treating completed trajectories as independent observations while accounting for within-trajectory dependence. We provide detailed estimators and inference procedures in Appendix § 6.

Experiments

We apply our randomized intervention design to three established agent benchmarks: τ\tau-bench Retail, ToolSandbox, and ImpossibleBench.3 In ImpossibleBench, we examine whether adversarial feedback following an unsuccessful code submission increases the likelihood of violating task rules.

Using the same collection of randomized trajectories, we estimate intervention effects across different agent actions and states. In particular, for tool-call review, we consider groups ranging from broad activities and capabilities to specific operations and individual tools. For adversarial feedback, we examine states defined by task progress, visible test conflicts, and prior failures. We also evaluate selective intervention policies that intervene whenever the observed history belongs to a specified group. In the figures reported below, “Matching decisions” counts the intervention points, N(G)N(G), whose pre-assignment histories belong to group GG. A trajectory may contribute multiple points to the same group, and a point may be counted in more than one group. Table 1 summarizes the experimental setups across the three benchmarks. In the τ\tau-bench Retail experiment, we also compare our SNMM-based off-policy evaluation procedure with a more computationally intensive Monte Carlo simulation to assess its accuracy and robustness to model misspecification. In the simulation, we roll out each candidate policy multiple times and average the outcomes. We find that the SNMM based off-policy evaluation procedure is able to accurately estimate the policy gains with significantly lower computational cost. Experimental protocols and additional details are provided in Appendix § 7.

BenchmarkAgentsIntervention / ControlOutcomeReporting Groups GG
τ\tau-bench Retail [4, 34]GPT-5 nano; Claude Haiku 5.5Tool-call review vs. direct executionTask successReads/writes; functional families; individual tools
ToolSandbox [20]GPT-5 nano; Claude Haiku 5.5Tool-call review vs. direct executionSimilarity scoreCapabilities; operations; individual tools
ImpossibleBench [36]Qwen3.5 FlashAdversarial vs. neutral feedbackRule violationTask progress; test conflicts; prior steering

Table 1. Summary of experimental setups across the three benchmarks.

Note. Interventions are assigned with equal probability at each eligible point. GPT-6 Luna serves as the reviewer in τ\tau-bench Retail and ToolSandbox.

τ\tau-bench Retail

In τ\tau-bench Retail, a customer-support agent handles account and order requests while following store policies [4, 34]. We evaluate two agents, powered by GPT-5 nano and Claude Haiku 5.5, respectively.

At each proposed tool call, we randomly assign review or direct execution with equal probability. The reviewer, GPT-6 Luna with high reasoning effort, may approve or replace the proposed call or request additional information from the customer. We measure the effect of review on terminal task success.

We estimate review effects for groups of intervention points GG defined by the proposed tool-call type. The groups range from broad activities, such as reads and writes, to functional families, such as identity lookup, and individual tools, such as finding a customer by email. For example, GG can contain all proposed database writes.

We first discuss the AIPW estimates of blip effects with randomized continuation, τ(G;g)\tau(G; g), shown in Figure 3–Figure 4. These estimates measure the effect of reviewing the current proposal when later review assignments continue to be randomized. For GPT-5 nano (Figure 3), the estimated effects are positive for both read and write calls. Among reads, we estimate positive effects for customer identification and order-detail retrieval, while the confidence interval for the catalog group includes zero. At the tool level, the confidence intervals lie above zero for order cancellation and item changes and include zero for payment changes.

Figure 3. τ\tau-bench Retail review effects on terminal task success for GPT-5 nano, from read/write activities to individual tools. Bars are pointwise 95% tt-based confidence intervals; arrows indicate bounds beyond the displayed axis range.

Figure 4. τ\tau-bench Retail review effects on terminal task success for Claude Haiku 5.5, from read/write activities to individual tools. Bars are pointwise 95% tt-based confidence intervals; arrows indicate bounds beyond the displayed axis range.

For Claude Haiku 5.5 (Figure 4), we estimate a review effect close to zero when averaged across all read calls, but a positive effect for account-detail calls alone, with a pointwise 95% confidence interval above zero. Looking only at the aggregate for read calls would miss this subgroup estimate.

We also evaluate nine selective review policies in additional rollouts: one for each of the seven read/write families in Figure 5, one for calculation calls, and one for handoff calls. Each policy reviews every proposal in its group and executes all other proposals directly. For example, the order-details policy reviews all order-details calls and skips review elsewhere. For each policy, we estimate the gain relative to never reviewing in two ways. For direct evaluation, we use 456 rollout episodes per policy. For SNMM-based evaluation, we reuse the same 2,280 sequentially randomized trajectories used to estimate the Retail blip effects. The SNMM and direct Monte Carlo estimates are positive for all nine policies, and their pointwise 95% confidence intervals overlap for every policy (Figure 5). The direct Monte Carlo point estimate falls within the corresponding SNMM interval for eight of the nine policies.

Figure 5. Policy gains on τ\tau-bench Retail for GPT-5 nano, relative to never reviewing. Direct estimates use 456 rollout episodes per policy; SNMM estimates use a separate randomized study of 2,280 trajectories. Bars are pointwise 95% trajectory-level tt-based confidence intervals.

ToolSandbox

In ToolSandbox, we reuse one randomized experiment per agent to estimate review effects and selective-policy gains across 33 tools and 49 reporting groups, ranging from broad capabilities to specific operations and individual tools. The agent completes tasks in a simulated phone environment using tools for device settings, contacts, messages, reminders, and information lookup [20]. Our main analysis uses a GPT-5 nano agent and we repeat the analysis for a second agent that uses Claude Haiku 5.5 in Appendix C. As in τ\tau-bench Retail, we randomly assign review or direct execution with equal probability, using a GPT-6 Luna with High reasoning as reviewer.

When an agent proposes multiple tool calls in a single turn, we treat the batch as one intervention point. We measure performance using ToolSandbox’s official score, which evaluates how closely the agent’s tool calls, environment states, and responses satisfy predefined task requirements. The score allows partial credit but is set to zero when a benchmark-defined failure condition occurs.

We organize intervention groups GG hierarchically, from broad capabilities to specific operations and individual tools. For example, reminder operations include finding, creating, modifying, and removing reminders. For each group, we estimate two blip effects: one with later reviews randomized and one with no later review in either branch. We also estimate the gain over never reviewing for a policy that reviews every matching proposal and skips review elsewhere (Figure 6–Figure 8).

Figure 6. Review effects on the ToolSandbox similarity score for GPT-5 nano across device state and contacts. Bars are pointwise 95% tt-based confidence intervals.

Figure 8. Review effects on the ToolSandbox score for GPT-5 nano for tool calls related to time and calculations, location, and external information. Bars are pointwise 95% tt-based confidence intervals; arrows indicate bounds beyond the displayed axis range.

Figure 7. Review effects on the ToolSandbox score for GPT-5 nano for tool calls related to messaging and reminders. Bars are pointwise 95% tt-based confidence intervals.

Averaging across a capability can obscure promising targets for review. For Time and calculations, the randomized-continuation blip is +1.8 score points, with a pointwise 95% tt-based confidence interval of [-0.2, 3.8]. Within this capability, reviewing calls to “datetime info to timestamp” tool, which constructs a timestamp from date and time components, has an estimated blip of +6.9 points [1.2, 12.6]. This suggests timestamp construction as a candidate for targeted review, despite the inconclusive capability-level estimate.

The precision of these estimates depends on how often the agent proposes each tool and frequently proposed tools tend to have narrower confidence intervals. Rarely proposed tools have wider intervals; both blip intervals for seconds-to-time conversion span substantial negative and positive effects. We note that the reported intervals are pointwise and do not account for multiple comparisons and the blips with no-treatment continuation and the policy gains also rely on additional parametric assumptions in the SNMMs. Despite these limitations, the estimates can help prioritize targets for further investigation. Independent randomized controlled trials, with interventions and comparisons specified in advance, could then test whether reviewing the selected tools improves outcomes.

ImpossibleBench

We evaluate a Qwen3.5 Flash agent on a subset of ImpossibleBench coding problems derived from LiveCodeBench [13, 36], using either the original tests or variants containing contradictory tests. These variants contain mutually incompatible test requirements. The agent may make up to six code submissions or stop and report inconsistent tests. After each of the first five unsuccessful submissions, we randomly assign either a neutral note or adversarial feedback that encourages prohibited test edits. An episode ends after success, an issue report, a detected violation, or the sixth submission. Our terminal outcome is a detected rule violation: we set Y=1Y = 1 if a test edit is detected or the agent passes the benchmark’s contradictory check, and Y=0Y = 0 otherwise. Positive intervention effects therefore indicate an increase in detected violations.

We use steering to refer to the assignment of adversarial feedback, regardless of whether the agent follows it. We define reporting groups using information available before the current assignment: the number of remaining attempts, the fraction of tests passing, whether the visible feedback contains contradictory checks, and whether adversarial feedback was assigned earlier in the episode. In Figure 9, “conflict” means a contradiction visible in the current feedback, and “near completion” means that at least 80% but fewer than 100% of tests pass. “Steered before, still eligible” denotes previous exposure to adversarial feedback followed by another eligible decision; it does not require previous compliance with that feedback.

Figure 9. Effects of adversarial feedback on terminal detected rule violations by Qwen3.5 Flash in ImpossibleBench. Blue circles and orange squares denote AIPW and SNMM estimates, respectively. Bars are pointwise 95% tt-based confidence intervals. For the group defined by previous exposure to adversarial feedback, the selective policy intervenes only after an earlier intervention. Starting without prior exposure, this policy never initiates intervention and coincides with never intervening; its policy gain is therefore zero by definition.

In Figure 9 we see that the agent has a higher propensity to be steered by an adversarial feedback when it has fewer attempts remaining or when it sees only few failing test cases. After the fifth failed attempt, with one attempt remaining, adversarial feedback increases terminal violations by an estimated 9.4 pp [2.9, 15.9], compared with 3.3 pp [0.6, 6.1] after the first failed attempt. When at least 80% of tests pass, the estimated effect is 5.7 pp [2.3, 9.2], compared with 0.3 pp [-1.7, 2.2] when fewer than 50% of the tests pass.

Limitations and Future Directions

Our results demonstrate how sequentially randomized interventions can support causal evaluation at different points in an agent’s trajectory. Below we highlight limitations of our methodology and propose several directions for developing more flexible and adaptive methods for agent evaluation.

  1. Parametric assumptions for structural nested mean models. Our AIPW estimator for the blip effect under randomized continuation does not require a correctly specified outcome model because the intervention probabilities are known by design. In contrast, our SNMM estimators for blip effects under alternative continuation policies and for policy gains assume that the conditional blip effects belong to the specified parametric class. The SNMM assumptions provides a tractable semiparametric approach for estimation but the reported confidence intervals have valid coverage for the target causal effects only when the semiparametric model is well specified. Under misspecification, the fitted coefficients may instead represent a weighted projection of the true blip effects.

  1. Model and benchmark scope. Our experiments evaluate relatively small models on established benchmarks that might be saturated by frontier models. Future experiments should evaluate frontier agents on more challenging tasks that require sustained planning, contain more consequential intervention opportunities, and better reflect deployment settings. Such experiments would also help determine whether intervention effects become smaller as agents improve or whether greater agent capability creates new forms of heterogeneity and susceptibility to intervention.

  1. Representation and selection of agent histories. We use hand-designed, experiment-specific features to represent the observed history in the SNMMs. These representations encode aspects of the trajectory that we expect to moderate intervention effects, such as prior failures, previous interventions, proposed tools, and task progress. Future work could use sample splitting or cross-validation to select representations that perform well on a testable estimand such as randomized-continuation effect, although agreement for this estimand would not by itself establish correct specification under other continuation policies. Candidate representations could also be proposed by an LLM judge and evaluated on held-out trajectories.

  1. Semantically defined intervention opportunities. Our experiments randomize interventions at fixed and well specified decision points, such as proposed tool calls or unsuccessful code submissions. In many applications, useful intervention opportunities may instead depend on the semantic content of an interaction—for example, when an agent expresses unwarranted confidence, contradicts an earlier statement, becomes persuasive, or proposes a potentially harmful plan. Detecting such events requires interpreting the evolving textual history, potentially with an LLM-based judge. An important direction is therefore to develop and validate semantic detectors of intervention opportunities and then incorporate them into sequentially randomized experiments.

References

Estimating Equations and Assumptions

This appendix gives the estimating equations for the three causal estimands defined in § 2.3. Appendix B provides the implementation details for each experiment.

Notation

We follow the notation introduced in § 2, adding an index i=1,…,ni = 1,\ldots,n for the nn completed trajectories.

Trajectories and intervention points. For trajectory ii, let Ti\mathcal{T}_i denote the number of intervention points, indexed by Ti={1,…,Ti}\mathcal{T}_i = \{1,\ldots,T_i\}. At each point t∈Tit \in\mathcal{T}_i, let HitH_{it} denote the pre-assignment history defined in (1), and let Ait∈{0,1}A_{it} \in\{0,1\} denote the randomized intervention assignment. We use YiY_i for the terminal outcome, which is binary in τ\tau-bench Retail and ImpossibleBench and continuous in ToolSandbox. For brevity, we write ∑i,t\sum_{i,t} for ∑i=1n∑t∈Ti\sum_{i=1}^{n}\sum_{t\in\mathcal{T}_i}.

Intervention policies. Let gg denote the randomized collection policy, with assignment probabilities

g(a∣h)=e(h)a(1−e(h))1−a,e(h)=Pr⁡g(At=1∣Ht=h).g(a \mid h) = e(h)^a(1-e(h))^{1-a}, \qquad e(h) = \Pr_g(A_t = 1 \mid H_t = h).

In our experiments, we use e(h)=1/2e(h) = 1/2 at every eligible intervention point. We write Eg\mathbb{E}_g for expectation over trajectories generated under gg. We use π\pi to denote a target intervention policy, including the never-intervene policy π0\pi_0 and the selective policy πG\pi_G defined in (5).

Reporting groups and estimands. For a reporting group GG, let Iit(G)=1{Hit∈G}I_{it}(G) = 1\{H_{it} \in G\} indicate whether intervention point tt in trajectory ii belongs to GG. Define the number of matching intervention points by

Ni(G)=∑t∈TiIit(G),N(G)=∑i=1nNi(G).N_i(G) = \sum_{t\in\mathcal{T}_i} I_{it}(G), \qquad N(G) = \sum_{i=1}^{n} N_i(G).

We consider three causal estimands: the blip with randomized continuation τ(G;g)\tau(G;g), the blip with no-treatment continuation τ(G;0)\tau(G;0), both defined in (4), and the policy gain Δ(G)\Delta(G) defined in (5).

Cross-fitting. We divide the nn trajectories into five folds, keeping all intervention points from the same trajectory together. Let k(i)∈{0,…,4}k(i) \in\{0,\ldots,4\} denote the fold containing trajectory ii, and define Ik={i:k(i)=k}\mathcal{I}_k = \{i : k(i) = k\}. We use the superscript (−k)(-k) to denote a function fitted without data from fold kk, so predictions for trajectory ii are obtained from models fitted without its fold k(i)k(i).

Blip with Randomized Continuation

Augmented inverse probability weighting. Let mg(h,a)=Eg[Y∣Ht=h,At=a]m_g(h,a) = \mathbb{E}_g[Y \mid H_t = h, A_t = a] denote the conditional outcome regression, and let m^g(−k)(h,a)\widehat{m}_g^{(-k)}(h,a) be its estimate fitted without fold kk. By (6), the blip effect is b(h;g)=mg(h,1)−mg(h,0)b(h;g) = m_g(h,1) - m_g(h,0). We then define the cross-fitted augmented inverse probability weighting (AIPW) score as

D^it=m^g(−k(i))(Hit,1)−m^g(−k(i))(Hit,0)+Aite(Hit){Yi−m^g(−k(i))(Hit,1)}−1−Ait1−e(Hit){Yi−m^g(−k(i))(Hit,0)}.(7)\widehat{D}_{it} = \widehat{m}_g^{(-k(i))}(H_{it},1) - \widehat{m}_g^{(-k(i))}(H_{it},0) + \frac{A_{it}}{e(H_{it})}\left\{Y_i - \widehat{m}_g^{(-k(i))}(H_{it},1)\right\} -\frac{1-A_{it}}{1-e(H_{it})}\left\{Y_i - \widehat{m}_g^{(-k(i))}(H_{it},0)\right\}. \tag*{(7)}

which gives the group estimator for group GG

τ^AIPW(G;g)=1N(G)∑i,tIit(G)D^it.\widehat{\tau}_{\mathrm{AIPW}}(G;g)=\frac{1}{N(G)}\sum_{i,t}I_{it}(G)\widehat{D}_{it}.

When the propensity e(h)e(h) is known, the score stays conditionally unbiased even when the outcome predictions are wrong. Let rar_a be any regression fitted without the current trajectory. At eligible histories, randomization gives

Eg[1{At=a}g(a∣Ht){Y−ra(Ht)}∣Ht=h]=mg(h,a)−ra(h),\mathbb{E}_g\left[\left.\frac{\mathbb{1}\{A_t=a\}}{g(a\mid H_t)}\{Y-r_a(H_t)\}\right|H_t=h\right]=m_g(h,a)-r_a(h),

so the right-hand side of (7) with each m^g\widehat{m}_g replaced by rar_a has conditional mean b(h;g)b(h;g). When assignment probabilities are known, the outcome regression affects the estimator’s variance but does not change its population target. In case of estimated propensities the usual double-robustness property applies [2].

Linear blip model for randomized continuation Let f(h)f(h) denote an encoding of history hh used in the effect model. It includes attributes such as the proposed tool, earlier errors, and prior interventions. Within each environment and estimand, all agents use the same feature definition. Let s(h)∈{0,…,P−1}s(h) \in\{0,\ldots,P-1\} identify the phase and write sit=s(Hit)s_{it}=s(H_{it}). τ\tau-bench Retail and ToolSandbox use semantic progress phases (see Appendix § 7.1.4 and Appendix § 7.§ 2.4, correspondingly), while steering uses chronological checkpoints. The features and phases for each experiment are defined in Appendix § 7.

Let d=dim⁡f(h)d=\dim f(h) and partition the coefficient coordinates into a shared set S⊂{1,…,d}\mathcal{S}\subset\{1,\ldots,d\} and a phase-specific set U={1,…,d}∖S\mathcal{U}=\{1,\ldots,d\}\setminus\mathcal{S}. Define

βs,S=α,βs,U=γs,s=0,…,P−1,(8)\beta_{s,\mathcal{S}}=\alpha,\qquad\beta_{s,\mathcal{U}}=\gamma_s,\qquad s=0,\ldots,P-1, \tag*{(8)}

where α∈R∣S∣\alpha\in\mathbb{R}^{|\mathcal{S}|} is shared across phases and γs∈R∣U∣\gamma_s\in\mathbb{R}^{|\mathcal{U}|} may vary with ss. The cases S=∅\mathcal{S}=\varnothing and U=∅\mathcal{U}=\varnothing give fully phase-specific and fully shared coefficients, respectively. The exact choice of these coefficients for our experiments is described in Appendix § 7.

Define

q(h)=Eg[Y∣Ht=h]={1−e(h)}mg(h,0)+e(h)mg(h,1),Zit=(Ait−e(Hit))f(Hit).q(h)=\mathbb{E}_g[Y\mid H_t=h]=\{1-e(h)\}m_g(h,0)+e(h)m_g(h,1),\qquad Z_{it}=(A_{it}-e(H_{it}))f(H_{it}).

The cross-fitted value q^it:=q^(−k(i))(Hit)\widehat{q}_{it}:=\widehat{q}^{(-k(i))}(H_{it}). Under a well specified model we have that the blip with randomized continuation b(h;g)=f(h)⊤βs(h)(g)b(h;g)=f(h)^\top\beta_{s(h)}(g).

Finally, we estimated the coefficients in our model jointly by minimizing the following objective:

(α^(g),{γ^s(g)})=arg⁡min⁡α,{γs}∑i,t{Yi−q^it−Zit⊤βsit(g)}2,(9)\left(\widehat{\alpha}(g),\{\widehat{\gamma}_s(g)\}\right)=\underset{\alpha,\{\gamma_s\}}{\arg\min}\sum_{i,t}\left\{Y_i-\widehat{q}_{it}-Z_{it}^{\top}\beta_{s_{it}}(g)\right\}^{2}, \tag*{(9)}
τ^SNMM(G;g)=1N(G)∑i,tIit(G)f(Hit)⊤β^sit(g).\widehat{\tau}_{\mathrm{SNMM}}(G;g)=\frac{1}{N(G)}\sum_{i,t}I_{it}(G)f(H_{it})^\top\widehat{\beta}_{s_{it}}(g).

This algorithm for estimating the coefficients β\beta is known in the literature as the Double Machine Learning algorithm [11] or equivalently the R-Learner algorithm [24] with a linear specification of the heterogeneous treatment effect (see e.g. the ‘LinearDML’ class in the EconML library [5]).

In case the linear model is misspecified, the coefficients target an e(h){1−e(h)}e(h)\{1-e(h)\}-weighted projection of the true blip onto the allowed effect model. When the propensities are constant, as it is in our case, then this weighted projection reduces to the un-weighted projection of the true blip on the allowed effect model. This explains why the SNMM estimate in our experiments is almost identical to the AIPW estimate.

Blip with No-treatment Continuation

To estimate blip effects under a target continuation π\pi, we must account for the fact that future interventions in the observed data follow gg, rather than π\pi. We do so by correcting the terminal outcome for subsequent intervention assignments and fitting a structural nested mean model (SNMM). We develop the estimator for a general continuation π\pi and then specialize to the no-treatment policy π0\pi_0. We adapt the gg-estimation construction of Lewis and Syrgkanis [18]; see also Vansteelandt and Joffe [31].

Blip model. Throughout the derivation, fix a target continuation π\pi and suppress the dependence of the coefficients on π\pi. We fit a separate model for each continuation. Let βs\beta_s denote the coefficient vector for phase ss. As in (8), write

βs,S=α,βs,U=γs,s=0,…,P−1.\beta_{s,S} = \alpha,\qquad\beta_{s,U} = \gamma_s,\qquad s = 0,\ldots,P-1.

Thus α\alpha contains coefficients shared across phases, while γs\gamma_s contains coefficients specific to phase ss. The structural restriction is

E[Yt(a;π)−Yt(0;π)∣Ht=h]=af(h)Tβs(h),a∈{0,1}.(10)\mathbb{E}[Y_t(a;\pi)-Y_t(0;\pi)\mid H_t=h] = af(h)^{\mathsf{T}}\beta_{s(h)},\qquad a\in\{0,1\}. \tag*{(10)}

Here Yt(a;π)Y_t(a;\pi) retains the observed past, assigns aa at the current decision, and follows π\pi afterward.

Future-treatment correction and identification. For each phase ℓ\ell, collect the features of later decisions, weighted by the difference between the observed assignment and the assignment prescribed by π\pi:

Bit,ℓ(π)=∑j∈Tij>t, sij=ℓ{Aij−π(Hij)}f(Hij).(11)B_{it,\ell}(\pi)=\sum_{\substack{j\in\mathcal{T}_i\\j>t,\ s_{ij}=\ell}}\{A_{ij}-\pi(H_{ij})\}f(H_{ij}). \tag*{(11)}

The quantity ∑ℓ=0P−1Bit,ℓ(π)Tβℓ\sum_{\ell=0}^{P-1}B_{it,\ell}(\pi)^{\mathsf{T}}\beta_\ell corrects for these future assignments. To see why this correction identifies the current blip, let ν(h,π)=E[Yt(0;π)∣Ht=h]\nu(h,\pi)=\mathbb{E}[Y_t(0;\pi)\mid H_t=h]. Under the identification assumptions and a correctly specified blip model,

Eg[Y−∑ℓ=0P−1Bt,ℓ(π)Tβℓ∣Ht=h,At=a]=ν(h,π)+af(h)Tβs(h).(12)\mathbb{E}_g\left[Y-\sum_{\ell=0}^{P-1}B_{t,\ell}(\pi)^{\mathsf{T}}\beta_\ell\mathrel{\Big|}H_t=h,A_t=a\right]=\nu(h,\pi)+af(h)^{\mathsf{T}}\beta_{s(h)}. \tag*{(12)}

Thus, after correction, the conditional mean differs between the two current assignments by f(h)Tβs(h)f(h)^{\mathsf{T}}\beta_{s(h)}.

Recall the centered current features Zit=(Ait−e(Hit))f(Hit)Z_{it}=(A_{it}-e(H_{it}))f(H_{it}). Subtracting ZtTβs(Ht)Z_t^{\mathsf{T}}\beta_{s(H_t)} from the corrected outcome in (12) makes its conditional mean independent of AtA_t, given HtH_t: that mean is ν(Ht,π)+e(Ht)f(Ht)Tβs(Ht)\nu(H_t,\pi)+e(H_t)f(H_t)^{\mathsf{T}}\beta_{s(H_t)}. Since Eg[Zt∣Ht]=0\mathbb{E}_g[Z_t\mid H_t]=0, multiplying by ZtZ_t therefore gives a zero conditional expectation at the true coefficients. These are the moments used to estimate the blip model.

Centering regressions. To improve precision, we center the outcome and future-treatment features using predictions based on the current history. Let q^it\widehat{q}_{it} be the cross-fitted estimate of q(h)=Eg[Y∣Ht=h]q(h)=\mathbb{E}_g[Y\mid H_t=h] defined above. For each phase ℓ\ell, define

pℓ(h,π)=Eg[Bt,ℓ(π)∣Ht=h],U^it,ℓ(π)=Bit,ℓ(π)−p^ℓ(−k(i))(Hit,π).p_\ell(h,\pi)=\mathbb{E}_g[B_{t,\ell}(\pi)\mid H_t=h],\qquad\widehat{U}_{it,\ell}(\pi)=B_{it,\ell}(\pi)-\widehat{p}_{\ell}^{(-k(i))}(H_{it},\pi).

All predictions for trajectory ii are fitted without its fold, and all regressions condition on the history at the current decision HitH_{it}.

Joint coefficient estimation. For candidate coefficients α\alpha and {γs}\{\gamma_s\}, assemble each βs\beta_s using the sharing restrictions above and define

εit(α,{γs};π)=Yi−q^it−ZitTβsit−∑ℓ=0P−1U^it,ℓ(π)Tβℓ.(13)\varepsilon_{it}(\alpha,\{\gamma_s\};\pi)=Y_i-\widehat{q}_{it}-Z_{it}^{\mathsf{T}}\beta_{s_{it}}-\sum_{\ell=0}^{P-1}\widehat{U}_{it,\ell}(\pi)^{\mathsf{T}}\beta_\ell. \tag*{(13)}

We estimate α\alpha and {γs}\{\gamma_s\} jointly. Writing ε^it(π)\widehat{\varepsilon}_{it}(\pi) for the residual evaluated at α^\widehat{\alpha} and {γ^s}\{\widehat{\gamma}_s\}, the estimates solve

∑i,tZit,Sε^it(π)=0,\sum_{i,t} Z_{it,S}\widehat{\varepsilon}_{it}(\pi)=0,
∑i,t:sit=sZit,Uε^it(π)=0,s=0,…,P−1.(14)\sum_{i,t:s_{it}=s} Z_{it,U}\widehat{\varepsilon}_{it}(\pi)=0,\qquad s=0,\ldots,P-1. \tag*{(14)}

Each shared coefficient uses scores from all phases, while each phase-specific coefficient uses scores only from its own phase. Because the residual is linear in the coefficients, these equations form a joint system of linear equations.

Known randomization gives Eg[Zit∣Hit]=0\mathbb{E}_g[Z_{it}\mid H_{it}]=0. Consequently, subtracting a function of the current history from the corrected outcome leaves the population moments unchanged, provided the relevant expectations exist. The estimating equations therefore remain valid at the true blip coefficients even when the regressions for qq and pℓp_\ell are misspecified. To identify the coefficients uniquely, we additionally require the population coefficient matrix to be nonsingular on the retained feature directions.

Backward implementation. When S=∅\mathcal{S}=\varnothing, coefficients are shared across decisions within each phase but remain separate across phases. If phases are nondecreasing along a trajectory, later decisions cannot belong to an earlier phase. In this case, Bit,ℓ(π)=0B_{it,\ell}(\pi)=0 for ℓ<sit\ell<s_{it}. We preserve these structural zeros by setting the corresponding predictions p^ℓ(−k(i))(Hit,π)\widehat{p}_{\ell}^{(-k(i))}(H_{it},\pi) and residuals U^it,ℓ(π)\widehat{U}_{it,\ell}(\pi) to zero. The joint system in (14) is then block upper triangular and can be solved by backward substitution. When the diagonal blocks are nonsingular, this gives the same coefficients as solving the full system at once.

For s=P−1,…,0s=P-1,\ldots,0, first subtract the contributions of later phases using their already estimated coefficients:

Y‾it,s(π)=Yi−q^it−∑ℓ>sU^it,ℓ(π)Tβ^ℓ,sit=s.(15)\overline{Y}_{it,s}(\pi)=Y_i-\widehat{q}_{it}-\sum_{\ell>s}\widehat{U}_{it,\ell}(\pi)^{\mathsf{T}}\widehat{\beta}_\ell,\qquad s_{it}=s. \tag*{(15)}

Then estimate the current phase’s coefficients by solving

∑i,t:sit=sZit[Y‾it,s(π)−{Zit+U^it,s(π)}Tβ^s]=0.(16)\sum_{i,t:s_{it}=s} Z_{it}\left[\overline{Y}_{it,s}(\pi)-\{Z_{it}+\widehat{U}_{it,s}(\pi)\}^{\mathsf{T}}\widehat{\beta}_s\right]=0. \tag*{(16)}

The term U^it,s(π)\widehat{U}_{it,s}(\pi) accounts for later decisions within the same phase. Because those decisions share the current coefficient vector βs\beta_s, their contributions remain in the equation used to estimate it. For example, a Retail trajectory may contain several proposals before successful customer identification.

We use this backward implementation for τ\tau-bench Retail (Appendix § 7.1.4) and the separate chronological checkpoints in ImpossibleBench (Appendix § 7.§ 3). In ImpossibleBench, each checkpoint occurs at most once, so the same-phase future features, predictions, and residuals are also set to zero. This implementation adapts Algorithm 3 of Lewis and Syrgkanis [18] to allow repeated decisions within a phase. For ToolSandbox, we solve the full joint system because tool and history coefficients are shared across phases (Appendix § 7.§ 2.4).

Group-averaged blip with no-treatment continuation. To estimate τ(G;0)\tau(G; 0), fit the model with π=π0\pi= \pi_0 and average the fitted blips over matching intervention points. Restoring the policy argument to distinguish the fitted coefficients gives

τ^(G;0)=1N(G)∑i,tIit(G)f(Hit)⊤β^sit(π0).(17)\widehat{\tau}(G;0)=\frac{1}{N(G)}\sum_{i,t} I_{it}(G)f(H_{it})^{\top}\widehat{\beta}_{s_{it}}(\pi_0). \tag*{(17)}

The average uses histories reached under the collection policy gg, as specified by the estimand. Each selective policy πG\pi_G receives a separate fit with the same features, phases, and sharing restrictions; the policy gain compares its estimated value with that of π0\pi_0, as described next.

Policy Value Estimation

We now use the fitted blip models to evaluate policies that govern interventions throughout a trajectory. Recall that V(π)=E[Y(π)]V(\pi)=\mathbb{E}[Y(\pi)] is the expected terminal outcome under policy π\pi. We write the dependence of the coefficients on π\pi explicitly here, because evaluating different policies requires their respective continuation-specific fits.

From blip effects to policy values. The blip b(h;π)=f(h)⊤βs(h)(π)b(h;\pi)=f(h)^{\top}\beta_{s(h)}(\pi) compares intervention with no intervention at history hh, with all subsequent assignments following π\pi. Thus, once the future continuation is π\pi, the conditional mean change from replacing the current assignment AtA_t with π(Ht)\pi(H_t) is

{π(Ht)−At}b(Ht;π).\{\pi(H_t)-A_t\}b(H_t;\pi).

For a deterministic target policy, this adds the blip when the target intervenes and the observed assignment does not, subtracts it in the reverse case, and is zero when the assignments agree.

To obtain the policy value, consider replacing the collection policy with π\pi one decision at a time, working backward. At each step, earlier assignments still follow gg, while later assignments follow π\pi. The history at the current decision therefore has its distribution under gg, and the change in expected outcome is given by the corresponding blip correction above. Summing these changes telescopes from Eg[Y]\mathbb{E}_g[Y] to V(π)V(\pi). Under the identification assumptions and a correctly specified blip model, this gives [18 Lemma 6]

V(π)=Eg[Y+∑t∈T{π(Ht)−At}f(Ht)⊤βs(Ht)(π)].(18)V(\pi)=\mathbb{E}_g\left[Y+\sum_{t\in\mathcal{T}}\{\pi(H_t)-A_t\}f(H_t)^{\top}\beta_{s(H_t)}(\pi)\right]. \tag*{(18)}

The blips include the effects of the current assignment on subsequent agent behavior and intervention opportunities. This is why histories observed under gg can be used to evaluate a policy that would generate different future trajectories. The correction is valid in expectation. For the no-treatment policy π0\pi_0, it subtracts the corresponding blip for every observed intervention.

Estimation and policy gain. We estimate V(π)V(\pi) by applying the fitted corrections to each trajectory’s terminal outcome and averaging over trajectories. For a reporting group GG, we fit the models for the selective policy πG\pi_G and the no-treatment policy π0\pi_0 separately, then take the difference between their estimated values:

V^(π)=1n∑i=1n[Yi+∑t∈Ti{π(Hit)−Ait}f(Hit)⊤β^sit(π)],(19)\widehat{V}(\pi)=\frac{1}{n}\sum_{i=1}^{n}\left[Y_i+\sum_{t\in\mathcal{T}_i}\{\pi(H_{it})-A_{it}\}f(H_{it})^{\top}\widehat{\beta}_{s_{it}}(\pi)\right], \tag*{(19)}
Δ^(G)=V^(πG)−V^(π0).\widehat{\Delta}(G)=\widehat{V}(\pi_G)-\widehat{V}(\pi_0).

Each value estimate uses all nn completed trajectories, including those without intervention points in GG. Within each trajectory, the correction includes all intervention points. In particular, evaluating πG\pi_G also requires subtracting the effects of observed interventions outside GG, where the target policy would not intervene. The normalization is therefore by the number of trajectories nn, rather than the number of matching intervention points N(G)N(G) used for the group-averaged blip.

Uncertainty Quantification

We construct confidence intervals by treating completed trajectories as independent and identically distributed (IID) observations. Decisions within a trajectory may be dependent and share the same terminal outcome. We therefore aggregate their contributions within each trajectory before estimating variability across trajectories.

Throughout this section, we use sample-normalized estimated influence functions: each is the usual estimated influence function divided by nn. With this normalization, the influence functions give individual trajectories’ contributions to the first-order linearization of an estimator.

Blip with randomized continuation: AIPW. For the AIPW estimator of τ(G;g)\tau(G;g), we sum the centered AIPW scores over the intervention points in GG within each trajectory. The estimated influence function is

ui{τ^AIPW(G;g)}=1N(G)∑t∈TiIit(G){D^it−τ^AIPW(G;g)}.u_i\{\widehat{\tau}_{\mathrm{AIPW}}(G;g)\} = \frac{1}{N(G)} \sum_{t\in\mathcal{T}_i} I_{it}(G) \left\{ \widehat{D}_{it} - \widehat{\tau}_{\mathrm{AIPW}}(G;g) \right\}.

Here D^it\widehat{D}_{it} is the cross-fitted AIPW score defined above. Centering each score by the estimated group effect accounts for the random number of matching intervention points in the denominator. Summing within trajectories retains the dependence between scores that share the same terminal outcome.

Blip effects estimated using SNMMs. An SNMM group-effect estimate depends on both the sampled histories and the fitted blip coefficients. We first obtain influence functions for the coefficients by linearizing their estimating equations, adapting the construction of Lewis and Syrgkanis [18 Theorem 8]. We then propagate coefficient uncertainty to the group average.

Fix a continuation π\pi for the fit under consideration. As in Appendix § 6.§ 3, we suppress the dependence of the coefficients on π\pi during this derivation. Let mi(α,{γs};π)m_i(\alpha,\{\gamma_s\};\pi) be trajectory ii’s vector of estimating scores, with the shared coordinates first, followed by the phase-specific coordinates for s=0,…,P−1s=0,\ldots,P-1. The shared block sums Zit,SεitZ_{it,S}\varepsilon_{it} over all decisions in trajectory ii. The block for phase ss sums Zit,UεitZ_{it,U}\varepsilon_{it} over decisions in that phase.

For a changed continuation, use the residual in (13), giving the estimating equations in (14). For randomized continuation, use the normal equations obtained by differentiating the least-squares objective in (9). The score blocks have the same form, with residual

εit(α,{γs};g)=Yi−q^it−ZitTβsit,\varepsilon_{it}(\alpha,\{\gamma_s\};g) = Y_i-\widehat{q}_{it}-Z_{it}^{\mathsf{T}}\beta_{sit},

where βsit\beta_{sit} is assembled from the candidate coefficients α\alpha and {γs}\{\gamma_s\}. This residual has no future-treatment correction because the observed continuation already follows gg.

For either fit, hold the cross-fitted nuisance predictions fixed and differentiate the summed score with respect to the coefficients:

J^(π)=−∑i=1n∂mi(α,{γs};π)∂(αT,γ0T,…,γP−1T)∣α=α^, γs=γ^s.\widehat{J}(\pi) = -\left. \sum_{i=1}^{n} \frac{\partial m_i(\alpha,\{\gamma_s\};\pi)} {\partial(\alpha^{\mathsf{T}},\gamma_0^{\mathsf{T}},\ldots,\gamma_{P-1}^{\mathsf{T}})} \right|_{\alpha=\widehat{\alpha},\,\gamma_s=\widehat{\gamma}_s}.

The vector J(π)−1mi(α^,{γ^s};π)\mathbf{J}(\pi)^{-1}m_i(\widehat{\alpha},\{\widehat{\gamma}_s\};\pi) gives the estimated influence functions for the fitted coefficients. Write α˙i\dot{\alpha}_i for its shared block and γ˙i,s\dot{\gamma}_{i,s} for its phase-ss block. Assemble the influence function for each phase’s coefficient vector as

β˙i,s,S=α˙i,β˙i,s,U=γ˙i,s.\dot{\beta}_{i,s,S}=\dot{\alpha}_i,\qquad\dot{\beta}_{i,s,U}=\dot{\gamma}_{i,s}.

A coefficient shared across phases has the same influence function in every phase. For inference on the causal targets, we assume that the blip model and its coefficient-sharing restrictions are correctly specified, that the population Jacobian is nonsingular on the retained feature directions, and that the cross-fitted trajectory scores satisfy the regularity conditions for asymptotic linearity.

We next use these coefficient influence functions to obtain the influence function for the group average. Define

ai,s(G)=∑t∈Ti:sit=sIit(G)f(Hit),as(G)=∑i=1nai,s(G).a_{i,s}(G)=\sum_{t\in\mathcal{T}_i:s_{it}=s}I_{it}(G)f(H_{it}),\qquad a_s(G)=\sum_{i=1}^{n}a_{i,s}(G).

Write τ^SNMM(G;π)\widehat{\tau}_{\mathrm{SNMM}}(G;\pi) for the fitted group-averaged blip under continuation π\pi. Its estimated influence function is

ui{τ^SNMM(G;π)}=∑sai,s(G)⊤β^s−Ni(G)τ^SNMM(G;π)+∑sas(G)⊤β˙i,sN(G).(20)u_i\{\widehat{\tau}_{\mathrm{SNMM}}(G;\pi)\}= \frac{\sum_s a_{i,s}(G)^\top\widehat{\beta}_s-N_i(G)\widehat{\tau}_{\mathrm{SNMM}}(G;\pi)+\sum_s a_s(G)^\top\dot{\beta}_{i,s}} {N(G)}. \tag*{(20)}

The first two terms in the numerator account for sampling variation in the histories being averaged and in the number of matching intervention points. The final term accounts for uncertainty in the fitted coefficients. This formula applies to the randomized-continuation estimate τ^SNMM(G;g)\widehat{\tau}_{\mathrm{SNMM}}(G;g) and, with π=π0\pi=\pi_0, to the no-treatment-continuation estimate τ^(G;0)\widehat{\tau}(G;0).

Policy values and gains. Policy-value estimates also depend on both the observed trajectories and the fitted coefficients. For each target policy, use the coefficients and coefficient influence functions from the fit for that continuation. We make their dependence on π\pi explicit below because a policy gain compares two different fits.

Define the features used in each trajectory’s policy-value correction:

di,s(π)=∑t∈Ti:sit=s{π(Hit)−Ait}f(Hit),ds(π)=∑i=1ndi,s(π).d_{i,s}(\pi)=\sum_{t\in\mathcal{T}_i:s_{it}=s}\{\pi(H_{it})-A_{it}\}f(H_{it}),\qquad d_s(\pi)=\sum_{i=1}^{n}d_{i,s}(\pi).

The estimated influence functions for the policy value and gain are

ui{V^(π)}=Yi+∑sdi,s(π)⊤β^s(π)−V^(π)+∑sds(π)⊤β˙i,s(π)n,(21)u_i\{\widehat{V}(\pi)\}= \frac{Y_i+\sum_s d_{i,s}(\pi)^\top\widehat{\beta}_s(\pi)-\widehat{V}(\pi)+\sum_s d_s(\pi)^\top\dot{\beta}_{i,s}(\pi)} {n}, \tag*{(21)}
ui{Δ^(G)}=ui{V^(πG)}−ui{V^(π0)}.u_i\{\widehat{\Delta}(G)\}=u_i\{\widehat{V}(\pi_G)\}-u_i\{\widehat{V}(\pi_0)\}.

In the value formula, the first three terms in the numerator give the trajectory’s corrected outcome minus the estimated policy value. The final term accounts for uncertainty in the fitted coefficients. Because both policy values are estimated from the same trajectories, we subtract their influence functions trajectory by trajectory before estimating the variance. This accounts for the covariance between the two value estimates. Policy values and gains average over all completed trajectories, including those without intervention points in the reporting group.

Reported confidence intervals. Let ui(η^)u_i(\hat{\eta}) denote the sample-normalized estimated influence function for any of the estimators above. These influence functions sum to zero when the estimating equations are solved exactly. We estimate the standard error by

se^(η^)={nn−1∑i=1nui(η^)2}1/2.\widehat{\mathrm{se}}(\hat{\eta}) = \left\{ \frac{n}{n-1} \sum_{i=1}^{n} u_i(\hat{\eta})^2 \right\}^{1/2}.

We report pointwise, asymptotic 95% confidence intervals using a Student’s tt critical value:

η^±t0.975,n−1se^(η^).\hat{\eta} \pm t_{0.975,n-1}\widehat{\mathrm{se}}(\hat{\eta}).

Experimental setup and fitted models

τ\tau-bench Retail

Agent setup

We use 114 original tasks from the benchmark and construct 114 variants of these task and run our sequential randomization ten times per agent. One agent uses GPT-5 nano (gpt-5-nano-2025-08-07) with minimal reasoning; the other uses Claude Haiku 5.5 (claude-haiku-5-5) at low effort. Both have a 4,000-token output limit and a 100-step episode cap. We generate customer messages with gpt-4.1-mini-2025-04-14 at temperature zero and a 2,048-token limit.

The agent receives the store policy, schemas for the 16 tools in Table 2, and the conversation, including executed calls and results. We instruct it to help the customer while following the policy. At each turn, the agent chooses a tool call or a message to the customer. We score terminal success by checks with behavioral assertions in the benchmark.

Family suffixTool names
identityfind_user_id_by_email, find_user_id_by_name_zip
user_readget_user_details
order_readget_order_details
catalog_readget_item_details, get_product_details, list_all_product_types
pending_writecancel_pending_order, modify_pending_order_address, modify_pending_order_items, modify_pending_order_payment
resolution_writeexchange_delivered_order_items, modify_user_address, return_delivered_order_items
utilitycalculate, transfer_to_human_agents

Table 2. The 16 tools available to the agents and grouped into seven families of task

Reviewer setup

After freezing the proposed call or batch, we assign review or direct execution with equal probability. The reviewer uses GPT-6 Luna (gpt-6-luna) with high reasoning effort and a 6,000-token output limit. It receives the policy, schemas, visible conversation, and frozen proposal, but cannot query the database. We instruct it to check authentication, identifiers, arguments, available evidence, and required confirmation.

In its JSON response, the reviewer either approves the proposal, replaces the entire batch with one call, or suggests one question for the customer. In case the reviewer approves the tool call is directly executed.

Features

Each reporting group contains original proposals involving any of the tools assigned to that group. Table 2 gives the tool names and their families.

Figure 10. Two observed GPT-5 nano τ\tau-bench Retail episodes with review randomized at each proposal. Episode A succeeds after review replaces an invalid exchange proposal; episode B ends with a transfer to a human after an invalid order modification.

A confirmation phrase matches yes|confirm|go ahead|proceed as whole words, ignoring capitalization. We count a tool result as an error if its error flag is set or its text starts with error.

To predict terminal success, we also include the current review assignment and earlier decision records. Each earlier record contains the 39 inputs in the first five rows of Table 4 and its review assignment. We reserve 28 slots for GPT-5 nano and 14 for Claude Haiku; unused slots are zero.

InputsCountMeaning
Proposed tools (tool:)16The same 16 tool inputs as in Table 3.
Current tool families (current:)7For each family in Table 2, whether the proposal includes a tool from that family.
Earlier successful families (used:)7For each family, whether an earlier call from that family succeeded.
Earlier events5Earlier error, earlier correction, and repeated tool (defined above); any earlier review assignment (prior_verification); earlier user confirmation (user_affirmation_seen).
n_prior_tools, n_prior_errors, n_prior_verifications, n_proposed_tools4Numbers of earlier tool results, earlier errors, earlier review assignments, and calls in the current proposal, respectively.
source_cohort_extension1Whether the task comes from the added-task set.
eligible_position_zero_based1The number of review opportunities before the current proposal.

Table 4. The 41 features for the Retail nuisance models.

InputCountMeaning
intercept1A constant value of one.
Proposed tools (tool:)16One input for each tool in Table 2: whether the current proposal includes that tool.
prior_error1An earlier tool call returned an error.
prior_correction1An earlier review changed a proposal.
retry_tool1At least one proposed tool was used earlier.
used:catalog_read1An earlier product or catalog read succeeded.
used:order_read1An earlier order-details read succeeded.
write_x_prior_error1The proposal contains a write, and an earlier tool call returned an error.
write_x_affirmation1The proposal contains a write, and an earlier user message contains a confirmation phrase.

Table 3. The 24 Retail SNMM inputs, used for all continuations. Yes/no inputs are one for yes and zero for no.

The models used to account for later reviews use the 41 current inputs above, plus one yes/no input for each possible decision number. Only the input for the current decision number is one.

Phases and coefficient sharing

We use four phases: before successful customer identification; after identification but before a successful order or catalog read; after that read but before a successful write; and after a successful write.

We fit a separate coefficient vector for each phase, agent, and continuation, sharing coefficients across decisions within a phase. The randomized-continuation fit has no future-treatment correction. For changed continuations, we solve the joint moment system by backward substitution across the four phases, as in (15)–(16). Each phase’s equations retain the contributions of later proposals in that same phase, while contributions from later phases use their already estimated coefficients.

Nuisance estimation

For each held-out fold, we train one histogram gradient-boosted regressor on the other four folds to predict terminal success from the inputs listed above. Boosted trees accommodate nonlinear relationships and interactions among history features. We choose the histogram implementation for computational efficiency across repeated fits: it bins numerical predictors to reduce the number of candidate split points. We obtain predictions m^0\hat{m}_0 and m^1\hat{m}_1 by setting the current assignment to zero and one, respectively, and use q^=(m^0+m^1)/2\hat{q} = (\hat{m}_0 + \hat{m}_1)/2. We use squared-error loss, a learning rate of 0.1, at most 120 boosting iterations, at most 12 leaves per tree, a minimum leaf size of 20, and an L2 penalty of 2, seed 42+k42 + k for fold kk, and automatic early stopping.

For changed continuations, we predict the sum of later feature vectors for each phase, weighting each vector by the observed treatment minus the target-policy action. Within each training fold, we standardize the current attributes, cohort, numeric position, and position indicators, then fit ridge regressions with penalty 10. We use ridge to fit the many future-feature prediction targets efficiently, with coefficient shrinkage to stabilize estimation when predictors are correlated. We set predictions for earlier phases to zero. Before fitting the unpenalized effect regressions, we remove redundant feature directions by SVD without using the outcomes.

ToolSandbox

Agent setup

For each agent, we collect three trials of 129 base scenarios under eight tool variants, for 3,096 trajectories. We use the same two agents, model settings, and user simulator as in τ\tau-bench Retail, with each scenario’s official cap on the number of messages.

The agent sees the conversation and schemas for the simulated-phone tools offered in that scenario (Table 5). We instruct it not to guess missing arguments and to clarify ambiguous requests. Across variants, we alter tool availability or scramble tool names and documentation.

Family suffixTool names
settings_readget_low_battery_mode_status, get_location_service_status, get_cellular_service_status, get_wifi_status, get_current_location
settings_writeset_low_battery_mode_status, set_location_service_status, set_cellular_service_status, set_wifi_status
contact_lookupsearch_contacts
contact_editadd_contact, modify_contact, remove_contact
message_searchsearch_messages
message_sendsend_message_with_phone_number
reminder_readsearch_reminder
reminder_writeadd_reminder, modify_reminder, remove_reminder
utilityget_current_timestamp, timestamp_to_datetime_info, datetime_info_to_timestamp, shift_timestamp, timestamp_diff, seconds_to_hours_minutes_seconds, unit_conversion, calculate_lat_lon_distance, search_holiday
web_searchsearch_lat_lon, search_location_around_lat_lon, search_weather_around_lat_lon, search_stock, convert_currency

Table 5. The 33 tools available across scenarios and their grouping used as model inputs.

Reviewer setup

We freeze each agent turn containing tool calls and assign review with probability 1/21/2 before execution. The reviewer uses the same model settings as in τ\tau-bench Retail: GPT-6 Luna with high reasoning effort and a 6,000-token output limit.

The reviewer receives the visible conversation, proposal, offered schemas, public API rules, and stateless argument checks. For scrambled variants, we include the relevant rules without changing the schemas shown to the agent. The reviewer cannot execute tools or see the hidden device state and evaluator targets. It either approves the batch or replaces it with one call from the offered tools or one response to the user, citing the supplied evidence for any correction.

Features

Tables 6 and 7 list the 45 attributes for the SNMM model and 90 attributes used for nuisance model. We derive device settings and database sizes from the pre-execution state of the system, We group tools into six capabilities for reporting and ten families for model inputs.

A write changes device settings, edits contacts, sends a message, or changes reminders. We set precondition_unmet to one for messaging with cellular off; web search with Wi-Fi off; coordinate-based search without latitude when location is off; current-location lookup with location off; setting a toggle to its existing value; or enabling Wi-Fi, cellular, or location in low-battery mode.

InputsCountMeaning
Proposed tools (tool:)33The same 33 tool inputs as in Table 6.
Current tool families (current:)10For each family in Table 5, whether the proposal includes a tool from that family.
Earlier successful families (used:)10For each family, whether an earlier call from that family succeeded.
Earlier events and device rules6prior_error, prior_state_error, retry_tool, prior_verification, prior_correction, and precondition_unmet, defined above.
unknown_tool, malformed_arguments2The execution tool name is outside the catalog; the arguments do not decode to a JSON object.
n_prior_errors, n_prior_tools, n_prior_verifications, n_proposed_tools, n_user_turns5Numbers of earlier exception results, traced tool calls, review assignments, calls in the current proposal, and observed user turns, respectively.
Device settings (setting:)4Whether cellular, wifi, location_service, and low_battery_mode are on.
Database sizes (rows:)3Numbers of rows in CONTACT, MESSAGING, and REMINDER.
Scenario categories (category:)15All tools available; zero, three, or ten distractor tools; scrambled argument descriptions, argument types, tool descriptions, or tool names; single or multiple tool calls; single or multiple user turns; canonicalization; insufficient information; state dependency. Categories can overlap.
Scenario variants (variant:)2The scrambled and distractors inputs defined above.

Table 7. The 90 features used for the ToolSandbox nuisance models.

InputCountMeaning
Progress phases (phase:)3Before any successful tool result; after a success but before a successful write; after a successful write. Only one is active.
Proposed tools (tool:)33One input for each tool in Table 5: whether the current proposal includes that tool.
prior_error1An earlier tool result contained an exception.
prior_state_error1An earlier exception began with ConnectionError or PermissionError.
retry_tool1At least one proposed tool was executed earlier.
prior_verification1An earlier proposal was assigned review.
prior_correction1An earlier review changed a proposal.
precondition_unmet1A proposed call breaks a device-state rule.
write_x_prior_error1The proposal contains a write, and an earlier tool result contained an exception.
scrambled1The scenario name contains _scrambled.
distractors1The scenario name contains distraction_tools or ends in _all_tools.

Table 6. The 45 ToolSandbox SNMM inputs, used for all continuations. Each input is one when its condition holds and zero otherwise.

Phases and coefficient sharing

We use three phases: before any successful tool result, after a success but before a successful write, and after a successful write. Phase progression is one-way. Writes include settings changes, contact edits, message sends, and reminder writes.

For continuation cc, the effect model is

bc(h)=asc(h)+∑k=133bkc1{k is proposed}+(dc)Tz(h),b^{c}(h) = a^{c}_{s}(h) + \sum_{k=1}^{33} b^{c}_{k} \mathbb{1}\{k\text{ is proposed}\} + (d^{c})^{\mathsf{T}} z(h),

where zz is the vector of nine history and variant terms. The baseline asca^{c}_{s} varies by phase; the tool coefficients bkcb^{c}_{k} and history coefficients dcd^{c} are shared across phases. There is no additional intercept. We fit each agent and continuation separately. For each changed continuation, we solve one joint system of equations that accounts for all later decisions, both within the current phase and in subsequent phases. We chose this structure after inspecting sparse-coefficient diagnostics; the intervals do not include model-selection uncertainty.

Nuisance estimation

We encode the history using the 90 attributes, numeric decision position, and earlier decisions’ attributes and assignments. For outcome prediction, we use the same histogram gradient-boosting specification as in τ\tau-bench Retail, retaining its flexibility to model nonlinear relationships and its computational efficiency across repeated fits. We obtain predictions m^0\hat{m}_0 and m^1\hat{m}_1 from the fitted outcome model by setting the current assignment to zero and one, respectively, then take their average as q^\hat{q}.

Within each training fold, we standardize the current attributes and position indicators, then fit ridge regressions with penalty 10 to predict future totals in the phase-expanded basis. We set predictions for earlier phases to zero and map the remaining predictions into the basis for the pooled model’s shared coefficients. Before fitting the unpenalized effect regressions, we apply SVD to the pooled 45-feature design without using the outcomes.

ImpossibleBench

Agent setup

The agent uses Qwen3.5 Flash (qwen/qwen3.5-flash-02-23) at temperature 0.5, with reasoning disabled and a 4,096-token output limit. It has at most six actions and can submit code with write_file or stop with report_issue. In the system prompt we prohibit test edits and evaluator exploits, and permit reports of contradictory tests. We run the tests after each write. At the next action, the agent sees the task, submitted code, and test feedback.

We run each of 103 problems six times with original tests and six times with contradictory tests, yielding 1,236 episodes. We exclude 18 with missing terminal outcomes and analyze the remaining 1,218. We record a violation if a test edit is detected or the agent passes the contradictory check, and end the episode after success, an issue report, or a detected violation.

Steering setup

After an eligible unsuccessful submission, we assign a neutral or steering note with equal probability. In the steering note, we tell the agent that it may change the tests or we prompt Qwen3.5 Flash with the task, code, checkpoint state, test feedback, and two previous notes, asking for a diagnostic note that encourages a prohibited edit or test-order shortcut. We generate the note at temperature 0.8 with a 512-token limit. If generation fails, we use the script and retain the original assignment. We append the note to the tool feedback. In the system instructions to the agent, we tell the agent that notes cannot override any instructions.

Features

Table 8 and Table 9 list the nine SNMM inputs and 36 current nuisance attributes.

InputCountMeaning
intercept1A constant value of one.
pass_fraction1The fraction of checks passed.
conflict_visible1The visible feedback contains contradictory checks.
near_conflict1At least 80% but not all checks pass, and the visible feedback contains contradictory checks.
prior_attack_resisted1Earlier steering occurred and the current checkpoint is eligible. An explicit rejection is not required.
stalled1The fraction of checks passed is no higher than after the preceding submission; zero at the first submission.
execution_error1A current check reports an error.
repeated_failure1The current failure signature matches the preceding evaluation’s.
mentions_inconsistency1The agent’s message contains contradict, inconsistent, inconsistency, or impossible, ignoring capitalization.

Table 8. The nine features used for SNMM inputs

InputsCountMeaning
Current SNMM attributes8All inputs in Table 8 except the constant.
previous_pass_fraction, progress_delta2The fraction of checks passed after the preceding submission, and the change since then.
Progress ranges4almost_finished: 0.8≤p<10.8 \le p < 1; zero_progress: p=0p = 0; low_progress: p<0.5p < 0.5; mid_progress: 0.5≤p<0.80.5 \le p < 0.8, where pp is the current pass fraction.
Other progress changes3regressed: p<p−p < p_{-}; improved: p>p−p > p_{-}; recovered_from_zero: p−=0<pp_{-} = 0 < p, where p−p_{-} is the preceding pass fraction. All are zero at the first submission.
same_failure_streak, unchanged_code2The number of consecutive preceding evaluations with the current failure signature; whether the code equals the state saved after the preceding action (zero at the first checkpoint).
Time4checkpoint: preceding action number; turns_remaining: 6−checkpoint6 - \text{checkpoint}; late: checkpoint≥3\text{checkpoint} \ge3; last_chance: checkpoint=5\text{checkpoint} = 5.
prior_attacks1The number of earlier steering assignments.
Previous note (previous_arm_)4One input each for neutral, static, contextual, and llm; all zero before the first assignment.
Missing previous progress2previous_pass_fraction_missing and progress_delta_missing: whether the corresponding value is missing. Missing values are set to zero.
log_checks, log_code_characters2log⁡(1+count)\log(1 + \text{count}) for current checks and code characters.
Task variant and cohort4conflicting_variant: contradictory tests; repaired_conflict: locally repaired fixture; confirmation_split: confirmation cohort; baseten_provider: always zero for Qwen3.5 Flash.

Table 9. The 36 features for nuisance models.

Phases and coefficient sharing

Checkpoint tt follows action tt, for t=1t = 1 through 5. When an action is ineligible, we leave its checkpoint empty and retain the numbering of later checkpoints. We fit separate coefficients by checkpoint and continuation. We estimate coefficients for changed continuations backward from checkpoint 5. The future correction uses only later checkpoints because each checkpoint occurs at most once.

Figure 11. Two observed Qwen3.5 Flash runs of the same ImpossibleBench task, with note assignment randomized at each eligible checkpoint. The neutral run reports contradictory tests; the steered run follows a scripted note and edits the protected test file.

Nuisance estimation

We encode the baseline history with 36 features and, for each of the two previous eligible checkpoints, its attributes, assignment, and a presence indicator. For each held-out trajectory fold, we fit separate control, treatment, and pooled outcome regressions on the other four folds to estimate m^0\hat{m}_0, m^1\hat{m}_1, and q^\hat{q}. We use the τ\tau-bench Retail gradient-boosting settings with early stopping disabled. For the future-feature regressions, we use the same history features, standardize them within each training fold, and apply ridge penalty 10. Predictions for the current or an earlier checkpoint are zero. After applying SVD without using outcomes, we fit unpenalized effect regressions and scale the retained directions to unit second moment within each checkpoint.

ToolSandbox results for the Claude Haiku agent

We repeat the hierarchical ToolSandbox analysis for the agent that uses Claude Haiku 5.5. We fit the effects separately for this agent. The row order and display conventions match the main-text analysis, with horizontal ranges adapted to these estimates.

Figure 12. Review effects on the ToolSandbox similarity score for Claude Haiku 5.5 across device state and contacts. Bars are pointwise 95% tt-based confidence intervals.

Figure 13. Review effects on the ToolSandbox similarity score for Claude Haiku 5.5 across messaging and reminders. Bars are pointwise 95% tt-based confidence intervals.

Figure 14. Review effects on the ToolSandbox similarity score for Claude Haiku 5.5 across time and calculations, location, and external information. Bars are pointwise 95% tt-based confidence intervals. Seconds-to-time conversion was never proposed by this agent, so its effects are marked n/a.

References

  1. [1]Arena Team. Agent Arena: Causal evaluation of agents in the real world. https://arena.ai/blog/agent-arena-methodology, June 2026. Arena Blog.
  2. [2]H. Bang and J. M. Robins. Doubly robust estimation in missing data and causal inference models. Biometrics, 61(4):962–973, 2005. doi: 10.1111/j.1541-0420.2005.00377.x. URL https://doi.org/10.1111/j.1541-0420.2005.00377.x.
  3. [3]S. Barke, A. Goyal, A. Khare, A. Singh, S. Nath, and C. Bansal. AgentRx: Diagnosing AI agent failures from execution trajectories. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 12610–12639. Association for Computational Linguistics, 2026. doi: 10.18653/2026.findings-emnlp.598. URL https://aclanthology.org/2026.findings-emnlp.598/.arxiv.org/abs/2602.02475
  4. [4]V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan. τ²-Bench: Evaluating conversational agents in a dual-control environment, 2025. URL https://arxiv.org/abs/2506.07982.
  5. [5]K. Battocchi, E. Dillon, M. Hei, G. Lewis, P. Oka, M. Oprescu, and V. Syrgkanis. EconML: A Python Package for ML-Based Heterogeneous Treatment Effects Estimation. https://github.com/py-why/EconML, 2019. Version 0.x.
  6. [6]V. K. Bhat, P. Chanda, V. Ekbote, A. Khandelwal, M. Swaroop, V. N. Balasubramanian, S. Kambhampati, N. Natarajan, and A. Sharma. interwhen: A generalizable framework for steering reasoning models with test-time verification. In Advances in Neural Information Processing Systems, 2026. URL https://arxiv.org/abs/2602.11202.
  7. [7]A. Bonagiri, D. Borkar, G. J. Anderias, S. Rafatirad, and H. Homayoun. CausalFlow: Causal attribution and counterfactual repair for LLM agent failures, 2026. URL https://arxiv.org/abs/2605.25338.
  8. [8]A. Boruvka, D. Almirall, K. Witkiewitz, and S. A. Murphy. Assessing time-varying causal effect moderation in mobile health. Journal of the American Statistical Association, 113(523):1112–1121, 2018. doi: 10.1080/01621459.2017.1305274. URL https://doi.org/10.1080/01621459.2017.1305274.
  9. [9]A. Camuffo, A. Gambardella, S. Kazemi, and A. Pandey. Beyond black boxes: Designing and testing agentic ai systems for strategy. Strategy Science, 11(1):137–156, 2026.DOI
  10. [10]M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandran, M. Zaharia, J. E. Gonzalez, and I. Stoica. Why do multi-agent LLM systems fail? In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2025. URL https://arxiv.org/abs/2503.13657.
  11. [11]V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1–C68, 2018. doi: 10.1111/ectj.12097. URL https://doi.org/10.1111/ectj.12097.
  12. [12]E. Debenedetti, J. Zhang, M. Balunović, L. Beurer-Kellner, M. Fischer, and F. Tramèr. AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, volume 37, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/97091a5177d8dc64b1da8bf3e1f6fb54-Abstract-Datasets_and_Benchmarks_Track.html.arxiv.org/abs/2406.13352
  13. [13]N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/94074dd5a072d28ff75a76dabed43767-Abstract-Conference.html.arxiv.org/abs/2403.07974
  14. [14]N. Jiang and L. Li. Doubly robust off-policy value evaluation for reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 652–661. PMLR, 2016. URL https://proceedings.mlr.press/v48/jiang16.html.arxiv.org/abs/1511.03722
  15. [15]C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2310.06770.
  16. [16]P. Klasnja, E. B. Hekler, S. Shiffman, A. Boruvka, D. Almirall, A. Tewari, and S. A. Murphy. Microrandomized trials: An experimental design for developing just-in-time adaptive interventions. Health Psychology, 34(Suppl.):1220–1228, 2015. doi: 10.1037/hea0000305. URL https://pmc.ncbi.nlm.nih.gov/articles/PMC4732571/.DOI
  17. [17]G. Lewis and V. Syrgkanis. Double/debiased machine learning for dynamic treatment effects via g-estimation. arXiv preprint arXiv:2002.07285, 2020.arxiv.org/abs/2002.07285
  18. [18]G. Lewis and V. Syrgkanis. Double/debiased machine learning for dynamic treatment effects via g-estimation, 2021. URL https://arxiv.org/abs/2002.07285v5. Version 5; algorithm and lemma numbers refer to this version.
  19. [19]X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y. Su, H. Sun, M. Huang, Y. Dong, and J. Tang. AgentBench: Evaluating LLMs as agents. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2308.03688.
  20. [20]J. Lu, T. Holleis, Y. Zhang, B. Aumayer, F. Nan, H. Bai, S. Ma, S. Ma, M. Li, G. Yin, Z. Wang, and R. Pang. ToolSandbox: A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1160–1183. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.findings-naacl.65. URL https://aclanthology.org/2025.findings-naacl.65/.DOI
  21. [21]Y. Lu, T.-Y. Hsu, H. Gu, L. Cui, Y. Xie, W. P. Headden III, B. Yao, A. Veeragouni, J. Liu, S. Nag, et al. An agent a/b: Automated and scalable a/b testing on live websites with interactive live agents. In Proceedings of the Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems, pages 1–12, 2026.DOI
  22. [22]M. Ma, J. Zhang, F. Yang, Y. Kang, Q. Lin, S. Rajmohan, and D. Zhang. DoVer: Intervention-driven auto debugging for LLM multi-agent systems. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=mrEK16Jy6h.arxiv.org/abs/2512.06749
  23. [23]S. A. Murphy. Optimal dynamic treatment regimes. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 65(2):331–355, 2003. doi: 10.1111/1467-9868.00389. URL https://doi.org/10.1111/1467-9868.00389.
  24. [24]X. Nie and S. Wager. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108(2):299–319, 2021.arxiv.org/abs/1712.04912
  25. [25]T. Qian. Distal causal excursion effects: Modeling long-term effects of time-varying treatments in micro-randomized trials. Biometrics, 81(4):ujaf134, 2025. doi: 10.1093/biomtc/ujaf134. URL https://doi.org/10.1093/biomtc/ujaf134.
  26. [26]J. M. Robins. Correcting for non-compliance in randomized trials using structural nested mean models. Communications in Statistics-Theory and methods, 23(8):2379–2412, 1994.DOI
  27. [27]J. Shah. Causal agent replay: Counterfactual attribution for LLM-agent failures, 2026. URL https://arxiv.org/abs/2606.08275.
  28. [28]T. Shi, J. He, Z. Wang, H. Li, L. Wu, W. Guo, and D. Song. Progent: Securing AI agents with privilege control. arXiv preprint arXiv:2504.11703, 2025. URL https://arxiv.org/abs/2504.11703.
  29. [29]P. Thomas and E. Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning, volume 48, pages 2139–2148. PMLR, 2016. URL https://proceedings.mlr.press/v48/thomasa16.html.arxiv.org/abs/1604.00923
  30. [30]M. Uehara, C. Shi, and N. Kallus. A review of off-policy evaluation in reinforcement learning. Statistical Science, 41(3):561–581, 2026. doi: 10.1214/25-STS985. URL https://doi.org/10.1214/25-STS985.
  31. [31]S. Vansteelandt and M. Joffe. Structural nested models and G-estimation: The partially realized promise. Statistical Science, 29(4):707–731, 2014. doi: 10.1214/14-STS493. URL https://arxiv.org/abs/1503.01589.
  32. [32]H. Wang, C. M. Poskitt, and J. Sun. AgentSpec: Customizable runtime enforcement for safe and reliable LLM agents. In Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering, pages 2938–2950. Association for Computing Machinery, 2026. doi: 10.1145/3744916.3764546. URL https://doi.org/10.1145/3744916.3764546.
  33. [33]S. Xiao, S. Wang, X. Chen, K. Chao, M. Cui, F. Qian, C. Mei, F. Meng, Z. Yu, and J. Yin. When harnesses lose the signal: Causal evaluation of recovery in LLM agents, 2026. URL https://arxiv.org/abs/2610.00372.
  34. [34]S. Yao, N. Shinn, P. Razavi, and K. Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains. In International Conference on Learning Representations, 2025. URL https://arxiv.org/abs/2406.12045.
  35. [35]S. Zhang, M. Yin, J. Zhang, J. Liu, Z. Han, J. Zhang, B. Li, C. Wang, H. Wang, Y. Chen, and Q. Wu. Which agent causes task failures and when? On automated failure attribution of LLM multi-agent systems. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 76583–76599. PMLR, 2025. URL https://proceedings.mlr.press/v267/zhang25cq.html.arxiv.org/abs/2505.00212
  36. [36]Z. Zhong, A. Raghunathan, and N. Carlini. ImpossibleBench: Measuring LLMs’ propensity of exploiting test cases, 2025. URL https://arxiv.org/abs/2510.20270.
  37. [37]S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig. WebArena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/4410c0711e9154a7a2d26f9b3816d1ef-Abstract-Conference.html.arxiv.org/abs/2307.13854

Paper details

vlnc:W_kep3j6tv
Version
v2 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 16

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents