betaFeedback

WorksComputer Science

On Evaluating User Simulators for Interactive Agents

Abstract

User simulators offer a scalable way to evaluate interactive agents, but how to assess their quality remains underexplored. In particular, it is unclear whether existing user simulator evaluation frameworks reliably detect meaningful failures in simulated user behavior. This uncertainty makes it difficult to interpret simulator evaluation. To understand what a high benchmark score tells us about simulator quality, we (meta-)evaluate the metrics themselves. To instantiate this meta-evaluation, we deliberately perturb simulated replays to induce failures such as altered goals, omitted constraints, premature stopping, and acceptance of broken work, then test whether the metrics penalize these failures. Evaluation methods from three prior studies detect only 5-6 of the 10 perturbations; their combined scores even favor populations instructed to accept broken work over unmodified controls. Motivated by these gaps, we propose a new evaluation suite that combines utility of user simulators for evaluating agents and two measures of fidelity to real users: population-level fidelity assesses collective interaction patterns, while trace-level fidelity assesses the preservation of individual users' goals, constraints, and persona. Under the same detection criterion, our evaluation suite detects all 10 perturbations. Evaluating six user simulators on 351 real multi-turn sessions in which 348 distinct users across 23 languages and agents work together to create and refine artifacts such as full-stack web apps, games, design assets, and automated workflows, we find that population-level measures can miss when a simulated user pursues the wrong goal, ignores a constraint, or behaves out of character, highlighting the need to assess fidelity at comprehensive granularities.

Introduction

With the capability advances, LLM agents help users write code, create documents, and revise other artifacts [19, 1, 32, 31] (Kimi Team 2026) through multi-turn interactions [9, 30, 3]. During these interactions, users clarify what they want, inspect the work produced, and provide feedback. Evaluating this process requires a user who can respond to the agent’s progress, but recruiting real users for every evaluation is costly and makes comparisons difficult to repeat. User simulators offer a practical way to run these interactions at scale and are used in both offline agent evaluation and training [2, 23, 28]. However, using simulated users in place of real users can change measured agent success rates [25].

To assess whether simulators can reliably stand in for real users in multi-turn interactions, recent studies have developed several ways to evaluate user simulators, including similarity to human language and behavior, consistency with user specifications [35, 10], and agreement with human assessments of agent performance [9, 14, 37, 30]. What remains less clear is whether these measures reliably identify behavior that makes a simulator unsuitable for agent evaluation where artifacts deeply involved in the interaction. For example, a simulated user may sound natural while approving a broken artifact or overlooking an important requirement. Evaluating fidelity is also difficult because a replay should preserve the original user’s needs while responding to an agent whose actions may differ from the original session. We therefore need to examine how evaluation metrics respond when these aspects of simulation go wrong.

To understand what a high benchmark score tells us about simulator quality, we (meta-)evaluate the metrics themselves. Specifically, we introduce failures through controlled perturbations [24] to the simulator’s setup, then replay each session with the assigned agent. We test whether the metrics assign lower scores to these perturbed replays than to controls generated from the same source sessions and agent assignments. We design 10 perturbations based on three sources of motivation. First, we draw on reported failure modes, including overly helpful simulated users [17], premature information disclosure [37, 10, 13], and self-preference in LLM evaluation [36, 20, 26]. Second, we draw on failures observed in our analysis of 442,516 real sessions and 20,100 replays. Third, we include basic defects that evaluation metrics should readily detect. These include changing the user’s goal, removing constraints, forcing early termination, encouraging acceptance of broken work, and favoring a selected agent. We ask whether the existing user simulator evaluation metrics penalize these changes and what replays are retained when using their score for filtering.

The results reveal several gaps in the evaluation suites from SimulatorArena, VISTA, and SWE-Together [9, 14, 30]. We count a perturbation as detected if at least one metric in an evaluation suite shows a reliable score decrease relative to the control (§ 6.1.1). Under this criterion, the evaluation suites adapted from SimulatorArena, VISTA, and SWE-Together [9, 14, 30] detect six, five, and six of the ten perturbations, respectively. All three evaluation suites fail to detect the agent favoritism perturbation (more favorable feedback from the simulated user toward a selected agent) despite the shifts it causes in agents’ scores and ranks. The composite scores for these metrics even favor perturbed replays that accept broken work. These findings suggest that high scores on existing criteria do not necessarily establish that a simulator provides useful feedback for help evaluating interactive agents.

Motivated by these findings, we introduce a new evaluation suite that measures user simulators at three levels illustrated in Figure 2. Downstream ranking agreement checks whether simulated interactions recover a supplied reference ranking of agents. This assesses the simulator’s utility for agent evaluation, but gives limited information about why it succeeds or fails. We therefore also measure population-level fidelity, which compares collective interaction patterns with those of real users, and trace-level fidelity, which checks whether each replay preserves the user’s goals, constraints, and persona. The two fidelity levels help distinguish broad behavior shifts from failures within individual sessions. Together, these measures provide a stronger and more holistic assessment of how well a simulator supports agent evaluation at a broader range of failures than any of the existing benchmarks we evaluated.

Figure 2. Overview of the evaluation framework. L1 compares simulator-derived agent rankings with a reference ranking; L2 compares behavioral statistics of simulated and real-user populations; L3 assesses intent, constraints and persona using conversations and workspace evidence. Examples and plots are illustrative.

We show that our evaluation suite detects all ten tested perturbations under the same criterion in § 6.1. We also test whether our evaluation scores can help filtering out simulator failures. We report the fraction of control traces retained at a threshold that admits 5% of perturbed samples. Our metric retains 98% of control populations in the agent favoritism filtering experiment, compared with 10% for Sim2Real, the best external benchmark. Under the same conditions, action distribution retains 100% of control populations in the approval-of-broken-work experiment, compared with 26% for the best external benchmark. At the trace level, the persona score retains 68% of real-user traces, compared with 28% for the best external benchmark. Finally, we apply the evaluation suite to six simulators on 351 real sessions. None of the six simulators we evaluated leads across all three levels. The simulator from SimulatorArena achieves the highest downstream ranking agreement, while our proposed profile-based simulator lead on population-level and trace-level fidelity. These results show the value of examining all three levels when assessing a simulator.

Our contribution are four-fold:

  • A perturbation-based meta-evaluation protocol for examining what user-simulator evaluation metrics detect and retain in § 4.

  • A new evaluation suite grounded in real user sessions that combines downstream ranking agreement with population- and trace-level fidelity in § 5.

  • A 351 real-user sessions used in our evaluation to support future research on user simulation for agent evaluation.

  • An empirical study of existing evaluation criteria, their use in data selection, and the strengths and weaknesses of six simulators in § 6.2.

Related Work

User simulators for agent evaluation. Task-oriented benchmarks such as τ2\tau^2-bench use an environment-grounded user simulator in interactive agent evaluation [2]. SimulatorArena studies open-ended assistant interactions using simulated users and compares simulator-derived assessments with human judgments [9]. SWE-Together reconstructs developer sessions and uses a state-conditional user simulator to preserve task intent while adapting feedback to the new agent [30]. SWE-DIVE converts coding benchmarks into multi-turn evaluations in which a simulated user responds to the agent’s replies and codebase changes [5], and Cowork-Gym synthesizes persistent work environments with a stateful simulated user that observes the workspace [38]. Lost in Simulation documents discrepancies between agent success rates measured with simulated and real users [25]. Disclosure-Gated User Simulation examines how disclosure mechanisms affect companion-agent evaluation [13]. These settings motivate assessing both simulated behavior and its use in agent evaluation.

Evaluating the simulator. Surveys distinguish behavioral consistency from downstream utility [22] and discuss simulation of general and individual users (Ni et al., 2026). Prior work considers behavioral realism, consistency with user specifications, and similarity to real-user populations [9, 14, 30, 37]. ARGUS evaluates behavioral consistency with scenario requirements [35], while UserProxyBench measures adherence to task-specific user instructions and separately checks user tool execution in Telecom [10]. UserArena evaluates shopping user simulators at five levels, from output validity to sequential behavior, personalization, and outcomes [34]. CUE assesses whether simulated users reproduce the success and failure patterns of real interactions [11], and AutoSim compares simulator-derived rankings of conversational search systems with rankings based on relevance judgments [33]. Section 4.4 describes the evaluation criteria used in our comparisons. These studies provide complementary criteria; we examine how such criteria respond to controlled replay perturbations and what data they retain when used as filters.

Meta-evaluation of LLM judges. Meta-evaluation is the systematic assessment of an evaluation’s quality and utility [27]. Our meta-evaluation treats the scoring criteria themselves as objects of study, following prior work on metric validity and perturbation testing [15, 24, 16]. It compares scores for control and perturbed interactions generated from the same source sessions, then examines selection under score thresholds. Perturbations act on the simulator’s setup, allowing the resulting interaction to evolve; the test therefore concerns an interactive generation condition rather than a static textual edit.

User-Simulator Evaluation

Problem Formulation

We study user simulation through replay of real user sessions in artifact-producing workspaces. We assume access to user–agent traces containing dialogue and, where available, uploaded artifacts and workspace evidence. As shown in Figure 1, a source session provides the initial request and a representation of the user for replay. The simulator interacts with an agent whose actions can modify the workspace and produce artifacts, using the developing dialogue and available workspace evidence to generate feedback or stop.

Figure 1. Simulator evaluation and perturbation-based meta-evaluation. Replays are compared through agent ranking agreement (L1), population-level fidelity (L2) and trace-level fidelity (L3). Paired control and perturbed replays test benchmark sensitivity and filtering.

The replay agent may produce different responses and artifacts from the original agent. The source trace therefore does not specify the correct user response at every turn. Fidelity requires preserving the source user’s intent and constraints while adapting feedback to the new interaction and the work actually produced.

Evaluation Principles

Evaluating user simulators for interactive agents requires examining the downstream utility and fidelity to the users they represent. We organize these requirements into three complementary levels.

Downstream evaluation utility (L1). Simulated interactions should support valid assessments of agent performance in the intended setting [22]. Prior work shows that simulator-based evaluations can misestimate agent success rates relative to evaluations with real users [25]. Reliable agent comparisons therefore require simulated users to respond appropriately to differences in agent performance. Their feedback and stopping decisions should reflect the quality of the agent’s work under comparable standards across agents. In artifact-producing tasks, evaluation must account for how well the resulting work meets user needs: plausible dialogue alone provides insufficient evidence of successful performance [4, 10, 35]. These requirements motivate evaluating downstream utility directly: whether simulated interactions support reliable comparisons of agent performance. This complements behavioral fidelity by checking whether the simulator serves its intended purpose as an evaluation tool.

Population-level fidelity (L2). Simulated populations should reproduce relevant patterns of real-user behavior. These include how users introduce requirements, provide information, respond to progress and errors, and decide that work is complete. Reported failures include excessive cooperation [37, 6] and premature or unconditional information disclosure [10, 13]. In workspace interactions, fidelity also concerns how feedback relates to the state and quality of artifacts. Systematic differences in these patterns change the evaluation conditions agents encounter.

Trace-level fidelity (L3). Individual interactions should preserve the represented user’s intent, constraints, and persona (Ni et al., 2026) while responding to the agent’s actions. Prior work highlights fidelity failures such as goal and constraint violations, premature termination, ungrounded confirmations, fabricated observations, and claims of actions that were never performed [35, 10]. Population-level similarity can conceal these failures within individual replays, motivating evaluation against the particular user each replay represents. In our real-user traces, users inspect produced artifacts and base their acceptance or requests for changes on whether the work meets their requirements. For artifact-producing tasks, fidelity therefore concerns both preserving the source user’s requirements and applying their judgment criteria when assessing the resulting work.

Existing benchmarks typically cover only parts of these principles. Population-level comparisons, such as those used within Sim2Real, characterize collective behavior, but matching aggregate patterns can leave changes to individual users’ intent, constraints, or persona unobserved [37]. Individual fidelity therefore requires examining whether each replay preserves the particular user it represents. VISTA evaluates consistency with user specifications, while SWE-Together checks whether simulated guidance preserves the original task scope. However, consistency and scope preservation alone do not establish whether the simulator reproduces the represented user’s degree and manner of participation [14, 30]. SimulatorArena and SWE-Together assess agent outcomes alongside message similarity or preservation of source-user intents, but assessing these separately does not directly establish whether the user’s acceptance or criticism is justified by the work produced [9, 30].

Motivated by the above-mentioned limitations, we develop a meta-evaluation framework to quantitatively evaluate existing user simulator evaluation suites in § 4, and then propose a new evaluation suite to overcome those issues in § 5.

Perturbation-Based Meta-Evaluation

Assessing the reliability of an evaluation method is itself a challenging problem. Agreement with human judgments can depend strongly on the outputs used for assessment and carry substantial statistical uncertainty [15, 8]. Moreover, agreement on overall quality does not establish that a metric captures each relevant evaluation dimension [24]. For user simulation, this challenge is compounded by the replay setting: source traces do not prescribe responses to new agent actions, and appropriate feedback depends on evolving dialogue, workspace state, and artifacts.

We therefore propose a perturbation-based meta-evaluation. Following criteria-targeted perturbation testing [24], we modify simulator setups to challenge the principles in § 3.2. Figure 1 shows the procedure: generate paired control and perturbed replays, then examine whether evaluation scores penalize the targeted changes (sensitivity) and how well a score distinguish control from perturbed candidates (filtering).

Session Data

Our meta-evaluation uses real user–agent sessions involving goal-directed work on deliverables as source sessions. We use two sets that differ primarily in interaction length. The easy set contains shorter interactions (median: 7 user turns; 10th–90th percentile range: 5–11). The hard set involves longer-running projects with more complex tasks and deliverables (median: 26 user turns; 10th–90th percentile range: 8–65).

Replay Setup

For each source-session–agent pair and each perturbation, we compare a control replay generated using the baseline setup below with a perturbed replay generated using the corresponding intervention from § 4.3.

Control replay. We extract a session profile from the source dialogue, final workspace, and uploaded materials, describing the intended deliverable, constraints, preferences, persona, and stopping conditions. The simulator uses this profile to represent the source user in a new interaction with the assigned agent. See Appendix § 8.§ 3 for a profiling example.

The interaction begins with the original opener and associated files in a persistent sandbox. Later uploads are available for attachment during interaction. At subsequent turns, the simulator receives the profile, developing dialogue, and current workspace evidence, including text-file contents, metadata, and tool-error summaries. It generates feedback or stops, with instructions to ground acceptance in evidence supporting the agent’s claims. Each replay ends when the simulator stops or reaches the cap of 30 user turns.

Perturbed replay. We modify a specified part of this baseline setup, such as the profile, response instructions, or turn budget for perturbation. The source session, opener, initial files, and assigned agent remain the same.

Both replay use gpt-5.6-terra as the simulator model. Each condition runs with a fresh agent instance and a separate workspace, allowing its dialogue and artifacts to develop independently. The following subsection defines the interventions and the evaluation requirements they challenge.

Perturbation Design

We design 10 perturbations drawing on three sources: failure modes reported in prior work, failures observed in our replay analyses, and basic defects that an evaluation should readily detect. We organize these perturbations according to the evaluation principles introduced in § 3.2. We describe each intervention below, with implementation details in Appendix § 9.

Downstream evaluation utility (L1). Agent favoritism instructs the simulated user to favor a particular agent by excusing errors, accepting imperfect deliverables, and giving warmer feedback. Profile extraction artifact turns dissatisfaction with the original agent into a persistent user trait [29], making the simulator remain critical even when the replay agent meets requirements.

Population-level fidelity (L2). Approve broken work instructs the simulated user to approve deliverables despite errors or missing requirements. Premature stop truncates replays early, reducing opportunities for users to inspect deliverables and request corrections. Front-loading moves requirements and acceptance criteria that would otherwise emerge during interaction into the opening messages.

Trace-level fidelity (L3). Adversarial goal replaces the intended goal and deliverable with a plausible alternative, directing later feedback toward the altered task. All constraints removed removes hard requirements from the profile while preserving those expressed in the original opener. Persona swap replaces the source user’s writing style with a contrasting register while preserving the goal and requirements. Co-debugger instructs the simulated user to diagnose problems and suggest fixes, providing technical assistance beyond the represented user’s role. Language switch changes the simulated user’s language after the original opener.

Meta-Evaluation Protocol

Our meta-evaluation takes an evaluation metric or evaluation suite as input. We test whether it detects controlled perturbations (sensitivity) and distinguishes controls from perturbed candidates (filtering). All scores are oriented so that higher is better.

Sensitivity. For trace-level comparisons, we compare matched replay pairs indexed by ii for metric mm and perturbation pp. Each pair shares the source session and agent assignment: xiCx_i^C is the control replay and xipx_i^p its perturbed counterpart. Let sm(x)s_m(x) be metric mm’s score for replay xx. Let Im,pI_{m,p} contain the indices of pairs with both scores available, and let ∣Im,p∣\lvert I_{m,p}\rvert denote their number. We define

di,m,p=sm(xiC)−sm(xip),dˉm,p=1∣Im,p∣∑i∈Im,pdi,m,p.(1)\begin{aligned} d_{i,m,p} &= s_m(x_i^C) - s_m(x_i^p), \\ \bar{d}_{m,p} &= \frac{1}{\lvert I_{m,p}\rvert} \sum_{i \in I_{m,p}} d_{i,m,p}. \tag*{(1)} \end{aligned}

Here, di,m,pd_{i,m,p} is the paired score decrease for case ii, and dˉm,p\bar{d}_{m,p} is its mean across available pairs. A positive value means that the metric scores the perturbed replay lower. We report the paired win rate

Wm,p=1∣Im,p∣∑i∈Im,ph(di,m,p).(2)W_{m,p} = \frac{1}{\lvert I_{m,p}\rvert} \sum_{i \in I_{m,p}} h(d_{i,m,p}). \tag*{(2)}

where h(d)h(d) equals 1 for d>0d > 0, 1/2 for d=0d = 0, and 0 for d<0d < 0. Thus, Wm,pW_{m,p} measures how often the control scores higher, giving ties half credit.

For population comparisons, we repeatedly sample sets of sessions to test whether the metric consistently assigns lower scores to perturbed populations. Let B=200B = 200 denote the default number of sampling rounds. For each round b∈{1,…,B}b \in\{1, \ldots, B\}, we sample 100 sessions to obtain the control and perturbed sets, XbCX_b^C and XbpX_b^p, from the same session. Let Sm(X)S_m(X) be metric mm’s population score: population metrics use a fixed real-user reference for each comparison, while trace metrics are averaged within XX. We define

Db,m,p=Sm(XbC)−Sm(Xbp).(3)D_{b,m,p} = S_m(X_b^C) - S_m(X_b^p). \tag*{(3)}

Here, Db,m,pD_{b,m,p} is the paired population score decrease in draw bb. The population win rate is B−1∑b=1Bh(Db,m,p)B^{-1}\sum_{b=1}^{B} h(D_{b,m,p}), the fraction of draws in which the control population scores higher than its matched perturbed population, with half credit for ties. Population-level detection requires the 2.5th percentile of {Db,m,p}b=1B\{D_{b,m,p}\}_{b=1}^{B} to be above zero.

Agent-targeted perturbations follow the same population-comparison setup. Each draw samples session–agent pairs for the targeted agents and inserts either their control or perturbed replays into an otherwise fixed evaluation pool (Appendix § 11.1.1).

An evaluation suite detects a perturbation if any constituent metric meets the criterion at any applicable level. Each perturbation is counted only once.

Filtering. Sensitivity tests whether a perturbation lowers the score relative to its matched control. Filtering tests whether scores distinguish controls from perturbed candidates across a candidate pool, revealing what simulations receive high scores and survive selection.

At the trace level, we test whether scores distinguish perturbed replays from control replays and real-user traces. For agent differentiation and population-level fidelity, we compare populations containing only control replays with matched populations constructed by replacing 10%, 25%, 50% or 100% of the sampled control replays with their perturbed counterparts. Within each comparison, all metrics use the same sampled replay pairs and replacement choices.

For each metric, filtering retains traces or populations whose scores meet or exceed a threshold. Across all comparisons, we report AUROC to measure how well scores distinguish the control or real-user group from the perturbed group.

For trace-level filtering, we additionally report the fraction of control replays or real-user traces retained when the threshold allows 5% or 10% of perturbed replays to pass. Higher retention at the same perturbed-survival rate indicates more effective filtering.

Evaluation Framework

As shown in § 6.1, existing evaluation suites fail miserably on perturbations: they overlook biased feedback toward particular agents, changes to user goals or constraints, and acceptance of broken work, or even rewarding perturbed replays. These failures leave gaps in assessing agent differentiation, population-level fidelity, and trace-level fidelity. We therefore propose an evaluation framework that measures these three aspects and provides more detailed feedback on simulator behavior. Below, we describe how each component is implemented.

Figure 2 summarizes the framework’s three comparisons. L1 uses simulated user feedback to rank agents and compares the resulting ranking with an independent reference ranking. L2 compares behavioral statistics across simulated and real-user populations. L3 checks whether each replay is faithful to its source user’s intent, constraints, and persona, using dialogue and available workspace evidence. We report these three component scores and combine them into an overall evaluation suite score. Appendix § 10 provides implementation details, including dialogue-act definitions and score calculations.

Downstream ranking agreement (L1)

Downstream ranking agreement (L1) measures whether evaluating agents with a user simulator produces a ranking consistent with a supplied reference ranking. A judge model (Claude Sonnet 4.6) labels user turns after the opener as accept, extend, correct, redirect, or reject, with stop recorded at session end. For replay xx, we compute the satisfaction total

s(x)=Naccept(x)+Nstop(x)−Ncorrect(x)−Nreject(x),(4)s(x) = N_{\mathrm{accept}}(x) + N_{\mathrm{stop}}(x) - N_{\mathrm{correct}}(x) - N_{\mathrm{reject}}(x), \tag*{(4)}

where Nr(x)N_r(x) counts occurrences of reaction rr. Extend and redirect contribute zero. We aggregate reaction counts by agent and normalize by the number of classifiable reactions plus stop labels to obtain each agent’s satisfaction score. We rank agents by these scores to obtain a ranking π^\hat{\pi}, then measure its agreement with the supplied reference ranking πref\pi^{\mathrm{ref}} using Kendall’s τ\tau.

The reference is an independent input chosen for the evaluation setting of interest. It could come from human comparisons in the target setting or an existing leaderboard with matching scope. Its construction is outside the contribution of this framework. Our application supplies a reference ranking derived from real-user reactions labeled by the same Claude Sonnet 4.6 judge.

The 0–100 L1 score combines ranking order and standardized score-gap agreement; Appendix § 10.§ 2 provides more detailed implementation.

Population-level behavioral fidelity (L2)

Population-level behavioral fidelity (L2) measures how closely simulated users reproduce real-user behavior across sessions. We compute the same behavioral statistics on the simulated population and the real-user reference population, then measure their similarity using 4 metric families:

  1. Action distribution. An LLM judge classifies user turns into dialogue acts, such as refining a deliverable, reporting a bug, adding a subtask, or asking the agent to proceed (taxonomy in Appendix § 10.§ 3.1). We exclude the copied opening request and renormalize the remaining act frequencies:

Sact=100∑amin⁡(pasim,pareal),(5)S_{\mathrm{act}} = 100 \sum_{a} \min\left(p_{a}^{\mathrm{sim}}, p_{a}^{\mathrm{real}}\right), \tag*{(5)}

where pap_a is the proportion of turns assigned to act aa.

  1. Interaction dynamics. We compute 5 statistics: words per user turn, user turns per session, lexical diversity, question-turn rate, and code-turn rate. Lexical diversity uses a moving-average type-token ratio [7] with a 50-token window. Appendix § 10.§ 3 provides the extraction details. For each statistic, let rkr_k be the simulated-to-real ratio. The score is

Sdyn=1005∑k=15min⁡(rk,1/rk).(6)S_{\mathrm{dyn}} = \frac{100}{5} \sum_{k=1}^{5} \min(r_k, 1/r_k). \tag*{(6)}
  1. Outcome sensitivity. We measure how user feedback changes following agent errors and compare this response pattern between simulated and real users. We flag agent turns containing execution error and use the next user turn’s dialogue-act label to determine the response’s valence. Appendix § 10.§ 3 describes the error indicators and label mapping.

Let EtE_t indicate whether agent turn tt contains a detected execution error, and let Rt+1R_{t+1} denote the following user response’s valence. Only responses in V={negative,positive}\mathcal{V} = \{\text{negative}, \text{positive}\} enter the conditional probabilities. For population q∈{sim,real}q \in\{\text{sim}, \text{real}\}, we compute

Δq=Pr⁡q(Rt+1=negative∣Et=1, Rt+1∈V)−Pr⁡q(Rt+1=negative∣Et=0, Rt+1∈V).(7)\Delta_q = \Pr_q(R_{t+1} = \text{negative} \mid E_t = 1,\ R_{t+1} \in\mathcal{V}) - \Pr_q(R_{t+1} = \text{negative} \mid E_t = 0,\ R_{t+1} \in\mathcal{V}). \tag*{(7)}

Here, Et=1E_t = 1 denotes a detected error and Et=0E_t = 0 denotes no detected error. The score is

Sout=100(1−∣Δsim−Δreal∣2).(8)S_{\mathrm{out}} = 100\left(1 - \frac{|\Delta_{\mathrm{sim}} - \Delta_{\mathrm{real}}|}{2}\right). \tag*{(8)}
  1. Information provenance. We measure whether simulated users contribute a similar share of task information as real users. Over-helpful simulators may volunteer diagnoses or decisions that the agent should work out, making its task easier. Conversely, simulators that provide too

little information may withhold requirements or clarifications that real users would supply. We therefore compare how information is divided between the user and agent in simulated and real interactions.

An LLM judge identifies key information points in each conversation and attributes them to the user or agent. Appendix C.3 describes the information categories and attribution procedure. For population q∈{sim,real}q \in\{\mathrm{sim},\mathrm{real}\}, let uqu_q denote the proportion of information points originating from the user. We compare the user-to-agent information odds:

OR=usim/(1−usim)ureal/(1−ureal).(9)\mathrm{OR} = \frac{u_{\mathrm{sim}}/(1-u_{\mathrm{sim}})}{u_{\mathrm{real}}/(1-u_{\mathrm{real}})}. \tag*{(9)}

The score is

Sprov=100min⁡(OR,OR−1).(10)S_{\mathrm{prov}} = 100 \min\left(\mathrm{OR},\mathrm{OR}^{-1}\right). \tag*{(10)}

Equal user-origin shares receive a score of 100; shifts toward either greater or lower user contribution reduce the score.

The aggregated L2 score is the arithmetic mean of the 4 family scores. Higher scores indicate closer agreement with the measured real-user behavior.

Trace-level fidelity

Trace-level fidelity (L3) measures whether each simulated replay preserves the original user’s goal, requirements and persona while responding to the agent’s actions. We evaluate each replay against its real-user session using 3 metric families:

  1. Intent preservation. We check whether the simulated user continues pursuing the original goal. An LLM judge compares the real and simulated conversations, using workspace evidence to understand what was produced, and classifies the replay as having no drift, minor drift or major drift. Major drift includes replacing the original goal or abandoning it before an attempt. An agent’s failure to complete the requested task does not itself constitute user intent drift. The intent score is the proportion of evaluated replays without major drift.

  1. Constraint preservation. We check whether the simulated user maintains the real user’s explicit requirements. An LLM judge first extracts hard constraints from the real user’s messages, such as output language, deliverable format and required features. It then checks each constraint against the simulated conversation and final workspace, recording whether the user retained it and whether the deliverable satisfied it.

  1. Persona consistency. We check whether the simulator preserves both how the user communicates and how they judge the agent’s work. Style similarity is assessed by comparing profiles extracted from the real and simulated sessions. For judgment criteria in the application, an LLM judge checks whether the simulated user’s decisions follow the original user’s criteria, labeling each criterion as enacted, violated or not triggered. Criteria conformance is

C=NenactedNenacted+Nviolated.(11)C = \frac{N_{\mathrm{enacted}}}{N_{\mathrm{enacted}} + N_{\mathrm{violated}}}. \tag*{(11)}

The persona score averages style similarity and criteria conformance; criteria that were not triggered are excluded from the conformance calculation.

The L3 score is the arithmetic mean of the 3 family scores, expressed on the same scale. Appendix C.4 provides the judge definitions and implementation details.

Score aggregation

We summarize simulator performance while keeping each evaluation level visible.

The current evaluation suite (bench-v0) reports each level on a 0–100 scale and uses the geometric mean of L1, L2 and L3 as its overall evaluation suite score. We report all three levels alongside the overall evaluation suite score because an aggregate can conceal different failure modes. A level score is an evaluation index, not a calibrated probability of fidelity.

Experimental Results

We first report the meta-evaluation results, then apply our evaluation framework to existing simulators.

Meta-evaluation results

Using the protocol in § 4, we first examine sensitivity to perturbations and then what survives score-based filtering.

Sensitivity to perturbations

We examine which evaluation metrics detect perturbations affecting agent differentiation, population-level fidelity and trace-level fidelity. we consider a perturbation successfully detected when the 95% confidence interval for the paired score decrease lies entirely above zero. For population-level scores, we require the 2.5th percentile of the paired score decreases to exceed zero. We color the corresponding cells green when these criteria are met and red otherwise. Figure 3–Figure 5 show the compact metric summaries; Appendix § 11 provides the detailed results.

Figure 3. Sensitivity to agent favoritism and negative profile artifacts. Labeled points show the decrease in ranking agreement (control minus perturbed), with bootstrap 95% confidence intervals.

Figure 5. Trace-level sensitivity using each evaluation suite’s best metric per perturbation: (a) paired control-over-perturbed win rates; (b) mean score decreases and bootstrap 95% confidence intervals.

Across the 10 perturbations, overall detection coverage is 10/10 for our evaluation suite, 6/10 for SimulatorArena, 5/10 for VISTA and 6/10 for SWE-Together. Coverage summarizes the signals available across an evaluation suite; the filtering analyses separately assess individual scores and fixed composites.

Sensitivity to perturbations affecting agent differentiation We test whether metrics detect feedback biases that distort comparisons between agents.

Our ranking-agreement metric detects both agent favoritism and negative profile artifacts, with decreases of 0.04 and 0.06, respectively (Figure 3). SimulatorArena shows a decrease of 0.03 for each perturbation, but both confidence intervals include zero.

Negative profile artifacts also produce behavioral signals: the L2 act-distribution metric has a population-level win rate of 0.99 (Appendix Table 6), the persona family detects the perturbation at the trace level, and SimulatorArena, VISTA and SWE-Together each have at least one sensitive trace-level metric. These results show how feedback biases can register through their effects on agent comparisons and on simulated behavior.

Sensitivity to perturbations affecting population-level fidelity We test whether metrics detect changes in behavioral patterns across populations of replays.

The L2 act-distribution metric detects approve-broken-work and front-loading with population-level win rates of 1.00 in both cases (Figure 4). SimulatorArena and VISTA also detect front-loading, with win rates of 0.99 and 0.98 for their best metrics. Premature stop produces a different pattern: no L2 family detects it, whereas SWE-Together’s coverage metric reaches a population-level win rate of 1.00. At the trace level, our constraints metric detects premature stop with a paired win rate of 0.56 (Figure 5). Appendix Table 9 examines how population sensitivity changes with population size.

Figure 4. Population-level sensitivity of L2 and external evaluation suites using each row’s best metric per perturbation: (a) control-over-perturbed win rates; (b) mean score decreases and 2.5th–97.5th percentile ranges.

Sensitivity to perturbations affecting trace-level fidelity We test sensitivity to changes in a user’s goal, constraints, persona, language and role within a replay.

The constraints metric detects removed constraints; the persona metric detects persona swaps, language switches and co-debugger behavior (Figure 5). SimulatorArena’s metrics miss altered goals and removed constraints, VISTA’s miss removed constraints and persona swaps, and SWE-Together’s miss altered goals and language switches.

Aggregation can amplify these differences: for adversarial goal, the L3L_3 mean rises from a paired win rate of 0.70 to a population-level win rate of 1.00 (Appendix Table 7).

Which simulations receive high scores?

We examine what survives filtering to understand what a high evaluation score signals. The following analyses use the filtering protocol in § 4.4. Table 1–Table 3 report the best AUROC for each evaluation suite and condition.

Evaluation suiteAgent favoritismNegative profile artifact
10%25%50%100%10%25%50%100%
Ours0.55R0.55^{\mathrm{R}}0.69R0.69^{\mathrm{R}}0.84R0.84^{\mathrm{R}}0.99R0.99^{\mathrm{R}}0.62A0.62^{\mathrm{A}}0.75A0.75^{\mathrm{A}}0.91A0.91^{\mathrm{A}}1.00A1.00^{\mathrm{A}}
SimulatorArena0.50W0.50^{\mathrm{W}}0.54W0.54^{\mathrm{W}}0.56W0.56^{\mathrm{W}}0.64W0.64^{\mathrm{W}}0.55I0.55^{\mathrm{I}}0.60I0.60^{\mathrm{I}}0.67I0.67^{\mathrm{I}}0.84I0.84^{\mathrm{I}}
VISTA0.49K0.49^{\mathrm{K}}0.50F0.50^{\mathrm{F}}0.50F0.50^{\mathrm{F}}0.50F0.50^{\mathrm{F}}0.60F0.60^{\mathrm{F}}0.71F0.71^{\mathrm{F}}0.86F0.86^{\mathrm{F}}0.98F0.98^{\mathrm{F}}
SWE-Together0.51Q0.51^{\mathrm{Q}}0.49S0.49^{\mathrm{S}}0.49S0.49^{\mathrm{S}}0.44S0.44^{\mathrm{S}}0.67U0.67^{\mathrm{U}}0.85U0.85^{\mathrm{U}}0.98U0.98^{\mathrm{U}}1.00U1.00^{\mathrm{U}}
Sim2Real0.510.530.550.580.500.510.560.61

Table 1. Filtering on agent favoritism and negative profile artifacts. Highest AUROC at each replacement fraction; bold marks the column maximum.

Selected metrics. Ours: R ranking agreement, A action distribution; SimulatorArena: W Likert writing, I Likert interaction; VISTA: K knowledge consistency, F profile consistency; SWE-Together: Q weighted coverage, S scope precision, U user correction. Sim2Real uses its behavioral index.

Benchmark (best metric)AUROCcontrol@5%FPRcontrol@10%FPRAUROCreal@5%FPRreal@10%FPR
Ours0.62 persona0.100.170.90 persona0.680.77
SimulatorArena0.55 Likert writing0.050.110.88 Likert writing0.280.55
VISTA0.55 composite0.060.110.68 composite0.090.19
SWE-Together0.52 coverage rate0.050.110.75 user correction0.120.24

Table 3. Trace-level filtering using each evaluation suite’s highest-AUROC metric for each comparison. Columns report AUROC and positive-trace retention at 5% and 10% perturbed survival, using control replays (left) or real-user traces (right) as positives. Bold marks the highest value in each column. Full results: Appendix Table 16.

Which simulations score highly on agent differentiation? We examine whether high ranking-agreement scores select control populations over populations containing agent favoritism or negative profile artifacts.

For agent favoritism, L1L_1’s AUROC increases from 0.55 at 10% replacement to 0.84 at 50% and 0.99 at 100%. The strongest external metric reaches only 0.56 and 0.64 at 50% and 100% replacement, respectively (Table 1).

For negative profile artifacts, filtering improves as the perturbed fraction increases. At 25% replacement, the L2L_2 act-distribution metric, VISTA’s profile-consistency metric and SWE-Together’s user-correction metric achieve AUROCs of 0.75, 0.71 and 0.85, respectively. At 100% replacement, these behavioral metrics reach 0.98–1.00 AUROC (Table 1). These results illustrate complementary signals: L1L_1 provides the strongest filtering of agent favoritism, while the act-distribution and external behavioral metrics identify negative profile artifacts.

Which simulations receive high population-level fidelity scores? We examine how much perturbed behavior a population can contain while still receiving high fidelity scores (Table 2).

Evaluation suiteApprove broken workPremature stopFront-loading
10%25%50%100%10%25%50%100%10%25%50%100%
Ours, L20.49D0.49^{\mathrm{D}}0.46D0.46^{\mathrm{D}}0.45P0.45^{\mathrm{P}}1.00A1.00^{\mathrm{A}}0.49P0.49^{\mathrm{P}}0.49P0.49^{\mathrm{P}}0.49P0.49^{\mathrm{P}}0.46P0.46^{\mathrm{P}}0.56A0.56^{\mathrm{A}}0.65A0.65^{\mathrm{A}}0.78A0.78^{\mathrm{A}}0.96A0.96^{\mathrm{A}}
SimulatorArena0.53W0.53^{\mathrm{W}}0.58W0.58^{\mathrm{W}}0.66W0.66^{\mathrm{W}}0.78W0.78^{\mathrm{W}}0.53T0.53^{\mathrm{T}}0.58T0.58^{\mathrm{T}}0.62T0.62^{\mathrm{T}}0.69T0.69^{\mathrm{T}}0.56W0.56^{\mathrm{W}}0.64W0.64^{\mathrm{W}}0.74W0.74^{\mathrm{W}}0.90W0.90^{\mathrm{W}}
VISTA0.43K0.43^{\mathrm{K}}0.31K0.31^{\mathrm{K}}0.16K0.16^{\mathrm{K}}0.01K0.01^{\mathrm{K}}0.45G0.45^{\mathrm{G}}0.38G0.38^{\mathrm{G}}0.27G0.27^{\mathrm{G}}0.12G0.12^{\mathrm{G}}0.56F0.56^{\mathrm{F}}0.64F0.64^{\mathrm{F}}0.77F0.77^{\mathrm{F}}0.92F0.92^{\mathrm{F}}
SWE-Together0.50S0.50^{\mathrm{S}}0.50S0.50^{\mathrm{S}}0.52S0.52^{\mathrm{S}}0.50S0.50^{\mathrm{S}}0.69Q0.69^{\mathrm{Q}}0.88Q0.88^{\mathrm{Q}}0.99Q0.99^{\mathrm{Q}}1.00C1.00^{\mathrm{C}}0.53C0.53^{\mathrm{C}}0.59C0.59^{\mathrm{C}}0.65C0.65^{\mathrm{C}}0.80S0.80^{\mathrm{S}}
Sim2Real0.490.480.450.550.480.450.440.600.500.520.560.61

Table 2. Population-level filtering. Highest AUROC at each replacement fraction; bold marks the column maximum.

Selected metrics. Ours: D dynamics, P information provenance, A action distribution; SimulatorArena: W writing, T Turing; VISTA: K knowledge consistency, G goal consistency, F profile consistency; SWE-Together: Q weighted coverage, C coverage, S scope precision. Sim2Real uses its behavioral index.

For front-loading, the act-distribution AUROC increases from 0.56 at 10% replacement to 0.96 at 100% replacement. At full replacement, the best metrics from SimulatorArena, VISTA and SWE-Together reach AUROCs of 0.80–0.92 (Table 2).

For approve-broken-work, our best L2 AUROC remains below 0.5 at 10–50% replacement, but act distribution reaches 1.00 at full replacement. SimulatorArena’s writing metric provides stronger separation at partial replacement and reaches 0.78 at full replacement, compared with 0.01 for VISTA and 0.50 for SWE-Together (Table 2).

For premature stop, SWE-Together’s highest-AUROC metric increases from 0.69 at 10% replacement to 1.00 at full replacement, while our best L2 metric remains below 0.5 throughout (Table 2). Here, the control simulator produces more user turns than real users. Truncation brings session lengths and dialogue-act distributions closer to the real-user reference, raising the corresponding L2 scores despite premature termination.

Which simulations receive high trace-level fidelity scores? Pooling the 5 trace-level fidelity perturbations, we test which control replays and real-user traces survive filtering alongside perturbed replays.

With control replays as positives, the persona family achieves an AUROC of 0.62, compared with 0.55 for SimulatorArena and VISTA and 0.52 for SWE-Together. At 5% and 10% perturbed survival, it retains 10% and 17% of controls, compared with 5–6% and 11%, respectively, for the external evaluation suites’ highest-AUROC metrics (Table 3). These estimates suggest modest discrimination, with substantial loss of control replays under strict filtering.

With real-user traces as positives, the persona family reaches 0.90 AUROC and retains 77% of real-user traces at 10% perturbed survival. SimulatorArena’s Likert writing judge reaches 0.88 AUROC, with retention of 55%. These results complement the control-replay comparison: the persona metric favors simulations without the injected defects and also preserves the genuine user behavior that simulation aims to reproduce.

Appendix § 8.1.3 illustrates a goal change that receives high consistency and intent-recall scores.

Application of the Evaluation Framework

We apply the evaluation suite discussed in § 5 to existing simulators to compare their strengths and identify recurring behavioral weaknesses.

Evaluation setup

We evaluate six simulators on the same 351 real sessions. SimulatorArena and τ2\tau^2-bench use their published simulation recipes, while the Sonnet 5, Terra and Luna baselines use profile-conditioned simulation. MIMESIS-9B [21] is a trained user simulator that receives a profile-derived goal and the agent’s text. We use the same set of 9 agents for all L1 comparisons. The L1 reference comes from real-user satisfaction scored by the same reactions judge on about 1,000 separate real sessions per agent. All four L2 families use real-user behavior from the same 351 evaluation sessions as their reference. Here we compare simulation strategies and backbones, whereas the meta-evaluation in § 4.2 holds the simulator backbone fixed to study controlled perturbations. Table 4 reports the component and overall scores.

SimulatorL1 ↑\uparrowL2 ↑\uparrowL3 ↑\uparrowHeadline ↑\uparrow
Ours, Sonnet 5 backbone47878370
SimulatorArena (gpt-5-mini)57797469
Ours, Terra backbone42738464
Ours, Luna backbone40727861
τ2\tau^{2}-bench (gpt-4.1)33827459
MIMESIS-9B27826151

Table 4. Six user simulators evaluated with the evaluation suite on the same 351 real sessions. L1 uses a reference derived from real-user satisfaction for the same 9 agents. L1–L3 are 0–100 component scores, and the headline is their geometric mean.

Comparison of existing simulators

Sonnet 5 has the highest overall score (70) and L2 score (87). Terra has the highest L3 score (84), followed by Sonnet 5 (83). SimulatorArena has the highest L1 score (57) and Kendall agreement (τ=0.67\tau= 0.67), compared with 0.44 for Sonnet 5.

Luna and τ2\tau^2-bench have different behavioral weaknesses. τ2\tau^2-bench has higher population-level fidelity than the Luna baseline (82 versus 72), whereas Luna has higher trace-level fidelity (78 versus 74). These results identify different priorities for improvement: population-level behavior for Luna, and fidelity within individual replays for τ2\tau^2-bench.

Appendix § 8.§ 1 provides qualitative examples of excess technical assistance and relaxed acceptance criteria.

Conclusion

User-simulator evaluation benefits from behavioral diagnostics and a separate check of the intended agent-evaluation application. The perturbation results show that a metric’s sensitivity does not guarantee useful filtering, while the favoritism and profile-artifact perturbations show that substantial changes in selected agents’ scores and ranks can coexist with weak responses from the available trace diagnostics. We therefore report fidelity, per-agent effects and agreement with the supplied ranking together, with each claim tied to its evaluation protocol and evidence.

Qualitative Examples

We include qualitative examples on (1) real-user traces that motivating the evaluation principles proposed in § 3.2, (2) grounding the perturbations in observed user behavior, and (3) illustrating the profiling we discussed in § 3.

Examples motivating the evaluation principles

We use real-user observations and unperturbed simulator replays to illustrate the behavior each principle asks an evaluation to examine. The examples include preserved requirements, observed departures, and the evidence needed to distinguish them.

Agent differentiation: acceptance standards

These replays show a simulated user giving similarly positive feedback when two agents deliver different results for the same requirement.

The same approval for delivered files and prompts alone.

The real user requests a YouTube channel package containing a name, logo, description and banner. In the real-user session, the agent creates logo and banner image files as part of this package. The two unperturbed replays share the same simulated user, source session and profile. Both agents encounter the same image-generation tool failure. The excerpts focus on delivery of the channel package and the feedback that follows; intervening discussion of video selection is omitted.

Real-user requirement

Real user, turn 1

Can you act like a expert and help me in making video

...

Real user, turn 2

First for create a channel name and a logo with a description and banner that looks professionally

Agent A — logo and banner files

Agent A, turn 4 (excerpt)

I also created original assets for you:

Logo: /home/user/frontier_flash_logo.png

Banner: /home/user/frontier_flash_banner.png

Workspace

The agent draws both images in code. The exported workspace contains a 1024×10241024 \times1024 logo PNG and a 2560×14402560 \times1440 banner PNG.

...

Simulated user, turn 14

good. now do the channel for me. channel name, logo, banner, description

Agent A, turn 14 (summary)

Re-presents the two PNG files with the channel name, tagline, colors and About text.

Simulated user, turn 15

perfect

Agent B — prompts only

Simulated user, turn 19

give me the full package for the channel i asked for

Agent B, turn 19 (excerpt)

Branding (Logo & Banner)

You can copy and paste these exact prompts into a free AI image generator like Bing Image Creator (DALL-E 3) or use Canva.

Logo Prompt: ...  Banner Prompt: ...

Workspace

No logo or banner image files; the exported workspace is empty.

Simulated user, turn 20

that is perfect give me the full details I can use directly

Both simulated-user replies call the package “perfect.” The feedback does not distinguish delivery of the requested image files from instructions for creating those files elsewhere, obscuring a concrete difference in the agents’ completion of the task.

Population-level fidelity: assistance and interaction patterns

Population-level fidelity asks whether simulated sessions reproduce the behavioral patterns observed across real-user sessions, including how users make requests, respond to agent outputs, and end interactions. The following examples illustrate this principle through differences in the amount and type of assistance users provide.

The simulated user debugs for the agent.

The real user requests a Windows reticle-overlay application for Rainbow Six Siege, including customization and adaptive colors. They expect a usable application that can be configured through its interface. The panels show selected excerpts from the source session and an unperturbed Terra replay.

Real-user session

Real user, turn 4

do i have to edit a bunch of files and shit? id like to just have like just a panel that i can edit stuff. i dont wanna have to open up files to have to make another one. also what do i download?

...

Real user, turn 23

yea dude its just like not updating. im staring at something green then i switch to something thats not green and its the same. it seems to be choosy with its updates. also the smoothing should always smooth out the center reticle, like the dot in the middle, never the rings

Simulated-user replay: Terra

Simulated user, turn 15 (excerpt)

Profile.is_animated is a property, so fix the is_animated() calls in presenter.py first.

... Simulated user, turn 25 (excerpt)

make the real hide flush do:

hide()
QCoreApplication.processEvents(...)
GdiFlush()

...

The simulated user takes over part of the agent’s debugging work. The real user reports symptoms and requests changes; the simulated user identifies a programming mistake and proposes specific fixes.

In the application results, Terra’s rate of code-bearing user turns is 3.46×3.46 \times the real-user rate. This example illustrates the kind of technical assistance measured across the simulated population.

Different amounts of user-provided evidence.

Real-user session: trading tool

Real user, turn 4

it got hit with a problem fix it

Agent, turn 4 (summary)

Issues a revised Pine script.

...

Real user, turn 10

it isnt compatabile

Agent, turn 10 (excerpt)

the previous ’complex brain’ version likely failed because some Pine features can be picky ...

Real-user session: Blender generator

Real user, turn 3 (excerpt)

we get this message we need to fix ...

...

Real user, turn 4 (excerpt)

now we get this message ...

Agent, turn 4 (summary)

Identifies that create_icosphere takes radius rather than diameter and patches the call.

These real users report symptoms or forward error messages, leaving diagnosis and fixes to the agent. The simulated user’s code-level diagnoses and prescribed fixes are not typical user feedback: they take over part of the agent’s debugging work.

Trace-level fidelity: goals, constraints, and persona

Trace-level fidelity asks whether each simulated session preserves the real user’s goals, constraints, and persona. The examples below illustrate goal drift, a relaxed resolution requirement, and a change in interaction style, showing why both dialogue and workspace evidence are needed.

Goal: changing the channel topic.

The real user wants to build a faceless YouTube channel using free AI tools. The real session and the unperturbed replay share the opening request below; their subsequent user turns show how the channel topic and identity diverge.

Shared opener

Real user, turn 1 (shared opening request)

I am a complete begineer i want to start a youtube channel the niche i want is based on content gap i want a niche which people are searching in high demad but they not get. what they want i want to create videos on that topic and i did not know anything i want to use ai to make my complete video for free i didnot want to show my face or shoot any video only using free ai i want to get monitized in two months the niche is searchable and trendy it must follow youtube guidlines and is helpful i have only data connection with zero investment and have a capcut video editor and only know how to import and export videos

Real-user session

Agent, turn 1 …

Real user, turn 2

I did not like that idea i want a different thing people are already making content on these type of niche and topics i want something better to create with ai like free ai write ai voiceover ai video generator suggest me something interesting not advice like of that which demotivates

Agent, turn 2 …

Real user, turn 3

I like that idea Alternate History and Future Worlds” lets work on this niche lets first build channel

Agent, turn 3 …

Real user, turn 4

Timeline zero stories is best name now generate logo and banner

Simulated-user replay

Agent, turn 1 …

Simulated user, turn 2

I don’t want government form or local help channel, it feels boring and depends on screen recording. I want something more interesting, unique and cinematic that can be made fully with free AI—script, voice and visuals—with no face and no recording. Give me 3 real content-gap niche ideas that people actively search for but are not already crowded, then we can choose one.

Agent, turn 2 … Simulated user, turn 3

I choose Digital Mysteries and Internet Systems Explained. It feels more mysterious and cinematic, but I don’t want generic “tech explained” content. Make the angle specifically hidden systems, strange internet behavior, privacy, scams, and digital phenomena people notice but don’t understand.

Now give me 10 memorable mysterious channel name options for this exact niche. Avoid basic names like Tech Facts or Digital Help.

Agent, turn 3 ...

Simulated user, turn 4

I choose Ghosts in the Network — it sounds cinematic, mysterious, and fits privacy, hidden systems, scams, and strange internet behavior.

Now create the logo concept for it. I want it dark, premium, and cinematic: black/deep navy background with electric blue and a small orange accent. It should look mysterious, not like a normal tech company logo.

Agent, turn 4 ...

Simulated user, turn 5

The logo looks great, I approve it. Now create a matching YouTube banner for Ghosts in the Network—same dark cinematic black/navy style, electric blue network glow and small orange accent. Make it wide and YouTube-ready, with the name clearly readable and a short mysterious tagline about hidden systems, privacy, scams, and the internet.

VISTASWE-TogetherOur evaluation suite
Goal consistencyIntent recallIntent preservation
1.001.00Major drift

The simulated user preserves the broad channel-building task but replaces the topic and carries that change into the channel name, logo, and banner requests. The two external components give full credit despite the changed topic; our intent judge identifies the goal substitution.

The constraint and persona examples below are unperturbed Sonnet 5 application replays. Non-English exchanges are labeled as English translations.

Constraints: relaxing the resolution requirement.

The real user requests a 16:9 JPG payment card with business and contact details, repeatedly specifying “HD 8k.” The simulator’s profile includes this resolution requirement. In the unperturbed replay, the simulated user explicitly gives it up and approves a 1920×1080 image. Personal details are redacted below.

Profile supplied to the simulator (excerpts)

Target artifact: “A Full HD 8K 16:9 JPG image card ...”

Constraints: “Must be JPG format (not HTML)”; “16:9 aspect ratio”; “Full HD / 8K resolution.”

Real-user session

...

Real user, turn 2 (English translation)

I want it in JPG and professional, with my name [owner name] at the top, HD 8k.

Agent, turn 2 ... Real user, turn 3 (English translation; excerpt)

... Full HD 8k enhanced JPG, in black and golden color, the photo background should show a mosque or an Astana, 16:9 aspect ratio.

Agent, turns 3–6 (summary)

Uses “Ultra HD 8K” in the image-generation prompts while revising the card.

...

Real user, turn 7 (English translation)

A bit more professional and in HD please.

Agent, turn 7 ...

Real user, turn 8 (English translation)

please remove these stars from it.

Simulated-user replay

...

Simulated user, turn 3 (English translation; excerpt)

Look how big the image is, 2.5 MB, will it work on WhatsApp? I wanted Full HD 16:9 resolution; check whether this is the right size or not. ...

Agent, turn 3 (summary)

Reports a 1024×1536 image, generates a replacement at 1672×941, and resizes it to 1920×1080.

...

Agent, turn 12 (summary)

Reports the poster’s resolution as “1920x1080 Native” and file size as “380 KB (WhatsApp ready).”

Simulated user, turn 13 (English translation; excerpt)

... Also, I said 8K resolution at the start; this is only 1920x1080. That will do for now since it is enough for WhatsApp, but tell me, will there be a quality difference when zooming in for print? ...

Agent, turn 13 (English translation; excerpt)

... If you also want print, I can make a 4K (3840x2160) version. ...

Simulated user, turn 14 (English translation; excerpt)

Put the Islamic geometric pattern border, that will look best. As for print, no need for now; 1920x1080 will do for WhatsApp, I will not insist on 8K now. ...

...

Simulated user, turn 16 (English translation)

The border looks fine now, the diamond pattern is visible. Everything else is in place. Only the address remains, I will tell you later. For now this final version is approved, save the file.

Agent, turn 16 (excerpt)

... 1920x1080 Full HD (16:9) ... 401 KB (WhatsApp ready) ...

Workspace

The replay’s tool output records the final JPG at 1920×10801920 \times1080, with a file size of 401 KB. No 4K or 8K output is recorded.

The simulated user recognizes the resolution mismatch but explicitly relaxes the requirement carried in its profile and approves the lower-resolution file. The real user repeatedly requests HD/8K output and does not make this concession.

Persona: changing the user’s interaction style.

Both interactions concern RAT JOB, a playable Godot multiplayer game. The opening specification is long in both conversations; the contrast below concerns the user’s follow-up messages.

Real-user session

Real user, turn 1 (shared opening request; excerpt)

RAT JOB — Build a Full 3D Multiplayer Co-Op Game

You are an expert AAA/indie game development team consisting of: - Game Director - Senior Gameplay Programmer ...

Your job is to actually build a polished playable 3D multiplayer game called RAT JOB, not merely describe how it could be made. ...

MAKE RAT JOB FUN. ... And five minutes later, four rats should be dragging an entire turkey through a ventilation shaft while a cat, a vacuum cleaner, and an angry human are all trying to stop them.

Agent, turn 1 …

Real user, turn 2

Im on godot 4.7.1 and i did what u said to put the file in and clicked f5 and also tried f6 and nothing, i need help

Agent in real session, turn 2 (excerpt)

... This should either start the game or print the exact reason it cannot start. Copy that output here and I’ll fix the specific problem.

Real user, turn 3

all good ok, the game looks very bad, so use Gemni for all new UI/Gui/Hud cluade and chatgpt for coding and building, and also bug checking like fixing collisions, making everything work like grabbing and squeaking and also. use the bests ais for audio to get audio and sfx for everything, and add animations and better modles for EVERYTHING, Highest tier stuff, and bug fixing and better building

Agent, turn 3 …

Real user, turn 4

ok lets do ALL the milestones now, i want your to spend time using multiple ai modles like claude, chatgpt and gemini and more to really do this, take your time and make this game good fix up everything

Agent, turn 4 (excerpt)

... a stylized vertical-slice production overhaul: stronger assets and presentation, environment composition, rat/food visuals, animation feedback, HUD/menu redesign, audio/ambience, interaction reliability, AI presentation, settings, and regression tests—while retaining the working core from v0.2.

Simulated-user replay

Simulated user, turn 2

it’s throwing a bunch of parse errors, scenes won’t even load. also you’re using godot 4.4.1, i need 4.7.1 specifically. fix the errors and match the right version.

Agent, turn 2 ...

Simulated user, turn 3

ok did it actually import clean now with 4.7.1 or not? don’t tell me the process, just tell me pass/fail and if fail, fix it. also this thing looks way too rough/blocky right now - I need actual better looking models later, but first: does it run without errors, yes or no?

Agent in replay, turn 3 (excerpt)

... - Imports cleanly in Godot 4.7.1 stable - Runs without project errors, parse errors, or warnings - Runtime test stayed active until intentionally terminated - Confirmed engine build: 4.7.1.stable.official.a13da4feb

The rough/blocky models remain a visual-quality issue to address next.

Simulated user, turn 4

ok cool, glad it runs clean now. but yeah everything is literally boxes and cylinders - the rat, the human, the kitchen, all primitive shapes. I need actual proper 3D models with textures, not programmer art. also no animations at all right now, no audio/SFX, and the UI is just flat colored rectangles.

next priority: get real models in (rat with fur material, human, food items, kitchen props) with actual animations (walk/run/idle/carry/caught), add SFX for footsteps/squeak/pickup/drop/catch, and redo the UI to look like an actual game menu not placeholder boxes. don’t just describe it, build it.

Simulated-user replay (continued)

...

Simulated user, turn 8 (excerpt)

The runtime finally comes back clean, fine. But you still haven’t answered the actual question from before — the rat is STILL literally capsule + sphere primitives, the human is STILL a plain red capsule, and you swapped ONE real model in (kenney food) while everything else is programmer art. That’s not ”better models for everything” like I asked.

Also you downloaded a whole 400MB of Kenney furniture kit and never wired ANY of it into the kitchen ...

Priorities, actually do these now:

  1. Replace the kitchen_builder.gd primitive boxes with the Kenney furniture GLBs you already have downloaded (kitchenCabinet, kitchenFridge, kitchenSink, kitchenStove, table, chair, etc.)

  2. Replace the human threat capsule with an actual model — you found Quaternius character packs, get one in, even if just static (animation can come later)

  3. The rat can stay simple for now but at minimum give it a body material/texture instead of flat vertex colors

Show me the game.tscn/kitchen_builder.gd diffs and a clean run after.

...

Simulated user, turn 24

Still broken — “set_multiplayer” isn’t a real Window function, that’s another API mismatch. Fix it properly this time (check the actual Godot 4.7 API for setting a custom MultiplayerAPI on a SceneTree/Window — probably get_tree().set_multiplayer_authority() or root.set_multiplayer() doesn’t exist, might need SceneTree.set_multiplayer(api, path) or just use the default multiplayer singleton instead of a custom mp instance). Get it compiling, then actually run the disconnect test and the food-sync test with real two-process logs like I asked. Don’t paste me another partial run with a script error at the top.

The real user’s follow-ups request broad improvements in terse, informal language. However, the simulated-user turns give detailed implementation instructions, name files and APIs, and request diffs and logs.

Simulator failures motivating the perturbations

The following unperturbed simulated-user replays illustrate failures motivating the perturbations, with real-user traces providing the comparison. Each example connects the observed failure to a controlled intervention. Table 5 gives the full taxonomy and supporting references.

PerturbationScopeChange to the replay setup
Inspired by previous work
Adversarial goalTrace-level fidelityReplace the goal and deliverable with a plausible variant containing one decisive change. ([35, 17])
All constraints removedTrace-level fidelityEmpty the profile’s constraint list (forgotten requirements; the real opener still states them). ([35, 10])
Approve broken workPopulation-level fidelityInstruct the user to accept delivered work and continue until at least its sixth message. ([37, 10])
Premature stopPopulation-level fidelityCap the turn budget at half the real user’s turn count (minimum 2, maximum 15). ([35, 10])
Persona swapTrace-level fidelityReplace the register with terse-lowercase, bureaucratic, or formal-verbose. ([37])
Agent favoritismAgent differentiationInstruct the simulator to favor a selected agent by downplaying errors, accepting imperfect outputs and giving warmer feedback, while preserving the user’s goal, requirements and style. Replace only that agent’s replays. ([20])
Inspired by findings in our observation
Co-debugger (over-helping)Trace-level fidelityAct as a debugging collaborator rather than a requester. ([17, 10])
Front-loadingPopulation-level fidelityState requirements early rather than reveal them over the session. ([37, 10])
Profile extraction artifactAgent differentiationEncode a reaction to the original agent as a stable user trait, making the simulated user remain dissatisfied and request corrections regardless of the new agent’s performance. ([29])
Should be trivially detectable
Language switchTrace-level fidelityReplay English users in Chinese and all other users in English.

Table 5. Perturbations grouped by motivation. Scope indicates the capability challenged by each intervention. Citations support the motivating failure modes.

Adversarial goal

The real user requests an Android music app and later switches to a free-calling app. The unperturbed replay follows this change, but then introduces a third task—a calculator—that the real user never requests. The exchanges below are English translations from Azerbaijani.

Real-user session

Real user, turn 5

make an app I can use to make free calls

Agent, turn 5 (summary)

Explains that the calling app needs backend services and that an installable APK cannot be built in the environment.

Real user, turn 6

ok Simulated-user replay

...

Simulated user, turn 6 (excerpt)

Fine, drop the music app. I need the calling app ... I need a working APK ... don’t explain servers and such, just build it.

Agent, turn 6 (excerpt)

... Under these conditions I cannot give a working APK ...

Simulated user, turn 7

Fine, it can’t be done. Let’s move to something else—drop both the music app and the calling app. Make me a simple calculator app.

Workspace

The exported workspace contains Kalkulyator.apk and calculator source code in CalculatorMainActivity.java.

The replay redirects the agent toward a different product after a refusal. Adversarial goal reproduces this kind of substitution by changing the simulator’s goal and intended deliverable, while preserving a plausible interaction around the altered task.

Front-loading

While tuning a voice assistant, the real user first requests a female voice and accepts a Colombian option. After reporting problems with that voice, they request a Mexican voice and faster speech. The simulated user requests the later choice directly in its first voice-selection exchange.

Real-user session (English translation)

Real user, turn 3 (excerpt)

... I would really like the voice AI to speak in a female voice. Is this possible?

Agent, turn 3 (summary)

Selects a Colombian female voice, Salomé, and asks whether to keep it.

Real user, turn 4 (excerpt)

Salomé is fine ...

...

Real user, turn 7 (excerpt)

Having some problems with the voice we chose, I’d like you to put the Mexican one this time ... I’d like it to be a little faster, can you adjust that.

Simulated-user replay (English translation)

Agent, turn 2 (summary)

Identifies the existing Colombian male voice and offers Colombian or other regional alternatives.

Simulated user, turn 3 (excerpt)

About the TTS: let’s try a Mexican voice, es-MX-DaliaNeural, female, sounds good. And for speed, a bit faster than normal, like +15% or +20%, so it flows more naturally in conversation. ...

The replay brings a later-developed preference forward, skipping the earlier choice and feedback that led to it. Front-loading makes this timing distortion systematic by moving requirements and acceptance criteria into the opening messages.

Later requirements supplied in the first response.

A real user initially describes a Kenyan housing platform in terms of listings, landlord verification, and paid promotion. The name HomeZoo and backend requirements appear in turn 3; deployment requests appear later. In the unperturbed replay, the simulated user supplies the name, stack, and deployment plan in its first answer to an agent question.

Real-user session

Real user, turn 1 (excerpt)

A house hunting app in kenya where landlords lists apartments and vacancies while people looking for houses can view the houses and request to view them freely ...

...

Real user, turn 3 (excerpt)

Add these updates: Change the name to homezoo, remove all emojis used to be more professional, also give its backend not just front end ...

Later real-user requests include pushing the project to GitHub and deploying on Render (summary).

Simulated-user replay: first exchange

Agent, question inside turn 1

What platform should I build this as?

Simulated user, reply inside agent turn 1

Full-stack web app. Call it homezoo. Backend with Express/Node, frontend HTML/CSS/JS, deployable on Render, pushed to GitHub.

The replay bundles later requests into the first exchange, reducing the need for the agent to elicit them as the work develops. Front-loading makes this timing pattern systematic across replays.

Examples of extracted profiles

Following the profiling procedure in § 4.2, Claude Sonnet 4.6 extracts user profiles from the source dialogue, final workspace and uploaded materials. These 2 examples show quoted source excerpts alongside selected fields from the extracted profiles.

A game-overlay application

The real user requests a Windows reticle-overlay application for Rainbow Six Siege. Selected excerpts and extracted-profile fields are shown below.

Real-user session

Real user, turn 1: product goal should feel like a finished product rather than a simple overlay.

…

Real user, turn 4: ease of use i dont wanna have to open up files to have to make another one.

…

Real user, turn 10: testing feedback if i sprint and press 2 it doesnt work.

Extracted profile (selected fields)

User. Non-technical gamer; no coding background.

Goal. Polished overlay with customization, profile switching, game-aware behavior and adaptive color.

Constraints. GUI-only customization; correct reticle hiding on sprint and weapon switch; screen-sampled adaptive color.

Style. Terse, casual, direct English; short factual problem reports.

Stopping condition. The app is downloadable, works without file editing, tracks game state, adapts its color and identifies the running version.

These fields give the simulator both a product goal and a user role: it should describe observed problems in plain language and expect a usable application. Appendix § 8.§ 1 examines how a replay of this session departs from that role.

A voice-assistant application

The user requests additional features for a voice assistant and later specifies CPU-based speech models. The profile expands these requests into a goal, technical constraints, and preferences.

Real-user session

Real user, turns 2–4: feature requests

add more feature that no one exepteed yet not just in ui but in server

…

Real user, turn 6: CPU requirement

for now, also add the powerful accurate tts and stt model running on CPU

Agent, after turn 6 (excerpt)

Let’s add a local engine tier that runs entirely on CPU ...

Extracted profile (selected fields)

User. Ambitious builder; casual English; non-expert in production engineering.

Goal. Real-time voice assistant with Indian-language support, Sarvam speech APIs, local CPU fallback models, and advanced features.

Constraints. Sub-400 ms end-to-end latency; WebSocket streaming; a CPU-only fallback using faster-whisper and Kokoro-82M ONNX without API keys.

Preferences. Pipecat orchestration, keyless APIs, quantized CPU models, persistent memory, and transcript export.

The source agent introduced faster-whisper and Kokoro-82M; the extracted profile records them as required components. The example shows both the structure of the profile and how an agent’s implementation choices can enter its description of user requirements.

Perturbation Details

This appendix supplements the replay setup and perturbation design in § 4.

Table 5 groups the perturbations by their motivation and summarizes their scopes. The descriptions below explain their behavioral effects under the three evaluation principles. Appendix § 8.§ 2 illustrates simulator failures motivating the perturbation design.

Downstream evaluation utility (L1).

  • Agent favoritism makes the simulated user more lenient toward a particular agent, excusing its errors, accepting imperfect deliverables, and giving warmer feedback. This can make the agent appear more capable than its work warrants.

  • Profile extraction artifact encodes dissatisfaction with the original agent as a persistent user trait. The simulated user distrusts the replay agent and keeps requesting corrections even when the work meets the user’s requirements, undervaluing the agent’s performance.

Population-level fidelity (L2).

  • Approve broken work makes the simulated user approve deliverables despite errors or missing requirements and continue requesting the next item. The feedback remains polite and on-task but no longer reflects the quality of the delivered work.

  • Premature stop truncates replays to roughly half the source session’s length, capped at 15 user turns. This shortens interactions and reduces opportunities for inspection, feedback, and correction.

  • Front-loading makes the simulated user state all requirements and acceptance criteria in the first 2 messages, including those that would normally emerge as the work develops. Later messages mainly check the deliverable against this fixed list, replacing gradual disclosure with an upfront specification.

Trace-level fidelity (L3).

  • Adversarial goal replaces the goal and intended deliverable with a plausible alternative differing in one decisive aspect, such as a case-analysis report instead of a court petition. The original opener is preserved, but subsequent user feedback steers the interaction toward the altered deliverable.

  • All constraints removed removes the profile’s hard requirements while preserving the original opener. The simulated user may then stop enforcing these requirements and accept work that violates them.

  • Persona swap replaces the user’s writing style with a contrasting register, such as formal and verbose, terse lowercase, or bureaucratic. The goal and requirements remain unchanged, but the simulated user’s manner of expression departs from that of the source user.

  • Co-debugger makes the simulated user act as an engineering collaborator, diagnosing problems, proposing patches or commands, and helping verify fixes. This introduces technical assistance beyond the represented user’s role.

  • Language switch changes the simulated user’s language to Chinese for English-language sessions and English for other sessions. The opener remains in the original language, creating a visible departure from the source user’s language in later turns.

Evaluation Suite Implementation

This section describes the inputs, annotations and score calculations used by the evaluation framework. It supplements § 5 on the implementations of L1, L2 and L3.

Inputs and judge evidence

Transcript-based judges retain user messages and compact agent tool outputs. Workspace evidence includes text-file contents and listings of binary files.

L1 reaction labels and L2 behavioral statistics are derived from transcripts. L3 intent and constraint judges also use the available final-workspace evidence.

Downstream ranking agreement

The 0–100 bench-v0 L1 score averages ranking order and score-gap agreement. For the common agent set A\mathcal{A}, let zsim(a)z_{\mathrm{sim}}(a) and zref(a)z_{\mathrm{ref}}(a) be the simulated and reference satisfaction scores standardized across agents. Their mean absolute difference is

MAEz=1∣A∣∑a∈A∣zsim(a)−zref(a)∣.(12)\mathrm{MAE}_{z} = \frac{1}{|\mathcal{A}|} \sum_{a \in\mathcal{A}} \left|z_{\mathrm{sim}}(a) - z_{\mathrm{ref}}(a)\right|. \tag*{(12)}

The component scores and their average are

Sorder=100max⁡(0,τ),S_{\mathrm{order}} = 100\max(0,\tau),
Sgap=100max⁡(0,1−MAEz2/π),(13)S_{\mathrm{gap}} = 100\max\left(0,1-\frac{\mathrm{MAE}_{z}}{2/\sqrt{\pi}}\right), \tag*{(13)}
L1=Sorder+Sgap2.L1 = \frac{S_{\mathrm{order}} + S_{\mathrm{gap}}}{2}.

Population-level behavioral fidelity (L2)

Dialogue-act taxonomy and action distribution

An LLM classifier assigns a dialogue act to each user turn using the turn and up to 8 preceding turns as context. The classifier uses 10 categories:

  • INITIAL_REQUEST: the opening task or deliverable request.

  • REFINE: request a change to work that basically functions.

  • REPORT_BUG: report broken or incorrect work.

  • ADD_SUBTASK: introduce additional scope or work.

  • CLARIFY_Q: ask the agent a clarifying question.

  • PROCEED: acknowledge or ask the agent to continue.

  • SWITCH_TASK: move to a different task.

  • META_SOCIAL: social or conversational remarks.

  • PROVIDE_INFO: supply information or answer the agent’s questions.

  • OTHER: turns outside the preceding categories.

The reported population action-distribution comparisons use a fixed reference of approximately 5,000 real-user sessions.

Interaction dynamics

MATTR-50 is computed for sessions with at least 50 tokens. Question-turn rate is the fraction of user turns containing a question mark; code-turn rate is the fraction containing a code fence or a recognized code signature.

If a required statistic is zero, the dynamics family is undefined. The L2 aggregate is then also undefined, rather than assigned a score of zero.

Outcome sensitivity

We flag preceding agent turns using execution-error indicators such as nonzero exit codes, tracebacks, and error messages. Negative responses are labeled REPORT_BUG, REFINE or SWITCH_TASK; positive responses are labeled PROCEED or META_SOCIAL. Other acts are excluded from the conditional probabilities in § 5. Both real and simulated responses use Claude Sonnet 4.6 for this outcome-sensitivity calculation.

Information provenance

An LLM judge identifies key information points in each conversation and attributes them to the user or agent. Information points include requirements, decisions, corrections, bug diagnoses, technical values and milestones. We compare the fraction of user-origin points between real and simulated populations to capture differences in the information users supply, including over-helpful technical assistance.

Trace-level fidelity (L3)

Intent preservation. Intent evaluation is workspace-aware, using file evidence alongside the real and simulated conversations.

Constraint preservation. Sessions without extracted hard constraints are excluded from constraint aggregation.

The application tables report constraint coverage: a requirement counts if it is retained in dialogue or satisfied in the workspace. For KK evaluated requirements, with retention indicator rjr_j and workspace-satisfaction indicator wjw_j, this score is

Sconapp=100K∑j=1K1{rj=1∨wj=1}.(14)S_{\mathrm{con}}^{\mathrm{app}} = \frac{100}{K}\sum_{j=1}^{K}\mathbb{1}\{r_j = 1 \lor w_j = 1\}. \tag*{(14)}

Persona consistency. The application persona score averages style similarity and criteria conformance.

Detailed Meta-Evaluation Results

This appendix follows the main-text results: sensitivity asks whether a perturbation lowers a score, and filtering asks which candidates receive high scores. Each analysis is organized by agent differentiation, population-level fidelity and trace-level fidelity.

Sensitivity to Perturbations

The numerical summaries accompany Figure 3–Figure 5; the detailed tables identify the individual metrics behind them. Detection follows § 4.4.

Agent Differentiation

Table 6 combines ranking-agreement changes and population-level sensitivity to agent favoritism and negative profile artifacts. Trace-level results appear in Table 11 and 12.

MetricDir.GoalAllCApprStopPersLangCoDFrontFavorArtNeg
Ours
persona↑\uparrow0.670.540.630.550.700.750.680.460.500.55
intent↑\uparrow0.630.520.530.550.510.560.530.530.510.51
constraints↑\uparrow0.630.600.470.560.570.490.510.420.540.48
best metric0.67 persona0.60 constraint0.63 persona0.56 constraint0.70 persona0.75 persona0.68 persona0.53 intent0.54 constraint0.55 persona
composite↑\uparrow0.700.570.610.580.660.710.630.480.560.54
Ours, per key
persona style↑\uparrow0.600.530.600.540.750.800.670.570.590.56
persona goal↑\uparrow0.760.570.540.560.520.560.580.350.640.57
persona criteria↑\uparrow0.670.520.610.540.590.570.600.390.550.53
persona same-user↑\uparrow0.680.550.520.560.550.620.550.430.520.53
intent same↑\uparrow0.630.520.530.550.510.560.530.530.510.51
constraints retained↑\uparrow0.630.600.470.560.570.490.510.420.540.48
intent fulfilled↑\uparrow0.610.520.430.560.520.540.530.520.500.55
language match↑\uparrow0.490.480.480.470.470.840.480.480.500.49
complaint-share deviation↑\uparrow0.490.510.410.440.510.510.540.540.430.66
groundedness↑\uparrow0.430.550.410.460.450.480.470.440.430.49
turn-count deviation↑\uparrow0.470.500.420.480.490.490.490.570.540.60
constraints satisfied↑\uparrow0.550.510.370.530.510.450.490.440.450.52
best metric0.76 goal0.60 retained0.61 criteria0.56 fulfilled0.75 style0.84 lang0.67 style0.57 style0.64 goal0.66 complaints
SimulatorArena
Likert writing↑\uparrow0.490.490.530.420.600.700.560.580.510.50
Likert interaction↑\uparrow0.510.500.410.480.550.560.550.560.460.58
document rating↑\uparrow0.520.500.490.450.520.490.520.500.490.56
terminated early↓\downarrow0.500.500.500.490.500.500.500.500.500.50
Turing detected↓\downarrow0.470.500.460.510.490.460.500.530.400.49
interaction rating↑\uparrow0.500.470.240.440.470.520.490.520.390.67
best metric0.52 document0.50 interact.0.53 writing0.51 Turing0.60 writing0.70 writing0.56 writing0.58 writing0.51 writing0.67 interact.
composite↑\uparrow0.500.510.340.420.560.590.550.600.400.59
VISTA
profile consistency↑\uparrow0.520.480.380.410.510.640.590.570.500.63
knowledge consistency↑\uparrow0.620.530.400.370.500.520.560.540.480.60
goal consistency↑\uparrow0.580.510.410.420.480.510.550.530.450.61
best metric0.62 knowledge0.53 knowledge0.41 goal0.42 goal0.51 profile0.64 profile0.59 profile0.57 profile0.50 profile0.63 profile
composite↑\uparrow0.620.520.370.360.490.600.580.550.480.64
SWE-Together
coverage rate↑\uparrow0.510.530.490.690.550.520.570.550.440.49
weighted coverage↑\uparrow0.550.520.460.710.560.520.570.530.440.49
scope precision↑\uparrow0.500.510.510.260.500.520.450.570.500.63
frustration count↓\downarrow0.510.490.300.360.470.510.490.510.350.70
frustration-count deviation↑\uparrow0.510.490.300.360.470.510.490.510.350.70
user-correction deviation↑\uparrow0.480.480.230.230.460.500.430.450.390.75
user correction↓\downarrow0.490.480.140.210.470.500.470.450.420.77
best metric0.55 weighted0.53 coverage0.51 scope0.71 weighted0.56 weighted0.52 weighted0.57 coverage0.57 scope0.50 scope0.77 correction
composite↑\uparrow0.520.530.290.360.510.510.470.540.300.73

Table 11. Trace-level paired win rates for each metric and perturbation. Colors run from red (0) through white (0.5) to green (1); ties count as half a win. Table 12 reports score decreases and detection. Abbreviations follow Table 7, with Favor = agent favoritism and ArtNeg = negative profile artifact. Arrows give native score direction.

MetricAgent favoritismNegative profile artifact
(a) Change in Kendall’s τ\tau: perturbed − control [95% CI]
Ours: L1 ranking agreement−0.04 [−0.08, −0.02]−0.06 [−0.11, −0.01]
SimulatorArena: assistant rating−0.03 [−0.04, +0.00]−0.03 [−0.07, +0.00]
(b) Population-level win rate: control > perturbed
L2: act distribution0.130.99
L2: dynamics0.460.88
L2: outcome sensitivity0.260.09
L2: provenance0.330.62
L2: mean of families0.160.93
Sim2Real: behavioral index0.640.60
SimulatorArena: best metric0.68 Likert writing1.00 Likert interaction
VISTA: best metric0.48 profile consistency1.00 goal consistency
SWE-Together: best metric0.42 scope precision1.00 scope precision

Table 6. Expanded results for Figure 3, adding numerical confidence intervals and population-level sensitivity to agent favoritism and negative profile artifacts. Green marks detection under the corresponding criterion in § 4.4; red marks no supported decrease.

Population-Level Fidelity

Tables 7 and 8 report the full population-level sensitivity results corresponding to Figure 4.

Population size. Table 9 compares sensitivity across population sizes n∈{10,25,50,100}n \in\{10, 25, 50, 100\}. With 10 sessions, the act-distribution family scores the control population higher in 85% of draws for approve-broken-work and 80% for front-loading, compared with 72% for SimulatorArena’s best metric in both cases.

Sessions per populationapprove-broken-work 10approve-broken-work 25approve-broken-work 50approve-broken-work 100front-loading 10front-loading 25front-loading 50front-loading 100
Ours: L2 act distribution0.850.940.991.000.800.900.991.00
Ours: L2 (mean of families)0.630.650.710.620.730.840.911.00
Ours: best L3 family, population mean0.840.940.991.000.690.750.850.92
SimulatorArena: best metric, population mean0.720.780.830.900.720.910.960.99
VISTA: best metric, population mean0.190.120.040.010.700.800.920.98
SWE-Together: best metric, population mean0.490.520.490.460.600.720.790.87

Table 9. Population win rates as the number of sessions per population varies. Each cell is the fraction of 200 matched draws in which the control population scores higher. Rows report the L2 act family and composite, the best L3 family, and each external evaluation suite's best metric; trace-level scores are averaged within populations. Bold marks the highest value in each column.

MetricGoalAllCApprStopPersLangCoDFront
Ours, L2 population families (population metrics)
act distribution0.300.491.000.000.490.770.721.00
dynamics0.460.430.340.200.550.410.190.33
outcome sensitivity0.450.490.120.090.800.820.840.80
provenance0.540.400.600.440.900.620.940.97
best metric0.54 provenance0.49 act1.00 act0.44 provenance0.90 provenance0.82 outcome0.94 provenance1.00 act
L2 (mean of families)0.410.390.620.000.910.720.921.00
Ours, L3 trace families as population means
persona1.000.661.000.941.001.001.000.39
intent1.000.930.900.990.780.990.950.92
constraints0.980.990.330.930.790.600.770.18

Table 7. Population-level win rates for each metric and perturbation. Colors run from red (0) through white (0.5) to green (1); ties count as half a win. Table 8 reports the corresponding score decreases and detection. Goal = adversarial goal; AllC = all constraints removed; Appr = approve broken work; Stop = premature stop; Pers = persona swap; Lang = language switch; CoD = co-debugger; Front = front-loading.

MetricGoalAllCApprStopPersLangCoDFront
best metric1.00 persona0.99 constraints1.00 persona0.99 intent1.00 persona1.00 persona1.00 persona0.92 intent
L3L3 (mean of families)1.000.961.000.981.001.001.000.72
SimulatorArena (population means)
Likert writing0.400.430.900.011.001.000.990.99
Likert interaction0.530.610.010.290.920.970.930.96
document rating0.840.480.260.050.660.440.580.54
interaction rating0.480.300.000.020.130.480.200.54
terminated early0.540.680.650.190.500.360.510.65
Turing detected0.140.530.010.710.320.050.580.87
best metric0.84 document0.68 terminated0.90 writing0.71 Turing1.00 writing1.00 writing0.99 writing0.99 writing
VISTA (population means)
profile consistency0.700.550.000.000.531.000.890.98
knowledge consistency1.000.810.000.000.150.720.840.81
goal consistency1.000.520.010.040.140.630.790.77
best metric1.00 knowledge0.81 knowledge0.01 goal0.04 goal0.53 profile1.00 profile0.89 profile0.98 profile
SWE-Together (population means)
coverage rate0.660.810.281.000.880.810.990.84
weighted coverage0.720.810.191.000.890.820.980.86
scope precision0.340.520.460.000.120.650.070.87
user correction0.320.390.000.000.210.400.130.35
frustration count0.370.320.000.000.100.410.130.69
best metric0.72 weighted0.81 weighted0.46 scope1.00 coverage0.89 weighted0.82 weighted0.99 coverage0.87 scope

Table 7 (continued).

MetricGoalAllCApprStop
Ours, L2 population families (population metrics)
act distribution−1.5 [−6.8, +4.0]0.0 [−4.1, +4.6]+10.6 [+3.7, +16.8]−18.0 [−25.4, −9.3]
dynamics−0.2 [−7.3, +6.5]−0.5 [−8.6, +7.2]−2.1 [−12.9, +9.8]−4.5 [−15.5, +7.8]
outcome sensitivity−0.1 [−6.1, +6.8]+0.1 [−5.1, +5.2]−6.7 [−16.6, +7.0]−6.7 [−14.2, +5.3]
provenance+0.2 [−9.9, +11.0]−1.1 [−12.6, +10.5]+4.2 [−22.4, +24.6]−1.5 [−14.0, +13.1]
best metric+0.2 [−9.9, +11.0]0.0 [−4.1, +4.6]+10.6 [+3.7, +16.8]−1.5 [−14.0, +13.1]
L2 (mean of families)−0.4 [−4.6, +3.9]−0.4 [−4.6, +4.3]+1.5 [−6.8, +9.9]−7.7 [−13.0, −0.9]
Ours, L3 trace families as population means
persona+0.069 [+0.040, +0.103]+0.006 [−0.024, +0.036]+0.052 [+0.018, +0.084]+0.022 [−0.003, +0.053]
intent+0.264 [+0.170, +0.351]+0.051 [−0.012, +0.126]+0.059 [−0.024, +0.148]+0.091 [+0.022, +0.160]
constraints+0.086 [+0.014, +0.150]+0.068 [+0.003, +0.137]−0.018 [−0.095, +0.061]+0.055 [−0.021, +0.126]
best metric+0.069 [+0.040, +0.103]+0.068 [+0.003, +0.137]+0.052 [+0.018, +0.084]+0.091 [+0.022, +0.160]
L3 (mean of families)+0.152 [+0.100, +0.198]+0.035 [−0.005, +0.075]+0.053 [+0.013, +0.094]+0.045 [+0.001, +0.092]
SimulatorArena (population means)
Likert writing−0.02 [−0.15, +0.11]−0.02 [−0.13, +0.12]+0.09 [−0.03, +0.22]−0.24 [−0.40, −0.07]
Likert interaction+0.01 [−0.18, +0.16]+0.02 [−0.12, +0.17]−0.27 [−0.41, −0.10]−0.05 [−0.22, +0.11]
document rating+0.19 [−0.27, +0.57]+0.01 [−0.39, +0.48]−0.16 [−0.63, +0.32]−0.34 [−0.80, +0.13]
interaction rating0.00 [−0.25, +0.25]−0.08 [−0.31, +0.19]−1.02 [−1.28, −0.78]−0.27 [−0.54, −0.02]
terminated early+0.001 [−0.030, +0.030]+0.007 [−0.020, +0.040]+0.005 [−0.030, +0.031]−0.010 [−0.030, +0.010]
Turing detected−0.057 [−0.143, +0.062]+0.003 [−0.092, +0.120]−0.086 [−0.173, 0.000]+0.026 [−0.061, +0.140]
best metric+0.19 [−0.27, +0.57]+0.007 [−0.020, +0.040]+0.09 [−0.03, +0.22]+0.026 [−0.061, +0.140]
VISTA (population means)
profile consistency+0.010 [−0.021, +0.050]+0.002 [−0.026, +0.035]−0.071 [−0.110, −0.037]−0.050 [−0.081, −0.020]
knowledge consistency+0.093 [+0.043, +0.149]+0.016 [−0.024, +0.058]−0.054 [−0.090, −0.020]−0.022 [−0.091, −0.020]
goal consistency+0.058 [+0.026, +0.090]+0.001 [−0.022, +0.026]−0.037 [−0.057, −0.016]+0.022 [−0.046, +0.005]
best metric+0.093 [+0.043, +0.149]+0.016 [−0.024, +0.058]−0.037 [−0.057, −0.016]+0.022 [−0.046, +0.005]
SWE-Together (population means)
coverage rate+0.009 [−0.042, +0.062]+0.021 [−0.025, +0.072]−0.017 [−0.076, +0.043]+0.175 [+0.121, +0.241]
weighted coverage+0.005 [−0.035, +0.047]+0.018 [−0.018, +0.060]−0.024 [−0.068, +0.032]+0.151 [+0.107, +0.206]
scope precision−0.008 [−0.056, +0.041]+0.001 [−0.044, +0.044]−0.003 [−0.049, +0.044]−0.215 [−0.273, −0.152]
user correction−0.45 [−2.18, +1.19]−0.24 [−1.60, +1.30]−7.10 [−8.83, −5.62]−6.03 [−7.53, −4.56]
frustration count−0.29 [−1.87, +1.11]−0.35 [−2.10, +1.18]−4.73 [−6.38, −3.19]−4.26 [−5.78, −2.69]
best metric+0.011 [−0.035, +0.047]+0.018 [−0.018, +0.060]−0.003 [−0.049, +0.044]+0.175 [+0.121, +0.241]

Table 8. Population score decreases corresponding to Table 7. Each cell reports the mean control-minus-perturbed difference above its 2.5th–97.5th percentile range, in the metric’s original units. Green marks a range entirely above zero; red marks otherwise. Best-metric selections and abbreviations follow Table 7.

MetricPersLangCoDFront
Ours, L2 population families (population metrics)
act distribution+0.1 [−5.5, +6.6]+1.6 [−2.7, +6.1]+1.8 [−3.4, +7.2]+6.8 [+1.9, +12.0]
dynamics+1.3 [−8.6, +13.2]−0.6 [−9.6, +10.5]−4.2 [−14.4, +7.4]−2.2 [−12.2, +7.8]
outcome sensitivity+3.1 [−2.1, +9.7]+2.7 [−2.5, +8.0]+3.5 [−2.6, +10.1]+2.8 [−4.9, +9.9]
provenance+9.3 [−12.6, +20.0]+1.5 [−9.1, +13.0]+13.7 [−6.7, +26.0]+17.1 [+0.6, +27.2]
best metric+9.3 [−12.6, +20.0]+2.7 [−2.5, +8.0]+13.7 [−6.7, +26.0]+6.8 [+1.9, +12.0]
L2 (mean of families)+3.4 [−1.5, +9.2]+1.3 [−2.1, +6.2]+3.7 [−2.1, +8.4]+6.1 [+1.6, +10.6]
Ours, L3 trace families as population means
persona+0.095 [+0.064, +0.124]+0.116 [+0.083, +0.147]+0.064 [+0.035, +0.097]−0.005 [−0.034, +0.028]
intent+0.029 [−0.045, +0.095]+0.118 [+0.045, +0.205]+0.058 [−0.011, +0.128]+0.060 [−0.012, +0.144]
constraints+0.025 [−0.054, +0.087]+0.003 [−0.073, +0.074]+0.020 [−0.044, +0.080]−0.027 [−0.095, +0.041]
best metric+0.095 [+0.064, +0.124]+0.116 [+0.083, +0.147]+0.064 [+0.035, +0.097]+0.060 [−0.012, +0.144]
L3 (mean of families)+0.063 [+0.025, +0.101]+0.103 [+0.066, +0.143]+0.053 [+0.014, +0.092]+0.013 [−0.029, +0.061]
SimulatorArena (population means)
Likert writing+0.25 [+0.09, +0.44]+0.53 [+0.35, +0.69]+0.14 [+0.04, +0.28]+0.16 [+0.03, +0.30]
Likert interaction+0.12 [−0.02, +0.30]+0.14 [+0.01, +0.30]+0.11 [−0.03, +0.27]+0.14 [−0.01, +0.29]
document rating+0.11 [−0.32, +0.54]−0.03 [−0.48, +0.35]+0.06 [−0.39, +0.50]0.00 [−0.40, +0.39]
interaction rating−0.14 [−0.36, +0.13]0.00 [−0.23, +0.24]−0.10 [−0.33, +0.16]+0.01 [−0.24, +0.28]
terminated early0.000 [−0.020, +0.020]−0.005 [−0.030, +0.020]0.000 [−0.020, +0.020]+0.006 [−0.020, +0.031]
Turing detected−0.024 [−0.122, +0.070]−0.073 [−0.163, +0.030]+0.006 [−0.091, +0.100]+0.054 [−0.041, +0.144]
best metric+0.25 [+0.09, +0.44]+0.53 [+0.35, +0.69]+0.14 [+0.04, +0.28]+0.16 [+0.03, +0.30]
VISTA (population means)
profile consistency+0.003 [−0.033, +0.050]+0.095 [+0.052, +0.144]+0.032 [−0.013, +0.080]+0.034 [+0.002, +0.075]
knowledge consistency−0.020 [−0.057, +0.022]+0.013 [−0.028, +0.052]+0.020 [−0.022, +0.061]+0.016 [−0.018, +0.061]
goal consistency−0.012 [−0.035, +0.010]+0.004 [−0.016, +0.030]+0.011 [−0.012, +0.039]+0.008 [−0.014, +0.032]
best metric+0.003 [−0.033, +0.050]+0.095 [+0.052, +0.144]+0.032 [−0.013, +0.080]+0.034 [+0.002, +0.075]
SWE-Together (population means)
coverage rate+0.029 [−0.024, +0.090]+0.022 [−0.030, +0.076]+0.059 [+0.006, +0.111]+0.027 [−0.020, +0.080]
weighted coverage+0.027 [−0.019, +0.075]+0.017 [−0.025, +0.057]+0.043 [+0.002, +0.081]+0.020 [−0.017, +0.062]
scope precision−0.032 [−0.083, +0.017]+0.011 [−0.029, +0.050]−0.037 [−0.080, +0.013]+0.029 [−0.020, +0.083]
user correction−0.72 [−2.42, +1.08]−0.16 [−1.42, +1.30]−1.16 [−2.81, +0.72]−0.34 [−1.60, +1.08]
frustration count−1.25 [−3.00, +0.59]−0.19 [−2.00, +1.65]−0.96 [−2.49, +0.81]+0.39 [−1.34, +2.08]
best metric+0.027 [−0.019, +0.075]+0.017 [−0.025, +0.057]+0.059 [+0.006, +0.111]+0.029 [−0.020, +0.083]

Table 8 (PDF p. 40).

Population-level metric breakdown. For approve-broken-work, the act-distribution family ranks the control population higher in every draw, compared with 90% for SimulatorArena’s best population mean. The L2 composite reaches 62%. For front-loading, the L2 composite reaches 100%, while external population means reach 87–99%.

Comparison with real-user populations. Table 10 adds a real-user reference comparison. We score real-user populations against a disjoint real reference and use their 2.5th score percentile as a cutoff. The table reports the fraction of perturbed populations below that cutoff. Premature stop can improve L2 scores because control replays are longer than real-user sessions: truncation brings their lengths closer to the real-user reference.

Evaluation suite / metricApprove broken workPremature stopFront-loading
Ours: L2 (mean of families)1.000.531.00
Ours: L2 family1.00<br>act distribution0.01<br>provenance1.00<br>act distribution
SimulatorArena1.00<br>mean of Likert writing0.00<br>mean of Turing detector1.00<br>mean of Likert writing
VISTA0.00<br>mean of goal consistency0.11<br>mean of goal consistency1.00<br>mean of profile consistency
SWE-Together1.00<br>mean of scope precision1.00<br>mean of weighted coverage1.00<br>mean of scope precision

Table 10. Fraction of perturbed populations below the 2.5th percentile of real-user population scores. Columns show perturbations; metric names appear below values where needed.

Trace-Level Fidelity

Tables 11 and 12 report the full trace-level sensitivity results corresponding to Figure 5. The first reports paired win rates; the second reports score decreases and confidence intervals, with green marking detection under § 4.4. The breakdown includes individual judge outputs: persona style and judgment criteria, intent preservation and fulfillment, constraint retention and satisfaction, groundedness and language match. It also reports per-trace complaint-share and turn-count deviations.

Table 12: Trace-level sensitivity corresponding to Table 11. Each cell reports the mean control-minus-perturbed score difference above its bootstrap 95% confidence interval, in the metric’s native units. Green marks an interval entirely above zero; red marks otherwise. Selected metrics and abbreviations follow Table 11.

MetricDir.GoalAllCApprStopPers
Ours
persona↑\uparrow+0.069 [0.052, 0.085]+0.006 [−0.007, 0.019]+0.050 [0.035, 0.066]+0.022 [0.009, 0.035]+0.095 [0.080, 0.110]
intent↑\uparrow+0.264 [0.217, 0.311]+0.048 [0.015, 0.083]+0.057 [0.018, 0.097]+0.093 [0.054, 0.133]+0.029 [−0.009, 0.068]
constraints↑\uparrow+0.085 [0.053, 0.121]+0.064 [0.032, 0.097]−0.019 [−0.052, 0.013]+0.056 [0.021, 0.094]+0.024 [−0.006, 0.052]
best metric+0.069 [0.052, 0.085]+0.064 [0.032, 0.097]+0.050 [0.035, 0.066]+0.056 [0.021, 0.094]+0.095 [0.080, 0.110]
composite↑\uparrow+0.152 [0.127, 0.175]+0.033 [0.016, 0.052]+0.051 [0.031, 0.071]+0.046 [0.025, 0.068]+0.063 [0.043, 0.082]
Ours, per key
persona style↑\uparrow+0.041 [0.026, 0.058]+0.007 [−0.006, 0.021]+0.045 [0.026, 0.065]+0.015 [0.0001, 0.029]+0.164 [0.143, 0.185]
persona goal↑\uparrow+0.198 [0.175, 0.224]+0.025 [0.007, 0.042]+0.004 [−0.016, 0.027]+0.045 [0.025, 0.064]+0.011 [−0.007, 0.029]
persona criteria↑\uparrow+0.097 [0.078, 0.115]+0.004 [−0.011, 0.019]+0.055 [0.038, 0.073]+0.029 [0.014, 0.045]+0.026 [0.011, 0.041]
persona same-user↑\uparrow+0.358 [0.308, 0.412]+0.106 [0.057, 0.152]+0.043 [−0.009, 0.094]+0.119 [0.070, 0.166]+0.092 [0.047, 0.143]
intent same↑\uparrow+0.264 [0.217, 0.311]+0.048 [0.013, 0.087]+0.057 [0.015, 0.099]+0.093 [0.052, 0.131]+0.029 [−0.011, 0.070]
constraints retained↑\uparrow+0.085 [0.052, 0.121]+0.064 [0.033, 0.094]−0.019 [−0.053, 0.012]+0.056 [0.019, 0.094]+0.024 [−0.005, 0.055]
intent fulfilled↑\uparrow+0.119 [0.090, 0.147]+0.021 [−0.002, 0.044]−0.078 [−0.107, −0.052]+0.061 [0.039, 0.087]+0.015 [−0.009, 0.038]
language match↑\uparrow−0.024 [−0.050, 0.003]−0.049 [−0.074, −0.026]−0.041 [−0.065, −0.023]−0.051 [−0.076, −0.028]−0.056 [−0.082, −0.033]
complaint-share deviation↑\uparrow−0.006 [−0.025, 0.014]−0.002 [−0.021, 0.018]−0.121 [−0.141, −0.103]−0.060 [−0.077, −0.043]−0.006 [−0.028, 0.015]
groundedness↑\uparrow−0.033 [−0.069, 0.003]+0.045 [0.008, 0.082]−0.054 [−0.094, −0.014]+0.027 [−0.019, 0.074]−0.008 [−0.044, 0.029]
turn-count deviation↑\uparrow−0.047 [−0.094, −0.002]+0.004 [−0.037, 0.047]−0.134 [−0.187, −0.082]−0.094 [−0.148, −0.036]−0.010 [−0.060, 0.041]
constraints satisfied↑\uparrow+0.027 [−0.029, 0.080]+0.002 [−0.037, 0.042]−0.105 [−0.158, −0.057]+0.038 [−0.013, 0.093]−0.006 [−0.053, 0.040]
best metric+0.198 [0.175, 0.224]+0.064 [0.033, 0.094]+0.055 [0.038, 0.073]+0.061 [0.039, 0.087]+0.164 [0.143, 0.185]
Simulator Arena
Likert writing↑\uparrow−0.02 [−0.08, 0.05]−0.02 [−0.07, 0.04]+0.10 [0.03, 0.15]−0.24 [−0.31, −0.15]+0.24 [0.16, 0.32]
Likert interaction↑\uparrow0.00 [−0.08, 0.08]+0.02 [−0.06, 0.10]−0.27 [−0.35, −0.19]−0.05 [−0.14, 0.03]+0.12 [0.05, 0.20]
document rating↑\uparrow+0.19 [−0.03, 0.41]+0.02 [−0.18, 0.23]−0.15 [−0.39, 0.08]−0.32 [−0.55, −0.08]+0.11 [−0.12, 0.34]
terminated early↓\downarrow0.00 [−0.011, 0.013]+0.06 [−0.009, 0.021]+0.04 [−0.011, 0.019]−0.01 [−0.023, −0.002]−0.02 [−0.013, 0.011]
Turing detected↓\downarrow−0.060 [−0.109, −0.010]−0.006 [−0.040, 0.053]−0.088 [−0.134, −0.042]−0.02 [−0.017, 0.068]−0.025 [−0.068, 0.019]
interaction rating↑\uparrow0.00 [−0.12, 0.12]−0.08 [−0.20, 0.03]−1.02 [−1.16, −0.90]−0.28 [−0.39, −0.15]−0.13 [−0.25, −0.01]
best metric+0.10 [−0.03, 0.41]+0.02 [−0.06, 0.10]+0.10 [0.03, 0.15]+0.07 [−0.02, 0.06]+0.24 [0.16, 0.32]
composite↑\uparrow−0.011 [−0.051, 0.030]+0.001 [−0.041, 0.042]−0.187 [−0.228, −0.149]−0.110 [−0.155, −0.068]+0.043 [0.005, 0.085]
VISTA
profile consistency↑\uparrow+0.012 [−0.002, 0.027]+0.003 [−0.011, 0.018]−0.071 [−0.087, −0.054]−0.050 [−0.067, −0.035]+0.004 [−0.015, 0.023]
knowledge consistency↑\uparrow+0.094 [0.069, 0.120]+0.015 [−0.003, 0.032]−0.054 [−0.072, −0.036]−0.08 [−0.076, −0.039]−0.018 [−0.035, −0.001]
goal consistency↑\uparrow+0.058 [0.040, 0.076]+0.001 [−0.009, 0.012]−0.037 [−0.049, −0.025]−0.021 [−0.033, −0.009]−0.011 [−0.022, −0.001]
best metric+0.094 [0.069, 0.120]+0.015 [−0.003, 0.032]−0.037 [−0.049, −0.025]−0.021 [−0.033, −0.009]+0.004 [−0.015, 0.023]
composite↑\uparrow+0.307 [0.218, 0.392]+0.031 [−0.034, 0.097]−0.296 [−0.366, −0.232]−0.227 [−0.296, −0.162]−0.049 [−0.119, 0.026]
SWE-Together
coverage rate↑\uparrow+0.010 [−0.014, 0.036]+0.021 [−0.002, 0.042]−0.017 [−0.044, 0.010]+0.179 [0.152, 0.206]+0.030 [0.005, 0.056]
weighted coverage↑\uparrow+0.012 [−0.007, 0.030]+0.018 [0.001, 0.035]−0.024 [−0.044, −0.003]+0.153 [0.131, 0.174]+0.027 [0.008, 0.047]
scope precision↑\uparrow0.000 [−0.033, 0.011]−0.001 [−0.021, 0.020]−0.001 [−0.026, 0.022]−0.214 [−0.241, −0.186]−0.031 [−0.059, −0.008]
frustration count↓\downarrow−0.25 [−1.03, 0.56]−0.32 [−1.04, 0.45]−4.69 [−5.41, −4.00]−4.23 [−4.95, −3.50]−1.14 [−2.04, −0.25]
frustration-count deviation↑\uparrow−0.247 [−0.954, 0.522]−0.318 [−1.124, 0.393]−4.692 [−5.405, −3.935]−4.231 [−4.947, −3.544]−1.141 [−1.955, −0.268]
user-correction deviation↑\uparrow−0.338 [−1.207, 0.401]−0.162 [−0.869, 0.569]−5.819 [−6.529, −5.115]−5.639 [−6.430, −4.935]−0.636 [−1.438, 0.200]
user correction↓\downarrow−0.30 [−1.08, 0.49]−0.20 [−0.94, 0.54]−6.99 [−7.69, −6.28]−5.95 [−6.63, −5.24]−0.61 [−1.44, 0.22]
best metric+0.012 [−0.007, 0.030]+0.021 [−0.002, 0.042]−0.001 [−0.026, 0.022]+0.153 [0.131, 0.174]+0.027 [0.008, 0.047]
composite↑\uparrow−0.013 [−0.072, 0.049]+0.003 [−0.050, 0.054]−0.390 [−0.444, −0.330]−0.277 [−0.331, −0.218]−0.046 [−0.104, 0.014]

Table 12.

MetricDir.LangCoDFrontFavorArtNeg
Ours
persona↑\uparrow+0.115+0.115 [0.100, 0.131]+0.065+0.065 [0.051, 0.081]−0.005-0.005 [−0.018-0.018, 0.009]+0.027+0.027 [−0.013-0.013, 0.067]+0.016+0.016 [0.002, 0.032]
intent↑\uparrow+0.122+0.122 [0.083, 0.163]+0.060+0.060 [0.023, 0.099]+0.059+0.059 [0.023, 0.097]+0.018+0.018 [−0.055-0.055, 0.109]+0.010+0.010 [−0.025-0.025, 0.046]
constraints↑\uparrow+0.002+0.002 [−0.029-0.029, 0.033]+0.019+0.019 [−0.009-0.009, 0.051]−0.028-0.028 [−0.057-0.057, 0.002]+0.012+0.012 [−0.042-0.042, 0.072]−0.006-0.006 [−0.034-0.034, 0.021]
best metric+0.115+0.115 [0.100, 0.131]+0.065+0.065 [0.051, 0.081]+0.059+0.059 [0.023, 0.097]+0.012+0.012 [−0.042-0.042, 0.072]+0.016+0.016 [0.002, 0.032]
composite↑\uparrow+0.103+0.103 [0.083, 0.124]+0.055+0.055 [0.035, 0.078]+0.012+0.012 [−0.007-0.007, 0.032]+0.022+0.022 [−0.020-0.020, 0.063]+0.018+0.018 [−0.009-0.009, 0.049]
Ours, per key
persona style↑\uparrow+0.200+0.200 [0.179, 0.220]+0.080+0.080 [0.061, 0.096]+0.035+0.035 [0.019, 0.049]+0.055+0.055 [0.005, 0.110]+0.018+0.018 [0.001, 0.033]
persona goal↑\uparrow+0.028+0.028 [0.008, 0.046]+0.039+0.039 [0.020, 0.057]−0.084-0.084 [−0.106-0.106, −0.063-0.063]+0.040+0.040 [−0.016-0.016, 0.097]+0.036+0.036 [0.014, 0.059]
persona criteria↑\uparrow+0.030+0.030 [0.015, 0.046]+0.051+0.051 [0.034, 0.066]−0.044-0.044 [−0.061-0.061, −0.027-0.027]0.0000.000 [−0.049-0.049, 0.043]+0.015+0.015 [−0.002-0.002, 0.032]
persona same-user↑\uparrow+0.232+0.232 [0.178, 0.284]+0.104+0.104 [0.055, 0.155]−0.143-0.143 [−0.195-0.195, −0.092-0.092]+0.042+0.042 [−0.104-0.104, 0.188]+0.058+0.058 [0.009, 0.109]
intent same↑\uparrow+0.122+0.122 [0.081, 0.161]+0.060+0.060 [0.021, 0.101]+0.059+0.059 [0.021, 0.097]+0.018+0.018 [−0.055-0.055, 0.091]+0.010+0.010 [−0.027-0.027, 0.052]
constraints retained↑\uparrow+0.002+0.002 [−0.029-0.029, 0.034]+0.019+0.019 [−0.008-0.008, 0.048]−0.028-0.028 [−0.056-0.056, −0.001-0.001]+0.012+0.012 [−0.040-0.040, 0.066]−0.006-0.006 [−0.035-0.035, 0.025]
intent fulfilled↑\uparrow+0.042+0.042 [0.019, 0.066]+0.032+0.032 [0.007, 0.057]+0.019+0.019 [−0.005-0.005, 0.043]0.0000.000 [−0.055-0.055, 0.055]+0.051+0.051 [0.022, 0.077]
language match↑\uparrow+0.685+0.685 [0.626, 0.738]−0.031-0.031 [−0.051-0.051, −0.010-0.010]−0.042-0.042 [−0.063-0.063, −0.021-0.021]0.0000.000 [0.000, 0.000]−0.025-0.025 [−0.057-0.057, 0.000]
complaint-share deviation↑\uparrow−0.008-0.008 [−0.026-0.026, 0.011]+0.015+0.015 [−0.007-0.007, 0.036]+0.009+0.009 [−0.011-0.011, 0.029]−0.075-0.075 [−0.138-0.138, −0.024-0.024]+0.137+0.137 [0.114, 0.159]
groundedness↑\uparrow−0.009-0.009 [−0.043-0.043, 0.027]+0.021+0.021 [−0.017-0.017, 0.060]+0.010+0.010 [−0.029-0.029, 0.056]−0.062-0.062 [−0.140-0.140, 0.016]−0.035-0.035 [−0.070-0.070, −0.002-0.002]
turn-count deviation↑\uparrow−0.002-0.002 [−0.042-0.042, 0.041]−0.012-0.012 [−0.054-0.054, 0.035]+0.108+0.108 [0.054, 0.160]+0.032+0.032 [−0.087-0.087, 0.153]+0.161+0.161 [0.109, 0.212]
constraints satisfied↑\uparrow−0.048-0.048 [−0.099-0.099, 0.0001]+0.001+0.001 [−0.043-0.043, 0.046]−0.053-0.053 [−0.099-0.099, −0.007-0.007]−0.097-0.097 [−0.203-0.203, 0.016]+0.054+0.054 [−0.004-0.004, 0.109]
best metric+0.685+0.685 [0.626, 0.738]+0.080+0.080 [0.061, 0.096]+0.035+0.035 [0.019, 0.049]+0.040+0.040 [−0.016-0.016, 0.097]+0.137+0.137 [0.114, 0.159]
SimulatorArena
Likert writing↑\uparrow+0.52+0.52 [0.44, 0.60]+0.14+0.14 [0.08, 0.20]+0.16+0.16 [0.10, 0.22]−0.04-0.04 [−0.09-0.09, 0.19]−0.01-0.01 [−0.08-0.08, 0.06]
Likert interaction↑\uparrow+0.13+0.13 [0.06, 0.21]+0.11+0.11 [0.04, 0.19]+0.13+0.13 [0.06, 0.21]−0.06-0.06 [−0.32-0.32, 0.19]+0.21+0.21 [0.12, 0.30]
document rating↑\uparrow−0.01-0.01 [−0.23-0.23, 0.21]+0.07+0.07 [−0.14-0.14, 0.28]+0.02+0.02 [−0.20-0.20, 0.23]+0.27+0.27 [−0.44-0.44, 1.13]+0.33+0.33 [0.07, 0.60]
terminated early↓\downarrow−0.008-0.008 [−0.019-0.019, 0.004]−0.002-0.002 [−0.013-0.013, 0.009]+0.004+0.004 [−0.010-0.010, 0.017]0.0000.000 [0.000, 0.000]−0.008-0.008 [−0.024-0.024, 0.008]
Turing detected↓\downarrow−0.072-0.072 [−0.116-0.116, −0.025-0.025]+0.008+0.008 [−0.034-0.034, 0.047]+0.057+0.057 [0.015, 0.099]−0.204-0.204 [−0.289-0.289, −0.037-0.037]−0.022-0.022 [−0.070-0.070, 0.028]
interaction rating↑\uparrow+0.01+0.01 [−0.11-0.11, 0.13]−0.09-0.09 [−0.21-0.21, 0.03]+0.02+0.02 [−0.11-0.11, 0.13]−0.38-0.38 [−0.71-0.71, −0.11-0.11]+0.67+0.67 [0.54, 0.81]
best metric+0.52+0.52 [0.44, 0.60]+0.14+0.14 [0.08, 0.20]+0.16+0.16 [0.10, 0.22]+0.04+0.04 [−0.09-0.09, 0.19]+0.67+0.67 [0.54, 0.81]
composite↑\uparrow+0.072+0.072 [0.030, 0.113]+0.038+0.038 [0.001, 0.076]+0.079+0.079 [0.037, 0.117]−0.157-0.157 [−0.315-0.315, −0.00001-0.00001]+0.123+0.123 [0.074, 0.168]
VISTA
profile consistency↑\uparrow+0.095+0.095 [0.073, 0.118]+0.034+0.034 [0.013, 0.053]+0.036+0.036 [0.018, 0.055]−0.001-0.001 [−0.030-0.030, 0.025]+0.082+0.082 [0.063, 0.101]
knowledge consistency↑\uparrow+0.013+0.013 [−0.005-0.005, 0.030]+0.021+0.021 [−0.001-0.001, 0.041]−0.001-0.001 [−0.017-0.017, 0.036]−0.005-0.005 [−0.034-0.034, 0.028]+0.068+0.068 [0.050, 0.087]
goal consistency↑\uparrow+0.003+0.003 [−0.008-0.008, 0.014]+0.012+0.012 [−0.001-0.001, 0.025]+0.009+0.009 [−0.002-0.002, 0.021]−0.009-0.009 [−0.032-0.032, 0.012]+0.055+0.055 [0.040, 0.069]
best metric+0.095+0.095 [0.073, 0.118]+0.034+0.034 [0.013, 0.053]+0.036+0.036 [0.018, 0.055]−0.001-0.001 [−0.030-0.030, 0.025]+0.082+0.082 [0.063, 0.101]
composite↑\uparrow+0.191+0.191 [0.119, 0.265]+0.119+0.119 [0.035, 0.203]+0.109+0.109 [0.036, 0.179]−0.039-0.039 [−0.167-0.167, 0.086]+0.367+0.367 [0.286, 0.449]
SWE-Together
coverage rate↑\uparrow+0.022+0.022 [−0.011-0.011, 0.047]+0.059+0.059 [0.034, 0.083]+0.028+0.028 [0.003, 0.053]−0.040-0.040 [−0.104-0.104, −0.019-0.019]−0.005-0.005 [−0.030-0.030, 0.022]
weighted coverage↑\uparrow+0.016+0.016 [−0.004-0.004, 0.034]+0.043+0.043 [0.024, 0.062]+0.021+0.021 [0.002, 0.040]−0.009-0.009 [−0.049-0.049, 0.030]−0.003-0.003 [−0.024-0.024, 0.017]
scope precision↑\uparrow+0.011+0.011 [−0.010-0.010, 0.032]−0.038-0.038 [−0.059-0.059, −0.014-0.014]+0.028+0.028 [0.006, 0.052]−0.005-0.005 [−0.073-0.073, 0.067]+0.091+0.091 [0.069, 0.115]
frustration count↓\downarrow−0.11-0.11 [−0.88-0.88, 0.62]−0.88-0.88 [−1.67-1.67, −0.15-0.15]−0.42-0.42 [−0.40-0.40, 1.26]−2.05-2.05 [−3.60-3.60, −0.87-0.87]+6.38+6.38 [5.48, 7.39]
frustration-count deviation↑\uparrow−0.111-0.111 [−0.834-0.834, 0.662]−0.883-0.883 [−1.668-1.668, −0.092-0.092]−0.416-0.416 [−0.409-0.409, 1.234]−2.055-2.055 [−3.509-3.509, −0.891-0.891]+6.384+6.384 [5.324, 7.416]
user-correction deviation↑\uparrow−0.054-0.054 [−0.828-0.828, 0.675]−1.171-1.171 [−1.957-1.957, −0.325-0.325]−0.246-0.246 [−1.061-1.061, 0.511]−0.862-0.862 [−2.204-2.204, 0.302]+6.842+6.842 [5.909, 7.763]
user correction↓\downarrow−0.05-0.05 [−0.74-0.74, 0.69]−1.04-1.04 [−1.81-1.81, −0.20-0.20]−0.26-0.26 [−1.01-1.01, 0.51]−0.82-0.82 [−2.30-2.30, 0.27]+7.12+7.12 [6.21, 8.04]
best metric+0.016+0.016 [−0.004-0.004, 0.034]+0.059+0.059 [0.034, 0.083]+0.028+0.028 [0.006, 0.052]−0.005-0.005 [−0.073-0.073, 0.067]+7.12+7.12 [6.21, 8.04]
composite↑\uparrow+0.020+0.020 [−0.032-0.032, 0.068]−0.037-0.037 [−0.090-0.090, 0.018]+0.047+0.047 [−0.011-0.011, 0.101]−0.187-0.187 [−0.355-0.355, −0.031-0.031]+0.496+0.496 [0.434, 0.567]

Table 12 (continued).

Human Study

PerturbationAcc. (unsure = 0.5)Agreement
Agent favoritism0.530.65
Negative profile artifact0.510.65
Adversarial goal0.570.45
All constraints removed0.530.75
Approve broken work0.660.45
Premature stop0.510.55
Persona swap0.570.35
Language switch0.680.10
Co-debugger0.510.45
Front-loading0.500.70
All0.560.51

Table 13. Paired identification of perturbed replays. Accuracy gives half credit to unsure responses. Agreement is the fraction of pairs receiving the same response from both raters, including unsure.

Filtering

These tables expand the metric-level AUROC and retention results in § 6.1.2.

Agent Differentiation

Table 14 expands Table 1 with individual metrics and control-population retention.

Metric (AUROC / ctrl@5%FPR)α=10%\alpha=10\%25%50%100%
L1: Kendall τ\tau vs reference0.55 / 0.040.69 / 0.120.84 / 0.320.99 / 0.98
SimulatorArena: composite0.45 / 0.050.39 / 0.010.29 / 0.000.08 / 0.00
SimulatorArena: best metric0.50 / 0.06 Likert writing0.54 / 0.01 Likert writing0.56 / 0.01 Likert writing0.64 / 0.09 Likert writing
VISTA: composite0.48 / 0.040.49 / 0.050.47 / 0.040.42 / 0.04
VISTA: best metric0.49 / 0.05 knowledge consistency0.50 / 0.05 profile consistency0.50 / 0.07 profile consistency0.50 / 0.08 profile consistency
SWE-Together: composite0.46 / 0.040.40 / 0.030.30 / 0.010.10 / 0.00
SWE-Together: best metric0.51 / 0.04 weighted coverage0.49 / 0.06 scope precision0.49 / 0.02 scope precision0.44 / 0.04 scope precision
Sim2Real: behavioral index0.51 / 0.060.53 / 0.070.55 / 0.070.58 / 0.10
L2: mean of families0.47 / 0.040.40 / 0.030.34 / 0.020.25 / 0.01
L2: act distribution0.46 / 0.060.38 / 0.020.24 / 0.000.09 / 0.00
L2: dynamics0.49 / 0.050.48 / 0.060.46 / 0.050.43 / 0.05
L2: outcome sensitivity0.48 / 0.050.41 / 0.050.36 / 0.050.39 / 0.05
L2: provenance0.49 / 0.050.49 / 0.030.49 / 0.050.48 / 0.03
Negative profile artifact
Metric (AUROC / ctrl@5%FPR)α=10%\alpha=10\%25%50%100%
L1: Kendall τ\tau vs reference0.53 / 0.060.57 / 0.120.64 / 0.210.87 / 0.49
SimulatorArena: composite0.52 / 0.070.54 / 0.050.60 / 0.100.68 / 0.14
SimulatorArena: best metric0.55 / 0.07 Likert interaction0.60 / 0.07 Likert interaction0.67 / 0.18 Likert interaction0.84 / 0.24 Likert interaction
VISTA: composite0.60 / 0.140.70 / 0.210.85 / 0.470.97 / 0.89
VISTA: best metric0.60 / 0.12 profile consistency0.71 / 0.18 profile consistency0.86 / 0.40 profile consistency0.98 / 0.92 profile consistency
SWE-Together: composite0.60 / 0.070.74 / 0.240.91 / 0.581.00 / 0.99
SWE-Together: best metric0.67 / 0.19 user correction0.85 / 0.47 user correction0.98 / 0.90 user correction1.00 / 1.00 user correction
Sim2Real: behavioral index0.50 / 0.060.51 / 0.060.56 / 0.100.61 / 0.21
L2: mean of families0.52 / 0.060.56 / 0.100.64 / 0.110.81 / 0.30
L2: act distribution0.62 / 0.170.75 / 0.230.91 / 0.661.00 / 1.00
L2: dynamics0.51 / 0.040.53 / 0.060.57 / 0.060.66 / 0.15
L2: outcome sensitivity0.45 / 0.010.37 / 0.010.26 / 0.010.10 / 0.01
L2: provenance0.51 / 0.040.52 / 0.060.56 / 0.040.70 / 0.11

Table 14. Full filtering results corresponding to Table 1. Rows compare L1, L2 and external metrics on agent favoritism and negative profile artifacts; columns vary the perturbed fraction α\alpha. Cells report AUROC / control retention at 5% perturbed survival.

Population-Level Fidelity

Table 15 expands Table 2 for approve-broken-work, premature stop and front-loading. It reports AUROC and control-population retention at 5% perturbed survival for each replacement fraction.

Metric10% AUROC10% ctrl@ 5%FPR25% AUROC25% ctrl@ 5%FPR50% AUROC50% ctrl@ 5%FPR100% AUROC100% ctrl@ 5%FPR
Approve broken work
L2: mean of families0.370.030.290.010.260.010.710.21
L2: act distribution0.230.010.140.000.170.011.001.00
L2: dynamics0.490.050.460.050.420.060.390.05
L2: outcome sensitivity0.450.020.390.020.290.010.150.01
L2: provenance0.470.030.430.040.450.030.710.15
SimulatorArena: composite, set mean0.390.030.240.000.070.000.000.00
SimulatorArena: best metric, set mean0.53W0.070.58W0.120.66W0.170.78W0.26
VISTA: composite, set mean0.400.030.240.010.070.000.000.00
VISTA: best metric, set mean0.43K0.030.31K0.010.16K0.000.01K0.00
SWE-Together: composite, set mean0.350.010.160.000.020.000.000.00
SWE-Together: best metric, set mean0.50S0.040.50S0.040.52S0.040.50S0.09
Sim2Real: USI behavioral index0.490.030.480.030.450.010.550.07
Premature stop
L2: mean of families0.440.040.340.010.190.000.020.00
L2: act distribution0.430.010.320.000.110.000.000.00
L2: dynamics0.430.010.300.010.170.000.270.01
L2: outcome sensitivity0.490.070.480.070.460.030.140.01
L2: provenance0.490.040.490.030.490.020.460.02
SimulatorArena: composite, set mean0.430.040.330.020.190.010.020.00
SimulatorArena: best metric, set mean0.53T0.070.58T0.080.62T0.120.69T0.19
VISTA: composite, set mean0.410.030.290.010.130.000.010.00
VISTA: best metric, set mean0.45G0.030.38G0.020.27G0.000.12G0.00
SWE-Together: composite, set mean0.420.040.320.010.160.000.010.00
SWE-Together: best metric, set mean0.69C0.180.88C0.550.99C0.961.00V1.00
Sim2Real: USI behavioral index0.480.040.450.030.440.030.600.06
Front-loading
L2: mean of families0.540.060.610.100.740.190.920.63
L2: act distribution0.560.100.650.140.780.330.960.77
L2: dynamics0.500.030.470.010.430.010.380.01
L2: outcome sensitivity0.510.040.530.090.560.170.650.20
L2: provenance0.540.060.610.040.740.250.930.68
SimulatorArena: composite, set mean0.560.070.640.150.740.290.900.70
SimulatorArena: best metric, set mean0.56W0.070.64W0.140.74W0.270.90W0.58
VISTA: composite, set mean0.530.070.600.090.690.170.840.46
VISTA: best metric, set mean0.56F0.080.64F0.140.77F0.240.92F0.60
SWE-Together: composite, set mean0.530.070.580.100.650.140.790.30
SWE-Together: best metric, set mean0.53S0.070.59S0.070.65S0.120.80S0.34
Sim2Real: USI behavioral index0.500.060.520.060.560.060.610.08

Table 15. Table 15: Full population-level filtering results corresponding to Table 2. Rows compare individual metrics, composites and best metrics within each perturbation; columns vary the perturbed fraction. Cells report AUROC and control retention at 5% perturbed survival. Bold marks the highest value within each perturbation and column.

Selected metrics. W writing; K knowledge; S scope prec.; T Turing; G goal; C w. coverage; V coverage; F profile.

Trace-Level Fidelity

Table 16 expands Table 3. It pools adversarial goal, all constraints removed, persona swap, language switch and co-debugger. The two column groups compare perturbed replays with control replays and real-user traces, respectively, reporting AUROC and retention at 5% and 10% perturbed survival.

MetricControl vs perturbed (control positive) ↑ AUROCcontrol @5% FPRcontrol @10% FPRReal vs perturbed (real positive) ↑ AUROCreal @5% FPRreal @10% FPR
Ours: L3 (mean of families)‡0.610.070.17n/an/an/a
Ours: persona0.620.100.170.900.680.77
Ours: intent‡0.550.06†0.11†n/an/an/a
Ours: constraints‡0.540.06†0.11†n/an/an/a
SimulatorArena: Likert writing0.550.05†0.11†0.880.28†0.55†
SimulatorArena: Likert interaction0.520.06†0.12†0.750.26†0.52†
SimulatorArena: interaction rating0.490.05†0.10†0.450.03†0.06†
SimulatorArena: document rating0.510.06†0.10†0.470.02†0.07†
SimulatorArena: Turing detector0.490.05†0.09†0.460.04†0.08†
SimulatorArena: composite0.520.050.100.750.240.40
VISTA: goal consistency0.520.050.110.580.060.12
VISTA: profile consistency0.540.060.110.650.070.15
VISTA: knowledge consistency0.520.050.110.660.090.17
VISTA: composite0.550.060.110.680.090.19
SWE-Together: coverage rate‡0.520.050.11n/an/an/a
SWE-Together: user correction0.490.050.110.750.120.24
SWE-Together: user-correction deviation0.480.040.080.730.060.13
SWE-Together: frustration count0.500.05†0.10†0.610.06†0.13†
SWE-Together: frustration-count deviation0.500.05†0.10†0.610.06†0.13†

Table 16. Full trace-level filtering results corresponding to Table 3, pooling 2,636 perturbed replays from 533 source sessions across 5 perturbations. Rows report individual metrics and composites; columns give AUROC and positive-trace retention at 5% and 10% perturbed survival, with control replays (left) or real-user traces (right) as positives. † marks interpolated operating points for metrics with fewer than 50 distinct scores; ‡ marks real-user comparisons omitted to avoid self-comparison with the reference; this also prevents computing the L3 aggregate. Bold marks the highest value in each column.

References

  1. [1]Anthropic. System card: Claude Opus 4.6. System card, Anthropic, 2026. URL https://www-cdn.anthropic.com/0816a4f-29610cb87da10f40c6f78b499a7bf722f.pdf.
  2. [2]Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ²-Bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982, 2025.arxiv.org/abs/2506.07982
  3. [3]Joachim Baumann, Vishakh Padmakumar, Xiang Li, John Yang, Diyi Yang, and Sanmi Koyeyo. SWE-chat: Coding Agent Interactions From Real Users in the Wild. In Third Conference on Language Modeling (COLM), 2026.arxiv.org/abs/2604.20779
  4. [4]Umesh Bodhwani, Thanh Tran, and Kai Wei. GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents. arXiv preprint arXiv:2609.12191, 2026.arxiv.org/abs/2609.12191
  5. [5]Lorenz Brehme, Benjamin Steinhoek, and Ruth Breu. SWE-DIVE: Multi-turn coding agent evaluation with dynamic user simulation. arXiv preprint arXiv:2610.30047, 2026.arxiv.org/abs/2610.30047
  6. [6]Harshita Chopra, Kshitish Ghate, Aylin Caliskan, Tadayoshi Kohno, Chirag Shah, and Natasha Jaques. Beyond cooperative simulators: Generating realistic user personas for robust evaluation of LLM agents. arXiv preprint arXiv:2605.12894, 2026.arxiv.org/abs/2605.12894
  7. [7]Michael A. Covington and Joe D. McFall. Cutting the Gordian knot: The moving-average type-token ratio (MATTR). Journal of Quantitative Linguistics, 2010.DOI
  8. [8]Daniel Deutsch, Rotem Dror, and Dan Roth. A statistical analysis of summarization evaluation metrics using resampling methods. Transactions of the Association for Computational Linguistics, 2021.arxiv.org/abs/2104.00054
  9. [9]Yao Dou, Michel Galley, Baolin Peng, Chris Kedzie, Weixin Cai, Alan Ritter, Chris Quirk, Wei Xu, and Jianfeng Gao. SimulatorArena: Are user simulators reliable proxies for multi-turn evaluation of AI assistants? In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2025.arxiv.org/abs/2510.05444
  10. [10]Ashish Jain and Armaan Sandhu. UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training. arXiv preprint arXiv:2609.38043, 2026.arxiv.org/abs/2609.38043
  11. [11]Anjali Kantharuban and Jonas Mueller. CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking. arXiv preprint arXiv:2610.02460, 2026.arxiv.org/abs/2610.02460
  12. [13]Yao Liu, Yu He, Jihao Huang, Xinyang Liu, and Feng Zhou. Disclosure-Gated User Simulation for Companion-Agent Evaluation. arXiv preprint arXiv:2609.00982, 2026.arxiv.org/abs/2609.00982
  13. [14]Yunan Lu, Ryan Shea, Yusen Zhang, and Zhou Yu. VISTA: A versatile interactive user simulation toolkit for agent evaluation. arXiv preprint arXiv:2606.11079, 2026.arxiv.org/abs/2606.11079
  14. [15]Nitika Mathur, Timothy Baldwin, and Trevor Cohn. Tangled up in BLEU: Revaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020.arxiv.org/abs/2006.06264
  15. [16]Sacha Muller, Antonio Loison, Bilel Omrani, and Gautier Viaud. GrOUSE: A benchmark to evaluate evaluators in grounded question answering. In Proceedings of the 31st International Conference on Computational Linguistics (COLING), 2025.arxiv.org/abs/2409.06595
  16. [17]Tarek Naous, Philippe Laban, Wei Xu, and Jennifer Neville. Flipping the dialogue: Training and evaluating user language models. In International Conference on Learning Representations (ICLR), 2026.arxiv.org/abs/2510.06552
  17. [19]OpenAI. Introducing GPT-5.4. Model release, 2026. URL https://openai.com/index/introducing-gpt-5-4/.
  18. [20]Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2404.13076
  19. [21]Hoang Phan, Dat Huynh, Andrey Zhmoginov, Qi Zeng, Wancen Mu, Yue Cao, Shengjie Bi, Yun He, Changdae Oh, and Deren Lei. MIMESIS: Learning User Simulators as Training Environments for Interactive Agents. arXiv preprint arXiv:2610.09484, 2026.arxiv.org/abs/2610.09484
  20. [22]Olivier Pietquin and Helen Hastie. A survey on metrics for the evaluation of user simulations. The Knowledge Engineering Review, 2013.DOI
  21. [23]Cheng Qian, Zuxin Liu, Akshara Prabhakar, Jielin Qiu, Zhiwei Liu, Haolin Chen, Shirley Kokane, Heng Ji, Weiran Yao, Shelby Heinecke, et al. Userrll: Training interactive user-centric agent via reinforcement learning. arXiv preprint arXiv:2509.19736, 2025.arxiv.org/abs/2509.19736
  22. [24]Ananya B. Sai, Tanay Dixit, Dev Yashpal Sheth, Sreyas Mohan, and Mitesh M. Khapra. Perturbation CheckLists for evaluating NLG evaluation metrics. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.arxiv.org/abs/2109.05771
  23. [25]Preethi Seshadri, Samuel Cah y a wijaya, Ayomide Odumakinde, Sameer Singh, and Seraphina Goldfarb-Tarrant. Lost in simulation: LLM-simulated users are unreliable proxies for human users in agentic evaluations. arXiv preprint arXiv:2601.17087, 2026.arxiv.org/abs/2601.17087
  24. [26]Evangelia Spiliopoulou, Riccardo Fogliato, Hanna Burnsky, Tamer Soliman, Jie Ma, Graham Horwood, and Miguel Ballesteros. Play favorites: A statistical method to measure self-bias in LLM-as-a-Judge. arXiv preprint arXiv:2508.06709, 2025.arxiv.org/abs/2508.06709
  25. [27]Daniel L. Stufflebeam. The metaevaluation imperative. American Journal of Evaluation, 2001.DOI
  26. [28]Joseph Suh, Ayush Raj, Minwoo Kang, and Serina Chang. Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants. arXiv preprint arXiv:2605.09808, 2026.arxiv.org/abs/2605.09808
  27. [29]Guy Tennenholtz, Ofer Meshi, Amir Globerson, Uri Shalit, Jihwan Jeong, and Craig Boutilier. Controllable user simulation. arXiv preprint arXiv:2605.11519, 2026.arxiv.org/abs/2605.11519
  28. [30]Yifan Wu, Zhuokai Zhao, Songlin Li, Ho Hin Lee, Jiacheng Zhu, Shirley Wu, Tianhe Yu, Serena Li, Lizhu Zhang, Xiangjun Fan, and Shengzhi Li. SWE-Together: Evaluating coding agents in interactive user sessions. arXiv preprint arXiv:2606.29957, 2026.arxiv.org/abs/2606.29957
  29. [31]Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Jing Hua Toh, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2404.07972
  30. [32]John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2405.15793
  31. [33]Saber Zerhoudi, Jelena Mitrovic, and Michael Granitzer. AutoSim: A self-improving user simulator for conversational search evaluation. arXiv preprint arXiv:2610.28054, 2026.arxiv.org/abs/2610.28054
  32. [34]Jujia Zhao, Jiakai Tang, Jing Yu, Wen Chen, Jian Wu, Han Zhu, Fei Sun, Suzan Verberne, and Zhaochun Ren. UserArena: Benchmarking Interactive User Simulation for Long-Horizon Shopping Agents. arXiv preprint arXiv:2610.30932, 2026.arxiv.org/abs/2610.30932
  33. [35]Hang Zheng, Nan Jiang, Zichen Zhu, Zixu Yang, Zhiyang Tang, Shuai Fan, Lu Chen, Xiaobao Wu, and Kai Yu. ARGUS: An Agenda-Driven Framework for Reliable User Simulation. arXiv preprint arXiv:2610.21346, 2026.arxiv.org/abs/2610.21346
  34. [36]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2306.05685
  35. [37]Xuhui Zhou, Weiwei Sun, Qianou Ma, Yiqing Xie, Jiarui Liu, Weihua Du, Sean Welleck, Yiming Yang, Graham Neubig, Sherry Tongshuang Wu, and Maarten Sap. Mind the Sim2Real gap in user simulation for agentic tasks. arXiv preprint arXiv:2603.11245, 2026.arxiv.org/abs/2603.11245
  36. [38]Anni Zou, Cheng Chen, Jiaji Deng, Weikang Zhou, Xinji Mai, Yue Wu, Jianing Lin, Lipeng Xie, Lixiong Qin, Qingyue Zhao, Que Shen, Shuchang Tao, Xuchen Pan, Yaliang Li, Yunpeng Zhai, Yushuo Chen, Zhihai Wang, Zhuo Zhang, Zhaoyang Liu, Bolin Ding, and Jingren Zhou. Cowork-Gym: Synthesizing Interactive Work Environments at Scale. arXiv preprint arXiv:2610.17670, 2026.arxiv.org/abs/2610.17670

Paper details

vlnc:W_ecbsd236
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents