Rethinking Verification in Agent Harnesses

Introduction

Coding agents built on large language models (LLMs) tackle long-horizon software engineering tasks, including resolving repository-level issues across large codebases [18, 24, 25, 26]. These agents run inside harnesses that let the model act through tools, observe the results, and iterate toward a task goal [22, 18]. Beyond exposing tools, the harness specifies the agent’s instructions, controls intervention timing, and coordinates verification before accepting its work [27, 28]. These choices allocate computation to checking and revising the agent’s work, making verification an important part of harness design [28].

Prior work develops learned verifiers for selecting solutions [8, 9], self-correction through critiques and reflections [1, 2], and tool-assisted checks that return execution evidence for repair [11, 12, 15]. In practice, verification in agent harnesses is often shaped by engineering heuristics and iterative experience: Claude Code provides hooks for completion checks [39], while Codex uses self-review and iteration [28]. Yet systematic evidence remains limited on how to design and scale verification, when a design improves performance, and whether its gains justify the additional computation. Without this guidance, current designs may be suboptimal for a given model or task.

In this paper, we study verification as an axis of test-time scaling in agent harnesses. We organize the design space around three questions: when to intervene, who should check the work, and how findings should lead to revision. Following the agent lifecycle, L0: Initial instructions and L1: Execution-time reminders shape what the agent is told before and during execution to guide its own checking; L2: Completion gates reopen its work for self-checking; and L3: Independent review assigns a separate critic to check the work and return findings for repair before it is accepted. This framework lets us compare prompting the agent to verify, extending its self-checking, and introducing an independent checking perspective. Within these layers, we vary message role, intervention timing, critic context, and review strategy, then scale independent review through additional rounds or critics.

We compare 17 verification designs on Terminal-Bench (TB) [7] and SWE-bench Verified (SWE) [4, 10] tasks using seven frontier coding models: Qwen3.7-Max [40], Qwen3.8-Max [41], GLM-5.2 [29], DeepSeek V4-Pro [32], DeepSeek V4-Flash [32], Kimi-K2.7-Code [30], and MiniMax-M2.5 [31]. We also introduce an internal VisualBench of 18 visual and interactive tasks, evaluated through blinded expert comparisons of functionality and visual quality. L0 initial instructions and L1 execution-time reminders yield modest, model-dependent average gains on the coding benchmarks, using $0.90$–$1.00\times$ and $0.75$–$0.93\times$ baseline output tokens, respectively. L2 completion gates can yield larger gains but may substantially increase cost: rejection-based designs cost $1.60$–$1.76\times$ baseline on TB and $1.48$–$1.56\times$ on SWE. L3 independent review by a separate critic offers a more favorable accuracy–cost tradeoff, improving average accuracy by $5.9$–$9.5$ percentage points on TB and $3.9$–$6.6$ on SWE at $1.13$–$1.25\times$ and $1.00$–$1.04\times$ baseline cost, respectively, including critic review and agent revision. On VisualBench, all four independent-review designs outperform the evaluated L0–L2 designs, with net preferences of $+13.2$–$+23.8$ points. Changes-based review outperforms trajectory-based review both with and without required visual inspection. Additional review rounds or independent critics further improve average TB accuracy by up to $3.5$ percentage points. Together, these results provide empirical guidance for verification design in agent harnesses and support independent review as a practical axis of test-time scaling, enabling users to configure verification effort for different tasks to improve agent performance.

Related work

Verification. There is a line of works exploring verification for assessing candidates, guiding self-correction, and checking outputs against external evidence. Learned verifiers score complete solutions or intermediate reasoning steps to guide candidate selection [8, 9, 33]. Code-focused assessment ranges from trained critics that highlight bugs [3] to review protocols that combine agent judgments or introduce structured disagreement [16, 17]. Self-correction methods use natural-language critiques and reflections to revise outputs or improve subsequent attempts [1, 2]. Training-based approaches further teach models to improve across successive attempts through fine-tuning or multi-turn reinforcement learning [34, 14]. Studies of reasoning correction and code repair examine the limits of self-generated feedback and show that reported gains depend on feedback quality and the cost of additional attempts [6, 5, 13]. Tool-assisted verification grounds error detection in evidence from code execution and external tools [12, 11], while dedicated test-generation and execution agents return test feedback to guide code revision [15]. Together, these strands provide the assessment and feedback mechanisms from which harness-level verification designs can be built.

Agent harnesses. Agent harnesses determine how models interact with tools and environments by organizing instructions, tool interfaces, execution flow, and context. Prior work studies interleaved reasoning and action [22], agent–computer interfaces [18], execution platforms [23], and structured repair workflows [19]. These studies examine how the surrounding execution design shapes agent behavior. Harness design also governs how computation and information are used across attempts, including agent sampling and selection [20] and reuse of coding rollouts through structured summaries [21]. Agent harnesses have become an essential part of modern coding agents, including Claude Code [24], Codex CLI [25], and Gemini CLI [26].

These works establish useful assessment mechanisms and agent workflows, but offer limited comparative guidance on how a harness should organize checking and revision. We systematically study verification design in agent harnesses across multiple models and benchmarks, examining when to intervene, who performs the checks, how feedback leads to revision, and how additional verification effort affects performance and cost.

Verification Designs

In this section, we introduce verification designs by following the agent lifecycle, from receiving a task to delivering a result as shown in Figure 1. Table 1 summarizes when each layer intervenes, who checks the work, and how execution continues.

Before execution begins, the agent receives the task and the instructions that guide its work. At this stage, L0: Initial instructions introduces verification guidance into the initial prompt. We vary the message role of the verification instructions: System or User. In the System variant, we append it to the system prompt as standing guidance. Because users do not always include verification instructions in their requests, we introduce a User variant that appends the same prompt to the initial user request. Both variants ask the agent to run the deliverable through its intended interface, inspect the output, and fix observed defects. The L0 verification instructions are in Appendix § 11.1. For visual or interactive artifacts, the instruction also asks the agent to render the result and exercise its core interactions. The L0 verification instructions for visual tasks in our VisualBench are in Appendix § 11.1.

As execution proceeds, changes to the codebase accumulate across turns. In our experiment, we find that only $6.2$–$13.6\%$ of detected editing turns are followed by a recognized checking command before the next editing turn or the end of the run, as reported in Appendix Table 7. We therefore introduce L1: Execution-time reminders to encourage the agent to check recent changes as it works, creating opportunities to identify and correct errors before further changes accumulate. These reminders arrive after tool results and before the next model call, so the agent can check its recent changes before continuing. We evaluate three designs. Periodic reminder (System) sends a reminder a few turns after the edit. After-edit reminder (System) sends the same reminder after a turn that changes a file. These two designs use a marker identifying the message as an automated self-verification checkpoint. After-edit reminder (User) uses the same edit-based trigger but delivers the reminder as a user message without that marker. Thus, the first comparison varies the trigger, while the second varies the message role and the presence of the automated marker. The L1 verification instructions are in Appendix § 11.2.

When the agent attempts to finish, L2: Completion gates reopen the agent’s work for checking and revision. To examine how the requested action guides subsequent checking and revision, we vary whether the gate asks the agent to inspect changes or run checks, or rejects completion and asks it to continue working. We vary the message role (System or User) and whether user messages carry an automated marker to study how these choices affect the agent’s response. We also vary intervention timing to test whether reserving time for checking and repair improves outcomes. The L2 prompts and triggering rules are in Appendix § 11.3.

Before final acceptance, the harness can also assign verification to a separate model, which we call the critic. This introduces L3: Independent review: the critic starts in a fresh context and assesses the agent’s deliverable. The critic receives the task specification and access to the live sandbox, where it can run its own checks. The critic returns its findings to the agent for revision. We vary the critic’s context and whether its checks are guided by task requirements alone (Task-first) or also by the agent’s assumptions and claims (Claim-guided). The Changes condition provides the critic with the task and the agent’s changes. The Trajectory condition provides the task, changes, and the agent’s execution trajectory. The critic instructions are in Appendix § 11.4.

Experimental setup

Models. We evaluate seven coding models through Alibaba Cloud Model Studio (DashScope): Qwen3.7-Max [40], Qwen3.8-Max [41], GLM-5.2 [29], DeepSeek V4-Pro [32], DeepSeek V4-Flash [32], Kimi-K2.7-Code [30], and MiniMax-M2.5 [31]. Qwen3.7-Plus serves as the independent critic for every coding model.

Benchmarks. We evaluate on task subsets from Terminal-Bench 2.1 (TB) [7] and SWE-bench Verified (SWE) [4, 10], both with automatic graders. Task selection aims to cover a range of difficulties, using baseline success rates for TB and human time-to-fix tiers followed by baseline evaluation for SWE. Appendix § 7.1 provides task counts and the selection procedure. Each coding condition is configured for three trials per model and task.

VisualBench contains 18 visual and interactive tasks. Each provides a task description and any supporting files, and asks the agent to produce an artifact, such as a playable game, an interactive 3D configurator, or a sales data explorer. For models without native visual capabilities, a look_at tool renders the artifact and returns a description from Qwen3.7-Plus. Raters from an internal company data team compare artifacts from a verification condition and the baseline for the same task and model, with conditions hidden and sides randomized. They receive the task brief and short interaction instructions, and judge functionality and visual quality against the task requirements. Outcomes are recorded as wins, losses, or ties; cases where both artifacts fail are recorded separately. We evaluate one representative design from each of L0–L2 and four L3 designs, with two trials configured per task, model, and condition. Appendix § 9 provides the full descriptions of all 18 tasks, expert instructions, and screenshots of the evaluation interface.

Environment. All experiments run in E2B cloud sandboxes [36], orchestrated with Harbor [35]. Each trial starts in a fresh sandbox containing the task files, dependencies, and execution tools. The agent interacts with this environment by reading and editing files, running commands, and inspecting outputs. For independent review, the critic accesses the same workspace to examine the deliverable and execute its own checks. The critic does not have permission to modify the deliverable. The agent can then revise the artifact using the returned feedback.

Metrics. For TB and SWE, we report accuracy (the percentage of tasks solved) and cost. We average accuracy and cost across trials for each model–task pair. Cost includes the coding agent, the critic, and subsequent revisions, priced separately by model. Figure 2 reports each design’s mean cost relative to its baseline, with per-model plots in Appendix § 8.2. Appendix § 7.2 lists the API cost prices, and Appendix § 8.3 separates agent and critic output tokens.

Results

In this section, we analyze L0 initial instructions in Section , L1 execution-time reminders in Section , L2 completion gates in Section , and L3 independent review in Section .

L0: Initial instructions

L0 verification instructions yield modest average coding gains, with substantial differences across models and benchmarks. Instructions with the system role improve accuracy by $+3.3$ points on TB and $+3.9$ points on SWE, compared with $+1.2$ and $-1.1$ points for instructions with the user role, as shown in Table 2 and Table 3. On SWE, MiniMax-M2.5 benefits from both variants ($+12.6/+8.0$ points for system/user), whereas Qwen3.8-Max declines under both ($-2.3/-2.3$). Qwen3.8-Max instead gains $+8.0/+4.0$ points on TB with the system/user roles. On VisualBench, L0 yields an average net preference of $-3.2$ points over the baseline, as shown in Figure 3. DeepSeek V4-Flash receives a net preference of $+5.6$ points, while GLM-5.2 receives $-8.3$ points despite its positive accuracy gains on both coding benchmarks.

Cost changes are modest: system/user instructions incur $1.08/1.03\times$ baseline cost on TB and $0.94/0.93\times$ on SWE, as shown in Figure 2. Average output tokens and agent turns remain close to or below baseline levels across both benchmarks and message roles, at $0.90$–$1.00\times$ and $0.94$–$0.99\times$, respectively, as reported in Appendix Table 6.

These average gains are not accompanied by consistently increased generation or longer interactions. Overall, we recommend concise verification instructions in the system prompt, as system instructions outperform user instructions on average in the coding benchmarks, while their uneven benefits favor leaving room for other harness instructions.

The case studies illustrate how L0 instructions can alter verification behavior that is already present. On bn-fit-modify, MiniMax’s baseline run inspects the generated files and their row counts but misses the required edge orientation. With the verification instruction, the agent revisits this requirement, corrects the CSV convention, and passes, as shown in Appendix Figure 25. On git-multibranch, Qwen3.7-Max checks deployment endpoints in both conditions. However, the instructed run deletes the repository to prepare for another test, removing the deployment hook, and fails despite subsequent reconstruction; the baseline passes, as shown in Appendix Figure 26.

\node[draw=layerzero, line width=0.8pt, rounded corners=5pt,
      fill=layerzero!11, inner xsep=10pt, inner ysep=10pt,
      text width=\dimexpr\linewidth-21pt\relax, align=left,
      font=\small] (takeaways) {%
      \setlength{\parindent}{0pt}%
      \vspace{2pt}%
      \noindent\ignorespaces
\textbf{Keep verification guidance in system prompt lightweight.}
Use concise instructions, leaving room for other harness guidance.
Use user message for task requirements and context.
\unskip\par
      };
    \node[anchor=west, fill=layerzero!70!black, text=white, rounded corners=2pt,
      inner xsep=7pt, inner ysep=3pt, font=\small\bfseries]
      at ([xshift=10pt]takeaways.north west) {L0 Takeaways};

L1: Execution-time reminders

Periodic and after-edit L1 reminders have similar average effects with the system message role: $+3.8$ versus $+3.5$ points on TB and $+3.1$ versus $+3.6$ on SWE. Switching the message role of after-edit reminders to user raises the average gain to $+5.4$ points on TB but lowers it to $-0.3$ on SWE, as shown in Table 2 and Table 3. On VisualBench, MiniMax-M2.5 receives $+12.5$ points, while GLM-5.2 and Qwen3.7-Max each receive $-12.5$ points. Across the three L1 designs, average output tokens decrease by $14$–$25\%$ on TB and $7$–$10\%$ on SWE, while cost ranges from $0.90$–$1.12\times$ and $0.85$–$0.89\times$ baseline, respectively, as shown in Figure 2 and Appendix Figure 15. Higher accuracy can also coexist with less generated output. On TB, Qwen3.7-Max gains $+7.1/+14.3$ points with after-edit system/user reminders while using $33\%/28\%$ fewer output tokens, respectively. On VisualBench, L1 yields an average net preference of $-3.2$ points over the baseline, as shown in Figure 3. Overall, we recommend verification reminders after code edits to focus the agent on task requirements and the resulting artifact alongside the code itself. The average accuracy gains with fewer output tokens on TB and SWE support using L1, while the VisualBench results motivate checks that also cover rendered output and interactive behavior.

Trajectory analysis distinguishes continued execution from task-relevant verification. On TB, $90$–$97\%$ of after-edit user reminders are followed by execution commands within two agent turns, as reported in Appendix Table 8. The checks themselves also differ in coverage. On nginx-request-logging, Qwen3.7-Max checks the Nginx configuration, HTTP responses, and access log after a reminder and passes, whereas the baseline stops with an unfinished plan, as shown in Appendix Figure 27. However, Kimi-K2.7-Code verifies compilation while overlooking invalid word substitutions on overfull-hbox, and DeepSeek V4-Flash verifies output equality yet misses the runtime criterion on query-optimize, as shown in Appendix Figure 28 and Appendix Figure fig:case-l1-query.

\node[draw=layerone, line width=0.8pt, rounded corners=5pt,
      fill=layerone!11, inner xsep=10pt, inner ysep=10pt,
      text width=\dimexpr\linewidth-21pt\relax, align=left,
      font=\small] (takeaways) {%
      \setlength{\parindent}{0pt}%
      \vspace{2pt}%
      \noindent\ignorespaces
\textbf{Use verification reminders after code edits to guide the agent in checking both the code and the resulting artifact against task requirements.}
Include checks of rendered output and interactive behavior when relevant.
\unskip\par
      };
    \node[anchor=west, fill=layerone!70!black, text=white, rounded corners=2pt,
      inner xsep=7pt, inner ysep=3pt, font=\small\bfseries]
      at ([xshift=10pt]takeaways.north west) {L1 Takeaways};

L2: Completion gates

L2 yields limited gains relative to its cost, even when rejection and earlier intervention improve average accuracy. Automated Reject completion yields $+4.0/+5.3$ points on TB/SWE, compared with $+0.7/+0.5$ for Inspect changes and $+2.6/-0.5$ for automated Run and check. Using the system role for Run and check raises the gains to $+4.5/+1.5$ points. Earlier triggering also improves both user rechecking and user rejection: rechecking rises from $+4.1/-0.3$ to $+4.8/+4.1$ points, while rejection rises from $+2.1/+4.1$ to $+6.0/+6.2$ points, the largest average gains among the tested L2 designs, as shown in Table 2 and Table 3. Across the rejection designs, cost reaches $1.60$–$1.76\times$ baseline on TB and $1.48$–$1.56\times$ on SWE. Early user rechecking achieves its gains at a lower cost of $1.13/1.20\times$ baseline, respectively, as shown in Figure 2. On VisualBench, early rechecking yields an average net preference of $+2.7$ points, as shown in Figure 3. GLM-5.2 receives $+9.4$ points, while DeepSeek V4-Flash receives $-8.0$ points. Overall, we recommend avoiding completion-time self-checking, as its gains are limited relative to the additional cost and reopening work does not consistently improve the final artifact.

The L2 cases illustrate both useful repair and invalid final states. On TB, model-level medians range from $12$–$19$ additional steps after user rejection, compared with $4$–$6$ after Inspect changes. File-changing actions follow user rejection in $86$–$96\%$ of L2 runs in which user rejection is triggered, as reported in Appendix Table 9. On regex-log, MiniMax-M2.5 resumes after early user rejection and finds that its regex misses dates placed before the IP address. It removes this ordering assumption and passes grading, whereas the baseline retains it and fails, as shown in Appendix Figure 30. On git-multibranch, Kimi-K2.7-Code responds to an early recheck with deployment tests that commit and push test content to the served branches. It leaves that content in place and fails, whereas the passing baseline restores the required branch contents, as shown in Appendix Figure 31. These cases show why further execution alone is insufficient: a completion gate creates an opportunity for repair, but the agent must still identify the relevant defect and preserve a valid final artifact.

\node[draw=layertwo, line width=0.8pt, rounded corners=5pt,
      fill=layertwo!11, inner xsep=10pt, inner ysep=10pt,
      text width=\dimexpr\linewidth-21pt\relax, align=left,
      font=\small] (takeaways) {%
      \setlength{\parindent}{0pt}%
      \vspace{2pt}%
      \noindent\ignorespaces
\textbf{Avoid completion-time self-checking.}
Asking the main agent to recheck its own work can improve accuracy, but the gains are limited relative to the additional cost.
\unskip\par
      };
    \node[anchor=west, fill=layertwo!70!black, text=white, rounded corners=2pt,
      inner xsep=7pt, inner ysep=3pt, font=\small\bfseries]
      at ([xshift=10pt]takeaways.north west) {L2 Takeaways};

Vector (PDF)

L3: Independent review

L3 delivers consistent improvement across all four designs on TB and SWE at modest additional cost. The gains range from $+5.9$–$9.5$ points on TB and $+3.9$–$6.6$ on SWE, as shown in Table 2 and Table 3. Cost, including critic review and agent revision, is $1.13$–$1.25\times$ baseline on TB and $1.00$–$1.04\times$ on SWE, as shown in Figure 2. Compared with the strongest L2 design, the strongest L3 design achieves a larger gain on TB ($+9.5$ versus $+6.0$ points) and a comparable gain on SWE ($+6.6$ versus $+6.2$), at lower cost on both benchmarks. On VisualBench, all four L3 designs receive positive net preference, ranging from $+13.2$ to $+23.8$ points and exceeding the evaluated L0–L2 designs, as shown in Appendix Table 12. Changes-based review outperforms trajectory-based review both without required visual inspection ($+21.2$ versus $+13.2$ points) and with it ($+23.8$ versus $+21.6$ points). Required visual inspection improves net preference for both contexts, with a larger increase for trajectory-based review. Both changes-based designs receive positive net preference for all seven models, while trajectory-based designs retain some negative model-level results. Across the three benchmarks, the full trajectory does not consistently improve outcomes, and trajectory-based designs incur $29$–$43\%$ higher critic cost on TB and SWE. Overall, we recommend task requirements and changes as the default context for a separate, low-cost critic.

The trajectories illustrate how independent review turns additional checks into concrete repairs. On regex-log, MiniMax-M2.5 tests its regex and declares completion, but the critic tests a case with the date appearing before the IP address and exposes a missed match. The agent then revises the regex and passes, whereas the baseline performs its own tests but still fails, as shown in Appendix Figure 32. On cobol-modernization, Qwen3.7-Max’s implementation passes its own checks for valid and invalid transactions. The critic tests an arithmetic underflow and finds that Python writes a negative sign that COBOL’s unsigned field omits, breaking the required record format. The agent reproduces the discrepancy, repairs the numeric formatting, and passes the external grader; the baseline declares success after its own comparisons but fails grading, as shown in Appendix Figure 33. In both examples, the critic tests a case that the agent overlooked and supplies an executable failure that guides revision. This illustrates how an independent checking perspective can address gaps in the agent’s self-verification. On cancel-async-tasks, GLM-5.2’s critic confirms that asynchronous cleanup remains incomplete after cancellation, but the run reaches its 45-minute limit without a completed repair after review, as shown in Appendix Figure 34.

Vector (PDF)

Scaling review. Increasing review depth (successive review rounds) or width (independent critics) further improves average accuracy on TB. One critic and one round achieve $62.7\%$ accuracy at 38.2k mean critic tokens, compared with $65.6\%$ at 79.5k for two rounds and $64.4\%$ at 69.7k for two critics. All larger settings outperform a single review, reaching $66.1\%$ with four rounds and $65.8\%$ with four critics, as shown in Figure 4. Four critics improve average accuracy by $3.1$ points over one critic at $1.33\times$ total cost, including revisions. Scaling verification through low-cost critics can limit repeated execution by the main agent. By consolidating independent feedback before revision, four critics approach four-round accuracy at $19\%$ lower total cost, as shown in Appendix Figure 23.

\node[draw=layerthree, line width=0.8pt, rounded corners=5pt,
      fill=layerthree!11, inner xsep=10pt, inner ysep=10pt,
      text width=\dimexpr\linewidth-21pt\relax, align=left,
      font=\small] (takeaways) {%
      \setlength{\parindent}{0pt}%
      \vspace{2pt}%
      \noindent\ignorespaces
\textbf{Use a separate, low-cost critic to check the deliverable and guide agent revision.} Scaling review through additional rounds or independent critics further improves average accuracy.
\unskip\par
      };
    \node[anchor=west, fill=layerthree!70!black, text=white, rounded corners=2pt,
      inner xsep=7pt, inner ysep=3pt, font=\small\bfseries]
      at ([xshift=10pt]takeaways.north west) {L3 Takeaways};

Conclusion

In this paper, we systematically study verification in agent harnesses, comparing 17 designs across four layers of the agent lifecycle and seven frontier coding models on TB and SWE, with evaluation on VisualBench. We examine when to intervene, who checks the work, and how feedback leads to revision. Initial instructions and reminders yield modest, model-dependent coding gains without substantially increasing average output token use. Completion gates can improve accuracy but may substantially increase cost. Independent review offers a more favorable accuracy–cost tradeoff on TB and SWE. On VisualBench, its four designs achieve net preferences of $13.2$–$23.8$ points over the baseline, exceeding the evaluated L0–L2 designs. Additional rounds or critics further improve average TB accuracy by up to $3.5$ percentage points. Together, these results provide empirical guidance for verification design in agent harnesses and support independent review as a practical axis of test-time scaling.

Ethics statement

VisualBench’s human evaluation received ethics approval. Raters are members of an internal company data team; they provided informed consent and were compensated under their existing contracts with the company. The interface explains the study’s purpose and assessment criteria, while hiding verification conditions and randomizing artifact placement. Appendix § 9 documents the evaluation protocol, task descriptions, rater instructions, and interface.

AI use statement

The authors developed the research ideas and system design and led the manuscript writing. Generative AI tools assisted with implementation and manuscript polishing. The authors are responsible for the final content.

Appendix guide

Additional experimental details

Task selection for Terminal-Bench and SWE-bench

We select tasks to balance coverage across easy, intermediate, and hard difficulty levels, enabling us to better observe both improvements and regressions.

Terminal-Bench. We began with a baseline-harness pilot over the 89-task Terminal-Bench registry recorded as version 2.1. The pilot used Qwen3.7-Max, GLM-5.2, DeepSeek V4-Pro, and Kimi-K2.7-Code. Their pass rates guided selection across three empirical difficulty bands: tasks that were usually solved, solved inconsistently, or usually failed. We aimed for balanced coverage of these bands, yielding the 31-task study set.

SWE-bench Verified. We initially selected 30 tasks evenly across three human time-to-fix tiers: under 15 minutes, 15 minutes to one hour, and over one hour. We then evaluated baseline performance and adjusted the task set to include both tasks with room for improvement and consistently solved tasks for detecting regressions.

API cost pricing

Table 4 lists the model rates used to calculate cost, with Alibaba Cloud Model Studio pricing as the reference [37, 38].

Additional coding results

We report accuracy, cost, and token use by verification design and coding model, followed by behavioral diagnostics for L0–L2.

Absolute accuracy by verification design

Figure 5 reports accuracy by verification design, with open markers for the baseline and filled markers for the design. The corresponding absolute accuracies are listed alongside.

Vector (PDF)

Figure 6–Figure 12 show the same comparison for each coding model.

Vector (PDF)

Vector (PDF)

Vector (PDF)

Vector (PDF)

Vector (PDF)

Vector (PDF)

Vector (PDF)

Accuracy and cost by coding model

Figure 13 and Figure 14 separate Figure 2 by coding model, using the same trials, prices, and baseline. Each figure includes an Average panel reproducing Figure 2, and each panel labels all 17 verification designs by ID. Black stars mark baseline accuracy, and dashed lines mark $1\times$ baseline cost. Axis ranges are shared across models within each benchmark.

Table 5 lists the design IDs and settings.

Vector (PDF)

Vector (PDF)

Agent and critic token use

Figure 15 shows mean output token use for all verification designs. Solid bars show agent generation, including revisions, and hatched segments show critic generation. Vertical ticks mark the baseline, with baseline and total values listed in thousands.

Vector (PDF)

Figure 16–Figure 22 show the same comparison for each coding model.

Vector (PDF)

Vector (PDF)

Vector (PDF)

Vector (PDF)

Vector (PDF)

Vector (PDF)

Vector (PDF)

Review scaling and total cost

Figure 23 reports accuracy against mean total cost, including agent execution, critic checks, and subsequent revisions. One critic and one round achieve $62.7\%$ accuracy at $0.47 per trial. Two rounds reach $65.6\%$ at $0.61, while two critics reach $64.4\%$ at $0.60. Four critics achieve $65.8\%$ at $0.62, compared with $66.1\%$ at $0.77 for four rounds. The critic costs for four critics and four rounds are both approximately $0.14 per trial, while their agent-side costs, including revisions, are $0.48 and $0.63, respectively. The cost difference therefore comes primarily from agent execution, consistent with wider review consolidating feedback before revision instead of repeatedly continuing the agent between review rounds. A second round raises Qwen3.7-Max from $59.5\%$ to $70.2\%$, but lowers Kimi-K2.7-Code from $60.7\%$ to $58.3\%$.

Vector (PDF)

L0: Output tokens and agent turns

Table 6 reports mean output tokens and agent turns relative to baseline on TB and SWE. Values above one indicate more generated output or longer interactions.

L1: Checking during execution

Table 7 reports how often TB baseline agents check their work after editing. We count recognized checking commands before the next editing turn or the recorded run end.

Table 8 reports reminder coverage and checking or execution within the next two agent turns. After-edit user reminders reach $80$–$100\%$ of runs, with recognized checks following $58$–$72\%$ of reminder events.

L2: Work after a completion gate

Table 9 reports subsequent steps and file-changing actions after L2 interventions. User rejection is followed by $12$–$19$ additional steps at the model median and file-changing actions in $86$–$96\%$ of runs where it is triggered.

VisualBench tasks and human evaluation

VisualBench contains 18 tasks spanning games, shaders, interactive three-dimensional scenes, applications, and other visual artifacts.

Complete task catalog

Table 10 and Table 11 summarize all 18 tasks and their requested deliverables. Full task descriptions and task-specific interaction instructions follow the overview.

The following entries reproduce all 18 English task briefs and their task-specific rater instructions from the evaluation kit. Line wrapping and dash typography are normalized for typesetting. The task descriptions are supplied to the agent; the interaction instructions guide human assessment of the delivered artifacts.

Blender desk scene

blender-object

Task description

Human interaction instructions

1. First, the objects in render.png: an articulated desk lamp - base, jointed arm, shade - with a mug and a stack of books beside it, arranged as a composition rather than a row? If the objects are wrong, the lighting below does not count

2. Look at render.png - lamp lights the scene, shadows and depth of field

3. Play turntable.mp4 - full rotation, lamp lit throughout

Interactive cloth

cloth-sim

Task description

Human interaction instructions

1. First: is it a rectangular sheet pinned along its top edge hanging under gravity, with the sphere in the scene and live energy / constraint-error readouts on screen? No sheet, no sphere or no readouts - nothing below counts

2. Grab a point and shake it hard, then let go - does it settle, or does the sheet detonate? Explosion on hard input is an outright fail

3. Watch the energy and constraint-error readouts while it settles: energy should fall and the error should go toward zero. If the numbers are missing or frozen, the claim is unsupported

4. Drape it over the sphere - does it rest ON the surface, or sink through it, or jitter?

5. Does it fold, or does it stretch like rubber? Pull one corner far away and see whether the sheet grows

6. Unpin a corner; turn the wind up and down; cut or tear it; change stiffness, gravity and damping while it runs, then reset

7. Raise the resolution - does it stay stable and keep its framerate, or does higher resolution make it unstable?

8. Toggle the structure view - are particles and constraints actually there? Then look at the shaded view: does it read as fabric with light on it?

Flowline kanban

data-app-kanban

Task description

Human interaction instructions

1. Open it cold - is it the Flowline board with Backlog / In Progress / Done, already populated with realistic cards (title, note, labels, assignee, due date visible without opening them), and In Progress already at its WIP limit? An empty board means you cannot test anything without first typing data in - nothing below counts

2. Add, edit in place, and delete a card - all three work, and column counts stay correct

3. Drag a card between and within columns - the order follows

4. Drag a fourth card into In Progress - refused visibly, with a reason given

5. Click the board first so it has focus. Select several cards and move or delete them in one go, then press Ctrl+Z ONCE - the whole batch comes back as one step; then try Ctrl+Z / Ctrl+Y on an add, an edit and a drag

6. Put the mouse away: can you move a card between columns using only the keyboard?

7. Type in the filter, then also narrow by a label or assignee - they compose; counts follow

8. Check due dates: is overdue vs due-soon obvious without opening a card?

9. Reload the page (F5) - cards, order, labels, filter and collapsed columns all survive

Sales data explorer

dataviz-explorer

Task description

Human interaction instructions

1. First: is it showing that dataset - six regions, about thirty months, both the online and retail channels? If not, nothing below counts

2. Give yourself two minutes: which region’s online/retail mix shifted the most, and roughly when did it turn? Could you answer from the page alone?

3. Try to compare two regions directly - does the page let you, or do you have to remember one while looking at the other?

4. Separate seasonality from trend - is there any way to tell a recurring dip from a real decline?

5. Click the excluded-row count: the panel length must match the count, every row must carry a reason, and the rows must genuinely be unusable (spot-check a few)

6. Between two candidates, prefer the one that repairs more and excludes less

7. Look for a break in a line - a single missing value must render as a gap, never as zero

8. Check it does not mislead: axes that start somewhere other than zero without saying so, aggregation you cannot identify, or rows dropped silently

Breakout

game-breakout

Task description

Human interaction instructions

1. First: is it breakout - paddle, ball and a brick field, paddle on both mouse and arrow keys, and several brick kinds (plain, two-hit, armoured, unbreakable)? If not, nothing below counts

2. Play a level - listen for the pitch rising through a rally

3. Catch a power-up; check what’s active is shown

4. Lose all balls, then restart; reload the page (F5) and check the high score is still there

2D platformer

game-platformer

Task description

Human interaction instructions

1. First: is it a 2D platformer - terrain, hazards, coins, a start and an exit, in a level larger than the screen? If not, nothing below counts

2. Tap vs hold jump - the height varies; jump just after a ledge - coyote time catches it

3. Run ahead - the camera follows smoothly

4. Touch a hazard - death, then respawn at the level start

5. Collect coins and reach the exit - counter, level 2, then a win screen

Tower defense

game-tower

Task description

Human interaction instructions

1. First: is it tower defense - a visible enemy path, towers placed by clicking (refused on the path or on another tower), at least four tower types, a HUD with wave / lives / money? If not, nothing below counts

2. Place different tower types; click one to see range and stats

3. Upgrade a tower, then sell one

4. Call the next wave early; use pause and speed-up

Voxel sandbox

game-voxel

Task description

Human interaction instructions

1. First: is it a voxel world - rolling terrain with hills, valleys and beaches, water in the low ground that reads as water, and visibly distinct block types (trees, sand, stone, dirt, grass, ore)? If not, nothing below counts

2. Click the canvas - pointer locks; WASD, mouse-look, space to jump

3. Walk a while - new terrain streams in; look for caves and water

4. Hold left click to break, right click to place; open the inventory

5. Dig out the block under sand; wait for night; try a torch and the crafting screen

6. Reload the page (F5) - the world and your changes should still be there

Generative art series

generative-art

Task description

Human interaction instructions

1. First: does the page show one piece at a time, with next / jump-to-seed controls? And an even mat of marks filling the whole canvas fails this task however fine the marks are - check that before anything else

2. Use the on-page next / jump-to-seed controls: go to a seed, note the picture, move away and come back - it must be exactly the same picture, not a reflowed one

3. Step through three different seeds. Do they read as one body of work - same hand - while being genuinely different compositions?

4. The failure to look for is the opposite pair: either all three are the same picture with the parts moved, or they have nothing in common but the palette

5. Take one piece: is there something the eye goes to? Is the empty space doing work, or is it just where the algorithm stopped?

6. Do the marks read as a medium - varying density, pooling where they cross, thinning out - or as uniform hairlines?

7. Look closely at the edges of the marks (use "open full $\nearrow$" for a bigger view) - still crisp, or mushy? The piece should fill the whole canvas, no blank strip or letterbox

8. Watch it draw - progressive, and the page still responds while it works?

Vector boolean editor

path-boolean

Task description

Human interaction instructions

1. First: is it a vector editor - closed paths drawn with a pen tool, rectangles and ellipses, cubic Bézier segments, and all four operations (union, intersection, difference, exclusion) available? If not, nothing below counts

2. Union two overlapping shapes, then run difference on the RESULT - an operation whose output cannot be operated on again fails the task

3. Nudge a point after an operation: does the outline follow, with handles intact? If the result is not editable it was faked

4. Difference a small shape out of a big one so the result has a HOLE - then move the outer shape and check the hole is still a hole

5. The two classic degenerate cases: shapes that only TOUCH (tangent, no crossing), and shapes sharing a straight edge along its length - no garbage, no dropped contour

6. An operation whose result is several disjoint pieces - are they all there, as one path?

7. Use curves, not just polygons: draw with Bézier segments and check the result curves are still curves, not polylines

8. Toggle the intersection-point and winding-direction overlay - do the points sit where curves actually cross?

City in a storm

shader-city

Task description

Human interaction instructions

1. First: is it that city - night, storm, seen from over the water: towers of varying height receding, windows lit unevenly, the city rising out of moving water with reflections, rain and low cloud? If not, nothing below counts

2. Wait for lightning - towers, cloud and reflection light together

3. Look at the water - reflections stretch and break with the swell

4. Move the mouse to swing the viewpoint; watch fog with distance

Stormy ocean

shader-scene

Task description

Human interaction instructions

1. First: is it open water in a storm - long rollers with chop riding on top, foam on the crests, clouds with depth, rain - not a blue sine wave? If not, nothing below counts

2. Watch thirty seconds - the storm builds and eases, lightning relights the scene

3. Find a crest between you and the sun - look for light through the wave

4. Move the mouse to swing the viewpoint; look straight down, then to the horizon

5. The scene should fill the whole canvas with no black bars; while it runs (and while you drag or move), it should stay smooth, not stutter

Gridwork spreadsheet

spreadsheet-engine

Task description

Human interaction instructions

1. First: is it a spreadsheet - at least 40 rows by 20 columns, A1-style addresses, a formula bar, arrow-key navigation, typing straight into a cell? If not, nothing below counts

2. Formulas chain: type 5 in B1, =B1*2 in C1, =C1+1 in D1 $\rightarrow$ they show 10 and 11; change B1 to 7 $\rightarrow$ C1 and D1 become 14 and 15 by themselves

3. Built-in functions: put 1, 2, 3 in A1-A3; A5 =SUM(A1:A3) $\rightarrow$ 6; A6 =IF(A5>5,"big","small") $\rightarrow$ big

4. Errors: E1 =1/0 and E2 =FOO(1) $\rightarrow$ both show a readable error and the sheet stays up; F1 =E1+1 $\rightarrow$ the error propagates instead of quietly becoming 0

5. Cycle: G1 =H1 and H1 =G1 $\rightarrow$ both cells report a circular reference; the page must not hang or freeze

6. Fill, range, sort: fill C1 down to C10 $\rightarrow$ references become B2, B3...; select A1:A3 $\rightarrow$ the status bar shows count 3, sum 6; sort a column and check the result

7. Undo and persistence: Ctrl+Z undoes one step, Ctrl+Y redoes; then reload the page (F5) $\rightarrow$ the sheet is still there

Animated concert poster

svg-poster

Task description

Human interaction instructions

1. First, check the three strings appear exactly: NEON TIDE / Live at Meridian Hall / Sep 12 2026; then whether it is a poster you would put on a wall rather than centred text on a gradient. If this fails, nothing below counts

2. Watch a full loop - look for the seam where it repeats

3. Zoom the browser to 300% - the type and edges stay crisp

4. Look for several things moving at different rates

3D lamp configurator

three-config

Task description

Human interaction instructions

1. First: is it the lamp that was asked for - base, arm, joints and shade all there, at least three shade shapes to switch between, arm height and angle adjustable? If this fails, nothing below counts

2. Change shade, finish and colour - transitions, not snaps

3. Switch the bulb on; move the dimmer and warmth

4. Drag to orbit and let go - it should glide to a stop with damping; wheel to zoom; the view should fill the whole canvas with no black bars

GPU particle field

three-particles

Task description

Human interaction instructions

1. First, the on-screen count and framerate: is it close to a million particles at a smooth rate, and does the motion read as a fluid (momentum, trails) rather than points on springs? If not, nothing below counts

2. Move the mouse through the field; toggle attract/repel

3. Watch a pile-up - brighter where particles gather

4. Watch it for a few seconds - it must keep running steadily, never freeze or stop, and fill the whole canvas

Product hero scene

three-scene

Task description

Human interaction instructions

1. First, the object: a wireless headphone - band, two cups, cushions, hinges, all there and in believable proportion? If the object is wrong, nothing below counts

2. Drag to orbit, then let go - it glides to a stop

3. Switch colour variants; toggle the exploded view

4. Click a hotspot on the object

5. Leave it alone for a few seconds - idle animation resumes

Fleet terminal dashboard

tui-dashboard

Task description

Human interaction instructions

1. First: is the recording a multi-pane terminal dashboard - a service list with status, a sparkline per service, an overall health summary, a scrolling log pane, and ok / warn / crit colour-coded? If not, nothing below counts

2. Play demo.cast - values move and the log scrolls

3. Watch for a service crossing into crit

4. Check sparklines and that problems sort to the top

Human evaluation protocol

The human comparison covers seven designs. For L0–L2, we select one representative intervention per layer: the system verification instruction, the after-edit user reminder, and the user recheck request with the 80% early trigger. These use the same intervention definitions as TB and SWE. For L3, we compare changes-based and trajectory-based review, each with and without required visual inspection. Each design is compared with the baseline for the same task and coding model. Review depth and width are studied separately on Terminal-Bench.

Each task supplies a description and any supporting files without a reference implementation. Tasks run in an environment with Python, Chromium, and ffmpeg, with additional tools where required. Browser artifacts run offline, with bundled libraries only where the task permits them.

Raters and briefing. The raters are members of an internal company data team. Before rating, the interface presents a briefing explaining the study’s purpose, the blinded comparison, the four response options, and a worked example of applying the criteria.

Consent, compensation, and ethics approval. The human evaluation received ethics approval. Raters provided informed consent and were compensated under their existing contracts with the company.

Blinded comparison. Raters compare artifacts for the same task and model with verification conditions hidden and sides randomized. The interface presents the original task brief, its judging criteria, and a short set of task-specific interaction steps. Both artifacts can be inspected side by side or opened individually at full size. Raters interact with live artifacts where available and inspect delivered recordings or renders where appropriate.

Instructions to raters. The rating interface gives the following guidance:

1. Read the original task brief before comparing the artifacts. Judge against what the task asks for, including its functional and visual requirements.

2. Follow the task-specific interaction checklist on both artifacts. Click, drag, type, or play as needed to check the requested behavior; use the full-size view when the side-by-side view is too small. For keyboard controls, first click inside the artifact to give it focus.

3. Choose A or B when one artifact better satisfies the task. Choose a tie when both work and you cannot distinguish their quality. Choose both fail only when neither makes a meaningful attempt at the task, such as when both are blank, broken, or unrelated to the request. An artifact that works poorly does not by itself justify both fail.

4. If only one side produces a usable deliverable, choose that side even if it is rough. If a page appears blank, refresh it once before judging. Visual polish should not outweigh missing functionality unless visual quality is itself the task’s objective.

5. Optionally leave a short comment explaining the deciding issue, especially when the two artifacts are difficult to distinguish.

Example interaction checklist. For a Breakout game, raters move the paddle with the arrow keys, launch the ball, clear several bricks, and deliberately let the ball drop. These actions check paddle control, collisions, scoring, and the loss condition. The judgment follows the requested game behavior rather than a screenshot alone.

Expert evaluation interface

Figure 24 shows the English rating interface for the Blender desk-scene task, with rendered images and turntable videos available on both sides. Raters can expand the task brief and judging guidance, follow the task-specific instructions, and inspect the two artifacts side by side or at full size. Four response buttons record a preference for A or B, a tie, or failure of both artifacts; an optional comment records the reason for the judgment.

Vector (PDF)

Detailed VisualBench results

L0 results. The system verification instruction yields a net preference of $-3.2$ points over the baseline, as shown in Table 12. DeepSeek V4-Flash and Qwen3.7-Max benefit ($+5.6$ and $+2.8$ points), while DeepSeek V4-Pro, GLM-5.2, and Kimi-K2.7-Code decline ($-10.0$, $-8.3$, and $-8.3$). Initial guidance therefore does not provide a general improvement on visual tasks.

L1 results. After-edit reminders with the user message role yield $-3.2$ points of net preference, as shown in Table 12. MiniMax-M2.5 gains $+12.5$ points, whereas GLM-5.2 and Qwen3.7-Max each lose $12.5$ points.

L2 results. Early user rechecking yields a small positive net preference of $+2.7$ points, as shown in Table 12. GLM-5.2 and DeepSeek V4-Pro gain $+9.4$ and $+8.9$ points, while DeepSeek V4-Flash and Qwen3.7-Max decline by $8.0$ and $4.3$ points. These comparisons suggest a modest benefit from reopening work, with substantial variation across models.

L3 results. All four review designs have positive net preference, ranging from $+13.2$ to $+23.8$ points, as shown in Table 12. Changes-based review with required visual inspection performs best at $+23.8$ points, compared with $+21.2$ points without required visual inspection. Trajectory-based review improves from $+13.2$ to $+21.6$ points with required visual inspection. Changes-based review therefore receives higher net preference in both comparisons.

Case study

Overview of the paired trajectories

We examine ten Terminal-Bench cases across the four verification layers, each paired with a baseline run of the same task and model, as shown in Table 13. Each case has a full-page paired trajectory with tool inputs, returned evidence, harness interventions, and final grading.

Vector (PDF)

Vector (PDF)

Vector (PDF)

Vector (PDF)

Vector (PDF)

Vector (PDF)

Vector (PDF)

Vector (PDF)

Vector (PDF)

Vector (PDF)

Verification instructions and execution rules

We reproduce the intervention templates and the rules that place them in the conversation. Prompt wording comes from the harness implementation; line wrapping and dash typography are normalized for typesetting. Braced fields such as {detail} are filled at runtime. The critic prompts are presented as an opening, a strategy-specific first step, and a shared remainder, so the four complete prompts can be reconstructed without repeating identical text.

L0: initial instructions

The verification instructions below are appended to the system prompt or the user’s initial task request.

Verification instructions

L1: execution-time reminders

Periodic reminders follow every third turn after the first edit; after-edit reminders follow each edit. Both stop at four. After tool results, reminders with the system role include the marker below; those with the user role contain only the shared body.

Marker for the system role

Shared reminder body

L2: completion gates

A turn without a tool call triggers the gate. Each single-pass condition sends one message and accepts the next stop. The automated designs use the user message role with the name system-reminder; the System design of Run and check uses the system message role. The User designs use the user message role without this marker. Early-trigger variants also fire at the next turn boundary after 80% of the task’s time limit if the agent has not stopped.

Inspect changes (automated)

Run and check (automated or system)

Recheck before finishing (user)

Reject completion (automated)

Reject completion (user)

L3: independent review

The critic receives the review instructions in a system message. A separate user message contains the task and changes; trajectory conditions also include the agent’s reasoning, messages, and actions. Each critic starts in a fresh context and can inspect the live sandbox. The standard single-pass design ends on approval; an objection returns findings to the agent, whose next delivery is accepted without another review.

Prompt assembly. Choose the opening for the design’s strategy and context, append Work in THREE steps:, then the matching first step and the shared steps below. The common remainder includes the non-mutation rule and the JSON output format.

Task-first opening: changes

Task-first opening: trajectory

Claim-guided opening: changes

Claim-guided opening: trajectory

Task-first step 1

Claim-guided step 1

Shared steps 2–3 and verdict format

Feedback after review. The agent receives an objection as a user message using the template below. The detail field contains unmet requirements and, where available, the critic’s feedback.

Feedback after an objection

Required visual inspection on VisualBench. The visual review conditions use the Claim-guided strategy with Changes or Trajectory context. Required-inspection variants enable the critic’s viewing tool and append the following two clauses; the second accompanies temporal capture support.

Required visual inspection

Temporal inspection for moving artifacts

Review depth and width

The TB scaling sweep uses claim-guided review with changes. Depth $n$ allows up to $n$ critic rounds, subject to the remaining budget. Subsequent rounds receive the critic’s prior checklist and feedback.

Review width. Width $n$ uses $n$ critics with separate contexts on the same delivery. The implementation runs the reviews sequentially in the shared sandbox; critics do not receive one another’s assessments. Any returned objection sends the work back to the agent, with dissenting findings grouped by critic. Acceptance requires no returned objection. The width sweep uses one panel round, so an agent’s subsequent revision is not reviewed again.

Prompts for additional review rounds

Prior-review context for subsequent critic rounds

Continuation after an objection