betaFeedback

WorksComputer Science

VeriCUA: Verification as Agentic Exploration for Computer-Using Agents

Abstract

Building and scaling computer-using agents (CUAs) across desktop, mobile, and web environments requires reliable task-completion verification for data curation, evaluation, and reinforcement learning. Typical automated approaches prompt vision-language models (VLMs) to judge trajectories or generate programs to inspect application state. However, both rely on partial views of complex digital environments. VLM judges are restricted to supplied observations, while generated programs depend on accessible state. These views can omit decisive evidence or leave completion judgments unsupported. Tailoring separate verifiers to different platforms and access settings adds further engineering cost. We present VeriCUA, a platform-agnostic framework for verification through agentic exploration. It treats verification as dynamic evidence gathering and assessment within a shared procedure for online and offline settings across platforms and observation modalities. After inspecting the available context to construct a task rubric, the verifier selects GUI or code checks for unresolved requirements and combines their findings under explicit evidence rules. Experiments show improved agreement with reference labels over the compared baselines across these settings, with the largest offline gains on hard cases and stronger rejection of false completion claims online. Further analyses connect these gains to complementary evidence and test-time exploration.

Introduction

Vision-language models (VLMs) have enabled computer-using agents (CUAs) to navigate the digital world through software interfaces (Rawles et al., [8]; Xie et al., 2024; Zhou et al., 2024). Given a natural-language instruction, an agent observes the interface, plans, and acts, leaving a trajectory of thoughts, actions, and observations. Digital tasks span applications in which success can depend on visual presentation, application records, or changes to persisted files. A trajectory can appear successful even when edits remain unsaved or an operation fails without a visible warning. The agent may also report completion after fulfilling only part of the instruction. Accepting such a run as successful produces a false positive, making the distinction between attempted actions and their actual effects central to verification. Developing these agents therefore requires reliable judgments throughout evaluation, data curation (Sun et al., [11]), and reinforcement learning (Xue et al., 2026a), including applications for which no task-specific checker has been supplied.

An execution can be verified online from the resulting machine state or offline from its retained trajectory. Evidence can come from rendered interfaces, structured observations, or artifacts such as files and database records. VLM judges assess trajectories from selected observations (Pan et al., [7]), while automatically generated checkers inspect application state through model-written programs (Xue et al., 2026b). A fixed set of frames may omit a decisive effect, and a generated query may target state that does not answer the verification question. Platform-native signals support more specialized judgments and validation (Chae et al., [1]; Sun et al., [12]), but their availability varies with the application and access setting. Agentic approaches actively inspect trajectories or environments (Cui et al., [2]; Sanayei et al., [9]; Wan et al., 2026). A shared procedure must seek relevant evidence under these access constraints and ground its conclusions in the observations.

Corresponding author(s): [email protected], [email protected]

To address these evidence and access limitations, we present VeriCUA, a platform-agnostic framework that formulates CUA verification as agentic exploration (Figure 1). It provides a shared procedure for offline and online verification using the sources and inspection tools available in each setting. Verification begins after the executing agent’s attempt ends. The verifier first explores the available context to understand the app and its task artifacts, then uses these observations with the instruction to construct a rubric of fine-grained requirements. This context helps interpret task concepts and locate potential evidence, especially when the application is unfamiliar. The verifier chooses GUI or code tools to check the rubric’s requirements. Findings from these checks reveal what remains unresolved and guide further inspection of relevant frames, artifacts, or another available source.

Figure 1. VeriCUA inspects the available environment or recording before constructing a rubric, then gathers evidence to verify its requirements.

VeriCUA’s unified design separates the verification workflow from the interfaces through which evidence is obtained. Offline verification navigates retained screen frames and action logs, using structured observations such as accessibility trees when recorded. Online verification can inspect the accessible machine state left by the executing agent through GUI or code tools, including files, databases, and system services. These sources answer different questions about the requested outcome. An action log records an attempted edit, while a file read can show whether the edited content persisted. A report that the task is complete remains a claim to verify. The framework links findings to their supporting observations and records source properties such as frame timing or state persistence. Explicit evidence rules combine the findings into a task verdict while retaining which requirements are supported, refuted, or unresolved.

We evaluate this shared procedure in offline and online scenarios on multiple platforms. VeriCUA achieves higher agreement with reference labels than the compared baselines in both settings. It also rejects more false completion claims than the VLM judge in the online study. The action-space study shows that adding code-based state reads enables more decisions without increasing false positives. At a matched number of verification passes, adding visual evidence improves agreement and decision coverage compared with repeating the same configuration.

Our contributions are summarized below:

  • General-Purpose Agentic Verification. VeriCUA formulates CUA verification as agentic exploration, providing a shared paradigm for offline and online tasks across platforms and available evidence sources.

  • Adaptive Evidence Gathering. Context inspection grounds task rubrics in the available environment or recording. The verifier selects GUI or code checks for unresolved requirements and combines findings under source-specific evidence rules.

  • Evaluation Across Platforms. Experiments show improved agreement with reference labels across the evaluated platforms. Analyses show how evidence access and complementary exploration affect decision coverage and reliability.

Related Work

Computer-Using Agents and Their Evaluation. Computer-using agents are evaluated on web, mobile, and desktop tasks ([8] (Xie et al. 2024; Zhou et al. 2024)), with dedicated environments covering scientific applications ([12]). These benchmarks typically use task-specific programmatic evaluators to check the resulting application state. Human-labeled trajectory datasets provide reference judgments across applications ([12]), while audits of benchmark evaluators identify errors in the checks used to produce reference labels ((Xie et al., 2025)). Benchmarks of environment-aware judges assess evidence acquisition and verification of states and execution processes ([10]). Completion judgments also support data curation and training beyond benchmark evaluation. Trajectory synthesis pipelines use verifier scores to select examples ([11]), and continual reinforcement learning relies on automated feedback during environment adaptation ((Xue et al., 2026a)). Rubric-based reward modeling further uses synthetic criteria derived from preferred and rejected responses ([6]). Safety-oriented evaluation additionally examines whether agent behavior satisfies workflow constraints ([12]). The range of applications and evaluation criteria creates demand for verification that can locate relevant evidence through available interfaces, including tasks without a dedicated evaluator.

Verifying Trajectories and Reward Models. VLM judges assess task completion from trajectory observations ([7]), while structured critics evaluate task requirements or intermediate milestones ((Li et al., 2026)). Task-adaptive rubrics can be generated from task descriptions to guide trajectory evaluation ([3]). Generated verifiers construct programs that inspect application state ((Xue et al., 2026b)). Reward benchmarks evaluate the reliability of trajectory judgments against reference labels across tasks and platforms ([5, 12]). Agent-based evaluators assess intermediate task requirements in code-generation workflows ((Zhuge et al., 2025)). Agentic approaches gather additional evidence by iteratively inspecting recorded trajectories ((Wan et al., 2026)) or interacting with the execution environment ([2]). Evidence-first verification separates investigation from validation, checking proposed claims against their state or trace sources before accepting rubric verdicts ([9]). These approaches motivate both seeking task-relevant evidence and assessing how it supports a judgment. VeriCUA builds its rubric after inspecting the available environment or recording, then selects GUI or code checks for unresolved requirements. Source-aware rules combine the resulting findings within a shared offline and online verification workflow.

Method

Method

Verification as Active Evidence Gathering

Given an instruction gg, an executing agent produces a trajectory τ=[o0,a0,…,oT]\tau= [o_0, a_0, \ldots, o_T] of observations and actions, together with any recorded reasoning, and leaves the environment in state sTs_T. VeriCUA begins after this attempt ends and assesses whether the execution fulfilled gg using the sources available to the verifier. Although task completion is binary, the verifier outputs y∈{SUCCESS,FAIL,ABSTAIN}y \in\{\mathrm{SUCCESS}, \mathrm{FAIL}, \mathrm{ABSTAIN}\}, allowing abstention when the gathered evidence is insufficient to support a decision.

The available sources S={S1,…,Sm}\mathcal{S} = \{S_1, \ldots, S_m\} can include a retained trajectory, recorded accessibility trees (a11ytrees), a frozen machine snapshot, or accessible post-execution machine state. Offline verification uses retained observations, while online verification can inspect exposed machine state alongside any retained execution records. When a requirement concerns a change made during execution, an available pre-execution reference can also serve as a source.

The shared workflow in Figure 1 begins with context inspection, which gathers information about the application and available observations into context cc. Rubric synthesis uses gg and cc to produce requirements RR that make the instruction checkable. During active exploration, the verifier selects checks for these requirements and uses their results to decide what to inspect next. Finally, rule-based arbitration combines the findings under the setting’s evidence rules and handles any unresolved requirements.

At step kk, an inspection action αk=(vk,xk)∈A(S)\alpha_k = (v_k, x_k) \in\mathcal{A}(\mathcal{S}) pairs a tool vkv_k with arguments xkx_k. Let Ek−1\mathcal{E}_{k-1} denote the inspection record before this step, initialized with any retained context-inspection outputs E0\mathcal{E}_0. The verifier selects and executes the next check according to

αk∼πV(⋅∣g,c,R,Ek−1),Execute⁡(αk,S)⟶(o~k,tk),(1)\alpha_k \sim\pi_V(\cdot\mid g,c,R,\mathcal{E}_{k-1}), \qquad\operatorname{Execute}(\alpha_k,\mathcal{S}) \longrightarrow(\widetilde{o}_k,t_k), \tag*{(1)}

where o~k\widetilde{o}_k is the returned tool observation and tkt_k records its evidence type and source attributes, assigned by the tool-execution layer (the harness). The executed check (αk,o~k,tk)(\alpha_k,\widetilde{o}_k,t_k) and its associated findings are added to form Ek\mathcal{E}_k. A finding assesses how an observation supports or refutes a particular rubric requirement. The verifier continues until the hard requirements are settled or its inspection budget is exhausted. After LL inspection steps, the arbiter derives a verdict from EL\mathcal{E}_L.

Environment-Aware Rubric Synthesis

An instruction specifies the requested result but may leave its application-specific interpretation implicit. A request to save a note, for example, does not necessarily identify the application’s storage directory or explain which view reveals whether the note persisted. Generating requirements without this context can introduce assumptions about files or interface elements that the application does not expose.

Before constructing the rubric, VeriCUA uses the available inspection tools to gather context cc. Online inspection can identify application packages, relevant directories, and exposed database schemas; offline inspection reviews the recorded steps, frames, and available structured observations. From the instruction gg and observed context cc, the verifier constructs a rubric R={(ri,hi)}i=1nR = \{(r_i,h_i)\}_{i=1}^{n}, where each rir_i is a checkable task requirement and hi∈{hard,soft}h_i \in\{\mathrm{hard},\mathrm{soft}\} indicates whether it is mandatory for success or an additional check. Requirements can concern a resulting state or a change relative to an available pre-execution reference.

Each hard requirement must be justified by the outcomes requested in the instruction, while context supplies its application-specific meaning and helps locate evidence. In the note-saving example, the requested text and persistence are task requirements; a discovered directory is a place to inspect for evidence. This distinction keeps evidence locations from becoming additional conditions for success. The rubric guides subsequent checks, which assign each requirement a status of MET, UNMET, or UNKNOWN. Soft checks provide additional findings without relaxing any hard requirement.

Adaptive Hybrid Action Space

The action space connects the rubric to the sources that can answer its requirements. Figure 2 distinguishes three views of task state:

Figure 2. VeriCUA gathers evidence from retained trajectories and, when accessible, the current interface and underlying machine state. Each read retains its source attributes.

  • Trajectory. Retained frames, actions, and reasoning record what the executing agent observed and attempted. The verifier can search the action log, inspect selected frames, and read recorded a11ytrees when available.

  • Surface. The current machine interface can be inspected through a new screenshot or a UI hierarchy read. A recorded screenshot belongs to the historical trajectory; this view provides a fresh observation of the accessible machine.

  • Persisted State. Files, application databases, and stored settings can retain effects that the current interface does not show. Code tools can inspect them through file reads, database queries, or queries to exposed services.

The harness records persistence for each read. Alongside stored artifacts, live machine access provides volatile observations such as current UI dumps and harness artifacts.

GUI tools support visual inspection through frame selection, region enlargement, and optical character recognition (OCR), while code tools query exposed machine state. The online verifier can also synthesize a checker: a sequence of permitted reads paired with a predicate that evaluates a requirement. The harness executes the reads; structured predicates use a restricted evaluator, while visual or semantic comparisons use a model judgment. A single online inspection action αk\alpha_k can execute a checker comprising several reads and its predicate evaluation.

The verifier chooses its next inspection from the findings already collected, focusing on requirements that remain unresolved within the available access. A text match may leave persistence unresolved and prompt a file read; an unclear frame may prompt a crop or inspection of a later frame. The harness records the executed tool, arguments, and source metadata, including frame timing or the persistence of queried state. The executed read determines these attributes, which are attached to the resulting findings. The complete tool inventory is given in Appendix § 8.§ 2.

Evidence Discipline

VeriCUA links each finding to an executed read, then applies source constraints and corroboration rules before assigning requirement statuses.

Grounded Findings. Each finding identifies a rubric requirement, states the support or refutation inferred from an observation, and cites the executed check that returned the observation. The harness rejects citations that match no executed call and attaches each valid citation’s observation and source attributes to the finding. Source attributes and corroborating probes then determine how each linked assessment contributes to a requirement’s status.

Source-Aware Evidence. The attested/asserted source labels distinguish observations of task state from records of the executing agent’s intent or claims. A post-action frame, a file read, or a database query can provide attested evidence. The executing agent’s completion message and a pre-action frame showing an intended operation are asserted evidence for that operation’s outcome. Asserted evidence can direct further inspection but cannot alone mark a requirement MET.

Source attributes further constrain what an observation can support, including whether a recorded image precedes or follows the action whose effect is being checked. Online reads distinguish persistent artifacts from volatile interface state, which can answer different questions about the same requirement. For example, a screen displaying edited text can support its visible contents, while a file read can check whether that text was saved. A finding must address the task requirement, and its observation must satisfy the source conditions for the inference being made.

Rule-Based Arbitration. The arbiter applies the setting’s confirmation and refutation rules to the stored findings ELE_L, assigning each requirement a status of MET, UNMET, or UNKNOWN. A MET status requires attested support, while UNMET requires corroborating refutations from separate probes. Evidence that does not settle the requirement leaves it UNKNOWN. The task verdict is SUCCESS when all hard requirements are MET, and FAIL when any is UNMET. If some remain UNKNOWN and none is UNMET, the task is unresolved and follows the setting’s abstention or fallback policy. Aggregation is deterministic for fixed findings and rules; the stored records allow confirmation thresholds to be re-evaluated without repeating inspection actions.

Online and Offline Instantiations

Online and offline verification share the same workflow over the sources illustrated in Figure 2. Access conditions determine the available checks and the treatment of unresolved tasks.

Online. Online machine access allows the verifier to combine GUI inspection with code-based checks of stored artifacts. Task-specific checkers can be reused across repeated executions of the same task with compatible application schemas. Persistent attested reads can confirm requirements, while volatile observations provide corroboration. The verifier abstains on unresolved tasks to avoid assigning training or filtering labels unsupported by the collected evidence.

Offline. VeriCUA audits fixed, time-stamped recordings through multiple paths, seeking support through forward inspection and refutations through backward inspection. A requirement is MET when every completed path provides attested support. For tasks left unresolved, the reported configuration uses an initial judgment: a single-pass VLM assessment of the retained trajectory computed before exploration and withheld from the audit paths. This fallback supplies the final verdict while unresolved requirements remain UNKNOWN. Detailed setting rules and inspection budgets are given in Appendices § 8.§ 1 and § 9.§ 4.

Experiments

Experimental Setup

Benchmarks. In online environments, we run AndroidWorld ([8]) on an Android emulator, OSWorld ((Xie et al., 2024)) in a virtual machine, and ScienceBoard ([12]) in its scientific desktop environment. Each episode is executed by the benchmark’s own agent scaffold, and the verifier probes the resulting machine state after the episode ends. For offline verification, we evaluate recorded WebArena trajectories from three executing agents ((Zhou et al., 2024)) and the human-labeled OSReward trajectories ([12]), which span four platforms and several agents.

Protocol. All verifiers and baselines run on gemini-3-flash-preview, and the backbone study re-runs them on gemini-3.1-pro-preview. In online environments, gold is each benchmark’s official evaluator and the executing agents also run on gemini-3-flash-preview. On WebArena we provide a11ytree variants of the input beyond the GUI, to the verifier and the judges alike. The remaining protocol details are provided in Appendix § 9.

Baselines. We compare against a VLM judge, an automatically generated verifier, and two structured judges. Implementation details are provided in Appendix § 9.§ 2.

  • VLM judge. A single-pass judge such as TRM ([11]), which assesses the supplied observations in one model call and returns a verdict or a score.

  • Auto-Generated Verifier ((Xue et al., 2026b)). A model-generated evaluator script per task, run once against the machine for a binary verdict.

  • CUAJudge ((Xue et al., 2026a)). A checklist judge that derives key points from the task and checks them against sampled frames in one call.

  • OS-Themis ((Li et al., 2026)). A milestone judge that checks ordered milestones and reviews them.

The VLM judge runs in both settings, the auto-generated verifier only in online verification, and CUAJudge and OS-Themis only in offline verification.

Metrics. We report agreement with gold, precision, recall, and F1 for the success class, and the false positive rate, with abstentions counted as FAIL. For verifiers that abstain we also report the decision rate and the accuracy on decided episodes. Verdict conventions are given in Appendix § 9.§ 5.

Online Verification

Accuracy and False Positives. Across the three online environments, VeriCUA has the highest accuracy and lowest false-positive rate (Table 1). The auto-generated verifier has higher accuracy than the VLM judge in each environment. The largest gap is on ScienceBoard, where generated checks are not reliably aligned with the target scientific software. Scripts constructed without context inspection can query incorrect state locations. VeriCUA inspects the application context before generating checks, and its false-positive rate is roughly a quarter of the generated verifier’s. The generated verifier has higher recall on OSWorld and ScienceBoard, while VeriCUA has higher precision and accuracy.

EnvironmentJudgePrecRecF1AccFalse Positive Rate ↓DecidedDecided Acc
AndroidWorldVLM judge79.369.774.278.415.4100%78.4%
AndroidWorldAuto-Generated verifier77.884.881.282.416.9100%82.4%
AndroidWorldVeriCUA90.687.989.290.57.782%95.1%
OSWorldVLM judge68.275.071.473.427.9100%73.4%
OSWorldAuto-Generated verifier70.678.174.275.925.9100%75.9%
OSWorldVeriCUA80.371.375.579.513.978%88.7%
ScienceBoardVLM judge42.960.050.071.624.8100%71.6%
ScienceBoardAuto-Generated verifier50.972.559.876.921.7100%76.9%
ScienceBoardVeriCUA77.460.067.686.45.470%91.6%

Table 1. Performance in online environments.

Deciding and Abstaining. Both baselines return a verdict on every episode, so their decided accuracy equals their overall accuracy. VeriCUA abstains when its findings cannot settle the task or when decided verification passes disagree. Its accuracy on decided episodes is higher across all three environments. Counting abstentions as FAIL can lower recall without adding false positives. This policy favors conservative selection of successful trajectories for data curation or reward assignment.

Offline Verification

Across Benchmarks and Difficulty. In offline verification, VeriCUA has the highest accuracy and balanced accuracy across the benchmarks in Table 2. On OSReward, its accuracy advantage over the strongest compared judge is larger on the hard subset than on the full set. The gains on hard trajectories support active evidence gathering when task completion is difficult to assess from fixed observations.

JudgeOSReward-HardOSRewardWebArenaAndroidWorldOSWorld
AccBalAccAccBalAccAccBalAccAccBalAccAccBalAcc
TRM57.464.882.283.179.679.888.389.680.373.7
CUAJudge63.462.583.782.681.381.891.290.783.478.2
OS-Themis65.763.280.279.875.474.389.489.082.179.0
VeriCUA70.468.387.286.382.783.194.394.186.381.2

Table 2. Performance in offline verification.

Robustness and Evidence Limits. On WebArena, VeriCUA remains ahead across the three rollout agents and when baseline judges receive accessibility trees alone. The per-agent and observation comparisons are reported in Appendix § 10.§ 2. Offline verification is limited by the observations retained in the recording. When these do not resolve the task, VeriCUA uses its initial-judgment fallback. This occurs on 23% of WebArena trajectories. Appendix § 11 presents online and offline examples with the evidence underlying their verdicts.

Analysis

GUI Versus Hybrid Action Space

GUI Only versus Hybrid. On online AndroidWorld, adding code-based state checks to GUI inspection increases the decision rate and recall while leaving false positives unchanged (Figure 3). In the paired analysis, every additional correct decision from the code channel is supported by a file or application-store read. Access to stored state therefore resolves requirements that GUI observations leave unsettled.

Figure 3. Each metric of VeriCUA with GUI actions alone and with the hybrid action space on the online AndroidWorld suite. Hollow markers are GUI only, filled markers the hybrid space, and the column at the right gives the change.

The Effect Of Observation

Different Observations. On WebArena, adding a11ytrees to screenshots raises VeriCUA’s decision rate while leaving accuracy unchanged. Trees alone yield the highest decision rate and lowest cost, at lower accuracy. Table C.2 in Appendix § 10.§ 2 reports the observation-input ablation. The comparison with baseline judges uses trajectories from three rollout agents. Their accuracy changes little between tree-only and combined inputs, and VeriCUA remains ahead under both settings (Figure 4b). Per-agent results are given in Table C.4.

Figure 4. More observation for a single-call judge. (a) The OSReward judge reading the last NN frames of each trajectory, with VeriCUA at the right; OSReward’s default is five. (b) The WebArena judges reading a11ytree(s) alone or with screenshots, against VeriCUA (dashed).

More Observations. We also vary the number of final frames given to the single-call OSReward judge (Figure 4a). Additional frames improve its accuracy on the hard subset, but the judge remains below VeriCUA. Accuracy on the full set changes only modestly.

Test-Time Exploration

Combining Passes. On the multi-pass AndroidWorld suite, three passes use the default configuration of code and structured-state checks. An additional pass enables visual inspection through screenshots and file-image reads, referred to as the vision fallback. All passes and the auto-generated baseline share one restored snapshot per episode. The combination rule retains a decided verdict and abstains when decided passes disagree. Combining the default passes gives modest gains over the strongest single pass. Adding the visual pass improves agreement and decision rate further, both reaching 87% (Figure 5a). The visual pass also improves these metrics when the number of passes is matched.

Figure 5. (a) Combining verifier passes improves agreement and decision rate. (b) Rejection of the agent’s false completion reports.

Gains From A Stronger Backbone

Combined verdicts produce fewer incorrect FAIL decisions than majority voting over the same passes (Table C.5, Appendix § 10.§ 4).

Matched Budget. We also compare accuracy under a matched verification budget on the flash backbone (Table 3). The protocol allocates repeated runs to TRM, a fixed step limit to CUAJudge, and an exploration limit to VeriCUA. VeriCUA retains higher accuracy on OSReward and OSReward-Hard. Restricting its budget produces a small accuracy decrease from the default configuration while preserving the lead over the compared judges.

JudgeOSRewardOSReward-Hard
TRM (N=10)83.558.5
CUAJudge84.864.5
VeriCUA86.468.7

Table 3. Accuracy under matched budgets.

Detecting False Completion Reports. We analyze false completion reports across the three online environments, using an AndroidWorld subset (Figure 5b). VeriCUA rejects about 80% of these reports and outperforms both baselines in each environment. The difference is largest on ScienceBoard, where the generated verifier rarely rejects an incorrect completion claim. Figure D.1 in Appendix § 11 illustrates an AndroidWorld case in which reads of the media store refute the agent’s report.

Gains From A Stronger Backbone

Replacing the flash backbone with the pro backbone improves VeriCUA’s accuracy and balanced accuracy on OSReward-Hard (Table 4). The compared judges show smaller or negative changes, indicating that stronger models benefit the verification methods differently. All frameworks improve in the online AndroidWorld comparison, with VeriCUA retaining the highest accuracy on both backbones (Table C.8, Appendix § 10.§ 5).

JudgeAccAccAccBalAccBalAccBalAcc
FlashProΔFlashProΔ
CUAJudge63.462.7−0.762.558.4−4.1
OS-Themis65.766.1+0.463.261.1−2.1
VeriCUA70.473.1+2.768.369.2+0.9

Table 4. Backbone comparison on OSReward-Hard.

Confident Failures and Gold-Label Audit

For offline verification, we compare initial FAIL judgments with the subset that exploration also resolves as FAIL. Selection raises failure precision on each benchmark while retaining fewer of the initial failure judgments (Figure 6).

Figure 6. Failure precision and retention.

Disagreements can also identify suspected errors in the reference labels. A blinded model adjudicator supports VeriCUA on 63% of the flagged trajectories, with the clearest support on OSWorld. These results suggest that verification can help identify failed trajectories and prioritize suspected label errors for inspection. Appendix § 10.§ 7 details the supporting analyses.

Conclusion

We presented VeriCUA, a platform-agnostic framework for verifying computer-using agents through agentic exploration. The verifier inspects available context before constructing a task rubric, then selects GUI or code checks to gather evidence for unresolved requirements. Source-aware arbitration combines the findings within a shared workflow for offline recordings and online environments. Across the evaluated platforms, VeriCUA improves agreement with reference labels over the compared baselines, with fewer false positives online and the largest offline gains on hard trajectories. Analyses show that complementary evidence sources and verification passes improve decision coverage, while stronger backbones further improve accuracy. These results support agentic verification for CUA evaluation and motivate its use in data curation and reinforcement learning.

Implementation Details

Verification Settings

Table A.1 lists what each setting determines at each stage of the procedure.

StageAspectOnline environmentRecorded trajectory
Inspect contextapplication contextpackages, directories, stores, and exposed schemasrecorded steps, frames, and structured observations
Synthesize rubricgrounded task requirementsinstruction and observed machine contextinstruction and observed recording context
Exploreevidence gatheringadaptive GUI reads and code checks, including synthesized checkersup to KK audit paths, seeking support and refutations
attested typesenvironment read, UI dump, harness artifactpost-effect frame, precise read, computation, structured state
source attributespersistence, persistent or volatileframe timing, before or after the relevant action
Arbitraterequirement confirmationone persistent attested confirmationattested support from every completed path
unresolved taskabstention, counted as FAIL in binary evaluationinitial-judgment fallback

Table A.1. What each setting determines in the shared procedure.

Action Space and Evidence Types

Table Table A.2 lists the verbs by state source and modality, together with the evidence type each one stamps.

State SourceModalityVerbsEvidence TypeClass
actions and reasoningtextfind_steps, inspect_actionagent claimasserted
actions and reasoningcodecomputesandboxed computationattested
screen framesGUIoverview, get_frame, zoom, ocrpost-effect frame; precise readattested
screen framesGUIpre-action frameasserted
a11ytreesstructuredread_a11y, search_a11ystructured stateattested
machinecodefile_exists, find, list_dir, file_stat, read_file, sqlite_query, content_queryenvironment read (persistent)attested
machinecodedumpsys, current_activity, settings_get, getprop, pm_listenvironment read (persistent or volatile)attested
machineGUI, structuredscreenshot, ui_dumpscreenshot; UI dump (volatile)attested
machinetextsubmitted_answer, clipboard_readharness artifactattested

Table A.2. The action space by state source and modality, with the evidence type each verb stamps.

The getter inventory has seventeen verbs, listed in Table A.3 with their persistence tags. The image-reading verbs are enabled in the vision fallback configuration, clipboard_read runs last because it disturbs the screen, and a miss from find counts only as weak evidence.

persistentsettings_get, getprop, file_exists, file_stat, read_file, list_dir, find, sqlite_query, content_query, pm_list, view_file_image
volatiledumpsys, ui_dump, current_activity, clipboard_read, submitted_answer, screencap (the screenshot verb of Table A.2)

Table A.3. Persistence tags of the online getter inventory.

Extension Contract. A new source enters as exactly (i) its read verbs, (ii) its evidence types with their attested or asserted classification, and (iii) an initial view for the cold-start judgment; every later stage consumes types, so nothing downstream changes. The accessibility channel entered this way, and the machine channel repeated the pattern with its own inventory.

Evidence Aggregation

On recorded trajectories, up to KK audit paths gather support through forward inspection and seek refutations through backward inspection. The default K=3K = 3 permits early stopping after two paths agree on every hard requirement. A requirement is MET only through attested support from every completed path and UNMET only with corroborating refutations from separate probes. If the evidence does not settle the task, the initial judgment supplies the fallback verdict without adding attested support to UNKNOWN requirements.

In online environments, initial inspection maps the device’s packages, stores, and schemas before rubric synthesis. The observed context informs the rubric and subsequent checker construction; each requirement receives checkers on more than one surface where possible; predicates are evaluated by a restricted evaluator or, for semantic comparisons, by a judge call; a requirement is MET by one persistent attested confirmation and UNMET by two corroborating refutations (the unmet_min threshold, which trades decided episodes for precision when raised). An unresolved task produces an abstention, which binary evaluation counts as FAIL. Across passes, the combination rule takes the decided verdict and abstains when two decided verdicts disagree.

Read-Only Enforcement

On recordings the read-only property is structural, since every verb is a pure function of files on disk. On a machine the verbs come from a fixed allowlist, arguments that could smuggle shell control are rejected, and a digest of the plan’s persistent targets is compared before and after execution.

Online Environment Adapters

AndroidWorld. In online environments, VeriCUA runs against AndroidWorld on an Android emulator. The harness has root access over adb, so every store the benchmark’s own evaluator can read is readable by the verifier. Each episode is executed by the benchmark’s agent scaffold; when it ends, the harness saves the device state as a snapshot and verification runs against that snapshot, so any verification pass can be replayed on the identical end state. Before the episode starts, the harness records a pre-state anchor of the application storage, against which requirements about a change are compared. The agent’s submitted answer is extracted by the harness and exposed through the submitted_answer getter as the artifact under test.

The emulator is a headless Pixel 6 image at API level 33 (Android 13) with root access over adb. Snapshots are taken through the emulator console once the official evaluator has run.

OSWorld and ScienceBoard. OSWorld runs in a virtual machine and ScienceBoard in its own scientific desktop environment. Both are verified through the same allowlisted read-only getters over the machine, and ScienceBoard’s evaluator reads the state of the scientific applications inside the VM. Both use the virtual machine images that the benchmarks ship, without modification.

Recorded Observation Processing

A recorded trajectory enters as the agent’s actions and reasoning, its screen frames, and, when the platform recorded them, its a11ytrees. Frame semantics, whether frame ii captures the state before or after action ii, is declared per dataset and determines the post-effect and pre-action evidence types. The a11ytrees are condensed before they are read. OSWorld trajectories are pure GUI with pre-action frame semantics; WebArena trajectories carry a11ytrees.

The condenser writes one line per node of the browser’s accessibility dump, indented by depth, with the node id, role, name, and ten retained properties such as value, checked, focused, and url. Ignored nodes and nameless nodes without properties are dropped, and names are cut to 110 characters. A read returns at most 150 lines with a note on how many nodes were left out, a query returns at most 40 matching lines, and a search across steps returns the first match in each of up to 12 steps. Each trajectory enters as one record that lists, per step, the agent’s thought and action, the page URL, and the paths of the screenshot and the tree.

Experimental Setup and Evaluation Protocol

Evaluation Sets

Table B.1 summarizes the evaluation sets.

SetSizeGoldNotes
OSReward-Hard284human labelshard subset of OSReward
OSReward full1019human labels
WebArena715official checkers239 / 235 / 241 rollouts by Claude Sonnet 4.6, Gemini 3.1 Pro, Gemini 3 Flash
AndroidWorld116official evaluatorM3A Agents
OSWorld, online361official evaluatorone rollout per task; actor success 44.3%
ScienceBoard, online169official evaluatoractor success 23.7%

Table B.1. Evaluation sets.

Multi-Pass Suite. The multi-pass suite keeps every snapshot on which the agent claimed completion but gold says FAIL, on which the configurations disagree, or on which any pass abstained.

Baseline Implementations

Offline Verification. TRM scores the trajectory from 1 to 5 in a single call from the instruction, every thought and action, and the last three frames; CUAJudge checks up to six task-derived key points against the first and last of up to six frames in a single call; OS-Themis verifies up to four ordered milestones over up to eight frames and reviews them in a second call.

The three offline judges run under one harness at temperature 0, with images downsampled to a long side of 1280 pixels, the first and last frames pinned, no self-consistency voting, and a parse failure mapped to FAIL. TRM thresholds its 1–5 score at 4 and is the only judge that sees the agent’s thoughts. CUAJudge collapses the original four-stage pipeline into one call, dropping the dedicated frame-selection and init-versus-final modules. OS-Themis collapses the multi-agent critic to two calls with the budget cut to four milestones and eight frames against the original six and twelve; Table C.6 shows that restoring the original budget changes little.

Online Environments. The online VLM judge answers in one word from the instruction and the final screenshot and never abstains; the auto-generated verifier writes one evaluator script per task with full device access and runs it once.

The auto-generated verifier receives the instruction with its concrete parameters, the same generic device context our synthesis receives, unrestricted host-side adb and Python, and the agent’s submitted answer on answer-scored tasks. It does not receive the trajectory, the results of context inspection, or the pre-state anchor, and it runs last on each end state.

Majority Voting. Majority voting combines the same three verifier draws by vote with abstentions folded to FAIL.

Verifier Configuration

Offline Verification. The default runs K=3K = 3 paths and stops after two when they agree on the status of every hard requirement, which happens on about two fifths of the trajectories. A path spends up to four tool calls gathering support and then at least four, or two per hard requirement, hunting refutations. Each call returns one observation, frames are downscaled to at most 1288 pixels on the long side, and only the last eight images stay in context.

The rubric comes from three candidate decompositions consolidated in a fourth call. The sealed initial judgment reads the last five frames together with the thought and action log and is reused only as the fallback.

Online Environments. Rubric synthesis draws two candidate rubrics at temperature 0.7 and consolidates them in a third call at temperature 0; checker synthesis makes two parallel calls at temperature 0.7 and asks for two checkers per hard requirement on different surfaces where the state allows it; context inspection uses one call over at most three packages and six databases, followed by one schema probe; a semantic predicate costs one judge call at temperature 0 over at most three images.

No call carries a token cap, and each has a 180-second timeout with up to six retries.

Evaluation Metrics

The binary evaluation treats SUCCESS as the positive class. Abstentions are counted as FAIL for agreement, precision, recall, F1, and the false positive rate. For verifiers that abstain, we also report the decision rate and the accuracy on decided episodes.

Additional Results and Analysis

Action Space and Evidence Rules

Action-Space Ablation. The values behind Figure 3 are in Table C.1.

Action SpacePrecRecF1AccFalse PositivesDecidedDecided Acc
GUI only86.457.669.177.03/4145%87.9%
Hybrid (default)90.687.989.290.53/4182%95.1%

Table C.1. Action-space ablation in the online AndroidWorld environment.

Evidence Rules And Recall. Figure C.1 shows which arbiter rules carry the result on the first AndroidWorld suite and how episode recall falls with the number of hard requirements.

Figure C.1. Evidence rules and recall on the first AndroidWorld suite.

Observation Inputs and Rollout Agents

VeriCUA By Observation Input. Table C.2 gives VeriCUA’s accuracy, decision rate, share of verdicts resting on structured evidence, and cost under each observation input on WebArena.

InputAccBalAccDecidedStructured EvidenceTokens
Screenshots80.080.033%0%234K
Screenshots + a11ytrees80.080.075%73%252K
a11ytrees76.776.785%93%178K

Table C.2. VeriCUA by observation input on WebArena.

WebArena Per Rollout Agent. The WebArena results combine trajectories collected from three rollout agents. Table C.3 gives the results for each agent and for the combined set.

Rollout AgentOverall
Claude Sonnet 4.6Gemini 3.1 ProGemini 3 Flash
AccBalAccAccBalAccAccBalAccAccBalAcc
VeriCUA (GUI + a11ytree)79.980.384.385.383.883.982.783.1
CUAJudge79.179.882.183.482.681.981.381.8
TRM76.277.079.180.883.481.079.679.8
OS-Themis70.369.880.079.475.974.175.474.3

Table C.3. WebArena results by rollout agent and overall.

Tree-Fed Baselines On WebArena. The judges are re-run with a11ytrees alone as their input, in place of the GUI and a11ytree input of Table C.3, per rollout agent and overall (Table C.4). VeriCUA retains its GUI and a11ytree input in this comparison.

Claude Sonnet 4.6Gemini 3.1 ProGemini 3 FlashOverall
AccBalAccAccBalAccAccBalAccAccBalAcc
VeriCUA (GUI + a11ytree)79.980.384.385.383.883.982.783.1
CUAJudge, tree-fed77.877.983.084.382.680.081.180.8
TRM, tree-fed74.574.980.481.480.977.478.678.1
OS-Themis, tree-fed74.573.579.678.778.074.977.375.7

Table C.4. Accuracy and balanced accuracy on WebArena with the judges reading a11ytrees.

Test-Time Scaling and Budget

Multi-Pass Combination In The Online Environment. The multi-pass AndroidWorld suite compares repeated runs of the default code and structured-state configuration with a run that enables visual inspection, called the vision fallback. These verifier runs and the auto-generated baseline share one restored snapshot per episode. Table C.5 reports individual and combined results. Adding visual inspection improves agreement and decision rate over repeated default runs, including at a matched pass count. The combination rule also produces fewer incorrect FAIL verdicts than majority voting.

PassAgreementDecision RateDecided AccFalse PositivesWrong FAILS
draw 175.772.291.634
draw 279.177.492.134
draw 377.467.094.813
draw with vision fallback76.573.090.535
three draws combined80.978.395.622
three draws + vision fallback87.087.096.022
draws 1, 2 + vision (equal cost)84.385.294.932
three draws, majority vote80.010080.0320
auto-generated verifier78.310078.3421

Table C.5. Multi-pass combination in the online environment.

Baseline Budget. Table C.6 reports the OS-Themis budget ablation.

ConfigurationBalAcc
OS-Themis, 4 milestones / 8 frames (used in the main text)74.2
OS-Themis, 6 milestones / 12 frames (original budget)75.9
CUAJudge76.8

Table C.6. OS-Themis budget ablation, mean balanced accuracy.

Backbone Comparisons

Single-Call Judges On Other Backbones. Results for the single-call judge of OSReward on several backbones appear alongside VeriCUA on gemini-3-flash-preview in Table C.7.

JudgeOSReward-Hard AccOSReward-Hard BalAccOSReward full AccOSReward full BalAcc
VeriCUA (Gemini 3 Flash)70.468.387.286.3
Gemini 3 Flash57.065.287.088.2
Gemini 3.1 Pro61.661.687.988.2
Claude Sonnet 4.659.268.187.788.9
Claude Opus 4.667.368.689.590.2
Claude Opus 4.869.769.789.790.0
GPT-5.567.367.089.589.8

Table C.7. The OSReward single-call judge by backbone, with VeriCUA for reference.

Backbone Study In The Online Environment. Table C.8 repeats the flash-to-pro comparison of Table 4 in the online AndroidWorld environment.

JudgeAcc FlashAcc ProAcc ΔBalAcc FlashBalAcc ProBalAcc Δ
VLM judge78.479.7+1.377.578.5+1.0
Auto-generated verifier82.485.1+2.782.785.4+2.7
VeriCUA90.591.9+1.490.391.8+1.5

Table C.8. Accuracy and balanced accuracy in the online AndroidWorld environment with each framework on the flash and pro tiers.

Computational Cost

Token usage per trajectory or episode is reported in Table C.9.

SettingMethodTokens
offline, WebArena (tree-fed, flash)three judges together (4 calls)43K–61K
offline, WebArena (GUI, pro)three judges together (4 calls)25K
offline, OSReward (GUI)three judges together (4 calls)33K–35K
offline, WebArenaVeriCUA150K–208K
offline, OSRewardVeriCUA345K–378K
online, first suiteauto-generated verifier16.2K (60 s)
online, first suiteVeriCUA39.2K (97 s)
online, multi-pass suiteauto-generated verifier21K
online, multi-pass suiteVeriCUA, per pass43K–51K

Table C.9. Cost per trajectory or episode.

On AndroidWorld, a snapshot load restores the end state in about fifteen seconds. One verifier pass takes about 100 seconds per episode, of which checker synthesis takes about half and predicate evaluation about a third, with getter execution and context inspection taking under ten seconds together.

Confident Failures and Gold-Label Audit

Confident Failure Verdicts. For offline verification, we start with all initial FAIL judgments and retain those for which exploration also yields a decided FAIL. Initial judgments use the last five available frames. Evidence aggregation uses three paths, a met threshold of 0.67, and an unmet fraction of 0.4.

Failure precision is the fraction of predicted FAIL verdicts that match the reference labels. Retention is the selected subset size divided by the number of initial FAIL judgments. Figure 6 connects these two selections within each cohort; Table C.10 provides their counts. WebArena uses its official reference labels, and OSReward uses human labels. OSReward-Hard is included in OSReward full.

DatasetTotal NNInitial FAIL: Correct / nnInitial FAIL: Prec. (%)Selected FAIL: Correct / nnSelected FAIL: Prec. (%)Retained (%)
WebArena715381/43188.4195/21292.049.2
OSReward full1019524/62583.8166/17992.728.6
OSReward-Hard284148/18580.048/5488.929.2

Table C.10. Initial failure judgments and the subset retained after exploration.

Gold-Label Audit. The gold-label audit uses 814 recorded trajectories from AndroidWorld, OSWorld, and WebArena. A trajectory is flagged when a decided VeriCUA verdict contradicts the benchmark’s gold label, which happens on 81 of them. A single-blind adjudicator, a model from a different family than the verifier’s backbone, then re-judges each flagged trajectory from scratch. It sees only the instruction, the agent’s thought and action log, and up to ten evenly spaced frames including the last one, never the verifier’s verdict, the gold label, or the verifier’s reasoning, and it may answer SUCCESS, FAIL, or UNSURE.

A flag counts as confirmed when the adjudicator sides with VeriCUA against gold. Of the 81 flags, 51 are confirmed, 20 refuted, and 10 left UNSURE, with UNSURE kept in the denominator. On OSWorld, the adjudicator confirms 37 of 57 flags. A second adjudicator, run under the same blind protocol, confirms 34 of the 51 flags it decides, excluding its UNSURE cases from that denominator. In the online OSWorld environment, about 18 of VeriCUA’s 28 false positives are episodes that the official evaluator marks FAIL in error.

Case Studies

Each card shows the instruction and the frames or machine reads the verifier cited. The summary below it gives the verdict against gold and the baselines, the rubric with each requirement’s final status, and the typed evidence the arbiter used.

Online Environments

The verifier reads the live machine after the episode ends.

Trajectory evidence and verification summary for a false completion claim

Figure D.1. TRAJECTORY EVIDENCE. INSTRUCTION: Take one video. MOBILE. STEP 5: First clip recording. · · · STEP 8: Saved, records again. · · · STEP 10: Idle, claims done. MEDIA STORE AND FILE READS. $ content_query video/media [_data, date_added] Movies/VID_20231015_153539.mp4 1697384139 Movies/VID_20231015_153638.mp4 1697384198 Movies/VID_20231015_153745.mp4 1697384265 $ list_dir /storage/emulated/0/Movies 612562 2023-10-15 15:36 VID_20231015_153539.mp4 2751463 2023-10-15 15:37 VID_20231015_153638.mp4 4219707 2023-10-15 15:38 VID_20231015_153745.mp4 # episode start 1697384045: three new files. VERIFIER READ: Three new videos, one asked. FOCUSED STATE. MISS REASON: The VLM judge sees an idle camera with a saved thumbnail, and the generated script returns on the first video it finds without counting; both certify the run. VERIFICATION SUMMARY: Gold FAIL; Agent claimed complete; VLM judge SUCCESS; Generated verifier SUCCESS; VeriCUA FAIL. RUBRIC: Exactly one new video since the episode began. The new video sits in a standard media directory. EVIDENCE: content_query on the video media store, persistent: three rows added after the start. list_dir and find over Movies, persistent: the same three files, written one minute apart. Three of the four verifier passes decide FAIL on this evidence; the fourth abstains. A false completion claim refuted from the media store. The agent recorded three clips for one requested video, and both baselines certified the run.

Trajectory evidence showing a mobile instruction, app screenshots, database readout, verifier read, focused state, verification summary, rubric, and evidence

TRAJECTORY EVIDENCE

INSTRUCTION Add the following songs, in order, City of Stars, Hidden Paths, Distant Memories to my playing queue in Retro music.

MOBILE

STEP 5 City of Stars added to the queue

STEP 14 Third song added

STEP 15 Claims done

DATABASE READ AFTER THE RUN

$ sqlite_query music_playback_state.db
"SELECT title FROM playing_queue"
{"title": "City of Stars"}
{"title": "Hidden Paths"}
{"title": "Distant Memories"}
$ ui_dump contains="City of Stars"
cat: /sdcard/_verifier_ui.xml: No such file or directory
# 3 database reads decide; 4 ui_dump reads unavailable

VERIFIER READ The queue, in order, from the app’s store

FOCUSED STATE

MISS REASON The queue lives in the app’s database and does not appear on the final screen; the generated script joined the wrong tables and read an empty queue, and the VLM judge saw only a search view.

VERIFICATION SUMMARY

Gold SUCCESS Agent claimed complete  VLM judge FAIL Generated verifier FAIL VeriCUA SUCCESS

RUBRIC

  • Each of the three songs is in the queue.

  • The songs follow each other in the asked order.

  • The queue holds exactly three songs.

EVIDENCE

  • sqlite_query on the playback database, located through the environment map, persistent: the three titles in the asked order, on two tables.

  • ui_dump, volatile, four attempts: no dump was produced, so the verdict rests on the store alone.

Figure D.2. A success that only the app’s store can attest. The queue is read from the playback database in the asked order, while both baselines fail the run.

Recorded Trajectories

Here the verifier has nothing but the frames and logs the recording kept.

Trajectory evidence for a desktop recording

Figure D.3. A long desktop recording that every judge passes on the finished export. The verifier reads the preview frame and finds the text at the top rather than centered.

Trajectory evidence for a web recording

Figure D.4. A web trajectory whose final screen shows cheap shoes. The a11ytree names the page a keyword search rather than the category page and lists prices above the limit.

Trajectory evidence from a mobile recording showing the instruction, maps, route, focused state, verification summary, rubric, and evidence

Figure D.5. A mobile recording whose answer reads well. The map 18 steps before the end shows a closer park than the one the agent chose, and two of three judges passed it.

Limitations and Broader Impacts

VeriCUA trades cost for evidence. A verification pass issues tens of tool calls and model calls per trajectory, an order of magnitude more than a single-call judge, so it suits filtering and reward assignment better than interactive use. On recorded trajectories it abstains wherever the recording holds no evidence for a requirement. Those abstentions are a coverage limit, and online access lifts it only for environments that can be instrumented. A stochastic verifier also needs its draw-to-draw spread reported beside every comparison, as we do. Reliable trajectory verification lowers the cost of training computer-using agents and of auditing benchmark labels. The same machinery could be misused to certify agents that act on real systems, which is one reason VeriCUA never writes to the environment it verifies.

Prompts

The system prompts below drive rubric synthesis in the two settings and checker synthesis in online environments. Task-specific inputs such as the instruction and the device context are appended at run time.

Recorded-Trajectory Verification

```` You decompose a computer-use task into atomic, individually checkable requirements for a verifier. Output requirements about the FINAL END-STATE the task demands (what a programmatic checker inspecting files/documents/settings afterwards would verify), NOT about the method or the steps taken.

Rules:

  • Each requirement is one concrete, verifiable condition.

  • Honor every quantifier/detail: "all"/"each" => ALL; exact names/values/paths => exact. Split compound asks into separate requirements.

  • Mark hard (must hold for success) or soft (minor/cosmetic).

  • Do NOT invent process requirements ("must have clicked X"); if the end-state is right, the route does not matter.

Output ONLY:

{"requirements": [{"id": "r1", "description": "...", "hard": true}, ...]}

````

Prompt F.1 | Rubric synthesis on recorded trajectories.

Online Verification

You author verification requirements for tasks executed on an Android device.
Given a task instruction, produce the END-STATE requirements a strict human
examiner would check AFTER the task supposedly finished.

Rules:
- END-STATE only: describe what must be true of the device state, never the
  procedure ("brightness setting equals maximum", not "user opened settings").
- Atomic: one checkable fact per requirement. Split name vs content vs location.
- Strict reading: if the instruction names a value, place, or format, it is
  required exactly; include implicit artifacts (a created note must exist AND
  contain the given text).
- hard=true for requirements whose failure fails the task; hard=false for
  cosmetic/soft ones.
- Only what the instruction implies: do NOT add adjacent settings, modes, or
  hygiene the instruction never mentions (no invented requirements).
- COUNTS ARE EXACT: "take one video" means exactly one new video was created,
  not at least one; "log the reimbursable transactions" means exactly that set
  was added — nothing missing, nothing extra. When the instruction quantifies,
  produce a requirement for the exact count or exact set of NEW artifacts
  (relative to the episode start), not mere existence.
- Do NOT bind platform-specific encodings or exact constants the instruction
  does not state: say "the minimum value the system allows", never "(0)".
  Resolving concrete encodings is the checkers' job, not yours.
- Do NOT bind storage locations or app conventions the instruction does not
  name ("a photo was taken" requires a new image in SOME standard media
  location, not specifically DCIM). Location is a requirement only when the
  instruction states one.
Output ONLY JSON:
```json
{"requirements": [{"id": "r1", "description": "...", "hard": true}]}

Prompt F.2 | Rubric synthesis in online environments.

You write CHECKERS that verify end-state requirements on an Android device. A checker is DATA, not a script: a plan of read-only getter calls plus one predicate over their outputs. You cannot write to the device; only these getters exist:

{getter_docs}

{device_context}

Design rules:

  • For each hard requirement produce 2 checkers probing DIFFERENT surfaces or routes where meaningful (e.g. settings vs dumpsys; direct path vs find). If only one sensible surface exists, 2 independent probes of it are fine.

  • Prefer broad reads + predicate (list_dir or find + check the listing) over a single narrow existence probe: the task may have been satisfied at a different valid location or filename than the literal guess.

  • predicate kinds: "code" (one Python expression over the getter outputs) or "semantic" (a plain-language criterion decided by a separate judge). [rules on polarity, exact-name matching, store representations, and screen-only state omitted for space]


Prompt F.3 | Checker synthesis in online environments, shortened where marked.

References

  1. [1]H. Chae, S. Kim, J. Cho, S. Kim, S. Moon, G. Hwangbo, D. Lim, M. Kim, Y. Hwang, M. Gwak, D. Choi, M. Kang, G. Im, B. Cho, H. Kim, J. H. Han, T. Kwon, M. Kim, B. woo Kwak, D. Kang, and J. Yeo. Web-shepherd: Advancing PRMs for reinforcing web agents. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=G2kMro09UV.arxiv.org/abs/2505.15277
  2. [2]C. Cui, J. Huang, S. Wang, L. Zheng, Q. Kong, and Z. Zeng. Agentic reward modeling: Verifying gui agent via progressive trajectory-grounded interaction, 2026. URL https://arxiv.org/abs/2602.00575.
  3. [3]L. Ding. Adarubric: Task-adaptive rubrics for reliable llm agent evaluation and reward learning. 2026. URL https://github.com/alphadl/AdaRubrics.arxiv.org/abs/2603.21362
  4. [5]H. Lin, X. Tan, Y. Qin, Z. Xu, Y. Shi, Z. Li, G. Li, S. Cai, S. Cai, C. Fu, K. Li, and X. Sun. Cuarewardbench: A benchmark for evaluating reward models on computer-using agent, 2025. URL https://arxiv.org/abs/2510.18596.
  5. [6]T. Liu, R. Xu, T. Yu, I. Hong, C. Yang, T. Zhao, and H. Wang. OpenRubrics: Towards scalable synthetic rubric generation for reward modeling and LLM alignment. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 17417–17437, San Diego, California, United States, July 2026. Association for Computational Linguistics. ISBN 979-8-89176-390-6. doi: 10.18653/v1/2026.acl-long.791. URL https://aclanthology.org/2026.acl-long.791/.DOI
  6. [7]J. Pan, Y. Zhang, N. Tomlin, Y. Zhou, S. Levine, and A. Suhr. Autonomous evaluation and refinement of digital agents. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=NPAQ6FKSmK.arxiv.org/abs/2404.06474
  7. [8]C. Rawles, S. Clinckemaillie, Y. Chang, J. Waltz, G. Lau, M. Fair, A. Li, W. E. Bishop, W. Li, F. Campbell-Ajala, D. K. Toyama, R. J. Berry, D. T. Yamgandulu, T. P. Lillicrap, and O. Riva. Androidworld: A dynamic benchmarking environment for autonomous agents. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=il5yUQsrjC.arxiv.org/abs/2405.14573
  8. [9]R. Sanayei, R.-G. Dumitru, S. Hou, M. Rezaei, H. Riaz, R. Vacareanu, M. Yang, Y. Liu, M. Surdeanu, and D. Y. Zhang. Evidence-first agentic verification, 2026. URL https://arxiv.org/abs/2610.17724v1.
  9. [10]W. Shi, Y. Wang, Y. Zhao, Y. Chen, F. Feng, X. Hao, X. Su, Q. GU, H. Su, X. Cai, and X. He. AJ-bench: Benchmarking agent-as-a-judge for environment-agent evaluation. In Findings of the Association for Computational Linguistics: ACL 2026, pages 25371–25413, San Diego, California, United States, July 2026. Association for Computational Linguistics. ISBN 979-8-89176-395-1. doi: 10.18653/v1/2026.findings-acl.1269. URL https://aclanthology.org/2026.findings-acl.1269/.DOI
  10. [11]Q. Sun, K. Cheng, Z. Ding, C. Jin, Y. Wang, F. Xu, Z. Wu, C. Jia, L. Chen, Z. Liu, et al. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5555–5579, 2025.
  11. [12]Q. Sun, K. Cheng, Y. Wang, B. Yang, H. Yan, L. Chen, F. Xu, Z. Ding, N. Chen, J. Cao, et al. Osreward: Instituting standardized evaluation for cross-platform computer-use reward models. arXiv preprint arXiv:2607.28609, 2026a.arxiv.org/abs/2607.28609

Paper details

vlnc:W_k4r6757m
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents