betaFeedback

WorksComputer Science

PACT: Plan-Action Consistency Training by Preserving Correct Plans and Repairing Actions in GUI Agents

Abstract

GUI agents can formulate a correct immediate plan in their reasoning yet emit an action that fails to execute it, creating an execution gap. This recurring mismatch undermines reliable task completion, as a single execution error can derail a multi-step workflow. In outcome-supervised reinforcement learning, broadcasting a negative response-level advantage across all generated tokens risks penalizing valid planning together with erroneous execution. To address this credit-assignment conflict, we introduce PACT (Plan-Action Consistency Training), a training framework that combines reference-aligned plan protection and plan-aligned action repair without altering the original scalar action reward. Using annotated reference actions to identify execution gaps, PACT protects the verified plan by masking negative policy-gradient contributions across its token span. Keeping the generated reasoning unchanged as context, it then favors the reference action over the emitted erroneous action through a preference surrogate with stop-gradient pair weights. Compared with the outcome-only baseline, PACT raises AndroidWorld task success from 60.34% to 68.97% (+8.63 pp), while improving rule-based plan-action consistency on AndroidControl High from 83.71% to 89.74% (+6.03 pp). Adapting PACT to desktop agents also improves task success from 40.92% to 44.17% (+3.25 pp) on the full 369-task OSWorld benchmark. Together, these results establish PACT as an effective credit-assignment framework for bridging execution gaps and improving task completion across mobile and desktop GUI environments.

Introduction

GUI agents must translate visual interfaces and user goals into executable interactions. Yet formulating a correct immediate plan in generated reasoning does not guarantee that the emitted action executes it. For example, an agent may state that it will tap a visible + button, yet emit an application-launch command instead (Figure 1). This execution gap is distinct from a planning error: the response already contains a valid immediate plan, but the final action fails to realize it. Building on prior diagnostic studies of reasoning–execution gaps in mobile agents [5], we investigate how training can preserve valid plans within failed responses while repairing their execution.

Figure 1. Execution gaps and targeted correction. Top: Both models continue the same supplied Pinterest reasoning; O launches the app, while PACT clicks the intended + control. Bottom: With independently generated reasoning and actions on Moon, O states a downward swipe but emits upward finger motion, while PACT executes the downward swipe. On-screen circles mark the click or gesture start; swipe arrows show finger motion. The separate training schematic illustrates reference-aligned plan protection of selected plans and additional action repair under a negative response advantage.

In reinforcement learning (RL), such execution gaps create a credit-assignment problem. Several GUI-agent methods use Group Relative Policy Optimization (GRPO) or its variants to learn from action feedback [7, 9]. In implementations that broadcast one response-level advantage across reasoning and action tokens [17], a negative advantage penalizes valid planning alongside erroneous execution. A scalar thought–action consistency reward [9] does not determine the direction of correction: higher agreement can result from adapting a plan to justify an incorrect action, and a flawed plan can be acted consistently. Effective correction therefore requires identifying which component the annotated reference action supports preserving.

Our central principle is to preserve a reference-supported plan and repair the action that fails to implement it. Here, a plan denotes the intended next GUI interaction expressed in the generated reasoning, and the plan span denotes the tokens expressing it. Verifying reference support for parameterized GUI actions requires checking spatial or textual arguments beyond action type alone, enabling selective credit assignment that shields valid plans while repairing failed actions.

We present PACT, which instantiates this principle within a group-relative RL pipeline while leaving the original scalar action reward intact. First, a conservative rule-based selector identifies responses featuring a reference-supported plan but an incorrect, inconsistent final action. Second, reference-aligned plan protection masks negative policy-gradient contributions within the plan span, leaving the advantages on action tokens and the remaining reasoning unchanged. Third, plan-aligned action repair pairs a reference-derived positive action suffix with the model’s emitted erroneous suffix under the same decoded reasoning trace via a stop-gradient pair-weight preference surrogate in the spirit of direct preference optimization (DPO) [15], outside the original GRPO rollout group. Both mechanisms operate strictly during training; inference retains the standard single-pass reasoning–action interface without extra reflection or re-planning.

PACT improves interactive task completion over the original GRPO baseline, reaching 68.97% success on AndroidWorld [16] (+8.63 pp) and 44.17% on the full OSWorld benchmark [21] (+3.25 pp). It also achieves 89.74% rule-based plan–action consistency on AndroidControl High [12] (+6.03 pp), alongside improved visual grounding on ScreenSpot-v2 Mobile [20].

Three complementary diagnostics examine why these gains arise. Fixed-plan continuation isolates action generation, showing that plan-aligned action repair improves execution of a valid plan. Aligned-state retention analyses demonstrate that reference-aligned plan protection reduces regressions on baseline-correct actions. Free-generation comparisons further show that protecting plans preserves more valid plans and enables more joint repairs, supporting the complementary roles of protection and repair beyond surface plan–action agreement.

Our main contributions are threefold:

  • Reference-aligned plan protection and plan-aligned action repair. PACT uses reference-supported plans to decouple plan preservation from action correction without altering the scalar action reward.

  • Task improvements across mobile and desktop environments. Comparisons against outcome-only and two repair baselines demonstrate consistent gains in dynamic task completion on AndroidWorld and OSWorld, supported by consistency and grounding evaluations on AndroidControl and ScreenSpot-v2 Mobile.

  • Mechanistic evidence on repair and retention. Through

fixed-plan continuation, free-generation, and baseline-correct retention analyses, we characterize the repair–retention trade-off while disentangling plan correctness, action validity, and plan–action consistency.

Related Work

Reasoning–execution alignment in GUI agents. Reliable GUI interaction demands both sound planning and faithful execution, motivating fine-grained evaluation beyond aggregate task success. VenusBench-Mobile [6] decomposes user-centric agent evaluation across perception, understanding, decision, action, and memory capabilities, while Dong et al. [5] specifically diagnose reasoning–execution gaps by separating the correctness of stated plans from that of emitted actions. On the training side, Faithful-Agent [9] combines evidence-grounded supervision, thought–action consistency rewards, and a guided advantage estimator, whereas MobileIPL [10] derives thinking-level preferences by propagating action feedback through reasoning trees to improve planning. In contrast, PACT uses reference-supported plans within unsuccessful responses to determine the direction of correction—preserving valid plans while repairing only their failed execution.

Fine-grained preference learning and action correction. Building on the pairwise objective of DPO [15], Step-DPO [11] targets individual reasoning steps conditioned on a correct prefix, whereas Agentic-DPO [2] contrasts expert actions against sampled mistakes at expert states. From a rollout-augmentation perspective, Octopus [4] recombines sampled trajectories and masks specific segments to separate self-correction from direct reasoning. PACT instead retains the model’s own decoded reasoning trace as the shared conditioning context to contrast the reference-derived action suffix against the emitted erroneous suffix.

Credit assignment within failed responses. Localized supervision also raises a credit-assignment question: which parts of an unsuccessful response should be penalized? In standard GRPO [17], a single group-relative advantage is broadcast across every token of a sampled response. Recent methods refine this assignment based on intermediate correctness, ranging from retrieval-calibrated negative advantages in CalibAdv [19] to teacher-identified first-error boundaries in Cliff [8]. In GUI agent training, UI-Venus-2 [18] introduces structured action-aware distillation to condition token-level supervision on action correctness and distinguish action-type errors from parameter errors. Within outcome-supervised GRPO, PACT masks negative policy-gradient contributions on reference-supported GUI plan spans and repairs failed action suffixes under the same plan, while keeping the scalar action reward intact.

Problem Formulation

Let xx denote the agent’s observable input context, comprising a user instruction, the current interface screenshot, and the available interaction history. Given xx, the policy πθ\pi_\theta autoregressively generates a reasoning trace zz followed by a GUI action aa:

πθ(z,a∣x)=πθ(z∣x)πθ(a∣x,z).(1)\pi_\theta(z,a \mid x) = \pi_\theta(z \mid x)\pi_\theta(a \mid x,z). \tag*{(1)}

During training, each context xx is paired with an annotated reference action a⋆a^\star.

The plan span in zz expresses the next intended GUI interaction. Building on reasoning–execution diagnostics [5], Table 1 defines TT, AA, CC and joint repair, and summarizes the four correctness regimes and their training implications in PACT.

MetricSuccess criterion
TTThe current plan is correct for the task.
AAThe action is valid for the current task and state.
CCThe action implements the model’s stated plan.
Joint repairT=A=C=1T = A = C = 1 for the same plan–action pair.
TTAATraining implication
11Original action-reward objective.
10Protect the plan; repair the action.
01Original objective; no forced plan alignment.
00Original objective; no reward for error agreement.

Table 1. Step-level judgments and training implications in PACT. (a) Definitions (1: satisfied; 0: otherwise). (b) Correctness regimes. Protection and repair apply only to eligible gaps in Eq. (2) (C^i=0\hat{C}_i = 0).

All sampled responses remain in outcome-supervised GRPO. PACT adds protection and repair for reference-supported plans whose actions fail to implement them. For incorrect plans, correction instead requires revising the reasoning; valid actions should not be redirected to follow an incorrect plan.

During training, a rule-based selector evaluates each sampled response ii relative to its reference action ai⋆a_i^\star. Let Ji∈{0,1}J_i \in\{0,1\} indicate whether the response admits the required reference-based judgments, and let T^i\hat{T}_i, A^i\hat{A}_i, C^i∈{0,1}\hat{C}_i \in\{0,1\} denote the resulting proxies for TT, AA, CC. The execution-gap eligibility flag is

Ei=Ji[T^i=1, A^i=0, C^i=0].(2)E_i = J_i \left[\hat{T}_i = 1,\ \hat{A}_i = 0,\ \hat{C}_i = 0\right]. \tag*{(2)}

Unjudgeable responses (Ji=0J_i = 0) remain in the base RL objective without auxiliary supervision. These reference-relative proxies govern training-time selection, whereas benchmark evaluation and mechanism diagnostics use independent rule-consistency and screenshot-based semantic assessments.

PACT

Figure 2 summarizes the original GRPO path and PACT’s two complementary training mechanisms.

Figure 2. PACT training framework. Reference-verified execution-gap selection identifies reference-supported plans paired with incorrect, inconsistent actions. Reference-aligned plan protection zeros negative policy-gradient coefficients on the plan span. Plan-aligned action repair favors reference-derived over erroneous action suffixes under the same input and reasoning trace. Both feed the GRPO actor update.

Retaining the original action reward

PACT retains the original scalar action reward without introducing any plan-consistency bonus. For a sampled response y=(z,a)y=(z,a), the standard parseable branch of the mobile scorer combines a format score, an action-type match indicator, and an action-specific parameter score:

R0(y,a⋆)=rfmt(y)+1[τ(a)=τ(a⋆)](1+P0(a,a⋆)),(3)R_0(y,a^\star)=r_{\mathrm{fmt}}(y)+\mathbf{1}[\tau(a)=\tau(a^\star)](1+P_0(a,a^\star)), \tag*{(3)}

where τ(a)\tau(a) denotes the action type. All original parsing, missing-action, and grounding branches remain unchanged, with action-specific scoring rules and coordinate tolerances detailed in the supplementary material.

For each prompt, the pre-update policy πold\pi_{\mathrm{old}} samples KK responses. Outcome-supervised GRPO [17] computes the group-standardized advantage

Ai0=R0,i−μg(i)sg(i)+10−6,(4)\mathcal{A}_i^0=\frac{R_{0,i}-\mu_{g(i)}}{s_{g(i)}+10^{-6}}, \tag*{(4)}

where g(i)g(i) is the prompt group of response ii, and μg(i)\mu_{g(i)} and sg(i)s_{g(i)} are its reward mean and sample standard deviation. The baseline broadcasts Ai0\mathcal{A}_i^0 to valid response tokens in a clipped token-mean policy loss regularized by a KL term against the fixed supervised fine-tuned (SFT) reference πref\pi_{\mathrm{ref}}. Only sampled rollouts enter these groups; repair suffixes never affect group statistics (μg(i),sg(i))(\mu_{g(i)},s_{g(i)}) or base rollout rewards.

Reference-verified execution-gap selection

The selector evaluates the eligibility flag EiE_i in Eq. (2). In the mobile implementation, it scans the final four sentences of ziz_i in reverse order for an explicit immediate plan, rejecting negation, multi-step ambiguity, and conflicting action types. Candidate plans are extracted solely from the reasoning text ziz_i prior to reference comparison, never inferred retrospectively from the emitted action aia_i.

Judgeability (Ji=1J_i=1) requires a parseable reference, a unique emitted action block, comparison metadata, and a non-terminal plan with checkable parameters: pointing actions require a unique explicit coordinate; scrolling actions require a single direction; text-entry and app-launch actions require a unique quoted argument; and supported parameter-free system actions require a valid action type. Grounding-only examples, terminal references, and unverifiable plans (Ji=0J_i=0) remain active in the base RL objective but are excluded from auxiliary selection.

For judgeable responses, T~i\widetilde{T}_i checks the plan parameters against ai⋆a_i^\star, A^i\widehat{A}_i tests whether the base content score reaches the configured full-credit threshold, and C^i\widehat{C}_i checks whether the emitted action aia_i matches the plan under distinct compatibility tolerances. When Ei=1E_i=1, span localization and pair construction proceed independently, with extraction and fallback rules in the supplement.

Reference-aligned plan protection

Let Mi,tD∈{0,1}M^D_{i,t}\in\{0,1\} indicate tokens within the localized plan span. PACT modifies the token-level policy advantage as

A~i,t={max⁡(Ai0,0),Ei=1 and Mi,tD=1,Ai0,otherwise.(5)\widetilde{\mathcal{A}}_{i,t}= \begin{cases} \max(\mathcal{A}_i^0,0), & E_i=1\text{ and }M^D_{i,t}=1,\\ \mathcal{A}_i^0, & \text{otherwise}. \end{cases} \tag*{(5)}

Without renormalizing group advantages, this clamp zeroes out negative policy-gradient coefficients on the reference-supported plan span while leaving positive advantages, action tokens, and the remaining reasoning tokens untouched. Crucially, protection acts strictly on local policy-gradient coefficients rather than freezing shared model parameters or masking the KL regularizer.

Plan-aligned action repair

For an eligible response yi−=(zi,ai−)y_i^-=(z_i,a_i^-), we construct a positive response yi+=(zi,ai+)y_i^+=(z_i,a_i^+) by replacing only the action block with the training reference ai⋆a_i^\star formatted in the model’s native action syntax and coordinate space, preserving the input xix_i and decoded reasoning text ziz_i. Let P\mathcal{P} denote the retained constructible pairs, where yi+y_i^+ is scored via teacher forcing; unconstructible or overlength replacements are omitted from P\mathcal{P}, while their original rollouts remain in the rollout batch B\mathcal{B}.

Let Mi,tA,±∈{0,1}M^{A,\pm}_{i,t}\in\{0,1\} select the action tags and action tokens, excluding reasoning and trailing tokens. We define the summed action-span log probability as

sθ,i±=∑tMi,tA,±log⁡πθ(yi,t±∣xi,yi,<t±),(6)s_{\theta,i}^{\pm}=\sum_t M^{A,\pm}_{i,t}\log\pi_\theta(y_{i,t}^{\pm}\mid x_i,y_{i,<t}^{\pm}), \tag*{(6)}

so that only action-span tokens enter the auxiliary objective while the decoded reasoning trace ziz_i serves as their shared conditioning context.

We evaluate the log-ratio margin using πold\pi_{\mathrm{old}} and πref\pi_{\mathrm{ref}}, and define the pair weight as

wi=λβB0σ(β[si−,old−si−,ref−si+,old+si+,ref]),(7)w_i=\frac{\lambda\beta}{B_0}\sigma\left(\beta\left[s_{i}^{-,\mathrm{old}}-s_{i}^{-,\mathrm{ref}}-s_{i}^{+,\mathrm{old}}+s_{i}^{+,\mathrm{ref}}\right]\right), \tag*{(7)}

where λ\lambda is the repair weight, β\beta is the preference scale, σ\sigma is the sigmoid function, and B0=∣B∣B_0=|\mathcal{B}| is the original rollout batch size (rather than ∣P∣|\mathcal{P}|). The suffix-repair surrogate is

Lrepairsur(θ)=∑i∈Pstopgrad⁡(wi)(sθ,i−−sθ,i+).(8)\mathcal{L}_{\mathrm{repair}}^{\mathrm{sur}}(\theta)=\sum_{i\in\mathcal{P}}\operatorname{stopgrad}(w_i)\left(s_{\theta,i}^{-}-s_{\theta,i}^{+}\right). \tag*{(8)}

Minimizing Eq. (8) favors the reference-derived suffix ai+a_i^+ over the erroneous suffix ai−a_i^- under the same decoded reasoning trace. With pairs and masks fixed, its gradient at θold\theta_{\mathrm{old}} matches that of the corresponding conditional logistic preference objective [15] when both suffix terms are evaluated at that parameter point; reusing frozen weights wiw_i across actor minibatches yields a linear log-probability surrogate rather than a repeatedly recomputed logistic loss, with the derivation and full rule specifications in the supplementary material.

Collecting all actor terms yields the complete objective:

Lactor(θ)=LPG(θ;A~,B)+ηLKL(θ;B)+Lrepairsur(θ),(9)\mathcal{L}_{\mathrm{actor}}(\theta)=\mathcal{L}_{\mathrm{PG}}(\theta;\widetilde{\mathcal{A}},\mathcal{B})+\eta\mathcal{L}_{\mathrm{KL}}(\theta;\mathcal{B})+\mathcal{L}_{\mathrm{repair}}^{\mathrm{sur}}(\theta), \tag*{(9)}

where η\eta is the unchanged KL coefficient, and λ\lambda is absorbed into wiw_i. Constructed positive responses contribute exclusively to Lrepairsur\mathcal{L}_{\mathrm{repair}}^{\mathrm{sur}}, without entering the rollout batch B\mathcal{B}, the modified policy loss LPG(θ;A~,B)\mathcal{L}_{\mathrm{PG}}(\theta;\widetilde{\mathcal{A}},\mathcal{B}), or the KL term. At inference, PACT uses the standard single-pass reasoning–action interface.

Training configurations

We compare four cumulative GRPO configurations in Table 2. In these configuration names, O denotes Original GRPO, A denotes Action Repair, and R denotes Reference-Verified Execution-Gap Selection.

ConfigurationAction RepairReference-Verified Execution-Gap SelectionReference-Aligned Plan Protection
O✗✗✗
OA✓✗✗
OAR✓✓✗
PACT✓✓✓

Table 2. Cumulative training configurations.

  • O (Original GRPO): no action repair or plan protection.

  • OA (O + A): adds action repair to O on supported, parseable, non-terminal actions below full credit (Ai=0A_i=0), without execution-gap selection or plan protection.

  • OAR (O + A + R): adds reference-verified execution-gap selection (Ei=1E_i=1) to OA, yielding plan-aligned action repair without advantage clamping (Ai,t=Ai0A_{i,t}=A_i^0).

  • PACT: adds reference-aligned plan protection to OAR.

All variants retain the original reward and reference KL. Repair variants construct pairs from their own rollouts using the same reference-action constructor and suffix surrogate. Optimization settings are in § 5 and the supplement.

Experimental Setup

Training. In our primary mobile study, O, OA, OAR, and PACT share the same Qwen3.5-9B vision–language model SFT initialization, whose frozen copy serves as the reference policy πref\pi_{\mathrm{ref}}, along with 10,000 mobile GUI state–action training examples with pixel-space reference coordinates. Each variant is trained for 1,250 training iterations over two epochs, sampling 16 prompts and K=8K=8 responses per prompt at each step to yield B0=128B_0=128 original rollout responses before auxiliary rows are added. Optimization uses one PPO epoch per rollout batch, a learning rate of 10−610^{-6}, rollout temperature 1.0, KL coefficient η=0.01\eta=0.01, and 32 accelerators with FSDP2; repair-enabled variants set λ=β=0.1\lambda=\beta=0.1 and retain at most 64 repair pairs per iteration. To evaluate cross-platform generality, we also train desktop-adapted counterparts of O, OA, OAR, and PACT. Platform-specific training settings, action-space adaptations, and optimization details are provided in the supplementary material.

Results and Analysis

Mobile task completion, consistency, and grounding

Benchmarks and roles. Our evaluation separates interactive task completion, plan–action consistency, and visual grounding. On AndroidWorld [16], we report the task success rate across its 116 interactive tasks for each checkpoint. On AndroidControl [12], we evaluate 8,444 states from 1,543 episodes in each of the High and Low settings: along with the current screenshot and available history, High provides only the task goal, whereas Low additionally supplies the step instruction. In both settings, we measure rule-based plan–action consistency and reference-action matching for retention and mechanism diagnostics. ScreenSpot-v2 Mobile [20] measures point-in-target hit rate across 501 examples (290 text and 211 icon targets). For desktop interaction, we report task success for each desktop checkpoint on the full 369-task OSWorld benchmark [21], counting a task as successful strictly when its native score equals 1.0.

Generation and measurement. Mobile evaluation uses greedy decoding (temperature 0), a 4,096-token output limit, and up to ten prior interaction steps. On AndroidControl, models observe recorded screenshots, with non-initial states conditioned on each model’s own generated history. OSWorld uses the same environment configuration, greedy decoding (temperature 0), a 100-step limit, and the official native evaluator. Distinct from the training-time selector, our AndroidControl consistency evaluator extracts a function-style or keyword-based action intent from the reasoning trace and compares the inferred action type and recoverable parameters with those of the emitted action. We report

Crule=Nconsistent/NjudgeableC_{\mathrm{rule}} = N_{\mathrm{consistent}}/N_{\mathrm{judgeable}}

with consistent and judgeable counts provided in the supplementary material. For action retention, we measure the proportion of O’s reference-matched states on which a compared variant also matches the reference.

In mechanism diagnostics, fixed-plan continuation conditions each model on the same input and the exact valid reasoning prefix generated by O to isolate action generation, whereas free generation requires each model to produce both reasoning and action from scratch. On these subsets, screenshot-based semantic assessments use language-model assistance and human verification to separately evaluate plan correctness (TT), action validity (AA), and semantic plan–action consistency (CC), complementing benchmark-wide rule-based scores; the supplementary material details sample selection, annotation protocols, and execution.

PACT improves interactive task completion. On AndroidWorld, PACT leads the matched RL configurations in Table 3(B) at 68.97% task success: +8.63 pp over O (60.34%), +6.04 pp over OAR, and +7.76 pp over OA. Table 3(A) provides published context. In a file-copy task, PACT selects the target folder, confirms copying, and finishes in 15 steps. OAR repeatedly types the folder name until the step limit (Supplement Figure S2).

**Task gains are accompanied by stronger plan–action consistency. On AndroidControl High, PACT leads in CruleC_{\mathrm{rule}} at 89.74% (+6.03 pp over O, +3.31 pp over OAR, +9.70 pp over OA), with the highest consistent-action count (7,046 states) and 41.1% fewer action-type mismatches than O, from 1,279 to 753 (Supplement Table S3). On Low, PACT raises CruleC_{\mathrm{rule}} over O from 86.39% to 87.56%, while OAR reaches 88.14% but incurs 217 more action regressions than PACT (§ 6.3).

Table S3. AndroidControl consistency and mismatch counts over 8,444 states per setting. Type, coordinate, and text columns partition the inconsistent judgeable outputs. Bold marks the best values within each setting, including ties.

SettingModelConsistent / judgeableType↓Coord.↓Text↓Crule(%)C_{\mathrm{rule}}(\%)↑
AC HighO6879 / 82181279194183.71
OA6440 / 80461559252280.04
OAR6997 / 80961032214686.43
PACT7046 / 7852753163789.74
AC LowO7182 / 83131081193186.39
OA6534 / 83271753241678.47
OAR7270 / 8248906244888.14
PACT7155 / 817296664587.56

Table S3. AndroidControl consistency and mismatch counts over 8,444 states per setting. Type, coordinate, and text columns partition the inconsistent judgeable outputs. Bold marks the best values within each setting, including ties.

Visual grounding improves alongside task performance. On ScreenSpot-v2 Mobile, PACT raises the point-in-target hit rate from 91.82% (460/501) to 96.01% (481/501), improving on both text and icon targets and exceeding OAR (95.41%; Table 3(B)). PACT has the highest grounding score in Table 3(A). Notably, while PACT and OA tie on aggregate grounding accuracy, PACT leads OA by 7.76 pp on AndroidWorld, indicating that PACT’s advantage in multi-step task completion extends beyond static point localization.

ModelSizeAndroidWorld SR (%)OSWorld SR (%)ScreenSpot-v2 Mobile (%)
UI-TARS-7B-SFT [13]7B33.017.793.61 [13]
UI-TARS-1.5-7B [1]7B30.0 [25]27.591.23 [26]
UI-Venus-Navi-7B [7]7B49.1––
GUI-Owl-7B [23]7B66.429.5 [3]–
Qwen3-VL-8B-Instruct [25]8B47.6–92.38 [25]
Step-GUI-8B [22]8B67.740.2–
Qwen3.5-9B [14]9B57.841.893.01 [24]
PACT (ours)9B68.9744.1796.01
ConfigurationAndroidWorldOSWorldAC High CruleC_{\mathrm{rule}}AC Low CruleC_{\mathrm{rule}}ScreenSpot-v2 Mobile
O60.3440.9283.7186.3991.82
OA61.2141.1980.0478.4796.01
OAR62.9342.5586.4388.1495.41
PACT68.9744.1789.7487.5696.01
Δ\Delta(PACT−O), pp+8.63+3.25+6.03+1.17+4.19

Table 3. Main benchmark results. (A) Published GUI-agent context. (B) Matched RL comparisons. (A) Published benchmark results: representative 7B–9B GUI agents. SR: task success rate. “–”: not listed. Published-result settings and aggregation details are provided in the supplement. (B) Main benchmark results under matched settings within each platform. Task success ↑. Plan–action consistency ↑. Grounding ↑. AC: AndroidControl; CruleC_{\mathrm{rule}}: rule consistency over judgeable states. Scores: %; Δ\Delta: pp.

Extending PACT to desktop GUI agents

The task-level gains extend to desktop interaction. On the full 369-task OSWorld benchmark (Table 3(A,B)), PACT solves 163 tasks (44.17%), surpassing O (151, 40.92%) by 12 tasks, OAR (157, 42.55%) by 6 tasks, and OA (152, 41.19%) by 11 tasks. Action repair adds one success over O, while OAR and PACT yield larger gains, matching AndroidWorld’s PACT > OAR > OA > O ordering. On LibreOffice CSV export, PACT selects Text CSV and finishes in 9 steps. OAR misclicks PNG and reaches the 100-step limit (Supplement Figure S3).

Comparison of OAR and PACT on an OSWorld CSV export task

Figure S3. OSWorld CSV export. PACT selects the requested format and completes the export with native success 1.0. OAR continues format selection through step 100. Green and red circles identify the actual PACT and OAR click locations, respectively.

Repairing failures while retaining baseline-correct actions

Effective repair must correct errors while retaining mastered behavior. Let K0K_0 denote states where O emits a reference-matching action. Table 4 reports retention, recovered errors, regressions, and net error change ΔE\Delta E relative to O; Figure 3 shows net correct-action gains over OAR.

SettingModelRet. (%)W→R ↑R→W ↓Net ΔE\Delta E ↓
AC HighOA87.89669638−31
AC HighOAR86.34591720+129
AC HighPACT89.51631553−78
AC LowOA90.03436656+220
AC LowOAR89.07380719+339
AC LowPACT92.37476502+26

Table 4. Reference-action retention on aligned AndroidControl states. ∣K0∣|K_0|: High 5,270; Low 6,580. Non-initial states use each model’s own generated history. W/R: wrong/right. Net error change vs. O: ΔE=(R→W)−(W→R)\Delta E = (R \to W) - (W \to R).

Figure 3. Retention and repair on aligned states (higher is better). Top: O-correct action retention. Bottom: net correct-action gains over OAR, computed as extra repairs plus avoided regressions. PACT leads retention across settings and net gains on AndroidControl.

Plan protection improves both retention and repair relative to OAR. PACT achieves the highest baseline-correct retention on AndroidControl at 89.51% on High and 92.37% on Low, versus 86.34% and 89.07% for OAR, preserving

167 and 217 additional baseline-correct states (23.2% and 30.2% fewer regressions) while recovering 40 and 96 more baseline errors, totaling +207 and +313 correct actions over OAR on High and Low, compared with +160 and +119 for OA (Figure 3).

Limiting regressions yields a superior trade-off over action repair. On AC High, OA recovers 38 more baseline errors than PACT (669 vs. 631), but PACT avoids 85 more regressions (553 vs. 638), yielding 47 more reference-action matches overall. On Low, PACT leads on both axes, preserving 154 more baseline-correct states and recovering 40 more errors for a net advantage of 194 matches. Thus, curbing regressions on mastered states translates directly into higher overall reference action accuracy than either repair baseline.

A similar retention advantage holds on ScreenSpot-v2 Mobile: PACT preserves 459/460 (99.78%) of O-correct targets (vs. 456/460 for OAR and 458/460 for OA), while all three variants acquire 22–23 new hits.

Fixed-plan execution versus free generation

We next examine execution on 23 identified AndroidControl Low initial states with empty history, where O stated a valid immediate plan but emitted an incompatible application-launch action (Table 5). Under fixed-plan continuation, all four models—including O—generate a fresh action continuation from the same screenshot, instruction, and exact O reasoning prefix, whereas free generation requires each model to produce both reasoning and action.

Metric (%; out of 23 states)OOAOARPACT
Fixed plan
Reference action accuracy56.5295.65100.0100.0
Free generation
Reference action accuracy0.0*95.6595.6595.65
Plan–action consistency (CC)0.0*100.0100.0100.0
Semantic joint repair0.0*100.095.6595.65

Table 5. Execution diagnostics on the 23-state execution-gap set. Asterisks (*) mark O’s selection criteria. Joint repair is defined in Table 1(a). Full semantic results are in the supplementary material.

Plan-aligned action repair resolves fixed-plan execution failures. Conditioned on the fixed valid plan, both PACT and OAR raise reference action accuracy from 56.52% (13/23) for O to 100.0% (23/23), while OA reaches 95.65% (22/23); Supplement Figure S4 shows cross-app examples. Holding the input and plan fixed isolates action generation, showing that suffix supervision translates a valid plan into the intended action.

Figure S4. Fixed-plan continuation on Pinterest, Google News, and Pepperfry. The plan excerpt is shared by O and PACT. Displayed actions are the actual fixed-plan continuations. Green circles mark PACT’s predicted clicks.

The repaired behavior transfers to free generation. When generating both plans and actions autonomously, OA, OAR, and PACT all achieve 95.65% (22/23) reference action accuracy; under semantic evaluation, PACT and

OAR achieve 95.65% (22/23) joint repair while OA reaches 100.0% (23/23). The additional semantic success for OA comes from a valid off-center product click outside the reference point-distance tolerance, as detailed in the supplementary material. Although all three reach 100.0% (23/23) plan–action consistency (CC), their joint-repair scores separate surface agreement with a generated plan from faithful execution of the requested step.

Preserving plans during free generation

To assess reference-aligned plan protection, we compare PACT with OAR in Table 6 on 30 natural execution-gap states covering action-type, scroll-direction, and input-order errors, with protocol details in the supplementary material.

SubsetModelT↑T \uparrowA↑A \uparrowC↑C \uparrowJoint repair↑\uparrow
All (N=30N = 30)OAR19/3013/3025/3013/30
PACT23/3019/3028/3019/30
Both models consistent C=1C = 1, N=24N = 24OAR13/2412/2424/2412/24
PACT18/2416/2424/2416/24

Table 6. Free-generation results on all 30 states and the 24 with COAR=CPACT=1C_{\mathrm{OAR}} = C_{\mathrm{PACT}} = 1, using Table 1(a) metrics.

PACT preserves more valid plans and completes more repairs. PACT leads OAR on all four metrics, raising joint repair from 43.33% to 63.33% (+20.00 pp). Among the 24 states where both models are plan–action consistent, PACT preserves five more correct plans (18/24 vs. 13/24) and completes four more joint repairs (16/24 vs. 12/24). For example, in Supplement Figure S1, PACT closes the blocking Myntra popup before typing, whereas OAR types into the covered field; Figure S5–Figure S8 illustrate additional single-step and boundary cases.

Figure S1. Free generation on the same Myntra state and history. PACT selects the required cancel action and achieves T=A=C=1T = A = C = 1. OAR plans and emits direct typing, yielding T=A=0,C=1T = A = 0, C = 1.

Figure S5. Executing a stated plan. O formulates a valid search plan but emits an incompatible LaunchApp action. PACT and OAR execute the intended on-screen search click.

Figure S8. Consistency without correctness. ac-high:45:3. Task: turn on location (switch is already ON). Observed response (T=0T=0, A=0A=0, C=1C=1). MODEL OUTPUT. PLAN SUMMARY (PARAPHRASED): The switch is off; toggle it again. ACTION: Click(box=(844, 314)). TT: wrong — Plan correctness. The enabled switch is read as off. AA: invalid — Action validity. The click toggles an enabled switch. CC: consistent — Plan-action consistency. The click implements the toggle plan. Coordinates use a 0–999 grid. Pred. Figure S8. Consistency without correctness (T=0T = 0, A=0A = 0, C=1C = 1). When instructed to enable location access on a screen where the switch is already ON, the model misreads the toggle state as disabled and clicks it, consistently executing an incorrect plan.

Discussion and Limitations

Why plan protection complements action repair. Across multi-step benchmarks, the complete PACT framework achieves substantially higher task success on AndroidWorld and the highest task success on OSWorld within the matched comparisons of Table 3(B), while preserving more baseline-correct actions across aligned states in Table 4, even though OAR and OA match or approach PACT on isolated single-step grounding in Table 3(B) and fixed-plan continuation in Table 5. These results clarify the complementary roles of our two mechanisms: plan-aligned action repair teaches the policy to translate a valid plan into the intended action, whereas reference-aligned plan protection shields that plan from negative policy-gradient updates on failed rollouts. Because each action in closed-loop interaction alters the context for subsequent turns, retaining mastered behaviors is as critical as recovering from errors, enabling PACT to drive stronger end-to-end task completion across mobile and desktop environments.

Limitations and future directions. First, our training-time selector requires explicit plans whose parameters can be verified against a single annotated reference. Consequently, valid but implicit plans or semantically equivalent alternative actions may fall outside the eligible set; incorporating semantic plan verifiers or multi-reference supervision could broaden the coverage of repairable failures. Second, whereas our OSWorld experiments establish task-level gains on desktop workflows, our fine-grained plan and fixed-plan continuation diagnostics focus on mobile states. Extending these mechanistic analyses to richer desktop action spaces—such as keyboard shortcuts, composite operations, and long-horizon workflows—will clarify how both mechanisms interact in complex environments.

Conclusion

We presented PACT, a training framework that couples reference-aligned plan protection with plan-aligned action repair to resolve GUI execution gaps without altering the original scalar action reward. It shields reference-supported plan spans from negative policy-gradient updates while optimizing corrected action suffixes under the same decoded reasoning trace. Compared with outcome-only RL, PACT substantially improves AndroidWorld task success and achieves higher task success on OSWorld, alongside mobile plan–action consistency and visual grounding. Aligned-state retention analyses further show that PACT preserves a higher proportion of baseline-correct actions than unprotected repair variants. Moreover, fixed-plan continuation and free-generation diagnostics confirm the complementary roles of both mechanisms while distinguishing true action correctness from surface-level plan–action agreement. Together, these findings establish a correctness-guided credit-assignment principle for GUI agents: preserve valid plans and selectively repair their execution, rather than optimizing consistency alone.

LLM Usage Statement. Large language models (GPT and Gemini) were used for manuscript copy-editing, figure-layout formatting, and assisting with initial candidate screening prior to human verification in our diagnostic semantic evaluations (§ 5). All experimental designs, implementations, analyses, and final judgments were conducted and verified by the authors.

Implementation Details

Sections A.1–A.5 detail the reward, selection and construction rules, gradient derivation, and optimization schedule. Section B provides benchmark breakdowns and diagnostic protocols, and Section C presents qualitative cases.

Original action reward and coordinate tolerances

Within each platform, all four training configurations share the scalar outcome scorer R0(y,a⋆)R_0(y, a^\star) defined in Eq. (3) of the main paper. In the mobile scorer’s standard parseable branch, the format term rfmt(y)r_{\mathrm{fmt}}(y) assigns +0.5 for valid <think>...</think><action>...</action> structure and −0.5 otherwise. Matching the reference action type τ(a)=τ(a⋆)\tau(a) = \tau(a^\star) contributes 1.0 point, followed by the action-specific parameter score P0(a,a⋆)P_0(a, a^\star). Missing actions, unparseable outputs, and grounding-only samples retain the base scorer’s original branches.

For mobile point-based actions (e.g., Click, LongPress), predicted coordinates are emitted on a normalized 0–999 grid and mapped back to native image pixels. Let dd denote the Euclidean pixel distance between the predicted and reference coordinates, and let m=min⁡(W,H)m = \min(W,H) be the shorter screen dimension. The parameter score is

Pclick(d)={1.0,d<0.07m,0.5,0.07m≤d<0.14m,0.0,d≥0.14m.(10)P_{\mathrm{click}}(d) = \begin{cases} 1.0, & d < 0.07m,\\ 0.5, & 0.07m \le d < 0.14m,\\ 0.0, & d \ge0.14m. \end{cases} \tag*{(10)}

For Scroll/Swipe actions with directional metadata, the scorer checks both gesture endpoints and assigns 1.5 parameter points when both satisfy the strict coordinate tolerance. The branch with endpoint coordinates alone uses its standard Boolean endpoint comparison. Text-based actions (Type, LaunchApp, Answer) receive full parameter credit under exact normalized string match or a token-set F1 score exceeding 0.5.

Reference-verified execution-gap selection and plan span localization

To compute the execution-gap eligibility flag Ei=Ji1[T^i=1,A^i=0,C^i=0]E_i = J_i \mathbb{1}[\hat{T}_i = 1, \hat{A}_i = 0, \hat{C}_i = 0] in Eq. (2) of the main paper, the programmatic selector first evaluates judgeability Ji∈{0,1}J_i \in\{0,1\}. Setting Ji=1J_i = 1 requires a unique emitted action block, a parseable reference action ai⋆a_i^\star with comparison metadata, and a non-terminal plan with checkable parameters. Grounding-only samples and terminal references (Finished, Answer, CallUser) are assigned Ji=0J_i = 0 and participate in the base GRPO objective.

The following rules define the three binary proxies for judgeable mobile responses (Ji=1J_i = 1).

  • Action correctness proxy (A^i\hat{A}_i): A^i=1\hat{A}_i = 1 if the base content score (action-type bonus plus parameter score) reaches the full-credit threshold (2.0 points for standard actions, or 2.5 points for normalized Scroll/Swipe with directional endpoints). Responses receiving only partial parameter credit are treated as A^i=0\hat{A}_i = 0.

  • Plan correctness proxy (T^i\hat{T}_i): The reasoning trace ziz_i is segmented into clauses using line breaks and English/Chinese punctuation, and the final four sentences are scanned in reverse order. A valid plan must specify a single immediate action without negation or multi-step ambiguity. Setting T^i=1\hat{T}_i = 1 requires the extracted plan parameters to match ai⋆a_i^\star. Pointing plans require a unique explicit 0–999 coordinate within the strict 0.07min⁡(W,H)0.07\min(W,H) pixel tolerance of ai⋆a_i^\star. Directional plans require a unique matching direction. Text and app-launch plans require a unique quoted argument matching the normalized reference string. Parameter-free system actions must match by action type.

  • Plan–action compatibility proxy (C^i\hat{C}_i): C^i=1\hat{C}_i = 1 if the emitted action aia_i matches the extracted plan in action type, lies within a grid distance of 1 on the normalized 0–999 coordinate space for pointing actions, and matches any specified direction, text, or application argument.

For pointing actions, T^i\hat{T}_i checks proximity to the reference coordinate, whereas C^i\hat{C}_i checks agreement with the explicitly stated coordinate. PACT selects repair candidates using the full eligibility condition Ei=1E_i = 1, including A^i=0\hat{A}_i = 0.

For reference-aligned plan protection (Ei=1E_i = 1), the character span of the verified plan is mapped to response token indices to form the binary mask Mi,tD∈{0,1}M^{D}_{i,t} \in\{0,1\}. The implementation uses tokenizer character offsets whenever the re-encoded reasoning prefix matches the active token sequence, and otherwise performs an exact token-subsequence search prior to the action tag. If a unique token span cannot be localized, we set Mi,tD=0M^{D}_{i,t} = 0 for all tt, leaving the base advantage Ai0A^0_i unchanged while still permitting suffix repair if pair construction succeeds.

Constructing positive action suffixes

For each eligible mobile response yi=(zi,ai−)y_i = (z_i, a_i^-), the positive action suffix ai+a_i^+ is synthesized from the training reference ai+a_i^+ in the model’s native action syntax and 0–999 coordinate grid. Because T^i=1\hat{T}_i = 1 already verifies that the extracted plan targets the reference UI element within the strict reference tolerance, synthesizing ai+a_i^+ from ai∗a_i^* under the same decoded reasoning trace ziz_i provides a reference-compatible execution of the stated plan. Reference pixel coordinates are converted to the normalized grid, rounded, and clamped to valid bounds. Scroll references with endpoint coordinates are formatted as Swipe actions. Reference actions with missing endpoints or text arguments that cannot be serialized in the native action syntax, as well as overlength replacements, are omitted from PP. Their original rollouts remain in B\mathcal{B}.

On both platforms, the constructed positive sequence yi+=(zi,ai+)y_i^+ = (z_i, a_i^+) retains the exact input context xix_i (instruction, screenshot, and interaction history) and the decoded reasoning text ziz_i, replacing only the <action>...</action> block. The binary action masks Mi,tA,−M_{i,t}^{A,-} and Mi,tA,+M_{i,t}^{A,+} cover only the action tags and action tokens, excluding all reasoning tokens and any trailing text. At most 64 constructible pairs are retained per training iteration in batch order. In the OA baseline (original GRPO with Action Repair), pairs are constructed using the identical reference formatter from parseable, non-terminal rollouts with A^i=0\hat{A}_i = 0 and ai−≠ai+a_i^- \ne a_i^+, without requiring T^i=1\hat{T}_i = 1 or C^i=0\hat{C}_i = 0.

First-order gradient derivation of the suffix-repair surrogate

Let sθ,i±s_{\theta,i}^{\pm} denote the summed action-span log probability defined in Eq. (6) of the main paper, and let the log-ratio margin be

ℓi(θ)=(sθ,i+−sref,i+)−(sθ,i−−sref,i−).(11)\ell_i(\theta) = \left(s_{\theta,i}^{+} - s_{\mathrm{ref},i}^{+}\right) - \left(s_{\theta,i}^{-} - s_{\mathrm{ref},i}^{-}\right). \tag*{(11)}

A conditional DPO-style logistic preference objective [11, 15] over the retained pair set PP is given by

Llog⁡(θ)=−λB0∑i∈Plog⁡σ(βℓi(θ)).(12)\mathcal{L}_{\log}(\theta) = -\frac{\lambda}{B_0}\sum_{i\in P}\log\sigma\left(\beta\ell_i(\theta)\right). \tag*{(12)}

Here, λ\lambda is the repair weight and β\beta is the preference scale. Taking the gradient with respect to θ\theta and evaluating at the pre-update policy θ=θold\theta= \theta_{\mathrm{old}} (where ℓi(θold)=ℓiold\ell_i(\theta_{\mathrm{old}}) = \ell_i^{\mathrm{old}}) yields

∇θLlog⁡(θ)∣θ=θold=∑i∈PλβB0σ(−βℓiold)⏟wi(∇θsθ,i−−∇θsθ,i+)∣θ=θold.(13)\nabla_\theta\mathcal{L}_{\log}(\theta)\big|_{\theta=\theta_{\mathrm{old}}} = \sum_{i\in P} \underbrace{\frac{\lambda\beta}{B_0}\sigma\left(-\beta\ell_i^{\mathrm{old}}\right)}_{w_i} \left(\nabla_\theta s_{\theta,i}^{-} - \nabla_\theta s_{\theta,i}^{+}\right)\big|_{\theta=\theta_{\mathrm{old}}}. \tag*{(13)}

Detaching the pre-update weight wiw_i (where −ℓiold=sold,i−−sref,i−−sold,i++sref,i+-\ell_i^{\mathrm{old}} = s_{\mathrm{old},i}^{-} - s_{\mathrm{ref},i}^{-} - s_{\mathrm{old},i}^{+} + s_{\mathrm{ref},i}^{+} matches Eq. (7) of the main paper) via stopgrad⁡(wi)\operatorname{stopgrad}(w_i) produces the linear suffix-repair surrogate Lrepairsur(θ)=∑i∈Pstopgrad⁡(wi)(sθ,i−−sθ,i+)\mathcal{L}_{\mathrm{repair}}^{\mathrm{sur}}(\theta) = \sum_{i\in P}\operatorname{stopgrad}(w_i)(s_{\theta,i}^{-} - s_{\theta,i}^{+}) in Eq. (8) of the main paper, whose gradient matches Eq. (13) at θ=θold\theta= \theta_{\mathrm{old}}.

Optimization schedule, hyperparameters, and desktop adaptation

Training data and optimization. Table S1 summarizes the RL configurations for both platforms. All four configurations (O, OA, OAR, and PACT; main Table 2) on both platforms start RL training from the same SFT checkpoint. We obtain this checkpoint by supervised fine-tuning the Qwen3.5-9B vision-language model on multimodal GUI trajectories to generate structured <think>...</think> <action>...</action> responses, with spatial action coordinates normalized to 0–999. A frozen copy of this checkpoint serves as the KL reference policy πref\pi_{\mathrm{ref}}. Within each platform, the four variants share the base-training configuration in Table S1. Desktop RL training uses 18,000 examples sampled without replacement (seed 42) and runs for one epoch with complete 32-prompt batches (562 iterations), preserving the original system instructions, interaction history, screenshots, and reference actions. All training data are strictly disjoint from the evaluation benchmarks.

SettingMobileDesktop
Initialization and fixed referenceQwen3.5-9B SFTQwen3.5-9B SFT
RL training examples10,00018,000
Training iterations / epochs1,250 / 2562 / 1
Prompts per iteration / responses per prompt (KK)16 / 832 / 8
Original rollout batch size (B0B_0)128256
OptimizerAdamWDistributed Adam
Learning rate / PPO epochs10−610^{-6} / 110−610^{-6} / 1
Learning-rate scheduleConstantConstant
Weight decay / gradient clipping0.01 / 1.00.01 / 1.0
Policy-loss aggregation / entropy coefficientToken mean / 0Token mean / 0
Probability-ratio clipping / dual clipping0.2 / 3.00.2 / 3.0
Sampling temperature / top-pp / top-kk1.0 / 1.0 / −1-11.0 / 1.0 / −1-1
Maximum prompt / response tokens36,864 / 2,04840,960 / 2,048
Accelerators / training framework32 / FSDP264 / Megatron
Rollout tensor parallelism44
Numerical precisionbfloat16bfloat16
KL coefficient (η\eta) / estimator0.01 / k3+0.01 / low_var_kl
Repair weight (λ\lambda) / preference scale (β\beta)0.1 / 0.10.1 / 0.1
Maximum repair pairs per iteration6464

Table S1. Training hyperparameters for the mobile and desktop studies. Repair parameters apply to OA, OAR, and PACT.

The desktop action interface supports pointing, keyboard, drag/scroll, and control actions. Training references are dominated by Click (43.0%), Hotkey (26.1%), and Type (21.2%). Each training example pairs the multimodal input context (system instructions, interaction history, and screenshots) with its ground-truth reference action.

Both platforms use token-mean policy loss and apply KL regularization in the actor objective with η=0.01\eta= 0.01. The mobile k3+k3+ estimator combines a clipped k3k_3 value with a straight-through squared-log-ratio gradient. Desktop training uses ed−d−1e^d - d - 1, with d=log⁡πref−log⁡πθd = \log\pi_{\mathrm{ref}} - \log\pi_\theta clipped to [−20,20][-20, 20] and the estimator clipped to [−10,10][-10, 10]. Original rollout groups determine GRPO advantages. Constructed positive responses carry the suffix-repair loss with zero PG and KL masks, and signed pair weights ±wi\pm w_i preserve the sum in Eq. (8) of the main paper under data-parallel averaging. On 32 accelerators, the mobile 1,250-iteration training loop takes 120.76 hours for O and 123.19 hours for PACT, excluding initial model loading, corresponding to approximately 2.0% overhead.

Algorithm S1 summarizes one mobile PACT training iteration. Selection, span localization, and pair construction use Sections A.2 and A.3.

Desktop plan verification and suffix construction. The desktop selector applies the same execution-gap condition to explicit plans. Table S2 sets out the supported action parameters. Reference support T^i\hat{T}_i compares the plan with the reference action, while compatibility C^i\hat{C}_i compares the plan with the emitted action. Both comparisons require matching action types. Desktop action correctness A^i=1\hat{A}_i = 1 requires the full native content score of 2.0, comprising one point for type and one for parameters.

ActionRequired parametersReference support / plan compatibilityPositive suffix construction
Click, DoubleClick, RightClickOne box coordinateT^\hat{T}: distance < 49.95 (0.05 × 999). C^\hat{C}: distance ≤ 1.Copy reference box.
Drag, Swipestart and end coordinatesT^\hat{T}: each endpoint distance < 139.86 (0.14 × 999). C^\hat{C}: each endpoint distance ≤ 1.Copy both reference endpoints.
TypeOne quoted content stringT^,C^\hat{T}, \hat{C}: exact string equality, preserving whitespace and escapes.Copy reference content.
HotkeyOne nonempty keys list or tupleT^,C^\hat{T}, \hat{C}: equality of normalized, ordered key tuples.Copy reference keys.
WaitNo argumentsT^,C^\hat{T}, \hat{C}: matching Wait action.Wait().
CallUser, FinishedOptional content stringTerminal actions receive native scoring with exact content matching.Excluded from auxiliary selection.

Table S2. Desktop training-action verification and reference-derived suffix construction. Spatial distances are Euclidean distances on the normalized 0–999 grid.

For desktop pointing actions, the native parameter score is 1.0 at grid distances below 49.95=0.05×99949.95 = 0.05 \times999, 0.5 from 49.95 up to but excluding 99.9 = 0.10 × 999, and zero otherwise. The reference-support test uses the full-credit threshold. Drag and Swipe plans specify exactly two coordinate pairs in start–end order. For Type, the native parameter score awards 1.0 to exact text matches and 0.5 to non-exact matches with the same trailing-whitespace category and token-set F1 greater than 0.5. Hotkey comparison strips surrounding whitespace and lowercases each key while preserving tuple order and the literal space key.

Desktop reference coordinates already use the model’s normalized grid. The positive suffix serializes the reference action name and parameters with Python literal representations and is checked by the same action parser. The resulting pair retains the original input and decoded reasoning context, following Section A.3.

Evaluation Protocols and Additional Quantitative Results

AndroidControl rule-based consistency

On AndroidControl, the rule-based consistency evaluator first searches the reasoning trace for the final function-style action expression; if none is present, it extracts the last English or Chinese action keyword after resolving keyword overlaps, and parses any accompanying coordinate, direction, or text arguments. Disagreements between the extracted intent and the emitted action are categorized into action-type or direction mismatches (NtypeN_{\mathrm{type}}), coordinate mismatches (NcoordN_{\mathrm{coord}}), and text-argument mismatches (NtextN_{\mathrm{text}}). Rule-based consistency is computed over all judgeable outAlgorithm S1 One mobile PACT training iteration

Require: Pre-update policy πold\pi_{\mathrm{old}}, reference πref\pi_{\mathrm{ref}}, 16 prompts with reference actions, K=8K = 8, pair cap 64

  1. Sample original rollout batch B\mathcal{B}, B0=128\mathcal{B}_0 = 128

  2. Compute R0R_0 and grouped Ai0A_i^0 using main Eqs. (3)–(4)

  3. Broadcast A~i,t←Ai0\widetilde{A}_{i,t} \leftarrow A_i^0; set P←∅\mathcal{P} \leftarrow\varnothing

  4. for each response ii in batch order do

  5. Compute JiJ_i, T^i\widehat{T}_i, A^i\widehat{A}_i, C^i\widehat{C}_i and EiE_i using main Eq. (2)

  6. if Ei=1E_i = 1 then

  7. Localize Mi,tDM_{i,t}^{D} (use a zero mask if no unique span is found)

  8. Clamp plan-token advantages using main Eq. (5)

  9. Independently construct yi+=(zi,ai+)y_i^{+} = (z_i, a_i^{+}) from ai−a_i^{-}

  10. if the pair is constructible and ∣P∣<64|\mathcal{P}| < 64 then

  11. Append the pair and its action masks to P\mathcal{P}

  12. end if

  13. end if

  14. end for

  15. if P≠∅\mathcal{P} \ne\varnothing then

  16. Pack positive responses into a 128-row auxiliary batch with zero-loss padding

  17. Score both suffixes under πold\pi_{\mathrm{old}} and πref\pi_{\mathrm{ref}}; compute and freeze wiw_i using main Eqs. (6)–(7)

  18. Assign +wi+w_i to sampled negative suffixes, −wi-w_i to constructed positive suffixes, and zero to padding

  19. Set auxiliary PG and KL masks to zero

  20. end if

  21. Update policy parameters θ\theta by minimizing the actor objective in main Eq. (9)

puts as

Crule=NconsistentNconsistent+Ntype+Ncoord+Ntext.(14)C_{\mathrm{rule}} = \frac{N_{\mathrm{consistent}}}{N_{\mathrm{consistent}} + N_{\mathrm{type}} + N_{\mathrm{coord}} + N_{\mathrm{text}}}. \tag*{(14)}

Responses lacking a reasoning block, an extractable action intent, or a valid reference label are excluded from the denominator of CruleC_{\mathrm{rule}}. Table S3 reports consistent and judgeable counts over 8,444 states per setting. These states comprise all annotated action steps in the 1,543 evaluation episodes, with identical episode–step pairs used for High and Low. On AC High, PACT achieves the highest CruleC_{\mathrm{rule}} (89.74%) and absolute consistent-action count (7,046 vs. O: 6,879; OAR: 6,997; OA: 6,440), remaining first with all 8,444 states as a common denominator. Action-type mismatches account for most rule inconsistencies. PACT reduces these mismatches from 1,279 to 753 on High, and has the fewest coordinate mismatches on both High (16) and Low (6).

Because CruleC_{\mathrm{rule}} relies on surface keyword and function extraction, it provides a scalable benchmark-wide proxy rather than a full semantic parse. For example, mentioning an in-app “Home” tab can trigger the system PressHome keyword rule, and discussing a subsequent step at the end of a multi-clause sentence can override an earlier immediate click intent. These boundary cases motivate our complementary screenshot-based semantic diagnostics in the main paper.

Interactive task evaluation on AndroidWorld and OSWorld

AndroidWorld. For each mobile checkpoint, we report the task success rate across AndroidWorld’s 116 interactive tasks, yielding 70/116 (60.34%) for O, 71/116 (61.21%) for OA, 73/116 (62.93%) for OAR, and 80/116 (68.97%) for PACT.

OSWorld. All four desktop-adapted variants (O, OA, OAR, and PACT) are evaluated on the full 369-task OSWorld benchmark. Evaluation uses a standardized Linux x86/Docker environment with a read-only Ubuntu base image and the official native evaluation scripts. All four variants use temperature 0 and a 100-step interaction limit. Input construction and generation parameters are given in Section B.5. At each step, the complete emitted action block is parsed and executed. A task is counted as successful strictly when its native evaluator returns a full score of 1.0:

SROSWorld=1369∑j=13691[native_scorej=1.0],(15)\mathrm{SR}_{\mathrm{OSWorld}} = \frac{1}{369}\sum_{j=1}^{369}\mathbb{1}[\mathrm{native\_score}_j = 1.0], \tag*{(15)}

without counting partial completion scores as successes. The totals are 151/369 (40.92%) for O, 152/369 (41.19%) for OA, 157/369 (42.55%) for OAR, and 163/369 (44.17%) for PACT. PACT completes 12 more tasks than O, 11 more than OA, and 6 more than OAR. The Step-GUI-8B and Qwen3.5-9B scores in main Table 3(A) use OSWorld-Verified.

ScreenSpot-v2 Mobile subgroup breakdown and target retention

Table S4 and Table S5 provide the target-category breakdown and baseline-correct retention statistics on ScreenSpot-v2 Mobile. Because ScreenSpot-v2 Mobile consists of single-step grounding queries without multi-turn history, Table S5 offers a history-independent test of capability preservation: PACT retains 99.78% (459/460) of O-correct targets while incurring only a single right-to-wrong regression. For the four cited ScreenSpot-v2 Mobile scores in main Table 3(A), text/icon accuracies are weighted by 290/211 targets.

Table S4. ScreenSpot-v2 Mobile point-in-target accuracy across all 501 examples, 290 text targets, and 211 icon targets.

ConfigurationAll targets (501)Text targets (290)Icon targets (211)
O460 (91.82%)271 (93.45%)189 (89.57%)
OA481 (96.01%)286 (98.62%)195 (92.42%)
OAR478 (95.41%)286 (98.62%)192 (91.00%)
PACT481 (96.01%)285 (98.28%)196 (92.89%)

Table S4. ScreenSpot-v2 Mobile point-in-target accuracy across all 501 examples, 290 text targets, and 211 icon targets.

ConfigurationRetained / 460Retention (%)Wrong to right (↑)Right to wrong (↓)
OA458 / 46099.57232
OAR456 / 46099.13224
PACT459 / 46099.78221

Table S5. Target-level retention and repair on ScreenSpot-v2 Mobile (460 O-correct targets).

Mechanism diagnostic protocols and complete results

For both diagnostic sets, screenshot-based semantic assessments use language-model assistance followed by human verification, with separate judgments of plan correctness (TT), action validity (AA), and semantic plan–action consistency (CC). Here, TT evaluates whether the current plan is correct for the task, AA evaluates action validity in the current task and state, and CC evaluates whether the action implements the model’s stated plan. Joint repair requires T=A=C=1T = A = C = 1 for the same plan–action pair, as in Table 1(a) of the main paper. When a valid plan (T=1T = 1) is paired with an invalid action (A=0A = 0), consistency is judged as C=0C = 0 if the action targets a different UI element or operation than stated (e.g., landing on an unrelated control in Figure S7), and C=1C = 1 if it targets the stated element or text but fails task validity due to boundary localization or interface state constraints.

Figure S7. Coordinate grounding error. Zara: view the results for the selected child size. ac-high:55:8. Observed response (T=1T=1, A=0A=0, C=0C=0). MODEL OUTPUT. PLAN SUMMARY (PARAPHRASED): Click VIEW RESULTS (6). ACTION: Click(box=(750, 585)). TT: correct — Plan correctness. Targets VIEW RESULTS. AA: invalid — Action validity. The click lands on the size option. CC: inconsistent — Plan-action consistency. The click does not implement the plan. Coordinates use a 0–999 grid. Pred. Figure S7. Coordinate-error illustration (T=1T = 1, A=0A = 0, C=0C = 0). The model identifies VIEW RESULTS as the intended target, but its click lands on a neighboring size option.

23-state fixed-plan continuation and free generation. The 23 diagnostic states were identified among AndroidControl Low initial steps (empty interaction history) where O articulated a valid immediate plan but emitted an erroneous application-launch action. Under the fixed-plan condition, each model receives the identical task instruction, screenshot, empty history, and O’s exact valid reasoning prefix terminating at the <action> opening tag, and deterministically generates the action continuation in a single turn. For a matched comparison with OA, OAR, and PACT, O also generates a fresh action continuation from the same fixed prefix, yielding 13/23; its 0/23 free-generation score comes from the original outputs used to select these states. Under free generation, the assistant prefix is removed and each model generates both reasoning and action from scratch. Reference-action matching and screenshot-based semantic evaluation are reported side by side in Table 5 of the main paper. On the single state where OA’s semantic action-validity and reference-matching judgments differ (a Nike product-card screen), OA clicks a valid off-center region of the product image outside the point-distance tolerance of the single reference coordinate, whereas PACT and OAR score 22/23 under both criteria. Table S6 reports the complete fixed-plan continuation and free-generation results, including the semantic breakdown. Following our coordinate-based action interface, reference-action matching evaluates predictions directly against each state’s annotated reference action a∗a^* (without accessibility-tree bounding boxes or cross-type equivalences), requiring matching action types and reference-compatible parameters

Condition / metricOOAOARPACT
Fixed plan: reference action accuracy56.52 (13/23)95.65 (22/23)100.0 (23/23)100.0 (23/23)
Free generation: reference action accuracy0.0 (0/23)*95.65 (22/23)95.65 (22/23)95.65 (22/23)
Free generation: plan correctness (TT)100.0 (23/23)*100.0 (23/23)95.65 (22/23)95.65 (22/23)
Free generation: action validity (AA)0.0 (0/23)*100.0 (23/23)95.65 (22/23)95.65 (22/23)
Free generation: plan–action consistency (CC)0.0 (0/23)*100.0 (23/23)100.0 (23/23)100.0 (23/23)
Free generation: semantic joint repair0.0 (0/23)*100.0 (23/23)95.65 (22/23)95.65 (22/23)

Table S6. Complete fixed-plan continuation and free-generation results on the 23-state execution-gap diagnostic set (%, with exact counts out of 23 in parentheses). Joint repair requires T=A=C=1T = A = C = 1. Asterisks (*) mark O’s selection criteria. Bold indicates row-wise maxima, including ties.

(0.14W2+H2(0.14\sqrt{W^2 + H^2} point-distance tolerance for clicks, matching swipe direction, and normalized text match).

30-state natural execution-gap set. This set comprises O-selected execution-gap candidates from AndroidControl High and Low, targeting valid immediate plans paired with invalid, inconsistent actions. Eleven states use empty history and 19 use O’s interaction history. OAR and PACT freely generate plans and actions from identical screenshots, instructions, and available O histories, without O’s current response or the reference action. This set and the 23-state fixed-plan diagnostic follow distinct selection protocols and are analyzed separately.

All 30 states remain in each denominator; only verified successes enter the numerator. The ambiguous-reference boundary case scores as a non-success for TT and joint repair, with CC evaluated independently. In Table S7, PACT raises TT from 63.33% (19/30) to 76.67% (23/30), CC from 83.33% (25/30) to 93.33% (28/30), and both AA and joint repair from 43.33% (13/30) to 63.33% (19/30), +20.00 pp. Among the 24 states with COAR=CPACT=1C_{\mathrm{OAR}} = C_{\mathrm{PACT}} = 1, PACT retains 18/24 correct plans versus 13/24 for OAR (+20.83 pp) and completes 16/24 joint repairs versus 12/24 (+16.67 pp). On the remaining six states, OAR generates six valid plans but only one consistent action and joint repair; PACT achieves consistency on four states and joint repair on three.

Candidate categoryStates
Click-versus-launch conflict8
Scroll-direction error2
Premature text entry / skipped selection17
Click-versus-wait conflict1
Swipe-versus-click/selection conflict1
Ambiguous-reference boundary case1
Total30
SubsetModelTT ↑AA ↑CC ↑Joint repair↑
All (N=30N = 30)OAR63.33 (19/30)43.33 (13/30)83.33 (25/30)43.33 (13/30)
All (N=30N = 30)PACT76.67 (23/30)63.33 (19/30)93.33 (28/30)63.33 (19/30)
Both models consistent C=1C = 1, N=24N = 24OAR54.17 (13/24)50.00 (12/24)100.0 (24/24)50.00 (12/24)
Both models consistent C=1C = 1, N=24N = 24PACT75.00 (18/24)66.67 (16/24)100.0 (24/24)66.67 (16/24)
Remaining (N=6N = 6)OAR100.0 (6/6)16.67 (1/6)16.67 (1/6)16.67 (1/6)
Remaining (N=6N = 6)PACT83.33 (5/6)50.00 (3/6)66.67 (4/6)50.00 (3/6)

Table S7. Natural execution-gap composition and free-generation results on all 30 states, the 24 with COAR=CPACT=1C_{\mathrm{OAR}} = C_{\mathrm{PACT}} = 1, and the remaining six. Bold marks the better value within each comparison, including ties.

Prompts, input construction, and generation settings

Training inputs. Both platforms retain the original system prompt, interaction history, and screenshots from each training example. Images fill the message placeholders in their recorded order. Historical assistant messages contain the associated reasoning and actions. The current reference action is stored in a separate reference field.

Mobile evaluation. The system prompt contains the task goal, action definitions, and the <think>...</think><action>...</action> response format. On AndroidControl Low, the goal is followed by Current step: and the low-level instruction. High uses the episode goal. The current user message contains Current Screenshot: and the image. The most recent ten interaction rounds contribute the model’s own reasoning and action text. AndroidControl uses the recorded state image at each step. AndroidWorld supplies the current environment screenshot and uses an interaction budget of ⌈10×task.complexity⌉\lceil10 \times\text{task.complexity} \rceil.

ScreenSpot-v2 uses an empty interaction history and the instruction In the screenshot, click the element that matches this description: followed by the quoted target description. Mobile generation uses temperature 0 and a maximum of 4,096 output tokens. For execution, normalized 0–999 coordinates (x,y)(x,y) are linearly rescaled to integer pixel coordinates on the W×HW \times H screen.

Desktop evaluation. The desktop system prompt embeds the task goal and defines Click, DoubleClick, RightClick, Drag, Swipe, Type, Hotkey, Wait, CallUser, and Finished. Each input contains the current 1920×1080 screenshot and the two most recent historical screenshots. Earlier steps retain historical reasoning and action blocks. All four desktop configurations use temperature 0, a maximum of 4,096 output tokens, and at most 100 interaction steps. Task success is determined strictly by the official native evaluator.

Fixed-plan continuation. The fixed-plan input supplies the same system message, screenshot, and verbatim valid O reasoning prefix to each model. The assistant prefix terminates immediately after the opening action tag.

system      [task and action definitions]
user        [Current Screenshot: and the state
→  image]
assistant   <think>[O's exact valid
→  reasoning]</think><action>

temperature=0, max_tokens=4096 continue_final_message=true

The completed response concatenates the supplied prefix and generated suffix. Free generation uses the same state input and generates both reasoning and action in a new assistant response. In the 30-state natural execution-gap set, 11 states use empty history and 19 supply the same recorded O-history steps to both OAR and PACT (e.g., two prior steps on Myntra in Figure S1), using identical mobile generation settings.

Qualitative Case Studies

Comparative Success Cases Across Mobile and Desktop

Preserving the next-step requirement. The Myntra example in Figure S1 uses the same screenshot and two preceding O-history steps for PACT and OAR. The task requires closing the modal before text entry. PACT preserves that ordering in its immediate plan and clicks the close button. OAR assumes that the covered search field is ready and emits Type. Both outputs have C=1C = 1, while only PACT meets the plan, action, and joint-repair criteria. This example illustrates how PACT preserves a valid plan and completes the repair during free generation.

Executing a complete mobile workflow. In Figure S2, all four configurations receive the same receipt-copy goal and task seed. PACT selects Download, confirms the copy, and emits Finished at step 15. OAR emits Type(content='Download') at steps 9, 10, 12, 14, and 16. The native final score is 1.0 for PACT and 0.0 for O, OA, and OAR.

Comparison of OAR and PACT on an AndroidWorld receipt-copy task

Figure S2. AndroidWorld receipt-copy task. PACT selects the destination, confirms copying, and terminates with native success. OAR repeatedly emits the folder name. Step numbers are one-based.

Completing a desktop export task. Figure S3 shows the same OSWorld task across the four configurations. PACT uses the Save As workflow, selects Text CSV, confirms the export options, and finishes in nine steps. Its exported file passes the official CSV-content comparison. O, OA, and OAR receive native scores of zero at the 100-step limit.

Action repair with a fixed valid plan. Figure S4 provides three examples from the 23-state set. Each O/PACT comparison uses the same task, screenshot, and verbatim O reasoning prefix ending at the <action> opening tag. The prompt-token sequences are identical between O and PACT within each case. O continues with an application-launch command in all three examples. PACT generates the on-screen click specified by the supplied plan. The corresponding PACT free-generation outputs also satisfy T=A=C=1T = A = C = 1.

Step-level plan execution and joint correction. Figures S5 and S6 further compare O, OAR, and PACT on individual AndroidControl states. In Figure S5, O states a valid plan to tap the search icon in Google News but emits an incompatible LaunchApp command, whereas both OAR and PACT execute the intended on-screen click. In Figure S6, where the instruction requests tapping Send in an email composer, O plans to edit the message body and long-presses the subject line, whereas OAR and PACT jointly correct both the immediate plan and the emitted click.

Figure S6. Joint plan and action correction. O plans to edit the email body and long-presses the subject line. PACT and OAR correct both the plan and action to click Send.

Plan-aligned action repair across apps

Within each case, O and PACT share the input and valid O plan.

Diagnostic Taxonomy Boundary Illustrations

Figures S7 and S8 illustrate two distinct TT, AA, CC configurations using observed agent responses, a correct plan with incorrect spatial execution (1, 0, 0) and an incorrect plan executed consistently (0, 0, 1). These examples distinguish plan correctness, action validity, and plan–action consistency.

References

  1. [1]ByteDance Seed. UI-TARS-1.5-7B: Official model card. Hugging Face, https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B, 2025.
  2. [2]Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories. arXiv preprint arXiv:2607.10601, 2026.
  3. [3]Yuhan Chen, Yuxuan Liu, Long Zhang, Pengzhi Gao, Jian Luan, and Wei Liu. STEP: Success-rate-aware trajectory-efficient policy optimization. In Findings of the Association for Computational Linguistics: ACL 2026, pages 30681–30692. Association for Computational Linguistics, 2026.
  4. [4]Yi Ding, Ziliang Qiu, Bolian Li, and Ruqi Zhang. Learning in Self-Correction in Vision-Language Models via Rollout Augmentation. In Proceedings of the 43rd International Conference on Machine Learning, pages 25096–25113. PMLR, 2026.arxiv.org/abs/2602.08503
  5. [5]Lingzhong Dong, Ziqi Zhou, Shuaibo Yang, Haiyue Sheng, Pengzhou Cheng, Zongru Wu, Zheng Wu, Gongshen Liu, and Zhuosheng Zhang. Say One Thing, Do Another? Diagnosing Reasoning-Execution Gaps in VLM-Powered Mobile-Use Agents. arXiv preprint arXiv:2510.02204, 2025.arxiv.org/abs/2510.02204
  6. [6]Yichen Gong, Zhuohan Cai, Sunhao Dai, Yuqi Zhou, Zhangxuan Gu, Changhua Meng, and Shuheng Shen. VenusBench-Mobile: A challenging and user-centric benchmark for mobile GUI agents with capability diagnostics. In Proceedings of the 43rd International Conference on Machine Learning, pages 35862–35887. PMLR, 2026.arxiv.org/abs/2604.06182
  7. [7]Zhangxuan Gu, Zhengwen Zeng, Zhenyu Xu, Xingran Zhou, Shuheng Shen, Yunfei Liu, Beitong Zhou, Changhua Meng, Tianyu Xia, Weizhi Chen, Yue Wen, Jingya Dou, Fei Tang, Jinzhen Lin, Yulin Liu, Zhenlin Guo, Yichen Gong, Heng Jia, Changlong Gao, Yuan Guo, Yong Deng, Zhenyu Guo, Liang Chen, and Weiqiang Wang. UI-Venus technical report: Building high-performance UI agents with RFT. arXiv preprint arXiv:2508.10833, 2025.
  8. [8]Peixuan Han, Runhui Wang, Ketan Ramaneti, Jie Hao, Gerald Friedland, and Chris Kong. Cliff: Learning Process Rewards from the First Mistake. arXiv preprint arXiv:2609.02817, 2026.arxiv.org/abs/2609.02817
  9. [9]Haowen Hu, Pengzhou Cheng, Zheng Wu, Lingzhong Dong, Gongshen Liu, and Zhuosheng Zhang. Faithful Mobile GUI Agents with Guided Advantage Estimator. In Proceedings of the 43rd International Conference on Machine Learning, pages 44838–44865. PMLR, 2026.arxiv.org/abs/2605.01208
  10. [10]Kun Huang, Weikai Xu, Yuxuan Liu, Quandong Wang, Pengzhi Gao, Wei Liu, Jian Luan, Bin Wang, and Bo An. MobileIPL: Enhancing Mobile Agents Thinking Process via Iterative Preference Learning. In International Conference on Learning Representations, 2026.arxiv.org/abs/2505.12299
  11. [11]Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs. arXiv preprint arXiv:2406.18629, 2024.arxiv.org/abs/2406.18629
  12. [12]Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the Effects of Data Scale on UI Control Agents. In Advances in Neural Information Processing Systems, pages 92130–92154, 2024.arxiv.org/abs/2406.03679
  13. [13]Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi. UI-TARS: Pioneering automated GUI interaction with native agents. arXiv preprint arXiv:2501.12326, 2025.arxiv.org/abs/2501.12326
  14. [14]Qwen Team. Qwen3.5-9B model card. Hugging Face, https://huggingface.co/Qwen/Qwen3.5-9B, 2026.
  15. [15]Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In Advances in Neural Information Processing Systems, pages 53728–53741, 2023.arxiv.org/abs/2305.18290
  16. [16]Chris Rawles, Sarah Clinckemaille, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. In International Conference on Learning Representations, 2025.arxiv.org/abs/2405.14573
  17. [17]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
  18. [18]Venus Team, Zhaohan Cai, Haoxing Chen, Jiaxuan Chen, Weizhi Chen, Changlong Gao, Zhangxuan Gu, Yuan Guo, Jiayi Wu, Ruobing Xie, Zqian Huang, Lei Jiang, Can Xu, Kangyang Luo, Bochen Lin, Ming Gao, and Xiang Li. Negative Advantage Is a Double-Edged Sword: Calibrating Advantages in GRPO for Search Agents. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 39986–40002. Association for Computational Linguistics, 2026.
  19. [20]Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Yu Liang, and Yu Qiao. OS-ATLAS: Foundation Action Model for Generalist GUI Agents. In International Conference on Learning Representations, 2025.arxiv.org/abs/2410.23218
  20. [21]Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. In Advances in Neural Information Processing Systems, pages 52040–52094, 2024.arxiv.org/abs/2404.07972
  21. [22]Haolong Yan, Jia Wang, Xin Huang, Yeqing Shen, Ziyang Meng, Zhimin Fan, Kaijun Tan, Jin Gao, Lieyu Shi, Mi Yang, Shiliang Yang, Zhirui Wang, Brian Li, Kang An, Chenyang Li, Lei Lei, Mengmeng Duan, Danxun Liang, Guodong Liu, Hang Cheng, Hao Wu, Jie Dong, Junhao Huang, Mei Chen, Renjie Yu, Shunshan Li, Xu Zhou, Yiting Dai, Yineng Deng, Yingdan Liang, Zelin Chen, Wen Sun, Chengxu Yan, Chunqin Xu, Dong Li, Fengqiong Xiao, Guanghao Fan, Guopeng Li, Guozhen Peng, Hongbing Li, Hang Li, Hongming Chen, Jingjing Xie, Jianyong Li, Jingyang Zhang, Jiaj u Ren, Jiayu Yuan, Jianpeng Yin, Kai Cao, Liang Zhao, Liguo Tan, Liying Shi, Mengqiang Ren, Min Xu, Manjiao Liu, Mao Luo, Mingxin Wan, Na Wang, Nan Wu, Ning Wang, Peiyao Ma, Qingzhou Zhang, Qiao Wang, Qinlin Zeng, Qiong Gao, Qiongyao Li, Shangwu Zhong, Shuli Gao, Shaofan Liu, Shisi Gao, Shuang Luo, Xingbin Liu, Xiaojia Liu, Xiaojie Hou, Xin Liu, Xuanti Feng, Xuedan Cai, Xuan Wen, Xianwei Zhu, Xin Liang, Xin Liu, Xin Zhou, Yifan Sui, Yingxiu Zhao, Yukang Shi, Yunfan Xu, Yuqing Zeng, Yixun Zhang, Zejia Weng, Zhonghao Yan, Zhiguo Huang, Zhuoyu Wang, Zihan Yan, Zheng Ge, Jing Li, Yibo Zhu, Binxing Jiao, Xiangyu Zhang, and Daxin Jiang. Step-GUI technical report. arXiv preprint arXiv:2512.15431, 2025.arxiv.org/abs/2512.15431
  22. [23]Jiabo Ye, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Zhaoqing Zhu, Ziwei Zheng, Feiyu Gao, Junjie Cao, Zhengxi Lu, Jitong Liao, Qi Zheng, Fei Huang, Jingren Zhou, and Ming Yan. Mobile-Agent-v3: Fundamental agents for GUI automation. arXiv preprint arXiv:2508.15144, 2025.arxiv.org/abs/2508.15144
  23. [24]Qize Yu, Lianrui Fan, Boyu Chen, Jiaqi Liang, Xini Ding, Yue Chen, Zetian Song, Yuran Wang, Yi Zou, Kaixuan Wang, Tianxing Chen, Wenxuan Song, Bohan Zhou, Mingleyang Li, Siqiao Huang, Yuqi Ye, Caigao Jiang, Wei Wei, Ruihai Wu, Hang Zhang, Xiaoao Ge, Shuchang Zhou, Shilong Liu, Xianming Liu, Ping Luo, and Shiyu Huang. GroundingPI: A grounding foundation model towards physical intelligence with visual primitives. arXiv preprint arXiv:2609.39601, 2026.arxiv.org/abs/2609.39601
  24. [25]Hanzhang Zhou, Xu Zhang, Panrong Tong, Jianan Zhang, Liangyu Chen, Quy u Kong, Chenglin Cai, Chen Liu, Yue Wang, Jingren Zhou, and Steven Hoi. MAI-GUI technical report: Real-world centric foundation GUI agents. arXiv preprint arXiv:2512.22047, 2025.arxiv.org/abs/2512.22047
  25. [26]Shijie Zhou, Viet Dac Lai, Hao Tan, Jihyung Kil, Wanrong Zhu, Changyou Chen, and Ruiyi Zhang. GUI-AIMA: Aligning intrinsic multimodal attention with a context anchor for GUI grounding. In Computer Vision – ECCV 2026, pages 79–97. Springer, 2026.arxiv.org/abs/2511.00810

Paper details

vlnc:W_c732uz9z
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents