WorksComputer Science
PACT: Plan-Action Consistency Training by Preserving Correct Plans and Repairing Actions in GUI Agents
Abstract
GUI agents can formulate a correct immediate plan in their reasoning yet emit an action that fails to execute it, creating an execution gap. This recurring mismatch undermines reliable task completion, as a single execution error can derail a multi-step workflow. In outcome-supervised reinforcement learning, broadcasting a negative response-level advantage across all generated tokens risks penalizing valid planning together with erroneous execution. To address this credit-assignment conflict, we introduce PACT (Plan-Action Consistency Training), a training framework that combines reference-aligned plan protection and plan-aligned action repair without altering the original scalar action reward. Using annotated reference actions to identify execution gaps, PACT protects the verified plan by masking negative policy-gradient contributions across its token span. Keeping the generated reasoning unchanged as context, it then favors the reference action over the emitted erroneous action through a preference surrogate with stop-gradient pair weights. Compared with the outcome-only baseline, PACT raises AndroidWorld task success from 60.34% to 68.97% (+8.63 pp), while improving rule-based plan-action consistency on AndroidControl High from 83.71% to 89.74% (+6.03 pp). Adapting PACT to desktop agents also improves task success from 40.92% to 44.17% (+3.25 pp) on the full 369-task OSWorld benchmark. Together, these results establish PACT as an effective credit-assignment framework for bridging execution gaps and improving task completion across mobile and desktop GUI environments.
Introduction
GUI agents must translate visual interfaces and user goals into executable interactions. Yet formulating a correct immediate plan in generated reasoning does not guarantee that the emitted action executes it. For example, an agent may state that it will tap a visible + button, yet emit an application-launch command instead (Figure 1). This execution gap is distinct from a planning error: the response already contains a valid immediate plan, but the final action fails to realize it. Building on prior diagnostic studies of reasoning–execution gaps in mobile agents [5], we investigate how training can preserve valid plans within failed responses while repairing their execution.
Figure 1. Execution gaps and targeted correction. Top: Both models continue the same supplied Pinterest reasoning; O launches the app, while PACT clicks the intended + control. Bottom: With independently generated reasoning and actions on Moon, O states a downward swipe but emits upward finger motion, while PACT executes the downward swipe. On-screen circles mark the click or gesture start; swipe arrows show finger motion. The separate training schematic illustrates reference-aligned plan protection of selected plans and additional action repair under a negative response advantage.
In reinforcement learning (RL), such execution gaps create a credit-assignment problem. Several GUI-agent methods use Group Relative Policy Optimization (GRPO) or its variants to learn from action feedback [7, 9]. In implementations that broadcast one response-level advantage across reasoning and action tokens [17], a negative advantage penalizes valid planning alongside erroneous execution. A scalar thought–action consistency reward [9] does not determine the direction of correction: higher agreement can result from adapting a plan to justify an incorrect action, and a flawed plan can be acted consistently. Effective correction therefore requires identifying which component the annotated reference action supports preserving.
Our central principle is to preserve a reference-supported plan and repair the action that fails to implement it. Here, a plan denotes the intended next GUI interaction expressed in the generated reasoning, and the plan span denotes the tokens expressing it. Verifying reference support for parameterized GUI actions requires checking spatial or textual arguments beyond action type alone, enabling selective credit assignment that shields valid plans while repairing failed actions.
We present PACT, which instantiates this principle within a group-relative RL pipeline while leaving the original scalar action reward intact. First, a conservative rule-based selector identifies responses featuring a reference-supported plan but an incorrect, inconsistent final action. Second, reference-aligned plan protection masks negative policy-gradient contributions within the plan span, leaving the advantages on action tokens and the remaining reasoning unchanged. Third, plan-aligned action repair pairs a reference-derived positive action suffix with the model’s emitted erroneous suffix under the same decoded reasoning trace via a stop-gradient pair-weight preference surrogate in the spirit of direct preference optimization (DPO) [15], outside the original GRPO rollout group. Both mechanisms operate strictly during training; inference retains the standard single-pass reasoning–action interface without extra reflection or re-planning.
PACT improves interactive task completion over the original GRPO baseline, reaching 68.97% success on AndroidWorld [16] (+8.63 pp) and 44.17% on the full OSWorld benchmark [21] (+3.25 pp). It also achieves 89.74% rule-based plan–action consistency on AndroidControl High [12] (+6.03 pp), alongside improved visual grounding on ScreenSpot-v2 Mobile [20].
Three complementary diagnostics examine why these gains arise. Fixed-plan continuation isolates action generation, showing that plan-aligned action repair improves execution of a valid plan. Aligned-state retention analyses demonstrate that reference-aligned plan protection reduces regressions on baseline-correct actions. Free-generation comparisons further show that protecting plans preserves more valid plans and enables more joint repairs, supporting the complementary roles of protection and repair beyond surface plan–action agreement.
Our main contributions are threefold:
Reference-aligned plan protection and plan-aligned action repair. PACT uses reference-supported plans to decouple plan preservation from action correction without altering the scalar action reward.
Task improvements across mobile and desktop environments. Comparisons against outcome-only and two repair baselines demonstrate consistent gains in dynamic task completion on AndroidWorld and OSWorld, supported by consistency and grounding evaluations on AndroidControl and ScreenSpot-v2 Mobile.
Mechanistic evidence on repair and retention. Through
fixed-plan continuation, free-generation, and baseline-correct retention analyses, we characterize the repair–retention trade-off while disentangling plan correctness, action validity, and plan–action consistency.
Related Work
Reasoning–execution alignment in GUI agents. Reliable GUI interaction demands both sound planning and faithful execution, motivating fine-grained evaluation beyond aggregate task success. VenusBench-Mobile [6] decomposes user-centric agent evaluation across perception, understanding, decision, action, and memory capabilities, while Dong et al. [5] specifically diagnose reasoning–execution gaps by separating the correctness of stated plans from that of emitted actions. On the training side, Faithful-Agent [9] combines evidence-grounded supervision, thought–action consistency rewards, and a guided advantage estimator, whereas MobileIPL [10] derives thinking-level preferences by propagating action feedback through reasoning trees to improve planning. In contrast, PACT uses reference-supported plans within unsuccessful responses to determine the direction of correction—preserving valid plans while repairing only their failed execution.
Fine-grained preference learning and action correction. Building on the pairwise objective of DPO [15], Step-DPO [11] targets individual reasoning steps conditioned on a correct prefix, whereas Agentic-DPO [2] contrasts expert actions against sampled mistakes at expert states. From a rollout-augmentation perspective, Octopus [4] recombines sampled trajectories and masks specific segments to separate self-correction from direct reasoning. PACT instead retains the model’s own decoded reasoning trace as the shared conditioning context to contrast the reference-derived action suffix against the emitted erroneous suffix.
Credit assignment within failed responses. Localized supervision also raises a credit-assignment question: which parts of an unsuccessful response should be penalized? In standard GRPO [17], a single group-relative advantage is broadcast across every token of a sampled response. Recent methods refine this assignment based on intermediate correctness, ranging from retrieval-calibrated negative advantages in CalibAdv [19] to teacher-identified first-error boundaries in Cliff [8]. In GUI agent training, UI-Venus-2 [18] introduces structured action-aware distillation to condition token-level supervision on action correctness and distinguish action-type errors from parameter errors. Within outcome-supervised GRPO, PACT masks negative policy-gradient contributions on reference-supported GUI plan spans and repairs failed action suffixes under the same plan, while keeping the scalar action reward intact.
Problem Formulation
Let denote the agent’s observable input context, comprising a user instruction, the current interface screenshot, and the available interaction history. Given , the policy autoregressively generates a reasoning trace followed by a GUI action :
During training, each context is paired with an annotated reference action .
The plan span in expresses the next intended GUI interaction. Building on reasoning–execution diagnostics [5], Table 1 defines , , and joint repair, and summarizes the four correctness regimes and their training implications in PACT.
| Metric | Success criterion | |
| The current plan is correct for the task. | ||
| The action is valid for the current task and state. | ||
| The action implements the model’s stated plan. | ||
| Joint repair | for the same plan–action pair. | |
| Training implication | ||
| 1 | 1 | Original action-reward objective. |
| 1 | 0 | Protect the plan; repair the action. |
| 0 | 1 | Original objective; no forced plan alignment. |
| 0 | 0 | Original objective; no reward for error agreement. |
Table 1. Step-level judgments and training implications in PACT. (a) Definitions (1: satisfied; 0: otherwise). (b) Correctness regimes. Protection and repair apply only to eligible gaps in Eq. (2) ().
All sampled responses remain in outcome-supervised GRPO. PACT adds protection and repair for reference-supported plans whose actions fail to implement them. For incorrect plans, correction instead requires revising the reasoning; valid actions should not be redirected to follow an incorrect plan.
During training, a rule-based selector evaluates each sampled response relative to its reference action . Let indicate whether the response admits the required reference-based judgments, and let , , denote the resulting proxies for , , . The execution-gap eligibility flag is
Unjudgeable responses () remain in the base RL objective without auxiliary supervision. These reference-relative proxies govern training-time selection, whereas benchmark evaluation and mechanism diagnostics use independent rule-consistency and screenshot-based semantic assessments.
PACT
Figure 2 summarizes the original GRPO path and PACT’s two complementary training mechanisms.
Figure 2. PACT training framework. Reference-verified execution-gap selection identifies reference-supported plans paired with incorrect, inconsistent actions. Reference-aligned plan protection zeros negative policy-gradient coefficients on the plan span. Plan-aligned action repair favors reference-derived over erroneous action suffixes under the same input and reasoning trace. Both feed the GRPO actor update.
Retaining the original action reward
PACT retains the original scalar action reward without introducing any plan-consistency bonus. For a sampled response , the standard parseable branch of the mobile scorer combines a format score, an action-type match indicator, and an action-specific parameter score:
where denotes the action type. All original parsing, missing-action, and grounding branches remain unchanged, with action-specific scoring rules and coordinate tolerances detailed in the supplementary material.
For each prompt, the pre-update policy samples responses. Outcome-supervised GRPO [17] computes the group-standardized advantage
where is the prompt group of response , and and are its reward mean and sample standard deviation. The baseline broadcasts to valid response tokens in a clipped token-mean policy loss regularized by a KL term against the fixed supervised fine-tuned (SFT) reference . Only sampled rollouts enter these groups; repair suffixes never affect group statistics or base rollout rewards.
Reference-verified execution-gap selection
The selector evaluates the eligibility flag in Eq. (2). In the mobile implementation, it scans the final four sentences of in reverse order for an explicit immediate plan, rejecting negation, multi-step ambiguity, and conflicting action types. Candidate plans are extracted solely from the reasoning text prior to reference comparison, never inferred retrospectively from the emitted action .
Judgeability () requires a parseable reference, a unique emitted action block, comparison metadata, and a non-terminal plan with checkable parameters: pointing actions require a unique explicit coordinate; scrolling actions require a single direction; text-entry and app-launch actions require a unique quoted argument; and supported parameter-free system actions require a valid action type. Grounding-only examples, terminal references, and unverifiable plans () remain active in the base RL objective but are excluded from auxiliary selection.
For judgeable responses, checks the plan parameters against , tests whether the base content score reaches the configured full-credit threshold, and checks whether the emitted action matches the plan under distinct compatibility tolerances. When , span localization and pair construction proceed independently, with extraction and fallback rules in the supplement.
Reference-aligned plan protection
Let indicate tokens within the localized plan span. PACT modifies the token-level policy advantage as
Without renormalizing group advantages, this clamp zeroes out negative policy-gradient coefficients on the reference-supported plan span while leaving positive advantages, action tokens, and the remaining reasoning tokens untouched. Crucially, protection acts strictly on local policy-gradient coefficients rather than freezing shared model parameters or masking the KL regularizer.
Plan-aligned action repair
For an eligible response , we construct a positive response by replacing only the action block with the training reference formatted in the model’s native action syntax and coordinate space, preserving the input and decoded reasoning text . Let denote the retained constructible pairs, where is scored via teacher forcing; unconstructible or overlength replacements are omitted from , while their original rollouts remain in the rollout batch .
Let select the action tags and action tokens, excluding reasoning and trailing tokens. We define the summed action-span log probability as
so that only action-span tokens enter the auxiliary objective while the decoded reasoning trace serves as their shared conditioning context.
We evaluate the log-ratio margin using and , and define the pair weight as
where is the repair weight, is the preference scale, is the sigmoid function, and is the original rollout batch size (rather than ). The suffix-repair surrogate is
Minimizing Eq. (8) favors the reference-derived suffix over the erroneous suffix under the same decoded reasoning trace. With pairs and masks fixed, its gradient at matches that of the corresponding conditional logistic preference objective [15] when both suffix terms are evaluated at that parameter point; reusing frozen weights across actor minibatches yields a linear log-probability surrogate rather than a repeatedly recomputed logistic loss, with the derivation and full rule specifications in the supplementary material.
Collecting all actor terms yields the complete objective:
where is the unchanged KL coefficient, and is absorbed into . Constructed positive responses contribute exclusively to , without entering the rollout batch , the modified policy loss , or the KL term. At inference, PACT uses the standard single-pass reasoning–action interface.
Training configurations
We compare four cumulative GRPO configurations in Table 2. In these configuration names, O denotes Original GRPO, A denotes Action Repair, and R denotes Reference-Verified Execution-Gap Selection.
| Configuration | Action Repair | Reference-Verified Execution-Gap Selection | Reference-Aligned Plan Protection |
| O | ✗ | ✗ | ✗ |
| OA | ✓ | ✗ | ✗ |
| OAR | ✓ | ✓ | ✗ |
| PACT | ✓ | ✓ | ✓ |
Table 2. Cumulative training configurations.
O (Original GRPO): no action repair or plan protection.
OA (O + A): adds action repair to O on supported, parseable, non-terminal actions below full credit (), without execution-gap selection or plan protection.
OAR (O + A + R): adds reference-verified execution-gap selection () to OA, yielding plan-aligned action repair without advantage clamping ().
PACT: adds reference-aligned plan protection to OAR.
All variants retain the original reward and reference KL. Repair variants construct pairs from their own rollouts using the same reference-action constructor and suffix surrogate. Optimization settings are in § 5 and the supplement.
Experimental Setup
Training. In our primary mobile study, O, OA, OAR, and PACT share the same Qwen3.5-9B vision–language model SFT initialization, whose frozen copy serves as the reference policy , along with 10,000 mobile GUI state–action training examples with pixel-space reference coordinates. Each variant is trained for 1,250 training iterations over two epochs, sampling 16 prompts and responses per prompt at each step to yield original rollout responses before auxiliary rows are added. Optimization uses one PPO epoch per rollout batch, a learning rate of , rollout temperature 1.0, KL coefficient , and 32 accelerators with FSDP2; repair-enabled variants set and retain at most 64 repair pairs per iteration. To evaluate cross-platform generality, we also train desktop-adapted counterparts of O, OA, OAR, and PACT. Platform-specific training settings, action-space adaptations, and optimization details are provided in the supplementary material.
Results and Analysis
Mobile task completion, consistency, and grounding
Benchmarks and roles. Our evaluation separates interactive task completion, plan–action consistency, and visual grounding. On AndroidWorld [16], we report the task success rate across its 116 interactive tasks for each checkpoint. On AndroidControl [12], we evaluate 8,444 states from 1,543 episodes in each of the High and Low settings: along with the current screenshot and available history, High provides only the task goal, whereas Low additionally supplies the step instruction. In both settings, we measure rule-based plan–action consistency and reference-action matching for retention and mechanism diagnostics. ScreenSpot-v2 Mobile [20] measures point-in-target hit rate across 501 examples (290 text and 211 icon targets). For desktop interaction, we report task success for each desktop checkpoint on the full 369-task OSWorld benchmark [21], counting a task as successful strictly when its native score equals 1.0.
Generation and measurement. Mobile evaluation uses greedy decoding (temperature 0), a 4,096-token output limit, and up to ten prior interaction steps. On AndroidControl, models observe recorded screenshots, with non-initial states conditioned on each model’s own generated history. OSWorld uses the same environment configuration, greedy decoding (temperature 0), a 100-step limit, and the official native evaluator. Distinct from the training-time selector, our AndroidControl consistency evaluator extracts a function-style or keyword-based action intent from the reasoning trace and compares the inferred action type and recoverable parameters with those of the emitted action. We report
with consistent and judgeable counts provided in the supplementary material. For action retention, we measure the proportion of O’s reference-matched states on which a compared variant also matches the reference.
In mechanism diagnostics, fixed-plan continuation conditions each model on the same input and the exact valid reasoning prefix generated by O to isolate action generation, whereas free generation requires each model to produce both reasoning and action from scratch. On these subsets, screenshot-based semantic assessments use language-model assistance and human verification to separately evaluate plan correctness (), action validity (), and semantic plan–action consistency (), complementing benchmark-wide rule-based scores; the supplementary material details sample selection, annotation protocols, and execution.
PACT improves interactive task completion. On AndroidWorld, PACT leads the matched RL configurations in Table 3(B) at 68.97% task success: +8.63 pp over O (60.34%), +6.04 pp over OAR, and +7.76 pp over OA. Table 3(A) provides published context. In a file-copy task, PACT selects the target folder, confirms copying, and finishes in 15 steps. OAR repeatedly types the folder name until the step limit (Supplement Figure S2).
**Task gains are accompanied by stronger plan–action consistency. On AndroidControl High, PACT leads in at 89.74% (+6.03 pp over O, +3.31 pp over OAR, +9.70 pp over OA), with the highest consistent-action count (7,046 states) and 41.1% fewer action-type mismatches than O, from 1,279 to 753 (Supplement Table S3). On Low, PACT raises over O from 86.39% to 87.56%, while OAR reaches 88.14% but incurs 217 more action regressions than PACT (§ 6.3).
Table S3. AndroidControl consistency and mismatch counts over 8,444 states per setting. Type, coordinate, and text columns partition the inconsistent judgeable outputs. Bold marks the best values within each setting, including ties.
| Setting | Model | Consistent / judgeable | Type↓ | Coord.↓ | Text↓ | ↑ |
| AC High | O | 6879 / 8218 | 1279 | 19 | 41 | 83.71 |
| OA | 6440 / 8046 | 1559 | 25 | 22 | 80.04 | |
| OAR | 6997 / 8096 | 1032 | 21 | 46 | 86.43 | |
| PACT | 7046 / 7852 | 753 | 16 | 37 | 89.74 | |
| AC Low | O | 7182 / 8313 | 1081 | 19 | 31 | 86.39 |
| OA | 6534 / 8327 | 1753 | 24 | 16 | 78.47 | |
| OAR | 7270 / 8248 | 906 | 24 | 48 | 88.14 | |
| PACT | 7155 / 8172 | 966 | 6 | 45 | 87.56 |
Table S3. AndroidControl consistency and mismatch counts over 8,444 states per setting. Type, coordinate, and text columns partition the inconsistent judgeable outputs. Bold marks the best values within each setting, including ties.
Visual grounding improves alongside task performance. On ScreenSpot-v2 Mobile, PACT raises the point-in-target hit rate from 91.82% (460/501) to 96.01% (481/501), improving on both text and icon targets and exceeding OAR (95.41%; Table 3(B)). PACT has the highest grounding score in Table 3(A). Notably, while PACT and OA tie on aggregate grounding accuracy, PACT leads OA by 7.76 pp on AndroidWorld, indicating that PACT’s advantage in multi-step task completion extends beyond static point localization.
| Model | Size | AndroidWorld SR (%) | OSWorld SR (%) | ScreenSpot-v2 Mobile (%) | |
| UI-TARS-7B-SFT [13] | 7B | 33.0 | 17.7 | 93.61 [13] | |
| UI-TARS-1.5-7B [1] | 7B | 30.0 [25] | 27.5 | 91.23 [26] | |
| UI-Venus-Navi-7B [7] | 7B | 49.1 | – | – | |
| GUI-Owl-7B [23] | 7B | 66.4 | 29.5 [3] | – | |
| Qwen3-VL-8B-Instruct [25] | 8B | 47.6 | – | 92.38 [25] | |
| Step-GUI-8B [22] | 8B | 67.7 | 40.2 | – | |
| Qwen3.5-9B [14] | 9B | 57.8 | 41.8 | 93.01 [24] | |
| PACT (ours) | 9B | 68.97 | 44.17 | 96.01 | |
| Configuration | AndroidWorld | OSWorld | AC High | AC Low | ScreenSpot-v2 Mobile |
| O | 60.34 | 40.92 | 83.71 | 86.39 | 91.82 |
| OA | 61.21 | 41.19 | 80.04 | 78.47 | 96.01 |
| OAR | 62.93 | 42.55 | 86.43 | 88.14 | 95.41 |
| PACT | 68.97 | 44.17 | 89.74 | 87.56 | 96.01 |
| (PACT−O), pp | +8.63 | +3.25 | +6.03 | +1.17 | +4.19 |
Table 3. Main benchmark results. (A) Published GUI-agent context. (B) Matched RL comparisons. (A) Published benchmark results: representative 7B–9B GUI agents. SR: task success rate. “–”: not listed. Published-result settings and aggregation details are provided in the supplement. (B) Main benchmark results under matched settings within each platform. Task success ↑. Plan–action consistency ↑. Grounding ↑. AC: AndroidControl; : rule consistency over judgeable states. Scores: %; : pp.
Extending PACT to desktop GUI agents
The task-level gains extend to desktop interaction. On the full 369-task OSWorld benchmark (Table 3(A,B)), PACT solves 163 tasks (44.17%), surpassing O (151, 40.92%) by 12 tasks, OAR (157, 42.55%) by 6 tasks, and OA (152, 41.19%) by 11 tasks. Action repair adds one success over O, while OAR and PACT yield larger gains, matching AndroidWorld’s PACT > OAR > OA > O ordering. On LibreOffice CSV export, PACT selects Text CSV and finishes in 9 steps. OAR misclicks PNG and reaches the 100-step limit (Supplement Figure S3).

Figure S3. OSWorld CSV export. PACT selects the requested format and completes the export with native success 1.0. OAR continues format selection through step 100. Green and red circles identify the actual PACT and OAR click locations, respectively.
Repairing failures while retaining baseline-correct actions
Effective repair must correct errors while retaining mastered behavior. Let denote states where O emits a reference-matching action. Table 4 reports retention, recovered errors, regressions, and net error change relative to O; Figure 3 shows net correct-action gains over OAR.
| Setting | Model | Ret. (%) | W→R ↑ | R→W ↓ | Net ↓ |
| AC High | OA | 87.89 | 669 | 638 | −31 |
| AC High | OAR | 86.34 | 591 | 720 | +129 |
| AC High | PACT | 89.51 | 631 | 553 | −78 |
| AC Low | OA | 90.03 | 436 | 656 | +220 |
| AC Low | OAR | 89.07 | 380 | 719 | +339 |
| AC Low | PACT | 92.37 | 476 | 502 | +26 |
Table 4. Reference-action retention on aligned AndroidControl states. : High 5,270; Low 6,580. Non-initial states use each model’s own generated history. W/R: wrong/right. Net error change vs. O: .
Figure 3. Retention and repair on aligned states (higher is better). Top: O-correct action retention. Bottom: net correct-action gains over OAR, computed as extra repairs plus avoided regressions. PACT leads retention across settings and net gains on AndroidControl.
Plan protection improves both retention and repair relative to OAR. PACT achieves the highest baseline-correct retention on AndroidControl at 89.51% on High and 92.37% on Low, versus 86.34% and 89.07% for OAR, preserving
167 and 217 additional baseline-correct states (23.2% and 30.2% fewer regressions) while recovering 40 and 96 more baseline errors, totaling +207 and +313 correct actions over OAR on High and Low, compared with +160 and +119 for OA (Figure 3).
Limiting regressions yields a superior trade-off over action repair. On AC High, OA recovers 38 more baseline errors than PACT (669 vs. 631), but PACT avoids 85 more regressions (553 vs. 638), yielding 47 more reference-action matches overall. On Low, PACT leads on both axes, preserving 154 more baseline-correct states and recovering 40 more errors for a net advantage of 194 matches. Thus, curbing regressions on mastered states translates directly into higher overall reference action accuracy than either repair baseline.
A similar retention advantage holds on ScreenSpot-v2 Mobile: PACT preserves 459/460 (99.78%) of O-correct targets (vs. 456/460 for OAR and 458/460 for OA), while all three variants acquire 22–23 new hits.
Fixed-plan execution versus free generation
We next examine execution on 23 identified AndroidControl Low initial states with empty history, where O stated a valid immediate plan but emitted an incompatible application-launch action (Table 5). Under fixed-plan continuation, all four models—including O—generate a fresh action continuation from the same screenshot, instruction, and exact O reasoning prefix, whereas free generation requires each model to produce both reasoning and action.
| Metric (%; out of 23 states) | O | OA | OAR | PACT |
| Fixed plan | ||||
| Reference action accuracy | 56.52 | 95.65 | 100.0 | 100.0 |
| Free generation | ||||
| Reference action accuracy | 0.0* | 95.65 | 95.65 | 95.65 |
| Plan–action consistency () | 0.0* | 100.0 | 100.0 | 100.0 |
| Semantic joint repair | 0.0* | 100.0 | 95.65 | 95.65 |
Table 5. Execution diagnostics on the 23-state execution-gap set. Asterisks (*) mark O’s selection criteria. Joint repair is defined in Table 1(a). Full semantic results are in the supplementary material.
Plan-aligned action repair resolves fixed-plan execution failures. Conditioned on the fixed valid plan, both PACT and OAR raise reference action accuracy from 56.52% (13/23) for O to 100.0% (23/23), while OA reaches 95.65% (22/23); Supplement Figure S4 shows cross-app examples. Holding the input and plan fixed isolates action generation, showing that suffix supervision translates a valid plan into the intended action.
Figure S4. Fixed-plan continuation on Pinterest, Google News, and Pepperfry. The plan excerpt is shared by O and PACT. Displayed actions are the actual fixed-plan continuations. Green circles mark PACT’s predicted clicks.
The repaired behavior transfers to free generation. When generating both plans and actions autonomously, OA, OAR, and PACT all achieve 95.65% (22/23) reference action accuracy; under semantic evaluation, PACT and
OAR achieve 95.65% (22/23) joint repair while OA reaches 100.0% (23/23). The additional semantic success for OA comes from a valid off-center product click outside the reference point-distance tolerance, as detailed in the supplementary material. Although all three reach 100.0% (23/23) plan–action consistency (), their joint-repair scores separate surface agreement with a generated plan from faithful execution of the requested step.
Preserving plans during free generation
To assess reference-aligned plan protection, we compare PACT with OAR in Table 6 on 30 natural execution-gap states covering action-type, scroll-direction, and input-order errors, with protocol details in the supplementary material.
| Subset | Model | Joint repair | |||
| All () | OAR | 19/30 | 13/30 | 25/30 | 13/30 |
| PACT | 23/30 | 19/30 | 28/30 | 19/30 | |
| Both models consistent , | OAR | 13/24 | 12/24 | 24/24 | 12/24 |
| PACT | 18/24 | 16/24 | 24/24 | 16/24 |
Table 6. Free-generation results on all 30 states and the 24 with , using Table 1(a) metrics.
PACT preserves more valid plans and completes more repairs. PACT leads OAR on all four metrics, raising joint repair from 43.33% to 63.33% (+20.00 pp). Among the 24 states where both models are plan–action consistent, PACT preserves five more correct plans (18/24 vs. 13/24) and completes four more joint repairs (16/24 vs. 12/24). For example, in Supplement Figure S1, PACT closes the blocking Myntra popup before typing, whereas OAR types into the covered field; Figure S5–Figure S8 illustrate additional single-step and boundary cases.
Figure S1. Free generation on the same Myntra state and history. PACT selects the required cancel action and achieves . OAR plans and emits direct typing, yielding .
Figure S5. Executing a stated plan. O formulates a valid search plan but emits an incompatible LaunchApp action. PACT and OAR execute the intended on-screen search click.
Figure S8. Consistency without correctness. ac-high:45:3. Task: turn on location (switch is already ON). Observed response (, , ). MODEL OUTPUT. PLAN SUMMARY (PARAPHRASED): The switch is off; toggle it again. ACTION: Click(box=(844, 314)). : wrong — Plan correctness. The enabled switch is read as off. : invalid — Action validity. The click toggles an enabled switch. : consistent — Plan-action consistency. The click implements the toggle plan. Coordinates use a 0–999 grid. Pred. Figure S8. Consistency without correctness (, , ). When instructed to enable location access on a screen where the switch is already ON, the model misreads the toggle state as disabled and clicks it, consistently executing an incorrect plan.
Discussion and Limitations
Why plan protection complements action repair. Across multi-step benchmarks, the complete PACT framework achieves substantially higher task success on AndroidWorld and the highest task success on OSWorld within the matched comparisons of Table 3(B), while preserving more baseline-correct actions across aligned states in Table 4, even though OAR and OA match or approach PACT on isolated single-step grounding in Table 3(B) and fixed-plan continuation in Table 5. These results clarify the complementary roles of our two mechanisms: plan-aligned action repair teaches the policy to translate a valid plan into the intended action, whereas reference-aligned plan protection shields that plan from negative policy-gradient updates on failed rollouts. Because each action in closed-loop interaction alters the context for subsequent turns, retaining mastered behaviors is as critical as recovering from errors, enabling PACT to drive stronger end-to-end task completion across mobile and desktop environments.
Limitations and future directions. First, our training-time selector requires explicit plans whose parameters can be verified against a single annotated reference. Consequently, valid but implicit plans or semantically equivalent alternative actions may fall outside the eligible set; incorporating semantic plan verifiers or multi-reference supervision could broaden the coverage of repairable failures. Second, whereas our OSWorld experiments establish task-level gains on desktop workflows, our fine-grained plan and fixed-plan continuation diagnostics focus on mobile states. Extending these mechanistic analyses to richer desktop action spaces—such as keyboard shortcuts, composite operations, and long-horizon workflows—will clarify how both mechanisms interact in complex environments.
Conclusion
We presented PACT, a training framework that couples reference-aligned plan protection with plan-aligned action repair to resolve GUI execution gaps without altering the original scalar action reward. It shields reference-supported plan spans from negative policy-gradient updates while optimizing corrected action suffixes under the same decoded reasoning trace. Compared with outcome-only RL, PACT substantially improves AndroidWorld task success and achieves higher task success on OSWorld, alongside mobile plan–action consistency and visual grounding. Aligned-state retention analyses further show that PACT preserves a higher proportion of baseline-correct actions than unprotected repair variants. Moreover, fixed-plan continuation and free-generation diagnostics confirm the complementary roles of both mechanisms while distinguishing true action correctness from surface-level plan–action agreement. Together, these findings establish a correctness-guided credit-assignment principle for GUI agents: preserve valid plans and selectively repair their execution, rather than optimizing consistency alone.
LLM Usage Statement. Large language models (GPT and Gemini) were used for manuscript copy-editing, figure-layout formatting, and assisting with initial candidate screening prior to human verification in our diagnostic semantic evaluations (§ 5). All experimental designs, implementations, analyses, and final judgments were conducted and verified by the authors.
Implementation Details
Sections A.1–A.5 detail the reward, selection and construction rules, gradient derivation, and optimization schedule. Section B provides benchmark breakdowns and diagnostic protocols, and Section C presents qualitative cases.
Original action reward and coordinate tolerances
Within each platform, all four training configurations share the scalar outcome scorer defined in Eq. (3) of the main paper. In the mobile scorer’s standard parseable branch, the format term assigns +0.5 for valid <think>...</think><action>...</action> structure and −0.5 otherwise. Matching the reference action type contributes 1.0 point, followed by the action-specific parameter score . Missing actions, unparseable outputs, and grounding-only samples retain the base scorer’s original branches.
For mobile point-based actions (e.g., Click, LongPress), predicted coordinates are emitted on a normalized 0–999 grid and mapped back to native image pixels. Let denote the Euclidean pixel distance between the predicted and reference coordinates, and let be the shorter screen dimension. The parameter score is
For Scroll/Swipe actions with directional metadata, the scorer checks both gesture endpoints and assigns 1.5 parameter points when both satisfy the strict coordinate tolerance. The branch with endpoint coordinates alone uses its standard Boolean endpoint comparison. Text-based actions (Type, LaunchApp, Answer) receive full parameter credit under exact normalized string match or a token-set F1 score exceeding 0.5.
Reference-verified execution-gap selection and plan span localization
To compute the execution-gap eligibility flag in Eq. (2) of the main paper, the programmatic selector first evaluates judgeability . Setting requires a unique emitted action block, a parseable reference action with comparison metadata, and a non-terminal plan with checkable parameters. Grounding-only samples and terminal references (Finished, Answer, CallUser) are assigned and participate in the base GRPO objective.
The following rules define the three binary proxies for judgeable mobile responses ().
Action correctness proxy (): if the base content score (action-type bonus plus parameter score) reaches the full-credit threshold (2.0 points for standard actions, or 2.5 points for normalized
Scroll/Swipewith directional endpoints). Responses receiving only partial parameter credit are treated as .Plan correctness proxy (): The reasoning trace is segmented into clauses using line breaks and English/Chinese punctuation, and the final four sentences are scanned in reverse order. A valid plan must specify a single immediate action without negation or multi-step ambiguity. Setting requires the extracted plan parameters to match . Pointing plans require a unique explicit 0–999 coordinate within the strict pixel tolerance of . Directional plans require a unique matching direction. Text and app-launch plans require a unique quoted argument matching the normalized reference string. Parameter-free system actions must match by action type.
Plan–action compatibility proxy (): if the emitted action matches the extracted plan in action type, lies within a grid distance of 1 on the normalized 0–999 coordinate space for pointing actions, and matches any specified direction, text, or application argument.
For pointing actions, checks proximity to the reference coordinate, whereas checks agreement with the explicitly stated coordinate. PACT selects repair candidates using the full eligibility condition , including .
For reference-aligned plan protection (), the character span of the verified plan is mapped to response token indices to form the binary mask . The implementation uses tokenizer character offsets whenever the re-encoded reasoning prefix matches the active token sequence, and otherwise performs an exact token-subsequence search prior to the action tag. If a unique token span cannot be localized, we set for all , leaving the base advantage unchanged while still permitting suffix repair if pair construction succeeds.
Constructing positive action suffixes
For each eligible mobile response , the positive action suffix is synthesized from the training reference in the model’s native action syntax and 0–999 coordinate grid. Because already verifies that the extracted plan targets the reference UI element within the strict reference tolerance, synthesizing from under the same decoded reasoning trace provides a reference-compatible execution of the stated plan. Reference pixel coordinates are converted to the normalized grid, rounded, and clamped to valid bounds. Scroll references with endpoint coordinates are formatted as Swipe actions. Reference actions with missing endpoints or text arguments that cannot be serialized in the native action syntax, as well as overlength replacements, are omitted from . Their original rollouts remain in .
On both platforms, the constructed positive sequence retains the exact input context (instruction, screenshot, and interaction history) and the decoded reasoning text , replacing only the <action>...</action> block. The binary action masks and cover only the action tags and action tokens, excluding all reasoning tokens and any trailing text. At most 64 constructible pairs are retained per training iteration in batch order. In the OA baseline (original GRPO with Action Repair), pairs are constructed using the identical reference formatter from parseable, non-terminal rollouts with and , without requiring or .
First-order gradient derivation of the suffix-repair surrogate
Let denote the summed action-span log probability defined in Eq. (6) of the main paper, and let the log-ratio margin be
A conditional DPO-style logistic preference objective [11, 15] over the retained pair set is given by
Here, is the repair weight and is the preference scale. Taking the gradient with respect to and evaluating at the pre-update policy (where ) yields
Detaching the pre-update weight (where matches Eq. (7) of the main paper) via produces the linear suffix-repair surrogate in Eq. (8) of the main paper, whose gradient matches Eq. (13) at .
Optimization schedule, hyperparameters, and desktop adaptation
Training data and optimization. Table S1 summarizes the RL configurations for both platforms. All four configurations (O, OA, OAR, and PACT; main Table 2) on both platforms start RL training from the same SFT checkpoint. We obtain this checkpoint by supervised fine-tuning the Qwen3.5-9B vision-language model on multimodal GUI trajectories to generate structured <think>...</think> <action>...</action> responses, with spatial action coordinates normalized to 0–999. A frozen copy of this checkpoint serves as the KL reference policy . Within each platform, the four variants share the base-training configuration in Table S1. Desktop RL training uses 18,000 examples sampled without replacement (seed 42) and runs for one epoch with complete 32-prompt batches (562 iterations), preserving the original system instructions, interaction history, screenshots, and reference actions. All training data are strictly disjoint from the evaluation benchmarks.
| Setting | Mobile | Desktop |
| Initialization and fixed reference | Qwen3.5-9B SFT | Qwen3.5-9B SFT |
| RL training examples | 10,000 | 18,000 |
| Training iterations / epochs | 1,250 / 2 | 562 / 1 |
| Prompts per iteration / responses per prompt () | 16 / 8 | 32 / 8 |
| Original rollout batch size () | 128 | 256 |
| Optimizer | AdamW | Distributed Adam |
| Learning rate / PPO epochs | / 1 | / 1 |
| Learning-rate schedule | Constant | Constant |
| Weight decay / gradient clipping | 0.01 / 1.0 | 0.01 / 1.0 |
| Policy-loss aggregation / entropy coefficient | Token mean / 0 | Token mean / 0 |
| Probability-ratio clipping / dual clipping | 0.2 / 3.0 | 0.2 / 3.0 |
| Sampling temperature / top- / top- | 1.0 / 1.0 / | 1.0 / 1.0 / |
| Maximum prompt / response tokens | 36,864 / 2,048 | 40,960 / 2,048 |
| Accelerators / training framework | 32 / FSDP2 | 64 / Megatron |
| Rollout tensor parallelism | 4 | 4 |
| Numerical precision | bfloat16 | bfloat16 |
| KL coefficient () / estimator | 0.01 / k3+ | 0.01 / low_var_kl |
| Repair weight () / preference scale () | 0.1 / 0.1 | 0.1 / 0.1 |
| Maximum repair pairs per iteration | 64 | 64 |
Table S1. Training hyperparameters for the mobile and desktop studies. Repair parameters apply to OA, OAR, and PACT.
The desktop action interface supports pointing, keyboard, drag/scroll, and control actions. Training references are dominated by Click (43.0%), Hotkey (26.1%), and Type (21.2%). Each training example pairs the multimodal input context (system instructions, interaction history, and screenshots) with its ground-truth reference action.
Both platforms use token-mean policy loss and apply KL regularization in the actor objective with . The mobile estimator combines a clipped value with a straight-through squared-log-ratio gradient. Desktop training uses , with clipped to and the estimator clipped to . Original rollout groups determine GRPO advantages. Constructed positive responses carry the suffix-repair loss with zero PG and KL masks, and signed pair weights preserve the sum in Eq. (8) of the main paper under data-parallel averaging. On 32 accelerators, the mobile 1,250-iteration training loop takes 120.76 hours for O and 123.19 hours for PACT, excluding initial model loading, corresponding to approximately 2.0% overhead.
Algorithm S1 summarizes one mobile PACT training iteration. Selection, span localization, and pair construction use Sections A.2 and A.3.
Desktop plan verification and suffix construction. The desktop selector applies the same execution-gap condition to explicit plans. Table S2 sets out the supported action parameters. Reference support compares the plan with the reference action, while compatibility compares the plan with the emitted action. Both comparisons require matching action types. Desktop action correctness requires the full native content score of 2.0, comprising one point for type and one for parameters.
| Action | Required parameters | Reference support / plan compatibility | Positive suffix construction |
| Click, DoubleClick, RightClick | One box coordinate | : distance < 49.95 (0.05 × 999). : distance ≤ 1. | Copy reference box. |
| Drag, Swipe | start and end coordinates | : each endpoint distance < 139.86 (0.14 × 999). : each endpoint distance ≤ 1. | Copy both reference endpoints. |
| Type | One quoted content string | : exact string equality, preserving whitespace and escapes. | Copy reference content. |
| Hotkey | One nonempty keys list or tuple | : equality of normalized, ordered key tuples. | Copy reference keys. |
| Wait | No arguments | : matching Wait action. | Wait(). |
| CallUser, Finished | Optional content string | Terminal actions receive native scoring with exact content matching. | Excluded from auxiliary selection. |
Table S2. Desktop training-action verification and reference-derived suffix construction. Spatial distances are Euclidean distances on the normalized 0–999 grid.
For desktop pointing actions, the native parameter score is 1.0 at grid distances below , 0.5 from 49.95 up to but excluding 99.9 = 0.10 × 999, and zero otherwise. The reference-support test uses the full-credit threshold. Drag and Swipe plans specify exactly two coordinate pairs in start–end order. For Type, the native parameter score awards 1.0 to exact text matches and 0.5 to non-exact matches with the same trailing-whitespace category and token-set F1 greater than 0.5. Hotkey comparison strips surrounding whitespace and lowercases each key while preserving tuple order and the literal space key.
Desktop reference coordinates already use the model’s normalized grid. The positive suffix serializes the reference action name and parameters with Python literal representations and is checked by the same action parser. The resulting pair retains the original input and decoded reasoning context, following Section A.3.
Evaluation Protocols and Additional Quantitative Results
AndroidControl rule-based consistency
On AndroidControl, the rule-based consistency evaluator first searches the reasoning trace for the final function-style action expression; if none is present, it extracts the last English or Chinese action keyword after resolving keyword overlaps, and parses any accompanying coordinate, direction, or text arguments. Disagreements between the extracted intent and the emitted action are categorized into action-type or direction mismatches (), coordinate mismatches (), and text-argument mismatches (). Rule-based consistency is computed over all judgeable outAlgorithm S1 One mobile PACT training iteration
Require: Pre-update policy , reference , 16 prompts with reference actions, , pair cap 64
Sample original rollout batch ,
Broadcast ; set
for each response in batch order do
Compute , , , and using main Eq. (2)
if then
Localize (use a zero mask if no unique span is found)
Clamp plan-token advantages using main Eq. (5)
Independently construct from
if the pair is constructible and then
Append the pair and its action masks to
end if
end if
end for
if then
Pack positive responses into a 128-row auxiliary batch with zero-loss padding
Score both suffixes under and ; compute and freeze using main Eqs. (6)–(7)
Assign to sampled negative suffixes, to constructed positive suffixes, and zero to padding
Set auxiliary PG and KL masks to zero
end if
Update policy parameters by minimizing the actor objective in main Eq. (9)
puts as
Responses lacking a reasoning block, an extractable action intent, or a valid reference label are excluded from the denominator of . Table S3 reports consistent and judgeable counts over 8,444 states per setting. These states comprise all annotated action steps in the 1,543 evaluation episodes, with identical episode–step pairs used for High and Low. On AC High, PACT achieves the highest (89.74%) and absolute consistent-action count (7,046 vs. O: 6,879; OAR: 6,997; OA: 6,440), remaining first with all 8,444 states as a common denominator. Action-type mismatches account for most rule inconsistencies. PACT reduces these mismatches from 1,279 to 753 on High, and has the fewest coordinate mismatches on both High (16) and Low (6).
Because relies on surface keyword and function extraction, it provides a scalable benchmark-wide proxy rather than a full semantic parse. For example, mentioning an in-app “Home” tab can trigger the system PressHome keyword rule, and discussing a subsequent step at the end of a multi-clause sentence can override an earlier immediate click intent. These boundary cases motivate our complementary screenshot-based semantic diagnostics in the main paper.
Interactive task evaluation on AndroidWorld and OSWorld
AndroidWorld. For each mobile checkpoint, we report the task success rate across AndroidWorld’s 116 interactive tasks, yielding 70/116 (60.34%) for O, 71/116 (61.21%) for OA, 73/116 (62.93%) for OAR, and 80/116 (68.97%) for PACT.
OSWorld. All four desktop-adapted variants (O, OA, OAR, and PACT) are evaluated on the full 369-task OSWorld benchmark. Evaluation uses a standardized Linux x86/Docker environment with a read-only Ubuntu base image and the official native evaluation scripts. All four variants use temperature 0 and a 100-step interaction limit. Input construction and generation parameters are given in Section B.5. At each step, the complete emitted action block is parsed and executed. A task is counted as successful strictly when its native evaluator returns a full score of 1.0:
without counting partial completion scores as successes. The totals are 151/369 (40.92%) for O, 152/369 (41.19%) for OA, 157/369 (42.55%) for OAR, and 163/369 (44.17%) for PACT. PACT completes 12 more tasks than O, 11 more than OA, and 6 more than OAR. The Step-GUI-8B and Qwen3.5-9B scores in main Table 3(A) use OSWorld-Verified.
ScreenSpot-v2 Mobile subgroup breakdown and target retention
Table S4 and Table S5 provide the target-category breakdown and baseline-correct retention statistics on ScreenSpot-v2 Mobile. Because ScreenSpot-v2 Mobile consists of single-step grounding queries without multi-turn history, Table S5 offers a history-independent test of capability preservation: PACT retains 99.78% (459/460) of O-correct targets while incurring only a single right-to-wrong regression. For the four cited ScreenSpot-v2 Mobile scores in main Table 3(A), text/icon accuracies are weighted by 290/211 targets.
Table S4. ScreenSpot-v2 Mobile point-in-target accuracy across all 501 examples, 290 text targets, and 211 icon targets.
| Configuration | All targets (501) | Text targets (290) | Icon targets (211) |
| O | 460 (91.82%) | 271 (93.45%) | 189 (89.57%) |
| OA | 481 (96.01%) | 286 (98.62%) | 195 (92.42%) |
| OAR | 478 (95.41%) | 286 (98.62%) | 192 (91.00%) |
| PACT | 481 (96.01%) | 285 (98.28%) | 196 (92.89%) |
Table S4. ScreenSpot-v2 Mobile point-in-target accuracy across all 501 examples, 290 text targets, and 211 icon targets.
| Configuration | Retained / 460 | Retention (%) | Wrong to right (↑) | Right to wrong (↓) |
| OA | 458 / 460 | 99.57 | 23 | 2 |
| OAR | 456 / 460 | 99.13 | 22 | 4 |
| PACT | 459 / 460 | 99.78 | 22 | 1 |
Table S5. Target-level retention and repair on ScreenSpot-v2 Mobile (460 O-correct targets).
Mechanism diagnostic protocols and complete results
For both diagnostic sets, screenshot-based semantic assessments use language-model assistance followed by human verification, with separate judgments of plan correctness (), action validity (), and semantic plan–action consistency (). Here, evaluates whether the current plan is correct for the task, evaluates action validity in the current task and state, and evaluates whether the action implements the model’s stated plan. Joint repair requires for the same plan–action pair, as in Table 1(a) of the main paper. When a valid plan () is paired with an invalid action (), consistency is judged as if the action targets a different UI element or operation than stated (e.g., landing on an unrelated control in Figure S7), and if it targets the stated element or text but fails task validity due to boundary localization or interface state constraints.
Figure S7. Coordinate grounding error. Zara: view the results for the selected child size. ac-high:55:8. Observed response (, , ). MODEL OUTPUT. PLAN SUMMARY (PARAPHRASED): Click VIEW RESULTS (6). ACTION: Click(box=(750, 585)). : correct — Plan correctness. Targets VIEW RESULTS. : invalid — Action validity. The click lands on the size option. : inconsistent — Plan-action consistency. The click does not implement the plan. Coordinates use a 0–999 grid. Pred. Figure S7. Coordinate-error illustration (, , ). The model identifies VIEW RESULTS as the intended target, but its click lands on a neighboring size option.
23-state fixed-plan continuation and free generation. The 23 diagnostic states were identified among AndroidControl Low initial steps (empty interaction history) where O articulated a valid immediate plan but emitted an erroneous application-launch action. Under the fixed-plan condition, each model receives the identical task instruction, screenshot, empty history, and O’s exact valid reasoning prefix terminating at the <action> opening tag, and deterministically generates the action continuation in a single turn. For a matched comparison with OA, OAR, and PACT, O also generates a fresh action continuation from the same fixed prefix, yielding 13/23; its 0/23 free-generation score comes from the original outputs used to select these states. Under free generation, the assistant prefix is removed and each model generates both reasoning and action from scratch. Reference-action matching and screenshot-based semantic evaluation are reported side by side in Table 5 of the main paper. On the single state where OA’s semantic action-validity and reference-matching judgments differ (a Nike product-card screen), OA clicks a valid off-center region of the product image outside the point-distance tolerance of the single reference coordinate, whereas PACT and OAR score 22/23 under both criteria. Table S6 reports the complete fixed-plan continuation and free-generation results, including the semantic breakdown. Following our coordinate-based action interface, reference-action matching evaluates predictions directly against each state’s annotated reference action (without accessibility-tree bounding boxes or cross-type equivalences), requiring matching action types and reference-compatible parameters
| Condition / metric | O | OA | OAR | PACT |
| Fixed plan: reference action accuracy | 56.52 (13/23) | 95.65 (22/23) | 100.0 (23/23) | 100.0 (23/23) |
| Free generation: reference action accuracy | 0.0 (0/23)* | 95.65 (22/23) | 95.65 (22/23) | 95.65 (22/23) |
| Free generation: plan correctness () | 100.0 (23/23)* | 100.0 (23/23) | 95.65 (22/23) | 95.65 (22/23) |
| Free generation: action validity () | 0.0 (0/23)* | 100.0 (23/23) | 95.65 (22/23) | 95.65 (22/23) |
| Free generation: plan–action consistency () | 0.0 (0/23)* | 100.0 (23/23) | 100.0 (23/23) | 100.0 (23/23) |
| Free generation: semantic joint repair | 0.0 (0/23)* | 100.0 (23/23) | 95.65 (22/23) | 95.65 (22/23) |
Table S6. Complete fixed-plan continuation and free-generation results on the 23-state execution-gap diagnostic set (%, with exact counts out of 23 in parentheses). Joint repair requires . Asterisks (*) mark O’s selection criteria. Bold indicates row-wise maxima, including ties.
point-distance tolerance for clicks, matching swipe direction, and normalized text match).
30-state natural execution-gap set. This set comprises O-selected execution-gap candidates from AndroidControl High and Low, targeting valid immediate plans paired with invalid, inconsistent actions. Eleven states use empty history and 19 use O’s interaction history. OAR and PACT freely generate plans and actions from identical screenshots, instructions, and available O histories, without O’s current response or the reference action. This set and the 23-state fixed-plan diagnostic follow distinct selection protocols and are analyzed separately.
All 30 states remain in each denominator; only verified successes enter the numerator. The ambiguous-reference boundary case scores as a non-success for and joint repair, with evaluated independently. In Table S7, PACT raises from 63.33% (19/30) to 76.67% (23/30), from 83.33% (25/30) to 93.33% (28/30), and both and joint repair from 43.33% (13/30) to 63.33% (19/30), +20.00 pp. Among the 24 states with , PACT retains 18/24 correct plans versus 13/24 for OAR (+20.83 pp) and completes 16/24 joint repairs versus 12/24 (+16.67 pp). On the remaining six states, OAR generates six valid plans but only one consistent action and joint repair; PACT achieves consistency on four states and joint repair on three.
| Candidate category | States |
| Click-versus-launch conflict | 8 |
| Scroll-direction error | 2 |
| Premature text entry / skipped selection | 17 |
| Click-versus-wait conflict | 1 |
| Swipe-versus-click/selection conflict | 1 |
| Ambiguous-reference boundary case | 1 |
| Total | 30 |
| Subset | Model | ↑ | ↑ | ↑ | Joint repair↑ |
| All () | OAR | 63.33 (19/30) | 43.33 (13/30) | 83.33 (25/30) | 43.33 (13/30) |
| All () | PACT | 76.67 (23/30) | 63.33 (19/30) | 93.33 (28/30) | 63.33 (19/30) |
| Both models consistent , | OAR | 54.17 (13/24) | 50.00 (12/24) | 100.0 (24/24) | 50.00 (12/24) |
| Both models consistent , | PACT | 75.00 (18/24) | 66.67 (16/24) | 100.0 (24/24) | 66.67 (16/24) |
| Remaining () | OAR | 100.0 (6/6) | 16.67 (1/6) | 16.67 (1/6) | 16.67 (1/6) |
| Remaining () | PACT | 83.33 (5/6) | 50.00 (3/6) | 66.67 (4/6) | 50.00 (3/6) |
Table S7. Natural execution-gap composition and free-generation results on all 30 states, the 24 with , and the remaining six. Bold marks the better value within each comparison, including ties.
Prompts, input construction, and generation settings
Training inputs. Both platforms retain the original system prompt, interaction history, and screenshots from each training example. Images fill the message placeholders in their recorded order. Historical assistant messages contain the associated reasoning and actions. The current reference action is stored in a separate reference field.
Mobile evaluation. The system prompt contains the task goal, action definitions, and the <think>...</think><action>...</action> response format. On AndroidControl Low, the goal is followed by Current step: and the low-level instruction. High uses the episode goal. The current user message contains Current Screenshot: and the image. The most recent ten interaction rounds contribute the model’s own reasoning and action text. AndroidControl uses the recorded state image at each step. AndroidWorld supplies the current environment screenshot and uses an interaction budget of .
ScreenSpot-v2 uses an empty interaction history and the instruction In the screenshot, click the element that matches this description: followed by the quoted target description. Mobile generation uses temperature 0 and a maximum of 4,096 output tokens. For execution, normalized 0–999 coordinates are linearly rescaled to integer pixel coordinates on the screen.
Desktop evaluation. The desktop system prompt embeds the task goal and defines Click, DoubleClick, RightClick, Drag, Swipe, Type, Hotkey, Wait, CallUser, and Finished. Each input contains the current 1920×1080 screenshot and the two most recent historical screenshots. Earlier steps retain historical reasoning and action blocks. All four desktop configurations use temperature 0, a maximum of 4,096 output tokens, and at most 100 interaction steps. Task success is determined strictly by the official native evaluator.
Fixed-plan continuation. The fixed-plan input supplies the same system message, screenshot, and verbatim valid O reasoning prefix to each model. The assistant prefix terminates immediately after the opening action tag.
system [task and action definitions]
user [Current Screenshot: and the state
→ image]
assistant <think>[O's exact valid
→ reasoning]</think><action>
temperature=0, max_tokens=4096 continue_final_message=true
The completed response concatenates the supplied prefix and generated suffix. Free generation uses the same state input and generates both reasoning and action in a new assistant response. In the 30-state natural execution-gap set, 11 states use empty history and 19 supply the same recorded O-history steps to both OAR and PACT (e.g., two prior steps on Myntra in Figure S1), using identical mobile generation settings.
Qualitative Case Studies
Comparative Success Cases Across Mobile and Desktop
Preserving the next-step requirement. The Myntra example in Figure S1 uses the same screenshot and two preceding O-history steps for PACT and OAR. The task requires closing the modal before text entry. PACT preserves that ordering in its immediate plan and clicks the close button. OAR assumes that the covered search field is ready and emits Type. Both outputs have , while only PACT meets the plan, action, and joint-repair criteria. This example illustrates how PACT preserves a valid plan and completes the repair during free generation.
Executing a complete mobile workflow. In Figure S2, all four configurations receive the same receipt-copy goal and task seed. PACT selects Download, confirms the copy, and emits Finished at step 15. OAR emits Type(content='Download') at steps 9, 10, 12, 14, and 16. The native final score is 1.0 for PACT and 0.0 for O, OA, and OAR.

Figure S2. AndroidWorld receipt-copy task. PACT selects the destination, confirms copying, and terminates with native success. OAR repeatedly emits the folder name. Step numbers are one-based.
Completing a desktop export task. Figure S3 shows the same OSWorld task across the four configurations. PACT uses the Save As workflow, selects Text CSV, confirms the export options, and finishes in nine steps. Its exported file passes the official CSV-content comparison. O, OA, and OAR receive native scores of zero at the 100-step limit.
Action repair with a fixed valid plan. Figure S4 provides three examples from the 23-state set. Each O/PACT comparison uses the same task, screenshot, and verbatim O reasoning prefix ending at the <action> opening tag. The prompt-token sequences are identical between O and PACT within each case. O continues with an application-launch command in all three examples. PACT generates the on-screen click specified by the supplied plan. The corresponding PACT free-generation outputs also satisfy .
Step-level plan execution and joint correction. Figures S5 and S6 further compare O, OAR, and PACT on individual AndroidControl states. In Figure S5, O states a valid plan to tap the search icon in Google News but emits an incompatible LaunchApp command, whereas both OAR and PACT execute the intended on-screen click. In Figure S6, where the instruction requests tapping Send in an email composer, O plans to edit the message body and long-presses the subject line, whereas OAR and PACT jointly correct both the immediate plan and the emitted click.
Figure S6. Joint plan and action correction. O plans to edit the email body and long-presses the subject line. PACT and OAR correct both the plan and action to click Send.
Plan-aligned action repair across apps
Within each case, O and PACT share the input and valid O plan.
Diagnostic Taxonomy Boundary Illustrations
Figures S7 and S8 illustrate two distinct , , configurations using observed agent responses, a correct plan with incorrect spatial execution (1, 0, 0) and an incorrect plan executed consistently (0, 0, 1). These examples distinguish plan correctness, action validity, and plan–action consistency.
References
- [1]ByteDance Seed. UI-TARS-1.5-7B: Official model card. Hugging Face, https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B, 2025.
- [2]Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories. arXiv preprint arXiv:2607.10601, 2026.
- [3]Yuhan Chen, Yuxuan Liu, Long Zhang, Pengzhi Gao, Jian Luan, and Wei Liu. STEP: Success-rate-aware trajectory-efficient policy optimization. In Findings of the Association for Computational Linguistics: ACL 2026, pages 30681–30692. Association for Computational Linguistics, 2026.
- [4]Yi Ding, Ziliang Qiu, Bolian Li, and Ruqi Zhang. Learning in Self-Correction in Vision-Language Models via Rollout Augmentation. In Proceedings of the 43rd International Conference on Machine Learning, pages 25096–25113. PMLR, 2026.arxiv.org/abs/2602.08503
- [5]Lingzhong Dong, Ziqi Zhou, Shuaibo Yang, Haiyue Sheng, Pengzhou Cheng, Zongru Wu, Zheng Wu, Gongshen Liu, and Zhuosheng Zhang. Say One Thing, Do Another? Diagnosing Reasoning-Execution Gaps in VLM-Powered Mobile-Use Agents. arXiv preprint arXiv:2510.02204, 2025.arxiv.org/abs/2510.02204
- [6]Yichen Gong, Zhuohan Cai, Sunhao Dai, Yuqi Zhou, Zhangxuan Gu, Changhua Meng, and Shuheng Shen. VenusBench-Mobile: A challenging and user-centric benchmark for mobile GUI agents with capability diagnostics. In Proceedings of the 43rd International Conference on Machine Learning, pages 35862–35887. PMLR, 2026.arxiv.org/abs/2604.06182
- [7]Zhangxuan Gu, Zhengwen Zeng, Zhenyu Xu, Xingran Zhou, Shuheng Shen, Yunfei Liu, Beitong Zhou, Changhua Meng, Tianyu Xia, Weizhi Chen, Yue Wen, Jingya Dou, Fei Tang, Jinzhen Lin, Yulin Liu, Zhenlin Guo, Yichen Gong, Heng Jia, Changlong Gao, Yuan Guo, Yong Deng, Zhenyu Guo, Liang Chen, and Weiqiang Wang. UI-Venus technical report: Building high-performance UI agents with RFT. arXiv preprint arXiv:2508.10833, 2025.
- [8]Peixuan Han, Runhui Wang, Ketan Ramaneti, Jie Hao, Gerald Friedland, and Chris Kong. Cliff: Learning Process Rewards from the First Mistake. arXiv preprint arXiv:2609.02817, 2026.arxiv.org/abs/2609.02817
- [9]Haowen Hu, Pengzhou Cheng, Zheng Wu, Lingzhong Dong, Gongshen Liu, and Zhuosheng Zhang. Faithful Mobile GUI Agents with Guided Advantage Estimator. In Proceedings of the 43rd International Conference on Machine Learning, pages 44838–44865. PMLR, 2026.arxiv.org/abs/2605.01208
- [10]Kun Huang, Weikai Xu, Yuxuan Liu, Quandong Wang, Pengzhi Gao, Wei Liu, Jian Luan, Bin Wang, and Bo An. MobileIPL: Enhancing Mobile Agents Thinking Process via Iterative Preference Learning. In International Conference on Learning Representations, 2026.arxiv.org/abs/2505.12299
- [11]Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs. arXiv preprint arXiv:2406.18629, 2024.arxiv.org/abs/2406.18629
- [12]Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the Effects of Data Scale on UI Control Agents. In Advances in Neural Information Processing Systems, pages 92130–92154, 2024.arxiv.org/abs/2406.03679
- [13]Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi. UI-TARS: Pioneering automated GUI interaction with native agents. arXiv preprint arXiv:2501.12326, 2025.arxiv.org/abs/2501.12326
- [14]Qwen Team. Qwen3.5-9B model card. Hugging Face, https://huggingface.co/Qwen/Qwen3.5-9B, 2026.
- [15]Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In Advances in Neural Information Processing Systems, pages 53728–53741, 2023.arxiv.org/abs/2305.18290
- [16]Chris Rawles, Sarah Clinckemaille, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. In International Conference on Learning Representations, 2025.arxiv.org/abs/2405.14573
- [17]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
- [18]Venus Team, Zhaohan Cai, Haoxing Chen, Jiaxuan Chen, Weizhi Chen, Changlong Gao, Zhangxuan Gu, Yuan Guo, Jiayi Wu, Ruobing Xie, Zqian Huang, Lei Jiang, Can Xu, Kangyang Luo, Bochen Lin, Ming Gao, and Xiang Li. Negative Advantage Is a Double-Edged Sword: Calibrating Advantages in GRPO for Search Agents. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 39986–40002. Association for Computational Linguistics, 2026.
- [20]Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Yu Liang, and Yu Qiao. OS-ATLAS: Foundation Action Model for Generalist GUI Agents. In International Conference on Learning Representations, 2025.arxiv.org/abs/2410.23218
- [21]Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. In Advances in Neural Information Processing Systems, pages 52040–52094, 2024.arxiv.org/abs/2404.07972
- [22]Haolong Yan, Jia Wang, Xin Huang, Yeqing Shen, Ziyang Meng, Zhimin Fan, Kaijun Tan, Jin Gao, Lieyu Shi, Mi Yang, Shiliang Yang, Zhirui Wang, Brian Li, Kang An, Chenyang Li, Lei Lei, Mengmeng Duan, Danxun Liang, Guodong Liu, Hang Cheng, Hao Wu, Jie Dong, Junhao Huang, Mei Chen, Renjie Yu, Shunshan Li, Xu Zhou, Yiting Dai, Yineng Deng, Yingdan Liang, Zelin Chen, Wen Sun, Chengxu Yan, Chunqin Xu, Dong Li, Fengqiong Xiao, Guanghao Fan, Guopeng Li, Guozhen Peng, Hongbing Li, Hang Li, Hongming Chen, Jingjing Xie, Jianyong Li, Jingyang Zhang, Jiaj u Ren, Jiayu Yuan, Jianpeng Yin, Kai Cao, Liang Zhao, Liguo Tan, Liying Shi, Mengqiang Ren, Min Xu, Manjiao Liu, Mao Luo, Mingxin Wan, Na Wang, Nan Wu, Ning Wang, Peiyao Ma, Qingzhou Zhang, Qiao Wang, Qinlin Zeng, Qiong Gao, Qiongyao Li, Shangwu Zhong, Shuli Gao, Shaofan Liu, Shisi Gao, Shuang Luo, Xingbin Liu, Xiaojia Liu, Xiaojie Hou, Xin Liu, Xuanti Feng, Xuedan Cai, Xuan Wen, Xianwei Zhu, Xin Liang, Xin Liu, Xin Zhou, Yifan Sui, Yingxiu Zhao, Yukang Shi, Yunfan Xu, Yuqing Zeng, Yixun Zhang, Zejia Weng, Zhonghao Yan, Zhiguo Huang, Zhuoyu Wang, Zihan Yan, Zheng Ge, Jing Li, Yibo Zhu, Binxing Jiao, Xiangyu Zhang, and Daxin Jiang. Step-GUI technical report. arXiv preprint arXiv:2512.15431, 2025.arxiv.org/abs/2512.15431
- [23]Jiabo Ye, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Zhaoqing Zhu, Ziwei Zheng, Feiyu Gao, Junjie Cao, Zhengxi Lu, Jitong Liao, Qi Zheng, Fei Huang, Jingren Zhou, and Ming Yan. Mobile-Agent-v3: Fundamental agents for GUI automation. arXiv preprint arXiv:2508.15144, 2025.arxiv.org/abs/2508.15144
- [24]Qize Yu, Lianrui Fan, Boyu Chen, Jiaqi Liang, Xini Ding, Yue Chen, Zetian Song, Yuran Wang, Yi Zou, Kaixuan Wang, Tianxing Chen, Wenxuan Song, Bohan Zhou, Mingleyang Li, Siqiao Huang, Yuqi Ye, Caigao Jiang, Wei Wei, Ruihai Wu, Hang Zhang, Xiaoao Ge, Shuchang Zhou, Shilong Liu, Xianming Liu, Ping Luo, and Shiyu Huang. GroundingPI: A grounding foundation model towards physical intelligence with visual primitives. arXiv preprint arXiv:2609.39601, 2026.arxiv.org/abs/2609.39601
- [25]Hanzhang Zhou, Xu Zhang, Panrong Tong, Jianan Zhang, Liangyu Chen, Quy u Kong, Chenglin Cai, Chen Liu, Yue Wang, Jingren Zhou, and Steven Hoi. MAI-GUI technical report: Real-world centric foundation GUI agents. arXiv preprint arXiv:2512.22047, 2025.arxiv.org/abs/2512.22047
- [26]Shijie Zhou, Viet Dac Lai, Hao Tan, Jihyung Kil, Wanrong Zhu, Changyou Chen, and Ruiyi Zhang. GUI-AIMA: Aligning intrinsic multimodal attention with a context anchor for GUI grounding. In Computer Vision – ECCV 2026, pages 79–97. Springer, 2026.arxiv.org/abs/2511.00810








