betaFeedback

WorksComputer Science

Learning to Look Beyond Screens: Transferable Visual Observation from GUI Grounding

Abstract

Visual perception is evolving from passive recognition toward active, task-oriented observation. We investigate GUI grounding as a scalable source of spatial supervision for learning observation behaviors that transfer beyond interfaces. We introduce LBS (Look Beyond Screens), a vision-language model trained through supervised initialization on teacher-generated GUI grounding trajectories, followed by reinforcement learning for multi-turn crop-and-inspect decisions. Under fixed inference settings, LBS models based on Qwen3.5-35B-A3B and Qwen3.8-27B consistently improve over their corresponding baselines. Gains beyond supervised fine-tuning transfer to PerceptionBench for both backbones and to HR-Bench and ChartQAPro for Qwen3.5-35B-A3B, supporting observation strategies that generalize beyond GUI-specific imitation. Further analysis suggests complementary roles for the two stages: SFT establishes visual inspection primitives, while RL improves their downstream effectiveness. These findings support GUI grounding as a practical supervision paradigm for transferable active visual perception and broader multimodal reasoning. Our code and model are publicly available at https://github.com/inclusionAI/UI-Venus/tree/LBS and https://huggingface.co/inclusionAI/LBS-27B, respectively.

Introduction

Code: https://github.com/inclusionAI/UI-Venus/tree/LBS

Model: https://huggingface.co/inclusionAI/LBS-27B

Active visual perception requires a vision-language model (VLM) to acquire evidence as well as interpret it. On a high-resolution image, the model must locate relevant regions, select an appropriate view, and relate local detail to the original scene [10, 2]. Crop-and-inspect and think-with-image methods demonstrate the value of additional visual observations [47, 36, 32] (Qi et al. 2025). A central training challenge is to connect these intermediate observation decisions to task success, particularly when a correct final answer can follow either informative or redundant inspection.

GUI grounding provides a concrete setting for learning this connection. Each example pairs an instruction with a spatial target, making both the final localization and intermediate crop geometry measurable ([3, 39, 13]). These annotations can therefore supervise intermediate observation decisions without specifying an ideal crop sequence for reinforcement learning. We hypothesize that such supervision can help elicit latent visual knowledge in pretrained VLMs by training them to put this knowledge into practice through sequential observation: deciding where to inspect, how much context to retain, and when further evidence is useful. Although the source images are interfaces, these decisions are also relevant to natural images and charts. This motivates our central question: can spatial supervision from GUI grounding elicit observation behavior that transfers to non-GUI perception? Such transfer would suggest a practical way to make pretrained visual knowledge more accessible through interaction, potentially reducing the need to construct specialized observation trajectories for each downstream domain.

We address this question with a two-stage framework. Supervised fine-tuning (SFT) on teacher-generated grounding trajectories initializes the crop-and-inspect protocol. Multi-turn reinforcement learning (RL) then refines the policy using grounding correctness and training-only target annotations ([7]). Outcome-gated spatial rewards connect intermediate inspections to task success, a small-target bonus encourages target coverage, and selective response masking controls which intermediate responses receive policy updates. In this way, spatial annotations provide supervision for the process of acquiring visual evidence as well as for the final grounding result. The same crop interface is used at inference across domains, with task-specific final answers. This design allows us to examine whether observation behavior learned under GUI supervision benefits tasks with different visual content and objectives.

Our results support transfer beyond supervised initialization. RL improves ScreenSpot-Pro ([13]) accuracy over SFT by 1.83 and 6.13 percentage points for Qwen3.5-35B-A3B and Qwen3.8-27B, and improves PerceptionBench ([14]) by approximately 3.0 and 7.10 points. The Qwen3.5-based policy also improves on HR-Bench 4K/8K ([34]) and ChartQAPro ([18]). The reward comparison reveals a useful distinction: outcome-only RL improves source-domain grounding but lowers PerceptionBench accuracy, whereas the spatially supervised configuration improves both. These comparisons show that higher GUI grounding accuracy does not necessarily imply stronger non-GUI transfer; the results highlight the value of supervising intermediate observation decisions. Comparisons of reconstructed interaction histories further examine how crop images and their presentation scale affect answer generation. Together, these findings support GUI grounding as a practical source of supervision for visual observation whose benefits extend beyond the source domain, consistent with the hypothesis that spatially grounded interaction training can help elicit and apply pretrained visual knowledge across tasks.

Our contributions are threefold:

  • We establish empirical evidence that GUI grounding can supervise visual observation policies that transfer beyond interfaces, with gains over SFT across two backbones on PerceptionBench and across additional high-resolution and chart benchmarks for Qwen3.5-35B-A3B.

  • We develop a two-stage framework that converts target annotations into outcome-gated spatial rewards and selective response updates for multi-turn crop-and-inspect training.

  • We show that improved grounding under outcome-only RL need not translate into improved non-GUI perception, and complement the training comparison with controlled edits to historical crop observations.

Related Work

Related Work

Agentic visual reasoning. Agentic visual reasoning treats observation as sequential evidence acquisition. ReAct interleaves reasoning with environment interaction [41], and Toolformer learns API use from self-supervised data [24]. Visual agents similarly use image manipulation and code-based tools during reasoning [29, 1, 6]. CogCoM integrates image manipulations into reasoning (Qi et al., 2025); V∗V^{*} uses guided visual search to select informative regions at inference time [36]. DeepEyes and DeepEyesV2 study reinforcement learning for visual and multimodal tool use [47, 8]. These methods establish the utility of additional observations and tool interaction. Our focus is the supervision that teaches inspection behavior and whether it transfers from GUI grounding to non-GUI perception.

Visually grounded reasoning. Visually grounded reasoning connects linguistic predictions to image regions [9, 20, 49]. Recent methods can be organized along two axes: by paradigm, into training-based approaches [23, 47, 33] and test-time approaches [10, 27, 44]; and by motivation, into suppressing irrelevant background [35, 15] and enhancing task-relevant foreground through zoom-in operations [5, 17, 42, 43, 47]. ViGoRL anchors reasoning to visual coordinates, while TreeVGR jointly supervises localization and reasoning [23, 32]. GUI grounding methods instead emphasize instruction-conditioned target localization and downstream action execution [3, 39, 31]. Our study connects these directions by using GUI target annotations to supervise intermediate observation actions and evaluating the resulting policy on non-GUI perception. The central distinction is the source of supervision and its transfer: one spatially annotated interaction domain supplies feedback for observation policies evaluated across different visual content and answer formats.

Method

Our framework turns GUI target annotations into supervision for sequential visual observation (Figure 1). SFT first teaches the interaction protocol; RL then scores crop geometry and filters intermediate response updates using grounding outcomes. The policy receives images and instructions, while target annotations remain training-only information. At inference, the crop interface is shared across domains and the final answer format is adapted to each task.

Figure 1. Learning transferable observation from GUI grounding. (a) Grounding success alone cannot distinguish target-relevant from redundant inspection. (b) Following SFT, spatially supervised RL uses training-only target annotations to score crops and mask irrelevant intermediate responses. (c) The observation policy applies the same crop-and-inspect protocol to non-GUI images. (d) The Qwen3.5-based model improves over SFT by 2.97 and 1.83 percentage points on PerceptionBench and ScreenSpot-Pro, respectively.

Grounding Supervision and Policy Initialization

Synthetic grounding tasks. We construct interfaces through code-guided generation, following prior work on synthetic multimodal and GUI data [40, 31]. An LLM produces executable HTML/CSS/JavaScript from a scenario, optionally conditioned on a persona or reference screenshot. Rendering associates each screenshot with DOM structure, element states, and geometry. Candidate targets are extracted from element semantics and accessibility attributes, filtered for visibility, clipping, occlusion, and correspondence to rendered content, and refined using text-node geometry where appropriate. Accepted targets are paired with natural-language interaction instructions; infeasible instructions provide negative examples. Both SFT and RL use only GUI grounding instructions, with no general-domain VQA training tasks. Reference screenshots used for synthetic image generation exclude images from the evaluation benchmarks.

A grounding task is d=(I,q,B)d = (I, q, B), where II is the screenshot, qq is the instruction, and B=(bx−,by−,bx+,by+)B = (b_x^{-}, b_y^{-}, b_x^{+}, b_y^{+}) is the target box in the original image’s [0,1000]2[0, 1000]^2 coordinate system. Infeasible tasks use B=(−1,−1,−1,−1)B = (-1, -1, -1, -1). The policy observes only (I,q)(I, q); BB is reserved for trajectory filtering and training feedback.

Teacher trajectories. We use Qwen3.8-Max (Qwen Team, [22]) to generate multi-turn trajectories at temperature 1.0, with a maximum generation length of 32,768 tokens. Each trajectory τ=(a1,o1,…,aT−1,oT−1,aT)\tau= (a_1, o_1, \ldots, a_{T-1}, o_{T-1}, a_T) contains assistant responses ata_t and tool observations oto_t. The grounding response predicts a point p^=(x^,y^)\hat{p} = (\hat{x}, \hat{y}). For a parseable finite prediction, correctness is

C(d,τ)=1{bx−≤x^≤bx+, by−≤y^≤by+}.(1)C(d, \tau) = 1\{b_x^{-} \le\hat{x} \le b_x^{+},\ b_y^{-} \le\hat{y} \le b_y^{+}\}. \tag*{(1)}

Invalid predictions receive zero. The same rule accepts only (−1,−1)(-1, -1) for an infeasible task. Although the prompt requests an element center, acceptance requires point-in-box agreement, including the boundary. Rejection sampling retains successful trajectories:

DCS={(d,τ):τ∼πteach(⋅∣I,q), C(d,τ)=1}.(2)\mathcal{D}_{\mathrm{CS}} = \{(d, \tau) : \tau\sim\pi_{\mathrm{teach}}(\cdot\mid I, q),\ C(d, \tau) = 1\}. \tag*{(2)}

Rejection sampling ensures grounding correctness but leaves intermediate inspection quality unconstrained, motivating the spatial feedback used during RL.

Supervised initialization. We train the student with Swift [46], supervising assistant outputs while conditioning on preceding images, responses, and tool feedback. With Y(τ)\mathcal{Y}(\tau) denoting assistant-token positions, the objective is

LCS(θ)∝−∑(d,τ)∈DCS∑k∈Y(τ)log⁡πθ(yk∣I,q,τ<k).(3)\mathcal{L}_{\mathrm{CS}}(\theta) \propto-\sum_{(d,\tau)\in\mathcal{D}_{\mathrm{CS}}}\sum_{k\in\mathcal{Y}(\tau)} \log\pi_{\theta}(y_k \mid I,q,\tau_{<k}). \tag*{(3)}

The resulting checkpoint initializes RL and provides an inspection-capable baseline, allowing us to evaluate improvement beyond imitation of the tool-use protocol.

Multi-Turn Visual Observation

At turn tt, the policy samples at∼πθ(⋅∣ht)a_t \sim\pi_{\theta}(\cdot\mid h_t), where hth_t contains the task, original image, and interaction history. Each response contains reasoning followed by either a final answer or one or two textual crop_image actions. A crop action specifies an available image and a rectangle in its local pixel coordinates; previously returned crops may be inspected recursively. Final grounding predictions always use original-image normalized coordinates.

The tool returns the crop image, its dimensions, and a coordinate mapping to the original image. Results from the same turn are added to the history in action order; failed executions return textual feedback. Earlier images and messages remain available, so each crop adds a local view while preserving access to the original context. Interaction ends with a final answer or a configured stopping limit. Every admitted crop attempt consumes a tool-call slot, including failed executions. Appendix § 9 details crop construction and coordinate tracking.

Outcome-Gated Spatial Rewards

Grounding correctness assigns the same outcome to successful trajectories with different inspection quality. We use target annotations in two complementary ways: a spatial reward evaluates the crop sequence, and the response mask in § 4.4 controls which intermediate outputs receive direct updates. Let R1,…,RKR_1,\ldots,R_K be the successful crop regions in execution order, mapped to original-image coordinates, and let uj=IoU⁡(Rj,B)u_j = \operatorname{IoU}(R_j,B).

Spatial process reward. We reward the best target overlap and penalize repeated crops, declining overlap, and tool use:

Prep=min⁡(0.4, 0.2∑j=2K1{IoU⁡(Rj−1,Rj)>0.9}),(4)P_{\mathrm{rep}} = \min\left(0.4,\ 0.2\sum_{j=2}^{K}\mathbf{1}\{\operatorname{IoU}(R_{j-1},R_j)>0.9\}\right), \tag*{(4)}
Preg=0.21{K≥2 and uK<u1},(5)P_{\mathrm{reg}} = 0.2\mathbf{1}\{K\ge2\ \text{and}\ u_K<u_1\}, \tag*{(5)}
Rproc=C(d,τ)clip⁡[0,1](max⁡juj−Prep−Preg−0.05K).(6)R_{\mathrm{proc}} = C(d,\tau)\operatorname{clip}_{[0,1]}\left(\max_j u_j-P_{\mathrm{rep}}-P_{\mathrm{reg}}-0.05K\right). \tag*{(6)}

The reward is zero for infeasible tasks or trajectories without successful crops. Repetition compares consecutive crops, whereas regression compares the final and initial target overlaps. The outcome gate restricts spatial rewards to successful trajectories, and clipping bounds this component between zero and one. Crop overlap supplies a proxy for spatial relevance; it does not directly evaluate interpretation of the returned image.

Small-target inspection bonus. IoU can be low when a useful contextual crop contains a small target. We therefore define a turn-level relevance score that combines overlap and target-center containment:

Qt=max⁡j:Rj produced at t[0.6IoU⁡(Rj,B)+0.41{cB∈Rj}],(7)Q_t=\max_{j:R_j\ \text{produced at }t}\left[0.6\operatorname{IoU}(R_j,B)+0.4\mathbf{1}\{c_B\in R_j\}\right], \tag*{(7)}

where cBc_B is the target center. A crop containing this center receives at least 0.4, even when its overlap is small. For targets occupying less than 5% of the image, we add

Rzoom=0.2C(d,τ)1{∣B∣/106<0.05}1{max⁡tQt≥0.2}.(8)R_{\mathrm{zoom}}=0.2C(d,\tau)\mathbf{1}\{|B|/10^6<0.05\}\mathbf{1}\{\max_t Q_t\ge0.2\}. \tag*{(8)}

This bonus is zero for infeasible tasks and trajectories without successful crops. It is a target-coverage incentive: a broad crop containing the target center can qualify, while crop selectivity is encouraged separately by the overlap-based process reward. The bonus depends on geometry rather than the image-resizing factor.

Format and textual rewards. The format reward RfmtR_{\mathrm{fmt}} scores the final response: canonical parsing receives 1, compatibility parsing receives 0.5, and parsing failure receives 0. We additionally examine a text-based critic that evaluates task focus, feedback use, reasoning–action consistency, incremental progress, and conciseness. Given the task and a textual trajectory, the critic returns scot∈[0,1]s_{\mathrm{cot}}\in[0,1], yielding

Rcot=C(d,τ)scot.(9)R_{\mathrm{cot}}=C(d,\tau)s_{\mathrm{cot}}. \tag*{(9)}

Trajectories without visible reasoning receive zero. The non-oracle critic receives neither images nor target annotations, so its score measures textual consistency rather than visual correctness. The general reward is

R(d,τ)=C(d,τ)+0.3Rproc+0.1Rfmt+Rzoom+0.3Rcot.(10)R(d,\tau)=C(d,\tau)+0.3R_{\mathrm{proc}}+0.1R_{\mathrm{fmt}}+R_{\mathrm{zoom}}+0.3R_{\mathrm{cot}}. \tag*{(10)}

All components contribute to one trajectory-level return.

Selective Response Updates and Optimization

Filtering intermediate responses. A shared trajectory reward can reinforce irrelevant actions in an otherwise successful rollout. We use the turn-level relevance score to construct a response mask:

mt={0,t<T, C(d,τ)=1, Qt exists, Qt<0.05,1,otherwise.(11)m_t= \begin{cases} 0, & t<T,\ C(d,\tau)=1,\ Q_t\ \text{exists},\ Q_t<0.05,\\ 1, & \text{otherwise}. \end{cases} \tag*{(11)}

Equation (11) defines the spatial mask. A second filter removes spatially eligible intermediate responses whose mean generated-token surprisal is below the median over eligible turns, using scaling factor η=1\eta=1. Let et∈{0,1}e_t\in\{0,1\} denote this second filter and m~t=mtet\widetilde{m}_t=m_te_t the resulting response mask, with m~T=1\widetilde{m}_T=1. Both filters act on direct response loss; all previous responses and observations remain in the conditioning history. Filtering is independent of the advantage sign; it selects direct response updates without assigning a separate causal value to each crop. Appendix B specifies the filtering procedure.

Group-relative optimization. We adapt GRPO ([26]) to multi-turn trajectories. For each task, G=8G=8 rollouts receive the reward in Equation (10). Each advantage subtracts the group mean without variance normalization and is shared across retained assistant tokens. Training uses a PPO-style clipped surrogate ([25]) with a fixed-reference KL penalty, averaging first over retained tokens and then over trajectories.

Experiments

Experimental Setup

Training configuration. We study Qwen3.5-35B-A3B (Team, 2026) and Qwen3.8-27B (Qwen Team, 2026). For both models, we freeze the vision encoder and vision–language projector throughout SFT and RL, updating only the language-model parameters. Qwen3.5-35B-A3B initially exhibits limited adherence to our prompting format and agentic observation protocol. To establish reliable protocol following, we use approximately 35,000 SFT trajectories for Qwen3.5-35B-A3B, and 10,000 for Qwen3.8-27B. Each SFT checkpoint then initializes the corresponding RL policy. The main configuration combines grounding correctness, gated spatial feedback, format compliance, small-target inspection bonus and the auxiliary critic. Optimization settings and prompts appear in Appendices B and F. We also keep the training recipe simple and do not extensively tune the remaining optimization hyperparameters, focusing our study on the proposed supervision design.

Interaction protocol. The crop tool uses a scale factor of α=4.0\alpha= 4.0 and a pixel cap of Pmax⁡=4×106P_{\max} = 4 \times10^{6} Equation (18). Original images are preprocessed with a pixel count between 65,536 and 16,777,216. We generate rollouts using vLLM [12]. Training allows up to three crop calls per example. Qwen3.8-27B evaluation also uses at most three crop calls, with reasoning_effort=xhigh and thinking_budget=4096 (4,096 reasoning tokens per assistant turn), passed through chat_template_kwargs. Within each benchmark, SFT and RL models use identical prompts, decoding settings, and inference limits.

Benchmarks and metrics. ScreenSpot-Pro [13] evaluates instruction-conditioned GUI localization using point-in-box accuracy. PerceptionBench [14] evaluates non-GUI perception across ten capabilities using reference-based answer judgment, with a locally deployed DeepSeek-V4-Flash [38] serving as the judge model. The Qwen3.8-27B evaluation uses Deepseekv4-flash-0731, with no failed or skipped judgments across all 3,000 records. We interpret stage-wise gains within each backbone rather than treating cross-backbone scores as a controlled ranking. We report overall accuracy and category-level scores. Additional evaluations cover OSWorld-G-R (Refine) [37], HR-Bench [34] at 4K and 8K resolutions, and ChartQAPro [18]. Further details on selected additional benchmarks are provided in Appendix A.

GUI Grounding Across Backbones

We compared our method with several mainstream GUI grounding methods [4, 45, 11, 19, 48], and the results are presented in Table 1. Table 1 shows that RL improves ScreenSpot-Pro accuracy over SFT for both backbones. Qwen3.5-35B-A3B increases from 78.56% to 80.39%, and Qwen3.8-27B increases from 75.02% to 81.15%. Since both SFT baselines already follow the crop-and-inspect protocol, these gains support the value of optimizing observation trajectories beyond teacher imitation. Qwen3.8-27B gains 11.89 points over base and improves all six application groups over SFT, led by CAD and engineering (+7.28), creative applications (+6.45), and scientific applications (+6.30). Qwen3.5-35B-A3B improves in five groups, led by development (+6.35), but declines in office applications (90.00% to 87.83%). These results are consistent with the utility of zoom-in inspection for high-resolution GUI grounding, where enlarged local views can make small interface elements and fine-grained visual details easier to distinguish and localize.

ModelZoomDev.Creat.CADSci.OfficeOSOverall
Holo2-235B-A22B (Agentic)✗74.672.180.579.985.283.278.5
MAI-UI-32B (MVP)✓79.369.576.676.091.375.577.5
AdaZoom (Qwen3.5-397B refine)✓75.372.472.474.093.576.576.8
KV-Ground-GuiOwl-1.5 (Zoom In)✓73.672.474.777.290.073.076.4
MAI-UI-32B (Zoom In)✓71.670.472.875.686.165.373.5
Base (Qwen3.5-35B-A3B)✓––––––1.271
Ours (+SFT, 35k trajectories)✓77.2670.6778.9381.1090.0077.0478.56
Ours (+Agentic RL)✓83.6171.5580.0883.0787.8379.0880.39
Base (Qwen3.8-27B)✓69.2361.8867.0567.3283.4870.9269.26
Ours (+SFT, 10k trajectories)✓76.9268.3370.1174.4186.9677.0475.02
Ours (+Agentic RL)✓82.6174.7877.3980.7193.0481.6381.15

Table 1. ScreenSpot-Pro accuracy (%) by application group.

Zoom indicates zoom-in use. Bold denotes the highest reported accuracy within each backbone and column; dashes indicate unreported scores. 1 The base Qwen3.5-35B-A3B model does not reliably follow our prompting format and interaction protocol.

Transfer to General Visual Perception

Gain is the accuracy difference between +RL and +SFT in percentage points.

Table 2 shows improvements beyond SFT for both backbones on PerceptionBench. Qwen3.5-35B-A3B increases from 38.6% at base to 42.1% after SFT and 45.1% after RL; Qwen3.8-27B follows the same progression, from 46.6% to 52.57% and 59.67%. The RL gains of approximately 3.0 and 7.10 points show that non-GUI improvement is not confined to one backbone. Both final policies exceed their respective base models across all ten capabilities.

CapabilityQwen3.5-35B-A3B Base+SFT+RLGainQwen3.8-27B Base+SFT+RLGain
Visual relation40.039.745.2+5.553.959.7068.48+8.78
Counting41.342.548.8+6.348.553.9461.82+7.88
Attribute40.347.348.5+1.250.655.4565.76+10.31
Depth / 3D perception33.137.637.60.039.750.9150.30-0.61
Localization37.051.350.0-1.348.565.4567.58+2.13
Comparison39.835.543.1+7.649.551.6163.44+11.83
Fine-grained recognition41.441.847.0+5.243.452.0757.59+5.52
Contextual integration34.238.538.1-0.438.844.3152.55+8.24
OCR53.058.560.4+1.950.653.7361.57+7.84
Hallucination26.228.131.8+3.741.033.2143.91+10.70
Overall38.642.145.1+3.046.652.5759.67+7.10

Table 2. PerceptionBench accuracy (%) across training stages for two backbones.

The capability profiles reveal shared benefits and model-dependent tradeoffs. Relative to SFT, both policies improve counting, comparison, fine-grained recognition, OCR, and hallucination assessment. For Qwen3.5-35B-A3B, the largest gains are in comparison (+7.6), counting (+6.3), and visual relation (+5.5); RL also reverses the SFT declines in comparison and visual relation. Qwen3.8-27B gains most in comparison (+11.83), hallucination assessment (+10.70), and attribute recognition (+10.31). Qwen3.5-35B-A3B improves seven of ten capabilities, while Qwen3.8-27B improves nine, with a slight decline in depth/3D perception (50.91% to 50.30%, −0.61 points). The shared gains extend beyond grounding to tasks requiring discrimination and comparison of visual evidence. These results support transfer of the trained observation policy while showing that its benefits depend on the perceptual capability. Appendix § 11 provides external reference scores.

Transfer Across Visual Domains

Figure 2(a) extends the stage-wise evaluation of Qwen3.5-35B-A3B to another GUI setting, high-resolution perception, and chart understanding. Scores increase from Base to SFT to RL on all four benchmarks. On OSWorld-G-R, SFT raises accuracy from 63.6% to 66.9%, and RL further improves it to 72.7%, yielding a 5.8-point gain over SFT. Figure 2(b) compares the incremental gains from the two training stages. RL yields larger gains than SFT on every benchmark. On the non-GUI tasks, RL improves over SFT by 2.2, 3.1, and 3.2 percentage points on HR-Bench 4K, HR-Bench 8K, and ChartQAPro, respectively, exceeding the corresponding SFT gains over Base of 1.1, 1.7, and 0.1 points. ChartQAPro illustrates this difference: SFT changes the score only from 63.4% to 63.5%, whereas RL raises it to 66.7%. Together with PerceptionBench, these results show that improvements beyond supervised initialization extend across general perception, high-resolution imagery, and charts under the reported evaluation settings.

Figure 2. Stage-wise results for Qwen3.5-35B-A3B.

Reward Design Ablation

Outcome-only RL reveals a distinction between source-domain improvement and cross-domain transfer. It reaches 79.19% on ScreenSpot-Pro and 40.87% on PerceptionBench, compared with 78.56% and 42.10% for SFT. Thus, a 0.63-point grounding gain accompanies a 1.23-point transfer decline. Our full configuration, combining the gated spatial process reward, small-target inspection bonus, and text-based critic, improves both benchmarks over SFT and exceeds outcome-only RL by 1.20 and 4.20 points, respectively. The configuration with the highest ScreenSpot-Pro accuracy is not the one with the strongest transfer. These results Outcome-only RL uses only correctness and format rewards. Process: spatial process reward; Bonus: small-target inspection bonus; Critic: text-based CoT critic.

indicate that better source-task performance need not translate into better non-GUI perception, motivating supervision of intermediate observation behavior beyond grounding correctness.

The component comparisons highlight the role of the combined reward design (Table 3). Our full configuration achieves the highest PerceptionBench accuracy (45.07%), whereas ungated process reward with the critic achieves the highest ScreenSpot-Pro accuracy (81.09%). Removing the critic from the full configuration lowers ScreenSpot-Pro/PerceptionBench accuracy by 0.69/1.81 points; removing the bonus lowers accuracy by 0.57/2.50 points. These comparisons support retaining both components, with larger observed gains on the transfer benchmark. The effect of outcome gating depends on the accompanying components: gating improves both benchmark scores when paired with the bonus, but lowers both when paired with the critic alone. Taken together, the results favor the full configuration for cross-domain transfer, while showing that gating alone does not consistently improve performance across the evaluated configurations.

ConfigurationProcessBonusCriticSSProPB
Outcome-only RL✗✗✗79.1940.87
Ungated process + criticUngated✗✓81.0942.83
Ungated process + bonusUngated✓✗79.5742.10
Gated processGated✗✗79.2541.63
Gated process + bonusGated✓✗79.7043.26
Gated process + criticGated✗✓79.8242.57
Gated process + bonus + critic (ours)Gated✓✓80.3945.07

Table 3. Source-domain and transfer accuracy (%) for Qwen3.5-35B-A3B reward configurations.

Effects of Visual Context on Grounding and Reasoning

We examine how visual context affects Qwen3.8-27B answer generation (Table 4). From a reference interaction, we reconstruct histories containing upsampled crops, crops without tool-side upsampling, or crop metadata only, while retaining the original image. These interventions modify supplied observations without rerunning crop selection (Appendix § 10); direct reasoning provides a separate baseline.

ConditionScreenSpot-ProPerceptionBench
Direct reasoning (base)63.446.6
Crop images without upsampling68.852.2
Crop metadata only70.649.9
Crop images with upsampling81.1559.67

Table 4. Qwen3.8-27B answer accuracy (%) under direct reasoning and reconstructed observation histories.

The relative value of spatial metadata and crop content differs across tasks. Metadata-only contexts outperform upsampled crops on ScreenSpot-Pro (70.6% vs. 68.8%), whereas crops perform better on PerceptionBench (52.2% vs. 49.9%). This pattern suggests that coordinate-level visual primitives support spatial prediction in grounding, while localized crop views may better support visual reasoning by reducing irrelevant background. Upsampled crops achieve the highest accuracy on both benchmarks (81.15%/59.67%), highlighting the importance of visual detail and presentation scale. These results characterize the use of supplied context, rather than isolating observation behavior acquired during RL.

Observation-feedback conditions on a text-recognition example

Figure 3. Observation-feedback conditions on a text-recognition example: original image only, native-resolution crops, crop metadata only, and crops with 4×4 \times upsampling. All agentic contexts retain the original image and interaction history.

Conclusion

GUI grounding provides a practical source of transferable supervision for active visual perception. Building on recent, strong open-weight VLMs and newly established baselines, we demonstrate improvements in both GUI grounding and non-GUI perception. Our results suggest that supervising intermediate observation decisions helps models inspect question-relevant regions, while crop upsampling makes fine-grained details more accessible for visual understanding. We hope this approach can be a step toward models that actively seek relevant evidence and develop a deeper understanding of the visual world.

Discussion

Our results suggest that agentic visual perception is becoming increasingly accessible to modern VLMs. With suitable supervision and a simple crop-and-inspect interface, these models can improve image-level perception without lots of additional general-domain VQA training. This progress motivates extending observation policies from individual images to video. Recent work such as Video-Index [28] highlights both the potential of agent tools and the remaining challenges in video understanding, while emphasizing evaluation that genuinely requires temporal evidence. A promising near-term direction is therefore to learn when and where to inspect across both space and time, including selecting informative frames and maintaining useful visual memory. Extending these capabilities to long videos and streaming inputs will further require balancing evidence acquisition with context, latency, and computation budgets.

Additional Benchmark Protocols

OSWorld-G [37] contains 564 finely annotated GUI grounding samples drawn from OSWorld rollouts, covering text matching (268), element recognition (337), layout understanding (252), fine-grained manipulation (154), and refusal of infeasible instructions (54). Unlike ScreenSpot-Pro, which uses extreme resolutions, OSWorld-G uses standard 720p/1080p screenshots and emphasizes software commonsense and layout reasoning.

HR-Bench [34] evaluates perception on high-resolution images at 4K and 8K resolutions, with 100 fine-grained single-instance perception (FSP) and 100 fine-grained cross-instance perception (FCP) questions per resolution. The benchmark uses a cyclic-permutation protocol that averages accuracy over cyclic rotations of the answer choices.

ChartQAPro [18] extends ChartQA with 1,341 charts from 157 diverse sources and 1,948 questions spanning factoid, multiple-choice, conversational, fact-checking, and hypothetical questions, including unanswerable cases. The benchmark uses an answer-type-specific score: numeric answers allow a 5% relative error, years and categorical answers require exact matching, and textual answers use average normalized Levenshtein similarity (ANLS), following § 5.3 and Appendix § 8.§ 6 of the benchmark paper.

Implementation and Evaluation Details

Training Configuration

Both Qwen3.5-35B-A3B and Qwen3.8-27B use the same RL optimization and rollout configuration. GRPO samples G=8G = 8 trajectories with temperature 0.8 and top-pp 0.95. We use AdamW [16] with learning rate 10−610^{-6}, a cosine schedule without warmup, weight decay 0.1, (β1,β2)=(0.9,0.95)(\beta_1, \beta_2) = (0.9, 0.95), and gradient clipping at norm 1.0. The global batch contains 512 trajectories (64 tasks with 8 rollouts each) across 64 NPUs, with a per-device micro-batch size of 1 and 8 gradient-accumulation steps. Training uses bfloat16 precision and DeepSpeed ZeRO-3 with CPU optimizer offloading, as in the released training launcher. Token-level importance ratios use asymmetric clipping bounds ϵ−=0.2\epsilon_{-} = 0.2 and ϵ+=0.28\epsilon_{+} = 0.28. The training launcher configures one epoch, with maximum sequence length 32,768, up to 12,288 completion tokens per trajectory, and 2,560 per assistant turn.

Selective Masking Details

After spatial filtering, a second filter masks intermediate turns with low mean generated-token surprisal. For each intermediate turn, this proxy is the negative mean log-probability of its generated tokens, nt=−1∣Yt∣∑k∈Ytlog⁡πold(yik∣hik)n_t = -\frac{1}{\lvert\mathcal{Y}_t\rvert}\sum_{k\in\mathcal{Y}_t}\log\pi_{\mathrm{old}}(y_{ik}\mid h_{ik}), where Yt\mathcal{Y}_t indexes the tokens of turn tt. Among the turns that survive the spatial filter, we mask those whose uncertainty falls below the median uncertainty across filter-eligible turns (equivalently, nt<median⁡snsn_t < \operatorname{median}_s n_s with an optional scaling factor η\eta; the main configuration uses η=1\eta= 1). This retains higher-uncertainty decisions under the token-probability proxy; uncertainty does not itself establish that a decision is useful or incorrect. The two filters are applied sequentially over intermediate turns only; the final assistant turn is always retained. Each intermediate mask applies to the entire response, including reasoning and action tokens. The spatial-masking criterion depends on terminal correctness and spatial relevance, and the uncertainty filter depends on turn-level uncertainty; neither depends on the sign of the group-relative advantage, so a successful trajectory can still have a negative advantage relative to its peers. Masked responses remain in the conditioning history but contribute no direct response loss. User messages and tool observations likewise provide context without response loss. Because QtQ_t takes a maximum, a turn containing two crops is retained if either satisfies the relevance threshold. Turns with only failed crops have no QtQ_t and are not removed by the spatial filter (the uncertainty filter still applies). Rollout log-probabilities are aligned with the retained tokens when constructing the training payload.

Group-Relative Optimization

We adapt group relative policy optimization (GRPO) [26] to multi-turn inspection, with the reward, token masks, and normalization specified below. For each task, we sample G=8G = 8 trajectories from the rollout policy and evaluate their rewards Ri=R(d,τi)R_i = R(d, \tau_i). Define

Rˉd=1G∑i=1GRi,A^i=Ri−Rˉd.(12)\bar{R}_d = \frac{1}{G}\sum_{i=1}^{G} R_i, \qquad\hat{A}_i = R_i - \bar{R}_d. \tag*{(12)}

The advantage subtracts the group-mean reward without variance normalization. We disable reward scaling in Swift by setting scale_rewards=none. The aggregate trajectory reward is formed by combining the enabled components of Equation (10) before the group mean is subtracted. This trajectory advantage is shared across the retained assistant tokens. In critic-enabled configurations, the critic enters through RiR_i, providing an additional within-group ranking signal rather than a separate supervised reasoning loss. Critic-enabled runs use the default settings of the critic integration in our Swift-based training pipeline, except for the activation and reward weights specified in Table 3 and Equation (10). Appendix § 13 describes the critic prompts and scoring procedure. Groups with identical aggregate rewards have zero policy advantages.

Let kk index assistant tokens across turns, hikh_{ik} denote the conditioning history for token yiky_{ik}, and Mik=m~t(k)M_{ik} = \tilde{m}_{t(k)} be the combined mask of its assistant turn t(k)t(k). The current-to-old policy ratio is

rik(θ)=πθ(yik∣hik)πold(yik∣hik).(13)r_{ik}(\theta) = \frac{\pi_\theta(y_{ik} \mid h_{ik})}{\pi_{\mathrm{old}}(y_{ik} \mid h_{ik})}. \tag*{(13)}

The policy term follows the clipped surrogate of PPO [25], with separately configured lower and upper bounds:

ℓikPG=−min⁡(rikA^i,clip⁡(rik,1−ϵ−,1+ϵ+)A^i),(14)\ell_{ik}^{\mathrm{PG}} = -\min\left(r_{ik}\hat{A}_i,\operatorname{clip}(r_{ik},1-\epsilon_{-},1+\epsilon_{+})\hat{A}_i\right), \tag*{(14)}

where ϵ−\epsilon_{-} and ϵ+\epsilon_{+} control the lower and upper clipping bounds. Let πref\pi_{\mathrm{ref}} be the fixed reference policy. The reference-policy KL estimator is

Δik=log⁡πref(yik∣hik)−log⁡πθ(yik∣hik),K^ik=exp⁡(Δik)−Δik−1,(15)\Delta_{ik} = \log\pi_{\mathrm{ref}}(y_{ik} \mid h_{ik}) - \log\pi_\theta(y_{ik} \mid h_{ik}), \qquad\hat{K}_{ik} = \exp(\Delta_{ik}) - \Delta_{ik} - 1, \tag*{(15)}

with the numerical safeguards of the selected trainer version. Writing Li=∑kMikL_i = \sum_k M_{ik}, the sample-normalized objective is

LRL=Ed[1G∑i=1G∑kMik(ℓikPG+βK^ik)max⁡(1,Li)].(16)\mathcal{L}_{\mathrm{RL}} = \mathbb{E}_d\left[\frac{1}{G}\sum_{i=1}^{G}\frac{\sum_k M_{ik}\left(\ell_{ik}^{\mathrm{PG}}+\beta\hat{K}_{ik}\right)}{\max(1,L_i)}\right]. \tag*{(16)}

We use Swift’s grpo loss, which first averages over retained tokens within each trajectory and then over trajectories. We use the default KL coefficient β\beta of the Swift version used for training. KL regularization enters the loss rather than the reward (kl_in_reward=false); hence it does not change the rewards used in Equation (12). The token-level ratio rikr_{ik} provides importance weighting between the current and old policies. Under the default rollout-correction configuration, no additional importance weight is applied to correct rollout-engine versus trainer probability differences. Appendix § 9.§ 1 gives the clipping parameters and reported training settings.

Crop Construction

A requested rectangle is rounded outward to pixel boundaries and clamped to its source image. The executor rejects degenerate crops and excessive aspect ratios. For a crop of size (wc,hc)(w_c,h_c), the returned size is

s=min⁡(α,Pmax⁡wchc),(17)s = \min\left(\alpha,\sqrt{\frac{P_{\max}}{w_ch_c}}\right), \tag*{(17)}
w′=max⁡(1,⌊swc⌋),h′=max⁡(1,⌊shc⌋),(18)w' = \max(1,\lfloor sw_c\rfloor), \qquad h' = \max(1,\lfloor sh_c\rfloor), \tag*{(18)}

where α\alpha is the requested scale factor and Pmax⁡P_{\max} is the pixel cap. Resizing uses Lanczos interpolation. A request on the original image reads its source pixels; a nested request reads the previously returned, resized crop. The executor maintains the mapping to the original image but does not re-extract nested crops from the original source.

Benchmark Evaluation Protocol

Stage-wise comparisons cover ScreenSpot-Pro and PerceptionBench for both backbones, and OSWorld-G-R, HR-Bench 4K/8K, and ChartQAPro for Qwen3.5-35B-A3B. Additional benchmark descriptions appear in Appendix A. We use benchmark-specific task definitions, reference answers, and scoring to assess transfer from point localization to question answering. For grounding, each example supplies a screenshot and an instruction, and the prediction is a point in the original-image coordinate system. Correctness is determined by inclusive point-in-box matching as in Equation (1), rather than crop IoU or exact center matching. Invalid coordinate predictions are incorrect. The sample-level accuracy is N−1∑i=1NC(di,τi)N^{-1}\sum_{i=1}^{N} C(d_i,\tau_i), with category reporting following the benchmark protocol. PerceptionBench (Lin et al., [14]) uses open-ended perception questions with short reference answers and reference-based judgment. We report each benchmark separately rather than combining heterogeneous scores.

Across tasks, inspection consists of alternating policy responses and crop observations. Benchmark adapters specify the task prompt and final answer interface: GUI tasks terminate with coordinates, whereas non-GUI tasks terminate with the required choice or textual answer. Ground-truth regions and training-time spatial rewards are not supplied during evaluation. Comparisons between cold-start and RL checkpoints use the same task interface and inference configuration within each benchmark. Qwen3.8-27B evaluation uses reasoning effort xhigh, a thinking budget of 4,096 tokens per assistant turn, and at most three crop calls.

Observation Feedback Protocols

We compare direct reasoning with three observation conditions constructed from an initial interaction history. All conditions receive the same question and original input image. Direct reasoning uses the base Qwen3.8-27B model to answer the question without crop-tool interaction. For the other three conditions, we first perform an initial inference run to obtain a reference interaction history. We then manually reconstruct the historical context to create condition-specific question–answer inputs and evaluate the model’s answers under these inputs.

The three reconstructed conditions provide crop images without tool-side upsampling, crop metadata without image delivery, or crop images with target scale α=4\alpha= 4. The original image remains available in all conditions; the metadata-only condition removes historical crop images without removing the original visual input.

History reconstruction. Let HH denote the reference interaction history obtained from the initial inference run. We construct three context variants:

H(c)=Tc(H),c∈{no-up,meta,up},(19)H^{(c)} = \mathcal{T}_{c}(H), \qquad c \in\{\text{no-up},\text{meta},\text{up}\}, \tag*{(19)}

where Tc\mathcal{T}_{c} modifies the historical crop observations according to condition cc. These variants share the initial history as their source. The comparison manipulates recorded observation content rather than evaluating three independently generated online trajectories. Direct reasoning is evaluated separately and is not constructed by editing HH.

Crop images and spatial metadata. For a successful crop observation at historical turn tt, let vt(α)v_{t}^{(\alpha)} denote the crop image constructed with scale parameter α\alpha, and let μt(α)\mu_{t}^{(\alpha)} denote its associated non-image metadata. The observation entries in the three reconstructed contexts are

otno-up=(vt(1),μt(1)),otmeta=(∅,μt(4)),otup=(vt(4),μt(4)).(20)\begin{aligned} o_{t}^{\text{no-up}} &= \left(v_{t}^{(1)},\mu_{t}^{(1)}\right),\\ o_{t}^{\text{meta}} &= \left(\varnothing,\mu_{t}^{(4)}\right),\\ o_{t}^{\text{up}} &= \left(v_{t}^{(4)},\mu_{t}^{(4)}\right). \tag*{(20)} \end{aligned}

Here, ∅\varnothing denotes the absence of a crop-image payload. Metadata include image identifiers, dimensions, original-image regions, resize information, and local-to-global coordinate mappings.

The metadata-only context retains the metadata associated with the upsampled crop observations while omitting their image payloads. This editing rule applies to historical observations from both original-image and nested crop requests; failure feedback is retained. The resulting comparison examines the value of additional crop images conditional on the original image, spatial metadata, and the supplied interaction history.

Crop presentation scale. Following Equation (18), a source crop of width wcw_c and height hch_c is constructed at scale

sα=min⁡(α,Pmax⁡wchc),α∈{1,4},(21)s_{\alpha} = \min\left(\alpha,\sqrt{\frac{P_{\max}}{w_c h_c}}\right), \qquad\alpha\in\{1,4\}, \tag*{(21)}

where Pmax⁡P_{\max} is the shared output-pixel cap. For the no-upsampling condition,

sno-up=min⁡(1,Pmax⁡wchc).(22)s_{\text{no-up}} = \min\left(1,\sqrt{\frac{P_{\max}}{w_c h_c}}\right). \tag*{(22)}

Crops within the pixel cap therefore retain their source dimensions, whereas larger crops may still be downsampled. The upsampling condition uses α=4\alpha= 4, enlarging each spatial dimension by up to four times when permitted by the cap.

Both image conditions retain the model’s usual image preprocessing. Metadata describe the actual constructed views, so image dimensions, scale factors, and coordinate mappings can differ between the two scale settings. Interpolation changes presentation scale without introducing new scene information.

Evaluation protocol. We evaluate the direct-reasoning baseline and the three reconstructed-context conditions on the same evaluation items, using the same decoding settings and applicable generation limits. All three observation-history conditions use the same post-RL Qwen3.8-27B checkpoint and identical decoding settings and generation limits. The upsampling condition reuses the final answers and scores from the original online evaluation of the RL model reported in the main results, whereas answers for the no-upsampling and metadata-only conditions are regenerated from their modified histories. Their scores consequently characterize answer generation under different historical observations, rather than the outcome of independently executed crop-selection policies.

The four evaluated conditions and their scores are reported in Table 4 in the main text.

Results and interpretation. For benchmark bb, let Ab(c)A_b^{(c)} denote accuracy under condition cc. We summarize the differences as

Δboverall=Ab(up)−Ab(direct),Δbimage=Ab(up)−Ab(meta),Δbscale=Ab(up)−Ab(no−up).(23)\begin{aligned} \Delta_b^{\mathrm{overall}} &= A_b^{(\mathrm{up})} - A_b^{(\mathrm{direct})}, \\ \Delta_b^{\mathrm{image}} &= A_b^{(\mathrm{up})} - A_b^{(\mathrm{meta})}, \\ \Delta_b^{\mathrm{scale}} &= A_b^{(\mathrm{up})} - A_b^{(\mathrm{no-up})}. \tag*{(23)} \end{aligned}

The overall contrast compares the evaluated configurations, including differences in model checkpoint and supplied context; it does not isolate crop-feedback effects. The image and scale contrasts compare context variants reconstructed from the same reference history.

Upsampled crops achieve the highest accuracy on both benchmarks, exceeding metadata-only contexts by approximately 10.55/9.77 percentage points on SSPro/PerceptionBench and unupsampled crops by 12.35/7.47 points. These results highlight the importance of both crop-image availability and presentation scale.

The comparison between metadata-only and unupsampled-crop contexts reveals a task-dependent pattern. Metadata-only contexts perform better on SSPro (70.6% versus 68.8%), whereas unupsampled crops perform better on PerceptionBench (52.2% versus 49.9%). This pattern suggests that coordinate-level visual primitives can support spatial prediction in grounding, while localized image content may aid non-GUI reasoning by reducing irrelevant background. The strongest results with upsampled crops suggest that both tasks benefit from suitably presented local visual evidence.

Extended PerceptionBench Reference Results

Table 5 reports external reference results on PerceptionBench.

CapabilityGPT-5.6-SolKimi K3Claude-Fable-5Gemini-3.1-ProGPT-5.5Seed-2.1-ProGemini-3.5-FlashQwen 3.7-Plus
VRel69.768.258.558.861.957.653.659.1
Count62.459.752.956.955.851.243.653.3
Attr62.159.460.961.860.958.254.555.8
Depth55.552.451.550.048.843.649.748.5
Loc76.770.370.452.765.850.050.652.7
Comp67.059.156.161.765.659.554.855.9
FGR55.955.951.654.847.256.651.746.8
Ctx60.053.359.861.258.060.453.352.2
OCR54.961.264.364.356.566.759.654.5
Hallu26.941.745.040.634.749.850.629.5
Overall59.758.557.256.255.855.052.051.1
CapabilityQwen 3.5-397B-A17BClaude-Opus-4.8Kimi K2.6Grok-4.5Gemma-4-31BGLM-5V-TurboMinimax-M3GLM-4.6V
VRel55.251.450.947.042.741.240.035.2
Count49.144.245.235.233.940.030.331.8
Attr53.049.443.639.440.341.234.635.2
Depth44.640.642.441.239.141.236.729.1
Loc46.758.845.239.744.943.933.330.6
Comp49.848.439.143.743.745.231.234.8
FGR44.840.734.536.239.036.626.629.3
Ctx50.244.740.439.645.932.931.033.7
OCR52.954.140.843.946.743.535.739.2
Hallu26.938.741.044.732.128.029.926.9
Overall47.547.242.641.040.739.633.132.5

Table 5. PerceptionBench accuracy (%) across ten perceptual capabilities. Panels (a,b) provide results for 16 external reference models. External scores: PerceptionBench [14], official leaderboard (accessed September 21, 2026). Evaluation and inference settings may differ. Bold in (a,b) marks the highest external score per capability across all 16 models. Our checkpoint results appear in Table 2. VRel: visual relation; Count: counting; Attr: attribute; Depth: depth/3D perception; Loc: localization; Comp: comparison; FGR: fine-grained recognition; Ctx: contextual integration; OCR: optical character recognition; Hallu: perception-related hallucination.

Example of a complete agentic grounding/reasoning trace

Figure 4. Recorded GUI observation trajectories with verbatim reasoning excerpts and all crop actions. Quartus relocates a missed region, Stata refines a truncated target, and AutoCAD recursively crops a returned view and repairs an incomplete response. I0 denotes the original image and C1 the first crop. GUI observations are reconstructed from logged actions; colored annotations are editorial.

Figure 5. Complete PerceptionBench action sequences with verbatim reasoning excerpts. OCR and vehicle recognition revise misplaced crops; C17 recursively inspects a returned observation. Bounds use parent-image pixels. All final answers match their references.

Training and Critic Prompts

System Prompt for Agentic RL

We train with ms-swift GRPO and a custom multi-turn scheduler. At rollout initialization, the scheduler constructs the system message below from the protocol template, expanding the crop description. Braced fields are filled with the task, image metadata, and tool budget; line wrapping is for presentation only. The original screenshot is supplied in the initial user message.

The {max_tool_calls} field is set to the configured tool-call budget, capped at 3 (default: 3). The scheduler executes textual crop actions and returns images and metadata in a user message, without a native function-calling schema. GRPO rewards and loss masks are computed by the framework and plugin, not specified through an additional training prompt.

Agentic RL system prompt pRLp_{\mathrm{RL}}

You are an agentic GUI grounding model.

Task instruction:
{instruction}

Original image metadata:
- image_path: {image_path}
- resolution: {width}x{height} pixels (width x height)
- origin: [0,0] at the top-left

Goal:
Output the center point of the position corresponding to the task
instruction. The final coordinate MUST be relative to the ORIGINAL
image and normalized to [0.00,1000.00], where [0.00,0.00] is the
original image's top-left and [1000.00,1000.00] is its bottom-right.
Never output absolute pixels or coordinates local to a crop.
Preserve sub-unit precision:
the two final coordinate values MUST each have exactly two digits
after the decimal point (for example, [512.34,76.50]). If the task
is infeasible (for example, the requested UI element or action is
absent from the image), use [-1.00,-1.00].

Available local action:
- crop_image: Crop an image that has already been supplied in this
  conversation. image_path must be the exact original path or a path
  returned by an earlier crop_image call. x1, y1, x2, y2 are
  absolute pixel coordinates in that referenced image's LOCAL
  coordinate system, with (x1,y1) at the top-left and (x2,y2) at the
  bottom-right. The result provides a new image path, dimensions,
  and the crop region mapped into the ORIGINAL image. Returned crops
  are resized for inspection, so nested crops must use the returned
  width and height. Crops may be nested. The final answer must
  always use relative coordinates of the ORIGINAL image normalized
  to the [0,1000] range, never crop-local coordinates. When writing
  a crop_image action, format each of x1, y1, x2, and y2 with
  exactly two digits after the decimal point.

Agentic RL system prompt pRLp_{\mathrm{RL}} (continued)

Text-action protocol:

Every turn must first emit visible reasoning inside <think>...</think>, then end with exactly one of these choices:

  • To inspect one region: <think>brief reasoning for the crop</think> <action>crop_image(image_path="exact path", x1=100.00, y1=200.00, x2=500.00, y2=600.00)</action>

  • To inspect two independent regions in the same turn: <think>brief reasoning for both crops</think> <action>crop_image(image_path="exact path", x1=100.00, y1=200.00, x2=500.00, y2=600.00)</action> <action>crop_image(image_path="exact path", x1=600.00, y1=200.00, x2=900.00, y2=600.00)</action>

  • To return the final point: <think>brief reasoning for the final point</think> [x.xx,y.yy]

crop_image is a textual action interpreted locally after your answer. It is NOT an API function call. Do not emit JSON, <tool_call> XML, or native tool_calls. Keep x1, y1, x2, and y2 as four separately named numeric arguments.

Tool rules:

  1. Before returning the final coordinate, you SHOULD make at least one successful crop_image call to inspect the target more precisely. A direct first-turn coordinate is discouraged. Skip cropping only when the task is clearly infeasible or cropping cannot reasonably improve localization.

  2. You may make at most {max_tool_calls} crop_image calls in total. Never exceed this budget; use each crop purposefully and return the final coordinate as soon as the target is localized. A single assistant turn may contain one or two crop_image actions, and every action counts separately toward the total budget. Never emit more than two actions in one turn or mix crop actions with a final coordinate.

  3. Tool arguments are local absolute pixels in the image_path passed to that call. Write x1, y1, x2, and y2 with exactly two digits after the decimal point, including trailing zeros (for example, x1=100.00). Two actions emitted in the same turn must refer only to image paths already available before that turn; the second action cannot depend on the result of the first action from the same turn.

  4. Each result states the crop's region both in original-image pixels and original-image normalized [0,1000] coordinates. Use that mapping to convert any point found in a crop back to the ORIGINAL image's normalized [0,1000] coordinate system.

  5. After crop_image actions, the following single user message contains all result metadata and all successful crop images in action order.

Agentic RL system prompt pRLp_{\mathrm{RL}} (continued)

Final-answer rule:
When ready, output exactly <think>...</think> followed by a newline
and [x.xx,y.yy], with no prose, Markdown, or action wrapper after
the coordinate. Both values must contain exactly two digits after
the decimal point, including trailing zeros. The point must be in
the ORIGINAL image's normalized [0,1000] coordinate system. If the
task is infeasible, use [-1.00,-1.00].

Initial image message and protocol correction. The initial user message uses ms-swift’s <image> marker, with the screenshot supplied through the request’s image field. Unparseable responses trigger the correction below as a user turn, with a default allowance of 2 corrections. Crop observations follow the system protocol.

Initial user message pimagep_{\mathrm{image}}

image_path: {image_path}<image>

Invalid-action correction pcorrectp_{\mathrm{correct}}

Your previous answer was not valid. Reply with <think>...</think>
followed by either one or two independent crop actions, each in its
own <action>crop_image(...)</action> block, or one final [x.xx,y.yy]
coordinate. Do not emit more than two actions or mix actions with a
final coordinate. Every coordinate value must have exactly two
digits after the decimal point.

Prompting and learned behavior. The system prompt explicitly encourages at least one successful crop and requests a final answer once the target is localized. These instructions are part of the interaction protocol; adherence alone is not evidence that RL learned when inspection is useful or when to stop.

Prompts for CoT Scoring

The critic receives a separate text-only request. In blind mode, it sees the task and transcript, with the five criteria below substituted for {criteria}, but no images or target annotations. It scores textual reasoning–action consistency and cannot verify visual claims. The boxes use the supplied English rendering; the Chinese implementation requests an 80-character comment, whereas this rendering specifies 80 words.

DeepSeek critic system prompt pcotsysp_{\mathrm{cot}}^{\mathrm{sys}} (blind mode)

You are a quality reviewer of the reasoning process of a GUI
grounding agent. The agent's task is to locate the center point of
the target described by a task instruction within a UI screenshot;
it may first crop and zoom into the image multiple times to inspect
it, and must finally output coordinates normalized to [0, 1000], or
[-1.00, -1.00] when the task is infeasible.

DeepSeek critic system prompt (continued)

You can only see text: the task instruction, the agent's per-turn thinking (think), action decisions (action), and the execution results of crop actions (feedback). You cannot see the images, and you do not need to solve the task yourself -- only evaluate the quality of the reasoning process.

Evaluation criteria:

  1. Goal orientation: whether each turn's thinking revolves around "how to locate the target of the task instruction";

  2. Feedback utilization: whether subsequent thinking leverages the results of previous crop executions (successful region ranges / failure reasons);

  3. Thought-action consistency: whether the regions and directions described in the thinking match the coordinate parameters in the actions;

  4. Incremental progress: whether each turn brings new information or new decisions, without repeating the same crop or reasoning;

  5. Conciseness: no meaningless filler or parroting.

Scoring anchors:

  • 0.9~1.0: The thinking stays focused on locating the target throughout, fully leverages the crop feedback from each turn, is consistent with the action parameters, converges step by step, and is concise without redundancy;

  • 0.6~0.8: The thinking is generally reasonable, with only minor flaws that do not affect the main line (e.g., slightly imprecise coordinate descriptions);

  • 0.3~0.5: There are obvious problems -- at least one of: ignoring tool feedback, inconsistency between thinking and actions, repeating the same ineffective operation, or reasoning detached from the task goal;

  • 0.0~0.2: Almost no effective reasoning, heavy verbatim repetition, or content completely irrelevant to the task.

First give a brief comment of no more than 80 words, then output the score alone on a new line, in exactly this format:

Score: 0.xx

(a float in 0.0~1.0, rounded to two decimal places)

Critic user prompt pcotuserp_{\mathrm{cot}}^{\mathrm{user}}

Task instruction: {task}

The agent's full trajectory (think = reasoning, action = action decision, feedback = crop execution result):

{transcript}

Transcript construction and reward use. Each serialized assistant turn contains [Turn N], think:, action:, and available feedback: fields. Feedback records crop geometry and dimensions or execution failures; images are excluded. Default character limits are 1,500 per reasoning segment, 600 per action summary, and 6,000 per transcript, so “full trajectory” is subject to truncation. The default reward multiplies the parsed score by terminal correctness (Equation (9)), without exposing that label to the blind critic. Trajectories without visible reasoning receive zero.

Optional oracle variant. When oracle mode is enabled, the sixth criterion and reference annotation below are appended to the system and user prompts, respectively. Other instructions are unchanged. The annotation is exposed to the critic, not the policy; the critic still receives no images.

Oracle addition to the critic system prompt

6. Annotation alignment: against the reference annotation (GT bbox or infeasibility flag), whether the cropped regions and coordinate directions are progressively approaching the correct location; for refusal samples, evaluate whether the justification for judging the task as infeasible is sufficient.

Oracle addition to the critic user prompt

Reference annotation (GT bbox in [0, 1000] normalized coordinates; [-1, -1, -1, -1] means the task is infeasible): {ground_truth}

References

  1. [1]Anthropic. Claude opus 4.8 system card. Technical report, Anthropic, May 2026. URL https://www-cdn.anthropic.com/0b4915911bb0d19eca5b5ee635c80fef830a37ea.pdf.
  2. [2]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunj iang Ge, et al. Qwen3-Vl technical report. arXiv preprint arXiv:2511.21631, 2025.arxiv.org/abs/2511.21631
  3. [3]Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing gui grounding for advanced visual gui agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9313–9332, 2024.
  4. [4]H Company. Holo2 - open foundation models for navigation and computer use agents, 2025. URL https://huggingface.co/collections/Hcompany/holo2.
  5. [5]Yue Fan, Xuehai He, Diji Yang, Kaizhi Zheng, Ching-Chen Kuo, Yuting Zheng, Xinze Guan, and Xin Wang. Grit: Teaching mllms to think with images. Advances in Neural Information Processing Systems, 38: 116522–116543, 2026.
  6. [6]Google. Introducing agentic vision in gemini 3 flash. Google Developers Blog, 2026. URL https://blog.google/innovation-and-ai/technology/developers-tools/agentic-vision-gemini-3-flash/. Accessed: 2026-09-20.
  7. [7]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Rouyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
  8. [8]Jack Hong, Chenxiao Zhao, ChengLin Zhu, Weiheng Lu, and Guohai Xu. Deepeyesv2: Toward agentic multimodal model. In International Conference on Learning Representations, volume 2026, pages 114851–114872, 2026.arxiv.org/abs/2511.05271
  9. [9]Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014.
  10. [10]Mahyar Khayatkhoei, Prateek Chhikara, Filip Ilievski, et al. Mllms know where to look: Training-free perception of small visual details with multimodal llms. In International Conference on Learning Representations, volume 2025, pages 68194–68213, 2025.
  11. [11]Keon Kim and Krish Chelikavada. Zoom consistency: A free confidence signal in multi-step visual grounding pipelines. arXiv preprint arXiv:2604.15376, 2026.
  12. [12]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626, 2023.
  13. [13]Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 8778–8786, 2025.
  14. [14]Zichao Lin, Yifeng Xie, Bowen Qu, Haiming Wang, Jia Li, Haoning Wu, Yuhao Dong, Zuhao Yang, Jingguo Zhu, Haoyu Lu, et al. Perceptionbench: Evaluating atomic visual perception in multimodal large language models. arXiv preprint arXiv:2607.24957, 2026.arxiv.org/abs/2607.24957
  15. [15]Xianjie Liu, Yiman Hu, Yixiong Zou, Liang Wu, Jian Xu, and Bo Zheng. Hide: Rethinking the zoom-in method in high resolution mlms via hierarchical decoupling. arXiv preprint arXiv:2510.00054, 2025.arxiv.org/abs/2510.00054
  16. [16]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
  17. [17]Tiange Luo, Lajanugen Logeswaran, Justin Johnson, and Honglak Lee. Visual test-time scaling for gui agent grounding. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 19989–19998. IEEE, 2025.
  18. [18]Ahmed Masry, Mohammed Saidul Islam, Mahir Ahmed, Aayush Bajaj, Firoz Kabir, Aaryaman Kartha, Md Tahmid Rahman Laskar, Mizanur Rahman, Shadıkur Rahman, Mehrad Shahmohammadi, et al. Chartaqpro: A more diverse and challenging benchmark for chart question answering. In Findings of the Association for Computational Linguistics: ACL 2025, pages 19123–19151, 2025.
  19. [19]Siqi Pei, Liang Tang, Tianan Duan, Long Chen, Shuxian Li, Kaer Huang, Yanzhe Jing, Yiqiang Yan, Bo Zhang, Chenghao Jiang, et al. Adazoom-gui: Adaptive zoom-based gui grounding with instruction refinement. arXiv preprint arXiv:2603.17441, 2026.arxiv.org/abs/2603.17441
  20. [20]Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Gaicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641–2649, 2015.
  21. [22]Qwen Team. Qwen3.8-max: A new bar for coding and cowork, August 2026. URL https://qwen.ai/blog?id=qwen3.8.
  22. [23]Gabriel Sarch, Snigdha Saha, Naitik Khandelwal, Ayush Jain, Michael Tarr, Aviral Kumar, and Katerina Fragkiadaki. Grounded reinforcement learning for visual reasoning. Advances in Neural Information Processing Systems, 38:150977–151013, 2026.
  23. [24]Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in neural information processing systems, 36:68539–68551, 2023.arxiv.org/abs/2302.04761
  24. [25]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347
  25. [26]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
  26. [27]Haozhan Shen, Kangjia Zhao, Tiancheng Zhao, Ruochen Xu, Zilun Zhang, Mingwei Zhu, and Jianwei Yin. Zoomeye: Enhancing multimodal llms with human-like zooming capabilities through tree-based image exploration. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 6613–6629, 2025.arxiv.org/abs/2411.16044
  27. [28]Enxin Song, Yinuo Xu, Shusheng Yang, Wenhao Chai, and Jiatao Gu. Video-index: A curated meta-benchmark for video understanding. arXiv preprint arXiv:2610.00960, 2026.arxiv.org/abs/2610.00960
  28. [29]Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Ziwei Chai, Y Charles, HS Che, Cheng Chen, et al. Kimi k2.5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026a.
  29. [31]Venus Team, Zhuohan Cai, Haoxing Chen, Jiaxuan Chen, Weizhi Chen, Changlong Gao, Zhangxuan Gu, Yuan Guo, Yusong Hu, Jianrong Jiang, et al. Ui-venus-2 technical report. arXiv preprint arXiv:2609.00028, 2026b.arxiv.org/abs/2609.00028
  30. [32]Haochen Wang, Xiangtai Li, Zilong Huang, Anran Wang, Jiacong Wang, Tao Zhang, Sule Bai, Zijian Kang, Jiashi Feng, Zhuochen Wang, et al. Traceable evidence enhanced visual grounded reasoning: Evaluation and method. In International Conference on Learning Representations, volume 2026, pages 148769–148794, 2026a.arxiv.org/abs/2507.07999
  31. [33]Jiacong Wang, Zijian Kang, Haochen Wang, Xiao Liang, Ya Wang, Jiawen Li, Bohong Wu, Jiao Ran, Haiyong Jiang, Chao Feng, et al. Vgr: Visual grounded reasoning. In International Conference on Learning Representations, volume 2026, pages 89506–89528, 2026b.arxiv.org/abs/2506.11991
  32. [34]Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, Wei Yu, and Dacheng Tao. Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 7907–7915, 2025.arxiv.org/abs/2408.15556
  33. [35]Lai Wei, Liangbo He, Jun Lan, Lingzhong Dong, Yutong Cai, Siyuan Li, Huijia Zhu, Weiqiang Wang, Linghe Kong, Yue Wang, et al. Zooming without zooming: Region-to-image distillation for fine-grained multimodal perception. arXiv preprint arXiv:2602.11858, 2026.arxiv.org/abs/2602.11858
  34. [36]Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal llms. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13084–13094. IEEE, 2024.arxiv.org/abs/2312.14135
  35. [37]Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, et al. Scaling computer-use grounding via user interface decomposition and synthesis. Advances in Neural Information Processing Systems, 38, 2026.arxiv.org/abs/2505.13227
  36. [38]Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chencheng Ling, et al. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026.arxiv.org/abs/2606.19348
  37. [39]Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous GUI interaction. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 69772–69805. PMLR, 2025.arxiv.org/abs/2412.04454
  38. [40]Yue Yang, Ajay Patel, Matt Deitke, Tanmay Gupta, Luca Weihs, Andrew Head, Mark Yatskar, Chris Callison-Burch, Ranjay Krishna, Aniruddha Kembhavi, et al. Scaling text-rich image understanding via code-guided synthetic multimodal data generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 17486–17505, 2025.arxiv.org/abs/2502.14846
  39. [41]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.arxiv.org/abs/2210.03629
  40. [42]Qianhao Yuan, Jie Lou, Xing Yu, Hongyu Lin, Le Sun, Xianpei Han, and Yaojie Lu. Vision-opd: Learning to see fine details for multimodal llms via on-policy self-distillation. arXiv preprint arXiv:2605.18740, 2026.arxiv.org/abs/2605.18740
  41. [43]Lu Zhang, Jiazou Yu, Haomiao Xiong, Ping Hu, Yunzhi Zhuge, Huchuan Lu, and You He. Finers: Fine-grained reasoning and segmentation of small objects with reinforcement learning. Advances in Neural Information Processing Systems, 38:67403–67426, 2026a.arxiv.org/abs/2510.21311
  42. [44]Xinyu Zhang, Yuxuan Dong, Lingling Zhang, Chengyou Jia, Zhuohang Dang, Basura Fernando, Jun Liu, and Mike Zheng Zhou. Gofft: Chain of foresight-focus thought for visual language models. Advances in Neural Information Processing Systems, 38:106182–106201, 2026b.arxiv.org/abs/2509.22010
  43. [45]Yunzhu Zhang, Zeyu Pan, Zhengwen Zeng, Shuheng Shen, Changhua Meng, and Linchao Zhu. Mvp: Multiple view prediction improves gui grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27482–27492, 2026c.arxiv.org/abs/2512.08529
  44. [46]Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, et al. Swift: a scalable lightweight infrastructure for fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 29733–29735, 2025.
  45. [47]Ziwei Zheng, Minghao Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, and Chao Shen. Deepeyes: Incentivizing "thinking with images" via reinforcement learning. In International Conference on Learning Representations, volume 2026, pages 126775–126798, 2026.arxiv.org/abs/2505.14362
  46. [48]Hanzhang Zhou, Xu Zhang, Panrong Tong, Jianan Zhang, Liangyu Chen, Quyuy Kong, Chenglin Cai, Chen Liu, Yue Wang, Jingren Zhou, et al. Mai-ui technical report: Real-world centric foundation gui agents. arXiv preprint arXiv:2512.22047, 2025.arxiv.org/abs/2512.22047
  47. [49]Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4995–5004, 2016.arxiv.org/abs/1511.03416

Paper details

vlnc:W_5g7bbvgz
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents