betaFeedback

WorksComputer Science

T2T Technical Report: Synthesizing GUI Agent Trajectories at Scale from Video Tutorials

Abstract

Training generalist GUI agents requires large volumes of interaction trajectories comprising screenshot sequences paired with grounded UI actions, yet collecting such data through human demonstration is expensive and difficult to scale. Although public software tutorial videos contain rich operational demonstrations, their continuous pixel streams lack discrete action boundaries, semantic intents, and precise spatial coordinates, making them difficult to use directly for model training. To address this, we present T2T (Tutorial-to-Trajectory), a fully automated pipeline for synthesizing GUI agent trajectories at scale. T2T spans desktop, web, and mobile platforms to convert unannotated video tutorials into structured, executable interaction trajectories. To guarantee trajectory quality, T2T combines active segment filtering, keyframe identification via localized UI change, shortcut-free observation frame selection, presence-aware grounding, and two-stage action-space normalization. These refined trajectories are then converted into multi-task training views spanning navigation, planning, forward dynamics, and grounding. We fine-tune Qwen3-VL-8B on a subset of the more than one million human-annotation-free training samples generated by this pipeline. Evaluations show that the model achieves consistent gains across multiple benchmarks covering desktop, web, and mobile environments, including OSWorld-G, ScreenSpot-Pro, AndroidControl, and AgentNetBench, indicating improved core capabilities in grounding, step-level instruction execution, and trajectory-level decision-making. Notably, all gains are achieved without any in-distribution training data from target domains. Learning exclusively from trajectories extracted from public videos via the T2T pipeline, the model exhibits cross-application and cross-task generalization capabilities.

Introduction

Large language models and vision-language models have demonstrated remarkable capabilities in understanding natural language, generating code, and reasoning over visual content [13, 14, 2], increasingly serving as general-purpose foundations for a wide range of intelligent applications. A natural next step is to translate these capabilities into concrete task execution in the digital world. Unlike web APIs or command-line interfaces that expose structured endpoints, the vast majority of software, from productivity suites and creative tools to enterprise systems and mobile applications, is designed primarily for human interaction through graphical user interfaces. This makes GUI the de facto universal interface to the digital world, but also one that demands perceptual and motor skills traditionally reserved for human users: recognizing UI elements from pixels, understanding spatial layouts, tracking state changes across screens, and executing precise sequences of clicks, keystrokes, and gestures [3, 4, 19]. GUI agents, autonomous systems built on top of LLMs and VLMs that perceive screen content and perform such actions on behalf of users, bridge this gap by operating directly at the interface level, automating tasks across arbitrary applications without requiring dedicated APIs or developer-side instrumentation. A substantial body of work has advanced this direction in recent years [17, 12], producing increasingly capable agents across desktop, mobile, and web platforms [7, 24, 9, 23, 11, 8] (ByteDance Seed 2025).

Despite rapid progress in model architecture and training recipes, the availability of high-quality trajectory data remains a central bottleneck. Training a capable GUI agent requires large collections of expert demonstrations: sequences of screenshots paired with precise actions that together capture realistic, multi-step interactions with diverse applications and interfaces. Curating such data through human annotation is inherently expensive and slow, and the cost grows sharply with task complexity and horizon length.

A natural alternative source of such data is video tutorials. Platforms such as YouTube host millions of screen recordings in which human experts demonstrate complex software workflows step by step, spanning a broad range of applications across desktop, mobile, and web platforms. These videos are inherently long-horizon, reflect realistic human operating patterns, and are available at a scale that manual annotation cannot easily match.

Several recent efforts have recognized the potential of web-sourced tutorials and videos, and explored converting them into agent trajectories [10, 22, 25, 16, 20]. However, bridging the gap from raw video to structured, training-ready trajectories presents several technical challenges: (1) Computational and Contextual Limits. Feeding unsegmented raw videos into vision-language models (VLMs) is computationally prohibitive and practically infeasible due to strict API limits and context windows. Pruning the vast majority of frames that contain non-interactive content (e.g., introductions, static slides, or idle pauses) and identifying informative frames are essential for efficient and effective video processing. (2) Visual Target Leakage and Shortcut Learning. Naively sampled screenshots often suffer from severe visual shortcuts. High-production tutorials frequently overlay colored boxes or arrows, training models to locate annotations rather than comprehend UI pixels. Similarly, taking screenshots at the exact action timestamp leaves the cursor directly hovering over the target, encouraging models to learn a trivial “read-the-cursor” shortcut. (3) Coordinate Hallucination under Target Absence. Standard grounding models may instead rely on layout priors (e.g., close buttons in corners) and hallucinate coordinates even when the intended target element is temporarily absent or occluded, injecting noisy coordinates into the training distribution. (4) Action Space Heterogeneity and Misalignment. Speech narration and screen activity are temporally asynchronous. Furthermore, raw video extraction produces a heterogeneous vocabulary of non-standardized action names, which must be mapped into unified, platform-conditioned tool-call schemas.

In this report, we present T2T (Tutorial-to-Trajectory), a fully automated pipeline designed to overcome these challenges. Starting from raw public videos, the pipeline proceeds through three stages: (1) Video Filtering and Preprocessing. We prune non-interactive video segments using cursor-based sparse probing as a user activity proxy, and enforce strict visual annotation filtering to eliminate superimposed highlights, arrows, and bounding boxes that cause label leakage. (2) Trajectory Extraction and Action Spatial Grounding. Instead of relying on noisy global frame differences or asynchronous audio transcripts, we pinpoint discrete action timestamps by scoring localized UI changes (explicitly discounting full-screen turnovers) and extracting settled before/after frame pairs. A vision-language model then derives task context, sub-step plans, and target referring expressions. To prevent the model from simply memorizing pointer positions, we perform a backward search for pre-action screenshots where the cursor is away from the target element. A three-tier presence-aware grounding cascade then localizes target elements, explicitly discarding unverified coordinates to prevent label noise while repurposing rejected steps as negative abstention samples. (3) Action Normalization and Multi-task Conversion. We normalize heterogeneous extractor action vocabularies through a two-stage normalization scheme into unified, platform-conditioned tool-call schemas (computer use, browser use, and mobile use). Finally, trajectories are rendered into complementary multi-task training views (navigation, planning, forward dynamics, and grounding) to maximize data efficiency.

Applied to a large-scale collection of public tutorial videos, T2T yields over one million training samples without human annotation. Supervised fine-tuning of Qwen3-VL-8B on a ∼\sim350K mixture drawn from this corpus, without utilizing any target-domain or benchmark training data, yields the fine-tuned model T2T-8B, which improves over the base model across multiple benchmarks: raising GUI grounding accuracy on OSWorld-G and ScreenSpot-Pro, boosting step success rate on AndroidControl, and outperforming the base model on AgentNetBench. These results show that a rigorous pipeline can enable raw video tutorials to serve as a scalable and effective data engine for training generalist GUI agents.

The T2T Pipeline

Overview

Figure 1 summarizes the pipeline. Given a screen recording and its transcript, T2T must answer three questions that the raw video does not state: when the user is interacting with the interface at all, that an action occurred and of what type, and where on the screen it was aimed. The first is answered during video filtering and segmentation (§ 2.2); the second and third during trajectory extraction and grounding (§ 2.3), which also groups the recovered actions into tasks. A final stage (§ 2.4) expresses the recovered actions in a single schema and renders them into training samples. Because the three inferences fail independently and at different rates for different action types, every stage records how each decision was reached, not only its outcome; this provenance (Section C) is what allows the corpus to be filtered and audited after the fact.

Figure 1. Overview of the T2T (Tutorial-to-Trajectory) pipeline. T2T converts GUI tutorial videos into structured interaction trajectories through video filtering, action extraction, spatial grounding, and action normalization. The resulting trajectories are transformed into four training views: navigation, planning, forward dynamics, and grounding.

From Video to Informative Segments: Video Preprocessing and Filtering

Web-sourced videos provide a large pool of potential training data, but their actual utility varies widely. Many clips feature no human-GUI interaction at all, including talking-head lectures, promotional teasers, and static slides. Even genuine screen tutorials are frequently padded with idle pauses, title cards, and off-topic discussion. Feeding raw footage directly into a vision-language model is computationally wasteful, clogging the model’s context with tens of thousands of uninformative frames. Worse, non-interactive clips dilute training quality by breaking the clear mapping between user actions, such as clicks or typing, and visual interface changes. This stage addresses both challenges by filtering out irrelevant videos and extracting informative segments—spans of continuous human–GUI interaction—from the videos that are retained. Figure 2 illustrates both steps, and Figure 3 traces the resulting data volumes through the full pipeline.

Figure 2. Video filtering and preprocessing. (a) Cursor-guided extraction of active interaction segments using sparse, adaptive cursor detection. (b) Multi-stage filtering of desktop, web, and mobile tutorials based on GUI activity, visual and semantic cues, and annotation overlays.

Cursor-based Segmentation and Filtering

In desktop and web tutorial videos, cursor movement serves as a primary indicator for human-GUI interaction. We track cursor activities to pinpoint active segments from raw video footage.

(1) Detecting Cursor Presence. Cursor glyphs typically span only a few dozen pixels on high-resolution displays, vary widely in shape from standard arrows and I-beams to hand pointers and custom icons, and often blend into complex UI backgrounds. Therefore, we deploy a dedicated cursor detector operating at high input resolutions to improve cursor detection accuracy. Given a single frame from a screen recording, the detector outputs the bounding box of the cursor pointer alongside a confidence score, or yields no detection when the cursor is absent. We apply a permissive confidence threshold to prioritize high recall, as a false positive merely extends an informative segment slightly, whereas a false negative prematurely truncates an ongoing human-GUI interaction.

User interactions usually span several seconds, making consecutive video frames nearly redundant as evidence of activity. Running cursor detection on every frame is therefore computationally wasteful. We replace full framerate inference with a sparse sampling schedule of roughly one frame per second. Furthermore, we make this probing adaptive by skipping the detector whenever the current frame matches the visual state of the last frame where a cursor was found. In these cases, the skipped frames are automatically assumed to contain a cursor. As a static screen provides no new interaction signals, this mechanism eliminates unnecessary computation during idle pauses.

(2) Extracting Informative Segments. The cursor detector outputs a series of discrete frames where a pointer is present, but downstream processing requires continuous segments. Defining segment boundaries merely by adjacent positive frames (those with detected cursors), is insufficient, as the pointer often vanishes intermittently during active usage, such as when occluded by dropdown menus, parked off screen, or kept still during reading pauses. To accommodate this intermittent visibility, we tolerate short non-detection gaps using a defined grace period. A segment opens at the first detected frame, remains active across transient non-detection gaps, and closes only after this specified period elapses without any new detection. Finally, we prune segments falling below a minimum duration threshold to eliminate accidental mouse jitters and video editing artifacts.

(3) Filtering Videos Based on GUI Activity Duration. To decide whether a video should be retained, we evaluate its cumulative informative segment coverage rather than its raw frame-level detection rate. Specifically, a video is kept if its extracted informative segments account for a sufficient fraction of its total duration. This coverage-based metric is essential because our segment extraction already bridges short pauses when the cursor is temporarily hidden or stationary. Scoring videos purely on raw cursor hits would falsely reject high-quality tutorials that contain minutes of continuous workflow. Finally, the temporal boundaries of these informative segments are passed downstream so that the subsequent trajectory extraction stage (§ 2.3) processes only valid interaction intervals. The operating points of all filters are listed in Table 10.

FilterSetting
Cursor detectorsingle-class YOLO on cursor glyphs; 1280 px input; confidence threshold 0.3
Probe strideevery 30th frame (one probe per second at 30 fps); every 10th frame for the 720p pool
Static-frame skipdetector skipped when mean absolute intensity difference to the last cursor-bearing frame < 10
Segment gap tolerancesegment closes after 2 s without a detection
Minimum segment length6 s
Desktop/web retentionunion of active segments ≥ 30% of video duration
Mobile visual filterCLIP zero-shot on 8 uniformly sampled frames; ≥ 4 classified as mobile UI
Mobile linguistic filter≥ 2 imperative instruction phrases in the transcript, plus a normalized instruction-density threshold
Annotated footage16 sampled frames; reject if colored box outline in ≥ 25% of frames, or mean annotation-color coverage > 15% of frame area

Table 10. Video preprocessing and filtering parameters.

Visual and Semantic Filtering for Mobile Videos

Mobile tutorial videos lack cursor signals because touch input replaces mouse pointers. To filter mobile recordings effectively, we enforce a dual-validation pipeline where a video must satisfy both visual and semantic criteria. First, the visual filter uses a zero-shot image classifier on uniformly sampled frames to ensure that at least half of the sampled frames actually display a mobile UI. Second, the semantic filter analyzes the video transcript to confirm that imperative instructional phrases occur with sufficient frequency (captions and phrase list in Section F.1). Enforcing both conditions eliminates two common failure modes, namely videos that show mobile screens while lacking actionable guidance, and videos that provide tutorial narration while mostly featuring presenter footage.

Visual Annotation Filtering

Beyond the above approaches, we apply an additional filter to prevent label leakage. High-production tutorials frequently superimpose bright rectangles, arrows, or highlights over target UI elements. These visual overlays effectively embed ground-truth answers directly into image pixels. As a result, a model may learn a superficial shortcut by tracking the annotations rather than understanding the interface, causing it to fail on clean screenshots during inference. To address this, we sample video frames and reject candidates using two complementary rules (thresholds in Table 10). First, we detect structured bounding-box outlines across a significant portion of frames. Second, we measure the total pixel area occupied by characteristic annotation colors. This dual mechanism is essential because the first rule identifies crisp box boundaries, while the second catches diffuse highlights and freehand markings that evade box detection.

Figure 3. Data flow through the T2T pipeline. (a) Video filtering and trajectory extraction. Some videos are processed at more than one resolution, so the nodes count preprocessing runs rather than unique videos: the 17,603 runs that yield trajectories correspond to the 16,974 unique source videos of § 2.6. (b) Conversion of extracted actions into retained steps, supervised samples, and the final training mixture. Supervised samples are counted over the navigation and grounding views, from which the training mixture is drawn; with the planning and forward-dynamics views the corpus yields 2,709,896 samples (§ 2.4). Brown branches indicate rejected data, while gray branches indicate data held in reserve.

From Video Segments to Grounded Trajectory: Trajectory Extraction and Action Spatial Grounding

Having isolated informative human-GUI interaction segments from raw videos, we now convert these visual clips into structured agent trajectories. On their own, unannotated video segments lack three essential elements required for a trajectory: pre-action visual states describing what the interface looks like before an interaction, textual action intents explaining why the action is performed, and spatial interaction coordinates specifying where the interaction occurs. To bridge this gap, this stage transforms the filtered segments into a discrete execution trajectory, mapping each step to a pre-action interface state StS_t, a textual action description AtA_t, and grounded spatial coordinates CtC_t. Figure 4 illustrates the main steps.

Figure 4. Trajectory extraction and spatial grounding. (a) Keyframe identification based on localized UI changes, combining high-resolution before–after pairs with low-resolution contextual frames. (b) Schema-constrained extraction of tasks and actions with semantic target descriptions. (c) Shortcut-free observation selection and presence-aware spatial grounding with confidence-based verification.

Keyframe Identification via Localized UI Change

To convert continuous video segments into discrete agent trajectories, we identify keyframes that capture the precise visual states immediately before and after each user interaction. The pre-action frame SbeforeS^{\mathrm{before}} serves as the visual context for agent decision-making, while the post-action frame SafterS^{\mathrm{after}} reveals the state transition required to infer action intent and verify execution. Pinpointing these transition moments is challenging because direct fixed-interval sampling blindly designates frames extracted at uniform temporal increments as trajectory steps. By ignoring natural action timing, this approach frequently captures mid-animation transient frames or skips brief interactions altogether, creating temporal misalignment between visual states and actual user actions. On the other hand, relying on pixel differences between consecutive full frames to detect action occurrences is unreliable. Global background movements, such as page scrolling or video cuts, generate massive frame-wide variance that easily drowns out the subtle localized changes produced by genuine UI interactions.

We resolve this through a two-stage strategy. First, we identify candidate change moments, defined as estimated timestamps in the video timeline where a GUI interaction is likely triggered. Rather than scanning every raw frame, we coarsely sample video frames and compute initial transition scores by partitioning each frame into a spatial grid, evaluating maximum localized pixel change while discounting full-screen turnovers. To refine these initial scores, we incorporate two multimodal priors. Candidate scores are multiplied by a boost factor if they fall within a temporal window surrounding transcript action verbs such as click or tap. Furthermore, an object detector locates the cursor bounding box, scaling up scores for candidates where UI changes occur directly adjacent to the pointer while discounting distant changes. All candidate moments are then re-ranked, and the top-scoring moments across the video timeline are selected for keyframe extraction. Second, for each selected top-scoring moment, we dynamically scan neighboring frames in the native video stream to locate stable boundaries where temporal pixel variations drop below a threshold, capturing clean SbeforeS^{\mathrm{before}} and SafterS^{\mathrm{after}} states once interface animations subside.

These sparse high-resolution keyframe pairs are interleaved with a dense stream of low-resolution frames sampled at fixed intervals across the entire video recording. While the high-resolution pairs supply fine-grained details for localized GUI interaction, the low-resolution stream captures comprehensive background context for the entire task. This interleaved sequence is subsequently passed to the downstream action extraction module.

Action Extraction and Schema-constrained Decoding

Given the interleaved sequence of sparse keyframe pairs and dense background context, action inference is powered by GPT-5.5 (prompts in Section F2). When an input frame sequence exceeds the context length limit, it is processed through sequential temporal windows. Compact structured summaries from previous windows are passed forward, ensuring that tasks spanning window boundaries maintain logical continuity without losing state history.

To prevent cross-domain action hallucinations, where models mistakenly output invalid actions from other environments such as mobile swipe gestures on web interfaces or platform-incompatible shortcuts, model decoding is constrained by a platform-specific schema that limits outputs strictly to valid native operations. The model generates structured action parameters, including action type, frame timestamps, typed text strings, key combinations, and scroll vectors, together with a referring expression for the target element. To eliminate spatial ambiguity during downstream grounding, each referring expression must identify the smallest actionable control using its literal interface text, such as the Export button, rather than referencing broad structural containers like toolbars or sidebars.

Individual actions are hierarchically grouped into sub-tasks, each labeled with a high-level user instruction and a concise execution plan. Because every action carries a specific task identifier, a single video containing multiple independent workflows is cleanly divided into separate trajectories.

Shortcut-Free Observation Frame Selection

Each extracted action must be paired with an appropriate pre-action observation frame OtO_t that reflects what an agent would observe before initiating an interaction. While the keyframe extraction stage captures SbeforeS^{\mathrm{before}} immediately preceding a state transition, directly using SbeforeS^{\mathrm{before}} as OtO_t introduces two critical failure modes that may impair trajectory dataset quality.

First, interaction-induced element disappearance occurs when the target UI control vanishes during the interaction itself. In modern user interfaces, elements such as dropdown menu items, context menus, hover tooltips, and confirmation popups immediately collapse or close upon being clicked. If the SbeforeS^{\mathrm{before}} frame is captured at the precise moment of action trigger or during a slight temporal overlap, the target element may already be absent from the display, leaving downstream agents without a visible interaction target in their observation frame. Second, cursor position leakage leads to shortcut learning, where the mouse pointer corrupts the visual observation. Because SbeforeS^{\mathrm{before}} is captured immediately prior to action execution, the cursor is often resting directly on top of the target UI element. When downstream models are trained on such frames, they may quickly learn a spurious shortcut by tracking the cursor location to identify the target, rather than parsing the visual interface and grounding textual referring expressions.

To resolve both failure modes, we search backward from the action timestamp for the latest video frame that satisfies three strict criteria. The candidate frame must maintain structural consistency with the action frame to ensure the overall interface layout has not shifted. The target UI element must remain fully visible and unoccluded. Finally, the cursor must be displaced away from the target element, ensuring that the pointer does not overlap with the target bounding box. When no frame satisfies all three criteria, the default SbeforeS^{\mathrm{before}} frame is retained as OtO_t and the step is flagged for quality control.

Presence-Aware Spatial Grounding

Given the pre-action observation frame OtO_t selected in the preceding step and the textual referring expression of the target UI control extracted from the action pipeline, mapping these inputs to spatial coordinates CtC_t is formulated as single-image visual grounding. Standard grounders often output coordinates even when the target element is absent by relying on layout priors, such as assuming a close button resides near the top-right corner, thereby injecting confident errors into the training distribution. We instead require explicit target-presence verification, allowing the system to decline prediction when an element is missing.

This mechanism is operationalized as a three-tier adaptive cascade (prompts for each tier in Section F.3). First, a fast pass feeds the full observation frame OtO_t and the target textual expression into Qwen3.7-Plus without chain-of-thought reasoning. This pass employs a high-recall prompt that encourages candidate generation whenever a plausible control is observed, rapidly producing a structured JSON response containing a presence flag, coordinate points, and a confidence score. When this pass yields a low-confidence prediction with a score below 0.6, a localized visual patch centered at the predicted coordinate is cropped from OtO_t and evaluated by a locally hosted Qwen3.6-27B model using a focused binary presence prompt. Predictions that are initially declined, as well as candidates explicitly rejected by the crop verification model, are escalated to a reasoning-enabled pass. Here, Qwen3.7-Plus re-evaluates the full observation frame OtO_t under a conservative prompt that explicitly instructs the model to assume a high likelihood of target absence. Guided by this strict prompt together with chain-of-thought reasoning, the model delivers a final presence and coordinate determination.

Verification fails when the escalated reasoning pass also declines presence, whether it was reached because the fast pass declined or because the cropped region did not contain the target. Some of these failures stem from source video characteristics, such as a transient dropdown menu or dialog already closing, a target element scrolling out of view, or a referring expression naming a structural container rather than an actionable control. Other failures arise from technical issues, such as unparseable model responses or transport errors, which are separated by a downstream filter before discarded steps are put to further use. Failure handling then depends strictly on action semantics. Pointing actions require precise coordinates and are removed from the trajectory, text entry actions degrade gracefully to ungrounded steps, and scroll operations are retained because their coordinates specify broad screen regions rather than localized controls. Finally, non-spatial operations such as hotkey combinations and wait commands bypass spatial grounding entirely.

Trajectory Assembly

Extracted steps are grouped by sub-task into ordered trajectories, discarding single-action tasks that offer no sequential learning signal. Action timestamps are aligned directly with the visual frames processed during extraction, enabling deterministic frame retrieval from the video stream for every step. Spatial coordinates are projected into a normalized coordinate space, decoupling trajectory representations from both the original capture resolution and any downstream resizing. When an intermediate action is removed due to failed spatial verification, its textual description is preserved within the interaction history of subsequent steps, maintaining chronological task continuity while preventing models from training on unverified coordinates. Figure 5 illustrates an assembled trajectory as stored in the final corpus; complete stored samples are given in Section D.

Figure 5. Example of an extracted interaction trajectory. Each step includes a pre-action observation, source-video timestamp, normalized tool call, and textual action description. Red markers indicate grounded interaction coordinates.

From Trajectories to Training-ready Tasks

Action-Space Normalization

While schema-constrained decoding during action extraction guarantees that extracted outputs conform to syntactical types, it cannot prevent semantically degenerate arguments or reconcile heterogeneous vocabularies across schema revisions and pipeline iterations. Decoupling raw action extraction from downstream training specifications allows action schemas and tool definitions to evolve without re-executing computationally expensive visual inference across the video corpus.

To achieve this, normalization proceeds through a two-stage pipeline. First, raw outputs from the extraction stage are preserved verbatim in trajectory logs as detailed in Table 11, ensuring complete trajectory auditability and supporting future schema revisions without re-annotating the underlying dataset. Second, each step is mapped to a JSON tool invocation under one of three platform-conditioned schema definitions, namely computer_use, browser_use, and mobile_use, as described in Table 12 and Table 13. This second stage canonicalizes variant synonyms into unified primitives (full mapping in Table 14), filters out degenerate operations such as zero-distance drags or zero-magnitude scrolls, and enforces platform-conditioned routing when the same high-level intent requires different platform-specific executions, such as resolving a back action to a native system gesture on mobile interfaces and a browser navigation control on web interfaces. Steps that lack valid schema representations or contain unresolvable arguments are discarded with explicitly logged reason codes rather than forced into approximate tool calls.

ActionStepsShareNote
left_click384,16040.96%} merged → click family
click152,42416.25%
type117,33212.51%} merged → type
write6,2810.67%
left_click_drag73,5877.85%
key45,5404.86%} merged → key
hotkey17,6571.88%
scroll40,2554.29%
wait34,7853.71%
swipe26,6752.84%intent-resolved
right_click14,0331.50%
double_click6,7350.72%
select_option5,2400.56%
mouse_move2,8450.30%
select1,9360.21%
open_url1,8520.20%
open_app1,8040.19%
navigate_back1,0000.11%
navigate_home8430.09%} merged → back / system button
go_backward5060.05%
go_back4420.05%
long_press7740.08%
hscroll5080.05%
system_button4500.05%
open2000.02%
go_forward28<0.01%
triple_click12<0.01%
middle_click8<0.01%

Table 11. Native extractor action vocabulary over all candidate steps. Synonym groups that are merged during normalization are marked.

ActionParametersGrd
clickcoordinate✓
long_presscoordinate, time✓
scrollcoordinate, direction, amount
swipecoordinate, coordinate2
typetext
system_buttonbutton ∈ {Back, Home, Menu, Enter}
opentext
keytext (adb keyevent syntax)
waittime
answertext
interacttext
terminatestatus ∈ {success, failure}

Table 12. mobile_use (12 actions).

ActionParametersGrd
left_clickcoordinate✓
right_clickcoordinate✓
middle_clickcoordinate✓
double_clickcoordinate✓
triple_clickcoordinate✓
mouse_movecoordinate✓
left_click_dragcoordinate, coordinate2✓
scrollcoordinate, clicks
hscrollcoordinate, clicks
typetext
keykeys
waittime
terminatestatus
answertext
interacttext
selectcoordinate, text✓ web
open_urltextweb
go_back—web
go_forward—web

Table 13. computer_use (desktop, 15 actions) and browser_use (web, 19 actions). The web schema is the desktop schema plus the four actions marked web.

Native actionMapped to (counts)Unconv.
left_clickleft_click 251,099; click 2,1431,746
clickleft_click 107,985; click 36,4791,038
typetype 81,991142
left_click_dragleft_click_drag 49,312; swipe 4207,074
waitwait 23,97453
keykey 22,42628
scrollscroll 20,120279
swipescroll 13,360; swipe 6,277; left_click_drag 3315,079
hotkeykey 17,609; system_button 345
right_clickright_click 9,79461
writetype 6,25823
select_optionselect 3,518; left_click 1,683; click 1722
double_clickdouble_click 4,36140
open_appopen 1,177627
open_urlopen_url 936321
navigate_homesystem_button 82716
navigate_backsystem_button 752; go_back 106142
long_presslong_press 55691
go_backwardgo_back 457; system_button 841
triple_clicktriple_click 110
go_forwardgo_forward 100
Mapping rate97.0%

Table 14. Native action →\to tool-call mapping, measured over the 720p and 4K source pools; native verbs that do not occur in these pools are omitted. “Unconverted” counts steps for which no defensible tool call could be produced; overall, 97.0% of steps map successfully.

Training-ready Multi-task Conversion

To construct training-ready datasets, trajectories are decomposed at the step level and formatted into several complementary task views as illustrated in Figure 6. Task views that retain sequential context model multi-turn interaction dynamics. Specifically, Navigation tasks predict the next action and tool invocation given the current state and interaction history. Planning tasks extend navigation by generating an explicit reasoning rationale prior to action prediction. Forward dynamics tasks forecast the resulting interface state change produced by a given action. Additionally, navigation includes a step-instruction variant that provides the specific low-level instruction for the current step alongside the task context, shifting the model objective from decision-making to direct instruction execution.

Figure 6. Multi-task training data conversion. A single interaction trajectory is transformed into four complementary training views: navigation, planning, forward dynamics, and grounding. The highlighted step illustrates their respective training formats, with gray text indicating shared context, black text indicating step-specific content, and green text indicating the target tool call.

In contrast, single-step Grounding tasks deliberately strip overall task instructions and interaction history. By pairing an isolated screenshot with a textual referring expression, these tasks force the model to resolve visual references purely from image context rather than inferring target positions from prior intent. Furthermore, steps where grounding failed due to target absence are repurposed as negative training examples, explicitly teaching models to decline prediction when targets are missing; these negative examples are retained in the corpus but are not part of the fine-tuning mixture of § 3.

Finally, the conversion pipeline binds system prompts and dialogue templates directly into the output files, producing self-contained and training-ready datasets. In total, the corpus yields 2,709,896 supervised samples: 787,616 navigation, 478,688 grounding, 658,109 planning and 785,483 forward-dynamics samples.

Corpus Statistics

The resulting corpus comprises 45,049 trajectories over 16,974 source videos, containing 937,912 interaction steps; each trajectory additionally carries one synthesized terminal step. The median trajectory is 13 steps long, with a ninetieth-percentile length of 48 and a maximum of 286. The corpus contains both short interactions confined to a single dialog and extended multi-application procedures. Figure 7 summarizes these statistics.

Figure 7. Overview of the dataset statistics. The dataset comprises 45,049 trajectories and 937,912 interaction steps from 16,974 source videos, spanning 5,696 applications and 2,349 web domains. The figure summarizes distributions across platforms, trajectory lengths, capture resolutions, applications, and action types.

Multi-Platform Coverage. The dataset covers three major computing environments including Web, Desktop, and Mobile, providing multi-platform diversity for GUI agent training. In terms of trajectory proportion, Web constitutes the largest share at 62.5%, Desktop accounts for 29.0%, and Mobile represents 8.5%. Measured by interaction steps, Web comprises 55.4%, Desktop represents 37.9%, and Mobile accounts for 6.7%, reflecting relatively longer interaction sequences within desktop applications. Furthermore, the corpus provides broad multi-resolution coverage, where 720p accounts for 50.0% at 22,505 trajectories, 4K represents 32.2% at 14,524 trajectories, and 1080p constitutes the remaining 17.8% at 8,020 trajectories.

Applications and Websites. Across all computing platforms, the corpus encompasses 5,696 distinct applications and 2,349 web domains. The application distribution varies by platform: Web interactions span 3,706 applications, Desktop encompasses 2,009 applications, and Mobile covers 895 applications (the per-platform counts exceed the total because some applications, such as Google Chrome, occur on more than one platform). On Desktop, the most frequent software includes Google Chrome, Visual Studio Code, Microsoft Excel, File Explorer, and FL Studio. Web workflows center on platforms such as Google Chrome, Visual Studio Code, File Explorer, Terminal, and Microsoft Edge. Mobile workflows focus primarily on utility and consumer applications, led by Settings, CapCut, Google Chrome, Safari, and Google Play Store.

Action Space Distribution. The unified action space defines 24 distinct action verbs managed across three platform-conditioned schemas. The realized interaction distribution is predominantly composed of pointing operations, where clicks of all forms represent 59.5% of total steps. Text entry accounts for 13.2%, view manipulations such as scrolling and swiping constitute 7.2%, and discrete keyboard actions make up 6.7%. Crucially, drag operations account for 7.8%, representing a substantial share for an action primitive that is typically scarce in annotation-driven corpora due to its high trajectory labeling complexity. The remaining 5.5% consists of specialized interaction types.

Experimental Setup

Training

Supervised fine-tuning is conducted on Qwen3-VL-8B-Instruct [15] for one epoch using a training mixture of approximately 350,000 samples drawn from the T2T corpus. This mixture is sampled primarily from the 1080p source pool, with mobile tasks supplemented from the 720p and 4K pools to compensate for the relative scarcity of mobile data. The exact mixture composition is detailed in Table 16 located in Section E. Across computing platforms, the training mixture allocates approximately 50% to desktop, 40% to web, and 10% to mobile interactions, adjusting the platform distribution relative to the full corpus presented in § 2.6 due to pool-specific sampling. Across task views, the mixture distributes roughly two-thirds of the samples to navigation, and one-third to single-step grounding tasks. The resulting fine-tuned checkpoint, designated as T2T-8B, serves as the primary model evaluated in § 4.

SubsetViewSourcePrompt templateSamples
Desktop navigationNav1080pDesktop navigation108,406
Web navigationNav1080pWeb navigation89,145
Desktop groundingGrd1080pDesktop grounding59,938
Web groundingGrd1080pWeb grounding50,535
Mobile navigation, SISI720pMobile navigation13,279
Mobile navigationNav1080pMobile navigation6,894
Mobile navigation, SISI1080pMobile navigation6,827
Mobile navigation, SISI4KMobile navigation5,102
Mobile groundingGrd1080pMobile grounding4,881
Desktop drag groundingGrd1080pDesktop grounding401
Web drag groundingGrd1080pWeb grounding177
Total345,585

Table 16. Fine-tuning mixture. Nav = navigation, Grd = grounding, SI = step-instruction navigation. Source gives the capture resolution of the pool each subset is drawn from; the two 4K pools are reported together.

Training hyperparameters and hardware configurations are summarized in Table 1. During fine-tuning, the vision encoder remains frozen while only the multimodal projector and the language model parameters are updated. This design choice relies on two key rationales. First, the pre-trained visual backbone already provides adequate spatial representation for pointing and element localization, allowing the corpus to focus on teaching procedural action execution and tool calling semantics rather than low-level visual recognition. Second, freezing the vision encoder eliminates a primary source of gradient variance caused by optimizing visual features across heterogeneous image resolutions ranging from 720p to 4K.

SettingValue
Base modelQwen3-VL-8B-Instruct
ObjectiveSFT, loss on assistant turns only
Trainableprojector + language model; vision encoder frozen
Epochs1
Global batch size48
Learning rate1×10−61 \times10^{-6}, cosine schedule, 3% warmup
Gradient clipping5
Precision / memorybf16, ZeRO-3, gradient checkpointing
Sequence length73,728 tokens, with sequence packing
Coordinate spacenormalized 1000×10001000 \times1000
Image resolutionnative, no downsampling

Table 1. Training configuration.

Evaluation

Evaluation Dimensions: To evaluate the capabilities imparted by the T2T corpus, we assess performance across three core dimensions, namely GUI grounding, instruction-guided step execution, and trajectory-level decision-making.

GUI grounding evaluates spatial localization given an instruction and a screenshot. The model predicts normalized coordinates for the target control. Benchmarks include OSWorld-G (Xie et al., [21]), comprising 564 desktop evaluation items alongside its refined-instruction variant, and ScreenSpot-Pro (Li et al., [5]), featuring 1,581 items across high-resolution professional software applications. A prediction is scored as correct if the coordinate falls within the ground-truth region according to each benchmark’s official evaluator.

Instruction-guided execution measures step-level action generation. Provided with a screenshot, an episode goal, interaction history, and a low-level step instruction, the model generates the complete next tool call. Evaluation is conducted on the official test split of AndroidControl (Li et al., [6]) in its low-level setting, which isolates step execution from task decomposition; action selection without step instructions is assessed on AgentNetBench below. We report type accuracy, which evaluates whether the predicted action primitive matches, and step success rate, which additionally requires exact agreement across all action parameters. For click actions, predicted points must fall inside bounding boxes extracted from the accessibility tree, falling back to Euclidean distance thresholds only when accessibility boxes are unavailable.

Trajectory-level decision-making assesses multi-step action selection, where models must autonomously determine appropriate action types without explicit step-level guidance. We evaluate on AgentNetBench (Wang et al., [18]), an offline desktop benchmark containing 100 trajectories and 1,208 interaction steps. Models receive a single screenshot per step alongside textual action history.

Evaluation Protocol: To eliminate silent evaluation artifacts and pipeline inconsistencies common in benchmark evaluations, we establish strict protocols governing coordinate resolution, prompt fidelity, and inference execution.

Evaluating models under unfamiliar prompt formats often introduces syntax confusion, causing artificial performance drops that reflect formatting errors rather than a lack of visual capability. To prevent this mismatch, we evaluate each model using a prompt structure it was trained for; the prompts are reproduced in Section F.5. Specifically, on the grounding benchmarks T2T-8B uses the grounding system prompt from its training pipeline, whereas the baseline model uses the computer-use prompt of the official Qwen3-VL cookbook, which requests the target as a computer_use function call with coordinates in a 1000×10001000 \times1000 space. On AndroidControl, both models receive the mobile navigation template from the T2T training pipeline, which presents the mobile_use tool schema of Section B in the function-calling format of the Qwen-VL agent recipes (tool signatures inside <tools> tags, actions returned inside <tool_call> tags); under an alternative T2T template that requires a reasoning block before each action, the baseline scored lower (70.32 versus 72.57 step success rate), so the shared template does not disadvantage it.

To ensure valid comparisons, our evaluation pipeline aligns model outputs and benchmark ground truths within a shared coordinate space. While both T2T-8B and Qwen3-VL-8B-Instruct output relative coordinates on a 0 to 1000 scale, individual benchmarks store ground-truth targets in varying formats, such as pixel boxes or normalized ratios. Our pipeline automatically converts model predictions into the format required by each benchmark. For example, to derive pixel positions, relative outputs are scaled against the original screenshot width WW and height HH.

All evaluations are deployed using vLLM with temperature set to zero. Generation token budgets are assigned according to task output requirements. Grounding benchmarks, ScreenSpot-Pro and OSWorld-G, use a limit of 128 tokens for single tool calls. AndroidControl receives a budget of 512 tokens to accommodate action descriptions and arguments. AgentNetBench uses a 1000 token limit, aligning with its official agent configuration.

Results

This section evaluates the effectiveness of the T2T corpus for GUI grounding, instruction-guided execution, and trajectory-level decision-making. T2T-8B denotes Qwen3-VL-8B-Instruct fine-tuned as described in § 3, while Base denotes the original checkpoint. As summarized in Table 2, a single epoch of fine-tuning on approximately 350K tutorial-derived samples improves performance across diverse benchmarks without using in-distribution training data. We further examine performance by target type, software domain, and action primitive, and test whether T2T supervision remains useful alongside in-domain data.

BenchmarkMetricBaseT2T-8BRel.Err. red.
OSWorld-Gaccuracy56.9161.52+8.1%10.7%
OSWorld-Gref_{\mathrm{ref}}accuracy65.9670.74+7.2%14.0%
ScreenSpot-Proaccuracy50.2857.87+15.1%15.3%
icon targets onlyaccuracy20.7030.30+46.4%12.1%
AndroidControl-Lowstep SR72.5780.36+10.7%28.4%
AndroidControl-Lowtype accuracy81.6091.79+12.5%55.4%
scroll steps onlystep SR19.0864.57+238%56.2%

Table 2. Summary of results. All benchmarks are out-of-distribution with respect to the training data. Rel. is the relative improvement over the base model. Err. red. is the fraction of the base model’s errors removed.

GUI Grounding

Overall Performance. As shown in Table 3, T2T-8B improves accuracy across all three grounding evaluations. Accuracy increases from 56.91% to 61.52% on OSWorld-G and from 65.96% to 70.74% on its refined-instruction variant, OSWorld-Gref_{\mathrm{ref}}. The largest absolute gain occurs on ScreenSpot-Pro, where accuracy rises from 50.28% to 57.87% (+7.59 percentage points). These results support the effectiveness of tutorial-derived trajectories for GUI grounding, particularly on visually demanding interfaces. A limitation remains on target-absent cases: T2T-8B correctly rejects none of the 54 such items in OSWorld-G, compared with three for Base. This is consistent with the exclusion of negative grounding examples from the fine-tuning mixture (§ 2.4). On the remaining 510 items, accuracy improves from 62.35% to 68.04%.

ModelOSWorld-G (564)OSWorld-Gref_{\mathrm{ref}} (564)ScreenSpot-Pro (1,581)
Base56.9165.9650.28
T2T-8B61.5270.7457.87
Δ\Delta+4.61+4.78+7.59

Table 3. GUI grounding accuracy (%). OSWorld-Gref_{\mathrm{ref}} is the refined-instruction variant: the same 564 items and identical target regions, with 507 of the instructions rewritten from functional intent into explicit appearance-and-position descriptions.

Performance by Target Type. Table 4 breaks down ScreenSpot-Pro performance by target type and software domain. T2T-8B improves all twelve domain-by-type subsets. Accuracy on text targets increases from 68.58% to 74.92% (+6.34 points), while icon-target accuracy rises from 20.70% to 30.30% (+9.60 points, or 46.4% relative). The larger gain on icons, which offer fewer explicit textual cues, suggests that T2T supervision is particularly useful for visually grounded element localization.

DomainnnText BaseText T2TIcon BaseIcon T2TAll BaseAll T2T
Office23077.4080.2332.0843.4066.9671.74
Scientific25478.4782.6426.3637.2755.9162.99
OS19671.0376.6419.1034.8347.4557.65
Dev29977.2780.5220.0029.6649.5055.85
CAD26148.2261.4215.6223.4440.2352.11
Creative34165.6672.7316.0820.9844.8751.03
All1,58168.5874.9220.7030.3050.2857.87
Δ\Delta+6.34+9.60+7.59

Table 4. ScreenSpot-Pro accuracy (%) by domain and target type. Every one of the twelve cells improves, as do all six domain totals and both target types.

Performance by Software Domain. Gains extend across all six domains, with the largest absolute improvements in CAD (+11.88 points) and OS interfaces (+10.20 points). In OS interfaces, icon accuracy increases from 19.10% to 34.83%. In CAD, both text targets (48.22% to 61.42%) and icon targets (15.62% to 23.44%) improve substantially. Together, these results highlight the value of tutorial-derived supervision for precise grounding in complex interfaces with dense controls and limited textual cues.

Instruction-guided Execution

Overall Performance. We next evaluate step-level execution on the official AndroidControl low-level test split, where the model receives a screenshot, task goal, action history, and step instruction and must generate the complete next tool call. This is an out-of-domain evaluation: no AndroidControl training examples are used for T2T-8B, and mobile data accounts for only approximately 10% of its fine-tuning mixture. Across 7,846 test steps (Table 5), step success rate increases from 72.57% to 80.36% (+7.79 points), while action-type accuracy rises from 81.60% to 91.79% (+10.19 points). These improvements remove 28.4% of baseline step errors and 55.4% of action-type errors. Gains also hold on unseen applications (+6.73 points), tasks (+6.06 points), and categories (+5.94 points), indicating transfer beyond the in-distribution split.

ModelType AccStep SRIDDAppunseen_{\text{unseen}}Taskunseen_{\text{unseen}}Cat.unseen_{\text{unseen}}
Base81.6072.5773.3371.5271.8670.99
T2T-8B91.7980.3683.0078.2577.9276.93
Δ\Delta+10.19+7.79+9.67+6.73+6.06+5.94

Table 5. AndroidControl, low-level setting, official test split (7,846 steps). Both models are evaluated under the same prompt template and the same element-box scoring criterion. IDD = in-distribution split.

Performance by Action Type. As shown in Table 6, the largest gain occurs on scroll, whose step success rate increases from 19.08% to 64.57% (+45.49 points). Scroll and swipe operations are well represented in tutorial videos (40,255 native scroll and 26,675 swipe steps; Table 11), providing supervision for an action that is difficult to annotate manually. Gains are also observed for open_app (+11.67 points) and input_text (+6.81 points). The long_press gain (+22.22 points) is promising but is measured on only nine examples. Click performance changes modestly (89.52% to 88.58%), whereas wait declines from 23.63% to 20.46%. The latter remains challenging because a single screenshot provides limited temporal evidence. Overall, the results point to better coverage of non-click action primitives without a large loss on the dominant click action.

ActionnnBaseT2T-8BΔ\Delta
scroll1,21119.0864.57+45.49
long_press966.6788.89+22.22
open_app60873.3685.03+11.67
input_text63288.1394.94+6.81
navigate_back34391.2592.42+1.17
click4,47689.5288.58−0.94
wait56723.6320.46−3.17

Table 6. AndroidControl-Low step success rate (%) by ground-truth action type.

Complementarity with In-domain Data

Experimental Setup. To test whether tutorial-derived supervision remains useful when target-domain data is available, we construct an approximately 6K-sample set from the AndroidControl training split using the conversion pipeline in § 2. We then perform one additional SFT epoch starting from either Base or T2T-8B. Because downward scrolls dominate the original data (78.5%), scroll examples are resampled to a more balanced direction distribution (45/25/18/12% for down/up/right/left; Section G). This setup helps distinguish improvements in choosing to scroll from improvements in predicting its direction.

Complementary Gains. Table 7 shows that T2T-8B, without AndroidControl training data, reaches 80.36% step success rate, compared with 72.57% for Base and 84.08% for Base fine-tuned on the 6K in-domain samples. Combining the two sources performs best: T2T-8B followed by in-domain fine-tuning reaches 86.01%, outperforming Base with the same in-domain fine-tuning by 1.93 points. Thus, tutorial-derived data contributes beyond what this modest amount of in-domain supervision provides.

ModelType AccStep SRscrollopen_appinput_textclick
Base81.6072.5719.0873.3688.1389.52
Base + 6K in-domain94.4784.0884.5688.4993.9983.71
T2T-8B91.7980.3664.5785.0394.9488.58
T2T-8B + 6K in-domain95.3286.0186.8788.9894.6286.15

Table 7. Complementarity with in-domain data, AndroidControl low-level setting. Same split and criterion as Table 5; the last four columns are step success rate by action type.

Scroll Direction Analysis. The additional in-domain epoch raises T2T-8B’s overall step success rate by 5.65 points, with notable gains on scroll and open_app. For scroll steps whose action type is already predicted correctly, direction accuracy rises from 67.9% to 87.8%. This indicates that the additional supervision improves the direction argument, rather than only the decision to scroll. The test split contains 78.2% downward scrolls, but T2T-8B initially predicts down on only 65.8% of test scroll steps. After the in-domain epoch, this prediction rate becomes 79.2%, close to the test distribution despite the more balanced training directions. This pattern is consistent with improved use of screen content to predict direction, although matching aggregate direction frequencies alone does not establish the underlying mechanism. Further distributions are provided in Section G.

Trajectory-level Evaluation

Evaluation Protocol. Finally, we evaluate action selection without step-level instructions on AgentNetBench, an offline desktop benchmark containing 100 trajectories and 1,208 steps. Both models receive one screenshot per step and the preceding action history as text. T2T-8B uses the desktop navigation template from its training pipeline, while Base uses the benchmark’s Qwen-VL agent prompt with its coordinate convention adapted to the same normalized 0–1000 space. The prompts are provided in Section F.5.

Overall and Per-action Results. T2T-8B improves the overall score from 63.56% to 65.23% (Table 8). The per-action breakdown in Table 9 shows higher scores on seven of ten reported action types. Particularly large differences appear for doubleClick (11.76% to 70.59%), terminate (0.00% to 21.65%), and moveTo (13.16% to 31.21%). These gains suggest broader support for GUI-specific operations, although some categories contain relatively few evaluation examples. Performance declines slightly on click and write, and more noticeably on hotkey. The modest aggregate improvement therefore indicates progress in action selection while leaving substantial room for stronger trajectory-level decision-making.

ModelScore
Base63.56
T2T-8B65.23

Table 8. AgentNetBench overall score (%).

ActionnnBaseT2T-8BΔ\Delta
doubleClick2011.7670.59+58.8
terminate970.0021.65+21.7
moveTo4513.1631.21+18.1
rightClick1714.2921.43+7.1
scroll184.6211.33+6.7
dragTo274.357.70+3.4
press2850.0052.63+2.6
click85074.9973.44−1.6
write13774.2472.51−1.7
hotkey3026.6716.67−10.0

Table 9. AgentNetBench per-action score (%) under the single-image protocol, ordered by margin. nn is the number of ground-truth actions of each type; a step may contain more than one action (e.g., moveTo followed by dragTo), so the counts sum to 1,269 over the 1,208 steps.

Conclusion

We present T2T, a fully automated pipeline for synthesizing GUI interaction trajectories from unannotated video tutorials across desktop, web, and mobile platforms. Through video filtering, action extraction, presence-aware spatial grounding, and action normalization, T2T produces over one million training samples without human annotation. Fine-tuning Qwen3-VL-8B on a subset of these samples yields consistent improvements across diverse GUI benchmarks without using in-distribution training data, demonstrating the potential of tutorial-derived trajectories as a scalable supervision source for generalist GUI agents.

Limitations. T2T dataset quality is assessed primarily through automated verification rather than systematic human annotation audits. Moreover, negative grounding examples are excluded from the current training mixture, limiting the model’s ability to abstain when targets are absent. Current evaluation focuses on step-level and offline trajectory-level benchmarks, leaving online task completion for future investigation. Mobile tutorial videos introduce visual challenges, including background clutter, occlusion, and perspective distortion in externally captured recordings.

Appendix

Video Filtering Parameters

Table 10 summarizes the thresholds used for video filtering and preprocessing (§ 2.2). All values were selected on a development sample and held fixed across source pools.

Action Space and Vocabulary

As described in § 2.4, action normalization preserves the extractor’s native outputs before mapping them to platform-specific tool calls. This appendix specifies the native vocabulary, the three tool schemas, and the observed mappings.

Stage 1: The Native Extractor Vocabulary

Table 11 reports the native extractor vocabulary, including synonym groups merged during normalization. The original action labels remain available in the trajectory logs.

Stage 2: The Platform-conditioned Tool-call Schemas

The schemas below define valid actions and arguments for each platform. “Grd” marks actions whose coordinates identify a spatial target rather than a scroll region or direction.

Direction convention. In mobile_use.scroll, direction denotes the direction of movement through content, not finger motion; down reveals content below the visible region. Desktop and web scrolling follow the same convention, with positive clicks moving toward the top. This convention is shared by the corpus, training prompts, and evaluation.

Native-to-tool-call Mapping

Table 14 summarizes native-to-schema conversion, including actions that could not be mapped to a valid tool call.

Per-Step Provenance Schema

Each reconstructed step records extraction, frame selection, grounding, and action-mapping decisions, as detailed in the table below. These provenance fields support filtering and auditing without rerunning the pipeline.

Table 15 | Recorded fields per step. Ext = written by the extraction model, Sel = observation-frame selector, Grd = grounder, Map = action-space mapper.

FieldByMeaning
action_typeExtNative verb (Table 11).
global_start/end_timeExtAction window over the source video.
timestamp_anchoredExtWhether the window was pinned to a detected event.
keyframes.start/end_frameExtPre-action and post-action frames.
transcripts.before/during/afterExtNarration windows around the action.
targetExtNatural-language name of the acted-on element.
drag_end_targetExtDestination name, drags only.
inner_monologue.thoughtExtFirst-person rationale, contemporaneous.
inner_monologue.action_descriptionExtOne-sentence imperative; becomes the <action> turn.
state_changeExtExpected post-condition.
parsed_actionsExtRaw parameters before normalization.
resized_width/heightExtFrame dimensions the coordinates are relative to.
observation_selection.methodSelPre-action frame, an earlier frame, or a cursor-guarded frame.
observation_selection.reasonSelWhy that frame was chosen.
observation_selection.t3_target_xySelCursor position used by the guard.
grounding.source, .toolGrdGrounder identity.
grounding.modeGrdpoint or bbox.
grounding.queryGrdThe phrase submitted to the grounder; equal to target unless rewritten.
grounding.qc.presentGrdFalse marks an absent verdict.
grounding.qc.confidenceGrdSelf-reported confidence of the grounder.
grounding.qc.frames_triedGrdNumber of frames attempted.
grounding.qc.escalatedGrdWhether a second (verification-path) attempt was made.
grounding.qc.verifiedGrdResult of the verification pass, where run.
grounding.point_norm/_pxGrdCoordinate, normalized and in pixels.
tool_call.action, .argumentsMapEmitted tool call.
tool_call.okMapWhether a defensible call could be produced.
tool_call.backfilledMapParameters recovered from a legacy field.
uncertainty_codesMapUncertainty codes raised during mapping.

Table 15. Recorded fields per step. Ext = written by the extraction model, Sel = observation-frame selector, Grd = grounder, Map = action-space mapper.

A compact per-step index retains the fields required for corpus statistics and can be joined to trajectories by video, task, and timestamp.

Corpus Samples

The following examples reproduce the stored training format. The first is a mobile navigation sample:

{
  "image": "-3msSQJ4_v8/task_0/step_01_start.jpg",
"width": 3840,
"height": 2160,
"conversations": [
    {"from": "human", "value":
        "<image>\nPlease generate the next move according to the UI screenshot,
        task and previous operations.\n\n
        Task: Update or open TikTok, browse sticker comments, and create a new
        live sticker from a video.\n\n
        Previous operations:\nStep 1: Scroll down on the comments list."},
    {"from": "gpt", "value":
        "<action>Scroll down on the comments list again.</action>\n
        <tool_call>\n
        {\"name\": \"mobile_use\", \"arguments\": {\"action\": \"scroll\",
        \"coordinate\": [498, 536], \"direction\": \"down\"}}\n
        </tool_call>"}
]
}

The second is a desktop grounding sample, without task-level context or action history:

{
    "image": "--LtGxJ8U4E/step_00_start.jpg",
    "width": 1920,
    "height": 1080,
    "conversations": [
        {"from": "human", "value":
            "<image>\nClick the '+ Create project' button in the large cyan banner."},
        {"from": "gpt", "value":
            "<tool_call>\n
            {\"name\": \"computer_use\", \"arguments\": {\"action\": \"left_click\",
            \"coordinate\": [578, 123]}}\n
            </tool_call>"}
    ]
}

Training Mixture

Table 16 details the training mixture used for supervised fine-tuning (§ 3).

Prompts

This appendix reproduces the pipeline prompts and their roles (Table 17). Training-time system prompts are generated from the same action-schema registry used for normalization.

StageModelPurposePrompt
FilteringCLIP (zero-shot)touch-surface gatingSection F.1
ExtractionGPT-5.5one call per video or windowSection F.2
GroundingQwen3.7-Plusfirst pass, no reasoningSection F.3
GroundingQwen3.7-Plusescalation, reasoning onSection F.3
GroundingQwen3.6-27B (local)crop verificationSection F.3
Training—tool schema in contextSection B

Table 17. Prompt roles across pipeline stages.

Touch-surface Gating

Mobile recordings are screened with a zero-shot CLIP classifier using eight sampled frames, two positive captions, and four negative captions. A frame counts as mobile UI when a positive caption scores above all negatives. Videos are retained when at least four frames pass and the transcript contains at least two instructional phrases, detected using a fixed imperative-pattern list.

Listing 1 | Positive and negative CLIP text prompts for mobile UI filtering.

# CLIP zero-shot text prompts (mobile_filter.py)
# a frame counts as UI when a positive scores above every negative

POSITIVE:
    a screen recording of a smartphone app
    a screenshot of a mobile phone user interface

NEGATIVE:
    a person talking to the camera
    a real-world outdoor scene
    a vlog of people
    a closeup of a product held in hands

Trajectory Extraction

The extraction model receives video metadata, transcript, and both frame streams in one call per video or temporal window. Desktop, web, and mobile prompts share task segmentation and action-extraction instructions but constrain outputs to their respective platforms. Additional rules discourage ambiguous target descriptions, incorrect drag endpoints, and unsupported scroll magnitudes.

Listing 2 | Trajectory extraction prompt for desktop tutorial videos.

You are an expert GUI-interaction annotator. You are given frames from a screen-recording tutorial
video, in chronological order. Two kinds of frames are provided:
- NARRATIVE frames (regular ~1s sampling, lower resolution) --- use them to follow the storyline:
  what application is used, what the user is trying to achieve, when one task ends and another
  begins.
- CANDIDATE frame pairs (BEFORE/AFTER around a detected screen change, higher resolution) --- use
  them to read fine details: exact button labels, typed text, menu items.
Every frame is labeled with its absolute video time "t=XX.XX". The video's speech transcript (if any)
is appended at the end --- narration often states typed text and keyboard shortcuts explicitly.

Your job:
  1. PLATFORM --- set platform_kind from what the FRAMES show, independently of any assumption: mobile = a phone/tablet touchscreen recording (no mouse cursor, usually portrait, system status bar at the top); web = a desktop web browser is the surface being operated (address bar / tab strip visible, the work happens on a page); desktop = a native desktop application is the surface (a browser may be visible in the background but is not what is being operated). Judge by the visible surface, not by what the software is capable of. Also fill platform with a short free-text description of the OS/environment you see.

  1. TASKS --- split the video into independent user task instances. Open a new task ONLY when the user's goal clearly switches (e.g. "create a pivot table" -> "export the sheet as PDF"). Side remarks and brief detours belong to the current task. Write each task's instruction as a natural-language command a user would give, and a short step plan.

  1. ACTIONS --- output every REAL GUI interaction the user performs, in time order, using EXACTLY these action types (desktop action space):

  • left_click / right_click / middle_click / double_click / triple_click: a mouse click at the target element.

  • left_click_drag: press the left button at target, drag to drag_end_target, release. Put target on the thing being grabbed --- the object, its handle, the slider knob, the edge being resized. drag_end_target must then say WHERE IT ENDS UP: a destination element or an empty slot ("the empty cell right of the chart", "the guide line at the 2-inch mark"). FOUR ways to get this wrong, all of which leave the end point undetermined: restating the same control ("the Saturation slider"); a bare direction ("rightward", "lower-right area"); describing the RESULT instead of the place ("the column boundary after widening", "a new point on the red curve", "the shapes centered farther apart") --- the question is WHERE the button is released, not what the screen looks like afterwards; and NAMING THE DESTINATION ONLY BY THE VALUE IT PRODUCES ("the Saturation slider at roughly 70%", "the knob position showing 2.86 s", "the timeline position at 00:02:29"). That last one is the most damaging: a rotary knob does not travel --- it turns in place, so the release point is wherever the mouse went and NO frame shows it; linear sliders and scrub bars fail the same way in practice. If the only way you can name the release point is by the reading it produces, then you cannot identify it --- DO NOT EMIT THE DRAG. Prefer anchoring the end to something else that is visible AT that spot (a tick mark, a label, a neighbouring element); a phrase that only positions it relative to the grabbed control ("the left end of that slider") often cannot be resolved. If the release point genuinely sits on blank space with nothing to anchor to, say so plainly ("empty canvas about one card-width right of the shape") rather than inventing a landmark. BOTH ends are REQUIRED --- a drag with only one end is not a valid action; if you cannot identify both, do not emit it.

  • mouse_move: move the cursor onto an element without clicking (e.g. to reveal a hover menu or tooltip). Only emit this when the hover itself visibly causes the change.

  • scroll: vertical scroll. hscroll: horizontal scroll. Provide target = A VISIBLE ELEMENT THE POINTER SITS OVER while scrolling --- a heading, a row, a card, a thumbnail inside the area whose content moves. NOT the page, pane or region itself ("About Us page", "the timeline area"): a page is not a point, the grounder correctly refuses to guess one, and the whole action becomes unusable. And clicks = the scroll AMOUNT measured off the frames: judge how far the CONTENT moved, as a fraction of the FULL SCREEN --- its height for scroll, its width for hscroll --- NOT of the panel, list or pane the scrolling happens inside; then set clicks to that fraction times 10 --- one full screen is 10, half a screen is 5, a fifth of a screen is 2. Give one decimal place when the shift is smaller than a tenth of the screen (a scroll inside a small panel is often 1.5 or 0.8). Never emit 0 (use 0.5 for the smallest visible shift). Do NOT try to count mouse-wheel notches: a notch is not visible in a recording and its size differs per device. The SIGN names the direction the CONTENT moves, not the wheel or finger: for scroll, positive scrolls up (back toward the top) and negative scrolls down (revealing content further down); for hscroll, positive reveals content further right.

  • type: type a string of text on the keyboard. Provide text and target (the field being typed into).

  • key: press keys. Provide keys --- a single key (e.g. ["Enter"]) or a combination pressed together (e.g. ["ctrl","s"]). There is no separate hotkey action; a combination is just a longer keys list.

  • wait: wait for the screen to change. No target.

IGNORE changes caused by video cuts, scene transitions, zoom/pan of the recording, overlays/captions, ads, webcam insets, or passive video playback --- those are NOT user actions.

  • timestamp: the time the action happens; ALWAYS reference a labeled frame time near the change.

  • target: a short description of the exact UI element acted on, for a grounding model. Base it STRICTLY on what is visible in the frames --- never on assumed screen layout or on why the action is done. Identify the element PRIMARILY by its visible label text, quoted exactly as shown (e.g. "the 'Export' button"); if it has no text, describe its icon shape and colour (e.g. "the gear-shaped settings icon"). Add a short positional cue (e.g. "in the top-right") ONLY when you are sure of it --- a wrong position hurts more than none. Do NOT include the element's purpose, the task goal, or any non-visual context. Three ways a target goes wrong, all of which leave the grounder pointing at the wrong pixel: (a) NAMING A CONTAINER instead of the control

inside it ("the slide navigation control", "the toolbar", "the right sidebar", "the timeline
area") --- a container has no single click point, so the grounder aims at its centre and lands
on whatever happens to sit there, usually body text. Name the SMALLEST thing whose activation
causes the state change: the arrow, the checkbox, the swatch, the X. Never offer two at once ("
checkbox/card", "button/area", "icon/menu") --- pick the smaller one. (b) SHORTENING ON REPEAT
--- when the same control is clicked several times in a row, write the SAME full description
every time; never write "again", "the same button", and never fall back to naming the container
it sits in. (c) NOT DISAMBIGUATING LOOK-ALIKES --- when several similar controls sit near each
other (a row of toolbar icons, one X per panel, a list of identical rows), say WHICH one, by the
row it sits on, the element beside it, or the label it belongs to ("the X on the 'Report.pptx'
row, below the toolbar icons"). REQUIRED for click/mouse_move/scroll/hscroll/drag and for type (
the input field). MUST be null for key/wait --- never invent a target for keyboard actions.
- `drag_end_target`: for left_click_drag only --- the drop location; REQUIRED there, null
otherwise.
- `text`: the typed string for type; null otherwise. `keys`: the key list for key; null otherwise.
`clicks`: the scroll amount for scroll/hscroll, as defined above; null otherwise.
- `action_description`: one first-person sentence ("I click ..."). `thought`: 1-2 first-person
sentences on why.
- `state_change`: one short sentence --- what visibly changed on screen after the action.
Only real, task-relevant actions. Every action's task_id must exist in tasks.
Listing 3

Figure 3 (PDF p. 28). Trajectory extraction prompt for web tutorial videos.

You are an expert WEB (desktop browser) GUI-interaction annotator. You are given frames from a screen
recording of a web-app / browser tutorial, in chronological order. Two kinds of frames are
provided:
- NARRATIVE frames (regular ~1s sampling, lower resolution) --- follow the storyline: which site/page
  , what the user is trying to achieve, when one task ends and another begins.
- CANDIDATE frame pairs (BEFORE/AFTER around a detected screen change, higher resolution) --- read
  fine details: exact button/link labels, typed text, menu items, URLs.
Every frame is labeled with its absolute video time "t=XX.Xs". The speech transcript (if any) is
appended at the end.

Your job:
0. PLATFORM --- set `platform_kind` from what the FRAMES show, independently of any assumption:
   `mobile` = a phone/tablet touchscreen recording (no mouse cursor, usually portrait, system status
   bar at the top); `web` = a desktop web browser is the surface being operated (address bar / tab
   strip visible, the work happens on a page); `desktop` = a native desktop application surface is the
   surface (a browser may be visible in the background but is not what is being operated). Judge by
   the visible surface, not by what the software is capable of. Also fill `platform` with a short
   free-text description of the OS/environment you see.
1. TASKS --- split the video into independent user task instances. Open a new task ONLY when the user
   's goal clearly switches. Write each task's `instruction` (natural-language command) and a short
   step `plan`.
2. ACTIONS --- output every REAL browser interaction the user performs, in time order, using EXACTLY
   these action types (web action space):
   - `left_click` / `right_click` / `middle_click` / `double_click` / `triple_click`: a mouse click on
     a button/link/element. Provide `target`.
   - `left_click_drag`: press the left button at `target`, drag to `drag_end_target`, release. Put
     `target` on the thing being grabbed --- the object, its handle, the slider knob, the edge being
     resized. `drag_end_target` must then say WHERE IT ENDS UP: a destination element or an empty slot
     ("the empty column right of the card", "the row above the header"). FOUR ways to get this wrong,
     all of which leave the end point undetermined: restating the same control ("the volume slider");
     a bare direction ("rightward", "lower-right area"); describing the RESULT instead of the place
     ("the column after widening", "the card moved up") --- the question is WHERE the button is
     released, not what the page looks like afterwards; and NAMING THE DESTINATION ONLY BY THE VALUE
     IT PRODUCES ("the volume slider at roughly 70%", "the knob position showing 2.86 s", "the
     timeline position at 00:02:29"). That last one is the most damaging: a rotary knob does not
     travel --- it turns in place, so the release point is wherever the mouse went and NO frame shows
     it; linear sliders and scrub bars fail the same way in practice. If the only way you can name
     the release point is by the reading it produces, then you cannot identify it --- DO NOT EMIT THE
     DRAG. Prefer anchoring the end to something else that is visible at that spot (a tick mark, a
     label, a neighbouring element); a phrase that only positions it relative to the grabbed control
     ("the left end of that slider") often cannot be resolved. If the release point genuinely sits on
     blank space with nothing to anchor to, say so plainly ("empty area about one card-width right
     of the tile") rather than inventing a landmark. BOTH ends are REQUIRED --- a drag with only one
     end is not a valid action; if you cannot identify both, do not emit it.

Figure and code content overlap? Image is a code listing, should not use figure container; I incorrectly added figure. Need body likely caption text then code listing. The image shows Listing 3 caption and shaded code. No figure markup; represent caption as paragraph maybe Listing 3 Trajectory... then code. Use body. Remove figure. Also first block likely continuation code. final.

- `mouse_move`: move the cursor onto an element without clicking (e.g. to open a hover menu). Only emit when the hover itself visibly causes the change.
- `scroll`: vertical scroll of the page/region. `hscroll`: horizontal scroll. Provide `target` = A VISIBLE ELEMENT THE POINTER SITS OVER while scrolling --- a heading, a row, a card, a thumbnail inside the area whose content moves. NOT the page, pane or region itself ("About Us page", "the results area", "the timeline area"): a page is not a point, the grounder correctly refuses to guess one, and the whole action is then unusable. And `clicks` = the scroll AMOUNT measured off the frames: judge how far the CONTENT moved, as a fraction of the FULL SCREEN --- its height for `scroll`, its width for `hscroll` --- NOT of the panel, list or pane the scrolling happens inside; then set `clicks` to that fraction times 10 --- one full screen is 10, half a screen is 5, a fifth of a screen is 2. Give one decimal place when the shift is smaller than a tenth of the screen (a scroll inside a small panel is often 1.5 or 0.8). Never emit 0 (use 0.5 for the smallest visible shift). Do NOT try to count mouse-wheel notches: a notch is not visible in a recording and its size differs per device. The SIGN names the direction the CONTENT moves, not the wheel or finger: for `scroll`, positive scrolls up (back toward the top) and negative scrolls down (revealing content further down); for `hscroll`, positive reveals content further right. There is no swipe action on web --- every scroll gesture is `scroll` or `hscroll`.
- `type`: type text into the focused field. Provide `text` and `target` (the input field).
- `select`: choose an option from a dropdown/`<select>`. Provide `target` (the dropdown) and `value` (the chosen option text).
- `open_url`: navigate to a URL (typed in the address bar). Provide `url`. No target.
- `go_back`: browser back button. `go_forward`: browser forward button. No fields.
- `key`: press keys. Provide `keys` --- a single key (e.g. ["Enter"]) or a combination pressed together (e.g. ["ctrl","t"]). There is no separate hotkey action; a combination is just a longer `keys` list. No target.
- `wait`: wait for the page to change. No target.
IGNORE changes from video cuts, transitions, zoom/pan of the recording, overlays/captions, ads, webcam insets, or passive playback --- those are NOT user actions.
Field rules per action:
- `timestamp`: the time the action happens; ALWAYS reference a labeled frame time near the change.
- `target`: for every click type / mouse_move / left_click_drag / scroll / hscroll / type / select --- a short, visually-grounded description by visible label text (quoted) or icon shape/colour.
Three ways a target goes wrong, all of which leave the grounder pointing at the wrong pixel: (a) NAMING A CONTAINER instead of the control inside it ("the pagination control", "the toolbar", "the right sidebar", "the results area") --- a container has no single click point, so the grounder aims at its centre and lands on whatever happens to sit there, usually body text. Name the SMALLEST thing whose activation causes the state change: the arrow, the checkbox, the Swatch, the X. Never offer two at once ("checkbox/card", "button/area", "icon/menu") --- pick the smaller one. (b) SHORTENING ON REPEAT --- when the same control is clicked several times in a row, write the SAME full description every time; never write "again", "the same button", and never fall back to naming the container it sits in. (c) NOT DISAMBIGUATING LOOK-ALIKES --- when several similar controls sit near each other (a row of toolbar icons, one X per card, a list of identical rows), say WHICH one, by the row it sits on, the element beside it, or the label it belongs to ("the checkbox on the WooCommerce card, top-right of that card"). MUST be null for open_url/back/go_forward/key/wait.
- `text` (type) / `url` (open_url) / `value` (select) / `keys` (key) / `clicks` (scroll, hscroll): as applicable, else null.
- `drag_end_target`: for left_click_drag only --- REQUIRED there, null otherwise.
- `action_description`: one first-person sentence. `thought`: 1-2 first-person sentences on why.
- `state_change`: one short sentence --- what visibly changed on screen after the action.
Only real, task-relevant actions. Every action's task_id must exist in tasks.

Listing 4 | Trajectory extraction prompt for mobile tutorial videos.

You are an expert MOBILE (touchscreen) GUI-interaction annotator. You are given frames from a screen-recording of a phone/tablet tutorial, in chronological order. Two kinds of frames are provided:
- NARRATIVE frames (regular ~1s sampling, lower resolution) --- follow the storyline: which app is used, what the user is trying to achieve, when one task ends and another begins.
- CANDIDATE frame pairs (BEFORE/AFTER around a detected screen change, higher resolution) --- read fine details: exact button labels, typed text, list items.
Every frame is labeled with its absolute video time "t=XX.Xs". The speech transcript (if any) is appended at the end.

Your job:
0. PLATFORM --- set `platform_kind` from what the FRAMES show, independently of any assumption: `mobile` = a phone/tablet touchscreen recording (no mouse cursor, usually portrait, system status bar at the top); `web` = a desktop web browser is the surface being operated (address bar / tab

strip visible, the work happens on a page); desktop = a native desktop application is the surface (a browser may be visible in the background but is not what is being operated). Judge by the visible surface, not by what the software is capable of. Also fill platform with a short free-text description of the OS/environment you see.

  1. TASKS --- split the video into independent user task instances. Open a new task ONLY when the user's goal clearly switches. Write each task's instruction as a natural-language command and a short step plan.

  1. ACTIONS --- output every REAL touchscreen interaction the user performs, in time order, using EXACTLY these action types (mobile action space):

  • click: a tap on an element. Provide target.

  • long_press: a press-and-hold on an element. Provide target and duration (seconds, default 1).

  • swipe: a swipe gesture. It drags whatever sits under the start point toward the end point, and covers TWO different intents --- decide which one you are seeing before describing the ends. (a) SCROLL / PAN the view: the user is moving a list, page or canvas, not an object. Put target anywhere inside the scrolled region that is not itself a draggable object --- an ordinary list row is fine, a dense list has no empty space ("the middle of the list", "over the third row of the feed"). swipe_to_target then encodes ONLY direction and distance --- it is NOT a destination element, so describe it as a screen position ("near the top of the list area"), never as a thing to land on. (b) MOVE AN OBJECT: the user is dragging something to a place. Put target on the object itself or its handle ("the selected cloud shape", "the corner handle of the text box", "the Opacity slider knob"). swipe_to_target must then name WHERE THE OBJECT ENDS UP --- a destination element or an empty slot ("the empty grid cell right of the cloud", "the row above the header"). FOUR ways to get this wrong, all of which leave the end point undetermined: restating the same control ("the Opacity slider"); a bare direction ("rightward", "lower-right area"); describing the RESULT instead of the place ("the shape moved up", "a new point on the curve") --- the question is WHERE the finger lifts off, not what the screen looks like afterwards; and NAMING THE DESTINATION ONLY BY THE VALUE IT PRODUCES ("the Opacity slider at roughly 40%", "the dial showing 2.86 s", "the timeline position at 00:02:29"). That last one is the most damaging: a rotary dial does not travel --- it turns in place, so the lift-off point is wherever the finger went and NO frame shows it; linear sliders and scrub bars fail the same way in practice. If the only way you can name the lift-off point is by the reading it produces, then you cannot identify it --- DO NOT EMIT THE SWIPE. Prefer anchoring the end to something else visible AT that spot (a tick, a label, a neighbouring element); a phrase that only positions it relative to the grabbed control ("the left end of that slider") often cannot be resolved. If it genuinely lands on blank space, say so plainly ("empty canvas about one tile-width right of the shape") rather than inventing a landmark.

Never take the direction from words like "scroll down" --- they name the reading direction, not the gesture. Fix it from evidence first: whatever is visible at your start point in the BEFORE frame must be the thing sitting at your end point in the AFTER frame; if that does not hold, your two ends are backwards. When you cannot check that --- no BEFORE/AFTER pair covers this moment, or the content travelled so far it left the screen --- fall back on the invariant it comes from: the content travels the same way the two points do. Revealing what was below the fold means the start point is LOWER on the screen than the end point; revealing what was above means the start point is HIGHER; the same holds left/right. This invariant is about case (a) only --- in case (b) the end point is fixed by the destination and may lie in any direction from the start. BOTH ends are REQUIRED --- a swipe with only one end is not a valid action; if you cannot identify both, do not emit swipe.

  • type: type text into the currently focused field. Provide text (the typed string). No target.

  • open: open an application from the home screen / app drawer. Provide app_name. No target.

  • system_button: press a hardware/system button. Provide button --- one of "Back", "Home", "Menu", "Enter". No target.

  • key: an ad hoc key event (e.g. "volume_up", "power"). Provide text. Use system_button for Back/Home/Menu/Enter instead. No target.

  • wait: wait for the screen to change. No target. IGNORE changes from video cuts, transitions, zoom/pan of the recording, overlays/captions, ads, webcam insets, or passive playback --- those are NOT user actions.

Field rules per action:

  • timestamp: the time the action happens; ALWAYS reference a labeled frame time near the change.

  • target: for click/long_press (REQUIRED) and swipe (the START point, REQUIRED) --- a short, visually-grounded description of the exact element, by its visible label text (quoted) or icon shape/colour. Three ways a target goes wrong, all of which leave the grounder pointing at the wrong pixel: (a) NAMING A CONTAINER instead of the control inside it ("the bottom navigation bar", "the toolbar", "the settings section", "the feed area") --- a container has no single tap point, so the grounder aims at its centre and lands on whatever happens to sit there, usually body text. Name the SMALLEST thing whose activation causes the state change: the tab icon, the

checkbox, the toggle, the X. Never offer two at once ("checkbox/row", "icon/tab") --- pick the smaller one. NOTE this rule does NOT apply to a `swipe` used to SCROLL: there the start point is deliberately a region, as described above. (b) SHORTENING ON REPEAT --- when the same control is tapped several times in a row, write the SAME full description every time; never write "again", "the same button", and never fall back to naming the container it sits in. (c) NOT DISAMBIGUATING LOOK-ALIKES --- when several similar controls sit near each other (a row of tab icons, one toggle per row, a list of identical cells), say WHICH one, by the row it sits on, the element beside it, or the label it belongs to ("the toggle on the 'Wi-Fi' row, right-hand side"). MUST be null for type/open/system_button/key/wait.
- `text`: the typed string for `type`, the key name for `key`; null otherwise. `app_name`: for `open`; null otherwise. `button`: for `system_button`; null otherwise.
- `swipe_to_target`: the END point --- REQUIRED for `swipe`, null otherwise. `duration`: for `long_press`; null otherwise.
- `action_description`: one first-person sentence ("I tap ..."). `thought`: 1-2 first-person sentences on why.
- `state_change`: one short sentence --- what visibly changed on screen after the action.
Only real, task-relevant actions. Every action's task_id must exist in tasks.

Spatial Grounding

Grounding uses a high-recall first pass and a more conservative reasoning-enabled escalation pass (§ 2.3.4). The latter explicitly allows target absence rather than inferring coordinates from layout priors. A separate local model verifies uncertain predictions using a crop around the proposed target.

Listing 5 | Initial spatial grounding prompt for point-coordinate prediction.

You are a GUI grounding model. Locate the target described below in the screenshot.
TARGET: {query}
Respond with ONLY a JSON object and nothing else:
{{"present": <true|false>, "point": [x, y], "confidence": <0..1>}}
- present: set FALSE only when the target is genuinely NOT visible anywhere in this screenshot. If a plausible match exists --- even a small icon or a partially described element --- prefer present =true and give your best-guess point.
- point: the ABSOLUTE PIXEL coordinates [x, y] (integers) of the CENTER of the target in the given image; use [0, 0] only when not present.
- confidence: your confidence in [0, 1].

Listing 6 | Initial spatial grounding prompt for bounding-box prediction of text-entry targets.

You are a GUI grounding model. Locate the target described below in the screenshot.
TARGET: {query}
Respond with ONLY a JSON object and nothing else:
{{"present": <true|false>, "box": [x1, y1, x2, y2], "confidence": <0..1>}}
- present: true if the target is visible in this screenshot, false if it is not.
- box: the ABSOLUTE PIXEL bounding box [x1, y1, x2, y2] (top-left, bottom-right; integers) tightly enclosing the target element; use [0,0,0,0] when not present.
- confidence: your confidence in [0, 1].

Listing 7 | Reasoning-enabled spatial grounding prompt for point-coordinate verification.

You are a GUI grounding model. Locate the target described below in the screenshot.
TARGET: {query}
Respond with ONLY a JSON object and nothing else:
{{"present": <true|false>, "point": [x, y], "confidence": <0..1>}}
- present: true only if the target is actually visible in THIS screenshot. A transient element (an open menu, a dropdown, a dialog) may have already closed --- answer false then. Do NOT infer a location from where such an element usually lives, and do not guess when unsure: false is a normal, expected answer.
- point: the ABSOLUTE PIXEL coordinates [x, y] (integers) of the CENTER of the target in the given image; use [0, 0] only when not present.
- confidence: your confidence in [0, 1].

Listing 8 | Reasoning-enabled spatial grounding prompt for bounding-box verification.

You are a GUI grounding model. Locate the target described below in the screenshot.
TARGET: {query}
Respond with ONLY a JSON object and nothing else:
{{"present": <true|false>, "box": [x1, y1, x2, y2], "confidence": <0..1>}}
- present: true only if the target is actually visible in THIS screenshot; answer false when it is
    not there. Do not guess when unsure.
- box: the ABSOLUTE PIXEL bounding box [x1, y1, x2, y2] (top-left, bottom-right; integers) tightly
    enclosing the target element; use [0,0,0,0] when not present.
- confidence: your confidence in [0, 1].

Listing 9 | Local crop-verification prompt for checking target presence.

Does this cropped screenshot region show the following UI target: "{query}"? Answer with ONLY "yes"
    or "no".

Training-time System Prompts

Training-time tool definitions are rendered from the action schema registry (§ 2.4), keeping prompts consistent with normalized tool calls. Evaluation uses the same rendering, and each result records a SHA-256 prompt hash for reproducibility.

Evaluation Prompts

The evaluation prompts are reproduced from the evaluation code (§ 3 and § 4.4). Table 18 identifies the prompt used by each model and benchmark. Braced fields are filled per example, and <image> marks screenshot insertion.

BenchmarkModelPrompt
OSWorld-G, ScreenSpot-ProBaseQwen3-VL cookbook computer use (Listing 10)
OSWorld-G, ScreenSpot-ProT2T-8BT2T desktop grounding (Listing 11)
AndroidControlBase, T2T-8BT2T mobile navigation (Listing 12)
AgentNetBenchBaseAgentNetBench Qwen-VL agent (Listing 13)
AgentNetBenchT2T-8BT2T desktop navigation (Listing 14)

Table 18. Evaluation prompt used for each benchmark and model.

Grounding. Base uses the official Qwen3-VL cookbook computer-use prompt with a 1000 ×\times 1000 display convention; T2T-8B uses its training-time grounding template.

Listing 10 | Evaluation prompt for the base model on GUI grounding benchmarks, adapted from the official Qwen3-VL computer-use cookbook.

[system]
You are a helpful assistant.

# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "computer_use", "description": "Use a mouse and keyboard to
    interact with a computer, and take screenshots.\n* This is an interface to a desktop GUI. You
    do not have access to a terminal or applications menu. You must click on desktop icons to start
    applications.\n* Some applications may take time to start or process actions, so you may need to
    wait and take successive screenshots to see the results of your actions. E.g. if you click on
Firefox and a window doesn't open, try wait and taking another screenshot.\n* The screen's resolution is 1000x1000.\n* Whenever you intend to move the cursor to click on an element like an icon, you should consult a screenshot to determine the coordinates of the element before moving the cursor.\n* If you tried clicking on a program or link but it failed to load, even after waiting, try adjusting your cursor position so that the tip of the cursor visually falls on the element that you want to click.\n* Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don't click boxes on their edges.", "parameters": {"properties": {"action": {"description": "The action to perform. The available actions are:\n* `key`: Performs key down presses on the arguments passed in order, then performs key releases in reverse order.\n* `type`: Type a string of text on the keyboard.\n* `mouse_move`: Move the cursor to a specified (x, y) pixel coordinate on the screen.\n* `left_click`: Click the left mouse button at a specified (x, y) pixel coordinate on the screen.\n* `left_click_drag`: Click and drag the cursor to a specified (x, y) pixel coordinate on the screen.\n* `right_click`: Click the right mouse button at a specified (x, y) pixel coordinate on the screen.\n* `middle_click`: Click the middle mouse button at a specified (x, y) pixel coordinate on the screen.\n* `double_click`: Double-click the left mouse button at a specified (x, y) pixel coordinate on the screen.\n* `triple_click`: Triple-click the left mouse button at a specified (x, y) pixel coordinate on the screen (simulated as double-click since it's the closest action).\n* `scroll`: Performs a scroll of the mouse scroll wheel.\n* `hscroll`: Performs a horizontal scroll (mapped to regular scroll).\n* `wait`: Wait specified seconds for the change to happen.\n* `terminate`: Terminate the current task and report its completion status.\n* `answer`: Answer a question.", "enum": ["key", "type", "mouse_move", "left_click", "left_click_drag", "right_click", "middle_click", "double_click", "triple_click", "scroll", "hscroll", "wait", "terminate", "answer"], "type": "string"}, "keys": {"description": "Required only by `action=key`.", "type": "array"}, "text": {"description": "Required only by `action=type` and `action=answer`.", "type": "string"}, "coordinate": {"description": "(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to.", "type": "array"}, "pixels": {"description": "The amount of scrolling to perform. Positive values scroll up, negative values scroll down. Required only by `action=scroll` and `action=hscroll`.", "type": "number"}, "time": {"description": "The seconds to wait. Required only by `action=wait`.", "type": "number"}, "status": {"description": "The status of the task. Required only by `action=terminate`.", "type": "string", "enum": ["success", "failure"]}}, "required": ["action"], "type": "object"}}
</tools>

For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call>

[user]
<image>{instruction}

Listing 11 | Evaluation prompt for T2T-8B on GUI grounding benchmarks, using the T2T desktop grounding template.

[system]
You are an autonomous GUI agent capable of operating on desktops, mobile devices, and web browsers.
    Your primary function is to analyze a screen capture, locate the target UI element, and issue
    the single appropriate action that fulfills the given instruction.

# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "computer_use", "description": "Use a mouse and keyboard to
    interact with a computer, and take screenshots.\n* This is an interface to a desktop GUI. You
    do not have access to a terminal or applications menu. You must click on desktop icons to start
    applications.\n* Some applications may take time to start or process actions, so you may need to
    wait and take successive screenshots to see the results of your actions. E.g. if you click
    Firefox and a window doesn't open, try wait and taking another screenshot.\n* The screen's
    resolution is 1000x1000.\n* Make sure to click any buttons, links, icons, etc with the cursor
    tip in the center of the element. Don't click boxes on their edges unless asked.", "parameters":
    {"properties": {"action": {"description": "The action to perform. The available actions are:\n* `
    mouse_move`: Move the cursor to a specified (x, y) pixel coordinate on the screen.\n* `left_click
`: Click the left mouse button at a specified (x, y) pixel coordinate on the screen.\n* `
left_click_drag`: Press the left mouse button at the start point with coordinate (x, y), drag to
the end point with coordinate2 (x2, y2), and release.\n* `right_click`: Click the right mouse
button at a specified (x, y) pixel coordinate on the screen.\n* `middle_click`: Click the middle
mouse button at a specified (x, y) pixel coordinate on the screen.\n* `double_click`: Double-
click the left mouse button at a specified (x, y) pixel coordinate on the screen.\n* `
triple_click`: Triple-click the left mouse button at a specified (x, y) pixel coordinate on the
screen.", "enum": ["mouse_move", "left_click", "left_click_drag", "right_click", "middle_click",
"double_click", "triple_click"], "type": "string"}, "coordinate": {"description": "(x, y): The x
(pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to.
Required only by `action=mouse_move`, `action=left_click`, `action=left_click_drag`, `action=
right_click`, `action=middle_click`, `action=double_click`, and `action=triple_click`.", "type":
"array"}, "coordinate2": {"description": "(x, y): The x (pixels from the left edge) and y (
pixels from the top edge) coordinates to drag the cursor to. Required only by `action=
left_click_drag`.", "type": "array"}}, "required": ["action"], "type": "object"}}}
</tools>
For each function call, return a json object with function name and arguments within <tool_call></
tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call>
## Input Specification
- A screenshot of the current screen + an instruction describing the action to perform.

## Output Format
- Emit exactly one <tool_call></tool_call> block containing a single JSON object with the function
name and its arguments.
- Coordinates are integers in a 1000x1000 space (x measured from the left edge, y from the top edge).

## Note
- Choose exactly one function whose name and arguments are defined in the tool schema above.
- Do not output anything other than the <tool_call></tool_call> block.
[user]
<image>
{instruction}

AndroidControl. Both models use the T2T mobile navigation template, with an added Current step field for the low-level instruction. The alternative template discussed in § 3 additionally requires a <think> block before the <action> line.

Listing 12 | Shared evaluation prompt for both models on AndroidControl, using the T2T mobile navigation template.

[system]
You are an autonomous GUI agent operating on the **Mobile Phone & Tablet** platform(s). Your primary
function is to analyze screen captures and perform appropriate UI actions to complete assigned
tasks.

# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "mobile_use", "description": "Use a touchscreen to interact
with a mobile device, and take screenshots.\n* This is an interface to a mobile device with
touchscreen. You can perform actions like clicking, typing, swiping, etc.\n* Some applications
may take time to start or process actions, so you may need to wait and take successive
screenshots to see the results of your actions.\n* The screen's resolution is 1000x1000.\n* Make
sure to click any buttons, links, icons, etc with the cursor tip in the center of the element.
Don't click boxes on their edges unless asked.", "parameters": {"properties": {"action": {
"description": "The action to perform. The available actions are:\n* `key`: Perform a key event on
the mobile device.\n  - This supports adb's `keyevent` syntax.\n  - Examples: \"volume_up\",
\"volume_down\", \"power\", \"camera\", \"clear\".
\n* `click`: Click the point on the screen with coordinate (x, y).\n* `long_press`: Press the point
on the screen with coordinate (x, y) for specified seconds.\n* `scroll`: Scroll the view at
coordinate (x, y) in the given `direction`. `direction` names the direction THE VIEW travels over
the content, never a finger or wheel motion: `down` advances downward through the content,
revealing what was below the fold; `up` goes back toward the top, revealing what was above;
`right` and `left` work the same way on the horizontal axis. This action does not describe a
gesture --- do not think in terms of which way a finger moves. Put (x, y) anywhere inside the
scrollable region that should move, which matters when a horizontally scrolling row sits inside
a vertically scrolling page. `amount` is optional --- it is how far to scroll as a fraction of
the screen, and defaults to 0.5 when omitted.\n* `swipe`: Drag whatever sits under coordinate (x,
y) to coordinate2 (x2, y2). Use this ONLY to move an object or a control: put (x, y) on the
object or its handle, and put (x2, y2) exactly where it must end up --- a destination slot,
another element, or the position on a track that encodes the wanted value. (x2, y2) is decided by
the destination alone and may lie in any direction from (x, y). To scroll or pan a view, use
`scroll` instead; this action never means scrolling.
\n* `type`: Input the specified text into the activated input box.\n* `system_button`: Press the
system button.\n* `open`: Open an app on the device.\n* `wait`: Wait specified seconds for the
change to happen.\n* `answer`: Terminate the current task and output the answer.\n* `interact`:
Resolve the blocking window by interacting with the user.\n* `terminate`: Terminate the current
task and report its completion status.", "enum": ["key", "click", "long_press", "scroll", "swipe
", "type", "system_button", "open", "wait", "answer", "interact", "terminate"], "type": "string
"}, "text": {"description": "Required only by `action=key`, `action=type`, `action=open`, `action
=answer`, and `action=interact`.", "type": "string"}, "coordinate": {"description": "(x, y): the
point on the screen the action acts on --- the element to touch, the region to scroll, or the
object to start dragging. x is measured from the left edge and y from the top edge, both on a
0-1000 scale, so y gets larger toward the bottom of the screen. Required only by `action=click`,
`action=long_press`, `action=scroll`, and `action=swipe`.", "type": "array"}, "type": "array"}, "time": {
"description": "The seconds to wait. Required only by `action=long_press` and `action=wait`.", "
type": "number"},
"direction": {"description": "The direction THE VIEW travels over the content, not a finger direction.
Required only by `action=scroll`.", "type": "string", "enum": ["up", "down", "left", "right"]},
"amount": {"description": "How far to scroll along that axis, as a fraction of the screen. 0.5
moves the view by about half a screen. Optional for `action=scroll`; it is never required. Omit
it to use the default 0.5.", "type": "number"}, "coordinate2": {"description": "(x2, y2):
where the dragged object must end up, on the same axes as coordinate. It is a destination, not a
direction. Required only by `action=swipe`.", "type": "array"}, "button": {"description": "Back
means returning to the previous interface, Home means returning to the desktop, Menu means
opening the application background menu, and Enter means pressing the enter. Required only by
`action=system_button`.", "type": "string", "enum": ["Back", "Home", "Menu", "Enter"]}, "status":
{"description": "The status of the task. Required only by `action=terminate`.", "type": "string",
"enum": ["success", "failure"]}], "required": ["action"], "type": "object"}}
</tools>

For each function call, return a json object with function name and arguments within <tool_call></
tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call>

## Input Specification
- A screenshot of the current screen + the task description + your past interaction history with the
  UI.

## Output Format
Response format for every step:
1) <action>...</action>: a short imperative sentence describing what to do in the UI.
2) A single <tool_call></tool_call> block containing one JSON object with the function name and its
   arguments.

## Note
- Output exactly in the order: the <action></action> block first, then the <tool_call></tool_call>
  block.
- The imperative sentence describing the action must be enclosed in <action></action> tags.
- Be brief: one sentence inside <action></action>.
- Coordinates are integers in a 1000x1000 space (x measured from the left edge, y from the top edge).
- The function call must use a function and arguments defined in the tool schema above.
- When the task is complete, call `terminate`; use `answer` when a textual answer must be returned.

[user]
<image>
Please generate the next move according to the UI screenshot, task and previous operations.

Task: {goal}

Current step: {step instruction}

Previous operations:
Step 1: {previous action}
...

AgentNetBench. Base uses the benchmark’s Qwen-VL agent prompt and Step {n}: Thought: prefill. T2T-8B uses its desktop navigation template. Both receive one current screenshot and textual action history.

Listing 13 | Evaluation prompt for the base model on AgentNetBench, using the benchmark’s official Qwen-VL agent template.

[system]
You are a helpful assistant

# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name_for_human": "computer_use", "name": "computer_use", "description": "Use a mouse and keyboard to interact with a computer, and take screenshots.\n* This is an interface to a desktop GUI. You do not have access to a terminal or applications menu.\n* You must click on desktop icons to start applications.\n* Some applications may take time to start or process actions, so you may need to wait and take successive screenshots to see the results of your actions. E.g. if you click on Firefox and a window doesn't open, try wait and taking another screenshot.\n* The screen's resolution is 1000x1000.\n* Whenever you intend to move the cursor to click on an element like an icon, you should consult a screenshot to determine the coordinates of the element before moving the cursor.\n* If you tried clicking on a program or link but it failed to load, even after waiting, try adjusting your cursor position so that the tip of the cursor visually falls on the element that you want to click.\n* Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don't click boxes on their edges unless asked.", "parameters": {"properties": {"action": {"description": "The action to perform. The available actions are:\n* `key`: Performs key down presses on the arguments passed in order, then performs key releases in reverse order.\n* `type`: Type a string of text on the keyboard.\n* `mouse_move`: Move the cursor to a specified (x, y) pixel coordinate on the screen.\n* `left_click`: Click the left mouse button.\n* `left_click_drag`: Click and drag the cursor to a specified (x, y) pixel coordinate on the screen.\n* `right_click`: Click the right mouse button.\n* `middle_click`: Click the middle mouse button.\n* `double_click`: Double-click the left mouse button.\n* `scroll`: Performs a scroll of the mouse scroll wheel.\n* `wait`: Wait specified seconds for the change to happen.\n* `terminate`: Terminate the current task and report its completion status.", "enum": ["key", "type", "mouse_move", "left_click", "left_click_drag", "right_click", "middle_click", "double_click", "scroll", "wait", "terminate"], "type": "string"}, "keys": {"description": "Required only by `action=key`.", "type": "array"}, "text": {"description": "Required only by `action=type`.", "type": "string"}, "coordinate": {"description": "(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to. Required only by `action=mouse_move` and `action=left_click_drag`.", "type": "array", "items": {"type": "number"}}, "pixels": {"description": "The amount of scrolling to perform. Positive values scroll up, negative values scroll down. Required only by `action=scroll`.", "type": "number"}, "time": {"description": "The seconds to wait. Required only by `action=wait`.", "type": "number"}, "status": {"description": "The status of the task. Required only by `action=terminate`.", "type": "string", "enum": ["success", "failure"]}}, "required": ["action"], "type": "object"}, "args_format": "Format the arguments as a JSON object."}
</tools>

For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags: <tool_call> {"name": <function-name>, "arguments": <args-json-object>} </tool_call>

[user] <image>

Please generate the next move according to the UI screenshot, instruction and previous actions.

Instruction: {task}

Previous actions: Step 1: {previous low-level instruction} ...

[assistant, prefilled] Step {n}: Thought:

Listing 14 |Evaluation prompt for T2T-8B on AgentNetBench, using the T2T desktop navigation template.

[system]
You are an autonomous GUI agent operating on the **desktop (Windows/Linux/macOS)** platform(s). Your
primary function is to analyze screen captures and perform appropriate UI actions to complete
assigned tasks.

# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "computer_use", "description": "Use a mouse and keyboard to
interact with a computer, and take screenshots.\n* This is an interface to a desktop GUI. You
do not have access to a terminal or applications menu. You must click on desktop icons to start
applications.\n* Some applications may take time to start or process actions, so you may need to
wait and take successive screenshots to see the results of your actions. E.g. if you click on
Firefox and a window doesn't open, try wait and taking another screenshot.\n* The screen's
resolution is 1000x1000.\n* Make sure to click any buttons, links, icons, etc with the cursor
tip in the center of the element. Don't click boxes on their edges unless asked.", "parameters":
{"properties": {"action": {"description": "The action to perform. The available actions are:\n* `
key`: Performs key down presses on the arguments passed in order, then performs key releases in
reverse order.\n* `type`: Type a string of text on the keyboard.\n* `mouse_move`: Move the cursor to a specified (x, y) pixel coordinate on the screen.\n* `left_click`: Click the left mouse button at specified (x, y) pixel coordinate on the screen.\n* `left_click_drag`: Press the left mouse button at the start point with coordinate2 (x2, y2), drag to the end point with coordinate2 (x2, y2), and release.", "type": "string"}, "coordinate": {"description": "(x, y) pixel coordinate on the screen.", "type": "array"}, "coordinate2": {"description": "(x2, y2) pixel coordinate on the screen. Required only by `action=left_click_drag`.", "type": "array"}}}}
"clicks": {"description": "The number of scroll clicks to perform. For `action=scroll`, positive values scroll up (back toward the top) and negative values scroll down (revealing content further down). For `action=hscroll`, positive values scroll right (revealing content further right) and negative values scroll left. The sign always names the direction THE VIEW travels over the content, never the direction the wheel or finger travels. Required only by `action=scroll` and `action=hscroll`.", "type": "number"}, "time": {"description": "The seconds to wait. Required only by `action=wait`.", "type": "number"}, "status": {"description": "The status of the task. Required only by `action=terminate`.", "type": "string", "enum": ["success", "failure"]}}, "required": ["action"], "type": "object"}}}
</tools>
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call>
## Input Specification
- A screenshot of the current screen + the task description + your past interaction history with the UI.

## Output Format
Response format for every step:
1) <action>...</action>: a short imperative sentence describing what to do in the UI.
2) A single <tool_call></tool_call> block containing one JSON object with the function name and its arguments.

Scroll-Direction Statistics

We report the scroll-direction distributions used in the in-domain experiment (§ 4.3). AndroidControl training scrolls are distributed 78.5/10.6/8.8/2.1% (down/up/right/left); the additional training subset is rebalanced to 45/25/18/12% without changing the proportion of scroll actions. In the test split, 1,211 scroll steps follow a 78.2/7.9/7.8/6.1% distribution. Among steps correctly identified as scrolls, direction accuracy increases from 67.9% to 87.8% after in-domain fine-tuning. The predicted share of down rises from 65.8% to 79.2%, close to the test share of 78.2%.

References

  1. [2]Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, et al. Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling. arXiv preprint arXiv:2412.05271, 2024.arxiv.org/abs/2412.05271
  2. [3]Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
  3. [4]Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents. In ICLR, 2025.arxiv.org/abs/2410.05243
  4. [5]Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use. arXiv preprint arXiv:2504.07981, 2025.arxiv.org/abs/2504.07981
  5. [6]Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the Effects of Data Scale on UI Control Agents. Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2406.03679
  6. [7]Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Stan Wexian Lei, Lijuan Wang, and Mike Zheng Shou. ShowUI: One Vision-Language-Action Model for GUI Visual Agent. In CVPR, 2025.arxiv.org/abs/2411.17465
  7. [8]Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, Iat Long Iong, Jiadai Sun, Jiaqi Wang, et al. AutoGLM: Autonomous Foundation Agents for GUIs. arXiv preprint arXiv:2411.00820, 2024.arxiv.org/abs/2411.00820
  8. [9]Zhaoyang Liu, Jingjing Xie, Zichen Ding, Zehao Li, Bowen Yang, Zhenyu Wu, Xuehui Wang, Qiushi Sun, Shi Liu, Weiyun Wang, et al. ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data. In ICLR, 2026.arxiv.org/abs/2509.15221
  9. [10]Dunjie Lu, Yiheng Xu, Junli Wang, Haoyuan Wu, Xinyuan Wang, Zekun Wang, Junlin Yang, Hongjin Su, Jixuan Chen, Junda Chen, Yuchen Mao, Jingren Zhou, Junyang Lin, Binyuan Hui, and Tao Yu. VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos. arXiv preprint arXiv:2510.19488, 2025.
  10. [11]Zhengzi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Pengxiang Zhao, Guangyi Liu, et al. UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning. In AAAI, 2026.arxiv.org/abs/2503.21620
  11. [12]Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, HanJia Lyu, Junda Wu, Ryan Aponte, Yu Xia, et al. GUI Agents: A Survey. In ACL Findings, 2025.
  12. [13]OpenAI. GPT-4o System Card. arXiv preprint arXiv:2410.21276, 2024.
  13. [14]Qwen Team. Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923, 2025a.
  14. [15]Qwen Team. Qwen3-VL Technical Report. arXiv preprint arXiv:2511.21631, 2025b.
  15. [16]Chan Hee Song, Yiwen Song, Palash Goyal, Yu Su, Oriana Riva, Hamid Palangi, and Tomas Pfister. Watch and Learn: Learning to Use Computers from Online Videos. In CVPR, 2026.arxiv.org/abs/2510.04673
  16. [17]Shuai Wang, Weiwen Liu, Jingxuan Chen, Yuqi Zhou, Weinan Gan, Xingshan Zeng, Yuhan Che, Shuai Yu, Xinlong Hao, Kun Shao, et al. GUI Agents with Foundation Models: A Comprehensive Survey. arXiv preprint arXiv:2411.04890, 2024.arxiv.org/abs/2411.04890
  17. [18]Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, et al. OpenCUA: Open Foundations for Computer-Use Agents. arXiv preprint arXiv:2508.09123, 2025.arxiv.org/abs/2508.09123
  18. [19]Qingsong Xie, Jiahao Tian, Jiacheng Tang, Zhifeng Li, Madalina Ciobanu, Navan Preet Singh, Qingqing Mao, and Ritankar Das. GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning. arXiv preprint arXiv:2610.19100, 2026a.arxiv.org/abs/2610.19100
  19. [20]Rui Xie, Zhi Gao, Chenrui Shi, Zirui Shang, Lu Chen, and Qing Li. GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotation. arXiv preprint arXiv:2603.26266, 2026b.arxiv.org/abs/2603.26266
  20. [21]Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, et al. Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis. arXiv preprint arXiv:2505.13227, 2025.arxiv.org/abs/2505.13227
  21. [22]Weimin Xiong, Shuhao Gu, Bowen Ye, Zihao Yue, Lei Li, Feifan Song, Sujian Li, and Hao Tian. Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining. In Proceedings of the 43rd International Conference on Machine Learning, 2026.arxiv.org/abs/2605.14747
  22. [23]Yifan Xu, Xiao Liu, Xinghan Liu, Jiaqi Fu, Hanchen Zhang, Bohao Jing, Shudan Zhang, Yuting Wang, Wenyi Zhao, and Yuxiao Dong. MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agents. In ICLR, 2026.arxiv.org/abs/2509.18119
  23. [24]Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction. In ICML, 2025.arxiv.org/abs/2412.04454
  24. [25]Bofei Zhang, Zirui Shang, Zhi Gao, Wang Zhang, Rui Xie, Xiaojian Ma, Tao Yuan, Xinxiao Wu, Song-Chun Zhu, and Qing Li. TongUI: Building Generalized GUI Agents by Learning from Multimodal Web Tutorials. arXiv preprint arXiv:2504.12679, 2025.

Paper details

vlnc:W_ypt73gw2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents