GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents

Introduction

A mobile graphical user interface (GUI) agent’s per-step cost is dominated by a 7B+ vision-language model (VLM) decoding its action token by token. On one NVIDIA H20, a Qwen2.5-VL-7B agent ([4]) LoRA-fine-tuned ([17]) on AndroidControl needs 207 ms per step under vLLM ([20]) and 507 ms under Hugging Face (HF) transformers to emit a JSON action of ∼12 tokens (median over 100 real steps, batch 1; Table 3), and 8B-class agents average 2.6–10.6 s

System	Serving	p50	p90	Mem.	× ours
7B SFT, merged	HF	506.7	588.9	15.9	6.8/9.8
7B SFT, merged	vLLM	206.7	247.9	≥16	2.8/4.0
GUI-JEPA, fp32	PyTorch	74.3	127.8	3.45	1
GUI-JEPA, bf16	PyTorch	51.5	79.7	2.48	1

per step in a live-device benchmark ((Sui et al., 2026)). Yet in GUI grounding a VLM fixes its target elements during prefill, with decoding only refining coordinates ([24]), and decoding on the critical path makes agents correct but late ([12]). Could a GUI agent score its actions in the latent space of a joint-embedding predictive architecture (JEPA; [1]), without decoding them at all?

GUI world models generate pixels, code or text and serve an LLM policy as simulator or verifier ([29, 31, 6] (Zheng et al. 2026; Xu et al. 2026b)), or as a pretraining target ([27]). On GUIs, JEPAs have so far served understanding and verification: UI-JEPA describes user intent through an LLM decoder ([14]), LGWM verifies GUI transitions ((Zhang et al., 2026a)), and next-frame feature prediction regularises the training of a VLM agent ((Yang et al., 2026b)). Candidate scorers over the accessibility tree (a11y) keep an LLM or VLM in the loop ([11 Zhang,] 2026; Christianos et al., [9]), and Agentic-JEPA, which acts with a JEPA, is text-only with 0% out-of-distribution (OOD) success (Rodrigo-Ginés, 2026). To our knowledge, no GUI agent has yet used a JEPA as the policy that selects its actions.

We bring JEPAs from perception and robotics to GUI control, moving action selection from the decoder into the latent space of a frozen JEPA encoder. GUI-JEPA encodes the screenshot once with a JEPA ViT-L pretrained on GUI screenshots, transforms that latent once per candidate (a11y element or global action) with a 39.9M residual predictor conditioned on the candidate’s action and region of interest (ROI), scores the candidate-conditioned latents with a cosine structural head, adds a 33M text-matcher score and takes the argmax, all in one batched pass (Figure 2). As in earlier candidate-scoring agents, the candidates come from the a11y tree, but GUI-JEPA never generates a token, so the cost of a step depends on how many candidates it scores.

Under the same inputs, judge and GPU as a Qwen2.5-VL-7B agent that we fine-tune with LoRA (SFT), GUI-JEPA is $2.8$–$4.0\times$ faster and reaches 92.4% / 86.1% of its in-distribution (IDD) Low / High accuracy with 6.0% of its parameters and $4.6$–$6.4\times$ less GPU memory (Figure 1).

Our contributions are:

• GUI-JEPA, the first GUI agent whose policy is built on a JEPA. It selects actions by scoring candidate-conditioned JEPA latents, with no decoder, and is $2.8$–$4.0\times$ faster than a same-protocol 7B agent that decodes token by token.

• Adapting a JEPA policy to GUIs: a11y candidates as actions, ROI- and action-conditioned residual prediction, a structural head whose only candidate-dependent visual input is the conditioned latent, and two text rankers that together add 3.8 out-of-distribution AC-High points (Table 2).

Stage	IDD	full	OOD	val.
Base policy	56.84	49.79	43.85	57.54
+ span ranker	59.69	52.44	46.34	60.43
+ click reranker	60.99	53.75	47.67	62.46
Δ total	+4.15	+3.96	+3.82	+4.92

• Mechanism: the scorer relies on the JEPA latent pathway, which is trained for the decision. A randomly initialised predictor matches a dynamics-pretrained one, so a JEPA policy for GUIs needs no dynamics pretraining.

Related Work

VLM GUI agents and their cost. The dominant mobile GUI agent is a VLM of 7B+ parameters that emits an action token by token from a screenshot ([4] (Wu et al. 2025; Xu et al. 2025; Qin et al. 2025; Zhang et al. 2025)); 3B agents report AndroidControl under incompatible protocols (Table 11). On-device agents need seconds per step (Park et al., [34]), and efficiency work cheapens the decoder (Bai et al., [3]); GUI-JEPA removes it altogether.

System (source)	Params	Input	Judge	Split	AC-Low	AC-High
HiconAgent-3B (Zhou et al., 2026, Tab. 1)	3B	img	OS-Atlas	unspec. (“OOD setting”)	–	52.40
UI-TARS-2B (Qin et al., 2025, Tab. 8)	2B	img	OS-Atlas	7,708 (inferred)	89.3	68.9
LAMO-3B (Wang et al., 2026, Tab. 2)	3B	img	own (type + value match)	unspec.	92.1	65.5
SecAgent-3B (Xie et al., 2026, Tab. 3)	3B	img	unspec.	unspec.	–	69.5
GUI-Libra-3B (Yang et al., 2026a, Tab. 3)	3B	img	own	AC-v2 (398)	85.9	57.3
Qwen2.5-VL-3B, re-run by GUI-Libra (Yang et al., 2026a, Tab. 3)	3B	img	own	AC-v2 (398)	71.1	36.4
UI-R1-3B (Lu et al., [28], Tab. 3)	3B	img	SR, clicks within 14% of screen	unspec.	66.4	–
LiMAC (AcT + Florence-2) (Christianos et al., [9], Tab. 1)	≈1.34B	img + UI tree	own (text Jaccard ≥0.5)	unspec.	–	63.1†
Florence-2 alone (LiMAC baseline) (Christianos et al., [9], Tab. 1)	0.82B	img + UI-tree boxes	own (text Jaccard ≥0.5)	unspec.	–	57.0†
Qwen2-VL-2B, LoRA (LiMAC baseline) (Christianos et al., [9], Tab. 1)	2B	img + UI-tree boxes	own (text Jaccard ≥0.5)	unspec.	–	52.2†

Discriminative and candidate-scoring GUI agents. Closest in input are the a11y-text PaLM 2S baselines of AndroidControl (§ 4.2). LiMAC selects UI elements from screenshots and UI trees with a small action transformer, calls Florence-2 (Xiao et al., 2024) for text arguments and reports its own metric (Christianos et al., [9]); V-Droid verifies a11y candidates with a prefill-only LLM (Dai et al., [11]), Jev-Mobile selects a11y identifiers with a typed decision model (Zhang, 2026), and IntentScore verifies a generative policy’s actions (Chen et al., [8]); each keeps a generative LLM or VLM in the loop. GUI-JEPA shares a11y enumeration and argmax but scores a candidate-conditioned JEPA latent plus a 33M text matcher, with no generative model in the loop.

World models for GUI agents. GUI world models predict the next state as pixels (Luo et al., [29]; Ning et al. [32]), code (Zheng et al., 2026; Xu et al., 2026a), text (Chae et al., [7]; Mei et al. [31]) or textual element sketches (Cao et al., [6]) and serve an LLM policy as simulator, verifier or data synthesiser; none of the four output modalities compared by Xu et al. (2026b) is joint-embedding. STP pretrains forward and inverse GUI dynamics in a unified multimodal model that still decodes actions (Liu et al., [27]), and Li et al. [22] train a generative text world model over a11y-tree states by predicted-state matching, whose predictions feed a separate reward model that ranks candidate actions. GUI-JEPA is decoder-free and is the policy, trained, like that web world model, for how it is used.

JEPA and latent world models as policies. In robotics, latent world models built on joint-embedding predictive architectures (Assran et al., [1, 2]) plan (Zhou et al., (2025a); Maes et al. [30]; Zholus et al. (2026)) or drive action policies (Sun et al., (2026); Lin et al. [25]; Zhang et al. (2026b)). On GUIs, UI-JEPA feeds a JEPA-pretrained video encoder into a fine-tuned LLM decoder that describes user intent, with a reported 6.6× lower latency than Claude 3.5 Sonnet on IIW [14]; the concurrent LGWM, a JEPA over GUI transitions, verifies whether an observed transition is expected (Zhang et al., 2026a), RSI-Anything’s world-model predictions rank actions that a separate policy proposes [10], and next-frame feature alignment serves as a training-time regulariser discarded at inference (Yang et al., 2026b); the recent Agentic-JEPA picks actions by the cosine between predicted next-state and goal embeddings in synthetic text-only environments (100% success in distribution, 0% out of distribution) (Rodrigo-Ginés, 2026). Decision-aligned work finds that the latent closest to the goal need not be the best action [26] and that predictors can ignore the action [15]; value-equivalent models likewise learn latents for planning without a reconstruction objective (Schrittwieser et al., 2020). In GUI-JEPA the JEPA is the policy itself: it scores all candidates through candidate-conditioned latents of a frozen GUI-pretrained encoder, with no decoder, whereas earlier GUI work uses a JEPA to describe intent, verify transitions or regularise training. Our mechanism result (§ 5.1) adds a GUI-domain data point to work finding that predictability and action relevance come apart [16] and to work that trains world models for the decision [22].

Method: GUI-JEPA

GUI-JEPA chooses each action by scoring candidates. At every step it enumerates the executable actions from the a11y tree, embeds the screenshot once with a frozen JEPA encoder, transforms that screen latent once per candidate with a residual predictor, scores each candidate-conditioned latent and the candidate’s text against the text condition, and executes the highest-scoring candidate (Figure 2). Since no token, pixel or line of code is generated, a decision is one batched forward pass whose cost grows with the number of candidates.

Problem and notation. We learn a step-level offline policy on AndroidControl [23] that predicts the recorded action from one step’s inputs and the ground-truth history. A step has a screenshot $x_t$, an a11y tree $T_t$, a text condition $g$ (the goal in AC-High, the step instruction in AC-Low) and the ground-truth history $h_t$ (the previous action vector and the mean latent change from the previous frame; the text matcher receives the previous actions as text). A frozen encoder gives the screen latent, patch tokens $z_t=f_\theta(x_t)\in\mathbb{R}^{196\times1024}$. The candidate set $\mathcal{C}=\operatorname{enumerate}(T_t)\cup\mathcal{G}$ joins element candidates (a click per enumerated node, an input_text per editable one) with a fixed set $\mathcal{G}$ of global actions. A candidate $c$ carries a 16-d action vector $a_c$ (type one-hot plus coordinates or scroll direction) and, if it has an element, its region $\operatorname{roi}(c)$ (global actions: zero region feature, whole-screen pooling). The policy picks $\hat{c}_t = \arg\max_{c \in\mathcal{C}_t} s(c)$.

Input step and candidates

Element-level candidates are the visible a11y nodes flagged clickable, long-clickable, editable or scrollable, capped at the 80 smallest by area, which keeps specific elements and bounds the batch; each carries the text of the node and its descendants, its bounding box and its type. Every enumerated node yields a click candidate, and every editable node also an input_text candidate whose text is filled in by the span ranker (§ 3.4). Global actions act on the whole screen: the four scroll directions, navigate_back, navigate_home, wait and one open_app, whose argument is the longest training-vocabulary app name in the text condition. The ground-truth element of a click is the smallest enumerated candidate that contains the recorded click point (details and coverage: Section B).

JEPA encoder

$f_\theta$ is a ViT-L/16 (303M; Dosovitskiy et al., [13]), ImageNet-initialised and pretrained (stage 1) with a JEPA objective adapted from I-JEPA (Assran et al., [1]) on 189K unlabelled mobile (GUI-Odyssey; Lu et al., 2025) and desktop (VideoCUA; Jian et al., [19]) screenshots, without action labels or AndroidControl data. The student encoder sees the screenshot $\tilde{x}$ with a random half $M$ of its patches zeroed and regresses, at the masked positions, the tokens that an exponential-moving-average (EMA) teacher $\bar{f}$ computes on the full screenshot:

with $\ell_{\mathrm{sL1}}$ the smooth-$L_1$ loss and $\operatorname{sg}$ the stop-gradient. Random masking suits screenshots, whose content is spread over many small elements, and the latent target spares the encoder from reproducing fonts or colours that do not matter for acting. After two epochs the encoder is frozen and its patch tokens are cached, so policy training never runs the backbone (recipe and hyperparameters: Section H; pretraining–test overlap: Section G.1).

Residual latent predictor

The predictor $g_\phi$ (39.9M) is a JEPA-style action-conditioned latent predictor that maps the screen latent and one candidate to a candidate-conditioned latent. Its action input concatenates $a_c$ with the ROI-pooled feature $\mathbf{u}_c = \operatorname{pool}_{\operatorname{roi}(c)}(\mathbf{z}_t) \in\mathbb{R}^{1024}$ of the candidate’s element (zero for global actions), so that two candidates of the same type, such as two buttons, are told apart by what their regions show as well as by their coordinates. An MLP of this pair is added to a learned query token, and a three-layer transformer attends over the 196 patch tokens and the query, giving the candidate-conditioned latent $\tilde{\mathbf{z}}_t^c = \mathbf{z}_t + \Delta(\mathbf{z}_t,a_c,\mathbf{u}_c)$. The residual form keeps the content of the current screen and lets the predictor express only a candidate-specific change, which preserves the detail needed to ground an element. All candidates of a step form one batch. $g_\phi$ is initialised from a dynamics predictor trained on AndroidControl transitions to predict the next step’s latent from $\mathbf{z}_t$ and the recorded action (Section D) and is then fine-tuned jointly with the scoring heads. Policy training adds no next-latent loss, and the policy loss alone shapes $\tilde{\mathbf{z}}_t^c$ into a decision feature (§ 5.1).

Score and select

Structural head. The head pools the candidate-conditioned latent over the candidate’s region (all 196 tokens for global actions) and over the whole screen, and compares an MLP of the pair by cosine with an MLP of the conditioning vector (learned scale $\kappa$, per-type bias):

with $\mathbf{v}_c = [\operatorname{pool}_{\operatorname{roi}(c)}(\tilde{\mathbf{z}}_t^c); \operatorname{mean}(\tilde{\mathbf{z}}_t^c)]$. cond joins the frozen CLIP embedding (63M; Radford et al., 2021) of $g$ with $h_t$. The head’s only candidate-dependent visual input is $\tilde{\mathbf{z}}_t^c = \mathbf{z}_t + \Delta$: $\mathbf{z}_t$ reaches it via the residual skip and, independently of the candidate, through cond. The constraint is deliberate. Everything the head sees of a candidate passes through the predictor, and training cannot bypass the JEPA latent pathway (§ 5.2).

MiniLM text matcher. The text matcher, a MiniLM-L12 cross-encoder (33M; Wang et al., 2020; Reimers and Gurevych, 2019), scores $g$ plus the textual action history against the text of each element candidate ($s_{\mathrm{text}} = 0$ for global actions); the scores are summed outside the structural head, $s(c) = s_{\mathrm{lat}}(c) + s_{\mathrm{text}}(c)$. The structural head arbitrates the action type and the eight global actions, which have no text to match, and also contributes at element level, while the text matcher carries most of the grounding among clicks (§ 5.1).

Text rankers (stage 3). Two small rankers refine the arguments without changing which kind of action is taken. The input_text argument is the top span of the goal-source span ranker, a MiniLM-L6 cross-encoder (22.7M) trained to rank spans of the goal, the text it reads at test time in AC-High (§ 4.4). If the argmax is a click, the type-gated click reranker $r$, a MiniLM-L12 listwise model (33.5M), re-scores the click candidates as $s(c) + \beta\mathbf{1}[\operatorname{conf}(r) > \tau]\tilde{r}(c)$, with $\tilde{r}$ its $z$-scored logit and $\beta,\tau$ set on validation.

Training

Training has three stages (circled numbers in Figure 2): JEPA pretraining of the encoder (§ 3.2), policy training, and the two text rankers. The base policy (predictor, structural head and text matcher, trained jointly; encoder and CLIP text encoder frozen) is trained on the 13,594 cached episodes (74,714 steps) of the 13,604-episode AndroidControl training split with a listwise cross-entropy over the candidates of each step, $\mathcal{L}_t = -w_t \log p_t(c_t^*)$ with $p_t = \operatorname{softmax}_{c \in C_t} s(c)$, where $c_t^*$ is the ground-truth candidate and $w_t=1.5$ for non-click steps and 1 for clicks, which offsets the predominance of clicks. The objective matches inference, where the recorded action must outscore every alternative the policy could take on that screen. We train with AdamW for 6 epochs, one model for both AC-Low and AC-High (hyperparameters: Table 13), and fine-tune the predictor jointly, which outperforms freezing it (§ 5.2). The two rankers are then trained on the fixed base policy. Of the 0.50B parameters, 366M are frozen (303M encoder, 63M CLIP text encoder) and $\approx132$M trained, including the 22.7M span ranker and the 33.5M click reranker.

Component	Params	Setting
Encoder pretraining (frozen afterwards)
JEPA encoder, ViT-L/16	303.3M	vit_large_patch16_224, 224 px, $14 \times14 = 196$ patch tokens; ImageNet initialisation; JEPA objective after I-JEPA with random-patch masking at ratio 0.5 (the student sees the screenshot with the masked patches zeroed and regresses the EMA target’s tokens at those positions, smooth-L1); 189,262 unlabelled screenshots (118,791 GUI-Odyssey mobile, 70,471 VideoCUA desktop); 2 epochs at batch 96, 3,942 optimiser steps (the policy uses the epoch-2 checkpoint); EMA target decay 0.996; peak learning rate $5 \times10^{-5}$ with warm-up; JEPA loss 1.28 $\to$ 0.68; target-token std 1.8–2.2 (min 1.82, final 2.17; no collapse); 2,739 s $\approx$ 0.76 GPU-hour on one H20
Policy (trained end to end with the listwise loss)
Action code $a_c$	–	16-d: action-type one-hot plus normalised coordinates or scroll direction
Residual latent predictor	39.9M	ROI-pooled encoder feature $u_c$ (1,024-d) $\oplus a_c \to$ MLP $\to$ added to a learned query token; 3 transformer layers over the 196 patch tokens and the query, 2-D sine–cosine positional embeddings; residual output $\tilde{z}_t = z_t + \Delta(z_t, a_c, u_c)$; initialised from a dynamics predictor trained on AndroidControl transitions, then fine-tuned with the policy loss
Structural head	2.1M	cosine, with a learned scale, between an MLP of $[\operatorname{pool}_{\mathrm{roi}(c)}(\tilde{z}_t); \operatorname{mean}(\tilde{z}_t)]$ and an MLP of the conditioning vector, plus a learned per-type bias; its only candidate-dependent visual input is $\tilde{z}_t$
Conditioning vector	63.2M	frozen CLIP text encoder: embedding of the goal (High, once per episode) or of the step instruction (Low, once per step), joined with the previous action code and the mean latent difference between the previous and the current frame
Text matcher	33.4M	MiniLM-L12 cross-encoder over the instruction (with the textual action history appended) and the candidate text; its score is added outside the structural head
Post-hoc modules
Goal-source span ranker	22.7M	MiniLM-L6 cross-encoder over spans of the goal, on which it is also trained; supplies the input_text argument in both settings
Type-gated click reranker	33.5M	MiniLM-L12 listwise reranker over the click candidates; final = base $+ \beta\tilde{r}$ where $\tilde{r}$ is the $z$-scored reranker logit, applied only when the base argmax is a click and the reranker confidence exceeds the gate, in both settings; variant, $\beta= 1$ and gate $= 1$ selected on the validation split
Optimisation of the base policy
Data	–	AndroidControl training split as cached, 13,594 of 13,604 episodes / 74,714 of 74,722 steps (Section B); cached frozen features; candidates capped at 80 per step
Loss	–	listwise cross-entropy over the enumerated candidates with the ground-truth candidate as positive; weight 1.5 on non-click ground-truth types, 1 on clicks
Optimiser	–	AdamW; learning rate $10^{-4}$ for predictor and heads, $2 \times10^{-5}$ for MiniLM; weight decay 0.01; no dropout; 6 epochs; 3 training seeds; one model for Low and High
Total	0.50B	498.0M: ViT-L 303.3M, CLIP text 63.2M, predictor 39.9M, head 2.1M, matcher 33.4M, click reranker 33.5M, span ranker 22.7M; 366.5M frozen (ViT-L and CLIP text) and 131.6M trained

Experiments

Setup

Benchmark and splits. We test on the four official AndroidControl subsplits ([23]): IDD (721 episodes) and the overlapping app_unseen (631), task_unseen (803) and category_unseen (700, contained in task_unseen). The full test set (1,543 episodes, 8,444 steps) and the OOD union (full test minus IDD, 4,586 steps) are the units of evidence. IDD results cover 716 of the 721 episodes (3,858 of 3,897 steps), because five IDD episodes also in OOD subsplits count as OOD (Section B). The 137-episode validation split selected the click reranker’s variant, weight $\beta$ and gate. Systems that state their split report all test subsets together, so we compare them with our full-test result and use IDD only against published IDD numbers.

Protocol. AC-Low conditions on the step instruction and AC-High on the episode goal, both with the ground-truth action history. Our exact-match judge keeps the original benchmark’s click rule (the chosen candidate’s centre lies inside the ground-truth element box) and departs from the original protocol in four places: it counts no equivalent open_app or navigation actions, requires the normalised typed text to match, accepts a click for a ground-truth long_press (9 IDD steps), and takes the ground-truth box from the smallest enumerated candidate (originally the smallest a11y node), scoring clicks without a candidate (1.6–2.6%) as errors where the original discards them (Section B). Candidates come from the a11y tree, a privileged input that pixel-only agents do not use.

Main results

Closest published protocol. The AndroidControl paper’s LoRA-tuned PaLM 2S over a11y text ([23], Table 5) shares our a11y input (we add the screenshot), click rule and subsplits, under the original relaxed judge. In distribution, GUI-JEPA matches it on AC-Low (84.06 vs 83.2) and trails on AC-High (60.99 vs 70.8; Table 6 and Table 7).

Condition	IDD AC-Low s1	IDD AC-Low s2	IDD AC-Low s3	IDD AC-High s1	IDD AC-High s2	IDD AC-High s3
Inference-time lesions (text matcher untouched unless stated)
Base policy (as is)	82.79	81.80	83.88	56.48	55.39	56.84
no predictor ($\tilde{z}_t := z_t$)	61.79	62.05	59.02	41.16	40.93	42.87
within-type action shuffle	72.01	71.12	72.65	46.47	44.95	46.06
all-candidate action shuffle	55.83	55.81	56.64	35.17	34.63	33.41
swapped, non-fine-tuned predictor	68.33	66.77	64.54	46.03	45.41	45.08
text matcher dropped	39.17	36.39	39.48	30.82	29.32	31.60
Training-time variants (identical full-scale recipe)
Jointly fine-tuned predictor (base)	82.71	81.88	83.72	56.35	55.21	56.84
Frozen dynamics-pretrained predictor	80.43	80.38	79.94	54.90	54.80	54.95
Randomly initialised predictor, joint	82.84	82.24	82.71	57.52	56.14	57.34
Identity map in place of the predictor	61.53	62.96	62.36	42.79	41.21	42.33
Matcher only (no latent pathway)	55.60	54.98	55.42	40.51	39.17	40.10
No predictor, action vector into a param.-matched head	82.37	81.96	83.89	56.45	56.82	41.08

Subsplit	Episodes / steps	Base	+ ranker	+ reranker	$\Delta$	PaLM 2S
IDD	716 / 3,858	$56.13 \pm0.84$	$59.14 \pm0.79$	$60.12 \pm0.95$	$+3.98$	70.8
app_unseen	631 / 3,475	$41.30 \pm0.27$	$43.39 \pm0.26$	$44.56 \pm0.83$	$+3.25$	58.5
task_unseen	803 / 4,464	$43.03 \pm0.70$	$45.79 \pm0.68$	$46.87 \pm1.00$	$+3.84$	59.6
category_unseen	700 / 3,891	$41.71 \pm0.45$	$44.06 \pm0.44$	$45.20 \pm0.83$	$+3.48$	57.4
OOD union	827 / 4,586	$43.21 \pm0.55$	$45.65 \pm0.60$	$46.74 \pm0.94$	$+3.52$	–
Full test	1,543 / 8,444	$49.12 \pm0.64$	$51.81 \pm0.62$	$52.85 \pm0.94$	$+3.73$	–
Validation	137 / 690	$57.39 \pm1.67$	$60.43 \pm1.74$	$61.30 \pm2.13$	$+3.91$	–

Subsplit	Base	+ ranker	+ reranker	PaLM 2S
IDD	82.77	82.68	$83.09 \pm1.01$	83.2
app_unseen	72.99	71.75	$72.37 \pm1.58$	78.5
task_unseen	73.66	72.65	$73.15 \pm1.46$	77.3
category_unseen	72.84	71.75	$72.30 \pm1.68$	76.8
OOD union	73.69	72.51	$73.04 \pm1.44$	–
Full test	77.84	77.16	$77.64 \pm1.23$	–
Validation	83.91	83.48	$84.16 \pm0.89$	–

Exact-match VLM agents. The pixel-only 7–8B exact-match entries of Table 1 decode their actions with 14–16× more (nominal) parameters than GUI-JEPA. On AC-Low our full-test 78.64 is above OS-Atlas-7B (67.3), MIRAGE-4B (77.6) and Qwen3-VL-8B (77.66), and none of these agents reports a comparable per-step latency.

System	Params	Input	Judge	Split	AC-Low	AC-High
EM scores reported by other papers (screenshot input)
Aguvis-7Bᵃ	7B	img	EM	unspec.	89.4	54.2
OS-Atlas-7Bᵃ	7B	img	EM	unspec.	67.3	56.5
Qwen3-VL-8Bᵐ	8B	img	EM*	all-split	77.7	60.8
Qwen2.5-VL-7B, releasedᵃ	7B	img	EM	unspec.	85.0	62.9
MIRAGE-4Bᵐ	4B	img	EM*	all-split	77.6	64.4
Qwen2.5-VL-72Bᵠ	72B	img	EM*	unspec.	93.7	67.4
AgentCPM-GUI-8Bᵃ	8B	img	EM	unspec.	90.2	69.2
MIRAGE-8Bᵐ	8B	img	EM*	all-split	83.8	72.4
UI-TARS-7Bᵃ	7B	img	EM	unspec.	91.8	74.4
UI-Venus-Navi-7Bᵐ	7B	img	EM*	all-split	85.1	75.2
Closest published protocol (a11y input, our click rule and split)
PaLM 2S, LoRA on a11y textᵖ	n/a	a11y	relaxed	IDD	83.2	70.8
Same inputs, judge and split (ours)
Qwen2.5-VL-7B, LoRA SFT (our reference)	8.29B	img+a11y	exact	IDD	91.0	70.9
GUI-JEPA (ours)	0.50B	img+a11y	exact	IDD	84.1	61.0
GUI-JEPA (ours)	0.50B	img+a11y	exact	full	78.6	53.8

System (source)	Params	Input	Judge	Split	AC-Low	AC-High
GPT-4o, zero-shot ((Wu et al., 2025, Tab. 5))	n/a	img	OS-Atlas	7,708	28.4	21.2
OS-Atlas-7B, zero-shot ((Wu et al., 2025, Tab. 5))	7B	img	OS-Atlas	7,708	50.9	29.8
M3A (GPT-4-Turbo, text-only), zero-shot ([23 Tab. 4])	n/a	a11y	relaxed	r500-IDD	55.0	42.1
Aguvis-7B, SFT ((Xu et al., 2025, Tab. 3))	7B	img	relaxed	(inferred) r500	80.5	61.5
Aguvis-72B, SFT ((Xu et al., 2025, Tab. 3))	72B	img	relaxed	(inferred) r500	84.4	66.4
OS-Atlas-7B, SFT ((Wu et al., 2025, Tab. 5))	7B	img	OS-Atlas	7,708	85.22	71.17
UI-TARS-7B, SFT ((Qin et al., 2025, Tab. 8))	7B	img	OS-Atlas	7,708 (inferred)	90.8	72.5
MAI-UI-8B / 32B ((Zhou et al., 2025b, Tab. 7))	8B / 32B	img	unspec.	unspec.	–	69.1 / 75.5
ScaleTrack-7B ([18 Tab. 3])	7B	img	unspec.	r800 (“out-of-domain”)	86.6	77.9

Examples. Figure 5 shows GUI-JEPA on three real IDD AC-High steps, one per action kind: it picks a menu entry among 15 candidates, opens the app named in the goal, and types a book title copied from the goal, each with one forward pass.

Same-protocol 7B reference

We also train a same-protocol reference: Qwen2.5-VL-7B-Instruct ([4]) fine-tuned with LoRA ([17]) on 4,000 training episodes for one epoch, one adapter per setting (GUI-JEPA: one model, 13,594 episodes for six; 377-token image), given the same screenshot, candidates and ground-truth history as GUI-JEPA and scored by the same judge on the same 716 IDD episodes. It reaches 91.01 / 70.87 Low / High, and GUI-JEPA reaches 84.06 / 60.99, 92.4% / 86.1% of it, with 6.0% of its parameters (0.498B vs 8.29B; 16.6× fewer) and 2.8–4.0× its vLLM speed (§ 4.5). The gap is 7.0 points when the step instruction is given and 9.9 with the goal alone. § 4.6 breaks the AC-High gap down by action type.

Gains from the two text rankers

Table 2 isolates the two text rankers. Switching the training data of the 22.7M input_text span ranker from step instructions to goal spans adds +2.85 IDD High (+2.49 OOD union) by raising IDD High input_text accuracy from 13.5 to 52.5. The type-gated click reranker then adds +1.30 IDD High (+1.33 OOD union) by re-ordering click candidates only. Together they lift

OOD-union High from 43.85 to 47.67 and every unseen subplit by 3.5–4.1 points (Section C), and the span-ranker gain replicates on the validation split (+2.89 vs +2.85 on IDD).

Efficiency

What the comparison fixes. The timing compares a scored step with a decoded step at fixed inputs, hardware and batch size. Both systems receive the same 100 real AC-High IDD screenshots, candidates and histories on one H20 at batch 1, and differ in model, training data and computation (§ 4.3). GUI-JEPA’s timing starts at JPEG decoding and covers CPU candidate and ROI preparation; the 7B agent’s starts at image processing and tokenisation. Screenshot capture, a11y retrieval and GUI-JEPA’s once-per-episode CLIP

Speed. The final GUI-JEPA pipeline (base policy, span ranker and click reranker) needs a median 74 ms per step in fp32 and 51.5 ms with bf16 autocast, against 207 ms for the 7B agent with vLLM and 507 ms with HF transformers (merged LoRA; Table 3). GUI-JEPA is thus 2.8× (fp32) to 4.0× (bf16) faster than vLLM and 6.8–9.8× faster than HF, with 4.6–6.4× less peak GPU memory. In bf16 it makes the same predictions as in fp32 on all 200 timed steps, including the typed text (§ 15).

Where the time goes. Figure 3 splits the mean step time on the same 100 steps. Decoding takes 51% (vLLM) to 60% (HF) of the 7B agent’s step, on top of its prefill, and the vLLM prefill alone outlasts a whole fp32 GUI-JEPA step, so the gain comes from both the smaller model and the absence of decoding. About half of a GUI-JEPA step goes to the batched predictor and head, which bf16 speeds up threefold, and its cost grows with the number of candidates (§ 15).

Error analysis

Clicks, 63% of the steps, account for 5.8 of the 9.9-point AC-High gap to the 7B agent, shared between choosing the click type and grounding the element, the part the text matcher carries (§ 3.4). On global actions the gap lies mainly in the type decision, and on open_app and input_text in the argument (per type: Table 15). The two systems also err on different steps (Figure 10): at least one of them is right on 78.7% of IDD AC-High steps, and the margin between GUI-JEPA’s top two scores flags its errors (AUROC 0.74), so uncertain steps can be routed to a same-protocol 7B agent where accuracy matters more than speed (Section J).

Action type	$n$	Step acc. 7B	Step acc. ours	Type acc. 7B	Type acc. ours	Gap
All	3,858	70.9	61.0	87.0	80.6	9.9
click	2,419	72.0	62.8	91.3	86.3	5.8
scroll	505	71.7	59.8	83.8	64.4	1.6
input_text	282	67.0	52.5	92.2	87.6	1.1
open_app	253	81.0	68.0	87.7	92.9	0.9
navigate_back	117	64.1	38.5	64.1	38.5	0.8
wait	273	57.9	59.3	57.9	59.3	−0.1
long_press	9	44.4	66.7	100	88.9	−0.1

Analysis

What the latent pathway does

Reliance. We lesion the trained base policy at evaluation time, text matcher untouched (Figure 4a; Table 5). On IDD, shuffling the action vectors across all candidates costs 27.2 Low / 23.4 High, shuffling them within a type 11.2 / 10.8 (mostly scroll directions; about 5 High on clicks), and replacing the candidate-conditioned latent by the current latent (no predictor) 24.9 / 14.0, mostly on the global actions, which share the full-screen ROI. The action vector reaches the structural head only through the predictor, and training cannot route around it. Retrained at full scale with the identity map in place of the predictor, the policy reaches 62.36 / 42.33, 21.4 / 14.5 points below the base policy, again on the global actions, which the identity map leaves indistinguishable (Table 8). The pathway thus carries chiefly action type and scroll direction, and also helps select among clicks.

Condition	IDD AC-Low	IDD AC-High	app_unseen Low / High	task_unseen Low / High	category_unseen Low / High
Base policy (as is)	82.8±1.0	56.2±0.8	72.9 / 40.8	73.6 / 42.6	72.7 / 41.3
no predictor ($\tilde{z}_t := z_t$)	61.0±1.7−21.9	41.7±1.1−14.6	53.8 / 32.6	53.7 / 33.1	52.8 / 32.2
within-type action shuffle	71.9±0.8−10.9	45.8±0.8−10.4	62.1 / 32.3	63.5 / 34.5	61.9 / 32.7
all-candidate action shuffle	56.1±0.5−26.7	34.4±0.9−21.8	49.9 / 26.7	50.3 / 27.6	49.3 / 26.8
swapped, non-fine-tuned predictor	66.5±1.9−16.3	45.5±0.5−10.7	–	–	–
text matcher dropped (structural head alone)	38.4±1.7−44.5	30.6±1.2−25.7	–	–	–
Frozen dynamics-pretrained predictor	80.3±0.3−2.5	54.9±0.1−1.3	–	–	–
Jointly fine-tuned predictor (base)	82.8±0.9	56.1±0.8	–	–	–
Randomly initialised predictor, joint	82.6±0.3−0.2	57.0±0.8+0.9	–	–	–
Identity map in place of the predictor, retrained	62.3±0.7−20.5	42.1±0.8−14.0	55.0 / 31.6	55.5 / 32.2	54.3 / 31.2
Matcher only (no latent pathway), retrained	55.3±0.3−27.4	39.9±0.7−16.2	47.7 / 29.6	47.2 / 30.5	46.8 / 29.6
No predictor, action vector into a param.-matched head (per seed)	82.4 / 82.0 / 53.9	56.5 / 56.8 / 41.1	72.4 / 40.4§	73.2 / 41.9§	72.4 / 40.9§
Per-patch next-latent loss	–	+0.26 (mean)	–	–	–
Pooled next-latent MSE	–	+0.6 / −13.4 / +1.3	–	–	–
Multi-horizon InfoNCE	–	+0.7 / −1.2	–	–	–
SIGReg-style (LeWorldModel loss) pretrained predictor	–	−1.0	–	–	–

The pathway is trained for the decision. A randomly initialised predictor trained with the identical full-scale recipe does as well as the dynamics-pretrained one (+0.50 High / −1.01 Low; Figure 4c), and predictive auxiliary objectives (per-patch (Sun et al., 2026)), pooled, multi-horizon, LeWorldModel-style ([30]) add nothing at 4K-episode scale (Table 8). The predictor thus learns what the decision needs, and forward-model probes show that it forecasts the next latent worse than a copy of the current one (Section D). It acts as an action- and ROI-conditioned transform of the current screen, so a JEPA policy for GUIs needs no dynamics pretraining. This agrees with findings that predictability and action relevance come apart ([16]) and with training world models for the decision ([22]).

Design choices and the encoder

At pilot scale without the text matcher, the residual form raises AC-Low from 40.6 to 43.9, whereas a predictor that re-synthesises the whole latent lowers AC-Low click grounding to 34.8, below the identity map’s 46.8. The structural head makes the policy use the predictor. Replacing the predictor by the identity map costs 8.9 AC-Low points (40.6 vs 31.7), whereas a concat-MLP head that also sees raw candidate features stays within 1.4 points with or without a predictor (Figure 4b). Conditioning on the previous action and frame change adds 8 High points; at full scale, joint fine-tuning beats a frozen predictor by 3.8 AC-Low / 1.9 AC-High IDD points (Figure 4c; alternatives: Table 10). The encoder is cheap to obtain. After 0.76 GPU-hours of JEPA pretraining on unlabelled screenshots, its frozen tokens score 67.22 on a coarse localisation probe over ScreenSpot-Pro (Li et al., [21]) (probe hit@8; random-init ViT-L: 58.8), on par with DINOv3-ViT-L (Siméoni et al., 2026) (65.28; Figure 7, Section G). In the policy, DINOv2-L (Oquab et al., [33]) features at 518 px instead cost 17.4 / 9.8 points at 4K-episode scale, though it also changes the input resolution (Table 10).

Variant	Setting	Reference	Variant	Reading
Scoring head
Concat-MLP head over $[z_t, \tilde{z}_t^c, c]$ replacing the structural head (ROI-conditioned predictor)	pilot, 2 ep	40.59 / 19.62	34.58 / 26.88	−6.0 Low, +7.3 High; its scores do not depend on the predictor
Encoder and resolution
DINOv2-L at 518 px as main features	4K, 4 ep	77.32 / 47.12	59.88 / 37.35	encoder and resolution change together

Conclusion

GUI-JEPA, to our knowledge the first GUI agent whose policy is built on a JEPA, replaces action decoding with one batched scoring pass over candidate-conditioned JEPA latents. With 0.50B parameters it needs 51.5–74 ms per step, 2.8–4.0× less than a vLLM-served 7B agent on the same inputs, and reaches 92.4% / 86.1% of its IDD Low / High accuracy with 6.0% of its parameters.

The JEPA latent pathway is essential to the scorer (lesions cost 10.8–23.4 IDD High points)

and is trained for the decision: a randomly initialised predictor does as well as a dynamics-pretrained one. A JEPA used as a GUI action scorer should thus be judged by the decisions it supports, not by next-latent fidelity, and can be trained for them directly. Extending such scorers from steps to tasks is the next step for JEPA-based GUI agents.

Limitations

GUI-JEPA consumes the ally tree, a privileged structured input that pixel-only agents do not use; the 1.6–2.6% of test clicks without a candidate count as errors, and every latency excludes screen capture and ally retrieval. The policy is a step-level scorer; online task completion, which needs a termination head and a planner, is left to future work, as is using the predictor as a forward model. Design choices other than the reranker’s variant, weight and gate were made on test IDD and rechecked on validation only for the span ranker and overall base-policy accuracy (§ 4.4). Speed-ups are measured at batch 1 against vLLM and HF serving of the 7B agent (§ 4.5). The same-protocol 7B reference is one LoRA adapter per setting trained on 4,000 of 13,594 episodes. The method is specialised to mobile a11y candidates; other candidate sources, such as web DOM nodes, need their own candidate texts and action vectors. Non-JEPA encoders are compared through a probe on the frozen tokens (Section G); a comparison inside the full-scale policy is future work.

Ethical Considerations

GUI-JEPA lowers the cost of GUI agents that act on a user’s device. A wrong action, such as sending a message, deleting data or confirming a payment, can be hard to undo, and step-level errors remain common; a deployment needs user confirmation before irreversible actions. The speed that makes on-device agents practical also makes it cheaper to automate interactions meant for humans, such as mass account creation. The policy reads the accessibility tree, which can contain personal on-screen data; running the 0.50B model on-device keeps that data local. We use only public datasets under their licences (AndroidControl Apache-2.0, GUIOdyssey CC-BY-4.0, VideoCUA and ScreenSpot-Pro MIT), collect no new data, and involve no human subjects, annotators or crowd workers.

Reproducibility statement

The encoder is a ViT-L/16 at 224 px, initialised from ImageNet weights and trained with a JEPA objective adapted from I-JEPA ([1]) on 189,262 unlabelled screenshots from GUI-Odyssey (Lu et al., 2025) and VideoCUA ([19]) for 2 epochs ($\approx0.76$ GPU-hour on one H20), with no action labels and no AndroidControl data (§ 3.2 and H). The policy is trained on the 13,594 cached episodes of the 13,604-episode AndroidControl training split ([23]) with a listwise cross-entropy, AdamW and 6 epochs, for three seeds (§ 3.5 and Table 13). We evaluate on the official test subsplits with the exact-match judge of § 4.1 and time both systems on the same 100 real IDD steps on one H20 at batch 1 (§ 1).

Protocol details

Splits and overlap. The official test_subsplits.json of the AndroidControl ([23]) lists 721 IDD, 631 app_unseen, 803 task_unseen and 700 category_unseen episodes, whose union is the full test set of 1,543 episodes and 8,444 steps. The subsplits are not disjoint: every category_unseen episode is also a task_unseen episode, and app_unseen shares 606 episodes with category_unseen and 607 with task_unseen. We therefore treat the full test set and the OOD union (full test minus IDD as evaluated: 827 episodes, 4,586 steps) as the units of evidence and report the three OOD subsplits, each evaluated in full, only as a breakdown (Section C). Five of the 721 IDD episodes also appear in OOD subsplits and count as OOD, so every IDD accuracy in the paper is computed on 716 episodes and 3,858 of the 3,897 steps (99.0%); the coverage analysis below uses all 721 episodes. The 137-episode validation split (690 steps) selected the click reranker’s variant, its weight $\beta$ and its confidence gate. Every other design choice, including both text rankers, was made on test IDD; the validation row of Table 6 checks how these choices carry over. We train on 13,594 of the 13,604 training episodes ([23 Tab. 3]) (74,714 of 74,722 steps).

Candidate enumeration and coverage. Element-level candidates are the visible accessibility-tree (a11y) nodes flagged clickable, long-clickable, editable or scrollable, plus containers that inherit clickability, de-duplicated by bounds and capped at the 80 smallest by area; editable nodes also yield input_text candidates. Global actions are the four scroll directions, navigate_back, navigate_home, wait and one open_app, whose argument is the longest training-vocabulary app name in the goal or instruction. A ground-truth click or long-press is mapped to the smallest enumerated candidate that contains its coordinate, whereas the original benchmark takes the smallest a11y node of any kind; the click rule itself (the chosen candidate’s centre inside the ground-truth box) is unchanged. When no candidate contains the coordinate the step is unmapped and counts as an error for our policy (the original benchmark discards such steps). The mapped fraction is 97.4–98.4% on the test subsplits (IDD 98.0%, 2,405 of 2,454 click and long-press steps; full test 97.6%, 4,961 of 5,083), 98.8% on validation and 98.4% on the training split. An IDD step has 23.6 a11y candidates on average (median 18, p95 57; 22.2 on the full test set and 23.2 on the training split), and the 80-candidate cap is reached on 16 IDD, 22 full-test and 272 training steps. Per-type IDD step counts are in Table 15.

Judge. A step is correct under our exact-match judge when the predicted action type equals the ground-truth type and the argument matches: a click when the centre of the chosen candidate lies inside the ground-truth element box, which is the click rule of the original benchmark ([23 App. D.3]); an input_text when the typed string equals the ground-truth string after normalisation; open_app and scroll when the argument matches exactly. The judge departs from the original protocol in four places, in both directions. (i) It is stricter in accepting no equivalent actions, whereas the original protocol treats open_app and a click on an element bearing the app name as equivalent, and likewise navigate_back and a click on the on-screen Back button. (ii) It is more lenient on input_text: it compares the normalised text only, whereas the original protocol scores typing as an element action that also requires the target element to lie inside the ground-truth box and the text to be identical. (iii) It is more lenient in accepting a click for a ground-truth long_press (9 IDD steps). (iv) It takes the ground-truth box from our enumerated candidates: a mapped click is judged against a box at least as large as the original’s smallest a11y node (more lenient), while an unmapped click counts as an error,

counts as an error, where Li et al. [23] discard it (stricter). On click steps, 62.7% of IDD steps, both judges thus apply the same centre-in-box rule. Exact-match (EM) re-evaluations in the literature are the closest published protocol; numbers under the original relaxed protocol are not interchangeable with ours.

Conditioning and seeds. One trained model serves both settings: AC-Low conditions on the step instruction and AC-High on the episode goal; both receive the ground-truth action history. We train three policy seeds. The main text reports seed 3, the best seed on every test split, at every stage and in both settings; full-scale results in this appendix are means ± sd over the three seeds unless a seed is named, and Table 4 and Table 5 list every seed. For $n=3$ the 95% $t$-interval half-width, $4.30\,\mathrm{sd}/\sqrt{3}$, is about ±2.4 points (IDD High ±2.37, full-test High ±2.34, IDD Low ±2.52). Seeds capture training variance only, so external comparisons also use episode-clustered bootstrap intervals (Section B.2).

Split	Stage	AC-High seed 1	AC-High seed 2	AC-High seed 3	AC-High mean ± sd	AC-Low seed 1	AC-Low seed 2	AC-Low seed 3	AC-Low mean ± sd
IDD	Base policy	56.35	55.21	56.84	56.13±0.84	82.71	81.88	83.72	82.77±0.92
	+ span ranker	59.49	58.24	59.69	59.14±0.79	82.63	81.78	83.64	82.68±0.93
	+ click reranker	60.26	59.10	60.99	60.12±0.95	83.18	82.04	84.06	83.09±1.01
Full test	Base policy	49.05	48.51	49.79	49.12±0.64	78.13	76.65	78.74	77.84±1.07
	+ span ranker	51.79	51.21	52.44	51.81±0.62	77.43	75.99	78.06	77.16±1.06
	+ click reranker	52.93	51.87	53.75	52.85±0.94	78.01	76.26	78.64	77.64±1.23
OOD union	Base policy	42.91	42.87	43.85	43.21±0.55	74.27	72.24	74.55	73.69±1.26
	+ span ranker	45.31	45.29	46.34	45.65±0.60	73.05	71.13	73.35	72.51±1.21
	+ click reranker	46.75	45.79	47.67	46.74±0.94	73.66	71.39	74.07	73.04±1.44
app_unseen	Base policy	41.29	41.04	41.58	41.30±0.27	73.55	71.57	73.84	72.99±1.24
	+ span ranker	43.37	43.14	43.65	43.39±0.26	72.32	70.33	72.60	71.75±1.24
	+ click reranker	44.95	43.60	45.12	44.56±0.83	73.12	70.56	73.44	72.37±1.58
task_unseen	Base policy	42.63	42.61	43.84	43.03±0.70	74.19	72.22	74.57	73.66±1.26
	+ span ranker	45.39	45.41	46.57	45.79±0.68	73.16	71.24	73.54	72.65±1.23
	+ click reranker	46.82	45.90	47.89	46.87±1.00	73.77	71.48	74.19	73.15±1.46
category_unseen	Base policy	41.40	41.51	42.23	41.71±0.45	73.45	71.19	73.89	72.84±1.45
	+ span ranker	43.74	43.87	44.56	44.06±0.44	72.35	70.11	72.78	71.75±1.43
	+ click reranker	45.28	44.33	45.98	45.20±0.83	73.04	70.37	73.48	72.30±1.68
Validation	Base policy	58.99	55.65	57.54	57.39±1.67	84.64	83.04	84.06	83.91±0.81
	+ span ranker	62.17	58.70	60.43	60.43±1.74	84.20	82.61	83.62	83.48±0.80
	+ click reranker	62.61	58.84	62.46	61.30±2.13	84.93	83.19	84.35	84.16±0.89

Per-seed results

Table 4 and Table 5 list every seed; the main-text values are the seed-3 columns, which lie 0.87 / 0.90 AC-High points above the three-seed means on IDD / the full test. Out-of-distribution lesion costs of the main-text model are given in Section D.

Episode-clustered bootstrap intervals

Because steps of one episode are correlated, the bootstrap for the external comparisons (§ 4.1, Table 1) resamples test episodes. Each of $B=2{,}000$ resamples draws the 716 IDD (or 1,543 full-test) episodes with replacement and keeps every step of a drawn episode; the same resample is applied to the per-step predictions of the three seeds of the final system, whose step accuracies are averaged, and the 2.5th and 97.5th percentiles of this seed-averaged accuracy form the 95% interval (the per-step predictions reproduce the reported accuracies exactly). The intervals are 60.12 [58.45, 61.85] High and 83.09 [81.72, 84.41] Low on IDD, and 52.85 [51.59, 54.04] High and 77.63 [76.62, 78.63] Low on the full test set (77.64 in Table 4 averages per-seed results rounded to two decimals). For seed 3 alone they are 60.99 [58.95, 62.97] High and 84.06 [82.46, 85.47] Low on IDD, and 53.75 [52.40, 55.06] High and 78.64 [77.61, 79.70] Low on the full test set. The bootstrap covers testepisode sampling and the $t$-interval over seeds covers training variance; neither covers the protocol differences listed in Table 1.

Per-subsplit waterfall

Table 6 and Table 7 and Figure 6 break the waterfall of Table 2 down by subsplit. The AC-High gains transfer: on the three OOD subsplits the span ranker adds 2.08–2.76 points, and the click reranker adds 1.04–1.17 on the full test set, the OOD union and every OOD subsplit, at least its 0.98 on IDD. Neither stage changes the action-type decision of the base policy, but AC-Low pays a small cost. The single span ranker, trained on goals, also supplies the typed argument for AC-Low step instructions and lowers AC-Low on every seed (by 1.18 points on the OOD union): on the full test set it raises High input_text accuracy from 17.7 to 53.7 (IDD: 12.3 to 53.4) but lowers Low input_text accuracy from 83.4 to 74.4. The click reranker, which raises IDD High click grounding given the type from 71.8 to 73.7, also runs in AC-Low and recovers 0.53 of the 1.18 OOD-union points, leaving OOD-union AC-Low 0.65 below the base policy.

On the validation split the waterfall keeps the test ordering and step sizes (+3.04 and +0.87 High). The span-ranker step is thus a held-out replication of a choice made on test IDD. The reranker step is not, because the reranker’s variant, $\beta$ and gate were selected on this split; its held-out evidence is the test-set gain.

The PaLM 2S columns give the only published baseline that consumes the a11y tree (text only, no screenshot) with our click rule and subsplits; its AC-Low setting receives the episode goal together with the step instruction [23], while ours receives the step instruction only. Averaged over the three seeds, GUI-JEPA matches it on IDD AC-Low (83.09 vs 83.2) and trails by 10.7 AC-High points; out of distribution it trails by 12.2–13.9 High and 4.2–6.1 Low points.

Mechanism analyses

Table 8 collects the evidence on the latent-predictor pathway summarised in § 5.1; the text below adds only what the table does not show.

Inference-time lesions. The lesions are applied at evaluation time to the trained full-scale base policies (text rankers off), so they measure how much the trained scorer relies on the pathway; the retrained controls below test whether it is needed. Removing the predictor ($\tilde{z}_t := z_t$) hurts mainly the global actions: of its 21.9 AC-Low / 14.6 AC-High IDD points, 19.0 / 14.1 fall on scroll, open_app and navigate_back (19.6 / 15.6 with wait). These share the full-screen ROI, so without the predictor their action vectors no longer reach the score and the three types are never chosen correctly, while click accuracy barely moves (83.9 to 83.2 Low, 58.4 to 60.6 High). The within-type shuffle, which permutes the four scroll directions, costs 8.6 / 6.3 points on scroll steps and 2.3 / 4.2 on clicks. Without the text matcher the structural head’s click grounding is 30.9 Low / 31.8 High against 89.8 / 71.8 for the full scorer: the matcher carries most of the click grounding, while the head alone still grounds about a third of the click steps whose type it gets right. The reliance holds on every OOD subsplit; for the main-text model, removing the predictor costs 21.7–23.1 Low / 7.9–9.7 High points there, the within-type shuffle 10.5–11.5 / 8.5–9.0 and the all-candidate shuffle 24.4–25.0 / 15.4–16.7 (means: Table 8).

Training-time variants. Freezing the dynamics-pretrained predictor costs 2.52 AC-Low / 1.25 AC-High IDD points on average (3.78 / 1.89 for seed 3; Table 5). A randomly initialised predictor trained jointly with the identical full-scale recipe matches the dynamics-pretrained one: its High is higher for every seed (+0.50 to +1.17) and its Low within 1.01 points. At pilot scale, the identity map gives 31.73 Low vs 40.59 for the ROI-conditioned predictor (structural head, no text matcher; Figure 4b), and training on shuffled dynamics with a frozen predictor gives 47.33 / 28.46 vs 62.70 / 38.96.

Retrained controls. Retrained from scratch at full scale with the identical recipe, the identity map in place of the predictor again never chooses scroll, open_app or navigate_back correctly (−11.3, −5.1 and −2.6 Low points), while clicks change by −1.8 Low / +0.2 High points. The matcher-only policy scores 0–12% on scroll, open_app, navigate_back, wait and input_text, and its click grounding is 88.4 / 68.2 against the base policy’s 89.8 / 71.8. A no-predictor control tests whether the predictor is needed at all once the action reaches the score: its structural head receives $[\operatorname{pool}_{\mathrm{roi}}(c)(z_t); \operatorname{mean}(z_t); a_c]$ through an MLP matched in parameters to the predictor and the original projection (41.09M against 39.91M + 1.18M), with matcher, history, type bias and recipe unchanged. It matches the base policy in two of three seeds (Table 5; within 1.4 points on app- and category-unseen) and collapses on clicks in the third (click grounding 48.3 Low against 89.6 for its base seed). Given the action vector directly, a parameter-matched head on the same JEPA latents thus reaches the base policy’s accuracy when its training succeeds, whereas the predictor trains stably in all three seeds; either way the action is scored on the JEPA latent, consistent with the predictor’s role as an action- and ROI-conditioned transform of it (§ 5.1).

Predictive objectives. The bottom block of Table 8 gives the predictive auxiliary objectives as AC-High changes against the matched 4K reference; none improves on it (Section E).

Forward-model probes. Table 9 shows that the deployed predictor is a poor forward model: its output is further from the true next latent than a copy of the current latent, and its predicted change has near-zero cosine to the true change and points the right way at chance rate. Feeding a shuffled action leaves the predicted latent at cosine 0.86 to the true-action prediction (IDD). The randomly initialised predictor of Table 8 (150 IDD episodes, 782 steps) is no more action-sensitive: its predicted change has cosine 0.002 to the true change, its ground-truth AUC equals the deployed predictor’s (0.5426 vs 0.5425; chance 0.5), and a shuffled action leaves its output at cosine 0.887 (0.954 for a within-type shuffle, probed for this predictor only). Free-running

	IDD	app_unseen	task_unseen
Probe steps / episodes	1,579 / 300	116 / 24	1,671 / 300
cos(prediction, true next latent)	0.467	0.467	0.460
cos(copy $z_t$, true next latent)	0.788	0.773	0.786
cos(predicted change, true change)	0.006	0.003	0.004
Right direction (%)	52.6	47.4	51.6

$K$-step rollouts of the dynamics-pretrained predictor that initialises the deployed one (400 episodes; 1,500 / 1,500 / 1,349 queries at $K=1 / 2 / 3$), scored by cosine to the true latent $K$ steps ahead, never beat persistence (keeping $z_t$): 0.745 against 0.793 at $K=1$ (0.733 with shuffled actions), 0.510 against 0.749 at $K=2$ and 0.322 against 0.741 at $K=3$. The rollout beats persistence on 39 / 10 / 2% of steps and a shuffled-action rollout on only 43 / 48 / 49%; only within-episode retrieval of the true frame at $K=1$ slightly favours it (R@5 79.7 vs 77.2 for persistence). Its one-step cosine (0.745) exceeds the deployed predictor’s 0.467, consistent with joint fine-tuning on the policy loss moving the predictor away from next-latent prediction; both are below copying (0.793 / 0.788).

Design alternatives not adopted

Table 10 lists the design alternatives not adopted; the paragraphs below add what it does not show.

Scoring head. The concat-MLP head trades AC-Low for AC-High accuracy: against the structural head it loses AC-Low and gains AC-High points with the ROI-conditioned predictor, and likewise loses 7.3 Low and gains 8.8 High with an $(x,y)$-conditioned predictor (40.51 / 18.43 vs 33.18 / 27.19). Its AC-Low accuracy is the same with either predictor or none (Figure 4b), so whatever it gains does not come through the predictor, whereas the structural head drops from 40.59 to 31.73 when the predictor is replaced by the identity map. We keep the structural head because it forces the policy through the candidate-conditioned latent; the higher pilot-scale High of the concat head shows that this constraint has a cost at that scale. Conditioning the structural head on the previous action and frame change (implicit action history) was adopted: it gains 8.0 High at pilot scale without the text matcher (23.35 to 31.39, at a cost of 4.4 Low) and 7.1 High with it (31.91 to 38.96, −1.6 Low), and the final system uses it together with the textual action history in the matcher query.

Encoder and resolution. Swapping in DINOv2-L features ([33]) changes the encoder and the resolution (518 px) together, so this comparison does not isolate the effect of resolution.

Predictor objectives. Predictive auxiliary losses on the predictor do not help (Table 8, bottom). The leakage-free pooled next-latent MSE has a low mean because one of its three runs collapses; the other two are within 1.3 points of their paired controls. Our SIGReg-style regulariser simplifies SIGReg ([5]): where LeJEPA uses the Epps–Pulley statistic, it penalises the mean and variance of 256 random projections of the pooled predicted latents, and it acts on frozen encoder features, whereas LeWorldModel ([30]) trains end to end.

Other-protocol baselines

Tables 11 and 12 list the AndroidControl results of small models and of larger or zero-shot systems reported under protocols that Table 1 does not include; their judge, split and input columns show why none except LiMAC is set against our numbers. None of the larger models is evaluated on the full test set we use.

LiMAC ([9]) is the closest small agent that also consumes the UI tree: an action transformer (ActT, $\approx520\mathrm{M}$) over UI-element embeddings combined with Florence-2 ((Xiao et al., 2024)) (820M as deployed by LiMAC), which receives the screenshot annotated with numbered element boxes and the goal. Its overall accuracy is a goal-conditioned per-step action accuracy, the analogue of our AC-High setting, under its own judge (a predicted element whose box lies within the target element is correct, typed text is accepted at a Jaccard index of at least 0.5) on “the test set”, with no sub-split or size given; its confusion matrix sums to 3,708 steps, matching none of the official sub-splits. LiMAC and Florence-2 alone are 10.3 and 4.2 points above our 3-seed mean full-test High (9.3 and 3.2 above seed 3) with 2.7× and 1.6× our parameters, and its fine-tuned Qwen2-VL-2B baseline is 0.65 below our mean with 4.0× our parameters; LiMAC’s more lenient text rule and unstated split keep the comparison approximate. LiMAC reports 0.34 s per step (Florence-2 alone: 0.50 s) on unspecified hardware, against our 51.5–74 ms per real step on one H20 (§ 15).

With the three-seed mean full-test AC-Low (77.64; seed 3: 78.64), GUI-JEPA is level with Qwen3-VL-8B (77.7) and MIRAGE-4B (77.6) of Table 1.

Encoder probe

Probe. Figure 7 compares pretraining recipes for the frozen encoder by a localisation probe on its patch tokens: a linear scorer per patch, a softmax over the $14 \times14$ grid and a soft-argmax give a point that is regressed (MSE) onto the centre of the target box; the encoder is fixed and only the probe is trained. hit@$k$ is the fraction of held-out examples whose predicted point lies within $k/14$ of the image side (normalised Euclidean distance) of the target centre, so hit@8 accepts errors of up to 128 px on the 224-px input. The probe is lenient: on the 1,581 held-out ScreenSpot-Pro (Li et al., [21]) examples the policy encoder’s hit@8 of 67.22 ± 0.91 comes with a hit@3 of 10.36 ± 0.36, a hit@1 of 0.92 ± 0.36 and a mean error of 0.89 of the image side (114 px); on 255 ScreenSpot-v2 (Wu et al., 2025) examples it reaches 90.59 ± 1.00 hit@8 with a mean error of 0.378 of the side (85 px). For calibration, under the same probe a randomly initialised ViT-L reaches 87.8 hit@8 on the same ScreenSpot-v2 split and 58.8 [56.4, 61.2] on ScreenSpot-Pro, and an ImageNet-initialised ViT-L without GUI pretraining 58.1. The probe thus measures coarse localisation.

Recipes. Under this probe, the mixed recipe, I-JEPA-style pretraining (Assran et al., [1]) of an ImageNet-initialised ViT-L on 119K GUI-Odyssey (Lu et al., 2025) mobile and 70K VideoCUA (Jian et al., [19]) desktop frames for 2 epochs, scores at least as high as every other recipe in Figure 7: either domain alone, a web anchor trained for 60 epochs on the 76,838 viewport and full-page screenshots of the class-balanced 7K split (Web-7k-Resampled) of WebUI (Wu et al., 2023), and DINOv3-ViT-L (Siméoni et al., 2026), trained on 1.7B images. DINOv3’s 95% bootstrap interval over probe examples, [62.9, 67.6], contains the policy encoder’s 67.22 (95% $t$-interval over probe seeds [66.1, 68.3]): under this probe the policy encoder is at DINOv3’s level, and since the two intervals cover different sources of variance (probe seeds against probe examples), they do not order the encoders. The replicate uses the same data, batch size and learning-rate schedule and continues to 6 epochs: after 2 epochs (JEPA loss 0.668 against 0.677 for the policy encoder) it gives 67.82, and its 4- and 6-epoch scores show that training longer does not help. The 0.6-point gap between the two 2-epoch encoders is a rough measure of pretraining variance, which the per-recipe sds, computed over probe seeds, do not capture.

The recipes also differ in data size (189K / 119K / 70K / ≈77K screenshots) and training length (2 to 60 epochs), so the comparison does not identify which ingredient matters. Two further recipes do not lift the probe mean: initialising JEPA pretraining from DINOv3 lowers it, and action-conditioned pretraining in the style of V-JEPA 2-AC ([2]) on VideoCUA stays at the level of the web anchor.

The probe measures frozen localisation features: the recipe is a 0.76 GPU-hour route to frozen GUI features that localise as well as the alternatives. In the policy, the 4K-scale swap to DINOv2-L at 518 px (Table 10) changes the encoder and the resolution together.

Pretraining–test near-duplicates

We hashed the 9,987 screenshots of the 1,543 AndroidControl test episodes (pHash and dHash, 64 bits) against the 189,262 pretraining screenshots (118,791 GUI-Odyssey, 70,471 VideoCUA). 58 test frames (0.58%) in 43 episodes have a verified near-duplicate (pHash distance $\le8$ and normalised cross-correlation $\ge0.9$), all from GUI-Odyssey and showing generic system screens. For calibration, the same pHash threshold matches 35.0% of test frames to the frames of 4,000 AndroidControl training episodes (different episodes of the same benchmark) and 9.5% to the pretraining set; random pairs have a median distance of 32. Removing the 43 episodes changes final High and Low by −0.2 (High 59.90 vs 60.12 on IDD, 52.64 vs 52.85 on the full test). Removing every episode with any pHash match within distance 8 (549 episodes), a criterion that also flags generic layouts, gives 58.74 IDD and 51.14 full-test High. At the app level, 29 app names match exactly and 26% of test episodes touch a package that also appears in GUI-Odyssey, with different devices and tasks and no shared screenshots. Pretraining–test leakage is thus negligible for the reported numbers.

Implementation details and hyperparameters

Table 13 lists every component with its measured parameter count and the settings of the reported models; the paragraphs below add only what the table does not state.

Encoder. The encoder is the ViT-L/16 of Dosovitskiy et al. [13] from the timm library. Where I-JEPA (Assran et al., [1]) uses multi-block masks, our student sees the screenshot with a random half of its patches zeroed and regresses the tokens that the EMA target computes on the full image at the masked positions. The encoder is frozen and its 196 patch tokens per frame are cached; the policy never updates it.

Predictor, head and matcher. All candidates of a step form one predictor batch. The residual form was chosen in IDD pilots without the text matcher: a predictor that re-synthesises the whole latent erases local detail, lowering AC-Low click grounding to 34.8 against 46.8 for the identity map, and the residual form raises pilot AC-Low from 40.59 to 43.93. At pilot scale (IDD) the structural head’s constraint, that its only candidate-dependent visual input is the candidate-conditioned latent, decides whether the pathway is used: a concat-MLP head over $[z_t,\tilde{z}_t^c,\mathrm{cand}]$ ignores the predictor (within 1.4 AC-Low points), while the structural head gains 8.8 points (Figure 4b; Section E). The conditioning vector uses the frozen CLIP text embedding (Radford et al., 2021). The MiniLM-L12 text matcher (Wang et al., 2020; Reimers and Gurevych, 2019) carries most of the click grounding: in the pilot, grounding rose from 37.4 to 70.2 when it was added. The input_text argument is copied as a goal span, which can be correct on at most 81.6% of IDD input_text steps.

Training. Training skips the 1.6% of training click steps whose ground-truth element is not enumerated. Neither text ranker changes the action-type decision of the base policy, but both act in AC-Low as well as AC-High (Section C).

Same-protocol 7B reference. Qwen2.5-VL-7B-Instruct (Bai et al., [4]) (8.29B parameters, measured) is fine-tuned with LoRA (Hu et al., [17]) on 4,000 of the 13,594 cached training episodes for one epoch, with one adapter per setting (3.5 / 4.4 GPU-h for High / Low on one H20); the AC-Low adapter’s prompt adds the step instruction to the goal and shares every other setting, whereas GUI-JEPA serves both settings with one model. It is prompted with the same screenshot (377 image tokens; at most 384 visual tokens), the same enumerated candidate list and the same ground-truth history as GUI-JEPA, decodes a JSON action greedily, and is scored by the byte-identical judge on the same 716 IDD episodes. Beyond Table 14, the High adapter reaches a click top-5 recall of 95.3, and the AC-Low adapter reaches 91.01 Low with type accuracy 97.4 (GUI-JEPA: 84.06, 92.4% of it). We report click grounding in two forms because the conditional form alone rewards predicting fewer clicks: our seed with the lowest type accuracy (74.9) has the highest conditional grounding (76.1). The 7B agent leads on every action type of Table 14 except wait; Table 15 converts the per-type differences into contributions to the overall gap. The fine-tuned number is close to published 7B SFT results on AndroidControl (69.7–71.2 High for Qwen2-VL-7B and OS-Atlas-7B fine-tuned on

caption: Same-protocol comparison with a 7B VLM agent on AC-High (IDD). All systems see the same screenshots, enumerated candidates and ground-truth history and are scored with the same step-accuracy rules. The click rows use the 2,379 click and long-press steps whose ground-truth element is enumerated: type recall is the share predicted as a click, grounding (all) the share answered correctly with type errors counted as misses, and grounding | type the share correct among the steps predicted as a click. ms/step: median over the 100 real IDD steps timed for all systems (§ 4.5). Ours: seed 3, reported in the main text, and the mean ± sd over three seeds.

:::

AndroidControl, Wu et al., 2025, Tab. 5), which use the OS-Atlas judge on 7,708 test samples; under our judge and split it serves as the same-protocol reference.

Latency measurement details

Common protocol. All latencies are measured on one NVIDIA H20 at batch 1 with a single request and no concurrent load on the GPU, with CUDA synchronisation before and after each timed step, and reported as medians. GUI-JEPA timing includes JPEG decoding and CPU candidate and ROI preparation, and 7B timing includes image preprocessing and tokenisation; screenshot capture, accessibility-tree retrieval and parsing, and action execution on the device are excluded for every system. Every latency and speed ratio in the main text comes from the 100 real IDD steps on which both systems were timed (Table 3 and Table 16). The 200-step medians include steps the 7B agent never

Configuration	Steps	p50 ms	p90 ms	mean ms	peak mem	vs 7B vLLM	vs 7B HF (merged)
Real steps, fp32 (precision of all accuracy tables)	shared 100	74.3	127.8	86.9	3.45 GB	2.8×	6.8×
Real steps, bf16 autocast	shared 100	51.5	79.7	56.2	2.48 GB	4.0×	9.8×
Real steps, fp32	all 200	74.8	134.0	87.5	3.45 GB	–	–
Real steps, bf16 autocast	all 200	48.0	79.0	53.4	2.48 GB	–	–
Synthetic fixed $N$, fp32, $N=5$ / 23 / 57 / 100	–	28.6 / 53.9 / 95.8 / 151.5	–	–	1.9–3.3 GB	–	–
Synthetic fixed $N$, bf16, $N=5$ / 10 / 23 / 57 / 100	–	19.4 / 22.2 / 28.3 / 43.6 / 63.8	–	–	1.8–2.4 GB	–	–

ran and are given for reference only. so our ratios use the merged model. With vLLM 0.11.2 (Kwon et al., [20]) in bf16 the merged model needs 206.7 ms end to end (generation alone 206.5 ms; prefill 96 ms, 10.2 ms per token) with prefix caching off, and 211.9 ms (p90 250) with prefix caching on, where the 15 ms prefill and 18.5 ms per token reflect the cached system prefix.

7B agent on real steps. The Qwen2.5-VL-7B agent is timed on the first 110 AC-High IDD test steps in test-set order (10 warm-up, 100 measured) with the prompt of its accuracy evaluation: the screenshot at a 377-token image budget, the goal, the ground-truth history and the enumerated candidate list in the Qwen chat template, 931 input tokens on average. It decodes greedily until the end-of-sequence token and emits 11.75 new tokens on average, its most favourable configuration. End to end, including CPU prompt preparation (chat template, image processor, tokenisation), HF transformers 4.57 in bf16 with flash-attention 2 needs 506.7 ms per step for the merged LoRA-SFT model (generation alone 499.1 ms; prefill 189 ms, decode 28.3 ms per token), 504.6 ms for the zero-shot base model (p90 580.6; prefill 187 ms, 27.5 ms per token; 15.9 GB peak) and 753.6 ms for the unmerged LoRA model used in the accuracy evaluation (p90 889.3; prefill 229 ms, 47.8 ms per token; 16.2 GB). Merging is the deployment configuration,

GUI-JEPA real-step serving path. We time the final pipeline end to end on the first 210 AC-High IDD test steps in test-set order (10 warm-up, 200 measured, 46 episodes), so the 7B agent’s 100 measured steps are a subset of ours. Each step executes the full pipeline, with the span ranker only on steps predicted as input_text (10% of steps) and the click reranker only on steps predicted as click (63.5%). The 200 steps have 24.4 ally candidates on average (median 18, p90 49) and 33.0 scored actions (median 28.5, p90 58). The served policy is the evaluated policy: its base-policy action matches the offline evaluation, which uses precomputed encoder features, on 198 of the 200 steps (99.0%), and on all 200 when both receive the same preprocessed frames. On the 200 steps the bf16 and fp32 pipelines make identical predictions, including the typed text. On all IDD steps, the base policy (base stage of Table 2) changes by +0.18 Low / −0.07 High in bf16 relative to fp32, with 99.8% step-level agreement on type, candidate and direction. The typed string agrees on 151 of the 322 High steps where both predict input_text; that ranker is rarely right in High, and High input_text accuracy moves from 15.6 to 14.2 (4 of 282 steps).

Where the GUI-JEPA step time goes. Over the 200 steps, the fp32 p50 decomposes into 8.3 ms JPEG decoding and resizing (9.6% of the mean), 11.4 ms ViT-L (13.0%), 1.2 ms candidate construction, 1.2 ms ROI encoding, 39.7 ms predictor and structural head (51.4%), 8.8 ms matcher (12.7%), 8.1 ms click reranker when it fires (6.8%) and 21.5 ms span ranker when it fires (3.3%); in bf16 the p50 components are 7.0 ms decoding, 7.7 ms ViT-L, 11.8 ms predictor and head and 10.4 ms matcher. By a11y candidate count the fp32 p50 is 58.9 ms below 15 candidates, 74.5 at 15–29, 125.1 at 30–59 and 168.7 at 60 or more. The CLIP goal embedding runs once per episode and is not included: its median is 5.6 ms (mean 10.8 ms, raised by one 245 ms first call), about 1.3 ms per step amortised at the median and 2.4 ms at the mean. In AC-Low the step instruction changes at every step, so the CLIP call recurs per step; all real-step timings are AC-High.

GUI-JEPA synthetic fixed-$N$ micro-benchmark. The synthetic rows of Table 16 and the lines of Figure 8 time the final pipeline in eager PyTorch on a random 224-px image tensor with random action and ROI vectors and dummy candidate texts and bounds, 100-iteration median. $N$ counts the scored actions in the predictor batch, that is, a11y candidates plus global actions. $N=23$ and 57 equal the mean and p95 a11y candidate counts of IDD, but a real step scores more actions than it has a11y candidates (33.0 against 24.4 on average on the timed steps), so the synthetic $N=23$ point understates a typical step and the real-step rows are the operating point. The encoder cost is fixed, and the predictor batch grows by about 1.2 ms per scored action in fp32 and 0.37 ms in bf16; torch.compile on top of bf16 lowers the bf16 times by a further 0.7–6.6%. The benchmark’s inputs are not real screenshots, so it only shows the linear scaling in $N$.

Literature latencies. 8B-class GUI agents average 2.6–10.6 s per step in a live-device benchmark (Sui et al., 2026, Tab. 2; means over valid sessions, BF16 on H200 GPUs with the GPU count and serving engine not stated, each model under its official inference protocol where one is released). V-Droid reports 0.7 s per decision, covering all candidates of a step (0.44 s batched prefill verification plus 0.30 s action completion), and 4.3 s per step end to end, 3.03 s of it in a GPT-4 working-memory update (verifier on two RTX 4090 GPUs; [11]). Neither shares our measurement boundary or statistic (means over live-device sessions with model-specific prompts and outputs, against our median over 100 real test steps).

Confidence routing to the 7B agent

GUI-JEPA and the same-protocol 7B agent err on different steps (Figure 10): on IDD AC-High at least one of them is right on 78.9% of steps on average over the three GUI-JEPA seeds (78.7–79.2), against 70.87 for the 7B agent and 60.12 for GUI-JEPA. We therefore test GUI-JEPA as the cheap first stage of a cascade: a step is answered by GUI-JEPA when the margin between its top two final scores is at least $\tau$ and escalated to the 7B SFT agent otherwise. The margin predicts whether GUI-JEPA is correct with AUROC 0.74–0.77, and on its most confident half of the steps GUI-JEPA alone is 77.6–79.2% accurate. Setting $\tau$ at coverage quantiles (the share of steps kept by GUI-JEPA) traces the curve of Figure 11 (main-text model). At 50% coverage the cascade reaches 69.48 High on average over seeds, 98% of the 7B agent’s 70.87, at 0.91 × its mean vLLM latency with fp32 GUI-JEPA and 0.76 × with bf16 (with HF serving, 86.9 + 0.5 × 515.7 ≈ 345 ms against 515.7 ms, 0.67×). Because $\tau$ is set on coverage quantiles of the same IDD steps, we report the cascade as a deployment mode. For the main-text model the margin AUROC is 0.74, at least one system is right on 78.7% of steps, and the 50%-coverage cascade reaches 69.67 (98% of 70.87) at the same expected latency, with GUI-JEPA 78.1% accurate on the steps it keeps.