betaFeedback

WorksComputer Science

JointGV: Structure-Aware Correction of Visual World Models with Parallel Graph Dynamics for Long-Horizon Modeling

Abstract

Visual world models predict future visual states conditioned on actions, but recursive errors in entity poses and spatial relations limit their long-horizon prediction for robot planning and control. We ask whether a separately predicted relational state can correct this drift without restructuring the visual model. This is challenging because graph nodes and visual states share neither elements nor a feature space. We propose Joint Graph–Visual Dynamics Correction (JointGV), a plug-in based on a cross-granularity dynamics hypothesis: despite their different representations, graph-node and visual-state changes describe the same physical transition. JointGV rolls out a task-conditioned scene graph alongside the visual world model, couples their predicted changes, and combines the resulting graph-motion context with graph-predicted node positions to compute a bounded residual. Graph-learning and correction gradients do not propagate into the original visual dynamics. We attach JointGV to DINO-WM (spatial tokens) and DreamerV3 (a global latent state), evaluating both on DMC Cheetah Run and Walker Walk across three seeds and extending the evaluation to contact-rich manipulation with DMC Finger Turn Hard. Across 32-step rollouts, JointGV improves structural and visual-prediction accuracy, reducing joint-position error by 11–28%. For spatial tokens, learned graph–visual change coupling gives lower error than broadcasting the same mean graph-motion context to every token, particularly when non-adjacent parts appear close together in the image. On real-robot inverted-pendulum sequences, JointGV reduces DINO-WM's 32-step pole-angle error from 12.56∘12.56^\circ to 6.69∘6.69^\circ and improves DreamerV3 and LeWorldModel as well. In simulated inverted-pendulum control, an actor trained with JointGV-corrected rollouts completes 46 of 50 episodes, compared with none using the original rollouts.

Code: https://github.com/JasonLvernex/Joint_Graph_Vedio_Latent_Dynamic_World_Model

Introduction

Visual world models predict how a robot and its environment change under an action. Robot planning evaluates multi-step action sequences, making recursive rollout necessary [7, 27, 34]. Reusing each predicted state as the next input, however, accumulates errors in entity poses and spatial relations. Figure 1 shows this structural drift in open-loop predictions initialized from RGB observations of a physical inverted-pendulum robot: the predicted pole progressively deviates and blurs, while its angle error increases with rollout horizon across three visual world models.

Motivation figure showing real-robot DINO-WM open-loop rollout and pole-angle error across rollout horizons

Figure 1. Motivation. (a) DINO-WM open-loop predictions initialized from real-robot inverted-pendulum observations increasingly deviate from the observed pole. (b) Pole-angle error rises with rollout horizon for all three models, while JointGV reduces this structural drift.

Existing approaches reduce long-horizon error by changing the native predictive representation [29, 33], using auxiliary trajectory supervision [31], learning residuals within the original latent pathway [9, 21], or replacing the visual state with object- or graph-centric dynamics [4, 11]. These approaches modify the visual representation, derive correction from the same latent pathway, or require a separately trained structured world model. They do not address how an independently predicted relational rollout can correct an existing visual state when their elements and feature spaces do not correspond.

Our key hypothesis is that, although graph nodes and visual states differ in granularity and content, their action-induced changes describe the same physical transition. We propose Joint Graph–Visual Dynamics Correction (JointGV), which rolls out a task-conditioned scene graph alongside an existing visual world model and couples their predicted changes in a shared space. Graph-predicted node positions provide an additional spatial reference, and a bounded, gated residual corrects the visual prediction. Graph-learning and correction gradients do not propagate into the original visual encoder or dynamics.

Our contributions are: (1) a cross-granularity graph–visual correction interface for predictive states without shared elements or feature spaces; (2) a plug-in architecture that uses a separately predicted relational rollout while preserving the original visual dynamics; and (3) three-seed evaluation across four DMC locomotion pairings and a contact-rich DMC manipulation task, real-robot prediction, simulated control, and analyses identifying when learned graph–visual coupling is most useful.

Related Work

Visual world models for robot learning. Visual world models learn action-conditioned states for planning and control. DayDreamer and DreamerV3 learn behaviors in latent imagination [7, 27]. DINO-WM predicts DINOv2 patch features, IRASim predicts robot–object videos, and LeWorldModel predicts image embeddings [13, 15, 34, 35].

Object-centric and graph-structured world models. Graph networks model physical systems through entity states and relational message passing [12, 17, 18]. C-SWM, G-SWM, SlotFormer, and DAFT-RL build object interactions into the predictive state [4, 8, 11, 28], while recent methods extend structured prediction to multimodal, recurrent, and evolving graphs [1, 5, 19, 26]. These methods use the structured state as part of the world model itself. JointGV instead preserves an existing visual state and uses an independently predicted graph rollout as a correction source.

Structure and residual correction of world models. GRWM and GeoWorld alter the geometry of the native predictive representation [29, 33]. 4D-WAM aligns spatially corresponding video and trajectory-field changes as auxiliary training supervision [31]. ReDRAW and Sandwich-Residuals instead learn residuals within the original latent pathway to adapt to dynamics or distribution shift [9, 21]. Structure-guided video prediction generates frames from predicted poses or keypoints [14, 24], and PhyDNet separates physical dynamics from residual factors [10]; these methods train the visual predictor around the structured state. JointGV addresses a different interface: it couples an independently predicted scene-graph rollout with visual states that have no shared indexing or feature space, then corrects the visual prediction without propagating correction gradients into the original visual model.

Method

Overview

Let Zt\mathbf{Z}_t denote a spatial-token or global-latent visual state, and let Z‾t+1\overline{\mathbf{Z}}_{t+1} be its next-state prediction from the original visual world model. JointGV predicts graph-node features N^t+1\widehat{\mathbf{N}}_{t+1} in parallel. Their one-step changes are

ΔZ‾t=Z‾t+1−Zt,ΔN^t=N^t+1−Nt(1)\Delta\overline{\mathbf{Z}}_t = \overline{\mathbf{Z}}_{t+1} - \mathbf{Z}_t,\qquad\Delta\widehat{\mathbf{N}}_t = \widehat{\mathbf{N}}_{t+1} - \mathbf{N}_t \tag*{(1)}

where the bar denotes the original visual prediction, the hat denotes the graph-dynamics prediction, and Δ\Delta denotes a one-step change.

JointGV is based on a cross-granularity dynamics hypothesis: although graph nodes and visual states share neither elements nor a feature space, their action-induced changes describe the same physical transition. JointGV couples these changes and combines the resulting graph-motion context with graph-predicted node positions to compute a bounded correction CtC_t:

Z^t+1=Z‾t+1+Ct.(2)\widehat{\mathbf{Z}}_{t+1} = \overline{\mathbf{Z}}_{t+1} + C_t. \tag*{(2)}

Graph-learning and correction losses do not update the original visual encoder or dynamics. Figure 2 shows the complete data flow.

Overview of JointGV

Figure 2. Overview of JointGV. Given the current RGB observation and action, the original visual world model (yellow) predicts the uncorrected next visual state Zˉt+1\bar{Z}_{t+1} and its change ΔZˉt\Delta\bar{Z}_t. In parallel, JointGV (blue) uses Relation Transformer (RelTR) features to construct a task-conditioned scene graph and predicts the node-feature change ΔN^t\Delta\hat{N}_t. For the spatial-token case shown here, learned projections map the visual-token and graph-node changes into a shared space, where a change-alignment adapter produces the graph-motion context. The predicted node positions provide a complementary position context. A bounded, gated correction combines these contexts with the original visual prediction to produce Z^t+1=Zˉt+1+Ct\hat{Z}_{t+1}=\bar{Z}_{t+1}+C_t. Setting Ct=0C_t=0 recovers the original prediction. Decoded images are shown only for visualization; the global-state adapter is described in § 3.4.

Task-Conditioned Scene Graph Construction

From RGB observation ot\mathbf{o}_t, JointGV constructs

Gt=(Nt,Xt,Rt),(3)\mathbf{G}_t = (\mathbf{N}_t,\mathbf{X}_t,\mathbf{R}_t), \tag*{(3)}

where Nt∈RJ×dg\mathbf{N}_t \in\mathbb{R}^{J \times d_g} contains JJ task-role node features, Xt∈[−1,1]J×2\mathbf{X}_t \in[-1,1]^{J \times2} contains their image positions, and Rt\mathbf{R}_t contains directed relation logits.

Learned task-role queries bind intermediate semantic and spatial RelTR features [2] to fixed graph slots. The resulting nodes exchange task-relevant information before a relation head predicts directed edges. Node identities, positions, visibility, and relations are training targets only; inference constructs the graph from RGB observations without simulator states. The feature binding, temporal query update, inter-node exchange, and relation head are detailed in § 7.

History-Aware Graph Dynamics

Let Hg≥1H_g \ge1 denote the number of most recent scene graphs supplied to the graph dynamics model. These graphs, together with the action at\mathbf{a}_t applied between time tt and t+1t+1, are used to predict the next scene graph. This design follows learned graph physics models that combine relation-aware message passing with short motion histories [17, 18]. For node ii, we define its one-step change as δnt,i=nt,i−nt−1,i\delta\mathbf{n}_{t,i} = \mathbf{n}_{t,i} - \mathbf{n}_{t-1,i}. The current node feature and its changes over the history window are encoded as

gt,i(0)=fh([nt,i;δnt−Hg+2,i;…;δnt,i]),(4)\mathbf{g}^{(0)}_{t,i} = f_h\left(\left[\mathbf{n}_{t,i};\delta\mathbf{n}_{t-H_g+2,i};\ldots;\delta\mathbf{n}_{t,i}\right]\right), \tag*{(4)}

where fhf_h is a multilayer perceptron (MLP), and the superscript (0) indicates the input to the first message-passing layer. When Hg=1H_g = 1, the input contains only the current node feature nt,i\mathbf{n}_{t,i}.

Before message passing, the relation logits Rt\mathbf{R}_t are normalized over the KrK_r relation classes. At each of the LGL_G graph layers, every active node sends an action-conditioned message to the other active nodes. The relation probabilities of each node pair are embedded into its message, while the probability of the no-relation class controls how strongly that message contributes to the receiving node.

The final graph features are decoded into bounded node and relation-logit increments, denoted by ΔN^t\Delta\widehat{\mathbf{N}}_t and ΔR^t\Delta\widehat{\mathbf{R}}_t, respectively. The next graph state is

N^t+1=Nt+ΔN^t,R^t+1=Rt+ΔR^t,(5)\begin{aligned} \widehat{\mathbf{N}}_{t+1} &= \mathbf{N}_t + \Delta\widehat{\mathbf{N}}_t,\\ \widehat{\mathbf{R}}_{t+1} &= \mathbf{R}_t + \Delta\widehat{\mathbf{R}}_t, \tag*{(5)} \end{aligned}

where each component of ΔN^t\Delta\widehat{\mathbf{N}}_t and ΔR^t\Delta\widehat{\mathbf{R}}_t is bounded by cn>0c_n > 0 and cr>0c_r > 0, respectively. The predicted node positions X^t+1\widehat{\mathbf{X}}_{t+1} are decoded from N^t+1\widehat{\mathbf{N}}_{t+1}, giving the predicted scene graph G^t+1=(N^t+1,X^t+1,R^t+1)\widehat{\mathbf{G}}_{t+1} = (\widehat{\mathbf{N}}_{t+1},\widehat{\mathbf{X}}_{t+1},\widehat{\mathbf{R}}_{t+1}). The complete message-passing and increment-decoding operations are provided in Sec. S2.

Self-fed dynamics training. One-step training always provides an observed graph as the input, whereas inference recursively uses previously predicted graphs. To reduce this mismatch, we train the graph dynamics through a KK-step self-fed rollout, where K≥1K \ge1 is the training horizon. Starting from a graph-history window constructed from observed image frames, rollout step k∈{1,…,K}k \in\{1,\ldots,K\} predicts

(N^t+k,R^t+k)=Dη(H^t+k−1g,at+k−1),(6)\left(\widehat{\mathbf{N}}_{t+k},\widehat{\mathbf{R}}_{t+k}\right) = \mathcal{D}_{\eta}\left(\widehat{\mathcal{H}}^{g}_{t+k-1},\mathbf{a}_{t+k-1}\right), \tag*{(6)}

where Dη\mathcal{D}_{\eta} is the graph dynamics model with parameters η\eta, at+k−1\mathbf{a}_{t+k-1} is the action applied during the current transition, and H^t+k−1g\widehat{\mathcal{H}}^{g}_{t+k-1} contains the most recent HgH_g observed or predicted graphs. The first transition uses an observed graph history, after which every predicted graph is inserted into the history window for the following transition. Observed future graphs and simulator states are used only as supervision targets, not as inputs to the recursive rollout. The complete masked losses and the matched self-fed control for the original visual world model are provided in Sec. S4.

Graph–Visual Correction Interface

The interface is defined on predicted changes rather than absolute features. Graph nodes and visual states retain different information, but the changes induced by the same action isolate the physical transition represented by both predictions. The graph rollout then supplies two complementary cues: node changes describe how task-relevant entities evolve, while predicted node positions provide their spatial reference. This separation allows the same correction principle to operate on both spatial tokens and a global latent state.

JointGV operationalizes the cross-granularity dynamics hypothesis by coupling projected graph-node and visual-state changes. The two representations need not share elements, cardinality, or a feature space, and the interface is not tied to a particular coupling operator. We use Sinkhorn optimal transport for spatial tokens and normalized attention pooling for a global latent state.

Spatial-token alignment. For a spatial visual state, we instantiate the alignment adapter with Sinkhorn optimal transport (OT). Let Δz^t,p=[Δz^t]p∈Rdz\Delta\widehat{\mathbf{z}}_{t,p} = [\Delta\widehat{\mathbf{z}}_t]_p \in\mathbb{R}^{d_z} denote the predicted change of visual token p∈{1,…,P}p \in\{1,\ldots,P\}, and let Δn^t,i=[Δn^t]i∈Rdg\Delta\widehat{\mathbf{n}}_{t,i} = [\Delta\widehat{\mathbf{n}}_t]_i \in\mathbb{R}^{d_g} denote the predicted change of active graph node i∈Ii \in\mathcal{I}. Here, PP is the number of visual tokens, dzd_z and dgd_g are the visual-token and graph-node feature dimensions, respectively, and I\mathcal{I} is the set of active task-role-bound nodes.

Because the two changes belong to different feature spaces, proj⁡v:Rdz→Rdc\operatorname{proj}_{\mathrm{v}}:\mathbb{R}^{d_z}\to\mathbb{R}^{d_c} and proj⁡g:Rdg→Rdc\operatorname{proj}_{\mathrm{g}}:\mathbb{R}^{d_g}\to\mathbb{R}^{d_c} project and ℓ2\ell_2-normalize them in a shared dcd_c-dimensional space. The visual-token and graph-node sets also have different sizes and no fixed one-to-one pairing. We use entropy-regularized OT for soft many-to-few change coupling; cosine attention provides an alternative learned coupling and is evaluated in Table 1. The OT matching cost and transport plan are

MethodYr.Joint-position error (px) ↓\downarrow H=8H=8H=16H=16H=32H=32Token MSE ↓\downarrow H=32H=32RGB MSE×103\times10^{3} ↓\downarrow H=32H=32SSIM ↑\uparrow H=32H=32LPIPS ↓\downarrow H=32H=32
Longer visual training
DINO-WM (Kv=1K_v=1) [34]20250.8300.9311.0360.4562.310.8410.0435
DINO-WM + JointGV (Kv=1K_v=1)Ours0.7300.7960.8520.3411.840.8610.0351
DINO-WM (Kv=8K_v=8)20250.8820.9400.9570.3392.130.8530.0420
DINO-WM + JointGV (Kv=8K_v=8)Ours0.7500.8150.8630.3141.800.8630.0352
Residual correction without a predicted graph
Matched DINO-WM backbone20250.7650.8470.8980.3772.060.8530.0366
Visual-only residual{sup}* {citep}[9, 21]`20260.7990.8600.8930.3432.010.8560.0380
Visual-only residual + node supervision{sup}*{sup}§20260.7710.8360.8830.3511.990.8560.0362
Correction using a predicted graph rollout
JointGV: mean graph-motion broadcastOurs0.7000.7650.8150.3221.730.8660.0326
JointGV: cosine change coupling{sup}†`Ours0.6690.7330.7930.3041.690.8690.0322
JointGV: OT change couplingOurs0.6930.7500.7910.3041.670.8690.0316

Table 1. DINO-WM results on 280 matched DMC Cheetah Run test windows. Values at horizon HH average steps 1 through HH; Token MSE denotes DINOv2-token MSE. Rows are single runs and are compared only within matched blocks. {sup}*The visual-residual controls adapt residual-correction principles rather than reproduce the cited methods. {sup}§The annotation-matched control uses the same auxiliary node targets but receives no predicted graph states or graph features. {sup}†The cosine run has its own matched backbone. Three-seed results and additional ablations are provided in {xref}Sec. S6`.

Mt,pi=∥proj⁡v(Δzˉt,p)−proj⁡g(Δn^t,i)∥22,Πtε=Sinkhorn⁡ε(Mt)(7)M_{t,pi}=\|\operatorname{proj}_{\mathrm{v}}(\Delta\bar{z}_{t,p})-\operatorname{proj}_{\mathrm{g}}(\Delta\hat{n}_{t,i})\|_2^2, \qquad \Pi_t^\varepsilon=\operatorname{Sinkhorn}_{\varepsilon}(M_t\text{)} \tag*{(7)}

where Mt∈RP×∣I∣M_t\in\mathbb{R}^{P\times|\mathcal{I}|} contains the matching costs between all visual tokens and active graph nodes, ε>0\varepsilon>0 is the entropic regularization coefficient, and Πtε∈RP×∣I∣\Pi_t^\varepsilon\in\mathbb{R}^{P\times|\mathcal{I}|} is the resulting transport plan [3]. We train the two projections using the standard debiased Sinkhorn divergence Lalign\mathcal{L}_{\mathrm{align}} [6]; implementation details are provided in the supplementary material.

The same transport plan assigns a graph-motion context to each visual token:

yt,pmot=∑i∈IΠ~t,pitanh⁡(WgvΔn^t,i)∈Rdz.(8)y_{t,p}^{\mathrm{mot}}=\sum_{i\in\mathcal{I}}\widetilde{\Pi}_{t,pi}\tanh\left(W_{gv}\Delta\hat{n}_{t,i}\right)\in\mathbb{R}^{d_z}. \tag*{(8)}

where Π~t,pi\widetilde{\Pi}_{t,pi} is the row-normalized transport weight and Wgv∈Rdz×dgW_{gv}\in\mathbb{R}^{d_z\times d_g} maps each graph-node change to the visual-token dimension. The resulting yt,pmoty_{t,p}^{\mathrm{mot}} contains the relation-aware graph change assigned to visual token pp.

Position-aware graph context. Graph–visual change alignment captures action-induced state evolution in the shared feature space but not absolute image location. JointGV therefore uses the next node positions predicted by the graph rollout to route predicted node features to nearby visual tokens, providing the spatial reference required for correction without ground-truth positions at inference. Let x^t+1,i∈[−1,1]2\hat{x}_{t+1,i}\in[-1,1]^2 be the predicted image position of node ii, and let up∈[−1,1]2u_p\in[-1,1]^2 be the fixed image-grid coordinate of visual token pp. The position-aware graph context is

yt,ppos=∑i∈Iexp⁡(−∥up−x^t+1,i∥222σpos2)⋅tanh⁡(Wposn^t+1,i).(9)y_{t,p}^{\mathrm{pos}}=\sum_{i\in\mathcal{I}}\exp\left(-\frac{\left\|u_p-\hat{x}_{t+1,i}\right\|_2^2}{2\sigma_{\mathrm{pos}}^2}\right)\cdot\tanh\left(W_{\mathrm{pos}}\hat{n}_{t+1,i}\right). \tag*{(9)}

The exponential term assigns a large weight when token pp is close to the predicted position of node ii. The projected node feature tanh⁡(Wposn^t+1,i)\tanh(\mathbf{W}_{\mathrm{pos}}\hat{\mathbf{n}}_{t+1,i}) is the graph information delivered to nearby tokens. Here, σpos>0\sigma_{\mathrm{pos}}>0 controls the spatial range over which each node influences nearby visual tokens, and Wpos∈Rdz×dg\mathbf{W}_{\mathrm{pos}}\in\mathbb{R}^{d_z\times d_g} maps the predicted node feature to the visual-token dimension.

Bounded visual correction. The motion-based and position-based graph contexts provide complementary information: yt,pmot\mathbf{y}^{\mathrm{mot}}_{t,p} identifies graph-node changes compatible with the visual-token change, whereas yt,ppos\mathbf{y}^{\mathrm{pos}}_{t,p} identifies graph nodes located near the visual token. For token pp, the correction module concatenates both contexts with the predicted visual change and the original next-token prediction:

χt,p=[Δzt,p;yt,pmot;yt,ppos;zˉt+1,p],(10)\boldsymbol{\chi}_{t,p}=\left[\Delta\mathbf{z}_{t,p};\mathbf{y}^{\mathrm{mot}}_{t,p};\mathbf{y}^{\mathrm{pos}}_{t,p};\bar{\mathbf{z}}_{t+1,p}\right], \tag*{(10)}

where χt,p\boldsymbol{\chi}_{t,p} is the complete input feature of the correction module for visual token pp.

The bounded correction for visual token pp is

ct,p=sc sigmoid⁡(fγ(χt,p))⊙[cmax⁡tanh⁡(fc(χt,p)cmax⁡)].(11)\mathbf{c}_{t,p}=s_c\,\operatorname{sigmoid}\left(f_\gamma(\boldsymbol{\chi}_{t,p})\right)\odot\left[c_{\max}\tanh\left(\frac{f_c(\boldsymbol{\chi}_{t,p})}{c_{\max}}\right)\right]. \tag*{(11)}

Here, fcf_c predicts the correction, fγf_\gamma predicts its token-wise gate, ⊙\odot denotes element-wise multiplication, sc∈[0,1]s_c\in[0,1] controls the overall correction strength, and cmax⁡>0c_{\max}>0 bounds each component of the correction. Stacking the corrections of all PP visual tokens gives Ct=[ct,1,…,ct,P]T\mathbf{C}_t=[\mathbf{c}_{t,1},\ldots,\mathbf{c}_{t,P}]^{\mathsf{T}}, which is applied to the original visual prediction through Eq. (2).

Global-vector alignment. When the visual state is a single global vector, e.g., in DreamerV3, there is no spatial token distribution on which to apply OT. In this case, proj⁡v\operatorname{proj}_v denotes a separately learned projection from the global-state dimension to the same alignment dimension dcd_c. JointGV uses the predicted global visual-state change ΔZt\Delta\mathbf{Z}_t to attend to graph-node changes. The attention weight of active node ii is

αt,i=softmax⁡i∈I(proj⁡v(ΔZt)Tproj⁡g(Δn^t,i)τ),(12)\alpha_{t,i}=\operatorname{softmax}_{i\in\mathcal{I}}\left(\frac{\operatorname{proj}_v(\Delta\mathbf{Z}_t)^{\mathsf{T}}\operatorname{proj}_g(\Delta\hat{\mathbf{n}}_{t,i})}{\tau}\right), \tag*{(12)}

where τ>0\tau>0 is the attention temperature. The pooled graph-node change and its alignment loss are

Δn^tpool=∑i∈Iαt,iΔn^t,i,\Delta\hat{\mathbf{n}}_t^{\mathrm{pool}}=\sum_{i\in\mathcal{I}}\alpha_{t,i}\Delta\hat{\mathbf{n}}_{t,i},
Lglobal=12∥proj⁡v(ΔZt)−proj⁡g(Δn^tpool)∥22.(13)\mathcal{L}_{\mathrm{global}}=\frac{1}{2}\left\|\operatorname{proj}_v(\Delta\mathbf{Z}_t)-\operatorname{proj}_g\left(\Delta\hat{\mathbf{n}}_t^{\mathrm{pool}}\right)\right\|_2^2. \tag*{(13)}

The pooled graph change is then used by the same bounded correction principle. For global correction, the predicted nodes’ projected features and two-dimensional positions are concatenated in fixed task-role order to form a global node context. This context and the pooled graph change are both supplied to the bounded, gated correction module. This adapter uses attention pooling followed by normalized half-squared ℓ2\ell_2 alignment; it does not perform a singleton Sinkhorn computation.

Training Objective

JointGV trains graph observation, graph dynamics, graph–visual alignment, and bounded correction while retaining the original visual-model objective. The uncorrected visual prediction is detached before alignment and correction, so these losses do not update the original visual encoder or dynamics. During recursive correction training, the visual model and graph dynamics are evaluated without gradients, and only the correction module is updated. The complete objective and recursive losses are provided in Sec. S4.

Experiments and Results

Experimental Setup

We evaluate DINO-WM and DreamerV3 on DMC Cheetah Run and Walker Walk across three seeds, and extend the DMC evaluation to contact-rich manipulation with Finger Turn Hard [22]. Table 1–Table 3 report DINO-WM on Cheetah Run, all four locomotion pairings over three seeds, and Finger Turn Hard; complete metrics are in Sec. S6. For DreamerV3, we reproduce its online actor–critic training, then train the backbone and JointGV on the same fixed dataset collected using the reproduced policy and random actions. This avoids exposing separately trained prediction models to different transition distributions [7]; details are provided in Sec. S5. We additionally evaluate DINO-WM, DreamerV3, and LeWorldModel on recordings of a physical inverted-pendulum system built with an RM65-B arm [16] and a rotary gear damper [20], using the same observed image history and recorded actions.

MethodH=8H=8H=16H=16H=32H=32
DINO-WM backbone [34]10.81312.47614.861
DINO-WM + JointGV9.4449.91110.754

Table 3. DINO-WM on 280 DMC Finger Turn Hard test windows. Values at HH average steps 1 through HH; additional metrics are provided in Sec. S6.

We report two-dimensional joint-position error, RGB mean squared error (MSE), structural similarity (SSIM) [25], Learned Perceptual Image Patch Similarity (LPIPS) [32], and DINOv2-token MSE for DINO-WM. Values at horizon HH average rollout steps 11 through HH. Joint positions are measured by a frozen RGB observer separate from JointGV’s graph branch. We also rescore the predictions with an independently trained heatmap-based keypoint detector to avoid dependence on the ReITR-based evaluator. Metric definitions and protocols are provided in Secs. S5 and § 12.9.

For repeated experiments, we report the mean and standard deviation across three training seeds; within each seed, 95% confidence intervals use 2,000 paired bootstrap resamples of the fixed test windows.

Controls compare longer self-fed visual training, visual-only residuals inspired by ReDRAW and Sandwich-Residuals [9, 21], mean graph-motion broadcasting, cosine and OT change coupling, and correction-gradient propagation. The residual controls adapt selected principles rather than reproduce the complete methods. Simulator annotations, and sensor-derived geometric node targets for the physical data, supervise graph observation and dynamics during training but are not supplied during recursive inference. The annotation-matched visual residual receives the same auxiliary node targets but no predicted graph states or graph features; its results are reported in Sec. S6. Actors trained with original or JointGV-corrected DINO-WM rollouts are compared in a matched MuJoCo model (Table 4).

Measure (50 unseen initial states)Original+ JointGV
Six-second completion ↑0/5046/50
Mean time before termination (s) ↑2.285.86
Shared reward-head MSE ↓0.08510.0375

Table 4. MuJoCo inverted-pendulum control. Actors are trained with original or JointGV-corrected DINO-WM rollouts; reward-head MSE is a diagnostic.

Long-Horizon Prediction with DINO-WM

Table 1 and Figure 3 show that JointGV reduces joint-position error across all reported horizons, improves the latent and decoded-image metrics, and better preserves articulated shape during rollout.

Qualitative comparison of DINO-WM and JointGV on a held-out DMC Cheetah Run sequence

Figure 3. Qualitative comparison of DINO-WM and JointGV on a held-out DMC Cheetah Run sequence. Columns show rollout steps 2, 8, 16, 24, and 32; adjacent maps show absolute RGB error amplified by four using the shared decoder. Red arrows mark representative structural differences. KvK_v denotes the self-fed visual training horizon and ω\omega the propagated correction-gradient fraction. Additional episodes and videos are provided in Sec. S8.

Longer self-fed visual training improves DINO-WM at later horizons but worsens its shorter-horizon joint-position error. JointGV improves both the Kv=1K_v = 1 and Kv=8K_v = 8 backbones, showing that its gain is not explained by longer visual training alone.

Contact-rich manipulation. On DMC Finger Turn Hard, JointGV reduces 32-step joint-position error by 27.6% and RGB MSE by 51.8%, while better preserving the finger–spinner configuration over the rollout (Table 3 and Fig. S7). Additional metrics are provided in Sec. S6.

Evidence for Graph–Visual Change Coupling

Table 1 first separates graph-conditioned correction from residual capacity and additional supervision. The visual-only residual changes the 32-step joint-position error only from 0.898 to 0.893 pixels, whereas JointGV reduces it to 0.791 pixels. Across three seeds, the corresponding reductions are 4.3% ± 4.3% and 16.0% ± 3.7%. Supplying the residual with the same node annotations, but no predicted graph, reaches only 0.883 pixels. The larger gain therefore comes from conditioning the correction on the predicted graph rollout, rather than from residual capacity or matched node supervision alone.

Mean graph-motion broadcasting improves the matched visual backbone, showing that the predicted graph rollout already provides a useful correction signal without token-specific routing. Cosine and OT coupling further reduce the 32-step error to 0.793 and 0.791 pixels, respectively. Their similar accuracy shows that the correction interface is not tied to one operator. The additional gain over broadcasting is largest when non-adjacent parts approach in image space, where assigning differentiated graph motion becomes more important. An independently trained heatmap-based keypoint network also measures a 13.6% reduction and finds improvement on 95.4% of the test windows. Complete paired comparisons and confidence intervals, and stratified analyses are provided in Secs. S6 and S6.8.

Portability to a Global Latent World Model

Table 2 evaluates JointGV with spatial-token and global-latent states on Cheetah Run and Walker Walk. It improves every model–task pairing in all three seeds, reducing average 32-step joint-position error by 11.1–28.0%. For DreamerV3, graph-node changes are pooled using the global visual-state change rather than spatial-token OT, showing that the correction principle does not depend on a visual-token grid. Figure 4 shows the corresponding Walker Walk rollout.

Qualitative comparison on a held-out DMC Walker Walk validation sequence

Figure 4. Qualitative comparison on a held-out DMC Walker Walk validation sequence. Columns show rollout steps 2, 8, 16, 24, and 32, with absolute RGB error amplified by a factor of four. JointGV uses the DreamerV3 global-state change to pool the graph-node changes and applies the result as a bounded global correction. The corrected rollout better preserves the articulated pose at later steps.

Visual world modelVisual stateTaskOriginal+ JointGVReduction (%) ↑
DINO-WMSpatial tokensCheetah Run0.942 ± 0.0460.790 ± 0.01116.0 ± 3.7
DreamerV3Global latentCheetah Run0.467 ± 0.0090.415 ± 0.00911.1 ± 2.6
DINO-WMSpatial tokensWalker Walk10.740 ± 0.9227.682 ± 0.15128.0 ± 7.8
DreamerV3Global latentWalker Walk5.835 ± 0.0934.336 ± 0.01925.7 ± 1.3

Table 2. JointGV across visual states and control tasks. Values are average joint-position errors over rollout steps 1–32, reported as mean and standard deviation across three independently trained seeds. Every model–task pairing improves in all three seeds. Per-horizon and visual metrics are provided in Sec. S6.

The per-window distributions in Figure 5 further show that these averages are not driven by a few favorable rollouts. JointGV improves 277 of 280 DINO-WM Cheetah Run windows, 186 of 280 DreamerV3 Cheetah Run windows, 250 of 280 DINO-WM Walker Walk windows, and 232 of 280 DreamerV3 Walker Walk windows. The broader DreamerV3 distributions also show that the correction magnitude remains task dependent, despite a consistent mean improvement across all three seeds.

Per-window effect across four model–task pairings

Figure 5. Per-window effect across four model–task pairings. Each value is the relative change in 32-step average joint-position error from the original prediction to JointGV; values below zero indicate improvement. JointGV improves 99%, 66%, 89%, and 83% of the respective test windows, showing that the aggregate gains are not produced by a small subset of rollouts.

Inverted-Pendulum Prediction and Control

Real-robot prediction. On 156 matched windows from real-robot sequences, JointGV reduces DINO-WM’s average 32-step pole-angle error from 12.56∘12.56^\circ to 6.69∘6.69^\circ and its pole-angle-change error from 3.09∘3.09^\circ to 1.46∘1.46^\circ, obtaining the lower change error on 82.1% of the windows. DreamerV3 and LeWorldModel [13] also improve.

Simulated control. We test whether the prediction gains support control in a MuJoCo model of the same RM65-B inverted-pendulum system. Two actors are trained in imagination with identical action constraints and a shared reward predictor; they differ only in using original or JointGV-corrected DINO-WM rollouts. On 50 unseen initial states, the JointGV-rollout actor completes 46 six-second episodes, compared with none for the original-rollout actor, and its mean time before termination rises from 2.28 to 5.86 s (Table 4).

Physical trial. Figure 6 shows a preliminary physical control trial from comparable initial pole angles. After 1.27 seconds, the original-rollout actor reaches −83.3∘-83.3^\circ, whereas the JointGV-rollout actor remains at −2.7∘-2.7^\circ. Stable six-second physical balancing has not yet been achieved because high-speed operation introduces non-negligible command delay.

Preliminary physical RM65-B control trial

Figure 6. Preliminary physical RM65-B control trial. The two actors start from comparable pole angles. At 1.27 seconds, the pole controlled using the original visual-world-model rollout has fallen to −83.3∘-83.3^\circ, whereas the actor using the JointGV-corrected rollout keeps it near upright at −2.7∘-2.7^\circ. This trial demonstrates short-duration physical deployment rather than stable six-second balancing.

and Table 4 as the aggregate control evaluation; further details are provided in Sec. S7.2.

Discussion and Conclusion

JointGV improves long-horizon prediction across spatial-token and global-latent visual models, DMC articulated locomotion, contact-rich DMC manipulation, and real-robot sequences. These results support the central cross-granularity hypothesis: graph nodes and visual states without shared elements can still be coupled through changes induced by the same action. The stronger coupling advantage when non-adjacent parts approach further identifies where learned graph–visual correspondence is most useful.

The coupling analysis clarifies the role of graph–visual alignment. Cosine and OT coupling obtain similar overall accuracy, indicating that the correction interface is not tied to a particular operator. Both outperform broadcasting the same mean graph motion to every visual token, and this difference grows when non-adjacent parts approach in image space. The benefit therefore comes from associating visual changes with differentiated graph motion, particularly when the projected structure becomes ambiguous.

The visual-only and supervision-matched residual controls show that the gain is not explained by correction capacity or additional node annotations. Longer visual training also does not reproduce JointGV’s short- and long-horizon improvements, while propagating correction gradients into the original visual dynamics degrades their native prediction. JointGV therefore uses the graph rollout as an external, gradient-isolated correction source rather than another branch of the original latent pathway.

JointGV adds computation and currently assumes supervised task-defined node roles. Its correction also depends on graph observation and prediction quality; overlapping links can cause node-localization errors. Automatic role discovery, confidence-aware correction, broader visual models, and stable physical closed-loop deployment remain future work. The current hardware system exhibits non-negligible control-command delay at high speed, preventing reliable six-second physical balancing despite the improved visual predictions.

Overall, JointGV provides a plug-in interface through which separately predicted relational dynamics correct long-horizon visual rollouts without restructuring the original visual world model.

References

Detailed Scene Graph Construction

This section provides the implementation details omitted from § 3.2. The observer constructs Gt=(Nt,Xt,Rt)G_t = (N_t, X_t, R_t) from the current RGB observation. Simulator states, per-frame object identities, and relation annotations are training targets only and are not supplied at inference.

RelTR Features and Task-Role Binding

Given RGB image oto_t, the pretrained Relation Transformer (RelTR) [2] provides sparse semantic features St∈RMs×dsS_t \in\mathbb{R}^{M_s \times d_s} and dense spatial features Dt=[dt,1,…,dt,Md]⊤∈RMd×ddD_t = [d_{t,1}, \ldots, d_{t,M_d}]^\top\in\mathbb{R}^{M_d \times d_d}. Here, MsM_s and MdM_d are the numbers of sparse and dense features, while dsd_s and ddd_d are their dimensions. Dense feature dt,md_{t,m} has a fixed normalized image coordinate um∈[−1,1]2\mathbf{u}_m \in[-1,1]^2.

RelTR’s final predicate predictions do not preserve the fixed task roles required by recursive graph prediction. We therefore retain its intermediate features and bind them to JJ task-defined role slots. Role ii has a learned query qi∈Rdg\mathbf{q}_i \in\mathbb{R}^{d_g}, where dgd_g is the node-feature dimension. At inference, the query locates its assigned role from RGB features without simulator states or per-frame object labels.

The sparse and dense attention operations follow standard multi-head and scaled dot-product attention [23]. A first pass with qi\mathbf{q}_i reads the current RGB features and produces an anchor feature ht,i(0)∈Rdg\mathbf{h}_{t,i}^{(0)} \in\mathbb{R}^{d_g}. This feature determines how much information the query receives from the previous node.

Bounded Temporal Query Correction

Each role is located again from the current RGB frame. The previous node supplies a bounded hint when the role is briefly occluded. Let nt−1,iLN=LN⁡(nt−1,i)\mathbf{n}_{t-1,i}^{\mathrm{LN}} = \operatorname{LN}(\mathbf{n}_{t-1,i}) be the layer-normalized previous node. The history-conditioned query is

γt,io=sigmoid⁡(fo[ht,i(0),nt−1,iLN]),\gamma_{t,i}^{o} = \operatorname{sigmoid}\left(f_o\left[\mathbf{h}_{t,i}^{(0)},\mathbf{n}_{t-1,i}^{\mathrm{LN}}\right]\right),
q~t,i=qi+socoγt,iotanh⁡(Whnt−1,iLN).(14)\widetilde{\mathbf{q}}_{t,i} = \mathbf{q}_i + s_o c_o \gamma_{t,i}^{o} \tanh\left(\mathbf{W}_h\mathbf{n}_{t-1,i}^{\mathrm{LN}}\right). \tag*{(14)}

Here, the superscript oo identifies the observer gate, fof_o predicts its scalar value, Wh∈Rdg×dg\mathbf{W}_h \in\mathbb{R}^{d_g \times d_g} projects the previous node, so∈[0,1]s_o \in[0,1] controls the historical contribution, and co>0c_o > 0 bounds it. Setting so=0s_o = 0 recovers independent per-frame role binding.

The history-conditioned query binds the RelTR features to task role ii:

ht,i=MHA⁡(q~t,i,St,St)+Attn⁡(q~t,i,Dt,Dt),(15)\mathbf{h}_{t,i} = \operatorname{MHA}(\widetilde{\mathbf{q}}_{t,i}, S_t, S_t) + \operatorname{Attn}(\widetilde{\mathbf{q}}_{t,i}, D_t, D_t), \tag*{(15)}

where the two terms provide semantic and spatial evidence, respectively.

The updated query reads the sparse and dense RelTR features again. Adding their attention outputs gives role feature ht,i\mathbf{h}_{t,i}. Let At,i,mA_{t,i,m} be its normalized dense-attention weight at location mm. The node position is

xt,i=clip⁡(∑m=1MdAt,i,mum+cxtanh⁡(fx(ht,i)),−1,1),(16)\mathbf{x}_{t,i} = \operatorname{clip}\left(\sum_{m=1}^{M_d} A_{t,i,m}\mathbf{u}_m + c_x\tanh\left(f_x(\mathbf{h}_{t,i})\right), -1, 1\right), \tag*{(16)}

where fx:Rdg→R2f_x:\mathbb{R}^{d_g}\to\mathbb{R}^2 predicts a position offset and cx>0c_x>0 bounds its magnitude.

Inter-Node Exchange and Relation Logits

The role queries initially read the image independently. We then apply one round of Iterative Message Passing (IMP) [30], allowing each active node to receive information from the other task-relevant nodes. For an ordered pair with subject node ss and object node oo, the initial pair feature is

et,s,o0=fe[ht,s;ht,o;xt,o−xt,s],(17)\mathbf{e}_{t,s,o}^{0} = f_e[\mathbf{h}_{t,s};\mathbf{h}_{t,o};\mathbf{x}_{t,o}-\mathbf{x}_{t,s}], \tag*{(17)}

where fef_e is a learned multilayer perceptron (MLP). A shared MLP sends message mt,s→o=fm([ht,s;et,s,o0])\mathbf{m}_{t,s\to o}=f_m([\mathbf{h}_{t,s};\mathbf{e}_{t,s,o}^{0}]). The received messages update node oo:

m‾t,o=1max⁡(Ja−1,1)∑s∈Is≠omt,s→o,(18)\overline{\mathbf{m}}_{t,o} = \frac{1}{\max(J_a-1,1)} \sum_{\substack{s\in\mathcal{I}\\s\ne o}}\mathbf{m}_{t,s\to o}, \tag*{(18)}
nt,o=LN⁡[GRU⁡(m‾t,o,ht,o)].\mathbf{n}_{t,o} = \operatorname{LN}[\operatorname{GRU}(\overline{\mathbf{m}}_{t,o},\mathbf{h}_{t,o})].

Here, I\mathcal{I} is the active-node set, Ja=∣I∣J_a=|\mathcal{I}| is its size, GRU is a gated recurrent unit, and LN is layer normalization.

IMP updates node features but does not assign semantic relations. For relation class kk, we use the VTransE-inspired translation score

Tk(nt,s,nt,o)=−1dg∥Wsnt,s+ρk−Wont,o∥22,(19)T_k(\mathbf{n}_{t,s},\mathbf{n}_{t,o}) = -\frac{1}{d_g}\left\|\mathbf{W}_s\mathbf{n}_{t,s}+\boldsymbol{\rho}_k-\mathbf{W}_o\mathbf{n}_{t,o}\right\|_2^2, \tag*{(19)}

where Ws\mathbf{W}_s and Wo\mathbf{W}_o project the subject and object features and ρk\boldsymbol{\rho}_k is the learned translation for relation class kk. We add a learned pair-specific residual:

rres=fr[nt,s;nt,o;et,s,o0;xt,o−xt,s],(20)\mathbf{r}^{\mathrm{res}} = f_r[\mathbf{n}_{t,s};\mathbf{n}_{t,o};\mathbf{e}_{t,s,o}^{0};\mathbf{x}_{t,o}-\mathbf{x}_{t,s}], \tag*{(20)}
[Rt]s,o,k=Tk(nt,s,nt,o)+[rres]k.[\mathbf{R}_t]_{s,o,k} = T_k(\mathbf{n}_{t,s},\mathbf{n}_{t,o}) + [\mathbf{r}^{\mathrm{res}}]_k.

The shared MLP frf_r outputs one residual for each of the KrK_r relation classes. Softmax over kk gives the relation distribution for this ordered pair; class k=0k=0 denotes no relation.

Detailed History-Aware Graph Dynamics

This section expands the message passing and bounded graph update summarized in § 3.3. The history input is defined in Eq. (4).

Relation-Aware Message Passing

For ordered pair (s,o)(s,o), let pt,s,o=softmax⁡([Rt]s,o,:)\mathbf{p}_{t,s,o}=\operatorname{softmax}([\mathbf{R}_t]_{s,o,:}) be its distribution over the KrK_r relation classes. We set βt,s,o=1−[pt,s,o]0\beta_{t,s,o}=1-[\mathbf{p}_{t,s,o}]_0 and et,s,o=WR(ℓ)pt,s,o\mathbf{e}_{t,s,o}=W_R^{(\ell)}\mathbf{p}_{t,s,o}, where WR(ℓ):RKr→RdgW_R^{(\ell)}:\mathbb{R}^{K_r}\to\mathbb{R}^{d_g} is learned at graph layer ℓ\ell. The no-relation probability controls message strength, while the projected distribution supplies the relation feature. The action-conditioned message and node update are

mt,s→o(ℓ)=βt,s,ofm(ℓ)[gt,s(ℓ),gt,o(ℓ),et,s,o(ℓ),at],\mathbf{m}_{t,s\to o}^{(\ell)}=\beta_{t,s,o}f_m^{(\ell)}\left[\mathbf{g}_{t,s}^{(\ell)},\mathbf{g}_{t,o}^{(\ell)},\mathbf{e}_{t,s,o}^{(\ell)},\mathbf{a}_t\right],
m‾t,o(ℓ)=∑s≠omt,s→o(ℓ)∑s≠oβt,s,o+εe,\overline{\mathbf{m}}_{t,o}^{(\ell)}=\frac{\sum_{s\ne o}\mathbf{m}_{t,s\to o}^{(\ell)}}{\sum_{s\ne o}\beta_{t,s,o}+\varepsilon_e},
gt,o(ℓ+1)=LN⁡[gt,o(ℓ)+fu(ℓ)[gt,o(ℓ),m‾t,o(ℓ),at]].(21)\mathbf{g}_{t,o}^{(\ell+1)}=\operatorname{LN}\left[\mathbf{g}_{t,o}^{(\ell)}+f_u^{(\ell)}\left[\mathbf{g}_{t,o}^{(\ell)},\overline{\mathbf{m}}_{t,o}^{(\ell)},\mathbf{a}_t\right]\right]. \tag*{(21)}

Here, εe>0\varepsilon_e>0 prevents division by zero, and fm(ℓ)f_m^{(\ell)} and fu(ℓ)f_u^{(\ell)} are learned MLPs. This update is repeated for LGL_G graph layers.

Bounded Graph Update and Self-Fed Rollout

The final graph features predict bounded node and relation-logit increments:

Δn^t,i=cntanh⁡(fn(gt,i(LG))),\Delta\widehat{\mathbf{n}}_{t,i}=c_n\tanh\left(f_n\left(\mathbf{g}_{t,i}^{(L_G)}\right)\right),
Δr^t,s,o=crtanh⁡(fR[gt,s(LG);gt,o(LG);at]).(22)\Delta\widehat{\mathbf{r}}_{t,s,o}=c_r\tanh\left(f_R\left[\mathbf{g}_{t,s}^{(L_G)};\mathbf{g}_{t,o}^{(L_G)};\mathbf{a}_t\right]\right). \tag*{(22)}

Here, fnf_n and fRf_R decode the increments, while cn>0c_n>0 and cr>0c_r>0 bound each component. Stacking these outputs gives the updates in Eq. (5). The position decoder fpos:Rdg→R2f_{\mathrm{pos}}:\mathbb{R}^{d_g}\to\mathbb{R}^2 gives x^t+1,i=tanh⁡(fpos(n^t+1,i))\widehat{\mathbf{x}}_{t+1,i}=\tanh(f_{\mathrm{pos}}(\widehat{\mathbf{n}}_{t+1,i})).

The KK-step training rollout follows Eq. (6). Its first prediction uses an observed graph-history window. Each later prediction uses the previous predicted graph instead of an observed future graph. Observed future graphs and simulator states serve only as training targets.

Detailed Graph–Visual Alignment and Correction

This section provides implementation details omitted from § 3.4. Sinkhorn iterations follow Cuturi [3]; we do not repeat their standard derivation.

Spatial-Token Alignment

The projection functions in Eq. (7) are learned bias-free linear layers followed by optional set centering and safe row-wise ℓ2\ell_2 normalization. They map visual-token changes Δzt,p∈Rdz\Delta\mathbf{z}_{t,p}\in\mathbb{R}^{d_z} and graph-node changes Δn^t,i∈Rdg\Delta\widehat{\mathbf{n}}_{t,i}\in\mathbb{R}^{d_g} into the same dcd_c-dimensional space.

We use the debiased Sinkhorn divergence of Feydy et al. [6] as Lalign\mathcal{L}_{\mathrm{align}} and clamp small negative numerical values to zero. The variance-floor loss Lvar\mathcal{L}_{\mathrm{var}} penalizes a projected dimension when its sample standard deviation falls below γσ>0\gamma_\sigma>0.

For each visual token, transport weights are normalized over active graph nodes before the motion context in Eq. (8) is computed. The position context in Eq. (9) is computed independently from graph-predicted node positions and does not come from the transport plan. The two contexts are concatenated in Eq. (10). The gate in Eq. (11) is one scalar per spatial token and is broadcast over its feature dimensions.

Global-Vector Alignment

For a global visual state, Eqs. (12) and (13) define attention pooling and the alignment loss. The global visual-state change is the query, graph-node changes are the keys, and the resulting weights pool the unprojected node changes. Since the projected changes are ℓ2\ell_2-normalized, the half-squared ℓ2\ell_2 alignment is equivalent to cosine distance. This adapter does not perform singleton Sinkhorn.

The global correction receives the original global prediction, its predicted change, the pooled graph-node change, and the node features and predicted positions concatenated in fixed task-role order. It applies the same bounded residual principle as the spatial-token path.

Detailed Training Objective

The complete training objective is

L=LWM+λobsLobs+λg,1Lg,1+λalignLalign+λvarLvar+λemb,KLemb,K+λstate,KLstate,K+λposLpos+λc,1Lc,1+λc,KLc,K.(23)\mathcal{L}=\mathcal{L}_{\mathrm{WM}}+\lambda_{\mathrm{obs}}\mathcal{L}_{\mathrm{obs}}+\lambda_{g,1}\mathcal{L}_{g,1}+\lambda_{\mathrm{align}}\mathcal{L}_{\mathrm{align}} +\lambda_{\mathrm{var}}\mathcal{L}_{\mathrm{var}}+\lambda_{\mathrm{emb},K}\mathcal{L}_{\mathrm{emb},K}+\lambda_{\mathrm{state},K}\mathcal{L}_{\mathrm{state},K} +\lambda_{\mathrm{pos}}\mathcal{L}_{\mathrm{pos}}+\lambda_{c,1}\mathcal{L}_{c,1}+\lambda_{c,K}\mathcal{L}_{c,K}. \tag*{(23)}

The terms supervise the original visual model, scene-graph observation and prediction, graph–visual alignment, node positions, and visual correction; each λ\lambda is a non-negative weight. For a global latent state, Lglobal\mathcal{L}_{\mathrm{global}} replaces Lalign\mathcal{L}_{\mathrm{align}}. The stop-gradient operator sg⁡(⋅)\operatorname{sg}(\cdot) identifies a fixed target that is not updated by the corresponding loss.

Visual and Graph Targets

For DINO-WM, the optional motion weight for visual token pp is

w~t,p=∥zt+1,p−zt,p∥2+εm,\widetilde{w}_{t,p}=\lVert\mathbf{z}_{t+1,p}-\mathbf{z}_{t,p}\rVert_2+\varepsilon_m,
wt,p=w~t,pP−1∑q=1Pw~t,q.(24)w_{t,p}=\frac{\widetilde{w}_{t,p}}{P^{-1}\sum_{q=1}^{P}\widetilde{w}_{t,q}}. \tag*{(24)}

Here, εm>0\varepsilon_m>0 retains a contribution from static tokens, and qq indexes the tokens in the denominator. The DINO-WM token loss applies weighted Smooth-L1L_1 error between original prediction zt+1,p\mathbf{z}_{t+1,p} and fixed target zt+1,p\mathbf{z}_{t+1,p}. DreamerV3 retains its native recurrent state-space-model objective. We denote the original visual model objective by LWM\mathcal{L}_{\mathrm{WM}}.

Let bt,i∈{0,1}b_{t,i} \in\{0,1\} indicate whether role ii is present and supervised. The observer loss Lobs\mathcal{L}_{\mathrm{obs}} uses masked Smooth-L1L_1 losses for physical-state readouts and image positions, binary cross-entropy for presence, and cross-entropy for relation classes. Simulator annotations supervise the observer but are not its inputs.

The KK-step graph-embedding loss is

Lemb,K=∑k=1K∑i=1Jbt+k,i∥n^t+k,i−sg⁡(nt+k,i)∥22dg∑k=1K∑i=1Jbt+k,i+εb.(25)\mathcal{L}_{\mathrm{emb},K} = \frac{ \sum_{k=1}^{K}\sum_{i=1}^{J} b_{t+k,i}\left\lVert\hat{\mathbf{n}}_{t+k,i}-\operatorname{sg}(\mathbf{n}_{t+k,i})\right\rVert_{2}^{2}} { d_g\sum_{k=1}^{K}\sum_{i=1}^{J} b_{t+k,i}+\varepsilon_b}. \tag*{(25)}

where εb>0\varepsilon_b>0 prevents division by zero. The one-step graph loss Lg,1\mathcal{L}_{g,1} is its K=1K=1 counterpart. When physical-state annotations are available, Lstate,K\mathcal{L}_{\mathrm{state},K} supervises an auxiliary readout from predicted nodes. The position loss Lpos\mathcal{L}_{\mathrm{pos}} supervises observed, one-step predicted, and recursively predicted node positions.

Recursive Correction Training

The one-step correction loss Lc,1\mathcal{L}_{c,1} compares the corrected prediction with a fixed next-state target. For spatial visual tokens, the KcK_c-step correction loss is

Lc,Kc=1KcP∑k=1Kc∑p=1Pwt+k−1,p⋅SL1⁡(z^t+k,p,sg⁡(zt+k,p)).(26)\mathcal{L}_{c,K_c} = \frac{1}{K_cP}\sum_{k=1}^{K_c}\sum_{p=1}^{P}w_{t+k-1,p} \cdot\operatorname{SL1}\left(\hat{\mathbf{z}}_{t+k,p},\operatorname{sg}(\mathbf{z}_{t+k,p})\right). \tag*{(26)}

The corrected prediction is fed into the following visual transition. The original visual world model and graph dynamics are evaluated without gradients during this correction rollout, so the loss updates only the correction pathway.

The Kv=8K_v=8 control follows a matched recursive schedule but disables graph correction and feeds the visual model its own predictions. It tests whether longer visual training alone explains JointGV’s improvement.

Implementation and Evaluation Details

Implementation Settings

The primary DMC runs were trained on an NVIDIA GeForce RTX 5090 D v2 GPU with 24 GB of memory; additional seeds were trained on RTX 4090 and 22 GB RTX 2080 Ti GPUs using the same model and optimization settings. All DMC tasks use eight graph slots of dimension dg=64d_g=64, three graph message-passing layers, and a graph-history window of Hg=4H_g=4. Graph dynamics and correction use K=Kc=8K=K_c=8. Node-feature increments and visual corrections are bounded by cn=0.25c_n=0.25 and cmax⁡=1.0c_{\max}=1.0.

For spatial-token alignment, dc=32d_c=32, the entropic regularization coefficient is ϵ=0.1\epsilon=0.1, Sinkhorn runs for 20 iterations, and σpos=0.15\sigma_{\mathrm{pos}}=0.15. The alignment-loss weight starts at 0.1 and is adjusted every 50 minibatches using an exponential moving average of gradient norms. DINO-WM and DreamerV3 use a learning rate of 10−410^{-4} and batch size 16. DINO-WM uses AdamW with 16-frame sequences; DreamerV3 uses Adam with 64-frame sequences. DINOv2-S/14 and ReiTR inputs are resized to 224×224224 \times224 pixels.

We first verify the DreamerV3 implementation using its online actor–critic protocol; the retained Walker Walk agent achieves an evaluation return of 737.8±15.2737.8 \pm15.2. We then freeze this policy and use it to collect a fixed dataset for the prediction experiments. In each data split, 80% of the episodes are collected with this policy and 20% use uniformly random actions to broaden the motion coverage. The DreamerV3 backbone and JointGV correction reported in this paper are trained offline from scratch on these fixed trajectories; the online checkpoint is used only for data collection. This protocol ensures that the original and corrected predictions are trained and evaluated on identical observations and actions, without confounding the comparison with online exploration, actor learning, or changes in the replay distribution.

Rollout Protocol

All compared methods receive the same observed frames and recorded actions. A rollout begins with three preceding RGB frames and the rollout-origin frame, followed by 32 recursive predictions. DINO-WM uses three visual-history frames and JointGV uses four graph-history frames. DreamerV3 filters the four observed context frames through its recurrent state-space-model posterior. No future RGB observation, graph, or simulator state is supplied after prediction begins.

The Cheetah Run and Walker Walk test sets each contain 20 episodes and 14 fixed windows per episode, giving 280 matched windows per task. Stored RGB images are 64 ×\times 64 pixels. Predictions within an experiment use the same decoder for visualization and image-based evaluation.

Metric Computation

At horizon HH, each metric is averaged over prediction steps 1 through HH within a window and then across windows. RGB MSE compares a decoded prediction with its target image. SSIM uses an 11 ×\times 11 Gaussian window with standard deviation 1.5. LPIPS uses its AlexNet feature network. DINOv2-token MSE compares predicted tokens with frozen-encoder tokens extracted from the observed future image.

Joint-position error measures articulated structure. The same frozen RGB observer extracts normalized two-dimensional positions from every predicted RGB frame; distances to target positions are averaged over visible nodes and converted to pixels. This evaluation observer was trained separately from JointGV’s internal graph observer and does not access JointGV’s graph features, predicted positions, or relations. Applying it to observed DMC frames gives measurement floors of 0.19 pixels for Cheetah Run and 1.46 pixels for Walker Walk. Graph-only errors use graph-predicted positions directly and therefore have a different measurement path.

Computational Cost

The DINO-WM backbone contains 22.7M parameters, of which 0.326M are trainable. JointGV adds 64.4M parameters through the graph branch and its RelTR-based observer, of which 0.531M are trainable. JointGV also increases rollout time because the graph observer, recursive graph dynamics, and correction are evaluated alongside the visual backbone. The graph observer is evaluated once on the context frames, while recursive graph dynamics and correction account for most of the additional cost.

Additional Experiments and Ablation Studies

Visual Training, Graph Prediction, and Gradient Coupling

Increasing the visual training horizon from Kv=1K_v = 1 to Kv=8K_v = 8 improves the original DINO-WM prediction at later steps, but JointGV further improves either model (Table 1).

We also test whether the correction loss should update the original visual dynamics. The detached plug-in uses ω=0\omega= 0, while partial and full coupling use ω=0.25\omega= 0.25 and ω=1\omega= 1. Table S1 separately reports the original visual output and corrected output from each coupled run. Gradient coupling does not improve upon the detached plug-in and degrades the original visual output. Graph-only rows measure recursive graph prediction before graph information is transferred to the visual state.

OutputJoint-position error (px) ↓DINOv2 token MSE ↓RGB MSE×10310^3 ↓SSIM ↑LPIPS ↓
H=8H=8H=16H=16H=32H=32H=32H=32H=32H=32H=32H=32H=32H=32
Detached plug-in: corrected output (ω=0\omega=0)0.7300.7960.8520.3411.840.8610.0351
Full gradient coupling: original visual output (ω=1\omega=1)0.9921.1841.3400.9032.960.7860.0592
Full gradient coupling: corrected output (ω=1\omega=1)0.7380.8120.8820.3811.970.8530.0374
Partial gradient coupling: original visual output (ω=0.25\omega=0.25)0.9251.0391.1530.7852.580.8130.0532
Partial gradient coupling: corrected output (ω=0.25\omega=0.25)0.7710.8380.8900.4491.990.8460.0364
Graph-only prediction (Kv=1K_v=1)0.5200.6090.704––––
Graph-only prediction (Kv=8K_v=8)0.5350.6300.754––––

Table S1. Gradient-coupling and graph-only controls. “Original visual output” is the native prediction from the visual model trained with the indicated correction-gradient fraction; “corrected output” includes JointGV. Graph-only errors are measured directly from graph-predicted positions and are not ranked against RGB-observer errors.

Correction and Change Alignment

The visual-only controls test whether residual capacity or additional node supervision explains the improvement. Mean graph-motion broadcasting tests whether the predicted graph rollout is useful without token-specific coupling. The cosine and OT variants then test two operators for coupling graph-node and visual-token changes; Table S2 reports the matched comparisons.

Correction variantOwn backboneCorrectedReduction
Residual controls
Visual-only residual0.89780.89350.0043
Visual-only residual + node supervision0.89780.88300.0148
Graph-conditioned correction
Mean graph-motion broadcast0.89780.81460.0832
Cosine change coupling0.90210.79340.1087
OT change coupling0.89780.79060.1072

Table S2. Matched within-run correction ablations on DMC Cheetah Run. Values are joint-position errors averaged over rollout steps 1–32. Each corrected result is compared with the visual backbone from the same training run; Reduction denotes the absolute decrease from that backbone. The visual-only controls test residual capacity and matched node supervision without providing a predicted graph. Mean broadcasting tests the graph correction signal without token-specific coupling, while cosine and OT couple graph-node and visual-token changes.

Across three seeds, the visual-only controls provide substantially smaller improvements than JointGV. Mean graph-motion broadcasting also improves the backbone, showing that the predicted graph rollout contains useful correction information. Cosine and OT coupling provide a further reduction by assigning differentiated graph motion to the visual tokens, while their similar accuracy shows that this interface does not depend on a single coupling operator.

Inter-Node Message Passing

The no-message-passing control disables observer IMP and cross-node message passing in graph dynamics. Its corrected 32-step joint-position error is 0.8588 pixels, compared with 0.9646 pixels for its own backbone. Full JointGV reaches 0.7906 pixels with its own backbone at 0.8978 pixels. The no-message-passing run therefore has higher absolute error, although its separately trained backbone prevents the difference from isolating only node-to-node communication.

Training Seeds and Paired Bootstrap

For DINO-WM on Cheetah Run, we repeat JointGV and the visual-only residual control with three independently trained seeds. Each corrected prediction is compared with the backbone from the same run on the same 280 test windows. We use 2,000 paired bootstrap resamples of these windows within each seed. These intervals measure test-window sampling variation, whereas the standard deviation across seeds measures training variation.

Table S3 shows that JointGV improves its corresponding backbone in every seed and produces a larger and more consistent reduction than the visual-only residual. The visual-only residual also improves two of the three runs, but its reduction is smaller and more variable; its seed-0 confidence interval includes zero. We also repeat the primary DreamerV3–Walker Walk comparison with three seeds. JointGV reduces the average 32-step joint-position error from 5.835±0.0935.835 \pm0.093 to 4.336±0.0194.336 \pm0.019 pixels, corresponding to a relative reduction of 25.7±1.3%25.7 \pm1.3\%. The reductions for seeds 0, 1, and 2 are 27.0%, 25.7%, and 24.4%, respectively.

MethodSeedBackboneCorrectedReduction95% CI
JointGV00.89780.790611.95%[11.08, 12.72]%
JointGV10.93770.779016.92%[16.17, 17.70]%
JointGV20.98990.800119.18%[18.20, 20.10]%
JointGVMean±std0.9418±0.04620.7899±0.010616.01±3.70%–
Visual-only residual00.89780.89350.49%[−0.27, 1.23]%
Visual-only residual10.95160.91943.38%[2.46, 4.35]%
Visual-only residual20.99980.90949.04%[8.45, 9.63]%
Visual-only residualMean±std0.9497±0.05100.9074±0.01314.30±4.35%–

Table S3. Three-seed results on the 280 fixed DMC Cheetah Run test windows. Backbone and corrected columns report average joint-position error over rollout steps 1–32. Reduction is computed relative to the corresponding backbone from the same run. Confidence intervals are obtained by paired bootstrap resampling within each seed; mean and standard deviation summarize variation across independently trained seeds.

Reversed Backbone–Task Pairings

We additionally evaluate DINO-WM on Walker Walk and DreamerV3 on Cheetah Run, reversing the backbone–task pairings in the main comparisons. Table S4 reports representative matched runs for all four pairings.

Model / taskOutputJoint-position error (px) ↓\downarrowRGB MSE×103\times10^3 ↓\downarrowSSIM↑\uparrowLPIPS↓\downarrow
H=8H=8H=16H=16H=32H=32H=32H=32H=32H=32H=32H=32
DINO-WM / Cheetah RunOriginal0.8300.9311.0362.310.8410.0435
+ JointGV0.7300.7960.8521.840.8610.0351
DreamerV3 / Cheetah RunOriginal0.3780.4270.4700.9780.9290.0178
+ JointGV0.3550.3940.4250.7980.9390.0153
DINO-WM / Walker WalkOriginal7.8079.62610.95313.520.5680.2705
+ JointGV6.3686.9967.67111.740.6250.2382
DreamerV3 / Walker WalkOriginal4.0634.7505.93815.120.6210.1722
+ JointGV3.7854.0224.33811.150.6940.1231

Table S4. Representative matched runs for all four DMC model–task pairings. Values at horizon HH average rollout steps 1 through HH. The three-seed joint-position results are summarized in Table 2.

Across three independently trained seeds, JointGV reduces the average 32-step joint-position error of DINO-WM on Walker Walk from 10.740 ±\pm 0.922 to 7.682 ±\pm 0.151 pixels, corresponding to a relative reduction of 28.0 ±\pm 7.8%. For DreamerV3 on Cheetah Run, the error decreases from 0.467 ±\pm 0.009 to 0.415 ±\pm 0.009 pixels, corresponding to a reduction of 11.1 ±\pm 2.6%. The improvement occurs in every seed for both reversed pairings.

Representative rollouts for DreamerV3 on Cheetah Run and DINO-WM on Walker Walk are shown in Figure S5 and Figure S6, respectively.

DreamerV3 rollout on a held-out DMC Cheetah Run episode

Figure S5. DreamerV3 rollout on a held-out DMC Cheetah Run episode. JointGV better preserves the articulated configuration and reduces visual error as the rollout progresses, showing that the correction also applies to this reversed backbone–task pairing.

DINO-WM rollout on a held-out DMC Walker Walk episode

Figure S6. DINO-WM rollout on a held-out DMC Walker Walk episode. The original prediction becomes increasingly blurred and loses limb definition, whereas JointGV retains a clearer articulated pose over the rollout.

Per-Step and Per-Window Analyses

The per-step curves in Figure S1 show that JointGV’s improvement is distributed across the rollout rather than arising only at the final prediction step. The corresponding per-window distributions are reported in Figure 5 of the main paper.

Per-step joint-position error curves for DINO-WM and DreamerV3 task pairing

Figure S1. Per-step joint-position error for one run of each DINO-WM and DreamerV3 task pairing. Dashed curves show the original predictions and solid curves show JointGV-corrected predictions. Shaded regions are episode-level 95% bootstrap intervals over the fixed test windows. JointGV reduces structural error over most of the rollout rather than only at its final step.

Supervision Scope

JointGV uses simulator-provided node identities, visibility, positions, and task relations as training targets for graph construction and graph prediction. The visual-only residual in Table 1 receives neither graph features nor node annotations. It therefore tests whether a generic bounded residual explains the gain, but does not by itself isolate the effect of additional structured supervision.

To control for supervision volume, we additionally train a visual-only residual with an auxiliary node head supervised by the same MuJoCo node annotations as JointGV. The correction module does not receive the predicted node states or graph features. The supervision-matched residual reduces the average 32-step joint-position error from per. 0.8978 to 0.8830 pixels, a 1.65% reduction with a paired-bootstrap 95% interval of [0.95, 2.33]%. JointGV reduces the same backbone error to 0.7906 pixels, corresponding to an 11.95% reduction. The additional node supervision alone therefore does not explain JointGV’s gain.

Stratified Analysis

We analyze when graph–visual coupling provides the largest additional benefit. For each DINO-WM test window, we compute two quantities over rollout steps 1–32: (i) the minimum ground-truth image distance between non-adjacent nodes, which measures how closely separate articulated parts approach in the image, and (ii) the mean error of the graph-predicted node positions. The 280 windows of each task are divided into four equal-sized groups according to each quantity.

Table S5 reports paired differences in average 32-step joint-position error. For Cheetah Run, we compare mean graph-motion broadcasting with OT change coupling to isolate the additional benefit of token-specific graph–visual coupling. For Walker Walk, we compare JointGV with its original visual backbone to examine how structural ambiguity and graph-prediction quality affect the complete correction. Confidence intervals use 2,000 paired bootstrap resamples over windows. Because windows from the same episode are not independent, these window-level intervals are optimistic.

Paired difference (px)Q1Q2Q3Q4
Minimum distance between non-adjacent nodes (Q1: closest)
Cheetah Run: mean broadcast − OT+0.038 [+0.024, +0.051]+0.036 [+0.025, +0.046]+0.022 [+0.010, +0.035]+0.001 [−0.013, +0.014]
Walker Walk: JointGV − original−3.92 [−4.28, −3.52]−3.39 [−3.80, −2.97]−3.34 [−3.80, −2.84]−2.48 [−3.10, −1.88]
Graph-predicted position error (Q1: most accurate)
Cheetah Run: mean broadcast − OT+0.025 [+0.012, +0.038]+0.024 [+0.011, +0.037]+0.030 [+0.017, +0.043]+0.017 [+0.004, +0.030]
Walker Walk: JointGV − original−4.69 [−4.92, −4.46]−4.34 [−4.58, −4.10]−3.07 [−3.52, −2.59]−1.02 [−1.50, −0.57]

Table S5. Stratified paired differences in average 32-step joint-position error. Each task contains 280 DINO-WM seed-0 test windows, with 70 windows per quartile. Brackets give paired-bootstrap 95% confidence intervals. Positive mean-broadcast − OT values indicate lower error with OT coupling; negative JointGV − original values indicate lower error after correction.

On Cheetah Run, the additional gain from OT coupling over mean broadcasting is concentrated in windows where non-adjacent nodes approach in image space. The difference decreases from 0.038 pixels in the closest quarter to 0.001 pixels in the farthest quarter, with Spearman correlation ρ=−0.22\rho=-0.22 and a 95% interval of [−0.33, −0.10]. Its relation to graph-position error is weak (ρ=−0.06\rho=-0.06, [−0.17, 0.06]), indicating that this coupling advantage is associated more strongly with projected structural ambiguity than with overall node-position accuracy.

On Walker Walk, JointGV also provides its largest reduction when non-adjacent nodes are closest. Its improvement decreases as graph-predicted position error grows: the paired difference changes from −4.69 pixels in the most accurate quarter to −1.02 pixels in the least accurate quarter. The correlation between graph-position error and the JointGV-minus-original difference is ρ=0.67\rho=0.67, with a 95% interval of [0.59, 0.73]. This result identifies inaccurate graph prediction, particularly under severe limb overlap, as a remaining failure mode.

Observer Robustness and Independent Evaluation

The internal graph observer is used only to initialize the graph history from the observed RGB context. To measure sensitivity to initialization errors, we add Gaussian noise N(0,σ2)\mathcal{N}(0,\sigma^2) to the normalized RGB seed frames read by the internal observer, using σ∈{0.02,0.05,0.1,0.2,0.3}\sigma\in\{0.02, 0.05, 0.1, 0.2, 0.3\}. The visual backbone, target frames, and evaluation observer always receive clean images. After initialization, the graph is predicted recursively and receives no additional image observations.

Figure S2 shows that both configurations remain more accurate than their corresponding backbones through σ=0.05\sigma=0.05. At σ=0.10\sigma=0.10, JointGV with Sinkhorn OT retains a 7.9% reduction in 32-step joint-position error, whereas the primary plug-in becomes 2.7% worse than its backbone. Both configurations degrade under the larger perturbations. JointGV therefore tolerates moderate graph-initialization error but currently lacks a fallback when the observed graph becomes severely inaccurate.

Sensitivity to errors in JointGV’s internal graph observer

Figure S2. Sensitivity to errors in JointGV’s internal graph observer. Gaussian noise is applied only to the RGB context read by this observer; the visual backbone and evaluation observer receive clean images. Negative values indicate lower error than the corresponding backbone. Bars show paired episode-level 95% bootstrap intervals.

The main joint-position metric is produced by a frozen RelTR-based RGB observer trained separately from JointGV. Because this evaluator and the internal graph observer originate from the same pretrained architecture, we additionally rescore the same predicted RGB rollouts using an independently trained heatmap-based keypoint network. This network is trained from scratch on observed training images and does not use RelTR features or model-generated images.

On observed target frames, the independent evaluator has a measurement error of 0.093 pixels, compared with 0.196 pixels for the RelTR-based evaluator. It measures an average 32-step joint-position error of 0.786 pixels for the original DINO-WM prediction and 0.679 pixels for JointGV, corresponding to a 13.6% reduction with a paired-bootstrap 95% interval of [12.7, 14.5]%; JointGV improves 95.4% of the test windows. Under the same evaluator, the visual-only residual is 3.4% worse than its backbone. The JointGV improvement and the relative ordering of the controls therefore do not depend on using a RelTR-based evaluator.

Contact-Rich Manipulation on DMC Finger Turn Hard

To extend the DMC evaluation beyond articulated locomotion, we evaluate JointGV on Finger Turn Hard, where a two-link finger must contact and rotate a spinner toward a target. The scene graph contains six task roles: the proximal finger, distal finger, fingertip, spinner center, spinner tip, and target. The task therefore introduces robot–object contact, object rotation, and a separate goal entity that are absent from Cheetah Run and Walker Walk.

We evaluate 280 fixed test windows using the same 32-step open-loop protocol as in the main experiments. The original and corrected predictions share the same epoch-100 checkpoint, DINO-WM backbone, decoder, observed context, and action sequence; only JointGV correction is enabled for the corrected output. Joint-position error is measured from decoded predictions by the same frozen RGB evaluation observer used in the other DMC experiments and averages all visible task-role nodes. The compact comparison is reported in Table 3; Table S6 provides the complete visual metrics. We additionally verify the result with an independently trained keypoint evaluator that uses neither RelTR features nor model-generated images during training.

JointGV reduces joint-position error by 12.7%, 20.6%, and 27.6% at H=8H = 8, 16, and 32, respectively. At H=32H = 32, Token MSE and RGB MSE decrease by 62.7% and 51.8%, while SSIM increases from 0.633 to 0.761 and LPIPS decreases from 0.367 to 0.244. The independently trained keypoint evaluator confirms the improvement, measuring a 43.8% reduction in 32-step error with a paired-bootstrap 95% interval of 41.7–45.8%; all 280 windows improve. The pairwise shape-error reduction also increases from 2.9% over steps 1–8 to 21.4% over steps 25–32. These results extend the evaluation from articulated locomotion to contact-rich manipulation involving an independently moving object and an explicit target.

MethodJoint-position error (px) ↓ H=8H=8Joint-position error (px) ↓ H=16H=16Joint-position error (px) ↓ H=32H=32Token MSE ↓ H=32H=32RGB MSE×103\times10^3 ↓ H=32H=32SSIM ↑ H=32H=32LPIPS ↓ H=32H=32
DINO-WM backbone [34]10.81312.47614.8612.20411.290.6330.3673
DINO-WM + JointGV9.4449.91110.7540.8235.440.7610.2444

Table S6. DINO-WM results on 280 DMC Finger Turn Hard test windows. Values at horizon HH average rollout steps 1 through HH. Joint-position error is measured from decoded predictions over visible task-role nodes, and Token MSE denotes DINOv2-token MSE. Both rows use the same checkpoint and differ only in whether JointGV correction is enabled.

Real-Robot Inverted-Pendulum Prediction and Control

Open-Loop Prediction on Real-Robot Recordings

We evaluate 156 matched windows from 96 recorded RM65-B inverted-pendulum episodes. For the physical dataset, task-role node targets are generated offline by posing a calibrated RM65-B geometry model from synchronized robot joint angles and SA100 pole-angle measurements, then projecting its representative robot-link and pole points into the image. Color-based pole-tip detections are used only to fit the camera extrinsics and timing and angle offsets. These targets supervise training only; recursive inference uses the observed RGB context and recorded actions. All models receive the same observed context and recorded actions, then predict 32 future steps without future images. Table S7 reports each visual world model before and after correction.

ModelOutputJoint (px) ↓Pole (deg) ↓RGB MSE×103\times10^3 ↓
DINO-WMOriginal2.59012.56410.659
DINO-WM+ JointGV2.1246.6898.067
DreamerV3Original1.1373.8054.342
DreamerV3+ JointGV0.8942.8252.217
LeWorldModel [13]Original1.5764.8235.805
LeWorldModel [13]+ JointGV1.4424.2425.055

Table S7. Open-loop prediction on 156 matched real-robot inverted-pendulum windows. Values average rollout steps 1–32; comparisons are within each visual world model. DINO-WM and LeWorldModel use matched checkpoints, while DreamerV3 uses separately selected original and corrected checkpoints.

JointGV reduces all three reported errors for DINO-WM and DreamerV3. LeWorldModel also improves under the same evaluation windows, although its 32-step joint-position and pole-angle reductions are smaller. These are open-loop prediction results, not physical balancing results. Representative real-robot rollouts for the three visual world models are shown in Figure S8–Figure S10.

DINO-WM rollout on a held-out real-robot inverted-pendulum sequence

Figure S8. DINO-WM rollout on a held-out real-robot inverted-pendulum sequence. The pole-axis overlays show how orientation error develops across the rollout. JointGV improves several intermediate predictions, although residual late-horizon deviation remains in this selected sequence.

LeWorldModel rollout on a held-out real-robot inverted-pendulum sequence

Figure S10. LeWorldModel rollout on a held-out real-robot inverted-pendulum sequence. The original model already retains much of the pole structure, and JointGV provides a smaller correction than for DINO-WM and DreamerV3, consistent with the aggregate results in Table S7.

MuJoCo Inverted-Pendulum Control and Hardware Status

We test whether the prediction improvement can support control in a MuJoCo model of the RM65-B inverted-pendulum system. Both actors use the same action constraints and a shared reward predictor trained on observed simulator RGB frames. The task reward decreases with absolute pole angle from vertical, and an episode terminates when the pole reaches the horizontal. The actors differ in the visual predictions used during imagined training: one uses the original DINO-WM prediction, while the other uses JointGV-corrected predictions.

Figure S3 shows one representative rollout, and Table 4 of the main paper reports the aggregate results. Across 50 unseen simulator initial states, the JointGV actor completes the six-second horizon in 46 trials, compared with none for the original-model actor. Its mean time before termination increases from 2.28 to 5.86 seconds. The shared reward predictor also has lower validation error on JointGV-corrected rollouts, although this is only a diagnostic; balancing is measured by executing each actor in the simulator.

Representative MuJoCo inverted-pendulum control rollout

Figure S3. Representative MuJoCo inverted-pendulum control rollout. Both actors start from the same unseen simulator state. The actor trained with original DINO-WM rollouts reaches the fall threshold after approximately 1.44 seconds, whereas the actor trained with JointGV-corrected rollouts completes the six-second simulation horizon. Aggregate results over 50 unseen initial states are reported in Table 4 of the main paper.

For DINO-WM, JointGV also reduces the average 32-step pole-angle-change error from 3.088∘3.088^\circ to 1.457∘1.457^\circ and obtains the lower error on 82.1% of the matched windows, indicating more accurate prediction of the pole’s tilt trend.

We also tested the actors on the physical RM65-B system. The earlier seven paired trials used preliminary checkpoints; both actors reached the angle-triggered stop, although JointGV produced a longer recorded control window in six pairs. The representative trial shown in Figure 6 starts from comparable pole angles: after 1.27 seconds, the original-rollout actor reaches −83.3∘-83.3^\circ, whereas the JointGV-rollout actor remains at −2.7∘-2.7^\circ. This demonstrates a short-duration physical-control advantage, but not stable balancing. High-speed operation introduces non-negligible command delay, and we have not yet obtained a controller that consistently maintains the physical pole for six seconds. Table S8 summarizes these hardware trials.

Hardware: seven paired trialsOriginal+ JointGV
Angle-triggered stops7/77/7
Mean recorded control window (s)0.850.98

Table S8. Preliminary hardware control trials. Hardware trials use earlier actor checkpoints and are not directly comparable with the MuJoCo results in Table 4.

@page 18/21 starts_mid=0 ends_mid=1

MethodJoint-position error (px) ↓ H=8H=8Joint-position error (px) ↓ H=16H=16Joint-position error (px) ↓ H=32H=32Token MSE ↓ H=32H=32RGB MSE×103\times10^3 ↓ H=32H=32SSIM ↑ H=32H=32LPIPS ↓ H=32H=32
DINO-WM backbone [34]10.81312.47614.8612.20411.290.6330.3673
DINO-WM + JointGV9.4449.91110.7540.8235.440.7610.2444

Table S6 (PDF p. 18). DINO-WM results on 280 DMC Finger Turn Hard test windows. Values at horizon HH average rollout steps 1 through HH. Joint-position error is measured from decoded predictions over visible task-role nodes, and Token MSE denotes DINOv2-token MSE. Both rows use the same checkpoint and differ only in whether JointGV correction is enabled.

Real-Robot Inverted-Pendulum Prediction and Control

Open-Loop Prediction on Real-Robot Recordings

We evaluate 156 matched windows from 96 recorded RM65-B inverted-pendulum episodes. For the physical dataset, task-role node targets are generated offline by posing a calibrated RM65-B geometry model from synchronized robot joint angles and SA100 pole-angle measurements, then projecting its representative robot-link and pole points into the image. Color-based pole-tip detections are used only to fit the camera extrinsics and timing and angle offsets. These targets supervise training only; recursive inference uses the observed RGB context and recorded actions. All models receive the same observed context and recorded actions, then predict 32 future steps without future images. Table S7 reports each visual world model before and after correction.

ModelOutputJoint (px) ↓Pole (deg) ↓RGB MSE×103\times10^3 ↓
DINO-WMOriginal2.59012.56410.659
DINO-WM+ JointGV2.1246.6898.067
DreamerV3Original1.1373.8054.342
DreamerV3+ JointGV0.8942.8252.217
LeWorldModel [13]Original1.5764.8235.805
LeWorldModel [13]+ JointGV1.4424.2425.055

Table S7 (PDF p. 18). Open-loop prediction on 156 matched real-robot inverted-pendulum windows. Values average rollout steps 1–32; comparisons are within each visual world model. DINO-WM and LeWorldModel use matched checkpoints, while DreamerV3 uses separately selected original and corrected checkpoints.

JointGV reduces all three reported errors for DINO-WM and DreamerV3. LeWorldModel also improves under the same evaluation windows, although its 32-step joint-position and pole-angle reductions are smaller. These are open-loop prediction results, not physical balancing results. Representative real-robot rollouts for the three visual world models are shown in Figure S8–Figure S10.

MuJoCo Inverted-Pendulum Control and Hardware Status

We test whether the prediction improvement can support control in a MuJoCo model of the RM65-B inverted-pendulum system. Both actors use the same action constraints and a shared reward predictor trained on observed simulator RGB frames. The task reward decreases with absolute pole angle from vertical, and an episode terminates when the pole reaches the horizontal. The actors differ in the visual predictions used during imagined training: one uses the original DINO-WM prediction, while the other uses JointGV-corrected predictions.

Figure S3 shows one representative rollout, and Table 4 of the main paper reports the aggregate results. Across 50 unseen simulator initial states, the JointGV actor completes the six-second horizon in 46 trials, compared with none for the original-model actor. Its mean time before termination increases from 2.28 to 5.86 seconds. The shared reward predictor also has lower validation error on JointGV-corrected rollouts, although this is only a diagnostic; balancing is measured by executing each actor in the simulator.

For DINO-WM, JointGV also reduces the average 32-step pole-angle-change error from 3.088∘3.088^\circ to 1.457∘1.457^\circ and obtains the lower error on 82.1% of the matched windows, indicating more accurate prediction of the pole’s tilt trend.

We also tested the actors on the physical RM65-B system. The earlier seven paired trials used preliminary checkpoints; both actors reached the angle-triggered stop, although JointGV produced a longer recorded control window in six pairs. The representative trial shown in Figure 6 starts from comparable pole angles: after 1.27 seconds, the original-rollout actor reaches −83.3∘-83.3^\circ, whereas the JointGV-rollout actor remains at −2.7∘-2.7^\circ. This demonstrates a short-duration physical-control advantage, but not stable balancing. High-speed operation introduces non-negligible command delay, and we have not yet obtained a controller that consistently maintains the physical pole for six seconds. Table S8 summarizes these hardware trials.

Hardware: seven paired trialsOriginal+ JointGV
Angle-triggered stops7/77/7
Mean recorded control window (s)0.850.98

Table S8 (PDF p. 18). Preliminary hardware control trials. Hardware trials use earlier actor checkpoints and are not directly comparable with the MuJoCo results in Table 4.

Additional Qualitative Rollouts

Figure S4 shows three additional DINO-WM rollouts on held-out Cheetah Run episodes, including occasional local failures. Figure S5 and Figure S6 provide qualitative results for the reversed DMC backbone–task pairings. Figure S7 shows a representative Finger Turn Hard rollout. Figure S8–Figure S10 show representative predictions on real-robot inverted-pendulum recordings.

Additional DINO-WM rollouts on three held-out DMC Cheetah Run test episodes

Figure S4. Additional DINO-WM rollouts on three held-out DMC Cheetah Run test episodes. Rows, displayed steps, decoder, and error-map scaling follow Figure 3. JointGV generally preserves articulated shape more consistently than the corresponding original prediction. Occasional local failures are retained.

DINO-WM rollout on a held-out DMC Finger Turn Hard episode

Figure S7. DINO-WM rollout on a held-out DMC Finger Turn Hard episode. Rows show the ground truth, the original DINO-WM prediction, and its JointGV-corrected prediction. JointGV better preserves the orientations and relative configuration of the finger and spinner as the rollout progresses, with lower visual error at later steps.

Reversed DMC backbone–task pairings. The following panels show predictions at rollout steps 2, 8, 16, 24, and 32. Each predicted RGB frame is followed by its absolute RGB error magnified by four. These examples complement the quantitative reversed-pairing results in Sec. S6.

Real-robot inverted-pendulum rollouts. All three models receive the same observed RGB context and recorded actions and predict 32 future steps without access to future images. The solid green and dashed red lines indicate the observed and predicted pole orientations, respectively. These selected sequences complement the aggregate results in Table S7.

Supplementary video. The accompanying video presents complete 32-step rollouts for DINO-WM and DreamerV3 on Cheetah Run and Walker Walk, including three test episodes for each model–task pair. It additionally shows two matched real-robot prediction episodes for each of DINO-WM, DreamerV3, and LeWorldModel, followed by a MuJoCo inverted-pendulum control demonstration and two physical-robot control episodes comparing the original and JointGV-corrected actors. Each prediction rollout displays ground-truth RGB, the original visual world model, and its JointGV-corrected counterpart using the same decoder and error-map scaling as the corresponding static figures.

DreamerV3 rollout on a held-out real-robot inverted-pendulum sequence

Figure S9. DreamerV3 rollout on a held-out real-robot inverted-pendulum sequence. JointGV more closely follows the observed pole orientation at later rollout steps, consistent with the aggregate reduction in pole-angle error.

References

  1. [1]Anaïs Berkes, Donna Vakalis, Yoshua Bengio, and David Rolnick. Graph dreamer: Temporal graph world models for sample-efficient and generalisable reinforcement learning. In Women in Machine Learning Workshop at NeurIPS, 2025. 2
  2. [2]Yuren Cong, Michael Ying Yang, and Bodo Rosenhahn. RelTR: Relation transformer for scene graph generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):11169–11183, 2023. 3, 1DOI
  3. [3]Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems, 2013. 4, 2
  4. [4]Fan Feng and Sara Magliacane. Learning dynamic attribute-factored world models for efficient multi-object reinforcement learning. In Advances in Neural Information Processing Systems, 2023. 1, 2arxiv.org/abs/2307.09205
  5. [5]Tao Feng, Yexin Wu, Guanyu Lin, and Jiaxuan You. Graph world model. In Proceedings of the 42nd International Conference on Machine Learning, pages 16931–16955. PMLR, 2025. 2arxiv.org/abs/2507.10539
  6. [6]Jean Feydy, Thibault Séjourné, François-Xavier Vialard, Shun-ichi Amari, Alain Trouvé, and Gabriel Peyré. Interpolating between optimal transport and MMD using sinkhorn divergences. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, pages 2681–2690. PMLR, 2019. 4, 2arxiv.org/abs/1810.08278
  7. [7]Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, 640(8059):647–653, 2025. 1, 2, 5DOI
  8. [8]Thomas Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured world models. In International Conference on Learning Representations, 2020. 2
  9. [9]JB Lanier, Kyungmin Kim, Armin Karamzade, Yifei Liu, Ankita Sinha, Kat He, Davide Corsi, and Roy Fox. Adapting world models with latent-state dynamics residuals. In Proceedings of the 8th Annual Learning for Dynamics and Control Conference, pages 117–144. PMLR, 2026. 1, 2, 6arxiv.org/abs/2504.02252
  10. [10]Vincent Le Guen and Nicolas Thome. Differentiable physical dynamics from unknown factors for unsupervised video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 2arxiv.org/abs/2003.01460
  11. [11]Zhixuan Lin, Yi-Fu Wu, Skand Peri, Bofeng Fu, Jindong Jiang, and Sungjin Ahn. Improving generative imagination in object-centric world models. In Proceedings of the 37th International Conference on Machine Learning, pages 6140–6149. PMLR, 2020. 1, 2arxiv.org/abs/2010.02054
  12. [12]Jiawei Liu, Senqiao Yang, Mingjun Wang, Yu Wang, and Bei Yu. Graph world models: Concepts, taxonomy, and future directions. arXiv preprint arXiv:2604.27895, 2026. 2arxiv.org/abs/2604.27895
  13. [13]Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann Le-Cun, and Randall Balestriero. LeworldModel: Stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312, 2026. 2, 8arxiv.org/abs/2603.19312
  14. [14]Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin Murphy, and Honglak Lee. Unsupervised learning of object structure and dynamics from videos. In Advances in Neural Information Processing Systems, 2019. 2arxiv.org/abs/1906.07889
  15. [15]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. 2
  16. [16]RealMan Intelligent Technology (Beijing) Co., Ltd. RM65-B Collaborative Robot: Hardware Preparation and Installation Guide. RealMan Intelligent Technology (Beijing) Co., Ltd., 2026. Accessed: 2026-10-10. 5
  17. [17]Alvaro Sanchez-Gonzalez, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. In Proceedings of the 35th International Conference on Machine Learning, pages 4470–4479. PMLR, 2018. 2, 3
  18. [18]Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In Proceedings of the 37th International Conference on Machine Learning, pages 8459–8468. PMLR, 2020. 2, 3
  19. [19]Alex Schutz, Nick Hawes, and Victor-Alexandru Darvariu. Beyond static graph world models: Learning stochastic latent dynamics over evolving topologies. arXiv preprint arXiv:2609.28670, 2026. 2arxiv.org/abs/2609.28670
  20. [20]Shanghai Toyou Industry Co., Ltd. TRD-CG5D-A 360-Degree Bidirectional Rotary Gear Damper. Shanghai Toyou Industry Co., Ltd., 2026. Accessed: 2026-10-10. 5
  21. [21]Krishnam Soni, Aditya Sehgal, Vedant Dave, and Elmar Rueckert. Sandwich-Residuals: Parameter-efficient test-time adaptation of world models. arXiv preprint arXiv:2609.21740, 2026. 1, 2, 6arxiv.org/abs/2609.21740
  22. [22]Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018.arxiv.org/abs/1801.00690
  23. [23]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.arxiv.org/abs/1706.03762
  24. [24]Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryul Sohn, Xunyu Lin, and Honglak Lee. Learning to generate long-term future via hierarchical prediction. In Proceedings of the 34th International Conference on Machine Learning. PMLR, 2017.arxiv.org/abs/1704.05831
  25. [25]Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004.pubmed.ncbi.nlm.nih.gov/15376593
  26. [26]Zizhao Wang, Kaixin Wang, Li Zhao, Peter Stone, and Jiang Bian. Dyn-O: Building structured world models with object-centric representations. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2507.03298
  27. [27]Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. DayDreamer: World models for physical robot learning. In Proceedings of the 6th Conference on Robot Learning, pages 2226–2240, 2023.arxiv.org/abs/2206.14176
  28. [28]Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg. SlotFormer: Unsupervised visual dynamics simulation with object-centric models. In International Conference on Learning Representations, 2023.arxiv.org/abs/2210.05861
  29. [29]Zaishuo Xia, Yukuan Lu, Xinyi Li, Yifan Xu, and Yubei Chen. Cloning deterministic worlds: The critical role of latent geometry in long-horizon world models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 32602–32612, 2026.arxiv.org/abs/2510.26782
  30. [30]Danfei Xu, Yuke Zhu, Christopher B. Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3097–3106, 2017.arxiv.org/abs/1701.02426
  31. [31]Lishan Yang, Wenxuan Song, Xi Wang, Pingyue Sheng, Zheng Fang, Ziyang Zhou, Junjie He, Haodong Yan, Jiayi Chen, Nan Sun, Qiao Sun, Pengwei Wang, Lingqiao Liu, Yan Wang, Yuxiang Gao, Feras Dayoub, and Haoang Li. 4D-WAM: Infusing spatiotemporal awareness into world action models through trajectory fields. arXiv preprint arXiv:2608.08023, 2026.arxiv.org/abs/2608.08023
  32. [32]Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 586–595, 2018.arxiv.org/abs/1801.03924
  33. [33]Zeyu Zhang, Danning Li, Ian Reid, and Richard Hartley. GeoWorld: Geometric world models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 30952–30963, 2026.arxiv.org/abs/2602.23058
  34. [34]Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning. In Proceedings of the 42nd International Conference on Machine Learning, pages 79115–79135. PMLR, 2025.arxiv.org/abs/2411.04983
  35. [35]Fangqi Zhu, Hongtao Wu, Song Guo, Yuxiao Liu, Chilam Cheang, and Tao Kong. IRASim: A fine-grained world model for robot manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9834–9844, 2025.arxiv.org/abs/2406.14540

Paper details

vlnc:W_s4wvt9qx
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents