WorksComputer Science
JointGV: Structure-Aware Correction of Visual World Models with Parallel Graph Dynamics for Long-Horizon Modeling
Abstract
Visual world models predict future visual states conditioned on actions, but recursive errors in entity poses and spatial relations limit their long-horizon prediction for robot planning and control. We ask whether a separately predicted relational state can correct this drift without restructuring the visual model. This is challenging because graph nodes and visual states share neither elements nor a feature space. We propose Joint Graph–Visual Dynamics Correction (JointGV), a plug-in based on a cross-granularity dynamics hypothesis: despite their different representations, graph-node and visual-state changes describe the same physical transition. JointGV rolls out a task-conditioned scene graph alongside the visual world model, couples their predicted changes, and combines the resulting graph-motion context with graph-predicted node positions to compute a bounded residual. Graph-learning and correction gradients do not propagate into the original visual dynamics. We attach JointGV to DINO-WM (spatial tokens) and DreamerV3 (a global latent state), evaluating both on DMC Cheetah Run and Walker Walk across three seeds and extending the evaluation to contact-rich manipulation with DMC Finger Turn Hard. Across 32-step rollouts, JointGV improves structural and visual-prediction accuracy, reducing joint-position error by 11–28%. For spatial tokens, learned graph–visual change coupling gives lower error than broadcasting the same mean graph-motion context to every token, particularly when non-adjacent parts appear close together in the image. On real-robot inverted-pendulum sequences, JointGV reduces DINO-WM's 32-step pole-angle error from to and improves DreamerV3 and LeWorldModel as well. In simulated inverted-pendulum control, an actor trained with JointGV-corrected rollouts completes 46 of 50 episodes, compared with none using the original rollouts.
Code: https://github.com/JasonLvernex/Joint_Graph_Vedio_Latent_Dynamic_World_Model
Introduction
Visual world models predict how a robot and its environment change under an action. Robot planning evaluates multi-step action sequences, making recursive rollout necessary [7, 27, 34]. Reusing each predicted state as the next input, however, accumulates errors in entity poses and spatial relations. Figure 1 shows this structural drift in open-loop predictions initialized from RGB observations of a physical inverted-pendulum robot: the predicted pole progressively deviates and blurs, while its angle error increases with rollout horizon across three visual world models.

Figure 1. Motivation. (a) DINO-WM open-loop predictions initialized from real-robot inverted-pendulum observations increasingly deviate from the observed pole. (b) Pole-angle error rises with rollout horizon for all three models, while JointGV reduces this structural drift.
Existing approaches reduce long-horizon error by changing the native predictive representation [29, 33], using auxiliary trajectory supervision [31], learning residuals within the original latent pathway [9, 21], or replacing the visual state with object- or graph-centric dynamics [4, 11]. These approaches modify the visual representation, derive correction from the same latent pathway, or require a separately trained structured world model. They do not address how an independently predicted relational rollout can correct an existing visual state when their elements and feature spaces do not correspond.
Our key hypothesis is that, although graph nodes and visual states differ in granularity and content, their action-induced changes describe the same physical transition. We propose Joint Graph–Visual Dynamics Correction (JointGV), which rolls out a task-conditioned scene graph alongside an existing visual world model and couples their predicted changes in a shared space. Graph-predicted node positions provide an additional spatial reference, and a bounded, gated residual corrects the visual prediction. Graph-learning and correction gradients do not propagate into the original visual encoder or dynamics.
Our contributions are: (1) a cross-granularity graph–visual correction interface for predictive states without shared elements or feature spaces; (2) a plug-in architecture that uses a separately predicted relational rollout while preserving the original visual dynamics; and (3) three-seed evaluation across four DMC locomotion pairings and a contact-rich DMC manipulation task, real-robot prediction, simulated control, and analyses identifying when learned graph–visual coupling is most useful.
Related Work
Visual world models for robot learning. Visual world models learn action-conditioned states for planning and control. DayDreamer and DreamerV3 learn behaviors in latent imagination [7, 27]. DINO-WM predicts DINOv2 patch features, IRASim predicts robot–object videos, and LeWorldModel predicts image embeddings [13, 15, 34, 35].
Object-centric and graph-structured world models. Graph networks model physical systems through entity states and relational message passing [12, 17, 18]. C-SWM, G-SWM, SlotFormer, and DAFT-RL build object interactions into the predictive state [4, 8, 11, 28], while recent methods extend structured prediction to multimodal, recurrent, and evolving graphs [1, 5, 19, 26]. These methods use the structured state as part of the world model itself. JointGV instead preserves an existing visual state and uses an independently predicted graph rollout as a correction source.
Structure and residual correction of world models. GRWM and GeoWorld alter the geometry of the native predictive representation [29, 33]. 4D-WAM aligns spatially corresponding video and trajectory-field changes as auxiliary training supervision [31]. ReDRAW and Sandwich-Residuals instead learn residuals within the original latent pathway to adapt to dynamics or distribution shift [9, 21]. Structure-guided video prediction generates frames from predicted poses or keypoints [14, 24], and PhyDNet separates physical dynamics from residual factors [10]; these methods train the visual predictor around the structured state. JointGV addresses a different interface: it couples an independently predicted scene-graph rollout with visual states that have no shared indexing or feature space, then corrects the visual prediction without propagating correction gradients into the original visual model.
Method
Overview
Let denote a spatial-token or global-latent visual state, and let be its next-state prediction from the original visual world model. JointGV predicts graph-node features in parallel. Their one-step changes are
where the bar denotes the original visual prediction, the hat denotes the graph-dynamics prediction, and denotes a one-step change.
JointGV is based on a cross-granularity dynamics hypothesis: although graph nodes and visual states share neither elements nor a feature space, their action-induced changes describe the same physical transition. JointGV couples these changes and combines the resulting graph-motion context with graph-predicted node positions to compute a bounded correction :
Graph-learning and correction losses do not update the original visual encoder or dynamics. Figure 2 shows the complete data flow.

Figure 2. Overview of JointGV. Given the current RGB observation and action, the original visual world model (yellow) predicts the uncorrected next visual state and its change . In parallel, JointGV (blue) uses Relation Transformer (RelTR) features to construct a task-conditioned scene graph and predicts the node-feature change . For the spatial-token case shown here, learned projections map the visual-token and graph-node changes into a shared space, where a change-alignment adapter produces the graph-motion context. The predicted node positions provide a complementary position context. A bounded, gated correction combines these contexts with the original visual prediction to produce . Setting recovers the original prediction. Decoded images are shown only for visualization; the global-state adapter is described in § 3.4.
Task-Conditioned Scene Graph Construction
From RGB observation , JointGV constructs
where contains task-role node features, contains their image positions, and contains directed relation logits.
Learned task-role queries bind intermediate semantic and spatial RelTR features [2] to fixed graph slots. The resulting nodes exchange task-relevant information before a relation head predicts directed edges. Node identities, positions, visibility, and relations are training targets only; inference constructs the graph from RGB observations without simulator states. The feature binding, temporal query update, inter-node exchange, and relation head are detailed in § 7.
History-Aware Graph Dynamics
Let denote the number of most recent scene graphs supplied to the graph dynamics model. These graphs, together with the action applied between time and , are used to predict the next scene graph. This design follows learned graph physics models that combine relation-aware message passing with short motion histories [17, 18]. For node , we define its one-step change as . The current node feature and its changes over the history window are encoded as
where is a multilayer perceptron (MLP), and the superscript (0) indicates the input to the first message-passing layer. When , the input contains only the current node feature .
Before message passing, the relation logits are normalized over the relation classes. At each of the graph layers, every active node sends an action-conditioned message to the other active nodes. The relation probabilities of each node pair are embedded into its message, while the probability of the no-relation class controls how strongly that message contributes to the receiving node.
The final graph features are decoded into bounded node and relation-logit increments, denoted by and , respectively. The next graph state is
where each component of and is bounded by and , respectively. The predicted node positions are decoded from , giving the predicted scene graph . The complete message-passing and increment-decoding operations are provided in Sec. S2.
Self-fed dynamics training. One-step training always provides an observed graph as the input, whereas inference recursively uses previously predicted graphs. To reduce this mismatch, we train the graph dynamics through a -step self-fed rollout, where is the training horizon. Starting from a graph-history window constructed from observed image frames, rollout step predicts
where is the graph dynamics model with parameters , is the action applied during the current transition, and contains the most recent observed or predicted graphs. The first transition uses an observed graph history, after which every predicted graph is inserted into the history window for the following transition. Observed future graphs and simulator states are used only as supervision targets, not as inputs to the recursive rollout. The complete masked losses and the matched self-fed control for the original visual world model are provided in Sec. S4.
Graph–Visual Correction Interface
The interface is defined on predicted changes rather than absolute features. Graph nodes and visual states retain different information, but the changes induced by the same action isolate the physical transition represented by both predictions. The graph rollout then supplies two complementary cues: node changes describe how task-relevant entities evolve, while predicted node positions provide their spatial reference. This separation allows the same correction principle to operate on both spatial tokens and a global latent state.
JointGV operationalizes the cross-granularity dynamics hypothesis by coupling projected graph-node and visual-state changes. The two representations need not share elements, cardinality, or a feature space, and the interface is not tied to a particular coupling operator. We use Sinkhorn optimal transport for spatial tokens and normalized attention pooling for a global latent state.
Spatial-token alignment. For a spatial visual state, we instantiate the alignment adapter with Sinkhorn optimal transport (OT). Let denote the predicted change of visual token , and let denote the predicted change of active graph node . Here, is the number of visual tokens, and are the visual-token and graph-node feature dimensions, respectively, and is the set of active task-role-bound nodes.
Because the two changes belong to different feature spaces, and project and -normalize them in a shared -dimensional space. The visual-token and graph-node sets also have different sizes and no fixed one-to-one pairing. We use entropy-regularized OT for soft many-to-few change coupling; cosine attention provides an alternative learned coupling and is evaluated in Table 1. The OT matching cost and transport plan are
| Method | Yr. | Joint-position error (px) | Token MSE | RGB MSE | SSIM | LPIPS | ||
| Longer visual training | ||||||||
| DINO-WM () [34] | 2025 | 0.830 | 0.931 | 1.036 | 0.456 | 2.31 | 0.841 | 0.0435 |
| DINO-WM + JointGV () | Ours | 0.730 | 0.796 | 0.852 | 0.341 | 1.84 | 0.861 | 0.0351 |
| DINO-WM () | 2025 | 0.882 | 0.940 | 0.957 | 0.339 | 2.13 | 0.853 | 0.0420 |
| DINO-WM + JointGV () | Ours | 0.750 | 0.815 | 0.863 | 0.314 | 1.80 | 0.863 | 0.0352 |
| Residual correction without a predicted graph | ||||||||
| Matched DINO-WM backbone | 2025 | 0.765 | 0.847 | 0.898 | 0.377 | 2.06 | 0.853 | 0.0366 |
Visual-only residual{sup}* {citep}[9, 21]` | 2026 | 0.799 | 0.860 | 0.893 | 0.343 | 2.01 | 0.856 | 0.0380 |
Visual-only residual + node supervision{sup}*{sup}§ | 2026 | 0.771 | 0.836 | 0.883 | 0.351 | 1.99 | 0.856 | 0.0362 |
| Correction using a predicted graph rollout | ||||||||
| JointGV: mean graph-motion broadcast | Ours | 0.700 | 0.765 | 0.815 | 0.322 | 1.73 | 0.866 | 0.0326 |
| JointGV: cosine change coupling{sup}†` | Ours | 0.669 | 0.733 | 0.793 | 0.304 | 1.69 | 0.869 | 0.0322 |
| JointGV: OT change coupling | Ours | 0.693 | 0.750 | 0.791 | 0.304 | 1.67 | 0.869 | 0.0316 |
Table 1. DINO-WM results on 280 matched DMC Cheetah Run test windows. Values at horizon average steps 1 through ; Token MSE denotes DINOv2-token MSE. Rows are single runs and are compared only within matched blocks. {sup}*The visual-residual controls adapt residual-correction principles rather than reproduce the cited methods. {sup}§The annotation-matched control uses the same auxiliary node targets but receives no predicted graph states or graph features. {sup}†The cosine run has its own matched backbone. Three-seed results and additional ablations are provided in {xref}Sec. S6`.
where contains the matching costs between all visual tokens and active graph nodes, is the entropic regularization coefficient, and is the resulting transport plan [3]. We train the two projections using the standard debiased Sinkhorn divergence [6]; implementation details are provided in the supplementary material.
The same transport plan assigns a graph-motion context to each visual token:
where is the row-normalized transport weight and maps each graph-node change to the visual-token dimension. The resulting contains the relation-aware graph change assigned to visual token .
Position-aware graph context. Graph–visual change alignment captures action-induced state evolution in the shared feature space but not absolute image location. JointGV therefore uses the next node positions predicted by the graph rollout to route predicted node features to nearby visual tokens, providing the spatial reference required for correction without ground-truth positions at inference. Let be the predicted image position of node , and let be the fixed image-grid coordinate of visual token . The position-aware graph context is
The exponential term assigns a large weight when token is close to the predicted position of node . The projected node feature is the graph information delivered to nearby tokens. Here, controls the spatial range over which each node influences nearby visual tokens, and maps the predicted node feature to the visual-token dimension.
Bounded visual correction. The motion-based and position-based graph contexts provide complementary information: identifies graph-node changes compatible with the visual-token change, whereas identifies graph nodes located near the visual token. For token , the correction module concatenates both contexts with the predicted visual change and the original next-token prediction:
where is the complete input feature of the correction module for visual token .
The bounded correction for visual token is
Here, predicts the correction, predicts its token-wise gate, denotes element-wise multiplication, controls the overall correction strength, and bounds each component of the correction. Stacking the corrections of all visual tokens gives , which is applied to the original visual prediction through Eq. (2).
Global-vector alignment. When the visual state is a single global vector, e.g., in DreamerV3, there is no spatial token distribution on which to apply OT. In this case, denotes a separately learned projection from the global-state dimension to the same alignment dimension . JointGV uses the predicted global visual-state change to attend to graph-node changes. The attention weight of active node is
where is the attention temperature. The pooled graph-node change and its alignment loss are
The pooled graph change is then used by the same bounded correction principle. For global correction, the predicted nodes’ projected features and two-dimensional positions are concatenated in fixed task-role order to form a global node context. This context and the pooled graph change are both supplied to the bounded, gated correction module. This adapter uses attention pooling followed by normalized half-squared alignment; it does not perform a singleton Sinkhorn computation.
Training Objective
JointGV trains graph observation, graph dynamics, graph–visual alignment, and bounded correction while retaining the original visual-model objective. The uncorrected visual prediction is detached before alignment and correction, so these losses do not update the original visual encoder or dynamics. During recursive correction training, the visual model and graph dynamics are evaluated without gradients, and only the correction module is updated. The complete objective and recursive losses are provided in Sec. S4.
Experiments and Results
Experimental Setup
We evaluate DINO-WM and DreamerV3 on DMC Cheetah Run and Walker Walk across three seeds, and extend the DMC evaluation to contact-rich manipulation with Finger Turn Hard [22]. Table 1–Table 3 report DINO-WM on Cheetah Run, all four locomotion pairings over three seeds, and Finger Turn Hard; complete metrics are in Sec. S6. For DreamerV3, we reproduce its online actor–critic training, then train the backbone and JointGV on the same fixed dataset collected using the reproduced policy and random actions. This avoids exposing separately trained prediction models to different transition distributions [7]; details are provided in Sec. S5. We additionally evaluate DINO-WM, DreamerV3, and LeWorldModel on recordings of a physical inverted-pendulum system built with an RM65-B arm [16] and a rotary gear damper [20], using the same observed image history and recorded actions.
| Method | |||
| DINO-WM backbone [34] | 10.813 | 12.476 | 14.861 |
| DINO-WM + JointGV | 9.444 | 9.911 | 10.754 |
Table 3. DINO-WM on 280 DMC Finger Turn Hard test windows. Values at average steps 1 through ; additional metrics are provided in Sec. S6.
We report two-dimensional joint-position error, RGB mean squared error (MSE), structural similarity (SSIM) [25], Learned Perceptual Image Patch Similarity (LPIPS) [32], and DINOv2-token MSE for DINO-WM. Values at horizon average rollout steps through . Joint positions are measured by a frozen RGB observer separate from JointGV’s graph branch. We also rescore the predictions with an independently trained heatmap-based keypoint detector to avoid dependence on the ReITR-based evaluator. Metric definitions and protocols are provided in Secs. S5 and § 12.9.
For repeated experiments, we report the mean and standard deviation across three training seeds; within each seed, 95% confidence intervals use 2,000 paired bootstrap resamples of the fixed test windows.
Controls compare longer self-fed visual training, visual-only residuals inspired by ReDRAW and Sandwich-Residuals [9, 21], mean graph-motion broadcasting, cosine and OT change coupling, and correction-gradient propagation. The residual controls adapt selected principles rather than reproduce the complete methods. Simulator annotations, and sensor-derived geometric node targets for the physical data, supervise graph observation and dynamics during training but are not supplied during recursive inference. The annotation-matched visual residual receives the same auxiliary node targets but no predicted graph states or graph features; its results are reported in Sec. S6. Actors trained with original or JointGV-corrected DINO-WM rollouts are compared in a matched MuJoCo model (Table 4).
| Measure (50 unseen initial states) | Original | + JointGV |
| Six-second completion ↑ | 0/50 | 46/50 |
| Mean time before termination (s) ↑ | 2.28 | 5.86 |
| Shared reward-head MSE ↓ | 0.0851 | 0.0375 |
Table 4. MuJoCo inverted-pendulum control. Actors are trained with original or JointGV-corrected DINO-WM rollouts; reward-head MSE is a diagnostic.
Long-Horizon Prediction with DINO-WM
Table 1 and Figure 3 show that JointGV reduces joint-position error across all reported horizons, improves the latent and decoded-image metrics, and better preserves articulated shape during rollout.

Figure 3. Qualitative comparison of DINO-WM and JointGV on a held-out DMC Cheetah Run sequence. Columns show rollout steps 2, 8, 16, 24, and 32; adjacent maps show absolute RGB error amplified by four using the shared decoder. Red arrows mark representative structural differences. denotes the self-fed visual training horizon and the propagated correction-gradient fraction. Additional episodes and videos are provided in Sec. S8.
Longer self-fed visual training improves DINO-WM at later horizons but worsens its shorter-horizon joint-position error. JointGV improves both the and backbones, showing that its gain is not explained by longer visual training alone.
Contact-rich manipulation. On DMC Finger Turn Hard, JointGV reduces 32-step joint-position error by 27.6% and RGB MSE by 51.8%, while better preserving the finger–spinner configuration over the rollout (Table 3 and Fig. S7). Additional metrics are provided in Sec. S6.
Evidence for Graph–Visual Change Coupling
Table 1 first separates graph-conditioned correction from residual capacity and additional supervision. The visual-only residual changes the 32-step joint-position error only from 0.898 to 0.893 pixels, whereas JointGV reduces it to 0.791 pixels. Across three seeds, the corresponding reductions are 4.3% ± 4.3% and 16.0% ± 3.7%. Supplying the residual with the same node annotations, but no predicted graph, reaches only 0.883 pixels. The larger gain therefore comes from conditioning the correction on the predicted graph rollout, rather than from residual capacity or matched node supervision alone.
Mean graph-motion broadcasting improves the matched visual backbone, showing that the predicted graph rollout already provides a useful correction signal without token-specific routing. Cosine and OT coupling further reduce the 32-step error to 0.793 and 0.791 pixels, respectively. Their similar accuracy shows that the correction interface is not tied to one operator. The additional gain over broadcasting is largest when non-adjacent parts approach in image space, where assigning differentiated graph motion becomes more important. An independently trained heatmap-based keypoint network also measures a 13.6% reduction and finds improvement on 95.4% of the test windows. Complete paired comparisons and confidence intervals, and stratified analyses are provided in Secs. S6 and S6.8.
Portability to a Global Latent World Model
Table 2 evaluates JointGV with spatial-token and global-latent states on Cheetah Run and Walker Walk. It improves every model–task pairing in all three seeds, reducing average 32-step joint-position error by 11.1–28.0%. For DreamerV3, graph-node changes are pooled using the global visual-state change rather than spatial-token OT, showing that the correction principle does not depend on a visual-token grid. Figure 4 shows the corresponding Walker Walk rollout.

Figure 4. Qualitative comparison on a held-out DMC Walker Walk validation sequence. Columns show rollout steps 2, 8, 16, 24, and 32, with absolute RGB error amplified by a factor of four. JointGV uses the DreamerV3 global-state change to pool the graph-node changes and applies the result as a bounded global correction. The corrected rollout better preserves the articulated pose at later steps.
| Visual world model | Visual state | Task | Original | + JointGV | Reduction (%) ↑ |
| DINO-WM | Spatial tokens | Cheetah Run | 0.942 ± 0.046 | 0.790 ± 0.011 | 16.0 ± 3.7 |
| DreamerV3 | Global latent | Cheetah Run | 0.467 ± 0.009 | 0.415 ± 0.009 | 11.1 ± 2.6 |
| DINO-WM | Spatial tokens | Walker Walk | 10.740 ± 0.922 | 7.682 ± 0.151 | 28.0 ± 7.8 |
| DreamerV3 | Global latent | Walker Walk | 5.835 ± 0.093 | 4.336 ± 0.019 | 25.7 ± 1.3 |
Table 2. JointGV across visual states and control tasks. Values are average joint-position errors over rollout steps 1–32, reported as mean and standard deviation across three independently trained seeds. Every model–task pairing improves in all three seeds. Per-horizon and visual metrics are provided in Sec. S6.
The per-window distributions in Figure 5 further show that these averages are not driven by a few favorable rollouts. JointGV improves 277 of 280 DINO-WM Cheetah Run windows, 186 of 280 DreamerV3 Cheetah Run windows, 250 of 280 DINO-WM Walker Walk windows, and 232 of 280 DreamerV3 Walker Walk windows. The broader DreamerV3 distributions also show that the correction magnitude remains task dependent, despite a consistent mean improvement across all three seeds.

Figure 5. Per-window effect across four model–task pairings. Each value is the relative change in 32-step average joint-position error from the original prediction to JointGV; values below zero indicate improvement. JointGV improves 99%, 66%, 89%, and 83% of the respective test windows, showing that the aggregate gains are not produced by a small subset of rollouts.
Inverted-Pendulum Prediction and Control
Real-robot prediction. On 156 matched windows from real-robot sequences, JointGV reduces DINO-WM’s average 32-step pole-angle error from to and its pole-angle-change error from to , obtaining the lower change error on 82.1% of the windows. DreamerV3 and LeWorldModel [13] also improve.
Simulated control. We test whether the prediction gains support control in a MuJoCo model of the same RM65-B inverted-pendulum system. Two actors are trained in imagination with identical action constraints and a shared reward predictor; they differ only in using original or JointGV-corrected DINO-WM rollouts. On 50 unseen initial states, the JointGV-rollout actor completes 46 six-second episodes, compared with none for the original-rollout actor, and its mean time before termination rises from 2.28 to 5.86 s (Table 4).
Physical trial. Figure 6 shows a preliminary physical control trial from comparable initial pole angles. After 1.27 seconds, the original-rollout actor reaches , whereas the JointGV-rollout actor remains at . Stable six-second physical balancing has not yet been achieved because high-speed operation introduces non-negligible command delay.

Figure 6. Preliminary physical RM65-B control trial. The two actors start from comparable pole angles. At 1.27 seconds, the pole controlled using the original visual-world-model rollout has fallen to , whereas the actor using the JointGV-corrected rollout keeps it near upright at . This trial demonstrates short-duration physical deployment rather than stable six-second balancing.
and Table 4 as the aggregate control evaluation; further details are provided in Sec. S7.2.
Discussion and Conclusion
JointGV improves long-horizon prediction across spatial-token and global-latent visual models, DMC articulated locomotion, contact-rich DMC manipulation, and real-robot sequences. These results support the central cross-granularity hypothesis: graph nodes and visual states without shared elements can still be coupled through changes induced by the same action. The stronger coupling advantage when non-adjacent parts approach further identifies where learned graph–visual correspondence is most useful.
The coupling analysis clarifies the role of graph–visual alignment. Cosine and OT coupling obtain similar overall accuracy, indicating that the correction interface is not tied to a particular operator. Both outperform broadcasting the same mean graph motion to every visual token, and this difference grows when non-adjacent parts approach in image space. The benefit therefore comes from associating visual changes with differentiated graph motion, particularly when the projected structure becomes ambiguous.
The visual-only and supervision-matched residual controls show that the gain is not explained by correction capacity or additional node annotations. Longer visual training also does not reproduce JointGV’s short- and long-horizon improvements, while propagating correction gradients into the original visual dynamics degrades their native prediction. JointGV therefore uses the graph rollout as an external, gradient-isolated correction source rather than another branch of the original latent pathway.
JointGV adds computation and currently assumes supervised task-defined node roles. Its correction also depends on graph observation and prediction quality; overlapping links can cause node-localization errors. Automatic role discovery, confidence-aware correction, broader visual models, and stable physical closed-loop deployment remain future work. The current hardware system exhibits non-negligible control-command delay at high speed, preventing reliable six-second physical balancing despite the improved visual predictions.
Overall, JointGV provides a plug-in interface through which separately predicted relational dynamics correct long-horizon visual rollouts without restructuring the original visual world model.
References
Detailed Scene Graph Construction
This section provides the implementation details omitted from § 3.2. The observer constructs from the current RGB observation. Simulator states, per-frame object identities, and relation annotations are training targets only and are not supplied at inference.
RelTR Features and Task-Role Binding
Given RGB image , the pretrained Relation Transformer (RelTR) [2] provides sparse semantic features and dense spatial features . Here, and are the numbers of sparse and dense features, while and are their dimensions. Dense feature has a fixed normalized image coordinate .
RelTR’s final predicate predictions do not preserve the fixed task roles required by recursive graph prediction. We therefore retain its intermediate features and bind them to task-defined role slots. Role has a learned query , where is the node-feature dimension. At inference, the query locates its assigned role from RGB features without simulator states or per-frame object labels.
The sparse and dense attention operations follow standard multi-head and scaled dot-product attention [23]. A first pass with reads the current RGB features and produces an anchor feature . This feature determines how much information the query receives from the previous node.
Bounded Temporal Query Correction
Each role is located again from the current RGB frame. The previous node supplies a bounded hint when the role is briefly occluded. Let be the layer-normalized previous node. The history-conditioned query is
Here, the superscript identifies the observer gate, predicts its scalar value, projects the previous node, controls the historical contribution, and bounds it. Setting recovers independent per-frame role binding.
The history-conditioned query binds the RelTR features to task role :
where the two terms provide semantic and spatial evidence, respectively.
The updated query reads the sparse and dense RelTR features again. Adding their attention outputs gives role feature . Let be its normalized dense-attention weight at location . The node position is
where predicts a position offset and bounds its magnitude.
Inter-Node Exchange and Relation Logits
The role queries initially read the image independently. We then apply one round of Iterative Message Passing (IMP) [30], allowing each active node to receive information from the other task-relevant nodes. For an ordered pair with subject node and object node , the initial pair feature is
where is a learned multilayer perceptron (MLP). A shared MLP sends message . The received messages update node :
Here, is the active-node set, is its size, GRU is a gated recurrent unit, and LN is layer normalization.
IMP updates node features but does not assign semantic relations. For relation class , we use the VTransE-inspired translation score
where and project the subject and object features and is the learned translation for relation class . We add a learned pair-specific residual:
The shared MLP outputs one residual for each of the relation classes. Softmax over gives the relation distribution for this ordered pair; class denotes no relation.
Detailed History-Aware Graph Dynamics
This section expands the message passing and bounded graph update summarized in § 3.3. The history input is defined in Eq. (4).
Relation-Aware Message Passing
For ordered pair , let be its distribution over the relation classes. We set and , where is learned at graph layer . The no-relation probability controls message strength, while the projected distribution supplies the relation feature. The action-conditioned message and node update are
Here, prevents division by zero, and and are learned MLPs. This update is repeated for graph layers.
Bounded Graph Update and Self-Fed Rollout
The final graph features predict bounded node and relation-logit increments:
Here, and decode the increments, while and bound each component. Stacking these outputs gives the updates in Eq. (5). The position decoder gives .
The -step training rollout follows Eq. (6). Its first prediction uses an observed graph-history window. Each later prediction uses the previous predicted graph instead of an observed future graph. Observed future graphs and simulator states serve only as training targets.
Detailed Graph–Visual Alignment and Correction
This section provides implementation details omitted from § 3.4. Sinkhorn iterations follow Cuturi [3]; we do not repeat their standard derivation.
Spatial-Token Alignment
The projection functions in Eq. (7) are learned bias-free linear layers followed by optional set centering and safe row-wise normalization. They map visual-token changes and graph-node changes into the same -dimensional space.
We use the debiased Sinkhorn divergence of Feydy et al. [6] as and clamp small negative numerical values to zero. The variance-floor loss penalizes a projected dimension when its sample standard deviation falls below .
For each visual token, transport weights are normalized over active graph nodes before the motion context in Eq. (8) is computed. The position context in Eq. (9) is computed independently from graph-predicted node positions and does not come from the transport plan. The two contexts are concatenated in Eq. (10). The gate in Eq. (11) is one scalar per spatial token and is broadcast over its feature dimensions.
Global-Vector Alignment
For a global visual state, Eqs. (12) and (13) define attention pooling and the alignment loss. The global visual-state change is the query, graph-node changes are the keys, and the resulting weights pool the unprojected node changes. Since the projected changes are -normalized, the half-squared alignment is equivalent to cosine distance. This adapter does not perform singleton Sinkhorn.
The global correction receives the original global prediction, its predicted change, the pooled graph-node change, and the node features and predicted positions concatenated in fixed task-role order. It applies the same bounded residual principle as the spatial-token path.
Detailed Training Objective
The complete training objective is
The terms supervise the original visual model, scene-graph observation and prediction, graph–visual alignment, node positions, and visual correction; each is a non-negative weight. For a global latent state, replaces . The stop-gradient operator identifies a fixed target that is not updated by the corresponding loss.
Visual and Graph Targets
For DINO-WM, the optional motion weight for visual token is
Here, retains a contribution from static tokens, and indexes the tokens in the denominator. The DINO-WM token loss applies weighted Smooth- error between original prediction and fixed target . DreamerV3 retains its native recurrent state-space-model objective. We denote the original visual model objective by .
Let indicate whether role is present and supervised. The observer loss uses masked Smooth- losses for physical-state readouts and image positions, binary cross-entropy for presence, and cross-entropy for relation classes. Simulator annotations supervise the observer but are not its inputs.
The -step graph-embedding loss is
where prevents division by zero. The one-step graph loss is its counterpart. When physical-state annotations are available, supervises an auxiliary readout from predicted nodes. The position loss supervises observed, one-step predicted, and recursively predicted node positions.
Recursive Correction Training
The one-step correction loss compares the corrected prediction with a fixed next-state target. For spatial visual tokens, the -step correction loss is
The corrected prediction is fed into the following visual transition. The original visual world model and graph dynamics are evaluated without gradients during this correction rollout, so the loss updates only the correction pathway.
The control follows a matched recursive schedule but disables graph correction and feeds the visual model its own predictions. It tests whether longer visual training alone explains JointGV’s improvement.
Implementation and Evaluation Details
Implementation Settings
The primary DMC runs were trained on an NVIDIA GeForce RTX 5090 D v2 GPU with 24 GB of memory; additional seeds were trained on RTX 4090 and 22 GB RTX 2080 Ti GPUs using the same model and optimization settings. All DMC tasks use eight graph slots of dimension , three graph message-passing layers, and a graph-history window of . Graph dynamics and correction use . Node-feature increments and visual corrections are bounded by and .
For spatial-token alignment, , the entropic regularization coefficient is , Sinkhorn runs for 20 iterations, and . The alignment-loss weight starts at 0.1 and is adjusted every 50 minibatches using an exponential moving average of gradient norms. DINO-WM and DreamerV3 use a learning rate of and batch size 16. DINO-WM uses AdamW with 16-frame sequences; DreamerV3 uses Adam with 64-frame sequences. DINOv2-S/14 and ReiTR inputs are resized to pixels.
We first verify the DreamerV3 implementation using its online actor–critic protocol; the retained Walker Walk agent achieves an evaluation return of . We then freeze this policy and use it to collect a fixed dataset for the prediction experiments. In each data split, 80% of the episodes are collected with this policy and 20% use uniformly random actions to broaden the motion coverage. The DreamerV3 backbone and JointGV correction reported in this paper are trained offline from scratch on these fixed trajectories; the online checkpoint is used only for data collection. This protocol ensures that the original and corrected predictions are trained and evaluated on identical observations and actions, without confounding the comparison with online exploration, actor learning, or changes in the replay distribution.
Rollout Protocol
All compared methods receive the same observed frames and recorded actions. A rollout begins with three preceding RGB frames and the rollout-origin frame, followed by 32 recursive predictions. DINO-WM uses three visual-history frames and JointGV uses four graph-history frames. DreamerV3 filters the four observed context frames through its recurrent state-space-model posterior. No future RGB observation, graph, or simulator state is supplied after prediction begins.
The Cheetah Run and Walker Walk test sets each contain 20 episodes and 14 fixed windows per episode, giving 280 matched windows per task. Stored RGB images are 64 64 pixels. Predictions within an experiment use the same decoder for visualization and image-based evaluation.
Metric Computation
At horizon , each metric is averaged over prediction steps 1 through within a window and then across windows. RGB MSE compares a decoded prediction with its target image. SSIM uses an 11 11 Gaussian window with standard deviation 1.5. LPIPS uses its AlexNet feature network. DINOv2-token MSE compares predicted tokens with frozen-encoder tokens extracted from the observed future image.
Joint-position error measures articulated structure. The same frozen RGB observer extracts normalized two-dimensional positions from every predicted RGB frame; distances to target positions are averaged over visible nodes and converted to pixels. This evaluation observer was trained separately from JointGV’s internal graph observer and does not access JointGV’s graph features, predicted positions, or relations. Applying it to observed DMC frames gives measurement floors of 0.19 pixels for Cheetah Run and 1.46 pixels for Walker Walk. Graph-only errors use graph-predicted positions directly and therefore have a different measurement path.
Computational Cost
The DINO-WM backbone contains 22.7M parameters, of which 0.326M are trainable. JointGV adds 64.4M parameters through the graph branch and its RelTR-based observer, of which 0.531M are trainable. JointGV also increases rollout time because the graph observer, recursive graph dynamics, and correction are evaluated alongside the visual backbone. The graph observer is evaluated once on the context frames, while recursive graph dynamics and correction account for most of the additional cost.
Additional Experiments and Ablation Studies
Visual Training, Graph Prediction, and Gradient Coupling
Increasing the visual training horizon from to improves the original DINO-WM prediction at later steps, but JointGV further improves either model (Table 1).
We also test whether the correction loss should update the original visual dynamics. The detached plug-in uses , while partial and full coupling use and . Table S1 separately reports the original visual output and corrected output from each coupled run. Gradient coupling does not improve upon the detached plug-in and degrades the original visual output. Graph-only rows measure recursive graph prediction before graph information is transferred to the visual state.
| Output | Joint-position error (px) ↓ | DINOv2 token MSE ↓ | RGB MSE× ↓ | SSIM ↑ | LPIPS ↓ | ||
| Detached plug-in: corrected output () | 0.730 | 0.796 | 0.852 | 0.341 | 1.84 | 0.861 | 0.0351 |
| Full gradient coupling: original visual output () | 0.992 | 1.184 | 1.340 | 0.903 | 2.96 | 0.786 | 0.0592 |
| Full gradient coupling: corrected output () | 0.738 | 0.812 | 0.882 | 0.381 | 1.97 | 0.853 | 0.0374 |
| Partial gradient coupling: original visual output () | 0.925 | 1.039 | 1.153 | 0.785 | 2.58 | 0.813 | 0.0532 |
| Partial gradient coupling: corrected output () | 0.771 | 0.838 | 0.890 | 0.449 | 1.99 | 0.846 | 0.0364 |
| Graph-only prediction () | 0.520 | 0.609 | 0.704 | – | – | – | – |
| Graph-only prediction () | 0.535 | 0.630 | 0.754 | – | – | – | – |
Table S1. Gradient-coupling and graph-only controls. “Original visual output” is the native prediction from the visual model trained with the indicated correction-gradient fraction; “corrected output” includes JointGV. Graph-only errors are measured directly from graph-predicted positions and are not ranked against RGB-observer errors.
Correction and Change Alignment
The visual-only controls test whether residual capacity or additional node supervision explains the improvement. Mean graph-motion broadcasting tests whether the predicted graph rollout is useful without token-specific coupling. The cosine and OT variants then test two operators for coupling graph-node and visual-token changes; Table S2 reports the matched comparisons.
| Correction variant | Own backbone | Corrected | Reduction |
| Residual controls | |||
| Visual-only residual | 0.8978 | 0.8935 | 0.0043 |
| Visual-only residual + node supervision | 0.8978 | 0.8830 | 0.0148 |
| Graph-conditioned correction | |||
| Mean graph-motion broadcast | 0.8978 | 0.8146 | 0.0832 |
| Cosine change coupling | 0.9021 | 0.7934 | 0.1087 |
| OT change coupling | 0.8978 | 0.7906 | 0.1072 |
Table S2. Matched within-run correction ablations on DMC Cheetah Run. Values are joint-position errors averaged over rollout steps 1–32. Each corrected result is compared with the visual backbone from the same training run; Reduction denotes the absolute decrease from that backbone. The visual-only controls test residual capacity and matched node supervision without providing a predicted graph. Mean broadcasting tests the graph correction signal without token-specific coupling, while cosine and OT couple graph-node and visual-token changes.
Across three seeds, the visual-only controls provide substantially smaller improvements than JointGV. Mean graph-motion broadcasting also improves the backbone, showing that the predicted graph rollout contains useful correction information. Cosine and OT coupling provide a further reduction by assigning differentiated graph motion to the visual tokens, while their similar accuracy shows that this interface does not depend on a single coupling operator.
Inter-Node Message Passing
The no-message-passing control disables observer IMP and cross-node message passing in graph dynamics. Its corrected 32-step joint-position error is 0.8588 pixels, compared with 0.9646 pixels for its own backbone. Full JointGV reaches 0.7906 pixels with its own backbone at 0.8978 pixels. The no-message-passing run therefore has higher absolute error, although its separately trained backbone prevents the difference from isolating only node-to-node communication.
Training Seeds and Paired Bootstrap
For DINO-WM on Cheetah Run, we repeat JointGV and the visual-only residual control with three independently trained seeds. Each corrected prediction is compared with the backbone from the same run on the same 280 test windows. We use 2,000 paired bootstrap resamples of these windows within each seed. These intervals measure test-window sampling variation, whereas the standard deviation across seeds measures training variation.
Table S3 shows that JointGV improves its corresponding backbone in every seed and produces a larger and more consistent reduction than the visual-only residual. The visual-only residual also improves two of the three runs, but its reduction is smaller and more variable; its seed-0 confidence interval includes zero. We also repeat the primary DreamerV3–Walker Walk comparison with three seeds. JointGV reduces the average 32-step joint-position error from to pixels, corresponding to a relative reduction of . The reductions for seeds 0, 1, and 2 are 27.0%, 25.7%, and 24.4%, respectively.
| Method | Seed | Backbone | Corrected | Reduction | 95% CI |
| JointGV | 0 | 0.8978 | 0.7906 | 11.95% | [11.08, 12.72]% |
| JointGV | 1 | 0.9377 | 0.7790 | 16.92% | [16.17, 17.70]% |
| JointGV | 2 | 0.9899 | 0.8001 | 19.18% | [18.20, 20.10]% |
| JointGV | Mean±std | 0.9418±0.0462 | 0.7899±0.0106 | 16.01±3.70% | – |
| Visual-only residual | 0 | 0.8978 | 0.8935 | 0.49% | [−0.27, 1.23]% |
| Visual-only residual | 1 | 0.9516 | 0.9194 | 3.38% | [2.46, 4.35]% |
| Visual-only residual | 2 | 0.9998 | 0.9094 | 9.04% | [8.45, 9.63]% |
| Visual-only residual | Mean±std | 0.9497±0.0510 | 0.9074±0.0131 | 4.30±4.35% | – |
Table S3. Three-seed results on the 280 fixed DMC Cheetah Run test windows. Backbone and corrected columns report average joint-position error over rollout steps 1–32. Reduction is computed relative to the corresponding backbone from the same run. Confidence intervals are obtained by paired bootstrap resampling within each seed; mean and standard deviation summarize variation across independently trained seeds.
Reversed Backbone–Task Pairings
We additionally evaluate DINO-WM on Walker Walk and DreamerV3 on Cheetah Run, reversing the backbone–task pairings in the main comparisons. Table S4 reports representative matched runs for all four pairings.
| Model / task | Output | Joint-position error (px) | RGB MSE | SSIM | LPIPS | ||
| DINO-WM / Cheetah Run | Original | 0.830 | 0.931 | 1.036 | 2.31 | 0.841 | 0.0435 |
| + JointGV | 0.730 | 0.796 | 0.852 | 1.84 | 0.861 | 0.0351 | |
| DreamerV3 / Cheetah Run | Original | 0.378 | 0.427 | 0.470 | 0.978 | 0.929 | 0.0178 |
| + JointGV | 0.355 | 0.394 | 0.425 | 0.798 | 0.939 | 0.0153 | |
| DINO-WM / Walker Walk | Original | 7.807 | 9.626 | 10.953 | 13.52 | 0.568 | 0.2705 |
| + JointGV | 6.368 | 6.996 | 7.671 | 11.74 | 0.625 | 0.2382 | |
| DreamerV3 / Walker Walk | Original | 4.063 | 4.750 | 5.938 | 15.12 | 0.621 | 0.1722 |
| + JointGV | 3.785 | 4.022 | 4.338 | 11.15 | 0.694 | 0.1231 |
Table S4. Representative matched runs for all four DMC model–task pairings. Values at horizon average rollout steps 1 through . The three-seed joint-position results are summarized in Table 2.
Across three independently trained seeds, JointGV reduces the average 32-step joint-position error of DINO-WM on Walker Walk from 10.740 0.922 to 7.682 0.151 pixels, corresponding to a relative reduction of 28.0 7.8%. For DreamerV3 on Cheetah Run, the error decreases from 0.467 0.009 to 0.415 0.009 pixels, corresponding to a reduction of 11.1 2.6%. The improvement occurs in every seed for both reversed pairings.
Representative rollouts for DreamerV3 on Cheetah Run and DINO-WM on Walker Walk are shown in Figure S5 and Figure S6, respectively.

Figure S5. DreamerV3 rollout on a held-out DMC Cheetah Run episode. JointGV better preserves the articulated configuration and reduces visual error as the rollout progresses, showing that the correction also applies to this reversed backbone–task pairing.

Figure S6. DINO-WM rollout on a held-out DMC Walker Walk episode. The original prediction becomes increasingly blurred and loses limb definition, whereas JointGV retains a clearer articulated pose over the rollout.
Per-Step and Per-Window Analyses
The per-step curves in Figure S1 show that JointGV’s improvement is distributed across the rollout rather than arising only at the final prediction step. The corresponding per-window distributions are reported in Figure 5 of the main paper.

Figure S1. Per-step joint-position error for one run of each DINO-WM and DreamerV3 task pairing. Dashed curves show the original predictions and solid curves show JointGV-corrected predictions. Shaded regions are episode-level 95% bootstrap intervals over the fixed test windows. JointGV reduces structural error over most of the rollout rather than only at its final step.
Supervision Scope
JointGV uses simulator-provided node identities, visibility, positions, and task relations as training targets for graph construction and graph prediction. The visual-only residual in Table 1 receives neither graph features nor node annotations. It therefore tests whether a generic bounded residual explains the gain, but does not by itself isolate the effect of additional structured supervision.
To control for supervision volume, we additionally train a visual-only residual with an auxiliary node head supervised by the same MuJoCo node annotations as JointGV. The correction module does not receive the predicted node states or graph features. The supervision-matched residual reduces the average 32-step joint-position error from per. 0.8978 to 0.8830 pixels, a 1.65% reduction with a paired-bootstrap 95% interval of [0.95, 2.33]%. JointGV reduces the same backbone error to 0.7906 pixels, corresponding to an 11.95% reduction. The additional node supervision alone therefore does not explain JointGV’s gain.
Stratified Analysis
We analyze when graph–visual coupling provides the largest additional benefit. For each DINO-WM test window, we compute two quantities over rollout steps 1–32: (i) the minimum ground-truth image distance between non-adjacent nodes, which measures how closely separate articulated parts approach in the image, and (ii) the mean error of the graph-predicted node positions. The 280 windows of each task are divided into four equal-sized groups according to each quantity.
Table S5 reports paired differences in average 32-step joint-position error. For Cheetah Run, we compare mean graph-motion broadcasting with OT change coupling to isolate the additional benefit of token-specific graph–visual coupling. For Walker Walk, we compare JointGV with its original visual backbone to examine how structural ambiguity and graph-prediction quality affect the complete correction. Confidence intervals use 2,000 paired bootstrap resamples over windows. Because windows from the same episode are not independent, these window-level intervals are optimistic.
| Paired difference (px) | Q1 | Q2 | Q3 | Q4 |
| Minimum distance between non-adjacent nodes (Q1: closest) | ||||
| Cheetah Run: mean broadcast − OT | +0.038 [+0.024, +0.051] | +0.036 [+0.025, +0.046] | +0.022 [+0.010, +0.035] | +0.001 [−0.013, +0.014] |
| Walker Walk: JointGV − original | −3.92 [−4.28, −3.52] | −3.39 [−3.80, −2.97] | −3.34 [−3.80, −2.84] | −2.48 [−3.10, −1.88] |
| Graph-predicted position error (Q1: most accurate) | ||||
| Cheetah Run: mean broadcast − OT | +0.025 [+0.012, +0.038] | +0.024 [+0.011, +0.037] | +0.030 [+0.017, +0.043] | +0.017 [+0.004, +0.030] |
| Walker Walk: JointGV − original | −4.69 [−4.92, −4.46] | −4.34 [−4.58, −4.10] | −3.07 [−3.52, −2.59] | −1.02 [−1.50, −0.57] |
Table S5. Stratified paired differences in average 32-step joint-position error. Each task contains 280 DINO-WM seed-0 test windows, with 70 windows per quartile. Brackets give paired-bootstrap 95% confidence intervals. Positive mean-broadcast − OT values indicate lower error with OT coupling; negative JointGV − original values indicate lower error after correction.
On Cheetah Run, the additional gain from OT coupling over mean broadcasting is concentrated in windows where non-adjacent nodes approach in image space. The difference decreases from 0.038 pixels in the closest quarter to 0.001 pixels in the farthest quarter, with Spearman correlation and a 95% interval of [−0.33, −0.10]. Its relation to graph-position error is weak (, [−0.17, 0.06]), indicating that this coupling advantage is associated more strongly with projected structural ambiguity than with overall node-position accuracy.
On Walker Walk, JointGV also provides its largest reduction when non-adjacent nodes are closest. Its improvement decreases as graph-predicted position error grows: the paired difference changes from −4.69 pixels in the most accurate quarter to −1.02 pixels in the least accurate quarter. The correlation between graph-position error and the JointGV-minus-original difference is , with a 95% interval of [0.59, 0.73]. This result identifies inaccurate graph prediction, particularly under severe limb overlap, as a remaining failure mode.
Observer Robustness and Independent Evaluation
The internal graph observer is used only to initialize the graph history from the observed RGB context. To measure sensitivity to initialization errors, we add Gaussian noise to the normalized RGB seed frames read by the internal observer, using . The visual backbone, target frames, and evaluation observer always receive clean images. After initialization, the graph is predicted recursively and receives no additional image observations.
Figure S2 shows that both configurations remain more accurate than their corresponding backbones through . At , JointGV with Sinkhorn OT retains a 7.9% reduction in 32-step joint-position error, whereas the primary plug-in becomes 2.7% worse than its backbone. Both configurations degrade under the larger perturbations. JointGV therefore tolerates moderate graph-initialization error but currently lacks a fallback when the observed graph becomes severely inaccurate.

Figure S2. Sensitivity to errors in JointGV’s internal graph observer. Gaussian noise is applied only to the RGB context read by this observer; the visual backbone and evaluation observer receive clean images. Negative values indicate lower error than the corresponding backbone. Bars show paired episode-level 95% bootstrap intervals.
The main joint-position metric is produced by a frozen RelTR-based RGB observer trained separately from JointGV. Because this evaluator and the internal graph observer originate from the same pretrained architecture, we additionally rescore the same predicted RGB rollouts using an independently trained heatmap-based keypoint network. This network is trained from scratch on observed training images and does not use RelTR features or model-generated images.
On observed target frames, the independent evaluator has a measurement error of 0.093 pixels, compared with 0.196 pixels for the RelTR-based evaluator. It measures an average 32-step joint-position error of 0.786 pixels for the original DINO-WM prediction and 0.679 pixels for JointGV, corresponding to a 13.6% reduction with a paired-bootstrap 95% interval of [12.7, 14.5]%; JointGV improves 95.4% of the test windows. Under the same evaluator, the visual-only residual is 3.4% worse than its backbone. The JointGV improvement and the relative ordering of the controls therefore do not depend on using a RelTR-based evaluator.
Contact-Rich Manipulation on DMC Finger Turn Hard
To extend the DMC evaluation beyond articulated locomotion, we evaluate JointGV on Finger Turn Hard, where a two-link finger must contact and rotate a spinner toward a target. The scene graph contains six task roles: the proximal finger, distal finger, fingertip, spinner center, spinner tip, and target. The task therefore introduces robot–object contact, object rotation, and a separate goal entity that are absent from Cheetah Run and Walker Walk.
We evaluate 280 fixed test windows using the same 32-step open-loop protocol as in the main experiments. The original and corrected predictions share the same epoch-100 checkpoint, DINO-WM backbone, decoder, observed context, and action sequence; only JointGV correction is enabled for the corrected output. Joint-position error is measured from decoded predictions by the same frozen RGB evaluation observer used in the other DMC experiments and averages all visible task-role nodes. The compact comparison is reported in Table 3; Table S6 provides the complete visual metrics. We additionally verify the result with an independently trained keypoint evaluator that uses neither RelTR features nor model-generated images during training.
JointGV reduces joint-position error by 12.7%, 20.6%, and 27.6% at , 16, and 32, respectively. At , Token MSE and RGB MSE decrease by 62.7% and 51.8%, while SSIM increases from 0.633 to 0.761 and LPIPS decreases from 0.367 to 0.244. The independently trained keypoint evaluator confirms the improvement, measuring a 43.8% reduction in 32-step error with a paired-bootstrap 95% interval of 41.7–45.8%; all 280 windows improve. The pairwise shape-error reduction also increases from 2.9% over steps 1–8 to 21.4% over steps 25–32. These results extend the evaluation from articulated locomotion to contact-rich manipulation involving an independently moving object and an explicit target.
| Method | Joint-position error (px) ↓ | Joint-position error (px) ↓ | Joint-position error (px) ↓ | Token MSE ↓ | RGB MSE ↓ | SSIM ↑ | LPIPS ↓ |
| DINO-WM backbone [34] | 10.813 | 12.476 | 14.861 | 2.204 | 11.29 | 0.633 | 0.3673 |
| DINO-WM + JointGV | 9.444 | 9.911 | 10.754 | 0.823 | 5.44 | 0.761 | 0.2444 |
Table S6. DINO-WM results on 280 DMC Finger Turn Hard test windows. Values at horizon average rollout steps 1 through . Joint-position error is measured from decoded predictions over visible task-role nodes, and Token MSE denotes DINOv2-token MSE. Both rows use the same checkpoint and differ only in whether JointGV correction is enabled.
Real-Robot Inverted-Pendulum Prediction and Control
Open-Loop Prediction on Real-Robot Recordings
We evaluate 156 matched windows from 96 recorded RM65-B inverted-pendulum episodes. For the physical dataset, task-role node targets are generated offline by posing a calibrated RM65-B geometry model from synchronized robot joint angles and SA100 pole-angle measurements, then projecting its representative robot-link and pole points into the image. Color-based pole-tip detections are used only to fit the camera extrinsics and timing and angle offsets. These targets supervise training only; recursive inference uses the observed RGB context and recorded actions. All models receive the same observed context and recorded actions, then predict 32 future steps without future images. Table S7 reports each visual world model before and after correction.
Table S7. Open-loop prediction on 156 matched real-robot inverted-pendulum windows. Values average rollout steps 1–32; comparisons are within each visual world model. DINO-WM and LeWorldModel use matched checkpoints, while DreamerV3 uses separately selected original and corrected checkpoints.
JointGV reduces all three reported errors for DINO-WM and DreamerV3. LeWorldModel also improves under the same evaluation windows, although its 32-step joint-position and pole-angle reductions are smaller. These are open-loop prediction results, not physical balancing results. Representative real-robot rollouts for the three visual world models are shown in Figure S8–Figure S10.

Figure S8. DINO-WM rollout on a held-out real-robot inverted-pendulum sequence. The pole-axis overlays show how orientation error develops across the rollout. JointGV improves several intermediate predictions, although residual late-horizon deviation remains in this selected sequence.

Figure S10. LeWorldModel rollout on a held-out real-robot inverted-pendulum sequence. The original model already retains much of the pole structure, and JointGV provides a smaller correction than for DINO-WM and DreamerV3, consistent with the aggregate results in Table S7.
MuJoCo Inverted-Pendulum Control and Hardware Status
We test whether the prediction improvement can support control in a MuJoCo model of the RM65-B inverted-pendulum system. Both actors use the same action constraints and a shared reward predictor trained on observed simulator RGB frames. The task reward decreases with absolute pole angle from vertical, and an episode terminates when the pole reaches the horizontal. The actors differ in the visual predictions used during imagined training: one uses the original DINO-WM prediction, while the other uses JointGV-corrected predictions.
Figure S3 shows one representative rollout, and Table 4 of the main paper reports the aggregate results. Across 50 unseen simulator initial states, the JointGV actor completes the six-second horizon in 46 trials, compared with none for the original-model actor. Its mean time before termination increases from 2.28 to 5.86 seconds. The shared reward predictor also has lower validation error on JointGV-corrected rollouts, although this is only a diagnostic; balancing is measured by executing each actor in the simulator.

Figure S3. Representative MuJoCo inverted-pendulum control rollout. Both actors start from the same unseen simulator state. The actor trained with original DINO-WM rollouts reaches the fall threshold after approximately 1.44 seconds, whereas the actor trained with JointGV-corrected rollouts completes the six-second simulation horizon. Aggregate results over 50 unseen initial states are reported in Table 4 of the main paper.
For DINO-WM, JointGV also reduces the average 32-step pole-angle-change error from to and obtains the lower error on 82.1% of the matched windows, indicating more accurate prediction of the pole’s tilt trend.
We also tested the actors on the physical RM65-B system. The earlier seven paired trials used preliminary checkpoints; both actors reached the angle-triggered stop, although JointGV produced a longer recorded control window in six pairs. The representative trial shown in Figure 6 starts from comparable pole angles: after 1.27 seconds, the original-rollout actor reaches , whereas the JointGV-rollout actor remains at . This demonstrates a short-duration physical-control advantage, but not stable balancing. High-speed operation introduces non-negligible command delay, and we have not yet obtained a controller that consistently maintains the physical pole for six seconds. Table S8 summarizes these hardware trials.
| Hardware: seven paired trials | Original | + JointGV |
| Angle-triggered stops | 7/7 | 7/7 |
| Mean recorded control window (s) | 0.85 | 0.98 |
Table S8. Preliminary hardware control trials. Hardware trials use earlier actor checkpoints and are not directly comparable with the MuJoCo results in Table 4.
@page 18/21 starts_mid=0 ends_mid=1
| Method | Joint-position error (px) ↓ | Joint-position error (px) ↓ | Joint-position error (px) ↓ | Token MSE ↓ | RGB MSE ↓ | SSIM ↑ | LPIPS ↓ |
| DINO-WM backbone [34] | 10.813 | 12.476 | 14.861 | 2.204 | 11.29 | 0.633 | 0.3673 |
| DINO-WM + JointGV | 9.444 | 9.911 | 10.754 | 0.823 | 5.44 | 0.761 | 0.2444 |
Table S6 (PDF p. 18). DINO-WM results on 280 DMC Finger Turn Hard test windows. Values at horizon average rollout steps 1 through . Joint-position error is measured from decoded predictions over visible task-role nodes, and Token MSE denotes DINOv2-token MSE. Both rows use the same checkpoint and differ only in whether JointGV correction is enabled.
Real-Robot Inverted-Pendulum Prediction and Control
Open-Loop Prediction on Real-Robot Recordings
We evaluate 156 matched windows from 96 recorded RM65-B inverted-pendulum episodes. For the physical dataset, task-role node targets are generated offline by posing a calibrated RM65-B geometry model from synchronized robot joint angles and SA100 pole-angle measurements, then projecting its representative robot-link and pole points into the image. Color-based pole-tip detections are used only to fit the camera extrinsics and timing and angle offsets. These targets supervise training only; recursive inference uses the observed RGB context and recorded actions. All models receive the same observed context and recorded actions, then predict 32 future steps without future images. Table S7 reports each visual world model before and after correction.
Table S7 (PDF p. 18). Open-loop prediction on 156 matched real-robot inverted-pendulum windows. Values average rollout steps 1–32; comparisons are within each visual world model. DINO-WM and LeWorldModel use matched checkpoints, while DreamerV3 uses separately selected original and corrected checkpoints.
JointGV reduces all three reported errors for DINO-WM and DreamerV3. LeWorldModel also improves under the same evaluation windows, although its 32-step joint-position and pole-angle reductions are smaller. These are open-loop prediction results, not physical balancing results. Representative real-robot rollouts for the three visual world models are shown in Figure S8–Figure S10.
MuJoCo Inverted-Pendulum Control and Hardware Status
We test whether the prediction improvement can support control in a MuJoCo model of the RM65-B inverted-pendulum system. Both actors use the same action constraints and a shared reward predictor trained on observed simulator RGB frames. The task reward decreases with absolute pole angle from vertical, and an episode terminates when the pole reaches the horizontal. The actors differ in the visual predictions used during imagined training: one uses the original DINO-WM prediction, while the other uses JointGV-corrected predictions.
Figure S3 shows one representative rollout, and Table 4 of the main paper reports the aggregate results. Across 50 unseen simulator initial states, the JointGV actor completes the six-second horizon in 46 trials, compared with none for the original-model actor. Its mean time before termination increases from 2.28 to 5.86 seconds. The shared reward predictor also has lower validation error on JointGV-corrected rollouts, although this is only a diagnostic; balancing is measured by executing each actor in the simulator.
For DINO-WM, JointGV also reduces the average 32-step pole-angle-change error from to and obtains the lower error on 82.1% of the matched windows, indicating more accurate prediction of the pole’s tilt trend.
We also tested the actors on the physical RM65-B system. The earlier seven paired trials used preliminary checkpoints; both actors reached the angle-triggered stop, although JointGV produced a longer recorded control window in six pairs. The representative trial shown in Figure 6 starts from comparable pole angles: after 1.27 seconds, the original-rollout actor reaches , whereas the JointGV-rollout actor remains at . This demonstrates a short-duration physical-control advantage, but not stable balancing. High-speed operation introduces non-negligible command delay, and we have not yet obtained a controller that consistently maintains the physical pole for six seconds. Table S8 summarizes these hardware trials.
| Hardware: seven paired trials | Original | + JointGV |
| Angle-triggered stops | 7/7 | 7/7 |
| Mean recorded control window (s) | 0.85 | 0.98 |
Table S8 (PDF p. 18). Preliminary hardware control trials. Hardware trials use earlier actor checkpoints and are not directly comparable with the MuJoCo results in Table 4.
Additional Qualitative Rollouts
Figure S4 shows three additional DINO-WM rollouts on held-out Cheetah Run episodes, including occasional local failures. Figure S5 and Figure S6 provide qualitative results for the reversed DMC backbone–task pairings. Figure S7 shows a representative Finger Turn Hard rollout. Figure S8–Figure S10 show representative predictions on real-robot inverted-pendulum recordings.

Figure S4. Additional DINO-WM rollouts on three held-out DMC Cheetah Run test episodes. Rows, displayed steps, decoder, and error-map scaling follow Figure 3. JointGV generally preserves articulated shape more consistently than the corresponding original prediction. Occasional local failures are retained.

Figure S7. DINO-WM rollout on a held-out DMC Finger Turn Hard episode. Rows show the ground truth, the original DINO-WM prediction, and its JointGV-corrected prediction. JointGV better preserves the orientations and relative configuration of the finger and spinner as the rollout progresses, with lower visual error at later steps.
Reversed DMC backbone–task pairings. The following panels show predictions at rollout steps 2, 8, 16, 24, and 32. Each predicted RGB frame is followed by its absolute RGB error magnified by four. These examples complement the quantitative reversed-pairing results in Sec. S6.
Real-robot inverted-pendulum rollouts. All three models receive the same observed RGB context and recorded actions and predict 32 future steps without access to future images. The solid green and dashed red lines indicate the observed and predicted pole orientations, respectively. These selected sequences complement the aggregate results in Table S7.
Supplementary video. The accompanying video presents complete 32-step rollouts for DINO-WM and DreamerV3 on Cheetah Run and Walker Walk, including three test episodes for each model–task pair. It additionally shows two matched real-robot prediction episodes for each of DINO-WM, DreamerV3, and LeWorldModel, followed by a MuJoCo inverted-pendulum control demonstration and two physical-robot control episodes comparing the original and JointGV-corrected actors. Each prediction rollout displays ground-truth RGB, the original visual world model, and its JointGV-corrected counterpart using the same decoder and error-map scaling as the corresponding static figures.

Figure S9. DreamerV3 rollout on a held-out real-robot inverted-pendulum sequence. JointGV more closely follows the observed pole orientation at later rollout steps, consistent with the aggregate reduction in pole-angle error.
References
- [1]Anaïs Berkes, Donna Vakalis, Yoshua Bengio, and David Rolnick. Graph dreamer: Temporal graph world models for sample-efficient and generalisable reinforcement learning. In Women in Machine Learning Workshop at NeurIPS, 2025. 2
- [2]Yuren Cong, Michael Ying Yang, and Bodo Rosenhahn. RelTR: Relation transformer for scene graph generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):11169–11183, 2023. 3, 1DOI
- [3]Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems, 2013. 4, 2
- [4]Fan Feng and Sara Magliacane. Learning dynamic attribute-factored world models for efficient multi-object reinforcement learning. In Advances in Neural Information Processing Systems, 2023. 1, 2arxiv.org/abs/2307.09205
- [5]Tao Feng, Yexin Wu, Guanyu Lin, and Jiaxuan You. Graph world model. In Proceedings of the 42nd International Conference on Machine Learning, pages 16931–16955. PMLR, 2025. 2arxiv.org/abs/2507.10539
- [6]Jean Feydy, Thibault Séjourné, François-Xavier Vialard, Shun-ichi Amari, Alain Trouvé, and Gabriel Peyré. Interpolating between optimal transport and MMD using sinkhorn divergences. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, pages 2681–2690. PMLR, 2019. 4, 2arxiv.org/abs/1810.08278
- [7]Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, 640(8059):647–653, 2025. 1, 2, 5DOI
- [8]Thomas Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured world models. In International Conference on Learning Representations, 2020. 2
- [9]JB Lanier, Kyungmin Kim, Armin Karamzade, Yifei Liu, Ankita Sinha, Kat He, Davide Corsi, and Roy Fox. Adapting world models with latent-state dynamics residuals. In Proceedings of the 8th Annual Learning for Dynamics and Control Conference, pages 117–144. PMLR, 2026. 1, 2, 6arxiv.org/abs/2504.02252
- [10]Vincent Le Guen and Nicolas Thome. Differentiable physical dynamics from unknown factors for unsupervised video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 2arxiv.org/abs/2003.01460
- [11]Zhixuan Lin, Yi-Fu Wu, Skand Peri, Bofeng Fu, Jindong Jiang, and Sungjin Ahn. Improving generative imagination in object-centric world models. In Proceedings of the 37th International Conference on Machine Learning, pages 6140–6149. PMLR, 2020. 1, 2arxiv.org/abs/2010.02054
- [12]Jiawei Liu, Senqiao Yang, Mingjun Wang, Yu Wang, and Bei Yu. Graph world models: Concepts, taxonomy, and future directions. arXiv preprint arXiv:2604.27895, 2026. 2arxiv.org/abs/2604.27895
- [13]Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann Le-Cun, and Randall Balestriero. LeworldModel: Stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312, 2026. 2, 8arxiv.org/abs/2603.19312
- [14]Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin Murphy, and Honglak Lee. Unsupervised learning of object structure and dynamics from videos. In Advances in Neural Information Processing Systems, 2019. 2arxiv.org/abs/1906.07889
- [15]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. 2
- [16]RealMan Intelligent Technology (Beijing) Co., Ltd. RM65-B Collaborative Robot: Hardware Preparation and Installation Guide. RealMan Intelligent Technology (Beijing) Co., Ltd., 2026. Accessed: 2026-10-10. 5
- [17]Alvaro Sanchez-Gonzalez, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. In Proceedings of the 35th International Conference on Machine Learning, pages 4470–4479. PMLR, 2018. 2, 3
- [18]Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In Proceedings of the 37th International Conference on Machine Learning, pages 8459–8468. PMLR, 2020. 2, 3
- [19]Alex Schutz, Nick Hawes, and Victor-Alexandru Darvariu. Beyond static graph world models: Learning stochastic latent dynamics over evolving topologies. arXiv preprint arXiv:2609.28670, 2026. 2arxiv.org/abs/2609.28670
- [20]Shanghai Toyou Industry Co., Ltd. TRD-CG5D-A 360-Degree Bidirectional Rotary Gear Damper. Shanghai Toyou Industry Co., Ltd., 2026. Accessed: 2026-10-10. 5
- [21]Krishnam Soni, Aditya Sehgal, Vedant Dave, and Elmar Rueckert. Sandwich-Residuals: Parameter-efficient test-time adaptation of world models. arXiv preprint arXiv:2609.21740, 2026. 1, 2, 6arxiv.org/abs/2609.21740
- [22]Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018.arxiv.org/abs/1801.00690
- [23]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.arxiv.org/abs/1706.03762
- [24]Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryul Sohn, Xunyu Lin, and Honglak Lee. Learning to generate long-term future via hierarchical prediction. In Proceedings of the 34th International Conference on Machine Learning. PMLR, 2017.arxiv.org/abs/1704.05831
- [25]Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004.pubmed.ncbi.nlm.nih.gov/15376593
- [26]Zizhao Wang, Kaixin Wang, Li Zhao, Peter Stone, and Jiang Bian. Dyn-O: Building structured world models with object-centric representations. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2507.03298
- [27]Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. DayDreamer: World models for physical robot learning. In Proceedings of the 6th Conference on Robot Learning, pages 2226–2240, 2023.arxiv.org/abs/2206.14176
- [28]Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg. SlotFormer: Unsupervised visual dynamics simulation with object-centric models. In International Conference on Learning Representations, 2023.arxiv.org/abs/2210.05861
- [29]Zaishuo Xia, Yukuan Lu, Xinyi Li, Yifan Xu, and Yubei Chen. Cloning deterministic worlds: The critical role of latent geometry in long-horizon world models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 32602–32612, 2026.arxiv.org/abs/2510.26782
- [30]Danfei Xu, Yuke Zhu, Christopher B. Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3097–3106, 2017.arxiv.org/abs/1701.02426
- [31]Lishan Yang, Wenxuan Song, Xi Wang, Pingyue Sheng, Zheng Fang, Ziyang Zhou, Junjie He, Haodong Yan, Jiayi Chen, Nan Sun, Qiao Sun, Pengwei Wang, Lingqiao Liu, Yan Wang, Yuxiang Gao, Feras Dayoub, and Haoang Li. 4D-WAM: Infusing spatiotemporal awareness into world action models through trajectory fields. arXiv preprint arXiv:2608.08023, 2026.arxiv.org/abs/2608.08023
- [32]Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 586–595, 2018.arxiv.org/abs/1801.03924
- [33]Zeyu Zhang, Danning Li, Ian Reid, and Richard Hartley. GeoWorld: Geometric world models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 30952–30963, 2026.arxiv.org/abs/2602.23058
- [34]Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning. In Proceedings of the 42nd International Conference on Machine Learning, pages 79115–79135. PMLR, 2025.arxiv.org/abs/2411.04983
- [35]Fangqi Zhu, Hongtao Wu, Song Guo, Yuxiao Liu, Chilam Cheang, and Tao Kong. IRASim: A fine-grained world model for robot manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9834–9844, 2025.arxiv.org/abs/2406.14540