WorksComputer Science
QCC-WM: Linking Macro-State Dynamics and Visual Generation via Quotient Causal Construction
Abstract
A key challenge in action-conditioned world modeling is to construct macro-states from high-dimensional observations that support recursive prediction under actions, while keeping their predicted evolution consistent with visual changes in generated observations. We propose QCC-WM, a visual world model based on Quotient-based Causal Construction. QCC-WM uses action responses to guide macro-variable construction for recursive action-conditioned prediction. A shared reference quotient measurement constrains the consistency between predicted states and observed changes, allowing macro-level predictions to provide structured constraints for future visual generation. Using translational and rotational actions and their predicted responses, the model aggregates local visual features into macro-nodes with spatial correspondence through causal abstraction, and recursively predicts node content and motion under a shared reference measurement. Macro-level predictions guide future observation generation, while predicted motion transports historical appearance information. Node-level errors between generated observations and macro-level targets are then mapped back to the corresponding spatial regions for correction, improving consistency between visual generation and macro-level predictions. On the US-VLA dataset, QCC-WM achieves an ROI-MSE of 0.01861 for four-step prediction. Compared with the best baseline at each prediction horizon, it reduces LPIPS by 48.59%, 41.68%, and 30.23% for four-step prediction and eight- and sixteen-step recursive extrapolation, respectively. Ablation studies and mechanism analyses show that motion-guided reference transport mainly improves pixel reconstruction, while node-level error feedback further improves perceptual quality and consistency between generated observations and macro-level predictions.
Introduction
Causal representation learning (CRL) aims to identify high-level variables related to the underlying generative mechanisms from high-dimensional observations, providing representations for analyzing action outcomes, intervention effects, and distribution shifts 1. For world models, this goal further requires learned states to evolve recursively under actions while maintaining their correspondence with observable changes. However, a fundamental challenge in high-dimensional visual settings is that macro-level causal variables suitable for recursive prediction are generally not given and must be constructed from fine-grained observations. Local features can describe visual appearance and changes, but they do not necessarily form macro-states with consistent measurement definitions and predictable transition rules. Building action-conditioned world models therefore requires addressing both
how macro-variables are formed and how they represent state evolution under successive actions.
Research on macro-variable construction and causal abstraction offers theoretical insights into this problem. Chalupka et al. studied how macro-level cause and effect variables can be constructed from the causal behavior of micro-level measurements 2. Hoel et al. showed that, for specific systems and under the effective information measure, suitable macro-level descriptions can exhibit greater causal effectiveness than micro-level descriptions 3. Rubenstein et al. further emphasized that causal models at different levels should yield consistent predictions under explicitly defined intervention mappings 4. These studies suggest that micro-to-macro mappings should be designed around the mechanisms and dynamics they preserve. For world models, we therefore consider a dynamics-oriented abstraction principle: macro-variable construction should be aligned with future prediction so that abstract state changes correspond to evolution at the observation level.
Existing visual world models have explored various state representations and prediction methods. DINO-WM learns action-conditioned dynamics in pretrained spatial features [5], while IRASim incorporates action trajectories into a diffusion model to generate future observations [6]. For structured modeling, SlotFormer learns recursive dynamics over object-level representations [7], and Slot-JEPA organizes prediction through semantic slots and spatial masks [8]. Motion-I2V uses predicted motion to transport reference features, connecting motion modeling with visual generation [9]. These studies provide important foundations for state prediction, structured representations, and spatially organized generation. Recursive macro-state prediction nevertheless raises several questions: whether nodes have comparable measurement definitions across time steps, whether predicted state changes are constrained by observed evolution, and whether macro-level predictions retain the correspondence needed to map them back to observation space. These questions directly affect the preservation of state meaning and the accumulation of prediction errors during recursive prediction.
To address these issues, we introduce QCC (Quotient-based Causal Construction) and develop the action-conditioned world model QCC-WM. QCC uses action-guided soft quotient mappings to aggregate local visual features into macro-nodes with spatial support. It recursively predicts content and motion states through inter-node dependencies and action-conditioned transitions. Within each prediction window, the model fixes the reference quotient mapping and its alignment, and constrains macro-state prediction through supervision on states and their temporal increments. This encourages consistency with observed evolution under the same quotient measurement.
To translate macro-level predictions into observation generation, the model uses predicted motion to transport reference features and latent variables, construct observable macro-level targets, and correct the initial generation. It then reads back the generated observations through the same quotient measurement, computes node-level errors relative to the macro-level targets, and maps these errors back to local latent space through the node–space correspondence. The reference quotient mapping thus connects node construction, macro-state prediction, and spatial correction, linking macro-level dynamics with future observation generation.
Our main contributions are as follows:
An action-guided quotient-based causal construction mechanism, QCC. We construct macro-nodes with spatial support through soft quotient mappings, discover a macro-level causal structure of temporal dependencies among nodes under a shared reference measurement, and use this structure to organize action-conditioned dynamics. This connects macro-node construction and structure discovery with recursive prediction of content and motion states.
A macro-level dynamics learning method under a fixed reference quotient measurement. We fix the reference quotient mapping and content alignment within each prediction window to provide consistent measurement definitions for macro-states across time steps. Supervision on
states and their temporal increments constrains recursive content and motion prediction, encouraging consistency between macro-level predictions and observed evolution under the same reference measurement.
A generation correction interface combining motion transport with readback under the same quotient measurement. We use predicted motion to transport historical appearance information, then map node-level errors obtained through readback to the generation latent space through spatial correspondence, improving consistency between future observations and macro-level predictions.
The remainder of this paper is organized as follows. Section 2 reviews related work on causal representation learning, structured world models, and motion-guided generation. Section 3 presents QCC-WM, including quotient-based causal construction, macro-state prediction, motion-guided reference transport, spatial correction, and staged optimization. Section 4 describes the experimental setup and evaluates model performance and module contributions through comparative experiments, ablation studies, and mechanism analyses. Section 5 concludes the paper and discusses future directions.
Related Work
Causal Representation Learning and Macro-Variable Construction
Causal representation learning seeks to identify latent variables related to generative mechanisms from high-dimensional observations, enabling mechanism analysis, intervention prediction, and generalization across distributions [1]. In dynamic visual settings, CITRIS uses temporal observations and known intervention targets to identify latent causal factors under specific assumptions [10]. BISCUIT studies causal variable learning with unknown binary interaction mechanisms and establishes corresponding identifiability conditions [11]. These studies show that causal interpretations depend closely on additional supervision and structural assumptions. QCC focuses on constructing macro-nodes for action-conditioned prediction. Its node measurements and dependency structures are learned and evaluated as predictive interfaces within the model; action inputs and interventions within the model alone do not provide such identifiability guarantees.
Macro-variable construction also concerns relationships between different levels of description. Chalupka et al. used causal partitions to connect micro-level measurements with macro-level cause and effect variables [2]. Hoel et al. analyzed macro-level causal effectiveness in specific systems using effective information [3]. Rubenstein et al. formalized intervention consistency across levels through exact transformations between structural equation models [4]. In reinforcement learning, DeepMDP learns state representations by predicting rewards and next-state distributions in latent space, and establishes connections to bisimulation [12]. These studies provide a basis for abstractions that preserve underlying dynamics. We apply this idea to spatial aggregation within visual observations: action-guided soft quotient mappings define macro-level measurements, whose future evolution is constrained to match observed changes under the same reference mapping. This spatial quotient construction addresses a different modeling object from full environment-state abstraction, and the prediction consistency imposed here does not imply strict intervention consistency.
Action-Conditioned Dynamics and Structured World Models
Latent world models support future prediction by learning transition rules in compact state spaces. PlaNet combines deterministic and stochastic transitions and introduces latent overshooting as a multi-step variational objective [13]. Dreamer optimizes behavior over imagined latent trajectories, using recursive state prediction for control [14]. DINO-WM learns action-conditioned patch dynamics from pretrained spatial features and performs planning through goal-feature prediction [5]. These methods connect latent state prediction with downstream decision-making. QCC further addresses how local observations are organized into macro-level prediction units. A shared reference measurement constrains node states and their temporal increments, providing comparable observation-based targets for recursive prediction in the abstract state space.
Structured world models explicitly organize prediction units within a scene. DINOSAUR learns object-level representations by reconstructing self-supervised pretrained features [15]. SlotFormer learns autoregressive dynamics over slots to predict future object states [7]. Slot-JEPA decomposes dense features into semantic slots and time-varying spatial masks, combining transition memory and transport objectives for prediction and planning [8]. These methods establish foundations for feature grouping, structured recursive prediction, and spatial correspondence. Our specific design incorporates action plans into quotient mapping construction and uses the same reference quotient mapping for macro-state measurement, future-state supervision, and mapping predictions back to spatial locations. QCC nodes are defined as action-conditioned aggregate measurements, whose predictive value must be assessed through appropriate controlled comparisons.
Generative World Models and Motion-Guided Spatial Feedback
Generative world models synthesize future observations from actions or structured states. IRASim models robotic interaction videos using a diffusion Transformer with frame-wise action conditioning [6]. To connect structured representations with generation, SlotDiffusion uses a slot-conditioned latent diffusion decoder and applies the learned representations to object dynamics modeling [16]. SlotDiT generates future slot trajectories from text-conditioned diffusion and then decodes them into images [17]. QCC uses macro-level content and motion as states for recursive prediction and converts these predictions into generation conditions in image latent space through explicit quotient mappings. Its focus is the interface between macro-level dynamics and observation generation.
Explicit motion modeling provides another way to connect state changes with visual synthesis. Motion-I2V first predicts pixel trajectories and then propagates reference image features through motion-enhanced temporal attention [9]. FLIP combines flow generation, flow-conditioned video generation, and value modeling for generative planning in visual space [18]. DeXplicit predicts explicit 3D states and deformations from actions and transports appearance information to guide video generation [19]. These studies provide direct precedents for motion prediction and reference content transport. We use a learned visual quotient space as an intermediate representation, where regional motion and content changes arise from recursive macro-state prediction and are mapped back to local space through the retained quotient mapping. Spatial feedback thus forms a prediction pipeline together with QCC node measurements and recursive dynamics. Its contribution is assessed through generation evaluation and interventions on the model interfaces.
Methods
Problem Formulation and QCC Overview
Given historical visual observations and a known action plan , we learn the conditional distribution of future observations . The central problem is to construct macro-state units that support action-conditioned prediction while preserving their connection to observation space, allowing their predictions to guide future generation.
We propose QCC (Quotient-based Causal Construction), which uses action-guided soft quotient mappings to organize local visual representations into macro-nodes with explicit spatial origins. QCC jointly defines the aggregation sources, reference measurements, and spatial correspondence of nodes, and establishes action-conditioned transitions and inter-node dependencies over the constructed macro-variables. We refer to these state units as macro-level causal nodes, which represent action-driven evolution and inter-node interactions within the model.
As shown in Figure 1, the framework consists of four components: quotient-based causal construction, action-conditioned macro-state prediction, motion-guided reference transport, and readback under the same quotient measurement followed by spatial correction:

Figure 1. Overview of QCC-WM. QCC first constructs macro-nodes with spatial origins and reference measurements. Action-conditioned dynamics recursively predict content and motion states. Predicted motion transports reference features and latent variables to construct observable macro-level targets and perform motion-guided reference correction, respectively. The model then reads back the current generation under the same quotient measurement and maps the remaining node-level errors to the corresponding spatial locations to obtain future observations. The reference quotient mapping connects node construction, macro-state prediction, and spatial generation.
where denotes the quotient construction branch guided by translational or rotational action components, and is the corresponding action guidance. Both dynamics models and the generator retain the complete action input. and denote the reference soft assignment and reference content alignment, respectively, while specify the dependency supports for the content and motion channels. contains node content, node motion memory, and their corresponding shared states; is the node control derived from the action responses predicted by QCC. denotes the generation conditions produced by macro-state prediction, is the initially generated latent variable, is the latent variable after motion-guided reference correction, is the remaining node-level error obtained through readback under the same quotient measurement, and is the observation decoder.
Action-Response-Guided Quotient-Based Causal Construction
We use the pretrained visual encoder of I-JEPA [20] to extract observation representations and obtain frame-wise content features through branch-specific mappings: . Historical features and their local temporal differences are used to organize nodes and initialize motion, while frame-wise content features measure node states at each time step. Each branch maintains global prototypes , with prototype indices identifying the nodes. QCC uses the action plan to modulate the affinity between historical local features and prototypes. Combined with spatial distances, these affinities determine soft assignments within the valid observation region. The assignments are normalized across nodes and refined by iteratively updating node centroids, thereby defining the local spatial support of each node.
Action-response-driven quotient refinement. To connect node organization with potential action-induced changes, QCC further predicts local action responses from historical descriptors. For future time step , let denote the cumulative action guidance:
where is the response base term conditioned on history, and parameterizes the interaction between local mechanisms and actions. The response includes cumulative motion and appearance changes relative to the reference latent variable, connecting feature grouping with the future changes to be represented.
We subtract the spatial mean of the action response term over the valid region, normalize it using training statistics, and concatenate it across time to form the local response descriptor . This descriptor compares relative responses across locations under the current action plan. Starting from the prototype grouping, QCC sequentially refines spatial assignments using history-conditioned response encodings and explicit mechanism–action response encodings. Both refinements alternate between response aggregation and assignment updates. For each refinement, let denote the input assignment and the corresponding response descriptor:
where is a bounded strength learned separately for each refinement layer, and the final assignment is denoted by . Node assignments are thus jointly constrained by historical spatial structure and predicted action responses, organizing local evolution information while preserving prototype identities. At inference, action responses are predicted entirely from history and the known action plan; observed future responses are used only for training supervision.
Reference quotient measurement and node initialization. QCC further uses historical descriptors and predicted action responses to produce a bounded reference content alignment , which remains fixed within the current prediction window. We define the normalized aggregation weights as
The corresponding reference quotient measurement is
This definition unifies spatial assignment, content alignment, and frame-wise measurement. The initial node content is set to . Motion memory is initialized by quotient aggregation of historical content changes, estimated historical optical flow, and local mechanism descriptors. Reference centroids are computed from the final assignments . Shared content and motion states are obtained through global aggregation over the valid region, providing a reference for overall changes to the local nodes.
Action-Conditioned Macro-State Prediction under a Fixed Quotient Measurement
Node dependencies and step-wise response control. We construct candidate node dependency supports through temporal dependency analysis of macro-level measurements from training observations, incorporating action and reference conditions [21]. The analysis controls for the complete action plan, reference support, and past measurements of shared states. Fixed directed supports organize inter-node messages for the content and motion channels. For channel , incoming edge weights are defined as
where is a learnable strength within the support, and the direction from to represents information transfer from node at the previous time step to node at the current time step. The directed supports remain fixed during recursive prediction, with content and motion using their respective message weights.
The action responses predicted by QCC also enter the recursive dynamics through step-wise node controls. We take temporal differences of the cumulative motion responses in (2) and aggregate them under the reference quotient mapping:
where is a bounded response encoding function. This control allows local action responses learned during construction to contribute to node updates at every step, jointly driving macro-state evolution with the complete action input.
Recursive evolution of shared and local states. To separate overall changes from local differences, we first update the shared states:
and define local residuals as and . Content messages combine node content, motion, reference positions, and fixed historical context, while motion messages are extracted from motion residuals:
where denote the message projections for the respective channels. Inter-node messages use the states at the start of each step, while each node continues to evolve within its residual update blocks.
Node content is updated through a residual mapping, while motion memory is updated through a gated recurrent unit:
where , is the node motion response, and is the parameterized residual update mapping for node content in branch . This update combines each node’s history, dependency messages, shared changes, and local action responses to recursively predict future sequences from the same initial macro-state without future observations.
Content dynamics are learned under a fixed reference quotient measurement, connecting recursively predicted states with corresponding observed changes:
where extracts node content. Initial and future observations are measured using the same , providing a common reference across prediction steps. Supervision is applied to both states and their temporal increments:
where denotes averaging over branches, and MSE is averaged over samples, prediction steps, nodes, and channels. This constraint aligns macro-states and their changes under a fixed observation reference, providing state sequences with a common measurement basis for subsequent spatial generation.
Quotient-Mapping-Based Motion-Guided Reference Transport
QCC first uses spatial correspondence to condition the initial generation on macro-states. The model performs frame-level and temporal readouts of node content and motion, and injects node content increments and motion geometry into the generator’s local features through the reference quotient mapping. For example, the spatial field corresponding to node content changes is
Together with macro-level motion conditions, this field forms and guides the generation of the initial latent block , while retaining historical visual information and direct action conditioning.
To further translate macro-level motion into changes in observation space, we use the same reference assignment to recover local displacement fields and accumulate them over prediction steps:
where converts normalized motion responses to the displacement scale required for transport. We define the reference transport operator through bilinear backward sampling as , with displacements interpolated and rescaled to the corresponding feature grid.
Reference information is transported separately in content feature space and generation latent space. For branch , the reference content features are transported using the predicted displacement and measured through the fixed quotient interface to form an observable macro-level target:
where represents the observable changes induced by predicted motion in the reference content and is used for comparison with generated observations under the same quotient measurement. The recursively predicted latent state supports macro-state prediction and generation conditioning. The two quantities thus serve latent prediction and observable correction, respectively.
In latent space, we average the displacement fields from the two branches to obtain and transport the latent variable of the last historical observation:
QCC also converts node motion memory into a spatial gate:
where is the node motion gating network, is the sigmoid function, and is a bounded strength. The gate depends on predicted motion memory and is expanded over the reference spatial support of each node. Motion-guided reference correction is defined as
where is the initially generated latent variable, is the reference latent variable transported using predicted motion, contains spatial gating weights, controls the overall correction strength, and denotes element-wise multiplication. The difference between the reference latent variable and the initial generation defines the correction direction. The spatial gate controls the correction magnitude at each location, incorporating transported historical appearance information into the generation.
Readback under the Same Quotient Measurement and Spatial Correction of Node Errors
After motion-guided reference transport, QCC uses the same reference quotient measurement to evaluate the discrepancy between the current generation and the observable macro-level target. It first decodes the motion-corrected latent variable and reads back its macro-level content:
Because target construction and generation readback share has fixed node definitions and spatial origins. It represents changes that the current generation has not yet captured under the macro-level measurement. Readback follows motion-guided reference correction, allowing error correction to operate on generation results that have already been adjusted using reference motion.
Spatial feedback expands errors through the node–location relationships defined by QCC. One path directly projects node errors, while the other modulates error representations using predicted content and content read back from the generation:
where , are channel projections and is a node content modulation function. The error fields are interpolated to the generation latent grid and combined with local context from the current and reference latent variables to produce spatial corrections:
where and include branch fusion, spatial interpolation, and local context modulation, and are collectively denoted by . This interface converts macro-level errors into generation adjustments with explicit spatial correspondence.
Learning Objectives and Staged Optimization
The model is trained for quotient construction, macro-level content prediction, conditional generation, and spatial correction. The main objective is summarized as
where encourages quotient construction to preserve observed response information, supervises macro-level content prediction under a fixed reference measurement, trains conditional diffusion generation, and trains spatial correction. Training also includes auxiliary supervision on local responses, node and local motion, shared states, and transport alignment, as detailed below. These objectives are applied to the corresponding modules in separate training stages. The spatial correction module is trained separately after the core model is frozen.
Local response prediction is supervised using cumulative optical flow and changes relative to the reference latent variable from training observations. Let denote the observed response normalized using training statistics. We supervise Eq. (2) with both response prediction error and local response error after removing the spatial mean. Response reconstruction after node aggregation further encourages quotient construction to preserve information relevant to future evolution:
where denotes the valid support, and regularizes node occupancy and near-duplicate normalized assignment columns. These objectives connect node organization with observed responses during training, while construction itself uses only history and actions.
Content learning in the macro-level dynamics uses the quotient consistency objective in Eq. (12). Node motion targets are obtained by aggregating observed optical flow under the same reference assignment:
where predictions and targets use the same motion scale. The corresponding losses compare node motion with and reconstructed local fields with , while also supervising shared content increments and shared motion. Construction and response adaptation additionally use local motion errors and alignment errors between transported references and true future latent variables to coordinate response control, macro-level motion, and reference transport.
Initial observations are generated in a pretrained visual latent space. Let . We construct diffusion targets using the reference residual :
Conditioned on history, complete actions, and CQCC, the generator minimizes a denoising objective weighted by the valid region:
where is the valid observation region weight, and the expectation averages over samples, diffusion steps, future steps, positions, and channels .
The spatial correction module is trained using initial samples generated by the frozen core model. For its training output , we combine image reconstruction, perceptual distance, structural similarity, and generation–prediction and generation–observation alignment under the same quotient measurement:
where is image MSE over the valid region, is perceptual distance, and is structural similarity loss. The two node-level alignment terms encourage generated observations to match the observable macro-level targets and the true future observations under a shared measurement, respectively.
Results
Ablation Study
To examine the effect of QCC branch configurations on prediction performance, we compare four settings: a direct action-conditioned baseline (Direct), a single-branch QCC model with only the translation-guided branch (T-only), a single-branch QCC model with only the rotation-guided branch (R-only), and the complete QCC-WM with both branches. The T-only and R-only models are trained independently, with six macro-nodes per branch. The complete system combines dual-branch macro-state prediction, motion-guided reference transport, and spatial correction of node-level errors. All configurations use the same evaluation split and sampling settings. Results are reported as the mean and sample standard deviation across three training seeds.
As shown in Table 1, both single-branch QCC variants outperform Direct in the mean values of all five metrics, indicating that action-guided macro-level modeling improves future observation prediction. The two single-branch configurations perform similarly overall: T-only achieves slightly lower ROI-MSE and temporal MSE, while R-only performs slightly better in SSIM and LPIPS. Both achieve a PSNR of approximately 24.98 dB.
| Setting | ROI-MSE () | PSNR (dB) | SSIM | LPIPS | Temporal MSE () |
| Direct | |||||
| Single QCC: T-only | |||||
| Single QCC: R-only | |||||
| Dual QCC: T and R |
Table 1. Prediction performance of different QCC configurations. Dual QCC is the complete system with macro-state prediction, motion-guided reference transport, and spatial correction of node-level errors. Note: Results are reported as the mean sample standard deviation across three training seeds. Both MSE metrics are reported in units of . Bold indicates the best mean in each column.
The complete dual-branch QCC system achieves the best mean values across all five metrics. Compared with Direct, it reduces ROI-MSE, LPIPS, and temporal MSE by 9.05%, 7.51%, and 6.64%, respectively. It also improves all metrics over both single-branch variants, indicating that combining dual-branch macro-state prediction with subsequent spatial correction improves pixel reconstruction, structural similarity, perceptual quality, and prediction of inter-frame changes.
Mechanism Analysis of QCC-WM
We analyze QCC from three perspectives: spatial node organization, motion prediction, and generation correction.
Spatial Organization of Macro-Nodes
QCC aggregates local visual features through action-guided soft quotient mappings while preserving the spatial support of each node in the reference observation. The mapping remains fixed within each prediction window, ensuring that node-state comparison and subsequent spatial feedback use the same spatial correspondence. Figure 2 shows that nodes in both branches cover the valid imaging region and form distinct local aggregation regions, providing explicit spatial origins for macro-states.

Figure 2. Spatial origins and effective occupancy of QCC macro-nodes. Left: reference observations and soft quotient assignments of the T and R branches for three samples. Colors indicate node indices within each branch. Right: effective number of occupied nodes, minimum node mass, and cosine similarity between assignment columns. Diamonds and error bars indicate the mean and sample standard deviation across three training seeds; circles show individual seed results. Dashed lines mark the six predefined nodes and the 1% low-occupancy threshold, respectively.
We define the effective number of nodes using normalized node masses as . With six nodes per branch, the effective node counts of the T and R branches are and , respectively. Their mean minimum node masses are and , and the cosine similarities between assignment columns are and . These results indicate that node capacity is well utilized and that spatial weights have limited overlap across nodes. This spatial organization allows macro-level motion and node-level errors to be mapped back to their corresponding local regions.
Macro-Level Motion and Local Displacement
The action-conditioned dynamics predict the motion response of each node and recover local displacement fields through the reference quotient mapping. Motion differences between macro-nodes can therefore produce different displacements across image regions, driving reference information transport. Figure 3(a,b) compares predicted displacements with zero-motion, local constant-velocity, and global constant-velocity extrapolation. Using observed optical flow as the reference, we compute the MSE of two-dimensional cumulative displacement over the valid region and the corresponding error after removing its spatial mean.
Figure 3. Macro-level motion prediction. (a) Cumulative displacement MSE. (b) MSE after removing the spatial mean of the cumulative error. Optical flow estimated from observations serves as the reference. Curves and shaded regions show the mean and sample standard deviation across three random seeds. (c,d) Cumulative responses of T and R nodes in Case 2: open circles indicate aggregated observed optical flow, and crosses indicate predictions. Successive markers correspond to consecutive prediction steps. Displacements are enlarged sixfold for visualization.
QCC achieves a four-step average cumulative displacement MSE of , lower than the three controls at , , and . After removing the spatial mean of the cumulative error, QCC achieves an MSE of , which remains below for both zero-motion and global constant-velocity extrapolation, and for local constant-velocity extrapolation. These results indicate that macro-level dynamics reduce cumulative displacement errors and further improve the prediction of local motion differences. Figure 3(c,d) illustrates differences in the direction and magnitude of node responses, showing how the fixed quotient mapping translates macro-level predictions into spatially varying motion fields.
Complementary Effects of Motion Transport and Error Correction
Motion-guided reference transport moves historical appearance information according to predicted displacements and corrects the initial generation through spatial gating. Readback under the same quotient measurement then measures the remaining node-level discrepancies and maps them back to local latent space. Figure 4 examines the two paths using four settings that independently enable or disable each correction. Node-level errors are recomputed from the generated results under each setting.
Figure 4. Complementary effects of motion transport and node-level error correction. (a–c) ROI-MSE, LPIPS, and generation–prediction node MSE under four interface settings. The horizontal axis indicates whether motion-guided reference correction is enabled. Gray and teal indicate disabled and enabled node-level error correction, respectively. Error bars show the sample standard deviation across three random seeds. Below: local crops at prediction step 4 in Case 2, showing the reference observation, transported reference, true future observation, initial generation, and generation with motion correction only.
Enabling motion-guided reference correction alone reduces ROI-MSE from to , indicating that transported historical appearance improves pixel reconstruction. However, Fig. 4(b,c) shows increases in both LPIPS and generation–prediction node error. Readback after motion transport is therefore needed to correct the remaining discrepancies from the macro-level targets. Adding node-level error correction reduces LPIPS from to , a relative reduction of 7.58% (paired 95% interval: [6.39%, 8.93%]). The two paths thus have complementary effects: motion transport provides spatially relevant appearance references, while node-level error correction mainly improves perceptual quality and macro-level alignment. Compared with disabling both corrections, the complete interface reduces ROI-MSE, LPIPS, and temporal MSE by 6.25%, 6.45%, and 4.55%, respectively.
Effects of Readback and Spatial Correspondence
Readback uses the fixed reference measurement to compare the observable macro-level target produced by motion transport with the current generation, giving node-level errors a common measurement definition and spatial origin. To evaluate correction, we compute node-level errors between generated observations and both macro-level targets and true future observations:
where and denote generated and true future observations, respectively, and the metrics are averaged over the T and R branches.
Adding node-level error feedback reduces generation–prediction node MSE from 0.004187 to 0.001604 and generation–observation node MSE from 0.010688 to 0.006570, corresponding to reductions of 61.69% and 38.53% (paired 95% intervals: [53.36%, 67.91%] and [30.34%, 46.14%], respectively). Figure 5(d,Figure 5e) shows that both errors decrease at all four prediction steps. Readback correction therefore improves both agreement with macro-level predictions and macro-level correspondence with true future observations.
Figure 5. Analysis of readback under the same quotient measurement and spatial feedback. (a–c) Reductions in image error, generation–prediction node error, and generation–observation node error for the complete model relative to four controls: no node-level error correction, global feedback, spatially shuffled feedback, and initial content targets. Positive values favor the complete model; horizontal intervals indicate paired 95% intervals. (d,e) Node MSE at each prediction step after motion correction and after adding node-level error correction. Curves and shaded regions show the mean and sample standard deviation across three random seeds. All settings use the same reference quotient measurement.
To examine the effect of error feedback locations, we replace the final feedback mapping with a spatially uniform assignment or shuffle its spatial positions, while keeping all other predictions, readback measurements, and correction conditions unchanged. Global feedback preserves the valid support and total node masses. Results for spatial shuffling are averaged over five predefined shuffles. Compared with these two controls, complete spatial feedback reduces generation–prediction node MSE by 8.28% and 7.79%, and generation–observation node MSE by 4.39% and 4.07%, respectively. ROI-MSE values for global, shuffled, and complete feedback are 0.018530, 0.018563, and 0.018609, respectively. Complete feedback has slightly higher pixel error, with its main advantage lying in macro-level alignment. These results indicate that preserving node–location correspondence helps translate macro-level errors into generation adjustments at the appropriate regions.
Compared with using the initial content measurement as the error decoder target, using the future observable target reduces generation–prediction node error by 1.20%. ROI-MSE changes by only 0.02%, with its paired interval including zero. Together, these results suggest that the current correction gains mainly arise from combining motion-guided reference transport with node-level error feedback, while correct spatial correspondence further improves consistency between macro-level predictions and generated observations.
Comparative Experiments
We evaluate action-conditioned ultrasound video prediction through short-window prediction and multi-step extrapolation, with prediction horizons . All methods use model weights trained at and recursively generate future observations in blocks of four frames. The and settings correspond to extrapolation over two and four prediction blocks, respectively. Each model starts from four real historical frames. Each prediction block receives the corresponding four-step sequence of six-dimensional actions, and generated observations are used as history for the next block. The results are taken from the first eight steps of the same sixteen-step trajectory to preserve the same noise and prediction prefix.
The compared methods include PredRNN-Act [22, 23], PredRNN-V2-Act [23], SimVP [24], IRASim-S/4 [6], DINO-WM [5], MAU [25], and MMVP [26]. The PredRNN models use the action-conditioned extensions released by their authors. SimVP, MAU, and MMVP use implementations from OpenSTL [27]. Each baseline retains its original backbone, with separate interfaces adapting action inputs and observation representations. The PredRNN models, SimVP, IRASim, MAU, and MMVP share the same frozen domain-specific VAE. DINO-WM uses its native DINOv2 feature dynamics and official RGB decoder. During recursive prediction, QCC feeds back the corrected latent variables and recomputes USFM features, JEPA features, and historical optical flow from generated frames to construct quotient states for the next prediction block. DINO-WM feeds back predicted patch features, while the remaining methods feed back predicted latent variables. All baselines are trained for 30 epochs and evaluated using the raw final-epoch weights.
Based on recorded timestamps, the median temporal spans for are approximately 0.133 s, 0.267 s, and 0.533 s, respectively.
Four-Step Prediction
Table 2 reports four-step prediction results. QCC achieves the best mean values across all five metrics. Compared with SimVP, the baseline with the lowest ROI-MSE, QCC reduces ROI-MSE by 2.70% and increases PSNR by 0.862 dB. Compared with MMVP, the strongest baseline in perceptual quality and temporal prediction, QCC reduces LPIPS and temporal MSE by 47.84% and 13.06%, respectively. These results indicate improvements in pixel reconstruction, perceptual quality, and prediction of inter-frame changes within the short prediction window, with the largest relative reduction observed in perceptual distance.
| Method | ROI-MSE | PSNR (dB) | SSIM | LPIPS | Temporal MSE |
| PredRNN-Act [22] | |||||
| PredRNN-V2-Act [23] | |||||
| SimVP+Act [24] | |||||
| IRASim-S/4 [6] | |||||
| DINO-WM+Dec. [5] | |||||
| MAU+Act [25] | |||||
| MMVP+Act [26] | |||||
| QCC-WM |
Table 2. Cumulative prediction results at .
Eight-Step Extrapolation
Table 3 reports results after recursively generating two prediction blocks. QCC maintains the best mean values in ROI-MSE, PSNR, LPIPS, and temporal MSE. Compared with the best baseline for each metric, QCC reduces ROI-MSE by 2.05% relative to SimVP, and reduces LPIPS and temporal MSE by 41.68% and 5.65%, respectively, relative to PredRNN-V2. QCC achieves an SSIM of 0.64475, close to MMVP at 0.64526. Reconstruction errors and perceptual distances increase for all methods compared with four-step prediction. QCC nevertheless retains a substantial LPIPS reduction beyond the training window, indicating a perceptual quality advantage in multi-step extrapolation.
| Method | ROI-MSE | PSNR (dB) | SSIM | LPIPS | Temporal MSE |
| PredRNN-Act [22] | |||||
| PredRNN-V2-Act [23] | |||||
| SimVP+Act [24] | |||||
| IRASim-S/4 [6] | |||||
| DINO-WM+Dec. [5] | |||||
| MAU+Act [25] | |||||
| MMVP+Act [26] | |||||
| QCC-WM |
Table 3. Cumulative prediction results at .
Sixteen-Step Extrapolation
Table 4 reports results after recursively generating four prediction blocks. QCC retains the best mean values in ROI-MSE,
| Method | ROI-MSE () | PSNR (dB) | SSIM | LPIPS | Temporal MSE () |
| PredRNN-Act [22] | |||||
| PredRNN-V2-Act [23] | |||||
| SimVP+Act [24] | |||||
| IRASim-S/4 [6] | |||||
| DINO-WM+Dec. [5] | |||||
| MAU+Act [25] | |||||
| MMVP+Act [26] | |||||
| QCC-WM |
Table 4. Cumulative prediction results at .
Note: All methods use the same 226 test starting points. Results are the mean sample standard deviation across three training seeds. Both MSE metrics are reported in units of ; bold indicates the best mean. Act denotes an action-conditioning interface, and Dec. denotes the official RGB decoder.
Note: All methods use the same 226 test starting points. Results are the mean sample standard deviation across three training seeds. Both MSE metrics are reported in units of ; bold indicates the best mean. Act denotes an action-conditioning interface, and Dec. denotes the official RGB decoder.
PSNR, LPIPS, and temporal MSE. Its LPIPS is 0.16209, a 30.23% reduction relative to the best baseline, PredRNN-V2. QCC achieves an ROI-MSE of 0.056069, which is 0.77% lower than MMVP at 0.056504, and a temporal MSE of 0.007315, which is 1.55% lower than PredRNN-V2. As the extrapolation horizon increases, QCC’s relative advantages in pixel and temporal errors decrease, while its advantage in perceptual distance remains substantial. For SSIM, MMVP achieves the best result at 0.56690. QCC achieves 0.55178, below MMVP, SimVP, and PredRNN-Act, indicating that these methods provide higher structural similarity at this horizon.
Across the three prediction horizons, QCC achieves the best mean values for all five metrics at four steps and maintains its lead in ROI-MSE, PSNR, LPIPS, and temporal MSE at eight and sixteen steps. Its LPIPS reductions relative to the best baseline at each horizon are 47.84%, 41.68%, and 30.23%, respectively, representing its most prominent performance advantage across prediction horizons.
As the prediction horizon increases from 4 to 16 steps, QCC’s cumulative ROI-MSE rises from 0.01861 to 0.05606, and LPIPS increases from 0.0731 to 0.1620, approximately 2.92 and 2.19 times their four-step values, respectively. PSNR decreases by 5.145 dB, and SSIM decreases by 0.19010. This trend reflects error accumulation during recursive prediction. Overall, QCC maintains favorable perceptual quality during multi-step extrapolation, while structural similarity at longer horizons remains an area for improvement.
Experiments on an Additional Dataset
To evaluate the transfer performance of QCC-WM on additional ultrasound data, we conduct supplementary experiments on the public Freehand US dataset [28, 29]. The dataset contains ultrasound scans of both forearms from 19 volunteers, with 12 scanning sequences per volunteer and 228 sequences in total. Acquisition covers straight, C-shaped, and S-shaped trajectories, with the probe oriented parallel or perpendicular to the forearm. The B-mode images have an original resolution of 480 × 640 and a sampling rate of approximately 20 Hz. The released data also include optically tracked poses and calibration information. We represent calibrated relative SE(3) transformations between adjacent frames using three-dimensional translation and three-dimensional axis-angle rotation, forming six-dimensional action conditions for predicting subsequent ultrasound images.
The training, validation, and test sets are split by subject and contain 13, 3, and 3 subjects, respectively. The training set provides 1437 four-step prediction windows. The validation set contains 207 windows with at least 16 consecutive future steps, and the test set contains 278 common prediction starting points. Images are resized to 224 × 224 and normalized to [−1, 1]. We retain the original temporal sampling and align images with poses using the official calibration information. All methods receive four historical frames and the corresponding action sequences and are trained with a supervision horizon of . During evaluation, the same model weights are reused to recursively generate predictions at and in four-frame blocks, with generated observations fed back beNote: All methods use the same 226 test starting points. Results are the mean sample standard deviation across three training seeds. Both MSE metrics are reported in units of ; bold indicates the best mean. Act denotes an action-conditioning interface, and Dec. denotes the official RGB decoder.
tween blocks. All three horizons are evaluated at the same 278 test starting points. Their median temporal spans are approximately 0.196, 0.391, and 0.798 s, respectively.
Table 5 reports the comparative results on Freehand US. QCC-WM achieves the best mean values across all five metrics at all three horizons. At , QCC-WM reduces ROI-MSE by 33.09% and increases PSNR by 2.101 dB relative to SimVP-Act, the baseline with the lowest pixel error. It reduces LPIPS by 23.57% relative to PredRNN-Act, the baseline with the lowest perceptual distance. These results indicate that QCC-WM improves pixel reconstruction, structural similarity, and perceptual prediction quality in this transfer setting.
| Horizon | Method | ROI-MSE | PSNR/dB | SSIM | LPIPS | Temporal MSE |
| PredRNN-Act [22] | 4.3820.150 | 29.7750.159 | 0.82780.0007 | 0.14480.0014 | 1.5530.014 | |
| PredRNN-V2-Act [23] | 4.5240.118 | 29.6420.107 | 0.82830.0007 | 0.14750.0008 | 1.5610.009 | |
| SimVP-Act [24] | 4.0560.015 | 30.0480.015 | 0.84080.0004 | 0.19330.0008 | 1.6630.007 | |
| IRASim-S/4 [6] | 20.2379.006 | 24.2852.010 | 0.63250.0457 | 0.21300.0292 | 14.1244.614 | |
| DINO-WM [5] | 8.1480.509 | 27.2200.208 | 0.75270.0010 | 0.25000.0061 | 2.6310.069 | |
| MAU-Act [25] | 4.2030.182 | 29.9220.187 | 0.83070.0013 | 0.20480.0010 | 1.5810.016 | |
| MMVP-Act [26] | 4.1390.093 | 30.0390.099 | 0.84500.0035 | 0.14690.0113 | 1.5220.050 | |
| QCC-WM | 2.7140.014 | 32.1490.021 | 0.86260.0002 | 0.11070.0009 | 1.1320.003 | |
| PredRNN-Act [22] | 6.0800.099 | 28.5500.094 | 0.78960.0004 | 0.17770.0013 | 1.3150.005 | |
| PredRNN-V2-Act [23] | 6.2630.241 | 28.4160.154 | 0.79000.0003 | 0.18100.0010 | 1.3150.004 | |
| SimVP-Act [24] | 5.3850.039 | 28.9990.022 | 0.81060.0003 | 0.22050.0008 | 1.4020.004 | |
| IRASim-S/4 [6] | 35.12719.700 | 22.3392.483 | 0.57070.0577 | 0.24820.0329 | 13.2974.581 | |
| DINO-WM [5] | 10.3520.792 | 26.3120.252 | 0.73600.0006 | 0.25710.0063 | 1.8940.035 | |
| MAU-Act [25] | 5.7970.309 | 28.7280.230 | 0.79790.0022 | 0.22380.0018 | 1.3390.007 | |
| MMVP-Act [26] | 5.7020.143 | 28.8510.108 | 0.80990.0036 | 0.18830.0121 | 1.3030.037 | |
| QCC-WM | 4.4030.005 | 30.4130.011 | 0.82510.0004 | 0.13910.0015 | 1.1060.001 | |
| PredRNN-Act [22] | 9.4240.632 | 26.9340.168 | 0.74470.0010 | 0.21820.0009 | 1.1960.008 | |
| PredRNN-V2-Act [23] | 9.5330.569 | 26.8410.229 | 0.74420.0014 | 0.22060.0013 | 1.1930.003 | |
| SimVP-Act [24] | 8.2600.102 | 27.4580.039 | 0.76980.0007 | 0.25000.0012 | 1.2760.003 | |
| IRASim-S/4 [6] | 69.11351.126 | 20.0353.209 | 0.49150.0785 | 0.29990.0387 | 13.4495.119 | |
| DINO-WM [5] | 15.3341.459 | 24.8490.319 | 0.70810.0021 | 0.27090.0070 | 1.5080.021 | |
| MAU-Act [25] | 9.4800.691 | 26.9500.291 | 0.75320.0054 | 0.24830.0045 | 1.2230.008 | |
| MMVP-Act [26] | 8.5500.249 | 27.3480.129 | 0.76770.0040 | 0.23440.0096 | 1.1940.025 | |
| QCC-WM | 7.7750.102 | 28.3490.024 | 0.77880.0003 | 0.18520.0031 | 1.0950.001 |
Table 5. Action-conditioned ultrasound video prediction on Freehand US. All methods use the same 278 test starting points. Results are the mean sample standard deviation across three random seeds. At , models trained for four-step prediction are reused for longer-horizon extrapolation. ROI-MSE and temporal MSE are reported in units of . Bold indicates the best mean at each prediction horizon.
As the prediction horizon increases, ROI-MSE generally rises and SSIM decreases across methods, reflecting error accumulation during recursive prediction. At and , QCC-WM still reduces ROI-MSE by 18.24% and 5.88%, respectively, relative to the best baseline at each horizon. At , QCC-WM achieves an SSIM of 0.7788, exceeding SimVP-Act at 0.7698, and reduces temporal MSE by 8.17% relative to PredRNN-V2-Act, the best baseline for this metric. Although the relative pixel-error advantage narrows as the extrapolation horizon increases, QCC-WM maintains favorable structural and temporal prediction performance on this additional dataset.
Conclusion
We presented QCC-WM, an action-conditioned ultrasound video prediction model that constructs macro-states with spatial correspondence through action-guided quotient mappings. By combining recursive state prediction, motion-guided reference transport, and readback correction under the same quotient measurement, QCC-WM connects macro-level dynamics with future observation generation. Comparative experiments on US-VLA and transfer experiments on Freehand US show improved ultrasound video prediction quality and a sustained perceptual advantage during recursive extrapolation beyond the training window. Ablation studies and mechanism analyses further show that motion-guided reference transport mainly improves pixel reconstruction, while node-level error correction further improves perceptual quality and consistency between generated observations and macro-level predictions. Future work will focus on improving structural preservation in long-horizon prediction and evaluating the framework for closed-loop ultrasound navigation.
References
- [1]B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, Y. Bengio, Toward causal representation learning, Proceedings of the IEEE 109 (5) (2021) 612–634. doi:10.1109/JPROC.2021.3058954. URL https://doi.org/10.1109/JPROC.2021.3058954
- [2]K. Chalupka, F. Eberhardt, P. Perona, Multi-level cause-effect systems, in: Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, Vol. 51 of Proceedings of Machine Learning Research, PMLR, 2016, pp. 361–369. URL https://proceedings.mlr.press/v51/chalupka16.htmlarxiv.org/abs/1512.07942
- [3]E. P. Hoel, L. Albantakis, G. Tononi, Quantifying causal emergence shows that macro can beat micro, Proceedings of the National Academy of Sciences 110 (49) (2013) 19790–19795. doi:10.1073/pnas.1314922110. URL https://doi.org/10.1073/pnas.1314922110
- [4]P. K. Rubenstein, S. Weichwald, S. Bongers, J. M. Mooij, D. Janzing, M. Grosse-Wentrup, B. Schölkopf, Causal consistency of structural equation models, in: Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence, 2017, pp. 151–160. URL https://proceedings.mlr.press/r15/rubenstein-17a.htmlarxiv.org/abs/1707.00819
- [5]G. Zhou, H. Pan, Y. LeCun, L. Pinto, DINO-WM: World models on pre-trained visual features enable zero-shotarxiv.org/abs/2411.04983


