betaFeedback

WorksComputer Science

QCC-WM: Linking Macro-State Dynamics and Visual Generation via Quotient Causal Construction

Abstract

A key challenge in action-conditioned world modeling is to construct macro-states from high-dimensional observations that support recursive prediction under actions, while keeping their predicted evolution consistent with visual changes in generated observations. We propose QCC-WM, a visual world model based on Quotient-based Causal Construction. QCC-WM uses action responses to guide macro-variable construction for recursive action-conditioned prediction. A shared reference quotient measurement constrains the consistency between predicted states and observed changes, allowing macro-level predictions to provide structured constraints for future visual generation. Using translational and rotational actions and their predicted responses, the model aggregates local visual features into macro-nodes with spatial correspondence through causal abstraction, and recursively predicts node content and motion under a shared reference measurement. Macro-level predictions guide future observation generation, while predicted motion transports historical appearance information. Node-level errors between generated observations and macro-level targets are then mapped back to the corresponding spatial regions for correction, improving consistency between visual generation and macro-level predictions. On the US-VLA dataset, QCC-WM achieves an ROI-MSE of 0.01861 for four-step prediction. Compared with the best baseline at each prediction horizon, it reduces LPIPS by 48.59%, 41.68%, and 30.23% for four-step prediction and eight- and sixteen-step recursive extrapolation, respectively. Ablation studies and mechanism analyses show that motion-guided reference transport mainly improves pixel reconstruction, while node-level error feedback further improves perceptual quality and consistency between generated observations and macro-level predictions.

Introduction

Causal representation learning (CRL) aims to identify high-level variables related to the underlying generative mechanisms from high-dimensional observations, providing representations for analyzing action outcomes, intervention effects, and distribution shifts 1. For world models, this goal further requires learned states to evolve recursively under actions while maintaining their correspondence with observable changes. However, a fundamental challenge in high-dimensional visual settings is that macro-level causal variables suitable for recursive prediction are generally not given and must be constructed from fine-grained observations. Local features can describe visual appearance and changes, but they do not necessarily form macro-states with consistent measurement definitions and predictable transition rules. Building action-conditioned world models therefore requires addressing both

how macro-variables are formed and how they represent state evolution under successive actions.

Research on macro-variable construction and causal abstraction offers theoretical insights into this problem. Chalupka et al. studied how macro-level cause and effect variables can be constructed from the causal behavior of micro-level measurements 2. Hoel et al. showed that, for specific systems and under the effective information measure, suitable macro-level descriptions can exhibit greater causal effectiveness than micro-level descriptions 3. Rubenstein et al. further emphasized that causal models at different levels should yield consistent predictions under explicitly defined intervention mappings 4. These studies suggest that micro-to-macro mappings should be designed around the mechanisms and dynamics they preserve. For world models, we therefore consider a dynamics-oriented abstraction principle: macro-variable construction should be aligned with future prediction so that abstract state changes correspond to evolution at the observation level.

Existing visual world models have explored various state representations and prediction methods. DINO-WM learns action-conditioned dynamics in pretrained spatial features [5], while IRASim incorporates action trajectories into a diffusion model to generate future observations [6]. For structured modeling, SlotFormer learns recursive dynamics over object-level representations [7], and Slot-JEPA organizes prediction through semantic slots and spatial masks [8]. Motion-I2V uses predicted motion to transport reference features, connecting motion modeling with visual generation [9]. These studies provide important foundations for state prediction, structured representations, and spatially organized generation. Recursive macro-state prediction nevertheless raises several questions: whether nodes have comparable measurement definitions across time steps, whether predicted state changes are constrained by observed evolution, and whether macro-level predictions retain the correspondence needed to map them back to observation space. These questions directly affect the preservation of state meaning and the accumulation of prediction errors during recursive prediction.

To address these issues, we introduce QCC (Quotient-based Causal Construction) and develop the action-conditioned world model QCC-WM. QCC uses action-guided soft quotient mappings to aggregate local visual features into macro-nodes with spatial support. It recursively predicts content and motion states through inter-node dependencies and action-conditioned transitions. Within each prediction window, the model fixes the reference quotient mapping and its alignment, and constrains macro-state prediction through supervision on states and their temporal increments. This encourages consistency with observed evolution under the same quotient measurement.

To translate macro-level predictions into observation generation, the model uses predicted motion to transport reference features and latent variables, construct observable macro-level targets, and correct the initial generation. It then reads back the generated observations through the same quotient measurement, computes node-level errors relative to the macro-level targets, and maps these errors back to local latent space through the node–space correspondence. The reference quotient mapping thus connects node construction, macro-state prediction, and spatial correction, linking macro-level dynamics with future observation generation.

Our main contributions are as follows:

  1. An action-guided quotient-based causal construction mechanism, QCC. We construct macro-nodes with spatial support through soft quotient mappings, discover a macro-level causal structure of temporal dependencies among nodes under a shared reference measurement, and use this structure to organize action-conditioned dynamics. This connects macro-node construction and structure discovery with recursive prediction of content and motion states.

  1. A macro-level dynamics learning method under a fixed reference quotient measurement. We fix the reference quotient mapping and content alignment within each prediction window to provide consistent measurement definitions for macro-states across time steps. Supervision on

states and their temporal increments constrains recursive content and motion prediction, encouraging consistency between macro-level predictions and observed evolution under the same reference measurement.

  1. A generation correction interface combining motion transport with readback under the same quotient measurement. We use predicted motion to transport historical appearance information, then map node-level errors obtained through readback to the generation latent space through spatial correspondence, improving consistency between future observations and macro-level predictions.

The remainder of this paper is organized as follows. Section 2 reviews related work on causal representation learning, structured world models, and motion-guided generation. Section 3 presents QCC-WM, including quotient-based causal construction, macro-state prediction, motion-guided reference transport, spatial correction, and staged optimization. Section 4 describes the experimental setup and evaluates model performance and module contributions through comparative experiments, ablation studies, and mechanism analyses. Section 5 concludes the paper and discusses future directions.

Related Work

Causal Representation Learning and Macro-Variable Construction

Causal representation learning seeks to identify latent variables related to generative mechanisms from high-dimensional observations, enabling mechanism analysis, intervention prediction, and generalization across distributions [1]. In dynamic visual settings, CITRIS uses temporal observations and known intervention targets to identify latent causal factors under specific assumptions [10]. BISCUIT studies causal variable learning with unknown binary interaction mechanisms and establishes corresponding identifiability conditions [11]. These studies show that causal interpretations depend closely on additional supervision and structural assumptions. QCC focuses on constructing macro-nodes for action-conditioned prediction. Its node measurements and dependency structures are learned and evaluated as predictive interfaces within the model; action inputs and interventions within the model alone do not provide such identifiability guarantees.

Macro-variable construction also concerns relationships between different levels of description. Chalupka et al. used causal partitions to connect micro-level measurements with macro-level cause and effect variables [2]. Hoel et al. analyzed macro-level causal effectiveness in specific systems using effective information [3]. Rubenstein et al. formalized intervention consistency across levels through exact transformations between structural equation models [4]. In reinforcement learning, DeepMDP learns state representations by predicting rewards and next-state distributions in latent space, and establishes connections to bisimulation [12]. These studies provide a basis for abstractions that preserve underlying dynamics. We apply this idea to spatial aggregation within visual observations: action-guided soft quotient mappings define macro-level measurements, whose future evolution is constrained to match observed changes under the same reference mapping. This spatial quotient construction addresses a different modeling object from full environment-state abstraction, and the prediction consistency imposed here does not imply strict intervention consistency.

Action-Conditioned Dynamics and Structured World Models

Latent world models support future prediction by learning transition rules in compact state spaces. PlaNet combines deterministic and stochastic transitions and introduces latent overshooting as a multi-step variational objective [13]. Dreamer optimizes behavior over imagined latent trajectories, using recursive state prediction for control [14]. DINO-WM learns action-conditioned patch dynamics from pretrained spatial features and performs planning through goal-feature prediction [5]. These methods connect latent state prediction with downstream decision-making. QCC further addresses how local observations are organized into macro-level prediction units. A shared reference measurement constrains node states and their temporal increments, providing comparable observation-based targets for recursive prediction in the abstract state space.

Structured world models explicitly organize prediction units within a scene. DINOSAUR learns object-level representations by reconstructing self-supervised pretrained features [15]. SlotFormer learns autoregressive dynamics over slots to predict future object states [7]. Slot-JEPA decomposes dense features into semantic slots and time-varying spatial masks, combining transition memory and transport objectives for prediction and planning [8]. These methods establish foundations for feature grouping, structured recursive prediction, and spatial correspondence. Our specific design incorporates action plans into quotient mapping construction and uses the same reference quotient mapping for macro-state measurement, future-state supervision, and mapping predictions back to spatial locations. QCC nodes are defined as action-conditioned aggregate measurements, whose predictive value must be assessed through appropriate controlled comparisons.

Generative World Models and Motion-Guided Spatial Feedback

Generative world models synthesize future observations from actions or structured states. IRASim models robotic interaction videos using a diffusion Transformer with frame-wise action conditioning [6]. To connect structured representations with generation, SlotDiffusion uses a slot-conditioned latent diffusion decoder and applies the learned representations to object dynamics modeling [16]. SlotDiT generates future slot trajectories from text-conditioned diffusion and then decodes them into images [17]. QCC uses macro-level content and motion as states for recursive prediction and converts these predictions into generation conditions in image latent space through explicit quotient mappings. Its focus is the interface between macro-level dynamics and observation generation.

Explicit motion modeling provides another way to connect state changes with visual synthesis. Motion-I2V first predicts pixel trajectories and then propagates reference image features through motion-enhanced temporal attention [9]. FLIP combines flow generation, flow-conditioned video generation, and value modeling for generative planning in visual space [18]. DeXplicit predicts explicit 3D states and deformations from actions and transports appearance information to guide video generation [19]. These studies provide direct precedents for motion prediction and reference content transport. We use a learned visual quotient space as an intermediate representation, where regional motion and content changes arise from recursive macro-state prediction and are mapped back to local space through the retained quotient mapping. Spatial feedback thus forms a prediction pipeline together with QCC node measurements and recursive dynamics. Its contribution is assessed through generation evaluation and interventions on the model interfaces.

Methods

Problem Formulation and QCC Overview

Given historical visual observations X=IL+1:0X = I_{L+1:0} and a known action plan A=(a0,…,aH−1)A = (a_0,\ldots,a_{H-1}), we learn the conditional distribution p(Y∣X,A)p(Y \mid X,A) of future observations Y=I1:HY = I_{1:H}. The central problem is to construct macro-state units that support action-conditioned prediction while preserving their connection to observation space, allowing their predictions to guide future generation.

We propose QCC (Quotient-based Causal Construction), which uses action-guided soft quotient mappings to organize local visual representations into macro-nodes with explicit spatial origins. QCC jointly defines the aggregation sources, reference measurements, and spatial correspondence of nodes, and establishes action-conditioned transitions and inter-node dependencies over the constructed macro-variables. We refer to these state units as macro-level causal nodes, which represent action-driven evolution and inter-node interactions within the model.

As shown in Figure 1, the framework consists of four components: quotient-based causal construction, action-conditioned macro-state prediction, motion-guided reference transport, and readback under the same quotient measurement followed by spatial correction:

Overview of QCC-WM

Figure 1. Overview of QCC-WM. QCC first constructs macro-nodes with spatial origins and reference measurements. Action-conditioned dynamics recursively predict content and motion states. Predicted motion transports reference features and latent variables to construct observable macro-level targets and perform motion-guided reference correction, respectively. The model then reads back the current generation under the same quotient measurement and maps the remaining node-level errors to the corresponding spatial locations to obtain future observations. The reference quotient mapping connects node construction, macro-state prediction, and spatial generation.

(Π0b,Γ0b,S0b)=Qϕb(X,Ab),S~k+1b=Fθb(S~kb,ak,Ab,ℓk+1b;Gzb,Gmb),x~1:H∼pψ(x~1:H∣X,A,CQCC),I~k=D(Xϵm+αERΠ0(ek;⋅)),(1)\begin{aligned} (\Pi_0^b,\Gamma_0^b,S_0^b) &= Q_\phi^b(X,A^b),\\ \widetilde{S}_{k+1}^b &= F_\theta^b(\widetilde{S}_k^b,a_k,A^b,\ell_{k+1}^b;G_z^b,G_m^b),\\ \widetilde{x}_{1:H} &\sim p_\psi(\widetilde{x}_{1:H}\mid X,A,C_{\mathrm{QCC}}),\\ \widetilde{I}_k &= D(X_\epsilon^m+\alpha E\mathcal{R}_{\Pi_0}(e_k;\cdot)), \tag*{(1)} \end{aligned}

where b∈B={T,R}b \in\mathcal{B} = \{\mathrm{T},\mathrm{R}\} denotes the quotient construction branch guided by translational or rotational action components, and AbA^b is the corresponding action guidance. Both dynamics models and the generator retain the complete action input. Π0b\Pi_0^b and Γ0b\Gamma_0^b denote the reference soft assignment and reference content alignment, respectively, while Gzb,GmbG_z^b,G_m^b specify the dependency supports for the content and motion channels. Skb=S_k^b = (Zkb,Mkb,gkb,mkb)(Z_k^b, M_k^b, g_k^b, m_k^b) contains node content, node motion memory, and their corresponding shared states; ℓk+1b\ell_{k+1}^b is the node control derived from the action responses predicted by QCC. CQCCC_{\mathrm{QCC}} denotes the generation conditions produced by macro-state prediction, x~k\tilde{x}_k is the initially generated latent variable, xkMx_k^M is the latent variable after motion-guided reference correction, eke_k is the remaining node-level error obtained through readback under the same quotient measurement, and DD is the observation decoder.

Action-Response-Guided Quotient-Based Causal Construction

We use the pretrained visual encoder EobsE_{\mathrm{obs}} of I-JEPA [20] to extract observation representations and obtain frame-wise content features through branch-specific mappings: Fkb=Φb(Eobs(Ik))F_k^b = \Phi^b(E_{\mathrm{obs}}(I_k)). Historical features and their local temporal differences are used to organize nodes and initialize motion, while frame-wise content features measure node states at each time step. Each branch maintains KK global prototypes {cjb}j=1K\{c_j^b\}_{j=1}^K, with prototype indices identifying the nodes. QCC uses the action plan to modulate the affinity between historical local features and prototypes. Combined with spatial distances, these affinities determine soft assignments Πpjb,proto\Pi_{pj}^{b,\mathrm{proto}} within the valid observation region. The assignments are normalized across nodes and refined by iteratively updating node centroids, thereby defining the local spatial support of each node.

Action-response-driven quotient refinement. To connect node organization with potential action-induced changes, QCC further predicts local action responses from historical descriptors. For future time step kk, let aˉkb=∑r=0k−1arb\bar{a}_k^b = \sum_{r=0}^{k-1} a_r^b denote the cumulative action guidance:

rˉk,pb=bϕ,kb(hpb)+Jϕ,kb(hpb,aˉkb)aˉkb,(2)\bar{r}_{k,p}^b = b_{\phi,k}^b(h_p^b) + J_{\phi,k}^b(h_p^b,\bar{a}_k^b)\bar{a}_k^b, \tag*{(2)}

where bϕ,kbb_{\phi,k}^b is the response base term conditioned on history, and Jϕ,kbJ_{\phi,k}^b parameterizes the interaction between local mechanisms and actions. The response includes cumulative motion and appearance changes relative to the reference latent variable, connecting feature grouping with the future changes to be represented.

We subtract the spatial mean of the action response term over the valid region, normalize it using training statistics, and concatenate it across time to form the local response descriptor χpb\chi_p^b. This descriptor compares relative responses across locations under the current action plan. Starting from the prototype grouping, QCC sequentially refines spatial assignments using history-conditioned response encodings and explicit mechanism–action response encodings. Both refinements alternate between response aggregation and assignment updates. For each refinement, let Πpjb,prior\Pi_{pj}^{b,\mathrm{prior}} denote the input assignment and χpb\chi_p^b the corresponding response descriptor:

χˉjb,(n)=∑pΠpjb,(n)χpb∑pΠpjb,(n)+ϵ,\bar{\chi}_j^{b,(n)} = \frac{\sum_p \Pi_{pj}^{b,(n)}\chi_p^b}{\sum_p \Pi_{pj}^{b,(n)}+\epsilon},
Π~pjb,(n+1)∝mpexp⁡[log⁡(Πpjb,prior+ϵ)−λr∥χpb−χˉjb,(n)∥22],\widetilde{\Pi}_{pj}^{b,(n+1)} \propto m_p \exp\left[\log\left(\Pi_{pj}^{b,\mathrm{prior}}+\epsilon\right)-\lambda_r\left\Vert\chi_p^b-\bar{\chi}_j^{b,(n)}\right\Vert_2^2\right],
Πpjb,new=(1−ζb)Πpjb,prior+ζbΠ~pjb,(3)\Pi_{pj}^{b,\mathrm{new}}=(1-\zeta_b)\Pi_{pj}^{b,\mathrm{prior}}+\zeta_b\widetilde{\Pi}_{pj}^b, \tag*{(3)}

where ζb\zeta_b is a bounded strength learned separately for each refinement layer, and the final assignment is denoted by Π0b\Pi_0^b. Node assignments are thus jointly constrained by historical spatial structure and predicted action responses, organizing local evolution information while preserving prototype identities. At inference, action responses are predicted entirely from history and the known action plan; observed future responses are used only for training supervision.

Reference quotient measurement and node initialization. QCC further uses historical descriptors and predicted action responses to produce a bounded reference content alignment Γ0b\Gamma_0^b, which remains fixed within the current prediction window. We define the normalized aggregation weights as

ω0,pjb=Π0,pjb∑qΠ0,qjb+ϵ,PΠ0b,Γ0b(Fj)=∑pω0,pjb(Γ0,pb⊙Fp).(4)\omega_{0,pj}^b=\frac{\Pi_{0,pj}^b}{\sum_q \Pi_{0,qj}^b+\epsilon}, \qquad\mathcal{P}_{\Pi_0^b,\Gamma_0^b}(F_j)=\sum_p\omega_{0,pj}^b\left(\Gamma_{0,p}^b\odot F_p\right). \tag*{(4)}

The corresponding reference quotient measurement is

M0b(I)=PΠ0b,Γ0b(Φb(Eobs(I))),Zkb,⋆=M0b(Ik).(5)\mathcal{M}_0^b(I)=\mathcal{P}_{\Pi_0^b,\Gamma_0^b}\left(\Phi^b(E_{\mathrm{obs}}(I))\right), \qquad Z_k^{b,\star}=\mathcal{M}_0^b(I_k). \tag*{(5)}

This definition unifies spatial assignment, content alignment, and frame-wise measurement. The initial node content is set to Z0b=Z0b,⋆Z_0^b=Z_0^{b,\star}. Motion memory is initialized by quotient aggregation of historical content changes, estimated historical optical flow, and local mechanism descriptors. Reference centroids are computed from the final assignments Π0b\Pi_0^b. Shared content and motion states are obtained through global aggregation over the valid region, providing a reference for overall changes to the local nodes.

Action-Conditioned Macro-State Prediction under a Fixed Quotient Measurement

Node dependencies and step-wise response control. We construct candidate node dependency supports through temporal dependency analysis of macro-level measurements from training observations, incorporating action and reference conditions [21]. The analysis controls for the complete action plan, reference support, and past measurements of shared states. Fixed directed supports Gzb,GmbG_z^b,G_m^b organize inter-node messages for the content and motion channels. For channel c∈{z,m}c\in\{z,m\}, incoming edge weights are defined as

Bijc,b=Gijbclip⁡(γijc,b,0,1)1+∑lGcl,jbclip⁡(γljc,b,0,1).(6)B_{ij}^{c,b}=\frac{G_{ij}^{b}\operatorname{clip}(\gamma_{ij}^{c,b},0,1)}{1+\sum_l G_{cl,j}^{b}\operatorname{clip}(\gamma_{lj}^{c,b},0,1)}. \tag*{(6)}

where γijc,b\gamma_{ij}^{c,b} is a learnable strength within the support, and the direction from ii to jj represents information transfer from node ii at the previous time step to node jj at the current time step. The directed supports remain fixed during recursive prediction, with content and motion using their respective message weights.

The action responses predicted by QCC also enter the recursive dynamics through step-wise node controls. We take temporal differences of the cumulative motion responses in (2) and aggregate them under the reference quotient mapping:

ℓk,jb=∑pω0,pjbψℓ(rˉk,p,motb−rˉk−1,p,motb),rˉ0,p,motb=0,(7)\ell_{k,j}^b=\sum_p\omega_{0,pj}^b\psi_\ell\left(\bar{r}_{k,p,\mathrm{mot}}^b-\bar{r}_{k-1,p,\mathrm{mot}}^b\right), \qquad\bar{r}_{0,p,\mathrm{mot}}^b=0, \tag*{(7)}

where ψℓb\psi_{\ell}^{b} is a bounded response encoding function. This control allows local action responses learned during construction to contribute to node updates at every step, jointly driving macro-state evolution with the complete action input.

Recursive evolution of shared and local states. To separate overall changes from local differences, we first update the shared states:

(gk+1z,b,gk+1m,b)=Fgb(gkz,b,gkm,b,ak,Ab),(8)\left(g_{k+1}^{z,b},g_{k+1}^{m,b}\right)=F_{g}^{b}\left(g_{k}^{z,b},g_{k}^{m,b},a_{k},A^{b}\right), \tag*{(8)}

and define local residuals as Z~k,jb=Zk,jb−gkz,b\widetilde{Z}_{k,j}^{b}=Z_{k,j}^{b}-g_{k}^{z,b} and M~k,jb=Mk,jb−gkm,b\widetilde{M}_{k,j}^{b}=M_{k,j}^{b}-g_{k}^{m,b}. Content messages combine node content, motion, reference positions, and fixed historical context, while motion messages are extracted from motion residuals:

mk,jz,b=∑iBijz,bVzb(Z~k,ib,M~k,ib,μib;X),m_{k,j}^{z,b}=\sum_{i}B_{ij}^{z,b}V_{z}^{b}\left(\widetilde{Z}_{k,i}^{b},\widetilde{M}_{k,i}^{b},\mu_{i}^{b};X\right),
mk,jm,b=∑iBijm,bVmb(M~k,ib),(9)m_{k,j}^{m,b}=\sum_{i}B_{ij}^{m,b}V_{m}^{b}\left(\widetilde{M}_{k,i}^{b}\right), \tag*{(9)}

where Vzb,VmbV_{z}^{b},V_{m}^{b} denote the message projections for the respective channels. Inter-node messages use the states at the start of each step, while each node continues to evolve within its residual update blocks.

Node content is updated through a residual mapping, while motion memory is updated through a gated recurrent unit:

Z~k+1,jb=Fzb(Z~k,jb,M~k,jb,mk,jz,b,ak,Ab,ϕk+1,jb,gkb;μb;X),\widetilde{Z}_{k+1,j}^{b}=F_{z}^{b}\left(\widetilde{Z}_{k,j}^{b},\widetilde{M}_{k,j}^{b},m_{k,j}^{z,b},a_{k},A^{b},\phi_{k+1,j}^{b},g_{k}^{b};\mu^{b};X\right),
Zk+1,jb=Z~k+1,jb+gk+1z,b,Z_{k+1,j}^{b}=\widetilde{Z}_{k+1,j}^{b}+g_{k+1}^{z,b},
M~k+1,jb=GRU⁡b(M~k,jb,ψmb(Z~k+1,jb,mk,jm,b,ak,ϕk+1,jb,μjb)),\widetilde{M}_{k+1,j}^{b}=\operatorname{GRU}^{b}\left(\widetilde{M}_{k,j}^{b},\psi_{m}^{b}\left(\widetilde{Z}_{k+1,j}^{b},m_{k,j}^{m,b},a_{k},\phi_{k+1,j}^{b},\mu_{j}^{b}\right)\right),
Mk+1,jb=M~k+1,jb+gk+1m,b,M_{k+1,j}^{b}=\widetilde{M}_{k+1,j}^{b}+g_{k+1}^{m,b},
u^k+1,jb=ρb(M~k+1,jb)+ρb(gk+1m,b),(10)\widehat{u}_{k+1,j}^{b}=\rho^{b}\left(\widetilde{M}_{k+1,j}^{b}\right)+\rho^{b}\left(g_{k+1}^{m,b}\right), \tag*{(10)}

where gkb=[gkz,b;gkm,b]g_{k}^{b}=[g_{k}^{z,b};g_{k}^{m,b}], u^k+1,jb\widehat{u}_{k+1,j}^{b} is the node motion response, and FzbF_{z}^{b} is the parameterized residual update mapping for node content in branch bb. This update combines each node’s history, dependency messages, shared changes, and local action responses to recursively predict future sequences from the same initial macro-state without future observations.

Content dynamics are learned under a fixed reference quotient measurement, connecting recursively predicted states with corresponding observed changes:

pr⁡zFθb,(k)(S0b,A)≈Mb(Ik),(11)\operatorname{pr}_{z}F_{\theta}^{b,(k)}\left(S_{0}^{b},A\right)\approx\mathcal{M}_{b}(I_{k}), \tag*{(11)}

where pr⁡z\operatorname{pr}_{z} extracts node content. Initial and future observations are measured using the same Π0b,Γ0b\Pi_{0}^{b},\Gamma_{0}^{b}, providing a common reference across prediction steps. Supervision is applied to both states and their temporal increments:

LQ=⟨λzMSE⁡(Z^b,Zb,⋆)+λnMSE⁡(LN⁡(Z^b),LN⁡(Zb,⋆))+λΔMSE⁡(ΔZ^b,ΔZb,⋆)⟩b,ΔZk=Zk−Zk−1,(12)\mathcal{L}_{\mathrm{Q}}=\left\langle\lambda_z\operatorname{MSE}(\widehat{Z}^{b},Z^{b,\star})+\lambda_n\operatorname{MSE}(\operatorname{LN}(\widehat{Z}^{b}),\operatorname{LN}(Z^{b,\star}))+\lambda_{\Delta}\operatorname{MSE}(\Delta\widehat{Z}^{b},\Delta Z^{b,\star})\right\rangle_b,\quad\Delta Z_k=Z_k-Z_{k-1}, \tag*{(12)}

where ⟨⋅⟩b\langle\cdot\rangle_b denotes averaging over branches, and MSE is averaged over samples, prediction steps, nodes, and channels. This constraint aligns macro-states and their changes under a fixed observation reference, providing state sequences with a common measurement basis for subsequent spatial generation.

Quotient-Mapping-Based Motion-Guided Reference Transport

QCC first uses spatial correspondence to condition the initial generation on macro-states. The model performs frame-level and temporal readouts of node content and motion, and injects node content increments and motion geometry into the generator’s local features through the reference quotient mapping. For example, the spatial field corresponding to node content changes is

ΔFkb(p)=∑jΠ0,pjb(Z^k,jb−Z^0,jb).(13)\Delta F_k^b(p)=\sum_j \Pi_{0,pj}^b(\widehat{Z}_{k,j}^b-\widehat{Z}_{0,j}^b). \tag*{(13)}

Together with macro-level motion conditions, this field forms CQCCC_{\mathrm{QCC}} and guides the generation of the initial latent block x~1:H\widetilde{x}_{1:H}, while retaining historical visual information and direct action conditioning.

To further translate macro-level motion into changes in observation space, we use the same reference assignment to recover local displacement fields and accumulate them over prediction steps:

ν~kb(p)=σf∑jΠ0,pjbμ~k,jb,dkb(p)=∑r=1kν~rb(p),(14)\widetilde{\nu}_k^b(p)=\sigma_f\sum_j\Pi_{0,pj}^b\widetilde{\mu}_{k,j}^b,\qquad d_k^b(p)=\sum_{r=1}^{k}\widetilde{\nu}_r^b(p), \tag*{(14)}

where σf\sigma_f converts normalized motion responses to the displacement scale required for transport. We define the reference transport operator through bilinear backward sampling as Wd(F)(p)=F(p−d(p))\mathcal{W}_d(F)(p)=F(p-d(p)), with displacements interpolated and rescaled to the corresponding feature grid.

Reference information is transported separately in content feature space and generation latent space. For branch bb, the reference content features F0bF_0^b are transported using the predicted displacement and measured through the fixed quotient interface to form an observable macro-level target:

C^kb=PΠ0b,Γ0b(Wdkb(F0b)),(15)\widehat{C}_k^b=\mathcal{P}_{\Pi_{0}^{b},\Gamma_{0}^{b}}\left(\mathcal{W}_{d_k^b}(F_0^b)\right), \tag*{(15)}

where C^kb\widehat{C}_k^b represents the observable changes induced by predicted motion in the reference content and is used for comparison with generated observations under the same quotient measurement. The recursively predicted latent state Z^kb\widehat{Z}_k^b supports macro-state prediction and generation conditioning. The two quantities thus serve latent prediction and observable correction, respectively.

In latent space, we average the displacement fields from the two branches to obtain dˉk\bar{d}_k and transport the latent variable x0x_0 of the last historical observation:

dˉk=1∣B∣∑bdkb,xkref=Wdˉk(x0).(16)\bar{d}_k=\frac{1}{|\mathcal{B}|}\sum_b d_k^b,\qquad x_k^{\mathrm{ref}}=\mathcal{W}_{\bar{d}_k}(x_0). \tag*{(16)}

QCC also converts node motion memory into a spatial gate:

gk(p)=κInterp⁡[1∣B∣∑b∑jΠ0,pjb⋅sigmoid⁡(hmb(M^k,jb))],(17)g_k(p)=\kappa\operatorname{Interp}\left[\frac{1}{|\mathcal{B}|}\sum_b\sum_j\Pi_{0,pj}^b\cdot\operatorname{sigmoid}\left(h_m^b(\widehat{M}_{k,j}^b)\right)\right], \tag*{(17)}

where hmbh_m^b is the node motion gating network, σ\sigma is the sigmoid function, and κ\kappa is a bounded strength. The gate depends on predicted motion memory and is expanded over the reference spatial support of each node. Motion-guided reference correction is defined as

xkM=x~k+αMgk⊙(xkref−x~k),(18)x_k^{\mathrm{M}}=\widetilde{x}_k+\alpha_{\mathrm{M}}g_k\odot\left(x_k^{\mathrm{ref}}-\widetilde{x}_k\right), \tag*{(18)}

where x~k\widetilde{x}_k is the initially generated latent variable, xkrefx_k^{\mathrm{ref}} is the reference latent variable transported using predicted motion, gkg_k contains spatial gating weights, αM\alpha_{\mathrm{M}} controls the overall correction strength, and ⊙\odot denotes element-wise multiplication. The difference between the reference latent variable and the initial generation defines the correction direction. The spatial gate controls the correction magnitude at each location, incorporating transported historical appearance information into the generation.

Readback under the Same Quotient Measurement and Spatial Correction of Node Errors

After motion-guided reference transport, QCC uses the same reference quotient measurement to evaluate the discrepancy between the current generation and the observable macro-level target. It first decodes the motion-corrected latent variable and reads back its macro-level content:

Ckb,M=M0b(D(xkM)),ekb=C^kb−Ckb,M.(19)C_k^{b,\mathrm{M}}=M_0^b(D(x_k^{\mathrm{M}})),\qquad e_k^b=\widehat{C}_k^b-C_k^{b,\mathrm{M}}. \tag*{(19)}

Because target construction and generation readback share Π0b,Γ0b,ekb\Pi_0^b,\Gamma_0^b,e_k^b has fixed node definitions and spatial origins. It represents changes that the current generation has not yet captured under the macro-level measurement. Readback follows motion-guided reference correction, allowing error correction to operate on generation results that have already been adjusted using reference motion.

Spatial feedback expands errors through the node–location relationships defined by QCC. One path directly projects node errors, while the other modulates error representations using predicted content and content read back from the generation:

Ekb,base(p)=∑jΠ0,pjbWbbek,jb,E_k^{b,\mathrm{base}}(p)=\sum_j\Pi_{0,pj}^bW_b^be_{k,j}^b,
rk,jb=Wrbek,jb⊙[1+ξb([C^k,jb;Ck,jb,M])],r_{k,j}^b=W_r^be_{k,j}^b\odot\left[1+\xi^b\left([\widehat{C}_{k,j}^b;C_{k,j}^{b,\mathrm{M}}]\right)\right],
Ekb,ctx(p)=∑jΠ0,pjbrk,jb.(20)E_k^{b,\mathrm{ctx}}(p)=\sum_j\Pi_{0,pj}^br_{k,j}^b. \tag*{(20)}

where WebW_e^b, WrbW_r^b are channel projections and ξb\xi^b is a node content modulation function. The error fields are interpolated to the generation latent grid and combined with local context from the current and reference latent variables to produce spatial corrections:

δxkE=Dbase({Eb;kb,base}b;xkM)+Dctx({Eb;kb,ctx}b;xkM,x0),x^k=xkM+αEδxkE,I^k=D(x^k),(21)\delta x_k^E = \mathcal{D}_{\mathrm{base}}\left(\{E_{b;k}^{b,\mathrm{base}}\}_{b};x_k^M\right) + \mathcal{D}_{\mathrm{ctx}}\left(\{E_{b;k}^{b,\mathrm{ctx}}\}_{b};x_k^M,x_0\right), \qquad \hat{x}_k = x_k^M + \alpha_E\delta x_k^E, \qquad \hat{I}_k = D(\hat{x}_k), \tag*{(21)}

where Dbase\mathcal{D}_{\mathrm{base}} and Dctx\mathcal{D}_{\mathrm{ctx}} include branch fusion, spatial interpolation, and local context modulation, and are collectively denoted by RΠ0\mathcal{R}_{\Pi_0}. This interface converts macro-level errors into generation adjustments with explicit spatial correspondence.

Learning Objectives and Staged Optimization

The model is trained for quotient construction, macro-level content prediction, conditional generation, and spatial correction. The main objective is summarized as

Lmain=λgroupLgroup+λQLQ+λgenLgen+λretLret,(22)\mathcal{L}_{\mathrm{main}} = \lambda_{\mathrm{group}}\mathcal{L}_{\mathrm{group}} + \lambda_Q\mathcal{L}_Q + \lambda_{\mathrm{gen}}\mathcal{L}_{\mathrm{gen}} + \lambda_{\mathrm{ret}}\mathcal{L}_{\mathrm{ret}}, \tag*{(22)}

where Lgroup\mathcal{L}_{\mathrm{group}} encourages quotient construction to preserve observed response information, LQ\mathcal{L}_Q supervises macro-level content prediction under a fixed reference measurement, Lgen\mathcal{L}_{\mathrm{gen}} trains conditional diffusion generation, and Lret\mathcal{L}_{\mathrm{ret}} trains spatial correction. Training also includes auxiliary supervision on local responses, node and local motion, shared states, and transport alignment, as detailed below. These objectives are applied to the corresponding modules in separate training stages. The spatial correction module is trained separately after the core model is frozen.

Local response prediction is supervised using cumulative optical flow and changes relative to the reference latent variable from training observations. Let rk⋆r_k^\star denote the observed response normalized using training statistics. We supervise Eq. (2) with both response prediction error and local response error after removing the spatial mean. Response reconstruction after node aggregation further encourages quotient construction to preserve information relevant to future evolution:

rˉk,jb,⋆=∑pω0,pjbrk,pb,⋆,Lgroup=⟨∥Π0brˉkb,⋆−rkb,⋆∥Ω2⟩b+λregLreg,(23)\bar{r}_{k,j}^{b,\star} = \sum_p \omega_{0,pj}^{b}r_{k,p}^{b,\star}, \qquad \mathcal{L}_{\mathrm{group}} = \left\langle\left\|\Pi_0^b\bar{r}_{k}^{b,\star}-r_k^{b,\star}\right\|_{\Omega}^{2}\right\rangle_b +\lambda_{\mathrm{reg}}\mathcal{L}_{\mathrm{reg}}, \tag*{(23)}

where Ω\Omega denotes the valid support, and Lreg\mathcal{L}_{\mathrm{reg}} regularizes node occupancy and near-duplicate normalized assignment columns. These objectives connect node organization with observed responses during training, while construction itself uses only history and actions.

Content learning in the macro-level dynamics uses the quotient consistency objective in Eq. (12). Node motion targets are obtained by aggregating observed optical flow fkf_k under the same reference assignment:

uk,jb,⋆=∑pω0,pjbfk,p,f^k,pb=∑jΠ0,pjbu^k,jb,(24)u_{k,j}^{b,\star} = \sum_p \omega_{0,pj}^{b}f_{k,p}, \qquad \hat{f}_{k,p}^{b} = \sum_j \Pi_{0,pj}^{b}\hat{u}_{k,j}^{b}, \tag*{(24)}

where predictions and targets use the same motion scale. The corresponding losses compare node motion u^b\hat{u}^{b} with ub,⋆u^{b,\star} and reconstructed local fields f^b\hat{f}^{b} with ff, while also supervising shared content increments and shared motion. Construction and response adaptation additionally use local motion errors and alignment errors between transported references and true future latent variables to coordinate response control, macro-level motion, and reference transport.

Initial observations are generated in a pretrained visual latent space. Let xk=Evae(Ik)x_k = E_{\mathrm{vae}}(I_k). We construct diffusion targets using the reference residual rkx=(xk−x0)/σxr_k^x=(x_k-x_0)/\sigma_x:

xs,k=αˉsrkx+1−αˉsϵk,vs,k⋆=αˉsϵk−1−αˉsrkx.(25)x_{s,k}=\sqrt{\bar{\alpha}_s}r_k^x+\sqrt{1-\bar{\alpha}_s}\epsilon_k, \qquad v_{s,k}^{\star}=\sqrt{\bar{\alpha}_s}\epsilon_k-\sqrt{1-\bar{\alpha}_s}r_k^x. \tag*{(25)}

Conditioned on history, complete actions, and CQCC, the generator minimizes a denoising objective weighted by the valid region:

Lgen=E[w(p)(v^ψ,s,k(p,C)−vs,k⋆(p,C))2]E[w(p)],(26)\mathcal{L}_{\mathrm{gen}}= \frac{\mathbb{E}\left[w(p)\left(\hat{v}_{\psi,s,k}(p,\mathcal{C})-v_{s,k}^{\star}(p,\mathcal{C})\right)^2\right]} {\mathbb{E}[w(p)]}, \tag*{(26)}

where w(p)w(p) is the valid observation region weight, and the expectation averages over samples, diffusion steps, future steps, positions, and channels cc.

The spatial correction module is trained using initial samples generated by the frozen core model. For its training output IkretI_k^{\mathrm{ret}}, we combine image reconstruction, perceptual distance, structural similarity, and generation–prediction and generation–observation alignment under the same quotient measurement:

Lret=λpixLpix(Iret,I)+λperLper(Iret,I)+λstrLstr(Iret,I)+λP⟨MSE⁡(M0b(Iret),C^b)⟩b+λO⟨MSE⁡(M0b(Iret),M0b(I))⟩b,(27)\mathcal{L}_{\mathrm{ret}} = \lambda_{\mathrm{pix}}\mathcal{L}_{\mathrm{pix}}(I^{\mathrm{ret}},I) +\lambda_{\mathrm{per}}\mathcal{L}_{\mathrm{per}}(I^{\mathrm{ret}},I) +\lambda_{\mathrm{str}}\mathcal{L}_{\mathrm{str}}(I^{\mathrm{ret}},I) +\lambda_P\left\langle\operatorname{MSE}(\mathcal{M}_0^b(I^{\mathrm{ret}}),\hat{\mathcal{C}}^b)\right\rangle_b +\lambda_O\left\langle\operatorname{MSE}(\mathcal{M}_0^b(I^{\mathrm{ret}}),\mathcal{M}_0^b(I))\right\rangle_b, \tag*{(27)}

where Lpix\mathcal{L}_{\mathrm{pix}} is image MSE over the valid region, Lper\mathcal{L}_{\mathrm{per}} is perceptual distance, and Lstr\mathcal{L}_{\mathrm{str}} is structural similarity loss. The two node-level alignment terms encourage generated observations to match the observable macro-level targets and the true future observations under a shared measurement, respectively.

Results

Ablation Study

To examine the effect of QCC branch configurations on prediction performance, we compare four settings: a direct action-conditioned baseline (Direct), a single-branch QCC model with only the translation-guided branch (T-only), a single-branch QCC model with only the rotation-guided branch (R-only), and the complete QCC-WM with both branches. The T-only and R-only models are trained independently, with six macro-nodes per branch. The complete system combines dual-branch macro-state prediction, motion-guided reference transport, and spatial correction of node-level errors. All configurations use the same evaluation split and sampling settings. Results are reported as the mean and sample standard deviation across three training seeds.

As shown in Table 1, both single-branch QCC variants outperform Direct in the mean values of all five metrics, indicating that action-guided macro-level modeling improves future observation prediction. The two single-branch configurations perform similarly overall: T-only achieves slightly lower ROI-MSE and temporal MSE, while R-only performs slightly better in SSIM and LPIPS. Both achieve a PSNR of approximately 24.98 dB.

SettingROI-MSE ↓\downarrow (×10−3\times10^{-3})PSNR ↑\uparrow (dB)SSIM ↑\uparrowLPIPS ↓\downarrowTemporal MSE ↓\downarrow (×10−3\times10^{-3})
Direct20.46±0.0920.46 \pm0.0924.91±0.0124.91 \pm0.010.7440±0.00030.7440 \pm0.00030.0790±0.00060.0790 \pm0.00066.17±0.046.17 \pm0.04
Single QCC: T-only19.90±0.0919.90 \pm0.0924.98±0.0224.98 \pm0.020.7450±0.00090.7450 \pm0.00090.0782±0.00060.0782 \pm0.00066.05±0.046.05 \pm0.04
Single QCC: R-only19.98±0.1019.98 \pm0.1024.98±0.0424.98 \pm0.040.7451±0.00110.7451 \pm0.00110.0779±0.00070.0779 \pm0.00076.06±0.056.06 \pm0.05
Dual QCC: T and R18.61±0.0818.61 \pm0.0825.17±0.0225.17 \pm0.020.7492±0.00010.7492 \pm0.00010.0731±0.00020.0731 \pm0.00025.76±0.025.76 \pm0.02

Table 1. Prediction performance of different QCC configurations. Dual QCC is the complete system with macro-state prediction, motion-guided reference transport, and spatial correction of node-level errors. Note: Results are reported as the mean ±\pm sample standard deviation across three training seeds. Both MSE metrics are reported in units of 10−310^{-3}. Bold indicates the best mean in each column.

The complete dual-branch QCC system achieves the best mean values across all five metrics. Compared with Direct, it reduces ROI-MSE, LPIPS, and temporal MSE by 9.05%, 7.51%, and 6.64%, respectively. It also improves all metrics over both single-branch variants, indicating that combining dual-branch macro-state prediction with subsequent spatial correction improves pixel reconstruction, structural similarity, perceptual quality, and prediction of inter-frame changes.

Mechanism Analysis of QCC-WM

We analyze QCC from three perspectives: spatial node organization, motion prediction, and generation correction.

Spatial Organization of Macro-Nodes

QCC aggregates local visual features through action-guided soft quotient mappings while preserving the spatial support of each node in the reference observation. The mapping remains fixed within each prediction window, ensuring that node-state comparison and subsequent spatial feedback use the same spatial correspondence. Figure 2 shows that nodes in both branches cover the valid imaging region and form distinct local aggregation regions, providing explicit spatial origins for macro-states.

Spatial origins and effective occupancy of QCC macro-nodes

Figure 2. Spatial origins and effective occupancy of QCC macro-nodes. Left: reference observations and soft quotient assignments of the T and R branches for three samples. Colors indicate node indices within each branch. Right: effective number of occupied nodes, minimum node mass, and cosine similarity between assignment columns. Diamonds and error bars indicate the mean and sample standard deviation across three training seeds; circles show individual seed results. Dashed lines mark the six predefined nodes and the 1% low-occupancy threshold, respectively.

We define the effective number of nodes using normalized node masses pjp_j as Keff=exp⁡(−∑jpjlog⁡pj)K_{\mathrm{eff}} = \exp(-\sum_j p_j \log p_j). With six nodes per branch, the effective node counts of the T and R branches are 5.42±0.175.42 \pm0.17 and 5.50±0.195.50 \pm0.19, respectively. Their mean minimum node masses are (7.53±0.73)%(7.53 \pm0.73)\% and (8.34±1.46)%(8.34 \pm1.46)\%, and the cosine similarities between assignment columns are 0.0689±0.00250.0689 \pm0.0025 and 0.0696±0.00670.0696 \pm0.0067. These results indicate that node capacity is well utilized and that spatial weights have limited overlap across nodes. This spatial organization allows macro-level motion and node-level errors to be mapped back to their corresponding local regions.

Macro-Level Motion and Local Displacement

The action-conditioned dynamics predict the motion response of each node and recover local displacement fields through the reference quotient mapping. Motion differences between macro-nodes can therefore produce different displacements across image regions, driving reference information transport. Figure 3(a,b) compares predicted displacements with zero-motion, local constant-velocity, and global constant-velocity extrapolation. Using observed optical flow as the reference, we compute the MSE of two-dimensional cumulative displacement over the valid region and the corresponding error after removing its spatial mean.

Figure 3. Macro-level motion prediction. (a) Cumulative displacement MSE. (b) MSE after removing the spatial mean of the cumulative error. Optical flow estimated from observations serves as the reference. Curves and shaded regions show the mean and sample standard deviation across three random seeds. (c,d) Cumulative responses of T and R nodes in Case 2: open circles indicate aggregated observed optical flow, and crosses indicate predictions. Successive markers correspond to consecutive prediction steps. Displacements are enlarged sixfold for visualization.

QCC achieves a four-step average cumulative displacement MSE of 0.5980±0.00910.5980 \pm0.0091, lower than the three controls at 0.78030.7803, 0.94510.9451, and 0.70710.7071. After removing the spatial mean of the cumulative error, QCC achieves an MSE of 0.3321±0.00530.3321 \pm0.0053, which remains below 0.35270.3527 for both zero-motion and global constant-velocity extrapolation, and 0.59060.5906 for local constant-velocity extrapolation. These results indicate that macro-level dynamics reduce cumulative displacement errors and further improve the prediction of local motion differences. Figure 3(c,d) illustrates differences in the direction and magnitude of node responses, showing how the fixed quotient mapping translates macro-level predictions into spatially varying motion fields.

Complementary Effects of Motion Transport and Error Correction

Motion-guided reference transport moves historical appearance information according to predicted displacements and corrects the initial generation through spatial gating. Readback under the same quotient measurement then measures the remaining node-level discrepancies and maps them back to local latent space. Figure 4 examines the two paths using four settings that independently enable or disable each correction. Node-level errors are recomputed from the generated results under each setting.

Figure 4. Complementary effects of motion transport and node-level error correction. (a–c) ROI-MSE, LPIPS, and generation–prediction node MSE under four interface settings. The horizontal axis indicates whether motion-guided reference correction is enabled. Gray and teal indicate disabled and enabled node-level error correction, respectively. Error bars show the sample standard deviation across three random seeds. Below: local crops at prediction step 4 in Case 2, showing the reference observation, transported reference, true future observation, initial generation, and generation with motion correction only.

Enabling motion-guided reference correction alone reduces ROI-MSE from 0.019849±0.0000420.019849 \pm0.000042 to 0.018627±0.0000960.018627 \pm0.000096, indicating that transported historical appearance improves pixel reconstruction. However, Fig. 4(b,c) shows increases in both LPIPS and generation–prediction node error. Readback after motion transport is therefore needed to correct the remaining discrepancies from the macro-level targets. Adding node-level error correction reduces LPIPS from 0.079051±0.0003750.079051 \pm0.000375 to 0.073059±0.0002430.073059 \pm0.000243, a relative reduction of 7.58% (paired 95% interval: [6.39%, 8.93%]). The two paths thus have complementary effects: motion transport provides spatially relevant appearance references, while node-level error correction mainly improves perceptual quality and macro-level alignment. Compared with disabling both corrections, the complete interface reduces ROI-MSE, LPIPS, and temporal MSE by 6.25%, 6.45%, and 4.55%, respectively.

Effects of Readback and Spatial Correspondence

Readback uses the fixed reference measurement M0\mathcal{M}_0 to compare the observable macro-level target C^k\widehat{C}_k produced by motion transport with the current generation, giving node-level errors a common measurement definition and spatial origin. To evaluate correction, we compute node-level errors between generated observations and both macro-level targets and true future observations:

EG−P(k)=MSE⁡(M0(U^k),C^k),(28)E_{\mathrm{G-P}}(k)=\operatorname{MSE}\left(\mathcal{M}_0(\widehat{U}_k),\widehat{C}_k\right), \tag*{(28)}
EG−O(k)=MSE⁡(M0(U^k),M0(Uk)),(29)E_{\mathrm{G-O}}(k)=\operatorname{MSE}\left(\mathcal{M}_0(\widehat{U}_k),\mathcal{M}_0(U_k)\right), \tag*{(29)}

where U^k\widehat{U}_k and UkU_k denote generated and true future observations, respectively, and the metrics are averaged over the T and R branches.

Adding node-level error feedback reduces generation–prediction node MSE from 0.004187 to 0.001604 and generation–observation node MSE from 0.010688 to 0.006570, corresponding to reductions of 61.69% and 38.53% (paired 95% intervals: [53.36%, 67.91%] and [30.34%, 46.14%], respectively). Figure 5(d,Figure 5e) shows that both errors decrease at all four prediction steps. Readback correction therefore improves both agreement with macro-level predictions and macro-level correspondence with true future observations.

Figure 5. Analysis of readback under the same quotient measurement and spatial feedback. (a–c) Reductions in image error, generation–prediction node error, and generation–observation node error for the complete model relative to four controls: no node-level error correction, global feedback, spatially shuffled feedback, and initial content targets. Positive values favor the complete model; horizontal intervals indicate paired 95% intervals. (d,e) Node MSE at each prediction step after motion correction and after adding node-level error correction. Curves and shaded regions show the mean and sample standard deviation across three random seeds. All settings use the same reference quotient measurement.

To examine the effect of error feedback locations, we replace the final feedback mapping with a spatially uniform assignment or shuffle its spatial positions, while keeping all other predictions, readback measurements, and correction conditions unchanged. Global feedback preserves the valid support and total node masses. Results for spatial shuffling are averaged over five predefined shuffles. Compared with these two controls, complete spatial feedback reduces generation–prediction node MSE by 8.28% and 7.79%, and generation–observation node MSE by 4.39% and 4.07%, respectively. ROI-MSE values for global, shuffled, and complete feedback are 0.018530, 0.018563, and 0.018609, respectively. Complete feedback has slightly higher pixel error, with its main advantage lying in macro-level alignment. These results indicate that preserving node–location correspondence helps translate macro-level errors into generation adjustments at the appropriate regions.

Compared with using the initial content measurement as the error decoder target, using the future observable target reduces generation–prediction node error by 1.20%. ROI-MSE changes by only 0.02%, with its paired interval including zero. Together, these results suggest that the current correction gains mainly arise from combining motion-guided reference transport with node-level error feedback, while correct spatial correspondence further improves consistency between macro-level predictions and generated observations.

Comparative Experiments

We evaluate action-conditioned ultrasound video prediction through short-window prediction and multi-step extrapolation, with prediction horizons H∈{4,8,16}H \in\{4, 8, 16\}. All methods use model weights trained at H=4H=4 and recursively generate future observations in blocks of four frames. The H=8H=8 and H=16H=16 settings correspond to extrapolation over two and four prediction blocks, respectively. Each model starts from four real historical frames. Each prediction block receives the corresponding four-step sequence of six-dimensional actions, and generated observations are used as history for the next block. The H=8H = 8 results are taken from the first eight steps of the same sixteen-step trajectory to preserve the same noise and prediction prefix.

The compared methods include PredRNN-Act [22, 23], PredRNN-V2-Act [23], SimVP [24], IRASim-S/4 [6], DINO-WM [5], MAU [25], and MMVP [26]. The PredRNN models use the action-conditioned extensions released by their authors. SimVP, MAU, and MMVP use implementations from OpenSTL [27]. Each baseline retains its original backbone, with separate interfaces adapting action inputs and observation representations. The PredRNN models, SimVP, IRASim, MAU, and MMVP share the same frozen domain-specific VAE. DINO-WM uses its native DINOv2 feature dynamics and official RGB decoder. During recursive prediction, QCC feeds back the corrected latent variables and recomputes USFM features, JEPA features, and historical optical flow from generated frames to construct quotient states for the next prediction block. DINO-WM feeds back predicted patch features, while the remaining methods feed back predicted latent variables. All baselines are trained for 30 epochs and evaluated using the raw final-epoch weights.

Based on recorded timestamps, the median temporal spans for H=4,8,16H = 4, 8, 16 are approximately 0.133 s, 0.267 s, and 0.533 s, respectively.

Four-Step Prediction

Table 2 reports four-step prediction results. QCC achieves the best mean values across all five metrics. Compared with SimVP, the baseline with the lowest ROI-MSE, QCC reduces ROI-MSE by 2.70% and increases PSNR by 0.862 dB. Compared with MMVP, the strongest baseline in perceptual quality and temporal prediction, QCC reduces LPIPS and temporal MSE by 47.84% and 13.06%, respectively. These results indicate improvements in pixel reconstruction, perceptual quality, and prediction of inter-frame changes within the short prediction window, with the largest relative reduction observed in perceptual distance.

MethodROI-MSE ↓\downarrow (×10−3)(\times10^{-3})PSNR ↑\uparrow (dB)SSIM ↑\uparrowLPIPS ↓\downarrowTemporal MSE ↓\downarrow (×10−3)(\times10^{-3})
PredRNN-Act [22]24.51±0.7624.51 \pm0.7623.22±0.1123.22 \pm0.110.7053±0.00090.7053 \pm0.00090.1530±0.00130.1530 \pm0.00136.92±0.096.92 \pm0.09
PredRNN-V2-Act [23]23.99±0.8123.99 \pm0.8123.30±0.1223.30 \pm0.120.7025±0.00170.7025 \pm0.00170.1515±0.00080.1515 \pm0.00086.85±0.096.85 \pm0.09
SimVP+Act [24]19.70±0.2419.70 \pm0.2424.14±0.0724.14 \pm0.070.7251±0.00220.7251 \pm0.00220.2110±0.00200.2110 \pm0.00207.35±0.087.35 \pm0.08
IRASim-S/4 [6]48.67±5.1948.67 \pm5.1920.28±0.3820.28 \pm0.380.6152±0.04290.6152 \pm0.04290.1677±0.03380.1677 \pm0.033828.43±8.2128.43 \pm8.21
DINO-WM+Dec. [5]37.84±2.3737.84 \pm2.3720.53±0.2820.53 \pm0.280.5977±0.00250.5977 \pm0.00250.2879±0.04170.2879 \pm0.041712.08±0.6112.08 \pm0.61
MAU+Act [25]23.54±0.2023.54 \pm0.2023.11±0.0923.11 \pm0.090.6964±0.00610.6964 \pm0.00610.2095±0.00370.2095 \pm0.00377.29±0.167.29 \pm0.16
MMVP+Act [26]22.98±0.2522.98 \pm0.2523.93±0.0423.93 \pm0.040.7294±0.00240.7294 \pm0.00240.1421±0.00700.1421 \pm0.00706.73±0.046.73 \pm0.04
QCC-WM18.61±0.0818.61 \pm0.0825.17±0.0225.17 \pm0.020.7492±0.00010.7492 \pm0.00010.0731±0.00020.0731 \pm0.00025.76±0.025.76 \pm0.02

Table 2. Cumulative prediction results at H=4H = 4.

Eight-Step Extrapolation

Table 3 reports results after recursively generating two prediction blocks. QCC maintains the best mean values in ROI-MSE, PSNR, LPIPS, and temporal MSE. Compared with the best baseline for each metric, QCC reduces ROI-MSE by 2.05% relative to SimVP, and reduces LPIPS and temporal MSE by 41.68% and 5.65%, respectively, relative to PredRNN-V2. QCC achieves an SSIM of 0.64475, close to MMVP at 0.64526. Reconstruction errors and perceptual distances increase for all methods compared with four-step prediction. QCC nevertheless retains a substantial LPIPS reduction beyond the training window, indicating a perceptual quality advantage in multi-step extrapolation.

MethodROI-MSE ↓\downarrow (×10−3)(\times10^{-3})PSNR ↑\uparrow (dB)SSIM ↑\uparrowLPIPS ↓\downarrowTemporal MSE ↓\downarrow (×10−3)(\times10^{-3})
PredRNN-Act [22]40.33±1.4440.33 \pm1.4421.04±0.1521.04 \pm0.150.6264±0.00180.6264 \pm0.00180.1958±0.00220.1958 \pm0.00227.51±0.067.51 \pm0.06
PredRNN-V2-Act [23]39.96±1.7339.96 \pm1.7321.09±0.1621.09 \pm0.160.6226±0.00330.6226 \pm0.00330.1913±0.00090.1913 \pm0.00097.43±0.067.43 \pm0.06
SimVP+Act [24]35.86±0.5835.86 \pm0.5821.68±0.0821.68 \pm0.080.6414±0.00180.6414 \pm0.00180.2542±0.00190.2542 \pm0.00198.27±0.098.27 \pm0.09
IRASim-S/4 [6]85.89±17.2085.89 \pm17.2017.97±0.6317.97 \pm0.630.5151±0.03540.5151 \pm0.03540.2147±0.02880.2147 \pm0.028828.45±7.4128.45 \pm7.41
DINO-WM+Dec. [5]50.65±3.5750.65 \pm3.5719.40±0.3019.40 \pm0.300.5601±0.00410.5601 \pm0.00410.2978±0.04030.2978 \pm0.040310.17±0.3610.17 \pm0.36
MAU+Act [25]40.11±0.8140.11 \pm0.8120.94±0.0720.94 \pm0.070.6199±0.00600.6199 \pm0.00600.2363±0.00480.2363 \pm0.00487.85±0.177.85 \pm0.17
MMVP+Act [26]38.10±0.3238.10 \pm0.3221.57±0.0821.57 \pm0.080.6453±0.00320.6453 \pm0.00320.2058±0.00870.2058 \pm0.00877.54±0.067.54 \pm0.06
QCC-WM35.12±0.3035.12 \pm0.3022.26±0.0522.26 \pm0.050.6448±0.00070.6448 \pm0.00070.1116±0.00030.1116 \pm0.00037.02±0.027.02 \pm0.02

Table 3. Cumulative prediction results at H=8H = 8.

Sixteen-Step Extrapolation

Table 4 reports results after recursively generating four prediction blocks. QCC retains the best mean values in ROI-MSE,

MethodROI-MSE ↓\downarrow (×10−3\times10^{-3})PSNR ↑\uparrow (dB)SSIM ↑\uparrowLPIPS ↓\downarrowTemporal MSE ↓\downarrow (×10−3\times10^{-3})
PredRNN-Act [22]58.05±2.2858.05 \pm2.2819.23±0.1819.23 \pm0.180.5553±0.00200.5553 \pm0.00200.2407±0.00370.2407 \pm0.00377.51±0.047.51 \pm0.04
PredRNN-V2-Act [23]58.17±2.1658.17 \pm2.1619.25±0.1619.25 \pm0.160.5506±0.00290.5506 \pm0.00290.2323±0.00150.2323 \pm0.00157.43±0.047.43 \pm0.04
SimVP+Act [24]56.79±1.3156.79 \pm1.3119.50±0.1019.50 \pm0.100.5633±0.00240.5633 \pm0.00240.2930±0.00250.2930 \pm0.00258.34±0.048.34 \pm0.04
IRASim-S/4 [6]147.50±50.71147.50 \pm50.7115.74±1.1615.74 \pm1.160.4200±0.02850.4200 \pm0.02850.2678±0.02330.2678 \pm0.023328.87±7.0628.87 \pm7.06
DINO-WM+Dec. [5]69.31±5.4469.31 \pm5.4418.10±0.3318.10 \pm0.330.5121±0.01010.5121 \pm0.01010.3134±0.03900.3134 \pm0.03908.82±0.218.82 \pm0.21
MAU+Act [25]64.12±2.8664.12 \pm2.8618.87±0.1518.87 \pm0.150.5423±0.00820.5423 \pm0.00820.2668±0.00600.2668 \pm0.00607.81±0.197.81 \pm0.19
MMVP+Act [26]56.50±1.5356.50 \pm1.5319.53±0.1319.53 \pm0.130.5669±0.0051\mathbf{0.5669} \pm0.00510.2715±0.00590.2715 \pm0.00597.61±0.087.61 \pm0.08
QCC-WM56.07±0.07\mathbf{56.07} \pm0.0719.86±0.02\mathbf{19.86} \pm0.020.5518±0.00100.5518 \pm0.00100.1621±0.0007\mathbf{0.1621} \pm0.00077.32±0.02\mathbf{7.32} \pm0.02

Table 4. Cumulative prediction results at H=16H = 16.

Note: All methods use the same 226 test starting points. Results are the mean ±\pm sample standard deviation across three training seeds. Both MSE metrics are reported in units of 10−310^{-3}; bold indicates the best mean. Act denotes an action-conditioning interface, and Dec. denotes the official RGB decoder.

Note: All methods use the same 226 test starting points. Results are the mean ±\pm sample standard deviation across three training seeds. Both MSE metrics are reported in units of 10−310^{-3}; bold indicates the best mean. Act denotes an action-conditioning interface, and Dec. denotes the official RGB decoder.

PSNR, LPIPS, and temporal MSE. Its LPIPS is 0.16209, a 30.23% reduction relative to the best baseline, PredRNN-V2. QCC achieves an ROI-MSE of 0.056069, which is 0.77% lower than MMVP at 0.056504, and a temporal MSE of 0.007315, which is 1.55% lower than PredRNN-V2. As the extrapolation horizon increases, QCC’s relative advantages in pixel and temporal errors decrease, while its advantage in perceptual distance remains substantial. For SSIM, MMVP achieves the best result at 0.56690. QCC achieves 0.55178, below MMVP, SimVP, and PredRNN-Act, indicating that these methods provide higher structural similarity at this horizon.

Across the three prediction horizons, QCC achieves the best mean values for all five metrics at four steps and maintains its lead in ROI-MSE, PSNR, LPIPS, and temporal MSE at eight and sixteen steps. Its LPIPS reductions relative to the best baseline at each horizon are 47.84%, 41.68%, and 30.23%, respectively, representing its most prominent performance advantage across prediction horizons.

As the prediction horizon increases from 4 to 16 steps, QCC’s cumulative ROI-MSE rises from 0.01861 to 0.05606, and LPIPS increases from 0.0731 to 0.1620, approximately 2.92 and 2.19 times their four-step values, respectively. PSNR decreases by 5.145 dB, and SSIM decreases by 0.19010. This trend reflects error accumulation during recursive prediction. Overall, QCC maintains favorable perceptual quality during multi-step extrapolation, while structural similarity at longer horizons remains an area for improvement.

Experiments on an Additional Dataset

To evaluate the transfer performance of QCC-WM on additional ultrasound data, we conduct supplementary experiments on the public Freehand US dataset [28, 29]. The dataset contains ultrasound scans of both forearms from 19 volunteers, with 12 scanning sequences per volunteer and 228 sequences in total. Acquisition covers straight, C-shaped, and S-shaped trajectories, with the probe oriented parallel or perpendicular to the forearm. The B-mode images have an original resolution of 480 × 640 and a sampling rate of approximately 20 Hz. The released data also include optically tracked poses and calibration information. We represent calibrated relative SE(3) transformations between adjacent frames using three-dimensional translation and three-dimensional axis-angle rotation, forming six-dimensional action conditions for predicting subsequent ultrasound images.

The training, validation, and test sets are split by subject and contain 13, 3, and 3 subjects, respectively. The training set provides 1437 four-step prediction windows. The validation set contains 207 windows with at least 16 consecutive future steps, and the test set contains 278 common prediction starting points. Images are resized to 224 × 224 and normalized to [−1, 1]. We retain the original temporal sampling and align images with poses using the official calibration information. All methods receive four historical frames and the corresponding action sequences and are trained with a supervision horizon of H=4H = 4. During evaluation, the same H=4H = 4 model weights are reused to recursively generate predictions at H=8H = 8 and H=16H = 16 in four-frame blocks, with generated observations fed back beNote: All methods use the same 226 test starting points. Results are the mean ±\pm sample standard deviation across three training seeds. Both MSE metrics are reported in units of 10−310^{-3}; bold indicates the best mean. Act denotes an action-conditioning interface, and Dec. denotes the official RGB decoder.

tween blocks. All three horizons are evaluated at the same 278 test starting points. Their median temporal spans are approximately 0.196, 0.391, and 0.798 s, respectively.

Table 5 reports the comparative results on Freehand US. QCC-WM achieves the best mean values across all five metrics at all three horizons. At H=4H = 4, QCC-WM reduces ROI-MSE by 33.09% and increases PSNR by 2.101 dB relative to SimVP-Act, the baseline with the lowest pixel error. It reduces LPIPS by 23.57% relative to PredRNN-Act, the baseline with the lowest perceptual distance. These results indicate that QCC-WM improves pixel reconstruction, structural similarity, and perceptual prediction quality in this transfer setting.

HorizonMethodROI-MSE ↓\downarrowPSNR/dB ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowTemporal MSE ↓\downarrow
H=4H = 4PredRNN-Act [22]4.382±\pm0.15029.775±\pm0.1590.8278±\pm0.00070.1448±\pm0.00141.553±\pm0.014
H=4H = 4PredRNN-V2-Act [23]4.524±\pm0.11829.642±\pm0.1070.8283±\pm0.00070.1475±\pm0.00081.561±\pm0.009
H=4H = 4SimVP-Act [24]4.056±\pm0.01530.048±\pm0.0150.8408±\pm0.00040.1933±\pm0.00081.663±\pm0.007
H=4H = 4IRASim-S/4 [6]20.237±\pm9.00624.285±\pm2.0100.6325±\pm0.04570.2130±\pm0.029214.124±\pm4.614
H=4H = 4DINO-WM [5]8.148±\pm0.50927.220±\pm0.2080.7527±\pm0.00100.2500±\pm0.00612.631±\pm0.069
H=4H = 4MAU-Act [25]4.203±\pm0.18229.922±\pm0.1870.8307±\pm0.00130.2048±\pm0.00101.581±\pm0.016
H=4H = 4MMVP-Act [26]4.139±\pm0.09330.039±\pm0.0990.8450±\pm0.00350.1469±\pm0.01131.522±\pm0.050
H=4H = 4QCC-WM2.714±\pm0.01432.149±\pm0.0210.8626±\pm0.00020.1107±\pm0.00091.132±\pm0.003
H=8H = 8PredRNN-Act [22]6.080±\pm0.09928.550±\pm0.0940.7896±\pm0.00040.1777±\pm0.00131.315±\pm0.005
H=8H = 8PredRNN-V2-Act [23]6.263±\pm0.24128.416±\pm0.1540.7900±\pm0.00030.1810±\pm0.00101.315±\pm0.004
H=8H = 8SimVP-Act [24]5.385±\pm0.03928.999±\pm0.0220.8106±\pm0.00030.2205±\pm0.00081.402±\pm0.004
H=8H = 8IRASim-S/4 [6]35.127±\pm19.70022.339±\pm2.4830.5707±\pm0.05770.2482±\pm0.032913.297±\pm4.581
H=8H = 8DINO-WM [5]10.352±\pm0.79226.312±\pm0.2520.7360±\pm0.00060.2571±\pm0.00631.894±\pm0.035
H=8H = 8MAU-Act [25]5.797±\pm0.30928.728±\pm0.2300.7979±\pm0.00220.2238±\pm0.00181.339±\pm0.007
H=8H = 8MMVP-Act [26]5.702±\pm0.14328.851±\pm0.1080.8099±\pm0.00360.1883±\pm0.01211.303±\pm0.037
H=8H = 8QCC-WM4.403±\pm0.00530.413±\pm0.0110.8251±\pm0.00040.1391±\pm0.00151.106±\pm0.001
H=16H = 16PredRNN-Act [22]9.424±\pm0.63226.934±\pm0.1680.7447±\pm0.00100.2182±\pm0.00091.196±\pm0.008
H=16H = 16PredRNN-V2-Act [23]9.533±\pm0.56926.841±\pm0.2290.7442±\pm0.00140.2206±\pm0.00131.193±\pm0.003
H=16H = 16SimVP-Act [24]8.260±\pm0.10227.458±\pm0.0390.7698±\pm0.00070.2500±\pm0.00121.276±\pm0.003
H=16H = 16IRASim-S/4 [6]69.113±\pm51.12620.035±\pm3.2090.4915±\pm0.07850.2999±\pm0.038713.449±\pm5.119
H=16H = 16DINO-WM [5]15.334±\pm1.45924.849±\pm0.3190.7081±\pm0.00210.2709±\pm0.00701.508±\pm0.021
H=16H = 16MAU-Act [25]9.480±\pm0.69126.950±\pm0.2910.7532±\pm0.00540.2483±\pm0.00451.223±\pm0.008
H=16H = 16MMVP-Act [26]8.550±\pm0.24927.348±\pm0.1290.7677±\pm0.00400.2344±\pm0.00961.194±\pm0.025
H=16H = 16QCC-WM7.775±\pm0.10228.349±\pm0.0240.7788±\pm0.00030.1852±\pm0.00311.095±\pm0.001

Table 5. Action-conditioned ultrasound video prediction on Freehand US. All methods use the same 278 test starting points. Results are the mean ±\pm sample standard deviation across three random seeds. At H=8,16H = 8, 16, models trained for four-step prediction are reused for longer-horizon extrapolation. ROI-MSE and temporal MSE are reported in units of 10−310^{-3}. Bold indicates the best mean at each prediction horizon.

As the prediction horizon increases, ROI-MSE generally rises and SSIM decreases across methods, reflecting error accumulation during recursive prediction. At H=8H = 8 and H=16H = 16, QCC-WM still reduces ROI-MSE by 18.24% and 5.88%, respectively, relative to the best baseline at each horizon. At H=16H = 16, QCC-WM achieves an SSIM of 0.7788, exceeding SimVP-Act at 0.7698, and reduces temporal MSE by 8.17% relative to PredRNN-V2-Act, the best baseline for this metric. Although the relative pixel-error advantage narrows as the extrapolation horizon increases, QCC-WM maintains favorable structural and temporal prediction performance on this additional dataset.

Conclusion

We presented QCC-WM, an action-conditioned ultrasound video prediction model that constructs macro-states with spatial correspondence through action-guided quotient mappings. By combining recursive state prediction, motion-guided reference transport, and readback correction under the same quotient measurement, QCC-WM connects macro-level dynamics with future observation generation. Comparative experiments on US-VLA and transfer experiments on Freehand US show improved ultrasound video prediction quality and a sustained perceptual advantage during recursive extrapolation beyond the training window. Ablation studies and mechanism analyses further show that motion-guided reference transport mainly improves pixel reconstruction, while node-level error correction further improves perceptual quality and consistency between generated observations and macro-level predictions. Future work will focus on improving structural preservation in long-horizon prediction and evaluating the framework for closed-loop ultrasound navigation.

References

  1. [1]B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, Y. Bengio, Toward causal representation learning, Proceedings of the IEEE 109 (5) (2021) 612–634. doi:10.1109/JPROC.2021.3058954. URL https://doi.org/10.1109/JPROC.2021.3058954
  2. [2]K. Chalupka, F. Eberhardt, P. Perona, Multi-level cause-effect systems, in: Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, Vol. 51 of Proceedings of Machine Learning Research, PMLR, 2016, pp. 361–369. URL https://proceedings.mlr.press/v51/chalupka16.htmlarxiv.org/abs/1512.07942
  3. [3]E. P. Hoel, L. Albantakis, G. Tononi, Quantifying causal emergence shows that macro can beat micro, Proceedings of the National Academy of Sciences 110 (49) (2013) 19790–19795. doi:10.1073/pnas.1314922110. URL https://doi.org/10.1073/pnas.1314922110
  4. [4]P. K. Rubenstein, S. Weichwald, S. Bongers, J. M. Mooij, D. Janzing, M. Grosse-Wentrup, B. Schölkopf, Causal consistency of structural equation models, in: Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence, 2017, pp. 151–160. URL https://proceedings.mlr.press/r15/rubenstein-17a.htmlarxiv.org/abs/1707.00819
  5. [5]G. Zhou, H. Pan, Y. LeCun, L. Pinto, DINO-WM: World models on pre-trained visual features enable zero-shotarxiv.org/abs/2411.04983

Paper details

vlnc:W_r7ucfyw5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents