WorksComputer Science
Householder Motion Embedding: Noise Modulation for Motion-Controllable Video Generation
Abstract
Diffusion models for video generation have achieved remarkable success, yet precise motion control under explicit user guidance remains challenging. Recent noise-warping methods introduce temporal correlations by warping the noise according to the user-provided motion signal. However, standard spatial warping generally disrupts the noise's i.i.d. Gaussian structure, requiring distribution correction, such as Jacobian-based correction, but at a substantial computational overhead. We propose Householder Motion Embedding (HME), which encodes the motion signal into the noise through a sequence of Householder reflections that together form an orthogonal transformation applied to each location's channel vector. HME exactly preserves the spatial Gaussian structure within each frame while introducing cross-frame correlations that promote temporally consistent video. HME is agnostic to the network architecture and integrates directly into standard video diffusion fine-tuning pipelines. Experiments on different scenarios show that HME matches or outperforms the state of the art, while achieving a 2.7x speedup over noise warping. These results demonstrate that channel-space Householder transformations offer a principled and efficient alternative to noise warping for motion-controllable video generation.
Figure 1. We propose Householder Motion Embedding (HME), a simple noise-modulation method for motion-controllable diffusion video generation. (a) Prior methods rely on spatial noise warping driven by motion fields. They preserve spatial Gaussianity only through costly correction steps. (b) HME instead applies per-location Householder transformations in channel space to the noise. It preserves spatial Gaussianity by construction, with a speed-up and better motion adherence. HME naturally supports (c1) motion transfer, (c2) local object motion control, and (c3) camera motion control.
Introduction
Diffusion based models have recently achieved remarkable success in video generation, demonstrating strong visual fidelity and semantic coherence across temporal dimension [59, 27, 6, 4, 19, 67, 49, 12]. However, generating temporally consistent videos under explicit motion controls (modeling complex camera motion or scene dynamics) remains a fundamental challenge. Existing approaches [64, 45, 24, 35] often suffer from temporal drift, geometric inconsistency, or limited controllability when motion deviates from patterns observed during training.
A common wisdom to inject motion cues into the neural network architecture is through feature modulation [65, 43] or motion conditioning tokens [14, 52, 20, 53], which operate downstream of the stochastic generation process and leave the noise motion-agnostic. An alternative is to modify the noise directly. Building on the observation by PYoCo [13] that shared noise components across frames improve temporal coherence, HIWYN [8], InfRes [10], and GWTF [7] demonstrate that constructing temporally correlated noise by transporting the a priori latent noise via diffeomorphic deformation fields parameterized by optical flow significantly enhances frame-to-frame consistency without modifying model architecture or training objectives. However, spatial resampling during this noise transport inherently destroys the i.i.d. Gaussian structure of the noise, since interpolation induces correlations across neighboring locations and alters the marginal distribution. This matters because the diffusion model is trained to denoise i.i.d. standard Gaussian noise: noise with spatial autocorrelation or altered variance falls outside this training distribution, which prior work has shown degrades generation quality [8, 10, 7]. Consequently, all existing noise-warping methods must introduce additional processing steps to restore spatial Gaussianity at the cost of non-trivial computational overhead.
In this paper, we propose Householder Motion Embedding (HME), a noise-modulation framework that resolves this spatial-Gaussianity bottleneck without sacrificing motion fidelity. Rather than warping noise via spatial deformation fields, we modulate the noise using the intended motion field by per-location orthogonal transformations in channel space, parameterized as the product of Householder reflections [17]. The Householder transformation, composed of a small number of reflections, yields expressive orthogonal operators at low computational cost, while the orthogonality ensures distribution preservation by construction.
Crucially, this distribution preservation guarantee is precisely what spatial noise warping methods lack. Modulating the noise through per-location Householder products therefore eliminates post-hoc correction entirely, requiring no Jacobian adjustments or distribution restoration steps, while inducing structured cross-frame temporal correlations that encode the geometric consistency of the underlying motion. Compared to deformation field based solutions, our method achieves exact spatial Gaussianity by construction, a speedup in noise processing time, and improved motion adherence on all three benchmarks.
Similar to existing noise warping methods, HME operates entirely at the noise initialization stage and requires no modifications to the denoising architecture, loss function, or training objective. The motion-embedded noise replaces the standard Gaussian noise at the start of the diffusion process, making the method plug-and-play across UNet-based and transformer-based video diffusion frameworks; we demonstrate it on the transformer-based CogVideoX. Our framework accepts any dense optical flow field as conditioning input regardless of its source: whether extracted from a reference video for motion transfer, rendered from a target camera trajectory for camera-controlled generation, or derived from user-specified drag interactions on the initial frame, thereby unifying diverse motion modalities within a single framework without motion-type-specific architectural modifications.
Contributions of this paper are as follows:
We propose a novel noise-modulation method for motion-controllable video diffusion that replaces spatial noise transport with per-location orthogonal transformations in latent channel space, parameterized as products of Householder reflections. Unlike existing noise-warping methods, this construction requires no post-hoc Gaussianity correction and operates in a single forward pass with no learnable parameters, making it a drop-in replacement for the noise initialization step in any video diffusion fine-tuning pipeline.
We derive the cross-frame covariance of the proposed noise, showing that the dependence between frames is an orthogonal transformation determined by the local motion, and analyse how the number of reflections controls the expressivity of this encoding.
We validate through extensive experiments on spatial Gaussianity preservation, motion-transfer and camera-controlled video generation benchmarks that HME matches or exceeds the state-of-the-art noise-warping baseline on nearly all motion-adherence and generation-quality metrics, while achieving a speedup in noise processing time.
Related Work
Motion-controllable video generation. Motion-controllable video generation methods broadly fall into three categories. Local object motion control methods [14, 24, 60] condition generation on user-specified trajectories via feature modulation or latent optimization, including Gaussian-filtered trajectory approaches [61, 53], two-stage sparse-to-dense motion field estimation [45, 37], region-wise trajectory and mask conditioning [65], and training-free latent optimization [36]. Global camera control methods [29, 54, 51] embed camera pose sequences into the denoising network through dedicated conditioning modules [20, 3, 52, 57, 55, 63] or learning Video-to-Video generation directly from imperfect 3D/4D renders [42, 18], which require large posed-video datasets for training. Motion transfer methods [1, 15, 28, 16, 40] propagate motion signals from a reference clip without additional supervision, either through temporal attention cloning [32], zero-shot feature guidance [60], inference-time schedule adaptation [46], motion disentanglement [43], or trajectory and optical flow conditioning [14, 30, 50]. Across all three paradigms, motion signals are injected downstream of the stochastic initialization, leaving the noise distribution motion-agnostic. Our method addresses this by modulating the noise through per-location orthogonal transformations, which preserve the i.i.d. Gaussian structure by construction while inducing cross-frame correlations that align per-frame noise distributions along the motion trajectory.
Noise Warping for Temporally Consistent Video Generation. Regardless of architectural difference between UNet-based and DiT-based video diffusion models, both are initialized from Gaussian noise whose correlation across frames can be influential to the temporal consistency of the generated output. PYoCo [13] first demonstrates that correlating the noise initialization across frames through a mixed noise prior substantially improves temporal coherence, motivating subsequent work on flow-guided noise warping. HIWYN [8] introduces integral noise, reinterpreting warped noise samples as integrals of a continuous noise field to preserve spatial Gaussianity during deformation field guided transport, at the cost of higher computational complexity. InfRes [10] resolves this by deriving an equivalent infinite-resolution formulation through Brownian bridge increments, reducing overhead by orders of magnitude while matching HIWYN’s theoretical guarantees. GWT [7] further improves efficiency with a linear-time warping algorithm and demonstrates motion-controllable fine-tuning of modern video diffusion models across object motion, camera control, and motion transfer. Despite these advances, all existing methods share a fundamental limitation: spatial resampling during deformation-field-based transport disrupts the i.i.d. Gaussian noise structure, necessitating costly post-hoc correction at higher resolutions. Our method resolves this bottleneck by replacing spatial transport with per-location orthogonal transformations in channel space that preserve spatial Gaussianity exactly by construction.
Our method makes use of Householder transformations, leveraging their distribution-preserving property. Householder transformations have previously been used for other deep learning problems in very different contexts, ranging from position encoding [58, 33] to parameter-efficient fine-tuning [2, 11, 62].
Method
Our goal is to construct motion-embedded noise that preserves spatial Gaussianity while inducing structured temporal correlations across video frames. We achieve this through per-location orthogonal transformations in latent channel space, parameterized as products of Householder reflections.
Householder Reflector. Householder reflectors are a standard, efficient numerical tool for constructing orthogonal transformations, requiring only an inner product and a vector update. Let denote the base model’s latent channel size. A Householder reflector is a simple orthogonal operator defined by a single non-zero vector :
Geometrically, reflects any vector across the hyperplane orthogonal to . As reflections are orthogonal transformations, they preserve Euclidean norms and, crucially, the Gaussian distribution: if , then for any choice of . Composing such reflections yields a more expressive orthogonal operator, and any orthogonal matrix admits an exact decomposition into at most reflections [17]. We exploit this property to construct orthogonal transformations that preserve spatial Gaussianity exactly.
As shown in Figure 2, our framework consists of three stages: (1) optical flow estimation, (2) a training-free Householder noise transformation module and (3) video diffusion model fine-tuning. The noise module generates temporally correlated noise during both training and inference, while the denoising architecture and training objective remain unchanged.

Figure 2. HME generates motion controlled videos in 3 stages: 1) optical flow estimation and spatial resampling, 2) temporally correlated noise construction via Householder Reflectors (see § 3.1 for details), 3) diffusion model fine-tuning and inference. At inference time, our method enables motion- and camera-controlled video generation given a text prompt, an initial frame, and a reference video as the motion signal.
Motion-Embedded Noise Construction
Problem Formulation and Notation
We distinguish between image space and latent noise space. In image space, a video consists of frames on the pixel grid, and optical flow assigns each pixel a displacement . The VAE encodes the video into latent frames on the latent grid and each latent location covers a pixel footprint (e.g., pixels for CogVideoX). In latent noise space, each carries a noise vector , drawn i.i.d. with . Operations in latent noise space act on the channel dimension only, without modifying spatial coordinates. In our framework, optical flow acts as a conditioning signal that parameterizes per-location orthogonal transformations of latent noise vectors, rather than spatially resampling them.
Flow-to-Latent Alignment
The optical flow between consecutive frames, , can be obtained with any off-the-shelf flow estimator ; we use RAFT [47] in our experiments. Since the diffusion model operates on a spatially down-sampled latent grid, we resample the estimated flow field to match the latent resolution:
where maps the flow field from the pixel grid to the latent grid by computing the weighted average displacement over the pixel footprint of each latent location, and denotes the flow normalized by the image resolution.
Optical Flow Conditioned Parameterization
We then convert each flow vector into transformations by applying fixed random linear projections:
The flow at each latent location is two-dimensional, whereas a Householder reflector acting on the latent noise requires a non-zero vector in ; the projections therefore lift each flow vector into distinct reflection directions in channel space, whose composition forms a orthogonal transformation. The projections and are randomly initialized once and frozen throughout training and inference, introducing no learnable parameters and yielding a deterministic, reproducible mapping from flow to transformation. The only requirement on these parameters is , since the Householder normalization is undefined for the zero vector; we use a small non-zero bias to avoid such degenerate projections. No orthogonality constraint on is needed, as orthogonality of the resulting operator is guaranteed by the Householder construction itself. Learning and is possible but unnecessary: their role is only to provide distinct deterministic embeddings of the flow into channel space. Because the projections are shared across spatial locations, the orthogonal transformation constructed below depends on only through the local flow . The projection parameters total only values, independent of spatial resolution.
Per-Location Orthogonal Noise Transformation
The motion-embedded noise is constructed sequentially: the noise at each frame is obtained by applying a flow-conditioned orthogonal transformation to the noise of the preceding frame. Formally, for each frame and latent location :
where frame 0 retains the unmodified Gaussian noise , and the reflection vectors are obtained from the resampled forward optical flow between frames and via Eq. (3).
Since each is a product of Householder reflections and therefore orthogonal, and the product of orthogonal matrices is orthogonal, the noise at frame is equivalently a single orthogonal transformation of the initial noise:
This accumulated operator is orthogonal by construction, which is the key property exploited in Section 3.2. The complete noise construction procedure is given in Algorithm 1.
Number of Reflections and Expressivity. Each Householder reflection is a rank-one perturbation of the identity, and the composed transformation is a full-rank orthogonal matrix for every . Expanding the product in Eq. (4) yields a series in the number of interacting reflections,
Algorithm 1 Motion-Embedded Noise via Householder Transformations
Require: Reference video , degradation level , number of reflections , fixed random projections
Ensure: Motion-embedded noise tensor
for all ;
Step 1: construct per frame noise
for do
a) Estimate flows and resample
▷ estimate and align flow to latent grid
b) apply orthogonal transformation to noise of previous frame
▷ per-location flow to orthogonal transformation
end for
Step 2: Stack and apply noise degradation
; , same shape as
return
where we omit the arguments of . Analogous to a Taylor series, the order- term captures interactions among flow-dependent reflection directions and, through Eq. (3), is a rational function of the flow of degree ; determines the highest order retained. Since each order- term is bounded in norm by and the random projections yield reflection directions with small pairwise inner products, higher-order terms are increasingly attenuated and the flow influences predominantly through the low-order terms. Adding reflections therefore yields diminishing gains in expressivity, while the cost grows linearly with , consistent with the saturation observed in Table 5.
| Method | Subject ↑ | Background ↑ | Temporal ↑ | Motion ↑ | FID ↓ | FVD ↓ | Flow EPE ↓ | CoTracker mIoU ↑ | PSNR ↑ | CLIP Image ↑ |
| Ours | 0.898 | 0.925 | 0.946 | 0.978 | 63.992 | 861.121 | 4.868 | 0.543 | 15.300 | 0.918 |
| Ours | 0.898 | 0.925 | 0.946 | 0.979 | 61.780 | 861.740 | 4.433 | 0.546 | 15.399 | 0.923 |
| Ours | 0.898 | 0.926 | 0.948 | 0.979 | 64.579 | 821.749 | 4.358 | 0.555 | 15.424 | 0.922 |
| Ours | 0.898 | 0.927 | 0.948 | 0.979 | 63.314 | 827.664 | 4.144 | 0.550 | 15.512 | 0.923 |
| Ours | 0.898 | 0.927 | 0.950 | 0.979 | 64.699 | 831.586 | 4.197 | 0.558 | 15.581 | 0.923 |
Table 5. Performance comparison across different numbers of Householder reflections (all with ), on DAVIS.
Theoretical Properties
The key theoretical guarantee of our noise construction is that the transformed noise remains exactly spatially Gaussian, regardless of the motion condition; a detailed proof and empirical results are given in Section E.1.
Proposition 1 (Exact Spatial Gaussianity). Let be i.i.d. across spatial locations, and let for , where each is a product of Householder reflections parameterized by the resampled optical flow . Then for all and all , and the vectors are mutually independent across latent locations .
Temporal Correlation Structure. While Proposition 1 concerns each frame in isolation, the sequential construction couples the frames. Using Eq. (5), the cross-frame covariance of the noise at a latent location is, for ,
while noise at different locations remains uncorrelated (Section E.2). (7) shows that the dependence between two frames is an orthogonal matrix determined entirely by the optical flow at between frames and : the noise vector at frame is a flow-dependent orthogonal transformation of the one at frame . Because the projections are shared across locations, latent locations undergoing the same motion receive the same cross-frame transformation, whereas locations with different motion receive different ones, so the spatial pattern of these transformations mirrors the motion field. The denoising model (the CogVideoX DiT) is trained with this structured noise in place of i.i.d. noise and learns to exploit it as an implicit motion-conditioning signal. A geometric comparison with spatial noise warping is given in § 5.
Fine-tuning with Motion-Embedded Noise
The key insight is that the motion conditioning is carried entirely by the cross-frame covariance structure of , not by its marginal distribution. Since each frame’s noise is marginally standard Gaussian (Proposition 1), the model trains with the standard diffusion objective without modification. The structured inter-frame dependencies, encoded by , act as an implicit motion signal that the denoiser learns to exploit during fine-tuning. More training details are provided in Section C.
Experiments
Implementation Details
We use the pretrained CogVideoX-5B-I2V as our base I2V model [59]. We fine-tune the base video diffusion model on 100K videos from RealCam-Vid [66], curated from multiple sources including RealEstate10K [68], MiraData [25], and DL3DV-10K [31], using optical flows estimated by RAFT [47] for training. All experiments are conducted at a spatial resolution of with 49 frames per video; the latent channel dimension is , determined by the CogVideoX VAE. We follow GWTF [7] to fine-tune the model for 30,000 iterations using a rank-2048 LoRA setup (learning rate , batch size 8), running on 4 NVIDIA A100 80GB GPUs for a total of 40 GPU-days.
Video Generation with Motion Control
Baselines and Evaluation Metrics. We evaluate HME on motion transfer, fine-grained motion control, object-level motion control and camera motion control for I2V generation. The performances on DAVIS [39], MC-Bench [65] are displayed in Table 1 and Table 2, respectively. Following GWTF, we adopt VBench [23] to assess the perceptual quality of generated video with related metrics such as Subject Consistency, Background Consistency, Temporal Flickering, and Motion Smoothness. To assess the motion adherence, we evaluate 1) CoTracker Distance and Background CoTracker distance for MC-Bench samples; 2) Optical Flow EPE, CoTracker mIoU, and PSNR against the reference videos for DAVIS samples. The input for all evaluations is a single reference image paired with a motion signal obtained from either the optical flow estimated from a reference video or derived from a user-dragged mask on the initial frame. We use the resulting forward optical flows between frames to construct the motion-embedded noise via Algorithm 1 (details in Section D).
| Method | Subject Consist. | BG Consist. | Temporal Flicker. | Motion Smooth. | FID | FVD () | Optical Flow (EPE) | CoTracker (mIoU) | PSNR (dB) | CLIP Image |
| MotionClone [32] | 0.754 | 0.862 | 0.874 | 0.928 | 133.852 | 1.771 | 26.451 | 0.373 | 10.032 | 0.814 |
| ImageConductor [30] | 0.791 | 0.891 | 0.913 | 0.939 | 128.087 | 1.830 | 20.382 | 0.372 | 12.117 | 0.811 |
| GWTF [7] | 0.888 | 0.920 | 0.942 | 0.978 | 68.385 | 0.987 | 5.644 | 0.464 | 14.748 | 0.912 |
| MotionPro [65] | 0.873 | 0.913 | 0.932 | 0.964 | 103.859 | 1.253 | 19.692 | 0.473 | 14.257 | 0.879 |
| DisMo [43] | 0.851 | 0.903 | 0.920 | 0.956 | 85.201 | 1.156 | 11.189 | 0.428 | 11.475 | 0.882 |
| TTM [46] | 0.869 | 0.916 | 0.935 | 0.970 | 70.658 | 1.093 | 13.871 | 0.412 | 13.942 | 0.901 |
| Ours | 0.896 | 0.924 | 0.951 | 0.979 | 65.343 | 0.830 | 4.816 | 0.540 | 15.662 | 0.920 |
| Ours | 0.898 | 0.925 | 0.946 | 0.979 | 61.780 | 0.862 | 4.433 | 0.546 | 15.399 | 0.923 |
| Ours | 0.898 | 0.926 | 0.945 | 0.979 | 62.799 | 0.855 | 4.486 | 0.548 | 15.162 | 0.923 |
Table 1. Quantitative comparison of motion controlled I2V generation on DAVIS. Best performance in Bold, best baseline performance in Underline.
| Method | Subject Cons. | Background Cons. | Motion Smooth. | Aesthetic Qual. | Imaging Qual. | CTD | Bg-Obj CTD |
| MotionPro [65] | 0.980 | 0.976 | 0.995 | 0.586 | 0.610 | 8.019 | 23.30 |
| TTM [46] | 0.984 | 0.981 | 0.995 | 0.601 | 0.645 | 8.432 | 36.32 |
| DisMo [43] | 0.985 | 0.973 | 0.991 | 0.608 | 0.601 | 8.954 | 36.16 |
| Noise-based methods | |||||||
| GWTF [7] | 0.975 | 0.974 | 0.993 | 0.589 | 0.645 | 15.747 | 42.44 |
| Ours | 0.976 | 0.977 | 0.995 | 0.594 | 0.645 | 13.879 | 39.53 |
Table 2. Quantitative comparison of Fine-grained and Object-level motion control on MC-Bench.
Results on Motion Transfer. We evaluate motion transfer I2V on 45 DAVIS videos, using the motion of each original video as reference. As shown in Table 1, we compare against six baselines: training-free methods (MotionClone [32], TTM [46]), noise-warping (GWTF), and supervised controllers (ImageConductor [30], MotionPro [65], DisMo [43]). Each of our three variants () outperforms all six baselines on all ten metrics, despite being fine-tuned on 100K videos, compared with approximately 4M for GWTF. It attains the highest scores on all four VBench metrics, as well as the best FID and FVD. At , Optical Flow EPE decreases from 5.644 to 4.486, CoTracker mIoU increases from 0.464 to 0.548, and PSNR improves from 14.748 to 15.162 dB. These gains require no post-hoc Gaussianity correction, indicating that motion embedding via orthogonal transformations is a more efficient alternative to spatial noise warping. We further evaluate across varying noise degradation levels, where all variants consistently outperform GWTF and other baselines.
Results on Fine-grained and Object-level Control. We evaluate object-level and fine-grained motion control on 965 MC-Bench samples against MotionPro, TTM, DisMo, and GWTF. Our approach outperforms GWTF on five metrics, achieving lower CTD and better Background Consistency and Motion Smoothness, indicating that Householder motion embedding captures motion priors more faithfully than spatial warping. The CTD gap with supervised methods reflects a trade-off between controllability and generative freedom: TTM, MotionPro, and DisMo enforce strict trajectory adherence via explicit supervision but degrade subject-background disentanglement, evidenced by lower Bg-Obj CTD (23.30, 36.32, 36.16) relative to our method (39.53) and GWTF (42.44). Our method also matches the best Motion Smoothness (0.995) without architectural changes, showing that HME is competitive with supervised motion control while better preserving subject–background disentanglement.
Results on Camera Motion Control. We evaluate HME on camera controlled I2V generation under pre-defined camera trajectories using 200 randomly sampled scenes from the DL3DV subset of the RealCam-Vid validation set. Frame-level alignment is assessed via PSNR, LPIPS, and SSIM, and camera motion consistency via Normalized Absolute Trajectory Error (ATE). As shown in Table 3, our method achieves the best performance across all seven metrics, outperforming noise-warping and other baselines. The leading ATE score confirms accurate replication of the target camera trajectory, while the superior PSNR, LPIPS, and SSIM scores demonstrate that our method maintains high visual fidelity alongside strong motion adherence, reflecting a great balance between appearance quality and temporal consistency under explicit camera control.
Table 3. Quantitative comparison of DL3DV camera motion control.
Qualitative Results. In Figure 3, we present a qualitative comparison between our method and all competing baselines on (a) motion transfer, (b) fine-grained, and (c) object-level motion controlled video generation. For (a), the red box highlights that our method more faithfully replicates the detailed motion of the bear across frames, accurately capturing the displacement of the subject in accordance with the reference motion signal, whereas competing methods fail to preserve fine motions. For (b), our method not only follows the prescribed motion signal faithfully but also maintains high visual fidelity, producing coherent details throughout the sequence, while other baselines exhibit visible artifacts. For (c), our method demonstrates precise spatial awareness by correctly animating the target ball along the intended trajectory while preserving the relative positioning between objects, whereas competing methods either lose track of the motion signal or produce spatially inconsistent object placements across frames.

Figure 3. Qualitative comparison of (a) motion-transfer, (b) fine-grained motion control, and (c) local object motion control I2V generation, evaluated on DAVIS and MC-Bench. We present the intermediate frame and last frame of generated videos by each method. Our method demonstrates strong appearance consistency and high generation fidelity while faithfully adhering to the provided motion control signals. Best viewed zoomed in; videos are provided in the Supplementary Material.
We further show camera-motion-controlled I2V results in Figure 4. Compared with the strongest baseline, TTM, our generated videos yield estimated camera trajectories closer to the target trajectory, indicating stronger camera-motion adherence while better preserving viewpoint changes, scene structure, and appearance consistency. Additional results are provided in Section F.

Figure 4. Qualitative Comparison of camera motion controlled I2V generation, evaluated on DL3DV. We present the intermediate frames and last frames of generated videos by each method. Best viewed zoomed in; videos are provided in the Supplementary Material.
Robustness to the Flow Estimator. We replace RAFT with UniMatch [56] at evaluation time only, while our model remains trained on RAFT flows; GWTF and our method receive identical UniMatch flows. Our method matches or outperforms GWTF on all ten metrics (Table 4), reducing Flow EPE from 5.857 to 4.707.
| Method | Subject Consist. ↑ | BG Consist. ↑ | Temporal Flicker. ↑ | Motion Smooth. ↑ | FVD FID ↓ | Optical Flow (EPE) ↓ | CoTracker (mIoU) ↑ | PSNR (dB) ↑ | CLIP Image ↑ | |
| GWTF [7] | 0.882 | 0.918 | 0.945 | 0.979 | 73.348 | 1.180 | 5.857 | 0.472 | 14.437 | 0.901 |
| Ours | 0.895 | 0.926 | 0.948 | 0.979 | 63.619 | 0.837 | 4.707 | 0.541 | 15.418 | 0.915 |
Table 4. Robustness to the flow estimator: DAVIS results with UniMatch [56] flows at evaluation time.
Ablation and Analysis
Choice of the Number of Householder Reflections. We ablate the number of Householder reflections in Table 5. Motion adherence is weakest at (Flow EPE 4.868) and improves with diminishing returns as grows (Flow EPE 4.433 4.144 from to ; CoTracker mIoU 0.558 and PSNR 15.581 dB at ), consistent with the attenuation of higher-order terms in Eq. (6), while runtime grows approximately linearly, from 2.27 ms/latent at to 14.78 ms/latent at (Table 6). At , our method runs at 2.80 ms/latent, a 2.7× speedup over GWTF and 13.6× over HIWYN. It is also memory-light: only projection values are stored, shared across frames and locations, and neither dense per-location matrices nor noise upsampling are needed, so the peak additional memory is 0.5–6 MB, versus 76 MB for GWTF, 285 MB for InfRes and 49,384 MB for HIWYN. We use as the default: it achieves the best FID with only a small latency increase over , while larger offers slightly stronger motion adherence at higher cost.
Table 6. Computational overhead of noise-based methods: GPU time and approximate peak additional GPU memory on an NVIDIA RTX PRO 6000 Blackwell GPU ( resolution, ; HIWYN with ).
Choice of Noise Degradation Level. The degradation parameter trades motion adherence against generative freedom. All three variants, at , outperform GWTF on all ten metrics (Table 1); lower strengthens motion adherence, while higher favours perceptual quality and subject consistency, with small differences overall. We adopt for all other experiments; qualitative comparisons are in Section G.
Discussion
Geometric Analysis of Temporal Noise Correlation. PYoCo [13] shows that correlating noise across frames promotes temporal coherence. GWTF [7] realises this by transporting noise along flow trajectories across locations, whereas our noise stays tied to its location: preserves its norm, and its change between frames, , depends only on the local flow. Since the projections are shared across locations, regions with the same motion receive the same transformation; this includes static regions, since for zero flow when . We hypothesise that this per-location encoding of motion in channel space provides a more direct conditioning signal than spatial transport, contributing to the improved motion adherence over GWTF.
Limitations. Despite its effectiveness, our method has several limitations. The choice of the number of reflections introduces an expressivity-efficiency trade-off and may require tuning for different settings. The method also relies on accurate optical flow for training, so noisy motion estimates can weaken motion adherence. Relatedly, since the motion signal is carried by per-location flow, subjects undergoing complex non-rigid geometric transformations (e.g., the rapidly rotating snowboarder in DAVIS) remain challenging, which we leave for future work. Moreover, while the noise construction is training-free and parameter-free, the video diffusion model still requires fine-tuning to fully exploit the motion-embedded noise. Finally, the baselines are evaluated with their released checkpoints rather than retrained on our training data and RAFT flows.
Conclusion
We presented Householder Motion Embedding (HME), a noise-modulation framework for motion-controllable image-to-video generation. By replacing spatial noise transport with per-location Householder orthogonal transformations in latent channel space, our method preserves spatial Gaussianity by construction while encoding motion-aligned temporal correlations. Experiments on motion transfer, object-level control, and camera motion control show that HME improves motion adherence, visual fidelity, and noise-processing efficiency over existing noise-warping methods. Beyond noise construction, we believe that geometric transformation-based conditioning offers a promising way for incorporating structured motion priors into video diffusion models.
Acknowledgments and Disclosure of Funding
This work was conducted in collaboration with Vertex Lab Technology Limited. This research is funded in part by an Australian Research Council (ARC) Discovery Grant DP220100800. Hongdong Li holds concurrent appointments with The Australian National University (full-time) and with Amazon as an Amazon Scholar (part-time).
Other Related Work
Image-to-Video Diffusion Models. Diffusion-based video generation has progressed rapidly from early UNet-based architectures [21, 5, 4, 19] that append temporal attention layers to pretrained image diffusion backbones, to large-scale Diffusion Transformer (DiT) models [38] that jointly model spatial and temporal dependencies through full spatio-temporal attention [59, 27, 49, 6]. Image-to-video (I2V) models such as Stable Video Diffusion [4] and CogVideoX [59] extend this capability by conditioning generation on a reference frame, enabling content-consistent animation from a single image. Despite their architectural differences, both UNet-based and DiT-based video diffusion models are initialized from Gaussian noise whose correlation across frames can be influential to the temporal consistency of the generated output [13]. Our method exploits this property by modulating the noise with optical flow to induce structured temporal correlations before denoising begins, without modifying the denoising architecture.
Application of Householder Reflections in Deep Learning. Householder reflections provide a numerically stable and efficient parameterization of orthogonal operators, with a growing range of applications in deep learning. Early work uses them to enforce orthogonality in RNN transition matrices as a remedy for vanishing and exploding gradients [34]. In generative modeling, products of Householder reflections serve as volume-preserving transformations in normalizing flows for VAEs [48, 22]. More recently, they have been adopted for parameter-efficient fine-tuning, where orthogonal weight updates preserve hyper-spherical energy during adaptation [41, 62, 2, 11], and for positional encoding, where data-dependent Householder products replace static RoPE rotations to improve positional expressiveness [58, 33]. In all of these settings, Householder products operate on weight matrices, transition operators, or attention mechanisms. Our work applies them to the diffusion noise, where orthogonality guarantees that each frame’s noise remains exactly i.i.d. Gaussian after noise modulation.
Preliminaries
Latent Video Diffusion. We build on CogVideoX [59], which encodes a video into a latent with a 3D VAE and learns to reverse the forward process , , where . A DiT denoiser , conditioned on the first frame and the text prompt , is trained with the -prediction objective
and sampling starts from . Noise-based motion-control methods, including noise warping and ours, replace by structured noise both in the training objective and as the initial noise at inference, leaving the architecture and loss unchanged.
Fine-tuning with Motion-Embedded Noise
Training Procedure. For each training video clip, we estimate pairwise forward optical flows with RAFT [47], construct via Algorithm 1, and replace the standard i.i.d. initialization with . The denoising objective and all other aspects of the training pipeline remain unchanged. At inference, the same noise construction is applied from a reference motion source, which can be obtained from a reference video, a predefined camera trajectory, or user-specified drag interactions on the initial frame.
Noise Degradation. Following GWTF [7], we interpolate between fully structured and fully unstructured noise via a degradation parameter :
where has the same shape as , retains full motion conditioning and approaches standard Gaussian noise. Since is drawn independently of and the denominator normalizes the variance, every frame of remains exactly i.i.d. across spatial locations and channels for any , so Proposition 1 carries over to the noise the model actually receives. During training, is sampled uniformly from ; at inference it is a user-controlled parameter governing the strength of motion adherence.
Evaluation Protocol
DAVIS (motion transfer). We estimate forward optical flows on each reference video with RAFT [47] and use them to construct . Motion adherence is measured by comparing RAFT flows of the generated video against those of the reference video (Flow EPE), together with CoTracker [26] mIoU and PSNR.
MC-Bench (fine-grained and object-level control). For each sample, we render a motion video from the provided mask and drag trajectory, estimate its flow with RAFT, and restrict the flow to the masked region.
DL3DV (camera control). Following TTM [46], we convert each camera trajectory into the corresponding optical flow. For ATE, camera poses are estimated with COLMAP [44] on both the generated and the reference video, and the normalized error is computed between the two trajectories.
Baseline inputs. All flow-conditioned methods (GWTF and ours) receive identical input flows, while the remaining baselines are conditioned on the same reference videos.
Extended Details of Spatial Gaussianity Preservation
Proof of Proposition 1
To prove Proposition 1, we establish its two claims using the unrolled form , where .
Proof.
Marginal Gaussianity. Each is a product of Householder reflections, each of which is orthogonal by construction. Since a finite product of orthogonal matrices is orthogonal, for all , and hence .
For any orthogonal and :
Applying this to yields for all and . For odd , ; this does not affect Proposition 1, since any orthogonal matrix preserves , and the results in Table 5 confirm that odd works in practice.
Spatial independence. The transformation at each location acts solely on without coupling to neighboring locations. For any two distinct locations :
where follows from the spatial independence of the initial noise. Since the vectors are jointly Gaussian (each is a linear function of the initial noise) with zero cross-location covariance, they are mutually independent.
Cross-Frame Covariance Derivation
We derive the full covariance structure of the motion-embedded noise. For frames , and latent locations :
where is the Kronecker delta, following from the zero mean and spatial independence of . Noise at distinct locations is therefore uncorrelated across all frames, and at the same location the covariance reduces to Eq. (7); for ,
an orthogonal matrix determined only by the flows between frames and at . Its Frobenius norm is for all , and , so the strength of the cross-frame dependence is reflected not in its norm but in its alignment with the identity: the average channel-wise correlation equals if and only if .
Experimental Results
Metrics and Baselines. To evaluate the Gaussianity preservation ability of our method, we follow InfRes [10] and GWTF [7] to present Moran’s I and Kolmogorov-Smirnov (K-S) test results. Moran’s I measures spatial autocorrelation of warped noise and the K-S test assesses the distributional fidelity of the warped noise against a standard normal distribution. We further report noise Std., skewness and excess kurtosis to characterize higher-order distributional properties such as the symmetry and shape of the resulting distribution compared to normal distribution. We also evaluate the noise processing efficiency of all methods using GPU time. Baselines mainly include optical flow driven warping methods (CaV, HIWYN [8], GWTF [7], InfRes [10]) designed for Gaussianity preservation. These benchmarks operate on the noise tensor alone and run at , , on an NVIDIA RTX PRO 6000 Blackwell GPU; this setting only measures noise properties, computational overhead and memory, and is independent of the video generation experiments in § 4.
Performance Analysis. As shown in Table 7, HME is successful in generating spatially Gaussian noise with Moran’s I index values close to 0, with corresponding p-values ranging from 0.466 to 0.534, indicating no significant spatial autocorrelation, an important property of i.i.d. noise. Small K-S index values and p-values above 0.05 indicate the spatial distribution of resulting noise matches a standard normal distribution. HME obtains a K-S statistic of and p-value of on average, demonstrating the distributional fit to a standard normal distribution. Furthermore, our method achieves near 1 Std., negligible Skewness and Excess Kurtosis, suggesting substantial normality. In contrast, all interpolation-based operations introduce significant spatial autocorrelation (Moran’s I p-values of 0.008–0.020); bilinear and bicubic interpolation additionally fail the K-S test, while nearest-neighbor sampling preserves the marginal distribution (K-S p-value = 0.115) but not the spatial independence of the noise.
| Method | Moran’s I (≈ 0) | Moran’s I p-value↑ | K-S Test (index) ↓ | K-S Test p-value ↑ | Std. (≈ 1.0) | Skewness (≈ 0) | Ex. Kurtosis (≈ 0) | GPU Time ms/Latent↓ |
| Warping + Interpolation Operation | ||||||||
| Nearest | 0.1150 | 0.012 | 0.0107 | 0.115 | 1.0015 | 0.0016 | -0.0116 | - |
| Bilinear | 0.6218 | 0.008 | 0.3318 | 0.000 | 0.2179 | 0.0720 | 0.4753 | - |
| Bicubic | -0.0671 | 0.020 | 0.0670 | 0.000 | 1.0489 | 0.0075 | 0.2194 | - |
| PYoCo [13] | 0.0002 | 0.478 | 0.0033 | 0.512 | 1.000 | 0.0005 | 0.0000 | - |
| Optical Flow Driven Noise Warping Methods | ||||||||
| CaV [9] | 0.6170 | 0.004 | 0.3379 | 0.000 | 0.2060 | 0.0590 | 4.065 | - |
| HIWYN [8] | 0.0001 | 0.502 | 0.0037 | 0.422 | 0.9957 | 0.0007 | -0.0001 | 38.00 |
| GWTF [7] | 0.0002 | 0.507 | 0.0035 | 0.494 | 0.9997 | -0.0006 | 0.0037 | 7.63 |
| InfRes [10] | 0.0003 | 0.512 | 0.0034 | 0.496 | 1.0001 | 0.0004 | 0.0011 | 23.94 |
| Ours (HME) | ||||||||
| 0.0003 | 0.482 | 0.0032 | 0.535 | 0.9998 | -0.0006 | 0.0014 | 2.80 | |
| 0.0003 | 0.475 | 0.0032 | 0.531 | 0.9999 | -0.0005 | 0.0014 | 5.18 | |
| 0.0003 | 0.466 | 0.0033 | 0.527 | 1.0001 | -0.0007 | 0.0015 | 7.59 | |
| 0.0003 | 0.493 | 0.0033 | 0.528 | 0.9998 | -0.0004 | 0.0011 | 10.01 | |
| 0.0002 | 0.534 | 0.0032 | 0.546 | 1.0001 | -0.0005 | 0.0013 | 14.78 |
Table 7. Comparison of Gaussianity preservation and computational efficiency between methods that produce temporally correlated noise ( resolution, ) driven by optical flow; statistics for our method are computed on (i.e., ), and all GPU times are measured on an NVIDIA RTX PRO 6000 Blackwell GPU.
Meanwhile, dedicated flow-based methods also recover near-zero Moran’s I, acceptable K-S p-values, and negligible skewness and excess kurtosis, but at runtimes of 7.63–38.00 ms/latent. Our method reduces the GPU runtime to 2.80 ms/latent at , a speedup over GWTF and over HIWYN. All our variants ( to 12) satisfy the Gaussianity criteria with runtime scaling gracefully from 2.80 to 14.78 ms/latent.
Extra Qualitative Comparisons
Video versions of the comparisons in this paper, along with additional results, are provided in the Supplementary Material; differences between methods are easier to spot in the videos.
Motion Transfer
For clarity, we show qualitative motion-transfer comparisons with GWTF, DisMo and MotionPro in Figure 5; quantitative comparisons with all baselines, including TTM, are given in Table 1.

Figure 5. Motion transfer example: we demonstrate strong motion adherence to the reference video while maintaining high generation quality and fidelity. Prompt used: "Cyberpunk rooftop at night, wet reflective ground, neon city skyline, three humanoid robots standing slightly left of center in a triangular group pose, aligned like three girls in an open field. White-blue robot, black-orange robot, white-red-green robot. Wide composition, large empty foreground and right-side space, glowing Japanese signs, holographic billboards, photorealistic, cinematic, ultra detailed."
Qualitative Comparisons From MC-Bench
Following the experiment described in § 4.2, we present additional results beyond those shown in Figure 3, further illustrating our method’s performance against leading approaches on the MC-Bench dataset.

Figure 6. Example from MC-Bench: we highlight the disentanglement between the moving subject (masked planet) and the background in our method.

Figure 7. Example from MC-Bench: we highlight the disentanglement between the moving astronaut and the background in our method.
Qualitative Comparisons From DAVIS
Following the experiment described in § 4.2, we present additional results beyond those shown in Figure 3, further illustrating our method’s performance against leading approaches on the DAVIS dataset.

Figure 8. Example from DAVIS: our method shows better motion adherence.

Figure 9. Example from DAVIS: our method shows better motion adherence.
Qualitative Comparisons From DL3DV
Following the experiment described in § 4.2, we present additional camera-control results beyond those shown in Figure 4.

Figure 10. Example from DL3DV: our method shows better camera motion adherence under optical flow control.

Figure 11. Example from DL3DV: our method shows better camera motion adherence under optical flow control.
Qualitative Results for Different Noise Degradation Levels
We qualitatively ablate the noise degradation level , as analyzed in § 4.3.

Figure 12. The effect of noise degradation level on generated videos. As degradation decreases, the video adheres more to the input flow.
References
- [1]Aira, L. S., Montanaro, A., Aiello, E., Valsesia, D., & Magli, E. (2024). Motioncraft: Physics-based zero-shot video generation. NeurIPS.
- [2]Alejandro Moreno Arcas, Albert Sanchis, Jorge Civera, and Alfons Juan. Hoft: Householder orthogonal fine-tuning. arXiv preprint arXiv:2505.16531, 2025.arxiv.org/abs/2505.16531
- [3]Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin-Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, David B. Lindell, and Sergey Tulyakov. VD3D: Taming large video diffusion transformers for 3D camera control. In ICLR, 2025. URL https://openreview.net/forum?id=0n4bS0R5MM.openreview.net/forum?id=0n4bS0R5MM
- [4]Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., & al., . et . (2023). Stable video diffusion: Scaling latent video diffusion models to large datasets. In arXiv preprint.arXiv
- [5]Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models. In CVPR, 2023.arxiv.org/abs/2304.08818
- [6]Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., Ng, C., Wang, R., & Ramesh, A. (2024). Video generation models as world simulators. In OpenAI technical report. <a href="https://doi.org/https://openai.com/research/video-generation-models-as-world-simulators" rel="noopener noreferrer" target="_blank">https://doi.org/https://openai.com/research/video-generation-models-as-world-simulators</a>
- [7]Burgert, R., Xu, Y., Xian, W., Pilarski, O., Clausen, P., He, M., Ma, L., Deng, Y., Li, L., Mousavi, M., & al., . et . (2025). Go-with-the-flow: Motion-controllable video diffusion models using real-time warped noise. CVPR, 13–23.
- [8]Pascal Chang, Jingwei Tang, Markus Gross, and Vinicius C Azevedo. How i warped your noise: a temporally-correlated noise prior for diffusion models. In ICLR. OpenReview, 2024.arxiv.org/abs/2504.03072
- [9]Weifeng Chen, Yatai Ji, Jie Wu, Hefeng Wu, Pan Xie, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-A-Video: Controllable Text-to-Video Diffusion Models with Motion Prior and Reward Feedback Learning. arXiv preprint arXiv:2305.13840, 2023.arxiv.org/abs/2305.13840
- [10]Yitong Deng, Winnie Lin, Lingxiao Li, Dmitriy Smirnov, Ryan D Burgert, Ning Yu, Vincent Dedun, and Mohammad H. Taghavi. Infinite-resolution integral noise warping for diffusion models. In ICLR, 2025. URL https://openreview.net/forum?id=Y6LPWBo2HP.openreview.net/forum?id=Y6LPWBo2HP
- [11]Wei Dong, Yuan Sun, Yiting Yang, Xing Zhang, Zhijun Lin, Qingsen Yan, Haokui Zhang, Peng Wang, Yang Yang, and Hengtao Shen. Efficient adaptation of pre-trained vision transformer via householder transformation. NeurIPS, 37:102056–102077, 2024.arxiv.org/abs/2410.22952
- [12]Gao, Y., Guo, H., Hoang, T., Huang, W., Jiang, L., Kong, F., Li, H., Li, J., Li, L., Li, X., & al., . et . (2025). Seedance 1.0: Exploring the boundaries of video generation models. In arXiv preprint.arXiv
- [13]Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming-Yu Liu, and Yogesh Balaji. Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models. In ICCV, 2023.arxiv.org/abs/2305.10474
- [14]Geng, D., Herrmann, C., Hur, J., Cole, F., Zhang, S., Pfaff, T., Lopez-Guevara, T., Aytar, Y., Rubinstein, M., Sun, C., & al., . et . (2025). Motion prompting: Controlling video generation with motion trajectories. CVPR, 1–12.
- [15]Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. In ICLR, 2024. URL https://arxiv.org/abs/2307.10373.
- [16]Ahmet Berke Gökmen, Yigit Ekin, Bahri Batuhan Bilecen, and Aysegul Dundar. Ropecraft: Training-free motion transfer with trajectory-guided rope optimization on diffusion transformers. In NeurIPS, 2025.arxiv.org/abs/2505.13344
- [17]Golub, G. H., & Van Loan, C. F. (2013). Matrix computations. In JHU press.
- [18]Gu, Z., & al., . et . (2025). Diffusion as shader. SIGGRAPH.
- [19]Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In ICLR, 2024.arxiv.org/abs/2307.04725
- [20]Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. CameraCtrl: Enabling camera control for video diffusion models. In ICLR, 2025.arxiv.org/abs/2503.10592
- [21]Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., & Fleet, D. J. (2022). Video Diffusion Models. NeurIPS.
- [22]Emiel Hoogeboom, Rianne van den Berg, and Max Welling. Emerging convolutions for generative normalizing flows. In ICML, pages 2771–2780. PMLR, 2019.arxiv.org/abs/1901.11137
- [23]Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., & al., . et . (2024). Vbench: Comprehensive benchmark suite for video generative models. CVPR.
- [24]Yash Jain, Anshul Nasery, Vibhav Vineet, and Harkirat Behl. Peekaboo: Interactive video generation via masked-diffusion. In CVPR, 2024.arxiv.org/abs/2312.07509
- [25]Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A large-scale video dataset with long durations and structured captions. NeurIPS, 37:48955–48970, 2024.arxiv.org/abs/2407.06358
- [26]Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. In ECCV, 2024. URL https://arxiv.org/abs/2307.07635.
- [27]Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., Xiong, J., Li, X., Wu, B., Zhang, J., & al., . et . (2024). Hunyuanvideo: A systematic framework for large video generative models. In arXiv preprint arXiv:2412.02603. 2412.02603
- [28]Max Ku, Cong Wei, Weiming Ren, Huan Yang, and Wenhu Chen. AnyV2V: A tuning-free framework for any video-to-video editing tasks. TMLR, 2024.arxiv.org/abs/2403.14468
- [29]Zhengfei Kuang, Shengqu Cai, Hao He, Yinghao Xu, Hongsheng Li, Leonidas Guibas, and Gordon Wetzstein. Collaborative video diffusion: Consistent multi-video generation with camera control. In NeurIPS, 2024.arxiv.org/abs/2405.17414
- [30]Yaowei Li, Xintao Wang, Zhaoyang Zhang, Zhouxia Wang, Ziyang Yuan, Jiangbin Xie, Ying Shan, and Yuexian Zou. Image conductor: Precision control for interactive video synthesis. In AAAI, volume 39, pages 5031–5038, 2025.arxiv.org/abs/2406.15339
- [31]Ling, L., Sheng, Y., Tu, Z., Zhao, W., Xin, C., Wan, K., Yu, L., Guo, Q., Yu, Z., Lu, Y., & al., . et . (2024). DI3dv-10k: A large-scale scene dataset for deep learning-based 3d vision. CVPR, 22160–22169.
- [32]Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, and Yi Jin. MotionClone: Training-free motion cloning for controllable video generation. In ICLR, 2025. URL https://openreview.net/forum?id=aY3L65HgHJ.openreview.net/forum?id=aY3L65HgHJ
- [33]Haiping Liu, Lijing Lin, Jingyuan Sun, Zhegong Shangguan, Mauricio A Alvarez, and Hong-peng Zhou. Rethinking rope: A mathematical blueprint for n-dimensional positional embedding. arXiv preprint arXiv:2504.06308, 2025.arXiv
- [34]Mhammedi, Z., Hellicar, A., Rahman, A., & Bailey, J. (2017). Efficient orthogonal parametrisation of recurrent neural networks using householder reflections. ICML, 2401–2409.
- [35]Chong Mou, Mingdeng Cao, Xintao Wang, Zhaoyang Zhang, Ying Shan, and Jian Zhang. Revideo: Remake a video with motion and content control. arXiv, 2024.arxiv.org/abs/2405.13865
- [36]Koichi Namekata, Sherwin Bahmani, Ziyi Wu, Yash Kant, Igor Gilitschenski, and David B. Lindell. SG-I2V: Self-guided trajectory control in image-to-video generation. In ICLR, 2025. URL https://openreview.net/forum?id=uQjySppU9x.openreview.net/forum?id=uQjySppU9x
- [37]Muyao Niu, Xiaodong Cun, Xintao Wang, Yong Zhang, Ying Shan, and Yinqiang Zheng. MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model. In ECCV, 2024.arxiv.org/abs/2405.20222
- [38]William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.arxiv.org/abs/2212.09748
- [39]F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In CVPR, 2016.DOI
- [40]Alexander Pondaven, Aliaksandr Siarohin, Sergey Tulyakov, Philip Torr, and Fabio Pizzati. Video motion transfer with diffusion transformers. In CVPR, 2025.arxiv.org/abs/2412.07776
- [41]Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. NeurIPS, 36:79320–79362, 2023.arxiv.org/abs/2306.07280
- [42]Xuanchi Ren, Tianchang Shen, Jiahui Huang, Huan Ling, Yifan Lu, Merlin Nimier-David, Thomas Müller, Alexander Keller, Sanja Fidler, and Jun Gao. Gen3c: 3d-informed world-consistent video generation with precise camera control. In CVPR, pages 6121–6132, 2025.arxiv.org/abs/2503.03751
- [43]Thomas Ressler-Antal, Frank Fundel, Malek Ben Alaya, Stefan Andreas Baumann, Felix Krause, Ming Gui, and Björn Ommer. Dismo: Disentangled motion representations for open-world motion transfer. In NeurIPS, 2025.arxiv.org/abs/2511.23428
- [44]Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. In CVPR, 2016.DOI
- [45]Shi, X., Huang, Z., Wang, F.-Y., Bian, W., Li, D., Zhang, Y., Meng, M., Cheung, K. C., See, S., Qin, H., Dai, J., & Li, H. (2024). Motion-i2V: Consistent and controllable image-to-video generation with explicit motion modeling. SIGGRAPH. https://arxiv.org/abs/2401.15977
- [46]Assaf Singer, Noam Rotstein, Amir Mann, Ron Kimmel, and Or Litany. Time-to-Move: Training-free motion-controlled video generation via dual-clock denoising. In ICLR, 2026.DOI
- [47]Zachary Teed and Jia Deng. RAFT: Recurrent all-pairs field transforms for optical flow. In ECCV, pages 402–419, 2020. URL https://arxiv.org/abs/2003.12039.
- [48]Jakub M Tomczak and Max Welling. Improving variational auto-encoders using householder flow. arXiv preprint arXiv:1611.09630, 2016.arXiv
- [49]Wan, T., Wang, A., Ai, B., Wen, B., Mao, C., Xie, C.-W., Chen, D., Yu, F., Zhao, H., Yang, J., & al., . et . (2025). Wan: Open and advanced large-scale video generative models. In arXiv preprint. arXiv:2503.20314
- [50]Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. In NeurIPS, 2024.arxiv.org/abs/2306.02018
- [51]Yiming Wang, Qihang Zhang, Shengqu Cai, Tong Wu, Jan Ackermann, Zhengfei Kuang, Yang Zheng, Frano Rajic, Siyu Tang, and Gordon Wetzstein. BulletTime: Decoupled control of time and camera pose for video generation. In CVPR, 2026.arxiv.org/abs/2512.05076
- [52]Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. MotionCtrl: A unified and flexible motion controller for video generation. In SIGGRAPH, pages 1–11, 2024.arxiv.org/abs/2312.03641
- [53]Weijia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. DragAnything: Motion control for anything using entity representation. In ECCV, 2024.arxiv.org/abs/2403.07420
- [54]Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xintao Wang, Tien-Tsin Wong, and Ying Shan. DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors. In ECCV, 2024.arxiv.org/abs/2310.12190
- [55]Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vahdat. Camco: Camera-controllable 3d-consistent image-to-video generation. arXiv preprint arXiv:2406.02509, 2024.arXiv
- [56]Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. Unifying flow, stereo and depth estimation. TPAMI, 45(11):13941–13958, 2023.DOI
- [57]Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user-directed camera movement and object motion. In SIGGRAPH, 2024.DOI
- [58]Songlin Yang, Yikang Shen, Kaiyue Wen, Shawn Tan, Mayank Mishra, Liliang Ren, Rameswar Panda, and Yoon Kim. Path attention: Position encoding via accumulating householder transformations. In NeurIPS, 2025.arxiv.org/abs/2505.16381
- [59]Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y., Hong, W., Zhang, X., Feng, G., Dong, D., & al., . et . (2025). CogVideoX: Text-to-video diffusion models with an expert transformer. ICLR. 2408.06072
- [60]Danah Yatim, Rafail Fridman, Omer Bar-Tal, Yoni Kasten, and Tali Dekel. Space-time diffusion features for zero-shot text-driven motion transfer. In CVPR, 2024.arxiv.org/abs/2311.17009
- [61]Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. DragNUWA: Fine-Grained Control in Video Generation by Integrating Text, Image, and Trajectory. arXiv preprint arXiv:2308.08089, 2023.arXiv
- [62]Shen Yuan, Haotian Liu, and Hongteng Xu. Bridging the gap between low-rank and orthogonal adaptation via householder reflection adaptation. NeurIPS, 37:113484–113518, 2024.arxiv.org/abs/2405.17484
- [63]David Junhao Zhang, Roni Paiss, Shiran Zada, Nikhil Karnad, David E. Jacobs, Yael Pritch, Inbar Mosseri, Mike Zheng Shou, Neal Wadhwa, and Nataniel Ruiz. ReCapture: Generative video camera controls for user-provided videos using masked video fine-tuning. In CVPR, 2025.arxiv.org/abs/2411.05003
- [64]Zhang, Y., Wei, Y., Jiang, D., Zhang, X., Zuo, W., & Tian, Q. (2025). ControlVideo: Training-Free Controllable Text-to-Video Generation. ICLR.
- [65]Zhongwei Zhang, Fuchen Long, Zhaofan Qiu, Yingwei Pan, Wu Liu, Ting Yao, and Tao Mei. Motionpro: A precise motion controller for image-to-video generation. In CVPR, pages 27957–27967, 2025.arxiv.org/abs/2505.20287
- [66]Guangcong Zheng, Teng Li, Xianpan Zhou, and Xi Li. Realcam-vid: High-resolution video dataset with dynamic scenes and metric-scale camera movements. arXiv preprint arXiv:2504.08212, 2025.arXiv
- [67]Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all. arXiv preprint arXiv:2412.20404, 2024.arxiv.org/abs/2412.20404
- [68]Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. ACM TOG (Proc. SIGGRAPH), 37 (4), 2018.arxiv.org/abs/1805.09817
