betaFeedback

WorksComputer Science

Improving the Diffusability of Motion Tokenizer

Abstract

Latent diffusion models paired with a variational auto-encoder (VAE) tokenizer have shown promising performance for efficient text-driven human motion generation. However, motion latent diffusion suffers from a generation-reconstruction trade-off: simply increasing tokenizer capacity enhances reconstruction but fails to proportionally improve text-conditioned generation quality. We assume this trade-off stems from a spectral mismatch inside the VAE tokenizer: the encoder yields a latent space biased toward high frequencies, and the decoder introduces temporal artifacts. In this paper, we propose the Diffusable Motion Tokenizer (DiMoT) to improve the diffusability of the tokenizer, rather than scaling up the diffusion backbone. To improve text-conditioned optimization, Critic Spectral Adaptation (CSA) leverages a pretrained diffusion model as the critic to suppress high-frequency latent energy. To preserve motion dynamics, the Alpha-Flow Motion Decoder (AMD) performs generative decoding via average-velocity prediction, enabling efficient one-step decoding. Extensive experiments demonstrate that DiMoT achieves state-of-the-art performance across various challenging text-to-motion benchmarks, yielding superior motion fidelity and text-motion alignment. Project page: https://githubhgh.github.io/dimot/.

Preprint.

Introduction

Generative modeling is entering a Token Era, with text [1, 2, 3] and video [4, 5, 6] tokenization becoming standard practice in large generalist models. Conversely, motion sequences present unique challenges due to their inherent kinematic structure and strict temporal sensitivity, which are also more expensive and time-consuming to collect. For example, human joints have strict rotation limits, and any minor temporal misalignment could result in obvious artifacts such as jitter and foot sliding. Consequently, developing a robust motion tokenizer under these constraints is critical for advancing downstream applications in humanoid robotics [7, 8, 9, 10] and virtual reality [11, 12, 13, 14].

Latent Diffusion Models (LDMs) [15, 16, 17, 18, 19, 20] paired with a pre-trained Variational Autoencoder (VAE) tokenizer have advanced rapidly in motion generation, extending from single-person [21, 22, 23, 24, 25] to multi-person [26, 27, 28] scenarios. Compared with the diffusion backbone, the tokenizer has been less explored, yet it is equally important for high-quality motion generation. While continuous representations are naturally suited for producing smooth motion [29], they struggle to capture the discrete nature of conditioning signals (e.g., text). Furthermore, the tokenizer and generative backbone are typically trained in a decoupled manner, where existing literature often relies on the VAE’s reconstruction accuracy as a proxy for generation quality. However, such an optimal latent space for reconstruction struggles to achieve the same superiority in conditional diffusion generation [30]. Previous studies [31, 32] attribute this gap to the spectral mismatch: the latent space contains abundant high-frequency energy, whereas raw motion is predominantly low-frequency, which degrades downstream generation. In this paper, we propose the Diffusable Motion Tokenizer (DiMoT). Specifically, DiMoT couples Critic Spectral Adaptation (CSA) to regularize the encoder’s latent space with Alpha-Flow Motion Decoder (AMD) to efficiently preserve motion dynamics.

Firstly, it is well-known that VAEs tend to over-smooth motion sequences into low-frequency content [33]. Surprisingly, we empirically find in Figure 1a that the encoder yields a latent space biased toward high frequencies absent from the raw motion. We interpret this as an ill-posed shortcut code in the VAE that easily amplifies back into motion space, while the diffusion backbone struggles to model the text-conditioned distribution with such artificial outliers [31]. However, naively applying a uniform high-frequency penalty also suppresses the necessary dynamics for motion reconstruction. We therefore introduce CSA to use the pretrained diffusion model as a text-conditioned critic regularizer [34]. Moreover, CSA re-weights the regularization loss mode by mode (i.e., Fourier) for spectral reorganization, which adaptively suppresses the VAE’s shortcut outliers. By reweighting the latent space critic regularization with the spectral profile, CSA improves the diffusability of the tokenizer while maintaining comparable reconstruction performance.

Spectral mismatch in motion tokenizers

Figure 1. Spectral mismatch in motion tokenizers. (a) The baseline encoder produces high-frequency shortcut outliers into the latent space. Our Critic Spectral Adaptation (CSA) uses the pre-trained diffusion as a frequency-specific critic to suppress this shortcut high-frequency energy. (b) The baseline deterministic decoder introduces severe high-frequency artifacts into the reconstructed motion. The Alpha-Flow Motion Decoder (AMD) calibrates the spectral profile to closely approximate the real motion dynamics.

Beyond the latent representation, the VAE decoder sets the performance ceiling for generation quality; decoupled training isolates it from the diffusion prior. Consequently, this mismatch implies that a well-optimized latent space does not guarantee accurate motion reconstruction during inference. Previous works [29, 35] employ iterative refinement within diffusion decoding, which uses the latent as conditioning signals to directly guide motion generation. However, the iterative nature of diffusion decoders introduces significant computational overhead, contradicting the primary efficiency of LDMs. To address it, we introduce the AMD, an efficient one-step generative decoder that learns the average velocity field [36, 37]. As illustrated in Figure 1b, AMD’s generative decoding matches the real motion frequency distribution, whereas deterministic decoding inherently introduces high-frequency artifacts. Therefore, the AMD improves the diffusability by calibrating the spectral profile to efficiently preserve the essential motion dynamics.

In summary, our core contributions are:

  • We introduce Critic Spectral Adaptation (CSA), leveraging a pre-trained LDM as a critic for tokenizer encoder regularization. Guided by the text-conditioned distribution, CSA adaptively suppresses shortcut high-frequency energy via spectral reweighting. Furthermore, we provide a theoretical analysis detailing how CSA improves generalized latent diffusion frameworks.

  • We propose the Alpha-Flow Motion Decoder (AMD) to replace the tokenizer decoder with the one-step generative decoder. By learning the latent-conditioned average velocity field, AMD calibrates essential motion dynamics without iterative refinement.

  • We integrate these modules into the Diffusable Motion Tokenizer (DiMoT) to conduct comprehensive evaluations across multi-person and single-person benchmarks. Extensive results show DiMoT demonstrates superior fidelity and text following ability over existing LDM frameworks.

Related Works

Tokenizer Diffusability Bottleneck. Recent works [31, 38, 39, 40] have identified the spectral inconsistency between the latent space and raw data as a primary bottleneck for LDM generation quality. Specifically, vanilla VAE encoders tend to push the latent space into biased high-frequency modes, which diffusion models struggle to fit [31]. Existing solutions attempt to mitigate this by penalizing latent DCT amplitudes [31], matching input and latent power spectra [40], aligning latents with external features [30, 38], or jointly training the encoder with a diffusion decoder [39, 41]. Other approaches regularize latents toward lower frequencies [32] or apply a frequency curriculum during diffusion training without altering the autoencoder [42]. In contrast, CSA leverages the same pre-trained diffusion as a per-frequency critic to fine-tune the encoder. This requires no extra module, external feature extractor, or joint training.

Motion Generation in the Latent Space. LDMs [16] are widely adopted in text-to-motion generation due to their effective trade-off between computational efficiency and generation quality [21, 22, 26, 27, 28, 29, 35, 43, 44, 45, 46]. Early works focus on single-person generation [21], which are further improved by consistency distillation for real-time inference [22]. Different from individual motion generation, multi-person generation is more challenging due to the extra relative orientation and position modeling. Geng et al. [26] applies a decoupled latent space via disentangling the interaction context and individual motion. Recently, the focus has shifted to unified latent spaces capable of handling variable agents and tasks via shared policies [46] or domain-specific motion priors [45]. However, coherence between the tokenizer and the diffusion backbone remains under-explored. Concurrent works bridge this coherence gap via Gaussian mixture regularization [44] or generative discrete decoding [29]. These methods either ignore the inherent high-frequency outliers in the decoder output or introduce significant computational overhead. Instead, we replace the VAE decoder with AMD to preserve the essential motion dynamics, achieving high-fidelity, one-step generative decoding [41].

Methodology

Problem Statement. Given a user-input text yy, the goal is to generate a natural and smooth motion sequence xx that aligns with the language semantics. This can be modeled with a text-conditioned Latent Diffusion Model (LDM), where motion is generated in a latent space and decoded into the final motion sequence. Standard LDMs use a decoupled approach: training a diffusion model fθf_\theta strictly on the frozen latents z=Eφ(x)z = E_\varphi(x) of a pre-trained VAE tokenizer (Eφ,Dψ)(E_\varphi, D_\psi). Rather than scaling up the diffusion backbone, we target improving the tokenizer (Eφ(E_\varphi and Dψ)D_\psi) to maximize its diffusability.

To achieve this, we present the Diffusable Motion Tokenizer (DiMoT), a framework that addresses the spectral mismatch and bridges the performance gap between tokenizer reconstruction and diffusion generation. As illustrated in Figure 2, DiMoT consists of two core components: (1) Critic Spectral Adaptation (CSA), which uses a pre-trained diffusion as the critic to adaptively regularize the latent spectral profile of encoder EφE_\varphi, and (2) Alpha-Flow Motion Decoder (AMD), which replaces the decoder DψD_\psi with one-step generative decoding to preserve temporal motion dynamics. In this way, DiMoT achieves high-quality text-conditioned generation while simultaneously preserving reconstruction fidelity.

Overview of the DiMoT framework

Figure 2. Overview of our DiMoT framework. Left: Critic Spectral Adaptation (CSA). CSA utilizes a pre-trained latent diffusion model as a text-conditioned critic, which adaptively suppresses shortcut high-frequency outliers via spectral reweighting. Right: Alpha-Flow Motion Decoder (AMD). By using curriculum learning over trajectory velocity fields in different time scales, AMD gradually learns one-step average-velocity generative decoding with latent conditions.

Critic Spectral Adaptation (CSA)

We introduce Critic Spectral Adaptation (CSA) to fine-tune the encoder EφE_\varphi via spectral reweighting, leveraging a pre-trained text-conditioned diffusion model as a critic [34, 47]. We begin by formulating the spectral critic loss, followed by a theoretical analysis detailing how CSA facilitates the optimization of the downstream diffusion objective.

Diffusion as a Critic. By employing the pre-trained diffusion model fθ0f_{\theta_0} as a generative critic, CSA optimizes the latent space manifold toward improved text-conditioned generability, rather than trivially fitting the reconstruction objective. Let EφE_\varphi be the pre-trained motion VAE encoder with the decoder Dψ0D_{\psi_0} pinned at its baseline parameters ψ0\psi_0, and let Lrec(φ)\mathcal{L}_{\mathrm{rec}}(\varphi) denote the reconstruction loss as a function of the encoder. The frozen text-conditioned rectified-flow latent diffusion model fθ0(xt,t,y)f_{\theta_0}(x_t,t,y) is trained on the baseline tokenizer’s latents [16]. For motion-prompt pair (x,y)∼pdata(x,y) \sim p_{\mathrm{data}} and latent z=Eφ(x)z = E_\varphi(x), sample interpolation time t∼U[0,1]t \sim\mathcal{U}[0,1] and Gaussian noise ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0,I). Set xt=(1−t)sg⁡(z)+tϵx_t = (1-t)\operatorname{sg}(z) + t\epsilon. The prediction loss/residual is

r(z)=sg⁡(fθ0(xt,t,y))−(ϵ−z),(1)r(z) = \operatorname{sg}(f_{\theta_0}(x_t,t,y)) - (\epsilon-z), \tag*{(1)}

where sg⁡\operatorname{sg} is the stop-gradient. The stop-gradients keep fθ0f_{\theta_0} a fixed critic rather than a trainable model to differentiate through, restricting parameter updates solely to the encoder. Finally, the expected squared residual gives the (plain) critic loss Lcrit(φ)=E(x,y),t,ϵ∥r(z)∥2\mathcal{L}_{\mathrm{crit}}(\varphi) = \mathbb{E}_{(x,y),t,\epsilon}\|r(z)\|^2 (See Figure 4a results).

Comparison of different design choices with DiMoT on the InterHuman dataset

Figure 4. Comparison of different design choices with DiMoT on the InterHuman dataset.

Spectral Reweighting. Inspired by [31], we further improve the critic loss in the frequency domain. The Fast Fourier Transform (FFT) F\mathcal{F} decomposes r(z)r(z) along the time axis into Fourier modes indexed by kk, where k=0k=0 is the static component and larger kk corresponds to faster temporal variations. The residual amplitude ∣F[r(z)](k)∣|\mathcal{F}[r(z)](k)| is small at modes the diffusion handles well, and large at the shortcut high-frequency modes the encoder introduces (Figure 1a). However, Lcrit\mathcal{L}_{\mathrm{crit}} weights every mode equally, so we reweight it across modes to concentrate the encoder’s response on the shortcut band:

LCSA(φ)=E(x,y),t,ϵ[1Nt∑kw~(k)∣F[r(z)](k)∣2].(2)\mathcal{L}_{\mathrm{CSA}}(\varphi) = \mathbb{E}_{(x,y),t,\epsilon}\left[\frac{1}{N_t}\sum_k \widetilde{w}(k)\left|\mathcal{F}[r(z)](k)\right|^2\right]. \tag*{(2)}

Here, the weight w~(k)=w(k)/w‾\widetilde{w}(k)=w(k)/\overline{w} with w(k)=1+κd(k)pw(k)=1+\kappa d(k)^p is a static spectral weight (d(k)∈[0,1]d(k)\in[0,1] denotes FFT distance from k=0k=0, pp denotes polynomial exponent). The strength knob κ\kappa controls how aggressively the encoder pressure is biased toward the high-frequency band. The gradient on zz at each mode is proportional to w~(k)r^(k)\widetilde{w}(k)\hat{r}(k) (Theorem 13, Appendix § 7.§ 4), so modes already handled by the diffusion receive little encoder gradient. Algorithm 1 summarizes the resulting training step.

Algorithm 1 CSA Training Loop.

Require: encoder EφE_\varphi, frozen diffusion fθ0f_{\theta_0}, FFT length NtN_t, total steps SS

  1. Pre-compute spectral weights w~(k)=(1+κd(k)p)/w‾\widetilde{w}(k)=(1+\kappa d(k)^p)/\overline{w} for k=0,…,Nt−1k=0,\ldots,N_t-1 ▷ static, FFT-geometry only

  2. for step i=1,…,Si=1,\ldots,S do

  3. Sample (x,y)∼pdata(x,y)\sim p_{\mathrm{data}}, t∼U[0,1]t\sim\mathcal{U}[0,1], ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0,I) ▷ joint motion-prompt batch

  4. z←Eφ(x)z\leftarrow E_\varphi(x), xt←(1−t)sg⁡(z)+tϵx_t\leftarrow(1-t)\operatorname{sg}(z)+t\epsilon ▷ noised interpolant, encoder takes xx only

  5. r←sg⁡(fθ0(xt,t,y))−(ϵ−z)r\leftarrow\operatorname{sg}(f_{\theta_0}(x_t,t,y))-(\epsilon-z) ▷ Eq. (1), text yy enters here

  6. LCSA←Nt−1∑kw~(k)∣F[r](k)∣2\mathcal{L}_{\mathrm{CSA}}\leftarrow N_t^{-1}\sum_k\widetilde{w}(k)|\mathcal{F}[r](k)|^2 ▷ Eq. (2), FFT along temporal axis

  7. Update φ\varphi on Lrec(φ)+λcmin⁡(1,i/W)LCSA(φ)\mathcal{L}_{\mathrm{rec}}(\varphi)+\lambda_c\min(1,i/W)\mathcal{L}_{\mathrm{CSA}}(\varphi) ▷ ψ0,θ0\psi_0,\theta_0 frozen, WW-step warmup

  8. end for

CSA Improves Downstream Diffusion. While EφE_\varphi (φ\varphi denotes the encoder parameters) operates solely on motion inputs, CSA guides the encoder’s gradients by incorporating the prompt yy via the critic’s text-conditioned prediction sg⁡(fθ0(xt,y))\operatorname{sg}(f_{\theta_0}(x_t,y)). Moreover, we provide the theoretical foundations detailing how it improves general conditional diffusion optimization within this learned latent space.

Define the text-conditioned diffusion training loss as Ldiff(θ;φ):=E(x,y),t,ϵ∥fθ(xt,t,y)−(ϵ−z)∥2\mathcal{L}_{\mathrm{diff}}(\theta;\varphi) := \mathbb{E}_{(x,y),t,\epsilon}\|f_\theta(x_t,t,y)-(\epsilon-z)\|^2. Under reasonable conditions, we analyze the CSA regularization from the bilevel optimization theory [47, 48, 49]. Consequently, we prove that CSA translates into a strict improvement in Ldiff\mathcal{L}_{\mathrm{diff}} as follows:

Theorem 1 (Diffusion-loss improvement). Adopt the assumptions and notation of Theorem 20 (Appendix § 7.6). Suppose also θ0=arg⁡min⁡θLdiff(θ;φ0)\theta_0=\arg\min_\theta\mathcal{L}_{\mathrm{diff}}(\theta;\varphi_0) and gcrit⊤H−1g>0g_{\mathrm{crit}}^\top H^{-1}g>0, where gcrit:=∇φLdiff(θ0;φ0)g_{\mathrm{crit}}:=\nabla_\varphi\mathcal{L}_{\mathrm{diff}}(\theta_0;\varphi_0) and H:=∇φ2Lrec(φ0)H:=\nabla_\varphi^2\mathcal{L}_{\mathrm{rec}}(\varphi_0). Then for all sufficiently small λc>0\lambda_c>0,

min⁡θLdiff(θ;φ∗(λc))<min⁡θLdiff(θ;φ0).(3)\min_\theta\mathcal{L}_{\mathrm{diff}}(\theta;\varphi^*(\lambda_c)) < \min_\theta\mathcal{L}_{\mathrm{diff}}(\theta;\varphi_0). \tag*{(3)}

The alignment condition is treated as a working assumption (Appendix § 7.7).

Intuitively, at the CSA-trained encoder φ∗(λc)\varphi^{*}(\lambda_c), training the diffusion from scratch yields a strictly lower training loss than at the baseline φ0\varphi_0. Meanwhile, when fine-tuning, reconstruction is theoretically preserved at first order in λc\lambda_c along the same trajectory (Theorem 20, Appendix § 7.6). Even without optimality of the baseline diffusion, Corollary 26 yields the same conclusion for fixed θ0\theta_0.

As a cross-modality check, we apply CSA to a class-conditional ImageNet [50, 51] pipeline (Appendix § 8). In line with Theorem 1, generation FID improves at the evaluated diffusion training checkpoints (i.e., 100k, 200k and 300k steps). Reconstruction FID remains close to baseline, suggesting information is reorganized rather than removed.

Alpha-Flow Motion Decoder

As shown in Figure 1, while deterministic decoding inherently produces this high-frequency noise, generative decoding can match the frequency distribution of real motion. Compared to the VAE encoder, the VAE decoder directly determines the upper bound of LDM generation performance, while it only plays the static role as a passive receiver in a mismatched way [41]. Such a mismatch becomes increasingly disruptive for the motion tokenizer. Iterative generative decoding can mitigate it, which also introduces prohibitive computational overhead. To this end, we propose the Alpha-Flow Motion Decoder (AMD) in Figure 2, a latent-conditioned one-step generative decoder that preserves the efficiency while restoring motion details lost by standard VAE decoding [31, 32].

Latent Condition Upsampling Projection. Given latent tokens z∈RNz×dzz \in\mathbb{R}^{N_z \times d_z} from a frozen diffusion backbone, we aim to recover a frame-aligned condition sequence cz∈RT×dCc_z \in\mathbb{R}^{T \times d_C}. Unlike [29], our continuous latent tokens avoid strict frame-to-frame correspondence, where each token is aligned with a coarse temporal segment. Rather than naively interpolating these tokens, we map the compressed memory to target frames using a Transformer decoder for upsampling projection. We first linearly project each latent token into the conditioning space, m=Proj⁡(z)∈RNz×dCm = \operatorname{Proj}(z) \in\mathbb{R}^{N_z \times d_C}, which serves as the memory sequence. Meanwhile, we construct a set of learnable frame queries q∈RT×dCq \in\mathbb{R}^{T \times d_C}, where each query represents one target frame position. After adding positional encoding to both the memory and query sequences, we feed them into a transformer decoder, where the queries interact with each other through self-attention and retrieve motion information from the latent memory through cross-attention: cz=TransDec⁡(q,m)∈RT×dCc_z = \operatorname{TransDec}(q, m) \in\mathbb{R}^{T \times d_C}. This global cross-attention ensures each condition vector czc_z captures the full semantic context, providing the fine-grained temporal alignment required for accurate motion generation.

Generative Decoding on Average Velocity Field. After obtaining the latent condition, we apply AMD [52] as a generative backbone to generate the motion sequences. Compared to vanilla rectified flow matching [53], AMD learns to predict the average velocity field in a one-step setting, without relying on costly iterative refinement. In detail, vanilla flow matching learns local denoising directions instantaneously, whereas AMD predicts a long-range transport toward motion target XX via velocity integration [37]. Moreover, AMD applies curriculum training to achieve smooth transitions between different trajectory flow matching, which shows better convergence and stability.

Following MeanFlow [37], we introduce the average velocity field for one-step diffusion decoding. Given a destination time rd<tr_d < t, the decoder uψu_\psi predicts an average transition velocity u(xt,rd,t,z)u(x_t, r_d, t, z) over the interval [rd,t][r_d, t] conditioned on latent space zz as follows:

u(xt,rd,t,z)=1t−rd∫rdtv(xτ,τ,z) dτ,(4)u(x_t, r_d, t, z) = \frac{1}{t-r_d}\int_{r_d}^{t} v(x_\tau, \tau, z)\,\mathrm{d}\tau, \tag*{(4)}

where xx is the clean motion sample and Gaussian noise ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0, I), xt=(1−t)x+tϵx_t = (1-t)x + t\epsilon, vt=ϵ−xv_t = \epsilon- x, where t∈[0,1]t \in[0,1], x0=xx_0 = x, and x1=ϵx_1 = \epsilon. By conditioning explicitly on the destination time, the model can represent large trajectory jumps that are necessary for direct one-step decoding based on MeanFlow identity. However, the original MeanFlow’s training target is not tractable and depends on the network prediction. By differentiating the integral definition (see Appendix § 9 for derivation), the average velocity derivative target is calculated via the Jacobian-vector product of the decoder network:

u(xt,rd,t,z)=v(xt,t,z)−(t−rd)ddtuψ−(xt,rd,t,z).(5)u(x_t, r_d, t, z) = v(x_t, t, z) - (t-r_d)\frac{\mathrm{d}}{\mathrm{d}t}u_{\psi^-}(x_t, r_d, t, z). \tag*{(5)}

In practice, MeanFlow training configuration sets t=rdt = r_d for 75% of samples, providing instantaneous flow-matching supervision [37]. AMD instead uses t=rdt = r_d for 50% of each batch, while the remaining samples with rd<tr_d < t learn interval-average velocities for one-step decoding. The α\alpha schedule and training details are provided in Appendix § 11.

Alpha-Flow Curriculum Learning To mitigate such reliance on trivial instantaneous velocity prediction, AMD performs one-step average velocity supervision by gradually shifting from fast-converging trajectory flow matching to intractable MeanFlow optimization. In practice, AMD enforces consistency between interval predictions defined over (t,s,rd)(t, s, r_d):

s=αrd+(1−α)t, α∈(0,1],xs=xt−(t−s)v~s,t,(6)s = \alpha r_d + (1-\alpha)t,\ \alpha\in(0,1], \qquad x_s = x_t - (t-s)\tilde{v}_{s,t}, \tag*{(6)}

where α\alpha is the time-variant scheduler, and xsx_s is the motion state at timestep ss. Based on this connection, we train the decoder with an α\alpha-curriculum rather than a fixed objective (e.g., s=ts=t for vanilla flow matching). The AMD objective is formally defined as:

LAMD(ψ)=Ex,ϵ,t,rd[1α∥uψ(xt,rd,t,z)−(αv~s,t+(1−α)uψ−(xs,rd,s,z))∥22],(7)\mathcal{L}_{\mathrm{AMD}}(\psi) = \mathbb{E}_{x,\epsilon,t,r_d}\left[\frac{1}{\alpha}\left\|u_\psi(x_t,r_d,t,z)-\left(\alpha\tilde{v}_{s,t}+(1-\alpha)u_{\psi^-}(x_s,r_d,s,z)\right)\right\|_2^2\right], \tag*{(7)}

where v~s,t=vt\tilde{v}_{s,t}=v_t is used to form the shift target in the endpoint value. Specifically, when α=1\alpha=1, it reduces to trajectory flow matching, which provides stable transition supervision. When α→0\alpha\rightarrow0, it recovers the MeanFlow regime, which is better aligned with the final goal of one-step generation.

Notably, the latent can come from text-conditioned diffusion backbones or VAE encoders. A decoder trained with a tokenizer latent may still face a train-test mismatch at inference time, while the diffusion backbone could produce training samples with high variance. To reduce this mismatch, we use a mixed training scheme by randomly replacing the latent samples produced.

At inference time, we instantiate decoding with a single interval from t=1t=1 to rd=0r_d=0, starting from Gaussian noise in raw motion space. Conditioned on the latent token, the decoder predicts the whole average velocity field of raw motion sequences in one step: x^=x1−uψ(x1,0,1,z)\hat{x}=x_1-u_\psi(x_1,0,1,z). In this way, we use the AMD to replace deterministic decoding with efficient but more effective motion realization, preserving the coherent high-frequency dynamics (See the video in the supplementary material).

DiMoT Justification

Training the Tokenizer for Generation While DiMoT optimizes the tokenizer, it is not limited to maximizing standard reconstruction accuracy. Instead, DiMoT enhances motion generation by improving the tokenizer’s diffusability. It structures the latent space to be more easily modeled by the diffusion process. We evaluate this by showing that DiMoT improves text-conditioned generation performance while maintaining comparable reconstruction fidelity. To isolate the tokenizer’s contribution, we keep the downstream diffusion backbone architecture and training configuration strictly fixed (See Appendix § 11).

Optimal Latent Spectral Profile High-frequency energy in the latent space is more challenging to suppress, as latent spaces lack the tractable ground truth for spectral alignment. While excessively high-frequency energy degrades diffusion generation, naively eliminating it also compromises sample quality [31]. Consequently, prior methods [32] rely on mathematical transformations (e.g., downsampling) as anchors. In contrast, CSA provides a flexible anchoring mechanism that adaptively penalizes high-frequency components, exclusively for samples misaligned with the critic distribution.

Experiments

Datasets

We utilize InterHuman [54] and Inter-X [55] datasets for the evaluation of text-conditioned interaction generation performance. We also provide HumanML3D [56] results in Table 3, but this paper focuses on the dual-person task. This setting imposes stricter constraints that more effectively expose the temporal artifacts and high-frequency outliers our method addresses.

MethodR-Top3↑\uparrowFID↓\downarrowMM Dist↓\downarrowMModality↑\uparrow
GT0.5110.0022.974-
MLD++ [65]0.5440.0492.8281.672
MLD++ [65] w. CSA0.5520.0472.8081.715
MLD++ [65] w. AMD0.5420.0422.8421.675
MLD++ [65] w. DiMoT0.5500.0382.8091.689

Table 3. Single-person results on HumanML3D.

The InterHuman and InterX datasets contain 7,779 and 11,388 interaction sequences, respectively, where each sequence is illustrated with 3 textual annotations. InterHuman follows the SMPL [57] skeleton representation with 22 joints, including the root joint. Each interaction sequence at the ii-th frame is represented by: xi=[jgp,jgv,jr,cf]x^i = [j_g^p,j_g^v,j^r,c^f] which is the collection of joint positions jgp∈R2×22×3j_g^p \in\mathbb{R}^{2 \times22 \times3}, joint velocities jgv∈R2×22×3j_g^v \in\mathbb{R}^{2 \times22 \times3}, 6D representation rotations jr∈R2×21×6j^r \in\mathbb{R}^{2 \times21 \times6}, and binary foot-ground contact features cf∈R2×4c^f \in\mathbb{R}^{2 \times4}, resulting in a total input dimension of 524. Inter-X follows the SMPL-X [58] skeleton representation, comprising 54 body and hand joints, accompanied by root orientation and translation. Each interaction sequence could be represented by xi=[rgp,jr]x^i = [r_g^p,j^r] comprising root joint positions rgp∈R2×3r_g^p \in\mathbb{R}^{2 \times3} and Euler angle representation rotations jr∈R2×55×3j^r \in\mathbb{R}^{2 \times55 \times3}, for a total input dimension of 336.

We employ the evaluation metrics following previous studies [54, 56]. (See Appendix § 10 for more details). The implementation of CSA, AMD, and diffusion backbone can be found in Appendix § 11.

Main Results

Table 1 shows DiMoT achieves state-of-the-art performance on the InterHuman [54] benchmark. Compared to the MLD++ [22] baseline, DiMoT improves Top-3 R-Precision from 0.688 to 0.739 and reduces FID by 6.5% (5.046 to 4.716). Furthermore, DiMoT consistently outperforms recent methods (e.g., DHVAE [26], TMotion [62]) across all R-Precision tiers while establishing the lowest overall FID. Similar to DHVAE, these methods based on LDMs suffer from low MModality, while our DiMoT still shows better diversity over the MLD++, with the Diversity metric closely matching the ground truth. In the Inter-X evaluations, compared to the MLD++ baseline, DiMoT improves R-Precision (0.732 to 0.746) and achieves a substantial 24.9% reduction in FID (0.446 to 0.335). DiMoT still outperforms the DHVAE across R-Precision metrics, achieving the lowest FID and MM Dist. These results confirm DiMoT’s generalizability to both SMPL [57] and SMPL-X [58] data. While suppressing high-frequency shortcuts with CSA slightly degrades reconstruction FID, the MPJPE improvement demonstrates enhanced kinematic accuracy.

Figure 3 qualitatively compares DiMoT with the MLD++ [65] baseline across diverse interactive text prompts. In the first scenario, MLD++ fails to capture the interaction semantics with interpenetration. Conversely, DiMoT synthesizes the first agent’s sprinting approach and the second’s synchronized dismissing gesture. For the dancing prompt, MLD++ yields generic, rigid movements that ignore the target style. DiMoT instead generates expressive, coordinated poses, capturing complex kinematics like "stomping feet" and "swaying hips". In the "bumping" scenario, MLD++ poorly models the interaction, yielding unnatural spatial proximity. DiMoT models spatial boundaries and momentum strictly with natural post-contact reactions. These visualization results demonstrate that DiMoT mitigates temporal artifacts, showing smooth and semantically accurate multi-person motions. Furthermore, Appendix § 14 reports a user study comparing DiMoT with InterMask.

DatasetMethodR Precision↑ Top 1R Precision↑ Top 2R Precision↑ Top 3FID↓MM Dist↓Diversity→MModality↑
InterHumanGround Truth0.452±0.0080.452 \pm0.0080.610±0.0090.610 \pm0.0090.701±0.0080.701 \pm0.0080.273±0.0070.273 \pm0.0073.755±0.0083.755 \pm0.0087.948±0.0647.948 \pm0.064-
InterHumanTEMOS [59]0.224±0.0100.224 \pm0.0100.316±0.0130.316 \pm0.0130.450±0.0180.450 \pm0.01817.375±0.04317.375 \pm0.0436.342±0.0156.342 \pm0.0156.939±0.0716.939 \pm0.0710.535±0.0140.535 \pm0.014
InterHumanT2M [56]0.238±0.0120.238 \pm0.0120.325±0.0100.325 \pm0.0100.464±0.0140.464 \pm0.01413.769±0.07213.769 \pm0.0725.731±0.0135.731 \pm0.0137.046±0.0227.046 \pm0.0221.387±0.0761.387 \pm0.076
InterHumanMDM [60]0.153±0.0120.153 \pm0.0120.260±0.0090.260 \pm0.0090.339±0.0120.339 \pm0.0129.167±0.0569.167 \pm0.0567.125±0.0187.125 \pm0.0187.602±0.0457.602 \pm0.0452.350±0.0802.350 \pm0.080
InterHumanComMDM [61]0.223±0.0090.223 \pm0.0090.334±0.0080.334 \pm0.0080.466±0.0100.466 \pm0.0107.069±0.0547.069 \pm0.0546.212±0.0216.212 \pm0.0217.244±0.0387.244 \pm0.0381.822±0.0521.822 \pm0.052
InterHumanInterGen [54]0.371±0.0100.371 \pm0.0100.515±0.0120.515 \pm0.0120.624±0.0100.624 \pm0.0105.918±0.0795.918 \pm0.0795.108±0.0145.108 \pm0.0147.387±0.0297.387 \pm0.0292.141±0.0632.141 \pm0.063
InterHumanTIMotion [62]0.491±0.0050.491 \pm0.0050.648±0.0040.648 \pm0.0040.724±0.0040.724 \pm0.0045.433±0.0805.433 \pm0.0803.775±0.0013.775 \pm0.0018.032±0.0308.032 \pm0.0300.952±0.0320.952 \pm0.032
InterHumanin2IN [63]0.425±0.0080.425 \pm0.0080.576±0.0080.576 \pm0.0080.662±0.0090.662 \pm0.0095.535±0.1205.535 \pm0.1203.803±0.0023.803 \pm0.0027.953±0.0477.953 \pm0.0471.215±0.0231.215 \pm0.023
InterHumanInterMask [64]0.449±0.0040.449 \pm0.0040.599±0.0050.599 \pm0.0050.683±0.0040.683 \pm0.0045.154±0.0615.154 \pm0.0613.790±0.0023.790 \pm0.0027.944±0.0337.944 \pm0.0331.737±0.0201.737 \pm0.020
InterHumanDHVAE [26]0.496±0.0040.496 \pm0.0040.647±0.0060.647 \pm0.0060.720±0.0050.720 \pm0.0055.015±0.0855.015 \pm0.0853.772±0.0023.772 \pm0.0027.952±0.0457.952 \pm0.0450.804±0.0300.804 \pm0.030
InterHumanMLD++† [65]0.412±0.0070.412 \pm0.0070.594±0.0030.594 \pm0.0030.688±0.0050.688 \pm0.0055.046±0.0845.046 \pm0.0843.787±0.0013.787 \pm0.0018.084±0.0238.084 \pm0.0230.613±0.0280.613 \pm0.028
InterHumanDiMoT0.519±0.0040.519 \pm0.0040.663±0.0080.663 \pm0.0080.739±0.0060.739 \pm0.0064.716±0.0804.716 \pm0.0803.775±0.0013.775 \pm0.0017.970±0.0357.970 \pm0.0350.650±0.0130.650 \pm0.013
InterXGround Truth0.429±0.0040.429 \pm0.0040.626±0.0030.626 \pm0.0030.736±0.0030.736 \pm0.0030.002±0.0020.002 \pm0.0023.536±0.0133.536 \pm0.0139.734±0.0789.734 \pm0.078-
InterXTEMOS [59]0.092±0.0030.092 \pm0.0030.171±0.0030.171 \pm0.0030.238±0.0020.238 \pm0.00229.258±0.06929.258 \pm0.0696.867±0.0136.867 \pm0.0134.738±0.0784.738 \pm0.0780.672±0.0410.672 \pm0.041
InterXT2M [56]0.184±0.0100.184 \pm0.0100.298±0.0060.298 \pm0.0060.396±0.0050.396 \pm0.0055.481±0.3825.481 \pm0.3829.576±0.0069.576 \pm0.0065.771±0.1515.771 \pm0.1512.761±0.0422.761 \pm0.042
InterXMDM [60]0.203±0.0090.203 \pm0.0090.329±0.0070.329 \pm0.0070.426±0.0050.426 \pm0.00523.701±0.05723.701 \pm0.0579.548±0.0149.548 \pm0.0145.856±0.0775.856 \pm0.0773.490±0.0613.490 \pm0.061
InterXMDM(GRU) [60]0.179±0.0060.179 \pm0.0060.299±0.0050.299 \pm0.0050.387±0.0070.387 \pm0.00732.617±0.12232.617 \pm0.1229.557±0.0199.557 \pm0.0197.003±0.1347.003 \pm0.1343.430±0.0353.430 \pm0.035
InterXComMDM [61]0.090±0.0020.090 \pm0.0020.165±0.0040.165 \pm0.0040.236±0.0040.236 \pm0.00429.266±0.06729.266 \pm0.0676.870±0.0176.870 \pm0.0174.734±0.0674.734 \pm0.0670.771±0.0530.771 \pm0.053
InterXInterGen [54]0.207±0.0040.207 \pm0.0040.335±0.0050.335 \pm0.0050.429±0.0050.429 \pm0.0055.207±0.2165.207 \pm0.2169.580±0.0119.580 \pm0.0117.788±0.2087.788 \pm0.2083.686±0.0523.686 \pm0.052
InterXInterMask [64]0.403±0.0050.403 \pm0.0050.595±0.0040.595 \pm0.0040.705±0.0050.705 \pm0.0050.399±0.0130.399 \pm0.0133.705±0.0173.705 \pm0.0179.046±0.0739.046 \pm0.0732.261±0.0812.261 \pm0.081
InterXDHVAE [26]0.442±0.0050.442 \pm0.0050.638±0.0050.638 \pm0.0050.745±0.0040.745 \pm0.0040.339±0.0150.339 \pm0.0153.604±0.0203.604 \pm0.0209.378±0.0659.378 \pm0.0652.505±0.0472.505 \pm0.047
InterXMLD++† [65]0.433±0.0050.433 \pm0.0050.625±0.0070.625 \pm0.0070.732±0.0060.732 \pm0.0060.446±0.0170.446 \pm0.0173.659±0.0283.659 \pm0.0284.664±0.0624.664 \pm0.0621.611±0.1151.611 \pm0.115
InterXDiMoT0.450±0.0070.450 \pm0.0070.642±0.0040.642 \pm0.0040.746±0.0040.746 \pm0.0040.335±0.0130.335 \pm0.0133.562±0.0303.562 \pm0.0304.648±0.0604.648 \pm0.0602.427±0.1122.427 \pm0.112

Table 1. Quantitative evaluation on the InterHuman [54] and InterX test sets. ±\pm indicates a 95% confidence interval (10 runs) and →\to means the closer to ground truth the better. The deep blue background indicates the best, while the light blue background indicates the second-best. ’†’ denotes self-reproduced results.

Ablation Study

Table 2 ablates the DiMoT components across Latent Diffusion Model (LDM) baselines (MLD [21], MLD++ [65]). Integrating DiMoT consistently improves generation quality independent of the base architecture. By applying our tokenizer design to MLD++, DiMoT achieves the best generative performance with minimum temporal errors (Foot Sliding: 0.0168). DiMoT improves the diffusability with a superior trade-off between reconstruction and generation. The tokenizer maintains its fidelity with stronger MPJPE results while maximizing motion fidelity and text alignment.

MethodTokenizer rFID↓Tokenizer rR-Top3↑Tokenizer MPJPE↓Diffusion FID↓Diffusion R-Top3↑Diffusion Foot Sliding↓
MLD [21]0.3170.62971.105.4770.6360.0209
GMM [44]0.1830.65158.155.1640.6650.0191
MLD++ [65]0.0800.68649.265.0460.6880.0186
MLD [21] w. CSA0.3150.66167.705.3220.6890.0203
MLD [21] w. AMD0.2220.65659.345.2090.6620.0194
MLD [21] w. DiMoT0.193 (-0.124)0.663 (+0.034)53.90 (-17.20)5.245 (-0.232)0.685 (+0.049)0.0197 (-0.0012)
MLD++ [65] w. CSA0.0860.71347.184.9670.7460.0182
MLD++ [65] w. AMD0.0770.67246.764.7810.7160.0175
MLD++ [65] w. DiMoT0.071 (-0.009)0.698 (+0.012)41.36 (-7.90)4.716 (-0.330)0.739 (+0.051)0.0168 (-0.0018)

Table 2. Ablation study of DiMoT components. Results are from the InterHuman [54] dataset.

Figure 4(a) compares different regularization methods applied to the tokenizer encoder. "No Critic" refers to the encoder that relies solely on a reconstruction loss, which evaluates the effectiveness of the reconstruction objective itself. CSA achieves optimal performance (R-Top3: 0.74) when fine-tuned with an additional adapter, which is further enhanced by spectral reweighting. Without an adapter, CSA functions as a parameter-free objective applied to the standard MLD++ [65] diffusion backbone, yet achieves better text-motion alignment. More spectral reweighting results are included in Appendix G.

Qualitative comparison between MLD++ and DiMoT

Figure 3. Qualitative comparison [66] (zoom in to see it better) between baseline MLD++ [22] and DiMoT, highlighting DiMoT’s superior generation quality. (See the video in supplementary material.)

Figure 4(b) evaluates various decoder generative schedulers in terms of latency and FID. We also highlight the inference step and FLOPs with the superscript and circle size, respectively. DisCoRD16NFE^{16\mathrm{NFE}} achieves better fidelity than the baseline but with excessive latency and FLOPs. In contrast, our AMD improves the generation fidelity with single-step efficiency, reducing computational overhead by orders of magnitude (See more results in Appendix F). Moreover, we compare linear and sigmoid curricula in Appendix G.1.

In Table 3, we also conduct an evaluation on the single-person benchmark (i.e., HumanML3D [56]). By integrating DiMoT into MLD++ [65], it also improves the baseline performance. DiMoT dominates in fidelity with the lowest FID, and also shows gains in R-Top3 and MModality. These results further confirm that DiMoT’s architectural benefits generalize to motion domains.

Conclusion

In this paper, we address the generation-reconstruction trade-off in motion latent diffusion models by improving diffusability of the tokenizer, rather than scaling the diffusion backbone. We observe a spectral mismatch wherein both the latent and output profiles exhibit an abnormally high proportion of high-frequency components, potentially degrading text-to-motion generation. To address this, we propose the Diffusable Motion Tokenizer (DiMoT), consisting of Critic Spectral Adaptation (CSA) and Alpha-Flow Motion Decoder (AMD). Firstly, we demonstrate, both theoretically and empirically, that CSA regularizes the latent space via a diffusion-based spectral critic, suppressing high-frequency shortcuts without sacrificing reconstruction. Concurrently, AMD replaces deterministic decoding with a one-step average-velocity prediction, selectively preserving the essential dynamics required for temporal realism. Evaluations on standard motion benchmarks demonstrate that DiMoT successfully balances generation fidelity and reconstruction accuracy, validating our tokenizer refinement strategy.

Limitation DiMoT addresses spectral mismatch in text-to-motion generation, but defining the optimal spectral profile of the latent space remains an open problem, where CSA relies on using the diffusion as a critic with empirical choices. Moreover, motion datasets are limited in scale and use heterogeneous representations. Our evaluations demonstrate improvements within each dataset, but do not establish whether a tokenizer adapted on one dataset retains its performance on another. Future work will explore reinforcement or adversarial learning to replace the polynomial spectral weight, bypassing these rigid priors and optimality assumptions.

Acknowledgments

This work was supported by the K-CSC funding. The authors acknowledge the use of King’s CREATE HPC. Retrieved November 05, 2026, from https://doi.org/10.18742/rnvf-m076.

Full theoretical development of CSA

Notation

SymbolMeaning
X,Z\mathcal{X},\mathcal{Z}data and latent spaces, Z=Rd\mathcal{Z} = \mathbb{R}^{d} with d=C′×Nd = C' \times N, N=n1⋯nDN = n_{1} \cdots n_{D}
Eφ,Dψ0E_{\varphi}, D_{\psi_{0}}encoder (with parameters φ\varphi) and decoder pinned at its pre-trained parameters ψ0\psi_{0}
pdatap_{\mathrm{data}}data distribution
pzφp_{z}^{\varphi}induced latent distribution (Eφ)#pdata(E_{\varphi})_{\#}p_{\mathrm{data}}
fθ,θ0f_{\theta}, \theta_{0}velocity model (rectified flow) and its frozen pre-trained parameters
sg⁡(⋅)\operatorname{sg}(\cdot)stop-gradient (identity on values, zero on gradients)
xt,ϵ,tx_{t},\epsilon,trectified-flow interpolant, noise, time
y,py,pz∣yφy,p_{y},p_{z\mid y}^{\varphi}text prompt, text marginal, conditional latent law (Eφ)#p(x∣y)(E_{\varphi})_{\#}p(x\mid y)
r(z)r(z)prediction residual sg⁡(vpred)−(ϵ−z)\operatorname{sg}(v_{\mathrm{pred}})-(\epsilon-z), with vpred:=fθ0(xt,t,y)v_{\mathrm{pred}}:=f_{\theta_{0}}(x_{t},t,y)
Lcrit\mathcal{L}_{\mathrm{crit}}plain critic loss
LCSA\mathcal{L}_{\mathrm{CSA}}static spectrally weighted CSA loss
Lrec\mathcal{L}_{\mathrm{rec}}reconstruction loss
Ldiff(θ;φ)\mathcal{L}_{\mathrm{diff}}(\theta;\varphi)diffusion training loss for encoder φ\varphi
F,F∗\mathcal{F},\mathcal{F}^{*}unitary DD-D Discrete Fourier Transform (DFT) and its conjugate transpose
d(k)d(k)normalized Euclidean FFT distance from DC, d(k)∈[0,1]d(k)\in[0,1]
w(k),w~(k),w‾w(k),\widetilde{w}(k),\overline{w}static spectral weight, mean-1-normalized weight, mean of ww
κ,p\kappa,pspectral hyperparameters: HF strength and distance exponent
λc\lambda_{c}weight on the spectral critic loss
MwM_{w}spectral mass operator F∗diag⁡(w~)F\mathcal{F}^{*}\operatorname{diag}(\widetilde{w})\mathcal{F}
NNtotal spatial DFT modes n1⋯nDn_{1}\cdots n_{D} (N=TN=T for the 1D motion setting of § 3.1)
HHLrec\mathcal{L}_{\mathrm{rec}} Hessian at the baseline φ0\varphi_{0}
ggstop-gradient pseudo-gradient of LCSA\mathcal{L}_{\mathrm{CSA}} at φ0\varphi_{0} (Assumption 18)
gcritg_{\mathrm{crit}}true gradient ∇φLdiff(θ0;φ0)\nabla_{\varphi}\mathcal{L}_{\mathrm{diff}}(\theta_{0};\varphi_{0}) at the baseline (not the stop-gradient pseudo-gradient)
S(k)S(k)per-mode power spectral density of the latent process
βeff(k,t)\beta_{\mathrm{eff}}(k,t)per-mode inverse effective temperature (Proposition 29)

Setup

Definition 2 (Stop-gradient operator). The stop-gradient sg⁡:RN→RN\operatorname{sg}:\mathbb{R}^{N}\to\mathbb{R}^{N} is the identity map on values: sg⁡(z)=z\operatorname{sg}(z)=z as elements of RN\mathbb{R}^{N}. In any expression where sg⁡(z(φ))\operatorname{sg}(z(\varphi)) appears, gradients with respect to φ\varphi treat sg⁡(z(φ))\operatorname{sg}(z(\varphi)) as a constant. Equivalently, introduce a frozen copy z‾=z\overline{z}=z with ∂z‾/∂φ≡0\partial\overline{z}/\partial\varphi\equiv0. Replace sg⁡(z(φ))\operatorname{sg}(z(\varphi)) by z‾\overline{z} in every gradient computation. Operationally, in PyTorch this is z.detach().

Let X=RC×H×W\mathcal{X}=\mathbb{R}^{C\times H\times W} and Z=Rd\mathcal{Z}=\mathbb{R}^{d} be the data and latent spaces. Set d=C′×n1×⋯×nDd=C'\times n_{1}\times\cdots\times n_{D} with D∈{1,2,3}D\in\{1,2,3\} for motion, image, and video respectively. An autoencoder is a pair (Eφ,Dψ0)(E_{\varphi},D_{\psi_{0}}) with text-agnostic encoder Eφ(x)E_{\varphi}(x) and a frozen pre-trained decoder Dψ0D_{\psi_{0}}. Data is sampled jointly (x,y)∼pdata(x,y)\sim p_{\mathrm{data}} where yy is the text prompt, with text marginal pyp_{y}. The conditional latent law is pz∣yφ:=(Eφ)#p(x∣y)p_{z\mid y}^{\varphi}:=(E_{\varphi})_{\#}p(x\mid y), and the marginal latent law is pzφ=(Eφ)#pdatap_{z}^{\varphi}=(E_{\varphi})_{\#}p_{\mathrm{data}}. The rectified-flow interpolant [67] is xt=(1−t)z0+tϵx_{t}=(1-t)z_{0}+t\epsilon with velocity v=ϵ−z0v=\epsilon-z_{0}. A text-conditioned velocity model fθ(xt,t,y)f_{\theta}(x_{t},t,y) is trained by

Ldiff(θ;φ)=E(x,y),t∼U[0,1],ϵ∼N(0,I)∥fθ((1−t)Eφ(x)+tϵ,t,y)−(ϵ−Eφ(x))∥2.(8)\mathcal{L}_{\mathrm{diff}}(\theta;\varphi)=\mathbb{E}_{(x,y),t\sim\mathcal{U}[0,1],\epsilon\sim\mathcal{N}(0,I)}\left\|f_{\theta}\left((1-t)E_{\varphi}(x)+t\epsilon,t,y\right)-\left(\epsilon-E_{\varphi}(x)\right)\right\|^{2}. \tag*{(8)}

Definition 3 (Plain critic loss). For z=Eφ(x)z=E_{\varphi}(x), sample (t,ϵ)(t,\epsilon) as above, set xt=(1−t)sg⁡(z)+tϵx_{t}=(1-t)\operatorname{sg}(z)+t\epsilon, and define the prediction residual r(z)=sg⁡(fθ(xt,t,y))−(ϵ−z)r(z)=\operatorname{sg}(f_{\theta}(x_{t},t,y))-(\epsilon-z). The plain critic loss is

Lcrit(φ;θ)=E(x,y),t,ϵ∥r(z)∥2.(9)\mathcal{L}_{\mathrm{crit}}(\varphi;\theta)=\mathbb{E}_{(x,y),t,\epsilon}\left\|r(z)\right\|^{2}. \tag*{(9)}

Lemma 4 (Value equivalence (values, not gradients)). For every φ\varphi and every frozen θ\theta, Lcrit(φ;θ)=Ldiff(θ;φ)\mathcal{L}_{\mathrm{crit}}(\varphi;\theta)=\mathcal{L}_{\mathrm{diff}}(\theta;\varphi) as values. Their gradients in φ\varphi differ under the stop-gradients (Remark 5, Assumption 18).

Proof. sg⁡\operatorname{sg} preserves values, so dropping sg⁡\operatorname{sg} from (9) gives

E(x,y),t,ϵ∥fθ((1−t)Eφ(x)+tϵ,t,y)−ϵ+Eφ(x)∥2,\mathbb{E}_{(x,y),t,\epsilon}\left\|f_{\theta}\left((1-t)E_{\varphi}(x)+t\epsilon,t,y\right)-\epsilon+E_{\varphi}(x)\right\|^{2},

which matches (8) with z0=Eφ(x)z_0 = E_\varphi(x). Equivalently, by the tower property E(x,y)=EyEx∣yE_{(x,y)} = E_y E_{x\mid y}, the same quantity reads Ey∼pyEz0∼pφ∣y,t,ϵ∥fθ(xt,t,y)−(ϵ−z0)∥2E_{y\sim p_y} E_{z_0\sim p_{\varphi\mid y},t,\epsilon}\left\|f_\theta(x_t,t,y)-(\epsilon-z_0)\right\|^2, the form used in conditional diffusion training.

Remark 5 (Stop-gradients: value equivalence and necessity of both stops). Lcrit\mathcal{L}_{\mathrm{crit}} and Ldiff\mathcal{L}_{\mathrm{diff}} agree as values (Lemma 4). Their gradients in φ\varphi differ under the stop-gradients. Autograd of Lcrit\mathcal{L}_{\mathrm{crit}} drops the Jacobian-of-fθ0f_{\theta_0} contribution that ∇φLdiff\nabla_\varphi\mathcal{L}_{\mathrm{diff}} retains. The pseudo-gradient gg (Assumption 18) is the autograd direction. The true gcrit=∇φLdiff(θ0;φ0)g_{\mathrm{crit}}=\nabla_\varphi\mathcal{L}_{\mathrm{diff}}(\theta_0;\varphi_0) is the Taylor coefficient in Theorem 25.

Both stops are required. Removing sg from vpredv_{\mathrm{pred}} adds a Jfθ0TJfθ0J_{f_{\theta_0}}^{\mathsf T}J_{f_{\theta_0}} term to the Hessian and breaks Theorem 10. Removing sg from xtx_t instead pushes the encoder to reshape the noised sample itself, a different objective.

FFT distance grid and CSA loss

Definition 6 (Normalized FFT distance grid). For spatial sizes (n1,…,nD)(n_1,\ldots,n_D) let fkℓ(ℓ)=kℓ/nℓf_{k_\ell}^{(\ell)}=k_\ell/n_\ell (mod 1) ∈[−1/2,1/2)\in[-1/2,1/2) in FFT layout. Define

d(k1,…,kD)=(∑ℓ(fkℓ(ℓ))2)1/2/dmax⁡,dmax⁡=max⁡k(∑ℓ(fkℓ(ℓ))2)1/2.d(k_1,\ldots,k_D)=\left(\sum_\ell\left(f_{k_\ell}^{(\ell)}\right)^2\right)^{1/2}/d_{\max},\qquad d_{\max}=\max_k\left(\sum_\ell\left(f_{k_\ell}^{(\ell)}\right)^2\right)^{1/2}.

Then d(0)=0d(0)=0 at DC, and max⁡kd(k)=1\max_k d(k)=1 by normalization. The maximum is attained at the Nyquist corner kℓ=nℓ/2k_\ell=n_\ell/2 when all nℓn_\ell are even. Moreover, d(kˉ)=d(k)d(\bar{k})=d(k) where kˉℓ:=nℓ−kℓ\bar{k}_\ell:=n_\ell-k_\ell is the per-axis reflection, since fnℓ−kℓ(ℓ)≡−fkℓ(ℓ)f_{n_\ell-k_\ell}^{(\ell)}\equiv-f_{k_\ell}^{(\ell)} (mod 1) in the FFT layout.

Definition 7 (CSA loss). With w(k)=1+κd(k)pw(k)=1+\kappa d(k)^p and w~(k)=w(k)/wˉ\widetilde{w}(k)=w(k)/\bar{w}, wˉ=N−1∑kw(k)\bar{w}=N^{-1}\sum_k w(k), and F\mathcal{F} the unitary DD-D DFT acting on the spatial axes,

LCSA(φ;θ,w)=E(x,y),t,ϵ[1N∑kw~(k)∣(Fr)(k)∣2].\mathcal{L}_{\mathrm{CSA}}(\varphi;\theta,w)=E_{(x,y),t,\epsilon}\left[\frac{1}{N}\sum_k\widetilde{w}(k)\left|(\mathcal{F}r)(k)\right|^2\right].

Here ∣(Fr)(k)∣2=∑c∣(Fr)(k,c)∣2\left|(\mathcal{F}r)(k)\right|^2=\sum_c\left|(\mathcal{F}r)(k,c)\right|^2 sums the squared magnitude over channels at each spatial mode kk, and N=∏ℓnℓN=\prod_\ell n_\ell is the number of spatial modes.

Lemma 8 (Parseval recovery). With κ=0\kappa=0, so w≡w~≡1w\equiv\widetilde{w}\equiv1, LCSA(φ;θ,1)=Lcrit(φ;θ)/N\mathcal{L}_{\mathrm{CSA}}(\varphi;\theta,1)=\mathcal{L}_{\mathrm{crit}}(\varphi;\theta)/N.

Proof. Plancherel for the unitary DFT gives ∑k∣(Fu)(k)∣2=∑iui2\sum_k|(\mathcal{F}u)(k)|^2=\sum_i u_i^2 for any u∈RNu\in\mathbb{R}^N. Apply per channel to u=ru=r.

Pointwise gradient, Hessian, strong convexity

Proposition 9 (Pointwise gradient). Fix (x,y,t,ϵ)(x,y,t,\epsilon) and let c=sg⁡(fθ(xt,t,y))c=\operatorname{sg}(f_\theta(x_t,t,y)), constant in zz under stop-gradient. Let Mw=F∗diag⁡(w~)F∈RN×NM_w=\mathcal{F}^{*}\operatorname{diag}(\widetilde{w})\mathcal{F}\in\mathbb{R}^{N\times N}. Then MwT=MwM_w^{\mathsf T}=M_w, MwM_w maps RN\mathbb{R}^N to RN\mathbb{R}^N, and

(i) ℓw(z):=1N∑kw~(k)∣(Fr(z))(k)∣2=1Nr(z)TMwr(z)\ell_w(z):=\frac{1}{N}\sum_k\widetilde{w}(k)|(\mathcal{F}r(z))(k)|^2=\frac{1}{N}r(z)^{\mathsf T}M_wr(z), with r(z)=c−ϵ+zr(z)=c-\epsilon+z.

(ii) ∇zℓw=2NMwr(z)=2NMw(z−(ϵ−c))\nabla_z\ell_w=\frac{2}{N}M_wr(z)=\frac{2}{N}M_w(z-(\epsilon-c)).

(iii) ∇z2ℓw=2NMw\nabla_z^2\ell_w=\frac{2}{N}M_w, constant in zz.

(iv) ∇φℓw=2NJEφ(x)TMwr(z)\nabla_\varphi\ell_w=\frac{2}{N}J_{E_\varphi(x)}^{\mathsf T}M_wr(z).

Proof. diag⁡(w~)\operatorname{diag}(\widetilde{w}) is real-diagonal. By Definition 6, d(kˉ)=d(k)d(\bar{k})=d(k), so w~(kˉ)=w~(k)\widetilde{w}(\bar{k})=\widetilde{w}(k) and w~\widetilde{w} is conjugate-symmetric in the FFT layout. For u∈RNu\in\mathbb{R}^N, u^:=Fu\hat{u}:=\mathcal{F}u satisfies the Nyquist symmetry u^(kˉ)=u^(k)‾\hat{u}(\bar{k})=\overline{\hat{u}(k)}. Then diag⁡(w~)u^\operatorname{diag}(\widetilde{w})\hat{u} inherits the symmetry and its inverse DFT is real, so Mwu∈RNM_wu\in\mathbb{R}^N. The unitary conjugation Mw=F∗diag⁡(w~)FM_w=\mathcal{F}^{*}\operatorname{diag}(\widetilde{w})\mathcal{F} of a real diagonal is Hermitian on CN\mathbb{C}^N. Restricting to RN\mathbb{R}^N gives MwT=MwM_w^{\mathsf T}=M_w, with spectrum {w~(k)}k\{\widetilde{w}(k)\}_k. For the multi-channel latent r∈RC′×Nr\in\mathbb{R}^{C'\times N}, the same operator applies along each channel with overall Hessian IC′⊗MwI_{C'}\otimes M_w.

For (i), ∑kw~(k)∣(Fr)(k)∣2=⟨Fr,diag⁡(w~)Fr⟩=rTMwr\sum_k\widetilde{w}(k)|(\mathcal{F}r)(k)|^2=\langle\mathcal{F}r,\operatorname{diag}(\widetilde{w})\mathcal{F}r\rangle=r^{\mathsf T}M_wr by unitarity. For (ii) and (iii), ∂r/∂z=I\partial r/\partial z=I and MwM_w is constant, so ∇z(rTMwr)/N=2Mwr/N\nabla_z(r^{\mathsf T}M_wr)/N=2M_wr/N and ∇z2ℓw=2Mw/N\nabla_z^2\ell_w=2M_w/N. For (iv), apply the chain rule at z=Eφ(x)z=E_\varphi(x).

Theorem 10 (Strong convexity of the detached surrogate in zz). Let Lw(z)=Et,ϵ[ℓw(z)]L_w(z)=\mathbb{E}_{t,\epsilon}[\ell_w(z)] at fixed (x,y)(x,y), where ℓw\ell_w is the pointwise loss of Proposition 9 with c=sg⁡(fθ(xt,t,y))c=\operatorname{sg}(f_\theta(x_t,t,y)) frozen via Definition 2. Then ∇z2Lw=2Mw/N\nabla_z^2 L_w=2M_w/N, and LwL_w is μ\mu-strongly convex and Λ\Lambda-smooth in zz with

μ=2N−1min⁡kw~(k),Λ=2N−1max⁡kw~(k).\mu=2N^{-1}\min_k\widetilde{w}(k),\qquad\Lambda=2N^{-1}\max_k\widetilde{w}(k).

For w(k)=1+κd(k)pw(k)=1+\kappa d(k)^p with κ≥0\kappa\ge0, w~>0\widetilde{w}>0, so μ>0\mu>0 and the global minimizer z∗z^* is unique. Gradient descent on LwL_w at fixed cc with η≤1/Λ\eta\le1/\Lambda contracts in the MwM_w-norm at rate 1−ημ1-\eta\mu.

Remark 11 (Strong convexity is a frozen-cc statement). Theorem 10 bounds the Hessian of the detached surrogate at fixed cc. Real training re-evaluates fθ0f_{\theta_0} at each step, so cc moves with zz and contributes a non-constant Jacobian term to the full Hessian. The contraction rate 1−ημ1-\eta\mu is therefore a property of the surrogate, not a guarantee on the full training dynamics. Training also acts on the encoder parameters φ\varphi, not on zz directly. The zz-space convexity at fixed (x,y,c)(x,y,c) does not transfer to a corresponding property in φ\varphi.

Proof. Proposition 9(iii) gives a constant Hessian at every (x,y,t,ϵ)(x,y,t,\epsilon), and linearity of expectation preserves it. MwM_w is the unitary conjugation of a positive diagonal, so its spectrum is exactly {w~(k)}k\{\widetilde{w}(k)\}_k, giving μ\mu and Λ\Lambda. Strict positivity of w~\widetilde{w} gives a unique minimizer. The contraction statement is the standard rate for quadratic objectives with constant Hessian in the MwM_w-norm.

Remark 12 (Fixed-point set is invariant under w~\widetilde{w}). z∗z^* minimizes LwL_w at fixed detached c=sg⁡(vpred)c=\operatorname{sg}(v_{\mathrm{pred}}) if and only if Et,ϵ[c−(ϵ−z∗)]=0\mathbb{E}_{t,\epsilon}[c-(\epsilon-z^*)]=0. This follows from setting ∇zLw=0\nabla_z L_w=0 in Proposition 9(ii) and inverting MwM_w. The fixed-point set is therefore the same under any positive weighting. Spectral weights steer the dynamics toward this set but do not change which z∗z^* are stationary. Plain critic and CSA share the fixed points. What differs is the gradient field at non-stationary zz. The practical benefit of CSA therefore arises on the way to convergence, not from changing the asymptotic optimum.

Theorem 13 (Per-frequency selectivity). Let r^=Fr\hat{r}=\mathcal{F}r. The CSA gradient in the Fourier domain is

(F∇zℓw)(k)=2Nw~(k)r^(k),(\mathcal{F}\nabla_z\ell_w)(k)=\frac{2}{N}\widetilde{w}(k)\hat{r}(k),

and the per-mode magnitude is 2w~(k)∣r^(k)∣/N2\widetilde{w}(k)|\hat{r}(k)|/N. The gradient at mode kk vanishes if and only if w~(k)=0\widetilde{w}(k)=0 or r^(k)=0\hat{r}(k)=0.

Proof. Apply F\mathcal{F} to Proposition 9(ii) and use unitarity FF∗=I\mathcal{F}\mathcal{F}^*=I, so FMw=diag⁡(w~)F\mathcal{F}M_w=\operatorname{diag}(\widetilde{w})\mathcal{F}.

Bayes-optimal sanity check

This subsection is stated for the marginal latent law pzφ0p_z^{\varphi_0} to keep the analysis self-contained. The conditional case follows by the same argument with pyEx∣yp_y\mathbb{E}_{x\mid y} replacing Ex\mathbb{E}_x and Δ(z,y)\Delta(z,y) replacing Δ(z)\Delta(z), but is not used by Theorem 1.

Remark 14 (Spring-force interpretation). The pointwise gradient ∇zℓw=2N−1Mw(z−(ϵ−vpred))\nabla_z\ell_w=2N^{-1}M_w(z-(\epsilon-v_{\mathrm{pred}})) acts as a spring pulling zz toward ϵ−vpred\epsilon-v_{\mathrm{pred}}, the diffusion’s implicit reconstruction of the clean code from xtx_t. Stiffness MwM_w upweights the modes selected by w~\widetilde{w}. The fixed point in Remark 12 is the configuration where encoder outputs and diffusion denoising agree on every mode.

Proposition 15 (Bayes-optimal sanity check). Let f∗(xt,t):=E[ϵ−z0∣xt,t]f^*(x_t,t):=\mathbb{E}[\epsilon-z_0\mid x_t,t] be the Bayes-optimal velocity predictor for pzφ0p_z^{\varphi_0}. Define the population residual

Δ(z):=Et,ϵ[f∗((1−t)z+te,t)−(ϵ−z)],\Delta(z):=\mathbb{E}_{t,\epsilon}[f^*((1-t)z+te,t)-(\epsilon-z)],

and the per-time slice

Δt(z):=Eϵ[f∗((1−t)z+te,t)−(ϵ−z)],\Delta_t(z):=\mathbb{E}_{\epsilon}[f^*((1-t)z+te,t)-(\epsilon-z)],

so Δ(z)=Et[Δt(z)]\Delta(z)=\mathbb{E}_t[\Delta_t(z)]. Then Ez∼pzφ0[Δ(z)]=0\mathbb{E}_{z\sim p_z^{\varphi_0}}[\Delta(z)]=0. Pointwise, Δ(z)≠0\Delta(z)\ne0 in general. For z∈supp⁡(pz)z\in\operatorname{supp}(p_z) with pz∈C2p_z\in C^2 and pz(z)>0p_z(z)>0, Δt(z)→0\Delta_t(z)\to0 as t→0+t\to0^+ at rate O(t)O(t), uniformly on compact subsets of supp⁡(pz)\operatorname{supp}(p_z).

The vanishing of Ez[Δ(z)]\mathbb{E}_z[\Delta(z)] is a sanity check on the residual: its mean over the training distribution is zero. It does not assert encoder-parameter stationarity. The actual encoder gradient at φ0\varphi_0 is, by Proposition 9(iv),

∇φLCSA(φ0;θ0,w)=2NEx[JEφ0(x)TMwΔ(Eφ0(x))],\nabla_\varphi\mathcal{L}_{\mathrm{CSA}}(\varphi_0;\theta_0,w)=\frac{2}{N}\mathbb{E}_x\left[J_{E_{\varphi_0}}(x)^{\mathsf{T}}M_w\Delta(E_{\varphi_0}(x))\right],

which need not equal zero even when Ez[Δ(z)]=0\mathbb{E}_z[\Delta(z)]=0, since the Jacobian-weighted expectation differs from the unweighted residual mean.

Proof. Population identity. Apply the tower property with respect to (xt,t)(x_t,t):

Ez,t,ϵ[f∗(xt,t)]=Ez,t,ϵ[E[ϵ−z∣xt,t]]=Ez,t,ϵ[ϵ−z].\mathbb{E}_{z,t,\epsilon}[f^*(x_t,t)] = \mathbb{E}_{z,t,\epsilon}[\mathbb{E}[\epsilon-z \mid x_t,t]] = \mathbb{E}_{z,t,\epsilon}[\epsilon-z].

This equals the right-hand side of E[ϵ−z]\mathbb{E}[\epsilon-z] inside Δ\Delta, so Ez[Δ(z)]=0\mathbb{E}_z[\Delta(z)] = 0.

Pointwise nonvanishing. At fixed zz and t>0t>0, f∗(xt,t)=E[ϵ′−z′∣xt,t]f^*(x_t,t)=\mathbb{E}[\epsilon'-z'\mid x_t,t] averages over all (z′,ϵ′)(z',\epsilon') pairs whose interpolant lands at xtx_t, not just (z,ϵ)(z,\epsilon). The posterior has nontrivial spread for t>0t>0, so Δt(z)≠0\Delta_t(z)\ne0 in general.

Slice-wise low-noise limit. Fix z∈supp⁡(pz)z\in\operatorname{supp}(p_z) with pz∈C2p_z\in C^2 and pz(z)>0p_z(z)>0. The Bayes posterior is p(z′∣xt,t)∝pz(z′)exp⁡(−∥xt−(1−t)z′∥2/(2t2))p(z'\mid x_t,t)\propto p_z(z')\exp\left(-\lVert x_t-(1-t)z'\rVert^2/(2t^2)\right), a Gaussian likelihood centered at xt/(1−t)x_t/(1-t) with covariance Σt=t2/(1−t)2I\Sigma_t=t^2/(1-t)^2I tilted by the prior. The Laplace expansion at small ∥Σt∥\lVert\Sigma_t\rVert gives

E[z′∣xt,t]=xt1−t+Σt∇log⁡pz(xt1−t)+O(∥Σt∥2),(10)\mathbb{E}[z'\mid x_t,t]=\frac{x_t}{1-t}+\Sigma_t\nabla\log p_z\left(\frac{x_t}{1-t}\right)+O(\lVert\Sigma_t\rVert^2), \tag*{(10)}

uniformly on compact K⊂supp⁡(pz)K\subset\operatorname{supp}(p_z) by pz∈C2(K)p_z\in C^2(K) and inf⁡Kpz>0\inf_Kp_z>0. Substituting xt=(1−t)z+tϵx_t=(1-t)z+t\epsilon, expanding xt/(1−t)=z+tϵ+t2ϵ+O(t3)x_t/(1-t)=z+t\epsilon+t^2\epsilon+O(t^3) and Σt=t2I+O(t3)\Sigma_t=t^2I+O(t^3), and Taylor of ∇log⁡pz\nabla\log p_z at zz gives E[z′∣xt,t]=z+tϵ+t2ϵ+t2∇log⁡pz(z)+O(t3)\mathbb{E}[z'\mid x_t,t]=z+t\epsilon+t^2\epsilon+t^2\nabla\log p_z(z)+O(t^3). Using ϵ′−z′=(xt−z′)/t\epsilon'-z'=(x_t-z')/t, f∗(xt,t)=−z−tϵ−t∇log⁡pz(z)+O(t2)f^*(x_t,t)=-z-t\epsilon-t\nabla\log p_z(z)+O(t^2). Averaging over ϵ\epsilon kills the tϵt\epsilon term, leaving

Δt(z)=−t∇log⁡pz(z)+O(t2),\Delta_t(z)=-t\nabla\log p_z(z)+O(t^2),

which is O(t)O(t) and vanishes pointwise on supp⁡(pz)\operatorname{supp}(p_z) as t→0+t\to0^+. For z∉supp⁡(pz)z\notin\operatorname{supp}(p_z), Δt(z)↛0\Delta_t(z)\not\to0 in general.

Information preservation and Pareto regime

Assumption 16. Lrec∈C3\mathcal{L}_{\mathrm{rec}}\in C^3 on a neighbourhood UU of φ0\varphi_0, with ∇φLrec(φ0)=0\nabla_\varphi\mathcal{L}_{\mathrm{rec}}(\varphi_0)=0 and Hessian H:=∇φ2Lrec(φ0)⪰μrecI≻0H:=\nabla_\varphi^2\mathcal{L}_{\mathrm{rec}}(\varphi_0)\succeq\mu_{\mathrm{rec}}I\succ0.

Remark 17 (Reading H≻0H\succ0 in over-parameterized networks). Strict positivity is a simplifying assumption. For over-parameterized encoders, Lrec\mathcal{L}_{\mathrm{rec}} has flat directions from reparameterization symmetries (permutation of hidden units, weight-norm rescaling, and similar things). The analysis should be read on the corresponding quotient, equivalently as H≻0H\succ0 restricted to the orthogonal complement of the symmetry-group tangent at φ0\varphi_0.

Assumption 18. Both LCSA(⋅;θ0,w)\mathcal{L}_{\mathrm{CSA}}(\cdot;\theta_0,w) and Lcrit(⋅;θ0)\mathcal{L}_{\mathrm{crit}}(\cdot;\theta_0) are C2C^2 on UU. Let g:=∇φLCSA(φ0;θ0,w)g:=\nabla_\varphi\mathcal{L}_{\mathrm{CSA}}(\varphi_0;\theta_0,w) denote the stop-gradient pseudo-gradient. It is computed by autograd with vpredv_{\mathrm{pred}} treated as constant in φ\varphi. Assume g≠0g\ne0.

Remark 19 ( gg is a vector field, not necessarily a scalar gradient). The pseudo-gradient gg is a vector field on Φ\Phi. The implicit function theorem (IFT) framework does not require gg to be the gradient of a scalar function. It only needs F(φ,λc)=∇φLrec(φ)+λc∇φLCSA(φ;θ0,w)F(\varphi,\lambda_c)=\nabla_\varphi\mathcal{L}_{\mathrm{rec}}(\varphi)+\lambda_c\nabla_\varphi\mathcal{L}_{\mathrm{CSA}}(\varphi;\theta_0,w) to be C1C^1 with invertible ∂φF\partial_\varphi F at (φ0,0)(\varphi_0,0). The trajectory φ∗(λc)\varphi^*(\lambda_c) in Theorem 20 is well-defined regardless.

Theorem 20 (First-order Pareto improvement). Under Assumptions 16 and 18, let φ∗(λc)\varphi^*(\lambda_c) be the local stationary point of ∇φLrec(φ)+λc∇φLCSA(φ;θ0,w)=0\nabla_\varphi\mathcal{L}_{\mathrm{rec}}(\varphi)+\lambda_c\nabla_\varphi\mathcal{L}_{\mathrm{CSA}}(\varphi;\theta_0,w)=0 near φ0\varphi_0. Throughout, gg denotes the stop-gradient pseudo-gradient of Assumption 18. For all sufficiently small λc>0\lambda_c>0:

(i) φ∗(λc)=φ0−λcH−1g+O(λc2)\varphi^*(\lambda_c)=\varphi_0-\lambda_cH^{-1}g+O(\lambda_c^2).

(ii) Lrec(φ∗)−Lrec(φ0)=λc22gTH−1g+O(λc3)\mathcal{L}_{\mathrm{rec}}(\varphi^*)-\mathcal{L}_{\mathrm{rec}}(\varphi_0)=\frac{\lambda_c^2}{2}g^\mathsf{T}H^{-1}g+O(\lambda_c^3).

(iii) gT(φ∗−φ0)=−λcgTH−1g+O(λc2)g^\mathsf{T}(\varphi^*-\varphi_0)=-\lambda_cg^\mathsf{T}H^{-1}g+O(\lambda_c^2).

Since gTH−1g>0g^\mathsf{T}H^{-1}g>0, the surrogate progress-to-cost ratio ∥gT(φ∗−φ0)∥/ΔLrec=2/λc+O(1)→∞\lVert g^\mathsf{T}(\varphi^*-\varphi_0)\rVert/\Delta\mathcal{L}_{\mathrm{rec}}=2/\lambda_c+O(1)\to\infty as λc→0+\lambda_c\to0^+.

Proof. Define F:Φ×R→ΦF:\Phi\times\mathbb{R}\to\Phi by

F(φ,λc):=∇φLrec(φ)+λc∇φLCSA(φ;θ0,w).F(\varphi,\lambda_c):=\nabla_\varphi\mathcal{L}_{\mathrm{rec}}(\varphi)+\lambda_c\nabla_\varphi\mathcal{L}_{\mathrm{CSA}}(\varphi;\theta_0,w).

FF is C1C^1 jointly in (φ,λc)(\varphi,\lambda_c) from Lrec∈C3\mathcal{L}_{\mathrm{rec}}\in C^3 (Assumption 16) and LCSA∈C2\mathcal{L}_{\mathrm{CSA}}\in C^2 (Assumption 18). Also F(φ0,0)=0F(\varphi_0,0)=0 and ∂φF(φ0,0)=H≻0\partial_\varphi F(\varphi_0,0)=H\succ0 by Assumption 16 (read on the symmetry quotient as in Remark 17). The IFT then applies. It gives open V⊂RV\subset\mathbb{R} around 00 and U′⊂UU'\subset U around φ0\varphi_0. For every λc∈V\lambda_c\in V, a unique φ∗(λc)∈U′\varphi^*(\lambda_c)\in U' satisfies F(φ∗(λc),λc)=0F(\varphi^*(\lambda_c),\lambda_c)=0. The map λc↦φ∗(λc)\lambda_c\mapsto\varphi^*(\lambda_c)

is C1C^1. Implicit differentiation at λc=0\lambda_c = 0 gives dϕ∗/dλc∣0=−H−1gd\phi^*/d\lambda_c|_0 = -H^{-1}g, hence ∥ϕ∗−ϕ0∥=O(λc)\|\phi^*-\phi_0\| = O(\lambda_c). To upgrade the remainder to O(λc2)O(\lambda_c^2), expand the gradients in the stationarity equation ∇ϕLrec(ϕ∗)+λc∇ϕLCSA(ϕ∗;θ0,w)=0\nabla_\phi L_{\mathrm{rec}}(\phi^*)+\lambda_c\nabla_\phi L_{\mathrm{CSA}}(\phi^*;\theta_0,w)=0 via Taylor at ϕ0\phi_0. Using ∇ϕLrec(ϕ0)=0\nabla_\phi L_{\mathrm{rec}}(\phi_0)=0, ∇ϕ2Lrec(ϕ0)=H\nabla_\phi^2L_{\mathrm{rec}}(\phi_0)=H, Lrec∈C3L_{\mathrm{rec}}\in C^3, and LCSA∈C2L_{\mathrm{CSA}}\in C^2,

H(ϕ∗−ϕ0)+λcg=O(∥ϕ∗−ϕ0∥2)+λcO(∥ϕ∗−ϕ0∥)=O(λc2),H(\phi^*-\phi_0)+\lambda_cg=O(\|\phi^*-\phi_0\|^2)+\lambda_cO(\|\phi^*-\phi_0\|)=O(\lambda_c^2),

where the last equality uses ∥ϕ∗−ϕ0∥=O(λc)\|\phi^*-\phi_0\|=O(\lambda_c). Inverting HH,

ϕ∗(λc)=ϕ0−λcH−1g+O(λc2),\phi^*(\lambda_c)=\phi_0-\lambda_cH^{-1}g+O(\lambda_c^2),

which is (i).

Taylor for LrecL_{\mathrm{rec}}. Using ∇ϕLrec(ϕ0)=0\nabla_\phi L_{\mathrm{rec}}(\phi_0)=0, ∇ϕ2Lrec(ϕ0)=H\nabla_\phi^2L_{\mathrm{rec}}(\phi_0)=H, and Lrec∈C3L_{\mathrm{rec}}\in C^3,

Lrec(ϕ∗)−Lrec(ϕ0)=12(ϕ∗−ϕ0)⊤H(ϕ∗−ϕ0)+O(∥ϕ∗−ϕ0∥3).L_{\mathrm{rec}}(\phi^*)-L_{\mathrm{rec}}(\phi_0)=\frac{1}{2}(\phi^*-\phi_0)^\top H(\phi^*-\phi_0)+O(\|\phi^*-\phi_0\|^3).

Substituting ϕ∗−ϕ0=−λcH−1g+O(λc2)\phi^*-\phi_0=-\lambda_cH^{-1}g+O(\lambda_c^2) and using HH−1=IHH^{-1}=I on the quadratic term gives (ϕ∗−ϕ0)⊤H(ϕ∗−ϕ0)=λc2g⊤H−1g+O(λc3)(\phi^*-\phi_0)^\top H(\phi^*-\phi_0)=\lambda_c^2g^\top H^{-1}g+O(\lambda_c^3), and ∥ϕ∗−ϕ0∥3=O(λc3)\|\phi^*-\phi_0\|^3=O(\lambda_c^3). Combining, (ii) holds with

∣Lrec(ϕ∗)−Lrec(ϕ0)−λc22g⊤H−1g∣≤Crecλc3.\left|L_{\mathrm{rec}}(\phi^*)-L_{\mathrm{rec}}(\phi_0)-\frac{\lambda_c^2}{2}g^\top H^{-1}g\right|\le C_{\mathrm{rec}}\lambda_c^3.

Surrogate pairing for (iii). Taking the inner product of ϕ∗−ϕ0=−λcH−1g+O(λc2)\phi^*-\phi_0=-\lambda_cH^{-1}g+O(\lambda_c^2) with gg gives (iii):

∣g⊤(ϕ∗−ϕ0)+λcg⊤H−1g∣≤CCSAλc2.\left|g^\top(\phi^*-\phi_0)+\lambda_cg^\top H^{-1}g\right|\le C_{\mathrm{CSA}}\lambda_c^2.

The constant Crec>0C_{\mathrm{rec}}>0 depends on third derivatives of LrecL_{\mathrm{rec}} on a closed ball Bˉ(ϕ0,r)⊂U′\bar{B}(\phi_0,r)\subset U' that contains ϕ∗(λc)\phi^*(\lambda_c) for all λc∈[0,λcmax⁡]\lambda_c\in[0,\lambda_c^{\max}], and CCSA>0C_{\mathrm{CSA}}>0 tracks the implicit constant in the trajectory expansion. H≻0H\succ0 and g≠0g\ne0 give g⊤H−1g>0g^\top H^{-1}g>0, and the surrogate progress-to-cost ratio follows from dividing (iii) by (ii).

Remark 21 (Reading of (iii)). Statement (iii) is the first-order pairing of the CSA direction gg with the trajectory displacement ϕ∗−ϕ0\phi^*-\phi_0. This is the surrogate progress that training reduces by construction. The corresponding decrease in LcritL_{\mathrm{crit}} is bridged in Theorem 25 through the alignment assumption Assumption 23. Lemma 4 then carries it over to LdiffL_{\mathrm{diff}}.

Diffusion loss improvement

Assumption 22. θ0=arg⁡min⁡θLdiff(θ;ϕ0)\theta_0=\arg\min_\theta L_{\mathrm{diff}}(\theta;\phi_0). The baseline diffusion model is optimal for the baseline encoder.

Assumption 23 (Critic-CSA gradient alignment). Let g:=∇ϕLCSA(ϕ0;θ0,w)g:=\nabla_\phi L_{\mathrm{CSA}}(\phi_0;\theta_0,w) be the stop-gradient pseudo-gradient that training optimizes. It is the gradient computed by autograd with sg⁡(vpred)\operatorname{sg}(v_{\mathrm{pred}}) treated as a constant in ϕ\phi, as in Proposition 9. Let gcrit:=∇ϕLdiff(θ0;ϕ0)g_{\mathrm{crit}}:=\nabla_\phi L_{\mathrm{diff}}(\theta_0;\phi_0) be the true gradient of the conditional diffusion loss with respect to the encoder. We assume

gcrit⊤H−1g>0.g_{\mathrm{crit}}^\top H^{-1}g>0.

The descent direction −H−1g-H^{-1}g in Theorem 20 comes from the stop-gradient pseudo-gradient. This is what training actually follows. The true gradient gcritg_{\mathrm{crit}} of the conditional diffusion loss is a different object. It includes a term involving the Jacobian of fθ0f_{\theta_0} that the stop-gradient drops from the training signal. Assumption 23 says that this dropped term does not flip the sign at ϕ0\phi_0. So the descent direction used by training also descends the true Ldiff(θ0;⋅)L_{\mathrm{diff}}(\theta_0;\cdot) at ϕ0\phi_0. We treat Assumption 23 as a working assumption. Empirically the reconstruction quality stays flat while the downstream generation quality improves on Motion and ImageNet, consistent with the Pareto regime predicted by the analysis.

Remark 24 (Decomposition of the alignment condition). Write δ:=gcrit−g\delta:=g_{\mathrm{crit}}-g. By construction δ\delta collects the Jacobian term that sg⁡\operatorname{sg} removes from the training signal. It also captures the structural difference between LdiffL_{\mathrm{diff}} and LCSAL_{\mathrm{CSA}}. Then

gcrit⊤H−1g=∥g∥H−12+δ⊤H−1g.g_{\mathrm{crit}}^\top H^{-1}g=\|g\|_{H^{-1}}^2+\delta^\top H^{-1}g.

The first term is non-negative and vanishes only at g=0g=0. So Assumption 23 is equivalent to δ⊤H−1g>−∥g∥H−12\delta^\top H^{-1}g>-\|g\|_{H^{-1}}^2. A simple sufficient (but not necessary) condition is δ⊤H−1g≥0\delta^\top H^{-1}g\ge0. A more testable condition is ∥δ∥H−1≤c∥g∥H−1\lVert\delta\rVert_{H^{-1}} \le c\lVert g\rVert_{H^{-1}} with c<1c < 1. Cauchy–Schwarz then gives ∣δTH−1g∣≤c∥g∥H−12|\delta^{\mathsf{T}}H^{-1}g| \le c\lVert g\rVert_{H^{-1}}^{2}. In standard diffusion architectures fθ0f_{\theta_0} is locally Lipschitz with ∥Jfθ0∥op≤L\lVert J_{f_{\theta_0}}\rVert_{\mathrm{op}} \le L on supp⁡(pxt,t)\operatorname{supp}(p_{x_t,t}). This gives the per-instance bound ∥δ∥≤2(1−t)L∥JEϕ∥op∥r∥\lVert\delta\rVert\le2(1-t)L\lVert J_{E_\phi}\rVert_{\mathrm{op}}\lVert r\rVert. The same baseline-residual scale appears in ∥g∥\lVert g\rVert, so ∥δ∥/∥g∥\lVert\delta\rVert/\lVert g\rVert stays bounded. Assumption 23 reduces to a non-extreme-anti-alignment condition. The ratio ∥δ∥/∥g∥\lVert\delta\rVert/\lVert g\rVert can be probed directly during fine-tuning. To probe the Euclidean alignment between gg and gcritg_{\mathrm{crit}}, we compared their minibatch estimates on 128 single-image batches from ImageNet [50, 51] across two seeds, using the baseline encoder and a frozen critic. Their Euclidean inner product was positive on every batch (mean cosine 0.93), and ∥δ∥2/∥g∥2\lVert\delta\rVert_2/\lVert g\rVert_2 was below 1 throughout (mean 0.40).

Theorem 25 (Conditional diffusion-loss improvement under gradient alignment; formal form of Theorem 1). Under Assumptions 16, 18, 22 and 23 and λc>0\lambda_c > 0 sufficiently small,

min⁡θLdiff(θ;ϕ∗(λc))<min⁡θLdiff(θ;ϕ0).\min_{\theta} \mathcal{L}_{\mathrm{diff}}(\theta;\phi^{*}(\lambda_c)) < \min_{\theta} \mathcal{L}_{\mathrm{diff}}(\theta;\phi_0).

Proof. Four-step chain.

  1. θ0\theta_0 is fixed (the baseline of Assumption 22). By definition of the minimum, min⁡θ′Ldiff(θ′;ϕ∗)≤Ldiff(θ0;ϕ∗)\min_{\theta'}\mathcal{L}_{\mathrm{diff}}(\theta';\phi^{*}) \le\mathcal{L}_{\mathrm{diff}}(\theta_0;\phi^{*}).

  2. By Lemma 4, Ldiff(θ0;ϕ∗)=Lcrit(ϕ∗;θ0)\mathcal{L}_{\mathrm{diff}}(\theta_0;\phi^{*}) = \mathcal{L}_{\mathrm{crit}}(\phi^{*};\theta_0).

  3. The IFT direction is the pseudo-gradient of LCSA\mathcal{L}_{\mathrm{CSA}}. From Theorem 20, ϕ∗(λc)−ϕ0=−λcH−1g+O(λc2)\phi^{*}(\lambda_c)-\phi_0=-\lambda_c H^{-1}g+O(\lambda_c^2) with g=∇ϕLCSA(ϕ0;θ0,w)g=\nabla_\phi\mathcal{L}_{\mathrm{CSA}}(\phi_0;\theta_0,w) from Assumption 18. This is what training follows, and we Taylor-expand Lcrit\mathcal{L}_{\mathrm{crit}}, not LCSA\mathcal{L}_{\mathrm{CSA}}. By Lemma 4, Lcrit(⋅;θ0)\mathcal{L}_{\mathrm{crit}}(\cdot;\theta_0) and Ldiff(θ0;⋅)\mathcal{L}_{\mathrm{diff}}(\theta_0;\cdot) agree as value functions of ϕ\phi. Their true gradients at ϕ0\phi_0 both equal gcrit=∇ϕLdiff(θ0;ϕ0)g_{\mathrm{crit}}=\nabla_\phi\mathcal{L}_{\mathrm{diff}}(\theta_0;\phi_0). Substituting,

Lcrit(ϕ∗;θ0)−Lcrit(ϕ0;θ0)=−λcgcritTH−1g+O(λc2).\mathcal{L}_{\mathrm{crit}}(\phi^{*};\theta_0)-\mathcal{L}_{\mathrm{crit}}(\phi_0;\theta_0)=-\lambda_c g_{\mathrm{crit}}^{\mathsf{T}}H^{-1}g+O(\lambda_c^2).

The form is mixed. gg comes from LCSA\mathcal{L}_{\mathrm{CSA}} as the training trajectory (Theorem 20). gcritg_{\mathrm{crit}} comes from Lcrit≡Ldiff\mathcal{L}_{\mathrm{crit}}\equiv\mathcal{L}_{\mathrm{diff}} in value as the Taylor coefficient (Lemma 4). LCSA\mathcal{L}_{\mathrm{CSA}} never enters the Taylor expansion. Assumption 23 assumes gcritTH−1g>0g_{\mathrm{crit}}^{\mathsf{T}}H^{-1}g > 0. The difference is then strictly negative for all sufficiently small λc>0\lambda_c > 0.

  1. By Lemma 4 and Assumption 22, Lcrit(ϕ0;θ0)=min⁡θLdiff(θ;ϕ0)\mathcal{L}_{\mathrm{crit}}(\phi_0;\theta_0)=\min_\theta\mathcal{L}_{\mathrm{diff}}(\theta;\phi_0).

Concatenating gives the strict inequality. □

Corollary 26 (Fixed-θ0\theta_0 improvement). Under Assumptions 16, 18 and 23 and λc>0\lambda_c > 0 sufficiently small,

Ldiff(θ0;ϕ∗)<Ldiff(θ0;ϕ0).\mathcal{L}_{\mathrm{diff}}(\theta_0;\phi^{*}) < \mathcal{L}_{\mathrm{diff}}(\theta_0;\phi_0).

Steps (i)–(iii) of the proof of Theorem 25 do not invoke Assumption 22 and give this directly.

Remark 27 (Fine-tuning inherits the bound). At fixed θ0\theta_0, the diffusion loss strictly decreases on the post-trained latents. No retraining of θ\theta is required. Theorem 25 adds Assumption 22 to upgrade this to a min-over-θ\theta statement. Fine-tuning θ\theta from θ0\theta_0 on ϕ∗\phi^{*} can only decrease the loss further. The practical fine-tune setting therefore inherits the strict bound either way.

Remark 28 (Approximate baseline optimality). If Ldiff(θ0;ϕ0)=min⁡θLdiff(θ;ϕ0)+ϵopt\mathcal{L}_{\mathrm{diff}}(\theta_0;\phi_0)=\min_\theta\mathcal{L}_{\mathrm{diff}}(\theta;\phi_0)+\epsilon_{\mathrm{opt}}, then Theorem 25 relaxes to

min⁡θLdiff(θ;ϕ∗)≤min⁡θLdiff(θ;ϕ0)+ϵopt−λcgcritTH−1g+O(λc2).\min_\theta\mathcal{L}_{\mathrm{diff}}(\theta;\phi^{*}) \le\min_\theta\mathcal{L}_{\mathrm{diff}}(\theta;\phi_0)+\epsilon_{\mathrm{opt}}-\lambda_c g_{\mathrm{crit}}^{\mathsf{T}}H^{-1}g+O(\lambda_c^2).

The improvement persists as long as λcgcritTH−1g>ϵopt\lambda_c g_{\mathrm{crit}}^{\mathsf{T}}H^{-1}g>\epsilon_{\mathrm{opt}}.

Frequency-dependent effective temperature (DenseAM extension)

Pham et al. [68] identify the empirical diffusion marginal

p(xt,t)=1K∑μ(2πσ2t)−N/2exp⁡(−∥xt−ξμ∥2/(2σ2t)).p(x_t,t)=\frac{1}{K}\sum_{\mu}(2\pi\sigma^2t)^{-N/2}\exp\left(-\lVert x_t-\xi_\mu\rVert^2/(2\sigma^2t)\right).

with a Dense Associative Memory at scalar inverse temperature β(t)=1/(2σ2t)\beta(t)=1/(2\sigma^2t). We extend this identification to a per-mode effective temperature under spatial stationarity.

Proposition 29 (Per-mode effective temperature). Assume the memories {ξμ}μ=1K\{\xi_\mu\}_{\mu=1}^{K} form a temporally wide-sense stationary (WSS) process: constant mean E[ξμ(s)]=μ0\mathbb{E}[\xi_\mu(s)] = \mu_0 for all ss, and autocovariance Cov⁡(ξμ(s),ξμ(t))\operatorname{Cov}(\xi_\mu(s),\xi_\mu(t)) depending only on the lag ∣s−t∣\lvert s-t\rvert. By Wiener–Khinchin, the population power spectral density (PSD) is the temporal Fourier transform of the autocovariance:

S(k):=F[Cov⁡(ξμ(0),ξμ(⋅))](k).S(k) := \mathcal{F}\left[\operatorname{Cov}(\xi_\mu(0),\xi_\mu(\cdot))\right](k).

The empirical estimator S^(k):=K−1∑μ=1K∣(Fξμ)(k)∣2\widehat{S}(k) := K^{-1}\sum_{\mu=1}^{K}\lvert(\mathcal{F}\xi_\mu)(k)\rvert^2 converges to S(k)S(k) as K→∞K \to\infty under ergodicity, with O(K−1/2)O(K^{-1/2}) deviation at finite KK.

Because F\mathcal{F} is unitary, Parseval gives ∥xt−ξμ∥2=∑k∣x^t,k−ξ^μ,k∣2\lVert x_t-\xi_\mu\rVert^2=\sum_k\lvert\widehat{x}_{t,k}-\widehat{\xi}_{\mu,k}\rvert^2, and each Boltzmann factor splits across modes:

p(xt,t)=1K∑μ(2πσt2)−N/2∏kexp⁡(−∣x^t,k−ξ^μ,k∣22σt2).p(x_t,t)=\frac{1}{K}\sum_\mu(2\pi\sigma_t^2)^{-N/2}\prod_k\exp\left(-\frac{\lvert\widehat{x}_{t,k}-\widehat{\xi}_{\mu,k}\rvert^2}{2\sigma_t^2}\right).

The sum over μ\mu does not factor across kk. WSS gives second-order spectral uncorrelatedness across modes, not joint independence. We treat modes as independent as a working approximation. The per-mode signal and noise energies are then Esig(k)=S(k)E_{\mathrm{sig}}(k)=S(k) and Enoise(k,t)=σt2E_{\mathrm{noise}}(k,t)=\sigma^2_t, giving a per-mode SNR:

βeff(k,t)∝Esig(k)/Enoise(k,t)=S(k)/(σ2t).(11)\beta_{\mathrm{eff}}(k,t)\propto E_{\mathrm{sig}}(k)/E_{\mathrm{noise}}(k,t)=S(k)/(\sigma^2t). \tag*{(11)}

Proof. Apply the scalar identification of Pham et al. [68] per mode to each Boltzmann factor above. The per-mode signal energy scales as S(k)S(k) by the Wiener–Khinchin definition.

Remark 30 (Reading of Proposition 29). The mode-independence step is a working approximation, not an exact identity. WSS gives only second-order uncorrelatedness across modes. Joint independence is a separate assumption that we adopt for tractability. The polynomial weight w(k)=1+κd(k)pw(k)=1+\kappa d(k)^p should be read as motivated by this approximation, not derived as a unique identity. The empirical spectral profile in Figure 1a supports this choice in our deployment regime.

Remark 31 (Static weight as SNR compensation). For a power-law PSD S(k)∝∥k∥−γS(k)\propto\lVert k\rVert^{-\gamma}, the SNR-compensation choice from (11) is w~(k)∝βeff−1∝∥k∥γ\widetilde{w}(k)\propto\beta_{\mathrm{eff}}^{-1}\propto\lVert k\rVert^\gamma. This makes w~(k)βeff(k,t)\widetilde{w}(k)\beta_{\mathrm{eff}}(k,t) approximately constant across kk. The encoder pressure 2Nw~(k)∣r^(k)∣\frac{2}{N}\widetilde{w}(k)\lvert\widehat{r}(k)\rvert from Theorem 13 then applies more force at modes with smaller per-mode SNR. For D>1D>1 the same construction applies axis-wise. The additive 1 in w(k)=1+κd(k)pw(k)=1+\kappa d(k)^p does not come from (11). It keeps w~(k)>0\widetilde{w}(k)>0, which Theorem 10 uses. It also keeps a baseline pressure on low-frequency modes.

Practical considerations for text-to-motion deployment

The setup in Appendix A.2 is already text-conditioned. This subsection collects practical remarks for deployment.

Remark 32 (Scope of the improvement). Theorem 25 states min⁡θLdiff(θ;φ∗)<min⁡θLdiff(θ;φ0)\min_\theta\mathcal{L}_{\mathrm{diff}}(\theta;\varphi^*)<\min_\theta\mathcal{L}_{\mathrm{diff}}(\theta;\varphi_0) where Ldiff\mathcal{L}_{\mathrm{diff}} is the joint-expectation loss E(x,y),t,ϵ[Ldiffy]\mathbb{E}_{(x,y),t,\epsilon}[\mathcal{L}_{\mathrm{diff}}^y] (equivalently Ey∼py[Ldiffy]\mathbb{E}_{y\sim p_y}[\mathcal{L}_{\mathrm{diff}}^y] via tower property, with Ldiffy\mathcal{L}_{\mathrm{diff}}^y the per-prompt loss). The result is an min-over-θ\theta statement about the prompt-averaged loss. It does not say a fixed θ\theta achieves this drop, nor does it say each fixed prompt yy improves. Text alignment, R-Precision, FID, and related metrics depend on the full sampler and decoder pipeline, so the theorem does not predict them.

Remark 33 (Classifier-free guidance and approximate optimality). Pretrained text-to-motion diffusion models are usually trained with classifier-free guidance (CFG) [69]. With probability puncondp_{\mathrm{uncond}} the text yy is replaced by a null token during training, so the actual training objective is a mixture of conditional and unconditional losses. Hence θ0\theta_0 does not exactly minimize the conditional Ldiff(θ;φ0)\mathcal{L}_{\mathrm{diff}}(\theta;\varphi_0) in Assumption 22. Remark 28 covers this case. The improvement of Theorem 25 persists when the conditional optimality gap is small relative to λcgcrit⊤H−1g\lambda_c g_{\mathrm{crit}}^\top H^{-1}g. CSA uses the raw conditional prediction fθ0(xt,t,y)f_{\theta_0}(x_t,t,y), not the CFG-guided combination used at sampling time.

Remark 34 (Encoder stays text-free). The conditioning on yy comes from the data distribution p(x∣y)p(x\mid y), not from the encoder. The map Eφ(x)E_\varphi(x) takes only motion. Different texts induce different latent distributions because they select different motions, and the critic gradient picks up the text signal through fθ0(⋅,⋅,y)f_{\theta_0}(\cdot,\cdot,y).

ImageNet generalization study

We apply the CSA pipeline to class-conditional ImageNet [50, 51]. The autoencoder is the Stable Diffusion VAE (SD-VAE) backbone [16], fine-tuned with CSA at κ=1.0\kappa= 1.0, λc=0.1\lambda_c = 0.1, p=2.0p = 2.0, for 5,000 fine-tune steps with 1,000-step warmup on λc\lambda_c, learning rate 10−610^{-6}, batch size 8. The diffusion model is a 200M-parameter class-conditional UNet at 256×256256 \times256 resolution. The architecture has channels (192,384,576,768)(192, 384, 576, 768), three layers per block, dropout 0.1, and 1,001 class embeddings. We train from scratch on the 1.28M ImageNet training images for 300,000 steps at batch size 128. Peak learning rate is 10−410^{-4} with linear warmup over 5,000 steps followed by cosine decay. We use exponential moving average (EMA) decay 0.9999 and CFG dropout probability 0.10. Reconstruction FID is measured on 50,000 validation images. Generation FID uses 50 samples per class generated via 50-step deterministic sampling at CFG scale w=2.0w = 2.0. We use a single seed (42).

Table 4 reports the paired baseline-vs-CSA result. Three observations stand out. Reconstruction FID stays close to baseline (Δ=+0.08\Delta= +0.08), and generation FID improves at all three checkpoints we evaluate (Δ=−2.24\Delta= -2.24 at 100,000 training steps, Δ=−1.69\Delta= -1.69 at 200,000 training steps, and Δ=−1.35\Delta= -1.35 at 300,000 training steps). Moreover, the diffusion training loss curves separate by about 0.30 from step 1,000 onward (Figure 5). Taken together, all three are consistent with the prediction of Theorem 1. The post-trained latent is easier for the diffusion to fit, and reconstruction is preserved at first order in λc\lambda_c.

MetricBaselineCSAΔ\Delta
Reconstruction FID2.792.87+0.08
Generation FID at 100k steps36.2934.05−2.24-2.24
Generation FID at 200k steps25.4623.77−1.69-1.69
Generation FID at 300k steps20.8219.47−1.35-1.35

Table 4. ImageNet generalization study. Single seed (42), 200M class-conditional UNet, CFG scale w=2.0w = 2.0. Reconstruction FID is computed on 50,000 validation images. Generation FID is computed from 50,000 CFG samples (50 per class).

Diffusion training loss on class-conditional ImageNet latents

Figure 5. Diffusion training loss on class-conditional ImageNet latents. CSA-fine-tuned latents (orange) yield consistently lower training loss than the SD-VAE baseline (blue) from step 1,000 onward. Faded curves show raw step-wise loss. Solid curves show 0.999-EMA-smoothed loss. The shaded band marks the sustained gap of about 0.30.

MeanFlow Identity Derivation

This appendix derives the JVP form of the average velocity (5) used in § 3.2 from the integral definition (4), following Geng et al. [37].

Recall the average velocity definition:

u(xt,rd,t,z)=1t−rd∫rdtv(xτ,τ,z) dτ.(12)u(x_t,r_d,t,z) = \frac{1}{t-r_d}\int_{r_d}^{t} v(x_\tau,\tau,z)\,\mathrm{d}\tau. \tag*{(12)}

Multiplying both sides by (t−rd)(t-r_d) removes the prefactor and gives an integral identity:

(t−rd)u(xt,rd,t,z)=∫rdtv(xτ,τ,z) dτ.(13)(t-r_d)u(x_t,r_d,t,z) = \int_{r_d}^{t} v(x_\tau,\tau,z)\,\mathrm{d}\tau. \tag*{(13)}

We now differentiate both sides of (13) with respect to tt, treating rdr_d as independent of tt. The right-hand side reduces by the fundamental theorem of calculus to

ddt∫rdtv(xτ,τ,z) dτ=v(xt,t,z).(14)\frac{\mathrm{d}}{\mathrm{d}t}\int_{r_d}^{t} v(x_\tau,\tau,z)\,\mathrm{d}\tau= v(x_t,t,z). \tag*{(14)}

The left-hand side expands by the product rule:

ddt[(t−rd)u(xt,rd,t,z)]=u(xt,rd,t,z)+(t−rd)ddtu(xt,rd,t,z).(15)\frac{\mathrm{d}}{\mathrm{d}t}\left[(t-r_d)u(x_t,r_d,t,z)\right] = u(x_t,r_d,t,z) + (t-r_d)\frac{\mathrm{d}}{\mathrm{d}t}u(x_t,r_d,t,z). \tag*{(15)}

Equating (14) and (15) and rearranging recovers the MeanFlow identity:

u(xt,rd,t,z)=v(xt,t,z)−(t−rd)ddtu(xt,rd,t,z).(16)u(x_t,r_d,t,z) = v(x_t,t,z) - (t-r_d)\frac{\mathrm{d}}{\mathrm{d}t}u(x_t,r_d,t,z). \tag*{(16)}

This matches (5) in the main text.

At training time, the total derivative ddtu(xt,rd,t,z)\frac{\mathrm{d}}{\mathrm{d}t}u(x_t,r_d,t,z) is computed via the Jacobian-vector product (JVP) of the decoder network with the tangent vector (v(xt,t,z),0,1,0)(v(x_t,t,z), 0, 1, 0) along the (x,rd,t,z)(x,r_d,t,z) axes. The instance of uu inside the time derivative is detached, denoted uψ−u_{\psi^-} in (5), to prevent recursive backpropagation through the network’s own prediction [37]. Without this stop-gradient, the loss would induce double backpropagation through the JVP and require higher-order optimization.

Generation Metric

Frechet Inception Distance (FID) The FID [70] measures the distribution distance between the generated and real interaction features.

FID=∥μgt−μpred∥2+Tr⁡(Σgt+Σpred−2(ΣgtΣpred)12)\mathrm{FID} = \lVert\mu_{gt}-\mu_{pred}\rVert^2 + \operatorname{Tr}\left(\Sigma_{gt}+\Sigma_{pred}-2(\Sigma_{gt}\Sigma_{pred})^{\frac{1}{2}}\right)

where μgt\mu_{gt} and μpred\mu_{pred} are the mean ground-truth and generated interaction features, and Σ\Sigma represents the covariance matrix.

Multimodal Distance (MM-Dist) This metric calculates the average Euclidean distance between each text feature and the generated interaction feature.

MM-Dist=1N∑i=1N∥ft,i−fm,i∥\mathrm{MM\text{-}Dist} = \frac{1}{N}\sum_{i=1}^{N}\lVert f_{t,i}-f_{m,i}\rVert

where ft,if_{t,i} and fm,if_{m,i} are the features of the iith text-interaction pair.

Diversity All generated interactions are randomly sampled to calculate the average Euclidean distance between two subsets.

Diversity=1Xd∑i=1Xd∥xi−xi′∥\mathrm{Diversity} = \frac{1}{X_d}\sum_{i=1}^{X_d}\lVert x_i-x'_i\rVert

Multimodality (MModality) This metric assesses the variability given multiple text descriptions by calculating the average pairwise Euclidean distance between motion features.

MModality=1Jm×Xm∑j=1Jm∑i=1Xm∥xj,i−xj,i′∥\mathrm{MModality} = \frac{1}{J_m \times X_m}\sum_{j=1}^{J_m}\sum_{i=1}^{X_m}\lVert x_{j,i}-x'_{j,i}\rVert

where xj,ix_{j,i} and xj,i′x'_{j,i} are the features of the jjth pair of the iith text description.

Implementation Details

Diffusion backbone Training. In DiMoT, we train the diffusion backbone for different tokenizers in a fixed way. Firstly, each motion sequence is encoded into a set of 16 latent tokens with a channel dimension of 32, and the diffusion backbone denoises them conditioned on text. The denoiser is implemented as a transformer with an encoder-only topology. Each latent token is first projected to a 512-dimensional hidden space, processed by an 11-layer skip-connected Transformer encoder with 8 attention heads, a feed-forward dimension of 1024, GELU activations, and dropout of 0.1, and then projected back to the original latent dimension. We use learned 1D positional embeddings for the latent token sequence. Conditioning is injected by prepending two condition tokens to the latent sequence: (1) a timestep token obtained from a sinusoidal timestep embedding followed by an MLP projection, and (2) a text token produced by a frozen CLIP text encoder followed by a 2-layer Transformer Encoder for feature refinement. Thus, the denoiser processes a joint sequence consisting of 16 motion latent tokens, one timestep token, and one global text token. Notably, the Transformer backbone is not a plain stacked encoder: it adopts a skip-encoder design, in which hidden states from the early encoder blocks are concatenated with their symmetric late blocks via learnable linear fusion, thereby improving cross-layer information propagation while preserving global self-attention. By default, the model is trained with a flow-matching velocity prediction objective and sampled with classifier-free guidance at guidance scale 3.0 in 10 steps. The training process uses a batch size of 192, a peak learning rate of 1e−41\mathrm{e}{-4}, a 1K-iteration linear warm-up, and a cosine decay schedule. In the CSA stage, we use the same denoiser backbone, which remains frozen.

Tokenizer Training. We reproduce MLD++ [65] and adopt it as our baseline with multi-latent-token training. The baseline tokenizer is trained for 6K epochs with a peak learning rate of 1×10−41 \times10^{-4}, a batch size of 96, and a cosine decay schedule following a 1K-epoch linear warmup. During the CSA stage, we train the encoder for 1K epochs and gradually introduce spectral reweighting over a 200-epoch warmup. The encoder is optimized with an ℓ2\ell_2 reconstruction objective, and the CSA loss is weighted by λc=0.1\lambda_c = 0.1. We evaluate CSA under both training-from-scratch and baseline fine-tuning settings. For the fine-tuning variant, we additionally study an encoder-side adapter implemented as a 2-layer residual block. The spectral hyperparameters are set to κ=1.0\kappa= 1.0 and p=2.0p = 2.0. The Critic is pretrained on the baseline tokenizer’s latents and remains frozen throughout CSA. After CSA, we freeze the adapted encoder and train a separate generative diffusion model from scratch on its latents, using the same architecture and training configuration as the baseline diffusion model. This generative model supplies diffusion-generated latent conditions for AMD training. In detail, we replace deterministic decoding with a latent-conditioned generative decoder. The decoder uses a 9-layer Transformer [71], while the latent-to-frame upsampling projection is implemented as a 2-layer Transformer. We train the AMD decoder from scratch for 2K epochs, where latent conditions are sampled from the pretrained diffusion output with probability 0.25 and from the encoder output with probability 0.75. For 50% of each batch, we apply marginal flow-matching supervision by setting t=rdt = r_d. For the remaining samples, we use an α\alpha-curriculum that linearly decreases the generative mixing factor from 1.0 to 0.005 over the first 1K epochs and then sets it to 0 for the final 1K epochs. We further apply single-step classifier-free guidance [37] with guidance scale 3.

Compute resources. All experiments were conducted on the High-Performance Computing cluster. We use a Linux server equipped with a 30-core Intel Xeon 7677P CPU, 235 GiB of system memory, and an NVIDIA H200 GPU.

AMD ablation

VariantProj.StepTargetα\alpha Sched. (Ratio)FID↓\downarrowR-Top3↑\uparrowMPJPE↓\downarrowFoot Sliding↓\downarrowAITS↓\downarrow
MLD++ (Baseline)––––5.0460.68849.260.01861.05
Target vv Flow Matching Velocity Baseline
V1: + Interpo.Interpolation1 NFEvvNone (0%)7.6250.55166.120.02553.05
V2: + Trans.Transformer16 NFEvvNone (0%)4.9510.70847.120.018145.16
V3: + Trans.Transformer1 NFEvvNone (0%)6.8420.58958.450.02214.82
Target uu Projection Search
V4: InterpolationInterpolation1 NFEuuNone (0%)5.2150.67549.850.01913.13
V5: MLPMLP1 NFEuuNone (0%)5.0120.69447.530.01783.64
Target uu + Marginal Distribution ratio t=rt = r
V6: Ratio 25%Transformer1 NFEuuAnnealed (25%)4.8320.71046.900.01834.80
V7: Ratio 50% (Default)Transformer1 NFEuuAnnealed (50%)4.7810.71646.760.01754.80
V8: Ratio 75%Transformer1 NFEuuAnnealed (75%)4.8050.71446.820.01764.80

Table 5. AMD ablation.

CSA ablation

Figure 6. Spectral Reweighting Curve.

Methodκ\kappappλc\lambda_crFID ↓\downarrowMPJPE ↓\downarrowFID ↓\downarrowR-Precision ↑\uparrow
Baseline--0.00.08049.265.0460.688
Plain Critic0.0-0.10.08248.505.0120.708
CSA0.51.00.10.08447.854.9850.728
CSA (Default)1.02.00.10.08647.184.9670.746
CSA2.02.00.10.12551.325.2150.712
CSA1.03.00.10.18858.455.6820.665
CSA1.04.00.10.24566.805.9320.621
CSA1.02.00.010.08148.605.0350.698
CSA1.02.00.50.14254.205.4100.685

Table 6. Ablation of CSA Spectral Weights (κ\kappa, pp) and Critic Scale (λc\lambda_c)

Effect of the Alpha Curriculum Schedule

We compare the default linear curriculum with an AlphaFlow-style sigmoid curriculum [52], keeping the remaining AMD settings unchanged. AMD is trained for 2K epochs. For samples with non-degenerate intervals (rd<tr_d < t), both schedules decrease α\alpha from 1 to αmin⁡=0.005\alpha_{\min} = 0.005 over the first 1K epochs, followed by the exact MeanFlow objective (α=0\alpha= 0) for the final 1K epochs. The degenerate-interval ratio remains 50% for both schedules.

Let ee denote training progress in epochs and q=e/1000q = e/1000. For schedule SS, we use

αS(e)={αmin⁡+(1−αmin⁡)gS(q),0≤e≤1000,0,1000<e≤2000.(17)\alpha_S(e) = \begin{cases} \alpha_{\min} + (1-\alpha_{\min})g_S(q), & 0 \le e \le1000, \\ 0, & 1000 < e \le2000. \end{cases} \tag*{(17)}

The linear schedule uses glin(q)=1−qg_{\mathrm{lin}}(q) = 1-q. For the sigmoid schedule, with σ(a)=(1+exp⁡(−a))−1\sigma(a) = (1+\exp(-a))^{-1} and γ=25\gamma= 25, we use

gsig(q)=σ(γ/2)−σ(γ(q−1/2))σ(γ/2)−σ(−γ/2).(18)g_{\mathrm{sig}}(q) = \frac{\sigma(\gamma/2)-\sigma(\gamma(q-1/2))} {\sigma(\gamma/2)-\sigma(-\gamma/2)}. \tag*{(18)}

This normalization gives the same initial and final α\alpha values for both curricula.

ScheduleFID↓\downarrowR-Top3↑\uparrowMPJPE↓\downarrowFoot Sliding↓\downarrow
Linear (default)4.7810.71646.760.0175
Sigmoid (γ=25\gamma= 25)4.8560.68749.520.0178

Table 7. Comparison of α\alpha curriculum schedules. Both schedules use the same training duration and endpoint values.

The linear schedule performs better on all four reported metrics in this comparison. The sigmoid schedule concentrates the transition around the middle of the first training stage, whereas the linear schedule changes α\alpha at a constant rate. These results support the default linear curriculum under the evaluated setting.

User Study against InterMask

We conduct a user study comparing DiMoT with InterMask [64]. The study involves 11 participants and 20 randomly selected samples. We evaluate preferences along four dimensions: text alignment, physical realism, interaction quality, and overall quality.

CriterionInterMaskDiMoT
Text alignment53.33%46.67%
Physical realism46.15%53.85%
Interaction quality43.80%56.20%
Overall quality47.39%52.61%

Table 8. User preferences for InterMask and DiMoT. Higher percentages indicate greater preference. Ties are excluded from the overall-quality comparison.

DiMoT receives higher preference percentages for physical realism, interaction quality, and overall quality, while InterMask receives higher preference for text alignment. The overall-quality difference is modest, and the study is limited in participant and sample size. We therefore interpret these results as descriptive evidence of user preference in this comparison, rather than as a statistically established advantage.

References

  1. [1]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186, 2019.arxiv.org/abs/1810.04805
  2. [2]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020.arxiv.org/abs/1910.10683
  3. [3]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.arxiv.org/abs/2302.13971
  4. [4]Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025.arxiv.org/abs/2503.20314
  5. [5]Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuyue Gu, et al. Language model beats diffusion–tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023.arxiv.org/abs/2310.05737
  6. [6]Dan Kondratyuk, Lijun Yu, Xiuyue Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125, 2023.
  7. [7]Wei-Jin Huang, Yue-Yi Zhang, Yi-Lin Wei, Zhi-Wei Xia, Juantao Tan, Yuan-Ming Li, Zhilin Zhao, and Wei-Shi Zheng. Learning whole-body human-humanoid interaction from human-human demonstrations. arXiv preprint arXiv:2601.09518, 2026.arxiv.org/abs/2601.09518
  8. [8]Zipeng Fu, Qingqing Zhao, Qi Wu, Gordon Wetzstein, and Chelsea Finn. Humanplus: Humanoid shadowing and imitation from humans. arXiv preprint arXiv:2406.10454, 2024.arxiv.org/abs/2406.10454
  9. [9]Tairan He, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Omnih2o: Universal and dexterous human-to-humanoid whole-body teleoperation and learning. arXiv preprint arXiv:2406.08858, 2024.arxiv.org/abs/2406.08858
  10. [10]Tairan He, Jiajwei Gao, Wenli Xiao, Yuanhang Zhang, Zi Wang, Jiashun Wang, Zhengyi Luo, Guanqi He, Nikhil Sobanbawa, Chaoyi Pan, et al. Asap: Aligning simulation and real-world physics for learning agile humanoid whole-body skills. arXiv preprint arXiv:2502.01143, 2025.arxiv.org/abs/2502.01143
  11. [11]Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, and Jitendra Malik. Humans in 4D: Reconstructing and tracking humans with transformers. In ICCV, 2023.arxiv.org/abs/2305.20091
  12. [12]Jiaxi Jiang, Paul Streli, Huajian Qiu, Andreas Fender, Larissa Laich, Patrick Snape, and Christian Holz. Avatarposer: Articulated full-body pose tracking from sparse motion sensing. In European conference on computer vision, pages 443–460. Springer, 2022.arxiv.org/abs/2207.13784
  13. [13]Yuming Du, Robin Kips, Albert Pumarola, Sebastian Starke, Ali Thabet, and Artsiom Sanakoyeu. Avatars grow legs: Generating smooth human motion from sparse tracking inputs with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 481–490, 2023.arxiv.org/abs/2304.08577
  14. [14]Jiaman Li, Karen Liu, and Jiajun Wu. Ego-body pose estimation via ego-head pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17142–17151, 2023.
  15. [15]Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023.arxiv.org/abs/2311.15127
  16. [16]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.arxiv.org/abs/2112.10752
  17. [17]Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479–36494, 2022.arxiv.org/abs/2205.11487
  18. [18]Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.arxiv.org/abs/2204.06125
  19. [19]Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.arxiv.org/abs/2307.01952
  20. [20]William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023.arxiv.org/abs/2212.09748
  21. [21]Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18000–18010, 2023.arxiv.org/abs/2212.04048
  22. [22]Wenxun Dai, Ling-Hao Chen, Jingbo Wang, Jinpeng Liu, Bo Dai, and Yansong Tang. Motionlcm: Real-time controllable motion generation via latent consistency model. arXiv preprint arXiv:2404.19759, 2024.arxiv.org/abs/2404.19759
  23. [23]Lei Jiang, Ye Wei, and Hao Ni. Motionpcm: Real-time motion synthesis with phased consistency model. arXiv preprint arXiv:2501.19083, 2025.arxiv.org/abs/2501.19083
  24. [24]Peng Jin, Yang Wu, Yanbo Fan, Zhongqian Sun, Wei Yang, and Li Yuan. Act as you wish: Fine-grained control of motion diffusion model with hierarchical semantic graphs. Advances in Neural Information Processing Systems, 36, 2024.arxiv.org/abs/2311.01015
  25. [25]Longbin Ji, Pengfei Wei, Yi Ren, Jinglin Liu, Chen Zhang, and Xiang Yin. C2g2: Controllable co-speech gesture generation with latent diffusion model. arXiv preprint arXiv:2308.15016, 2023.arxiv.org/abs/2308.15016
  26. [26]Zichen Geng, Zeeshan Hayder, Bo Miao, Jian Liu, Wei Liu, and Ajmal Mian. Disentangled hierarchical vae for 3d human-human interaction generation. arXiv preprint arXiv:2603.00144, 2026.arxiv.org/abs/2603.00144
  27. [27]Guanhe Huang and Oya Celiktutan. Unified number-free text-to-motion generation via flow matching. arXiv preprint arXiv:2603.27040, 2026.arxiv.org/abs/2603.27040
  28. [28]Boyuan Li, Xihua Wang, Ruihua Song, and Wenbing Huang. Two-in-one: Unified multi-person interactive motion generation by latent diffusion transformer. arXiv preprint arXiv:2412.16670, 2024.arxiv.org/abs/2412.16670
  29. [29]Jungbin Cho, Junwan Kim, Jisoo Kim, Minseo Kim, Mingu Kang, Sungeun Hong, Tae-Hyun Oh, and Youngjae Yu. Discord: Discrete tokens to continuous motion via rectified flow decoding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14602–14612, 2025.arxiv.org/abs/2411.19527
  30. [30]Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 15703–15712, 2025.arxiv.org/abs/2501.01423
  31. [31]Ivan Skorokhodov, Sharath Girish, Benran Hu, Willi Menapace, Yanyu Li, Rameen Abdal, Sergey Tulyakov, and Aliaksandr Siarohin. Improving the diffusability of autoencoders. arXiv preprint arXiv:2502.14831, 2025.arxiv.org/abs/2502.14831
  32. [32]Shizhan Liu, Xinran Deng, Zhuoyi Yang, Jiayan Teng, Xiaotao Gu, and Jie Tang. Delving into latent spectral biasing of video vaes for superior diffusability. arXiv preprint arXiv:2512.05394, 2025.arxiv.org/abs/2512.05394
  33. [33]Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.arxiv.org/abs/1312.6114
  34. [34]Hanyu Wang, Saksham Suri, Yixuan Ren, Hao Chen, and Abhinav Shrivastava. Larp: Tokenizing videos with a learned autoregressive generative prior. arXiv preprint arXiv:2410.21264, 2024.arxiv.org/abs/2410.21264
  35. [35]Chenyang Gu, Mingyuan Zhang, Haozhe Xie, Zhongang Cai, Lei Yang, and Ziwei Liu. Bridging semantic and kinematic conditions with diffusion-based discrete motion tokenizer. arXiv preprint arXiv:2603.19227, 2026.arxiv.org/abs/2603.19227
  36. [36]Zhengyang Geng, Yiyang Lu, Zongze Wu, Eli Shechtman, J Zico Kolter, and Kaiming He. Improved mean flows: On the challenges of fastforward generative models. arXiv preprint arXiv:2512.02012, 2025.arxiv.org/abs/2512.02012
  37. [37]Zhengyang Geng, Mingyang Deng, Xingjian Bai, J Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling. arXiv preprint arXiv:2505.13447, 2025.arxiv.org/abs/2505.13447
  38. [38]Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng. Repa-e: Unlocking vae for end-to-end tuning of latent diffusion transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18262–18272, 2025.arxiv.org/abs/2504.10483
  39. [39]Yinbo Chen, Rohit Girdhar, Xiaolong Wang, Sai Saketh Rambhatla, and Ishan Misra. Diffusion autoencoders are scalable image tokenizers. arXiv preprint arXiv:2501.18593, 2025.arxiv.org/abs/2501.18593
  40. [40]Mang Ning, Mingxiao Li, Le Zhang, Lanmiao Liu, Matthew B Blaschko, Albert Ali Salah, and Itir Onal Ertugrul. Spectrum matching: a unified perspective for superior diffusability in latent diffusion. arXiv preprint arXiv:2603.14645, 2026.arxiv.org/abs/2603.14645
  41. [41]Long Zhao, Sanghyun Woo, Ziyu Wan, Yandong Li, Han Zhang, Boqing Gong, Hartwig Adam, Xuhui Jia, and Ting Liu. Epsilon-vae: Denoising as visual decoding. arXiv preprint arXiv:2410.04081, 2024.
  42. [42]Bolin Lai, Xudong Wang, Saketh Rambhatla, James M Rehg, Zsolt Kira, Rohit Girdhar, and Ishan Misra. Toward diffusible high-dimensional latent spaces: A frequency perspective. arXiv preprint arXiv:2511.22249, 2025.arxiv.org/abs/2511.22249
  43. [43]Guy Tevet, Sigal Raab, Setareh Cohan, Daniele Reda, Zhengyi Luo, Xue Bin Peng, Amit H Bermano, and Michiel van de Panne. Closd: Closing the loop between simulation and diffusion for multi-task character control. arXiv preprint arXiv:2410.03441, 2024.arxiv.org/abs/2410.03441
  44. [44]Linnan Tu, Lingwei Meng, Zongyi Li, Hefei Ling, and Shijuan Huang. Autoregressive motion generation with gaussian mixture-guided latent sampling. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.DOI
  45. [45]Yaoqin Ye, Yiteng Xu, Qin Sun, Xinge Zhu, Yujing Sun, and Yuexin Ma. Remogen: Real-time human interaction-to-reaction generation via modular learning from diverse data. arXiv preprint arXiv:2604.01082, 2026.arxiv.org/abs/2604.01082
  46. [46]Liang Pan, Zeshi Yang, Zhiyang Dou, Wenjia Wang, Buzhen Huang, Bo Dai, Taku Komura, and Jingbo Wang. Tokenhsi: Unified synthesis of physical human-scene interactions through task tokenization. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 5379–5391, 2025.arxiv.org/abs/2503.19901
  47. [47]Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988, 2022.arxiv.org/abs/2209.14988
  48. [48]Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International conference on artificial intelligence and statistics, pages 1540–1552. PMLR, 2020.arxiv.org/abs/1911.02590
  49. [49]Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31, 2018.arxiv.org/abs/1810.04650
  50. [50]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE, 2009.
  51. [51]Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–252, 2015. doi: 10.1007/s11263-015-0816-y.DOI
  52. [52]Huijie Zhang, Aliaksandr Siarohin, Willi Menapace, Michael Vasilkovsky, Sergey Tulyakov, Qing Qu, and Ivan Skorokhodov. Alphaflow: Understanding and improving meanflow models. arXiv preprint arXiv:2510.20771, 2025.arxiv.org/abs/2510.20771
  53. [53]Patrick Esser, Smith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.arxiv.org/abs/2403.03206
  54. [54]Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu. Intergen: Diffusion-based multi-human motion generation under complex interactions. International Journal of Computer Vision, pages 1–21, 2024.DOI
  55. [55]Liang Xu, Xintao Lv, Yichao Yan, Xin Jin, Shuwen Wu, Congsheng Xu, Yifan Liu, Yizhou Zhou, Fengyun Rao, Xingdong Sheng, et al. Inter-x: Towards versatile human-human interaction analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22260–22271, 2024.arxiv.org/abs/2312.16051
  56. [56]Chuan Guo, Shihаo Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5152–5161, 2022.DOI
  57. [57]Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1–248:16, October 2015.
  58. [58]Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 10975–10985, 2019.arxiv.org/abs/1904.05866
  59. [59]Mathis Petrovich, Michael J Black, and Gü l Varol. Temos: Generating diverse human motions from textual descriptions. arXiv preprint arXiv:2204.14109, 2022.arxiv.org/abs/2204.14109
  60. [60]Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209.14916, 2022.arxiv.org/abs/2209.14916
  61. [61]Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Bermano. Human motion diffusion as a generative prior. arXiv preprint arXiv:2303.01418, 2023.arxiv.org/abs/2303.01418
  62. [62]Yabiao Wang, Shuo Wang, Jiangning Zhang, Ke Fan, Jiafu Wu, Zhucun Xue, and Yong Liu. Timotion: Temporal and interactive framework for efficient human-human motion generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 7169–7178, 2025.arxiv.org/abs/2408.17135
  63. [63]Pablo Ruiz Ponce, German Barquero, Cristina Palmero, Sergio Escalera, and Jose Garcia-Rodriguez. in2in: Leveraging individual information to generate human interactions. arXiv preprint arXiv:2404.09988, 2024.
  64. [64]Muhammad Gohar Javed, Chuan Guo, Li Cheng, and Xingyu Li. Intermask: 3d human interaction generation via collaborative masked modelling. arXiv preprint arXiv:2410.10010, 2024.
  65. [65]Wenxun Dai, Ling-Hao Chen, Yufei Huo, Jingbo Wang, Jinpeng Liu, Bo Dai, and Yansong Tang. Real-time controllable motion generation via latent consistency model.
  66. [66]Manuel Kaufmann, Velko Vechev, and Dario Mylonopoulos. aitviewer, 7 2022. URL https://github.com/eth-ait/aitviewer.DOI
  67. [67]Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022.arxiv.org/abs/2209.03003
  68. [68]Bao Pham, Gabriel Raya, Matteo Negri, Mohammed J Zaki, Luca Ambrogioni, and Dmitry Krotov. Memorization to generalization: Emergence of diffusion models from associative memory. arXiv preprint arXiv:2505.21777, 2025.
  69. [69]Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.arxiv.org/abs/2207.12598
  70. [70]Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
  71. [71]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1706.03762

Paper details

vlnc:W_dptp6mmg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents