betaFeedback

WorksComputer Science

Modaliff: A Unified Token Diffusion Model for Any-to-Any Generation

Abstract

Any-to-any generation seeks a single model that translates flexibly among text, image, and audio, yet existing approaches either require fully aligned multimodal tuples or a shared representation that suppresses modality-specific structure. We introduce Modaliff, a novel token-based diffusion model that learns from widely available bimodal pairs while preserving each modality in its native latent sequence. Modaliff assembles conditions and targets into a unified token sequence, with per-token attributes to disambiguate token identities and role-based positional embedding to preserve each modality's internal structure across sequence layouts. This design enables a single denoiser to support diverse generation tasks without task-specific architectural changes or a shared cross-modal latent space. Beyond one-to-one generation, Modaliff supports multiple conditions either by combining single-condition predictions at inference or by lightweight finetuning on captions that jointly describe multiple inputs. Experiments across text, image, and audio show that Modaliff achieves the best fidelity among any-to-any models on most generation tasks while using substantially fewer parameters and training data.

Introduction

A long-standing goal of generative modeling is to build a single model that translates freely among text, image, and audio, such as captioning a photograph, imagining the scene behind a sound, or synthesizing audio from a description. Recent diffusion and flow-based models have brought this any-to-any goal closer [3, 39, 20], building on progress in text-to-image [32] and text-to-audio [23] generation. Yet flexible and efficient any-to-any generation remains challenging because different modalities have fundamentally different structures, with text is sequential and symbolic, image is spatially structured, and audio is temporally continuous.

Existing any-to-any diffusion models typically follow one of two approaches. The first approach, exemplified by UniDiffuser [3] and OmniFlow [20], concatenates tokens from all modalities and model their joint density over concatenated multimodal tokens with a single transformer. This design is general, but it often requires fully aligned multimodal tuples, which are scarce in practice, and is less flexible for conditional tasks such as Image + Text →\to Image. The second, represented by CoDi [39] and FlowBind [4], instead maps all modalities

into a shared representation space and condition generation there. This reduces the tuple requirement, but it either requires contrastive pretraining or forces all modalities into the same vector format, limiting representation expressivity.

We introduce Modaliff, a novel token-based diffusion model for any-to-any generation that learns from widely available bimodal pairs without requiring fully aligned multimodal tuples or a shared cross-modal latent space. Our key idea is to represent each modality as a latent token sequence and assemble condition and target modalities into a unified token sequence across generation tasks. To this end, each token of the sequence carries a rich attribute tuple that specifies its modality (e.g., text, image, or audio), role (source or target), and noise level. A role-based positional embedding further organizes token positions across task-varying sequence layouts. Together, these components allow a single denoiser to support flexible source and target combinations without architectural modification, while preserving expressive conditioning through attention over native token sequences.

Beyond one-to-one generation, Modaliff supports multiple conditions through two complementary mechanisms, namely inference-time score composition and lightweight caption-based adaptation. Score composition adopts a product-of-experts formulation to combine the single-condition predictions learned from paired data, enabling multi-condition generation without additional training. In contrast, caption-based adaptation expresses the joint semantics of multiple conditions in language and finetunes Modaliff on a compact set of examples constructed from existing paired data, better capturing interactions among conditions.

Extensive experiments across text, image, and audio demonstrate that Modaliff achieves the best fidelity among any-to-any models on most generation tasks. Compared to OmniFlow and CoDi, Modaliff uses six to eight times fewer parameters and about 5% of OmniFlow’s training data while maintaining competitive results.

Our contributions are as follows:

  • We propose Modaliff, an any-to-any diffusion model that learns from bimodal paired data while preserving each modality in its native latent token space. To our knowledge, Modaliff is the first any-to-any model that requires neither a shared cross-modal representation nor fully aligned multimodal tuples for training.

  • We develop two approaches for multi-condition gen-

eration through inference-time score composition and lightweight adaptation on caption-based supervision.

  • Across text, image, and audio, Modaliff achieves the best fidelity among any-to-any models on most generation tasks while using substantially fewer parameters and less training data than OmniFlow and CoDi.

Background

Diffusion Models

In diffusion models [34, 36] (Ho, Jain, and Abbeel 2020; Song, Meng, and Ermon 2021), a clean sample xx is progressively corrupted by Gaussian noise through a predefined process:

xt=atx+σtϵ,ϵ∼N(0,I)(1)x_t = a_t x + \sigma_t \epsilon,\quad\epsilon\sim\mathcal{N}(0,I) \tag*{(1)}

where ata_t decreases from 1 to 0 and σt\sigma_t increases from 0 to 1 as tt goes from 0 to TT.

The generative (reverse) process starts from xT∼N(0,I)x_T \sim\mathcal{N}(0,I) and iteratively denoises using pθ(xt−1∣xt)p_\theta(x_{t-1}|x_t) – a Gaussian transition whose mean depends on the network ϵθ\epsilon_\theta that predicts the noise. The model is trained with the denoising objective:

LDM(θ)=Ex,t,ϵ[∥ϵθ(xt,t)−ϵ∥22](2)\mathcal{L}_{\mathrm{DM}}(\theta) = \mathbb{E}_{x,t,\epsilon}\left[\left\|\epsilon_\theta(x_t,t)-\epsilon\right\|_2^2\right] \tag*{(2)}

For conditional generation, classifier-free guidance (CFG) is usually applied:

ϵ~θ(xt,t,y)=wϵθ(xt,t,y)+(1−w)ϵθ(xt,t,∅)(3)\tilde{\epsilon}_\theta(x_t,t,y) = w\epsilon_\theta(x_t,t,y) + (1-w)\epsilon_\theta(x_t,t,\varnothing) \tag*{(3)}

where ϵθ(xt,t,y)\epsilon_\theta(x_t,t,y) and ϵθ(xt,t,∅)\epsilon_\theta(x_t,t,\varnothing) are the conditional and unconditional noise predictions, and ww controls the guidance.

Diffusion Router

Diffusion Router (DR) [13] is a variant of diffusion models designed for the Universal Multi-Domain Translation (UMDT) problem. Given K−1K-1 paired datasets

Dk,c={(xk,xc)n}n=1Nk\mathcal{D}_{k,c} = \left\{(x^k,x^c)_n\right\}_{n=1}^{N_k}

between a central domain XcX^c and K−1K-1 non-central domains XkX^k (k≠ck \ne c), DR trains a unified noise predictor ϵθ(xttgt,t,xsrc,tgt,src)\epsilon_\theta(x_t^{\mathrm{tgt}},t,x^{\mathrm{src}},\mathrm{tgt},\mathrm{src}) which supports bidirectional central-noncentral translations Xc↔XkX^c \leftrightarrow X^k using the objective:

Lpair(θ)=E(xk,xc)∼Dk,c,t,ϵ,ζ[ζ∥ϵθ(xtk,t,xc,k,c)−ϵ∥22+(1−ζ)∥ϵθ(xtc,t,xk,c,k)−ϵ∥22](4)\mathcal{L}_{\mathrm{pair}}(\theta) = \mathbb{E}_{(x^k,x^c)\sim\mathcal{D}_{k,c},t,\epsilon,\zeta}\left[\zeta\left\|\epsilon_\theta(x_t^k,t,x^c,k,c)-\epsilon\right\|_2^2 +(1-\zeta)\left\|\epsilon_\theta(x_t^c,t,x^k,c,k)-\epsilon\right\|_2^2\right] \tag*{(4)}

At inference, DR can translate between two non-central domains XiX^i, XjX^j indirectly via the central domain XcX^c. To enable direct translation, an additional objective is introduced:

Lunpair(θ)=Extj∼p(xtj∣xc)[∥ϵθ(xtj,t,xi,j,i)−sg⁡(ϵθ(xtj,t,xc,j,c))∥22](5)\mathcal{L}_{\mathrm{unpair}}(\theta) = \mathbb{E}_{x_t^j\sim p(x_t^j|x^c)}\left[\left\|\epsilon_\theta(x_t^j,t,x^i,j,i)-\operatorname{sg}\left(\epsilon_\theta(x_t^j,t,x^c,j,c)\right)\right\|_2^2\right] \tag*{(5)}

where sg⁡(⋅)\operatorname{sg}(\cdot) denotes the stop gradient operator. The final training objective is:

Lfinal(θ)=λ1Lunpair(θ)+λ2Lpair(θ)(6)\mathcal{L}_{\mathrm{final}}(\theta) = \lambda_1\mathcal{L}_{\mathrm{unpair}}(\theta) + \lambda_2\mathcal{L}_{\mathrm{pair}}(\theta) \tag*{(6)}

Despite promising results, DR is primarily designed for image-to-image translation, assuming all domains share the same dimensionality. This limits its applicability to multimodal settings, where domains may differ in representation and structure. In addition, DR focuses on pairwise translation and does not naturally support composing multiple conditions for generation.

Modaliff

Unified Token-based Representation

A fundamental challenge in multimodal generation is that latent representations of different modalities differ substantially in both structure and semantic abstraction. Existing unified diffusion and flow models typically address this heterogeneity either by modeling a joint distribution over concatenated multimodal tokens [3, 20] or by mapping modalities into a shared representation space [39, 4]. The former often requires fully aligned multimodal datasets [20] and struggles to support flexible conditioning patterns such as Image + Text →\rightarrow Image, while the latter typically discards modality-specific structure and restricts cross-modal interaction to operations between fixed-dimensional vectors.

To address these limitations, we represent each modality as its native latent token sequence and concatenate arbitrary condition and target modalities into a unified sequence. Specifically, let M={1,…,K}\mathcal{M} = \{1,\ldots,K\} denote the modality set, instantiated with text, image, and audio, following prior multimodal settings [4]. For each modality i∈Mi \in\mathcal{M}, a frozen encoder Enc⁡i(⋅)\operatorname{Enc}^i(\cdot) encodes a sample xix^i to a token sequence, zi=Enc⁡i(xi)z^i = \operatorname{Enc}^i(x^i), zi∈RNi×diz^i \in\mathbb{R}^{N_i \times d_i}, where NiN_i and did_i denote the number and dimensionality of tokens. Given a generation task, let C,T⊆M\mathcal{C},\mathcal{T}\subseteq\mathcal{M} denote the condition and target modality sets, respectively. We concatenate the token sequences of the context and target set as:

Ht=H({zi}i∈C,{ztj}j∈T)=Concat⁡({zi⋅Wini}i∈C,{ztj⋅Winj}j∈T),(7)\begin{aligned} H_t = \mathcal{H}\left(\left\{z^i\right\}_{i\in\mathcal{C}},\left\{z_t^j\right\}_{j\in\mathcal{T}}\right) \\ &=\operatorname{Concat}\left(\left\{z^i\cdot W_{\mathrm{in}}^i\right\}_{i\in\mathcal{C}},\left\{z_t^j\cdot W_{\mathrm{in}}^j\right\}_{j\in\mathcal{T}}\right), \tag*{(7)} \end{aligned}

where Wini∈Rdi×dW_{\mathrm{in}}^i\in\mathbb{R}^{d_i\times d} is a trainable projection matrix. Here, each condition modality i∈Ci\in\mathcal{C} contributes clean sequence ziz^i, while each target modality j∈Tj\in\mathcal{T} contributes a noisy sequence ztjz_t^j at diffusion timestep tt, following Eq. (1). This design reduces flexible generation to rearranging condition and target segments, while the shared network and modality-native representations remain unchanged.

Processing all tasks through a unified sequence introduces two challenges. First, HtH_t does not specify each token’s modality, condition or target status, or diffusion timestep, leaving the generation task ambiguous to the shared model. Second, the sequence length and segment layout vary across tasks, making conventional positional indexing unsuitable. Global indexing over HtH_t [3] binds modalities to a fixed segment order, limiting flexibility, whereas local indexing avoids this constraint but creates artificial correspondences between unrelated tokens sharing the same index, introducing false positional biases into the model.

To resolve the first challenge, we define an attribute tuple (r,i,t)(r, i, t) for each token, encoding its generation role, modality type, and diffusion timestep. Specifically, r∈{0,1}r \in\{0, 1\}, where r=0r = 0 denotes a condition token and r=1r = 1 denotes a target token, i∈Mi \in\mathcal{M} identifies the modality type, and tt records its noise level, with t=0t = 0 for clean condition tokens and tt set to the current denoising timestep for target tokens. These attributes are embedded and injected into each transformer block through a token-wise AdaLN (Peebles and Xie [27]), which produces token-specific scale, shift, and gating parameters, enabling the shared network to adapt its computation to each token without modality-specific branches.

We address the second challenge with role-based positional embedding (PE) built on RoPE (Su et al. 2024), in which the positions used in attention depend on whether the attending token belongs to a condition or target segment. Condition queries use local positions and attend only within their own segment, preserving internal structure and avoiding false correspondences across conditions. Target queries retain local target positions while treating all condition segments as an unordered set under a shared positional reference, removing spurious cross-segment biases and supporting flexible condition-target layouts.

Figure 1 illustrates the overall pipeline, transformer block, and per-token attributes with role-based positional encoding.

Figure 1. Fine-grain illustration of our proposed Modaliff.

Learning Any-to-Any with Paired Data

Training Objective We adopt the training objective of DR (Eq. (4)) to learn all one-to-one mappings directly from paired datasets. For any two modalities i,j∈Mi, j \in\mathcal{M} with a paired dataset Di,j\mathcal{D}_{i,j}, Modaliff learns both denoising directions, i→ji \to j

and j→ij \to i, through a symmetric objective:

Lpair(θ)=E(xi,xj),t,ϵ,ζ[ζ∥zθ(zti,t,zj)−zi∥22+(1−ζ)∥zθ(ztj,t,zi)−zj∥22],(8)\mathcal{L}_{\mathrm{pair}}(\theta)=\mathbb{E}_{(x^i,x^j),t,\epsilon,\zeta}\left[\zeta\left\lVert z_\theta\left(z_t^i,t,z^j\right)-z^i\right\rVert_2^2 +\left(1-\zeta\right)\left\lVert z_\theta\left(z_t^j,t,z^i\right)-z^j\right\rVert_2^2\right], \tag*{(8)}

where ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0, I), t∼U(1,T)t \sim\mathcal{U}(1, T), ζ∼B(0.5)\zeta\sim\mathcal{B}(0.5), and zti=αtzi+σtϵz_t^i = \alpha_t z^i + \sigma_t \epsilon. Here, zθ(zti,t,zj)z_\theta\left(z_t^i,t,z^j\right) abbreviates Modaliff’s clean-latent prediction, computed on the unified sequence H(zj,zti)\mathcal{H}\left(z^j,z_t^i\right) of Eq. (7). The output tokens of the target segment, denoted h^θi∈RNi×di\hat{h}_\theta^i \in\mathbb{R}^{N_i \times d_i}, are projected back to the native latent space as:

zθ(zti,t,zj)=h^θi⋅Wouti,Wouti∈Rd×di(9)z_\theta\left(z_t^i,t,z^j\right)=\hat{h}_\theta^i \cdot W_{\mathrm{out}}^i,\qquad W_{\mathrm{out}}^i \in\mathbb{R}^{d \times d_i} \tag*{(9)}

where WoutiW_{\mathrm{out}}^i is a trainable output projection matrix. Unlike the noise-prediction in Eq. (4), Modaliff predicts the clean latents, which is more robust across heterogeneous pretrained latents with varying scales and heavy-tailed dimensions that can destabilize noise prediction, as shown in Appx. B.2. During training, we replace the condition with a null token ∅\varnothing with probability 0.2, enabling CFG (Ho and Salimans 2021).

To generate xix^i from xjx^j, we encode zj=Enc⁡j(xj)z^j = \operatorname{Enc}^j(x^j), initialize zti∼N(0,I)∈RNi×diz_t^i \sim\mathcal{N}(0,I) \in\mathbb{R}^{N_i \times d_i}, assemble H(zj,zti)\mathcal{H}(z^j,z_t^i), and run diffusion sampler (Song, Meng, and Ermon 2021) with CFG (Eq. (3)) in the noise space with ϵθ(zti,t,⋅)=(zti−αtzθ(zti,t,⋅))/σt\epsilon_\theta\left(z_t^i,t,\cdot\right)=\left(z_t^i-\alpha_t z_\theta\left(z_t^i,t,\cdot\right)\right)/\sigma_t. The final prediction z^i\hat{z}^i is decoded with the frozen decoder, i.e., xi=Dec⁡i(z^i)x^i=\operatorname{Dec}^i(\hat{z}^i).

Learning with Missing Pairwise Data In practice, some modality pairs are difficult to collect and provide far less supervision than others. For example, image-audio dataset (Di,j)(D_{i,j}) [5] are typically scarce compared with image-text (Di,c)(D_{i,c}) [33] and audio-text (Dj,c)(D_{j,c}) [7] datasets, in which case Modaliff can exploit the shared text modality cc to learn the missing Xj→XiX^j \to X^i mappings. To learn Xj→XiX^j \to X^i (or similarly Xj←XiX^j \leftarrow X^i) without access to Di,jD_{i,j}, we reformulate Eq. (5) under clean-latent prediction as:

Lunpairj→i(θ)=E(xj,xc),t,zti∼pθ(zti∣xc)[∥zθ(zti,t,zj)−sg⁡(zθ(zti,t,zc))∥22],(10)\mathcal{L}_{\mathrm{unpair}}^{j\to i}(\theta)=\mathbb{E}_{(x^j,x^c),t,z_t^i\sim p_\theta(z_t^i\mid x^c)}\left[\left\|z_\theta(z_t^i,t,z^j)-\operatorname{sg}\left(z_\theta(z_t^i,t,z^c)\right)\right\|_2^2\right], \tag*{(10)}

where sg⁡(⋅)\operatorname{sg}(\cdot) denotes stop gradient. We jointly optimize Eq. (8) and Eq. (10) via Eq. (6), enabling Xj→XiX^j \to X^i without Di,jD_{i,j}.

Multi-Condition Generation via Score Composition

Modaliff enables multi-condition generation by composing the single-condition predictions learned from pairwise training. Let ii denote the target modality and S⊂(M∖i)\mathcal{S} \subset(\mathcal{M}\setminus i) the conditioning modalities. Assuming the conditions are conditionally independent given the noisy target, i.e., p({zj}j∈S∣zti)=∏j∈Sp(zj∣zti)p\left(\{z^j\}_{j\in\mathcal{S}}\mid z_t^i\right)=\prod_{j\in\mathcal{S}}p(z^j\mid z_t^i), the product-of-experts formulation [26] yields multi-condition noise prediction as:

ϵ~θ(zti,t,{zj}j∈S)=ϵθ(zti,t,∅)+∑j∈Swj[ϵθ(zti,t,zj)−ϵθ(zti,t,∅)],(11)\widetilde{\epsilon}_\theta\left(z_t^i,t,\{z^j\}_{j\in\mathcal{S}}\right)=\epsilon_\theta(z_t^i,t,\varnothing)+\sum_{j\in\mathcal{S}}w_j\left[\epsilon_\theta(z_t^i,t,z^j)-\epsilon_\theta(z_t^i,t,\varnothing)\right], \tag*{(11)}

where ϵθ(zti,t,⋅)=(zti−atzθ(zti,t,⋅))/σt\epsilon_\theta(z_t^i,t,\cdot)=(z_t^i-a_tz_\theta(z_t^i,t,\cdot))/\sigma_t and {wj}j∈S\{w_j\}_{j\in\mathcal{S}} control the contribution of each condition. Eq. (11) extends single-condition predictions to multi-condition generation without training on fully aligned multimodal datasets and reduces to standard CFG when only one condition is provided.

Improving Multi-Condition Generation via Caption-Based Finetuning

Score composition (Eq. (11)) enables inference-time multi-condition generation but limits the ability to capture interactions among conditions and struggles with tasks such as Xi+Xj→XiX^i+X^j\to X^i, where the required same-modality prediction ϵθ(zti,t,zi)\epsilon_\theta(z_t^i,t,z^i) is unavailable. To mitigate the gap, we propose a novel approach that constructs a compact finetuning set for multi-condition generation and uses it to adapt Modaliff. Our key idea is to encode joint semantics of multiple conditions in language and transfer them through the text-to-target mappings learned from one-to-one training. For example, image generation from image and audio conditions can be reframed as text-to-image generation using a caption that combines semantics of both conditions.

Formally, we consider two-condition generation Xi+Xj→XkX^i+X^j\to X^k, with i,j,k∈Mi,j,k\in\mathcal{M}, following prior any-to-any settings [39, 20]. Given two pairs (xi,xc)∼Di,c(x^i,x^c)\sim D_{i,c} and (xj,x′c)∼Dj,c(x^j,x^{\prime c})\sim D_{j,c}, where cc denotes text, an instruction-tuned language model [45] merges xcx^c and x′cx^{\prime c} into a natural joint description x~c\widetilde{x}^c that preserves information from both xix^i and xjx^j, yielding a finetuning triplet (xi,xj,x~c)(x^i,x^j,\widetilde{x}^c). Random pairing, however, provides a poor finetuning signal because unrelated captions yield unnatural descriptions, while near-duplicate captions contribute limited complementary information. We therefore compute their cosine similarity as:

sim⁡(xc,x′c)=⟨Emb⁡(xc),Emb⁡(x′c)⟩∥Emb⁡(xc)∥2∥Emb⁡(x′c)∥2,(12)\operatorname{sim}(x^c,x^{\prime c})=\frac{\langle\operatorname{Emb}(x^c),\operatorname{Emb}(x^{\prime c})\rangle}{\|\operatorname{Emb}(x^c)\|_2\|\operatorname{Emb}(x^{\prime c})\|_2}, \tag*{(12)}

where Emb⁡(⋅)\operatorname{Emb}(\cdot) denotes a sentence-embedding model [31], and retain only pairs satisfying sim⁡(xc,x′c)∈[γmin⁡,γmax⁡]\operatorname{sim}(x^c,x^{\prime c})\in[\gamma_{\min},\gamma_{\max}], where γmin⁡\gamma_{\min} and γmax⁡\gamma_{\max} control the similarity range. The resulting triplets supervise task (Xi+Xj)→Xk(X^i+X^j)\to X^k with the objective:

Lmulti(i+j)→k(θ)=E(xi,xj,x~c),ztk,ztk[∥zθ(ztk,t,{zi,zj})−ztk∥22],(13)\mathcal{L}_{\mathrm{multi}}^{(i+j)\to k}(\theta)=\mathbb{E}_{(x^i,x^j,\widetilde{x}^c),z_t^k,z_t^k}\left[\left\|z_\theta\left(z_t^k,t,\{z^i,z^j\}\right)-z_t^k\right\|_2^2\right], \tag*{(13)}
s.t.{k=c:ztk=atz~c+σtϵ,z‾k=z~ck≠c:ztk∼pθ(ztk∣z~c),z‾k=sg⁡(zθ(ztk,t,z~c))(14)\text{s.t.}\quad \begin{cases} k=c:\quad z_t^k=a_t\widetilde{z}^c+\sigma_t\epsilon,\quad\overline{z}^k=\widetilde{z}^c\\ k\ne c:\quad z_t^k\sim p_\theta(z_t^k\mid\widetilde{z}^c),\quad\overline{z}^k=\operatorname{sg}\left(z_\theta(z_t^k,t,\widetilde{z}^c)\right) \end{cases} \tag*{(14)}

where z~c=Enc⁡c(x~c)\widetilde{z}^c=\operatorname{Enc}^c(\widetilde{x}^c). To preserve the pretrained one-to-one mappings and finetune efficiently, we optimize Eq. (13) using LoRA (Hu et al. 2022).

Experiments

Experiment Setup

Training Datasets For image-text dataset, we train Modaliff on 800K pairs sampled from LAION-Aesthetics +6.25 [33], replacing noisy captions containing URLs or HTML fragments with captions generated by LLaVA [24]. For audio-text dataset, we use 92K human-annotated pairs from AudioCaps v2 [15]. Although Modaliff does not strictly require image-audio pairs, we include 184K image-audio pairs from VGGSound [5] for fair comparison with baselines. We additionally train a variant without VGGSound to assess image↔audio generation through text as the shared modality. Full dataset details are provided in Appx. A.1.

Implementation Details We use CLIP-ViT-L/16 [28] to encode an image into zimg∈R1×768z^{\mathrm{img}}\in\mathbb{R}^{1\times768} and Stable-UnCLIP-small [30] to decode zimgz^{\mathrm{img}}. For text, we build an encoder based on Qwen3-0.6B Embedding [47] to map each text sequence to a fixed-length latent ztxt∈R32×256z^{\mathrm{txt}}\in\mathbb{R}^{32\times256}, and use a fine-tuned Qwen3-1.7B [45] to generate text from ztxtz^{\mathrm{txt}}. Appx. A.2 provides details of the text encoder and text decoder. For audio, we use CLAP-HTSAT (Wu et al. 2023) to encode each 10-second waveform into zaud∈R1×512z^{\mathrm{aud}}\in\mathbb{R}^{1\times512} and AudioLDM-S [23] to decode zaudz^{\mathrm{aud}}. Modality-specific encoders and decoders are frozen during training.

Following DiT-L [27], Modaliff uses 24 transformer blocks, hidden dimension d=1024d=1024, and 16 attention heads, yielding 558M trainable parameters to fit a H100 80GB GPU. We train Modaliff with Eq. (8) for up to 500K iterations with a batch size of 512, bfloat16 precision, and a constant learning rate of 3e-5 with 5K warmup steps. With no VGGSound setting, we finetune Modaliff following Sec. 3.2 to enable image↔audio generation, with reduced learning rate of 5e−65\mathrm{e}{-6}, and λ1=λ2=1\lambda_1 = \lambda_2 = 1 in Eq. (6), as recommended by DR [13]. For multi-condition generation, we construct 50K composed examples with γmin⁡=0.4\gamma_{\min} = 0.4, γmax⁡=0.8\gamma_{\max} = 0.8, and finetune Modaliff with LoRA of rank 16 and scaling 32 for 60K iterations with a learning rate of 5e−65\mathrm{e}{-6}.

Evaluation settings We evaluate cross-modal generation across image (I), text (T), and audio (A), using metrics for both fidelity and semantic alignment. Image fidelity is measured by FID [9] on 256×256256 \times256 images, using 30K MSCOCO validation samples [22] for T→I and VGGSound test samples for A→I. Text quality is evaluated with CIDEr [41] on MSCOCO for I→T, where each image has five reference captions, and AudioCaps for A→T, with five captions per audio clip. Audio fidelity is measured by FAD [14] on AudioCaps for T→A and VGGSound for I→A. Semantic alignment is assessed using CLIP score [28] for image-text pairs, CLAP score (Wu et al. 2023) for audio-text pairs, and ImgBind score [8] for audio-image pairs. For multi-condition evaluation, we hold out 1K samples from the finetuning set, using γmin⁡=0.4\gamma_{\min} = 0.4, γmax⁡=0.8\gamma_{\max} = 0.8 to encourage semantically distinct conditions.

Baselines We compare Modaliff against state-of-the-art specialist models and any-to-any diffusion models, including CoDi [39], OmniFlow [20], and FlowBind [4]. For specialists, we adopt the results from FlowBind. For any-to-any models, we use the official checkpoints, sampling scripts, recommended hyperparameters, and fixing latent generation to 50 sampling steps across all methods for a fair comparison.

Experiment Results

One-to-One Cross-Modality Generation Table 1 and Table 2 report fidelity and semantic alignment for one-to-one cross-modal generation across text, image, and audio, while Appdx. A.1 compares the training-data scale and trainable parameters of the generalists. We observe that Modaliff outperforms the generalists in fidelity on most generation tasks, with particularly large gains on T→I and T→A, highlighting the effectiveness of preserving modality-native token representations within a unified denoising sequence. CoDi’s inferior performance typically stems from combining multiple outdated U-Net architectures and training under a multi-task objective, making it difficult to balance learning across modalities. OmniFlow exhibits strong T→I alignment, which is expected given its initialization from SD3-Medium, a strong foundation model trained on billions of high-quality samples. However, OmniFlow underperforms in FID, which we attribute to the substantial amount of synthetic and stylized imagery in the training data of SD3-Medium, causing its generations to differ in visual style from the realistic images in MSCOCO. Moreover, both CoDi and OmniFlow denoise high-dimensional image and audio VAE latents [32, 23], requiring larger denoisers and higher training cost. In contrast, Modaliff operates on compact semantic latents produced by contrastive encoders (CLIP for images and CLAP for audio), shifting low-level synthesis to frozen decoders and matching these baselines with a substantially smaller denoiser and fewer computational resources. Although FlowBind improves performance over CoDi by jointly training all velocity models with a shared latent encoder, its fixed shared vector representation discards the token-level structure of text, causing FlowBind to underperform in both text generation quality and alignment on the text-conditioned routes T→I and T→A.

TaskT→IA→II→TA→TT→AI→A
MethodFID↓CIDEr↑FAD↓
SD3-Medium25.40-----
Sound2Vision-42.55----
LLaVA-Next--109.3---
Qwen2-Audio---4.64--
AudioLDM-L----1.96-
Seeing&Hearing-----5.31
CoDi*24.6856.6414.016.619.7216.34
OmniFlow*23.8472.3143.6031.792.483.69
FlowBind*17.3936.4837.3536.055.192.52
Modaliff13.7719.0244.4741.301.311.89
Modaliff-noVGG14.1857.2346.6941.571.733.16

Table 1. Fidelity metrics for cross-modal generation. * indicates results that we reproduced using the official repository.

TaskT→II→TA→TT→AA→II→A
MethodCLIP↑CLAP↑ImgBind↑
SD3-Medium31.60-----
Sound2Vision----26.71-
LLaVA-Next-32.14----
Qwen2-Audio--17.09---
AudioLDM-L---36.05--
Seeing&Hearing-----30.69
CoDi*30.3226.117.8610.2315.965.53
OmniFlow*31.7027.848.6533.138.5613.91
FlowBind*27.6928.3841.3142.3024.4321.11
Modaliff30.6528.1140.0145.7124.8327.99
Modaliff-noVGG30.3728.4539.4244.0112.8313.79

Table 2. Alignment scores for cross-modal generation. * indicates results that we reproduced using the official repository.

When VGGSound is excluded from training, Modaliff predictably degrades on A→I and I→A, as Modaliff must learn these mappings indirectly through text rather than from direct image-audio supervision. Nevertheless, Modaliff-noVGG still outperforms CoDi on I→A and OmniFlow on both A→I and I→A in fidelity, while remaining comparable in alignment despite both baselines using millions of image-audio pairs, indicating that Modaliff can effectively use text as the central modality to learn image-audio generation.

Ablation Studies

Impact of Role-based Positional Embedding We compare role-based PE against local indexing using two Modaliff models that differ only in the positional scheme, each reduced to 159M parameters for efficiency. As shown in Figure 4, role-based PE consistently outperforms local indexing on all generation routes, with the largest gain on text generation. We attribute this improvement to the positional conflict introduced by local indexing in the attention projections shared across all generation tasks. Under local indexing, each condition and target segment resets its positions from the same origin, creating artificial correspondences between unrelated tokens that share similar indices. The shared projections must then suppress this bias while preserving the positional structure of the target sequence, which is particularly important for text generation. Role-based PE removes these cross-segment biases by presenting condition tokens to target queries under a shared positional reference, enabling more effective routing across variable generation tasks.

Learning curves comparing role-based positional encoding and local indexing

Figure 4. Learning curves comparing role-based positional encoding and local indexing.

Multi-Condition Generation

Table 3 reports multi-condition alignment for generalists and Modaliff using score composition (§ 3.2) and composed caption finetuning (§ 3.4). For each task, we measure the ImageBind score between the generated output and each condition separately. Without multi-condition training, Modaliff remains competitive with the baselines and performs particularly well on (T+I)→A, demonstrating that the learned one-to-one mappings can be composed effectively at inference. Finetuning with composed captions further improves five of the six alignment scores and achieves the best performance on four, confirming that language provides effective supervision for modeling interactions between conditions. The only exception is the lower A-I score of finetuned variant on (T+A)→I relative to Modaliff (CFG), which arises because static images are inherently limited in depicting transient acoustic events such as speaking or chirping, even when this information is preserved in the composed caption, thereby limiting their transfer from the T→I teacher …

Method(T+A)→I T-I(T+A)→I A-I(I+A)→T I-T(I+A)→T A-T(T+I)→A T-A(T+I)→A I-A
CoDi22.9722.9315.0519.518.213.30
FlowBind27.7120.8220.2120.5412.929.60
Modaliff (CFG)23.3920.9321.9217.1320.8913.89
Modaliff (Finetuned)29.4116.3624.2818.3522.1014.08

Table 3. ImgBind scores for multi-condition generation. This capability is unavailable in OmniFlow’s released code.

This behavior is consistent with Figure 2, where fine-tuning improves visual quality and text alignment, while actions suggested by the audio, such as giving a speech, remain difficult to depict in a single image. For (T + I)→I, Figure 3 shows that Modaliff produces higher-quality images and follows both conditions more accurately, demonstrating the effectiveness of the proposed finetuning strategy.

Figure 2. Qualitative results of Modaliff and generalists on (T + A)→I and (I + A)→T.

Qualitative results of Modaliff and generalists on text and image inputs

Figure 3. Qualitative results of Modaliff and generalists on (T + I)→I.

Impact of the Similarity Band [γmin⁡,γmax⁡\gamma_{\min}, \gamma_{\max}] To evaluate similarity filtering for multi-condition triplet construction, we fix γmax⁡=0.8\gamma_{\max} = 0.8 to exclude near-duplicate pairs, vary γmin⁡∈{0,0.2,0.4,0.6}\gamma_{\min} \in\{0, 0.2, 0.4, 0.6\}, and finetune Modaliff on 10K composed samples per setting. Table 4 reports the alignment between the generated target and each of the two conditions across three multi-condition tasks. Across all tasks, increasing γmin⁡\gamma_{\min} improves alignment with one condition but reduces alignment with the other. This is because a high γmin⁡\gamma_{\min} retains pairs with highly similar captions, causing x~c\tilde{x}^{c} to emphasize shared content and provide limited supervision for composing distinct conditions. Conversely, a lower γmin⁡\gamma_{\min} admits more diverse pairs and strengthens the use of complementary information but may introduce details supported by only one input, thereby reducing alignment with the other. These results support our hypothesis for similarity-based filtering and we use γmin⁡=0.4\gamma_{\min} = 0.4 to balance individual-condition alignment with effective multi-condition fusion.

γmin⁡\gamma_{\min}T-IA-II-TA-TT-AI-A
026.8016.4223.2314.9816.2812.71
0.227.1316.3623.5514.3516.4312.33
0.427.2515.6023.9013.6516.8411.78
0.627.7815.3723.9812.1317.0411.20

Table 4. ImgBind score between the target and each conditions with γmin⁡∈{0,0.2,0.4,0.6}\gamma_{\min} \in\{0, 0.2, 0.4, 0.6\} and γmax⁡=0.8\gamma_{\max} = 0.8. Each setting finetunes on 10K samples.

Related Work

Building on the success in pairwise cross-modal generation ([23, 6] (Le et al. 2026)), diffusion models ((Song et al. 2021)) have recently been extended to unified multimodal generation. Versatile Diffusion (Xu et al. 2023) extends latent diffusion into a unified image-text framework with a modular cross-modal architecture, but its model size grows with the number of modalities and the framework remains limited to the image-text setting. UniDiffuser (Bao et al. [3]) instead models the joint distribution of all modalities using a single transformer-based noise prediction network that represents each modality as a token sequence. This design provides a general formulation for multimodality, but it requires fully aligned multimodal tuples and substantial training time. CoDi (Tang et al. [39]) extends any-to-any generation to diverse modalities, including text, image, audio, and video, through a two-stage pipeline that first learns modality-specific encoders with contrastive

learning and then trains diffusion models for conditional generation. While this supports flexible and multi-source generation, it requires contrastive pretraining and multiple diffusion models whose number scales linearly with the number of modalities. OmniFlow (Li et al. [20]) replaces diffusion with flow matching to model a joint distribution across modalities and adapts a pretrained SD3 backbone (Esser et al. [6]) to reduce training cost. However, it still depends on fully aligned multimodal tuples, and missing modalities must be synthesized during training. More recently, FlowBind (Cha et al. [4]) adopts flow matching to learn a shared latent prior and modality-specific velocity models that map between the prior and each modality’s latent space. Although FlowBind jointly optimizes all models from paired data, it requires matched dimensions between the prior and modality representations, scales one model per modality, and assumes paired supervision for every modality pair.

In summary, most of existing multimodal diffusion models rely on fully aligned tuples, contrastive pretraining, or modality-specific generators whose parameters scale linearly with the number of modalities. FlowBind reduces reliance on aligned tuples through single-stage paired-data training, but couples all modalities to a shared prior’s dimensionality, uses one velocity model per modality, and assumes supervision for every modality pair. By contrast, our method learns directly from paired data, preserves modality-specific structure in native latent tokens, uses a single denoiser for all generation tasks to avoid modality-dependent parameter growth, and supports settings with missing modality pairs.

Conclusion

We presented Modaliff, a token-based diffusion framework for any-to-any generation that requires neither a shared representation space nor fully aligned multimodal tuples. By representing modality-native condition and target tokens under a unified token sequence, Modaliff supports diverse one-to-one and multi-condition tasks using a single denoiser trained on paired data. Extensive experiments across text, image, and audio demonstrate that Modaliff achieves the best fidelity among any-to-any models on most generation tasks while using substantially fewer parameters and training data.

Additional Details of the Proposed Framework

Comparison between existing diffusion-based multi-modal generation frameworks and ours

In Table 5 and Table 6, we detail the framework designs and the training data used by representative multimodal diffusion based methods, including Versatile Diffusion (Xu et al. 2023), UniDiffuser ([3]), CoDi ([39]), OmniFlow ([20]), and FlowBind ([4]). For each framework, we highlight important architectural designs that determine how modalities are processed and modeled, including the modality specific encoders and decoders, and the backbone model for parameterizing the score function of diffusion models. We also list the supervision format required by each framework, such as fully aligned multimodal tuples or pairwise datasets, together with the corresponding training dataset and the number of training samples for each pairwise generation task.

MethodText EncoderText DecoderImage EncoderImage DecoderAudio EncoderAudio DecoderDiffusion Model
Versatile Diffusion (Xu et al. 2023)Optimus (Li et al. 2020) (227M)VQ-VAE ([32]) (84M)--2D+1D UNet (2.6B)
UniDiffuser ([3])CLIP-L ([28]) (124M)GPT2 ([29]) (124M)VQ-VAE ([32]) (84M)--U-ViT ([2]) (952M)
CoDi ([39])Optimus (Li et al. 2020) (227M)VQ-VAE ([32]) (84M)AudioLDM-VAE ([23]) (111M)2D+1D UNet (4.3B)
OmniFlow ([20])CLIP-L + CLIP-G + Flan-T5-L ([20]) (783M)TinyLlama ([46]) (1.1B)VQ-VAE ([32]) (84M)AudioLDM-VAE ([23]) (111M)SD3-DiT (3.4B)
FlowBind ([4])Embedding Gemma ([42]) (300M)Gemma3 [40] (1B)CLIP-L Image Enc ([28]) (304M)Stable-unCLIP-small ([30]) (1.6B)CLAP-HTSAT + Autoencoder (Wu et al. 2023) (200M)AudioLDM-M ([23]) (652M)MLP (568M)
Modaliff (ours)Qwen3-Embedding ([47]) (600M)Qwen3 ([45]) (1.7B)CLIP-L Image Enc ([28]) (304M)Stable-unCLIP-small ([30]) (1.6B)CLAP-HTSAT (Wu et al. 2023) (158M)AudioLDM-S ([23]) (421M)Multimodal Diffusion Transformer (558M)

Table 5. Comparison between the components for each modality used in our methods and baselines.

MethodData formatText-Image NameText-Image SizeText-Audio NameText-Audio SizeImage-Audio NameImage-Audio Size
Versatile Diffusion (Xu et al. 2023)PairwiseLAION-2B-en COYO-700M2.7B----
UniDiffuser ([3])PairwiseLAION-2B-en LAION-high-resolution LAION-aesthetics v2 5+2.4B----
CoDi ([39])Aligned tuples + PairwiseLAION-400M400MAudioSet AudioCaps Freesound BBCSoundEffect3.5MAudioSet SoundNet2M
OmniFlow ([20])Aligned tuples + PairwiseLAION-aesthetics-3M CC12M COYO-700M (Subset) LAION-COCO T2I-2M28MAudioSet AudioCaps WavCaps2.4MVGGSound SoundNet2.2M
FlowBind ([4])PairwiseLAION-COCO Flickr-30K272KAudioCaps v292KVGGSound184K
Modaliff (ours)PairwiseLAION-aesthetics 6.25+800KAudioCaps v292KVGGSound (Optional)184K (0K)

Table 6. Comparison between the multi-modal training datasets used in our methods and baselines.

From Table 5 and Table 6, Modaliff uses a 558M trainable denoiser and approximately 1.1M paired samples for the full text-image-audio setting, requiring substantially fewer trainable parameters and less data than large-scale any-to-any models such as CoDi and OmniFlow. This comparison highlights the efficiency of Modaliff in predicting compact semantic latents from frozen contrastive pretrained encoders such as CLIP and CLAP while delegating low-level synthesis to frozen modality-specific decoders. FlowBind provides the closest comparison in trainable model scale, but requires a common vector dimensionality across the shared prior and modality representations and uses a separate MLP for each modality. Modaliff instead preserves each modality’s separate latent token sequence and aligns only the transformer width through lightweight projections, allowing one shared denoiser to process heterogeneous token lengths and feature dimensions. Among methods supporting image-audio generation, CoDi, OmniFlow, and FlowBind rely on direct image-audio pairs, whereas Modaliff can learn image-to-audio and audio-to-image mappings through text as the shared modality when VGGSound is omitted.

Details of the Proposed Text Auto-Encoder

High-fidelity Text Autoencoder Since diffusion-based multimodal generation frameworks usually operate in latent space, their performance depends on the quality of the underlying auto-encoders for each modality. While effective auto-encoders exist for images and audio (e.g., [32] and [23]), comparable solutions for text remain underdeveloped. We empirically observe that current frameworks ([39]; [3]; [20]) often produce repetitive and unnatural text. We attribute this to the use of outdated or underpowered text models. For example, Versatile Diffusion (Xu et al. 2023) adopts Optimus (Li et al. 2020), which pairs a BERT encoder with a GPT-2 decoder, while OmniFlow ([20]) employs a CLIP-L + CLIP-G + Flan-T5-L encoder with a TinyLlama-1.1B decoder ([46]).

To address this limitation, we introduce a text auto-encoder that uses Qwen3-0.6B Embedding ([47]) as the encoder and Qwen3-1.7B ([45]) as the decoder. As the latest generation in the Qwen family, Qwen3 exhibits strong multilingual generation and reasoning capabilities, while the embedding counterpart achieves competitive performance on text representation and retrieval benchmarks. Together, they form a high-fidelity text auto-encoding backbone.

Given an input sequence x=(x1,…,xL)x = (x_{1}, \ldots, x_{L}), we first obtain token-level representations h∈RL×dench \in\mathbb{R}^{L \times d_{\mathrm{enc}}} from the Qwen3 Embedding model without applying any pooling operation. Then, we employ a Flamingo-style resampler ([1]) to compress the variable-length sequence into a fixed-size latent representation z∈RM×dlatentz \in\mathbb{R}^{M \times d_{\mathrm{latent}}}. A lightweight projector subsequently maps zz to a sequence of prefix embeddings r∈RM×ddecr \in\mathbb{R}^{M \times d_{\mathrm{dec}}}, which are prepended to the language model decoder as soft prompts for reconstruction. During training, we freeze the embedding encoder while jointly optimizing the resampler, projector, and decoder. To improve parameter efficiency, we finetune the decoder using LoRA (Hu et al. 2022) while keeping the remaining decoder weights frozen. The model is trained with the standard next-token prediction objective:

L(ϕ)=−∑l=1Llog⁡pϕ(xl∣r,x<l).(15)\mathcal{L}(\phi) = -\sum_{l=1}^{L} \log p_{\phi}(x_l \mid r, x_{<l}). \tag*{(15)}

Training Configurations We use Qwen3-0.6B Embedding without a pooling layer to encode a text sequence x=(x1,…,xL)x = (x_{1}, \ldots, x_{L}) into token-level representations h∈RL×dench \in\mathbb{R}^{L \times d_{\mathrm{enc}}}, where LL is the sequence length and denc=1024d_{\mathrm{enc}} = 1024 follows the Qwen3-0.6B Embedding specification. We then apply a linear projection to obtain h′∈RL×dlatenth' \in\mathbb{R}^{L \times d_{\mathrm{latent}}}, reducing the feature dimension to dlatent=256d_{\mathrm{latent}} = 256 while retaining sufficient information for accurate reconstruction. To convert variable-length token sequences into a fixed-length latent, we adopt the resampler module from Flamingo ([1]) to compress h′h' into zz containing MM learned query tokens, i.e., z∈RM×dlatentz \in\mathbb{R}^{M \times d_{\mathrm{latent}}}, with M=32M = 32 following common practice in BLIP-2 (Li et al. 2023) and OmniFlow ([20]). The resampler outputs zz serve as the text latent representation for downstream diffusion models. For decoding, we map zz through a linear projector to obtain a 32×204832 \times2048 prefix embedding that is fed into a Qwen3-1.7B decoder. The full architecture is illustrated in Figure 5.

Figure 5. Architecture of Text Auto-Encoder: a) Overall pipeline of the proposed text auto-encoder. b) Details of the resampler module to produce latent embedding of text.

We train the text autoencoder using the objective in Eq. (15) with a two-stage procedure. In the first stage, we freeze the Qwen3-0.6B Embedding encoder, train the resampler and projection layers, and finetune the Qwen3-1.7B decoder with LoRA using rank r=16r = 16 and scaling α=32\alpha= 32. We train for five epochs on captions from the text-image dataset and exclude the text-audio captions, since they are often short phrases rather than well-formed sentences. We use a learning rate of 1e-5 and a global batch size of 64 with the AdamW optimizer ([25]). In the second stage, we freeze the resampler and continue finetuning the Qwen3-1.7B decoder with LoRA for three additional epochs on the merged caption corpus of Table 6.

Further Experimental Results

Study on different text encoders

We examine how the choice of text encoder affects the performance of our text auto-encoder model by training the same architecture with three encoder options, Qwen3-Embedding-0.6B, Qwen3-0.6B, and CLIP-ViT-B/32, while keeping the resampler design and the Qwen3-1.7B text decoder fixed. Figure 6 reports reconstruction metrics on the validation set throughout training, including Perplexity, BLEU-1, and BLEU-4. Overall, the Qwen3 based encoders consistently achieve lower perplexity and higher BLEU scores than CLIP, indicating stronger reconstruction quality. This is expected because the decoder is a Qwen3 model, so encoders from the same model family produce representations that are better aligned with the decoder’s representation space. Consequently, the resampler can form effective prefix tokens from the encoder features more easily, whereas the CLIP-ViT-B/32 text encoder introduces a larger representation mismatch that the resampler must learn to compensate for. Furthermore, we observe that Qwen3-Embedding-0.6B reaches strong reconstruction performance earlier than Qwen3-0.6B, which is likely because Qwen3-Embedding is explicitly optimized to produce information dense embeddings for downstream use, providing a more suitable input representation for reconstruction. These results empirically justify the configuration of text auto-encoder models.

Figure 6. Validation reconstruction metrics under different text encoder choices with the same resampler and Qwen3-1.7B decoder. We report Perplexity, BLEU-1, and BLEU-4 over training for Qwen3-Embedding-0.6B, Qwen3-0.6B, and CLIP-ViT-B/32.

Ablation Studies

Impact of Different Parameterizations Modaliff supports multiple parameterizations of the denoising model. We study three diffusion parameterizations, including ϵ\epsilon-prediction, x0x_0-prediction, and vv-prediction, where v=atϵ−σtx0v = a_t\epsilon- \sigma_t x_0. We also

include a Rectified Flow (RF) variant that uses the ϵ−x0\epsilon- x_0 parameterization and the RF noise schedule [20, 4]. For ϵ\epsilon-prediction, we normalize each modality using training set mean and standard deviation, whereas x0x_0-prediction and vv-prediction are trained on raw latents. Moreover, we find that ϵ\epsilon-prediction with a cosine schedule yields corrupted samples due to SNR(T)≈0SNR(T) \approx0, consistent with prior work [21], so we instead use a linear βt\beta_t schedule and clip normalized values to [−8,8][-8,8] for stability. To save computational resources, we reduce the denoiser to 159M parameters and evaluate both FID for T→I and CIDEr for I→T on 5K MSCOCO validation samples.

As shown in Table 7, x0x_0-prediction achieves the best performance on 4 out of 6 generation tasks, with the largest improvement on T→I and A→I, while ϵ\epsilon-prediction remains less stable and produces corrupted samples on image generation tasks (see Figure 7). We attribute this behavior to the CLIP image latent space, where several dimensions remain heavy-tailed and contain large-magnitude outliers even after per-modality standardization (Figure 8). These latent statistics make ϵ\epsilon-prediction difficult to learn, whereas x0x_0-prediction directly predicts the clean latent and provides a more data-aligned objective. The mixed noise-data parameterizations, including vv-prediction and RF, underperform x0x_0-prediction on most tasks. This difference is likely due to the added complexity of mixed noise-data targets, which can be harder to learn than direct x0x_0-prediction under the same model capacity, consistent with findings in [20]. These results support our choice of x0x_0-prediction as a robust default for Modaliff across modalities with diverse latent statistics.

PredictionT→II→TT→AA→TA→II→A
FID↓CIDEr↑FAD↓CIDEr↑FID↓FAD↓
ϵ−x0\epsilon- x_032.3746.011.6644.2636.101.69
ϵ\epsilon74.2922.311.8241.29159.461.91
atϵ−σtx0a_t\epsilon- \sigma_t x_028.3929.571.7441.0834.271.87
x0x_024.4536.911.3340.4329.851.49

Table 7. Fidelity metrics of Modaliff with different diffusion parameterizations.

Failure cases of ε-prediction on T→I generation

Figure 7. Failure cases of ϵ\epsilon-prediction on T→I generation.

Heavy-tailed distributions in normalized CLIP image latent dimensions

Figure 8. Heavy-tailed distributions in normalized CLIP image latent dimensions.

Impact of the Attribute Tuple and Token-wise AdaLN The attribute tuple (r,i,t)(r, i, t) and its token-wise adaptive normalization are Modaliff’s key mechanisms for resolving task ambiguity in the unified sequence (§ 3.1). We isolate their contribution with two ablated variants of the 159M model, evaluating FID for T→I and CIDEr for I→T on 5K MSCOCO samples. w/o attributes removes the role and modality embeddings from both the AdaLN conditioning and the token-content injection, reducing the per-token conditioning vector to the timestep embedding, with t=0t = 0 for condition tokens. Seq.-level AdaLN keeps the full attribute tuple but replaces the per-token conditioning vectors with their unweighted mean over the sequence, producing a single set of scale, shift, and gating parameters broadcast to all tokens, following the conditioning of DiT [27].

Tab. 8 reports the results after 400K iterations. We observe that removing either mechanism degrades performance on all reported tasks, and removing the attributes is the most damaging. The collapse caused by removing the token attributes is expected from the sequence layout, since T→I, I→T, T→A, and A→T share the same number of tokens in the unified input sequence, making it difficult for the shared model to distinguish between task configurations and hindering the learning process, as reflected in Figure 9. Seq.-level AdaLN recovers much of this gap because the pooled vector still identifies the task, but it remains behind the full model due to a single set of modulation parameters to serve clean condition tokens and noisy target tokens at once, limiting its ability to adapt the computation across different generation configurations. These results support the design of Modaliff, in which the attribute tuple supplies the task information and token-wise AdaLN delivers it to each token, allowing one shared denoiser to serve heterogeneous condition and target layouts.

Learning curves of the architecture ablation on T→I, I→T, T→A, and A→T

Figure 9. Learning curves of the architecture ablation on T→IT \to I, I→TI \to T, T→AT \to A, and A→TA \to T.

VariantT→\toII→\toTT→\toAA→\toT
FID↓\downarrowCIDEr↑\uparrowFAD↓\downarrowCIDEr↑\uparrow
Full model24.4536.911.3340.43
w/o attributes54.3525.537.2126.54
Seq.-level AdaLN28.5731.223.7631.32

Table 8. Ablation of the attribute tuple and token-wise adaptive normalization.

Impact of Guidance Scale for One-to-One Mappings We study the effect of classifier free guidance by varying the guidance scale w∈{1,3,5,7}w \in\{1, 3, 5, 7\} and measuring both fidelity and semantic alignment across one to one generation tasks. For evaluation efficiency, we evaluate FID for T→IT \to I and CIDEr for I→TI \to T on 5K MSCOCO samples. Figure 10 reports the corresponding quality metrics and alignment scores for T→IT \to I, I→TI \to T, T→AT \to A, and A→TA \to T. Across all directions, increasing ww consistently improves alignment, as reflected by the rise of CLIP and CLAP scores, indicating that stronger guidance enforces closer semantic agreement with the conditioning signal. In contrast, fidelity follows a non monotonic trend that reveals an over guidance regime. For T→IT \to I generation, FID improves as guidance increases from w=3w = 3 to w=5w = 5, but at w=7w = 7 it begins to saturate, matching the trade-off pattern observed in T→AT \to A generation where FAD improves most at w=3w = 3 and then degrades at higher guidance. For captioning tasks, including I→TI \to T and A→TA \to T, CIDEr improves sharply with ww and then saturates, with a slight drop at w=7w = 7 even as alignment continues to improve. Overall, these results indicate that moderate guidance offers the best balance between fidelity and alignment across modalities, while overly large guidance can over constrain sampling and reduce generation quality.

Effect of classifier-free guidance scale on the quality and alignment metrics across one-to-one generation tasks

Figure 10. Effect of classifier-free guidance scale on the quality and alignment metrics across one-to-one generation tasks.

Sensitivity to the Composition Weights wjw_j (11) assigns a guidance weight wjw_j to each condition, motivating an analysis of sensitivity to different weight configurations. We evaluate (T+A)→I(T+A) \to I by varying wtw_t and waw_a, and (A+I)→T(A+I) \to T by varying wiw_i and waw_a. We consider wa∈{1,1.5,2,3}w_a \in\{1, 1.5, 2, 3\} and wt,wi∈{2,3,4,5,6}w_t, w_i \in\{2, 3, 4, 5, 6\}, yielding 20 configurations per task. Each configuration generates 1K test samples, which are evaluated using the ImageBind alignment between the generated output and each condition. Figure 11 shows the task-dependent trade-off between the two conditions. For (T+A)→I(T+A) \to I, increasing wtw_t improves text alignment at the expense of audio alignment, while increasing waw_a produces the opposite effect. Configurations with similar ratios wa/wtw_a/w_t tend to occupy nearby regions of the trade-off frontier, suggesting that relative weighting largely determines the balance between the conditions. A similar trend is also reflected in (A+I)→T(A+I) \to T, where increasing wiw_i generally favors image alignment but degrades the audio alignment. Interestingly, for (A+I)→T(A+I) \to T, we observe that wa/wi>0.5w_a/w_i > 0.5 reduces alignment with both conditions relative to the corresponding configuration with wa/wi=0.5w_a/w_i = 0.5, further indicating that the relative guidance ratio largely controls the

Sensitivity plots and alignment score heatmaps for all configurations

Figure 11. Sensitivity to the composition weights on (T+A)→I(T+A)\rightarrow I and (A+I)→T(A+I)\rightarrow T. In (a) and (b), each curve fixes waw_a, marker size encodes the weight of the other condition, and the star marks the selected configuration. In (c), each cell reports the ImageBind alignment between the generated output and one condition, with the selected configuration outlined in black.

balance between conditions and that large waw_a can push the model into an over-guided regime that degrades overall caption quality. Based on these results, we use (wt,wa)=(3,2)(w_t, w_a) = (3, 2) for (T+A)→I(T+A)\rightarrow I and (wa,wi)=(1.5,4)(w_a, w_i) = (1.5, 4) for (A+I)→T(A+I)\rightarrow T, marked by stars in Figure 11, keeping alignment to both conditions competitive rather than maximizing either alone. For (T+I)→A(T+I)\rightarrow A, we keep the default wt=wi=2w_t = w_i = 2, which already achieves competitive alignment to both conditions in Table 3.

Qualitative Results

We present qualitative results across a broad range of one-to-one and multi-condition generation tasks in Figure 12–Figure 18, covering diverse people, objects, scenes, and semantic content. Modaliff consistently produces high-quality outputs that remain faithful to the provided conditions across text, image, and audio modalities.

Figure 12. Qualitative results on T→IT \to I generation.

Qualitative results table showing Input, Ours (Finetuned), Ours (CFG), FlowBind, and CoDi descriptions for waterfall, bear, boat, and clock examplesVector (PDF)
InputOurs (Finetuned)Ours (CFG)FlowBindCoDi
A waterfall with a rocky bottom, with the water flowing downstream. [water flowing and falling]A waterfall with a rocky bottom, with the water flowing downstream.waterfall with rocky cliffope[z] of water and rocks flowing waterfallwater is falling into the water and coming into the water
A brown bear swims in the water as the stream flows rapidly. [water flowing down a stream]A brown bear swims in the water as the stream flows rapidly.porous bear is swimming in the water and splashing.A bear clawing in water streamriver is walking by brown white water.
A boat on a lake reflects the mountains in the background as a boat floats on the water. [a boat is sailing on the water]A boat on a lake reflects the mountains in the background as a boat floats on the water.as a quiet rowboat floats on the water followed by mountain in the background and water noise.A boat rider moving in the water with mountains.wind on a mountain with a wave crashing on a sea with a blue <unk> in the water
A painting of a city with a castle and a large clock tower in the center. [a clock is ticking]A painting of a city with a castle and a large clock tower in the center.a painting of a city with a large building and a clock tower.An oil painting of a clock and tomatoesclock in the eye of a hall by the starlight bridge

Figure 18. Qualitative results on (I + A)→T generation.

Figure 13. Qualitative results on I→TI \to T generation.

Qualitative results showing Audio, Ours (w/ VGG), Ours (w/o VGG), FlowBind, CoDi, and OmniFlow across growling, pheasant crowing, subway, female singing, sewing machine, oriole calling, singing choir, and stream burbling

Figure 14. Qualitative results on A→\toI generation.

Figure 15. Qualitative results on A→TA \to T generation.

Figure 16. Qualitative results on (T+A)→I(T + A)\to I generation.

Figure 17. Qualitative results on (T+I)→I(T + I) \to I generation.

References

  1. [1]Alayrac, J.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; Ring, R.; Rutherford, E.; Cabi, S.; Han, T.; Gong, Z.; Samangooei, S.; Monteiro, M.; Menick, J. L.; Borgeaud, S.; Brock, A.; Nematzadeh, A.; Sharifzadeh, S.; Binkowski, M.; Barreira, R.; Vinyals, O.; Zisserman, A.; and Simonyan, K. 2022. Flamingo: a Visual Language Model for Few-Shot Learning. In NeurIPS.arxiv.org/abs/2204.14198
  2. [2]Bao, F.; Nie, S.; Xue, K.; Cao, Y.; Li, C.; Su, H.; and Zhu, J. 2023a. All are worth words: A vit backbone for diffusion models. In CVPR, 22669–22679.arxiv.org/abs/2209.12152
  3. [3]Bao, F.; Nie, S.; Xue, K.; Li, C.; Pu, S.; Wang, Y.; Yue, G.; Cao, Y.; Su, H.; and Zhu, J. 2023b. One transformer fits all distributions in multi-modal diffusion at scale. In ICML, 1692–1717.arxiv.org/abs/2303.06555
  4. [4]Cha, Y.; Kim, S.; Kwon, J.; and Hong, S. 2026. FlowBind: Efficient Any-to-Any Generation with Bidirectional Flows. In ICLR.arxiv.org/abs/2512.15420
  5. [5]Chen, H.; Xie, W.; Vedaldi, A.; and Zisserman, A. 2020. Vggsound: A large-scale audio-visual dataset. In ICASSP, 721–725.arxiv.org/abs/2004.14368
  6. [6]Esser, P.; Kулal, S.; Blattmann, A.; Entezari, R.; Müller, J.; Saini, H.; Levi, Y.; Lorenz, D.; Sauer, A.; Boesel, F.; Podell, D.; Dockhorn, T.; English, Z.; and Rombach, R. 2024. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. In ICML, 12606–12633.arxiv.org/abs/2403.03206
  7. [7]Gemmeke, J. F.; Ellis, D. P.; Freedman, D.; Jansen, A.; Lawrence, W.; Moore, R. C.; Plakal, M.; and Ritter, M. 2017. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP, 776–780.DOI
  8. [8]Girdhar, R.; El-Nouby, A.; Liu, Z.; Singh, M.; Alwala, K. V.; Joulin, A.; and Misra, I. 2023. Imagebind: One embedding space to bind them all. In CVPR, 15180–15190.arxiv.org/abs/2305.05665
  9. [9]Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; and Hochreiter, S. 2017. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In NIPS, 6626–6637.
  10. [13]Kieu, D.; Do, K.; Hoang, T.; Le, T. M.; Kieu, T.; Nguyen, D.; and Nguyen, T. 2026. Universal Multi-Domain Translation via Diffusion Routers. In ICLR.arxiv.org/abs/2510.03252
  11. [14]Kilgour, K.; Zuluaga, M.; Roblek, D.; and Sharifi, M. 2019. Fréchet Audio Distance: A Reference-Free Metric for Evaluating Music Enhancement Algorithms. In INTERSPEECH, 2350–2354.DOI
  12. [15]Kim, C. D.; Kim, B.; Lee, H.; and Kim, G. 2019. Audiocaps: Generating captions for audios in the wild. In NAACL–HLT, 119–132.DOI
  13. [20]Li, T.; and He, K. 2025. Back to basics: Let denoising generative models denoise. arXiv preprint arXiv:2511.13720.arxiv.org/abs/2511.13720
  14. [21]Lin, S.; Liu, B.; Li, J.; and Yang, X. 2024. Common diffusion noise schedules and sample steps are flawed. In WACV, 5404–5411.arxiv.org/abs/2305.08891
  15. [22]Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014. Microsoft coco: Common objects in context. In ECCV, 740–755.arxiv.org/abs/1405.0312
  16. [23]Liu, H.; Chen, Z.; Yuan, Y.; Mei, X.; Liu, X.; Wang, D.; Wang, W.; and Plumbley, M. D. 2023a. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models. In ICML, 21450–21474.
  17. [24]Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023b. Visual instruction tuning. In NeurIPS, 34892–34916.
  18. [25]Loshchilov, I.; and Hutter, F. 2019. Decoupled Weight Decay Regularization. ICLR.
  19. [26]Nguyen, T.; Do, K.; Kieu, D.; and Nguyen, T. 2025. h-Edit: Effective and Flexible Diffusion-Based Editing via Doob’s h-Transform. In CVPR, 28490–28501.
  20. [27]Peebles, W.; and Xie, S. 2023. Scalable diffusion models with transformers. In CVPR, 4195–4205.arxiv.org/abs/2212.09748
  21. [28]Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In ICML, 8748–8763.arxiv.org/abs/2103.00020
  22. [29]Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8): 9.
  23. [30]Ramesh, A.; Dharival, P.; Nichol, A.; Chu, C.; and Chen, M. 2022. Hierarchical Text-Conditional Image Generation with CLIP Latents. arXiv preprint arXiv:2204.06125.arxiv.org/abs/2204.06125
  24. [31]Reimers, N.; and Gurevych, I. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In EMNLP.arxiv.org/abs/1908.10084
  25. [32]Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022. High-resolution image synthesis with latent diffusion models. In CVPR, 10684–10695.arxiv.org/abs/2112.10752
  26. [33]Schuhmann, C.; Beaumont, R.; Vencu, R.; Gordon, C.; Wightman, R.; Cherti, M.; Coombes, T.; Katta, A.; Mullis, C.; Wortsman, M.; et al. 2022. LAION-5B: an open large-scale dataset for training next generation image-text models. In NeurIPS, 25278–25294.arxiv.org/abs/2210.08402
  27. [34]Sohl-Dickstein, J.; Weiss, E.; Maheswaranathan, N.; and Ganguli, S. 2015. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2256–2265.
  28. [36]Song, Y.; and Ermon, S. 2019. Generative modeling by estimating gradients of the data distribution. NeurIPS, 11895–11907.
  29. [39]Tang, Z.; Yang, Z.; Zhu, C.; Zeng, M.; and Bansal, M. 2023. Any-to-any generation via composable diffusion. In NeurIPS, 16083–16099.arxiv.org/abs/2305.11846
  30. [40]Team, G. 2025. Gemma 3 Technical Report. arXiv preprint arXiv:2503.19786.
  31. [41]Vedantam, R.; Zitnick, C. L.; and Parikh, D. 2015. CIDEr: Consensus-based image description evaluation. In CVPR, 4566–4575.arxiv.org/abs/1411.5726
  32. [42]Vera, H. S.; Dua, S.; Zhang, B.; Salz, D.; Mullins, R.; Panyam, S. R.; Smoot, S.; Naim, I.; Zou, J.; Chen, F.; et al. 2025. Embeddinggamma: Powerful and lightweight text representations. arXiv preprint arXiv:2509.20354.arxiv.org/abs/2509.20354
  33. [45]Yang, A.; Li, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Gao, C.; Huang, C.; Lv, C.; et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.arxiv.org/abs/2505.09388
  34. [46]Zhang, P.; Zeng, G.; Wang, T.; and Lu, W. 2024. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385.arxiv.org/abs/2401.02385
  35. [47]Zhang, Y.; Li, M.; Long, D.; Zhang, X.; Lin, H.; Yang, B.; Xie, P.; Yang, A.; Liu, D.; Lin, J.; et al. 2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176.arxiv.org/abs/2506.05176

Paper details

vlnc:W_b6xhmgms
Version
v1 · latest
Published
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents