WorksComputer Science
From Tokens to Trajectories: Parameter-Efficient Continuous Adaptation of Unified Multimodal Models
Abstract
While discrete tokenization in Unified Multimodal Models (UMMs) establishes strong semantic reasoning, it introduces a strict quantization bottleneck that degrades fine-grained visual details, often resulting in structurally accurate but artifact-heavy images. We introduce FlowShift, a parameter-efficient post-training framework that bypasses this bottleneck by converting pre-trained discrete UMMs into continuous rectified-flow generators. Instead of discarding the backbone's semantic priors, FlowShift replaces discrete visual interfaces with continuous latent projections and applies LoRA to the attention layers. To efficiently boost visual quality without semantic drift, we propose Dual-Objective Velocity Vector Distillation (DOVD), which aligns the model with a high-fidelity continuous teacher and ground-truth continuous trajectories. This approach drastically improves perceptual quality while preserving prompt adherence and reasoning. When applied to leading discrete UMMs, FlowShift significantly improves COCO-30K FID and aesthetic scores, while retaining over 99.5% of multimodal understanding capabilities. Ablations confirm DOVD drives generation quality. Ultimately, FlowShift achieves high-fidelity continuous generation without sacrificing pre-trained semantic grounding.

Figure 1. FlowShift converts discrete unified generators to continuous flow, improving fidelity while preserving compositional control.
Introduction
The landscape of multimodal AI is shifting toward Unified Multimodal Models (UMMs) that jointly handle understanding and generation within a single framework [3, 5, 30, 37, 43, 48, 53, 55]. Practical UMMs require strong semantic grounding and high-quality visual synthesis [33, 41], leading to two main architectural paradigms. Models such as Chameleon [37] and Show-o [48] map visual data into discrete tokens, integrating naturally with the autoregressive next-token prediction paradigm of large language models. Conversely, hybrid models like Transfusion [62] combine autoregressive text modeling with continuous image generation.
While discrete image representations integrate naturally with language models and preserve robust semantic reasoning, their reliance on quantization and categorical prediction fundamentally restricts fine-grained reconstruction [15, 61]. At a fixed codebook size and token resolution, multiple continuous visual features may map to the same discrete representation, limiting the exact preservation of fine-grained appearance variations such as texture and local shading. Consequently, models fail to learn the smooth gradients required for photorealism, producing structurally accurate but artifact-heavy images. Although expanding codebook sizes mitigates quantization error [15], it triggers vocabulary explosion and exacerbates the data-hungry nature of visual modeling, offering diminishing returns compared to continuous approaches [32, 62].
Conversely, continuous latent flows offer smoother visual geometry but commonly require massive retraining from scratch to establish semantic understanding [51, 60]. Their unbounded, high-dimensional spaces make learning strict semantic constraints (e.g., exact counting, rigid spatial relations) mathematically difficult, increasing the risk of out-of-distribution hallucinations. Furthermore, continuous models heavily optimized for visual appeal, often via Direct Preference Optimization (DPO) [27] frequently suffer from reward hacking, thus sacrificing faithful instruction adherence for scalar aesthetic rewards [14].
Figure 2 contrasts discrete and continuous state generative transport. In the discrete setting, samples lie on a finite or countable support and evolve through categorical transitions, yielding jump-like trajectories over quantized spaces. In contrast, continuous Flow Matching transports samples smoothly via a time-dependent velocity field in a Euclidean latent space, avoiding the quantization bottleneck and supporting fine-grained visual variation [20], finding applications in 3D generation, video generation, and video editing [11, 29, 38, 47, 54]. Motivated by this distinction, we propose a post-training framework FlowShift that converts a pre-trained discrete backbone from categorical token prediction to continuous latent-vector regression while preserving its semantic priors. We further introduce Dual-Objective Velocity Vector Distillation (DOVD), which transfers transport dynamics from a high-fidelity continuous teacher while maintaining the student’s semantic controllability. This enables improved high-frequency rendering and aesthetic fidelity without requiring full retraining or substantial parameter scaling. Our contributions are:
Figure 2. Discrete vs. continuous generative transport: discrete models jump between categorical states, while Flow Matching follows smooth ODE trajectories from prior to target.
We propose FlowShift, a parameter-efficient post-training framework that converts a pre-trained discrete Unified Multimodal Model (UMM) into a continuous rectified-flow generator without retraining the backbone, thereby preserving the semantic and compositional priors learned during discrete multimodal pretraining.
We introduce Dual-Objective Velocity Vector Distillation (DOVD), a distillation objective that operates over a shared continuous VAE latent space. By jointly optimizing for ground-truth flow matching and continuous velocity guidance from a frozen teacher model, DOVD overcomes the quantization bottleneck to recover fine-grained structural and photometric details.
We validate FlowShift across multiple discrete UMM backbones and continuous teachers, demonstrating consistent improvements in perceptual quality with minimal semantic degradation. Across the evaluated backbones, COCO-30K FID improves by 33.1%–42.5% and the aesthetic score by 1.50–1.78 points, while more than 99.5% of the original multimodal-understanding performance is retained across five benchmarks.
Related Work
Unified Multimodal Models (UMMs) have evolved through distinct architectural paradigms to jointly support text and image generation within a single framework:
1. Discrete autoregressive models. By mapping modalities into a shared discrete vocabulary, early models enabled unified training via standard next-token prediction. Chameleon [37] and Emu3 [43] pioneered this early-fusion of interleaved sequences, which Janus later refined by decoupling the visual encoding pathways for understanding and generation. Furthermore, Unified-IO 2 [22] expanded this discrete paradigm to encompass audio, actions, and structured outputs.
2. Discrete-diffusion models. To address the slow, uni-directional nature of autoregressive decoding, subsequent models adopted discrete diffusion. Show-o [48] combines autoregressive text and discrete diffusion over visual tokens within a single transformer. UniDisc [36] extended this by employing discrete diffusion as a joint formulation for both text and images, allowing for bidirectional denoising. MMaDA [50] similarly applies a modality-agnostic discrete diffusion objective to unify textual reasoning and visual synthesis. More recently, FUDOKI [42] replaces autoregressive modeling with kinetic-optimal discrete flow matching, enabling unified understanding and iterative image generation. UniDFlow [35] further unifies understanding, generation, and editing using task-specific adapters and preference alignment to preserve reasoning and controllability.
Continuous AR–diffusion
To achieve higher visual fidelity, the field is rapidly transitioning from shared discrete spaces to continuous visual dynamics. Transfusion [62] pairs text autoregression with continuous image diffusion for superior scaling, while MonoFormer [59] leverages a shared transformer for causal text and bidirectional continuous image diffusion. Similarly, recent architectures are directly upgrading their discrete predecessors: Janus-Flow [24] replaces the discrete autoregression of Janus [44] with continuous rectified flow, and Show-o2 [49] evolves Show-o from discrete diffusion to continuous flow matching over VAE representations.
While the progression to continuous hybrids highlights high-fidelity generation, training them from scratch demands exorbitant compute and vast datasets to establish semantic alignment. As an efficient alternative, our method FlowShift post-trains existing discrete models to capture aesthetic benefits of continuous flow-matching, bypassing the massive retraining costs of native hybrids while fully preserving the backbone’s original semantic properties.
Quantization Bottleneck
The shift toward continuous hybrids stems from the severe information loss inherent in discrete tokenization pipelines established by VQ-VAE [40] and VQGAN [6], which map continuous images into finite codebooks. To mitigate the resulting quantization error, recent advancements have focused on enhancing tokenizer capacity [34]. MAGVIT-v2 [52] introduced lookup-free quantization and massively expanded vocabularies to make discrete generation competitive with continuous diffusion. Further efforts like Finite Scalar Quantization [25] address codebook collapse.
Despite tokenizer improvements, discretization fundamentally throttles generative quality due to a granularity conflict: TokenFlow [26] shows that while semantics benefit from discrete compression, generation requires continuous information. Consequently, VTBench [17] confirms discrete tokenizers lag behind continuous VAEs in reconstructing fine textures and structural integrity. Furthermore, Transfusion [62] proves continuous diffusion scales significantly better than quantized next-token prediction. By employing DOVD to transfer smooth trajectory dynamics, our framework bypasses this bottleneck, recovering the fine-grained details and high-frequency textures that discrete vocabularies erase.
Method
Overview
Discrete models offer strong semantics but lack perceptual fidelity due to quantization and categorical training objectives. Conversely, continuous models achieve high fidelity but demand massive training for semantic alignment and often sacrifice instruction adherence when optimized for aesthetics. While existing continuous hybrids bridge this gap, resolving the trade-off by training from scratch remains computationally prohibitive.
We address this with FlowShift, a post-training framework that converts a pretrained discrete unified model into a continuous rectified-flow generator without updating pretrained weights (Figure 3). FlowShift comprises three components: (1) replacing discrete visual interfaces with continuous ones to enable velocity regression in a VAE latent space shared with a frozen teacher; (2) applying LoRA to attention projections while keeping all backbone and teacher parameters frozen; and (3) Dual-Objective Velocity Vector Distillation (DOVD), which combines flow-matching loss with teacher velocity distillation.
Figure 3. FlowShift: a frozen discrete UMM is adapted to continuous latent flow using lightweight trainable modules and dual-objective velocity distillation from a frozen continuous teacher.
Problem Formulation
Let be a pretrained UMM with frozen parameters , which encodes an image into categorical tokens using a tokenizer of codebook size . For a model that generates images by autoregressively predicting and decoding these tokens, the objective can be written as
where denotes the prompt. This categorical formulation has two limitations: high-frequency content mapped to codebook centroids is unrecoverable by any decoder, and cross-entropy loss penalizes all incorrect codes equally.
We instead adopt the rectified-flow formulation [20]. Given a VAE with encoder , decoder , and latents , noise and data are connected by the linear interpolant .
where and (with as data and as noise). The conditional velocity is directly given by . By regressing this target, the model learns the marginal velocity field, denoted as . This field defines an Ordinary Differential Equation (ODE), . To generate samples, this ODE is integrated from to and the final latent is decoded via . Unlike Eq. (1), this Euclidean regression provides gradients directly proportional to the prediction error.
We formulate our problem as follows: given a frozen discrete unified backbone , a frozen rectified-flow teacher , and an image-text dataset , we learn a lightweight parameter set (where ) such that the augmented model (i) defines a continuous velocity field with improved perceptual fidelity; (ii) inherits semantic and compositional control via the backbone’s native language pathway; and (iii) requires only parameter-efficient post-training, bypassing full multimodal pretraining.
Hereafter, denotes the backbone equipped with the trainable modules in , and the frozen parameters are left implicit. The next subsection presents how is constructed and trained.
FlowShift
FlowShift utilizes two strictly frozen pretrained networks: the unified backbone , which forms the frozen core of the student and the rectified-flow teacher . The trainable parameters are a continuous input projector , a timestep layer , attention LoRA adapters, and a velocity output head .
For an image-prompt pair , the teacher’s frozen VAE encoder acts as a shared interface, extracting the continuous clean latent . Following, noise is added to produce , which is then passed to both the student and teacher . Sharing the exact clean latent, noise, and timestep ensures both networks predict velocities at the identical trajectory point.
Simultaneously, each model processes the prompt through its native text encoder to yield conditioning signals (student) and (teacher). We share VAE latents to enable pointwise comparison of the teacher’s and student’s velocity predictions. Text encoders remain separate because each model requires its native conditioning distribution; cross-model embeddings could corrupt supervision and weaken the student’s semantic grounding.
Of the two networks, only the student requires architectural modification; the teacher is inherently a continuous velocity network that directly predicts from the shared noisy latent. Since the frozen student backbone natively expects and predicts discrete tokens, we adapt its forward pass at three distinct points: the input interface, attention computation, and output interface.
Input Adaptation. At input, is patchified (size ) and projected to model width via trainable layer , producing continuous tokens that match the discrete sequence length of Eq. (1). By mapping these to the original discrete positional indices, we directly reuse the backbone’s pretrained spatial priors. Simultaneously, timestep is injected by adding its sinusoidally transformed embedding, followed by the two-layer network , to every visual token.
Backbone Adaptation. The visual tokens then enter the frozen backbone together with . Inside , we attach LoRA adapters [10] to the query, key, value, and output projections of every self-attention layer. For layer and projection type , the adapted projection is , where is frozen, and are trainable low-rank matrices, is the LoRA rank, and is the LoRA scaling coefficient. We restrict adaptation to attention projections because the representation transition primarily changes how continuous visual tokens interact with one another and with the textual condition. Updates to the query and key projections adapt the attention geometry, while updates to the value and output projections adapt how continuous visual information is aggregated and propagated. Freezing the feed-forward layers and remaining backbone parameters preserves pretrained semantic representations and reduces semantic drift.
Output Adaptation. At the output, the hidden states at the visual positions are mapped by the linear head and unpatchified into the student prediction , replacing the original softmax head over the codebook. This completes the architecture. Trainable parameters are therefore given by
All parameters of the source text-conditioning pathway, unified backbone, teacher text encoder, teacher transformer, and VAE remain frozen.
Training Objective: With both networks predicting velocity vectors over the same noisy latent, relying on a single training signal is suboptimal. Standard conditional flow matching ties the student to the target data distribution, but recovering high-frequency generation behavior purely from data is highly inefficient. Conversely, pure teacher distillation efficiently transfers continuous rendering dynamics but risks transferring the teacher’s instruction-adherence errors.
Dual-Objective Velocity Vector Distillation (DOVD) therefore jointly optimizes the student against the analytic rectified-flow target and the frozen teacher prediction. With expectations taken over , , and , the objective is
where denotes stop-gradient, weights the distillation term. The two objectives serve different purposes. The Flow Matching (FM) loss, , preserves direct supervision from the prompt–image training pair and prevents the student from merely imitating the teacher. In contrast, the Velocity Distillation (VD) loss, , regularizes the student toward the smooth velocity field learned by a high-capacity continuous generator. Together, they help the student inherit continuous rendering without relying solely on the teacher for visual or semantic supervision.
Experiments
We train FlowShift on a curated corpus of approximately 6 million image-text pairs sourced from LAION-20M Recap, OpenGPT-4o-Image, and DRAGON. To ensure fair and rigorous comparisons against all discrete and continuous baselines, we maintain strictly consistent evaluation protocols across all models, utilizing identical evaluation prompt sets, standardized sampling steps, and matching generation resolutions. During inference, generations employ an Euler discrete scheduler with a uniform classifier-free guidance scale of 8.0. Comprehensive details regarding the training data, evaluation benchmarks, metrics, and implementation settings are provided in Appendix § 10.
Quantitative Results
Image Generation
Table 1 compares FlowShift-adapted models (listed under DFM → CFM Unified Models) against three baselines: native continuous generators (CFM based Native Image Generation Models), scratch-trained continuous UMMs (CFM based Unified Models), and discrete UMMs (DFM based Unified Models). Results show FlowShift effectively balances semantic control (prompt adherence) and perceptual fidelity (visual quality).
| Baselines | Params (B) | Inf. Time (s) | GenEval [7] Accuracy ↑ | GenEval [7] FID ↓ | COCO-30K [18] DINOv2-FID ↓ | COCO-30K [18] FID ↓ | Aesthetic-Eval [58] CLIPScore ↑ | Aesthetic-Eval [58] Aesthetic Score ↑ |
| Continuous Flow Matching (CFM) based Native Image Generation Models | ||||||||
| Flux1 [12] | 14 | 16 | 0.66 | 9.15 | 10.43 | 39.41 | 33.42 | 5.51 |
| FLUX.2 [klein] 9B [13] | 9 | 12 | 0.75 | 8.92 | 9.17 | 38.47 | 35.41 | 6.28 |
| Qwen-Image RL [45] | 22 | 21 | 0.88 | 8.55 | 9.32 | 39.09 | 33.98 | 6.43 |
| Flux-2 32B [13] | 32 | 33 | 0.89 | 7.12 | 7.77 | 33.14 | 39.21 | 7.56 |
| Z-Image [2] | 6 | 8 | 0.87 | 8.18 | 8.28 | 37.79 | 36.29 | 5.85 |
| Continuous Flow Matching (CFM) based Unified Models | ||||||||
| OmniGen2 [46] | 3+4 | 7 | 0.85 | 8.89 | 9.48 | 40.08 | 37.32 | 5.18 |
| BAGEL [5] | 7+7 | 12 | 0.88 | 9.32 | 9.19 | 41.14 | 36.61 | 5.92 |
| MammothModa-2 [28] | 8+3+2 | 11 | 0.87 | 10.02 | 9.95 | 40.44 | 35.92 | 6.05 |
| Discrete Flow Matching (DFM) based Unified Models | ||||||||
| MUDDIT [31] | 2 | 5 | 0.67 | 10.11 | 16.68 | 50.16 | 33.36 | 3.15 |
| FUDOKI [42] | 2 | 7 | 0.81 | 16.16 | 13.29 | 48.93 | 38.77 | 4.42 |
| UniDFlow-4B [35] | 4 | 5 | 0.95 | 13.33 | 15.23 | 44.78 | 38.01 | 4.43 |
| DFM → CFM Unified Models | ||||||||
| FlowShift (FUDOKI) | 2 | 8 | 0.77−0.04 | 9.29−6.87 | 10.15−3.14 | 38.42−10.51 | 37.17−1.60 | 5.92+1.50 |
| FlowShift (UniDFlow-4B) | 4 | 7 | 0.92−0.03 | 8.92−4.41 | 9.53−5.70 | 38.32−6.46 | 38.23+0.22 | 6.21+1.78 |
Table 1. Image generation performance, model size, and inference time. Subscript values denote the score change of each FlowShift model relative to its corresponding DFM source model.
As baselines demonstrate, discrete unified models excel in compositional fidelity (e.g., UniDFlow-4B [35] scores 0.95 on GenEval) but suffer on perceptual metrics due to quantization-induced high-frequency loss. Conversely, native continuous models leverage smooth latent geometry for superior visuals. For instance, FLUX-2 32B [13] achieves the best COCO-30K FID (7.12) and Aesthetic scores (7.56) yet struggles with complex instructions (GenEval 0.89).
When observing recent continuous unified models trained from scratch, such as OmniGen2 [46] and BAGEL [5], the empirical results highlight the difficulty of optimizing textual reasoning and continuous generation simultaneously. Without a discrete semantic foundation, the data shows these models compromise on both fronts: they fail to match the rigorous logical grounding of discrete models (with GenEval scores hovering between 0.85 and 0.88) and fall short of the pristine visual polish of native continuous generators (yielding Aesthetic scores below 6.0).
Rather than relearning these capabilities from scratch, FlowShift transfers a learned discrete generation policy into a continuous space. By restricting adaptation to visual generation and leveraging teacher trajectory supervision, the model preserves its semantic backbone while improving perceptual modeling. Bypassing the quantization bottleneck yields substantial improvements in visual fidelity across architectures: FUDOKI [42] improves COCO-30K FID from 16.16 to 9.29 and Aesthetic Score from 4.42 to 5.92, while UniDFlow-4B improves FID from 13.33 to 8.92 and Aesthetic Score from 4.43 to 6.21. Compositional performance remains largely preserved, with GenEval scores decreasing by 0.04 for FUDOKI and 0.03 for UniDFlow-4B. These results suggest that our framework improves visual fidelity and aesthetics while largely retaining discrete compositional priors.
Multimodal Understanding
Table 2 evaluates whether adapting a discrete flow-matching (DFM) unified model to a continuous flow-matching (CFM) space compromises pretrained semantic understanding. The DFM→CFM transition introduces only minor performance changes. For UniDFlow-4B, performance decreases by 0.28, 0.24, 0.30, 0.70, and 0.40 percentage points on MMBench, MMMU, MM-Vet, MathVista, and MMVP, respectively. Averaging the relative changes across these five benchmarks yields approximately 0.46% degradation, corresponding to 99.54% retention of the original performance, with relative degradation below 1% on each benchmark. FUDOKI similarly exhibits approximately 0.27% average relative degradation, corresponding to 99.73% retention based on the displayed scores, while improving on MMVP from 67.72 to 68.01. Overall, results indicate that the representation transition modifies visual generation while largely preserving pretrained multimodal understanding and performance on reasoning tasks.
| Baselines | MMBench [21] | MMMU [57] | MM-VET [56] | MathVista [23] | MMVP [39] |
| Continuous Flow Matching (CFM) based Unified Models | |||||
| BAGEL [5] | 85.00 | 55.30 | 67.20 | 73.10 | 69.30 |
| OmniGen2 [46] | 79.10 | 53.10 | 61.80 | 69.42 | 68.26 |
| MammothModa-2 [28] | 86.60 | 71.23 | 79.40 | 81.80 | 77.50 |
| Discrete Flow Matching (DFM) based Unified Models | |||||
| MUDDIT [31] | 55.48 | 49.87 | 40.14 | 70.15 | 66.66 |
| Semantic Preservation under DFM → CFM Conversion | |||||
| FUDOKI (Source) [42] | 81.27 | 58.01 | 65.23 | 74.17 | 67.72 |
| FlowShift (FUDOKI)99.73% | 81.19 | 58.00 | 64.91 | 73.29 | 68.01 |
| UniDFlow-4B (Source) [35] | 91.20 | 74.50 | 82.70 | 85.90 | 80.20 |
| FlowShift (UniDFlow-4B)99.54% | 90.92 | 74.26 | 82.40 | 85.20 | 79.80 |
Table 2. Multimodal understanding performance. Subscript values beside FlowShift models denote the average percentage of source-model performance retained after conversion.
Qualitative Results
Figure 4 and Figure 5 compare the original discrete UniDFlow-4B and FUDOKI models with their continuous counterparts obtained using our FlowShift respectively. Across diverse prompts and complex scenes, the continuous variants consistently exhibit improved local fidelity while preserving the semantic composition of the corresponding discrete models. For FUDOKI (food example), the continuous variants better resolve the croissant’s flaky layers. In the UniDFlow portrait example, continuous generation produces more refined facial details, hair strands, and accessories. Similar improvements are visible for the hummingbird, where feather structure, specular coloration, and flower details become sharper and more coherent. Importantly, such improvements are observed for both FUDOKI and UniDFlow-4B rather than being specific to a single backbone.

Figure 4. Qualitative comparison of discrete and continuous generation with UniDFlow-4B.

Figure 5. FUDOKI (Discrete) and FUDOKI (Continuous). croissant beside a ceramic cup of cappuccino | rustic wooden table | crisp flaky layers | butter highlights | leaf-shaped latte-art pattern | Soft morning light passes through a nearby window, illuminating rising steam and subtle wood grain. peregrine falcon banking sharply above sunlit desert cliffs | wings fully extended | subtle motion blur at the tips | detailed talons | sharp hooked beak | focused eye | Warm low-angle sunlight should illuminate the underside of the bird | authentic natural colors. nighttime photograph | tropical beach | glowing with bioluminescent plankton | breaking wave should produce electric-blue light along the foam | wet sand reflecting the stars | faint footprints | dark palm silhouettes | small shells | clear Milky Way overhead. underwater image of a translucent giant jellyfish | finely textured bell | branching internal structures | hundreds of thin trailing tentacles | bioluminescent violet and cyan patterns should glow from within its body and illuminate nearby suspended particles | soft volumetric beams | deep black negative space. macro photograph of monarch butterfly | orange-and-black wings with intricate scales | delicate veins | slight moisture | naturally crumpled edges | fine hairs on the butterfly’s body | droplets of morning dew on the surrounding leaves | creamy green bokeh. lone explorer walking through a vast blue glacier cave | walls should contain layered ice | trapped air bubbles | mineral streaks | cracks | translucent sections illuminated by daylight filtering through the surface | scattered ice crystals | reflective wet floor | frost accumulating on the explorer’s clothing. Figure 5. Qualitative comparison of discrete and continuous generation of FUDOKI.
Additionally, Figure 6 qualitatively confirms that the DFMCFM transition using FlowShift preserves the model’s multimodal understanding, retaining strong performance across diverse reasoning and counting tasks.
Figure 6. VQA/Image understanding using continuous UniDFlow-4B. The examples demonstrate that FlowShift successfully preserves complex reasoning capabilities despite the transition from a discrete to a continuous visual generation space.
Ablations
Table 3 evaluates the key design choices of FlowShift: (i) the necessity of distillation beyond standard flow matching; (ii) sensitivity to teacher choice and capacity; and (iii) the role of parameter-efficient attention adaptation in accommodating representation shift while preserving pretrained backbone semantics. By default, we utilize UniDFlow-4B as the discrete unified backbone, Qwen Image as the continuous teacher, and LoRA adapters for adaptation.
| Variants | GenEval Accuracy ↑ | COCO-30K FID ↓ | Aesthetic-Eval CLIP Score ↑ | Aesthetic Score ↑ | MMBench Accuracy ↑ | MMMU Accuracy ↑ |
| Main Baseline (Qwen as teacher, LoRA adapters, and UniDFlow as base unified model) | ||||||
| Main Baseline | 0.92 | 8.92 | 38.23 | 6.21 | 90.92 | 74.26 |
| Impact of Distillation | ||||||
| No Teacher Model | ||||||
| Impact of Different Teacher Models | ||||||
| Flux 1.0 [12] | ||||||
| Flux 2 4B [13] | ||||||
| Flux 2 9B [13] | ||||||
| Flux 2 32B [13] | ||||||
| Full Fine-Tuning vs. LoRA | ||||||
| No Teacher Model and No Adapters | ||||||
| No Adapters |
Table 3. Ablation study on the impact of distillation, teacher models, and parameter-efficient adaptation. Subscript values denote changes relative to the Main Baseline.
Impact of velocity distillation. Removing the continuous teacher to train the student solely against the rectified-flow target (using without the velocity-distillation term ) consistently degrades all generation metrics: GenEval drops from 0.92 to 0.89, COCO-30K FID increases from 8.92 to 10.69, CLIPScore decreases from 38.23 to 36.31, and aesthetic score falls from 6.21 to 5.58. These results validate DOVD: while both terms supervise velocity, regresses the conditional velocity , forcing the student to recover the marginal field from a high-variance signal where noise cancels only in expectation. Conversely, directly transfers continuous rendering behavior by matching a learned, low-variance field rather than re-estimating it from data. Consequently, isolating reduces perceptual quality (Table 3).
Impact of the teacher models. Replacing the default teacher with Flux-1.0 and Flux-2 variants yields nearly unchanged semantic performance: GenEval varies only between 0.91–0.92, MMBench between 90.23–90.58, and MMMU between 74.17–74.67. Conversely, generation quality is more teacher-sensitive, with Flux-2 32B achieving the best COCO-30K FID (8.33) and aesthetic score (6.24). This confirms our hypothesis: the teacher primarily transfers continuous rendering dynamics, while the frozen unified backbone preserves semantic grounding. Because no single teacher is uniformly optimal, we adopt Qwen Image to provide the best overall balance between semantic retention and perceptual quality.
Full fine-tuning vs. LoRA. We examine whether the representation transition requires full-backbone updates. Full fine-tuning yields only marginal generation gains (COCO-30K FID 8.92 8.76, CLIPScore 38.23 38.52), while substantially degrading semantic understanding, with MMBench dropping from 90.92 to 86.01 and MMMU from 74.26 to 69.28. Similar degradation without teacher distillation indicates semantic drift from unrestricted updates. Thus, LoRA provides sufficient flexibility for the discrete-to-continuous representation shift while better preserving pretrained semantic knowledge.
Conclusion
We introduce a framework (FlowShift) to transform pretrained discrete unified multimodal models into continuous rectified-flow generators without relearning established semantic knowledge. By freezing the backbone and adapting only lightweight continuous interfaces and low-rank attention modules, our approach transfers high-fidelity rendering dynamics from a continuous teacher via Dual-Objective Velocity Vector Distillation. Evaluations confirm FlowShift yields improvements in visual distribution similarity and aesthetics with negligible degradation in precise compositional control. Ultimately, this establishes a scalable pathway for future unified architectures to achieve superior continuous visual synthesis while retaining the rigorous logical grounding of discrete language modeling.
Acknowledgments
This research is based upon work supported by the NSF CAREER Award #2542328 and the U.S. Defense Advanced Research Projects Agency (DARPA) under award HR001125C0303. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of NSF, DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

Figure 7. Qualitative results generated by FlowShift.
Experimental Details
Training Data
For training, we construct a large-scale image–text corpus from three datasets: LAION-20M Recap [16], OpenGPT-4o-Image [4], and DRAGON [1], comprising approximately 6M image-text pairs in total. LAION-20M Recap and OpenGPT-4o-Image provide high-quality images paired with detailed, information-rich captions, offering strong supervision for learning fine-grained text-image correspondence. We further incorporate DRAGON, a synthetically curated dataset generated using text-to-image models, to increase the diversity of visual concepts, compositions, and generation patterns represented during training.
Implementation Details
We train FlowShift on 8 NVIDIA H100 GPUs with 80 GB memory each. Training uses a per-GPU batch size of 32 with 4 gradient-accumulation steps, resulting in an effective batch size of 1,024. We optimize the model for 5K steps using the Muon [19] optimizer with an initial learning rate of (). The learning rate is linearly warmed up for the first 2K steps and subsequently decayed using a cosine annealing schedule. For an efficient parameter adaptation, we employ LoRA with rank () and scaling factor (). Image-generation captions are truncated to a maximum sequence length of 500 tokens.
During training, we preserve the native aspect ratios of images rather than resizing all samples to a fixed resolution. This exposes the model to diverse spatial configurations and improves the generalization of its rotary positional embeddings (RoPE) across different resolutions and aspect ratios.
For sampling, we use an Euler discrete scheduler with a classifier-free guidance (CFG) scale of (8.0), a true-CFG scale of (4.0), and negative-prompt guidance with a scale of (4.0). Both training and inference are implemented using the Hugging Face diffusers framework.
Evaluation Datasets and Metrics
We evaluate our model across both image generation and multimodal understanding, as summarized in Table 4. For image generation, we consider three complementary benchmarks. GenEval [7] consists of 553 compositional prompts designed to evaluate object presence, counting, color attribution, spatial relationships, and attribute binding. COCO-30K [18] contains 30,000 caption-based generation prompts sampled from MS-COCO and is used to measure distribution-level image quality using FID [9] and DINOv2-FID. We additionally evaluate on Aesthetic-Eval [58], comprising 2,781 high-resolution image–text pairs, to assess perceptual quality, text–image alignment, and aesthetic fidelity using FID, CLIPScore [8], and Aesthetic Score [58].
| Benchmark | # Examples | Format | Task | Evaluation Metric |
| Image Generation | ||||
| GenEval [7] | 553 | Text Prompt | Compositional T2I | GenEval Acc. |
| COCO-30K [18] | 30,000 | Image Caption | Text-to-Image | FID , DINOv2-FID |
| Aesthetic-Eval [58] | 2,781 | Image–Text Pair | High-Resolution T2I | FID , CLIPScore , Aesthetic Score |
| Multimodal Understanding | ||||
| MMBench v1.1 [21] | 3,217 | Multiple Choice | General Multimodal QA | CircularEval Acc. |
| MMMU [57] | 11,550 | MCQ + Open-ended | Expert-level Reasoning | Overall Acc. |
| MM-Vet [56] | 218 | Open-ended | Integrated Multimodal QA | GPT-4 Score |
| MathVista [23] | 6,141 | MCQ + Open-ended | Visual Mathematical Reasoning | Accuracy |
| MMVP [39] | 300 | Multiple Choice | Fine-grained Visual Perception | Pair Accuracy |
Table 4. Summary of evaluation benchmarks. We report the number of evaluation examples, input–output format, task category, and evaluation metric. For image-generation benchmarks, each example denotes a text prompt, while for multimodal-understanding benchmarks, each example corresponds to a visual question-answering instance. Arrows indicate whether higher () or lower () values are preferred.
For multimodal understanding, we evaluate on MMBench v1.1 [21], which contains 3,217 multiple-choice visual reasoning questions, and MMMU [57], comprising
11,550 expert-level multimodal questions spanning both multiple-choice and open-ended formats. We further evaluate on MM-Vet [56], which contains 218 open-ended questions covering integrated vision-language capabilities, and MathVista [23], which contains 6,141 multiple-choice and free-form questions requiring visual mathematical reasoning. Finally, MMVP [39] contains 300 multiple-choice questions constructed from 150 visually challenging image pairs and evaluates fine-grained visual perception. Together, these benchmarks measure whether continuous adaptation preserves the semantic perception and reasoning capabilities of the unified backbone while improving its image-generation ability.
References
- [1]Giulia Bertazzini, Daniele Baracchi, Dasara Shullani, Isao Echizen, and Alessandro Piva. Dragon: A large-scale dataset of realistic images generated by diffusion models. arXiv preprint arXiv:2505.11257, 2025. 12arxiv.org/abs/2505.11257
- [2]Huanqia Cai, Sihan Cao, Ruoyi Du, Peng Gao, Aiming Hao, Steven Hoi, Zhaohui Hou, Shijie Huang, Dengyang Jiang, Yuming Jiang, et al. Z-image: An efficient image generation foundation model with single-stream diffusion transformer. arXiv preprint arXiv:2511.22699, 2025. 5arxiv.org/abs/2511.22699
- [3]Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811, 2025. 1arxiv.org/abs/2501.17811
- [4]Zhihong Chen, Xuehai Bai, Yang Shi, Chaoyou Fu, Huanyu Zhang, Haotian Wang, Xiaoyan Sun, Zhang Zhang, Liang Wang, Yuanxing Zhang, et al. Opengpt-4o-image: A comprehensive dataset for advanced image generation and editing. arXiv preprint arXiv:2509.24900, 2025. 12arxiv.org/abs/2509.24900
- [5]Chaouri Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihou Yu, Xiaonian Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683, 2025. 1, 5, 6arxiv.org/abs/2505.14683
- [6]Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In 2021 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 12868–12878. IEEE, 2021. 3arxiv.org/abs/2012.09841
- [7]Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36:52132–52152, 2023. 5, 13arxiv.org/abs/2310.11513
- [8]Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 7514–7528, 2021. 13
- [9]Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. 13
- [10]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 4arxiv.org/abs/2106.09685
- [11]Adheesh Sunil Juvekar, Onkar Kishor Susladkar, Kiet A. Nguyen, Muntasir Wahed, Nabeel Bashir, Xiaona Zhou, Tianjiao Yu, Vedant Shah, and Ismini Lourentzou. One editor, many edits: A unified training-free framework for diverse video editing. arXiv preprint arXiv:2609.04190, 2026. 2
- [12]Black Forest Labs. Flux.1, 2024. 12 billion parameter rectified flow transformer. 5, 8
- [13]Black Forest Labs. FLUX.2: Frontier Visual Intelligence. https://bfl.ai/blog/flux-2, 2025. 5, 6, 8
- [14]Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, et al. Flux.1 kontext: Flow matching for in-context image generation and editing in latent space. arXiv preprint arXiv:2506.15742, 2025. 2arxiv.org/abs/2506.15742
- [15]Jindong Li, Yali Fu, Jiahong Liu, Linxiao Cao, Wei Ji, Menglin Yang, Irwin King, and Ming-Hsuan Yang. Discrete tokenization for multimodal llms: A comprehensive survey. arXiv preprint arXiv:2507.20290, 2025. 2
- [16]Zejian Li, Chenye Meng, Yize Li, Ling Yang, Shengyuan Zhang, Jiarui Ma, Jiayi Li, Guang Yang, Changyuan Yang, Zhiyuan Yang, et al. Laion-sg: An enhanced large-scale dataset for training complex image-text models with structural annotations. arXiv preprint arXiv:2412.08580, 2024. 12arxiv.org/abs/2412.08580
- [17]Huawei Lin, Tong Geng, Zhaozhuo Xu, and Weijie Zhao. Vt-bench: Evaluating visual tokenizers for autoregressive image generation. arXiv preprint arXiv:2505.13439, 2025. 3arxiv.org/abs/2505.13439
- [18]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014. 5, 13arxiv.org/abs/1405.0312
- [19]Jingyuan Jiang, Jian Lian, Su Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for llm training. arXiv preprint arXiv:2502.16982, 2025. 13arxiv.org/abs/2502.16982
- [20]Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022. 2, 4arxiv.org/abs/2209.03003
- [21]Yuan Liu, Haodong Duan, Yuhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer, 2024. 5, 13arxiv.org/abs/2307.06281
- [22]Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26429–26445. IEEE, 2024. 2arxiv.org/abs/2312.17172
- [23]Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuanyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jinfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations, pages 23439–23554, 2024.
- [24]Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhendong Xie, Haowei Zhang, Xingkai Yu, et al. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7739–7751. IEEE, 2025.arxiv.org/abs/2411.07975
- [25]Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. In International Conference on Learning Representations, pages 51772–51783, 2024.arxiv.org/abs/2309.15505
- [26]Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2545–2555. IEEE, 2025.
- [27]Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023.arxiv.org/abs/2305.18290
- [28]Tao Shen, Xin Wan, Taicai Chen, Rui Zhang, Junwen Pan, Dawei Lu, Fanding Lei, Zhilin Lu, Yunfei Yang, Chen Cheng, et al. Mammothmoda2: A unified ar-diffusion framework for multimodal understanding and generation. arXiv preprint arXiv:2511.18262, 2025.arxiv.org/abs/2511.18262
- [29]Ying Shen, Jerry Xiong, Tianjiao Yu, and Ismini Lourentzou. PHANTOM: Physics-infused video generation via joint modeling of visual and latent physical dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11185–11194, 2026.arxiv.org/abs/2604.08503
- [30]Ying Shen, Zhiyang Xu, Jiuhai Chen, Shizhe Diao, Jiaxin Zhang, Yuguang Yao, Joy Rimchala, Ismini Lourentzou, and Lifui Huang. LaTTe-Flow: Layerwise timestep-expert flow-based transformer. In International Conference on Machine Learning, 2026.arxiv.org/abs/2506.06952
- [31]Qingyu Shi, Jinbin Bai, Zhuoran Zhao, Wenhao Chai, Kaidong Yu, Jianzhong Wu, Yunhai Tong, Xiangtai Li, Xuelong Li, and Shuicheng Yan. Muddit: Liberating generation beyond text-to-image with a unified discrete diffusion model. arXiv preprint arXiv:2505.23606, 2025.arxiv.org/abs/2505.23606
- [32]Mustafa Shukor, Enrico Fini, Victor Guilherme Turrisi da Costa, Matthieu Cord, Joshua Susskind, and Alaaeldin El-Nouby. Scaling laws for native multimodal models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12–23, 2025.arxiv.org/abs/2504.07951
- [33]Onkar Susladkar, Dong-Hwan Jang, Tushar Prakash, Adheesh Juvekar, Vedant Shah, Ayush Barik, Nabeel Bashir, Muntasir Wahed, Ritish Shrir ao, and Ismini Lourentzou. RewardFlow: Generate images by optimizing what you reward. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026.arxiv.org/abs/2604.08536
- [34]Onkar Susladkar, Tushar Prakash, Adheesh Juvekar, Kiet A. Nguyen, Dong-Hwan Jang, Inderjit S Dhillon, and Ismini Lourentzou. PyraTok: Language-aligned pyramidal tokenizer for video understanding and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026.arxiv.org/abs/2601.16210
- [35]Onkar Kishor Susladkar, Tushar Prakash, Gayatri Deshmukh, Kiet A. Nguyen, Jiaxun Zhang, Adheesh Sunil Juvekar, Tianshu Bao, Lin Chai, Sparsh Mittal, Inderjit S Dhillon, and Ismini Lourentzou. Best of both worlds: Multimodal reasoning and generation via unified discrete flow matching. In International Conference on Machine Learning, 2026.arxiv.org/abs/2602.12221
- [36]Alexander Swerdlow, Mihir Prabhudesai, Siddharth Gandhi, Deepak Pathak, and Katerina Fragkiadaki. Unified multimodal discrete diffusion. arXiv preprint arXiv:2503.20853, 2025.arxiv.org/abs/2503.20853
- [37]Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024.arxiv.org/abs/2405.09818
- [38]Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025.arxiv.org/abs/2503.20314
- [39]Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyles wide shut? exploring the visual shortcomings of multimodal llms. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9568–9578. IEEE, 2024.arxiv.org/abs/2401.06209
- [40]Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems, 2017.arxiv.org/abs/1711.00937
- [41]Kavana Venkatesh, Yusuf Dalva, Ismini Lourentzou, and Pinar Yanardag. RAVEL: Rare concept generation and editing via graph-driven relational guidance. In Advances in Neural Information Processing Systems, 2026.
- [42]Jin Wang, Yao Lai, Aoxue Li, Shifeng Zhang, Jiacheng Sun, Ning Kang, Chengyue Wu, Zhenguo Li, and Ping Luo. Fudoki: Discrete flow-based unified understanding and generation via kinetic-optimal velocities. Advances in Neural Information Processing Systems, 38:81402–81440, 2025.arxiv.org/abs/2505.20147
- [43]Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024.arxiv.org/abs/2409.18869
- [44]Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12966–12977. IEEE, 2025.arxiv.org/abs/2410.13848
- [45]Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al. Qwen-image technical report. arXiv preprint arXiv:2508.02324, 2025.arxiv.org/abs/2508.02324
- [46]Chenyuan Wu, Jiahao Wang, Pengfei Zheng, Ruiran Yan, Shitao Xiao, Xin Luo, Yueze Wang, Wanli Li, Xiyan Jiang, Yexin Liu, et al. Omnigen2: Towards instruction-aligned multimodal generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21964–21975, 2026. 5, 6arxiv.org/abs/2506.18871
- [47]Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3D latents for scalable and versatile 3D generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21469–21480, 2025. 2arxiv.org/abs/2412.01506
- [48]Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. In International Conference on Learning Representations, pages 28240–28264, 2025. 1, 2arxiv.org/abs/2408.12528
- [49]Jinheng Xie, Zhenheng Yang, and Mike Zheng Shou. Show-o2: Improved native unified multimodal models. Advances in Neural Information Processing Systems, 38:47490–47518, 2025. 3arxiv.org/abs/2506.15564
- [50]Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models. Advances in Neural Information Processing Systems, 38:138867–138907, 2025. 3arxiv.org/abs/2505.15809
- [51]Yan Yang, Haochen Tian, Yang Shi, Wulin Xie, Yi-Fan Zhang, Yuhao Dong, Yibo Hu, Liang Wang, Ran He, Caifeng Shan, et al. A survey of unified multimodal understanding and generation: Advances and challenges. Authorea Preprints, 2025. 2
- [52]Lijun Yu, José Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kiyoh Suk Hohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuyue Gu, Alexander G Hauptmann, et al. Language Model Beats Diffusion – Tokenizer is Key to Visual Generation. In International Conference on Learning Representations, pages 765–783, 2024. 3arxiv.org/abs/2310.05737
- [53]Tianjiao Yu, Xinzhou Li, Yifan Shen, Yuanzhe Liu, and Ismini Lourentzou. CoRe3D: Collaborative reasoning as a foundation for 3d intelligence. arXiv preprint arXiv:2512.12768, 2025. 1arxiv.org/abs/2512.12768
- [54]Tianjiao Yu, Xinzhou Li, Yifan Shen, Onkar Susladkar, Yuanzhe Liu, Xiaona Zhou, and Ismini Lourentzou. SILSA: Sliding-window slice latents for topology-preserving high-resolution 3d generation. In Advances in Neural Information Processing Systems, 2026. 2arxiv.org/abs/2610.02201
- [55]Tianjiao Yu, Xinzhou Li, Yifan Shen, Onkar Susladkar, Yuanzhe Liu, Xiaona Zhou, and Ismini Lourentzou. ELSA3D: Elastic semantic anchoring for unified 3d understanding and generation. In Advances in Neural Information Processing Systems, 2026. 1arxiv.org/abs/2607.06565
- [56]Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. 5, 13arxiv.org/abs/2308.02490
- [57]Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9556–9567, 2024. 5, 13arxiv.org/abs/2311.16502
- [58]Jinjin Zhang, Qiuyu Huang, Junjie Liu, Xiefan Guo, and Di Huang. Diffusion-4k: Ultra-high-resolution image synthesis with latent diffusion models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23464–23473. IEEE, 2025. 5, 13DOI
- [59]Chuyang Zhao, Yuxing Song, Wenhao Wang, Haocheng Feng, Errui Ding, Yifan Sun, Xinyan Xiao, and Jingdong Wang. Monoformer: One transformer for both diffusion and autoregression. arXiv preprint arXiv:2409.16280, 2024. 3arxiv.org/abs/2409.16280
- [60]Shanshan Zhao, Xinjie Zhang, Jintao Guo, Jiakui Hu, Lunhao Duan, Minghao Fu, Yong Xien Chng, Guo-Hua Wang, Qing-Guo Chen, Zhao Xu, et al. Unified multimodal understanding and generation models: Advances, challenges, and opportunities. arXiv preprint arXiv:2505.02567, 2025. 2arxiv.org/abs/2505.02567
- [61]Peng Zheng, Junke Wang, Yi Chang, Yizhou Yu, Rui Ma, and Zuxuan Wu. Rethinking discrete tokens: Treating them as conditions for continuous autoregressive image synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17390–17400, 2025. 2arxiv.org/abs/2507.01756
- [62]Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. In International Conference on Learning Representations, pages 6446–6469, 2025. 2, 3arxiv.org/abs/2408.11039


