betaFeedback

WorksComputer Science

ViLFlow: Parallel Vision-Language Generation with Continuous Latent Flows

Abstract

Vision-language generation has predominantly relied on autoregressive and discrete diffusion models. Autoregressive models generate tokens sequentially, while discrete diffusion models typically predict tokens independently within each denoising step. We propose ViLFlow (Vision-Language Flow), a continuous-flow vision-language model that combines joint modeling of output embeddings with parallel token decoding. ViLFlow adapts pretrained continuous language generation to visual question answering and image captioning through learned image guidance and prediction-aware token decoding. A visually adapted teacher provides image-guided predictions as additional denoising supervision. The denoiser learns to reproduce these predictions directly, retaining one image-conditioned prediction per generation step. We train the token decoder on denoiser predictions and paired reference text to improve the conversion of continuous outputs into words. The decoder can be adapted independently, improving answering and captioning without further denoiser training. For captioning, it also supplies text feedback during late denoising, allowing decoded words to inform subsequent refinement. ViLFlow outperforms autoregressive and masked-diffusion baselines by up to 8.46 percentage points on VQAv2 and 4.50 points on GQA without GQA training. On COCO Karpathy test, it improves CIDEr by 16.25 over masked diffusion and 17.81 over sampled autoregression.

Introduction

Vision–language models have conventionally followed two paradigms. Autoregressive models [21] sequentially generate tokens left-to-right, which limits parallel decoding. More recent discrete diffusion models [15, 33, 35], despite generating tokens in parallel, must still ensure these tokens express mutually consistent and coherent responses. However, masked diffusion typically predicts tokens independently within each denoising step, conditioned on the current sequence [27]. This raises the question: how can vision–language models jointly model output dependencies while retaining parallel decoding and incorporating visual information?

Continuous language generation offers a way to combine joint sequence modeling with parallel decoding [9]. Rather than unmasking tokens, it progressively refines a continuous representation of the entire response before converting that representation into text. This allows different parts of the response to develop together while retaining parallel generation. For vision–language generation, visual information must guide this joint refinement and carry through to the final text. Hence, visual adaptation requires addressing two key aspects: how the model generates continuous predictions and how it learns to decode those predictions into accurate answers and captions.

We thus introduce ViLFlow (Vision–Language Flow), a continuous-flow vision–language model that adapts pretrained language generation through learned image guidance and prediction-aware token decoding (Figure 1). Alongside reference text, ViLFlow uses model predictions to train both image-conditioned denoising and token decoding. To learn image guidance, a visually adapted teacher predicts the same noisy text with and without image features. We amplify the difference between these predictions to construct a guided training target [23]. By learning to reproduce this target, the denoiser incorporates image guidance into a single image-conditioned prediction at each generation step. Computing the same guidance during inference would require two predictions.

Figure 1. Three approaches to vision-language generation, illustrated with the same image and question. (a) Autoregressive models decode left-to-right. (b) Masked diffusion models replace masks [M] with token predictions, highlighted in the second row. (c) ViLFlow repeatedly denoises continuous text embeddings before decoding tokens in parallel. Each narrow outline represents one output position, with schematic coordinates shown inside. Gray embeddings denote initial noise and green embeddings the final state.

Adapting the token decoder improves both answering and captioning without changing the trained denoiser’s weights. The decoder initially learns to reconstruct text from corrupted reference embeddings, whereas generation supplies denoiser predictions. We therefore train the decoder on these predictions to improve their conversion into words. This adapted decoder also supplies text feedback during late denoising, using its clean predictions to refine the continuous response before final decoding.

We evaluate ViLFlow against autoregressive and masked-diffusion baselines on visual question answering (VQA) and image captioning. On the shared VQAv2 evaluation set [6], ViLFlow achieves 63.39% accuracy, exceeding our GPT-2-based autoregressive baseline by 3.34 percentage points. It also generalizes to GQA [10] without further training, achieving 39.47% accuracy and a 2.83-point improvement over the autoregressive baseline. For captioning, ViLFlow achieves 111.16 CIDEr on the COCO Karpathy test split [12, 19], outperforming our 32-step masked-diffusion baseline by 16.25 points. Our contributions are as follows:

  • We introduce ViLFlow, a continuous-flow vision–language model that adapts pretrained language generation to visual tasks through joint embedding generation and parallel token decoding.

  • We distill image guidance from a visually adapted teacher into the denoiser, enabling image-guided generation with one image-conditioned prediction per step.

  • We develop prediction-aware token decoding that improves answering and captioning without further denoiser training. The adapted decoder also provides text feedback during caption generation.

  • We show that ViLFlow outperforms our autoregressive baseline on VQA and generalizes to GQA without further training. Captioning gains over masked diffusion are complemented by controlled evidence that our training choices improve performance beyond additional training alone.

Related Work

Language Models

Autoregressive language models such as GPT-2 [25] and LLaMA [30] predict each token from the preceding sequence. Masked diffusion language models, including MDLM [27] and LLaDA [24], instead recover masked tokens in parallel using bidirectional context. Both approaches represent the evolving output as discrete tokens or masks.

Continuous language models differ in their text representations and token decoding. Earlier approaches explore continuous text denoising for controlled and conditional generation [5, 11, 17, 20, 28]. CoLa [7] learns a text variational autoencoder and a block-causal continuous prior, with preliminary experiments on image-conditioned generation. ELF [9] instead denoises embeddings from a frozen contextual text encoder and decodes tokens after denoising. ELF-S2T [16] applies this formulation to speech. Sigma [34] uses vocabulary probabilities to construct embedding predictions during denoising, while HC-DLM [26] uses provisional tokens as inputs to subsequent continuous updates. ViLFlow adopts ELF’s frozen text representation and pretrained full-sequence denoiser as the foundation for visual adaptation.

Vision-Language Models

LLaVA [21] adapts autoregressive language modeling to visual tasks through a learned projector that maps image features into the language model’s embedding space. Other pretrained VLMs connect vision and language models through learned interfaces [1, 4, 13]. LLaDA-V [35], LaViDa [15], and MMaDA [33] instead condition masked-token generation on images. Discrete diffusion has also been applied to captioning-specific and unified multimodal models [18, 37]. For text generation, these models represent the evolving output with discrete tokens or masks. ViLFlow denoises continuous text embeddings and decodes the output tokens in parallel at the final readout.

Continuous diffusion has also been studied directly for image captioning. Analog Bits and SCD-Net apply continuous diffusion to bit representations of caption tokens [3, 22]. DiffCap [8] denoises learned caption embeddings, and LaDiC [32] uses a split BERT encoder and decoder around an image-conditioned diffuser. UniDiffuser [2] studies unified image–text diffusion, while Multimodal Flow [29] and JUICE [14] use continuous flows in pretrained representation spaces for unified multimodal modeling. ViLFlow adapts a pretrained continuous language generator, whereas Multimodal Flow and JUICE train their flow backbones from scratch. ViLFlow combines learned image guidance with a separate token decoder trained on predictions from the frozen denoiser. The adapted decoder also supplies text feedback during late caption-denoising updates. Matched controls separate the effects of guidance supervision, decoder inputs and inference feedback.

Preliminaries

We adopt ELF’s continuous text representation and its denoising and token-reconstruction objectives [9]. A frozen text encoder maps a reference sequence padded to LL positions to normalized dd-dimensional embeddings X∈RL×dX \in\mathbb{R}^{L \times d}. Let ϵ∈RL×d\epsilon\in\mathbb{R}^{L \times d} contain independent standard Gaussian entries and let σ>0\sigma> 0 be the noise scale. The noisy embeddings at time t∈[0,1]t \in[0,1] are

Zt=tX+(1−t)σϵ.(1)Z_t = tX + (1-t)\sigma\epsilon. \tag*{(1)}

The denoiser, parameterized by θ\theta, uses a transformer and denoising head to predict clean embeddings X^\widehat{X} from ZtZ_t at all positions in parallel. For flow matching, we convert the predicted and reference embeddings into a predicted velocity vθv_\theta and target velocity v∗v_*. Let δ(t)\delta(t) denote 1−t1-t bounded away from zero for numerical stability. These velocities are

vθ=X^−Ztδ(t),v∗=X−Ztδ(t).(2)v_\theta= \frac{\widehat{X}-Z_t}{\delta(t)}, \qquad v_* = \frac{X-Z_t}{\delta(t)}. \tag*{(2)}

Let mm mark the supervised token positions and ∥⋅∥m2\lVert\cdot\rVert_m^2 denote mean squared error over those positions. The denoising loss averages the velocity error over training examples, sampled times and noise: Ldata=E∥vθ−v∗∥m2\mathcal{L}_{\mathrm{data}} = \mathbb{E}\lVert v_\theta-v_*\rVert_m^2. For token reconstruction, the same transformer uses a vocabulary head to predict reference tokens in parallel from corrupted embeddings. We train this branch with cross-entropy.

Method

ViLFlow is a continuous diffusion vision-language model that generates captions or answers conditioned on images and optional questions. It combines full-sequence embedding-space denoising with parallel token decoding. Training comprises three stages: image-conditioned adaptation, image guidance distillation, and prediction-adapted decoding. For captioning, inference additionally incorporates text feedback into the denoising trajectory.

Model Architecture

ViLFlow performs image captioning and visual question answering through continuous diffusion in the text embedding space (Figure 2(a)). Alongside a frozen image encoder FF and a frozen text encoder EE, the model contains a visual projector, a denoising transformer, and a token decoder. The projector maps image features into the text embedding space, and the projected features are concatenated with the encoded question, when present, to form the conditioning sequence. Starting from Gaussian noise, the denoising transformer iteratively refines all output positions jointly under this conditioning. During Stage I, denoising and token reconstruction share a transformer; Stage III initializes a separate decoder from its guided counterpart. This decoder also receives the multimodal conditioning and predicts tokens in parallel. For captioning, it is applied at selected late denoising steps: the decoded text is re-encoded by EE and used in the current sampler update and as self-conditioning for the next step. After the final update, the decoder produces the output text.

Figure 2. Overview of ViLFlow for parallel vision–language generation through continuous latent flow. (a) Starting from Gaussian noise, the image-guided denoiser jointly refines all output positions. At selected late captioning steps, decoded text undergoes adjacent-word repetition removal and is re-encoded by frozen T5 for the current update and subsequent self-conditioning; other steps use the denoiser prediction directly. Final tokens are decoded in parallel. (b) Left: image-conditioned and image-zeroed teacher predictions provide a guided target for student training alongside paired-text supervision. Right: a separate decoder learns from denoiser predictions and clean reference embeddings using token cross-entropy, with the denoiser and visual projector frozen. Orange dashed arrows indicate supervision.

Stage I: Image-Conditioned Adaptation

Given an image–text pair (I,y)(I,y) and an optional question qq, let FF denote the frozen image encoder and EE the frozen text encoder with output normalization. We introduce a learnable visual projector PϕP_\phi to map image features into the text embedding space. The multimodal conditioning sequence CC and target text embeddings XX are defined as

C=[Pϕ(F(I));E(q)],X=E(y),(3)C = [P_\phi(F(I)); E(q)], \qquad X = E(y), \tag*{(3)}

where [ ; ][\,;\,] denotes sequence concatenation and E(q)E(q) is omitted for captioning. The target embeddings X∈RL×dX \in\mathbb{R}^{L \times d} occupy LL output positions with embedding dimension dd. They are corrupted to ZtZ_t according to Eq. (1), where t=0t=0 denotes the noise endpoint and t=1t=1 the clean endpoint. The conditioning sequence remains uncorrupted.

We adapt a shared transformer with a denoising head and a token reconstruction head. In denoising mode, the model predicts clean embeddings from ZtZ_t, conditioned on CC and a self-conditioning estimate SS. In reconstruction mode, it predicts reference tokens from separately corrupted reference embeddings X~\widetilde{X}:

X^θ=Dθ(Zt,t;C,S),πθ=Rθ(X~,1;C).(4)\widehat{X}_\theta= D_\theta(Z_t,t;C,S),\qquad\pi_\theta= R_\theta(\widetilde{X},1;C). \tag*{(4)}

Here, SS is a preliminary clean prediction computed without gradients, or zero when self-conditioning is omitted, and πθ\pi_\theta denotes the token distributions. Reconstruction uses a fixed decoder time input of 1, irrespective of the corruption used to construct X~\widetilde{X}. This fixed decoder time input is omitted in subsequent notation. Bidirectional attention allows the shared transformer to jointly process the conditioning features and all output positions. At each update, we sample either the denoising or reconstruction mode, optimizing continuous denoising supervision or token-level cross-entropy, respectively.

We initialize the shared transformer and both output heads from pretrained ELF and train them jointly with PϕP_\phi, keeping the image and text encoders frozen. To support subsequent image guidance, we apply image dropout by zeroing the projected visual features, yielding C∅=[0;E(q)]C_{\varnothing}=[0;E(q)]. This exposes the model to both image-present and image-zeroed conditioning. The corruption settings, scale-conditioning settings, and training-mode sampling probabilities are provided in the supplementary material.

Stage II: Image Guidance Distillation

Classifier-free guidance combines image-conditioned and image-zeroed predictions to strengthen visual conditioning, but requires two denoiser evaluations per generation step. We distill this guidance into a single student prediction [23]. A frozen teacher vψv_\psi and a trainable student vθv_\theta, including their respective visual projectors, are initialized from the adapted model (Figure 2(b), left). Here, vψv_\psi and vθv_\theta denote the velocity predictions obtained from clean-embedding estimates via Eq. (2).

Let CψC_\psi and CθC_\theta denote the teacher and student conditioning sequences constructed as in Eq. (3), using their respective projectors. The teacher additionally uses Cψ∅C^\varnothing_\psi, obtained by zeroing the projected image features while retaining the question conditioning. For a noisy text state ZtZ_t at time tt, we compute

vc=vψ(Zt,t;Cψ,S),v_c = v_\psi(Z_t,t;C_\psi,S),
vu=vψ(Zt,t;Cψ∅,S).(5)v_u = v_\psi(Z_t,t;C^\varnothing_\psi,S). \tag*{(5)}

Here, SS is a preliminary clean prediction from the student, computed without gradients, or zero when self-conditioning is omitted. The student and both teacher evaluations share this self-conditioning input. The two teacher evaluations also use identical ZtZ_t, tt, and question conditioning, differing only in whether the explicit visual conditioning features are present or zeroed.

Given a guidance scale ww, the teacher’s guided velocity is

vg=vc+(w−1)(vc−vu).(6)v_g = v_c + (w-1)(v_c-v_u). \tag*{(6)}

For w>1w > 1, this amplifies the velocity difference induced by explicit image conditioning. The student predicts vθ(Zt,t;Cθ,S,w)v_\theta(Z_t,t;C_\theta,S,w) using image-present conditioning. We provide ww through the backbone’s existing scale-conditioning interface, which adds an embedding of the scale to learned conditioning tokens. In this stage, the scale specifies the image guidance to be distilled. The student’s denoising objective combines reference supervision with regression toward the teacher’s guided velocity:

Lstudent=(1−λ)Ldata\mathcal{L}_{\mathrm{student}} = (1-\lambda)\mathcal{L}_{\mathrm{data}}
+λE[∥vθ(Zt,t;Cθ,S,w)−sg⁡(vg)∥m2],(7)\qquad+ \lambda\mathbb{E}\left[\left\|v_\theta(Z_t,t;C_\theta,S,w)-\operatorname{sg}(v_g)\right\|_m^2\right], \tag*{(7)}

where λ∈[0,1]\lambda\in[0,1] controls the distillation weight, sg⁡(⋅)\operatorname{sg}(\cdot) denotes stop-gradient, and ∥⋅∥m2\|\cdot\|_m^2 denotes masked mean squared error. The reference objective Ldata\mathcal{L}_{\mathrm{data}} supervises the student velocity using the paired target embeddings, while the teacher term transfers the scale-dependent image guidance. Token reconstruction training is retained in the separate reconstruction mode. At inference, the student incorporates the distilled guidance through a single image-conditioned denoiser evaluation per step. Hereafter, DθD_\theta denotes the distilled student’s clean-embedding predictor.

Stage III: Prediction-Adapted Decoding

Continuous generation ultimately requires mapping predicted embeddings to discrete tokens. However, the token reconstruction objective uses corrupted reference embeddings, whereas inference requires reading embeddings produced by the denoiser. Prediction errors can shift these inputs away from the representations encountered during reconstruction training, so accurate reconstruction of reference embeddings does not necessarily imply reliable readout of generated states. We therefore adapt the decoder directly to denoiser predictions, using their paired reference tokens as supervision.

To isolate readout adaptation from continuous generation, we initialize a separate decoder RωR_\omega by copying the guided model’s transformer and vocabulary head, while keeping the denoiser DθD_\theta and visual projector frozen (Figure 2(b), right). The decoder retains multimodal conditioning CC and predicts all output tokens in parallel.

Given reference embeddings X=E(y)X = E(y), we sample a timestep tt and obtain the noisy state ZtZ_t using Eq. (1). The frozen denoiser first produces a preliminary clean estimate X^(0)\widehat{X}^{(0)}, which is then used for self-conditioning in a second prediction:

X^(0)=Dθ(Zt,t;C,0,w),X^=Dθ(Zt,t;C,X^(0),w).(8)\begin{aligned} \widehat{X}^{(0)} &= D_\theta(Z_t, t; C, 0, w), \\ \widehat{X} &= D_\theta(Z_t, t; C, \widehat{X}^{(0)}, w). \tag*{(8)} \end{aligned}

Both evaluations use the same ZtZ_t, tt, conditioning CC, and guidance scale ww. The resulting X^\widehat{X} provides the decoder input, while the reference tokens yy serve as cross-entropy targets. This exposes RωR_\omega to denoiser predictions without requiring a complete generation trajectory.

We add reconstruction supervision on clean embeddings XX so that the decoder can also recover tokens directly from the text encoder representations. The decoder objective combines prediction-adapted decoding and clean-embedding reconstruction:

Ldec=(1−λclean) E[CE⁡(Rω(X^;C),y)]+λclean E[CE⁡(Rω(X;C),y)],(9)\mathcal{L}_{\mathrm{dec}} = (1-\lambda_{\mathrm{clean}})\,\mathbb{E}\left[\operatorname{CE}(R_\omega(\widehat{X};C),y)\right] + \lambda_{\mathrm{clean}}\,\mathbb{E}\left[\operatorname{CE}(R_\omega(X;C),y)\right], \tag*{(9)}

where λclean∈[0,1]\lambda_{\mathrm{clean}} \in[0, 1] controls the contribution of clean-embedding reconstruction. Only the decoder parameters ω\omega are updated, while DθD_\theta and the visual projector remain frozen.

The resulting prediction-adapted decoder serves as the parallel token readout module at inference. For captioning, we also reuse this decoder during late denoising steps: its text predictions are re-encoded and fed back into continuous generation, as described in § 4.5. This feedback is applied with all model weights fixed and requires no additional decoder or denoiser training. The supplementary material provides the corruption ranges, guidance-scale settings, loss weights, and task-specific training details.

Inference: Parallel Readout and Text Feedback

Given multimodal conditioning CC, generation starts from Zt0=σinitϵZ_{t_0} = \sigma_{\mathrm{init}}\epsilon, where ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0,I) and σinit\sigma_{\mathrm{init}} controls the inference initialization scale. The sampler jointly updates all LL output positions over KK denoising steps. At step kk, the denoiser receives the current state concatenated with the previous clean estimate SkS_k, initialized as S0=0S_0 = 0, and predicts X^k=Dθ(Ztk,tk;C,Sk,w)\widehat{X}_k = D_\theta(Z_{t_k}, t_k; C, S_k, w). Image and question features remain fixed throughout generation.

We use the prediction-adapted decoder RωR_\omega trained in § 4.4 for parallel token readout. For VQA, we use final-only readout: the decoder converts the final generated embeddings into text without modifying the preceding trajectory. For captioning, we additionally apply RωR_{\omega} at selected late denoising steps and feed its text predictions back into the sampler through re-encoding. Let TT denote parallel text readout from token logits. The clean estimate used by the sampler is

Xˉk={E(T(Rω(X^k;C))),k∈F,X^k,otherwise,(10)\bar{X}_k = \begin{cases} E\left(T\left(R_{\omega}(\hat{X}_k; C)\right)\right), & k \in\mathcal{F}, \\ \hat{X}_k, & \text{otherwise}, \end{cases} \tag*{(10)}

where F\mathcal{F} specifies the feedback steps and is empty for final-only readout. Before applying the normalized frozen text encoder EE, the decoded text is retokenized and padded or truncated to LL positions, yielding an estimate in the denoiser’s embedding space.

The sampler uses Xˉk\bar{X}_k to compute velocity via Eq. (2) and advance the noisy state, while setting Sk+1=XˉkS_{k+1} = \bar{X}_k for subsequent self-conditioning. Thus, decoded text influences both the current update and later predictions without fixing a token prefix; all output positions remain editable.

After the final update, RωR_{\omega} reads the resulting embeddings to produce the output text in parallel. All model weights remain fixed. This base inference configuration follows a single trajectory and uses the final readout without candidate selection. Training and inference noise scales, sampling settings, and implementation details are provided in the supplementary material.

Experiments

We evaluate ViLFlow on visual question answering (VQA) and image captioning against autoregressive and masked diffusion baselines, exemplifying two existing paradigms for language modeling.

Experimental Setup

Models and Baselines. Our main experiments use ViLFlow with a language backbone initialized from pretrained ELF-B [9]. We compare against autoregressive GPT-2 [25] and masked-diffusion MDLM [27], chosen for their comparable language-backbone sizes. All models use frozen SigLIP image features and the same downstream training data. To adapt each baseline, a learned visual projector maps the image features into its language embedding space and prepends them to the text input. We train the projector and fine-tune the language model using next-token prediction for AR and the masked-diffusion objective for MDLM. VQA baselines use greedy decoding, while captioning comparisons include both greedy and sampled outputs. Unless otherwise stated, ViLFlow uses 32 denoising steps, with text feedback during the final 10 steps. Hardware specifications and additional training and inference settings are provided in the supplementary material.

Datasets and Evaluation. For each baseline, we train separate models for VQA and image captioning. VQA models are trained on VQAv2 [6] and report soft-answer accuracy on a subset of its validation split. We further evaluate generalization to GQA [10] and report exact-answer accuracy. All captioning comparisons and ablations use the COCO Karpathy split [19], reporting BLEU-4, CIDEr [31], and METEOR on the test set. The supplementary material details the dataset splits and image-overlap exclusions, as well as further captioning results on Flickr30k [36].

Main Results

We discuss our main results on VQA and image captioning. On VQA, Table 1 demonstrates that ViLFlow outperforms both autoregressive and masked-diffusion baselines. On VQAv2, our method improves accuracy by 3.34 percentage points over AR and 8.47 points over MDLM. This advantage extends to GQA without further training: the same VQAv2-trained checkpoint exceeds AR by 2.83 points and MDLM by 4.50 points.

ModelBackbone (M)VQAv2 (%)GQA (%)
MDLM17054.9134.97
GPT-2 AR12460.0536.64
ViLFlow10563.39±0.03\pm0.0339.47±0.40\pm0.40

Table 1. ViLFlow outperforms baselines on VQAv2 and GQA. ViLFlow uses 32 denoising steps with text feedback during the final 10 steps. Results are means over three inference seeds, with standard deviations in subscripts. MDLM and GPT-2 use greedy decoding and a single run.

Table 2 compares ViLFlow with autoregressive and masked-diffusion baselines on image captioning. Because an image can admit multiple valid descriptions, we evaluate both greedy decoding and stochastic sampling. ViLFlow achieves 111.16 CIDEr and outperforms sampled AR and both MDLM decoding settings across all three metrics; it is furthermore competitive with greedy AR’s 114.66 CIDEr, with similar BLEU-4 and METEOR scores.

ModelBLEU-4 ↑\uparrowCIDEr ↑\uparrowMETEOR ↑\uparrow
ViLFlow33.00±0.4533.00\pm0.45111.16±0.69111.16\pm0.6927.34±0.0827.34\pm0.08
+ Retrieval33.40±0.29\mathbf{33.40}\pm0.29114.69±0.81\mathbf{114.69}\pm0.8127.47±0.08\mathbf{27.47}\pm0.08
Greedy decoding
MDLM27.6094.9123.82
GPT-2 AR33.29114.6627.36
Sampling: temperature 0.7, top-p 0.95
MDLM27.15±0.1427.15\pm0.1493.44±0.1193.44\pm0.1123.80±0.0323.80\pm0.03
GPT-2 AR25.04±0.4125.04\pm0.4193.35±0.3993.35\pm0.3925.08±0.1225.08\pm0.12

Table 2. ViLFlow outperforms sampled AR on COCO Karpathy test. Sampled baselines use temperature 0.7 and top-pp 0.95. Both ViLFlow and MDLM use 32 denoising steps; ViLFlow uses text feedback in the last ten steps. Subscripts indicate standard deviations over three inference seeds; greedy results use a single run. All metrics are reported on a ×100\times100 scale.

Retrieval-based draft selection. Unlike left-to-right decoding, whose intermediate states are incomplete prefixes, every ViLFlow readout is a complete caption, so a trajectory provides alternative drafts without extra samples or beams. We augment intermediate readouts with single-word variants and rank the candidates by similarity-weighted CIDEr-D against captions of the 20 nearest training images, retrieved using cosine similarity of SigLIP image embeddings. We replace the final readout only when the score improves by at least 2, with settings selected on validation; details are provided in the supplementary material.

Effect of Image Guidance. Our distillation stage uses a frozen teacher to provide image-guided denoising targets, which the generator learns to match using a single image-conditioned prediction at inference. To test the benefit of these targets, we compare caption training with and without the distillation term in Eq. (7). Both conditions start from the same checkpoint after initial image-conditioned adaptation and use identical data, update budgets, and learning-rate schedules. Before token decoder adaptation, guided training raises CIDEr from 92.05 to 96.97 (Table 3). The 4.93-point gain under matched training budgets supports distilling image guidance into the generator rather than relying on additional training alone.

Token Decoder Adaptation. We compare two ways of constructing decoder-training inputs from training captions: denoiser predictions and noisy reference embeddings. Both conditions use the same reference-text

ContinuationBLEU-4 ↑CIDEr ↑METEOR ↑
Without guided target26.80±0.1426.80\pm_{0.14}92.05±0.3792.05\pm_{0.37}25.64±0.0725.64\pm_{0.07}
With guided target27.28±0.1127.28\pm_{0.11}96.97±0.2296.97\pm_{0.22}27.09±0.0927.09\pm_{0.09}

caption: tbl:3: Guided targets improve captioning under matched training budgets. Only regression to the teacher’s guided prediction changes. Both models use 32 denoising steps without token decoder adaptation or text feedback on Karpathy test. Scores are means over three seeds, with standard deviations in subscripts.

supervision, decoder architecture, and update budget, with the guided denoiser fixed. We also evaluate the decoder without adaptation (Table 4).

Training on denoiser predictions yields 63.39% VQAv2 accuracy, compared with 62.48% for noisy-reference training and 62.52% without adaptation. The 0.87-point gain over the matched noisy-reference control supports training the decoder to recover text from denoiser predictions rather than from artificially corrupted reference embeddings.

Decoder adaptationVQAv2 (%) ↑
None62.52±0.0362.52\pm_{0.03}
Noisy references62.48±0.0362.48\pm_{0.03}
Denoiser predictions63.39±0.0363.39\pm_{0.03}

caption: tbl:4: Training the token decoder on denoiser predictions improves VQA. The denoiser is fixed, and the trained decoders share the same architecture and update budget. This component comparison uses eight denoising steps without text feedback. Scores are means over three inference seeds, with standard deviations in subscripts.

The same comparison on captioning also favors training on denoiser predictions (Table 5). With the generator fixed, prediction-adapted decoding reaches 107.17 CIDEr, compared with 104.79 for noisy-reference training and 104.90 without adaptation. The 2.38-point gain over the matched noisy-reference control shows that the benefit extends beyond short answers to caption generation.

Decoder adaptationBLEU-4CIDErMETEOR
None30.57±0.2430.57\pm_{0.24}104.90±0.46104.90\pm_{0.46}27.22±0.0827.22\pm_{0.08}
Noisy references30.57±0.2630.57\pm_{0.26}104.79±0.49104.79\pm_{0.49}27.19±0.0727.19\pm_{0.07}
Denoiser predictions31.09±0.1431.09\pm_{0.14}107.17±0.25107.17\pm_{0.25}27.37±0.0527.37\pm_{0.05}

caption: tbl:5: Prediction-adapted decoding improves captioning. The generator is fixed; trained decoders share the same architecture and update budget. All rows use 32 denoising steps without text feedback on COCO Karpathy test. Subscripts give standard deviations over three inference seeds.

Effect of Text Feedback. Text feedback re-encodes the adapted token decoder’s intermediate predictions to refine the late denoising updates. With model weights and 32 denoising steps fixed, the feedback procedure raises CIDEr from 107.17 to 111.16 and BLEU-4 from 31.09 to 33.00, while METEOR remains similar (Table 6). This 3.99-point CIDEr gain requires no additional training. The comparison measures the complete feedback procedure, including reduced late-step re-noising; it does not attribute the entire gain to text re-encoding alone.

ModelBLEU-4 ↑\uparrowCIDEr ↑\uparrowMETEOR ↑\uparrow
Without feedback31.09±0.1431.09 \pm0.14107.17±0.25107.17 \pm0.2527.37±0.0527.37 \pm0.05
With feedback33.00±0.4533.00 \pm0.45111.16±0.69111.16 \pm0.6927.34±0.0827.34 \pm0.08

Table 6. Text feedback improves captioning without further training. Both rows use the same model, 32 steps, initial noise and final phrase cleanup on Karpathy test. The second row adds text feedback and reduced late-step re-noising. Subscripts give standard deviations over three inference seeds.

As Table 7 shows, text feedback provides a further VQAv2 gain after token decoder adaptation. With the denoiser, token decoder, and noise settings fixed, accuracy increases by 0.42 percentage points, with a paired 95% confidence interval of [0.37,0.46][0.37,0.46]. GQA accuracy remains nearly unchanged. The feedback benefit is therefore task-dependent: it improves VQAv2 answering but does not provide a measurable transfer gain on GQA.

InferenceVQAv2 (%) ↑\uparrowGQA (%) ↑\uparrow
Without text feedback62.97±0.0162.97 \pm0.0139.46±0.2139.46 \pm0.21
With text feedback63.39±0.0363.39 \pm0.0339.47±0.4039.47 \pm0.40

Table 7. Effect of text feedback on VQA. The denoiser, token decoder, 32 steps, noise settings and removal of adjacent repetitions are fixed. Only text feedback changes. Scores are means over three inference seeds, with standard deviations in subscripts.

Number of Denoising Steps

We vary the number of denoising steps with text feedback disabled, keeping the denoiser and adapted token decoder fixed. As Table 8 shows on image captioning, CIDEr is 107.17 at the default 32-step budget, compared with 104.50 at 16 steps and 106.02 at 64 steps. METEOR increases from 26.80 at 16 steps to 27.44 at 64 steps, so additional denoising does not uniformly improve all metrics.

StepsCIDErMETEOR
16104.50±0.26104.50 \pm0.2626.80±0.0926.80 \pm0.09
32107.17±0.25107.17 \pm0.2527.37±0.0527.37 \pm0.05
64106.02±0.19106.02 \pm0.1927.44±0.0327.44 \pm0.03

Table 8. Effect of denoising steps on captioning without text feedback. The denoiser and token decoder are fixed. Evaluation uses COCO Karpathy test, with means and standard deviations over three inference seeds.

Analysis and Discussion

Tables 3 and 4 isolate the contributions of image guidance and token decoder adaptation. Guided targets improve captioning under matched training budgets, while training the decoder on denoiser predictions improves VQA with the denoiser fixed. Both comparisons retain reference supervision and the same update budget, so additional training alone cannot explain the gains. The decoder comparison further demonstrates that adapting the mapping from continuous predictions to text can improve output quality without further denoiser training.

Text feedback extends the adapted decoder’s role beyond final token recovery by using its intermediate readouts to refine subsequent denoising updates (Table 6 and Table 7). For captioning, the feedback procedure improves output quality with model weights and the 32-step denoising budget held fixed. This procedure combines re-encoding of decoder predictions with reduced late-step re-noising. The answering comparison keeps the noise settings unchanged and therefore isolates the contribution of text feedback. These results show how the independently adapted decoder can improve both the final readout and the trajectory that produces it.

Qualitative Results

Figure 3 illustrates counting and scene interpretation. ViLFlow correctly counts the two overlapping giraffes in (a) and identifies the kites casting the shadows in (b), while both baselines give incorrect answers.

Figure 3. VQA examples. (a) Question: How many giraffe are standing on the field? Human: 2; ViLFlow: 2; MDLM: 1; AR: 1. (b) Question: What is making the shadows? Human: kites; ViLFlow: kites; MDLM: fish; AR: people. Human denotes the most frequent annotated answer. The examples use the ablation models without text feedback; AR and MDLM decode greedily.

Figure 4 compares descriptions of the same scene. ViLFlow describes the elephant next to the car. AR omits the car, whereas MDLM places the elephant in its hood. This comparison illustrates a difference in the described spatial relation, not just the objects mentioned.

Figure 4. COCO captioning example. Reference: a big elephant makes its way across the road with onlookers in the car. ViLFlow: an elephant walking down a road next to a car. MDLM: an elephant standing in the hood of a car. AR: an elephant standing in the middle of a road. ViLFlow describes the elephant’s position relative to the car. Outputs use the ablation models without text feedback; AR and MDLM decode greedily.

Conclusion

We present ViLFlow, a continuous vision–language model for question answering and captioning. Our results show that visual adaptation benefits from both learning image guidance and training the token decoder to interpret denoiser predictions. Decoder adaptation improves both tasks without further denoiser training, while text feedback further improves captioning. These findings establish token decoding as more than a final conversion into words: it is an independently trainable component that can also improve continuous generation. Our evaluation focuses on short answers and captions; extending the approach to longer, open-ended visual responses remains future work.

References

  1. [1]Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikołaj Bińkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems, pages 23716–23736, 2022.arxiv.org/abs/2204.14198
  2. [2]Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. In Proceedings of the 40th International Conference on Machine Learning, pages 1692–1717. PMLR, 2023.arxiv.org/abs/2303.06555
  3. [3]Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. In The Eleventh International Conference on Learning Representations, 2023.arxiv.org/abs/2208.04202
  4. [4]Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems, pages 49250–49267, 2023.
  5. [5]Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. DiffuSeq: Sequence to sequence text generation with diffusion models. In The Eleventh International Conference on Learning Representations, 2023.arxiv.org/abs/2210.08933
  6. [6]Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904–6913, 2017.
  7. [7]Hongcan Guo, Qinyu Zhao, Yian Zhao, Shen Nie, Rui Zhu, Qiushan Guo, Feng Wang, Tao Yang, Hengshuang Zhao, Guoqiang Wei, and Yan Zeng. Cola DLM: Continuous latent diffusion language model. arXiv preprint arXiv:2605.06548, 2026.arxiv.org/abs/2605.06548
  8. [8]Yufeng He, Zefan Cai, Xu Gan, and Baobao Chang. DiffCap: Exploring continuous diffusion on image captioning. arXiv preprint arXiv:2305.12144, 2023.arxiv.org/abs/2305.12144
  9. [9]Keya Hu, Linlu Qiu, Yiyang Lu, Hanhong Zhao, Tianhong Li, Yoon Kim, Jacob Andreas, and Kaiming He. ELF: Embedded language flows. In Advances in Neural Information Processing Systems, 2026.arxiv.org/abs/2605.10938
  10. [10]Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6700–6709, 2019.arxiv.org/abs/1902.09506
  11. [11]Rabeeh Karimi Mahabadi, Hamish Ivison, Jaesung Tae, James Henderson, Iz Beltagy, Matthew E. Peters, and Arman Cohan. TESS: Text-to-text self-conditioned simplex diffusion. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2347–2361. Association for Computational Linguistics, 2024.
  12. [12]Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3128–3137, 2015.
  13. [13]Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, pages 19730–19742. PMLR, 2023.arxiv.org/abs/2301.12597
  14. [14]Jiahan Li, Huangyuan Su, Shengbang Tong, Shusheng Yang, Ruihan Guo, Suozhi Huang, Hexu Zhao, Pratyusha Sharma, Sherry Yang, Jiatao Gu, Rob Fergus, and Kunpeng Hu. JUICE: Unified multimodal modeling in pretrained representation spaces. arXiv preprint arXiv:2610.28801, 2026.arxiv.org/abs/2610.28801
  15. [15]Shufan Li, Konstantinos Kallidromitis, Hritik Bansal, Akash Gokul, Yusuke Kato, Kazuki Kozuka, Jason Kuen, Zhe Lin, Kai-Wei Chang, and Aditya Grover. LaViDa: A large diffusion language model for multimodal understanding. In Advances in Neural Information Processing Systems, pages 105101–105134, 2025.arxiv.org/abs/2505.16839
  16. [16]Xu anchen Li, Tianrui Wang, Yuheng Lu, Zikang Huang, Yu Jiang, Chenghan Lin, Chenrui Cui, Ziyang Ma, Xingyu Ma, Chunyu Qiang, Guochen Yu, Xie Chen, Longbiao Wang, and Jianwu Dang. Speech meets ELF: Audio conditional continuous-target diffusion for speech recognition and translation. arXiv preprint arXiv:2606.10368, 2026.arxiv.org/abs/2606.10368
  17. [17]Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. Diffusion-LM improves controllable text generation. In Advances in Neural Information Processing Systems, pages 4328–4343, 2022.arxiv.org/abs/2205.14217
  18. [18]Zijie Li, Henry Li, Yichun Shi, Amir Barati Farimani, Yuval Kluger, Linjie Yang, and Peng Wang. Dual diffusion for unified image generation and understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2779–2790, 2025.arxiv.org/abs/2501.00289
  19. [19]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In Computer Vision – ECCV 2014, pages 740–755. Springer, 2014.arxiv.org/abs/1405.0312
  20. [20]Zhenghao Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zihao Fan, Chen Lin, Nan Duan, and Weizhu Chen. Text generation with diffusion language models: A pre-training approach with continuous paragraph denoise. In Proceedings of the 40th International Conference on Machine Learning, pages 21051–21064. PMLR, 2023.arxiv.org/abs/2212.11685
  21. [21]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, pages 34892–34916, 2023.
  22. [22]Jianjie Luo, Yehao Li, Yingwei Pan, Ting Yao, Jianlin Feng, Hongyang Chao, and Tao Mei. Semantic-conditional diffusion networks for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23359–23368, 2023.arxiv.org/abs/2212.03099
  23. [23]Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik P. Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14297–14306, 2023.arxiv.org/abs/2210.03142
  24. [24]Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2502.09992
  25. [25]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019.
  26. [26]Hui Ren, Zihan Li, Chang Liu, Huidong Liu, and Alexander Schwing. Hierarchical continuous diffusion language models. arXiv preprint arXiv:2610.02193, 2026.arxiv.org/abs/2610.02193
  27. [27]Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In Advances in Neural Information Processing Systems, pages 130136–130184, 2024.arxiv.org/abs/2406.07524
  28. [28]Robin Strudel, Corentin Tallec, Florent Altché, Yilun Du, Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre, and Rémi Leblond. Self-conditioned embedding diffusion for text generation. arXiv preprint arXiv:2211.04236, 2022.arxiv.org/abs/2211.04236
  29. [29]Hongyuan Tao, Xinggang Wang, Lianghui Zhu, Yongkang Li, Yunchao Wei, Bin Feng, Shaoyu Chen, Qian Zhang, Chang Huang, and Kai Yu. Multimodal flow: Unified flow modeling of language and vision in embedding spaces. arXiv preprint arXiv:2609.40362, 2026.arxiv.org/abs/2609.40362
  30. [30]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.arxiv.org/abs/2302.13971
  31. [31]Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus-based image description evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4566–4575, 2015.arxiv.org/abs/1411.5726
  32. [32]Yuchi Wang, Shuhuai Ren, Rundong Gao, Linli Yao, Qingyan Guo, Kaikai An, Jianhong Bai, and Xu Sun. LaDiC: Are diffusion models really inferior to autoregressive counterparts for image-to-text generation? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6699–6715. Association for Computational Linguistics, 2024.arxiv.org/abs/2404.10763
  33. [33]Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. MMaDA: Multi-modal large diffusion language models. In Advances in Neural Information Processing Systems, pages 138867–138907, 2025.
  34. [34]Zhihan Yang, Wei Guo, Jean-Marie Lemercier, Simon Welker, Mohammad Mahdi Kamani, Yonggan Fu, Sajjad Norouzi, Julius Berner, Tomas Geffner, Karsten Kreis, Yongxin Chen, Molei Tao, John Thickstun, Pavlo Molchanov, Ante Jukić, Arash Vahdat, and Morteza Mardani. Large language continuous diffusion models, 2026.arxiv.org/abs/2610.02665
  35. [35]Zebin You, Shen Nie, Xiaolu Zhang, Jun Zhou, Zhiwu Lu, Ji-Rong Wen, and Chongxuan Li. LLaDA-V: Large language diffusion models with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10093–10105, 2026.arxiv.org/abs/2505.16933
  36. [36]Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78, 2014.
  37. [37]Zixin Zhu, Yixuan Wei, Jianfeng Wang, Zhe Gan, Zheng Zhang, Le Wang, Gang Hua, Lijuan Wang, Zicheng Liu, and Han Hu. Exploring discrete diffusion models for image captioning. arXiv preprint arXiv:2211.11694, 2022.arxiv.org/abs/2211.11694

Paper details

vlnc:W_hhp42nht
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents