betaFeedback

WorksComputer Science

MMFlash: Faster Speculative Decoding for Vision-Language Models

Abstract

Speculative decoding accelerates generation by letting a lightweight drafter propose tokens for parallel verification by a target model. Diffusion drafters such as DFlash propose an entire token block in a single forward pass, but mainly focus on text-only tasks, leaving multimodal drafting underexplored. To study this setting, we first introduce a metric of token-level visual dependence based on the KL divergence between next-token distributions with and without visual input. We find that acceptance rates decrease from 46% in the lowest visual-dependence quartile to 22% in the highest, with this disparity persisting after multimodal training. Motivated by this observation, we propose MMFlash, a simple and effective framework that adaptively upweights visually dependent tokens that the drafter predicts poorly. For long video inputs, MMFlash further employs sparse attention to reduce redundant attention computation and improve drafting quality. Across various multimodal benchmarks, MMFlash achieves lossless decoding speedups of up to 3.58×3.58\times, outperforming EAGLE-3 and DFlash by up to 195.4% and 17.1%, respectively.

§ Code: MMFlash Models: MMFlash

Introduction

Vision-language models (VLMs) [1, 4, 58, 62] have demonstrated remarkable capabilities across multimodal tasks such as visual question answering (VQA) [2, 61, 72], video understanding [15, 36, 64], and visual reasoning [11, 12, 45, 69]. However, their inference efficiency remains constrained by autoregressive generation, which produces tokens sequentially. This bottleneck is further exacerbated by the growing adoption of reasoning models that generate longer responses [10, 19, 25]. Speculative decoding [6, 34, 66] has emerged as a prominent solution: a small model drafts multiple tokens, which a larger target model then verifies in parallel, substantially accelerating decoding.

The performance of speculative decoding largely depends on draft quality and drafting cost. Early approaches include Medusa [5], which augments the target model with multiple decoding heads, and the EAGLE series [37, 38, 39, 57], which drafts from target model features and uses tree-based verification to increase acceptance lengths. For autoregressive drafters, however, each drafting step requires a forward pass, making drafting cost grow linearly with draft length. This favors lightweight architectures, often at the expense of draft quality. Inspired by diffusion language models (DLMs) [9, 49, 71], recent methods [23, 42, 65, 73] such as DFlash [7] use block diffusion [3] to generate an entire draft block in a single forward pass, decoupling the number of sequential drafting steps from the number of proposed tokens. The computational budget previously spent on repeated sequential passes can thus support deeper, higher-capacity drafters, improving draft quality. Building on these advances, we extend diffusion drafting to multimodal models, a direction that remains largely unexplored.

To better extend to the vision field, we first introduce a token-level measure of visual dependence: the KL divergence between the target model’s distributions with and without visual input, conditioned on the same textual prefix. Using this metric, we analyze two DFlash drafters for Qwen3.5-4B (Figure 3): the official text-trained drafter and a counterpart trained on multimodal data following the official recipe. Acceptance rates decrease monotonically with visual dependence, from 46% in the lowest quartile to 22% in the highest. Moreover, the highest quartile accounts for 44% of first rejections within draft blocks, compared with only 7% for the lowest. This disparity persists after training with multimodal data.

Figure 3. Empirical studies on Qwen3.5-4B with three benchmarks. For each benchmark, we randomly select 200 samples and sample up to 128 anchor positions per sample. Top: acceptance rate of drafted tokens. Bottom: share of truncation points falling in each quartile. Visual dependence increases from Q1 to Q4.

To address this challenge, we propose MMFlash (Figure 2), a simple and effective framework for VLMs. Its core is a visually aware loss reweighting scheme that uses the above visual dependence scores to emphasize visually dependent tokens in the DFlash cross-entropy objective. The additional weight scales with the drafter’s prediction error probability, emphasizing tokens it has yet to learn and gradually returning to the baseline weight as predictions improve. For long videos, drafter attention concentrates on nearby positions, with substantial redundancy among visual tokens. We thus equip the MMFlash drafter with a simple sparse attention pattern that retains a few attention sinks, strided visual tokens, all text tokens, and a local window, consistently across training and inference.

Figure 2. Overview of MMFlash. (a) The target VLM is run twice on the same prefix x<tx_{<t}, with and without the image II, and the visual dependence score dtd_t is the KL divergence between the two next-token distributions. Each masked position of a block keeps its base weight bkb_k and gains a visual bonus that depends on dtd_t. (b) The MMFlash drafter adopts a simple sparse attention pattern for video inputs, shrinking a context of more than 40K tokens to about 1.6K.

We evaluate MMFlash primarily on Qwen3.5 series [51] across seven image benchmarks covering text and document understanding, general VQA, and STEM reasoning, as well as three long video benchmarks (Figure 1). Baselines include native MTP, EAGLE-3, and DFlash. On image benchmarks, MMFlash achieves the longest average acceptance lengths and highest decoding speedups across all

Figure 1. Decoding speedup comparison with SOTA speculative decoding methods (EAGLE-3 & DFlash) on SGLang.

settings. Under greedy decoding, average speedups reach 3.00–3.05×\times, exceeding DFlash by 10.3–10.9%. On long video benchmarks, sparse attention combined with minor video fine-tuning yields average speedups of 2.33–2.71×\times, outperforming DFlash by 11.2–23.2%. MMFlash remains effective under high concurrency and generalizes to other model families and larger scales.

Overall, we introduce MMFlash, a novel diffusion drafting framework for speculative decoding in VLMs. By prioritizing poorly predicted, visually dependent tokens during training, MMFlash addresses a key bottleneck in draft acceptance. Comprehensive experiments demonstrate its effectiveness and generalizability, establishing MMFlash as a strong baseline for speculative decoding in VLMs.

Related Work

We briefly review the most relevant work here and provide a more detailed discussion in Appendix § 8.

Speculative Decoding for VLMs. Speculative decoding [6, 34] lets a lightweight drafter propose tokens that the target model verifies in parallel, leaving its output distribution unchanged and thus achieving lossless acceleration. Drafters range from decoding heads [5] to feature-level drafters with tree-structured verification [37, 38, 39, 57]. For VLMs, existing methods differ mainly in how the drafter handles visual input: it receives the full visual input [17, 24, 41, 60], only distilled visual information [20, 26, 28, 30, 47, 76], or no visual tokens at all [16, 67, 74], and some methods switch among these strategies adaptively [21, 32, 33]. MMFlash follows the third and most efficient paradigm, since the target has already fused the visual content into the hidden states that the drafter reads. Orthogonal efforts relax verification [27], speculate at the agent level [22], or specialize in document OCR [40].

Speculative Decoding with Diffusion Drafters. Diffusion language models (DLMs) [29, 50, 53, 56, 71, 78] generate text by denoising many positions in parallel, and block diffusion [3] applies this denoising within blocks that are generated autoregressively. Since a DLM can propose a whole block in one forward pass, early work explores diffusion-style drafting by mixing diffusion and autoregressive objectives in one model [43], adapting autoregressive large language models (LLMs) for parallel drafting [54], or drafting with large pre-trained DLMs [35, 55]. DFlash [7] makes diffusion drafting simple and effective with a lightweight drafter that emits an entire block in one forward pass, and follow-up work explores related designs [14, 23, 42] and dynamic position-wise loss weights [18, 65, 70].

Preliminaries

As a baseline for our method, we briefly introduce DFlash [7], highlighting its inherent advantages for speculative decoding. Unlike autoregressive drafters, which require multiple sequential forward passes to propose draft tokens, DFlash generates an entire draft block in a single forward pass. This parallelism decouples the number of sequential drafting passes from the number of draft tokens, allowing the computational budget otherwise spent on the drafting passes to support a deeper, higher-capacity drafter.

Concretely, at each drafting step, DFlash conditions on context features extracted from the target model and processes a block of BB positions, one anchor token followed by B−1B - 1 mask tokens, filling all masks in parallel. During training, anchors are randomly sampled from a response generated by the target model, and the B−1B - 1 tokens after each anchor are masked for reconstruction. This mirrors inference, where each block starts with the bonus token from the preceding verification step. Let zk∗z_k^* denote the token that the target decoding policy selects at the kk-th masked position, and let qkq_k be the probability that the drafter assigns to it given the prefix. Training reduces to a cross-entropy fit to the target’s own choices, except that the B−1B - 1 positions do not contribute equally to the loss:

LDFlash=∑k=1B−1wk(−log⁡qk),wk=exp⁡(−k−1γ),\mathcal{L}_{\mathrm{DFlash}} = \sum_{k=1}^{B-1} w_k(-\log q_k), \quad w_k = \exp\left(-\frac{k-1}{\gamma}\right),

where the hyperparameter γ>0\gamma> 0 sets how fast the weight decays along the block. This decay mirrors how a block is consumed at inference time. Verification accepts a prefix of the block, so a rejection at position kk discards every token after it, and position kk lengthens the accepted sequence only when positions 1,…,k−11,\ldots,k-1 have all been accepted.

Method

In this section, we first identify a key limitation of extending DFlash to multimodal speculative decoding, which motivates us to propose our MMFlash. Then, we introduce our MMFlash in detail.

Motivation

DFlash has been thoroughly validated in the text setting. Since our target is VLMs on multimodal data, the natural first question is how DFlash behaves on tokens that depend on the multimodal inputs (e.g., images). Answering it requires a per-token measure of visual dependence, and we take this measure from the target model rather than from the drafter or from attention maps. For each generated token, we run the target model twice, once with the image and once with the image removed while keeping the text prompt and the generated prefix fixed, and define the token’s visual dependence as the KL divergence between the two resulting next-token distributions:

d=KL⁡(p(⋅∣prefix,I) ∥ p(⋅∣prefix)),(1)d = \operatorname{KL}\left(p(\cdot\mid\mathrm{prefix}, I) \,\|\, p(\cdot\mid\mathrm{prefix})\right), \tag*{(1)}

where pp denotes the next-token distribution of the target model and II the input image. This choice follows directly from how speculative decoding works. The sole objective of the drafter is to imitate the distribution of the target model, and a draft token is accepted only when it matches the target model’s own prediction. What matters is therefore how much of the prediction that the drafter must reproduce is contributed by the image, which is exactly what the divergence above measures.

Then, we conduct an empirical study on Qwen3.5-4B with two DFlash drafters: (1) the officially released drafter2, trained on text-only data; and (2) a drafter we train following the official recipe on roughly 900K multimodal samples drawn from LLaVA-OV-1.5-Instruct-Data3. Every drafted token is scored by the visual dependence defined above and assigned to a quartile from Q1 (least visual-dependent) to Q4 (most). We then report two statistics per quartile: the acceptance rate of drafted tokens; and the share of truncation points that fall in each quartile, where the truncation point of a block is the first token at which the drafter’s prediction disagrees with the target’s. As shown in Figure 3, across benchmarks and drafters the acceptance rate falls monotonically with visual dependence (46% in Q1 vs. 22% in Q4 on average, a 2.1× gap), while truncation points pile up at the image-dependent end: Q4 holds 44% of them against 7% in Q1, and Q3-Q4 together account for 72-80%. We further report the acceptance rate of drafted blocks grouped by the visual dependence of their first predicted token in

Appendix § 9 (Figure 5). The block-level view shows the same trend: a block that opens on an image-dependent token retains far less of its draft, with acceptance falling from 33% in Q1 to 14% in Q4 on average (a 2.4× gap). Overall, these results indicate that the acceptance loss of DFlash on vision-language targets concentrates on visually dependent tokens and that training on multimodal data alone does not close this gap. This observation motivates MMFlash, which explicitly directs training toward such tokens rather than treating all positions alike.

Figure 5. Block acceptance rate on Qwen3.5-4B with three benchmarks, under the same setup as Figure 3. Each drafted block is grouped by the visual dependence quartile of its first predicted token. Visual dependence increases from Q1 to Q4.

MMFlash

The study above shows where a DFlash drafter loses its acceptance under the multimodal setting: on the tokens whose prediction the image decides. DFlash’s objective is blind to this. Its weight wkw_k depends only on the position inside a block, so a token that merely continues a phrase and a token that reads a number off a chart receive the same gradient. Yet the quantity that exposes the gap, the visual dependence dd, is computed from the target alone, needs no drafter, and can be scored once offline for the training data. MMFlash therefore turns the diagnostic of § 4.1 into a training signal: it reweights the block loss so that the drafter spends its capacity where the image matters, and only while it has not yet learned those tokens.

For every token tt of a training response, we compute its visual dependence dtd_t with Equation (1), i.e., with the target model, the image II, and the prefix x<tx_{<t} fixed. The score dtd_t is precomputed once offline and stored with the training data, incurring no additional training cost. Following the analysis in § 4.1, we likewise bucket the tokens in the training set D\mathcal{D} by this score, and denote by g~t∈[0,1]\tilde{g}_t \in[0,1] the quantile of dtd_t among all tokens in D\mathcal{D} (see Appendix § 10.§ 1 for details). Visual dependence tells us where the image matters; it does not tell us whether the target has settled on a prediction there. The drafter is trained to reproduce the target’s choice, and extra effort pays off only where there is a definite choice to reproduce: on a token whose with-image distribution is still flat, the target picks among near-tied candidates, and that pick is not predictable from its features no matter how the token is weighted. We therefore scale the score by the target’s confidence with the image, measured by the entropy HtH_t of its with-image distribution p(⋅∣x<t,I)p(\cdot\mid x_{<t}, I):

gt=g~t⋅exp⁡(−Ht).(2)g_t = \tilde{g}_t \cdot\exp(-H_t). \tag*{(2)}

Hence gt∈[0,1]g_t \in[0,1] is highest where the image both changes the prediction and settles it, which is exactly where additional effort can turn into acceptance.

The score says which tokens deserve effort. Two more things decide whether effort at a block position is useful at all: whether verification will ever reach that position, and whether the drafter still has something to learn there. MMFlash therefore keeps the cross-entropy fit of DFlash and replaces its positional weight by a product of three factors. For the B−1B-1 masked positions of a block, each position kk inherits the static visual grounding gk∈[0,1]g_k \in[0,1] of its target token from Equation (2), which marks whether the token is worth extra effort at all. We combine it with two further factors in the training loss, where sg⁡(⋅)\operatorname{sg}(\cdot) denotes the stop-gradient operator:

wk=bk(1+αgk(1−sg⁡(qk))),w_k=b_k\left(1+\alpha g_k\left(1-\operatorname{sg}(q_k)\right)\right),
LMMFlash=∑k=1B−1wk(−log⁡qk),(3)\mathcal{L}_{\mathrm{MMFlash}}=\sum_{k=1}^{B-1}w_k\left(-\log q_k\right), \tag*{(3)}

where bkb_k and (1−sg⁡(qk))\left(1-\operatorname{sg}(q_k)\right) play complementary roles. bkb_k is a verification-aware positional weight: since verification accepts only a prefix of the block, the gradient at position kk matters only if positions 1,…,k1,\ldots,k all survive. We estimate this chain survival from the drafter’s own confidence,

bk=∏i=1kqˉi,qˉi=(1−λ)sg⁡(qi)+λ,(4)b_k=\prod_{i=1}^{k}\bar{q}_i,\qquad\bar{q}_i=(1-\lambda)\operatorname{sg}(q_i)+\lambda, \tag*{(4)}

where the smoothing λ∈[0,1]\lambda\in[0,1] keeps every factor above λ\lambda, so that a single miss weakens the later positions of a block without silencing them. Finally, (1−sg⁡(qk))\left(1-\operatorname{sg}(q_k)\right) adds a dynamic component: it is the drafter’s current probability of missing the token, so the boost is largest while a visually grounded token is still unlearned and fades to zero as the drafter fits it, shifting the extra effort elsewhere instead of over-fitting visual tokens at the expense of textual ones.

Unless stated otherwise we use λ=0.5\lambda=0.5 and α=1\alpha=1; further implementation details are given in Appendix § 10.

Sparse Attention in MMFlash Drafter for Video-Length Contexts. We further examine video benchmarks, where the average input length increases from 477 tokens across seven image benchmarks to 41,218 tokens. Diffusion drafters such as DFlash attend to the entire target context at every drafting step, raising the question of whether full-context attention is necessary for a small drafter. We therefore analyze the attention patterns of the trained MMFlash drafter on video inputs. Figure 4(a) shows that attention concentrates on the question and generated answer, while visual tokens receive diffuse attention with no frame standing out. Across 60 questions, Figure 4(b) confirms this locality: the most recent 256 positions account for only 0.8% of the context but receive 50.1% of the attention. In contrast, distant frames occupy over 96% of the context and collectively absorb 46% of the attention despite low attention per token, suggesting substantial redundancy in full-context drafting. Furthermore, we apply a simple attention window to the trained MMFlash drafter. As shown in Figure 4(c), windowing reduces acceptance length on image benchmarks but improves it on video benchmarks, supporting the benefit of sparse attention in long video contexts. A plain window discards visual evidence outside its range, potentially removing information needed to answer the question. Based on the observed attention patterns, we design a simple sparse attention pattern specifically for video benchmarks. It retains the first 4 sink tokens, visual tokens sampled with a stride of 32, all text tokens, and a local window of 256 tokens. We use the same pattern during training and inference to ensure consistent context access. Implementation details are provided in § 5.2.

Figure 4. Empirical studies on Qwen3.5-4B across video benchmarks (60 random samples per benchmark). (a) Per-token attention of the MMFlash drafter on one LongVideoBench question, relative to a uniform spread over the visible context; rows are decoding steps. (b) Cumulative share of draft attention within a given distance of the current position (solid), against the share of context tokens (dashed). (c) Acceptance length with the full draft context and with a window of the most recent 512 positions.

Experiments

Experimental Setup

We summarize the experimental setup below and provide more details in Appendix § 11.

Training Data. To ensure data diversity, we collect 1M samples from LLaVA-OV-1.5-Instruct-Data and regenerate their responses using the target model for better alignment. We modify selected prompts to elicit longer responses for training efficiency (Appendix § 11.§ 1). We follow the official generation settings, with temperature set to 0.0, top-p to 0.95, top-k to 20, and a maximum generation length of 4,096 tokens. After discarding samples whose total sequence length exceeds 4,096 tokens, we obtain approximately 900K training samples.

Training Details. We adopt Qwen3.5-4B and Qwen3.5-9B [51], two popular open-source VLMs, as our target models. For training, we mainly follow the official setting of DFlash. Specifically, we set the block size to 16 and the number of anchors to 512, use a batch size of 2, and train the drafter for 6 epochs. The learning rate is set to 6.0×10−46.0 \times10^{-4} and follows a cosine annealing schedule that decays to zero, preceded by a linear warmup over the first 4% of training steps. All training is conducted on 8 NVIDIA H100 GPUs.

Baselines. We compare MMFlash against the following baselines: (1) MTP, the multi-token prediction module natively supported by the Qwen3.5 series, for which we report results under different numbers of draft steps; (2) EAGLE-3, the state-of-the-art speculative decoding method [39], which we train on the same data as MMFlash; (3) DFlash-z, the officially released DFlash drafter, which consists of 6 layers, extracts hidden states from 8 layers of the target model, and is trained on text-only data4; and (4) DFlash, which we train on the same data as MMFlash following the official setting, i.e., a 5-layer drafter that extracts hidden states from 5 target layers. For a fair comparison, all trained baselines share the training data of MMFlash, and all other unrelated hyperparameters are kept identical.

Inference Details. We follow the official inference settings released with each target model, with thinking mode disabled and a maximum of 4,096 generated tokens. For all baselines and MMFlash, we set top-pp to 0.95 and top-kk to 20, and evaluate under two temperatures, T=0T = 0 and T=1T = 1. For the main experiments, the block size of the drafter is set to 16, consistent with training, and the concurrency is fixed to 1. All inference experiments are run on a single NVIDIA H200 GPU with SGLang [79] framework.

Evaluations. We evaluate our method on seven multimodal benchmarks covering text recognition and document understanding (ChartQA [48] and CharXiv [63]), general visual question answering (MMStar [8] and MMBench [44]), and STEM tasks (DynaMath [80], MathVista [46], and MathVerse [75]). All evaluations are conducted in a zero-shot setting on the same 200 random samples per benchmark (Appendix § 11.§ 3), following the LMMs-Eval framework.5

Main Results

MMFlash Achieves the Highest Acceptance Length and Speedup. As shown in Table 1, MMFlash consistently outperforms all baselines in both acceptance length and speedup across benchmarks, target models, and temperatures. It achieves average decoding speedups of 3.00–3.05× under greedy decoding (T=0T = 0) and 2.54–2.70× at T=1T = 1 across the two target models, 6.7–10.9% higher than DFlash. Overall, these results demonstrate the effectiveness and robustness of MMFlash.

ModelMethodChartQA τ\tauChartQA SpeedupCharXiv τ\tauCharXiv SpeedupMMStar τ\tauMMStar SpeedupMMBench τ\tauMMBench SpeedupDynaMath τ\tauDynaMath SpeedupMathVista τ\tauMathVista SpeedupMathVerse τ\tauMathVerse SpeedupAvg. τ\tauAvg. Speedup
Qwen3.5-4BMTP (3)3.181.69×3.271.76×2.951.61×2.431.40×3.271.69×3.181.68×3.171.69×3.061.64×
MTP (7)4.261.68×4.641.87×3.751.57×2.581.24×4.421.69×4.291.67×4.271.69×4.031.63×
MTP (15)4.671.26×5.371.43×3.991.14×2.600.87×4.761.23×4.651.21×4.611.22×4.381.19×
EAGLE-3 (16)4.311.74×4.381.86×3.491.49×2.611.08×4.291.67×4.021.56×3.951.68×3.861.58×
DFlash-z (16)4.922.76×5.633.10×3.812.35×5.012.62×4.822.56×4.432.45×4.342.50×4.712.62×
DFlash (16)5.432.97×5.603.13×4.052.31×5.172.93×5.572.57×5.162.53×4.842.62×5.122.72×
MMFlash (16)5.973.28×6.173.46×4.492.55×5.683.19×6.262.87×6.002.78×5.352.88×5.703.00×
Qwen3.5-9BMTP (3)3.321.83×3.281.87×3.011.73×2.881.57×3.381.80×3.261.80×3.241.80×3.201.77×
MTP (7)4.661.88×4.601.94×3.911.66×3.551.41×4.741.77×4.521.78×4.431.77×4.341.74×
MTP (15)5.271.38×5.271.47×4.201.18×3.750.99×5.251.30×5.081.32×4.901.30×4.811.27×
EAGLE-3 (16)3.931.81×3.851.93×3.061.55×2.721.32×3.601.76×3.501.63×3.481.75×3.451.68×
DFlash-z (16)5.223.03×5.273.15×3.932.46×3.672.11×4.962.75×4.772.75×4.522.67×4.622.70×
DFlash (16)5.593.20×5.173.09×4.152.46×3.792.20×5.622.71×5.382.83×4.922.79×4.952.75×
MMFlash (16)6.193.58×5.673.42×4.582.67×4.202.39×6.443.02×6.493.17×5.603.10×5.603.05×
Qwen3.5-4BMTP (3)3.041.58×3.071.60×2.741.48×2.411.34×3.171.59×3.001.54×3.001.57×2.921.53×
MTP (7)3.931.53×4.261.68×3.411.40×2.581.14×4.221.55×3.971.49×3.911.52×3.751.48×
MTP (15)4.331.14×4.761.24×3.570.99×2.620.86×4.551.14×4.331.08×4.221.11×4.061.08×
EAGLE-3 (16)3.971.59×3.971.60×3.231.38×2.601.01×4.091.49×3.871.45×3.681.53×3.631.44×
DFlash-z (16)4.492.42×5.002.66×3.432.06×4.812.12×4.642.41×4.142.19×4.012.28×4.362.30×
DFlash (16)4.872.56×4.952.66×3.712.04×4.922.48×5.072.31×4.662.22×4.472.37×4.672.38×
MMFlash (16)5.232.74×5.392.86×3.962.18×5.252.58×5.752.56×5.232.37×4.692.49×5.072.54×
Qwen3.5-9BMTP (3)3.191.72×3.081.72×2.861.61×2.781.49×3.251.70×3.111.69×3.111.71×3.051.66×
MTP (7)4.351.72×4.151.71×3.601.49×3.371.34×4.461.64×4.261.65×4.171.66×4.051.60×
MTP (15)4.841.24×4.571.27×3.811.06×3.530.93×4.921.22×4.651.18×4.561.21×4.411.16×
EAGLE-3 (16)3.601.67×3.561.72×2.841.42×2.591.23×3.461.61×3.321.50×3.291.62×3.241.54×
DFlash-z (16)4.772.64×4.702.76×3.662.25×3.421.97×4.712.57×4.492.51×4.262.49×4.292.46×
DFlash (16)4.942.75×4.502.65×3.742.18×3.552.02×5.242.52×4.852.52×4.532.57×4.482.46×
MMFlash (16)5.543.22×4.852.85×4.132.37×3.822.17×6.072.77×5.712.74×5.102.81×5.032.70×

Table 1. Decoding speedup over baseline and average acceptance length (τ\tau) on Qwen3.5 models across various benchmarks. Parenthesized values indicate the number of draft steps for MTP, the draft tree size for EAGLE-3, and the diffusion block size for DFlash & MMFlash.

MMFlash Remains Superior on Video Benchmarks. We further evaluate MMFlash on three video benchmarks: LongVideoBench [64], MVBench [36], and Video-MME [13]. We fine-tune the MMFlash drafter for 3 epochs on 2K samples from LLaVA-Video-178K [77] using the

proposed sparse attention pattern. As shown in Table 2, sparse attention alone, without any training, improves acceptance length by 11.1% and speedup by 4.7% on average. Fine-tuning increases these gains to 20.0% and 13.7%, respectively, yielding average speedups of 2.33–2.71×, 11.2–23.2% higher than DFlash, at minimal fine-tuning cost.

MethodLongVideoBenchMVBenchVideo-MME
τ\tauSpeedupτ\tauSpeedupτ\tauSpeedup
Qwen3.5-4B
MTP (15)3.991.31×3.801.09×3.501.00×
EAGLE-3 (4)2.041.63×1.911.45×1.921.43×
DFlash (16)3.142.11×3.011.82×2.811.75×
MMFlash (16)3.232.23×2.911.83×2.861.82×
+ sparse attn.3.812.44×3.502.05×3.201.93×
+ fine-tuning4.092.67×3.832.24×3.452.09×
Qwen3.5-9B
MTP (15)4.071.34×3.941.26×3.731.10×
EAGLE-3 (4)1.881.29×1.671.06×1.791.25×
DFlash (16)3.562.71×3.192.37×3.212.22×
MMFlash (16)3.632.79×3.232.42×3.282.27×
+ sparse attn.3.812.81×3.462.49×3.422.21×
+ fine-tuning4.163.07×3.742.66×3.632.39×

Table 2. Average acceptance length (τ\tau) and decoding speedup over baseline on three video benchmarks at temperature = 0. The gray number under each benchmark gives its average context length.

MMFlash Generalizes Across Model Families and Scales. We assess MMFlash’s generalization across model families and scales using Gemma4-E4B-it [58] and Qwen3.8-27B [52]. We train the EAGLE-3 and MMFlash drafters on the same data regenerated by the target model. As shown in Table 5, MMFlash achieves the highest acceptance length and decoding speedup on all three benchmarks for both models, with speedups of 2.10–3.02× on Gemma and 3.54–4.71× on Qwen3.8-27B.

MethodChartQA τ\tauChartQA SpeedupMMStar τ\tauMMStar SpeedupDynaMath τ\tauDynaMath Speedup
Gemma4-E4B-it
MTP (5, official)3.811.78×3.191.48×4.041.89×
MTP (15)4.731.66×3.741.33×5.171.75×
EAGLE-3 (16)4.872.27×4.041.50×6.231.90×
MMFlash (16)6.392.87×4.832.10×7.153.02×
Qwen3.8-27B
MTP (7)5.192.90×4.552.59×5.483.07×
MTP (15)6.362.47×5.252.16×6.632.68×
MMFlash (16)6.374.18×5.363.54×7.574.71×

Table 5. Average acceptance length (τ\tau) and decoding speedup on Gemma4-E4B-it and Qwen3.8-27B at temperature = 0.

Diagnostic Results

In this section, we first evaluate MMFlash under the high serving concurrency of real-world deployment, and then ablate three key design choices of the MMFlash drafter: the test-time block size, the drafter scale, and the number of target hidden states. All ablated drafters are trained on the same training data described in Section 5.1.

MMFlash Remains Effective under High Serving Concurrency. As shown in Table 3, MMFlash accelerates decoding at all concurrency levels from 1 to 32, peaking at 3.58× on Qwen3.5-9B. Although the speedup narrows as decoding becomes compute-bound, MMFlash at concurrency 16 still delivers higher throughput than the baseline at concurrency 32 on all six tasks, confirming its effectiveness in real-world serving. Results on vLLM [31] are consistent (as shown in Table 8 in Appendix § 12.§ 1).

TaskMethod1481632Avg. τ\tau
Qwen3.5-4B
CharXivBaseline238794130620942733–
MMFlash826<br>3.46×2054<br>2.59×3102<br>2.37×3943<br>1.88×4395<br>1.61×6.23
MMBenchBaseline24245367211411748–
MMFlash772<br>3.19×1032<br>2.28×1451<br>2.16×1929<br>1.69×2364<br>1.35×5.69
DynaMathBaseline239836145622763440–
MMFlash685<br>2.87×1991<br>2.38×3081<br>2.12×4066<br>1.79×4643<br>1.35×6.27
Qwen3.5-9B
ChartQABaseline174633104816941952–
MMFlash623<br>3.58×1732<br>2.73×2438<br>2.33×3080<br>1.82×3685<br>1.89×6.21
MMStarBaseline172633106416972462–
MMFlash460<br>2.67×1380<br>2.18×1964<br>1.85×2511<br>1.48×2814<br>1.14×4.59
DynaMathBaseline172655115518842665–
MMFlash520<br>3.02×1645<br>2.51×2541<br>2.20×3308<br>1.76×3792<br>1.42×6.49

Table 3. Decode throughput (tok/s), speedup over the baseline, and average acceptance length (τ\tau) on SGLang under increasing serving concurrency. For each MMFlash block the upper line reports throughput and the lower line the speedup at the same concurrency.

TaskMethod1481632Avg. τ\tau
Qwen3.5-4B
CharXivBaseline2841022178326903772–
MMFlash907<br>3.19×2555<br>2.50×3634<br>2.04×4590<br>1.71×5514<br>1.46×6.17
MMBenchBaseline27447970510511768–
MMFlash923<br>3.37×836<br>1.75×1347<br>1.91×1717<br>1.63×2859<br>1.62×5.74
DynaMathBaseline2951052182931994159–
MMFlash821<br>2.78×2430<br>2.31×3685<br>2.02×4705<br>1.47×5323<br>1.28×6.27
Qwen3.5-9B
ChartQABaseline188648124119112133–
MMFlash701<br>3.72×1975<br>3.05×2795<br>2.25×3318<br>1.74×3500<br>1.64×6.17
MMStarBaseline190700125320803076–
MMFlash529<br>2.79×1663<br>2.38×2461<br>1.96×2893<br>1.39×3606<br>1.17×4.59
DynaMathBaseline189708127221283127–
MMFlash602<br>3.18×1860<br>2.63×2810<br>2.21×3604<br>1.69×4001<br>1.28×6.57

Table 8. Decode throughput (tok/s), speedup over the baseline, and average acceptance length (τ\tau) on vLLM under increasing serving concurrency, complementing Table 3. For each MMFlash block the upper line reports throughput and the lower line the speedup at the same concurrency.

Ablation on Test-Time Block Size. Since the block size is a core design choice of MMFlash, we first ablate the test-time block size by evaluating MMFlash with block sizes of 8 and 16, where 16 matches the block length used during training. As shown in Table 4, MMFlash performs best with a block size of 16, which is aligned with the training configuration. When the test-time block size is smaller than that used in training, MMFlash remains robust: the average speedup decreases by only 5.3% and still surpasses all baselines in Table 1. In contrast, the average acceptance length drops by 18.7%, since a smaller block caps the number of tokens accepted per draft block and thus under-utilizes the drafter. For instance, 28.8% and 27.6% of draft blocks saturate the maximum acceptance length of 8 on ChartQA and CharXiv, respectively. Overall, the MMFlash drafter generalizes well to smaller test-time block sizes, which supports dynamic block-size scheduling in real-world serving.

ModelMethodChartQA τ\tauChartQA SpeedupCharXiv τ\tauCharXiv SpeedupMMStar τ\tauMMStar SpeedupMMBench τ\tauMMBench SpeedupDynaMath τ\tauDynaMath SpeedupMathVista τ\tauMathVista SpeedupMathVerse τ\tauMathVerse SpeedupAvg. τ\tauAvg. Speedup
Qwen3.5-4BMMFlash (8)4.703.01×4.773.13×3.802.49×4.522.89×4.922.80×4.682.74×4.432.83×4.552.84×
MMFlash (16)5.973.28×6.173.46×4.492.55×5.683.19×6.262.87×6.002.78×5.352.88×5.703.00×
Qwen3.5-9BMMFlash (8)5.073.22×4.723.17×4.012.65×3.772.30×5.182.94×5.022.96×4.682.95×4.642.89×
MMFlash (16)6.193.58×5.673.42×4.582.67×4.202.39×6.443.02×6.493.17×5.603.10×5.603.05×

Table 4. Effect of the test-time block size on MMFlash. Parenthesized values indicate the test-time block size for MMFlash. All drafters are trained with block size 16.

Ablation on the Scale of MMFlash Drafter. As shown in Table 6, increasing the depth of the MMFlash drafter from 3 to 8 layers improves the acceptance length, and the same trend holds at temperature = 1 (Table 9 in Appendix § 12). This indicates that the capability of the MMFlash drafter scales with its size. However, a deeper drafter also costs more per drafting pass: the average speedup rises by 6.7-8.1% from 3 to 5 layers but changes by less than 1% from 5 to 8 layers at both temperatures, so we default to 5 layers. Overall, the drafter scale should be chosen according to the application scenario to achieve the best trade-off between draft quality and drafting cost.

SettingCharXiv τ\tauCharXiv SpeedupMMBench τ\tauMMBench SpeedupDynaMath τ\tauDynaMath Speedup
3-L5.563.25×4.692.86×5.782.70×
5-L6.173.46×5.683.19×6.262.87×
8-L6.393.42×6.213.21×6.692.88×

Table 6. Effect of the number of drafter layers on MMFlash for Qwen3.5-4B at temperature = 0. nn-L denotes a drafter with nn layers; all drafters are trained with block size 16 and read hidden features from 5 layers of the target model.

SettingCharXivMMBenchDynaMath
τ\tauSpeedupτ\tauSpeedupτ\tauSpeedup
3-L4.912.59×4.502.48×5.342.43×
5-L5.392.86×5.252.58×5.752.56×
8-L5.592.83×5.882.67×5.942.54×

Table 9. Effect of the number of drafter layers on MMFlash for Qwen3.5-4B at temperature = 1 (top-pp = 0.95, top-kk = 20), complementing Table 6. nn-L denotes a drafter with nn layers; all drafters are trained with block size 16 and read hidden features from 5 layers of the target model.

Ablation on the Number of Target Hidden States. As shown in Table 7 (and Table 10 in Appendix § 12 for TT = 1), increasing the number of tapped target hidden states from 5 to 8 consistently improves acceptance length, suggesting that access to more layers provides richer semantic information. However, the speedup gain is marginal, at most 0.01×. Additional taps also increase training costs, requiring more storage for offline training and more time for online training (Appendix § 11.§ 2). In our setting, increasing the number of taps from 5 to 8 increases online training time by 19.9%, so we use 5 taps by default.

SettingCharXiv τ\tauCharXiv SpeedupMMBench τ\tauMMBench SpeedupDynaMath τ\tauDynaMath Speedup
5-H6.173.46×5.683.19×6.262.87×
8-H6.183.47×6.013.20×6.372.88×

Table 7. Effect of the number of tapped target hidden states on MMFlash for Qwen3.5-4B at temperature = 0. nn-H denotes a drafter that reads hidden features from nn layers of the target model; all drafters have 5 layers and are trained with block size 16.

SettingCharXivMMBenchDynaMath
τ\tauSpeedupτ\tauSpeedupτ\tauSpeedup
5-H5.392.86×5.252.58×5.752.56×
8-H5.402.86×5.402.59×5.832.57×

Table 10. Effect of the number of tapped target hidden states on MMFlash for Qwen3.5-4B at temperature = 1, complementing Table 7. nn-H denotes a drafter that reads hidden features from nn layers of the target model.

Conclusion

We present MMFlash, a novel diffusion drafting framework for speculative decoding in VLMs. We find that draft acceptance drops on visually dependent tokens, even after multimodal training. MMFlash therefore adaptively upweights these tokens while the drafter still predicts them poorly, and adopts sparse attention for long videos. It achieves the best average decoding speedups across seven image and three video benchmarks, remains effective under high serving concurrency, and generalizes to other model families, offering a strong baseline for future work.

LLM Usage Statement

We used Claude only to polish the writing of this paper, improving the grammar, wording, and readability of the text in all sections. All research ideas, methods, experiments, and results are by the authors, who reviewed all text and take full responsibility for the content of this paper.

Extended Related Work

Speculative Decoding for VLMs. Speculative decoding [5, 7, 34, 37] accelerates autoregressive generation by having a lightweight draft model propose several candidate tokens that the target model verifies in a single parallel forward pass, leaving the target’s output distribution unchanged. Work has since pursued drafters that are cheaper yet better aligned with the target, ranging from auxiliary decoding heads [5] to feature-level drafting with tree-structured verification [37, 38, 39] and, most recently, drafters built on diffusion language models [7, 42].

Bringing speculative decoding to VLMs introduces a design choice absent from the text-only setting: how the drafter should handle visual input. Existing methods fall into three paradigms according to this choice. The first exposes the drafter to the full visual input, strengthening its visual perception so as to raise the acceptance rate [17, 24, 41, 60]. The second supplies only distilled visual information, preserving the essential multimodal semantics while substantially reducing the drafter’s computation [20, 26, 28, 30, 47, 76]. The third, which this work adopts, removes visual tokens from the drafter altogether [16, 67, 74]. Since the target model has already fused the visual content into its hidden states during prefill, the drafter need not re-encode the visual input; its cost therefore no longer scales with the number of visual tokens, making this the fastest of the three paradigms. Rather than committing to a single paradigm, several methods select the visual-input strategy adaptively at inference time according to the task at hand [21, 32, 33]. Orthogonal to the visual-input axis, other efforts relax the verification criterion to admit more draft tokens [27], lift speculation from the token level to the agent level [22], or tailor speculative decoding to vertical domains such as document OCR [40].

Speculative Decoding with Diffusion Drafters. Diffusion large language models (dLLMs) [50, 53, 71, 78] recast text generation as iterative denoising: starting from a fully masked sequence, the model predicts every masked position in parallel and, over a small number of refinement steps, commits to confident tokens while re-masking uncertain ones. Because generation is not bound to a left-to-right order, a dLLM can emit many tokens in a single forward pass, offering test-time flexibility and efficiency that autoregressive decoding lacks, and recent large-scale DLMs achieve quality competitive with autoregressive LLMs [29, 50, 56]. Block diffusion [3] bridges the two families by running diffusion within fixed-size blocks that are themselves generated autoregressively, recovering KV caching and variable-length generation while retaining intra-block parallelism. These properties make DLMs natural drafters: whereas an autoregressive drafter spends one forward pass per token, a DLM can propose a whole block at once. Early explorations either blend the two paradigms within a single model or retrofit autoregressive models for parallel drafting. Liu et al. [43] jointly trains diffusion and autoregressive objectives to “think” in parallel and “talk” sequentially, at the cost of exact losslessness, while Samragh et al. [54] exploit the future-token information implicitly encoded in autoregressive LLMs, adding a LoRA adapter for parallel drafting and keeping the base model as verifier. DiffuSpec [35] and SpecDiff-2 [55] instead draft directly with large pre-trained DLMs, relying on inference-time search or train-test alignment to raise acceptance. DFlash [7] shows that the idea can be made simple and effective: a lightweight diffusion drafter emits an entire block of candidates in one forward pass, which the target then verifies as in standard speculative decoding. Subsequent work refines this recipe: DART [42] pursues a similar design; D-PACE [65] turns DFlash’s fixed position-wise loss weights into a dynamic schedule; Domino [23] adds causal dependencies within a block so that later positions see earlier committed tokens rather than only masks; and Nemotron [14] unifies autoregressive, diffusion, and self-speculative decoding in a single tri-mode model that drafts with diffusion and verifies autoregressively.

Block-Level Results of the Empirical Study

Figure 5 extends the token-level study in Figure 3 to drafted blocks, grouped by the visual-dependence quartile of their first predicted token. The trend is the same: averaged over benchmarks and drafters, block acceptance drops steadily from 33% in Q1 to 14% in Q4, a 2.4× gap. Training on multimodal data does not close this gap: it raises block acceptance by about 5 points in Q1 but by less than 3 points in Q4. These results further support the observation in § 4.1 that the acceptance loss of DFlash on VLMs concentrates on visually dependent tokens.

Additional Details of MMFlash

Quantile Normalization of Visual Dependence

Raw KL values are unbounded and heavy-tailed, so we first map them to the unit interval by their rank in the corpus D\mathcal{D} of all scored tokens,

g~t=FD(dt)=1∣D∣∑t′∈D1[dt′≤dt],(5)\widetilde{g}_t = F_{\mathcal{D}}(d_t) = \frac{1}{\lvert\mathcal{D}\rvert}\sum_{t' \in\mathcal{D}} \mathbf{1}[d_{t'} \le d_t], \tag*{(5)}

which is distribution-free: the quartiles of Figure 3 correspond to g~t∈[0,0.25)\tilde{g}_t \in[0, 0.25), [0.25,0.5)[0.25, 0.5), [0.5,0.75)[0.5, 0.75) and [0.75,1][0.75, 1].

Additional Experimental Setup

Training Data Construction

We regenerate all responses in the training set with the target model (greedy decoding, thinking disabled, at most 4,096 new tokens), and before regeneration we rewrite part of the prompts so that the target produces longer responses, which makes training more efficient. During training, the draft model is supervised on blocks that start at anchor positions sampled from the response tokens, with up to 512 anchors per sample. A short response therefore yields only a few training blocks, even though its image and prompt still cost a full forward pass of the target model, and samples whose response is shorter than 32 tokens are discarded altogether. However, for multimodal data, many source prompts restrict the answer to a single word, a short phrase, or an option letter, and with the original prompts 12.5% of the regenerated responses contain at most 8 tokens. We therefore replace such instructions with “reason step by step” or ”explain each option”, and append the same instruction to math problems and to questions from short-answer subsets such as chart, document, and table QA. Prompts that prescribe a specific output format, generation-style tasks such as captioning and OCR, and video samples are left unchanged, and the same rewriting is applied for every target model. For Qwen3.5-4B, 37.0% of the prompts are rewritten. For the prompts whose answer-format instruction is replaced, the median response length grows from 3 to 130 tokens, and over the whole training set, the share of responses with at most 8 tokens drops from 12.5% to 1.0%.

Offline and Online Training

Training the drafter requires the hidden states of the target model on the training data, which can be obtained in

two ways. Offline training runs the target model over the training set in advance and stores the tapped hidden states, which the drafter then reads during training. Online training instead runs the target model alongside the drafter, computing these hidden states for each batch on the fly and passing them directly to the drafter. Offline training computes the hidden states only once, but its storage grows with both the number of training tokens and the number of tapped layers, and typically reaches tens of terabytes for our data. We therefore adopt online training, which requires no extra storage at the cost of running the target model and transferring its tapped hidden states at every training step.

Inference and Evaluation

For each benchmark, we randomly select 200 samples and measure the acceptance length and decoding speedup of all methods on this same subset.

Additional Experimental Results

This section reports the experimental results that do not fit in the main paper.

Serving Concurrency on vLLM

Table 8 repeats the concurrency study of Table 3 on vLLM with the FlashAttention backend, keeping the benchmarks, prompts, generation cap, and MMFlash drafters unchanged; vLLM serves the drafter through its built-in DFlash speculator. MMFlash accelerates decoding on both target models at every concurrency level, by 1.17–3.72×, with acceptance lengths close to those on SGLang. On Qwen3.5-4B, the speedups are generally lower than on SGLang because vLLM improves the baseline more than MMFlash: averaged over the seven benchmarks of Table 1, vLLM raises the throughput of the baseline by 21–38% and that of MMFlash by 17–28% relative to SGLang. On Qwen3.5-9B, vLLM instead improves MMFlash more than the baseline at low concurrency, so the speedups at concurrency 1 and 4 exceed those on SGLang.

Drafter Ablations at Temperature 1

Tables 9 and 10 repeat the ablations on the number of drafter layers and on the number of target hidden states (Table 6 and Table 7) under sampling with temperature TT = 1, top-pp = 0.95, and top-kk = 20.

Limitations & Future Work

MMFlash is designed and evaluated for VLMs that take images or videos as input and generate text. For videos, it relies on a simple sparse attention pattern and lightweight fine-tuning rather than a drafter built specifically for video, and drafting on long videos remains harder than on images. A promising direction is therefore to tailor speculative decoding to videos, for example by selecting the drafter’s visual context according to the question, compressing the redundancy across frames, and measuring visual dependence at the frame level so that training focuses on tokens grounded in specific moments. Another direction is to extend MMFlash to omni-modal models [59, 68], which also take audio as input and generate streaming speech. This would require generalizing visual dependence to tokens that depend on audio or on joint audio-visual cues, handling even longer interleaved audio-visual contexts, and accelerating speech generation in addition to text.

References

  1. [1]J Achiam. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 1arxiv.org/abs/2303.08774
  2. [2]Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015. 1
  3. [3]Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. In ICLR, 2025. 1, 3
  4. [4]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Kevin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025. 1arxiv.org/abs/2511.21631
  5. [5]Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774, 2024. 1, 2arxiv.org/abs/2401.10774
  6. [6]Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023. 1, 2arxiv.org/abs/2302.01318
  7. [7]Jian Chen, Yesheng Liang, and Zhijian Liu. Dnash: Block diffusion for flash speculative decoding. arXiv preprint arXiv:2602.06036, 2026. 1, 3arxiv.org/abs/2602.06036
  8. [8]Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zhang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? In NeurIPS, 2024. 7arxiv.org/abs/2403.20330
  9. [9]Zizeng Chen, Gongfan Fang, Xinyin Ma, Ruonan Yu, and Xinchao Wang. Dmax: Aggressive parallel decoding for dllms. Advances in Neural Information Processing Systems, 2026. 1arxiv.org/abs/2604.08302
  10. [10]Sicheng Feng, Gongfan Fang, Xinyin Ma, and Xinchao Wang. Efficient reasoning models: A survey. arXiv preprint arXiv:2504.10903, 2025. 1arxiv.org/abs/2504.10903
  11. [11]Sicheng Feng, Kaiwen Tuo, Song Wang, Lingdong Kong, Jianke Zhu, and Huan Wang. Rewardmap: Tackling sparse rewards in fine-grained visual reasoning via multi-stage reinforcement learning. In ICLR, 2026. 1arxiv.org/abs/2510.02240
  12. [12]Sicheng Feng, Song Wang, Shuiyi Ouyang, Lingdong Kong, Zikai Song, Jianke Zhu, Huan Wang, and Xinchao Wang. Reasonmap: Towards fine-grained visual reasoning from transit maps. In CVPR, 2026. 1arxiv.org/abs/2505.18675
  13. [13]Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In CVPR, 2025. 7arxiv.org/abs/2405.21075
  14. [14]Yonggan Fu, Lexington Whalen, Abhinav Garg, Chengyue Wu, Maksim Khadkevich, Nicolai Oswald, Enze Xie, Daniel Egert, Sharath Turuvekere Sreenivas, Shizhe Diao, et al. Nemotron-labs-diffusion: A tri-mode language model unifying autoregressive, diffusion, and self-speculation decoding. arXiv preprint arXiv:2607.05722, 2026. 3, 1arxiv.org/abs/2607.05722
  15. [15]Yuming Fu, Weijia Wu, Jing Chen, Jiahao Tang, Feifei Chen, Hongyu Zhu, Xin Jin, and Alex Jinpeng Wang. Labinstruct: Benchmarking situated instructional video generation for lab procedures. arXiv preprint arXiv:2610.14020, 2026. 1arxiv.org/abs/2610.14020
  16. [16]Mukul Gagrani, Raghav Goel, Wonsook Jeon, Junyoung Park, Mingu Lee, and Christopher Lott. On speculative decoding for multimodal large language models. arXiv preprint arXiv:2404.08856, 2024. 2, 1arxiv.org/abs/2404.08856
  17. [17]Mugilan Ganesan, Shane Segal, Ankur Aggarwal, Nish Sindurai, Sean Lie, and Vithursan Thangarasa. Massv: Multimodal adaptation and self-data distillation for speculative decoding of vision-language models. In EMNLP, 2025. 2, 1
  18. [18]Geonmo Gu, Byeongho Heo, HeeJae Jun, Yoohoon Kang, Sangmin Lee, Sangdoo Yun, and Dongyoon Han. Verification-aware training for speculative decoding. arXiv preprint arXiv:2608.30135, 2026. 3arxiv.org/abs/2608.30135
  19. [19]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihua Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 1DOI
  20. [20]Yunhai Hu, Tianhua Xia, Zining Liu, Rahul Raman, Xingyu Liu, Bo Bao, Eric Sather, Vithursan Thangarasa, and Sai Qian Zhang. Dream: Drafting with refined target features and entropy-adaptive cross-attention fusion for multimodal speculative decoding. In NeurIPS, 2025. 2, 1arxiv.org/abs/2505.19201
  21. [21]Haiduo Huang, Fuwei Yang, Zhenhua Liu, Xuanwu Yin, Dong Li, Pengju Ren, and Emad Barsoum. Specvlm: Fast speculative decoding in vision-language models. arXiv preprint arXiv:2509.11815, 2025. 2, 1arxiv.org/abs/2509.11815
  22. [22]Haoyu Huang, Jinfa Huang, Zhongwei Wan, Xiawu Zheng, Rongrong Ji, and Jiebo Luo. Specueyes: Accelerating agentic multimodal llms via speculative perception and planning. arXiv preprint arXiv:2603.23483, 2026. 3, 1arxiv.org/abs/2603.23483
  23. [23]Jianuo Huang, Yaojie Zhang, Qituan Zhang, Hao Lin, Hanlin Xu, and Linfeng Zhang. Domino: Decoupling causal modeling from autoregressive drafting in speculative decoding. arXiv preprint arXiv:2605.29707, 2026. 1, 3arxiv.org/abs/2605.29707
  24. [24]Mingxiao Huo, Jiayi Zhang, Hewei Wang, Jinfeng Xu, Zheyu Chen, Huilin Tai, and Yijun Chen. Spec-Jlava: Accelerating vision-language models with dynamic tree-based speculative decoding. arXiv preprint arXiv:2509.11961, 2025. 2, 1arxiv.org/abs/2509.11961
  25. [25]Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024. 1arxiv.org/abs/2412.16720
  26. [26]Yicheng Ji, Jun Zhang, Jipeng Chen, Jinpeng Chen, Lidan Shou, Gang Chen, and Huan Li. Specvlm: Enhancing speculative decoding of video llms via verifier-guided token pruning. In EMNLP, 2025. 2, 1arxiv.org/abs/2508.16201
  27. [27]Yicheng Ji, Jun Zhang, Jinpeng Chen, Cong Wang, Lidan Shou, Gang Chen, and Huan Li. See the forest for the trees: Loosely speculative decoding via visual-semantic guidance for efficient inference of video llms. In ACL, 2026. 3, 1
  28. [28]Jialiang Kang, Han Shu, Wenshuo Li, Yingjie Zhai, and Xinghao Chen. Vispec: Accelerating vision-language models with vision-aware speculative decoding. In NeurIPS, 2025. 2, 1arxiv.org/abs/2509.15235
  29. [29]Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palecha, Stefano Ermon, et al. Mercury: Ultra-fast language models based on diffusion. arXiv preprint arXiv:2506.17298, 2025. 3, 1arxiv.org/abs/2506.17298
  30. [30]Quan Kong, Yuhao Shen, Yicheng Ji, Huan Li, and Cong Wang. Parallellvm: Lossless video-llm acceleration with visual alignment aware parallel speculative decoding. arXiv preprint arXiv:2603.19610, 2026. 2, 1arxiv.org/abs/2603.19610
  31. [31]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. 8arxiv.org/abs/2309.06180
  32. [32]Minjae Lee, Wonjun Kang, Minghao Yan, Christian Classen, Hyung Il Koo, and Kangkook Lee. In-batch ensemble drafting: Toward fast and robust speculative decoding for multimodal language models. https://openreview.net/forum?id=8o7131Lm83, 2024. 2, 1
  33. [33]Minjae Lee, Wonjun Kang, Byeongkeun Ahn, Christian Classen, Kevin Galim, Seunghyuk Oh, Minghao Yan, Hyung Il Koo, and Kangkook Lee. Tabed: Test-time adaptive ensemble drafting for robust speculative decoding in lvlms. In EACL, 2026. 2, 1
  34. [34]Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In ICML, 2023. 1, 2arxiv.org/abs/2211.17192
  35. [35]Guanghao Li, Zhihui Fu, Min Fang, Qibin Zhao, Ming Tang, Chun Yuan, and Jun Wang. Diffuspec: Unlocking diffusion language models for speculative decoding. In Findings of the Association for Computational Linguistics: ACL 2026, 2026. 3, 1arxiv.org/abs/2510.02358
  36. [36]Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In CVPR, 2024. 1, 7arxiv.org/abs/2311.17005
  37. [37]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077, 2024. 1, 2arxiv.org/abs/2401.15077
  38. [38]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. In EMNLP, 2024.arxiv.org/abs/2406.16858
  39. [39]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference acceleration of large language models via training-time test. NeurIPS, 2026. 1, 2, 6arxiv.org/abs/2503.01840
  40. [40]Wenhui Liao, Hongliang Li, Pengyu Xie, Xinyu Cai, Yufan Shen, Yi Xin, Qi Qin, Shengl ong Ye, Tianbin Li, Ming Hu, et al. Hsd: Training-free acceleration for document parsing vision-language models with hierarchical speculative decoding. arXiv preprint arXiv:2602.12957, 2026. 3, 1arxiv.org/abs/2602.12957
  41. [41]Luxi Lin, Zhihang Lin, Zhanpeng Zeng, and Rongrong Ji. Speculative decoding reimagined for multimodal large language models. arXiv preprint arXiv:2505.14260, 2025. 2, 1arxiv.org/abs/2505.14260
  42. [42]Fuliang Liu, Xue Li, Ketai Zhao, Yinxi Gao, Ziyan Zhou, Zhonghui Zhang, Zhibin Wang, Wanchun Dou, Sheng Zhong, and Chen Tian. Dart: Diffusion-inspired speculative decoding for fast llm inference. arXiv preprint arXiv:2601.19278, 2026. 1, 3arxiv.org/abs/2601.19278
  43. [43]Jingyu Liu, Xin Dong, Zhifan Ye, Rishabh Mehta, Yongan Fu, Vartika Singh, Jan Kautz, Ce Zhang, and Pavlo Molchanov. Tidar: Think in diffusion, talk in autoregression. arXiv preprint arXiv:2511.08923, 2025. 3, 1arxiv.org/abs/2511.08923
  44. [44]Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In ECCV, 2024. 7arxiv.org/abs/2307.06281
  45. [45]Ziyu Liu, Zeyi Sun, Yuhang Zhang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. In ICCV, 2025. 1arxiv.org/abs/2503.01785
  46. [46]Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR, 2024. 7arxiv.org/abs/2310.02255
  47. [47]Qitan Lv, Tianyu Liu, Wen Wu, Xuenan Xu, Bowen Zhou, Feng Wu, and Chao Zhang. Hippo: Accelerating video large language models inference via holistic-aware parallel speculative decoding. arXiv preprint arXiv:2601.08273, 2026. 2, 1arxiv.org/abs/2601.08273
  48. [48]Ahmed Masry, Jia Qing Tan, Shafiq Joty, Enamul Hoque, et al. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the association for computational linguistics: ACL 2022, 2022. 7
  49. [49]Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025. 1arxiv.org/abs/2502.09992
  50. [50]Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. In NeurIPS, 2026. 3, 1arxiv.org/abs/2502.09992
  51. [51]Qwen Team. Qwen3.5: Towards native multimodal agents, 2026. 2, 6
  52. [52]Qwen Team. Qwen3.8-Max: A new bar for coding and cowork, 2026. 7
  53. [53]Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In NeurIPS, 2024. 3, 1arxiv.org/abs/2406.07524
  54. [54]Mohammad Samragh, Arnav Kundu, David Harrison, Kumari Nishu, Devang Naik, Minsik Cho, and Mehrdad Farajt̄abar. Your llm knows the future: Uncovering its multi-token prediction potential. arXiv preprint arXiv:2507.11851, 2025. 3, 1arxiv.org/abs/2507.11851
  55. [55]Jemason Sandler, Jacob Christopher, Tom Hartvigsen, and Ferdinando Fioretto. Specdiff-2: Scaling diffusion drafter alignment for faster speculative decoding. Proceedings of Machine Learning and Systems, 2026. 3, 1arxiv.org/abs/2511.00606
  56. [56]Yuxuan Song, Zeng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang, Hongli Yu, Xingwei Qu, et al. Seed diffusion: A large-scale diffusion language model with high-speed inference. arXiv preprint arXiv:2508.02193, 2025. 3, 1arxiv.org/abs/2508.02193
  57. [57]EAGLE-4 Team. EAGLE-4: One flexible drafter for all batch sizes and hardware, 2026. Preprint. 1, 2
  58. [58]Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cárbune, Michelle Casbon, et al. Gemma 4 technical report. arXiv preprint arXiv:2607.02770, 2026. 1, 7
  59. [59]Qwen Team. Qwen3.5-omni: Scaling up, toward native omni-modal agi, 2026. 3
  60. [60]Yujia Tong, Tian Zhang, Yunyang Wan, Kaiwei Lin, Jingling Yuan, and Chuang Hu. Sage: Accelerating vision-language models via entropy-guided adaptive speculative decoding. arXiv preprint arXiv:2602.00523, 2026. 2, 1arxiv.org/abs/2602.00523
  61. [61]Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. In NeurIPS, 2024. 1arxiv.org/abs/2402.14804
  62. [62]Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. 1arxiv.org/abs/2508.18265
  63. [63]Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. In NeurIPS, 2024. 7arxiv.org/abs/2406.18521
  64. [64]Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. In NeurIPS, 2024. 1, 7arxiv.org/abs/2407.15754
  65. [65]Tianyu Wu, Yu Yao, Zhenting Qi, Han Zheng, Zhuohan Wang, Haoran Ma, Lawrence Liao, Himabindu Lakkaraju, Ju Li, and Yilun Du. D-pace: Dynamic position-aware cross-entropy for parallel speculative drafting. arXiv preprint arXiv:2605.18810, 2026. 1, 3arxiv.org/abs/2605.18810
  66. [66]Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. In Findings of the Association for Computational Linguistics: ACL 2024, 2024. 1arxiv.org/abs/2401.07851
  67. [67]Zhihan Xie, Peisong Wang, Shuang Qiu, and Jian Cheng. Hivis: Hiding visual tokens from the drafter for speculative decoding in vision-language models. In CVPR, 2026. 2, 1arxiv.org/abs/2509.23928
  68. [68]Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report. arXiv preprint arXiv:2503.20215, 2025. 3arxiv.org/abs/2503.20215
  69. [69]Weiye Xu, Jiahui Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, et al. Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models. In ICLR, 2026. 1
  70. [70]Tianjian Yang and Meng Li. Spec-auf: Accept-until-fail training under train-inference misalignment for masked block drafters. arXiv preprint arXiv:2607.01893, 2026. 3arxiv.org/abs/2607.01893
  71. [71]Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487, 2025. 1, 3arxiv.org/abs/2508.15487
  72. [72]Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR, 2024. 1arxiv.org/abs/2311.16502
  73. [73]Haohui Zhang, Keyu Chen, Haocheng Sun, Weibo Gu, Ruizhi Qiao, Xing Sun, and Bo Jiang. Draft in parallel, condition through depth: Adjacent causal injection for speculative decoding. arXiv preprint arXiv:2609.36173, 2026. 1arxiv.org/abs/2609.36173
  74. [74]Libo Zhang, Zhaoning Zhang, Peng Qiao, Dongsheng Li, et al. Sparrow: Text-anchored window attention with visual-semantic glimpsing for speculative decoding in video llms. In ACL, 2026. 2, 1
  75. [75]Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Do your multi-modal llm truly see the diagrams in visual math problems? In ECCV, 2024. 7arxiv.org/abs/2403.14624
  76. [76]Xuan Zhang, Cunxiao Du, Sicheng Yu, Jiawei Wu, Fengzhuo Zhang, Wei Gao, and Qian Liu. Sparse-to-dense: A free lunch for lossless acceleration of video understanding in llms. In ACL, 2025. 2, 1
  77. [77]Yuan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024. 7arxiv.org/abs/2410.02713
  78. [78]Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccu rate categorical sampling. arXiv preprint arXiv:2409.02908, 2024. 3, 1arxiv.org/abs/2409.02908
  79. [79]Liamin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody H Y Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. In NeurIPS, 2024. 7
  80. [80]Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. In ICLR, 2025. 7arxiv.org/abs/2411.00836

Paper details

vlnc:W_stsxpuuj
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude

View statement
Looking for related papers…
Contents