betaFeedback

WorksComputer Science

CoTA++: Understanding the Repeat Curse in dMLLMs from an Information Flow Perspective

Abstract

Diffusion-based multimodal large language models (dMLLMs) decode by iterative denoising with bidirectional attention, which precludes the standard autoregressive key–value cache. Practical deployments therefore rely on approximate caching that reuses intermediate states across denoising steps. Such caching, however, often makes a response repeat one token at adjacent positions in a run that breaks the sentence, a failure we term the Repeat Curse. Tracing the context-token information flow reveals three disruptions. Within a step, caching disperses the attention that anchors a position on nearby context tokens. Across steps, similarity-based refresh is not staleness-aware, so some positions are decoded against stale key–value states. Across layers, the deep-layer entropy of the context fails to converge where repetition arises. We accordingly propose CoTA++, a training-free, plug-and-play framework that counteracts each disruption where it originates: Context-Token Attention Re-anchoring re-forms the attention routing against the current context, Decode-Aware Refresh reserves part of the cache's refresh budget for the positions about to be committed, and Context-Token Entropy-Guided Voting discounts the confidence of a position by the deep-layer entropy of its context, which reorders the unmasking. The three components act at non-overlapping stages of a decoding step and compose freely with existing dMLLMs and caches. Across three dMLLMs and two caching backends, CoTA++ reduces the sample repetition rate of long responses from up to 99.6% to at most 17.0%, recovers most of the accuracy lost to caching on nine benchmarks, and keeps the speedup of caching at under 15% overhead, making approximate caching reliable for dMLLMs.

Introduction

Diffusion language models (dLLMs) [2, 3, 4, 5] have recently been extended with a vision encoder and vision-instruction tuning [6] into diffusion-based multimodal large language models (dMLLMs) [7, 8, 9, 10, 11, 12, 13], which now rival their autoregressive counterparts [14, 15] on multimodal benchmarks. They decode by iterative denoising: starting from a fully masked response, each step predicts all masked positions in parallel and commits the most confident ones. This parallelism promises efficient decoding, but in practice each step recomputes the whole sequence. Bidirectional attention lets every update perturb the key–value (KV) states of all tokens, so the exact KV cache of autoregressive decoding does not apply [16, 17] and inference remains slow. Recent work instead reuses intermediate states across adjacent steps, an approximate caching guided by cross-step feature similarity [18, 19, 20], which yields large speedups and has become the de facto acceleration recipe for dMLLMs [7, 8, 9].

Q. Zhao, X.-Y. Zhang, S. Zhu and D.-H. Wang are with the State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China.

X. Zhang is with Shanghai Jiao Tong University, Shanghai 200240, China.

S. Han and Z. Li are with the Faculty of Business and Economics, The University of Hong Kong, Hong Kong, China.

Q. Zhao and X.-Y. Zhang are the School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS), Beijing 100049, China.

Corresponding authors: D.-H. Wang (e-mail: [email protected]) and X.-Y. Zhang (e-mail: [email protected]).

An earlier version of this work appeared at ICLR 2026 [1].

Code is publicly available at https://github.com/ErikZ719/CoTA.

Figure 1. Motivation. LLaDA-V [7] responses to the same query: vanilla decoding is normal (top); dLLM-Cache [18] induces the Repeat Curse (repetitions in red, middle); CoTA++ removes them while retaining the acceleration (bottom).

This recipe has a severe side effect. Under caching, a response often repeats one token at adjacent positions, a run that breaks the sentence (Fig. 1). We term this failure the Repeat Curse. Measured as the share of responses that contain such a run (SRR, ∣IV|\mathrm{IV}-A) on 500 MS-COCO [21] captions, it affects up to 99.6% of long responses and arises across dMLLMs (LLaDA-V [7], MMaDA [8], LaViDa [9]) and caching backends (dLLM-Cache [18], SlowFast [19]; Figure 2(a)). Its model- and backend-agnostic nature points to a mechanistic disruption rather than a sampling artifact. The question this paper answers is therefore: why does approximate caching cause the Repeat Curse in dMLLMs?

The Repeat Curse and its three information-flow signatures

Figure 2. The Repeat Curse and its three information-flow signatures. (a) Sample repetition rate on 500 MS-COCO captions, L=512L=512. (b)–(d) LLaDA-V, L=128L=128, vanilla against dLLM-Cache. (b) Within-step attention of suffix queries: caching disperses local anchoring (F1). (c) KV staleness at the decode moment along one response: most repeats (red) decode stale (F2). (d) Cross-layer entropy after generation at that response’s repeat positions and at normal positions: repeats fail to consolidate (F3).

This question calls for information-flow interpretability [22, 23, 24], which has already served to diagnose abnormal generation [25, 26]. The inference of a dMLLM is indexed by token, step and layer, so information flows along exactly three axes. Along each axis, we trace the flow of the context tokens, the tokens around a position being decoded, in the vanilla model and in its cached variant. The result is the thesis of this paper (Figure 3): the Repeat Curse is the convergent symptom of three distinct disruptions. (F1) Routing. Within a step, a position draws its attention mainly from nearby context tokens, which anchor locally coherent generation. Caching disperses this anchoring where repetition arises (Figure 2(b)). (F2) Timeliness. Across steps, a position reads its context tokens through cached KV states. The cache refreshes by feature similarity rather than by staleness, so some positions are committed on a view of their context that is several steps old, and repetition concentrates on them (Figure 2(c)). (F3) Consolidation. Across layers, the deep-layer entropy of context tokens normally converges as the model consolidates its prediction. Under caching it fails to converge at precisely the tokens that repeat, to the end of generation (Figure 2(d)).

Figure 3. Overview. Caching disrupts the information flow of context tokens on the three axes of dMLLM inference, and CoTA++ counteracts each disruption where it originates.

Each disruption prescribes its own remedy, and CoTA++1 applies the three in one training-free, plug-and-play decoding loop. Context-Token Attention Re-anchoring (CTAR) reforms the attention of a position against the current context, restoring the anchoring that caching disperses (against F1). Decode-Aware Refresh (DAR) reserves part of the cache’s refresh budget for the positions closest to commitment, so that tokens are not committed against stale states (against F2).

Context-Token Entropy-Guided Voting (CTEV) discounts the confidence of a candidate by the deep-layer entropy of its context, so that the resulting voting score favors candidates in consolidated contexts (against F3). They act on different objects at different moments of a step, the attention output, the refresh set and the voting score, and therefore compose freely with each other and with existing dMLLMs and caches.

Extension over the conference version. This paper extends our ICLR 2026 work [1], which introduced the Repeat Curse, identified context tokens as anchors and proposed CTAR2 and CTEV. The cross-step axis and F2 are new, DAR follows from them, and the study grows to three dMLLMs, two caching backends, three lengths and nine benchmarks, with per-component efficiency and design analyses.

Our contributions are three-fold:

  1. We characterize the Repeat Curse [1] at scale and explain it: tracing the context-token information flow along the token, step and layer axes localizes the failure to attention routing within a step, cache timeliness across steps and entropy consolidation across layers, and controlled comparisons separate the three.

  1. We derive CoTA++ from this diagnosis: a training-free, plug-and-play framework whose components CTAR, DAR and CTEV counter each disruption where it originates and compose with existing dMLLMs and caches.

  1. Across three dMLLMs and two caching backends, CoTA++ reduces the sample repetition rate of long-form generation from up to 99.6% to at most 17.0%, recovers most of the accuracy lost to caching on nine benchmarks, and adds under 15% latency overhead.

Related Work

Diffusion Language Models and dMLLMs

Following the success of diffusion models in image generation [27, 28], diffusion has been brought to language modeling in continuous [29, 30] and discrete [31, 32, 33] forms. The masked discrete branch has proved the most scalable: recent dLLMs [2, 3, 4, 5, 34, 35] reach 8B parameters and beyond, with performance comparable to autoregressive LLMs [14, 15]. Pairing a dLLM with a vision encoder [36, 37] and a projector, and training it with vision-instruction tuning [6], yields dMLLMs [7, 8, 9, 10, 11, 12, 38, 39, 40, 41]. Our study concerns the property that defines this class: inference unrolls many denoising passes over the entire sequence, which makes caching both indispensable and, as we show, hazardous.

Inference Acceleration for Diffusion Multimodal LLMs

For dMLLMs, inference remains the principal deployment bottleneck: every denoising step recomputes the attention and hidden states of the entire multimodal sequence [7, 8, 9]. The KV cache of autoregressive decoding [42] is exact because causal attention never changes past states. Bidirectional attention perturbs all states at every update, so acceleration resorts to approximate caching, which trades an approximation error for the reuse of states across adjacent steps [18].

Existing designs differ in what they reuse and when they refresh. Prefix-KV caching [9] caches the prompt states once, kept exact by its prefix attention mask, and recomputes the whole suffix at every step, which D3^{3}ToM [17] extends with adaptive visual-token merging. Fast-dLLM [20] caches the committed blocks and recomputes from the current block onward. dLLM-Cache [18] caches per position, pairing a long refresh interval for the slowly varying prompt with a short one for the suffix and recomputing at every step only the suffix positions whose features change most. Others refine the schedule or the store: SlowFast [19] separates cautious

MethodCache granularityRefresh criterionDecode-awareIntervention level
Fast-dLLM [20]blockblock boundary×\timesschedule
Prefix-KV [9]prefix onlynone (exact)n/aschedule
dLLM-Cache [18]per positionperiodic + similarity×\timesschedule
SlowFast [19]per positionperiodic + similarity×\timesschedule + decoding
dky-cache [47]per positiondelayed commit×\timesschedule + decoding
CoTA [1]plug-ininherited×\timesattention + decoding
CoTA++ (Ours)plug-ininherited + commitment rank✓\checkmarkattention + cache + decoding

Table I. Positioning of CoTA++ among dMLLM caching methods. Only CoTA++ makes its refresh decode-aware.

exploration from fast parallel decoding, and MaskKV [43] evicts cache entries to curb memory. Further variants continue to appear [44, 45, 46, 47].

Across these designs, the refresh schedule is a global, per-step budget: none consults the decoder’s ranking of the positions it is about to commit, and none intervenes in the flow that consumes those states. They are also evaluated mainly by task accuracy, which overlooks repetition that leaves the final answer intact, rather than on the open-ended long-form generation in which the approximation error compounds (§VI-B). CoTA++ (Table I) is a training-free plug-in that inherits the backend’s cache, makes its refresh decode-aware, and intervenes where the cached states are consumed.

Information-Flow Interpretability

Information-flow analysis [48, 49, 50] offers an interpretable lens on the internal mechanisms of black-box models. Its instruments include attention-map analysis [51, 52, 53], entropy-based readouts [54, 55], saliency attribution [23, 56], Grad-CAM [57, 58] and massive-value analyses [59, 60]; our diagnosis uses the first two. In autoregressive (M)LLMs, such analyses have traced abnormal outputs to their internal mechanisms, including hallucination [61, 62] and text degeneration [25, 63]. For dLLMs, interpretability is only beginning to emerge, with recent analyses of dynamic attention sinks [64, 65] and attribution methods for dMLLMs [66]. We bring this lens to cached dMLLMs and trace the flow of context tokens along all three axes of their inference.

Token Repetition in Generation

Early research on token repetition and degeneration focused on n-gram penalties [67], decoding strategies [68, 69] and training-time objectives [70, 71], and later work extended the study to modern LLMs [72, 73, 74]. Most of them mitigate repetition without explaining it. The exceptions analyze autoregressive models, attributing repetition to latent features [54] or to specific neurons [75]. In dMLLMs, our conference version [1] was, to our knowledge, the first to identify and analyze repetitive generation, tracing it to approximate caching. A more recent study [76] shows that cache-free decoding also repeats under aggressive step reduction and attributes this to mask-prior drift. This paper deepens the caching line: the three-axis information-flow analysis of §IV explains the Repeat Curse, and a mitigation follows for each axis.

Preliminaries

This section formalizes the building blocks on which our analysis and method rest: the dMLLM architecture (§III-A), iterative denoising inference (§III-B) and approximate KV caching (§III-C). Throughout the paper we adopt LLaDA-V [7] and dLLM-Cache [18] as the running examples of a representative dMLLM and caching backend.

Architecture of dMLLMs

A dMLLM F\mathcal{F} such as LLaDA-V comprises three components: a pretrained vision encoder Fv\mathcal{F}_v, a diffusion-based language model (dLLM) Ft\mathcal{F}_t and a projector ff that maps visual features into the text embedding space. Given an image IvI_v and an instruction prompt IpI_p (e.g., “Please describe the image in detail”), the image is encoded and projected into visual tokens Sv=f(Fv(Iv))S_v = f(\mathcal{F}_v(I_v)) and the instruction is tokenized into instruction tokens SpS_p. Their concatenation {Sv,Sp}\{S_v, S_p\} constitutes the prefix. A fully masked suffix Sm={[MASK]i}i=1MS_m = \{[\mathrm{MASK}]_i\}_{i=1}^{M} of maximum response length MM, written LL when we vary it in the experiments, is then appended, and the complete sequence S={Sv,Sp,Sm}S = \{S_v, S_p, S_m\} is fed to Ft\mathcal{F}_t for iterative denoising. All tokens share the same hidden dimensionality dd.

Iterative Denoising Inference

Let St={Svt,Spt,Smt}S^t = \{S_v^t, S_p^t, S_m^t\} denote the sequence state before decoding step t∈{0,1,…,N−1}t \in\{0,1,\ldots,N-1\}, initialized as S0=SS^0 = S with its suffix entirely masked. The prefix tokens are fixed throughout (Svt=SvS_v^t = S_v, Spt=SpS_p^t = S_p), and only the suffix evolves. Over the NN steps, the model progressively commits tokens at masked positions, and the suffix of the final state, SmNS_m^N, is the generated response. At each step tt, the model estimates, for every masked position ii, a categorical distribution over the vocabulary V\mathcal{V}, and the corresponding greedy prediction is

S^(i)t=arg⁡max⁡v∈V pθ(S(i)=v∣St),i∈Imaskt,(1)\hat{S}_{(i)}^t = \underset{v\in\mathcal{V}}{\arg\max}\ p_\theta\left(S_{(i)}=v\mid S^t\right),\quad i\in\mathcal{I}_{\mathrm{mask}}^t, \tag*{(1)}

where S(i)S_{(i)} denotes the token at suffix position ii and Imaskt\mathcal{I}_{\mathrm{mask}}^t the set of masked suffix positions in StS^t. Since a single parallel pass yields unreliable predictions at many positions, dMLLMs commit only the most confident predictions at each step and re-predict the rest, a strategy known as confidence-based remasking. The confidence score of each masked position is

c(i)t=pθ(S(i)=S^(i)t∣St),i∈Imaskt.(2)c_{(i)}^t = p_\theta\left(S_{(i)}=\hat{S}_{(i)}^t\mid S^t\right),\quad i\in\mathcal{I}_{\mathrm{mask}}^t. \tag*{(2)}

The model selects the kk positions with the highest c(i)tc_{(i)}^t (k=M/Nk=M/N under the uniform schedule used throughout), forming the update set Iupdatet\mathcal{I}_{\mathrm{update}}^t, and produces

S(i)t+1={S^(i)t,if i∈Iupdatet,S(i)t,otherwise,(3)S_{(i)}^{t+1} = \begin{cases} \hat{S}_{(i)}^t, & \text{if } i\in\mathcal{I}_{\mathrm{update}}^t,\\ S_{(i)}^t, & \text{otherwise}, \end{cases} \tag*{(3)}

with Imaskt+1=Imaskt∖Iupdatet\mathcal{I}_{\mathrm{mask}}^{t+1}=\mathcal{I}_{\mathrm{mask}}^t\setminus\mathcal{I}_{\mathrm{update}}^t. We drop the step superscript of c(i)tc_{(i)}^t where the step is clear from the context.

Approximate KV Caching for dMLLMs

Because dMLLM attention is bidirectional rather than causal, no token’s KV states are guaranteed to stay fixed across denoising steps, and practical schemes therefore rely on approximate caching. They share a common starting point: the features of prefix tokens {Sv,Sp}\{S_v,S_p\} and of most suffix tokens vary slowly across adjacent steps [18, 19]. dLLM-Cache, our running backend, accordingly organizes per-step recomputation by token type:

Recompute at step tt:

{{Svt,Spt},if t mod Ep=0,Smt,if t mod Es=0,S(i)t,if i∈Pαt,(4)\begin{cases} \{S_v^t,S_p^t\}, & \text{if } t\bmod\mathcal{E}_p=0,\\ S_m^t, & \text{if } t\bmod\mathcal{E}_s=0,\\ S_{(i)}^t, & \text{if } i\in\mathcal{P}_\alpha^t, \tag*{(4)} \end{cases}

where prefix states are refreshed every Ep\mathcal{E}_p steps and the entire suffix every Es\mathcal{E}_s steps. At all other steps, Pαt\mathcal{P}_\alpha^t collects the αM\alpha M suffix positions with the lowest cross-step similarity L(i)t\mathcal{L}_{(i)}^t, the cosine between position ii’s current and cached features, so only the tokens whose features have drifted most are recomputed, and the rest are reused.3

Motivation and Analysis

This section analyzes how approximate KV caching (§III-C) triggers the Repeat Curse. After defining metrics (§IV-A), it takes the three axes of Section I in turn, routing within a step (F1, §IV-B), timeliness across steps (F2, §IV-C) and consolidation across layers (F3, §IV-D), and closes with a joint account that separates the three disruptions (§IV-E). Each finding is summarized in a box of the same form: observation, mechanism, evidence and implication.

Quantifying the Repeat Curse

Repeat-Curse metrics. To quantify the token repetition illustrated in Fig. 1, we introduce the Adjacent Repetition Rate (ARR), computed on the content tokens y=(y1,…,y∣y∣)y=(y_1,\ldots,y_{\lvert y\rvert}) of a generated response (the decoded suffix SmNS_m^N of §III-B, truncated at its first end-of-text token):

ARR=1∣y∣−1∑i=1∣y∣−11(yi=yi+1),(5)\mathrm{ARR}=\frac{1}{\lvert y\rvert-1}\sum_{i=1}^{\lvert y\rvert-1}\mathbf{1}(y_i=y_{i+1}), \tag*{(5)}

where 1(⋅)\mathbf{1}(\cdot) is the indicator function. The second token of each adjacent identical pair marks a repeat position, the criterion followed by every position-level analysis in this section. The Sample Repetition Rate (SRR) measures the prevalence of repetition over a test set of nn responses:

SRR=nrep/n,(6)\mathrm{SRR}=n_{\mathrm{rep}}/n, \tag*{(6)}

where nrepn_{\mathrm{rep}} counts the responses with ARR >0>0. Three additional metrics characterize the severity of repetition: with {rk}\{r_k\} collecting the lengths of all maximal runs of identical tokens (rk≥2r_k\ge2) over the nn responses, we report the maximum, average and 95th-percentile repetition lengths MRL=max⁡krk\mathrm{MRL}=\max_k r_k, ARL=meankrk\mathrm{ARL}=\mathrm{mean}_k r_k and 95pRL=Quantile0.95{rk}\mathrm{95pRL}=\mathrm{Quantile}_{0.95}\{r_k\}. Classical degeneration metrics measure windowed or nn-gram recurrence [68, 77]. ARR instead targets the runs of identical tokens that dominate the Repeat Curse.

Severity across models, backends and settings. Figure 4 visualizes the Repeat Curse on LLaDA-V and MMaDA under two caching backends at L=64L=64 and L=128L=128, on detailed captioning over 500 MS-COCO images [21] (full values in Table VI). Each axis is scaled by its value under dLLM-Cache, printed on the outer ring, and overbars denote run-length metrics averaged per response. Without a cache both models are essentially repetition-free, with an SRR of at most 0.6%0.6\%. dLLM-Cache raises the SRR of LLaDA-V to 70.8%70.8\% at L=64L=64 and 96.8%96.8\% at L=128L=128, a 484×484\times increase at the longer length, and that of MMaDA to 98.6%98.6\% and 88.6%88.6\%. ARR follows, rising from below 0.2%0.2\% to between 1.6%1.6\% and 11.1%11.1\%. SlowFast is markedly milder on LLaDA-V, yet still lifts its SRR by nearly two orders of magnitude, to 31.2%31.2\% and 17.4%17.4\%, and on MMaDA it is barely milder than dLLM-Cache (96.4%96.4\% and 83.0%83.0\%). Which backend is milder therefore depends on the model, and neither avoids the curse.

Figure 4. Radar charts of the Repeat Curse. Near zero without caching, every metric surges with it (labels: vanilla→cached).

ModelMethodL=512L=512 ARR↓SRR↓MRL↓ARL↓95pRL↓L=128L=128 ARR↓SRR↓MRL↓ARL↓95pRL↓L=64L=64 ARR↓SRR↓MRL↓ARL↓95pRL↓Avg. ARR↓SRR↓
LLaDA-V [7]Vanilla0.012.432.072.60.000.222.002.00.010.422.002.00.011.0
dLLM-Cache [18]14.7099.64555.903.07.0196.8732.883.01.6170.8382.122.07.7789.1
+ CoTA [1]0.1747.432.002.00.1011.422.002.00.042.422.002.00.1020.4
+ CoTA++0.038.232.123.00.022.422.002.00.010.422.002.00.023.7
SlowFast [19]0.2817.62322.892.00.2217.432.032.00.5531.222.002.00.3522.1
+ CoTA [1]0.156.82868.332.00.043.222.002.00.031.422.002.00.073.8
+ CoTA++0.023.232.103.00.021.622.002.00.010.422.002.00.021.7
MMaDA [8]Vanilla0.000.422.002.00.010.622.002.00.190.65720.3351.50.070.5
dLLM-Cache [18]11.3597.44093.225.06.1888.6342.504.011.0898.6302.484.09.5494.9
+ CoTA [1]0.1814.272.123.00.119.632.022.00.2513.032.062.60.1812.3
+ CoTA++0.000.222.002.00.021.622.002.00.021.222.002.00.011.0
SlowFast [19]8.6896.04002.975.04.7883.0352.414.09.1596.4352.404.07.5491.8
+ CoTA [1]0.1412.232.032.00.1513.822.002.00.1910.432.022.00.1612.1
+ CoTA++0.011.222.002.00.010.822.002.00.010.422.002.00.010.8
LaViDa [9]Vanilla0.022.232.273.00.054.632.263.00.021.022.002.00.032.6
dLLM-Cache [18]6.6298.44653.363.01.5675.6892.212.01.6424.6472.563.03.2766.2
+ CoTA [1]0.3843.832.022.00.1618.432.022.00.042.232.092.50.1921.5
+ CoTA++0.046.622.002.00.079.232.022.00.031.822.002.00.055.9
SlowFast [19]5.6785.64282.723.01.4179.6412.102.00.696.4503.493.02.5957.2
+ CoTA [1]0.3428.432.022.00.2017.8162.312.00.031.432.142.70.1915.9
+ CoTA++0.1617.032.012.00.078.632.022.00.031.622.002.00.099.1

Table VI. Repeat Curse across dMLLMs and caching backends on the 500 COCO images at three response lengths LL, under the instruction “Please describe the image in detail.”, on content tokens (§VI-A). ARR and SRR are in percent. MRL, ARL and 95pRL are the maximum, mean and 95th percentile of the lengths of all repeated runs in the 500 responses. Avg.: mean over the three lengths. ↓: lower is better. Bold: best within a backend group. Shaded: CoTA++.

Localizing the responsible cache component. To identify which cache component drives the failure, Table II sweeps the three design choices of Eq. (4), namely the prefix interval Ep\mathcal{E}_p, the suffix interval Es\mathcal{E}_s and the similarity fraction α\alpha, and contrasts dLLM-Cache with a prefix-only reuse policy [9]. Varying Ep\mathcal{E}_p barely moves the SRR, which stays between 96.8%96.8\% and 99.0%99.0\%. Stretching Es\mathcal{E}_s from 1 to 7 drives it from 0.6%0.6\% to 96.8%96.8\%, and lowering α\alpha from 1 to 0 drives it from 0.4%0.4\% to 99.8%99.8\%. The two ends agree: Es=1\mathcal{E}_s=1, which is the prefix-only policy, and α=1\alpha=1 both recompute every suffix position at every step, and both return the SRR to the uncached level. The Repeat Curse is thus driven mainly by the approximate caching of suffix tokens, and the suffix is the segment that our subsequent analysis and method target.

Routing Within a Step: Dispersed Anchoring (F1)

We begin with the within-step routing axis and visualize the attention of the suffix queries. In the uncached model, the attention maps show a pronounced diagonal that persists across layers and decoding steps (Figure 5(a)): queries concentrate on their context tokens, the tokens at neighboring positions. The pattern resembles the anchor tokens of autoregressive LLMs, such as attention sinks [78] and label words [23], which absorb a large share of attention and relay information. The anchors here, however, are relative: under bidirectional attention every query has its own, set by distance rather than by token identity. Approximate caching disperses this anchoring where repetition arises (Figure 2(b)): attention there moves away from the context tokens and spreads over distant positions.

Four attention and anchoring plots: (a) Within-step information flow of vanilla LLaDA-V; (b) Regional loss of anchoring; (c) Compounding along decoding; (d) Loss versus missed anchors

Figure 5. Context tokens are anchors, and caching disperses them (Finding F1). (a) Attention of vanilla LLaDA-V (brighter is higher). (b)–(d) Change in local anchoring under dLLM-Cache on 94 MS-COCO images, Δw5\Delta w_5 (cached − vanilla), where w5w_5 is the share of a position’s suffix attention within ±5\pm5 positions. Repeat regions: repeat positions ±5\pm5. Distant regions: all other positions. Numbers in (b): ratio of the two losses. Error bars: 95% intervals over images.

Quantifying the dispersion. Local anchoring is scored by w5w_5, the share of a position’s suffix attention that falls within ±5\pm5 positions. We compare the cached with the vanilla run at each position’s own decoding step, on the 94 of 100 MS-COCO images that repeat under dLLM-Cache. The loss of anchoring is regional (Figure 5(b)): repeat regions (repeat positions ±5\pm5) lose 2.8 to 7.6 times as much as distant regions, most in the deep band. It also compounds along decoding (Figure 5(c)), growing with every quarter of the trajectory in the middle and deep bands. Both hold across images (all p<10−6p < 10^{-6}). Yet even in repeat regions the loss averages about a tenth of the vanilla anchoring mass: anchoring is weakened where repetition emerges, not abolished.

Dispersion grows with the anchors a position has missed. The cache reuses the routing of a position until it is formed again, so a context token within ±5\pm5 committed in the meantime is an anchor that this routing predates, a missed anchor. The deep-band loss grows with the number of missed anchors (Figure 5(d)), and repeat regions miss more than twice as many as distant ones (1.01 against 0.40 per position). Both observations hold across images (both p<10−10p < 10^{-10}).

Component sweepSRR (%)↓
Prefix interval Ep\mathcal{E}_p1: 99.05: 97.215: 97.625: 96.8
Suffix interval Es\mathcal{E}_s1: 0.63: 68.45: 90.87: 96.8
Similarity fraction α\alpha0: 99.80.25: 96.80.75: 25.21: 0.4
Reuse policyprefix-only: 0.6dLLM-Cache: 96.8——

Table 2. Ablation of dLLM-Cache components on LLaDA-V (L=128L=128; SRR, %). Each row sweeps one component with the others at their defaults (shaded).

F1: Context tokens act as information-flow anchors. Observation. Neighboring context tokens anchor the within-step attention routing. Caching disperses this anchoring where repetition arises. Mechanism. A cached routing misses the anchors committed since it was formed. Evidence. The loss is regional, compounds along decoding and grows with the missed anchors (Figure 5). Implication. The routing must be re-formed against the current context tokens, which is what CTAR does (§V-A).

Timeliness Across Steps: Staleness-Driven Fracture (F2)

We next turn to the cross-step timeliness axis. Across steps, each prediction reads the evolving sequence state through the KV states of the surrounding positions. Once these states are cached and reused (§III-C), the cache carries the flow across steps, and its refresh policy decides how timely that flow is. Approximate caches such as dLLM-Cache refresh by feature similarity, a criterion that is not staleness-aware: a minority of suffix positions go unrefreshed for several consecutive steps and are decoded against outdated KV states. The analysis below formalizes this lag and traces it to the repeat positions.

Cache staleness Δit\Delta_i^t. Let Rt⊆{1,…,M}R^t \subseteq\{1,\ldots,M\} be the set of suffix positions recomputed at step tt under the cache’s refresh policy. Refresh decisions are made per layer. RtR^t is their union across layers, and all statistics below are computed from it. The staleness of position ii at step tt,

Δit=t−max⁡{t′≤t:i∈Rt′},(7)\Delta_i^t = t - \max\{t' \le t : i \in R^{t'}\}, \tag*{(7)}

counts the decoding steps since the position was last recomputed, so that Δit=0\Delta_i^t=0 exactly when i∈Rti \in R^t. At staleness Δ\Delta, a position is thus decoded against KV states that reflect St−ΔS^{t-\Delta} or an earlier state. Because RtR^t is a union, Δ\Delta lower-bounds the staleness of each layer, and the bound is tight in over 99% of the layers at the decode moment (see the supplementary material). Vanilla dMLLMs recompute every position at every step (Δ≡0\Delta\equiv0), so each prediction is conditioned only on the current state. Approximate caching breaks this guarantee wherever Δit>0\Delta_i^t > 0.

Cache staleness concentrates on repeat positions. Staleness is read at each position’s decode moment, the step at which it is committed. On the same 94 images, normal positions concentrate at low staleness (median 1) and repeat positions in the stale tail (median 4, Figure 6(a)). Half of the repeat positions are decoded beyond τ=3\tau=3, the midpoint of the staleness range, against a quarter of the normal ones (p<10−14p < 10^{-14} across images). Staleness and repetition both grow along the trajectory, yet the contrast is not merely one of lateness: in every quarter after the first, repeat positions remain staler than normal ones (all p≤2.7×10−5p \le2.7 \times10^{-5}, Table III), most markedly in the second, where repetition sets in (0.63 against 0.19).

Figure 6. Cache staleness concentrates on repeat positions (Finding F2). * (a) Staleness at the decode moment on 94 images. (b)–(d) Reference sample, on shared axes: (b) staleness of each position up to its decode (faded afterwards), (c) and (d) how each KV state is read at layer 22 (attention received, averaged over heads and queries). Triangles: repeat positions.

PositionsΔ>τ\Delta>\tau
(a) Quarter (steps)RepeatNormalRepeatNormalImagespp
1st (0–31)152,8990.200.023/130.54
2nd (32–63)1102,8980.630.1943/517.8×10−97.8 \times10^{-9}
3rd (64–95)3372,6710.470.4051/702.7×10−52.7 \times10^{-5}
4th (96–127)4842,5240.500.4264/865.1×10−65.1 \times10^{-6}
All94610,9920.500.2585/943.4×10−153.4 \times10^{-15}
(b) Recomputed per step (%)1st2nd3rd4thAll
Committed at the previous step100100100100100
Committed, all60.859.040.328.740.0
Still-masked20.05.1<0.1<0.1<0.1<0.110.2
(c) Steps after a full refresh0123456
Repeat positions (%)4.96.67.58.69.79.49.1
Steps 4–6 against 0: 83/94 images, p=1.2×10−13p=1.2 \times10^{-13}

Table 3. Staleness, refresh and repetition on 94 images. Δ>τ\Delta>\tau: share decoded beyond τ=3\tau=3. Images: where repeat exceeds normal. pp: Wilcoxon signed-rank test over images.

How the cross-step flow loses timeliness. dLLM-Cache recomputes the whole suffix every seventh step and, in between, only the quarter whose features changed most. Committing a position replaces its mask token with a word, so it is always recomputed at the next step, whereas a still-masked position’s input is unchanged. The selection thus favors committed positions (40% against 10% per step) and, in the second half of decoding, reaches a still-masked one in under 0.1% of the steps (Table III(b)). The KV states of the positions about to be committed thus grow stale until the next full refresh (Figure 6(b)–(Figure 6d)). Repetition follows this cycle independently of lateness, nearly doubling from the step of a full refresh to four steps later (4.9% against 9.7%, p=1.2×10−13p=1.2 \times10^{-13}, Table III(c)).

RepeatNormalImagespp
Deep-window entropy (bits)13.710.192/944.6×10−174.6 \times10^{-17}
Halved not before layer 31 (%)793689/942.0×10−152.0 \times10^{-15}
Top fifthRestImagespp
Repetition rate (%)21.24.666/941.5×10−71.5 \times10^{-7}
Share of all repetitions (%)5446

Table 4. TABLE IV: Consolidation and repetition on 94 images. Top fifth: highest context entropy. Images: where the first value exceeds the second. pp: Wilcoxon signed-rank test over images.

F2: Cache refresh is not staleness-aware. Observation. Cached KV states carry the information flow of the context tokens across steps. Repetition concentrates where those states are stale, beyond what lateness explains. Mechanism. Refresh by similarity favors committed positions, so the KV states of those about to be committed grow stale until the next full refresh. Evidence. Repeat positions concentrate in the stale tail in every quarter after the first, and repetition rises along the refresh cycle (Figure 6, Table III). Implication. The flow into a position must be refreshed before it is committed, which DAR enforces (§V-B).

Consolidation Across Layers: Failed Consolidation (F3)

Finally, we turn to the cross-layer consolidation axis: whether the information carried by the flow condenses, layer

by layer, into a confident prediction. We measure it by the entropy of the per-layer prediction distribution. Reading the layer-ℓ\ell hidden state of a token through the output head of the model (a logit-lens readout [[79]]) yields a distribution pv(ℓ)p_v^{(\ell)} over the vocabulary VV, whose entropy in bits is

E(ℓ)=−∑v∈Vpv(ℓ)log⁡2pv(ℓ).(8)E^{(\ell)}=-\sum_{v\in V}p_v^{(\ell)}\log_2p_v^{(\ell)}. \tag*{(8)}

Low deep-layer entropy indicates a consolidated prediction. Repeat positions remain unconsolidated. In a normal response, the entropy of a token stays on a plateau through the shallow and middle layers and converges over the deep window W={26,…,30}W=\{26,\ldots,30\}, as the model aggregates cross-layer information into a confident prediction [[79]] (Figure 2(d), blue). Under approximate caching, normal positions keep this profile (Figure 7(a)), whereas at repeat positions the entropy stays on the plateau throughout the deep window and collapses only in the last two layers (Figure 2(d), red). The contrast holds on the 94 images (Figure 7(a)): at the end of generation, 79% of the repeat positions have not shed half of their plateau entropy before layer 31, against 36% of the normal ones (p<10−14p < 10^{-14} across images, Table IV). A repeated token thus remains unconsolidated even after generation has finished.

Figure 7. Consolidation fails at repeat positions (Finding F3). (a) Entropy per layer (shaded: deep window). (b) Repetition rate by decile of the deep-window entropy of the committed neighbors within ±5. Intervals: 95%, over 94 images.

Failed consolidation predicts repetition. The failure shows in the local context before a repetition is committed. We score each position by its context entropy, the entropy of its committed neighbors within ±5\pm5 averaged over the deep window and read at its decode moment, and bin the positions by decile of this score (Figure 7(b)). The repetition rate stays at or below the overall rate (7.9%) in the first eight deciles and rises steeply in the last two. The fifth of positions with the highest context entropy repeats 4.6 times as often as the rest and holds more than half of all repetitions (p<10−6p < 10^{-6} across images, Table IV). An unconsolidated local context thus signals that a repetition is about to be committed.

F3: The information flow fails to consolidate at repeat positions. Observation. A normal token’s deep-layer entropy typically converges, whereas a repeated token’s stays high through the deep window. Mechanism. A repetition is committed from an unconsolidated context. Evidence. Most repeat positions keep over half their entropy through the deep window, and the top fifth by context entropy holds over half of all repetitions (Figure 7, Table IV). Implication. The context entropy should discount the confidence, which CTEV does in its voting score (§V-C).

Joint Diagnosis

The three findings describe one failure along the three axes of the information flow (Figure 3), each with its own object: routing in the attention (F1), timeliness in the KV states (F2) and consolidation in the deep-layer entropy (F3). They are not redundant: staleness accounts for most of the excess anchoring loss of repeat regions in the shallow band (81%) but for almost none in the deep band (5%), and failed consolidation persists at equal staleness. They also compound: tokens committed with dispersed routing, stale KV states and an unconsolidated context repeat at 55%, against 2.4% with none of them (see the supplementary material). Each disruption thus calls for an intervention on its own object, which Section V derives.

Method: The CoTA++ Framework

CoTA++ (Figure 8) is a training-free, plug-and-play framework that turns the three implications of Section IV into three components, one per disruption. Its design follows one principle: intervene where the disruption arises, on the object it corrupts. CTAR re-forms the routing of the within-step flow (F1, §V-A), DAR restores the timeliness of the cross-step flow (F2, §V-B), and CTEV brings the consolidation of the cross-layer flow into the order of commitment (F3, §V-C). Section V-D places them in one loop and examines their cost.

The CoTA++ framework

Figure 8. The CoTA++ framework. Top: a cached dMLLM reuses the feed-forward, attention and KV states of Step KK at Step K−1K-1 and unmasks the response in order of confidence. Badges mark where each component acts. Bottom: each component acts on its own object. CTAR re-anchors the attention of the positions its monitor selects, DAR refreshes the KV states of the positions about to commit, and CTEV discounts the confidence by the deep-layer entropy of the context.

Context-Token Attention Re-anchoring (CTAR)

For a position it does not recompute, the cache replays the attention output stored at an earlier step, whose routing predates every anchor committed since (§IV-B). The dispersion thus lies in the routing itself, so CTAR re-forms the routing against the current context. Let hi(ℓ)h_i^{(\ell)} denote the hidden state that suffix position ii carries into layer ℓ\ell at the current step, and K(ℓ)K^{(\ell)} and V(ℓ)V^{(\ell)} the keys and values the cache currently holds. CTAR recomputes the attention distribution of the position against these keys,

qi(ℓ)=WQ(ℓ)LN⁡(hi(ℓ)),Ai(ℓ)=softmax⁡(qi(ℓ)K(ℓ)T/dh),(9)q_i^{(\ell)} = W_Q^{(\ell)} \operatorname{LN}\left(h_i^{(\ell)}\right), \qquad A_i^{(\ell)} = \operatorname{softmax}\left(q_i^{(\ell)} K^{(\ell)\mathsf{T}} / \sqrt{d_h}\right), \tag*{(9)}

and reads its attention output through this distribution,

oi(ℓ)=WO(ℓ)(Ai(ℓ)V(ℓ)),(10)o_i^{(\ell)} = W_O^{(\ell)}\left(A_i^{(\ell)} V^{(\ell)}\right), \tag*{(10)}

where WQ(ℓ)W_Q^{(\ell)} and WO(ℓ)W_O^{(\ell)} are the query and output projections of the layer, LN its input normalization and dhd_h the dimension of one attention head. The result replaces the stored output of position ii, in the current pass and in the cache. The position is thus routed against the context as it now stands.

Re-anchoring every position at every step would recompute the attention of the whole suffix. The dispersion grows with the anchors a position has missed (Figure 5(d)), so CTAR re-anchors a position only once it has missed one, that is, once one of its context tokens has been committed (§IV-B). A context-anchor monitor reads this from the decoding schedule alone: for every masked position it counts the context tokens within ±w\pm w committed since its last re-anchoring, or since decoding began, and at each step CTAR evaluates Eqs. (9)–(10) for the set Tt\mathcal{T}^{t} of masked positions whose count has reached a threshold κ\kappa. With κ=1\kappa=1, a position is re-anchored as soon as one of its context tokens is committed. The count ignores the recomputations of the backend, which keeps the monitor independent of whichever refresh policy is in use.

CTAR acts in the deep band D={25,…,32}\mathcal{D}=\{25,\ldots,32\}, where the dispersion is most concentrated on repeat regions (Figure 5(b)). It adds no learned parameter and no operation of its own: the routing is re-formed by the attention of the model, while the keys, values and feed-forward outputs stay cached. Every masked position thus stays anchored to its context tokens in the deep band as they are committed (against F1).

Decode-Aware Refresh (DAR)

A similarity-based cache refreshes the positions whose features have changed most, and the positions still masked are rarely among them (Table III(b)). The positions about to be committed are thus decoded on stale KV states (§IV-C). Timeliness matters at the step of decision: a stale state is inconsequential at a position that stays masked and decisive at one about to be committed. DAR therefore makes the refresh aware of the decoding schedule and reserves part of its budget for the positions about to be committed, which the confidence ranking of the decoder already identifies.

Let It\mathcal{I}^{t} be the rr masked positions with the highest confidence c(i)c(i) of Eq. (2) at step t−1t-1. DAR refreshes them at step tt and leaves the rest of the refresh set to the cache. With bb the number of positions that the cache refreshes at a step and s(i)ts^{t}_{(i)} the similarity by which it selects them (§III-C), the set RtR^{t} of positions recomputed at step tt (Eq. (7)) becomes

Rt=It∪Bottom⁡-k({s(i)t:i∉It}),(11)R^{t}=\mathcal{I}^{t}\cup\operatorname{Bottom}\text{-}k\left(\{s^{t}_{(i)}:i\notin\mathcal{I}^{t}\}\right), \tag*{(11)}

with Bottom-kk the k=b−rk=b-r positions of lowest similarity. The refresh set keeps its size bb, so DAR spends no more than the cache and only redirects what it spends. It leaves the selection rule of the backend in place and has a single hyperparameter, rr. By Eq. (7), the reserved positions have staleness Δ=0\Delta=0 at step tt. The positions about to be committed are thus decoded on fresh KV states (against F2).

Context-Token Entropy-Guided Voting (CTEV)

A dMLLM commits a token on its confidence alone, which reports how peaked the prediction is, not whether the context behind it has consolidated. Under caching the two diverge where repetition arises (§IV-D). F3 located the signature of repetition in the context a position commits from, so CTEV brings the context entropy into the decision. For a masked position ii, it averages the entropy E(ℓ)E^{(\ell)} of Eq. (8) over the deep window W={26,…,30}\mathcal{W}=\{26,\ldots,30\} and over the context tokens already committed,

Eˉctx(i)=1∣N(i)∣∣W∣∑j∈N(i)∑ℓ∈WE(ℓ)(j),(12)\bar{E}_{\mathrm{ctx}}(i)=\frac{1}{|\mathcal{N}(i)||\mathcal{W}|}\sum_{j\in\mathcal{N}(i)}\sum_{\ell\in\mathcal{W}}E^{(\ell)}(j), \tag*{(12)}

where N(i)={j∉Tmaskt:0<∣j−i∣≤w}\mathcal{N}(i)=\{j\notin\mathcal{T}^{t}_{\mathrm{mask}}:0<|j-i|\le w\} collects the committed context tokens within ±w\pm w, and Eˉctx(i)=0\bar{E}_{\mathrm{ctx}}(i)=0 if no context token has been committed yet.

This is the quantity whose top fifth repeats 4.6 times as often as the rest in Figure 7(b). CTEV subtracts it from the confidence,

Score⁡(i)=c(i)−λEˉctx(i)log⁡2∣V∣,λ>0,(13)\operatorname{Score}(i)=c(i)-\lambda\frac{\bar{E}_{\mathrm{ctx}}(i)}{\log_{2}|\mathcal{V}|},\qquad\lambda>0, \tag*{(13)}

where dividing by log⁡2∣V∣\log_{2}|\mathcal{V}|, the largest possible entropy, puts the penalty on the confidence scale. The resulting voting score replaces c(i)c(i) in the top-kk selection of Eq. (3). Each committed context token thus votes through its deep-layer entropy, and a candidate in an unconsolidated context yields its turn to one of comparable confidence in a consolidated context.

The penalty is graded, so CTEV never forbids a token and only changes the order in which tokens are committed. It adds no learned parameter, reuses the readout of §IV-D and leaves the cache untouched. Positions whose context has converged are thus committed ahead of the rest (against F3).

Unified Algorithm and Complexity

Algorithm 1 places the three components in a single decoding loop. DAR acts at the step boundary, on the refresh set.

Image not extracted

Figure 1.

brings the context entropy into the decision. For a masked position ii, it averages the entropy E(ℓ)E^{(\ell)} of Eq. (8) over the deep window W={26,…,30}\mathcal{W}=\{26,\ldots,30\} and over the context tokens already committed,

Eˉctx(i)=1∣N(i)∣∣W∣∑j∈N(i)∑ℓ∈WE(ℓ)(j),(14)\bar{E}_{\mathrm{ctx}}(i)=\frac{1}{|\mathcal{N}(i)||\mathcal{W}|}\sum_{j\in\mathcal{N}(i)}\sum_{\ell\in\mathcal{W}}E^{(\ell)}(j), \tag*{(14)}

where N(i)={j∉Tmaskt:0<∣j−i∣≤w}\mathcal{N}(i)=\{j\notin\mathcal{T}^{t}_{\mathrm{mask}}:0<|j-i|\le w\} collects the committed context tokens within ±w\pm w, and Eˉctx(i)=0\bar{E}_{\mathrm{ctx}}(i)=0 if no context token has been committed yet.

This is the quantity whose top fifth repeats 4.6 times as often as the rest in Figure 7(b). CTEV subtracts it from the confidence,

Score⁡(i)=c(i)−λEˉctx(i)log⁡2∣V∣,λ>0,(15)\operatorname{Score}(i)=c(i)-\lambda\frac{\bar{E}_{\mathrm{ctx}}(i)}{\log_{2}|\mathcal{V}|},\qquad\lambda>0, \tag*{(15)}

where dividing by log⁡2∣V∣\log_{2}|\mathcal{V}|, the largest possible entropy, puts the penalty on the confidence scale. The resulting voting score replaces c(i)c(i) in the top-kk selection of Eq. (3). Each committed context token thus votes through its deep-layer entropy, and a candidate in an unconsolidated context yields its turn to one of comparable confidence in a consolidated context.

The penalty is graded, so CTEV never forbids a token and only changes the order in which tokens are committed. It adds no learned parameter, reuses the readout of §IV-D and leaves the cache untouched. Positions whose context has converged are thus committed ahead of the rest (against F3).

Unified Algorithm and Complexity

Algorithm 1 places the three components in a single decoding loop. DAR acts at the step boundary, on the refresh set.

CTAR acts inside the forward pass, on the attention output of the positions its monitor selects. CTEV acts once the pass is over, on the voting score. They share no state, so that any subset of them can be enabled, and the rest of the loop is left to the backend. The framework requires only what a cached dMLLM already provides, a confidence-ordered unmasking and a cache that exposes its states and its refresh set.

Their costs differ in kind (Table V). DAR adds a top-rr selection per step, and CTAR a row of attention with its two projections for each selected position in each layer of the band. CTEV adds the deep-window readout, which dominates since each of its ∣W∣|\mathcal{W}| projections spans the vocabulary.

ComponentIntervention levelExtra cost per step
CTARAttention routing, in-passO(∣D∣∣Tt∣(S+d))O(|\mathcal{D}||\mathcal{T}^{t}|(S+d))
DARRefresh set, at the step boundaryO(M)O(M) index ops
CTEVVoting score, after the passO(∣W∣Md∣V∣)O(|\mathcal{W}|Md|\mathcal{V}|)

Table V. TABLE V: Per-step cost of the three components. D\mathcal{D}: re-anchored layer band; Tt\mathcal{T}^{t}: positions selected by the monitor; W\mathcal{W}: deep window of the entropy readout; MM: suffix length; SS: sequence length; dd: hidden size; ∣V∣|\mathcal{V}|: vocabulary size.

Experiments

We evaluate CoTA++ on three dMLLMs, each under two caching backends (§VI-A). The experiments address three questions: whether CoTA++ mitigates the Repeat Curse across models, backends and response lengths without a cost in response quality (§VI-B); whether it preserves the general multimodal ability of the model, on short answers and on long ones (§VI-C); and what each component contributes, at what cost in latency and memory, and whether it acts on the axis where its finding located the disruption (§VI-D).

Experimental Setup

Models and caching backends. We evaluate three dMLLMs, LLaDA-V-8B [7], MMaDA-8B [8] and LaViDa-8B [9], each under two approximate caches, dLLM-Cache [18] and SlowFast [19]. LLaDA-V and LaViDa share the LLaDA-8B-Instruct language tower [2] and a two-layer MLP projector, over a siglip2-so400m-patch14-384 [36] and a siglip-so400m-patch14-384 vision tower respectively. MMaDA [8] is instead a single diffusion model over a joint vocabulary, where an image enters as discrete MAGVIT-v2 tokens. Unless stated otherwise, one token is committed per step, and the caches run at the released dLLM-Cache setting on LLaDA-V (Ep=25\mathcal{E}_p=25, Es=7\mathcal{E}_s=7, α=0.25\alpha=0.25). Each response is truncated at its first end-of-text token before any metric is computed.

Benchmarks. The Repeat Curse is measured on the descriptions of 500 MS-COCO val2014 images [21], whose list we release with our code, at response lengths of 64, 128 and 512 tokens. General multimodal ability is measured on nine benchmarks, DocVQA [80], ChartQA [81], MMStar [82], MME [83], SEED-Bench [84], MMBench [85], LLaVA-Bench (in-the-wild) [6], MathVista [86] and MathVerse [87], each evaluated under its released protocol.

Metrics. Repetition is measured by ARR, SRR, MRL, ARL and 95pRL (§IV-A). ARR and SRR are averaged over the responses, and the run lengths are pooled over all of them. Since adjacent runs miss repetition at the phrase level, we also report seq-rep-nn, the share of the nn-grams of a response that repeat an earlier one, and distinct-nn, its complement, both read against the uncached model. To verify that repetition is not removed at the cost of quality, every description is rated by a judge on a 1–10 scale for accuracy against the five human captions of the image, level of detail and coherence. The judge is gpt-5.6-luna. Its protocol, the statistical tests and the hardware are given in the supplementary material.

Hyperparameters. CTAR re-anchors the deep band D={25,…,32}\mathcal{D}=\{25,\ldots,32\} with κ=1\kappa=1 and w=5w=5, DAR reserves r=4r=4 positions, and CTEV uses λ=0.25\lambda=0.25 over the deep window W={26,…,30}\mathcal{W}=\{26,\ldots,30\}, with the same ww for its context tokens.

ModelMethodL=512L=512 Judge↑\uparrowL=512L=512 LenL=128L=128 Judge↑\uparrowL=128L=128 LenL=64L=64 Judge↑\uparrowL=64L=64 LenAvg. Judge↑\uparrow
LLaDA-VVanilla3.934754.771274.34634.35
dLLM-Cache3.035093.591274.17633.60
+ CoTA3.895104.731274.38634.33
+ CoTA++4.005104.771274.40634.39
SlowFast4.122544.34993.94564.13
+ CoTA4.291584.27864.24574.27
+ CoTA++4.302384.35974.18584.28
MMaDAVanilla1.881081.95981.93631.92
dLLM-Cache1.381061.52841.39541.43
+ CoTA1.841021.88921.88611.87
+ CoTA++1.841031.88951.94611.89
SlowFast1.411031.48861.44561.44
+ CoTA1.891031.87921.97611.91
+ CoTA++1.881041.88941.96621.91
LaViDaVanilla4.601704.761273.74464.36
dLLM-Cache3.071954.281262.51263.28
+ CoTA4.631624.781253.51394.31
+ CoTA++4.671624.781263.50404.32
SlowFast3.011224.171262.21153.13
+ CoTA3.80914.741253.38363.97
+ CoTA++3.85954.731253.49424.02

Table VII. TABLE VII: Response quality and length in the setting of Table VI. Judge is the mean score of the 500 descriptions on a 1–10 scale (§VI-A), Len the mean response length in tokens. Bold: best within a backend group. Shaded: CoTA++.

Repeat Curse Evaluation

The Repeat Curse extends to a third model and to long responses. Table VI completes §IV-A with LaViDa and with L=512L=512. Without a cache, no model repeats in more than 4.6% of its responses at any length. Under dLLM-Cache, all three models repeat in more than 97% of them at L=512L=512, and a single run reaches 465 tokens. On LLaDA-V and LaViDa, whose responses lengthen with the budget (Table VII), the SRR rises with LL, from 70.8% and 24.6% at L=64L=64. MMaDA, whose responses level off near 100 tokens, repeats at every length. On average over the lengths, SlowFast is markedly milder than dLLM-Cache on LLaDA-V alone.

The damage lies in a few runs that consume the response. Under dLLM-Cache the mean run is 2–6 tokens and 95% of the runs are no longer than 5, so most runs are as short as those that occur without a cache. The longest run, however, grows with the budget, from 30–47 tokens at L=64L=64 to 409–465 at L=512L=512, that is, 80–91% of the 512-token budget; under SlowFast it reaches 232–428 tokens at L=512L=512. The Repeat Curse is not a matter of a few extra pairs but of responses that break into a single run.

CoTA++ mitigates the repetition across models, backends and lengths. Averaged over the three lengths and both backends, CoTA++ reduces the SRR of the cached model by 95% on LLaDA-V, 88% on LaViDa and 99% on MMaDA, and its ARR by 99.5%, 97.7% and 99.9%. No run longer than 3 tokens remains in any setting. On average, it also repeats less than CoTA in every model-backend pair, and on LLaDA-V the margin widens as the responses lengthen: under dLLM-Cache,

CoTA leaves an SRR of 2.4% at L=64L=64 and 47.4% at L=512L=512, where CoTA++ leaves 0.4% and 8.2%, and under SlowFast CoTA still leaves a run of 286 tokens at L=512L=512. The repetition is not displaced to longer spans. A remedy that acts on adjacent repeats could push the repetition into phrases. Table VIII shows that it does not: under dLLM-Cache at L=512L=512, the share of repeated 4-grams rises from 24.6%, 2.2% and 2.0% without a cache to 37.9%, 7.7% and 14.5% on LLaDA-V, MMaDA and LaViDa, and CoTA++ brings it back to 20.5%, 2.2% and 1.9%, at or below the uncached value. Distinct-2 returns the same way.

MethodLLaDA-V Rep-4↓\downarrowLLaDA-V Dist-2↑\uparrowMMaDA Rep-4↓\downarrowMMaDA Dist-2↑\uparrowLaViDa Rep-4↓\downarrowLaViDa Dist-2↑\uparrow
Vanilla24.658.22.289.32.086.8
dLLM-Cache37.945.77.778.514.573.6
+ CoTA21.957.72.488.22.086.6
+ CoTA++20.559.12.289.21.986.9

Table VIII. TABLE VIII: Phrase-level repetition at L=512L=512 under dLLM-Cache: repeated 4-grams (Rep-4) and distinct 2-grams (Dist-2) in percent (§VI-A). Bold: best among the cached rows. Shaded: CoTA++.

The repetition is mitigated at no cost in quality. Averaged over the three lengths, the cache lowers the judge score by 0.22 to 1.23 points (Table VII). CoTA++ raises it in all six model-backend pairs, to within 0.34 of the uncached model, and to within 0.04 in four of them. On the same average, it scores at least as high as CoTA in every pair. The gains are not bought with shorter responses: on LLaDA-V under SlowFast, the one pair where the cache halves the response at L=512L=512, CoTA++ answers at the length of the SlowFast baseline, 238 against 254 tokens, so its lower SRR is not an effect of length.

Case study. Fig. 9 shows two responses of LLaDA-V under dLLM-Cache. In (a), at L=128L=128, the cached model breaks into a run of 13 identical tokens, and CoTA++ returns a response free of repetition. In (b), at L=512L=512, the cache breaks into a run of 150 copies of the, and CoTA++ recovers the description except for a doubled period at a sentence boundary. This is the form of the curse that survives: where CoTA++ still repeats, no run is longer than 3 tokens (Table VI), and the repeated tokens are punctuation, an article or correct text, such as a trash can can be seen or the digits of a number in the image.

Performance on General Multimodal Benchmarks

Short answers are left as they are. MMStar, MME, SEED-Bench and MMBench ask for an option letter or a single word, which leaves no room for repetition. On these benchmarks CoTA++ stays within 0.4 points of the cached model, or 14 points on the scale of MME, on all three models and under both backends (Table IX). An intervention aimed at long responses thus costs nothing where there is nothing to repair.

Longer answers are largely recovered. The cache costs most where the answer is long. Under dLLM-Cache (Table IX(a)), LLaDA-V loses 13.4 points on DocVQA and 21.5 on LaVA-Bench, and CoTA++ returns both to within 1.8 points of the uncached model. On MathVerse, CoTA++ recovers 2.2 of the 3.1 points lost on LLaDA-V and 3.8 of the 6.3 lost on LaViDa.

ModelDocChartMMSMMEP^{P}MMEC^{C}SEEDMMBLLaVAW^{W}MVistaMVerse
LLaDA-V [7]83.978.360.2150849274.883.051.760.429.2
+ dLLM-Cache70.578.260.0150349774.883.030.255.926.1
+ CoTA [1]81.678.660.0150549974.883.250.057.027.5
+ CoTA++82.178.660.0150549974.883.250.457.228.3
MMaDA [8]—10.137.3108728556.642.629.734.218.8
+ dLLM-Cache—9.037.1108227456.742.133.234.520.4
+ CoTA [1]—9.637.1108627756.742.533.934.520.8
+ CoTA++—9.637.1108627756.742.535.035.321.4
LaViDa [9]—67.345.9134636866.471.750.544.922.1
+ dLLM-Cache—64.945.4133933466.271.445.844.615.8
+ CoTA [1]—66.445.5134533866.171.645.945.218.8
+ CoTA++—66.445.5134533866.171.647.145.619.6

Table IX. General multimodal benchmarks under (a) dLLM-Cache and (b) SlowFast: DocVQA, ChartQA, MMStar, MME (perception, cognition), SEED, MMBench, LLaVA-Bench (in-the-wild), MathVista and MathVerse. MMaDA uses the MMaDA-8B-MixCoT checkpoint, the only one with a released evaluation protocol.

Figure 9. Case study, LLaDA-V under dLLM-Cache: the uncached model, the cached model, and the cached model with CoTA++. (a) The typical case, L=128L=128. (b) A residual failure, L=512L=512: a doubled period at a sentence boundary.

Recovery is partial on MathVista, where CoTA++ returns 1.3 of the 4.5 points LLaDA-V loses, and on LLaVA-Bench for LaViDa, 1.3 of 4.7. On MMaDA the cache costs at most 1.1 points on these benchmarks, and CoTA++ scores at or above the cached model on each. Under SlowFast (Table IX(b)) the pattern is the same: on LLaDA-V, LLaVA-Bench returns from 37.6 to 53.5, the same uncached 51.7, and DocVQA to within 3.1 points; LaViDa recovers 2.9 of the 6.3 points on ChartQA, 5.4 of 9.1 on LLaVA-Bench and 3.8 of 7.0 on MathVerse; MMaDA scores at or above the backend on every benchmark.

and CTEV the weakest: at L=128L=128 they leave a repeat in 4.4% and 59.6% of the responses, CTAR in 20.6%, against 96.8% under the cache. DAR and CTAR lower ARP by more than an order of magnitude at every length, CTEV by a factor of two to nine. None of them alone removes the tail at L=512L=512, where a run of 165, 160 and 420 tokens survives.

Component-wise Analysis of CoTA++

Ablation Study: Each component alone mitigates the curse.

Table X evaluates every subset of the three components on LLaDA-V under dLLM-Cache. Alone, DAR is the strongest

Pairs remove the tail, and the full method is the most consistent across lengths. Any two components remove the tail at L=512L=512, where no pair leaves a run longer than 3 tokens, although DAR with CTEV still leaves one of 25 at L=128L=128. The pairs differ in how many responses still repeat: DAR with CTAR leaves 9.6% at L=512L=512, CTAR with CTEV, the conference method, 47.4%. The full method is the strongest configuration at L=128L=128 and L=512L=512 and matches the best pair at L=64L=64, DAR with CTEV (0.4%): it leaves no run longer than

L=64L=64L=128L=128L=512L=512
DARCTARCTEVARRSRRMRLARRSRRMRLARRSRRMRL
1.6170.8387.0196.87314.7099.6455
✓0.021.420.044.420.1011.2165
✓0.138.020.2120.630.3162.0160
✓0.7735.6340.8659.6561.7294.0420
✓✓0.020.620.023.020.059.63
✓✓0.090.420.116.6250.0312.23
✓✓0.042.420.1011.420.1747.43
✓✓✓0.010.420.022.420.038.23
Vanilla0.010.420.000.220.012.43

Table X. Component grid (LLaDA-V with dLLM-Cache, 500 images): every subset of the three components at three response lengths. Shaded: CoTA++.

3 tokens and an SRR of 0.4, 2.4 and 8.2%, against 0.6, 3.0 and 9.6% for DAR with CTAR; the uncached model repeats in 0.4, 0.2 and 2.4%.

  1. Efficiency: The components run at the speed of the backend they correct. Table XI times each configuration alone on one A100 at L=64L=64. dLLM-Cache decodes 2.4 times faster than the uncached model. On top of it, DAR is free (1.01×), since it changes which positions the backend recomputes and not how many. CTAR costs 6% for the rows its monitor re-forms in the eight deep layers, and CTEV 7% for five logit-lens projections per step over the committed positions. Together they bring CoTA++ to 1.15× the latency of the cache, with 0.9 GiB more memory, which is still 2.1 times faster than decoding without a cache.

ConfigurationLatency (s/img)↓Throughput (tok/s)↑Relative latencyMemory (GiB)
Vanilla21.42.9–30.6
dLLM-Cache8.87.11.00×32.5
+ DAR9.07.01.01×32.5
+ CTAR9.46.71.06×32.5
+ CTEV9.46.71.07×33.4
+ CoTA++10.26.21.15×33.4

Table XI. Efficiency on LLaDA-V at L=64L=64, 20 images, each configuration alone on one A100: latency in seconds per image, throughput in decoded tokens per second, latency relative to dLLM-Cache, and peak GPU memory.

  1. CTAR: Routing at the Decode Moment: F1 located the Repeat Curse in the routing of the within-step flow: under the cache, a position about to commit anchors less on its context tokens. Figure 10 shows this at the decode moment. Without CTAR, the diagonal band of anchoring on the neighbors fades at the positions that repeat, and their attention spreads into vertical streaks on a few distant positions; with CTAR, the routing is re-formed against the current keys and the band returns as sharp as in the uncached model. The same loss and recovery appear on both models, so the dispersion belongs to cached decoding rather than to a model, and re-forming the routing restores the anchoring that the cache weakened.

Figure 10. Attention at the decode moment, with and without CTAR, layers 25–32; triangles: repeat positions. (a)–(b) LLaDA-V and (c)–(d) LaViDa under dLLM-Cache, L=128L=128.

  1. DAR: Timeliness of the KV State: Figure 11 reads the staleness of the KV state at the decode moment. Under the cache it follows the refresh interval, rising to six steps between refreshes; with DAR it is zero at 99% of the decode moments.

Staleness of the KV state at the decode moment with and without DAR

Figure 11. Staleness of the KV state at the decode moment, with and without DAR. Mean over 100 images at each decoding step, dLLM-Cache, L=128L=128.

  1. CTEV: Consolidation of the Context: F3 located the Repeat Curse in the consolidation of the cross-layer flow: under the cache, repetitions arise in unconsolidated contexts. Figure 12 shows this at the decode moment. The repeat rate of the committed position stays low over most of the range of its context entropy and climbs steeply at the top of the range; at the decisions CTEV changes, the position that confidence alone would commit lies in that region, and the position CTEV commits instead lies where the repeat rate is low. CTEV thus moves the commitment out of the unconsolidated contexts in which F3 found the repetitions.

Figure 12. Context entropy at the decode moment, with and without CTEV, dLLM-Cache, L=128L=128. Top: repeat rate of the committed position. Bottom: the decisions CTEV changes.

Discussion

What Repeats

Figure 13 collects the tokens that open a repeated run over the cached rows of Table VI. Function words open 54% of the runs, punctuation 33% and content words 14%; what repeats is the token whose identity is fixed by its context, the first to go wrong when the anchoring on that context weakens (F1).

Figure 13. Tokens that open a repeated run under the cache, sized by run count; a leading hyphen marks a word piece.

ConfigurationARR↓SRR↓MRL↓Rep-4↓
Vanilla0.000.000.037
dLLM-Cache7.5296.0730.083
CTAR: re-weight the stored routing, or re-form it
Additive distance bias6.4198.0730.073
Gated gain9.0799.0820.086
Re-formed, layers 25–32 (CTAR)0.3221.030.035
DAR: how many slots rr, and what fills six slots
r=2r=20.3936.020.022
r=4r=4 (DAR)0.044.020.026
r=16r=160.021.020.030
Dispersed anchoring (F1)7.6698.0720.083
Unconsolidated entropy (F3)1.3157.0770.038
Confidence (DAR)0.033.020.025
CTEV: whose entropy, and how strongly
Own entropy, λ=0.25\lambda=0.255.1498.0470.047
Context entropy, veto7.6696.0730.085
Context entropy, λ=0.25\lambda=0.25 (CTEV)0.7161.030.031
Context entropy, λ=0.5\lambda=0.50.3636.020.044
Context entropy, layers 1–103.3079.040.039

Table 12. Design decisions of the three components, LLaDA-V under dLLM-Cache, L=128L=128, 100 images, each component alone.

Design Alternatives

CTAR: re-forming the routing, not re-weighting it. In Table XII, an additive distance bias leaves the curse in place and a gated gain raises ARR, whereas re-forming the routing against the current keys lowers ARR by more than an order of magnitude and removes the tail of long runs. The deep band follows F1; shallower bands trade a higher phrase-level statistic, and the full stack four times the rows, for a few more repeat-free responses (see the supplementary material).

DAR: how many slots, and chosen by what. Two reserved slots leave a third of the responses with a repeat, four bring SRR to 4%, and sixteen lower it further while the phrase-level statistic begins to rise (Table XII). The slots are worth reserving only for the positions about to commit: filled by the positions of most dispersed anchoring (F1), the cache is left

as it was; by the least consolidated (F3), more than half of the responses still repeat; by confidence, 3% do.

CTEV: whose entropy, and how strongly. Penalizing a candidate by its own entropy leaves the curse in place, and a veto on the context entropy reproduces the cache (Table XII). The graded penalty mitigates the curse, and λ\lambda trades the two statistics: 0.5 lowers the share of responses with a repeat from 61% to 36% but raises the phrase-level statistic above the uncached level. Read from the shallow layers instead of the deep window, the same penalty leaves ARR above 3.

Fixed budgets and adaptive budgets. A single token-level risk score could instead merge the three signals and allocate one compute budget adaptively. The three components already act per position, each steered by a signal of its own disruption, the monitor count of CTAR, the confidence rank of DAR and the context entropy of CTEV, and only the amount spent per step, set by rr, τ\tau and λ\lambda, is fixed. Such a score presumes that the signals are interchangeable, which Table XII rules out: DAR slots filled by the F1 or F3 signal leave the curse largely in place, and a veto on the F3 signal reproduces the cache, because the disruptions act on different objects. A risk score could still set how much of each budget to spend at a step, for instance more reserved slots when more positions fall in the top fifth by context entropy, which we leave to future work.

Decoding-Time Alternatives and Settings

A decoding-time penalty caps the runs and leaves the curse. Applying the no-repeat nn-gram rule of autoregressive decoding [88] to LLaDA-V under dLLM-Cache, which blocks any token that would complete a 2-gram or 3-gram already in the response, caps the longest run at a few tokens, yet nearly all responses still carry a repeat and the judge score does not improve, whereas CoTA++ leaves a repeat in a few percent of them at a higher score (see the supplementary material).

Block-wise decoding leaves the picture unchanged. In blocks of 16 to 128 positions, the cache gets worse the shorter the block, and CoTA++ returns every setting close to the uncached model (see the supplementary material).

Conclusion

This work traces the Repeat Curse of cached dMLLMs to three disruptions of the context-token information flow: position about to commit anchors less on its context tokens (F1), is decoded on a stale KV state (F2), and commits from an unconsolidated context (F3). CoTA++ counteracts each where it originates, without training, and across three dMLLMs, two caching backends and three response lengths it mitigates the curse at no cost in quality and keeps the speedup of caching. The broader lesson is that an approximate cache must preserve the information flow it accelerates, not only the freshness of its states. A refresh schedule chosen by feature similarity keeps the states fresh on average, yet lets the flow break at the positions about to be committed, and the repair has to follow the flow: within the step, across steps and across layers.

Open-weight dMLLMs and caching methods designed for them are still few, and our compute is limited, so CoTA++ has not yet been evaluated on further caching methods or larger dMLLMs. These are the first steps of future work. Broader ones are to allocate the three budgets adaptively from their signals (§VII-B) and to let the three findings inform how dMLLMs are trained and how their caches are designed.

Acknowledgment

Generative AI tools, including Grammarly and Claude, were used solely for grammar checking and language polishing. The originality of this work was not influenced by generative AI. All core designs, data analysis, experimental findings, and technical contributions were independently developed by the human authors.

References

  1. [1]Q. Zhao, X. Zhang, S. Chang, Q. Chen, X. Chen, L. Liu, J. Zhang, X.-Y. Zhang, and D.-H. Wang, “Context tokens are anchors: Understanding the repeat curse in dMLLMs from an information flow perspective,” in The Fourteenth International Conference on Learning Representations, 2026.arxiv.org/abs/2610.23290
  2. [2]S. Nie, F. Zhu, X. You, X. Zhang, J. Ou, J. Hu, J. ZHOU, Y. Lin, J.-R. Wen, and C. Li, “Large language diffusion models,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.arxiv.org/abs/2502.09992
  3. [3]F. Zhu, R. Wang, S. Nie, X. Zhang, C. Wu, J. Hu, J. Zhou, J. Chen, Y. Lin, J.-R. Wen, and C. Li, “Llada 1.5: Variance-reduced preference optimization for large language diffusion models,” 2025.
  4. [4]J. Ye, Z. Xie, L. Zheng, J. Gao, Z. Wu, X. Jiang, Z. Li, and L. Kong, “Dream 7b,” 2025. [Online]. Available: https:////khnulp.github.io/blog/2025/dreamarxiv.org/abs/2508.15487
  5. [5]T. Bie, M. Cao, K. Chen, L. Du, M. Gong, Z. Gong, Y. Gu, J. Hu, Z. Huang, C. Lan et al., “Llada 2.0: Scaling up diffusion language models to 100b,” arXiv preprint arXiv:2512.15745, 2025.
  6. [6]H. Liu, C. Li, Y. Li, and Y. J. Lee, “Improved baselines with visual instruction tuning,” 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26 286–26 296, 2023.arxiv.org/abs/2310.03744
  7. [7]Z. You, S. Nie, X. Zhang, J. Hu, J. Zhou, Z. Lu, J.-R. Wen, and C. Li, “Llada-v: Large language diffusion models with visual instruction tuning,” arXiv preprint arXiv:2505.16933, 2025.arxiv.org/abs/2505.16933
  8. [8]L. Yang, Y. Tian, B. Li, X. Zhang, K. Shen, Y. Tong, and M. Wang, “MMaDA: Multimodal large diffusion language models,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.arxiv.org/abs/2505.15809
  9. [9]S. Li, K. Kallidromitis, H. Bansal, A. Gokul, Y. Kato, K. Kozuka, J. Kuen, Z. Lin, K.-W. Chang, and A. Grover, “Lavida: A large diffusion model for vision-language understanding,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
  10. [10]S. Li, J. Gu, K. Liu, Z. Lin, Z. Wei, A. Grover, and J. Kuen, “Lavida-0: Elastic large masked diffusion models for unified multimodal understanding and generation,” in International Conference on Learning Representations, vol. 2026, 2026, pp. 50 571–50 603.
  11. [11]L. Zeng, J. Yao, B. Liao, H. Tao, W. Liu, and X. Wang, “Diffusionvl: Translating any autoregressive models into diffusion vision language models,” arXiv preprint arXiv:2512.15713, 2025.arxiv.org/abs/2512.15713
  12. [12]I. Ai, T. Bie, H. Chen, T. Chen, Z. Cheng, L. Cui, K. Gan, Z. Huang, Z. Lan, H. Li et al., “Llada2. 0-uni: Unifying multimodal understanding and generation with diffusion large language model,” arXiv preprint arXiv:2604.20796, 2026.
  13. [13]Y. Xin, Q. Qin, S. Luo, K. Zhu, J. Yan, Y. Tai, J. Lei, Y. Cao, K. Wang, Y. Wang et al., “Lumina-dimoo: An omni diffusion large language model for multi-modal generation and understanding,” arXiv preprint arXiv:2510.06308, 2025.arxiv.org/abs/2510.06308
  14. [14]A. Grattafiori, A. Dubey, A. Jauhari, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024.arxiv.org/abs/2407.21783
  15. [15]A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li et al., “Qwen2.5 technical report,” CoRR, vol. abs/2412.15115, 2024.arxiv.org/abs/2412.15115
  16. [16]R. Xu, J. Xu, Q. Zhao, and X. Zhang, “Fixing semantic blind spots in anchor tokens of dmllms,” in Findings of the Association for Computational Linguistics: ACL 2026, 2026, pp. 19 359–19 374.DOI
  17. [17]S. Chang, X. Zhang, Q. Liu, and L. Niu, “D3Tom: Decider-guided dynamic token merging for accelerating diffusion mlms,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 24, 2026, pp. 19 961–19 969.
  18. [18]Z. Liu, Y. Yang, Y. Zhang, J. Chen, C. Zou, Q. Wei, S. Wang, Y. Zhu, and L. Zhang, “dLLM-cache: Accelerating diffusion large language models with adaptive caching,” in Forty-third International Conference on Machine Learning, 2026.arxiv.org/abs/2506.06295
  19. [19]Q. Wei, Y. Zhang, Z. Liu, P. Zeng, Y. Wang, B. Qi, D. Liu, and L. Zhang, “Accelerating diffusion large language models with slowfast sampling: The three golden principles,” in International Conference on Learning Representations, vol. 2026, 2026, pp. 4340–4354.arxiv.org/abs/2506.10848
  20. [20]C. Wu, H. Zhang, S. Xue, Z. Liu, S. Diao, L. Zhu, P. Luo, S. Han, and E. Xie, “Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding,” in International Conference on Learning Representations, vol. 2026, 2026, pp. 57 027–57 051.arxiv.org/abs/2505.22618
  21. [21]T.-Y. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European Conference on Computer Vision, 2014.arxiv.org/abs/1405.0312
  22. [22]Z. Yu, Z. Wang, Y. Fu, H. Shi, K. Shaikh, and Y. C. Lin, “Unveiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration,” in Forty-first International Conference on Machine Learning, 2024.arxiv.org/abs/2406.15765
  23. [23]L. Wang, L. Li, D. Dai, D. Chen, H. Zhou, F. Meng, J. Zhou, and X. Sun, “Label words are anchors: An information flow perspective for understanding in-context learning,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 9840–9855.arxiv.org/abs/2305.14160
  24. [24]L. Chen, H. Zhao, T. Liu, S. Bai, J. Lin, C. Zhou, and B. Chang, “An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models,” in European Conference on Computer Vision, 2024.arxiv.org/abs/2403.06764
  25. [25]I. Yona, I. Shumailov, J. Hayes, and Y. Gandelsman, “Interpreting the repeated token phenomenon in large language models,” in Forty-second International Conference on Machine Learning, 2025.arxiv.org/abs/2503.08908
  26. [26]X. Zhang, Y. Quan, C. Shen, C. Gu, X. Yuan, S. Yan, J. Cao, H. Cheng, K. Wu, and J. Ye, “Shallow focus, deep fixes: Enhancing shallow layers vision attention sinks to alleviate hallucination in lvlms,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025, pp. 3512–3534.
  27. [27]D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach, “Sdxl: Improving latent diffusion models for high-resolution image synthesis,” in International Conference on Learning Representations, vol. 2024, 2024, pp. 1862–1874.
  28. [28]P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel et al., “Scaling rectified flow transformers for high-resolution image synthesis,” in Forty-first international conference on machine learning, 2024.arxiv.org/abs/2403.03206
  29. [29]R. ZHANG, S. Zhai, Y. Zhang, J. Thornton, Z. Ou, J. M. Susskind, and N. Jaitly, “Target concrete score matching: A holistic framework for discrete diffusion,” in Forty-second International Conference on Machine Learning, 2025.arxiv.org/abs/2504.16431
  30. [30]S. Gong, M. Li, J. Feng, Z. Wu, and L. Kong, “Diffuseq: Sequence to sequence text generation with diffusion models,” in The Eleventh International Conference on Learning Representations, 2023.arxiv.org/abs/2210.08933
  31. [31]K. Zheng, Y. Chen, H. Chen, M.-Y. Liu, J. Zhu, and Q. Zhang, “Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling,” in International Conference on Learning Representations, vol. 2025, 2025, pp. 63 186–63 227.arxiv.org/abs/2409.02908
  32. [32]J. Ou, S. Nie, K. Xue, F. Zhu, J. Sun, Z. Li, and C. Li, “Your absorbing discrete diffusion secretly models the conditional distributions of clean data,” in International Conference on Learning Representations, vol. 2025, 2025, pp. 64972–65009.arxiv.org/abs/2406.03736
  33. [33]S. Nie, F. Zhu, C. Du, T. Pang, Q. Liu, G. Zeng, M. Lin, and C. Li, “Scaling up masked diffusion models on text,” in International Conference on Learning Representations, vol. 2025, 2025, pp. 82974–82997.arxiv.org/abs/2410.18514
  34. [34]Z. Xie, J. Ye, L. Zheng, J. Gao, J. Dong, Z. Wu, X. Zhao, S. Gong, X. Jiang, Z. Li et al., “Dream-coder 7b: An open diffusion language model for code,” arXiv preprint arXiv:2509.01142, 2025.arxiv.org/abs/2509.01142
  35. [35]F. Zhu, Z. You, Y. Xing, Z. Huang, L. Liu, Y. Zhuang, G. Lu, K. Wang, X. Wang, L. Wei et al., “Llada-moe: A sparse moe diffusion language model,” arXiv preprint arXiv:2509.24389, 2025.arxiv.org/abs/2509.24389
  36. [36]M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Bever, Y. Xia, B. Mustafa et al., “Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features,” arXiv preprint arXiv:2502.14786, 2025.arxiv.org/abs/2502.14786
  37. [37]S. Bai, K. Chen, X. Liu, J. Wang et al., “Qwen2.5-vl technical report,” 2025.arxiv.org/abs/2502.13923
  38. [38]X. Zhang, Y. Zhu, C. Gu, X. Yuan, Q. Zhao, J. Cao, F. Tang, S. Fan, Y. Shen, C. Shen et al., “Hallucination begins where saliency drops,” arXiv preprint arXiv:2601.20279, 2026.arxiv.org/abs/2601.20279
  39. [39]Q. Zhao, X. Zhang, B. Liu, M. Chen, W. Xiong, J. Chen, G. Ye, W. Yu, X. Yuan, S. Han et al., “Locational-aware skill reliability for self-evolving agents,” arXiv preprint arXiv:2608.22610, 2026.arxiv.org/abs/2608.22610
  40. [40]Y. Zhao, Q. Zhao, J. Zhang, X. Zhang, X. Yuan, and P. Cheng, “Context-aware cluster decoding: Semantic anchor-driven coherence in dllms,” arXiv preprint arXiv:2608.22367, 2026.arxiv.org/abs/2608.22367
  41. [41]X. Zhang, Y. Zhu, Q. Zhao, X. Yuan, J. Cao, and X. Chen, “Token-Penalty: Alleviating attention sinks and positional decay in LVLMs,” in Findings of the Association for Computational Linguistics: ACL 2026. San Diego, California, United States: Association for Computational Linguistics, Jul. 2026, pp. 12 894–12 906.DOI
  42. [42]R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean, “Efficiently scaling transformer inference,” in Proceedings of Machine Learning and Systems, vol. 5, 2023, pp. 606–624.arxiv.org/abs/2211.05102
  43. [43]J. Huang, Y. Zhang, Y. Yang, B. Huang, and L. Zhang, “Mask tokens as prophet: Fine-grained cache eviction for efficient dllm inference,” in Findings of the Association for Computational Linguistics: ACL 2026, 2026, pp. 3456–3479.arxiv.org/abs/2510.09309
  44. [44]H. Hu, J. Meng, Y. Akhauri, M. Abdelfattah, J.-s. Seo, Z. Zhang, and U. Gupta, “Flashdlm: Accelerating diffusion language model inference via efficient kv caching and guided diffusion,” in International Conference on Learning Representations, vol. 2026, 2026, pp. 153 529–153 544.arxiv.org/abs/2505.21467
  45. [45]Y. Song, Z. Zhang, C. Luo, P. Gao, F. Xia, H. Luo, Z. Li, Y. Yang, H. Yu, X. Qu et al., “Seed diffusion: A large-scale diffusion language model with high-speed inference,” arXiv preprint arXiv:2508.02193, 2025.arxiv.org/abs/2508.02193
  46. [46]W. Wang, B. Fang, C. Jing, Y. Shen, Y. Shen, Q. Wang, H. Ouyang, H. Chen, and C. Shen, “Time is a feature: Exploiting temporal dynamics in diffusion language models,” in International Conference on Learning Representations, vol. 2026, 2026, pp. 72 909–72 935.
  47. [47]X. Ma, R. Yu, G. Fang, and X. Wang, “Dkv-cache: The cache for diffusion language models,” Advances in Neural Information Processing Systems, vol. 38, pp. 149 009–149 033, 2026.arxiv.org/abs/2505.15781
  48. [48]S. Abnar and W. Zuidema, “Quantifying attention flow in transformers,” in Proceedings of the 58th annual meeting of the association for computational linguistics, 2020, pp. 4190–4197.arxiv.org/abs/2005.00928
  49. [49]J. Ferrando and E. Voita, “Information flow routes: Automatically interpreting language models at scale,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 17 432–17 445.arxiv.org/abs/2403.00824
  50. [50]Z. Zhang, S. Yadav, F. Han, and E. Shutova, “Cross-modal information flow in multimodal large language models,” in 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025, pp. 19 781–19 791.arxiv.org/abs/2411.18620
  51. [51]Z. Qiu, Z. Wang, B. Zheng, Z. Huang, K. Wen, S. Yang, R. Men, L. Yu, F. Huang, S. Huang et al., “Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free,” Advances in Neural Information Processing Systems, vol. 38, pp. 100 092–100 118, 2026.
  52. [52]Y. Song, X. Liu, R. Li, Z. Liu, Z. Huang, Q. Guo, Z. He, and X. Qiu, “Sparse-dllm: Accelerating diffusion llms with dynamic cache eviction,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 39, 2026, pp. 33 038–33 046.arxiv.org/abs/2508.02558
  53. [53]X. Chen, S. Huang, C. Guo, C. Wei, Y. He, J. Zhang, H. Li, and Y. Chen, “Dpad: Efficient diffusion language models with suffix dropout,” in International Conference on Learning Representations, vol. 2026, 2026, pp. 57 286–57 316.arxiv.org/abs/2508.14148
  54. [54]J. Yao, S. Yang, J. Xu, L. Hu, M. Li, and D. Wang, “Understanding the repeat curse in large language models from a feature perspective,” in Findings of the Association for Computational Linguistics: ACL 2025, 2025, pp. 7787–7815.DOI
  55. [55]D. Nguyen, A. Pavani, and B. Mirzasoleiman, “Beyond semantic entropy: Boosting llm uncertainty quantification with pairwise semantic similarity,” in Findings of the Association for Computational Linguistics: ACL 2025, 2025, pp. 4530–4540.arxiv.org/abs/2506.00245
  56. [56]Y. Zhang, C. Wang, Y. Chen, C. Fan, J. Jia, and S. Liu, “The fragile truth of saliency: Improving llm input attribution via attention bias optimization,” Advances in Neural Information Processing Systems, vol. 38, pp. 154 395–154 421, 2026.DOI
  57. [57]X. Zhang, Y. Quan, C. Shen, X. Yuan, S. Yan, L. Xie, W. Wang, C. Gu, H. Tang, and J. Ye, “From redundancy to relevance: Information flow in lvms across reasoning tasks,” in Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2025, pp. 2289–2299.
  58. [58]Y. Li, H. Wang, X. Ding, H. Wang, and X. Li, “Token activation map to visually explain multimodal llms,” in 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025, pp. 48–58.arxiv.org/abs/2506.23270
  59. [59]C. Gan, Y. Tu, X. Chen, T. Chen, Y. Li, M. Harandi, and W. Lin, “Unleashing diffusion transformers for visual correspondence by modulating massive activations,” Advances in Neural Information Processing Systems, vol. 38, pp. 114 432–114 462, 2026.arxiv.org/abs/2505.18584
  60. [60]C. Gan, Z. Zhao, Y. Tu, X. Chen, Z. Qin, T. Chen, M. Harandi, and W. Lin, “Massive activations are the key to local detail synthesis in diffusion transformers,” in International Conference on Learning Representations, vol. 2026, 2026, pp. 104 855–104 883.arxiv.org/abs/2510.11538
  61. [61]Q. Huang, X. Wen Dong, P. Zhang, B. Wang, C. He, J. Wang, D. Lin, W. Zhang, and N. H. Yu, “Opera: Alleviating hallucination in multimodal large language models via over-trust penalty and retrospection-allocation,” 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13 418–13 427, 2023.DOI
  62. [62]Q. Zhao, X. Zhang, Y. Li, X. Ying, X. Yuan, F. Tang, S. Fan, X. Chen, D.-H. Wang, and X.-Y. Zhang, “Mca-llava: Manhattan causal attention for reducing hallucination in large vision-language models,” in Proceedings of the 33rd ACM International Conference on Multimedia, 2025, pp. 3981–3990.arxiv.org/abs/2507.09184
  63. [63]C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen et al., “In-context learning and induction heads,” arXiv preprint arXiv:2209.11895, 2022.arxiv.org/abs/2209.11895
  64. [64]M. E. Rulli, S. Petruzzi, E. Michielon, F. Silvestri, S. Scardapane, and A. Devoto, “Attention sinks in diffusion language models,” in Findings of the Association for Computational Linguistics: ACL 2026, 2026, pp. 43 244–43 256.arxiv.org/abs/2510.15731
  65. [65]Z. Zhang, Z. Xie, L. Zhong, H. Liu, and S. Cao, “One token is enough: Improving diffusion language models with a sink token,” in Findings of the Association for Computational Linguistics: ACL 2026, 2026, pp. 6479–6490.arxiv.org/abs/2601.19657
  66. [66]H. Zuo, Y. Li, L. Yang, and X. Zhang, “Diffusion-cam: Faithful visual explanation for dllms,” in Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2026, pp. 12 085–12 101.
  67. [67]W. Zhu, H. Hao, and R. Wang, “Penalty decoding: Well suppress the self-reinforcement effect in open-ended text generation,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 1218–1228.arxiv.org/abs/2310.14971
  68. [68]Y. Su, T. Lan, Y. Wang, D. Yogatama, L. Kong, and N. Collier, “A contrastive framework for neural text generation,” Advances in neural information processing systems, vol. 35, pp. 21 548–21 561, 2022.arxiv.org/abs/2202.06417
  69. [69]Z. Fu, W. Lam, A. M.-C. So, and B. Shi, “A theoretical analysis of the repetition problem in text generation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 14, 2021, pp. 12 848–12 856.arxiv.org/abs/2012.14660
  70. [70]E. Lagutin, D. Gavrilov, and P. Kalaidin, “Implicit unlikelihood training: Improving neural text generation with reinforcement learning,” in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, 2021, pp. 1432–1441.arxiv.org/abs/2101.04229
  71. [71]J. Xu, X. Liu, J. Yan, D. Cai, H. Li, and J. Li, “Learning to break the loop: Analyzing and mitigating repetitions for neural text generation,” Advances in Neural Information Processing Systems, vol. 35, pp. 3082–3095, 2022.
  72. [72]H. Li, T. Lan, Z. Fu, D. Cai, L. Liu, N. Collier, T. Watanabe, and Y. Su, “Repetition in repetition out: Towards understanding neural text degeneration from the data perspective,” Advances in Neural Information Processing Systems, vol. 36, pp. 72 888–72 903, 2023.arxiv.org/abs/2310.10226
  73. [73]H. Yang, D. Cai, H. Li, W. Lam, S. Shi et al., “A frustratingly simple decoding method for neural text generation,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024, pp. 536–557.arxiv.org/abs/2305.12675
  74. [74]A. A. Ginart, N. Kodali, J. Lee, C. Xiong, S. Savarese, and J. R. Emmons, “LZ penalty: An information-theoretic repetition penalty for autoregressive language models,” arXiv preprint arXiv:2504.20131, 2025.arxiv.org/abs/2504.20131
  75. [75]W. Wang, Z. Li, D. Lian, C. Ma, L. Song, and Y. Wei, “Mitigating the language mismatch and repetition issues in llm-based machine translation via model editing,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 15 681–15 700.
  76. [76]S. Hong, C. Yoon, and S. J. Hwang, “Mitigating mask prior drift and positional attention collapse in large diffusion vision-language models,” in Forty-third International Conference on Machine Learning, 2026.
  77. [77]S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston, “Neural text generation with unlikelihood training,” arXiv preprint arXiv:1908.04319, 2019.arxiv.org/abs/1908.04319
  78. [78]G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis, “Efficient streaming language models with attention sinks,” in International Conference on Learning Representations, vol. 2024, 2024, pp. 21 875–21 895.arxiv.org/abs/2309.17453
  79. [79]R. Ali, F. Caso, C. Irwin, and P. Liò, “Entropy-lens: Uncovering decision strategies in Llms,” arXiv preprint arXiv:2502.16570, 2025.arxiv.org/abs/2502.16570
  80. [80]M. Mathew, D. Karatzas, and C. Jawahar, “Docvqa: A dataset for vqa on document images,” in 2021 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2021, pp. 2199–2208.arxiv.org/abs/2007.00398
  81. [81]A. Masry, J. Q. Tan, S. Joty, E. H. Hoque et al., “Chartqa: A benchmark for question answering about charts with visual and logical reasoning,” in Findings of the association for computational linguistics: ACL 2022, 2022, pp. 2263–2279.arxiv.org/abs/2203.10244
  82. [82]L. Chen, J. Li, X. Dong, P. Zhang, Y. Zhang, Z. Chen, H. Duan, J. Wang, Y. Qiao, D. Lin et al., “Are we on the right way for evaluating large vision-language models?” Advances in Neural Information Processing Systems, vol. 37, pp. 27 056–27 087, 2024.arxiv.org/abs/2403.20330
  83. [83]C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun et al., “Mme: A comprehensive evaluation benchmark for multimodal large language models,” Advances in Neural Information Processing Systems, vol. 38, 2026.
  84. [84]B. Li, R. Wang, G. Wang, Y. Ge, and Y. Shan, “Seed-bench: Benchmarking multimodal llms with generative comprehension,” arXiv preprint arXiv:2307.16125, 2023.arxiv.org/abs/2307.16125
  85. [85]Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu et al., “Mmbench: Is your multi-modal model an all-around player?” in European conference on computer vision. Springer, 2024, pp. 216–233.arxiv.org/abs/2307.06281
  86. [86]P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao, “Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts,” in International Conference on Learning Representations, vol. 2024, 2024, pp. 23 439–23 554.
  87. [87]R. Zhang, D. Jiang, Y. Zhang, H. Lin, Z. Guo, P. Qiu, A. Zhou, P. Lu, K.-W. Chang, Y. Qiao et al., “Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?” in European Conference on Computer Vision. Springer, 2024, pp. 169–186.
  88. [88]R. Paulus, C. Xiong, and R. Socher, “A deep reinforced model for abstractive summarization,” arXiv preprint arXiv:1705.04304, 2017.arxiv.org/abs/1705.04304

Paper details

vlnc:W_ed4c2rpz
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Grammarly, Claude

View statement
Looking for related papers…
Contents