betaFeedback

WorksComputer Science

Calibrating Attention-Based Token Pruning for Efficient Multi-Image Synthesis in UMMs

Abstract

Unified Multimodal Models (UMMs) enable powerful multi-image synthesis but incur substantial computation from long visual contexts. While token pruning offers a direct route to accelerating inference, existing methods are primarily adapted from understanding tasks, suffering from insufficient reference image exploration, confused conditional guidance, and pooling-induced ambiguity that degrades generation quality. To address these issues, we propose CATP, the first training-free context-compression method tailored to UMM generation, which Calibrates the discrepancies of general Attention-based Token Pruning for better quality-efficiency trade-off. Concretely, building on independent layer-wise pruning during denoising, we introduce Guidance-Consistent Pruning (GCP) to synchronize pruning decisions across conditioning branches, and a Taylor-Calibrated Attention Metric (TCAM) that derives a theoretically grounded second-order correction for pooling-induced bias. In addition, a Fast Scoring Operator (FSO) fuses the calibration into tiled online Softmax to reduce pruning cost. Extensive experiments across representative UMMs and benchmarks demonstrate that CATP consistently outperforms state-of-the-art token-reduction methods, achieving up to 2.09×\times attention-core speedup.

Introduction

Unified Multimodal Models (UMMs), which integrate understanding and generation within a shared backbone (Deng et al., [12]; Diao et al. [13]), have emerged as a prevalent paradigm for enhancing both capabilities (Zhao et al., [39]). Leveraging their strength in handling complex inputs, multi-image synthesis, which generates a target image from multiple references (Chen et al., [9]; Wei et al. [29]), has recently attracted growing interest in broad applications including fashion design (Choi et al., [10]), personalized content production (Wang et al., [28]), etc. However, this typically requires concatenating visual tokens from multiple high-resolution images, yielding long sequences that incur substantial attention computation during denoising (Chen et al., [9]; He et al. [17]).

In general, such long-context designs are originally intended to support diverse capabilities of foundation models (Deng et al., [12]), whereas a specific task may exploit only a subset of the available information (Zhang et al., [37]). Therefore, reducing the effective sequence length is a direct route to improving inference efficiency (Bolya et al., [5]; Chen et al. [8]). Among strategies, attention-based token pruning offers an effective and general baseline, leveraging native attention score to discard less informative tokens (Yao et al., [35]; Wu et al. [33]). Nevertheless, existing schemes are primarily developed for understanding-oriented models (Chen et al., [8]; Wu et al. [32]), while the few direct adaptations to UMMs overlook generation-specific characteristics (Ke et al., [19]; Chen et al. [9]), potentially discarding visual regions critical to faithful synthesis.

Specifically, we identify three key limitations: 1) Static pruning across timesteps and layers. Understanding tasks usually produce short responses (Liu et al., [21]), with most information aggregated into a few tokens at early layers (Wu et al., [32]), making one-shot pruning throughout inference feasible (Chen et al., [8]). In contrast, visual generation attends to complementary regions across denoising timesteps and layers, as in Figure 1(a). Especially under aggressive token budgets, a fixed pruning fails to cover the full reference content, yielding insufficient exploration of visual cues. 2) Inconsistent compression across conditioning branches. Classifier-Free Guidance (CFG) (Ho & Salimans, [18]), natively instantiated in UMMs (Deng et al., [12]), performs multiple forward passes with different input sequences, naturally producing branch-specific attention scores and motivating independent pruning. However, as shown in Figure 1(b), this yields mismatched token selections whose symmetric difference carries useless information yet is amplified in the CFG guidance residual, perturbing guided velocity prediction and compromising generation fidelity. 3) Ambiguous token selection under pooled query representations. Adapting pruning to varying context exploration requires frequent importance estimation, while exact attention-based metric with the still-lengthy denoising queries incurs substantial overhead, undermining the goal of acceleration. Average pooling on queries is commonly adopted to reduce this cost [36]. However, as observed in Figure 1(c), it smooths fine-grained attention responses to critical regions, obscuring their distinction from less relevant content and thus yielding less reliable context retention.

Figure 1. Empirical observations on Bagel using the general attention metric. (a) Top-10% tokens at timesteps 1/31 across layers 9/18/27, with their union in the last column. (b) Top-10% tokens from the (Left) full-condition and (Middle) image-only branches, with (Right) their symmetric difference; red boxes indicate branch-specific tokens. (c) (Left) Reference images with manually selected synthesis-critical regions highlighted in red boxes. (Middle/Right) Heatmaps of the maximum attention from generated-image locations to these regions under dense and pooled attention, respectively.

To address these issues, we propose CATP, a novel training-free framework for efficient multi-image synthesis in UMMs. Starting from the general attention-based token pruning, CATP identifies generation-specific discrepancies and calibrates them at both conditioning-branch and importance-estimation levels, thereby improving generation fidelity. Specifically, CATP prunes reference-image tokens in the KV cache across all layers at predefined timesteps, adapting compression to varying reference explorations. Building on this, we introduce Guidance-Consistent Pruning (GCP), which formulates the cross-branch mismatch of independent compression and synchronizes pruning decisions with the full-condition branch to preserve clean guidance. A second-order Taylor expansion then characterizes the bias of attention metric with pooled-query, leading to a Taylor-Calibrated Attention Metric (TCAM) for more reliable identification of critical reference tokens. Additionally, a Fast Scoring Operator (FSO) is developed with a custom Triton kernel. It integrates the calibration into tiled online Softmax computation, substantially reducing the scoring overhead.

In summary, our main contributions are three-fold:

  1. We analyze discrepancies in general attention-based token pruning for UMM multi-image synthesis and propose a training-free acceleration framework, dubbed CATP. To the best of our knowledge, CATP makes the first investigation of token pruning specifically tailored to UMM generation.

  1. We develop Guidance-Consistent Pruning (GCP), Taylor-Calibrated Attention Metric (TCAM) and Fast Scoring Operator (FSO), which address the rigidity, inconsistency, and ambiguity of general schemes, jointly enabling a flexible, accurate and efficient online context compression.

  1. We conduct extensive experiments on representative UMMs and benchmarks, showing that CATP delivers superior generation quality, with up to 2.09×2.09 \times attention-core acceleration.

Related Work

Unified Multimodal Models and Multi-Image Synthesis

Unified Multimodal Models (UMMs) enable understanding and generation within a single architecture [39]. Recent prevalent paradigm adopts a Large Language Model (LLM) as the backbone [39], where text generation remains autoregressive process, while images are synthesized via iterative flow matching in latent [12] or pixel space [13]. For image generation, conditioning inputs are encoded and concatenated into a token sequence, cached as fixed KV states, and repeatedly attended to by generated visual tokens throughout denoising [12, 6]. Multi-image synthesis leverages UMMs’ strengths in complex instruction understanding and high-quality generation ([29, 25]). However, multiple reference images substantially lengthen the input sequence, underscoring the practical necessity of efficient inference, especially in attention computation ([9]).

Efficient Inference via Token Reduction

Token Pruning. Token pruning has presented clear efficiency gains for long-context tasks ([35]), exploiting the task-irrelevant redundancy inherent in input sequences ([32]). Initially explored in understanding tasks, methods develop various criteria for token selection ([34, 1]). Among them, attention-based pruning offers a strong baseline across LLMs ([32]), Multimodal Large Language Models (MLLMs) ([8]), and the understanding branch of UMM ([19]), accelerating either prefilling through early token removal ([8]) or decoding via KV cache pruning ([27]).

Token Reduction for Visual Generation. Efficiency in visual generation has been studied from multiple perspectives, such as timestep reduction ([14]) and cross-step cache reusing ([22, 38]), while token reduction is orthogonal to them. Most methods originally target Diffusion Transformers (DiTs) ([26]), typically reducing the computed sequence by merging and subsequently unmerging denoising tokens ([15] (Bolya & Hoffman 2023)). ToPi ([20]) more closely follows the aforementioned token pruning paradigm, but shares pruning decisions across layers, overlooking layer-wise differences in information demand. The few attempts on UMMs largely transfer understanding-oriented techniques ([9]), neglecting generation-specific characteristics. In this work, we systematically analyze these distinctive properties and develop token pruning specifically tailored to UMM generation.

Methodology

Framework Overview

In context of mainstream UMMs, multiple reference images and an instruction are encoded into visual and text token sequences II and TT, respectively, and concatenated to form a context C=[I;T]C = [I; T] of length NN. Prefilling CC through the LL-layer, θ\theta-parameterized backbone fθ(⋅)f_\theta(\cdot) produces layer-wise representations {Cl}l=1L\{C^l\}_{l=1}^{L}, cached as fixed KV states. Starting from noisy visual tokens ZTZ_T of length MM, generation proceeds for TT iterative flow-based denoising steps as Eq. (1), where the intermediate visual states ZtlZ_{t}^{l} attend to ClC^l within the attention blocks at each layer, and Z0Z_0 is ultimately decoded into the output image.

Zt−1=fθ(Zt;t,{Cl}l=1L),t=T,…,1.(1)Z_{t-1} = f_\theta(Z_t; t, \{C^l\}_{l=1}^{L}), \quad t = T, \ldots, 1. \tag*{(1)}

Token pruning shortens ClC^l at a predefined timestep tpt^p, reducing attention cost in the current and subsequent denoising steps. Motivated by the dynamic reference exploration discussed in § 1, we adopt the common independent layer-wise pruning ([27]) as the baseline framework. Specifically, each token CjlC_j^l is assigned an importance score sjls_j^l, and the top-BB candidates are retained accordingly as:

C^t<tp,l={Cjl∣j∈TopK⁡({sjl}j=1N,B)}.(2)\hat{C}^{t<t^p,l} = \{C_j^l \mid j \in\operatorname{TopK}(\{s_j^l\}_{j=1}^{N}, B)\}. \tag*{(2)}

As established in prior work ([20]), the attention from denoising visual tokens to reference tokens effectively reflects the native information flow during image synthesis, providing a principled criterion for token pruning. Formally, let Q={qm}m=1MQ = \{q_m\}_{m=1}^{M} denote the queries projected from ZtplZ_{t^p}^{l}, and K={kj}j=1NK = \{k_j\}_{j=1}^{N} the keys of ClC^l, with the superscript ll omitted for clarity. Accordingly, the baseline importance score sjls_j^l is defined as the average attention received by token jj over all queries:

sjl=1M∑q∈QAttn⁡(q)j,Attn⁡(q)=Softmax⁡(qKTd)∈RN,(3)s_j^l = \frac{1}{M}\sum_{q\in Q}\operatorname{Attn}(q)_j,\quad\operatorname{Attn}(q)=\operatorname{Softmax}\left(\frac{qK^{\mathsf{T}}}{\sqrt{d}}\right)\in\mathbb{R}^{N}, \tag*{(3)}

where dd denotes head dimension, and we omit the head-wise averaging from the notation for brevity.

Building upon this general framework, CATP focuses on accelerating the computation-dominant denoising process, as illustrated in Figure 2. By further investigating the aforementioned limitations of vanilla attention-based pruning, we develop GCP to determine C^t<tp,l\hat{C}^{t<t^p,l} across condition branches and TCAM with FSO to refine sjls_j^l, thereby enabling more accurate and efficient context compression.

Framework overview showing the UMM inference pipeline and the GCP, TCAM, and FSO modules

Figure 2. Framework overview. (Top) UMM inference pipeline with CATP. It accelerates denoising by pruning visual KV cache tokens across layers at predefined timesteps. (Bottom) Illustration of the main modules. (a) GCP synchronizes pruning decisions from the full-condition to image-only branch, avoiding condition confusion from independent pruning. (b) TCAM estimates token importance using pooled-query attention and calibrates it with a theoretically grounded second-order term. (c) FSO efficiently realizes calibrated scoring via a two-pass tiled online-Softmax kernel.

Guidance-Consistent Pruning

The native CFG mechanism [18] instantiates Eq. (1) into three condition branches, each performing a complete forward pass under full, image-only or text-only conditioning:

Vtfull=vθ(Zt,t;I,T),Vtimg=vθ(Zt,t;I,∅),Vttext=vθ(Zt,t;∅,T),(4)V_{t}^{\mathrm{full}} = v_{\theta}(Z_t,t;I,T), \qquad V_{t}^{\mathrm{img}} = v_{\theta}(Z_t,t;I,\varnothing), \qquad V_{t}^{\mathrm{text}} = v_{\theta}(Z_t,t;\varnothing,T), \tag*{(4)}

where VV denotes the velocity field, and vθ(⋅)v_{\theta}(\cdot) is the velocity predictor of fθ(⋅)f_{\theta}(\cdot). This yields modality-specific guidance residuals Gtimg=Vtfull−VttextG_{t}^{\mathrm{img}}=V_{t}^{\mathrm{full}}-V_{t}^{\mathrm{text}} and Gttext=Vtfull−VtimgG_{t}^{\mathrm{text}}=V_{t}^{\mathrm{full}}-V_{t}^{\mathrm{img}}, capturing the incremental contribution of each modality. Accordingly, the guided velocity can be equivalently formulated as VtcfgV_{t}^{\mathrm{cfg}}, controlled by model-inherent weights λtext\lambda^{\mathrm{text}} and λimg\lambda^{\mathrm{img}}, and then used for denoising update:

Vtcfg=Vtfull+λimgGtimg+λtextGttext,Zt−1=Zt−ΔtVtcfg.(5)V_{t}^{\mathrm{cfg}} = V_{t}^{\mathrm{full}} + \lambda^{\mathrm{img}}G_{t}^{\mathrm{img}} + \lambda^{\mathrm{text}}G_{t}^{\mathrm{text}}, \qquad Z_{t-1}=Z_t-\Delta t V_{t}^{\mathrm{cfg}}. \tag*{(5)}

Notably, reference image tokens, which are the primary pruning targets, are shared by both the full-condition and image-only branches. However, as shown in Figure 1(b), the straightforward strategy of pruning each branch using its native online attention scores yields mismatched token selections. Formally, let IfullI^{\mathrm{full}} and IimgI^{\mathrm{img}} denote the visual tokens retained in the corresponding branches, with the layer index omitted for simplicity. The resulting text-guidance can be decomposed as:

G~ttext=vθ(Zt,t;Ifull,T)−vθ(Zt,t;Ifull,∅)+vθ(Zt,t;Ifull,∅)−vθ(Zt,t;Iimg,∅).(6)\widetilde{G}_{t}^{\mathrm{text}} = v_{\theta}(Z_t,t;I^{\mathrm{full}},T)-v_{\theta}(Z_t,t;I^{\mathrm{full}},\varnothing)+v_{\theta}(Z_t,t;I^{\mathrm{full}},\varnothing)-v_{\theta}(Z_t,t;I^{\mathrm{img}},\varnothing). \tag*{(6)}

Here, the first term remains clean and valid text guidance, whereas an unexpected condition confusion arises from the second term δtconf\delta_{t}^{\mathrm{conf}}, as it introduces a spurious guidance signal from the mismatched visual token sets, further amplified by the non-negligible scale λtext\lambda^{\mathrm{text}}.

To address this issue, we adopt a simple yet effective strategy that synchronizes pruning across branches sharing the same visual context. Concretely, token importance is estimated from the full-condition branch, which has access to complete context, and the resulting pruning decision is reused by the image-only branch, yielding:

I^img,t,l←I^full,t,l,∀ t,l.(7)\widehat{I}^{\mathrm{img},t,l} \leftarrow\widehat{I}^{\mathrm{full},t,l}, \quad\forall\, t,l. \tag*{(7)}

This eliminates the confusion term as δtconf=0\delta_{t}^{\mathrm{conf}}=0, ensuring the clean textual guidance as intended.

Taylor-Calibrated Attention Metric

Pooling-Induced Bias. Without loss of generality, we focus on a single layer, writing sjls_{j}^{l} in Eq. (3) as sjs_j for brevity. Although faithful, the sequence length of QQ incurs non-negligible computation, undetermining the acceleration objective. A common remedy is to spatially downsample QQ [9, 36]. Concretely, QQ is partitioned into G=M/w2G=M/w^2 non-overlapping w×ww \times w windows {Wg}g=1G\{W_g\}_{g=1}^{G}, each represented by an average-pooled query qˉg=1w2∑q∈Wgq\bar{q}_g=\frac{1}{w^2}\sum_{q\in W_g}q. This replaces the costly scoring over QQ in Eq. (3) with a compact query set, yielding sˉj=1G∑g=1GAttn⁡(qˉg)j\bar{s}_j=\frac{1}{G}\sum_{g=1}^{G}\operatorname{Attn}(\bar{q}_g)_j.

However, interchanging averaging with the nonlinear Softmax operation introduces a window-wise numerical bias between sjs_j and sˉj\bar{s}_j. Considering a single window WgW_g, the exact discrepancy is:

Δg,j=1w2∑q∈WgAttn⁡(q)j−Attn⁡(qˉg)j.(8)\Delta_{g,j}=\frac{1}{w^2}\sum_{q\in W_g}\operatorname{Attn}(q)_j-\operatorname{Attn}(\bar{q}_g)_j. \tag*{(8)}

Taylor-Based Metric Calibration. Given Eq. (8), our goal is to derive a lightweight estimate of Δg,j\Delta_{g,j} that compensates sˉj\bar{s}_j toward sjs_j, thereby improving the accuracy of token selection.

First, let ϵg,i=qg,i−qˉg\epsilon_{g,i}=q_{g,i}-\bar{q}_g denote the deviation of each query qg,i∈Wgq_{g,i}\in W_g from qˉg\bar{q}_g. Due to the spatial locality of visual features [23], ϵg,i\epsilon_{g,i} is typically small, motivating a Taylor approximation of Attn⁡(⋅)j\operatorname{Attn}(\cdot)_j around qˉg\bar{q}_g. Defining pg,j=Attn⁡(qˉg)jp_{g,j}=\operatorname{Attn}(\bar{q}_g)_j, and the second-order expansion gives:

Attn⁡(qg,i)j≈pg,j+∇pg,j⊤ϵg,i+12ϵg,i⊤Hg,jϵg,i,(9)\operatorname{Attn}(q_{g,i})_j \approx p_{g,j}+\nabla p_{g,j}^{\top}\epsilon_{g,i}+\frac{1}{2}\epsilon_{g,i}^{\top}H_{g,j}\epsilon_{g,i}, \tag*{(9)}

where ∇pg,j\nabla p_{g,j} and Hg,jH_{g,j} are the gradient vector and Hessian matrix of Attn⁡(⋅)j\operatorname{Attn}(\cdot)_j at qˉg\bar{q}_g, respectively. Subsequently, averaging Eq. (9) over WgW_g yields an approximation of Eq. (8) as:

Δg,j≈12Tr⁡(Hg,jΣg),Σg=1w2∑i=1w2ϵg,iϵg,i⊤,(10)\Delta_{g,j}\approx\frac{1}{2}\operatorname{Tr}(H_{g,j}\Sigma_g),\qquad\Sigma_g=\frac{1}{w^2}\sum_{i=1}^{w^2}\epsilon_{g,i}\epsilon_{g,i}^{\top}, \tag*{(10)}

where Tr⁡(⋅)\operatorname{Tr}(\cdot) denotes the trace of a matrix, and Σg\Sigma_g is the covariance matrix of query deviations.

Furthermore, define the attention-weighted key mean as kˉg=∑j=1Npg,jkj\bar{k}_g=\sum_{j=1}^{N}p_{g,j}k_j and denote the diagonal entries of Σg\Sigma_g by (σg,12,…,σg,d2)(\sigma_{g,1}^{2},\ldots,\sigma_{g,d}^{2}). As detailed in Sec. B.1, Hg,jH_{g,j} admits a closed-form expression in Eq. (11), while the weak cross-channel correlations enable Σg≈σg2I\Sigma_g\approx\sigma_g^2I, where σg2=1d∑c=1dσg,c2\sigma_g^2=\frac{1}{d}\sum_{c=1}^{d}\sigma_{g,c}^{2}.

Hg,j=pg,jd[(kj−kˉg)(kj−kˉg)⊤−∑n=1Npg,n(kn−kˉg)(kn−kˉg)⊤].(11)H_{g,j}=\frac{p_{g,j}}{d}\left[(k_j-\bar{k}_g)(k_j-\bar{k}_g)^{\top}-\sum_{n=1}^{N}p_{g,n}(k_n-\bar{k}_g)(k_n-\bar{k}_g)^{\top}\right]. \tag*{(11)}

Substituting these results into Eq. (10) yields an efficient computable approximation:

Δg,j≈pg,j2d(rg,j−rˉg),rg,j=σg2∥kj−kˉg∥22,rˉg=∑n=1Npg,nrg,n.(12)\Delta_{g,j}\approx\frac{p_{g,j}}{2d}(r_{g,j}-\bar{r}_g),\qquad r_{g,j}=\sigma_g^2\lVert k_j-\bar{k}_g\rVert_2^2,\qquad\bar{r}_g=\sum_{n=1}^{N}p_{g,n}r_{g,n}. \tag*{(12)}

Finally, aggregating the window-wise corrections formulates the calibrated importance score s~j\tilde{s}_j as Eq. (13), where we introduce the hyperparameter λ\lambda to control the strength of correction.

s~j=1G∑g=1Gp~g,j,p~g,j=pg,j+λpg,j2d(rg,j−rˉg).(13)\tilde{s}_j=\frac{1}{G}\sum_{g=1}^{G}\tilde{p}_{g,j},\qquad\tilde{p}_{g,j}=p_{g,j}+\lambda\frac{p_{g,j}}{2d}(r_{g,j}-\bar{r}_g). \tag*{(13)}

Remark. This derivation indicates that pooling-induced bias is jointly governed by local query variation and key dispersion within the corresponding attention view. Larger query variance implies greater information loss from pooling, while larger centered key distance makes attention response more sensitive to such variation, both requiring stronger correction. This also echoes prior heuristic use of variance for regional importance [15, 31], while our derivation further reveals its theoretical role in modulating importance rather than directly estimating it.

Fast Scoring Operator

While parallelizable across windows and keys, the calibration still involves a sequential dependency pg,j→kˉg→rˉg→p~g,jp_{g,j}\rightarrow\bar{k}_g\rightarrow\bar{r}_g\rightarrow\tilde{p}_{g,j}, where the repeated pg,jp_{g,j}-weighted aggregations require multiple accesses to large-scale matrices, incurring non-negligible memory traffic. Therefore, we develop a tiled window-wise scoring operator that fuses the correction into the native computation of pg,jp_{g,j}.

First, as detailed in the Sec. B.2, p~g,j\tilde{p}_{g,j} can be equivalently reformulated as:

p~g,j=pg,j[1+λσg22d(∥kj∥22−2kj⊤kˉg+2∥kˉg∥22−∑n=1Npg,n∥kn∥22)].(14)\tilde{p}_{g,j}=p_{g,j}\left[1+\frac{\lambda\sigma_g^2}{2d}\left(\lVert k_j\rVert_2^2-2k_j^{\top}\bar{k}_g+2\lVert\bar{k}_g\rVert_2^2-\sum_{n=1}^{N}p_{g,n}\lVert k_n\rVert_2^2\right)\right]. \tag*{(14)}

This avoids the explicit dependence on rˉg\bar{r}_g, allowing the required statistics to be accumulated alongside online Softmax. Concretely, as summarized in Algo. 1, we adopt a tiled two-pass procedure inspired by FlashAttention [11] and Sec. B.2. Notably, without materializing intermediate logits, two passes are minimally required to explicitly recover the normalized probabilities pg,jp_{g,j}.

During the first pass, the native logit maximum mbm_b and normalizer ℓb\ell_b are updated online in Static Random-Access Memory (SRAM), and the attention-weighted key mean vv and squared-norm statistic uu are simultaneously accumulated under the same rescaling scheme, as Lines 6∼8. After traversing all key blocks, normalization by ℓTK\ell_{T_K} yields the exact kˉg\bar{k}_g and ∑n=1Npg,n∥kn∥2=u\sum_{n=1}^{N}p_{g,n}\lVert k_n\rVert^2=u, as Line 10.

The second pass reloads each key block and recomputes the exact pg,jp_{g,j} utilizing the finalized statistics mTKm_{T_K} and ℓTK\ell_{T_K}, as shown in Line 14. With all window-level statistics available, the correction then depends only on the current key-specific terms ∥kj∥2\lVert k_j\rVert^2 and kjTkˉgk_j^{\mathsf{T}}\bar{k}_g, enabling p~g,j\tilde{p}_{g,j} to be computed immediately as Line 15 and accumulated back into High Bandwidth Memory (HBM) score tile.

Overall, only calibrated scores are materialized, avoiding separate attention and correction matrices.

Experimental Results and Analysis

Experimental Settings

Models. Following recent advances in UMMs, we evaluate methods on two representative models: Bagel [12] and SenseNova-U1 [13], with generation performed in latent and pixel spaces, respectively. For Bagel, we use the multi-image fine-tuned checkpoint from Chen et al. [9], while SenseNova-U1 adopts its official checkpoint with native multi-image support.

Datasets and Evaluation Metrics. We conduct experiments on three widely used multi-image synthesis benchmarks, following their official splits and metric aggregation rules. Concretely, we report the geometric mean of Image Consistency and Prompt Following scores on the MacroBench [9]. Customization subset across four reference-number groups (1–3, 4–5, 6–7, and 8–10), the Weighted-Ref-VIEScore for Object-Centric, Human-Centric, Human-Object Interaction (HOI), and Decomposition-and-Recomposition (De&Re) tasks on MICo-Bench [29], and the geometric mean of Prompt Following and Subject Consistency scores for the Single/Multiple/Scene categories of OmniContext [30], each further divided into Character (Char.)/Object (Obj.)/Char.+Obj. subcategories. We also include the average score (Avg.) for MacroBench and OmniContext, and the Overall score for MICo-Bench. For metrics involving MLLM-based evaluation, we utilize the open-source Qwen3-VL-32B-Instruct [2] as the evaluator.

Implementation Details. We implement all methods on top of the TorchUWM library [24], with FlashAttention [11] the acceleration baseline. Since visual tokens dominate the

Algorithm 1Inner Loop of the Fast Scoring Operator (FSO) (Native attention; Taylor correction).
Input:Pooled query qˉg\bar{q}_g, variance σg2\sigma_g^2, key matrix K∈RN×dK \in\mathbb{R}^{N \times d} in HBM; KK is partitioned into TK=⌈N/BK⌉T_K=\lceil N/B_K\rceil blocks {Kb}b=1TK\{K_b\}_{b=1}^{T_K}, each of size BK×dB_K \times d; hyperparameter λ\lambda; 1d\frac{1}{\sqrt{d}} is absorbed into KK and omitted for brevity.
Output:Calibrated importance scores p~g∈RN\tilde{p}_g \in\mathbb{R}^{N} in HBM.
1:Load qˉg\bar{q}_g, σg2\sigma_g^2 from HBM to on-chip SRAM.
2:On chip, initialize (m0,ℓ0)=(−∞,0)(m_0,\ell_0)=(-\infty,0) and (v,u)=(0,0)(v,u)=(0,0) in SRAM.
3:Pass 1: online Softmax with key-moment accumulation.
4:for b=1,…,TKb=1,\ldots,T_K do
5:Load KbK_b from HBM to on-chip SRAM.
6:On chip, compute zb←qˉgKbTz_b\leftarrow\bar{q}_gK_b^{\mathsf{T}}, mb←max⁡(mb−1,max⁡(zb))m_b\leftarrow\max(m_{b-1},\max(z_b)).
7:On chip, compute eb←exp⁡(zb−mb)e_b\leftarrow\exp(z_b-m_b), α←exp⁡(mb−1−mb)\alpha\leftarrow\exp(m_{b-1}-m_b), ℓb←αℓb−1+sum⁡(eb)\ell_b\leftarrow\alpha\ell_{b-1}+\operatorname{sum}(e_b).
8:On chip, compute v←αv+ebKbv\leftarrow\alpha v+e_bK_b, u←αu+ebRowSqNorm⁡(Kb)u\leftarrow\alpha u+e_b\operatorname{RowSqNorm}(K_b).
9:end for
10:On chip, compute kˉg←v/ℓTK\bar{k}_g\leftarrow v/\ell_{T_K}, u←u/ℓTKu\leftarrow u/\ell_{T_K}.
11:Pass 2: probability recomputation with Taylor calibration.
12:for b=1,…,TKb=1,\ldots,T_K do
13:Reload KbK_b from HBM to on-chip SRAM.
14:On chip, compute zb←qˉgKbTz_b\leftarrow\bar{q}_gK_b^{\mathsf{T}}, pg,b←exp⁡(zb−mTK)/ℓTKp_{g,b}\leftarrow\exp(z_b-m_{T_K})/\ell_{T_K}.
15:On chip, compute Ab←RowSqNorm⁡(Kb)−2kˉgKbT+2∥kˉg∥2−uA_b\leftarrow\operatorname{RowSqNorm}(K_b)-2\bar{k}_gK_b^{\mathsf{T}}+2\lVert\bar{k}_g\rVert^2-u, p~g,b←pg,b⊙(1+λσg2Ab)\tilde{p}_{g,b}\leftarrow p_{g,b}\odot(1+\lambda\sigma_g^2A_b).
16:Accumulate p~g,b\tilde{p}_{g,b} into the bb-th tile of p~g\tilde{p}_g in HBM.
17:end for
18:return p~g\tilde{p}_g

Table 1.

MethodMacroBench 1–3MacroBench 4–5MacroBench 6–7MacroBench 8–10MacroBench Avg.↑MICo-Bench ObjectMICo-Bench HumanMICo-Bench HOIMICo-Bench De&ReMICo-Bench Overall↑
Full Model8.898.517.977.208.1447.5240.6842.4753.6946.66
retain 25% tokens
Text-Aligned8.218.228.077.447.9852.1135.7746.9552.9247.64
VisionZip8.188.078.006.977.8149.6522.9938.7956.4043.38
SDTM8.137.997.687.507.8338.9624.7925.8641.5232.90
ToPi8.618.398.127.498.1553.4840.1746.0063.6951.97
CATP (Ours)8.768.698.517.878.4654.8139.8749.4963.6153.22
retain 10% tokens
Text-Aligned7.097.307.536.957.2249.4223.6039.7255.1843.35
VisionZip5.446.026.255.705.8537.4313.9227.5846.9533.00
SDTM7.217.086.816.476.8931.4515.1818.8228.8123.41
ToPi7.327.187.286.867.1646.8929.2938.8156.6044.20
CATP (Ours)8.408.338.317.548.1552.0129.5142.8858.0446.84

Table 1 (PDF p. 7). Comparison of multi-image synthesis performance scores by various methods for Bagel, evaluated on the MacroBench and MICo-Bench datasets. The best results are highlighted in bold.

multimodal context, pruning is conducted exclusively on them, including both ViT and VAE tokens in Bagel and patch-level visual tokens in SenseNova-U1. By following Lin et al. [20], pruning is performed over the full sequence at denoising timesteps tp∈{49,39,29,19,9}t^{p} \in\{49, 39, 29, 19, 9\}. Identical retention ratio is applied to each reference image. In all experiments, we fix λ=0.5\lambda= 0.5 and w=4w = 4 for TCAM and BK=64B_{K} = 64 for FSO. All other inference settings, e.g., sampling and guidance parameters, follow the original model configurations. All experiments are conducted on one high-performance GPU.

Main Results

Since token pruning for UMM generation remains rarely explored, we adapt representative baselines from closely related paradigms underlying UMMs, i.e., MLLMs and DiTs. Specifically, Text-Aligned Pruning is previously applied to UMM-based multi-image synthesis by Chen et al. [9]. Both it and VisionZip (Yang et al. [34]) are originally developed for MLLM understanding tasks. SDTM (Fang et al. [15]) and ToPi (Lin et al. [20]) are DiT-oriented methods. For fair comparison, we restrict all methods to attention acceleration and adjust each to an equivalent computation reduction as CATP under predefined token retention ratios.

Quantitative Results. As shown in Tab. 1, CATP consistently outperforms compared methods across varying numbers of reference images and diverse generation tasks. Specifically, it surpasses the second-best methods by 0.31 and 1.25 points on MacroBench and MICo-Bench, respectively at a 25% retention ratio, and further increases the advantages to 0.93 and 2.64 points respectively at 10% retention ratio, demonstrating more pronounced robustness under aggressive context compression.

Notably, CATP even outperforms the original full model on MacroBench Avg. at a 25% retention ratio, with gains primarily arising in multi-image settings (≥6\ge6 images) and reaching up to 0.67 points. This stems from the effective redundancy removal. As discussed in prior work, the intrinsic normalization property of attention can dilute individual weights in excessively long sequences [32], making critical reference regions harder to distinguish and thereby yielding confused spatial layouts or degraded fine-grained details. Accordingly, pruning redundant context mitigates this effect by reducing unnecessary attention competition [9], while CATP further amplifies this benefit by more accurately retaining generation-critical regions.

Generalization Across Denoising Spaces. Given recent advances in pixel-space UMMs, we further adapt methods to the representative SenseNova-U1 and evaluate on OmniContext. As shown in Table 2, CATP maintains notable advantages of 0.26 and 0.42 Avg. points on Bagel and SenseNova-U1 at 25% token retention, respectively, with the margins further widening to 0.5 and 1.2 at 10% retention in Tab. A. These results demonstrate its impressive generalizability across denoising spaces.

MethodSingle Char.Single Obj.Multiple Char.Multiple Obj.Multiple Char.+Obj.Scene Char.Scene Obj.Scene Char.+Obj.Avg.↑
Bagel (Latent Space)
Full Model8.509.287.979.228.607.498.218.158.43
Text-Aligned8.168.777.569.108.257.468.237.648.14
VisionZip7.067.906.788.647.977.408.507.587.73
SDTM7.168.326.447.107.566.327.456.977.16
ToPi6.988.616.828.898.407.088.428.027.90
CATP (Ours)8.459.107.558.988.597.578.728.268.40
SenseNova-U1 (Pixel Space)
Full Model7.757.787.718.858.107.127.067.067.68
Text-Aligned7.116.967.698.607.436.977.807.327.48
VisionZip7.347.517.318.667.927.067.216.897.49
SDTM7.027.736.848.017.466.896.696.227.11
ToPi6.096.686.587.407.186.287.946.816.87
CATP (Ours)7.428.067.768.768.407.527.697.637.91

Table 2. Comparison of performance scores by various methods for Bagel/SenseNova-U1 on the OmniContext dataset with 25% tokens retained. The best results are highlighted in bold.

Qualitative Results. As shown in Figure 3, CATP delivers more favorable overall generation quality, particularly in spatial composition, human identity preservation, and critical background details. This stems from the joint effects of our designs, where GCP improves generation fidelity through coherent guidance, and TCAM better preserves fine-grained identity cues by assigning stronger importance corrections to regions with more details, making them more likely to survive pruning.

Qualitative comparison of various methods on Bagel at a 25% token retention ratio

Figure 3. Qualitative comparison of various methods on Bagel at a 25% token retention ratio. The failure/well-preserved regions are marked with red/green boxes.

Further Remarks. Text-Aligned relies on attention signals from text or within reference images, which primarily reflect multimodal understanding rather than direct generation relevance. VisionZip merges similar visual regions, favoring coarse structural preservation but potentially losing identity-critical details. SDTM compresses only denoising tokens, making it better suited to simple generation scenarios but prone to substantial degradation when targeting higher acceleration. While ToPi explicitly targets context compression for generation, it reuses pruning decisions across layers to reduce scoring overhead, overlooking layer-wise differences in context exploration, and the fully computed anchor timesteps further limit the compression scale. In contrast, CATP develops a more accurate and efficient scoring process, enabling layer-wise token evaluation and selection. Under aggressive compression, different layers can exploit complementary critical regions rather than maintaining complete target content throughout, thereby maintaining overall generation quality.

Efficiency. We further assess the acceleration effect of CATP, with FlashAttention as the baseline. For each image count, we fix 1024-resolution inputs, sample 10 examples and report the average latency of the accelerated attention cores across all denoising calls. As shown in Figure 4(a), attention computation becomes increasingly dominant with more reference images, translating context compression into larger practical gains, reaching 2.09×2.09 \times with 10 reference images at 25% retention.

Figure 4. (a) Attention core latency of the full model and CATP, along with the relative speedup of CATP, across varying numbers of reference images. (b) Visualization of (Left) window-wise query variance and (Right) the resulting aggregated calibration term over reference images in TCAM.

Ablation Study

On GCP. As shown in Table 3, GCP effectively improves conditional-guidance consistency, yielding average gains of 0.23 and 1.16 points at 25% and 10% retention ratios, respectively. Intuitively, more aggressive pruning makes the symmetric difference between conditioning branches increasingly dominated by less important tokens, introducing stronger irrelevant signals into the guidance residual. Therefore, GCP becomes more beneficial at lower retention ratios.

GCPTCAMRetain 25% Tokens: 1–34–56–78–10Avg.↑Retain 10% Tokens: 1–34–56–78–10Avg.↑
8.328.198.147.628.076.656.877.256.856.91
✓8.758.658.367.458.308.238.288.397.388.07
✓✓8.768.698.517.878.468.408.338.317.548.15

Table 3. Ablation of GCP and TCAM modules for Bagel on MacroBench at two retention ratios.

On TCAM. Table 3 also indicates that TCAM further improves gains under both settings. As visualized in Figure 4(b), high-variance denoising regions concentrate around object boundaries and identity-critical details, consistent with the widely recognized properties in prior studies [15]. TCAM exploits this signal via second-order calibration to strengthen corresponding reference regions, thus promoting the retention of identity-critical tokens for finer-grained synthesis.

On FSO. We further assess the effect of FSO in Table 4, by measuring scoring latency with ten 1024-resolution reference images. Concretely, dense attention achieves the best generation quality but incurs prohibitive pruning cost, while query pooling substantially reduces latency at the cost of degraded performance. Given TCAM’s additional calibration cost, FSO finally mitigates this by algebraically reformulating it into a fused operator, reducing latency to 5.39 ms without sacrificing quality. Notably, FSO is even faster than the pooling baseline, which owes to fine-grained HBM-level optimization enabling BF16 multiplication with FP32 accumulation.

PoolingTCAMFSOLatency↓Avg.↑
102.978.48
✓7.248.30
✓✓30.128.46
✓✓✓5.398.46

Table 4. Effect of TCAM and FSO modules on scoring latency (ms) and MacroBench average score (Avg.).

Conclusion

In this work, we conduct a pioneering investigation of visual token pruning for efficient UMM multi-image synthesis, and develop a novel training-free method, namely CATP. By performing layer-wise pruning on KV cache tokens, CATP enables adaptive utilization of different reference information. Furthermore, we explicitly preserve clean conditional guidance through Guidance-Consistent Pruning (GCP) and improve the discriminability of critical regions with a theoretically derived Taylor-Calibrated Attention Metric (TCAM), together with a Fast Scoring Operator (FSO) that reduces the scoring cost. Experiments on prevalent latent- and pixel-space UMMs across three multi-image benchmarks demonstrate the effectiveness of CATP, consistently outperforming competitive token-reduction approaches across retention ratios while achieving up to 2.09×2.09 \times attention-core speedup.

AI Use Statement

In this work, we used generative AI tools for assist in the writing of proofs, assist with translation. We have not used generative AI tools for help develop theoretical models or conceptual frameworks, formulate mathematical claims, provide critical ingredients for proving mathematical claims, propose or refine hypotheses, design or provide feedback on research methodology or experiments, implement methods, interpret results, and generate synthetic data sets, clean and reformat dataset, support qualitative and thematic data analysis are not applicable to this work. Additionally, we used generative AI tools for sourcing/searching for information, identify relevant literature.

We have reviewed all AI-assisted work. Specifically, we independently verified all AI-polished mathematical formulations and proof-related content step by step to ensure their correctness and consistency with the underlying mathematical definitions. AI-assisted translations were manually reviewed by 2 authors to preserve the intended meaning. Information obtained through AI-assisted search was cross-checked against reliable sources, and all AI-identified literature was verified by consulting the corresponding original publications before being cited or discussed in this manuscript.

We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

References

Under review

More Experimental Results and Analysis

Additional Quantitative Results on OmniContext

To complement the 25% retention results reported in Table 2, we further evaluate all methods on OmniContext [30] under a more aggressive 10% token retention ratio. As shown in Tab. A, CATP continues to achieve superior performance in most scenarios on both Bagel and SenseNova-U1, further widening its margin over the second-best methods to 0.5 and 1.2 points, respectively. These results provide additional evidence for the robustness of CATP under aggressive acceleration and its generalization across various UMM generation spaces.

MethodSingleMultipleSceneAvg.↑\uparrow
Char.Obj.Char.Obj.Char.+Obj.Char.Obj.Char.+Obj.
Bagel (Latent Space)
Full Model8.509.287.979.228.607.498.218.158.43
Text-Aligned7.067.365.178.337.336.137.957.267.07
VisionZip4.384.623.677.504.014.277.116.065.20
SDTM6.576.235.895.776.685.316.315.756.06
ToPi5.336.654.167.926.905.158.206.896.38
CATP (Ours)6.748.196.787.948.256.898.147.617.57
SenseNova-U1 (Pixel Space)
Full Model7.757.787.718.858.107.127.067.067.68
Text-Aligned5.724.786.217.886.566.407.946.586.51
VisionZip6.356.195.547.906.515.697.396.226.47
SDTM6.367.276.466.906.636.015.765.416.35
ToPi4.705.644.456.965.385.137.545.875.71
CATP (Ours)7.317.457.848.558.187.047.867.427.71

Table A. Comparison of performance scores by various methods for Bagel/SenseNova-U1 on the OmniContext dataset with 10% tokens retained. The best results are highlighted in bold.

Further Ablations

On Calibration Strength λ\lambda. Since TCAM adopts approximations to avoid computing higher-order terms, its second-order estimate captures the correction trend rather than an exact discrepancy. Therefore, we heuristically introduce λ\lambda in Eq. (13) to control the calibration strength, whose effect is ablated in Tab. B. As λ\lambda decreases, TCAM approaches the vanilla pooling baseline and provides insufficient emphasis on fine-grained reference regions, whereas an excessively large value biases selection toward high-variance regions and may suppress smoother yet relevant content. Given the overall performance peak at λ=0.5\lambda= 0.5, we empirically fix this value for all experiments.

λ\lambda1–34–56–78–10Avg.↑\uparrow
08.758.658.367.458.30
0.258.738.708.347.738.38
0.508.768.698.517.878.46
0.758.798.768.497.778.45
1.008.818.678.377.748.40
1.258.838.658.357.688.38

Table B. Effect of the correction strength λ\lambda on MacroBench score for Bagel at a 25% token retention ratio. The default setting is underlined.

On Window Size ww. We assess the effect of the pooling window size ww in TCAM. As reported in Tab. C, smaller windows incur less loss of fine-grained cues and therefore yield better generation quality, but at the cost of a longer query sequence with higher scoring latency. Larger windows exhibit the opposite trade-off, reducing scoring overhead while sacrificing more local details. Accordingly, we empirically choose w=4w = 4 as a balanced configuration and fix it across all experiments.

Table C: Effect of the window size ww on MacroBench score and scoring latency (ms) for Bagel at a 25% token retention ratio. The default setting is underlined.

ww1–34–56–78–10Avg.↑Latency↓
28.758.738.497.988.4917.87
48.768.698.517.878.465.39
88.688.578.277.688.302.31

Table C.

Qualitative Ablation on Main Modules. Beyond the quantitative ablations in Table 3, we further visualize the effects of GCP and TCAM in Fig. A. With the general pruning scheme, independent token selection across conditioning branches amplifies irrelevant visual cues through CFG guidance, leading to implausible rendering or distorted content. GCP alleviates this issue by enforcing consistent pruning across branches. TCAM further improves generation by emphasizing information-rich reference regions, yielding local patterns and details that better match the references.

Qualitative ablation of GCP and TCAM modules, showing Reference Images, Baseline, + GCP, and + GCP + TCAM results

Figure A. Reference Images; Baseline; + GCP; + GCP + TCAM. Place the red panda on the edge of the bed, curling up against the decorative pillows as it gazes out the window, soaking in the warm sunlight. A … lighting fixture from image 1 hangs above a dining table, illuminating a … bowl from image 2 placed at the center. Nearby, a travel guide app for digital devices from image 3 is open on a smartphone, while a … pill from image 4 rests on the table, hinting at a mindful gathering. Qualitative ablation of GCP and TCAM modules on Bagel at a 25% token retention ratio. The failure/well-preserved regions are marked with red/green boxes.

On Alternatives for Pooling-Induced Bias. For the pooling-induced bias discussed in § 3.3, we further compare our solution (i.e. TCAM + FSO) with several alternative designs in Tab. D. intuitively, the inferior performance of the baseline Average Pooling stems from indiscriminate within-window averaging that smooths fine-grained cues. Therefore, a direct alternative is Max Pooling, however, channel-wise maxima may originate from different spatial tokens, disrupting the semantic coherence of the pooled representation and yielding even worse performance. Inspired by our derivation in Eq. (13), which suggests strengthening the contribution of high-variance denoising regions, we further consider an intuitive alternative that leaves queries from the highest-variance windows unpooled, denoted as Mixed Query. Its improved generation quality in Tab. D provides additional empirical support for our motivation, but the enlarged query sequence incurs increased scoring latency. In contrast, TCAM together with FSO achieves gains in both generation quality and scoring efficiency, further validating the effectiveness of our designs.

Robustness Across MLLM Evaluators

Noting that MLLM-based evaluation can exhibit evaluator-specific biases and judgment inconsistencies [7], we therefore re-evaluate all methods on MacroBench utilizing two other representative MLLMs: the open-source Qwen2.5-VL-72B-Instruct [3] and the closed-source Gemini-3-Flash [16]. As shown in Tab. E, CATP still consistently achieves the best performance across both evaluators and distinct token retention ratios, despite noticeable differences in their scoring distributions. This further suggests that CATP more reliably preserves generation quality rather than benefiting from a particular evaluator.

Method1–34–56–78–10Avg.↑\uparrowLatency↓\downarrow
Average Pooling8.758.658.367.458.307.24
Max Pooling8.598.568.297.508.247.23
Mixed Query8.808.748.487.668.4231.47
TCAM + FSO8.768.698.517.878.465.39

Table D. Effect of various design choices for the pooling-induced bias on MacroBench for Bagel at a 25% retention ratio. The best results are highlighted in bold and the default setting is underlined.

Additionally, it is noteworthy that, under the more capable evaluator, performance with larger numbers of reference images remains far from the maximum score (i.e. 10.0). While context compression provides partial recovery, we consider that further gains in such challenging scenarios should primarily stem from stronger native multi-reference generation capabilities of the foundation model, motivating continued research along this direction.

MethodQwen2.5-VL-72B-Instruct: 1–3Qwen2.5-VL-72B-Instruct: 4–5Qwen2.5-VL-72B-Instruct: 6–7Qwen2.5-VL-72B-Instruct: 8–10Qwen2.5-VL-72B-Instruct: Avg.↑\uparrowGemini-3-Flash: 1–3Gemini-3-Flash: 4–5Gemini-3-Flash: 6–7Gemini-3-Flash: 8–10Gemini-3-Flash: Avg.↑\uparrow
Full Model8.437.656.455.426.999.018.486.934.457.21
retain 25% tokens
Text-Aligned8.077.526.265.346.808.058.346.764.826.99
VisionZip7.817.365.885.026.527.437.716.363.726.31
SDTM7.686.755.283.885.907.847.696.424.596.63
ToPi8.107.456.315.186.768.478.336.804.667.06
CATP (Ours)8.447.846.265.627.048.878.707.545.257.59
retain 10% tokens
Text-Aligned7.346.465.754.636.056.136.455.413.895.47
VisionZip5.575.054.574.184.844.434.073.172.113.45
SDTM6.715.404.083.054.816.746.375.423.795.58
ToPi7.136.395.054.645.806.946.344.713.205.30
CATP (Ours)8.087.556.035.156.708.108.207.014.997.07

Table E. Comparison of multi-image synthesis performance scores by various methods for Bagel on MacroBench, evaluated by Qwen2.5-VL-72B-Instruct and Gemini-3-Flash. The best results are highlighted in bold.

Statistical Stability

We further assess the statistical stability of methods on Bagel at a 25% token retention ratio. Specifically, all methods are evaluated on MacroBench over five random seeds for image generation, with the mean and standard deviation reported. As shown in Tab. F, CATP consistently maintains superior performance, improving the Avg. over the second-best method by 0.306 points while exhibiting the lowest Avg. standard deviation of 0.02. These results further demonstrate both the effectiveness and stability of our method.

Compatibility with Orthogonal Acceleration Methods

CATP accelerates inference from the perspective of context compression, reducing attention computation by removing redundant conditional tokens. As discussed in § 2, several methods optimize efficiency along orthogonal dimensions, motivating an exploration of their joint effect with CATP. Specifically, we combine CATP at a 25% token retention ratio with two representative approaches,

Table F: Comparison of MacroBench scores (mean ±\pm std) by various methods on Bagel at a 25% token retention ratio. The best results are highlighted in bold.

Method1–34–56–78–10Avg.↑\uparrow
Full Model8.810 ±\pm 0.0458.412 ±\pm 0.0838.048 ±\pm 0.0787.256 ±\pm 0.0998.132 ±\pm 0.047
Text-Aligned8.210 ±\pm 0.0418.164 ±\pm 0.0608.064 ±\pm 0.0487.406 ±\pm 0.0647.960 ±\pm 0.027
VisionZip8.128 ±\pm 0.0658.010 ±\pm 0.0917.966 ±\pm 0.1687.012 ±\pm 0.1537.780 ±\pm 0.065
SDTM8.110 ±\pm 0.2228.048 ±\pm 0.1207.876 ±\pm 0.1577.252 ±\pm 0.1657.822 ±\pm 0.066
ToPi8.570 ±\pm 0.0568.278 ±\pm 0.0738.240 ±\pm 0.0877.532 ±\pm 0.1138.152 ±\pm 0.032
CATP (Ours)8.814 ±\pm 0.0738.640 ±\pm 0.1048.540 ±\pm 0.0397.838 ±\pm 0.0588.458 ±\pm 0.020

Table F.

SDTM (Fang et al., [15] and TaylorSeer (Liu et al., [22], both adopting their official default settings. SDTM shortens the denoising sequence, whereas TaylorSeer performs block-wise computation skipping across timesteps, and we therefore report the end-to-end speedup ratio of the entire denoising process relative to the full model, with 10 input reference images.

As shown in Tab. G, both methods alone already improve average generation quality over the full model. Here, SDTM fails to provide a meaningful acceleration, as the fixed context in UMM paradigm renders computation over its targeted denoising sequence non-dominant, leaving the overhead of frequent selection operation insufficiently amortized. Furthermore, when combined with these methods, CATP continues to reduce computational cost, while even improving generation performance by 0.19 and 0.26 points, respectively, primarily due to the elimination of interfering contextual redundancy. We also include representative generation results in Fig. B, where both methods produce erroneous outputs, yet their combination with CATP corrects these failures. These results demonstrate the favorable adaptability of CATP to prevalent techniques, and suggest that appropriate compression may itself provide a promising direction for improving downstream generation quality.

Table G: MacroBench scores and overall denoising speedup of SDTM and TaylorSeer, individually and in combination with CATP, on Bagel. The best results are highlighted in bold.

Method1–34–56–78–10Avg.↑\uparrowSpeedup↑\uparrow
Full Model8.898.517.977.208.141.00×\times
SDTM8.958.798.417.658.451.04×\times
SDTM + CATP (Ours)8.918.988.638.068.641.41×\times
TaylorSeer8.768.488.217.708.292.39×\times
TaylorSeer + CATP (Ours)8.788.958.657.818.553.06×\times

Table G.

Qualitative generation results by SDTM and TaylorSeer, individually and combined with CATP, on Bagel

Figure B. Qualitative generation results by SDTM and TaylorSeer, individually and combined with CATP, on Bagel. The failure/well-preserved regions are marked with red/green boxes.

Detailed Derivations

In this section, we present detailed derivations for TCAM and FSO. Sec. B.1 first formulates the pooling-induced attention discrepancy, then derives its second-order Taylor expansion with combined Hessian and query variance, yielding the calibrated importance score. Sec. B.2 reformulates this calibration into a simplified form, follow by a clarification of the theoretical basis of exact tiled online Softmax and the analogous online accumulation of the required intermediate statistics.

Derivation of the Taylor Calibration

Pooling-Induced Bias. We begin with the exact attention-based metric sjs_j computed over the complete query sequence QQ, corresponding to Eq. (3) in the main text:

sj=1M∑q∈QAttn⁡(q)j,Attn⁡(q)=Softmax⁡(qK⊤d)∈RN.(15)s_j = \frac{1}{M}\sum_{q\in Q}\operatorname{Attn}(q)_j,\qquad\operatorname{Attn}(q)=\operatorname{Softmax}\left(\frac{qK^{\top}}{\sqrt{d}}\right)\in\mathbb{R}^{N}. \tag*{(15)}

Afterward, partitioning QQ into G=M/w2G=M/w^2 non-overlapping w×ww\times w windows and applying average pooling yields, for the gg-th window,

qˉg=1w2∑i=1w2qg,i,(16)\bar{q}_g=\frac{1}{w^2}\sum_{i=1}^{w^2}q_{g,i}, \tag*{(16)}

which further produces the coarsened window-level score:

pg,j=Attn⁡(qˉg)j.(17)p_{g,j}=\operatorname{Attn}(\bar{q}_g)_j. \tag*{(17)}

Accordingly, grouping the queries by window, the original dense attention metric of Eq. (B.1) can be equivalently written as:

sj=1G∑g=1Gp^g,j,p^g,j=1w2∑i=1w2Attn⁡(qg,i)j,(18)s_j=\frac{1}{G}\sum_{g=1}^{G}\hat{p}_{g,j},\qquad\hat{p}_{g,j}=\frac{1}{w^2}\sum_{i=1}^{w^2}\operatorname{Attn}(q_{g,i})_j, \tag*{(18)}

whereas the metric with pooled queries becomes:

sˉj=1G∑g=1GAttn⁡(qˉg)j=1G∑g=1Gpg,j.(19)\bar{s}_j=\frac{1}{G}\sum_{g=1}^{G}\operatorname{Attn}(\bar{q}_g)_j=\frac{1}{G}\sum_{g=1}^{G}p_{g,j}. \tag*{(19)}

Although sˉj\bar{s}_j can be computed more efficiently, the difference from the exact sjs_j goes beyond a simple reordering of window-wise averaging and Softmax. Due to the nonlinearity of Softmax, each window incurs an additional non-negligible bias:

Δg,j=p^g,j−pg,j=1w2∑i=1w2Attn⁡(qg,i)j−pg,j,(20)\Delta_{g,j}=\hat{p}_{g,j}-p_{g,j}=\frac{1}{w^2}\sum_{i=1}^{w^2}\operatorname{Attn}(q_{g,i})_j-p_{g,j}, \tag*{(20)}

which is identical to Eq. (8) of Sec. 3.3, and aggregating these window-level discrepancies gives:

sj=sˉj+1G∑g=1GΔg,j.(21)s_j=\bar{s}_j+\frac{1}{G}\sum_{g=1}^{G}\Delta_{g,j}. \tag*{(21)}

Therefore, recovering the accurate importance score reduces to efficiently estimating the window-level correction Δg,j\Delta_{g,j}, which motivates the following derivation.

Taylor Expansion for Discrepancy Approximation. For the gg-th window, denote the deviation of each query qg,iq_{g,i} from the average-pooled representation qˉg\bar{q}_g as:

ϵg,i=qg,i−qˉg,(22)\epsilon_{g,i}=q_{g,i}-\bar{q}_g, \tag*{(22)}

Given the definition in Eq. (B.2), these deviations sum to zero within the corresponding window:

∑i=1w2ϵg,i=∑i=1w2qg,i−w2qˉg=0.(23)\sum_{i=1}^{w^2}\epsilon_{g,i}=\sum_{i=1}^{w^2}q_{g,i}-w^2\bar{q}_g=0. \tag*{(23)}

Due to the spatial locality of visual representations, queries within each window typically lie close to their pooled center. This motivates a local Taylor expansion of Attn⁡(qg,i)j\operatorname{Attn}(q_{g,i})_j around qˉg\bar{q}_g. Using Eq. (B.3), the expansion simplifies to:

Attn⁡(qg,i)j=Attn⁡(qˉg+ϵg,i)j=pg,j+∇pg,j⊤ϵg,i+12ϵg,i⊤Hg,jϵg,i+R(3)≈pg,j+∇pg,j⊤ϵg,i+12ϵg,i⊤Hg,jϵg,i,(24)\begin{aligned} \operatorname{Attn}(q_{g,i})_j &= \operatorname{Attn}(\bar{q}_g+\epsilon_{g,i})_j = p_{g,j}+\nabla p_{g,j}^{\top}\epsilon_{g,i}+\frac{1}{2}\epsilon_{g,i}^{\top}H_{g,j}\epsilon_{g,i}+R^{(3)}\\ &\approx p_{g,j}+\nabla p_{g,j}^{\top}\epsilon_{g,i}+\frac{1}{2}\epsilon_{g,i}^{\top}H_{g,j}\epsilon_{g,i}, \tag*{(24)} \end{aligned}

where ∇pg,j=∇qAttn⁡(q)j∣q=qˉg\nabla p_{g,j}=\left.\nabla_q\operatorname{Attn}(q)_j\right|_{q=\bar{q}_g} and Hg,j=∇q2Attn⁡(q)j∣q=qˉgH_{g,j}=\left.\nabla_q^2\operatorname{Attn}(q)_j\right|_{q=\bar{q}_g} denote the first-order derivative and the second-order Hessian matrix, respectively. Terms beyond the second order, represented by R(3)R^{(3)}, are omitted due to their negligible magnitude.

Furthermore, subtracting pg,jp_{g,j} from both sides of Eq. (B.10) and averaging over all queries within the window yields:

Δg,j=1w2∑i=1w2(Attn⁡(qg,i)j−pg,j)≈1w2∑i=1w2(∇pg,jTεg,i+12εg,iTHg,jεg,i)=∇pg,jT1w2∑i=1w2εg,i+12w2∑i=1w2(εg,iTHg,jεg,i).(25)\begin{aligned} \Delta_{g,j} &= \frac{1}{w^2}\sum_{i=1}^{w^2}\left(\operatorname{Attn}(q_{g,i})_j-p_{g,j}\right)\\ &\approx\frac{1}{w^2}\sum_{i=1}^{w^2}\left(\nabla p_{g,j}^{\mathsf{T}}\varepsilon_{g,i}+\frac{1}{2}\varepsilon_{g,i}^{\mathsf{T}}H_{g,j}\varepsilon_{g,i}\right)\\ &=\nabla p_{g,j}^{\mathsf{T}}\frac{1}{w^2}\sum_{i=1}^{w^2}\varepsilon_{g,i}+\frac{1}{2w^2}\sum_{i=1}^{w^2}\left(\varepsilon_{g,i}^{\mathsf{T}}H_{g,j}\varepsilon_{g,i}\right). \tag*{(25)} \end{aligned}

Given Eq. (B.9), the first-order term vanishes exactly. Meanwhile, by defining the population covariance of the within-window query deviations as:

Σg=1w2∑i=1w2εg,iεg,iT,(26)\Sigma_g=\frac{1}{w^2}\sum_{i=1}^{w^2}\varepsilon_{g,i}\varepsilon_{g,i}^{\mathsf{T}}, \tag*{(26)}

and applying the identity xTAx=Tr⁡(AxxT)x^{\mathsf{T}}Ax=\operatorname{Tr}(Axx^{\mathsf{T}}), where Tr⁡(⋅)\operatorname{Tr}(\cdot) is the matrix trace, the second-order term can be rewritten as 12Tr⁡(Hg,jΣg)\frac{1}{2}\operatorname{Tr}(H_{g,j}\Sigma_g). Therefore, Eq. (B.11) yields

Δg,j≈12Tr⁡(Hg,jΣg).(27)\Delta_{g,j}\approx\frac{1}{2}\operatorname{Tr}(H_{g,j}\Sigma_g). \tag*{(27)}

Hessian of the Softmax Probability. For notational convenience, we denote the attention logit associated with kjk_j as:

zj(q)=qTkjd,(28)z_j(q)=\frac{q^{\mathsf{T}}k_j}{\sqrt{d}}, \tag*{(28)}

such that the Softmax-normalized attention probability in Eq. (B.1) becomes:

pj≡Attn⁡(q)j=exp⁡(zj(q))∑n=1Nexp⁡(zn(q)).(29)p_j\equiv\operatorname{Attn}(q)_j=\frac{\exp(z_j(q))}{\sum_{n=1}^{N}\exp(z_n(q))}. \tag*{(29)}

Here for brevity, we temporarily omit the window index gg and evaluate all quantities at a generic query qq. The derivative of Softmax w.r.t its logits is:

∂pj∂zn=pj(I(j=n)−pn),(30)\frac{\partial p_j}{\partial z_n}=p_j\left(\mathbb{I}(j=n)-p_n\right), \tag*{(30)}

where I(⋅)\mathbb{I}(\cdot) is the indicator function. Applying the chain rule to Eq. (B.16) with Eq. (B.14) gives:

∇qpj=∑n=1N∂pj∂zn∇qzn=∑n=1Nkndpj(I(j=n)−pn)=pjd(kj−∑n=1Npnkn)=pjdcj,(31)\begin{aligned} \nabla_q p_j&=\sum_{n=1}^{N}\frac{\partial p_j}{\partial z_n}\nabla_q z_n=\sum_{n=1}^{N}\frac{k_n}{\sqrt{d}}p_j\left(\mathbb{I}(j=n)-p_n\right)\\ &=\frac{p_j}{\sqrt{d}}\left(k_j-\sum_{n=1}^{N}p_nk_n\right)\\ &=\frac{p_j}{\sqrt{d}}c_j, \tag*{(31)} \end{aligned}

where we define the attention-weighted mean key kˉ\bar{k} and the centered key vectors cjc_j as:

kˉ=∑n=1Npnkn,cj=kj−kˉ.(32)\bar{k}=\sum_{n=1}^{N}p_nk_n,\qquad c_j=k_j-\bar{k}. \tag*{(32)}

Adopting Eq. (B.17) and Eq. (B.18), we further prepare the Jacobian of kˉ\bar{k} w.r.t qq as:

∇qkˉ=∑n=1Nkn(∇qpn)T=1d∑n=1Npnkn(kn−kˉ)T=1d(∑n=1NpnknknT−kˉkˉT),(33)\begin{aligned} \nabla_q\bar{k}&=\sum_{n=1}^{N}k_n(\nabla_qp_n)^{\mathsf{T}}\\ &=\frac{1}{\sqrt{d}}\sum_{n=1}^{N}p_nk_n(k_n-\bar{k})^{\mathsf{T}}\\ &=\frac{1}{\sqrt{d}}\left(\sum_{n=1}^{N}p_nk_nk_n^{\mathsf{T}}-\bar{k}\bar{k}^{\mathsf{T}}\right), \tag*{(33)} \end{aligned}

Differentiating Eq. (B.17) to the second-order gradient and noting that ∇qcj=−∇qkˉ\nabla_q c_j=-\nabla_q\bar{k} now gives:

∇q2pj=1d∇q(pjcj)=1d(cj(∇qpj)⊤+pj∇qcj)=1d(cj(∇qpj)⊤−pj∇qkˉ)=pjd(cjcj⊤−(∑n=1Npnknkn⊤−kˉkˉ⊤))=pjd[(kj−kˉ)(kj−kˉ)⊤−∑n=1Npn(kn−kˉ)(kn−kˉ)⊤].(34)\begin{aligned} \nabla_q^2p_j&=\frac{1}{\sqrt{d}}\nabla_q(p_jc_j)=\frac{1}{\sqrt{d}}\left(c_j(\nabla_qp_j)^\top+p_j\nabla_qc_j\right)\\ &=\frac{1}{\sqrt{d}}\left(c_j(\nabla_qp_j)^\top-p_j\nabla_q\bar{k}\right)\\ &=\frac{p_j}{d}\left(c_jc_j^\top-\left(\sum_{n=1}^{N}p_nk_nk_n^\top-\bar{k}\bar{k}^\top\right)\right)\\ &=\frac{p_j}{d}\left[(k_j-\bar{k})(k_j-\bar{k})^\top-\sum_{n=1}^{N}p_n(k_n-\bar{k})(k_n-\bar{k})^\top\right]. \tag*{(34)} \end{aligned}

Restoring the window index and evaluating at qˉg\bar{q}_g gives pj→pg,jp_j\to p_{g,j} and kˉ→kˉg=∑n=1Npg,nkn\bar{k}\to\bar{k}_g=\sum_{n=1}^{N}p_{g,n}k_n. This yields the following closed-form Hessian, which is identical to Eq. (11) in § 3.3.

Hg,j=pg,jd[(kj−kˉg)(kj−kˉg)⊤−∑n=1Npg,n(kn−kˉg)(kn−kˉg)⊤].(35)H_{g,j}=\frac{p_{g,j}}{d}\left[(k_j-\bar{k}_g)(k_j-\bar{k}_g)^\top-\sum_{n=1}^{N}p_{g,n}(k_n-\bar{k}_g)(k_n-\bar{k}_g)^\top\right]. \tag*{(35)}

Contracting Hessian with Query Covariance. We first denote the diagonal entries of Σg\Sigma_g by σg,c2\sigma_{g,c}^2, where c=1,…,dc=1,\ldots,d, which also corresponds to the channel-wise variances as:

σg,c2=1w2∑i=1w2εg,i,c2.(36)\sigma_{g,c}^2=\frac{1}{w^2}\sum_{i=1}^{w^2}\varepsilon_{g,i,c}^2. \tag*{(36)}

Since cross-channel correlations among the query deviations are typically weak, the off-diagonal covariance terms can be neglected. To further reduce sensitivity to channel-wise variance fluctuations, we approximate the covariance with an isotropic form:

Σg≈σg2I,σg2=1d∑c=1dσg,c2.(37)\Sigma_g\approx\sigma_g^2I,\qquad\sigma_g^2=\frac{1}{d}\sum_{c=1}^{d}\sigma_{g,c}^2. \tag*{(37)}

Substituting Eq. (B.23) and Eq. (B.20) into Eq. (B.13), and adopting the identity Tr⁡(xx⊤)=∥x∥22\operatorname{Tr}(xx^\top)=\lVert x\rVert_2^2, the approximated discrepancy gives:

Δg,j≈σg22Tr⁡(Hg,j)=pg,jσg22dTr⁡[(kj−kˉg)(kj−kˉg)⊤−∑n=1Npg,n(kn−kˉg)(kn−kˉg)⊤]=pg,j2d(σg2∥kj−kˉg∥22−∑n=1Npg,nσg2∥kn−kˉg∥22)=pg,j2d(rg,j−rˉg),(38)\begin{aligned} \Delta_{g,j}&\approx\frac{\sigma_g^2}{2}\operatorname{Tr}(H_{g,j})\\ &=\frac{p_{g,j}\sigma_g^2}{2d}\operatorname{Tr}\left[(k_j-\bar{k}_g)(k_j-\bar{k}_g)^\top-\sum_{n=1}^{N}p_{g,n}(k_n-\bar{k}_g)(k_n-\bar{k}_g)^\top\right]\\ &=\frac{p_{g,j}}{2d}\left(\sigma_g^2\lVert k_j-\bar{k}_g\rVert_2^2-\sum_{n=1}^{N}p_{g,n}\sigma_g^2\lVert k_n-\bar{k}_g\rVert_2^2\right)\\ &=\frac{p_{g,j}}{2d}(r_{g,j}-\bar{r}_g), \tag*{(38)} \end{aligned}

where we define:

rg,j=σg2∥kj−kˉg∥22,rˉg=∑n=1Npg,nrg,n.(39)r_{g,j}=\sigma_g^2\lVert k_j-\bar{k}_g\rVert_2^2,\qquad\bar{r}_g=\sum_{n=1}^{N}p_{g,n}r_{g,n}. \tag*{(39)}

Calibrated Importance Score. Revisiting the global calibration target in Eq. (B.7), the approximated window-wise discrepancy in Eq. (B.24) yields the calibrated window-level score:

p~g,j=pg,j+λpg,j2d(rg,j−rˉg),(40)\tilde{p}_{g,j}=p_{g,j}+\lambda\frac{p_{g,j}}{2d}(r_{g,j}-\bar{r}_g), \tag*{(40)}

where λ\lambda controls the correction strength. Finally, the overall calibrated importance score formulates:

s~j=1G∑g=1Gp~g,j.(41)\tilde{s}_j=\frac{1}{G}\sum_{g=1}^{G}\tilde{p}_{g,j}. \tag*{(41)}

This corresponds to Eq. (13) in § 3.3, augmenting the efficient pooled attention with a modest additional second-order correction to improve token discriminability for more reliable pruning.

Derivation of the Fast Scoring Operator

Simplified Window-Wise Correction. We further simplify Eq. (13), equivalently given by Eq. (B.26), in preparation for practical implementation.

First, the centered squared distance rg,jr_{g,j} can be expanded as:

rg,j=σg2∥kj−kˉg∥22=σg2(∥kj∥22−2kjTkˉg+∥kˉg∥22).(42)r_{g,j} = \sigma_g^2 \lVert k_j - \bar{k}_g \rVert_2^2 = \sigma_g^2(\lVert k_j \rVert_2^2 - 2 k_j^\mathsf{T}\bar{k}_g + \lVert\bar{k}_g \rVert_2^2). \tag*{(42)}

Then, given that ∑n=1Npg,n=1\sum_{n=1}^{N} p_{g,n} = 1 and ∑n=1Npg,nkn=kˉg\sum_{n=1}^{N} p_{g,n} k_n = \bar{k}_g, the weighted mean distance rˉg\bar{r}_g is simplified as:

rˉg=∑n=1Npg,nrg,n=σg2(∑n=1Npg,n∥kn∥22−2(∑n=1Npg,nkn)Tkˉg+∥kˉg∥22∑n=1Npg,n)=σg2(∑n=1Npg,n∥kn∥22−∥kˉg∥22).(43)\begin{aligned} \bar{r}_g &= \sum_{n=1}^{N} p_{g,n}r_{g,n} \\ &= \sigma_g^2\left(\sum_{n=1}^{N} p_{g,n}\lVert k_n\rVert_2^2 - 2\left(\sum_{n=1}^{N} p_{g,n}k_n\right)^\mathsf{T}\bar{k}_g + \lVert\bar{k}_g\rVert_2^2\sum_{n=1}^{N}p_{g,n}\right) \\ &= \sigma_g^2\left(\sum_{n=1}^{N} p_{g,n}\lVert k_n\rVert_2^2 - \lVert\bar{k}_g\rVert_2^2\right). \tag*{(43)} \end{aligned}

Subtracting Eq. (B.29) from Eq. (B.28) and substituting the result into Eq. (B.26) ultimately yields:

p~g,j=pg,j[1+λσg22d(∥kj∥22−2kjTkˉg+2∥kˉg∥22−∑n=1Npg,n∥kn∥22)].(44)\tilde{p}_{g,j} = p_{g,j}\left[1+\frac{\lambda\sigma_g^2}{2d}\left(\lVert k_j\rVert_2^2 - 2k_j^\mathsf{T}\bar{k}_g + 2\lVert\bar{k}_g\rVert_2^2 - \sum_{n=1}^{N}p_{g,n}\lVert k_n\rVert_2^2\right)\right]. \tag*{(44)}

This is algebraically equivalent to the TCAM formulation. It eliminates the explicit computation of rˉg\bar{r}_g, requiring only the window-level statistics kˉg\bar{k}_g and ug=∑n=1Npg,n∥kn∥22u_g = \sum_{n=1}^{N}p_{g,n}\lVert k_n\rVert_2^2, both of which can be accumulated online as in Algo. 1.

Exactness of Online Softmax. To establish the basis for the tiled implementation, we first review the exactness of online Softmax. Without loss of generality, we consider a single query qq within window gg and let z={zj}j=1Nz = \{z_j\}_{j=1}^{N} denote its attention logits.

Softmax is invariant to a uniform shift of its logits, i.e. for any scalar cc,

Softmax⁡(z)j=exp⁡(zj−c)∑n=1Nexp⁡(zn−c).(45)\operatorname{Softmax}(z)_j = \frac{\exp(z_j-c)}{\sum_{n=1}^{N}\exp(z_n-c)}. \tag*{(45)}

In particular, choosing c=max⁡jzjc = \max_j z_j improves numerical stability without changing the resulting probabilities. Online Softmax exploits this property by dynamically updating the reference maximum as key blocks are processed.

Specifically, consistent with the notations in Algo. 1, let the key sequence KK be partitioned into TKT_K blocks, each of size BK×dB_K \times d. After processing the first bb blocks, define the running maximum mbm_b and the normalization factor ℓb\ell_b as:

mb=max⁡j=1,…,bBKzj,ℓb=∑j=1bBKexp⁡(zj−mb).(46)m_b = \max_{j=1,\ldots,bB_K} z_j,\qquad\ell_b = \sum_{j=1}^{bB_K}\exp(z_j-m_b). \tag*{(46)}

When processing a new block, the running maximum changes from mb−1m_{b-1} to mbm_b, and all previously accumulated terms can then be exactly rescaled to the new reference maximum using:

α=exp⁡(mb−1−mb),(47)\alpha= \exp(m_{b-1}-m_b), \tag*{(47)}

yielding:

ℓb=αℓb−1+∑j=(b−1)BK+1bBKexp⁡(zj−mb).(48)\ell_b = \alpha\ell_{b-1}+\sum_{j=(b-1)B_K+1}^{bB_K}\exp(z_j-m_b). \tag*{(48)}

This recurrence preserves the exact normalization throughout block-wise accumulation. After the first pass till the final block, the resulting mTKm_{T_K} and ℓTK\ell_{T_K} recover each attention probability as:

pj=exp⁡(zj−mTK)ℓTK=Softmax⁡(z)j.(49)p_j = \frac{\exp(z_j-m_{T_K})}{\ell_{T_K}} = \operatorname{Softmax}(z)_j. \tag*{(49)}

Thus, tiled online computation is algebraically equivalent to standard Softmax, differing only in the accumulation procedure.

Online Accumulation of TCAM Statistics. Given the shift invariance of Eq. (B.31), any statistic aggregated with normalized Softmax weights is likewise invariant to a uniform logit shift, e.g. for arbitrary {aj}j=1N\{a_j\}_{j=1}^{N} and scalar cc,

∑j=1Npjaj=∑j=1Nexp⁡(zj−c)aj∑n=1Nexp⁡(zn−c).(50)\sum_{j=1}^{N} p_j a_j = \frac{\sum_{j=1}^{N}\exp(z_j-c)a_j}{\sum_{n=1}^{N}\exp(z_n-c)}. \tag*{(50)}

Building on this, the two attention-weighted statistics required by TCAM, i.e. kˉg=∑j=1Npg,jkj\bar{k}_g=\sum_{j=1}^{N}p_{g,j}k_j and ∑n=1Npg,n∥kn∥22\sum_{n=1}^{N}p_{g,n}\lVert k_n\rVert_2^2, can be accumulated jointly within the first online pass. Concretely, after processing the first bb key blocks, we maintain the corresponding unnormalized running statistics:

vb=∑j=1bBKexp⁡(zj−mb)kj,ub=∑j=1bBKexp⁡(zj−mb)∥kj∥22.(51)v_b=\sum_{j=1}^{bB_K}\exp(z_j-m_b)k_j,\qquad u_b=\sum_{j=1}^{bB_K}\exp(z_j-m_b)\lVert k_j\rVert_2^2. \tag*{(51)}

Both follow the same rescaling rule when the running maximum changes, i.e.,

vb=αbvb−1+∑j=(b−1)BK+1bBKexp⁡(zj−mb)kj,ub=αbub−1+∑j=(b−1)BK+1bBKexp⁡(zj−mb)∥kj∥22.(52)\begin{aligned} v_b&=\alpha_bv_{b-1}+\sum_{j=(b-1)B_K+1}^{bB_K}\exp(z_j-m_b)k_j,\\ u_b&=\alpha_bu_{b-1}+\sum_{j=(b-1)B_K+1}^{bB_K}\exp(z_j-m_b)\lVert k_j\rVert_2^2. \tag*{(52)} \end{aligned}

After the final block, normalization by ℓTK\ell_{T_K} exactly recovers the required statistics as:

kˉg=vTKℓTK=∑j=1Npg,jkj,∑n=1Npg,n∥kn∥22=uTKℓTK.(53)\bar{k}_g=\frac{v_{T_K}}{\ell_{T_K}}=\sum_{j=1}^{N}p_{g,j}k_j,\qquad\sum_{n=1}^{N}p_{g,n}\lVert k_n\rVert_2^2=\frac{u_{T_K}}{\ell_{T_K}}. \tag*{(53)}

With these finalized statistics, a second traversal over the key blocks recomputes the exact probabilities and directly applies the calibration, yielding the final TCAM scores. Consequently, neither the full attention matrix nor the centered-distance matrix needs to be materialized.

References

  1. [1]Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. Divprune: Diversity-based visual token pruning for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9392–9401, 2025.arxiv.org/abs/2503.02175
  2. [2]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025a.arxiv.org/abs/2511.21631
  3. [3]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025b.arxiv.org/abs/2502.13923
  4. [5]Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In Proceedings of the International Conference on Learning Representations, 2023.arxiv.org/abs/2210.09461
  5. [6]Siyu Cao, Hangting Chen, Peng Chen, Yiji Cheng, Yutao Cui, Xinchi Deng, Ying Dong, Kipper Gong, Tianpeng Gu, Xiusen Gu, et al. Hunyuanimage 3.0 technical report. arXiv preprint arXiv:2509.23951, 2025.arxiv.org/abs/2509.23951
  6. [7]Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. In Proceedings of the International Conference on Machine Learning, volume 235, pp. 6562–6595, 2024a.arxiv.org/abs/2402.04788
  7. [8]Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In Proceedings of the European Conference on Computer Vision, pp. 19–35. Springer, 2024b.arxiv.org/abs/2403.06764
  8. [9]Zhekai Chen, Yuqing Wang, Manyuan Zhang, and Xihui Liu. Macro: Advancing multi-reference image generation with structured long-context data. arXiv preprint arXiv:2603.25319, 2026.arxiv.org/abs/2603.25319
  9. [10]Yisol Choi, Sangkyung Kwak, Kyungmin Lee, Hyungwon Choi, and Jinwoo Shin. Improving diffusion models for authentic virtual try-on in the wild. In Proceedings of the European Conference on Computer Vision, pp. 206–235. Springer, 2024.arxiv.org/abs/2403.05139
  10. [11]Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In Proceedings of the International Conference on Learning Representations, 2024.arxiv.org/abs/2307.08691
  11. [12]Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683, 2025.arxiv.org/abs/2505.14683
  12. [13]Haiwen Diao, Penghao Wu, Hanming Deng, Jiahao Wang, Shihai Bai, Silei Wu, Weichen Fan, Wenjie Ye, Wenwen Tong, Xiangyu Fan, et al. Sensenova-u1: Unifying multimodal understanding and generation with neo-unify architecture. arXiv preprint arXiv:2605.12500, 2026.arxiv.org/abs/2605.12500
  13. [14]Xiangyu Fan, Zesong Qiu, Zhuguanv Wu, Fanzhou Wang, Zhiqian Lin, Tianxiang Ren, Dahua Lin, Ruihao Gong, and Lei Yang. Phased dmd: Few-step distribution matching distillation via score matching within subintervals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 41667–41676, 2026.arxiv.org/abs/2510.27684
  14. [15]Haipeng Fang, Sheng Tang, Juan Cao, Enshuo Zhang, Fan Tang, and Tong-Yee Lee. Attend to not attended: Structure-then-detail token merging for post-training dit acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18083–18092, 2025.arxiv.org/abs/2505.11707
  15. [16]Google DeepMind. Gemini 3 flash: Model card. Google DeepMind Model Card, 2025. URL https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-Flash-Model-Card.pdf.
  16. [17]Xin He, Longhui Wei, Jianbo Ouyang, Minghui Liao, Lingxi Xie, and Qi Tian. Emma: Efficient multimodal understanding, generation, and editing with a unified architecture. arXiv preprint arXiv:2512.04810, 2025.arxiv.org/abs/2512.04810
  17. [18]Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS Workshop on Deep Generative Models and Downstream Applications, 2021.arxiv.org/abs/2207.12598
  18. [19]Junlong Ke, Zichen Wen, Boxue Yang, Yantai Yang, Xuyang Liu, Chenfei Liao, Zhaorun Chen, Shaobo Wang, and Linfeng Zhang. Flash-unified: A training-free and task-aware acceleration framework for native unified models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Findings, pp. 9131–9142, 2026.arxiv.org/abs/2603.15271
  19. [20]Junqing Lin, Xingyu Zheng, Pei Cheng, Bin Fu, Jingwei Sun, and Guangzhou Sun. Token pruning for in-context generation in diffusion transformers. arXiv preprint arXiv:2602.01609, 2026.arxiv.org/abs/2602.01609
  20. [21]Haotian Liu, ChunYuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26286–26296, 2024.arxiv.org/abs/2310.03744
  21. [22]Jiacheng Liu, Chang Zou, Yuanhuiyi Lyu, Junjie Chen, and Linfeng Zhang. From reusing to forecasting: Accelerating diffusion models with taylorsers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15853–15863, 2025.arxiv.org/abs/2503.06923
  22. [23]Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF international conference on computer vision (ICCV), pp. 9992–10002. IEEE, 2021.arxiv.org/abs/2103.14030
  23. [24]Yinyi Luo, Wenwen Wang, Hayes Bai, Hongyu Zhu, Hao Chen, Pan He, Marios Savvides, Sharon Li, and Jindong Wang. Torchumm: A unified multimodal model codebase for evaluation, analysis, and post-training. arXiv preprint arXiv:2604.10784, 2026.arxiv.org/abs/2604.10784
  24. [25]Yuta Oshima, Daiki Miyake, Kohsei Matsutani, Yusuke Iwasawa, Masahiro Suzuki, Yutaka Matsuo, and Hiroki Furuta. Multibanana: A challenging benchmark for multi-reference text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 448–460, 2026.arxiv.org/abs/2511.22989
  25. [26]William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195–4205, 2023.arxiv.org/abs/2212.09748
  26. [27]Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference. In Findings of the Association for Computational Linguistics, pp. 4065–4078, 2024.arxiv.org/abs/2406.18139
  27. [28]Xierui Wang, Siming Fu, Qihan Huang, Wanggui He, and Hao Jiang. Ms-diffusion: Multi-subject zero-shot image personalization with layout guidance. In Proceedings of the International Conference on Learning Representations, volume 2025, pp. 95118–95146, 2025.arxiv.org/abs/2406.07209
  28. [29]Xinyu Wei, Kangrui Cen, Hongyang Wei, Zhen Guo, Bairui Li, Zeqing Wang, Jinrui Zhang, and Lei Zhang. Mico-150k: A comprehensive dataset advancing multi-image composition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 29695–29706, 2026.arxiv.org/abs/2512.07348
  29. [30]Chenyuan Wu, Jiahao Wang, Pengfei Zheng, et al. Omnigen2: Towards instruction-aligned multimodal generation. arXiv preprint arXiv:2506.18871, 2025a.arxiv.org/abs/2506.18871
  30. [31]Zimeng Wu, Jiaxin Chen, and Yunhong Wang. Unified knowledge maintenance pruning and progressive recovery with weight recalling for large vision-language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 8550–8558, 2025b.DOI
  31. [32]Zimeng Wu, Donghao Wang, Chaozhe Jin, Jiaxin Chen, and Yunhong Wang. Probe and skip: Self-predictive token skipping for efficient long-context llm inference. arXiv preprint arXiv:2601.13155, 2026a.arxiv.org/abs/2601.13155
  32. [33]Zimeng Wu, Yunhong Wang, Donghao Wang, and Jiaxin Chen. Collaborative multi-mode pruning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 39561–39571, 2026b.arxiv.org/abs/2604.02956
  33. [34]Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19792–19802, 2025.arxiv.org/abs/2412.04467
  34. [35]Linli Yao, Long Xing, Yang Shi, Sida Li, Yuanxin Liu, Yuhao Dong, Yi-Fan Zhang, Lei Li, Qingxiu Dong, Xiaoyi Dong, et al. Towards efficient multimodal large language models: A survey on token compression. 2026.DOI
  35. [36]Jintao Zhang, Chendong Xiang, Haofeng Huang, Haocheng Xi, Jun Zhu, Jianfei Chen, et al. Spargeattention: Accurate and training-free sparse attention accelerating any model inference. In Proceedings of the International Conference on Machine Learning, 2025a.arxiv.org/abs/2502.18137
  36. [37]Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis A Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparselvm: Visual token sparsification for efficient vision-language model inference. In Proceedings of the International Conference on Machine Learning, 2025b.
  37. [38]Qicheng Zhao, Qi Sun, and Zheyu Yan. Seeing the end at step zero: Accelerating diffusion mllms via mlp sparsity-aware truncation. arXiv preprint arXiv:2607.14557, 2026.arxiv.org/abs/2607.14557
  38. [39]Shanshan Zhao, Xinjie Zhang, Jintao Guo, Jiakui Hu, Lunhao Duan, Minghao Fu, Yong Xien Chng, Guo-Hua Wang, Qing-Guo Chen, Zhao Xu, et al. Unified multimodal understanding and generation models: Advances, challenges, and opportunities. arXiv preprint arXiv:2505.02567, 2025.arxiv.org/abs/2505.02567

Paper details

vlnc:W_ustkt279
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents