Introduction
In recent years, video diffusion transformers have been applied to a wide range of generation tasks, from text-to-video synthesis [34 ] to controlled generation [24 ] and neural rendering [15 ] , gradually moving from many-step samplers toward few-step distilled generators [6 , 44 ] . Sparse attention makes these generators fast without retraining: since attention grows quadratically with video length, it computes a few key blocks exactly and compensates for the rest, keeping near-dense fidelity (Figure 1 ). However, existing methods tie compensation to block selection: the groups that pick the exact blocks also summarize the skipped keys, so compensation is costly at every step of a few-step sampler and impossible once the selection is fixed offline.
Figure 1. DecoupAttn keeps dense-attention quality at a lower latency. Frames from dense attention and DecoupAttn on Cosmos-Predict2.5 (4 steps) and HunyuanVideo (50 steps), with latency, speedup on all ten models (filled: these two), and PSNR and VBench against speedup for DecoupAttn and the seven baselines of Table 1, averaged over the ten models.
Sparse attention for video generation has followed three main lines of work, as shown in Figure 2 . First, online routing (dynamic sparse attention) identifies critical blocks at every step from the current queries and keys, by scoring pooled blocks [39 , 45 ] or clustering tokens by semantics [33 , 41 ] , treating sparsity as a property of each input. Second, offline calibration (static sparse attention) fixes sparse patterns before deployment, measured on a few prompts [43 ] or derived from how attention decays with distance [13 ] , treating sparsity as a property of the model. Third, compensation approximates the skipped blocks instead of dropping them, from cluster centroids [53 ] , block-wise Taylor expansions [12 ] , or fitted residuals [32 ] , recov ering much of the attention that selection leaves out.
Figure 2. Three lines of sparse attention and DecoupAttn. (a) Online routing re-picks exact blocks each step and drops the rest, (b) offline calibration fixes them but drops the rest, and (c) compensation reuses them and reroutes every step. (d) DecoupAttn fixes them offline and compensates every key by semantic clusters. Right: schematic error and time against DecoupAttn (dashed), excess in red.
However, these methods still face three challenges. (i) Identification overhead. Online routing and the compensation built on its groups identify critical blocks at every denoising step [11 , 12 , 17 , 32 , 33 , 37 , 39 , 40 , 41 , 45 , 48 , 49 , 53 ] , an overhead that can offset the gains of sparsity in few-step samplers, while static patterns fixed in advance [13 , 43 , 46 ] discard the remaining blocks and incur information loss. (ii) Suboptimal budget allocation. Existing methods set sparsity budgets across heads, layers, and timesteps uniformly or by local metrics such as retained attention mass [20 , 41 , 43 , 48 , 53 ] , or by the effect of dropping blocks on the output [38 , 51 , 52 ] . Retained attention mass, however, does not determine the output error [32 ] , which depends on the residual left after compensation and on how unevenly later layers and steps amplify it, so exact computation is spent where it barely changes the final video. (iii) Cross-step error accumulation. To avoid redundant recomputation, recent methods reuse masks, clusters, or earlier centroids across steps, or predict masks ahead [2 , 10 , 16 , 18 , 20 , 29 , 40 , 41 ] , but attention patterns evolve across denoising steps, so the reused groups go stale and their errors accumulate, while refreshing them repeats the identification.
Our key observation is that once every key is compensated, the residuals of the skipped blocks point in many directions and cancel each other like dithered quantization errors [27 ] , so blocks fixed once offline match blocks reselected at every step. Based on this observation, we propose DecoupAttn, a new paradigm of decoupled sparse attention: semantic clusters compensate every key, and exact attention corrects their residual on a few blocks fixed once offline by position (Figure 2 d). Selecting blocks costs nothing at inference, and the residual of every fixed block can be priced offline. Few-step samplers amplify this residual unevenly, so we introduce Distortion Budget, a rate–distortion objective for budget allocation that prices every timestep, layer, and head by the distortion the residual leaves in the final latent. Built on this budget, we further propose Reassign Cache, which limits the growth of these residuals by reassigning the drifting keys to cached centroids at every step, while the blocks and the budget stay fixed. DecoupAttn thus moves block identification and budget allocation into one plan compiled offline with no training, and inference only refreshes the clusters, running standard block-sparse kernels and one fused compensation kernel.
We evaluate DecoupAttn on ten video generation, controlled generation, and neural rendering models, where it outperforms online-routing, offline-calibrated, and compensation-based methods on nine of them and delivers a substantial speedup over dense attention.
Related Work
Summarizing skipped attention. Transformers summarize attention by clusters, near- and far-field splits, or sparse-plus-low-rank estimators [1 , 14 , 21 , 25 , 35 , 50 ] , and video sparse attention compensates the skipped keys by cluster or block means [17 , 53 ] , Taylor terms [12 , 48 ] , or fitted or cached residuals [2 , 32 , 40 ] . Masks frozen early or predicted ahead [16 , 18 ] drop the keys outside them, and routers that reuse centroids [10 , 20 , 41 ] still let the clusters pick the blocks. DecoupAttn summarizes the skipped keys by semantic clusters that never pick blocks.
Allocating computation by sensitivity. Coders and quantizers allocate bits by rate–distortion cost or layer sensitivity [3 , 30 , 31 ] , and sparse attention sets density by retained attention mass, profiling, sampler curvature, or block error [20 , 41 , 43 , 48 , 53 ] , or by each unit’s effect on the output [38 , 51 , 52 ] . Distortion Budget instead prices the residual left after compensation, not the error of dropping a block, by its effect on the final latent.
Preliminaries
Definition 1: Block-sparse attention call. A self-attention call at denoising step t t t , layer l l l , and head h h h maps the queries q i q_i q i , keys k j k_j k j , and values v j v_j v j of its L L L tokens to outputs y i y_i y i , with the tokens partitioned by position into query tiles g g g and key blocks b b b , the units of block-sparse kernels [42 ] :
y i = N i Z i = ∑ b N i b ∑ b Z i b , N i b = ∑ j ∈ b e s i j v j , Z i b = ∑ j ∈ b e s i j (1) y_i = \frac{N_i}{Z_i} = \frac{\sum_b N_{ib}}{\sum_b Z_{ib}}, \qquad N_{ib} = \sum_{j \in b} e^{s_{ij}}v_j, \qquad Z_{ib} = \sum_{j \in b} e^{s_{ij}} \tag*{(1)} y i = Z i N i = ∑ b Z ib ∑ b N ib , N ib = j ∈ b ∑ e s ij v j , Z ib = j ∈ b ∑ e s ij (1)
where s i j = q i T k j / d s_{ij}=q_i^\mathsf{T}k_j/\sqrt{d} s ij = q i T k j / d and d d d is the head dimension. A sparse call computes N i b N_{ib} N ib and Z i b Z_{ib} Z ib exactly on a block set S g S_g S g and summarizes the rest. Existing compensation writes N ^ i = ∑ b ∈ S g N i b + ∑ b ∉ S g N ~ i b \hat{N}_i=\sum_{b\in S_g}N_{ib}+\sum_{b\notin S_g}\tilde{N}_{ib} N ^ i = ∑ b ∈ S g N ib + ∑ b ∈ / S g N ~ ib with N ~ i b \tilde{N}_{ib} N ~ ib from the statistics of block b b b itself, and every change of S g S_g S g rebuilds the compensation terms [12 , 53 ] .
Definition 2: Sparse attention configuration. A sampler runs T T T denoising steps. Each head of each call is a unit u = ( t , l , h ) u=(t,l,h) u = ( t , l , h ) of the unit set U U U , and the heads of a call ( t , l ) (t,l) ( t , l ) share one compensation estimator. A configuration, or plan, π = ( { o u } u ∈ U , { η t l } ) \pi=(\{o_u\}_{u\in U},\{\eta_{tl}\}) π = ({ o u } u ∈ U , { η tl }) assigns every unit an option and every call an estimator, which determines the final latent and the density:
π ⇒ x ^ 0 ( π ) , X ( π ) = ∑ u w u X u ( o u ) ∑ u w u , (2) \pi\Rightarrow\hat{x}_0(\pi), \qquad X(\pi)=\frac{\sum_u w_uX_u(o_u)}{\sum_u w_u}, \tag*{(2)} π ⇒ x ^ 0 ( π ) , X ( π ) = ∑ u w u ∑ u w u X u ( o u ) , (2)
where o u ∈ O o_u\in\mathcal{O} o u ∈ O is the unit’s option (Eq. (8) ), η t l ∈ { F U L L , C E N T R O I D } \eta_{tl}\in\{\mathrm{FULL},\mathrm{CENTROID}\} η tl ∈ { FULL , CENTROID } the call’s estimator (Eq. (4) ), X u X_u X u the density of an option, the fraction of blocks it executes with every block padded to the kernel block size and the DENSE option counted as one, and w u w_u w u the number of sampler steps the unit represents. The first step and the first layer always stay dense and lie outside U U U , and the estimator adds a fixed overhead to X X X (Appendix § 8 .9).
Problem statement. Given a pretrained video diffusion transformer with frozen weights and a few calibration clips, we seek a plan that is fixed before deployment, that keeps the density within a target ρ \rho ρ , and that keeps the final latent closest to the dense one:
π ⋆ = arg min π E x ∥ x ^ 0 ( π ) − x 0 ∥ 2 2 s . t . X ( π ) ≤ ρ , (3) \pi^\star=\underset{\pi}{\arg\min}\ \mathbb{E}_x\lVert\hat{x}_0(\pi)-x_0\rVert_2^2 \quad\mathrm{s.t.}\quad X(\pi)\leq\rho, \tag*{(3)} π ⋆ = π arg min E x ∥ x ^ 0 ( π ) − x 0 ∥ 2 2 s.t. X ( π ) ≤ ρ , (3)
where x x x is the input (prompt, conditions, and seed) and x 0 x_0 x 0 the dense final latent for the same input.
Method
As Figure 3 shows, DecoupAttn centers on one quantity, the residual that compensation leaves on each key block: the decoupled estimator makes this residual the only error that reaches the output and keeps it small (§ 4.1 ), Distortion Budget spends exact computation where the residual most distorts the final latent (§ 4.2 ), and Reassign Cache limits its growth as keys drift (§ 4.3 ).
Figure 3. How DecoupAttn computes one attention call (Cosmos-Predict2.5). (a) Reassign Cache clusters the keys once, caches the centroids, and at every later step moves each key to its nearest cached centroid and refreshes the cluster statistics, (b) Distortion Budget prices the options of every unit offline and keeps the cheapest, which fixes the exact blocks S g S_g S g , and (c) exact attention on S g S_g S g and centroid compensation of every other key meet only through the counts n g c n_{gc} n g c and share one softmax denominator (pink: approximation, blue: exact).
Decoupled Sparse Attention
To keep compensation when the selection is fixed offline, decoupled sparse attention follows the difference estimators of survey sampling [26 ] : a cheap algorithm predicts the contribution of every key, and exact attention on a few blocks replaces the prediction there (Figure 3 c).
Centroid compensation with two corrections. At the first sparse step, the keys of every head are clustered by k-means into clusters c c c , and every later call summarizes each cluster by its centroid with two second-order corrections computed from the cluster’s current members:
ω i c = ϕ i c ( 1 + γ i c ) , a i c = ω i c v ˉ c + ϕ i c H q i d , (4) \omega_{ic}=\phi_{ic}(1+\gamma_{ic}), \qquad a_{ic}=\omega_{ic}\bar{v}_c+\phi_{ic}\frac{Hq_i}{\sqrt{d}}, \tag*{(4)} ω i c = ϕ i c ( 1 + γ i c ) , a i c = ω i c v ˉ c + ϕ i c d H q i , (4)
where ϕ i c = e q i T k ˉ c / d \phi_{ic}=e^{q_i^\mathsf{T}\bar{k}_c/\sqrt{d}} ϕ i c = e q i T k ˉ c / d is the centroid weight, and k ˉ c \bar{k}_c k ˉ c , v ˉ c \bar{v}_c v ˉ c , and σ c 2 \sigma_c^2 σ c 2 are the cluster’s mean key, mean value, and per-dimension key variance. The variance correction γ i c = 1 2 d ∑ p q i p 2 σ c p 2 \gamma_{ic}=\frac{1}{2d}\sum_p q_{ip}^2\sigma_{cp}^2 γ i c = 2 d 1 ∑ p q i p 2 σ c p 2 is the diagonal part of the second-order term of e q i T k / d e^{q_i^\mathsf{T}k/\sqrt{d}} e q i T k / d around k ˉ c \bar{k}_c k ˉ c and restores this part of the mass that a centroid misses on a spread-out cluster (Lemmas A.3 and A.4). The within-cluster cross-covariance H = 1 L ( ∑ j v j k j T − ∑ c ∣ c ∣ v ˉ c k ˉ c T ) H=\frac{1}{L}\left(\sum_jv_jk_j^\mathsf{T}-\sum_c|c|\bar{v}_c\bar{k}_c^\mathsf{T}\right) H = L 1 ( ∑ j v j k j T − ∑ c ∣ c ∣ v ˉ c k ˉ c T ) , pooled over the clusters of a head, gives their shared first-order value correction (Lemma A.5). The CENTROID estimator drops both corrections.
Decoupled estimator. Each query tile g g g computes its exact blocks S g S_g S g in full and covers every other key with its cluster centroid, and both parts share one softmax denominator:
y ^ i = N ^ i Z ^ i = ∑ b ∈ S g N i b + ∑ c n g c a i c ∑ b ∈ S g Z i b + ∑ c n g c ω i c , (5) \hat{y}_i=\frac{\hat{N}_i}{\hat{Z}_i}=\frac{\sum_{b\in S_g}N_{ib}+\sum_c n_{gc}\,a_{ic}}{\sum_{b\in S_g}Z_{ib}+\sum_c n_{gc}\,\omega_{ic}}, \tag*{(5)} y ^ i = Z ^ i N ^ i = ∑ b ∈ S g Z ib + ∑ c n g c ω i c ∑ b ∈ S g N ib + ∑ c n g c a i c , (5)
where a i c a_{ic} a i c and ω i c \omega_{ic} ω i c are what one member of cluster c c c contributes to the numerator and the denominator (Eq. (4) ). The count n g c = ∣ c ∣ − ∑ b ∈ S g n b c n_{gc}=|c|-\sum_{b\in S_g}n_{bc} n g c = ∣ c ∣ − ∑ b ∈ S g n b c , with n b c = ∣ b ∩ c ∣ n_{bc}=|b\cap c| n b c = ∣ b ∩ c ∣ , keeps only the members of c c c that the exact blocks do not already cover. The exact blocks are fixed offline from position and calibration statistics, the clusters come from the keys alone, and the counts n g c n_{gc} n g c are the only place the two meet, so no block identification runs at inference.
Compensate every key, correct some blocks. Substituting n g c n_{gc} n g c into Eq. (5) splits the estimate into two steps. First, the centroids compensate all keys as if no block were exact. Then, on each exact block, the exact value replaces the centroids’ prediction, as in Scatterbrain and EVA [1 , 50 ] :
N ^ i = ∑ c ∣ c ∣ a i c + ∑ b ∈ S g Δ N i b , y ^ i − y i = − 1 Z ^ i ∑ b ∉ S g ( Δ N i b − y i Δ Z i b ) (6) \widehat{N}_i=\sum_c |c|a_{ic}+\sum_{b\in S_g}\Delta N_{ib},
\qquad
\widehat{y}_i-y_i=-\frac{1}{\widehat{Z}_i}\sum_{b\notin S_g}\left(\Delta N_{ib}-y_i\Delta Z_{ib}\right) \tag*{(6)} N i = c ∑ ∣ c ∣ a i c + b ∈ S g ∑ Δ N ib , y i − y i = − Z i 1 b ∈ / S g ∑ ( Δ N ib − y i Δ Z ib ) (6)
where Δ N i b = N i b − ∑ c n b c a i c \Delta N_{ib}=N_{ib}-\sum_c n_{bc}a_{ic} Δ N ib = N ib − ∑ c n b c a i c and Δ Z i b = Z i b − ∑ c n b c ω i c \Delta Z_{ib}=Z_{ib}-\sum_c n_{bc}\omega_{ic} Δ Z ib = Z ib − ∑ c n b c ω i c are the gaps between exact value and centroid prediction on block b b b , the residual r i b r_{ib} r ib is the error block b b b passes to the output, and skipped blocks leave error.
Why fixed blocks suffice. By Eq. (6) , the error is a sum of block residuals, so its size depends on the estimated mass, on how large the residuals are, and on how they align:
∥ y ^ i − y i ∥ 2 2 = 1 Z ^ i 2 ∥ ∑ b ∉ S g r i b ∥ 2 2 = κ i R i Z ^ i 2 , (7) \left\lVert\widehat{y}_i-y_i\right\rVert_2^2
=\frac{1}{\widehat{Z}_i^2}\left\lVert\sum_{b\notin S_g}r_{ib}\right\rVert_2^2
=\frac{\kappa_iR_i}{\widehat{Z}_i^2}, \tag*{(7)} ∥ y i − y i ∥ 2 2 = Z i 2 1 b ∈ / S g ∑ r ib 2 2 = Z i 2 κ i R i , (7)
where R i = ∑ b ∉ S g ∥ r i b ∥ 2 2 R_i=\sum_{b\notin S_g}\lVert r_{ib}\rVert_2^2 R i = ∑ b ∈ / S g ∥ r ib ∥ 2 2 is the residual energy of the skipped blocks, and the coherence ratio κ i = ∥ ∑ b ∉ S g r i b ∥ 2 2 / R i \kappa_i=\left\lVert\sum_{b\notin S_g}r_{ib}\right\rVert_2^2/R_i κ i = ∑ b ∈ / S g r ib 2 2 / R i , set to one when R i = 0 R_i=0 R i = 0 , is one for mutually orthogonal residuals and grows as they align. Dropped blocks each remove their whole attention and push the output the same way, whereas a compensated residual comes from its members’ deviations from their centroids, which sum to zero over the cluster, and the residuals of the skipped blocks cancel each other like dithered quantization errors [27 ] (Theorem A.10). With Z ^ i ≈ Z i \widehat{Z}_i\approx Z_i Z i ≈ Z i (Corollary A.2) and κ i \kappa_i κ i kept small, the residual energy R i R_i R i is what Sections 4.2 and 4.3 then control.
Proposition 1. For count-based compensation, decoupled sparse attention equals compensating every key and correcting the residual of each exact block, and its error is determined by the summed residual of the other blocks for any blocks and clusters, while for blocks fixed offline the deviation parts of these residuals add like random errors and nearly cancel each other. (Proof in Appendices A.2 and A.4, evidence in § 5.3 .)
Distortion Budget
Section 4.1 leaves the residual energy R i R_i R i to control, and by Proposition 1 a block’s residual does not depend on which other blocks are exact (Appendix A.2). Distortion Budget measures each residual once on the calibration set and solves Eq. (3) as a rate–distortion bit allocation [30 ] : units act as quantizers, options as their resolutions, density as the rate, and final-latent error as the distortion (Figure 3 b).
Options. Each unit u = ( t , l , h ) u=(t,l,h) u = ( t , l , h ) (Definition 2) picks one of three kinds of option, which differ in how many key blocks it computes exactly. Dense computes every block, as dense attention does. Staticθ _\theta θ computes the fraction θ \theta θ of key blocks that rank highest for the query tile and compensates the rest. CompOnly computes only the diagonal block and compensates everything else, the analog of the skip mode in rate–distortion mode decision [31 ] . The option set and the exact blocks of Staticθ _\theta θ are
O = { D e n s e , C o m p O n l y } ∪ { S t a t i c θ } θ ∈ Θ , \mathcal{O}=\{\mathrm{Dense},\mathrm{CompOnly}\}\cup\{\mathrm{Static}_\theta\}_{\theta\in\Theta}, O = { Dense , CompOnly } ∪ { Static θ } θ ∈ Θ ,
S g ( S t a t i c θ ) = { b g } ∪ topK θ − 1 ( D u [ g , ⋅ ] ) , (8) S_g(\mathrm{Static}_\theta)=\{b_g\}\cup\operatorname{topK}_{\theta-1}\left(D_u[g,\cdot]\right), \tag*{(8)} S g ( Static θ ) = { b g } ∪ topK θ − 1 ( D u [ g , ⋅ ] ) , (8)
where Θ \Theta Θ is the set of density levels (Appendix B), K θ = max ( 1 , ⌈ θ N B ⌉ ) K_\theta=\max(1,\lceil\theta N_B\rceil) K θ = max ( 1 , ⌈ θ N B ⌉) for N B N_B N B key blocks, D u [ g , ⋅ ] D_u[g,\cdot] D u [ g , ⋅ ] the ranking of the other key blocks for query tile g g g (Eq. (9) ), and b g b_g b g its diagonal block, which every option computes. The ranking orders these blocks by how badly compensation handles them, measured by their residual in Eq. (6) on the calibration set with the model run densely:
D u [ q , b ] = E x E i ∈ g ∥ r i b ∥ 2 2 Z i 2 ∥ y i ∥ 2 2 , (9) D_u[q,b] = \mathbb{E}_{x}\mathbb{E}_{i\in g}\frac{\lVert r_{ib}\rVert_2^2}{Z_i^2\lVert y_i\rVert_2^2}, \tag*{(9)} D u [ q , b ] = E x E i ∈ g Z i 2 ∥ y i ∥ 2 2 ∥ r ib ∥ 2 2 , (9)
where r i b r_{ib} r ib is taken under full compensation, and dividing by Z i 2 ∥ y i ∥ 2 2 Z_i^2\lVert y_i\rVert_2^2 Z i 2 ∥ y i ∥ 2 2 puts large and small outputs on one scale.
Distortion of an option. Distortion Budget measures, unit by unit on dense calibration runs, the projected error of the head under every option o o o and estimator η \eta η :
ε u ( o , η ) = E x ⟨ M h , ( O ^ u − O u ) T ( O ^ u − O u ) ⟩ , \varepsilon_u(o,\eta)=\mathbb{E}_x\left\langle M_h,(\widehat{O}_u-O_u)^{\mathsf{T}}(\widehat{O}_u-O_u)\right\rangle, ε u ( o , η ) = E x ⟨ M h , ( O u − O u ) T ( O u − O u ) ⟩ ,
M h = W o , h T W o , h , (10) M_h=W_{o,h}^{\mathsf{T}}W_{o,h}, \tag*{(10)} M h = W o , h T W o , h , (10)
where O u O_u O u and O ^ u \widehat{O}_u O u are the head’s dense output and its output under o o o and η \eta η (Eq. (5) ), the expectation runs over calibration inputs x x x , the inner product is the squared error after projection, ∥ ( O ^ u − O u ) W o , h T ∥ F 2 \lVert(\widehat{O}_u-O_u)W_{o,h}^{\mathsf{T}}\rVert_F^2 ∥( O u − O u ) W o , h T ∥ F 2 , and W o , h W_{o,h} W o , h is the head’s slice of the output projection into the residual stream.
Sensitivity. The distortion of each unit is further scaled by a sensitivity J u J_u J u , the squared gain from the head to the final latent through the remaining layers and sampler steps (Appendix § 8 .10), which plays the role of Hessian sensitivity in mixed-precision quantization [3 ] . To reduce the number of measurements, units are measured in groups G \mathcal{G} G , each covering a range of steps and of layers (Appendix § 9 ): the units of G \mathcal{G} G run sparse at a reference level θ 0 \theta_0 θ 0 with full compensation while the rest stays dense, so the injected errors are the real sparse errors, and the extra final-latent error is divided by the group’s summed distortion:
J u = MSE G − MSE 0 ∑ u ′ ∈ G w u ′ ε u ′ ( STATIC θ 0 , FULL ) , (11) J_u=\frac{\operatorname{MSE}_{\mathcal{G}}-\operatorname{MSE}_0}{\sum_{u'\in\mathcal{G}}w_{u'}\varepsilon_{u'}(\operatorname{STATIC}_{\theta_0},\operatorname{FULL})}, \tag*{(11)} J u = ∑ u ′ ∈ G w u ′ ε u ′ ( STATIC θ 0 , FULL ) MSE G − MSE 0 , (11)
where MSE G \operatorname{MSE}_{\mathcal{G}} MSE G is the final-latent error of this run against the dense output, and the subtracted MSE 0 \operatorname{MSE}_0 MSE 0 is the run-to-run error of the same dense attention backend. This reference gain prices every option of the unit (Lemma A.17).
Allocation. With a price J u ε u J_u\varepsilon_u J u ε u for every unit and option, choosing the plan becomes a budgeting problem under the density target ρ \rho ρ :
min π ∑ u w u J u ε u ( o u , η t l ) s.t. X ( π ) ≤ ρ , \min_{\pi}\sum_u w_uJ_u\varepsilon_u(o_u,\eta_{t_l})\quad\text{s.t.}\quad X(\pi)\le\rho, π min u ∑ w u J u ε u ( o u , η t l ) s.t. X ( π ) ≤ ρ ,
L μ ( π ) = ∑ u w u ( J u ε u ( o u , η t l ) + μ X u ( o u ) ) , (12) \mathcal{L}_{\mu}(\pi)=\sum_u w_u\left(J_u\varepsilon_u(o_u,\eta_{t_l})+\mu X_u(o_u)\right), \tag*{(12)} L μ ( π ) = u ∑ w u ( J u ε u ( o u , η t l ) + μ X u ( o u ) ) , (12)
where the first line approximates Eq. (3) (Proposition 2). The second line is the Lagrangian relaxation of Everett [4 ] , which splits the search over plans into one choice per call: under each estimator every head takes the option with the lowest J u ε u + μ X u J_u\varepsilon_u+\mu X_u J u ε u + μ X u , and the call keeps the lower-total estimator. Bisection on μ \mu μ finds the densest plan that still fits ρ \rho ρ , and a final greedy pass spends the density left over.
Proposition 2. The summed priced distortion approximates the final-latent error, and for every nonnegative multiplier, the plan that minimizes L μ \mathcal{L}_{\mu} L μ has the lowest priced distortion among all plans that are no denser, so bisection returns the best plan at its own density. (Proof in Appendix § 8 .7, evidence in § 5.3 .)
Reassign Cache
The plan prices the residuals of clusters built at the first sparse step, and a cluster’s residual is of second order in the spread of its members around the centroid (Proposition A.6). As keys drift between denoising steps, the spread grows and the residual outgrows what the plan priced. Reassign Cache bounds this spread at every step at little cost (Figure 3 a).
Cached centroids. We treat the centroids of the first sparse step as a fixed codebook and, at every later step, run the assignment step of Lloyd’s algorithm [19 ] : each key moves to the cluster of its nearest cached centroid, and the statistics of Eq. (4) are recomputed from the current members:
c t ( j ) = arg min c ∥ k j ( t ) − m c ∥ 2 2 , m c = k ˉ c ( t 1 ) , (13) c_t(j)=\arg\min_c\left\lVert k_j^{(t)}-m_c\right\rVert_2^2,\qquad m_c=\bar{k}_c^{(t_1)}, \tag*{(13)} c t ( j ) = arg c min k j ( t ) − m c 2 2 , m c = k ˉ c ( t 1 ) , (13)
where t 1 t_1 t 1 is the first sparse step, k j ( t ) k_j^{(t)} k j ( t ) is key j j j at step t t t , and m c m_c m c is the cached centroid of cluster c c c . One call costs one key–centroid product and an argmin (Appendix § 8 .9).
Assignment rule. One measurement on the calibration set decides whether reassignment pays:
reassign = 1 [ δ s t a l e > τ s ] , (14) \operatorname{reassign}=\mathbb{1}\left[\delta_{\mathrm{stale}}>\tau_s\right], \tag*{(14)} reassign = 1 [ δ stale > τ s ] , (14)
where 1 [ ⋅ ] \mathbb{1}[\cdot] 1 [ ⋅ ] is the indicator function, δ s t a l e \delta_{\mathrm{stale}} δ stale is the PSNR lost when the cluster assignments of the first sparse step stay frozen, and τ s \tau_s τ s is a threshold (Appendix § 9 ).
Proposition 3. Reassigning the keys at a step gives the lowest total squared distance to the cached centroids of any assignment, and it changes only the compensation: the exact blocks and every unit’s budget stay as planned. (Proof in Appendix § 8 .8, evidence in § 5.3 .)
Experiments
We ask whether DecoupAttn is more accurate than prior sparse attention at equal density yet faster than dense attention (RQ1), whether one procedure fits all families and samplers (RQ2), what each component adds (RQ3), the quality–speed trade-off (RQ4), and what the plan costs (RQ5).
Experimental Setup
Models. We test ten models in three families (Table 5 ). Video generation covers Cosmos-Predict1 (CP1) [23 ] , Cosmos-Predict2.5 (CP2.5) [24 ] , Wan2.1-14B at 480p (Wan2.1) and Wan2.2-A14B (Wan2.2) [34 ] , HunyuanVideo (HY) [9 ] , and image-to-video Cosmos3-Nano (Nano) [22 ] .
Table 5. Models, samplers, and test splits. Calibration, validation, and test clips are pairwise disjoint.
Controlled generation uses edge-conditioned Cosmos-Transfer2.5 (T2.5) [24 ] , and neural rendering uses DiffusionRenderer (DR) [15 ] , AlayaRenderer (Alaya) [8 ] , and UniRelight (UniRel.) [5 ] . Five run four-step distilled samplers and five their original many-step samplers. Each model has disjoint calibration (at most 16 clips), validation (8), and test (8–39) splits, all with seed 1000.
Baselines. We compare with seven methods from three lines of work, all run from their official code: offline calibration (CalibAtt [43 ] ), online routing by clustering (SVOO [20 ] , SVG2 [41 ] ) or by block scoring (Sol-Attn [11 ] ), and compensation-based sparse attention (SVG-EAR [53 ] , PISA [12 ] , SparsePR [32 ] ). For a fair comparison, every method keeps the first step and layer dense, runs the other calls at density 0.20 or its official minimum (Appendix § 10 , 0.23–0.42 with padding in Table 14 ), and is scored and timed against the same dense outputs in one process.
Table 14. Density accounting of the deployed plans. Shares of attention work are weighted by sampler steps; computed blocks count 128×96 kernel blocks per video, in millions.
Metrics and implementation. Quality is the PSNR of each video against the dense output with the same seed, together with SSIM [36 ] and LPIPS [47 ] and six VBench dimensions [7 ] (Appendix § 10 ), and speed is the latency of the denoising loop (DiT) and its speedup over dense attention. DecoupAttn sparsifies the spatiotemporal self-attention of every model and keeps cross-attention dense. Exact blocks run on FlashInfer’s variable-block FlashAttention-3 kernel [28 , 42 ] , and one kernel merges the cluster terms (Appendix § 9 ).
Main Results
DecoupAttn is accurate and fast at once (RQ1). As shown in Table 1 , DecoupAttn has the highest PSNR on nine of the ten models, 2.18 dB above the strongest baseline on average, while running 1.31× faster than dense attention and up to 1.65×. Among the methods faster than dense attention it is the most accurate on all ten and leads every baseline in pairwise PSNR (Figure 5 f), with the lowest LPIPS on all ten, VBench within 0.007 of dense attention (Table 7 and Table 28 ), and dense-level error to ground truth (Table 35 , cases in Appendix § 12 ). The three lines of prior work show where this comes from: dropping the skipped keys is fastest but least accurate, compensating with cheap position blocks lands in between, and compensation that re-clusters at every step falls below dense speed on all four-step models (Figure 5 a, Table 27 ). Decoupling removes this trade-off: blocks and clusters meet only through the counts n g c n_{gc} n g c , so it fixes the blocks offline, clusters once, and compensates accurately at fixed-mask speed.
Table 1. Quality and speed at density 0.20. Each model shows PSNR (dB) and speedup over the fastest dense attention backend (top) and SSIM and LPIPS (bottom) against same-seed dense outputs, with test-prompt standard deviations below. Floor compares two dense attention backends, bold marks the best sparse method, and Δ \Delta Δ is DecoupleAtt minus the baseline mean (§ 5.1 ).
Figure 5. Where the speed and accuracy come from, and how they hold. (a) Medians of interleaved runs; per step covers assignment, statistics, counts, and merging. (b–d) DecoupAttn sweeps its density (labels; shaded: ±1 standard error). (e) Top: our exact mask, bottom: our compensator (test prompts; MSE ×10 3 10^{3} 1 0 3 ; small ±: standard deviation). (f) Row minus column PSNR (dB), mean over ten models.
Table 7. VBench for the models of Table 1 : mean of subject consistency, background consistency, temporal flickering, motion smoothness, imaging quality, and aesthetic quality. SSIM and LPIPS with their standard deviations are in Table 1 . Bold: best sparse method.
Table 27. Time breakdown of one video (ms, median over interleaved runs). k-means is the first clustering, Counts the block–cluster counts, and E2E covers encoders, DiT, VAE decoding, and transfers. SVG-EAR has no merge, and its Counts column holds its error map and top-p block selection. Peak GiB is the maximum allocated GPU memory over the whole generation, measured with one method per process and the text embeddings precomputed.
Table 28. VBench dimensions. Subject and background consistency, temporal flickering, motion smoothness, dynamic degree, imaging and aesthetic quality, and overall consistency (text–video alignment).
Table 35. Condition adherence and ground-truth error. T2.5: Cosmos-Transfer2.5; DR: DiffusionRenderer; Alaya: AlayaRenderer.
One procedure serves every family and sampler (RQ2). With the same procedure and constants, DecoupAttn leads the best baseline in all three families and both sampler types, by up to 3.79 dB, with each model’s sampler setting its own J u J_u J u (Eq. (11) ).
Component Analysis (RQ3)
We validate each component by removing it from the full method and re-solving the plan at the same density (Table 2 ). Every removal lowers PSNR in all 59 cells.
Table 2. Component ablations (RQ3). Variant minus the full method (first row) on eight validation clips per model, in PSNR (dB) and latency (ms rows). Supp. counts the models where the variant loses at least 0.3 dB on average and on at least 6 of 8 clips (Table 6 ).
Decoupling matters most. Grouping compensation by the selection blocks (Tied), as in Definition 1, costs 2.85 dB on average at almost unchanged speed, and 3.52 dB under our plan (Tied†). Position blocks group keys by place, not content, so their members spread far from the centroid and leave large residuals: under one exact mask, semantic clusters raise PSNR from 18.36 to 23.47 dB (Figure 5 e, Figure 6 ). A centroid underestimates the mass of a spread-out cluster, so dropping both corrections (Centroid) costs 1.49 dB (Appendix § 11 .§ 3 ).
Figure 6. Group count. Projected attention error against compensation time as the number of position or semantic groups per head varies, with both corrections on (Cosmos-Predict2.5, Wan2.1-14B). Time: CUDA time per attention call of the diagnostic implementation, exact part and compensation, without the assignment refresh. Bars: ±1 standard deviation over eight clips.
With compensation, fixed blocks are as good as routing. Under the same compensator, routing by attention mass at every call matches our fixed blocks on Cosmos-Predict2.5 at 650 ms of routing and falls behind on Wan2.1-14B (Figure 5 e, Table 16 ). The cancellation of § 4.1 explains this: compensated residuals cancel each other, cutting the median coherence ratio on Cosmos-Predict2.5 from 10.37 to 2.69 (Figure 4 a–Figure 4 c, Table 11 ). Ranking blocks offline by this residual also beats ranking by mass or CalibAtt’s masks (Figure 5 e).
Figure 4. Compensated block residuals cancel like dithered quantization errors, and conditioning damps error amplification. (a) For one Cosmos-Predict2.5 query, the small residuals align with the summed error without compensation but point randomly with DecoupAttn, so the coherence ratio κ \kappa κ of Eq. (7) stops growing after the 16 largest (ticks: these 16, 90% of the energy). (b) Per-unit κ \kappa κ drops 2.9–3.9-fold on three models (bars: quartiles, dots: medians, ÷: drop in median), and (c) under one compensator, blocks fixed by their residual match per-step routing by attention mass at no routing cost during inference. (d) Two dense attention backends diverge along the sampler, less under per-pixel conditions (blue) than under prompts (pink). (e) Adding edges to Cosmos-Predict2.5 halves our lead over SVG-EAR and cuts the value of the sensitivity fourfold (lines: 95% bootstrap CI), and (f) on ten models, it falls as the dense-backend floor rises.
Table 11. Coherence ratio κ \kappa κ of Eq. (7) as quartiles over the units of the deployed plans, with the median of the normalized ratio ζ \zeta ζ .
Table 16. Block selection under a fixed semantic compensator. The starred row reads the dense attention of the current call (diagnostic), and Random keeps every diagonal block. Small ±: standard deviation of PSNR over the test prompts of each model.
Pricing the residual by its effect on the output matters next. Setting every sensitivity to one (U) costs 1.68 dB on average, and pricing options by their error before compensation (D), which buys exact blocks where compensation already works, costs 1.18 dB, both at unchanged speed (Table 8 ). The price orders 21 candidate plans by their held-out error as Proposition 2 expects, and the deployed plan is the second best (Figure 8 , Table 20 ). The sensitivity pays most where the sampler amplifies errors: removing it costs 2.30 dB on Cosmos-Predict2.5 but 0.55 dB with edges (Figure 4 e,Figure 4 f, Appendix § 11 .§ 4 ).
Table 8. Speedup over dense attention of every ablation variant (validation clips, same process as its dense anchor). DiffusionRenderer already freezes its cluster assignments.
Figure 8. Predicted versus actual distortion of candidate plans at one density; predicted on calibration clips, measured on held-out clips (Cosmos-Predict2.5, Cosmos-Transfer2.5).
Table 20. How well the priced distortion ranks plans. CP2.5: Cosmos-Predict2.5; T2.5: Cosmos-Transfer2.5. Rank: actual rank of the selected plan among the candidates; regret: its excess actual distortion over the best candidate.
Reassignment keeps the clusters current at little cost. Drifting keys spread frozen clusters beyond what the plan priced, so freezing them (Frozen) costs 1.23 dB on average over the nine models that reassign (Table 26 ). Reassignment never increases this spread (Proposition 3, Figure 11 ), so per-step k-means (Rebuild) changes PSNR by at most 0.95 dB for 11–43% more latency (Tables 9, Table 8 , and Table 25 ).
Table 25. Assignment update schemes under one exact plan. All schemes recompute cluster statistics from current keys and values, and Changed is the share of keys changing assignment per sparse step. DiT ms: median of interleaved runs (periodic rebuild: mean of two).
Figure 11. Assignment schemes across sparse steps (Cosmos-Predict2.5, DiffusionRenderer, UniRelight). Reassignment stays at or below frozen assignments in distance to the cached centroids and cuts the within-cluster and attention errors that frozen assignments let grow. Step 1 builds the clusters.
Table 26. Assignment rule on test clips. Gain and cost of the selected scheme over the other; agrees: whether the test-clip staleness falls on the same side of τ s \tau_s τ s as the calibration staleness. For HunyuanVideo and Cosmos3-Nano this table re-runs the calibration from scratch, with a reassign PSNR of 27.25 and 21.95 dB.
Trade-off and Offline Plan (RQ4, RQ5)
The lead holds across densities (RQ4). Sweeping its density from 0.10 (0.05 on Wan2.1-14B) to 0.45, every baseline lies below its curve within its speed range (Figure 5 b–Figure 5 d), since each point is its density’s lowest-priced plan (Proposition 2), and at the latency of the strongest baseline it is 5.74–8.40 dB more accurate (Figure 12 , Table 29 ).
Table 29. Matched-latency and matched-quality comparison against the strongest baseline of Figure 12 (CP2.5: Cosmos-Predict2.5; Wan2.1: Wan2.1-14B). –: the baseline does not reach the target PSNR at any operating point.
Figure 12. LPIPS against end-to-end latency for the operating points of Figure 5 b–Figure 5 d; dashed: dense attention. End-to-end time covers text encoding, DiT, and VAE decoding (UniRelight: the whole generation).
The offline plan is cheap and holds on new content (RQ5). Calibration takes 5.3–19.6 GPU-hours per model, which the faster inference repays after 2.1k–23.7k videos (Figure 5 a, Table 30 ). As the frozen plan fixes only where exact attention runs and each video clusters its own keys, it beats SVG-EAR on all held-out content (Table 32 ), and Appendix § 11 .8 varies calibration, resolution, and cluster count.
Table 30. Offline cost and break-even count (CP2.5: Cosmos-Predict2.5; Wan2.1: Wan2.1-14B). Calib.: block ranking, distortion table, and error calibration of the deployed plan, without the sensitivity measurement. Peak memory: GPU memory during calibration. Saving: end-to-end time saved per video.
Table 32. Frozen plan on held-out content (Cosmos-Predict2.5; PSNR, dB). vs. SVG-EAR: mean paired difference.
Conclusion
We introduced DecoupAttn, a decoupled sparse attention that keeps compensation when the selection is fixed offline: semantic clusters compensate every key, exact attention corrects their residual on a few fixed blocks, Distortion Budget prices that residual, and Reassign Cache keeps the clusters current. On ten video generation, controlled generation, and neural rendering models, DecoupAttn is the most accurate method faster than dense attention, at up to 1.65× its speed. Our results argue for organizing exact computation and approximation separately: position blocks for the kernel, semantic clusters for the approximation error.
LLM Usage Statement
We used LLMs (Claude) to polish the grammar and word choice of the paper and to refine the plotting scripts of the figures. The authors designed the method and the experiments, ran all experiments, and checked every statement, number, and reference in the paper.
Theory of Decoupled Sparse Attention
This appendix develops the theory behind the three components of § 4 and proves its three propositions. Section A.1 fixes the notation and the assumptions. Section A.2 shows that the decoupled estimator is an exact rearrangement of attention (Proposition 1) whose error is additive over blocks. Section A.3 expands the centroid compensation to second order and identifies what each correction removes. Section A.4 explains why fixed blocks suffice: compensation subtracts exactly the part of the block residuals that aligns, and for blocks fixed independently of the cluster membership the remaining deviation part is no more coherent than random errors. Section A.5 turns this into a rule for choosing the exact blocks, and Section A.6 relates the estimator to difference estimators. Section A.7 analyzes the plan objective, the sensitivity estimator, and the Lagrangian relaxation (Proposition 2). Section A.8 proves Proposition 3 and the monotone chain behind Reassign Cache, Section A.9 counts the cost of one call, and Section A.10 traces how errors travel through the sampler. Algebraic identities hold on every attention call, the expansions hold under the bounded-deviation assumption (A3), their orders in the deviations under its joint form (A3′ ^\prime ′ ), and the statements about expectations under the exchangeability model (A4).
Setting and Notation
Fix one self-attention call and one query i i i , and drop the indices of step, layer, and head. The logits are s i j = q i T k j / d s_{ij}=q_i^\mathsf{T}k_j/\sqrt{d} s ij = q i T k j / d , and the dense quantities N i b N_{ib} N ib , Z i b Z_{ib} Z ib , N i N_i N i , Z i Z_i Z i , and y i = N i / Z i y_i=N_i/Z_i y i = N i / Z i are those of Eq. (1) . The key blocks b b b and the clusters c c c are two partitions of the same L L L keys, and n b c = ∣ b ∩ c ∣ n_{bc}=|b\cap c| n b c = ∣ b ∩ c ∣ counts the keys they share, so that
∑ b n b c = ∣ c ∣ , ∑ c n b c = ∣ b ∣ . \sum_b n_{bc}=|c|,\qquad\sum_c n_{bc}=|b|. b ∑ n b c = ∣ c ∣ , c ∑ n b c = ∣ b ∣.
The query tile g g g of query i i i has exact blocks S g S_g S g and skipped blocks K g K_g K g , the complement of S g S_g S g . Every member j j j of cluster c c c is written as a deviation from the cluster means,
k j = k ˉ c + δ j , v j = v ˉ c + ν j , k_j=\bar{k}_c+\delta_j,\qquad v_j=\bar{v}_c+\nu_j, k j = k ˉ c + δ j , v j = v ˉ c + ν j ,
∑ j ∈ c δ j = 0 , ∑ j ∈ c ν j = 0 , \sum_{j\in c}\delta_j=0,\qquad\sum_{j\in c}\nu_j=0, j ∈ c ∑ δ j = 0 , j ∈ c ∑ ν j = 0 ,
and x i j = q i T δ j / d x_{ij}=q_i^\mathsf{T}\delta_j/\sqrt{d} x ij = q i T δ j / d is the logit deviation, so that e s i j = ϕ i c e x i j e^{s_{ij}}=\phi_{ic}e^{x_{ij}} e s ij = ϕ i c e x ij with ϕ i c = e q i T k ˉ c / d \phi_{ic}=e^{q_i^\mathsf{T}\bar{k}_c/\sqrt{d}} ϕ i c = e q i T k ˉ c / d . The key covariance and the value–key cross-covariance of a cluster are
C c = 1 ∣ c ∣ ∑ j ∈ c δ j δ j T , B c = 1 ∣ c ∣ ∑ j ∈ c ν j δ j T , C_c=\frac{1}{|c|}\sum_{j\in c}\delta_j\delta_j^\mathsf{T},\qquad B_c=\frac{1}{|c|}\sum_{j\in c}\nu_j\delta_j^\mathsf{T}, C c = ∣ c ∣ 1 j ∈ c ∑ δ j δ j T , B c = ∣ c ∣ 1 j ∈ c ∑ ν j δ j T ,
so the per-dimension variance σ c 2 \sigma_c^2 σ c 2 of Eq. (4) is the diagonal of C c C_c C c and the variance correction reads
γ i c = 1 2 d q i T diag ( C c ) q i . \gamma_{ic}=\frac{1}{2d}q_i^\mathsf{T}\operatorname{diag}(C_c)q_i. γ i c = 2 d 1 q i T diag ( C c ) q i .
The per-member predictions of Eq. (4) are ω i c = ϕ i c ( 1 + γ i c ) \omega_{ic}=\phi_{ic}(1+\gamma_{ic}) ω i c = ϕ i c ( 1 + γ i c ) and a i c = ω i c v ˉ c + ϕ i c H q i / d a_{ic}=\omega_{ic}\bar{v}_c+\phi_{ic}Hq_i/\sqrt{d} a i c = ω i c v ˉ c + ϕ i c H q i / d . We use four assumptions, the third in a weak and a joint form. Table 3 collects the symbols used in the proofs.
Table 3. Notation of Appendix A.
(A1) Count-based compensation. Every member of cluster c c c that lies outside the exact blocks contributes a i c a_{ic} a i c to the numerator and ω i c \omega_{ic} ω i c to the denominator, and these terms depend only on the cluster and the query.
(A2) Current statistics. k ˉ c \bar{k}_c k ˉ c , v ˉ c \bar{v}_c v ˉ c , σ c 2 \sigma_c^2 σ c 2 , and H H H are computed from the current members of each cluster, at every call.
(A3) Bounded deviations. ∣ x i j ∣ ≤ η \lvert x_{ij}\rvert\leq\eta ∣ x ij ∣ ≤ η for every query and every member of every cluster.
(A3′ ^\prime ′ ) Small joint deviations. ∥ q i ∥ 2 ∥ δ j ∥ 2 / d ≤ η \lVert q_i\rVert_2\lVert\delta_j\rVert_2/\sqrt{d}\leq\eta ∥ q i ∥ 2 ∥ δ j ∥ 2 / d ≤ η and ∥ ν j ∥ 2 ≤ η V \lVert\nu_j\rVert_2\leq\eta V ∥ ν j ∥ 2 ≤ η V , with V = max j ∥ v j ∥ 2 V=\max_j\lVert v_j\rVert_2 V = max j ∥ v j ∥ 2 , for every query and every member of every cluster.
(A4) Exchangeable membership. Within each cluster, the members are distributed over the blocks by a uniformly random permutation that keeps the counts n b c n_{bc} n b c , independently across clusters and of the members’ keys and values.
(A1) and (A2) describe the implementation and hold exactly. (A3) is the regime in which the expansions of Section A.3 are accurate, and clustering makes η \eta η small because k-means groups nearby keys. (A3′) implies (A3) and adds that the values of a cluster vary on the same small scale, as when values move with their keys. Under it every correction term is quadratic and every remainder cubic in η \eta η . (A4) is a statistical model of the membership under which Section A.4 computes expectations: it states that which block a member falls into carries no information about the direction of its deviation once its cluster is known. Blocks fixed offline never read the membership of a new input, so the skipped set is independent of the membership, and Figure 9 tests the exchangeability itself on saved attention inputs.
Figure 9. Residual correlation against block distance. Mean pairwise cosine of the residuals of two skipped blocks, binned by the distance between the blocks, without compensation, with plain centroids, and with full DecoupAttn, and of the deviation terms D b D_b D b of Proposition A.8 (dashed), which are the same under every compensator.
The Decoupled Estimator (Proposition 1)
Proof of Proposition 1. The key blocks partition the keys, so ∣ c ∣ = ∑ b n b c |c| = \sum_b n_{bc} ∣ c ∣ = ∑ b n b c and n g c = ∑ b ∉ S g n b c n_{gc} = \sum_{b\notin S_g} n_{bc} n g c = ∑ b ∈ / S g n b c . Substituting into Eq. (5) gives
N ^ i = ∑ b ∈ S g N i b + ∑ b ∉ S g ∑ c n b c a i c = ∑ c ∣ c ∣ a i c + ∑ b ∉ S g ( N i b − ∑ c n b c a i c ) , \begin{aligned}
\widehat{N}_i = \sum_{b\in S_g} N_{ib} + \sum_{b\notin S_g}\sum_c n_{bc}a_{ic} \\
&= \sum_c |c|a_{ic} + \sum_{b\notin S_g}\left(N_{ib}-\sum_c n_{bc}a_{ic}\right),
\end{aligned} N i = b ∈ S g ∑ N ib + b ∈ / S g ∑ c ∑ n b c a i c = c ∑ ∣ c ∣ a i c + b ∈ / S g ∑ ( N ib − c ∑ n b c a i c ) ,
which is the first identity of Eq. (6) , and the denominator follows in the same way with Z i b Z_{ib} Z ib and ω i c \omega_{ic} ω i c . Since N i = ∑ b N i b N_i=\sum_b N_{ib} N i = ∑ b N ib and Z i = ∑ b Z i b Z_i=\sum_b Z_{ib} Z i = ∑ b Z ib (Eq. (1) ), the same rearrangement gives N i − N ^ i = ∑ b ∉ S g Δ N i b N_i-\widehat{N}_i=\sum_{b\notin S_g}\Delta N_{ib} N i − N i = ∑ b ∈ / S g Δ N ib and Z i − Z ^ i = ∑ b ∉ S g Δ Z i b Z_i-\widehat{Z}_i=\sum_{b\notin S_g}\Delta Z_{ib} Z i − Z i = ∑ b ∈ / S g Δ Z ib . Using N i = y i Z i N_i=y_iZ_i N i = y i Z i ,
y ^ i − y i = N ^ i − y i Z ^ i Z ^ i = − 1 Z ^ i ∑ b ∉ S g ( Δ N i b − y i Δ Z i b ) . \widehat{y}_i-y_i=\frac{\widehat{N}_i-y_i\widehat{Z}_i}{\widehat{Z}_i}=-\frac{1}{\widehat{Z}_i}\sum_{b\notin S_g}\left(\Delta N_{ib}-y_i\Delta Z_{ib}\right). y i − y i = Z i N i − y i Z i = − Z i 1 b ∈ / S g ∑ ( Δ N ib − y i Δ Z ib ) .
The identity holds for any fixed blocks and any clusters. □
The residual has a per-member form that the rest of this appendix uses. For a member j j j of cluster c c c , let
u i j = e s i j ( v j − y i ) − ( a i c − y i ω i c ) u_{ij}=e^{s_{ij}}(v_j-y_i)-(a_{ic}-y_i\omega_{ic}) u ij = e s ij ( v j − y i ) − ( a i c − y i ω i c )
be the error that compensating this one key leaves. Expanding Δ N i b \Delta N_{ib} Δ N ib and Δ Z i b \Delta Z_{ib} Δ Z ib over the members of block b b b shows that the residual of a block is the sum of the errors of its keys:
r i b = ∑ j ∈ b u i j . r_{ib}=\sum_{j\in b}u_{ij}. r ib = j ∈ b ∑ u ij .
Lemma A.1 (Additive block removal). Let E i ( S ) = N ^ i − y i Z ^ i E_i(S)=\widehat{N}_i-y_i\widehat{Z}_i E i ( S ) = N i − y i Z i be the unnormalized error of the estimator with exact blocks S S S . Then E i ( S ) = − ∑ b ∉ S r i b E_i(S)=-\sum_{b\notin S}r_{ib} E i ( S ) = − ∑ b ∈ / S r ib , and for every block b ∉ S b\notin S b ∈ / S ,
E i ( S ∪ { b } ) = E i ( S ) + r i b , Z ^ i ( S ∪ { b } ) = Z ^ i ( S ) + Δ Z i b . E_i(S\cup\{b\})=E_i(S)+r_{ib},\qquad\widehat{Z}_i(S\cup\{b\})=\widehat{Z}_i(S)+\Delta Z_{ib}. E i ( S ∪ { b }) = E i ( S ) + r ib , Z i ( S ∪ { b }) = Z i ( S ) + Δ Z ib .
Proof. The first claim is the last display of the proof of Proposition 1 multiplied by Z ^ i \widehat{Z}_i Z i . Making b b b exact removes its term from the sum over b ∉ S b\notin S b ∈ / S , and replaces its compensated mass ∑ c n b c ω i c \sum_c n_{bc}\omega_{ic} ∑ c n b c ω i c by Z i b Z_{ib} Z ib in the denominator. □
Lemma A.1 says that every block owns a fixed share r i b r_{ib} r ib of the unnormalized error, whatever the other blocks do. This is what makes a residual a property of a block that can be measured once and priced offline, while the coherence ratio and the mass Z ^ i \widehat{Z}_i Z i decide how the shares combine into the output error (Corollary A.2).
Corollary A.2 (Three factors of the error). With R i = ∑ b ∉ S g ∥ r i b ∥ 2 R_i=\sum_{b\notin S_g}\|r_{ib}\|_2 R i = ∑ b ∈ / S g ∥ r ib ∥ 2 and the coherence ratio κ i \kappa_i κ i of Eq. (7) , set to one when R i = 0 R_i=0 R i = 0 and between 0 and ∣ K g ∣ |K_g| ∣ K g ∣ by Cauchy–Schwarz,
∥ y ^ i − y i ∥ 2 2 = κ i R i Z ^ i 2 , ∥ y ^ i − y i ∥ 2 ∥ y i ∥ 2 ≤ 1 1 − ζ i κ i R i Z ^ i ∥ y i ∥ 2 \|\widehat{y}_i-y_i\|_2^2=\frac{\kappa_iR_i}{\widehat{Z}_i^2},\qquad
\frac{\|\widehat{y}_i-y_i\|_2}{\|y_i\|_2}\leq\frac{1}{1-\zeta_i}\frac{\sqrt{\kappa_iR_i}}{\widehat{Z}_i\|y_i\|_2} ∥ y i − y i ∥ 2 2 = Z i 2 κ i R i , ∥ y i ∥ 2 ∥ y i − y i ∥ 2 ≤ 1 − ζ i 1 Z i ∥ y i ∥ 2 κ i R i
whenever ζ i = ∣ Z ^ i / Z i − 1 ∣ < 1 \zeta_i=|\widehat{Z}_i/Z_i-1|<1 ζ i = ∣ Z i / Z i − 1∣ < 1 and y i ≠ 0 y_i\ne0 y i = 0 . Proof. The equality combines Proposition 1 with the definition of κ i \kappa_i κ i . For the bound, Z ^ i ≥ ( 1 − ζ i ) Z i \widehat{Z}_i\geq(1-\zeta_i)Z_i Z i ≥ ( 1 − ζ i ) Z i . □
The three factors are the residual energy R i R_i R i , its coherence κ i \kappa_i κ i , and the mass ratio Z ^ i / Z i \widehat{Z}_i/Z_i Z i / Z i , which are the columns of Table 24 . Only the variance correction acts on ζ i \zeta_i ζ i , because the value correction enters N ^ i \widehat{N}_i N i but not Z ^ i \widehat{Z}_i Z i : on DiffusionRender the median ∣ Z ^ i / Z i − 1 ∣ |\widehat{Z}_i/Z_i-1| ∣ Z i / Z i − 1∣ is 0.27 without compensation, 0.100 with plain centroids or the value correction alone, and 0.035 with the variance correction (Table 24 ).
Table 24. Factors of the output error (medians over queries on held-out clips, deployed bf16 kernel). Columns are defined in the text above, and the gap is the PSNR of online routing minus fixed blocks under the deployed compensator.
Second-Order Analysis of the Compensation
Lemma A.3 (A centroid underestimates the mass). For every cluster and every query, ∑ j ∈ c e s i j ≥ ∣ c ∣ ϕ i c \sum_{j\in c}e^{s_{ij}}\geq|c|\phi_{ic} ∑ j ∈ c e s ij ≥ ∣ c ∣ ϕ i c , with equality only if all logit deviations x i j x_{ij} x ij of the cluster are zero.
Proof. The exponential is strictly convex and the members’ keys average to k ˉ c \bar{k}_c k ˉ c , so Jensen’s inequality gives
1 ∣ c ∣ ∑ j ∈ c e s i j ≥ exp ( 1 ∣ c ∣ ∑ j ∈ c s i j ) = exp ( q i T k ˉ c / d ) = ϕ i c , \frac{1}{|c|}\sum_{j\in c}e^{s_{ij}}\geq\exp\left(\frac{1}{|c|}\sum_{j\in c}s_{ij}\right)=\exp\left(q_i^\mathsf{T}\bar{k}_c/\sqrt{d}\right)=\phi_{ic}, ∣ c ∣ 1 j ∈ c ∑ e s ij ≥ exp ( ∣ c ∣ 1 j ∈ c ∑ s ij ) = exp ( q i T k ˉ c / d ) = ϕ i c ,
with equality exactly when the s i j s_{ij} s ij are equal, that is, when the x i j x_{ij} x ij vanish. □
Lemma A.3 needs no expansion. It shows that a centroid underestimates the mass of a spread-out cluster and, under (A4), in expectation that of the members the exact blocks leave, since each member is skipped with equal probability.
Lemma A.4 (The variance correction is the diagonal second-order term). Under (A3),
∑ j ∈ c e s i j = ∣ c ∣ ϕ i c ( 1 + 1 2 d q i T C c q i ) + ϕ i c ρ i c , ∣ ρ i c ∣ ≤ ∣ c ∣ η 3 e η 6 , \sum_{j\in c}e^{s_{ij}}=|c|\phi_{ic}\left(1+\frac{1}{2d}q_i^\mathsf{T}C_cq_i\right)+\phi_{ic}\rho_{ic},\qquad|\rho_{ic}|\leq\frac{|c|\eta^3e^\eta}{6}, j ∈ c ∑ e s ij = ∣ c ∣ ϕ i c ( 1 + 2 d 1 q i T C c q i ) + ϕ i c ρ i c , ∣ ρ i c ∣ ≤ 6 ∣ c ∣ η 3 e η ,
and therefore the corrected prediction misses the mass of the cluster by
∑ j ∈ c e s i j − ∣ c ∣ ω i c = ∣ c ∣ ϕ i c 2 d q i T ( C c − diag ( C c ) ) q i + ϕ i c ρ i c . (15) \sum_{j\in c}e^{s_{ij}}-|c|\omega_{ic}=\frac{|c|\phi_{ic}}{2d}q_i^\mathsf{T}\left(C_c-\operatorname{diag}(C_c)\right)q_i+\phi_{ic}\rho_{ic}. \tag*{(15)} j ∈ c ∑ e s ij − ∣ c ∣ ω i c = 2 d ∣ c ∣ ϕ i c q i T ( C c − diag ( C c ) ) q i + ϕ i c ρ i c . (15)
Proof. Taylor’s theorem gives e x = 1 + x + x 2 / 2 + ϱ ( x ) e^x = 1 + x + x^2/2 + \varrho(x) e x = 1 + x + x 2 /2 + ϱ ( x ) with ∣ ϱ ( x ) ∣ ≤ ∣ x ∣ 3 e ∣ x ∣ / 6 |\varrho(x)| \le|x|^3e^{|x|}/6 ∣ ϱ ( x ) ∣ ≤ ∣ x ∣ 3 e ∣ x ∣ /6 . Summing e s i j = ϕ i c e x i j e^{s_{ij}} = \phi_{ic}e^{x_{ij}} e s ij = ϕ i c e x ij over the cluster, the linear term vanishes because ∑ j ∈ c x i j = q i T ∑ j ∈ c δ j / d = 0 \sum_{j\in c}x_{ij} = q_i^{\mathsf T}\sum_{j\in c}\delta_j/\sqrt{d}=0 ∑ j ∈ c x ij = q i T ∑ j ∈ c δ j / d = 0 , and the quadratic term is
1 2 ∑ j ∈ c x i j 2 = 1 2 d q i T ( ∑ j ∈ c δ j δ j T ) q i = ∣ c ∣ 2 d q i T C c q i . \frac{1}{2}\sum_{j\in c}x_{ij}^2 = \frac{1}{2d}q_i^{\mathsf T}\left(\sum_{j\in c}\delta_j\delta_j^{\mathsf T}\right)q_i = \frac{|c|}{2d}q_i^{\mathsf T}C_cq_i. 2 1 j ∈ c ∑ x ij 2 = 2 d 1 q i T ( j ∈ c ∑ δ j δ j T ) q i = 2 d ∣ c ∣ q i T C c q i .
The remainders add up to ρ i c = ∑ j ∈ c ϱ ( x i j ) \rho_{ic}=\sum_{j\in c}\varrho(x_{ij}) ρ i c = ∑ j ∈ c ϱ ( x ij ) , which (A3) bounds term by term. Subtracting ∣ c ∣ ω i c = ∣ c ∣ ϕ i c ( 1 + γ i c ) |c|\omega_{ic}=|c|\phi_{ic}(1+\gamma_{ic}) ∣ c ∣ ω i c = ∣ c ∣ ϕ i c ( 1 + γ i c ) with γ i c = q i T diag ( C c ) q i / ( 2 d ) \gamma_{ic}=q_i^{\mathsf T}\operatorname{diag}(C_c)q_i/(2d) γ i c = q i T diag ( C c ) q i / ( 2 d ) gives the second display. □
The CENTROID estimator therefore leaves the whole second-order term ∣ c ∣ ϕ i c q i T C c q i / ( 2 d ) ≥ 0 |c|\phi_{ic}q_i^{\mathsf T}C_cq_i/(2d)\ge0 ∣ c ∣ ϕ i c q i T C c q i / ( 2 d ) ≥ 0 , in agreement with Lemma A.3. The variance correction removes its diagonal part and leaves only the covariance between different key dimensions within a cluster, of order η 2 \eta^2 η 2 under (A3′), plus a remainder of order η 3 \eta^3 η 3 .
Lemma A.5 (The value correction is the pooled cross-covariance). (i) The matrix of Eq. (4) is the size-weighted average of the clusters’ cross-covariances,
H = ∑ c ∣ c ∣ L B c . H=\sum_c\frac{|c|}{L}B_c. H = c ∑ L ∣ c ∣ B c .
(ii) Under (A3), with V = max j ∥ v j ∥ 2 V=\max_j\|v_j\|_2 V = max j ∥ v j ∥ 2 and V ν = max j ∥ ν j ∥ 2 V_\nu=\max_j\|\nu_j\|_2 V ν = max j ∥ ν j ∥ 2 ,
∑ j ∈ c e s i j v j = ∣ c ∣ ϕ i c [ v ˉ c ( 1 + q i T C c q i 2 d ) + B c q i d ] + ϕ i c ρ i c ′ , \sum_{j\in c}e^{s_{ij}}v_j=|c|\phi_{ic}\left[\bar v_c\left(1+\frac{q_i^{\mathsf T}C_cq_i}{2d}\right)+\frac{B_cq_i}{\sqrt d}\right]+\phi_{ic}\rho'_{ic}, j ∈ c ∑ e s ij v j = ∣ c ∣ ϕ i c [ v ˉ c ( 1 + 2 d q i T C c q i ) + d B c q i ] + ϕ i c ρ i c ′ ,
with ∥ ρ i c ′ ∥ 2 ≤ ∣ c ∣ ( η 2 V ν / 2 + η 3 e η V / 6 ) \|\rho'_{ic}\|_2\le|c|(\eta^2V_\nu/2+\eta^3e^\eta V/6) ∥ ρ i c ′ ∥ 2 ≤ ∣ c ∣ ( η 2 V ν /2 + η 3 e η V /6 ) .
Proof. (i) Within one cluster the cross terms vanish because the deviations have zero mean:
∑ j ∈ c v j κ j T = ∣ c ∣ v ˉ c κ ˉ c T + ∑ j ∈ c ν j δ j T = ∣ c ∣ v ˉ c κ ˉ c T + ∣ c ∣ B c . \sum_{j\in c}v_j\kappa_j^{\mathsf T}=|c|\bar v_c\bar\kappa_c^{\mathsf T}+\sum_{j\in c}\nu_j\delta_j^{\mathsf T}=|c|\bar v_c\bar\kappa_c^{\mathsf T}+|c|B_c. j ∈ c ∑ v j κ j T = ∣ c ∣ v ˉ c κ ˉ c T + j ∈ c ∑ ν j δ j T = ∣ c ∣ v ˉ c κ ˉ c T + ∣ c ∣ B c .
Summing over clusters and dividing by L L L gives H = 1 L ( ∑ j v j κ j T − ∑ c ∣ c ∣ v ˉ c κ ˉ c T ) = ∑ c ∣ c ∣ L B c H=\frac{1}{L}\left(\sum_jv_j\kappa_j^{\mathsf T}-\sum_c|c|\bar v_c\bar\kappa_c^{\mathsf T}\right)=\sum_c\frac{|c|}{L}B_c H = L 1 ( ∑ j v j κ j T − ∑ c ∣ c ∣ v ˉ c κ ˉ c T ) = ∑ c L ∣ c ∣ B c . (ii) Multiply the expansion of e x i j e^{x_{ij}} e x ij by v j = v ˉ c + ν j v_j=\bar v_c+\nu_j v j = v ˉ c + ν j and sum over the cluster. The terms ∑ j ν j \sum_j\nu_j ∑ j ν j and v ˉ c ∑ j x i j \bar v_c\sum_jx_{ij} v ˉ c ∑ j x ij vanish, ∑ j x i j ν j = ( ∑ j ν j δ j T ) q i / d = ∣ c ∣ B c q i / d \sum_jx_{ij}\nu_j=(\sum_j\nu_j\delta_j^{\mathsf T})q_i/\sqrt d=|c|B_cq_i/\sqrt d ∑ j x ij ν j = ( ∑ j ν j δ j T ) q i / d = ∣ c ∣ B c q i / d , and 1 2 v ˉ c ∑ j x i j 2 \frac{1}{2}\bar v_c\sum_jx_{ij}^2 2 1 v ˉ c ∑ j x ij 2 is the variance term of Lemma A.4. What remains is 1 2 ∑ j x i j 2 ν j + ∑ j ϱ ( x i j ) v j \frac{1}{2}\sum_jx_{ij}^2\nu_j+\sum_j\varrho(x_{ij})v_j 2 1 ∑ j x ij 2 ν j + ∑ j ϱ ( x ij ) v j , which (A3) bounds by the stated amount. □
Proposition A.6 (The residual of a whole cluster is second-order). Let U i c = ∑ j ∈ c u i j U_{ic}=\sum_{j\in c}u_{ij} U i c = ∑ j ∈ c u ij be the error that compensation leaves on all members of cluster c c c together. Under (A3),
U i c = ∣ c ∣ ϕ i c [ ( v ˉ c − y i ) q i T ( C c − diag ( C c ) ) q i 2 d + ( B c − H ) q i d ] + ϕ i c ( ρ i c ′ − y i ρ i c ) . U_{ic}=|c|\phi_{ic}\left[(\bar v_c-y_i)\frac{q_i^{\mathsf T}(C_c-\operatorname{diag}(C_c))q_i}{2d}+\frac{(B_c-H)q_i}{\sqrt d}\right]+\phi_{ic}(\rho'_{ic}-y_i\rho_{ic}). U i c = ∣ c ∣ ϕ i c [ ( v ˉ c − y i ) 2 d q i T ( C c − diag ( C c )) q i + d ( B c − H ) q i ] + ϕ i c ( ρ i c ′ − y i ρ i c ) .
Under (A3′), moreover, ∥ U i c ∥ 2 ≤ ∣ c ∣ ϕ i c V ( 3 η 2 + η 3 e η ) \|U_{ic}\|_2\le|c|\phi_{ic}V(3\eta^2+\eta^3e^\eta) ∥ U i c ∥ 2 ≤ ∣ c ∣ ϕ i c V ( 3 η 2 + η 3 e η ) .
Proof. U i c = ( ∑ j e s i j v j − ∣ c ∣ a i c ) − y i ( ∑ j e s i j − ∣ c ∣ ω i c ) U_{ic}=\left(\sum_je^{s_{ij}}v_j-|c|a_{ic}\right)-y_i\left(\sum_je^{s_{ij}}-|c|\omega_{ic}\right) U i c = ( ∑ j e s ij v j − ∣ c ∣ a i c ) − y i ( ∑ j e s ij − ∣ c ∣ ω i c ) . Insert Lemma A.5(ii) and ∣ c ∣ a i c = ∣ c ∣ ϕ i c [ v ˉ c ( 1 + γ i c ) + H q i / d ] |c|a_{ic}=|c|\phi_{ic}[\bar v_c(1+\gamma_{ic})+Hq_i/\sqrt d] ∣ c ∣ a i c = ∣ c ∣ ϕ i c [ v ˉ c ( 1 + γ i c ) + H q i / d ] in the first bracket, and Lemma A.4 in the second. For the bound, (A3′) gives ∣ x i j ∣ ≤ η |x_{ij}|\le\eta ∣ x ij ∣ ≤ η and ∑ p q i p 2 δ j p 2 ≤ ∥ q i ∥ 2 2 ∥ δ j ∥ 2 2 ≤ d η 2 \sum_pq_{ip}^2\delta_{jp}^2\le\|q_i\|_2^2\|\delta_j\|_2^2\le d\eta^2 ∑ p q i p 2 δ j p 2 ≤ ∥ q i ∥ 2 2 ∥ δ j ∥ 2 2 ≤ d η 2 , so q i T C c q i / ( 2 d ) q_i^{\mathsf T}C_cq_i/(2d) q i T C c q i / ( 2 d ) and q i T diag ( C c ) q i / ( 2 d ) q_i^{\mathsf T}\operatorname{diag}(C_c)q_i/(2d) q i T diag ( C c ) q i / ( 2 d ) both lie in [ 0 , η 2 / 2 ] [0,\eta^2/2] [ 0 , η 2 /2 ] . It also gives ∥ B c q i ∥ 2 / d ≤ η 2 V \|B_cq_i\|_2/\sqrt d\le\eta^2V ∥ B c q i ∥ 2 / d ≤ η 2 V and ∥ H q i ∥ 2 / d ≤ η 2 V \|Hq_i\|_2/\sqrt d\le\eta^2V ∥ H q i ∥ 2 / d ≤ η 2 V , because B c q i / d B_cq_i/\sqrt d B c q i / d averages x i j ν j x_{ij}\nu_j x ij ν j and H H H averages the B c B_c B c . With ∥ v ˉ c − y i ∥ 2 ≤ 2 V \|\bar v_c-y_i\|_2\le2V ∥ v ˉ c − y i ∥ 2 ≤ 2 V and ∥ y i ∥ 2 ≤ V \|y_i\|_2\le V ∥ y i ∥ 2 ≤ V , the bracket is at most 3 η 2 V 3\eta^2V 3 η 2 V , and the remainders of Lemmas A.4 and A.5 add at most ∣ c ∣ η 3 V ( 1 / 2 + e η / 3 ) ≤ ∣ c ∣ η 3 e η V |c|\eta^3V(1/2+e^\eta/3)\le|c|\eta^3e^\eta V ∣ c ∣ η 3 V ( 1/2 + e η /3 ) ≤ ∣ c ∣ η 3 e η V . □
Proposition A.6 says what the two corrections achieve. On every cluster, compensation is exact up to three terms: the off-diagonal key covariance, the gap between the cluster’s own cross-covariance B c B_c B c and the head average H H H , and a remainder, of orders η 2 \eta^2 η 2 , η 2 \eta^2 η 2 , and η 3 \eta^3 η 3 under (A3′). The two corrections of Eq. (4) are therefore the second-order terms that a diagonal, head-shared model can capture. A full covariance per cluster and a per-cluster H c H_c H c remove the two second-order terms, and Table 18 measures what these additions gain on saved attention inputs.
Table 18. Richer corrections on saved attention inputs (Cosmos-Predict2.5). Extra ms: added time per call (–: offline implementation).
Why Fixed Blocks Suffice
Lemma A.7 (First-order cancellation). Under (A3′), the per-member error splits as u i j = u i j ( 1 ) + u i j ( 2 ) u_{ij}=u_{ij}^{(1)}+u_{ij}^{(2)} u ij = u ij ( 1 ) + u ij ( 2 ) with ∥ u i j ( 2 ) ∥ 2 ≤ ( 3 + e η ) ϕ i c V η 2 \|u_{ij}^{(2)}\|_2\le(3+e^\eta)\phi_{ic}V\eta^2 ∥ u ij ( 2 ) ∥ 2 ≤ ( 3 + e η ) ϕ i c V η 2 and the first-order part
u i j ( 1 ) = ϕ i c ( ν j + ( v ˉ c − y i ) x i j ) , u_{ij}^{(1)}=\phi_{ic}\left(\nu_j+(\bar v_c-y_i)x_{ij}\right), u ij ( 1 ) = ϕ i c ( ν j + ( v ˉ c − y i ) x ij ) ,
which sums to zero over every cluster. Consequently the first-order parts r i b ( 1 ) = ∑ j ∈ b u i j ( 1 ) r_{ib}^{(1)}=\sum_{j\in b}u_{ij}^{(1)} r ib ( 1 ) = ∑ j ∈ b u ij ( 1 ) of the block residuals sum to zero over all blocks, and
∑ b ∉ S g r i b ( 1 ) = − ∑ b ∈ S g r i b ( 1 ) . \sum_{b\notin S_g}r_{ib}^{(1)}=-\sum_{b\in S_g}r_{ib}^{(1)}. b ∈ / S g ∑ r ib ( 1 ) = − b ∈ S g ∑ r ib ( 1 ) .
Proof. With e s i j = ϕ i c e x i j e^{s_{ij}}=\phi_{ic}e^{x_{ij}} e s ij = ϕ i c e x ij , v j = v ˉ c + ν j v_j=\bar v_c+\nu_j v j = v ˉ c + ν j , and a i c − y i ω i c = ϕ i c ( 1 + γ i c ) ( v ˉ c − y i ) + ϕ i c H q i / d a_{ic}-y_i\omega_{ic}=\phi_{ic}(1+\gamma_{ic})(\bar v_c-y_i)+\phi_{ic}Hq_i/\sqrt d a i c − y i ω i c = ϕ i c ( 1 + γ i c ) ( v ˉ c − y i ) + ϕ i c H q i / d , the zeroth-order terms ϕ i c ( v ˉ c − y i ) \phi_{ic}(\bar v_c-y_i) ϕ i c ( v ˉ c − y i ) cancel and
u i j ( 2 ) = ϕ i c [ x i j ν j + ( e x i j − 1 − x i j ) ( v j − y i ) − γ i c ( v ˉ c − y i ) − H q i / d ] . u_{ij}^{(2)}=\phi_{ic}\left[x_{ij}\nu_j+(e^{x_{ij}}-1-x_{ij})(v_j-y_i)-\gamma_{ic}(\bar v_c-y_i)-Hq_i/\sqrt d\right]. u ij ( 2 ) = ϕ i c [ x ij ν j + ( e x ij − 1 − x ij ) ( v j − y i ) − γ i c ( v ˉ c − y i ) − H q i / d ] .
Under (A3′) the four terms are at most η 2 V \eta^2V η 2 V , η 2 e η V \eta^2e^\eta V η 2 e η V , η 2 V \eta^2V η 2 V , and η 2 V \eta^2V η 2 V , by ∣ e x − 1 − x ∣ ≤ x 2 e ∣ x ∣ / 2 |e^x-1-x|\le x^2e^{|x|}/2 ∣ e x − 1 − x ∣ ≤ x 2 e ∣ x ∣ /2 , ∥ v j − y i ∥ 2 ≤ 2 V \|v_j-y_i\|_2\le2V ∥ v j − y i ∥ 2 ≤ 2 V , and the bounds in the proof of Proposition A.6. The first-order part sums to zero because ∑ j ∈ c ν j = 0 \sum_{j\in c}\nu_j=0 ∑ j ∈ c ν j = 0 and ∑ j ∈ c x i j = 0 \sum_{j\in c}x_{ij}=0 ∑ j ∈ c x ij = 0 . Since r i b = ∑ j ∈ b u i j r_{ib}=\sum_{j\in b}u_{ij} r ib = ∑ j ∈ b u ij and the blocks partition the keys, ∑ b r i b ( 1 ) = ∑ c ∑ j ∈ c u i j ( 1 ) = 0 \sum_b r_{ib}^{(1)}=\sum_c\sum_{j\in c}u_{ij}^{(1)}=0 ∑ b r ib ( 1 ) = ∑ c ∑ j ∈ c u ij ( 1 ) = 0 . □
Dropping a block has an analogous identity: the dropped residuals r i b d r o p = ∑ j ∈ b e s i j ( v j − y i ) r_{ib}^{\mathrm{drop}}=\sum_{j\in b}e^{s_{ij}}(v_j-y_i) r ib drop = ∑ j ∈ b e s ij ( v j − y i ) also sum to zero over all blocks, because ∑ j e s i j ( v j − y i ) = N i − y i Z i = 0 \sum_j e^{s_{ij}}(v_j-y_i)=N_i-y_iZ_i=0 ∑ j e s ij ( v j − y i ) = N i − y i Z i = 0 . What separates the two is not the sum over all blocks but how the residuals of the skipped blocks align with each other. The next proposition makes this precise without any expansion.
Proposition A.8 (Compensation subtracts the aligned part). Let z ˉ i c = 1 ∣ c ∣ ∑ j ∈ c e s i j ( v j − y i ) \bar{z}_{ic}=\frac{1}{|c|}\sum_{j\in c}e^{s_{ij}}(v_j-y_i) z ˉ i c = ∣ c ∣ 1 ∑ j ∈ c e s ij ( v j − y i ) be the mean contribution of a member of cluster c c c , and w i j = e s i j ( v j − y i ) − z ˉ i c w_{ij}=e^{s_{ij}}(v_j-y_i)-\bar{z}_{ic} w ij = e s ij ( v j − y i ) − z ˉ i c its deviation, which sums to zero over the cluster. Then for every block,
r i b d r o p = ∑ c n b c z ˉ i c ⏟ cluster means + ∑ j ∈ b w i j ⏟ deviations , r i b = ∑ j ∈ b w i j ⏟ deviations + ∑ c n b c U i c ∣ c ∣ ⏟ cluster bias β i b . r_{ib}^{\mathrm{drop}}=\underbrace{\sum_c n_{bc}\bar{z}_{ic}}_{\text{cluster means}}+\underbrace{\sum_{j\in b}w_{ij}}_{\text{deviations}},\qquad r_{ib}=\underbrace{\sum_{j\in b}w_{ij}}_{\text{deviations}}+\underbrace{\sum_c n_{bc}\frac{U_{ic}}{|c|}}_{\text{cluster bias }\beta_{ib}}. r ib drop = cluster means c ∑ n b c z ˉ i c + deviations j ∈ b ∑ w ij , r ib = deviations j ∈ b ∑ w ij + cluster bias β ib c ∑ n b c ∣ c ∣ U i c .
Proof. The first identity splits each member’s contribution into z ˉ i c \bar{z}_{ic} z ˉ i c and w i j w_{ij} w ij and groups the means by cluster. For the second, r i b = r i b d r o p − ∑ c n b c ( a i c − y i ω i c ) r_{ib}=r_{ib}^{\mathrm{drop}}-\sum_c n_{bc}(a_{ic}-y_i\omega_{ic}) r ib = r ib drop − ∑ c n b c ( a i c − y i ω i c ) , and averaging u i j u_{ij} u ij over the cluster gives a i c − y i ω i c = z ˉ i c − U i c / ∣ c ∣ a_{ic}-y_i\omega_{ic}=\bar{z}_{ic}-U_{ic}/|c| a i c − y i ω i c = z ˉ i c − U i c /∣ c ∣ .
The cluster-mean term of a dropped block is a sum of the same vectors z ˉ i c \bar{z}_{ic} z ˉ i c for every block that contains members of c c c . Blocks with similar cluster composition therefore push the output in the same direction. Compensation removes this term and replaces it by the cluster bias β i b \beta_{ib} β ib , built from the U i c / ∣ c ∣ U_{ic}/|c| U i c /∣ c ∣ that Proposition A.6 shows to be of second order under (A3′) and that the variance correction shrinks in practice (Appendix § 11 .§ 5 ). What remains is the deviation term, whose behavior the next lemma computes.
Lemma A.9 (Sampling identities). Let a finite population w 1 , … , w n w_1,\ldots,w_n w 1 , … , w n of vectors, n ≥ 2 n\ge2 n ≥ 2 , have ∑ j w j = 0 \sum_j w_j=0 ∑ j w j = 0 and τ 2 = 1 n ∑ j ∥ w j ∥ 2 2 \tau^2=\frac{1}{n}\sum_j\lVert w_j\rVert_2^2 τ 2 = n 1 ∑ j ∥ w j ∥ 2 2 , and let disjoint subsets A A A and A ′ A' A ′ of sizes a a a and a ′ a' a ′ be drawn by a uniformly random permutation. Then
E ∑ j ∈ A w j = 0 , E ∥ ∑ j ∈ A w j ∥ 2 2 = a ( n − a ) n − 1 τ 2 , \mathbb{E}\sum_{j\in A}w_j=0,\qquad\mathbb{E}\left\lVert\sum_{j\in A}w_j\right\rVert_2^2=\frac{a(n-a)}{n-1}\tau^2, E j ∈ A ∑ w j = 0 , E j ∈ A ∑ w j 2 2 = n − 1 a ( n − a ) τ 2 , E ⟨ ∑ j ∈ A w j , ∑ j ∈ A ′ w j ⟩ = − a a ′ n − 1 τ 2 . \mathbb{E}\left\langle\sum_{j\in A}w_j,\sum_{j\in A'}w_j\right\rangle=-\frac{aa'}{n-1}\tau^2. E ⟨ j ∈ A ∑ w j , j ∈ A ′ ∑ w j ⟩ = − n − 1 a a ′ τ 2 .
Proof. Each w j w_j w j belongs to A A A with probability a / n a/n a / n , so the mean is a n ∑ j w j = 0 \frac{a}{n}\sum_jw_j=0 n a ∑ j w j = 0 . For the second moment,
E ∥ ∑ j ∈ A w j ∥ 2 = ∑ j Pr [ j ∈ A ] ∥ w j ∥ 2 + ∑ j ≠ j ′ Pr [ j , j ′ ∈ A ] ⟨ w j , w j ′ ⟩ \mathbb{E}\left\lVert\sum_{j\in A}w_j\right\rVert^2=\sum_j\Pr[j\in A]\lVert w_j\rVert^2+\sum_{j\ne j'}\Pr[j,j'\in A]\langle w_j,w_{j'}\rangle E j ∈ A ∑ w j 2 = j ∑ Pr [ j ∈ A ] ∥ w j ∥ 2 + j = j ′ ∑ Pr [ j , j ′ ∈ A ] ⟨ w j , w j ′ ⟩
with Pr [ j ∈ A ] = a / n \Pr[j\in A]=a/n Pr [ j ∈ A ] = a / n and Pr [ j , j ′ ∈ A ] = a ( a − 1 ) / ( n ( n − 1 ) ) \Pr[j,j'\in A]=a(a-1)/(n(n-1)) Pr [ j , j ′ ∈ A ] = a ( a − 1 ) / ( n ( n − 1 )) . Because ∑ j ≠ j ′ ⟨ w j , w j ′ ⟩ = ∥ ∑ j w j ∥ 2 − ∑ j ∥ w j ∥ 2 = − n τ 2 \sum_{j\ne j'}\langle w_j,w_{j'}\rangle=\left\lVert\sum_jw_j\right\rVert^2-\sum_j\lVert w_j\rVert^2=-n\tau^2 ∑ j = j ′ ⟨ w j , w j ′ ⟩ = ∑ j w j 2 − ∑ j ∥ w j ∥ 2 = − n τ 2 , this equals
a τ 2 − a ( a − 1 ) n − 1 τ 2 = a ( n − a ) n − 1 τ 2 . a\tau^2-\frac{a(a-1)}{n-1}\tau^2=\frac{a(n-a)}{n-1}\tau^2. a τ 2 − n − 1 a ( a − 1 ) τ 2 = n − 1 a ( n − a ) τ 2 .
Applying the second identity to A A A , A ′ A' A ′ , and A ∪ A ′ A\cup A' A ∪ A ′ and expanding ∥ ∑ A ∪ A ′ w j ∥ 2 \left\lVert\sum_{A\cup A'}w_j\right\rVert^2 ∥ ∑ A ∪ A ′ w j ∥ 2 gives the cross term:
2 E ⟨ ∑ A w j , ∑ A ′ w j ⟩ = τ 2 n − 1 [ ( a + a ′ ) ( n − a − a ′ ) − a ( n − a ) − a ′ ( n − a ′ ) ] = − 2 a a ′ n − 1 τ 2 . \begin{aligned}
2\mathbb{E}\left\langle\sum_A w_j,\sum_{A'}w_j\right\rangle=\frac{\tau^2}{n-1}\left[(a+a')(n-a-a')-a(n-a)-a'(n-a')\right] \\
&=-\frac{2aa'}{n-1}\tau^2.
\end{aligned} 2 E ⟨ A ∑ w j , A ′ ∑ w j ⟩ = n − 1 τ 2 [ ( a + a ′ ) ( n − a − a ′ ) − a ( n − a ) − a ′ ( n − a ′ ) ] = − n − 1 2 a a ′ τ 2 .
Theorem A.10 (Deviation terms add like random errors). Under (A4), let K K K be a set of skipped blocks fixed independently of the permutation, let τ i c 2 = 1 ∣ c ∣ ∑ j ∈ c ∥ w i j ∥ 2 2 \tau_{ic}^2=\frac{1}{|c|}\sum_{j\in c}\lVert w_{ij}\rVert_2^2 τ i c 2 = ∣ c ∣ 1 ∑ j ∈ c ∥ w ij ∥ 2 2 for clusters with ∣ c ∣ ≥ 2 |c|\ge2 ∣ c ∣ ≥ 2 (a singleton has w i j = 0 w_{ij}=0 w ij = 0 and drops out), let m c = ∑ b ∈ K n b c m_c=\sum_{b\in K}n_{bc} m c = ∑ b ∈ K n b c count the members of c c c in K K K , and let D b = ∑ j ∈ b w i j D_b=\sum_{j\in b}w_{ij} D b = ∑ j ∈ b w ij be the deviation term of Proposition A.8 . Then E D b = 0 \mathbb{E}\,D_b=0 E D b = 0 and
E ∥ ∑ b ∈ K D b ∥ 2 2 = ∑ c m c ( ∣ c ∣ − m c ) ∣ c ∣ − 1 τ i c 2 \mathbb{E}\left\lVert\sum_{b\in K}D_b\right\rVert_2^2=\sum_c\frac{m_c(|c|-m_c)}{|c|-1}\tau_{ic}^2 E b ∈ K ∑ D b 2 2 = c ∑ ∣ c ∣ − 1 m c ( ∣ c ∣ − m c ) τ i c 2 ≤ ∑ c ∑ b ∈ K n b c ( ∣ c ∣ − n b c ) ∣ c ∣ − 1 τ i c 2 \le\sum_c\sum_{b\in K}\frac{n_{bc}(|c|-n_{bc})}{|c|-1}\tau_{ic}^2 ≤ c ∑ b ∈ K ∑ ∣ c ∣ − 1 n b c ( ∣ c ∣ − n b c ) τ i c 2 = ∑ b ∈ K E ∥ D b ∥ 2 2 . =\sum_{b\in K}\mathbb{E}\lVert D_b\rVert_2^2. = b ∈ K ∑ E ∥ D b ∥ 2 2 . The expected coherence ratio κ ˉ D = E ∥ ∑ b ∈ K D b ∥ 2 2 ∑ b ∈ K E ∥ D b ∥ 2 2 \bar{\kappa}_D=\frac{\mathbb{E}\lVert\sum_{b\in K}D_b\rVert_2^2}{\sum_{b\in K}\mathbb{E}\lVert D_b\rVert_2^2} κ ˉ D = ∑ b ∈ K E ∥ D b ∥ 2 2 E ∥ ∑ b ∈ K D b ∥ 2 2 is therefore at most one, and distinct blocks are negatively correlated: E ⟨ D b , D b ′ ⟩ = − ∑ c n b c n b ′ c τ i c 2 / ( ∣ c ∣ − 1 ) ≤ 0 \mathbb{E}\langle D_b,D_{b'}\rangle=-\sum_c n_{bc}n_{b'c}\tau_{ic}^2/(|c|-1)\le0 E ⟨ D b , D b ′ ⟩ = − ∑ c n b c n b ′ c τ i c 2 / ( ∣ c ∣ − 1 ) ≤ 0 . With the cluster bias β i b \beta_{ib} β ib of Proposition A.8 , the full residual satisfies
E ∥ ∑ b ∈ K r i b ∥ 2 2 = E ∥ ∑ b ∈ K D b ∥ 2 2 + E ∥ ∑ c m c ∣ c ∣ U i c ∥ 2 2 . \mathbb{E}\left\lVert\sum_{b\in K}r_{ib}\right\rVert_2^2=\mathbb{E}\left\lVert\sum_{b\in K}D_b\right\rVert_2^2+\mathbb{E}\left\lVert\sum_c\frac{m_c}{|c|}U_{ic}\right\rVert_2^2. E b ∈ K ∑ r ib 2 2 = E b ∈ K ∑ D b 2 2 + E c ∑ ∣ c ∣ m c U i c 2 2 . Proof. By (A4), the members of each cluster that fall into each block form disjoint random subsets of sizes n b c n_{bc} n b c , and different clusters are independent. Writing D b = ∑ c D b c D_b=\sum_cD_{bc} D b = ∑ c D b c with D b c = ∑ j ∈ b ∩ c w i j D_{bc}=\sum_{j\in b\cap c}w_{ij} D b c = ∑ j ∈ b ∩ c w ij , every D b c D_{bc} D b c has zero mean (Lemma A.9 ), so cross terms between different clusters vanish. Within a cluster, ∑ b ∈ K D b c \sum_{b\in K}D_{bc} ∑ b ∈ K D b c is the sum over a random subset of size m c m_c m c , and Lemma A.9 gives the left side. Each block alone is a subset of size n b c n_{bc} n b c , which gives the right side and the cross term. The inequality holds cluster by cluster because m c ≥ n b c m_c\ge n_{bc} m c ≥ n b c for every b ∈ K b\in K b ∈ K :
m c ( ∣ c ∣ − m c ) = ∑ b ∈ K n b c ( ∣ c ∣ − m c ) ≤ ∑ b ∈ K n b c ( ∣ c ∣ − n b c ) . m_c(|c|-m_c)=\sum_{b\in K}n_{bc}(|c|-m_c)\le\sum_{b\in K}n_{bc}(|c|-n_{bc}). m c ( ∣ c ∣ − m c ) = b ∈ K ∑ n b c ( ∣ c ∣ − m c ) ≤ b ∈ K ∑ n b c ( ∣ c ∣ − n b c ) . For the full residual, r i b = D b + β i b r_{ib}=D_b+\beta_{ib} r ib = D b + β ib , and ∑ b ∈ K β i b = ∑ c m c U i c / ∣ c ∣ \sum_{b\in K}\beta_{ib}=\sum_cm_cU_{ic}/|c| ∑ b ∈ K β ib = ∑ c m c U i c /∣ c ∣ depends only on the counts because U i c U_{ic} U i c sums over the whole cluster, so the cross term vanishes with $\mathbb{E},D_b=0.
Corollary A.11 (The selection enters only through counts). Under (A4), for exact sets fixed independently of the permutation, the expected deviation error E ∥ ∑ b ∉ S g D b ∥ 2 2 \mathbb{E}\lVert\sum_{b\notin S_g}D_b\rVert_2^2 E ∥ ∑ b ∈ / S g D b ∥ 2 2 depends on the exact set S g S_g S g only through the numbers m c = ∑ b ∉ S g n b c m_c=\sum_{b\notin S_g}n_{bc} m c = ∑ b ∈ / S g n b c of skipped members of each cluster. Two exact sets that skip the same number of members of every cluster have the same expected deviation error, whichever blocks they contain.
Proof. The left side in Theorem A.10 is ∑ c m c ( ∣ c ∣ − m c ) τ i c 2 / ( ∣ c ∣ − 1 ) \sum_c m_c(|c|-m_c)\tau_{ic}^2/(|c|-1) ∑ c m c ( ∣ c ∣ − m c ) τ i c 2 / ( ∣ c ∣ − 1 ) , which contains S g S_g S g only through the m c m_c m c .
Blocks fixed offline never read the keys of a new input, so they meet this independence by construction. Among such sets, a change of blocks under the same budget changes the deviation error only through the m c m_c m c , and with hundreds of clusters spread over many blocks each m c m_c m c moves by little. A selection made anew at every call reads the current keys and can correlate with the deviations, which moves the error at equal counts in either direction, and Table 16 measures the net effect: under one compensator, routing by attention mass at every call reaches 24.65 dB against 24.70 dB for fixed blocks on Cosmos-Predict2.5.
Corollary A.12 (Dropped residuals align). Under (A4), the dropped residuals have mean μ b = ∑ c n b c z ˉ i c \mu_b = \sum_c n_{bc}\bar{z}_{ic} μ b = ∑ c n b c z ˉ i c , and for every set K \mathcal{K} K fixed independently of the permutation
E ∥ ∑ b ∈ K r i b d r o p ∥ 2 2 = ∥ ∑ b ∈ K μ b ∥ 2 2 + E ∥ ∑ b ∈ K D b ∥ 2 2 . \mathbb{E}\left\|\sum_{b\in\mathcal{K}}r_{ib}^{\mathrm{drop}}\right\|_2^2
=
\left\|\sum_{b\in\mathcal{K}}\mu_b\right\|_2^2
+
\mathbb{E}\left\|\sum_{b\in\mathcal{K}}D_b\right\|_2^2. E b ∈ K ∑ r ib drop 2 2 = b ∈ K ∑ μ b 2 2 + E b ∈ K ∑ D b 2 2 . If the skipped blocks share one cluster composition, n b c = α b p c n_{bc}=\alpha_b p_c n b c = α b p c with ∑ c p c z ˉ i c ≠ 0 \sum_c p_c\bar{z}_{ic}\ne0 ∑ c p c z ˉ i c = 0 , the cluster-mean terms are parallel and their coherence ratio ∥ ∑ b μ b ∥ 2 2 / ∑ b ∥ μ b ∥ 2 2 = ( ∑ b α b ) 2 / ∑ b α b 2 \left\|\sum_b\mu_b\right\|_2^2/\sum_b\|\mu_b\|_2^2 = \left(\sum_b\alpha_b\right)^2/\sum_b\alpha_b^2 ∥ ∑ b μ b ∥ 2 2 / ∑ b ∥ μ b ∥ 2 2 = ( ∑ b α b ) 2 / ∑ b α b 2 reaches ∣ K ∣ |\mathcal{K}| ∣ K ∣ for blocks of equal size.
Proof. By Proposition A.8, r i b d r o p = μ b + D b r_{ib}^{\mathrm{drop}}=\mu_b+D_b r ib drop = μ b + D b with E D b = 0 \mathbb{E}D_b=0 E D b = 0 , so the cross term between the two parts vanishes in expectation. Under a shared composition, μ b = α b ∑ c p c z ˉ i c \mu_b=\alpha_b\sum_c p_c\bar{z}_{ic} μ b = α b ∑ c p c z ˉ i c for every b b b . □
Theorem A.10 and Corollary A.12 give the two regimes of the coherence ratio. Dropped blocks carry a cluster-mean term that adds coherently over the blocks that share clusters, so its coherence ratio grows with the number of such blocks. Compensated blocks carry only the deviation term, whose expected coherence ratio κ ˉ D \bar{\kappa}_D κ ˉ D is at most one, plus the cluster bias, of second order under (A3′) and the coherent part of the residual in practice (Appendix § 11 .§ 5 ). This is the mechanism of subtractive dither in quantization [27 ] : subtracting a known prediction leaves zero-mean errors that add like random ones. The measured medians follow these regimes, from 4.69, 5.03, and 10.37 without compensation to 1.61, 1.65, and 2.69 with both corrections on DiffusionRenderer, Wan2.1-14B, and Cosmos-Predict2.5 (Table 11 ). Theorem A.10 holds for every set of blocks fixed independently of the membership, so blocks fixed offline by position keep κ ˉ D \bar{\kappa}_D κ ˉ D at most one, as observed in § 4.1 .
Choosing the Exact Blocks
Proposition A.13 (Residual ranking). Fix a query tile and a budget of K K K exact blocks besides its diagonal block b g b_g b g , and take expectations over calibration inputs. If the residuals of distinct blocks are orthogonal in expectation, E ⟨ r i b , r i b ′ ⟩ = 0 \mathbb{E}\langle r_{ib},r_{ib'}\rangle=0 E ⟨ r ib , r i b ′ ⟩ = 0 for b ≠ b ′ b\ne b' b = b ′ , the set of K K K blocks that minimizes the expected unnormalized error
E ∥ ∑ b ∉ S r i b ∥ 2 2 , \mathbb{E}\left\|\sum_{b\notin\mathcal{S}}r_{ib}\right\|_2^2, E b ∈ / S ∑ r ib 2 2 , summed over the queries of the tile, consists of the K K K blocks with the largest ∑ i ∈ g E ∥ r i b ∥ 2 2 \sum_{i\in g}\mathbb{E}\|r_{ib}\|_2^2 ∑ i ∈ g E ∥ r ib ∥ 2 2 . If instead E ⟨ r i b , r i b ′ ⟩ ≤ 0 \mathbb{E}\langle r_{ib},r_{ib'}\rangle\le0 E ⟨ r ib , r i b ′ ⟩ ≤ 0 , as for the deviation terms of Theorem A.10, the same set minimizes an upper bound on this error.
Proof. With orthogonal residuals, E ∥ ∑ b ∉ S r i b ∥ 2 = ∑ b E ∥ r i b ∥ 2 − ∑ b ∈ S E ∥ r i b ∥ 2 \mathbb{E}\left\|\sum_{b\notin\mathcal{S}}r_{ib}\right\|^2=\sum_b\mathbb{E}\|r_{ib}\|^2-\sum_{b\in\mathcal{S}}\mathbb{E}\|r_{ib}\|^2 E ∑ b ∈ / S r ib 2 = ∑ b E ∥ r ib ∥ 2 − ∑ b ∈ S E ∥ r ib ∥ 2 for every query, and summing over the query tile leaves a total that the largest K K K terms minimize. With nonpositive cross moments, E ∥ ∑ b ∉ S r i b ∥ 2 ≤ ∑ b ∉ S E ∥ r i b ∥ 2 \mathbb{E}\left\|\sum_{b\notin\mathcal{S}}r_{ib}\right\|^2\le\sum_{b\notin\mathcal{S}}\mathbb{E}\|r_{ib}\|^2 E ∑ b ∈ / S r ib 2 ≤ ∑ b ∈ / S E ∥ r ib ∥ 2 , and the same choice minimizes this upper bound. □
Equation (9) implements this rule with each query weighted by 1 / ( Z i 2 ∥ y i ∥ 2 2 ) 1/(Z_i^2\|y_i\|_2^2) 1/ ( Z i 2 ∥ y i ∥ 2 2 ) , for queries with y i ≠ 0 y_i\ne0 y i = 0 . The weight is the scale of the relative error of Corollary A.2 with the dense mass Z i Z_i Z i in place of Z ^ i \hat{Z}_i Z ^ i , which depends on the exact set, so the ranking minimizes a residual-energy proxy of the relative error. The rule ranks blocks by how badly compensation handles them, not by the attention they hold. A block with large attention mass that compensation already predicts well has a small residual and stays skipped.
Relation to Difference Estimators The decoupled estimator has the form of a difference estimator from survey sampling [26 ] : a model predicts every term cheaply, and the exact values correct the prediction on a chosen subset. Scatterbrain [1 ] and control-variate attention [50 ] use the same structure with a support chosen for each input, whereas DecoupAttn keeps the clusters and the exact blocks as two groupings joined by counts and fixes the blocks offline. This view explains why compensation changes what the selection has to capture.
Proposition A.14 (Random exact blocks). Let every block be made exact independently with probability π b ∈ ( 0 , 1 ] \pi_b\in(0,1] π b ∈ ( 0 , 1 ] , let I b I_b I b indicate the event, and weight each correction of the numerator and the mass by 1 / π b 1/\pi_b 1/ π b :
( N ^ i H T , Z ^ i H T ) = ∑ c ∣ c ∣ ( a i c , ω i c ) + ∑ b I b π b ( Δ N i b , Δ Z i b ) . (\hat{N}_i^{\mathrm{HT}},\hat{Z}_i^{\mathrm{HT}})
=
\sum_c |c|\,(a_{ic},\omega_{ic})
+
\sum_b\frac{I_b}{\pi_b}\,(\Delta N_{ib},\Delta Z_{ib}). ( N ^ i HT , Z ^ i HT ) = c ∑ ∣ c ∣ ( a i c , ω i c ) + b ∑ π b I b ( Δ N ib , Δ Z ib ) . *Then E N ^ i H T = N i \mathbb{E}\hat{N}_i^{\mathrm{HT}}=N_i E N ^ i HT = N i , E Z ^ i H T = Z i \mathbb{E}\hat{Z}_i^{\mathrm{HT}}=Z_i E Z ^ i HT = Z i , the output y ^ i H T = N ^ i H T / Z ^ i H T \hat{y}_i^{\mathrm{HT}}=\hat{N}_i^{\mathrm{HT}}/\hat{Z}_i^{\mathrm{HT}} y ^ i HT = N ^ i HT / Z ^ i HT has the error y ^ i H T − y i = ∑ b ( I b / π b − 1 ) r i b / Z ^ i H T \hat{y}_i^{\mathrm{HT}}-y_i=\sum_b(I_b/\pi_b-1)r_{ib}/\hat{Z}_i^{\mathrm{HT}} y ^ i HT − y i = ∑ b ( I b / π b − 1 ) r ib / Z ^ i HT , and with the mass at its mean Z i Z_i Z i
E ∥ 1 Z i ∑ b ( I b π b − 1 ) r i b ∥ 2 2 = 1 Z i 2 ∑ b 1 − π b π b ∥ r i b ∥ 2 2 . \mathbb{E}\left\|\frac{1}{Z_i}\sum_b\left(\frac{I_b}{\pi_b}-1\right)r_{ib}\right\|_2^2
=
\frac{1}{Z_i^2}\sum_b\frac{1-\pi_b}{\pi_b}\|r_{ib}\|_2^2. E Z i 1 b ∑ ( π b I b − 1 ) r ib 2 2 = Z i 2 1 b ∑ π b 1 − π b ∥ r ib ∥ 2 2 . Proof. By Proposition 1 with every block exact, N i = ∑ c ∣ c ∣ a i c + ∑ b Δ N i b N_i=\sum_c|c|\,a_{ic}+\sum_b\Delta N_{ib} N i = ∑ c ∣ c ∣ a i c + ∑ b Δ N ib and Z i = ∑ c ∣ c ∣ ω i c + ∑ b Δ Z i b Z_i=\sum_c|c|\,\omega_{ic}+\sum_b\Delta Z_{ib} Z i = ∑ c ∣ c ∣ ω i c + ∑ b Δ Z ib . Hence N ^ i H T − N i = ∑ b ( I b / π b − 1 ) Δ N i b \hat{N}_i^{\mathrm{HT}}-N_i=\sum_b(I_b/\pi_b-1)\Delta N_{ib} N ^ i HT − N i = ∑ b ( I b / π b − 1 ) Δ N ib and Z ^ i H T − Z i = ∑ b ( I b / π b − 1 ) Δ Z i b \hat{Z}_i^{\mathrm{HT}}-Z_i=\sum_b(I_b/\pi_b-1)\Delta Z_{ib} Z ^ i HT − Z i = ∑ b ( I b / π b − 1 ) Δ Z ib , and with N i = y i Z i N_i=y_iZ_i N i = y i Z i the error ( N ^ i H T − y i Z ^ i H T ) / Z ^ i H T (\hat{N}_i^{\mathrm{HT}}-y_i\hat{Z}_i^{\mathrm{HT}})/\hat{Z}_i^{\mathrm{HT}} ( N ^ i HT − y i Z ^ i HT ) / Z ^ i HT collects Δ N i b − y i Δ Z i b = r i b \Delta N_{ib}-y_i\Delta Z_{ib}=r_{ib} Δ N ib − y i Δ Z ib = r ib . The weights I b / π b − 1 I_b/\pi_b-1 I b / π b − 1 have zero mean, are independent, and have E ( I b / π b − 1 ) 2 = ( 1 − π b ) / π b \mathbb{E}(I_b/\pi_b-1)^2=(1-\pi_b)/\pi_b E ( I b / π b − 1 ) 2 = ( 1 − π b ) / π b . □
Without compensation the same estimator has a i c = ω i c = 0 a_{ic}=\omega_{ic}=0 a i c = ω i c = 0 , so r i b r_{ib} r ib becomes the dropped residual N i b − y i Z i b N_{ib}-y_iZ_{ib} N ib − y i Z ib and the error is driven by the attention a block holds. With
compensation it is driven by the residual r i b r_{ib} r ib that the clusters leave, which is the quantity that Eq. (9) ranks and Distortion Budget prices.
Proposition A.15 (Optimal inclusion follows the residual). Let every ∥ r i b ∥ 2 \lVert r_{ib}\rVert_2 ∥ r ib ∥ 2 be positive. Among inclusion probabilities with 0 < π b ≤ 1 0 < \pi_b \le1 0 < π b ≤ 1 and ∑ b π b = K \sum_b \pi_b = K ∑ b π b = K exact blocks in expectation, the error of Proposition A.14 is minimized by π b = min ( 1 , λ ∥ r i b ∥ 2 ) \pi_b = \min(1,\lambda\lVert r_{ib}\rVert_2) π b = min ( 1 , λ ∥ r ib ∥ 2 ) , with λ \lambda λ chosen to meet the budget.
Proof. For the blocks with π b < 1 \pi_b < 1 π b < 1 , Cauchy–Schwarz gives ( ∑ b ∥ r i b ∥ ) 2 ≤ ∑ b π b ⋅ ∑ b ∥ r i b ∥ 2 / π b \left(\sum_b \lVert r_{ib}\rVert\right)^2 \le\sum_b \pi_b \cdot\sum_b \lVert r_{ib}\rVert^2/\pi_b ( ∑ b ∥ r ib ∥ ) 2 ≤ ∑ b π b ⋅ ∑ b ∥ r ib ∥ 2 / π b , with equality exactly when π b ∝ ∥ r i b ∥ \pi_b \propto\lVert r_{ib}\rVert π b ∝ ∥ r ib ∥ . The error is Z i − 2 ( ∑ b ∥ r i b ∥ 2 / π b − ∑ b ∥ r i b ∥ 2 ) Z_i^{-2}\left(\sum_b \lVert r_{ib}\rVert^2/\pi_b-\sum_b \lVert r_{ib}\rVert^2\right) Z i − 2 ( ∑ b ∥ r ib ∥ 2 / π b − ∑ b ∥ r ib ∥ 2 ) , so it is minimized by the proportional rule. When the rule asks for π b > 1 \pi_b > 1 π b > 1 , the constraint is active, those blocks become certain, and the rule applies to the remaining budget, which is the stated form.
A block with r i b = 0 r_{ib}=0 r ib = 0 adds no error at any π b > 0 \pi_b>0 π b > 0 , so with such blocks the optimum is approached as their π b ↓ 0 \pi_b\downarrow0 π b ↓ 0 and the rule applies to the others.
The blocks with the largest residuals become certain, which is the residual ranking of Proposition A.13. DecoupAttn makes this rule deterministic, π b ∈ { 0 , 1 } \pi_b \in\{0,1\} π b ∈ { 0 , 1 } without the weights 1 / π b 1/\pi_b 1/ π b : the blocks with the largest residuals are exact, and the others are left to compensation. Its error is then the deterministic sum of the remaining residuals (Proposition 1), and Theorem A.10 shows that the deviation part of this sum adds like random errors.
The Plan Objective (Proposition 2)
Lemma A.16 (Projected distortion). For a head output error A = O ^ u − O u A=\hat{O}_u-O_u A = O ^ u − O u and the head’s slice W o , h W_{o,h} W o , h of the output projection,
∥ A W o , h ⊤ ∥ F 2 = ⟨ W o , h ⊤ W o , h , A ⊤ A ⟩ , \lVert A W_{o,h}^{\top}\rVert_F^2 = \langle W_{o,h}^{\top}W_{o,h}, A^{\top}A\rangle, ∥ A W o , h ⊤ ∥ F 2 = ⟨ W o , h ⊤ W o , h , A ⊤ A ⟩ , so ε u \varepsilon_u ε u of Eq. (10) is the expected squared error that the head adds to the residual stream, and it vanishes exactly when every row of the error lies in the null space of W o , h W_{o,h} W o , h almost surely.
Proof. ∥ A W o , h ⊤ ∥ F 2 = tr ( W o , h A ⊤ A W o , h ⊤ ) = tr ( A ⊤ A W o , h ⊤ W o , h ) \lVert AW_{o,h}^{\top}\rVert_F^2 = \operatorname{tr}(W_{o,h}A^{\top}AW_{o,h}^{\top}) = \operatorname{tr}(A^{\top}AW_{o,h}^{\top}W_{o,h}) ∥ A W o , h ⊤ ∥ F 2 = tr ( W o , h A ⊤ A W o , h ⊤ ) = tr ( A ⊤ A W o , h ⊤ W o , h ) by the cyclic property of the trace. The value is zero exactly when A W o , h ⊤ = 0 AW_{o,h}^{\top}=0 A W o , h ⊤ = 0 . □
The projected errors of one call’s heads add in the residual stream, with cross terms as in Lemma A.17.
Lemma A.17 (First-order additivity). Let the final latent be x ^ 0 = Φ ( e ) \hat{x}_0=\Phi(e) x ^ 0 = Φ ( e ) , where e = ( e u ) u e=(e_u)_u e = ( e u ) u collects the projected head errors injected at the units and Φ ( 0 ) \Phi(0) Φ ( 0 ) is the dense output. If Φ \Phi Φ is twice continuously differentiable with ∥ ∇ 2 Φ ∥ ≤ β \lVert\nabla^2\Phi\rVert\le\beta ∥ ∇ 2 Φ ∥ ≤ β on the segment from 0 0 0 to e e e , then with G u G_u G u the Jacobian of Φ \Phi Φ with respect to e u e_u e u at zero,
x ^ 0 − x 0 = ∑ u G u e u + R , ∥ R ∥ 2 ≤ β 2 ∥ e ∥ 2 2 . \hat{x}_0-x_0=\sum_u G_u e_u+R,\qquad\lVert R\rVert_2\le\frac{\beta}{2}\lVert e\rVert_2^2. x ^ 0 − x 0 = u ∑ G u e u + R , ∥ R ∥ 2 ≤ 2 β ∥ e ∥ 2 2 .
If the propagated errors of distinct units, and of the sampler steps that one unit stands for, are orthogonal in expectation, E ⟨ G u e u , G u ′ e u ′ ⟩ = 0 \mathbb{E}\langle G_u e_u,G_{u'}e_{u'}\rangle=0 E ⟨ G u e u , G u ′ e u ′ ⟩ = 0 for u ≠ u ′ u\ne u' u = u ′ , and the joint Jacobian G = ( G u ) u G=(G_u)_u G = ( G u ) u has ∥ G ∥ ≤ M \lVert G\rVert\le M ∥ G ∥ ≤ M , then
E ∥ x ^ 0 − x 0 ∥ 2 2 − ∑ u E ∥ G u e u ∥ 2 2 ≤ M β E ∥ e ∥ 2 3 + β 2 4 E ∥ e ∥ 2 4 . \mathbb{E}\lVert\hat{x}_0-x_0\rVert_2^2-\sum_u\mathbb{E}\lVert G_u e_u\rVert_2^2
\le M\beta\mathbb{E}\lVert e\rVert_2^3+\frac{\beta^2}{4}\mathbb{E}\lVert e\rVert_2^4. E ∥ x ^ 0 − x 0 ∥ 2 2 − u ∑ E ∥ G u e u ∥ 2 2 ≤ M β E ∥ e ∥ 2 3 + 4 β 2 E ∥ e ∥ 2 4 .
Proof. The first display is Taylor’s theorem with the integral remainder R = ∫ 0 1 ( 1 − s ) ∇ 2 Φ ( s e ) [ e , e ] d s R=\int_0^1(1-s)\nabla^2\Phi(se)[e,e]\,ds R = ∫ 0 1 ( 1 − s ) ∇ 2 Φ ( se ) [ e , e ] d s . Squaring it gives ∥ e ∥ 2 + 2 ⟨ G e , R ⟩ + ∥ R ∥ 2 \lVert e\rVert^2+2\langle Ge,R\rangle+\lVert R\rVert^2 ∥ e ∥ 2 + 2 ⟨ G e , R ⟩ + ∥ R ∥ 2 , the cross terms between units in ∥ G e ∥ 2 \lVert Ge\rVert^2 ∥ G e ∥ 2 vanish in expectation by assumption, and 2 ∣ ⟨ G e , R ⟩ ∣ ≤ M β ∥ e ∥ 3 2\lvert\langle Ge,R\rangle\rvert\le M\beta\lVert e\rVert^3 2 ∣⟨ G e , R ⟩∣ ≤ M β ∥ e ∥ 3 and ∥ R ∥ 2 ≤ β 2 ∥ e ∥ 4 / 4 \lVert R\rVert^2\le\beta^2\lVert e\rVert^4/4 ∥ R ∥ 2 ≤ β 2 ∥ e ∥ 4 /4 . □
Writing E ∥ G u e u ∥ 2 = J u ( o , η ) w u ε u ( o , η ) \mathbb{E}\lVert G_u e_u\rVert^2=J_u(o,\eta)w_u\varepsilon_u(o,\eta) E ∥ G u e u ∥ 2 = J u ( o , η ) w u ε u ( o , η ) defines the sensitivity of a unit under each option as its gain from projected error to final-latent error, with w u w_u w u counting the sampler steps the unit stands for. The gain depends on the option only through the direction of the error, and Distortion Budget prices all options of a unit with the one gain J u = J u ( S T A T I C θ o , F U L L ) J_u=J_u(\mathrm{STATIC}_{\theta_o},\mathrm{FULL}) J u = J u ( STATIC θ o , FULL ) measured at the reference option of Eq. (11) , so the gaps J u ( o , η ) − J u J_u(o,\eta)-J_u J u ( o , η ) − J u enter the modeling error δ m \delta_m δ m below. Lemma A.17 then turns the final-latent error of Eq. (3) into the first line of Eq. (12) . Table 21 measures directly the cross terms that the lemma sets to zero.
Lemma A.18 (The sensitivity estimator). Suppose the units of a group G \mathcal{G} G share one gain J G J_{\mathcal{G}} J G , their propagated errors are orthogonal as in Lemma A.17, the run and the dense output differ, besides the injected errors, by a zero-mean numerical error ξ \xi ξ , the run-to-run difference of the same dense attention backend, independent of the injected errors and with E ∥ ξ ∥ 2 = M S E 0 \mathbb{E}\lVert\xi\rVert^2=\mathrm{MSE}_0 E ∥ ξ ∥ 2 = MSE 0 , and the group’s summed distortion is positive. Then, to first order, Eq. (11) returns J G J_{\mathcal{G}} J G .
Proof. The run of the group gives x ^ 0 − x 0 = ∑ u ∈ G G u e u + ξ \hat{x}_0-x_0=\sum_{u\in\mathcal{G}}G_u e_u+\xi x ^ 0 − x 0 = ∑ u ∈ G G u e u + ξ up to second order. Independence and zero mean remove the cross terms with ξ \xi ξ , so
M S E G = ∑ u ∈ G E ∥ G u e u ∥ 2 2 + M S E 0 = J G ∑ u ∈ G w u ε u + M S E 0 . \mathrm{MSE}_{\mathcal{G}}=\sum_{u\in\mathcal{G}}\mathbb{E}\lVert G_u e_u\rVert_2^2+\mathrm{MSE}_0=J_{\mathcal{G}}\sum_{u\in\mathcal{G}}w_u\varepsilon_u+\mathrm{MSE}_0. MSE G = u ∈ G ∑ E ∥ G u e u ∥ 2 2 + MSE 0 = J G u ∈ G ∑ w u ε u + MSE 0 .
Subtracting M S E 0 \mathrm{MSE}_0 MSE 0 and dividing by the summed distortion leaves J G J_{\mathcal{G}} J G . □
Without the subtraction, the estimate would carry the bias M S E 0 / ∑ u w u ε u \mathrm{MSE}_0/\sum_u w_u\varepsilon_u MSE 0 / ∑ u w u ε u , which is largest exactly where two dense attention backends already disagree, on the prompt-driven few-step models. In this model M S E G ≥ M S E 0 \mathrm{MSE}_{\mathcal{G}}\ge\mathrm{MSE}_0 MSE G ≥ MSE 0 and the estimate is nonnegative, so a negative cell of Table 22 marks a group whose injected error lies below the run-to-run variation of the two measured errors.
Table 22. Stability under the dense reference (CP2.5: Cosmos-Predict2.5; T2.5: Cosmos-Transfer2.5). Correlations between the two references; negative cells: J J J cells whose floor-subtracted error is below zero under the second reference.
Lemma A.19 (Measuring one unit at a time). Let the projected error of a unit be e u = E u ( z ) e_u=E_u(z) e u = E u ( z ) , a function of the unit’s input z z z , and let the attention probe be measured at the dense input z d z_d z d , as in § 4.2 . If E u E_u E u is ℓ u \ell_u ℓ u -Lipschitz on the inputs that
occur, and a sparse run perturbs the input to z = z d + Δ z z=z_d+\Delta z z = z d + Δ z , then
∣ ( E ∥ E u ( z ) ∥ 2 2 ) 1 / 2 − ε u 1 / 2 ∣ ≤ ℓ u Δ , Δ = ( E ∥ Δ z ∥ 2 2 ) 1 / 2 . \left|\left(\mathbb{E}\lVert E_u(z)\rVert_2^2\right)^{1/2}-\varepsilon_u^{1/2}\right|\le\ell_u\Delta,\qquad\Delta=\left(\mathbb{E}\lVert\Delta z\rVert_2^2\right)^{1/2}. ( E ∥ E u ( z ) ∥ 2 2 ) 1/2 − ε u 1/2 ≤ ℓ u Δ , Δ = ( E ∥ Δ z ∥ 2 2 ) 1/2 .
and therefore
E ∥ E u ( z ) ∥ 2 2 − ε u ≤ 2 ε u 1 / 2 ℓ u Δ + ℓ u 2 Δ 2 . \mathbb{E}\lVert E_u(z)\rVert_2^2-\varepsilon_u\le2\varepsilon_u^{1/2}\ell_u\Delta+\ell_u^2\Delta^2. E ∥ E u ( z ) ∥ 2 2 − ε u ≤ 2 ε u 1/2 ℓ u Δ + ℓ u 2 Δ 2 .
Proof. ∥ E u ( z ) − E u ( z d ) ∥ ≤ ℓ u ∥ Δ z ∥ \lVert E_u(z)-E_u(z_d)\rVert\le\ell_u\lVert\Delta z\rVert ∥ E u ( z ) − E u ( z d )∥ ≤ ℓ u ∥ Δ z ∥ pointwise, and the triangle inequality in L 2 L^2 L 2 gives the first bound. The second follows from a 2 − b 2 = ( a − b ) ( a + b ) a^2-b^2=(a-b)(a+b) a 2 − b 2 = ( a − b ) ( a + b ) with a + b ≤ 2 b + ℓ u Δ a+b\le2b+\ell_u\Delta a + b ≤ 2 b + ℓ u Δ .
The deviation of the measured distortion is thus led by the product of the unit’s own error ε u 1 / 2 \varepsilon_u^{1/2} ε u 1/2 and the upstream error ℓ u Δ \ell_u\Delta ℓ u Δ , where Δ z \Delta z Δ z is the accumulated error of the units upstream and ℓ u \ell_u ℓ u bounds how fast the unit’s error changes with its input. Measuring every unit at the dense input therefore captures its distortion up to a product of two errors whenever ℓ u \ell_u ℓ u is bounded. It also keeps the measurement of a unit independent of the plan, so the distortion table is built once and every candidate plan is priced from it.
Cost of solving. For one multiplier, the per-call minimization visits every unit and option once under each of the two estimators, which costs O ( ∣ U ∣ ∣ O ∣ ) O(|\mathcal{U}||\mathcal{O}|) O ( ∣ U ∣∣ O ∣ ) operations on a table that is already measured. Bisection repeats this B μ B_\mu B μ times and the greedy pass sorts the candidate changes once, for O ( ∣ U ∣ ∣ O ∣ B μ + ∣ U ∣ log ∣ U ∣ ) O(|\mathcal{U}||\mathcal{O}|B_\mu+|\mathcal{U}|\log|\mathcal{U}|) O ( ∣ U ∣∣ O ∣ B μ + ∣ U ∣ log ∣ U ∣ ) in total. This is why plan solving takes under a minute in Table 30 , while measuring ε u \varepsilon_u ε u and J u J_u J u takes the GPU hours.
Proof of Proposition 2. Plans π \pi π (Definition 2) range over a finite set, F ( π ) = ∑ u w u J u ε u ( o u , η t l ) F(\pi)=\sum_u w_uJ_u\varepsilon_u(o_u,\eta_{tl}) F ( π ) = ∑ u w u J u ε u ( o u , η tl ) is the priced distortion, and X ( π ) = ∑ u w u X u ( o u ) / w Σ X(\pi)=\sum_u w_uX_u(o_u)/w_\Sigma X ( π ) = ∑ u w u X u ( o u ) / w Σ the density of Eq. (2) , with w Σ = ∑ u w u w_\Sigma=\sum_u w_u w Σ = ∑ u w u . For a multiplier μ ≥ 0 \mu\ge0 μ ≥ 0 , let π μ \pi_\mu π μ be a global minimizer of the Lagrangian of Eq. (12) , L μ ( π ) = F ( π ) + μ w Σ X ( π ) L_\mu(\pi)=F(\pi)+\mu w_\Sigma X(\pi) L μ ( π ) = F ( π ) + μ w Σ X ( π ) .
Exact minimization. L μ L_\mu L μ is a sum over calls ( t , l ) (t,l) ( t , l ) of ∑ h w u ( J u ε u ( o u , η t l ) + μ X u ( o u ) ) \sum_h w_u\left(J_u\varepsilon_u(o_u,\eta_{tl})+\mu X_u(o_u)\right) ∑ h w u ( J u ε u ( o u , η tl ) + μ X u ( o u ) ) , and the estimator couples only the heads of one call. Minimizing each call over η t l \eta_{tl} η tl of ∑ h min o u ( ⋅ ) \sum_h\min_{o_u}(\cdot) ∑ h min o u ( ⋅ ) therefore minimizes L μ L_\mu L μ exactly, which is the procedure of Section 4.2. Adding the two minimality inequalities shows that any minimizers at μ 1 < μ 2 \mu_1<\mu_2 μ 1 < μ 2 have X ( π μ 2 ) ≤ X ( π μ 1 ) X(\pi_{\mu_2})\le X(\pi_{\mu_1}) X ( π μ 2 ) ≤ X ( π μ 1 ) , whichever ties are taken, so bisection on μ \mu μ is well defined.
Optimality at its own density. Let π \pi π be any plan with X ( π ) ≤ X ( π μ ) X(\pi)\le X(\pi_\mu) X ( π ) ≤ X ( π μ ) . Because π μ \pi_\mu π μ minimizes L μ L_\mu L μ ,
F ( π μ ) ≤ F ( π ) + μ w Σ ( X ( π ) − X ( π μ ) ) ≤ F ( π ) , F(\pi_\mu)\le F(\pi)+\mu w_\Sigma\left(X(\pi)-X(\pi_\mu)\right)\le F(\pi), F ( π μ ) ≤ F ( π ) + μ w Σ ( X ( π ) − X ( π μ ) ) ≤ F ( π ) ,
so π μ \pi_\mu π μ has the lowest priced distortion among all plans at most as dense as itself [4 ] . Bisection returns the feasible π μ \pi_\mu π μ of largest density, and the greedy pass then lowers F F F within the remaining slack ρ − X ( π μ ) \rho-X(\pi_\mu) ρ − X ( π μ ) , which Lemma A.20(i) covers. □
Lemma A.20 (Gap of the relaxation). Let F ∗ ( ρ ) F^*(\rho) F ∗ ( ρ ) be the lowest priced distortion over plans with X ( π ) ≤ ρ X(\pi)\le\rho X ( π ) ≤ ρ . (i) If bisection returns a feasible exact minimizer π b \pi_b π b of L μ L_\mu L μ for some μ ≥ 0 \mu\ge0 μ ≥ 0 , then
F ( π b ) − F ∗ ( ρ ) ≤ μ w Σ ( ρ − X ( π b ) ) , F(\pi_b)-F^*(\rho)\le\mu w_\Sigma(\rho-X(\pi_b)), F ( π b ) − F ∗ ( ρ ) ≤ μ w Σ ( ρ − X ( π b )) ,
and every plan that the greedy pass reaches from π b \pi_b π b by feasible changes that do not raise F F F keeps this bound. (ii) Let W max W_{\max} W m a x be the largest total weight ∑ h w u \sum_h w_u ∑ h w u of one call and μ ∗ \mu^* μ ∗ the multiplier at which the minimizers of L μ L_\mu L μ cross the budget. If, at μ ∗ \mu^* μ ∗ , the minimizers are switched one call at a time, in any order, from an infeasible one toward a feasible one and the first feasible plan is kept, the bound of (i) is below μ ∗ W max \mu^*W_{\max} μ ∗ W m a x .
Proof. (i) For every feasible π \pi π , minimality of π b \pi_b π b gives F ( π ) ≥ F ( π b ) + μ w Σ ( X ( π b ) − X ( π ) ) ≥ F ( π b ) − μ w Σ ( ρ − X ( π b ) ) F(\pi)\ge F(\pi_b)+\mu w_\Sigma(X(\pi_b)-X(\pi))\ge F(\pi_b)-\mu w_\Sigma(\rho-X(\pi_b)) F ( π ) ≥ F ( π b ) + μ w Σ ( X ( π b ) − X ( π )) ≥ F ( π b ) − μ w Σ ( ρ − X ( π b )) , since X ( π ) ≤ ρ X(\pi)\le\rho X ( π ) ≤ ρ . Taking the minimum over feasible π \pi π gives the bound, and a later plan with no larger F F F inherits it. (ii) If a minimizer of L 0 = F L_0=F L 0 = F is feasible, it attains F ∗ ( ρ ) F^*(\rho) F ∗ ( ρ ) and the gap is zero. Otherwise, as a function of μ \mu μ , the set of minimizers of L μ L_\mu L μ changes only at finitely many breakpoints, and at the breakpoint μ ∗ \mu^* μ ∗ it contains a plan π − \pi_- π − with X ( π − ) > ρ X(\pi_-)> \rho X ( π − ) > ρ and a plan π + \pi_+ π + with X ( π + ) ≤ ρ X(\pi_+)\le\rho X ( π + ) ≤ ρ . Because L μ ∗ L_{\mu^*} L μ ∗ separates over calls, every plan that takes a minimizing choice in each call is again a minimizer, so π − \pi_- π − turns into π + \pi_+ π + one call at a time through minimizers. The first feasible plan of this chain follows an infeasible one that differs from it in one call, so its slack ρ − X \rho-X ρ − X is below W max / w Σ W_{\max}/w_\Sigma W m a x / w Σ , and (i) bounds its gap by μ ∗ W max \mu^*W_{\max} μ ∗ W m a x . □
The bound of (i) is known once the plan is returned, and (ii) caps it at the price of a single call’s density for a solver that switches tied minimizers one call at a time. Our solver keeps the whole minimizer at the feasible end of bisection, and on the deployed plans of the eight models it solves the bound of (i) at most 0.54% of the priced distortion (Cosmos-Predict2.5) and at most 0.094% on the others, with at most 0.026% left after the greedy pass. With every step, layer, and head of a video in the plan, one call holds a small share of w Σ w_\Sigma w Σ , so the relaxation is nearly tight.
The gap concerns the priced distortion F F F , and it carries over to the final-latent error D ( π ) = E x ∥ x ^ 0 ( π ) − x 0 ∥ 2 2 D(\pi)=\mathbb{E}_x\lVert\hat{x}_0(\pi)-x_0\rVert_2^2 D ( π ) = E x ∥ x ^ 0 ( π ) − x 0 ∥ 2 2 of Eq. (3) through the modeling error. If ∣ D ( π ) − F ( π ) ∣ ≤ δ m |D(\pi)-F(\pi)|\le\delta_m ∣ D ( π ) − F ( π ) ∣ ≤ δ m on every feasible plan and F ( π ^ ) ≤ F ∗ ( ρ ) + ϵ o F(\hat{\pi})\le F^*(\rho)+\epsilon_o F ( π ^ ) ≤ F ∗ ( ρ ) + ϵ o , then for every feasible π \pi π
D ( π ^ ) ≤ F ( π ^ ) + δ m ≤ F ( π ) + ϵ o + δ m ≤ D ( π ) + ϵ o + 2 δ m , D(\hat{\pi})\le F(\hat{\pi})+\delta_m\le F(\pi)+\epsilon_o+\delta_m\le D(\pi)+\epsilon_o+2\delta_m, D ( π ^ ) ≤ F ( π ^ ) + δ m ≤ F ( π ) + ϵ o + δ m ≤ D ( π ) + ϵ o + 2 δ m ,
so π ^ \hat{\pi} π ^ is within ϵ o + 2 δ m \epsilon_o+2\delta_m ϵ o + 2 δ m of the best feasible plan in final-latent error. Lemmas A.17–A.19 trace where δ m \delta_m δ m comes from: the cross terms between units, the shared gain of a group and of the options of a unit, and the measurement at the dense input.
Reassign Cache (Proposition 3)
Proof of Proposition 3. With the centroids m c m_c m c fixed, the total squared distance ∑ j ∥ k j ( t ) − m c ( j ) ∥ 2 2 \sum_j\lVert k_j^{(t)}-m_{c(j)}\rVert_2^2 ∑ j ∥ k j ( t ) − m c ( j ) ∥ 2 2 is a sum of one term per key, and Eq. (13) minimizes each term separately. Hence, for any other assignment c ′ c' c ′ , including the frozen assignment of the first sparse step,
∑ j ∥ k j ( t ) − m c t ( j ) ∥ 2 2 ≤ ∑ j ∥ k j ( t ) − m c ′ ( j ) ∥ 2 2 . \sum_j \lVert k_j^{(t)}-m_{c_t(j)}\rVert_2^2 \le\sum_j \lVert k_j^{(t)}-m_{c'(j)}\rVert_2^2. j ∑ ∥ k j ( t ) − m c t ( j ) ∥ 2 2 ≤ j ∑ ∥ k j ( t ) − m c ′ ( j ) ∥ 2 2 .
which is the assignment step of Lloyd’s algorithm [19 ] . The exact blocks S g S_g S g come from the offline ranking of Eq. (9) , and the options and estimators from the offline solution of Eq. (12) . Neither reads the assignments at inference, so reassignment leaves both unchanged. □
Lemma A.21 (Monotone chain with recomputed statistics). Let c t c_t c t be the assignment after reassignment at step t t t , k ˉ c ( t ) \bar{k}_{c}^{(t)} k ˉ c ( t ) the means of the reassigned clusters computed from their current keys, and c f c_f c f the frozen assignment. Then
∑ j ∥ k j ( t ) − k ˉ c t ( j ) ( t ) ∥ 2 2 ≤ ∑ j ∥ k j ( t ) − m c t ( j ) ∥ 2 2 ≤ ∑ j ∥ k j ( t ) − m c f ( j ) ∥ 2 2 . \begin{aligned}
\sum_j \lVert k_j^{(t)}-\bar{k}_{c_t(j)}^{(t)}\rVert_2^2 \le\sum_j \lVert k_j^{(t)}-m_{c_t(j)}\rVert_2^2 \\
&\le\sum_j \lVert k_j^{(t)}-m_{c_f(j)}\rVert_2^2.
\end{aligned} j ∑ ∥ k j ( t ) − k ˉ c t ( j ) ( t ) ∥ 2 2 ≤ j ∑ ∥ k j ( t ) − m c t ( j ) ∥ 2 2 ≤ j ∑ ∥ k j ( t ) − m c f ( j ) ∥ 2 2 .
Proof. Within every cluster the mean minimizes the sum of squared distances, which gives the first inequality, and Proposition 3 gives the second. □
The left side is the within-cluster spread ∑ c ∣ c ∣ tr ( C c ) \sum_c |c|\operatorname{tr}(C_c) ∑ c ∣ c ∣ tr ( C c ) that enters the compensation error. By Lemma A.4, the second-order mass term of a cluster is ∣ c ∣ ϕ i c q i T C c q i / ( 2 d ) |c|\phi_{ic}q_i^{\mathsf T}C_cq_i/(2d) ∣ c ∣ ϕ i c q i T C c q i / ( 2 d ) , and for Gaussian queries q i ∼ N ( 0 , I ) q_i\sim\mathcal{N}(0,I) q i ∼ N ( 0 , I ) its mean is ∣ c ∣ e ∥ a c ∥ 2 / 2 ( tr C c + a c T C c a c ) / ( 2 d ) |c|e^{\lVert a_c\rVert^2/2}(\operatorname{tr}C_c+a_c^{\mathsf T}C_ca_c)/(2d) ∣ c ∣ e ∥ a c ∥ 2 /2 ( tr C c + a c T C c a c ) / ( 2 d ) with a c = k ˉ c / d a_c=\bar{k}_c/\sqrt{d} a c = k ˉ c / d , which grows with the spread tr C c \operatorname{tr}C_c tr C c and with its part along the centroid. Reassignment thus lowers an upper bound on the spread, the distance to the cached centroids, at the cost of one product between the keys and the cached centroids, L C d LCd L C d multiply–adds, followed by an argmin. A cluster that loses all of its members has ∣ c ∣ = n g c = 0 |c|=n_{gc}=0 ∣ c ∣ = n g c = 0 and drops out of the estimator and of H H H , and its cached centroid stays available to later assignments. Reassignment with recomputed statistics is one step of k k k -means whose centroids for the next assignment stay at the cached m c m_c m c , so the assignments cannot drift away from the clusters that the plan was priced on. Proposition 3 concerns the distance to the cached centroids, and the Frozen ablation, which keeps the assignments and recomputes the cluster statistics, measures the quality gain of reassignment directly (Table 2 and Table 25 ).
Cost of One Call
Count multiply–adds per head for L L L tokens, head dimension d d d , C C C clusters, and density X X X (block padding included). Dense attention costs 2 L 2 d 2L^2d 2 L 2 d , for Q K T QK^{\mathsf T} Q K T and for the weighted sum of values. The exact part costs 2 X L 2 d 2XL^2d 2 X L 2 d . The compensation computes the centroid logits q i T k ˉ c q_i^{\mathsf T}\bar{k}_c q i T k ˉ c for every query and cluster (L C d LCd L C d ), the variance corrections ∑ p q i p 2 σ c p 2 \sum_p q_{ip}^2\sigma_{cp}^2 ∑ p q i p 2 σ c p 2 (L C d LCd L C d ), the numerator ∑ c n g c a i c \sum_c n_{gc}a_{ic} ∑ c n g c a i c (L C d LCd L C d ), and the products H q i Hq_i H q i (L d 2 Ld^2 L d 2 ). The statistics k ˉ c \bar{k}_c k ˉ c , v ˉ c \bar{v}_c v ˉ c , σ c 2 \sigma_c^2 σ c 2 , and H H H cost O ( L d + L d 2 ) O(Ld+Ld^2) O ( L d + L d 2 ) , reassignment costs L C d LCd L C d , and the counts n g c n_{gc} n g c are bookkeeping over the N B × C N_B\times C N B × C table of n b c n_{bc} n b c . Relative to dense attention, one call therefore costs
r = X + ( 2 + 1 F + 1 R ) C 2 L + 1 F d L . r=X+(2+1_F+1_R)\frac{C}{2L}+\mathbf{1}_F\frac{d}{L}. r = X + ( 2 + 1 F + 1 R ) 2 L C + 1 F L d .
where 1 F = 1 \mathbf{1}_F=1 1 F = 1 for the Full estimator, which adds the variance corrections and H H H , and 1 R = 1 \mathbf{1}_R=1 1 R = 1 when Reassign Cache reassigns. On Cosmos-Predict2.5, the latent has L = 24 ⋅ 44 ⋅ 80 = 84,480 L=24\cdot44\cdot80=84{,}480 L = 24 ⋅ 44 ⋅ 80 = 84 , 480 tokens, with C = 880 C=880 C = 880 and d = 128 d=128 d = 128 , so for the full method the two overhead terms add about 0.021 and 0.0015 to X X X . In multiply–adds, compensation, counts, and reassignment thus cost about one tenth of the exact part at X = 0.20 X=0.20 X = 0.20 , and the k k k -means of the first sparse step adds I ⋅ L C d I\cdot LCd I ⋅ L C d once per video for R R R rounds. If attention takes a share f f f of the latency and runs r t i m e r_{\mathrm{time}} r time times as long as dense attention, the speedup is
S = 1 ( 1 − f ) + f r t i m e . S=\frac{1}{(1-f)+fr_{\mathrm{time}}}. S = ( 1 − f ) + f r time 1 .
where the measured time ratio r t i m e r_{\mathrm{time}} r time adds to r r r the memory traffic and kernel launches that multiply–adds leave out. The share f f f differs across models, which is why one density gives different speedups in Table 1 . The measured components of Table 27 map onto these terms: Exact is 2 X L 2 d 2XL^2d 2 X L 2 d , Statistics, Counts, and Merge are the compensation, Assign is reassignment, and k k k -means is the one-time clustering.
Error Propagation Through the Sampler
The sensitivity J u J_u J u measures how much of a unit’s error reaches the final latent. This section shows where its spread across steps and models comes from, and why the disagreement between two dense attention backends proves it.
Lemma A.22 (Amplification along the sampler). Let the sampler update the latent by x s − 1 = Ψ s ( x s ) x_{s-1}=\Psi_s(x_s) x s − 1 = Ψ s ( x s ) for s = T , … , 1 s=T,\ldots,1 s = T , … , 1 , where Ψ s \Psi_s Ψ s contains the network evaluation of step s s s and is L s L_s L s -Lipschitz on the latents that occur. An error e e e added to the latent after step t t t changes the final latent by at most
∥ x ^ 0 − x 0 ∥ 2 ≤ ( ∏ s = 1 t − 1 L s ) ∥ e ∥ 2 . \lVert\hat{x}_0-x_0\rVert_2\le\left(\prod_{s=1}^{t-1}L_s\right)\lVert e\rVert_2. ∥ x ^ 0 − x 0 ∥ 2 ≤ ( s = 1 ∏ t − 1 L s ) ∥ e ∥ 2 .
If the Ψ s \Psi_s Ψ s are also differentiable along the dense trajectory, the change is, to first order, ( ∏ s = 1 t − 1 ∂ Ψ s ) e \left(\prod_{s=1}^{t-1}\partial\Psi_s\right)e ( ∏ s = 1 t − 1 ∂ Ψ s ) e , the product of the remaining step Jacobians applied to e e e .
Proof. Let x ^ t − 1 = x t − 1 + e \hat{x}_{t-1}=x_{t-1}+e x ^ t − 1 = x t − 1 + e . For every later step, ∥ x ^ s − 1 − x s − 1 ∥ = ∥ Ψ s ( x ^ s ) − Ψ s ( x s ) ∥ ≤ L s ∥ x ^ s − x s ∥ \lVert\hat{x}_{s-1}-x_{s-1}\rVert=\lVert\Psi_s(\hat{x}_s)-\Psi_s(x_s)\rVert\le L_s\lVert\hat{x}_s-x_s\rVert ∥ x ^ s − 1 − x s − 1 ∥ = ∥ Ψ s ( x ^ s ) − Ψ s ( x s )∥ ≤ L s ∥ x ^ s − x s ∥ , and induction down to s = 1 s=1 s = 1 gives the bound. The first-order form follows from the chain rule. □
An attention error e u e_u e u at step t t t enters the latent through the network evaluation inside Ψ t \Psi_t Ψ t , to first order as B u e u B_ue_u B u e u with B u B_u B u the Jacobian of Ψ t \Psi_t Ψ t with respect to the unit’s projected output, and then follows the same products, so G u = G_u= G u = ( ∏ s ′ = 1 t − 1 ∂ Ψ s ′ ) B u (\prod_{s'=1}^{t-1} \partial\Psi_{s'})B_u ( ∏ s ′ = 1 t − 1 ∂ Ψ s ′ ) B u and J u J_u J u is the squared gain of G u G_u G u in the direction of the unit’s projected error, averaged over inputs and normalized as in Lemma A.17. Few-step samplers take large steps, and each Jacobian of a large step can stretch some directions strongly, so the products, and with them the sensitivities, differ by orders of magnitude across steps and models. A per-pixel condition anchors the output to the input, so the step maps contract the directions the condition determines and the products stay small.
Corollary A.23 (The backend floor probes the same products). Let two dense attention backends differ at step s s s by a numerical error ξ s \xi_s ξ s , the difference of their updates of x s − 1 x_{s-1} x s − 1 , which given x s x_s x s has zero mean and covariance Σ s \Sigma_s Σ s and is uncorrelated across steps. Then, to first order and with P s = ∏ s ′ = 1 s − 1 ∂ Ψ s ′ P_s = \prod_{s'=1}^{s-1} \partial\Psi_{s'} P s = ∏ s ′ = 1 s − 1 ∂ Ψ s ′ the Jacobian product from x s − 1 x_{s-1} x s − 1 to the output along the dense trajectory,
E ∥ x 0 ( 1 ) − x 0 ( 2 ) ∥ 2 2 = ∑ s tr ( P s Σ s P s T ) \mathbb{E}\left\|x_0^{(1)}-x_0^{(2)}\right\|_2^2 = \sum_s \operatorname{tr}\left(P_s\Sigma_sP_s^{\mathsf{T}}\right) E x 0 ( 1 ) − x 0 ( 2 ) 2 2 = s ∑ tr ( P s Σ s P s T ) ≤ ∑ s ( ∏ s ′ = 1 s − 1 L s ′ ) 2 tr Σ s . \le\sum_s \left(\prod_{s'=1}^{s-1} L_{s'}\right)^2 \operatorname{tr}\Sigma_s. ≤ s ∑ ( s ′ = 1 ∏ s − 1 L s ′ ) 2 tr Σ s .
Proof. By Lemma A.22, each ξ s \xi_s ξ s reaches the output as P s ξ s P_s\xi_s P s ξ s to first order, and E ∥ P s ξ s ∥ 2 2 = tr ( P s Σ s P s T ) \mathbb{E}\|P_s\xi_s\|_2^2 = \operatorname{tr}(P_s\Sigma_sP_s^{\mathsf{T}}) E ∥ P s ξ s ∥ 2 2 = tr ( P s Σ s P s T ) . The terms between steps vanish, and ∥ P s ∥ ≤ ∏ s ′ < s L s ′ \|P_s\| \le\prod_{s'<s} L_{s'} ∥ P s ∥ ≤ ∏ s ′ < s L s ′ bounds each term. □
The floor weights the Jacobian products by the covariance of the numerical noise, and the sensitivities weight the same products by the directions of the sparse errors, so the floor probes the sensitivities. The Floor of Table 1 reads this disagreement on the decoded frames, as a PSNR averaged over clips. Figure 4 d–Figure 4 f and Table 12 measure this: the cost of removing the sensitivity falls as the floor rises, with rank correlation − 0.61 -0.61 − 0.61 on ten models.
Table 12. Spread of the sensitivity (coefficient of variation over the cells the plan uses), the cost of removing it (Table 2 ), and the dense-backend floor column (Table 1 ).
Constants and Implementation
Shared constants. All ten models use the constants of Table 4 , and what differs between them is measured on their own clips: the block shape, the block ranking D u D_u D u , the distortion table ϵ u \epsilon_u ϵ u , the sensitivity J u J_u J u , and the assignment rule of Eq. (14) . The per-model plans come from Distortion Budget’s measurements, not tuning, so one procedure serves every family and sampler (§ 5.2 ).
Table 4. Constants of DecoupAttn, shared by all models.
Block shape. Five block shapes are scored by the attention density-0.20 static mask kept on eight calibration clips; the default and the best challenger are compared on eight validation clips, a gap of at least 0.1 dB decides, and otherwise the faster shape is kept.
Sensitivity groups. J J J is measured on groups of sampler steps (one step for four-step models) and one third of the layers, run at level θ 0 \theta_0 θ 0 with full compensation while the rest stays dense. Every unit in a group shares its J u J_u J u , and J J J is normalized to mean one, so J u > 1 J_u > 1 J u > 1 marks a unit whose errors reach the final latent more than the average unit’s. Cells that the floor subtraction leaves negative are set to zero before this normalization, and for the models measured by injection a group without a positive excess stops the plan build. No cell of the deployed plans is affected under the reference backend. DiffusionRenderer uses one weight per step from an earlier sensitivity map of the same quantity.
Speed check. The implementation also measures S max S_{\max} S m a x , the dense latency divided by the lowest latency any sparse plan reaches, and runs dense attention when S max < τ g S_{\max} < \tau_g S m a x < τ g . The experiments in this paper do not use this branch.
Kernels. The exact blocks run on FlashInfer’s variable-block FlashAttention-3 kernel. Each call computes the cluster means, the per-dimension key variances, and the matrix H H H (two GEMMs) from the current keys and values, and a fused Triton kernel adds the cluster terms of Eq. (5) and merges them with the exact part through the log-sum-exp. Centroid-only calls skip the variance pass and H H H .
Experimental Details
Models. Table 5 covers five four-step distilled samplers and five samplers with 35 or 50 steps, four of them with classifier-free guidance, with 8 to 39 test clips per model. Both sampler types run with the constants of Table 4 , so what adapts to each model is its calibrated plan.
Density accounting and baseline settings. Density counts every computed block after padding to the kernel block size, with dense calls counted as one. Baselines run from their official code with their density knob set to 0.20; where the official configuration sets a higher per-head minimum, the baseline runs at that minimum (SVG2 and SVG-EAR at 0.225 and 0.224 on Wan2.2-A14B). On Cosmos-Predict1 and UniRelight, which have no official CalibAtt schedule, its energy schedule follows the official 50-step Wan schedule, scaled to density 0.20 on the calibration set.
Timing. Each timing process runs dense attention, the method, and dense attention again, with one warm-up and one timed generation each, measured with CUDA events; latency is the wall-clock time of the denoising loop, and the speedup is the mean dense latency divided by that of the method. Plan building, clustering, and reassignment are all counted inside the measured latency.
Ablation variants. Tied groups compensation by the selection blocks; Centroid drops both corrections; U sets every J u J_u J u to one; D replaces ε u \varepsilon_u ε u by the error of the same option without compensation; Frozen keeps the assignments of the first sparse step on the nine models where Eq. (14) reassigns them; Rebuild runs k-means at every sparse step (Table 9). Tied{sup}†applies the same grouping swap under our plan without re-solving, and every other variant is re-solved at density 0.20. DiffusionRenderer already keeps its assignments under the rule of {xref}Eq. (14), where reassigning them changes PSNR by +0.05 dB. In {xref}Table 6` the Full column gives the full method’s PSNR with its standard deviation across clips.
Ablation results. Table 6 adds the spread and the clip count behind each cell of Table 2 . The decoupling variants lose most consistently: Tied and Tied{sup}†lose on at least six of the eight clips of every model and Centroid on at least seven, since all three enlarge the residual that every query carries ({xref}Proposition 1). The cost of U depends more on the model, because it follows how unevenly the sampler amplifies the errors of different units ({xref}Table 12): U costs 4.10 dB on HunyuanVideo but 0.55 dB on Cosmos-Transfer2.5, whose edges damp the error growth ({xref}Figure 4d,e`). Frozen costs at most 1.14 dB on the video models and most on AlayaRenderer and UniRelight (2.59 and 4.45 dB).
Table 6. Component ablations with standard deviations and clip counts (full version of Table 2 ). Each cell is the PSNR of the variant minus that of the full method (dB) on eight validation clips, with the standard deviation of this difference (small ± \pm ± ) and the number of clips on which the full method is better. Variants are defined in Appendix § 10 , their latencies are in Table 2 , and their speedups are in Table 8 .
Speed of the variants. Table 8 shows what each component costs in time. U and D change only the prices, so their speedup stays within 0.03 of the full method’s, and Distortion Budget gains its 1.18–1.68 dB at no cost in speed. Tied and Tied{sup}†group by the selection blocks and skip the clustering, which makes them slightly faster (0.04–0.15 in speedup) but costs 0.85–8.59 dB ({xref}Table 6), so the semantic clusters of {xref}Section 4.1` are worth their time. Frozen saves the per-step assignment, at most 0.08 in speedup, and Rebuild, which runs k-means at every sparse step, falls below dense speed on five of the ten models.
Re-clustering and reassignment. Rebuild changes PSNR by at most 0.95 dB (UniRelight) and by less than 0.4 dB on the other nine models, while adding 0.8–96.1 s per video (Table 9). Reassignment already moves every key to its nearest cached centroid (Eq. (13) ), which never increases the spread (Proposition 3), so fresh centroids gain little, and the full method wins on five of the eight clips of HunyuanVideo and Cosmos3-Nano. In the timing session of Table 10 , reassignment adds 108–227 ms (2–3%) over frozen assignments, whereas Rebuild adds 0.8–1.8 s (14–21%) and makes DiffusionRenderer and Wan2.1-14B slower than dense attention, which is why Reassign Cache keeps the cached centroids instead.
Table 10. Latency (ms) of the three assignment schemes in one timing session; dense is 4,329, 12,170, and 13,747 ms.
VBench. Table 7 scores each video on its own, without the dense output. DecoupAttn stays within 0.007 of dense attention on every model and is the best or tied-best sparse method on four of them, and on Cosmos3-Nano, where SVOO and SVG-EAR fall to 0.7988 and 0.8100, it keeps the dense score (0.8616 against 0.8612).
Coherence measurement. Table 11 takes the (clip, step, layer, head) units of the deployed plans on four calibration clips with seed 1000 (conditional branch, sparse steps, layers after the first): 10,040 units on DiffusionRenderer, 17,968 on Wan2.1-14B, and 4,948 on Cosmos-Predict2.5. Each unit pools the sums over its queries, κ = ∑ i ∥ ∑ b r i b ∥ 2 2 / ∑ i ∑ b ∥ r i b ∥ 2 2 \kappa= \sum_i \lVert\sum_b r_{ib} \rVert_2^2 / \sum_i \sum_b \lVert r_{ib} \rVert_2^2 κ = ∑ i ∥ ∑ b r ib ∥ 2 2 / ∑ i ∑ b ∥ r ib ∥ 2 2 , in value space with dense probabilities (Z i = 1 Z_i = 1 Z i = 1 ), so high-energy queries weigh most. The normalized ratio ζ = ( κ − 1 ) / ( K g − 1 ) \zeta= (\kappa- 1)/(K_g - 1) ζ = ( κ − 1 ) / ( K g − 1 ) divides out the number K g K_g K g of compensated key blocks b ∈ S g b \in\mathcal{S}_g b ∈ S g per query (medians 198, 319, and 594) and equals the mean pairwise cosine of the residuals when their norms are equal. Compensation lowers κ \kappa κ on all three models, and the corrections lower it further: the median falls 2.9–3.9-fold, from 4.69–10.37 without compensation to 2.67–5.12 with plain centroids and 1.61–2.69 with DecoupAttn, and the median ζ \zeta ζ falls to 0.002–0.003, so the residuals of a few hundred compensated blocks are nearly orthogonal. This is the cancellation of Proposition 1: a compensated residual comes from member deviations that sum to zero over each cluster,
so the skipped blocks share almost no common direction, and blocks fixed offline suffice (§ 5.3 ).
Sensitivity spread. Table 12 relates the cost of U U U to how unevenly J J J spreads over the cells a plan uses. On the four-step video models the cost tracks the spread (coefficient of variation 1.11–2.23, cost 1.06–2.30 dB). A flat J J J misprices units when their true sensitivities differ. Cosmos-Transfer2.5 and UniRelight spread J J J widely yet lose less than 0.9 dB without it, and both have high dense-backend floors (41.93 and 55.49 dB): their conditions damp the error growth along the sampler, which leaves less for the sensitivity to steer (Figure 4 d–Figure 4 f).
Supplementary Analyses
This appendix isolates each mechanism of § 4 and tests whether its gain carries over to matched latency, unseen inputs, and task metrics. Every comparison shares prompts, seeds, conditioning inputs, and the dense reference, and runs on clips that took no part in calibration or validation, after every variant is fixed; Table 13 lists the models that each analysis uses. Cosmos-Predict2.5 is in every analysis, and the other models add another video family (Wan2.1-14B), many- and four-step rendering (UniRelight, DiffusionRenderer), and edge control (Cosmos-Transfer2.5), and the task metrics cover nine of the ten models, all but UniRelight.
Table 13. Models of each supplementary analysis , by the subsections of Appendix D that use them.
Statistics. Mechanism diagnostics record the queries, keys, and values of selected (step, layer, head) units on eight held-out clips without storing attention maps, plan screening uses eight held-out clips, and the final comparisons use 8 to 39 prompts or conditions per model with one seed each. We report paired differences with 95% bootstrap confidence intervals, resampling prompts, together with the win rate and the worst-decile difference; frames, heads, and layers are never counted as independent samples.
Density Accounting
Table 14 separates the always-dense first step and first layer from the units the plan sparsifies: f d e n s e f_{\mathrm{dense}} f dense is their share of the attention work, X s p X_{\mathrm{sp}} X sp is the density of the remaining units, and the whole run computes
ρ g l o b a l = f d e n s e + ( 1 − f d e n s e ) X s p . \rho_{\mathrm{global}} = f_{\mathrm{dense}} + (1 - f_{\mathrm{dense}}) X_{\mathrm{sp}}. ρ global = f dense + ( 1 − f dense ) X sp .
Both guidance branches of the CFG models, the first step, the first layer, and every diagonal block count as computed, and every block is padded to the kernel block size. All ten plans spend density 0.200 on the units they sparsify, and the global density differs only through the dense share: the first step and the first layer take 27–28% of the attention work on the four-step models but 3.6–6.3% on the many-step models, giving global densities of 0.415–0.421 and 0.229–0.251. All methods keep these parts dense (§ 5.1 ), so this share bounds every method’s speedup.
Decoupling
Grouping under a fixed mask. Table 15 holds the exact mask, the per-unit exact budget, the block size, the assignment scheme, and the precision fixed, and changes only how the skipped keys are summarized: by nothing, by position (the selection blocks), or by semantic clusters, each with plain centroids or with both corrections of Eq. (4) . Figure 6 varies the number of groups of both schemes and plots the projected attention error against the compensation time. With the mask fixed and plain centroids, semantic clusters raise PSNR over position groups by 5.11 dB on Cosmos-Predict2.5 and 3.72 dB on Wan2.1-14B (Table 15 ), and with both corrections they lower the projected error at every group count in the diagnostic implementation of Figure 6 , 3.6-fold and 7.1-fold at the deployed 880 clusters. The corrections help both groupings, yet position groups with both corrections (22.33 and 20.66 dB) remain below semantic clusters without them (23.47 and 23.41 dB): the corrections repair the mass and the first-order value of each group but cannot pull far-apart members together, and the spread of the members is what sets the residual (Proposition A.6). The semantic clusters cost 0.7–1.4 s more DiT time than position groups, which buys 2.37 and 4.04 dB with both corrections on, so decoupling the grouping from the selection pays on both models.
Table 15. Compensation grouping under a fixed exact mask. Only the summary of the skipped keys changes, A computes the exact blocks alone, and Both adds the variance and value corrections of Eq. (4) . Small ±: standard deviation of PSNR over the test prompts.
Selection under a fixed compensator. Table 16 keeps the semantic compensator, the cluster assignments, the per-unit exact budget, and the kernels fixed and changes only how the exact blocks are chosen. Online residual selection ranks the blocks of each call by their residual under that call’s dense attention, information that no deployed method reads. The last row attaches the same compensator to the masks of CalibAtt [43 ] . Under one compensator, ranking blocks by their residual beats ranking them by attention mass by 0.32 and 0.33 dB at the same cost (Table 16 ), because the plan keeps exact the blocks that compensation handles worst, while a block with high attention mass may be one the centroids already predict well. Routing by mass at every call gains 0.27 dB over offline mass ranking on Cosmos-Predict2.5 for 650 ms of routing, loses 0.42 dB on Wan2.1-14B for 1,288 ms, and stays below our offline blocks on both, since with every key compensated the residuals cancel, and the remaining error depends on the blocks that compensation handles worst, which calibration measures once (Proposition 1). The blocks themselves matter: random blocks lose 7.44 and 8.34 dB, and the masks of CalibAtt reach 21.22 and 20.79 dB under our compensator, 3.48 and 3.91 dB below our blocks. The diagnostic online residual gains 2.33–3.27 dB at 86–143 times the latency, while our fixed blocks add no routing time.
Compensation Corrections
Factorial ablation. Table 17 switches the variance correction γ i c \gamma_{ic} γ i c and the first-order value correction H H H of Eq. (4) on and off. Under a fixed plan, the exact mask, the per-unit options, the assignments, and the set of FULL calls are held, so the solver cannot route around a removed term; under a re-solved plan, every variant is re-solved at the same density. The mass error ∣ Z ~ i / Z i − 1 ∣ \lvert\tilde{Z}_i/Z_i-1\rvert ∣ Z ~ i / Z i − 1 ∣ isolates what a centroid misses in the denominator, and the projected error of Eq. (10) measures the resulting output error. Under a fixed plan, each correction alone moves PSNR by −0.16 to +0.28 dB, but together they add 1.23 and 1.29 dB (Table 17 ). The variance correction restores the mass that a centroid misses and cuts the median mass error about threefold (0.0512 to 0.0162 and 0.0641 to 0.0228), and the value correction fixes the numerator; alone, each changes the latent error little (0.0541 to 0.0560 and 0.0546 on Cosmos-Predict2.5, 0.0364 to 0.0362 and 0.0325 on Wan2.1-14B), while together they lower it to 0.0412 and 0.0260. Under re-solved plans each correction alone already recovers 0.76–1.16 dB, because the solver moves exact blocks to the units that the missing term hurts most, so the fixed-plan rows measure the terms themselves, and both terms of Eq. (4) are needed.
Table 17. Variance and value corrections , 2×2. Fixed holds the exact mask, options, assignments, and FULL calls, re-solved re-solves each variant at equal density, and mass errors are query medians and 99th percentiles. Small ±: standard deviation over test prompts.
Where each correction acts. Figure 7 bins clusters by key variance and by key–value covariance and plots the residual energy each correction removes from the clusters of each bin. The variance correction removes 10–25% of the centroid error in every bin. The value correction applies the head average H H H to every cluster, so on the clusters whose own B c B_c B c is smallest the gap B c − H B_c-H B c − H of Proposition A.6 exceeds B c B_c B c and the correction adds error, and elsewhere it changes the per-cluster error by less than 1%. Over the whole output it lowers the projected attention error from 0.0108 to 0.00916 under a fixed plan (Table 17 ), and a per-cluster H c H_c H c closes the gap (Table 18 ). Table 18 replaces the diagonal variance by the full covariance and the shared H H H by one H c H_c H c per cluster on saved attention inputs. Full covariance lowers the attention error by only 5% (0.0059 to 0.00559), so the diagonal variance already captures nearly all the mass a centroid misses, whereas one H c H_c H c per cluster lowers it by 27% (to 0.00428), in line with Figure 7 b. The
Figure 7. Where the corrections act (Cosmos-Predict2.5). Per-cluster residual energy removed by the variance correction, binned by cluster key variance (a), and by the value correction, binned by key–value covariance (b). Each bin holds one eighth of the clusters, and a negative bar means the correction adds error.
deployed compensator keeps the diagonal variance and the shared H H H of Eq. (4) at two GEMMs per call, and H c H_c H c extends the same correction.
Distortion Budget
Pricing and sensitivity. Table 19 crosses the error that prices a unit, measured before or after compensation, with the sensitivity, set to one or measured, at one compensator, block candidate set, assignment scheme, and density. Shuffling J J J across units while keeping its values tests whether the gain comes from locating sensitive units, and pricing the head output without the W o W_o W o projection of Eq. (10) tests the projection. The sensitivity also pays at the density 0.10 of Figure 5 b–Figure 5 d, where setting it to one costs 1.54 dB on Cosmos-Predict2.5. Two time-only rows compare the curvature schedule of PASA [48 ] , ported into DecoupAttn, with Distortion Budget restricted to one weight per step, and the density of every row is re-measured after padding. Both factors of the price J u ε u J u_{\varepsilon} u J u ε u matter, and they pay most together (Table 19 ): on Cosmos-Predict2.5, pricing the compensated error gains 0.41 dB over the uncompensated error with J = 1 J = 1 J = 1 , the measured J J J gains 1.94 dB, and both together gain 2.99 dB (24.70 against 21.71 dB), with the same ordering on Cosmos-Transfer2.5. Shuffling J J J keeps its values but not their places and falls to 20.05 dB, below J = 1 J = 1 J = 1 , so the gain comes from locating the units whose errors reach the final latent. Pricing the head output without W o W_o W o changes PSNR by at most 0.14 dB, and the two time-only schedules reach 21.79 and 22.01 dB, near J = 1 J = 1 J = 1 , so most of the gain comes from pricing layers and heads, not steps alone.
Table 19. Pricing error and sensitivity. U and D are the ablations of Table 2 under this protocol, and density is measured after padding. Cosmos-Transfer2.5 comes from one rerun with saved latents, in which the full method reaches 33.90 dB (33.81 dB in Table 1 ).
Predicted versus actual distortion. For each model, 21 candidate plans at one density (the deployed plan, uniform allocation, compensation error alone, a time-only schedule, small budget swaps of the deployed plan, and the ablation plans) are priced on the calibration set by
D ^ ( π ) = ∑ u w u J u ε u ( o u , η u ) , \hat{D}(\pi)=\sum_u w_u J_u\varepsilon_u(o_u,\eta_u), D ^ ( π ) = u ∑ w u J u ε u ( o u , η u ) ,
and their actual distortion
D a c t u a l ( π ) = E x ∥ x ^ 0 ( π ) − x 0 ∥ 2 2 D_{\mathrm{actual}}(\pi)=\mathbb{E}_x\lVert\hat{x}_0(\pi)-x_0\rVert_2^2 D actual ( π ) = E x ∥ x ^ 0 ( π ) − x 0 ∥ 2 2
is measured on held-out clips (Figure 8 ). Table 20 reports the rank correlations within the budget, the actual rank of the plan the objective selects, and its regret against the best candidate. On both models the price ranks the 21 plans nearly as the held-out error does (Spearman 0.944 and 0.917), and the plan the objective selects is the second best, 0.6% and 1.2% above the best candidate, as Proposition 2 expects: the solver minimizes the price, and the price follows the actual distortion.
Additivity. Eq. (12) sums the priced distortion over units. For two groups of units A A A and B B B , sparse error is injected into A A A alone, B B B alone, and both, giving final-latent errors e A e_A e A , e B e_B e B , and e A B e_{AB} e A B ; Table 21 reports the nonlinearity ∥ e A B − e A − e B ∥ 2 / ( ∥ e A ∥ 2 + ∥ e B ∥ 2 ) \lVert e_{AB}-e_A-e_B\rVert_2/(\lVert e_A\rVert_2+\lVert e_B\rVert_2) ∥ e A B − e A − e B ∥ 2 / (∥ e A ∥ 2 + ∥ e B ∥ 2 ) and the cross term 2 e A e B / ( ∥ e A ∥ 2 + ∥ e B ∥ 2 ) 2e_Ae_B/(\lVert e_A\rVert_2+\lVert e_B\rVert_2) 2 e A e B / (∥ e A ∥ 2 + ∥ e B ∥ 2 ) that summed squared errors omit. Errors of different unit groups interact moderately: the cross term is 0.12–0.45 of the summed energies and smallest between the first and the last sparse step (Table 21 ). The summed price ranks the candidate plans as their held-out error does (Table 20 ), all the plan choice of Eq. (12) needs.
Table 21. Interaction between unit groups in the final latent (CP2.5: Cosmos-Predict2.5; T2.5: Cosmos-Transfer2.5). Mean over the test clips (32 and 39), small ± \pm ± : standard deviation.
Reference backend. The sensitivity of Eq. (11) subtracts the run-to-run floor of the same dense attention backend; Table 22 recomputes J J J and the method ranking against a second dense attention backend as reference and counts the cells where the subtraction turns negative. J J J keeps its order under the second reference (Spearman 0.884 and 0.967), the method ranking stays the same (Kendall 1.000), and at most 6 of 81 cells turn negative (Table 22 ), so Distortion Budget measures the sensitivity of the model itself.
Residual Structure
Exchangeability of the cluster members. Assumption (A4) of Appendix § 8 .§ 1 treats the block a cluster member falls into as independent of its deviation. Figure 9 tests it by the mean pairwise cosine of the skipped blocks’ residuals, and of their deviation terms D b D_b D b of Proposition A.8 , against their distance. Theorem A.10 bounds the coherence of the deviation terms by one under (A4), and the measured deviation terms add like random errors: their median coherence ratio is 0.97–1.72, against 4.86–8.25 for the residuals without compensation (Table 23 ), and their correlation decays with the distance between the blocks (Figure 9 ). The cluster bias β i b = r i b − D b \beta_{ib}=r_{ib}-D_b β ib = r ib − D b is the coherent part of the residual, and the corrections shrink it from 0.73–1.34 to 0.54–0.83 of the residual norm, with the variance correction giving nearly all of this reduction (Table 23 ).
Table 23. Coherence ratio and cluster bias on saved attention inputs: medians over queries of the coherence ratio of D b D_b D b and of the residual r i b r_{ib} r ib , and over skipped blocks of ∥ β i b ∥ 2 / ∥ r i b ∥ 2 \lVert\beta_{ib}\rVert_2/\lVert r_{ib}\rVert_2 ∥ β ib ∥ 2 / ∥ r ib ∥ 2 .
Energy, coherence, and mass. Eqs. (6) and (7) give
∥ y ^ i − y i ∥ 2 = κ i R i / Z ^ i 2 \lVert\hat{y}_i-y_i\rVert^2=\kappa_i R_i/\hat{Z}_i^2 ∥ y ^ i − y i ∥ 2 = κ i R i / Z ^ i 2
with R i = ∑ b ∉ S g ∥ r i b ∥ 2 R_i=\sum_{b\notin S_g}\lVert r_{ib}\rVert^2 R i = ∑ b ∈ / S g ∥ r ib ∥ 2 , so the output error splits into residual energy, coherence, and estimated mass. Table 24 reports the three factors and the error for five compensators on the same held-out inputs and masks, next to the gap between fixed blocks and online routing of Table 16 . By Eq. (7) , κ i \kappa_i κ i grows with the pairwise cosine of the skipped-block residuals, and Figure 10 shows that compensation moves the median cosine from 0.31–0.36 to 0.012–0.027. Unlike Table 11 , κ i \kappa_i κ i is taken per query through the deployed bf16 kernel with the error normalized by the estimated mass Z ^ i \hat{Z}_i Z ^ i , the residual energy is normalized as R ~ i = R i / ( Z ^ i 2 ∥ y i ∥ 2 ) \tilde{R}_i=R_i/(\hat{Z}_i^2\lVert y_i\rVert^2) R ~ i = R i / ( Z ^ i 2 ∥ y i ∥ 2 ) , and the error is ∥ y ^ i − y i ∥ 2 / ∥ y i ∥ 2 \lVert\hat{y}_i-y_i\rVert^2/\lVert y_i\rVert^2 ∥ y ^ i − y i ∥ 2 / ∥ y i ∥ 2 . Compensation lowers all three factors, and the corrections act mainly on coherence and mass (Table 24 ): from plain centroids to both corrections, the residual energy falls by at most 10%, while κ i \kappa_i κ i falls from 2.73–4.43 to 1.56–2.29 and the mass error from 0.051–0.100 to 0.016–0.035, so the error falls 2.2–2.6-fold. The variance correction fixes the mass and also lowers κ i \kappa_i κ i , the value correction lowers κ i \kappa_i κ i without changing the mass, and both together reach the lowest error on all three models, as Eq. (7) predicts. Figure 10 shows these shifts across the whole distributions: with both corrections, κ i \kappa_i κ i and Z ^ i / Z i \hat{Z}_i/Z_i Z ^ i / Z i concentrate closer to one, the error falls at every quantile from the 25th to the 99th, and the distribution of the residual energy stays in place.
Figure 10. Distributions of the error factors of Table 24 for its five compensators: residual energy R ~ i \widetilde{R}_i R i , coherence κ i \kappa_i κ i , mass ratio Z ^ i / Z i \widehat{Z}_i/Z_i Z i / Z i , output error, and the cosine between the residuals of two skipped blocks of the same query tile (held-out clips, 10,000 queries and 100,000 random pairs per compensator). Bars: interquartile range, dots: medians.
Reassign Cache
Assignment update schemes. Every scheme in Table 25 recomputes the cluster statistics from the current keys and values (Eq. (4) ) and runs one exact plan; the schemes differ only in the cluster assignments. Frozen keeps the assignments of the first sparse step, Reassign assigns keys to the cached centroids (Eq. (13) ), Rebuild runs k-means at every sparse step, and Periodic rebuild runs it every 5 steps on the many-step models. Figure 11 follows Frozen, Reassign, and Rebuild across the sparse steps: the distance to the cached centroids (the quantity of Proposition 3), the within-cluster error to the current means, the projected attention error, and the share of keys whose assignment changes. Reassignment keeps the distance to the cached centroids at or below its frozen value, as Proposition 3 states, and keeps each cluster tight around its current mean, so its attention error stays close to Rebuild’s and far below Frozen’s. Reassign recovers most of what Rebuild gains over Frozen at a fraction of its cost (Table 25 ): on UniRelight it lifts PSNR from 39.13 to 42.75 dB against 43.18 dB for Rebuild, changes 22.5% of the assignments per step against 37.0%, and adds 4.8 s of DiT time instead of 36.7 s; on Cosmos-Predict2.5 it recovers 0.47 of Rebuild’s 0.52 dB for 0.15 s instead of 1.09 s. Periodic rebuild reaches Rebuild’s accuracy (43.16 dB) for 10.0 s, still twice the cost of Reassign, and on DiffusionRenderer, where stale assignments cost only 0.46 dB on calibration clips (Table 26 ), the three schemes lie within 0.07 dB of each other.
Deployment rule. Table 26 checks the choices of Eq. (14) on test clips: the staleness measured on calibration clips, the scheme the rule selects, the test PSNR gain and latency cost of that scheme over the other, and whether the staleness measured on test clips falls on the same side of τ s \tau_s τ s . On test clips the selected scheme gains 0.14–3.63 dB on all nine models that reassign, and DiffusionRenderer, which keeps its assignments, gives up 0.06 dB for 102 ms. The calibration staleness overstates the test gain, so the two fall on the same side of τ s \tau_s τ s on five of the ten models.
Efficiency
Quality–latency curves. Each method runs at four or five operating points of its own density knob, with the density of every point measured. Figure 5 b–Figure 5 d plots PSNR against DiT speedup, and Figure 12 plots LPIPS against end-to-end latency for the same points. In LPIPS the picture is the same: at every latency of DecoupAttn all baselines lie above its curve, and the lower LPIPS of SVG-EAR and SVG2 at top-p p p 0.99 costs 1.5–1.8 times the dense end-to-end time. Table 29 reads two numbers off them: the PSNR gain at the latency of the strongest baseline, and the speedup at a target PSNR fixed before testing. At the latency of the strongest baseline, DecoupAttn is 5.74–8.40 dB more accurate, and the strongest baseline does not reach our deployed PSNR at any of its operating points, so its speedup at equal PSNR is undefined (Table 29 ).
baseline, DecoupAttn is 5.74–8.40 dB more accurate, and the strongest baseline does not reach our deployed PSNR at any of its operating points, so its speedup at equal PSNR is undefined (Table 29 ).
Time breakdown. Table 27 splits the time of one video into attention (exact blocks, first k-means, assignment, cluster statistics, block–cluster counts, and the fused merge), the DiT, and the full generation (encoders, DiT, VAE decoding, and transfers). Each entry is the median of five interleaved runs on one NVIDIA H800 GPU with batch size 1 and bf16 precision, without compilation or cold start; Table 30 accounts offline calibration, plan solving, and plan loading separately and gives the break-even count
N_{\mathrm{break\mathchar`-even}}=\left\lceil\frac{T_{\mathrm{calibration}}}{T_{\mathrm{dense,E2E}}-T_{\mathrm{ours,E2E}}}\right\rceil,
the smallest number of videos whose savings repay the calibration, for T o u r s , E 2 E < T d e n s e , E 2 E T_{\mathrm{ours,E2E}}<T_{\mathrm{dense,E2E}} T ours , E2E < T dense , E2E .
Where the time goes. The gap to SVG-EAR comes mostly from the per-step work (Table 27 ): SVG-EAR re-clusters at every step and rebuilds its error map and top-p p p selection, which costs 4.7–18.4 s of k-means and 0.8–17.6 s of selection per video, while DecoupAttn clusters once (0.6–1.2 s) and later only reassigns, counts, and merges. Its attention therefore takes 1.1–1.7 times less time than dense attention and 1.9–2.8 times less than SVG-EAR’s, and every video finishes faster end to end. Calibration takes 5.3–19.6 GPU-
hours and the plan solve under a minute (Table 30 ), and the break-even count follows the end-to-end saving per video:
UniRelight, which saves the most, breaks even after 2.1k videos, and Wan2.1-14B, which saves the least, after 23.7k.
Generalization of the Offline Plan
Calibration set size. Table 31 calibrates on 1, 2, 4, 8, and 16 training clips with three draws each and reports the test PSNR, the intersection over union of the exact blocks with those of the 16-clip plan, and the calibration time. PSNR rises from 24.59 to 24.87 dB as the set grows from 1 to 16 clips, and its spread over draws shrinks from 0.22 dB with one clip to 0.04 dB with eight (Table 31 ); with the deployed four clips, the exact blocks already overlap 94.8% with the 16-clip plan after 0.33 GPU-hours of error calibration.
Table 31. Calibration set size (Cosmos-Predict2.5; mean ± \pm ± standard deviation over three draws). Only the error calibration uses the drawn clips, and the sensitivity and the distortion table are those of the deployed plan. The 4-clip draws are training clips 0–3, 4–7, and 8–11, and the first is the deployed plan. Block overlap: intersection over union of the exact blocks with the 16-clip plan. Calibration counts error calibration alone (301 s per clip), while Table 30 adds block ranking and the distortion table.
Unseen inputs and settings. With the plan frozen, Table 32 groups held-out prompts by content and reports the mean and the worst decile. The plan is tied to the token grid of one resolution, so a new resolution runs a new calibration, and Table 33 reports its cost for Wan2.1-14B at 720p. The frozen plan leads SVG-EAR on every content group by 1.22–3.79 dB, including categories absent from calibration (+3.39 dB) and complex lighting (+3.79 dB, Table 32 ). Its worst decile stays at 19.4–27.6 dB in every group (Table 32 ). The plan fixes only where exact attention runs, while each video clusters its own keys at its first sparse step, so new content changes the compensation, not the plan. At 720p the recalibrated plan reaches 23.89 dB with the same procedure and constants after 18.70 GPU-hours of calibration, 3.5 times that of the 480p plan (Table 33 ).
Table 33. Recalibration cost for a new resolution (Wan2.1-14B). 480p: the deployed plan of Table 1 , with its cost from Table 30 .
Cluster count and k-means rounds. Table 34 varies the clusters per head over 220, 440, 880, and 1,760 and the k-means rounds of the first sparse step. More clusters per head raise PSNR steadily, from 23.88 to 25.18 dB, because smaller clusters have members closer to their centroid and leave smaller residuals (Proposition A.6), while k-means time grows from 404 to 927 ms and DiT time from 7.78 to 8.78 s; 880 clusters keep all but 0.48 dB of the 1,760-cluster gain at 0.59 s less DiT time. The k-means rounds matter little, 3 rounds losing 0.06 dB and 20 rounds gaining 0.01 dB, since every later call recomputes the cluster statistics.
Table 34. Clusters per head and k-means rounds (Cosmos-Predict2.5). k-means ms: the first clustering per video.
Task Quality
VBench dimensions. Table 28 reports each VBench dimension [7 ] together with dynamic degree, the amount of motion, and overall consistency, the text–video alignment score. DecoupAttn tracks dense attention in every dimension on the four-step models (Table 28 ), and on Cosmos3-Nano it keeps every dimension within 0.003 of dense attention, while SVG-EAR drops subject consistency from 0.932 to 0.844 and imaging quality from 0.718 to 0.661. Dynamic degree stays within one video of the dense value on every model but Cosmos-Predict1, where it drops by three of 39 videos (SVG-EAR: two), so the sparse videos keep nearly all the motion of the dense outputs.
Conditions and ground truth. Table 35 measures edge adherence on Cosmos-Transfer2.5, the F1 score between the input edges and the edges of the output, and the error against ground truth where the rendering test sets provide it. DecoupAttn keeps edge adherence at the dense level (0.423 against 0.421) and matches dense attention against ground truth on both renderers (27.48 against 27.49 dB and 12.08 against 12.17 dB), above SVG-EAR on all three (Table 35 ), so its fidelity to the dense output carries over to the input conditions and to the ground truth.
Case Studies
Four clips show where the method is stressed: two are preselected by a measurable property before generation (fast motion with fine texture, and occlusion), and two are the clips on which DecoupAttn scores lowest. Each figure shows the same frames from dense attention, SVG-EAR, and DecoupAttn, generated with the main-table configuration at density 0.20 and seed 1000.
Case 1: Fast motion (UniRelight, syn_test_0004). It has the largest mean absolute frame difference of the dense output among the 39 test clips, and the largest mean Laplacian variance, so it combines fast motion with fine texture. Figure 13 shows frames 0, 28, and 56 of this clip.
Figure 13. Case 1: fast motion (UniRelight, syn_test_0004, frames 0, 28, and 56). Rows: dense attention, SVG-EAR, and DecoupAttn, with per-frame PSNR and SSIM against dense attention.
Key Observations: Both sparse outputs are indistinguishable from the dense frames: DecoupAttn scores 38.4–38.9 dB per frame and SVG-EAR 39.9–40.2 dB (38.76 and 39.94 dB over the clip). The remaining error of both methods lies on the edges of the cobblestones and the furniture, the fine texture that the motion carries across blocks. SSIM stays at 0.990–0.992 on every frame for DecoupAttn and 0.993–0.995 for SVG-EAR, so both keep the structure of the dense frames. The error of DecoupAttn has no block shape although its exact blocks are fixed: the compensated residuals of the skipped blocks cancel like dithered quantization errors (Proposition 1, Figure 4 a–Figure 4 c), so what remains spreads as fine noise over the texture instead of gathering at block borders. UniRelight is also the model on which keeping the clusters current matters most (Table 25 ), and on its fastest clip DecoupAttn, which reassigns on this model, keeps every frame above 38 dB.
Case 2: Occlusion (UniRelight, syn_test_0006). It has the largest share of pixels whose G-buffer depth changes by more than 10% of its range between frames, so objects keep covering and uncovering each other. Figure 14 shows frames 0, 28, and 56 of this clip.
Figure 14. Case 2: occlusion (UniRelight, syn_test_0006, frames 0, 28, and 56). Rows: dense attention, SVG-EAR, and DecoupAttn, with per-frame PSNR and SSIM against dense attention.
Key Observations: SVG-EAR stays closest to dense here (45.7–46.3 dB per frame against 37.7–38.4 dB for DecoupAttn, 46.02 and 38.18 dB over the clip), the largest gap of the four cases. The difference of DecoupAttn is a faint brightness offset spread evenly over the floor, together with thin lines along the outlines of the stool, the cans, and the box where the objects occlude one another. The objects and the floor texture match the dense output in every frame. The brightness offset is the visible form of an error in the estimated mass: the exact and the compensated parts share one softmax denominator (Eq. (5) ), so a small error in Z ^ i \hat{Z}_i Z ^ i scales a whole region evenly instead of leaving a local artifact, and SSIM stays at 0.993–0.994 on every frame. The variance correction of Eq. (4) targets exactly this mass error and cuts it about threefold on the three models of Table 24 , so the offset stays faint.
Case 3: Worst clip of a few-step model (Cosmos-Predict2.5, test_19). It is the clip on which DecoupAttn has its lowest PSNR among the 32 test clips (16.39 dB). Figure 15 shows frames 0, 46, and 92 of this clip.
Figure 15. Case 3: worst clip of a few-step model (Cosmos-Predict2.5, test_19, frames 0, 46, and 92). Rows: dense attention, SVG-EAR, and DecoupAttn, with per-frame PSNR and SSIM against dense attention.
Key Observations: SVG-EAR scores lower still on this clip (15.85 dB over the clip, 15.6–15.9 dB per frame against 16.2–16.4 dB for DecoupAttn). Both sparse outputs keep the greenhouse, the hanging baskets, and the person bend-
ing over the plants, but they redraw the dense canopy of small leaves and flowers at shifted positions, so the error covers the whole canopy evenly. The four-step sampler amplifies early differences the most (Figure 4 d), and a prompt with this much fine texture turns them into a low PSNR for every sparse method. DecoupAttn keeps a higher SSIM on every frame (0.630–0.647 against 0.586–0.602), so even where both methods redraw the leaves it keeps more of the dense layout. The even spread of its error is what the cancellation of Proposition 1 predicts for compensated residuals, and it keeps the worst clip of Cosmos-Predict2.5 free of local failures.
Case 4: Worst clip of a distilled 14B model (Wan2.1-14B, test 26). It is the clip on which DecoupAttn has its lowest PSNR among the 32 test clips (19.95 dB). Figure 16 shows frames 0, 40, and 80 of this clip.
Figure 16. Case 4: worst clip of a distilled 14B model (Wan2.1-14B, test_26, frames 0, 40, and 80). Rows: dense attention, SVG-EAR, and DecoupAttn, with per-frame PSNR and SSIM against dense attention.
Key Observations: DecoupAttn again stays ahead of SVG-EAR (19.95 against 18.24 dB over the clip). The first frame is the hardest for both (16.96 and 16.38 dB), and later frames recover (20.94 and 18.90 dB at frame 40). Both outputs keep the goblin, the car, and the coaster and shift the thin rails and struts of the background, while SVG-EAR also departs more on the goblin’s face and the front car, which DecoupAttn keeps close to the dense frames. Per frame, DecoupAttn leads by 0.58, 2.04, and 1.66 dB and keeps SSIM at 0.732–0.801 against 0.673–0.729, so its lead holds on every frame of the clip.
Summary. Across the four cases, from two video models and a renderer, the error of DecoupAttn is spread, not local: a fine texture shift under fast motion, an even brightness offset under occlusion, and redrawn leaves on the hardest prompt, never a broken object or a block-shaped artifact. DecoupAttn leads SVG-EAR on every frame of both four-step cases, and on both UniRelight cases its frames exceed 37.6 dB with SSIM ≥ 0.99 \ge0.99 ≥ 0.99 and match the dense frames. These are the behaviors that decoupling predicts: the residuals of the fixed blocks cancel and spread (§ 4.1 ), and the clusters of each video follow its own content as objects move and occlude each other (Appendix § 11 .8).
Components at work. The cases show the three components of DecoupAttn in the frames. Decoupling lets fixed exact blocks coexist with semantic clusters that cover every key, so the textures of Cases 1 and 3 change as spread noise, not at block borders (§ 4.1 ). One offline plan per model sets the exact blocks of all four clips and each clip clusters its own keys, so a single frozen plan serves fast motion, occlusion, and dense texture, as it serves held-out content (Table 32 ). Distortion Budget matters on the four-step models of Cases 3 and 4, where removing its sensitivity costs 2.30 and 2.14 dB on average (Table 2 ). On UniRelight, the model of Cases 1 and 2, reassignment gains 3.63 dB over frozen assignments on test clips, the largest gain of the ten models (Table 26 ), so the clusters follow the moving and occluding objects.
Stability along the clip. The per-frame PSNR of DecoupAttn varies by at most 0.75 dB within Cases 1–3 (38.44–38.87, 37.68–38.43, and 16.16–16.39 dB), so its error does not build up from first to last frame, and in Case 4 it is highest after the first frame (20.94 dB, frame 40). SVG-EAR is as flat (39.88–40.24, 45.71–46.32, and 15.58–15.87 dB), so the per-frame scores follow the prompt, not the position of the frame in the clip.
Hard prompts. The prompts of Cases 3 and 4 are hard for every sparse method: SVG-EAR falls on them from its test means of 21.85 and 22.95 dB to 15.85 and 18.24 dB (Table 1 ), and DecoupAttn leads on both (16.39 and 19.95 dB). Their dense canopy of small leaves and their thin rails and struts are fine, repeated structures, and on them DecoupAttn keeps the layout, the people, and the vehicles of the dense video: the gardener and the goblin keep their poses and positions in every frame (Figure 15 and Figure 16 ).
Structure and quality. DecoupAttn leads in SSIM on every frame of Cases 3 and 4, by 0.04–0.07, so it keeps more of the dense layout where both methods redraw fine detail, and its LPIPS is the lowest of all sparse methods on the three models of the cases (0.102, 0.057, and 0.006, Table 1 ). VBench stays within 0.007 of dense attention on every model (Table 7 ), and the error against ground truth matches that of dense attention on both renderers that provide it (Table 35 ), so the fidelity seen in the four cases holds across the test sets of all ten models.