WorksComputer Science
Every Query Routes Itself: Accelerating Video Diffusion Transformers with Per-Query Routed Sparse Attention
Abstract
Self-attention over long token sequences dominates the inference cost of video diffusion transformers. Clustering-based sparse attention clusters both queries and keys and lets all queries in a cluster share one selection of key clusters. We show that this sharing limits accuracy: with key clusters and density fixed, attention grows more accurate as the query clusters are refined, and is most accurate when each query selects on its own, which needs no query clustering at all. We therefore propose Per-Query Routed Sparse Attention (PQRS), a training-free method that clusters only keys, lets each query select its own key clusters, and compensates for the unselected ones with their centroids. Per-query selections leave no query tile with a shared key list, so PQRS traverses key clusters instead of queries, buckets queries with identical selections into dense tasks, and merges their partial results in one pass under a fixed softmax offset carried from the previous denoising step. On four text-to-video and image-to-video models from the Wan and HunyuanVideo families, PQRS is the fastest of all the sparse attention methods we evaluate and the closest to the dense output, reaching dB PSNR on Wan2.1-T2V and dB on HunyuanVideo at – end-to-end speedup.
Introduction
Video diffusion transformers denoise token sequences whose length sets their cost: Wan2.1 [28] at 720p with 81 frames denoises tokens, and self-attention over them is quadratic in and accounts for most of the 28 minutes a five-second clip takes on an H100 [25, 35]. Sparse attention reduces this cost by computing only the token pairs that carry most of the attention mass. Most methods select over blocks of contiguous tokens, and contiguity in a video latent does not imply similarity: tokens that respond to the same keys are often far apart in raster order. Clustering-based methods [21, 25, 35, 45] remove this mismatch by grouping tokens by content, and among training-free methods they currently give the best fidelity at a given cost.
Keys and queries are clustered for different reasons. Keys are clustered so that a query can select at cluster granularity. Queries are clustered so that the selection can be executed: when a whole query cluster shares one selection, each selected pair of clusters becomes a dense block for a block-sparse kernel. But the shared selection also fixes what each query may see, and a query that attends differently from its cluster mean is handed the mean’s keys. Holding the key clusters and the density fixed, we find that recall rises and compensated output error falls as the query clusters are refined, with one query per cluster the most accurate. Even when each query cluster is scored by the attention mass it actually receives, so that the score carries no estimation error, a shared selection still trails per-query routing. The sharing itself is what costs accuracy.
Per-query routing needs no query clusters at all, so it removes a step rather than adding one. Query clustering is not free either: a -means pass compares every token against every centroid, the same order of cost as scoring every query against the key centroids once. The finest granularity is therefore not the more expensive option. What stands in the way is the shape of the work rather than its volume: tensor cores multiply a tile of queries against a tile of keys, and such a tile pays off only when its rows attend to the same keys, which is why two-sided clustering groups queries before they select [35].
We therefore propose Per-Query Routed Sparse Attention (PQRS), which clusters only the keys and lets every query select on its own. Each query scores the key centroids, ranks the clusters by estimated attention mass, and keeps the top clusters whose cumulative mass reaches ; the selected clusters are computed densely, and the rest are compensated through their centroid summaries using the scores the routing has already produced. Since no tile of consecutive queries now shares a key list, our kernel traverses key clusters instead: within a group of similar clusters it buckets the queries by which members they chose, so that queries with the same selection form a dense task, and execution is shared only where the hardware requires it. Partial results accumulate in place under a fixed offset taken from the log-sum-exp of the previous denoising step, which removes the rescaling of online softmax and lets the kernel finish in a single pass.
We evaluate PQRS on four 720p video DiTs covering text-to-video and image-to-video generation from the Wan and HunyuanVideo families, with one configuration shared by all models and no training or per-model calibration. Against five training-free sparse attention baselines, PQRS attains the best PSNR, SSIM and LPIPS on every model while also being the fastest, with end-to-end speedups of to on a single H100. On VBench, which judges a video without reference to the dense output, we detect no degradation relative to dense attention. Figure 1 compares frames from dense attention and PQRS. Our contributions are the following.
Figure 1. PQRS accelerates 720p video generation while preserving visual quality. Each block shows frames from dense attention (top) and PQRS (bottom) generated with the same prompt and seed. On a single H100, PQRS reaches 1.75× and 2.10× end-to-end speedup on Wan2.1 and HunyuanVideo, with PSNRs of 27.61 and 33.58 dB on the shown samples. Prompts are truncated.
We locate the accuracy cost of clustering-based sparse attention in the shared selection rather than in the clustering.
We propose PQRS, a training-free method in which every query routes over key clusters on its own and compensates the rest through their centroids, together with a cluster-major kernel that executes these irregular selections at tile efficiency in a single pass.
On four evaluated 720p video DiTs, PQRS attains the best PSNR, SSIM and LPIPS against five training-free baselines while also being the fastest, with end-to-end speedups of to and no degradation detected on VBench.
Related Work
Sparse attention for video DiTs. Static methods exploit spatiotemporal locality in video DiTs [10, 28] with head-wise spatial or temporal masks [31], sliding tiles [4] or energy-decay masks [16], while dynamic methods estimate block importance online [33, 39] or search the mask at a few steps and reuse it [32]. Following content-based clustering in language models [22, 27], clustering-based methods permute similar tokens into contiguous blocks: SVG2 [35] clusters queries and keys separately and selects cluster pairs by centroid scores, and SVOO [21], AdaCluster [25], Hyper-VAattention [12] and SCOPE [43] refine its clustering. SVG-EAR [45], PISA [14] and Sol-Attn [13] approximate rather than drop unselected blocks, and trainable methods learn the routing [30, 40]. To use block-sparse kernels, most training-free methods share one key selection across the queries of a block or cluster; TRA [1] assigns per-query budgets but still executes a block-sparse mask, and SparsePR [24] regroups queries by their responses to key clusters. PQRS instead removes query grouping: each query routes over shared key clusters on its own.
Query-level sparse attention and its kernels. Per-query selection is common in LLMs, both in single-query decoding over KV pages or clusters [18, 26] and in trainable architectures that route each token by key blocks or tokens [3, 11, 20, 37]; Squeezed Attention [6] also scores key-cluster centroids per query, but averages the scores over a query block in prefill. To execute per-query selections, MoBA [20] reorders queries by key block and merges partial outputs with online softmax, while FSA [34] and MSA [11] loop over key blocks outermost and merge buffered partial outputs in a separate kernel. PQRS follows this key-outer order but accumulates partial outputs in place with atomic additions, replacing online-softmax rescaling with a fixed per-row exponent offset. The offset is the log-sum-exp of the previous denoising step, which AdaSpa [32] observed to be stable across steps, analogous to the forward log-sum-exp reused in the FlashAttention-2 backward pass [2] and the shared maximum of FlashDecoding++ [5]. Within a group of similar key clusters, queries with identical selections share one tile rather than the union of their selections, much as Hydragen [9] batches queries over a shared prefix.
Other acceleration of diffusion models. Orthogonal to attention sparsity, fast solvers [19, 23] and distillation [36] reduce the number of sampling steps, feature caching reuses computation across steps [17, 44], and quantization lowers numerical precision [15, 38].
Motivation
Clustering-based sparse attention makes all queries in a cluster share one selection of key clusters, whereas per-query routing lets every query select its own, as Figure 2 illustrates. After § 3.1 formalizes the shared selection, § 3.2 shows that accuracy improves as the query clusters are refined down to single queries, which yields per-query routing. § 3.3 shows that per-query routing also avoids clustering the queries, but that its irregular selections do not map onto the tiles of GPU attention kernels.
Figure 2. Shared selection versus per-query routing. Darker cells carry more attention mass, outlined cells are selected, and bars show the recall . (a) A shared selection fits some queries in a cluster poorly. (b) Per-query routing keeps a similar recall for every query with one fewer selected pair.
Preliminaries: Clustered Sparse Attention
Sparse attention. Consider one attention head over tokens, whose queries, keys, and values have rows , , and . Let be the attention logit and the dense attention probability, so the dense output is . Sparse attention restricts the softmax to the pairs selected by a mask . Its cost scales with the density , and its accuracy depends on the attention recall .
Two-sided clustering. SVG2 [35] builds by running k-means on both sides, which partitions the key indices into clusters with means and the query indices into clusters with means , where query belongs to . Replacing every member with its cluster mean gives the cluster-level logit , from which SVG2 estimates the attention mass that places on as
It then selects the top- set , the fewest key clusters whose scores sum to at least , and lets every query in attend to all keys of these clusters, so queries in one cluster share one mask row:
This fixes both which key clusters and how many keys each query attends to, as Figure 2 shows. In exchange, the mask maps onto a block-sparse kernel: once the tokens are permuted by cluster, every selected pair of query and key clusters is a dense block [35]. Other clustering-based methods for video DiTs [21, 25, 45] also share the selection.
Centroid compensation. For each unselected key cluster , SVG-EAR [45] replaces the logits of the keys in with the per-query logit and their values with the value mean , which gives
The compensation thus uses each query’s own logits , whereas the selection uses the cluster-level logits .
Query Granularity and Accuracy
To measure how much accuracy the shared selection in Equation (2) costs, we refine the query clusters at a fixed density. At , every query forms its own cluster with , so no selection is shared and we obtain per-query routing, as illustrated in Figure 2(b).
We record , , and along the dense sampling trajectories of Wan2.1-T2V-14B [28] at 720p for eight of the augmented VBench prompts used by SVG2 [7, 35]. All configurations share the key clusters () and run their own query k-means, with calibrated per head and configuration to a density of . To separate the error of the centroid estimate from the cost of sharing, we also select key clusters with Oracle scores, which replace with the true attention mass averaged over the queries of the cluster:
We measure the recall and the relative output error of the compensated output in Equation (3).
Finer query clusters improve accuracy. Figure 3a, Figure 3b shows that refining the query clusters raises recall and lowers output error under both scoring rules, with per-query routing the most accurate. In a smaller cluster, fewer queries share one selection, which then fits each member better.
Figure 3. Accuracy versus query granularity. All configurations share the key clusters () and the target density () and differ only in the number of query clusters , where is per-query routing. (a) Mean recall under centroid and Oracle scoring. (b) Mean relative error of the compensated output. (c) Recall percentiles over the queries of a head under centroid scoring, averaged over heads.
Oracle scores do not remove the cost of sharing. The gap between Oracle and centroid scoring narrows as grows, since a smaller cluster is better represented by its mean. Yet at every shared granularity we test, even Oracle scoring is less accurate than per-query routing with centroid scores, so the remaining loss comes from the sharing itself rather than from the error of the centroid estimate.
The gain concentrates in the low-recall tail. Figure 3c shows that refining the query clusters lifts the low recall percentiles the most. P1 rises from 0.560 at to 0.865 under per-query routing, whereas above the median the curves nearly coincide, so per-query routing yields an almost flat profile. This follows from how selections are sized: per-query routing adds key clusters until each query’s estimated mass reaches , whereas a shared selection is sized by the cluster mean and under-covers the queries far from it, as the recall bars of Figure 2 show.
Cost of Query Granularity
Section 3.2 shows that finer query granularity improves accuracy. We now compare the routing cost of reaching it by two-sided clustering and by per-query routing. For one head in one attention call, we count the matrix multiplications relative to the FLOPs that sparse attention spends on the logits and weighted values of the selected pairs, and leave out the key clustering that both share.
Refining query clusters. Two-sided clustering adds the query k-means and the cluster-level scores of Equation (1). With centroids warm-started from the previous denoising step [35], the k-means runs only iterations per call, each comparing queries with centroids. Relative to sparse attention, this costs
where we drop the scoring term, which is only of the k-means term.
Routing each query directly. Per-query routing needs no query clustering and only scores each query against each key mean, which costs
For Wan2.1-T2V-14B at 720p () with and , this cost is 1.8% of the sparse attention.
Dividing Equation (5) by Equation (6) gives
which depends neither on the video length nor on the density. Per-query routing is therefore cheaper whenever . With the iterations that SVG2 uses and , this holds for , so every shared configuration in Figure 3 except costs more than per-query routing and is also less accurate.
Algorithm 1 PQRS attention for one head.
Require: $Q, K, V \in \mathbb{R}^{N \times D}$; $C_k$; $p$; $f$
Ensure: $\hat{o}_1, \ldots, \hat{o}_N$
1: $K_1, \ldots, K_{C_k} \gets$ k-means on the keys
2: Compute $|K_c|$, $\bar{k}_c$, and $\bar{v}_c$ for every cluster $c$
3: $Z \gets Q\bar{K}^{\mathsf{T}}/\sqrt{D}$ ▷ $z_{ic}$ for all $i,c$
4: for all queries $i$ in parallel do
5: $\hat{P}_{ic} \gets |K_c|e^{z_{ic}}/\sum_{c'}|K_{c'}|e^{z_{ic'}}$ for every $c$
6: $S_i \gets$ top clusters by $\hat{P}_{ic}$, added until $\sum_{c\in S_i}\hat{P}_{ic}\ge p$ and $\sum_{c\in S_i}|K_c|\ge fN$
7: $(u_i,l_i)\gets\sum_{c\notin S_i}|K_c|e^{z_{ic}}(\bar{v}_c,1)$ ▷ means
8: $(u_i,l_i)+=\sum_{c\in S_i}\sum_{j\in K_c}e^{s_{ij}}(v_j,1)$ ▷ exact
9: $\hat{o}_i\gets u_i/l_i$
10: end for
Beyond matrix multiplications. FLOPs alone do not determine the speed on GPUs, where tensor cores multiply a tile of typically 64 queries with a tile of keys, so a tile is efficient only when all of its queries attend to the same keys. Two-sided clustering meets this requirement by grouping the queries before they select keys, but refining these groups is costly: at fixed , raising from 50 to 400 lifts the PSNR of SVG2 [35] by 3.9 dB but cuts its speedup from 1.90× to 1.25×. Under per-query routing the queries of a tile generally attend to different keys, so PQRS reverses this order: each query selects its own key clusters, and queries with the same selection then share a tile.
Per-Query Routed Sparse Attention
Section 4.1 defines the output that PQRS computes for each query. Section 4.2 computes these outputs by traversing the key clusters, packing queries with identical selections into dense tasks, and merging the partial results of each query in a single pass, and Section 4.3 shows that this merging is numerically safe.
Per-Query Routing and Compensation
Algorithm 1 summarizes one sparse attention call of PQRS for one head in three steps.
Key clustering. PQRS clusters only the keys. At every sparse attention call, it runs k-means on the keys of each head and stacks the key means as the rows of .
Per-query routing. Each query scores the key clusters with Equation (1), using its own logit in place of , which gives . It selects key clusters in descending order of until their scores sum to at least and, following the official implementation of SVG2 [35], until they cover at least keys. Query attends to all keys of the selected set , which, unlike , depends on query alone.
Compensation. PQRS keeps the centroid compensation of (3), with in place of . Since routing has already computed for every key cluster, the compensation adds only one weighted sum of the value means. In Algorithm 1, and accumulate the numerator and denominator of (3), and appends a one to a value vector so that both are accumulated together.
Cluster-Major Execution
Since the selections differ across queries, PQRS computes Line 8 of Algorithm 1 in cluster-major order, following KV-outer sparse attention kernels [11, 20, 34]: it traverses the key clusters and gathers, for each one of them, the queries that selected it.
Grouping clusters. A key cluster holds few keys, so a task over one cluster is short along the keys, yet each of its queries must still be gathered and have its partial output atomically merged. PQRS therefore executes groups of clusters, so that a query pays this cost once per group it hits rather than once per selected cluster. PQRS groups them greedily by the cosine similarity of their key means. The grouping runs before routing, and and are permuted so that each group is contiguous in memory.
Routing each query. Since consists of the clusters whose scores reach a row-specific threshold, the router finds this threshold by bisection instead of sorting. It records the selection of query in group as a -bit code in member order and writes the compensation into the fp32 accumulators as their initial value.
Bucketing by code. A query often selects only some members of a group it hits. Running the whole group for this query would count the unselected members twice, once exactly and once in the compensation, and masking them would waste tensor-core work. PQRS instead buckets the queries of each group by code, so all queries in a bucket select the same members and run as dense tasks of up to 128 queries over the contiguous key ranges of these members. In Figure 4, , , and select members and of group and share the code 1100, so runs as three dense tasks without a union or a mask. A group can split its queries into up to buckets, which leaves some tasks partially filled, and Section 5.3 evaluates this trade-off over the group width.
Figure 4. Execution pipeline of PQRS for one attention head. (a) Key clusters with similar centroids are grouped and stored contiguously. (b) Each query records its selection in each group as a 4-bit code and initializes its accumulators with the compensation. (c) The queries are bucketed by (group, code). (d) Each bucket runs as dense tasks that atomically add their partial results under a fixed offset . (e) The rows are normalized, and rare rows outside the safe range are recomputed. Bottom: each row’s LSE is its offset at the next step.
Cluster-major attention. The partial results of a query come from tasks that run on different SMs in no fixed order. Online softmax [2] rescales the accumulated output whenever the running maximum grows and therefore combines them in sequence, and existing KV-outer kernels either recompute for the softmax statistics [34] or merge the stored partial results in another kernel [11]. PQRS instead fixes a per-row offset before the kernel starts and subtracts it from every logit of row , including , which scales the numerator and denominator of Equation (3) by the same factor . The partial results then simply add: each task computes over its query–key pairs and adds to the accumulators of its queries.
Offsets from the previous step. Let be the natural logarithm of the denominator of Equation (3) at step , and let . The kernel accumulates , and every term and partial sum of lies in , so the numerical range of a row depends only on . The ideal offset is therefore the current LSE, which is unknown before the kernel runs. Since the LSE is stable across denoising steps [32], PQRS sets from the same layer and classifier-free guidance [4] branch, and the first sparse step uses the LSE that flash attention returns at the last dense step. A cached maximum logit would instead add to the gap between the LSE and the maximum up to . FlashDecoding++ [5] also fixes the offset in advance and recomputes on overflow, but uses one offset for all rows.
Finalization. A final pass computes and and caches the LSE for the next step. Rows with are recorded in device memory and recomputed by single-row online softmax over the same selection and compensation, without host synchronization. Implementation details are given in the supplementary material.
Numerical Safety of Cross-Step Offsets
In exact arithmetic the offset cancels in and only sets the range of the fp32 accumulators. Within the range accepted by the final pass, , the accumulators stay far from overflow, and the mass lost to underflow stays below the bf16 rounding error of the output. Other rows are recomputed, so the output is accurate for any offset, and a good offset only means fewer repaired rows.
Why the previous LSE suffices. The tolerance needs only a coarse prediction. For a fixed selection, the LSE of a row changes, to first order, by the attention-weighted average of the changes in its logits, so it drifts far only when these logits move together, and refreshing the cache at every step keeps the error from accumulating.
Stability in practice. On Wan2.1-T2V-14B, Figure 5 shows that fewer than 0.01% of the rows cross the repair threshold, almost all of them in the last two steps. It indicates that the LSE is stable enough to reuse across steps.
Figure 5. The LSE changes little between consecutive steps. Percentiles of over the sparse rows at each denoising step on Wan2.1-T2V-14B at 720p, pooled over two prompts. Rows above the dashed repair threshold, , are recomputed.
Experiments
Setup
Models. We evaluate PQRS on four widely used video generation models: Wan2.1-T2V-14B, Wan2.2-T2V-A14B [28] and HunyuanVideo-T2V-13B [10] for text-to-video (T2V) generation, and Wan2.1-I2V-14B [28] for image-to-video (I2V) generation.
Metrics. For quality, we measure the similarity between the videos generated with sparse and dense attention by PSNR, SSIM [29] and LPIPS [42], and report the VBench score [7] averaged over subject consistency, background consistency, motion smoothness, aesthetic quality and imaging quality. For efficiency, we report the attention density, the fraction of query–key pairs computed exactly in the sparse attention calls, together with the end-to-end latency and the speedup over dense attention.
Datasets. Following prior work [21, 35, 45], we randomly sample 50 of the augmented VBench prompts [7] for T2V and 50 image–prompt pairs from VBench++ [8] for I2V.
Baselines. We compare PQRS with state-of-the-art training-free sparse attention methods for video generation: SVG [31], SVG2 [35], SVG-EAR [45], SVOO [21] and PISA [14]. All baselines use their official configurations, except for the shared warm-up below and the density hyperparameter. Table 1 lists the resulting densities.
| Model | Method | PSNR↑ | SSIM↑ | LPIPS↓ | VBench↑ | Density | Latency↓ | Speedup↑ |
| Wan2.1-T2V-14B (720P, 81 Frames) | Dense | – | – | – | 83.64% | 100% | 1682s | 1.00× |
| SVG | 22.17 | 0.769 | 0.210 | 83.77% | 40.7% | 1053s | 1.60× | |
| SVG2 | 25.73 | 0.858 | 0.138 | 83.54% | 31.4% | 1091s | 1.54× | |
| SVOO | 26.02 | 0.866 | 0.130 | 83.58% | 35.8% | 1137s | 1.48× | |
| PISA | 24.05 | 0.824 | 0.164 | 83.78% | 35.5% | 1107s | 1.52× | |
| SVG-EAR | 27.19 | 0.884 | 0.116 | 83.58% | 21.7% | 1032s | 1.63× | |
| PQRS | 28.86 | 0.909 | 0.094 | 83.70% | 22.0% | 960s | 1.75× | |
| Wan2.1-I2V-14B (720P, 81 Frames) | Dense | – | – | – | 83.74% | 100% | 1362s | 1.00× |
| SVG | 23.98 | 0.790 | 0.183 | 83.50% | 40.7% | 858s | 1.59× | |
| SVG2 | 26.59 | 0.857 | 0.140 | 83.47% | 30.4% | 878s | 1.55× | |
| SVOO | 27.27 | 0.873 | 0.128 | 83.57% | 33.2% | 919s | 1.48× | |
| PISA | 26.85 | 0.869 | 0.130 | 83.69% | 35.5% | 866s | 1.57× | |
| SVG-EAR | 28.06 | 0.889 | 0.117 | 83.59% | 21.2% | 838s | 1.62× | |
| PQRS | 29.97 | 0.917 | 0.097 | 83.74% | 21.6% | 781s | 1.74× | |
| Wan2.2-T2V-A14B (720P, 81 Frames) | Dense | – | – | – | 84.26% | 100% | 1424s | 1.00× |
| SVG | 17.66 | 0.594 | 0.328 | 84.12% | 40.7% | 926s | 1.54× | |
| SVG2 | 21.15 | 0.744 | 0.222 | 84.24% | 33.6% | 986s | 1.44× | |
| SVOO | 20.87 | 0.735 | 0.227 | 84.21% | 39.3% | 1037s | 1.37× | |
| PISA | 19.13 | 0.665 | 0.276 | 84.32% | 35.5% | 931s | 1.53× | |
| SVG-EAR | 22.25 | 0.779 | 0.194 | 84.19% | 22.6% | 923s | 1.54× | |
| PQRS | 23.47 | 0.814 | 0.170 | 84.26% | 22.5% | 856s | 1.66× | |
| HunyuanVideo-T2V-13B (720P, 129 Frames) | Dense | – | – | – | 84.45% | 100% | 1701s | 1.00× |
| SVG | 27.34 | 0.892 | 0.125 | 84.37% | 36.6% | 961s | 1.77× | |
| SVG2 | 31.09 | 0.933 | 0.084 | 84.30% | 26.8% | 961s | 1.77× | |
| SVOO | 29.95 | 0.922 | 0.093 | 84.25% | 25.0% | 881s | 1.93× | |
| PISA | 28.27 | 0.909 | 0.109 | 84.46% | 35.5% | 1085s | 1.57× | |
| SVG-EAR | 32.95 | 0.948 | 0.068 | 84.38% | 19.8% | 929s | 1.83× | |
| PQRS | 35.44 | 0.961 | 0.053 | 84.48% | 19.6% | 809s | 2.10× |
Table 1. Quality and efficiency on four models. Among sparse methods, the best result is in bold and the second best is underlined.
Implementation. PQRS uses key clusters, a routing threshold of and a routing floor of on all four models. Following [14, 45], every method computes dense attention in the first 20% of denoising steps and in the first block of each transformer stack. All methods, including dense attention with PyTorch SDPA, share the same non-attention operators, so latency differences come from attention alone. Videos are generated at 720p with the official sampling configuration of each model on a single NVIDIA H100 GPU, and further details are given in the supplementary material.
Main Results
Table 1 compares PQRS with five sparse attention baselines on four video diffusion models, covering both text-to-video and image-to-video generation. PQRS consistently achieves the best PSNR, SSIM, and LPIPS on all four models, surpassing SVG-EAR, which runs at a comparable density, as well as SVG, SVG2, SVOO, and PISA, which compute more attention. Since each query selects key clusters according to its own attention distribution, PQRS retains the attention mass that a shared selection misses, and the generated videos stay closest to the dense outputs. On the reference-free VBench metric, PQRS remains on par with dense attention across all four models, and the qualitative comparisons in Figure 1 further confirm that it preserves the visual quality of the dense outputs.
PQRS also delivers the highest inference efficiency on every model, reaching 1.66–2.10× end-to-end speedup. Although per-query routing produces an irregular selection pattern, the cluster-major kernel in § 4.2 executes it efficiently, so the finer selection does not come at the cost of speed. As a result, PQRS improves fidelity and inference speed at the same time on all evaluated models, achieving a better quality–efficiency trade-off than existing sparse attention methods.
Ablation Study
Group width. We sweep the group width from 1 to 6, keep every other setting at its main-table value, and time all widths in one process on the same , , and tensors in alternating order. Neither the routing rule nor the estimator depends on , so all widths select the same clusters and their outputs differ only in floating-point summation order.
As shown in Figure 6, is the fastest width, and removing grouping () makes a call 1.4× slower, almost entirely in attend, as the stages outside it vary by less than 4 ms across the sweep. A group is built from similar clusters, which queries tend to select together: at , a query that touches a group selects 2.3 of its four clusters, and one task serves them with a single Q gather and a single merge, rather than one of each per cluster. A wider group saves more of both, but only queries that select the same subset of the group can share a task, and clusters have non-empty subsets. At an average task fills only 77% of its 128 query rows and computes the rest anyway, so beyond the emptier tasks outweigh the saving.
Figure 6. Latency versus group width . Each bar is one sparse attention call over all 40 heads of Wan2.1-T2V-14B, averaged over 3 prompts, 3 denoising steps and 39 layers. Other stages covers cluster, route, build and finalize.
Selection versus compensation. SVG-EAR [45] applies the same centroid compensation as PQRS, but to a selection shared within each query cluster. To separate the two components, we cross shared or per-query selection with compensation off or on and keep every other setting at its main-table value. The shared cells cluster the queries into
clusters, select key clusters for each query cluster with the SVG2 rule [35], and recalibrate to the density of per-query selection.
As shown in Table 2, each component improves every metric on its own, and per-query selection beats shared selection by 0.9 dB of PSNR even without compensation. The two gains also reinforce each other: compensation adds 1.7 dB on top of shared selection but 3.1 dB on top of per-query selection, and SSIM and LPIPS follow the same pattern. A likely reason is that the centroid estimate works best for clusters that carry little of a query’s attention. Per-query selection leaves the estimate only each query’s light tail, whereas a shared selection under-covers the queries far from the cluster mean, as shown in § 3.2, and leaves some of their heavily attended clusters to the estimate.
| Selection | Comp. | Density | |||
| Shared | 22.5% | 24.91 | 0.842 | 0.151 | |
| Shared | 21.9% | 26.61 | 0.874 | 0.123 | |
| Per-query | 22.4% | 25.81 | 0.861 | 0.134 | |
| Per-query | 22.0% | 28.86 | 0.909 | 0.094 |
Table 2. Selection granularity centroid compensation. Results on Wan2.1-T2V-14B, where shared selection uses query clusters and the last row is PQRS.
Conclusion
We propose PQRS, a training-free sparse attention method for video DiTs. Since sharing one selection across each query cluster limits accuracy, PQRS clusters only the keys, lets every query select its own key clusters, and approximates the unselected ones by their centroids. A cluster-major kernel packs queries with identical selections into dense tasks and merges their partial results in a single pass. On four video generation models, PQRS is faster and more faithful to dense attention than all five baselines, reaching 1.66–2.10 end-to-end speedup.
Implementation Details of PQRS
Kernels. The attend and compensation kernels are written with CuTe and CUTLASS for H100, selection and task construction in CUDA, and the remaining stages in Triton. All inputs are BF16.
Clustering and grouping. Each sparse call clusters the keys of each head, after QK normalization and RoPE, into clusters with the k-means kernel of SVG2 [35]. The first sparse call runs 50 iterations. Each later call runs 2 iterations from the centroids of the previous call of the same layer, in either guidance branch. Grouping is greedy: the largest unassigned cluster opens a group, which then repeatedly takes the unassigned cluster with the largest summed cosine similarity to its centroids until it holds clusters. Keys and values are permuted so that each group is contiguous in memory, and a task reads the selected members of a group as at most two contiguous segments.
Routing and execution. One matrix multiplication scores all queries against all key centroids. Selection assigns one warp to each query, finds the threshold of § 4.1 by bisection over the scores held in registers, and writes the 4-bit code of every group with a warp ballot. Task construction then buckets the queries of each head and band of query rows by group and code. A separate kernel first writes the compensation into the FP32 accumulators. The persistent attend kernel then loads keys and values with TMA, computes with wgmma and adds its partial results to the accumulators with vectorized atomic reductions.
Offsets and row repair. At the last dense step, PQRS calls the FlashAttention kernel behind SDPA directly. It returns the same output together with the LSE, which gives the offsets of the first sparse step. Rows outside the safe range are appended to a list in device memory. The repair kernel reads the length of this list on the device and returns at once when the list is empty, so the host never synchronizes. As Table 3 shows, repair is rare. On the two prompts of Figure 5, almost all repaired rows underflow, and they concentrate in the last two layers and the last two denoising steps.
| Model | Rows | Calls |
| Wan2.1-T2V-14B | 8.7% | |
| Wan2.1-I2V-14B | 9.1% | |
| Wan2.2-T2V-A14B | 9.3% | |
| HunyuanVideo-T2V-13B | 11 rows in total | 3 calls in total |
Table 3. Row repair on the 50 evaluation videos. Rows is the fraction of (head, row) pairs in sparse calls that are repaired, and Calls is the fraction of sparse calls with at least one repaired row.
Wan2.2. The two experts have separate layers, and only the expert active at the last dense step receives the dense LSE. In the other expert, the first sparse call of each layer therefore computes its offsets exactly with a pre-pass over the selected keys and runs k-means for 50 iterations. All later calls use the cached LSE.
HunyuanVideo. HunyuanVideo attends jointly over the video tokens and 256 text tokens, of which the trailing ones are padding. PQRS sparsifies only the attention of video queries to video keys, and the density counts only this part. Each video query also attends to all valid text keys with one FlashAttention call, and the two parts are merged exactly through their LSEs. Text queries attend densely to all valid keys and padding rows only to padding, as in the official SVG2 code. The offsets of the first sparse step are the dense LSEs minus the contribution of the valid text keys.
Experimental Setup
Sampling. The sampling configurations in Table 5 follow the official SVG inference scripts for Wan2.1 and HunyuanVideo and the official configuration of Wan2.2, whose two guidance scales belong to its two experts. HunyuanVideo uses embedded guidance and thus one transformer pass per step. On Wan2.2, every method offloads the two experts to the CPU and keeps the text encoder and the VAE on the GPU.
Table 5. Sampling configurations. Dense steps and dense blocks use dense attention for every method. counts video tokens. Euler denotes the flow-matching Euler sampler.
| Model | Frames (fps) | Steps | Sampler | Guidance | Dense steps | Dense blocks | |
| Wan2.1-T2V-14B | 81 (16) | 75,600 | 50 | UniPC, shift 5.0 | CFG 5.0 | 1–10 | first of 40 |
| Wan2.1-I2V-14B | 81 (16) | 75,600 | 40 | UniPC, shift 5.0 | CFG 5.0 | 1–8 | first of 40 |
| Wan2.2-T2V-A14B | 81 (16) | 75,600 | 40 | UniPC, shift 12.0 | CFG 4.0 / 3.0 | 1–8 | first per expert |
| HunyuanVideo-T2V-13B | 129 (24) | 118,800 | 50 | Euler, shift 7.0 | embedded 6.0 | 1–10 | first of each stream type |
Table 5.
Prompts and metrics. The augmented VBench prompts are the versions that the VBench team rewrote with prompt optimization. The I2V images are cropped to 16:9. PSNR, SSIM and LPIPS compare each frame with the same frame of the dense video generated from the same prompt and seed, and are averaged over frames and then over videos. SSIM uses an uniform window, and LPIPS uses the VGG backbone. VBench 0.1.5 runs in its custom-input mode with the default preprocessing.
Baseline Configurations
Implementations. Every baseline uses its official attention kernels. SVG [31], SVG2 [35], SVG-EAR [45] and SV00 [21] run in an open-source framework that calls these kernels, and we checked its reimplementation of the other steps against the official code. PISA [14] is called directly, as described below.
Hyperparameters. In Table 6, three density hyperparameters differ from the official values. SVG-EAR uses a lower top-p to match the density of PQRS, and SVG and PISA are set to an end-to-end speedup of about on Wan2.1-T2V-14B, comparable to the other baselines.
Table 6. Baseline hyperparameters. Each baseline uses the same density hyperparameter on all four models, except where noted, and its official values for all other hyperparameters.
| Method | Density hyperparameter | Official value | Other hyperparameters (official) |
| SVG | sparsity 0.37 | 0.30 (HunyuanVideo 0.25) | 64 sampled query rows, FlexAttention |
| SVG2 | top- 0.90 | 0.90 | (HunyuanVideo 400/1000), k-means 50 / 2 iterations, floor 0.10 |
| SVG-EAR | top- 0.80 | 0.85 | as SVG2, floor 0.05, |
| SVOO | top- 0.90 (HunyuanVideo 0.88) | 0.90 (HunyuanVideo 0.88) | , k-means 2 / 2 iterations, per-model schedule and profile |
| PISA | density 0.355 | 0.125 | block size 64, no bias term, sink: none (Wan), last block (HunyuanVideo) |
| PQRS | 0.848 | – | , floor , k-means 50 / 2 iterations, |
Table 6.
Shared protocol. Wherever attention is dense, in the dense baseline and in the dense steps and blocks of every method, it runs PyTorch SDPA with the FlashAttention-2 backend, as in the official SVG code. For the other operators, all methods including the dense baseline use the fused Triton kernels of SVG and SVOO, in which we fixed a LayerNorm bug that included zero-padded columns in the variance. The speedups reported by the SVG family use the original Diffusers operators for the dense pipeline and these fused kernels for the sparse one, so they also count the gain of kernel fusion. Our speedups use the fused kernels on both sides.
Adaptations. Three baseline configurations needed adaptation.
SVG-EAR on HunyuanVideo. The official code passes the zero-filled centroids of the padding text tokens to the compensation, which adds a term absent from dense attention to the denominator of every video row. We exclude the prompt and padding clusters from the compensation and compute the prompt columns exactly.
PISA. PISA provides no inference script for Wan or HunyuanVideo, so we call its kernel directly. Because the kernel takes no mask, on HunyuanVideo we remove the padding tokens and, as its documentation recommends for MMDiT models, use the last block as the attention sink. When the valid text spans more than one block, only the last block is guaranteed to be exact, which slightly disadvantages PISA.
SVOO on Wan2.2. The official profile of SVOO for Wan2.2 was collected with a different shift and guidance scale. We reuse it because the profile only sets a per-head lower bound on the budget, which is clipped to [0.20, 0.22] and lies well below the density of SVOO in Table 1.
Additional Results
VBench Dimensions
Table 4 breaks the VBench average of Table 1 into its five dimensions. PQRS stays close to dense attention in all five dimensions on all four models.
| Method | Subj. | Back. | Motion | Aesth. | Imag. | Avg. |
| Wan2.1-T2V-14B | ||||||
| Dense | 93.86 | 95.73 | 98.62 | 62.22 | 67.76 | 83.64 |
| SVG | 93.80 | 95.77 | 98.66 | 62.63 | 68.01 | 83.77 |
| SVG2 | 93.71 | 95.69 | 98.60 | 62.11 | 67.56 | 83.54 |
| SVOO | 93.86 | 95.55 | 98.62 | 62.12 | 67.73 | 83.58 |
| PISA | 93.88 | 95.91 | 98.64 | 62.47 | 67.99 | 83.78 |
| SVG-EAR | 93.72 | 95.51 | 98.63 | 62.18 | 67.88 | 83.58 |
| PQRS | 93.81 | 95.92 | 98.63 | 62.43 | 67.70 | 83.70 |
| Wan2.1-I2V-14B | ||||||
| Dense | 94.57 | 95.46 | 98.59 | 60.60 | 69.47 | 83.74 |
| SVG | 94.08 | 94.86 | 98.56 | 60.24 | 69.73 | 83.50 |
| SVG2 | 94.44 | 95.29 | 98.51 | 59.86 | 69.23 | 83.47 |
| SVOO | 94.50 | 95.33 | 98.54 | 60.03 | 69.46 | 83.57 |
| PISA | 94.55 | 95.40 | 98.57 | 60.43 | 69.52 | 83.69 |
| SVG-EAR | 94.50 | 95.33 | 98.57 | 60.15 | 69.41 | 83.59 |
| PQRS | 94.56 | 95.49 | 98.57 | 60.63 | 69.46 | 83.74 |
| Wan2.2-T2V-A14B | ||||||
| Dense | 93.12 | 95.20 | 98.16 | 64.36 | 70.45 | 84.26 |
| SVG | 93.12 | 95.20 | 98.18 | 63.69 | 70.41 | 84.12 |
| SVG2 | 93.22 | 95.05 | 98.16 | 64.23 | 70.55 | 84.24 |
| SVOO | 93.30 | 95.21 | 98.18 | 64.01 | 70.37 | 84.21 |
| PISA | 93.25 | 95.38 | 98.21 | 64.33 | 70.41 | 84.32 |
| SVG-EAR | 93.05 | 95.15 | 98.19 | 64.41 | 70.16 | 84.19 |
| PQRS | 93.23 | 95.25 | 98.17 | 64.19 | 70.48 | 84.26 |
| HunyuanVideo-T2V-13B | ||||||
| Dense | 94.11 | 96.00 | 99.38 | 62.26 | 70.49 | 84.45 |
| SVG | 94.15 | 96.05 | 99.37 | 62.01 | 70.25 | 84.37 |
| SVG2 | 93.97 | 95.93 | 99.35 | 62.23 | 70.03 | 84.30 |
| SVOO | 94.02 | 95.92 | 99.35 | 62.10 | 69.88 | 84.25 |
| PISA | 94.15 | 96.00 | 99.37 | 62.35 | 70.40 | 84.46 |
| SVG-EAR | 94.04 | 96.09 | 99.37 | 62.12 | 70.30 | 84.38 |
| PQRS | 94.12 | 96.06 | 99.38 | 62.32 | 70.49 | 84.48 |
Table 4. VBench scores per dimension (%). Subj., Back., Motion, Aesth. and Imag. are subject consistency, background consistency, motion smoothness, aesthetic quality and imaging quality, and Avg. is their mean as reported in Table 1.
Quality at Different Budgets
Figure 7 and Table 7 add two settings for each of PQRS, SVG-EAR and SVG2 on Wan2.1-T2V-14B, one of them the official top- of SVG-EAR. In both panels of Figure 7, PQRS at lies above and to the left of every baseline point, and the paired bootstrap 95% interval of its PSNR gain over each of them excludes zero.
Table 7. Results of Figure 7. The setting is for PQRS and top- for the baselines, and the middle row of each method is its setting in Table 1. Speedups are over dense attention (1682 s).
| Method | Setting | Density | Latency | Speedup | PSNR | SSIM | LPIPS |
| SVG2 | 0.80 | 22.3% | 994s | 1.69× | 24.66 | 0.836 | 0.157 |
| SVG2 | 0.90 | 31.4% | 1091s | 1.54× | 25.73 | 0.858 | 0.138 |
| SVG2 | 0.95 | 50.8% | 1268s | 1.33× | 26.67 | 0.875 | 0.123 |
| SVG-EAR | 0.75 | 19.3% | 1017s | 1.65× | 27.00 | 0.880 | 0.120 |
| SVG-EAR | 0.80 | 21.7% | 1032s | 1.63× | 27.19 | 0.884 | 0.116 |
| SVG-EAR | 0.85 | 25.2% | 1081s | 1.56× | 27.31 | 0.886 | 0.114 |
| PQRS | 0.75 | 17.6% | 933s | 1.80× | 28.39 | 0.903 | 0.100 |
| PQRS | 0.848 | 22.0% | 960s | 1.75× | 28.86 | 0.909 | 0.094 |
| PQRS | 0.90 | 26.0% | 989s | 1.70× | 29.43 | 0.916 | 0.088 |
Table 7.
Figure 7. Fidelity versus budget on Wan2.1-T2V-14B. Each method runs at three settings, and circles mark the settings of Table 1. (a) PSNR versus density. (b) PSNR versus end-to-end latency.
Per-Call Latency Breakdown
To time all methods on the same inputs, we record , and along the dense trajectories of three prompts. Because these inputs come from dense trajectories, the methods with adaptive budgets run at slightly higher densities than in Table 1. As Table 8 shows, PQRS routes every query individually, yet its stages outside attend take about as long as those of SVG2, which routes query clusters. The last row runs PQRS with the exact-offset pass instead of the cached LSE.
| Method | Wall time | Speedup | Density | Attend | Outside attend | Largest stages outside attend |
| Dense (SDPA) | 329.1 | 1.00× | 100% | 329.1 | – | – |
| SVG | 123.3 | 2.67× [2.66, 2.68] | 40.7% | 112.4 | 10.9 (8.8%) | head profiling 5.7 |
| SVG2 | 138.6 | 2.51× [2.45, 2.57] | 32.9% | 106.0 | 32.6 (23.6%) | k-means 17.8, plan 7.2, permutation 3.3 |
| SVOO | 152.6 | 2.25× [2.20, 2.30] | 37.2% | 116.3 | 36.3 (23.8%) | plan 27.6 |
| SVOO, reclustering call | 425.3 | 0.78× | 37.2% | 116.9 | 308.5 | co-clustering 272.2 |
| SVOO, amortized | 159.4 | 2.15× | – | – | 43.2 (27.1%) | – |
| PISA | 129.8 | 2.54× [2.53, 2.55] | 35.5% | 122.2 | 7.7 (5.9%) | pooling 2.7 |
| SVG-EAR | 122.7 | 2.82× [2.76, 2.88] | 22.9% | 71.9 | 50.8 (41.4%) | k-means 16.9, compensation 13.8, routing 6.5, plan 6.4 |
| PQRS | 97.8 | 3.52× [3.45, 3.59] | 23.2% | 66.0 | 31.8 (32.5%) | selection 12.3, k-means 6.5, compensation 4.7, scoring 3.1 |
| PQRS, exact offsets | 163.3 | 2.14× [2.09, 2.19] | 23.2% | 65.9 | 97.4 | exact-offset pass 53.3, task tables 5.5 |
Table 8. Latency of one sparse attention call on Wan2.1-T2V-14B. Times are in milliseconds per call over all 40 heads, averaged over 351 calls. Speedups are means of per-call ratios with bootstrap 95% intervals. Outside attend is the wall time minus the attend kernel, with its share of the wall time in parentheses. The reclustering row times an SVOO call that recomputes its clusters, and the amortized row spreads one such call over 40 calls.
Qualitative Results
Figure 8 to Figure 11 compare dense attention and PQRS on the four models. For each model we show the prompt on which the PSNR of PQRS is closest to its median over the 50 prompts, excluding the prompts of Figure 1 and two prompts that describe copyrighted characters. Each strip shows five evenly spaced frames, and the latencies are the means of Table 1.
Figure 8. Wan2.1-T2V-14B, 720p Text-to-Video. Dense Attention | Latency = 28.0 min. PQRS (Ours) | PSNR = 28.57 dB | Latency = 16.0 min (1.75x). A breathtaking coastal beach in spring, with golden sands stretching beneath a clear blue sky, is captured in stunning HD. The scene begins with a close-up of delicate seashells and smooth pebbles scattered across the shore. As the camera racks focus, gentle waves roll in, their white foam contrasting against the sunlit sand. The focus shifts to reveal vibrant wildflowers blooming along the dunes, their colors vivid against the backdrop of the sparkling ocean. Seagulls glide gracefully overhead, their calls blending with the soothing sound of the waves. The entire scene exudes tranquility and the rejuvenating essence of spring. Wan2.1-T2V-14B. Dense attention (top) and PQRS (bottom) with the same prompt and seed.
Figure 11. HunyuanVideo-T2V-13B, 720p Text-to-Video. Dense Attention | Latency = 28.4 min. PQRS (Ours) | PSNR = 35.34 dB | Latency = 13.5 min (2.10x). A determined individual in a sleek, black athletic outfit jogs along a winding forest trail, surrounded by towering trees and dappled sunlight filtering through the leaves. Their rhythmic strides create a sense of purpose and focus, with the soft crunch of leaves underfoot adding to the serene ambiance. As they run, the camera captures close-ups of their focused expression, beads of sweat forming on their brow, and the gentle sway of their ponytail. The scene transitions to a wider shot, revealing the lush greenery and the tranquil beauty of the forest, emphasizing the harmony between the jogger and nature. HunyuanVideo-T2V-13B. Dense attention (top) and PQRS (bottom) with the same prompt and seed.
Prompts of Figure 1. Wan2.1 (left): “A charming raccoon, wearing a tiny sailor hat and a striped shirt, strums a miniature guitar while sitting in a small wooden boat. The boat gently rocks on the calm, azure ocean under a clear, sunny sky. The raccoon’s nimble fingers pluck the strings with surprising skill, creating a cheerful melody that echoes across the water. Seagulls fly overhead, and the distant horizon is dotted with fluffy white clouds. The raccoon’s eyes sparkle with joy as it plays, its bushy tail swaying in time with the music, creating a whimsical and heartwarming scene.” HunyuanVideo (right): “A sleek, modern bicycle with a matte black frame and aerodynamic design begins its journey on a smooth, sunlit road. The rider, clad in a fitted, neon green cycling suit and helmet, leans forward, gripping the handlebars tightly. The camera captures the initial slow pedal strokes, the wheels spinning with increasing speed. As the bicycle accelerates, the background blurs, emphasizing the rapid motion. The rider’s muscles tense and flex, showcasing the effort and determination. The sunlight glints off the bike’s frame and the rider’s helmet, creating a dynamic interplay of light and shadow. The sound of the wind rushing past and the rhythmic clicking of the gears enhance the sensation of speed and exhilaration.”
Figure 9. Wan2.1-I2V-14B, 720p Image-to-Video. Dense Attention | Latency = 22.7 min. PQRS (Ours) | PSNR = 30.20 dB | Latency = 13.0 min (1.74x). a reflection of a city with buildings in the water. Wan2.1-I2V-14B. Dense attention (top) and PQRS (bottom) with the same input image, prompt and seed.
Figure 10. Wan2.2-T2V-A14B, 720p Text-to-Video. Dense Attention | Latency = 23.7 min. PQRS (Ours) | PSNR = 23.38 dB | Latency = 14.3 min (1.66x). A serene, steady shot captures the iconic Bund in Shanghai at twilight, with the historic colonial buildings on one side and the modern skyline of Pudong on the other, all bathed in the soft glow of city lights. The camera glides smoothly along the waterfront promenade, showcasing the bustling activity of locals and tourists alike, framed by the majestic Huangpu River. The scene transitions to a close-up of the intricate architectural details of the historic buildings, then pans out to reveal the vibrant contrast of the futuristic skyscrapers, including the Oriental Pearl Tower, against the evening sky. The video concludes with a tranquil view of the river, reflecting the shimmering lights of the city, encapsulating the harmonious blend of old and new in this dynamic metropolis. Wan2.2-T2V-A14B. Dense attention (top) and PQRS (bottom) with the same prompt and seed.
References
- [1]Jiayu Chen, Zhikun Jiang, Maoliang Li, Jiayi Luo, Jiawei Yang, Zihao Zheng, Hengyi Zhang, Guojie Luo, and Xiang Chen. Token radius attention for efficient video generation, 2026. 2arxiv.org/abs/2608.02504
- [2]Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024. 3, 5arxiv.org/abs/2307.08691
- [3]DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models, 2025. 2arxiv.org/abs/2512.02556
- [4]Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. 6arxiv.org/abs/2207.12598
- [5]Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Yuhan Dong, and Yu Wang. Flashdecoding++: Faster large language model inference with asynchronization, flat gemm optimization, and heuristics. In Proceedings of Machine Learning and Systems, pages 148–161, 2024. 3, 6
- [6]Coleman Richard Charles Hooper, Sehoon Kim, Hiva Mohammad-zadeh, Monishwaran Maheswaran, Sebastian Zhao, June Paik, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami. Squeezed attention: Accelerating long context length LLM inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32631–32652, Vienna, Austria, 2025. Association for Computational Linguistics. 2arxiv.org/abs/2411.09688
- [7]Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive Benchmark Suite for Video Generative Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21807–21818, 2024. 3, 7arxiv.org/abs/2311.17982
- [8]Ziqi Huang, Fan Zhang, Xiaojie Xu, Yinan He, Jiashuo Yu, Ziyue Dong, Qianli Ma, Nattapol Chanpaisit, Chenyang Si, Yuming Jiang, Yaohui Wang, Xinyuan Chen, Ying-Cong Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(3):3268–3285, 2026. 7DOI
- [9]Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y. Fu, Christopher Ré, and Azalia Mirhoseini. Hydragen: High-throughput llm inference with shared prefixes, 2024. 3arxiv.org/abs/2402.05099
- [10]Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, Kathrina Wu, Qin Lin, Junkun Yuan, Yanxin Long, Aladdin Wang, Andong Wang, Changlin Li, Duojun Huang, Fang Yang, Hao Tan, Hongmei Wang, Jacob Song, Jiawang Bai, Jianbing Wu, Jinbao Xue, Joey Wang, Kai Wang, Mengyang Liu, Pengyu Li, Shuai Li, Weiyan Wang, Wenqing Yu, Xinchi Deng, Yang Li, Yi Chen, Yutao Cui, Yuanbo Peng, Zhentao Yu, Zhiyu He, Zhiyong Xu, Zixiang Zhou, Zunnan Xu, Yangyu Tao, Qinglin Lu, Songtao Liu, Dax Zhou, Hongfa Wang, Yong Yang, Di Wang, Yuhong Liu, Jie Jiang, and Caesar Zhong. Hunyuanvideo: A systematic framework for large video generative models, 2025. 2, 7arxiv.org/abs/2412.03603
- [11]Xunhao Lai, Weiqi Xu, Yufeng Yang, Qiaorui Chen, Yang Xu, Lunbin Zeng, Xiaolong Li, Haohai Sun, Haichao Zhu, Vito Zhang, Jinkai Hu, Jiayao Li, Rui Gao, Zekun Li, Songquan Zhu, Jingkai Zhou, and Pengyu Zhao. Minimax sparse attention, 2026. 2, 5, 6arxiv.org/abs/2606.13392
- [12]Dongyeun Lee, Amir Zandieh, Vahab Mirrokni, Junmo Kim, and Insu Han. Hypertention: Efficient sparse attention with spatio-temporal clustering for video diffusion, 2026. 2arxiv.org/abs/2607.03012
- [13]Haopeng Li, Yitong Li, Junsong Chen, Tian Ye, Haozhe Liu, Jincheng Yu, Duomin Wang, Ruihua Zhang, Zeke Xie, Enze Xie, and Song Han. Solt-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification, 2026. 2arxiv.org/abs/2607.24027
- [14]Haopeng Li, Shitong Shao, Wenliang Zhong, Zikai Zhou, Lichen Bai, Hui Xiong, and Zeke Xie. PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers, 2026. 2, 7, 12arxiv.org/abs/2602.01077
- [15]Muyang Li, Yujun Lin, Zhekai Zhang, Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng, Jun-Yan Zhu, and Song Han. Qdiffquant: Absorbing outliers by low-rank component for 4-bit diffusion models. In International Conference on Learning Representations, pages 97831–97858, 2025. 3arxiv.org/abs/2411.05007
- [16]Xingyang Li, Muyang Li, Tianle Cai, Haochen Xi, Shuo Yang, Yujun Lin, Lvmin Zhang, Songlin Yang, Jinbo Hu, Kelly Peng, Maneesh Agrawala, Ion Stoica, Kurt Keutzer, and Song Han. Radial attention: O(n log n) sparse attention with energy decay for long video generation. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. 2
- [17]Feng Liu, Shiwei Zhang, Xiaofeng Wang, Yujie Wei, Haonan Qiu, Yuzhong Zhao, Yingya Zhang, Qixiang Ye, and Fang Wan. Timestep embedding tells: It’s time to cache for video diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7353–7363, 2025. 3arxiv.org/abs/2411.19108
- [18]Guangda Liu, Chengwei Li, Jieru Zhao, Chenqi Zhang, and Minyi Guo. ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression. In 2025 62nd ACM/IEEE Design Automation Conference (DAC), pages 1–7, 2025. 2arxiv.org/abs/2412.03213
- [19]Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In Advances in Neural Information Processing Systems, 2022. 3arxiv.org/abs/2206.00927
- [20]Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Eming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Yutao Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang, and Jiezhong Qiu. MoBA: Mixture of block attention for long-context LLMs. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. 2, 5arxiv.org/abs/2502.13189
- [21]Jiayi Luo, Jiayu Chen, Jiankun Wang, Cong Wang, Hanxin Zhu, Qingyun Sun, Chen Gao, Zhibo Chen, and Jianxin Li. Attention sparsity is input-stable: Training-free sparse attention for video generation via offline sparsity profiling and online QK co-clustering. In Forty-third International Conference on Machine Learning, 2026. 1, 2, 3, 7, 12arxiv.org/abs/2603.18636
- [22]Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient Content-Based Sparse Attention with Routing Transformers. Transactions of the Association for Computational Linguistics, 9:53–68, 2021. 2
- [23]Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. 3arxiv.org/abs/2010.02502
- [24]Pardis Taghavi, Reza Langari, and Gaurav Pandey. Partition the support, reconstruct the residual: Training-free sparse attention for video generation and world models, 2026. 2arxiv.org/abs/2608.18484
- [25]Haoyue Tan, Shengnan Wang, Yulin Qiao, Juncheng Zhang, Youhui Bai, Ping Gong, Zewen Jin, and Cheng Li. Adacluster: Adaptive query-key clustering for sparse attention in video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 43249–43259, 2026. 1, 2, 3arxiv.org/abs/2604.18348
- [26]Jiaming Tang, Yiling Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. QUEST: Query-aware sparsity for efficient long-context LLM inference. In Forty-first International Conference on Machine Learning, 2024. 2arxiv.org/abs/2406.10774
- [27]Apoorv Vyas, Angelos Katharopoulos, and Francois Fleuret. Fast transformers with clustered attention. In Advances in Neural Information Processing Systems, pages 21665–21674. Curran Associates, Inc., 2020. 2arxiv.org/abs/2007.04825
- [28]Team Wan, Ang Wang, Baoai Bi, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxiang Wang, Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang, Wei Yang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models, 2025. 1, 2, 3, 7arxiv.org/abs/2503.20314
- [29]Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004. 7pubmed.ncbi.nlm.nih.gov/15376593
- [30]Jianzong Wu, Liang Hou, Haotian Yang, Ye Tian, Pengfei Wan, Di ZHANG, and Yunhai Tong. VMoBA: Mixture-of-block attention for video diffusion models. In The Fourteenth International Conference on Learning Representations, 2026. 2arxiv.org/abs/2506.23858
- [31]Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, Jianfei Chen, Ion Stoica, Kurt Keutzer, and Song Han. Sparse video-gen: Accelerating video diffusion transformers with spatial-temporal sparsity. In Forty-second International Conference on Machine Learning, 2025. 2, 7, 12arxiv.org/abs/2502.01776
- [32]Yifei Xia, Suhan Ling, Fangcheng Fu, Yujie Wang, Huixia Li, Xuefeng Xiao, and Bin Cui. Training-free and adaptive sparse attention for efficient long video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15982–15993, 2025. 2, 6arxiv.org/abs/2502.21079
- [33]Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. XAttention: Block sparse attention with antidiiagonal scoring. In Forty-second International Conference on Machine Learning, 2025. 2arxiv.org/abs/2503.16428
- [34]Ran Yan, Youhe Jiang, Zhuoming Chen, Haohui Mai, Beidi Chen, and Binhang Yuan. FSA: An alternative efficient implementation of native sparse attention kernel. In The Fourteenth International Conference on Learning Representations, 2026. 2, 5arxiv.org/abs/2508.18224
- [35]Shuo Yang, Haocheng Xi, Yilong Zhao, Muyang Li, Jintao Zhang, Han Cai, Yujun Lin, Xiuyu Li, Chenfeng Xu, Kelly Peng, Jianfei Chen, Song Han, Kurt Keutzer, and Ion Stoica. Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. 1, 2, 3, 4, 5, 7, 8, 12
- [36]Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Frédo Durand, William T. Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6613–6623, 2024. 3arxiv.org/abs/2311.18828
- [37]Jingyang Yuan, Huazao Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse attention: Hardware-aligned and natively trainable sparse attention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23078–23097, Vienna, Austria, 2025. Association for Computational Linguistics. 2arxiv.org/abs/2502.11089
- [38]Jintao Zhang, Jia Wei, Pengle Zhang, Jun Zhu, and Jianfei Chen. Sagattention: Accurate 8-bit attention for plug-and-play inference acceleration. In The Thirteenth International Conference on Learning Representations, 2025. 3arxiv.org/abs/2410.02367
- [39]Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. Spargattention: Accurate and training-free sparse attention accelerating any model inference. In Forty-second International Conference on Machine Learning, 2025. 2arxiv.org/abs/2502.18137
- [40]Peiyuan Zhang, Yongqi Chen, Haofeng Huang, Will Lin, Zhengzhong Liu, Ion Stoica, Eric P. Xing, and Hao Zhang. Faster video diffusion with trainable sparse attention. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. 2DOI
- [42]Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 586–595, Los Alamitos, CA, USA, 2018. IEEE Computer Society. 7arxiv.org/abs/1801.03924
- [43]Qi Zhao, Qirui Lu, Hanlin Tang, Yiduo Li, Zhen Guo, Cuifeng Shen, Chao Xu, Zhaosheng Chi, Xiaojin Lu, Kan Liu, Tao Lan, Lin Qu, and Xi Li. Scope: Subspace clustering with online per-head top-k estimation for sparse video attention, 2026. 2arxiv.org/abs/2608.12780
- [44]Xuanlei Zhao, Xiaolong Jin, Kai Wang, and Yang You. Real-time video generation with pyramid attention broadcast. In The Thirteenth International Conference on Learning Representations, 2025. 3arxiv.org/abs/2408.12588
- [45]Xuanyi Zhou, Qiuyang Mang, Shuo Yang, Haocheng Xi, Jintao Zhang, Huanzhi Mao, Joseph E. Gonzalez, Kurt Keutzer, Ion Stoica, and Alvin Cheung. SVG-EAR: parameter-free linear compensation for sparse video generation via error-aware routing. In Computer Vision - ECCV 2026 - 19th European Conference, Malmö, Sweden, September 8-12, 2026, Proceedings, Part L, pages 167–185. Springer, 2026. 1, 2, 3, 7, 8, 12arxiv.org/abs/2603.08982










