WorksComputer Science
Spark-H3: Adaptive Block Sparse Attention for Video Diffusion Models
Abstract
Long video sequences make attention a substantial cost in video diffusion models. Block sparse attention (BSA) reduces this cost by computing selected blocks exactly while compressing the remaining blocks. However, fixed token grouping can place tokens with dissimilar attention preferences in the same group, while uniform key averaging can underestimate the attention mass of compressed blocks. We introduce Spark-Attn, a training-free BSA method comprising Spark-Reblock and Spark-Reweight. Spark-Reblock recursively groups tokens by attention preference while preserving regular block shapes for efficient GPU execution. Spark-Reweight uses the global mean query as an anchor to construct attention-weighted key-value representations and logit biases for the compressed blocks, calibrating their attention mass. We instantiate Spark-Attn in MiniMax-H3 as Spark-H3. At 10% exact-block density, Spark-H3 delivers a 2.49 DiT speedup and reaches 24.81 dB PSNR against the corresponding dense outputs on MiniMax-H3 (20 denoising steps, 345 frames, 1344768). Increasing the density to 20% further improves PSNR to 26.76 dB while still achieving a 2.26 DiT speedup. These results demonstrate that Spark-H3 achieves a state-of-the-art quality-efficiency trade-off for training-free BSA on video diffusion models.
*Equal contribution. †Corresponding author.
Email: [email protected]
Source Code: https://github.com/zechengtang/Spark-H3
Figure 1. MiniMax-H3 and Spark-H3. (a,b) Qualitative comparisons on 14.4s, 1344 × 768 videos generated using 20 denoising steps. Spark-H3 uses exact-block density in these examples. (c) DiT speedup vs. PSNR for five state-of-the-art sparse-attention methods and two Spark-H3 configurations () under the same inference setting. (d) DiT latency and speedup across video durations. The fully sparse Spark-H3 variant omits the 4-step dense warm-up and provides an optimistic upper bound on speedup.
Introduction
Video diffusion transformers (DiTs) [16] generate videos by repeatedly processing spatiotemporal token sequences throughout the denoising process. Dense self-attention [20] compares every query token with every key token, incurring quadratic complexity with respect to sequence length. The FlashAttention family [2, 3, 17, 29] improves memory efficiency and hardware utilization but retains this dense all-pairs computation. Attention cost therefore grows rapidly with video resolution and duration and can become a major bottleneck at inference time. Reducing this cost while preserving generation quality is important for efficient high-quality video generation.
Recent open-source video diffusion models, including HunyuanVideo [9], the Wan family [19], the LTX-2 family [6], and MiniMax-H3 [15], demonstrate the rapid progress of video generation. Within this landscape, we study attention acceleration mainly on MiniMax-H3, whose strong generative capability makes it a practically relevant testbed for assessing whether attention acceleration can provide substantial speedups while preserving output fidelity. Unlike trainable sparse or hybrid attention methods such as VSA and Video DeltaNet [24, 34], we consider a training-free setting in which the pretrained parameters and denoising schedule remain unchanged while dense attention is replaced by sparse attention during inference. Because the model is trained with dense attention, preserving its inference behavior helps limit the training–inference discrepancy introduced by this replacement. Our goal is therefore to reduce attention latency while keeping the generated outputs close to their dense-attention counterparts.
Early training-free sparse attention methods for video diffusion models exploit structured spatial–temporal redundancy through head-specific patterns or localized sliding tiles [23, 35]. Prior block sparse attention (BSA) methods [10, 11, 26, 31] often use fixed token grouping, which can place tokens with dissimilar attention preferences in the same group. Cluster sparse attention (CSA) methods [14, 27, 38] instead regroup tokens adaptively to better align grouping with attention preferences. Within BSA, fixed grouping spreads important query–key interactions across more blocks, making them harder to capture at a fixed exact-block density. Meanwhile, uniform key averaging replaces the mean of the exponentiated logits within a block with the exponential of their mean. The resulting Jensen gap can cause compressed blocks to receive too little attention mass. We identify these two failure modes as preference mismatch and compression bias, respectively.
We introduce Spark-Attn, a training-free BSA method that addresses both failure modes. As illustrated in Figure 2, Spark-Reblock uses recursive balanced splits to group query and key–value (KV) tokens with similar attention preferences while preserving regular block shapes. Spark-Reweight uses the global mean query as an anchor to construct attention-weighted KV representations and logit biases for the compressed blocks, calibrating their attention mass. We instantiate Spark-Attn in MiniMax-H3 as Spark-H3 without parameter training, adapters, or learned auxiliary modules. Our key contributions are as follows:
Figure 2. BSA, CSA, and Spark-Reblock. Colors denote attention preferences, and denotes the number of video tokens. Spark-Reblock uses recursive balanced splits to construct preference-aligned, uniformly sized tiles in .
We introduce Spark-Attn, a training-free BSA for video diffusion models, which comprising Spark-Reblock and Spark-Reweight.
We propose Spark-Reblock, a preference-aware hierarchical partitioning method that aligns token grouping with attention structure while preserving regular block shapes for efficient GPU execution.
We develop Spark-Reweight, which uses a global query anchor to calibrate the attention mass of compressed blocks.
We instantiate Spark-Attn in MiniMax-H3 as Spark-H3 and evaluate it across several video diffusion models, demonstrating consistent state-of-the-art quality–efficiency trade-offs for training-free BSA on video diffusion models.
Related work
Training-free sparse attention. Training-free sparse attention accelerates pretrained video diffusion models without updating model parameters or training auxiliary modules. Early pattern-based methods exploit spatial–temporal redundancy using fixed layouts or by selecting head-specific layouts from predefined pattern families. Sparse VideoGen (SVG) [23] profiles spatial and temporal attention heads online, Sliding Tile Attention (STA) [35] computes dense attention within local spatiotemporal windows, and PAROAttention [37] selects per-head layouts from six spatiotemporal axis permutations through offline profiling. Block sparse attention (BSA) makes regular blocks the unit of routing, computing selected blocks exactly and handling unselected blocks through a compressed branch, with dropping as the zero-contribution special case. SpargeAttention [31] and XAttention [26] focus on block-importance estimation, while AdaSpa [25] searches block masks at selected denoising steps and reuses sparse patterns across steps. PISA [11] and Sol-Attn [10] approximate the contributions of unselected blocks through block-wise Taylor correction and compressed KV representations, respectively. Cluster sparse attention (CSA) methods instead use adaptive token grouping to expose sparse structure. SVG2 [27] uses semantic-aware permutation, SVOO [14] combines offline profiling with online query–key co-clustering, and SVG-EAR [38] adds error-aware routing and centroid compensation. Spark-Reblock combines adaptive token grouping with regular block shapes, while Spark-Reweight uses the global mean query as an anchor to calibrate the attention mass of compressed blocks.
Training-based attention adaptation. Training-based approaches either train models natively with efficient attention or adapt pretrained models. Representative native architectures include MoBA [13], which formulates block routing as a mixture-of-experts process, and NSA [28], which combines coarse-grained token compression with fine-grained token selection. VMoBA [22] and Log-linear Sparse Attention [39] extend trainable sparse attention to video DiTs, while SANA-Video [1] adopts a linear DiT with autoregressive state for long-video generation. VSA [34] supports both pretraining with sparse attention and adapting pretrained models. Post-training adaptation methods include Veda [7] and SpargeAttention2 [32], which adapt BSA, and SLA and SLA2 [30, 33], which combine sparse and linear attention. VORTA [18] learns a lightweight timestep-conditioned router over three predefined attention patterns while freezing the pretrained backbone. For MiniMax-H3 specifically, FastH3 [4, 5] combines trained sparse attention with few-step distillation, whereas Video DeltaNet [24] learns a hybrid of local softmax attention and linear memory. Because these approaches train attention parameters or auxiliary modules for a target model, applying them to another LoRA of the same model or to a different model may require further optimization. Spark-Attn instead restructures attention computation itself, requiring neither model-specific adaptation nor retraining when applied to different LoRAs of the same model or to different models.
Method
Spark-Attn accelerates dense self-attention without adapting the pretrained parameters. We call a group of query or KV tokens a tile, and the query–key score matrix between a query tile and a KV tile an attention block. Given the query, key, and value matrices , , for one attention head, Spark-Attn uses a block router to select a configured fraction , for example, 20%, of candidate KV tiles per query tile for exact computation; a compressed branch approximates the remaining blocks. As shown in Figure 3, Spark-Reblock addresses preference mismatch by constructing uniformly sized tiles with similar attention preferences, allowing the block router to capture more attention mass at the same exact-block density . Spark-Reweight addresses compression bias by constructing attention-weighted KV representatives and a logit bias for each unselected KV tile. The two branches share one softmax normalization, and the output is restored to the original query order. We next formalize the two-branch computation, then detail preference-aware tile construction in Spark-Reblock and anchor-based compression in Spark-Reweight.
Figure 3. Spark-Attn overview. (1) Block routing: Spark-Reblock groups query and key–value (KV) tokens into preference-aware, uniformly sized tiles; tile means guide selection of a fraction of KV tiles per query tile. (2) Two-branch attention: Selected blocks are computed exactly, while the rest are approximated using Spark-Reweight’s weighted KV representatives and logit biases. Both branches share one softmax normalization, with compressed entries for selected blocks masked to avoid double counting. , , and denote the number of video tokens, the number of KV tiles, and the exact-block density, respectively.
Two-Branch Block Sparse Attention
For one attention head over video tokens, let , , be the query, key, and value matrices, with head dimension . For a query and a key , define the scaled dot-product score as . Dense attention [20] exponentiates these scores to weight the corresponding values . For subsequent block-wise accumulation, we equivalently write dense attention as
Here is the value-weighted numerator and is the total attention mass.
Tiles and attention blocks. The scores form a matrix . FlashAttention [3] computes these scores block by block on chip while updating the numerator and denominator in Eq. (1). Let and denote query and key tiles of sizes and , respectively. The values aligned with the latter form , giving the KV tile . Their scaled matrix product defines an attention block, whose entries are the individual query–key scores.
Block routing. Block sparse attention (BSA) reduces the cost of Eq. (1) by computing token-level scores only in selected blocks. Following MoBA [13], we use tile means for block routing. For query tile and KV tile , let and denote their respective mean query and key; the router scores the pair by .
For each query tile , it selects a set containing the highest-scoring fraction of the candidate KV tiles, which we call the configured exact-block density. Every query in tile shares this selection. The tile layout therefore determines which token-level scores can be computed together under a given exact-block budget, directly affecting how much dense attention mass the selected blocks capture.
Approximating unselected blocks. Unselected blocks can still carry substantial attention mass in aggregate. We therefore use a two-branch formulation in which selected blocks contribute their token-level scores exactly, whereas unselected blocks contribute through a compressed approximation.
For a query , the exact contribution of a KV tile to Eq. (1) is , where indexes the token positions in the tile. The exact branch computes this contribution from token-level scores for each selected tile. For each unselected tile, the compressed branch approximates its contribution by . Accumulating the two branches gives
Each block contributes through exactly one branch, and compressed entries corresponding to selected blocks are masked to avoid double counting. The final output is , following Eq. (1). Because both branches contribute to , compressed blocks also affect the normalized weights of the selected keys.
This two-branch formulation leaves two design choices: the tile layout used for block routing and the approximation of unselected blocks. We address them next with Spark-Reblock and Spark-Reweight, respectively.
Spark-Reblock
Fixed token grouping couples block routing to the spatiotemporal layout rather than to the attention structure. When a tile mixes tokens with dissimilar attention preferences, the tile means used by the block router become less representative of the token-level scores, increasing the chance that high-mass blocks are unselected at a given exact-block density. Spark-Reblock addresses this preference mismatch by using the current and to construct a token permutation and uniformly sized tiles that group tokens with similar attention preferences. It thus combines the adaptive grouping of cluster sparse attention (CSA) with the regular block shapes of block sparse attention (BSA). As illustrated in Figure 2, its construction is analogous to median-pivot quicksort: each node uses a landmark-derived direction to define scalar split scores, and equal-capacity splits partition them at the median rank. The objective, however, is uniformly sized tiles rather than a globally sorted sequence.
Attention preference. We define attention preference as the ranking induced by scaled dot-product scores: all keys for a query, or all queries for a key. For a query , the corresponding row of lists its scores against all keys. Normalizing this row gives . Rather than comparing raw query features by Euclidean distance, we use cosine similarity between normalized score vectors as a surrogate for attention-preference similarity. The non-centered second moment expresses this comparison as a cosine distance under a key-induced Mahalanobis-type inner product:
The distance between keys follows symmetrically by comparing their interactions with the current query set using . Before recursion, we estimate from sampled tokens and reuse the resulting metric throughout the partition tree, avoiding construction of the full attention matrix.
Node-specific split direction. A single global split direction may not capture the preference structure remaining within every node. Each internal node therefore fits two representatives and from its Algorithm 1 Recursive tile construction in Spark-Reblock
1: Inputs: token features , tile size , estimated second moment , and child capacities in multiples of . 2: Output: ordered list of tiles. 3: Init: for each 4: return 5: procedure 6: if then 7: return 8: end if 9: Read the prescribed left child capacity for 10: Initialize from a dissimilar landmark pair 11: Refine both representatives on the local landmarks 12: Compute for by Eq. (4) 13: indices of the smallest 14: 15: return followed by 16: end procedure
local landmarks. For a token with transformed feature , we define the scalar split score
The difference between the two unit-normalized representatives gives the node’s split direction, and the resulting scores provide the ordering used for rank partitioning.
Recursive rank partitioning. Recursive splitting terminates when , yielding uniformly sized tiles. Child capacities are imposed through rank cutoffs, reducing to median-rank splits when the two capacities are equal. Algorithm 1 summarizes the procedure.
Batched construction. A full partitioning round batches four binary levels: a landmark tree constructs 15 split directions for 16 child groups, and one batched matrix product evaluates their split scores before rank cutoffs and token permutation. With fixed feature dimension and landmark budget, linear-time rank selection gives construction time in the token count. Spark-H3 uses 32 landmarks, two refinement updates per split, and 64 tokens per tile, by default.
Partitioning and routing. Spark-H3 derives a single partition from the keys using and applies its permutation to queries, keys, and values, avoiding a second partitioning pass. The attention output is restored to the original query order. Spark-H3† instead uses separate query and key–value partitions constructed with and , respectively. After reblocking, the block router scores each query–KV tile pair using the query and key tile means and selects the top fraction of KV tiles per query tile for the exact branch.
Spark-Reweight
Consider a KV tile left unselected for query tile , i.e., . For any query , the compressed branch must approximate both the tile’s total attention mass and its value-weighted numerator . Mean-pooled KV compression uses and estimates the log-mass by . Jensen’s inequality gives the nonnegative gap
which can make compressed blocks receive too little attention mass. Spark-Reweight addresses this compression bias by constructing attention-weighted key and value representatives together with a scalar logit bias, using the global mean query as a shared anchor.
Spark-Reweight first computes the global mean query for each attention head. For any query , let denote the exponentiated scores within KV tile , with the exponential applied elementwise. Evaluating these scores at the anchor and normalizing within the tile gives the pooling weights
These weights are shared by all queries and assign greater weight to keys with higher scores at the anchor. Using these weights, Spark-Reweight constructs the key and value representatives and . It pairs them with the logit bias
The bias measures the difference between the tile’s exact log-mass and the score of its weighted key representative at the anchor. For an unselected block, the resulting representation gives
These contributions enter Eq. (2) directly. The representatives and bias are computed once from the current , , and reused across queries, while the estimated mass remains query-dependent through .
The weight–bias construction admits a simple mathematical interpretation as a first-order approximation to the tile’s log-sum-exp. Mean-pooled compression corresponds to expanding at a zero anchor: it uses uniform weights and , giving and hence the Jensen gap in Eq. (5). Spark-Reweight instead expands at the observed mean query. Since , the first-order expansion is
Equation (8) therefore gives exactly . At , the tangent matches both the log-mass and its gradient, and equals the tile’s normalized value aggregate; consequently, and . Away from the anchor, the construction remains a first-order approximation rather than an exactness guarantee.
Experiments
Evaluation protocol
Benchmark. We evaluate all sparse-attention methods on the VBench benchmark suite [8]. Following prior work [14, 27, 38], we use GPT-enhanced prompts from the Penguin Benchmark. For MiniMax-H3, we adapt these prompts to its prompt template [15].
Metrics. We evaluate quality and efficiency. For efficiency, we measure DiT latency, excluding text encoding and VAE decoding, and report speedup over the DiT latency of dense-attention results under identical generation settings. For quality, we use PSNR, SSIM [21], and LPIPS [36] to measure fidelity to dense-attention outputs, capturing the inference discrepancy introduced by sparse-attention replacement. We also report the average VBench score over the five core dimensions used in prior work [11, 14, 27, 38].
Baselines. We denote the original model outputs as Dense and compare Spark-H3 and Spark-H3{sup}† with five recent state-of-the-art training-free sparse-attention baselines: SVG2 {citep}[27], SVG-EAR {citep}[38], SVOO {citep}[14], PISA {citep}[11], and Sol-Attn {citep}[10]`.
Quantitative Results
We evaluate Spark-Attn on MiniMax-H3 [15] and LTX-2.5 [6], two leading video diffusion models. These experiments assess quality–efficiency trade-offs across distinct model architectures and generation settings.
Quantitative Results on MiniMax-H3. Table 1 summarizes the MiniMax-H3 results. We generate 14.4s videos at resolution using a 20-step denoising schedule. All sparse-attention methods use four initial dense denoising steps as warm-up before switching to sparse attention. At 10% exact-block density, Spark-H3 achieves the lowest DiT latency and a 2.49× speedup, with a PSNR within 0.07 dB of the best baseline result. Increasing the density to 20% improves PSNR to 26.76 dB while still achieving a 2.26× speedup. Spark-H3 improves PSNR over two BSA methods at similar density, with gains of 4.61 dB over PISA at 10% and 5.50 dB over Sol-H3 at 20%. At 20% exact-block density, both Spark-H3 variants achieve higher PSNR and SSIM, lower LPIPS, and lower DiT latency than all three CSA baselines. These results establish a state-of-the-art quality–efficiency trade-off among the evaluated methods.
| Method | Category | Density | DiT latency (s) ↓ | Speedup ↑ | PSNR (dB) ↑ | SSIM ↑ | LPIPS ↓ | VBench ↑ |
| Dense | – | 100% | 4061.8 | 1.00× | ∞ | 1.00 | 0.00 | 84.38 |
| SVG2 [27] | CSA | ≈ 17.8% | 1876.1 | 2.16× | 23.70 | 0.79 | 0.15 | 84.39 |
| SVG-EAR [38] | CSA | ≈ 17.8% | 1929.3 | 2.11× | 24.88 | 0.82 | 0.13 | 84.50 |
| SVOO [14] | CSA | ≈ 19.7% | 1847.6 | 2.20× | 23.73 | 0.79 | 0.14 | 84.56 |
| PISA [11] | BSA | 10% | 1643.4 | 2.47× | 20.20 | 0.71 | 0.23 | 84.60 |
| Sol-H3 [10] | BSA | ≈ 20.8% | 1781.9 | 2.28× | 21.26 | 0.74 | 0.19 | 84.47 |
| Spark-H3 | BSA | 10% | 1629.6 | 2.49× | 24.81 | 0.83 | 0.12 | 84.56 |
| Spark-H3 | BSA | 20% | 1796.1 | 2.26× | 26.76 | 0.86 | 0.09 | 84.50 |
| Spark-H3† | BSA | 20% | 1818.0 | 2.23× | 26.89 | 0.87 | 0.08 | 84.56 |
Table 1. Table 1. Quantitative Results on MiniMax-H3. Green shading marks the first, second, and third distinct scores among methods. † uses separate query and KV partitions.
Quantitative Results on LTX-2.5. LTX-2.5 uses a two-stage denoising process. We generate 10s videos at resolution using an 11-step schedule: 8 steps at , followed by 3 steps at the target resolution after latent upsampling. Following Sol-Attn [10], we keep the first stage dense and apply sparse-attention methods only to the second stage. Table 2 therefore reports DiT latency of the second stage, excluding the first stage and latent upsampling.
| Method | Category | Density | DiT latency (s) ↓ | Speedup ↑ | PSNR (dB) ↑ | SSIM ↑ | LPIPS ↓ | VBench ↑ |
| Dense | – | 100% | 1223.6 | 1.00× | ∞ | 1.00 | 0.00 | 82.10 |
| SVG2 | CSA | ≈ 54.0% | 869.6 | 1.41× | 30.80 | 0.94 | 0.09 | 82.30 |
| SVG-EAR | CSA | ≈ 53.9% | 878.3 | 1.39× | 34.20 | 0.96 | 0.06 | 82.38 |
| SVOO | CSA | ≈ 56.0% | 891.9 | 1.37× | 29.90 | 0.93 | 0.10 | 82.93 |
| PISA | BSA | 10% | 313.4 | 3.90× | 29.20 | 0.92 | 0.12 | 82.00 |
| Sol-Attn | BSA | ≈ 18.5% | 385.7 | 3.17× | 24.72 | 0.88 | 0.21 | 81.21 |
| Spark-Attn | BSA | 10% | 312.0 | 3.92× | 30.84 | 0.93 | 0.09 | 82.38 |
| Spark-Attn | BSA | 20% | 415.1 | 2.95× | 32.68 | 0.94 | 0.07 | 82.39 |
| Spark-Attn | BSA | 50% | 736.9 | 1.66× | 36.79 | 0.97 | 0.04 | 82.33 |
| Spark-Attn† | BSA | 50% | 737.8 | 1.66× | 35.16 | 0.96 | 0.05 | 82.44 |
Table 2. Table 2. Quantitative Results on LTX-2.5 (High-Resolution Stage). Notation and shading follow Table 1.
Spark-Attn achieves a strong quality–efficiency trade-off on LTX-2.5. At 10% density, it matches PISA’s DiT latency while improving PSNR by 1.64 dB, achieving a 3.92× speedup. Increasing the density to 20% and 50% progressively improves PSNR. At 50% density, Spark-Attn achieves higher PSNR and lower latency than all three CSA baselines. Together with the MiniMax-H3 results, these experimental results support the applicability of Spark-Attn across distinct video diffusion models.
Compatibility with Few-Step Distillation
Quantitative Results on MiniMax-H3-Turbo. We further evaluate whether Spark-H3 can be combined with few-step diffusion distillation using MiniMax-H3-Turbo [12], a community-released 8-step LoRA for MiniMax-H3. We generate 14.4-second videos at resolution using eight denoising steps, with two initial dense denoising steps as warm-up before switching to sparse attention.
As shown in Table 3, Spark-H3 achieves a 2.27 DiT speedup at 10% exact-block density and 24.80 dB PSNR against Dense. Increasing the density to 20% further improves PSNR to 26.48 dB while achieving a 2.08 speedup. At 20% density, Spark-H3 also achieves higher PSNR and SSIM and lower LPIPS than all five baselines, while remaining faster than the three CSA baselines. These results demonstrate the effectiveness of Spark-H3 when combined with few-step distillation.
| Method | Category | Density | DiT latency (s) | Speedup | PSNR (dB) | SSIM | LPIPS | VBench |
| Dense | – | 100% | 1625.0 | 1.00 | 1.00 | 0.00 | 84.32 | |
| SVG2 | CSA | 17.6% | 846.7 | 1.92 | 24.01 | 0.78 | 0.15 | 84.32 |
| SVG-EAR | CSA | 17.6% | 864.8 | 1.88 | 25.01 | 0.81 | 0.13 | 84.31 |
| SVOO | CSA | 19.1% | 829.0 | 1.96 | 24.33 | 0.80 | 0.14 | 84.54 |
| PISA | BSA | 10% | 718.7 | 2.26 | 21.28 | 0.71 | 0.22 | 84.39 |
| Sol-H3 | BSA | 20.9% | 775.6 | 2.10 | 22.44 | 0.75 | 0.17 | 84.43 |
| Spark-H3 | BSA | 10% | 717.2 | 2.27 | 24.80 | 0.81 | 0.13 | 84.37 |
| Spark-H3 | BSA | 20% | 780.2 | 2.08 | 26.48 | 0.85 | 0.09 | 84.36 |
| Spark-H3 | BSA | 20% | 785.0 | 2.07 | 26.65 | 0.85 | 0.09 | 84.34 |
Table 3. Quantitative Results on MiniMax-H3-Turbo. Notation and shading follow Table 1.
Ablation studies
Spark-Reblock and attention-mass recall. Table 4 compares configurations with and without Spark-Reblock while keeping Spark-Reweight enabled. Attention-mass recall is the fraction of dense-softmax attention mass captured by selected blocks. Spark-Reblock increases recall by 18.33 and 14.58 percentage points at 10% and 20% density, respectively, and improves PSNR by 4.51 and 5.11 dB. Moreover, Spark-Reblock at achieves higher recall and PSNR with lower DiT latency than the configuration without Reblock at , demonstrating a better quality–efficiency trade-off than doubling the exact-block budget. These results show that Spark-Reblock effectively addresses preference mismatch through preference-aware token grouping.
| Spark-Reblock | Density | DiT Latency (s) | Recall (%) | PSNR (dB) |
| 10% | 1598.6 | 61.36 2.71 | 20.30 | |
| ✓ | 10% | 1629.6 | 79.69 0.91 | 24.81 |
| 20% | 1763.2 | 74.05 2.12 | 21.65 | |
| ✓ | 20% | 1796.1 | 88.63 0.61 | 26.76 |
Table 4. Ablation of token grouping methods. Attention-mass recall is the fraction of dense-softmax attention mass captured by selected blocks (mean sample SD).
Spark-Reweight and compressed-block error. Table 5 isolates Spark-Reweight at 20% density with and without Spark-Reblock. Without Spark-Reblock, it reduces log-mass MAE from 3.38 to 2.52 and increases PSNR from 21.24 to 21.65 dB. With Spark-Reblock, it reduces log-mass MAE from 1.39 to 1.16 and increases PSNR from 26.24 to 26.76 dB. In both configurations, lower compressed-block mass error is accompanied by higher dense-reference fidelity. These results show that Spark-Reweight mitigates compression bias by reducing the Jensen gap in compressed-block attention-mass estimation.
| Spark-Reblock | Spark-Reweight | DiT Latency (s) | MAE | PSNR (dB) |
| 1765.0 | 3.38 | 21.24 | ||
| ✓ | 1763.2 | 2.52 | 21.65 | |
| ✓ | 1796.0 | 1.39 | 26.24 | |
| ✓ | ✓ | 1796.1 | 1.16 | 26.76 |
Table 5. Ablation of compression methods. Log-mass MAE is the mean absolute error between predicted and dense block log-masses.
Qualitative results
We present qualitative comparisons of MiniMax-H3 outputs for two prompts. In the attic scene (Figure 4), Spark-H3 and SVG-EAR produce more plausible body poses and motion transitions between the two consecutive frames spanning the shot cut than the other sparse methods. Spark-H3 also retains finer hair detail than SVG-EAR. In the party scene (Figure 5), the two BSA baselines, PISA and Sol-Attn, exhibit visible inconsistencies in the red hoodie graphic across the two views, whereas all three CSA methods and Spark-H3 preserve its structure more consistently. These examples illustrate Spark-H3’s ability to preserve temporal coherence and fine visual details.
Figure 4. Motion incoherency. Zoom in to compare body poses between adjacent frames and hair details.
Figure 5. Graphic inconsistency. Cyan and yellow boxes mark the hoodie regions enlarged below; zoom in to compare the graphics.
Conclusion
We presented Spark-Attn, a training-free BSA method comprising Spark-Reblock and Spark-Reweight. Spark-Reblock uses quicksort-inspired recursive balanced splits to group tokens by attention preference into uniformly sized tiles, combining BSA’s efficiency with CSA’s adaptivity. Spark-Reweight reinterprets mean pooling as a first-order Taylor expansion at the zero query and shifts the expansion point to the global mean query, the center of the observed query distribution, yielding a new, principled, and lightweight compression approach. Extensive experiments across video diffusion models demonstrate Spark-Attn’s effectiveness and generalizability, establishing it as a leading approach to block sparse attention optimization.
References
References
- [1]Junsong Chen, Yuyang Zhao, Jincheng Yu, Ruihang Chu, Junyu Chen, Shuai Yang, Xianbang Wang, Yicheng Pan, Daquan Zhou, Huan Ling, Haozhe Liu, Hongwei Yi, Hao Zhang, Muyang Li, Yukang Chen, Han Cai, Sanja Fidler, Ping Luo, Song Han, and Enze Xie. SANA-Video: Efficient video generation with block linear diffusion transformer. arXiv preprint arXiv:2509.24695, 2025.arxiv.org/abs/2509.24695
- [2]Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations, 2024.arxiv.org/abs/2307.08691
- [3]Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2205.14135
- [4]FastVideo Team. FastH3 Preview v1: Four open-weight H3 models in four steps. Project blog, 2026. Accessed 2026-10-19.
- [5]FastVideo Team. FastVideo-FastH3-8-Step-V2. Hugging Face model repository, 2026. Accessed 2026-11-17.
- [6]Yoav HaCohen, Benny Brazowski, Nisan Chiprut, Yaki Bitterman, Andrew Kovchko, et al. LTX-2: Efficient joint audio-visual foundation model. arXiv preprint arXiv:2601.03233, 2026.arxiv.org/abs/2601.03233
- [7]Shihao Han, Hao Yang, Xinting Hu, Xiaofeng Mei, Yi Jiang, and Xiaojuan Qi. Veda: Scalable video diffusion via distilled sparse attention. In Proceedings of the International Conference on Machine Learning, 2026.arxiv.org/abs/2605.30325
- [8]Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chengyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. arXiv preprint arXiv:2311.17982, 2023.arxiv.org/abs/2311.17982
- [9]Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. HunyuanVideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024.arxiv.org/abs/2412.03603
- [10]Haopeng Li, Yitong Li, Junsong Chen, Tian Ye, Haozhe Liu, Jincheng Yu, Duomin Wang, Ruihua Zhang, Zeke Xie, Enze Xie, and Song Han. Sol-Attn: Accelerating video generation inference via on-the-fly attention sparsification. arXiv preprint arXiv:2607.24027, 2026.arxiv.org/abs/2607.24027
- [11]Haopeng Li, Shitong Shao, Wenliang Zhong, Zikai Zhou, Lichen Bai, Hui Xiong, and Zeke Xie. PISA: Piecewise sparse attention is wiser for efficient diffusion transformers. arXiv preprint arXiv:2602.01077, 2026.arxiv.org/abs/2602.01077
- [12]lightx2v. MiniMax-H3-Turbo. Hugging Face model repository, 2026. Accessed 2026-11-16.
- [13]Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Neo Y. Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang, and Jiezhong Qiu. MoBA: Mixture of block attention for long-context LLMs. arXiv preprint arXiv:2502.13189, 2025.arxiv.org/abs/2502.13189
- [14]Jiayi Luo, Jiayu Chen, Jiankun Wang, Cong Wang, Hanxin Zhu, Qingyun Sun, Chen Gao, Zhibo Chen, and Jianxin Li. Attention sparsity is input-stable: Training-free sparse attention for video generation via offline sparsity profiling and online QK co-clustering. arXiv preprint arXiv:2603.18636, 2026.arxiv.org/abs/2603.18636
- [15]MiniMax-AI. MiniMax H3. Official model repository, 2026. Accessed 2026-10-17.
- [16]William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023.arxiv.org/abs/2212.09748
- [17]Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. FlashAttention-3: Fast and accurate attention with asynchronous and low-precision. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2407.08608
- [18]Wenhao Sun, Rong-Cheng Tu, Yifu Ding, Zhao Jin, Jingyi Liao, Shunyu Liu, and Dacheng Tao. VORTA: Efficient video diffusion via routing sparse attention. arXiv preprint arXiv:2505.18809, 2025.arxiv.org/abs/2505.18809
- [19]Team Wan et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025.arxiv.org/abs/2503.20314
- [20]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.arxiv.org/abs/1706.03762
- [21]Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004.pubmed.ncbi.nlm.nih.gov/15376593
- [22]Jianzhong Wu, Liang Hou, Haotian Yang, Xin Tao, Ye Tian, Pengfei Wan, Di Zhang, and Yunhai Tong. VMoBA: Mixture-of-block attention for video diffusion models. arXiv preprint arXiv:2506.23858, 2025.arxiv.org/abs/2506.23858
- [23]Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, Jianfei Chen, Ion Stoica, Kurt Keutzer, and Song Han. Sparse VideoGen: Accelerating video diffusion transformers with spatial-temporal sparsity. In Proceedings of the International Conference on Machine Learning, 2025.arxiv.org/abs/2502.01776
- [24]Haocheng Xi, Yiming Xie, Hexu Zhao, Yiwen Zhang, Michael Liu, Thomas Creavin, Kurt Keutzer, Xiuyu Li, Zhaoyang Lv, Chenfeng Xu, and Haiwen Feng. Video deltanet: A video-native hybrid attention for livestream video generation. arXiv preprint arXiv:2609.20744, 2026.arxiv.org/abs/2609.20744
- [25]Yifei Xia, Suhan Ling, Fangcheng Fu, Yujie Wang, Huixia Li, Xuefeng Xiao, and Bin Cui. Training-free and adaptive sparse attention for efficient long video generation. arXiv preprint arXiv:2502.21079, 2025.arxiv.org/abs/2502.21079
- [26]Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. XAttention: Block sparse attention with antidiagonal scoring. In Proceedings of the International Conference on Machine Learning, pages 69819–69831, 2025.arxiv.org/abs/2503.16428
- [27]Shuo Yang, Haocheng Xi, Yilong Zhao, Muyang Li, Jintao Zhang, Han Cai, Yujun Lin, Xiuyu Li, Chenfeng Xu, Jianfei Chen, Song Han, Kurt Keutzer, and Ion Stoica. Sparse VideoGen2: Accelerate video generation with sparse attention via semantic-aware permutation. In Advances in Neural Information Processing Systems, 2025.
- [28]Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wandang Zeng. Native sparse attention: Hardware-aligned and natively trainable sparse attention. In Proceedings of the Association for Computational Linguistics, 2025.arxiv.org/abs/2502.11089
- [29]Ted Zadouri, Markus Hoehnbach, Jay Shah, Timmy Liu, Vijay Thakkar, and Tri Dao. FlashAttention-4: Algorithm and kernel pipelining co-design for asymmetric hardware scaling. arXiv preprint arXiv:2603.05451, 2026.arxiv.org/abs/2603.05451
- [30]Jintao Zhang, Haoxu Wang, Kai Jiang, Shuo Yang, Kaiwen Zheng, Haocheng Xi, Ziteng Wang, Hongzhou Zhu, Min Zhao, Ion Stoica, Joseph E. Gonzalez, Jun Zhu, and Jianfei Chen. SLA: Beyond sparsity in diffusion transformers via fine-tunable sparse-linear attention. arXiv preprint arXiv:2509.24006, 2025.arxiv.org/abs/2509.24006
- [31]Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. SpargeAttention: Accurate and training-free sparse attention accelerating any model inference. In Proceedings of the International Conference on Machine Learning, 2025.arxiv.org/abs/2502.18137
- [32]Jintao Zhang, Kai Jiang, Chendong Xiang, Weiqi Feng, Yuezhou Hu, Haocheng Xi, Jianfei Chen, and Jun Zhu. SpargeAttention2: Trainable sparse attention via hybrid top-k+top-p masking and distillation fine-tuning. arXiv preprint arXiv:2602.13515, 2026.arxiv.org/abs/2602.13515
- [33]Jintao Zhang, Haoxu Wang, Kai Jiang, Kaiwen Zheng, Youhe Jiang, Ion Stoica, Jianfei Chen, Jun Zhu, and Joseph E. Gonzalez. SLA2: Sparse-linear attention with learnable routing and quantization-aware training. arXiv preprint arXiv:2602.12675, 2026.arxiv.org/abs/2602.12675
- [34]Peiyuan Zhang, Yongqi Chen, Haofeng Huang, Will Lin, Zhengzhong Liu, Ion Stoica, Eric Xing, and Hao Zhang. VSA: Faster video diffusion with trainable sparse attention. arXiv preprint arXiv:2505.13389, 2025.arxiv.org/abs/2505.13389
- [35]Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, Ion Stoica, Zhengzhong Liu, and Hao Zhang. Fast video generation with Sliding Tile Attention. arXiv preprint arXiv:2502.04507, 2025.arxiv.org/abs/2502.04507
- [36]Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 586–595, 2018.arxiv.org/abs/1801.03924
- [37]Tianchen Zhao, Ke Hong, Xinhao Yang, Xuefeng Xiao, Huixia Li, Feng Ling, Ruiqi Xie, Siqi Chen, Hongyu Zhu, Yichong Zhang, and Yu Wang. PAROAttention: Pattern-aware ReOrdering for efficient sparse and quantized attention in visual generation models. arXiv preprint arXiv:2506.16054, 2025.arxiv.org/abs/2506.16054
- [38]Xuan yi Zhou, Qiyang Mang, Shuo Yang, Haocheng Xi, Jintao Zhang, Huanzhi Mao, Joseph E. Gonzalez, Kurt Keutzer, Ion Stoica, and Alvin Cheung. SVG-EAR: Parameter-free linear compensation for sparse video generation via error-aware routing. arXiv preprint arXiv:2603.08982, 2026.arxiv.org/abs/2603.08982
- [39]Yifan Zhou, Zeqi Xiao, Tianyi Wei, Shuai Yang, and Xingang Pan. Trainable log-linear sparse attention for efficient diffusion transformers. arXiv preprint arXiv:2512.16615, 2025.arxiv.org/abs/2512.16615




