WorksComputer Science
CLiS: Plug-and-Play Attention Acceleration for Video Diffusion on TPUs via Certified Stabilization
Abstract
High-resolution video diffusion relies on attention over long token sequences, making attention efficiency critical to generation latency. Our analysis identifies repeated maximum reductions and accumulator rescaling in online softmax as a bottleneck in long-context TPU attention. We propose CLiS (Certified Loop-invariant Stabilization), a training-free, plug-and-play optimization for video diffusion inference. CLiS fixes per-row stabilizers before the KV loop and certifies numerical safety per head, falling back to online softmax when certification fails. It integrates into existing fused-attention kernels without changing model weights or architecture. On Wan 2.2 and LTX-2 across TPU v6e and TPU7x, CLiS reduces diffusion transformer (DiT) step latency by up to 19.6% relative to tuned online softmax without measurable quality loss. Controlled TPU adaptations of prior GPU stabilization methods show performance or numerical-safety limitations. Preliminary causal LLM experiments also show up to 13.5% higher training throughput in continued pretraining and 7.7% in supervised fine-tuning.
Figure 1. CLiS accelerates long-sequence video diffusion attention without visually altering generation quality. Left: speedup on TPU v6e. Right: frames 0, 20, 40, 60, 80, 100 from a 720p, 7.5s (16fps, 40-step) video generated by WAN 2.2 (Team Wan, [41]).
Introduction
Video Diffusion Transformers (DiTs) repeatedly apply self-attention over spatiotemporal tokens during generation (Peebles & Xie, [34]; Team Wan, [41]; Kong et al., [26]; HaCohen et al., [14]; Brooks et al., [3]; Google DeepMind, [12]). As resolution and duration increase, these sequences become extremely long: a five-second 720p video can require approximately 75K–120K tokens (Team Wan, [41]; Kong et al., [26]). Attention efficiency is therefore a major determinant of generation latency. FLOP counts, however, can obscure an emerging hardware bottleneck. Although matmuls dominate attention FLOPs, softmax can rival them in cycle cost because exponential throughput is far lower (Figure 2) and has scaled more slowly (Shah et al., [39]).
Figure 2. (1) Each unit’s modeled share of cycles per attention score (work divided by peak throughput). At (dotted line, used by Wan 2.2 and LTX-2), exponentiation alone matches the matmuls. On TPU7x, vector-unit max reduction and rescaling push softmax past them. (2) Exposed softmax in the compiled VLIW bundle schedule for Wan 2.2 DiT self-attention (one KV-block iteration) on TPU7x.
Motivation. This matmul–non-matmul throughput imbalance, previously studied on GPUs (Shah et al., [39]; Zadouri et al., [46]), also arises on TPUs. Schedules from the TPU compiler’s low-level optimizer (LLO), which emits the final VLIW instruction bundles (Jouppi et al., [22]), show that exponentiation alone rivals the matmuls in modeled cycle cost, while maximum reduction and accumulator rescaling push the full softmax cost beyond them. TPU architecture adds another constraint: within each KV iteration, online-softmax state competes with score and probability tiles for a limited set of shared vector registers. Because this maximum-and-rescaling chain is re-executed for every KV block, its total cost grows with sequence length, and runtime ablations confirm that it remains on the critical path (§ 3).
Challenges. Softmax subtracts a per-row stabilizer from the logits to keep exponentials in floating-point range. In exact arithmetic, any choice gives the same result. Online softmax uses the largest logit seen so far, so every KV block that raises it forces a rescale of the accumulated sums. A loop-invariant stabilizer, fixed before the KV loop, removes both the per-block maximum reduction and the rescaling. Prior GPU work moves partway toward this: FlashDecoding++ fixes a shared shift but recomputes rows that violate it (Hong et al., [16]), VFA freezes an approximate maximum without a safety guarantee (Sun et al., [40]), and FlashAttention-4 rescales only past a threshold but still computes every block’s maximum (Zadouri et al., [46]). Yet if a fixed stabilizer sits too far below a row’s maximum, exponentials overflow, and if it sits too far above, they underflow to zero (Figure 3). Computing the exact maximum in advance avoids both but costs an extra pass over all attention scores. The challenge is therefore to certify a safe loop-invariant stabilizer at negligible cost.
Figure 3. Approximate numerical limits of for FP16, BF16, and FP32 [24].
Our Method. We introduce CLiS (Certified Loop-invariant Stabilization), a training-free, plug-and-play optimization for existing video diffusion models. It replaces online softmax’s running maximum with a loop-invariant per-row stabilizer whenever numerical safety can be certified. Using inexpensive query and key statistics, CLiS derives an upper bound on each row’s attention logits before the key-block loop. Certified heads skip maximum reduction and accumulator rescaling on every KV block, while uncertified heads fall back to standard online softmax. By changing only attention stabilization, CLiS integrates into existing fused-attention kernels without additional training or changes to model weights or architecture. It preserves standard attention in exact arithmetic.
Results. We evaluate CLiS on Wan 2.2 and LTX-2 using TPU v6e and TPU7x, varying video resolution and duration over attention lengths from 17K to 184K tokens. Relative to tuned online softmax, CLiS reduces attention-kernel latency by up to 24.0% and DiT-step latency by up to 19.6%, without measurable degradation in generation quality. TPU adaptations of prior GPU methods either fail to deliver speedups or produce invalid outputs. A preliminary exploration of causal LLM training also finds up to 13.5% higher training throughput in continued pretraining and 7.7% in supervised fine-tuning (§ 9), suggesting potential applicability beyond diffusion attention.
Contributions.
(1) We identify repeated maximum reduction and accumulator rescaling as a bottleneck in long-context diffusion attention on latest-generation TPUs.
(2) We propose CLiS, a training-free, plug-and-play method that removes repeated stabilization through per-head numerical certification.
(3) We achieve up to 19.6% lower DiT-step latency on Wan 2.2 and LTX-2 without measurable quality loss. Preliminary LLM experiments also show speedups in continued pretraining and supervised fine-tuning.
Preliminaries
FlashAttention and Online Softmax
Given queries, keys, and values , scaled dot-product attention computes
where is the sequence length and is the head dimension. FlashAttention partitions into query blocks and into key-value blocks , avoiding materialization of the matrices and (Dao et al., [5]).
For each query row, online softmax maintains a running maximum , a normalization accumulator , and an output accumulator . Given the logits for key block , it updates
The final attention output is .
Numerically Stable Softmax. Softmax is invariant to a uniform shift:
for any scalar in exact arithmetic. Conventional online softmax chooses the running row maximum as this shift, ensuring that every exponent is non-positive [30]. As shown in Figure 3, in floating-point arithmetic, replacing the running maximum with an arbitrary fixed shift can cause overflow, underflow, or increased numerical error [2].
TPU Architecture and Low-Level Execution
Attention combines matrix multiplications with reductions, exponentials, and elementwise updates. These operations run on different TPU units, so performance depends on how their execution overlaps.
Compute units. A TPU’s main compute core, called a TensorCore, contains specialized units for these operations. As shown in Figure 4, the Matrix Multiply Unit (MXU) computes the two attention matrix products, and . The Vector Processing Unit (VPU) performs elementwise operations, including exponentiation and accumulator rescaling. The Cross-Lane Unit (XLU) supports data movement across vector lanes, which combines with vector operations to implement row-wise reductions. These units can execute concurrently when their data dependencies permit [22, 20].
Figure 4. Simplified TPU7x chiplet architecture and dataflow for attention [11, 20]. Arrows show logical dataflow. Each TPU7x chip contains two chiplets.
Registers and memory. These computations share vector registers (VREGs), which hold intermediate results such as score tiles, probabilities, and online-softmax accumulators. On-chip vector scratchpad memory (VMEM) provides additional storage, while off-chip high-bandwidth memory (HBM) holds the larger input and output tensors. When live intermediate values exceed VREG capacity, the compiler temporarily stores them in VMEM (VREG spills) and reloads them into VREGs when needed (fills) [20]. We refer to these transfers as spill/fill traffic.
Instruction scheduling. Pallas, a JAX kernel language, lowers TPU kernels through Mosaic to LLO, a hardware-specific representation for TPU compilation [19]. The compiler coordinates the functional units using a very long instruction word (VLIW) execution model. It groups instructions into bundles: instructions for different units that can be issued together [22]. A bundle can therefore issue matrix work alongside independent vector work. We perform LLO analysis of the compiled instruction bundles and their generation-specific modeled cycle costs to examine where matrix and vector work overlap, where spill/fill instructions occur, and where dependencies prevent further MXU work from being issued.
Motivation: What Online Softmax Costs on TPUs
Online softmax repeats a row-maximum reduction and accumulator rescaling for every KV block. We first examine this stabilization cost, then distinguish the effects of sequence length and tile selection, and finally identify the numerical constraint on removing it.
Online Stabilization Adds Removable Vector Work
We compare compiled schedules and run latency ablations to isolate the cost of online stabilization. We replace the running maximum with a loop-invariant stabilizer, removing the maximum reduction and accumulator rescaling while retaining exponentiation. Figure 5 shows the removed dependency chain.

Figure 5. Online-stabilization overhead for one compiled KV-block iteration. Wan 2.2 on TPU7x, generating a 5 s, 720p video (81 frames), at identical tile sizes. (a) A loop-invariant stabilizer removes the reduction and rescaling chain. (b–c) Compiled VLIW schedules. Fixed stabilization reduces VPU-only issue bundles and shortens the schedule by 24.3%. (d–e) Scheduled bundles, VPU vmax reductions, VPU-only issue bundles, and VREG spill stores. Counts describe instruction issue, not elapsed unit-active cycles.
We examine WAN 2.2 () [41] generating a 5-second 720p video. At identical tile sizes, we compare one compiled KV-block iteration. The compiled schedule shrinks from 23,185 to 17,556 instruction bundles (24.3%). The fixed stabilizer also cuts spill stores by 48%. Bundles issuing VPU work without an MXU instruction fall from 6,858 to 384 (94%).
In runtime ablations on this workload, removing maximum reduction and rescaling reduces DiT-step latency by 7.5%. Removing all softmax work reduces it by 8.2%. Therefore, stabilization accounts for 91% of the removable softmax overhead in this ablation.
Takeaway 1: Stabilization is a major removable cost. Maximum reduction and accumulator rescaling account for most removable softmax overhead in the measured workload. A fixed per-row stabilizer removes both while retaining exponentiation.
Sequence Length and Tile Size Have Different Effects
To separate the effects of tile size and sequence length on stabilization overhead, we compare online softmax with a loop-invariant stabilizer in WAN 2.2 on TPU7x (Figure 6).
Figure 6. (a) Query-tile sweep at (720p, 81 frames). Stars mark fastest tiles. Spill/fill traffic is reported in bytes per attention score. (b) Sequence-length sweep at fixed tiles (720p). Top: total spill/fill traffic per call. Bottom: spill/fill share of vector loads and stores. Figure 6. Loop-invariant stabilization reduces spill/fill traffic across tile sizes and sequence lengths. One ring attention step of WAN 2.2 on TPU7x (20 heads, , BF16). Times are medians of 20 warm calls. Spill/fill traffic is derived from static LLO counts.
Tile selection. In Pallas, the TPU compiler maps configurable query and key tiles to vector registers and VMEM and schedules work across execution units ([19 b]). We therefore tune tile sizes empirically to balance data reuse, memory demand, and scheduling efficiency.
At a fixed sequence length (Figure 6a), larger query tiles increase spill/fill traffic per KV block, but traffic per score stays nearly constant. A loop-invariant stabilizer roughly halves this per-score traffic across all tiles. Online softmax also exhibits latency peaks at certain tile sizes that persist after removing spill/fill code, indicating a scheduling cost beyond memory traffic. A loop-invariant stabilizer avoids these peaks, providing more consistent latency and a 6.6% speedup at each method’s fastest tile.
Sequence length. At fixed tile sizes (Figure 6b), longer sequences add more query–key tile pairs. Because each pair incurs similar spill/fill traffic, total traffic grows with sequence length while remaining a nearly constant share of vector loads and stores. A loop-invariant stabilizer roughly halves this traffic across all measured lengths.
Takeaway 2: Loop-invariant stabilization reduces spill/fill traffic and tile sensitivity. A loop-invariant stabilizer roughly halves spill/fill traffic per score and reduces latency sensitivity to tile choice, whereas longer sequences increase total traffic by adding query–key tile pairs.
Removing Stabilization Requires Numerical Safety
Removing row-maximum reduction and accumulator rescaling speeds up attention, but can introduce numerical errors. Although softmax is shift-invariant in exact arithmetic, the choice of stabilizer matters in floating point. As shown in Figure 3, a stabilizer too far below the row maximum can cause exponentiation to overflow. One too far above it can cause underflow.
Simple alternatives each have limitations. Omitting the shift risks overflow, while clipping changes the attention probabilities. A shared stabilizer must accommodate different logit ranges across rows and heads. Computing each row’s exact maximum avoids this difficulty but requires an extra pass over the attention scores. We therefore need a per-row stabilizer that is safe, inexpensive to compute, and fixed before the KV loop.
Certified Loop-Invariant Stabilization
Certification is performed independently for each attention head. All statistics below refer to one head, and we suppress the head index unless needed. Processing multiple heads in parallel does not combine their certificates.
Following FlashAttention’s block notation Dao et al., [5], let be a query block and a key–value block. Orange denotes query-indexed variables and blue denotes key–value-indexed variables, while pairwise arrays retain colored indices. We absorb the attention scale and into , so the score block is and softmax uses .
Bounding and centering. Let denote the number of valid key tokens, excluding padding. We compute their mean and subtract it from each key, giving . For each query , this subtracts the same scalar from every logit in its row and preserves attention [49]. In exact arithmetic, the centered logits in each row have mean zero, so their maximum satisfies . Let
By the Cauchy–Schwarz inequality, . For an integer recentering constant , CLiS uses the loop-invariant shift
Using directly as the stabilizer can cause underflow when the bound is loose. We therefore subtract a constant , raising every exponent argument by . This allows positive exponent arguments while keeping them bounded above by . We choose subject to the numerical safety conditions below to avoid an all-zero denominator and prevent overflow.
Numerical certificate. Before using the loop-invariant path, we check that each row retains a nonzero denominator and that exponentiation and accumulation stay within range. Let be a conservative output interval over which produces nonzero, finite outputs. Let be the safe FP32 accumulator magnitude limit expressed in base-2 logarithmic units. We require to bound the computed logits, including rounding error. The supplied margin must guarantee that each computed centered row has maximum at least . Our implementation uses . For head , let denote the bound for query row . We define , taking the maximum only over rows within that head. The value-magnitude bound also uses that head’s values. We seek a nonnegative integer recentering constant satisfying
If a solution exists, we choose the largest feasible integer and use the loop-invariant path. Otherwise, the head uses online softmax. The first condition keeps at least one exponential in each row nonzero, so the denominator remains positive. The second prevents individual exponentials from overflowing. The third reserves enough range to sum over keys and accumulate the value-weighted outputs. For eligible heads, the fixed per-row shift preserves attention in exact arithmetic. Algorithm 1 summarizes this dispatch.

Figure 1 (PDF p. 6). CLiS forward pass for one dense attention head.
Compiler-aware dispatch. We precompute each head’s key mean , norm bound , and eligibility, reducing across sequence shards when needed. The kernel computes and once per query block while is on chip. Each head selects its execution path before the KV loop, keeping eligibility checks outside it.
For Mosaic scheduling, both paths use an online update on any final partial KV block, rescaling existing accumulators as needed to preserve the result in exact arithmetic. Certified heads perform row-maximum reduction only for this block.
Experiments
In this section, we evaluate CLiS on two video diffusion models, Wan 2.2 (Team Wan, [41]) and LTX-2 (HaCohen et al., [15]), across TPU v6e (Google Cloud, [10]) and TPU7x (Jouppi et al., [23]). Our results show that CLiS consistently accelerates attention and diffusion inference over tuned online softmax while preserving numerical accuracy and generation quality, whereas competing methods either provide no speedup or produce invalid outputs.
Experimental Setup
Baselines. We compare CLiS with the following baselines:
Online softmax (Milakov & Gimelshein, [30]; Dao et al., [5]): A tuned, open-sourced Pallas (JAX Team, [18]) implementation of the standard running-maximum algorithm.
FlashDecoding++-style (FD++) (Hong et al., [16]): Uses an empirically calibrated shared shift and recomputes an affected row with online softmax when the supported exponent range is violated.
VFA-style (Sun et al., [40]): Initializes each row’s maximum from approximate key-block representatives, updates it on sink and local blocks, and freezes it for the remaining blocks.
FlashAttention-4-style (FA4) (Zadouri et al., [46]): Keeps the exact per-block row maximum and rescales only when its increase exceeds the threshold. FA4 decides per row but coalesces correction over 32-row warps. Our port compares row-wise and full-tile control and reports both.
The latter three methods include GPU-specific scheduling and dataflow optimizations that do not transfer directly to TPUs. Our style baselines adapt their stabilization mechanisms, including safety checks and recovery behavior, within a common Pallas kernel. See Appendix § 12.
Implementations. Wan 2.2 and LTX-2 use TorchAX (Qi et al., 2025). All workloads use eight TensorCores (TPU v6e-8 or TPU7x-8), sharing classifier-free guidance (CFG) and ring attention as CFG=2×Ring=4 (Ulysses=1). Appendix § 14 varies this layout. For each workload, we tune each variant’s tile sizes under otherwise matched execution settings. Reported latency includes preprocessing, safety checks, and recomputation. We measure attention-kernel and DiT-step latency after warmup using repeated runs.
Experiments. We vary video duration at each fixed resolution to evaluate how CLiS speedup changes with sequence length. For Wan 2.2, we use 41, 81, 121, 161, and 201 frames (2.5, 5, 7.5, 10, and 12.5 s). For LTX-2, we use 81, 161, 241, 321, and 401 frames (3.4, 6.7, 10.0, 13.4, and 16.7 s at 24 fps).
Figure 7. Normalized attention-kernel latency for CLiS and baselines, relative to online softmax (1.0×). Lower is better. All runs use eight TensorCores with CFG=2×Ring=4 (Ulysses=1). Panels (a1)–(d2) use focused axes. Out-of-range bars are clipped and labeled with their measured ratios. Crosses mark corrupted outputs.
Kernel and DiT-Step Performance
Figure 7 compares attention-kernel and DiT-step latency across both models, TPU generations, and sequence lengths. All results are normalized to the corresponding online-softmax control, with lower values indicating lower latency. As shown in Table 8 and Table 9 in Appendix § 15, we also report the corresponding absolute latencies.
| Frames | Tokens | Online | FD++ | VFA | FA4 tile | FA4 row | CLiS |
| Wan 2.2, TPU v6e, 480p | |||||||
| 41 | 17,160 | 4.089 | 4.809 | 6.931† | 7.624 | 4.086 | 3.344 |
| 81 | 32,760 | 14.156 | 18.320 | 12.926† | 21.098 | 14.152 | 11.012 |
| 121 | 48,360 | 30.666 | 37.374 | 26.123† | 47.123 | 30.645 | 24.169 |
| 161 | 63,960 | 52.009 | 61.793 | 43.164† | 64.258 | 52.045 | 40.841 |
| 201 | 79,560 | 80.151 | 97.233 | 64.993 | 97.864 | 80.280 | 61.110 |
| Wan 2.2, TPU v6e, 720p | |||||||
| 41 | 39,600 | 20.986 | 24.855 | 17.963† | 42.107 | 20.783 | 16.819 |
| 81 | 75,600 | 73.002 | 89.220 | 60.533† | 90.772 | 73.034 | 57.629 |
| 121 | 111,600 | 159.764 | 186.821 | 126.678† | 188.396 | 157.149 | 122.610 |
| 161 | 147,600 | 274.548 | 325.357 | 216.361† | 323.929 | 274.832 | 211.768 |
| 201 | 183,600 | 419.815 | 496.323 | 327.618† | 495.442 | 414.598 | 320.727 |
| Wan 2.2, TPU7x, 480p | |||||||
| 41 | 17,160 | 3.231 | 3.552 | 3.004† | 3.446 | 3.214 | 2.754 |
| 81 | 32,760 | 10.834 | 12.145 | 9.172† | 11.207 | 10.827 | 8.472 |
| 121 | 48,360 | 23.413 | 25.905 | 19.695† | 24.245 | 23.356 | 18.336 |
| 161 | 63,960 | 40.277 | 39.525 | 32.358† | 85.607 | 40.324 | 31.262 |
| 201 | 79,560 | 54.483 | 60.511 | 49.217 | 150.986 | 54.604 | 47.605 |
| Wan 2.2, TPU7x, 720p | |||||||
| 41 | 39,600 | 13.863 | 15.399 | 12.782† | 36.532 | 13.864 | 12.302 |
| 81 | 75,600 | 48.894 | 55.156 | 44.813† | 148.243 | 49.270 | 43.712 |
| 121 | 111,600 | 105.769 | 117.437 | 95.412† | 261.834 | 106.638 | 92.399 |
| 161 | 147,600 | 210.893 | 205.882 | 166.704† | 429.516 | 210.815 | 162.913 |
| 201 | 183,600 | 286.316 | 315.458 | 256.481† | 683.688 | 288.964 | 245.522 |
| LTX-2, TPU v6e, 960p | |||||||
| 81 | 17,160 | 3.331 | 4.062 | 5.745 | 6.489 | 3.478 | 2.715 |
| 161 | 32,760 | 11.607 | 14.892 | 10.609† | 17.291 | 11.601 | 8.822 |
| 241 | 48,360 | 24.538 | 29.641 | 20.910† | 37.707 | 24.522 | 19.083 |
| 321 | 63,960 | 41.601 | 48.902 | 34.534† | 51.396 | 41.632 | 32.696 |
| 401 | 79,560 | 64.113 | 76.991 | 51.996† | 78.271 | 64.207 | 48.900 |
| LTX-2, TPU v6e, 1440p | |||||||
| 81 | 39,600 | 16.793 | 19.733 | 14.388† | 33.689 | 16.632 | 13.055 |
| 161 | 75,600 | 58.398 | 70.617 | 48.425† | 72.615 | 58.419 | 46.125 |
| 241 | 111,600 | 127.808 | 148.051 | 101.338† | 150.723 | 125.716 | 98.126 |
| 321 | 147,600 | 219.500 | 257.058 | 173.001† | 258.949 | 219.732 | 169.437 |
| 401 | 183,600 | 335.787 | 393.414 | 262.044† | 396.267 | 331.613 | 256.590 |
| LTX-2, TPU7x, 960p | |||||||
| 81 | 17,160 | 2.609 | 2.878 | 2.446 | 2.797 | 2.615 | 2.260 |
| 161 | 32,760 | 8.625 | 9.648 | 7.337† | 8.918 | 8.622 | 6.801 |
| 241 | 48,360 | 18.721 | 20.827 | 15.823† | 19.484 | 18.730 | 14.654 |
| 321 | 63,960 | 28.120 | 31.074 | 25.712† | 65.615 | 28.179 | 25.024 |
| 401 | 79,560 | 43.124 | 48.246 | 39.284† | 119.369 | 43.598 | 38.160 |
| LTX-2, TPU7x, 1440p | |||||||
| 81 | 39,600 | 11.141 | 13.915 | 10.251† | 13.197 | 11.139 | 10.091 |
| 161 | 75,600 | 39.171 | 44.231 | 35.899† | 118.927 | 39.478 | 35.073 |
| 241 | 111,600 | 84.743 | 94.343 | 76.590† | 210.342 | 85.611 | 73.919 |
| 321 | 147,600 | 168.894 | 165.060 | 134.189† | 345.569 | 168.920 | 130.405 |
| 401 | 183,600 | 229.212 | 252.108 | 205.362† | 546.896 | 231.421 | 196.345 |
Table 8. Absolute attention-kernel latency (ms) for the 40 workloads in Figure 7. Values are means. All runs use eight TensorCores with CFG=2×Ring=4 (Ulysses=1). FD++ and FA4 denote the style baselines. †Corrupted output.
Table 9. Absolute DiT-step latency (s) for the 40 workloads in Figure 7. Values are means. All runs use eight TensorCores with CFG=2×Ring=4 (Ulysses=1). FD++ and FA4 denote the style baselines. †Corrupted output.
| Frames | Tokens | Online | FD++ | VFA | FA4 tile | FA4 row | CLiS |
| Wan 2.2, TPU v6e, 480p | |||||||
| 41 | 17,160 | 0.606 | 0.635 | 0.768† | 0.754 | 0.606 | 0.574 |
| 81 | 32,760 | 1.159 | 1.325 | 1.233† | 1.445 | 1.159 | 1.039 |
| 121 | 48,360 | 2.055 | 2.306 | 1.873† | 2.696 | 2.053 | 1.797 |
| 161 | 63,960 | 3.159 | 3.527 | 2.807† | 3.643 | 3.160 | 2.719 |
| 201 | 79,560 | 4.550 | 5.203 | 3.935 | 5.254 | 4.555 | 3.799 |
| Wan 2.2, TPU v6e, 720p | |||||||
| 41 | 39,600 | 1.541 | 1.685 | 1.432† | 2.359 | 1.533 | 1.376 |
| 81 | 75,600 | 4.211 | 4.834 | 3.721† | 4.921 | 4.213 | 3.607 |
| 121 | 111,600 | 8.340 | 9.369 | 6.985† | 9.494 | 8.235 | 6.890 |
| 161 | 147,600 | 13.506 | 15.480 | 11.216† | 15.506 | 13.516 | 11.042 |
| 201 | 183,600 | 19.855 | 22.811 | 16.154† | 22.900 | 19.648 | 15.986 |
| Wan 2.2, TPU7x, 480p | |||||||
| 41 | 17,160 | 0.305 | 0.312 | 0.296† | 0.418 | 0.304 | 0.284 |
| 81 | 32,760 | 0.735 | 0.746 | 0.671† | 1.052 | 0.735 | 0.642 |
| 121 | 48,360 | 1.376 | 1.406 | 1.231† | 2.791 | 1.378 | 1.175 |
| 161 | 63,960 | 2.199 | 2.221 | 1.872† | 4.024 | 2.201 | 1.839 |
| 201 | 79,560 | 2.898 | 3.203 | 2.691 | 6.745 | 2.903 | 2.629 |
| Wan 2.2, TPU7x, 720p | |||||||
| 41 | 39,600 | 0.936 | 1.014 | 0.893† | 1.838 | 0.936 | 0.872 |
| 81 | 75,600 | 2.633 | 2.943 | 2.472† | 6.597 | 2.646 | 2.430 |
| 121 | 111,600 | 5.203 | 5.806 | 4.797† | 11.448 | 5.237 | 4.686 |
| 161 | 147,600 | 9.710 | 9.790 | 7.999† | 18.488 | 9.715 | 7.809 |
| 201 | 183,600 | 13.048 | 14.531 | 11.867† | 28.796 | 13.160 | 11.474 |
| LTX-2, TPU v6e, 960p | |||||||
| 81 | 17,160 | 0.715 | 0.745 | 0.767 | 0.806 | 0.715 | 0.693 |
| 161 | 32,760 | 1.378 | 1.542 | 1.521† | 1.647 | 1.379 | 1.274 |
| 241 | 48,360 | 2.309 | 2.568 | 2.158† | 2.643 | 2.309 | 2.063 |
| 321 | 63,960 | 3.506 | 3.881 | 3.159† | 4.117 | 3.508 | 3.099 |
| 401 | 79,560 | 5.026 | 5.678 | 4.441† | 5.887 | 5.028 | 4.310 |
| LTX-2, TPU v6e, 1440p | |||||||
| 81 | 39,600 | 1.736 | 1.920 | 1.707† | 1.968 | 1.729 | 1.599 |
| 161 | 75,600 | 4.517 | 5.139 | 4.026† | 5.380 | 4.517 | 3.952 |
| 241 | 111,600 | 8.709 | 9.759 | 7.448† | 10.144 | 8.612 | 7.367 |
| 321 | 147,600 | 13.809 | 15.751 | 11.562† | 15.711 | 13.817 | 11.399 |
| 401 | 183,600 | 20.240 | 23.190 | 16.702† | 23.805 | 20.047 | 17.203 |
| LTX-2, TPU7x, 960p | |||||||
| 81 | 17,160 | 0.330 | 0.330 | 0.327 | 0.393 | 0.331 | 0.311 |
| 161 | 32,760 | 0.761 | 0.760 | 0.711† | 1.023 | 0.762 | 0.687 |
| 241 | 48,360 | 1.433 | 1.427 | 1.351† | 2.282 | 1.434 | 1.250 |
| 321 | 63,960 | 2.042 | 2.171 | 1.937† | 3.863 | 2.048 | 1.925 |
| 401 | 79,560 | 2.986 | 3.213 | 2.813† | 6.292 | 3.002 | 2.758 |
| LTX-2, TPU7x, 1440p | |||||||
| 81 | 39,600 | 0.955 | 1.003 | 0.917† | 1.787 | 0.955 | 0.922 |
| 161 | 75,600 | 2.665 | 2.885 | 2.520† | 6.425 | 2.676 | 2.498 |
| 241 | 111,600 | 5.196 | 5.624 | 4.836† | 10.599 | 5.234 | 4.689 |
| 321 | 147,600 | 9.656 | 9.536 | 8.075† | 17.720 | 9.659 | 7.890 |
| 401 | 183,600 | 12.916 | 14.041 | 11.812† | 27.445 | 13.031 | 11.495 |
Table 9. Absolute DiT-step latency (s) for the 40 workloads in Figure 7. Values are means. All runs use eight TensorCores with CFG=2×Ring=4 (Ulysses=1). FD++ and FA4 denote the style baselines. †Corrupted output.
Attention-kernel latency. CLiS reduces kernel latency by 9.4–24.0% across all measured workloads. The benefit appears on both Wan 2.2 and LTX-2 and on both TPU v6e and TPU7x, showing that it extends beyond a single model or accelerator. The reduction varies across configurations and does not grow monotonically with sequence length.
DiT-step latency. The kernel improvements translate into 3.1–19.6% lower DiT-step latency. The smaller reductions reflect the other computation in each step, which attention optimization does not accelerate. Longer sequences often show larger step-level gains, but the trend varies by model and TPU generation.
Comparison with prior stabilization mechanisms. The FlashDecoding++-style and FlashAttention-4-style baselines provide little benefit or increase latency in most configurations. Row-wise FA4 remains close to online softmax, while full-tile correction incurs larger overhead. VFA can reduce latency but produces corrupted outputs on most Wan 2.2 and LTX-2 runs (9 of 10 workloads each). Latency and output quality must therefore be considered together when comparing stabilization methods.
Certification and Output Quality
Numerical safety. Table 1 shows that CLiS produces valid outputs on all ten tested Wan 2.2 workloads on TPU7x, as do the FlashDecoding++-style and FA4-style baselines. VFA-style fails on nine workloads, highlighting the risk of using an uncertified stabilizer without a recovery mechanism. CLiS preserves attention in exact arithmetic. measured attention output differs from online softmax by in relative , comparable to valid baselines.
| Method | Certification | Attention output | VBench vs. online () | ||||||
| Valid | Safe-cond. rate | Max abs. | Rel. | Imaging | Subject cons. | Temp. flicker | Motion smooth. | Dynamic degree | |
| Online softmax | 10/10 | – | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| VFA-style | 1/10 | 40.1 | – | – | – | – | – | ||
| FD++-style | 10/10 | 91.0 | +0.0011 | −0.0005 | +0.0012 | −0.0001 | −0.1000 | ||
| FA4-style (tile) | 10/10 | – | +0.0032 | +0.0013 | −0.0001 | +0.0010 | 0 | ||
| CLiS | 10/10 | 97.9–98.3 | +0.0110 | +0.0015 | +0.0004 | +0.0002 | 0 |
Table 1. Safety and output quality on Wan 2.2. Certification and attention error cover ten TPU7x workloads (17K–184K tokens). Valid counts workloads without NaN/Inf or all-zero outputs. Safe-condition rates use different units and are not directly comparable. The CLiS rate spans the five 720p workloads, from the same run as Table 2 and Figure 8. Errors and VBench deltas are relative to online softmax. VBench uses 110 prompts with matched seeds at 720p×81f. VFA videos have all-zero frames and are unscored.
Figure 7 (PDF p. 8). Normalized DiT-step latency (continued). Panels (e1)–(h2) follow the same order and encoding as panels (a1)–(d2).
Generation quality. We evaluate the generated videos using selected VBench [17] metrics with matched prompts and seeds. CLiS shows small differences from online softmax across the reported dimensions, supporting that its attention-level numerical differences do not materially degrade generation quality in this evaluation. Full VBench scores, paired differences, and visual comparisons appear in Appendix § 12.§ 3 (Table 5 and Table 6, and Figure 11).
| Dimension | Online softmax | VFA-style | FD++-style | FA4-style (tile) | FA4-style (row) | CLiS |
| Imaging quality | 0.6658 | –{sup}† | 0.6668 | 0.6690 | 0.6672 | 0.6768 |
| Subject consistency | 0.9237 | –{sup}† | 0.9231 | 0.9250 | 0.9224 | 0.9252 |
| Temporal flickering | 0.9693 | –{sup}† | 0.9705 | 0.9692 | 0.9710 | 0.9697 |
| Motion smoothness | 0.9454 | –{sup}† | 0.9453 | 0.9465 | 0.9452 | 0.9456 |
| Dynamic degree | 1.0000 | –{sup}† | 0.9000 | 1.0000 | 0.9000 | 1.0000 |
| Aesthetic quality | 0.6226 | –{sup}† | 0.6240 | 0.6223 | 0.6216 | 0.6287 |
| Background consistency | 0.9178 | –{sup}† | 0.9275 | 0.9264 | 0.9273 | 0.9206 |
| Overall consistency | 0.2644 | –{sup}† | 0.2656 | 0.2629 | 0.2662 | 0.2651 |
| Appearance style | 0.2197 | –{sup}† | 0.2201 | 0.2193 | 0.2192 | 0.2180 |
| Temporal style | 0.2397 | –{sup}† | 0.2395 | 0.2376 | 0.2406 | 0.2400 |
| Object class | 0.7375 | –{sup}† | 0.7188 | 0.7312 | 0.7875 | 0.7688 |
| Multiple objects | 0.6500 | –{sup}† | 0.6312 | 0.6687 | 0.6375 | 0.7063 |
| Spatial relationship | 0.8370 | –{sup}† | 0.8442 | 0.8568 | 0.8310 | 0.8356 |
| Human action | 0.6000 | –{sup}† | 0.6000 | 0.7000 | 0.8000 | 0.6000 |
| Scene | 0.2875 | –{sup}† | 0.2875 | 0.2188 | 0.1938 | 0.3000 |
| Color | 0.9609 | –{sup}† | 0.8542 | 0.9554 | 0.8264 | 0.9844 |
Table 5. Table 5. Absolute VBench scores for Wan 2.2 T2V at 720p81f on 110 prompts, using VBench’s per-dimension aggregates (higher is better). The first five dimensions appear in Table 1. VFA-style, FD++-style, and FA4-style (row) use TPU v6e. Other methods use TPU7x. †All 110 VFA-style videos have all-zero frames and are unscored.
| Dimension | FD++-style | FA4-style (tile) | FA4-style (row) | CLiS | |
| Imaging quality | +0.0011 [−0.0158, +0.0198] | +0.0032 [−0.0061, +0.0149] | +0.0014 [−0.0124, +0.0181] | +0.0110 [+0.0015, +0.0239] | 10 |
| Subject consistency | −0.0005 [−0.0106, +0.0096] | +0.0013 [−0.0029, +0.0061] | −0.0013 [−0.0082, +0.0068] | +0.0015 [−0.0043, +0.0076] | 10 |
| Temporal flickering | +0.0012 [−0.0005, +0.0035] | −0.0001 [−0.0004, +0.0002] | +0.0017 [−0.0001, +0.0041] | +0.0004 [+0.0000, +0.0010] | 10 |
| Motion smoothness | −0.0001 [−0.0028, +0.0026] | +0.0010 [−0.0006, +0.0032] | −0.0003 [−0.0031, +0.0032] | +0.0002 [−0.0010, +0.0016] | 10 |
| Dynamic degree | −0.1000 [−0.3000, +0.0000] | +0.0000 [+0.0000, +0.0000] | −0.1000 [−0.3000, +0.0000] | +0.0000 [+0.0000, +0.0000] | 10 |
Table 6. Paired VBench differences from online softmax with 95% percentile-bootstrap confidence intervals over prompts. counts prompts scored for both methods.

Figure 11. Visual examples of video generation on Wan 2.2 (720p×81f, 40 denoising steps, with frames 0, 16, 32, 48, 64, 80 shown) across VBench prompts comparing tuned online softmax and CLiS.
| Outcome | Max logit | Sites |
| No effect | ≤ 108 | 60 |
| Graceful flush | 108–122 | 1 |
| NaN | > 122 | 23 |
Table 2b. Had the 84 fallback sites run fixed-.
Figure 8a. Fallback rate across attention sequence lengths.
Figure 8b. Fallback rate across denoising steps.
Fallback Behavior
Fallback is rare. As shown in Table 2a, 98.22% of head evaluations use loop-invariant stabilization and 1.78% fall back to online softmax in the 40-step Wan 2.2 run at 720p with 81 frames. As shown in Figure 8, fallback remains near 2% across sequence lengths and denoising steps.
| Expert | Eligible ↑ | Fallback ↓ | Sites | Max-reduce ↓ | Score rows ↓ |
| High-noise | 98.54% | 1.46% | 36 / 1600 | 6.65% | 3.90% |
| Low-noise | 98.07% | 1.93% | 48 / 1600 | 7.09% | 4.36% |
| Total | 98.22% | 1.78% | 84 / 3200 | 6.95% | 4.21% |
Table 2. Certified-head eligibility and the cost of the fallback path. Wan 2.2 T2V, 1280 × 720, 81 frames, 75,600 tokens. (a) Eligibility and maximum-reduction work removed.
Figure 8. Runtime fallback rate of CLiS on Wan 2.2 (TPU7x) as a function of sequence length (left) and denoising step (right).
Fallback prevents numerical failures. As shown in Table 2b, replaying the 84 rejected head sites without fallback would produce NaNs at 23 sites and flush a ring shard’s weights to zero at one site. The remaining 60 sites would be numerically harmless in this replay. The certificate is therefore conservative, but it catches cases that require fallback for numerical safety.
Most maximum reductions are eliminated. As shown in Table 2a, the hybrid implementation removes 93.05% of reduce_max operations and 95.79% of score-row reductions relative to online softmax. Therefore, the small fallback fraction still allows most of the repeated maximum-reduction work to be removed.
Preliminary Exploration of Causal LLM Training
To explore applicability beyond diffusion attention, we conduct preliminary experiments with causal Splash and Ring Attention for autoregressive LLM training on TPU7x. The forward kernel uses loop-invariant stabilization for certified heads and stores a compact log-sum-exp table for the unchanged backward pass.
We evaluate four model families across 8K scratch pretraining for 1,500 steps on C4 ([38]), ClimbMix ([7]), and FineWeb-Edu ([35]); 8K supervised fine-tuning on UltraChat ([8]); 128K–256K continual pretraining with Ring Attention on PG-19 ([37]); and 512-chip FP8 MLA training of DeepSeek-V3-671B ([6]).
Training throughput. Figure 9 and Table 3 show that CLiS accelerates end-to-end training steps by 6.7% to 8.9% at 8K context on Qwen3-8B ([43]) and Llama-3.1-8B ([13]). At 128K and 256K context, where Ring Attention accounts for a larger share of step time, CLiS reduces forward attention kernel latency by 33.4% to 43.6% and improves end-to-end step throughput by 11.6% to 19.0%. On 512-chip DeepSeek-V3-671B, CLiS reduces forward Splash kernel time by 22.1% and cuts wall-clock time-to-train to the MLPPerf validation target ([31]) by 6.2%. In Gemma-3-12B ([9]), five of every six layers use 1,024-token sliding windows, so applying CLiS to the global 8K layers yields a modest +0.6% step speedup.
| Model (Dataset) | Task | Seq. | Fwd. Attn. | Train Speedup | Loss (Base / CLiS) | Loss | Cert. |
| Qwen3-8B (C4) | Pretrain | 8K | −37.9% | +7.8% | 3.182 / 3.185 | +0.10% | 100% |
| Qwen3-8B (ClimbMix) | Pretrain | 8K | −37.9% | +8.9% | 2.678 / 2.673 | −0.18% | 100% |
| Qwen3-8B (UltraChat) | SFT | 8K | −36.2% | +7.7% | 0.807 / 0.806 | −0.03% | 95.8% |
| Qwen3-8B (PG-19) | Continual PT | 256K | −35.6% | +12.4% | 2.182 / 2.182 | −0.02% | 95.5% |
| Llama-3.1-8B (UltraChat) | SFT | 8K | −37.8% | +6.7% | 0.752 / 0.752 | −0.07% | 100% |
| Llama-3.1-8B (PG-19) | Continual PT | 128K | −43.3% | +11.6% | 1.992 / 1.992 | −0.03% | 99.8% |
| Llama-3.1-8B (PG-19) | Continual PT | 256K | −43.6% | +13.5% | 2.013 / 2.013 | −0.01% | 99.8% |
| Gemma-3-12B (C4) | Pretrain | 8K | −9.6% | +0.6% | 3.423 / 3.425 | +0.06% | 100% |
| DSv3-671B (MLPerf) | Continual PT (loss ) | 4K | −22.1% | +6.2% (TTT) | 3.589 / 3.593 | +0.11% | 99.9% |
| DSv3-12L (Ring MLA) | Pretrain | 128K | −33.4% | +19.0% | 10.70 / 10.70 | 0.00% | 100% |
Table 3. Representative causal LLM pretraining, continual pretraining, and SFT results on TPU7x (64 chips, or 512 chips for DSv3-671B). Forward attention kernel latency change, training speedup (step throughput or wall-clock time-to-train, TTT), and final-50 loss are paired against online softmax.
Figure 9. End-to-end training speedup of CLiS over online softmax across sequence lengths from 4K to 256K tokens on TPU7x.
Loss convergence and certificate behavior. As shown in Figure 10, final-50 training loss under CLiS remains within −0.18% to +0.11% of online softmax across all certified runs. This variation is smaller than the to loss shift caused by changing the online-softmax KV reduction block size from 512 to 256. Models with bounded query-key norms, either from architectural QK-Norm (Qwen3-8B and Gemma-3-12B) or from pretrained checkpoints (Llama-3.1-8B and DeepSeek-V3), maintain 95.5% to 100% head certification throughout training. In contrast, training Llama-3.1-8B from random initialization Table 3. Representative causal LLM pretraining, continual pretraining, and SFT results on TPU7x (64 chips, or 512 chips for DSv3-671B). Forward attention kernel latency change, training speedup (step throughput or wall-clock time-to-train, TTT), and final-50 loss are paired against online softmax.
Figure 10. Relative final-50 training loss difference of CLiS against online softmax, compared with hardware tile-size reduction noise.
without QK-Norm allows unconstrained logit growth past the FP32 exponent budget by step 40, causing the certificate to fall back to online softmax and prevent NaNs.
Related Work
Exact attention kernels. FlashAttention and FlashAttention-2 optimize IO and work partitioning [5, 4]. FlashAttention-3 and FlashAttention-4 optimize matrix and softmax schedules for Hopper and Blackwell [39, 46]. These kernels retain blockwise maximum reductions. Ragged Paged Attention uses Pallas for variable-length TPU LLM serving [21].
Reducing softmax overhead. Online softmax retains a running maximum [30]. FlashDecoding++ uses calibrated shared shifts with recovery [16], while VFA predicts and freezes maxima [40]. FlashAttention-4 skips small rescales, while FLASH-D uses a normalization recurrence [46, 1]. TurboAttention quantizes and approximates exponentials [25]. CLiS certifies fixed per-row shifts, removing loop-level maximum reductions and rescaling with online-softmax fallback.
Video diffusion attention. SageAttention uses low-bit quantization, while SparseAttention adds sparsity [49 a;b;d]. BLASST skips negligible blocks [45]. Sliding Tile Attention, Sparse VideoGen, and Radial Attention exploit spatiotemporal sparsity [51] (Xi et al. 2025; Li et al. 2025). Sparse VideoGen2 and HyperVAttention use clustering [44, 27]. CLiS complements these methods while preserving dense attention in exact arithmetic.
Conclusion
We presented CLiS, a training-free, plug-and-play method that removes repeated online-softmax stabilization work in TPU attention. It uses certified loop-invariant stabilizers when safe and otherwise falls back to online softmax. CLiS integrates into existing attention kernels without changing model weights or architecture. On Wan 2.2 and LTX-2 across TPU v6e and TPU7x, CLiS reduces attention-kernel latency by up to 24.0% and DiT-step latency by up to 19.6% relative to tuned online softmax, without measurable generation-quality loss. Preliminary causal LLM experiments also show up to 13.5% higher end-to-end training throughput in continued pretraining and 7.7% in supervised fine-tuning.
| Method | Retained stabilizer | Row max | Rescale rule | Controlled TPU adaptation |
| Online softmax | Exact running maximum | Every | Every | Tuned exact reference in the common kernel |
| FD++-style | Calibrated shared shift | No* | No* | Range check and exact row recomputation without flat-GEMM or CUDA scheduling |
| VFA-style | Predicted maximum, then frozen | Sink/local only | Sink/local only | Locality mapped to flattened video tokens, including representative construction |
| FA4-style | Running maximum with thresholded updates | Every | Row-wise and full-tile control without Blackwell pipelining or exp emulation | |
| CLiS | Certified fixed shift | No† | No† | Per-head certification with unchanged online fallback |
Table 4. Controlled comparison of the stabilization mechanisms and their TPU adaptations. “Block” denotes one streamed KV block. CLiS still computes a row maximum and rescales on the final partial KV block, if one exists (Algorithm 1, lines 22–28). *FD++ fast path only, with full-row online recomputation after a range violation. †Certified heads only. Rejected heads use online softmax.
Baseline Selection and TPU Adaptations
Our baselines are controlled ports of softmax-side mechanisms rather than end-to-end reproductions of GPU systems. Every method uses the same Pallas attention kernel, tiling, sharding, masking, precision, and ring schedule ([29]). Only the stabilization rule and its required checks or preprocessing change. This design attributes differences to the mechanism under study rather than to a different matmul pipeline, tile, or communication schedule.
Why These Baselines
Tuned online softmax is the primary reference because it is exact, robust, and differs from CLiS only in its stabilization logic. FlashDecoding++, VFA, and FlashAttention-4 provide three distinct ways to reduce that logic: use a calibrated shift with recovery, predict a maximum and later freeze it, or keep the exact running maximum but skip small rescalings, respectively ([16, 40, 46]). These baselines test whether a heuristic plus recovery, approximate maximum prediction, or conditional rescaling is sufficient without certification. For each style baseline, we preserve the published softmax decision and its safety or recovery behavior, but replace the architecture-specific kernel around it with the common TPU implementation.
Implementation Details
Tuned online softmax. For every KV block, the control computes the block row maximum, updates the running maximum, and rescales the normalizer and output accumulator. It uses the same tiles, layouts, ring communication, BF16 inputs, and FP32 accumulators as CLiS. It is therefore the appropriate denominator for both latency and numerical comparisons.
FlashDecoding++-style unified shift. FlashDecoding++ assigns all independently processed score partitions a unified shift , chosen from empirical score ranges, and falls back to the synchronized online-softmax computation when the supported exponent interval is violated. Our port retains this shared-shift fast path, the runtime range check, and exact recomputation, while excluding the paper’s asynchronous CUDA schedule and flat-GEMM optimization. We calibrate on prompts and seeds disjoint from evaluation. The fast path accumulates without a running maximum or per-block rescaling. On a violation, the complete affected row is recomputed with online softmax, and that cost is included in latency. This baseline therefore tests whether empirical calibration plus recovery can replace a proof of safety.
VFA-style predicted maximum. VFA initializes each row’s running maximum from signed absolute-maximum representatives of the key blocks, processes the sink and local blocks with ordinary online updates, and freezes the maximum for the remaining blocks. Our port retains all three steps. We map “local” to the key block covering the same flattened spatiotemporal range as the video query block, and include representative construction in latency. We record any amount by which the frozen maximum is exceeded but do not repair the primary result: VFA does not specify a recovery path, and adding one would define a different method. The baseline therefore measures both the performance and the safety consequences of transferring its approximate initialization and freezing rule to long video sequences.
FA4-style conditional rescaling. FlashAttention-4 retains the exact row-maximum reduction for every block. For each query row, let be the increase from the retained reference to the candidate maximum for block . The reference and accumulators are rescaled only when , with $\tau = 8 in base-2 exponent units. Otherwise, the row continues to accumulate under the previous reference. The published rule is therefore row-wise. Its Blackwell kernel coalesces only the execution of correction: one thread owns each row, and a 32-thread warp runs the correction path when any of its rows needs it, using a unit scale for the other rows. This is neither an independently branched implementation nor a single decision for the full 128-row query tile.
TPUs have no direct analogue of this CUDA warp-level control. We therefore evaluate two Pallas mappings: a row-wise variant that follows the published threshold semantics, and a tile-coalesced variant that promotes an any-row trigger to the full query tile. We report both variants separately in the performance and VBench comparisons, labeling their control granularity as row or tile. Both mappings retain the per-block row-maximum reduction and isolate conditional rescaling from FlashAttention-4’s tensor-memory pipeline, warp specialization, and polynomial exponential approximation.
Uncertified control. Replaying the rejected heads with an uncertified fixed shift shows the failures the certificate prevents (Table 2b).
Safety and Numerical Measurements
Safe-condition rates. The rates in Table 1 use each method’s native decision unit and are therefore not directly comparable. For CLiS, the rate is the fraction of (call, head) pairs that pass certification in the 40-step 720p run behind Table 2 and Figure 8, over 25,600 calls at each of the five 720p sequence lengths (39.6K–184K tokens). For FD++, the rate is the fraction of ring steps that avoid recomputation. At 720p81f, 691 of 7,680 steps trigger recomputation. For VFA, we sample one attention call per denoising step and report the fraction of (row, head) pairs whose frozen covers the true row maximum. This coverage ranges from 23.8–54.6% over 40 steps (3.2 sampled pairs) as the maximum logit grows from 7.1 to 70.7 in base-2 units. FA4 is unconditionally correct and requires no certificate.
Attention-output error. We compare each method with online softmax on identical Q/K/V captured from a Wan 2.2 720p81f denoising step (one ring shard, (1, 20, 37800, 128) BF16), using each method’s end-to-end tile. The online kernel itself differs from an exact FP32 reference by 6.09 in relative at this shape. The common maximum error of 3.1 is one BF16 ULP at the output magnitude and therefore does not discriminate among the non-online variants. Table 1 summarizes the full-tile-control FA4-style TPU adaptation, labeled FA4-style (tile). Its row-wise counterpart has relative error 2.30 and maximum absolute error 6.3 . Both variants are reported separately in the performance comparisons (Figure 7 and Table 8 and Table 9) and VBench results (Table 5 and Table 6).
VBench. We generate one Wan 2.2 T2V video per prompt of a 110-prompt stratified VBench subset (720p81f, 40 UniPC [52] steps, the same random seed, flow shift 3.0, BF16 with FP8 attention and linears disabled) for each method, and score all 16 VBench dimensions with the upstream VBench evaluator. Each dimension is scored on the prompts tagged with it, 10 per dimension. Color additionally drops videos in which the detector finds no target object. Table 5 reports VBench’s aggregate scores, Table 6 the paired deltas behind Table 1, and Figure 11 provides side-by-side visual comparisons across representative VBench prompts. Online softmax, CLiS, and FA4-style (tile) videos were generated on TPU7x, and FD++-style, FA4-style (row), and VFA-style videos on TPU v6e. VBench scores each video independently, and generating the same method on the other TPU changes rounding, not the method. Because online softmax, CLiS, FD++-style, and FA4-style all compute the same attention in exact arithmetic, their deltas measure how far BF16 rounding, amplified over the 40-step trajectory, moves the sampled video. Differences of this size between two exact methods set the noise floor against which Table 1 should be read. Every VFA-style video decodes to all-zero frames and is therefore not scored.
Why GPU Kernels Do Not Transfer Directly to TPUs
The limitation is not that the GPU algorithms are invalid. Their complete implementations are co-designed with a different execution model, so neither their source code nor their performance assumptions carry over unchanged.
Different execution substrate. GPU kernels are organized around CUDA thread blocks, warps, shared memory, and architecture-specific asynchronous tensor-core pipelines [32, 33]. TPU kernels are written in Pallas, lowered through Mosaic/XLA, and compiled into VLLIW LLO bundles that coordinate the MXU, XLU, and VPU through a shared vector-register file [22, 19]. GPU instructions, synchronization, and memory primitives therefore have no direct TPU implementation.
Different dataflow and control costs. FlashDecoding++ couples its shift heuristic with GPU-specific flat-GEMM and asynchronous execution. FlashAttention-4 relies on a Blackwell-oriented pipeline and memory hierarchy. VFA’s sink/local block reordering can conflict with the arrival order of ring-sharded KV blocks on TPU, requiring prefetching or buffering. Moreover, a row-wise branch that is cheap on a GPU may compile to masked work or execute both paths on a TPU, so whether it removes work must be verified in LLO.
Different remaining critical paths. After adaptation, FA4-style execution still performs a row-maximum reduction on every block, preserving the XLU-to-vector dependency that dominates the TPU stabilization chain. FD++-style execution adds range checks and potentially expensive recomputation. VFA removes more work, but its approxi-
mate shift can under-estimate long-video logits and leave the safe exponent range. As a result, the adaptations may be correct yet slower, or fast yet unsafe. Our controlled ports retain each paper’s softmax-side idea on a common TPU kernel, allowing us to measure this transfer gap without claiming to reproduce the original GPU systems.
Parallelism and Attention-Kernel Ablation
To verify that CLiS provides consistent speedups across distinct attention kernel implementations and context-parallelism decompositions rather than benefiting only a single sharding point, Table 7 evaluates five 8-TensorCore configurations on TPU7x-8 (four chips) for Wan 2.2 at 720p×81f (, , ): data-parallel execution without context parallelism (CP=1, DP=8) using standalone Splash attention, pure Ulysses head-sharded context parallelism (Ulysses=8, Ring=1) combining Splash attention with all-to-all collectives, two hybrid 2D context-parallel layouts (Ulysses=4, Ring=2 and Ulysses=2, Ring=4) combining ring attention with all-to-all collectives, and pure sequence-sharded ring attention (Ulysses=1, Ring=8). The main evaluation (Figure 7 and Table 8–Table 9) uses the same eight TensorCores as two classifier-free-guidance groups of four-way ring attention (CFG=2, Ring=4, Ulysses=1). To expose more Ulysses–Ring factorizations, the context-parallel settings here instead disable CFG and use eight-way context parallelism (CFG=1, Ulysses×Ring= 8). In this layout, each TensorCore holds half the attention work, and each DiT step runs one guidance pass instead of two, so absolute latencies are lower than the 720p×81f entries in Table 8–Table 9 and are not directly comparable. For every setting, we sweep candidate tile shapes independently for both online softmax and CLiS and report each method at its fastest tile. Across all five configurations, CLiS consistently reduces attention-kernel latency by 6.6–11.3% (1.07–1.13×) and full 40-block DiT denoising-step latency by 2.9–7.5% (1.03–1.08×), with pure ring attention (Ulysses=1, Ring=8) achieving the lowest overall kernel (24.66 ms) and DiT-step (1.308 s/step) latency.
| Parallelism | CP=1, DP=8 | Ulysses=8, Ring=1 | Ulysses=4, Ring=2 | Ulysses=2, Ring=4 | Ulysses=1, Ring=8 |
| Kernel family | Splash | Splash + A2A | 2D Ring | 2D Ring | Pure Ring |
| Local shard (, ) | (40, 75,600) | (5, 75,600) | (10, 37,800) | (20, 18,900) | (40, 9,450) |
| Attention Kernel Microbenchmark (ms) | |||||
| Online softmax | 185.85 | 31.62 | 32.95 | 28.26 | 27.67 |
| CLiS | 173.32 | 29.54 | 29.74 | 25.06 | 24.66 |
| Latency change | −6.7% | −6.6% | −9.8% | −11.3% | −10.9% |
| DiT Denoising Step Latency (s/step) | |||||
| Online softmax | 10.041 | 1.519 | 1.595 | 1.430 | 1.415 |
| CLiS | 9.544 | 1.474 | 1.504 | 1.351 | 1.308 |
| Latency change | −5.0% | −2.9% | −5.7% | −5.6% | −7.5% |
Table 7. Parallelism and attention-kernel ablation on TPU7x-8 (Wan 2.2, 720p×81f, , , ). Online softmax and CLiS use independently tuned tiles for each setting. All settings use the same eight TensorCores (four chips). The four context-parallel columns disable CFG and use eight-way context parallelism (CFG=1, Ulysses×Ring= 8), whereas Figure 7 and Table 8–Table 9 use CFG=2×Ring=4. Absolute latencies are therefore not comparable across them.
Absolute Latency Measurements
As shown in Table 8 and Table 9, we report absolute attention-kernel and DiT-step latencies for all workloads in Figure 7. The tables use the same recorded means and online-softmax normalization controls as the figure.
References
- [1]Alexandridis, K., Tjotipoulos, V., and Dimitrakopoulos, G. FLASH-Attention with hidden softmax division. In 2025 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED), pp. 1–7. IEEE, 2025. doi: 10.1109/ISLPED65674.2025.11261805. URL https://arxiv.org/abs/2505.14201.
- [2]Blanchard, P., Higham, D. J., and Higham, N. J. Accurately computing the log-sum-exp and softmax functions. IMA Journal of Numerical Analysis, 41(4):2311–2330, 2021. doi: 10.1093/imanum/draa038. URL https://academic.oup.com/imanum/article/41/4/2311/5893596.DOI
- [3]Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., Ng, C., Wang, R., and Ramesh, A. Video generation models as world simulators. Technical report, 2024. URL https://openai.com/index/video-generation-models-as-world-simulators/. Technical report.
- [4]Dao, T. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/989c2b503b20c31dac6b24bbcf263e3d4a7–Abstract-Conference.html.arxiv.org/abs/2307.08691
- [5]Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/67d57c32e20f0da7a302cb81d36e40d5–Abstract-Conference.html.arxiv.org/abs/2205.14135
- [6]DeepSeek-AI, Liu, A., Feng, B., et al. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024. URL https://arxiv.org/abs/2412.19437.
- [7]Diao, S., Yang, Y., Fu, Y., Dong, X., Su, D., Kliegl, M., Chen, Z., Belcak, P., Suhara, Y., Yin, H., Patwary, M., Lin, Y., Kautz, J., and Molchanov, P. Nemotron-CLIMB: Clustering-based iterative data mixture bootstrapping for language model pre-training. arXiv preprint arXiv:2504.13161, 2025. URL https://arxiv.org/abs/2504.13161.
- [8]Ding, N., Chen, Y., Xu, B., Qin, Y., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3029–3051. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.183. URL https://aclanthology.org/2023.emnlp-main.183/.DOI
- [9]Gemma Team, Kamath, A., Ferret, J., et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. URL https://arxiv.org/abs/2503.19786.
- [10]Google Cloud. TPU v6e. Google Cloud documentation, 2024. URL https://docs.cloud.google.com/tpu/docs/v6e. Accessed October 29, 2026.
- [11]Google Cloud. TPU7x (Ironwood). Google Cloud documentation, 2025. URL https://docs.cloud.google.com/tpu/docs/tpu7x. Accessed October 29, 2026.
- [12]Google DeepMind. Veo: A text-to-video generation system. Technical report, Google DeepMind, 2025. URL https://storage.googleapis.com/deepmind-media/veo/Veo-3-Tech-Report.pdf. Veo 3 technical report.
- [13]Grattafiori, A., Dubey, A., Jauhri, A., et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. URL https://arxiv.org/abs/2407.21783.
- [14]HaCohen, Y., Chiprut, N., Brazowski, B., Shalem, D., Moshe, D., Richardson, E., Levin, E., Shiran, G., Zabari, N., Gordon, O., et al. LTX-Video: Realtime video latent diffusion. arXiv preprint arXiv:2501.00103, 2024. URL https://arxiv.org/abs/2501.00103.
- [15]HaCohen, Y., Brazowski, B., Chiprut, N., Bitterman, Y., Kvochko, A., Berkowitz, A., Shalem, D., Lifschitz, D., Moshe, D., Porat, E., Richardson, E., Shiran, G., Chachy, I., Chetboun, J., Finkelson, M., Kupchik, M., Zabari, N., Guetta, N., Kotler, N., Bibi, O., Gordon, O., Panet, P., Benita, R., Armon, S., Kulikov, V., Inger, Y., Shiftan, Y., Melumian, Z., and Farbman, Z. LTX-2: Efficient joint audio-visual foundation model. arXiv preprint arXiv:2601.03233, 2026. URL https://arxiv.org/abs/2601.03233.
- [16]Hong, K., Dai, G., Xu, J., Mao, Q., Li, X., Liu, J., Chen, K., Dong, Y., and Wang, Y. FlashDecoding++: Faster large language model inference with asynchronization, flat GEMM optimization, and heuristics. In Proceedings of Machine Learning and Systems, volume 6, pp. 148–161, 2024. URL https://proceedings.mlsys.org/paper_files/paper/2024/hash/5321b1dabcd2be188d796c21b733e8c7–Abstract-Conference.html.
- [17]Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., Wang, Y., Chen, X., Wang, L., Lin, D., Qiao, Y., and Liu, Z. VBench: Comprehensive benchmark suite for video generative models. In* Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807–21818, 2024. URL https://openaccess.thecvf.com/content/CVPR2024/html/Huang_VBench_Comprehensive_Benchmark_Suite_for_Video_Generative_Models_CVPR_2024_paper.html.
- [18]JAX Team. Pallas: A JAX kernel language. JAX documentation, 2023. URL https://docs.jax.dev/en/latest/pallas/index.html. Accessed October 29, 2026.
- [19]JAX Team. Pallas design. JAX documentation, 2026a. URL https://docs.jax.dev/en/latest/pallas/design/design.html. Accessed October 29, 2026.
- [20]JAX Team. Writing TPU kernels with Pallas. JAX documentation, 2026b. URL https://docs.jax.dev/en/latest/pallas/tpu/details.html. Accessed October 29, 2026.
- [21]Jiang, J., Chen, Y., Hechtman, B. A., Zhang, F., and Mu, Y. Ragged paged attention: A high-performance and flexible LLM inference kernel for TPU. arXiv preprint arXiv:2604.15464, 2026. URL https://arxiv.org/abs/2604.15464.
- [22]Jouppi, N. P., Yoon, D. H., Ashcraft, M., Gottscho, M., Jablin, T. B., Kurian, G., Laudon, J., Li, S., Ma, P., Ma, X., et al. Ten lessons from three generations shaped google’s tpuv4i: Industrial product. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), pp. 1–14. IEEE, 2021. doi: 10.1109/ISCA52012.2021.00010. URL https://doi.org/10.1109/ISCA52012.2021.00010.
- [23]Jouppi, N. P., Lakshmanamurthy, S., Young, C., and Patterson, D. Google’s training supercomputers from TPUv2 to Ironwood: Architectural stability, scale, resilience, power efficiency, and sustainability across five generations. IEEE Micro, 46(4):27–37, 2026. doi: 10.1109/MM.2026.3699647. URL https://arxiv.org/abs/2606.15870.
- [24]Kalamkar, D., Mudigere, D., Mellemputi, N., et al. A study of BFLOAT16 for deep learning training. arXiv preprint arXiv:1905.12322, 2019. URL https://arxiv.org/abs/1905.12322.
- [25]Kang, H., Bharadwaj, S., Hensman, J., Krishna, T., Rühle, V., and Rajmohan, S. TurboAttention: Efficient attention approximation for high throughputs LLM. In Proceedings of Machine Learning and Systems, volume 7, 2025. URL https://proceedings.mlsys.org/paper_files/paper/2025/hash/f4f55846501f3336f293fd8b6de10770–Abstract–Conference.html.arxiv.org/abs/2412.08585
- [26]Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., Xiong, J., Li, X., Xu, W., Zhang, J., et al. Hunyuan-Video: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024. URL https://arxiv.org/abs/2412.03603.
- [27]Lee, D., Zandieh, A., Mirrokni, V., Kim, J., and Han, I. HyperVAttention: Efficient sparse attention with spatio-temporal clustering for video diffusion. arXiv preprint arXiv:2607.03012, 2026. URL https://arxiv.org/abs/2607.03012.
- [29]Liu, H., Zaharia, M., and Abbeel, P. RingAttention with blockwise transformers for near-infinite context. In International Conference on Learning Representations (ICLR), 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/1119587863e78451f080da2a768c4935–Abstract–Conference.html.arxiv.org/abs/2310.01889
- [30]Milakov, M. and Gimelshein, N. Online normalizer calculation for softmax. arXiv preprint arXiv:1805.02867, 2018. URL https://arxiv.org/abs/1805.02867.
- [31]MLCommons. MLPerf training benchmarks. Benchmark documentation, 2026. URL https://mlcommons.org/benchmarks/training/. Accessed October 29, 2026.
- [32]NVIDIA Corporation. NVIDIA H100 Tensor Core GPU Architecture. White paper, March 2022. URL https://resources.nvidia.com/en-us-hopper-architecture. Accessed October 29, 2026.
- [33]NVIDIA Corporation. NVIDIA Blackwell Architecture Technical Brief. Technical brief, 2025. URL https://resources.nvidia.com/en-us-blackwell-architecture. Version 2.1; accessed October 29, 2026.
- [34]Peebles, W. and Xie, S. Scalable diffusion models with transformers. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4195–4205. IEEE, 2023. URL https://openaccess.thecvf.com/content/ICCV2023/html/Peebles_Scalable_Diffusion_Models_with_Transformers_ICCV_2023_paper.html.arxiv.org/abs/2212.09748
- [35]Penedo, G., Kydlíček, H., Ben allal, L., Lozhkov, A., Mitchell, M., Raffel, C., von Werra, L., and Wolf, T. The FineWeb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems, volume 37, pp. 30811–30849, 2024. doi: 10.52202/079017-0970. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/370df50ccfdf8bde18f8f9c2d9115bda-Abstract-Datasets_and_Benchmarks_Track.html.DOI
- [37]Rae, J. W., Potapenko, A., Jayakumar, S. M., Hillier, C., and Lillicrap, T. P. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations, 2020. URL https://iclr.cc/virtual_2020/poster_SylKikSYDH.html.arxiv.org/abs/1911.05507
- [38]Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 (140):1–67, 2020. URL https://www.jmlr.org/papers/v21/20-074.html.arxiv.org/abs/1910.10683
- [39]Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. FlashAttention-3: Fast and accurate attention with asynchrony and low-precision. In Advances in Neural Information Processing Systems, volume 37, pp. 68658–68685, 2024. doi: 10.52202/079017-2193. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/97ed79c3e082c6df10a8d6103a2eebd2-Abstract-Conference.html.
- [40]Sun, Y., Li, Y., Zou, Z., Du, B., Zhang, Z., Dong, H., Fan, G., and Wang, H. VFA: Relieving vector operations in flash attention with global maximum pre-computation. arXiv preprint arXiv:2604.12798, 2026. URL https://arxiv.org/abs/2604.12798.
- [41]Team Wan. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025. URL https://arxiv.org/abs/2503.20314.
- [43]Yang, A., Li, A., Yang, B., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a. URL https://arxiv.org/abs/2505.09388.
- [44]Yang, S., Xi, H., Zhao, Y., Li, M., Zhang, J., Cai, H., Lin, Y., Li, X., Xu, C., Peng, K., Chen, J., Han, S., Keutzer, K., and Stoica, I. Sparse VideoGen2: Accelerate video generation with sparse attention via semantic-aware permutation. In Advances in Neural Information Processing Systems, volume 38, pp. 96965–96991, 2025b. doi: 10.52202/085713-3243. URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/8bd5148caced2d73cea7b6961a874a49-Abstract-Conference.html.DOI
- [45]Yuan, J., Shinn, C., Xu, K., Cui, J., Klimiashvili, G., Xiao, G., Zheng, P., Li, B., Zhou, Y., Ye, Z., You, W., Zheng, T., Brown, D., Wang, P., Hoenerbach, M., Cai, R., Demouth, J., Owens, J. D., Hu, X., Han, S., Liu, T., and Mao, H. BLASST: Dynamic BLocked attention sparsity via softmax thresholding. In Proceedings of Machine Learning and Systems, volume 8, pp. 843–859, 2026. URL https://proceedings.mlsys.org/paper_files/paper/2026/hash/cee784cbe46d854843e4c883a3321ef-Abstract-Conference.html.arxiv.org/abs/2512.12087
- [46]Zadouri, T., Hoehnerbach, M., Shah, J., Thakkar, V., and Dao, T. FlashAttention-4: Algorithm and kernel pipelining co-design for asymmetric hardware scaling. In Proceedings of Machine Learning and Systems, volume 8, pp. 912–926, 2026. URL https://proceedings.mlsys.org/paper_files/paper/2026/hash/ae8b0b5838ba510daff1198474e7b984-Abstract-Conference.html.arxiv.org/abs/2603.05451
- [49]Zhang, J., Wei, J., Zhang, P., Zhu, J., and Chen, J. SageAttention: Accurate 8-bit attention for plug-and-play infer- ence acceleration. In International Conference on Learn- ing Representations, volume 2025, pp. 71566–71585, 2025c. URL https://proceedings.iclr.cc/ paper_files/paper/2025/hash/b286c344 d38e10d2466c0514b78e2f36-Abstract-Con ference.html.arxiv.org/abs/2410.02367
- [51]Zhang, P., Chen, Y., Su, R., Ding, H., Stoica, I., Liu, Z., and Zhang, H. Fast video generation with sliding tile attention. In Proceedings of the 42nd International Con- ference on Machine Learning, volume 267 of Proceed- ings of Machine Learning Research, pp. 74714–74731. PMLR, 2025e. URL https://proceedings.mlr. press/v267/zhang25m.html.arxiv.org/abs/2502.04507
- [52]Zhao, W., Bai, L., Rao, Y., Zhou, J., and Lu, J. UniPC: A unified predictor-corrector framework for fast sampling of diffusion models. In Advances in Neural Information Processing Systems, volume 36, pp. 49842–49869, 2023. URL https://arxiv.org/abs/2302.04867.











