betaFeedback

WorksComputer Science

dit.xpu : Taming DiT Sparse Inference on Mobile SoCs

Abstract

Diffusion Transformers (DiTs) enable on-device image generation and editing but incur high attention costs at high resolutions. Existing sparse attention reduces computation, yet dynamic sparsity introduces preprocessing and irregular data accesses that poorly fit mobile NPUs. We present DiT.XPU, a sparse DiT inference framework for heterogeneous mobile SoCs. DiT.XPU combines tile-aligned sparse attention with anchor-based error compensation by exploiting slowly varying dense-sparse output residuals across denoising steps. Heterogeneous XPU collaboration overlaps NPU dense attention with GPU sparse calibration, while mask-guided dataflow reduces data transfers and layout conversion by overlapping data movement with computation. Across five DiT models on two Snapdragon smartphones, DiT.XPU achieves up to 1.56x end-to-end denoising speedup over the Hexagon-NPU backend and 11.5x attention-kernel speedup over the fastest alternative, while preserving generation quality close to dense attention. Code is available at https://github.com/IF-LAB-PKU/dit.xpu.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Conference’17, Washington, DC, USA

© 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM.

ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM

https://doi.org/10.1145/nnnnnnn.nnnnnnn

ACM Reference Format:

Jiayu Chen, Jinye Hu, Maoliang Li, Anqian Cen, Hailong Zou, Zihao Zheng, Xinming Wei, Guojie Luo, Jie Zhang, Xuanzhe Liu, and Xiang Chen. 2026. dit.xpu : Taming DiT Sparse Inference on Mobile SoCs. In . ACM, New York, NY, USA, 15 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

Introduction

Diffusion Transformer (DiT) architectures [13, 35, 56] have enabled remarkable image generation capabilities with broad applications in digital content creation, image editing [3], and visual design. Yet these capabilities are not widely accessible through on-device execution on mobile devices, a pervasive platform for multimodal interaction, because their limited compute resources struggle to support the substantial demands of DiT inference.

To understand this barrier, consider how DiTs perform inference. Unlike autoregressive LLMs [1, 29], which process new tokens using a KV cache, DiTs recompute attention over the entire visual token grid at every denoising step. As resolution increases, the visual token count grows, and global attention cost scales quadratically with it. This cost recurs throughout denoising, making inference compute-intensive. As shown in Figure 1, Stable Diffusion 3.5 Medium requires nearly half an hour of denoising to generate a single 2048 × 2048 image on the Snapdragon 8Gen4 NPU, with attention accounting for 86% of the latency.

Stacked bar chart showing DiT latency breakdown across resolutions

Figure 1. Attention dominates high-resolution DiT latency. It accounts for 32%, 60%, and 86% of the SD3.5 Medium DiT runtime at 512, 1024, and 2048 resolution, respectively, and 64% for Z-Image Turbo at 2048.

Sparse token interactions in DiT attention offer an opportunity to reduce computation. Sparse attention retains important interactions while omitting those with little effect on the output [2, 10, 20, 23, 57]. Because their importance varies with denoising state, these methods dynamically map selected token-block computations onto hardware [27, 62]. This mapping introduces preprocessing overhead and irregular memory accesses, so existing approaches use custom CUDA kernels to exploit GPU parallelism. For example, SpargeAttn [62] uses fused CUDA kernels to reduce the combined preprocessing and mapping overhead.

However, mobile SoCs struggle to efficiently execute DiT inference, even with sparse attention. Compared with NVIDIA GPUs, mobile SoCs offer limited compute resources fragmented across heterogeneous CPUs, GPUs, and NPUs. Even the NPU, their most capable processor for neural workloads, faces restrictive tensor layout requirements and limited on-chip memory [33]. Supporting sparse attention under these constraints requires layout conversion and carefully scheduled memory transfers, adding preprocessing and mapping overhead. As shown in Figure 5, SpargeAttn spends 47.4% of its total attention latency mapping the sparsity pattern.

Figure 5. Dynamic and static sparse attention kernel breakdown for SD3.5 Medium at 1024 × 1024 on 8Gen4 NPU.

Based on our analysis of DiT inference mechanisms and mobile SoC constraints, we identify three progressively more complex challenges.

(1) Heterogeneous XPU. Given the limited capacity of mobile PUs, collaboration among heterogeneous cores has become mainstream, commonly termed the XPU computing paradigm. These XPUs differ in computational capability and available resources, so DiT tasks must be assigned based on their execution characteristics. This requires partitioning the inference graph into tasks suited to different XPUs and arranging their execution as a pipeline. The resulting pipeline must preserve task dependencies while efficiently exchanging intermediate tensors between XPUs.

(2) Fragmented Dataflow. Although XPU collaboration alleviates workload capacity constraints, fragmented computing resources further complicate dataflow. A partition that balances computation may still create mismatches in memory capacity and data layout. To fit within an XPU’s local memory, a task may need to be split into smaller blocks, leading to repeated transfers of intermediate tensors. When tensors move between XPUs, they may need to be rearranged to match the receiving unit’s data layout. Such overhead can erode the gains from XPU collaboration.

(3) Approximation Errors. Even after the workload is deployed, adapting the algorithm to hardware constraints can introduce approximation errors that threaten generation quality. For sparse attention, mapping an adaptive interaction pattern onto fixed-granularity computation limits its ability to track changes in interaction importance across denoising steps. Interactions omitted by the pattern may become important later, introducing approximation errors that can degrade quality. Compensating for these errors is necessary to retain quality under efficient XPU execution.

To address these challenges, we present dit.xpu, a training-free framework that leverages mobile XPU resources for efficient sparse DiT inference. dit.xpu introduces an approximate tile-aligned attention scheme that provides flexibility for hardware mapping through predictable computation and data access. The framework incorporates the following key designs:

Hardware-Aligned Sparse Attention Algorithm. To characterize the approximation errors caused by mapping an adaptive interaction pattern onto fixed-granularity computation, we compare sparse and dense attention outputs across denoising steps. Although omitted interactions may become important later, our analysis shows that the output residual changes relatively slowly between adjacent steps. Accordingly, dit.xpu computes both attention outputs on the same inputs at anchor steps and caches their difference as a calibration residual. The dense output serves the current step, while subsequent sparse steps reuse the residual to compensate for approximation errors, helping retain quality.

Heterogeneous XPU Collaboration. To mitigate the latency overhead of anchor-based compensation, dit.xpu orchestrates heterogeneous XPU collaboration by partitioning the anchor-step inference graph according to task characteristics and available resources: the NPU computes dense attention for the current step, while the GPU concurrently computes sparse attention for calibration. Asynchronous submission overlaps the two branches within a cross-XPU pipeline, while shared memory mappings give both processors direct access to intermediate tensors without explicit cross-processor copies. The runtime synchronizes the branches when updating the calibration residual or reusing buffers, preserving task dependencies while limiting coordination overhead.

Mask-Guided Dataflow Optimization. To mitigate the transfer and rearrangement overhead caused by fragmented XPU resources, dit.xpu derives a dataflow schedule from the static attention mask. By grouping contiguous accesses and reusing loaded blocks across neighboring computations, the schedule reduces repeated transfers when attention is partitioned to fit local memory. dit.xpu further overlaps data movement with computation and produces intermediate tensors directly in the layout required by attention execution, avoiding separate rearrangement stages. Together, these mechanisms reduce the dataflow cost of XPU collaboration.

Implementation and evaluation. We implement dit.xpu in over 5,000 lines of C++ and Python and evaluate five DiT models on the OnePlus Ace 5 and OnePlus 13 at resolutions from 256 × 256 to 1024 × 1024. dit.xpu achieves up to an 11.0× attention-kernel speedup over the fastest alternative and a 1.56× end-to-end denoising speedup over Stable-Diffusion.cpp’s [25] Hexagon-NPU backend. Across the five models, GenEval improves by 0.54 percentage points on average, while HPSv2.1 and ImageReward decrease by only 0.06 and 0.01, respectively, relative to dense attention.

Our contributions are summarized as follows:

  • We analyze sparse DiT inference on mobile SoCs and identify three challenges: heterogeneous XPU execution, fragmented dataflow, and approximation errors.

  • We present three techniques: anchor-based error compensation, heterogeneous XPU collaboration, and mask-guided dataflow optimization for tile-aligned sparse attention.

  • We design and implement dit.xpu, the first DiT sparse inference framework that coordinates mobile XPUs through asynchronous execution and shared memory.

  • We demonstrate that, across five DiT models on two mobile devices, dit.xpu achieves up to an 11.0× attention-kernel speedup over the fastest alternative and a 1.56× end-to-end denoising speedup over Stable-Diffusion.cpp’s Hexagon NPU backend, while maintaining generation quality close to dense attention.

Background

Inference Mechanism of DiTs

A DiT [35] repeatedly applies its denoising network to the full visual token grid. For NN visual tokens, feed-forward layers and projections scale linearly with NN, whereas self-attention scales quadratically. With 8× latent downsampling and 2 × 2 patches, a 1024 × 1024 image contains 4096 visual tokens. As Figure 1 shows, attention’s share of denoising latency in SD3.5 Medium [13] rises from 32% at 512 × 512 to 86% at 2048 × 2048. Reducing attention computation is therefore key to accelerating high-resolution denoising.

DiT attention exhibits sparsity. Dense attention computes every query–key interaction, although their weights

are highly uneven. Figure 2 visualizes this imbalance: its first column shows the full QQ–KK attention map of a single layer, dominated by a narrow diagonal band, with only a few off-diagonal streaks reaching comparable strength; the last three columns show the spatial attention distributions of three selected query tokens over the spatial key grid. In both models the selected queries concentrate their weight on a compact, spatially local region and leave the rest of the grid nearly dark. This concentration motivates retaining influential interactions and skipping weaker ones [28, 47].

Figure 2. Attention patterns across DiT models. The first column shows the full Q–K attention map of a representative layer, while the remaining columns visualize the spatial attention distributions of selected query tokens.

Sparse attention workflow. Sparse attention retains selected key blocks for each query block [50, 62]. In dynamic sparse attention (Figure 3(a)), the query QQ and key KK are partitioned into blocks QiQ_i and KjK_j. Averaging the tokens in each block yields representatives qˉi\bar{q}_i and kˉj\bar{k}_j, whose similarity provides a coarse block score sijs_{ij}. High-scoring block pairs form a binary mask MM, where Mij=1M_{ij}=1 denotes a retained pair. Token-level attention is then computed only over selected blocks as Oi=softmax⁡(QiK[Mi]Td)V[Mi]O_i = \operatorname{softmax}\left(\frac{Q_iK[M_i]^{\mathsf{T}}}{\sqrt{d}}\right)V[M_i], where K[Mi]K[M_i] and V[Mi]V[M_i] contain the key and value blocks selected for QiQ_i. Re-computing the block means, scores, and mask at each step allows the selected interactions to change with the attention pattern.

Figure 3. (a) Dynamic sparse attention workflow. (b) Static sparse attention workflow. The image K/V blocks are requested from the dynamically selected blocks in (a), and from the fixed mask in (b).

In static sparse attention (Figure 3(b)), MM is defined before sampling and reused across steps [55]. The same attention computation applies, but block averaging, scoring, and selection are skipped during denoising. This removes repeated selection while leaving interactions outside MM unavailable.

Hardware Constraints of Mobile SoCs

Fragmented XPU Resources in a Shared-Memory SoC. Compute capacity in a mobile SoC is distributed across complementary processing units: an ARM CPU cluster, an Adreno GPU, and a Hexagon NPU [33], as shown in Figure 4. The CPU handles control flow, graph scheduling, and operators without dedicated kernels; the GPU handles workloads with irregular access patterns; and the NPU provides high throughput for dense matrix multiplication. All three share physical DRAM and its bandwidth through the system interconnect. Because DiT inference combines dense computation, irregular operations, and control tasks, efficient execution requires coordination across these processing units.

Figure 4. Architecture of a mobile SoC. Left: the CPU, GPU, and Hexagon NPU with their compute and memory resources. Right: the tensor stored in tile-first order, and the storage order within a 32×3232 \times32 tile.

Limited On-Chip Storage and Operand Staging. HMX operands must be staged in limited on-chip memory [33]. The left side of Figure 4 shows the NPU’s 1 MiB L2 cache and 8 MiB TCM, together with the activation and weight memories inside HMX. Part of the TCM can be configured as VTCM to supply data to HMX. The NPU moves data from DRAM through the L2fetch path and an asynchronous UserDMA engine. When operands cannot fit in VTCM at once, they must be loaded in batches; insufficient reuse between batches causes repeated transfers.

Tile-First Operand Layout and Conversion. Within the Hexagon NPU, scalar cores execute control code, HVX vector cores perform SIMD operations, and the HMX matrix core performs tensor computation [33]. HMX accesses operands in tile-first order: in the upper-right illustration of Figure 4, it visits tiles 0–3 before tiles 4–7, rather than traversing the entire matrix row by row. The tiles supplied to HMX must therefore be stored contiguously. CPUs and GPUs typically use row-major tensors, so passing their results to HMX requires layout conversion.

Fixed Tile Granularity and Padding Overhead. HMX performs matrix operations using fixed-size tiles [33]. For FP16 operands, each tile contains 32×3232 \times32 elements and occupies 2 KiB, as illustrated on the right side of Figure 4. When an operand dimension is not a multiple of 32, its boundary regions must be padded to full tiles.

Motivation and Challenges

We first examine why dynamic sparse attention maps poorly to a mobile NPU. We then identify an opportunity in the residual between static sparse and dense attention, and characterize the hardware costs of exploiting it.

Empirical Motivation

As discussed in § 2.1, dynamic sparse attention must select and map token-block interactions before the attention kernel can run. This preparation lies on the critical path. As shown in Figure 5, at 40% sparsity, SparseAttn [62] spends 146.0 ms on block-mean computation and block selection before its 164.6 ms attention kernel. At 80% sparsity, PISA [27] spends 176.4 ms on block-mean computation, block selection, and K/V preparation before its 569.1 ms kernel. Preparation thus accounts for 47.0% and 23.7% of their respective attention latencies, consuming a substantial part of the benefit from skipping token interactions.

The cost extends into kernel execution. Selected K/V blocks can differ across query blocks and denoising steps, so their transfers and VTCM staging cannot follow a schedule fixed in advance. This is consequential for Hexagon HMX, which requires operands in a compatible tile layout. At the same 80% sparsity, PISA’s block-sparse attention kernel takes 569.1 ms, compared with 41.6 ms for static block-sparse attention, even after preparation time is excluded. This kernel-level gap is consistent with the difficulty of executing dynamic block access efficiently on the mobile NPU.

By contrast, static block-sparse attention fixes its interaction mask before inference. Guided by the attention pattern in Figure 2 and prior local-window designs [2], it retains a predefined diagonal band of token-block interactions. This fixed pattern removes per-step block selection and makes K/V accesses predictable. This predictable execution is reflected in the 41.6 ms attention-kernel latency shown in Figure 5. Its fixed mask, however, cannot track changes in interaction importance across inputs and denoising steps, and may omit interactions needed for generation. Relative to dense attention, static sparsity reduces GenEval and HPSv2.1 by 41.54% and 8.72%, respectively, on Stable Diffusion 3.5 Medium. Across five DiT models, the mean relative reductions are 34.22% and 16.61%, respectively, as shown in Table 1.

ModelGenEval (%) ↑ DenseGenEval (%) ↑ StaticHPSv2.1 ↑ DenseHPSv2.1 ↑ StaticImageReward ↑ DenseImageReward ↑ Static
SD3.5 Medium68.3726.8329.1820.460.8774−0.7486
SD3.5 Large71.3123.6629.9119.350.8781−1.3234
Z-Image75.3565.8827.7425.500.69360.2371
Z-Image-Turbo75.4963.1429.8328.590.69600.4741
FLUX.2 [klein]82.6670.6030.0928.390.91080.5522

Table 1. Static block-sparse attention substantially degrades generation quality across five DiT models.

Algorithmic Opportunity

Inspired by Re-ttention [9] and DiTFastAttn [55], we examine dense and static sparse attention computed from the same QtQ_t, KtK_t, and VtV_t at each denoising step tt. Sparse attention omits some keys and normalizes the retained scores over a smaller set. It therefore removes the omitted keys’ contributions while changing the weights assigned to retained keys. For query token ii, we quantify this normalization shift with

ρt,i=exp⁡(LSE⁡t,is−LSE⁡t,id),(1)\rho_{t,i} = \exp\left(\operatorname{LSE}^{s}_{t,i} - \operatorname{LSE}^{d}_{t,i}\right), \tag*{(1)}

where the log-sum-exp terms cover the retained and all keys, respectively. The coefficient ρt,i\rho_{t,i} is the fraction of dense attention probability retained by the static mask; multiplying the sparse output Ot,isO^{s}_{t,i} by ρt,i\rho_{t,i} restores the dense normalization of those retained interactions. Figure 6(a) shows that ρ\rho remains below one but changes gradually across denoising steps. The normalization shift is therefore substantial, while its coefficient varies smoothly.

Figure 6. Calibration coefficient ρ\rho and residual MSE across denoising steps for SD3.5 Medium at 1024 × 1024 resolution.

We define the remaining residual as

Rt,i=Ot,id−ρt,iOt,is,(2)R_{t,i} = O^{d}_{t,i} - \rho_{t,i}O^{s}_{t,i}, \tag*{(2)}

which captures the omitted keys’ contribution under dense attention. Figure 6(b) reports the MSE of this residual. It stays low for most denoising steps, although it rises near the beginning and end. Thus, after correcting the normalization of retained interactions, the measured output discrepancy is small through most of the denoising trajectory. Together,

the gradual variation in ρ\rho and the low residual MSE motivate periodically computing these terms at anchor steps to compensate subsequent static sparse attention outputs.

Deployment Challenges

The temporal stability of the dense–sparse residual offers a way to compensate for the approximation errors of static sparse attention. Realizing this opportunity on a mobile SoC, however, requires addressing two deployment challenges: coordinating calibration across XPUs and feeding sparse computation from limited on-chip memory.

Heterogeneous XPU execution. As Figure 4 shows, the CPU, GPU, and NPU share unified memory through the system interconnect but differ in execution engines and local storage. Compensating for static sparse attention requires dense and sparse outputs computed from the same QQ, KK, and VV, since the sparse output alone cannot reveal the approximation error. Computing both serially adds dense attention work and may diminish the benefit of sparsity. Distributing them across XPUs can overlap computation, but placement must account for device capabilities and shared input access. The correction can be formed only after both outputs are ready, so task submission, tensor movement, and synchronization may offset that overlap; unified physical memory alone does not guarantee compatible device mappings, layouts, or buffer visibility.

Fragmented dataflow. The NPU in Figure 4 has only 8 MiB of on-chip TCM, whereas one layer’s image K/V in SD3.5 Medium occupies 24.0 MiB. Q tiles, text K/V, intermediate results, and staging buffers further reduce the space available for image K/V. The simplest policy loads the blocks required by one query block, computes its output, and releases that storage before moving to the next query block. This limits the live working set, but neighboring queries need overlapping image K/V blocks and repeatedly access text K/V, causing the same data to be fetched again. As Table 2 shows, this policy reads 456.3 MB from DRAM. Cross-block reuse retains shared K/V blocks while neighboring queries are processed and reduces reads to 216.4 MB. However, retained blocks compete with QQ, text K/V, and staging buffers for TCM space, limiting how far reuse and prefetching can extend. The remaining DMA requests average only 5.32 KB, and fetched blocks still require conversion to HMX’s tile-first layout. Consequently, reuse alone leaves 2.11 ms of data wait and 3.40 ms of repacking. The challenge is to coordinate block residency, query order, data movement, and layout under the same limited TCM budget.

MeasurementPer-query loadingCross-query reusedit.xpu
DRAM read volume (MB)456.3216.451.36
Mean DMA request size (KB)15.335.327.42
Data-wait time (ms)4.522.110.02
K/V repacking time (ms)5.453.400
Kernel latency (ms)24.9621.3513.30

Table 2. K/V data movement and kernel latency for SD3.5 Medium static sparse attention on the 8Gen4 NPU.

System Design

Design Overview

As shown in Figure 7, we present dit.xpu, a training-free framework for sparse DiT inference on heterogeneous mobile SoCs. To translate reduced attention computation into end-to-end denoising speedup, its design addresses three challenges: approximation errors, heterogeneous XPU execution, and fragmented dataflow.

Figure 7. Overview of dit.xpu.

Hardware-Aligned Sparse Attention Algorithm. dit.xpu uses a static, tile-aligned block-sparse attention mask so that retained interactions execute as regular matrix tiles on the NPU. The fixed mask avoids per-step pattern selection and makes computation and data access predictable. Because omitted interactions may become important during denoising, dit.xpu periodically computes dense and sparse attention from the same Q/K/V at anchor steps. It caches their output difference as a calibration residual: the dense output serves the current step, while subsequent sparse steps reuse the residual to compensate for approximation errors.

Heterogeneous XPU Collaboration. At each anchor step, the NPU computes dense attention for the generation path

while the GPU computes sparse attention for calibration. Asynchronous submission overlaps the two branches in a cross-XPU pipeline. Shared memory mappings let both processors access the same intermediate tensors without cross-processor copies. The runtime synchronizes the branches before updating the calibration residual, preserving task dependencies while limiting anchor-step overhead.

Mask-Guided Dataflow Optimization. The static mask specifies which K/V blocks each query block accesses, allowing dit.xpu to plan transfers before execution. When the K/V working set exceeds VTCM capacity, the schedule groups contiguous transfers and reuses loaded blocks across neighboring query blocks. Double buffering overlaps loading the next blocks with NPU computation on the current blocks. Intermediate tensors are produced directly in the layout required by attention execution, avoiding separate rearrangement stages and reducing dataflow stalls.

Hardware-Aligned Sparse Attention Algorithm

As image resolution increases, DiTs recompute global attention over a growing visual token grid at every denoising step, with cost quadratic in token count. Sparse attention reduces this cost, but mapping it to mobile NPUs presents two issues:

  • Fine-grained sparsity does not match fixed NPU matrix tiles, while dynamic block selection incurs per-step preprocessing and mapping overhead.

  • A static mask makes computation and data access predictable, but omitted interactions introduce approximation errors through a normalization shift and missing key contributions.

We therefore combine a tile-aligned static mask with periodic anchor calibration. The mask executes retained interactions as regular matrix blocks; anchor steps compute the normalization coefficient and residual used to calibrate subsequent sparse outputs.

Tile-Aligned Static Halo Mask. Dynamic block-sparse attention first estimates the importance of query and K/V blocks, then selects K/V blocks for each query block using Top-kk or Top-pp. Designed largely for NVIDIA GPUs, these methods use programmable CUDA kernels for block reduction and selection, then access selected K/V blocks through indices generated at runtime.

As discussed in Section 3, the Hexagon NPU’s HMX executes fixed-layout matrix tiles in VTCM. Computing Q/K block means and performing Top-kk/Top-pp selection require reductions, comparisons, and index generation, which rely mainly on the lower-throughput HVX. Even if HMX computes inter-block scores, block means must first be prepared and results exchanged between compute units. This preprocessing and mapping must finish before attention and recur at every layer and denoising step.

We avoid this repeated work by fixing, before inference, the image K/V blocks accessed by each image query block.

We group 32 spatially adjacent image tokens into a block, so every retained query–K/V block pair forms a 32×3232 \times32 attention matrix tile aligned with HMX’s FP16 granularity. Because these block pairs are predetermined, the system can plan K/V transfer order and VTCM placement in advance, then follow that plan throughout denoising.

Specifically, image blocks form a two-dimensional grid. For image-to-image attention, a query block at (y,x)(y,x) retains K/V blocks satisfying ∣y−y′∣≤1|y-y'| \le1 and ∣x−x′∣≤1|x-x'| \le1, clipped at grid boundaries. Each query block therefore computes at most 3×33 \times3 image block pairs. Neighboring query blocks share much of their K/V working set, allowing the system to merge transfers, reuse loaded blocks in VTCM, and skip matrix tiles outside the halo.

Anchor-Based Error Compensation. A static mask can omit interactions that become important later in denoising, introducing approximation errors. It removes the omitted keys’ contributions and changes the Softmax normalization of retained keys. As shown in § 3.2, ρt,i\rho_{t,i} changes gradually across steps, while the reported MSE of Rt,iR_{t,i} stays low through most of the denoising trajectory. These observations motivate periodic anchor steps.

We designate an anchor step aa every pp denoising steps and compute dense and static sparse attention from the same QaQ_a, KaK_a, and VaV_a. The dense output Oa,idO^d_{a,i} serves the current step. For each query token ii, we obtain ρa,i\rho_{a,i} by exponentiating the difference between LSE⁡a,is\operatorname{LSE}^{s}_{a,i} and LSE⁡a,id\operatorname{LSE}^{d}_{a,i}, then compute the omitted-key residual Ra,iR_{a,i} as defined above. A subsequent sparse step tt reuses the terms from the most recent anchor:

O~t,i=ρa,iOt,is+Ra,i.(3)\widetilde{O}_{t,i} = \rho_{a,i} O^s_{t,i} + R_{a,i}. \tag*{(3)}

This approximately compensates for the normalization shift and omitted-key contributions without recomputing dense attention at every step.

Heterogeneous XPU Collaboration

At an anchor step aa, the NPU dense branch and GPU sparse branch read the same (Qa,Ka,Va)(Q_a, K_a, V_a) within each attention layer. The GPU uses the same static mask MM as subsequent NPU sparse steps. Both branches produce attention outputs and row-wise log-sum-exp values; once both results are ready, the runtime updates (ρa,i,Ra,i)(\rho_{a,i}, R_{a,i}) for each query ii. The dense output Oa,idO^d_{a,i} serves the current step. A later sparse step computes Ot,isO^s_{t,i} on the NPU and applies the latest calibration terms to obtain O~t,i=ρa,iOt,is+Ra,i\widetilde{O}_{t,i} = \rho_{a,i}O^s_{t,i} + R_{a,i}. Serializing the two anchor branches would place the auxiliary sparse computation on the generation path. dit.xpu instead overlaps their execution and shares their intermediate tensors.

NPU–GPU Co-Execution. Once (Qa,Ka,Va)(Q_a, K_a, V_a) is ready, the dense and sparse branches have no dependency on each other. The runtime submits dense attention to the NPU for the current step and static sparse attention to the GPU for calibration. After producing Oa,idO^d_{a,i}, the NPU proceeds with the main path while the GPU completes its branch. The runtime waits for both outputs before publishing the new calibration terms; a subsequent sparse step reads only a fully published correction. It also keeps the input buffers available until the GPU has finished reading them. Figure 8Figure 1 and Figure 2 illustrate how this schedule moves the auxiliary sparse branch out of serial NPU execution and overlaps it with the main path.

Figure 8. An illustration of the heterogeneous XPU collaboration.(1) sequential execution with CPU fallback, (2) NPU–GPU co-execution with explicit data copies, and (3) NPU–GPU co-execution through shared memory mappings.

Shared Buffer Mapping. Co-execution would require copies at each anchor step if the NPU and GPU used separate tensor storage. Figure 8 shows a 51.5 MB Q/K/V transfer costing 3.2 ms and a 25.3 MB output transfer costing 2.5 ms. To remove these transfers, dit.xpu allocates Q/K/V, branch outputs, and calibration terms in shared physical memory, then maps the buffers into the CPU, NPU, and GPU address spaces. The GPU reads the NPU-produced Q/K/V from the shared buffers, and the calibration update reads both branch outputs without a staging copy. Device events ensure that readers see completed writes and that buffers are not reused while a branch still needs them. The runtime retains mappings across blocks to avoid repeated mapping setup. Figure 8Figure 3 illustrates the execution without copy stages.

Mask-Guided Dataflow Optimization

Heterogeneous XPU collaboration in § 4.3 limits anchor-step calibration overhead, but NPU attention must still move Q, K, and V from DRAM into VTCM and arrange them in HMX’s tile layout. The full layer’s K/V working set cannot reside on-chip. Loading retained blocks independently therefore repeats transfers of overlapping image-halo and text K/V data. Figure 9(a) depicts these per-block requests and the separate layout conversion, both of which can leave HMX waiting for data.

Figure 9. (a) Naive Dataflow with repeated K/V loading and layout conversion; (b) Mask-Guided Dataflow with K/V reuse, prefetching, and fused QKV packing.

Table 2 shows that with independent per-query loading, data wait and K/V repacking take 4.52 ms and 5.45 ms, respectively. Cross-query reuse reduces these costs to 2.11 ms and 3.40 ms, but both remain significant. dit.xpu therefore uses the predictable access pattern of the static mask to group and prefetch K/V blocks, while producing Q/K/V in the layout required by attention.

Asynchronous KV Block Loading. The static halo mask in § 4.2 determines each query block’s K/V access set before inference, but transferring blocks individually still creates many small requests. dit.xpu groups query blocks in the same grid row into a query row, gathers their required K/V blocks, and merges contiguous addresses into transfer intervals. Using tensor strides, block width, and target VTCM addresses, it constructs two-dimensional UserDMA requests that move multiple token rows with one submission. Figure 9(b) shows these per-row requests in place of the per-block requests.

Merged requests alone cannot prevent adjacent query rows from loading the same blocks again. dit.xpu retains image K/V needed by later rows and keeps shared text K/V resident when VTCM capacity permits. It loads the remaining image K/V by row or small batch, reusing available blocks while keeping the on-chip working set within capacity.

To overlap transfers with computation, dit.xpu uses two buffers for transfers and two for the packed K/V cache. While HMX processes the current query row, UserDMA prefetches K/V missing from the next row into the other buffer. The runtime waits for that transfer before consuming the prefetched blocks, then alternates between buffers. As illustrated in Figure 9(b), this schedule preserves blocks in use while loading the next row alongside NPU computation.

Fused NPU Packing and Conversion Kernel. Double buffering hides transfer time, but HMX still requires K/V in a tile-first layout. With unpacked inputs, the system must pack loaded data and cache it for neighboring query rows. If QKV projection first writes row-major tensors and the CPU then applies normalization or RoPE, separate rearrangement adds memory accesses and cross-processor synchronization. dit.xpu instead moves packing to the QKV projection’s write-out stage and fuses projection with its post-processing on the NPU. The operator converts FP32 activations to FP16, performs matrix multiplication and bias addition, applies Q/K RMSNorm and the required RoPE, and writes packed Q, K, and V. It processes image and text inputs separately, placing them in a unified layout at 32-token tile boundaries without an intermediate row-major concatenation. Figure 9(b) shows packing and conversion alongside QKV projection, replacing the separate conversion stage in (a). Subsequent DMA requests transfer packed data, which the attention kernel accesses through tile offsets without further packing.

Compared with block reuse alone, dit.xpu reduces DRAM reads from 216.4 MB to 51.36 MB and transfer stalls from 2.11 ms to 0.02 ms while eliminating rearrangement before attention. Compute time remains largely unchanged; the gain comes primarily from lower data preparation costs.

Implementation

dit.xpu is implemented on the ggml [15] compute graph of stable-diffusion.cpp [25], with HTP kernels compiled using the LLVM [24] toolchain in Hexagon SDK 6.0.0.2 [39]. Our implementation consists of three modules: Cross-XPU Execution, Dataflow Optimization, and Error Compensation. The CPU executes and schedules the graph. The HTP runs operators built around HMX and VTCM [33] through the open-source htp-ops-lib [17], and OpenCL [22] runs sparse attention asynchronously. Keeping the graph in ggml instead of QNN [38] allows the mask format, block size, and sparse plan to change across denoising steps.

Cross-XPU Execution. The CPU, CDSP, and OpenCL device operate on shared buffers. We allocate HTP buffers with rpcmem_alloc [44], obtain their descriptors with rpcmem_to_fd, and map them into the CDSP address space with fastrpc_mmap [37]. Tensor addresses are passed as descriptor–offset pairs. The OpenCL side imports the same descriptors as cl_mem using CL_MEM_DMABUF_HOST_PTR_QCOM and CL_MEM_USE_HOST_PTR [21], avoiding copies between processors. Imported objects are cached for buffers reused across graphs. We track mapping lifetimes and backing-store references separately, and unmap a buffer only after its last graph alias is released.

Dataflow Optimization. We instantiate a fixed block-sparse mask from the token layout and static-halo policy, and store its block size, mask format, and halo parameters as attention-node attributes for reuse across denoising steps. On the HTP, dynamic KV sparse attention constructs 2D UserDMA descriptors for the retained K/V blocks and transfers them from shared DRAM to VTCM. Its transfer plan follows each query block’s mask, while two staging banks prefetch the next K/V group during the current matrix computation. A second operator fuses QKV projection, bias, and Q/K normalization, applies rotary embedding on branches that use it, and writes Q, K, and V directly in the packed layout expected by attention. This removes the intermediate tensors and row-major round trip.

Error Compensation. At each dense residual anchor, the attention node evaluates dense and sparse branches. Exact correction computes ρ\rho from their LSEs. Otherwise, we use the preset scaling factor. We cache ρ\rho and the difference between the dense output and corrected sparse output once per anchor, then apply the cached correction to subsequent sparse steps without rerunning dense attention. To overlap the anchor branches, the graph thread submits sparse attention to one ordered OpenCL queue, flushes it, publishes its completion event, and starts dense attention on the HTP. A second queue submits the ρ\rho and residual kernels with the sparse event as a dependency. The shared Q/K/V range remains leased until sparse attention completes. Sparse O/LSE staging buffers and the dense-output lease remain held until the residual completes.

Experiments

Experiment Setup

Hardware platforms. We evaluate dit.xpu on two mobile devices equipped with different generations of Snapdragon SoCs. All devices run Android 13. The first device is a OnePlus Ace 5 with a Snapdragon 8 Gen 3 SoC, 24 GB of memory, and 1 TB of UFS 4.0 storage. The second is a OnePlus 13 with a Snapdragon 8 Gen 4 SoC, 24 GB of memory, and 1 TB of UFS 4.0 storage. These platforms cover multiple generations of Qualcomm mobile hardware and let us assess the portability of dit.xpu across heterogeneous devices.

Models and resolutions. We select representative text-to-image DiT models with different parameter scales, network architectures, and denoising schedules. The MMDiT family includes Stable Diffusion 3.5 Medium [13], Stable Diffusion 3.5 Large, and FLUX.2 [klein] [3]. The S3-DiT family includes Z-Image [56] and its distilled variant, Z-Image-Turbo. All models are evaluated at 256 × 256, 512 × 512, and 1024 × 1024 output resolutions. Unless otherwise stated, each model uses its standard sampling schedule and precision on all backends. Table 3 summarizes the architecture and the sampling configuration of the five models.

ModelArchitectureParamsHeadsHead DimLayersSteps
SD3.5 MediumMMDiT-X2.5B24642420
SD3.5 LargeMMDiT8.1B38643820
Z-ImageS3-DiT6B301283215
Z-Image TurboS3-DiT6B30128328
FLUX.2 [klein]RF Transformer4B24128254

Table 3. Architecture and sampling configuration of the evaluated DiT models.

Datasets and metrics. To quantify the effect of sparse inference on generation quality, we use several established text-to-image benchmarks. GenEval [16] measures compositional generation ability. HPSv2 [46] and ImageReward [49] measure alignment with human preferences. On MJHQ-30K [26], we report FID [19] and CLIP Score [18] to evaluate the distributional quality of generated images and text–image consistency, respectively. All generation-quality evaluations are conducted on four NVIDIA L40 GPUs.

Comparison baselines. We compare dit.xpu with baselines at two levels. For end-to-end inference, we use the

CPU, OpenCL-GPU, and Hexagon-NPU backends of Stable-diffusion.cpp [25], representing execution on the mobile CPU, GPU, and NPU, respectively. For sparse attention, we compare with PISA [27] at 80% sparsity and SpargeAttn [62] at 40% sparsity, following their evaluated configurations.

Settings. We use dense attention for the first 10% of denoising steps as a warm-up, then switch to sparse attention with dense anchors at five-step intervals. Sparse attention uses 32 × 32 query–key tiles and a fixed 3 × 3 image halo (r=1r = 1), while all channel connections involving text tokens remain dense. All reported latencies are averaged over five runs.

Latency Evaluation

End-to-End Denoising Latency

End-to-End Denoising Latency. We compare dit.xpu with the CPU, OpenCL-GPU, and Hexagon-NPU backends of SDcpp. As shown in Figure 10, dit.xpu outperforms Hexagon-NPU in all 30 model–resolution–device settings and achieves the lowest latency in 29 of them. At 1024 × 1024, it delivers a 1.15–1.45× speedup over Hexagon-NPU on the 8 Gen 3 device and a 1.21–1.56× speedup on the 8 Gen 4 device. The gains are most pronounced at the highest resolution, where the larger attention workload provides more opportunity for sparse execution to reduce denoising time.

Figure 10. End-to-end denoising latency of dit.xpudit.xpu and baseline backends across DiT models and mobile SoCs.

End-to-End Latency of Sparse-Attention Strategies

End-to-End Latency of Sparse-Attention Strategies. We compare the end-to-end denoising latency of dit.xpu with dense attention, PISA [27], SpargeAttn [62], and a static sparse-attention baseline across the five models. Table 4 shows that dit.xpu achieves an average 1.40× speedup over dense attention, exceeding the speedups of PISA and SpargeAttn. This result suggests that a fixed, hardware-aligned sparsity pattern and amortized anchor calibration translate attention savings into end-to-end gains more effectively than per-step sparse selection.

Latency (s) ↓DensePISASpargeStaticOursSpeedup
SD3.5 Medium57914468763253711.56×
SD3.5 Large157931832185138512061.31×
Z-Image919303011445366311.46×
Z-Image-Turbo48312396162923261.48×
FLUX.2-klein1864332951471541.21×
Mean Speedup1.00×0.40×0.72×1.51×1.40×

Table 4. End-to-end denoising latency (s) of dense attention and three sparse-attention strategies at 1024×1024 resolution on the Snapdragon 8 Gen 4.

Kernel Performance Evaluation

Attention Kernel

Attention Kernel. We benchmark a single attention kernel on two devices over five models at different output image resolutions, with all implementations sharing the same Q/K/V tensors, data precision, block size and tensor layout. Figure 11 shows that dit.xpu is the fastest implementation in every measured configuration on both devices, and that its margin over the best alternative grows with the resolution, reaching 2.50–3.27× on the 8 Gen 3 and 2.68–3.38× on the

Figure 11. Attention kernel latency of dit.xpudit.xpu and the baseline backends across DiT models and mobile SoCs.

ModelCPU DenseGPU DenseGPU SparseNPU DenseGPU/NPU AnchorNPU Sparse
SD3.5 Medium18.322.350.480.450.760.05
SD3.5 Large12.883.770.770.791.180.08
FLUX.2-klein12.174.200.230.530.720.07
Z-Image15.165.370.280.730.900.09
Z-Image-Turbo14.985.320.270.710.850.08

Table 5. Single-kernel energy consumption across DiT models and execution configurations.

8 Gen 4 device at 512×512512 \times512 and up to 11.5× and 7.0× at 1024×10241024 \times1024.

Energy Efficiency

6.3.2 Energy Efficiency. We measure the energy of a single attention kernel on the five models at 1024×10241024 \times1024 resolution. Table 5 shows that dit.xpudit.xpu consumes 0.72–1.18 J, an order of magnitude below CPU-dense and 2–7× below GPU-dense, and removing the dense anchor correction lowers it to 0.05–0.09 J, the lowest of all implementations.

Generation Quality

We evaluate generation quality at 1024×10241024 \times1024 across five models, comparing dit.xpudit.xpu with dense attention and sparse-attention baselines. As shown in Table 6, dit.xpudit.xpu closely matches dense attention on GenEval: it scores 0.35–1.05 points higher on four models and only 0.18 points lower on SD3.5 Medium. It also achieves the highest ImageReward among sparse methods on SD3.5 Large and FLUX.2 [klein], with scores of 0.8878 and 0.9243, respectively. Figure 12 provides representative visual comparisons. Together, these results suggest that dit.xpu largely preserves generation quality while accelerating inference through sparse attention.

Figure 12. Qualitative comparison of dit.xpu and the baselines on three prompts, one per row: SD3.5-Medium, SD3.5-Large and Z-Image.

MetricDensePISASpargeStaticOursΔ
GenEval ↑
SD3.5 Medium68.3764.5568.1926.8368.19-0.18
SD3.5 Large71.3170.0670.5423.6671.88+0.57
Z-Image75.3573.3075.1365.8876.26+0.91
Z-Image-Turbo75.4972.4975.4763.1476.54+1.05
FLUX.2-klein82.6684.8782.4070.6083.01+0.35
Avg. Δ–-1.58-0.29-24.61+0.54
HPSv2.1 ↑
SD3.5 Medium29.1827.6029.3520.4629.53+0.35
SD3.5 Large29.9128.5129.8119.3530.12+0.21
Z-Image27.7426.9427.7525.5027.72-0.02
Z-Image-Turbo29.8330.0129.8628.5929.86+0.03
FLUX.2-klein30.0929.8130.1028.3929.21-0.88
Avg. Δ–-0.78+0.02-4.89-0.06
ImageReward ↑
SD3.5 Medium0.880.860.85-0.750.86-0.02
SD3.5 Large0.880.690.88-1.320.89+0.01
Z-Image0.690.600.660.240.66-0.03
Z-Image-Turbo0.700.770.690.470.69-0.01
FLUX.2-klein0.910.910.920.550.92+0.01
Avg. Δ–-0.05-0.01-0.97-0.01
FID ↓
SD3.5 Medium118.70104.92150.24139.90107.60-11.10
SD3.5 Large119.07102.11142.20190.74108.07-11.00
Z-Image121.23105.55105.38125.54104.97-16.26
Z-Image-Turbo127.11111.00112.17129.37114.59-12.53
FLUX.2-klein122.52116.90118.37125.42114.90-7.61
Avg. Δ–-13.63+3.95+20.47-11.70
CLIP ↑
SD3.5 Medium81.1780.0781.3368.8481.28+0.11
SD3.5 Large81.4580.8481.6765.5081.65+0.20
Z-Image80.5680.2580.5278.4880.81+0.25
Z-Image-Turbo79.3579.9879.3879.1780.25+0.90
FLUX.2-klein79.6780.1379.7278.8283.72+4.05
Avg. Δ–-0.19+0.08-6.28+1.10

Table 6. Generation quality comparison across attention strategies. FID and CLIP are averaged over Landscape, People, and Food. Δ denotes the absolute change relative to Dense. For ↑ metrics, positive values indicate improvement; for FID↓, negative values indicate improvement.

Ablation Study

6.5.1 Component Ablation. We enable the proposed techniques one at a time on SD3.5 Medium at 1024×10241024 \times1024. Figure 13 reports the GenEval score and the end-to-end latency of the five versions: the complete dit.xpu reduces the latency from 578.6 s to 371.1 s, a 1.56× speedup, with GenEval essentially unchanged at 68.19% against 68.37%, whereas removing the anchor compensation collapses GenEval to 26.83%.

Component ablation

Figure 13. Component ablation. Each build adds one technique to the previous one.

6.5.2 Anchor Step Interval. We vary only the anchor interval pp, which inserts one anchor step every pp sparse steps, on SD3.5 Medium at 1024×10241024 \times1024, and report, for p=2p = 2 to p=6p = 6, GenEval, HPSv2.1, ImageReward and the end-to-end latency on the 8 Gen 4 device in Table 7. A sparser anchoring trades a small quality loss for a modest latency gain: from p=2p = 2 to p=5p = 5 the latency falls from 412.32 s to 371.11 s while GenEval moves only from 68.36% to 68.19%.

ppGenEval ↑HPSv2.1 ↑ImageReward ↑Latency (s) ↓
268.3629.630.8813412.32
368.3129.600.8725394.88
468.2429.550.8604386.40
568.1929.530.8575371.11
666.0729.420.8219367.09

Table 7. Generation quality and latency of dit.xpu under different anchor step intervals pp on SD3.5 Medium at 1024×10241024 \times1024 resolution and 20 steps.

6.5.3 Heterogeneous Strategy. We fix one denoising step on the Snapdragon 8 Gen 4 device and compare four alternative execution strategies against dit.xpu: npu divided computes the attention twice, separately, on the NPU; npu fused executes a single fused dense-and-sparse attention on the NPU; cpu + gpu executes the sparse attention on the CPU and the dense attention on the GPU; and npu + cpu executes the dense attention on the NPU and the sparse attention on the CPU. Figure 14 shows that dit.xpu is the fastest strategy in 13 of the 15 configurations: at 1024×10241024 \times1024 it is 1.05–1.23× faster than the best of the four alternatives.

Figure 14. Heterogeneous-strategy ablation for a single denoising step on the Snapdragon 8 Gen 4 device.

6.5.4 KV Load/Unload. We compare four NPU sparse-attention loading schemes, including chunk and baseline: chunk computes one Q block at a time and then loads the corresponding KV blocks; baseline first attempts to load all KV blocks and loads them if the VTCM capacity is sufficient, otherwise attempts to load all K blocks and loads them if the VTCM capacity is sufficient, and falls back to chunk otherwise; and no packed refers to our method without the repacking optimization proposed in §4.4.2. Figure 15 shows that dit.xpu reduces the kernel latency against dense loading by 9.9–12.3× at 1024×10241024 \times1024, with a median of 3.25× over the 27 configurations, and that in the median it is 1.44× faster than chunk and 1.18× faster than baseline.

Figure 15. KV load ablation of the attention kernel on the Snapdragon 8 Gen 4 device.

6.5.5 Performance at Different Resolutions. We measure the end-to-end latency of the SD3.5-medium model at different resolutions on the 8 Gen 4 device. We cover two

pairs of resolutions that contain the same number of pixels but differ in spatial arrangement, together with a high-resolution case at 2048×20482048 \times2048. Table 8 reports the end-to-end latency of dit.xpu: it is 1.99× and 1.94× faster than dense attention on the first pair, 1.52× and 1.56× on the second.

1920 × 10801080 × 1920896 × 11521152 × 896
Dense1565.881556.36493.32495.24
dit.xpu786.14803.97324.48317.02

Table 8. Latency of SD3.5 Medium at different resolutions.

Orthogonality with Other Acceleration Methods

To verify orthogonality with other acceleration methods, we combine dit.xpu with TeaCache [30], DBCache [36] and TaylorSeer [31]. As Table 9 shows, adding TaylorSeer lowers the GenEval score only slightly, to 0.6423, while the speedup on the 8 Gen 4 reaches 1.74×. The results show that dit.xpu is compatible with existing optimization methods.

MethodGenEval ↑\uparrowLatency (s) ↓\downarrowSpeedup ↑\uparrow
Dense0.68375791.00×\times
ours0.6819371.111.56×\times
w/ TeaCache0.6451361.571.60×\times
w/ DBCache0.6399333.891.73×\times
w/ TaylorSeer0.6423333.141.74×\times

Table 9. GenEval scores, inference latency and speedup of orthogonal caching on the DiT.xPU backend for the SD3.5-medium model at 1024×10241024 \times1024 resolution.

Related Work

Inference Optimization on Mobile SoCs

On-device inference has received growing attention. Mobile SoCs integrate CPUs, GPUs, and NPUs, and existing work primarily accelerates LLM inference by distributing computation across these processors [7, 48, 51]. For example, HeteroLLM partitions computation between the GPU and NPU to reduce data copies, but still runs most attention operations on the GPU. llm.npu assigns quantized linear layers to the NPU while leaving attention on the CPU or GPU. PowerInfer-2 also uses the NPU for dense attention, but does not reduce the amount of attention computation. More closely related to our work, shadowAttn [53] uses the NPU to identify important tokens and then performs sparse attention on the CPU or GPU. It updates token selection as the input changes, while the sparse computation itself remains outside the NPU. For DiTs [35], which process the full visual token grid at every denoising step, existing engines such as stable-diffusion.cpp [25] support on-device inference but do not target block-sparse attention on mobile NPUs. dit.xpu maps block-sparse attention to the NPU’s fixed compute granularity to improve execution efficiency.

Sparse Inference Optimization for DiTs

Prior work has reduced the cost of DiT inference through techniques such as caching [30, 31, 36] and distillation [34]. Sparse attention directly reduces the attention computation at each denoising step by omitting selected token interactions [28, 47, 62]. These methods can be grouped by whether the retained interactions are updated at runtime.

Static sparse attention determines which interactions to retain before inference, using criteria such as spatial proximity or similarity across denoising steps [55]. The resulting computation and memory accesses are predictable, enabling regular execution. However, important interactions vary with the input and denoising state, so a fixed pattern may omit them as sparsity increases.

Dynamic sparse attention such as PISA [27] and SpargeAttn estimates the importance of attention blocks at runtime and selects which interactions to compute. Others adapt the sparsity pattern to differences across heads and layers [58]. This flexibility introduces overhead from pattern selection, data rearrangement, and irregular memory accesses. Existing implementations rely primarily on custom GPU kernels to manage these costs and are difficult to map directly onto mobile NPUs with fixed compute granularity and limited on-chip memory. Trainable variants instead learn or fine-tune a fixed pattern before inference [59, 60, 63] and

permutation-based methods reorder the tokens so that the retained blocks become contiguous [52]. Unlike prior methods, ours aligns static block sparsity with NPU matrix tiles and calibrates sparse outputs using dense attention at selected anchor steps.

Conclusion

We presented DiT.XPU, a sparse inference system for diffusion transformers on mobile heterogeneous SoCs. DiT.XPU combines an NPU-aligned halo mask with periodic dense anchors to compensate for omitted attention contributions, while cross-XPU execution and mask-guided K/V dataflow limit the cost of calibration and data movement. Across five models, three resolutions, and two mobile SoCs, it reduces generation latency relative to the NPU baseline in every evaluated setting, achieving up to 1.56×\times speedup. Ablations further show that anchor calibration is critical to generation quality. Future work will explore input-adaptive anchor scheduling in place of the fixed schedule used here.

References

  1. [1]Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609 (2023).arxiv.org/abs/2309.16609
  2. [2]Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long-Document Transformer. arXiv preprint arXiv:2004.05150 (2020).arxiv.org/abs/2004.05150
  3. [3]Black Forest Labs. 2025. FLUX.2. https://bfl.ai/models/flux-2. Official inference repository: https://github.com/black-forest-labs/flux2.
  4. [7]Le Chen, Dahu Feng, Erhu Feng, Rong Zhao, Yingrui Wang, Yubin Xia, Haibo Chen, and Pinjie Xu. 2025. HeteroLLM: Accelerating Large Language Model Inference on Mobile SoCs with Heterogeneous AI Accelerators. arXiv preprint arXiv:2501.14794 (2025).
  5. [9]Ruichien Chen, Keith G. Mills, Liyao Jiang, Chao Gao, and Di Niu. 2025. Re-tention: Ultra Sparse Visual Generation via Attention Statistical Reshape. arXiv preprint arXiv:2505.22918 (2025).arxiv.org/abs/2505.22918
  6. [10]Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating Long Sequences with Sparse Transformers. arXiv preprint arXiv:1904.10509 (2019).arxiv.org/abs/1904.10509
  7. [13]Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. 2024. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. In Proceedings of the 41st International Conference on Machine Learning (ICML).
  8. [15]ggml-org. 2022. ggml: Tensor Library for Machine Learning. https://github.com/ggml-org/ggml.
  9. [16]Dhruba Ghosh, Hanna Hajishirzi, and Ludwig Schmidt. 2023. GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track.arxiv.org/abs/2310.11513
  10. [17]haozixu. 2025. htp-ops-lib: Self-implemented NN Operators for Qualcomm’s Hexagon NPU. https://github.com/haozixu/htp-ops-lib.
  11. [18]Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  12. [19]Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances in Neural Information Processing Systems (NeurIPS).
  13. [20]Huiqiang Jiang, Yucheng Li, Chengruizhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention. In Advances in Neural Information Processing Systems (NeurIPS), 528481–52515.arxiv.org/abs/2407.02490
  14. [21]Khronos Group. 2018. OpenCL Extension: cl_qcom_ion_host_ptr. https://registry.khronos.org/OpenCL/extensions/qcom/cl_qcom_ion_host_ptr.txt. Version 7.
  15. [22]Khronos Group. 2023. The OpenCL Specification. https://registry.khronos.org/OpenCL/specs/3.0-unified/html/OpenCL_API.html. Version 3.
  16. [23]Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The Efficient Transformer. arXiv preprint arXiv:2001.04451 (2020).arxiv.org/abs/2001.04451
  17. [24]Chris Lattner and Vikram Adve. 2004. LLVM: A Compilation Framework for Lifelong Program Analysis and Transformation. In Proceedings of the International Symposium on Code Generation and Optimization (CGO), 75–86. doi:10.1109/CGO.2004.1281665DOI
  18. [25]leejet. 2024. stable-diffusion.cpp: Stable Diffusion and Flux in pure C/C++. https://github.com/leejet/stable-diffusion.cpp.
  19. [26]Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. 2024. Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation. arXiv preprint arXiv:2402.17245 (2024).
  20. [27]Haopeng Li, Shitong Shao, Wenliang Zhong, Zikai Zhou, Lichen Bai, Hui Xiong, and Zeke Xie. 2026. PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers. arXiv preprint arXiv:2602.01077 (2026).arxiv.org/abs/2602.01077
  21. [28]Xingyang Li, Muyang Li, Tianle Cai, Haocheng Xi, Shuo Yang, Yujun Lin, Lvmin Zhang, Songlin Yang, Jinbo Hu, Kelly Peng, Maneesh Agrawala, Ion Stoica, Kurt Keutzer, and Song Han. 2025. Radial Attention: O(n log n) Sparse Attention with Energy Decay for Long Video Generation. arXiv preprint arXiv:2506.19852 (2025).arxiv.org/abs/2506.19852
  22. [29]Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyuan Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024).arxiv.org/abs/2412.19437
  23. [30]Feng Liu, Shiwei Zhang, Xiaofeng Wang, Yujie Wei, Haonan Qiu, Yuzhong Zhao, Yingya Zhang, Qixiang Ye, and Fang Wan. 2024. Timestep Embedding Tells: It’s Time to Cache for Video Diffusion Model. arXiv preprint arXiv:2411.19108 (2024).arxiv.org/abs/2411.19108
  24. [31]Jiacheng Liu, Chang Zou, Yuanhuiyu Lv, Junjie Chen, and Linfeng Zhang. 2025. From Reusing to Forecasting: Accelerating Diffusion Models with TaylorSeers. arXiv preprint arXiv:2503.06923 (2025).arxiv.org/abs/2503.06923
  25. [33]Eric Mahurin. 2023. Hexagon NPU architecture and on-chip memory. In 2023 IEEE Hot Chips 35 Symposium (HCS), 1–19. doi:10.1109/HCS59251.2023.10254715
  26. [34]Cheng Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. 2023. On Distillation of Guided Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 14297–14306.arxiv.org/abs/2210.03142
  27. [35]William Peebles and Saining Xie. 2023. Scalable Diffusion Models with Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 4195–4205.arxiv.org/abs/2212.09748
  28. [36]Yanjun Qiu, Zhangqiu Yu, Hua Xiao, Xiuquan Hao, Dengfeng Zhang, Yunqing Li, Guiping Du, Wangpeng Zhan, Xiaoyu Zhang, and Song Lin. 2025. Cache-DiT: A Unified PyTorch-Native Inference Engine for Diffusion Transformers. Tech report, https://github.com/vipshop/cache-dit. Introduces DBCache (Dual Block Caching); design doc: https://cache-dit.readthedocs.io/en/latest/user_guide/DBCACHE_DESIGN/.
  29. [37]Qualcomm Technologies, Inc. 2024. FastRPC: Userspace Library for Remote Procedure Calls between the CPU and the DSP. https://github.com/qualcomm/fastrpc.
  30. [38]Qualcomm Technologies, Inc. 2024. Qualcomm AI Engine Direct SDK (QNN). https://www.qualcomm.com/developer/software/qualcomm-ai-engine-direct-sdk.
  31. [39]Qualcomm Technologies, Inc. 2025. Qualcomm Hexagon NPU SDK. https://www.qualcomm.com/developer/software/hexagon-npu-sdk.
  32. [44]The Linux Kernel Community. 2026. Buffer Sharing and Synchronization (dma-buf). Linux Kernel Documentation, https://docs.kernel.org/driver-api/dma-buf.html.
  33. [46]Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. 2023. Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis. arXiv preprint arXiv:2306.09341 (2023).arxiv.org/abs/2306.09341
  34. [47]Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, Jianfei Chen, Ion Stoica, Kurt Keutzer, and Song Han. 2025. Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity. arXiv preprint arXiv:2502.01776 (2025).arxiv.org/abs/2502.01776
  35. [48]Daliang Xu, Hao Zhang, Liming Yang, Ruiqi Liu, Gang Huang, Mengwei Xu, and Xuanzhe Liu. 2024. Fast On-device LLM Inference with NPUs. arXiv preprint arXiv:2407.05858 (2024).arxiv.org/abs/2407.05858
  36. [49]Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. 2023. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. In Advances in Neural Information Processing Systems (NeurIPS).
  37. [50]Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. 2025. XAttention: Block Sparse Attention with Antidiagonal Scoring. arXiv preprint arXiv:2503.16428 (2025).
  38. [51]Zhenliang Xue, Yixin Song, Zeyu Mi, Xinrui Zheng, Yubin Xia, and Haibo Chen. 2024. PowerInfer-2: Fast Large Language Model Inference on a Smartphone. arXiv preprint arXiv:2406.06282 (2024).arxiv.org/abs/2406.06282
  39. [52]Shuo Yang, Haocheng Xi, Yilong Zhao, Muyang Li, Jintao Zhang, Han Cai, et al. 2025. Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation. arXiv preprint arXiv:2505.18875 (2025).arxiv.org/abs/2505.18875
  40. [53]Wangsong Yin, Daliang Xu, Mengwei Xu, Gang Huang, and Xuanhe Lu. 2025. Dynamic Sparse Attention on Mobile SoCs. arXiv preprint arXiv:2508.16703 (2025).
  41. [55]Qinghan Yuan, Hanling Zhang, Pu Lu, Xuefei Ning, Linfeng Zhang, Tianchen Zhao, Shengen Yan, Guohao Dai, and Yu Wang. 2024. DiT-FastAttn: Attention Compression for Diffusion Transformer Models. arXiv preprint arXiv:2406.08552 (2024).
  42. [56]Z-Image Team, Huanqia Cai, Sihan Cao, Ruoyi Du, Peng Gao, Aiming Hao, Steven Hoi, Zhaohui Hou, Shijie Huang, Dengyang Jiang, Yuming Jiang, Xin Jin, Liangchen Li, Zhen Li, Zhong-Yu Li, David Liu, Dongyang Liu, Qilong Wu, Feng Yu, Zechao Zhan, Chi Zhang, Shifeng Zhang, Ruikai Zhou, and Shilin Zhou. 2025. Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer. arXiv preprint arXiv:2511.22699 (2025).arxiv.org/abs/2511.22699
  43. [57]Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big Bird: Transformers for Longer Sequences. In Advances in Neural Information Processing Systems (NeurIPS). 17283–17297.
  44. [58]Hanling Zhang, Rundong Su, Zhihang Yuan, Pengtao Chen, Mingzhu Shen, Yibo Fang, Sheng'en Yang, Guohao Dai, and Yu Wang. 2025. DiT-FastAttnV2: Head-wise Attention Compression for Multi-Modality Diffusion Transformers. arXiv preprint arXiv:2503.22796 (2025).arxiv.org/abs/2503.22796
  45. [59]Jinda Zhang, Kai Jiang, Chendong Xiang, Weiqi Feng, Yuezhou Hu, Haocheng Xi, Jianfei Chen, and Jun Zhu. 2026. SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuning. arXiv preprint arXiv:2602.13515 (2026).
  46. [60]Jintao Zhang, Haoxu Wang, Kai Jiang, Shuo Yang, Kaiwen Zheng, Haocheng Xi, et al. 2025. SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention. arXiv preprint arXiv:2509.24006 (2025).arxiv.org/abs/2509.24006
  47. [62]Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. 2025. SpargeAttn: Accurate Sparse Attention Accelerating Any Model Inference. arXiv preprint arXiv:2502.18137 (2025).
  48. [63]Peiyuan Zhang, Yongqi Chen, Haofeng Huang, Will Lin, Zhengzhong Liu, Ion Stoica, Eric Xing, and Hao Zhang. 2025. VSA: Faster Video Diffusion with Trainable Sparse Attention. arXiv preprint arXiv:2505.13389 (2025).arxiv.org/abs/2505.13389

Paper details

vlnc:W_yr5w6r4s
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents