betaFeedback

WorksComputer Science

TileWeave: Heterogeneous Tiling for Sparse Prefill on Mobile SoCs

Abstract

Smartphone applications increasingly require large language models to process long contexts, whereas high prefill latency and memory demands constrain local inference. Block-sparse attention reduces computation and memory traffic, but its irregular selections conflict with mobile NPUs' preference for large, regular tiles. Efficient execution must exploit cross-query reuse while avoiding redundant computation and preserving selections needed for accuracy. In this paper, we present TileWeave, which partitions sparse attention by this cross-query reuse: highly shared key blocks run in coarse NPU tiles, while low-reuse blocks run on the GPU only for the query blocks that selected them. In addition, we design an NPU-native pooled-key selector that keeps selection inexpensive, and a runtime that publishes GPU partials in the order the NPU consumes them and merges them in its existing output store. On RULER, HeteroSparse matches XAttention's accuracy with Qwen3-1.7B and improves it with Llama-3.2-3B-Instruct; its mean LongBench scores match XAttention as well. On Snapdragon SM8750 and SM8850 phones, it achieves 1.35–3.23×\times faster full-model prefill than XAttention over 1k–16k tokens and up to 1.58×\times faster than dense attention.

Introduction

Large language models (LLMs) support applications ranging from document understanding and question answering to personal assistants. These applications increasingly rely on long contexts to incorporate documents, retrieved information, and extended interaction histories [11, 3]. Demand for these capabilities extends to edge devices, especially smartphones, where local inference keeps personal data on the device and enables operation without a network connection [25]. Compact models such as Qwen3-1.7B Yang et al., [26], Llama-3.2-1B/3B [18], and Phi-3-mini [1] make local deployment feasible, but long contexts remain expensive even for small models. Processing a longer prompt increases the delay before generation begins, while its growing key/value (K/V) cache consumes more of the device’s limited memory [12] (Xu et al. 2024). Dense attention compounds this pressure by computing interactions across the prompt, with quadratic work and substantial memory traffic. Block-sparse attention is therefore an important approach to efficient long-context inference: it selects informative groups of tokens and skips the remaining attention interactions, reducing computation and the associated data movement [11, 13] (Xu et al. 2025). Enabling these savings on mobile hardware is essential for making long-context applications responsive within constrained device resources.

Despite progress in sparse attention, efficient execution on mobile NPUs remains challenging. Existing sparse prefill methods primarily target GPUs, whereas mobile NPU systems emphasize efficient dense computation and heterogeneous placement of model components [4] (Xu et al. 2025; Xu et al. 2024). An NPU obtains high throughput by processing large, regular tiles and reusing their inputs, whereas neighboring queries can select different, scattered key blocks. Executing each small selection separately repeats data staging and limits reuse; combining selections into a large tile improves hardware utilization but computes extra interactions. Discarding selections that do not fit the shared tile can also remove information needed for accurate outputs. Thus, fewer selected blocks do not automatically translate into lower latency: execution must preserve useful selections while exploiting the reuse that the hardware needs.

We introduce TileWeave, a system that addresses such a mismatch through heterogeneous sparse prefill (Figure 1). Our key idea is to assign work according to how widely selected key blocks are shared across queries. The NPU handles shared work in efficient coarse tiles, while the GPU handles query-specific work at fine granularity. Together with inexpensive selection and coordinated execution, this split reduces wasted computation while retaining useful query-specific contributions.

The coarse-tile dilemma comparing fine NPU tiles, coarse tiles, and TileWeave

Figure 1. Thin blue frames mark NPU query tiles: 64 queries in (1) and 256 in (2)–(4). Small tiles preserve selections but repeat K/V staging and coordination, limiting NPU efficiency. Coarse tiles improve reuse, but dropping exceptions risks accuracy (2), while coarse masked tiles compute unwanted interactions (3). TileWeave keeps shared keys on coarse NPU tiles and preserves exceptions through fine GPU execution (4).

In particular, our contributions span the selection policy, work partition, and runtime needed to make this approach practical. First, we identify cross-query reuse as a criterion for choosing execution granularity, supported by measurements of NPU efficiency and the accuracy contribution of query-specific selections. Second, we design an NPU-native selector using compact query and key summaries, coupled with a reuse-based partition that preserves every selected block while assigning it to coarse NPU or fine GPU execution. Third, we build a runtime that overlaps the two engines and combines their partial results with low coordination overhead. We implement TileWeave in llama.cpp (Gerganov & llama.cpp Contributors, [8]) and evaluate it with Qwen3-1.7B and Llama-3.2-3B-Instruct on RULER and LongBench, and on Snapdragon SM8750 and SM8850 phones. TileWeave matches XAttention’s RULER accuracy with Qwen3 and slightly exceeds it with Llama; its LongBench means remain within 0.5 points of XAttention and 0.6 points of dense attention on both models. With Qwen3 over 1k–16k tokens, it achieves 1.351.35–3.23×3.23\times faster full-model prefill than XAttention and up to 1.58×1.58\times over dense attention; attention itself is up to 2.14×2.14\times faster than dense attention, including selection and coordination.

Background and Related Work

Attention and sparsity. Prefill processes prompt tokens in parallel and writes their keys and values to the K/V cache; decode generates one token at a time ([12]). Block-sparse prefill selects pairs of query and key blocks; attention density is the retained fraction of causal query-key pairs. Grouped-query attention (GQA) shares each K/V head among several query heads, allowing the kernel to reuse loaded K/V rows ([2]). FlashAttention computes tiled attention using online-softmax statistics, which can be combined across disjoint key subsets to normalize their outputs jointly ([5, 19]).

Phone NPU architecture. Snapdragon SoCs integrate a Hexagon NPU and an Adreno GPU sharing DRAM. Figure 2 shows Hexagon’s scalar control threads, Hexagon Vector eXtensions (HVX), and Hexagon Matrix Extensions (HMX) (Mahurin, [17]). In attention, HMX executes matrix products, while HVX handles masks, reductions, and softmax. FP16 HMX operates on 32×3232 \times32 hardware tiles in a dedicated memory layout (Hao et al., [9]). These are building blocks for larger query tiles; sparse selection uses its own granularity. HMX operands must be staged from DRAM into software-managed on-chip tightly coupled memory (VTCM), whose buffers are reused as computation advances. Direct memory access (DMA) transfers can

Figure 2. Hexagon NPU architecture and attention mapping. Larger query tiles amortize K/V staging and coordination.

malize their outputs jointly ([5, 19]).

Phone NPU architecture. Snapdragon SoCs integrate a Hexagon NPU and an Adreno GPU sharing DRAM. Figure 2 shows Hexagon’s scalar control threads, Hexagon Vector eXtensions (HVX), and Hexagon Matrix Extensions (HMX) (Mahurin, [17]). In attention, HMX executes matrix products, while HVX handles masks, reductions, and softmax. FP16 HMX operates on 32×3232 \times32 hardware tiles in a dedicated memory layout (Hao et al., [9]). These are building blocks for larger query tiles; sparse selection uses its own granularity. HMX operands must be staged from DRAM into software-managed on-chip tightly coupled memory (VTCM), whose buffers are reused as computation advances. Direct memory access (DMA) transfers can overlap data movement with computation [17]. Coarse query tiles reuse each staged K/V block across more queries and amortize staging and synchronization; fine tiles repeat these steps for smaller amounts of useful work. The GPU supports indexed loads for traversing selected blocks, while the NPU favors large, regular matrix operations [4]. Shared allocations avoid copying the K/V cache between engines, but their separate caches and shared memory bandwidth require explicit publication and coordination.

Sparse attention methods. For prefill, MInference assigns sparse patterns to heads offline and constructs their indices at run time [11]; XAttention estimates block importance from strided antidiagonal scores (Xu et al., 2025). SpargeAttn predicts sparsity from compressed query/key blocks [29]; FlexPrefill probes recent queries [13], while SampleAttention samples query chunks across the sequence [30]. FlashPrefill scores pooled keys and selects blocks relative to each row’s maximum [6]. Learned gates and architectures such as SeerAttention, NSA, and MoBA incorporate sparse selection during training [7, 28] (Lu et al. 2025). For decoding, sparse attention reduces K/V-cache reads through query-aware retrieval or adaptive token selection [22, 15, 14] (Ni et al. 2026).

Heterogeneous mobile inference. Prior NPU inference systems use prompt chunking and heterogeneous scheduling (Xu et al., 2024); PowerInfer-2 distributes neuron clusters across CPU and NPU [25]. HeteroInfer combines layer-level placement with tensor-level GPU/NPU parallelism, using profiled performance to choose weight, activation, or hybrid partitions [4]. ShadowAttn, part of ShadowNPU, estimates token importance with dense INT8 scores on the NPU, then selects keys per query row and evaluates their attention on the CPU or GPU, overlapping stages across heads [27].

Motivation

The cost of sparse attention on an NPU depends on how selected key blocks are shared across queries. Large query tiles improve execution efficiency, but neighboring query blocks can select different keys. Executing their union either introduces additional attention pairs or computes contributions that masking later discards. Cross-query reuse identifies where finer execution can reduce this excess work.

Coarse Execution Couples Query Selections

The query selection block bqb_q, key selection block bkb_k, and NPU query tile BrB_r serve different purposes. The first two de-

termine which query-key pairs can be selected together; the last determines how the kernel shares staging and computation. FlashAttention-style prefill keeps BrB_r queries resident while DMA stages K/V chunks into VTCM, HMX computes matrix products, and HVX applies masks and online softmax. A larger query tile amortizes staging and pipeline coordination over more queries.

Coarse tiles improve NPU efficiency. For NqN_q queries, HqH_q query heads, head dimension DD, and SS selected keys per query, nominal matrix work is 4HqNqSD4H_qN_qSD, independent of BrB_r. At a fixed K/V chunk width CC, however, the kernel visits (Nq/Br)⌈S/C⌉(N_q/B_r)\lceil S/C\rceil chunks per K/V head. Reducing BrB_r from 256 to 64 quadruples these visits. In the equal-work experiment of Table 1, with S=2048S = 2048, 64-, 128-, and 256-query tiles take 4.121, 3.277, and 2.344 ms. The largest tile is 1.76× faster at the same selected work.

Query tile BrB_rS=1024S = 1024S=1536S = 1536S=2048S = 2048
642.62 (1.00×)3.18 (1.00×)4.12 (1.00×)
1282.15 (1.22×)2.53 (1.25×)3.28 (1.26×)
2561.51 (1.73×)1.82 (1.74×)2.34 (1.76×)

Table 1. NPU attention latency in ms (speedup over Br=64B_r = 64). Larger query tiles improve efficiency, giving up to 1.76× speedup at fixed selected work.

Sharing selections changes work or attention. Serving four adjacent 64-query selections from one 256-query tile requires the tile to traverse their union of key blocks. Masking the tile back to each query block’s selection preserves its attended pairs, but the matrix products still compute the unwanted interactions. An unmasked union includes those additional keys in attention; under a fixed density budget, this fill-in also competes with query-specific selections.

Low reuse does not imply low importance. Dropping rarely shared selections reduces work but can remove useful contributions. Table 2 holds the final selector and threshold fixed on Llama-3.2-3B-Instruct over 6,500 RULER 4k prompts. The pool retains blocks requested by at least three of four query blocks; the pool-plus-diagonal control removes only non-diagonal exceptions, preserving sink blocks, head-start tiles, and dense tails. Restoring those query-specific exceptions raises accuracy from 85.57 to 86.32, a 0.75-point gain (paired 95% bootstrap interval: [0.37, 1.14]). The union obtains a similar score but retains 73.13% of causal pairs, against 62.19% with exceptions. These are fixed-configuration policies, not density-matched selectors; each recomputes its pools from its own activations. The result supports preserving low-reuse contributions while reducing their excess coarse work.

Attention policyScoreDensity (%)
Pool + diagonal85.5755.42
Pool + exceptions (ours)86.3262.19
Union86.2973.13

Table 2. Llama-3.2-3B on RULER 4k. Non-diagonal exceptions add 0.75 points; the union retains more pairs for a similar score.

Selection axes have different effects. Coarsening selection itself is another option, whose quality depends on which axis changes. At matched density on Qwen3 RULER 4k, XAttention scores 86.39 with bq=bk=64b_q=b_k=64 and 86.29 with bq=256b_q=256, bk=64b_k=64; coarsening both axes to 256 lowers the score to 84.00. This control supports retaining fine key resolution, but does not establish a universal need for fine query blocks. Our objective is to execute query-specific selections efficiently where they are useful, while retaining coarse execution where selections overlap.

Cross-Query Reuse Determines Excess Work

The extra computation of a coarse tile is unevenly distributed across its key blocks. For one K/V head, let W0,…,W3W_0,\ldots,W_3 be the selections of four adjacent 64-query blocks, UU their union, and c(b)c(b) the number of query blocks selecting key block bb. For fully visible blocks, coarse execution performs four 64×6464 \times64 interactions for every b∈Ub \in U, while the fine selections request only c(b)c(b). The number of interactions beyond those selections is

ΔW=∑b∈U(4−c(b)).(1)\Delta W = \sum_{b \in U} \left(4-c(b)\right). \tag*{(1)}

A fully shared block uses all four coarse interactions; a singly requested block creates three additional interactions. Fine execution replaces four coarse NPU interactions with four requested interactions for the shared block, but only one for the singly requested block.

Cross-query reuse thus assigns execution granularity: high-reuse blocks retain coarse execution, while low-reuse blocks save more coarse work per requested interaction moved. Their quality contribution persists with diagonal attention preserved (Table 2), motivating fine execution that retains these contributions while reducing excess coarse work.

Reuse Partitioning Creates an Execution Opportunity

The mobile GPU provides a second execution path for blocks whose low reuse makes coarse NPU execution expensive. We quantify the opportunity on SM8750 with Llama-3.2-3B-Instruct, timing NPU attention with fixed Q/K/V tensors and group-mean selections, then removing whole key blocks selected by at most rr of the four query blocks.

blocks. Let TallT_{\mathrm{all}} be the original attention time and Tremain(r)T_{\mathrm{remain}}(r) the measured time for the retained blocks. If removed contributions complete at zero execution, transfer, contention, and merge cost, the ideal speedup is

Sideal(r)=TallTremain(r).(2)S_{\mathrm{ideal}}(r)=\frac{T_{\mathrm{all}}}{T_{\mathrm{remain}}(r)}. \tag*{(2)}

The bound holds the NPU kernel, inputs, and tile geometry fixed; the complete attention result still requires the removed contributions.

In this fixed-input study, singly selected blocks account for about 8% of requested block interactions but 22% of coarse NPU interactions, counted before token-level causal masking. Removing blocks selected by at most two query blocks yields ideal NPU speedups of 1.49× at 4k and 1.59× at 8k (Table 3). The measurements cover all 28 layers and all sparse chunks of the first variable-tracking prompt at each length; they exclude selection, host dispatch, dense tails, and every cost of completing the removed work.

Selection count offloaded4k8k
None (baseline)1.00×1.00×
1 of 41.21×1.25×
At most 2 of 41.49×1.59×
At most 3 of 41.90×2.12×

Table 3. Ideal NPU speedup for Llama-3.2-3B assuming removed blocks complete for free. Offloading blocks selected by at most two query blocks gives 1.49×–1.59× speedup.

Realizing this opportunity requires inexpensive selection and a second execution path whose computation, publication, and merge costs remain below the NPU work saved. TileWeave addresses these requirements with an NPU-native selector, a reuse-based partition, and coordinated execution within each attention operator (§ 4).

Selecting Finely, Executing Coarsely

§ 3 motivates preserving useful query-specific contributions while reducing their excess coarse work. TileWeave selects at 64-query granularity and assigns execution by reuse: with c∗=3c^*=3, keys requested by three or four query blocks run in a coarse 256-query NPU tile; the rest run on the GPU only for requesting blocks. Figure 3 outlines inexpensive selection (§ 4.1), reuse-based partitioning (§ 4.2), and coordinated execution (§ 4.3); § 5 evaluates the alternatives.

Figure 3. TileWeave overview. (1) Four sampled query rows and four pooled key vectors per 64-token block produce compact scores; causal softmax, GQA maximum, and relative thresholding with forced sink/diagonal blocks produce wanted lists WiW_i per K/V head. (2) With c∗=3c^* = 3, reuse across four query blocks assigns key blocks to the shared NPU pool PP (c(b)≥3c(b) \ge3, blue) or query-specific GPU exceptions Ei=Wi∖PE_i = W_i \setminus P (1≤c(b)<31 \le c(b) < 3, teal). The masks illustrate the partition; plus signs mark shared-pool fill-in. (3) Coarse NPU tiles and fine GPU blocks execute together, and a fused output store rescales, combines, and normalizes their online-softmax partials into OO.

Fine Selection at NPU Cost

Keep selection within the execution budget. GPU-designed selectors use reorders, copies, and reductions that run as separate NPU vector operators, moving operands between DRAM and VTCM (§ 2). Profiling Qwen3-1.7B on one RULER prompt per length, XAttention spends 154 of its 274 ms of selection at 4k on SM8750 reordering queries and copying keys, against 83 ms on matrix products. MInference’s vertical-slash estimate spends 262 of 435 ms on softmax and element-wise operators (Figure 4). Their per-query-head lists also require separate execution lanes or a denser union: the GQA kernel stages K/V once per group and executes one list per K/V head. Our selector uses compact matrix products and emits that shared list directly.

Figure 4. Qwen3-1.7B NPU selection cost relative to dense attention (totals in ms). TileWeave reduces selection to 6–10% of dense attention’s time.

Compact representatives. One compact matrix product scores 64-query/64-key block pairs using four sampled queries and four pooled keys. For query block ii, the samples are rows 0, 16, 32, 48 of every query head; for K/V block jj, each pool is the mean of 16 consecutive key rows. For K/V head hh, GQA group size GG, query-head offset gg, and sample and pool indices u,v∈{0,1,2,3}u,v \in\{0,1,2,3\}, define

Qihgu=q64i+16u,hG+g,Q_{ihgu}=q_{64i+16u,hG+g},
Kjhv=116∑t=015k64j+16v+t,h,K_{jhv}=\frac{1}{16}\sum_{t=0}^{15} k_{64j+16v+t,h},
zijhguv=QihguTKjhv/D.(3)z_{ijhguv}=Q_{ihgu}^{\mathsf{T}}K_{jhv}/\sqrt{D}. \tag*{(3)}

Q indices are local to the current prefill chunk; K indices address the visible cache.

Pool keys, sample queries. By linearity, zijhguvz_{ijhguv} is the mean logit over all 16 keys in pool vv, so every key contributes; sampling can miss a block’s few target tokens. Neighboring queries attend to similar keys, so four samples retain most of their ranking; pooling queries gains under half a point at more NPU time (§ 5.4.1). A strided Q view and a sixteenth of K avoid query reordering and key copying. Each score costs one DD-dimensional product, versus 16D16D for XAttention’s antidiagonals over the same score-matrix size. Key pools reside beside the K/V cache; each prefill chunk pools only newly written blocks.

Normalize, then take each group’s maximum. Averaging query samples or heads before scoring can collapse distinct query directions. Instead, we normalize each sample and head separately over the key pools it can see, then aggregate their probabilities. Sample uu of query block ii sees every pool of earlier blocks and, inside block ii, the pools w≤uw \le u that start no later than it does; with Viu={(b,w):b<i}∪{(i,w):w≤u},V_{iu} = \{(b,w): b < i\} \cup\{(i,w): w \le u\},

aijhguv=exp⁡(zijhguv)∑(b,w)∈Viuexp⁡(zibhguw),(4)a_{ijhguv} = \frac{\exp(z_{ijhguv})}{\sum_{(b,w)\in V_{iu}} \exp(z_{ibhguw})}, \tag*{(4)}
pijh=max⁡0≤g<G13∑u=03∑v:(j,v)∈Viuaijhguv.p_{ijh} = \max_{0 \le g < G} \frac{1}{3} \sum_{u=0}^{3} \sum_{v:(j,v)\in V_{iu}} a_{ijhguv}.

Summing visible pool probabilities and averaging query samples gives each query head’s block score. Taking the maximum merges their selections: a block enters the K/V head’s list whenever any query head selects it. Averaging the heads instead halves (G=2G = 2) the score of a block that only one head needs, which at 16k drops retrieval blocks (Appendix § 9.§ 1). Ai={0,…,i}A_i = \{0,\ldots,i\} is the set of blocks block ii can reach; the attention executor later enforces exact token causality.

Select the wanted blocks. We compare each probability with the average probability over reachable blocks:

Wi,h={j∈Ai:pijh∣Ai∣>τ}∪{0,di},(5)W_{i,h} = \{j \in A_i : p_{ijh}|A_i| > \tau\} \cup\{0,d_i\}, \tag*{(5)}

where did_i is the chunk-offset diagonal block and block zero is the sink. The threshold lets the number of selected blocks follow the estimated distribution, and one τ\tau serves every layer, head, and length. Sink and diagonal membership are forced after thresholding so that they do not distort the other probabilities.

Execution Granularity from Cross-Query Reuse

Cross-query reuse determines a selected block’s excess coarse work (§ 3.2). For one K/V head, suppress hh and count requests across four adjacent wanted lists W0,…,W3W_0,\ldots,W_3:

U=⋃i=03Wi,c(b)=∑i=031[b∈Wi],U = \bigcup_{i=0}^{3} W_i,\qquad c(b) = \sum_{i=0}^{3} \mathbf{1}[b \in W_i],
P={b∈U:c(b)≥c∗},P = \{b \in U : c(b) \ge c^*\},
Ei=Wi∖P.(6)E_i = W_i \setminus P. \tag*{(6)}

Assignments follow each tile’s current wanted lists, even at fixed shapes and c∗c^*. The NPU evaluates PP for all 256 queries, retaining coarse tiles and K/V reuse; the GPU evaluates EiE_i only for its requesting 64-query block. At c∗=3c^* = 3, exceptions requested by one or two query blocks require one or two GPU interactions before causal masking. Setting c∗=1c^* = 1 executes the union UU entirely on the NPU; larger thresholds reduce pool fill-in and move more selections to the GPU. The NPU-only masked tile executes UU but retains only each WiW_i, computing and discarding fill-in. We use c∗=3c^* = 3; § 5.4.2 compares these executions at a fixed selector and threshold.

The effective key set is P∪EiP \cup E_i, with disjoint partitions and Wi⊆P∪Ei⊆UW_i \subseteq P \cup E_i \subseteq U. Every wanted block is retained, but shared blocks can add neighboring queries’ selections, unlike the naive 64-block baseline’s direct execution of WiW_i.

One Operator on Two Engines

The NPU and GPU compute disjoint key subsets within one layer’s attention over a prefill chunk. The split pays off when GPU execution, coordination, and merging cost less than the coarse work removed. Figure 5 compares the three schedules using historical timing calibration (Appendix § 11.§ 3). For a query and selected key subset SS, an engine emits

Fine-grained schedules on a shared time axis

Figure 5. Fine-grained schedules on a shared time axis (1024 queries, eight K/V heads). Endpoints are measured two-round means from the historical 4k Qwen3-1.7B/SM8750 study using the earlier threshold scorer; selection finishes before t=0t = 0 and other model operators are excluded. Tile intervals are approximate; store/merge marks encode no duration. GPU completion spans 0.37–1.47 ms from a nearby trace, with intermediate groups interpolated. Timing calibration illustrates overlap and relative latency, not a single event trace.

mS=max⁡j∈Ssj,ℓS=∑j∈Sesj−mS,m_S = \max_{j \in S} s_j,\qquad\ell_S = \sum_{j \in S} e^{s_j-m_S},
aS=∑j∈Sesj−mSvj.(7)a_S = \sum_{j \in S} e^{s_j-m_S}v_j. \tag*{(7)}

Let m=max⁡(mN,mG)m = \max(m_N,m_G) and we=eme−mw_e = e^{m_e-m} for engine ee. The output is

O=wNaN+wGaGwNℓN+wGℓG.(8)O = \frac{w_N a_N + w_G a_G}{w_N \ell_N + w_G \ell_G}. \tag*{(8)}

An empty subset contributes zero weight and accumulator. This standard online-softmax composition ([19, 5]) is algebraically exact for the effective selected set, with numerical differences determined by the two implementations; it does not make the sparse policy equivalent to dense attention. The runtime overlaps the engines, merges in the existing store, and publishes shared data explicitly; Appendix § 11.§ 3 records the supporting timing and correctness diagnostics.

Overlap startup and publish in consumption order. GPU work depends on the NPU’s selections. The first eight execution tiles therefore keep the union UU on the NPU to cover GPU startup; later tiles use Equation (6). The accuracy runs include these tiles’ union fill-in in the effective mask. GPU work follows the NPU’s store order in query-block/K/V-head stages: persistent workgroups process count-bounded exception queues and publish each stage when its last workgroup finishes. The NPU waits only for the stage its current store needs, avoiding a whole-operator GPU wait or a separate launch per stage.

Merge in the existing store. HTP store workers combine the GPU’s statistics with the raw HMX accumulator and normalize once, avoiding a separate pass over the NPU output. They handle HMX’s transformed base-2 maximum, and both engines apply the same causal mask. The GPU writes F16 partials in NPU tile-row order for vectorized merging; tile-ahead prefetches bring completed rows in before the store needs them.

Make each publication visible. The GPU imports shared rpcmem allocations through OpenCL without copying the K/V cache, using kernel layouts and an import path derived from the local mllm research fork (mllm Contributors, 2026). An atomic ready word alone does not make new payloads visible through the engines’ separate caches. After observing it, a host relay submits a GPU operation referencing the newly written Q and membership buffers before opening the GPU gate. NPU readers invalidate published rows unless a whole-L2 flush already did so, and prefetch exact row ranges to avoid caching future partials.

Bound the GPU’s share. Both engines share DRAM bandwidth, so excess GPU work can delay the NPU. The reuse threshold limits offload to low-reuse blocks; fully shared selections leave no GPU work. An optional per-tile budget returns excess exceptions to the NPU, masked to their requesting query blocks so the effective mask is unchanged. Incomplete prefill tails use dense attention.

Evaluation

We first compare TileWeave with dense attention, XAttention, and ShadowAttn in accuracy, on RULER and LongBench with two models (§ 5.2), and in performance, as attention and full-model prefill latency on two phones from 1k to 16k tokens (§ 5.3); we then ablate the selector and its execution, including NPU-only executions with the group-maximum selector and threshold held fixed (§ 5.4).

Experimental Setup

Models and platforms. We evaluate Qwen3-1.7B [26] and Llama-3.2-3B-Instruct [18]: 28 layers, eight K/V heads, and head dimension 128, with 16 and 24 query heads. Both use Q4_0 GGU​F weights (Qwen3 with a Q6_K output projection, Llama with a tied Q6_K embedding) and F16 K/V. Accuracy uses CUDA reference implementations with these weights dequantized to F16 and dense greedy decode in every method, so only prefill attention differs. Performance runs use Qwen3-1.7B, and Llama-3.2-3B-Instruct for the comparison of executions, in llama.cpp [8] on Snapdragon SM8750 phones (Hexagon HTpV79 NPU, Adreno 830 GPU), and Qwen3-1.7B’s latency also on SM8850 (HTPV81, Adreno 840).

Methods. We compare dense attention with two training-free sparse methods. XAttention (Xu et al., 2025) keeps, for each query block and query head, the 64-token key blocks with the highest antidiagonal scores up to a cumulative threshold; on the phone it runs on our block-sparse NPU kernel. Our ShadowAttn reimplementation [27] keeps a fixed ratio of each query row’s keys and evaluates NPU–CPU and NPU–GPU execution. The CPU version gathers selected keys; the GPU version applies the per-row selections as a FlashAttention mask. Phone estimation uses F16 HMX products mapped to 8-bit scores, and estimation and attention run sequentially, without the published head-wise pipeline (Appendix § 11.§ 1). TileWeave is pooled-key selection with the NPU-GPU split at c∗=3c^{*} = 3, with the tile-row merge and GPU kernel of § 4.3; the union and the masked tile run the same selector on the NPU alone (§ 4.2). Each sparse method has one setting per model, kept at every length and benchmark: XAttention’s threshold, ShadowAttn’s ratio, and our τ\tau are 0.9268, 0.486, and 0.4277 for Qwen3-1.7B, and 0.8736, 0.488, and 0.5582 for Llama-3.2-3B-Instruct.

Benchmarks. RULER ([10]) has 13 tasks with 500 prompts each, at 4k, 8k, and 16k tokens (Qwen3-1.7B also at 1k and 2k, Appendix § 8); we report the equal-task mean score. LongBench v1 ([3]) adds single-document QA (Qasper, MultiFieldQA-en/MFQA), multi-document QA (HotpotQA/HQA, 2Wiki-MultihopQA/2WQA, MuSiQue/MSQ), and query-based summarization (QMSum), 200 prompts each (150 for MultiFieldQA-en) truncated to 16k tokens, with the official metrics.

Measurement protocol. Phone prefill latency uses five RULER variable-tracking prompts per length, with one warmup followed by two timed prefills per prompt from an empty cache; ShadowAttn uses a 1k-token warmup and one timed prefill per prompt. We run two or three interleaved rounds and report the median across rounds of the mean per-prompt median latency. Attention latency directly times each chunk’s attention on captured inputs, including selection and coordination, with one warmup prompt and one timed run per chunk (ShadowAttn CPU: two prompts at 16k). Attention and full-model prefill are measured in separate runs (Appendix § 11.§ 2). All phone runs use two CPU workers pinned to the two fastest cores (6–7, affinity mask 0xc0).

Accuracy

RULER. TileWeave matches XAttention with Qwen3-1.7B from 1k to 16k tokens and slightly exceeds it with Llama-3.2-3B-Instruct at every tested length (Table 4, Appendix § 8). Relative to dense attention, its mean scores decrease by at most 2.3 points with Qwen3 and 0.5 points with Llama. ShadowAttn’s per-row selection generally achieves higher accuracy, with a notable advantage on Qwen3’s common-words extraction task, where both block-sparse methods lose accuracy. At 16k, however, TileWeave slightly exceeds ShadowAttn with Qwen3. The group-maximum rule’s contribution to long-context retrieval is examined in Appendix § 9.§ 1.

MethodQwen3-1.7B 4kQwen3-1.7B 8kQwen3-1.7B 16kLlama-3.2-3B-Instruct 4kLlama-3.2-3B-Instruct 8kLlama-3.2-3B-Instruct 16k
Dense87.1279.4870.8686.7783.3980.61
XAttention86.3177.1670.0886.0082.6480.13
ShadowAttn86.9578.5169.5486.7283.4980.71
TileWeave86.3777.1969.9986.3282.9880.19

Table 4. RULER mean accuracy. TileWeave matches XAttention on Qwen3-1.7B and is slightly more accurate on Llama-3.2-3B-Instruct at every tested length.

LongBench. Across the question-answering and summarization tasks, TileWeave’s mean scores remain within 0.5 points of XAttention and 0.6 points of dense attention with both models (Table 5). It slightly trails XAttention with Qwen3 and slightly exceeds it with Llama, while ShadowAttn achieves higher means with both models. These comparisons use the same method settings as RULER, without retuning for individual datasets.

MethodQasperMFQAHQA2WQAMSQQMSumAvg.QasperMFQAHQA2WQAMSQQMSumAvg.
Dense16.9842.2335.5729.3813.7522.8226.7937.7150.2550.0938.2022.1924.1337.10
XAttention18.1342.3536.1528.3314.7322.6327.0637.3250.2448.1536.1922.8623.4736.37
ShadowAttn18.8642.3635.3828.5314.4322.7327.0539.3951.0450.2038.7521.8924.1037.56
TileWeave17.4643.3635.2428.2713.0322.1826.5938.3648.8150.4137.2021.0323.4136.54

Table 5. LongBench accuracy (F1; ROUGE-L for QMSum). TileWeave’s mean stays within 0.6 points of dense attention on both models.

Performance

Prefill. With Qwen3-1.7B on SM8750, TileWeave matches dense attention at 1k tokens and is faster from 2k on: 1.05×, 1.17×, 1.37×, and 1.58× at 2k, 4k, 8k, and 16k (Figure 7). The gain grows with length, consistent with the quadratic growth of attention work. XAttention is 1.33–1.77× slower than dense attention at every length, so TileWeave is 1.35–2.79× faster than it, and the gap widens with length. ShadowAttn’s CPU version is slower still: its prefill takes 1.04 s at 1k and 539 s at 16k, 2.2–30.8× dense attention’s, so TileWeave is 2.2–48.6× faster than it. On SM8850 TileWeave is 1.03×, 1.08×, 1.17×, 1.29×, and 1.44× faster than dense attention from 1k to 16k, 1.41–3.23× faster than XAttention, and 3.2–63.2× faster than ShadowAttn’s CPU version. The GPU version is faster than the CPU version from 2k onward, but TileWeave remains 2.3–12.6× faster on SM8750 and 2.0–14.0× on SM8850.

Qwen3-1.7B full-model prefill latency on SM8750 and SM8850

Figure 7. Qwen3-1.7B full-model prefill latency over 1k–16k tokens. TileWeave achieves speedups of up to 1.58× over dense attention, 3.23× over XAttention, 63.23× over ShadowAttn (CPU), and 13.98× over ShadowAttn (GPU).

Attention. Figure 6 reports directly measured attention latency. At 1k its selection costs about what the sparse kernels save, so TileWeave’s attention matches dense attention; at 2k, 4k, 8k, and 16k it is 1.14×, 1.51×, 1.92×, and 2.14× faster. On SM8850 it takes 9% longer at 1k and is 1.10×, 1.35×, 1.61×, and 1.88× faster from 2k to 16k. XAttention’s attention is 2.2–3.0× slower than dense attention on SM8750 and 3.1–3.4× on SM8850, although it executes fewer blocks than TileWeave (Appendix Table 13). In a profiled prefill its selection is only 12–18% of that time and about three times TileWeave’s; the rest runs its per-head masks as 64-query tiles in two GQA lanes, the small tiles that § 3.1 shows are inefficient on the NPU. ShadowAttn’s CPU attention takes 5.7–46× as long as dense attention and 5.6–98× as long as TileWeave’s on SM8750, and 6.9–69× and 6.3–130× on SM8850.

Figure 6. Qwen3-1.7B attention latency, including selection and coordination. TileWeave achieves speedups of up to 2.14× over dense attention, 5.82× over XAttention, 130.14× over ShadowAttn (CPU), and 24.53× over ShadowAttn (GPU).

Ablation Study

We ablate the two parts of our design: the selector (§ 4.1) and the heterogeneous split (§ 4.2–§ 4.3).

Selector

Pooled keys, sampled queries. Table 6 changes one part of the selector at a time on Qwen3-1.7B RULER 4k, keeping the group maximum and c∗=3c^{*} = 3 and calibrating each variant’s threshold to TileWeave’s density. Sampling keys instead of pooling them, each representative takes the average of two of its 16 keys, lowers the score by 0.56, most of it on UUID multi-key and multi-value retrieval (84.60 against 89.00 and 85.70 against 88.30), where two sampled keys of 16 miss the target tokens; in full-model prefill the pooled keys add only 0.5% to prefill time at 4k and 8k. Pooling the four query samples as well gains 0.43 points, nearly all on UUID multi-key retrieval (92.80 against 89.00), and XAttention’s antidagonal scores in the same range 0.30. Both cost more NPU time: pooling reads all 64 query rows of a block instead of four, which takes 1.5× the selection’s NPU time and 3% more prefill time at 4k and 8k, and an antidiagonal score takes 16 products instead of one, so the selector keeps sampled queries and pooled keys.

Table 6. Selector ablation at matched density (Qwen3-1.7B, RULER 4k). Pooled keys improve accuracy, while pooled queries and antidiagonal scores trade additional NPU time for small gains.

SelectorScoreDensity (%)
TileWeave86.3757.30
Keys sampled, not pooled85.8157.31
Queries pooled as well86.8057.21
XAttention’s antidiagonal scores86.6757.26
Dense attention87.12100.00

Table 6. Selector ablation at matched density (Qwen3-1.7B, RULER 4k). Pooled keys improve accuracy, while pooled queries and antidiagonal scores trade additional NPU time for small gains.

Selection cost. For Qwen3-1.7B on SM8750, we fix execution to naive 64-block attention with one list per K/V head. Replacing XAttention’s scorer with ours makes attention, selection included, 1.47×/1.46× faster at 8k/16k; it is 2.85×/2.83× faster than XAttention with its per-head masks, and the scorer makes the split adds 1.52×/1.63× (Table 7). End to end, the scorer makes prefill 1.20×/1.27× faster and the split another 1.24×/1.34×. In a profiled prefill at 4k/8k with each group’s mean, ours costs 3.4/10.0 ms of NPU time per layer over a prompt, against 9.5/27.8 ms for XAttention’s scoring and cumulative selection.

Table 7. Qwen3-1.7B scorer and execution ablations; gain is TileWeave’s speedup over each row. The scorer and heterogeneous split make comparable contributions to the latency reduction.

ScorerExecution8k prompts Time8k prompts Gain16k prompts Time16k prompts Gain
Attention (ms)
XAttentionnaive, per K/V head3,7622.24×12,9782.37×
XAttentionnaive, per head7,2634.33×25,2074.61×
Pooled keysnaive2,5521.52×8,8921.63×
Pooled keyssplit, c∗=2c^{*}=21,8261.09×6,0071.10×
Pooled keyssplit, c∗=3c^{*}=31,677–5,470–
Prefill (s)
XAttentionnaive, per K/V head6.611.48×18.781.69×
XAttentionnaive, per head10.122.27×30.952.79×
Pooled keysnaive5.501.24×14.841.34×
Pooled keyssplit, c∗=2c^{*}=24.611.03×11.661.05×
Pooled keyssplit, c∗=3c^{*}=34.45–11.10–

Table 7. Qwen3-1.7B scorer and execution ablations; gain is TileWeave’s speedup over each row. The scorer and heterogeneous split make comparable contributions to the latency reduction.

Heterogeneous Split

Comparable accuracy at different densities. We hold the group-maximum selector and threshold fixed across the three executions on Llama-3.2-3B-Instruct and measure latency on SM8750 (Table 8). The masked tile retains only selected pairs (51.1% causal-pair density), TileWeave also retains fill-in from its shared pool and head start (58.0%), and the union retains all blocks selected within each tile (70.1%). On RULER 8k, TileWeave scores 0.18 points below the union and 0.34 above the masked tile; extra keys improve variable tracking but reduce common-words extraction accuracy. An earlier group-mean control with independently calibrated thresholds compares executions at matched density (Appendix § 10).

Table 8. Llama-3.2-3B executions at a fixed selector and threshold. TileWeave gives 1.11–1.18× faster prefill with competitive RULER 8k accuracy.

ExecutionRULER 8k ScoreRULER 8k Dens.Attention (ms) 8kAttention (ms) 16kPrefill (s) 8kPrefill (s) 16k
TileWeave82.9858.02,3757,8646.7716.60
Masked tile82.6451.13,06610,5377.6319.63
Union83.1670.12,94910,0927.5219.18
Speedup over the masked tile1.29×1.34×1.13×1.18×
Speedup over the union1.24×1.28×1.11×1.16×

Table 8. Llama-3.2-3B executions at a fixed selector and threshold. TileWeave gives 1.11–1.18× faster prefill with competitive RULER 8k accuracy.

Full-model prefill gains. At 8k and 16k, TileWeave accelerates full-model prefill by 1.11–1.16× over the union and 1.13–1.18× over the masked tile; the corresponding attention speedups are 1.24–1.28× and 1.29–1.34×. Both NPU-only paths compute every block selected by any of a tile’s four query blocks, with the masked tile discarding fill-in. TileWeave confines this coarse work to the shared pool and executes low-reuse blocks concurrently on the GPU, preserving the benefit through full-model prefill.

Reuse threshold. At each model’s single setting, c∗=3c^{*}=3 executes fewer blocks than c∗=2c^{*}=2 (Appendix Table 13) and makes attention 1.09× and 1.10× faster at 8k and 16k and prefill 1.03× and 1.05× (Table 7), while RULER changes by −0.51 to −0.12 points with Qwen3-1.7B and −0.25 to +0.15 with Llama-3.2-3B-Instruct, and LongBench by at most 0.06.

Conclusion

TileWeave uses cross-query reuse to determine the execution granularity of sparse attention. Highly shared key blocks run in efficient coarse NPU tiles, while low-reuse blocks run on the GPU only for the query blocks that select them. An NPU-native selector and coordinated runtime make this partition practical. Across two models, TileWeave maintains accuracy comparable to XAttention on RULER and LongBench. With Qwen3-1.7B on two Snapdragon platforms, it accelerates full-model prefill by 1.35–3.23× over XAttention and up to 1.58× over dense attention. With Llama-3.2-3B-Instruct at 8k/16k and a fixed selector and threshold, attention is 1.24–1.28× faster than the NPU-only union and 1.29–1.34× faster than masked tiles, with comparable RULER 8k accuracy. Cross-query reuse reduces excess work while retaining efficient NPU execution.

AI USAGE DISCLOSURE

We used OpenAI Codex and Anthropic Claude Code as generative AI assistants during research development and manuscript preparation. Both tools assisted with code generation and writing; Codex also assisted with literature search, experimental design, experiment execution, and result analysis. Their use was iterative and included substantive code and prose generation as well as revision.

Ideation and literature. Codex helped refine the motivation and reasoning about selection granularity and cross-query reuse, compare alternative explanations of results, and propose follow-up controls. For example, it assisted with designing the diagonal-preserving exception ablation reported in Table 2. It also helped locate and summarize related work, compare sparse-attention methods, and check citation metadata against primary sources.

Code and experiments. Codex and Claude Code assisted with generating and revising implementation and evaluation code. Codex’s work included sparse-attention evaluation code, experiment-launch and monitoring scripts, result collection, numerical and mask validation, and debugging. Through command-line tools, it also launched and monitored authorized GPU and phone experiments and checked saved outputs. Validation included comparisons with reference computations, reproduction of saved baseline predictions, and checks of input, model, and configuration identities, as documented in the experimental protocols.

Data and analysis. Benchmark prompts and reference answers came from the documented RULER and LongBench datasets or their prescribed generators. The evaluated language models generated the predictions used for accuracy scoring; latency measurements came from hardware executions. Codex assisted with data-preparation and analysis scripts, rescoring predictions, computing aggregate scores, densities, speedups, and paired confidence intervals, and interpreting these results. Reported numerical results were derived from saved experiment outputs, rather than generated as prose by the assistants.

Writing and figures. Both tools assisted with drafting and revising manuscript text. Codex also helped organize and condense sections, revise captions and technical explanations, and write and modify plotting and diagram-generation scripts. This included performance charts, method and pipeline illustrations, editable figure exports, and LaTeX layout and PDF checks. Codex drafted this disclosure from the recorded workflow and author-provided information about tool use. The authors directed the work and are responsible for the implementation, experimental evidence, citations, technical claims, and final manuscript.

Per-Task Comparison

Tables 11 and 12 expand the RULER 4k comparison of Table 4 for each model, Table 9 its 16k comparison for Qwen3, and Table 10 adds 1k and 2k. Each task contains the same 500 prompts in all methods, following the accuracy protocol in § 5.1. The 1k/2k means cover 12 tasks, excluding qa_2 because its documents do not fit these lengths. Density counts retained causal attention pairs after the complete prefill policy, not just the selector’s wanted mask; the macro density is the equal-task mean. One setting per method leaves task densities free: at 4k they differ from TileWeave’s by up to 7.2 points for XAttention and 15.1 for ShadowAttn with Qwen3, so task-level score differences cannot be attributed to ranking alone. At 16k, with the same threshold and c∗=3c^{*}=3, the group mean keeps fewer pairs than the group maximum (36.8% against 43.8%) and scores 68.41 against 69.99, losing most on multi-key and multi-value retrieval (43.0 against 49.2 on niah_multikey_2, 81.4 against 83.1 on niah_multivalue) and on question answering (41.0 against 44.2 on qa_1).

TaskDense ScoreDense DensityXAttention ScoreXAttention DensityTileWeave ScoreTileWeave DensityGroup mean ScoreGroup mean Density
cwe1.66100.02.1039.22.4448.52.8040.6
fwe91.67100.093.5344.793.0754.293.4045.9
niah_multikey_196.80100.093.4032.994.4040.192.6033.9
niah_multikey_250.40100.051.8045.749.2053.643.0046.0
niah_multikey_337.20100.032.2042.334.0050.930.8043.3
niah_multiquery97.15100.096.1533.096.5040.395.5534.0
niah_multivalue86.20100.084.2032.283.0539.681.3533.3
niah_single_1100.00100.0100.0038.1100.0044.9100.0037.0
niah_single_2100.00100.0100.0032.0100.0039.3100.0033.1
niah_single_3100.00100.099.8033.3100.0040.4100.0034.2
qa_147.00100.046.4029.444.2036.141.0030.2
qa_234.80100.033.0029.934.0036.531.4030.4
vt78.32100.078.5237.978.9644.277.4436.4

Table 9. RULER 16k task scores and density (%) for Qwen3-1.7B. Taking the GQA-group maximum improves retrieval over the group mean at the same threshold.

Method1k Score1k Dens.2k Score2k Dens.
Dense97.3010091.13100
XAttention96.8886.390.7368.9
TileWeave97.0082.190.6969.2

Table 10. RULER 1k/2k mean scores and density (%) for Qwen3-1.7B. TileWeave remains within 0.12 points of XAttention.

Additional Selector Analysis

Sharing One List across a Group’s Query Heads

The GQA kernel runs one list per K/V head, so the query heads of a group must agree on it; we compare the two rules at c∗=2c^{*}=2. Averaging their probabilities halves the score of a block that only one head needs, and at 16k this loses retrieval: on the 500 RULER 16k multi-key prompts with distractor needles (niah_multikey_2), the group mean scores 44.0 against XAttention’s 51.8, while one list per query head scores 50.2 and the group maximum 51.0. Swapping in XAttention’s scorer (43.6) or its cumulative rule (46.0) changes little, so the gap comes from the sharing, not from the estimate. Over all 6500 prompts the maximum adds 1.46 points at 16k (70.50 against 69.04, at 46.9% against 39.6% of pairs) and 0.07 at 4k (86.49 against 86.42); with Llama3.2-3B-Instruct it adds 0.53 and 0.24 points on RULER 4k and 8k, and on LongBench it adds 0.25 points with Llama3.2-3B-Instruct and 0.08 with Qwen3-1.7B. It keeps a list per K/V head, so the NPU runs it with the same kernel; its extra blocks cost 8%, 12%, and 15% attention time at 4k, 8k, and 16k on SM8750, and 3%, 5%, and 7% prefill.

Density at One Setting

Each method keeps the setting calibrated at RULER 4k, so its density follows its own rule at other lengths (Table 13). Indented rows show our c∗=2c^{*}=2 controls with the group maximum and group mean. Both block methods keep a smaller share of longer prompts, TileWeave more slowly: from 1k to 16k our share of causal pairs falls from 82.1% to 43.8% and XAttention’s from 86.3% to 36.2%. Our threshold was calibrated with the group mean at c∗=2c^{*}=2, which keeps 53.2% at 4k; the group maximum keeps 60.4% at the same threshold, and 57.3% with c∗=3c^{*}=3. On the phone XAttention executes fewer blocks than TileWeave at 1k–8k, yet its attention is 2.2–3.0× slower than dense attention (§ 5.3), because it runs them as per-head 64-query tiles.

Method1k2k4k8k16kLB
RULERTileWeave82.169.257.349.643.841.4
c∗=2c^{*}=284.772.560.452.746.944.8
group mean80.966.153.245.239.638.2
XAttention86.368.953.443.636.234.4
ShadowAttn––53.451.249.850.1
PhoneTileWeave76.867.756.848.543.7–
c∗=2c^{*}=284.973.360.751.646.3–
group mean80.165.651.842.4––
XAttention73.961.449.942.136.5–

Table 13. Density (%) of retained causal pairs and executed phone blocks for Qwen3-1.7B (LB: LongBench). Both block methods become sparser on longer prompts, TileWeave more slowly.

Executions at One Density

Before the group maximum, we compared the three executions with each GQA group’s mean selections and gave each execution its own threshold for the same 53% density (Table 14; Llama-3.2-3B-Instruct, attention of the sparse chunks on SM8750 with the optimized split operator of that study). At that density the union scores 85.12 on RULER 4k, a point below TileWeave with c∗=3c^{*}=3 (86.10) and the masked tile (86.15), which preserve each query block’s own selection; common-words extraction carries most of the loss, 23.8 for the union against 32.3 with c∗=3c^{*}=3 and 33.1 for dense attention. With c∗=2c^{*}=2, every block that two of the four query blocks request stays in the shared tile, so it carries much of the same fill-in: 85.64, and 24.4 on common words. TileWeave is the fastest of the three: the union takes 1.14× (4k) and 1.12× (8k) its attention time, and the masked tile, which executes every block its four query blocks select (69% and 65% of causal blocks at 4k and 8k), 1.45× and 1.54×.

ExecutionRULER 4k ScoreRULER 4k Dens.Attention (ms) 4kAttention (ms) 8k
TileWeave, c∗=3c^{*}=386.1053.13741277
Masked tile86.1553.1542 (1.45)1973 (1.54)
Union85.1253.2426 (1.14)1426 (1.12)

Table 14. Llama-3.2-3B at matched density: TileWeave matches masked-tile accuracy and reduces attention latency. Parentheses show latency relative to TileWeave.

Implementation and Measurement Details

ShadowAttn Execution

We reimplement ShadowAttn’s per-query-row selection with one retention ratio per model, a policy also evaluated in its published ablation [27]. The accuracy reference quantizes Q/K to INT8 and ranks their integer products. For phone timing, the NPU instead computes F16 HMX products and emits F32 scores; the CPU or GPU maps each row’s scores to 8 bits before selecting its highest-scoring keys, with ties broken by key index. These estimation paths share the selection rule but need not select identical keys. Estimation precedes selection and attention in the model graph; we do not reproduce the published pipeline across heads.

The CPU version gathers only selected K/V rows and computes attention with F16 products and F32 accumulation. The GPU version uses OpenCL selection followed by masked FlashAttention with F16 K/V and F32 accumulation: it computes causal tiles and masks unselected pairs, retaining dense tile computation. On SM8850, a diagnostic using the first variable-tracking prompt per length found that only 0.27%, 0.45%, and 1.05% of causal 64×3264 \times32 query-key tiles contain no selected pairs at 4k, 8k, and 16k. A prototype GPU kernel that gathers individual selected tokens had 9.3× and 17.0× higher attention-kernel latency than the masked kernel at 4k and 8k in this diagnostic. We therefore report the faster masked implementation; the results characterize our implementations under the stated model, precision, and resource settings.

Attention Timing

We directly time each chunk’s attention graph across all layers using captured queries and resident K/V, including estimation, selection, attention, and device coordination and merging. Q/K/V projections, cache writes, and the other model operators are outside this timer. Each prompt’s attention latency is the sum over its chunks; ShadowAttn’s inputs are captured during dense prefill. Full-model prefill is timed separately with the complete model graph, so Figure 6 and Figure 7 are separate measurements rather than an additive latency breakdown. For example, on SM8750, ShadowAttn’s CPU attention at 4k takes 18.42 s when measured separately, while its full-model prefill takes 15.36 s. Same-device checks find that CPU attention chunks with 2–4k cached keys take 1.3–1.5× longer when measured separately than within prefill; later chunks agree within 1%. We retain both measurements and use the full-model prefill timings for end-to-end speedup claims.

Runtime Timing and Publication Diagnostics

These diagnostics motivated the runtime in § 4.3. They come from historical execution studies and isolated harnesses, rather than the current full-model performance cohort; their costs are not additive.

Pipeline calibration. Figure 5 covers 1024 queries and eight K/V heads, with endpoints set to two-round means from the historical 4k Qwen3-1.7B/SM8750 study using the earlier threshold scorer. Selection finishes before the illustrated interval; other model operators are excluded. Tile intervals are approximate, and store/merge marks encode no duration. GPU completion spans 0.37–1.47 ms from a nearby trace, with intermediate groups interpolated. The figure illustrates overlap and relative latency rather than a single event trace.

TaskDenseXAttentionShadowAttnTileWeave
ScoreDensityScoreDensityScoreDensityScoreDensity
cwe95.78100.093.2251.295.0851.994.4857.4
fwe64.13100.064.7358.964.6752.164.2066.1
niah_multikey_199.80100.099.8054.699.2055.899.8057.1
niah_multikey_285.00100.083.6062.084.6052.384.2067.4
niah_multikey_392.20100.088.6055.391.0050.989.0060.7
niah_multiquery99.65100.099.8051.099.6053.599.7054.3
niah_multivalue90.50100.087.6552.690.0554.588.3055.6
niah_single_1100.00100.0100.0050.7100.0051.6100.0054.1
niah_single_2100.00100.0100.0055.7100.0056.6100.0057.7
niah_single_399.80100.099.8052.999.8055.199.8055.3
qa_168.60100.068.4050.869.4053.667.2053.1
qa_243.00100.041.6044.343.0052.540.8047.1
vt94.04100.094.8854.693.9654.095.2859.0

Table 11. RULER 4k task scores and density (%) for Qwen3-1.7B. TileWeave gains on multi-key retrieval but loses on question answering relative to XAttention.

TaskDenseXAttentionShadowAttnTileWeave
ScoreDensityScoreDensityScoreDensityScoreDensity
cwe33.10100.030.7652.128.9051.527.8463.0
fwe82.40100.086.4055.582.3352.384.8066.4
niah_multikey_199.20100.099.4051.099.4052.899.6060.4
niah_multikey_299.80100.099.6059.599.8051.699.8071.1
niah_multikey_399.20100.095.8055.099.0050.399.0065.1
niah_multiquery98.35100.097.6552.398.0053.797.7061.2
niah_multivalue99.55100.098.9551.099.7052.899.3060.2
niah_single_1100.00100.0100.0047.6100.0051.7100.0056.7
niah_single_2100.00100.0100.0057.0100.0056.899.8064.4
niah_single_399.80100.099.8052.799.8054.299.8061.2
qa_178.40100.079.6053.779.0054.379.4061.2
qa_252.80100.051.0048.855.2054.450.8055.3
vt85.44100.079.0451.786.2054.384.2862.0

Table 12. RULER 4k task scores and density (%) for Llama-3.2-3B-Instruct. TileWeave improves variable tracking and multi-key retrieval over XAttention.

Startup and progressive completion. The GPU’s first partial is ready 0.35–0.48 ms after the NPU publishes the operator’s inputs. A conventional launch over all exceptions finishes the first of its 32 stages only after 2.5 ms of a 3.9 ms execution; one launch per stage instead adds about 65 μ\mus at each boundary. Persistent workgroups publish successive stages about 27–30 μ\mus apart in the recorded trace. These observations motivate the eight-tile NPU head start and progressive publication in consumption order.

Merge placement and memory traffic. A separate vector pass over the NPU output adds 890–920 μ\mus per operator, exceeding the historical split’s 0.67–0.80 ms saving over naive 64-block attention at 4k and 8k. In the isolated fold harness, changing the HMX normalization adds 290–350 μ\mus, while merging in the existing store with tile-ahead prefetch adds 113–186 μ\mus; omitting that prefetch raises the overhead to about 980 μ\mus. The GPU writes F16 partials to halve their transfer size relative to F32, reads K/V as 16-byte texels, and reads Q through an image. In a separate 8k profile, GPU exceptions take about 2.2 ms per operator and expose 218 μ\mus of NPU wait; NPU matrix multiplication is 7% slower with concurrent GPU traffic.

Payload visibility. With only an atomic ready word, pre-enqueued GPU work read the previous layer’s queries from a reused buffer in 10–20% of checked rows; fences, atomic loads, and added delays did not prevent it. The host relay’s buffer-referencing submission makes the GPU observe current Q and membership data in the tested driver path. NPU readers invalidate published rows unless a whole-L2 flush already did so, and prefetch exact row ranges so that a future stage is not cached before the GPU writes it. These mechanisms describe the tested shared-buffer implementation, not a portable guarantee for arbitrary OpenCL aliases.

References

  1. [1]Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., A., et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. URL https://arxiv.org/abs/2404.14219.
  2. [2]Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., and Sanghai, S. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4895–4901, 2023. URL https://aclanthology.org/2023.emnlp-main.298/.
  3. [3]Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J., and Li, J. LongBench: A bilingual, multi-task benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023. URL https://arxiv.org/abs/2308.14508.
  4. [4]Chen, L., Feng, D., Feng, E., Wang, Y., Zhao, R., Xia, Y., Xu, P., and Chen, H. Characterizing mobile SoC for accelerating heterogeneous LLM inference. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, pp. 359–374, 2025. doi: 10.1145/3731569.3764808. URL https://doi.org/10.1145/3731569.3764808.
  5. [5]Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. arXiv preprint arXiv:2205.14135, 2022. URL https://arxiv.org/abs/2205.14135.
  6. [6]Fan, Q., Huang, H., Wu, Z., Wang, J., Wang, B., and He, R. FlashPrefill: Instantaneous pattern discovery and thresholding for ultra-fast long-context prefilling. arXiv preprint arXiv:2603.06199, 2026. URL https://arxiv.org/abs/2603.06199.
  7. [7]Gao, Y., Zeng, Z., Du, D., Cao, S., Zhou, P., Qi, J., Lai, J., So, H. K.-H., Cao, T., Yang, F., et al. SEER Attention: Learning intrinsic sparse attention in your LLMs. arXiv preprint arXiv:2410.13276, 2024. URL https://arxiv.org/abs/2410.13276.
  8. [8]Gerganov, G. and llama.cpp Contributors. llama.cpp: LLM inference in C/C++. Software repository, 2023. URL https://github.com/ggerganov/llama.cpp.
  9. [9]Hao, Z., Wei, J., Wang, T., Huang, M., Jiang, H., Jiang, S., Cao, T., and Ren, J. Scaling LLM test-time compute with mobile NPU on smartphones. In Proceedings of the 21st European Conference on Computer Systems, pp. 2157–2172, 2026. doi: 10.1145/3767295.3769382. URL https://arxiv.org/abs/2509.23324.
  10. [10]Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg, B. RULER: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024. URL https://arxiv.org/abs/2404.06654.
  11. [11]Jiang, H., Li, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L. MLInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention. arXiv preprint arXiv:2407.02490, 2024. URL https://arxiv.org/abs/2407.02490.
  12. [12]Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611–626, 2023. doi: 10.1145/3600006.3613165. URL https://doi.org/10.1145/3600006.3613165.
  13. [13]Lai, X., Lu, J., Luo, Y., Ma, Y., and Zhou, X. FlexPrefill: A context-aware sparse attention mechanism for efficient long-sequence inference. In International Conference on Learning Representations, 2025. URL https://arxiv.org/abs/2502.20766.
  14. [14]Lin, C., Tang, J., Yang, S., Wang, H., Tang, T., Tian, B., Stoica, I., Han, S., and Gao, M. Twilight: Adaptive attention sparsity with hierarchical top-p pruning. arXiv preprint arXiv:2502.02770, 2025. URL https://arxiv.org/abs/2502.02770.
  15. [15]Liu, G., Li, C., Zhao, J., Zhang, C., and Guo, M. ClusterKV: Manipulating LLM KV cache in semantic space for recallable compression. In 2025 62nd ACM/IEEE Design Automation Conference (DAC), pp. 1–7, 2025. doi: 10.1109/DAC63849.2025.11132479. URL https://doi.org/10.1109/DAC63849.2025.11132479.
  16. [17]Mahurin, E. Qualcomm Hexagon NPU. Qualcomm Technologies, Hot Chips presentation, 2023.
  17. [18]Meta. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models. Meta AI Blog, September 2024. URL https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/.
  18. [19]Milakov, M. and Gimelshein, N. Online normalizer calculation for softmax. arXiv preprint arXiv:1805.02867, 2018. URL https://arxiv.org/abs/1805.02867.
  19. [22]Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasikci, B., and Han, S. Quest: Query-aware sparsity for efficient long-context LLM inference. arXiv preprint arXiv:2406.10774, 2024. URL https://arxiv.org/abs/2406.10774.
  20. [25]Xue, Z., Song, Y., Mi, Z., Zheng, X., Xia, Y., and Chen, H. PowerInfer-2: Fast large language model inference on a smartphone. arXiv preprint arXiv:2406.06282, 2024. URL https://arxiv.org/abs/2406.06282.
  21. [26]Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. URL https://arxiv.org/abs/2505.09388.
  22. [27]Yin, W., Xu, D., Xu, M., Huang, G., and Liu, X. Shad-owNPU: System and algorithm co-design for NPU-centric on-device LLM inference. In Proceedings of the 24th Annual International Conference on Mobile Systems, Applications and Services (MobiSys), pp. 294–308, 2026. doi: 10.1145/3745756.3809205. URL https://arxiv.org/abs/2508.16703. arXiv:2508.16703.
  23. [28]Yuan, J., Gao, H., Dai, D., Luo, J., Zhao, L., Zhang, Z., Xie, Z., Wei, Y., Wang, L., Xiao, Z., et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 23078–23097, 2025. doi: 10.18653/v1/2025.acl-long.1126. URL https://aclanthology.org/2025.acl-long.1126/.DOI
  24. [29]Zhang, J., Xiang, C., Huang, H., Wei, J., Xi, H., Zhu, J., and Chen, J. SpargeAttn: Accurate sparse attention accelerating any model inference. arXiv preprint arXiv:2502.18137v1, 2025. URL https://arxiv.org/abs/2502.18137v1.
  25. [30]Zhu, Q., Duan, J., Chen, C., Li, X., Liu, S., Feng, G., Lv, X., Xiao, C., Lin, D., and Yang, C. SampleAttention: Near-lossless acceleration of long context LLM inference with adaptive structured sparse attention. In Proceedings of Machine Learning and Systems, volume 7, 2025. URL https://proceedings.mlsys.org/paper_files/paper/2025/file/2d04d97593c83d3415337f408ed0e1b0-Paper-Conference.pdf.

Paper details

vlnc:W_v34vv2md
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Codex, Claude Code

View statement
Looking for related papers…
Contents