betaFeedback

WorksComputer Science

TriAttention: Unified Sparse Mixed-Precision Attention for Long-Context LLM Inference

Abstract

Sparsity and quantization reduce the cost of attention by skipping interactions and lowering the precision of those retained. However, useful interactions differ in their precision requirements: some tolerate low-precision scores while remaining sensitive to removal. We therefore introduce TriAttention, which assigns query-key (QK) tiles to skipping, 4-bit, or 8-bit computation, using 4-bit to retain quantization-tolerant interactions at lower cost. To represent compactly, we encode each head's computation regions with two diagonal-distance boundaries, keeping policy storage independent of context length. Offline calibration then evaluates candidate boundary pairs using normalized projected-output errors and complete-tile execution costs. Using these measurements, a model-wide allocator selects each head's boundaries to minimize calibration error under a shared computation budget. During prefill, the selected boundaries generate precision-grouped traversals for a fused kernel that skips omitted tiles and combines contributions from both QK precisions into one normalized attention output. The quantized keys and values prepared during prefill are then reused in a compressed key-value (KV) cache for decoding, reducing repeated reads of prompt data. Experiments show that TriAttention achieves end-to-end generation speedups of up to 7.88x over the Torch attention and 2.65x over the BF16 baseline. Meanwhile, TriAttention maintains quality comparable to the original models on general-purpose and long-context tasks. These results show how a compact retention-and-precision policy translates local sensitivities into attention computation and repeated KV-cache access. The implementation is https://github.com/HanzhiZhang-Ulrica/TriAttention.

Introduction

Efficient deployment of large language models (LLMs) requires reducing inference latency while preserving model quality across conversational assistance, code generation, reasoning, and document summarization ([10, 1]). For attention, this involves both matrix computation during prompt processing (prefill) and repeated key-value (KV) cache reads during token generation (decode) ([3, 24]). Sparsity reduces attention computation by skipping selected interactions, while quantization accelerates query-key (QK) multiplication through lower-precision arithmetic ([13, 30]). However, attention interactions differ in their sensitivity to skipping and reduced precision: some can be removed with little effect on the output, while other useful contributions tolerate lower QK precision but remain sensitive to removal. Model quality therefore depends on both which interactions are retained and the precision used to compute their contributions.

Figure 1 contrasts efficient attention strategies and illustrates their long-context QA and inference-throughput differences. Dense attention kernels reduce memory transfers and improve GPU utilization, yet still evaluate a quadratic number of causal interactions ([3, 2]). To reduce the interaction count, structured sparse methods select attention patterns or spans, but retained regions still incur floating-point computation. Therefore, further savings from sparsity require excluding more context, and discarding useful interactions degrades model quality ([15, 13, 33] (Fu et al. 2024)). Quantized attention increases QK multiplication throughput with lower-bit arithmetic. However, using a uniformly higher precision protects sensitive regions but spends unnecessary computation on regions that tolerate lower precision, while reducing precision to suit tolerant regions introduces excessive error in sensitive ones and degrades model quality ([30, 28]). Sparse quantized attention combines skipping with low-bit QK computation, yet retained regions still share one precision even though some require higher score accuracy and others tolerate further quantization ([29]). This leads to our central question: how can attention jointly allocate retention and precision for efficient GPU execution?

Figure 1. Attention computation choices and inference performance. Top left: attention maps and token-interaction graphs compare dense, sparse, quantized, and sparse-quantized designs with TriAttention, which jointly assigns skipping, 4-bit, and 8-bit. Bottom left: LooGLE single-hop QA F1 at 64K. Right: throughput across input lengths. Both evaluations use Llama-3.2-3B-Instruct.

Our key insight is that attention regions differ in their sensitivity to skipping and reduced score precision. Exploiting these differences lets us preserve useful context at lower cost while reserving higher precision for sensitive regions. However, translating this flexibility into efficient inference requires resolving two coupled challenges. For allocation, comparable reductions in computation produce different output errors across layers and heads, so a compression level tolerated by some regions introduces excessive errors in less tolerant regions and degrades model quality. For execution, an explicit tile-action map grows quadratically with context length. Query-block traversal metadata also competes with staged K/V operands for shared memory.

To address these challenges, we introduce TriAttention, a sparse mixed-precision attention method that jointly assigns skipping, 4-bit, and 8-bit QK computation within each head. To control policy-metadata growth, we define each layer–head configuration using two diagonal-distance boundaries that describe entire computation regions with storage independent of context length. To compare sensitivity across layers and heads, offline calibration measures each candidate boundary pair’s error after output projection and normalizes it by the layer’s reference attention-output energy. We pair these errors with complete-tile costs, so allocation accounts for the full computation performed on retained tiles. Using these comparisons, a global allocator selects a boundary pair under a shared budget, preserving more computation in sensitive regions and concentrating savings in more tolerant ones. During prefill, the selected boundaries generate only the current query block’s retained tile indices, keeping traversal metadata local to the work staged in shared memory. The fused kernel processes these indices in precision groups to reuse query operands, skips omitted tiles, and combines retained contributions into one normalized attention output. At prompt completion, the selected ranges determine which prepared quantized keys and values are packed into the decode cache, extending the allocation’s savings from prefill to repeated prompt-data reads.

Across the evaluated workloads, TriAttention achieves end-to-end generation speedups of up to 7.88×7.88\times over models using Torch attention and 2.65×2.65\times over the BF16 baseline. These efficiency gains are achieved while maintaining quality comparable to the original models on general-purpose tasks and long-context question answering.

Our contributions are:

  1. Three-Level QK Computation. We develop a tile-level organization of skipping, 4-bit, and 8-bit QK computation that matches graded sensitivity with graded computational cost. The shared tile unit makes retention and precision jointly selectable within each head.

  1. Compact Distance Encoding. We represent each head’s computation regions with two diagonal-distance boundaries, providing a common representation for offline calibration and local, precision-grouped GPU traversal while keeping policy storage independent of context length.

  1. Sensitivity-Guided Global Allocation. We develop an offline allocation rule that compares normalized output errors and execution costs of alternative three-level assignments. A model-wide budget coordinates these choices across layers and heads, prioritizing computation where it most reduces projected-output error.

  1. Cross-Phase Execution. Fused prefill executes the selected computation regions, while decoding reuses the prepared quantized keys and values in a compressed KV cache to reduce repeated prompt-data reads.

Related Work

Efficient attention connects GPU execution, interaction selection, and numerical precision.

Hardware-efficient attention execution improves data reuse, work partitioning, and kernel specialization. Tiled fusion keeps score computation, online softmax, and value aggregation on chip, reducing intermediate transfers while dense query-key computation remains quadratic [3, 2] (Ye et al. 2025). Workload-aware partitioning of query blocks or KV segments across thread blocks improves occupancy and load balance, although KV splitting adds coordination and partial-output reduction [2, 18] (Ye et al. 2025). Asynchronous designs overlap transfers, matrix multiplication, and reduce partial-softmax synchronization to improve pipeline parallelism [19, 11]. Meanwhile, programmable systems specialize fused kernels for score transformations, masks, and KV-cache layouts, supporting attention variants while leaving interaction selection to applications [5] (Ye et al. 2025).

Structured sparse attention selects tokens, spans, heads, or interaction blocks. Streaming windows retain initial attention sinks and recent tokens, bounding KV working sets but denying windowed heads direct access to excluded distant evidence [21] (Xiao et al. 2025). For heterogeneous heads, calibration selects per-head spans or full-context retrieval and streaming roles, leaving retained QK precision fixed (Fu et al., 2024; Xiao et al., 2025). Input-dependent methods use sampled-query attention and lightweight block-importance estimates to select columns, diagonals, or blocks, adding online scoring and sparse-index construction [13, 15, 33] (Xu et al. 2025). Estimated cumulative attention mass adapts retention per input but omits output errors from quantizing retained scores [15, 33] (Xu et al. 2025). Learning-based methods distill block-importance predictors from attention maps, requiring extra predictor training changes [9, 26].

Quantized and mixed-precision attention controls arithmetic precision and operand representations. Blockwise scales and integer kernels accelerate 8-bit QK, but outlier-dominated scales reduce resolution for smaller values [30, 14]. Smoothing and orthogonal transforms reduce outlier error, while finer groups, probability rescaling, and accumulation control enable lower-bit integer and floating-point attention with uniform QK precision within heads despite regional sensitivities [28, 19, 31]. Beyond scale selection, spatial mixed precision gives diagonal neighborhoods higher precision, yet computes all unmasked interactions, including low-impact ones [4, 20]. KV-cache quantization tailors scale axes, quantization levels, and outlier handling to key and value distributions, reducing bytes per entry while cached tokens and per-step reads grow with context [16, 12]. Joint sparse-quantized systems add block skipping to low-bit QK, but retained regions share precision [29]. Other systems combine selective KV access or eviction with cache quantization, reducing traffic but requiring online page or token scoring during generation [24, 32].

Preliminaries

Prefill computes attention for all prompt positions, allowing many queries to reuse each key and value. Decode processes one new query per request and rereads the accumulated KV cache each step. For a context of length LL, dense attention evaluates O(L2)\mathcal{O}(L^2) causal interactions during prefill and O(L)\mathcal{O}(L) per decode step [24].

Both phases use the same attention operation. Let Q∈RLq×dQ \in\mathbb{R}^{L_q \times d} contain the queries and K,V∈RLk×dK,V \in\mathbb{R}^{L_k \times d} the keys and values, where dd is the head dimension. The output is

P=softmax⁡(QKTd+M),O=PV,(1)P = \operatorname{softmax}\left(\frac{QK^{\mathsf{T}}}{\sqrt{d}} + M\right), \qquad O = PV, \tag*{(1)}

where MM applies the causal mask and softmax operates. QK multiplication produces attention scores, and PV multiplication aggregates values using the attention probabilities in PP. Grouped-query attention (GQA) allows several query heads to share the same KK and VV [24].

Integer operands and scales. Integer quantization represents a group of values XX through integer codes X^\hat{X} and a scale sXs_X, with X≈sXX^X \approx s_X\hat{X} [30]. The corresponding pre-scaled values restore the numerical magnitude of integer products. Products from different groups or precisions can then be accumulated in common units.

Tiled attention and partial aggregation. Fused attention kernels pair query and key blocks into tiles, stage operands in shared memory, and keep intermediate scores and probabilities on chip [3]. For one query row, a subset of keys produces a partial state (mr,lr,or)(m_r,l_r,o_r): its maximum score, the sum of exponentiated scores relative to that maximum, and the correspondingly weighted value sum. Partial states over disjoint key subsets are combined by aligning their score maxima:

m=max⁡rmr,m = \max_r m_r,
l=∑rexp⁡(mr−m)lr,l = \sum_r \exp(m_r-m)l_r,
o=∑rexp⁡(mr−m)or,O=o/l.(2)o = \sum_r \exp(m_r-m)o_r, \qquad O = o/l. \tag*{(2)}

This rule supports both online accumulation across successive tiles and parallel reduction across key segments. The partials use common score and value units so that the denominator and numerator describe the same attention weights.

TriAttention

TriAttention uses regional sensitivity to assign attention tiles to skipping, 4-bit, or 8-bit QK computation. The diagonal-distance boundaries define each layer–head configuration for both offline candidate evaluation and precision-grouped traversal. Offline calibration evaluates candidate boundary pairs, and a model-wide allocator uses their errors and costs to select a boundary table under a shared computation budget Figure 2. During inference, this boundary table guides fused prefill and compressed KV-cache for decode.

Offline calibration and compact distance encoding

Figure 2. Offline calibration and compact distance encoding. Left: candidate boundary pairs are evaluated by normalized projected-output error. Center: errors and complete-tile execution costs guide allocation under a shared computation budget. Full-model prediction checks select the final boundary table. Right: two diagonal-distance boundaries per layer and query head encode 8-bit, 4-bit, and skipped QK regions, with the first key block and diagonal tiles protected.

Compact Distance Encoding

Long contexts enlarge the attention tile map, while each GPU thread block has a limited shared-memory budget for traversal metadata and staged operands. We organize the three QK actions by distance from the attention diagonal. Each boundary pair describes a query head’s retained context and QK precision regions.

For query block ii and key block jj, let Δ=i−j\Delta= i-j be their distance in blocks. A configuration c=(w8,w4)c=(w_8,w_4) specifies the extent of the 8-bit QK region and the overall retained region, with 1≤w8≤w41 \le w_8 \le w_4. We protect the first key block and the diagonal tile with 8-bit QK, preserving access to the prompt beginning and the query’s own block. For the remaining tiles, the configuration assigns

ac(Δ)={8-bit,1≤Δ<w8,4-bit,w8≤Δ<w4,skip,Δ≥w4.(3)a_c(\Delta)= \begin{cases} \text{8-bit}, & 1 \le\Delta< w_8,\\ \text{4-bit}, & w_8 \le\Delta< w_4,\\ \text{skip}, & \Delta\ge w_4. \end{cases} \tag*{(3)}

Extending w4w_4 retains additional context at 4-bit precision, while increasing w8w_8 assigns 8-bit precision. We select a separate pair for every layer and query head from a candidate set C\mathcal{C} that includes full-context configurations.

The compact distance encoding in Figure 2 stores two integers per layer and query head, keeping policy storage independent of context length. At execution time, each thread block uses these boundaries to generate only the retained key-block indices for its query block. The indices are grouped by precision, so their group boundaries specify the arithmetic paths. Calibration evaluates these boundary pairs, and allocation selects one pair per layer and query head to form the boundary table.

Per-layer QK sensitivity in Llama-3.2-3B-Instruct

Figure 3. Per-layer QK sensitivity in Llama-3.2-3B-Instruct. Rows are query heads; columns are diagonal-distance bands ordered near to far. Colors show layer-normalized projected-output error from 4-bit QK or skipping, with nearer bands kept in 8-bit.

Distribution of high-sensitivity QK bands in Llama-3.2-3B-Instruct

Figure 4. Distribution of high-sensitivity QK bands in Llama-3.2-3B-Instruct. Ten bars per layer. Each bar reports the percentage of heads whose error lies in the model-wide top 20% of layer–head-band errors, ranked separately for 4-bit and skipping.

Sensitivity-Guided Global Allocation

Figures 3 and 4 show that output errors from skipping and 4-bit QK vary across layers, heads, and diagonal-distance bands. We therefore evaluate candidate boundary pairs for each layer and query head and select a boundary table under a shared computation budget.

Measuring the effect on the layer output. To compare these configurations, we measure how each one changes a head’s contribution to the representation passed through the model by the attention output projection. So we evaluate its error after this projection. We normalize each head’s projected-output error by the layer’s reference attention-output energy, making errors comparable across layers.

For calibration input xx, let Oℓhc(x)O_{\ell h}^{c}(x) and Oℓhref(x)O_{\ell h}^{\mathrm{ref}}(x) denote the candidate and dense 16-bit reference outputs of query head hh in layer ℓ\ell at the scored calibration positions. Let WℓhOW_{\ell h}^{O} be the block of the output projection that maps this head into the model’s hidden dimension, and let Yℓref=∑hOℓhrefWℓhOY_{\ell}^{\mathrm{ref}} = \sum_h O_{\ell h}^{\mathrm{ref}} W_{\ell h}^{O} be the complete reference attention output. Over calibration prompts D\mathcal{D}, we score the candidate by

eℓh(c)=1∣D∣∑x∈D∥(Oℓhc(x)−Oℓhref(x))WℓhO∥F2∥Yℓref(x)∥F2.(4)e_{\ell h}(c) = \frac{1}{|\mathcal{D}|} \sum_{x \in\mathcal{D}} \frac{\left\| \left(O_{\ell h}^{c}(x) - O_{\ell h}^{\mathrm{ref}}(x)\right) W_{\ell h}^{O} \right\|_{F}^{2}}{\left\|Y_{\ell}^{\mathrm{ref}}(x)\right\|_{F}^{2}}. \tag*{(4)}

A lower score identifies a configuration that preserves the head’s projected contribution more closely. All candidates’ reference output continues to the next layer. Their scores thus compare the combined effects of context retention and quantized computation under common inputs.

Allocating a shared computation budget. Each error score is paired with the computation needed to produce that output. At a common target context length, let n8(c)n_{8}(c) and n4(c)n_{4}(c) count the candidate’s retained 8-bit and 4-bit QK tiles, excluding the protected tiles shared by all candidates. Profiling complete tiles, including QK, softmax, and PV, gives the relative cost γ\gamma of a tile using 4-bit QK compared with one using 8-bit QK. We estimate normalized cost as C(c)=[n8(c)+γn4(c)]/n8\mathcal{C}(c) = [n_{8}(c) + \gamma n_{4}(c)]/n_{8}, where n8n_{8} is the corresponding count when all eligible tiles use 8-bit QK. This cost prices both context extension and increased precision.

For each layer and head, we choose cℓhc_{\ell h} to minimize summed error under a shared normalized-cost budget BB:

min⁡{cℓh∈C}∑ℓ,heℓh(cℓh)s.t.∑ℓ,hC(cℓh)≤B.(5)\min_{\{c_{\ell h}\in\mathcal{C}\}} \sum_{\ell,h} e_{\ell h}(c_{\ell h}) \quad\text{s.t.} \quad\sum_{\ell,h} \mathcal{C}(c_{\ell h}) \le B. \tag*{(5)}

We generate candidate boundary tables by sweeping a shared cost weight λ≥0\lambda\ge0, with each head selecting cℓh(λ)=arg⁡min⁡c∈C[eℓh(c)+λC(c)]c_{\ell h}(\lambda) = \arg\min_{c\in\mathcal{C}} [e_{\ell h}(c) + \lambda\mathcal{C}(c)]. The shared weight assigns the same price to computation in every head, giving wider context or higher precision priority where it yields greater error reduction per added cost. Per budget, we retain the feasible sampled table with the lowest summed error.

The resulting tables are evaluated in the complete model on separate selection prompts. We choose the lowest-cost table that satisfies the prescribed thresholds for mean next-token KL divergence from the reference model and the mean increase in teacher-forced response negative log-likelihood within every context-length group. Appendix § 7.§ 1 specifies the calibration prompts, candidate grid, cost-weight sweep, and selection thresholds.

Fused Prefill with Compact Traversal

Given the selected boundary table and a layer’s QQ, KK, and VV, prefill prepares the required operand formats and constructs a compact traversal for each query block. The traversal determines which K/V tiles enter shared memory and which arithmetic path processes them. The kernel accumulates their contributions into one attention output. Figure 5 follows the selected tiles through this execution.

Figure 5. Fused prefill with compact, precision-grouped traversal. The illustrated query block visits protected tiles 0 and 6, 8-bit QK tiles 4 and 5, and 4-bit QK tile 3. Protected tiles use 16-bit PV, while other retained tiles use 8-bit PV. Shared-memory staging and query reuse feed one register-resident attention state, followed by normalization, value restoration, and 16-bit output.

Preparing reusable operands. After rotary position embedding, the runtime centers keys and applies a common orthogonal signed Hadamard transform to queries and keys to control their dynamic range before quantization. It prepares the 8-bit and 4-bit Q/K formats for the selected table, with their scales. Value formats share channelwise centering and scaling statistics, allowing reconstruction after aggregation. Preparation produces 16-bit and 8-bit values for prefill and 4-bit values for the compressed prompt KV cache. GPU memory holds the packed operands and statistics.

Constructing the shared-memory traversal. Each thread block reads its head’s two boundaries and combines them with its query-block index to generate the retained key-block indices. Threads write these indices cooperatively in shared memory, with consecutive segments for protected tiles, 8-bit QK tiles, and 4-bit QK tiles. Segment boundaries specify the arithmetic path, so each list entry carries only a key-block index. The kernel consumes this list to stage the corresponding packed K/V tiles and reuses the query representation within each pass. Skipping removes tiles from the list and their associated QK, softmax, and PV work. This organization places the current query block’s traversal data alongside its operand buffers, linking the compact distance policy directly to on-chip execution.

Executing the precision groups. Protected tiles use 8-bit QK with 16-bit PV. Both remaining QK precisions use 8-bit probabilities and 8-bit values for PV. Tile-local probability scaling preserves resolution within each retained tile. The integer path uses the same probability codes for its normalization sum and PV product, and their scale places both contributions in the shared state. All groups update the same attention state in registers, keeping QK, probability formation, and PV accumulation fused across the traversal. After the final tile, the kernel normalizes and restores the value coordinates to produce OO for the layer’s output projection. Quantization scales, probability coding, and shared-state updates are detailed in Appendix § 7.§ 2. Prepared prompt K/V and statistics then supply the cache.

Decoding from the Compressed Cache

To reduce repeated prompt-data reads, we use the selected boundaries to compact the prepared quantized K/V into a compressed prompt KV cache. Cache construction applies each head’s boundaries to the final prompt query block, identifies the retained prompt tiles, and packs this selection once for reuse throughout generation. Each new query attends to this compressed prompt KV cache and a 16-bit tail containing generated-token keys and values. Figure 6 shows the computation over the retained prompt data.

Figure 6. Mixed-precision decode over the compressed prompt KV cache. Alternating shared-memory buffers B0/B1 stage retained KV tiles for 8-bit (blue) or 4-bit (green) computation. The kernel reuses query fragments and maintains attention state in registers across tiles. Each key segment writes partial states for subsequent merging with other segments and the 16-bit generated-token tail.

Mapping head decisions to shared storage. The policy specifies context per query head, while GQA shares physical K/V storage across a group of query heads. We retain the union of the group’s requested tiles and store each key tile once at the highest requested precision. A head mask identifies which query heads retain the tile, and participating heads compute QK at its stored precision. Compact buffers store retained K/V tiles with indices, masks, and reconstruction statistics. Values use 8-bit storage for the first prompt block and a recent window at the prompt’s end, with 4-bit storage for the other retained blocks. This combines shared storage with each head’s selected context and reduces prompt-data reads at each generation step.

Sharing reads and parallelizing the retained context. The incoming query receives the same rotation and quantization used in prefill, allowing it to consume the stored prompt keys directly. Queries sharing a KV head form rows of one matrix operation, so each loaded KV tile serves the participating heads in the group. Since decode supplies one query position per request, the kernel splits the retained key tiles across GPU thread blocks to expose parallelism along the context. Each block processes its assigned segment using the stored key precision and probability codes matched to the value precision, returning (mr,lr,or)(m_r, l_r, o_r) for merging.

Producing the output and extending the context. Newly generated keys and values are appended to the 16-bit tail. The merge kernel computes attention over this tail using the prompt’s key-centering offset. It combines the prompt partials and restores their value coordinates with the stored statistics. With both sources in common score and value units, the kernel merges their contributions and normalizes the result to produce the current query’s output. Appendix § 7.§ 3 details compact traversal, shared cache indexing, and prompt–tail merging.

Experiments

Experimental Setup

We evaluate inference efficiency and model quality for general and long-context tasks, with allocation controls examining the value of head-specific retention and precision. The main evaluations focus on Llama-3.2-3B-Instruct, with computation-level ablations also covering Qwen2-7B-Instruct and Ministral-3-8B-Instruct-2512-BF16. Appendix § 12.§ 2 reports additional quality results for the latter two models.

Datasets and Metrics. We evaluate language modeling using WikiText-2 word perplexity and language understanding using accuracy on HellaSwag, MMLU, ARC-Challenge, LAMBADA, PIQA, and WinoGrande. Long-context question answering is assessed on nine LV-Eval (Yuan et al., [27]) datasets at the 16K, 32K, and 64K length levels, using the benchmark’s task-specific F1 and ROUGE-L metrics.

Configuration and Comparisons. TriAttention uses 64 × 64 attention tiles. For the full method, the target mean normalized computation cost is 0.80. BF16 provides the original-model reference with scaled dot-product attention (SDPA), while Torch attention uses the math backend. The low-bit reference, all-int4, retains all causal tiles and uses INT4 QK outside the protected first and diagonal tiles. We compare against MInference, FlexPrefill, SageAttention2, MoA, and SpargeAttn (Jiang et al., [13]; Lai et al., [15]; Zhang et al., [28]; Fu et al., 2024; Zhang et al., [29]). Layer-shared, model-shared, and randomized boundary assignments test allocation granularity and sensitivity-guided placement. Appendix § 8.§ 1 provides model configurations, task settings, and allocation-control definitions.

Hardware and Measurement. Primary efficiency measurements use an NVIDIA A100-SXM4 40 GB GPU, with additional system evaluations on H100 NVL and RTX 4090 GPUs. We report time per output token (TPOT), complete-request generation throughput, peak GPU memory, and energy per 1,000 generated tokens. The workload sweeps vary prompt length, output length, and batch size, with complete-request measurements covering both prefill and decode. Timing, energy, and memory-measurement protocols are specified in Appendix § 8.§ 2.

Inference Efficiency

We first examine whether the selected attention representation improves complete generation requests. On Llama with a 64K prompt, TriAttention reaches 22.80 tokens/s, compared with 14.06 for BF16 and 17.39 for FlexPrefill, yielding 1.62×1.62 \times and 1.31×1.31 \times throughput, respectively (Table 1). The improvement accompanies a reduction in TPOT from 26.64 to 10.06 ms relative to BF16. Peak allocation decreases from 15.65 to 11.76 GiB, and energy per generated token falls by 36.5%. These measurements establish that the attention implementation’s benefits extend to inference with the rest of the model included.

MetricUnitUpper Ref. BF16Ours TriAttentionMInferenceFlexPrefillSageAttnMoASpargeAttn
TPOT ↓\downarrowms/token26.6410.0626.9126.7926.93176.6426.92
Throughput ↑\uparrowtok/s14.0622.8014.2317.3914.594.6413.97
Peak VRAM ↓\downarrowGiB15.6511.7615.7716.4015.6515.7716.82
Energy ↓\downarrowJ/1K tokens24788.2615737.6221536.4818358.3123459.1671430.9923365.47

Table 1. End-to-end inference efficiency for Llama-3.2-3B-Instruct on A100-SXM4-40GB, with 65,536 input tokens, 128 output tokens, and batch size one. Peak VRAM covers generation including prefill. Cell colors are relative to BF16 under each metric’s direction: green denotes equal or better results, blue worse results, and gray BF16. Bold and underlined values mark the best and worst results among Ours and Others; displayed ties share the mark.

The comparison with FlexPrefill highlights the importance of carrying attention savings into generation. Its TPOT is 26.79 ms, compared with 10.06 ms for TriAttention, alongside the complete-request throughput improvement. Figure 7 evaluates this combination across different batch sizes and input/output lengths. Prompt processing occurs once per request, while each generated token reuses the compressed prompt cache. The workload sweep therefore evaluates how the prefill and decode paths contribute as the balance between prompt processing and generation changes. Appendix § 12.§ 1 extends the throughput and TPOT comparisons across models, request shapes, and GPU platforms.

Generation-throughput ratio across batch sizes, input lengths, and output lengths

Figure 7. Generation-throughput ratio on A100-SXM4-40GB. Llama-3.2-3B-Instruct. Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048, 8,192, 16,384 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×; higher is better. Method colors and order are fixed across panels.

FINDINGS Joint retention and precision improve complete-request throughput while reducing token-generation latency, peak memory, and energy.

General-Purpose Performance

Having established inference gains, we examine whether the same boundary table preserves general prediction quality across tasks. The table selected through summary and retrieval calibration is reused across the benchmarks, testing whether its retention and precision choices support broader model capabilities. On Llama, TriAttention has word perplexity 12.50, compared with 12.49 for BF16 (Table 2). All six accuracy scores exceed the BF16 reference by less than 0.7 percentage points, with MMLU increasing from 60.66% to 61.32%. Near-reference quality therefore extends across both language modeling and the evaluated understanding tasks under one allocation.

BenchmarkMetricUpper Ref.Lower Ref.OursOthers
WikiText-2PPL ↓BF16all_int4TriAttentionMInferenceFlexPrefillSageAttnMoASpargeAttn
WikiText-2PPL ↓12.4912.5912.5012.5312.7412.5212.6412.43
HellaSwagAcc. ↑71.5271.5372.0371.6271.6371.8271.6571.61
MMLUAcc. ↑60.6660.2661.3260.6560.6460.8661.2860.79
ARC-ChallengeAcc. ↑52.5651.7152.9752.7352.4751.7953.0752.05
LAMBADAAcc. ↑64.2364.3364.8364.3964.4964.2564.8264.39
PIQAAcc. ↑77.3177.1577.6576.7177.2676.8877.0476.71
WinoGrandeAcc. ↑71.4371.1171.8571.5171.5971.8270.8070.24

Table 2. General-task results for Llama-3.2-3B-Instruct. Accuracy is reported in percent (higher is better); WikiText-2 uses word perplexity (lower is better). BF16 is the upper reference; all_int4 is the lower reference. Colors compare with BF16: green denotes equal or better results, blue worse results, and gray BF16. Bold marks the best result among Ours and Others; displayed ties share the mark.

The full-context all_int4 reference provides a complementary comparison by retaining all causal tiles and applying 4-bit QK outside the protected regions. Its perplexity is 12.59 and MMLU accuracy is 60.26%, while TriAttention achieves lower perplexity and higher accuracy on all six tasks. These results support jointly selecting retained context and score precision to preserve predictions alongside the measured inference gains. The following long-context QA evaluation examines whether this quality extends from prefill predictions to answer generation with repeated access to the compressed prompt cache.

FINDINGS One model-level retention-and-precision policy preserves general-task quality across benchmarks while supporting efficient inference.

Long-Context Question Answering

Long-context QA evaluates whether the context retained during prefill remains useful throughout answer generation. Table 3 reports nine tasks at the 16K, 32K, and 64K dataset levels using the same calibrated boundary table, alongside controls for boundary sharing and placement. Dataset identifiers are ① cmrc_mixup, ② dureader_mixup, ③ hotpotwikiqa_mixup, ④ lic_mixup, ⑤ loogle_CR_mixup, ⑥ loogle_MIR_mixup, ⑦

DatasetLevelUpper Ref.Lower Ref.OursOthersAblations
①16K21.3318.1021.9426.9327.1129.7913.5420.8520.6620.0519.65
32K18.4915.3418.0711.5511.7413.5010.0216.4016.5017.4713.38
64K13.9511.3214.615.315.395.776.2113.7211.636.474.30
②16K16.2215.4816.8414.6114.9216.6714.5715.3315.8415.5616.20
32K17.5915.1617.0112.0111.6414.5914.9616.8715.1016.3115.14
64K17.0015.2517.927.966.8111.2213.3616.3915.3514.2014.37
③16K23.7620.2624.2213.9713.1615.2715.9120.9418.6821.9822.66
32K29.5728.7828.817.817.209.1119.0428.4321.6323.2524.61
64K22.6622.6825.534.263.865.5019.2022.7522.8913.5417.63
④16K11.0510.1211.539.829.2810.697.8810.7510.539.5011.09
32K12.0310.0112.674.884.485.696.729.7212.0211.357.47
64K9.577.3310.851.121.071.444.159.427.414.471.93
⑤16K25.2723.8625.6415.5914.0519.9419.2825.5323.8524.7523.87
32K25.7023.4125.7512.2310.9515.9218.1525.3923.0021.2923.05
64K24.8324.1725.467.856.6711.2418.4322.1520.5715.5914.62
⑥16K19.1218.9020.1612.3211.3913.9315.4419.7817.2117.6417.17
32K20.1118.8119.179.267.4010.5413.6818.8516.1213.5813.11
64K19.0918.8919.657.245.089.459.7417.5817.387.9410.00
⑦16K54.0151.3154.4238.1635.2442.1237.2051.2249.3451.5950.49
32K49.8846.8450.7420.0921.4022.5029.3847.2144.0335.3941.88
64K48.4044.8247.9410.2410.6913.2119.6345.4545.5117.0720.17
⑧16K34.1933.1233.2623.2823.4325.1823.5936.2732.0833.3933.07
32K32.9530.5531.5012.7010.2813.9420.7632.9930.6823.1827.54
64K36.7733.6035.199.427.6813.3723.9833.4435.3622.5321.71
⑨16K18.4219.7919.5415.9912.6418.1916.0219.5620.5619.3719.26
32K12.8111.8614.5311.3310.2514.577.4313.6110.8714.3112.67
64K11.5910.6412.233.104.025.796.669.3411.294.978.64

Table 3. LV-Eval QA results for Llama-3.2-3B-Instruct. Dataset identifiers follow § 5.4. Scores use official task-specific metrics on a 0–100 scale; higher is better. Length labels follow the official dataset levels. BF16 is the upper reference; all_int4 is the lower reference. Colors compare with BF16: green denotes equal or better results, blue worse results, and gray BF16. Bold marks the best result among Ours, Others, and Ablations; displayed ties share the mark.

loogle_SD_mixup, ⑧ multifieldqa_en_mixup, and ⑨ multifieldqa_zh_mixup.

LooGLE-SD shows how this policy preserves answer quality as context grows. TriAttention scores 54.42, 50.74, and 47.94 at 16K, 32K, and 64K, respectively, following BF16’s 54.01, 49.88, and 48.40 with differences below one point at every level. At 64K, its score also exceeds 44.82 for all_int4 and 45.45 for SpargeAttn. The comparison with all_int4, which retains every causal tile, supports selecting score precision together with retained context. Selective mixed-precision attention preserves stronger answer quality here than retaining the full context with uniformly low QK precision outside the protected regions.

Tasks requiring multiple pieces of evidence provide a complementary check. At 64K, TriAttention scores 19.65 on LooGLE-MIR, compared with 19.09 for BF16 and 17.58 for SpargeAttn. On LooGLE-CR, the corresponding scores are 25.46, 24.83, and 22.15. HotpotWikiQA also reaches 25.53, compared with 22.66 for BF16 and 22.75 for SpargeAttn. These comparisons extend quality preservation across different QA tasks under one model-level allocation. They assess

the complete prefill-and-decode path: the retained context must support generated answers after being packed into the compressed prompt cache.

The allocation controls examine how this behavior depends on the freedom to assign different boundaries. Layer Same shares one boundary pair across the query heads of each layer, while All Same uses a common 32K 8-bit window throughout the model, with the first block protected. On LooGLE-SD at 64K, their scores are 45.51 and 17.07, compared with 47.94 for TriAttention. On LooGLE-CR, the corresponding scores are 20.57, 15.59, and 25.46. Sharing boundaries couples the retained ranges and precision choices of heads with different sensitivities. Head-specific boundaries let the allocator preserve wider ranges or higher precision where they most reduce projected-output error, supporting stronger QA performance in these comparisons.

Random then tests the value of matching the selected configurations to head sensitivity. It permutes the deployed boundary pairs across layers and heads, preserving both their collection and the aggregate modeled computation cost. On LooGLE-SD, its gap from TriAttention widens. from 3.93 points at 16K to 27.77 points at 64K, where Random scores 20.17 against 47.94. LooGLE-MIR shows a corresponding separation at 64K, with 10.00 for Random and 19.65 for TriAttention. The growing separation links long-context quality to which heads receive the wider ranges and higher precision. Sensitivity-guided placement therefore makes the selected computation more effective for QA within the same modeled budget.

FINDINGS The selected ranges preserve useful context throughout answer generation, while sensitivity-guided placement achieves stronger long-context QA than randomized assignment at the same modeled computation cost.

Ablation and Analysis

The preceding QA controls examine the placement and sharing of boundary configurations. We now examine the computation choices available to the allocator: whether combining skipping with two retained precision levels reduces the prefill time needed to meet the same quality-selection requirements.

Computation-level controls. Without 4-bit imposes w8=w4w_8 = w_4, leaving each non-protected region a choice between 8-bit QK and skipping. Without Skip imposes w4=∞w_4 = \infty, retaining the full causal context while assigning its 4-bit and 8-bit regions. Each control is independently reallocated and selected under the same quality-selection thresholds, allowing its remaining choices to adapt to the validation requirements. Table 4 compares the resulting prefill attention latency on A100 across three models and contexts from 32K to 128K.

ModelContextTriAttention (s)Without 4-bit (s)Without Skip (s)
Llama32K0.6540.6600.682
64K2.2422.2592.647
128K7.3477.56810.324
Qwen232K0.7810.8100.845
64K2.7522.9243.232
128K10.13411.15113.010
Ministral32K1.0661.1061.086
64K3.8564.0784.182
128K14.26615.44916.402

Table 4. Effect of three-level computation on A100 prefill attention latency. Latency includes Q/K preparation, V packing, and attention across all layers. Ablations are independently reallocated under the same quality-selection thresholds. Lower is better.

Lowering the cost of retained context. TriAttention achieves the lowest latency in all nine settings. At 128K, the full design reduces Qwen2 latency from 11.151 s without 4-bit to 10.134 s, and Ministral latency from 15.449 to 14.266 s, reductions of 9.1% and 7.7%, respectively. Without the intermediate level, retaining an interaction also commits it to 8-bit QK computation. The 4-bit region separates these choices, allowing lower-cost retention alongside an 8-bit region for more sensitive interactions. Since the measurements include Q/K preparation and V packing across all layers, these reductions reflect the benefit over complete attention calls after preparing the mixed-precision operands.

Removing work as context grows. Skipping reduces latency by eliminating QK, normalization, and PV for omitted tiles. For Llama, the reduction relative to Without Skip increases from 4.1% at 32K to 28.8% at 128K, where latency falls from 10.324 to 7.347 s. Qwen2 shows the same length-dependent pattern, with reductions increasing from 7.6% to 22.1%; the relative benefit also grows across the three lengths for Ministral. Full retention adds interactions throughout the causal map as context expands. Allowing selected distance regions to be skipped removes their complete tile cost, complementing the lower QK cost of interactions that remain.

Adapting the retention–precision trade-off. The faster restricted configuration also changes with the workload. For Ministral at 32K, Without Skip takes 1.086 s and Without 4-bit takes 1.106 s. At 128K, their ordering reverses, with 16.402 and 15.449 s, respectively, while the full design remains fastest at both lengths. Thus, the relative efficiency of retaining all context at mixed precision and retaining selected context at 8 bits varies even within one model. Independently adjustable boundaries let the allocator combine these alternatives. The measured advantage of the full design supports jointly choosing how far attention extends and which precision is used within that range.

FINDINGS Three computation levels reduce both the cost of retained interactions and the number of executed tiles, lowering prefill latency under common quality-selection requirements.

Conclusion

TriAttention demonstrates that useful attention interactions can be preserved at lower cost by jointly allocating retention and QK precision according to regional sensitivity. Compact distance boundaries carry these choices through fused prefill and compressed-cache decoding, yielding faster generation while preserving general-task and long-context quality. Allocation controls further show that computation placement affects long-context quality at a fixed modeled cost. These findings support a broader principle for efficient attention: evaluate which interactions to retain together with the precision they require, and make that joint policy govern both GPU computation and repeated context access.

Calibration and Implementation Details

The boundary table connects offline sensitivity measurements to the operands used during inference. This appendix details how calibration selects that table, how low-bit products recover attention scores and outputs, and how execution addresses and merges the resulting representations.

Calibration and Boundary-Table Selection

Calibration inputs and scored positions. We combine document summarization and archive-record retrieval to evaluate both distributed context use and access to specific evidence. The 16 calibration prompts span 16K, 32K, 64K, and 128K contexts. Summary prompts use Multi-News (Fabbri et al. [6]) documents and reference summaries capped at 256 response tokens. Retrieval prompts place a unique numeric record at document depths 0.2, 0.5, or 0.8 and request its value at the end. Each prompt uses the model’s chat template and reference response, with a 129,024-token budget for the 128K construction to accommodate both. Calibration and selection use separate documents, disjoint from LV-Eval and LongEval.

We score the final 1,024 positions, covering the prompt ending and teacher-forced response, so the evaluated queries attend over the long input. A dense BF16 pass supplies common Q/K/V for all candidates and propagates the reference output to the next layer. Candidate errors therefore reflect their retention and precision choices under the same layer input. Equation (4) normalizes each prompt’s projected-output error before averaging, giving every prompt equal weight.

Candidate ranges and computation costs. Geometrically spaced boundaries cover local through full-context configurations with a compact candidate set. Measured in 64-token blocks, the grid is

W8={1,2,4,…,512,∞},\mathcal{W}_8 = \{1, 2, 4, \ldots, 512, \infty\},
W4={1,2,4,…,2048,∞},\mathcal{W}_4 = \{1, 2, 4, \ldots, 2048, \infty\},
C={(w8,w4):w8∈W8,w4∈W4,  w8≤w4},\mathcal{C} = \{(w_8,w_4) : w_8 \in\mathcal{W}_8, \\ w_4 \in\mathcal{W}_4, \; w_8 \le w_4\},

These 86 pairs jointly vary retained context and QK precision. Setting w4=∞w_4 = \infty retains the full causal context, while w8=w4w_8 = w_4 assigns 8-bit QK to retained non-protected tiles.

The allocator evaluates non-protected tile costs at 65,536 tokens using the measured complete-tile ratio γ=0.905\gamma= 0.905. For NHN_H layer–query-head pairs, a target mean cost τ\tau gives the total budget B=NHτB = N_H\tau in Equation (5). We evaluate λ=0\lambda= 0 and 400 logarithmically spaced positive values from 10−910^{-9} to 10110^1, retaining the feasible sampled table with the lowest summed calibration error for each target budget. The deployed mean-cost targets are 0.80 for Llama-3.2-3B-Instruct and 0.85 for Qwen2-7B-Instruct and Ministral-3-8B-Instruct-2512-BF16.

Complete-model selection. The assembled boundary tables are evaluated on separate summary and retrieval prompts. KL divergence measures changes in the BF16 next-token distribution, while response NLL measures the likelihood assigned to the reference answers. We average each metric over response tokens within a prompt, then over prompts in the 16K/32K, 64K, and 128K groups. Each group must satisfy mean KL at most 0.002 and mean NLL increase at most 0.01 nats. This applies the quality requirements separately across context lengths before selecting the lowest-cost qualifying table for inference.

Operand Quantization and Shared Normalization

Queries and keys in common score coordinates. Key centering removes the prompt-wide key offset, and a shared orthogonal transform redistributes channel magnitudes before quantization (Zhang et al. [28]; ); Shah et al. [19]; ); Zhang et al. [31]). These operations prepare the same Q/K inputs for both precision paths. After rotary position embedding, let μK\mu_K be the mean prompt key vector for one KV head and UU the fixed orthogonal signed Hadamard transform. The prepared operands are Q~=QU\widetilde{Q} = QU and K~=(K−1μKT)U\widetilde{K} = (K - 1\mu_K^{\mathsf{T}})U. Since UUT=IUU^{\mathsf{T}} = I,

Q~K~T=QKT−(QμK)1T.(6)\widetilde{Q}\widetilde{K}^{\mathsf{T}} = QK^{\mathsf{T}} - (Q\mu_K)1^{\mathsf{T}}. \tag*{(6)}

Centering subtracts the same quantity from every legal score in a query row and therefore preserves its softmax before quantization. The orthogonal transform redistributes channel magnitudes without changing the centered dot products.

For bit-width r∈{4,8}r \in\{4,8\}, symmetric quantization uses ar=2r−1−1a_r = 2^{r-1} - 1 and absmax scale sX=max⁡∣X∣/ars_X = \max|X|/a_r per quantization group (Zhang et al. [30]). Codes round X/sX(r)X/s_X^{(r)} to the nearest integer, with zero-valued groups producing zero codes. Queries use one scale per token and query head; keys use one scale per KV head and group of 16 consecutive tokens across their channels. The 4-bit and 8-bit codes represent the same transformed operands with separate scales. For query-token index tt and key-token index kk, the restored score is

s~tk(r)=sQ,t(r)sK,g(k)(r)d⟨Q^t(r),K^g(k)(r)⟩,(7)\widetilde{s}_{tk}^{(r)} = \frac{s_{Q,t}^{(r)}s_{K,g(k)}^{(r)}}{\sqrt{d}}\left\langle\widehat{Q}_t^{(r)}, \widehat{K}_{g(k)}^{(r)} \right\rangle, \tag*{(7)}

where g(k)g(k) is key kk’s 16-token group. Restored scores at both precisions enter the same probability computation.

Values in shared reconstruction coordinates. Sharing channel statistics across value formats lets the kernel accumulate their contributions together and reconstruct the output once. We center and scale each prompt value channel to prepare the FP16, INT8, and INT4 representations [28]. For each KV head and channel cc, preparation computes the prompt mean μV,c\mu_{V,c} and

σV,c=max⁡k∣Vk,c−μV,c∣127.\sigma_{V,c} = \frac{\max_k \lvert V_{k,c} - \mu_{V,c}\rvert}{127}.

A constant channel uses scale one. The normalized values are V′=(V−μV)⊘σVV^\prime= (V-\mu_V)\oslash\sigma_V. Protected prefill tiles store V′V^\prime in FP16, while other retained prefill tiles use the rounded INT8 codes V^8\widehat{V}_8. For distant decode tiles, V^4=round⁡(7V′/127)\widehat{V}_4=\operatorname{round}(7V^\prime/127) is stored in signed 4-bit form and reconstructed in the same normalized coordinates as (127/7)V^4(127/7)\widehat{V}_4.

Probability coding and accumulation. Tile-local scaling gives each tile its own probability-coding range, while a separate factor preserves its weight relative to other tiles. For one query row, we encode the restored scores in base-two units and merge their contributions into the shared attention state. For one query row and the current key tile, let sks_k be the restored score multiplied by log⁡2e\log_2 e, and let b=max⁡k∈Kskb=\max_{k\in\mathcal{K}}s_k, where K\mathcal{K} contains this row’s legal keys in the tile.

Using the row’s query scale and the four key-group scales at QK precision rr, the largest score multiplier is

ηmax⁡=log⁡2edsQ(r)max⁡u∈{0,1,2,3}sK,u(r).(8)\eta_{\max}=\frac{\log_2 e}{\sqrt{d}s_Q^{(r)}}\max_{u\in\{0,1,2,3\}}s_{K,u}^{(r)}. \tag*{(8)}

We use the probability-coding base

εtile=0.75ηmax⁡+2.4×10−7∣b∣,\varepsilon_{\mathrm{tile}}=0.75\eta_{\max}+2.4\times10^{-7}\lvert b\rvert,
b′=b+εtile.(9)b^\prime=b+\varepsilon_{\mathrm{tile}}. \tag*{(9)}

The shared base provides rounding headroom while keeping all four key groups on the same probability scale.

The probability codes are

p^k=⌊round⁡1/256(aP2sk−b′+ρk)⌋,\widehat{p}_k=\left\lfloor\operatorname{round}_{1/256}\left(a_P2^{s_k-b^\prime}+\rho_k\right)\right\rfloor,

where round⁡1/256\operatorname{round}_{1/256} denotes rounding to the nearest multiple of 1/2561/256 in the FP32 packing step. Prefill uses ρk=1/2\rho_k=1/2. Decode uses ρk=(Dk mod 256)/256\rho_k=(D_k\bmod256)/256, where DkD_k is the integer QK dot product for key kk. The maximum code is aP=255a_P=255 for 8-bit PV and aP=15a_P=15 for 4-bit PV. Masked keys receive zero probability.

Let zkz_k be the stored value expressed in V′V^\prime coordinates. The factors α\alpha and β\beta align the accumulated state and the current tile to the updated maximum [3]:

m′=max⁡(m,b),α=2m−m′,β=2b′−m′/aP,l′=αl+β∑kp^k,o′=αo+β∑kp^kzk.(10)\begin{aligned} m^\prime&= \max(m,b),\\ \alpha&= 2^{m-m^\prime}, \qquad\beta= 2^{b^\prime-m^\prime}/a_P,\\ l^\prime&= \alpha l+\beta\sum_k\widehat{p}_k,\\ o^\prime&= \alpha o+\beta\sum_k\widehat{p}_kz_k. \tag*{(10)} \end{aligned}

The same quantized weights update the denominator and weighted value sum. Protected tiles contribute through FP16 PV into the same FP32 attention state. After the retained tiles have been processed, value restoration gives

O=μV+σV⊙(o/l).(11)O=\mu_V+\sigma_V\odot(o/l). \tag*{(11)}

Compact Traversal and Compressed-Cache Execution

From boundaries to local work. The contiguous distance ranges let each query block construct its traversal from the two stored boundaries. Processing the protected, 8-bit QK, and 4-bit QK groups separately reuses the corresponding query representation while successive K/V tiles are staged. On RTX 4090, direct index calculation and reduced value staging lower the per-thread-block shared-memory footprint. Across thread blocks, KV-group-major scheduling places heads sharing K/V together and visits query blocks in descending order to group accesses to common operands.

Mapping logical retention to physical storage. GQA query heads share physical K/V, so cache construction combines their requested prompt tiles into Rg=⋃h∈HgRhR_g=\bigcup_{h\in\mathcal{H}_g}R_h, where RhR_h is head hh’s retained set at the final prompt query block. Each tile is stored once at the highest requested key precision. A head mask preserves the participating query heads, and per-KV-head offsets locate the packed tiles in contiguous buffers. This shares stored operands while retaining each head’s selected context.

Values use INT8 for the first and most recent 16 prompt blocks and packed INT4 elsewhere in the retained set. At head dimension 128, a 64-token INT4 K or V tile uses 4 KiB, half the 8 KiB required by INT8, before scales and metadata. The packed prompt cache is reused throughout generation, while new K/V enter the BF16 tail.

Combining prompt and generated context. The prompt numerator is accumulated in normalized V′V^\prime coordinates, whereas the BF16 tail numerator uses original values. They must be reconciled before the final normalization. Let (mp,lp,op)(m_p,l_p,o_p) be the prompt state obtained by aligning and merging the thread-block partials, and let (mt,lt,ot)(m_t,l_t,o_t) be the tail state. Tail scores use the prompt’s key-centering offset. With m=max⁡(mp,mt)m=\max(m_p,m_t), the final output is

O=2mp−m(σV⊙op+μVlp)+2mt−mot2mp−mlp+2mt−mlt.(12)O=\frac{2^{m_p-m}(\sigma_V\odot o_p+\mu_Vl_p)+2^{m_t-m}o_t}{2^{m_p-m}l_p+2^{m_t-m}l_t}. \tag*{(12)}

The term σV⊙op+μVlp\sigma_V\odot o_p+\mu_Vl_p restores the unnormalized prompt numerator; the exponential factors align prompt and tail maxima before combining numerators and denominators.

Experimental Protocols

Models, Tasks, and Comparison Settings

The evaluated checkpoints are Llama-3.2-3B-Instruct, Qwen2-7B-Instruct, and Ministral-3-8B-Instruct-2512-BF16. Their respective numbers of query heads, KV heads, and decoder layers are (24,8,28)(24, 8, 28), (28,4,28)(28, 4, 28), and (32,8,34)(32, 8, 34), with head dimension 128. The original projection weights are retained, and the attention implementation consumes their Q/K/V outputs.

General quality. General evaluation uses the Language Model Evaluation Harness (Gao et al., [8]) with the recorded task configurations and dataset snapshots. MMLU uses five-shot prompts and subject-macro accuracy. These likelihood-based evaluations run the prefill attention path with the model’s native positional configuration.

Long-context QA. LV-Eval (Yuan et al., [27]) uses the official dataset prompts and task-specific scorers at the 16K, 32K, and 64K dataset levels, with batch size one, greedy generation, and each task’s configured response limit. Each prompt is processed in a single prefill pass before answer generation. Scores are reported separately for each dataset–length pair on a 0–100 scale, using keyword-aware F1 or DuReader ROUGE-L as specified by the benchmark. The length levels identify the supplied datasets; actual tokenized lengths follow the model’s tokenizer.

References and allocation controls. BF16 uses default PyTorch SDPA dispatch, while Torch attention restricts SDPA to its math backend. The all_int4 reference retains all causal tiles and changes non-protected QK computation to INT4; the protected first and diagonal tiles retain INT8 QK and FP16 PV. MoA loads FP16 model weights, while the other quality comparisons use BF16 weights. The H100 quality environment uses Python 3.12, PyTorch 2.10 with CUDA 12.6, and Transformers 5.18.

Layer Same selects one boundary pair per layer by aggregating its heads’ candidate errors and sharing the resulting pair across those heads. All Same assigns (w8,w4)=(512,512)(w_8, w_4) = (512, 512) throughout the model, corresponding to a 32K local 8-bit range with the first block protected. Random uses seed zero to permute the deployed boundary pairs across layers and heads, preserving the configuration collection and its aggregate modeled prefill cost.

Measurement and Ablation Protocols

Complete-request measurements. System evaluation uses an NVIDIA A100-SXM4 40 GB GPU, H100 NVL, and GeForce RTX 4090. Within each model, comparisons reuse the same stored input token IDs across methods and GPUs. The core grid varies prompt length over {128,2048,8192,16384,32768,65536,131072}\{128, 2048, 8192, 16384, 32768, 65536, 131072\}, batch size over {1,2,4,8}\{1, 2, 4, 8\}, and output length over {128,256,512,1024}\{128, 256, 512, 1024\}. The figures display the indicated prompt groups, with out-of-memory outcomes marked separately. Qwen2 system runs apply fourfold YaRN scaling (Peng et al., [17]) across methods. For prompts of at least 64K tokens, feed-forward computation is processed in 16K-position chunks across methods.

Two warm-up runs precede five timed repetitions. Each repetition generates 1,024 tokens greedily, with synchronized checkpoints at the reported output lengths. For a checkpoint containing LoutL_{\mathrm{out}} generated tokens per request, TPOT is the elapsed time from the first token to that checkpoint divided by Lout−1L_{\mathrm{out}} - 1. Complete-request throughput divides the batch’s generated-token count by elapsed time from request start, including prefill. Latency, throughput, and energy use medians across repetitions, while peak GPU allocation uses the maximum observed allocation. NVIDIA Management Library (NVML) measures GPU energy during generation, reported in joules per 1,000 generated tokens.

Before timing, CUDA graphs capture decode and prefill with prompts up to 16K tokens. Timed requests include operand preparation, attention, and prompt-cache construction. Baselines use released generation interfaces.

Attention-level timing and storage. The efficiency ablations use seeded BF16 Q/K/V tensors with each model’s head geometry and per-layer boundary table. CUDA events time complete attention calls, and per-layer median times are summed over the indicated layers. Prefill timing includes Q/K preparation, V packing, and attention computation; decode timing measures one attention step after prompt-cache construction. The PV comparison uses the first, middle, and last decoder layers. Prompt-cache storage counts retained K/V and associated metadata after prefill, excluding model weights, generated token storage, and temporary workspace.

Design Validation

Head-specific allocation determines where attention computation is retained. Its execution cost also depends on the available precision levels, operand-staging space, and the representation accessed at every decode step.

Value-Aggregation Precision

Value-aggregation precision. Retained tiles still require PV computation, making its precision another source of execution cost. With retained tiles and 8-bit QK fixed, Table 5 changes non-protected PV from 16 to 8 bits over the first, middle, and last layers. The 8-bit path is faster in all settings, reducing Ministral’s 128K latency from 882.1 to 762.2 ms. Lower-bit value aggregation therefore complements the savings from retention and QK precision.

ModelContext8-bit PV (ms)16-bit PV (ms)
Llama32K72.979.2
64K251.7275.3
128K865.0992.7
Qwen232K77.183.6
64K269.4295.1
128K985.21,089.4
Ministral32K64.569.7
64K216.0236.1
128K762.2882.1

Table 5. Effect of value-aggregation precision on A100 prefill latency. The comparison covers the first, middle, and last attention layers with identical retained tiles and 8-bit QK computation. Non-protected tiles use either 8-bit or 16-bit PV. Lower is better.

FINDINGS Low-bit PV reduces value-aggregation cost with retained tiles and QK precision fixed.

Compact On-Chip Execution and Scheduling

MetricOriginalCompact
Shared memory/CTA (KiB)3730
Resident CTAs/SM23
Resident warps/SM812

Table 6. (a) On-chip resource usage

ModelContextOriginal (s)Compact (s)
Llama32K0.5050.466
128K5.6095.149
Qwen232K0.5880.546
128K7.5657.036
Ministral32K0.8100.743
128K10.7259.813

Table 6. (b) Prefill attention latency

Shared-memory organization. Traversal metadata and staged K/V share the thread block’s limited on-chip storage, so reducing their combined footprint allows more blocks to reside on an SM. On RTX 4090, direct boundary-based indexing together with reduced value staging lowers shared memory per thread block from 37 to 30 KiB (Table 6). The resource reduction changes how many thread blocks fit on an SM: a 7 KiB decrease per block raises residency from two blocks to three. At 128K, the compact layout reduces prefill latency by 8.2%, 7.0%, and 8.5% for Llama, Qwen2, and Ministral, respectively; Ministral latency falls from 10.725 to 9.813 s. This links traversal metadata and value staging through their shared resource budget; reducing their combined footprint enables a different execution occupancy for the same attention computation.

Ordering work that shares K/V. Query heads sharing a KV head repeatedly access common operands, motivating KV-group-major scheduling. With the selected tiles fixed and outputs bitwise identical, this ordering reduces 128K prefill latency from 7.803 to 7.347 s for Llama and from 15.092 to 14.266 s for Ministral (Table 7).

ModelContextKV-Group-Major (s)Heaviest-First (s)
Llama32K0.6540.681
64K2.2422.359
128K7.3477.803
Qwen232K0.7810.782
64K2.7522.790
128K10.13410.339
Ministral32K1.0661.102
64K3.8564.053
128K14.26615.092

Table 7. Effect of KV-group-aware CTA scheduling on A100 prefill attention latency. Both schedules execute the same selected tiles and arithmetic operations, producing bitwise-identical outputs. KV-group-major scheduling places query heads sharing a KV head together to improve K/V data reuse. Lower is better.

FINDINGS Reducing the combined footprint of traversal metadata and operand staging increases thread-block residency and accelerates the same computation.

Prompt-Cache Storage and Decode

Persistent prompt storage. Cache size depends on both the number of retained tiles and their stored precision.

At 128K, Llama’s prompt cache occupies 14.00 GiB in BF16, 5.27 GiB with all tiles retained using 8-bit keys and mixed-bit values, and 3.74 GiB with the selected policy (Table 8). Low-bit representation supplies the first reduction; retention and key-precision selection save a further 29.0% over the full-retention compressed cache.

ModelContextBFI6 (GiB)Full-Retention Compressed (GiB)TriAttention (GiB)
Llama32K3.501.331.21
64K7.002.642.10
128K14.005.273.74
Qwen232K1.750.660.66
64K3.501.321.28
128K7.002.642.51
Ministral32K4.251.611.47
64K8.503.212.80
128K17.006.405.45

Table 8. Prompt KV-cache storage after prefill. The full-retention compressed cache stores all prompt tiles with 8-bit keys and mixed-bit values. TriAttention additionally applies its selected retention and key-precision policy. Sizes exclude model weights and temporary inference workspace. Lower is better.

Repeated decode work. The compressed prompt cache is read at every generation step, so its retention and precision choices also affect recurring attention cost. Full Retention keeps all prompt tiles with 8-bit keys, while 8-bit Values preserves the selected context and key precision and uses 8-bit values with matching 8-bit probability codes. At 128K, Ministral takes 5.19 ms with mixed-bit values and 6.52 ms with 8-bit Values (Table 9).

ModelContextTriAttention (ms)Full Retention (ms)8-bit Values (ms)
Llama32K1.621.681.95
Llama64K2.392.692.93
Llama128K3.794.774.72
Qwen232K0.930.931.32
Qwen264K1.811.822.15
Qwen2128K2.792.843.41
Ministral32K1.962.072.37
Ministral64K3.023.293.73
Ministral128K5.195.826.52

Table 9. Effect of prompt-cache retention and value precision on decode attention latency on A100. Each entry measures one decode step summed across all layers. Full Retention keeps all prompt tiles with 8-bit keys. The 8-bit Values control retains the selected context and key precision and uses 8-bit values with corresponding 8-bit probability codes on every retained tile. Lower is better.

The additional benefit of the selected cache policy grows with context length. For Llama, its storage reduction over the full-retention compressed cache increases from 9.0% at 32K to 29.0% at 128K. Over the same lengths, decode attention latency decreases from 1.68 to 1.62 ms and from 4.77 to 3.79 ms, corresponding to reductions of 3.6% and 20.5%. The paired measurements link the policy’s smaller persistent prompt cache to lower per-step attention cost.

FINDINGS Applying the selected policy to compressed KV storage reduces the persistent prompt working set and the attention cost repeated during generation.

Extended Evaluation

The selected boundary table is reused across tasks and request shapes. Additional models test whether reuse preserves prediction and evidence use, while the workload grids show how complete-request gains vary with generation length and GPU platform.

Workload and Hardware Scaling

The workload grids vary prompt processing, generation length, and batch-level parallelism. Each comparison uses the same model, GPU, and request shape. Torch attention provides the SDPA math-backend reference. Generation-throughput ratios divide method throughput by BF16 throughput, while TPOT speedups divide BF16 TPOT by method TPOT.

A100. Output length changes the balance between one-time prompt processing and repeated token generation. For Llama with a 16K prompt and batch size eight, increasing the output length from 128 to 1,024 tokens changes complete-request throughput from 137.07 to 394.97 tokens/s, compared with 88.55 to 156.58 tokens/s for BF16. The throughput advantage grows from 1.55×1.55\times to 2.52×2.52\times as the prompt cost is distributed across a longer continuation.

Figure 10. TPOT speedup on A100-SXM4-40GB. Qwen2-7B-Instruct. Rows use batch sizes 1 and 2; columns use 128, 2,048, 8,192 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×1\times; higher is better. Method colors and order are fixed across panels.

Figure 10 (continued). Rows use batch sizes 4 and 8; panel settings follow the first part.

Figure 11. Generation-throughput ratio on A100-SXM4-40GB. Ministral-3-8B-Instruct-2512-BF16. Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048, 8,192 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×1 \times; higher is better. Method colors and order are fixed across panels.

The long-context sweep reaches 42.95 tokens/s for Llama with a 128K prompt, 1,024 generated tokens, and batch size one, compared with 16.19 tokens/s for BF16, giving the 2.65×2.65 \times speedup. Figure 8 complements the main-text throughput grid, and Figure 9–Figure 12 cover the other models.

Figure 8. TPOT speedup on A100-SXM4-40GB. Llama-3.2-3B-Instruct. Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048, 8,192, 16,384 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×; higher is better. Method colors and order are fixed across panels.

Figure 9. Generation-throughput ratio on A100-SXM4-40GB. Qwen2-7B-Instruct. Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048, 8,192 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×1\times; higher is better. Method colors and order are fixed across panels.

Figure 12. TPOT speedup on A100-SXM4-40GB. Ministral-3-8B-Instruct-2512-BF16. Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048, 8,192 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×1 \times; higher is better. Method colors and order are fixed across panels.

Figure 13. Generation-throughput ratio on H100 NVL. Llama-3.2-3B-Instruct (prompt group 1 of 2). Rows use batch sizes 1 and 2; columns use 128, 2,048, 8,192 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×1 \times; higher is better. Method colors and order are fixed across panels.

Figure 13 (continued). Continued. Rows use batch sizes 4 and 8; panel settings follow the first part.

Figure 14. Generation-throughput ratio on H100 NVL. Llama-3.2-3B-Instruct (prompt group 2 of 2). Rows use batch sizes 1, 2, 4, and 8; columns use 128, 16,384, 32,768 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×; higher is better. The 128-token column repeats the first group as a shared reference. Panels share colors and order.

Figure 15. Figure 15. TPOT speedup on H100 NVL. Llama-3.2-3B-Instruct (prompt group 1 of 2). Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048, 8,192 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×; higher is better. Method colors and order are fixed across panels.

H100 NVL. On H100 NVL, Llama reaches 318.36 tokens/s with an 8K prompt, 128 generated tokens, and batch size four, compared with 40.42 tokens/s for Torch attention and 223.61 tokens/s for BF16 (Figures 13–Figure 18 and 21–24). These complete-request measurements give throughput gains of 7.88× and 1.42× over the respective references.

Figure 18. TPOT speedup on H100 NVL. Qwen2-7B-Instruct (prompt group 1 of 2). Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048, 8,192 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×; higher is better. Method colors and order are fixed across panels.

TPOT speedup on H100 NVL for Llama-3.2-3B-Instruct, prompt group 2 of 2

Figure 16. Figure 16. TPOT speedup on H100 NVL. Llama-3.2-3B-Instruct (prompt group 2 of 2). Rows use batch sizes 1 and 2; columns use 128, 16,384, 32,768 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×; higher is better. The 128-token column repeats the first group as a shared reference. Method colors and order are fixed across panels.

Figure 16 (continued). Rows use batch sizes 4 and 8; panel settings follow the first part.

Figure 17. Generation-throughput ratio on H100 NVL. Qwen2-7B-Instruct (prompt group 1 of 2). Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048, 8,192 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×; higher is better. Method colors and order are fixed across panels.

Generation-throughput ratio on RTX 4090 across batch sizes, input lengths, and output lengths

Figure 19. Generation-throughput ratio on RTX 4090. Llama-3.2-3B-Instruct. Rows use batch sizes 1 and 2; columns use 128, 2,048, 8,192 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×; higher is better. Method colors and order are fixed across panels.

RTX 4090. The compact-layout implementation also improves complete-request throughput on RTX 4090 (Figures 19–Figure 20 and 25–28). With an 8K prompt and batch size four, Llama reaches 167.69 tokens/s for 128 output tokens and 316.54 tokens/s for 1,024 output tokens, compared with 111.39 and 164.97 for BF16. The corresponding ratios increase from 1.51×1.51 \times to 1.92×1.92 \times, extending this trend to this GPU.

Figure 20. Figure 20. TPOT speedup on RTX 4090. Llama-3.2-3B-Instruct. Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048, 8,192 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×1 \times; higher is better. Method colors and order are fixed across panels.

Figure 19 (continued). Figure 19. Continued. Rows use batch sizes 4 and 8; panel settings follow the first part.

FINDINGS In A100, H100 NVL, and RTX 4090 comparisons, throughput gains over BF16 grow with continuation length. Longer continuations increase decode efficiency’s role in complete-request performance.

Figure 21. Generation-throughput ratio on H100 NVL. Qwen2-7B-Instruct (prompt group 2 of 2). Rows use batch sizes 1, 2, 4, and 8; columns use 128, 16,384, 32,768, 65,536 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×1\times; higher is better. The 128-token column repeats the first group as a shared reference. Panels share colors and order.

Figure 22. TPOT speedup on H100 NVL. Qwen2-7B-Instruct (prompt group 2 of 2). Rows use batch sizes 1, 2, 4, and 8; columns use 128, 16,384, 32,768, 65,536 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×1\times; higher is better. The 128-token column repeats the first group as a shared reference. Method colors and order are fixed across panels.

Figure 23. Generation-throughput ratio on H100 NVL. Ministral-3-8B-Instruct-2512-BF16. Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048, 8,192, 16,384 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×1 \times; higher is better. Method colors and order are fixed across panels.

Figure 24. TPOT speedup on H100 NVL. Ministral-3-8B-Instruct-2512-BF16. Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048, 8,192, 16,384 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×1 \times; higher is better. Method colors and order are fixed across panels.

Figure 25. Generation-throughput ratio on RTX 4090. Qwen2-7B-Instruct. Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×1 \times; higher is better. Method colors and order are fixed across panels.

Figure 27. Generation-throughput ratio on RTX 4090. Ministral-3-8B-Instruct-2512-BF16. Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×1 \times; higher is better. Panels share colors and order.

Figure 26. TPOT speedup on RTX 4090. Qwen2-7B-Instruct. Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×1 \times; higher is better. Method colors and order are fixed across panels.

Figure 28. TPOT speedup on RTX 4090. Ministral-3-8B-Instruct-2512-BF16. Rows use batch sizes 1, 2, 4, and 8; columns use 128, 2,048 input tokens. Each panel varies output length over 128, 256, 512, and 1024 tokens. BF16 is normalized to 1×1 \times; higher is better. Method colors and order are fixed across panels.

General and Long-Context Quality

General prediction across models. Each additional model reuses its calibrated boundary table across tasks. Qwen2-7B-Instruct and Ministral-3-8B-Instruct-2512-BF16 match BF16 word perplexity at the reported precision, at 9.68 and 9.73, respectively, and all six accuracy gaps are below 0.6 percentage points (Table 10 and Table 12). Near-reference predictions support model-level allocation across language modeling, reasoning, and understanding tasks.

BenchmarkMetricUpper Ref.Lower Ref.OursOthers
BF16all_int4TriAttentionMInferenceFlexPrefillSageAttnMoASpargeAttn
WikiText-2PPL ↓\downarrow9.689.739.689.6710.159.689.759.70
HellaSwagAcc. ↑\uparrow80.6280.6281.1280.6380.6580.6580.6580.62
MMLUAcc. ↑\uparrow71.6271.1672.0771.4571.6071.3471.6471.51
ARC-ChallengeAcc. ↑\uparrow62.3762.2962.9662.3762.5462.4662.3762.29
LAMBADAAcc. ↑\uparrow70.0469.9470.4470.0669.8469.9470.1170.06
PIQAAcc. ↑\uparrow80.5280.5281.0280.5280.4780.7980.5280.52
WinoGrandeAcc. ↑\uparrow76.5676.3276.8276.2476.4876.8076.2476.95

Table 10. General-task results for Qwen2-7B-Instruct. Accuracy is reported in percent (higher is better); WikiText-2 uses word perplexity (lower is better). BF16 is the upper reference; all_int4 is the lower reference. Colors compare with BF16: green denotes equal or better results, blue worse results, and gray BF16. Bold marks the best result among Ours and Others; displayed ties share the mark.

BenchmarkMetricUpper Ref. BF16Lower Ref. all_int4Ours TriAttentionOthers MInferenceOthers FlexPrefillOthers SageAttnOthers MoAOthers SpargeAttn
WikiText-2PPL ↓9.739.759.739.729.849.739.899.75
HellaSwagAcc. ↑78.7978.8079.3078.8478.8578.9578.8778.85
MMLUAcc. ↑77.8577.6878.3577.8277.9277.9477.6577.60
ARC-ChallengeAcc. ↑69.0369.1169.6169.2069.0369.0368.9468.94
LAMBADAAcc. ↑70.9971.0771.5371.1271.0571.2271.4371.12
PIQAAcc. ↑80.5280.3680.8680.3080.3080.5880.3680.30
WinoGrandeAcc. ↑77.9877.1178.4077.5177.1977.3577.5877.51

Table 12. General-task results for Ministral-3-8B-Instruct-2512-BF16. Accuracy is reported in percent (higher is better); WikiText-2 uses word perplexity (lower is better). BF16 is the upper reference; all_int4 is the lower reference. Colors compare with BF16: green denotes equal or better results, blue worse results, and gray BF16. Bold marks the best of Ours and Others, including displayed ties.

Long-context answer generation. LV-Eval additionally exercises the compressed prompt cache throughout generation. At 64K, Qwen2 scores 29.70 on CMRC, compared with 27.46 for BF16 and 26.15 for SpargeAttn; Ministral scores 20.71 on LooGLE-MIR, compared with 19.03 and 18.89, respectively (Table 11 and Table 13). The same model-specific boundaries therefore preserve useful context for generated answers as well as likelihood-based prediction.

DatasetLevelUpper Ref.Lower Ref.OursOthersAblations
BF16all_int4TriAttentionMInferenceFlexPrefillSageAttnMoASpargeAttnLayer SameAll SameRandom
①16K23.8522.2223.4929.0026.1332.6016.3220.8522.9421.7823.12
32K24.7419.9925.6515.0011.8916.9016.5521.1923.0123.2620.49
64K27.4620.6829.706.408.989.1019.4026.1524.0517.0721.15
②16K16.0014.0716.6214.1015.7414.2014.8615.8714.5614.7413.59
32K16.4214.9416.8811.5015.2413.5016.2015.6115.8715.6016.03
64K13.7012.2014.459.5012.6510.8013.4913.1913.8911.8313.07
③16K12.4316.6413.3614.3012.2112.1211.9112.7411.8912.2712.76
32K9.3314.049.9411.4615.169.7610.508.739.857.9810.51
64K11.8213.2112.1811.1513.1410.849.9510.977.6910.247.09
④16K14.8914.8216.8715.7915.4715.6513.7515.6714.6215.0715.37
32K16.0412.5314.1114.8814.1713.6610.4413.6111.9513.3012.15
64K14.0112.9114.6912.7813.9613.7011.6212.9412.909.9010.95
⑤16K18.2418.9420.3114.9017.0915.8016.7417.1418.1515.3015.12
32K18.0418.1518.5812.6016.0813.8017.7818.4815.3716.4616.37
64K17.5716.0117.709.0010.2911.4014.4914.7516.6012.8816.31
⑥16K17.3315.0217.2715.2013.6515.1014.0816.0615.9315.3517.47
32K15.4114.8815.9110.8012.2610.8013.9915.4414.1911.7615.55
64K14.6714.7914.318.109.469.5010.8313.3711.878.7310.91
⑦16K38.4934.9638.6528.5029.0731.4030.6337.3837.9437.1537.40
32K39.4533.0539.0318.1020.7121.0029.0238.0536.2829.4934.35
64K34.8932.0232.7711.4014.5313.7021.0433.1333.1715.3224.09
⑧16K30.1429.0831.3421.2023.8922.6022.2429.6528.5928.7326.49
32K25.8324.1526.3115.3017.6717.9022.7322.9224.6219.1423.06
64K25.5524.1727.6612.8015.9515.9021.1024.3727.4019.5823.33
⑨16K35.4235.8636.8128.7022.7431.4028.4532.2634.4132.4632.15
32K29.5227.2828.9419.3015.9823.5020.4828.9825.2927.3026.35
64K27.0522.5525.3813.0015.2216.1023.7926.1823.7421.5123.52

Table 11. LV-Eval QA results for Qwen2-7B-Instruct. Dataset identifiers follow § 5.4. Scores use official task-specific metrics on a 0–100 scale; higher is better. Length labels follow the official dataset levels. BF16 is the upper reference; all_int4 is the lower reference. Colors compare with BF16: green denotes equal or better results, blue worse results, and gray BF16. Bold marks the best result among Ours, Others, and Ablations; displayed ties share the mark.

DatasetLevelUpper Ref. BF16Lower Ref. all_int4Ours TriAttentionOthers MInferenceOthers FlexPrefillOthers SageAttnOthers MoAOthers SpargeAttnAblations Layer SameAblations All SameAblations Random
①16K39.5139.1440.5239.3038.4438.2221.5340.0239.0338.0131.35
①32K41.2039.4239.7742.3540.2939.4823.6139.2338.8735.7227.81
①64K40.1136.2639.5342.0236.7239.7922.3739.3739.6817.4620.14
②16K13.6813.9014.1113.5112.7812.6413.2312.9113.0513.3713.04
②32K15.2613.9315.8514.8415.6214.5113.8412.6814.4215.0312.82
②64K17.4615.4216.9114.8616.4415.6814.7815.8715.5714.2414.22
③16K31.1529.4832.0430.4826.8230.6931.4029.9531.4631.3629.52
③32K32.5833.2134.3429.9325.3430.7026.3331.8931.4025.5229.03
③64K25.9027.6327.8122.7023.2626.1122.7025.3523.0215.4621.49
④16K21.6520.7021.2219.6318.6119.9314.4920.1420.4720.3715.62
④32K21.2120.3421.1319.9820.1120.5214.6320.0920.7217.3815.51
④64K20.7720.3920.0420.2719.4319.5815.0019.4619.2010.3712.50
⑤16K27.6228.5230.4626.9127.3626.7928.8728.5428.2928.5627.51
⑤32K26.4626.9829.2225.9725.5725.3525.1427.8126.5522.6723.85
⑤64K26.9326.0327.3426.2325.7125.9920.8025.7227.0615.7919.82
⑥16K21.0720.0421.6420.7521.2519.6217.7919.4819.5118.6317.51
⑥32K19.4919.7620.0219.2319.9619.0816.3219.0818.4213.9215.93
⑥64K19.0319.3520.7118.1817.8717.9913.4418.8919.298.2711.44
⑦16K57.8956.5360.2657.9257.3655.7645.5057.0858.9056.9653.15
⑦32K57.3155.0457.5655.9253.8456.5442.8256.5356.3644.5539.54
⑦64K55.0353.1555.3356.5953.3254.9833.7454.0055.7522.0524.72
⑧16K33.1430.0534.9231.4130.1833.7723.8131.3931.0431.0026.39
⑧32K31.9030.0630.5027.6031.1230.2825.8830.1229.3725.1825.17
⑧64K31.2729.0533.8231.3928.9930.1125.4229.3233.8119.6421.01
⑨16K41.9744.1843.9541.9340.4941.4628.7944.5643.4941.8741.83
⑨32K41.7643.1042.7239.4136.0340.1826.8240.9139.1739.2534.34
⑨64K44.4343.8945.8045.8838.3440.6928.6840.6244.4326.5322.25

Table 13. LV-Eval QA results for Ministral-3-8B-Instruct-2512-BF16. Dataset identifiers follow § 5.4. Scores use official task-specific metrics on a 0–100 scale; higher is better. Length labels follow the official dataset levels. BF16 is the upper reference; all_int4 is the lower reference. Colors compare with BF16: green denotes equal or better results, blue worse results, and gray BF16. Bold marks the best result among Ours, Others, and Ablations; displayed ties share the mark.

Sensitivity-guided placement

Sensitivity-guided placement. Random preserves the selected boundary pairs and their aggregate modeled cost but changes which heads receive them. On LooGLE-SD at 64K, Qwen2 scores 32.77 with TriAttention and 24.09 with Random, while Ministral scores 55.33 and 24.72. All Same, which applies one 32K local configuration throughout the model, scores 15.32 and 22.05. The randomized control establishes the value of sensitivity-guided placement at fixed modeled cost, while the shared-range control examines restricting all heads to one context range.

The separation from Random widens at longer dataset levels. On LooGLE-SD, the gap increases from 1.25 points at 16K to 8.68 at 64K for Qwen2, and from 7.11 to 30.61 points for Ministral. Because Random preserves the boundary-pair collection and aggregate modeled cost, this trend identifies configuration placement as increasingly consequential in these longer-context evaluations. The same amount of modeled computation supports stronger QA quality when its retained ranges and precisions follow head sensitivity.

FINDINGS. At the same modeled computation cost, sensitivity-guided boundary placement preserves long-context QA quality better than random placement.

References

  1. [1]Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., et al. Longbench: A bilingual, multilingual benchmark for long context understanding. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), pp. 3119–3137, 2024.
  2. [2]Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations, volume 2024, pp. 35549–35562, 2024.arxiv.org/abs/2307.08691
  3. [3]Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344–16359, 2022.arxiv.org/abs/2205.14135
  4. [4]Ding, Y., Zhang, X., Wang, Y., Chen, Y., and Guo, J. Diagonal-tiled mixed-precision attention for efficient low-bit mxfp inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3420–3428, 2026.
  5. [5]Dong, J., Feng, B., Guessous, D., Liang, Y., and He, H. Flex-attention: A programming model for generating fused attention variants. Proceedings of Machine Learning and Systems, 7, 2025.
  6. [6]Fabbri, A. R., Li, I., She, T., Li, S., and Radev, D. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th annual meeting of the association for computational linguistics, pp. 1074–1084, 2019.arxiv.org/abs/1906.01749
  7. [8]Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. The language model evaluation harness, 07 2024a. URL https://zenodo.org/records/12608602.
  8. [9]Gao, Y., Zeng, Z., Du, D., Cao, S., Zhou, P., Qi, J., Lai, J., So, H. K.-H., Cao, T., Yang, F., et al. Seerattention: Learning intrinsic sparse attention in your llms. arXiv preprint arXiv:2410.13276, 2024b.arxiv.org/abs/2410.13276
  9. [10]Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.arxiv.org/abs/2407.21783
  10. [11]Hong, K., Dai, G., Xu, J., Mao, Q., Li, X., Liu, J., Chen, K., Dong, Y., and Wang, W. Flashdecoding++: Faster large language model inference with asynchronization, flat gemm optimization, and heuristics. Proceedings of Machine Learning and Systems, 6:148–161, 2024.
  11. [12]Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. Kvquant: Towards 10 million context length llm inference with kv cache quantization. Advances in Neural Information Processing Systems, 37:1270–1303, 2024.arxiv.org/abs/2401.18079
  12. [13]Jiang, H., Li, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y., et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. Advances in Neural Information Processing Systems, 37:52481–52515, 2024.arxiv.org/abs/2407.02490
  13. [14]Kang, H., Bharadwaj, S., Hensman, J., Krishna, T., Rühle, V., and Rajmohan, S. Turboattention: Efficient attention approximation for high throughputs llm. Proceedings of Machine Learning and Systems, 7, 2025.arxiv.org/abs/2412.08585
  14. [15]Lai, X., Lu, J., Luo, Y., Ma, Y., and Zhou, X. Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference. arXiv preprint arXiv:2502.20766, 2025.arxiv.org/abs/2502.20766
  15. [16]Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024.DOI
  16. [17]Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Efficient context window extension of large language models. In International Conference on Learning Representations, volume 2024, pp. 31932–31951, 2024.arxiv.org/abs/2309.00071
  17. [18]Sanovar, R., Bharadwaj, S., St Amant, R., Rühle, V., and Rajmohan, S. Leanattention: Hardware-aware scalable attention mechanism for the decode-phase of transformers. Proceedings of Machine Learning and Systems, 7, 2025.DOI
  18. [19]Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. Advances in Neural Information Processing Systems, 37:68658–68865, 2024.arxiv.org/abs/2407.08608
  19. [20]Sharratt, J. Thriftattention: Selective mixed precision for long-context fp4 attention. arXiv preprint arXiv:2605.23081, 2026.arxiv.org/abs/2605.23081
  20. [21]Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. In International Conference on Learning Representations, volume 2024, pp. 21875–21895, 2024.arxiv.org/abs/2309.17453
  21. [24]Yang, S., Guo, J., Tang, H., Hu, Q., Xiao, G., Tang, J., Lin, Y., Liu, Z., Lu, Y., and Han, S. Lserve: Efficient long-sequence llm serving with unified sparse attention. Proceedings of Machine Learning and Systems, 7, 2025.arxiv.org/abs/2502.14866
  22. [26]Yuan, J., Gao, H., Dai, D., Luo, J., Zhao, L., Zhang, Z., Xie, Z., Wei, Y., Wang, L., Xiao, Z., et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 23078–23097, 2025.arxiv.org/abs/2502.11089
  23. [27]Yuan, T., Ning, X., Zhou, D., Yang, Z., Li, S., Zhuang, M., Tan, Z., Yao, Z., Lin, D., Li, B., et al. Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k. arXiv preprint arXiv:2402.05136, 2024.
  24. [28]Zhang, J., Huang, H., Zhang, P., Wei, J., Zhu, J., and Chen, J. Sageattention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization. arXiv preprint arXiv:2411.10958, 2024a.
  25. [29]Zhang, J., Xiang, C., Huang, H., Wei, J., Xi, H., Zhu, J., and Chen, J. Spargeattention: Accurate and training-free sparse attention accelerating any model inference. arXiv preprint arXiv:2502.18137, 2025a.arxiv.org/abs/2502.18137
  26. [30]Zhang, J., Zhang, P., Zhu, J., Chen, J., et al. Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration. In International Conference on Learning Representations, volume 2025, pp. 71566–71585, 2025b.
  27. [31]Zhang, J., Wang, H., Zhang, P., Xu, X., Huang, H., Jiang, K., Chen, J., Zhu, J., et al. Sageattention3: Microscaling fp4 attention for inference and an exploration of 8-bit training. Advances in Neural Information Processing Systems, 38:53901–53931, 2026.
  28. [32]Zhang, Z., Liu, S., Chen, R., Kailkhura, B., Chen, B., and Wang, Z. Q-hitter: A better token oracle for efficient llm inference via sparse-quantized kv cache. Proceedings of Machine Learning and Systems, 6:381–394, 2024b.
  29. [33]Zhu, Q., Duan, J., Chen, C., Liu, S., Li, X., Feng, G., Lv, Y., Chuanfu, X., Lin, D., and Yang, C. Sampleattention: Near-lossless acceleration of long context llm inference with adaptive structured sparse attention. Proceedings of Machine Learning and Systems, 7, 2025.

Paper details

vlnc:W_tcwgsbk3
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 6

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents