WorksComputer Science
FOSCAR: Accelerating Decode Attention over 2- and 4-Bit KV Caches through Affine Folding
Abstract
Large-language-model decode attention streams the key-value (KV) cache for every generated token. Packing that cache into 2-bit or 4-bit quantized integers reduces memory traffic but does not automatically translate into latency saving. Existing 2-bit fused kernels reach under 31%/15% of peak HBM throughput on H100/B200. Converting each value and applying its scale and offset before feeding Tensor Cores is a major bottleneck because instruction-issue throughput is scarce on modern GPUs. Instead of applying scale and offset to each KV value, FOSCAR uses affine folding to apply them to the other operand or the output of decode-attention matrix products, depending on the quantization scheme (token-wise or channel-wise). This reduces scale-and-offset operations by up to two orders of magnitude while preserving the dequantized attention output in exact arithmetic. FOSCAR also uses bit manipulation to feed KV operands to Tensor Cores efficiently, requiring only 1 and 0.19 ALU instructions per INT2 element on H100 and B200, respectively. Finally, FOSCAR builds warp-specialized GPU pipelines for Hopper (H100) and Blackwell (B200) using wgmma and tcgen05. On H100, FOSCAR INT2 and INT4 kernels achieve geo-mean speedups of and over FlashInfer BF16, and and over TensorRT-LLM FP8. On B200, the corresponding INT2 and INT4 speedups are and over FlashInfer BF16, and over TensorRT-LLM FP8, and and over TensorRT-LLM NVFP4. We integrate FOSCAR into SGLang to benchmark serving efficiency. On long-context serving workloads, FOSCAR increases peak aggregate throughput by up to 170% and 341% over 2-bit and 16-bit baselines, respectively. We open-source FOSCAR as a kernel library at https://github.com/FutureMLS-Lab/FOSCAR.
Introduction
Autoregressive language-model decoding reads each active request’s key–value (KV) cache on every decode step. The cache footprint grows with batch size and context length; storing it consumes high-bandwidth memory (HBM) capacity, and streaming it from HBM often dominates decode-attention latency. Low-bit KV quantization reduces that traffic directly. KIVI [18], Kitty [31], OSCAR [40], QKVQuant (Hong et al., 2024), and QServe [17] store each cached value in 2–4 bits with a group-shared scale and offset.
Compressing the cache in memory does not eliminate on-chip operand preparation. KIVI [18], QServe [17], TurboAttention [13], BitDecoding (Du et al., 2026), SAW-INT4 [11], and OSCAR [40] all convert or reconstruct low-bit values before each matrix product. This preparation consumes instruction-issue throughput, registers, and shared-memory bandwidth even when fused into attention without writing dequantized K and V to HBM. Although bit-packing reduces HBM traffic, per-element operand preparation can still limit the resulting latency speedup.
Preprint.
FOSCAR applies affine folding to the query–key (QK) and probability–value (PV) matrix products, which reduce over channels and tokens. For token-wise K, the kernel scales each group’s partial QK result and adds the group’s offset times the matching query sum. For token-wise V, the kernel scales the attention weights before PV and adds a weighted sum of offsets to the output. Channel-wise K instead scales the query before QK, while channel-wise V scales each group’s partial PV result, and both add the corresponding offset-weighted sum. These per-group operations replace element-wise scale multiplication and offset addition, while bit manipulation constructs Tensor Core KV operands directly from packed integers. In exact arithmetic, affine folding produces the same attention output as dequantizing every KV value before the matrix products.
This paper makes four contributions:
Identify decode attention performance bottleneck. We compare per-byte reconstruction instruction demand against peak instruction-issue and HBM throughputs on H100 and B200, showing how per-element conversion and affine arithmetic limit low-bit decode speedups (§ 3).
Efficient Tensor Core operand preparation via affine folding. We formulate affine folding for independent token-wise or channel-wise K and V quantization, paired with bit-level Tensor Core operand construction, to feed Tensor Cores efficiently with lower instruction demand (§ 4.1 and § 4.3).
Fast GPU kernels for Hopper and Blackwell. We design warp-specialized decode pipelines for H100 register accumulators and B200 tensor-memory (TMEM) accumulators (§ 4.4).
Kernel and serving-engine evaluation. We evaluate FOSCAR in standalone kernel benchmarks and integrated into SGLang (Zheng et al., [39]). FOSCAR INT2 achieves geometric-mean kernel-latency speedups over FlashInfer BF16 of 3.36× on H100 and 2.67× on B200. For long-context serving workloads, FOSCAR INT2 increases peak aggregate serving throughput by up to 170% and 341% over 2-bit and 16-bit baselines, respectively (§ 6).
Background and Motivation
Decode attention with a quantized KV cache
Each decode step streams the cached K and V tensors from HBM, so low-bit storage reduces the bytes that attention must read. Asymmetric quantization stores each cached element as a -bit quantized value . Each quantization group shares a scale and an additive offset to reconstruct an element as
For either K or V, a token-wise group shares a scale and offset across channels within one token (Yao et al., [32]), whereas a channel-wise group shares them across tokens within one channel (Banner et al., [1]).
Limited latency savings in low-bit decode
Low-bit storage reduces the bytes that an attention kernel reads from HBM, but existing kernels still convert and dequantize each cached value before the matrix product. In Figure 1, two- and four-bit kernels achieve a much smaller fraction of peak HBM throughput on their KV traffic than BF16 kernels, whether they fuse dequantization into attention (Lin et al., [17]; Kang et al., [13]; Du et al., 2026; Jia et al., [11]; Zhou et al., [40]) or into separate matrix-vector kernels (Liu et al., [18]).

Figure 1. Effective KV access throughput of baseline kernels and FOSCAR at and as a percentage of peak HBM throughput.
Unpacking each integer value and applying its group scale and offset consume instruction-issue slots. As the stored bit width drops, HBM transfer time falls while this per-value instruction cost stays fixed, shifting the bottleneck from memory bandwidth to instruction issue. § 3 quantifies this instruction-issue cost and shows how affine folding removes per-value affine arithmetic.
Reconstruction limits low-bit decode attention speedups
A low-bit KV cache reduces HBM traffic, but the kernel must still construct floating-point operands fast enough to supply the Tensor Cores. In existing baselines, reconstruction involves converting integer values into floating-point operands, applying the affine transformation , and optionally performing one additional conversion to match the Tensor Core operand format (e.g., BF16 to FP8).
Low-bit baselines fall short of peak HBM throughput
We benchmark FlashInfer (Ye et al., 2025), TensorRT-LLM (NVIDIA Corporation, [27]), FlashAttention-2 (Dao, [2]), FlashAttention-3 (Shah et al., [30]), FlashAttention-4 (Zadouri et al., [34]), BitDecoding (Du et al., 2026), KIVI (Liu et al., [18]), Kitty (Xia et al., [31]), SAW-INT4 (Jia et al., [11]), and OSCAR (Zhou et al., [40]) across KV-cache storage formats. We compute effective KV access throughput by dividing the bytes in one read of the stored values, scales, and offsets by decode time. Figure 1 expresses this rate as a fraction of each GPU’s peak HBM throughput at batch size and context length . FlashInfer BF16 exceeds 90% of peak on both H100 and B200. On B200, hardware-native TensorRT-LLM NVFP4 reaches 72.4% of peak in the 4-bit group. Among integer 4-bit baselines, BitDecoding is highest at 70.0% on H100 and 36.1% on B200, while its INT2 kernel reaches only 30.7% and 14.5%, respectively. FOSCAR reaches 91.3% and 60.6% at INT4 and INT2 on H100, and 68.6% and 44.4% on B200. Although low-bit baselines read fewer KV-cache bytes, their effective KV access throughput remains far below peak because kernel latency is bounded by the slowest pipeline stage. § 3.2 shows how operand reconstruction can demand more instruction-issue throughput than the GPU provides at peak HBM throughput.
Reconstruction consumes instruction-issue slots
HBM transfers, operand preparation, softmax, and Tensor Core products all require instructions from the warp schedulers.
From H100 to B200, peak HBM throughput grows , while peak instruction-issue throughput grows only 11% (NVIDIA Corporation, [21, 23]). Peak dense Tensor Core throughput grows when comparing H100 FP8 with B200 FP4 (NVIDIA Corporation, [21]). Figure 2a compares these peak throughputs across GPU generations1.

Figure 2. GPU throughput scaling and per-byte instruction-issue demand for operand construction and affine evaluation (FOSCAR shown at four query heads per KV head and group size 128).
Narrower stored values reduce HBM traffic without proportionally reducing reconstruction work. We count the instructions to convert packed integers to BF16, evaluate in BF16, and convert the results to FP8 or FP4 when the Tensor Core instructions require those formats. One lane-slot corresponds to one instruction issued for one warp lane (NVIDIA Corporation, [24]). Let denote the lane-slots required to construct an operand and evaluate for one quantized value, the GPU’s peak instruction-issue throughput in lane-slots/s, and its peak HBM throughput in bytes/s. Each quantized value occupies bytes in INT4 and 0.25 bytes in INT2, excluding scales and offsets. Streaming the cache at peak HBM throughput remains within peak instruction-issue throughput only when
Figure 2b compares this per-byte issue demand against the hardware peak from Eq. (2) on H100 and B200.
Preparing four BF16 Tensor Core operands on H100 requires two SHF and two LOP3 instructions for INTFP conversion. Two BF16 HFMA2 instructions then evaluate the affine transformation , each producing two BF16 values in a 32-bit register (Frantar et al., [7]). The total is six instructions, or six lane-slots per warp lane. Four values occupy two bytes in INT4 or one byte in INT2, giving or lane-slots/byte in Figure 2b, respectively. Therefore, reconstruction alone can consume 30% and 60% of H100’s peak hardware budget of 9.99 lane-slots/byte, leaving little room for other parts of the kernel.
On B200, two F2FP instructions convert the four BF16 values directly to FP8 or FP4 and pack them into registers.2 Reconstructing four values therefore takes lane-slots, or lane-slots/byte for INT4 and lane-slots/byte for INT2. At peak HBM throughput, INT2 reconstruction exceeds B200’s hardware peak of 4.65 lane-slots/byte, and INT4 reconstruction alone consumes 86% of that peak.
FOSCAR removes per-value affine evaluation and never materializes a reconstructed K or V tile. Instead, it constructs Tensor Core operands directly from , applies each group’s scale to the other matrix input or partial product, and adds the offset as an output correction. At four query heads per KV head and group size 128, affine folding reduces issue demand in Figure 2b to 2.13 and 4.25 lane-slots/byte (INT4 and INT2) on H100, and to 0.88 and 1.00 on B200.

Figure 3b. Acc-fold scales a partial output column and adds the offset-weighted row sums of (Eq. (4)).
Figure 3. Affine folding of , where is the reduction axis.

Figure 3. Operand-fold scales a column of and adds the offset correction to every output column (Eq. (3)).
FOSCAR DESIGN
FOSCAR consists of three components:
Affine folding moves shared scales and offsets from individual KV values to matrix-product inputs and results so that Tensor Cores consume operands built directly from packed quantized values, reducing the affine demand in Figure 2b (§ 4.1).
Bit-level Tensor Core operand construction builds Tensor Core operands directly from packed quantized values, reducing the INT→FP conversion demand in Figure 2b (§ 4.3).
Warp-specialized GPU pipelines overlap data movement, operand preparation, matrix products, and affine corrections on Hopper and Blackwell (§ 4.4).
The affine folding mechanism
Consider , where , holds quantized values, and indexes the reduction. When each row of shares a scale and offset across columns , distributing reconstruction through the product gives
We call this operand-fold (Figure 3a): the scale multiplies the other input , and the offset contributes a separate reduction of without reconstructing individual entries of .
When each column of shares a scale and offset across the reduction index , both terms factor out of the sum:
We call this acc-fold (Figure 3b): the scale multiplies the accumulated product, and the offset multiplies the corresponding row sum of . Both operand-fold and acc-fold preserve the result of applying element-wise in exact arithmetic.
In attention, and arrive as tiles, with cached tokens and head dimension . Token-wise groups share a scale and offset along within each token, whereas channel-wise groups share them along within each channel. Because reduces over , token-wise uses acc-fold and channel-wise uses operand-fold into (Figure 4a). Because reduces over , token-wise uses operand-fold into and channel-wise uses acc-fold (Figure 4b). FOSCAR allows and to select their quantization and folding schemes independently according to the cache’s quantization policy. Figure 5 illustrates where scales and offsets are applied in the attention algorithm. FOSCAR retains online softmax [4, 3, 2] and changes only the QK and PV operand paths and their output corrections.

Figure 4. Affine folding modes for token-wise and channel-wise K and V tiles.

Figure 5. Affine folding in attention. Dashed arrows show where each K and V quantization scheme applies its scales.
With query heads per KV head and group size , affine folding reduces scalar scale-and-offset operations from one multiply-add per stored value to (Appendix C).
Affine folding spans multiple quantization groups
Under acc-fold, an unscaled partial product cannot cross a quantization-group boundary. When a reduction spans multiple groups, the kernel applies each group’s scale and offset to its partial product before adding the result to the running accumulator (Eq. (4). We call this partial product group-local because it sums only over the values that share one scale and offset. Operand-fold instead scales the other matrix input before multiplication, so the accumulator can span multiple groups without rescaling; its offset correction still follows Eq. (3).
Both the Hopper and Blackwell kernels enforce the same ordering constraints on these corrections, although they assign the work to different warpgroups (Sec. 4.4). For operand-fold, the kernel can accumulate offset contributions for the same output separately and add their sum later. QK corrections must precede softmax, while PV corrections must precede division by the accumulated softmax denominator.
Constructing Tensor Core operands from packed integers
FOSCAR constructs floating-point Tensor Core operands directly from packed integer bit patterns.3 Hopper combines each integer value with a fixed BF16 bit pattern, a technique known as mantissa injection [15, 7] (Du et al. 2026). Combining with via bitwise OR produces the exact BF16 value .
On Blackwell, FOSCAR can convert integer values into floating point more efficiently by directly placing each integer’s bit pattern in the low bits of a floating-point operand and filling the remaining bits with zeros. For example, the bit patterns of the INT2 values represent in FP4 (E2M1), giving . INT4 similarly yields FP8 (E4M3) operands with [16].
These constructions produce operands of the form () whose numerical values differ from the stored integers. To account for without extra per-element instructions, we fold both constants into each group’s shared scale and offset. Substituting into (Eq. (1)) gives
Affine folding then applies and in place of and , leaving the attention result unchanged in exact arithmetic (Sec. 4.1). and are precomputed in FP32 when the quantized KV cache is stored in HBM. Table 1 summarizes the mappings, the pairs used for compensation, and the ALU cost of converting each integer KV element into a Tensor Core operand.
| Mapping | ALU/elem. | |||
| Hopper | ||||
| INT4BF16 | 1.00 | |||
| INT2BF16 | 1.00 | |||
| Blackwell | ||||
| INT4E4M3 | 0.38 | |||
| INT2E2M1 | 0.19 |
Table 1. Tensor Core operand mappings and their static ALU costs per stored value. The column gives the bit-level construction of from .
Fig. 6a shows how FOSCAR packs INT2 K values in each 32-bit register to match these bit-level constructions. On Hopper, each byte interleaves pairs of 2-bit values so that one bit shift and one masked merge (LOP3) produce two BF16 operands in a 32-bit register. On Blackwell, each 32-bit register holds 16 two-bit values arranged as eight channel pairs; one bit shift and two masks unpack the register directly into two 32-bit registers holding 16 FP4 operands. Both kernels use the Tensor Memory Accelerator (TMA) to copy packed tiles from global to shared memory (Fig. 6b). Hopper consumers build K operands in registers and collectively issue wgmma; Blackwell prepares a shared-memory operand tile that a separate issue thread supplies to Tensor Cores by issuing tcgen05 [25]. Appendix § 9 extends the INT2 K example to the packing and dataflow of INT2 V and INT4 KV operands on both GPUs.
Warp-specialized GPU pipelines
We design separate warp-specialized pipelines for Hopper and Blackwell. Hopper’s wgmma accumulates into registers owned by its issuing warpgroup, whereas Blackwell’s tcgen05 accumulates into tensor memory (TMEM) accessible to other warpgroups in the CTA ([25]).
On Hopper, one producer warpgroup issues TMA copies, and two consumer warpgroups process alternating even and odd KV tiles (Figure 7a). Each consumer builds KV operands and runs its own –softmax– chain, keeping online-softmax state and output accumulators in registers. Each consumer applies scales and offset corrections according to Figure 5: K corrections precede softmax, and V corrections update the output as described in § 4.2. For channel-wise K, the producer warpgroup prepares the scaled inputs. At the end of the sequence, one consumer merges the two warpgroups’ softmax states and partial outputs into the final CTA output.

Figure 7. FOSCAR’s Hopper and Blackwell pipelines. FP(X) constructs X’s floating-point Tensor Core operand from packed integers (§ 4.3). corr. denotes the output correction, and WGx W[y] denotes warpgroup x, warp y. Dashed boxes execute only for the corresponding K or V quantization axis.
Blackwell assigns copy and matrix-product issue, operand preparation, softmax, and output accumulation to separate warpgroups (Figure 7b). The issue warpgroup starts TMA transfers and tcgen05 products, while the preparation warpgroup constructs KV operands in shared memory. The softmax warpgroup reads accumulators from TMEM, applies K offset corrections (and K scales for token-wise K) in FP32, computes the softmax weights , and scales for token-wise V (quantizing folded and tiles to E4M3 with a per-row scale). The output warpgroup reads accumulators from TMEM, maintains the running output, and applies V offset corrections (and V scales for channel-wise V); for channel-wise K, it also prepares the scaled inputs. Shared-memory buffers pass operands between warpgroups, while TMEM passes matrix-product accumulators.
Multi-stage buffers in shared memory and TMEM allow producer warpgroups to work on later tiles while downstream warpgroups process earlier tiles. We tune the buffer depth and producer lead distance empirically for each workload; for clarity, Figure 7b illustrates a one-step lead.
Serving Engine Integration
We integrate FOSCAR into SGLang as a backend for paged decode attention (Zheng et al., [39]). FOSCAR incorporates common optimization techniques, including split-KV, which divides long contexts across CTAs to increase parallelism. To better understand how FOSCAR benefits existing serving systems with quantized KV caches, we integrate FOSCAR with OSCAR (Zhou et al., [40]) as a drop-in replacement for its INT2 kernel and keep OSCAR’s quantization policy, rotations, and BF16 windows (§ 6.4) unchanged.
Evaluation
We implement FOSCAR in CuTe DSL (CUTLASS 4.8, CUDA 13.0) and evaluate it on NVIDIA H100 (80 GB HBM3) and B200 (180 GB HBM3e) GPUs with , , , page size 64, and token-wise K and V quantization groups of values with FP32 scales and offsets. Token-wise K and V exercise both acc-fold on QK and operand-fold on PV; Appendix § 10 evaluates all four combinations of token-wise and channel-wise K and V, where FOSCAR achieves 1.75–4.00× speedups over FlashInfer BF16 on H100 and 1.50–2.74× on B200. We evaluate batch sizes 1 and 8 at context lengths of 8K, 32K, 128K, and 512K, and batch size 64 at 8K, 16K, 32K, 64K, and 128K. Each measurement takes the median of five timed iterations (excluding an L2 flush before each call) after five warmups, and includes all decode kernels and split reduction. Plotted speedups normalize each kernel’s latency to FlashInfer BF16, whereas speedups quoted in the text compare FOSCAR directly against each named baseline within the same benchmark series (using five-run medians for B200 INT4 vendor baselines and three-run medians for BitDecoding and SAW).
Kernel performance
Figure 8 and Figure 9 compare FOSCAR against sub-byte and wider-format baselines. On H100, FOSCAR INT2 achieves a 3.36× geometric-mean speedup over FlashInfer BF16 and 2.03× over TensorRT-LLM FP8, with maxima of 4.15× and 2.44×. On B200, the geometric-mean speedups are 2.67× over BF16, 1.48× over TensorRT-LLM FP8, and 1.16× over TensorRT-LLM NVFP4, with maxima of 3.25×, 1.64×, and 1.26×. These vendor kernels are invoked through FlashInfer, and FOSCAR INT2 outperforms them across all tested batch sizes and context lengths. Against OSCAR INT2 and BitDecoding INT2, FOSCAR INT2 achieves geometric-mean speedups of 2.31× and 1.77× on H100, and 3.44× and 2.78× on B200.

Figure 8. Kernel speedups of FOSCAR and 2- and 4-bit baselines over FlashInfer BF16.

Figure 9. Kernel speedups of FOSCAR and 8- and 16-bit baselines over FlashInfer BF16. The dashed line marks parity.
On H100, FOSCAR INT4 achieves geometric-mean speedups of 2.83× over FlashInfer BF16, 1.71× over TensorRT-LLM FP8, 1.32× over BitDecoding INT4, and 4.19× over SAW INT4. On B200, FOSCAR INT4 achieves geometric-mean speedups of 2.30× over FlashInfer BF16, 1.28× over TensorRT-LLM FP8, 1.94× over BitDecoding INT4, and 6.93× over SAW INT4. It matches native NVFP4 overall (1.00×, ranging from 0.88× to 1.20×; faster at 32K–128K for batch 64), while returning BF16 output instead of NVFP4’s FP8 output. Appendix § 11 further evaluates how FOSCAR INT2 speedup scales across .
Kernel accuracy
Figure 11 compares the output error of BF16 decode (FlashInfer), FOSCAR INT4, FOSCAR INT2, and OSCAR INT2. To stress-test quantization, our synthetic input amplifies eight K channels, ramps K’s per-token scale along the sequence, and rescales K so that pre-softmax attention logits have unit standard deviation. For INT2, FOSCAR and OSCAR decode the same quantized cache written by OSCAR’s clipped quantizer without BF16 windows. Both INT2 kernels yield errors of 0.52–0.61, compared with 0.16–0.22 for FOSCAR INT4 and (2.13–2.25) × for BF16 decode. Under unclipped min–max INT2 quantization, the error rises to 0.86–1.03. Compared directly against the dequantized INT2 cache, FOSCAR’s kernel arithmetic error is only (1.68–1.76) × on both H100 and B200, matching BF16 decode and confirming that the 0.52–0.61 error stems from 2-bit quantization rather than affine folding.
Figure 11. L2 output error of BF16 decode (FlashInfer), FOSCAR INT4, FOSCAR INT2, and OSCAR INT2 on the synthetic input (logarithmic scale).
Kernel ablation
On INT2 caches with equal split-KV counts within each pair, Figure 10 ablates affine folding against per-value reconstruction on H100 and B200, and FP4 (E2M1) against FP8 (E4M3) KV Tensor Core operands on B200. On H100, FOSCAR’s speedup over external baselines combines its warp-specialized kernel design with affine folding, which adds a 1.07–1.15× speedup over per-value reconstruction (Figure 10a) as expected when H100 reconstruction remains within the hardware issue peak (Figure 2b). On B200, the reconstruction baseline follows the standard approach of dequantizing INT2 values into FP8 (E4M3) Tensor Core operands before each matrix product; because this per-value reconstruction exceeds B200’s issue peak, affine folding achieves a 7.36–8.86× speedup (Figure 10b). Switching the folded B200 INT2 kernel from FP8 (E4M3) to FP4 (E2M1) KV operands and their matching packing layout provides an additional 1.17–1.21× speedup (Figure 10c).

Figure 10. FOSCAR INT2 kernel latency ratios between ablation variants at equal split-KV counts (first variant’s latency divided by the second’s). Panels (b) and (c) decode an INT2 cache into E4M3 or E2M1 Tensor Core operands on B200; error bars show paired 95% confidence intervals across ten runs.
Serving throughput
We serve Qwen3-8B with 100K input tokens, 1K output tokens, and 99% or 95% prefix-cache hits (one cached prefix per request), sharing a BF16 prefill backend across configurations. We evaluate two INT2 configurations to show both realistic accuracy-preserving integration and pure 2-bit kernel speed: OSCAR INT2 + Ours replaces OSCAR’s INT2 decode kernel with FOSCAR while keeping OSCAR’s rotations and BF16 windows (128 initial and 512 recent tokens, < 0.7% of the 100K context), whereas Ours INT2 uses FOSCAR’s token-wise INT2 K and V cache without recent windows or rotations. We vary concurrency in powers of two up to 16 on H100 and 32 (plus 48) on B200 to measure per-user (median request output tokens/s) and aggregate (total output tokens/s) throughput (Figure 12).

Figure 12. Per-user and aggregate Qwen3-8B serving throughput on a single GPU with 100K input and 1K output tokens.
| Configuration | GPQA-Diamond (%) | GSM8K (%) |
| FlashInfer BF16 | 56.36 ± 1.84 | 95.71 ± 0.11 |
| OSCAR INT2 | 54.65 ± 2.18 | 95.74 ± 0.38 |
| OSCAR INT2 + Ours | 55.45 ± 1.49 | 95.92 ± 0.15 |
Table 2. Qwen3-8B serving accuracy on H100 (mean ± sample standard deviation across five runs).
At a 99% prefix-cache hit rate, OSCAR INT2 + Ours reaches a peak aggregate throughput of 487 tokens/s on H100 and 868 tokens/s on B200, compared with 269 and 333 tokens/s for OSCAR (81% and 161% higher). Ours INT2 reaches 506 tokens/s on H100 and 897 tokens/s on B200, improving over OSCAR by 88% and 170%; this also exceeds FlashInfer BF16’s peaks of 115 tokens/s on H100 (at concurrency 2) and 355 tokens/s on B200 (at concurrency 8) by 341% and 153%. At 95% prefix hits, OSCAR INT2 + Ours and Ours INT2 reach 337 and 346 tokens/s versus OSCAR’s 214 tokens/s on H100, and 738 and 755 tokens/s versus 297 tokens/s on B200. Across cache formats, peak throughput reflects both decoder speed and the number of concurrent requests that fit in memory.
Serving accuracy
Table 2 compares FlashInfer BF16, OSCAR INT2, and OSCAR INT2 + Ours on GPQA-Diamond (Rein et al., [29]) and GSM8K on H100, reporting the mean and sample standard deviation across five runs per configuration. Every question-paired 95% confidence interval for the accuracy difference between configurations spans zero, so five runs show no measurable accuracy gain or loss (though five runs do not establish formal equivalence). Median prompt lengths are 243 tokens for GPQA-Diamond and 91 tokens for GSM8K; because OSCAR INT2 + Ours decodes the identical quantized cache as OSCAR with < 1.8 × arithmetic error (§ 6.2), it preserves OSCAR’s cache accuracy while accelerating decode.
Related Work
KV-cache quantization algorithms
KIVI (Liu et al., [18]), KVQuant (Hooper et al., [10]), and Kitty (Xia et al., [31]) establish low-bit KV formats and motivate channel-wise grouping for K and token-wise grouping for V. Atom (Zhao et al., [38]) and QServe (Lin et al., [17]) co-design 4-bit quantization formats and serving kernels, while GEAR (Kang et al., [12]) adds low-rank and sparse error corrections. FOSCAR retains uniform affine quantization and changes how a Tensor Core kernel evaluates the quantized cache. Nonlinear lookup formats and sparse outlier representations cannot be expressed as a single affine transform and require separate kernel support.
Quantized attention kernels
BitDecoding provides low-bit Tensor Core attention kernels across several GPU generations (Du et al., 2026). On Hopper, it converts stored integer values and applies scales and offsets before the matrix products. On Blackwell, it uses block-scaled FP4 instructions that restrict scales to the contraction axis, omit offsets, and quantize and each tile to FP4. TensorRT-LLM’s XQA kernel folds a scalar INT8 or FP8 KV scale into the QK and PV accumulators [27]. QServe fuses low-bit KV dequantization into CUDA-core batched matrix-vector products [17]. TurboAttention converts sub-8-bit values into INT8 operands for Tensor Core attention [13]. The SageAttention kernels quantize full-precision , , and inputs for prefill and general attention workloads, including FP4 Tensor Core execution on Blackwell [37 a;b]. SAW-INT4 [11] combines token-wise affine INT4 caches with a block-diagonal Hadamard rotation, while OSCAR [40] combines a mixed INT2/BF16 cache with offline spectral rotations; both kernels reconstruct cached values before their matrix products. FOSCAR instead applies affine folding across independent token-wise or channel-wise and caches, folding group scales and non-zero offsets into the or inputs or into group-local partial products while keeping and in BF16 or FP8. Our serving evaluation combines affine folding with OSCAR’s cache policy and rotations (§ 6.4).
Attention pipelines and kernel reparameterization
FlashAttention-2 introduces work partitioning for tiled attention [2], FlashAttention-3 overlaps producer and consumer warpgroups through warp specialization [30], FlashInfer provides paged-cache attention kernels for serving engines (Ye et al., 2025), and FlashAttention-4 studies how Blackwell’s higher Tensor Core throughput shifts bottlenecks toward on-chip staging and non-matmul instructions [34]. RepVGG [5] folds training-time convolution branches into inference kernels, CODA [9] rewrites Transformer operators into GEMM epilogues, and quantized GEMM kernels [38, 17] rescale group partial sums or subtract zero-point activation sums in their epilogues. FOSCAR combines warp-specialized attention pipelining with scale-and-offset folding across both QK and PV under online softmax, where the eliminated per-element work grows with context length.
Quantized inference architecture
Oaken combines a KV quantization policy with custom hardware for online and offline compression [14]. Weight-only kernels such as Marlin [7, 16], FLUTE [8], and LUT-GEMM [28] optimize single matrix products over quantized model weights. FOSCAR instead targets the QK and PV matrix products of decode attention, which are linked by online softmax, on unmodified Hopper and Blackwell Tensor Cores.
Conclusion
FOSCAR accelerates decode attention over low-bit KV caches through affine folding, which moves each quantization group’s shared scale to a matrix-product input or partial result and applies its offset as an output correction. Its Hopper and Blackwell kernels construct Tensor Core KV operands from packed quantized values through bit manipulation, while affine folding preserves the attention output of the reconstructed cache in exact arithmetic. On H100 and B200, FOSCAR INT2 outperforms FlashInfer BF16 and TensorRT-LLM FP8 across all tested batch sizes and context lengths on the main benchmark grid. Qwen-8B serving experiments also show throughput gains of up to 170% over baselines in settings with long contexts and high cache hit rates that align with popular agentic workloads.
KV PACKING AND DATAFLOW
Figure 6 illustrates how FOSCAR packs INT2 K values and constructs Tensor Core operands on Hopper and Blackwell. The same approach extends to INT2 V and INT4 K and V because cache packing arranges quantized values to match each Tensor Core operand layout, and the decode kernel constructs floating-point operands with shifts and masks. Quantizing K or V along tokens or channels changes how scales and offsets are grouped and where affine folding applies them (§ 4.1), whereas the integer packing below depends only on the stored bit width and matrix operand layout.
Figure 6. Register packing and global-to-Tensor-Core dataflow for INT2 K in FOSCAR.
Hopper: packing for register operands
Hopper packs K and V according to the register fragments that each consumer thread supplies to wgmma. For QK, the K operand has token-by-channel coordinates, as shown in Figure 6a. For PV, the V operand has channel-by-token coordinates, so FOSCAR packs V values according to the transposed V tile. Both layouts let each consumer thread load contiguous packed words from shared memory and construct its register operands without gathering individual values.
Each 32-bit packed word places the two quantized values for a pair of BF16 operands in fields sixteen bits apart. An INT2 word holds sixteen values, forming eight pairs; an INT4 word holds eight values, forming four pairs. A shift aligns both fields with their BF16 mantissas, and a masked merge combines them with a fixed BF16 bit pattern. The stored bit width sets the field spacing and the number of pairs per word. K and V differ only in the logical coordinates of those pairs and share the same pairwise register construction. Affine folding compensates for the resulting operand mapping through Eq. (5).
The Hopper dataflow in Figure 6b therefore also applies to INT2 V and INT4 K and V. TMA copies the packed tiles from global to shared memory. Consumer threads construct BF16 operands in registers, and all four warps of a consumer warpgroup collectively issue wgmma. The QK product consumes K fragments, and the PV product consumes the transposed V fragments.
Blackwell: packing for shared-memory operands
The Blackwell kernel uses the same packed-word layout for K and V at each stored bit width. Both tensors store channels of a token together, with the values interleaved within each word for direct operand construction. The preparation warpgroup writes the constructed operands to the shared-memory layouts required by QK and PV.
INT2 K and V. Each 32-bit word holds sixteen integer values in eight four-bit groups. Group places in its low two bits and in its high two bits, for . For a packed word , the kernel constructs two operand words:
Each output holds eight E2M1 operands, with the unused bits of each four-bit operand set to zero. The patterns for represent , giving . Affine folding compensates with and . The two masks and one shift construct sixteen operands without numeric conversion, accounting for the INT2 cost in Table 1. This is the Blackwell construction shown for K in Figure 6a; V uses the same bit operations.
INT4 K and V. Each 32-bit word holds eight integer values in four bytes. Byte places in its low four bits and in its high four bits, for . The corresponding construction is
Each output holds four E4M3 operands, with each integer in the low four bits of its byte. For , these patterns represent . Affine folding compensates with and . The two masks and one shift construct eight operands, accounting for the INT4 cost in Table 1. Marlin WA48 uses the same INT4-to-E4M3 mapping for weights ([16]); FOSCAR applies it to unsigned KV values, whose construction needs no sign extraction.
Seven of the sixteen INT4 bit patterns represent subnormal E4M3 values, for which the PTX specification leaves matrix-operand behavior undefined ([25]). We test these subnormal values on B200 across both matrix-operand positions and both shared-memory layouts used by the kernel, and all matrix-instruction outputs match FP64 reference results.
For both stored bit widths, the Blackwell dataflow follows Figure 6b. TMA copies packed K and V tiles to shared memory, the preparation warpgroup constructs the QK and PV operands in shared memory, and a separate issue thread launches tcgen05. The operand format changes from E2M1 for INT2 to E4M3 for INT4, while the division of work across copying, operand preparation, and matrix-product issue remains the same.
Quantization-axis combinations
The main evaluation uses token-wise K and token-wise V with group size , which applies acc-fold to QK and operand-fold to PV (§ 4.1). To evaluate all four combinations of token-wise and channel-wise K and V, we benchmark FOSCAR INT2 and INT4 on H100 and B200 across and at , , , and , following the timing methodology of § 6 and verifying every output against an FP32 reference.
Figure 13 compares the decode speed of the three channel-wise K and V combinations against token-wise K and V (1.00) at the same stored bit width and shape. On H100 (Figure 13a), the channel-wise combinations run at 0.84–1.04× the speed of token-wise K and V at INT2 and 0.91–1.07× at INT4, corresponding to 2.08–4.00× (INT2) and 1.75–3.35× (INT4) speedups over FlashInfer BF16 across all four axis combinations. On B200 (Figure 13b), the single-axis channel-wise combinations (K-ch / V-tok and K-tok / V-ch) run at 0.77–0.87× the speed of token-wise K and V at INT2 and 0.80–0.92× at INT4, while channel-wise K and V (K-ch / V-ch) runs at 0.57–0.73× at INT2 and 0.67–0.79× at INT4, corresponding to 1.50–2.74× (INT2) and 1.50–2.36× (INT4) speedups over FlashInfer BF16 across all four combinations.
Figure 13. Decode kernel speed across channel-wise K and V quantization combinations relative to token-wise K and V (1.0, dashed line) at the same stored bit width and shape (, , , group size ).
Channel-wise K and channel-wise V read more scales and offsets and perform per-group work along the token dimension: operand-fold on channel-wise K scales for each token group, while acc-fold on channel-wise V rescales group-local PV partial results across token groups. The relative gap is larger on B200 than on H100 for two architectural reasons: first, the Blackwell kernel tiles the sequence at 256 tokens rather than 64, so splits each sequence tile into two sub-tile groups and runs both the -plane builder and the per-group PV accumulator drain inside the correction warpgroup; second, Blackwell’s tcgen05 QK product consumes an FP8 E4M3 query operand rather than Hopper’s BF16 operand, so each channel-wise K group must compute a per-row maximum and power-of-two scale to requantize to E4M3. With KIVI-style channel-wise K and token-wise V (Liu et al., [18]; Hooper et al., [10]), FOSCAR runs at 0.89–1.04× of token-wise speed on H100 (2.42–3.56× over FlashInfer BF16) and 0.85–0.87× on B200 (2.11–2.38× over FlashInfer BF16) at INT2, and 0.96–1.07× on H100 (2.03–3.31× over FlashInfer BF16) and 0.87–0.92× on B200 (1.94–2.05× over FlashInfer BF16) at INT4.
Head-ratio and group-size trade-off
When query heads share one KV head and each quantization group spans values, affine folding performs one scale multiplication and one offset multiply-add per group for each query head, or scalar affine operations per stored value. Reconstructing each cached value once with and reusing it across the query heads costs one multiply-add per stored value. Figure 14 compares the scalar affine operations per value as and vary. At and , affine folding uses operations per value, a reduction. Break-even occurs at ; because standard grouped-query attention models use group sizes , affine folding requires up to fewer scalar affine operations than per-element reconstruction.

Figure 14. Scalar scale and offset operations per stored K or V value for affine folding versus element-wise reconstruction (dashed).
Figure 15 measures this trade-off empirically by varying from 1 to 32 ( from 32 down to 1) with , , and at 128K context for batches 1 and 8. At (), the smaller KV cache and higher ratio limit the gain over FlashInfer BF16 (0.91 and 1.10 on H100; 1.32 and 0.77 on B200). As grows, KV traffic increases while decreases, raising FOSCAR INT2’s speedup over FlashInfer BF16 to 3.89 at batch 1 and 7.68 at batch 8 on H100, and to 2.75 and 3.12 on B200 (Figure 15a and Figure 15b).
Figure 15. FOSCAR INT2 speedup over FlashInfer BF16 across at 128K context (, ).
References
- [2]Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=mZn2Xyh9Ec.arxiv.org/abs/2307.08691
- [3]Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/ 67d57c32e20fd0a7a302cb81d36e40d5–Abstract–Conference.html.arxiv.org/abs/2205.14135
- [4]Dao, T., Haziza, D., Massa, F., and Sizov, G. Flash-Decoding for long-context inference. PyTorch Blog, 2023. URL https://pytorch.org/blog/flash-decoding/. Accessed 2026-08-14.
- [5]Ding, X., Zhang, X., Ma, N., Han, J., Ding, G., and Sun, J. Repvgg: Making vgg-style convnets great again. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 13733–13742. Computer Vision Foundation / IEEE, 2021. doi: 10.1109/CVPR46437.2021.01352. URL https://openaccess.thecvf.com/content/CVPR2021/html/Ding_RepVGG_Making_VGG-Style_ConvNets_Great_Again_CVPR_2021_paper.html.DOI
- [7]Frantar, E., Castro, R. L., Chen, J., Hoefler, T., and Alistarh, D. MARLIN: mixed-precision auto-regressive parallel inference on large language models. In Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, PPoPP 2025, Las Vegas, NV, USA, March 1-5, 2025, pp. 239–251. ACM, 2025. doi: 10.1145/3710848.3710871. URL https://doi.org/10.1145/3710848.3710871.
- [8]Guo, H., Brandon, W., Cholakov, R., Ragan-Kelley, J., Xing, E. P., and Kim, Y. Fast matrix multiplications for lookup table-quantized llms. In Al-Onaizan, Y., Bansal, M., and Chen, Y. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, volume EMNLP 2024 of Findings of ACL, pp. 12419–12433. Association for Computational Linguistics, 2024. doi: 10.18653/V1/2024.FINDINGS-EMNLP.724. URL https://doi.org/10.18653/v1/2024.findings-emnlp.724.
- [9]Guo, H., Zhang, J., Menon, A., Guessous, D., Thakkar, V., Kim, Y., and Dao, T. CODA: rewriting transformer blocks as gemm-epilogue programs. CoRR, abs/2605.19269, 2026. doi: 10.48550/ARXIV.2605.19269. URL https://doi.org/10.48550/arXiv.2605.19269.
- [10]Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. Kvquant: Towards 10 million context length LLM inference with KV cache quantization. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J. M., and Zhang, C. (eds.), Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URL http://papers.nips.cc/paper_files/paper/2024/hash/028fcbcf85435d39a40c4d61b42c99a4–Abstract–Conference.html.arxiv.org/abs/2401.18079
- [11]Jia, J., Li, J., Zhou, Z., Heo, J. H., Wang, J., Dao, T., Song, S. L., Athiwatkun, B., Xu, C., Zhang, T., and Wu, X. SAW-INT4: system-aware 4-bit kv-cache quantization for real-world LLM serving. CoRR, abs/2604.19157, 2026. doi: 10.48550/ARXIV.2604.19157. URL https://doi.org/10.48550/arXiv.2604.19157.
- [12]Kang, H., Zhang, Q., Kundu, S., Jeong, G., Liu, Z., Krishna, T., and Zhao, T. GEAR: an efficient KV cache compression recipe for near-lossless generative inference of LLM. CoRR, abs/2403.05527, 2024. doi: 10.48550/ARXIV.2403.05527. URL https://doi.org/10.48550/arXiv.2403.05527.
- [13]Kang, H., Bharadwaj, S., Hensman, J., Krishna, T., Rühle, V., and Rajmohan, S. Turboattention: Efficient attention approximation for high throughputs llm. In Zaharia, M., Joshi, G., and Lin, Y. C. (eds.), Proceedings of the Eighth Conference on Machine Learning and Systems, MLSys 2025, Santa Clara, CA, USA, May 12-15, 2025. OpenReview.net/mlsys.org, 2025. URL https://openreview.net/forum?id=bPfpSO2zX4.arxiv.org/abs/2412.08585
- [14]Kim, M., Hong, S., Ko, R., Choi, S., Lee, H., Kim, J., Kim, J., and Park, J. Oaken: Fast and efficient LLM serving with online-offline hybrid KV cache quantization. In Proceedings of the 52nd Annual International Symposium on Computer Architecture, ISCA 2025, Tokyo, Japan, June 21-25, 2025, pp. 482–497. ACM, 2025. doi: 10.1145/3695053.3731019. URL https://doi.org/10.1145/3695053.3731019.
- [15]Kim, Y. J., Henry, R., Fahim, R., and Awadalla, H. H. Who says elephants can’t run: Bringing large scale moe models into cloud scale production. CoRR, abs/2211.10017, 2022. doi: 10.48550/ARXIV.2211.10017. URL https://doi.org/10.48550/arXiv.2211.10017.
- [16]Lin, J. Add W4A8 support for the Marlin kernel. vLLM pull request #24722, 2025. URL https://github.com/vllm-project/vllm/pull/24722. Accessed 2026-08-19.
- [17]Lin, Y., Tang, H., Yang, S., Zhang, Z., Xiao, G., Gan, C., and Han, S. QServe: W4A8KV4 quantization and system co-design for efficient LLM serving. In Zaharia, M., Joshi, G., and Lin, Y. C. (eds.), Proceedings of the Eighth Conference on Machine Learning and Systems, MLSys 2025, Santa Clara, CA, USA, May 12-15, 2025. OpenReview.net/mlsys.org, 2025. URL https://openreview.net/forum?id=1FfmStySS1.
- [18]Liu, Z., Yuan, J., Jin, H., Zhong, S. H., Xu, Z., Braverman, V., Chen, B., and Hu, X. KVII: A tuning-free asymmetric 2bit quantization for KV cache. In Salakhutdinov, R., Kolter, Z., Heller, K. A., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, volume 235 of Proceedings of Machine Learning Research, pp. 32332–32344. PMLR / OpenReview.net, 2024. URL https://proceedings.mlr.press/v235/liu24bz.html.DOI
- [19]NVIDIA Corporation. GPU Performance Background User’s Guide. NVIDIA Corporation, 2023. URL https://docs.nvidia.com/deeplearning/performance/dl-performance-gpu-background/index.html. Accessed 2026-08-12.
- [21]NVIDIA Corporation. NVIDIA H100 Tensor Core GPU. NVIDIA Corporation, 2025b. URL https://www.nvidia.com/en-us/data-center/h100/. Accessed 2026-08-05.
- [23]NVIDIA Corporation. CUDA C++ Best Practices Guide. NVIDIA Corporation, 2026a. URL https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html#arithmetic-instructions-throughput-native-arithmetic-instructions. Accessed 2026-07-30.
- [24]NVIDIA Corporation. CUDA C++ Programming Guide. NVIDIA Corporation, 2026b. URL https://docs.nvidia.com/cuda/cuda-c-programming-guide/. Accessed 2026-08-20.
- [25]NVIDIA Corporation. Parallel Thread Execution ISA, Version 9.4. NVIDIA Corporation, 2026c. URL https://docs.nvidia.com/cuda/parallel-thread-execution/index.html. Accessed 2026-10-29.
- [26]NVIDIA Corporation. NVIDIA Vera Rubin NVL72. NVIDIA Corporation, 2026d. URL https://www.nvidia.com/en-us/data-center/vera-rubin-nvl72/. Accessed 2026-08-05.
- [27]NVIDIA Corporation. TensorRT-LLM. NVIDIA Corporation, 2026e. URL https://github.com/NVIDIA/TensorRT-LLM. Accessed 2026-08-19.
- [28]Park, G., Park, B., Kim, M., Lee, S., Kim, J., Kwon, B., Kwon, S. J., Kim, B., Lee, Y., and Lee, D. LUT-GEMM: quantized matrix multiplication based on LUTs for efficient inference in large-scale generative language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=gLARhFLE0F.
- [29]Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. GPQA: A graduate-level google-proof q&a benchmark. CoRR, abs/2311.12022, 2023. doi: 10.48550/ARXIV.2311.12022. URL https://doi.org/10.48550/arXiv.2311.12022.
- [30]Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. In GloBersons, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J. M., and Zhang, C. (eds.), Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URL http://papers.nips.cc/paper_files/paper/2024/hash/7ede97c3e082c6df10a8d6103a2eebd2-Abstract-Conference.html.arxiv.org/abs/2407.08608
- [31]Xia, H., Wu, X., Li, J., Wu, R., Wang, J., Wang, J., Li, C., Singhal, A., Shah, A. D., Ariyak, A., Zhuang, D., Zhou, Z., Athiwaratkun, B., Zheng, Z., and Song, S. L. Kitty: Accurate and efficient 2-bit KV cache quantization with dynamic channel-wise precision boost. CoRR, abs/2511.18643, 2025. doi: 10.48550/ARXIV.2511.18643. URL https://doi.org/10.48550/arXiv.2511.18643.
- [32]Yao, Z., Aminabadi, R. Y., Zhang, M., Wu, X., Li, C., and He, Y. ZeroQuant: Efficient and affordable post-training quantization for large-scale transformers. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/adf7fa39d65e2983d724ff7da57f00ac-Abstract-Conference.html.
- [34]Zadouri, T., Hoehnerbach, M., Shah, J., Liu, T., Thakkar, V., and Dao, T. Flashattention-4: Algorithm and kernel pipelining co-design for asymmetric hardware scaling. CoRR, abs/2603.05451, 2026. doi: 10.48550/ARXIV.2603.05451. URL https://doi.org/10.48550/arXiv.2603.05451.
- [37]Zhang, J., Wei, J., Zhang, P., Zhu, J., and Chen, J. Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025c. URL https://openreview.net/forum?id=OL44KtasKc.arxiv.org/abs/2410.02367
- [38]Zhao, Y., Lin, C., Zhu, K., Ye, Z., Chen, L., Zheng, S., Ceze, L., Krishnamurthy, A., Chen, T., and Kasikci, B. Atom: Low-bit quantization for efficient and accurate LLM serving. In Gibbons, P. B., Pekhimenko, G., and Sa, C. D. (eds.), Proceedings of the Seventh Annual Conference on Machine Learning and Systems, MLSys 2024, Santa Clara, CA, USA, May 13-16, 2024. mlsys.org, 2024. URL https://proceedings.mlsys.org/paper_files/paper/2024/hash/5edb57c05c81d04beb716ef1d542fe9e-Abstract-Conference.html.arxiv.org/abs/2310.19102
- [39]Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C. W., and Sheng, Y. Sglang: Efficient execution of structured language model programs. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J. M., and Zhang, C. (eds.), Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10-15, 2024, 2024. URL http://papers.nips.cc/paper_files/paper/2024/hash/724be4472168f31ba1c9ac630f15dec8-Abstract-Conference.html.arxiv.org/abs/2312.07104
- [40]Zhou, Z., Zhuang, D., Li, J., Chen, Z., Song, S. L., Athiwaratkun, B., and Wu, X. OSCAR: offline spectral covariance-aware rotation for 2-bit KV cache quantization. CoRR, abs/2605.17757, 2026. doi: 10.48550/ARXIV.2605.17757. URL https://doi.org/10.48550/arXiv.2605.17757.



