WorksComputer Science
Beyond Compression Ratio: High-Throughput KV Compression for Disaggregated LLM Serving
Abstract
KV cache compression reduces the communication volume of disaggregated large language model serving, but its practical benefit depends not only on compression ratio and model quality, but also on how efficiently the compression pipeline executes. This requirement becomes increasingly important as modern serving systems adopt high-bandwidth interconnects, where inefficient compression can offset a substantial portion of the saved communication time. This paper presents FastKVComp, a high-throughput KV cache compression data path built on top of KVServe. FastKVComp improves compression execution in two complementary ways. First, it fuses the distribution transformation and low-bit quantization stages into a KVaware TileLang kernel, eliminating intermediate tensor materialization and redundant memory accesses. Second, it replaces the original lossless compression backend with an open, extensible, and searchable backend based on LC, enabling high-performance compression operator pipelines. The fused kernel achieves up to 5.9x speedup, while the new lossless backend increases compression throughput from below 90 GB/s to over 500 GB/s. Across the evaluated workloads, FastKVComp reduces job completion time by up to 2.04x. End-to-end evaluation shows that these optimizations allow KV compression to retain meaningful communication benefits under modern high-bandwidth network.
Introduction
Disaggregated large language model (LLM) serving increasingly separates computation and storage responsibilities across devices and nodes. Representative scenarios include prefill–decode disaggregation, remote KV cache offloading [15, 16, 11]. In these systems, the key-value (KV) cache is no longer only local inference state; it also becomes a major communication object whose volume grows with model size, batch size, and context length.
Recent KV cache compression methods reduce this communication volume through low-bit quantization, mixed-precision representations, distribution transformations, and lossless encoding [14, 7, 11, 13]. Most existing studies focus primarily on the tradeoff among compression ratio, model quality, and workload-specific compression policies. These studies show that KV caches are highly compressible, but they do not fully address the efficiency of executing the resulting compression pipeline.
Compression reduces end-to-end communication latency only when the saved transfer time is larger than the encoding and decoding overhead. This condition becomes more demanding as serving systems adopt 100–400 Gbps networks and high-performance remote direct memory access (RDMA) data paths [15, 16, 13]. Reducing KV volume is therefore not sufficient: compression must also execute fast enough to keep pace with modern serving networks.
We identify two major optimization opportunities in the execution path of practical KV cache compression. First, distribution transformation and quantization are commonly implemented as separate GPU kernels. Their intermediate tensor is written to high-bandwidth memory (HBM) and immediately read by the following quantization kernel, introducing redundant memory traffic, temporary storage, and kernel-launch overhead. Second, the original lossless compression backend used by KVServe [13] provides limited openness and extensibility for integrating and combining high-performance compression primitives. Its effective throughput is therefore insufficient to fully exploit high-bandwidth serving data paths.
To address these issues, we present FastKVComp, a high-throughput KV cache compression data path integrated into KVServe. FastKVComp fuses transformation and quantization into a single KV-aware TileLang kernel [18], avoiding the materialization of intermediate transformed tensors. It also introduces an open and searchable lossless compression backend based on LC [3, 1], which composes high-performance primitives into efficient compression pipelines for quantized KV data. These optimizations preserve the compression policies and model-quality characteristics of KVServe while substantially improving their execution efficiency.
Our preliminary implementation achieves up to acceleration for fused transformation and quantization. The new lossless compression backend improves effective throughput from below 90 GB/s to over 500 GB/s.
In summary, this paper makes the following contributions:
We characterize the execution bottlenecks of practical KV cache compression, revealing substantial overhead from fragmented GPU operators and the inflexible lossless compression path.
We design a KV-aware TileLang frontend that fuses distribution transformation and low-bit quantization, while optimizing the inverse reconstruction path through tiled execution and workspace reuse.
We develop an open and searchable lossless compression backend based on LC, which composes GPU primitives into pipelines optimized for end-to-end KV transfer.
We integrate the optimized data path into KVServe and evaluate it on both host-staged and cross-node InfiniBand deployments, achieving up to 2.04× JCT speedup and 2.01× throughput improvement over the original KVServe implementation.
Background and Gap Analysis
KV Transfer and Compression Pipeline
KV Cache Compression. KV caches grow linearly with context length and can occupy substantial memory for long-context requests. Existing compression methods reduce this footprint through low-bit quantization, mixed-precision representations, distribution transformations, and lossless coding [11]. While originally motivated largely by memory capacity, these techniques are increasingly important when KV states must move across devices or nodes.
Compression in KV Transfer. In disaggregated serving, KV caches are transferred between prefill and decode workers, offloaded to remote memory or storage, and reused across requests or serving sessions [16, 11]. These scenarios turn the KV cache into a large communication payload, often reaching gigabytes for large models and long contexts. Systems such as KVServe [13] therefore apply compression directly to the KV transfer path, reducing communication volume while preserving serving quality.
A typical compression path first transforms the KV distribution, quantizes the transformed values into a compact low-bit representation, and optionally applies lossless compression before transfer. The receiver performs the inverse operations to reconstruct the KV states required by attention. For an original payload of size , compressed size , and effective network bandwidth , compression is beneficial when the saved transfer time exceeds its encoding and decoding overhead:
As network bandwidth increases, this condition leaves progressively less room for compression overhead. Practical KV compression must therefore optimize not only compressed size, but also the efficiency of the compression data path itself.
Optimization Opportunities
Fragmented Transformation and Quantization. In the original execution path, transformation and quantization are implemented as separate GPU kernels. The transformation kernel reads the original KV tensor and writes a full intermediate tensor to HBM. The quantization kernel then reads this intermediate tensor, computes or applies the quantization parameters, and writes the low-bit output.
This design introduces two avoidable costs. First, the intermediate tensor incurs an additional HBM write and read, even though it is consumed only by the immediately following quantization stage. Second, separate kernels introduce additional launch and synchronization overheads. These costs are especially important because both stages are closely coupled and process the same KV tensor layout.
Figure 1. Original KVServe and the proposed FastKVComp execution paths.
Limited Lossless Compression Extensibility. The original KVServe implementation uses nvCOMP through its available integration path. Although nvCOMP provides several GPU compression algorithms, the original integration offers limited flexibility for inspecting, extending, and composing the underlying compression operations. In particular, it is difficult to incorporate new high-performance primitives or construct specialized operator pipelines for quantized KV data.
As a result, the effective end-to-end throughput of the original lossless compression stage remains below 100 GB/s in our initial measurements. This leaves substantial room for improving the execution efficiency of KV compression, particularly for systems equipped with modern high-bandwidth interconnects.
Based on these observations, we establish two design goals:
G1: Eliminate redundant memory traffic. Closely coupled transformation and quantization operations should execute within a single GPU kernel whenever possible.
G2: Build a high-throughput and extensible backend. The lossless compression layer should expose composable high-performance primitives and support efficient pipeline construction for quantized KV data.
Design Overview
FastKVComp is a high-throughput implementation of the KV compression data path below the compression-policy layer of KVServe. It preserves the original strategy-selection and compression semantics while optimizing how a selected strategy is executed.
FastKVComp consists of two components: a tiled TileLang frontend that streamlines transformation, quantization, and reconstruction using reusable GPU workspaces, and an LC-based lossless backend that provides composable compression primitives and offline pipeline search. At the sender, quantized KV data is written directly into a preallocated output buffer before lossless encoding. At the receiver, lossless decoding is followed by the optimized reconstruction path. The design prioritizes high throughput, low execution overhead, compatibility with existing KVServe policies, and extensibility to new compression components.
Ultra-fast KV Compression
Fused Transform–Quantization (Addressing G1)
To eliminate the redundant memory traffic identified as G1, we fuse transformation and quantization into a single tiled TileLang kernel (D1). The fused frontend combines the randomized Hadamard transformation, quantization-statistics computation, and low-bit conversion within a single GPU kernel.
Each KV tile is loaded once and processed using on-chip storage. Intermediate transformed values are directly consumed by the quantization stage instead of being materialized as a full-size tensor in global memory. The operator then writes the quantized representation directly into a preallocated all-layer output buffer, reducing temporary allocation, head splitting, per-layer stacking, layout conversion, and framework-level kernel-launch overhead.
The fused operator preserves the original KVServe compression policy, including the deterministic randomized Hadamard transformation, head-importance-aware mixed precision, channel-wise quantization for keys, token-wise quantization for values, and the original min–max quantization rule. Transformation signs, precision assignments, compiled TileLang operators, and auxiliary workspaces are cached and reused across layers and requests.
Fused Reconstruction. On the receiver, the TileLang reconstruction operator combines dequantization, inverse Hadamard transformation, normalization, and sign restoration in a tiled execution pipeline. It avoids materializing a full-size dequantized intermediate tensor and writes only the reconstructed KV representation to global memory. This optimization is independent of the selected lossless backend and therefore applies to both nvCOMP and LC.
Searchable High-Throughput Lossless Backend (Addressing G2)
To meet G2, we replace the fixed nvCOMP path with a searchable high-throughput lossless backend based on LC (D2). nvCOMP provides a readily available GPU-resident codec and achieves a high compression ratio. Such a design is effective when network transfer dominates the KV delivery path, because the reduction in transferred data can compensate for the additional encoding and decoding cost.
However, this tradeoff changes under high-bandwidth communication and after frontend optimization. Once transformation and quantization overhead is reduced, lossless encoding and decoding account for a larger fraction of the remaining critical path. Moreover, nvCOMP is distributed as a binary library with a fixed public interface, which limits the ability to inspect, recombine, or specialize its internal operations for low-bit KV data.
Unlike a fixed general-purpose codec, LC [3, 1] constructs compression algorithms from composable GPU primitives, including tuple reordering, bit shuffling, and redundancy encoding. This allows the backend to be specialized for the statistical properties of quantized KV buffers. It also enables pipeline selection to consider encoding latency, decoding latency, and compressed size jointly rather than optimizing compression ratio alone.
Offline Pipeline Search. We evaluate candidate two- and three-stage LC pipelines offline using quantized KV buffers generated by the optimized frontend. For each candidate, we measure GPU encoding time, GPU decoding time, and compressed size. We then retain the Pareto-efficient candidates and rank them for the evaluated high-bandwidth communication setting.
This procedure selects . Tuple reordering and bit shuffling expose regularities in the low-bit KV representation, while zero-oriented encoding removes the resulting redundancy. Although the selected LC pipeline provides a lower compression ratio than nvCOMP ANS, its substantially lower encoding and decoding latency makes it competitive on the end-to-end critical path. Once selected, the pipeline remains fixed during online serving, so no search is performed at runtime.
Native GPU Integration. We generate an in-process GPU runtime specialized for the selected LC pipeline. The runtime directly consumes the CUDA buffer produced by the fused TileLang frontend and returns a GPU-resident compressed bitstream. The online serving path therefore avoids LC’s file-based command-line interface, host-side payload serialization, and CPU–GPU movement of the compressed payload. Only a small compressed-size scalar is returned to the host during encoding.
The runtime exposes device-pointer encoding and decoding APIs directly to the KV transfer path. Persistent runtime scratch space and a reusable encode staging buffer reduce repeated allocation on the critical path. The component-level effects of this integration are evaluated in § 5.2.
Extensibility. Because LC is component based, additional transformations, encoders, and architecture-specific implementations can be incorporated into the offline search without modifying the upper-level KVServe control logic. The same selection method can therefore adapt the lossless backend to future KV formats, models, communication environments, and GPU architectures.
D1 removes the dominant transform/quantization overhead (G1), while D2 prevents lossless coding from becoming the next bottleneck (G2); together they keep the whole path fast enough for high-bandwidth transfer.
Evaluation
We evaluate FastKVComp in terms of end-to-end serving performance, component-level efficiency, and the contribution of individual optimizations.
Experimental setup. Our local experiments use two NVIDIA RTX 5090 GPUs (32 GB each) for prefill and decode, respectively, without tensor parallelism. KV payloads are transferred through the host-staged NCCL SHM path over PCIe Gen5 x16, with measured bandwidth up to 306.5 Gbps. The software stack includes CUDA 12.8, PyTorch 2.10.0, and vLLM 0.18.0 [9].
We additionally evaluate a two-node deployment connected by 200-Gb/s InfiniBand, using an NVIDIA RTX PRO 6000 for prefill and an NVIDIA H100 for decode. KV caches are compressed on the prefill node, transferred over InfiniBand, and reconstructed on the decode node.
Workloads. We evaluate Llama-3.1-8B-Instruct [5] and Qwen2.5-7B-Instruct [19] on HotpotQA, Qasper, and 2WikiMQA from LongBench [2]. Inputs contain up to 20K tokens under a 32K context window. Each configuration processes 50 requests, generating 8 tokens locally and up to 64 tokens in the remote deployment. Compilation, initialization, and warm-up costs are excluded.
Baselines and metrics. We compare Raw Transfer, Original KVServe, Fused Only, LC Only, and FastKVComp. All compressed configurations use identical compression policies, isolating the effects of operator fusion and the lossless backend. We report JCT, output throughput, KV transfer cost, compression ratio, and component-level compression and reconstruction overhead.
End-to-End Results
Figure 2 reports the end-to-end performance over the host-staged NCCL SHM path. Compared with Raw Transfer, FastKVComp improves output throughput by 1.83–2.04 and reduces JCT by 45.3–51.1% across both models and all workloads. Over Original KVServe, throughput improves by 1.87–2.01. Although Original KVServe reduces KV volume by 9.4–10.1, its fragmented compression path offsets most of the communication savings. In contrast, the fused configurations nearly double throughput, showing that efficient compression execution is critical at high network bandwidths.
Figure 2. End-to-end results over the host-staged NCCL SHM path.
Figure 3 further evaluates FastKVComp in the two-node 200-Gb/s InfiniBand deployment. Across the remote workloads, FastKVComp reduces JCT by 30.2–38.6% and improves output throughput by 44.2–62.6% over Raw Transfer. These results confirm that the optimized compression path remains effective in heterogeneous, high-bandwidth cross-node serving.
Figure 3. End-to-end JCT and output throughput over 200-Gb/s InfiniBand. Red annotations show the improvement of FastKVComp over Raw Transfer.
Component-Level Results
We isolate the costs of the fused frontend and the lossless backend using two representative KV requests.
Fused Transform–Quantization. The original transformation–quantization path takes 27.309–34.394 ms, whereas the fused TileLang implementation reduces this to 5.876–6.926 ms, yielding a 3.94–5.85 speedup and 74.6–82.9% lower preprocessing latency. The gain comes primarily from eliminating full-size intermediate KV materialization and reusing tiled workspaces, while preserving the original transformation and quantization semantics.
Lossless Compression Backend. Using the LC pipeline selected in § 4.2, lossless encoding latency drops from 3.179–3.274 ms with nvCOMP ANS to 0.443–0.531 ms, corresponding to a 6.0–7.4 speedup. Decoding similarly decreases from 2.543–2.666 ms to 0.488–0.575 ms, or about 4.9 on average. As shown in Figure 4, after frontend fusion, lossless encoding accounts for 31.7–35.1% of the encoding path with nvCOMP but only 6.1–8.2% with LC. This lower codec overhead comes at a moderate compression-ratio tradeoff: 8.850–9.168 for nvCOMP versus 6.344–6.360 for LC.
Figure 4. Encoding-time breakdown for the original and fused frontends using nvCOMP and the searched LC pipeline.
Complete Compression Path. Table 1 summarizes the complete compression and reconstruction paths on the GPUs used in the remote deployment. The critical transfer direction compresses KV caches on the RTX PRO 6000 prefill node and reconstructs them on the H100 decode node; results on both GPUs are reported for completeness.
| GPU | Path | KV comp. ratio (×) | Comp. throughput (GB/s) | Comp. latency (ms) | Decomp. throughput (GB/s) | Decomp. latency (ms) |
| RTX PRO 6000 | Original + nvCOMP | 8.867 | 34.773 | 30.879 | 32.284 | 33.260 |
| RTX PRO 6000 | Original + LC | 6.474 | 41.591 | 25.817 | 37.336 | 28.759 |
| RTX PRO 6000 | Fused + LC | 6.276 | 139.218 | 7.713 | 155.841 | 6.890 |
| H100 | Original + nvCOMP | 8.804 | 27.659 | 38.820 | 26.698 | 40.219 |
| H100 | Original + LC | 6.478 | 30.195 | 35.560 | 29.038 | 36.977 |
| H100 | Fused + LC | 6.280 | 91.606 | 11.721 | 103.104 | 10.414 |
Table 1. Component-level compression and decompression performance in the 200-Gb/s InfiniBand deployment. The RTX PRO 6000 and H100 serve as the prefill and decode GPUs, respectively.
On the RTX PRO 6000 prefill GPU, FastKVComp reaches 139.218 GB/s compression throughput with 7.713 ms latency, providing a 4.00 throughput improvement and 75.0% latency reduction over Original KVServe. On the H100 decode GPU, it reaches 103.104 GB/s decompression throughput with 10.414 ms latency, corresponding to a 3.86 throughput improvement and 74.1% lower latency. These results demonstrate substantial overhead reduction on both ends of the remote KV-transfer path.
For the two representative requests, the complete encoding path drops from 30.583–37.599 ms with Original KVServe to 6.400–7.376 ms with FastKVComp, yielding a 4.15–5.88 speedup and 75.9–83.0% latency reduction. Together, the component-level results explain the end-to-end gains: operator fusion removes the dominant frontend overhead, while LC prevents lossless coding from becoming the next bottleneck.
Ablation Study
Across both models and all three datasets, Fused Only improves output throughput by 1.87–2.01 over Original KVServe, confirming operator fusion as the primary source of end-to-end improvement under the evaluated SHM bandwidth. LC Only remains within 2.1% of Original KVServe; its lower codec latency is largely offset by the increased transfer volume from a lower compression ratio.
Fused Only and FastKVComp differ by at most 1.3% in output throughput. The former benefits from the higher compression ratio of nvCOMP, whereas FastKVComp favors the substantially lower codec overhead of LC. This tradeoff shows that the optimal lossless backend depends on the target communication regime rather than compression ratio alone.
Related Work
KV Cache Compression Existing KV cache compression methods reduce memory and communication costs through low-bit quantization [14, 7, 8], mixed precision [20], distribution transformations [12], sparsification [22, 10], and lossless coding [11, 13]. They improve the compression–quality tradeoff by reshaping KV distributions, allocating precision, or exploiting redundancy in quantized representations [21]. Recent systems combine KV compression with runtime management or GPU-friendly lossless coding [6], but complete pipeline efficiency remains less explored. FastKVComp is complementary: it preserves the quantization policy selected by KVServe while optimizing its execution.
Operator Fusion Operator fusion reduces kernel launches, intermediate tensors, and memory traffic. Frameworks such as Triton [17], TileLang, and TVM [4] support specialized fused kernels. FastKVComp applies these principles to the tightly coupled transformation–quantization pair, consuming transformed values within one tiled GPU kernel without global-memory materialization.
Conclusion
Compression throughput is a first-class metric for KV cache compression in disaggregated LLM serving. FastKVComp combines a KV-aware fused transformation–quantization kernel with an open, searchable lossless backend, addressing overhead in both the frontend and codec. The fused kernel achieves up to speedup, while the new backend exceeds 500 GB/s. These optimizations preserve the end-to-end benefits of KV compression over high-bandwidth networks.
Acknowledgments. This work was supported by the National Natural Science Foundation of China through the Excellent Young Scientists Fund Program (Overseas) (Project No. E401470), the National Key Research and Development Program of China (Grant Nos. 2020YFA0907000 and 2025YFB30037002), the National Natural Science Foundation of China (Grant Nos. 62032023 and T2125013), and Ant Group. The AI-driven experiments, simulations, and model training were performed on the robotic AI-Scientist platform of the Chinese Academy of Sciences.
Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article.
References
- [1]Azami, N., Fallin, A., Burtscher, M.: Efficient lossless compression of scientific floating-point data on CPUs and GPUs. In: Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. pp. 395–409 (2025). https://doi.org/10.1145/3669940.3707280
- [2]Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J., Li, J.: LongBench: A bilingual, multitask benchmark for long context understanding. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. pp. 3119–3137 (2024). https://doi.org/10.18653/v1/2024.acl-long.172
- [3]Burtscher, M., et al.: LC: Automatic generation of high-speed reliable lossy and lossless data compressors. https://userweb.cs.txstate.edu/~burtscher/LC/ (2025), accessed 2026-07-29
- [4]Chen, T., Moreau, T., Jiang, Z., Zheng, L., Yan, E., Shen, H., Cowan, M., Wang, L., Hu, Y., Ceze, L., Guestrin, C., Krishnamurthy, A.: TVM: An automated end-to-end optimizing compiler for deep learning. In: 13th USENIX Symposium on Operating Systems Design and Implementation. pp. 578–594. USENIX Association (2018), https://www.usenix.org/conference/osdi18/presentation/chenarxiv.org/abs/1802.04799
- [5]Dubey, A., et al.: The Llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024), https://arxiv.org/abs/2407.21783
- [6]Guo, Y., Joshi, S.: Splitzip: Ultra fast lossless kv compression for disaggregated llm serving. arXiv preprint arXiv:2605.01708 (2026)arxiv.org/abs/2605.01708
- [7]Hooper, C., Kim, S., Mohammadzeh, H., Mahoney, M.W., Shao, Y.S., Keutzer, K., Gholami, A.: KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In: Advances in Neural Information Processing Systems. vol. 37 (2024). https://doi.org/10.52202/079017-0040
- [8]Kang, H., Zhang, Q., Kundu, S., Jeong, G., Liu, Z., Krishna, T., Zhao, T.: GEAR: An efficient KV cache compression recipe for near-lossless generative inference of LLM. arXiv preprint arXiv:2403.05527 (2024), https://arxiv.org/abs/2403.05527
- [9]Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C.H., Gonzalez, J.E., Zhang, H., Stoica, I.: Efficient memory management for large language model serving with PagedAttention. In: Proceedings of the 29th Symposium on Operating Systems Principles. pp. 611–626 (2023). https://doi.org/10.1145/3600006.3613165
- [10]Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., Chen, D.: SnapKV: LLM knows what you are looking for before generation. In: Advances in Neural Information Processing Systems. vol. 37 (2024). https://doi.org/10.52202/079017-0722
- [11]Liu, Y., Li, H., Cheng, Y., Ray, S., Huang, Y., Zhang, Q., Du, K., Yao, J., Lu, S., Anantharayanan, G., Maire, M., Hoffmann, H., Holtzman, A., Jiang, J.: CacheGen: KV cache compression and streaming for fast large language model serving. In: Proceedings of the ACM SIGCOMM 2024 Conference. pp. 38–56 (2024). https://doi.org/10.1145/3651890.3672274
- [12]Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Krishnamoorthi, R., Chandra, V., Tian, Y., Blankevoort, T.: SpinQuant: LLM quantization with learned rotations. In: International Conference on Learning Representations (2025), https://arxiv.org/abs/2405.16406
- [13]Liu, Z., Ma, X., Luo, D., Zhao, H., Lu, B., Huang, W., Gu, Y., Liu, X., Wei, Z., Liu, J., Tao, D., Tan, G.: KVServe: Service-aware KV cache com pression for communication-efficient disaggregated LLM serving. arXiv preprint arXiv:2605.13734 (2026), https://arxiv.org/abs/2605.13734
- [14]Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., Hu, X.: KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In: Proceedings of the 41st International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 235, pp. 32332–32344 (2024), https://proceedings.mlr.press/v235/liu24bz.htmlDOI
- [15]Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, I., Maleki, S., Bianchini, R.: Splitwise: Efficient generative LLM inference using phase splitting. In: 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (2024). https://doi.org/10.1109/ISCA59077.2024.00019
- [16]Qin, R., Li, Z., He, W., Cui, J., Ren, F., Zhang, M., Wu, Y., Zheng, W., Xu, X.: Mooncake: Trading more storage for less computation—a KVCache-centric architecture for serving LLM chatbot. In: 23rd USENIX Conference on File and Storage Technologies. pp. 155–170. USENIX Association (2025), https://www.usenix.org/conference/fast25/presentation/qin
- [17]Tillet, P., Kung, H.T., Cox, D.: Triton: An intermediate language and compiler for tiled neural network computations. In: Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages. pp. 10–19 (2019). https://doi.org/10.1145/3315508.3329973
- [18]Wang, L., Cheng, Y., Shi, Y., Tang, Z., Mo, Z., Xie, W., Ma, L., Xia, Y., Xue, J., Yang, F., Yang, Z.: TileLang: A composable tiled programming model for AI systems. arXiv preprint arXiv:2504.17577 (2025), https://arxiv.org/abs/2504.17577
- [19]Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., et al.: Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 (2024), https://arxiv.org/abs/2412.15115
- [20]Yang, J.Y., Kim, B., Bae, J., Kwon, B., Park, G., Yang, E., Kwon, S.J., Lee, D.: No token left behind: Reliable KV cache compression via importance-aware mixed precision quantization. arXiv preprint arXiv:2402.18096 (2024), https://arxiv.org/abs/2402.18096
- [21]Yuan, J., Liu, H., Zhong, S., Chuang, Y.N., Li, S., Wang, G., Le, D., Jin, H., Chaudhary, V., Xu, Z., Liu, Z., Hu, X.: KV cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches. In: Findings of the Association for Computational Linguistics: EMNLP 2024. pp. 4623–4648. Association for Computational Linguistics (2024). https://doi.org/10.18653/v1/2024.findings-emnlp.266, https://aclanthology.org/2024.findings-emnlp.266/
- [22]Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., Wang, Z., Chen, B.: H2O: Heavy-hitter oracle for efficient generative inference of large language models. In: Advances in Neural Information Processing Systems. vol. 36 (2023). https://doi.org/10.52202/075280-1506



