Vision-language models (VLMs) applied to 3D medical scans face a structural memory wall. A single CT or MRI volume produces tens of thousands of vision tokens, and the resulting key-value (KV) cache scales linearly with both batch and context. Existing KV compression methods operate on the cache's feature or token statistics, or apply 1D and 2D transforms, leaving the three-axis spatial structure in the cache unexploited. We propose TACT-KV (Tri-Axis Cosine Transform), a calibration-free KV compression method that exploits strong spatial correlations in visual KV states along all three anatomical axes of 3D scans. This correlation concentrates the KV cache energy under a frequency-domain transform. TACT-KV applies a separable 3D discrete cosine transform to the KV cache, allocates a per-layer bit budget across frequency bins via reverse water-filling (more bits to high-energy components), and quantizes each bin with a fixed scalar codebook. We derive a close-form relation: the distortion reduction from adaptive bit allocation is dominated by the spectral concentration of the transformed cache. Across six medical VLMs and two 3D CT benchmarks, TACT-KV achieves the lowest 1-bit prediction-flip rate on eleven of twelve model and dataset pairs in our main comparison. The same advantage reproduces on four general-purpose VLMs evaluated on video tasks, indicating that the benefit of exploiting three-axis structure generalizes beyond medical imaging. At equal high-bandwidth memory (HBM), TACT-KV scales the decoding batch size on a single GPU by up to 8×.
https://github.com/Chongyu1117/TACT-KV
Introduction
Vision-language models (VLMs) are increasingly used for 3D medical image analysis [31], with applications spanning computed tomography (CT) report generation, magnetic resonance imaging (MRI) anomaly detection, and multimodal diagnostic assistance [33, 20, 27]. Unlike 2D natural images, a 3D medical scan contains hundreds of slices that capture fine-grained anatomical detail such as small lesions, subtle nodules, and vascular structures driving clinical diagnosis [16, 5, 2]. The scan then produces tens of thousands of vision tokens after the vision encoder, and the resulting key-value (KV) cache scales linearly with both batch size and sequence length. Clinical sites typically operate on single-card or small-cluster GPUs [25], where this cache quickly dominates GPU memory and becomes the bottleneck during inference. Compressing the visual KV cache without sacrificing diagnostic fidelity is therefore a gating problem for clinical deployment.
Preprint. KV cache compression directly addresses this bottleneck, and existing methods split into two categories by their target. (1) LLM-targeted methods, including KIVI [29], KVQuant [19], and TurboQuant [44], quantize the cache along its 1D token or channel axis. (2) VLM-targeted methods, including AKVQ-VL [35] and VL-Cache [38], exploit attention-derived saliency to allocate precision or evict cache entries. A separate line of work instead transforms the cache into the frequency domain before compression. FreqKV [21] applies a 1D discrete cosine transform (DCT) along the token sequence, FlashCache [43] extracts a low-frequency baseline and handles outliers via IDCT and uses the reconstruction residual to identify outlier KV pairs for retention, KVTC [34] performs feature-dimension PCA with adaptive bit allocation, and Fourier-VLM [41] applies a 2D DCT to vision encoder features before they enter the cache. None of these methods exploit the spatial continuity that 3D scans carry along all three axes.
A closer analog is video KV cache compression, which exploits redundancy between adjacent 2D frames [32, 40, 36]. Medical VLMs commonly treat a 3D scan as a sequence of 2D slices [33, 20], but the slice axis differs from the temporal axis. The temporal axis is motion-induced, with attention concentrated on a few salient frames, while the slice axis is anatomically continuous, with diagnostic signal spread across many slices rather than localized in a sparse subset. Directly applying video-domain compression to medical scans is therefore questionable.
3D medical scans carry a distinctive structural property. Nearby slices are highly similar, just like neighboring pixels within a single image. We observe that this structure also appears in the visual KV cache of the language model decoder: tokens at the same spatial location across adjacent slices have high similarity (Figure 1), and the cache shows strong autocorrelation along all three spatial axes. This spatial correlation concentrates the cache energy in a small set of low-frequency components, which a frequency-domain transform can compress.
Figure 1. Motivation. Tokens at the same in-plane spatial position across adjacent slices in the visual KV cache exhibit high pairwise similarity, and within each slice neighboring tokens are also strongly correlated. The cache energy then concentrates in a small set of low-frequency components along every axis, a three-axis structure that previous work leaves unexploited.
Motivated by this observation, we propose TACT-KV (Tri-Axis Cosine Transform for KV Cache). TACT-KV reshapes the visual KV cache into a 3D tensor with dimensions (slice, height, width), then applies a separable 3D DCT across these three spatial axes. After transforming the cache into the frequency domain, TACT-KV allocates bits to different frequency components using a reverse water-filling strategy and compresses them with standard scalar quantization. The method requires no calibration data and introduces no additional learnable parameters.
We further prove that the gain of water-filling over uniform bit allocation grows as the cache energy concentrates in fewer frequency components, and verify this empirically. We evaluate TACT-KV on six medical VLMs using two 3D CT benchmarks, RSNA-ICH [11] and CT-RATE [17]. Even at 1-bit quantization, TACT-KV performs close to the uncompressed cache, while existing LLM and VLM compression methods often fail. We also test TACT-KV on Video-MME [13, 20] using four general-purpose VLMs and observe similar improvements, suggesting that the method generalizes beyond medical imaging. At the same GPU memory budget, TACT-KV can increase the maximum decoding batch size by up to 8×. Our contributions are:
We propose TACT-KV, to our knowledge the first method to treat the visual KV cache as a volumetric tensor and exploit the three-axis spatial correlation from 3D scans.
We prove a closed-form expression for the distortion gain of water-filling over uniform bit allocation in terms of the spectral concentration of the cache, which we verify empirically.
We evaluate TACT-KV across six medical VLMs (4B-32B), two 3D CT benchmarks, and four general-purpose VLMs on Video-MME. TACT-KV achieves the lowest 1-bit flip rate on eleven of twelve medical VLM and dataset pairs in our main comparison, while delivering up to 8× larger decoding batch at equal HBM.
Method
We introduce TACT-KV, a calibration-free method for compressing the visual portion of the KV cache held by a decoder-only vision–language model during generation. For a backbone with L transformer layers and nh attention heads per layer, a single sequence at layer ℓ produces a key tensor K(ℓ) and a value tensor V(ℓ) with per-head feature dimension d. In a vision–language model, tokens originate from two sources, a vision encoder and a text tokenizer. When the input is a volumetric medical scan or a video, visual tokens dominate the count and inherit a known three-dimensional grid S×H×W from the vision encoder, where S is the slice or temporal axis token count and H, W are the spatial pooling dimensions. We exploit this structural prior by allocating the per-layer bit budget for the visual KV cache in a frequency-domain representation of this grid, retaining the information required by downstream tasks at low bit precision. The pipeline introduces no learned parameters and runs once per layer per sequence in a single forward pass. Figure 2 illustrates the four stages.
Figure 2. TACT-KV pipeline. A medical VLM encodes a volumetric CT scan and a clinical prompt into visual and text embeddings ev, et. Inside each decoder block, visual keys/values Kv, Vv enter the four-stage pipeline below, while text Kt, Vt bypass at bf16. Volumetric KV Layout: the visual cache is reshaped into a (S,H,W) grid inherited from the vision encoder. 3-D DCT-II: a separable transform along the three axes concentrates energy at the DC corner (magma colormap, log2σ2). Reverse Water-Filling: a per-layer bit budget is allocated above water level λ, with bins below dropped (bar color = bit width 4/2/1/0). Codebook Encoding: surviving bins are gain-shape encoded against the corresponding Lloyd–Max codebook (1/2/4-bit tiles shown). All spectral statistics are estimated on the fly, requiring no calibration data.
Volumetric KV Layout
Let Kv∈RB×nh×nv×d denote the visual portion of the layer-ℓ key cache and Kt∈RB×nh×nt×d denote the text portion, with B the batch dimension, nv=S⋅H⋅W the visual token count, and nt the text token count. The value tensors Vv, Vt are defined analogously. The grid parameters (S,H,W) are emitted by the vision encoder for each input and are read directly by our pipeline. We do not estimate any per-sample structural quantity. We reshape the visual key tensor into the volumetric grid:
X=reshape(Kv)∈RB×nh×S×H×W×d,(1)
and reshape the visual value tensor identically. The remaining pipeline (3-D DCT, water-filling allocation, codebook encoding) runs independently on Kv and Vv with separate per-bin variance estimates and bit budgets. The batch and head dimensions (B,nh) act as outer indices for the remainder of the pipeline and are omitted from subsequent equations. The text portion Kt,Vt is held at bf16 (bfloat16) [22] throughout the pipeline.
3-D DCT-II
The discrete cosine transform of type II is an orthonormal energy-compacting transform [1]. For a real sequence of length M, its M×M matrix D has entries indexed by output frequency k and input sample m,
D[k,m]=αkcos[Mπ(m+21)k],0≤k,m≤M−1,(2)
with normalization constants α0=1/M and αk≥1=2/M chosen so that DDT=I. For a first-order Markov source, the DCT-II asymptotically approaches the Karhunen–Loève transform, and variants of it underpin the block transform in JPEG and successor image and video codecs [39].
We extend the DCT-II to the reshaped tensor X∈RS×H×W×d. Let Ds∈RS×S, Dh∈RH×H, and Dw∈RW×W be the DCT-II matrices on the corresponding axis lengths, each constructed by (2). The frequency-domain tensor F∈RS×H×W×d is defined by the mode product [23]:
F=X×1Ds×2Dh×3Dw,(3)
where ×n denotes matrix multiplication along the n-th tensor axis. The separable three-axis structure reduces the per-layer cost from O(nv2⋅d) for a non-separable transform to O((S+H+W)⋅nv⋅d), and the parameter count from (SHW)2 to S2+H2+W2. The feature axis d is left untransformed and is handled by the scalar quantizer (SQ) in Section 2.4. Because Ds,Dh,Dw are each orthonormal, (3) is a lossless invertible change of variable that preserves the ℓ2 norm. Compression is introduced in the next subsection by allocating the per-layer bit budget across frequency bins.
Reverse Water-Filling Bit Allocation
Let the per-bin variance of the frequency tensor in (3), averaged over the batch, head, and feature dimensions, be
σi2=EB,nh,d[Fi2],i=1,…,N,N=SHW.(4)
Given a per-layer bit budget Rℓ, our goal is to allocate non-negative integer bits bi to each frequency bin so as to minimize the total reconstruction MSE. Under the high-rate scalar-quantization distortion model for a Gaussian source [15], the per-bin distortion is Di(bi)=c⋅σi2⋅2−2bi with c a constant determined by the quantizer geometry [7], and the problem reduces to
bi∈Z≥0mini∑σi2⋅2−2bi,s.t.i∑bi≤Rℓ.(5)
A Lagrangian relaxation with a non-negativity constraint yields the reverse water-filling solution
bi=max(0,⌊21log2(σi2/λ)⌋),(6)
where the water level λ is determined by a bisection search on the budget equation. The max(0,⋅) in (6) enforces non-negativity, the floor produces integer bits, and a greedy ±1 correction enforces ∑ibi=Rℓ exactly. In our implementation bi is capped at 8.
The following theorem expresses the distortion improvement of reverse water-filling over uniform allocation as a closed-form function of the spectral shape. Let Duniform and Dwater−fill denote the total distortion summed over all bins under the two allocations.
References
[1]Nasir Ahmed, T_ Natarajan, and Kamisetty R Rao. Discrete cosine transform. IEEE transactions on Computers, 100(1):90–93, 1974.DOI
[2]Fan Bai, Yuxin Du, Tiejun Huang, Max Q-H Meng, and Bo Zhao. M3d: Advancing 3d medical image analysis with multi-modal large language models. arXiv preprint arXiv:2404.00578, 2024.arxiv.org/abs/2404.00578
[4]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianhao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025.arxiv.org/abs/2502.13923
[5]Louis Blankemeier, Joseph Paul Cohen, Ashwin Kumar, Dave Van Veen, Syed Jamal Safdar Gardezi, Magdalini Paschali, Zhihong Chen, Jean-Benoit Delbrouck, Eduardo Reis, Cesar Truyts, et al. Merlin: A vision language foundation model for 3d computed tomography. Research Square, pages rs–3, 2024.DOI
[6]Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in neural information processing systems, 36:4396–4429, 2023.arxiv.org/abs/2307.13304
[7]Thomas M Cover, Joy A Thomas, et al. Entropy, relative entropy and mutual information. Elements of information theory, 2(1):12–13, 1991.
[8]Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344–16359, 2022.arxiv.org/abs/2205.14135
[9]Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems, 35:30318–30332, 2022.
[10]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088–10115, 2023.arxiv.org/abs/2305.14314
[11]Adam E Flanders, Luciano M Prevedello, George Shih, Safwan S Halabi, Jayashree Kalpathy-Cramer, Robyn Ball, John T Morgan, Anouk Stein, Felipe C Kitamura, Matthew P Lungren, et al. Construction of a machine learning dataset through collaboration: the rsna 2019 brain ct hemorrhage challenge. Radiology: Artificial Intelligence, 2(3):e190211, 2020.
[12]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022.arxiv.org/abs/2210.17323
[13]Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24108–24118, 2025.arxiv.org/abs/2405.21075
[14]Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonnell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Zenodo, 2021.DOI
[15]Allen Gersho and Robert M Gray. Vector quantization and signal compression, volume 159. Springer Science & Business Media, 2012.
[16]Ibrahim Ethem Hamamci, Sezgin Er, Suprousanna Shit, Hadrien Reynaud, Dong Yang, Pengfei Guo, Marc Edgar, Daguang Xu, Bernhard Kainz, and Bjoern Menze. Better tokens for better 3d: Advancing vision-language modeling in 3d medical imaging. arXiv preprint arXiv:2510.20639, 2025.arxiv.org/abs/2510.20639
[17]Ibrahim Ethem Hamamci, Sezgin Er, Chenyu Wang, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgün, Irem Dogan, Omer Faruk Durugol, Benjamin Hou, Suprousanna Shit, et al. Developing generalist foundation models from a multimodal dataset for 3d computed tomography. arXiv preprint arXiv:2403.17834, 2024.DOI
[18]Ibrahim Ethem Hamamci, Sezgin Er, Chenyu Wang, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgün, Irem Dogan, Omer Faruk Durugol, Benjamin Hou, Suprousanna Shit, et al. Generalist foundation models from a multimodal dataset for 3d computed tomography. Nature Biomedical Engineering, pages 1–19, 2026.DOI
[19]Coleman Hooper, Sehoon Kim, Hiva Mohammad zadeh, Michael W Mahoney, Yakun S Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length llm inference with kv cache quantization. Advances in Neural Information Processing Systems, 37:1270–1303, 2024.arxiv.org/abs/2401.18079
[20]Songtao Jiang, Yuan Wang, Sibo Song, Tianxiang Hu, Chenyu Zhou, Bin Pu, Yan Zhang, Zhibo Yang, Yang Feng, Joey Tianyi Zhou, et al. Hulu-med: A transparent generalist model towards holistic medical vision-language understanding. arXiv preprint arXiv:2510.08668, 2025.arxiv.org/abs/2510.08668
[21]Jushi Kai, Boyi Zeng, Yixuan Wang, Haoli Bai, Ziwei He, Bo Jiang, and Zhouhan Lin. Freqkv: Frequency domain key-value compression for efficient context window extension. arXiv preprint arXiv:2505.00570, 2025.arxiv.org/abs/2505.00570
[22]Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jiany u Huang, Hector Yuen, et al. A study of bfloat16 for deep learning training. arXiv preprint arXiv:1905.12322, 2019.
[23]Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 51(3):455–500, 2009.DOI
[24]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626, 2023.
[25]Qizhen Lan, Aaron Choi, Jun Ma, Bo Wang, Zhaogming Zhao, Xiaoqian Jiang, and Yu-Chun Hsu. From performance to practice: Knowledge-distilled segmentator for on-premises clinical workflows. arXiv preprint arXiv:2601.09919, 2026.
[27]Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Li, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36:28541–28564, 2023.
[28]Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of machine learning and systems, 6:87–100, 2024.
[29]Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024.
[30]Stuart Lloyd. Least squares quantization in pcm. IEEE transactions on information theory, 28(2):129–137, 1982.DOI
[31]Yizhen Luo, Jiah uan Zhang, Siqi Fan, Kai Yang, Yushuai Wu, Mu Qiao, and Zaiqing Nie. Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine. arXiv preprint arXiv:2308.09442, 2023.
[32]Vishnu Sai, Dheeraj Sai, Girish Varma, Priyesh Shukla, et al. Dual-signal adaptive kv-cache optimization for long-form video understanding in vision-language models. arXiv preprint arXiv:2602.14236, 2026.
[33]Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. Medgemma technical report. arXiv preprint arXiv:2507.05201, 2025.DOI
[34]Konrad Staniszewski and Adrian Łańcucki. Kv cache transform coding for compact storage in llm inference. arXiv preprint arXiv:2511.01815, 2025.
[35]Zunhai Su, Wang Shen, Linge Li, Zhe Chen, Hanyu Wei, Huangqi Yu, and Kehong Yuan. Akvq-vl: Attention-aware kv cache adaptive 2-bit quantization for vision-language models. In 2025 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2025.
[36]Keda Tao, Haoxuan You, Yang Sui, Can Qin, and Huan Wang. Plug-and-play 1. x-bit kv cache quantization for video large language models. arXiv preprint arXiv:2503.16257, 2025.
[37]Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025.
[38]Dezhan Tu, Danylo Vashchilenko, Yuzhe Lu, and Panpan Xu. VI-cache: Sparsity and modality-aware kv cache compression for vision-language model inference acceleration. arXiv preprint arXiv:2410.23317, 2024.arxiv.org/abs/2410.23317
[39]Gregory K Wallace. The jpeg still picture compression standard. Communications of the ACM, 34(4):30–44, 1991.
[40]Zhongwei Wan, Xiang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4065–4078, 2024.arxiv.org/abs/2406.18139
[41]Huanyu Wang, Jushi Kai, Haoli Bai, Lu Hou, Bo Jiang, Ziwei He, and Zhouhan Lin. Fourier-vlm: Compressing vision tokens in the frequency domain for large vision-language models. arXiv preprint arXiv:2508.06038, 2025.
[42]Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning, pages 38087–38099. PMLR, 2023.arxiv.org/abs/2211.10438
[43]Yaoxin Yang, Peng Ye, Xudong Tan, Chongjun Tu, Maosen Zhao, Jia Hao, and Tao Chen. Revisiting multimodal kv cache compression: A frequency-domain-guided outlier-kv-aware approach. arXiv preprint arXiv:2511.16786, 2025.arxiv.org/abs/2511.16786