betaFeedback

WorksComputer Science

MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference

Abstract

Key–value (KV) cache compression is an effective way to reduce the memory overhead of large language model (LLM) inference, particularly for long-context workloads. However, existing compression methods make different trade-offs among accuracy, inference latency, and peak KV cache memory utilization, making a single fixed configuration unsuitable across different prompts and resource constraints. We introduce MetaKV, an adaptive framework that selects a KV cache compression configuration for each input prompt based on user-specified latency and peak memory budgets. MetaKV uses lightweight prediction models to estimate the end-to-end latency, peak memory, and probability of a correct response for each candidate configuration, and selects the configuration that best satisfies the latency-memory constraints while preserving accuracy. We evaluate MetaKV across ten configurations from three representative KV cache compression methods, KVQuant, H2_2O, and RocketKV, together with an uncompressed FP16 configuration, on four datasets covering mathematics, science, commonsense reasoning, and reading comprehension. Across a wide range of latency and peak memory constraints, MetaKV consistently outperforms the best static configuration, improving constrained success rate (CSR), the fraction of prompts answered correctly while satisfying both constraints, by approximately 0.07 on average and up to 0.135. These results demonstrate the benefit of adapting KV cache compression to individual prompts and latency-memory constraints. Code is available at https://github.com/MichaelWang0505/MetaKV.git

Introduction

Large Language Models (LLMs) have become widely used for applications such as conversational assistants and code generation [?, 5]. Their growing adoption has increased the demand for GPU memory and efficient LLM inference [?, ?, 1]. This challenge is especially important for long-context inference, where the KV cache, which stores the key and value vectors of previously processed tokens, can consume up to 80% of GPU memory [?]. To reduce this memory overhead and serve more requests on limited GPU resources, prior work has proposed KV cache compression methods. These methods can be broadly categorized as lossless or lossy. Lossless methods reduce memory usage while preserving the original KV cache values and, therefore, model accuracy. Lossy methods reduce precision or discard parts of the KV cache, achieving larger memory savings at the cost of some accuracy loss. Since long-context inference often requires substantial memory reduction and lossless compression provides limited savings, we focus on lossy KV cache compression in this work.

State-of-the-art lossy KV cache compression methods use different approaches. These include (i) quantization, reducing the bits used to store KV values [?, ?, ?]; (ii) token eviction, permanently removing less important tokens from the KV cache [?]; and (iii) sparse attention, attending to only a subset of cached tokens during computation [2, ?]. Each approach makes different trade-offs among peak KV cache memory utilization (peak memory), end-to-end latency, and accuracy.

Quantization can substantially reduce peak memory by representing values with only a few bits instead of full 16-bit floating-point precision (FP16). But the additional quantization and dequantization operations can increase inference latency. In contrast, token eviction and sparse attention reduce computation by removing or skipping tokens, often resulting in lower latency. However, the retained KV values typically remain at higher precision, leading to larger peak memory usage than aggressive quantization. So, no single compression approach is consistently superior across all efficiency and accuracy objectives.

Different LLM deployment environments also impose different requirements on accuracy, end-to-end latency, and peak memory. For example, LLMs are increasingly deployed on edge devices such as smartphones, where limited memory makes peak memory usage a major constraint [?]. In contrast, interactive applications such as customer-service chatbots require fast responses, making inference latency a key constraint [6]. These scenarios can favor different KV cache compression methods. Quantization may be more suitable for memory-constrained environments, while token eviction may better suit latency-constrained settings. Sparse attention can also be effective because it has the least latency on shorter-context prompts [2]. Compression methods can also perform differently across datasets because the datasets emphasize different capabilities. Since different compression methods preserve or remove information from the KV cache in different ways, their relative performance can vary across these task types. For example, token eviction can achieve higher accuracy than quantization on math prompts, while quantization can perform better on scientific-reasoning prompts. This difference can occur because token eviction removes entire token representations, while quantization retains all tokens at reduced precision, causing the two approaches to preserve task-relevant information differently. Per-dataset results can be viewed in Appendix A.

Since the relative performance of compression methods can vary across prompt types and deployment constraints, we propose METAKV, an adaptive framework that selects the most suitable KV cache compression configuration for each input prompt. Given end-to-end latency and peak memory constraints from the user, METAKV predicts the accuracy, end-to-end latency, and peak KV cache memory of each candidate configuration and selects the one that best satisfies the constraints while preserving accuracy. This allows METAKV to adapt its compression strategy to both the input prompt and the deployment environment.

In summary, this paper makes three main contributions. First, we show that existing KV cache compression methods exhibit different trade-offs across accuracy, end-to-end latency, and peak memory utilization, motivating the need for adaptive selection. Second, we introduce METAKV, a framework that predicts the accuracy, latency, and peak KV cache memory of candidate compression configurations and selects the most suitable one under given constraints. Third, we evaluate METAKV across multiple KV cache compression methods and datasets, showing that it consistently outperforms the best static configuration across all tested latency and peak memory constraint pairs, improving constrained success rate, the fraction of prompts answered correctly while satisfying both constraints, by approximately 0.07 on average and up to 0.135.

Designing MetaKV

In this section, we first review the KV cache compression methods considered in our study. We then compare their performance and show that they make different trade-offs among accuracy, latency, and peak KV cache memory, with no single method performing best across all settings. Motivated by this observation, we present METAKV, which selects the most suitable compression configuration for each prompt under given resource constraints.

Baseline KV cache compression methods

We consider KVQuant [?], H₂O [?], and RocketKV [2] because they represent three major categories of lossy KV cache compression: quantization, token eviction, and sparse attention. These methods provide METAKV with a diverse set of trade-offs among accuracy, latency, and peak memory usage.

KVQuant [9]: KVQuant reduces KV cache memory by storing Key and Value vectors with fewer bits instead of full 16-bit precision (FP16). Lower precision can reduce model accuracy, so KVQuant includes techniques to limit this loss. For example, it quantizes Keys per channel based on the observation that outliers are concentrated in consistent channels, and stores outlier values in uncompressed FP16 to avoid large quantization errors. We denote a KVQuant setting as bb-bit and evaluate b∈{2,3,4}b \in\{2, 3, 4\}, where bb is the number of bits used to store KV cache values.

H2_2O [18]: Instead of reducing precision, H2_2O removes the Key and Value vectors of less important tokens. It tracks an accumulated attention score for each token, reflecting how much attention the token receives from subsequent tokens. When the KV cache exceeds a given budget, H2_2O evicts older tokens with the lowest accumulated attention scores. We denote an H2_2O setting as p%p\% and evaluate p∈{20,40,60}p \in\{20, 40, 60\}, where pp is the percentage of tokens retained in the KV cache relative to the tokens in the input prompt.

RocketKV [2]: RocketKV combines token pruning with sparse attention. In its first stage, SnapKV uses aggregated attention scores between the input context and an observation window at the end of the prompt to identify and retain the most relevant tokens. In its second stage, Hybrid Sparse Attention attends to only a subset of earlier tokens during each decoding step. We denote a RocketKV setting as r×r\times and evaluate r∈{8,16,32}r \in\{8, 16, 32\}, where rr is the post-prefill compression ratio, meaning that the effective KV cache is reduced to approximately 1/r1/r of its original size after prefill.

Motivating MetaKV

To compare the three compression methods with the uncompressed baseline, we measure three key metrics for each configuration: accuracy, end-to-end latency, and peak KV cache memory. These results reveal clear trade-offs across methods and motivate the need for MetaKV.

Table 1 summarizes these trade-offs over 8,192 prompts across four datasets, with experimental details provided in subsection 3.1. Across the tested configurations, KVQuant has approximately 8.0×\times the latency of H2_2O and 5.7×\times the latency of RocketKV, while using about 29% less peak KV cache memory than H2_2O and 71% less than RocketKV on average.

MethodSettingAccuracyLatency (s)Peak Memory (MB)
Uncompressed–0.6452.0134.47
KVQuant4-bit0.63021.6711.71
KVQuant3-bit0.60920.748.91
KVQuant2-bit0.53619.686.39
H2_2O60%0.6272.4819.09
H2_2O40%0.6022.5812.70
H2_2O20%0.5072.706.29
RocketKV8×\times0.4643.5630.77
RocketKV16×\times0.3523.5330.65
RocketKV32×\times0.2833.8330.64

Table 1. Accuracy, average end-to-end latency, and average peak memory for each configuration over the 1,232 held-out test prompts. The uncompressed configuration uses the full KV cache in FP16. Setting denotes the quantization bit-width for KVQuant, fraction of tokens retained for H2_2O, and post-prefill compression ratio for RocketKV.

These results show a clear trade-off between end-to-end latency, peak memory usage, and accuracy. KVQuant is more suitable for memory-constrained settings because it achieves the lowest peak KV cache memory, but it incurs substantially higher latency. H2_2O provides much lower latency while still reducing memory usage, making it more suitable when latency is the main constraint. When neither latency nor peak memory is constrained and accuracy is the primary objective, the uncompressed FP16 configuration performs best. On average, RocketKV did not outperform the other methods. However, these averages hide an important point: on individual datasets, the fastest method changes. For example, on multiple-choice scientific reasoning, RocketKV is the fastest, although FP16 is fastest on average. Accuracy can shift by task as well: under aggressive compression, H2_2O keeps more accuracy than KVQuant on math problems, while KVQuant is more accurate on multiple-choice scientific reasoning. As mentioned in the introduction, compression methods can perform differently across datasets, since they preserve or remove information from the KV cache in different ways, meaning that their performance can vary based on the task or dataset. Performance can also vary across individual prompts because of differences in complexity and context length, so METAKV adapts to different constraints and prompts by choosing different method configurations.

MetaKV

METAKV selects the best KV cache compression configuration for a given prompt. Its goal is to satisfy user-defined limits for maximum end-to-end latency (Lmax⁡L_{\max}) and peak memory utilization (Mmax⁡M_{\max}) while maximizing the likelihood of a correct response. As shown in Figure 1, the system follows a straightforward pipeline. First (Steps 1 and 2), when a user submits a prompt alongside their constraints, METAKV uses Light Gradient-Boosting Machine (LightGBM) [?] to predict the execution latency and peak memory utilization for all candidate configurations listed in Table 1. Next, a Budget Filter evaluates these predictions and discards any configurations that exceed the user’s limits (Step 3).

The METAKV pipeline

Figure 1. The METAKV pipeline. Given a user’s prompt and latency and peak memory budgets, the latency and peak memory LightGBMs predict each configuration’s resource usage, and only those within budget are kept. The accuracy LightGBM then predicts which surviving candidate is most likely to answer correctly. The selected configuration is executed with Hugging Face Transformers.

METAKV then selects the final configuration using a logical hierarchy (Steps 4–7). If the uncompressed FP16 baseline survives the filter, it is selected immediately because it avoids information loss from KV cache compression and achieves the highest overall accuracy. If FP16 is too expensive, METAKV narrows the remaining candidates by keeping only the least compressed setting for each method to preserve as much information as possible. If only one candidate is left, it is selected. If multiple remain, an Accuracy LightGBM predicts their correctness probabilities, and the highest scorer is chosen. Finally, the selected configuration and the user’s prompt are sent to the LLM inference engine (Hugging Face Transformers)[?] to generate the response (Step 8).

If the user’s limits are so strict that no configuration passes the Budget Filter, METAKV calculates a relative violation score (vcv_c) to identify which configuration exceeds the limits the least: vc=L^c/Lmax⁡+M^c/Mmax⁡v_c = \hat{L}_c/L_{\max} + \hat{M}_c/M_{\max}, where L^c\hat{L}_c and M^c\hat{M}_c are the predicted end-to-end latency and peak memory utilization for configuration cc. METAKV then selects the configuration with the smallest combined violation, using the accuracy model to break any ties.

Evaluation

In this section, we evaluate METAKV’s performance and examine how its configuration choices change under different latency and peak memory constraints. We plan to open-source our tool.

Experimental setup

Model, Hardware, and Datasets. We evaluated METAKV using Meta Llama 3.1 8B [7] on a single NVIDIA RTX PRO 6000 Blackwell Server Edition GPU (96 GB VRAM) to ensure fair comparisons. To cover a diverse range of workloads, we used four datasets: GSM8K [4] (math), HellaSwag [?] (commonsense), ARC-Challenge [3] (scientific reasoning), and SQuAD 1.1 [?] (reading comprehension). For generative tasks, accuracy was measured using the exact match; for multiple-choice tasks, we selected the answer with the lowest negative log-likelihood and measured whether the selected answer was correct.

Data Collection and Model Training. For each dataset, we collected 2,048 prompts and profiled ten candidate configurations: uncompressed FP16; KVQuant at 2-, 3-, and 4-bit precision; H2O with 20%, 40%, and 60% token budgets; and RocketKV at 8×, 16×, and 32× compression ratios. The three settings we chose for each method span mild, moderate, and aggressive compression. This provides clear resource-to-accuracy trade-offs while ensuring the number of configurations is balanced across methods. We randomly split this data into 85% training and 15% testing, yielding 308 test prompts per dataset for a total of 1,232 evaluation prompts. METAKV’s LightGBM predictors for latency, peak memory, and accuracy were tuned on the training split using 5-fold cross-validation. The peak memory model achieved high accuracy (MAPE 7.7% and R2R^2 0.9981), the latency model performed well overall despite natural variance in RocketKV’s dynamic token selection (MAPE 38.19% and R2R^2 0.9651), and the accuracy model demonstrated moderate predictive power (pooled AUROC of 0.70). Although running the additional LightGBM predictors before LLM inference introduces extra latency, their average overhead is only 10.96 miliseconds per prompt, equivalent to 0.34% of MetaKV’s average LLM inference latency, and is therefore negligible.

Evaluation Metrics. We tracked accuracy, end-to-end latency, and peak KV cache memory. To capture METAKV’s ability to balance performance and efficiency, we introduce the Constrained Success Rate (CSR). CSR measures the proportion of test prompts answered correctly without exceeding the user’s constraints: CSR=1N∑i=1NI(correcti∧ℓi≤Lmax⁡∧mi≤Mmax⁡)\mathrm{CSR} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}\left(\mathrm{correct}_{i} \land\ell_{i} \leq L_{\max} \land m_{i} \leq M_{\max}\right) where for NN test prompts, the indicator function checks if the response is correct (correcti\mathrm{correct}_{i}) and whether the actual latency (ℓi\ell_{i}) and peak memory (mim_{i}) do not exceed their respective budgets (Lmax⁡L_{\max} and Mmax⁡M_{\max}).

Baselines. We compared METAKV against two baselines: (i) STATICBASELINE represents the single best fixed configuration a user could commit to in advance; for any given (Lmax⁡L_{\max}, Mmax⁡M_{\max}) constraint pair, we selected the configuration with the highest CSR on the training set and evaluated it on the test set; (ii) ORACLE represents METAKV’s theoretical upper bound under perfect predictions; it scores a success if any configuration that physically satisfies the true latency and peak memory constraints produces the correct answer.

Evaluating MetaKV

METAKV consistently outperforms STATICBASELINE. As shown in Table 1, static compression methods struggle to balance accuracy, end-to-end latency, and peak memory utilization. To quantify the value of adaptive selection, we measured METAKV’s Constrained Success Rate (CSR) improvement over the STATICBASELINE (ΔCSR=CSRMETAKV−CSRSTATICBASELINE\Delta\mathrm{CSR} = \mathrm{CSR}_{\mathrm{METAKV}} - \mathrm{CSR}_{\mathrm{STATICBASELINE}}) across a grid of latency (Lmax⁡L_{\max}) and peak memory (Mmax⁡M_{\max}) limits.

Figure 2 shows that METAKV achieves positive ΔCSR\Delta\mathrm{CSR} across all tested constraint pairs, with an average improvement of approximately 0.07. The largest and most consistent gains occur at the 30 MB peak memory threshold, where METAKV improves CSR by 0.10–0.13 across all latency constraints. Under the tightest peak memory constraint of 5 MB, METAKV improves CSR by approximately 0.06 on average, while under the tightest latency constraint of 2 s, it improves CSR by approximately 0.073.

Heatmap of METAKV’s constrained success rate improvement over STATICBASELINE across latency and peak memory constraints

Figure 2. METAKV’s constrained success rate (CSR) improvement over STATICBASELINE across latency and peak memory constraints, shown as proportions: ΔCSR=CSRMETAKV−CSRSTATICBASELINE\Delta\mathrm{CSR} = \mathrm{CSR}_{\mathrm{METAKV}} - \mathrm{CSR}_{\mathrm{STATICBASELINE}}. Positive values indicate METAKV answers more prompts correctly within budget.

Table 2 shows that the best static configuration under tighter peak memory budgets of 5–30 MB is typically H2O 20% or 40%. Across these tighter peak memory budgets, ΔCSR\Delta\mathrm{CSR} increases as the peak memory budget grows from 5 MB to 30 MB. Because the static policy must use the same configuration for every prompt, it cannot take advantage of prompts that could satisfy the constraints with a less aggressive and more accurate configuration. METAKV instead adapts its choice per prompt, allowing it to use lighter compression when the prompt requires fewer resources while using stronger compression when necessary. Once the peak memory budget reaches 30 MB, the STATICBASELINE can finally afford H2O 60%. This increases the STATICBASELINE’s accuracy, narrowing the performance gap (ΔCSR\Delta\mathrm{CSR} drops to ~0.01).

Latency5 MB10 MB30 MB80 MB
2 sUncompressedH₂O 20%H₂O 20%H₂O 60%
5 sH₂O 20%H₂O 40%H₂O 20%H₂O 60%
10 sH₂O 20%H₂O 40%H₂O 20%H₂O 60%
50 sH₂O 20%H₂O 40%H₂O 20%H₂O 60%

Table 2. The static configuration with the highest training-set CSR for each latency and peak memory constraint pair. This is the configuration STATICBASELINE commits to for that constraint.

Along the latency axis, METAKV’s gains depend on the peak memory budget. At 5–30 MB, tightening the latency threshold does not produce a consistent trend because both METAKV and the STATICBASELINE are restricted to similar low peak memory configurations. At 80 MB, however, METAKV’s gain increases from 0.01 at 50 s to 0.052 at 2 s, since the larger peak memory budget gives METAKV more flexibility to select faster configurations for latency-sensitive prompts while preserving accuracy on others.

Key Takeaways: METAKV improves CSR across every tested latency and peak memory constraint, with the largest gains when resource constraints create meaningful trade-offs among configurations. Its per-prompt selection allows it to use stronger compression when needed and preserve more accuracy when resources allow.

METAKV adapts to different resource bottlenecks. To understand why METAKV achieves higher CSR than the best static configuration, we compare accuracy and constraint violations under four representative constraint settings shown in fig:3a: tight latency with loose peak memory (Lmax⁡=2L_{\max} = 2 s, Mmax⁡=80M_{\max} = 80 MB), loose latency with tight peak memory (Lmax⁡=50L_{\max} = 50 s, Mmax⁡=5M_{\max} = 5 MB), moderate latency and peak memory (Lmax⁡=10L_{\max} = 10 s, Mmax⁡=30M_{\max} = 30 MB), and loose latency and peak memory (Lmax⁡=50L_{\max} = 50 s, Mmax⁡=80M_{\max} = 80 MB).

Under tight latency and loose peak memory constraints, METAKV improves CSR from 0.314 to 0.366. As shown in fig:3b, METAKV reduces the constraint violation rate from 45.9% to 35.8%, while its accuracy is only 2.2 percentage points lower than the static configuration (fig:3c). The STATICBASELINE selects H₂O 60% for all prompts, which violates the tight latency constraint more often. In contrast, METAKV can select lower-latency configurations for prompts where H₂O 60% would exceed the budget.

Under tight peak memory and loose latency constraints, METAKV improves accuracy by 5.3 percentage points over the STATICBASELINE, while increasing the constraint violation rate by only 3.1 percentage points. The STATICBASELINE achieves fewer violations because it uses H₂O 20% for every prompt, an aggressive compression setting that reduces memory usage but also lowers accuracy. But METAKV can select less aggressive and more accurate configurations for shorter prompts that still satisfy the peak memory budget.

Under moderate latency and peak memory constraints, METAKV improves accuracy by 10.6 percentage points while increasing the constraint violation rate by only 0.9 percentage points compared to the STATICBASELINE. The STATICBASELINE uses H₂O 20% for all prompts, which keeps latency and peak memory low but sacrifices accuracy. But METAKV can select more accurate configurations for prompts that still satisfy the constraints, leading to a large accuracy gain with only a small increase in violations.

Under loose latency and peak memory constraints, all methods achieve a 0% constraint violation rate because a feasible configuration exists for every prompt. In this setting, METAKV improves accuracy by 1.2 percentage points over the STATICBASELINE. The STATICBASELINE uses H2_2O 60% for all prompts, while METAKV can select more accurate configurations for shorter prompts that have enough latency and peak memory headroom.

Figure 3. Comparison of STATICBASELINE, METAKV, and ORACLE across four representative latency–memory constraint settings. (a) Constrained success rate (CSR), shown as a proportion. (b) Constraint violation rate, the percentage of prompts exceeding the latency or peak memory budget. (c) Accuracy, shown as a percentage.

Although violation rates reach up to 38% in some settings, most are not caused by METAKV’s selection pipeline. As shown in fig:3b, the ORACLE also violates up to 35% of prompts, which occurs only when none of the available configurations can satisfy the given constraints. METAKV’s violation rate is only 0–6 percentage points higher than the ORACLE, indicating that most violations are due to limitations of the current compression method pool rather than prediction errors. We discuss how to mitigate this issue further in section 4. Similarly, METAKV’s accuracy is only 0.2–1.8 percentage points lower than the ORACLE, showing that its selection policy performs close to the best achievable accuracy among the available configurations.

Key Takeaways: METAKV improves CSR by adapting to the active resource bottleneck, reducing violations under tight constraints, and improving accuracy when more configurations are feasible.

METAKV relies on multiple compression configurations. To understand how METAKV adapts its selections across deployment settings, Figure 4 shows the fraction of prompts assigned to each configuration under the four representative constraint pairs.

Stacked bar chart showing the proportion of prompts assigned to each compression configuration under four latency–memory constraint pairings

Figure 4. Proportion of prompts assigned to each configuration by METAKV under four representative latency–memory constraint pairings.

Under loose latency and peak memory constraints (Lmax⁡=50L_{\max} = 50 s, Mmax⁡=80M_{\max} = 80 MB), METAKV selects the uncompressed FP16 configuration for 74.9% of prompts, prioritizing accuracy when compression is unnecessary. The remaining prompts use H2_2O 60% (16.4%) or KVQuant 4-bit (8.7%), mainly when longer contexts require mild compression to remain within the peak memory budget.

When latency is tight (Lmax⁡=2L_{\max} = 2 s, Mmax⁡=80M_{\max} = 80 MB), FP16 usage drops to 63.6% and METAKV shifts toward low-overhead methods. Some prompts exceed the 80 MB budget under FP16, so METAKV routes them to H2_2O 60%, taking up 22.8% of selections. For other prompts, where FP16 would instead exceed the tight latency budget, METAKV leverages RocketKV’s lowest latency on ARC-Challenge (shown in Appendix A), selecting it 9.8% of the time.

Under tight peak memory constraints (Lmax⁡=50L_{\max}=50 s, Mmax⁡=5M_{\max}=5 MB), FP16 selection drops to 18.5% because it can only meet the peak memory threshold for the shortest prompts while using FP16. Instead, METAKV distributes selections across aggressive compression settings: H2_2O 20% (29.2%), KVQuant 2-bit (28.6%), and KVQuant 4-bit (15.3%). This balanced split illustrates how METAKV chooses different methods based on a prompt’s needs: it routes to KVQuant when prompt context requires preserving key tokens and to H2_2O when it needs lower latency.

Finally, under moderate constraints (Lmax⁡=10L_{\max}=10 s, Mmax⁡=30M_{\max}=30 MB), METAKV returns to FP16 for 68.4% of prompts, while relying on H2_2O 20% (25.0%) and KVQuant 4-bit (5.4%) to handle longer inputs without exceeding mid-tier budgets.

Key Takeaways: METAKV does not rely on a single dominant strategy; instead, it dynamically reconfigures its selection to FP16 under relaxed peak memory limits, fast sparse-attention pruning under strict latency, and dense quantization or token eviction under tight peak memory ceilings.

Discussion and Limitations

Supporting new compression methods. METAKV is not tied to a fixed set of KV cache compression methods. New methods can be added to the candidate pool by profiling them offline and retraining the prediction models, without changing METAKV’s core selection logic.

Adapting to new models and hardware. METAKV’s latency and peak memory predictors are specific to the deployment environment. Using another LLM or hardware platform requires reprofiling the candidate configurations and retraining the predictors. Reducing this recalibration cost through transfer learning or lightweight online calibration is an important direction for future work.

Handling constraint violations. METAKV relies on predicted latency and peak memory, so some unseen prompts may still violate the given constraints. Our results show that most violations occur when none of the available configurations can satisfy the constraints, while a smaller fraction is caused by prediction error. In practice, violations can be reduced by expanding the candidate pool, using conservative prediction margins, relaxing the constraints, or provisioning additional resources.

Supporting additional constraints. This work focuses on end-to-end latency and peak KV cache memory, but METAKV can be extended to other deployment metrics such as time-to-first-token, throughput, inference cost, or energy consumption. These metrics can be collected during offline profiling, predicted at runtime, and incorporated into the selection policy as additional constraints or objectives.

Scope of the evaluation. Our evaluation covers one LLM, one hardware platform, four datasets, and three KV cache compression methods. Therefore, the results should not be interpreted as showing that any individual compression method is universally better than the others. Instead, they demonstrate that different configurations provide different accuracy, latency, and peak memory trade-offs, and that adapting the configuration at the prompt level can improve constrained performance. Evaluating METAKV across additional models, hardware platforms, workloads, and compression methods remains future work. Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Temanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ramachandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqian Zhang, Shuqian Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenar u, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024.

Per-Dataset Results

DatasetConfigurationAccuracyLatency (s)Peak Memory (MB)
GSM8KUncompressed0.64611.197795.9923
GSM8KKVQuant 2-bit0.389614.034017.3521
GSM8KKVQuant 3-bit0.590914.994423.9535
GSM8KKVQuant 4-bit0.613615.587830.8855
GSM8KH2_2O 20%0.53251.677617.2394
GSM8KH2_2O 40%0.62341.711234.5990
GSM8KH2_2O 60%0.63641.691251.9269
GSM8KRocketKV 8×0.62015.233187.2135
GSM8KRocketKV 16×0.50654.806786.7212
GSM8KRocketKV 32×0.34095.891586.7212
ARC-ChallengeUncompressed0.56172.12654.5763
ARC-ChallengeKVQuant 2-bit0.526019.59331.1000
ARC-ChallengeKVQuant 3-bit0.526020.88381.6426
ARC-ChallengeKVQuant 4-bit0.555221.82622.4448
ARC-ChallengeH2_2O 20%0.33772.83060.8287
ARC-ChallengeH2_2O 40%0.51302.67551.7792
ARC-ChallengeH2_2O 60%0.55522.54282.7013
ARC-ChallengeRocketKV 8×0.39941.36183.8121
ARC-ChallengeRocketKV 16×0.30841.39243.8121
ARC-ChallengeRocketKV 32×0.32471.41423.8121
HellaSwagUncompressed0.79874.614910.5698
HellaSwagKVQuant 2-bit0.772742.98052.1627
HellaSwagKVQuant 3-bit0.795544.82443.0977
HellaSwagKVQuant 4-bit0.798746.97754.2995
HellaSwagH2_2O 20%0.72086.14722.0341
HellaSwagH2_2O 40%0.78575.79644.1778
HellaSwagH2_2O 60%0.79555.51886.3109
HellaSwagRocketKV 8×0.42867.25816.1530
HellaSwagRocketKV 16×0.27927.48636.1530
HellaSwagRocketKV 32×0.29877.47166.1530
SQuAD 1.1Uncompressed0.57470.113926.7459
SQuAD 1.1KVQuant 2-bit0.45452.10504.9320
SQuAD 1.1KVQuant 3-bit0.52272.27656.9417
SQuAD 1.1KVQuant 4-bit0.55192.27149.2091
SQuAD 1.1H2_2O 20%0.43830.13625.0657
SQuAD 1.1H2_2O 40%0.48700.145110.2557
SQuAD 1.1H2_2O 60%0.51950.148415.4213
SQuAD 1.1RocketKV 8×0.40580.388625.8920
SQuAD 1.1RocketKV 16×0.31490.438425.9054
SQuAD 1.1RocketKV 32×0.16880.542625.8892

Table 3. Accuracy, average end-to-end latency, and average peak KV-cache memory for each configuration on the held-out test split of each dataset (308 prompts per dataset).

References

  1. [1]Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, and Ramachandran Ramjee. Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills, 2023.arxiv.org/abs/2308.16369
  2. [2]Payman Behnam, Yaosheng Fu, Ritchie Zhao, Po-An Tsai, Zhiding Yu, and Alexey Tumanov. Rocketkv: Accelerating long-context llm inference via two-stage kv cache compression, 2025.arxiv.org/abs/2502.14051
  3. [3]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Ovyind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.arxiv.org/abs/1803.05457
  4. [4]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.arxiv.org/abs/2110.14168
  5. [5]Yihong Dong, Xue Jiang, Jiaru Qian, Tian Wang, Kechi Zhang, Zhi Jin, and Ge Li. A survey on code generation with llm-based agents, 2025.arxiv.org/abs/2508.00083
  6. [6]Ulrich Gnewuch, Stefan Morana, Marc T. P. Adam, and Alexander Maedche. Opposing effects of response time in human–chatbot interaction: The moderating role of prior experience. Business & Information Systems Engineering, 64:773–791, 2022.
  7. [7]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Diewuke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Ilyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnston, Joshua Xie, Juntao Jia, Kalyan Va-suden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kam-badur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hos-seini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Son, Soniya Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damljan, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqian Zhang, Shuqian Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijay Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024.

Paper details

vlnc:W_3wcuejy3
Version
v1 · latest
Published
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents