betaFeedback

WorksComputer Science

RIFT: KV Cache Compression for Draft Models with Complementary Compressibility Training

Abstract

Speculative decoding is a popular decode acceleration method. As such, it is particularly relevant to long-output generation, which requires an increasingly large KV cache. KV cache compression offers a way to constrain costs in long-context drafting. However, compression methods that focus on prefill information can lose relevance as the output length grows. We term this phenomenon decode staleness and show that it degrades draft model accuracy under selection-based KV cache compression. We introduce RIFT, a family of selection-based KV cache compression strategies that balance prefill information and recent generated tokens. We show that refreshing the KV cache selection during token generation significantly improves the drafter's quality at the cost of additional decode latency. To alleviate the refresh costs, we propose an adaptive KV refresh strategy that reduces the number of score calculations while retaining the drafter's quality. We further demonstrate how a compressibility-driven drafter training phase, based on sparse attention masks, can improve the drafter's quality under a compressed KV cache. Combined with compressibility training, RIFT enables Qwen3.5-4B, drafting for Qwen3.5-122B, to retain over 98% of its full-cache acceptance length on 32K-token inputs, 4K-token outputs, and an 8K-token KV cache budget.

Introduction

Speculative decoding accelerates autoregressive language models by verifying multiple draft proposals in a single target-model pass while preserving the target distribution [23, 8], shifting inference towards a compute-bound regime. Its effectiveness depends on a draft that is both inexpensive and closely aligned with the target. Reasoning and agentic applications increase the pressure on this balance,

combining long input sequence lengths (ISLs) with long output sequence lengths (OSLs) as models generate reasoning traces and accumulate interaction history [33, 44]. As context grows, key-value (KV) cache access can increasingly bottleneck drafting, limiting the achievable speedup from speculation [36]. As a decode acceleration method, speculative decoding targets long OSL, further magnifying the KV cache bottleneck. This bottleneck is still evident in hybrid attention architectures [10, 43]: even when only 25% of a small hybrid model’s layers use full attention, its KV footprint can significantly exceed its weights’ memory footprint at long contexts with only a few concurrent requests (See Appendix § 9 for details). Existing literature considers various forms of KV compression [18, 29, 22]. Here, we focus on selection-based compression, where decode tokens attend only a compact subset of their past KVs.

Tail-augmented drafters, e.g., MTP, Eagle or DFlash [25, 9], are considered superior to external draft models on GPU-based deployments due to their minimal latency overhead. However, contemporary hardware-disaggregated inference and edge-assisted serving renews the relevance of small external drafts [2, 34] over tail-augmented blocks, such as MTPs or DFlash [9]. In these settings, memory capacity poses a constraint, as memory is shared between the model weights and growing KV cache [20]. Considering all the above, we study draft KV cache compression under a fixed cache budget that is ISL-independent. KV compression also enables KV-based self-speculation: the target model itself drafts with a compressed cache and verifies with full cache [36, 47] (Sun et al. 2024). Together, these regimes motivate the question:

How can we best compress the drafter’s KV cache to sustain target alignment in long contexts?

We study this question through the trade-off between acceptance length (AL) and the KV cache budget size, measuring drafting quality under a fixed memory constraint. A fixed KV-budget introduces a tradeoff between KVs selected during prefill and KVs accumulated during decode. Prompt tokens can provide valuable information, while the generated prefix records the target’s recent context. A prefill-based selection becomes stale as generation evolves, while a sliding window (SW) approach limits the model’s view to the latest prefix tokens and discards remote information. Decode-time refresh of selected KVs allows to adaptively determine the relevant KVs, at the cost of additional latency. This motivates two complementary approaches: allocate capacity to both prefill positions and generated tokens, and refresh the selected history as relevance changes.

Our analysis characterizes decode staleness and introduce RIFT (Recency–Importance Fusion and Training), a family of selection-based KV compression strategies combining recency allocation with decode-time refresh of importance-selected context. We further propose compression-aware knowledge distillation (KD) to reduce the training–inference mismatch and improve draft alignment under compression. Our contributions are:

  1. Characterizing decode staleness: We show how prefill selection loses its initial advantage over SWA as generation progresses, and show this gap increases in hybrid-attention models.

  1. RIFT: We address decode staleness and propose RIFT, a family of KV compression methods for draft models, balancing implementation complexity with resulting drafting quality.

  1. Draft-compressibility training: We develop a knowledge-distillation scheme with compression-driven attention masks to improve draft alignment under a restricted KV budget.

Together, our methods retain most of the draft’s acceptance length, while using a fraction of the total KV cache. In SPEED-Bench ([1]) with 32K ISL and 4×4 \times compression ratio, Qwen3.5-4B retains up to 96% of its full cache AL (Qwen3.5-122B target), and Nemotron3.5-Lightning retains 97% of its full-cache AL (Nemotron3-Ultra target), while SnapKV retains 63 – 67% AL in both cases.

Background and Setting

Scope. We focus on two settings: 1) external autoregressive draft models and 2) KV-based self-speculative decoding. We leave tail-augmented drafters outside the scope of this work, as our preliminary studies find negligible benefit from KV compression in these settings (Appendix B).

Draft-side Compression. Rejection sampling preserves the target distribution regardless of the draft’s KV-compression policy. Since compressing the target cache can instead degrade output quality, we compress only the draft cache. We measure accuracy using average acceptance length (AL), the expected number of accepted tokens plus the extra target-generated token. We consider two strategies representing the endpoints of the spectrum of static KV selection methods under a budget BB:

SW attention. Pure SW attention (SWA) retains BB recent tokens, updating the cache at every decode step, independently of token importance.

Prefill-based KV compression. Prefill-based methods select KVs using importance scores aggregated during prefill. A widely used method, SnapKV (Li et al., [24]), aggregates attention scores from a set of last prompt queries, termed the observation window. For each KV head, SnapKV retains the highest-scoring KV positions alongside a SW the size of the observation window. Herein, we take an observation window of 64 tokens.

Decode Staleness In KV Compression

While prefill-only KV compression has proven beneficial for speculative decoding (Sadhukhan et al., [36]), prior evaluations do not fully reflect long OSLs and modern model architectures. We start by investigating prefill-based methods in speculative decoding. We use SnapKV to compress the KV cache of Qwen3.5-4B, drafting for Qwen3.5-122B and evaluate on SPEED-Bench 32K ISL low entropy split (Abramovich et al., [1]), with reasoning enabled. As seen in Figure 1(b), SWA outperforms SnapKV given the same KV budget. This result is surprising: SnapKV incorporates importance information, while SWA naively keeps a fixed tail of previous tokens. We make the following observations about the KV-compression literature:

AL improvement of RIFT and compressibility training

Figure 1. AL improvement of RIFT and compressibility training on a Qwen3.5-4B draft (Qwen3.5-122B target), SPEED-Bench 32K ISL low-entropy, draft length 11. (a) AL vs. generation interval at 8K cache budget. (b) AL vs. KV cache budget. AL is measured vs. the full-KV draft AL. Diagonal pattern denotes compression-aware training.

KV cache compression benchmarks often overlook long OSL. Evaluations centered on long inputs primarily test which prompt KVs to retain ([3, 19]). Regarding decode tokens, one can either 1) retain all generated KVs, 2) maintain a small sliding window alongside the prefill-selected cache, or 3) rescore and update the selection at every decoding step. The tradeoffs between these options become particularly important in speculative decoding, where the draft must remain aligned with a growing target-verified prefix under a fixed KV budget. Retaining all generated KVs violates the budget constraint, while a small sliding window can compromise alignment later in generation (as seen in Figure 1(a)). Rescoring at every decode step adds significant overhead to the drafting process.

To understand why a fixed prefill selection becomes less effective during generation, we examine how its relevance degrades, a phenomenon we term decode staleness, which motivates the following hypothesis:

Decode staleness: prefill-based KV selections become stale as generation progresses.

Additionally, we note that KV cache compression methods are rarely evaluated on hybrid-attention models. Yet, contemporary models interleave full attention with efficient recurrent components, including linear attention in Qwen3.5 (Yang et al., [43]; Qwen Team [35]) and Mamba layers in Nemotron 3 (Dao and Gu, [10]; NVIDIA [31]).

Output-position dependence. We analyze average draft AL across generation positions with an 8K-token budget (4×4 \times compression). As shown in Figure 1(a), SnapKV’s AL decreases by 36.7% while SW improves by 41% as generation evolves. We term the evolving SnapKV-SW difference the staleness gap, which we consider a performance-based proxy for the growing irrelevance of SnapKV’s selected KVs during generation. We attribute the growth of the staleness gap to two factors. First, in speculative decoding the recent decode prefix is produced by the target model, making access to it highly important for draft-target alignment. Second, recent decode tokens often contain reasoning traces, which potentially act as a running prompt summary ([33, 44]). Indeed, preliminary study shows that disabling reasoning reduces the staleness gap from 2.11 to 0.60 AL (See Figure 11 in Appendix § 11).

Interval AL versus generated-token position for Qwen3.5-4B drafting Qwen3.5-122B responses

Figure 11. Interval AL versus generated-token position for Qwen3.5-4B drafting Qwen3.5-122B responses, with thinking enabled and disabled. Reasoning magnifies the late staleness gap, but the SnapKV-SW crossover persists even without it.

Architecture dependence. We empirically observe that decode staleness is exacerbated in hybrid-attention architectures, which are often overlooked in the KV compression literature. Due to lack of space, we refer the reader to Appendix § 11, where we show that 1) a full-attention Llama3.1 draft retains the expected ordering: SnapKV consistently outperforms SWA throughout generation, while 2) in Nemotron3 (which is hybrid-SSM) and GPT-OSS (which interleaves hard-coded SWA layers with full attention), SWA again outperforms SnapKV.

Mechanistic evidence. We measure the fraction of the full-cache draft’s attention assigned to positions retained by each cache policy (SnapKV and SWA), under an 8K budget. Figure 2 shows that this fraction drops from 67.0% to 25.4% for SnapKV, while increasing from 43.5% to 68.7% for SWA as generation progresses. This crossover is consistent with attention shifting toward generated context that the prefill-selected cache largely omits.

Fraction of full-KV attention mass retained by SW and SnapKV

Figure 2. Fraction of full-KV attention mass retained by SW and SnapKV (with 64 token SW) in the Qwen3.5-4B draft on 32K-ISL data, with an 8K budget. SnapKV retains less attention as generation proceeds, while SW retains more.

RIFT: Staleness-Aware KV cache Compression

Motivated by the staleness gap, in this section we propose RIFT, a family of KV compression policies for draft models. RIFT comprises several solutions, ranging by their implementation complexity.

Effect of Sliding Window Ratio on Prefill-Based Methods

Our analysis emphasizes the importance of recent tokens to the draft’s KV cache. Thus, our first solution, which we call RIFT-stat, is a simple modification to a given static KV selection method.2 RIFT-stat balances a given budget BB between a prefill-based token selection algorithm (e.g. SnapKV) and a SW component. For a given SW ratio ρ∈(0,1]\rho\in(0, 1] and a cache budget BB, we reserve r=⌊ρB⌋r = \lfloor\rho B \rfloor positions for the most recent tokens and use the remaining k=B−rk = B - r positions for attention-based KV allocation. The SW ratio ρ\rho controls the trade-off between long-range retrieval and recent streaming context. Figure 3 shows the draft’s AL as a function of ρ\rho combining RIFT-stat with SnapKV (§ 4.4 presents results on several KV selection methods). With an 8K budget, the highest AL is achieved at ρ=0.5\rho= 0.5 (AL 5.614), an improvement of 37.8% over SnapKV (AL 4.073), with pure SWA achieving 5.230. In Appendix § 12.§ 1 we provide additional analysis on the effect of the SW ratio in RIFT-stat.

Effect of the sliding-window allocation ratio

Figure 3. Effect of the sliding-window allocation ratio. 4K OSL and 8K KV budget. Qwen3.5-4B drafts Qwen3.5-122B on SPEED-Bench 32K ISL data. ρ=0\rho= 0 denotes SnapKV with a fixed 64-token SW, and ρ=1\rho= 1 corresponds to pure SWA.

Block-Level RIFT-stat. Serving systems store KVs in pages ([21]). We therefore divide the cache into contiguous blocks, each scored by the mean of its keys’ scores. For an analysis of the block size effect on drafting performance, we refer the reader to Appendix § 12.§ 3.

RIFT-dyn: KV Cache Refresh During Decode

While RIFT-stat introduces gains in the draft AL, its chosen prefill KVs are still selected only once before decode. However, as generation progresses, the relevant prompt KVs dynamically change. We therefore assume access to the full draft KV cache in HBM or offloaded memory for full KV rescoring. We propose RIFT-dyn, which iteratively rescores the working KV cache every Δ\Delta decode steps. The resulting working KV cache is a combination of the decode-updated KV cache and the SW component of size ⌊ρB⌋\lfloor\rho B \rfloor. We focus on SnapKV-based scoring. However, RIFT-dyn is compatible with any alternative KV scoring functions. RIFT-dyn refreshes the draft’s working memory. Its effectiveness stems from two main advantages: (i) it scores the entire available KV cache (excluding the SW component), and (ii) it computes scores using a recent observation window, so they better reflect the model’s current state.

RIFT-dyn introduces a significant and consistent improvement in the draft’s AL (see Section 4.4 for results). For example, with a refresh interval of Δ=128\Delta= 128 and a Qwen3.5-4B drafter at 100K ISL, RIFT-dyn retains 95.7% of full-KV AL, compared with 85.6% for RIFT-stat. More frequent KV cache refreshes improve draft quality, with projected rescoring latency overhead (As seen in Figure 4). We next consider adaptive refresh to improve the AL-refresh tradeoff.

Effect of RIFT-dyn’s refresh interval on AL and projected rescoring overhead

Figure 4. Effect of RIFT-dyn’s refresh interval on AL (blue) and projected rescoring overhead relative to measured draft decode time (orange). Qwen3.5-4B, 100K ISL, 8K cache budget, and 4K OSL. Appendix § 12.§ 2 describes the latency projection.

Adaptive KV cache Refresh

RIFT-dyn improves draft accuracy by periodically refreshing the KV cache, even when the current selection remains useful. To avoid unnecessary refreshes, we propose RIFT-ada, which adaptively chooses when to refresh the KV cache based on signals collected at the draft during decode.

At decode step tt, let τ\tau be the decode position of the most recent KV scoring and at=t−τa_t = t - \tau the age of the current cache selection. Then, a KV rescore occurs if a threshold policy π(zt)\pi(z_t) is activated or ata_t exceeds a maximal no-refresh interval amax⁡a_{\max}:

ut=1{(at≥amax⁡)∨(π(zt)∧at≥amin⁡)},u_t = \mathbb{1}\{(a_t \ge a_{\max}) \lor(\pi(z_t) \land a_t \ge a_{\min})\},

where ut=1u_t = 1 triggers a rescore. The minimum age amin⁡a_{\min} prevents consecutive refreshes, while the maximum age amax⁡a_{\max} bounds staleness. We evaluate utu_t every d=8d = 8 steps.

Threshold Policies. At each dd-token decision interval, we aggregate several signals ztz_t: an exponentially weighted mean of previous measured acceptance lengths AtA_t, draft entropy HtH_t, and query drift, which is given by Dtq=1−qˉt⋅qˉτD_t^q = 1 - \bar{q}_t \cdot\bar{q}_\tau and qˉ\bar{q} denotes a normalized query summary. We average DtqD_t^q over full-attention layers and KV-head groups. We compare each of these signals with a calibrated threshold (See Appendix § 13.§ 3 for details on the calibration process), and aggregate multiple scores into a threshold policy using (and/or) gates or a fitted weighted score. Figure 5 illustrates how RIFT-ada varies refresh frequency during generation, compared with RIFT-dyn’s periodic schedule, showing how various stages in generation require different rescoring cadence. The evaluated policies are detailed in § 4.4.

KV rescore patterns comparison

Figure 5. KV rescore patterns comparison on a candidate sample from SPEED-Bench. RIFT-ada attains a higher AL and requires less KV refreshes. It can be seen that KV refreshes are not periodic and depend on generation location.

Results

We evaluate RIFT on SPEED-Bench, which comprises 8 different generation benchmarks, focusing on its throughput splits. Unless stated otherwise, we use draft length of 11, maximum OSL of 4K and decode temperatures T∈{0,1}T \in\{0,1\}. The KV cache budget is 8K and accuracy is measured in average AL. Additional results are provided in Appendix § 12, where we cover more drafter-target pairs, datasets and generation setups. All RIFT variations use ρ=0.5\rho= 0.5. For RIFT-ada, we use the same refresh framework throughout and consider a threshold that results in strictly fewer refreshes than RIFT-dyn (see Appendix § 13.§ 3 for more details). We also provide preliminary results on RIFT applied beyond speculative decoding in Appendix § 14: we evaluate RIFT-stat on a long-input, long-output reasoning benchmark, showing that its advantage extends beyond speculative decoding.

AL vs. Budget. We start by demonstrating the accuracy of the various RIFT variants across several evaluation settings. First, we consider Qwen3.5-4B drafting for Qwen3.5-122B. Figure 1(b) compares RIFT with SnapKV and SWA on SPEED-Bench 32K ISL across cache budgets. Figure 1(a) depicts AL vs. decode position, showing RIFT-stat benefits from both worlds: it preserves SnapKV’s high early-generation AL while benefiting from the SW component later, as expected from § 3. Table 1 extends the comparison to 100K ISL and additional model pairs. Specifically, we consider Nemotron3.5-Lightning (self-speculation and drafting for Nemotron3-Ultra) and

Draft (ISL)TTSWSnapKVRIFT statRIFT dynRIFT ada
Qwen3.5 (32K)05.234.075.626.126.19
Qwen3.5 (32K)14.703.694.995.485.55
Qwen3.5 (100K)06.124.466.407.147.17
Qwen3.5 (100K)15.403.825.556.216.36
Llama3.1 (32K)02.245.196.076.917.48
Llama3.1 (32K)11.553.834.545.055.52
N3.5 (self) (32K)05.734.836.728.268.71
N3.5 (self) (32K)15.564.516.848.708.93
N3.5 (ext.) (32K)04.313.644.354.994.98
N3.5 (ext.) (32K)13.863.223.934.454.99

Table 1. AL on low-entropy SPEED-Bench 32K/100K ISL, with an 8K budget, 4K OSL, and draft length 11. Qwen3.5: 4B → 122B; Llama3.1: 8B → 70B; Nemotron3.5-Lightning in self-speculation and as an external draft to Nemotron3-Ultra target.

Llama3.1-8B (Llama3.1-70B target), which is a full-attention model. For Qwen3.5 at 100K ISL, an 8K budget, and T=0T = 0, RIFT-stat, RIFT-dyn, and RIFT-ada retain 85.6%, 95.5%, and 96.0%, respectively, compared with 59.7% for SnapKV. Appendix D provides additional results, encompassing various budgets, models, OSLs and workloads. Appendix G demonstrates RIFT’s advantage across various generation tasks.

Beyond attention scores. While we demonstrate RIFT-stat on top of SnapKV, it can be used with any other prefill-based KV compression method. We combine several additional KV selection scores with RIFT-stat: 1) CriticalKV’s output-perturbation score Feng et al., [15], 2) Expected Attention’s distribution over future queries Devoto et al., [11], and 3) CurDKV’s leverage scores Sengupta et al., [37]. We measure the benefit of applying RIFT-stat on top of each method. As seen in Table 2, all three methods improve substantially at both budgets (see Table 14 for T=1T = 1). The proposed observations naturally extend to direct variants of SnapKV, such as Ada-KV and PyramidKV, which focus on layer/head budget allocation [14, 6].

BudgetSelection scoreBaseline+ RIFT-stat
4KCriticalKV3.575.02 (+40.6%)
Expected Attention3.595.03 (+40.2%)
CurDKV3.294.84 (+47.1%)
8KCriticalKV4.135.62 (+36.2%)
Expected Attention4.065.56 (+36.9%)
CurDKV3.765.35 (+42.2%)

Table 2. Absolute AL at T=0T = 0 (relative gains in parentheses): fixed 64-token SW versus RIFT-stat (ρ=0.5\rho= 0.5). Qwen3.5-4B drafts Qwen3.5-122B on SPEED-Bench 32K ISL, 4K OSL, draft length 11.

BudgetSelectorT=0T = 0T=1T = 1
4KCriticalKV3.57 → 5.02 (+40.6%)3.15 → 4.33 (+37.3%)
4KExpected Attention3.59 → 5.03 (+40.2%)3.09 → 4.30 (+39.0%)
4KCurDKV3.29 → 4.84 (+47.1%)2.84 → 4.14 (+45.5%)
8KCriticalKV4.13 → 5.62 (+36.2%)3.74 → 4.98 (+33.1%)
8KExpected Attention4.06 → 5.56 (+36.9%)3.56 → 4.83 (+35.9%)
8KCurDKV3.76 → 5.35 (+42.2%)3.26 → 4.63 (+42.1%)

Table 14. RIFT-stat extends beyond SnapKV: Qwen3.5-4B draft (Qwen3.5-122B target), 32K ISL, 4K OSL with draft length of 11. Each entry shows baseline AL → AL with RIFT-stat, followed by the relative improvement.

Cross-Family Drafting We consider a cross-family external drafting setting. The main challenge in this setup is that the draft and target models do not share the same vocabulary. We adapt drafters through a simple tokenizer matching step followed by target-model KD (see Appendix § 16 for tokenizer-adaptation details). We consider several cross-family draft-target pairs and apply RIFT in a long-ISL drafting setting. Specifically, we consider Llama3.2-1B and Qwen3.5-4B drafts and DeepSeek-V4-Flash/Pro and Flash as target models. As seen in Table 3, RIFT extends to the cross-model scenario, supporting external drafting when an in-family smaller model is unavailable.

DraftTargetSnapKV + RIFT-stat+ RIFT-dyn
Qwen3.5-4BDSV4-Flash3.504.265.35
DSV4-Pro3.434.265.07
Llama3.2-1BDSV4-Flash2.503.243.50
DSV4-Pro2.373.073.26

Table 3. Absolute AL for adapted drafters of DeepSeek-V4-Flash/Pro on low-entropy SPEED-Bench: 32K ISL, 8K budget, 4K OSL, T=0T = 0, draft length 11.

The benefit of adaptive refresh. We compare RIFT-ada with RIFT-dyn’s periodic refresh through their AL–refresh tradeoff, measuring AL gains at matched refresh counts and refresh reductions at matched AL. The policy triggers a refresh if any of the following conditions holds: acceptance is at or below its threshold, entropy is at or above its threshold, acceptance has fallen or entropy has risen sufficiently relative to post-refresh values, or query drift exceeds its threshold. As seen in Figure 6, this policy, which sweeps the query drift threshold, achieves 2.03× fewer refreshes (or up to +0.122 AL at the same refresh count). We refer the reader to Appendix § 13 for additional settings, parameter-transfer analysis and policy information.

RIFT-ada AL-refresh tradeoff plot

Figure 6. RIFT-ada AL-refresh tradeoff at T=1T = 1 for Qwen3.5-4B drafting Qwen3.5-122B on SPEED-Bench 32K low-entropy data, with an 8K budget and 4K OSL.

Comparison with additional baselines. We compare RIFT with StreamingLLM Xiao et al., [41], H2_2O Zhang et al., [46] and TriAttention Mao et al., [30], which are decode-time KV selection methods. As seen in Table 4, StreamingLLM is outperformed by all compression methods, H2O outperforms RIFT-stat, while RIFT-dyn and RIFT-ada achieve higher AL than either baseline across the two context lengths and temperatures. Appendix § 17 provides comparison across additional settings.

Method32K ISL, T=0T = 032K ISL, T=1T = 1100K ISL, T=0T = 0100K ISL, T=1T = 1
StreamingLLM5.344.735.604.98
H2O5.735.036.655.77
TriAttention5.434.776.105.11
RIFT-stat5.624.996.405.55
RIFT-dyn6.125.487.146.21
RIFT-ada6.175.517.176.36

Table 4. Absolute AL against decode-time baselines. Qwen3.5-4B draft, low-entropy SPEED-Bench, 8K budget, 4K OSL, and draft length 11.

Improving Draft KV Compressibility with Training

The previous section improves KV compression at inference with a fixed draft. A natural question is:

Can we further improve the drafter’s KV compressibility using dedicated training?

Inspired by recent results in compressibility training of pretrained models [16], we propose a draft compressibility post-training phase. Our goal is to minimize the gap between the draft’s working KV cache in training and during inference. In what follows, we focus on compressibility using attention scores. However, the training method itself is score-agnostic. The resulting objective promotes target-aligned proposals under a restricted KV budget.

Compression-aware knowledge distillation. We propose a modified KD forward pass. We begin with a dense scoring pass by applying a forward pass to the frozen draft model. We capture

post-RoPE queries and keys in each attention layer. We score keys by summing their attention scores against a fixed observation window. We define a compression ratio pp and a SW ratio ρ\rho. During training, the effective cache budget is B=⌊pL⌋B = \lfloor pL \rfloor, of which r=⌊ρB⌋r = \lfloor\rho B \rfloor positions are reserved for a SW component, where LL is the sample’s sequence length. We build an attention mask that combines the SWA with the highest-scoring prompt positions, obtained from the dense scoring pass. We then perform an additional student forward pass, in which we apply the LoRA adapters and the built attention masks to get a sparse forward pass. We combine cross-entropy and KL distillation using precomputed teacher rollouts and logits:

L=λ1CE⁡(qθ,y)+λ2DKL(p∥qθ),\mathcal{L} = \lambda_1 \operatorname{CE}(q_\theta,y) + \lambda_2 D_{\mathrm{KL}}(p\mathbin{\|}q_\theta),

where yy is the teacher-generated next token, and qθq_\theta and pp are the student and teacher distributions. The student uses the sparse attention mask. Figure 7 provides an overview. Our method extends to self-speculation by toggling the adapters and using the corresponding selection method during inference.

Compression-aware distillation diagram

Figure 7. Compression-aware distillation. The snowflake and fire symbols denote frozen and trainable passes, respectively. The mask combines recent positions (purple) with importance-selected context (green).

Results

Setting. We train for 10K steps using the Nemotron V2 post-training-dataset,3 with batch size 64 and SW ratio ρ=0.25\rho= 0.25. To increase budget robustness we randomly sample compression ratios for each sample. Evaluation uses SPEED-Bench 32K low-entropy data, 4K OSL, and draft length 11. Appendix § 18.§ 2 gives the full recipe.

Compression-aware gains. We evaluate Qwen3.5 and Nemotron3.5-Lightning, considering both self speculation and external drafting. As seen in Table 5, compressibility training improves the drafter’s performance under RIFT. For example, we see gains of up to 16.19% for Qwen3.5-4B (RIFT-stat, 2K budget, T=1T = 1) and 16.42% gains for Nemotron3.5-Lightning (RIFT-dyn at 2K and T=1T = 1). Both comparisons measure improvement over the untrained drafter under the same compression policy and budget. We note that gains are smaller at larger budgets, as the baseline method often already retains more than 95% of the full-KV AL, leaving a very small gap for improvement.

Draft →\to targetTTRIFT-stat 2KRIFT-stat 4KRIFT-stat 8KRIFT-dyn 2KRIFT-dyn 4KRIFT-dyn 8K
Qwen3.5 (4B →\to 122B)0+7.70%+6.87%+5.70%+7.48%+5.50%+3.72%
1+16.19%+14.88%+12.28%+15.51%+12.25%+9.21%
Lightning →\to Nemotron3-Ultra0+7.51%+7.48%+7.59%+7.88%+7.86%+5.59%
1+15.69%+15.63%+15.08%+16.42%+15.84%+13.19%
Lightning self-speculation0+5.3%+3.4%+2.5%+6.5%+2.8%+1.0%
1+11.0%+7.8%+4.2%+10.3%+5.0%+1.5%

Table 5. Relative AL improvement from compression-aware KD over the base model under the same KV compression policy and budget, evaluated on SPEED-Bench 32k ISL with draft length 11.

Benefit over no-mask training. To isolate the benefit of compression-aware training from that of the post-training data, we compare with the same pipeline without attention masking. As seen in Table 6, compressibility training offers a significant improvement on top of the baseline training scheme, especially in the lower budget regimes.

MethodTT2K4K8K
RIFT-stat0+5.44%+4.68%+3.71%
1+7.31%+6.37%+4.66%
RIFT-dyn0+5.34%+3.51%+1.82%
1+7.25%+4.73%+2.37%

Table 6. Relative AL improvement from compression-aware training over a similar training recipe with no compression-driven masks. Qwen3.5-4B→\toQwen3.5-122B, SPEED-Bench 32k ISL, draft length 11.

Related Work

Due to the amount of existing methods, we describe the most closely related work to our methods. We provide a broader discussion of literature in Appendix K. SnapKV estimates prompt importance from observation-window attention and PyramidKV and AdaKV refine allocation across layers and heads [24, 6, 14]. These methods optimize which KVs survive compression, while we study how prefill decisions become stale during long generation. Decode-time policies continually adapt the working cache, e.g., StreamingLLM retains attention sinks and recent tokens, while H2O combines recent tokens with accumulated-attention heavy hitters [41, 46]. TriForce uses a retrieved target cache as an intermediate drafter, refreshed periodically or by acceptance gating (Sun et al., 2024). MagicDec studies sparse-KV self-speculation at high concurrency. SpecExtend, SparseSpec, and Vegas study how verification can be used for KV cache compression [36, 7, 47, 45]. In compressibility training, KV-CAT [16] proposes a continual pre-training phase with a learned KV scoring model.

Conclusion

In this work, we identified decode staleness, where prefill-selected context loses relevance as OSL grows, especially in hybrid attention architectures. We show that decode staleness degrades the draft-target alignment in speculative decoding. We proposed RIFT: RIFT-stat combines recency with important prompt positions, RIFT-dyn improves KV selection by rescoring the cache during decode and RIFT-ada allows for adaptive KV rescore to minimize the induced latency overhead. To improve RIFT we propose a compression-aware training phase. Together, our results demonstrate the value of adapting both KV cache selection policies and draft models to retain acceptance lengths under fixed KV cache budgets. A promising future direction is to combine RIFT’s draft-side staleness signals with target-verification attention, as initial signals show promising results in this direction [e.g. 7, 45].

Limitations

At the time of writing, external draft models generally provide lower end-to-end speedups in GPU-only deployments than tail-augmented drafters such as MTP, EAGLE, and DFlash. Our work is therefore motivated by future heterogeneous systems in which drafting runs on a memory-constrained, SRAM-based accelerator, an emerging direction for which [2] provides an early signal. In such a system, KV-selection scores could be computed on the GPU using a copy of the draft model, while only the selected KV blocks are transferred to the drafting device. Our evaluation focuses on acceptance-length trade-offs under a fixed active draft KV cache budget. Because decode-time refresh assumes that the full draft KV history remains available in GPU HBM or offloaded memory, this active budget constrains the drafting device’s memory footprint but not the system’s total stored KV memory. Furthermore, on a GPU setup with naive measurements, we see a relative cost overhead of up to 6.0% of KV cache rescoring compared to the total draft decode latency. This overhead may become more significant when drafting is accelerated on an SRAM-based device, although its magnitude in disaggregated deployments remains to be measured.

References

Appendix for “RIFT: KV Cache Compression for Draft Models with Complementary Compressibility Training”

KV cache Costs in Hybrid Draft Models

We quantify the KV cache footprint in Qwen3.5-4B and Nemotron-3-Nano (30B-A3B) relative to the combined memory (of weights, KV cache and recurrent states). Concurrent requests share the same weights but retain separate caches. KV memory grows with context length and concurrency while recurrent states (such as SSM and GDN states) grow with concurrency but not with ISL or OSL. We assume bf16 weights and KV entries, fp32 recurrent matrices, and bf16 convolution buffers, giving approximately 49.5 MiB of state per request for Qwen and 47.1 MiB for Nemotron3-Nano. Figure 8 shows that long-context KV storage can still dominate. For Qwen3.5-4B, four requests at 64K tokens use 51.1% of the occupied memory, despite only 25% of layers using full attention. Nano’s smaller per-token cache and larger resident weight footprint delay the crossover. Nonetheless, at 256K ISL and 64 concurrent requests, its KV cache accounts for 62.0% of the accounted memory. Consequently, it is clear that the KV memory bottleneck remains evident in standard serving loads.

Attention KV cache share of total memory in two hybrid architectures, evaluated at the end of prefill.

Figure 8. Attention KV cache share of total memory in two hybrid architectures, evaluated at the end of prefill.

KV Cache Compression in Tail-Augmented Drafters

We provide additional information on our initial results for KV compression’s effect on MTP. Specifically, we examined Nemotron3-Super’s MTP draft, which went through a post-training phase with a 1K SW. The training used the Nemotron post training V2 dataset, using a standard KD scheme for 10K training steps. The resulting MTP was evaluated on SPEED-Bench 32k and 100k ISL data. The MTP’s ALs were 4.24 (94% of full KV AL) at 32K ISL and 5.07 (95% of full KV AL) at 100K ISL. Consequently, a simple SWA shows sufficient AL retention in the settings of interest, rendering it a sufficient and simpler option over RIFT for MTPs. Furthermore, the overall reduction in the system’s TPS due to the MTP’s KV compression was negligible, implying that in a tail-augmented drafting setting, the MTP’s KV cache may not pose as a significant bottleneck compared to external drafting and self-speculation.

Additional Staleness Analysis

We provide additional evidence on draft staleness. Figure 10 extends the comparison across model families: SW outperforms SnapKV late in generation for GPT-OSS and Nemotron, as for Qwen3.5 in Figure 1(a), while Llama-3.1 consistently favors SnapKV over SW. Qwen3, which is a full-attention model, shows ambiguous results in terms of one method’s dominance over the other. Next, we evaluate the effect of reasoning. Figure 11 compares Qwen3.5-4B’s drafting accuracy with thinking enabled and disabled (at both the draft and target models). Disabling thinking reduces the staleness gap in the final position bucket from 2.11 to 0.60 AL, although the SnapKV-SW crossover persists. All results use T=0T = 0, low-entropy SPEED-Bench 32k ISL data with a fixed 8K KV cache budget.

Interval AL across model families with full-KV references

Figure 10. Interval AL across model families, with full-KV references. Target/draft pairs (left to right): GPT-OSS-120B/20B, Nemotron-Super-120B/Nano-30B, Qwen3-235B/8B, and Llama-3.1-70B/8B. Each panel uses its own AL scale. The corresponding Qwen3.5 comparison appears in Figure 1(a).

Full-attention mass shifts from prompt tokens to the generated prefix

Figure 9. Full-attention mass shifts from prompt tokens to the generated prefix in the Qwen3.5-4B draft on 32K-ISL data.

Additional Staleness-Aware Compression Results

RIFT-stat Sliding-Window Ratio

This experiment uses the base Qwen3.5-4B draft without fine-tuning. Figure 12 supplements Figure 3 with five additional output-cap and KV-budget combinations, evaluated on the same precomputed Qwen3.5-122B responses at T=0T = 0, with draft length 11, block size 128, and observation window 64. The best observed ratios range from 0.3 to 0.7 across these settings, with modest gains over the default ρ=0.5\rho= 0.5. In each panel, the best intermediate ratio outperforms both fixed-64 SnapKV and pure SWA. Output length is a cap, so naturally shorter responses retain their original lengths; comparisons across panels change the output cap and/or KV budget.

Additional sliding-window ratio ablations

Figure 12. Additional sliding-window ratio ablations with the same base draft model, requests, and evaluation settings as Figure 3. Panel titles specify the output-length cap and KV budget. Stars mark ρ=0.5\rho= 0.5 in the first four panels and the optimum ρ=0.6\rho= 0.6 in the 1K-output, 1K-budget panel (AL 3.648, versus 3.597 at ρ=0.5\rho= 0.5); the zero-labeled endpoint denotes fixed-64 SnapKV, and ρ=1\rho= 1 denotes pure SWA. Broken diagonals indicate omitted AL range.

Rescoring Overhead Estimation

Figure 4 in the main paper projects the added rescoring overhead for Qwen3.5-4B with 100K ISL and 4K OSL. For a given refresh interval, we begin by calculating the drafter’s AL under RIFT-dyn with the corresponding refresh interval. Then, we run the speculative decoding loop in vLLM with the corresponding forced AL and a fixed 8K draft KV cache to measure decode latency. We add the estimated vLLM rescoring time for the given interval, and report the added time as a percentage of measured decode latency.

Block-Size Ablation

As described in § 4, to accommodate real-world serving scenarios, we work with block-level KV cache compression. We examine the effect of block size on the draft’s performance. We consider Qwen3.5-4B with draft length 11, drafting for Qwen3.5-122B using the SPEED-Bench dataset, 100K and 32K throughput splits. In each setting, we apply RIFT-stat with the corresponding block size. We consider two generation temperatures and sweep various block sizes. As seen in Figure 15, when the block size is significantly smaller than the overall KV cache, the effect is negligible. However, each setting has a "critical block size" beyond which the decision space is too coarse and the draft’s performance degrades.

Six line plots showing average acceptance length versus RIFT-stat block size for different entropy, context length, and temperature settings

Figure 15. Effect of SnapKV block size for Qwen3.5-4B drafting for Qwen3.5-122B with a 50/50 local–remote cache split. The 32K experiments use an 8K cache budget; the 100K experiments use a 16K budget. The top and bottom rows report T=0T = 0 and T=1T = 1, respectively. Moderate block sizes are broadly robust, whereas very coarse blocks reduce selection resolution and lower AL.

Budget, Context, and Cross-Family Comparisons

We supplement the results in § 4.4 with additional cache-budget sweeps in various settings. Additional Qwen Family Results Table 7 expands the Qwen3.5 results across context length, output entropy, temperature, and cache budget. The gains from allocating cache capacity to recent tokens and refreshing the remote selection remain consistent throughout these settings. Table 8 presents additional results with a Qwen3.5-0.8B draft model. As can be seen, the performance extends to smaller models.

ISLEntropyTTBudgetSWSnapKVRIFT-statRIFT-dynRIFT-ada
32KLow02K64.8%49.7%70.3%79.5%81.5%
32KLow04K71.3%54.5%77.7%88.5%90.2%
32KLow08K80.9%63.0%86.9%94.6%95.8%
32KLow12K61.8%47.1%65.9%74.7%77.0%
32KLow14K68.0%52.0%72.5%84.1%86.3%
32KLow18K78.6%61.8%83.4%91.7%92.8%
32KHigh02K79.1%61.9%81.8%88.5%90.1%
32KHigh04K84.0%64.4%88.8%94.1%94.7%
32KHigh08K85.9%66.9%93.1%97.6%97.7%
100KLow02K64.1%50.7%72.1%81.7%84.7%
100KLow04K75.5%55.5%77.8%90.3%92.4%
100KLow08K81.8%59.7%85.6%95.5%96.0%
100KHigh02K72.7%60.0%80.2%86.9%88.6%
100KHigh04K81.1%62.5%86.7%92.1%93.2%
100KHigh08K84.7%65.2%90.7%95.9%96.3%

Table 7. retained AL (%) of full-KV, Qwen3.5-4B draft with, Qwen3.5-122B target across SPEED-Bench workloads (ISL and entropy category) and cache budgets. The absolute full-KV ALs for 32K low TT = 0/1, 32K high TT = 0, 100K low TT = 0, and 100K high TT = 0 are 6.466/5.978, 4.617, 7.472, and 4.506, respectively. All settings use draft length 11 and maximum output length 4K.

ISLEntropyTTSWSnapKVRIFT-statRIFT-dynRIFT-ada
32KLow085.5%67.3%89.0%95.9%98.1%
32KLow185.5%68.2%86.9%94.2%96.4%
32KHigh091.5%75.0%95.6%97.9%98.6%
32KHigh192.6%77.7%94.9%97.1%97.9%
100KLow082.6%61.0%87.5%94.6%97.4%
100KLow184.6%62.0%86.9%92.7%97.1%
100KHigh090.6%71.9%94.6%96.3%97.7%

Table 8. Base Qwen3.5-0.8B draft performance at an 8K cache budget, expressed as a percentage of the matching Full KV AL for Qwen3.5-122B target responses. The absolute full-KV ALs for 32K low TT = 0/1 and high TT = 0/1 are 4.807/4.148 and 3.272/2.836; for 100K low TT = 0/1 and high TT = 0, they are 5.763/4.748 and 3.225.

Additional model families and cross-family drafting We extend our experiments beyond the Qwen3.5 family and the pairs considered in § 4.4. We provide several settings:

  1. Nemotron3 family - Due to their sizes, we consider Nemotron3-Super and Nemotron3-Ultra as target models, with Nemotron3-Nano and Nemotron3.5-Lightning serving as their draft models. Additionally, we consider a self-speculation setting using Nemotron3.5-Lightning. Numerics are given in Table 9.

DraftTargetTTBudgetSWSnapKVRIFT-statRIFT-dyn
NanoSuper02K53.0%41.8%52.6%62.1%
NanoSuper04K57.1%45.1%57.2%73.2%
NanoSuper08K63.4%52.8%63.9%83.4%
NanoUltra02K69.1%56.0%69.9%76.6%
NanoUltra04K74.4%60.5%75.2%85.9%
NanoUltra08K81.3%68.4%82.8%93.2%
LightningSuper02K52.6%42.1%53.3%66.9%
LightningSuper04K56.3%45.1%56.9%78.2%
LightningSuper08K62.2%51.9%62.2%90.7%
LightningSuper12K56.1%44.8%57.1%68.4%
LightningSuper14K60.7%48.6%61.7%78.7%
LightningSuper18K68.2%56.2%68.1%90.2%
LightningUltra02K67.0%54.6%69.6%77.2%
LightningUltra04K72.7%59.3%74.0%84.8%
LightningUltra08K79.9%67.4%80.7%92.6%
LightningUltra12K66.1%53.8%69.8%76.7%
LightningUltra14K71.8%58.1%74.1%83.9%
LightningUltra18K79.5%66.2%80.8%91.6%
LightningLightning02K43.0%35.8%46.8%57.6%
LightningLightning04K49.7%40.5%54.1%70.2%
LightningLightning08K57.3%48.3%67.2%82.6%
LightningLightning12K35.3%29.2%40.2%51.2%
LightningLightning14K41.4%33.4%47.1%64.2%
LightningLightning18K49.5%40.2%60.9%77.5%

Table 9. Full-KV AL retention in Nemotron3-family, low-entropy SPEED-Bench 32K split. The absolute full-KV ALs are 6.944 (Nano → Super, T=0T = 0), 4.994 (Nano → Ultra, T=0T = 0), 6.647/5.520 (Lightning → Super, T=0/1T = 0/1), 5.394/4.859 (Lightning → Ultra, T=0/1T = 0/1), and 10.009/11.222 (Lightning self-speculation, T=0/1T = 0/1).

  1. Additional Model families - Llama3.1 (8B drafting for 70B) serving as a full-attention example, and GPTOSS (20B drafting for 120B) serving as a mixed-attention example. Numerics are given in Table 10.

Draft → targetBudgetSWSnapKVRIFT-statRIFT-dyn
Llama-3.1-8B → Llama-3.1-70B2K26.6%53.7%64.3%70.5%
Llama-3.1-8B → Llama-3.1-70B4K27.6%58.1%70.1%79.7%
Llama-3.1-8B → Llama-3.1-70B8K28.6%66.1%77.4%88.0%
GPT-OSS-20B → GPT-OSS-120B2K62.3%32.0%56.4%67.7%
GPT-OSS-20B → GPT-OSS-120B4K65.5%32.6%64.9%76.8%
GPT-OSS-20B → GPT-OSS-120B8K69.6%34.0%71.4%87.8%

Table 10. Llama3.1 and GPT-OSS cache-budget sweeps on the T=0T = 0, low-entropy SPEED-Bench 32K workload. Values are AL as a percentage of the matching full-KV AL. The absolute full-KV AL is 7.849 for Llama and 5.746 for GPT-OSS. All settings use draft length 11 and maximum output length 4K.

  1. We provide additional cross-family drafting results. First, we consider additional draft models for the DeepSeek-V4 targets (Flash and Pro) in Table 11 and Table 12. We consider Qwen3.5-4B, Qwen3.5-0.8B and Llama3.2-1B. Second, we consider a Kimi-K3 target model and a Qwen3.5-4B draft in Table 13.

ISLDraftBudgetSnapKVRIFT-statRIFT-dyn
32KQwen3.5-0.8B2K48.4%63.3%78.8%
32KQwen3.5-0.8B4K54.7%69.9%87.6%
32KQwen3.5-0.8B8K63.2%77.6%93.4%
32KQwen3.5-0.8B16K70.5%84.7%97.5%
32KQwen3.5-4B2K49.3%63.5%78.9%
32KQwen3.5-4B4K54.7%70.0%88.2%
32KQwen3.5-4B8K63.7%79.0%94.1%
32KQwen3.5-4B16K70.1%85.2%98.0%
32KLlama3.2-1B2K51.6%66.8%70.6%
32KLlama3.2-1B4K54.1%71.3%75.9%
32KLlama3.2-1B8K59.8%77.3%82.1%
32KLlama3.2-1B16K67.4%84.7%90.5%
100KQwen3.5-0.8B2K57.9%76.1%84.6%
100KQwen3.5-0.8B4K63.1%81.2%92.4%
100KQwen3.5-0.8B8K66.8%88.9%96.0%
100KQwen3.5-0.8B16K69.4%92.2%97.7%
100KQwen3.5-4B2K56.5%74.2%83.3%
100KQwen3.5-4B4K62.3%79.5%91.5%
100KQwen3.5-4B8K65.9%87.6%95.2%
100KQwen3.5-4B16K68.5%91.5%97.1%
100KLlama3.2-1B2K56.1%77.5%80.4%
100KLlama3.2-1B4K58.6%82.2%85.0%
100KLlama3.2-1B8K61.8%89.4%91.4%
100KLlama3.2-1B16K65.0%92.3%94.4%

Table 11. DeepSeek V4 Pro with trained external drafters on the T=0T = 0, low-entropy SPEED-Bench workloads. For Qwen3.5-0.8B, Qwen3.5-4B, and Llama3.2-1B, respectively, the absolute full-KV ALs are 4.343/5.389/3.969 at 32K and 4.263/5.427/3.674 at 100K.

ISLDraftBudgetSnapKVRIFT-statRIFT-dyn
32KQwen3.5-0.8B2K40.9%54.4%75.0%
32KQwen3.5-0.8B4K50.6%62.6%85.5%
32KQwen3.5-0.8B8K59.8%72.4%92.7%
32KQwen3.5-0.8B16K67.9%79.3%97.2%
32KQwen3.5-4B2K42.7%55.3%75.1%
32KQwen3.5-4B4K51.3%63.1%86.4%
32KQwen3.5-4B8K61.3%74.5%93.7%
32KQwen3.5-4B16K68.9%80.7%97.8%
32KLlama3.2-1B2K46.6%59.3%64.1%
32KLlama3.2-1B4K50.1%64.9%71.2%
32KLlama3.2-1B8K56.7%73.6%79.6%
32KLlama3.2-1B16K65.7%81.0%89.4%
100KQwen3.5-0.8B2K55.0%62.2%78.7%
100KQwen3.5-0.8B4K63.4%71.1%89.0%
100KQwen3.5-0.8B8K70.4%86.1%95.0%
100KQwen3.5-0.8B16K76.3%92.7%97.3%
100KQwen3.5-4B2K56.0%62.5%79.8%
100KQwen3.5-4B4K65.0%71.2%89.6%
100KQwen3.5-4B8K71.8%86.4%95.1%
100KQwen3.5-4B16K76.4%92.9%97.6%
100KLlama3.2-1B2K49.2%61.4%64.8%
100KLlama3.2-1B4K54.0%69.6%75.1%
100KLlama3.2-1B8K60.2%84.4%87.9%
100KLlama3.2-1B16K67.0%91.5%93.6%

Table 12. DeepSeek V4 Flash with trained external drafters on the T=0T = 0, low-entropy SPEED-Bench workloads. For Qwen3.5-0.8B, Qwen3.5-4B, and Llama3.2-1B, respectively, the absolute full-KV ALs are 4.830/5.714/4.401 at 32K and 5.308/6.338/4.896 at 100K.

TTRIFT-stat 2KRIFT-stat 4KRIFT-stat 8KRIFT-dyn 2KRIFT-dyn 4KRIFT-dyn 8KRIFT-ada 2KRIFT-ada 4KRIFT-ada 8K
060.5%66.0%72.7%84.3%93.4%97.1%88.4%94.8%98.3%
156.6%62.5%69.3%81.5%91.6%96.1%87.2%94.7%98.2%

Table 13. Post-training AL as a percentage of the matching full-KV AL for cross-tokenizer Qwen3.5-4B → Kimi-K3 drafting. The absolute full-KV AL is 4.834/5.138 at T=0/1T = 0/1.

It can be seen that the performance of RIFT is maintained across all considered settings. It can also be seen that some setups benefit less from RIFT-stat,

showing that similar performance can be obtained with SWA at the same budget (see, for example, Nemotron3-Nano drafting for Nemotron3-Ultra or Super in Table 9). However, the benefit of dynamically rescoring and updating the KV cache during decode dramatically increases the draft’s performance, regardless of the specific setup considered. As a result, the advantage of the decode-level methods grows with OSL. We note that forcing high OSLs produces highly repetitive generated text, which may affect both the measured ALs and the relative gaps between methods. Thus, the observed trends should therefore be interpreted as results for forced continuations rather than naturally long, meaningful responses.

Additional Prefill Selectors

Table 14 reports the absolute ALs underlying the relative gains in Table 2.

Effect of draft length

We evaluate the effect of draft length on performance in Full-KV, SnapKV, SW, RIFT-stat, and RIFT-dyn at T∈{0,1}T \in\{0,1\}. As seen in Figure 13, the performance gap persists across draft lengths: RIFT-stat and RIFT-dyn consistently outperform SnapKV, with RIFT-dyn closest to Full-KV. The relative AL gain of RIFT-dyn over SnapKV grows from 17.7% to 60.0% at T=0T = 0 and from 17.9% to 54.8% at T=1T = 1 as draft length increases.

Effect of draft length for Qwen3.5-4B drafting for Qwen3.5-122B-A10B

Figure 13. Effect of draft length for Qwen3.5-4B drafting for Qwen3.5-122B-A10B on low-entropy SPEED-Bench 32K prompts, with a 4K output cap. Compressed methods use an 8K KV budget and block size 128; Full-KV retains the full context. Each point is mean per-request AL.

AL vs. OSL

We evaluate how acceptance changes with OSL on SPEED-Bench 32K low-entropy data. We force each target response to meet OSL shown in Figure 14. In this forced-output setting, the advantage of RIFT-stat over SWA diminishes as OSL grows: their mean-AL gap narrows from 0.42 to 0.09 at T=0T = 0 and from 0.29 to 0.03 at T=1T = 1 between 2K and 64K OSL. In contrast, the advantage of the decode-level methods grows with OSL. We note that forcing high OSLs produces highly repetitive generated text, which may affect both the measured ALs and the relative gaps between methods. Thus, the observed trends should therefore be interpreted as results for forced continuations rather than naturally long, meaningful responses.

Mean per-request acceptance length versus forced output sequence length for Qwen3.5-4B drafting for Qwen3.5-122B-A10B on SPEED-Bench 32K low-entropy prompts

Figure 14. Mean per-request acceptance length versus forced output sequence length for Qwen3.5-4B drafting for Qwen3.5-122B-A10B on SPEED-Bench 32K low-entropy prompts. All compressed methods use an 8K KV budget and draft length 11.

RIFT-Ada: Additional Results and Analysis

We provide additional numerical results for RIFT-ada, considering additional datasets, budgets and models. Furthermore, we provide additional information on the calibration and uncertainty of the calibrated threshold. The example refresh schedule and its score trace are shown in Figure 5.

Additional Results

Qwen3.5-4B Figure 16 compares AL–refresh tradeoffs across temperatures, KV budgets, and input lengths using setting-specific controllers, with parameters held fixed across budgets.

Six plots of mean acceptance length versus refreshes per response comparing RIFT-dyn and RIFT-ada

Figure 16. RIFT-ada AL-refresh tradeoffs for Qwen3.5-4B drafting Qwen3.5-122B on SPEED-Bench low-entropy data, with a 4K maximum OSL. Titles specify ISL, temperature, and KV cache budget.

Adaptive Refresh for Nemotron Lightning Self-Speculation

We provide additional analysis of RIFT-ada considering a self-speculative decoding setup with Nemotron3.5-Lightning ([32]). We consider SPEED-Bench 32k low entropy data, with draft length of 11 and 4K OSL. Following model-specific calibration, we refresh when aggregated acceptance falls below a threshold at T=0T = 0, or when either acceptance is low or draft entropy is high at T=1T = 1. Aggregated acceptance uses the last four completed verification steps with exponential smoothing (α=0.4\alpha= 0.4). Additionally, we use minimum and maximum refresh ages of 32 and 768 tokens respectively. As seen in Figure 17, RIFT-ada significantly improves upon RIFT-dyn,

Nemotron3.5-Lightning self-speculation RIFT-ada refresh-AL tradeoff

Figure 17. Nemotron3.5-Lightning self-speculation RIFT-ada refresh - AL tradeoff. SPEED-Bench 32k ISL, low entropy category.

allowing for similar ALs with a fraction of the overall number of refreshes. These additional results further highlight the necessity of adaptive gating for KV recomputation.

Adaptive-Refresh Policies and Calibration

Calibration and policy selection. We organize controller development around three sets: calibration, policy selection, and confirmation. We begin by collecting the signals on the calibration set, from which we determine the threshold search sets. The set of thresholds is determined from the corresponding percentiles which are derived from the signals’ statistics. Next, the policy-selection set compares measured AL-refresh tradeoffs for the various threshold candidates. We calibrate the controller separately for each draft/target-model and temperature setting. For the reported budget sweeps, controller parameters are kept fixed across budgets.

Selected policies. Generation settings change draft entropy, acceptance statistics, and the useful refresh operating range, motivating separate controller calibration. The selected policies therefore differ across models and temperatures, and may be further tailored to a particular generation setting. Acceptance-based rules use exponential weighting.

of 4 completed verification steps with coefficient 0.4. Entropy is averaged over the latest decision stride. Signals are updated after each completed decision stride, before deciding whether to refresh. Query drift compares the current grouped query summary with the summary stored at the last refresh.

The RIFT-ada curve in Figure 6 uses an or-combined rule. Let AtA_t be the four-completed-step exponentially weighted acceptance length (α=0.4\alpha= 0.4), HtH_t the mean draft entropy over the latest decision stride, and DtqD_t^q the mean query drift. Let ArefA_{\mathrm{ref}} and HrefH_{\mathrm{ref}} denote the post-refresh references. We denote the collection of signals with ztz_t. Thus, the signal trigger is given by

π(zt)=(Ht≥2.11)or (At≤2.19)or (Ht−Href≥1.35)or (Aref−At≥4.84)or (Dtq≥τD).\begin{aligned} \pi(z_t) ={}& (H_t \ge2.11) \\ &\text{or } (A_t \le2.19) \\ &\text{or } (H_t - H_{\mathrm{ref}} \ge1.35) \\ &\text{or } (A_{\mathrm{ref}} - A_t \ge4.84) \\ &\text{or } (D_t^q \ge\tau_D). \end{aligned}

To obtain the curve, we keep the other thresholds fixed and sweep τD\tau_D over {0.5,0.43,0.33,0.26,0.22,0.2,0.16}\{0.5, 0.43, 0.33, 0.26, 0.22, 0.2, 0.16\}.

Uncertainty and evaluation scope. We calibrate separate controllers for each temperature, using disjoint calibration, evaluation, and confirmation subsets during development. We compare fitted monotonic-score policies with acceptance/entropy level and deterioration rules, including and/or combinations with query drift. For the Qwen results in Figure 16(c) (T=0T = 0) and Figure 6 (T=1T = 1), the selected policies use level and deterioration signals at T=0T = 0, and an additional query-drift or trigger at T=1T = 1, reducing refreshes at matched AL by 53.5% and 51.1%, respectively. Paired-bootstrap 95% confidence intervals exclude zero for both improvements over periodic refresh, conditional on the selected policies. The final results include the development subsets, so these results describe selected-policy performance rather than an untouched held-out evaluation.

Measuring improvement in the refresh-AL tradeoff. For an adaptive operating point (R,A)(R, A), where RR is the average number of refreshes and AA is AL, we report

ΔA∣R=A−Afix(R),\Delta A|_R = A - A_{\mathrm{fix}}(R),
ΔR∣A=100(1−RRfix(A))%.(1)\Delta R|_A = 100\left(1 - \frac{R}{R_{\mathrm{fix}}(A)}\right)\%. \tag*{(1)}

Here Afix(R)A_{\mathrm{fix}}(R) is periodic refresh’s AL at matched refresh count, and Rfix(A)R_{\mathrm{fix}}(A) is its refresh count at matched AL.

Beyond Speculative Decoding: Long-Context Quality Evaluation

We provide preliminary evidence for RIFT-stat’s performance beyond speculative decoding in a long-OSL setting. We evaluate Qwen3.5-4B on LongBench-v1 and LongBench-v2 [3, 4].

Evaluation protocol. To meet a long OSL setup, we enforce a 4K reasoning phase. Following prefill, the model greedily generates up to 4,096 reasoning tokens. Then, it receives a task-specific final-answer request. Generation continues with the same cache, without re-prefill or reselection. Answer limits are 32-512 tokens for LongBench-v1 and 128 for LongBench-v2. We report the average of task scores. We count parse failures as incorrect (e.g., no valid choice in a multi-option question). We compare SnapKV (with a 64 token SW) and RIFT-stat.

Results. As seen in Table 15 RIFT-stat uniformly outperforms SnapKV at every tested budget. At 8K, it retains 96.4% and 89.9% of full-KV scores on v1 and v2, respectively. These results suggest that recency allocation can also benefit standalone generation. We note that, while promising, these are preliminary results, and a comprehensive evaluation beyond speculative decoding is left to future work.

BenchmarkBudget (Avg. compression)SnapKVRIFT-stat
LongBench-v11K (9.95×)60.776.2 (+15.5)
LongBench-v12K (4.98×)61.781.8 (+20.1)
LongBench-v14K (2.49×)63.489.5 (+26.1)
LongBench-v18K (1.24×)67.296.4 (+29.2)
LongBench-v21K (119.41×)60.361.9 (+1.6)
LongBench-v22K (59.70×)67.770.4 (+2.6)
LongBench-v24K (29.85×)68.375.7 (+7.4)
LongBench-v28K (14.93×)69.389.9 (+20.6)

Table 15. Full-KV score retention (%) for Qwen3.5-4B under the modified LongBench protocol described above. Absolute full-KV scores are 47.607 for LongBench-v1 and 38.569 for LongBench-v2.

RIFT Performance: Task Breakdown

To provide an additional point of view into RIFT’s performance, we break down acceptance length by SPEED-Bench task. We consider Qwen3.5-4B drafting for Qwen3.5-122B and consider data from ISL splits 16K, 32K and 100K. Table 16 presents nine selected categories. It can be seen that the order of performance (SnapKV<SW<RIFT-stat <RIFT-dyn) is maintained across all categories, across various generation modes induces by different tasks

TaskSnapKVSWARIFT-statRIFT-dynRIFT-ada
T=0T = 0
Sorting3.653.964.845.355.61
Creative writing3.904.865.796.296.28
NIAH4.304.865.506.166.18
Code debugging3.765.025.586.676.80
Creative writing3.033.914.184.394.42
Domain QA — Math3.265.645.825.915.91
Domain QA — Chemistry3.756.226.456.476.50
Domain QA — Biology/Medicine3.505.785.986.036.03
Domain QA — Engineering4.396.256.446.556.58
T=1T = 1
Sorting3.183.574.224.584.78
Creative writing3.134.144.855.405.39
NIAH3.664.224.725.355.32
Code debugging3.254.524.865.776.07
Creative writing2.763.553.763.903.93
Domain QA — Math2.835.195.325.395.39
Domain QA — Chemistry3.265.845.956.016.01
Domain QA — Biology/Medicine3.155.345.455.525.53
Domain QA — Engineering3.795.906.006.096.10

Table 16. Mean AL using SPEED-Bench data of ISLs 16K, 32K and 100K, at T=0T = 0 and T=1T = 1. Draft model is Qwen3.5-4B drafting for Qwen3.5-122B. Draft length is 11 and max OSL is 4K.

Tokenizer Adaptation for Cross-Family Drafting

We adopt the following tokenizer-adaptation procedure to initialize a cross-family draft before further draft–target alignment:

  1. Resize. Resize draft input embeddings and the LM-head to the target’s configured vocabulary size, retaining the draft’s hidden dimension.

  2. Copy matching tokens. For each target token, first try a manually role-matched control token (for special tokens). If the token is no a special token, look for an exact match, and if such a match is not found look for a normalized match in the draft vocabulary.

  3. Initialize remaining tokens. Retokenize unmatched text with the draft tokenizer and combine up to four embeddings using geometrically decreasing weights.

  4. Align outputs. Apply the same mapping to an untied output head, preserve tied embeddings, and adopt the target tokenizer’s beginning-of-sequence, end-of-sequence, and padding IDs.

Once the draft’s vocabulary and corresponding embedding matrices are aligned, we perform a knowledge distillation scheme over the Nemotron post-training V2 dataset over 10K steps.

Comparison With Additional KV Compression Policies

We provide additional comparisons with KV compression methods, complementing the results in the main paper. We evaluate on SPEED-Bench with Qwen3.5-4B drafting for Qwen3.5-122B and, in Table 17, Llama3.1-8B drafting for Llama3.1-70B. We consider 4K OSL and a draft length of 11. We consider the following methods:

ModelISLTTStreamingLLMRIFT-statRIFT-dynRIFT-ada
Qwen3.532K05.345.586.086.17
14.734.925.405.49
100K05.605.776.616.73
14.985.075.765.92
Llama3.132K05.875.876.587.08
14.204.414.895.41
100K04.825.325.656.23
13.353.583.703.84

Table 17. Mean AL on SPEED-Bench low-entropy data. Qwen3.5-4B drafts for Qwen3.5-122B and Llama3.1-8B for Llama3.1-70B. All rows use an 8K budget, draft length 11, and a 4K maximum OSL. StreamingLLM retains an 8K window plus four sinks.

  1. StreamingLLM (Xiao et al., [41]), which retains a SW of recent tokens together with four initial attention-sink tokens.

  2. H₂O (Zhang et al., [46]), which updates accumulated attention scores at each decoding step and retains a combination of recent tokens and heavy hitters, permanently evicting other positions.

  3. TriAttention (Mao et al., [30]), which uses calibrated pre-RoPE query statistics to estimate key importance through RoPE-induced trigonometric distance preferences and norm-based scoring. Specifically, we estimate the mean pre-RoPE query vectors from full-prefill statistics and hold them fixed during decod-

ing, and consider a refresh interval similar to RIFT-dyn of 128 tokens.

Tables 17 and 18 show higher AL for RIFT than StreamingLLM across the reported settings. For the other decode-time baselines, RIFT-dyn and RIFT-ada outperform both H2O and TriAttention in every reported setting, while H2O generally outperforms RIFT-stat (Table 18).

TTBudgetStreamingLLMH2_2OTriAttentionRIFT-statRIFT-dynRIFT-ada
02K4.524.684.544.545.145.28
4K4.795.254.975.035.735.82
8K5.345.735.435.626.126.17
12K3.904.063.863.944.474.54
4K4.174.534.274.345.035.10
8K4.735.034.774.995.485.51
02K4.845.355.175.396.116.33
4K5.305.925.645.816.756.91
8K5.606.656.106.407.147.17
12K4.284.474.254.545.165.46
4K4.675.064.694.985.806.09
8K4.985.775.115.556.216.36

Table 18. Mean AL on SPEED-Bench low entropy for Qwen3.5-4B drafting Qwen3.5-122B, with draft length 11 and a 4K maximum OSL. StreamingLLM uses the stated window plus four sinks.

Draft Compressibility Training

Mask Construction

To generate a SnapKV attention mask, we begin with a frozen dense forward pass and capture the post-RoPE queries and keys in every full-attention layer. Let O\mathcal{O} denote the observation window’s queries and G(h)\mathcal{G}(h) the query heads that share KV head hh. We denote the dense post-softmax attention weight from query qq in head gg to key position kk in KV head hh by Ag,h(q,k)A_{g,h}(q,k). The importance of key position kk is

ah(k)=1∣G(h)∣∑g∈G(h)∑q∈OAg,h(q,k).(2)a_h(k) = \frac{1}{|\mathcal{G}(h)|} \sum_{g \in\mathcal{G}(h)} \sum_{q \in\mathcal{O}} A_{g,h}(q,k). \tag*{(2)}

Training Configuration

We consider 10K training steps using teacher-generated rollouts from the Nemotron V2 post-training dataset, using a batch size of 64. The teacher responses and logits were generated at T=1T = 1. We train with a categorical distribution over p∈{0.1,0.3,0.5,0.7,0.9}p \in\{0.1, 0.3, 0.5, 0.7, 0.9\} whose values are {0.05,0.10,0.20,0.25,0.40}\{0.05, 0.10, 0.20, 0.25, 0.40\} respectively. We use a SW ratio ρ=0.25\rho= 0.25 during training. We consider KD temperature TKD=2T_{\mathrm{KD}} = 2, and (λCE,λKD)=(0.25,0.75)(\lambda_{\mathrm{CE}}, \lambda_{\mathrm{KD}}) = (0.25, 0.75). In evaluation, we consider an OSL of 4K and a draft length of 11, drafting on SPEED-Bench’s 32k throughput split, low entropy category. Additional results and ablation studies follow below.

Ablation Studies

We present ablation studies on various components in the draft compressibility training scheme, motivating the design choices in this work. Unless stated otherwise, we train Qwen3.5-4B drafting for Qwen3.5-122B and evaluate ALs on SPEED-Bench 32K ISL split, low entropy category.

Masking Strategy We test the gain of the attention-score-based mask by comparing it with similar training runs that use different masking strategies, such as random or SWA-based masks. As seen in Table 19, the attention-based masking scheme outperforms every random and sliding-window control at temperatures 0 and 1 and at all three considered KV budgets. It is worth noting that using an attention-based mask induces additional complexity to the overall training scheme, as we need to perform an additional dense forward pass.

Training configurationT=0T = 0 2KT=0T = 0 4KT=0T = 0 8KT=1T = 1 2KT=1T = 1 4KT=1T = 1 8K
Random, dynamic pp4.675.155.704.324.715.34
Random, p=0.1p = 0.14.314.765.203.994.374.87
Random, p=0.3p = 0.34.565.045.564.224.625.21
Random, p=0.5p = 0.54.595.095.674.244.665.29
Random, p=0.7p = 0.74.635.125.704.254.665.31
Random, p=0.9p = 0.94.625.125.694.234.645.29
SWA, B=64B = 644.224.474.743.944.144.47
SWA, B=128B = 1284.364.634.934.074.294.65
SWA, B=256B = 2564.494.785.104.184.434.81
SWA, B=512B = 5124.644.975.364.324.605.04
Attention-based mask4.895.375.944.584.985.60

Table 19. AL of the trained draft model (Qwen3.5-4B) under various masking strategies. ’dynamic pp’ refers to the same pp allocation used in the main draft-compressibility results.

Attention mask composition We study the effect of the distribution of the sparsity parameter, which determines the ratio of attention to mask out during training. Specifically, we consider dynamic pp values, in which pp is sampled from a given categorical distribution on a subset of [0,1][0, 1]. All distributions sample p∈{0.1,0.3,0.5,0.7,0.9}p \in\{0.1, 0.3, 0.5, 0.7, 0.9\}. High biased samples with probabilities {0.05,0.10,0.20,0.25,0.40}\{0.05, 0.10, 0.20, 0.25, 0.40\}, “soft” and “sharp” use probabilities {0.10,0.15,0.20,0.25,0.30}\{0.10, 0.15, 0.20, 0.25, 0.30\} and {0.03,0.07,0.15,0.25,0.50}\{0.03, 0.07, 0.15, 0.25, 0.50\}, respectively. Additionally, we test the effect of the training-time SW ratio value ρ\rho. As seen in Table 20, the high-biased pp distribution attains the best results, while the value of ρ\rho does not have a significant effect on performance.

Training configurationRIFT-stat 2KRIFT-stat 4KRIFT-stat 8KRIFT-dyn 2KRIFT-dyn 4KRIFT-dyn 8K
(a) Sliding-window ratio ρ\rho during training
ρ=0.10\rho= 0.104.675.195.805.365.926.28
ρ=0.25\rho= 0.254.695.215.825.395.926.29
ρ=0.50\rho= 0.504.725.225.815.405.936.29
ρ=0.75\rho= 0.754.725.215.795.385.936.30
ρ=0.90\rho= 0.904.725.205.785.355.886.28
(b) Sampling distribution over pkeepp_{\mathrm{keep}}
Soft high-biased4.735.215.815.415.926.28
High-biased4.725.225.815.405.936.29
Sharp high-biased4.705.195.805.385.936.32

Table 20. Training-mask hyperparameter ablations for Qwen3.5-4B drafting Qwen3.5-122B on SPEED-Bench 32K low-entropy. All checkpoints are evaluated with ρ=0.5\rho= 0.5 with T=0T = 0.

Optimization and Parameterization We examine the effect of the fine-tune hyperparameters on the training scheme. Specifically, we are examine the effect of the number of trainable parameters on the resulting model’s performance and the learning rate behavior. Table 21, compares LoRA rank and learning-rate schedule for the same Qwen pair, averaging over 2K, 4K, and 8K budgets. As expected, full model fine tune results in slightly higher ALs than LoRA. However, the computational overhead of a full FT is significant. To that end, when prioritizing performance and the resulting task does not involve self-speculation, we suggest full-FT, while LoRA is preferable when we are interested in a lightweight modification.

Training configurationT=0T = 0 RIFT-statT=0T = 0 RIFT-dynT=1T = 1 RIFT-statT=1T = 1 RIFT-dyn
LoRA-32, constant 10−510^{-5}5.30 ± 0.135.85 ± 0.134.93 ± 0.115.46 ± 0.12
LoRA-128, decay from 10−510^{-5}5.29 ± 0.135.85 ± 0.134.93 ± 0.125.45 ± 0.13
Full FT, constant 5×10−65 \times10^{-6}5.29 ± 0.135.82 ± 0.134.91 ± 0.125.43 ± 0.13
Full FT, decay from 10−510^{-5}5.30 ± 0.135.85 ± 0.134.94 ± 0.125.46 ± 0.13

Table 21. Training hyperparameter comparison. Results are reported in average ALs with 95% confidence intervals.

Additional Draft Compressibility Training Results

We present additional results on draft compressibility, demonstrating its effect in additional settings.

Additional evaluations on long context Table 22 extends the Qwen3.5-4B draft (Qwen3.5-122B target) training result to long contexts by evaluating on SPEED-Bench 100k ISL split. It can be seen that the gains are also evident in longer context drafting, with higher gains in non-greedy decoding settings.

TTRIFT-stat 2KRIFT-stat 4KRIFT-stat 8KRIFT-dyn 2KRIFT-dyn 4KRIFT-dyn 8KRIFT-ada 2KRIFT-ada 4KRIFT-ada 8K
0+4.05%+2.24%+0.97%+4.90%+2.54%+0.97%+2.57%+0.37%−0.34%
1+12.50%+9.87%+7.03%+12.04%+7.95%+5.12%+9.80%+5.49%+3.79%

Table 22. Relative AL improvement from compressibility training for Qwen3.5-4B drafting Qwen3.5-122B on SPEED-Bench 100K.

Effect of compressibility training to SnapKV. We demonstrate the effect of the proposed draft compressibility training on the baseline SnapKV method, considering Qwen3.5-4B drafting for Qwen3.5-122B. As seen in Table 23, SnapKV benefits dramatically from the proposed compressibility training scheme. However, we attribute the magnitude of that gain to the initial method’s AL, which was significantly lower than RIFT-stat (As reported in the main paper).

TT2K4K8K
0+15.57%+13.50%+11.52%
1+26.54%+23.80%+21.07%

Table 23. Relative AL improvement from draft compressibility training for SnapKV: Qwen3.5-4B draft (Qwen3.5-122B target), 100K low-entropy split.

Additional draft training We present the effect of draft compressibility training on additional draft models. Table 24 presents the effect on Qwen3.5-0.8B (drafting for Qwen3.5-122B). Gains are modest at T=0T = 0 but reach up to 10% at T=1T = 1. The corresponding untrained policy comparison appears in Table 8. Next, we consider a larger draft model. Specifically, we train Nemotron3.5-Lightning with a Nemotron-3-Ultra teacher. The training recipe follows the one described in § 5, with a single slight change: Following a empirical sweep, we replace the KL loss term with a skewed KL loss, which replaces the teacher’s logits with a linear interpolation of the student and teacher’s logits. The underlying potential reason for the performance improvement of skewed KL is that skewed KL compensates for large model distribution discrepancies, which may follow due to the large parameter count ratio between the draft and the target. As shown in Table 25, compressibility training improves Lightning’s AL by up to approximately 16.7% (3.72 to 4.34 for RIFT-dyn at 2K and T=1T = 1). At 8K and T=0T = 0, its trained RIFT-dyn AL of 5.27 corresponds to 97.8% of the original draft’s full-KV AL (5.39), or 91.7% of the trained draft’s full-KV AL (5.75).

ISLEntropyTTFull KVSWRIFT-statRIFT-dyn
32KLow0+0.55%+0.90%+0.84%+0.81%
32KLow1+7.77%+6.43%+9.71%+8.87%
32KHigh0+0.82%+0.80%+0.98%+0.84%
32KHigh1+8.95%+7.30%+9.33%+9.32%
100KLow0+0.00%+1.73%+0.90%+0.77%
100KLow1+5.90%+5.70%+8.24%+7.68%
100KHigh0+0.65%−0.25%+0.71%+0.52%

Table 24. Relative AL improvement from training the Qwen3.5-0.8B draft, measured on SPEED-Bench at an 8K cache budget against the corresponding untrained draft.

Inference methodBudgetT=0T = 0T=1T = 1
BaseTrainedBaseTrained
Full KV–5.395.754.865.49
Sliding window2K3.623.873.213.73
Sliding window4K3.924.193.494.02
Sliding window8K4.314.603.864.43
RIFT-stat2K3.754.043.393.92
RIFT-stat4K3.994.293.604.16
RIFT-stat8K4.354.683.934.52
RIFT-dyn2K4.164.493.724.34
RIFT-dyn4K4.574.934.084.72
RIFT-dyn8K4.995.274.455.04

Table 25. Average accepted length before and after compressibility training of Nemotron3.5-Lightning drafting Nemotron-3-Ultra on SPEED-Bench 32K low-entropy. The trained checkpoint uses skewed KL with α=0.5\alpha= 0.5. Static and Dynamic SnapKV use a sliding-window ratio of ρ=0.5\rho= 0.5.

Additional RIFT-ada Results on Draft Compressibility

We provide additional results on the effect on draft compressibility training on RIFT-ada. We consider Qwen3.5-4B and Qwen3.5-0.8B, drafting for Qwen3.5-122B. As seen in Table 26 training improves AL in every reported setting.

Draft → targetISLEntropyTT2K4K8K
Qwen3.5-4B → Qwen3.5-122B32KLow0+3.88%+2.38%+1.69%
Qwen3.5-4B → Qwen3.5-122B32KLow1+10.03%+7.88%+6.70%

Table 26. Relative AL improvement from compressibility training on RIFT-ada.

Extended Related Work

We review the work most directly connected to frozen prefill selection, decode-time cache management, and compressed-cache drafting.

Prefill-based KV cache compression. A major line of work focuses on a single compression of the prompt KV cache, right before decode takes place. SnapKV estimates prefill-token importance from attention scores produced by an observation window, often at the end of prefill [24]. PyramidKV and AdaKV build SnapKV’s selection paradigm while heterogeneously allocating the cache budget across layers and heads, respectively [6, 14]. Other static KV cache selectors change the importance signal. For example, CriticalKV scores the output perturbation caused by eviction [15], Expected Attention models attention under a distribution of future queries [11] and CurDKV uses query-free key and value-leverage scores [37]. DefensiveKV instead challenges the usual mean aggregation of importance over an observation window and introduces a worst-case-risk-aware aggregation rule. Nevertheless, its eviction is still performed during prefill, leaving the retained prompt set fixed during decode [13]. ChunkKV changes the selection unit from individual tokens to contiguous chunks [26]. Collectively, these methods primarily optimize which entries survive compression. Our work studies a common failure mode: even a strong prefill decision can become stale as a long generated prefix changes what the model needs, and we do that through the lenses of speculative decoding. In the context of compressibility training, KV-CAT [16] demonstrates that training under KV sparsification can improve the post-hoc compressibility of pretrained language models, enabling additional gains through training. They consider an additional, trainable masker module, based on which sparse attention masks are built. In this work we directly consider specific KV scoring methods.

Decode-time KV compression for long generation. Decode-time strategies continually adapt the working cache. For example, ScissorHands exploits persistence of token importance, StreamingLLM retains attention sinks and recent tokens, and H₂O allocates half of its cache budget to recent tokens and half to heavy-hitters identified by accumulated attention scores [28, 41, 46]. R-KV utilizes reasoning trace information sparsity [5]. QUEST proposes lightweight page-level retrieval through aggregated metrics [39]. Its page summaries are complementary to our study of when and from which tempo-

ral region a drafter should refresh its active cache. LongFlow derives a low-overhead importance signal from the current query for continuous eviction, whereas KARA restricts decode-time scoring to a recent window and expands selected tokens into flexible chunks [17] (Su et al. 2026). Compression-aware speculative decoding. Tri-Force uses a dynamically retrieved target cache as an intermediate drafter, refreshed periodically or by acceptance gating (Sun et al., 2024). MagicDec studies sparse-KV self-speculation in large concurrencies, while SpecExtend and SparseSpec reuse verification attention to select KVs [36, 7, 47]. LongSpec trains a constant-context drafter and Windowed-MTP and Attention Drift study SWA-based MTP drafting and stepwise attention changes in tail-augmented drafting [42, 40, 12]. RIFT complements these approaches by operating directly on external or self-speculative draft caches and studying recency allocation and draft-side refresh signals under a fixed active KV cache budget.

References

  1. [1]Talor Abramovich, Maor Ashkenazi, Carl (Izzy) Putterman, Benjamin Chislett, Tiyasa Mitra, Bita Darvish Rouhani, Ran Zilberstein, and Yonatan Geifman. 2026. SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding.
  2. [2]Kyle Aubrey and Farshad Ghodsian. 2026. Inside NVIDIA Groq 3 LPX: The Low-Latency Inference Accelerator for the NVIDIA Vera Rubin Platform. NVIDIA Technical Blog. Published March 16, 2026. Accessed: 2026-09-09.
  3. [3]Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2024. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119–3137.arxiv.org/abs/2308.14508
  4. [4]Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazhen Xu, Lei Hou, Yuxiao Dong, et al. 2025. LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-Context Multitasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3639–3664.arxiv.org/abs/2412.15204
  5. [5]Zefan Cai, Wen Xiao, Hanshi Sun, Cheng Luo, Yikai Zhang, Ke Wan, Yucheng Li, Yeyang Zhou, Li-Wen Chang, Jiuxiang Gu, Zhen Dong, Anima Anandkumar, Abdelkadir Asi, and Junjie Hu. 2025. R-KV: Redundancy-aware KV Cache Compression for Reasoning Models. arXiv preprint arXiv:2505.24133.arxiv.org/abs/2505.24133
  6. [6]Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, and Wen Xiao. 2024. PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling. arXiv preprint arXiv:2406.02069.arxiv.org/abs/2406.02069
  7. [7]Jungyoun Cha, Hyuniong Kim, and Sungzoon Cho. 2025. SpecExtend: A Drop-in Enhancement for Speculative Decoding of Long Sequences. arXiv preprint arXiv:2505.20776.arxiv.org/abs/2505.20776
  8. [8]Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating Large Language Model Decoding with Speculative Sampling. arXiv preprint arXiv:2302.01318.arxiv.org/abs/2302.01318
  9. [9]Jian Chen, Yesheng Liang, and Zhijian Liu. 2026. DFlash: Block Diffusion for Flash Speculative Decoding. arXiv preprint arXiv:2602.60036.
  10. [10]Tri Dao and Albert Gu. 2024. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. arXiv preprint arXiv:2405.21060.arxiv.org/abs/2405.21060
  11. [11]Alessio Devoto, Maximilian Jeblick, and Simon Jégou. 2025. Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution. arXiv preprint arXiv:2510.00636.arxiv.org/abs/2510.00636
  12. [12]Doğaç Eldenk, Payal Mohapatra, Yigitcan Comlek, Kaan Oktay, Hongyang Zhang, and Stephen Xia. 2026. Attention Drift: What Autoregressive Speculative Decoding Models Learn. arXiv preprint arXiv:2605.09992.arxiv.org/abs/2605.09992
  13. [13]Yuan Feng, Haoyu Guo, JunLin Lv, S. Kevin Zhou, and Xike Xie. 2025a. Taming the Fragility of KV Cache Eviction in LLM Inference. arXiv preprint arXiv:2510.13334.arxiv.org/abs/2510.13334
  14. [14]Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S. Kevin Zhou. 2025b. Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference. In Advances in Neural Information Processing Systems.arxiv.org/abs/2407.11550
  15. [15]Yuan Feng, Junlin Lv, Haoyu Guo, Yukun Cao, S. Kevin Zhou, and Xike Xie. 2026. CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective. In International Conference on Machine Learning.arxiv.org/abs/2502.03805
  16. [16]Yoav Gelberg, Yam Eitan, Michael Bronstein, Yarin Gal, and Haggai Maron. 2026. Training Transformers for KV Cache Compressibility. arXiv preprint arXiv:2605.05971.arxiv.org/abs/2605.05971
  17. [17]Shen Han, Yuyang Wu, Junpu Yu, and Oxlandr Isayev. 2026. KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression. arXiv preprint arXiv:2607.01237.arxiv.org/abs/2607.01237
  18. [18]Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun S Shao, Kurt Keutzer, and Amir Gholami. 2024. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. Advances in Neural Information Processing Systems, 37:1270–1303.arxiv.org/abs/2401.18079
  19. [19]Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. RULER: What’s the Real Context Size of Your Long-Context Language Models? arXiv preprint arXiv:2404.06654.arxiv.org/abs/2404.06654
  20. [20]Eunjeong Kim, Yeong Jun Jeon, and Myeonggyun Han. 2026. MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices. In Proceedings of the ACM SIG-PLAN/SIGBED International Conference on Languages, Compilers, and Tools for Embedded Systems, pages 180–192.DOI
  21. [21]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Xing Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.arxiv.org/abs/2309.06180
  22. [22]Hannah Laus, Claudio Mayrink Verdun, Hao Wang, Flavio du Pin Calmon, and Felix Krahmer. 2026. KV Cache Compression Through the Lens of Transform Coding. arXiv preprint arXiv:2608.14191.arxiv.org/abs/2608.14191
  23. [23]Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast Inference from Transformers via Speculative Decoding. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19274–19286. PMLR.arxiv.org/abs/2211.17192
  24. [24]Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. SnapKV: LLM Knows What You Are Looking for Before Generation. arXiv preprint arXiv:2404.14469.arxiv.org/abs/2404.14469
  25. [25]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2025. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. Advances in Neural Information Processing Systems, 38:136737–136756.arxiv.org/abs/2503.01840
  26. [26]Xiang Liu, Zhenheng Tang, Peijie Dong, Zeyu Li, Yue Liu, Bo Li, Xuming Hu, and Xiaowen Chu. 2025.
  27. [28]Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2023. ScissorHands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time. arXiv preprint arXiv:2305.17118.arxiv.org/abs/2305.17118
  28. [29]Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. arXiv preprint arXiv:2402.02750.DOI
  29. [30]Weian Mao, Xi Lin, Wei Huang, Yuxin Xie, Tianfu Fu, Bohan Zhuang, Song Han, and Yukang Chen. 2026. TriAttention: Efficient Long Reasoning with Trigonometric KV Compression. arXiv preprint arXiv:2604.04921.arxiv.org/abs/2604.04921
  30. [31]NVIDIA. 2026a. Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv preprint arXiv:2604.12374.
  31. [32]NVIDIA. 2026b. Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI. Technical report, NVIDIA.
  32. [33]Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. 2021. Show Your Work: Scratchpads for Intermediate Computation with Language Models. arXiv preprint arXiv:2112.00114.arxiv.org/abs/2112.00114
  33. [34]Jinwoo Park, Seunggeun Cho, and Dongsu Han. 2025. SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMs. arXiv preprint arXiv:2505.17052.arxiv.org/abs/2505.17052
  34. [35]Qwen Team. 2026. Qwen3.5-122B-A10B. Hugging Face model card.
  35. [36]Ranojoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisht Tiwari, Ruihang Lai, Jinyuan Shi, Jan En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen. 2025. MagiCDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding. In International Conference on Learning Representations.arxiv.org/abs/2408.11049
  36. [37]Ayan Sengupta, Siddhant Chaudhary, and Tanmoy Chakraborty. 2025. Value-Guided KV Compression for LLMs via Approximated CUR Decomposition. arXiv preprint arXiv:2509.15038.arxiv.org/abs/2509.15038
  37. [39]Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 47901–47911.arxiv.org/abs/2406.10774
  38. [40]Alagappan Valliappan. 2026. Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context. arXiv preprint arXiv:2607.21535.arxiv.org/abs/2607.21535
  39. [41]Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient Streaming Language Models with Attention Sinks. In International Conference on Learning Representations.arxiv.org/abs/2309.17453
  40. [42]Penghui Yang, Cunxiao Du, Fengzhou Zhang, Haonan Wang, Tianyu Pang, Chao Du, and Bo An. 2026. LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1826–1844.
  41. [43]Songlin Yang, Jan Kautz, and Ali Hatamizadeh. 2024. Gated Delta Networks: Improving Mamba2 with Delta Rule. arXiv preprint arXiv:2412.06464.arxiv.org/abs/2412.06464
  42. [44]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations.arxiv.org/abs/2210.03629
  43. [45]Yikang Yue, Yuqi Xue, and Jian Huang. 2026. Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attention. arXiv preprint arXiv:2602.07223.arxiv.org/abs/2602.07223
  44. [46]Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. H₂O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. arXiv preprint arXiv:2306.14048.arxiv.org/abs/2306.14048
  45. [47]Yilong Zhao, Jiaming Tang, Kan Zhu, Zihao Ye, Chi-Chih Chang, Chaofan Lin, Jongseok Park, Guangxuan Xiao, Mohamed S. Abdelfattah, Mingyu Gao, Baris Kasikci, Song Han, and Ion Stoica. 2025. Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding. arXiv preprint arXiv:2512.01278.arxiv.org/abs/2512.01278

Paper details

vlnc:W_bjhedjt6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents