WorksComputer Science
Where Should a KV-Cache Repair Budget Go? Distractor Structure and Contextual Compatibility in Non-Prefix Reuse
Abstract
Non-prefix KV-cache reuse encodes retrieved chunks independently and concatenates their caches. To find where a small repair budget should be spent, we vary the chunks around the answer-bearing one (the target) across three 7B-8B models and four datasets of dialogue memory, open-domain QA and API documentation. Reuse loses 19-100% of the context-dependent questions that full prefill answers, and it loses more when the other chunks are of the same kind as the target. Because the recoverable damage is spread over these other chunks, restoring a few of them recovers little, and the budget is better spent on the target. This works through contextual compatibility: recomputing the target against the reused cache beats inserting its full-prefill KV in all seven settings tested, although the recomputed KV stays far from the full-prefill KV. Question attention over reused caches, however, selects the target in at most 18.8% of development questions. A retriever finds it more often, and recomputing its top chunk after moving it next to the question significantly beats InfoFlow-style reordering at the same 5% budget in six of nine open-domain QA and API-documentation settings. On dialogue memory, though, recomputing chunks never significantly beats the token-level selector chosen for each setting.
Introduction
Retrieval-augmented generation, long-term conversational memory and tool-use agents build each prompt from retrieved units such as passages, dialogue sessions or API documents. Because the retrieved combination changes from request to request, prefix caching rarely applies. Non-prefix KV-cache reuse instead encodes each chunk once, stores its keys and values, and concatenates the stored caches at query time after assigning or correcting positions ([9, 21, 22, 35 c]). This saves the prefill over the retrieved context, but no chunk is ever encoded in the presence of the others (Figure 1). Repair methods either select a small fraction of tokens and recompute them online ([37, 11, 31, 30]), which we call token-level selectors, or encode chunks after a fixed surrogate prefix (SurroKV; [1]). Most of them choose scattered tokens by KV deviation, by position or by the question’s attention over the reused cache, and most use full prefill either as the reference that a repair should approach or to explain what a repair restores (Appendix Table 9).

Figure 1. Reuse cuts each chunk off from the chunks before it, and similar chunks then mislead the model. (a) Full prefill encodes each chunk after the earlier ones, whereas non-prefix reuse encodes each chunk alone and joins the caches, so only the question and the answer attend across chunks. (b) An NQ-Open question answered by Qwen3-8B with 30 passages (actual outputs and passages, both shortened). (c) Where a small repair budget goes. The inset shows that, when the target is unknown, the retriever’s top chunk is moved last (§ 3–§ 5).
To find out where a small repair budget should be spent, we trace which parts of a retrieved context account for the accuracy that reuse loses. A retriever returns the chunks most similar to the query, and these chunks are often similar to one another: several passages about the same entity, sessions from the same conversation, or documentation for APIs of the same kind. In Figure 1, Qwen3-8B answers correctly from the full-prefill cache, but when the same passages are reused it names the actress from a similar passage. For each question we fix the answer-bearing chunk (the target) and its position, and we draw the other chunks (the distractors) from off-domain prose, from unrelated chunks of the same corpus, or from retrieved chunks similar to the target (§ 2). We evaluate Qwen3-8B, Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.2 on dialogue memory (LoCoMo), open-domain QA (NQ-Open and TriviaQA) and API documentation (Gorilla), and we call each model–dataset pair a setting. In each setting we use the questions that full prefill answers and that require the retrieved context ().
The loss grows as the other chunks become more like the target. Position-corrected reuse loses 19–100% of these questions. Part of the loss remains even when the distractors are off-domain prose. Unrelated chunks of the same kind add to the loss significantly in nine of twelve settings, and on open-domain QA, passages similar to the target reduce accuracy further in all six settings, significantly in five (§ 3). A model that Ma et al. [22] fine-tune to encode documents as independent blocks still loses 60% of these questions on NQ-Open, and most of this loss comes from the similar chunks (Appendix § 11).
The damage is spread over the distractors, and a small budget does not undo it. Restoring the full-prefill caches of all distractors recovers more than restoring the target’s cache, in all twelve settings and significantly in eleven. By contrast, restoring a single distractor, the most similar or a random one in each setting, changes retention significantly in only two of these 24 cases. The effects of the distractors add up. In a simple model, each reused distractor adds a weight to the odds of losing the answer. Calibrated only at its two measured end points, with no distractor restored and with every distractor restored, this model predicts how retention grows as random distractors are restored, and its prediction beats a linear one in five of the six open-domain QA settings (§ 3.2). Restoring the distractors’ caches needs the full prefill that reuse avoids, and a small budget spent on a few similar chunks does worse than the same budget spent on selected tokens (§ 5).
Spending the budget on the target works through contextual compatibility. When the question is processed, it attends to the target together with the reused KV of the other chunks.
Recomputing the target against the reused cache encodes it in that reused context. This recomputation beats inserting the target’s full-prefill KV in all seven settings tested (four after Holm’s correction), although the recomputed KV stays at a median relative distance of 0.23–0.30 from the full-prefill KV. Recomputing only the known target chunk, which makes up a median of 2.5–3.3% of the context, beats the token-level selector chosen for each setting with a 5% budget in all nine open-domain QA and API-documentation settings, by 0.10 to 0.49.
Question attention rarely finds the target; a retriever finds it more often. In deployment the target is unknown. Over reused caches, question attention selects the target in at most 18.8% of development questions, and the question’s summed attention selects the last chunk before the question in 58–100% of them (§ 4). Because a recomputed chunk attends only to the chunks before it, we move the retriever’s top-ranked chunk to the position just before the question and recompute it there. At the same 5% budget, this significantly beats InfoFlow-style reordering (Teng et al. [30]) in six of the nine open-domain QA and API-documentation settings (five after Holm’s correction). On the same reordered prompt, recomputing the moved chunk is never significantly worse than recomputing selected tokens, and it is significantly better in four settings (§ 5). These gains are bounded by retrieval. On questions where the retriever’s top chunk is not the target, recomputing the moved chunk does worse than recomputing selected tokens on the same prompt. Moving the top three chunks extends the gain to some of these questions, and with a 10% budget its gain over moving only the top chunk matches a prediction from the retriever’s hit rate in seven of the nine settings. With this budget, however, selected tokens recomputed on the same prompt remain competitive with the three chunks, which are significantly better in only three settings and significantly worse in two. On dialogue memory, chunk-level recomputation brings the 7B–8B models no significant gain over the token-level selector chosen for each setting. Qwen3-14B and Qwen3-32B each reproduce all five results that we test for replication (§ 5 and Table 2).
| Dataset | Model | Loss | Same | Sim. | Target | Chunk moved |
| LoCoMo | 14B | 0.36 | +0.20 | +0.03 | +0.14 | – |
| 32B | 0.46 | +0.16 | +0.18 | +0.07 | – | |
| NQ-Open | 14B | 0.79 | +0.17 | +0.08 | +0.22 | +0.34 |
| 32B | 0.78 | +0.37 | +0.08 | +0.38 | +0.35 | |
| Gorilla | 14B | 0.90 | +0.58 | −0.10† | +0.45 | +0.38 |
| 32B | 0.93 | +0.90 | −0.07† | +0.43 | +0.30 | |
| TriviaQA | 14B | 0.77 | +0.32 | +0.20 | +0.20 | +0.24 |
| 32B | 0.72 | +0.24 | +0.25 | +0.25 | +0.22 |
Table 2. Qwen3-14B and Qwen3-32B on (67–157 questions per setting): the five results tested for replication. Loss is , and Same and Sim. are the same-category and similarity terms of Equation (1). Target and Chunk moved are the gains over the token-level selector (ProphetKV) at 5% when the budget recomputes the target or the retriever’s top chunk moved to the last position. Bold: the 95% interval excludes zero. †Retention under RAND is zero, so the term cannot be positive and is not set in bold. Appendix Table 32 gives the intervals.
Setup
Datasets. We use four datasets in which the answer is contained in one retrieved unit (the target chunk); Appendix § 8 describes how they are built and gives the prompts.
LoCoMo (dialogue memory; Maharana et al., [23]): each chunk is one conversation session, with 13–16 sessions per prompt (about 12K tokens), and we use the questions whose evidence lies in a single session.
NQ-Open (open-domain QA; Kwiatkowski et al., [18]; Liu et al., [20]): the multi-document version of Liu et al. [20], with one gold passage among up to 30 Wikipedia passages (a median of about 4.6K tokens), scored by best-subspan exact match.
TriviaQA (open-domain QA; Joshi et al., [15]): one gold passage together with 29 of the hard negatives released with DPR (Karpukhin et al., [17]), also scored by best-subspan exact match.
Gorilla (API documentation; Patil et al., [26]): HuggingFace APIBench requests, each with 31 API documents (a median of about 6.5K tokens), one of which documents the requested API.
Each dataset has 70 development questions and a disjoint test set (Appendix § 8). We evaluate Qwen3-8B (Yang et al., [34]) with thinking disabled, Llama-3.1-8B-Instruct (Grattafiori et al., [10]) and Mistral-7B-Instruct-v0.2 (Jiang et al., [14]) in bfloat16 with greedy decoding.
Distractor sources. For each question, the target chunk and its position in the prompt are fixed. The other chunks come from one of three sources.
SIM, retrieved chunks similar to the target: other sessions of the same conversation (LoCoMo), the retrieved hard negatives of the original datasets (NQ-Open, TriviaQA) and, for Gorilla, the API documents closest to the request and its reference solution under BGE-M3 (Chen et al., [6]). A distractor list built from the request alone (SIM-q) gives the same near-zero retention under reuse (Appendix § 8), and the retriever used in § 4 and § 5 is likewise given only the request.
RAND, unrelated chunks of the same kind: sessions of other conversations, passages retrieved for other questions, and API documents of other task categories.
OTHER, off-domain prose from Thoreau’s Walden, cut into pieces of the same lengths as the chunks they replace.
Conditions. Full prefill encodes the whole prompt in one pass. Position-corrected reuse encodes every chunk alone after the shared instruction prefix, concatenates the caches with corrected rotary positions, and then appends the question. The no-context baseline omits all chunks. We write for the prefix of tokens, for the chunks in prompt order, for the first position of and for . In full prefill, a token of attends to , and the earlier tokens of . Under reuse, it attends only to and the earlier tokens of . The chunk is encoded starting at position , and its keys are then rotated by .
Evaluation set and decomposition. Following the meaningful subset of Cestola et al. [5], we evaluate each setting on its context-dependent subset : the questions that the no-context baseline answers incorrectly and that full prefill answers correctly under SIM, RAND and OTHER (Appendix § 9 lists how our subset differs from theirs). Full-prefill accuracy on is therefore 1 by construction, and the accuracy of position-corrected reuse on is its retention . The loss under SIM splits into three terms:
The universal term is the loss that remains when the distractors are off-domain prose. The same-category term is the extra loss from chunks of the same kind, and the similarity term is the further change when those chunks are also similar to the target. The terms are differences between ordered interventions, and they need not correspond to independent mechanisms, because the three sources can differ in more than one property. Intervals are 95% paired bootstrap intervals with 2,000 resamples of questions (of conversations for LoCoMo), and we call a difference significant when its interval excludes zero.
Anatomy of the Reuse Loss
Size and decomposition
The loss of position-corrected reuse under SIM ranges from 0.19 to 1.00, and its interval excludes zero in all twelve settings (Figure 2 and Appendix Table 10; Appendix Table 3 reports accuracy on all evaluated questions). The universal term, the loss that remains when every distractor is off-domain prose, ranges from 0.11 to 0.92.
Figure 2. Retention of position-corrected reuse on under each distractor source (top), and the three terms of Equation (1) (bottom): universal , same-category and similarity . Bars are 95% paired bootstrap intervals, with questions resampled (conversations for LoCoMo). In the bottom row, open markers show two cases: intervals that include zero, and the similarity terms on Gorilla, which cannot be positive because retention under RAND is zero. Appendix Table 10 gives the values.
Unrelated chunks of the same kind usually cost more than off-domain prose. The same-category term is significantly positive in nine of the twelve settings and never significantly negative. It is largest for Qwen3-8B and Llama-3.1-8B on Gorilla (0.82 and 0.62). There, retention under RAND is 0.00 for every model, while retention under OTHER reaches 0.82.
Whether chunks similar to the target add to the loss depends on the domain. On open-domain QA, the similarity term is positive in all six settings and significant in five. In dialogue memory, only Qwen3-8B has a significant similarity term (+0.16, [+0.08, +0.24]).
Competing answers. On open-domain QA, the wrong answers point to the similar passages. Under SIM, the answer produced with reuse names an entity (a capitalised name or a number) that occurs in some distractor passage but not in the target passage on 40–63% of the questions, compared with 8–40% under RAND. The paired difference is significant in all six settings (Appendix Table 16; Appendix § 13 gives abstentions, examples and the corresponding counts of distractor APIs on Gorilla). These counts suggest that the loss beyond the universal term grows with the number of competing answers that the distractors offer. Replacing the competing entities of the answer’s type with generic descriptors such as “a person” recovers only part of the loss (Appendix Table 17).
Target versus distractor replacement
To locate the recoverable damage, we take the position-corrected cache under SIM and replace the KV of selected chunks with their KV from the full-prefill cache, that is, the KV those chunks would have had if they had been encoded in context (Appendix Tables 13 and 14). Replacing the distractors recovers more than replacing the target. With all of them replaced, retention is higher than with only the target replaced in all twelve settings, significantly so in eleven, and 41–73% of the loss is recovered on NQ-Open, TriviaQA and Gorilla. Compared with no replacement, replacing every distractor raises retention significantly in ten settings and lowers it significantly in one, while target replacement helps significantly in six and hurts significantly in one. By contrast, replacing a single distractor, the most similar one or a random one, changes retention significantly in only two of 24 cases (Appendix § 13).
Distractors add to the odds of losing the answer. A simple model fits both the gain from restoring every distractor and the small change from restoring a single one. In this model, each reused distractor adds a weight to the odds of failure,
where is the target, restoring the full-prefill KV of removes , and is what remains with every distractor restored. If a random set of of the distractors carries of the total weight, restoring it gives , with no parameter beyond the two measured end points and . We test this prediction against two alternatives, linear in and linear in the probit of , by restoring random distractors on open-domain QA. At , where the predictions differ most, the prediction of the odds model lies within the 95% interval of the measured retention and the linear prediction lies outside it in five of the six settings; in the sixth setting, all three predictions lie within the interval (Appendix Table 7 and Figure 4).
Figure 4. Retention after restoring the full-prefill KV of random distractors under SIM. Filled markers show the measured retention with 95% paired bootstrap intervals. Open markers show the two end points, and (every distractor restored), taken from the main experiments. The lines show the three predictions made from these two end points: additive odds of failure (Equation (2)), linear in the probit of , and linear in . The vertical line marks , where the predictions differ most and are tested (Table 7).
The target’s KV depends on the chunks before it. Target replacement inserts KV that was computed against the full-prefill KV of the earlier chunks, but the question then attends to it together with the reused KV of those chunks. We call a chunk’s KV contextually compatible with a cache if it was computed against the KV of the earlier chunks in that cache. On 270 questions in seven settings, inserting the target’s full-prefill KV raises retention by +0.11 [+0.06, +0.17] when the earlier chunks stay reused and by +0.25 [+0.20, +0.31] when their full-prefill KV is restored as well, so the interaction is +0.14 [+0.08, +0.20]. With the earlier chunks reused, the inserted KV helps significantly only in the three Mistral-7B settings (Appendix § 13).
Recomputing the target against the reused cache makes its KV compatible. The recomputed target significantly beats the inserted full-prefill KV in all seven settings, by 0.13 to 0.38, and four of these differences remain significant after Holm’s correction (Appendix Tables 33 and 34). This holds although the recomputed KV stays at a median relative distance of 0.23–0.30 from the full-prefill KV, compared with 0.34–0.45 for the unrepaired KV.
Where the deviation starts. Under reuse, every chunk except the first, whether a distractor or the target, differs from full prefill in two ways: it cannot attend to the chunks before it, and it is encoded at positions closer to the prefix. In the first layer, where every token enters with its embedding, these two differences account exactly for the change in each attention head.
Proposition 1. Fix a head and a token of , and suppose that and every token it attends to under either encoding have the same hidden states under both. Let be the attention of on under full prefill, and and the head’s outputs renormalised over and the tokens of up to , and over , at the full-prefill positions. Then
and the second term vanishes when . Both terms vanish for , whose KV is the same under the two encodings in every layer.
The proof splits the softmax normaliser (Appendix § 10). In the first layer of the three models, averaged over heads and the tokens of a chunk () rises with the chunk’s position in all nine open-domain QA and API-documentation settings (Appendix Table 6). A chunk thus misses more of its context the more chunks precede it.
Question Attention over Reused Caches
Several online repair methods choose what to recompute from the attention of the question tokens over the reused cache ([31, 1]). If similar chunks compete with the target, that attention may be a poor guide to the target-bearing chunk. We test three detectors of this answer-bearing chunk on development questions that full-prefill answers correctly (Appendix Table 12). The first, , sums the question’s attention over each chunk. The second, , averages this attention over the upper half of the layers after dropping the first 16 tokens of each chunk, to discount the attention that independently encoded chunks place on their first tokens. The third, , is the cosine similarity between the BGE-M3 embeddings of the question and of each chunk, computed without the model. For Gorilla, the question is the user request alone (Appendix § 8).
Over reused caches, the question’s summed attention peaks most often on the chunk just before the question. Detector selects that chunk in 58–100% of questions, and in every question in seven of the twelve settings. The target is selected in at most 12.5% of questions by and in at most 18.8% by , while chance is 3.2–6.4%. Over full-prefill caches, the same detectors do better in 20 of the 24 combinations of detector and setting, with scores of up to 89.3%. The retriever finds the target in 50–64% of open-domain QA questions, 50–52% of dialogue-memory questions and 36–56% of Gorilla questions.
Implications for Budgeted Repair
Spending a small budget on chunks. Restoring the distractors costs almost a full prefill (§ 3.2), and a small budget spent on a few of them does worse than the token-level selector. With a 10% budget, recomputing the four chunks most similar to the target, without the target itself, is significantly worse than the selector in eleven of the twelve settings (Appendix Table 22). We therefore spend a small budget on the chunk that holds the answer, which question attention rarely finds (§ 4). For each setting, we use the development questions to choose a token-level rule from CacheBlend, EPIC, ProphetKV and question attention ([37, 11, 31]). The token-level selector applies this rule to all chunks. Each chunk-level allocation spends the same budget on its chosen chunks and ranks the tokens within them by the same rule (Appendix § 9 describes where this selector differs from the published methods). Target recomputes the known target chunk, an oracle reference that holds 2.5–3.3% of the context on open-domain QA and API documentation; Target+4 also recomputes the four chunks most similar to the target, and Retrieved and Retrieved+4 use the retriever’s top chunk in place of the target. Table 1 gives the paired gains, and Appendix Tables 20–22 give intervals and controls.
| Dataset | Model | 5% budget Retention Token | 5% budget Gain over Token Target | 5% budget Gain over Token Target +4 | 5% budget Gain over Token Retrieved | 5% budget Gain over Token Retrieved +4 | 10% budget Retention Token | 10% budget Gain over Token Target | 10% budget Gain over Token Target +4 | 10% budget Gain over Token Retrieved | 10% budget Gain over Token Retrieved +4 | |
| LoCoMo | Qwen3-8B | 105 | 0.71 | −0.04 | −0.06 | −0.10 | −0.10 | 0.72 | −0.04 | −0.04 | −0.10 | −0.09 |
| Llama-3.1-8B | 102 | 0.85 | +0.01 | +0.01 | −0.01 | −0.05 | 0.86 | +0.00 | −0.02 | −0.02 | −0.05 | |
| Mistral-7B | 36 | 0.78 | +0.03 | −0.03 | −0.03 | +0.03 | 0.75 | +0.00 | +0.00 | +0.00 | +0.08 | |
| NQ-Open | Qwen3-8B | 146 | 0.16 | +0.35 | +0.29 | +0.22 | +0.21 | 0.24 | +0.27 | +0.33 | +0.14 | +0.24 |
| Llama-3.1-8B | 128 | 0.21 | +0.15 | +0.18 | +0.08 | +0.12 | 0.38 | −0.02 | +0.00 | −0.09 | −0.07 | |
| Mistral-7B | 101 | 0.28 | +0.31 | +0.21 | +0.14 | +0.15 | 0.34 | +0.25 | +0.25 | +0.08 | +0.14 | |
| Gorilla | Qwen3-8B | 89 | 0.06 | +0.17 | +0.08 | +0.09 | +0.08 | 0.19 | +0.03 | +0.02 | −0.04 | +0.00 |
| Llama-3.1-8B | 100 | 0.12 | +0.10 | +0.06 | +0.02 | +0.01 | 0.31 | −0.09 | −0.11 | −0.17 | −0.17 | |
| Mistral-7B | 108 | 0.10 | +0.49 | +0.34 | +0.19 | +0.21 | 0.30 | +0.30 | +0.23 | +0.00 | +0.09 | |
| TriviaQA | Qwen3-8B | 151 | 0.32 | +0.20 | +0.23 | +0.14 | +0.17 | 0.49 | +0.03 | +0.08 | −0.03 | +0.03 |
| Llama-3.1-8B | 127 | 0.46 | +0.11 | +0.09 | −0.02 | +0.02 | 0.55 | +0.02 | +0.04 | −0.11 | −0.06 | |
| Mistral-7B | 117 | 0.44 | +0.23 | +0.18 | +0.10 | +0.11 | 0.52 | +0.15 | +0.13 | +0.03 | +0.07 |
Table 1. Recomputing chosen chunks instead of tokens. Token: retention of the token-level selector at the budget. The other columns give paired gains over Token when the same budget is spent on the target chunk alone (Target), on the target and its four most similar chunks (Target+4), or on the BGE-M3 top-1 chunk alone or with its four most similar chunks (Retrieved, Retrieved+4). On open-domain QA and API documentation, a single chunk is about 3% of the context and is recomputed in full, except on a few questions at 5%. Target and Retrieved therefore score identically at both budgets outside LoCoMo, and their 10% gains differ from their 5% gains only because the 10% Token baseline is stronger. Bold gains have 95% intervals that exclude zero (Appendix Tables 20 and 21). A LoCoMo session is longer than a passage, and a whole session fits within the 5% budget on few questions (§ 5).
The target chunk. With a 5% budget, recomputing the known target alone beats the token-level selector in all nine open-domain QA and API-documentation settings (Table 1), and it usually uses less than the budget. When the budget doubles to 10%, the target remains significantly ahead of the selector in four of them and is never significantly behind. The gain depends on where the target sits. Averaged over the nine settings, recomputing the target raises retention by 0.13 when the target lies in the first third of the prompt, by 0.35 when it lies in the middle third and by 0.47 when it lies in the last third (Appendix Table 23 and Figure 5). On average, therefore, the more chunks precede the target, the more its recomputation adds.
Figure 5. Change in retention on from recomputing the known target chunk at 5% instead of applying no repair, by the slot of the target (Table 23). In slot 1 the recomputed KV equals the reused KV. Targets in later slots miss more of their context under reuse, and recomputing them raises retention more on average.
| Dataset | Model | Slot 1 | First third | First third Gain | Middle third | Middle third Gain | Last third | Last third Gain |
| NQ-Open | Qwen3-8B | 6 | 50 | +0.20 | 36 | +0.47 | 54 | +0.48 |
| Llama-3.1-8B | 5 | 43 | +0.05 | 36 | +0.22 | 44 | +0.34 | |
| Mistral-7B | 1 | 29 | +0.14 | 32 | +0.47 | 39 | +0.56 | |
| Gorilla | Qwen3-8B | 9 | 23 | +0.09 | 30 | +0.23 | 27 | +0.41 |
| Llama-3.1-8B | 8 | 23 | +0.04 | 33 | +0.12 | 36 | +0.36 | |
| Mistral-7B | 7 | 11 | +0.18 | 37 | +0.38 | 53 | +0.79 | |
| TriviaQA | Qwen3-8B | 4 | 42 | +0.02 | 53 | +0.28 | 52 | +0.42 |
| Llama-3.1-8B | 2 | 34 | +0.15 | 53 | +0.53 | 38 | +0.34 | |
| Mistral-7B | 4 | 32 | +0.34 | 36 | +0.47 | 45 | +0.49 |
Table 23. Change in retention on from recomputing the target chunk (Target, 5%) instead of applying no repair, by the slot of the target in the prompt. Slot 1: the number of questions whose target is the first chunk after the instruction prefix; for these questions the recomputed KV equals the reused KV and the output does not change. The other questions are split into thirds of the prompt by the slot of the target; gives their number and Gain the change in retention.
The retriever’s choice. The retriever ranks the target first on only 47–64% of the questions in the nine open-domain QA and API-documentation settings. Even so, recomputing its top chunk with a 5% budget significantly beats the token-level selector in five of these settings, and recomputing the four chunks most similar to it as well raises the count to seven.
Moving the retrieved chunk last. Under reuse, the chunk order is a free choice, because every cache is re-rotated when the prompt is assembled. InfoFlow KV ([30]) uses this freedom to move the chunks in which question attention is densest toward the question. Since a recomputed chunk gains more on average the more chunks precede it, we move the retriever’s top chunk to the position just before the question (Appendix Tables 27 and 28). Recomputing the chunk in its new position gives significantly higher retention than recomputing it in place in all nine open-domain QA and API-documentation settings, by 0.07 to 0.18. The move also significantly raises the retention of the token-level selector, by 0.11–0.25, in the seven of these settings whose rule ranks tokens by question attention, which over reused caches peaks on the last chunk (§ 4). To separate the effect of the move from that of the recomputation, we compare the moved chunk with selected tokens recomputed on the same reordered prompt, and with InfoFlow-style reordering at the same budget. Against selected tokens, the moved chunk is never significantly worse, and in four settings it is significantly better, by 0.09 to 0.31. In the others the difference lies between −0.07 and +0.04. Compared with InfoFlow-style reordering at the same 5% budget, the moved chunk wins significantly in six of the nine and never loses significantly.
Moving the top three chunks. Moving the retriever’s top three chunks instead, with the top one last, extends the gain to questions on which the top chunk is not the target. With a 10% budget, recomputing only the moved top chunk leaves the rest of the budget unspent, and recomputing all three chunks does better in all nine settings, by 0.06 to 0.16, significantly in seven. If the moved chunks help only when they include the target, the retriever’s hit rates predict the size of this gain (Appendix § 13). The prediction lies within the interval of the measured gain in seven of the nine and above it in the other two (Appendix Table 30, and Table 31 with BM25 as the retriever). Selected tokens recomputed on the same reordered prompt remain competitive at this budget. The three chunks beat them significantly in only three settings and lose significantly in two, both with Llama-3.1-8B.
| Dataset | Model | BGE-M3 | BM25 | Top-1 moved, retention at 10% Predicted | Top-1 moved, retention at 10% Measured | Top-3 − top-1 moved at 10% Predicted | Top-3 − top-1 moved at 10% Measured | Top-1 moved, BM25 − BGE | |
| NQ-Open | Qwen3-8B | 146 | 55/75 | 23/47 | 0.31 | 0.32 [+0.25, +0.40] | +0.19 | +0.18 [+0.12, +0.26] | −0.24 [−0.33, −0.15] |
| Llama-3.1-8B | 128 | 47/70 | 23/46 | 0.29 | 0.28 [+0.20, +0.35] | +0.16 | +0.19 [+0.11, +0.27] | −0.18 [−0.27, −0.09] | |
| Mistral-7B | 101 | 50/71 | 25/41 | 0.35 | 0.38 [+0.28, +0.48] | +0.12 | +0.07 [−0.02, +0.15] | −0.17 [−0.28, −0.05] | |
| Gorilla | Qwen3-8B | 89 | 53/74 | 54/72 | 0.27 | 0.31 [+0.22, +0.42] | +0.08 | +0.11 [+0.01, +0.21] | +0.03 [−0.06, +0.12] |
| Llama-3.1-8B | 100 | 53/69 | 52/67 | 0.27 | 0.28 [+0.19, +0.37] | +0.06 | +0.07 [−0.02, +0.16] | +0.01 [−0.06, +0.09] | |
| Mistral-7B | 108 | 47/72 | 49/72 | 0.44 | 0.46 [+0.37, +0.56] | +0.21 | +0.06 [−0.01, +0.15] | +0.04 [−0.06, +0.12] | |
| TriviaQA | Qwen3-8B | 151 | 64/85 | 65/83 | 0.61 | 0.65 [+0.57, +0.72] | +0.12 | +0.11 [+0.04, +0.17] | +0.05 [−0.02, +0.11] |
| Llama-3.1-8B | 127 | 60/79 | 60/76 | 0.60 | 0.59 [+0.50, +0.67] | +0.12 | +0.08 [−0.01, +0.16] | −0.01 [−0.09, +0.08] | |
| Mistral-7B | 117 | 62/85 | 67/83 | 0.65 | 0.67 [+0.59, +0.75] | +0.11 | +0.08 [+0.03, +0.13] | +0.05 [−0.03, +0.14] |
Table 31. Moving the top chunks of a second retriever, BM25. Predicted values use the BM25 hit rates (top-1) and (top-3) and the retention of the moved BGE-M3 chunk at 5% on the questions where it is and is not the target ( and , listed in Table 30): top-1 retention , and the gain of moving three chunks at 10%, . Measured values: the retention of the moved BM25 top chunk when all its tokens are recomputed at the 10% budget, and the paired gain of the three moved chunks at 10%, with 95% bootstrap intervals. Top-1 moved, BM25 − BGE: the paired difference in retention between the moved BM25 and BGE-M3 top chunks. Bold differences have intervals that exclude zero.
Dialogue memory. A whole LoCoMo session fits within the 5% budget on only 11–21% of the questions, but within the 10% budget on 89–94%. Relative to the token-level selector, recomputing the target session changes retention by −0.04 to +0.03, never significantly, even with the 10% budget, so the larger size of LoCoMo chunks does not explain the lack of gain. Against a ProphetKV selector, recomputing the target session gains significantly for Qwen3-14B (+0.14) but not for Qwen3-32B (+0.07; Appendix § 13).
Larger models. Qwen3-14B and Qwen3-32B, run on the main conditions with ProphetKV as the token-level rule, reproduce all five results that we test for replication (Table 2; Appendix § 13 reports each result).
Related Work
Non-prefix and parallel KV reuse. Prompt Cache (Gim et al., [9]), TurboRAG (Lu et al., [21]), Block-Attention (Ma et al., [22]), KVLink (Yang et al., [35]) and APE (Yang et al., [36]) encode chunks independently or in parallel and concatenate their caches with position adjustments; some of them adapt the model to this layout with fine-tuning or modified attention. Zhang et al. (2025) trace the loss of parallel encoding to unusually high attention entropy and reduce it with attention sinks and selective attention. Proposition 1 instead splits the first-layer deviation exactly into a cross-chunk term and a position term, and our distractor sources show that the loss also depends on what the other chunks contain.
Repairing reused caches. Token-level repairs such as CacheBlend (Yao et al., [37]), EPIC (Hu et al., [11]), ProphetKV (Wang et al., [31]) and CacheTune (Li et al., [19]) differ in how they choose the tokens to recompute (Appendix § 12 and Table 9). InfoFlow KV (Teng et al., [30]) selects tokens by question attention; before recomputing, it moves the chunks in which the selected tokens are densest to positions next to the question. § 5 compares with it at the same budget. SurroKV (Anonymous, [1]), a concurrent submission, encodes chunks after a fixed surrogate prefix and then recomputes tokens selected by question attention; Appendix Table 11 evaluates surrogate prefixes, with and without online recomputation, in our settings.
Chunk and suffix repair. SAFE-KV (Zhang, [39]) starts at a selected document and re-encodes the rest of the prompt over the retained reused cache of the earlier documents. It takes full prefill of the prompt in its original order as the reference and does not reorder documents. On a reordered prompt, recomputing the chunk moved last is the SAFE-KV suffix repair that starts at the last chunk; unlike SAFE-KV, we use the freedom of chunk order to shrink the suffix that needs repair to one chunk. Budgeted cache repair (BCR; Kang and Yoo, [16]) recomputes cache rows or short slices rather than whole retrieved documents (Appendix § 12). Both in SAFE-KV and after small document edits (Mao et al., [24]), copying full-prefill states into a cache and recomputing them in place give nearly the same result for a whole suffix or a contiguous window. The gap of § 3.2 arises when the target alone is inserted while the chunks before it stay reused (Appendix § 13). Appendix § 12 covers further work.
Conclusion
Encoding retrieved chunks independently makes models lose a large share of the questions that need the retrieved context. The loss grows as the other chunks become more like the target, even in a model fine-tuned for block-wise encoding. Because the recoverable damage is spread over the distractors and accumulates in the odds of failure, restoring the distractors costs nearly a full prefill, and a small budget is better spent on the target. When the budget is spent on the target, our controls show that contextual compatibility, rather than closeness to full prefill, explains what helps: recomputing the target against the reused cache beats inserting its full-prefill KV while the chunks before it stay reused. Question attention over reused caches mostly goes to the chunk next to the question, so the target has to be found by other means. With a 5% budget on open-domain QA and API documentation, recomputing the known target chunk beats token-level selectors. Moving the retriever’s top chunk next to the question and recomputing it there beats InfoFlow-style reordering at the same budget in most settings, and it never does significantly worse than recomputing selected tokens on the same prompt. These gains are bounded by retrieval: when the retriever does not rank the target first, the moved chunk does worse than selected tokens, and the gain from moving the top three chunks follows the retriever’s hit rate in most settings. Two further limits do not depend on retrieval. With a 10% budget, token-level selectors remain competitive, and for the 7B–8B models on dialogue memory, recomputing chunks never significantly improves on the token-level selector chosen for each setting. Evaluations of cache reuse should therefore use contexts built by retrieval, and on open-domain QA and API documentation, repair methods should locate the answer-bearing chunk rather than rely on attention over the reused cache.
Limitations
Our experiments use 7B–8B instruction-tuned models on four English datasets. We repeat the main conditions with two larger Qwen3 models, but larger models and other languages are not covered. Models fine-tuned for chunked encoding are represented by a single released model, and we do not test repairs on it. The chunk-level allocation needs either the known target or a retriever, so its gains in deployment depend on retrieval quality. We study only small budgets and count them in recomputed tokens. Latency is measured only in our research engine, and in an optimised serving system, recomputing a contiguous chunk and recomputing scattered tokens may take different amounts of time. Finally, we compare with InfoFlow KV through our own re-implementation, on our models and datasets.
AI Usage Statement
We used AI assistants, including OpenAI’s GPT models and Anthropic’s Claude, to help implement and check the experiment and analysis code, check the references, and edit the text. The authors take full responsibility for the content of this paper.
Data and Prompts
Figure 3 shows the example of Figure 1 chunk by chunk.

Figure 3. The example of Figure 1, shown chunk by chunk. The bottom line of each box states what that chunk attends to when its keys and values are computed; under reuse, every document after the first cannot attend to the documents before it.
Splits and sources. LoCoMo answers are scored by token F1 and counted as correct when F1 . Gorilla prompts contain the user request only, and an answer is correct when it contains the ground-truth api_name. Each dataset is split by document or conversation into 70 development questions and a disjoint test set of 250 questions (800 for TriviaQA, where many questions are answerable without context). For NQ-Open and Gorilla, where every model has fewer than 80 context-dependent test questions, we add a second set of 250 questions built in the same way. In every question, the slot of the target is drawn uniformly at random from all slots with a fixed seed and is the same under SIM, RAND and OTHER. Replacement chunks are chosen to match the length of the SIM chunk in the same position (within 10% for 92–100% of chunks). In each open-domain QA or API dataset, a normalised gold answer occurs in some RAND or OTHER chunk for at most 1.9% of the questions, and in some SIM distractor for at most 4.6% of the questions. SIM is the kind of context that a retriever returns; Gorilla is discussed below.
LoCoMo. Each chunk is one session of a conversation, headed by its date, for example “Session 3 (9:27 am on 7 February, 2022):”. We keep questions whose evidence lies in a single session and drop unanswerable ones. SIM uses up to 15 other sessions of the same conversation, drawn in random order among those that do not contain the answer; RAND uses sessions of other conversations. Development and test questions come from different conversations; the test set covers eight conversations.
NQ-Open. We use the 30-document version of Liu et al. [20]: one passage that contains the answer and up to 29 passages retrieved by Contriever (Izacard et al., [13]) that do not contain it. RAND uses passages retrieved for other questions.
TriviaQA. We use the DPR release (Karpukhin et al., [17]), which provides one positive passage and 29 hard negatives, and we remove any passage that contains an answer alias. Because the no-context baseline answers many TriviaQA questions, the test set has 800 questions.
Gorilla. Requests come from the HuggingFace part of APIBench. Each document lists an API’s name, domain, functionality, description and example code. The prompt contains the user request only, without the reference API call or code. SIM uses the 30 documents whose BGE-M3 embeddings are closest to that of the request together with its reference solution, whereas the detector (§ 4) and the retriever of § 5 embed the request alone. RAND uses documents from other task categories. Retention on Gorilla is already zero under RAND, so building SIM with the reference solution cannot create a positive similarity term. Once the user request is extracted on its own, none of the request texts contains the reference call or code, but four requests still name their API, and we exclude them. Because SIM uses the reference solution, we also build the document list from the request alone (SIM-q), keeping the same corpus, filters, number of documents and target position. For each question, on average 40–42% of the SIM-q distractor documents also appear in SIM. Full prefill answers 68–78% of under SIM-q; on those questions, retention under reuse is 0.01–0.03, against 0.00–0.07 under SIM and 0.00 under RAND. The paired differences from SIM are +0.01 [+0.00, +0.05] with Qwen3-8B, −0.03 [−0.07, +0.00] with Llama-3.1-8B and −0.04 [−0.10, +0.01] with Mistral-7B. The differences from RAND (+0.01 to +0.03) are significant for none of the models.
All evaluated questions. Table 3 gives accuracy on all evaluated questions. On open-domain QA, reuse lowers accuracy by 0.33–0.46 relative to full prefill and answers significantly fewer questions than the model without context in all six settings, because it also loses 47–75% of the questions that the model answers without context. On LoCoMo and Gorilla, where the no-context baseline is low, reuse stays above it.
| Dataset | Model | No context | Full prefill | Reuse | Full − reuse | Reuse − no context | middle | Retention middle | |
| LoCoMo | Qwen3-8B | 250 | 0.11 | 0.61 | 0.36 | +0.26 [+0.19, +0.32] | +0.24 [+0.18, +0.32] | 129 | 0.49 |
| Llama-3.1-8B | 250 | 0.10 | 0.58 | 0.50 | +0.08 [+0.04, +0.13] | +0.40 [+0.32, +0.50] | 123 | 0.74 | |
| Mistral-7B | 250 | 0.01 | 0.20 | 0.15 | +0.04 [+0.00, +0.09] | +0.14 [+0.08, +0.20] | 48 | 0.60 | |
| NQ-Open | Qwen3-8B | 500 | 0.29 | 0.56 | 0.22 | +0.34 [+0.30, +0.39] | −0.07 [−0.11, −0.03] | 164 | 0.15 |
| Llama-3.1-8B | 500 | 0.41 | 0.61 | 0.15 | +0.46 [+0.41, +0.51] | −0.26 [−0.30, −0.21] | 136 | 0.15 | |
| Mistral-7B | 500 | 0.36 | 0.53 | 0.20 | +0.33 [+0.28, +0.37] | −0.15 [−0.20, −0.10] | 128 | 0.18 | |
| Gorilla | Qwen3-8B | 496 | 0.00 | 0.19 | 0.02 | +0.18 [+0.14, +0.21] | +0.02 [+0.01, +0.03] | 95 | 0.01 |
| Llama-3.1-8B | 496 | 0.00 | 0.23 | 0.03 | +0.19 [+0.15, +0.23] | +0.03 [+0.02, +0.05] | 112 | 0.04 | |
| Mistral-7B | 496 | 0.00 | 0.30 | 0.02 | +0.28 [+0.24, +0.32] | +0.02 [+0.01, +0.03] | 147 | 0.05 | |
| TriviaQA | Qwen3-8B | 800 | 0.63 | 0.75 | 0.39 | +0.36 [+0.33, +0.40] | −0.24 [−0.28, −0.20] | 168 | 0.25 |
| Llama-3.1-8B | 800 | 0.72 | 0.79 | 0.33 | +0.45 [+0.42, +0.49] | −0.39 [−0.42, −0.35] | 138 | 0.20 | |
| Mistral-7B | 800 | 0.68 | 0.72 | 0.36 | +0.36 [+0.33, +0.40] | −0.33 [−0.36, −0.29] | 130 | 0.25 |
Table 3. Accuracy on every evaluated question under SIM (the Questions column of Table 4): without context, with full prefill and with position-corrected reuse; the differences have paired 95% bootstrap intervals. middle: the number of questions answered wrongly without context and correctly with full prefill under SIM, before the RAND and OTHER conditions of are applied. Retention middle: the share of these questions that reuse still answers correctly.
Context-dependent subset. Table 4 shows how the evaluated questions are narrowed down to . Retention on is the share of these questions that reuse still answers correctly.
| Dataset | Model | Questions | No context wrong | + full prefill right (SIM) | + right under RAND, OTHER () |
| LoCoMo | Qwen3-8B | 250 | 222 | 129 | 105 |
| Llama-3.1-8B | 250 | 225 | 123 | 102 | |
| Mistral-7B | 250 | 247 | 48 | 36 | |
| NQ-Open | Qwen3-8B | 500 | 354 | 164 | 146 |
| Llama-3.1-8B | 500 | 296 | 136 | 128 | |
| Mistral-7B | 500 | 322 | 128 | 101 | |
| Gorilla | Qwen3-8B | 496 | 496 | 95 | 89 |
| Llama-3.1-8B | 496 | 496 | 112 | 100 | |
| Mistral-7B | 496 | 496 | 147 | 108 | |
| TriviaQA | Qwen3-8B | 800 | 299 | 168 | 151 |
| Llama-3.1-8B | 800 | 226 | 138 | 127 | |
| Mistral-7B | 800 | 254 | 130 | 117 |
Table 4. Number of questions left after each condition, from the evaluated questions to (test set and, where used, the second question set).
Prompts. The user message consists of the instruction, the chunks separated by blank lines, and the question. It is wrapped in each model’s chat template with the generation prompt, and Qwen3 runs with thinking disabled. The instructions are:
LoCoMo: “Answer the question based on the following conversation sessions. Only provide the short, concise answer without any other words.”
NQ-Open and TriviaQA: “Write a high-quality answer for the given question using only the provided search results (some of which might be irrelevant). Only output the concise answer without any other words.”
Gorilla: “Below is a list of API documentations. Select and output the exact api_name that best matches the instruction.”
The question follows as “Question: . . . Answer:” (LoCoMo, NQ-Open, TriviaQA) or “Instruction: . . . Answer with api_name:” (Gorilla). Passages are headed “Document [1] (Title: . . . ):”, “Document [2] (Title: . . . ):” and so on, and API documents are headed “API Document [1]:”, “API Document [2]:” and so on. The number in square brackets is the position of the chunk in the prompt, counting the target chunk.
Implementation and Scoring
Reuse. The shared instruction prefix is encoded once. Each chunk is encoded directly after the prefix, its keys are re-rotated to the chunk’s position in the full prompt, and the caches are concatenated; the question tokens and the answer are then processed against this cache. Full prefill encodes the same tokens in one pass. When no chunk is replaced, the cache of the prefix is the same under reuse and under full prefill. Surrogate repairs encode each chunk after the prefix and a 512- or 1,024-token surrogate, and then discard the surrogate’s KV. SurroKV full is our re-implementation of the full method from its description in Anonymous [1]: it uses the 1,024-token Wikipedia surrogate and then recomputes the 10% of tokens ranked highest by question attention. Online methods and chunk-level allocations recompute the selected token positions through all layers. Each recomputed token attends to the prefix and to all earlier positions of the current cache (reused, or already recomputed in this pass); the KV of later chunks is not updated; and the question and answer tokens then attend to the whole cache. Our CacheBlend implementation selects the tokens to recompute by the deviation of their keys and values at the second layer, a one-step version of the gradual filtering of Yao et al. [37]. Budgets are fractions of the context tokens.
Scoring. Each output has at most 32 new tokens (64 on Gorilla). LoCoMo is scored by the token F1 of the first output line against the gold answer; NQ-Open and TriviaQA by whether a normalised gold answer occurs in the normalised first output line; and Gorilla by whether the gold api_name occurs in the output. Answers on LoCoMo, NQ-Open and TriviaQA are scored on their first line, as in the original setups. When Qwen3-8B begins with a copied document header, the answer is therefore scored wrong even if the correct answer follows; this happens on at most 4.6% of the questions in any condition. All scores are binarised at 0.5 before , retention and every difference are computed. If the api_name must match exactly instead of only being contained in the output, the same-category term on Gorilla remains significant for three models, and the similarity term is significantly positive for none of them.
Difference from the meaningful subset. differs from the meaningful subset of Cestola et al. [5] in three ways: correctness is a binarised score rather than F1 for full prefill and F1 without context; full prefill must be correct under all three sources; and we do not drop yes/no or either-or questions. Yes/no questions account for at most 2 LoCoMo questions per model, and none of the questions in the other datasets are yes/no questions.
Token-level selectors. The selector chosen on development questions is ProphetKV for LoCoMo with Qwen3-8B and Llama-3.1-8B, for NQ-Open, and for TriviaQA with Qwen3-8B and Llama-3.1-8B; question attention for LoCoMo and TriviaQA with Mistral-7B; EPIC for Gorilla with Qwen3-8B and Mistral-7B; and CacheBlend for Gorilla with Llama-3.1-8B. Question attention scores each chunk token by the attention that the question tokens place on it over the reused cache: in every layer, this attention is averaged over heads and question tokens and normalised over the chunk tokens, and the scores are then averaged over all layers. In the Token 10% column of Table 11, EPIC recomputes the first tokens of every chunk of tokens at budget (the original EPIC recomputes a fixed number of initial tokens of every chunk except the first), and CacheBlend selects tokens by their KV deviation. The allocation tables instead apply the setting’s ranking rule to all chunks jointly. For EPIC, the tokens of all chunks are ranked by their offset within their chunk, so every chunk receives the same number of initial tokens and the rest of the budget goes to the earliest chunks. For Gorilla with Llama-3.1-8B, question attention is used in place of CacheBlend. The token-level baselines of all three Gorilla settings therefore differ between Table 11 and the allocation tables. The in-domain pad of Table 11, the text placed before every chunk, is built from the first SIM chunks of two development questions and cut to 512 tokens.
| Dataset | Model | None | Pad in-domain | Pad SurroKV neutral | Pad neutral | SurroKV full | Token 10% | Token 10% + pad | |
| LoCoMo | Qwen3-8B | 105 | 0.53 | 0.72 | 0.66 | 0.59 | 0.90 | 0.72 | 0.91 |
| Llama-3.1-8B | 102 | 0.81 | 0.85 | 0.80 | 0.85 | 0.95 | 0.86 | 0.97 | |
| Mistral-7B | 36 | 0.72 | 0.78 | 0.81 | 0.81 | 0.83 | 0.75 | 0.92 | |
| NQ-Open | Qwen3-8B | 146 | 0.15 | 0.22 | 0.17 | 0.18 | 0.41 | 0.24 | 0.38 |
| Llama-3.1-8B | 128 | 0.16 | 0.17 | 0.13 | 0.15 | 0.63 | 0.38 | 0.55 | |
| Mistral-7B | 101 | 0.18 | 0.22 | 0.23 | 0.24 | 0.48 | 0.34 | 0.42 | |
| Gorilla | Qwen3-8B | 89 | 0.00 | 0.13 | 0.00 | 0.00 | 0.34 | 0.12 | 0.39 |
| Llama-3.1-8B | 100 | 0.04 | 0.06 | 0.08 | 0.04 | 0.33 | 0.16 | 0.25 | |
| Mistral-7B | 108 | 0.06 | 0.24 | 0.03 | 0.03 | 0.23 | 0.24 | 0.33 | |
| TriviaQA | Qwen3-8B | 151 | 0.27 | 0.34 | 0.36 | 0.36 | 0.68 | 0.49 | 0.64 |
| Llama-3.1-8B | 127 | 0.20 | 0.42 | 0.42 | 0.38 | 0.78 | 0.55 | 0.76 | |
| Mistral-7B | 117 | 0.25 | 0.40 | 0.37 | 0.36 | 0.73 | 0.52 | 0.68 |
Table 11. Retention on under existing repairs, under SIM. None: no repair. Pad: each chunk encoded after 512 tokens of in-domain text, of the SurroKV Wikipedia surrogate, or of neutral prose. SurroKV full: a 1,024-token surrogate followed by recomputing 10% of the tokens ranked by question attention. Token 10%: the online method chosen for the setting on development questions, with EPIC applied to each chunk and CacheBlend in a one-step version (Appendix § 9). Token 10% + pad: this method combined with the in-domain pad.
Allocation rule. Within the chosen chunks, tokens are ranked by the setting’s token-level rule (EPIC’s first-token rule, ProphetKV or question attention), and the token-level baseline applies the same rule to all chunks, so that an allocation differs from its baseline only in which chunks may be recomputed. CacheBlend selects tokens by comparing recomputed and reused KV and has no within-chunk form; in the one setting where it is the best selector on development questions (Gorilla with Llama-3.1-8B), question attention is used instead for the baseline and the allocations. When the chosen chunks hold fewer tokens than the budget, all their tokens are recomputed.
Statistics. Intervals are percentile intervals of 2,000 paired bootstrap resamples with a fixed seed. A resample draws questions (conversations for LoCoMo) with replacement and is shared by all conditions of a setting and by every table that reports the same questions. For the competing-answer comparison (Table 16), the same resample of questions is used for SIM and RAND. The abstention comparison (Table 18) contrasts two different sets of questions and resamples them independently. Intervals are reported per comparison without correction for multiple comparisons.
| Dataset | Model | SIM | RAND | SIM − RAND | Abstain SIM | Abstain RAND | Conditional (entity) | Conditional (phrase) | |
| NQ-Open | Qwen3-8B | 146 | 61.6 | 17.1 | +44.5 [+34.9, +53.4] | 4.0 | 25.0 | +40.9 [+27.5, +53.3] | +47.6 [+35.1, +60.1] |
| Llama-3.1-8B | 128 | 63.3 | 39.8 | +23.4 [+13.3, +33.6] | 0.0 | 14.9 | +6.8 [−5.8, +19.1] | +8.3 [−3.5, +19.8] | |
| Mistral-7B | 101 | 58.4 | 12.9 | +45.5 [+33.7, +57.4] | 10.8 | 72.6 | +14.7 [−7.8, +40.4] | +32.8 [+8.9, +56.8] | |
| TriviaQA | Qwen3-8B | 151 | 40.4 | 17.2 | +23.2 [+13.2, +32.5] | 8.2 | 17.9 | +22.7 [+7.9, +37.5] | +31.2 [+16.6, +44.7] |
| Llama-3.1-8B | 127 | 43.3 | 19.7 | +23.6 [+12.6, +33.1] | 3.0 | 9.8 | +10.7 [−5.3, +25.4] | +21.9 [+7.7, +35.9] | |
| Mistral-7B | 117 | 40.2 | 7.7 | +32.5 [+23.1, +41.9] | 26.1 | 59.3 | +31.4 [+8.5, +54.0] | +40.7 [+17.6, +62.8] |
Table 16. Reuse answers under SIM and RAND that name an entity found in some distractor chunk but not in the target chunk. SIM and RAND: share of all questions (%); SIM − RAND: paired difference in percentage points with 95% bootstrap interval. Abstain: share of abstentions among wrong reuse answers. Conditional (entity): the SIM − RAND difference among wrong answers that are not abstentions. Conditional (phrase): the same difference when any 1–5-word phrase, rather than only an entity, counts as a match. Bold differences have intervals that exclude zero.
| Dataset | Model | reuse | Reuse (%) | full | Full prefill (%) | Difference |
| NQ-Open | Qwen3-8B | 124 | 4.0 | 218 | 1.4 | +2.7 [−0.9, +6.3] |
| Llama-3.1-8B | 107 | 0.0 | 193 | 0.5 | −0.5 [−1.6, +0.0] | |
| Mistral-7B | 83 | 10.8 | 235 | 6.8 | +4.0 [−3.3, +11.6] | |
| TriviaQA | Qwen3-8B | 110 | 8.2 | 199 | 3.0 | +5.2 [+0.0, +10.7] |
| Llama-3.1-8B | 101 | 3.0 | 171 | 4.1 | −1.1 [−5.4, +3.4] | |
| Mistral-7B | 88 | 26.1 | 222 | 8.6 | +17.6 [+8.0, +26.9] |
Table 18. Share of abstentions among wrong outputs, for position-corrected reuse under SIM (on ) and for full prefill under SIM (on the questions that full prefill answers wrongly).
Holm’s correction. We also apply Holm’s correction within each family of settings whose count the text reports, with two-sided -values computed from the bootstrap standard error with a normal approximation. Table 5 lists the 23 families, which contain 196 comparisons in total, with their counts before and after the correction. The counts of 13 families do not change.
| Comparison | Reported in | Uncorrected | Holm | |
| Loss, | Table 10 | 12 | 12 / 0 | 12 / 0 |
| Same-category term | Table 10 | 12 | 9 / 0 | 9 / 0 |
| Similarity term, open-domain QA | Table 10 | 6 | 5 / 0 | 5 / 0 |
| Competing answers, SIM − RAND | Table 16 | 6 | 6 / 0 | 6 / 0 |
| All distractors replaced − target replaced | Table 13 | 12 | 11 / 0 | 11 / 0 |
| Reuse − no context, all questions, open-domain QA | Table 3 | 6 | 0 / 6 | 0 / 6 |
| Target − token-level selector, 5% | Table 1 | 9 | 9 / 0 | 9 / 0 |
| Target − token-level selector, 10% | Table 1 | 9 | 4 / 0 | 4 / 0 |
| Retrieved − token-level selector, 5% | Table 1 | 9 | 5 / 0 | 3 / 0 |
| Retrieved+4 − token-level selector, 5% | Table 1 | 9 | 7 / 0 | 5 / 0 |
| Recomputed target − inserted full-prefill KV | Tables 33, 34 | 7 | 7 / 0 | 4 / 0 |
| Moved chunk − token-level selector | Table 27 | 9 | 9 / 0 | 9 / 0 |
| Moved chunk − the same chunk recomputed in place | Table 27 | 9 | 9 / 0 | 9 / 0 |
| Selector on the reordered − original prompt | Table 28 | 9 | 7 / 0 | 7 / 0 |
| Moved chunk − selected tokens, reordered prompt | Table 28 | 9 | 4 / 0 | 3 / 0 |
| Moved chunk − InfoFlow-style reordering, 5% | Table 29 | 9 | 6 / 0 | 5 / 0 |
| Moved chunk at 5% − InfoFlow-style reordering at 15% | Table 29 | 9 | 1 / 2 | 1 / 1 |
| Three moved chunks − one moved chunk, 10% | Table 30 | 9 | 7 / 0 | 6 / 0 |
| Three moved chunks − selected tokens, 10% | Table 30 | 9 | 3 / 2 | 3 / 2 |
| One moved chunk − selected tokens, 10% | Appendix F | 9 | 2 / 4 | 0 / 1 |
| Gap closed by removing competing entities | Table 17 | 6 | 4 / 0 | 3 / 0 |
| Gap closed by removing other entities | Table 17 | 6 | 1 / 0 | 1 / 0 |
| Competing − other entities removed | Table 17 | 6 | 3 / 0 | 2 / 0 |
Table 5. Families of settings whose count of significant differences the text reports. : number of settings in the family. Uncorrected and Holm: settings in which the difference is significantly positive / significantly negative, before and after Holm’s correction within the family. Each difference is the first condition named minus the second.
Leaving out conversations. On LoCoMo, we also leave out each of the eight conversations in turn and recompute the bootstrap intervals. Whichever conversation is left out, 10 of the 12 terms of Table 10 keep their significance, and the other two change in at most one of these analyses. None of the six comparisons of Target with the selector at 5% and 10% becomes significant (point estimates to ), and the loss from recomputing the retrieved chunk for Qwen3-8B stays significant in all eight analyses. Against a ProphetKV selector, the gain of Target stays significant in eight of the eight analyses for Qwen3-14B ( to ), but in only five of them for Mistral-7B ( to ).
| Dataset | Model | Loss | Universal | Same-category | Similarity | |
| LoCoMo | Qwen3-8B | 105 | 0.47 | 0.11 | +0.19 [+0.10, +0.26] | +0.16 [+0.08, +0.24] |
| Llama-3.1-8B | 102 | 0.19 | 0.14 | +0.11 [+0.04, +0.18] | −0.06 [−0.13, +0.02] | |
| Mistral-7B | 36 | 0.28 | 0.31 | −0.11 [−0.30, +0.08] | +0.08 [−0.03, +0.20] | |
| NQ-Open | Qwen3-8B | 146 | 0.85 | 0.43 | +0.23 [+0.14, +0.32] | +0.19 [+0.12, +0.27] |
| Llama-3.1-8B | 128 | 0.84 | 0.72 | −0.04 [−0.13, +0.06] | +0.16 [+0.07, +0.25] | |
| Mistral-7B | 101 | 0.82 | 0.65 | +0.07 [−0.02, +0.16] | +0.10 [−0.01, +0.21] | |
| Gorilla | Qwen3-8B | 89 | 1.00 | 0.18 | +0.82 [+0.74, +0.89] | +0.00 [+0.00, +0.00]† |
| Llama-3.1-8B | 100 | 0.96 | 0.38 | +0.62 [+0.53, +0.71] | −0.04 [−0.08, −0.01]† | |
| Mistral-7B | 108 | 0.94 | 0.92 | +0.08 [+0.04, +0.14] | −0.06 [−0.10, −0.02]† | |
| TriviaQA | Qwen3-8B | 151 | 0.73 | 0.30 | +0.25 [+0.17, +0.33] | +0.17 [+0.08, +0.26] |
| Llama-3.1-8B | 127 | 0.80 | 0.25 | +0.23 [+0.14, +0.32] | +0.31 [+0.22, +0.41] | |
| Mistral-7B | 117 | 0.75 | 0.33 | +0.13 [+0.03, +0.22] | +0.29 [+0.19, +0.40] |
Table 10. Reuse loss and its decomposition on . Loss is , the universal term is , the same-category term is and the similarity term is , all for position-corrected reuse. Intervals are 95% paired bootstrap intervals, with questions resampled (conversations for LoCoMo); bold intervals exclude zero. †Retention under RAND is zero, so the term cannot be positive; these intervals are not set in bold.
Determinism. Running the first 60 NQ-Open test questions a second time with Qwen3-8B reproduced all 787 generated outputs exactly. The LoCoMo rows of Table 1 and Tables 3, 7, 24, 27–30, 33 and 34 come from separate runs. Wherever one of these runs repeats a condition of the main experiments or of another of these runs, we compared the two on every shared question. The scores are identical, and so are the generated outputs wherever both runs store them, except in Table 24 when the chosen chunks exceed the budget and their tokens are ranked by a different rule than in the main experiments.
Hardware and software. Model runs use bfloat16 weights and greedy decoding. Each process runs on one NVIDIA RTX 4090 D (24 GB), except that Qwen3-14B uses two of these GPUs and Qwen3-32B uses one NVIDIA RTX PRO 6000 (96 GB). Most runs use PyTorch 2.8 and Transformers 4.56. A few supplementary experiments ran on a second server with the same GPUs, PyTorch 2.7 and Transformers 4.52.
Proposition 1
Proof. Fix a head, write for the attention logit of on key under full prefill, and for a set of keys . The keys that sees under full prefill split into , where are the tokens of up to , and . With ,
and subtracting gives Equation (3). Under reuse, sees the keys of only. If the hidden states are the same, the queries, keys and values coincide up to the rotary position. Values carry no rotation, and logits depend on positions only through their difference. Within the differences are the same under both encodings, so the logits on agree. On they differ, because reuse shortens the distance between and every prefix token by . The second term of Equation (3) therefore vanishes when . For we also have , so the two encodings give the same output for every token of in the first layer. By induction over layers, the hidden states, keys and values of then agree in every layer. In the first layer every token enters with its embedding, so the assumption of equal hidden states holds for all tokens. In later layers the hidden states of and differ between the two encodings, and the second term of Equation (3) is then no longer only a position effect.
First-layer measurement. For Table 6, on the first 30 questions of each open-domain QA and API setting (ordered by identifier), we compute both sides of Equation (3) in the first layer for every head and every chunk token under SIM, in float32 on CPU. The largest difference between the two sides is . Both terms are zero for the first chunk in every question. The attention mass that a chunk places on the earlier chunks rises with the chunk’s position in all nine settings (Spearman correlation 0.94–1.00), from 0.17–0.24 for the second chunk to 0.34–0.62 for the last. The position term is 0.31–0.63 times the size of the cross-chunk term, and the two terms partly cancel.
| Dataset | Model | 2nd chunk | first third | last third | last chunk | Spearman slot, | Position term / cross term | Max. residual | |
| NQ-Open | Qwen3-8B | 30 | 0.19 | 0.27 | 0.36 | 0.36 | 0.99 | 0.45 | 2e-7 |
| Llama-3.1-8B | 30 | 0.24 | 0.40 | 0.59 | 0.60 | 1.00 | 0.63 | 1e-7 | |
| Mistral-7B | 30 | 0.21 | 0.38 | 0.60 | 0.62 | 1.00 | 0.38 | 5e-7 | |
| Gorilla | Qwen3-8B | 30 | 0.17 | 0.26 | 0.34 | 0.34 | 0.94 | 0.39 | 2e-7 |
| Llama-3.1-8B | 30 | 0.20 | 0.39 | 0.58 | 0.58 | 0.98 | 0.61 | 2e-7 | |
| Mistral-7B | 30 | 0.21 | 0.38 | 0.58 | 0.60 | 1.00 | 0.31 | 6e-7 | |
| TriviaQA | Qwen3-8B | 30 | 0.19 | 0.27 | 0.36 | 0.36 | 1.00 | 0.45 | 2e-7 |
| Llama-3.1-8B | 30 | 0.24 | 0.40 | 0.59 | 0.60 | 1.00 | 0.63 | 2e-7 | |
| Mistral-7B | 30 | 0.21 | 0.38 | 0.60 | 0.62 | 1.00 | 0.37 | 4e-7 |
Table 6. First-layer decomposition of Proposition 1 under SIM, on the first 30 questions of each setting (CPU, float32). : full-prefill attention mass on the earlier chunks, averaged over heads and the tokens of a chunk, then over questions, for the second chunk, the chunks in the first and last thirds of the prompt, and the last chunk. Spearman: rank correlation between the slot of a chunk and its . Position term / cross term: the ratio of the norm of to the norm of , averaged over the chunks after the first. Residual: largest absolute difference between the two sides of the identity.
Restoring distractors. Figure 4 plots the results of Table 7. All three predictions use only and from the main experiments, and the odds prediction applies Equation (2) to the retention of a setting rather than to each question. For each question, the distractors are put in a random order fixed by a seed and the first are restored, so the restored sets are nested; the target chunk itself stays reused. Over to 24, the odds model has the smallest squared error in five of the six settings, and summed over them its squared error is 0.06, against 0.18 for the probit model and 0.20 for the linear model. At , the odds prediction falls below the interval of the measured retention in three of them, and the measured retention exceeds it by up to 0.10, so the curve is convex as predicted but rises faster near the end. Restoring the 16 distractors most similar to the target first, instead of 16 random distractors, changes retention by −0.05 to +0.03, and the change is never significant. Gorilla is left out because is 0.00–0.06 there, and at such values the odds are undefined or extreme.
| Dataset | Model | 4 | 8 | 16 | 24 | Odds | Probit | Linear | Most similar first, 16 | ||||
| NQ-Open | Qwen3-8B | 146 | 0.15 | 0.19 | 0.23 | 0.32 | [+0.25, +0.40] | 0.51 | 0.71 | 0.27 | 0.44 | 0.46 | 0.27 |
| Llama-3.1-8B | 128 | 0.16 | 0.20 | 0.20 | 0.30 | [+0.22, +0.38] | 0.55 | 0.70 | 0.28 | 0.44 | 0.46 | 0.29 | |
| Mistral-7B | 101 | 0.18 | 0.25 | 0.29 | 0.32 | [+0.23, +0.42] | 0.49 | 0.66 | 0.30 | 0.43 | 0.45 | 0.29 | |
| TriviaQA | Qwen3-8B | 151 | 0.27 | 0.31 | 0.30 | 0.43 | [+0.35, +0.51] | 0.60 | 0.80 | 0.43 | 0.58 | 0.56 | 0.40 |
| Llama-3.1-8B | 127 | 0.20 | 0.29 | 0.35 | 0.43 | [+0.34, +0.51] | 0.60 | 0.74 | 0.34 | 0.49 | 0.50 | 0.46 | |
| Mistral-7B | 117 | 0.25 | 0.27 | 0.31 | 0.35 | [+0.26, +0.44] | 0.50 | 0.71 | 0.39 | 0.50 | 0.50 | 0.31 |
Table 7. Replacing the KV of distractors with their full-prefill KV under SIM, with the target left reused and the distractors taken in a random order fixed per question. The columns and (every distractor replaced) come from the main experiments. Predicted is computed from and alone in three ways: additive odds of failure (Equation (2)), linear in the probit of , and linear in . Last column: retention when the 16 distractors most similar to the target are replaced first.
Models Fine-Tuned for Chunked Encoding
Results. The Tulu3 model that Ma et al. [22] fine-tune to encode documents as independent blocks loses 0.60 of its questions on NQ-Open in block mode, compared with 0.82 for position-corrected reuse with its full-attention counterpart. Its universal term falls to 0.23 (0.61 for the counterpart), but its similarity term is +0.40 [+0.29, +0.50] (Table 8, which also covers Gorilla), so the similarity term accounts for most of its loss on NQ-Open.
| Dataset | Model () | Reuse | OTHER | RAND | SIM | Loss | Universal | Same-category | Similarity |
| NQ-Open | Tulu3-RAG (83) | Pos.-corr. | 0.39 | 0.40 | 0.18 | +0.82 [+0.73, +0.90] | +0.61 [+0.51, +0.71] | −0.01 [−0.13, +0.10] | +0.22 [+0.08, +0.35] |
| Block | 0.07 | 0.08 | 0.10 | +0.90 [+0.83, +0.96] | +0.93 [+0.87, +0.98] | −0.01 [−0.08, +0.06] | −0.01 [−0.10, +0.06] | ||
| Tulu3-Block-FT (111) | Pos.-corr. | 0.74 | 0.60 | 0.20 | +0.80 [+0.73, +0.87] | +0.26 [+0.18, +0.34] | +0.14 [+0.03, +0.24] | +0.41 [+0.31, +0.51] | |
| Block | 0.77 | 0.79 | 0.40 | +0.60 [+0.51, +0.69] | +0.23 [+0.16, +0.32] | −0.03 [−0.12, +0.05] | +0.40 [+0.29, +0.50] | ||
| Gorilla | Tulu3-RAG (102) | Pos.-corr. | 0.44 | 0.00 | 0.04 | +0.96 [+0.92, +0.99] | +0.56 [+0.46, +0.66] | +0.44 [+0.34, +0.54] | −0.04† [−0.08, −0.01] |
| Block | 0.03 | 0.00 | 0.01 | +0.99 [+0.97, +1.00] | +0.97 [+0.93, +1.00] | +0.03 [+0.00, +0.07] | −0.01† [−0.03, +0.00] | ||
| Tulu3-Block-FT (102) | Pos.-corr. | 0.76 | 0.00 | 0.06 | +0.94 [+0.89, +0.98] | +0.24 [+0.16, +0.32] | +0.76 [+0.68, +0.84] | −0.06† [−0.11, −0.02] | |
| Block | 0.45 | 0.02 | 0.14 | +0.86 [+0.79, +0.92] | +0.55 [+0.45, +0.65] | +0.43 [+0.34, +0.53] | −0.12 [−0.20, −0.05] | ||
| Method | Sparse tokens | Full prefill as reference or explanation | Attention locates | ||||||
| CacheBlend | ✓ | ✓ | ✗ | ||||||
| EPIC | ✓ | ✗ | ✗ | ||||||
| ProphetKV | ✓ | partly{sup}h` | ✓ | ||||||
| InfoFlow KV | ✓ | partly{sup}a` | ✓ | ||||||
| CacheTune | ✓ | ✓{sup}b` | ✗ | ||||||
| SurroKV | ✓{sup}c` | ✓{sup}d` | ✓{sup}c` | ||||||
| KVCOMM | –{sup}e` | ✓ | ✗ | ||||||
| BCR | partly{sup}f` | ✓{sup}g` | ✓ | ||||||
| SAFE-KV | ✗ | ✓ | ✗ |
Table 8. Models fine-tuned by Ma et al. [22] on the same RAG data, one with full attention (Tulu3-RAG) and one with block attention (Tulu3-Block-FT), both starting from Tulu3-SFT (Llama-3.1-8B). Each model is evaluated on its own with our prompts, which use the Tulu chat tags without BOS, as in its training data. Pos.-corr.: position-corrected reuse, with each chunk encoded after the prefix as in the rest of the paper. Block: each chunk encoded alone from position 0, as in the released inference code. Retention : , and , the retention under each distractor source. Terms of Equation (1) with 95% paired bootstrap intervals. Bold: the interval excludes zero. †Retention under RAND is zero, so the term cannot be positive; these intervals are not set in bold.
Models and prompts. For Table 8 we use the released Tulu3-RAG and Tulu3-Block-FT checkpoints of Ma et al. [22] and their inference procedure for block mode, in which each block is encoded alone from position 0 without BOS, its rotation is removed and reapplied at its final position, and the question attends to all blocks. Our instruction, chunks and question are wrapped in the Tulu tags without BOS, as in the models’ training data. The settings use the NQ-Open and Gorilla test sets and second question sets, with the distractor sources of § 2. None of our NQ-Open questions occurs in their training data (0 matches among the questions extracted from its RAG part). The two models are evaluated on their own and are not compared question by question.
Prompt format and number of documents. We also rebuild the prompts of the 111 NQ-Open questions of Tulu3-Block-FT in the format of its training data (their instruction, documents as “-Title:” entries, their question block), once with the same 30 documents and once with the 10-document version of the same prompt. With the 30-document and the 10-document prompt, full prefill answers 68% and 67% of these questions, respectively, and block mode answers 35% and 50%; the gaps are 0.32 [+0.21, +0.44] and 0.16 [+0.05, +0.27]. Both gaps are significant, and the 10-document gap is half the 30-document one: the loss persists when the prompt follows the training format, and it shrinks with fewer documents.
Additional Related Work
Repair methods. CacheBlend (Yao et al., [37]) recomputes the tokens whose KV deviates most, EPIC (Hu et al., [11]) recomputes the first tokens of each chunk, ProphetKV (Wang et al., [31]) selects tokens by question attention, and CacheTune (Li et al., [19]) selects them offline from a frequency-domain analysis of each chunk; Table 9 compares their design choices. CKV (Du et al., [8]) learns compressed caches that can be concatenated at new positions. Budgeted cache repair (BCR; Kang and Yoo, [16]) ranks cache rows by the attention of two draft tokens. On multi-agent GSM8K, its task accuracy does not differ significantly between repairing 64-token slices and repairing single rows (Table 9 gives its attention-output errors). In the diagnostic of SAFE-KV (Zhang, [39]) on 2Wiki-100, copying full-prefill states into the selected document alone keeps 38.0% of the full-prefill answers, because the documents after it keep states that were computed without it.
Other work
Cestola et al. [5] show that aggregate scores overstate how well reuse preserves full-prefill accuracy, and they evaluate on a meaningful subset of questions that full prefill answers, that need the context and whose answers are hard to guess; our follows this design (§ 2). KVCOMM (Ye et al., [38]) corrects the offsets of caches reused across agents, AgentKVShift (Pandey et al., [25]) corrects the retrieved memory chunks of an agent by an offset estimated from a few recomputed probe tokens, and CacheReforge (Cao et al., [4]) bounds the recomputation that recovers an updated adapter’s outputs from stale caches. StaleSpec (Anonymous, [3]) guarantees exact outputs after context edits through self-speculative verification. KVShareArena (Shi et al., [29]) benchmarks repairs across contexts and checkpoints, KV-Repurpose (Ikeda et al., [12]) reports that segment reuse struggles on cross-document reasoning, and decision margins have been used to model output preservation under KV compression (Anonymous, [2]). RelaxKV (Qi et al., [27]) chooses both the tokens to recompute and the context against which they are recomputed, and it restricts that context to a query-relevant part. Transition Overlap (Xu et al., [33]) reports that recomputing non-contiguous tokens can add discontinuities between neighbouring states, and it instead encodes each segment after a short temporary overlap whose KV is not stored. SemPIC (Xie et al., [32]) trains a LoRA writer that compiles document caches for the unchanged model. Latent-Remedy (Chen et al., [7]) attributes part of the loss of reuse across agents to the agent’s difficulty in reading the reused context under its own instructions, and it places a few learned latent tokens after the reused block. SurroKV, StaleSpec and the decision-margin model are concurrent anonymous submissions under review at ICLR 2027.
Additional Results
Similarity term on Gorilla. On Gorilla, retention under SIM is slightly higher than under RAND (by at most 0.06), which makes the similarity term significantly negative for two models. Because retention under RAND is zero on Gorilla, the three distractor sources cannot show an additional loss from similar API documents on this dataset.
Existing repairs. Table 11 reports existing repairs applied to the SIM condition. Encoding every chunk after 512 tokens of in-domain text matches or beats encoding it after the SurroKV Wikipedia surrogate of the same length in eight of twelve settings. The strongest repairs combine a surrogate prefix with 10% online recomputation (SurroKV’s full method, or the best token-level selector on top of the in-domain prefix). In each setting, the stronger of the two raises retention on LoCoMo close to full prefill (0.91–0.97) and on TriviaQA to 0.68–0.78, but leaves retention at 0.41–0.63 on NQ-Open and at 0.33–0.39 on Gorilla.
Allocation and detection details. On LoCoMo, Target+4 does not change retention significantly, and recomputing the retrieved session lowers retention for Qwen3-8B (−0.10 at 5%). For the detectors on Gorilla, the question is the user request alone (Appendix § 8). The Gorilla development settings contain only 11–16 questions that full prefill answers, so the estimates of detector accuracy on Gorilla are coarse. On questions where the retriever does not rank the target first, recomputing the retrieved chunk together with its four most similar chunks (Retrieved+4) gives higher retention than recomputing the retrieved chunk alone, significantly in seven of nine settings at 10%, but it never significantly beats the token-level selector (Table 19). At 5%, a gate that applies Retrieved+4 only when the retriever is confident amounts to choosing one method per domain (Table 25).
| Dataset | Model | Token 10% | Retrieved+4 | Retrieved | Retrieved+4 − Retrieved | Retrieved+4 − Token | |
| NQ-Open | Qwen3-8B | 66 | 0.24 | 0.26 | 0.14 | +0.12 [+0.03, +0.21] | +0.02 [−0.09, +0.12] |
| Llama-3.1-8B | 68 | 0.31 | 0.21 | 0.16 | +0.04 [−0.01, +0.12] | −0.10 [−0.22, +0.01] | |
| Mistral-7B | 50 | 0.34 | 0.36 | 0.20 | +0.16 [+0.02, +0.30] | +0.02 [−0.14, +0.16] | |
| Gorilla | Qwen3-8B | 42 | 0.14 | 0.12 | 0.00 | +0.12 [+0.02, +0.24] | −0.02 [−0.14, +0.10] |
| Llama-3.1-8B | 47 | 0.26 | 0.11 | 0.06 | +0.04 [−0.02, +0.13] | −0.15 [−0.30, −0.02] | |
| Mistral-7B | 57 | 0.30 | 0.23 | 0.09 | +0.14 [+0.04, +0.25] | −0.07 [−0.19, +0.04] | |
| TriviaQA | Qwen3-8B | 54 | 0.41 | 0.35 | 0.24 | +0.11 [+0.02, +0.22] | −0.06 [−0.20, +0.09] |
| Llama-3.1-8B | 51 | 0.45 | 0.27 | 0.14 | +0.14 [+0.06, +0.24] | −0.18 [−0.33, +0.00] | |
| Mistral-7B | 44 | 0.57 | 0.43 | 0.23 | +0.20 [+0.07, +0.34] | −0.14 [−0.32, +0.05] |
Table 19. Questions on which the retriever misses the target chunk (10% budget): retention of the token-level selector, of Retrieved+4 and of Retrieved (the retrieved chunk alone), with paired differences.
| Dataset | Model | 5% budget: Retrieved +4 (%) | 5% budget: Gated − Token | 5% budget: Gated − Retrieved+4 | 10% budget: Retrieved +4 (%) | 10% budget: Gated − Token | 10% budget: Gated − Retrieved+4 | |
| LoCoMo | Qwen3-8B | 105 | 0 | +0.00<br>[+0.00, +0.00] | +0.10<br>[+0.04, +0.20] | 1 | −0.01<br>[−0.03, +0.00] | +0.08<br>[+0.01, +0.16] |
| Llama-3.1-8B | 102 | 0 | +0.00<br>[+0.00, +0.00] | +0.05<br>[−0.02, +0.13] | 1 | +0.00<br>[+0.00, +0.00] | +0.05<br>[−0.04, +0.14] | |
| Mistral-7B | 36 | 0 | +0.00<br>[+0.00, +0.00] | −0.03<br>[−0.19, +0.12] | 3 | +0.00<br>[+0.00, +0.00] | −0.08<br>[−0.24, +0.06] | |
| NQ-Open | Qwen3-8B | 146 | 79 | +0.17<br>[+0.11, +0.24] | −0.03<br>[−0.08, +0.00] | 84 | +0.23<br>[+0.16, +0.32] | −0.01<br>[−0.03, +0.01] |
| Llama-3.1-8B | 128 | 77 | +0.11<br>[+0.02, +0.20] | −0.02<br>[−0.04, +0.00] | 84 | −0.08<br>[−0.16, +0.01] | −0.01<br>[−0.05, +0.03] | |
| Mistral-7B | 101 | 79 | +0.13<br>[+0.03, +0.23] | −0.02<br>[−0.08, +0.03] | 84 | +0.12<br>[+0.02, +0.23] | −0.02<br>[−0.07, +0.03] | |
| Gorilla | Qwen3-8B | 89 | 93 | +0.07<br>[+0.01, +0.12] | −0.01<br>[−0.03, +0.00] | 25 | −0.01<br>[−0.06, +0.03] | −0.01<br>[−0.09, +0.07] |
| Llama-3.1-8B | 100 | 91 | +0.00<br>[−0.08, +0.08] | −0.01<br>[−0.03, +0.00] | 23 | −0.07<br>[−0.13, −0.03] | +0.10<br>[+0.02, +0.18] | |
| Mistral-7B | 108 | 93 | +0.20<br>[+0.10, +0.31] | −0.01<br>[−0.03, +0.00] | 21 | +0.08<br>[+0.02, +0.15] | −0.01<br>[−0.08, +0.07] | |
| TriviaQA | Qwen3-8B | 151 | 87 | +0.15<br>[+0.07, +0.23] | −0.02<br>[−0.05, +0.01] | 48 | −0.03<br>[−0.09, +0.03] | −0.05<br>[−0.11, +0.01] |
| Llama-3.1-8B | 127 | 87 | +0.01<br>[−0.08, +0.09] | −0.01<br>[−0.05, +0.02] | 47 | −0.06<br>[−0.13, −0.01] | +0.00<br>[−0.07, +0.07] | |
| Mistral-7B | 117 | 91 | +0.09<br>[+0.01, +0.16] | −0.03<br>[−0.07, +0.01] | 54 | +0.03<br>[−0.03, +0.10] | −0.03<br>[−0.11, +0.03] |
Table 25. A gate that recomputes Retrieved+4 when the cosine similarity of the retriever’s top-ranked chunk exceeds that of its second-ranked chunk by at least a threshold, and uses the token-level selector otherwise. The questions are split into two halves (by conversation for LoCoMo). A threshold fitted on each half, pooled over the three models of a dataset, is applied to the other half, so every question is evaluated once. Retrieved+4 (%): share of questions sent to Retrieved+4; differences are paired, with 95% bootstrap intervals that keep the gate’s decisions fixed.
Universal term and KV replacement. Replacing every distractor raises retention for Qwen3-8B on NQ-Open from 0.15 to 0.71. The universal term is largest for Gorilla with Mistral-7B (0.92) and on NQ-Open (0.43–0.72); in the Gorilla setting, 93% of the wrong answers under OTHER describe the off-domain prose instead of naming an API. Replacing a single distractor, the most similar one or a random one, changes retention significantly in two of 24 cases (by at most 0.06; Table 13 and Table 14), and most of the recoverable loss is restored only by replacing many distractors at once. On LoCoMo (dialogue memory), replacing all distractors raises retention significantly only for Qwen3-8B, and it lowers retention for Mistral-7B (−0.08, [−0.19, −0.02]).
| Dataset | Model | Target KV | All distr. | 1 similar | 1 random | All distr. − Target | 1 similar − Target | ||
| LoCoMo | Qwen3-8B | 105 | 0.53 | 0.61 | 0.79 | 0.54 | 0.58 | +0.18 [+0.07, +0.29] | −0.07 [−0.11, −0.03] |
| Llama-3.1-8B | 102 | 0.81 | 0.71 | 0.82 | 0.79 | 0.82 | +0.12 [+0.03, +0.20] | +0.09 [+0.00, +0.17] | |
| Mistral-7B | 36 | 0.72 | 0.61 | 0.64 | 0.78 | 0.72 | +0.03 [−0.10, +0.18] | +0.17 [+0.05, +0.31] | |
| NQ-Open | Qwen3-8B | 146 | 0.15 | 0.18 | 0.71 | 0.15 | 0.14 | +0.53 [+0.44, +0.62] | −0.03 [−0.10, +0.03] |
| Llama-3.1-8B | 128 | 0.16 | 0.23 | 0.70 | 0.18 | 0.16 | +0.46 [+0.36, +0.56] | −0.05 [−0.13, +0.02] | |
| Mistral-7B | 101 | 0.18 | 0.46 | 0.66 | 0.17 | 0.17 | +0.21 [+0.08, +0.35] | −0.29 [−0.39, −0.19] | |
| Gorilla | Qwen3-8B | 89 | 0.00 | 0.02 | 0.43 | 0.01 | 0.02 | +0.40 [+0.29, +0.52] | −0.01 [−0.03, +0.00] |
| Llama-3.1-8B | 100 | 0.04 | 0.17 | 0.43 | 0.03 | 0.03 | +0.26 [+0.15, +0.38] | −0.14 [−0.21, −0.07] | |
| Mistral-7B | 108 | 0.06 | 0.32 | 0.53 | 0.06 | 0.06 | +0.20 [+0.08, +0.32] | −0.27 [−0.35, −0.19] | |
| TriviaQA | Qwen3-8B | 151 | 0.27 | 0.31 | 0.80 | 0.25 | 0.30 | +0.49 [+0.40, +0.58] | −0.06 [−0.13, +0.01] |
| Llama-3.1-8B | 127 | 0.20 | 0.33 | 0.74 | 0.23 | 0.24 | +0.41 [+0.31, +0.51] | −0.10 [−0.17, −0.03] | |
| Mistral-7B | 117 | 0.25 | 0.55 | 0.71 | 0.24 | 0.26 | +0.16 [+0.03, +0.29] | −0.31 [−0.39, −0.21] |
Table 13. KV replacement under SIM. Columns give retention with no replacement (), after replacing the target chunk (Target KV), all distractor chunks (All distr.), the single distractor most similar to the target (1 similar) and one random distractor (1 random) with their full-prefill KV. The last two columns are paired differences. The random distractor coincides with the most similar one on 0–6% of questions.
| Dataset | Model | Target KV | All distr. | 1 similar | 1 random | |
| LoCoMo | Qwen3-8B | 105 | +0.08 [+0.03, +0.12] | +0.26 [+0.17, +0.35] | +0.01 [+0.00, +0.03] | +0.05 [+0.01, +0.08] |
| Llama-3.1-8B | 102 | −0.11 [−0.16, −0.04] | +0.01 [−0.07, +0.07] | −0.02 [−0.05, +0.02] | +0.01 [−0.02, +0.04] | |
| Mistral-7B | 36 | −0.11 [−0.26, +0.00] | −0.08 [−0.19, −0.02] | +0.06 [+0.00, +0.12] | +0.00 [−0.09, +0.07] | |
| NQ-Open | Qwen3-8B | 146 | +0.03 [−0.03, +0.10] | +0.56 [+0.48, +0.64] | +0.00 [−0.02, +0.02] | −0.01 [−0.03, +0.01] |
| Llama-3.1-8B | 128 | +0.07 [+0.00, +0.15] | +0.53 [+0.44, +0.62] | +0.02 [−0.01, +0.05] | −0.01 [−0.03, +0.02] | |
| Mistral-7B | 101 | +0.28 [+0.18, +0.38] | +0.49 [+0.38, +0.59] | −0.01 [−0.06, +0.04] | −0.01 [−0.05, +0.02] | |
| Gorilla | Qwen3-8B | 89 | +0.02 [+0.00, +0.06] | +0.43 [+0.33, +0.53] | +0.01 [+0.00, +0.03] | +0.02 [+0.00, +0.06] |
| Llama-3.1-8B | 100 | +0.13 [+0.06, +0.20] | +0.39 [+0.29, +0.49] | −0.01 [−0.03, +0.00] | −0.01 [−0.04, +0.02] | |
| Mistral-7B | 108 | +0.27 [+0.19, +0.35] | +0.47 [+0.37, +0.57] | +0.00 [+0.00, +0.00] | +0.01 [+0.00, +0.03] | |
| TriviaQA | Qwen3-8B | 151 | +0.04 [−0.04, +0.12] | +0.53 [+0.44, +0.62] | −0.02 [−0.05, +0.01] | +0.03 [−0.01, +0.06] |
| Llama-3.1-8B | 127 | +0.13 [+0.06, +0.19] | +0.54 [+0.43, +0.63] | +0.02 [−0.02, +0.06] | +0.04 [+0.01, +0.08] | |
| Mistral-7B | 117 | +0.30 [+0.21, +0.39] | +0.46 [+0.36, +0.56] | −0.01 [−0.04, +0.02] | +0.01 [−0.03, +0.05] |
Table 14. KV replacement under SIM: change in retention against no replacement (), with 95% paired bootstrap intervals.
Competing answers. RAND passages also contain many names, so the share of reuse answers that name a distractor entity under RAND serves as the baseline, rather than zero. Under RAND the model also abstains more often: abstentions make up 10–73% of its wrong answers, against 0–26% under SIM. When the comparison is restricted to wrong answers that are not abstentions, the difference remains positive in all six settings and is significant in three. Table 15 lists, for each model and open-domain QA dataset, one case like that of Figure 1, chosen by a fixed rule. Table 16 gives the share of all questions whose reuse answer names an entity that occurs in some distractor chunk but not in the target chunk, the share of abstentions among wrong answers, and the same comparison restricted to wrong answers that are not abstentions. Entities are capitalised spans (allowing short connectors such as “of” and “the”) and numbers of at least three digits, excluding spans that occur in the question; spans are matched as token sequences. A broader rule that counts any 1–5-word phrase gives larger differences in the same direction (last column). Abstentions are counted
| Model, data | Question | Gold answer | Reuse answer | Similar passage (title: sentence) |
| Qwen3-8B, NQ-Open | Who was the actor that played ben stone on law and order | Michael Moriarty | Philip Winchester | Peter Stone (Chicago and Law & Order character): . . . saw it develop and we saw the actor, it made sense. Peter Stone (Chicago and Law & Order character) Peter Stone is a fictional character portrayed by Philip Winchester. |
| Llama-3.1-8B, NQ-Open | Who won the most medals in the 1924 winter olympics | Norway | Ville Ritola | Ville Ritola: . . . win the 3000 m team race. |
| Mistral-7B, NQ-Open | Dendrites and cell bodies are components of what type of matter found in the brain | gray | Neuronal matter | Neuron: Other structural proteins that are important for neuronal function are actin and the tubulin of microtubules. |
| Qwen3-8B, TriviaQA | Liberty Bell was the theme tune for which British TV show? | Monty Python’s Flying Circus | EastEnders | EastEnders theme tune: . . . the theme is still popular and in the national psyche. The theme tune was used in the 2012 Summer Olympics opening ceremony. |
| Llama-3.1-8B, TriviaQA | What Tory MP was found to have claimed unsuccessfully on his parliamentary expenses for a floating duck island? | Sir Peter Viggers | Gordon Brown | United Kingdom parliamentary expenses scandal: Among those who had to repay expenses were Prime Minister Gordon Brown who claimed £12,415 for cleaning and gardening costs and Liberal Democrat leader Nick Clegg who was. . . |
| Mistral-7B, TriviaQA | What was the character name of Commander Shore’s daughter in the 60’s series Stingray (voiced by Lois Maxwell) | Atlanta | Lois Lane | Lois Lane: . . . before was one of Luthor’s robotic duplicates. |
Table 15. For each model and open-domain QA dataset, the first question in (taking questions in identifier order, and the test set before the second set) that meets two conditions: its reuse answer under SIM is a short wrong answer (at most six words, with no reference to a document) that names an entity occurring in a similar passage but not in the target passage, and its reuse answer under RAND is correct. Long sentences are cut around that entity. Questions keep the capitalisation and punctuation of the datasets.
separately, and Table 18 compares their share with that among the errors of full prefill. On Gorilla, a wrong output counts as naming a distractor API when it contains the api_name of a document other than the target. Under SIM and RAND this holds for 45–91% of wrong reuse outputs, and the remaining wrong outputs either cite a document number or name no API; under OTHER, none of the wrong outputs names a distractor API.
Competing-entity control. For Table 17, we edit only the bodies of the SIM distractor chunks, on the questions of NQ-Open and TriviaQA; the target chunk, the chunk order, the headers and the question are the same as in the main experiments. Named entities are found with spaCy is contained in a gold answer. The answer’s type (en_core_web_lg) and grouped into persons, places, organisations, groups, works, events, languages, dates and quantities. An entity is a candidate when its text occurs neither in the target chunk nor in the question and neither contains nor
| Dataset | Model | SIM Reuse | SIM Full | Competing removed Reuse | Competing removed Full | Others removed Reuse | Others removed Full | Gap closed Competing | Gap closed Others | Competing − Others | |
| NQ-Open | Qwen3-8B | 131 | 0.13 | 0.94 | 0.22 | 0.94 | 0.16 | +0.15 [+0.08, +0.22] | +0.09 [+0.04, +0.15] | +0.06 [−0.01, +0.14] | |
| Llama-3.1-8B | 109 | 0.15 | 0.98 | 0.34 | 0.97 | 0.11 | +0.21 [+0.12, +0.31] | −0.01 [−0.06, +0.04] | +0.22 [+0.13, +0.31] | ||
| Mistral-7B | 86 | 0.19 | 0.93 | 0.22 | 0.95 | 0.17 | +0.10 [+0.01, +0.20] | +0.03 [−0.05, +0.12] | +0.07 [−0.03, +0.17] | ||
| TriviaQA | Qwen3-8B | 122 | 0.22 | 0.99 | 0.28 | 0.97 | 0.23 | +0.07 [+0.00, +0.13] | +0.04 [−0.01, +0.10] | +0.02 [−0.04, +0.10] | |
| Llama-3.1-8B | 96 | 0.16 | 0.94 | 0.24 | 0.99 | 0.17 | +0.15 [+0.06, +0.24] | +0.02 [−0.02, +0.06] | +0.12 [+0.04, +0.22] | ||
| Mistral-7B | 88 | 0.19 | 0.95 | 0.23 | 0.97 | 0.16 | +0.08 [+0.00, +0.16] | +0.00 [−0.06, +0.06] | +0.08 [+0.01, +0.15] |
Table 17. Competing-entity control on (questions with an answer type). In the SIM distractors, the named entities of the answer’s type (Competing removed) or the same number of entities of other types (Others removed) are replaced by generic descriptors such as “a person”, with the target chunk, the order and the headers unchanged. Reuse and Full: accuracy of reuse and of full prefill. SIM Reuse: accuracy of reuse on the unedited prompt, where full prefill answers every question. Gap closed: how much the gap between full prefill and reuse decreases relative to the unedited prompt. Differences are paired, with 95% bootstrap intervals.
the group of the gold answer’s mention in the target chunk or, if there is no such mention, is inferred from the question word; the 138 of 770 questions without a type are left out. One edit replaces every candidate of the answer’s type; the other replaces the same number of candidates of other types, drawn from the same chunk where possible. Each replaced entity is rewritten as a generic descriptor of its group (for instance “a person” or “that year”), and the descriptor is chosen per model so that the token count of each chunk stays close to the original. On a sample of 40 questions checked by hand, the answer’s type is right for 34, and 178 of 204 replaced entities have the right group on those questions. The run also encodes the unedited prompts of the questions with an answer type among the first ten of each setting, and all 49 of them reproduce the outputs of the main experiments. When the entities of the answer’s type in the SIM distractors are replaced, reuse answers 22–34% of the questions that have such a type instead of 13–22%, while full prefill still answers 93–99%. The gap narrows significantly in four of the six settings (three after Holm’s correction), but in only one when the same number of entities of other types are replaced. Replacing the competing entities lowers the share of wrong reuse answers (abstentions excluded) that name a distractor entity from 64–87% to 48–67%, against 62–83% when other entities are replaced. The difference between the two edits is positive everywhere but significant in only three (two after Holm’s correction), so this
control does not clearly separate the effect of the competing entities from that of editing. Leaving out the questions for which too few entities of other types were available makes the difference significant in four, and leaving out those whose token count changes by more than 1% keeps it significant in three.
Allocation controls. Table 22 adds two controls. Recomputing the four most similar chunks without the target is significantly worse than the token-level selector in eleven of twelve settings. Replacing these four chunks in Target+4 with four random chunks changes retention significantly in two settings, both times in favour of the random chunks. At 5%, Target+4 shares the budget among five chunks, so the target is no longer guaranteed to be recomputed in full. On Gorilla, Target+4 at 10% changes retention relative to the token-level selector by +0.23 for Mistral-7B and by −0.11 for Llama-3.1-8B, so we report it separately for each model.
| Dataset | Model | Target+4 − Target (5%) | Target+4 − Target (10%) | Target+4 − random (10%) | Similar − Token (10%) | |
| LoCoMo | Qwen3-8B | 105 | −0.02 [−0.06, +0.04] | +0.00 [−0.05, +0.05] | −0.01 [−0.07, +0.03] | −0.14 [−0.21, −0.07] |
| Llama-3.1-8B | 102 | +0.00 [−0.05, +0.05] | −0.02 [−0.06, +0.02] | −0.05 [−0.08, −0.02] | −0.13 [−0.21, −0.04] | |
| Mistral-7B | 36 | −0.06 [−0.17, +0.05] | +0.00 [−0.13, +0.10] | +0.00 [−0.13, +0.10] | −0.03 [−0.13, +0.06] | |
| NQ-Open | Qwen3-8B | 146 | −0.05 [−0.12, +0.02] | +0.05 [+0.00, +0.11] | +0.05 [−0.01, +0.11] | −0.09 [−0.14, −0.03] |
| Llama-3.1-8B | 128 | +0.03 [−0.02, +0.09] | +0.02 [−0.03, +0.08] | −0.03 [−0.10, +0.04] | −0.20 [−0.29, −0.12] | |
| Mistral-7B | 101 | −0.10 [−0.19, −0.01] | +0.00 [−0.07, +0.07] | −0.01 [−0.09, +0.07] | −0.16 [−0.26, −0.06] | |
| Gorilla | Qwen3-8B | 89 | −0.09 [−0.17, −0.01] | −0.01 [−0.09, +0.07] | −0.04 [−0.12, +0.03] | −0.16 [−0.24, −0.08] |
| Llama-3.1-8B | 100 | −0.04 [−0.10, +0.02] | −0.02 [−0.08, +0.04] | −0.04 [−0.12, +0.04] | −0.29 [−0.39, −0.20] | |
| Mistral-7B | 108 | −0.15 [−0.23, −0.07] | −0.06 [−0.16, +0.02] | +0.07 [−0.01, +0.17] | −0.24 [−0.33, −0.16] | |
| TriviaQA | Qwen3-8B | 151 | +0.03 [−0.03, +0.09] | +0.05 [+0.00, +0.10] | +0.00 [−0.05, +0.05] | −0.23 [−0.32, −0.15] |
| Llama-3.1-8B | 127 | −0.02 [−0.08, +0.05] | +0.02 [−0.04, +0.09] | −0.02 [−0.08, +0.05] | −0.32 [−0.41, −0.24] | |
| Mistral-7B | 117 | −0.05 [−0.12, +0.02] | −0.03 [−0.09, +0.03] | −0.07 [−0.13, −0.01] | −0.31 [−0.39, −0.21] |
Table 22. Controls for the chunk-level allocation (paired differences in retention on ). Target+4 − Target: the four most similar chunks added to the target, at 5% and 10%. Target+4 − random: the target with its four most similar chunks against the target with four random chunks (10%). Similar − Token: the four most similar chunks without the target against the token-level selector (10%).
The first slot. By Proposition 1, recomputing a target in the first slot reproduces the KV that it already has, and the outputs of all 46 such questions are unchanged. Figure 5 shows the change in retention from recomputing the target when the target is in a later slot.
Moving the chosen chunk last. We move either the retriever’s top chunk (chosen in the original order) or the target to the last slot. All headers are renumbered by position, every chunk is encoded alone as before, and the moved chunk is recomputed under the 5% budget with the setting’s rule. Table 27 gives the results for the retrieved chunk, and the results for the moved target are given below. Table 28 spends the same 5% budget with the setting’s token-level rule on the same reordered prompts. Recomputing the moved chunk uses 48–
| Dataset | Model | Hit (%) | Token 5% | In place − Token | Moved − Token | Moved only, no recomp. | Moved − in place | Moved − Token (hits) | Moved − Token (misses) | |
| NQ-Open | Qwen3-8B | 146 | 55 | 0.16 | +0.22 [+0.14, +0.29] | +0.40 [+0.31, +0.49] | +0.00 [−0.02, +0.02] | +0.18 [+0.11, +0.25] | +0.75 [+0.65, +0.85] | −0.03 [−0.12, +0.06] |
| Llama-3.1-8B | 128 | 47 | 0.21 | +0.08 [+0.00, +0.16] | +0.25 [+0.15, +0.35] | +0.04 [−0.01, +0.09] | +0.17 [+0.10, +0.24] | +0.63 [+0.50, +0.75] | −0.09 [−0.18, +0.00] | |
| Mistral-7B | 101 | 50 | 0.28 | +0.14 [+0.02, +0.25] | +0.27 [+0.14, +0.39] | +0.09 [+0.02, +0.16] | +0.13 [+0.03, +0.22] | +0.65 [+0.49, +0.78] | −0.12 [−0.26, +0.00] | |
| Gorilla | Qwen3-8B | 89 | 53 | 0.06 | +0.09 [+0.00, +0.18] | +0.21 [+0.12, +0.30] | +0.02 [+0.00, +0.06] | +0.12 [+0.03, +0.22] | +0.43 [+0.28, +0.57] | −0.02 [−0.07, +0.00] |
| Llama-3.1-8B | 100 | 53 | 0.12 | +0.02 [−0.07, +0.10] | +0.15 [+0.06, +0.24] | +0.00 [−0.04, +0.04] | +0.13 [+0.04, +0.22] | +0.34 [+0.23, +0.47] | −0.06 [−0.17, +0.02] | |
| Mistral-7B | 108 | 47 | 0.10 | +0.19 [+0.09, +0.30] | +0.32 [+0.21, +0.44] | +0.01 [−0.03, +0.05] | +0.13 [+0.04, +0.22] | +0.84 [+0.75, +0.94] | −0.14 [−0.23, −0.05] | |
| TriviaQA | Qwen3-8B | 151 | 64 | 0.32 | +0.14 [+0.06, +0.22] | +0.28 [+0.19, +0.36] | −0.05 [−0.09, −0.01] | +0.14 [+0.07, +0.22] | +0.46 [+0.36, +0.57] | −0.06 [−0.15, +0.04] |
| Llama-3.1-8B | 127 | 60 | 0.46 | −0.02 [−0.11, +0.08] | +0.14 [+0.04, +0.24] | +0.13 [+0.06, +0.20] | +0.16 [+0.09, +0.23] | +0.39 [+0.28, +0.51] | −0.24 [−0.37, −0.10] | |
| Mistral-7B | 117 | 62 | 0.44 | +0.10 [+0.02, +0.20] | +0.17 [+0.07, +0.27] | +0.02 [−0.04, +0.09] | +0.07 [+0.01, +0.14] | +0.42 [+0.32, +0.53] | −0.25 [−0.39, −0.11] |
Table 27. Recomputing a chunk at the 5% budget after moving it to the last slot before the question (headers renumbered, chunks encoded alone as before). In place: the chunk recomputed where the retriever found it (Retrieved in Table 1). Moved only: the reordered cache without recomputation, against the unrepaired cache. Hits and misses: questions on which the retriever does or does not rank the target first. Paired differences with 95% bootstrap intervals.
| Dataset | Model | Token original | Token moved | Chunk moved | Token moved − Token | All | Hits | Misses | |
| NQ-Open | Qwen3-8B | 146 | 0.16 | 0.27 | 0.56 | +0.11 [+0.05, +0.17] | +0.29 [+0.21, +0.37] | +0.56 [+0.45, +0.68] | −0.05 [−0.12, +0.03] |
| Llama-3.1-8B | 128 | 0.21 | 0.46 | 0.46 | +0.25 [+0.16, +0.34] | +0.00 [−0.09, +0.09] | +0.08 [−0.05, +0.22] | −0.07 [−0.18, +0.03] | |
| Mistral-7B | 101 | 0.28 | 0.51 | 0.54 | +0.24 [+0.13, +0.34] | +0.03 [−0.08, +0.13] | +0.22 [+0.08, +0.35] | −0.16 [−0.30, −0.02] | |
| Gorilla | Qwen3-8B | 89 | 0.06 | 0.06 | 0.27 | +0.00 [−0.03, +0.03] | +0.21 [+0.12, +0.31] | +0.45 [+0.30, +0.60] | −0.05 [−0.12, +0.00] |
| Llama-3.1-8B | 100 | 0.12 | 0.34 | 0.27 | +0.22 [+0.12, +0.32] | −0.02 [−0.16, +0.02] | −0.09 [−0.25, +0.06] | −0.04 [−0.13, +0.04] | |
| Mistral-7B | 108 | 0.10 | 0.12 | 0.43 | +0.02 [−0.03, +0.07] | +0.31 [+0.19, +0.42] | +0.82 [+0.73, +0.92] | −0.16 [−0.26, −0.07] | |
| TriviaQA | Qwen3-8B | 151 | 0.32 | 0.52 | 0.60 | +0.19 [+0.12, +0.26] | +0.09 [+0.02, +0.15] | +0.15 [+0.07, +0.25] | −0.04 [−0.13, +0.07] |
| Llama-3.1-8B | 127 | 0.46 | 0.61 | 0.60 | +0.16 [+0.06, +0.24] | −0.02 [−0.09, +0.06] | +0.09 [+0.00, +0.18] | −0.18 [−0.29, −0.08] | |
| Mistral-7B | 117 | 0.44 | 0.57 | 0.62 | +0.13 [+0.04, +0.21] | +0.04 [−0.03, +0.12] | +0.16 [+0.08, +0.26] | −0.16 [−0.30, −0.05] |
Table 28. Token-level selection after the move. On the prompt with the retriever’s top chunk moved to the last slot (Table 27), Token moved spends the 5% budget with the setting’s token-level rule over all chunks, and Chunk moved recomputes the moved chunk. Token: the token-level selector on the original prompt. Paired differences with 95% bootstrap intervals.
67% as many tokens as this rule, as one contiguous block after the reused chunks. Because our headers carry position numbers, they are renumbered in the reordered prompts and every chunk is encoded alone again. With headers that carry no position, the stored caches could be reused in the new order without re-encoding.
| Dataset | Model | Target − Token | Target +4 − Token | Retrieved − Token | Retrieved +4 − Token | |
| LoCoMo | Qwen3-8B | 105 | −0.04 [−0.13, +0.03] | −0.06 [−0.15, +0.01] | −0.10 [−0.19, −0.03] | −0.10 [−0.20, −0.04] |
| Llama-3.1-8B | 102 | +0.01 [−0.06, +0.07] | +0.01 [−0.06, +0.08] | −0.01 [−0.09, +0.07] | −0.05 [−0.13, +0.02] | |
| Mistral-7B | 36 | +0.03 [−0.09, +0.15] | −0.03 [−0.19, +0.13] | −0.03 [−0.14, +0.10] | +0.03 [−0.12, +0.19] | |
| NQ-Open | Qwen3-8B | 146 | +0.35 [+0.27, +0.43] | +0.29 [+0.22, +0.38] | +0.22 [+0.14, +0.29] | +0.21 [+0.13, +0.28] |
| Llama-3.1-8B | 128 | +0.15 [+0.06, +0.23] | +0.18 [+0.09, +0.27] | +0.08 [+0.00, +0.16] | +0.12 [+0.04, +0.21] | |
| Mistral-7B | 101 | +0.31 [+0.18, +0.43] | +0.21 [+0.09, +0.34] | +0.14 [+0.02, +0.25] | +0.15 [+0.04, +0.26] | |
| Gorilla | Qwen3-8B | 89 | +0.17 [+0.08, +0.26] | +0.08 [+0.02, +0.13] | +0.09 [+0.00, +0.18] | +0.08 [+0.02, +0.15] |
| Llama-3.1-8B | 100 | +0.10 [+0.01, +0.19] | +0.06 [−0.02, +0.14] | +0.02 [−0.07, +0.10] | +0.01 [−0.07, +0.09] | |
| Mistral-7B | 108 | +0.49 [+0.38, +0.59] | +0.34 [+0.24, +0.44] | +0.19 [+0.09, +0.30] | +0.21 [+0.11, +0.31] | |
| TriviaQA | Qwen3-8B | 151 | +0.20 [+0.11, +0.29] | +0.23 [+0.14, +0.32] | +0.14 [+0.06, +0.22] | +0.17 [+0.09, +0.25] |
| Llama-3.1-8B | 127 | +0.11 [+0.02, +0.20] | +0.09 [+0.01, +0.19] | −0.02 [−0.11, +0.08] | +0.02 [−0.08, +0.11] | |
| Mistral-7B | 117 | +0.23 [+0.15, +0.32] | +0.18 [+0.10, +0.26] | +0.10 [+0.02, +0.20] | +0.11 [+0.03, +0.20] |
Table 20. Paired gains over the token-level selector at a 5% budget, with 95% bootstrap intervals (point estimates in Table 1).
| Dataset | Model | Target − Token | Target +4 − Token | Retrieved − Token | Retrieved +4 − Token | |
| LoCoMo | Qwen3-8B | 105 | −0.04 [−0.14, +0.06] | −0.04 [−0.15, +0.05] | −0.10 [−0.20, −0.01] | −0.09 [−0.18, −0.01] |
| Llama-3.1-8B | 102 | +0.00 [−0.06, +0.06] | −0.02 [−0.08, +0.04] | −0.02 [−0.08, +0.05] | −0.05 [−0.14, +0.04] | |
| Mistral-7B | 36 | +0.00 [−0.11, +0.12] | +0.00 [−0.18, +0.16] | +0.00 [−0.11, +0.11] | +0.08 [−0.06, +0.24] | |
| NQ-Open | Qwen3-8B | 146 | +0.27 [+0.20, +0.36] | +0.33 [+0.25, +0.41] | +0.14 [+0.07, +0.23] | +0.24 [+0.16, +0.32] |
| Llama-3.1-8B | 128 | −0.02 [−0.12, +0.06] +0.00 [−0.09, +0.09] | −0.09 [−0.19, −0.01] | −0.07 [−0.17, +0.02] | ||
| Mistral-7B | 101 | +0.25 [+0.14, +0.36] | +0.25 [+0.14, +0.37] | +0.08 [−0.03, +0.20] | +0.14 [+0.02, +0.26] | |
| Gorilla | Qwen3-8B | 89 | +0.03 [−0.06, +0.13] | +0.02 [−0.06, +0.11] | −0.04 [−0.13, +0.06] | +0.00 [−0.09, +0.09] |
| Llama-3.1-8B | 100 | −0.09 [−0.19, +0.00] | −0.11 [−0.20, −0.02] | −0.17 [−0.27, −0.08] | −0.17 [−0.27, −0.08] | |
| Mistral-7B | 108 | +0.30 [+0.19, +0.41] | +0.23 [+0.12, +0.35] | +0.00 [−0.11, +0.11] | +0.09 [−0.01, +0.19] | |
| TriviaQA | Qwen3-8B | 151 | +0.03 [−0.05, +0.11] | +0.08 [+0.00, +0.16] | −0.03 [−0.11, +0.05] | +0.03 [−0.05, +0.11] |
| Llama-3.1-8B | 127 | +0.02 [−0.08, +0.11] | +0.04 [−0.05, +0.13] | −0.11 [−0.20, −0.02] | −0.06 [−0.16, +0.03] | |
| Mistral-7B | 117 | +0.15 [+0.06, +0.24] | +0.13 [+0.04, +0.21] | +0.03 [−0.08, +0.13] | +0.07 [−0.03, +0.16] |
Table 21. Paired gains over the token-level selector at a 10% budget, with 95% bootstrap intervals (point estimates in Table 1).
Whether the moved chunk beats the selector on the reordered prompt depends on whether the retriever found the target. On hits, the questions where the retriever ranks the target first, recomputing the moved chunk gives significantly higher retention than the selector in six of the nine settings. On misses it gives lower retention everywhere, significantly so in four. On Gorilla with Qwen3-8B and Mistral-7B, the setting’s rule is EPIC, whose selection does not depend on the order, so the gains of the moved chunk there are measured against EPIC. With ProphetKV as the token-level rule instead, the move raises the retention of the selector by +0.20 and +0.16 over ProphetKV on the original prompt, and the recomputed moved chunk beats this selector on the reordered prompt with Mistral-7B (+0.15 [ +0.06, +0.23]) but not with Qwen3-8B (−0.02 [−0.11, +0.08]).
The moved target gains 0.18–0.41 over the target recomputed in place. Without recomputation, a late slot can help by itself ([20]). Moving the target without recomputing it raises retention significantly, by 0.09 to 0.17, only with Llama-3.1-8B and Mistral-7B on NQ-Open and with Llama-3.1-8B on TriviaQA. Moving the retrieved chunk without recomputing it changes retention by −0.05 to +0.13. On hits this is a significant gain in only two settings, and on misses never a significant loss.
Reordering also changes full prefill, which answers 79–98% of on the reordered prompts, while retention is still measured against full prefill in the original order. Restricting the comparison to the questions that full prefill also answers in the moved order (83–98% of ) gives the same counts as on all of : the moved chunk significantly beats the selector in nine settings (by 0.12 to 0.40), the move significantly raises the retention of the selector in seven, and on the reordered prompt the moved chunk is significantly better than selected tokens in four settings and significantly worse in none.
InfoFlow-style reordering.
Teng et al. [30] also move chunks toward the question. We re-implemented the reordering procedure of their released code (commit 725b75c) in our engine (Table 29). It scores each context token by the L2 norm of the question tokens’ head-averaged attention on it in layers 22–25, the default of their run script for Qwen and Llama models, and we use the same layers for Mistral-7B. The first scoring pass reads the cache with chunk-local positions, as their paper (its HL–TP setting) and code do, so that no chunk is favoured for its current slot. The procedure then orders the chunks by the fraction of their tokens that are among the top-scoring tokens and places the chunk with the highest fraction last. Finally, it scores the tokens again on the reordered prompt and recomputes the top-scoring tokens. Their paper describes the score as a column sum of prompt attention; we follow the code. We use our 5% budget (their default is 15%) and our question suffix as the query. The comparison matches the token budget, not latency.
| Dataset | Model | Retrieved | InfoFlow | Token 5% | InfoFlow 5% | InfoFlow 15% | Chunk moved 5% | at 5% | at 15% | InfoFlow 5% − Token InfoFlow | InfoFlow chunk − Chunk moved | |
| NQ-Open | Qwen3-8B | 146 | 55 | 52 | 0.16 | 0.40 | 0.66 | 0.56 | +0.16 [+0.07, +0.26] | −0.10 [−0.20, +0.00] | +0.23 [+0.16, +0.32] | −0.01 [−0.11, +0.08] |
| Llama-3.1-8B | 128 | 47 | 39 | 0.21 | 0.40 | 0.59 | 0.46 | +0.06 [−0.04, +0.16] | −0.12 [−0.22, −0.02] | +0.19 [+0.10, +0.27] | −0.08 [−0.17, +0.02] | |
| Mistral-7B | 101 | 50 | 43 | 0.28 | 0.38 | 0.53 | 0.54 | +0.17 [+0.02, +0.31] | +0.01 [−0.13, +0.16] | +0.10 [+0.00, +0.20] | −0.07 [−0.20, +0.07] | |
| Gorilla | Qwen3-8B | 89 | 53 | 24 | 0.06 | 0.10 | 0.28 | 0.27 | +0.17 [+0.07, +0.27] | −0.01 [−0.11, +0.09] | +0.04 [−0.02, +0.11] | −0.13 [−0.24, −0.03] |
| Llama-3.1-8B | 100 | 53 | 15 | 0.12 | 0.13 | 0.27 | 0.27 | +0.14 [+0.05, +0.23] | +0.00 [−0.11, +0.11] | +0.01 [−0.08, +0.10] | −0.15 [−0.24, −0.06] | |
| Mistral-7B | 108 | 47 | 12 | 0.10 | 0.08 | 0.14 | 0.43 | +0.34 [+0.25, +0.44] | +0.29 [+0.19, +0.40] | −0.02 [−0.09, +0.06] | −0.30 [−0.39, −0.20] | |
| TriviaQA | Qwen3-8B | 151 | 64 | 51 | 0.32 | 0.60 | 0.78 | 0.60 | +0.00 [−0.09, +0.09] | −0.18 [−0.25, −0.11] | +0.28 [+0.20, +0.36] | −0.09 [−0.17, −0.01] |
| Llama-3.1-8B | 127 | 60 | 37 | 0.46 | 0.51 | 0.66 | 0.60 | +0.09 [−0.01, +0.18] | −0.06 [−0.16, +0.03] | +0.06 [−0.03, +0.14] | −0.16 [−0.25, −0.06] | |
| Mistral-7B | 117 | 62 | 29 | 0.44 | 0.43 | 0.62 | 0.62 | +0.19 [+0.09, +0.29] | −0.01 [−0.10, +0.09] | −0.02 [−0.09, +0.06] | −0.25 [−0.35, −0.15] |
Table 29. Comparison with InfoFlow-KV-style reordering [30], re-implemented in our engine at the 5% budget. InfoFlow style: tokens are scored by the L2 norm of question attention (layers 22–25) on the cache with chunk-local positions; chunks are ordered by the fraction of their tokens among the top 5%, with the highest fraction last; tokens are scored again on the reordered prompt; and the top 5% are recomputed. Token 5%: the token-level selector at 5% on the original prompt. Chunk moved: the retriever’s top chunk moved last and recomputed (Table 27). InfoFlow chunk: the last chunk of the InfoFlow order recomputed with the setting’s rule. InfoFlow 15%: the same procedure at its default budget of 15% in both passes. Hit: the share of questions on which the chunk placed last at 5% is the target. Differences are paired, with 95% bootstrap intervals.
At this budget, moving the retriever’s top chunk last and recomputing it beats the procedure significantly in six of the nine settings, by 0.14 to 0.34, and five of these differences remain
Holm’s correction. In the other three, Llama-3.1-8B on NQ-Open and TriviaQA and Qwen3-8B on TriviaQA, the differences are +0.00 to +0.09, so the moved chunk is never significantly worse. The procedure itself improves on the token-level selector significantly in three settings, and on the same scoring applied without reordering in six. At its default budget of 15% in both passes, the procedure recomputes 4.5–6.2 times as many tokens as the moved chunk (medians), and its retention rises by 0.06 to 0.26 over the 5% version, significantly in eight of the nine. Even with this larger budget, the procedure is significantly better than the moved chunk at 5% in only two settings and significantly worse in one (one of each after Holm’s correction).
The difference between the two methods has two parts: which chunk is placed last, and whether all of its tokens are recomputed. The chunk that the procedure places last is the target on 12–52% of the questions, against 47–64% for the retriever’s top chunk, and this share is significantly lower in six settings. Recomputing all tokens of that last chunk gives lower retention than recomputing the retriever’s chunk everywhere, significantly so in six of the nine. On the questions where both orders put the target last (10 to 55 per setting), recomputing all tokens of the last chunk gives significantly different retention under the two orders in only one setting. On these questions, recomputing all tokens of the moved chunk still beats recomputing the procedure’s selected tokens significantly in six settings, by 0.11 to 0.70. On the prompts with the retriever’s chunk moved last, the token-level rules based on question attention spend 13–24% of the budget on the moved chunk (one of 30 or 31 chunks), whereas EPIC spends about 3%.
The order also affects full prefill, which answers 58–95% of in the InfoFlow order and 83–98% with the retriever’s chunk moved. On the questions that full prefill answers in both reordered prompts (52–93% of ), the moved chunk still scores higher than the procedure in eight of the nine settings and the same in the remaining one (+0.00 to +0.55), significantly higher in six. Recomputing all tokens of the last chunk of the InfoFlow order again gives lower retention than recomputing the moved chunk in all nine settings, significantly so in six.
Moving the top three chunks. Let be the share of questions whose target is among the top chunks, and and the retention of the moved top chunk at 5% on the questions where it is and is not the target. If moving chunks gives retention when the target is among them and otherwise, the gain of moving three chunks over one is . The predictions in Table 30 use the values of and listed there, which come from the experiments of Table 27, and range from +0.07 to +0.23. The chunks are ranked by the BGE-M3 cosine similarity with the question alone. The top three are moved to the last three slots in reverse order of rank, so that the top one is last; the other chunks keep their order, the headers are renumbered, and every chunk is encoded alone as before. With one chunk moved, the prompt is the same as in Table 27, and the results at 5% are those of Table 27 and Table 28. Full prefill of the reordered prompts answers 68–97% of , and retention is still measured against full prefill in the original order.
When the moved chunks hold more tokens than the budget, the tokens within them are ranked by the setting’s rule. At 10%, the three chunks fit entirely within the budget on 41–73% of the ques-
| Dataset | Model | Hit (%) top-1/3 | chunks | chunks | tokens | chunks | Predicted | Chunks − tokens, | chunks − Token, orig. | ||
| NQ-Open | Qwen3-8B | 146 | 55/75 | 0.93<br>0.12 | 0.56 | 0.67 | 0.50 | +0.11<br>[+0.04, +0.18] | +0.16 | +0.17<br>[+0.09, +0.25] | +0.43<br>[+0.34, +0.53] |
| Llama-3.1-8B | 128 | 47/70 | 0.83<br>0.13 | 0.46 | 0.52 | 0.65 | +0.06<br>[−0.02, +0.14] | +0.16 | −0.12<br>[−0.20, −0.05] | +0.14<br>[+0.04, +0.23] | |
| Mistral-7B | 101 | 50/71 | 0.92<br>0.16 | 0.54 | 0.66 | 0.66 | +0.12<br>[+0.03, +0.21] | +0.16 | +0.00<br>[−0.09, +0.09] | +0.33<br>[+0.21, +0.46] | |
| TriviaQA | Qwen3-8B | 151 | 64/85 | 0.84<br>0.19 | 0.60 | 0.74 | 0.69 | +0.13<br>[+0.07, +0.20] | +0.14 | +0.05<br>[−0.03, +0.12] | +0.25<br>[+0.15, +0.34] |
| Llama-3.1-8B | 127 | 60/79 | 0.89<br>0.16 | 0.60 | 0.69 | 0.74 | +0.09<br>[+0.01, +0.17] | +0.14 | −0.06<br>[−0.13, +0.02] | +0.13<br>[+0.03, +0.24] | |
| Mistral-7B | 117 | 62/85 | 0.88<br>0.18 | 0.62 | 0.77 | 0.74 | +0.15<br>[+0.09, +0.22] | +0.15 | +0.03<br>[−0.04, +0.09] | +0.25<br>[+0.15, +0.35] | |
| Gorilla | Qwen3-8B | 89 | 53/74 | 0.47<br>0.05 | 0.28 | 0.44 | 0.20 | +0.16<br>[+0.07, +0.26] | +0.09 | +0.24<br>[+0.12, +0.36] | +0.25<br>[+0.15, +0.36] |
| Llama-3.1-8B | 100 | 53/69 | 0.47<br>0.04 | 0.27 | 0.37 | 0.51 | +0.10<br>[+0.00, +0.20] | +0.07 | −0.14<br>[−0.23, −0.04] | +0.06<br>[−0.05, +0.17] | |
| Mistral-7B | 108 | 47/72 | 0.90<br>0.00 | 0.43 | 0.56 | 0.28 | +0.13<br>[+0.04, +0.22] | +0.23 | +0.28<br>[+0.17, +0.39] | +0.26<br>[+0.13, +0.39] |
caption: Table 30: Moving the retriever’s top- chunks to the slots before the question, in reverse order of rank (top-1 last), and recomputing them at the 10% budget. Hit: share of questions whose target is the top-1 chunk or among the top three. (upper) and (lower): retention of the moved top-1 chunk at 5% on the questions where it is and is not the target (Table 27), the values the prediction uses. Chunks: the moved chunks recomputed with the setting’s rule; tokens: the setting’s token-level rule over all chunks on the same reordered prompt. Predicted: the gain predicted from the hit rates and from the retention of the moved top-1 chunk on hits and misses (Appendix § 13). Token, orig.: the token-level selector at 10% on the original prompt. Paired 95% bootstrap intervals; bold intervals exclude zero.
:::
tions. With one chunk moved and a 10% budget, only that chunk is recomputed and the rest of the budget is left unspent, as for Target and Retrieved in Table 1. Its scores then equal those at 5% in Table 27 on every question except one, on Gorilla with Qwen3-8B, where the moved chunk exceeds the 5% budget.
The prediction lies within the interval of the measured gain in seven of the nine settings and above it for NQ-Open with Llama-3.1-8B and Gorilla with Mistral-7B. On questions where the top chunk is not the target but one of the three moved chunks is, the gain is 0.43 to 0.62. Moving more chunks helps even at 5%, although two moved chunks then fit entirely within the budget on at most 43% of the questions and three on at most 4%. Moving two or three chunks beats moving one everywhere, significantly in four and five settings, respectively.
Against the other methods, the three moved chunks recomputed at 10% are significantly better than InfoFlow-style reordering at its default 15% in three settings and never significantly worse. At 5%, they beat the token-level selector on the same reordered prompt significantly in six and never lose to it significantly, whereas the single moved chunk does so in four. At 10%, the single moved chunk beats the selector on its own reordered prompt significantly in two settings but loses significantly in four (−0.21 to +0.13).
The two losses from recomputing the three chunks at 10% with Llama-3.1-8B remain on the questions whose target is among the three chunks and where all three fit within the budget: −0.18 [−0.28, −0.08] on NQ-Open and −0.19 [−0.33, −0.04] on Gorilla. This is consistent with Table 1, where recomputing the known target at 10% does not beat the selector with Llama-3.1-8B on these two datasets.
A second retriever. Table 31 repeats these experiments with BM25 (Robertson and Zaragoza, [28]) in place of BGE-M3. We use Okapi BM25 with and over the chunk texts without their position headers, take document frequencies from all chunks of the dataset, and break ties by position. Prompts, rules and budgets are unchanged. On NQ-Open, BM25 ranks the target first on 23–25% of , against 47–55% for BGE-M3; on Gorilla and TriviaQA, the two hit rates are close. From the BM25 hit rates and the and of BGE-M3, we predict the retention of the moved BM25 chunk, , and the gain of moving three chunks at 10%, as in Table 30. The first prediction lies within the interval of the measured retention in every setting, and the second lies within the interval of the measured gain in all but one, Gorilla with Mistral-7B, where the prediction for BGE-M3 is also above the measured gain. On NQ-Open, where the hit rate halves, the moved BM25 chunk changes retention by −0.24 to −0.17 relative to the moved BGE-M3 chunk, significantly with all three models, as the lower hit rate predicts. In none of the nine settings is the moved BM25 chunk significantly better than the moved BGE-M3 chunk. The gain of moving a retriever’s chunk thus follows the retriever’s hit rate rather than a property of BGE-M3. Figure 6 plots both.
Figure 6. Predicted and measured effects of moving a retriever’s top chunks last at the 10% budget, on NQ-Open, Gorilla and TriviaQA. (a) Retention of the moved BM25 top chunk against , where is the BM25 top-1 hit rate and and are the BGE-M3 values from Table 27. (b) Gain of moving the top three chunks over the top one chunk at 10%, , where is the top-3 hit rate, for BGE-M3 (filled; Table 30) and BM25 (open; Table 31). Bars are 95% paired bootstrap intervals, and the dashed line is .
predictions against the measurements.
Time to first token. Table 26 reports the time to first token of each condition with Qwen3-8B in our engine, where every condition runs through the same attention kernels. Reuse reaches the first token in 4–6% of the full-prefill time, and the repairs do so in 9–24%. At 5%, Retrieved+4 takes a median of 6–9 ms per question longer than the token-level selector, which is the cost of encoding the question with BGE-M3 and ranking the chunks. Recomputing the retrieved chunk alone is 11–57 ms faster than the token-level selector. On open-domain QA and API documentation, the chunk usually holds fewer tokens than the budget and needs no attention scoring, whereas on LoCoMo the session usually fills the budget and its tokens are still ranked.
| Condition | LoCoMo (10.8K) | NQ-Open (4.6K) | Gorilla (5.7K) | TriviaQA (4.7K) | LoCoMo | NQ-Open | Gorilla | TriviaQA |
| Full prefill | 2554 (3136) | 840 (898) | 1072 (1938) | 842 (959) | 1.00 | 1.00 | 1.00 | 1.00 |
| Reuse | 94 (120) | 53 (66) | 58 (81) | 53 (57) | 0.04 | 0.06 | 0.06 | 0.06 |
| Token 5% | 309 (363) | 162 (198) | 129 (189) | 165 (176) | 0.12 | 0.19 | 0.12 | 0.19 |
| Target 5% | 268 (324) | 92 (114) | 103 (149) | 94 (104) | 0.11 | 0.11 | 0.09 | 0.11 |
| Retrieved 5% | 282 (352) | 105 (130) | 117 (165) | 108 (114) | 0.11 | 0.13 | 0.11 | 0.13 |
| Retrieved+4 5% | 316 (368) | 169 (183) | 137 (190) | 173 (186) | 0.12 | 0.20 | 0.13 | 0.20 |
| Token 10% | 415 (510) | 198 (237) | 173 (263) | 200 (215) | 0.16 | 0.23 | 0.16 | 0.23 |
| Retrieved+4 10% | 417 (491) | 202 (237) | 177 (252) | 206 (223) | 0.17 | 0.24 | 0.16 | 0.24 |
Table 26. Time to first token (TTFT) with Qwen3-8B in our engine on one RTX 4090 D, measured on the first 50 questions of each dataset (taken from the second question set for NQ-Open and Gorilla). The time for each question is the median of three runs; the table gives the median over questions and, in parentheses, the 95th percentile (p95). The median context length of these questions is given in parentheses below each dataset name. Reuse assembles stored chunk caches (RoPE shift and concatenation); the repairs add selection (question-attention scoring where the rule needs it, and BGE-M3 question encoding and chunk ranking for Retrieved) and recomputation. All conditions use the same attention kernels; an optimised serving system would change the absolute times.
A fixed token-level rule. The token-level baseline uses the rule chosen on development questions for each setting. Table 24 repeats the 5% allocation with ProphetKV as the rule in the five settings where another rule was chosen: Gorilla with all three models, TriviaQA with Mistral-7B and LoCoMo with Mistral-7B. Recomputing the target alone beats the ProphetKV baseline in all five, by 0.11 to 0.46, so it beats the ProphetKV baseline on every open-domain QA and API-documentation setting. On LoCoMo with Mistral-7B, by contrast, the target alone does not beat the setting’s own rule (Table 1). There, ProphetKV has a retention of 0.69, against 0.78 for question attention, so the result on LoCoMo depends on which token-level rule serves as the baseline.
| Dataset | Model | Token 5% (own rule) | Token 5% (ProphetKV) | Target − Token | Target+4 − Token | Retrieved − Token | Retrieved +4 − Token | |
| LoCoMo | Mistral-7B | 36 | 0.78 | 0.69 | +0.11 [+0.02, +0.24] | +0.08 [−0.06, +0.24] | +0.06 [−0.05, +0.16] | +0.08 [−0.05, +0.23] |
| Gorilla | Qwen3-8B | 89 | 0.06 | 0.09 | +0.13 [+0.04, +0.22] | +0.15 [+0.07, +0.24] | +0.06 [−0.02, +0.13] | +0.12 [+0.04, +0.20] |
| Gorilla | Llama-3.1-8B | 100 | 0.12 | 0.07 | +0.16 [+0.08, +0.25] | +0.12 [+0.04, +0.20] | +0.08 [+0.01, +0.16] | +0.07 [−0.01, +0.14] |
| Gorilla | Mistral-7B | 108 | 0.10 | 0.12 | +0.46 [+0.35, +0.56] | +0.35 [+0.24, +0.46] | +0.18 [+0.08, +0.27] | +0.19 [+0.10, +0.29] |
| TriviaQA | Mistral-7B | 117 | 0.44 | 0.38 | +0.30 [+0.21, +0.38] | +0.24 [+0.15, +0.32] | +0.17 [+0.09, +0.26] | +0.18 [+0.09, +0.26] |
Table 24. The 5% allocation of Table 1 with ProphetKV as the token-level rule, in the settings where the rule selected on development questions is not ProphetKV. Token (own rule) repeats the baseline of Table 1; the gains (with 95% bootstrap intervals) are over the ProphetKV baseline. A chosen chunk that exceeds the budget has its tokens ranked by ProphetKV here; on LoCoMo the session exceeds it on 32 of the 36 questions. Target and Retrieved nevertheless give the same scores as in Table 1, except for 3 scores, each on a question where a chosen chunk exceeds the budget.
Larger models. We repeated the main conditions with Qwen3-14B and Qwen3-32B (Table 32). These experiments cover the four test sets, with the second question sets of NQ-Open and Gorilla added by the rule of Appendix § 8, which gives 67–157 questions per dataset and model. They include the three distractor sources and, on , the token-level selector, Target, Retrieved and Retrieved+4 at 5% with ProphetKV as the rule. On open-domain QA and API documentation, they also include the two moved-chunk conditions of Table 27 and Table 28. For Qwen3-14B, the 40 layers are split over two GPUs, and each layer’s cache is kept on the GPU that holds that layer. When Qwen3-8B is split in the same way, it reproduces all 260 outputs of its single-GPU runs on 20 NQ-Open questions. Qwen3-32B runs on one RTX PRO 6000. Run on that GPU, Qwen3-8B gives the same score as on the RTX 4090 for 247 of the 249 outputs that both runs produce on 20 NQ-Open questions, but the same text for only 210 of them. Greedy decoding turns small numerical differences between the two GPU architectures into different wordings. Each model runs all its conditions on one kind of GPU, so these differences do not favour any condition.
| Dataset | Model | Loss | Same | Sim. | Target − Token | Retrieved − Token | Chunk moved − Token | Token moved − Token | Chunk moved − Token moved | |
| LoCoMo | 14B | 74 | +0.36 [+0.21, +0.51] | +0.20 [+0.08, +0.31] | +0.03 [−0.07, +0.12] | +0.14 [+0.05, +0.22] | +0.05 [−0.05, +0.17] | – | – | – |
| 32B | 67 | +0.46 [+0.35, +0.54] | +0.16 [+0.02, +0.30] | +0.18 [+0.05, +0.31] | +0.07 [−0.01, +0.16] | +0.00 [−0.04, +0.04] | – | – | – | |
| 8B | 105 | +0.47 | +0.19 | +0.16 | −0.04 | −0.10 | – | – | – | |
| NQ-Open | 14B | 150 | +0.79 [+0.72, +0.85] | +0.17 [+0.09, +0.25] | +0.08 [−0.01, +0.17] | +0.22 [+0.13, +0.30] | +0.13 [+0.05, +0.21] | +0.34 [+0.24, +0.43] | +0.24 [+0.16, +0.32] | +0.10 [+0.03, +0.17] |
| 32B | 138 | +0.78 [+0.71, +0.85] | +0.37 [+0.28, +0.46] | +0.08 [−0.01, +0.18] | +0.38 [+0.28, +0.46] | +0.23 [+0.14, +0.31] | +0.35 [+0.25, +0.44] | +0.25 [+0.17, +0.33] | +0.09 [+0.01, +0.17] | |
| 8B | 146 | +0.85 | +0.23 | +0.19 | +0.35 | +0.22 | +0.40 | +0.11 | +0.29 | |
| Gorilla | 14B | 111 | +0.90 [+0.85, +0.95] | +0.58 [+0.49, +0.68] | −0.10† [−0.15, −0.05] | +0.45 [+0.36, +0.54] | +0.28 [+0.18, +0.38] | +0.38 [+0.28, +0.49] | +0.17 [+0.09, +0.25] | +0.21 [+0.13, +0.30] |
| 32B | 122 | +0.93 [+0.88, +0.97] | +0.90 [+0.84, +0.95] | −0.07† [−0.12, −0.03] | +0.43 [+0.34, +0.52] | +0.21 [+0.11, +0.30] | +0.30 [+0.19, +0.40] | +0.15 [+0.05, +0.25] | +0.15 [+0.08, +0.22] | |
| 8B | 89 | +1.00 | +0.82 | +0.00† | +0.17 | +0.09 | +0.21 | +0.00 | +0.21 | |
| TriviaQA | 14B | 157 | +0.77 [+0.71, +0.83] | +0.32 [+0.24, +0.40] | +0.20 [+0.11, +0.29] | +0.20 [+0.13, +0.28] | +0.10 [+0.03, +0.18] | +0.24 [+0.14, +0.34] | +0.22 [+0.15, +0.30] | +0.01 [−0.05, +0.08] |
| 32B | 123 | +0.72 [+0.64, +0.80] | +0.24 [+0.15, +0.33] | +0.25 [+0.16, +0.34] | +0.25 [+0.15, +0.36] | +0.15 [+0.05, +0.25] | +0.22 [+0.11, +0.32] | +0.24 [+0.16, +0.33] | −0.02 [−0.10, +0.06] | |
| 8B | 151 | +0.73 | +0.25 | +0.17 | +0.20 | +0.14 | +0.28 | +0.19 | +0.09 |
Table 32. Qwen3-14B (bf16, with its layers split over two RTX 4090 D) and Qwen3-32B (bf16, on one RTX PRO 6000) on the test sets, with the second question sets for NQ-Open and Gorilla, compared with Qwen3-8B. Loss, Same and Sim.: the loss and the same-category and similarity terms, as in Table 10; allocation columns as in Table 1; moved-chunk columns as in Table 27 and Table 28. The token-level rule is ProphetKV in every Qwen3-14B and Qwen3-32B setting; for Qwen3-8B it is the development-selected rule, which differs from ProphetKV only on Gorilla. The five replication criteria cover the loss, the same-category and similarity terms, Target − Token and Chunk moved − Token; the last two columns are reported in addition. Retrieved+4 was also evaluated but is not shown; it beats Token by +0.12 to +0.27 on open-domain QA and API documentation. Qwen3-14B and Qwen3-32B: paired differences with 97.5% bootstrap intervals; Qwen3-8B: point estimates. Bold: the interval excludes zero. †Retention under RAND is zero, so the term cannot be positive; these intervals are not set in bold.
Both models reproduce all five results that we test for replication. The loss is significant on all four datasets (0.36–0.93), and the same-category term is significantly positive on all four. The similarity term is positive on both open-domain QA datasets but significant only on TriviaQA. On NQ-Open, it is +0.08 [−0.01, +0.17] with Qwen3-14B and +0.08 [−0.01, +0.18] with Qwen3-32B. On Gorilla, retention under RAND is again zero. Recomputing the target beats the token-level selector at 5% on all three open-domain QA and API-documentation datasets (0.20 to 0.45), and so does recomputing the retrieved chunk after moving it last (0.22 to 0.38). Beyond these five results, the move also raises the retention of the token-level selector on all three datasets (0.15 to 0.25). On the reordered prompt, recomputing the moved chunk instead of selected tokens is significantly better on NQ-Open and Gorilla but not on TriviaQA, where the differences are +0.01 with Qwen3-14B and −0.02 with Qwen3-32B.
On LoCoMo the two models differ. Recomputing the target session beats the selector by +0.14 [+0.05, +0.22] with Qwen3-14B, and by +0.07 [−0.01, +0.16], not significantly, with Qwen3-32B. With the same rule, Qwen3-8B shows no gain (−0.04), whereas Mistral-7B does (+0.11, Table 24). The whole session fits within the budget on 22% and 21% of these questions, about as often as for the smaller models, and the bootstrap intervals resample only eight conversations. Recomputing the retrieved session gains +0.05 [−0.05, +0.17] with Qwen3-14B and +0.00 [−0.04, +0.04] with Qwen3-32B.
Full-prefill KV versus recomputation. Inserting the target’s full-prefill KV into the reused cache (§ 3.2) and recomputing the target against the reused cache (§ 5) both restore the same chunk, yet the latter recovers more. Table 33 compares them on the same questions in seven settings. As a reference, recomputing the target against the full-prefill cache gives the full-prefill score on 265 of 270 questions and the same output text on 244, below the distances reported next. On these questions, the inserted full-prefill KV changes retention significantly in three of the seven settings, and on all of it raises retention significantly in six of the twelve settings (§ 3.2). Recomputation,
| Dataset | Model | None | Target KV | Recomp. | Recomp. on full | Recomp. − Target KV | None | Recomp. | None | Target KV | Rec. | Full | |
| NQ-Open | Qwen3-8B | 40 | 0.15 | 0.15 | 0.53 | 0.97 | +0.38 [+0.23, +0.53] | 0.36 | 0.26 | 0.035 | 0.021 | 0.050 | 0.060 |
| NQ-Open | Llama-3.1-8B | 30 | 0.20 | 0.20 | 0.33 | 1.00 | +0.13 [+0.03, +0.27] | 0.40 | 0.28 | 0.029 | 0.020 | 0.025 | 0.064 |
| NQ-Open | Mistral-7B | 40 | 0.23 | 0.42 | 0.62 | 0.97 | +0.20 [+0.05, +0.35] | 0.40 | 0.27 | 0.031 | 0.019 | 0.030 | 0.059 |
| Gorilla | Qwen3-8B | 40 | 0.00 | 0.05 | 0.25 | 0.95 | +0.20 [+0.10, +0.33] | 0.42 | 0.30 | 0.040 | 0.027 | 0.065 | 0.053 |
| Gorilla | Mistral-7B | 40 | 0.05 | 0.38 | 0.55 | 1.00 | +0.17 [+0.03, +0.33] | 0.45 | 0.29 | 0.044 | 0.034 | 0.045 | 0.068 |
| TriviaQA | Qwen3-8B | 40 | 0.35 | 0.30 | 0.50 | 1.00 | +0.20 [+0.07, +0.33] | 0.34 | 0.23 | 0.040 | 0.026 | 0.061 | 0.076 |
| TriviaQA | Mistral-7B | 40 | 0.25 | 0.50 | 0.65 | 0.97 | +0.15 [+0.03, +0.28] | 0.39 | 0.25 | 0.048 | 0.040 | 0.058 | 0.092 |
| Dataset | Model | Earlier chunks reused: Target reused | Earlier chunks reused: Target KV | Earlier chunks full: Target reused | Earlier chunks full: Target KV | Target recomp. | Full target effect (reused) | Full target effect (full) | Interaction | ||||
| NQ-Open | Qwen3-8B | 40 | 0.15 | 0.15 | 0.28 | 0.55 | 0.53 | +0.00 [−0.12, +0.12] | +0.28 [+0.12, +0.42] | +0.28 [+0.15, +0.40] | |||
| NQ-Open | Llama-3.1-8B | 30 | 0.20 | 0.20 | 0.27 | 0.43 | 0.33 | +0.00 [−0.17, +0.17] | +0.17 [+0.00, +0.33] | +0.17 [−0.03, +0.37] | |||
| NQ-Open | Mistral-7B | 40 | 0.23 | 0.42 | 0.33 | 0.55 | 0.62 | +0.20 [+0.03, +0.40] | +0.23 [+0.10, +0.35] | +0.03 [−0.12, +0.17] | |||
| Gorilla | Qwen3-8B | 40 | 0.00 | 0.05 | 0.15 | 0.45 | 0.25 | +0.05 [+0.00, +0.12] | +0.30 [+0.17, +0.45] | +0.25 [+0.12, +0.38] | |||
| Gorilla | Mistral-7B | 40 | 0.05 | 0.38 | 0.07 | 0.60 | 0.55 | +0.33 [+0.17, +0.47] | +0.20 [+0.38, +0.68] | +0.20 [+0.03, +0.35] | |||
| TriviaQA | Qwen3-8B | 40 | 0.35 | 0.30 | 0.42 | 0.45 | 0.50 | −0.05 [−0.20, +0.10] | +0.03 [−0.10, +0.15] | +0.07 [−0.05, +0.20] | |||
| TriviaQA | Mistral-7B | 40 | 0.25 | 0.50 | 0.42 | 0.65 | 0.65 | +0.25 [+0.07, +0.40] | +0.23 [+0.07, +0.38] | −0.03 [−0.20, +0.17] |
Table 33. Repairing the target chunk in the reused SIM cache, for the first 40 questions (30 for Llama-3.1-8B) in identifier order whose target is not in the first slot. Retention with the target unrepaired (None), with its full-prefill KV inserted (Target KV), recomputed against the reused cache (Recomp.), and recomputed against the full-prefill cache (Recomp. on full; by causality it equals full prefill up to numerical differences). KV dist.: median relative distance between the target KV from its full-prefill value. Attention: the share of the question’s attention mass on the target chunk, aggregated as for the detector (Table 12), for the three caches (None, Target KV, and Rec. for Recomp.) and for full prefill (Full).
| Dataset | Model | Chance | reuse | reuse | full | full | reuse: last chunk | ||
| LoCoMo | Qwen3-8B | 29 | 6.4 | 10.3 | 17.2 | 20.7 | 31.0 | 51.7 | 86.2 |
| Llama-3.1-8B | 28 | 6.4 | 0.0 | 14.3 | 46.4 | 89.3 | 50.0 | 100.0 | |
| Mistral-7B | 14 | 6.3 | 7.1 | 7.1 | 14.3 | 42.9 | 50.0 | 92.9 | |
| NQ-Open | Qwen3-8B | 36 | 3.4 | 11.1 | 16.7 | 11.1 | 27.8 | 50.0 | 77.8 |
| Llama-3.1-8B | 38 | 3.3 | 10.5 | 13.2 | 23.7 | 55.3 | 55.3 | 100.0 | |
| Mistral-7B | 33 | 3.4 | 6.1 | 6.1 | 9.1 | 54.5 | 63.6 | 100.0 | |
| Gorilla | Qwen3-8B | 16 | 3.2 | 12.5 | 18.8 | 18.8 | 12.5 | 56.2 | 68.8 |
| Llama-3.1-8B | 11 | 3.2 | 0.0 | 9.1 | 27.3 | 36.4 | 36.4 | 100.0 | |
| Mistral-7B | 13 | 3.2 | 0.0 | 0.0 | 0.0 | 7.7 | 38.5 | 100.0 | |
| TriviaQA | Qwen3-8B | 50 | 3.3 | 6.0 | 14.0 | 16.0 | 6.0 | 54.0 | 58.0 |
| Llama-3.1-8B | 52 | 3.3 | 1.9 | 7.7 | 50.0 | 63.5 | 51.9 | 100.0 | |
| Mistral-7B | 49 | 3.3 | 2.0 | 10.2 | 26.5 | 55.1 | 55.1 | 100.0 |
Table 12. Top-1 accuracy (%) of target-chunk detectors on development questions answered correctly by full prefill. Chance: the accuracy of choosing a chunk at random. : the sum of question attention over a chunk; : its mean over the upper half of the layers, with the first 16 tokens of each chunk excluded. Both are computed on the reused (position-corrected) KV and on the full-prefill KV. : the cosine similarity between the BGE-M3 embeddings of the question and the chunk. Last column: the share of questions on which over the reused cache selects the chunk next to the question.
in contrast, gains 0.13–0.38 over the inserted KV, significantly in every setting, and it is better on 60 questions and worse on four. The recomputed target stays far from its full-prefill KV (median relative distance 0.23–0.30, against 0.34–0.45 for the unrepaired target), so closeness to the full-prefill KV does not explain the gain.
One explanation is that the recomputed target is encoded against the same reused chunks that the question sees when it reads the target, whereas the full-prefill KV of the target was encoded against different KV for those chunks. Table 34 tests this explanation by also restoring the full-prefill KV of every chunk before the target, while the chunks after it stay reused. It uses the same questions, which include three Mistral-7B settings where the inserted KV helps on all of . On 270 questions in seven settings, inserting the target’s full-prefill KV raises retention by +0.11 [+0.06, +0.17] when the earlier chunks stay reused and by +0.25 [+0.20, +0.31] when their full-prefill KV is restored too. With the earlier chunks reused, the inserted target KV raises retention significantly only in the three Mistral-7B settings, and with them restored it does so in five. The interaction is positive in six of the seven and significant in three, and pooled over all seven it is +0.14 [+0.08, +0.20].
The models differ in how much the inserted KV helps on its own. With Qwen3-8B and Llama-3.1-8B (four settings), it changes retention by −0.05 to +0.05, and it gains more when the earlier chunks are also restored, in all four settings (significantly in two). With Mistral-7B, it helps on its own (+0.20 to +0.33), and restoring the earlier chunks adds a significant further gain only on Gorilla. Recomputing the target against the restored earlier chunks reproduces the inserted full-prefill target, as causality requires. It gives the same score on 269 of 270 questions, with a median relative KV distance of at most 0.02. Attention mass does not follow these effects. Restoring the earlier chunks raises the question’s attention on the target from 0.03–0.05 to 0.04–0.08 of its mass, and adding the target’s full-prefill KV lowers it again to 0.03–0.06 while retention rises.
Copied and recomputed states in other settings. Two studies of stale caches find little difference between copying full-prefill states into a cache and recomputing them in place. After a document edit, Mao et al. [24] find that the two recover the answer margin to within 0.02 of each other when the repaired positions form a contiguous window starting at the edit, whereas copying recovers far more for the scattered positions where copying has the largest effect. SAFE-KV keeps 99.8% of the full-prefill answers on HotpotQA with copied suffix states and 100.0% with recomputed ones, at an operating point that re-encodes 94.8% of the context (Zhang, [39]). In both studies, the copied states are read next to states whose context is close to their own: the stale text differs from the edited text in a span of at most six tokens, and the copied suffix covers most of the prompt. In our setting, the target is read next to chunks that were each encoded alone and are not restored, and Table 34 shows that the inserted KV gains more when the chunks before it are restored. Our explanation in terms of contextual compatibility is limited to this configuration, in which the target’s full-prefill KV is inserted while the chunks before it stay reused.
References
- [1]Anonymous. 2026a. Construct before you repair: Surrogate contexts for non-prefix KV cache reuse. Submitted to ICLR 2027, OpenReview https://openreview.net/forum?id=OWX14VkewV.
- [2]Anonymous. 2026b. Predicting output preservation and first divergence under KV-cache compression. Submitted to ICLR 2027, OpenReview https://openreview.net/forum?id=KzXUf16XnW.
- [3]Anonymous. 2026c. Speculating on the past: Loss-less reuse of stale KV caches via self-speculative verification. Submitted to ICLR 2027, OpenReview https://openreview.net/forum?id=96mkAG7qgh1.
- [4]Yuhang Cao, Yanzhou Mu, Chunrong Fang, and Zhenyu Chen. 2026. CacheReforge: Bounded recovery for stale KV caches under evolving adapters. Preprint, arXiv:2609.30884.arxiv.org/abs/2609.30884
- [5]Samuel Cestola, Tianxiang Xia, Pengfei Zheng, Weiyan Zheng, Bo Wang, Yi Zhao, and Diego Didona. 2026. Evaluating the accuracy of KV cache reuse techniques. Preprint, arXiv:2609.31415.arxiv.org/abs/2609.31415
- [6]JianlYu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2318–2335.arxiv.org/abs/2402.03216
- [7]Zhiwei Chen, Zhongtao Rao, Waikit Xiu, Yuxiang Zhang, Shuting Cui, Jian Chen, and Tianxiang Zhao. 2026. LatentRemedy: Efficient remedy for non-prefix KV cache reuse in multi-agent systems. Preprint, arXiv:2610.18014.arxiv.org/abs/2610.18014
- [8]Chuheng Du, Junyi Chen, Hanlin Tang, Kan Liu, Tao Lan, Lin Qu, Chaoyue Niu, Shengzhong Liu, Guihai Chen, and Fan Wu. 2026. C²KV: Compressed and composable KV cache reuse for efficient LLM inference. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pages 991–1002.DOI
- [9]In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. 2024. Prompt cache: Modular attention reuse for low-latency inference. In Proceedings of Machine Learning and Systems (MLSys).arxiv.org/abs/2311.04934
- [10]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, et al. 2024. The Llama 3 herd of models. Preprint, arXiv:2407.21783.arxiv.org/abs/2407.21783
- [11]Junhao Hu, Wenrui Huang, Weidong Wang, Haoyi Wang, Tiancheng Hu, Qin Zhang, Hao Feng, Xusheng Chen, Yizhou Shan, and Tao Xie. 2025. EPIC: Efficient position-independent caching for serving large language models. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 24391–24402. PMLR.arxiv.org/abs/2410.15332
- [12]Taiga Ikeda, Daisuke Miyashita, and Jun Deguchi. 2026. KV-Repurpose: A testbed for evaluating non-prefix KV-cache reuse. Preprint, arXiv:2610.13060.arxiv.org/abs/2610.13060
- [13]Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research.arxiv.org/abs/2112.09118
- [14]Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B. Preprint, arXiv:2310.06825.arxiv.org/abs/2310.06825
- [15]Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of ACL, pages 1601–1611.arxiv.org/abs/1705.03551
- [16]Haeyong Kang and Chang D. Yoo. 2026. Budgeted cache repair for cross-context KV-cache reuse. Preprint, arXiv:2610.02233.arxiv.org/abs/2610.02233
- [17]Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of EMNLP, pages 6769–6781.arxiv.org/abs/2004.04906
- [18]Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken\-ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466.
- [19]Fei Li, Song Liu, Yan Liu, Jinhua Cui, Shiqiang Nie, Jinyu Wang, and Weiguo Wu. 2026. Adaptive KV cache reuse for fast long-context LLM serving. Preprint, arXiv:2605.24022.arxiv.org/abs/2605.24022
- [20]Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paran\-jape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.arxiv.org/abs/2307.03172
- [21]Songshuo Lu, Hua Wang, Yutian Rong, Zhi Chen, and Yaohua Tang. 2025. TurboRAG: Accelerating retrieval-augmented generation with precomputed KV caches for chunked text. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 6588–6601.arxiv.org/abs/2410.07590
- [22]Dongyang Ma, Yan Wang, and Tian Lan. 2025. Block-attention for efficient prefilling. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2409.15355
- [23]Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024. Evaluating very long-term conversational memory of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13851–13870.
- [24]Mingyang Mao, Wyatt Mackey, and Xiaomin Lin. 2026. Contiguity, not importance: Budgeted repair of stale KV caches after document edits. Preprint, arXiv:2609.17983.arxiv.org/abs/2609.17983
- [25]Nilesh Prasad Pandey, Jason Kong, Lanxiang Hu, Quan\-ling Zhao, Yujie Zhao, Onat Gungor, Hao Zhang, and Tajana Rosing. 2026. AgentKVShift: Efficient KV cache reuse for agentic memory systems. Preprint, arXiv:2607.21604.arxiv.org/abs/2607.21604
- [26]Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2024. Gorilla: Large language model connected with massive APIs. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/2305.15334
- [27]Ruoling Qi, Yirui Liu, Xuaner Wu, Yuxin Jin, Jian Chen, Jiayu Qin, Yin Chen, and Jiawei Shao. 2026. RelaxKV: Recomputation guided by the query with sparse context attention for efficient KV cache reuse. Preprint, arXiv:2609.33503.arxiv.org/abs/2609.33503
- [28]Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval.
- [30]Xin Teng, Canyu Zhang, Shaoyi Zheng, Danyang Zhuo, Tianyi Zhou, and Shenji Wan. 2026. InfoFlow KV: Information-flow-aware KV recomputation for long context. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 120070–120088.arxiv.org/abs/2603.05353
- [31]Shihao Wang, Jiahao Chen, Yanqi Pan, Hao Huang, Yichen Hao, Xiangyu Zou, Wen Xia, Chongyang Qiu, Wentao Zhang, and Pengfei Wang. 2026. ProphetKV: User-query-driven selective recomputation for efficient KV cache reuse in retrieval-augmented generation. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 125415–125434.arxiv.org/abs/2602.02579
- [32]Hui Xie, Peng Xiao, Yutong Deng, Shuoran Dou, Jian Yang, and Jinyang Guo. 2026. SemPIC: Learning semantic position-independent KV caches. Preprint, arXiv:2607.28069.arxiv.org/abs/2607.28069
- [33]Liang Xu, Zhichun Wang, and Xuetao Tian. 2026. Transition overlap: Construction-time conditioning for position-independent KV cache reuse. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 32366–32386.DOI
- [34]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengeng Huang, Chenxu Lv, Chujie Zheng, Day\-iheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, et al. 2025a. Qwen3 technical report. Preprint, arXiv:2505.09388.arxiv.org/abs/2505.09388
- [35]Jingbo Yang, Bairu Hou, Wei Wei, Yujia Bao, and Shiyu Chang. 2025b. KVLink: Accelerating large language models via efficient KV cache reuse. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/2502.16002
- [36]Xinyu Yang, Tianqi Chen, and Beidi Chen. 2025c. APE: Faster and longer context-augmented generation via adaptive parallel encoding. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2502.05431
- [37]Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Yuntai Du, Shan Lu, and Junchen Jiang. 2025. CacheBlend: Fast large language model serving for RAG with cached knowledge fusion. In Proceedings of the Twentieth European Conference on Computer Systems (EuroSys ’25), pages 94–109.
- [38]Hancheng Ye, Zhengqi Gao, Mingyuan Ma, Qinsi Wang, Yuzhe Fu, Ming-Yu Chung, Yueqian Lin, Zhijian Liu, Jianyi Zhang, Danyang Zhuo, and Yiran Chen. 2025. KVCOMM: Online cross-context KV-cache communication for efficient LLM-based multi-agent systems. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/2510.12872
- [39]Xingjian Zhang. 2026. SAFE-KV: Causal suffix repair for behavior-preserving non-prefix KV cache reuse in RAG. Preprint, arXiv:2610.15186. To appear in Findings of AACL-IJCNLP 2026.arxiv.org/abs/2610.15186



