WorksComputer Science
RECAP: Context-Aware Prediction Recycling with Request-Local and Persistent Memories for Multimodal Speculative Decoding
Abstract
Recycling a target model's own predictions offers a training-free source of proposals for speculative decoding. In vision-language models, however, existing recycling indexes predictions by the last token alone, conflating predictions computed under different images and prefixes; we find that such reuse fails most often at image-sensitive decoding states. Longer keys reduce this aliasing but match fewer states. We introduce \method, which stores target predictions under one-, two-, and three-token keys: contextual memories live only for the current request and are thus implicitly conditioned on its image, while token-level memory persists across requests to retain coverage. At each lookup, entropy-confidence weighting fuses the retrieved predictions according to their key specificity and concentration. Best-first search composes the fused transitions into a candidate tree, supplemented by suffix-matched continuations, and one target pass verifies the tree under greedy decoding. Without training or an auxiliary draft model, \method\ achieves mean speedups over autoregressive decoding of 1.66\speed and 1.56\speed across seven single-turn benchmarks on Qwen2.5-VL-7B and LLaVA-1.6-7B, exceeding the strongest training-free baseline, Token Recycling (1.51\speed and 1.37\speed). Our implementation is available in https://github.com/Senmo996/RECAP-official-code.
Introduction
Vision-language models (VLMs) generate text one token at a time, so every output token requires a sequential target-model pass. Speculative decoding reduces the number of sequential passes by proposing several tokens and verifying them in one parallel pass [14, 5]. Multimodal methods usually obtain proposals from
a separately trained drafter [9, 12], adding training and serving cost for every target model. Tree verification offers a cheaper source: each pass computes full next-token distributions at tens of candidate nodes yet commits only a few tokens, discarding the rest. Prediction recycling stores these target predictions and reuses them as later proposals (Luo et al., 2025), requiring neither a draft model nor training.
A recycled prediction is valid for the condition under which the target computed it: the image and the textual prefix. Existing recycling indexes predictions by the last token alone; Token Recycling, for example, keeps one Top- prediction row per vocabulary token in an adjacency matrix that every verification pass overwrites and that is retained across requests (Luo et al., 2025). Such a key merges all conditions that end in the same token, a mismatch we call condition aliasing. Along the prefix, her→head, cached after “a red hat covers her”, is reused after “blue shoes protect her”, where the target selects feet (Figure 1). Along the image, a prefix such as “the sign reads” must continue with whatever text the current image shows. Reuse also fails at image-sensitive states: in counterfactual diagnostics on 700 Qwen2.5-VL images, replacing the image changes the root prediction for 34.9% of fully rejected drafts but for only 12.4% of drafts that accept five or more tokens.
Figure 1. The role of context in prediction recycling. Top: a cached transition fails when reused after a different prefix. Middle: shorter accepted paths accompany stronger root image sensitivity (left), and source preference varies with candidate visual alignment (right). Bottom: RECAP retrieves and fuses predictions at multiple contextual resolutions.
Longer keys reduce aliasing but recur less often: they recall the target token more reliably when they match, yet match far fewer decoding states. We call this tension the specificity–coverage trade-off. Suffix-matching methods such as Prompt Lookup Decoding and SAM Decoding condition on longer contexts ([10] (Somasundaram et al. 2025)), but each match copies a single observed continuation rather than combining target predictions obtained under several context lengths.
The two parts of the condition call for separate controls. Key length distinguishes prefixes. Longer keys separate prefixes that a token key aliases, while shorter keys remain necessary for coverage. Neither resolution dominates: whether contextual or token-level candidates yield longer accepted prefixes depends on which carries more visual support (Appendix § 11), so candidate selection should combine resolutions. Memory lifetime distinguishes images. A lexical key cannot encode the image, but the image is fixed within a request, so request-local memory conditions its predictions on that image implicitly. Token transitions, in contrast, recur across requests, so token-level memory remains useful when retained across a stream.
We propose RECAP (REcycling Context-Aware Predictions), a training-free framework built on these two controls (Figure 2). It stores target predictions under one-, two-, and three-token keys; the two contextual maps live for a single request, whereas the token-level map persists across requests. Because a specific row may be diffuse while a shorter row is sharply peaked, RECAP does not simply prefer the longest key: it weights each retrieved row by a specificity prior scaled by an entropy-based confidence and mixes the rows, reinforcing shared candidates while retaining confident alternatives. Best-first search composes the fused transitions into a candidate tree, and a suffix automaton adds short continuations from the prompt and committed history. One target pass verifies
Figure 2. The RECAP framework. (A) Three-, two-, and one-token keys retrieve Top- prediction rows from , , and ; only survives request boundaries. (B) Entropy-confidence weighting fuses the retrieved rows under specificity priors and normalizes the retained candidates. (C) Best-first tree search repeats retrieval and fusion under each node’s path context. (D) Suffix retrieval inserts a contiguous continuation from the prompt and committed history. (E) One packed target pass commits the matching path plus one target-selected token. All evaluated nodes refresh prediction memory; only committed tokens extend sequence history. Tokens and scores are illustrative.
the tree, commits the longest matching path, and writes the predictions of every evaluated node back to memory.
Our contributions are as follows:
We identify condition aliasing in multimodal prediction recycling and show that token-indexed reuse fails most often at image-sensitive states.
We propose RECAP, a training-free framework that distinguishes prefixes through multi-length keys with entropy-confidence fusion and images through request-local and persistent memories.
RECAP achieves mean speedups of 1.66× and 1.56× on Qwen2.5-VL-7B and LLaVA-1.6-7B across seven single-turn benchmarks, and 1.89× and 1.59× on MMSpec, exceeding the strongest training-free baseline in mean speedup.
Related Work
Speculative decoding. Speculative decoding separates proposal generation from parallel target verification ([14, 5]). Learned proposers exploit target representations: Medusa adds future-token heads with tree verification ([4]), and Hydra models dependencies between head predictions ([2]). EAGLE drafts autoregressively in feature space ((Li et al., 2024b)), EAGLE-2 adapts draft trees to context-dependent confidence ((Li et al., 2024a)), and EAGLE-3 combines multi-layer features with direct token prediction ((Li et al., 2026)). RECAP keeps the propose–verify interface but drafts from memory of predictions the target has already evaluated.
Multimodal speculative decoding. Visual evidence adds a conditioning requirement to multimodal drafting: a drafter must propose tokens that are consistent with both the textual context and the visual input. Existing approaches address this challenge through different mechanisms. MSD separates text and visual token processing and trains a drafter with a text-to-multimodal curriculum ((Lin et al., 2025)); DREAM transfers refined target features through cross-attention ([9]); ViSpec adds a vision adaptor to drafting ([12]); SpecVLM combines EAGLE-style drafting, adaptive visual compression, and online-logit distillation ([11]). Together, these methods explore how visual information can be incorporated into a learned drafting process. MMSpec evaluates such approaches across multimodal task categories ((Shen et al., 2026)). RECAP instead obtains visual conditioning from cached target predictions and request-local memory, reusing information produced under the current multimodal context without training a multimodal proposer.
Training-free speculative decoding. Training-free methods derive proposals from existing sequences and target computations. Prompt Lookup Decoding reuses repeated prompt substrings ((Somosundaram et al., 2025)); REST retrieves suffix-matched continuations from an external datastore ([8]); SAM Decoding finds longest-suffix matches in static corpora and generation history ([10]); SuffixDecoding builds frequency-scored trees from current and previous outputs ((Oliaro et al., 2026)). Lookahead generates and verifies -grams through parallel iteration ([6]), and Token Recycling forms draft trees from token-indexed predictions ((Luo et al., 2025)). Related memory mechanisms include neural caches ([7]), kNN-LMs ([13]), and hierarchical language models that support sparse longer contexts with shorter-context distributions ((Teh, 2006)). RECAP combines multi-resolution lexical keys with request-local and persistent scopes, while the target retains control of commitment.
Context-Aware Prediction Recycling
Motivation
We examine how reuse depends on visual context through the 700-image counterfactual diagnostics in Figure 1 (Appendix § 11). With each text prefix fixed, target predictions are recomputed under mean-image and matched wrong-image controls, and results are averaged equally across benchmarks.
Evidence 1 relates draft acceptance to root sensitivity. With iterations grouped by accepted length, bars show root Top-1 changes under either control, and the curve gives the smaller of the two decreases in the original root token’s log probability. Rejected drafts score higher on both measures than 5+-token acceptances.
Evidence 2 compares sources at equal candidate budgets. With the prefix and candidate set fixed, visual support is the smaller of the two reductions, under the image controls, in the log of the total target probability assigned to source ’s candidates. Relative support between contextual () and token-only () candidates defines within-benchmark quintiles from token-only-aligned Q1 to context-aligned Q5. Bars show softmax-normalized support shares, and the curve shows contextual minus token-only accepted-prefix length. Its sign reversal indicates that the better source changes with candidate alignment, motivating memory that retains multiple context resolutions and fuses their evidence.
Problem Formulation
Let be the image, the prompt, and the generated prefix, with textual history . Greedy decoding with the VLM over vocabulary selects
Speculative decoding verifies a proposed sequence or tree in parallel, commits the longest prefix matching Equation (1), and emits one additional target-selected token ([14]), so accepting a draft tokens yields a output tokens per pass. RECAP preserves this commitment rule while building proposals from recycled target predictions.
Context-Stratified Prediction Memory
At each evaluated position, the target assigns logit to token . We retain the highest-scoring tokens as . A stored prediction row consists of and its normalized distribution
with zero mass on all other tokens. Each row thus inherits the conditioning of its source image and prefix.
Rows are addressed by the last one, two, or three tokens of . For suffix , these keys retrieve , , and . Thus, covers her, warms her, and protect her share the token key her but access distinct contextual rows (Figure 1), reducing condition aliasing.
Context refinement. Let be the final history tokens, and let denote the maximum expected recall of the target’s greedy token for selectors using only and at most candidates. Since is a deterministic function of ,
A finer-context selector can reproduce every coarse-context decision, but in a finite memory finer keys are observed less often; retaining shorter rows supplies coverage where longer keys are missing, as in hierarchical language models ((Teh, 2006)). Appendix F gives the proof.
Memory scope. Contextual maps and reset per request, making the current image and prompt implicit parts of the retrieval condition, whereas persists within an inference stream and supplies earlier transitions when local memory is sparse. Prefill seeds , tree verification refreshes all three maps, and each key retains its most recently assigned row.
Entropy-Confidence-Weighted Fusion
Let denote the rows available for history , ordered from most to least specific (, , ). RECAP fuses them by entropy-confidence weighting: each row’s weight is the product of a specificity prior, with , and an entropy-based confidence, defined as the row’s inverse effective support:
where . The prior favors specific matches, and confidence favors concentrated predictions. Priors follow specificity rank among available rows: if only and match, they receive and . Weights are recomputed at every tree node without additional target evaluations or learned parameters. Confidence measures concentration under the source condition, not direct alignment with the current image.
The normalized weights define the mixture
We bound branching by selecting the Top- tokens , using the same as row storage, and renormalizing:
with all other probabilities set to zero. A single retrieved row recovers ordinary lookup, and no match terminates expansion. Setting for all available rows yields the fixed-prior special case, the w/o confidence configuration analyzed in Appendix I.
Weighted KL center. For fixed , the weights do not depend on the candidate distribution , so the mixture is the weighted Kullback–Leibler (KL) center ([1]):
where is the probability simplex. Top- retention maximizes the preserved mixture mass, and the normalized proposal minimizes over distributions supported on at most tokens (Appendix § 12).
Tree Construction and Verification
At every node, RECAP retrieves and fuses rows using the established prefix together with proposed ancestors. Best-first construction orders paths by the score
where supplies the path context and controls depth-dependent expansion, until the node or depth budget is reached. Composing local predictions yields candidate sequences never stored as complete continuations.
A suffix automaton over the prompt and committed history retrieves the continuation following a match to the current suffix (Hu et al., [10]) and inserts it into the tree by sharing common prefixes, preserving observed sequence dependencies without modifying the fused transition scores.
The target verifies the joint tree in one packed pass, with ancestral attention confining each node to the established prefix and its own path. The longest path matching Equation (1) is committed together with one target-selected token; if no candidate agrees, the target directly supplies the next token.
Predictions from all evaluated nodes, including uncommitted branches, update , , and under their path contexts, whereas only committed tokens enter the suffix automaton. An uncommitted branch thus contributes a target prediction valid under its hypothetical prefix, which may become useful when a compatible context recurs. Appendix § 7 provides implementation settings and hyperparameters.
Experimental Evaluation
Setup
Models and workloads. Evaluation uses Qwen2.5-VL-7B-Instruct (Bai et al., [3]) and LLaVA-1.6 Vicuna-7B (Liu et al., 2024a) on an NVIDIA A100-SXM4-80GB GPU. Generation is greedy at batch size one, with at most 200 new tokens. We fix 1,000 inputs for each single-turn benchmark: MMT-Bench (Ying et al., 2024), SEED-Bench (Li et al., 2023), ScienceQA (Lu et al., 2022), OCRBench (Liu et al., 2024b), ChartQA (Masry et al., 2022), MathVista (Lu et al., 2024), and TextVQA (Singh et al., 2019). MMSpec adds 600 inputs from six categories, extending coverage to multi-turn conversations (Shen et al., 2026). For the main results, each benchmark, including MMSpec, forms one. inference stream: persistent is cleared after warmup and before each benchmark, then carried across that benchmark’s inputs, whereas and reset per request.
Baselines. We compare six draft-model-free approaches: Prompt Lookup Decoding (PLD; Somasundaram et al., 2025), Lookahead Decoding (Fu et al., [6]), Token Recycling (Luo et al., 2025), Tree Recycling, SAM Decoding (Hu et al., [10]), and SAM+ViSkip (Shen et al., 2026). Token Recycling and Tree Recycling run token-indexed recycling (Luo et al., 2025) on the same prefill and verification path as RECAP. Token Recycling follows the published configuration: a Top-8 adjacency matrix retained across requests drafts a static 80-node tree, and the matrix is cleared after warmup and at each benchmark boundary, as is . Tree Recycling is a lighter variant that expands a smaller static tree from a request-local Top-4 matrix. All methods use fixed input manifests and an autoregressive reference from the same backbone. Their settings are listed in Appendix § 7.
Metrics. We report speedup as the mean of paired throughput ratios, defined by
For each generation unit, and are the output-token count and total generation time for method . The autoregressive reference is denoted AR, and is the number of generation units.
Our acceptance length includes the additional target-selected token:
where averages draft tokens accepted per verification iteration for unit .
When target evaluations are exact, RECAP preserves greedy decoding without loss.
Main Results
RECAP leads every single-turn model–benchmark comparison in speedup and acceptance (Table 1). Qwen’s mean speedup across seven benchmarks is 1.66×, compared with 1.51× for Token Recycling, the strongest baseline by mean speedup. LLaVA’s corresponding values are 1.56× and 1.37×. The gains cover reasoning, chart understanding, and text-centric inputs: Qwen reaches 2.19× on MathVista and 1.75× on ChartQA, while LLaVA reaches 1.81× on OCRBench and 1.71× on MathVista. Alongside its consistent lead in , RECAP has the highest throughout. Its mean acceptance lengths of 3.02 and 2.92 exceed Token Recycling’s 2.71 and 2.65, the strongest baseline acceptance averages.
| Models | Methods | MMT | SEED | SciQA | OCR | ChartQA | MathVista | TextVQA | Average | ||||||||
| Qwen2.5-VL 7B-Instruct | PLD | 1.16 | 1.19 | 0.89 | 1.05 | 1.24 | 1.13 | 1.22 | 1.20 | 1.14 | 1.25 | 1.49 | 1.22 | 0.97 | 1.09 | 1.16 | 1.16 |
| Lookahead | 1.15 | 1.60 | 0.93 | 1.35 | 1.44 | 2.14 | 1.11 | 1.62 | 1.10 | 1.62 | 1.20 | 1.60 | 0.94 | 1.26 | 1.12 | 1.60 | |
| Tree Recycling | 1.20 | 1.47 | 1.07 | 1.49 | 1.23 | 1.54 | 1.29 | 1.68 | 1.27 | 1.72 | 1.41 | 1.70 | 1.06 | 1.41 | 1.22 | 1.57 | |
| Token Recycling | 1.59 | 2.57 | 1.25 | 2.58 | 1.60 | 2.60 | 1.51 | 2.82 | 1.54 | 2.89 | 1.86 | 2.84 | 1.21 | 2.63 | 1.51 | 2.71 | |
| SAM | 1.11 | 1.24 | 1.06 | 1.34 | 1.15 | 1.36 | 1.20 | 1.33 | 1.23 | 1.57 | 1.21 | 1.38 | 1.05 | 1.26 | 1.14 | 1.35 | |
| SAM+ViSkip | 1.04 | 1.25 | 0.99 | 1.27 | 1.07 | 1.29 | 1.04 | 1.28 | 1.16 | 1.50 | 1.14 | 1.36 | 0.99 | 1.22 | 1.06 | 1.31 | |
| RECAP | 1.69 | 2.81 | 1.27 | 2.78 | 1.72 | 2.84 | 1.67 | 3.31 | 1.75 | 3.30 | 2.19 | 3.27 | 1.34 | 2.81 | 1.66 | 3.02 | |
| LLaVA-1.6 Vicuna-7B | PLD | 1.01 | 1.34 | 0.91 | 1.14 | 1.09 | 1.23 | 1.28 | 1.53 | 1.02 | 1.34 | 1.19 | 1.40 | 0.89 | 1.11 | 1.06 | 1.30 |
| Lookahead | 0.80 | 1.74 | 0.70 | 1.48 | 0.94 | 2.11 | 0.91 | 2.11 | 0.74 | 1.66 | 0.79 | 1.70 | 0.64 | 1.31 | 0.79 | 1.73 | |
| Tree Recycling | 1.23 | 1.48 | 1.15 | 1.49 | 1.22 | 1.43 | 1.40 | 1.82 | 1.28 | 1.60 | 1.37 | 1.69 | 1.13 | 1.34 | 1.25 | 1.55 | |
| Token Recycling | 1.40 | 2.48 | 1.24 | 2.49 | 1.36 | 2.47 | 1.49 | 2.69 | 1.42 | 2.66 | 1.49 | 2.65 | 1.23 | 2.41 | 1.37 | 2.55 | |
| SAM | 1.13 | 1.26 | 1.11 | 1.34 | 1.12 | 1.24 | 1.24 | 1.45 | 1.18 | 1.36 | 1.23 | 1.40 | 1.07 | 1.19 | 1.16 | 1.32 | |
| SAM+ViSkip | 1.10 | 1.26 | 1.07 | 1.34 | 1.09 | 1.24 | 1.20 | 1.45 | 1.15 | 1.36 | 1.20 | 1.40 | 1.04 | 1.19 | 1.12 | 1.32 | |
| RECAP | 1.63 | 2.77 | 1.31 | 2.82 | 1.56 | 2.70 | 1.81 | 3.40 | 1.58 | 3.06 | 1.71 | 3.11 | 1.29 | 2.59 | 1.56 | 2.92 |
Table 1. Speedup (×) and acceptance length across seven single-turn benchmarks at temperature 0. SciQA denotes ScienceQA and OCR denotes OCRBench. averages paired generation-throughput ratios against autoregressive decoding; averages accepted draft tokens plus one target-selected token. Higher is better for both metrics.
On MMSpec (Table 2), RECAP reaches overall speedups of 1.89× on Qwen and 1.59× on LLaVA, exceeding Token Recycling, the strongest baseline on both models, at 1.56× and 1.43×. RECAP leads in both speedup and acceptance length across catcategories except Qwen text VQA, where Token Recycling reaches 1.31× against 1.26×, while both methods round to . Overall acceptance lengths for RECAP are 2.77 and 2.85. Multi-turn conversation yields 2.09× on Qwen, second only to complex reasoning at 2.16×, and 1.73× on LLaVA, its largest category speedup.
| Models | Methods | General VQA | General VQA | Text VQA | Text VQA | Captioning | Captioning | Charts | Charts | Reasoning | Reasoning | Multi-turn | Multi-turn | Overall | Overall |
| Qwen2.5-VL 7B-Instruct | PLD | 1.06 | 1.05 | 1.21 | 1.13 | 1.14 | 1.15 | 1.20 | 1.31 | 1.28 | 1.34 | 1.34 | 1.27 | 1.23 | 1.22 |
| Qwen2.5-VL 7B-Instruct | Lookahead | 1.11 | 1.37 | 1.04 | 1.29 | 1.44 | 1.64 | 1.21 | 1.77 | 1.91 | 2.28 | 1.39 | 1.80 | 1.36 | 1.71 |
| Qwen2.5-VL 7B-Instruct | Tree Recycling | 1.12 | 1.40 | 1.10 | 1.32 | 1.22 | 1.37 | 1.21 | 1.64 | 1.38 | 1.66 | 1.25 | 1.56 | 1.22 | 1.51 |
| Qwen2.5-VL 7B-Instruct | Token Recycling | 1.51 | 2.57 | 1.31 | 2.49 | 1.80 | 2.51 | 1.38 | 2.60 | 1.77 | 2.64 | 1.57 | 2.60 | 1.56 | 2.57 |
| Qwen2.5-VL 7B-Instruct | SAM | 1.05 | 1.14 | 1.13 | 1.13 | 1.03 | 1.07 | 1.14 | 1.54 | 1.18 | 1.31 | 1.08 | 1.23 | 1.10 | 1.24 |
| Qwen2.5-VL 7B-Instruct | SAM+ViSkip | 0.97 | 1.16 | 1.00 | 1.12 | 0.95 | 1.06 | 1.06 | 1.55 | 1.07 | 1.29 | 0.99 | 1.24 | 1.00 | 1.24 |
| Qwen2.5-VL 7B-Instruct | RECAP | 1.77 | 2.69 | 1.26 | 2.49 | 2.00 | 2.60 | 1.86 | 3.02 | 2.16 | 2.97 | 2.09 | 2.80 | 1.89 | 2.77 |
| LLaVA-1.6 Vicuna-7B | PLD | 1.03 | 1.10 | 1.06 | 1.08 | 1.09 | 1.17 | 1.21 | 1.54 | 1.37 | 1.79 | 1.33 | 1.82 | 1.21 | 1.48 |
| LLaVA-1.6 Vicuna-7B | Lookahead | 0.93 | 1.41 | 0.85 | 1.25 | 0.87 | 1.34 | 1.02 | 1.66 | 1.03 | 1.72 | 0.90 | 1.43 | 0.92 | 1.46 |
| LLaVA-1.6 Vicuna-7B | Tree Recycling | 1.12 | 1.43 | 1.17 | 1.40 | 1.19 | 1.34 | 1.34 | 1.70 | 1.37 | 1.77 | 1.35 | 1.64 | 1.27 | 1.56 |
| LLaVA-1.6 Vicuna-7B | Token Recycling | 1.21 | 2.42 | 1.35 | 2.48 | 1.50 | 2.50 | 1.43 | 2.58 | 1.46 | 2.57 | 1.53 | 2.65 | 1.43 | 2.55 |
| LLaVA-1.6 Vicuna-7B | SAM | 1.07 | 1.23 | 1.06 | 1.16 | 1.04 | 1.09 | 1.24 | 1.46 | 1.22 | 1.49 | 1.16 | 1.28 | 1.14 | 1.29 |
| LLaVA-1.6 Vicuna-7B | SAM+ViSkip | 1.04 | 1.23 | 1.04 | 1.16 | 1.02 | 1.09 | 1.21 | 1.46 | 1.20 | 1.49 | 1.14 | 1.29 | 1.12 | 1.29 |
| LLaVA-1.6 Vicuna-7B | RECAP | 1.26 | 2.60 | 1.44 | 2.64 | 1.65 | 2.63 | 1.65 | 3.10 | 1.67 | 3.09 | 1.73 | 2.93 | 1.59 | 2.85 |
Table 2. MMSpec speedup (×) and acceptance length at temperature 0, with the metric definitions of Table 1. Each category contains 100 inputs; multi-turn conversations contribute 223 generation turns, giving 723 total turns. Overall values are turn-weighted averages of both metrics, rather than unweighted category means.
Comparison with trained drafters. Table 3 contrasts RECAP with released DREAM Hu et al., [9] and ViSpec Kang et al., [12] drafters on the same single-turn inputs. Trained drafters accept longer continuations, with mean of 3.68–3.76 against 2.92 and 3.02 for RECAP, which reaches 78–80% of their acceptance length. Their speedups of 1.05–1.26× fall below those of RECAP, but they were measured against autoregressive baselines of 41.9–61.2 tokens/s rather than 29.9–39.4 tokens/s, so this gap does not establish that RECAP decodes faster. Their acceptance advantage requires a training stage for each target model and adds 451M–890M parameters, or 0.84–1.66 GiB of 16-bit weights, to the serving GPU. RECAP adds no parameters or GPU-resident state; at the default budget, its proposal state occupies at most 8.70 MiB of host memory (Table 10). It therefore transfers to a new VLM without data collection, training, or a drafter checkpoint.
| Model | Method | Params | GiB | AR | ||
| LLaVA | DREAM | 607M | 1.13 | 3.68 | 1.26 | 41.9 |
| ViSpec | 451M | 0.84 | 3.74 | 1.05 | 61.2 | |
| RECAP | 0 | 0 | 2.92 | 1.56 | 29.9 | |
| Qwen | ViSpec | 890M | 1.66 | 3.76 | 1.18 | 57.0 |
| RECAP | 0 | 0 | 3.02 | 1.66 | 39.4 |
Table 3. Deployment cost and decoding performance on the seven single-turn benchmarks in Table 1. Params and GiB denote drafter parameter counts and 16-bit GPU weight storage; AR is mean autoregressive throughput (tokens/s). DREAM and ViSpec use released checkpoints with top-4, depth-6, 32-node trees and a faster AR harness, so their is not directly comparable with that of RECAP.
| Model | Cap | Mean nodes | Prep. ms | ms | Host MiB | GPU GiB |
| Qwen | 31 | 26.5 | 0.591 | 0.722 | 5.79 | 19.32 |
| Qwen | 63 | 55.2 | 0.907 | 1.227 | 7.88 | 19.32 |
| Qwen | 127 | 113.4 | 2.202 | 2.617 | 8.74 | 19.32 |
| LLaVA | 31 | 29.1 | 0.721 | 0.882 | 7.46 | 19.98 |
| LLaVA | 63 | 59.2 | 1.121 | 1.385 | 8.70 | 20.01 |
| LLaVA | 127 | 121.0 | 2.485 | 2.729 | 12.68 | 20.16 |
Table 10. Budget sensitivity with shared inputs and orders. Candidate counts include suffixes and exclude the root. Preparation covers lookup through packing; is the request-average 95th percentile. Host memory is the largest proposal-state snapshot. GPU memory is peak model-process allocation, including weights and generation workspaces.
Analysis of Prediction Reuse
Component Contributions
Matched-budget ablations test which parts of the design contribute to throughput (Table 4). The MMSpec subset comprises 120 inputs, 20 per category, expanding to 160 generation turns, and is evaluated with Qwen2.5-VL-7B. Policies share a tree builder and a 63-node transition budget. Reported gains average input-paired throughput changes after pooling each input’s turn-level token counts and times. Except for the confidence comparison, these controls use the fixed-prior special case (), so each comparison isolates one component.
| Comparison | Gain (%, ↑) |
| Candidate search | |
| Best-first vs. fixed expansion | +4.90 |
| Memory design | |
| Context backoff vs. unigram | +3.03 |
| Uniform fusion vs. backoff | +0.75 |
| Specificity priors vs. uniform fusion | +2.80 |
| Entropy confidence vs. fixed priors† | +0.73/+1.09 |
| Persistent vs. reset | +5.42 |
| Sequence reuse | |
| Suffix insertion vs. none† | +0.61 |
Table 4. Paired throughput effects of individual components. Unmarked rows use the 120-input MMSpec subset with Qwen2.5-VL-7B; the persistence row averages five runs. † marks four-dataset controls (200 inputs per model for confidence, reported as Qwen/LLaVA; 400 inputs for suffix insertion). Backoff selects the most specific row available at a query.
Contextual backoff improves throughput over unigram recycling by 3.03%, and an independent repeat of the same 120-input comparison yields 2.42%. Uniform fusion improves backoff by 0.75%, and specificity priors improve uniform fusion by 2.80%. Across five runs, retaining token memory across requests improves throughput over per-request reset by 5.42% on average, ranging from 4.36% to 7.18%. Request-stream controls are detailed in Appendix § 10.
In four-dataset controls, entropy-confidence weighting adds a smaller gain over fixed priors: 0.73% mean throughput on Qwen and 1.09% on LLaVA. Confidence-based single-source gating decreases throughput by 0.66% and 1.90%, respectively. The contrast supports combining source evidence in these settings while qualifying the size of the confidence benefit. Appendix § 15 details the adaptive rules and fixed-prior sensitivity.
Suffix complementarity is tested separately on 100 inputs each from ChartQA, TextVQA, OCR-Bench, and MMT-Bench. Adding suffix insertion to fusion improves mean paired throughput by 0.61%, with dataset-specific changes from −0.67% to +2.23% (Table 7).
| Dataset | Fusion | + Suffix | Difference | Gain (%, ↑) | : before → after |
| ChartQA | 1.123 | 1.313 | 0.190 | +0.83 | 2.80 → 2.88 |
| TextVQA | 0.355 | 0.413 | 0.059 | +0.07 | 2.18 → 2.18 |
| OCRBench | 1.417 | 1.538 | 0.120 | −0.67 | 2.66 → 2.62 |
| MMT-Bench | 0.750 | 0.866 | 0.116 | +2.23 | 2.40 → 2.44 |
Table 7. Suffix-retrieval controls. Offline matching omits root and correction tokens; online gains and acceptance length use Equations (9)–(10).
Specificity Improves Matching; Backoff Retains Coverage
Replay comprises 23,811 states from 400 inputs, with 100 each from ChartQA, TextVQA, OCRBench, and MMT-Bench. Retrieval begins with empty memories and accesses only rows from earlier states. Query coverage is the fraction of states with a matching row. Conditional Recall@8 is recall of the reference next token among those covered states.
Stored rows preserve useful source-image information. To test this, we fix the query image, question, and prefix and vary the image used to compute earlier predictions. On jointly covered queries with eight candidates per source, matching-image rows outperform swapped-image rows in fused Recall@8 by 0.89–6.35 percentage points across datasets (Appendix § 9).
All four datasets show higher conditional recall and lower coverage with finer keys (Figure 3(a)). On OCRBench, coverage is 54.9% and coverage is 18.6%, while recall rises from 63.6% to 90.6%. In a prompt-seeded variant of this replay, restricting evaluation to states where both and match, improves Recall@8 over by 5.00–29.54 percentage points across datasets.
Figure 3. The specificity–coverage trade-off and contextual backoff. (a) Query coverage and conditional Recall@8 across 23,811 states from 400 inputs. Each benchmark connects (circle), (square), and (diamond). Finer keys raise conditional recall while reducing coverage. (b) Paired online throughput gains of backoff against .
At equal actual candidate counts, backoff also improves matched continuation length on every dataset, from 0.516 to 0.709 tokens on ChartQA and 0.632 to 1.135 on OCRBench (Appendix § 9). This offline metric measures the longest tree prefix agreeing with the saved target continuation, excluding the root and target correction token.
Online controls yield 1.11–3.64% paired throughput gains from backoff (Figure 3(b)). Adding improves offline matching further on ChartQA and MMT-Bench, with only small throughput changes online. Context refinement therefore supplies more precise candidates, and backoff preserves the coverage needed for their practical reuse.
Complementary Sequence Reuse
Suffix retrieval contributes sequence continuations beyond those formed by local transition composition. Starting with prompt-seeded , we compare fusion alone with suffix-augmented fusion at equal actual candidate counts. The added suffix increases mean matched length by 0.059–0.190 tokens across four replay datasets (Appendix § 9). The matched-budget result attributes the improvement to complementary sequence evidence.
Online throughput changes from suffix insertion range between −0.67% and +2.23% across the four datasets. Timing includes retrieval, insertion, and overlap handling. At the default budget, retrieval, fusion, tree construction, and packing together require 0.907 ms per Qwen iteration and 1.121 ms per LLaVA iteration. Including memory refresh, suffix-index updates, token commitment, and other host-side decoding work, all stages outside target verification take 1.778 and 1.974 ms, or 8.20% and 5.61% of decoding wall time (Appendix § 13).
Conclusion
RECAP addresses contextual specificity and reuse scope jointly in multimodal prediction recycling, fusing request-local contextual memories with persistent token-level memory through entropy-confidence weighting without training or an auxiliary draft model. It reaches mean generation-throughput speedups of and across seven single-turn benchmarks on Qwen2.5-VL-7B and LLaVA-1.6-7B, and and on MMSpec. Controlled analyses connect these gains to more precise contextual matching and reuse across request boundaries.
Limitations
Our evidence covers greedy, batch-one generation with image-based inputs on two 7B VLMs. Larger models, concurrent serving, and video remain untested operating regimes. Default specificity priors, confidence weighting, and tree settings may yield different latency–memory trade-offs across hardware and request lengths. The reported efficiency concerns generation throughput and does not establish downstream task accuracy or application-level latency distributions.
Ethical considerations
This work studies inference efficiency using existing models and public evaluation data and introduces no new human-subject data collection. Deployments should follow the applicable model and dataset licenses and isolate online prediction memories at appropriate user or service boundaries. Faster inference does not remove the need to assess outputs for the intended application.
Implementation and Baseline Settings
Model execution. Qwen and LLaVA run in bfloat16 and float16, respectively, on the workloads of § 4.1. Timing covers prefill and decoding, excluding data loading and report aggregation.
Recap defaults. We use Top-8 rows, width 8, depth 10, and 63 transition candidates excluding the root. Available rows receive priors (0.70, 0.20, 0.10) by decreasing specificity; entropy-confidence weighting multiplies each prior by the row’s confidence and normalizes the products (Equation (4)). Depth priors are (0.68, 0.63, 0.50, 0.43, 0.40, 0.38), then 0.38. Prefix-sharing suffix insertion adds at most four tokens beyond the transition cap, giving at most 68 packed nodes including the root; controls described as suffix-inclusive instead count suffix nodes within the 63-node cap. Special image tokens and duplicate children are filtered.
Memory updates and packed verification. Prefill seeds from each eligible prompt token’s first occurrence; start empty. All evaluated rows refresh memory, retaining each key’s first batch occurrence and replacing older entries. Packed positions follow path depth; accepted-path key/value entries continue decoding. Stream and policy states are isolated and reset after warmup; in the main results, is also cleared at each benchmark boundary, so no predictions carry over between benchmarks.
Baselines. PLD uses 4-token keys and at most 10 draft tokens; Lookahead uses decoding/branch lengths 64/12. Token Recycling follows the official implementation (Luo et al., 2025): a zero-initialized Top-8 adjacency matrix fills the static imbalanced “2.2.2” tree (80 candidates, depth 6) layer by layer, and every verified node writes its Top-8 target predictions back to the matrix. Prefill logits are not written; the matrix persists across requests and is cleared only after warmup and at benchmark boundaries. Tree Recycling expands a static tree of width 2 and depth 4, giving 30 candidates under a 31-node cap, from a request-local Top-4 matrix. SAM uses total-token setting 64 and draft lengths 2–40. SAM+ViSkip caps drafts at 40 tokens and gates at 0.35 on head-averaged image attention at final-layer query zero.
Timing and Analysis Protocols
Fusion rule. The main results (Table 1 and Table 2) and the MMSpec latency analysis (Appendix § 16) use entropy-confidence weighting. Component analyses in § 5 and Appendices § 9, § 10, § 13, and § 14 use its fixed-prior special case (), so that each control varies one component; Appendix § 15 compares the two rules directly.
Pairing and timed boundaries. For the recycling methods (RECAP, Tree Recycling, and Token Recycling), SAM, and SAM+ViSkip, autoregressive references are interleaved with method runs; PLD and Lookahead pair generation units. Main methods each have a separate run and reference, while component controls share a run. Main recycling and MMSpec ablations synchronize at target forwards. SAM+ViSkip and four-dataset controls synchronize at generation boundaries; the four-dataset controls also share timing paths that include CPU candidate counters.
MMSpec controls. In MMSpec ablations, the ratio for input pools its multi-turn records:
Gains are reported as . Resampling and repeated request orders reuse the same inputs. Their variation therefore reflects this fixed evaluation set, not independent-dataset uncertainty.
Offline replay. Offline replay queries before updating, includes unmatched empty-memory states, and compares proposals with saved native target continuations. Trees respect a 63-node cap and are rebuilt to the smallest actual count among policies. Matched length excludes the root and target correction token. Suffix additions are included in the cap for controls; the default configuration allows four extra nodes.
Offline means weight replay states within each dataset and use 2,000 paired image-cluster resamples: 97 ChartQA clusters and 100 clusters elsewhere. Online resampling pairs inputs, using 100 per dataset or the first 50 for scope controls.
Candidate-Level Results
Table 5 gives the coverage and recall values in Figure 3; the corresponding online gains are reported below.
| Dataset | Coverage (%, ↑) | Recall@8 (%, ↑) | ||||
| ChartQA | 44.5 | 23.2 | 14.8 | 64.5 | 81.7 | 86.6 |
| TextVQA | 26.3 | 7.4 | 3.4 | 52.5 | 73.5 | 78.3 |
| OCRBench | 54.9 | 30.4 | 18.6 | 63.6 | 84.2 | 90.6 |
| MMT-Bench | 45.6 | 19.7 | 11.2 | 56.9 | 77.9 | 84.4 |
Table 5. Coverage and conditional Recall@8 during empty-initialization replay. Queries are evaluated before their corresponding memory updates.
With 100 paired inputs per dataset, shared initialization, and a 63-node cap, backoff improves throughput over by 2.29%, 1.11%, 3.64%, and 1.65% on ChartQA, TextVQA, OCRBench, and MMT-Bench, respectively.
Backoff versus fusion. Matched-node comparisons favor backoff over fusion on ChartQA and OCRBench, with similar continuation lengths elsewhere (Table 6).
| Policy | Matched continuation length () | |||
| ChartQA | TextVQA | OCR | MMT | |
| 0.516 | 0.224 | 0.632 | 0.444 | |
| backoff | 0.709 | 0.253 | 1.135 | 0.600 |
| backoff | 0.778 | 0.252 | 1.128 | 0.638 |
| Uniform fusion | 0.737 | 0.253 | 1.041 | 0.629 |
| Prior-weighted fusion | 0.751 | 0.253 | 1.066 | 0.639 |
Table 6. Matched continuation length in empty-initialization replay. Each state uses equal actual node counts across the compared policies. Prior-weighted fusion uses fixed priors ().
Fusion in online controls. With empty initialization and fixed priors, prior-weighted/uniform throughput ratios are 1.0043, 1.0015, 1.0006, and 1.0037 on ChartQA, TextVQA, OCRBench, and MMT-Bench. These near-neutral ratios qualify the gain observed in the prompt-seeded MMSpec control (Table 4) and show that specificity priors have setting-dependent benefits.
Suffix complementarity. At equal actual candidate counts under a 63-node cap, prompt-seeded suffix insertion adds first-token hits in 3.33%, 1.46%, 1.95%, and 2.31% of states (Table 7). Online policies are interleaved, bridge extension is disabled, and all suffix-related operations are timed.
Image-conditioned source predictions. Holding queries fixed and using eight candidates per source, jointly available matching-image rows increase fused Recall@8 over swapped-image rows by 5.55, 6.35, 1.91, and 0.89 points on ChartQA, TextVQA, OCRBench, and MMT-Bench. Donor images are category-matched on ChartQA/OCRBench and pooled on the other datasets. The context-only/trigram-only changes of / points on OCRBench show that longer-key sources do not benefit uniformly. No image-similarity gate is used.
Prediction Reuse across Request Streams
We test cross-task reuse by changing token-memory lifetime while holding contextual memory request-local. Persistent is compared with per-request reset, and and reset in both cases. The same 200 inputs are scheduled either in benchmark blocks or by alternating benchmarks.
Persistent memory improves throughput on ChartQA, OCRBench, and MMT-Bench under both schedules, with near-neutral effects on TextVQA (Figure 4). MMT-Bench gains are 7.98% for blocked streams and 7.49% for alternating streams. A task-boundary reset retains reuse within benchmarks but clears on each benchmark switch. It becomes equivalent to per-request reset under alternation, while persistent retains predictions through intervening tasks.
Figure 4. Token-level reuse across task schedules. Persistent is paired with per-request reset on identical inputs to measure throughput under blocked and alternating benchmark streams.
Within-dataset orders use seeds 17, 29, 43. The three memory policies maintain isolated states, reset per request, and omit suffix insertion.
Table 8 averages paired input ratios separately for each seed. TextVQA’s pooled ratios are 0.987 under blocked scheduling and 0.990 under alternation, indicating near-neutral effects sensitive to aggregation. Alternating task-reset and per-request-reset acceptance is identical across 600 pairs. Memory snapshots include final prediction objects and exclude peak process RAM.
| Dataset | Blocked | Alternating |
| ChartQA | 5.69 ± 0.59 | 3.91 ± 0.77 |
| TextVQA | 0.92 ± 1.10 | 0.68 ± 1.13 |
| OCRBench | 4.77 ± 1.04 | 4.32 ± 0.13 |
| MMT-Bench | 7.98 ± 1.12 | 7.49 ± 1.77 |
Table 8. Mean throughput gain ± sample SD for persistent versus reset, measured across three orders of the same 200 inputs.
The persistent map keeps only the latest Top- row for each token, bounding storage by per stream. On cyclic replay of a fixed Qwen trajectory pool, its host footprint remains near 2.67 MiB through 10,000 request updates.
Motivation Figure Diagnostics
Figure 1 uses Qwen2.5-VL-7B traces from 100 images per single-turn benchmark, totaling 700. We compute equal-benchmark averages and 10,000 within-benchmark image-cluster bootstrap resamples. The counterfactuals hold text prefixes fixed while recomputing representations with mean-image and matched wrong-image controls.
Accepted length and root sensitivity (left). Visual dependence is the smaller counterfactual decrease in the original root token’s log probability, and change rate records Top-1 changes under either control. Valid full-image anchors yield 15,677 drafts across 695 image clusters, grouped by accepted length into 0–4 and 5+. From rejection to 5+ acceptance, dependence declines 0.620 → 0.198 and root-change rate declines 34.9% → 12.4%.
Candidate-source alignment (right). For token-only candidates and contextual candidates , define the log candidate mass and visual support as
We retain 1,125 eligible states with valid full-image anchors, both sources, row overlap of at most 25%, and matched budgets of 2–8 candidates. Within-benchmark quintiles of run from -aligned Q1 to context-aligned Q5. Bars show . The corresponding contextual acceptance gains are −1.292, −0.375, 0.709, 1.239, and 2.084 tokens. Acceptance is not used to define the bins.
Properties of Prediction Reuse
Context refinement. Let be the greedy target token under fixed tie-breaking and the last history tokens, start-padded as needed. Optimal recall is
For , any coarse selector can be reproduced as , proving . Dropping the earliest token gives Equation (3). Finite memories still require shorter-key coverage.
Fusion and sparse selection. At a fixed query, let combine normalized rows with fixed positive weights summing to one. The KL decomposition
makes the unique minimizer in Equation (7), without row independence (Abbas, [1]).
Let retain the largest positive masses of (all if fewer), with deterministic ties, and . Then
For nonempty and supported on ,
Thus attains , and Top- maximizes retained mass , allowing tied optimal supports. These properties characterize proposals; target verification determines commitment.
Overhead and Memory Scaling
Latency and Workspace
Profiling uses the same A100-SXM4-80GB GPU and AMD EPYC 9654 CPU for both models, with PyTorch 2.7.1, CUDA 12.6, and four CPU threads. Ten inputs per diagnostic dataset follow Appendix A. Orders 17/29 interleave caps 31/63/127, resetting isolated states after warmup
and between repeats. Profiling uses the fixed-prior special case (); entropy-confidence weighting additionally evaluates one row entropy per retrieved row at multi-source queries. In an isolated host microbenchmark of the reference Python fusion over 512 multi-source queries, this raises fusion time from 7.3 to 14.5 s per query; the C++ builder used for the main results was not profiled separately.
We pool exclusive CUDA-synchronized phases over decoding iterations, excluding prefill and image encoding (Table 9). Packing includes topology, masks, positions, and verification inputs. Memory traversal and serialization are untimed. An uninstrumented 63-node stream measures instrumentation overhead, and four 64-token traces per model separately attribute CUDA kernel and memory-operation time.
| Stage | Qwen Wall | Qwen CPU | LLaVA Wall | LLaVA CPU |
| Suffix lookup | 0.010 | 0.016 | 0.010 | 0.016 |
| Lookup, fusion, search | 0.506 | 0.510 | 0.592 | 0.598 |
| Packing | 0.391 | 0.409 | 0.519 | 0.538 |
| Memory refresh | 0.461 | 0.468 | 0.414 | 0.421 |
| Suffix index update | 0.027 | 0.034 | 0.028 | 0.034 |
| Token commitment | 0.256 | 0.273 | 0.286 | 0.304 |
| Other decoding | 0.127 | 0.148 | 0.125 | 0.145 |
| Target verification | 19.912 | 19.928 | 33.241 | 33.265 |
Table 9. Decoding-phase times in milliseconds, pooled over iterations at a 63-node cap with 40 inputs per model and two orders. Wall times are exclusive; CPU time includes synchronization polling.
Stage costs. Preparation takes 0.907/1.121 ms on Qwen/LLaVA, or 4.18%/3.18% of decoding wall time. Adding memory refresh and suffix-index updates gives 1.395/1.563 ms (6.43%/4.44%), and all stages outside target verification total 1.778/1.974 ms (8.20%/5.61%). Instrumentation adds 1.01%/1.36% median latency. Retrieval, fusion, and search launch no CUDA work. Packing consumes 4.46/3.29 s of device time per iteration, while verification consumes 16.60/28.04 ms, representing 98.88%/99.61% of decoding-device time.
Budget sensitivity and workspace. For candidates, the ancestor submatrix contains entries. Table 10 deduplicates host objects shared across , the suffix index, and topology state.
Persistent-Memory Growth
One latest Top- row per token bounds persistent memory by . The resulting space cost is per stream and for streams, independent of request count. Request-local scale with the contexts evaluated within each request.
Replay of observed prediction rows. Cyclic replay reshuffles 400 native Qwen trajectories, 100 per diagnostic dataset, under seeds 17/29/43 at each pass. Top-8 memory reaches 3,821 keys and 2.67 MiB after 400 requests and stays stable through 10,000 updates (Figure 5). This result characterizes replacement on a fixed pool, not saturation under unseen inputs.
Figure 5. Persistent over cyclic replay of the same 400 trajectories. Curves report the mean and min–max range across three replay orders.
Vocabulary-capacity stress test. The capacity stress test fills synthetic Top-8 rows at three occupancies in fresh processes (Table 11). At full vocabulary occupancy, Qwen/LLaVA host storage is 109.14/23.36 MiB and RSS increases by 126.48/27.55 MiB. These capacity figures do not represent natural-stream footprints. Persistent rows contain no CUDA tensors.
| Model | Host MiB | Lookup ns | |
| Qwen | 1,000 | 0.67 | 109 |
| 10,000 | 7.03 | 141 | |
| 152,064 | 109.14 | 386 | |
| LLaVA | 1,000 | 0.67 | 109 |
| 10,000 | 7.03 | 140 | |
| 32,064 | 23.36 | 183 |
Table 11. Synthetic occupancy stress test: host-object storage and median lookup latency, measured over seven repeats of 100,000 seeded ID/score hits with a checksum.
Across all six occupancies, 63-node tree construction takes 0.242–0.249 ms, measured as the median of seven repeats of 128 queries with packing excluded.
Alternative Root-Level Retrieval
Qwen fallbacks use a fixed six-category MMSpec subset and activate only when all root lexical lookups fail; descendants retain lexical retrieval. Semantic fallback selects up to three rows by cosine similarity of verified and predecessor states, with no extra target forwards. A request-local FIFO index holds at most 2,048 surviving rows; recency fallback selects the latest. Both use fixed priors (0.70, 0.20, 0.10) without confidence () and Top-8 candidates.
Interleaved policies share target-generated histories, depth 10, a 63-candidate cap including suffixes but excluding the root, and isolated persistent and request-local . Timing includes image encoding, prefill, index maintenance, and decoding; metrics average turns.
Both increase but reduce speedup by 1.46% (semantic) and 1.34% (recency; Table 12). Semantic lookup activates in 4.87% of iterations despite continuous index maintenance. Neither fallback is enabled by default.
| Variant | (, ) | () |
| RECAP (lexical) | 1.63 | 2.67 |
| RECAP + semantic fallback | 1.61 | 2.78 |
| RECAP + recency fallback | 1.61 | 2.75 |
Table 12. Root fallback on MMSpec with Qwen2.5-VL-7B. All variants use fixed-prior fusion (). Longer accepted continuations accompany lower speedup. Best column values are bold.
Fusion Ablations
In Equation (4), w/o confidence sets (reference), w/o prior sets , and uniform fusion removes both factors. Hard gating selects the most confident row, breaking ties by specificity. All rules share retrieved rows, Top-8 truncation, and renormalization.
Offline matching. Replay uses earlier rows, prompt-seeded , and empty . RECAP changes all-query Recall@8 by less than 0.02 points; gating loses 0.14–1.99 points (Table 14). Without confidence, soft (0.55, 0.30, 0.15) and sharp (0.85, 0.10, 0.05) priors change recall by at most 0.17 points.
| Variant | ChartQA | TextVQA | OCR | MMT |
| RECAP | +0.00 | +0.00 | +0.01 | −0.01 |
| w/o confidence | +0.00 | +0.00 | +0.00 | +0.00 |
| w/o prior | −0.08 | +0.00 | −0.31 | +0.00 |
| Uniform | −0.08 | +0.00 | −0.30 | −0.02 |
| Confidence gate | −1.49 | −0.14 | −1.99 | −0.95 |
| Specificity backoff | −0.11 | +0.00 | +1.29 | −0.24 |
| w/o conf., soft prior | −0.04 | +0.00 | −0.16 | +0.00 |
| w/o conf., sharp prior | −0.02 | +0.03 | +0.15 | −0.05 |
Table 14. Changes in all-query Recall@8 relative to w/o confidence (percentage points), evaluated on 23,811 Qwen states from 400 inputs. Uncovered states count as misses. Shading identifies RECAP.
Online comparison. The 4,000 generations in Table 13 compare five rules with isolated memories, suffix insertion, and Appendix § 7’s settings. Policies rotate per request; timing includes image encoding, prefill, generation, and adaptation. Repeats reuse inputs to assess order variation. RECAP gains 0.73%/1.09% mean throughput on Qwen/LLaVA, with median paired changes of −0.12%/−0.07%. W/o prior (confidence only) and uniform fusion have mixed effects; gating lowers throughput on both models. Gains are modest and task-dependent. Confidence measures concentration; learned gates and visual-dependence signals remain untested.
| Variant | Qwen ChartQA | Qwen TextVQA | Qwen OCR | Qwen MMT | Qwen Mean | Qwen | LLaVA ChartQA | LLaVA TextVQA | LLaVA OCR | LLaVA MMT | LLaVA Mean | LLaVA |
| RECAP | +0.65 | +1.24 | +1.43 | −0.41 | +0.73 | 2.93 | +2.30 | +0.34 | +1.71 | +0.01 | +1.09 | 2.90 |
| w/o confidence | +0.00 | +0.00 | +0.00 | +0.00 | +0.00 | 2.91 | +0.00 | +0.00 | +0.00 | +0.00 | +0.00 | 2.88 |
| w/o prior | +0.37 | +0.56 | +1.38 | −0.93 | +0.35 | 2.91 | +0.26 | −0.67 | +0.31 | −0.92 | −0.25 | 2.85 |
| Uniform | +0.01 | +0.37 | +2.34 | +0.22 | +0.73 | 2.90 | −0.47 | +0.18 | +2.29 | −0.06 | +0.48 | 2.86 |
| Confidence gate | −1.10 | +1.52 | −1.33 | −1.72 | −0.66 | 2.83 | −2.27 | −1.56 | −1.25 | −2.53 | −1.90 | 2.76 |
Table 13. Online fusion ablations, reporting mean input-paired throughput changes (%) against w/o confidence and acceptance length . Each model processes 200 inputs, 50 per dataset, under two orders. Policies share memory scope, candidate limits, and suffix insertion; timing includes weighting. Full RECAP is shaded.
Disagreement among sources. We further examine replay states where at least two retrieved rows have different Top-1 predictions. On these states, RECAP exceeds confidence gating by 9.62–17.33 Recall@8 points (Figure 7; Table 16). Paired image-cluster intervals exclude zero on ChartQA, OCRBench, and MMT-Bench; TextVQA has only 52 such states and a wider interval crossing zero. These results support retaining complementary candidates when sources disagree. They isolate the benefit over hard source selection; the effect of confidence weighting relative to fixed priors is assessed separately above.
Figure 7. Fusion gains over confidence gating on Qwen replay states with source disagreement. Points show Recall@8 differences; bars give 95% confidence intervals from 2,000 paired image-cluster bootstrap resamples. The dashed line marks zero gain, and is the number of disagreement states.
| Dataset | States | RECAP | Confidence gate | Difference | 95% CI |
| ChartQA | 427 | 84.31 | 66.98 | +17.33 | [12.08, 22.31] |
| TextVQA | 52 | 75.00 | 65.38 | +9.62 | [−2.33, 18.42] |
| OCRBench | 857 | 62.43 | 46.91 | +15.52 | [9.69, 20.70] |
| MMT-Bench | 510 | 77.65 | 62.55 | +15.10 | [10.27, 20.14] |
Table 16. Recall@8 (%) on Qwen replay states with source disagreement. Differences and confidence intervals are in percentage points. Intervals use 2,000 paired image-cluster bootstrap resamples of the fixed evaluation set.
Observed fusion weights. Table 17 summarizes normalized weights from the online RECAP arm, grouping calls by the number of available rows. Each candidate-tree fusion call contributes equally, pooling both request orders; indices follow decreasing specificity among available sources. With three sources, pooled weights are (0.693, 0.199, 0.108) for Qwen and (0.688, 0.200, 0.112) for LLaVA, close to the prior (0.700, 0.200, 0.100). With two sources, the normalized prior is (0.778, 0.222), compared with observed means of (0.764, 0.236) and (0.763, 0.237). Thus confidence adjusts the allocation while preserving the average preference for specific sources. These averages describe aggregate weighting behavior; they do not imply small adjustments at every query.
| Model | Dataset | Two available sources | Three available sources | ||
| Calls | Mean | Calls | Mean | ||
| Qwen | ChartQA | 15,101 | (0.764, 0.236) | 12,072 | (0.697, 0.197, 0.105) |
| TextVQA | 12,689 | (0.772, 0.228) | 8,353 | (0.700, 0.197, 0.103) | |
| OCRBench | 20,883 | (0.758, 0.242) | 19,394 | (0.693, 0.199, 0.108) | |
| MMT-Bench | 44,809 | (0.765, 0.235) | 32,106 | (0.689, 0.200, 0.110) | |
| Pooled | 93,482 | (0.764, 0.236) | 71,925 | (0.693, 0.199, 0.108) | |
| LLaVA | ChartQA | 33,922 | (0.764, 0.236) | 20,304 | (0.694, 0.198, 0.108) |
| TextVQA | 17,023 | (0.770, 0.230) | 6,563 | (0.700, 0.199, 0.102) | |
| OCRBench | 29,778 | (0.759, 0.241) | 32,512 | (0.687, 0.200, 0.113) | |
| MMT-Bench | 44,589 | (0.763, 0.237) | 36,050 | (0.685, 0.201, 0.115) | |
| Pooled | 125,312 | (0.763, 0.237) | 95,429 | (0.688, 0.200, 0.112) |
Table 17. Mean normalized fusion weights in the online ablation, averaged over calls at all candidate-tree positions. Each model uses 200 inputs under two orders. Source indices denote specificity ranks among available rows. Single-source calls have weight one and are omitted.
Per-Input Runtime Distributions
MMSpec latency. Following MMSpec (Shen et al., 2026), we compare paired AR/RECAP generation times from Table 2. Input contributes over turns. Each includes prefill and decoding, excluding prompt preparation and output text decoding. All 600 inputs enter without filtering.
RECAP lowers median and upper-tail latency (Figure 6; Table 15). Qwen/LLaVA P95 falls by 47.69%/42.17%, and 87.33%/97.83% of inputs are faster than paired AR.
| Model | Method | P50 | P90 | P95 | P99 |
| Qwen | AR | 2.70 | 8.44 | 11.63 | 15.07 |
| Qwen | RECAP | 1.52 | 4.46 | 6.08 | 7.56 |
| LLaVA | AR | 4.02 | 9.42 | 14.27 | 18.43 |
| LLaVA | RECAP | 2.38 | 5.45 | 8.25 | 10.67 |
Table 15. MMSpec per-input generation-latency percentiles in seconds, with lower values indicating faster generation. denotes the th percentile; RECAP rows are shaded.
Figure 6. Per-input generation-latency CDFs on MMSpec for paired RECAP and AR runs. A leftward curve indicates lower latency. Multi-turn inputs contribute summed generation time across turns.
Single-turn speedup. For each single-turn benchmark in Table 1, we summarize all 1,000 paired throughput ratios from Equation (9) without filtering (Figure 8; Table 18). These are empirical input-level percentiles, not uncertainty intervals. Median speedup exceeds one on every benchmark for both models, and 81.9–98.8% of Qwen inputs and 88.2–99.2% of LLaVA inputs have higher throughput than paired AR. The distributions nevertheless vary across tasks: Qwen reaches a median of 1.66× on MMT-Bench, with 98.8% of inputs benefiting, versus 1.15× and 81.9% on TextVQA. The latter also has a P10 of 0.59×, showing that positive average gains coexist with substantial slowdowns on some inputs.
Figure 8. Per-input throughput speedup on the seven single-turn benchmarks (1,000 inputs per model and benchmark). Points mark medians and segments span P10–P90 of the input-level ratios. Dashed lines mark parity with AR; the percentages report inputs with strictly higher throughput than paired AR.
| Dataset | Qwen | LLaVA | ||||||
| P10 | P50 | P90 | (%) | P10 | P50 | P90 | (%) | |
| MMT-Bench | 1.20 | 1.66 | 2.20 | 98.8 | 0.96 | 1.53 | 2.46 | 88.2 |
| SEED-Bench | 1.05 | 1.24 | 1.52 | 96.9 | 1.09 | 1.29 | 1.60 | 96.9 |
| ScienceQA | 1.15 | 1.66 | 2.50 | 98.5 | 0.99 | 1.49 | 2.36 | 89.4 |
| OCRBench | 1.00 | 1.37 | 2.97 | 90.0 | 0.99 | 1.66 | 2.86 | 88.9 |
| ChartQA | 0.89 | 1.49 | 3.19 | 88.4 | 1.23 | 1.56 | 1.97 | 99.2 |
| MathVista | 1.10 | 2.02 | 3.55 | 91.3 | 1.25 | 1.65 | 2.26 | 98.4 |
| TextVQA | 0.59 | 1.15 | 2.40 | 81.9 | 1.06 | 1.26 | 1.56 | 95.0 |
Table 18. Distributions of paired throughput ratios for RECAP on the seven single-turn benchmarks (1,000 inputs per dataset and model). P10/P50/P90 report speedup factors; the final column for each model gives the percentage of inputs with strictly higher throughput than paired AR.
References
- [1]Ali E. Abbas. 2009. A kullback-leibler view of linear and log-linear pools. Decision Analysis, 6(1):25–37.DOI
- [2]Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan-Kelley, and William Brandon. 2024. Hydra: Sequentially-dependent draft heads for medusa decoding. arXiv preprint arXiv:2402.05109.arxiv.org/abs/2402.05109
- [3]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. Qwen2.5-vl technical report. Preprint, arXiv:2502.13923.arxiv.org/abs/2502.13923
- [4]Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774.arxiv.org/abs/2401.10774
- [5]Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318.arxiv.org/abs/2302.01318
- [6]Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024. Break the sequential dependency of llm inference using lookahead decoding. arXiv preprint arXiv:2402.02057.arxiv.org/abs/2402.02057
- [7]Edouard Grave, Armand Joulin, and Nicolas Usunier. 2016. Improving neural language models with a continuous cache. arXiv preprint arXiv:1612.04426.arxiv.org/abs/1612.04426
- [8]Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. 2024. Rest: Retrieval-based speculative decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1582–1595.
- [9]Yunhai Hu, Tianhua Xia, Zining Liu, Rahul Raman, Xingyu Liu, Bo Bao, Eric Sather, Vithursan Thangarasa, and Sai Qian Zhang. 2026. Dream: Drafting with refined target features and entropy-adaptive cross-attention fusion for multimodal speculative decoding. Advances in Neural Information Processing Systems, 38:167592–167612.arxiv.org/abs/2505.19201
- [10]Yuxuan Hu, Ke Wang, Xiaokang Zhang, Fanjin Zhang, Cuiping Li, Hong Chen, and Jing Zhang. 2025. Sam decoding: Speculative decoding via suffix automaton. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12187–12204.arxiv.org/abs/2411.10666
- [11]Haiduo Huang, Fuwei Yang, Zhenhua Liu, Xuanwu Yin, Dong Li, Pengju Ren, and Emad Barsoum. 2025. Specvlm: Fast speculative decoding in vision-language models. arXiv preprint arXiv:2509.11815.arxiv.org/abs/2509.11815
- [12]Jialiang Kang, Han Shu, Wenshuo Li, Yingjie Zhai, and Xinghao Chen. 2026. Vispec: Accelerating vision-language models with vision-aware speculative decoding. Advances in Neural Information Processing Systems, 38:115511–115532.arxiv.org/abs/2509.15235
- [13]Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172.arxiv.org/abs/1911.00172
- [14]Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International conference on machine learning, pages 19274–19286. PMLR.arxiv.org/abs/2211.17192







