betaFeedback

WorksComputer Science

RECAP: Context-Aware Prediction Recycling with Request-Local and Persistent Memories for Multimodal Speculative Decoding

Abstract

Recycling a target model's own predictions offers a training-free source of proposals for speculative decoding. In vision-language models, however, existing recycling indexes predictions by the last token alone, conflating predictions computed under different images and prefixes; we find that such reuse fails most often at image-sensitive decoding states. Longer keys reduce this aliasing but match fewer states. We introduce \method, which stores target predictions under one-, two-, and three-token keys: contextual memories live only for the current request and are thus implicitly conditioned on its image, while token-level memory persists across requests to retain coverage. At each lookup, entropy-confidence weighting fuses the retrieved predictions according to their key specificity and concentration. Best-first search composes the fused transitions into a candidate tree, supplemented by suffix-matched continuations, and one target pass verifies the tree under greedy decoding. Without training or an auxiliary draft model, \method\ achieves mean speedups over autoregressive decoding of 1.66\speed and 1.56\speed across seven single-turn benchmarks on Qwen2.5-VL-7B and LLaVA-1.6-7B, exceeding the strongest training-free baseline, Token Recycling (1.51\speed and 1.37\speed). Our implementation is available in https://github.com/Senmo996/RECAP-official-code.

Introduction

Vision-language models (VLMs) generate text one token at a time, so every output token requires a sequential target-model pass. Speculative decoding reduces the number of sequential passes by proposing several tokens and verifying them in one parallel pass [14, 5]. Multimodal methods usually obtain proposals from

a separately trained drafter [9, 12], adding training and serving cost for every target model. Tree verification offers a cheaper source: each pass computes full next-token distributions at tens of candidate nodes yet commits only a few tokens, discarding the rest. Prediction recycling stores these target predictions and reuses them as later proposals (Luo et al., 2025), requiring neither a draft model nor training.

A recycled prediction is valid for the condition under which the target computed it: the image and the textual prefix. Existing recycling indexes predictions by the last token alone; Token Recycling, for example, keeps one Top-kk prediction row per vocabulary token in an adjacency matrix that every verification pass overwrites and that is retained across requests (Luo et al., 2025). Such a key merges all conditions that end in the same token, a mismatch we call condition aliasing. Along the prefix, her→head, cached after “a red hat covers her”, is reused after “blue shoes protect her”, where the target selects feet (Figure 1). Along the image, a prefix such as “the sign reads” must continue with whatever text the current image shows. Reuse also fails at image-sensitive states: in counterfactual diagnostics on 700 Qwen2.5-VL images, replacing the image changes the root prediction for 34.9% of fully rejected drafts but for only 12.4% of drafts that accept five or more tokens.

Figure 1. The role of context in prediction recycling. Top: a cached her→headher \to head transition fails when reused after a different prefix. Middle: shorter accepted paths accompany stronger root image sensitivity (left), and source preference varies with candidate visual alignment (right). Bottom: RECAP retrieves and fuses predictions at multiple contextual resolutions.

Longer keys reduce aliasing but recur less often: they recall the target token more reliably when they match, yet match far fewer decoding states. We call this tension the specificity–coverage trade-off. Suffix-matching methods such as Prompt Lookup Decoding and SAM Decoding condition on longer contexts ([10] (Somasundaram et al. 2025)), but each match copies a single observed continuation rather than combining target predictions obtained under several context lengths.

The two parts of the condition call for separate controls. Key length distinguishes prefixes. Longer keys separate prefixes that a token key aliases, while shorter keys remain necessary for coverage. Neither resolution dominates: whether contextual or token-level candidates yield longer accepted prefixes depends on which carries more visual support (Appendix § 11), so candidate selection should combine resolutions. Memory lifetime distinguishes images. A lexical key cannot encode the image, but the image is fixed within a request, so request-local memory conditions its predictions on that image implicitly. Token transitions, in contrast, recur across requests, so token-level memory remains useful when retained across a stream.

We propose RECAP (REcycling Context-Aware Predictions), a training-free framework built on these two controls (Figure 2). It stores target predictions under one-, two-, and three-token keys; the two contextual maps live for a single request, whereas the token-level map persists across requests. Because a specific row may be diffuse while a shorter row is sharply peaked, RECAP does not simply prefer the longest key: it weights each retrieved row by a specificity prior scaled by an entropy-based confidence and mixes the rows, reinforcing shared candidates while retaining confident alternatives. Best-first search composes the fused transitions into a candidate tree, and a suffix automaton adds short continuations from the prompt and committed history. One target pass verifies

Figure 2. The RECAP framework. (A) Three-, two-, and one-token keys retrieve Top-KK prediction rows from G\mathcal{G}, C\mathcal{C}, and U\mathcal{U}; only U\mathcal{U} survives request boundaries. (B) Entropy-confidence weighting fuses the retrieved rows under specificity priors and normalizes the retained candidates. (C) Best-first tree search repeats retrieval and fusion under each node’s path context. (D) Suffix retrieval inserts a contiguous continuation from the prompt and committed history. (E) One packed target pass commits the matching path plus one target-selected token. All evaluated nodes refresh prediction memory; only committed tokens extend sequence history. Tokens and scores are illustrative.

the tree, commits the longest matching path, and writes the predictions of every evaluated node back to memory.

Our contributions are as follows:

  • We identify condition aliasing in multimodal prediction recycling and show that token-indexed reuse fails most often at image-sensitive states.

  • We propose RECAP, a training-free framework that distinguishes prefixes through multi-length keys with entropy-confidence fusion and images through request-local and persistent memories.

  • RECAP achieves mean speedups of 1.66× and 1.56× on Qwen2.5-VL-7B and LLaVA-1.6-7B across seven single-turn benchmarks, and 1.89× and 1.59× on MMSpec, exceeding the strongest training-free baseline in mean speedup.

Related Work

Speculative decoding. Speculative decoding separates proposal generation from parallel target verification ([14, 5]). Learned proposers exploit target representations: Medusa adds future-token heads with tree verification ([4]), and Hydra models dependencies between head predictions ([2]). EAGLE drafts autoregressively in feature space ((Li et al., 2024b)), EAGLE-2 adapts draft trees to context-dependent confidence ((Li et al., 2024a)), and EAGLE-3 combines multi-layer features with direct token prediction ((Li et al., 2026)). RECAP keeps the propose–verify interface but drafts from memory of predictions the target has already evaluated.

Multimodal speculative decoding. Visual evidence adds a conditioning requirement to multimodal drafting: a drafter must propose tokens that are consistent with both the textual context and the visual input. Existing approaches address this challenge through different mechanisms. MSD separates text and visual token processing and trains a drafter with a text-to-multimodal curriculum ((Lin et al., 2025)); DREAM transfers refined target features through cross-attention ([9]); ViSpec adds a vision adaptor to drafting ([12]); SpecVLM combines EAGLE-style drafting, adaptive visual compression, and online-logit distillation ([11]). Together, these methods explore how visual information can be incorporated into a learned drafting process. MMSpec evaluates such approaches across multimodal task categories ((Shen et al., 2026)). RECAP instead obtains visual conditioning from cached target predictions and request-local memory, reusing information produced under the current multimodal context without training a multimodal proposer.

Training-free speculative decoding. Training-free methods derive proposals from existing sequences and target computations. Prompt Lookup Decoding reuses repeated prompt substrings ((Somosundaram et al., 2025)); REST retrieves suffix-matched continuations from an external datastore ([8]); SAM Decoding finds longest-suffix matches in static corpora and generation history ([10]); SuffixDecoding builds frequency-scored trees from current and previous outputs ((Oliaro et al., 2026)). Lookahead generates and verifies nn-grams through parallel iteration ([6]), and Token Recycling forms draft trees from token-indexed predictions ((Luo et al., 2025)). Related memory mechanisms include neural caches ([7]), kNN-LMs ([13]), and hierarchical language models that support sparse longer contexts with shorter-context distributions ((Teh, 2006)). RECAP combines multi-resolution lexical keys with request-local and persistent scopes, while the target retains control of commitment.

Context-Aware Prediction Recycling

Motivation

We examine how reuse depends on visual context through the 700-image counterfactual diagnostics in Figure 1 (Appendix § 11). With each text prefix fixed, target predictions are recomputed under mean-image and matched wrong-image controls, and results are averaged equally across benchmarks.

Evidence 1 relates draft acceptance to root sensitivity. With iterations grouped by accepted length, bars show root Top-1 changes under either control, and the curve gives the smaller of the two decreases in the original root token’s log probability. Rejected drafts score higher on both measures than 5+-token acceptances.

Evidence 2 compares sources at equal candidate budgets. With the prefix and candidate set fixed, visual support ESE_S is the smaller of the two reductions, under the image controls, in the log of the total target probability assigned to source ss’s candidates. Relative support EGC−EUE_{GC}-E_U between contextual (G/C\mathcal{G}/\mathcal{C}) and token-only (U\mathcal{U}) candidates defines within-benchmark quintiles from token-only-aligned Q1 to context-aligned Q5. Bars show softmax-normalized support shares, and the curve shows contextual minus token-only accepted-prefix length. Its sign reversal indicates that the better source changes with candidate alignment, motivating memory that retains multiple context resolutions and fuses their evidence.

Problem Formulation

Let II be the image, pp the prompt, and y<ty_{<t} the generated prefix, with textual history h=(p,y<t)h=(p,y_{<t}). Greedy decoding with the VLM PθP_{\theta} over vocabulary V\mathcal{V} selects

yt=arg⁡max⁡z∈VPθ(z∣I,h),(1)y_t=\arg\max_{z\in\mathcal{V}}P_{\theta}(z\mid I,h), \tag*{(1)}

Speculative decoding verifies a proposed sequence or tree in parallel, commits the longest prefix matching Equation (1), and emits one additional target-selected token ([14]), so accepting a draft tokens yields a a+1a+1 output tokens per pass. RECAP preserves this commitment rule while building proposals from recycled target predictions.

Context-Stratified Prediction Memory

At each evaluated position, the target assigns logit ℓz\ell_z to token zz. We retain the KK highest-scoring tokens as SKS_K. A stored prediction row consists of SKS_K and its normalized distribution

P^(z)=exp⁡(ℓz)∑v∈SKexp⁡(ℓv),z∈SK,(2)\widehat{P}(z)=\frac{\exp(\ell_z)}{\sum_{v\in S_K}\exp(\ell_v)},\quad z\in S_K, \tag*{(2)}

with zero mass on all other tokens. Each row thus inherits the conditioning of its source image and prefix.

Rows are addressed by the last one, two, or three tokens of hh. For suffix (u,v,x)(u,v,x), these keys retrieve U[x]\mathcal{U}[x], C[v,x]\mathcal{C}[v,x], and G[u,v,x]\mathcal{G}[u,v,x]. Thus, covers her, warms her, and protect her share the token key her but access distinct contextual rows (Figure 1), reducing condition aliasing.

Context refinement. Let HjH_j be the final jj history tokens, and let RK∗(Hj)R^*_K(H_j) denote the maximum expected recall of the target’s greedy token for selectors using only HjH_j and at most KK candidates. Since HjH_j is a deterministic function of Hj+1H_{j+1},

RK∗(Hj)≤RK∗(Hj+1),j=1,2.(3)R^*_K(H_j)\le R^*_K(H_{j+1}),\quad j=1,2. \tag*{(3)}

A finer-context selector can reproduce every coarse-context decision, but in a finite memory finer keys are observed less often; retaining shorter rows supplies coverage where longer keys are missing, as in hierarchical language models ((Teh, 2006)). Appendix F gives the proof.

Memory scope. Contextual maps C\mathcal{C} and G\mathcal{G} reset per request, making the current image and prompt implicit parts of the retrieval condition, whereas U\mathcal{U} persists within an inference stream and supplies earlier transitions when local memory is sparse. Prefill seeds U\mathcal{U}, tree verification refreshes all three maps, and each key retains its most recently assigned row.

Entropy-Confidence-Weighted Fusion

Let (M1,…,Mr)(M_1,\ldots,M_r) denote the rows available for history hh, ordered from most to least specific (G\mathcal{G}, C\mathcal{C}, U\mathcal{U}). RECAP fuses them by entropy-confidence weighting: each row’s weight is the product of a specificity prior, with α1≥α2≥α3>0\alpha_1\ge\alpha_2\ge\alpha_3>0, and an entropy-based confidence, defined as the row’s inverse effective support:

cj(h)=exp⁡[−H(P^Mj)],wj(h)=αjcj(h)∑i=1rαici(h).(4)\begin{aligned} c_j(h)&=\exp\left[-H\left(\widehat{P}_{M_j}\right)\right],\\ w_j(h)&=\frac{\alpha_jc_j(h)}{\sum_{i=1}^{r}\alpha_ic_i(h)}. \tag*{(4)} \end{aligned}

where H(P)=−∑zP(z)log⁡P(z)H(P)=-\sum_zP(z)\log P(z). The prior favors specific matches, and confidence favors concentrated predictions. Priors follow specificity rank among available rows: if only C\mathcal{C} and U\mathcal{U} match, they receive α1\alpha_1 and α2\alpha_2. Weights are recomputed at every tree node without additional target evaluations or learned parameters. Confidence measures concentration under the source condition, not direct alignment with the current image.

The normalized weights define the mixture

m(z∣h)=∑j=1rwj(h)P^Mj(z).(5)m(z\mid h)=\sum_{j=1}^{r}w_j(h)\widehat{P}_{M_j}(z). \tag*{(5)}

We bound branching by selecting the Top-KK tokens S(h)S(h), using the same KK as row storage, and renormalizing:

q(z∣h)=m(z∣h)∑v∈S(h)m(v∣h),z∈S(h),(6)q(z\mid h)=\frac{m(z\mid h)}{\sum_{v\in S(h)}m(v\mid h)},\quad z\in S(h), \tag*{(6)}

with all other probabilities set to zero. A single retrieved row recovers ordinary lookup, and no match terminates expansion. Setting cj(h)=1c_j(h)=1 for all available rows yields the fixed-prior special case, the w/o confidence configuration analyzed in Appendix I.

Weighted KL center. For fixed hh, the weights do not depend on the candidate distribution ρ\rho, so the mixture is the weighted Kullback–Leibler (KL) center ([1]):

m(⋅∣h)=arg⁡min⁡ρ∈Δ(V)∑j=1rwj(h)DKL(P^Mj∥ρ),(7)m(\cdot\mid h) = \underset{\rho\in\Delta(\mathcal{V})}{\arg\min} \sum_{j=1}^{r} w_j(h)D_{\mathrm{KL}}(\widehat{P}_{M_j} \mathbin{\|} \rho), \tag*{(7)}

where Δ(V)\Delta(\mathcal{V}) is the probability simplex. Top-KK retention maximizes the preserved mixture mass, and the normalized proposal minimizes DKL(q∥m)D_{\mathrm{KL}}(q \mathbin{\|} m) over distributions supported on at most KK tokens (Appendix § 12).

Tree Construction and Verification

At every node, RECAP retrieves and fuses rows using the established prefix together with proposed ancestors. Best-first construction orders paths π=(z1,…,zd)\pi= (z_1, \ldots, z_d) by the score

ϕ(π)=∏j=1dηjq(zj∣hj),(8)\phi(\pi) = \prod_{j=1}^{d} \eta_j q(z_j \mid h_j), \tag*{(8)}

where hj=(h,z<j)h_j = (h, z_{<j}) supplies the path context and ηj\eta_j controls depth-dependent expansion, until the node or depth budget is reached. Composing local predictions yields candidate sequences never stored as complete continuations.

A suffix automaton over the prompt and committed history retrieves the continuation following a match to the current suffix (Hu et al., [10]) and inserts it into the tree by sharing common prefixes, preserving observed sequence dependencies without modifying the fused transition scores.

The target verifies the joint tree in one packed pass, with ancestral attention confining each node to the established prefix and its own path. The longest path matching Equation (1) is committed together with one target-selected token; if no candidate agrees, the target directly supplies the next token.

Predictions from all evaluated nodes, including uncommitted branches, update UU, C\mathcal{C}, and G\mathcal{G} under their path contexts, whereas only committed tokens enter the suffix automaton. An uncommitted branch thus contributes a target prediction valid under its hypothetical prefix, which may become useful when a compatible context recurs. Appendix § 7 provides implementation settings and hyperparameters.

Experimental Evaluation

Setup

Models and workloads. Evaluation uses Qwen2.5-VL-7B-Instruct (Bai et al., [3]) and LLaVA-1.6 Vicuna-7B (Liu et al., 2024a) on an NVIDIA A100-SXM4-80GB GPU. Generation is greedy at batch size one, with at most 200 new tokens. We fix 1,000 inputs for each single-turn benchmark: MMT-Bench (Ying et al., 2024), SEED-Bench (Li et al., 2023), ScienceQA (Lu et al., 2022), OCRBench (Liu et al., 2024b), ChartQA (Masry et al., 2022), MathVista (Lu et al., 2024), and TextVQA (Singh et al., 2019). MMSpec adds 600 inputs from six categories, extending coverage to multi-turn conversations (Shen et al., 2026). For the main results, each benchmark, including MMSpec, forms one. inference stream: persistent U\mathcal{U} is cleared after warmup and before each benchmark, then carried across that benchmark’s inputs, whereas C\mathcal{C} and G\mathcal{G} reset per request.

Baselines. We compare six draft-model-free approaches: Prompt Lookup Decoding (PLD; Somasundaram et al., 2025), Lookahead Decoding (Fu et al., [6]), Token Recycling (Luo et al., 2025), Tree Recycling, SAM Decoding (Hu et al., [10]), and SAM+ViSkip (Shen et al., 2026). Token Recycling and Tree Recycling run token-indexed recycling (Luo et al., 2025) on the same prefill and verification path as RECAP. Token Recycling follows the published configuration: a Top-8 adjacency matrix retained across requests drafts a static 80-node tree, and the matrix is cleared after warmup and at each benchmark boundary, as is U\mathcal{U}. Tree Recycling is a lighter variant that expands a smaller static tree from a request-local Top-4 matrix. All methods use fixed input manifests and an autoregressive reference from the same backbone. Their settings are listed in Appendix § 7.

Metrics. We report speedup S=1N∑i=1NrimS = \frac{1}{N}\sum_{i=1}^{N}r_i^m as the mean of paired throughput ratios, defined by

rim=nim/TimniAR/TiAR.(9)r_i^m = \frac{n_i^m/T_i^m}{n_i^{\mathrm{AR}}/T_i^{\mathrm{AR}}}. \tag*{(9)}

For each generation unit, nimn_i^m and TimT_i^m are the output-token count and total generation time for method mm. The autoregressive reference is denoted AR, and NN is the number of generation units.

Our acceptance length τ\tau includes the additional target-selected token:

τ=1+1N∑i=1Naˉi,(10)\tau= 1 + \frac{1}{N}\sum_{i=1}^{N}\bar{a}_i, \tag*{(10)}

where aˉi\bar{a}_i averages draft tokens accepted per verification iteration for unit ii.

When target evaluations are exact, RECAP preserves greedy decoding without loss.

Main Results

RECAP leads every single-turn model–benchmark comparison in speedup and acceptance (Table 1). Qwen’s mean speedup across seven benchmarks is 1.66×, compared with 1.51× for Token Recycling, the strongest baseline by mean speedup. LLaVA’s corresponding values are 1.56× and 1.37×. The gains cover reasoning, chart understanding, and text-centric inputs: Qwen reaches 2.19× on MathVista and 1.75× on ChartQA, while LLaVA reaches 1.81× on OCRBench and 1.71× on MathVista. Alongside its consistent lead in SS, RECAP has the highest τ\tau throughout. Its mean acceptance lengths of 3.02 and 2.92 exceed Token Recycling’s 2.71 and 2.65, the strongest baseline acceptance averages.

ModelsMethodsMMTSEEDSciQAOCRChartQAMathVistaTextVQAAverage
SSτ\tauSSτ\tauSSτ\tauSSτ\tauSSτ\tauSSτ\tauSSτ\tauSSτ\tau
Qwen2.5-VL 7B-InstructPLD1.161.190.891.051.241.131.221.201.141.251.491.220.971.091.161.16
Lookahead1.151.600.931.351.442.141.111.621.101.621.201.600.941.261.121.60
Tree Recycling1.201.471.071.491.231.541.291.681.271.721.411.701.061.411.221.57
Token Recycling1.592.571.252.581.602.601.512.821.542.891.862.841.212.631.512.71
SAM1.111.241.061.341.151.361.201.331.231.571.211.381.051.261.141.35
SAM+ViSkip1.041.250.991.271.071.291.041.281.161.501.141.360.991.221.061.31
RECAP1.692.811.272.781.722.841.673.311.753.302.193.271.342.811.663.02
LLaVA-1.6 Vicuna-7BPLD1.011.340.911.141.091.231.281.531.021.341.191.400.891.111.061.30
Lookahead0.801.740.701.480.942.110.912.110.741.660.791.700.641.310.791.73
Tree Recycling1.231.481.151.491.221.431.401.821.281.601.371.691.131.341.251.55
Token Recycling1.402.481.242.491.362.471.492.691.422.661.492.651.232.411.372.55
SAM1.131.261.111.341.121.241.241.451.181.361.231.401.071.191.161.32
SAM+ViSkip1.101.261.071.341.091.241.201.451.151.361.201.401.041.191.121.32
RECAP1.632.771.312.821.562.701.813.401.583.061.713.111.292.591.562.92

Table 1. Speedup SS (×) and acceptance length τ\tau across seven single-turn benchmarks at temperature 0. SciQA denotes ScienceQA and OCR denotes OCRBench. SS averages paired generation-throughput ratios against autoregressive decoding; τ\tau averages accepted draft tokens plus one target-selected token. Higher is better for both metrics.

On MMSpec (Table 2), RECAP reaches overall speedups of 1.89× on Qwen and 1.59× on LLaVA, exceeding Token Recycling, the strongest baseline on both models, at 1.56× and 1.43×. RECAP leads in both speedup and acceptance length across catcategories except Qwen text VQA, where Token Recycling reaches 1.31× against 1.26×, while both methods round to τ=2.49\tau= 2.49. Overall acceptance lengths for RECAP are 2.77 and 2.85. Multi-turn conversation yields 2.09× on Qwen, second only to complex reasoning at 2.16×, and 1.73× on LLaVA, its largest category speedup.

ModelsMethodsGeneral VQA SSGeneral VQA τ\tauText VQA SSText VQA τ\tauCaptioning SSCaptioning τ\tauCharts SSCharts τ\tauReasoning SSReasoning τ\tauMulti-turn SSMulti-turn τ\tauOverall SSOverall τ\tau
Qwen2.5-VL 7B-InstructPLD1.061.051.211.131.141.151.201.311.281.341.341.271.231.22
Qwen2.5-VL 7B-InstructLookahead1.111.371.041.291.441.641.211.771.912.281.391.801.361.71
Qwen2.5-VL 7B-InstructTree Recycling1.121.401.101.321.221.371.211.641.381.661.251.561.221.51
Qwen2.5-VL 7B-InstructToken Recycling1.512.571.312.491.802.511.382.601.772.641.572.601.562.57
Qwen2.5-VL 7B-InstructSAM1.051.141.131.131.031.071.141.541.181.311.081.231.101.24
Qwen2.5-VL 7B-InstructSAM+ViSkip0.971.161.001.120.951.061.061.551.071.290.991.241.001.24
Qwen2.5-VL 7B-InstructRECAP1.772.691.262.492.002.601.863.022.162.972.092.801.892.77
LLaVA-1.6 Vicuna-7BPLD1.031.101.061.081.091.171.211.541.371.791.331.821.211.48
LLaVA-1.6 Vicuna-7BLookahead0.931.410.851.250.871.341.021.661.031.720.901.430.921.46
LLaVA-1.6 Vicuna-7BTree Recycling1.121.431.171.401.191.341.341.701.371.771.351.641.271.56
LLaVA-1.6 Vicuna-7BToken Recycling1.212.421.352.481.502.501.432.581.462.571.532.651.432.55
LLaVA-1.6 Vicuna-7BSAM1.071.231.061.161.041.091.241.461.221.491.161.281.141.29
LLaVA-1.6 Vicuna-7BSAM+ViSkip1.041.231.041.161.021.091.211.461.201.491.141.291.121.29
LLaVA-1.6 Vicuna-7BRECAP1.262.601.442.641.652.631.653.101.673.091.732.931.592.85

Table 2. MMSpec speedup SS (×) and acceptance length τ\tau at temperature 0, with the metric definitions of Table 1. Each category contains 100 inputs; multi-turn conversations contribute 223 generation turns, giving 723 total turns. Overall values are turn-weighted averages of both metrics, rather than unweighted category means.

Comparison with trained drafters. Table 3 contrasts RECAP with released DREAM Hu et al., [9] and ViSpec Kang et al., [12] drafters on the same single-turn inputs. Trained drafters accept longer continuations, with mean τ\tau of 3.68–3.76 against 2.92 and 3.02 for RECAP, which reaches 78–80% of their acceptance length. Their speedups of 1.05–1.26× fall below those of RECAP, but they were measured against autoregressive baselines of 41.9–61.2 tokens/s rather than 29.9–39.4 tokens/s, so this gap does not establish that RECAP decodes faster. Their acceptance advantage requires a training stage for each target model and adds 451M–890M parameters, or 0.84–1.66 GiB of 16-bit weights, to the serving GPU. RECAP adds no parameters or GPU-resident state; at the default budget, its proposal state occupies at most 8.70 MiB of host memory (Table 10). It therefore transfers to a new VLM without data collection, training, or a drafter checkpoint.

ModelMethodParamsGiBτ\tauSSAR
LLaVADREAM607M1.133.681.2641.9
ViSpec451M0.843.741.0561.2
RECAP002.921.5629.9
QwenViSpec890M1.663.761.1857.0
RECAP003.021.6639.4

Table 3. Deployment cost and decoding performance on the seven single-turn benchmarks in Table 1. Params and GiB denote drafter parameter counts and 16-bit GPU weight storage; AR is mean autoregressive throughput (tokens/s). DREAM and ViSpec use released checkpoints with top-4, depth-6, 32-node trees and a faster AR harness, so their SS is not directly comparable with that of RECAP.

ModelCapMean nodesPrep. msp95p_{95} msHost MiBGPU GiB
Qwen3126.50.5910.7225.7919.32
Qwen6355.20.9071.2277.8819.32
Qwen127113.42.2022.6178.7419.32
LLaVA3129.10.7210.8827.4619.98
LLaVA6359.21.1211.3858.7020.01
LLaVA127121.02.4852.72912.6820.16

Table 10. Budget sensitivity with shared inputs and orders. Candidate counts include suffixes and exclude the root. Preparation covers lookup through packing; p95p_{95} is the request-average 95th percentile. Host memory is the largest proposal-state snapshot. GPU memory is peak model-process allocation, including weights and generation workspaces.

Analysis of Prediction Reuse

Component Contributions

Matched-budget ablations test which parts of the design contribute to throughput (Table 4). The MMSpec subset comprises 120 inputs, 20 per category, expanding to 160 generation turns, and is evaluated with Qwen2.5-VL-7B. Policies share a tree builder and a 63-node transition budget. Reported gains average input-paired throughput changes after pooling each input’s turn-level token counts and times. Except for the confidence comparison, these controls use the fixed-prior special case (cj=1c_j = 1), so each comparison isolates one component.

ComparisonGain (%, ↑)
Candidate search
Best-first vs. fixed expansion+4.90
Memory design
Context backoff vs. unigram+3.03
Uniform fusion vs. backoff+0.75
Specificity priors vs. uniform fusion+2.80
Entropy confidence vs. fixed priors†+0.73/+1.09
Persistent vs. reset UU+5.42
Sequence reuse
Suffix insertion vs. none†+0.61

Table 4. Paired throughput effects of individual components. Unmarked rows use the 120-input MMSpec subset with Qwen2.5-VL-7B; the persistence row averages five runs. † marks four-dataset controls (200 inputs per model for confidence, reported as Qwen/LLaVA; 400 inputs for suffix insertion). Backoff selects the most specific row available at a query.

Contextual backoff improves throughput over unigram recycling by 3.03%, and an independent repeat of the same 120-input comparison yields 2.42%. Uniform fusion improves backoff by 0.75%, and specificity priors improve uniform fusion by 2.80%. Across five runs, retaining token memory across requests improves throughput over per-request reset by 5.42% on average, ranging from 4.36% to 7.18%. Request-stream controls are detailed in Appendix § 10.

In four-dataset controls, entropy-confidence weighting adds a smaller gain over fixed priors: 0.73% mean throughput on Qwen and 1.09% on LLaVA. Confidence-based single-source gating decreases throughput by 0.66% and 1.90%, respectively. The contrast supports combining source evidence in these settings while qualifying the size of the confidence benefit. Appendix § 15 details the adaptive rules and fixed-prior sensitivity.

Suffix complementarity is tested separately on 100 inputs each from ChartQA, TextVQA, OCR-Bench, and MMT-Bench. Adding suffix insertion to fusion improves mean paired throughput by 0.61%, with dataset-specific changes from −0.67% to +2.23% (Table 7).

DatasetFusion+ SuffixDifferenceGain (%, ↑)τ\tau: before → after
ChartQA1.1231.3130.190+0.832.80 → 2.88
TextVQA0.3550.4130.059+0.072.18 → 2.18
OCRBench1.4171.5380.120−0.672.66 → 2.62
MMT-Bench0.7500.8660.116+2.232.40 → 2.44

Table 7. Suffix-retrieval controls. Offline matching omits root and correction tokens; online gains and acceptance length τ\tau use Equations (9)–(10).

Specificity Improves Matching; Backoff Retains Coverage

Replay comprises 23,811 states from 400 inputs, with 100 each from ChartQA, TextVQA, OCRBench, and MMT-Bench. Retrieval begins with empty memories and accesses only rows from earlier states. Query coverage is the fraction of states with a matching row. Conditional Recall@8 is recall of the reference next token among those covered states.

Stored rows preserve useful source-image information. To test this, we fix the query image, question, and prefix and vary the image used to compute earlier predictions. On jointly covered queries with eight candidates per source, matching-image rows outperform swapped-image rows in fused Recall@8 by 0.89–6.35 percentage points across datasets (Appendix § 9).

All four datasets show higher conditional recall and lower coverage with finer keys (Figure 3(a)). On OCRBench, U\mathcal{U} coverage is 54.9% and G\mathcal{G} coverage is 18.6%, while recall rises from 63.6% to 90.6%. In a prompt-seeded variant of this replay, restricting evaluation to states where both G\mathcal{G} and U\mathcal{U} match, G\mathcal{G} improves Recall@8 over U\mathcal{U} by 5.00–29.54 percentage points across datasets.

Figure 3. The specificity–coverage trade-off and contextual backoff. (a) Query coverage and conditional Recall@8 across 23,811 states from 400 inputs. Each benchmark connects U\mathcal{U} (circle), C\mathcal{C} (square), and G\mathcal{G} (diamond). Finer keys raise conditional recall while reducing coverage. (b) Paired online throughput gains of U+C\mathcal{U} + \mathcal{C} backoff against U\mathcal{U}.

At equal actual candidate counts, U+C\mathcal{U} + \mathcal{C} backoff also improves matched continuation length on every dataset, from 0.516 to 0.709 tokens on ChartQA and 0.632 to 1.135 on OCRBench (Appendix § 9). This offline metric measures the longest tree prefix agreeing with the saved target continuation, excluding the root and target correction token.

Online controls yield 1.11–3.64% paired throughput gains from U+C\mathcal{U} + \mathcal{C} backoff (Figure 3(b)). Adding G\mathcal{G} improves offline matching further on ChartQA and MMT-Bench, with only small throughput changes online. Context refinement therefore supplies more precise candidates, and backoff preserves the coverage needed for their practical reuse.

Complementary Sequence Reuse

Suffix retrieval contributes sequence continuations beyond those formed by local transition composition. Starting with prompt-seeded U\mathcal{U}, we compare fusion alone with suffix-augmented fusion at equal actual candidate counts. The added suffix increases mean matched length by 0.059–0.190 tokens across four replay datasets (Appendix § 9). The matched-budget result attributes the improvement to complementary sequence evidence.

Online throughput changes from suffix insertion range between −0.67% and +2.23% across the four datasets. Timing includes retrieval, insertion, and overlap handling. At the default budget, retrieval, fusion, tree construction, and packing together require 0.907 ms per Qwen iteration and 1.121 ms per LLaVA iteration. Including memory refresh, suffix-index updates, token commitment, and other host-side decoding work, all stages outside target verification take 1.778 and 1.974 ms, or 8.20% and 5.61% of decoding wall time (Appendix § 13).

Conclusion

RECAP addresses contextual specificity and reuse scope jointly in multimodal prediction recycling, fusing request-local contextual memories with persistent token-level memory through entropy-confidence weighting without training or an auxiliary draft model. It reaches mean generation-throughput speedups of 1.66×1.66 \times and 1.56×1.56 \times across seven single-turn benchmarks on Qwen2.5-VL-7B and LLaVA-1.6-7B, and 1.89×1.89 \times and 1.59×1.59 \times on MMSpec. Controlled analyses connect these gains to more precise contextual matching and reuse across request boundaries.

Limitations

Our evidence covers greedy, batch-one generation with image-based inputs on two 7B VLMs. Larger models, concurrent serving, and video remain untested operating regimes. Default specificity priors, confidence weighting, and tree settings may yield different latency–memory trade-offs across hardware and request lengths. The reported efficiency concerns generation throughput and does not establish downstream task accuracy or application-level latency distributions.

Ethical considerations

This work studies inference efficiency using existing models and public evaluation data and introduces no new human-subject data collection. Deployments should follow the applicable model and dataset licenses and isolate online prediction memories at appropriate user or service boundaries. Faster inference does not remove the need to assess outputs for the intended application.

Implementation and Baseline Settings

Model execution. Qwen and LLaVA run in bfloat16 and float16, respectively, on the workloads of § 4.1. Timing covers prefill and decoding, excluding data loading and report aggregation.

Recap defaults. We use Top-8 rows, width 8, depth 10, and 63 transition candidates excluding the root. Available rows receive priors (0.70, 0.20, 0.10) by decreasing specificity; entropy-confidence weighting multiplies each prior by the row’s confidence and normalizes the products (Equation (4)). Depth priors are (0.68, 0.63, 0.50, 0.43, 0.40, 0.38), then 0.38. Prefix-sharing suffix insertion adds at most four tokens beyond the transition cap, giving at most 68 packed nodes including the root; controls described as suffix-inclusive instead count suffix nodes within the 63-node cap. Special image tokens and duplicate children are filtered.

Memory updates and packed verification. Prefill seeds UU from each eligible prompt token’s first occurrence; C/GC/G start empty. All evaluated rows refresh memory, retaining each key’s first batch occurrence and replacing older entries. Packed positions follow path depth; accepted-path key/value entries continue decoding. Stream and policy states are isolated and reset after warmup; in the main results, UU is also cleared at each benchmark boundary, so no predictions carry over between benchmarks.

Baselines. PLD uses 4-token keys and at most 10 draft tokens; Lookahead uses decoding/branch lengths 64/12. Token Recycling follows the official implementation (Luo et al., 2025): a zero-initialized Top-8 adjacency matrix fills the static imbalanced “2.2.2” tree (80 candidates, depth 6) layer by layer, and every verified node writes its Top-8 target predictions back to the matrix. Prefill logits are not written; the matrix persists across requests and is cleared only after warmup and at benchmark boundaries. Tree Recycling expands a static tree of width 2 and depth 4, giving 30 candidates under a 31-node cap, from a request-local Top-4 matrix. SAM uses total-token setting 64 and draft lengths 2–40. SAM+ViSkip caps drafts at 40 tokens and gates at 0.35 on head-averaged image attention at final-layer query zero.

Timing and Analysis Protocols

Fusion rule. The main results (Table 1 and Table 2) and the MMSpec latency analysis (Appendix § 16) use entropy-confidence weighting. Component analyses in § 5 and Appendices § 9, § 10, § 13, and § 14 use its fixed-prior special case (cj=1c_j = 1), so that each control varies one component; Appendix § 15 compares the two rules directly.

Pairing and timed boundaries. For the recycling methods (RECAP, Tree Recycling, and Token Recycling), SAM, and SAM+ViSkip, autoregressive references are interleaved with method runs; PLD and Lookahead pair generation units. Main methods each have a separate run and reference, while component controls share a run. Main recycling and MMSpec ablations synchronize at target forwards. SAM+ViSkip and four-dataset controls synchronize at generation boundaries; the four-dataset controls also share timing paths that include CPU candidate counters.

MMSpec controls. In MMSpec ablations, the ratio for input jj pools its multi-turn records:

Rja/b=∑i∈jnia∑i∈jTia/∑i∈jnib∑i∈jTib(11)R_j^{a/b} = \frac{\sum_{i \in j} n_i^a}{\sum_{i \in j} T_i^a} \bigg/ \frac{\sum_{i \in j} n_i^b}{\sum_{i \in j} T_i^b} \tag*{(11)}

Gains are reported as 100(mean⁡jRja/b−1)100(\operatorname{mean}_j R_j^{a/b} - 1). Resampling and repeated request orders reuse the same inputs. Their variation therefore reflects this fixed evaluation set, not independent-dataset uncertainty.

Offline replay. Offline replay queries before updating, includes unmatched empty-memory states, and compares proposals with saved native target continuations. Trees respect a 63-node cap and are rebuilt to the smallest actual count among policies. Matched length excludes the root and target correction token. Suffix additions are included in the cap for controls; the default configuration allows four extra nodes.

Offline means weight replay states within each dataset and use 2,000 paired image-cluster resamples: 97 ChartQA clusters and 100 clusters elsewhere. Online resampling pairs inputs, using 100 per dataset or the first 50 for scope controls.

Candidate-Level Results

Table 5 gives the coverage and recall values in Figure 3; the corresponding online gains are reported below.

DatasetCoverage (%, ↑)Recall@8 (%, ↑)
ChartQA44.523.214.864.581.786.6
TextVQA26.37.43.452.573.578.3
OCRBench54.930.418.663.684.290.6
MMT-Bench45.619.711.256.977.984.4

Table 5. Coverage and conditional Recall@8 during empty-initialization replay. Queries are evaluated before their corresponding memory updates.

With 100 paired inputs per dataset, shared initialization, and a 63-node cap, U+CU + C backoff improves throughput over UU by 2.29%, 1.11%, 3.64%, and 1.65% on ChartQA, TextVQA, OCRBench, and MMT-Bench, respectively.

Backoff versus fusion. Matched-node comparisons favor backoff over fusion on ChartQA and OCRBench, with similar continuation lengths elsewhere (Table 6).

PolicyMatched continuation length (↑\uparrow)
ChartQATextVQAOCRMMT
U\mathcal{U}0.5160.2240.6320.444
U+C\mathcal{U} + \mathcal{C} backoff0.7090.2531.1350.600
U+C+G\mathcal{U} + \mathcal{C} + \mathcal{G} backoff0.7780.2521.1280.638
Uniform fusion0.7370.2531.0410.629
Prior-weighted fusion0.7510.2531.0660.639

Table 6. Matched continuation length in empty-initialization replay. Each state uses equal actual node counts across the compared policies. Prior-weighted fusion uses fixed priors (cj=1c_j = 1).

Fusion in online controls. With empty initialization and fixed priors, prior-weighted/uniform throughput ratios are 1.0043, 1.0015, 1.0006, and 1.0037 on ChartQA, TextVQA, OCRBench, and MMT-Bench. These near-neutral ratios qualify the gain observed in the prompt-seeded MMSpec control (Table 4) and show that specificity priors have setting-dependent benefits.

Suffix complementarity. At equal actual candidate counts under a 63-node cap, prompt-seeded suffix insertion adds first-token hits in 3.33%, 1.46%, 1.95%, and 2.31% of states (Table 7). Online policies are interleaved, bridge extension is disabled, and all suffix-related operations are timed.

Image-conditioned source predictions. Holding queries fixed and using eight candidates per source, jointly available matching-image rows increase fused Recall@8 over swapped-image rows by 5.55, 6.35, 1.91, and 0.89 points on ChartQA, TextVQA, OCRBench, and MMT-Bench. Donor images are category-matched on ChartQA/OCRBench and pooled on the other datasets. The context-only/trigram-only changes of −2.96-2.96/−2.50-2.50 points on OCRBench show that longer-key sources do not benefit uniformly. No image-similarity gate is used.

Prediction Reuse across Request Streams

We test cross-task reuse by changing token-memory lifetime while holding contextual memory request-local. Persistent U\mathcal{U} is compared with per-request reset, and C\mathcal{C} and G\mathcal{G} reset in both cases. The same 200 inputs are scheduled either in benchmark blocks or by alternating benchmarks.

Persistent memory improves throughput on ChartQA, OCRBench, and MMT-Bench under both schedules, with near-neutral effects on TextVQA (Figure 4). MMT-Bench gains are 7.98% for blocked streams and 7.49% for alternating streams. A task-boundary reset retains reuse within benchmarks but clears U\mathcal{U} on each benchmark switch. It becomes equivalent to per-request reset under alternation, while persistent U\mathcal{U} retains predictions through intervening tasks.

Figure 4. Token-level reuse across task schedules. Persistent U\mathcal{U} is paired with per-request reset on identical inputs to measure throughput under blocked and alternating benchmark streams.

Within-dataset orders use seeds 17, 29, 43. The three memory policies maintain isolated states, reset C/G\mathcal{C}/\mathcal{G} per request, and omit suffix insertion.

Table 8 averages paired input ratios separately for each seed. TextVQA’s pooled ratios are 0.987 under blocked scheduling and 0.990 under alternation, indicating near-neutral effects sensitive to aggregation. Alternating task-reset and per-request-reset acceptance is identical across 600 pairs. Memory snapshots include final prediction objects and exclude peak process RAM.

DatasetBlockedAlternating
ChartQA5.69 ± 0.593.91 ± 0.77
TextVQA0.92 ± 1.100.68 ± 1.13
OCRBench4.77 ± 1.044.32 ± 0.13
MMT-Bench7.98 ± 1.127.49 ± 1.77

Table 8. Mean throughput gain ± sample SD for persistent UU versus reset, measured across three orders of the same 200 inputs.

The persistent map keeps only the latest Top-KK row for each token, bounding U\mathcal{U} storage by O(∣V∣K)O(|\mathcal{V}|K) per stream. On cyclic replay of a fixed Qwen trajectory pool, its host footprint remains near 2.67 MiB through 10,000 request updates.

Motivation Figure Diagnostics

Figure 1 uses Qwen2.5-VL-7B traces from 100 images per single-turn benchmark, totaling 700. We compute equal-benchmark averages and 10,000 within-benchmark image-cluster bootstrap resamples. The counterfactuals hold text prefixes fixed while recomputing representations with mean-image and matched wrong-image controls.

Accepted length and root sensitivity (left). Visual dependence is the smaller counterfactual decrease in the original root token’s log probability, and change rate records Top-1 changes under either control. Valid full-image anchors yield 15,677 drafts across 695 image clusters, grouped by accepted length into 0–4 and 5+. From rejection to 5+ acceptance, dependence declines 0.620 → 0.198 and root-change rate declines 34.9% → 12.4%.

Candidate-source alignment (right). For token-only candidates CUC_U and contextual candidates CGCC_{GC}, define the log candidate mass and visual support as

Ls(J)=log⁡∑z∈CsPθ(z∣J,p,y<t),(12)L_s(J) = \log\sum_{z \in C_s} P_\theta(z \mid J, p, y_{<t}), \tag*{(12)}
Es=min⁡J∈{Imean,Iwrong}[Ls(I)−Ls(J)].(13)E_s = \min_{J \in\{I_{\mathrm{mean}}, I_{\mathrm{wrong}}\}} \left[L_s(I) - L_s(J)\right]. \tag*{(13)}

We retain 1,125 eligible states with valid full-image anchors, both sources, row overlap of at most 25%, and matched budgets of 2–8 candidates. Within-benchmark quintiles of EGC−EUE_{GC} - E_U run from UU-aligned Q1 to context-aligned Q5. Bars show exp⁡(Es)/(exp⁡(EU)+exp⁡(EGC))\exp(E_s)/(\exp(E_U) + \exp(E_{GC})). The corresponding contextual acceptance gains are −1.292, −0.375, 0.709, 1.239, and 2.084 tokens. Acceptance is not used to define the bins.

Properties of Prediction Reuse

Context refinement. Let ZZ be the greedy target token under fixed tie-breaking and HjH_j the last jj history tokens, start-padded as needed. Optimal recall is

RK∗(Hj)=E[max⁡S⊆V∣S∣≤KPr⁡(Z∈S∣Hj)].(14)R_K^*(H_j) = \mathbb{E}\left[\max_{\substack{S \subseteq\mathcal{V}\\ |S| \le K}} \Pr(Z \in S \mid H_j)\right]. \tag*{(14)}

For Ha=f(Hb)H_a = f(H_b), any coarse selector can be reproduced as Sb(Hb)=Sa(f(Hb))S_b(H_b) = S_a(f(H_b)), proving RK∗(Ha)≤RK∗(Hb)R_K^*(H_a) \le R_K^*(H_b). Dropping the earliest token gives Equation (3). Finite memories still require shorter-key coverage.

Fusion and sparse selection. At a fixed query, let m=∑jwjPjm = \sum_j w_j P_j combine normalized rows Pj=P^MjP_j = \hat{P}_{M_j} with fixed positive weights summing to one. The KL decomposition

∑jwjDKL(Pj∥ρ)=∑jwjDKL(Pj∥m)+DKL(m∥ρ).(15)\sum_j w_j D_{\mathrm{KL}}(P_j \mathbin{\|} \rho) = \sum_j w_j D_{\mathrm{KL}}(P_j \mathbin{\|} m) + D_{\mathrm{KL}}(m \mathbin{\|} \rho). \tag*{(15)}

makes mm the unique minimizer in Equation (7), without row independence (Abbas, [1]).

Let SKS_K retain the largest KK positive masses of mm (all if fewer), with deterministic ties, and q=m(⋅∣SK)q = m(\cdot\mid S_K). Then

q∈arg⁡min⁡ρ∈Δ(V)∣supp⁡(ρ)∣≤KDKL(ρ∥m).(16)q \in\underset{\substack{\rho\in\Delta(V)\\ |\operatorname{supp}(\rho)| \le K}}{\arg\min} D_{\mathrm{KL}}(\rho\Vert m). \tag*{(16)}

For nonempty S⊆supp⁡(m)S \subseteq\operatorname{supp}(m) and ρ\rho supported on SS,

DKL(ρ∥m)=DKL(ρ∥m(⋅∣S))−log⁡m(S).(17)D_{\mathrm{KL}}(\rho\Vert m) = D_{\mathrm{KL}}(\rho\Vert m(\cdot\mid S)) - \log m(S). \tag*{(17)}

Thus ρ=m(⋅∣S)\rho= m(\cdot\mid S) attains −log⁡m(S)-\log m(S), and Top-KK maximizes retained mass m(S)m(S), allowing tied optimal supports. These properties characterize proposals; target verification determines commitment.

Overhead and Memory Scaling

Latency and Workspace

Profiling uses the same A100-SXM4-80GB GPU and AMD EPYC 9654 CPU for both models, with PyTorch 2.7.1, CUDA 12.6, and four CPU threads. Ten inputs per diagnostic dataset follow Appendix A. Orders 17/29 interleave caps 31/63/127, resetting isolated states after warmup

and between repeats. Profiling uses the fixed-prior special case (cj=1c_j = 1); entropy-confidence weighting additionally evaluates one row entropy per retrieved row at multi-source queries. In an isolated host microbenchmark of the reference Python fusion over 512 multi-source queries, this raises fusion time from 7.3 to 14.5 μ\mus per query; the C++ builder used for the main results was not profiled separately.

We pool exclusive CUDA-synchronized phases over decoding iterations, excluding prefill and image encoding (Table 9). Packing includes topology, masks, positions, and verification inputs. Memory traversal and serialization are untimed. An uninstrumented 63-node stream measures instrumentation overhead, and four 64-token traces per model separately attribute CUDA kernel and memory-operation time.

StageQwen WallQwen CPULLaVA WallLLaVA CPU
Suffix lookup0.0100.0160.0100.016
Lookup, fusion, search0.5060.5100.5920.598
Packing0.3910.4090.5190.538
Memory refresh0.4610.4680.4140.421
Suffix index update0.0270.0340.0280.034
Token commitment0.2560.2730.2860.304
Other decoding0.1270.1480.1250.145
Target verification19.91219.92833.24133.265

Table 9. Decoding-phase times in milliseconds, pooled over iterations at a 63-node cap with 40 inputs per model and two orders. Wall times are exclusive; CPU time includes synchronization polling.

Stage costs. Preparation takes 0.907/1.121 ms on Qwen/LLaVA, or 4.18%/3.18% of decoding wall time. Adding memory refresh and suffix-index updates gives 1.395/1.563 ms (6.43%/4.44%), and all stages outside target verification total 1.778/1.974 ms (8.20%/5.61%). Instrumentation adds 1.01%/1.36% median latency. Retrieval, fusion, and search launch no CUDA work. Packing consumes 4.46/3.29 μ\mus of device time per iteration, while verification consumes 16.60/28.04 ms, representing 98.88%/99.61% of decoding-device time.

Budget sensitivity and workspace. For BB candidates, the ancestor submatrix contains (B+1)2(B+1)^2 entries. Table 10 deduplicates host objects shared across U/C/G\mathcal{U}/\mathcal{C}/\mathcal{G}, the suffix index, and topology state.

Persistent-Memory Growth

One latest Top-KK row per token bounds persistent memory by ∣U∣≤∣V∣|\mathcal{U}| \leq|\mathcal{V}|. The resulting space cost is O(∣V∣K)O(|\mathcal{V}|K) per stream and O(ns∣V∣K)O(n_{\mathrm{s}}|\mathcal{V}|K) for nsn_{\mathrm{s}} streams, independent of request count. Request-local C/G\mathcal{C}/\mathcal{G} scale with the contexts evaluated within each request.

Replay of observed prediction rows. Cyclic replay reshuffles 400 native Qwen trajectories, 100 per diagnostic dataset, under seeds 17/29/43 at each pass. Top-8 memory reaches 3,821 keys and 2.67 MiB after 400 requests and stays stable through 10,000 updates (Figure 5). This result characterizes replacement on a fixed pool, not saturation under unseen inputs.

Figure 5. Persistent U\mathcal{U} over cyclic replay of the same 400 trajectories. Curves report the mean and min–max range across three replay orders.

Vocabulary-capacity stress test. The capacity stress test fills synthetic Top-8 rows at three occupancies in fresh processes (Table 11). At full vocabulary occupancy, Qwen/LLaVA host storage is 109.14/23.36 MiB and RSS increases by 126.48/27.55 MiB. These capacity figures do not represent natural-stream footprints. Persistent rows contain no CUDA tensors.

Model∣U∣|U|Host MiBLookup ns
Qwen1,0000.67109
10,0007.03141
152,064109.14386
LLaVA1,0000.67109
10,0007.03140
32,06423.36183

Table 11. Synthetic occupancy stress test: host-object storage and median lookup latency, measured over seven repeats of 100,000 seeded ID/score hits with a checksum.

Across all six occupancies, 63-node tree construction takes 0.242–0.249 ms, measured as the median of seven repeats of 128 queries with packing excluded.

Alternative Root-Level Retrieval

Qwen fallbacks use a fixed six-category MMSpec subset and activate only when all root lexical lookups fail; descendants retain lexical retrieval. Semantic fallback selects up to three rows by cosine similarity of verified and predecessor states, with no extra target forwards. A request-local FIFO index holds at most 2,048 surviving U/C/GU/C/G rows; recency fallback selects the latest. Both use fixed priors (0.70, 0.20, 0.10) without confidence (cj=1c_j = 1) and Top-8 candidates.

Interleaved policies share target-generated histories, depth 10, a 63-candidate cap including suffixes but excluding the root, and isolated persistent UU and request-local C/GC/G. Timing includes image encoding, prefill, index maintenance, and decoding; metrics average turns.

Both increase τ\tau but reduce speedup by 1.46% (semantic) and 1.34% (recency; Table 12). Semantic lookup activates in 4.87% of iterations despite continuous index maintenance. Neither fallback is enabled by default.

VariantSS (×\times, ↑\uparrow)τ\tau (↑\uparrow)
RECAP (lexical)1.632.67
RECAP + semantic fallback1.612.78
RECAP + recency fallback1.612.75

Table 12. Root fallback on MMSpec with Qwen2.5-VL-7B. All variants use fixed-prior fusion (cj=1c_j = 1). Longer accepted continuations accompany lower speedup. Best column values are bold.

Fusion Ablations

In Equation (4), w/o confidence sets cj(h)=1c_j(h) = 1 (reference), w/o prior sets αj=1\alpha_j = 1, and uniform fusion removes both factors. Hard gating selects the most confident row, breaking ties by specificity. All rules share retrieved rows, Top-8 truncation, and renormalization.

Offline matching. Replay uses earlier rows, prompt-seeded UU, and empty C/GC/G. RECAP changes all-query Recall@8 by less than 0.02 points; gating loses 0.14–1.99 points (Table 14). Without confidence, soft (0.55, 0.30, 0.15) and sharp (0.85, 0.10, 0.05) priors change recall by at most 0.17 points.

VariantChartQATextVQAOCRMMT
RECAP+0.00+0.00+0.01−0.01
w/o confidence+0.00+0.00+0.00+0.00
w/o prior−0.08+0.00−0.31+0.00
Uniform−0.08+0.00−0.30−0.02
Confidence gate−1.49−0.14−1.99−0.95
Specificity backoff−0.11+0.00+1.29−0.24
w/o conf., soft prior−0.04+0.00−0.16+0.00
w/o conf., sharp prior−0.02+0.03+0.15−0.05

Table 14. Changes in all-query Recall@8 relative to w/o confidence (percentage points), evaluated on 23,811 Qwen states from 400 inputs. Uncovered states count as misses. Shading identifies RECAP.

Online comparison. The 4,000 generations in Table 13 compare five rules with isolated memories, suffix insertion, and Appendix § 7’s settings. Policies rotate per request; timing includes image encoding, prefill, generation, and adaptation. Repeats reuse inputs to assess order variation. RECAP gains 0.73%/1.09% mean throughput on Qwen/LLaVA, with median paired changes of −0.12%/−0.07%. W/o prior (confidence only) and uniform fusion have mixed effects; gating lowers throughput on both models. Gains are modest and task-dependent. Confidence measures concentration; learned gates and visual-dependence signals remain untested.

VariantQwen ChartQAQwen TextVQAQwen OCRQwen MMTQwen MeanQwen τ\tauLLaVA ChartQALLaVA TextVQALLaVA OCRLLaVA MMTLLaVA MeanLLaVA τ\tau
RECAP+0.65+1.24+1.43−0.41+0.732.93+2.30+0.34+1.71+0.01+1.092.90
w/o confidence+0.00+0.00+0.00+0.00+0.002.91+0.00+0.00+0.00+0.00+0.002.88
w/o prior+0.37+0.56+1.38−0.93+0.352.91+0.26−0.67+0.31−0.92−0.252.85
Uniform+0.01+0.37+2.34+0.22+0.732.90−0.47+0.18+2.29−0.06+0.482.86
Confidence gate−1.10+1.52−1.33−1.72−0.662.83−2.27−1.56−1.25−2.53−1.902.76

Table 13. Online fusion ablations, reporting mean input-paired throughput changes (%) against w/o confidence and acceptance length τ\tau. Each model processes 200 inputs, 50 per dataset, under two orders. Policies share memory scope, candidate limits, and suffix insertion; timing includes weighting. Full RECAP is shaded.

Disagreement among sources. We further examine replay states where at least two retrieved rows have different Top-1 predictions. On these states, RECAP exceeds confidence gating by 9.62–17.33 Recall@8 points (Figure 7; Table 16). Paired image-cluster intervals exclude zero on ChartQA, OCRBench, and MMT-Bench; TextVQA has only 52 such states and a wider interval crossing zero. These results support retaining complementary candidates when sources disagree. They isolate the benefit over hard source selection; the effect of confidence weighting relative to fixed priors is assessed separately above.

Figure 7. Fusion gains over confidence gating on Qwen replay states with source disagreement. Points show Recall@8 differences; bars give 95% confidence intervals from 2,000 paired image-cluster bootstrap resamples. The dashed line marks zero gain, and nn is the number of disagreement states.

DatasetStatesRECAPConfidence gateDifference95% CI
ChartQA42784.3166.98+17.33[12.08, 22.31]
TextVQA5275.0065.38+9.62[−2.33, 18.42]
OCRBench85762.4346.91+15.52[9.69, 20.70]
MMT-Bench51077.6562.55+15.10[10.27, 20.14]

Table 16. Recall@8 (%) on Qwen replay states with source disagreement. Differences and confidence intervals are in percentage points. Intervals use 2,000 paired image-cluster bootstrap resamples of the fixed evaluation set.

Observed fusion weights. Table 17 summarizes normalized weights from the online RECAP arm, grouping calls by the number of available rows. Each candidate-tree fusion call contributes equally, pooling both request orders; indices follow decreasing specificity among available sources. With three sources, pooled weights are (0.693, 0.199, 0.108) for Qwen and (0.688, 0.200, 0.112) for LLaVA, close to the prior (0.700, 0.200, 0.100). With two sources, the normalized prior is (0.778, 0.222), compared with observed means of (0.764, 0.236) and (0.763, 0.237). Thus confidence adjusts the allocation while preserving the average preference for specific sources. These averages describe aggregate weighting behavior; they do not imply small adjustments at every query.

ModelDatasetTwo available sourcesThree available sources
CallsMean (w1,w2)(w_1,w_2)CallsMean (w1,w2,w3)(w_1,w_2,w_3)
QwenChartQA15,101(0.764, 0.236)12,072(0.697, 0.197, 0.105)
TextVQA12,689(0.772, 0.228)8,353(0.700, 0.197, 0.103)
OCRBench20,883(0.758, 0.242)19,394(0.693, 0.199, 0.108)
MMT-Bench44,809(0.765, 0.235)32,106(0.689, 0.200, 0.110)
Pooled93,482(0.764, 0.236)71,925(0.693, 0.199, 0.108)
LLaVAChartQA33,922(0.764, 0.236)20,304(0.694, 0.198, 0.108)
TextVQA17,023(0.770, 0.230)6,563(0.700, 0.199, 0.102)
OCRBench29,778(0.759, 0.241)32,512(0.687, 0.200, 0.113)
MMT-Bench44,589(0.763, 0.237)36,050(0.685, 0.201, 0.115)
Pooled125,312(0.763, 0.237)95,429(0.688, 0.200, 0.112)

Table 17. Mean normalized fusion weights in the online ablation, averaged over calls at all candidate-tree positions. Each model uses 200 inputs under two orders. Source indices denote specificity ranks among available rows. Single-source calls have weight one and are omitted.

Per-Input Runtime Distributions

MMSpec latency. Following MMSpec (Shen et al., 2026), we compare paired AR/RECAP generation times from Table 2. Input ii contributes Ti(m)=∑r=1Ritir(m)T_i^{(m)} = \sum_{r=1}^{R_i} t_{ir}^{(m)} over RiR_i turns. Each tir(m)t_{ir}^{(m)} includes prefill and decoding, excluding prompt preparation and output text decoding. All 600 inputs enter F^m(t)=N−1∑i=1N1[Ti(m)≤t]\hat{F}_m(t) = N^{-1}\sum_{i=1}^{N} 1[T_i^{(m)} \le t] without filtering.

RECAP lowers median and upper-tail latency (Figure 6; Table 15). Qwen/LLaVA P95 falls by 47.69%/42.17%, and 87.33%/97.83% of inputs are faster than paired AR.

ModelMethodP50P90P95P99
QwenAR2.708.4411.6315.07
QwenRECAP1.524.466.087.56
LLaVAAR4.029.4214.2718.43
LLaVARECAP2.385.458.2510.67

Table 15. MMSpec per-input generation-latency percentiles in seconds, with lower values indicating faster generation. PkP_k denotes the kkth percentile; RECAP rows are shaded.

Figure 6. Per-input generation-latency CDFs on MMSpec for paired RECAP and AR runs. A leftward curve indicates lower latency. Multi-turn inputs contribute summed generation time across turns.

Single-turn speedup. For each single-turn benchmark in Table 1, we summarize all 1,000 paired throughput ratios from Equation (9) without filtering (Figure 8; Table 18). These are empirical input-level percentiles, not uncertainty intervals. Median speedup exceeds one on every benchmark for both models, and 81.9–98.8% of Qwen inputs and 88.2–99.2% of LLaVA inputs have higher throughput than paired AR. The distributions nevertheless vary across tasks: Qwen reaches a median of 1.66× on MMT-Bench, with 98.8% of inputs benefiting, versus 1.15× and 81.9% on TextVQA. The latter also has a P10 of 0.59×, showing that positive average gains coexist with substantial slowdowns on some inputs.

Figure 8. Per-input throughput speedup on the seven single-turn benchmarks (1,000 inputs per model and benchmark). Points mark medians and segments span P10–P90 of the input-level ratios. Dashed lines mark parity with AR; the percentages report inputs with strictly higher throughput than paired AR.

DatasetQwenLLaVA
P10P50P90ri>1r_i > 1 (%)P10P50P90ri>1r_i > 1 (%)
MMT-Bench1.201.662.2098.80.961.532.4688.2
SEED-Bench1.051.241.5296.91.091.291.6096.9
ScienceQA1.151.662.5098.50.991.492.3689.4
OCRBench1.001.372.9790.00.991.662.8688.9
ChartQA0.891.493.1988.41.231.561.9799.2
MathVista1.102.023.5591.31.251.652.2698.4
TextVQA0.591.152.4081.91.061.261.5695.0

Table 18. Distributions of paired throughput ratios for RECAP on the seven single-turn benchmarks (1,000 inputs per dataset and model). P10/P50/P90 report speedup factors; the final column for each model gives the percentage of inputs with strictly higher throughput than paired AR.

References

  1. [1]Ali E. Abbas. 2009. A kullback-leibler view of linear and log-linear pools. Decision Analysis, 6(1):25–37.DOI
  2. [2]Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan-Kelley, and William Brandon. 2024. Hydra: Sequentially-dependent draft heads for medusa decoding. arXiv preprint arXiv:2402.05109.arxiv.org/abs/2402.05109
  3. [3]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. Qwen2.5-vl technical report. Preprint, arXiv:2502.13923.arxiv.org/abs/2502.13923
  4. [4]Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774.arxiv.org/abs/2401.10774
  5. [5]Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318.arxiv.org/abs/2302.01318
  6. [6]Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024. Break the sequential dependency of llm inference using lookahead decoding. arXiv preprint arXiv:2402.02057.arxiv.org/abs/2402.02057
  7. [7]Edouard Grave, Armand Joulin, and Nicolas Usunier. 2016. Improving neural language models with a continuous cache. arXiv preprint arXiv:1612.04426.arxiv.org/abs/1612.04426
  8. [8]Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. 2024. Rest: Retrieval-based speculative decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1582–1595.
  9. [9]Yunhai Hu, Tianhua Xia, Zining Liu, Rahul Raman, Xingyu Liu, Bo Bao, Eric Sather, Vithursan Thangarasa, and Sai Qian Zhang. 2026. Dream: Drafting with refined target features and entropy-adaptive cross-attention fusion for multimodal speculative decoding. Advances in Neural Information Processing Systems, 38:167592–167612.arxiv.org/abs/2505.19201
  10. [10]Yuxuan Hu, Ke Wang, Xiaokang Zhang, Fanjin Zhang, Cuiping Li, Hong Chen, and Jing Zhang. 2025. Sam decoding: Speculative decoding via suffix automaton. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12187–12204.arxiv.org/abs/2411.10666
  11. [11]Haiduo Huang, Fuwei Yang, Zhenhua Liu, Xuanwu Yin, Dong Li, Pengju Ren, and Emad Barsoum. 2025. Specvlm: Fast speculative decoding in vision-language models. arXiv preprint arXiv:2509.11815.arxiv.org/abs/2509.11815
  12. [12]Jialiang Kang, Han Shu, Wenshuo Li, Yingjie Zhai, and Xinghao Chen. 2026. Vispec: Accelerating vision-language models with vision-aware speculative decoding. Advances in Neural Information Processing Systems, 38:115511–115532.arxiv.org/abs/2509.15235
  13. [13]Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172.arxiv.org/abs/1911.00172
  14. [14]Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International conference on machine learning, pages 19274–19286. PMLR.arxiv.org/abs/2211.17192

Paper details

vlnc:W_m74yjyvw
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents