WorksComputer Science
Text Speeding Itself Up: Recency and Restraint in Session-Local Drafting
Abstract
Speculative decoding lets a cheap draft model propose tokens that a large language model verifies in a single forward pass. When the verifier keeps a KV cache, verification is cheap and drafting must be nearly free. Session-local drafters, which predict from the prompt and the text generated so far, meet that bar without training. We compare a recency-weighted, backing-off, confidence-gated session memory against prompt lookup decoding in its reference form, which copies from the earliest earlier match, and in a variant that copies from the most recent one, with paired per-prompt bootstrap intervals throughout. On general, medical, and legal text with a Mistral-7B verifier, recency accounts for the speed: most-recent lookup gains up to 0.09 in speedup over earliest-match lookup on 500-token generations, and session memory's weighting and backoff add nothing measurable on top, at 30 to 40 times the drafting cost. Energy tells a different story. On medical and legal text, unselective drafters are 14% faster than plain decoding yet use as much energy per token, while drafters that abstain on weak evidence save 3 to 4 joules per 100 tokens; a three-token minimum match length on prompt lookup matches the confidence-gated session memory. Outputs are token-identical to greedy decoding in 94% of generations, with every divergence at a numerical near-tie. Under a repetition penalty, long-generation speedups shrink from 1.4-1.7x to 1.1-1.2x, but recency still helps and the length floor is the only drafter that saves energy in both domains.
Introduction
Speculative decoding [16, 3] lets a cheap draft model propose several tokens that a large verifier checks in one forward pass. When the verifier keeps a KV cache, so that each new token attends to stored keys and values instead of re-encoding the sequence, verification is cheap and the cost of drafting becomes the binding constraint: a draft token that takes longer to produce than the verification it saves is a net loss (van den Bosch, 2026). The field has moved accordingly, from separate neural draft models toward attached prediction heads [1, 17], retrieval from a corpus [9], and lookup in the prompt itself (Saxena, 2023).
Trained draft models that are cheap enough for this regime exist, such as the CPU-resident memory-based language models of Van den Bosch et al. (2025), but they are fixed at training time. van den Bosch (2026) report that a memory-based draft model trained on legal text has 38% of its drafts accepted on legal input and 5.9% on medical input, and falls below break-even under a cached verifier when the domain does not match.
The alternative is a drafter that needs no training data because it learns from the session itself, from the prompt and from the tokens the verifier has already accepted. Prompt lookup decoding (Saxena, 2023) is the simplest such drafter: it copies the continuation of an exact n-gram match found earlier in the sequence. We introduced a richer variant, a session memory: a recency-weighted variable-order n-gram store with longest-match backoff and an entropy-based confidence gate. On 500-token greedy generations it reaches 1.4–1.7×, though, as we show, mostly by copying the verifier’s own repetition. The question this paper answers is which of its ingredients do the work, and it does so by comparing against prompt lookup decoding in two forms: the reference behaviour, which copies from the earliest match, and a one-line change that copies from the most recent match. That comparison isolates recency; adding session memory on top isolates weighting and backoff; switching the gate on isolates abstention. All comparisons are paired over prompts against a same-session baseline with bootstrap intervals, because differences of a few percent are the object of study.
The answers are not the ones we expected when we built the mechanism, and they change what is worth recommending:
Recency is what matters for speed. Copying from the most recent match instead of the earliest one adds +0.016 to +0.027 to the speedup on 100-token general text and +0.08 on 500-token generations. Recency weighting and backoff add nothing measurable on top: session memory and most-recent lookup are within 0.02 of each other in every cell, and lookup drafts at a thirtieth to a fortieth of the cost (§ 4).
Selectivity is what matters for energy. At draft length 5, unselective drafters run about 14% faster than plain decoding on medical and legal text but use as much energy or more, because they raise mean power. The confidence gate loses a little speed and saves 2.7–3.9 joules per 100 tokens, and a plain order floor on prompt lookup, drafting only on matches of three or more tokens, saves the same amount (§ 5). The gate’s value is abstention on short contexts, not confidence, and the floor is the only drafter that still saves energy when a repetition penalty suppresses the verifier’s self-repetition.
Speculative output is greedy output, up to numerical ties. Across 2,700 generations, 94% are token-identical to greedy decoding. Every divergence sits at a position where the verifier’s top two candidates differ by at most 0.023 logits, against a median gap of 3.0, and it is independent of the drafter and the draft length (§ 6). This replaces the ROUGE-based argument of an earlier version of this work.
Long-generation gains are mostly repetition. Drafting value rises with position in a greedy generation, and the gap between most-recent and earliest-match lookup widens with it. Under a repetition penalty that removes most of the verifier’s self-repetition, every drafter falls from 1.4–1.7× to 1.08–1.16×, while recency still helps (§ 4.4).
Session-Local Drafters
Speculative decoding with a cached verifier
A draft model proposes tokens; the verifier checks them in a single pass and accepts the longest prefix matching its own greedy choices. Our verifier decodes greedily over a persistent KV cache with a rejection fast-path: the first draft token is verified against the logits left over from the previous step, which are available for free, so a step whose first token is rejected costs exactly one plain decoding step. The full -token verification pass is only performed when the first token is accepted, and on rejection the cache is truncated to the accepted prefix. A draft call that proposes nothing also costs one plain step; we call this abstaining. Acceptance is decided at token-ID level against the verifier’s greedy choice, so the emitted text is the verifier’s greedy output up to the numerical effect of batched versus single-token verification, which § 6 measures.
Prompt lookup
Prompt lookup decoding (Saxena, 2023) drafts by copying. Let be the prompt followed by every verifier-accepted token. For from 3 down to 1, the last tokens form a query; an earlier position where the query occurs is a match, and the tokens that followed it are the draft. The reference implementation accepts a match only if a full -token continuation follows it and it does not overlap the query, and returns the first valid match in index order, that is, the earliest occurrence. If no order matches, it abstains. We reimplemented it and checked it token for token against the reference function on random sequences, and use it in four forms (Table 1): earliest (the reference behaviour), most recent (identical, but returning the last valid match), an -token floor (most recent, with no backoff below order , for ), and agreement (most recent; at the matched order, draft only if every valid occurrence continues with the same token).
| Drafter | Draws on | Abstains if |
| Lookup, earliest | first match, | no full match |
| Lookup, most recent | last match, | no full match |
| Lookup, -token floor | last match, | no match |
| Lookup, agreement | all matches, top | successors differ |
| Session memory gated () | decayed counts, | nothing matches |
Table 1. The session-local drafters compared. All draw only on the prompt and verifier-accepted tokens.
Session memory
The session memory is a recency-weighted, variable-order n-gram store built from the generation context, over the verifier’s own SentencePiece vocabulary. Let be the tokens produced so far (the prompt followed by every verifier-accepted token). We use maximum order and decay .
Decayed counts. For each position and each order , the -gram is a context in which the token was observed. Each such observation contributes a weight that decays geometrically with its distance from the current end of the sequence,
so the most recent occurrence of a context–token pair carries weight and each earlier occurrence is discounted by a further factor .
Backoff prediction. Prediction is based on finding the longest match between stored -grams and the current. With as the last tokens of the current context, let
be the highest order at which the trailing context has been seen. The predictive distribution of the next token is the normalized decayed counts at that single order,
and the drafted token is . Orders are not interpolated: the single highest order is used as draft token; lower orders are ignored. If no order fires the store abstains.
Confidence. Each prediction also carries a scalar confidence that the gate (Section 2.4) thresholds. is heuristically defined as the match-order score times one minus the normalized entropy of ,
where is the entropy of over its support , the ratio normalizes it by its maximum, and a singleton support () takes the entropy term to be 0. The match-order score takes the values for orders . Confidence is thus highest for a long deterministic match (order 4, single successor, ) and lowest for a short high-entropy one (order 1, near-uniform successors, ). The score defines , not : it governs whether the prediction is used, and not which token is drafted.
Statelessness and cost. The store is rebuilt from the context at the start of each draft call by one left-to-right pass (Eq. 1) in time with hashed lookups, and holds no state between calls. It learns only from verifier-accepted tokens: the tokens drafted within a call extend the query context for later positions of that same draft but are never added to , so a rejected draft leaves no trace. Its cost grows with the context: 280 μs per call at 100 generated tokens and 600 μs at 500, against 10–14 μs for prompt lookup (Section 4.3).
How it differs from prompt lookup. Three differences separate the two. First, aggregation: lookup copies the continuation of one occurrence, while session memory pools every occurrence of the matched context into a distribution, discounted by distance (Eq. 1), and drafts its most probable successor. Second, the drafting unit: lookup copies a contiguous span of tokens from one place and needs tokens of continuation to exist, while session memory re-predicts each position from the extended context, so a draft can splice continuations from different places and can run past the end of any single match. Third, reach: session memory matches contexts of up to four tokens, lookup up to three. Both back off to shorter contexts and both abstain when nothing matches; gating is an option for either (Section 2.4).
The gate
A threshold on the confidence turns session memory into a selective drafter. At each draft position, if the store’s argmax is drafted; otherwise the draft stops there (gated session memory). Without a threshold (), session memory drafts whenever any order matches. We use , the value of the first version of this work.
The threshold has a simple reading. With and , requires (Eq. 4); at $n^\star=2 it additionally requires a single observed successor, at a normalized entropy of at most , and at at most . The gate is therefore an order floor of two, tightened by a consistency requirement that relaxes as the matched context grows. § 5 tests which half of that rule does the work.
Experimental Setup
Verifier and hardware. Mistral-7B-v0.1 Jiang et al., [12]1 in float16 decodes greedily on the MPS accelerator of a Mac Studio (Apple M1 Max, 32 GB RAM), with the persistent KV cache and rejection fast-path of § 2. All drafters run on CPU.
Prompts. We evaluate on the test splits of WikiText-103 (Merity et al., 2017) (general), PubMed Summarization (Cohan et al., [6]) (medical), and EurLex (Chalkidis et al., [2]) (legal). From each split we take, in its native order, the first 100 documents of at least 60 words and use each document’s first 50 whitespace-delimited words as a prompt, which averages 67, 72, and 100 Mistral tokens on general, medical, and legal text.2 The verifier generates 100 tokens unless stated otherwise.
Experiments. The prompt-lookup comparison (§ 4.1) runs plain decoding, both lookup forms, and ungated session memory on all 300 prompts at . The 500-token comparison (§ 4.2) uses the first 30 general and legal prompts at and adds gated session memory. The energy sweeps (§ 5) repeat both with system power logged, and add the gated lookup forms. The within-generation analysis (§ 4.4) uses the per-step traces of the 500-token runs.
Protocol. Each domain’s sweep starts with plain decoding of its prompts in the same session, and speedup is the per-prompt ratio of a drafter’s throughput to that baseline. Differences between drafters are paired over prompts, with 95% percentile intervals from 10,000 bootstrap resamples of prompts. Ungated session memory, re-run alongside the new drafters, reproduces the first version of this work within 0.2 points of acceptance in all nine configurations and within 0.03 of speedup at (0.05 at ; per-prompt correlation 0.97), although operating-system and library updates since lowered absolute throughput by about 23%; we therefore compare only ratios measured within one session.
Metrics. We report acceptance (accepted over proposed draft tokens), tokens per verifier step (generated tokens over decoding steps, which sets throughput), abstention (the share of draft calls proposing nothing), drafting time as a share of wall time, energy per 100 tokens (§ 5), and exact token agreement with greedy decoding (§ 6).
Speed: Recency, Not Weighting
Prompt lookup versus session memory
Table 2 compares the three drafters on 100-token generations. All lie within 0.03 of each other, between 1.10× and 1.25×, while per-prompt speedups spread widely (standard deviations near 0.25 on general text), so we read differences from paired intervals.
| Domain | Lookup, earliest | Lookup, most recent | Session memory | recency | weighting | |
| General | 3 | 1.153 [1.122, 1.183] | 1.169 [1.139, 1.200] | 1.165 [1.134, 1.197] | +0.016* | -0.004* |
| General | 5 | 1.216 [1.170, 1.266] | 1.236 [1.187, 1.286] | 1.227 [1.179, 1.280] | +0.020* | -0.009* |
| General | 8 | 1.217 [1.159, 1.282] | 1.245 [1.181, 1.311] | 1.243 [1.177, 1.313] | +0.027* | -0.002 |
| Medical | 3 | 1.110 [1.089, 1.132] | 1.120 [1.100, 1.143] | 1.116 [1.095, 1.139] | +0.011* | -0.004* |
| Medical | 5 | 1.139 [1.106, 1.174] | 1.152 [1.119, 1.188] | 1.148 [1.115, 1.184] | +0.013 | -0.004* |
| Medical | 8 | 1.127 [1.085, 1.175] | 1.139 [1.095, 1.190] | 1.131 [1.087, 1.179] | +0.012 | -0.008* |
| Legal | 3 | 1.130 [1.115, 1.146] | 1.128 [1.113, 1.143] | 1.123 [1.108, 1.138] | -0.002 | -0.005* |
| Legal | 5 | 1.147 [1.125, 1.171] | 1.143 [1.123, 1.164] | 1.135 [1.116, 1.156] | -0.003 | -0.008* |
| Legal | 8 | 1.141 [1.113, 1.174] | 1.134 [1.106, 1.165] | 1.122 [1.098, 1.148] | -0.007 | -0.012* |
Table 2. Prompt lookup decoding versus session memory, KV-cached Mistral-7B, 100-token generations, 100 prompts per domain. Speedup is the per-prompt ratio to plain decoding in the same session, mean with 95% bootstrap interval over prompts. recency is most-recent minus earliest match (what recency alone buys); weighting is session memory minus most-recent lookup (what recency weighting and backoff add on top). *: interval excludes zero.
Recency helps where text repeats. Most-recent lookup beats earliest-match lookup by +0.016 to +0.027 on general text at every draft length and by about +0.01 on medical text, and makes no difference on legal text, where early and late occurrences of formulaic contexts tend to continue identically. Weighting and backoff do not help. Session memory is below most-recent lookup in all nine cells, by 0.002 to 0.012, and never reliably above it. Its tokens per verifier step match lookup’s to within 0.02 (for example 1.743 against 1.740 on general text at ), so the small deficit is its drafting cost, 0.28% of wall time against 0.01%.
Long generations
Over 500-token generations (Table 3) the recency effect grows about fourfold: most-recent lookup is +0.084 [+0.060, +0.111] faster than earliest-match lookup on legal text, winning on all 30 prompts, and +0.089 [+0.055, +0.126] on general text. As a generation lengthens, the earliest occurrence of a context is increasingly likely to predate the text’s current phrasing, while the most recent one is where the verifier is repeating itself now. Session memory still adds nothing: ungated, it is −0.015 [−0.029, +0.003] below most-recent lookup on legal text and −0.009 [−0.016, −0.003] on general text, with the same tokens per step.
| Domain | Drafter | Acc. % | Abstain % | Tok./step | Speedup | Draft % wall |
| General | Lookup, earliest | 49.9 | 30 | 2.82 | 1.622 [1.500, 1.740] | 0.01 |
| General | Lookup, most recent | 56.3 | 32 | 2.98 | 1.711 [1.585, 1.829] | 0.01 |
| General | Session memory | 55.4 | 31 | 2.98 | 1.703 [1.578, 1.821] | 0.39 |
| General | Session memory, gated | 76.4 | 55 | 2.81 | 1.682 [1.550, 1.806] | 0.41 |
| Legal | Lookup, earliest | 30.7 | 28 | 2.11 | 1.366 [1.316, 1.419] | 0.01 |
| Legal | Lookup, most recent | 35.9 | 30 | 2.27 | 1.450 [1.384, 1.517] | 0.01 |
| Legal | Session memory | 36.1 | 30 | 2.28 | 1.435 [1.368, 1.503] | 0.50 |
| Legal | Session memory, gated | 60.9 | 66 | 2.01 | 1.406 [1.346, 1.467] | 0.56 |
Table 3. 500-token generations, , 30 prompts per domain. Acceptance is per proposed token; abstain is the share of draft calls that proposed nothing; tokens per step counts verifier calls. The gate raises acceptance by abstaining, which lowers tokens per step and speed.
The gate raises acceptance from 36% to 61% on legal text and from 55% to 76% on general text, but it does so by abstaining on 55–66% of draft calls against 30% for the ungated store. Fewer drafts mean fewer tokens per verifier step (2.01 against 2.28 on legal text), and the gated store is 0.02–0.03 slower than the ungated one. When drafters abstain at different rates, acceptance per proposed token does not measure drafting value; tokens per verifier step does.
Drafting cost
Session memory rebuilds its store at every draft call in time, so its cost per call grows with the context and its cumulative cost with the square of the output length. We measure 280 s per call at 100 generated tokens (0.28% of wall time) and 600 s at 500 tokens (0.45–0.48%). Our unoptimized Python prompt lookup costs 10 and 14 s (0.01%). Both are small against a verifier step of about 75 ms at these lengths, but session memory’s share would reach several percent at a few thousand tokens without an incremental, per token update, which we leave to future work.
Within-generation dynamics and repetition
Session-local drafters learn from the generation as it accumulates. Table 5 tracks accepted draft tokens per verifier step over 500-token greedy generations: on legal text most-recent lookup rises from 0.43 in the first hundred tokens to 2.68 in the last, and gated session memory’s acceptance per proposed token climbs from 34% to 78% (47% to 86% on general text). The gap between the two lookup forms widens with position, from 0.02 to 0.46 tokens per step on legal text, which is where the fourfold growth of the recency effect between 100 and 500 tokens comes from. Much of the climb is the verifier repeating itself, a known property of maximisation-based decoding [10]: positions whose preceding 4-gram occurred earlier in the output rise from 7% of the first hundred tokens to 81% of the last on legal text, which is also why recency-aware copying captures the effect as well as the full store does.
| Drafter | Generated tokens 1–100 | –200 | –300 | –400 | –500 |
| Legal Lookup, earliest | 0.41 | 0.79 | 1.34 | 1.62 | 2.22 |
| Legal Lookup, most recent | 0.43 | 0.87 | 1.50 | 1.92 | 2.68 |
| Legal Session memory, gated | 0.30 | 0.62 | 1.24 | 1.60 | 2.29 |
| General Lookup, earliest | 0.44 | 1.55 | 2.06 | 2.07 | 2.34 |
| General Lookup, most recent | 0.44 | 1.77 | 2.52 | 2.39 | 2.79 |
| General Session memory, gated | 0.34 | 1.47 | 2.14 | 2.04 | 2.59 |
Table 5. Accepted draft tokens per verifier step by position within 500-token greedy generations (, 30 prompts per domain). Drafting value rises as the output accumulates, and the gap between most-recent and earliest-match lookup widens.
Without the repetition. To test whether the gains survive text that does not repeat itself, we rerun the 500-token comparison with the repetition penalty of Keskar et al. [13] () on the verifier. Verification stays exact: each draft position is checked against the penalised argmax given the context up to it, and penalised plain decoding reproduces the library’s penalised generation. The penalty works as intended (Table 4): in the last 100 tokens, repeated 4-gram contexts fall from 81% to 28% on legal and from 84% to 44% on general text, distinct bigrams rise from 0.39 to 0.70 and from 0.25 to 0.72, and perplexity under the unpenalised verifier stays low (1.58 and 2.35, against 1.19 and 1.30 for the degenerate greedy output); 11–12 of 30 outputs end before 500 tokens. Without the loops, every drafter falls to 1.08–1.16×: most of the long-generation gain was the verifier repeating itself. Recency still helps, with most-recent lookup ahead of earliest-match lookup by +0.029 in both domains (legal [+0.018, +0.044], general [−0.000, +0.056]), and session memory still adds nothing over it.
| Drafter | Legal Speedup greedy | Legal Speedup penalised | Legal J/100 penalised | General Speedup greedy | General Speedup penalised | General J/100 penalised |
| Plain decoding | 1 | 1 | 106.4 | 1 | 1 | 101.7 |
| Lookup, earliest | 1.37 | 1.09 | 107.7 | 1.62 | 1.14 | 99.9 |
| Lookup, most recent | 1.45 | 1.12 | 105.9 | 1.71 | 1.16 | 98.7 |
| Lookup, 3-token floor | 1.40 | 1.09 | 102.0* | 1.67 | 1.16 | 95.3* |
| Session memory, gated | 1.41 | 1.08 | 105.9 | 1.68 | 1.14 | 96.5* |
Table 4. 500-token generations with a repetition penalty () on the verifier, , 30 prompts per domain, against greedy decoding. The penalty lowers the share of positions in the last 100 tokens whose preceding 4-gram occurred earlier from 81% to 28% (legal) and from 84% to 44% (general). Energy in joules per 100 tokens; *: paired saving over plain decoding with a 95% bootstrap interval excluding zero.
Energy: Selectivity, Not Speed
Energy per token is not speed inverted. van den Bosch (2026) showed that under a cached verifier mean system power rises with draft length, because dense -token verification passes work the accelerator harder than one-token steps, so a faster configuration can cost more energy. We measure whole-system power (CPU, GPU and neural engine) at 1 Hz with powermetrics, integrate it over each generation, and report joules per 100 generated tokens, paired per prompt against plain decoding in the same session. Table 6 covers draft length 5, where the gate’s effect on speed is largest, at 100 tokens on all three domains and at 500 tokens on two.
| Drafter | Abstain % | 100 tokens, General | 100 tokens, Medical | 100 tokens, Legal | 500 tokens, General | 500 tokens, Legal |
| Plain decoding | – | 92.7 | 93.7 | 98.2 | 105.3 | 108.2 |
| Lookup, most recent | 41 | 89.5* | 94.6* | 99.2* | 78.2 | 90.0* |
| Lookup, 2-token floor | 69 | 87.5* | 92.0* | 96.8* | 77.7 | 88.4 |
| Lookup, 3-token floor | 79 | 86.7 | 91.2 | 96.1* | 78.4 | 88.6 |
| Lookup, agreement | 64 | 88.6* | 93.4* | 96.7* | 78.7* | 89.7* |
| Session memory, gated | 72 | 86.8 | 91.0 | 95.3 | 77.5 | 87.9 |
Table 6. Energy in joules per 100 generated tokens (CPU+GPU+ANE power integrated over each generation; mean over prompts). *: paired difference from gated session memory has a 95% bootstrap interval excluding zero. A 3-token order floor on lookup comes within 1 J of gated session memory in every column.
Unselective drafting saves time, not energy. On medical and legal text at 100 tokens, most-recent lookup is 1.14× faster than plain decoding and uses 94.6 and 99.2 J per 100 tokens against plain’s 93.7 and 98.2: no saving at all. Mean power rises from 12.5–12.7 W for plain to 14.2–14.6 W, and the rise eats the time saved. Ungated session memory behaves the same way. Only on general text, where acceptance is higher, does an unselective drafter save energy (89.5 versus 92.7 J).
The gate converts speed into energy. Gated session memory has the lowest energy in every column of Table 6. Against most-recent lookup it saves 2.7, 3.6 and 3.9 J per 100 tokens on general, medical and legal text, and 2.1 J on 500-token legal generations, with bootstrap intervals well clear of zero; on 500-token general text the difference is within noise. It pays for this with 0.02–0.03 of speedup (§ 4). The mechanism is visible in the abstention column: the gate declines to draft on 78–82% of calls at 100 tokens and 55–66% at 500, and the tokens it declines to draft are the ones a verifier would most likely have rejected. Verifying a rejected draft costs a dense forward pass and returns one token, which is the expensive case; abstaining costs a cheap single-token pass and returns one token.
An order floor on lookup does the same job. If abstention is the mechanism, prompt lookup should gain as much from a rule that declines short matches. It does. Drafting only on matches of two or more tokens recovers about three quarters of the gate’s saving; drafting only on matches of three tokens matches gated session memory within noise in all five columns (the paired difference is at most 0.9 J and its interval includes zero in four of five). Consistent with the reading of the gate in Section 2.4, an order floor of two tightened by a consistency requirement, the lookup rule that matches it best is the stricter floor of three. A gate on the agreement of earlier continuations, the closest lookup analogue of session memory’s entropy term, abstains as often but saves less, so what matters is the length of the matched context, not the consensus of what followed it. The relation is monotone: across the six drafters, more abstention means less energy, at a speed cost that never exceeds 0.03. At draft length 3 the effect shrinks to under 1 J per 100 tokens, as expected when a rejected step wastes less.
Without repetition. Under the repetition penalty (Table 4), unselective lookup saves no energy at all (105.9 against 106.4 J per 100 tokens on legal text, 98.7 against 101.7 on general text, intervals including zero). The three-token floor is the one drafter that saves reliably in both domains, 4.4 J [2.6, 6.2] on legal and 6.5 J [2.2, 11.2] on general text, and on legal text it beats gated session memory by 3.9 J [2.0, 6.2], while the gate’s own saving there is not measurable.
Recommendation. Under a cached verifier, draft with session-local lookup from the most recent match and refuse contexts shorter than three tokens: no parameters, no training, 10–14 μs per call, the energy saving of the gated session memory without its store, and the only saving that holds whether or not the output repeats itself.
Output Agreement
Acceptance compares token IDs with the verifier’s greedy choice, so speculative decoding should reproduce greedy output exactly. It need not quite, because a -token verification pass and single-token passes compute marginally different float16 logits. We measure the consequence directly on the 100-token generations of § 4.1 (breakdown in Table 7, Appendix § 9). Two non-speculative references, the library’s greedy generation function and our decoder with drafting disabled, agree on all 300 prompts. Against them, 2,539 of 2,700 speculative generations (94.0%) are token-identical. Agreement does not depend on the drafter (93.4–94.4%) or the draft length (93.8–94.2%), and the 161 divergent generations reduce to 47 distinct divergence points in 45 prompts: the same prompt diverges at the same position, to the same token, whichever drafter runs. The first divergence falls at a median of token 38; two thirds of divergent sequences share an 8-token run with the reference again later, and the rest stay split.
| Identical | % | ||
| Domain | General | 840/900 | 93.3 |
| Domain | Medical | 838/900 | 93.1 |
| Domain | Legal | 861/900 | 95.7 |
| Drafter | Lookup, earliest | 841/900 | 93.4 |
| Drafter | Lookup, most recent | 850/900 | 94.4 |
| Drafter | Session memory | 848/900 | 94.2 |
| 3 | 847/900 | 94.1 | |
| 5 | 844/900 | 93.8 | |
| 8 | 848/900 | 94.2 |
Table 7. Exact token agreement between speculative output and greedy decoding (2,700 100-token generations). The 47 distinct divergence points have a top-two logit margin of at most 0.023 (median 0.0) against a median of 3.03 over all generated positions; in 45 of 47 the speculative token is the reference’s runner-up.
Every divergence is a numerical tie. Teacher-forcing the greedy reference through the verifier, the gap between the top two logits at the 47 divergence points has a median of 0 (25 are exact float16 ties) and a maximum of 0.023, against a median of 3.0 over all 30,000 generated positions; only 1.1% of positions have a gap that small. In 45 of the 47 cases the speculative token is the reference’s runner-up. A float32 re-run, which would break these ties, does not fit in the machine’s 32 GB.
Related Work
Speculative decoding was introduced with a smaller neural model as the draft [16, 3]. Trained drafting heads attached to the verifier [1, 17], with dynamic draft trees in EAGLE-2 [17], give the strongest speedups on accelerators but must be trained per verifier and add accelerator work, which places them outside the training-free, CPU-only setting studied here. Training-free drafters retrieve continuations from a datastore [9], a reference document (Yang et al., 2023), the prompt (Saxena, 2023), or suffix structures over the prompt and earlier outputs [11] (Oliaro et al. 2025). Our results support that design direction: what separates good drafts from wasted verification is the recency and length of the matched context, not a weighted distribution over its continuations. Memory-based language models [7] (Van den Bosch et al. 2025) provide a trained, CPU-resident draft at similar cost; van den Bosch (2026) evaluate them as static drafters, with energy under cached and uncached verification.
That a text’s recent history predicts its continuation is old knowledge: words recur in bursts [5, 4], cache language models exploit this [15] (Rosenfeld 1996), and neural models retrieve recent context through caches, pointers, and nearest neighbours [8, 14] (Merity et al. 2017). These target perplexity; session-local drafting uses the same device to cut cost.
Online speculative decoding (Liu et al., 2024) adapts a neural draft across queries by distillation; session-local drafting adapts within one generation.
Conclusion
Session-local drafting suits a cached verifier on constrained hardware: it needs no training, no accelerator, and almost no time, and having no training data, it cannot meet the wrong domain. Of its ingredients, only two matter. For speed, copy from the most recent occurrence of the current context; recency weighting and backoff add nothing, at 30 to 40 times the drafting cost. For energy, refuse to draft on contexts shorter than three tokens, which reproduces the gated session memory’s saving and, once the verifier’s repetition is suppressed and speedups shrink to 1.1–1.2×, is the only rule that still saves energy. Output matches greedy decoding in 94% of generations and differs elsewhere only at numerical ties. The resulting drafter is a scan over the text so far and a length check.
Limitations
Scope. All measurements use one verifier, Mistral-7B, on one consumer platform. Verifiers with other attention designs, and batched serving, shift the balance between drafting and verification cost that drives both our speed and our energy results. Decoding is greedy, with and without a repetition penalty; sampling changes the acceptance rule and is untested. The penalty shows that long-generation speedups depend heavily on the verifier’s self-repetition (Section 4.4), and we test a single penalty strength. We measure 100- and 500-token outputs from prompts cut at a fixed 50-word length; longer contexts, conversational and code workloads, and non-English text are untested.
Implementation. Session memory rebuilds its store at every call, with cumulative cost quadratic in output length (Section 4.3); an incremental store would remove this but we did not implement it. The prompt-lookup forms are straightforward Python, not optimized; since they are already 30 to 40 times cheaper per call, optimization would only widen the gap.
Measurement. Energy figures are whole-system and hardware-specific, and support relative comparison on identical hardware, not absolute cross-platform claims. Background load and backup activity were logged every 30 s during all timed runs, and blocks that overlapped load bursts were re-run. Divergences from greedy output were characterized by logit margins, not by a float32 re-run. Code and scripts to reproduce every table are available at https://github.com/antalvdb/olifant-speculative.
Ethical Considerations
This work targets the efficiency of language-model inference, and its main impact is a reduction in the energy cost of on-device text generation: the selective drafters we recommend use 2–7% less energy per token than plain decoding at 100 tokens, 18–26% less at 500 tokens of greedy output, and 4–6% less at 500 tokens under a repetition penalty. Speculative output is token-identical to greedy decoding in 94% of generations and differs elsewhere only where the verifier’s top candidates are numerically tied, so the methods alter the cost of generation, not its content, and introduce no capability or misuse surface beyond that of the underlying verifier. Session-local drafters store only tokens the verifier has itself already produced, adding no external data and no new privacy exposure; by keeping drafting on CPU with no accelerator footprint, they support fully on-device generation in privacy-sensitive settings such as the medical and legal domains we study. Two caveats apply: cheaper inference can raise aggregate usage, a rebound effect that may partially offset per-token savings, and our energy figures are whole-system and hardware-specific.
Token Agreement by Domain, Drafter, and Draft Length
References
- [1]Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Proceedings of the 41st International Conference on Machine Learning (ICML 2024), volume 235 of Proceedings of Machine Learning Research, pages 5209–5235. PMLR.arxiv.org/abs/2401.10774
- [2]Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. 2022. LexGLUE: A benchmark dataset for legal language understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4310–4330, Dublin, Ireland. Association for Computational Linguistics.
- [3]Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318.arxiv.org/abs/2302.01318
- [4]Kenneth W. Church. 2000. Empirical estimates of adaptation: The chance of two Noriegas is closer to p/2 than p². In Proceedings of the 18th International Conference on Computational Linguistics (COLING 2000), pages 180–186.
- [5]Kenneth W. Church and William A. Gale. 1995. Poisson mixtures. Natural Language Engineering, 1(2):163–190.DOI
- [6]Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 615–621, New Orleans, Louisiana. Association for Computational Linguistics.arxiv.org/abs/1804.05685
- [7]Walter Daelemans and Antal Van den Bosch. 2005. Memory-Based Language Processing. Cambridge University Press.
- [8]Edouard Grave, Armand Joulin, and Nicolas Usunier. 2017. Improving neural language models with a continuous cache. In 5th International Conference on Learning Representations (ICLR 2017).arxiv.org/abs/1612.04426
- [9]Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. 2024. REST: Retrieval-based speculative decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1852–1595, Mexico City, Mexico. Association for Computational Linguistics.
- [10]Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.
- [11]Xuxuan Hu and 1 others. 2024. SAM decoding: Speculative decoding via suffix automaton. arXiv preprint arXiv:2411.10666.arxiv.org/abs/2411.10666
- [12]Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, and 1 others. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.arxiv.org/abs/2310.06825
- [13]Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858.
- [14]Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations.arxiv.org/abs/1911.00172
- [15]Roland Kuhn and Renato De Mori. 1990. A cache-based natural language model for speech recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(6):570–583.
- [16]Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19274–19286. PMLR.arxiv.org/abs/2211.17192
- [17]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024a. EAGLE-2: Faster inference of language models with dynamic draft trees. In Proceedings of EMNLP 2024.arxiv.org/abs/2406.16858