betaFeedback

WorksComputer Science

Give the Drafter a Second Chance to Guess: Parallel Reconsideration for Speculative Decoding

Abstract

Speculative decoding accelerates large language model inference with a small drafter, whose per-guess acceptance rate determines the achievable speedup. Feature-level drafters share a design choice: every token, easy or hard, is drafted under a single forward pass. Many of the tokens the target rejects are ones the drafter is itself unsure about yet the target answers decisively. To address this, we introduce Parallel Reconsideration: guided by its own uncertainty, the drafter reconsiders its hesitant guesses with extra passes that run in parallel with the one-pass draft, adding negligible latency. A co-designed training recipe and serving-time drafting algorithm append these reconsidered guesses to the original draft tree as refined candidates. Across models and benchmarks, on average, Parallel Reconsideration improves over the baselines by 5.3% to 10.2% on acceptance length and by 3.5% to 9.2% on speedup.

Introduction

Large language models (LLMs) have become a pivotal component of modern computing, powering assistants, search, and the fast-growing class of agentic systems that plan, call tools, and write code over long horizons ([3, 19] (Moonshot AI 2026; OpenAI 2025)). Yet this shift exposes a stubborn cost: in the agentic era even a simple coding task can unfold into generated tokens across many chained model calls, and each of those tokens is produced one at a time. This is not an implementation artifact but the nature of autoregressive decoding: every token is conditioned on all previous tokens, so a single forward pass of a multi-billion-parameter model yields exactly one token, and end-to-end latency scales with the length of the generation ((Pope et al., 2023)). As outputs grow longer, this sequential bottleneck, rather than raw compute, increasingly governs how fast a model can respond.

Speculative decoding directly mitigates this bottleneck ([22] (Xia et al. 2023)). A cheap drafter proposes several candidate tokens, and the target model verifies them in parallel in a single forward pass. A modified rejection sampling step keeps the longest run of drafted tokens the target agrees with, provably preserving its output distribution exactly ([22]); the saving comes from parallelism, as kk accepted tokens cost one target pass instead of kk expensive ones. The drafter thus decides everything: the more tokens it gets right per round, the greater the speedup. One notable class is the EAGLE family ((Li et al., 2024a,b, 2025)), which trains a lightweight drafter that predicts at the feature level rather than the token level, reusing the target’s hidden states to draft with a single small decoder layer and setting the standard for modern speculative decoding.

This efficiency, however, comes with a design choice shared by feature-level drafters: every token, easy or hard, is drafted with the same fixed budget of a single forward pass through a small decoder.1 Some proposed tokens the target rejects are not ones the draft gets confidently wrong; they are tokens the draft is itself unsure about even though the target has a clear answer there. We hypothesize that part of this loss stems from how much the draft can compute: a single pass of one small layer may not suffice to settle these hard tokens, yet the draft must commit and move on regardless.

The obvious fix is to give the drafter more capacity, e.g. more layers. Yet a bigger drafter slows the entire round and eats into the speedup that is the whole point of speculative decoding. This exposes the real tension: how can the drafter spend more compute on the tokens that need it, without paying for that compute on the tokens that do not? A viable answer must be selective, spending the extra compute only at the uncertain positions; (near-)free, adding negligible wall-clock latency over the ordinary draft; and lossless, leaving the verification untouched.

We answer this with Parallel Reconsideration, which grants the drafter a second chance to guess on the tokens it is unsure about. The drafter’s own uncertainty is the signal: where it is already decisive, the guess is committed as usual; where it hesitates, the drafter is given more attempts to revise its guess into a better one before committing. Moreover, this reconsideration is nearly free: the additional attempts are batched into the same forward pass as the ordinary draft, adding only negligible parallel overhead at inference time. Our contributions are as follows:

  • Parallel Reconsideration. We introduce uncertainty-guided draft-time scaling to feature-level drafters: instead of a fixed one-pass-per-token budget, the drafter scales its computation per position according to its own uncertainty. Parallel Reconsideration reconsiders the uncertain guesses with additional passes that run in parallel with the ordinary draft, so the added computation is nearly free (§ 3).

  • A co-design of training and serving. We design a training recipe under which the reconsideration branch deliberately learns to correct the drafter’s mistakes without degrading its ordinary drafting branch (§ 4). In serving, we design a drafting algorithm that grafts the reconsidered guesses onto the ordinary draft tree as additional candidates (§ 5).

  • Extensive empirical evaluation. Across models and benchmarks, Parallel Reconsideration improves acceptance length over the baselines by 5.3–10.2% on average across models, draft budgets, and sampling temperatures, at a nearly identical per-round drafting cost. We further show that spending the same compute on a wider draft or a second decoder layer does not match it, and ablate each component of the design (§ 6).

Background and Motivation

Preliminaries and the cost model

We consider an EAGLE-style drafter ((Li et al., 2024a)) that predicts at the feature level. Let fp0f_p^0 denote the draft feature at speculative position pp produced by a single forward pass of the drafter’s single decoder layer MM, let e(t)e(t) be the embedding of token tt, let fc⁡(g,e)\operatorname{fc}(g,e) be EAGLE’s fusion of a feature gg with a token embedding ee (concatenate, then project), and let head⁡(⋅)\operatorname{head}(\cdot) be the LM head. To advance one position, the drafter samples a token tp+1∼head⁡(fp0)t_{p+1} \sim\operatorname{head}(f_p^0) from the current feature and predicts the next feature by fp+10=M(fc⁡(fp0,e(tp+1)))f_{p+1}^0 = M(\operatorname{fc}(f_p^0,e(t_{p+1}))), one draft step. Each round, the drafter generates candidate tokens in γ\gamma steps sequentially, incurring a drafting cost

Tdraft=γ⋅tstep,(1)T_{\mathrm{draft}} = \gamma\cdot t_{\mathrm{step}}, \tag*{(1)}

where tstept_{\mathrm{step}} is the latency of a single forward pass of MM; the drafting cost therefore grows linearly with γ\gamma. The target model then verifies the candidates in one forward pass, costing TverifyT_{\mathrm{verify}}, and accepts the longest prefix it agrees with. The expected number of tokens accepted per round is the *acceptance length τ\tau, and the amortized cost per accepted token is

Tdraft+Tverifyτ=γ tstep+Tverifyτ.(2)\frac{T_{\mathrm{draft}} + T_{\mathrm{verify}}}{\tau} = \frac{\gamma\, t_{\mathrm{step}} + T_{\mathrm{verify}}}{\tau}. \tag*{(2)}

The goal is therefore to raise the acceptance length τ\tau while holding Tdraft+TverifyT_{\mathrm{draft}} + T_{\mathrm{verify}} on par with prior methods: a drafter that raises τ\tau by inflating γ tstep\gamma\,t_{\mathrm{step}} can easily lose more in the numerator than it gains in the denominator. Underlying this formulation is an implicit assumption that prior work has left unquestioned: each step runs MM exactly once, so every token, easy or hard, is drafted from a single forward pass with no opportunity to revise.

Where does the drafter lose acceptance?

To understand where acceptance is lost, we analyze a trained EAGLE-3.1 drafter for Llama-3.1-8B-Instruct on a held-out split. For each drafted position pp, we read off two quantities: the drafter confidence cpd=max⁡thead⁡(fp0)tc_p^{\mathrm{d}} = \max_t \operatorname{head}(f_p^0)_t, its own top-1 probability, and the target confidence cpt=max⁡tπtgt(t∣x<p)c_p^{\mathrm{t}} = \max_t \pi_{\mathrm{tgt}}(t \mid x_{<p}), the top-1 probability of the target’s next-token distribution at the same position. The drafter confidence measures how sure the drafter is; the target confidence measures how resolvable the position is, i.e. when cptc_p^{\mathrm{t}} is high, a single correct token exists and can, in principle, be predicted by the drafter. Figure 1 summarizes two observations relating these quantities to acceptance.

Acceptance rate versus drafter confidence and draft versus target confidence

Figure 1. Two observations on where an EAGLE-style drafter loses acceptance. (a) Acceptance rate versus the drafter’s own top-1 confidence, which falls from 95.6% on confident positions to 7.3% on the least confident. (b) Drafter confidence against target confidence over the rejected tokens (% of the reject pool): the drafter is unsure on 62.1% of them, and 17.0% of the pool is unsure at a decisive target (cpt≥0.9c_p^{\mathrm{t}} \ge0.9).

Observation 1: confidence measures the opportunity cost of a drafted token. Acceptance tracks closely (Figure 1a): positions the drafter is confident about (cpd≥0.9c_p^{\mathrm{d}} \ge0.9) are accepted 95.6% of the time, and acceptance falls steadily as its confidence drops: to 0.52 at 0.5–0.6 and 0.07 in the lowest bucket. The drafter is thus seldom confidently wrong; where it loses acceptance, it was already unsure. The drafter’s own confidence, available at draft time, therefore indicates how likely the target is to reject the token it drafts.

Observation 2: a fair share of rejections is recoverable, but confidence does not say which. Among rejected tokens (Figure 1b), the drafter is unsure on 62.1% of them, and 43.6% fall at a target that still favors one token (cpt≥0.5c_p^{\mathrm{t}} \ge0.5), 17.0% of rejected tokens at a decisive one (cpt≥0.9c_p^{\mathrm{t}} \ge0.9), against a comparable 18.5% where the target is unsure (cpt<0.5c_p^t < 0.5). A fair share of the acceptance the drafter loses is thus recoverable: at those positions a token the target would have accepted was there to be drafted. But the drafter cannot tell which: its confidence is equally low at a decisive target and at an ambiguous one.

Spend compute where the drafter is unsure

The two observations compose into a single reading: the drafter loses acceptance on tokens it is itself unsure about (Observation 1), on a fair share of which the target nonetheless has a decisive answer (Observation 2). It is not confidently wrong at these positions, and a correct token is there to be drafted, yet it is committed prematurely: a single draft step is not enough to resolve the uncertain position pp before the drafter must commit its guess and move to the next position p+1p + 1. What the drafter is missing, then, is not a better objective but more computation:

Give the drafter more compute where it is unsure, and none extra where it is not.

Within feature-level drafters, prior work mostly refines what the drafter computes in a single step rather than how much it computes, which stays fixed at one forward pass of MM per position. A naive bigger drafter with more compute is self-defeating under Eq. (2), since it raises tstept_{\mathrm{step}} on every position, including the confident majority that one step already resolves, inflating TdraftT_{\mathrm{draft}} faster than it lifts τ\tau (see § 6.5 for details). We therefore scale draft-time compute selectively, spending extra draft steps only on the uncertain positions where the acceptance lost is recoverable, and leaving the confident majority at one forward pass, as Parallel Reconsideration (PR). § 6 revisits these observations and analyzes whether Parallel Reconsideration indeed helps on the raised challenges.

Parallel Reconsideration

We introduce Parallel Reconsideration with the reconsideration step (§ 3.1), the learned self-query that lets the drafter refine the single-pass draft rather than growing the next one (§ 3.2), the learned gate that decides which positions have a chance of drafting a better token through reconsideration (§ 3.3), and the parallel execution that keeps it cheap (§ 3.4). We describe the method with a single reconsideration step for clarity, though in practice, the drafter may take more than one.

The reconsideration step

Recall that the ordinary draft step at position pp reads the one-pass feature fp0f_p^0, samples the next token tp+1∼head⁡(fp0)t_{p+1} \sim\operatorname{head}(f_p^0), and advances to position p+1p + 1 via fp+10=M(fc⁡(fp0,e(tp+1)))f_{p+1}^0 = M(\operatorname{fc}(f_p^0, e(t_{p+1}))). Given an uncertain feature fp0f_p^0 and Q a learned self-query (defined below), Reconsideration spends one additional forward pass to refine it into fp1f_p^1, from which the token is re-sampled:

fp1=M(fc⁡(fp0,Q(fp0))),t~p+1∼head⁡(fp1),(3)f_p^1 = M\left(\operatorname{fc}\left(f_p^0,Q(f_p^0)\right)\right), \qquad\tilde{t}_{p+1} \sim\operatorname{head}(f_p^1), \tag*{(3)}

The self-query

In an EAGLE-style drafter, current feature fp0f_p^0 does not determine the next feature on its own, since different sampled tokens tp+1t_{p+1} continue into different fp+10f_{p+1}^0 ((Li et al., 2024a)), so the drafter fuses $e(t_{p+1}) to tell MM which branch to advance along. Reconsideration, however, stays at pp without advancing to re-decide the very token tp+1t_{p+1} the drafter is unsure about. We therefore give reconsideration a steering embedding: a learned self-query Q(fp0)Q(f_p^0), with Q:Rd→RdQ : \mathbb{R}^d \to\mathbb{R}^d, encoding what a second look at an uncertain state should attend to in order to refine fp0f_p^0 instead of the naive one-pass draft of next token’s feature.

The reconsideration gate

Not every uncertain position is worth another pass, and cpdc_p^d does not say which are: it reads the same where the target is decisive and where it is ambiguous (Observation 2). Recoverability has to be learned from the feature itself. Writing fp0,…,fpRf_p^0, \ldots, f_p^R for the features at each reconsideration level, up to at most RR, a small gate at level rr reads each and emits a halt probability:

λr=σ(gate⁡(fpr)),q(r)=λr∏s<r(1−λs).(4)\lambda_r = \sigma\left(\operatorname{gate}(f_p^r)\right), \qquad q(r) = \lambda_r \prod_{s<r}(1-\lambda_s). \tag*{(4)}

Every level asks the same question, whether another pass still has a chance of drafting a better token, so q(r)q(r) is the chance a position reaches level rr and stops there.

Reconsidering in parallel

A naive Reconsideration adds forward passes, and run naively in series each one stalls the draft at pp until fp1f_p^1 is computed: a draft of γ\gamma positions with NN reconsidered ones costs (γ+N) tstep(\gamma+ N)\,t_{\mathrm{step}} instead of γ tstep\gamma\,t_{\mathrm{step}} (Figure 3, top). However, the one-pass draft step and its reconsideration,

Reconsideration in parallel: sequential versus co-forwarded drafting

Figure 3. Reconsidering in parallel. Diamonds are features produced by a forward pass of the shared block MM and circles are the tokens sampled from them; gray marks ordinary one-pass drafting (feature fif_i, token xix_i) and blue marks reconsideration (deepened feature fi′f_i^{\prime}, reconsidered token xi′x_i^{\prime}). Top (sequential): running reconsideration in series spends a forward of MM that advances no new position, so within a fixed draft budget the drafter reaches fewer tokens. Bottom (co-forward): reconsideration of a position depends only on its one-pass feature, which is already available, so it is co-forwarded with the ordinary draft step in a single batched pass of MM. The one-pass chain still reaches x3x_3 and the reconsidered token x2′x_2^{\prime} is obtained for negligible additional latency.

fp+10=M(fc⁡(fp0,e(tp+1))),fp1=M(fc⁡(fp0,Q(fp0))),(5)f_{p+1}^0 = M\left(\operatorname{fc}(f_p^0,e(t_{p+1}))\right), \qquad f_p^1 = M\left(\operatorname{fc}(f_p^0,Q(f_p^0))\right), \tag*{(5)}

are disjoint, so the two can be batched into a single forward of MM: co-forwarding rather than serializing (Figure 3, bottom). At the small batch sizes of latency-oriented decoding that forward is memory-bandwidth bound, so the added rows cost a negligible ϵ\epsilon overhead (Appendix § 9.§ 1). Drafting thus becomes interleaved: the ordinary steps grow the draft tree as before, and in parallel, a reconsidered position adds its t~p+1\tilde{t}_{p+1} as a reconsidered child beside the one-pass ones. Both kinds are draft candidates, pruned and verified together under the standard speculative-decoding verification in serving time.

Training Parallel Reconsideration

The single decoder layer MM must serve two heads at once: the one-pass head (fp0)(f_p^0) that guesses and advances the draft to position p+1p + 1, and the reconsidered head (fp1)(f_p^1) that revises a premature guess. Neither can be trained merely as an intermediate step toward the other, because at serving both are read out and committed to the tree: the one-pass head guesses the token at most positions and the reconsidered head revises it at the uncertain ones. Training must therefore keep the one-pass head as accurate as an ordinary drafter while pushing the reconsidered head to win on the recoverable tokens single pass misses, and doing so is delicate because the two share the feature fp0f_p^0. Supervise only the reconsidered head and the one-pass head degrades, overloading all tokens to be reconsidered and losing the ordinary-drafter quality on the easy tokens; supervise both to match the target with equal weight and gradient descent finds the trivial solution fp1≈fp0f_p^1 \approx f_p^0, so the reconsidered token merely echoes the same one-pass token. Either failure would undercut speculative-decoding quality (Appendix § 18). We therefore hold the one-pass head to ordinary-drafter quality with an unconditional loss, and supervise each reconsidered feature only at the positions and depths where it is actually used.

One loss, every depth. Thus during training, we predict token at every depth rr and score it with ℓpr=CE⁡(head⁡(fpr),tp+1⋆)\ell_p^r = \operatorname{CE}(\operatorname{head}(f_p^r), t_{p+1}^{\star}), then weight each depth’s loss by the mass q(r)q(r) the gate routes to it, while leaving the one-pass depth unweighted. The gate has already judged the chance each depth is the one that settles the token (Eq. (4)), so q(r)q(r) is the natural weight: a depth earns gradient in proportion to how often it is the one asked to answer. Each depth’s loss also shapes where the gate halts, pulling its mass toward whichever depth reads out the token best. Weighting alone, however, leaves the gate free to abandon a depth entirely (a depth it stops routing to receives no gradient), so we add an entropy bonus that keeps every depth in use. With sg⁡[⋅]\operatorname{sg}[\cdot] the stop-gradient, we have the training objective:

L=∑p(ℓp0+∑r≥1sg⁡[q(r)]ℓpr+∑r≥0q(r)sg⁡[ℓpr]−βH(Ep[q]))(6)\mathcal{L} = \sum_p \left( \ell_p^0 + \sum_{r \ge1} \operatorname{sg}[q(r)]\ell_p^r + \sum_{r \ge0} q(r)\operatorname{sg}[\ell_p^r] - \beta H(\mathbb{E}_p[q]) \right) \tag*{(6)}

The mixed cache. A reconsidered child commits the key/value of its parent’s halt level rather than of parent’s one-pass feature, so a position deeper in the draft attends over a mixture: one-pass keys and values at the confident ancestors that were never reconsidered, reconsidered ones at the uncertain ancestors that were. To adapt to this serving phenomenon, each position jj we sample a halt depth dj∼qd_j \sim q and commit the key/value that depth produces,

κj=kv⁡(fjdj),dj∼q,(7)\kappa_j = \operatorname{kv}\left(f_j^{d_j}\right), \qquad d_j \sim q, \tag*{(7)}

so position pp attends over the prefix κ<p\kappa_{<p}, resampled afresh each step.

Appendix § 11 gives the full training recipe, including the two on-policy rollouts the two heads are trained on, the per-depth weighting, and the analysis of the balance term.

Serving a Reconsidered Tree

EAGLE-3.1 serves by building a dynamic draft tree, grown over every round’s γ\gamma draft steps. Each step forwards a batch of BwB_w frontier nodes; every forwarded node offers children sampled from its feature, and each node vv carries a score equal to its cumulative path probability from the root, s(v)=∑u∈path⁡(v)log⁡πone(u∣parent⁡(u))s(v) = \sum_{u \in\operatorname{path}(v)} \log\pi_{\mathrm{one}}(u \mid\operatorname{parent}(u)), where πone\pi_{\mathrm{one}} is the one-pass distribution of the node’s parent. After γ\gamma steps, a single global cut keeps the BB highest-scoring nodes and prunes the rest.

Parallel Reconsideration keeps that procedure and has a second one in parallel. Alongside the frontier-growing nodes the drafter holds a reconsideration pool CC of nodes that have not yet halted. A draft step now has two things to do, advance the frontier and reconsider the pool, and it does them at once: the BwB_w highest-scoring frontier nodes move the draft to the next position, and the mm highest-bidding nodes of CC are reconsidered, within the same forward batch of MM (§ 3.4), so the γ\gamma-step growth of Eq. (1) is untouched.

Which mm nodes are reconsidered follows from what reconsideration changes: on the tree, reconsidering a node vv leaves vv itself fixed and revises the distribution it offers its children (§ 3.1). A node is therefore worth reconsidering when the draft is likely to reach it and its one-pass continuation is likely to be wrong, which together are the probability that the draft breaks immediately after vv. We rank the pool by

bid⁡(v)=s(v)⏟log⁡Pr⁡[reach v]+log⁡(1−max⁡tπone(t∣v))⏟log⁡Pr⁡[v’s one-pass child is wrong]=log⁡Pr⁡[the draft breaks at v’s children],(8)\operatorname{bid}(v) = \underbrace{s(v)}_{\log\Pr[\text{reach }v]} + \underbrace{\log\left(1-\max_t \pi_{\mathrm{one}}(t \mid v)\right)}_{\log\Pr[v\text{’s one-pass child is wrong}]} = \log\Pr[\text{the draft breaks at }v\text{’s children}], \tag*{(8)}
Co-forwarding reconsideration on the draft tree

Figure 2. Co-forwarding reconsideration on the draft tree. Each token produces a feature (diamond) from which the next token is sampled (∼\sim); the one-pass draft (gray, level 0) forms the main branches, and tree width comes free because a node’s top-kk children are all sampled from its single feature. The reconsideration pass (blue, level 1) deepens a node’s feature within the same batched forward as the next draft step so it proposes a corrected token (here x2′x'_2) with no extra sequential draft step. The one-pass draft still reaches x4x_4 while the reconsidered branch grows in parallel.

and reconsider the top mm. We note that the gate and the bid are two complementary mechanisms: The gate asks whether another pass has a chance of drafting a better token, which confidence cannot answer (Observation 2); a node it declines to halt, meaning it has a chance of drafting a better child. The bid then asks, among the nodes that have that chance, which are the most valuable to reconsider this step, and confidence is what answers it: the less confident vv’s one-pass guess, the likelier the first draft guesses it wrong, and the more there is to gain by reconsidering it (Observation 1).

A reconsidered node re-draws its children from the reconsidered feature, and the token t~v∼head⁡(fvdv)\tilde{t}_v \sim\operatorname{head}(f_v^{d_v}) it now proposes enters the tree as an additional child beside the ones its one-pass feature offered. Once node vv halts at dvd_v with πdv=head⁡(fvdv)\pi_{d_v} = \operatorname{head}(f_v^{d_v}), the draft has a new unified scoring over its original vs. reconsidered children,

πmix(t∣v)=πone(t∣v)+q(dv)(πdv(t∣v)−πone(t∣v)),(9)\pi_{\mathrm{mix}}(t \mid v) = \pi_{\mathrm{one}}(t \mid v) + q(d_v)\left(\pi_{d_v}(t \mid v) - \pi_{\mathrm{one}}(t \mid v)\right), \tag*{(9)}

where the gate’s own q(dv)q(d_v) sets how far the reconsidered distribution is trusted over the one-pass one, mimicking the weighting the training objective of Eq. (6) optimizes. Reconsideration thus corrects the probability of an edge, upward where a further pass favors the token and downward where it does not; since s(⋅)s(\cdot) sums edge log-probabilities along the path, the change log⁡πmix−log⁡πone\log\pi_{\mathrm{mix}} - \log\pi_{\mathrm{one}} on an edge is added to the score of every node beneath it. A node the bid never selects has no πdv\pi_{d_v}, so Eq. (9) loses its second term and returns πone\pi_{\mathrm{one}}. The global cut then ranks reconsidered and one-pass nodes alike, and the target verifies the final tree. Appendix § 12 gives the full discussion on serving algorithm of Parallel Reconsideration.

Experiments

We describe the setup (§ 6.1), then report acceptance length and speedup against EAGLE-3.1 at a matched budget (§ 6.2), the gain under batched serving (§ 6.3), and an ablation of each design choice and of a larger drafter (§ 6.5). Two further studies appear in Appendix § 13: a comparison against spending the same compute on a wider draft (Appendix § 13.§ 2), and a measurement of which of the drafter’s failures reconsideration recovers (Appendix § 13.§ 5).

Experimental setup

We compare Parallel Reconsideration against EAGLE-3.1, the state-of-the-art feature-level speculative decoder, on three target models, Llama-3.1-8B-Instruct, Qwen3-8B, and Gemma-3-12B, and across seven benchmarks spanning math (GSM8K, MATH-500, AIME25), code (HumanEval, MBPP, LiveCodeBench), and chat (MT-Bench). Both drafters use a single decoder layer and share an identical target model, both are trained in SpecForge (Li et al., 2026) under EAGLE-3’s released recipe, and both are served through SGLang (Zheng et al., 2024) under the same budget configuration, so every number we report is measured end to end on a production serving engine. Because SGLang already removes the system bottlenecks that an under-optimized autoregressive baseline suffers from, the speedups we measure are lower than those reported in the original EAGLE-3 paper. As an external check on the baseline, our EAGLE-3.1 acceptance lengths agree with those reported independently for the same models and benchmarks by Chen et al. [7]. Our primary metrics are the acceptance length τ\tau and wall-clock speedup over autoregressive decoding on one H200. Appendix § 13.§ 1 gives the models, benchmarks, training and serving details, and the budget configuration in full.

Acceptance length and speedup

Table 1 reports acceptance length and speedup across all seven benchmarks. Parallel Reconsideration improves average acceptance length over EAGLE-3.1 at every budget, on all three models, and at both temperatures, by 5.3–10.2%, and the gain carries through to wall-clock speedup (3.5–9.2%) at a nearly identical per-round drafting cost (Appendix § 17). The gain is consistent across domains: LiveCodeBench reaches +10.1% speedup on Gemma-3, and +10.2% speedup on GSM8K for Llama3.1.

ModelMethodGSM8K τ\tauGSM8K SpeedupMATH-500 τ\tauMATH-500 SpeedupAIME25 τ\tauAIME25 SpeedupHumanEval τ\tauHumanEval SpeedupMBPP τ\tauMBPP SpeedupLCB τ\tauLCB SpeedupMT-Bench τ\tauMT-Bench SpeedupAvg. τ\tauAvg. Speedup
Llama-3.1-8BEAGLE-3.1 (24)4.411.90×4.702.14×5.412.29×5.152.25×4.972.24×4.311.78×4.201.69×4.742.04×
PR (24)4.732.03×4.772.16×5.362.28×5.452.38×5.282.37×4.781.95×4.651.85×5.002.15×
Δ\Delta+7.3%+6.8%+1.5%+0.9%−0.9%−0.4%+5.8%+5.8%+6.2%+5.8%+10.9%+9.6%+10.7%+9.5%+5.5%+5.4%
EAGLE-3.1 (48)4.842.05×4.992.13×5.282.27×5.622.49×5.322.33×4.701.95×4.602.03×5.052.18×
PR (48)5.402.26×5.312.23×5.532.37×5.872.56×5.572.41×5.112.11×4.772.05×5.372.28×
Δ\Delta+11.6%+10.2%+6.4%+4.7%+4.7%+4.4%+4.4%+2.8%+4.7%+3.4%+8.7%+8.2%+3.7%+1.0%+6.3%+4.6%
Qwen3-8BEAGLE-3.1 (24)4.391.94×3.962.04×3.811.86×4.462.11×4.342.11×3.661.78×3.361.57×4.001.92×
PR (24)4.832.11×4.262.16×4.202.00×4.792.26×4.842.32×3.991.91×3.591.66×4.362.06×
Δ\Delta+10.0%+8.8%+7.6%+5.9%+10.2%+7.5%+7.4%+7.1%+11.5%+10.0%+9.0%+7.3%+6.8%+5.7%+9.0%+7.3%
EAGLE-3.1 (48)4.842.18×4.302.24×4.231.99×4.852.41×4.752.22×4.001.86×3.661.71×4.382.09×
PR (48)5.132.29×4.502.32×4.602.10×5.032.50×5.072.33×4.181.93×3.821.77×4.622.18×
Δ\Delta+6.0%+5.0%+4.7%+3.6%+8.7%+5.5%+3.7%+3.7%+6.7%+5.0%+4.5%+3.8%+4.4%+3.5%+5.5%+4.3%
Gemma-3-12BEAGLE-3.1 (24)4.923.08×4.212.54×4.292.68×6.373.40×5.293.08×5.282.78×3.251.79×4.802.76×
PR (24)5.283.16×4.652.78×4.602.86×7.073.78×5.653.30×5.903.05×3.751.98×5.272.99×
Δ\Delta+7.3%+2.6%+10.5%+9.4%+7.2%+6.7%+11.0%+11.2%+6.8%+7.1%+11.7%+9.7%+15.4%+10.6%+9.8%+8.3%
EAGLE-3.1 (48)5.433.23×4.532.60×4.672.93×6.733.68×5.683.20×5.813.27×3.581.94×5.202.98×
PR (48)5.963.53×4.892.80×4.933.07×7.103.84×6.083.33×6.463.60×3.942.09×5.623.18×
Δ\Delta+9.8%+9.3%+7.9%+7.7%+5.6%+4.8%+5.5%+4.3%+7.0%+4.1%+11.2%+10.1%+10.1%+7.7%+8.1%+6.7%
Llama-3.1-8BEAGLE-3.1 (24)3.471.45×4.132.02×4.492.06×4.612.16×4.401.99×2.821.21×2.071.17×3.711.72×
PR (24)4.011.61×4.452.12×4.512.08×4.802.18×4.552.03×3.151.32×2.291.24×3.971.80×
Δ\Delta+15.6%+10.6%+7.7%+5.0%+0.4%+1.0%+4.1%+0.9%+3.4%+2.0%+11.7%+9.1%+10.6%+6.0%+7.3%+4.7%
EAGLE-3.1 (48)3.991.67×4.442.06×4.512.13×5.132.31×4.792.16×3.021.29×2.721.45×4.091.87×
PR (48)4.561.84×4.792.19×4.582.16×5.422.36×5.082.23×3.491.47×2.831.49×4.391.96×
Δ\Delta+14.3%+10.2%+7.9%+6.3%+1.6%+1.4%+5.7%+2.2%+6.1%+3.2%+15.6%+14.0%+4.0%+2.8%+7.3%+4.8%
Qwen3-8BEAGLE-3.1 (24)4.471.72×3.621.74×3.251.52×4.321.94×4.081.78×3.381.52×3.121.47×3.751.73×
PR (24)4.762.27×3.901.84×3.431.57×4.501.96×4.301.84×3.531.57×3.251.50×3.951.79×
Δ\Delta+6.5%+4.6%+7.7%+5.7%+5.5%+3.3%+4.2%+1.0%+5.4%+3.4%+4.4%+3.3%+4.2%+2.0%+5.3%+3.5%
EAGLE-3.1 (48)4.842.20×3.941.94×3.591.70×4.732.26×4.482.15×3.671.70×3.371.57×4.091.93×
PR (48)5.212.36×4.192.00×3.701.72×4.952.40×4.692.24×3.921.78×3.631.65×4.332.02×
Δ\Delta+7.6%+7.3%+6.3%+3.1%+3.1%+1.2%+4.9%+6.2%+4.7%+4.7%+6.8%+4.7%+7.7%+5.1%+5.9%+4.7%
Gemma-3-12BEAGLE-3.1 (24)5.002.79×4.122.41×3.792.00×6.413.33×5.443.17×5.173.17×3.121.48×4.722.62×
PR (24)5.382.90×4.382.54×4.292.24×7.513.86×5.833.32×5.593.41×3.281.54×5.182.83×
Δ\Delta+7.6%+3.9%+6.3%+5.4%+13.2%+12.0%+17.2%+15.9%+7.2%+4.7%+8.1%+7.6%+5.1%+4.1%+9.7%+8.0%
EAGLE-3.1 (48)5.442.81×4.472.36×4.542.14×6.703.89×5.832.98×5.543.13×3.401.64×5.102.71×
PR (48)6.223.19×4.822.53×4.862.33×7.254.20×6.663.39×5.993.37×3.551.70×5.622.96×
Δ\Delta+14.3%+13.5%+7.8%+7.2%+11.7%+8.9%+8.2%+8.0%+14.2%+13.8%+8.1%+7.7%+4.4%+3.7%+10.2%+9.2%

Table 1. Acceptance length τ\tau and decoding speedup over autoregressive decoding. Parenthesized values give the total draft budget BB (number of draft nodes verified per round), held equal for both methods along with the offer width, grow width, and number of draft steps. Δ\Delta rows give the relative gain of PR over EAGLE-3.1 at the same budget.

Serving at larger batch sizes

Speculative decoding is hardest to justify at large batch sizes, where the target’s forward is already compute-bound and every drafted node competes for the same verification capacity. We therefore benchmark Parallel Reconsideration at batch sizes n∈{1,2,4,8,16}n \in\{1, 2, 4, 8, 16\} against autoregressive decoding at the same nn (Table 2).

Batch size nn
ModelBenchmarkMethod124816
LlamaGSM8KEAGLE-3.12.05×1.68×1.62×1.26×1.12×
LlamaGSM8KPR2.26×1.91×1.91×1.46×1.22×
LlamaMT-BenchEAGLE-3.12.03×1.71×1.71×1.50×1.41×
LlamaMT-BenchPR2.05×1.79×1.77×1.56×1.51×
LlamaLCBEAGLE-3.11.95×1.61×1.60×1.40×1.31×
LlamaLCBPR2.11×1.84×1.78×1.33×1.29×
QwenGSM8KEAGLE-3.12.18×2.16×2.01×1.68×1.39×
QwenGSM8KPR2.29×2.26×2.25×1.94×1.64×
QwenMT-BenchEAGLE-3.11.71×1.72×1.71×1.35×1.11×
QwenMT-BenchPR1.77×1.77×1.69×1.41×1.24×
QwenLCBEAGLE-3.11.86×1.81×1.80×1.60×1.31×
QwenLCBPR1.93×1.87×1.95×1.66×1.43×
GemmaGSM8KEAGLE-3.13.23×2.79×2.54×2.50×1.71×
GemmaGSM8KPR3.53×2.99×2.72×2.70×1.94×
GemmaMT-BenchEAGLE-3.11.94×1.92×1.83×1.71×1.50×
GemmaMT-BenchPR2.09×2.02×1.91×1.91×1.40×
GemmaLCBEAGLE-3.13.27×2.79×2.72×2.55×2.04×
GemmaLCBPR3.60×3.02×2.97×2.85×2.14×

Table 2. Speedup as requests are batched. Wall-clock speedup of EAGLE-3.1 and Parallel Reconsideration (PR) over autoregressive decoding at a matched batch size nn, measured on the same serving engine (B=48B = 48, temperature 0).

Gemma-3-12B holds 2.85× on LiveCodeBench at n=8n = 8 and 2.14× at n=16n = 16, and Qwen3-8B is flat through n=4n = 4 (2.25× on GSM8K, 1.95× on LiveCodeBench). Reconsideration buys acceptance out of the draft budget rather than the verification budget (§ 5), so the target still verifies the same BB nodes per round at every nn, and the co-forwarded rows are absorbed by the drafter’s own

Each component, and the alternative of a bigger drafter

batch rather than the target’s.

Each component, and the alternative of a bigger drafter

Table 3 removes one design choice at a time and retrains. It also answers a question the cost model raises: if reconsideration is worth a second pass of MM, why not simply give the drafter a second layer and spend that capacity on every position?

DrafterGSM8K τ\tauGSM8K SpeedupLiveCodeBench τ\tauLiveCodeBench SpeedupMBPP τ\tauMBPP SpeedupAvg. τ\tauAvg. Speedup
EAGLE-3.14.842.05×4.701.95×5.322.33×4.952.11×
Parallel Reconsideration5.402.26×5.112.11×5.572.41×5.362.26×
no balance term (β=0\beta= 0)5.172.14×5.032.04×5.582.41×5.262.20×
Δ\Delta−4.3%−5.3%−1.6%−3.3%+0.2%+0.0%−1.9%−2.8%
no self-query (Q→e(tp)Q \to e(t_p))5.032.10×4.922.02×5.472.35×5.142.16×
Δ\Delta−6.9%−7.1%−3.7%−4.3%−1.8%−2.5%−4.1%−4.6%
EAGLE-3.1, two decoder layers4.871.90×4.911.85×5.372.14×5.051.96×
Δ\Delta−9.8%−15.9%−3.9%−12.3%−3.6%−11.2%−5.8%−13.1%

Table 3. Removing each design choice, and spending the compute on depth instead. Llama-3.1-8B at B=48B = 48, temperature 0, reporting τ\tau / speedup. The first two rows remove one component of Parallel Reconsideration and retrain; the third is EAGLE-3.1 with a second decoder layer and no reconsideration, spending its additional capacity on every position rather than on the uncertain ones. Δ\Delta is relative to Parallel Reconsideration.

The self-query. Replacing Q(fp0)Q(f_p^0) with e(tp)e(t_p), the embedding of the token that produced fp0f_p^0, costs 4.1% of average acceptance and 4.6% of speedup. Steering reconsideration with a committed token rather than the feature’s own query recovers less, which is what § 3.2 predicts: a token embedding directs MM to advance along that branch, while reconsideration needs to stay in place.

The balance term. Setting β=0\beta= 0 costs 1.9% of average acceptance. The term keeps a depth from collapse before the head’s term can train it, so removing it degrades the drafter without disabling it.

A bigger drafter instead. Giving EAGLE-3.1 a second decoder layer raises its acceptance, by 2.0% on average over the one-layer EAGLE baseline, and still loses 7.0% of its speedup, because the extra layer raises tstept_{\mathrm{step}} on every draft step. Against Parallel Reconsideration the gap is 13.1% of speedup at 5.8% lower acceptance.

Related Work

Speculative decoding accelerates autoregressive generation by drafting several tokens cheaply and verifying them in one target forward pass, accepting the longest prefix the target agrees with so the output distribution is preserved exactly [22] (Xia et al. 2023); since the acceptance rule guarantees that distribution, subsequent work buys speed either by raising the acceptance length or by lowering the drafting cost. Self-drafting methods derive the draft from the target’s own feature, as in the EAGLE family, which autoregresses on target features with a single lightweight decoder layer [15] (Li et al. 2024a; Li et al. 2025), or in the prediction heads attached to the target’s last hidden state [2, 6]. Training-free drafters remove the auxiliary model altogether and verify nn-gram continuations generated in parallel (Fu et al., 2024; Le et al., 2026), while parallel drafters attack the cost axis directly by emitting an entire block per forward pass [7, 10] (An et al. 2025). A separate line changes what the drafter optimizes, recasting draft training as variational inference over draft paths so that acceptance, rather than token likelihood, is the objective (Zou et al., 2026). Parallel Reconsideration is orthogonal to most of these: it adds a second guess at the uncertain positions of an existing one-pass draft, and applies to any drafter that exposes a reusable draft feature. Finally, spending extra passes to sharpen a prediction is itself familiar, from looped and universal transformers to adaptive-compute methods that learn how many steps each token needs [5, 13, 18] (Schuster et al. 2022; Zhu et al. 2025); we loop not to make an autoregressive model reason better, but to revise the failure of autoregressive drafting, and we run the extra pass in parallel so it costs no additional step. Appendix § 14 provides a more detailed discussion of related work.

Conclusion

We presented Parallel Reconsideration, which grants the drafter a second chance to guess on the tokens it is unsure about: where it is already decisive the guess is committed as usual, and where it hesitates the drafter is given a few extra attempts to revise it. The attempts are batched into the same forward pass as the ordinary draft, so they add no sequential draft step and cost only negligible parallel overhead. This brings draft-time scaling to feature-level drafters, letting the drafter vary how much it computes per position rather than how well it computes in one step. Across Llama-3.1-8B-Instruct, Qwen3-8B, and Gemma-3-12B, it raises average acceptance length over EAGLE-3.1 by 5.3–10.2% and wall-clock speedup by 3.5–9.2% on math, code, and chat benchmarks at a matched draft budget.

Details on Parallel Reconsideration

Reconsidering in parallel

Reconsideration adds forward passes, and naively those passes are sequential. Running the second pass in series stalls the draft at pp until fp1f_p^1 is computed, so a draft of γ\gamma positions with NN reconsidered ones takes (γ+N) tstep(\gamma+ N)\,t_{\mathrm{step}} instead of γ tstep\gamma\,t_{\mathrm{step}} Eq. (1), Figure 3, top). Under a fixed drafting-time budget the drafter reaches NN fewer positions, shortening the original total number of guesses.

The stall, however, is an artifact of ordering, not of dependency. Writing the draft step and its reconsideration side by side Eq. (5), both take the same feature fp0f_p^0 as input, and their outputs go to separate places: fp+10f_{p+1}^0 advances the ongoing draft to p+1p+1, whereas fp1f_p^1 only re-samples t~p+1\tilde{t}_{p+1} and later seeds the reconsidered branch’s own continuation, which the ongoing draft never reads. Concretely, the reconsideration row keeps position index pp and attends to the committed prefix κ<p\kappa_{<p} and its own key, not to kv⁡(fp0)\operatorname{kv}(f_p^0); its key/value replaces slot pp on the reconsidered branch only. The reconsideration

Details on Training Parallel Reconsideration

is therefore independent of the draft step that advances past it, so the two can be batched together into a single forward of MM — co-forwarding rather than serializing (Figure 3, bottom).

At the small batch sizes of latency-oriented decoding a forward of MM is memory-bandwidth bound: its latency is set by streaming the drafter’s weights from HBM, and the extra arithmetic from enlarging the batch is hidden beneath that load. A co-forwarded pass therefore costs tstep+ϵt_{\mathrm{step}}+\epsilon for a negligible ϵ\epsilon, so a draft of γ\gamma positions costs γ(tstep+ϵ)≈γtstep\gamma(t_{\mathrm{step}}+\epsilon)\approx\gamma t_{\mathrm{step}}, against (γ+N)tstep(\gamma+N)t_{\mathrm{step}} for the serial version. The drafter reaches exactly as many positions as it would without reconsideration; we detail the batched execution in § 5. Measured end to end, carrying the reconsideration rows costs 1.015–1.022× the per-cycle time of the one-pass drafter, i.e., roughly 2% of cycle time (Appendix § 17 for details).

The result is an interleaved drafting strategy: the ordinary steps form the main branches of one-pass guesses, and each gated position might add a reconsidered candidate t~p+1\tilde{t}_{p+1} (Eq. (3)) beside its one-pass sibling. Verification scores both, so at that position the reconsidered token takes nothing from its one-pass sibling: it is an extra candidate rather than a substitute, and it can supply an accepted token only where the one-pass guess would have been rejected. The two share the verification budget, however, and § 5 is where they are ranked against each other for it.

Details on Training Parallel Reconsideration

This appendix gives a more detailed discussion on the training recipe of Parallel Reconsideration. Section 4 states the objective of Eq. (6) and the mixed cache of Eq. (7); we expand here on the two failure modes the recipe is designed against (Appendix § 11.§ 1), on what a reconsidered feature is actually supervised to do (Appendix § 11.§ 2), on the rollouts the losses are computed over, and on why the balance term regularizes the gate’s marginal usage rather than its per-position halt distribution (Appendix § 11.§ 3).

Why the one-pass head cannot be scaffolding

Loop transformers, which reuse a layer across depths much as we do, read out only at a single depth and are free to supervise the intermediate depths purely as scaffolding for that one output (Appendix § 14). Parallel Reconsideration cannot do this, because at serving both depths are read out and committed to the tree: the one-pass head guesses the token at most positions and the reconsidered head revises it at the uncertain ones. The difficulty is that the two share the feature fp0f_p^0. On one hand, fp0f_p^0 is the feature every later draft step conditions on, so it must retain enough of the target distribution to keep proposing acceptable tokens down the whole draft; on the other hand, reconsideration deepens fp0f_p^0 into fp1f_p^1, and for the reconsidered token to differ from the one-pass token the objective must let fp1f_p^1 diverge from fp0f_p^0 rather than reproduce it.

This is what makes the two failure modes of § 4 opposite rather than two versions of the same error. Supervising only the reconsidered head leaves the one-pass head untrained, so the draft it proposes degrades and shortens, and since a reconsidered candidate is only reached once the tokens before it are accepted, the reconsidered branches are left with shorter and shorter prefixes to extend. Supervising both to match the target with equal weight instead removes any incentive for the two features to differ, and gradient descent finds the trivial optimum fp1≈fp0f_p^1 \approx f_p^0. Appendix § 18 discusses both in more detail.

The recipe therefore holds the one-pass draft to ordinary EAGLE quality with an unconditional loss,

Lpone=CE⁡(head⁡(fp0),tp+1⋆),(10)\mathcal{L}_p^{\mathrm{one}} = \operatorname{CE}\left(\operatorname{head}(f_p^0), t_{p+1}^{\star}\right), \tag*{(10)}

against the teacher token tp+1⋆t_{p+1}^{\star}, which appears as the unweighted first term of Eq. (6). Supervising it at every position frees reconsideration to specialize on the positions the one-pass guess hesitates over, without eroding the ones it already resolves.

What a reconsidered feature is supervised to do

Each reconsideration pass reapplies MM to the previous depth’s feature through the self-query, fpr=M(fc⁡(fpr−1,Q(fpr−1)))f_p^r = M(\operatorname{fc}(f_p^{r-1}, Q(f_p^{r-1}))) (Eq. (3)), so depth rr is a strictly deeper reading of the same position pp. Such a feature does two jobs at serving, and the two are supervised differently. Every depth is read out to propose a candidate token, so we supervise the head at all depths. Only one depth, the halt depth dpd_p where the gate stops, has its key/value written to the cache and drafts the next position, so the continuation is supervised at that depth alone. Which depth halts is therefore central: it decides what the deeper positions read and what the recurrence is trained on. We sample it, dp∼qd_p \sim q (Eq. (4)), by walking the depths and halting at the first pass the gate declines, the same way in training and at serving.

The mixed cache, concretely. Section 4 introduces the mixed cache of Eq. (7). Training naively on a fixed one-pass cache, reading fj0f_j^0 at every ancestor, would leave a gap: that cache never appears at serving, where each ancestor commits at its own halt depth. For a position 5 whose ancestors halt at depths d1:4=(0,1,0,2)d_{1:4}=(0,1,0,2),

one-pass cache:(kv⁡(f10),kv⁡(f20),kv⁡(f30),kv⁡(f40)),mixed cache κ<5:(kv⁡(f10),kv⁡(f21),kv⁡(f30),kv⁡(f42)),(11)\begin{aligned} \text{one-pass cache:}\quad&(\operatorname{kv}(f_1^0),\operatorname{kv}(f_2^0),\operatorname{kv}(f_3^0),\operatorname{kv}(f_4^0)),\\ \text{mixed cache }\kappa_{<5}:\quad&(\operatorname{kv}(f_1^0),\operatorname{kv}(f_2^1),\operatorname{kv}(f_3^0),\operatorname{kv}(f_4^2)), \tag*{(11)} \end{aligned}

the two agree at the confident ancestors 1, 3 (halt depth 0) but differ at the uncertain 2, 4, where the mixed cache carries the reconsidered key/value the drafter will actually commit. Sampling dj∼qd_j\sim q deepens the ancestors of a single rollout faithfully under the halting distribution, while a served branch under a budget deepens few; both are prefixes the same per-position commit rule produces, so a reconsidered branch never reads one the drafter was not trained for.

On the halt depths seen at training and at serving. The depths qq favors at training are not the depths the drafter meets at serving. Training samples dp∼qd_p\sim q at every position, so any position may deepen; at serving the reconsideration budget mm admits only a few eligible nodes per step, so most positions are committed from depth 0 however willing qq is to go deeper. The weighting does not turn this into a train–serve gap, because it reallocates supervision rather than withholding it. Every depth is read out and scored at every position, and only the size of its gradient varies with q(r)q(r), so no depth is trained solely on the positions qq favors and each stays accurate wherever it is eventually read. A position served from depth 0 under a tight budget is one the one-pass term trains unconditionally; a position the budget does deepen is one the gated terms have trained at that depth. The same holds for the cache those losses are computed over, by the argument above.

Two on-policy rollouts. EAGLE-3 (Li et al., 2025) trains by unrolling an on-policy draft chain from every position of the training sequence: starting from a position’s target feature, the drafter speculates several steps forward on its own predictions, and each drafted position is scored against the teacher token. We unroll two such chains from every starting position. The one-pass chain is EAGLE-3’s own, advancing from each one-pass feature to the next, fp0→fp+10f_p^0\to f_{p+1}^0. The reconsideration chain branches off at the starting position, then runs on its own features: at each of its positions it deepens the current feature to the halt feature fpdpf_p^{d_p} and drafts the next position from it,

fp+10=M(fc⁡(fpdp,e(tp+1)∣κ<p+1)),(12)f_{p+1}^0=M\bigl(\operatorname{fc}(f_p^{d_p},e(t_{p+1})\mid\kappa_{<p+1})\bigr), \tag*{(12)}

reading the mixed cache assembled above and reconsidering afresh from there.

Both chains are scored, position by position, by the head losses of Eq. (6), so each learns its own predictions on its own rollout. They share parameters through MM but couple in the backward pass only at the branch point fp0f_p^0, which feeds both the one-pass step and the reconsideration chain. Its gradient is therefore the sum of two signals: the one-pass head loss at fp0f_p^0 and the losses of the entire reconsideration rollout it seeds. Thus fp0f_p^0 is optimized jointly to minimize the one-pass token error and to encode a state from which reconsideration can recover the tokens that error misses. The continuation of Eq. (12) needs no term of its own: keeping each chain accurate at every position already trains its features to draft the next one.

Which entropy the balance term rewards

The balance term of Eq. (6) is not the obvious entropy. The halt distribution induces two: the per-position entropy averaged over positions, Ep[H(q)]\mathbb{E}_p[H(q)], and the entropy of the marginal usage qˉ=Ep[q]\bar{q}=\mathbb{E}_p[q]. We reward the marginal H(qˉ)H(\bar{q}), and the decomposition of Proposition 1 (Appendix § 19) makes the choice precise.

The point is what each term does not do. Rewarding per-position entropy Ep[H(q)]\mathbb{E}_{p}[H(q)] would push every position to split its mass across depths and actively destroy the specialization we are trying to build. Rewarding the marginal H(qˉ)H(\bar{q}) leaves that specialization alone: it charges only for letting a depth fall out of use, and is neutral as to whether the surviving mass is realized by confident per-position decisions or by undecided ones. The specialization itself comes from the head term of Eq. (6), which trains each depth on the tokens routed to it; the balance term’s job is only to keep a depth from going dark before that term can train it. It is also symmetric in the depths: a gate that always halts at depth 00 and one that always runs to depth RR both give H(qˉ)=0H(\bar{q}) = 0, so it resists collapse onto any single depth rather than pulling toward deeper ones (Zhu et al., 2025).

No price on depth. No term of Eq. (6) charges for reconsidering. With the losses detached, ∑rq(r)sg⁡[ℓpr]\sum_r q(r)\operatorname{sg}[\ell_p^r] is minimized by routing each position to its lowest-loss depth, and nothing opposes going deeper. The objective therefore asks the gate only whether a further pass improves the token, which is what § 3.3 wants it to judge; how many passes the drafter can afford is set where the resource binds, by the depth cap RR and the mm reconsideration rows a forward carries (§ 5).

Details on Serving a Reconsidered Tree

This appendix gives a more detailed discussion on the serving algorithm of Parallel Reconsideration, summarized in Algorithm 1. We expand on how the reconsideration pool is maintained across draft steps (Appendix § 12.§ 1), on what the repricing rule of Eq. (9) does to a node’s existing children as well as its new ones (Appendix § 12.§ 2), on why the round degenerates exactly to EAGLE-3.1 when no node is eligible (Appendix § 12.§ 3), and on why verification, and hence losslessness, is unaffected (Appendix § 12.§ 4).

Maintaining the reconsideration pool

The gate is consulted once per level, always on the feature the last forward produced, so no evaluation of it costs a pass of its own. At level 00 this is what admits a node to the pool: the step that forwards uu reads λ0=σ(gate⁡(fu0))\lambda_0 = \sigma(\operatorname{gate}(f_u^0)) off the one-pass feature it has just produced, and uu enters the pool with probability 1−λ01-\lambda_0, sampled as in training (§ 3.3). Eligibility is therefore settled before any reconsideration pass is spent, which is what lets the pool be filtered for free.

At level r≥1r \ge1 the same test decides how much further a node already in the pool goes. Each step the bid selects the top mm of the pool (Eq. (8)); those nodes are carried one level deeper in the same batched forward, and the gate then reads λr\lambda_r off the feature furf_u^r that forward produced. A node it halts leaves the pool with its halt depth dud_u fixed and reprices its edges; a node it declines to halt has rur_u incremented and remains for a later step, so a single position may be reconsidered across several steps up to the cap RR, at which it halts regardless. Two consequences follow. First, the bid is re-evaluated every step over a changing pool, so a node that loses the competition early may still be selected once shallower competitors have halted. Second, the mm rows a step carries are not tied to any one position: they are spent on whichever mm pooled nodes currently bid highest, which is what makes mm a per-step budget rather than a per-position one.

Algorithm 1: One draft round of Parallel Reconsideration. A forwarded node joins the reconsideration pool CC only if the gate declines to halt it at depth 0. The bid then selects the mm eligible nodes whose reconsideration is most worth the pass, and the gate sets how deep each goes. With CC held empty the procedure is the EAGLE-3.1 loop.

input: budget BB; grow width BwB_w; offer width kk; reconsideration budget mm; reconsideration cap RR; draft steps γ\gamma

output: draft tree T\mathcal{T}, ∣T∣≤B\lvert\mathcal{T}\rvert\le B

T←{root}\mathcal{T} \leftarrow\{\mathrm{root}\}; U←{root}\mathcal{U} \leftarrow\{\mathrm{root}\}; C←∅C \leftarrow\varnothing // tree; un-forwarded nodes; reconsideration pool

for step =1,…,γ=1,\ldots,\gamma do

A←\mathcal{A} \leftarrow top-BwB_w of U\mathcal{U} by s(⋅)s(\cdot); U←U∖A\mathcal{U} \leftarrow\mathcal{U}\setminus\mathcal{A} \quad S←\mathcal{S} \leftarrow top-mm of CC by bid⁡(⋅)\operatorname{bid}(\cdot) // (8); the mm most promising of the pool

{fu0}u∈A∪{fvrv+1}v∈S←M(A∪S)\{f_u^0\}_{u\in\mathcal{A}}\cup\{f_v^{r_v+1}\}_{v\in\mathcal{S}}\leftarrow M(\mathcal{A}\cup\mathcal{S}) // one level deeper per selected node

foreach v∈Sv\in\mathcal{S} do

halt with probability λrv=σ(gate⁡(fvrv))\lambda_{r_v}=\sigma(\operatorname{gate}(f_v^{r_v})), Eq. (4) // sampled, as in training

if halted or rv=Rr_v=R then

dv←rvd_v\leftarrow r_v; remove vv from CC

else

rv←rv+1r_v\leftarrow r_v+1; keep vv in CC for a later step

/* two chains: a reconsidered node adds to what it already offered */

foreach u∈Au\in\mathcal{A} do

offer the top kk tokens of πone(⋅∣u)\pi_{\mathrm{one}}(\cdot\mid u) as children of uu, scoring each by

s(c)=s(u)+log⁡πone(c∣u)s(c)=s(u)+\log\pi_{\mathrm{one}}(c\mid u)

foreach vv halted this step do

/* vv now prices its edges by πmix\pi_{\mathrm{mix}}: reprice the children it offered when it was advanced, then admit the ones the halt distribution adds */

foreach existing child cc of vv do

δc←log⁡πmix(c∣v)−log⁡πone(c∣v)\delta_c\leftarrow\log\pi_{\mathrm{mix}}(c\mid v)-\log\pi_{\mathrm{one}}(c\mid v), Eq. (9)

add δc\delta_c to s(c)s(c) and to every node beneath c//s(⋅)c // s(\cdot) sums along the path

foreach tt with q(dv)(πdv(t∣v)−πone(t∣v))>0q(d_v)(\pi_{d_v}(t\mid v)-\pi_{\mathrm{one}}(t\mid v))>0 and tt not already a child of vv do

offer tt as a child cc of vv with s(c)=s(v)+log⁡πmix(t∣v)s(c)=s(v)+\log\pi_{\mathrm{mix}}(t\mid v)

U←U∪{children offered this step}\mathcal{U}\leftarrow\mathcal{U}\cup\{\text{children offered this step}\}; T←T∪{children offered this step}\mathcal{T}\leftarrow\mathcal{T}\cup\{\text{children offered this step}\}

foreach u∈Au\in\mathcal{A} do

ru←1r_u\leftarrow1; add uu to CC with probability 1−λ0=1−σ(gate⁡(fu0))1-\lambda_0=1-\sigma(\operatorname{gate}(f_u^0)) // gate eligibility at depth 0, free from fu0f_u^0

return top-BB of T\mathcal{T} by s(⋅)s(\cdot) // one global cut, one-pass and reconsidered alike

What the repricing rule covers

Once vv halts at dvd_v, Eq. (9) prices every edge out of it, not only the token the reconsidered feature newly proposes. The children its one-pass feature offered when it was advanced are repriced by the same rule, and because the path score s(⋅)s(\cdot) is a sum of edge log-probabilities, that correction carries into the subtree those children have already grown: in the algorithm this is the increment δc=log⁡πmix(c∣v)−log⁡πone(c∣v)\delta_c = \log\pi_{\mathrm{mix}}(c \mid v) - \log\pi_{\mathrm{one}}(c \mid v) added to s(c)s(c) and to every node beneath cc. A node the bid never selects has no πdv\pi_{\mathrm{dv}}, so Eq. (9) loses its second term and returns πone\pi_{\mathrm{one}}, leaving its edges exactly as the one-pass draft priced them.

This is where reconsideration does more than widen the tree. On a token vv has not proposed, a positive correction admits a new candidate. On one it already holds, the correction moves that child instead, up where the deeper pass agrees with it and down where it does not, against its siblings and against the rest of the tree at the cut.

Degenerating to the one-pass baseline

The round reduces to EAGLE-3.1’s exactly when the pool is empty. With C=∅C = \varnothing no node is ever selected by the bid, no forward carries reconsideration rows, every node keeps πone\pi_{\mathrm{one}} by the degenerate case of Eq. (9), and the global cut ranks a tree grown entirely by one-pass steps. This is the sense in which reconsideration is additive at serving: the gate controls whether the second mechanism runs at all, and where it declines, the drafter behaves as the baseline it was built on.

Verification is unchanged, and the output is lossless

Reconsideration changes which tokens enter the draft tree, but not how the tree is verified. The verifier receives only the tree’s tokens and its topology. It never receives a draft distribution: neither πone\pi_{\mathrm{one}}, nor πdv\pi_{\mathrm{dv}}, nor πmix\pi_{\mathrm{mix}}, nor any path score s(⋅)s(\cdot). These quantities, together with the sampled halting decisions, are used only on the drafter side, to decide what to propose and what survives the global cut. We therefore serve Parallel Reconsideration through SGLang’s tree verification for EAGLE, unmodified, the same path used for the EAGLE-3.1 and HASS baselines. At temperature 0 it accepts the longest path that matches the target’s greedy tokens. At temperature T>0T > 0 it applies target-only multi-candidate rejection sampling, which treats every drafted child as a deterministic proposal.

Losslessness. Fix a verified node with prefix xx, and let π=πtgt(⋅∣x)\pi= \pi_{\mathrm{tgt}}(\cdot\mid x) be the target’s distribution at temperature TT. Let c1,…,ckc_1, \ldots, c_k be the node’s children in the order the verifier visits them. The verifier sets r0=πr_0 = \pi. For i=1,…,ki = 1, \ldots, k, it accepts cic_i with probability ri−1(ci)r_{i-1}(c_i). On rejection it sets ri=ri−1(⋅∣⋅≠ci)r_i = r_{i-1}(\cdot\mid\cdot\ne c_i), that is, ri−1r_{i-1} with cic_i removed and the rest renormalized. If every child is rejected, it samples the emitted token from rkr_k. Sampling from ri−1r_{i-1} is the same as emitting cic_i with probability ri−1(ci)r_{i-1}(c_i) and otherwise sampling from ri−1r_{i-1} conditioned on ≠ci\ne c_i, which is rir_i. By induction on ii, the emitted token is therefore distributed exactly as r0=πr_0 = \pi, for every choice of c1,…,ckc_1, \ldots, c_k, duplicates included (a repeated child has zero residual mass and is never accepted twice). The only requirement is that the children are fixed before the verifier draws its coins, which holds because drafting finishes before verification begins. Applying the argument node by node along the accepted path shows that each emitted token is an exact sample from the target given its prefix. The output sequence is thus distributed autoregressive sampling from the target.

Nothing in this argument refers to how the candidates were produced. Whether a child was drawn from πone\pi_{\mathrm{one}} or from head(fjdv)(f_j^{\mathrm{dv}}), whether its edge was repriced by Eq. (9), and whether the gate halted stochastically, all of this changes only the candidate set, and hence how many tokens are accepted per round, never which distribution they are drawn from. The same holds for EAGLE-3.1, whose tree is also chosen by drafter-side scores that the verifier never sees. Reconsideration therefore adds nothing to verification that a standard speculative decoder does not already rely on, and the temperature-1 results in Table 1 are lossless in the same sense as the baseline’s.

Details on Experiments

This appendix gives the full experimental detail behind § 6. We state the setup in full (Appendix § 13.§ 1), then report four further studies: whether spending the same compute on a wider draft reproduces the gain (Appendix § 13.§ 2), whether the gain holds across tree configurations (Appendix § 13.§ 3) and on top of a different base drafter (Appendix § 13.§ 4), and which of the drafter’s failures reconsideration actually recovers, returning to the diagnosis of Section 2.2 (Appendix § 13.§ 5).

Experimental setup

Models and baseline. We evaluate on three target models, Llama-3.1-8B-Instruct, Qwen3-8B, and Gemma-3-12B, and compare Parallel Reconsideration against EAGLE-3.1, the state-of-the-art feature-level speculative decoder. We use EAGLE-3.1 to denote the current form of the EAGLE-3 drafter (Li et al., 2025) with the normalization fixes of Eldenk et al. [15]. Both drafters use a single decoder layer and share an identical target model, and both are trained on UltraChat (Ding et al., [14]) under EAGLE-3’s released recipe, unchanged (Appendix § 15). Parallel Reconsideration takes at most two reconsideration steps per position (depth cap R=2R = 2) on every model.

Implementation. We train both methods in SpecForge (Li et al., 2026), publicly available training recipe of EAGLE-3.1, and add reconsideration as a second head and gate within the same trainer. Both are then served through SGLang (Zheng et al., 2024), and every acceptance length and speedup we report is measured on SGLang serving engine, faithfully reporting realistic scenarios. As an external check on the baseline, our EAGLE-3.1 acceptance lengths agree with those reported independently for the same models and benchmarks by DFlash (Chen et al., [7]).

Benchmarks and metrics. We evaluate on seven benchmarks in three categories: math (GSM8K (Cobbe et al., [12]), MATH-500 (Lightman et al., [27]), AIME25 (Mathematical Association of America, [29])), code (HumanEval (Chen et al., [8]), MBPP (Austin et al., [4]), LiveCodeBench (Jain et al., [20])), and chat (MT-Bench (Zheng et al., 2023)). Our primary metric is the acceptance length τ\tau, the mean number of tokens the target accepts per verification round, which is the drafter’s contribution to end-to-end speedup at fixed target and verifier cost (Eq. (2)); we also report the wall-clock speedup over autoregressive decoding, measured on one H200. We report both greedy decoding (temperature 0) and sampling at temperature 1, with max generation length of 2048 tokens. Benchmark sizes and sources are in Appendix § 15.

Budget configuration. Both drafters serve a dynamic draft tree bounded by a total budget BB (the number of draft nodes verified per round), grown over γ\gamma draft steps with offer width kk and grow width BwB_w (§ 5). We hold all of these equal across the two drafters: B∈{24,48}B \in\{24, 48\}, shown in parentheses in Table 1, with k=6k = 6, Bw=6B_w = 6, and γ=8\gamma= 8. Parallel Reconsideration additionally batches at most m=3m = 3 reconsidered nodes per draft step, which share the co-forwarded batch; Appendix § 13.§ 2 grants the baseline the same enlarged batch and spends it on one-pass breadth instead.

Reconsidering against drafting wider

Holding BB, kk, BwB_w, and γ\gamma equal leaves one asymmetry: a draft step of Parallel Reconsideration batches Bw+mB_w + m nodes where EAGLE-3.1 batches BwB_w. If enlarging the batch is nearly free, then EAGLE-3.1 can enlarge its own by mm and spend the additional nodes on one-pass breadth, and if that recovers the same acceptance then the gain belongs to the wider batch rather than to reconsideration.

Sensitivity to the tree configuration

Table 4 | Reconsidering the additional nodes against drafting wider with them. Gemma-3-12B at B=48B = 48, temperature 1, reporting τ\tau / speedup. The two EAGLE-3.1 controls batch 9 nodes per draft step: the first takes 3 nodes of the previous step’s batch, matching where reconsideration draws from, and forwards each one’s best not-yet-forwarded one-pass child (a new sibling branch); the second widens the grow width outright, so all 9 nodes extend the current frontier, which is more drafting breadth than Parallel Reconsideration itself gets. Parallel Reconsideration with m=0m = 0 runs our full serving path with no node ever reconsidered and batches the same 6 nodes as EAGLE-3.1; adding 3 reconsidered nodes brings it to 9. Δ\Delta is relative to EAGLE-3.1.

DrafterGSM8K τ\tauGSM8K SpeedupLiveCodeBench τ\tauLiveCodeBench SpeedupMT-Bench τ\tauMT-Bench SpeedupAvg. τ\tauAvg. Speedup
EAGLE-3.1 (Bw=6B_w = 6)5.442.81×5.543.13×3.401.64×4.792.53×
+ advance 3 more5.512.83×5.683.17×3.431.64×4.872.55×
Δ\Delta+1.3%+0.7%+2.5%+1.3%+0.9%+0.0%+1.7%+0.8%
widen to Bw=9B_w = 95.492.82×5.613.15×3.421.64×4.842.54×
Δ\Delta+0.9%+0.4%+1.3%+0.6%+0.6%+0.0%+1.0%+0.4%
PR without reconsideration (m=0m = 0)5.442.81×5.553.13×3.381.63×4.792.52×
Δ\Delta+0.0%+0.0%+0.2%+0.0%−0.6%−0.6%−0.1%−0.4%
+ reconsider 3 (PR)6.223.19×5.993.37×3.551.70×5.252.75×
Δ\Delta+14.3%+13.5%+8.1%+7.7%+4.4%+3.7%+9.6%+8.7%

Table 4. Reconsidering the additional nodes against drafting wider with them.

We therefore grant the baseline exactly that, in two ways. The first control draws mm nodes from the previous step’s batch and, for each such node vv, forwards with an ordinary draft step its best one-pass child not yet forwarded, i.e., a sibling of the child already advanced. These rows take new inputs and so compute new features, growing a second branch beneath vv; the only difference is tree growing against reconsideration. Three properties are held fixed. The source set is the same, the previous step’s batch rather than the current frontier, since reconsideration cannot reach further either. The final pruning is the same, in that the children they grow compete for the same places in the tree on the same score. And the batch is the same size, Bw=6B_w = 6 advanced nodes plus m=3m = 3 more.

The second control drops the first of those constraints and simply widens the grow width to Bw=9B_w = 9. All nine nodes now extend the current frontier, which is more drafting breadth than Parallel Reconsideration has: our own mm nodes are confined to positions already forwarded, whereas these are free to advance the best of the live frontier.

As a sanity check in the other direction, we also run Parallel Reconsideration with m=0m = 0: the same drafter and serving path, but with no node ever reconsidered, so each step batches the same Bw=6B_w = 6 nodes as EAGLE-3.1.

Without reconsideration, Parallel Reconsideration matches EAGLE-3.1 to within noise (−0.1% acceptance and −0.4% speedup on average), as Appendix § 12.§ 3 predicts. The second head and gate therefore neither help nor hurt on their own, and the serving path adds no measurable overhead when it is idle. Neither control recovers the gain either. Advancing 3 more lifts average acceptance by 1.7% over EAGLE-3.1 and widening to Bw=9B_w = 9 by 1.0%, against 9.6% for reconsidering the same three, and speedup splits the same way (+0.8% and +0.4% against +8.7%). That the freer of the two controls rules out breadth as the explanation: the additional capacity pays only when it is spent on a second look at an uncertain feature. Appendix § 13.§ 5 examines which of the one-pass drafter’s misses reconsideration recovers.

Table 5 | Sensitivity to the tree configuration. Acceptance length τ\tau and speedup over autoregressive decoding for EAGLE-3.1 and Parallel Reconsideration (PR) under identical budgets, varying one of the total budget BB, grow width BwB_w and offer width kk at a time around our default (B=48B = 48, Bw=6B_w = 6, k=6k = 6, marked * and repeated in each block); γ=8\gamma= 8 and m=3m = 3 throughout, on Qwen3-8B, GSM8K, temperature 0. Δτ\Delta\tau is PR minus EAGLE-3.1. Bold marks the best value per column.

BBBwB_wkkτ\tauSpeedup
EAGLE-3.1PRΔτ\Delta\tauEAGLE-3.1PR
Total budget
24664.394.83+0.441.94×2.11×
32664.6064.869+0.2632.077×2.208×
48*664.8355.127+0.2922.184×2.291×
64664.9615.222+0.2612.195×2.280×
Grow width
48464.7975.069+0.2722.105×2.232×
48*664.8355.127+0.2922.184×2.291×
48864.8265.126+0.3002.138×2.266×
481064.8195.124+0.3042.096×2.227×
Offer width
48644.7675.061+0.2942.162×2.286×
48*664.8355.127+0.2922.184×2.291×
48684.8375.124+0.2872.080×2.250×

Table 5. Sensitivity to the tree configuration.

Sensitivity to the tree configuration

Our main results fix one tree configuration (B=48B = 48, Bw=6B_w = 6, k=6k = 6) for both drafters. To check that the gain is not an artifact of that choice, we vary the total budget BB, the grow width BwB_w and the offer width kk one at a time around it, again holding every knob equal across the two drafters (Table 5).

The acceptance gain holds across all nine configurations: Parallel Reconsideration adds +0.26 to +0.30 tokens per round over EAGLE-3.1, or 5.3–6.3% in relative terms, from B=32B = 32 to B=64B = 64 and whether the tree is narrow (Bw=4B_w = 4) or wide (Bw=10B_w = 10), and the gain grows to +0.44 tokens (+10.0%) at the smallest budget (B=24B = 24). Reconsideration therefore neither depends on a particular tree shape nor gets absorbed by a larger one. Growing the budget raises τ\tau for both drafters, but past B=48B = 48 the added verification cost outweighs the added acceptance, so speedup peaks at our default for Parallel Reconsideration (2.29×) and saturates for EAGLE-3.1 (2.18× to 2.20×). Widening BwB_w beyond 6 leaves τ\tau unchanged for both and only costs speedup; narrowing it to 4 costs speedup most for Parallel Reconsideration, whose speedup margin nearly vanishes there (2.112× against 2.105×). Offer width behaves similarly: k=4k = 4 and k=8k = 8 keep the acceptance gain but lose speedup relative to k=6k = 6. The default is thus the best configuration for Parallel Reconsideration, and it beats EAGLE-3.1 at every configuration tried, including the baseline’s own best (2.29× against 2.20× at B=64B = 64).

Reconsideration on top of a different drafter

Reconsideration adds a second head and gate to a one-pass drafter, so it is not tied to EAGLE-3.1. We test it on top of another speculative decoding method, HASS (Zhang et al., 2025). We train HASS and HASS with reconsideration on Llama-3.1-8B, compare them under the tree configuration of Table 1 (B=48B = 48, k=6k = 6, Bw=6B_w = 6, γ=8\gamma= 8), and evaluate on all seven benchmarks at temperature 0 (Table 6).

MethodGSM8KMATH-500AIME25HumanEvalMBPPLCBMT-BenchAvg.
τ\tauSpeedupτ\tauSpeedupτ\tauSpeedupτ\tauSpeedupτ\tauSpeedupτ\tauSpeedupτ\tauSpeedupτ\tauSpeedup
HASS4.812.03×4.842.11×5.212.27×5.592.38×5.342.38×4.681.90×4.641.97×5.022.15×
HASS + PR5.172.12×5.212.21×5.882.49×5.862.45×5.672.44×4.991.98×4.972.02×5.392.24×
Δ\Delta+7.6%+4.6%+7.7%+4.6%+12.7%+9.6%+4.7%+2.9%+6.0%+2.4%+6.6%+3.9%+7.2%+2.6%+7.5%+4.4%

Table 6. Parallel Reconsideration on top of HASS. Acceptance length τ\tau and speedup over autoregressive decoding on Llama-3.1-8B at temperature 0, both drafters taken at the same training checkpoint and served under the tree configuration of Table 1 (B=48B = 48). Δ\Delta is the relative gain of adding reconsideration to HASS.

Parallel Reconsideration lifts HASS on every benchmark, by 7.5% in average acceptance length (5.02 →\to 5.39) and 4.4% in speedup (2.15× →\to 2.24×), comparable to its gain over EAGLE-3.1 on the same model (Table 1). The gain is therefore not specific to one base drafter. The largest gain is on AIME25 (+12.7% acceptance, +9.6% speedup).

Which failures does reconsideration recover?

Figure 1 is correlational: a decisive target does not by itself show that a one-layer drafter could match it with more computation. As a partial check on the reading of § 2.3, we measure which of the drafter’s failures a trained second pass recovers. Taking every token the one-pass drafter rejects (30.2% of 3.1M scored positions on the held-out split) and bucketing them by the same cpd×cptc_p^d \times c_p^t bands, we report the fraction of each bucket reconsideration, trained on the same model, turns from wrong to right (Table 7).

drafter cpdc_p^d / target confidence cptc_p^t<0.5< 0.50.50.5–0.90.9≥0.9\ge0.9all
recovered, % of bucket (bucket’s share of pool)
<0.5< 0.517.7% (18.5%)29.5% (26.6%)39.7% (17.0%)28.8% (62.1%)
0.50.5–0.90.911.3% (6.7%)23.4% (13.6%)42.0% (15.1%)29.0% (35.4%)
≥0.9\ge0.93.7% (0.4%)6.9% (1.3%)27.1% (0.8%)12.9% (2.5%)
all16.4% (25.6%)26.7% (41.5%)39.9% (32.9%)28.5% (100%)
Δ\Delta top-1 probability vs. own one-pass distribution
recovered+0.008+0.024+0.061+0.039
not recovered+0.000+0.002+0.004+0.001

Table 7. What reconsideration recovers, by bucket. Top: percentage of each bucket’s rejected tokens that Parallel Reconsideration gets right, with the bucket’s share of the reject pool in parentheses. Buckets are the cpd×cptc_p^d \times c_p^t bands of Section 2.2. Bottom: mean change in top-1 probability against the position’s own one-pass distribution, on the tokens reconsideration recovers and on those it does not.

Reconsideration recovers 28.5% of all rejected tokens, and where it recovers them supports the hypothesis. Recovery rises monotonically along the target axis (16.4→26.7→39.9%16.4 \to26.7 \to39.9\%): it is governed by whether the target had a decisive answer, not by how unsure the drafter sounded. The premature commitments of Observation 2, drafted with cpd<0.5c_p^d < 0.5 where the target was decisive, are recovered 39.7% of the time, while the both-uncertain tokens that Observation 2 identified as largely unrecoverable move least (17.7%). A mechanism that merely sharpened the existing distribution would help most where the drafter was already close; reconsideration instead spends itself on the determined-but-missed tokens, which is the band § 2.3 set out to reach.

The added confidence is selective in the same way (Table 7, bottom). On the tokens it recovers, reconsideration raises the top-1 probability by +0.039, rising to +0.061 where the target is decisive; on the tokens in the same pool it fails to recover, the same measurement gives +0.001. Each token is measured against its own one-pass distribution, which keeps cpdc_p^d, the bucketing variable, out of the difference. This suggests the deeper pass does not simply sharpen indiscriminately.

Details on Related Work

Speculative decoding. Speculative decoding accelerates autoregressive generation by drafting several tokens cheaply and verifying them in a single target forward pass, accepting the longest prefix the target agrees with so the output distribution is preserved exactly ([22] (Xia et al. 2023)). Because the acceptance rule preserves that distribution by construction, subsequent work improves speed along two axes: the acceptance length, how many drafted tokens survive verification per round, and the drafting cost, how much latency the draft itself adds. Our cost model (§ 2) is the standard amortized form of this trade-off, and Parallel Reconsideration targets the first axis without paying on the second.

Feature-level drafting. The original formulation pairs the target with a smaller standalone language model as the drafter, which must be trained or found separately for each target. Self-drafting instead derives the draft from the target’s own computation. The EAGLE family drafts at the feature level, autoregressing on a combination of hidden features from several target layers with a single lightweight decoder layer and verifying candidates with dynamic draft trees ((Li et al., 2024a,b, 2025)), most recently corrected for the attention drift that degrades the drafter as the speculation chain lengthens ((Eldenek et al., 2026)); a related line attaches lightweight prediction heads directly to the target’s last hidden state, either independent (Medusa heads) ([6]) or made sequentially dependent to raise acceptance (Hydra heads) ([2]). Parallel Reconsideration is built on the EAGLE-style feature-level drafter but is orthogonal to this design axis: it adds a second guess at uncertain positions on top of the existing one-pass draft, and applies to any drafter that exposes a reusable draft feature.

Adaptive drafter compute and draft budgets. Several lines of work already vary how much the drafter computes per token. Self-speculative decoding drafts with a subset of the target’s own layers: Draft & Verify skips intermediate layers during drafting ((Zhang et al., 2024)), LayerSkip trains the target for early exit and drafts from its shallow layers ([16]), and Kangaroo drafts from a shallow sub-network and exits drafting early once confidence drops ([28]). These methods adapt the depth of a sequential draft, so extra compute on a token is extra sequential latency; Parallel Reconsideration keeps the one-layer feature-level drafter and runs the extra pass beside the ordinary step. ReDrafter replaces the draft heads with a recurrent drafter conditioned on the target’s hidden state ([11]), which still uses one recurrent step per drafted token. Sequoia and SpecExec allocate a fixed draft-tree budget unevenly across positions, placing more candidates where they are most likely to be accepted [9] (Svirschevski et al. 2024), as EAGLE-2’s dynamic trees do with drafter confidence (Li et al., 2024b). These methods vary how many candidates each position receives, not how each candidate is computed; reconsidered candidates enter the same tree and so compose with them. A direct empirical comparison with these methods is left to future work.

Training-free drafting. A complementary direction removes the trained drafter entirely. Lookahead decoding generates and verifies nn-gram continuations in parallel through Jacobi iteration, requiring no auxiliary model or datastore (Fu et al., 2024), and FAFO extends this draftless nn-gram candidate-pool paradigm by letting a compressed KV cache produce lossy guesses that the same forward pass verifies (Le et al., 2026). Such methods trade draft quality for zero training overhead; Parallel Reconsideration takes the opposite stance, spending a small amount of otherwise-idle draft compute to make each speculated position more accurate.

Parallel drafting. A drafter that autoregresses over γ\gamma positions is itself sequential, so a line of work replaces it with one that emits an entire block in a single forward pass. PARD trains a target-independent draft model to predict several future tokens at once (An et al., 2025), and diffusion-based drafters push this further: DFlash conditions a block diffusion model on target features to produce a draft block in one pass and holds the strongest reported speedups on this benchmark suite [7], while DSpark couples a parallel backbone with a lightweight sequential module to restore intra-block dependencies [10]. These methods reduce the cost axis by removing the drafter’s sequential steps, whereas Parallel Reconsideration raises the acceptance axis by improving each drafted position; a block-parallel drafter still commits to every position from a single pass, so reconsideration composes with it.

Draft training objectives. Orthogonal to the drafter’s architecture is what it is trained to optimize. Standard drafters minimize per-token likelihood against the target, which is misaligned with the sequence-level acceptance that verification actually rewards. Variational Speculative Decoding recasts draft training as variational inference over draft paths, maximizing the marginal probability of target acceptance rather than token likelihood (Zou et al., 2026). Our training (Section 4) shares this acceptance-first view but pursues it through a different mechanism: rather than reweighting a single draft distribution, we train a second reconsideration pass to recover the specific positions the one-pass draft gets wrong.

Looped and adaptive-compute transformers. The idea of spending extra passes to sharpen a prediction has a long history. Looped transformers reuse a fixed block to emulate iterative computation [18], universal transformers add a recurrent depth with a per-token halting mechanism [13], and adaptive-compute methods such as PonderNet and CALM learn how many steps or layers each token needs [5] (Schuster et al. 2022). A recent looped language model, Ouro, applies this recurrence to the target model during pre-training to improve its own generation quality (Zhu et al., 2025). Our use of recurrence differs in purpose: we loop not to make an autoregressive model reason better, but to repair the specific failure of autoregressive drafting, where the drafter must commit to a token before it has seen enough context to place it, and we run the extra pass in parallel so it costs no additional sequential step.

Implementation details

Training. Both drafters follow EAGLE-3’s released recipe on UltraChat ([14]), with no hyperparameter retuning for reconsideration. We use AdamW (β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, no weight decay) at a peak learning rate of 10−410^{-4}, cosine-annealed to zero after linear warmup over the first 1.5% of steps, with gradients clipped to norm 0.5. Training runs in BF16 with FP32 master weights and gradient checkpointing, sequences capped at 4,096 tokens, per-device batch size 1 at data-parallel degree 4 and no accumulation, giving an effective batch of four sequences (roughly 16k tokens) per step. The balance coefficient of Eq. (6) is β=0.01\beta= 0.01.

Benchmarks. We evaluate each benchmark on its full set.

Table 8 | Evaluation benchmarks. Each is evaluated on its full set.

CategoryBenchmarkPromptsSource
ChatMT-Bench80FastChat mt_bench
MathGSM8K1,319openai/gsm8k (main)
MathMATH-500500HuggingFaceH4/MATH-500
MathAIME2530math-ai/aime25
CodeHumanEval164openai/openai_humaneval
CodeMBPP500google-research-datasets/mbpp (full)
CodeLiveCodeBench400livecodebench/code_generation
Total2,993

Table 8. Evaluation benchmarks. Each is evaluated on its full set.

Limitations

Reconsideration, in principle, composes with any drafter that exposes a reusable draft feature, including the block-parallel drafters that are currently state of the art (Appendix § 14). DFlash (Chen et al., [7]) is the natural next target: it reduces the cost of drafting while reconsideration raises the acceptance of each drafted position, so the two act on different terms of Eq. (2) and there is no obvious reason they would conflict. Unlike EAGLE-style drafters, however, DFlash does not rely on a token embedding to guide its feature drafting, so there is no embedding input for the self-query to take the place of (Eq. (5)); applying reconsideration on top would require substantial redesign, which we leave for future work. Doing so also requires retraining a DFlash drafter with the extra head and gate, and its training recipe has not been released, so we cannot reproduce the baseline or train reconsideration on top of it. We therefore demonstrate on EAGLE-3.1, whose recipe is publicly released, and leave the block-parallel combination open.

The cost of co-forwarding, measured

§ 3.4 argues that a reconsideration pass shares the batch that advances the draft, costing ϵ\epsilon rather than a full draft step. We check that claim directly. For each cell we form the per-cycle time tcycle=τ/throughputt_{\mathrm{cycle}} = \tau/\mathrm{throughput}, the wall-clock milliseconds per verification step, and compare Parallel Reconsideration against EAGLE-3.1 run for run at temperature 0. Table 9 reports the ratio alongside the serial reference of § 3.4: run in series, the NN reconsidered positions of a γ\gamma-step draft add NN sequential draft steps, so drafting takes (γ+N)tstep(\gamma+ N)t_{\mathrm{step}} instead of γtstep\gamma t_{\mathrm{step}}, a factor of (γ+N)/γ(\gamma+ N)/\gamma. factor inflates only the drafting phase, so the serial version’s whole-cycle cost would lie between 1 and (γ+N)/γ(\gamma+ N)/\gamma depending on drafting’s share of the cycle.

BBBwB_wNNEAGLE-3.1 tcyclet_{\mathrm{cycle}}PR tcyclet_{\mathrm{cycle}}measured whole-cycle ratioserial drafting (γ+N)/γ(\gamma+N)/\gamma
486326.0326.601.022×1.375×
246325.5525.941.015×1.375×

Table 9. Per-cycle cost of co-forwarding on Qwen3-8B. Measured ratio of tcycle=τ/throughputt_{\mathrm{cycle}} = \tau/\text{throughput} (time per verification step), Parallel Reconsideration over EAGLE-3.1 run for run at temperature 0; this is a whole-cycle ratio (drafting plus verification). With γ=8\gamma= 8 draft steps and NN reconsidered positions per draft, serializing would stretch drafting by (γ+N)/γ(\gamma+ N)/\gamma (§ 3.4), an upper bound on the serial whole-cycle ratio. Co-forwarding keeps the measured ratio near 1.

With γ=8\gamma= 8 and N=3N = 3 reconsidered positions, serializing would lengthen drafting by 37.5%; co-forwarded, the same reconsiderations lengthen the whole cycle by only 1.5–2.2% (ratios 1.015 and 1.022 at B=24B = 24 and B=48B = 48). The co-forwarded rows thus add roughly 2% to total cycle time (drafting plus verification), consistent with the memory-bandwidth argument of § 3.4.

Supervising both heads

This section expands on the two degenerate training objectives of § 4 and why each collapses serving quality.

Supervising only the reconsidered head. Training only head⁡(fpr)\operatorname{head}(f_p^r) for r≥1r \ge1 is the naive way to apply a looped transformer (Dehghani et al., [13]; Giannou et al., [18]) to drafting: run the layer to its final depth and supervise the last head, exactly as one would optimize a recurrent model for its own output. It fails here for a reason specific to speculative decoding. The one-pass head head⁡(fp0)\operatorname{head}(f_p^0) is never trained, so the one-pass draft is free to drift and nothing holds it to ordinary-drafter quality. Because the served tree is grown from the one-pass feature and the reconsidered candidates only extend prefixes the one-pass draft already reaches, a degraded one-pass front leaves the reconsidered branches with short or empty prefixes to build on. At serving, this can pull acceptance length below the one-pass-only baseline even if the reconsidered head looks accurate in isolation under teacher forcing. This is why adapting loop transformer idea to the draft is non-trivial: unlike a looped language model, whose only output is the final pass, the draft must keep every intermediate pass serving-ready, since the tree is grown from the one-pass feature and reconsideration only adds to it.

Supervising both heads symmetrically. If instead both head⁡(fp0)\operatorname{head}(f_p^0) and head⁡(fp1)\operatorname{head}(f_p^1) are trained to match the target with equal weight, gradient descent has no incentive to make the two features differ, and it is free to settle on the trivial optimum fp1≈fp0f_p^1 \approx f_p^0. The reconsidered token then echoes the one-pass token: the second pass is real compute that adds no new candidate, and acceptance length can do no better than the one-pass baseline.

The recipe of § 4 avoids both failures by holding the one-pass draft to full quality with an unconditional loss while supervising the reconsidered feature only at the positions and depths where it is actually used, so fp1f_p^1 is pushed away from fp0f_p^0 exactly where a second guess can help.

Marginal vs. per-position entropy

The balance term of Eq. (6) is not the obvious entropy. The halt distribution induces two: the per-position entropy averaged over positions, Ep[H(q)]\mathbb{E}_{p}[H(q)], and the entropy of the marginal usage qˉ=Ep[q]\bar{q} = \mathbb{E}_{p}[q]. We reward the marginal H(qˉ)H(\bar{q}), and a clean decomposition says why.

Proposition 1 (Marginal vs. per-position entropy). For any distribution of per-position halt distributions qq,

H(qˉ)=I(d;p)⏟spread across depths+Ep[H(q)]⏟per-position indecision≥Ep[H(q)],(13)\begin{aligned} H(\bar{q}) &= \underbrace{I(d;p)}_{\text{spread across depths}} + \underbrace{\mathbb{E}_{p}[H(q)]}_{\text{per-position indecision}} \\ &\ge\mathbb{E}_{p}[H(q)], \tag*{(13)} \end{aligned}

where d∼qd \sim q is the halt depth and I(d;p)≥0I(d;p) \ge0 is its mutual information with the position. Consequently H(qˉ)H(\bar{q}) depends on the routing only through the marginal qˉ\bar{q}: it is indifferent between a deterministic routing that sends different positions to different depths (I(d;p)I(d;p) large, Ep[H(q)]=0\mathbb{E}_{p}[H(q)] = 0) and an undecided one that splits every position across depths (I(d;p)=0I(d;p) = 0, Ep[H(q)]\mathbb{E}_{p}[H(q)] large), so long as the two induce the same qˉ\bar{q}.

Proof of Proposition 1. Fix a batch of positions pp, each with a halt distribution q(⋅∣p)q(\cdot\mid p) over depths r∈{0,1,…,R}r \in\{0, 1, \ldots, R\}, and let pp be drawn uniformly over the batch. Define the joint distribution of position and halt depth dd,

Pr⁡(p,d=r)=Pr⁡(p) q(r∣p),(14)\Pr(p,d=r) = \Pr(p)\,q(r \mid p), \tag*{(14)}

and the marginal usage qˉ(r)=Ep[q(r∣p)]=Pr⁡(d=r)\bar{q}(r) = \mathbb{E}_{p}[q(r \mid p)] = \Pr(d=r).

Decomposition. The entropy of the marginal is the entropy of dd, which by the chain rule of mutual information decomposes as

H(qˉ)=H(d)=I(d;p)+H(d∣p)=I(d;p)+Ep[H(q(⋅∣p))],(15)\begin{aligned} H(\bar{q}) &= H(d) = I(d;p) + H(d \mid p) \\ &= I(d;p) + \mathbb{E}_{p}[H(q(\cdot\mid p))], \tag*{(15)} \end{aligned}

since H(d∣p)=∑pPr⁡(p)H(q(⋅∣p))=Ep[H(q)]H(d \mid p) = \sum_{p}\Pr(p)H(q(\cdot\mid p)) = \mathbb{E}_{p}[H(q)]. Because I(d;p)≥0I(d;p) \ge0, this gives H(qˉ)≥Ep[H(q)]H(\bar{q}) \ge\mathbb{E}_{p}[H(q)], which is exactly the Jensen inequality H(Ep[q])≥Ep[H(q)]H(\mathbb{E}_{p}[q]) \ge\mathbb{E}_{p}[H(q)] applied to the concave entropy functional; (15) identifies the gap as the mutual information I(d;p)I(d;p). This proves (13).

Indifference. Both terms in (15) are nonnegative and H(qˉ)≤log⁡(R+1)H(\bar{q}) \le\log(R+1), with equality iff qˉ\bar{q} is uniform. Since H(qˉ)H(\bar{q}) is a function of qˉ\bar{q} alone, it cannot distinguish two routings that induce the same marginal, however differently they allocate mass per position. Two extremes make this concrete. The deterministic routings — q(⋅∣p)q(\cdot\mid p) a point mass at some depth d(p)d(p), with d(p)d(p) distributed uniformly across the batch — attain H(qˉ)=log⁡(R+1)H(\bar{q}) = \log(R+1) with Ep[H(q)]=0\mathbb{E}_{p}[H(q)] = 0 and I(d;p)=log⁡(R+1)I(d;p) = \log(R+1). The fully undecided routing — every q(⋅∣p)q(\cdot\mid p) uniform — attains the same H(qˉ)=log⁡(R+1)H(\bar{q}) = \log(R+1) with the decomposition reversed, Ep[H(q)]=log⁡(R+1)\mathbb{E}_{p}[H(q)] = \log(R+1) and I(d;p)=0I(d;p) = 0. The balance term scores both identically, so it neither rewards nor penalizes per-position indecision; it penalizes only a marginal that has collapsed onto a subset of the depths. By contrast, regularizing Ep[H(q)]\mathbb{E}_{p}[H(q)] directly is maximized only by the second extreme, forcing per-position indecision. □

Practical note. We estimate qˉ\bar{q} per minibatch, so the balance term rewards spread across the depths used within a batch rather than a global marginal; with the small coefficient β=0.01\beta= 0.01 this is enough to keep every depth populated without measurably softening the per-position halt decisions.

References

  1. [2]Z. Ankner, R. Parthasarathy, A. Nrusimha, C. Rinard, J. Ragan-Kelley, and W. Brandon. Hydra: Sequentially-dependent draft heads for Medusa decoding. arXiv preprint arXiv:2402.05109, 2024.arxiv.org/abs/2402.05109
  2. [3]Anthropic. Claude code. https://claude.com/product/claude-code, 2025.
  3. [4]J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.arxiv.org/abs/2108.07732
  4. [5]A. Banino, J. Balaguer, and C. Blundell. PonderNet: Learning to ponder. In ICML Workshop on Automated Machine Learning (AutoML), 2021.arxiv.org/abs/2107.05407
  5. [6]T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In International Conference on Machine Learning (ICML). PMLR, 2024.arxiv.org/abs/2401.10774
  6. [7]J. Chen, Y. Liang, and Z. Liu. DFlash: Block diffusion for flash speculative decoding. In International Conference on Machine Learning (ICML), 2026.arxiv.org/abs/2602.06036
  7. [8]M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.arxiv.org/abs/2107.03374
  8. [9]Z. Chen, A. May, R. Svirschevski, Y. Huang, M. Ryabinin, Z. Jia, and B. Chen. Sequoia: Scalable, robust, and hardware-aware speculative decoding. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2402.12374
  9. [10]X. Cheng, X. Yu, C. Shao, J. Li, et al. DSpark: Confidence-scheduled speculative decoding with semi-autoregressive generation. arXiv preprint arXiv:2607.05147, 2026.arxiv.org/abs/2607.05147
  10. [11]Y. Cheng, A. Zhang, X. Zhang, C. Wang, and Y. Wang. Recurrent drafter for fast speculative decoding in large language models. arXiv preprint arXiv:2403.09919, 2024.arxiv.org/abs/2403.09919
  11. [12]K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
  12. [13]M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and Ł. Kaiser. Universal transformers. In International Conference on Learning Representations (ICLR), 2019.arxiv.org/abs/1807.03819
  13. [14]N. Ding, Y. Chen, B. Xu, Y. Qin, Z. Zheng, S. Hu, Z. Liu, M. Sun, and B. Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.arxiv.org/abs/2305.14233
  14. [15]D. Eldenk, P. Mohapatra, Y. Comlek, K. Oktay, H. Zhang, and S. Xia. Attention drift: What autoregressive speculative decoding models learn. arXiv preprint arXiv:2605.09992, 2026.arxiv.org/abs/2605.09992
  15. [16]M. Elhoushi, A. Shrivastava, D. Liskovich, B. Hosmer, B. Wasti, L. Lai, A. Mahmoud, B. Acun, S. Agarwal, A. Roman, A. Aly, B. Chen, and C.-J. Wu. LayerSkip: Enabling early exit inference and self-speculative decoding. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2024.
  16. [18]A. Giannou, S. Rajput, J.-y. Sohn, K. Lee, J. D. Lee, and D. Papailiopoulos. Looped transformers as programmable computers. In International Conference on Machine Learning (ICML). PMLR, 2023.arxiv.org/abs/2301.13196
  17. [19]Google. Google antigravity. https://antigravity.google/, 2025.
  18. [20]N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2403.07974
  19. [22]Y. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning (ICML), pages 19274–19286. PMLR, 2023.arxiv.org/abs/2211.17192
  20. [27]H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Gobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023.arxiv.org/abs/2305.20050
  21. [28]F. Liu, Y. Tang, Z. Liu, Y. Ni, K. Han, and Y. Wang. Kangaroo: Lossless self-speculative decoding via double early exiting. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2404.18911
  22. [29]Mathematical Association of America. American invitational mathematics examination (AIME). https://maa.org/math-competitions/american-invitational-mathematics-examination-aime, 2025.

Paper details

vlnc:W_wtqf52yf
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents