WorksComputer Science
HyFlash: A Prefix-Memory Prior for On-Device Speculative Decoding
Abstract
Parallel block drafting accelerates speculative decoding by predicting multiple future positions in a single backbone pass, but on-device deployment constrains both drafter capacity and context storage. Under tight memory budgets, drafters must model long-range dependencies and fine-grained draft-position interactions without the growing KV-cache overhead of full-context attention. We propose HyFlash, a compact parallel drafter that introduces a prefix-memory prior to separate long-range context modeling from fine-grained draft-position interactions with limited capacity. Local attention captures recent context and draft-position interactions, while recurrent memory compresses historical target features into fixed-size states. This inductive bias enables effective local and long-range modeling with bounded draft-side memory. Experiments show that HyFlash achieves higher acceptance than full-context attention with substantially less KV storage and with larger gains for shallow drafters and long contexts.
Introduction
Speculative decoding accelerates autoregressive language model inference by using a lightweight drafter to propose tokens that a larger target model verifies in parallel [26]. Recent parallel block drafters, such as DFlash [7] and DSpark [9], further reduce drafting latency by predicting multiple future positions in a single backbone pass. However, maintaining high draft acceptance requires effectively modeling both historical context and interactions among draft positions while keeping drafting overhead small.
This challenge becomes particularly pronounced in on-device deployment [33, 2, 34], where memory is severely constrained [30] (Xu et al. 2024a). Target weights, KV cache, and runtime overhead leave only a small residual budget for drafter parameters and context storage (Tiwari et al., 2025). Under limited model capacity, jointly modeling long-range dependencies and fine-grained intra-block interactions becomes difficult, even with full-context attention. Moreover, full attention incurs growing KV memory, while windowed attention restricts direct access to distant information (Yang et al., 2026; Yuan et al., 2026). These limitations are especially relevant to long-horizon conversations and tool-using agents.
Existing studies address memory-efficient speculative decoding through different approaches. LongSpec (Yang et al., 2026) bounds draft-side KV memory, while MASW (Yuan et al., 2026) compresses historical KV representations. H-Spec [24] combines recurrent drafting with target-KV reuse, whereas ReaDy [38] leverages recurrent states from hybrid target models. These advances motivate a complementary question: how can a capacity-constrained parallel drafter effectively model both local interactions and long-range context without relying on additional depth or full-context attention?
We propose HyFlash, a compact parallel drafter built around a prefix-memory prior for capacity-constrained drafting. HyFlash maintains persistent memory of committed target features while using local attention for recent context and bidirectional draft-position interactions. Rather than updating recurrent states through speculative positions, HyFlash lets them query a shared, read-only prefix memory. Our contributions are threefold:
We characterize the capacity–context tension in on-device parallel drafting, highlighting the challenges of modeling long-range dependencies with shallow drafters.
We introduce a prefix-memory prior that separates historical context modeling from local and intra-block interactions, enabling effective draft-
ing with bounded context memory.
We demonstrate improved acceptance and decoding efficiency across two target models, with capacity scaling and targeted interventions validating the benefits of this inductive bias under limited model capacity.
Memory Constraints in On-Device Parallel Drafting
Parallel block drafters must accommodate their weights and context alongside the target model. On-device memory constraints therefore limit both drafter capacity and access to historical context.
Memory pressure constrains capacity. On-device speculative decoding must accommodate target weights, KV cache, and runtime overhead before allocating memory to the drafter. We consider an illustrative 12 GiB inference budget for 16 GB-class devices, reserving approximately 2.9 GiB for the operating system and other workloads (Appendix B). Using Phi-4-reasoning [1] as a geometry reference for our 14B-family prototype, the target KV cache alone requires 6.25 GiB at 32K context. Even with ideal 2-bit target weights, at most 0.84 GiB remains for the drafter and other inference overheads (Figure 1a). Such tight budgets severely restrict drafter capacity, motivating architectures that make effective use of only a few layers.

Figure 1. Motivation for memory-constrained drafting. (a) Analytical target memory usage under an illustrative 12 GiB inference budget for 16 GB-class devices (Appendix B). (b) Acceptance across evidence distances for independently trained two-layer drafters with equal 5 MiB logical context budgets.
Limited capacity challenges context modeling. Parallel drafting requires modeling both long-range dependencies and fine-grained interactions among draft positions, which becomes challenging under limited model capacity. Even with full-context attention, a compact drafter may struggle to effectively utilize historical information, while retaining the full KV cache incurs substantial memory overhead. Windowed attention reduces this cost but limits direct access to distant information, leading to substantial acceptance degradation as shown in Figure 1b. These observations highlight two complementary limitations: windowed attention restricts access to distant information, while full-context attention incurs growing memory costs without necessarily overcoming the modeling limitations of shallow drafters. These challenges motivate a prefix-memory prior that combines recurrent modeling of historical context with local attention for recent and intra-block interactions.
Figure 2. HyFlash framework. Target-conditioned hybrid layers produce representations for a separate proposal readout. The same target verifies proposals and returns the next anchor; only newly committed target features update memory.
HyFlash
We propose HyFlash, a compact parallel drafter combining local attention and recurrent memory within each layer to efficiently capture local interactions and long-range context with limited capacity.
Drafting from a committed prefix. At boundary , the target has processed and selected a known anchor , whose feature is not yet available. An adapter projects selected target hidden layers into , . Given these features, the anchor, and masked future positions, a single backbone pass produces representations for draft positions, which can be used with different proposal generation strategies.
Intra-layer hybrid attention. Each layer implements the prefix-memory prior by combining sliding-window attention (SWA) for recent and draft-position interactions with recurrent memory that compresses historical target features. Unlike conventional hybrid sequence modeling (Dong et al., [15]), HyFlash explicitly separates committed history from speculative positions: recurrent states are updated only from committed target features and remain read-only during parallel drafting. We instantiate the recurrent memory with Mamba2 (Dao and Gu, [12]), whose selective state update follows
Here, is an input-dependent step size, is a learned head-specific decay rate, and and are bottleneck projections of committed target features. For each draft representation , the local and recurrent branches retrieve recent and historical context, respectively. We define:
where denotes the draft-block representations. The two branches are computed and fused as
Local attention attends to the last committed target features and all draft-block positions bidirectionally, while the recurrent branch queries the compressed prefix state without modifying it. Here, is the shared read-query projection, projects concatenated head outputs, and are learned channel scales. LN and RN denote LayerNorm (Ba et al., [5]) and RMSNorm (Zhang and Sennrich, 2019), respectively.
Persistent prefix memory. Rather than relying solely on recent target features, HyFlash explicitly accumulates committed target features in a persistent recurrent state across drafting rounds. After verification, only newly processed, committed features update the memory, excluding unverified and rejected proposals. Parallel draft positions query this memory without modifying it, preserving historical context without rescanning previous tokens.
Memory footprint. With bounded-window KV storage, HyFlash maintains a constant-size logical context footprint as the prefix grows. In our Phi-4-family configuration, local KV and FP32 recurrent states each require 2.5 MiB, together matching the 5 MiB logical context budget of a two-layer windowed drafter. This enables long-range context modeling without increasing drafter depth or the total logical context footprint. These estimates exclude model weights, target KV, and temporary buffers.
Experimental Evaluation
Experimental Setup
We evaluate HyFlash on Qwen3-4B (Yang et al., 2025) and Phi-4-reasoning. All drafters have two layers and use a block size of 7 with chain-based drafting. For training, we follow Cheng et al. [9] and use Open-PerfectBlend, an open-source version of PerfectBlend (Xu et al., 2024b) containing 1.3 million samples. We replace the original responses with those generated by each target model to improve target alignment. Evaluation covers three task categories: Math (GSM8K; Cobbe et al. [11], MATH; Lightman et al. [29], AIME25), Code (HumanEval; Chen et al. [8], MBPP; Austin et al. [4], LiveCodeBench; Jain et al. [23]), and Chat (MT-Bench; Zheng et al. 2023). We report average acceptance length () and end-to-end decoding speedup relative to autoregressive decoding, using standard speculative decoding (Leviathan et al., [26]) at a sampling temperature of 1.0. The detailed experimental setups are in Appendix § 11.
Main Results
General benchmarks. Table 1 summarizes speculative decoding performance across math, code, and chat benchmarks. HyFlash consistently improves both decoding speedup and acceptance length across Qwen3-4B and Phi-4-reasoning, outperforming DFlash and DSpark across all seven benchmarks. Compared with DSpark, HyFlash increases average decoding speedup from to on Qwen3-4B and from to on Phi-4-reasoning, corresponding to improvements of 10.0% and 16.2%, respectively. Average acceptance length similarly increases from 3.67 to 4.10 and from 3.28 to 3.79. On Phi-4-reasoning, HyFlash also outperforms the SWA variant, suggesting that recurrent prefix memory effectively complements local attention to improve draft quality and decoding efficiency. These results suggest that the prefix-memory prior improves draft quality sufficiently to yield end-to-end decoding gains despite the additional recurrent computation.
| Target | Drafter | Math | Code | Chat | |||||||||||||
| GSM8K | MATH-500 | AIME25 | HumanEval | MBPP | LCB | MT-Bench | Avg. | ||||||||||
| Speedup | Speedup | Speedup | Speedup | Speedup | Speedup | Speedup | Speedup | ||||||||||
| Q3-4B | DFlash | 1.89 | 3.19 | 1.80 | 3.28 | 1.64 | 3.12 | 1.70 | 3.02 | 1.63 | 2.89 | 1.36 | 2.75 | 1.53 | 2.63 | 1.64 | 2.98 |
| DSpark | 2.28 | 3.92 | 2.11 | 4.02 | 1.93 | 3.82 | 2.04 | 3.73 | 1.91 | 3.54 | 1.66 | 3.46 | 1.79 | 3.19 | 1.95 | 3.67 | |
| HyFlash | 2.53 | 4.38 | 2.30 | 4.48 | 2.16 | 4.38 | 2.23 | 4.13 | 2.08 | 3.94 | 1.82 | 3.83 | 1.98 | 3.59 | 2.15 | 4.10 | |
| P4-14B | DFlash | 2.09 | 3.13 | 2.38 | 3.58 | 2.04 | 3.10 | 2.21 | 3.26 | 2.20 | 3.17 | 1.75 | 2.78 | 1.62 | 2.28 | 2.03 | 3.04 |
| DSpark | 2.39 | 3.48 | 2.51 | 3.79 | 2.14 | 3.29 | 2.47 | 3.53 | 2.30 | 3.33 | 2.01 | 3.02 | 1.84 | 2.50 | 2.22 | 3.28 | |
| + SWA | 2.26 | 3.34 | 2.45 | 3.64 | 2.05 | 3.15 | 2.31 | 3.41 | 2.24 | 3.26 | 1.75 | 2.79 | 1.70 | 2.45 | 2.09 | 3.15 | |
| HyFlash | 2.80 | 4.02 | 2.87 | 4.21 | 2.52 | 3.78 | 2.78 | 4.06 | 2.95 | 4.12 | 2.18 | 3.36 | 2.13 | 3.01 | 2.58 | 3.79 |
Table 1. Decoding speedup over autoregressive decoding and average acceptance length () across math, code, and chat benchmarks on Qwen3-4B and Phi-4-reasoning (14B). Both targets use thinking mode, with a maximum generation length of 4,096 tokens.
Long-context evaluation. We further evaluate HyFlash on three LongBench (Bai et al., [6]) tasks, including HotpotQA (Yang et al., 2018), QASPER (Dasigi et al., [13]), and GovReport (Huang et al., [21]), with context lengths ranging from 4K to 32K tokens (Table 2). Across all tasks and context lengths, HyFlash consistently outperforms both full-context and sliding-window attention. Compared with SWA, HyFlash achieves higher acceptance by preserving historical information through recurrent memory while maintaining bounded context storage. Meanwhile, full-context
| Context | HotpotQA | QASPER | GovReport | ||||||
| Full | SWA | Hy. | Full | SWA | Hy. | Full | SWA | Hy. | |
| 2K–4K | 2.78 | 2.72 | 3.02 | 2.61 | 2.45 | 2.70 | 2.43 | 1.88 | 2.50 |
| 4K–8K | 3.23 | 2.57 | 3.25 | 2.52 | 2.31 | 2.64 | 2.50 | 1.66 | 2.61 |
| 8K–16K | 1.24 | 2.25 | 3.11 | 1.79 | 2.12 | 2.77 | 1.19 | 1.57 | 2.60 |
| 16K–32K | 1.02 | 2.13 | 2.72 | 0.97 | 1.81 | 2.83 | 0.86 | 1.58 | 2.44 |
Table 2. Average acceptance length () of Phi-4-reasoning on LongBench across context lengths.
attention exhibits substantial degradation beyond the 8K training length, whereas HyFlash maintains robust acceptance with bounded context memory.
Ablation Studies
We further analyze HyFlash using Phi-4, with additional results provided in Appendix § 12.
Role of hybrid context modeling. We examine the complementary roles of recurrent memory and local attention through inference-time ablations (Table 3). Last 512 and Last feature restrict the SSM history to recent committed target features, while Cross-slot off disables attention among draft positions while preserving attention to recent target features. Restricting recurrent history degrades acceptance on distant evidence, while retaining the recent 512 features largely preserves nearby and late-slot acceptance. In contrast, disabling cross-slot attention impairs nearby and late-slot predictions even with full recurrent history. These complementary effects indicate that recurrent history and intra-block attention serve distinct functional roles during drafting, supporting the proposed separation of long-range context modeling and local interactions under limited capacity.
| Intervention | Nearby (↑) | Distance (↑) | Late-slot Acc. (↑) |
| HyFlash | 6.32 | 5.62 | 0.77 |
| Recurrent history | |||
| SSM: Last 512 | 6.32 | 3.09 | 0.77 |
| SSM: Last feature | 4.66 | 2.19 | 0.68 |
| Intra-block interactions | |||
| Cross-slot off | 5.68 | 4.95 | 0.71 |
| Cross-slot off + SSM512 | 5.69 | 2.88 | 0.71 |
Table 3. Ablations of recurrent memory and intra-block attention. Nearby/Distance: average acceptance length on synthetic retrieval probes; Late-slot Acc.: conditional acceptance at draft positions 4–7 on natural prefixes.
Effect of drafter capacity. We investigate the effect of drafter capacity by varying backbone depth, with approximately matched parameter counts between HyFlash and full-attention drafters at each depth. As shown in Figure 3, HyFlash consistently outperforms full-context attention-only drafters, with the performance gap narrowing as depth increases. Notably, a one-layer HyFlash is competitive with two-layer full-attention drafters, while its two-layer variant approaches the performance of five-layer counterparts. These results suggest that the prefix-memory prior provides an effective inductive bias for capacity-constrained drafting, with larger benefits in shallow models.
Figure 3. Average acceptance length () of HyFlash and full-context attention drafters across different depth.
Robustness to weight quantization. To assess the practicality of HyFlash under tight memory budgets, we quantize the drafter weights to 4 bits using GPTQ [16], while keeping the target model unchanged. As shown in Table 4, the quantized drafter retains 99.2% of the BF16 acceptance length on average, while improving decoding speedup by 4.3%. These results demonstrate that the benefits of the prefix-memory prior are largely preserved under aggressive weight quantization, strengthening the case for HyFlash under constrained drafter weight budgets.
| Method | MATH-500 | AIME25 | MBPP | LCB | ||||
| Spd. | Spd. | Spd. | Spd. | |||||
| BF16 | 2.87 | 4.21 | 2.52 | 3.78 | 2.95 | 4.12 | 2.18 | 3.36 |
| INT4 | 2.98 | 4.18 | 2.61 | 3.72 | 3.08 | 4.09 | 2.30 | 3.35 |
| Rel. (%) | 3.8 | −0.7 | 3.6 | −1.6 | 4.4 | −0.7 | 5.5 | −0.7 |
Table 4. Effect of 4-bit weight quantization on decoding speedup (Spd.) and average acceptance length () of HyFlash across math and code benchmarks. Rel. denotes performance relative to BF16.
Generality of Prefix-Memory Prior
Mamba-2 vs. Gated DeltaNet-2. To examine whether HyFlash depends on the choice of recurrent operator, we replace Mamba-2 with Gated DeltaNet-2 (GDN-2; [18]), while keeping local attention and fusion unchanged.
Unlike Mamba-2, GDN-2 employs channel-wise decay and separate erase and write gates, updating its recurrent state as
where and control erasure and writing, respectively. The state is updated only from committed target features and remains frozen during drafting. As shown in Table 5, both operators achieve comparable acceptance lengths and decoding speedups, with Mamba-2 performing slightly better on average. These results suggest that the benefits of the prefix-memory prior are not specific to Mamba-2.
| Method | MATH-500 | AIME25 | MBPP | LCB | ||||
| Spd. | Spd. | Spd. | Spd. | |||||
| Mamba-2 | 2.87 | 4.21 | 2.52 | 3.78 | 2.95 | 4.12 | 2.18 | 3.36 |
| GDN-2 | 2.85 | 4.14 | 2.47 | 3.59 | 2.76 | 4.01 | 2.18 | 3.38 |
Table 5. Effect of recurrent operators on decoding speedup (Spd.) and average acceptance length ().
DSpark vs. DFlash2. To examine the generality of HyFlash across different proposal mechanisms, we replace the DSpark readout with that of DFlash2 ([22]), including its dynamic local convolutions and pairwise path selector, while keeping the hybrid backbone unchanged. As shown in Table 6, HyFlash achieves strong acceptance lengths and decoding speedups with both readouts, with DFlash2 providing modest but consistent improvements across all benchmarks. These results suggest that the prefix-memory prior is compatible with different proposal mechanisms rather than being tied to the DSpark readout.
| Method | MATH-500 | AIME25 | MBPP | LCB | ||||
| Spd. | Spd. | Spd. | Spd. | |||||
| DSpark | 2.87 | 4.21 | 2.52 | 3.78 | 2.95 | 4.12 | 2.18 | 3.36 |
| DFlash2 | 2.92 | 4.48 | 2.61 | 3.94 | 3.05 | 4.30 | 2.26 | 3.58 |
Table 6. Effect of proposal readout mechanisms on decoding speedup (Spd.) and average acceptance length (). Both methods use the same HyFlash backbone, differing only in their proposal readout mechanisms.
Conclusion
On-device memory constraints limit drafter capacity and context storage. HyFlash introduces a prefix-memory prior that combines recurrent modeling of historical context with local attention for recent and intra-block interactions. Capacity scaling and targeted interventions suggest that this inductive bias is particularly effective for shallow parallel drafters, improving acceptance over attention-only baselines while maintaining bounded draft-side context memory.
Limitations
Our study focuses on shallow parallel drafters under tight memory constraints. While HyFlash combines local attention with a fixed-size recurrent state, the latter provides a compressed representation of historical information rather than exact token-level access to the entire prefix. Furthermore, we primarily investigate two-layer drafters with a fixed attention window, leaving alternative capacity allocations and adaptive memory configurations for future exploration.
Acknowledgments
We would like to thank Rohith Kukkala, Karthik Vijayan, Sara Abdali, Rhea Sukthanker, and Michael Sapienza for valuable discussions and support during the course of this work.
References
Related Work
Here, we discuss prior work on speculative decoding, efficient long-context modeling, and memory-efficient drafting. We highlight how these research directions motivate our design and distinguish HyFlash from existing approaches.
Speculative decoding. Speculative decoding accelerates generation by verifying tokens proposed by a lightweight drafter [26]. While EAGLE [27] and EAGLE-3 [28] employ autoregressive drafting, recent methods such as DFlash [7], Domino [20], DSpark [9], and DFlash2 [22] explore parallel drafting with improved inter-token dependencies. In contrast, HyFlash focuses on efficient context modeling for parallel drafting.
Efficient long-context modeling. SSMs such as Mamba [17] and Mamba-2 [12] maintain compact recurrent states for efficient sequence modeling. Hybrid architectures, including Griffin [14], Jamba [25], and Hymba [15], combine recurrence with attention for efficient language modeling. Building on this principle, HyFlash adapts hybridization to parallel drafting: committed target features update persistent recurrent memory, while speculative draft positions query the read-only state and interact through local attention over recent context and other draft positions.
Memory-efficient speculative drafting. MagicDec [37] reduces draft KV-cache overhead, while Mamba Drafters [10] employ recurrent drafters. More closely related, H-Spec [24] combines Mamba and attention for parallel drafting, initializing recurrent states from the target’s last-token features and reusing its KV cache. Unlike H-Spec, which initializes recurrent states from the target’s last-token features and scans draft positions, HyFlash maintains a drafter-owned persistent state updated only from committed target features.
Memory Budget and Synthetic Diagnostic
Hardware context for memory budget. Our illustrative 12 GiB inference budget in Figure 1 is motivated by the memory capacities of commercially available on-device AI platforms.
As summarized in Table 7, representative devices offer 8–32 GB of physical memory, with 16 GB configurations available across consumer and embedded platforms, including Microsoft Surface Laptop [33], Apple Mac mini [2], ASUS Vivobook S 15 [3], NVIDIA Jetson Orin NX [32], and Qualcomm Snapdragon Dev Kit [36]. Since physical memory is shared with the operating system and other workloads, the entire capacity cannot be dedicated to inference. We therefore adopt a 12 GiB inference budget for 16 GB-class devices, leaving approximately 2.9 GiB of headroom. Within this budget, target weights, KV cache, runtime overhead, and drafter weights and context must all coexist.
| Platform (Processor) | RAM | Peak AI Compute |
| Microsoft Surface Laptop 7 (X Elite) | 16 GB | 45 TOPS |
| Apple Mac mini (M4) | 16 GB | 38 TOPS |
| ASUS Vivobook S 15 (X Elite) | 16 GB | 45 TOPS |
| NVIDIA Jetson Orin NX | 16 GB | 157 TOPS |
| NVIDIA Jetson Orin Nano Super | 8 GB | 67 TOPS |
| Qualcomm Snapdragon Dev Kit (X Elite) | 32 GB | 45 TOPS |
Table 7. Representative on-device AI hardware configurations. RAM denotes total physical memory, while peak AI compute is reported in tera operations per second (TOPS). Vendor-reported TOPS may reflect different compute units, numerical precisions, and sparsity settings and are not directly comparable. All specifications are taken from publicly available manufacturer documentation [33, 2, 3, 32, 36].
Synthetic retrieval diagnostic. For Figure 1b, we construct label-copy tasks by inserting eight randomly generated three-digit numbers into a prefix of irrelevant filler text. We fix the prefix length at 6,144 tokens and vary the evidence distance over tokens. We evaluate 47 jointly filtered prefixes with eight stochastic verification draws per prefix at temperature 1.0. All drafters use identical target-generated prefixes and anchors, and acceptance counts only accepted draft proposals.
Detailed Experimental Settings
In this section, we provide additional details on the drafter configurations, training procedures, and evaluation protocols used in our experiments.
Drafter configuration. We use two-layer drafters with a block size of for both Qwen3-4B and Phi-4-reasoning. We build HyFlash on DSpark [9], retaining its target-feature conditioning, parallel block drafting, and rank-256 Markov proposal head. We extract hidden features from five uniformly spaced target layers, spanning the second to the third-to-last layer. Adapted target features are provided to each drafter layer, while only draft-slot representations propagate between layers. We replace the attention mixer with our hybrid design combining SWA () and Mamba-2 (Dao and Gu, [12]). We retain the target model’s hidden size and feed-forward configuration while splitting the attention projection width between local attention and a Mamba-2 recurrent branch, keeping the overall parameter count approximately matched to the attention-only baselines. For Phi-4-reasoning, local attention uses 20 query heads and 5 KV heads with a head dimension of 128, while the recurrent branch uses 20 heads with a head dimension and state size of 128. We apply the same parameter-matching design to Qwen3-4B. The target remains frozen, while the feature adapter, drafter backbone, and proposal head are trained.
Baseline configuration. We compare against DFlash (Chen et al., [7]), DSpark (Cheng et al., [9]), and its SWA variant (). All methods use the same target model, proposal length, and speculative verification protocol.
Data preprocessing. Training conversations are tokenized using the corresponding target tokenizer and chat template, with a maximum sequence length of 8,192 tokens. We sample up to 512 valid assistant-token positions per conversation as drafting anchors, each conditioned only on preceding target features. Multiple blocks are processed in a single drafter forward pass. During training, we causally scan the adapted target features and extract the recurrent state immediately before each sampled anchor. All speculative positions within a draft block query this shared prefix state without updating it.
Optimization. We use AdamW (Loshchilov and Hutter, [31]) with a peak learning rate of , , , and weight decay of 0.1. We apply cosine decay with 4% linear warmup and gradient clipping at 1.0. Training uses BF16 weights, FP32 optimizer states, a global batch size of 512 conversations, and 10 epochs.
Training objective. Following Cheng et al. [9], we combine cross-entropy (CE) loss for ground-truth token prediction with total variation (TV) loss for matching draft and target distributions:
where and denote the draft and target distributions, respectively. We use , , and position weights (Chen et al., [7]).
Inference configuration. Unless otherwise specified, we use batch-one decoding without top- (Holtzman et al., [19]) or top- truncation. The proposal head samples tokens sequentially from a single backbone pass. For HyFlash, recurrent states are updated only with newly committed target features after verification and remain read-only during drafting. Rejected proposals do not modify the states.
Inference hardware and runtime. We measure batch-one decoding throughput on an NVIDIA RTX PRO 6000 Blackwell GPU using BF16 target and drafter weights. End-to-end speedup is measured against autoregressive decoding on the same hardware.
Model size and memory accounting. For Phi-4-reasoning, the two-layer HyFlash drafter contains 865.5M parameters, compared with 864.1M for the full-attention and SWA baselines. For Qwen3-4B, the corresponding counts are 303.3M and 312.4M. These counts exclude the input embedding and output vocabulary head, which are reused from the frozen target. The resulting BF16 weight footprints of HyFlash are 1,650.8 MiB and 578.5 MiB, respectively, with GPTQ 4-bit footprints of 465.9 MiB and 205.6 MiB (estimated for Qwen3-4B). In the Phi-4 configuration, local KV storage and FP32 recurrent states each require approximately 2.5 MiB, yielding a constant 5 MiB logical context footprint matching the SWA baseline. Both methods physically evict out-of-window KV entries. These estimates exclude target KV and temporary buffers, and do not represent measured peak memory.
Metric computation. For general benchmarks, average acceptance length () counts newly generated tokens per verification round, including eligible bonus tokens but excluding the initial anchor. Decoding speedup is the ratio of generated-token throughput to autoregressive decoding, including drafting, verification, and recurrent-state updates. For frozen-prefix evaluations, including synthetic retrieval (Figure 1) and LongBench (Table 2), we instead report the average number of accepted draft proposals at shared target-generated prefixes, excluding correction and bonus tokens. For aggregate results, we report the geometric mean of decoding speedups and the arithmetic mean of acceptance lengths across benchmarks.
Additional Experimental Discussion
Tree-based Drafting. We evaluate HyFlash under fixed and adaptive tree drafting with an identical verification budget of 29 nodes. The fixed strategy uses four depth-7 branches, while the adaptive strategy follows the confidence-guided tree construction of BASTION [35]. As shown in Table 8, HyFlash consistently improves both acceptance length and decoding speedup over DSpark under both strategies. These results suggest that the benefits of the prefix-memory prior extend beyond chain-based drafting to different tree construction strategies.
| Method | AIME25 | LCB | ||||||
| Fixed | Adaptive | Fixed | Adaptive | |||||
| Spd. | Spd. | Spd. | Spd. | |||||
| DSpark | 2.15 | 3.42 | 2.40 | 3.94 | 1.82 | 2.97 | 2.03 | 3.42 |
| HyFlash | 2.52 | 3.81 | 2.74 | 4.29 | 2.17 | 3.44 | 2.41 | 3.94 |
Table 8. Effect of tree drafting strategies on decoding speedup (Spd.) and average acceptance length (). Fixed uses width 4 and depth 7, while Adaptive follows BASTION [35].
Sensitivity to draft block size. We vary the inference-time draft block size over while using the same drafters trained with . As shown in Table 9, HyFlash consistently outperforms attention-only baselines in acceptance length and decoding speedup across different block sizes. Notably, HyFlash maintains its advantage at , beyond the training block size, suggesting that the prefix-memory prior remains effective for longer draft blocks.
| Method | ||||||||
| Spd. | Spd. | Spd. | Spd. | |||||
| DSpark | 1.88 | 2.73 | 2.12 | 3.10 | 2.22 | 3.28 | 2.18 | 3.27 |
| DSpark + SWA | 1.55 | 2.28 | 1.93 | 2.89 | 2.09 | 3.15 | 2.03 | 3.04 |
| HyFlash | 1.95 | 2.89 | 2.45 | 3.68 | 2.58 | 3.79 | 2.62 | 3.98 |
Table 9. Sensitivity to inference-time draft length. Average acceptance length () and decoding speedup for , using drafters trained with .
Sensitivity to sampling parameters. We evaluate HyFlash under greedy (), moderate (, ), and untruncated (, ) sampling, using the same trained drafter without modification. As shown in Table 10, greedy decoding achieves the highest acceptance length and speedup, while increasing sampling stochasticity reduces both metrics across all benchmarks. Nevertheless, HyFlash maintains decoding speedups under all sampling configurations, demonstrating its applicability across different sampling distributions.
| Method | MATH-500 | AIME25 | MBPP | LCB | ||||
| Spd. | Spd. | Spd. | Spd. | |||||
| Greedy | 3.35 | 4.86 | 2.90 | 4.29 | 4.09 | 5.77 | 3.13 | 4.69 |
| Moderate | 3.22 | 4.75 | 2.79 | 4.25 | 3.00 | 4.39 | 2.54 | 3.97 |
| Untruncated | 2.87 | 4.21 | 2.52 | 3.78 | 2.95 | 4.12 | 2.18 | 3.36 |
Table 10. Sensitivity to sampling parameters on decoding speedup (Spd.) and average acceptance length ().
Sensitivity to context length. We evaluate HyFlash under 4K and 16K context lengths while keeping the drafter architecture and training configuration unchanged. As shown in Table 11, acceptance length increases at 16K on MATH-500 and LCB but decreases on AIME25 and MBPP, while decoding speedup declines across all four benchmarks. As further examined in the latency analysis below, this speedup degradation is primarily associated with the increasing cost of full-attention target verification rather than drafter computation.
| Method | MATH-500 | AIME25 | MBPP | LCB | ||||
| Spd. | Spd. | Spd. | Spd. | |||||
| 4K | 2.87 | 4.21 | 2.52 | 3.78 | 2.95 | 4.12 | 2.18 | 3.36 |
| 16K | 2.00 | 5.46 | 1.41 | 2.95 | 1.64 | 3.69 | 1.78 | 4.15 |
Table 11. Sensitivity to context length on decoding speedup (Spd.) and average acceptance length ().
Latency analysis. To investigate the reduced decoding speedup at longer contexts, we profile the latency of speculative decoding at 4K and 16K. As shown in Figure 4, target verification latency on AIME25 nearly doubles from 44.76 to 87.80 ms, while draft-backbone latency remains nearly constant (2.95 vs. 3.02 ms). Unlike autoregressive decoding, which processes one query position per forward pass, speculative verification processes eight positions, incurring greater attention overhead as the context grows. Meanwhile, explicit KV-cache eviction and incremental recurrent-state updates keep the drafter computation nearly independent of context length. These results indicate that target verification becomes the dominant bottleneck at longer contexts.
Figure 4. Latency analysis at 4K and 16K context lengths, excluding prefill. (a) Per-round latency breakdown on AIME25. (b) Draft-backbone latency across four benchmarks.
References
- [1]Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, et al. 2025. Phi-4-reasoning technical report. arXiv preprint arXiv:2504.21318.arxiv.org/abs/2504.21318
- [2]Apple. 2024. Mac mini (2024): Tech specs. Apple Support. Accessed October 29, 2026. https://support.apple.com/en-us/121555.
- [3]ASUS. 2024. ASUS Debuts the Vivobook S 15, the Company’s First Copilot+ PC. ASUS News.
- [4]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.arxiv.org/abs/2108.07732
- [5]Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.arxiv.org/abs/1607.06450
- [6]Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A bilingual, multi-task benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119–3137. Association for Computational Linguistics.
- [7]Jian Chen, Yesheng Liang, and Zhijian Liu. 2026. DFlash: Block diffusion for flash speculative decoding. arXiv preprint arXiv:2602.06036.arxiv.org/abs/2602.06036
- [8]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
- [9]Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, Yunfan Xiong, Yi Qian, Jiaqi Zhu, Shirong Ma, Xiaokang Zhang, Jiasheng Ye, Qinyu Chen, Chengqi Deng, Jiping Yu, Damai Dai, Zhengyuan Zhang, Yixuan Wei, Yixuan Tan, Wenkai Yang, Runxin Xu, and 14 others. 2026. DSpark: Confidence-scheduled speculative decoding with semi-autoregressive generation. arXiv preprint arXiv:2607.05147.arxiv.org/abs/2607.05147
- [10]Daewon Choi, Seunghyuk Oh, Saket Dingliwal, Jihoon Tack, Kyuyoung Kim, Woomin Song, Seojin Kim, Insu Han, Jinwoo Shin, Aram Galstyan, Shubham Katiyar, and Sravan Babu Bodapati. 2025. Mamba drafters for speculative decoding. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 21404–21418. Association for Computational Linguistics.arxiv.org/abs/2506.01206
- [11]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.arxiv.org/abs/2110.14168
- [12]Tri Dao and Albert Gu. 2024. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 10041–10071. PMLR.arxiv.org/abs/2405.21060
- [13]Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4599–4610, Online. Association for Computational Linguistics.
- [14]Soham De et al. 2024. Griffin: Mixing gated linear recurrences with local attention for efficient language models. Preprint, arXiv:2402.19427.arxiv.org/abs/2402.19427
- [15]Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Mahabaleshwarkar, Shih-Yang Liu, Matthijs Van Keirsbilck, Min-Hung Chen, Yoshi Suhara, Yingyan Lin, Jan Kautz, and Pavlo Molchanov. 2025. Hymba: A hybrid-head architecture for small language models. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2411.13676
- [16]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. GPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations.arxiv.org/abs/2210.17323
- [17]Albert Gu and Tri Dao. 2024. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling.arxiv.org/abs/2312.00752
- [18]Ali Hatamizadeh, Yejin Choi, and Jan Kautz. 2026. Gated deltanet-2: Decoupling erase and write in linear attention. arXiv preprint arXiv:2605.22791.arxiv.org/abs/2605.22791
- [19]Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.
- [20]Jianuo Huang, Yaojie Zhang, Qituan Zhang, Hao Lin, Hanlin Xu, and Linfeng Zhang. 2026. Domino: Decoupling causal modeling from autoregressive drafting in speculative decoding. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 23346–23356, Budapest, Hungary. Association for Computational Linguistics.arxiv.org/abs/2605.29707
- [21]Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419–1436, Online. Association for Computational Linguistics.arxiv.org/abs/2104.02112
- [22]Inco AI. 2026. DFlash 2: Keep Drafting Parallel.
- [23]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2403.07974
- [24]Weifan Jiang, Krishna Teja Chitty-Venkata, Megan Flynn, Reed Meyerson, Zhenting Qi, Tianyu Wu, Eldar Kurtic, Minlan Yu, and Alexandre Marques. 2026. H-Spec: Parallel speculative decoding without a drafter-side KV cache. arXiv preprint arXiv:2609.24197v1.arxiv.org/abs/2609.24197
- [25]Barak Lenz, Opher Lieber, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, Daniel Gissin, Daniel Jannai, Dor Muhlgay, Dor Zimberg, Edden M. Gerber, Elad Dolev, Eran Krakowsky, Erez Safahi, Erez Schwartz, and 42 others. 2025. Jamba: Hybrid transformer-mamba language models. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2403.19887
- [26]Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19274–19286. PMLR.arxiv.org/abs/2211.17192
- [27]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024. EAGLE: Speculative sampling requires rethinking feature uncertainty. In Forty-first International Conference on Machine Learning.arxiv.org/abs/2401.15077
- [28]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2025. EAGLE-3: Scaling up inference acceleration of large language models via training-time test. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.arxiv.org/abs/2503.01840
- [29]Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2305.20050
- [30]Zechun Liu, Changsheng Zhao, Forrest Landola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, Liangzhen Lai, and Vikas Chandra. 2024. MobileLLM: Optimizing sub-billion parameter language models for on-device use cases. In Forty-first International Conference on Machine Learning.arxiv.org/abs/2402.14905
- [31]Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.
- [32]Shashank Maheshwari and Chen Su. 2025. NVIDIA JetPack 6.2 Brings Super Mode to NVIDIA Jetson Orin Nano and Jetson Orin NX Modules. NVIDIA Technical Blog. January 16, 2025. Accessed: 2026-11-04.
- [33]Microsoft. 2024. Surface Laptop 13.8-inch and 15-inch, 7th Edition: Fact Sheet. Microsoft Surface Product Specifications.
- [34]NVIDIA. 2026. Jetson orin: Module technical specifications. Official product specifications. Accessed October 29, 2026. https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/.
- [35]Soowon Oh, Nam Cao, Yujin Kim, Hojung Jung, Huzama Ahmad, Sangmin Bae, and Se-Young Yun. 2026. BASTION: Budget-aware speculative decoding with tree-structured block diffusion drafting. In AdapTFM: Resource-Adaptive Foundation Model Inference.arxiv.org/abs/2605.29727
- [36]Qualcomm. 2024. Snapdragon Dev Kit for Windows: Product Brief. Qualcomm Product Brief, Document 87-77834-1 Rev. C.
- [37]Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisht Tiwari, Ruihang Lai, Jinyuan Shi, Ian En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen. 2025. Magicdec: Breaking the latency-throughput tradeoff for long context generation with speculative decoding. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2408.11049
- [38]Bohan Sun, Zunhai Su, Xialie Zhuang, Jianjin Li, Ngai Wong, and Chuan-Wei Kuo. 2026. ReaDy: Recurrent states as dynamic context for speculative drafting in hybrid linear attention large language models. arXiv preprint arXiv:2610.20503v2.arxiv.org/abs/2610.20503


