betaFeedback

WorksComputer Science

Weights That Stay: On-Chip Megakernels for Serial Heads in Speculative Decoding

Abstract

Parallel speculative drafters use a lightweight serial head to recover token dependencies within a block generated in parallel. Each head step updates only a small state, yet projects that state through the same vocabulary-sized matrix. Step-wise execution repeatedly transfers this large, immutable projection, making the head a meaningful source of latency despite its low arithmetic cost. We present Weights That Stay, an execution design that keeps projection shards on chip for the entire serial draft walk. A persistent megakernel couples weight residency with GPU-side token selection and state advancement, reusing resident weights while exchanging only compact state between steps. The design supports full or partial residency and accommodates both Markov and recurrent heads on H100 and B200. Integrated into SGLang, Weights That Stay improves throughput for Markov drafting on Qwen3-4B and Qwen3-8B, with gains of up to 12.3%. At the head level, the DSpark walk is up to 11.6×\times faster than the fastest measured SGLang reference path. These results show that aligning weight residency with the full draft walk translates cross-step reuse into end-to-end serving gains.

Introduction

Speculative decoding accelerates language-model generation by drafting multiple tokens and verifying them in parallel with the target model [3] (Leviathan et al. 2023). Parallel drafters produce a block in one backbone pass, but predictions made without their realized predecessors can lose important intra-block dependencies. DSpark [6] and Domino [13] recover these dependencies with a lightweight serial head after the parallel backbone. Each head step uses tokens selected earlier in the block to refine the next proposal.

The serial head performs little arithmetic, but can still add substantial latency. Although its projection is low-rank, it spans more than 150,000 vocabulary rows and is larger than the H100 L2 cache. Reapplying this projection at every dependent step makes the head bandwidth-oriented at small batch. In SGLang, the head accounts for up to 12% of a Qwen3-4B speculative round. Low rank reduces the projection’s footprint, but does not eliminate its repeated movement along the serial path.

The root cause is a mismatch between the execution boundary and the lifetime of reusable data. The vocabulary projection is unchanged across the entire walk, while the cross-step dependency is carried by a much smaller token or recurrent state. Conventional operator boundaries confine the projection’s software-managed lifetime to one operation or step: registers and shared memory do not persist across kernel launches.

The same projection is used at every step, so its shards can remain resident throughout the walk. A whole-walk kernel exposes this opportunity but does not realize it automatically: it can still reload the same weights at every iteration. We therefore assign fixed row shards to a co-resident grid for the complete walk. Each CTA loads retained weight tiles once, scores the same vocabulary rows at every step, and exchanges only the compact information needed to advance the recurrence. This decouples traffic for the resident weights from the number of steps in the walk.

This organization creates three coupled design requirements. First, the projection needs a stable mapping at two levels: vocabulary rows to SMs, and each SM’s shard to the storage locations consumed by its matrix instructions. Second, global token selection and state advancement must complete without relinquishing that mapping. Third, the design must exploit partial fit when a shard modestly exceeds the operand storage left after reserving the execution workspace.

We realize this design in WEIGHTS THAT STAY, a family of persistent megakernels for serial draft heads (Figure 1). A GPU-side collective advances the recurrence while preserving fixed shard ownership. Resource-specific layouts provide full or partial residency on H100 and B200. The same execution structure supports heads with Markov, RNN, and GRU-based state updates.

Figure 1. Walk-resident execution across GPU memory tiers. Each SM retains a shard of WW throughout the walk, exchanging only tokens or recurrent state. Weights are int8 on H100 and bf16 on B200; optional loads stream nonresident tiles.

On H100, the DSpark walk is up to 11.6× faster than the fastest measured SGLang reference path. With block-16 Markov drafting, SGLang client throughput improves by up to 12.3% for Qwen3-4B and 8.6% for Qwen3-8B. Head-level results extend across Markov and recurrent heads and from H100 to B200.

This paper makes three contributions:

  • We characterize serial draft heads by a large immutable projection shared across steps and a small evolving state, and show why step-wise execution fails to exploit this temporal reuse (§ 2).

  • We develop walk-resident execution as a hierarchical tile-mapping design space, together with a GPU-side protocol that advances dependent steps without breaking the mapping (§ 3).

  • We implement the design on H100 and B200, achieving up to 11.6× head-level speedup across Markov and recurrent heads and 12.3% higher SGLang client throughput for Markov drafting (§ 4–§ 5).

Serial Draft Heads and Their Execution Mismatch

Speculative Decoding with Serial Draft Heads

Speculative decoding accelerates autoregressive generation by using a lightweight drafter to propose multiple future tokens and the target model to verify them in parallel ([3] (Leviathan et al. 2023)). In each round, the drafter proposes a token block and the target scores its positions in one pass. Greedy verification accepts the longest prefix that agrees with the target’s choices. For sampling, a proposed token yy drawn from the draft distribution qq is accepted with probability min⁡(1,p(y)/q(y))\min(1,p(y)/q(y)), where pp is the target distribution. On rejection, the replacement is sampled from a distribution proportional to max⁡(p−q,0)\max(p-q,0). This correction preserves the target distribution as long as drafting and verification use the same qq.

Drafter quality therefore affects efficiency rather than target correctness. Better proposals produce longer accepted prefixes, allowing each target pass to advance generation farther. Drafter latency and accepted length together determine the end-to-end benefit.

Parallel drafters reduce drafting latency by computing the base logits and slot features for all draft positions in one backbone forward pass. Because the positions are computed together, their base predictions cannot depend on the tokens actually selected at earlier positions in the block. The systems we study add a serial correction head. After selecting token yky_k, the head carries that token or a recurrent state into the next step to refine the next proposal. This restores intra-block dependence, but places a dependent head walk after the parallel backbone.

We use γ\gamma for the number of serial head steps executed in a walk, which can differ from the checkpoint’s nominal block size. Denote the backbone outputs for step kk by base logits bk∈RVb_k \in\mathbb{R}^{V} and slot features xkx_k, for k=1,…,γk=1,\ldots,\gamma. The relevant heads can be expressed as

uk=F(sk−1,xk),u_k = F(s_{k-1},x_k),
ℓk=bk+Wuk,\ell_k = b_k + Wu_k,
yk=Select⁡(ℓk),y_k = \operatorname{Select}(\ell_k),
sk=G(sk−1,xk,yk).s_k = G(s_{k-1},x_k,y_k).

The vocabulary projection W∈RV×rW \in\mathbb{R}^{V \times r} is fixed across steps (Figure 2). The latent uk∈Rru_k \in\mathbb{R}^{r} and state sks_k depend on the generated prefix. Selection is an argmax for greedy decoding or a sample from the temperature-scaled proposal distribution.

Figure 2. Serial refinement with a shared vocabulary projection (V=12V = 12, r=4r = 4, three steps). Each selected token advances the next step. Base-logit addition and explicit state-update nodes are omitted; the full recurrence is given in Equations 1–4.

For DSpark’s Markov head, sk−1s_{k-1} is the previous token and FF gathers its row from an embedding table. Domino (Huang et al., [13]) and DeLS-Spec (Zheng & Li, 2026) maintain recurrent state to summarize more of the prefix before the low-rank projection. Their state updates differ in how they compute uku_k, but retain the repeated vocabulary-wide operation in Equation 2. We target this shared structure. Purely parallel heads and methods whose serial phase is already confined to a small candidate lattice occupy a different regime.

Why a Lightweight Head Is Still Expensive

These heads are small relative to the backbone, but conventional execution reads their vocabulary projection once for every dependent step. At batch BB, the projection performs approximately 2BVr2BVr floating-point operations per step. If a weight uses pp bytes, reading the table transfers VrpVrp bytes per step and γVrp\gamma Vrp bytes over the walk. Ignoring other traffic, the arithmetic intensity is therefore about 2B/p2B/p operations per byte. At small batch, this is a bandwidth-oriented operation despite its large output dimension. The dependency in Equation 4 prevents evaluating the realized sequence of latents independently.

With r=256r = 256, the bf16 projection occupies 77.8 MB at V=151,936V = 151{,}936 and 127.1 MB at V=248,320V = 248{,}320. Both exceed H100 L2 capacity (Figure 3a). A conventional walk repeatedly streams the projection through the memory hierarchy, with additional gather, epilogue, and selection operations at every step. A profile of the larger-vocabulary SGLang head shows that the vocabulary projection takes 44.4 μ\mus per step and dominates its latency (Figure 3b).

Figure 3. Memory footprint and latency cost of serial draft heads. (a) Rank-256 projection size versus nominal GPU capacities (MB). (b) One SGLang step at B=1B=1, V=248V=248k under CUDA-graph replay. (c) Head microbenchmark latency divided by measured serving-round latency at C=1C=1, T=0T=0; DSpark uses the default folded path, and Domino uses the full-vocabulary head.

The round-level impact depends on the workload (Figure 3c). Longer walks repeat the head cost more often; larger targets increase verification time and dilute the same saving. Larger batches also increase target compute and change reuse across requests. Our evaluation therefore varies walk length, target size, and batch size, and reports both head-level and serving-level effects.

Aligning Execution with Reuse

The analysis above exposes a mismatch between algorithmic reuse and execution lifetime: the projection weights remain unchanged for the entire walk, but conventional operator execution retains each tile in registers or shared memory for only one projection. The next dependent step must stage the same tile again from L2 or HBM for a new latent.

Walk-resident execution reverses this dataflow. A persistent CTA retains a weight shard and brings each successive latent to it. The recurrence remains serial, but the dominant weights now outlive the step boundary. Section 3 develops the required shard mapping, state-advance protocol, and partial-residency path.

Walk-Resident Execution

Extending weight lifetime across a walk creates three problems: mapping the projection across SMs and their storage tiers, advancing a distributed recurrence without losing that mapping, and handling shards that do not fit alongside the execution workspace.

A Hierarchical Tile-Mapping Space

Per-step fusion reduces intermediate traffic and operator launches within each step. Fusing the entire walk further removes kernel boundaries between steps. The kernel can still repeat the same weight transfers at every iteration. We retain tiles across steps to reduce this recurring copy work. Figure 4 shows the weight reads and execution boundaries in these strategies.

Figure 4. Whole-walk fusion removes launch boundaries, while residency removes repeated fetches of WW from L2 or HBM (V=151,936V=151{,}936, B=1B=1, greedy). Timings are fitted from the complete-call measurements in § 5.4. The SGLang row is truncated after one step, and the within-step segments are illustrative. The resident kernel’s entry-cost excess over streaming is drawn as a one-time shard load.

Residency requires a stable mapping at two levels. Across SMs, we partition WW by vocabulary rows and assign one shard to a persistent CTA on each participating SM. A CTA owns the same token rows, spanning the full latent dimension, for the complete walk. Within an SM, the shard is divided among storage tiers, with instruction-specific paths to supply operands. Both mappings remain fixed while the latent and selected token change between steps.

Together, these choices define a hierarchical tile-mapping space. For each weight tile, a layout chooses its persistent SM owner, its storage tier within that SM, and the matrix instruction that consumes it. Figure 5 distinguishes copies between storage tiers from operand accesses by computation instructions. Copy instructions move tiles between tiers, whereas matrix instructions consume operands from prescribed locations ((NVIDIA Corporation, 2026)). A tile is resident when it remains in software-managed on-chip storage across dependent steps, avoiding repeated fetches from L2 or HBM. Residency does not eliminate all on-chip movement: shared-memory tiles may still need to be loaded into registers before computation.

Figure 5. Operand storage and data paths for GPU matrix instructions ((NVIDIA Corporation, 2026)). Thin arrows denote copies; thick arrows denote operand and result access. mma.sync uses registers; wgmma reads AA from RF/SMEM and BB from SMEM, accumulating in RF; tcgen05.mma reads AA from TMEM/SMEM and BB from SMEM, accumulating in TMEM.

The mapping therefore determines both the resident capacity and the recurring operand-staging cost. Making another tile resident removes its repeated fetch from L2 or HBM but consumes on-chip storage. Changing the matrix instruction changes the operand locations and staging paths, affecting both usable capacity and per-step cost.

A Distributed State-Advance Protocol

Fixed ownership distributes the projection: CTA jj produces scores only for the rows in its shard WjW_j. The recurrence, however, has one logical state, and no CTA can begin the next step until that state reflects a token selected from the full vocabulary. We therefore execute each step as a distributed state transition with four ordered phases: prepare, score, select, and commit.

In the prepare phase, the head-specific update exposes the latent components needed by every weight owner. Each CTA then scores its local rows without communication. The select phase computes one token from the distributed scores and publishes it to every CTA, using an argmax for greedy decoding or categorical selection for sampling. Finally, the commit phase applies the head-specific state update and establishes the state for the next step. No CTA enters the next prepare phase before commit completes. Weight ownership remains unchanged across all four phases. Only the selected token and required state fragments cross CTA boundaries.

This protocol permits different state layouts. A Markov head needs no state exchange beyond selection: every CTA uses the common token to gather the next latent locally. A replicated recurrent update is likewise recomputed by every CTA, while a sharded update adds collectives for the required state fragments during prepare or commit. § 4.2 gives the concrete mappings, and § 5.5 measures their critical paths.

The collectives impose a progress condition. If waiting CTAs occupy resources needed by CTAs that have not been scheduled, the grid can deadlock. We therefore use a co-operative launch with one CTA on each participating SM so that every weight owner remains live for the complete walk. § 4.2 describes the implemented exchange.

When the Shard Does Not Fit

The mapping in § 3.1 must share each SM with the execution workspace: accumulators, the current latent and state, selection scratch space, and buffers for data movement. After reserving this workspace, the remaining operand storage may not hold the complete weight shard. Larger vocabularies increase the shard, while larger batches and more complex state updates increase the workspace.

We use two complementary responses to this capacity gap. A compact weight representation reduces the bytes in every row and can make the complete shard resident. If the shard still does not fit, the layout retains a fixed subset of tiles and streams the rest from L2 or HBM at every step. These choices can be combined: representation determines the shard size, while the resident fraction determines how much recurring traffic remains.

Resident tiles, staging buffers, and execution state compete for on-chip storage. A larger resident subset removes recurring reads but leaves less buffering to overlap streamed transfers or less workspace for batched execution and state. § 4 instantiates these choices on H100 and B200.

GPU Kernel Realization

We instantiate walk-resident execution by mapping each weight shard to GPU operand storage and advancing the recurrence without relinquishing that mapping.

Weight Placement and Execution

The mapping in § 3.1 must satisfy the operand locations of its matrix instructions after reserving the workspace identified in § 3.2.1. Each implementation therefore chooses a weight representation, an on-chip placement, and a schedule for any tiles that remain outside that placement. The resulting paths are summarized below.

CaseRFSMEMTMEMStreamed per step
H100 int8, small BB768384––
H100 int8, large BB–704–448
H100 int8, large VV–704–1,216
B200 bf16, standard VV336224480–
B200 bf16, large VV336264440640

Table 1.

H100 weight representation and execution. The standard bf16 projection occupies 77.8 MB, already exceeding H100’s 65.4 MB aggregate register and shared-memory capacity before reserving execution workspace. Full residency therefore requires a one-byte weight format. Both int8 and FP8 provide this footprint, but allocate their bits differently. After per-row scaling, the inspected weight rows are dense and approximately bell-shaped, without the heavy tails that would require a wide exponent range (Figure 6). Int8 distributes its levels uniformly across the bounded row range, whereas FP8 E4M3 spends bits on exponent range and has coarser spacing at the larger magnitudes that contribute most to projection error. We therefore store WW in row-scaled int8, reducing the table to approximately 38.9 MB before scale metadata and making full residency feasible.

Figure 6. Cumulative distributions of normalized weight magnitudes (band: two released 4B heads) and representable positive values (quantization levels). Left of the dashed line, E4M3 has finer spacing than int8 but places 75% of its levels in a region containing only 12–17% of the weights.

To avoid dequantizing the resident weights at every step, we execute the projection with native int8 tensor-core instructions. These instructions require the latent operand to be int8 as well. Each request moves only one rank-256 latent per step, so carrying a second int8 vector adds little traffic. We spend this small extra traffic to retain more latent precision, encoding uu in two int8 byte planes. The high plane qhq_h encodes the coarse value, while the low plane qlq_l encodes its residual at one 256th of the high-plane scale,

u≃shqh+slql,sl=sh/256.(1)u \simeq s_h q_h + s_l q_l,\qquad s_l = s_h/256. \tag*{(1)}

For a weight row Wi≃aiqiW_i \simeq a_i q_i, the projection becomes

Wiu≃ai(sh⟨qi,qh⟩+sl⟨qi,ql⟩).(2)W_i u \simeq a_i\left(s_h\langle q_i,q_h\rangle+ s_l\langle q_i,q_l\rangle\right). \tag*{(2)}

Both dot products now use int8 tensor cores without expanding the weight tiles to floating point at each step. We place qhq_h and qlq_l in adjacent columns of the MMA operand, allowing the kernel to compute their projections together. Each request contributes one such column pair. The int8 dot products accumulate in int32, after which the epilogue applies the two scales, sums the results, and adds the base logits (Figure 7).

Figure 7. Vocabulary projection using int8 tensor cores on H100. Two int8 vectors represent the latent uu and are multiplied by WW together. Their scaled results are summed, added to the base logits, and rounded once to bf16.

The resulting int8 operand tiles are split between registers and shared memory within each CTA’s fixed shard. Register-resident tiles remain in mma.sync operand fragments; shared-memory tiles are loaded into operand registers when used. Both remain on chip across steps. The table above gives the row allocation after reserving execution workspace.

Partial residency on H100. When the H100 shard does not fit for a given batch size and vocabulary size, the implementation selects a partial-residency kernel. Both its

resident weights and the tiles delivered by TMA reside in shared memory, so the kernel uses wgmma to consume them there directly while keeping the latent in registers. Shared memory is divided between a fixed set of resident tiles and staging space that TMA refills at each step; both sets of tiles then follow the same wgmma path.

B200 layouts. B200 adds tensor memory (TMEM) as an on-chip operand store accessed through the tcgen05 instruction family. Combining TMEM with register and shared-memory placement provides enough operand capacity to keep the standard-vocabulary projection in bf16. We divide each CTA’s row shard among these three locations: TMEM-resident tiles use tcgen05, while the other tiles use mma.sync. The ownership of vocabulary rows remains unchanged.

For the larger vocabulary, B200 uses the same partial-residency organization as H100: a fixed subset of bf16 tiles remains on chip, while TMA stages the rest in shared memory at each step.

Selection and State Advancement

The projection leaves each CTA with scores only for the vocabulary rows it owns. Advancing the walk requires all CTAs to agree on one token and then materialize the state or latent consumed by the next projection.

Cross-CTA token selection. Each CTA first reduces its rows to a local score and token identifier. It packs them into a 64-bit key whose score bits preserve numerical order and whose inverted token bits break ties toward the smaller identifier. An integer maximum over these keys therefore implements the desired global selection. One thread in each CTA atomically updates the shared maximum and then atomically increments an arrival counter (Figure 8). The increment uses release ordering so that the key update precedes the CTA’s arrival. The cooperative grid keeps every contributing CTA resident while they wait.

Figure 8. Cross-CTA token selection. Each CTA scores its rows ❶, packs its local winner ❷, atomically updates the maximum key and arrival count ❸, waits for all CTAs ❹, and uses the common winner ❺. The {key, count} observation uses the hardware assumption stated in the text.

After contributing, each CTA waits until the arrival count reaches the number of participating CTAs and then reads the maximum key. Our H100 and B200 kernels combine these two operations by polling the aligned 16-byte {key, count} pair with one relaxed vector load. When a load returns the final count, the CTA uses the key returned by the same load, avoiding a dependent L2 round trip. This relies on behavior observed on both GPUs: the aligned load sees both fields from one L2 snapshot. PTX (NVIDIA Corporation, 2026) does not guarantee this property for vector loads. A portable implementation can instead poll the count with volatile loads until completion, execute an acquire fence, and then load the key separately. This increases measured walk latency by 6.7%.

Greedy and stochastic selection. The final bf16 logits from the epilogue define the proposal distribution used by both selection and verification. Greedy requests pack these logits directly into the keys above.

For stochastic selection, we use two standard implementations of categorical sampling. Inverse-CDF sampling draws one uniform variate and locates it in the cumulative probability mass. Gumbel-max instead adds independent Gumbel noise to each log mass and selects the maximum, turning categorical sampling into a reduction. Batched sampling combines the two methods in two levels. Each eight-row group first produces a sample conditioned on that group, and a second sample selects one group according to its total probability mass. At the first level, one thread holds all eight logits, so inverse CDF needs only one uniform variate and a register-local prefix sum. Gumbel-max would generate and transform one variate per row. At the second level, group masses are distributed across threads and CTAs, so inverse CDF would require a global prefix scan. Gumbel-max over the log masses instead uses the same maximum reduction as greedy selection. The selected group returns its conditional sample, yielding a sample from the original categorical distribution. Philox (Salmon et al., 2011) counters key the noise by request, vocabulary row, step, and graph replay, so repeated graph executions draw fresh samples.

From the selected token to the next latent. A Markov head maps the selected token directly to the next latent, so token selection is its only collective. DeLS-Spec and Domino instead follow

hk=GRU⁡(E[yk−1],hk−1),h_k = \operatorname{GRU}(E[y_{k-1}], h_{k-1}),
uk=Pk(hk),ℓk=bk+Wuk.(3)u_k = P_k(h_k), \qquad\ell_k = b_k + W u_k. \tag*{(3)}

Here hkh_k is the recurrent history, uku_k is the low-rank input to the vocabulary projection WW, and PkP_k denotes the head-specific projection from one to the other.

We precompute the GRU input contribution for every token and shard the recurrent matrix across CTAs. While token selection is in flight, each CTA computes the part that depends only on the previous hidden state. Once the token is known, it looks up the corresponding precomputed input and finishes its slice of the new hidden state. The following latent projection needs the complete hidden state, so the CTAs write their slices to a global-memory buffer and wait until all contributors have arrived. Every CTA then reads the complete hidden state.

The new hidden state must then be mapped to the latent consumed by the vocabulary projection. The matrix in PkP_k is also row-sharded, so each CTA produces only a slice. Every vocabulary-weight owner needs the complete latent to score its rows, so the CTAs write their latent slices to the buffer and synchronize a second time. Every weight owner then reads and quantizes the complete latent. The first gather therefore supplies the hidden state to the latent projection, and the second supplies the latent to the vocabulary projection. When an update matrix is small enough to replicate in every CTA, each CTA can compute the complete latent locally and needs no gather. The DSpark RNN instead produces latent slices and gathers them once. Figure 13b separates this state-update cost from the shared projection.

Figure 13. Latency and per-step breakdown across serial heads on H100 (B=1B = 1, greedy). (a) Complete calls: Markov runs 16 steps, recurrent heads 15; labels give speedup over each head’s reference. *DSpark RNN uses random recurrent weights. (b) Disjoint per-step timings, excluding entry cost. The fixed-latent probe repeats projection and selection with an unchanged latent.

In SGLang (Zheng et al., 2024), the H100 Markov kernel replaces only the serial draft walk within the existing draft-and-verify round; backbone execution and target verification remain unchanged.

Evaluation

Experimental Setup

Platforms. We use NVIDIA H100 SXM GPUs (80 GB, 132 SMs) and B200 GPUs (148 SMs) with CUDA 13. H100 is the primary platform, and each serving configuration runs on a single H100. The B200 experiments evaluate head execution only, with reference and optimized paths measured in separate B200 containers.

Workload. The main serving experiments use 3,030 prompts from nine math, code, and conversation datasets: GSM8K [9], MATH500 (Lightman et al., 2023), AIME25 [10], HumanEval [5], MBPP [2], LiveCodeBench [14], MT-Bench (Zheng et al., 2023), Alpaca (Taori et al., 2023), and Arena-Hard-v2 (Li et al., 2024), with at most 500 prompts per dataset drawn with a fixed seed. We apply the Qwen3 chat template with thinking disabled, allow normal EOS termination, and cap generation at 2,048 new tokens. At concurrency CC, the client keeps CC requests in flight over one queue of all prompts. The Domino serving check and acceptance experiments use a fixed 20% per-dataset subset (605 prompts).

Baselines and measurement. Markov serving compares client throughput against default SGLang, the same build with the optimized walk disabled. Both systems run back to back on the same GPU, and runs affected by one-time kernel compilation are discarded. Both include an upstream SGLang fix for draft-attention boundaries at batch sizes above one, which affects the 8B drafter. Markov head measurements use the fastest measured SGLang path per configuration. Recurrent-head references are SGLang’s DSpark RNN module, the released DeLS-Spec graph runner, and SGLang’s full-vocabulary Domino rollout with candidate pruning disabled. The Domino serving check uses this same full-vocabulary baseline.

Head-level measurements execute captured CUDA graphs and include the inputs and outputs required by the caller. Before each measurement, we flush L2 and refresh the base logits, excluding this preparation from the reported latency.

Serving Speedup

Figure 9 reports throughput with Qwen3-4B-DSpark-b16 (shreybirmiwal, 2026) and Qwen3-8B-Markov-b16 (Wang et al., 2026). Gains peak at 12.3% and 8.6%, respectively. The optimized walk improves throughput in every measured configuration, but the gain decreases at high concurrency. Larger decode batches reduce the relative head-level advantage, as discussed in § 5.3.

Figure 9. End-to-end throughput gains for Qwen3-4B and Qwen3-8B over default SGLang: block-16 Markov drafting on H100, using all 3,030 prompts.

The gain also decreases when the target grows from 4B to 8B. The two configurations use the same vocabulary size, rank, and walk length, so the optimized head removes a similar amount of work. Target-model computation is more expensive for 8B, making the saved head time a smaller fraction of end-to-end generation. This single-GPU trend may reverse under tensor parallelism, where target computation is partitioned but the head remains replicated on each rank and can occupy a larger share of the critical path.

Head-Level Speedup

Figure 10 measures the complete Markov head call while varying batch size, walk length, vocabulary, and sampling mode. Longer walks reuse the resident projection for more steps and therefore yield larger gains, reaching 11.6×11.6\times at B=1B = 1. Sampling adds epilogue work to both implementations without changing this trend.

Figure 10. Markov-head latency across batch sizes, walk lengths, and vocabularies on H100. Top: T=0T = 0; bottom: T=1T = 1. Latency uses a log scale; labels give speedup over the fastest measured SGLang path. The 152k path is fully resident at B≤4B \le4 and partially resident above; the 248k path is always partially resident.

For the standard-vocabulary panels, increasing BB has two effects. Accumulators and per-request state leave less on-chip capacity for resident weights, so more of the projection must be streamed. At the same time, each fetched weight serves more requests, increasing the arithmetic intensity of the reference projection. Both effects narrow the relative advantage of residency.

We additionally study the larger-vocabulary head from the Qwen3.6-27B DSpark checkpoint. Its 248,320-row vocabulary projection does not fit on chip even at B=1B = 1, so the kernel retains a subset and fetches the rest at every step. This fixed transfer is about 41 MB per step, independent of BB, and limits the B=1B = 1 speedup to 4.4×4.4\times relative to the 127 MB bf16 reference. As BB grows, the transfer is amortized across requests while the reference’s vocabulary-wide per-request work grows, increasing the speedup until computation and selection costs become significant at the largest batches.

Ablations

Fusion and residency. At V=151,936V = 151{,}936 and B=1B = 1, Figure 11 places three int8 implementations against SGLang: per-step kernels, a whole-walk kernel that streams the projection each step, and the resident whole-walk kernel. Fusion reduces the fitted per-step cost by 1.49×1.49\times; retaining the weights reduces it by another 2.53×2.53\times. Thus a single launch alone does not remove the repeated projection traffic.

Separating fusion and residency gains on H100

Figure 11. Separating fusion and residency gains on H100 (B=1B = 1, V=152V = 152k, greedy). Points measure complete calls; least-squares fits are labeled by their slopes in µs/step. The axis break spans 210–260 µs.

RF/SMEM placement. The fully resident path keeps the same 72 weight tiles per CTA while varying their placement between the register file and shared memory. More RF tiles eliminate recurring shared-memory operand loads, but leave fewer registers for execution workspace and eventually cause spills. Figure 12 shows the resulting tradeoff. Per-step latency falls as tiles move into RF, reaches a shallow minimum at the 5:4 and 6:3 splits, and rises at either extreme. At 6:3, the additional RF-resident tiles consume the registers needed to double-buffer shared-memory operands, so the implementation uses a single buffer.

RF/SMEM placement tradeoff on H100

Figure 12. RF/SMEM placement tradeoff on H100 (B=1B = 1, greedy), with 72 weight tiles per CTA. Points are per-step costs fitted from complete calls at γ=1,7,16\gamma= 1, 7, 16. Shading marks register spills.

Different Serial Recurrences

Figure 13 evaluates how the execution structure changes when the selected token advances a recurrent state. We use released heads for DeLS-Spec and Domino. DSpark defines an RNN recurrence but does not release a trained checkpoint for it. Our DSpark RNN experiment therefore uses randomly initialized recurrent weights. The Markov head achieves 11.6×11.6\times at γ=16\gamma= 16. At γ=15\gamma= 15, DSpark RNN, DeLS-Spec, and Domino achieve 8.9×8.9\times, 8.2×8.2\times, and 8.5×8.5\times, respectively, against their full-vocabulary references.

Figure 13b shows that projection, epilogue, and token selection remain nearly unchanged across heads. The added cost follows the number of cross-CTA round trips. The fixed-latent probe communicates across CTAs only to select a token. Updating the recurrent state adds one gather for DSpark RNN and two for DeLS-Spec and Domino. Every gather publishes CTA-local slices, waits for all contributors, and then reads the assembled state through L2, so much of its cost comes from global synchronization and the L2 round trip rather than recurrent arithmetic. This explains why DeLS-Spec and Domino have nearly identical per-step latency despite different update equations: both perform three cross-CTA exchanges per step. Once the vocabulary projection is resident, these exchanges become the main remaining cost.

As a serving check for a recurrent head, the Qwen3-4B Domino round falls from 6.68 to 5.72 ms against exact full-vocabulary SGLang.

B200 Capacity Regimes

The B200 implementation tests the same organization with bf16 weights and a different set of operand-storage resources. The 152k projection is fully resident, while the 248k projection keeps about two-thirds of its rows on chip and streams the remainder. Figure 14 separates these capacity regimes.

Figure 14. B200 bf16 head latency under full (152k) and partial (248k) residency (B=1B = 1, greedy).

B200’s larger L2 also strengthens the reference path. The 152k projection fits comfortably in L2, allowing SGLang to reuse it after the first step. At 248k, our kernel streams only the nonresident portion, while SGLang processes the complete larger projection. Partial residency is therefore slower in absolute time than full residency, but retains a similar relative gain because the large-vocabulary reference is also slower.

Numerical Effects and Acceptance

The H100 implementation changes the representation of the vocabulary projection. Consistent with Figure 6, row-scaled int8 gives 0.58% Frobenius weight error and 0.26% projection error across two released 4B heads ([6] (UIUC-SSAIL 2026)) and 4,096 sampled latents. E4M3 with the same per-row scaling gives 2.64% and 1.20%, respectively.

Figure 15 shows no systematic acceptance change across the two walk lengths and temperatures. Every estimate lies near zero, and every confidence interval includes zero. Quantization can therefore change individual draft proposals without measurably changing acceptance on this workload.

Figure 15. Effect of int8 execution on Qwen3-4B draft acceptance length relative to default SGLang. Estimates pool runs across concurrency settings; lines give 95% prompt-cluster bootstrap intervals on 605 prompts.

Related Work

Speculative drafting and causal refinement. Blockwise decoding ((Stern et al., 2018)) and speculative sampling ([3] (Leviathan et al. 2023)) reduce the cost of sequential target generation. Parallel drafting, including DFlash ([4]), shifts computation across proposed positions into a parallel backbone. DSpark ([6]), Domino ([13]), and DeLS-Spec ((Zheng & Li, 2026)) retain lightweight prefix-dependent refinement. Earlier dependent-head designs such as Hydra ([1]) and Re-Drafter ([8]) also expose serial structure, but their weight matrices can exceed the on-chip capacity targeted by our design.

Persistent execution and weight reuse. Persistent RNNs (Diamos et al., [11]) established the use of persistent GPU kernels to retain recurrent weights across time steps. Sparse Persistent RNNs (Zhu et al., 2018) and GRNN (Holmes et al., [12]) further explored on-chip recurrent execution. Our contribution builds on this established idea. The draft-head setting combines low-rank vocabulary projection with per-step discrete selection and optional sampling. Its implementation must jointly account for batch-dependent capacity and distributed state updates. We evaluate these requirements across multiple head structures and two GPU memory hierarchies.

Megakernels and execution abstractions. MPK (Cheng et al., [6]) represents tensor programs as SM-level tasks and schedules them within persistent megakernels; Event Tensor (Jin et al., [15]) expresses dependencies for dynamic megakernel compilation. We focus on retaining a weight table over a specific data-dependent recurrence and measuring residency beyond fusion.

Serving and selection primitives. SGLang (Zheng et al., 2024) and vLLM (Kwon et al., [16]) provide the scheduling and memory-management context for efficient language-model serving. Our integration replaces a local draft-head execution path within that context. GPU sampling kernels in FlashInfer (Ye et al., 2025) and FlashSampling (Ruiz et al., 2026) optimize categorical selection. Here selection is also the synchronization boundary between dependent steps, so its placement inside the resident recurrence is central. The optimized head must expose the proposal distribution actually used by selection to preserve exact speculative sampling.

Conclusion

Serial draft heads combine a small evolving state with a large projection reused across dependent steps. WEIGHTS THAT STAY retains weight shards throughout the walk and advances the recurrence through GPU-side collectives, supporting Markov and recurrent heads with full or partial residency. Experiments show gains beyond fusion and higher serving throughput, demonstrating the value of aligning GPU execution with the lifetime of reusable weights.

AI Usage Disclosure

We used Anthropic Claude Code and OpenAI Codex iteratively to assist with literature search and reference checking, experiment and ablation planning, code implementation and debugging, analysis of measured results, and manuscript drafting, restructuring, and revision. Code assistance covered GPU kernels, benchmarking and profiling scripts, and data-analysis and plotting scripts. The tools also assisted with figure preparation and consistency checks among notation, claims, and reported values. The reported performance results come from executed GPU experiments. The authors directed the work, retained final control over research decisions, and take responsibility for the experimental methodology, validation of implementations and measurements, interpretation of results, references, and final manuscript.

References

  1. [1]Anker, Z., Parthasarathy, R., Nrusimha, A., Rinard, C., Ragan-Kelley, J., and Brandon, W. Hydra: Sequentially-dependent draft heads for medusa decoding, 2024. URL https://arxiv.org/abs/2402.05109.
  2. [2]Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., and Sutton, C. Program synthesis with large language models, 2021. URL https://arxiv.org/abs/2108.07732.
  3. [3]Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling, 2023. URL https://arxiv.org/abs/2302.01318.
  4. [4]Chen, J., Liang, Y., and Liu, Z. Dflash: Block diffusion for flash speculative decoding, 2026. URL https://arxiv.org/abs/2602.06036.
  5. [5]Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code, 2021. URL https://arxiv.org/abs/2107.03374.
  6. [6]Cheng, X., Yu, X., Shao, C., Li, J., Xiong, Y., Qian, Y., Zhu, J., Ma, S., Zhang, X., Ye, J., Chen, Q., Deng, C., Yu, J., Dai, D., Zhang, Z., Wei, Y., Tan, Y., Yang, W., Xu, R., Wu, Y., Xu, Z., Wang, X., Chen, M., Tian, R., Bi, X., Hao, Z., Chen, S., Cao, H., Zhang, W., Xu, A., Zhang, H., Zhao, D., and Liang, W. Dspark: Confidence-scheduled speculative decoding with semi-autoregressive generation, 2026. URL https://arxiv.org/abs/2607.05147.
  7. [8]Cheng, Y., Zhang, A., Zhang, X., Wang, C., and Wang, Y. Recurrent drafter for fast speculative decoding in large language models, 2024. URL https://arxiv.org/abs/2403.09919.
  8. [9]Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168.
  9. [10]Dekoninck, J., Jovanović, N., Gehrunger, T., Røgnvaldsson, K., Petrov, I., Sun, C., and Vechev, M. Beyond benchmarks: Matharena as an evaluation platform for mathematics with llms, 2026. URL https://arxiv.org/abs/2605.00674.
  10. [11]Diamos, G., Sengupta, S., Catanzaro, B., Chrzanowski, M., Coates, A., Elsen, E., Engel, J., Hannun, A., and Satheesh, S. Persistent rnns: Stashing recurrent weights on-chip. In Balcan, M. F. and Weinberger, K. Q. (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 2024–2033, New York, New York, USA, 20–22 Jun 2016. PMLR. URL https://proceedings.mlr.press/v48/diamos16.html.
  11. [12]Holmes, C., Mawhirter, D., He, Y., Yan, F., and Wu, B. Grnn: Low-latency and scalable rnn inference on gpus. In Proceedings of the Fourteenth EuroSys Conference 2019, EuroSys ’19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450362818. doi: 10.1145/3302424.3303949. URL https://doi.org/10.1145/3302424.3303949.
  12. [13]Huang, J., Zhang, Y., Zhang, Q., Lin, H., Xu, H., and Zhang, L. Dominio: Decoupling causal modeling from autoregressive drafting in speculative decoding, 2026. URL https://arxiv.org/abs/2605.29707.
  13. [14]Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024. URL https://arxiv.org/abs/2403.07974.
  14. [15]Jin, H., Hou, B., Wang, G., Lai, R., Chen, J., Ye, Z., Cai, Y., Dong, Y., Cheng, X., Zhang, Z., Zhao, Y., Huang, Y., Yang, L., Jiang, J., Oliaro, G., Ji, J., Miao, X., Grover, V., Mowry, T. C., Jia, Z., and Chen, T. Event tensor: A unified abstraction for compiling dynamic megakernel. In Ninth Conference on Machine Learning and Systems, 2026. URL https://openreview.net/forum?id=PJqFgFhABuHA.arxiv.org/abs/2604.13327
  15. [16]Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, pp. 611–626, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400702297. doi: 10.1145/3600006.3613165. URL https://doi.org/10.1145/3600006.3613165.

Paper details

vlnc:W_8qy27kf3
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude Code, Codex

View statement
Looking for related papers…
Contents