betaFeedback

WorksComputer Science

Split-KV: Efficient Context Parallelism for Training Diffusion Drafters

Abstract

Diffusion drafters accelerate speculative decoding by denoising multiple draft tokens in parallel. Their training uses anchor sampling to construct compact draft queries and performs cross-attention with a block-sparse mask over a much longer context. We empirically show that acceptance length remains stable as context scales with a fixed query budget, while increasing the query budget yields diminishing quality gains, jointly establishing a practical regime with queries far shorter than the context. No existing context parallelism (CP) method jointly exploits this query–context asymmetry and meets the nonlinear, sample-dependent load-balancing requirement created by anchor sampling. We present Split-KV, an exact CP method that keeps key/value shards local and limits per-layer attention communication to query-sized tensors, making this communication independent of context length. Its global query view further enables a cost-aware longest-processing-time-first (LPT) strategy that balances context key/value chunks by their realized attention cost. At a CP size of 64, Split-KV retains 85–91% weak-scaling efficiency and achieves up to 3.67×3.67\times speedup over the fastest available CP baseline. LPT achieves <<1% imbalance and plans three orders of magnitude faster than the best load-balancing baseline. Code is available at https://github.com/zxgx/verl.

main_arxiv.pdf

Introduction

Speculative decoding accelerates large language model inference by using a lightweight drafter to propose multiple tokens and verifying them in one target-model forward pass (Leviathan et al., 2023; Chen et al., 2023). Diffusion drafters generate proposals by denoising a block of tokens in parallel, avoiding the sequential forward passes required by autoregressive drafters (Arriola et al., 2025; Christopher et al., 2025; Li et al., 2026a). Diffusion drafter training uses anchor sampling to construct compact draft queries and performs cross-attention with a block-sparse mask over a potentially much longer context.

Diffusion drafter training decouples the attention query length QQ, determined by the anchor budget, from the context length SS, determined by the training samples. However, how the relationship between QQ and SS affects drafter quality is still underexplored: scaling SS could require proportionally more queries to preserve acceptance length, eliminating the systems benefit of this separation. Our controlled SFT studies instead show that acceptance length remains stable as context scales with a fixed query budget, while increasing the query budget yields diminishing quality gains. Together, these results establish a practical Q≪SQ \ll S regime for long-context diffusion drafter training.

Existing context parallelism (CP) methods do not fully exploit this training regime. Ulysses, Ring Attention, and RingX retain attention communication that grows with SS, even when QQ remains fixed (Jacobs et al., 2024; Liu et al., 2024; Zhu, 2024; Yin et al., 2025). LV-XAttn keeps key/value tensors stationary and makes its attention communication independent of context length, but streams query-side state through a sequential hop loop and preserves contiguous key/value shards (Chang & Venkataraman, 2025). Thus, context-independent attention communication alone does not address the distinct load-balancing requirement of diffusion-drafter training.

Anchor sampling also changes how attention work is distributed. Unlike causal self-attention, whose per-position cost follows a regular linear profile, sampled anchors produce a nonlinear and sample-dependent context-access profile. Zigzag balances causal self-attention by pairing mirrored positions (Zhu, 2024; Yin et al., 2025), but this position-based rule cannot capture anchor-induced context access. Dynamic Context Parallelism (DCP) handles general attention masks through hypergraph partitioning but incurs substantial planning overhead, while RingX and LV-XAttn provide no load-balancing mechanism (Jiang et al., 2025; Yin et al., 2025; Chang & Venkataraman, 2025). Ex-

Preprint. isting CP methods therefore either balance this workload inaccurately, plan it expensively, or do not balance it at all.

We introduce Split-KV to address both limitations. Split-KV keeps key/value shards local throughout the forward and backward passes and limits per-layer attention communication to query-sized tensors. It exactly reconstructs the original attention while making this communication independent of context length. The resulting global query view decouples context key/value work from query ownership, enabling a cost-aware longest-processing-time-first (LPT) strategy that assigns key/value chunks by their realized attention cost.

Our contributions are summarized as follows:

  • We show that acceptance length remains stable as context scales with a fixed query budget, while increasing the query budget yields diminishing quality gains, jointly establishing a practical Q≪SQ \ll S regime.

  • We design Split-KV, an exact CP method that keeps key/value shards local and limits per-layer attention communication to query-sized tensors, making this communication independent of context length.

  • We design a cost-aware LPT strategy that uses Split-KV’s global query view to balance the nonlinear, sample-dependent attention work.

  • At a CP size of 64, Split-KV retains 85–91% weak-scaling efficiency and achieves up to 3.67× speedup over the fastest available CP baseline; LPT achieves < 1% imbalance and plans three orders of magnitude faster than the best load-balancing baseline, DCP.

Related Work

Speculative Decoding with Learnable Drafters

Speculative decoding accelerates a target model by generating a multi-token draft at low cost and verifying it in one target-model forward pass, thereby alleviating memory-bound decoding (Leviathan et al., 2023; Chen et al., 2023). Extensive work studies autoregressive drafters, including the EAGLE family (Li et al., 2026b) and MTP (DeepSeek-AI et al., 2025). These drafters follow the target-model architecture but use only a few layers to reduce draft-generation cost. Diffusion drafters instead denoise a block of draft tokens in each forward pass. DFlash [3] adopts a block-diffusion architecture (Arriola et al., 2025) and uses fewer drafter layers than DiffuSpec (Li et al., 2026a) and SpecDiff (Christopher et al., 2025; Sandler et al., 2026) while retaining high acceptance length. Domino (Huang et al., 2026) and DSpark (Cheng et al., 2026) modify the dependency structure within diffusion blocks to improve acceptance length, but retain the same anchor-based block-diffusion training pattern targeted by Split-KV.

Context Parallelism

Context parallelism (CP) distributes long sequences across devices to reduce per-device computation and memory at the cost of additional communication. Sequence parallelism distributes sequence-dependent activations across devices (Korthikanti et al., 2023). Ulysses uses all-to-all communication to exchange sequence shards for attention-head shards (Jacobs et al., 2024). Although Ulysses balances attention work across devices, its CP size is limited by the model’s attention-head count. Ring Attention instead partitions the sequence dimension and circulates key/value shards during local attention computation (Li et al., 2023; Liu et al., 2024). Zigzag Ring Attention pairs chunks at mirrored sequence positions to balance the triangular workload induced by causal self-attention (Yin et al., 2025; Zhu, 2024). Dynamic Context Parallelism (DCP) uses hypergraph partitioning to balance dynamic attention workloads while accounting for communication constraints (Jiang et al., 2025). Beyond methods for self-attention, RingX¹ (Yin et al., 2025) and LV-XAttn (Chang & Venkataraman, 2025) distribute bidirectional cross-attention by broadcasting queries in the forward pass. These methods target BERT-like (Liu et al., 2019) or multimodal (Ye et al., 2025) workloads without causal query–key dependencies and do not address the load imbalance introduced by anchor sampling.

Motivation

Diffusion Drafter Training

Diffusion drafters use cross-attention with a block-sparse mask, a pattern distinct from the self-attention used by LLMs and autoregressive drafters. We use DFlash [3] as the concrete architecture throughout the paper. The same attention pattern appears in subsequent diffusion drafters such as Domino (Huang et al., 2026) and DSpark (Cheng et al., 2026), so the Split-KV design applies to them as well.

Specifically, a micro-batch of bb prompt–response samples is packed into SS context tokens y1,…,ySy_1,\ldots,y_S and fed to the target model to extract intermediate hidden states. DFlash fuses and projects these hidden states into context features c∈RS×Hc \in\mathbb{R}^{S \times H} with hidden size HH, then projects cc into the context keys and values at each drafter layer. For the attention queries, anchor sampling selects AA response positions per sequence. Each anchor opens a block whose remaining B−1B-1 positions are mask tokens to be denoised as draft tokens.

The DFlash block-sparse mask for one micro-batch with two draft blocks

Figure 1. The DFlash block-sparse mask for one micro-batch with two draft blocks (B=3B=3). One cell is one query–key pair. Each block’s anchor row aia_i and its B−1B-1 masked rows ([M]) attend the context window [si,ai)[s_i,a_i) of their own sample and their own block.

Across the micro-batch, this process produces N=bAN=bA blocks and a query length of Q=NBQ=NB. Anchor ii lies at position aia_i in the sequence beginning at sis_i and attends to its context window Wi=[si,ai)W_i=[s_i,a_i) and all BB tokens in its own block. The attention is therefore causal over the context and bidirectional within each block. As shown in Figure 1, FlexAttention processes all NN blocks in one pass with QQ queries and a key/value sequence comprising SS context tokens and QQ mask tokens (Dong et al., 2024). Throughout this paper, SS denotes the packed context length, QQ denotes the packed query length, and the key/value length is S+QS+Q.

Limitation 1: Communication Ignores Query-Context Asymmetry

Query–context asymmetry Diffusion drafter training decouples the packed context length SS, determined by the training samples, from the packed query length QQ, determined by the anchor budget. The impact of the two factors on the final drafter quality is still underexplored. To determine whether this separation creates a useful Q≪SQ \ll S regime, we conduct two controlled SFT studies using DFlash drafters for Qwen3.5-9B and Qwen3.5-35B-A3B. Fix QQ–Scale SS holds the query budget fixed while increasing the maximum per-sequence context length. Fix SS–Scale QQ instead fixes the maximum context length and increases the query budget through the number of sampled anchors. Figure 2 reports mean drafter acceptance length across GSM8K (Cobbe et al., 2021), MATH-500 (Lightman et al., 2024), HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), and MT-Bench (Zheng et al., 2023). Appendix A provides the detailed setup and results.

Figure 2. Mean acceptance length over five downstream tasks for two controlled SFT studies. (a) Fix QQ–Scale SS varies the maximum context length at a fixed query budget. (b) Fix SS–Scale QQ varies the query budget at a fixed maximum context length.

Figure 2(a) shows that, with QQ fixed, scaling SS changes the five-task mean acceptance length by at most 0.82% for either model. Thus, increasing the context does not require a proportional increase in the query budget to preserve drafter quality. Figure 2(b) then fixes a long context and scales QQ. Acceptance length increases monotonically, confirming that query budget matters, but the gain from each doubling diminishes: the final doubling improves the mean by only 0.98% and 1.57% for two models separately. The first study shows that QQ need not grow with SS, while the second identifies a quality-saturation region at a query bud get substantially smaller than the context. Together, they establish our core observation: acceptance length remains stable as context scales with a fixed query budget, while increasing the query budget yields diminishing quality gains, creating a practical Q≪SQ \ll S regime.

Limitations of existing context parallelisms Table 1 shows that most existing CP methods do not exploit this query–context asymmetry. Ulysses communicates query/key/value tensors through all-to-all collectives ((Jacobs et al., 2024)), Ring Attention circulates key/value shards ((Liu et al., 2024; Zhu, 2024)), and RingX broadcasts key/value tensors and reduces their gradients in the backward pass ((Yin et al., 2025)). Their total communication therefore contains an SS-dependent term and grows with the context even when QQ is fixed. LV-XAttn makes attention communication independent of SS, but its P2P loop circulates running output and gradient state along with QQ and provides no mechanism for the anchor-induced load imbalance discussed next ((Chang & Venkataraman, 2025)).

Limitation 2: Inadequate Load Balancing

Nonlinear context access With the anchor sampling, context tokens at different positions incur different amounts of attention works, creating a unique load balance requirement different from causal self-attention.

Let the per-position load c(x)c(x) be the number of query rows that attend the key at absolute context position xx. For causal self-attention on a sequence of length LL, this load is linear:

cself(x)=L−x.(1)c_{\mathrm{self}}(x) = L - x. \tag*{(1)}

Zigzag relies on cself(x)+cself(L−1−x)=L+1c_{\mathrm{self}}(x) + c_{\mathrm{self}}(L - 1 - x) = L + 1 to achieve perfect load balance ((Zhu, 2024)).

Diffusion drafters instead sample anchors over the response. Each anchor contributes BB queries that read only the context prefix before it, so

canchor(x)=B∑i=0N−11[si≤x<ai],Cp=∑x∈Kpcanchor(x),(2)\begin{aligned} c_{\mathrm{anchor}}(x) &= B \sum_{i=0}^{N-1} \mathbf{1}[s_i \le x < a_i],\\ C_p &= \sum_{x \in\mathcal{K}_p} c_{\mathrm{anchor}}(x), \tag*{(2)} \end{aligned}

where Kp\mathcal{K}_p is the set of context positions assigned to rank pp and CpC_p is its context-dependent attention work. As Figure 3 shows, this load is flat over the prompt and drops only at sampled anchors in the response. Therefore, Anchor sampling changes the linear causal profile into a sample-dependent, nonlinear context-access profile.

Figure 3. Number of queries attending key xx in a 16-token sample. Causal self-attention has a linear profile whose mirrored positions have constant total load. Diffusion drafter anchor sampling (B=2B = 2, anchors {6,10,11,12}\{6, 10, 11, 12\}) produces a piecewise profile.

Limitations of existing balancing strategies Table 1 summarizes the balancing strategy coupled to each CP method. Ulysses evenly shards attention heads, but its CP size PP is bounded by the number of heads and thus cannot scale to arbitrary CP sizes ((Jacobs et al., 2024)). Ring Attention adopts Zigzag ((Zhu, 2024)), whose mirrored-position rule is inaccurate for the nonlinear, sample-dependent profile as shown in Figure 3. DCP can represent arbitrary sparse masks through hypergraph partitioning, but its per-step planning is expensive (§ 6.4) ((Jiang et al., 2025)). The bidirectional cross-attention methods RingX and LV-XAttn do not provide any load-balancing mechanism and preserve contiguous context shards ((Yin et al., 2025; Chang & Venkataraman, 2025)).

Consequently, existing methods neither fully exploit the query–context asymmetry in their parallel design nor account for the nonlinear attention workload induced by anchor sampling in their load balancing.

Split-KV: Context Parallelism for Diffusion Drafters

This section introduces Split-KV, a context-parallel method designed for diffusion-drafter training with Q≪SQ \ll S. Section 4.1 presents the parallelism design and analyzes its communication, computation and memory overhead. Section 4.2 analyzes the load skew introduced by anchor sampling and presents a low-cost longest-processing-time-first (LPT) load-balancing strategy built upon Split-KV’s parallelism.

Split-KV Parallelism Design

Motivated by Q≪SQ \ll S, Split-KV starts from the asymmetry between the QQ query-source rows X∈RQ×HX \in\mathbb{R}^{Q \times H} and the S+QS + Q key/value-source rows U∈R(S+Q)×HU \in\mathbb{R}^{(S+Q) \times H}.

Split-KV partitions X=⨆p=0P−1XpX = \bigsqcup_{p=0}^{P-1} X_p and U=⨆p=0P−1UpU = \bigsqcup_{p=0}^{P-1} U_p by rows across PP devices, but replicates only the projected queries q=QProj⁡(X)q = \operatorname{QProj}(X) while keeping kp,vp=KeyValueProj⁡(Up)k_p, v_p = \operatorname{KeyValueProj}(U_p) at rank pp. For one query and attention head, let zp=qkp⊤/d+Mpz_p = q k_p^{\top} / \sqrt{d} + M_p denote the masked scores over rank pp’s local keys, with MpM_p being the block-sparse mask and masked entries set to −∞-\infty, the local attention result and its log-sum-exp (LSE) are

ℓp=log⁡∑jexp⁡(zp,j),op=∑jexp⁡(zp,j−ℓp)vp,j(3)\ell_p = \log\sum_j \exp(z_{p,j}), \qquad o_p = \sum_j \exp(z_{p,j} - \ell_p)v_{p,j} \tag*{(3)}

Because the local key sets form a disjoint partition of the complete key/value sequence, the single-device result satisfies

ℓ=log⁡∑p=0P−1exp⁡(ℓp),o=∑p=0P−1exp⁡(ℓp−ℓ)op=∑p=0P−1oˉp(4)\ell= \log\sum_{p=0}^{P-1} \exp(\ell_p), \qquad o = \sum_{p=0}^{P-1} \exp(\ell_p-\ell)o_p = \sum_{p=0}^{P-1}\bar{o}_p \tag*{(4)}

where oˉp=exp⁡(ℓp−ℓ)op\bar{o}_p = \exp(\ell_p-\ell)o_p is rank pp’s additive contribution to the globally normalized output. Writing OProj⁡\operatorname{OProj} as its linear map and adding any bias once after reduction, linearity further gives

Yp=[∑p=0P−1OProj⁡(oˉp)]p=[OProj⁡(∑p=0P−1oˉp)]p(5)Y_p = \left[\sum_{p=0}^{P-1}\operatorname{OProj}(\bar{o}_p)\right]_p = \left[\operatorname{OProj}\left(\sum_{p=0}^{P-1}\bar{o}_p\right)\right]_p \tag*{(5)}

where [⋅]p[\cdot]_p selects the query rows owned by rank pp and is implemented together with the summation by a sum reduce-scatter. Equations 3–5 therefore reconstruct the single-device attention exactly without moving any key/value rows;

Algorithm 1 Split-KV for one draft layer on rank pp. AG, AR, and RS denote all-gather, all-reduce, and reduce-scatter.

  1. Input: Xp,Up,MpX_p, U_p, M_p ▷ local query rows, key/value-source rows, attention mask

  2. Forward pass:

  3. qp←QProj⁡(Xp)q_p \leftarrow\operatorname{QProj}(X_p)

  4. launch q←AG⁡(qp)q \leftarrow\operatorname{AG}(q_p)

  5. kp,vp←KeyValueProj⁡(Up)k_p, v_p \leftarrow\operatorname{KeyValueProj}(U_p) ▷ overlaps AG

  6. wait for qq

  7. (op,ℓp)←Attn⁡(q,kp,vp;Mp)(o_p, \ell_p) \leftarrow\operatorname{Attn}(q, k_p, v_p; M_p)

  8. (ℓ,oˉp)←MERGE⁡AR⁡(op,ℓp)(\ell, \bar{o}_p) \leftarrow\operatorname{MERGE}_{\operatorname{AR}}(o_p, \ell_p)

  9. yˉp←OProj⁡(oˉp)\bar{y}_p \leftarrow\operatorname{OProj}(\bar{o}_p)

  10. Yp←RS⁡sum⁡(yˉp)Y_p \leftarrow\operatorname{RS}_{\operatorname{sum}}(\bar{y}_p)

  11. return YpY_p

  12. Backward pass:

  13. g←AG⁡(dYp)g \leftarrow\operatorname{AG}(dY_p)

  14. doˉp←OProjBwd⁡(g)d\bar{o}_p \leftarrow\operatorname{OProjBwd}(g)

  15. (dop,dℓp)←MERGEBwd⁡AR⁡(doˉp)(do_p, d\ell_p) \leftarrow\operatorname{MERGEBwd}_{\operatorname{AR}}(d\bar{o}_p)

  16. (dq(p),dkp,dvp)←AttnBwd⁡p(dq^{(p)}, dk_p, dv_p) \leftarrow\operatorname{AttnBwd}_p

  17. launch dqp←RS⁡sum⁡(dq(p))dq_p \leftarrow\operatorname{RS}_{\operatorname{sum}}(dq^{(p)})

  18. dUp←KeyValueProjBwd⁡(dkp,dvp)dU_p \leftarrow\operatorname{KeyValueProjBwd}(dk_p, dv_p) ▷ overlaps RS

  19. wait for dqpdq_p

  20. dXp←QProjBwd⁡(dqp)dX_p \leftarrow\operatorname{QProjBwd}(dq_p)

  21. return dXp,dUpdX_p, dU_p

the forward pass communicates only qq and its projected output, and the backward pass communicates their corresponding gradients as depicted in Figure 4.

Comparison of Ulysses, Ring Attention, RingX, LV-XAttn, and Split-KV communication patterns

Figure 4. Split-KV exploits Q≪SQ \ll S by all-gathering the global query while keeping the long key/value shards local. The global query view decouples key/value-chunk work from query ownership and enables the cost-aware balancing design.

RingX Yin et al., 2025 and LV-XAttn Chang & Venkataraman, 2025 expose one query shard at each communication step. Under the anchor sampling, a key/value chunk’s cost varies with the active query shard, so balancing aggregate work does not balance every synchronized communication step. Split-KV instead all-gathers the complete qq before local attention, making every rank evaluate the full realized query workload against its local key/value chunks at once. This collapses the step-dependent workload into one destination-independent cost per key/value chunk, enabling the cost-aware assignment in § 5.2, while Q≪SQ \ll S keeps the additional query memory small. The complete block-sparse attention therefore evaluates Tattn=B∑i=0N−1(ai−si)+NB2T_{\mathrm{attn}} = B\sum_{i=0}^{N-1}(a_i-s_i)+NB^2 pairs, and balanced Split-KV assigns an average operation cost of TattnH/PT_{\mathrm{attn}}H/P to each rank. At the forward-pass peak, each rank simultaneously holds a local key/value shard of length (S+Q)/P(S+Q)/P, the all-gathered queries, and the partial projected outputs, giving a peak of 2(S+Q)H/P+2QH2(S+Q)H/P+2QH. The backward pass has the same peak memory.

Algorithm 1 instantiates this decomposition on rank pp, where XpX_p, UpU_p, and MpM_p are the local query rows, the local key/value-source rows comprising the context shard and owned block rows, and the corresponding block-sparse mask. In the forward pass, the query all-gather is overlapped with the key/value projection (lines 4–6), after which lines 7–10 evaluate Equations (3)–(5). The backward pass applies the dual schedule (lines 13–21), overlapping the query-gradient reduce-scatter with the key/value-projection backward computation while keeping dkpd k_p and dvpd v_p local.

Table 1 summarizes the communication and peak-memory costs of Split-KV. The forward-pass all-gather and reduce-scatter each move P−1PQH\frac{P-1}{P}QH elements per rank, and their backward-pass duals move the same amount. Thus Split-KV communicates P−1P4QH\frac{P-1}{P}4QH elements per layer. For a fixed micro-batch size and anchor budget, QQ remains fixed as SS grows, so this traffic is independent of the context length; the LSE statistics contribute only a lower-order term. For block ii, its BB queries attend to ai−sia_i-s_i context rows and all BB rows in the same block, producing B(ai−si)+B2B(a_i-s_i)+B^2 unmasked query–key pairs.

communicationpeak memoryload balancing
no CP (P=1P=1)—(2(S+Q)+2Q)H(2(S+Q)+2Q)HN/A
UlyssesP−1P⋅4H(S+2Q)\frac{P-1}{P}\cdot4H(S+2Q)(4(S+Q)+2Q)HP(4(S+Q)+2Q)\frac{H}{P}Head sharding
Ring Attention2(3P−2)P⋅(S+Q)H\frac{2(3P-2)}{P}\cdot(S+Q)H(4(S+Q)+2Q)HP(4(S+Q)+2Q)\frac{H}{P}Zigzag
RingXP−1P⋅2H(2S+3Q)\frac{P-1}{P}\cdot2H(2S+3Q)(4(S+Q)+2Q)HP(4(S+Q)+2Q)\frac{H}{P}None
LV-XAttn2(3P−2)P⋅QH\frac{2(3P-2)}{P}\cdot QH(2(S+Q)+8Q)HP(2(S+Q)+8Q)\frac{H}{P}None
Split-KVP−1P⋅4QH\frac{P-1}{P}\cdot4QH(2(S+Q)+2PQ)HP(2(S+Q)+2PQ)\frac{H}{P}Cost-aware LPT

Table 1. Per-rank training cost for a diffusion drafter layer. Multi-head attention is used for brevity. HH is hidden size, PP is CP size. Communication includes both forward and backward passes. Peak memory counts input and output tensors and communication buffers. Red highlights mark baseline limitations jointly addressed by Split-KV.

Having established context-independent communication and destination-independent chunk costs, we now turn to assigning those chunks across ranks.

Cost-Aware LPT Load Balancing for Split-KV

Cost-aware LPT Using the global queries exposed by Split-KV, we address the nonlinear workload identified in § 4.3 with a cost-aware heuristic based on longest processing time (LPT) first strategy. Our cost-aware LPT partitions the packed context globally into PJPJ equal chunks of m=S/(PJ)m=S/(PJ) rows, where each rank must receive JJ chunks. This one-dimensional assignment relies on Split-KV’s global query view: every rank evaluates the complete query set against whichever context chunks it owns. The aggregate cost of each key/value chunk is therefore independent of its destination rank and can be represented by one scalar, whereas hop-based methods expose a different cost for each active query shard and require a joint assignment over chunks and communication steps. Because every rank has the global anchor metadata A={(si,ai)}i=0N−1\mathcal{A}=\{(s_i,a_i)\}_{i=0}^{N-1}, it can independently price chunk jj, whose left boundary is λj\lambda_j, as

c^j=B∑i=0N−1(min⁡(ai,λj+m)−max⁡(si,λj))+(6)\hat{c}_j = B \sum_{i=0}^{N-1} \left(\min(a_i,\lambda_j+m)-\max(s_i,\lambda_j)\right)^+ \tag*{(6)}

where (⋅)+(\cdot)^+ denotes the positive part. We sort chunks by c^j\hat{c}_j, the context work of chunk jj, and repeatedly assign the remaining chunk with the longest estimated processing time (the highest cost) to the least-loaded rank with fewer than J\mathcal{J} chunks. A final locality matching maps the resulting bundles to physical ranks without changing their loads.

The plan itself requires no communication; one all-to-all then redistributes the context-source rows according to the assignment once per training step, before any draft layer executes. This step-level key/value redistribution is separate from the per-layer communication reported in Table 1 and is amortized across all draft layers. The draft blocks in Figure 1 remain local. With balanced block ownership they add the same D=(N/P)B2D=(N/P)B^2 work to every rank. Thus rank pp has total work Wp=Cp+DW_p=C_p+D. We define the load-imbalance ratio as ρ=max⁡pWp/W‾\rho=\max_p W_p/\overline{W}, where W‾=P−1∑pWp\overline{W}=P^{-1}\sum_p W_p.

A concrete example Figure 5 compares Zigzag, the position-based strategy discussed in § 4.3, with cost-aware LPT on one realized micro-batch. Panel (a) shows two packed samples with different lengths and anchor locations, which induce the nonlinear per-position costs from Equation (2); here u∈{0,1}u\in\{0,1\} indexes the sample. Panel (b) applies Zigzag independently to each sample, where Zj(u)Z_j^{(u)} denotes chunk jj, and pairs mirrored positions without considering their realized costs. Although every rank receives the same number of chunks, their full loads become [32,32,38,42][32,32,38,42]. Panel (c) instead forms global chunks LjL_j, prices them with Equation (6), and assigns them by LPT, producing [36,36,36,36][36,36,36,36]. Because the block-local contribution D=8D=8 is identical across ranks, equalizing the context contribution CpC_p reduces ρ\rho from 1.167 to 1.000.

Load balancing of the context keys in Split-KV

Figure 5. Load balancing of the context keys in Split-KV for the same two packed samples and sampled anchors. A cell shows its position and realized context cost canchor(x)c_{\mathrm{anchor}}(x); chunk cells show the chunk label above its summed context cost. Zigzag independently cuts each sample into 2P2P chunks and pairs mirrored positions, yielding (C0,…,C3)=[24,24,30,34](C_0,\ldots,C_3) = [24,24,30,34]. LPT cuts the global packed context into PJPJ chunks and assigns them by realized chunk cost, yielding (C0,…,C3)=[28,28,28,28](C_0,\ldots,C_3) = [28,28,28,28]. Adding the identical D=2B2=8D = 2B^2 = 8 gives (W0,…,W3)=[32,32,38,42](W_0,\ldots,W_3) = [32,32,38,42] and (W0,…,W3)=[36,36,36,36](W_0,\ldots,W_3) = [36,36,36,36], respectively.

Implementation Details

Our implementation is based on VeOmni, a PyTorch-native backend that composes FSDP2 ((Zhao et al., 2023)) with sequence parallelism and expert parallelism on multidimensional device meshes ((Ma et al., 2025)). Our implementation comprises approximately 4.3K lines of Python for constructing the drafter, capturing target-model hidden states, sampling anchors, building block-sparse masks, balancing the key/value chunks, computing the denoising loss and metrics, and integrating optimizer and checkpointing.

Beyond supporting diffusion-drafter training and implementing Split-KV, efficient long-context execution requires an additional system optimization. Packed, padding-free batches reduce computation but expose variable local key/value lengths that trigger repeated FlexAttention compilation. We therefore round each local key/value length up to a multiple of 4096, forming static-shape buckets whose compiled kernels can be reused. Padding is masked out before attention and loss computation.

Evaluation

Experiment Setup

Testbed We conduct all experiments on up to eight nodes, each equipped with eight accelerators. Within a node, each accelerator has 600 GB/s of aggregate bidirectional intra-node bandwidth; across nodes, communication goes through RDMA with 400 Gb/s of inter-node bandwidth.

Workload We evaluate DFlash drafters for Qwen3.5-35B-A3B and Qwen3.5-27B. The target models are from the official Qwen3.5 model collection3, and the drafter model architectures are summarized in Table 2 following the original DFlash configurations4 ([3]).

TargetLayersdmodeld_{\mathrm{model}}dffnd_{\mathrm{ffn}}nqn_qnkvn_{kv}dhd_h
Qwen3.5-35B-A3B820486144324128
Qwen3.5-27B5512017408328128

Table 2. Architectures of the evaluated DFlash drafters, where nqn_q, nkvn_{kv}, and dhd_h denote the numbers of query heads, key/value heads, and the head dimension, respectively.

Evaluation protocol Because diffusion drafters are typically trained for multiple epochs ([3] (Huang et al. 2026; Cheng et al. 2026)), we follow the offline training setting to directly feed target-model hidden states to the drafter and measure only drafter training step time. This offline setting excludes target-model feature extraction time. To eliminate variable-length sequences as a confounding factor, we use randomly generated fixed-length samples with a 1:3 prompt-to-response ratio. Following DFlash, we set the diffusion block size to B=16B=16 and vary the anchor budget AA to control the query length. We report per-sample context and query lengths when describing each benchmark configuration. For micro-batch size bb, the packed SS and QQ defined in § 4 are obtained by multiplying the corresponding per-sample lengths by bb.

To isolate context parallelism, we set the DP size to one and world size equal to CP size P∈{8,16,32,64}P\in\{8,16,32,64\}. We report end-to-end drafter training step time averaged over three steps after five warmup steps.

Baselines To evaluate CP methods, we compare Split-KV with four representative baselines. Ulysses (Jacobs et al.,5

  1. shards attention heads through all-to-all communication. Ring Attention (Liu et al., 2024; Zhu, 2024) rotates key/value shards with a zigzag-balanced schedule. RingX (Yin et al., 2025) moves queries in the forward pass, then broadcasts and reduces key/value gradients in the backward pass. LV-XAttn (Chang & Venkataraman, 2025) circulates query-side attention state through P2P rings.

To fairly evaluate load balancing, all baselines use Split-KV and differ only in how context key/value chunks are assigned to ranks. No balance preserves the contiguous chunk assignment. Zigzag applies a mirrored chunk order (Zhu, 2024). DCP invokes KaHyPar for hypergraph partitioning (Jiang et al., 2025; Schlag et al., 2016). LPT is our cost-aware strategy.

Weak Scaling of Drafter Training Step Time

Following a weak-scaling configuration, we increase PP from 8 to 64 and scale the per-sample context length proportionally from 128k to 1M while fixing the per-sample query length at 8k and b=8b = 8. Consequently, the packed context length SS grows from 1M to 8M, while the global packed query length remains Q=64kQ = 64\mathrm{k}. The local context shard remains S/P=128kS/P = 128\mathrm{k} rows per rank. Figure 6 reports the drafter training step time of every method at each scale.

Figure 6. Drafter training step time under weak scaling. Each bar is labeled with absolute time in seconds and weak-scaling efficiency Eweak(P)=T8/TPE_{\mathrm{weak}}(P) = T_{8}/T_{P}, separated by a slash. Lower time and higher efficiency are better; 100% efficiency denotes unchanged drafter training step time as the context and PP grow proportionally. N/A denotes an unsupported configuration.

Split-KV has the lowest reported drafter training step time in six of the eight model-scale configurations, ties Ulysses in a seventh, and provides 1.01×–2.81× speedup where it is strictly faster. Across the 8× scale-out, its drafter training step time remains within 3.67–4.02 s for Qwen3.5-35B-A3B and 3.33–3.94 s for Qwen3.5-27B. At P=64P = 64, these results correspond to 91% and 85% weak-scaling efficiency, respectively, while Split-KV is 2.81× and 2.19× faster than the fastest available baseline. The near-flat times reflect that its attention communication is independent of context length; the remaining efficiency loss comes from collective latency and synchronization as PP increases, even though the query payload stays fixed.

Ring Attention and RingX degrade rapidly because their training communication still carries context-sized tensors, whose size grows with SS in this experiment. Ulysses remains competitive at small scales but is unsupported at P=64P = 64 because of its head-sharding constraint. LV-XAttn’s attention communication is also independent of context length, but its sequential hop loop and contiguous, unbalanced key/value placement keep it slower than Split-KV.

Strong Scaling of Drafter Training Step Time

Following a strong-scaling configuration, we fix the total workload per training step at b=4b = 4, with a per-sample context length of 256k and a per-sample query length of 8k, while increasing PP from 8 to 64. The packed lengths are therefore S=1MS = 1\mathrm{M} and Q=32kQ = 32\mathrm{k}, and the local context shard S/PS/P decreases from 128k to 16k rows per rank. Figure 7 reports the drafter training step time of every method as the fixed workload is distributed over more ranks.

Figure 7. Drafter training step time under strong scaling. Each bar is labeled with absolute time in seconds and strong-scaling efficiency Estrong(P)=(T8/TP)/(P/8)E_{\mathrm{strong}}(P) = (T_8/T_P)/(P/8), separated by a slash. Lower time and higher efficiency are better; the gap from 100% quantifies the scale-out loss. N/A denotes an unsupported configuration.

Split-KV has the lowest reported drafter training step time in seven of the eight model-scale configurations and provides 1.04×–3.67× speedup over the fastest available baseline in those settings. It reduces drafter training step time from 3.46 to 0.78 s for Qwen3.5-35B-A3B and from 2.98 to 0.87 s for Qwen3.5-27B, corresponding to 4.44× and 3.43× speedups under an 8× increase in parallelism. At P=64P = 64, Split-KV therefore retains 55% and 43% strong-scaling efficiency, respectively. The efficiency gap arises because local attention work shrinks with S/PS/P, whereas global query payload remains fixed and collective latency and synchronization grow with PP.

Ulysses is comparable to Split-KV at the smallest scale for Qwen3.5-27B but is unsupported at P=64P = 64. Ring Attention and RingX retain context-dependent traffic, while LV-XAttn avoids that traffic but executes a sequential query-state hop loop without balancing key/value work. Consequently, even though LV-XAttn is the fastest available baseline at P=64P = 64, Split-KV remains 3.67×3.67\times and 2.55×2.55\times faster for Qwen3.5-35B-A3B and Qwen3.5-27B, respectively.

Query-to-Context Ratio Ablation

We evaluate query-to-context sensitivity on a fixed four-node cluster with P=32P = 32 (32 accelerators). Across all configurations, we hold the per-sample context length at Ssample=128kS_{\mathrm{sample}} = 128\mathrm{k} and the packed query length at Q=64kQ = 64\mathrm{k}. We vary the per-sample query length QsampleQ_{\mathrm{sample}} together with the micro-batch size bb using (Qsample,b)=(32k,2)(Q_{\mathrm{sample}}, b) = (32\mathrm{k}, 2), (16k,4)(16\mathrm{k}, 4), (8k,8)(8\mathrm{k}, 8), and (4k,16)(4\mathrm{k}, 16), so that Q=bQsampleQ = bQ_{\mathrm{sample}} remains fixed. The packed context length S=bSsampleS = bS_{\mathrm{sample}} is therefore 256k256\mathrm{k}, 512k512\mathrm{k}, 1M1\mathrm{M}, and 2M2\mathrm{M}, giving Q:SQ:S ratios of 1:4, 1:8, 1:16, and 1:32. Because each query attends only within its own sample, the nominal query–context attention work QSsampleQS_{\mathrm{sample}} also remains fixed.

Figure 8 demonstrates the central property of Split-KV: decreasing Q:SQ:S does not increase its attention communication because only query-sized tensors are exchanged. Because the figure reports the full training step rather than attention communication alone, residual growth can still come from context-side computation and the larger micro-batch. For Qwen3.5-35B-A3B, increasing S/QS/Q from 4 to 32 raises Split-KV time by only 1.11×1.11\times, compared with 2.18×2.18\times for Ulysses, 5.63×5.63\times for Ring Attention, and 4.35×4.35\times for RingX; Split-KV is fastest throughout, and its speedup over Ulysses grows from 1.01×1.01\times to 1.97×1.97\times. For Qwen3.5-27B, Split-KV grows by 1.44×1.44\times, versus 2.05×2.05\times for Ulysses, 6.32×6.32\times for Ring Attention, and 4.09×4.09\times for RingX. Ulysses is 1.10×1.10\times faster at Q:S=1:4Q:S = 1:4, but Split-KV overtakes it at 1:8 and reaches a 1.29×1.29\times speedup at 1:32. LV-XAttn grows by only 1.17×1.17\times and 1.22×1.22\times for the two models because its communication is also query-sized, but its sequential hop schedule leaves it 1.98×1.98\times–3.48×3.48\times slower than Split-KV across all settings.

Figure 8. Drafter training step time as the query-to-context ratio decreases. Lower is better; labels report absolute time in seconds.

Load-Balancing Quality and Planning Cost

We isolate load balancing under a common Split-KV transport, so the strategies differ only in how they assign context chunks. We fix the per-sample query length at 8k and b=4b = 4, and evaluate two scaling studies: context-length scaling from 16k to 256k per sample at P=8P = 8, and CP-size scaling from P=8P = 8 to 64 at a fixed per-sample context length of 256k. We report percentage imbalance as 100(ρ−1)100(\rho- 1), where ρ=max⁡pWp/W‾\rho= \max_p W_p/\overline{W} is the load-imbalance ratio, together with the planning latency required to construct each assignment. Both drafters use the same anchors, block partition, and query-head count and therefore induce identical work distributions.

Figure 9 summarizes each strategy by its average and standard deviation because their relative ordering is invariant across both scaling studies. No balance ignores the nonlinear access profile and averages 46.49% and 55.26% imbalance under context-length and CP-size scaling. Zigzag reduces these values to 7.94% and 8.01%, but its position-based pairing leaves residual skew. The cost-aware methods are substantially more accurate: DCP averages 0.66% and 0.80%, while LPT achieves the best balance at 0.38% and 0.21%.

Figure 9. Average work imbalance under varying context lengths and CP sizes. Each label reports avg (std), with both values measured in percentage points. The lower panel magnifies the cost-aware strategies.

Figure 10 shows that comparable assignment quality does not imply comparable planning cost. For Qwen3.5-35B-A3B, Zigzag takes 0.01–0.05 ms and LPT takes 0.38–1.24 ms across both scaling studies. DCP instead grows from 16.92 to 148.70 ms with context length and from 148.38 to 1164.64 ms with PP because it invokes a general hypergraph partitioner; Qwen3.5-27B exhibits the same trend.

Figure 10. Planning latency across context-length and CP-size scaling for Qwen3.5-35B-A3B. The shared logarithmic axis labels every decade explicitly so that all three planners remain visible. No balance has zero planning cost and is omitted.

Combining Figure 9 and Figure 10, LPT achieves the best load balance with low planning overhead: it is nearly three orders

of magnitude faster to plan than DCP while further reducing imbalance. End to end, LPT achieves a 1.15×–1.42× speedup in drafter training step time over No balance, whose details are extended to Appendix § 10.

Conclusion

Diffusion drafter training exposes a distinct CP regime: acceptance length remains stable as context scales with a fixed query budget, while increasing the query budget yields diminishing quality gains, yielding Q≪SQ \ll S; meanwhile, anchor sampling induces nonlinear, sample-dependent attention work. Split-KV addresses both properties by keeping key/value shards local and limiting per-layer attention communication to query-sized tensors, making this communication independent of context length. The global query view enables cost-aware LPT assignment that balances the realized workload. At a CP size of 64, Split-KV retains 85–91% weak-scaling efficiency and achieves up to 3.67× speedup over the fastest available CP baseline; LPT maintains sub-1% imbalance with planning latency nearly three orders of magnitude lower than the best available load-balancing baseline, DCP. These results show that context parallelism for structured attention should jointly design communication and load balancing around the model’s training semantics rather than inherit schedules designed for self-attention.

appendix.pdf

Drafter Quality across Query and Context Lengths

Training setup We train DFlash drafters for the frozen Qwen3.5-9B and Qwen3.5-35B-A3B target models on their model-specific regenerated SFT corpora, a mixture of 800K samples from NVIDIA Nemotron Post-Training Dataset V22 and CodeAlpaca2.3 Following the original DFlash recipe (Chen et al., [3]), training uses cross-entropy loss, block size B=16B = 16, loss-decay factor 7.0, weight decay 0.1, gradient clipping at 1.0, and a cosine learning-rate schedule with a 4% warmup. The 35B-A3B drafter uses the original learning rate 6×10−46 \times10^{-4}. For the 9B drafter, this rate causes recurrent gradient spikes, so we reduce it to 2×10−42 \times10^{-4} while leaving the remaining optimization settings unchanged.

Both studies keep the global batch size at 256 and follow the same six-epoch recipe. Fix QQ–Scale SS uses 512 anchors per sequence, giving Q=8kQ = 8\mathrm{k} at B=16B = 16, and varies the maximum per-sequence context length over {4k,16k,64k}\{4\mathrm{k}, 16\mathrm{k}, 64\mathrm{k}\}. Fix SS–Scale QQ fixes the maximum per-sequence context length at 64k and varies the anchor budget over {128,256,512,1024}\{128, 256, 512, 1024\}, giving Q∈{2k,4k,8k,16k}Q \in\{2\mathrm{k}, 4\mathrm{k}, 8\mathrm{k}, 16\mathrm{k}\}. Each maximum context length is a right-truncation limit rather than a requirement that every sample fill the sequence. All runs terminate at step 18,684 for Qwen3.5-9B and step 18,744 for Qwen3.5-35B-A3B; the 64k, 512-anchor run is shared by the two studies. All variants use Split-KV for drafter training.

Evaluation protocol We evaluate GSM8K, MATH-500, HumanEval, MBPP, and MT-Bench with the official DFlash benchmark and task-specific prompt formatters. Each drafter–task pair starts a fresh vLLM server on two accelerators with tensor parallelism two, 15 speculative tokens, and at most 32,768 batched tokens. We issue 128 sequential requests with thinking enabled, at most 2,048 generated tokens, temperature zero, top-pp one, and top-kk one. The reported acceptance length equals one plus accepted draft tokens divided by draft steps, including the target model’s bonus token. We measure acceptance length rather than downstream answer accuracy.

Results For Fix QQ–Scale SS, Table 1 shows no monotonic relationship between maximum training context length and acceptance length. Across 4k–64k, the relative task-average span, computed as the difference between the maximum and minimum divided by the minimum, is only 0.82% for Qwen3.5-35B-A3B and 0.70% for Qwen3.5-9B, and the best context length varies by task. Because each variant is evaluated once, we treat these sub-1% differences as equivalent rather than as statistically significant rankings. Thus, scaling SS does not require scaling QQ to preserve drafter quality over the tested range.

For Fix SS–Scale QQ, acceptance length increases monotonically with the query budget on both target models, so reducing QQ is not free. However, the marginal gain decreases across successive doublings of QQ: the mean gains are 0.16, 0.14, and 0.06 for Qwen3.5-35B-A3B and 0.13, 0.12, and 0.09 for Qwen3.5-9B. Consequently, the final doubling of QQ improves mean acceptance length by only 0.98% and 1.57%, respectively. Together, the two studies show that the query budget need not grow with the context and that quality exhibits diminishing returns as that budget increases, establishing a practical Q≪SQ \ll S regime.

Impact of Load Balancing on Drafter Training Step Time

To evaluate the end-to-end effect of load balancing, we fix the per-sample context and query lengths at 256k and 8k, respectively, with b=4b = 4, and measure how assignment quality and planning latency jointly affect drafter training step time as PP increases. Figure 1 reports speedup over No balance, so values below one indicate that planning costs more than balancing saves.

LPT is the only strategy that improves both drafters at every scale, yielding 1.15×–1.42× speedup over No balance. Zigzag remains beneficial except for a slight regression on Qwen3.5-27B at P=64P = 64, but its residual imbalance prevents it from matching LPT. DCP performs competitively at small PP yet falls below No balance from P=32P = 32 for Qwen3.5-35B-A3B and from P=16P = 16 for Qwen3.5-27B as partitioning dominates the shrinking compute time. At P=64P = 64, DCP is 3.11× and 3.24× slower than LPT for the two drafters. Thus, LPT converts near-perfect balance into reductions in drafter training step time without placing an expensive planner on the training critical path. Although planning can be moved into data preparation for offline drafter training, it remains on the critical path when the drafter is trained online during reinforcement learning (Chen et al., [1]; Li et al. (2026)).

Target modelSSQQGSM8KMATH-500HumanEvalMBPPMT-BenchAvg.
Fix QQ–Scale SS
Qwen3.5-35B-A3B4k8k5.986.997.005.964.486.08
16k8k6.097.057.035.984.526.13
64k8k6.096.957.055.974.546.12
Qwen3.5-9B4k8k5.716.596.305.504.405.70
16k8k5.706.516.445.564.475.74
64k8k5.776.566.425.534.375.73
Fix SS–Scale QQ
Qwen3.5-35B-A3B64k2k5.846.586.735.664.315.82
64k4k5.976.786.915.834.405.98
64k8k6.096.957.055.974.546.12
64k16k6.197.047.116.014.556.18
Qwen3.5-9B64k2k5.486.216.165.234.315.48
64k4k5.646.346.305.444.325.61
64k8k5.776.566.425.534.375.73
64k16k5.796.706.545.584.515.82

Table 1 (PDF p. 14). Acceptance length on five downstream tasks. Fix QQ–Scale SS holds 512 anchors per sequence and varies the maximum context length; Fix SS–Scale QQ holds the maximum context length at 64k and varies the anchor budget. With block size B=16B = 16, the anchor budgets {128,256,512,1024}\{128, 256, 512, 1024\} correspond to query lengths {2k,4k,8k,16k}\{2\mathrm{k}, 4\mathrm{k}, 8\mathrm{k}, 16\mathrm{k}\}. Avg. is the unweighted mean of the five displayed task values.

Figure 1 (PDF p. 14). Speedup in drafter training step time for each load-balancing strategy over No balance with per-sample context and query lengths of 256k and 8k, respectively. The dashed line denotes performance equal to No balance.

References

  1. [1]Chen, Q., Liu, Z., Sun, P., Li, S., Wang, G., Liu, Z., Wen, Y., Feng, S., and Zhang, T. Respec: Towards optimizing speculative decoding in reinforcement learning systems, 2025.arxiv.org/abs/2510.26475
  2. [3]Chen, J., Liang, Y., and Liu, Z. Dflash: Block diffusion for flash speculative decoding. arXiv preprint arXiv:2602.06036, 2026.

Paper details

vlnc:W_pe7xw4ht
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents