betaFeedback

WorksComputer Science

The Hidden Cost of MoE Diffusion Decoding: Profiling and Fixing the Sampler

Abstract

Masked diffusion language models (dLLMs) decode by iteratively denoising a full canvas of tokens, so every forward pass runs a per-token sampling and stopping pipeline over the entire canvas-by-vocabulary logits tensor. We profile single-user decoding of a 26B-parameter diffusion Mixture-of-Experts model on a bandwidth-bound edge accelerator (NVIDIA GB10) and report a result that is easy to miss: after weight quantisation, this logits pipeline, not the expert GEMMs, is the single largest consumer of decode time. Unlike autoregressive decoding, where sampling touches one position's logits per committed token, diffusion decoding processes S times as many logit entries per output token, where S is the number of denoising steps. We show this pipeline is largely redundant or fusable and remove it in two lossless tiers: a bit-exact tier whose output is byte-identical to the reference decoder, and an accuracy-parity tier using a fused single-pass entropy kernel and Gumbel-max sampling. Combined with a W4A16 expert kernel, the full stack reaches 2.4-2.7 times end-to-end speedup over bf16 at unchanged benchmark accuracy on GSM8K, HumanEval, and MATH, on a single 273 GB/s device. We confirm the finding on a second, independently built diffusion MoE (LLaDA2.0-mini: pipeline 28.8% of decode, fused W4A16 3.3 times at accuracy parity) and show it disappears on a dense diffusion model (3.6%). The cost is specific to MoE diffusion, where quantisation makes the backbone cheap. Code and quantised weights are released.

Introduction

Masked diffusion language models [1, 8, 14, 12] generate text by repeatedly denoising a fixed-length canvas: each forward pass predicts a distribution over the vocabulary for every canvas position, commits a subset of positions according to a confidence criterion, and re-noises the rest for the next step. This differs fundamentally from autoregressive (AR) decoding, where each forward pass produces logits for a single next position. The difference is usually discussed in terms of parallelism and quality; we point out that it also reshapes where inference time goes.

In AR decoding, the logits post-processing pipeline—temperature scaling, softmax, sampling, and any stopping heuristics—runs once per committed token, over a [1,V][1, V] logits vector, and is negligible next to the backbone matrix multiplications. In diffusion decoding, the same pipeline runs on a [C,V][C, V] tensor at every denoising step, where CC is the canvas length and VV the vocabulary. For a model with V=262,144V = 262{,}144 and C=256C = 256, that is a 67M-element, 268 MiB tensor touched several times per step, and there are many steps per generated block. Per output token, the logits pipeline processes roughly S×S \times more entries than AR, where SS is the number of denoising steps (Section 2). This is a structural cost of diffusion decoding, and it has gone largely unmeasured because dLLMs are rarely served under latency pressure.

We study this cost concretely on DiffusionGemma-26B-A4B, an open-weights masked-diffusion Mixture-of-Experts (MoE) model (128 experts, top-8 routing, 30 layers, hidden size 2816), decoded single-user on an NVIDIA GB10 (DGX Spark: 273 GB/s unified LPDDR, 119 GB, one user, batch size 1)—a regime where decoding is bandwidth-bound and every wasted pass is visible in wall-clock latency. Our contributions:

  • A profiling result. After standard W4A16 expert quantisation removes the dominant weight-bandwidth cost, the logits/sampling pipeline becomes the largest single bucket of decode time (37% averaged over 40 queries), exceeding the expert GEMMs (26%) on every query we measured. We give the O(S)O(S)-per-token analysis that explains why (§ 2).

  • Two lossless optimisations that target this pipeline. These apply known techniques (deduplication, online softmax, Gumbel-max); the profiling observation is the paper’s main contribution, and the optimisations are what it implies (§ 3): (i) a bit-exact elimination of a redundant entropy recomputation, with byte-identical output; (ii) an accuracy-parity fused single-pass entropy kernel and Gumbel-max sampler that replace a multi-pass softmax/multinomial chain.

  • An integrated edge-serving stack for a 26B diffusion MoE on one 273 GB/s device, reaching 2.4–2.7× over bfloat16 at unchanged accuracy, with a strictly bit-exact sub-tier (§ 4). Weights and code are released.1

  • Generality. We reproduce the profiling result and the fused-W4A16 speedup (3.3×, accuracy parity) on a second diffusion MoE, and show a dense diffusion model is the counterexample—so the pipeline cost is a property of MoE diffusion, and its fusable portion depends on the sampling algorithm (§ 5).

Where Does Diffusion Decode Time Go?

Setup. We serve DiffusionGemma-26B-A4B with its reference decoding loop (model.generate) and quantise the MoE experts to asymmetric 4-bit weights with 16-bit activations (W4A16, group sizes 128/64), reducing expert storage from 85.1 to 21.3 GiB. Quantisation is applied through a single fused grouped-GEMM Triton kernel [15] that serves all 128 experts in one launch with a deterministic (atomic-free) combine; this is standard practice [3, 7] and is not our contribution. It is, rather, the tool that exposes the finding: once expert weight traffic is cut by 4×, whatever remains as the bottleneck is what matters for diffusion serving.

The pipeline dominates. Table 1 breaks down quantised decode time over 20 GSM8K and 20 HumanEval queries (canvas CC=256), measured with CUDA-event hooks around each submodule. The experts, despite being the bulk of the parameters, take only about a quarter of the time. The largest single bucket is the logits/sampling pipeline together with its host-side glue: 34–40% on average, more than the experts on all 40 queries (minimum 31.2% against a maximum of 26.6%), and roughly double attention. One GSM8K run hit a host-side stall (456 ms per forward against a median of 219); without it, the GSM8K pipeline share is 38.2 ± 2.4%. The share is lower on HumanEval because its queries take half as many forwards (5.8 versus 11.6), so the one-off prompt encoding claims more of the total. An operator-level trace over a representative GSM8K run attributes this bucket to a small number of full-vocabulary passes over the [256, 262144] logits tensor: the token-entropy computation used by the acceptance and stopping criteria (≈ 29 ms/forward), the softmax feeding multinomial sampling (≈ 20 ms), the multinomial draw (≈ 10 ms), and an argmax (≈ 2.5 ms). Critically, the entropy is computed twice per step—once inside the token-acceptance sampler and once inside the stopping criterion—on the same tensor.

ComponentGSM8K (%)HumanEval (%)ms/fwd
Logits / sampling pipeline + host glue39.8±7.439.8 \pm7.434.4±2.734.4 \pm2.787.087.0
Expert GEMMs (W4A16 MoE)25.3±3.025.3 \pm3.025.8±0.525.8 \pm0.557.957.9
Attention16.3±1.916.3 \pm1.917.1±0.417.1 \pm0.437.937.9
Encoder (prompt KV, amortised)5.9±2.15.9 \pm2.19.8±2.39.8 \pm2.317.917.9
RMSNorms5.9±0.75.9 \pm0.75.9±0.25.9 \pm0.213.313.3
Shared MLP3.5±0.43.5 \pm0.43.8±0.13.8 \pm0.18.38.3
LM head3.3±0.43.3 \pm0.43.2±0.13.2 \pm0.17.47.4

Table 1. Decode-time breakdown of the W4A16 model (GB10, canvas 256): share of wall-clock time per bucket, mean ±\pm std over n=20n=20 queries per benchmark, and mean ms per forward over all 40. Buckets are disjoint. The logits/sampling pipeline (with host glue) is the largest bucket on every query.

Why: an O(S)O(S) per-token asymmetry. Consider generating a block of CC tokens. An AR decoder runs CC forward passes, each sampling over a [1,V][1,V] vector, for C⋅VC \cdot V logit entries of pipeline work, i.e. VV per output token. A diffusion decoder runs SS denoising steps, each running the full pipeline over the [C,V][C,V] canvas, for S⋅C⋅VS \cdot C \cdot V entries, i.e. S⋅VS \cdot V per output token. The logits pipeline is therefore ≈ S×S\times more expensive per token under diffusion, where SS is the step count. In our runs SS is roughly 5–12 (we observe 11.8 and 5.7 forward passes per query on GSM8K and HumanEval respectively), so the pipeline that is free in AR becomes a first-order serving cost. This asymmetry is intrinsic to canvas-based diffusion decoding and independent of the specific acceptance rule. The per-token argument explains why the pipeline costs more than in AR decoding; its share of each forward comes from a second asymmetry. In a bandwidth-bound forward the expert weights are read once and shared by all CC canvas positions, so backbone time is set mostly by weight bytes, while the logits pipeline touches C⋅VC \cdot V elements that nothing amortises. Quantisation cuts only the first term, which is why the pipeline overtakes the experts after W4A16.

Lossless Pipeline Optimisations

We define two levels of “lossless.” Bit-exact: output token IDs and per-query step counts are identical to the reference decoder for every query. Accuracy-parity: benchmark accuracy is statistically unchanged, though individual token trajectories may differ (e.g. a different but equivalent random stream). We target the pipeline of § 2 at both levels.

Tier 1 (bit-exact): remove the duplicate entropy. The reference loop computes the per-position token entropy twice on the same logits—once to decide which tokens to accept, once to decide whether to stop. We compute it once and pass it through. Because the second consumer now receives the exact tensor the first produced, and floating-point reductions are run-to-run deterministic for fixed shape, dtype, and device, the outputs are unchanged bit-for-bit. The only subtlety is that when finished sequences rebuild the logits tensor, the cached entropy no longer describes it; that branch falls back to recomputation. We verified byte-identical text and step counts on all queries (§ 4), and have upstreamed this optimisation to the model’s reference framework.

Tier 2 (accuracy-parity): fuse the entropy and sampling passes. The remaining pipeline is several separate full-vocabulary kernels. We replace them with two fused Triton kernels.

(a) One-pass entropy. The entropy H=log⁡∑jexj−∑jxjexj∑jexjH = \log\sum_j e^{x_j} - \frac{\sum_j x_j e^{x_j}}{\sum_j e^{x_j}} is computed in a single streaming pass using an online-softmax accumulation of the running max, ∑e\sum e, and ∑xe\sum x e, replacing a five-kernel chain (a ≈16×\approx16\times reduction in this op’s cost) at <10−6< 10^{-6} relative error.

(b) Gumbel-max sampling. Categorical sampling as softmax → multinomial materialises a 268 MiB probability tensor and makes several passes. The Gumbel-max identity [4, 11, 5] arg⁡max⁡j(xj+gj)\arg\max_j(x_j + g_j) with gj∼Gumbel⁡(0,1)g_j \sim\operatorname{Gumbel}(0,1), samples exactly the same categorical distribution as the softmax in one reduction. We generate the Gumbel noise inline with a counter-based RNG so it is never stored, and recover the greedy argmax (needed by the stopping criterion) from the same pass at no extra cost. Sampling drops from ≈32\approx32 ms to ≈1.2\approx1.2 ms per forward. Because the RNG stream differs from the reference multinomial, this tier is accuracy-parity rather than bit-exact: the distribution is identical, the specific draws are not.

A third, orthogonal fused-RMSNorm kernel (folding the reference ≈6\approx6-kernel normalisation into one launch per norm) contributes to the full “turbo” configuration. All three are gated on the definitions above.

Table 3: Attribution of pipeline optimisations, GSM8K n=20n=20, same load. Accuracy is stable (19–20/20) throughout. The bit-exact tier’s outputs are byte-identical to the quantised reference (texts and step counts match 20/20 on GSM8K and HumanEval).

Configurationtok/stier
W4A16 experts (reference loop)117.5—
+ entropy dedup132.4bit-exact
+ fused entropy & Gumbel-max165.1accuracy-parity
+ fused RMSNorm (turbo)200.3accuracy-parity

Table 3.

Results

End-to-end. Table 2 reports the full stack against a bfloat16 baseline, both arms run in the same process on the reference model.generate loop, GSM8K/HumanEval/MATH at n=100n=100 each. The bit-exact tier alone reaches 1.81.8–2.0×2.0\times while producing byte-identical output to the quantised reference decoder; the accuracy-parity “turbo” configuration reaches 2.42.4–2.7×2.7\times, with an accuracy delta of +1/−1/−1+1/-1/-1 across the 300 turbo questions. At n=100n=100 the 95% binomial confidence interval on each pass rate has a half-width of 4–7 points (±3.8 at 96%, ±6.6 at 87%), so these differences are well inside sampling error; we do not claim equality beyond that resolution. Throughput is measured as decoded output tokens per wall-clock second (diffusion commits multiple tokens per forward).

Table 2: End-to-end decode throughput on GB10, single user, reference model.generate; each cell is tok/s (speedup over the bfloat16 baseline). Quantising the experts alone gives ∼1.6\sim1.6–1.7×1.7\times; the two logits-pipeline tiers of § 3 contribute the rest, reaching 2.42.4–2.7×2.7\times. Accuracy is preserved at every step: on a same-session n=100n=100 comparison, bfloat16 scores 96/89/88 and turbo scores 97/88/87 (GSM8K / HumanEval / MATH, exact-match or pass@1); the bit-exact tier is token-identical to the W4A16 decoder, and W4A16 quantisation is accuracy-neutral versus bfloat16 (96→96, 87→87 on GSM8K / MATH).

ConfigurationGSM8KHumanEvalMATH
bfloat16 baseline70.484.369.7
W4A16 experts (reference loop)118.1 (1.68×)140.6 (1.67×)110.2 (1.58×)
+ bit-exact pipeline2134.4 (1.91×)167.6 (1.99×)123.9 (1.78×)
+ turbo pipeline186.2 (2.64×)224.9 (2.67×)168.4 (2.42×)

Table 2.

Attribution. Table 3 isolates each pipeline optimisation on GSM8K (n=20n=20, same model load). Starting from the quantised model at 117.5 tok/s, the bit-exact entropy dedup adds +13%, the fused entropy and Gumbel-max sampler take it to 165.1, and fused norms to 200.3—cumulative 1.7× on top of quantisation, entirely from the logits/normalisation pipeline rather than the backbone. The same stack contributes 1.58× in Table 2 (118.1 → 186.2). The two numbers come from different query sets (the first 20 versus the first 100 GSM8K problems) and separate runs; per-query throughput varies by more than 2× with generation length and step count, so we read the pipeline tiers’ contribution as roughly 1.6–1.7× rather than a single value.

Bit-exactness. On both GSM8K and HumanEval (n=20n=20), the bit-exact tier produces token-identical output and identical per-query step counts to the quantised reference decoder (20/20), confirming it changes speed without changing the model’s behaviour.

Generality Across MoE Diffusion Models

To test whether these findings are specific to DiffusionGemma, we profile two further masked-diffusion LMs on the same device: LLaDA2.0-mini [2], an independently built diffusion MoE (256 experts, top-8, 20 layers, V=157,184V=157{,}184), and LLaDA-8B [12], a dense diffusion model (V=126,464V=126{,}464). Table 4 shows the logits-pipeline share of decode: it is a first-order cost on both MoE models (28.8–37.1%) but negligible on the dense model (3.6%). The mechanism is the same as § 2: on a dense model the per-canvas projections are compute-bound GEMMs and the backbone dominates; MoE routing makes each expert bandwidth-bound, so once experts are cheap the pipeline surfaces. Vocabulary size is a confound we cannot fully remove with three models: LLaDA-8B’s VV is about half of DiffusionGemma’s. A smaller VV alone would shrink the pipeline’s share by at most about half, not ten-fold, and LLaDA2.0-mini (V=157KV=157\mathrm{K}, close to LLaDA-8B’s) still shows a 28.8% share, though under a heavier top-pp sampler than LLaDA-8B’s greedy loop; we therefore attribute most of the gap to the backbone regime, with vocabulary and sampler as secondary factors. A controlled sweep over VV at fixed architecture is left to future work.

ModelArch.Pipeline share
DiffusionGemma-26B (W4)MoE37.1%
LLaDA2.0-mini (bf16)MoE28.8%
LLaDA-8B (bf16)dense3.6%
LLaDA2.0-minibf16W4A16
GSM8K38/50 @ 18.741/50 @ 61.8 (3.3×)
HumanEval38/50 @ 57.335/50 @ 191.9 (3.4×)
MATH-50028/50 @ 20.327/50 @ 68.7 (3.4×)

Table 4. Logits-pipeline share of per-forward decode across three masked-diffusion LMs (GB10). It is first-order on MoE models, negligible on the dense one. Below: porting the fused W4A16 MoE kernel to LLaDA2.0-mini gives 3.3–3.4× over its bfloat16 reference at accuracy parity (n=50n=50; Δ\Delta within run-to-run sampling noise).

The fused W4A16 MoE kernel (grouped GEMM, SiLU epilogue) transfers directly: on LLaDA2.0-mini it is 3.3–3.4× faster than the reference per-expert loop at accuracy parity across GSM8K, HumanEval, and MATH-500 (Table 4, lower), reproducing the DiffusionGemma result on a second, independently trained MoE. The pipeline optimisations, however, transfer only in proportion to the pipeline’s composition. DiffusionGemma’s loop is sort-free (duplicated entropy plus multinomial), so it fuses cleanly for the +40% of § 3. LLaDA2.0-mini instead top-pp-filters over a 157K vocabulary every step; Gumbel-max removes its multinomial but the top-pp sort remains, so the accuracy-neutral variant nets ≈0. In a same-session run on MATH-500 (n=50n=50), W4A16 alone gives 26/50 at 67.8 tok/s and the Gumbel-max sampler with top-pp kept gives 27/50 at 71.3; across two runs the difference between them ranged from −5% to +5%, which we read as no reliable gain. Dropping top-pp removes the sort but bought no reliable speed (70.5 tok/s) and scored lower in both runs (24/50 and 23/50, versus 26–27/50 for every other arm), so we do not recommend it. The pipeline is first-order across MoE diffusion, but which of its sub-flows depends on the sampling algorithm.

Related Work

Work on accelerating diffusion LLMs has mostly reduced the number or cost of forward passes: approximate KV caching and confidence-aware parallel decoding [16, 10]. These leave the per-step logits pipeline untouched and are complementary to ours. Closest in diagnosis, Lou et al. [2026] also find that sampling takes a large share of dLLM inference latency (up to 71% on an A6000) and propose a hardware remedy; we reach a compatible conclusion on commodity hardware and address it in software. Fused sampling kernels are standard in autoregressive serving frameworks such as FlashInfer and vLLM [17, 6], and FlashSampling [13] likewise uses Gumbel-max for exact single-pass sampling. Those systems target autoregressive decoding, where sampling covers one position per step; our point is that diffusion decoding multiplies this cost by the canvas and step count, which makes the same techniques first-order there.

Discussion and Limitations

The takeaway is not any single kernel but where to look: in masked-diffusion serving, the logits pipeline is a first-order cost that grows with the number of denoising steps, and much of it is redundant or fusable without touching model behaviour. Optimisations that are pointless for AR serving—deduplicating a full-vocabulary reduction, one-pass entropy, single-reduction sampling—pay back in proportion to SS here.

Our study spans three diffusion LMs but one device and single-user, batch-size-1 decoding; the O(S)O(S) analysis is general, but the absolute breakdown will shift on datacenter GPUs with more compute headroom and on models with smaller vocabularies. Batching is the most consequential open question: it amortises expert weight reads across requests, but the logits pipeline grows linearly with batch size, so we expect its share to stay first-order or grow under batching, though we have not measured it. We measured the eager execution path. We attempted a torch.compile baseline, but the reference loop’s compiled path recompiles for every prompt length and exceeded the compiler’s recompilation limit on our benchmark mix, so we report no compiled numbers; a compiled path would fuse more of the redundant work and shrink the bit-exact tier’s margin, while the accuracy-parity kernels remove passes a compiler cannot. Finally, the accuracy-parity tier is lossless in distribution and in benchmark accuracy but not bit-exact by construction; we keep the two tiers separate so that applications requiring reproducible sampling can use the bit-exact path. We hope the profiling result encourages dLLM serving work to treat the sampling pipeline as a primary target rather than an afterthought.

References

  1. [1]Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems, 2021.arxiv.org/abs/2107.03006
  2. [2]Tiwei Bie, Maosong Cao, Kun Chen, et al. LLaDA2.0: Scaling up diffusion language models to 100b. arXiv preprint arXiv:2512.15745, 2025.arxiv.org/abs/2512.15745
  3. [3]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. In International Conference on Learning Representations, 2023.arxiv.org/abs/2210.17323
  4. [4]Emil Julius Gumbel. Statistical theory of extreme values and some practical applications. National Bureau of Standards Applied Mathematics Series, 33, 1954.
  5. [5]Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations, 2017.arxiv.org/abs/1611.01144
  6. [6]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), 2023.arxiv.org/abs/2309.06180
  7. [7]Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for on-device llm compression and acceleration. In Proceedings of Machine Learning and Systems (MLSys), 2024.DOI
  8. [8]Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In International Conference on Machine Learning, 2024.arxiv.org/abs/2310.16834
  9. [10]Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. dKV-Cache: The cache for diffusion language models. In Advances in Neural Information Processing Systems (NeurIPS), 2025.arxiv.org/abs/2505.15781
  10. [11]Chris J Maddison, Daniel Tarlow, and Tom Minka. A* sampling. In Advances in Neural Information Processing Systems, 2014.
  11. [12]Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2502.09992
  12. [13]Tomas Ruiz, Zhen Qin, Yifan Zhang, Xuyang Shen, Yiran Zhong, and Mengdi Wang. FlashSampling: Fast and memory-efficient exact sampling. arXiv preprint arXiv:2603.15854, 2026.arxiv.org/abs/2603.15854
  13. [14]Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2406.07524
  14. [15]Philippe Tillet, Hsiang-Tsung Kung, and David Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, 2019.DOI
  15. [16]Chengyue Wu, Hao Zhang, Shuchen Xue, et al. Fast-dLLM: Training-free acceleration of diffusion LLM by enabling KV cache and parallel decoding. In International Conference on Learning Representations (ICLR), 2026.arxiv.org/abs/2505.22618
  16. [17]Zihao Ye, Lequn Chen, Ruihang Lai, et al. FlashInfer: Efficient and customizable attention engine for LLM inference serving. In Proceedings of Machine Learning and Systems (MLSys), 2025.arxiv.org/abs/2501.01005

Paper details

vlnc:W_etstqw8q
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents