betaFeedback

WorksComputer Science

High-Throughput Serving of Nemotron 3.5 Transcribe with Regime-Aware Megakernels and Kernel Fusion

Abstract

Nemotron 3.5 Transcribe ranks first among open-weight models on the Hugging Face Open ASR leaderboards. Serving it efficiently takes more than fast operators: continuous batching changes the amount and mix of work at every iteration, and the best fusion granularity changes with it. We present a serving engine that, for every pass, chooses between persistent megakernels and chains of fused kernels at crossovers measured offline. On an H100 SXM GPU, throughput of our accuracy-preserving serving peaks at 4,345x real time, 1.39x the peak of tuned vLLM, and is up to 4.39x faster at low concurrency; on B200 it peaks at 6,367x, 1.64x vLLM at the highest concurrency we measured. Word error rate (WER) is 1.91 to 1.93% against 1.88 to 1.93% for vLLM. Within a 400 ms budget for p95 end-to-end completion time, it serves 8x as many concurrent requests. Choosing per pass beats either level alone: 23% higher throughput than always using megakernels at 128 concurrent requests, and 9% higher than always using fused kernels at 8. Opt-in switches, including greedy MoE layer skipping, add 13 to 16% throughput on B200 for offline jobs that can accept higher WER.

CCS Concepts

• Computing methodologies → Parallel computing methodologies; Machine learning; Speech recognition.

Keywords

ML Inference, Megakernel, Kernel fusion, Speculative decoding, GPU performance modeling

Introduction

Speech transcription is a core operation in speech-to-speech (S2S) systems and modern user-facing applications. Nemotron 3.5 Transcribe is the leading open speech transcription model at the time of its release in October 2026. It combines speech and speaker encoders with a hybrid Mamba Transformer mixture-of-experts (MoE) based language model, supports transcription in 40 languages and processes recordings up to one hour long.

Although inference serving frameworks like vLLM [15] provide optimized kernels, continuous batching and speculative decoding, vLLM’s existing Nemotron 3.5 Transcribe integration leaves significant opportunities for optimization across operator boundaries.

Separate kernels for normalization, quantization, projections, expert routing and output processing introduce intermediate memory traffic and synchronization. Fusion can reduce these costs but constrain registers, occupancy and compute efficiency. The challenge is to choose fusion granularity as serving workload size changes.

We address this problem with a regime-aware serving design which reduces p95 end-to-end completion time by 1.50 to 4.10× on

H100 SXM, 1.45 to 2.19× on H100 NVL and 1.93 to 3.75× on B200 across 32 to 1,280 concurrent requests (Figure 1). Our approach combines persistent megakernels, which execute a model region within one launch, with specialized kernel chains that fuse selected operations while retaining independent launch configurations. We select between these implementations using measured behavior across token lengths. The resulting native serving engine also processes prompt and verification work together, adapts expert computation to routed populations and keeps speculative drafting and token selection on the GPU.

p95 end-to-end completion time across client concurrency

Figure 1. p95 end-to-end completion time across client concurrency.

Our major contributions include:

  • A design methodology for prefill-dominated, megakernel-receding architectures using regime-aware megakernels and kernel fusion.

  • An engine inside a production inference server that, for every pass, chooses between megakernels and fused kernels at measured crossovers.

  • Regime-aware kernels designed for Nemotron 3.5 Lightning apply to other models built on it.

  • Optimizations for applications that can tolerate a higher word error rate (WER), including greedy selection of MoE layers to skip within a WER budget.

Industrial context

As of 6 November 2026, it ranks first among open-weight models on the English, multilingual and long-form Hugging Face Open ASR leaderboards [19, 32] (5.33%, 4.62% and 10.98% WER) and first among 14 systems on VoiceArena’s speaker-attributed transcription benchmark [34].

Nemotron 3.5 Transcribe model

Nemotron 3.5 Transcribe [23] is NVIDIA’s language model for transcription. One model, steered by a natural-language instruction, produces verbatim or written-form, contextual, speaker-attributed and rich transcripts, translates speech and identifies the spoken language in 40 languages.

Its audio path is a Parallel Expert Encoder: a 32-layer Transformer ASR encoder (d=1,280d=1{,}280, 0.6B parameters) and a frozen Sortformer [27] diarization model (0.1B) run side by side on 128-bin log-mel features and the per-frame activities of up to eight speakers are added to the ASR states through a fixed sinusoidal kernel. A linear adapter projects the result to the LLM width (1,280 →\to 2,688). The LLM backbone, Nemotron 3.5 Lightning 30B-A3B, has 52 layers: 23 Mamba-2 [8], 23 MoE layers with 128 routed experts (top-6) and a shared expert and only 6 grouped-query attention (GQA) layers so 3.5B of its 31.6B parameters are active per token. A multi-token prediction (MTP) module (1.3B parameters) serves as the draft model for speculative decoding and a CTC [12] adapter (50M parameters) on the ASR encoder aligns words to audio. The complete model contains approximately 34B parameters, with approximately 4.3B active parameters including the audio-processing components (Figure 2).

Figure 2. Nemotron 3.5 Transcribe model architecture.

Target workload and serving requirements

We target transcription at high concurrency. Our benchmark is NVIDIA’s English Bias corpus (test split): 4,820 single-speaker recordings totaling 15.36 hours (3.5 to 58.8 s, 11.5 s on average, 25 words per transcript on average). The recordings are split evenly across four domains (telecom and finance, technical, alphanumeric codes and numbers, and general) and six US and UK accents, with an even gender split. Our primary evaluation uses the full Bias corpus; we additionally evaluate the complete LibriSpeech [26] test-clean and test-other splits. Expert and layer rankings for the opt-in mode (§ 7.1) come from a separate calibration set of 512 LibriSpeech dev utterances.

We report end-to-end RTFx, the seconds of audio transcribed per second over a whole pass including ramp-up and drain; latency per request (p95); and WER and character error rate (CER). We measure on single GPUs: H100 SXM (700 W), H100 NVL (400 W) and B200 (144 SMs, 1,000 W). Every comparison is a matched run on the same GPU.

Online serving maximizes concurrent requests while keeping p95 end-to-end completion time below 400 ms, within a 1-second S2S budget allocating 100 ms to TTS and 500 ms to the LLM. Offline serving maximizes RTFx across batch sizes.

Deployment and use cases

Nemotron 3.5 Transcribe was released as open weights on Hugging Face on 28 October 2026 under the OpenMDW-1.1 license, with BF16, FP8 and NVFP4 checkpoints. vLLM, SGLang, Hugging Face Transformers and NeMo-Speech.cpp support it, and inference providers including Baseten, Deeplnfra and Simplismart have hosted it since release day. NVIDIA’s reference service is a Rust inference server with OpenAI-compatible routes in front of a vLLM engine. It is packaged per GPU family, from Jetson Thor and DGX Spark to H100/H200 and GB200/GB300. Our engine runs in process inside that server through a C ABI with no Python in the serving path. It keeps the same API, word timestamps and speaker labels, and has parity tests against the vLLM reference. Our optimized serving engine will be available in a future release of NVIDIA Nemotron 3.5 Transcribe NIM [24].

One checkpoint serves two workloads. Offline, it transcribes call-center, meeting and media archives of up to an hour per recording, where transcription throughput (RTFx) decides deployment. Online, it is the recognizer in voice agents and speech-to-speech systems, where a short utterance must be transcribed within about 400 ms.

Regime-aware megakernel architecture

The vLLM baseline executes Nemotron 3.5 Transcribe at operator granularity: each operator, or a small group fused by torch.compile, is a separate kernel, launched by vLLM’s model runner or replayed from a CUDA graph, and consecutive kernels pass intermediate results through global memory. Our optimized engine replaces every component with one of two fusion levels:

  • A megakernel runs the whole region in one persistent launch.

  • Fused kernels run the region as a short chain of custom kernels, each fusing several operators, as CUTLASS [22] GEMM epilogues and chained with programmatic dependent launch (PDL) [21].

Why is one megakernel not enough?

Prior work on megakernel composition [3] found that a single megakernel is rarely the fastest design. That work describes each fused region by how its gain from a megakernel changes with batch size. A megakernel-stable region gains at every size. A megakernel-receding region gains up to a crossover and loses beyond it.

With in-flight batching, the batch changes at every iteration; we size each region by the work of one pass: tokens per encoder pass or rows per decoder step.

For a region which recedes beyond a crossover point, the engine keeps both a megakernel and a conventional implementation and switches between them. Conventional means one kernel launch per operation inside our engine, with CUTLASS or cuBLASLt GEMMs and our custom CUDA kernels.

Table 1 lists the result. Two megakernels are stable and always run. Five are receding: three switch at runtime and two are selected by the serving configuration.

MegakernelClassSelection threshold
ASR encoderRecedingT = 384 tokens per encoder pass (input T × 1,280) (runtime)
SortformerRecedingT = 384 tokens per encoder pass (input T × 512) (runtime)
PrefillRecedingPrompt-only execution; at most 64 requests and 2,048 packed tokens per launch. Configuration-dependent availability.
Decoder - MambaRecedingVerification specialization supporting at most four rows per request.
Decoder - GQAStableRetained across the evaluated eligible shapes.
Decoder - MoERecedingR = 448 rows per decoder step (input R × 2,688) (runtime)
MTP draftStableRetained across the evaluated eligible shapes.

Table 1. Megakernel regimes and selection thresholds.

Nemotron 3.5 Transcribe is a prefill-dominated megakernel-receding architecture

The five megakernels, classified as megakernel-receding, dominate the end-to-end computation time, which makes the Nemotron 3.5 Transcribe model a megakernel-receding architecture.

The thresholds were measured offline and are applied from the pass size alone. At B128 most passes are large, so the encoder and MoE take the conventional path, while the GQA and draft megakernels run on every pass. At low load, and while the batch fills or drains, the megakernels take over.

Neither level alone matches this choice (Table 2). Megakernels slow down large passes (7.66 against 6.05 ms per encoder pass at 1,024 tokens), and fused kernels slow down small ones (549 launches instead of four per encoder pass). A megakernel-only design would also need separate prompt steps, because the prefill megakernel accepts only prompt-only batches, so each iteration would stream the 29.8 GB of FP8 expert weights twice.

RegionPass size: speedup
ASR encoder (tokens)64: 1.8× · 128: 1.6× · 256: 1.25× · 384: 1.0× · 1,024: 0.79×
MoE layer (rows)64: 1.16× · 128: 1.12× · 256: 1.07× · 448: 1.0× · 1,024: 0.88×

Table 2. Megakernel speedup over fused kernels on H100; values above 1× favor megakernels.

The workload is prefill-dominated at serving loads. Prefill rows consume 24.1% to 46.2% of end-to-end time across batch sizes 1 to 512. Decode rows dominate prefill rows only at batch size 1 where decode consumes 35.3% while prefill is at 24.1%.

Speech encoder across load regimes

In the speech encoder, each iteration encodes the audio of all requests it admits in one pass. One second of audio becomes 12.5 encoder tokens, so the size of a pass is the amount of audio admitted in that iteration.

One encoder pass

Figure 3 shows a pass in B128 serving. A front-end kernel computes log-mel features for all requests. The two branches of the Parallel Expert Encoder [23] run concurrently: the ASR branch (32 layers, d = 1,280, FP8 projections) on the main stream and the Sortformer diarization branch (31 layers, d=512, BF16) on a side stream. A tail kernel adds the speaker activities to ASR states and projects the result to the LLM width (modality adapter). The CTC timestamp head runs last, on the lowest-priority stream, overlapping the decoder step; its output feeds the draft hints.

Figure 3. Encoder execution across the 384-token crossover; CTC span measured during B128 serving (median 1,690 tokens per pass).

The pass is queued behind the previous iteration’s MTP draft, so the host builds the next batch while the GPU encodes. In steady-state B128 serving, the 2,048-row prompt budget of a decoder step limits a pass to about 1,800 tokens: passes carry 1,218 to 1,798 tokens (median 1,690 or 135 s of audio) and take 8.8 ms of the 55 ms iteration on H100 NVL. Under light load, a pass holds one or a few requests; a 5- to 10-second utterance is 63 to 125 tokens.

Megakernels for small passes

We have designed two separate megakernels for ASR and Sortformer. Below 384 tokens, the ASR and Sortformer megakernels launch on separate streams with persistent grids of 94 and 38 CTAs, respectively.

In every CTA, a producer warp loads tiles of weights and activations with TMA into a six-stage shared-memory ring and two consumer warpgroups multiply them with wgmma instructions [25], in FP8 for ASR and BF16 for Sortformer.

Each layer is a sequence of phases separated by software grid barriers. Figure 4 illustrates the phases for ASR. The Sortformer megakernel follows the same plan and ends with the speaker head.

Figure 4. ASR layer phases in the megakernel and fused-kernel paths.

Small passes have too few output tiles to fill the GPU. At 64 tokens, the projections expose 10 to 40 output tiles. Under a one-working-CTA-per-SM mapping, this provides useful tile work for approximately 8 to 30% of 132 SMs. Split-K tile scheduling increases the available parallel work to 50 to 80 tasks, corresponding to approximately 53 to 85% of the 94 CTA ASR worker pool.

Each phase boundary costs a grid barrier instead of a kernel launch, and a whole pass takes four launches instead of 549. We synchronize at phase boundaries instead of scheduling tiles across operators. In tile-level dataflow, as in recent LLM megakernels [5, 31], each tile of the next operator waits on counters for only the tiles it reads, so consecutive phases can overlap. Our encoder passes leave little to overlap. Below 384 tokens a pass has at most three 128-row blocks, so each phase fits in one or two waves of tiles on

94 SMs, and every phase reads whole rows: LayerNorm needs all ten column tiles of a row and their split-K partial sums, GQA needs every key and value of a request, and the down-projection needs all 40 tiles of the 5,120-wide FFN row. The tiles that one next-phase tile reads are spread over most CTAs and finish at about the same time, so per-tile counters would start the next phase little earlier than a barrier, while adding release and acquire cost to every tile. Based on experiments, a megakernel with tile-level scheduling is 4 to 10% slower at every B128 shape than the megakernel with grid barriers: its dependency fences added latency to every task.

Kernel fusion for large passes

Beyond 384 tokens, each ASR layer runs as eight kernels (Figure 4). The four projections for the H100 GPU are CUTLASS SM90 FP8 GEMMs (TMA, warp-specialized, 128×128×128 tiles) whose epilogue visitor trees apply the per-channel weight scales; the output and FFN down-projections read the residual stream and add bias and residual in place, and the FFN up-projection applies bias, GELU and FP8 quantization for the next GEMM. Four custom kernels complete the layer: QK LayerNorm with RoPE, a variable-length FlashAttention-3 [29] that writes FP8 for the output projection and two LayerNorm-plus-FP8 kernels. The ASR kernels launch with programmatic dependent launch (PDL): each kernel is launched while its producer still runs and waits for the producer’s results so the chain of 258 kernels per pass has significantly fewer launch gaps.

The Sortformer branch runs as 289 kernels: cuBLASLt BF16 GEMMs, row kernels for RoPE, attention, bias with residual and LayerNorm, and GELU, and the speaker head. Instead of a fixed SM split, the two branches share all SMs: they are queued layer by layer on two streams of equal priority, so each branch fills the SMs the other leaves idle.

The front end and the speaker fusion are one fused kernel each, in both regimes.

Nemotron 3.5 Lightning backbone

Prefill and decode phases share the same Nemotron 3.5 Lightning backbone which results in 45% to 61% of the end-to-end computation time being consumed by this backbone.

Prefill and decode share one backbone

The backbone does two jobs:

  • Prefill processes a new request’s prompt once (the instruction, 85 audio embeddings and a short text suffix) to fill the request’s Mamba states and KV cache and produce its first token: about 99 rows for a typical 6.8 s request after the shared prefix is reused.

  • Decode extends the transcript. With speculative decoding, each decode pass verifies the last accepted token plus up to 15 drafted tokens per request, about 9 rows on average, and a request needs three to four passes: about 30 decode rows for the same request.

The engine runs both jobs in the same step: each step reads every weight once, for the prefill rows of the requests admitted in that iteration and the decode rows of all requests in flight.

Each second of input audio gives the backbone about 21 prompt tokens to prefill (12.5 audio embeddings and, on average, 8.6 instruction and template tokens) but yields only about 3.6 transcript tokens, which speculative decoding verifies in about 4.4 decode rows. Even after prefix reuse, prompt rows are 76% of the backbone’s rows, and although a transcript token costs about four times as much backbone time as a prompt token (60 against 16 μ\mus at B128), prefill takes 1.6 times as long as decode: 36% against 22% of serving time in tuned vLLM.

Prefill megakernel design

The prefill megakernel is one cooperative launch of 132 persistent CTAs, one per SM in the H100 GPU, that runs a packed batch of up to 64 prompts and 2,048 tokens through all 52 layers. The producer streams 128×128 FP8 tiles of weights and activations from HBM into a six-stage shared-memory ring (6 × 32 KB) with cp.async.bulk.tensor, and per-stage mbarriers carry the transaction byte counts. Its distinguishing choice is one pipeline for three operator families (Mamba-2, GQA and MoE) inside one launch (Figure 5).

Figure 5. Prefill megakernel phases and shared execution pipeline.

Phases end at software grid barriers. For MoE, expert tiles are swap-AB: the weight takes wgmma’s M side and the expert’s 8 to 128 tokens its N side, chosen per expert, so small expert populations no longer pad to 128 rows. Per-expert readiness counters replace the barrier between up and down projections.

For a fixed cohort, all new incoming requests are computed using the prefill megakernel. In-flight batching, prefill rarely runs alone as a monolithic prefill megakernel: at B128, 90% of engine steps mix the prompt rows of newly admitted requests with the decode rows of running ones, 9% carry only decode rows (the drain at the end of a pass) and 1% carry only prompt rows. New prompts join the mixed step following the encoder pass and take the same kernels as the decode rows; that is the decoder’s regime-aware design.

Decoder regime-aware design

The decoder kernel is a combination of three megakernels and corresponding custom fused CUTLASS/cuBLAS kernels (Figure 6). During runtime, the GQA megakernel is executed for each iteration while the Mamba and MoE megakernels are replaced by corresponding custom fused kernels depending on the tensor dimensions.

Figure 6. Regime-aware kernel composition for the decoder step.

The decode step launches its kernels layer by layer on a non-blocking stream at the highest priority. Our kernels and the CUTLASS GEMMs use programmatic dependent launch (PDL). The CUTLASS GEMMs are compiled with grid-dependency control. Prompt and decode rows share every weight read, so a B128 step streams its 29.8 GB of FP8 expert weights once for both kinds of rows. The dense projections are CUTLASS SM90 FP8 GEMMs (TMA, warp-specialized ping-pong mainloops, FP8 GEMMs with FP32 accumulator types, scale epilogues) and each layer starts with one row kernel for the residual add, RMSNorm and FP8 cast, which the MoE finalize absorbs after MoE layers. In a Mamba layer, the prompt rows run a fused convolution and chunk-parallel SSD on the main stream while a side stream, forked and joined with events, replays the decode rows: each request’s 1 MB of BF16 state per layer is read into registers, up to 16 rows are verified from 16-entry xx, dtdt and BB rings on CUDA cores, and the state is written back only when the ring fills. The Mamba megakernel fuses the same replay into one launch but handles at most four rows per request.

The GQA megakernel quantizes queries, writes keys and values into the paged FP8 cache and computes attention with mma.sync m16n8k16 BF16 tiles whose 16-row M dimension holds the 16 query heads that share a KV head: each warp handles one query row for all 16 heads, and one CTA per GQA block and KV head stages the keys and values in shared memory once. In an MoE layer, a cuBLAS BF16 router and a top-6 kernel feed the MoE megakernel below 448 rows: one persistent launch that writes each token’s FP8 copy straight to its six expert positions and the shared expert, runs both as swap-AB wgmma tiles of 8 to 64 tokens from nine 24 KB ring slots, and issues weight loads before a tile’s inputs are ready. From 448 rows, CUTLASS ptr-array grouped FP8 GEMMs take over with the shared expert as a 129th group and ReLU2 and FP8 fused into the FC1 epilogue; above 1,200 rows they use 128 × 128 tiles whose 2×1 TMA multicast halves the token-tile reads, and they stream expert weights at 2.5 to 2.8 TB/s.

One finalize kernel combines the experts and applies the next layer’s residual add, RMSNorm and FP8 cast, and the LM head is a cuBLAS BF16 GEMM with a greedy argmax kernel.

Optimizing the draft model

The draft model for MTP is megakernel-stable.

Draft model

The model’s multi-token prediction (MTP) module has one attention block and one MoE block with shared weights, predicts the next token from the backbone’s last hidden state and can be applied repeatedly. For each iteration, it proposes KK tokens per request and the next decoder step verifies them.

Mean acceptance length MAL⁡(K)=1+∑i=1K∏j=1ipj\operatorname{MAL}(K) = 1 + \sum_{i=1}^{K} \prod_{j=1}^{i} p_j counts emitted tokens per verification for KK drafts, accepting one bonus or correction token. Here, pjp_j is acceptance at position jj conditional on preceding drafts’ acceptance. Measured probabilities of 0.971, 0.977, 0.939, 0.931 and 0.931 yield MAL⁡(5)=5.41\operatorname{MAL}(5) = 5.41 tokens.

We use one megakernel for the draft model which works in composition with other custom fused kernels (Figure 7).

MTP draft kernels and CTC hint flow

Figure 7. MTP draft kernels and CTC hint flow; CTC timing uses a 1,152-token pass.

Draft megakernel design

The draft model runs as one megakernel. Each pass is one cooperative launch of 264 persistent CTAs, two per SM. Cooperative-groups grid barriers, up to 16 per pass, separate the phases of the MTP block.

The routed experts swap A and B: an expert’s weights fill the 64-row M side and up to 32 of its tokens the N side, on an eight-stage ring with 64 × 32 × 16 wgmma; above 4 rows, the shared expert joins this GEMM as one more expert, which removes two phases and their barriers. Attention puts the 16 query heads of a KV head on the M=16 side of mfma.sync m16n8k16: FP8 queries are converted straight into A-fragment registers, 32-key FP8 tiles of the cache are double-buffered into shared memory with cp.async and the CTA’s four warps split the context and merge their softmax maxima and sums in shared memory; long contexts are split into KV segments that one more phase merges. Activations stay under 3 MB per buffer at 128 rows, so a pass’s HBM traffic is its weights plus 512 B of cache per context token and request.

All drafts of an iteration, K×128K \times128 token IDs, reach the host in one 6 KB copy at the end. The gaps between passes total 0.13 ms per iteration, so we kept one launch per pass.

Custom fused kernels

Six custom kernels feed and finish the megakernel, and the CTC head drafts from the encoder side (Table 3). Three carry the main ideas: the prefix KV kernel computes only what the prompt rows need, the head GEMM reduces its logits to an argmax inside its epilogue, and the embedding kernel reads the head’s token IDs in place, which keeps all KK passes on the GPU. The head also never dequantizes: each input row is scaled so its absolute maximum maps to 448 (FP8 E4M3 maximum) and the scale is discarded, because a common positive scale across the vocabulary logits does not change their argmax.

KernelLaunch details
Prefix KV copyGrid: chunks × tensors × slots, 256 threads, PDL.
Embed + gatherrows × 128 or rows × 256 threads, PDL. One CTA per row.
Prefix KV kernel132 × 288 threads, one CTA per SM, 203 KB shared memory, cooperative. Two consumer warpgroups and one TMA loader warp; five-stage ring of 40 KB stages.
FP8 input quantizerows × 256 threads, 32 B shared memory.
Head GEMM, argmax epilogue132 × 384 threads, persistent, one CTA per SM. CUTLASS SM90 FP8 GEMM with a TMA warp-specialized mainloop and fast accumulation.
Winner select⌈rows/32⌉\lceil\mathrm{rows} / 32 \rceil × 256 threads, 2 KB shared memory.
CTC hint headRow kernels ⌈tokens/8⌉\lceil\mathrm{tokens} / 8 \rceil × 256; attention requests × 16 × ⌈frames/64⌉\lceil\mathrm{frames} / 64 \rceil × 128; lowest-priority stream.

Table 3. Custom kernels surrounding the MTP draft megakernel.

What optimizations did not land?

Optimizations impacting accuracy

As some optimizations trade WER for throughput, the engine exposes them only as opt-in features.

Expert masking removes the 32 least-routed experts per layer from routing, so their weights are never streamed (+3.0% throughput at +4.2% relative WER). Fewer experts per token means fewer routing rounds per row (top-5: +1.5% at +1.7%; top-4: +3.5% at +6.2%). Diarization bypass skips the Sortformer branch for single-speaker audio (+2.1% with no WER change on LibriSpeech dev).

Layer skipping removes whole MoE layers, chosen by greedy forward selection on 512 LibriSpeech dev utterances: each round adds the layer with the smallest WER increase and stops when calibration WER would exceed 2.55%. It selects 7 of 23 MoE layers (6, 8, 13, 20, 27, 38 and 43) for 11% more throughput; the Bias corpus is never used for selection.

We use the methodology in Algorithm 1 to select which MoE layers to skip.

Together the switches add 13 to 16% throughput on B200 but raise WER from 1.92% to 3.44 to 3.48%, so they suit only cost-bound offline jobs; a light preset of diarization bypass and top-5 routing adds 3.6% for 1.7% relative WER.

Optimizations that lowered performance

An expert-major MoE kernel, a persistent warp-specialized FP8 GEMM that kept token tiles resident, was 1 to 3% slower than the grouped CUTLASS GEMMs, because the SM clock fell to 0.82 to 1.18 GHz while weights streamed under the 400 W cap in H100 NVL.

Algorithm 1 SᴋɪᴘLᴀʏᴇʀs(LL, DD, XX)

Inputs: L=NL = N MoE layers; DD = calibration set; XX = allowed WER increase over the unskipped model

Output: SS = layers to skip

  1. S←∅S \leftarrow\varnothing; C←LC \leftarrow L

  2. while C≠∅C \ne\varnothing do

  3. H←\quad H \leftarrow empty min-heap ordered by error

  4. \quad for each layer ll in CC do

  5. el←WERD(S∪{l})−WERD(∅)\qquad e_l \leftarrow\mathrm{WER}_D(S \cup\{l\}) - \mathrm{WER}_D(\varnothing)

  6. \qquad push (el,l)(e_l,l) onto HH

  7. (e∗,l∗)←pop⁡(H)\quad(e^*,l^*) \leftarrow\operatorname{pop}(H)

  8. \quad if e∗>Xe^* > X: break

  9. S←S∪{l∗}\quad S \leftarrow S \cup\{l^*\}

  10. C←∅\quad C \leftarrow\varnothing

  11. while H≠∅H \ne\varnothing and top⁡(H).e≤X\operatorname{top}(H).e \le X do

  12. (e,l)←pop⁡(H)\quad(e,l) \leftarrow\operatorname{pop}(H)

  13. C←C∪{l}\quad C \leftarrow C \cup\{l\}

  14. return SS

INT4 experts halved the expert bytes, but their mixed-input grouped GEMMs took 400 to 630 μ\mus per FC1 call against 230 to 245 μ\mus in FP8.

Green-context SM partitions between the draft and the encoder never beat sharing all 132 SMs (the best split, 1,818 RTFx, was within noise of 1,809).

Decode megakernels inside vLLM sped decode-only steps by 7.6 to 16.4% but moved throughput by ×0.9965 because those steps carried 0.3% of the tokens.

A faster Sortformer could not shorten encoder passes in which the ASR branch was longer, and larger encoder GEMM tiles left SMs idle with only 90 tiles at 1,152 tokens. Two experts per token reached 5.4% WER and ran slower than three. Each failure traces to one of three principles, the lessons we would carry to another hybrid model:

  1. Power wall: the board always draws 400 W, so time follows energy, t≈(ebB+efΦ)/Pt \approx(e_b B + e_f \Phi)/P; fewer bytes or FLOPs shorten work and then diminishingly lowers the clock (expert-major kernel, INT4 experts, green contexts).

The classic roofline [36] says a kernel needs at least the longer of its memory time and its compute time:

t≥max⁡(BBWpeak,ΦFpeak).t \ge\max\left(\frac{B}{BW_{\mathrm{peak}}}, \frac{\Phi}{F_{\mathrm{peak}}}\right).

This assumes the GPU can read and arithmetic at their peaks simultaneously, so the shorter one hides under the longer one. The energy-roofline model of Choi et al. [6, 7] adds a third limit for power-capped processors: the total energy divided by the power available. When that limit binds, memory and compute compete for the same watts, so their times add instead of overlapping:

t≈ebB+efΦP.t \approx\frac{e_b B + e_f \Phi}{P}.

Here, ebe_b is the energy needed to move a unit of data, BB is the number of bytes moved, efe_f is the energy per unit of computation, Φ\Phi is the total computation and PP is the power available for use:

P=Pcap−Pstatic.P = P_{\mathrm{cap}} - P_{\mathrm{static}}.

Here, PcapP_{\mathrm{cap}} is the power cap of the GPU and PstaticP_{\mathrm{static}} is the minimum power consumed while the GPU is on. Over a full pass at 1,280 requests on H100 NVL, board power stays within 2% of the 400 W cap in 96% of 10 Hz samples, and the SM clock averages 1.32 GHz against a 1.785 GHz maximum.

(2) Critical path: parallel branches finish with the longest, so shortening the longer branch provides the most direct reduction in completion time.

(3) Filling the GPU: tiles must match the work; too few tiles leave SMs idle, which is why megakernels win below 384 tokens and 448 rows and larger encoder tiles lost above.

Evaluation

We evaluate end-to-end transcription on the English Bias corpus, comprising 4,820 recordings and 15.36 hours of audio, using single H100 SXM and H100 NVL GPUs with 700 W and 400 W power limits, respectively. Table 4 reports mean RTFx across three measured passes following a full-corpus warmup.

RTFxH100 SXM 700 WH100 NVL 400 W
Concurrent requestsvLLMOptimizedvLLMOptimized
324672,0527991,815
647882,9901,0252,451
1281,3723,7541,5843,033
2562,2414,1812,0543,316
5122,9734,2872,3363,351
7683,1274,3242,4103,369
1,0243,1164,3422,3853,374
1,2803,0204,3452,4023,381

Table 4. End-to-end transcription throughput (RTFx) on H100.

The vLLM baseline is NVIDIA’s production configuration of vLLM 0.29.0 (vllm/vllm-openai:v0.29.0, build 98dff2a8) with the NeMo speech plugin and CUDA 13.0.2. It serves the ModelOpt FP8 checkpoint (FP8 linear layers, experts and KV cache) with 3-token MTP speculation. It uses FlashInfer attention [37], CUTLASS dense GEMMs, FlashInfer-CUTLASS MoE and a BF16 Mamba state with 16-step ReplaySSM decoding. It also uses torch.compile with Inductor graph partitioning (decode sizes 512 to 768 precompiled), CUDA graphs up to 768 sequences, and the product’s encoder graphs, packed Sortformer and head graphs. Limits are 768 concurrent sequences, 19,456 batched tokens per step, a 20 GiB KV-cache budget, 0.88 GPU memory utilization and four API servers. B200 uses the same limits with its SM100 FP8 profile. Both systems serve the same FP8 checkpoint on one GPU behind the same HTTP client. The client prepares all 4,820 requests before starting the clock, sends them in descending duration order with a 4 s ramp, and computes RTFx over the whole pass, including ramp and drain.

Our accuracy-preserving serving achieves 1.38 to 4.39× higher throughput on SXM and 1.40 to 2.39× on NVL, reaching 4,345× and 3,381× RTFx, or 1.39× and 1.40× vLLM’s peak. At 128 concurrent requests, p95 end-to-end completion time decreases from 1.501 to 0.545 seconds on SXM and from 1.308 to 0.712 seconds on NVL. WER is 1.91 to 1.92%, compared with 1.88 to 1.93% for vLLM using the same checkpoint, workload, and text normalization; neither system is bit-exact, because batch composition changes FP8 GEMM tiling and reduction order.

vLLM is slower on H100 SXM than on H100 NVL below 256 requests because it is host-bound there: at 32 requests its mean GPU utilization is 45% on the SXM host and 78% on the NVL host, against 98% for our engine on both. The hosts differ in CPU generation (Zen3 EPYC 7413 versus Zen4 EPYC 9124), CPU allocation and kernel; we did not isolate which of these slows vLLM’s host path on SXM.

The regime-aware engine is 9% faster than fused kernels alone at 8 requests, shrinking to 2% at 1,280 as passes grow and 23% faster than megakernels alone at 128 requests, growing to 32% at 1,280.

At 32 requests, where passes fall on both sides of the switch points, it beats both (5% and 9%). Even with fused kernels only, our engine is 4.2× faster than vLLM at 32 requests and 1.41× at 1,280; that gap comes from the native host path without Python, GPU feature extraction, fused epilogues and deeper speculation with CTC hints.

B200 runs a separate SM100 build of the same engine, not a recompile. Blackwell has no wgmma, so every tensor-core path in the megakernels, the draft head and the fused GEMMs is reimplemented with UMMA (tcgen05.mma), which accumulates in tensor memory. The SM100 code allocates TMEM, waits on UMMA-completion barriers and drains accumulators before each epilogue; register-level mma.sync paths such as attention carry over. Persistent grids are sized from the SM count queried at start-up (144, not H100’s 132), so the concurrent ASR and diarization megakernels split the GPU 72/72 instead of 94/38. The scheduler, switch points (384 tokens, 448 rows), admission limits (255 resident requests; 6,144 prompt tokens and 8,192 rows per step), MTP depth and numerical recipes are unchanged. Re-measured on B200, the crossovers stay within 32 tokens and 32 rows of the H100 values, so we keep the same thresholds. One B200-specific change mattered at high load. The draft’s folded expert path now covers 256 rows instead of 128. That removed a slowdown once a draft pass exceeded 128 requests and raised throughput by 26 to 36% from 256 concurrent requests up (4,688× to 6,367× at 1,280).

On NVIDIA B200 (Table 5) (144 SMs, 1,000 W limit), our engine is 3.1×3.1 \times faster than vLLM at 32 concurrent requests and 3.9×3.9 \times at 64. The gap narrows to 1.6 to 1.8×1.8 \times from 512 requests, where our engine levels off above 6,200×6{,}200 \times real time (6,367× and 555 requests/s at 1,280) while vLLM is still climbing toward 3,894×3{,}894 \times. The p95 end-to-end completion time is 1.9 to 3.8×3.8 \times lower at every concurrency (0.21 s against 0.63 s at 32), and peak GPU memory is 56.0 GiB against 73.3 GiB. Our optimized serving holds WER at 1.92 to 1.93% against vLLM’s 1.89 to 1.91%, within 0.04 percentage points. Enabling all opt-in optimizations adds 13 to 16% throughput, reaching 7,414×7{,}414 \times at 1,280 requests, but raises WER to 3.44 to 3.48%.

RTFx on B200
Concurrent requestsvLLMOptimizedSpeedup
328702,6753.07×
641,0414,0113.85×
1281,5915,1243.22×
2562,5655,9582.32×
5123,5496,2201.75×
7683,7776,2971.67×
1,0243,8506,3501.65×
1,2803,8946,3671.64×

Table 5. End-to-end transcription throughput (RTFx) on B200.

With a 400 ms limit on p95 end-to-end completion time, our engine serves 64 concurrent requests on H100 SXM (0.365 s; 0.545 s at 128) and 128 on B200 (0.368 s; 0.595 s at 256), while vLLM meets the limit only up to 8 on H100 SXM and 16 on B200: 8× as many requests within the budget. Our optimized serving on H100 SXM is more performant than vLLM serving on B200, bridging the hardware improvement over a GPU generation with software serving optimizations only.

We additionally evaluate the complete LibriSpeech [26] test-clean and test-other splits: 5,559 recordings totaling 10.75 hours, with a mean duration of 6.96 seconds compared with 11.47 seconds for the Bias corpus. Requests follow a frozen shuffled order, and results are averaged over two measured passes after a full-corpus warmup, retaining the same end-to-end timing boundaries. Across 32 to 1,280 concurrent requests, our regime-aware serving achieves 1.60×1.60 \times to 3.52×3.52 \times vLLM’s throughput on H100 SXM 700 W and 1.82×1.82 \times to 3.90×3.90 \times on B200, reaching 3,739×3{,}739 \times and 5,219×5{,}219 \times real time, respectively. At 128 concurrent requests, p95 end-to-end completion time decreases from 1.587 to 0.454 seconds on H100 SXM and from 1.356 to 0.328 seconds on B200. Combined WER is 2.437 to 2.457%, compared with 2.449 to 2.485% for vLLM. These results extend the throughput and latency benefits to a public corpus with shorter recordings and shuffled arrivals, supporting the applicability of the serving design beyond the original industrial workload.

Related work

Persistent kernels keep a computation resident across steps [10, 14], and recent LLM megakernels schedule tiles across operators in a single launch [5, 31], whereas path tracing found a monolithic megakernel harmful [16]. Following our earlier analysis [3], we run a megakernel only where a pass is small enough for it to win, and fused kernels above a measured crossover. Speculative decoding [4, 17, 18] drafts with a smaller model, extra decoding heads [2] or multi-token prediction modules [9, 11]; in hybrid models, verification must also restore the state-space states [35].

Mixing prompt and decode rows in one step follows chunked prefill in Sarathi-Serve [1], which vLLM also uses; we apply it to a hybrid model whose verification rows also need Mamba state replay. Megakernel serving engines [31, 33] mostly target decode; we target a prefill-dominated workload and switch per pass. RaMP [30] dispatches MoE kernels from the live routing histogram, whereas we dispatch from pass size alone.

State-space and hybrid Mamba-Transformer models [8, 13, 20] and speech LLMs such as Nemotron 3.5 Transcribe [23] are usually served with stacks tuned for chat, while ASR serving has centered on batched inference for encoder-decoder models such as Whisper [28]. We know of no published study of serving a prefill-dominated hybrid speech LLM with mixture-of-experts layers.

Open problems and conclusion

Our regime-aware engine serves Nemotron 3.5 Transcribe at 2.74 times the audio per second of tuned vLLM at 128 concurrent requests on H100 SXM (1.38 to 4.39× across 32 to 1,280), cuts p95 end-to-end completion time from 1.50 s to 0.55 s, and cuts GPU time per audio hour by 63% at 128 requests and 28% at peak; opt-in switches add 13 to 16% on B200. We expect the lesson to carry to other hybrid models on power-capped GPUs: when the board draws its limit, speed comes from fewer bytes and fewer operations, not from better overlap.

This work raises three open problems:

  1. The crossover points are determined offline. Different routing and admission limits may move the crossover points at runtime. What is the optimal online approach to determine crossover points?

  2. Our approach of regime-aware megakernels and kernel fusion results in dynamic granularity at the operator level. Is this the optimal performance model for all megakernel-receding architectures?

  3. Nemotron 3.5 Transcribe is a prefill-dominated megakernel-receding architecture. How do the optimization strategy and operator granularity change depending on different components becoming dominant?

References

References

  1. [1]Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Rajee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 117–134. https://www.usenix.org/conference/osdi24/presentation/agrawal (Cited on p. 9.)arxiv.org/abs/2403.02310
  2. [2]Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). PMLR, 5209–5235. https://proceedings.mlr.press/v235/cai24b.html (Cited on p. 9.)
  3. [3]Aditya Chatterjee, Anand Joseph X. M, and Sanjay Singh Choudhary. 2026. One Megakernel is Not Enough: Predicting Where to Cut for Streaming Inference. arXiv:2610.35336. https://arxiv.org/abs/2610.35336 (Cited on pp. 3 and 9.)
  4. [4]Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating Large Language Model Decoding with Speculative Sampling. arXiv:2302.01318. https://arxiv.org/abs/2302.01318 (Cited on p. 9.)
  5. [5]Xinhao Cheng, Zihao Zhang, Yu Zhou, Jianan Ji, Jinchen Jiang, Zepeng Zhao, Ziruo Xiao, Zihao Ye, Yingyi Huang, Ruihang Lai, Hongyijun, Bohan Hou, Mengdi Wu, Yixin Dong, Anthony Yip, Zihao Ye, Songting Wang, Wenqin Yang, Xupeng Miao, Tianqi Chen, and Zhihao Jia. 2026. MPK: A Compiler and Runtime for Mega-Kernelizing Tensor Programs. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26). USENIX Association, Seattle, WA, 1909–1926. https://www.usenix.org/conference/osdi26/presentation/cheng (Cited on pp. 4 and 9.)arxiv.org/abs/2512.22219
  6. [6]Jee Whan Choi, Daniel Bedard, Robert Fowler, and Richard Vuduc. 2013. A Roofline Model of Energy. In 2013 IEEE 27th International Symposium on Parallel and Distributed Processing (IPDPS). IEEE, 661–672. doi:10.1109/IPDPS.2013.77 (Cited on p. 7.)
  7. [7]Jee Whan Choi, Marat Dukhan, Xing Liu, and Richard Vuduc. 2014. Algorithmic Time, Energy, and Power on Candidate HPC Compute Building Blocks. In 2014 IEEE 28th International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 447–457. doi:10.1109/IPDPS.2014.54 (Cited on p. 7.)DOI
  8. [8]Tri Dao and Albert Gu. 2024. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). PMLR, 10041–10071. https://proceedings.mlr.press/v235/dao24a.html (Cited on pp. 2 and 9.)arxiv.org/abs/2405.21060
  9. [9]DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. arXiv:2412.19437. https://arxiv.org/abs/2412.19437 (Cited on p. 9.)
  10. [10]Greg Diamos, Shubho Sengupta, Bryan Catanzaro, Mike Chrzanowski, Adam Coates, Erich Elsen, Jesse Engel, Awni Hannun, and Sanjeev Satheesh. 2016. Persistent RNNs: Stashing Recurrent Weights On-Chip. In Proceedings of the 33rd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 48). PMLR, New York, New York, USA, 2024–2033. https://proceedings.mlr.press/v48/diamos16.html (Cited on p. 9.)
  11. [11]Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. 2024. Better & Faster Large Language Models via Multi-token Prediction. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). PMLR, 15706–15734. https://proceedings.mlr.press/v235/gloeckle24a.html (Cited on p. 9.)arxiv.org/abs/2404.19737
  12. [12]Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks. In Proceedings of the 23rd International Conference on Machine Learning (ICML ’06). Association for Computing Machinery, New York, NY, USA, 369–376. doi:10.1145/1143844.1143891 (Cited on p. 2.)DOI
  13. [13]Albert Gu and Tri Dao. 2024. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. In Proceedings of the First Conference on Language Modeling (COLM 2024). OpenReview, Philadelphia, PA, USA. https://arxiv.org/abs/2312.00752 (Cited on p. 9.)
  14. [14]Kshitij Gupta, Jeff A. Stuart, and John D. Owens. 2012. A Study of Persistent Threads Style GPU Programming for GPGPU Workloads. In 2012 Innovative Parallel Computing (InPar). IEEE, San Jose, CA, USA, 1–14. doi:10.1109/InPar.2012.6339596 (Cited on p. 9.)DOI
  15. [15]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23). Association for Computing Machinery, New York, NY, USA, 611–626. doi:10.1145/3600006.3613165 (Cited on p. 1.)DOI
  16. [16]Samuli Laine, Tero Karras, and Timo Aila. 2013. Megakernels Considered Harmful: Wavefront Path Tracing on GPUs. In Proceedings of the 5th High-Performance Graphics Conference (HPG ’13). Association for Computing Machinery, New York, NY, USA, 137–143. doi:10.1145/2492045.2492060 (Cited on p. 9.)DOI
  17. [17]Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast Inference from Transformers via Speculative Decoding. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202). PMLR, 19274–19286. https://proceedings.mlr.press/v202/leviathan23a.html (Cited on p. 9.)arxiv.org/abs/2211.17192
  18. [18]Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). PMLR, 28935–28948. https://proceedings.mlr.press/v235/li24bt.html (Cited on p. 9.)arxiv.org/abs/2401.15077
  19. [19]Maryam Motamedi, Piotr Żelasko, Kunal Dhawan, Nikolay Karpov, Nune Tadevosyan, Meline Mkrtchyan, Nithin Rao Koluguri, and Francesco Ciannella. 2026. NVIDIA Nemotron 3.5 Transcribe and Granary v2: Intelligent Multilingual Transcription You Can Prompt and Adapt. NVIDIA article on Hugging Face. Retrieved November 15, 2026 from https://huggingface.co/blog/nvidia/nemotron-transcribe-granary (Cited on p. 1.)
  20. [20]NVIDIA. 2025. Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models. arXiv:2504.03624. https://arxiv.org/abs/2504.03624 (Cited on p. 9.)
  21. [21]NVIDIA. 2026. CUDA C++ Programming Guide. Retrieved November 14, 2026 from https://docs.nvidia.com/cuda/cuda-c-programming-guide/ (Cited on p. 2.)
  22. [22]NVIDIA. 2026. CUTLASS: CUDA Templates for Linear Algebra Subroutines. Retrieved November 14, 2026 from https://github.com/NVIDIA/cutlass (Cited on p. 2.)
  23. [23]NVIDIA. 2026. Nemotron 3.5 Transcribe Technical Report. Technical Report. NVIDIA. https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3.5-Transcribe-Technical-Report.pdf (Cited on pp. 2, 3, and 9.)
  24. [24]NVIDIA. 2026. NVIDIA Nemotron 3.5 Transcribe NIM. NVIDIA NGC Catalog. Retrieved November 15, 2026 from https://catalog.ngc.nvidia.com/orgs/nim/containers/nemotron-transcribe/latest (Cited on p. 2.)
  25. [25]NVIDIA. 2026. Parallel Thread Execution ISA. Retrieved November 14, 2026 from https://docs.nvidia.com/cuda/parallel-thread-execution/ (Cited on p. 3.)
  26. [26]Vassil Panayotov, Guoguo Chen, Dan Povey, and Sanjeev Khudanpur. 2015. LibriSpeech: An ASR Corpus Based on Public Domain Audio Books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 5206–5210. doi:10.1109/ICASSP.2015.7178964 (Cited on pp. 2 and 9.)DOI
  27. [27]Taejin Park, Ivan Demidov, Kunal Dhawan, Weiqing Wang, He Huang, Nithin Rao Koluguri, Krishna C. Puvvada, Jagadeesh Balam, and Boris Ginsburg. 2025. Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267). PMLR, 48153–48169. https://proceedings.mlr.press/v267/park25h.html (Cited on p. 2.)
  28. [28]Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust Speech Recognition via Large-Scale Weak Supervision. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202). PMLR, 28492–28518. https://proceedings.mlr.press/v202/radford23a.html (Cited on p. 9.)
  29. [29]Jay Shah, Ganesh Bikhchandani, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. 2024. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision. In Advances in Neural Information Processing Systems, Vol. 37. Curran Associates, Inc., 68658–68685. doi:10.52202/97917-2193 (Cited on p. 4.)arxiv.org/abs/2407.08608
  30. [30]Vyom Sharma and Debajyoti Datta. 2026. RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Experts. arXiv:2604.26039. https://arxiv.org/abs/2604.26039 (Cited on p. 9.)
  31. [31]Benjamin Spector, Jordan Juravsky, Stuart Sul, Owen Dugan, Dylan Lim, Dan Fu, Simran Arora, and Chris Ré. 2025. Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B. Hazy Research Blog. Retrieved November 14, 2026 from https://hazyresearch.stanford.edu/blog/2025-05-27-no-bubbles (Cited on pp. 4 and 9.)
  32. [32]Vaibhav Srivastav, Steven Zheng, Eric Bezzam, Eustache Le Bihan, Nithin Rao Koluguri, Piotr Żelasko, Somshubra Majumdar, Adel Moumen, and Sanchit Gandhi. 2025. Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation. arXiv:2510.06961. https://arxiv.org/abs/2510.06961 (Cited on p. 1.)
  33. [33]Xiaochun Tong, Conway Zhu, and Donglu Wang. 2026. Inside the Megakernel Serving Engine for North Mini Code. Cohere Blog. Retrieved November 14, 2026 from https://cohere.com/blog/megakernels (Cited on p. 9.)
  34. [34]Voice Arena. 2026. Voice Arena Diarization Bench: Joint Speaker Diarization and ASR Evaluation with cpWER. Retrieved November 14, 2026 from https://voicearena.com/research/diarization-bench/diarization-asr-track-cpwer-results (Cited on p. 2.)
  35. [35]Junxiong Wang, Daniele Paliotta, Avner May, Alexander M. Rush, and Tri Dao. 2024. The Mamba in the Llama: Distilling and Accelerating Hybrid Models. In Advances in Neural Information Processing Systems, Vol. 37. Curran Associates, Inc., 62432–62457. doi:10.52202/97917-1996 (Cited on p. 9.)arxiv.org/abs/2408.15237
  36. [36]Samuel Williams, Andrew Waterman, and David Patterson. 2009. Roofline: An Insightful Visual Performance Model for Multicore Architectures. Commun. ACM 52, 4 (April 2009), 65–76. doi:10.1145/1498765.1498785 (Cited on p. 7.)DOI
  37. [37]Zihai Zhao, Leq qn Chen, Ruihang Lai, Wuwei Lin, Yining Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, and Luis Ceze. 2025. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving. In Proceedings of the Eighth Conference on Machine Learning and Systems (MLSys 2025), Vol. 7. MLSys, Santa Clara, CA, USA, 17 pages. https://arxiv.org/abs/2501.01005 (Cited on p. 8.)

Paper details

vlnc:W_j73pksxe
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents