EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL

Introduction

Low-rank adapters (LoRA) (Hu et al., 2022) have become the standard way to specialize a large language model without copying it. An adapter adds a low-rank update $BA$ to each frozen weight matrix $W$ of the model, where $A$ projects the layer’s input down to a small rank $r$ (typically 16–128) and $B$ projects it back up; together they hold a fraction of a percent of the model’s parameters. One base model can therefore carry thousands of fine-tunes. Multi-tenant serving engines exploit this by keeping a single copy of the base in GPU memory and batching the requests of many tenants, the users or applications that each bring their own adapter [5, 52]. Adapters are also what LoRA reinforcement learning (RL) trains: LoRA matches full fine-tuning at a fraction of the memory [49], and several services now train adapters for many users on shared bases [53, 38] (Yu et al. 2026a).

At the same time, base models are moving to low precision. NVIDIA’s Ada (e.g., L40S) and Hopper (H100 and H200) GPUs compute in 8-bit floating point (FP8) at twice their

rate in the 16-bit bf16 format (Micekievicius et al., 2022), and Blackwell GPUs compute 4-bit formats such as NVFP4 at four times that rate [48, 40]. Serving systems therefore quantize the base model’s general matrix multiplications (GEMMs) to these formats [27, 7] (Lin et al. 2025), and RL frameworks increasingly generate their training samples, called rollouts, with FP8 inference engines [30, 3] (Xi et al. 2026). Lower-precision bases make inference cheaper, but the cost of serving LoRA adapters remains largely unchanged in current engines.

The reason is that the adapter path stays outside the low-precision GEMM. Merging an adapter into a quantized base rounds the update onto the base’s coarse grid and can erase it [35] (Li et al. 2026), so engines keep adapters separate and compute them in their own GPU kernels beside the base GEMM [21, 5]. During decoding, when the engine generates one token per request at each step, this adapter path is a chain of small dependent kernels whose latency does not shrink with the base. The cheaper the base, the larger the adapter’s share of each step Figure 2.

Existing systems address other parts of the problem. Punica and S-LoRA made batching many adapters practical [5, 52], and later systems decide where adapters are stored and which tenants share a batch ([4] (Wu et al. 2024; Li et al. 2024a)). SVDQuant fuses one low-rank branch, shared by every token, into a 4-bit GEMM ([23]), and LoRAFusion fuses several adapters into a bf16 GEMM for finetuning ([75]). However, none computes many tenants’ adapters in a low-precision GEMM for serving.

We present EpiLoRA, a plugin for the vLLM engine that computes every tenant’s adapter together with the low-precision base GEMM. A GEMM kernel streams the weights through its main loop and applies scales and writes its output in its epilogue. Our key observation is that the GEMM needs an adapter’s down-projection only in that epilogue, after the main loop has finished (Figure 1). During decoding, EpiLoRA therefore launches the down-projection first and its own FP8 or NVFP4 GEMM as a dependent kernel: the GEMM streams its weights while the down-projection runs, and then adds each tenant’s up-projection in its epilogue. The down-projection’s latency is then mostly hidden: on full-size layers with rank-64 adapters, at most about 2 $\mu$s of it stays exposed. On Ada GPUs, whose standard FP8 GEMM is slow during decoding, EpiLoRA instead fuses all resident adapters into two kernels of its own. The kernels quantize activations exactly as vLLM does, so the base GEMM receives the same FP8 inputs as in vLLM.

The low-precision base also benefits RL training, where frameworks that place the inference engine and the trainer on the same GPUs keep a separate bf16 copy of the base for the trainer ([15, 51, 33]). Because the FP8 engine then samples from a slightly different model than the trainer optimizes, FP8 rollout pipelines reweight each sample with an importance weight ([30]). EpiLoRA’s trainer instead computes the engine’s own FP8 weights, which saves enough memory to train a 27B model on one 48 GB L40S, where its bf16 weights alone do not fit, and lets one copy serve both the engine and the trainer. A matched engine also keeps its activations in bf16, as the trainer does, which removes most of the mismatch that remains once the weights agree.

We evaluate EpiLoRA on models from 8B to 72B parameters on Ada, Hopper and Blackwell GPUs, against the strongest configuration of each baseline we could build. Serving a single request (batch size 1), EpiLoRA takes 0.76–0.88× the time per output token of vLLM’s fastest configuration for 8B–9B models on Hopper GPUs, is faster than SGLang and TensorRT-LLM, and keeps its lead on a 72B model, on one GPU and across two. On the L40S it matches vLLM’s fastest configuration. With 32 tenants, whose batches are too large for the decode path, it delivers 1.05–1.09× vLLM’s throughput on 7 of 9 GPU–model pairs, from its adapter kernels on Hopper GPUs and fused adapters on the L40S.

In summary, our contributions are:

• The decode path, which removes most of an adapter’s latency by adding the up-projection in the GEMM’s epilogue and overlapping the down-projection with its main loop, and a simple condition that bounds when the overlap hides the down-projection (§ 3.1 and § 5).

• Multi-adapter kernels for FP8 and NVFP4 bases in a drop-in vLLM plugin that works with CUDA graphs and tensor parallelism (§ 3.2, § 5 and § 5.1).

• A LoRA RL system on the same low-precision base, with a decomposition of the engine–trainer mismatch showing what each design choice removes (§ 3.3 and § 5.2).

Background and motivation

Why engines do not merge adapters. Merging an adapter into the base ($W \rightarrow W + BA$) produces one model per adapter, so the engine can no longer batch tenants. On a low-precision base, merging is also lossy: rounding $W + BA$ to its grid changes only entries whose update exceeds half a quantization step, erasing small fine-tunes.

The cost of the separate adapter path. Each decoding step runs every linear layer (projection) of the model over the tokens of the current batch. Consider a projection over $M$ tokens, each belonging to a request that uses one of $L$ resident adapters. On an FP8 base, vLLM 0.28 runs four kernels for it ([5]): it quantizes the activations $x$ to FP8, runs the FP8 GEMM, computes the down-projection $h = xA^{\top}$ for each adapter’s tokens (the shrink), and adds the up-projection $hB^{\top}$ to the output (the expand). The activations are read twice and the output is written, read and written again, and the adapter weights are read by kernels that cannot share the GEMM’s work. None of these costs shrinks with the base. Tuning the LoRA kernels’ launch configurations, which we do for every vLLM baseline, reduces them, but with tuned kernels or vLLM’s dual-stream mode the adapter still adds 42–55% to a layer’s time at batch size 1 on Hopper GPUs (Table 1). On a Blackwell partition, one rank-64 adapter adds 16% to Qwen3-8B’s latency at batch size 1 with a bf16 base, 26% with an FP8 base and 43% with an NVFP4 base with vLLM’s default kernels (Figure 2).

$\mu$s $\downarrow$	L40S	H100	H200	Blackwell
Base GEMM, vLLM	413	126	104	286
Base GEMM, EpiLoRA	416	121	102	276
Adapter, vLLM	97	53	57	54
Adapter, EpiLoRA	46	12	10	17
EpiLoRA’s path	fused	overlap	overlap	overlap
Layer, EpiLoRA / vLLM $\downarrow$	0.91×	0.75×	0.70×	0.86×

Programmatic dependent launch. A GPU kernel runs as many thread blocks, which the GPU schedules onto its streaming multiprocessors (SMs). On Hopper and Blackwell GPUs, programmatic dependent launch (PDL) lets a kernel’s successor start before it finishes: the dependent’s thread blocks start as soon as the producer signals that they may, and wait for its results only at an explicit wait instruction. Inference engines use PDL to hide the launch latency between consecutive kernels, placing the wait at the start of the dependent kernel, so that only its setup overlaps the producer. EpiLoRA moves the wait to the end of the GEMM, just before its epilogue, so that the GEMM’s entire main loop overlaps the adapter’s down-projection.

LoRA RL on a low-precision engine. RL post-training alternates between generating rollouts with an inference engine and updating the model with a trainer, and it spends much of its time generating ([51] (Fu et al. 2025)). An FP8 inference engine and a bf16 trainer, however, produce slightly different token distributions because they compute at different precisions. Existing quantized rollout pipelines correct this mismatch with importance weights, the ratio of the trainer’s to the engine’s probability of each sampled token ([30, 44, 73]). Placing engine and trainer on one GPU also duplicates the base in memory: two bf16 copies of a 27B model exceed an 80 GB GPU, and moving the trainer’s copy to host memory (offloading) copies it every step and pins it there. As LoRA RL updates only the adapter, § 3.3 addresses both problems at their source.

Design

EpiLoRA has three parts: kernels that compute every adapter together with the low-precision base GEMM, a vLLM plugin that replaces vLLM’s LoRA layers, and an RL system on the plugin; Figure 3 compares it with existing systems.

Multi-adapter kernels

Fusing the adapters of many tenants into a low-precision GEMM is harder than fusing one low-rank branch that every token shares ((Li et al., 2025)), for three reasons. First, a GEMM divides its output into tiles, each computed by one thread block, and tokens of different adapters share a tile, so the epilogue must apply a different up-projection to each group of rows. Second, the kernels must quantize the activations exactly as the engine’s own GEMM does, so the base GEMM sees the same FP8 inputs. Third, during decoding the adapter path is a chain of small dependent kernels whose latency, rather than its memory traffic, sets its cost. We address the first two with two kernels, K1 and K2, that fuse the adapters’ down- and up-projections into the base GEMM, and the third with the decode path, which overlaps the down-projection with the GEMM’s main loop.

K1 and K2. K1 (Figure 4b) processes the tokens grouped by adapter, using the batching metadata vLLM already maintains. It quantizes the layer’s activations once for the base GEMM, with the same per-token scale and rounding as vLLM, and computes each adapter’s down-projection $h_\ell= xA_\ell^{\mathsf{T}}$ from the bf16 activations. Because a decode batch fills only a few dozen thread blocks, K1 splits the sum over the input dimension across several blocks (split-K) to use more of the GPU. K2 then runs the low-precision GEMM over the tokens in adapter order, so that each output tile meets only a few contiguous groups of adapters. Its epilogue applies the scales and, for each adapter group in the tile, adds $h_\ell B_\ell^{\mathsf{T}}$, reading only the tile’s slice of $B_\ell$. Adapters may also be stored in FP8 or NVFP4 and are dequantized in registers, which halves (FP8) or nearly quarters (NVFP4) the adapter bytes the epilogue reads, though 6 of 24 accuracy checks of stored adapters failed.

When fusion pays. The fused path removes vLLM’s separate adapter kernels but replaces the vendor’s GEMM with ours. This pays on Ada GPUs against their default FP8 GEMM, slow during decoding; on Hopper GPUs our GEMM only matches vLLM’s CUTLASS GEMM (Table 1), and the fused path runs the down-projection first.

The decode path. During decoding, the base GEMM streams megabytes of weights, while the adapter is a chain of dependent steps: K2’s epilogue cannot add $h_\ell B_\ell^{\mathsf{T}}$ before the down-projection has finished. On the H100 this chain costs about 13 $\mu$s per projection with vLLM’s tuned kernels (Table 1), although reading $A_\ell$ and $B_\ell$ takes only about a microsecond. For decode batches of up to 32 tokens, EpiLoRA therefore launches the down-projection first as a small kernel of its own, without PDL so that it starts only after $x$ is complete, and launches K2 as its dependent with PDL (§ 2; Figure 1c). A few extra thread blocks of this kernel, or for some shapes K2 itself, quantize the activations exactly as vLLM does. K2 starts as soon as the down-projection has begun, prefetches $B_\ell$, streams its weights once the activations are quantized, and waits for $h$ only before its epilogue. Reductions are summed in a fixed order, so results are deterministic for a given batch. The same design runs on Blackwell with NVFP4 bases.

Why not a second stream? vLLM’s dual-stream mode and TensorRT-LLM also overlap the adapter with the GEMM, by running the adapter on a second CUDA stream, a kernel queue that the GPU can run concurrently (Figure 1b). Because their GEMM writes its output before the adapter’s result is added, the two streams must join after the GEMM, and a separate kernel (vLLM’s up-projection, TensorRT-LLM’s add) then reads and writes the output again. Launching the GEMM as the down-projection’s dependent avoids this join: the GEMM itself adds the up-projection in its epilogue, so the output is written once and nothing follows.

Portability. The decode path needs only PDL and a GEMM epilogue that can add a low-rank product for groups of rows, both available outside Triton, the GPU language of our kernels, so a CUTLASS-based engine could adopt it for any base format whose GEMM’s main loop streams its weights.

When the overlap hides the adapter. With the decode path, a projection’s adapter cost is approximately $t_{\mathrm{epi}}+\max(0,t_{\mathrm{shrink}}-t_{\mathrm{main}})$; the epilogue’s extra work plus whatever part of the down-projection outlasts K2’s main loop. At decode, $t_{\mathrm{main}}$ is the time to stream the projection’s weights, which falls with the GPU’s memory bandwidth and when tensor parallelism splits each layer’s weights across GPUs, while $t_{\mathrm{shrink}}$, the latency of a small kernel, hardly changes, so the condition is easiest to meet for large projections on GPUs of modest bandwidth; Figure 5 measures where it holds. Where the down-projection lasts most of the main loop, the two kernels also compete for SMs, and a few microseconds stay exposed (§ 5). The condition therefore bounds the exposure from below: when the GEMM launches more thread blocks than the GPU has SMs, only its first blocks start before the down-projection ends, so less than the whole main loop can hide it.

The vLLM plugin

EpiLoRA installs as a vLLM plugin that replaces vLLM’s LoRA linear layers; any layer it cannot serve falls back to vLLM. The kernels read vLLM’s own adapter buffers and batching metadata, so adapters are never copied, and the fused call is a registered custom operator that vLLM can capture in its CUDA graphs, which record a step’s kernel launches once and replay them to avoid launch overhead.

Timed dispatch. When the model loads, the plugin times up to three paths for each projection shape and batch size and keeps the fastest: the fused kernels, run as the decode path on Hopper GPUs; the hybrid path, the vendor GEMM followed by our adapter kernels; and the vendor GEMM followed by vLLM’s own adapter kernels. On Hopper GPUs it selects the decode path for decode batches of 8B–9B models. For larger batches, including prefill, which processes a request’s prompt, it selects the vendor GEMM with Epi-LoRA’s or vLLM’s adapter kernels, whichever is faster. On the L40S it selects the fused path (Table 1).

Memory. vLLM sizes its KV cache, which stores every request’s attention keys and values, from the memory left after the model loads, so the plugin shares scratch buffers across layers, reuses vLLM’s weight codes on NVFP4 checkpoints and stores low-precision adapters in smaller slots; with bf16 adapters, its load-time warm-up fragments memory and leaves $0.85\times$ vLLM’s cache on the L40S (Figure 12).

Implementation and correctness. EpiLoRA is about 6,500 lines of Python and Triton, including 48 Triton kernels and the RL trainer’s kernels, and needs no changes to vLLM. Under tensor parallelism it shards the adapters as vLLM does: projections whose outputs are split across GPUs split the up-projection’s output, and projections whose inputs are split split the down-projection’s input. We test correctness at three levels: EpiLoRA’s FP8 and NVFP4 quantization reproduces vLLM’s codes and scales bit for bit, its kernels match vLLM’s LoRA path and are bitwise reproducible, and served adapters keep vLLM’s accuracy (Figure 8).

LoRA RL on the same base

The RL system places a trainer and a vLLM rollout engine on one GPU and trains with GRPO, a policy-gradient algorithm that compares groups of sampled responses to the same prompt ([50, 51]). After each optimizer step the trainer publishes only its adapter, which the engine loads into an adapter slot.

A trainer on the engine’s FP8 weights. A bf16 trainer differentiates a slightly different model than the FP8 engine samples from. Our FP8 trainer instead computes on the engine’s FP8 weights with bf16 activations, which removes the weight part of this mismatch and shrinks the frozen base weights from 16.7 to 10.2 GiB for a 9B model (Table 36). With one copy, the trainer maps the engine’s weights directly through CUDA’s inter-process memory sharing instead of loading its own, and the engine releases its KV cache while the trainer runs, so no weights move between phases. The trainer’s projections run a training GEMM that reads the FP8 weights in place and adds the LoRA update in its epilogue, as K2 does, so no 16-bit copy of the weights is ever made.

Matched-precision rollouts. To first order, the engine’s and the trainer’s log-probabilities of a sampled token $y$ differ by three terms, from rounding the weights, rounding the activations, and differences between the two sides’ kernels:

The FP8 trainer removes $\delta_W$, but not $\delta_A$, the engine’s rounding of each GEMM’s input activations. Importance corrections compensate for this difference (Liu et al., [30]); we remove it in the arithmetic instead. In the matched engine, every FP8 layer computes with 8-bit weights and 16-bit activations (W8A16) instead of rounding both to 8 bits (W8A8) as vLLM’s FP8 engine does, using the trainer’s arithmetic on the same weights: FP8 weights widened to bf16 in registers, bf16 activations and fp32 accumulation. vLLM’s weight-only FP8 path, Marlin (Frantar et al., [7]), also leaves activations unrounded but repacks the weights into a layout the trainer cannot share.

Evaluation

We ask what the kernels save (§ 5), what that buys end to end (§ 5.1), and whether the same base trains as well as bf16 with less memory (§ 5.2).

Setup. We run vLLM 0.28 with CUDA graphs and PyTorch’s compiler, kernels cached before we measure. We serve Qwen3-8B (Qwen Team, [45]), Llama-3.1-8B (Llama Team, AI @ Meta, [31]), Qwen3.5-9B (Qwen Team, [46]), which replaces most attention layers with linear attention, and Qwen2.5-72B, with FP8 bases that vLLM quantizes when the model loads, and our NVFP4 checkpoint of Qwen3-8B; RL trains Qwen3.5-9B and Qwen3.8-27B ([47]). The GPUs are the L40S (Ada), the H100, H200 and GH200 (Hopper), and a 48 GB partition of an RTX PRO 6000 (Blackwell), which we call the Blackwell slice. For serving, the baselines are vLLM’s LoRA path on its fastest base for each GPU, with LoRA kernels we tuned for each model, SGLang ([72]) and TensorRT-LLM ((NVIDIA, 2026)). For RL, they are FlashRL’s FP8 engine beside a bf16 trainer with truncated importance sampling (TIS), which multiplies each token’s gradient by its importance weight capped at 2 ([30]), bf16 training with the trainer’s weights offloaded during rollouts ([51]), and one bf16 copy shared by engine and trainer, as in Unsloth ([55]) but keeping its KV cache. We measure latency as the time per output token (TPOT) of a single request, and throughput with 256 requests of 512 prompt and 256 output tokens, using random adapters of rank 16–128, since speed depends only on the adapters’ shapes. Each engine is started four times and we report the mean of the last three unless a table says otherwise; the appendices give every measurement. Kernel timings use CUDA graphs with the L2 cache flushed; unless noted, adapters have rank 64, and bold in tables marks the best value, with ties within 2%.

Kernels

The adapter’s cost falls by 2–6× on every GPU (Table 1). On Hopper GPUs, one rank-64 adapter costs 53–57 µs per layer with vLLM’s tuned kernels and 10–12 µs with the decode path, so the layer takes 0.70–0.75× vLLM’s time. On the H100, nine tenths of the saving is in the adapter and a tenth in EpiLoRA’s own GEMM. The overlap pays because the GEMM needs the down-projection only in its epilogue, so it streams its weights while the adapter is computed; vLLM’s dual-stream mode overlaps the adapter only with the GEMM as a whole (Figure 1b). On the L40S, fusing adapters into EpiLoRA’s own GEMM halves their cost, and

on Blackwell the decode path cuts it from 54 to 17 µs.

What stays exposed follows the overlap condition (Figure 5). The measured exposure tracks the condition with a window we fitted after the data: the down-projection stays hidden while it finishes within the main loop of the GEMM’s first wave of thread blocks, when the first epilogues need it. With a rank-64 adapter on full-size layers, the down-projection ends within the GEMM’s main loop on every projection we measured, and at most 2.1 µs of it remains exposed. At ranks 16 and 128 and on tensor-parallel shards, up to 9 µs per projection stays exposed, mostly where the GEMM launches more blocks than the GPU has SMs.

The lead holds from 1 to 16 decode tokens (Table 4). With the decode path, one adapter adds 4–9% to the layer on Hopper GPUs, against 35–55% with vLLM’s tuned kernels or dual-stream mode, because the down-projection finishes within the GEMM’s main loop (Figure 5); run serially, the layer takes 1.13–1.31× as long, depending on the down-projection kernel (Table 11, Table 12), so at one decode token the overlap accounts for a third to two thirds of the saving.

Table 4. From 1 to 16 decode tokens, the decode path keeps one adapter’s cost to a few percent of the layer. One Qwen3-8B decoder layer, in microseconds; cost: the time over the layer without an adapter (vLLM: the better of its tuned kernels and dual-stream mode).

μs ↓	No adapter	vLLM		EpiLoRA		Cost (%) ↓
Tokens		tuned	dual	serial	overlap	vLLM	EpiLoRA
H100, FP8 base
1	128	179	180	163	134	42	6
4	128	179	172	162	133	35	4
16	128	180	174	167	136	36	7
H200, FP8 base
1	104	161	167	144	112	55	8
4	106	162	157	147	112	48	6
16	106	162	159	150	115	49	9

Table 11. The overlap beats serializing even a library-fast down-projection, and comes within 2% of a serialized kernel that does no work (0.99–1.02$\times$). One Qwen3-8B decoder layer’s four projections at one decode token with one adapter, microseconds summed over the projections (CUDA graphs, L2 cache flushed), on one H200 and one H100. Serialized: a kernel run before the same GEMM without overlap, which does no work (empty kernel), uses cuBLAS’s GEMM or GEMV, or is our own down-projection. GEMM, $h$ given: the GEMM with the down-projection’s result already in memory, a lower bound for any overlap. Shrink alone: the down-projection by itself.

µs ↓		Serialized				GEMM, $h$ given	Shrink alone		Empty
GPU	Rank	Decode path	empty kernel	cuBLAS GEMM	cuBLAS GEMV	ours		cuBLAS	ours	kernel
H200	16	108.1	110.2	135.0	134.6	138.9	106.0	37.6	48.6	12.8
	64	112.4	114.0	140.1	139.6	144.6	110.4	38.6	50.3	13.3
	128	117.9	120.7	146.1	146.3	150.8	117.3	40.5	51.7	12.9
H100	16	129.3	132.0	154.9	154.5	156.8	128.7	42.0	49.4	20.3
	64	133.4	134.8	158.0	158.5	163.0	130.9	42.1	53.4	19.6
	128	139.2	138.2	163.5	163.0	164.8	135.3	44.6	52.2	19.9

GPU	rank	decode path	serialized	ratio	per projection	shrink alone
		µs ↓	µs ↓			its split	fastest
H200	16	109.6	118.3	1.08$\times$	1.05–1.12$\times$	11.1–14.6	6.1–8.6
H200	64	112.1	130.8	1.17$\times$	1.11–1.24$\times$	9.0–17.8	6.6–9.0
H200	128	117.9	146.1	1.24$\times$	1.17–1.33$\times$	9.7–18.2	6.9–9.7
H100	16	128.8	140.3	1.09$\times$	1.06–1.15$\times$	9.3–15.2	7.4–9.7
H100	64	133.1	149.9	1.13$\times$	1.09–1.19$\times$	10.1–18.1	7.5–9.9
H100	128	137.9	162.0	1.17$\times$	1.11–1.26$\times$	10.6–18.2	7.9–10.5

End-to-end serving

At batch 1, EpiLoRA is faster than every system we ran on Hopper GPUs (Table 2 and Table 3). For 8B–9B models it takes 0.76–0.88× the time per token of vLLM’s fastest configuration, including Llama-3.1-8B with both systems’ kernels tuned for its shapes. SGLang and TensorRT-LLM take 1.4–1.8× EpiLoRA’s time on Qwen3-8B. Profiles show why (Table 3): per token on the H100, their adapter kernels add 2.9–3.0 ms of GPU time, vLLM’s dual-stream mode 1.9 and EpiLoRA’s 0.4, leaving EpiLoRA 9–14% slower than without LoRA (Table 18).

Table 2. EpiLoRA against vLLM’s fastest configuration. Time per output token with one adapter at batch 1, and throughput with 32 tenants; vLLM runs its fastest base (NVFP4 on the Blackwell slice, FP8 elsewhere) with LoRA kernels tuned for each model (Qwen3.5-9B’s on the H100 for all its Hopper rows). On 2×H100, both systems use tiles searched for the shards; the H200 72B row comes from an earlier job than Table 29, which gives 0.94×.

GPU	Model	vLLM	EpiLoRA	Ratio
H100	Qwen3.5-9B	6.26	5.14	0.82×
H100	Qwen3-8B	6.02	4.81	0.80×
H100	Llama-3.1-8B	5.59	4.76	0.85×
H200	Qwen3.5-9B	6.11	4.81	0.79×
H200	Qwen3-8B	5.94	4.51	0.76×
GH200	Qwen3.5-9B	6.70	5.91	0.88×
H200	Qwen2.5-72B	28.00	26.80	0.96×
2×H100	Qwen2.5-72B	20.31	17.75	0.87×
L40S	Qwen3.5-9B	15.39	14.97	0.97×
L40S	Qwen3-8B	14.05	13.95	0.99×
Blackwell	Qwen3-8B	10.27	9.46	0.92×
32 tenants, thousand tokens per second ↑
H100	Qwen3.5-9B	5.30	5.64	1.06×
H100	Qwen3-8B	5.89	6.22	1.05×
H100	Llama-3.1-8B	6.26	6.32	1.01×
H200	Qwen3.5-9B	5.29	5.78	1.09×
H200	Qwen3-8B	5.83	6.16	1.06×
GH200	Qwen3.5-9B	5.57	6.00	1.08×
L40S	Qwen3.5-9B	1.47	1.59	1.08×
L40S	Qwen3-8B	1.63	1.76	1.08×
Blackwell	Qwen3-8B	2.08	2.11	1.02×

Table 3. EpiLoRA against three serving systems on Qwen3-8B, each pair measured in one job. Time per output token at batch 1 and throughput with 32 tenants, relative to EpiLoRA’s, each system in its fastest LoRA configuration (TRT-LLM: TensorRT-LLM’s PyTorch backend). ^aAt its fastest KV-cache share, 0.6. ^bWithout LoRA, 1.09–1.14× vLLM’s batch-1 time and 0.95× its throughput; H200: one start. ^cFrom NVIDIA’s Nsight Systems profiler (Table 21); vLLM in dual-stream mode with its default kernels.

	EpiLoRA	vLLM	SGLang^b	TRT-LLM
Batch 1: time per token relative to EpiLoRA’s ↓
H100	1.00×	1.25×	1.65×	1.43×
H200	1.00×	1.32×	1.81×	1.52×
Batch 1: GPU time the adapter adds per token (ms)^c ↓
H100	0.40	1.87	3.01	2.87
32 tenants: throughput relative to EpiLoRA’s ↑
H100	1.00×	0.95×	0.94×	0.97×^a
H200	1.00×	0.95×	0.93×	1.05×

Engine, LoRA path	W1 (ms) ↓	W2 (tok/s) ↑	W1 ↓	W2 ↑	W1 ↓	W2 ↑
			1 adapter (%)		32 adapters (%)
SGLang 0.5.9, csgmv	4.91	10,125	+64.9	−43.8	+64.8	−43.7
SGLang 0.5.9, triton	4.91	10,125	+67.4	−77.2	+66.6	−77.2
vLLM 0.28, default kernels	4.52	10,663	+52.1	−27.5	+83.5	−54.3
vLLM 0.28, dual-stream	4.52	10,663	+39.3	−30.6	+67.8	−54.4
ours, decode path	4.52	10,663	+8.6	−28.1	+8.1	−43.4

The lead is largest on the H200, whose faster memory shortens the base GEMM so the adapter is a larger share of each step, and smallest on the GH200, where both systems run slower than on the H100. On the L40S, where vLLM is fastest with Marlin’s weight-only GEMM, EpiLoRA’s weight-only mode (bf16 activations) matches it; on the Blackwell slice’s NVFP4 base, it takes 0.92× vLLM’s time.

At batch 1, the lead extends to a 72B model (Table 2). A larger model leaves less to save, as its GEMMs grow with the square of the hidden size and the adapter linearly: Qwen2.5-72B takes 0.96× the time per token of tuned vLLM on one H200 and 0.87× across two H100s.

With many tenants, EpiLoRA keeps more of the base’s throughput (Table 2). With 32 rank-64 tenants it delivers 1.05–1.09× the throughput of vLLM’s best configuration on 7 of the 9 GPU–model pairs. These batches exceed the 32 tokens the decode path covers, so the gain comes from EpiLoRA’s adapter kernels after the vendor GEMM: on the H200, swapping in vLLM’s turns 1.10× into 0.97× in one job (Table 14). They beat vLLM’s on projections with three or four outputs of unequal widths, such as attention’s query, key and value, where vLLM’s up-projection costs 1.7× as much per output column (Table 15); at 32 tokens,

Projection	1–32 tokens	64–2,048 tokens	64	128	256	512
linear attention, input (4 outputs)	decode path	EpiLoRA’s kernels	1.31	1.17	1.62	1.57
attention, query/key/value (3 outputs)	decode path	EpiLoRA’s kernels	1.45	1.25	1.78	1.89
MLP, gate/up (2 outputs)	decode path	vLLM’s kernels; EpiLoRA’s kernels at 256	0.87	0.89	1.00	0.99
attention output	decode path	vLLM’s kernels	0.81	0.72	0.84	0.70
MLP, down	decode path	vLLM’s kernels	0.84	0.74	0.89	0.78

Outputs (projections)	Columns	vLLM	EpiLoRA	vLLM	EpiLoRA
3–4 unequal (32)	376,832	8.14	4.71	21.6	12.5
gate/up, 2 equal (32)	786,432	10.54	9.52	13.4	12.1
one output (64)	262,144	2.08	3.42	7.9	13.1

the lead per layer shrinks as a batch mixes more distinct adapters (Figure 11). On the L40S the gain comes from fusing adapters into EpiLoRA’s GEMM; TensorRT-LLM beats EpiLoRA on the H200 (Table 3).

Online, EpiLoRA lowers the time per output token at most loads (Figure 6). When CUDA graphs cover the prefill sizes, its median time per output token with 32 tenants is 0.77–0.95× vLLM’s on the H100 and 0.92–0.99× on the L40S, and its 99th-percentile (p99) time to first token (TTFT) stays within 1.1× of vLLM’s at 6 of these 8 GPU–rate points. On a WildChat trace, EpiLoRA takes 0.70–0.86× vLLM’s time per output token on the H200 and 0.77–0.98× on the H100 at vLLM’s defaults, and serves at least vLLM’s goodput at every rate (Table 5). Its lead is largest at low load, where batches fit the decode path.

Req/s	vLLM	EpiLoRA	vLLM	EpiLoRA	vLLM	EpiLoRA
Median TPOT (ms) ↓			p99 TTFT (s) ↓		Goodput (req/s) ↑
H200
2	9.1	7.0	0.09	0.09	1.98	1.99
4	12.8	8.9	0.10	0.09	3.88	3.92
8	18.0	14.6	0.12	0.11	7.26	7.43
16	30.6	26.4	0.16	0.16	12.08	12.70
H100
2	9.9	7.6	0.16	0.18	1.98	1.99
4	14.3	11.1	0.18	0.19	3.87	3.91
8	21.8	19.0	1.371	0.22	7.15	7.35
16	33.1	32.4	0.24	0.27	12.00	12.38

At vLLM’s default graph sizes (dotted), prefills longer than 512 tokens run outside CUDA graphs, where every kernel

Table 5. On a real trace, EpiLoRA lowers the time per output token at every load. Per start, 1,000 of 3,383 first turns of WildChat conversations (short prompts: median 23 tokens) with gamma arrivals (burstiness 0.5), 32 tenants, Qwen3.5-9B at vLLM’s default CUDA-graph settings; goodput counts requests meeting a 2 s TTFT and a 50 ms TPOT. ²One of vLLM’s three starts took 3.7 s (the others 0.19–0.20 s), so neither value is bold.

:::

launch costs host time: EpiLoRA then trails vLLM slightly on the L40S at 4–6 requests/s and by 1.18× on the H100 at 16 requests/s, and its p99 TTFT is 1.1–1.2× vLLM’s on the H100; Table 24 and Table 25 add the H200 and long prompts.

GPU	Rate	Median TPOT (ms) ↓			p99 TTFT (ms) ↓
		vLLM	EpiLoRA	Ratio	vLLM	EpiLoRA	Ratio
H100	2	9.9	7.9	0.79	213	254	1.20
	4	15.5	13.0	0.84	238	274	1.15
	8	28.9	29.0	1.00	268	305	1.14
	16	54.8	64.7	1.18	360	400	1.11
H100, new tiles	2	9.9	7.7	0.78	214	242	1.13
	4	15.5	12.5	0.81	243	262	1.08
	8	29.0	27.7	0.95	269	289	1.08
	16	54.9	61.1	1.11	361	386	1.07
H100, new tiles, own kernels	2	9.9	7.5	0.76	214	213	1.00
	4	15.5	11.8	0.76	243	240	0.99
	8	29.0	24.9	0.86	269	266	0.99
	16	54.9	58.6	1.07	361	386	1.07
H200	2	8.8	6.5	0.74	96	111	1.17
	4	12.0	8.9	0.74	120	138	1.14
	8	21.2	17.6	0.83	166	164	0.99
	16	46.6	43.1	0.93	270	243	0.90
L40S	1	23.7	21.9	0.93	235	241	1.03
	2	32.0	31.1	0.97	255	262	1.03
	4	51.4	52.2	1.02	332	339	1.02
	6	86.9	88.5	1.02	3,062	4,054	1.32

GPU	Requests/s	vLLM	ours	ratio	vLLM	ours	ratio
		Median TPOT (ms) ↓			p99 TTFT (ms) ↓
Qwen3.5-9B, 32 tenants, 2,048-token prompts
H200	1	8.00	5.88	0.73	175	173	0.99
	2	9.79	7.08	0.72	235	210	0.90
	4	16.70	11.09	0.66	338	277	0.82
H100	1	8.08	6.41	0.79	242	250	1.03
	2	10.02	7.84	0.78	295	297	1.01
	4	16.89	13.04	0.77	402	384	0.95
Qwen2.5-72B, 8 tenants, 512-token prompts
H200	0.5	35.71	34.75	0.97	422	416	0.99
	1	41.84	39.70	0.95	541	505	0.93
	2	59.30	54.57	0.92	703	694	0.99
Qwen2.5-72B at TP2 on two H100s, vLLM tuned for the shards
H100	0.5	23.91	20.78	0.87	219	249	1.14
	1	26.14	23.26	0.89	272	280	1.03
	2	31.66	28.57	0.90	641	368	0.57

LoRA RL on the same base

GRPO (Shao et al. [50]) trains a rank-64 adapter on competition math (DAPO-Math (Yu et al. (2025)) with engine and trainer on one GPU: Qwen3.5-9B for 200 steps of 16 problems and Qwen3.8-27B for 100 steps of 8 problems, 8 samples each, with samples and answers capped at 2,048 tokens. We report accuracy on OlympiadBench (He et al. (2024)) and Minerva (Lewkowycz et al. [22]), with 3–9 seeds and 95% intervals (Table 30, Table 31); RL lifts the 9B from 50.5% to 60–61% (Figure 8b).

Code	Engine	Trainer, correction	GPU	Runs	GPU-h ↓	Accuracy (%) pre-RL	Accuracy (%) final [95% CI]	Δ (points) ↑	≤ pre-RL
S0G	bf16	bf16	H100a	5	6.9	49.8–50.5	57.4–61.8	reference	0/5
S0T	bf16, asleep	bf16	H100	3	5.6	50.5	61.0–61.7	+1.5 [−0.4, +3.5]	0/3
S8G	W8A8	FP8 (deq.)	H100	5	5.6	50.6–51.5	2.3–60.7	−12.9 [−43.9, +18.0]	1/5
S4F	W8A8	bf16	H100	3	5.3	49.7–51.0	4.3–52.4	−32.7 [−92.4, +27.0]	2/3
S8T	W8A8	FP8 (deq.), TIS	H100	4	5.7	50.6–51.0	57.6–61.0	−0.4 [−2.8, +2.1]	0/4
S0TF	bf16, asleep	bf16 + FLA	H100	9	3.2	49.8–50.5	58.2–63.5	+0.9 [−1.2, +2.9]	0/9
S8C	W8A8	FP8 (ours), centering	H100	4	4.0	49.9–50.6	57.1–62.8	+0.7 [−2.9, +4.4]	0/4
S8W	W8A16, Marlin	FP8 (ours)	H100	4	3.9	49.5–50.4	58.9–62.1	+0.3 [−2.2, +2.7]	0/4
S8TL	W8A8	FP8 (deq.), TIS	L40S	3	13.4	49.9–50.5	60.5–60.9	+1.0 [−1.0, +2.9]	0/3
S8ML	W8A16, matched	FP8 (ours)	L40S	3	8.8	50.2–50.6	60.0–62.0	+0.9 [−1.5, +3.3]	0/3
S8WA	W8A16, Marlin	FP8 (deq.) + FLA	A100	8	6.1	49.2–50.6	58.8–63.4	+0.8 [−1.3, +2.8]	0/8
S9P	W8A8	FP8 (deq.)	H100	3	5.7	50.7–51.5	56.2–58.5	−2.3 [−4.8, +0.1]	0/3
S9B	W8A8	FP8 (deq.)	H100	3	5.8	49.6–50.2	47.7–59.7	−6.3 [−20.4, +7.8]	1/3
S8TF	W8A8	FP8 (ours), TIS	H100	9	3.1	49.4–51.0	57.7–62.1	−0.3 [−2.0, +1.3]	0/9
S4TF	W8A8	bf16 + FLA, TIS	H100	9	3.0	50.2–51.0	60.1–62.8	+0.6 [−0.9, +2.0]	0/9
S8MH	W8A16, matched	FP8 (ours)	H100	9	4.2	50.7–50.8	58.2–64.4	+0.1 [−1.6, +1.9]	0/9
S8UF	W8A8	FP8 (ours)	H100	6	3.1	49.9–51.5	54.2–57.2	−4.9 [−6.5, −3.3]	0/6
S8TLF	W8A8	FP8 (ours), TIS	L40S	3	9.3	49.6–50.5	57.7–60.2	−2.0 [−5.1, +1.1]	0/3
S0OL	bf16	bf16 + FLA	L40S	7	8.4	49.6–51.1	59.9–64.8	reference	0/7
S8OL	W8A8	FP8 (ours), TIS	L40S	7	7.2	49.5–50.7	58.1–62.4	−1.1 [−3.1, +0.9]	0/7
S4OL	W8A8	bf16 + FLA, TIS	L40S	3	7.2	50.2–50.8	59.2–63.6	−0.3 [−4.6, +4.1]	0/3
S8RA	W8A8	FP8 W8A8 (ours)	L40S	3	8.2	49.6–51.3	51.1–60.5	−6.5 [−17.5, +4.5]	1/3
S8ULF	W8A8	FP8 (ours)	L40S	3	8.5	49.5–50.8	52.5–60.1	−6.3 [−15.4, +2.8]	0/3

Table 31. Every LoRA-RL configuration (continued from Table 30): the 27B, one copy shared by the engine and the trainer, and Qwen3-4B. Columns as in Table 30; each block’s header names its benchmarks and, for the 27B, the accuracy before RL. The 27B is scored on the bf16 engine (on the L40S also on OlympiadBench and Minerva only, as the 9B), except where a header says otherwise. One FP8 copy of the 27B with its KV cache kept ran out of memory on the L40S.

Code	Engine	Trainer, correction	GPU	Runs	GPU-h ↓	Accuracy (%) final [95% CI]	Δ (points) ↑	≤ pre-RL
Qwen3.8-27B math, MATH-500, OlympiadBench and Minerva (before RL 61.6–62.8%); against Q27
Q27	bf16	bf16	2×A100	3	15.7	70.8–74.6	reference	0/3
S8T1g	W8A16, Marlin	FP8 (deq.), TIS	A100	3	15.3	71.9–73.3	−0.4 [−4.4, +3.6]	0/3
S8T27H	W8A8	FP8 (ours), TIS	H100	2 + 1ᵇ	4.4	72.4–73.5	+0.2 [−3.9, +4.2]	0/2
S0T27A	bf16, offloaded	bf16 + FLA	A100	3	7.3	71.2–72.0	−1.2 [−5.6, +3.1]ᶜ	0/3
S8T27A	W8A16, Marlin, offloaded	FP8 (deq.) + FLA, TIS	A100	3	6.5	70.6–72.0	−1.3 [−5.3, +2.7]ᶜ	0/3
S8S27H	W8A8	FP8 (ours), one copy, TIS	H100	5	3.3	68.1–72.6	−2.5 [−6.2, +1.2]	0/5
Qwen3.8-27B math on the H100, the same benchmarks; against S0T27HO
S0T27HO	bf16, offloaded	bf16 + FLA	H100	3	4.0	70.4–72.6	reference	0/3
S8T27HO	W8A8, offloaded	FP8 (ours), TIS	H100	3	3.4	70.6–73.7	+0.7 [−2.5, +3.9]	0/3
S4T27HO	W8A8, offloaded	bf16 + FLA, TIS	H100	3	3.2	70.7–72.4	−0.1 [−2.4, +2.3]	0/3
Qwen3.8-27B math on the H100, the same benchmarks, one copy, KV cache released for training; against S8T27HO
S8K27H	W8A8	FP8 (ours), one copy, TIS	H100	3	3.3	70.9–73.6	−0.1 [−3.5, +3.3]	0/3
Qwen3.8-27B math on one L40S, OlympiadBench and Minerva (before RL 49.7–51.1%); against Q27 on the same benchmarks
Q27	bf16	bf16	2×A100	3	15.7	59.7–63.7	reference	0/3
S8T27L	W8A8, offloaded	FP8 (ours), TIS	L40S	3	11.5	58.7–63.0	−0.9 [−5.7, +3.8]	0/3
Qwen3.8-27B math on one L40S, the same benchmarks on each run’s own FP8 engine, one copy, KV cache released; against S8T27L
S8K27L	W8A8	FP8 (ours), one copy, TIS	L40S	3	11.2	57.8–61.1	+0.0 [−3.9, +3.9]	0/3
Qwen3.5-9B math on the L40S, one copy; against S8TLF (two copies)
S8S9L	W8A8	FP8 (ours), one copy, TIS	L40S	3	7.1	60.4–61.7	+2.4 [−0.2, +5.1]	0/3
Qwen3.5-9B math on the L40S, one copy, matched engine; against S8ML (two copies)
S8MS9L	W8A16, matched	FP8 (ours), one copy	L40S	5	7.3	56.1–60.1	−2.3 [−4.7, +0.1]	0/5
Qwen3.5-9B math on the L40S, one copy, KV cache released for training; against S8OL
S8K9L	W8A8	FP8 (ours), one copy, TIS	L40S	3	7.1	56.6–61.8	−0.3 [−5.8, +5.1]	0/3
Qwen3-4B math; against S0L
S0L	bf16	bf16	L40S	3	4.5	49.7–50.2	reference	0/3
S8L	W8A8	FP8 (deq.)	L40S	3	3.6	49.7–51.4	+0.7 [−1.3, +2.7]	0/3

Matching the weights removes the largest source of mismatch, and leaving activations unrounded removes most of the rest (Figure 7). We measure the mismatch as the KL divergence between the engine’s and the trainer’s next-token distributions. Rounding the weights raises it the most, to 12–21× that of a bf16 engine and trainer, and the FP8 trainer removes this part by computing on the engine’s own weights. Almost all of the rest comes from rounding activations: engines that round them retain 6–14×, while our matched engine and vLLM’s Marlin retain 1.0–1.4×, and the order holds on a dense second model (Table 33).

Adapter	W8A8	matched W8A16	Marlin W8A16	share	over W16 trainer
bf16-trained, seed 0	6.47×	1.35×	1.34×	0.76	1.35×
bf16-trained, seed 1	6.80×	1.20×	1.49×	0.74	1.36×
bf16-trained, seed 2	7.41×	1.51×	1.60×	0.86	1.50×
W8A8-trained, seed 0	6.31×	1.14×	1.30×	0.79	1.49×
W8A8-trained, seed 1	7.16×	1.38×	1.30×	0.76	1.37×

On the 9B, corrected or matched FP8 RL ends where bf16 training does (Figure 8b). We call two configurations equivalent when the 95% interval of their difference lies within ±5 points, about half of RL’s gain. With nine seeds, W8A8 rollouts with TIS, FlashRL’s recipe and the matched engine without any correction end within 0.6 points of bf16 training, every interval within ±2.6 points (Table 32), while uncorrected W8A8 rollouts end 4.9 points below it; on an L40S, the matched engine with one copy that keeps its KV cache ended 3.4 points below bf16 training ([−5.7, −1.1]), but against two copies one copy moved it by −2.3 and W8A8 rollouts with TIS by +2.4 (Welch intervals spanning zero, paired ones excluding it in opposite directions), so the one-copy effect is inconclusive (Appendix § 8). The matched engine needs no correction because it leaves the mismatch at the bf16 pair’s level (Figure 7) and keeps it there throughout training (Table 34). On the 27B, EpiLoRA’s configurations end at 72.0–72.1% (with MATH-500) against 71.4% for bf16 training and FlashRL, every interval within ±4 points with three seeds (Table 31).

Table 32. Equivalence of the RL configurations with their references, three ways. Final accuracy (OlympiadBench and Minerva pooled) of each configuration minus its reference: on each policy’s own engine, the Welch 95% interval over all runs and the $t$ interval of the per-seed differences over the seeds both ran (a seed fixes the problem order); and the Welch interval with every policy re-scored on one common bf16 engine (avg@8). $p_{\ne0}$: Welch’s test of a zero difference; $p_{\pm5}$: the two one-sided tests of the ±5-point margin; both Holm-corrected over these twelve comparisons (a family chosen after the data). Runs: the configuration’s and the reference’s.

Comparison	GPU	Runs	Each policy’s own engine: Welch 95% CI	Each policy’s own engine: $p_{\ne0}$	Each policy’s own engine: $p_{\pm5}$	Paired by seed: Seeds	Paired by seed: interval	One bf16 engine, avg@8: Welch 95% CI	One bf16 engine, avg@8: $p_{\ne0}$	One bf16 engine, avg@8: $p_{\pm5}$
W8A8 + TIS against tuned bf16	H100	9/9	−0.3 [−2.0, +1.3]	1.000	<0.001	9	−0.3 [−2.6, +1.9]	−0.3 [−1.6, +1.1]	1.000	<0.001
FlashRL’s recipe against tuned bf16	H100	9/9	+0.6 [−0.9, +2.0]	1.000	<0.001	9	+0.6 [−1.3, +2.4]	+0.5 [−0.6, +1.6]	1.000	<0.001
matched engine against tuned bf16	H100	9/9	+0.1 [−1.6, +1.9]	1.000	<0.001	9	+0.1 [−1.3, +1.5]	+0.1 [−1.4, +1.7]	1.000	<0.001
W8A8 + TIS, offloaded, against bf16 offloaded	L40S	7/7	−1.1 [−3.1, +0.9]	1.000	0.005	7	−1.1 [−3.5, +1.3]	−0.3 [−2.0, +1.5]	1.000	<0.001
FlashRL’s recipe, offloaded, against bf16 offloaded	L40S	3/7	−5.3 [−4.6, +4.1]	1.000	0.106	3	−1.2 [−8.6, +6.6]	−0.1 [−3.8, +3.6]	1.000	0.046
one copy against two copies (W8A8 + TIS)	L40S	3/3	+2.4 [−0.2, +5.1]	0.601	0.108	3	+2.4 [+0.3, +4.5]	+2.1 [−0.3, +4.4]	0.748	0.046
one copy against two copies (matched)	L40S	5/3	−2.3 [−4.7, +0.1]	0.601	0.106	3	−2.8 [−5.3, −0.3]	−2.3 [−5.3, +0.7]	0.902	0.046
one copy, KV released, against offloading (W8A8 + TIS)	L40S	3/7	−1.1 [−6.8, +4.6]	1.000	0.151	3	−0.3 [−4.2, +3.6]	−1.6 [−4.5, +1.3]	1.000	0.046
two copies (W8A8 + TIS) against bf16 offloaded	L40S	3/7	−3.0 [−5.5, −0.5]	0.305	0.151	3	−3.9 [−11.3, +3.5]	−1.6 [−3.8, +0.6]	1.000	0.026
one copy (W8A8 + TIS) against bf16 offloaded	L40S	3/7	−0.6 [−2.4, +1.3]	1.000	0.002	3	−1.5 [−9.1, +6.1]	+0.5 [−1.1, +2.0]	1.000	<0.001
two copies (matched) against bf16 offloaded	L40S	3/7	−1.1 [−3.3, +1.2]	1.000	0.017	3	−2.0 [−6.5, +2.5]	+0.4 [−2.4, +3.1]	1.000	0.026
one copy (matched) against bf16 offloaded	L40S	5/7	−3.4 [−5.7, −1.1]	0.108	0.151	5	−3.6 [−5.8, −1.5]	−1.9 [−3.5, −0.3]	0.304	0.006

Configuration	Runs	1–10	91–110	181–200	Late/early ↓
bf16 engine and trainer	9	0.0055	0.0061	0.0064	1.18×
FlashRL’s recipe (W8A8, bf16 trainer, TIS)	9	0.0272	0.0308	0.0348	1.28×
W8A8, FP8 trainer, TIS	9	0.0173	0.0192	0.0220	1.27×
matched engine, no correction	9	0.0070	0.0068	0.0075	1.07×
W8A8, FP8 trainer, no correction	6	0.0174	0.0244	0.0394	2.26×

An FP8 trainer fits the 27B on one 48 GB GPU. Training on the engine’s FP8 weights saves 6.5 GiB for the 9B and 22.7 GiB for the 27B (Table 36). So the 27B trains on one 48 GB L40S, which its bf16 weights (50 GiB) exceed: with the trainer’s copy offloaded, EpiLoRA reaches 60.8%, against 61.7% for bf16 training on two A100s (Figure 8b).

With long rollouts, one FP8 copy comes close to the fastest placement with a fraction of its host memory (Table 6 and Table 7). With rollouts of up to 8,192 tokens, two FP8 copies fit neither model, and one bf16 copy that keeps its KV cache fits only the 9B, at $2.2\times$ the step time ($1.9\times$ that of an FP8 copy that also keeps it, Table 38); Unsloth releases that cache while training, a variant we did not run (Appendix § 8). One FP8 copy then takes $0.94$–$1.00\times$ the time of FlashRL’s offloaded recipe and is $1.3$–$1.6\times$ as fast as bf16 training with offloading, while pinning 9 GiB of host memory instead of 32–160 GiB; in full runs of the 27B it takes $1.02$–$1.05\times$ the GPU-hours of the offloaded FP8 recipes, or $0.98$–$1.02\times$ per trained token (Table 7). The difference is in generation: FP8 weights take half the memory of bf16 weights and leave the engine a KV cache almost three times as large (Figure 9), and with long rollouts the cache sets how many sequences decode at once; FlashRL’s FP8 engine gets the same cache. Where EpiLoRA does not pay (Appendix § 8): where two copies fit, FlashRL’s faster bf16 trainer makes its recipe slightly cheaper; at 2,048 tokens a bf16 copy that keeps its KV cache ties one FP8 copy on the 9B; a trainer rounding activations like the engine trains $1.2\times$ faster at a larger mismatch; and the matched engine takes $1.35\times$ (vLLM’s Marlin, also unrounded, $1.26\times$) the GPU-hours of corrected W8A8 rollouts on the H100.

9B, one L40S		27B, one H100
Placement of the trainer’s base	s/step ↓	GiB ↓	s/step ↓	GiB ↓
One bf16 copy, KV keptᵃ	2,216	8	OOM
Two FP8 copies	OOM		OOM
bf16, trainer offloaded	1,314	48	659	160
FlashRL, trainer offloaded	1,076	41	421	126
EpiLoRA, FP8 trainer offloaded	1,025	32	418	94
EpiLoRA, one FP8 copy	1,012	9	422	9

Placement	GPU-h ↓ run	GPU-h ↓ per M tok.	Gen. s ↓	Host ↓	Tokens
bf16, offloaded	14.67	0.91	434	160	162k
FlashRL, offloaded	9.84	0.60	247	127	164k
EpiLoRA, FP8 offloaded	9.56	0.62	245	95	155k
EpiLoRA, one FP8 copy	10.05	0.61	257	10	165k

Placement of the trainer’s base	Host, training 27B (GiB) ↓	48 GB L40S 27B, 2,048 ↓	48 GB L40S 9B, 8,192 ↓	80 GB H100, the 27B 8,192 ↓	80 GB H100, the 27B 16,384 ↓
FlashRL: FP8 engine, bf16 trainer, two copies	–	×c	×a	×a	×a
FP8 engine and trainer, two copies	9	×c	OOM	OOM	×a
bf16, one shared copy, KV cache kept	6b	×c	OOM	OOM	OOM
bf16, trainer’s base offloaded (vLLM-style)	159	×c	1314	659	OOM
FlashRL, trainer’s base offloaded	125b	×c	1076	421	606d
FP8, trainer’s base offloaded	93	415	1025	418	673
FP8, one shared copy	10	OOM	1167	476	n/r
FP8, one copy, KV cache released (EpiLoRA)	8	404	1012	422	701

Related Work

Multi-adapter serving. Other systems accelerate per-request and per-token adapters ([61, 20]), tile adapters of mixed ranks ((Mi et al., 2025)), multiply each group of rows by its own matrix ((Ye et al., 2025), or cache, pool, compress and co-quantize adapters or share KV caches across them ([64, 18, 66, 68, 1, 59] (Yu et al. 2026b)). All of them compute adapters beside the base GEMM (Table 9); EpiLoRA can be their kernel layer.

Serving system	Many tenants	Fused	Low-precision adapters
Punica, S-LoRA	✓	×	×
vLLM, SGLang	✓	×	×
TensorRT-LLM	✓	×	✓
SVDQuant	×	✓	×
LoRＡFusion2	✓	✓	×
FlexLLM1	✓	×	×
EpiLoRA	✓	✓	✓
RL system	Low-precision rollouts	One copy	Matched
FlashRL	✓	×	×
Jet-RL3	✓	×	×
QeRL	✓	×	✓
Unsloth4	✓	✓	✓
Unsloth, FP83	✓	✓	×
EpiLoRA	✓	✓	✓

Low-precision serving. AWQ and QServe quantize weights to 4 bits with 16- or 8-bit activations ([27, 28]), Marlin provides fast mixed-precision GEMMs ([7]), and Hopper and Blackwell GPUs support FP8 and block-scaled 4-bit formats natively ([37, 48, 40]); EpiLoRA keeps each engine’s quantization bit for bit and changes only how adapters are computed around the GEMM.

Fusing low-rank branches into GEMMs. SVDQuant’s Nunchaku fuses one low-rank branch, shared by every token, into the 4-bit GEMMs of diffusion models ([23]). For fine-tuning, LoRAFusion fuses the up-projection into the bf16 base GEMM and routes the tiles of a mixed-adapter batch to their adapters ([75]). K2 does the same for serving over FP8 and NVFP4 GEMMs, gathering rows sorted by adapter as mixture-of-experts GEMMs gather rows sorted by expert ([9]). The decode path applies a known principle, overlapping a producer with its consumer’s main loop, as systems that overlap a GEMM with communication between GPUs do tile by tile ([58, 42, 2]) and NanoFlow does across a batch’s operations ([74]). Unlike weight prefetches issued before a dependent kernel’s wait, our producer is a side branch that needs only in its epilogue, so its whole main loop overlaps it; LoRAFusion needs the down-projection before its GEMM starts, and two-stream modes add the adapter’s output after the GEMM ends. Compressing adapters ([43, 29, 39] (Yadav et al. 2023)) and training them for low-precision merges ([6, 11] (Li et al. 2024b; Xu et al. 2024; Li et al. 2026)) are complementary: they change the adapters.

Low-precision RL. FlashRL, our RL baseline, pairs FP8 rollouts with truncated importance sampling beside a bf16 trainer ([30]). Quantized rollouts also speed up full-weight RL, which must re-quantize the weights at every step (Hu et al., 2026; Miah & Belilovsky, 2026); other systems train and generate at one low precision, rounding activations on both sides, which leaves a mismatch unless both sides’ kernels agree bit for bit ([44, 3] (Xi et al. 2026; Gu et al. 2026)), correct the rounding of 4-bit rollouts ([76, 60]), or sample from 4-bit weight-only kernels ([17]). Adaptive importance sampling tunes the correction ([73]) and score centering removes the drift a mismatched sampler adds to the policy gradient ([32]); the matched engine instead removes most of the mismatch itself. FlexLLM co-serves inference and fine-tuning on one bf16 base ([36]), batch-invariant kernels make engine and trainer agree bit for bit (He & Thinking Machines Lab, 2025; vLLM Team, 2025), Unsloth shares one 4-bit or FP8 copy of the base between engine and trainer and releases its KV cache while training (Unsloth AI, 2025b;c), and a TRL patch shares FP8 weights with vLLM (ingambe, 2026). QLoRA first trained adapters on a frozen quantized base ([6]). EpiLoRA trains on the engine’s FP8 codes with bf16 activations, and its matched engine keeps them unrounded, as Marlin and QeRL do, in the trainer’s layout.

Conclusion

Adapters have stayed outside the low-precision GEMM, so a cheaper base makes them relatively costlier to serve. EpiLoRA, a drop-in vLLM plugin, brings them back into the GEMM: on Ada GPUs it fuses every tenant’s adapter into its own low-precision GEMM, and on Hopper and Blackwell GPUs it launches the down-projection first and lets the GEMM stream its weights while the adapter is computed, waiting for it only in the epilogue. This change makes one adapter at batch 1 faster than vLLM, SGLang and TensorRT-LLM on Hopper GPUs, and with many tenants its adapter kernels keep more of the base’s throughput than vLLM’s on most GPU-model pairs we tested. The same low-precision weights also serve RL: a trainer that computes on them needs no bf16 copy of the model, so a 27B model trains on one 48 GB GPU, and an engine that leaves activations unrounded needs no importance correction. Served adapters keep the accuracy of vLLM’s LoRA path, and on the 9B, corrected or matched FP8 RL ends where bf16 training does. Sharing one FP8 copy between engine and trainer saves most of a second copy’s memory, which matters most for models that do not fit twice on a GPU; with long rollouts it nears the fastest placement with a fraction of its host memory. More broadly, a kernel can wait for its producer where it first needs the data, not where it starts.

These appendices hold the measurements behind the paper’s claims. Appendix A discusses where EpiLoRA helps and its limitations, and Appendix B what each part of the RL system buys; Appendices C to F give the closest systems and the kernel, serving and RL measurements; Appendix G gives the code and the software.

Discussion and Limitations

Where a single low-precision base helps. EpiLoRA helps where the adapter’s cost or the base’s memory is the bottleneck. At small decode batches, vLLM’s LoRA kernels add 35–55% to a Hopper layer, and the decode path removes most of it. With many tenants, EpiLoRA’s adapter kernels keep more of the base’s throughput, and adapters stored in low precision let more of them stay resident. In RL, an FP8 trainer removes the bf16 copy of the base, which lets larger models train on smaller GPUs. Other engines could adopt the decode path by adding the up-projection in a CUTLASS epilogue that waits for a down-projection launched first. Computing the down-projection inside the normalization kernel that already reads the input would hide it for the query/key/value and gate/up projections, whose inputs come from a normalization, but not for the output and down projections. Its gain is bounded by what stays exposed today: at one decode token, the exposed down-projection and the second kernel’s launch together take 1.4–2.2 µs of the 10–12 µs a rank-64 adapter still adds to a Hopper layer Table 13; the rest is the epilogue’s extra work, so a leaner epilogue is the next step.

GPU	Tokens	GEMM vs. CUTLASS	Epilogue	Exposed + launch	Adapter total
H100	1	−5.0	10.2	2.2	12.5
H100	4	−5.9	9.6	1.8	11.4
H100	8	−5.9	9.0	1.8	10.8
H100	16	−2.3	7.7	3.1	10.8
H200	1	−1.9	8.8	1.4	10.2
H200	4	−3.3	7.6	1.6	9.1
H200	8	−2.4	7.8	1.5	9.3
H200	16	+0.1	6.7	2.5	9.2

Where it does not. Where neither memory nor latency is the bottleneck, simpler designs are as good or better. Where two copies fit, FlashRL’s recipe is the cheapest: on the 9B at 2,048-token rollouts on an H100 it takes 2.96 GPU-hours per run, against 3.10 for our FP8 trainer with TIS (the difference is in training) and 3.22 for bf16 training. At 2,048 tokens one copy saves memory, not time, on the 9B: one bf16 copy that keeps its KV cache trains as fast as one FP8 copy on the L40S (174 against 173 s per step), while on the 27B the FP8 copy is 1.33× faster Table 37. With rollouts of up to 8,192 tokens, the bf16 copy that keeps its KV cache takes 2.2× EpiLoRA’s step time on the 9B, or 1.9× that of an FP8 copy that also keeps it Table 38; Unsloth’s bf16 mode, which releases the cache while training and which we did not run, holds bf16 offloading’s memory without its weight transfers, so it would take at most bf16 offloading’s 1.3×. The matched engine samples more slowly than a W8A8 engine: it takes 1.35× the GPU-hours of corrected W8A8 rollouts on the H100, almost all in generation, and 0.95–1.03× on the L40S Table 35 and Table 36. vLLM’s Marlin engine, which also keeps activations in 16 bits, reaches the same accuracy at 3.9 GPU-hours, so the matched engine’s advantage, a layout the trainer can share, is unproven; on the L40S one copy ended 3.4 points below bf16 training ([−5.7, −1.1]). With long rollouts on the 27B, FlashRL’s steps trained 19% more tokens than ours in the 12-step probes Table 39, and in the full 100-step runs its recipe needed 2–3% fewer GPU-hours per trained token than ours Table 7, since its bf16 trainer is faster; at 16,384 tokens the bf16 trainers run out of memory, and FlashRL’s recipe fits only with chunked log-probabilities and then takes 0.86× our step time while training fewer tokens per step Table 38. A trainer that rounds its activations as the engine does (our proxy for Unsloth’s FP8 trainer; S8RA in Table 30) trains 1.2× as fast as ours on the L40S (35–39 against 45–46 s of training per step, from the runs’ logs), at 1.4–1.6× our trainer’s mismatch with the engine. With 32 tenants, TensorRT-LLM delivers 1.05× EpiLoRA’s throughput on the H200 in its own job, SGLang 0.5.9 delivers 1,803 tokens/s on Qwen3-8B on the L40S against EpiLoRA’s 1,761 in Table 2’s job (1.02×, in separate jobs; SGLang’s engine without LoRA delivers 3,179), and vLLM’s kernels tuned for Llama-3.1-8B match ours.

Configuration	Runs	Generation ↓	Training ↓	Total ↓	Generation share
bf16 engine and trainer	9	2.00	1.22	3.22	62%
FlashRL’s recipe	9	1.77	1.19	2.96	60%
W8A8 with TIS, FP8 trainer	9	1.74	1.36	3.10	56%
matched engine, FP8 trainer	9	2.81	1.39	4.20	67%

Placement	s/step ↓	generation (s) ↓	response (tokens)	trained tokens	KV (GiB) ↑
9B, one L40S
One bf16 copy, KV kepta	2,216	1,886	7,992	527k	4.2
bf16, trainer offloaded	1,314	963	7,902	558k	15.3
FlashRL, trainer offloaded	1,076	732	7,906	549k	21.4
EpiLoRA, FP8 trainer offloaded	1,025	725	7,888	544k	21.4
EpiLoRA, one FP8 copy	1,012	722	7,795	531k	21.4
27B, one H100
bf16, trainer offloaded	659	521	5,575	244k	12.6
FlashRL, trainer offloaded	421	286	5,549	248k	34.8
EpiLoRA, FP8 trainer offloaded	418	291	5,549	206k	34.8
EpiLoRA, one FP8 copy	422	290	5,545	207k	34.8

Limitations.

1. Hardware. Datacenter Blackwell GPUs (sm_100) are untested; our Blackwell results come from a MIG slice of an RTX PRO 6000 (sm_120). sm_100’s shorter main loops would leave more of the down-projection exposed. The first-wave window used in Figure 5 is an empirical fit chosen after the data, and it overestimates the exposure beyond two waves. Our Triton GEMM trails CUTLASS on the H100 above 32 tokens, and at one token by up to 28% on narrow tensor-parallel shards and on small models Table 16.

Shapes	$M$	$n$	K2 / CUTLASS $\downarrow$
			min	median	max
Qwen3-8B	1–16	16	0.93	0.96	1.00
Qwen3-1.7B	1	8	0.92	1.08	1.12
Qwen3-0.6B	1	8	1.15	1.16	1.24
Qwen3-8B, TP2	1	4	0.98	1.15	1.28
Qwen3-8B, TP4	1	4	0.97	1.06	1.23
Qwen3-8B, TP8	1	4	0.99	1.19	1.22
Qwen2.5-72B, TP2	1–32	20	0.93	0.99	1.07

2. Scope of serving. The tensor-parallel path is opt-in and was tested only at two-way parallelism on one 72B model, where its bf16-head requires tiles searched for the shards and where it ties vLLM with 8 tenants. Mixture-of-experts LoRA and block-scaled FP8 bases are untested, and speculative decoding was tested only with n-gram drafts on one GPU, where the decode path serves verification steps in 0.81–0.82× vLLM’s time per token except in runs whose drafts stopped matching after a first token that run-to-run nondeterminism flipped, as in some of vLLM’s own runs. TensorRT-LLM ran on one model, with its PyTorch backend and a single-process worker.

3. Workloads. Serving speed uses random adapters and, except for one WildChat trace, synthetic requests; storage accuracy uses only RL-trained adapters; and every model except Llama-3.1-8B is a Qwen model.

4. Default CUDA-graph sizes. At vLLM’s default capture sizes, EpiLoRA’s operators run eagerly for prefill steps beyond 512 tokens. It still leads online at every rate on the H200, on the 72B with tensor parallelism and on the

H100 with long prompts, but with 512-token prompts it trails vLLM on the H100 at 16 requests/s ($1.18\times$ its time per output token, p99 TTFT $1.11$–$1.20\times$) and trails it slightly on the L40S at 4–6 requests/s ($1.02\times$, p99 TTFT up to $1.32\times$; Table 24). The kernel-level account (Table 27): the timed dispatch, which times its candidates on CUDA-graph replays, mixes vLLM’s adapter kernels into some projections and so pays both systems’ launch costs on the H100’s host CPU, and EpiLoRA’s own down-projection took twice vLLM’s GPU time at these batch sizes. Handing every batch above 32 tokens to vLLM’s kernels widens the gap ($1.23\times$ and $1.26\times$ at 8 and 16 requests/s). Two opt-in settings chosen after these diagnostics, EpiLoRA’s own kernels above the capture bound and tiles searched for large batches, narrow it to $1.07\times$ at 16 requests/s with p99 TTFT within $1.1\times$. Timing the dispatch’s candidates as they run outside the graphs, host time included, would let it choose these settings where they help.

	H100 vLLM	H100 EpiLoRA	H100 own kernels	H100 own, new tiles	H200 vLLM	H200 EpiLoRA	H200 own kernels	H200 own, new tiles
Step time (ms)	66.7	71.0	71.7	67.4	65.5	61.4	68.6	64.5
Host time, profiled (ms)	75.3	88.2	74.3	71.9	65.7	59.2	66.2	62.8
GPU time (ms)	65.6	60.6	67.3	62.9	68.0	59.0	66.5	61.1
CUTLASS FP8 GEMM	22.6	22.7	22.7	22.7	22.4	22.3	22.4	22.4
down-projection (shrink)	5.0	5.2	11.5	6.8	5.6	6.8	13.0	7.3
up-projection (expand)	20.9	17.5	17.8	18.1	22.7	16.0	17.2	17.4
activation quantization and norms	11.5	9.6	9.8	9.9	11.9	8.6	8.7	8.8
everything else	5.6	5.7	5.5	5.5	5.3	5.3	5.2	5.2

• (v) Co-serving. Serving tenants from the copy that RL trains on works but does not pay off: on one L40S it saves only 5–8% of the GPU-hours of two separate deployments ($0.92$–$0.95\times$), RL’s steps take $1.75\times$ as long, and the tenants’ p99 time to first token rises from 0.26 to 2.87 s.

• (vi) Batch-dependent numerics. A request’s tokens are computed by the decode path in decode batches of up to 32 tokens and by the vendor GEMM with adapter kernels above, so greedy outputs can change with the batch’s composition, as they do between vLLM’s own kernel configurations: on 16 prompts, every alternative configuration, vLLM’s tuned and dual-stream kernels included, diverges from vLLM’s default path on 5–9 in the output check of our H100 end-to-end test, where EpiLoRA’s decode path diverges on at most one more; without adapters, the two engines’ outputs are identical. Deployments that need batch-invariant outputs need batch-invariant kernels ([13]).

What Each Part of the RL System Buys

Table 8 summarizes, for each part of the RL system of § 3.3, what it buys and what it costs, measured on Qwen3.5-9B on one L40S unless noted; Table 36 gives the full per-configuration accounting. Running the matched engine, one copy and the KV release together in a 30-step probe takes 153 s per step, against 155 s with the KV cache kept, with twice the KV cache (23 against 12 GiB).

Part	Benefit	Cost
FP8 trainer weights	6.5 GiB less GPU memory for the 9B and 22 GiB for the 27B; the 27B trains on one 48 GB L40S	None measurable: 7.2 GPU-hours, the same as FlashRL’s bf16 trainer
One shared copy	Pinned host memory falls from 48–159 GiB to 7–11 GiB	None measurable: 7.1 against 7.2 GPU-hours with offloading
KV-cache release	A 21 GiB instead of a 10 GiB KV cache during rollouts	None measurable at 2,048 tokens; at 8,192 tokens the 27B’s KV-releasing copy took 1.05× FP8 offloading’s GPU-hours
Unrounded (W8A16) rollouts, ours or Marlin’s	No correction needed: uncorrected runs end where W8A8 with TIS does	On the H100, 4.2 GPU-hours (Marlin 3.9) against 3.1 for W8A8 with TIS; on the L40S, 8.8 against 9.3 resident and 7.2 offloaded; with one copy at engine share 0.55, −3.4 points [−5.7, −1.1] against bf16

The closest systems

Table 9 compares EpiLoRA’s capabilities with those of the closest serving and RL systems. Services that train and serve many users’ adapters on shared bases (Tinker, MinT, MARLaaS) do not document their base precision or kernels, so they are not in the table.

Kernel details

One output tile of K2 (Figure 10). K2 runs the low-precision GEMM over tokens sorted by adapter, so each output tile meets a few contiguous groups of rows, and its epilogue adds each group’s up-projection.

Table 1’s L40S column comes from the weight-only kernel benchmark: Marlin’s W8A16 GEMM with vLLM’s tuned LoRA kernels against EpiLoRA’s fused weight-only kernel, one Qwen3-8B layer at one decode token with one rank-64 adapter, CUDA graphs with the L2 cache flushed; its other columns come from the rows of Table 10.

Table 10. On the H100, the H200 and the Blackwell slice, the decode path brings one adapter’s cost to a few percent of the layer. One Qwen3-8B decoder layer (the four projections summed; FP8 base on the Hopper GPUs, our NVFP4 checkpoint on the Blackwell slice), one rank-64 adapter, CUDA graphs with the L2 cache flushed, microseconds (lower is better); each path at its best tile configuration. Previous: the plugin’s best path before the decode path (Hopper: hybrid or swap-AB; Blackwell: fused or hybrid). Bold: the fastest path with the adapter. Last column: the PDL path’s time over CUTLASS alone.

µs ↓		vLLM’s LoRA path			Ours			PDL over
Tokens	CUTLASS alone	default	tuned	dual-stream	previous	no PDL	PDL	CUTLASS (%)
H100, FP8 base
1	126	205	179	180	188	163	134	+6
4	128	195	179	172	192	162	133	+4
8	128	195	179	172	191	157	133	+4
16	128	196	180	174	181	167	136	+7
H200, FP8 base
1	104	190	161	167	171	144	112	+8
4	106	180	162	157	174	147	112	+6
8	106	180	162	156	175	144	113	+7
16	106	180	162	159	164	150	115	+9
Blackwell slice, NVFP4 base
1	286	388	342	340	316	330	293	+3
4	285	360	339	325	317	339	298	+5
8	288	359	343	326	320	342	298	+3
16	291	359	348	334	324	345	312	+7

The decode path across decode batch sizes (Table 10). From one to sixteen tokens, one rank-64 adapter adds 4–9% to a Qwen3-8B layer on Hopper GPUs and 3–7% on the Blackwell slice with the decode path, against 34–61% and 13–20% for vLLM’s tuned or dual-stream kernels.

Controls for the overlap (Table 11 and Table 12). Serializing a library-fast down-projection (cuBLAS’s GEMM or GEMV) before the same GEMM leaves the layer 1.17–1.25× slower than the decode path, and our own down-projection at its fastest split-K 1.13–1.17×; only a serialized kernel that does no work comes within 2%.

Where the adapter’s remaining cost goes (Table 13). Under the decode path, most of what the adapter still adds to a layer is the epilogue’s extra work; the exposed down-projection and its launch take 1–3 μs per layer, and EpiLoRA’s GEMM without an adapter is up to 6 μs faster than CUTLASS with vLLM’s separate quantization.

Which path the dispatch picks (Table 14). At engine start, the plugin times every candidate path per projection shape and batch-size bucket, on CUDA-graph replays with one adapter, and keeps the fastest. On the H100 with 32 rank-64 adapter slots it picks the decode path for every projection up to 32 tokens; above, it picks EpiLoRA’s adapter kernels after the vendor GEMM for the projections that combine three or four outputs, where the projection, vendor GEMM included, takes 0.53–0.85× its time with vLLM’s kernels, and vLLM’s own adapter kernels for the others.

**Why EpiLoRA’s adapter kernels win on these projec-

Table 12. Our own down-projection at its fastest split-K, run before the same GEMM without overlap, still trails the decode path. Table 11’s cases: one Qwen3-8B layer’s four projections at one decode token with one adapter, the down-projection at every split-K from 1 to 32, serialized before the GEMM; per projection the fastest split, microseconds summed over the projections (CUDA graphs). Ratio: serialized over the decode path, for the layer and per projection. Shrink alone: the down-projection at the decode path’s split and at the fastest.

[tions](Table 15). In the profiled step of Table 27, EpiLoRA’s GPU time on the H100 is 5.0 ms below vLLM’s: 3.4 ms in the up-projection (expand) and 1.9 ms in activation quantization and norms, which EpiLoRA’s base path computes with vLLM’s CUDA quantization kernel rather than the kernels vLLM’s compiler generates (as much with EpiLoRA’s own adapter kernels throughout, so not from them), while its down-projection takes 0.2 ms more. We attribute the up-projection’s difference to how the kernels index a projection’s outputs. vLLM’s expand kernel treats them as slices and launches one program per adapter, slice and tile of the widest slice, so the programs of narrower slices past their width exit without work. With several slices it reads each slice’s starting column in the output from memory at run time, and when their widths differ, as for attention’s query, key and value under grouped-query attention (8,192, 1,024 and 1,024 columns in Qwen3.5-9B) and for the linear-attention input (2,048, 2,048, 4,096 and 4,096), also each slice’s strides, which the compiler then cannot specialize. EpiLoRA’s expand-add tiles the concatenated output instead, each tile inside one slice, and receives the slice boundaries as kernel arguments. The costs per output column are consistent with this account: with tiles tuned for these shapes, vLLM’s expand takes 7.9 ns on the projections with one output, 13.4 ns on gate/up’s two equal ones and 21.6 ns on the three or four unequal ones, while EpiLoRA’s takes 12.1–13.1 ns on every kind, so on a single output vLLM’s one kernel is the faster and the dispatch keeps it there. On the H200, vLLM’s expand takes 17.4 ns per output column on the projections with two to four outputs against EpiLoRA’s 11.7, and 8.8 against 13.1 ns on those with one. We did not time vLLM’s kernel launched once per slice or with the slice offsets as constants, either of which might close the gap, and these costs come from one model’s 2,048-token step outside CUDA graphs; at the 64–512-token batches of the many-tenant runs, Table 14 shows the same ordering by kind of projection. The gain also depends on the model: on Llama-3.1-8B, whose query, key and value have Qwen3-8B’s widths, vLLM’s tuned kernels tie EpiLoRA with 32 tenants (Table 2). EpiLoRA’s kernels read vLLM’s own adapter buffers, so vLLM could adopt them.

When K2 is slower than CUTLASS (Table 16). Without an adapter, K2 matches CUTLASS on the 8B’s and the 72B’s shapes and trails it on small and narrow ones, which lengthens its main loop and so widens the window that hides the down-projection.

The exposure model (Table 17). On every measured projection, the exposed time against the down-projection’s time and the GEMM’s main loop. The window of the GEMM’s first wave of thread blocks, assuming one block per SM, was fitted after the data; with the resident blocks per SM that the CUDA occupancy API reports (two in most Hopper cases), the same window tracks the exposure less well.

GPU	Shapes	Cases	$t_{\mathrm{shrink}}$ ($\mu$s)	$t_{\mathrm{main}}$ ($\mu$s)	median	max	whole loop	first wave
H100	rank 64	24	10.4–18.3	16.7–50.5	0.6	1.3	0.6	0.6
H100	rank 16	8	11.8–15.5	18.7–50.8	2.6	6.1	2.6	0.7
H100	rank 128	8	14.0–18.5	18.7–50.8	5.0	9.3	5.0	2.3
H100	TP 2 shards	4	10.0–14.5	15.0–33.3	3.7	5.1	3.7	2.1
H100	TP 4 shards	4	10.7–14.2	14.4–20.6	2.5	4.3	2.5	1.1
H100	TP 8 shards	4	10.4–14.3	12.6–19.3	3.1	5.4	2.5	1.9
H200	rank 64	16	9.2–17.8	14.9–41.5	0.4	1.2	0.4	0.4
H200	rank 16	8	8.4–14.7	15.3–41.4	0.2	1.0	0.2	0.2
H200	rank 128	8	9.8–18.2	15.3–43.0	1.5	5.7	1.5	1.5
H200	TP 2 shards	4	9.5–13.7	14.2–28.2	3.2	4.8	3.2	1.7
H200	TP 4 shards	4	9.9–13.7	13.1–17.6	1.5	4.8	1.5	1.1
H200	TP 8 shards	4	9.7–13.5	11.7–17.4	3.2	5.7	2.7	1.9
GH200	TP 8 shards	3	9.4–13.8	11.4–17.0	4.3	4.9	2.5	2.5
GH200	Qwen3-0.6B, -1.7B	16	7.8–11.5	12.3–18.0	0.0	0.5	0.1	0.1
Blackwell slice	rank 64	16	15.0–29.7	31.6–126.0	0.7	2.1	0.9	0.9

Many distinct adapters (Figure 11). With one adapter in a 32-token batch, EpiLoRA adds 11% to the H100 layer where vLLM’s tuned kernels add 38%; with 32 distinct adapters the adapters’ own weights dominate the step, and storing them in FP8 or NVFP4 nearly halves the overhead on the L40S.

End-to-end serving

Batch 1 against throughput (Figure 12). Every vLLM configuration and every EpiLoRA path on each GPU, one point per configuration.

SGLang and TensorRT-LLM (Table 18 to Table 20). Each system ran in its own job beside EpiLoRA, which gives the same-job ratios of Table 3; Table 21 gives the GPU profiles behind its Nsight row.

	H100		H200
	TRT-LLM	EpiLoRA	TRT-LLM	EpiLoRA
Time per token, no LoRA (ms) ↓	4.59ᵇ	4.52	4.08ᵇ	4.11
Time per token, one adapter (ms) ↓	6.93ᵃ	4.85	6.89ᵃ	4.53
Throughput, one adapter (tok/s) ↑	6,858ᵇ	7,703	7,256ᵇ	7,654
Throughput, 32 adapters (tok/s) ↑	5,901ᶜ	6,072ᶜ	6,269ᵇ	5,985

System	Time per token (ms) ↓		Added GPU time (ms) ↓
	no adapter	one adapter
vLLM, two streams	4.39	6.52	1.87
SGLang	5.10	8.23	3.01
TensorRT-LLM	4.60	7.50	2.87
EpiLoRA	4.39	4.92	0.40

Served adapters’ accuracy (Table 22). The test behind Figure 8.

	Accuracy (%) ↑			One	Difference
Benchmark	vLLM	EpiLoRA		only	(95%)	$p$
bf16-trained adapter
MATH-500	91.4	90.4		14/9	−1.0 ± 1.9	0.40
Olympiad	61.8	63.4		36/47	+1.6 ± 2.6	0.27
Minerva	54.4	54.4		7/7	+0.0 ± 2.7	1.00
W8A8-trained adapter
MATH-500	89.0	90.2		13/19	+1.2 ± 2.2	0.38
Olympiad	58.4	58.5		40/41	+0.1 ± 2.6	1.00
Minerva	55.5	55.9		8/9	+0.4 ± 3.0	1.00

With CUDA graphs over the prefill sizes (Table 23). Every point of Figure 6’s solid series, and the same test with rank-16 tenants.

GPU	Rate	Median TPOT (ms) ↓				p99 TTFT (ms) ↓
		vLLM	EpiLoRA	Ratio	vLLM	EpiLoRA	Ratio
H100	2	9.9	7.9	0.79	213	254	1.20
	4	15.5	13.0	0.84	238	274	1.15
	8	28.9	29.0	1.00	268	305	1.14
	16	54.8	64.7	1.18	360	400	1.11
H100, new tiles	2	9.9	7.7	0.78	214	242	1.13
	4	15.5	12.5	0.81	243	262	1.08
	8	29.0	27.7	0.95	269	289	1.08
	16	54.9	61.1	1.11	361	386	1.07
H100, new tiles, own kernels	2	9.9	7.5	0.76	214	213	1.00
	4	15.5	11.8	0.76	243	240	0.99
	8	29.0	24.9	0.86	269	266	0.99
	16	54.9	58.6	1.07	361	386	1.07
H200	2	8.8	6.5	0.74	96	111	1.17
	4	12.0	8.9	0.74	120	138	1.14
	8	21.2	17.6	0.83	166	164	0.99
	16	46.6	43.1	0.93	270	243	0.90
L40S	1	23.7	21.9	0.93	235	241	1.03
	2	32.0	31.1	0.97	255	262	1.03
	4	51.4	52.2	1.02	332	339	1.02
	6	86.9	88.5	1.02	3,062	4,054	1.32

The WildChat trace (Table 5). Of 4,000 first turns, vLLM’s benchmark keeps the 3,383 with at most 1,024 prompt and 2,048 total tokens: prompts have a median of 23 tokens (90th percentile 247), replies 253 (759), and each request generates its recorded reply length. Each start draws 1,000 of them, a different sample per start and the same for both systems, with 85–102 prompt and 334–345 reply tokens on average. vLLM’s p99 TTFT of 1.37 s on the H100 at 8 requests/s comes from one of its three starts (3.7 s; the other two 0.19–0.20 s); EpiLoRA’s p99 TTFT there is 1.06–1.13× vLLM’s at the other three rates.

At vLLM’s default CUDA-graph sizes (Table 24). Steps that prefill more than 512 tokens run outside CUDA graphs; with 512-token prompts and 32 tenants, EpiLoRA lowers the time per output token at every rate on the H200 and at low rates on the H100 and the L40S.

Long prompts, a 72B model and host time (Table 25 and Table 26). At vLLM’s default CUDA-graph sizes, with direct launches, EpiLoRA lowers the time per output token with 2,048-token prompts on both Hopper GPUs and on a 72B model; Table 26 gives each path’s host and GPU time per layer for the steps that run outside CUDA graphs.

Path, $\mu\mathrm{s}$ ↓	H100 512 tokens host	H100 512 tokens GPU	H100 2,048 host	H100 2,048 GPU	H200 512 tokens host	H200 512 tokens GPU	H200 2,048 host	H200 2,048 GPU	L40S 512 tokens host	L40S 512 tokens GPU	L40S 2,048 host	L40S 2,048 GPU
vLLM’s quantize, GEMM and LoRA kernels	648	916	639	2211	299	982	287	2332	460	1579	454	4320
the vendor GEMM alone	136	208	138	732	68	215	69	730	101	713	100	2952
ours, hybrid	541	870	532	1769	260	875	258	1756	405	1689	402	4211
ours, hybrid, direct launches	245	873	243	1768	125	876	125	1756	180	1686	181	4354
ours, fused	350	1699	345	3921	174	1804	171	4164	265	1554	259	3490
ours, fused, direct launches	106	1700	106	3919	59	1802	59	4142	80	1559	81	3474

Where a step outside CUDA graphs spends its time (Table 27). In a step that prefills 2,048 tokens, EpiLoRA’s default configuration takes less GPU time than vLLM but more host time, because its timed dispatch picks vLLM’s adapter kernels for some projections and pays both systems’ launch costs, which the H100’s host CPU makes larger.

Restricted to its own kernels, EpiLoRA matches vLLM’s host time but its down-projection takes about twice vLLM’s GPU time at this batch size. Tiles searched for large batches remove most of that difference, bringing the step to 1.01× vLLM’s time on the H100 and 0.98× on the H200. Online on the H100, the two settings together narrow the gap at 16 requests/s from 1.18× to 1.07× and bring the p99 time to first token within 1.1× at every rate (Table 24). On the H200, whose host CPU is faster, EpiLoRA’s default dispatch, which mixes in vLLM’s kernels, is faster still (0.94× vLLM’s step time), so the settings stay opt-in.

The 72B model (Table 28 and Table 29). Across two H100s with tensor parallelism, against vLLM’s LoRA kernels tuned for the shard shapes, and on one H200 with the timed dispatch and with the decode path forced on every projection.

Path	W1 (ms) ↓	over no LoRA ↓	W2 (tok/s) ↑
vLLM, tuned	20.40 [20.31, 20.54]	+24%	2,141
vLLM, dual, tuned	20.31 [20.25, 20.34]	+24%	2,017
EpiLoRA, searched	17.75 [17.70, 17.77]	+8%	2,124

Path	W1 (ms) ↓	over no LoRA ↓	W2, one (tok/s) ↑	W2, 8 (tok/s) ↑
vLLM, tuned for the 72B	28.20 [28.06, 28.27]	+21%	1,042	980
vLLM, tuned, two streams	29.18 [29.17, 29.19]	+25%	994	926
EpiLoRA, timed dispatch (searched tiles)	26.50 [26.48, 26.51]	+13%	1,038	974
EpiLoRA, decode path forced	27.32 [27.31, 27.32]	+17%	1,041	975

LoRA RL

Every configuration (Figure 13, Table 30 and Table 31). Final accuracy of each RL configuration against bf16 training on the same GPU, with its cost; Table 32 tests each comparison for equivalence within ±5 points, also paired by seed and on one common engine.

Engine, LoRA path	W1 (ms) ↓	W2 (tok/s) ↑	W1 ↓	W2 ↑	W1 ↓	W2 ↑
			1 adapter (%)		32 adapters (%)
SGLang 0.5.9, csgmv	4.69	10,192	+74.9	−45.3	+74.9	−45.3
SGLang 0.5.9, triton	4.69	10,192	+78.4	−79.1	+78.1	−79.0
vLLM 0.28, default kernels	4.11	10,742	+67.9	−28.8	+106.9	−56.6
vLLM 0.28, dual-stream	4.11	10,742	+52.1	−30.8	+90.7	−55.9
ours, decode path	4.11	10,742	+10.4	−28.6	+12.5	−44.4

The decomposition on a second model (Table 33). On Qwen3-4B, a dense model, with five of its math RL adapters, the ordering of Figure 7 holds: rounding activations leaves 6.3–7.4× the bf16 pair’s KL, most of it the activations’ rounding itself (the fake-quant control), and the unrounded engines retain 1.1–1.6×.

The mismatch over training (Table 34). Two hundred steps cannot show a collapse that comes later, but the mismatch’s trend can: with the matched engine and no correction it stays at the level of a bf16 engine and trainer, while uncorrected W8A8 rollouts more than double theirs.

Where the GPU-hours go on the H100 (Table 35). Generation takes most of each step, and the matched engine’s extra cost on the H100 is almost all in generation.

What each part costs and buys (Table 36 and Table 37; Figure 9 shows the memory of each placement).

Software and Hardware

Tuning vLLM’s LoRA kernels. For each model and GPU, we record the shape of every shrink and expand call of a serving run and time, for each shape, rank (64, for some models also 16), slot count (1 and 32; 8 for the 72B) and token count (1 to 8,192), vLLM’s default configuration and the 12 to 47 others that run (block sizes, split-K, warps) with CUDA graphs and the L2 cache flushed; we keep the fastest whose output matches the default’s, a median $1.2$–$1.4\times$ faster than the default over 140 to 400 keys per model and GPU. vLLM looks the files up by GPU name: for Qwen3.5-9B, the H200 and GH200 use the H100’s, as we did not tune on those GPUs (on Qwen3-8B, tuning on the H200 itself kept the H100’s configuration for 112 of 160 keys). Software: vLLM 0.28.0, PyTorch 2.13.0, Triton 3.7.1, FlashInfer 0.6.16, Transformers 5.16.1, SGLang 0.5.9 and TensorRT-LLM in their own environments; CUDA 12.8 and 13.1. Hardware: NVIDIA L40S (48 GB, Ada); H100 SXM (80 GB), H200 NVL (141 GB) and GH200 (96 GB, Hopper); A100 (80 GB, RL only); a 2g.48gb MIG slice of an RTX PRO 6000 (Blackwell).