WorksComputer Science
Harvesting Expert-Parallel Bubbles for Fine-Tuning
Abstract
GPUs are expensive, yet latency-critical large language model (LLM) inference leaves part of them idle. Colocation reclaims this capacity by running best-effort work, such as fine-tuning, alongside serving. Many leading LLMs now use a Mixture-of-Experts (MoE) architecture, which adds a new source of idle time. Under expert parallelism, MoE experts are spread across GPUs, and because the router sends uneven numbers of tokens to each GPU, lightly loaded GPUs wait for the most loaded one. Even after applying load balancing solutions, such bubbles still take 8–13% of prefill time. We present BubbleTea, the first colocation system that leverages expert-parallel bubbles for fine-tuning. Unlike existing colocation systems, which make one scheduling decision for all GPUs per forward pass, BubbleTea predicts each GPU's bubble at every layer from the router's token assignment, and fills it with short chunks of fine-tuning work that reuse the weights already loaded for serving. We also introduce compiler passes that automatically derive these chunks from the served LLM as-is, without porting each model architecture to a separate training engine. Our evaluation on three MoE models, served on four A100 GPUs, shows that fine-tuning throughput with BubbleTea is 1.9–6.2 faster than with LLMStation, a state-of-the-art colocation system. BubbleTea keeps p90 inference latency within 10% of serving alone in 50 of 63 workloads, compared to 44 for LLMStation. On bursty production traces, BubbleTea trains 3.3 faster than LLMStation and 14 faster than a stock trainer colocated under CUDA MPS, with p90 inference latency staying within 7% of serving alone.
Introduction
GPU resources are a dominant cost of serving large language models (LLMs), motivating systems that improve accelerator utilization [23, 1, 47, 30, 12, 41] (Yu et al. 2022). Latency-critical inference must meet stringent response-time service-level objectives (SLOs), yet rarely uses every GPU resource at every instant. Colocation systems reclaim this otherwise-idle capacity by running best-effort work, such as fine-tuning, alongside serving as long as its SLOs are preserved [37] (He et al. 2025; Xu et al. 2025).
Many recent frontier LLMs use a Mixture-of-Experts (MoE) architecture [6, 10, 34, 22, 29, 39, 17]. An MoE layer contains many feed-forward experts, and a router sends each token to only a few of them [13], increasing model capacity without proportionally increasing per-token computation. Since for large LLMs the total size of experts is beyond what a single GPU can store, such models are typically served with expert parallelism (EP), which places different experts on different GPUs [6, 36].
EP, however, introduces a new source of underutilization. Because token routing is input-dependent, GPUs receive different amounts of expert work. Lightly loaded GPUs finish early and wait for the results of the most-loaded GPU before computation can proceed. We call the time a GPU spends waiting for the most-loaded GPU a bubble. This load imbalance is well known and heavily optimized through techniques such as custom token routing objectives, strategic expert replication and placement, token redistribution (Fedus et al., 2022; Lepikhin et al., 2021; Zhou et al., 2022; DeepSeek-AI, 2024b; He et al., 2022; DeepSeek-AI, 2025b,c; LMSYS Org, 2026; Doucet et al., 2026), and specialized communication and expert kernels [7, 32, 15]. We build on these mechanisms and ask the question they leave open:
Figure 1. Once load balancing and execution optimizations have done their job, how much idle GPU capacity remains, and can it be leveraged to perform useful work?
Our analysis in § answers the first half of this question and reveals that substantial idle capacity remains. After applying state-of-the-art EP load balancing techniques, bubbles
McGill University ETH Zurich EPFL. Correspondence to: Ramya Narasimha Prabhu <[email protected]>. still account for 8–13% of prefill time. The second half of the question, whether this capacity can perform useful work, is more challenging. Existing colocation systems are natural candidates [28, 2] (He et al. 2025; Xu et al. 2025). They place best-effort work, typically parameter-efficient fine-tuning (PEFT) such as low-rank adaptation (LoRA) (Hu et al., 2022), on the same GPUs as inference. However, they mostly exploit GPU underutilization during output generation (decode) and schedule work at the granularity of an inference or training step, whereas bubbles occur mostly during initial prompt processing (prefill), inside a single MoE layer, and differ in length across GPUs. Existing systems are therefore ineffective in leveraging such bubbles.
Figure 1 (PDF p. 2). BUBBLETEA trains faster than LLMSTATION and faster than uncoordinated colocation on a production trace, while keeping p90 TTFT within 3% of serving alone. Load (top) and fine-tuning throughput (bottom) of BUBBLETEA and baselines over time on a QWEN3-30B-A3B model serving a production (Azure) trace at its original rate (with 30 s bins). Legend values give each system’s p90 TTFT over the whole trace, relative to serving alone.
We introduce BUBBLETEA, a colocation system that leverages bubbles to fine-tune LLMs with LoRA while serving them. The key insight behind BUBBLETEA is that the length of a bubble is predictable. During inference, the router decides how many tokens each GPU receives before the experts run, and using this count, BUBBLETEA predicts each GPU’s bubble duration with a median error of 8.9 µs, about 7% of the median bubble. We introduce the first scheduler that admits fine-tuning work per GPU inside an MoE layer, unlike existing colocation systems: BUBBLETEA runs short chunks of fine-tuning work on each GPU only inside that GPU’s predicted bubble window.
We also design a compiler that turns the served model into training code. A serving engine only runs the model’s forward pass, which pushes tokens through the layers to predict the next one. Training also needs a backward pass, which propagates the error back through the model to compute the gradients that update the LoRA adapter. The compiler takes the serving engine’s forward pass, derives the matching backward pass, and splits the resulting training step into chunks short enough to fit in a single bubble. The generated code operates directly on the weights already loaded for serving, so BUBBLETEA fine-tunes the model without keeping a second copy of its weights in GPU memory. Existing colocation systems either train a second copy of the model (Xu et al., 2025) or require porting each model architecture into their own training engine [28] (He et al. 2025), and they split training no finer than a layer or a serving iteration. Our compiler instead works on the serving engine’s model as-is, including its inference-only MoE kernels and expert placement, so supporting a new architecture requires no hand-written kernels for backward passes.
We implement BUBBLETEA and compare it against two baselines. Since there are no colocation systems specifically designed for MoE, we port an existing colocation system, LLMSTATION (He et al., 2025), to support inference with MoE models, and also compare with uncoordinated colocation, a stock trainer running beside serving without coordination. Figure 1 shows the fine-tuning throughput of BUBBLETEA and both baselines while serving the QWEN3-30B-A3B LLM on a bursty production trace. Over the trace, BUBBLETEA trains 3.3× faster than LLMSTATION and 14× faster than uncoordinated colocation, and it sustains this advantage throughout, including during load bursts. Meanwhile, the 90th-percentile time-to-first-token (TTFT) (p90 TTFT) of BUBBLETEA stays within 3% of serving alone, whereas uncoordinated colocation increases it by 18%. Part of this gain comes from the fine-grained training runtime that bubbles require, which also makes better use of the decode and idle time that LLMSTATION already exploits. Bubbles add the most at high load, when little decode and idle time is left for training, raising throughput by up to 27% (§ 4.3). Our further evaluation across three MoE models and two production traces confirms that BUBBLETEA turns idle time in MoE serving into substantial fine-tuning throughput, while keeping both p90 TTFT and time-per-output-token (TPOT) within 10% of serving alone in 50 of 63 workloads.
This paper makes the following contributions:
A characterization of expert-parallel bubbles, showing that they survive load balancing, are predictable, and need per-GPU planning inside the MoE layer (§ 2).
BUBBLETEA, a novel colocation system that predicts each GPU’s bubble from the router’s decision, schedules short fine-tuning chunks only inside that window,
and uses a compiler to derive those chunks automatically from the served LLM § 3.
An evaluation of BUBBLETEA and baselines on three MoE models and two production traces that breaks the gains down by source of capacity and by mechanism § 4.
Characterizing Expert-Parallel Bubbles
We next characterize expert-parallel bubbles and conclude that they are long enough, persistent enough, and predictable enough to schedule fine-tuning work inside.
MoE models and load imbalance
An MoE model replaces the dense feed-forward network in each transformer block with an MoE layer, consisting of experts and a router that sends each token to its top- experts ([13, 24]). Large MoEs commonly combine tensor parallelism (TP) for dense computation with expert parallelism (EP) for the experts. EP places different experts on different GPUs and routes individual tokens to the GPUs hosting the appropriate expert (dispatch) ([6, 36]). In a TP–EP deployment, every GPU observes the same routing decision but executes only its local experts; the expert outputs are then combined across GPUs before the layer computation proceeds (gather).
Figure 2 shows a timeline of one MoE layer executing on three GPUs. Because routing is input-dependent, experts receive different numbers of tokens ([13, 48, 6]). Under EP, this results in a per-GPU work imbalance: GPUs hosting more heavily loaded experts finish later, while lightly loaded GPUs finish earlier but cannot proceed as they wait for the processed tokens from other GPUs. We call this GPU-local waiting interval a bubble. Specifically, a bubble is defined as the interval between completion of a GPU’s local expert computation and the dependency that allows the computation to proceed. We provide additional details on this in §A.1.
Figure 2. Expert-parallel bubbles. Token routing assigns different amounts of expert work to different GPUs. GPU 0 finishes its local expert computation (in green) before the more heavily loaded GPU 1 but cannot proceed until the cross-GPU dependency clears. We define this waiting interval (in red) as a bubble.
LLM inference has two phases. Prefill processes the whole prompt and produces the first output token; its latency is measured by TTFT, the time from a request’s arrival to its first output token. Decode then generates the remaining tokens one at a time; its latency is measured by TPOT, the time between consecutive output tokens. Serving systems enforce SLOs on both. In an MoE model, the two phases give experts very different amounts of work. During prefill, all prompt tokens pass through each MoE layer in parallel, so expert computation is long and compute-bound. During decode, each request contributes one token per step, so expert computation is short and memory-bandwidth-bound (et al., 2022; Kwon et al., 2023)). As we show next, this difference determines whether routing imbalance turns into bubbles long enough to leverage for useful work.
The characteristics of bubbles
To understand the characteristics of expert-parallel bubbles and how we can best leverage them, we run three experiments and instrument MoE layer execution with CUDA events. We measure, for each GPU and layer, the bubble duration. For each experiment, we provide key details of the experiment setup, and provide additional details in §C.1.
Observation 1: Prefill creates bubbles we can leverage, and that grow with prompt length. Routing imbalance occurs during both prefill and decode, but whether a bubble can be leveraged depends on the resulting absolute difference in expert computation time across GPUs. We first measure routing skew, which we define as the most-loaded GPU’s routed tokens divided by the mean. On the Qwen3-30B-A3B model with experts split across two GPUs and no load balancing, and using arXiv-summarization prompts ([4]), we find that this skew is 1.28× during prefill and 1.75× during decode. Thus, token imbalance is more pronounced during decode than prefill on this dataset. However, because decode expert computation is short, this larger imbalance yields only small bubbles. Across batch sizes 1–128 and arXiv-summarization prompts of 32–512 tokens in length, while generating 30 tokens per request, the median bubble duration during decode is 5 µs to 25 µs per layer. This is insufficient opportunity to effectively schedule work (in § 3 we use training chunks of 50 µs). When performing prefill with arXiv-summarization prompts of length 8k, in contrast, the bubble length is 0.29 ms per layer, which is about 14% of the layer’s expert computation time. Thus, prefill offers far more opportunities than decode to schedule work in bubbles.
The bubble duration during prefill also grows rapidly with the prompt length. Figure 3 shows the median bubble duration during prefill, for varying prompt lengths, on the DEEPSEEK-V2-LITE model, with and without load balancing optimizations. In the Base configuration (no optimizations), the median per-layer bubble duration increases substantially when increasing prompt length from 8k to 128k tokens. The same trend also holds for other LLMs (QWEN3-30B-A3B and QWEN1.5-MOE-A2.7B, see §A.2). Thus, longer prompts create more opportunities to fill bubbles.
Figure 3. Load balancing optimizations reduce bubbles but do not eliminate them. The median per-layer bubble duration on DEEPSEEK-V2-LITE, 8×H100 GPUs, and with BF16 precision, for prompts with varying lengths, with and without (combinations of) load balancing optimizations.
Observation 2: Load balancing optimizations do not eliminate bubbles. We next quantify how much bubble duration remains after applying the following three load balancing optimizations. (i) Expert-parallel load balancing (EPLB) uses historical expert loads to place experts and replicate hot ones across GPUs (DeepSeek-AI, [6, 8]), giving the runtime multiple expert replicas to which tokens can be sent. (ii) linear-programming-based load balancer (LPLB) (DeepSeek-AI, [9]) further balances routed tokens across those replicas at dispatch time using linear programming. (iii) WATERFILL (LMSYS Org, [25]) assigns shared-expert work to the least-loaded GPUs. WATERFILL composes naturally with the other two optimizations: EPLB and LPLB route tokens to physical expert replicas, then WATERFILL balances the shared-expert workload. We therefore also consider their combination.
We evaluate these mechanisms on the DEEPSEEK-V2-LITE LLM using the SGLANG serving engine, 8×H100 with experts split across eight GPUs, and with BF16 precision. Figure 3 shows the median bubble duration across prompt lengths. Load balancing reduces the bubble duration substantially compared to Base but does not eliminate it. With prompt lengths of 128 k, Base exposes 4.79 ms/layer, while LPLB, the most competitive load balancing system, still leaves 3.53 ms of idle time per layer. Load balancing is not free either: replicating experts consumes GPU memory, and redistributing tokens at dispatch time can increase communication volume. For example, WATERFILL raises median dispatch time from 2.50 ms/layer to 2.65 ms/layer (6.0%); correspondingly, LPLB+WATERFILL leaves a 3.76 ms/layer bubble, compared to 3.53 ms/layer with LPLB alone. Thus, minimizing token imbalance by itself does not necessarily eliminate bubbles and these optimizations must be weighed against their memory, communication, and runtime overhead. Lower-precision execution does not close the gap either: our experiments in §A.3 show that FP8 leaves the bubble essentially unchanged relative to BF16.
Observation 3: Serving-level colocation systems miss per-GPU bubbles.
We next investigate whether existing inference–fine-tuning colocation systems can already exploit the bubbles occurring during prefill. LLMSTATION (He et al., 2025) and HARLI (Xu et al., 2025) primarily exploit decode and idle-time underutilization instead of prefill. FLEXLLM (Oliaro et al., [28]) and DELTASERVE (Chen et al., [2]) can place training work alongside prefill, but only when the serving iteration itself leaves sufficient headroom. As the prompt length increases, however, GPU compute saturates quickly (Agrawal et al., [1]) and under a stringent serving SLO, their planners see little or no capacity in which to admit training.
The limitation is not simply the granularity of their training work. These systems measure slack once per serving iteration, using signals such as batch composition, iteration latency, predicted request latency, or aggregate engine headroom (see Table 1 in §B). That is reasonable when idle capacity is spread evenly across the GPUs. EP, however, breaks this: within an MoE layer, the iteration as a whole looks saturated because it runs at the pace of its most heavily loaded GPU, while the other GPUs finish their experts early and sit idle waiting for it. A planner with a single headroom budget per iteration cannot see this per-GPU idle time, so it concludes that prefill has no usable capacity even though substantial bubbles remain on individual GPUs. In a trace-driven experiment, a planner that knows the exact mean bubble of each layer but assigns the same budget to every GPU uses only about 1% of bubble time, whereas planning per GPU from the current routing decision uses 43–45% (Appendix § 9). Thus, leveraging expert-parallel bubbles requires a different admission signal. The scheduler must estimate slack separately for each GPU at each MoE layer, using the current routing decision to determine which GPUs can safely execute work.
| System | Capacity targeted | Work unit | Admission scope | Signal |
| FLEXLLM (Oliaro et al., [28]) | serving headroom | token slot | serving iteration | batch composition |
| LLMSTATION (He et al., 2025) | decode / idle headroom | tasklet | serving iteration | iteration timing |
| HARLI (Xu et al., 2025) | decode underutilization | SM partition | serving iteration | predicted latency |
| DELTASERVE (Chen et al., [2]) | serving / idle headroom | LoRA batch | serving iteration | predicted latency |
| BUBBLETEA | EP bubble | fine-tuning chunk | GPU MoE layer | router top- assignment |
Table 1. Existing colocation systems reason about serving-level headroom, whereas BUBBLETEA identifies slack per GPU inside an MoE layer.
Design of BubbleTea
Observations 1 and 2 establish that bubbles expose usable GPU capacity even after token load balancing. Observation 3 establishes two requirements for using this capacity: (i) admission must be decided per GPU and MoE layer from the router state; and (ii) fine-tuning must expose work short enough to fit inside the resulting bubbles, which determines how much of each bubble can be used (Appendix § 9).
BUBBLETEA addresses these requirements in two phases. During the offline phase, a fine-tuning compiler transforms the LLM already used by the serving engine into a fine-tuning runtime (§ 3.2). Specifically, it derives the LoRA backward pass over the same base weights, makes cross-GPU dependencies explicit, and partitions the remaining computation into fine-tuning chunks, which are independently admissible units of GPU work. Sequences of these chunks and their dependency waits are captured as CUDA graphs so that they can be efficiently launched.
During the online phase, BUBBLETEA uses each request’s current MoE router token assignment to predict the bubble duration on every GPU, then admits the precompiled fine-tuning work only inside those predicted bubbles (§ 3.3). As Figure 4 shows, each GPU runs two processes: a serving worker and a fine-tuning worker. A device-side gate lets the serving worker open and close an admission window inside each GPU’s bubble, while the fine-tuning worker keeps eligible work queued behind the gate. The serving worker closes admission early enough that fine-tuning does not delay the serving-critical GPU, and the fine-tuning worker keeps work ready despite cross-GPU dependencies.
Figure 4. The system architecture of BUBBLETEA. On each GPU, a serving worker and a fine-tuning worker run side by side and share the same copy of the base model weights.
BUBBLETEA by example
Figure 5 shows how BUBBLETEA schedules fine-tuning work during one MoE prefill layer across three GPUs. BUBBLETEA predicts each GPU’s bubble directly from the router’s expert assignment. As soon as a GPU’s experts finish, BUBBLETEA runs queued chunks (derived with our compiler) back to back inside its bubble. Because a running chunk cannot be preempted, BUBBLETEA stops admitting new chunks a margin before the predicted end of the bubble, so that no chunk is still running when the all-reduce starts, which would prolong inference.
Figure 5. Scheduling fine-tuning work with BUBBLETEA on one MoE prefill layer. Uneven expert work gives each GPU a different predicted bubble. GPUs that finish early (GPU 0 and 1) execute queued chunks of work until shortly before the bubble ends; the most-loaded GPU (GPU 2) admits none.
As a result, GPU 0 in Figure 5, which has the largest bubble, executes more chunks than GPU 1, and GPU 2 has no bubble and runs no fine-tuning work at all. Thus, fine-tuning progresses unevenly across GPUs. In practice, we set to 60 μs, as described in Appendix § 11.§ 6.
Offline phase: deriving fine-tuning work
Why off-the-shelf autograd does not suffice. The technical challenge is to execute the LoRA backward pass inside the serving deployment within bubbles. Training frameworks such as PyTorch derive the backward pass automatically with autograd: during the forward pass, they record each operation and keep the intermediate results it needs, and then traverse this record in reverse to compute the gradients. Applying autograd to the served model fails in three ways. (i) The serving forward cannot be differentiated. For speed, the serving engine (vLLM (Kwon et al., [23]) in our case) runs expert computation, attention, and cross-GPU communication with specialized kernels that only produce outputs: they keep no intermediate results and have no backward pass, so autograd cannot run through them. Running autograd on a second, trainable copy of the model instead, as TORCHTUNE does in uncoordinated colocation, duplicates the LLM weights. This takes another 15 GB memory per GPU for QWEN3-30B-A3B that serving could otherwise use for the key-value (KV) cache. (ii) The backward pass is too coarse. A bubble lasts under a millisecond and a kernel cannot be interrupted once started, so fine-tuning may only start work that finishes before the bubble ends. Autograd, however, runs the whole backward pass at once, one operator at a time, and a single expert matrix multiplication over a sample’s tokens can already outlast a bubble. Even when work fits, larger units leave more of each bubble unused (§B). (iii) The backward pass advances in lockstep. Each GPU gets a different bubble, but the backward pass exchanges gradients across GPUs at every layer, and each exchange waits until all GPUs reach it. GPUs with large bubbles therefore sit idle at each exchange until the GPU with the least bubble time catches up, so most of the bubble capacity goes unused.
Existing colocation systems do not close this gap for expert-parallel MoE models. LLMSTATION splits training into suspendable tasklets, but at layer granularity with a host synchronization after each (§ 4.2). Its fusion engine also assumes dense projections, which expert-parallel MoE does not have, and fixed weight placement on GPUs, which EPLB further breaks by moving experts across GPUs.
Deriving a bubble-compatible fine-tuning runtime. BUBBLETEA addresses the three limitations above directly through the following three innovations. (i) Differentiate the serving LLM in place. It replaces inference-only operations with differentiable equivalents over the same serving weights and uses AOTAUTOGRAD to derive the backward graph, preserving the serving LLM’s TP/EP layout and expert placement. (ii) Partition backward into bubble-sized work. BUBBLETEA makes cross-GPU exchanges explicit and partitions the remaining compute with a roofline cost model into chunks of at most , splitting operators such as grouped expert general matrix multiplications (GEMMs) when necessary. Chunks are compiled with INDUCTOR and sequences of gated chunks are captured in CUDA graphs to amortize launch overhead. Cross-GPU exchanges are annotated as communication chunks. (iii) Allow uneven progress across GPUs. Instead of making every GPU wait for all others at each cross-GPU exchange, BUBBLETEA lets a GPU wait only for the data it needs to keep progressing. One-sided mailbox messages make this possible: a GPU writes its data into a peer’s mailbox in GPU memory and continues immediately. A GPU with larger bubbles can thus run ahead, while one with smaller bubbles falls a few chunks behind and catches up in later bubbles. We provide additional details on our compiler passes in §E.
Online phase: Predicting and admitting work
At runtime, BUBBLETEA uses the current MoE routing decision to predict how long the bubble on each GPU will last and then schedules the precompiled fine-tuning work only inside those bubbles. We next explain this process.
Predicting each GPU’s bubble. In an EP deployment, every GPU observes the full top- assignment, so it can directly estimate the expert work assigned to every GPU.
BUBBLETEA introduces a predictor that estimates bubble durations from this assignment (Step 2 in Figure 4). The serving engine computes all experts on a GPU with one grouped matrix-multiplication kernel, which processes each expert’s tokens in tiles: fixed-size blocks of tokens that each take roughly the same time. An expert with routed tokens therefore needs tiles, and the predictor sums these tile counts over the experts placed on each GPU. The predicted bubble duration for GPU is then , where is the execution time of a single tile, so the most-loaded GPU has no bubble. We derive once per model and GPU configuration from a profiling run; porting the predictor to other hardware or GEMM kernels only requires setting the corresponding tile height and re-estimating .
The predictor runs as a small GPU kernel immediately after routing and keeps its result in device memory for the admission gate. Our predictor estimates bubble durations accurately and efficiently. Its median error is , about of the median bubble duration at a layer (§A.4), and its serving-path overhead is at most (§D.11). For prefills with little computation, however, the available bubbles are too small to justify the prediction and gating overhead. For prompts with tokens, enabling bubble filling reduces fine-tuning throughput by – (§D.7). BUBBLETEA therefore disables bubble filling below tokens and leaves those requests to conventional decode and idle-time colocation.
Admitting work inside the predicted bubble. Each GPU has a gate flag in device memory that the serving worker sets and clears. The fine-tuning worker enqueues the precompiled CUDA graphs ahead of time; each fine-tuning chunk waits on this gate before it executes. The wait is a device-side stream wait and consumes no GPU compute. When a GPU finishes its local expert computation and reaches the layer’s gather operation, the serving stream opens its gate (Step 4 in Figure 4), allowing already-queued fine-tuning chunks to execute in the bubble. Opening the gate is a single flag write, so serving never waits for fine-tuning: fine-tuning chunks wait on serving, never the reverse, and run only while the GPU would otherwise sit idle in its bubble.
Keeping work ready. A bubble is only useful if a chunk can start the moment it opens. A chunk can start only when its gate is open and its inputs have arrived, and some inputs come from cross-GPU exchanges with a slower GPU. If the host waited for each such message, the dependent chunk would reach the GPU only after a host round trip, which is a significant fraction of a sub-millisecond bubble. BUBBLETEA instead resolves these dependencies on the device (device-side readiness). The host queues all chunks ahead of time, and the stream waits on the sending GPU’s mailbox flag before each dependent chunk. This wait occupies no SMs, so the chunk starts as soon as both its input and its gate are ready, without host involvement.
We call the ordered sequence of compute and communication chunks for one fine-tuning sample a fine-tuning chain. A chain whose next chunk waits on a lagging GPU still cannot use an open bubble. BUBBLETEA therefore keeps a second, independent fine-tuning chain in flight (paired chains), with its own stream, buffers, and mailboxes. While one fine-tuning chain is blocked on an exchange, the other can run its chunks, and both chains’ gradients are accumulated into one optimizer step. Because two concurrent chains increase contention for memory bandwidth, BUBBLETEA runs them concurrently only when prefill took less than 70% of the last second. Otherwise, it serializes their execution.
Closing the window early. Because an admitted chunk cannot be preempted, one admitted near the end of a bubble may delay the all-reduce and thus increase inference latency. BUBBLETEA therefore stops admitting new chunks a margin before the predicted end of each bubble (early close). If GPU opens its gate at time , it closes the gate at
where is its predicted bubble length. A side-stream kernel closes the gate at , while completion of the serving all-reduce also clears it as a safeguard if the true bubble ends earlier than predicted. No window is opened for bubbles shorter than , so the most-loaded GPU never admits work. We use , chosen to exceed a typical fine-tuning chunk and absorb prediction error (§D.6).
The same precompiled fine-tuning work and device-side gate are reused for conventional decode and idle-time colocation. Only the admission bound differs: during prefill, each GPU’s gate is limited by its router-predicted intra-layer bubble; during decode steps and idle periods, serving leaves capacity unused, so the gate stays open for the whole step. This is the iteration-level colocation that existing systems such as LLMSTATION (He et al., 2025) already perform.
Evaluation
We evaluate BUBBLETEA and baselines on three MoE models and ask the following questions. First, how much fine-tuning throughput does BUBBLETEA obtain and what is the impact on inference latency, compared with an iteration-level colocation system (LLMSTATION) and with uncoordinated colocation (§ 4.2)? Second, where does the throughput in BUBBLETEA come from, and what execution policies are needed to leverage intra-layer idle time safely (§ 4.3, § 4.4)? Third, does BUBBLETEA still provide benefits once the serving engine balances expert load itself (§ 4.5)? Appendix C details our experimental setup, and Appendix D reports more extensive experiments: all models and prompt lengths, fewer GPUs, a second production trace, the uncoordinated colocation streaming multiprocessor (SM)-cap sweep, the choice of design parameters, bubble utilization, and overheads. Appendix B isolates the value of per-GPU planning with a trace-driven replay.
Experimental setup
Hardware and models
We run all experiments on a single compute node equipped with four NVLINK-connected A100-SXM4-80GB GPUs, serving with vLLM in BF16 precision. We evaluate three MoE models: Qwen3-30B-A3B (Owen Team, [34]), which has only routed experts, Qwen1.5-MoE-A2.7B (Team, [39]) and DeepSeek-V2-Lite (DeepSeek-AI, [5]), which additionally have shared experts that process every token and therefore add the same work to every GPU.
Workloads and metrics
For inference, we use arXiv summarization prompts (Cohan et al., [4]) with lengths of 1k–32k tokens and generate 32 output tokens for each prompt. We set vLLM’s maximum number of sequences to 256. For this controlled workload, we determine three load regimes. With a high load, the arrival rate is just below the point at which serving alone starts queueing. Medium and low load use lower rates, which we specify per experiment and workload in the appropriate plots. We also replay two production traces, code and conversation, from the Azure LLM inference trace (Patel et al., [30]), each over a 10-minute window with the trace’s own arrival times. For fine-tuning, we use the ALPACA dataset, consisting of 52 000 instruction-demonstration prompts (Taori et al., [38]).
Baselines
We compare against LLMSTATION (He et al., 2025), a state-of-the-art system for colocating fine-tuning with LLM inference on shared GPUs. LLMSTATION transforms a fine-tuning job into a suspendable pipeline and interleaves it with inference at iteration granularity, batching tuning work alongside decode steps. The system leverages the compute left unused by memory-bound decode iterations, and the idle periods in which no inference work is pending, which makes it the strongest representative of serving-level colocation. FLEXLLM (Oliaro et al., [28]) and DELTASERVE (Chen et al., [2]) instead place fine-tuning work alongside prefill, but only when their per-iteration planner sees headroom in the serving iteration. At our prompt lengths, prefill saturates GPU compute (Agrawal et al., [1]), and an iteration-level view cannot see the per-GPU slack inside MoE layers (§ 2.2). Their planners therefore admit almost no work into prefill. Since their training progress is tied to these admissions, they also make little use of decode slack, which leaves them mostly the idle time that LLMSTATION already exploits. This makes iteration-level colocation by LLMSTATION the more appropriate point of comparison. We also use Uncoordinated colocation as a baseline, which runs a stock LoRA trainer beside VLLM.
Implementation. We implement BUBBLETEA in Python in about 8k lines on vLLM 0.21.1 (v1 engine), PyTorch 2.11, and Triton 3.6. The offline compiler builds on AOTAutograd and Inductor. At runtime, hooks in vLLM’s MoE layer predict and open each window. A separate trainer process per GPU runs the compiled chunks under CUDA MPS and accesses the serving weights through CUDA IPC, so they are not duplicated.
For LLMSTATION, we port its artifact to our vLLM/PyTorch versions while preserving its scheduling policy. Because its released fusion engine assumes dense projections with fixed weight placement, which does not support expert-parallel MoE, we use its separate PyTorch training path under CUDA MPS. We evaluate several MoE expert-execution implementations and report the fastest, using padded batched GEMMs through cuBLAS. LLMSTATION’s latency is normalized to the same serving engine with training disabled. Uncoordinated colocation runs a stock torchtune LoRA trainer beside vLLM under CUDA MPS, with its SM share capped at 40%.
Metrics. For fine-tuning, we report throughput in samples/s. For serving, we report p90 TTFT and TPOT, normalized to serving alone, which capture prefill and per-token decode latency, respectively. Each experiment is repeated three times and we average all results.
End-to-end performance of BUBBLETEA and baselines
We measure end-to-end performance of BUBBLETEA and baselines on the QWEN3-30B-A3B LLM.
Steady-state workloads. Figure 6 shows fine-tuning throughput of BUBBLETEA and baselines, for varying serving loads and 4k–32k prompt lengths. Across all settings, BUBBLETEA provides more fine-tuning throughput than LLMSTATION and trains 2.7–5.0× faster than LLMSTATION, with a median improvement of 2.9× (1.9–6.2× on all models, §D.1). Uncoordinated colocation obtains a throughput of just 0.26–0.43 samples/s, 2.7–12.4× less than BUBBLETEA, and lifting its SM cap does not help: the off-the-shelf trainer is bound by its own per-expert kernel launches. While BUBBLETEA and LLMSTATION both fine-tune during decode and idle time, BUBBLETEA uses fine-grained compiled chunks, which enables more efficient scheduling.

Figure 6. Fine-tuning throughput for BUBBLETEA, LLMSTATION, and uncoordinated colocation (40% cap) on QWEN3-30B-A3B across prompt lengths and serving loads. The labels above bars report BUBBLETEA’s fine-tuning throughput relative to LLMSTATION.
Figure 7 shows the p90 TTFT and p90 TPOT in the same settings as in Figure 6, relative to a serving-only setting. Across all 63 configurations, BUBBLETEA keeps both p90 TTFT and p90 TPOT within 10% of serving alone in 50 configurations (79%), compared with 44 (70%) for LLMSTATION. The median increase of BUBBLETEA is 4% and 6% for p90 TTFT and p90 TPOT, respectively. Uncoordinated colocation meets this 10% bound in only 22 of the 63 configurations: it slows every serving step by a similar amount, which raises p90 latency by 10–20% over serving alone when serving has slack, but at high load pushes serving past its saturation point, so requests queue and p90 TTFT reaches 6–12× that of serving alone. Thus, we conclude that BUBBLETEA across most configurations achieves higher fine-tuning throughput without systematically increasing serving latency.

Figure 7. p90 TTFT (top) and p90 TPOT (bottom) ratio, relative to the corresponding serving-only execution, for BUBBLETEA, LLMSTATION, and uncoordinated colocation (40% cap) on QWEN3-30B-A3B across prompt lengths and serving loads. The dashed line marks serving alone and the dotted line a 10% increase.
Production traces. We additionally replay two production traces with bursty, time-varying arrivals. BUBBLETEA shows 3.2–3.3× higher fine-tuning throughput than LLMSTATION while keeping p90 TTFT and TPOT within 7% of serving alone. Figure 1 in § 1 shows one representative trace. Thus, even on production traces, BUBBLETEA achieves superior fine-tuning throughput with low effect on inference latency.
Where does BUBBLETEA’s throughput come from?
Next, we investigate where the fine-tuning throughput in BUBBLETEA comes from. Figure 8 shows fine-tuning throughput for varying serving loads, and for different colocation mechanisms, while using prompts with length 8k. We first compare decode colocation, which uses LLMSTATION-like scheduling at the granularity of an iteration, with a version of BUBBLETEA, named TEA, that schedules work at finer granularity but does not schedule work in bubbles. This comparison isolates the benefit of BUBBLETEA’s fine-grained scheduling while holding the available serving capacity fixed to conventional decode and idle-time headroom.
Figure 8. Fine-tuning throughput for decode colocation (coarse-grained work scheduling), TEA (fine-grained work scheduling but not within bubbles), and BUBBLETEA on QWEN3-30B-A3B with prompts of 8k length across serving loads.
TEA improves fine-tuning throughput by 36–60% over decode colocation, with improvements more pronounced in the low load regime. This highlights that the fine-grained scheduling mechanism in BUBBLETEA is able to significantly increase fine-tuning throughput compared to conventional approaches. TEA keeps p90 TTFT and TPOT within 10% of serving alone in most configurations.
Next, we compare the fine-tuning throughput of TEA with that of BUBBLETEA that additionally schedules fine-tuning work in bubbles. Figure 8 shows that at a medium and high load, BUBBLETEA achieves 24% and 27% higher throughput, respectively, compared to TEA. This is because with higher load, there are more abundant bubbles during prefill which BUBBLETEA effectively leverages. At low load, where conventional headroom is abundant, leveraging bubbles does not help, and slightly diminishes throughput by 3%. We attribute this decrease to the additional overhead required by the BUBBLETEA scheduling mechanism.
How does BUBBLETEA protect serving latency?
The BUBBLETEA scheduler should not increase inference latency. To better understand how our scheduler protects this latency, we progressively enable subcomponents of our scheduler and report in Figure 9 the fine-tuning throughput (left), p90 TTFT (middle) and p90 TPOT (right) for varying serving loads and scheduler configurations, when using prompts with length 4k. We highlight the results of prompt lengths of 4k because the effects of these different optimizations are most pronounced at this prompt length, making it easiest to distinguish their individual contributions.
Figure 9. Fine-tuning throughput (left), p90 TTFT (middle) and p90 TPOT (right) relative to serving alone, for TEA, unconstrained scheduling, constrained scheduling, and BUBBLETEA on QWEN3-30B-A3B with 4k prompts across serving loads. The dashed line marks serving alone and the dotted line a 10% increase.
Unconstrained scheduling admits fine-tuning work on every GPU during idle time until that GPU has completed the all-reduce operation. This maximizes the amount of training exposed inside prefill, but can compete for compute needed by serving and can run training on the most-loaded GPU that determines the layer’s critical path.
Constrained scheduling adds four mechanisms that protect serving latency. First, waiting for admission becomes a stream-ordered device wait that consumes no GPU compute. Second, the two fine-tuning chains run concurrently only while prefill takes less than 70% of recent serving time and are serialized otherwise (§ 3.3). Third, prefills shorter than 4k tokens are skipped because their bubbles are too small to justify admission. Fourth, each GPU stops admitting new chunks a margin before its bubble ends (early close), so that no running chunk delays the all-reduce (§ 3.3).
Figure 9 shows that at a high load, unconstrained scheduling increases fine-tuning throughput from 0.73 to 2.11 samples/s compared to TEA, but p90 TTFT increases to 5.03× serving alone and p90 TPOT to 1.19×. Fine-tuning that reaches the serving-critical GPU extends the prefill and can queue subsequent requests. Exposing intra-layer capacity is therefore not sufficient; admission must avoid interfering with the critical path. Constrained scheduling reduces p90 TTFT to 1.03× and p90 TPOT to 1.13×, while retaining 1.33 samples/s. Therefore, our scheduling constraints are key to protect serving latency.
Constrained scheduling, however, can still admit fine-tuning work on the most-loaded GPU because it does not distinguish GPUs by their predicted slack. BUBBLETEA uses the router-derived budget to prevent that GPU from admitting work and to stop other GPUs before their predicted bubbles end. Figure 9 shows that this reduces p90 TTFT and TPOT further to 1.01× and 1.04× compared to serving only, respectively, at 1.00 samples/s of fine-tuning throughput.
Composition with load balancing optimizations
We next ask whether BUBBLETEA remains useful when serving already uses an expert load-balancing optimization. We use VLLM’s EPLB, a load-balancing mechanism that periodically replicates and re-places experts across GPUs based on their observed load, so that each GPU receives a more even share of expert work. We enable it on QWEN3-30B-A3B with 16 redundant experts. Across our workloads, EPLB reduces the fraction of prefill spent in bubbles from 11–17% to 8–13%. Because EPLB also changes how much decode and idle-time headroom serving leaves, we break down the contribution of placing fine-tuning in decode and idle time with TEA, and in the bubbles with BUBBLETEA.
Figure 10 shows the results for 8k-token prompts at medium and high load. At high load and across prompt lengths, bubble filling increases fine-tuning throughput over TEA by 12–23% with EPLB, compared to 27–37% without it. This is because EPLB balances expert load averaged over recent steps, while the routing of each individual prefill still deviates from that average, so some GPUs still finish their experts early in every layer. At high load, where decode and idle-time headroom is scarce, these residual bubbles are the main remaining source of fine-tuning capacity. p90 TTFT and TPOT stay within 7% of serving alone.
Figure 10. Fine-tuning throughput (left), p90 TTFT (middle) and p90 TPOT (right) on QWEN3-30B-A3B with prompts of 8k length at medium (2 requests/s) and high (4 requests/s) serving load. Left: each bar shows the throughput of TEA, with the additional throughput of BUBBLETEA from bubble filling stacked on top; labels give the gain from bubble filling. Middle and right: latency of serving alone without and with EPLB, and of BUBBLETEA with EPLB.
At medium load, bubble filling changes fine-tuning throughput by −2 to +0.5%. As expected, decode and idle-time headroom is plentiful at this load, and TEA already uses it. The only workload where latency exceeds 7% is also at medium load: at 16k-token prompts and 1 request/s, p90 TPOT reaches 1.32× serving alone (Appendix D.10). This suggests BUBBLETEA should leave residual bubbles unused when conventional headroom already suffices.
Overall, EPLB and bubble filling are complementary: EPLB shrinks bubbles but cannot remove the imbalance within each prefill, and BUBBLETEA turns what remains into 12–23% more fine-tuning throughput at high load while protecting serving latency. Bubble filling therefore pays off most when serving is busiest, which is exactly when conventional decode and idle-time headroom runs out.
Related Work
MoE routing and serving optimizations. Load imbalance in sparse MoE execution is well known and has been addressed through routing objectives and capacity constraints [13, 24], expert replication and placement, and serving-time mechanisms such as EPLB [8], LPLB [9], and WATERFILL [25]. DEEPEP [7], PPLX-kernels [32], and lower-precision execution further reduce communication and expert compute. BUBBLETEA is complementary: it uses the residual idle capacity that remains after these optimizations are applied.
Inference–fine-tuning colocation and GPU multiplexing. FLEXLLM [28], LLMSTATION (He et al., 2025), HARLI (Xu et al., 2025), and DELTASERVE [2] colocate latency-critical inference with best-effort fine-tuning by exploiting prefill and decode underutilization or serving-level headroom alongside idle periods. BUBBLETEA instead targets per-GPU slack that appears inside expert-parallel prefill, requiring admission from the current router state rather than serving-level signals. More broadly, GPU- and LLM-multiplexing systems use temporal sharing, operator interleaving, spatial partitioning, or model-residency management [42, 43, 37, 26, 27, 12, 31, 41] (Fu et al. 2024; Yu et al. 2025). These systems allocate resources across workloads or models, whereas BUBBLETEA reclaims per-GPU slack inside an ongoing MoE forward pass. CONSERVE [33] co-serves online and offline inference of one model and argues against lending idle cycles to training because of process- or container-switch costs; BUBBLETEA avoids that cost by training on the served weights in place.
Multi-tenant LoRA serving. PUNICA ([3]), S-LORA ([35]), and DLORA (Wu et al., 2024) improve inference across many already-trained LoRA adapters sharing one base model. They optimize adapter serving, whereas BUBBLETEA uses otherwise-idle serving capacity to train adapters and produces ordinary checkpoints compatible with such downstream systems.
Conclusion
Expert-parallel MoE serving leaves GPUs idle in a way dense serving does not: routing skew makes lightly loaded GPUs wait for the most-loaded one at every layer, and these bubbles persist even after applying load balancing optimizations. BUBBLETEA predicts each GPU’s bubble from the router’s decision, admits fine-tuning only inside it, and fills it with compiler-derived chunks with fine-tuning work.
AI Usage Disclosure
We used large language model assistants to help write this paper: to improve wording and spell checking, to generate and edit plots, to check claims and numbers against our data and the cited works, to search the literature, and to verify bibliographic metadata. AI was used to implement several parts of the system, write and run scripts for the experiments and to generate the plots. All AI-assisted content was reviewed and edited by the authors, who take full responsibility for the paper.
Characterization details
Dense and expert-parallel MoE layers
Figure 11 contrasts a dense Transformer layer with an MoE layer on two GPUs. Under TP, a dense layer shards attention heads and feed-forward weights evenly, so every GPU performs the same work on all tokens and reaches each all-reduce at the same time. In the MoE layer, attention is still sharded by TP, and every GPU computes the same routing decision, but EP places whole experts on different GPUs. Each GPU then runs only the tokens routed to its local experts. When the router sends more tokens to one GPU’s experts, the other GPU finishes early and waits at the all-reduce that combines the expert outputs: this wait is the bubble of Figure 2.
Figure 11. Where expert-parallel bubbles come from. Time runs downward within each GPU. (a) A dense layer under TP splits work evenly, so no GPU waits. (b) An MoE layer under TP=EP places different experts on different GPUs; the router sends GPU 0’s experts fewer tokens (squares), so GPU 0 finishes early and idles until GPU 1 reaches the all-reduce that combines the expert outputs. Schematic; sizes are illustrative.
Bubble size across models
Figure 12 shows how the per-layer bubble grows with prefill size for the three MoE models of § 2.2, in the default setup of § 2. From 8k to 128k tokens, the median bubble grows from 0.22 to 6.41 ms/layer on Qwen3-30B-A3B, from 0.28 to 4.34 ms/layer on Qwen1.5-MoE-A2.7B, and from 0.20 to 2.70 ms/layer on DeepSeek-V2-Lite.
Figure 12. Per-layer bubble duration grows with prefill size. Median bubble across three MoE architectures under BF16 execution without load balancing. Error bars show min/max across repetitions.
Lower-precision execution
Lower-precision execution attacks a different part of the problem. FP8 reduces the cost of expert computation, but does not change how the router distributes expert work across GPUs. Nor, in our measured regime, does it reduce expert computation enough to remove the compute saturation during prefill: additional routed work on one GPU still extends its completion time. Figure 13 compares BF16 and FP8 in the load-balancing setup of Figure 3 (DeepSeek-V2-Lite, SGLang, H100, EP = 8). At 128k, moving from BF16 to FP8 changes the bubble from 4.79 to 4.81 ms/layer
Figure 13. Lower precision leaves the bubble unchanged. Median per-layer bubble under BF16 and FP8 (hatched), without load balancing (Base) and with LPLB, in the setup of Figure 3: DeepSeek-V2-Lite, SGLang, H100, EP = 8. Context denotes aggregate tokens across the 8-way EP group.
without load balancing and from 3.53 to 3.50 ms/layer with LPLB, and it changes by at most 0.05 ms/layer at every other context. Faster expert execution therefore does not, by itself, remove the GPU-to-GPU timing differences caused by uneven routed work.
Bubble predictability
We measure how well the router’s token assignment predicts each GPU’s bubble on Qwen3-30B-A3B, served by vLLM on A100 with TP=EP = 4 and no expert load balancing, at five loads (4k tokens at 8 requests/s, 8k at 4 and 2, 12k at 2, and 2k at 1; 400 ARXIV requests each). For every prefill layer and GPU (193920 samples), we compare the measured bubble with two predictors: the deployed router-aware predictor (§ 3.3), and a static profile that assigns every GPU the mean bubble of that layer from the first half of the run. Both are evaluated on the second half. Figure 14 shows that the router-aware predictor reduces the median absolute error from 123 μs to 8.9 μs and the p95 from 452 μs to 45 μs. With EPLB enabled, its median error is 9.5 μs (p95 43 μs). Our probe resolves durations to 4 μs to 8 μs, so the router-aware median error is at the measurement resolution.
Figure 14. Figure 14. The router’s assignment predicts each GPU’s bubble. Absolute error of the router-aware predictor and of a static per-layer profile on Qwen3-30B-A3B, 4×A100, TP=EP = 4, pooled over five serving loads.
Colocation systems and the planner experiment
Colocation runs best-effort work on resources that latency-critical serving leaves unused while preserving its SLOs [28, 2] (He et al. 2025; Xu et al. 2025). Fine-tuning is a natural candidate because it is throughput-oriented and delay-tolerant, and PEFT methods such as LoRA (Hu et al., 2022) train only small adapters while reusing the serving model’s resident weights. Because GPU kernels are difficult to preempt once launched [18], colocation systems must both identify spare capacity and expose work units short enough to fit inside it.
Existing inference–fine-tuning systems already decompose training for this purpose, but make admission decisions at serving boundaries using signals such as batch composition, iteration timing, or engine headroom Table 1. Expert-parallel bubbles have a different granularity: they open and close inside one MoE layer and differ across GPUs. Using them therefore requires both bubble-sized work and a per-GPU admission decision that reveals where slack exists and for how long. The current router assignment provides this missing signal § 3.3.
We first isolate the planning requirement. We replay a Qwen3-30B-A3B serving trace (TP=EP=2, 128k-token prefill) against the same 486 measured fine-tuning chunks (median 567 s) while varying only the information available to the planner. A router-blind planner assigns every GPU a historical layer-level slack estimate; an oracle GPU-agnostic planner instead receives the true mean bubble of the current layer but still assigns the same budget to every GPU; and a router-aware planner estimates slack separately for each GPU from the current routing decision. We compare their bubble utilization, the fraction of available bubble GPU-time occupied by training, at approximately the same 1% overrun rate.
Table 2 shows that router-aware planning recovers 42.7–45.1% of bubble capacity, compared with about 1.3% for either GPU-agnostic planner. Even knowing the exact average slack of the current layer is therefore insufficient: the planner must know which GPU owns that slack.
| Planner | Bubble utilization | Overrun rate |
| Router-blind | 1.28–1.31% | 1.04% |
| Oracle GPU-agnostic | 1.22–1.30% | 1.04% |
| Router-aware | 42.74–45.14% | 0.83–1.04% |
Table 2. Per-GPU planning is required to use bubbles. All planners use the same fine-grained fine-tuning chunks and operate near a 1% overrun rate.
Shorter work units determine how much of this per-GPU capacity can be used. With router-aware planning, reducing the median chunk duration from 4.13 ms (97 chunks) to 567 s (486 chunks) raises bubble utilization from 37.95–38.46% to 42.74–45.14%. Yet the router-blind planner remains near 1–1.5% under both partitions. Thus, finer execution helps only after the planner identifies the right GPU: chunk size determines how much work fits, while router state determines where it can safely run.
Experimental setup details
Experimental setup for Section 2.2
Observation 1. Routing skew and decode bubbles use Qwen3-30B-A3B on vLLM with TP=EP=2 on two A100 GPUs, eager mode, and no load balancing. Skew is the most-loaded GPU’s routed tokens divided by the mean, summed over all MoE layers, from vLLM’s EPLB balancedness logging with no redundant experts and no rebalancing. Each dataset sends 30 requests one at a time, so each request contributes one prefill step and one decode step per generated token. Decode bubbles are measured from GPU kernel timestamps recorded with torch.profiler.
Observation 2. Load-balancing results (Figure 3) use DeepSeek-V2-Lite on SGLang with 8×H100, EP = 8, BF16, and ArXiv prompts. Each point is the median over 1,248 samples: six prefill forwards, 26 MoE layers, and eight GPUs.
Observation 3. The colocation systems and the planner experiment are described in Appendix § 9.
Evaluation setup
This section gives the full configuration summarized in § 4.1.
Hardware and software. All experiments run on one node with four NVIDIA A100-SXM4-80GB GPUs connected over NVLink. Unless stated otherwise, serving uses vLLM with TP=EP=4, BF16 weights, eager execution, chunked prefill and prefix caching disabled, and a 0.85 GPU-memory budget. BubbleTea runs one fine-tuning worker per GPU under CUDA MPS, as described in § 3, with a 50 s chunk budget and an early-close margin of s (§ 3.3).
Models. We evaluate three MoE architectures with different expert organizations. Qwen3-30B-A3B (48 layers, 128 experts, top-8) uses only routed experts, while Qwen1.5-MoE-A2.7B (24 layers, 60 routed experts, top-4) and DeepSeek-V2-Lite (27 layers, 64 routed experts, top-6) also contain shared experts. Qwen1.5-MoE’s configuration declares an 8k context; for longer prompts we extend its rotary cache to the served length identically for every system.
Serving workloads and metrics. Our controlled workloads use arXiv summarization prompts truncated to an exact input length between 1k and 32k tokens, each generating 32 output tokens. Short outputs make the workload prefill-dominated, where bubbles arise, and resemble shortform tasks such as extraction or classification; § 4.2 also replays production traces.
For every model and prompt length, we choose three arrival rates: high is just below the point at which serving alone begins sustained queueing, while medium and low are fixed fractions of that rate. Each steady-state run serves 400 requests and is repeated thrice unless stated otherwise. Latency statistics discard the first and last 10% of requests and pool the remaining requests across repetitions. We report fine-tuning throughput as completed samples/s while requests are being served, together with tail TTFT and TPOT relative to the corresponding serving-only execution.
Fine-tuning workload. Every system trains the same job: LoRA rank 16 on the attention projections, using alpacacleaned samples padded to 128 tokens and AdamW with learning rate . We adapt for Qwen3-30B-A3B and Qwen1.5-MoE-A2.7B, and for DeepSeek-V2-Lite.
Baselines. We compare four systems.
Serving only provides the inference-latency reference.
• Uncoordinated colocation runs a stock torchtune LoRA trainer using FSDP alongside vLLM under CUDA MPS. Because its interference depends on the amount of GPU compute available to the trainer, we run it with the trainer’s SM share capped at 6% and 40% and uncapped. The main text shows the 40% cap; Appendix § 11.§ 4 compares all three.
• LLMStation (He et al., 2025) represents iteration-level colocation through suspendable tasklets. We update its implementation to the current vLLM (v1 engine) and PyTorch versions, and extend its training forward to Qwen3-MoE with shared serving weights. Its released fusion engine assumes dense projections with fixed weight placement, which does not hold for expert-parallel MoE, especially with EPLB. We therefore use its separate PyTorch training path alongside inference under CUDA MPS with EPLB disabled. To obtain a representative baseline, we evaluated several expert-execution paths including a naive per-expert forward, reuse of BUBBLETEA’s compiler-generated chunks and a packed-GEMM implementation. Executing padded batched GEMMs through cuBLAS gave the best LLMStation performance. This path does not use BUBBLETEA’s compiler generated chunks or A100-specific grouped-GEMM kernel. LLMStation’s latency ratios are measured relative to the same engine with training disabled, so differences between serving engines do not count against either system.
• BUBBLETEA is the full system. It combines conventional decode/idle-time colocation with router-aware intra-layer bubble filling. Unless stated otherwise, it uses all mechanisms of § 3, including device-side readiness, paired chains, and early close.
We do not compare against colocation systems that use compute headroom during prefill: prefill saturates the GPU at a few hundred tokens (Agarwal et al., 2024), so at our prompt lengths that headroom is gone, and the remaining serving-level opportunities, idle and decode time, are the ones LLMStation targets.
Additional experiments
This section extends the evaluation of § 4 and follows the order of its subsections. For the end-to-end comparison (§ 4.2), §D.1 reports all three models and 1k–32k prompts, §D.2 repeats the Qwen3-30B-A3B comparison on two GPUs, §D.3 describes both production traces and their full results, and §D.4 sweeps the SM cap of uncoordinated colocation. For the sources of throughput and the latency protections (§ 4.3 and § 4.4), §D.5 repeats both breakdowns at other prompt lengths, §D.6 selects the early-close margin, §D.7 tests the 4k-token bubble-filling threshold, §D.8 measures the decode and idle-time headroom that serving leaves at each load, and §D.9 shows where the bubble capacity that BUBBLETEA leaves unused goes. §D.10 extends the composition with EPLB (§ 4.5) to all workloads. Finally, §D.11 isolates the overhead of the BUBBLETEA runtime and verifies that chunked execution preserves training behavior.
End-to-end results for all models
§ 4.2 compares BUBBLETEA with the baselines on Qwen3-30B-A3B at 4k–32k prompts. Figure 15 and Figure 16 extend Figure 6 and Figure 7 to Qwen1.5-MoE-A2.7B and DeepSeek-V2-Lite, and to 1k, 2k, 8k, and 24k prompts. Both models follow the same trends as Qwen3-30B-A3B: across all three models and 1k–32k prompts, BUBBLETEA trains 1.9–6.2× faster than LLMSTATION (median 2.8×). At 1k and 2k tokens, prefills are too short for BUBBLETEA to open useful windows, so its gains there come from its runtime mechanisms on decode and idle time (§ 4.3).
Figure 15. Fine-tuning throughput while serving, all models. Samples/s for BUBBLETEA, LLMStation, and uncoordinated colocation (40% cap) across three MoE models, prompt lengths, and serving loads. Labels report BUBBLETEA’s throughput relative to LLMStation.
Figure 16. Serving latency under colocation, all models. p90 TTFT (top) and p90 TPOT (bottom), relative to the corresponding serving-only execution. The dashed line marks serving alone and the dotted line a 10% increase. Bars above are clipped, with their value printed beside them. Uncoordinated colocation uses the 40% cap.
BUBBLETEA keeps both p90 TTFT and TPOT within 10% of serving alone in 50 of the 63 workloads. Of the other 13, eight are the main exception, TPOT on Qwen1.5-MoE-A2.7B at long prompts: from 16k to 32k tokens, BUBBLETEA’s p90 TPOT reaches 1.14–1.47× serving alone in every workload but 16k at medium load (1.09×), highest at 32k and medium load, while TTFT stays within 6%. Its short decode iterations overlap with the fine-tuning worker’s cross-GPU exchanges and increase TPOT. Three more exceed the bound by at most three points: Qwen1.5-MoE-A2.7B at 8k and medium load (p90 TTFT 1.13×, TPOT 1.12×, where LLMStation’s TPOT is 1.15×) and high load (TPOT 1.10×), and Qwen3-30B-A3B at 2k and low load (TPOT 1.11×).
The last two are TTFT at 1k-token prompts and high load: Qwen1.5-MoE-A2.7B at 32 requests/s reaches 3.61× serving alone and Qwen3-30B-A3B at 16 requests/s 1.13×, while TPOT stays within 7%. These prompts are below the 4k-token threshold, so BUBBLETEA trains only in decode and idle time, and the server is already near saturation: on Qwen1.5-MoE-A2.7B each serving step takes about 29 ms, so 32 requests/s keep the GPU about 93% busy, and serving alone completes only 29 requests/s. Decode and idle-time training leaves the median step unchanged but lengthens the slowest ones (p90 from 30 to 34 ms in one run), and at this utilization that builds a request queue. This is a limitation of the current policy: BUBBLETEA does not pause decode-and idle-time training while requests are queued, and we leave such a queue-aware cutoff to future work.
Smaller expert-parallel degree
§ 4.2 uses four GPUs; we test whether BUBBLETEA’s advantage holds with fewer expert-parallel GPUs, where each GPU hosts more experts. Table 3 repeats the Qwen3-30B-A3B comparison with half the GPUs, at TP=EP=2. High load is half the TP=EP=4 high load of each prompt length, and medium load about a quarter of it. BUBBLETEA trains 3.3–3.9× faster than LLMStation and keeps both p90 TTFT and TPOT within 10% of serving alone in every workload; the highest is p90 TPOT at 8k tokens and high load (1.10×, against 1.11× for LLMStation).
| Len. | Load | req/s | FT LLMS | FT BT | FT Ratio | LLMS TTFT | LLMS TPOT | BT TTFT | BT TPOT |
| 4k | med. | 2 | 1.58 | 5.60 | 3.5× | 1.05 | 1.08 | 1.04 | 1.05 |
| 4k | high | 4 | 0.95 | 3.46 | 3.6× | 1.02 | 1.01 | 1.02 | 1.00 |
| 8k | med. | 1 | 1.52 | 5.22 | 3.4× | 1.04 | 1.05 | 1.02 | 1.01 |
| 8k | high | 2 | 0.91 | 3.51 | 3.9× | 1.03 | 1.11 | 1.02 | 1.10 |
| 16k | med. | 0.5 | 1.46 | 4.86 | 3.3× | 1.02 | 1.02 | 1.02 | 1.04 |
| 16k | high | 0.875 | 0.90 | 3.18 | 3.5× | 1.02 | 1.01 | 1.01 | 1.01 |
Table 3. TP=EP=2. Qwen3-30B-A3B, two repetitions. LLMS: LLMStation; BT: BUBBLETEA. FT: fine-tuning samples/s; TTFT and TPOT: p90 relative to serving alone.
Production traces
§ 4.2 summarizes BUBBLETEA’s behavior under bursty production arrivals, and Figure 1 shows one trace; here we describe both traces and report full results. Constant-rate workloads let us control prompt length and serving load, but production arrivals are bursty. We therefore replay two traces from the Azure LLM inference trace ([30]), code and conversation, on Qwen3-30B-A3B, each over a 10-minute window (minutes 18–28 and 17–27) at its original arrival rate. The traces give each request’s arrival time and its input and output lengths; we fill prompts with arXiv text of that length. The code trace has 2,183 requests with long prompts and short outputs (median input 1,506 tokens, p90 5,823; median output 13, p90 52) and very bursty arrivals: 3.9 requests/s on average but 0–14.6 per 10 s interval, with a coefficient of variation of inter-arrival times of 10.3. The conversation trace has 3,716 requests at a steadier 6.2 requests/s (3.9–8.2 per interval; coefficient of variation 1.0), with shorter prompts (median 1,024, p90 4,080) and longer outputs (median 111, p90 421). Prompts
of at least 4k tokens carry 47% of prompt tokens in the code trace and 10% in the conversation trace.
Figure 17 shows that BUBBLETEA keeps its advantage under bursty arrivals: it trains at 5.9 and 5.0 samples/s across the two traces, compared with 1.8 and 1.5 samples/s for LLMStation, while p90 TTFT and TPOT remain within 7% of serving alone.
Figure 17. Trace-driven serving. Request arrivals (10 s bins, dashed: mean rate), fine-tuning throughput, and p90 serving latency relative to serving alone under two production traces. In the latency panels, the dashed line marks serving alone and the dotted line a 10% increase.
Uncoordinated colocation sweep
§ 4.2 runs uncoordinated colocation with a 40% SM cap and notes that lifting the cap does not help; here we justify that choice by sweeping the cap. Uncoordinated colocation’s interference depends on how much GPU compute its trainer may use. Figure 18 and Figure 19 compare three settings of the trainer’s SM cap under CUDA MPS: 6%, 40%, and uncapped. Raising the cap from 6% to 40% raises the median fine-tuning throughput across the 63 workloads from 0.80 to 1.05 samples/s, but the number of workloads that keep both p90 TTFT and p90 TPOT within 10% of serving alone falls from 51 of 63 to 22 of 63. Without a cap, median throughput does not rise further (1.00 samples/s), and only 6 of 63 workloads stay within that bound. In the median workload, BUBBLETEA trains 5.9× faster than uncoordinated colocation at the 6% cap and 4.4× faster than at the 40% cap. In a few high-load workloads uncoordinated colocation trains faster than BUBBLETEA, almost always by inflating p90 TTFT by 5–31×. The exception is DeepSeek-V2-Lite at 1k tokens and high load, where uncoordinated colocation at the 40% cap or uncapped trains about 1.2× faster than BUBBLETEA while staying within 3% of serving alone.
Figure 18. Uncoordinated colocation sweep: fine-tuning throughput. Samples/s of the torchtune trainer colocated with vLLM, with the trainer’s SM share capped at 6%, at 40%, or uncapped.
Figure 19. Figure 19. Uncoordinated colocation sweep: serving latency. p90 TTFT (top) and p90 TPOT (bottom) relative to serving alone, for the three SM caps of Figure 18. The dashed line marks serving alone and the dotted line a 10% increase. Bars above are clipped, with their value printed beside them.
Ablation details
§ 4.3 and § 4.4 break down BUBBLETEA’s throughput and its latency protections at 8k and 4k prompts; here we check that both hold at other prompt lengths. Figure 21 extends Figure 8 to 4k and 16k prompts. The pattern is the same at every length: BUBBLETEA (without leveraging bubbles) raises throughput over decode colocation by 36–60% at low and medium load, bubble filling changes throughput by −7 to −2% at low load and adds 13–37% at medium and high load. At 16k and high load all three configurations train at most 0.53 samples/s, so the relative gain of BUBBLETEA (without leveraging bubbles) there (+159%) is large but small in absolute terms.
Figure 21. Sources of BUBBLETEA’s fine-tuning throughput, 4k–16k prompts. Configurations as in Figure 8.
Figure 20 extends Figure 9 to 8k and 16k prompts. Un-throttled bubbles hurt serving only near saturation, and most at 16k prompts and 2 requests/s, where p90 TTFT reaches 9.15× serving alone. At 8k and 4 requests/s the server keeps enough headroom that the same configuration keeps p90 TTFT within 1.08×, but its p90 TPOT reaches 1.22×. At 16k and high load, BUBBLETEA keeps TTFT at 1.01× but its p90 TPOT is 1.21×.
Figure 20. Mechanisms that protect serving latency, 4k–16k prompts. Rows: fine-tuning throughput, p90 TTFT and p90 TPOT relative to serving alone. Configurations as in Figure 9.
Sensitivity to the early-close margin
In § 4.4, the early close is one of the protections that keep training off the serving critical path; this subsection selects its margin. BUBBLETEA closes each admission window a margin before its predicted end ( § 3.3 ). We select with the sweep in Table 4, which varies from 0 to 120 s on Qwen3-30B-A3B at three high-load workloads, where bubble filling matters most ( § 4.3 ); all other settings are those of BUBBLETEA. The mechanism sets a floor on : the margin should cover one 50 chunk, so that a chunk admitted just before the close ends before the predicted end, and the predictor’s p95 error of 45 (§ 3.3). The sweep separates two regimes. Margins above 60 cost 10–19% of fine-tuning throughput without lowering serving latency: p90 TTFT stays within 6% of serving alone at every margin, and p90 TPOT does not fall (at 16k it rises to 1.22–1.25). Between 0 and 60 , throughput and latency differ by no more than run-to-run variation. Closing the window at the predicted end is what protects serving (§ 4.4); the margin absorbs prediction error. We therefore pick , the largest margin that costs no measurable throughput: it clears the floor and gives the most headroom against mispredictions at no cost.
| () | 4k, 8 req/s FT | 4k, 8 req/s TPOT | 8k, 4 req/s FT | 8k, 4 req/s TPOT | 16k, 2 req/s FT | 16k, 2 req/s TPOT |
| 0 | 0.96 | 1.05 | 1.59 | 1.10 | 0.53 | 1.09 |
| 30 | 0.98 | 1.05 | 1.55 | 1.11 | 0.51 | 1.13 |
| 60 | 0.97 | 1.05 | 1.63 | 1.08 | 0.51 | 1.08 |
| 90 | 0.86 | 1.06 | 1.39 | 1.13 | 0.43 | 1.22 |
| 120 | 0.83 | 1.05 | 1.47 | 1.09 | 0.42 | 1.25 |
Table 4. Early-close margin sweep. Qwen3-30B-A3B, TP=EP=4. FT: fine-tuning samples/s; TPOT: p90 TPOT relative to serving alone. Two repetitions at 4k, one at 8k and 16k; at 4k and 60 , one run with a queueing episode (p90 TTFT 559 ms against 192 ms) is excluded. Bold: the margin selected for BUBBLETEA, 60 .
Bubble-filling threshold
Throttled bubbles and BUBBLETEA skip prefills shorter than 4k tokens (§ 4.4, § 3.3); here we test whether this threshold is needed. To do so, we rerun Qwen3-30B-A3B at 2k-token prompts, where it never fires, with bubble filling enabled in every prefill step. At low load (3 requests/s), removing the threshold lowers fine-tuning throughput from 5.11 to 4.69 samples/s (), and p90 TTFT and TPOT rise from and serving alone to and . At high load (12 requests/s), throughput falls from 1.57 to 1.39 samples/s (), and p90 TTFT and TPOT rise from and to and . At 2k tokens the bubbles are too short to amortize prediction and gating, so filling them only adds contention. Both settings use two repetitions, with latency relative to serving alone in the same repetition.
Spare capacity while serving
§ 4.3 argues that bubbles matter more as serving load consumes conventional headroom; here we measure that headroom directly. Figure 22 decomposes the GPU capacity a colocated job could use while vLLM serves alone, for every model, prompt length, and load from a quarter of the high load up to the high load. Idle time is wall time with no forward step running. Decode headroom is the fraction of the GPU a co-running process can use inside decode steps without measurably slowing them, measured with a throttled GEMM probe under CUDA MPS. Prefill bubbles are the per-GPU waits before each layer’s all-reduce, inside which BUBBLETEA opens windows for fine-tuning. For prompts up to 16k tokens, decode headroom accounts for 35–74% of a GPU at the lightest load but only 3–11% at high load, while bubble capacity grows with prefill activity.
Figure 22. Spare GPU capacity while serving alone, as a share of one GPU: idle time (whole GPU free), spare compute inside decode steps that a co-running process can use without slowing them, and prefill bubbles. vLLM alone, TP=EP= 4, 400 requests per run; x labels give the request rate, from the high load down to a quarter of it for each prompt length.
Bubble utilization
§ 4.4 notes that BUBBLETEA’s protections leave some bubble capacity unused; here we break down where that capacity goes. We define bubble utilization as the fraction of measured non-straggler bubble GPU-time during which fine-tuning computation executes. Unlike the trace-driven experiment in Appendix § 9, which isolates what different planners could admit into recorded bubbles, this measurement accounts for the complete BUBBLETEA runtime: prediction, admission, training dependencies, chunk readiness, and early close.
At 8k tokens and 2 requests/s, BUBBLETEA executes useful fine-tuning work during 36% of the available bubble time. Approximately 50% passes with an admission window open but no fine-tuning chunk executing: the fine-tuning worker is waiting on one of its own cross-GPU dependencies or has no ready chunk. The remaining roughly 15% is deliberately left unused because BUBBLETEA closes each window before its predicted end.
Open-but-idle time arises because BUBBLETEA lets GPUs progress unevenly (§ 3.2) and does not try to equalize them. Equalizing would rarely help: on Qwen3-30B-A3B at 16k tokens, the most-loaded GPU changes between consecutive layers 70% of the time, and each of the four GPUs is the most-loaded in 14–39% of layers, so a GPU that falls behind in one layer often has a bubble to catch up in the next, as well as during decode and idle time, which open on every GPU. A GPU that runs ahead, however, can reach one of the training step’s cross-GPU exchanges before a lagging GPU has produced its input, and its window then passes with nothing to run.
These losses have different causes. Open-but-idle time is a readiness limitation: serving has exposed usable capacity, but training cannot always supply work at the same fine granularity. Device-side readiness and paired chains target this loss (§ 3.3), but they do not remove the dependencies between fine-tuning chunks. Early-close time, in contrast, is capacity that BUBBLETEA gives up on purpose to avoid extending the serving critical path.
Collocated training could also create the capacity it later uses. Unthrottled bubbles increases total bubble time by 17% relative to serving alone; training delays the most-loaded GPU, so its peers wait longer. BUBBLETEA increases it by only 4% (2% with one fine-tuning chain), because training is admitted primarily on GPUs that would otherwise be waiting. The windows also behave conservatively: 98% close before the serving dependency clears, with a median 57 s of remaining slack, and the straggling GPU opens a window in only 4% of layers.
Composition with EPLB, all workloads
§ 4.5 shows that BUBBLETEA composes with expert load balancing at 8k prompts; here we report all six workloads. Figure 24 extends Figure 10 to 4k, 8k, and 16k prompts at medium and high load, with latency relative to serving alone with EPLB. The only workload outside 7% of that baseline is 16k prompts at 1 request/s, where p90 TPOT reaches 1.32× (p90 TTFT 1.06×); there, bubble filling adds no throughput (−2%), so the residual bubbles are better left unused. The tallest latency bar, 1.34× p90 TPOT at 16k prompts and 2 requests/s, is serving alone without EPLB: it reflects EPLB’s own benefit to serving, not a cost of BUBBLETEA, which stays at 1.06× there.
Figure 24. BUBBLETEA composes with expert load balancing, all workloads. Top: fine-tuning throughput without and with EPLB (hatched), split into BUBBLETEA (without leveraging bubbles) and what bubble filling adds; labels give the change from filling, and a dashed outline marks a loss. Middle and bottom: p90 TTFT and p90 TPOT of serving alone and BUBBLETEA, relative to serving alone with EPLB (dashed line; dotted: 10% increase).
Overheads and correctness
§4 measures BUBBLETEA with fine-tuning admitted; here we isolate the cost of the runtime itself, independent of the fine-tuning work it admits, and verify that its chunked execution preserves training behavior.
Scheduling overhead. BUBBLETEA places prediction and admission logic on the serving path, so even a fine-tuning worker that never executes could in principle perturb inference. To isolate this overhead, we run the complete runtime with the fine-tuning worker resident but never admit a fine-tuning chunk. Each prefill MoE layer still predicts its bubble, opens an admission window, arms early close, and closes the window.
Together, these hooks consume approximately of GPU time per layer. Their host-side work takes approximately per layer, but overlaps GPU execution; across all three models, measured prefill-step duration changes by at most 0.5%. Correspondingly, p50 and p90 TTFT and TPOT remain within 2.5% of serving alone in all but two configurations. On Qwen3-30B-A3B at medium load, p90 TPOT rises by 6.8% while p50 TPOT and both TTFT percentiles stay within 1%. The other exception is DeepSeek-V2-Lite at its highest tested load, where the server is busy approximately 95% of the time and p50 and p90 TTFT increase by 6.4% and 7.0%. Reducing the load from 7 to 6 requests/s removes this effect, indicating that even sub-percent runtime overhead can amplify once the server is already at the edge of queueing. Figure 25 reports the per-model results and the per-layer hook cost.
Figure 25. Scheduling overhead with an idle fine-tuning worker. Left three panels: p50 and p90 TTFT and TPOT with BUBBLETEA’s hooks enabled but no training admitted, relative to serving alone, at 8k tokens and three loads per model (400 requests per run, pooled over runs). Right: per-layer cost of the prediction and admission hooks on one A100; solid bars are GPU time on the serving stream, hatched bars host time.
Memory overhead. Keeping a separate fine-tuning worker resident reduces the memory available to the serving KV cache by 4.3–5.8 GiB per GPU, or 5–7% of an 80 GB A100, compared with vLLM alone. The fine-tuning worker itself holds 3.0–4.6 GiB of this (Figure 27); the rest is buffers, CUDA contexts, and exchange mailboxes outside its allocator. The LoRA parameters, gradients, and optimizer state take at most 0.15 GiB; most of the fine-tuning worker’s memory is the activation pool backing the compiled training graphs and its separate CUDA context. For comparison, LLMStation reduces the KV cache by 1.5–2.4 GiB per GPU because its trainer executes inside the serving process and can share the CUDA context and allocator. Figure 26 compares these reductions, including the additional cost when EPLB is enabled.
Figure 26. Memory overhead and training correctness. Left: GPU memory taken per GPU, as a share of 80 GB, by LLMStation (LLMS), BUBBLETEA (BT), EPLB’s redundant experts, and both together (Both). vLLM has no EPLB for Qwen1.5-MoE. Right: 25-step moving-average training loss on Qwen3-30B-A3B for BUBBLETEA and a standalone HF+PEFT trainer with the same data, LoRA configuration, and learning rate.
Figure 27. Fine-tuning worker memory per GPU. Breakdown of the resident BUBBLETEA fine-tuning worker’s footprint on Qwen3-30B-A3B and DeepSeek-V2-Lite.
Training correctness and runtime reliability. Chunking a training step across short windows and two fine-tuning chains must not change the optimization being performed.
We therefore train Qwen3-30B-A3B with BUBBLETEA and with a standalone HF+PEFT trainer using the same data, LoRA configuration, and learning rate. Across 476 steps, the 25-step moving-average losses differ by 1.2% on average and at most 4.5%. We additionally observed 239,218 device-side exchanges across 320 BUBBLETEA runs with no mismatch, fallback, or timeout (Figure 26, right). Thus, the fine-grained execution required to exploit short bubbles preserves the expected training behavior while operating reliably across long-running experiments.
Compiler Passes
Section 3.2 summarizes how BUBBLETEA derives bubble-sized training work from the serving model; this appendix lists the five passes. Operation substitution replaces each inference-only operation with a differentiable one over the same weights: paged attention becomes causal SDPA over the full sequence, MLA expands its latent keys and values, fused MoE becomes grouped GEMM, and collectives get explicit gradient rules. Scaling, positional encoding, routing, and expert layout are kept, and LoRA deltas are inserted at the adapted projections. Graph capture traces this forward with AOTAutograd to obtain the backward graph, with the frozen base weights as graph inputs so that the fine-tuning worker runs on the live serving weights. Exchange annotation marks the partial gradients each GPU produces for cross-GPU summation, with the ranks that take part. Partitioning separates forward from backward and compute from communication, then uses a roofline cost model to cut compute into chunks of at most on the target GPU. Cuts need not follow layer or operator boundaries: the MoE lowering splits the token–expert pairs of one grouped GEMM into several smaller ones. Lowering compiles compute chunks with Inductor, captures sequences of gated chunks in CUDA graphs to avoid per-chunk launch cost, and hands communication to BUBBLETEA’s communicator.
Uneven progress is safe. Cross-GPU exchanges use one-sided mailbox messages tagged by exchange and round: a producer publishes without waiting, while consumers wait for the corresponding messages from their peers. Because every GPU encounters exchanges in the same order and publishing never blocks, uneven progress cannot create a cyclic wait; a faster GPU can only wait until a slower peer reaches the same dependency. Fine-tuning chains (§ 3.3) synchronize when their gradients are combined for the optimizer step, which bounds how far GPUs drift apart and makes mailbox reuse safe. Uneven progress may still leave some bubbles unused when a required peer result is not yet ready, but this affects utilization rather than correctness (Appendix § 11.9).
New models built from supported operations need no new code; the same passes support Qwen1.5-MoE, Qwen3-MoE, DeepSeek-V2-Lite, and OLMoE. A new operation needs a local lowering and a gradient rule, which can be tested on its own (Appendix § 11.11).
Figure 23. Using expert-parallel bubbles. Top: accounting of available non-straggler bubble time at 8k tokens and 2 requests/s. Bottom: total bubble time created while different colocated execution policies run, normalized to serving alone, for decode colocation (Coloc.), Unthrottled bubbles (Unthr.), Throttled bubbles (Thrott.), BUBBLETEA (BT), and BUBBLETEA with one fine-tuning chain (BT, 1 ch.); § 4.4 defines the configurations. The accounting categories add up to 104%: 46 ms (3.8% of bubble time) fall into two categories.
References
- [1]Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., Tumanov, A., and Ramjee, R. Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 117–134, 2024.arxiv.org/abs/2403.02310
- [2]Chen, J., She, J., Yuan, Y., Ghosh, R., Gupta, K., Ho, Q., Liu, X., and Balmau, O. DeltaServe: Host-agnostic co-serving of inference and fine-tuning for LLMs, arXiv:2607.28848, 2026.arxiv.org/abs/2607.28848
- [3]Chen, L., Ye, Z., Wu, Y., Zhuo, D., Ceze, L., and Krishnamurthy, A. Punica: Multi-tenant LoRA serving. Proceedings of Machine Learning and Systems, 6:1–13, 2024.arxiv.org/abs/2310.18547
- [4]Cohan, A., Dernoncourt, F., Kim, D. S., Bui, T., Kim, S., Chang, W., and Goharian, N. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 615–621, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-2097.DOI
- [5]DeepSeek-AI. DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model, arXiv:2405.04434, 2024a.arxiv.org/abs/2405.04434
- [6]DeepSeek-AI. DeepSeek-V3 technical report, arXiv:2412.19437, 2024b.arxiv.org/abs/2412.19437
- [7]DeepSeek-AI. DeepEP: an efficient expert-parallel communication library. https://github.com/deepseek-ai/DeepEP, 2025a. GitHub repository.
- [8]DeepSeek-AI. EPLB: Expert parallelism load balancer. https://github.com/deepseek-ai/EPLB, 2025b. GitHub repository.
- [9]DeepSeek-AI. LPLB: an expert-parallel load balancer for MoE models based on linear programming. https://github.com/deepseek-ai/LPLB, 2025c. GitHub repository, early research stage.
- [10]DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence, arXiv:2606.19348, 2026.arxiv.org/abs/2606.19348
- [12]Duan, J., Lu, R., Duanmu, H., Li, X., Zhang, X., Lin, D., Stoica, I., and Zhang, H. MuxServe: Flexible spatial-temporal multiplexing for multiple LLM serving. In Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research, pp. 11905–11917, 2024.arxiv.org/abs/2404.02015
- [13]Fedus, W., Zoph, B., and Shazeer, N. Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022.arxiv.org/abs/2101.03961
- [15]Gale, T., Narayanan, D., Young, C., and Zaharia, M. MegaBlocks: Efficient sparse training with mixture-of-experts. In Proceedings of Machine Learning and Systems (MLSys), volume 5, 2023.arxiv.org/abs/2211.15841
- [17]Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, arXiv:2507.06261, 2025.
- [18]Han, M., Zhang, H., Chen, R., and Chen, H. Microsecond-scale preemption for concurrent GPU-accelerated DNN inferences. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 539–558, 2022.
- [22]Kimi Team. Kimi K2: Open agentic intelligence, arXiv:2507.20534, 2025.
- [23]Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), pp. 611–626, 2023. doi: 10.1145/3600006.3613165.DOI
- [24]Lepikhin, D., Lee, H., Xu, Y., Chen, D., Fırat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations (ICLR), 2021.
- [25]LMSYS Org. Improving DeepEP MoE load balance in SGLang with waterfill and LPLB. https://www.lmsys.org/blog/2026-06-26-waterfill-lplb/, 2026. Blog post; built on two SGLang PRs (shared-expert fusion under EP, Waterfill dispatch balancing).
- [26]NVIDIA. NVIDIA multi-process service. https://docs.nvidia.com/deploy/mps/index.html, 2013.
- [27]NVIDIA. NVIDIA multi-instance GPU. https://www.nvidia.com/en-us/technologies/multi-instance-gpu/, 2020.
- [28]Oliaro, G., Miao, X., Cheng, X., Kada, V., Wu, M., Gao, R., Huang, Y., Delacourt, R., Yang, A., Wang, Y., Unger, C., and Jia, Z. FlexLLM: Token-level co-serving of LLM inference and finetuning with SLO guarantees. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26), pp. 1359–1379, Renton, WA, 2026. USENIX Association.
- [29]OpenAI. gpt-oss-120b & gpt-oss-20b model card, arXiv:2508.10925, 2025.
- [30]Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., and Bianchini, R. Splitwise: Efficient generative LLM inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pp. 118–132, 2024. doi: 10.1109/ISCA59077.2024.00019.DOI
- [31]Patke, A., Reddy, D., Jha, S., Qiu, H., Pinto, C., Narayanaswami, C., Kalbarczyk, Z., and Iyer, R. Queue management for SLO-oriented large language model serving. In Proceedings of the 2024 ACM Symposium on Cloud Computing, pp. 18–35, 2024. doi: 10.1145/3698038.3698523.DOI
- [32]Perplexity AI. pplx-kernels: Perplexity GPU kernels for MoE communication. https://github.com/perplexityai/pplx-kernels, 2025. GitHub repository; device-initiated all-to-all via NVSHMEM/IBGDA, integrated into vLLM and SGLang as an AllToAll backend.
- [33]Qiao, Y., Anzai, S., Yu, S., Ma, H., Yang, S., Wang, Y., Kim, M., Wu, Y., Zhou, Y., Xing, J., Gonzalez, J. E., Stoica, I., and Xu, H. ConServe: Fine-grained GPU harvesting for LLM online and offline co-serving. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.
- [34]Qwen Team. Qwen3 technical report, arXiv:2505.09388, 2025.
- [35]Sheng, Y., Cao, S., Li, D., Hooper, C., Lee, N., Yang, S., Chou, C., Zhu, B., Zheng, L., Keutzer, K., Gonzalez, J., and Stoica, I. S-LoRA: Scalable serving of thousands of LoRA adapters. Proceedings of Machine Learning and Systems, 6:296–311, 2024.DOI
- [36]Smith, T. M. [RFC]: Data parallel attention and expert parallel MoEs. https://github.com/vllm-project/vllm/issues/16037, 2025. vLLM project GitHub issue #16037, accessed 2026-09-01.
- [37]Strati, F., Ma, X., and Klimovic, A. Orion: Interference-aware, fine-grained GPU sharing for ML applications. In Proceedings of the Nineteenth European Conference on Computer Systems (EuroSys), pp. 1075–1092, 2024. doi: 10.1145/3627703.3629578.DOI
- [38]Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023.
- [39]Team, Q. Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters. https://huggingface.co/Qwen/Qwen1.5-MoE-A2.7B, 2024. Accessed: 2026-10-29.
- [41]Xiang, Y., Li, X., Qian, K., Yang, Y., Zhu, D., Yu, W., Zhai, E., Liu, X., Jin, X., and Zhou, J. Ageaon: Effective GPU pooling for concurrent LLM serving on the market. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP), pp. 1030–1045, 2025. doi: 10.1145/3731569.3764815.
- [42]Xiao, W., Bhardwaj, R., Ramjee, R., Sivathanu, M., Kwatra, N., Han, Z., Patel, P., Peng, X., Zhao, H., Zhang, Q., Yang, F., and Zhou, L. Gandiva: Introspective cluster scheduling for deep learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pp. 595–610, 2018.
- [43]Xiao, W., Ren, S., Li, Y., Zhang, Y., Hou, P., Li, Z., Feng, Y., Lin, W., and Jia, Y. AntMan: Dynamic scaling on GPU clusters for deep learning. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pp. 533–548, 2020.
- [47]Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., Guestrin, C., Liang, P., and Hashimoto, T. B. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 193–210, 2024.arxiv.org/abs/2401.09670
- [48]Zhou, Y., Lei, T., Liu, H., Du, N., Huang, Y., Zhao, V., Dai, A. M., Chen, Z., Le, Q. V., and Laudon, J. Mixture-of-experts with expert choice routing. In Advances in Neural Information Processing Systems, volume 35, pp. 7103–7114, 2022.arxiv.org/abs/2202.09368
























