betaFeedback

WorksComputer Science

AdaFloor: Adaptive Floor Planning and Rank-Time Reuse for Long-Tail MoE RL Rollouts

Abstract

Synchronous MoE reinforcement-learning post-training suffers from response-length imbalance. Existing methods reduce request tails, yet completed ranks remain bound to Expert Parallelism (EP) collectives and expert ownership. In a representative 16-accelerator Qwen3-30B-A3B rollout, this dummy run traps 32.83% of rank-time. AdaFloor converts this EP-coupled capacity into bounded leases for auxiliary work to improve system resource efficiency through rank-time reuse. Historical prompt-tail prediction and a calibrated guard construct the KV-safe layout. AdaFloor then selects the floor-specific safe quorum schedule with the largest predicted release area. Its runtime shrinks the EP group and leases detached ranks until full-world restoration. Across three updated-policy seeds on DeepScaleR mathematical reasoning rollouts, its capacity-oriented Natural floor2 configuration with TailGuard reduces Vanilla's 29393.9 rank-seconds of dummy run by 96.15%. Rollout shortening eliminates 18693.6 rank-seconds, while coordinated exits convert another 9568.6 rank-seconds into reusable capacity, leaving 1131.7 rank-seconds. For sidecar reuse, its Planned residency floor4 configuration without TailGuard serves 3134 Qwen3-8B GSM8K queries per epoch at 99.46% accuracy and incurs only a 2 to 3% primary throughput reduction against the same configuration with sidecar execution disabled. Its latency-oriented Planned residency floor4 configuration with TailGuard reduces rollout time by 24.09% relative to Vanilla and by 2.31% relative to LengthSort with TailGuard. On public Eurus-2-RL code rollouts with the same model and 16-rank topology, AdaFloor reduces rollout time by 20.56% and dummy run by 89.13% while generated work and mean reward remain stable.

Introduction

Reinforcement-learning post-training improves reasoning and instruction following in large language models [6, 38, 47, 52, 80]. In parallel, sparse MoE architectures increase parameter capacity without activating every parameter for each token [5, 7, 9, 20, 24, 48]. When RL post-training is applied to a sparse MoE model, variable response lengths produce uneven rank completion times during synchronous rollout, while distributed expert execution prevents completed ranks from leaving independently. Aggregate throughput does not reveal this rank-level tail.

Each rollout step generates multiple responses per prompt and waits for the slowest rank, turning response-length heterogeneity into rank-level imbalance. Under conventional fixed-world EP execution, a rank that finishes its local responses cannot leave the rollout group independently. It may still host experts selected by active tokens elsewhere and must remain in an all-to-all dispatch, expert computation, and combine communication until the barrier [10, 18, 24, 45]. We call this interval dummy run. The problem is how to turn this EP-coupled interval into reusable rank-time while preserving the primary rollout objective without violating survivor memory or collective correctness. In a representative Qwen3-30B rollout on 16 accelerators, dummy run consumes 32.83% of total rank-time. It is not independently idle because EP collectives, expert placement, KV state, and communication groups still bind each rank to the rollout.

Recent systems reduce completion skew across rollout workers through tail-aware batching or prediction [11, 15], while TLT uses the remaining inter-worker slack for auxiliary work [17]. These methods address imbalance between workers. AdaFloor instead targets residual rank-time within the EP group of an active worker. Even after a rank completes its local requests, collectives and expert ownership can bind it to active tokens on other ranks. This interval is not free capacity visible to a cluster scheduler. Reclaiming it introduces three coupled requirements. The survivor group must remain KV-safe. Prompt placement must advance completion quorums rather than only balance local work. The runtime must preserve expert availability and collective order through shrink and full-world restoration.

Our key insight is that synchronized completions create predictable quorums at which the EP world can change safely. Coordinating prompt placement with expert availability and survivor KV capacity turns the remaining EP-coupled suffix into reusable rank-time.

AdaFloor combines guarded epoch construction with elastic EP execution. LengthSort+TailGuard, its non-elastic ablation, accounts for most of the measured rollout speedup. Elastic execution exposes residual EP-coupled rank-time for auxiliary work when safe completion quorums form. Within a selected guarded layout, the planner maximizes feasible release area while preserving its predicted step tails. Floor selection and prompt placement are jointly planned under survivor memory and expert constraints.

AdaFloor realizes this ordered policy through a planner and an EP runtime. For each fixed 32-prompt step, the planner builds floor-specific KV-safe pair graphs and solves a quota-aware binary ILP over quorum schedules. The runtime maps the result to physical ranks, prepares expert availability, progressively shrinks the EP group, and exposes restore-bounded communication until the barrier [10, 18, 24, 45]. We call this interval dummy run. The problem is how to turn this EP-coupled interval into reusable rank-time while preserving the primary rollout objective without violating survivor memory or collective correctness. In a representative Qwen3-30B rollout on 16 accelerators, dummy run consumes 32.83% of total rank-time. It is not independently idle because EP collectives, expert placement, KV state, and communication groups still bind each rank to the rollout.

Recent systems reduce completion skew across rollout workers through tail-aware batching or prediction [11, 15], while TLT uses the remaining inter-worker slack for auxiliary work [17]. These methods address imbalance between workers. AdaFloor instead targets residual rank-time within the EP group of an active worker. Even after a rank completes its local requests, collectives and expert ownership can bind it to active tokens on other ranks. This interval is not free capacity visible to a cluster scheduler. Reclaiming it introduces three coupled requirements. The survivor group must remain KV-safe. Prompt placement must advance completion quorums rather than only balance local work. The runtime must preserve expert availability and collective order through shrink and full-world restoration.

Our key insight is that synchronized completions create predictable quorums at which the EP world can change safely. Coordinating prompt placement with expert availability and survivor KV capacity turns the remaining EP-coupled suffix into reusable rank-time.

AdaFloor combines guarded epoch construction with elastic EP execution. LengthSort+TailGuard, its non-elastic ablation, accounts for most of the measured rollout speedup. Elastic execution exposes residual EP-coupled rank-time for auxiliary work when safe completion quorums form. Within a selected guarded layout, the planner maximizes feasible release area while preserving its predicted step tails. Floor selection and prompt placement are jointly planned under survivor memory and expert constraints.

AdaFloor realizes this ordered policy through a planner and an EP runtime. For each fixed 32-prompt step, the planner builds floor-specific KV-safe pair graphs and solves a quota-aware binary ILP over quorum schedules. The runtime maps the result to physical ranks, prepares expert availability, progressively shrinks the EP group, and exposes restore-bounded leases. This co-design is necessary because a favorable completion order is not reusable until the corresponding survivor world is both memory-feasible and executable.

On the DeepScaleR mathematical reasoning workload, AdaFloor exposes separate operating points for capacity, sidecar reuse, and rollout latency. Across three updated-policy seeds, its capacity-oriented Natural floor2 configuration with TailGuard reduces Vanilla’s 29393.9 rank-seconds of dummy run by 96.15%. Rollout shortening eliminates 18693.6 rank-seconds, while coordinated exits convert another 9568.6 rank-seconds into reusable capacity, leaving 1131.7 rank-seconds. Its sidecar-oriented Planned residency floor4 configuration without TailGuard serves 3134 Qwen3-8B GSM8K queries at 99.46% accuracy and incurs only a 2 to 3% primary throughput reduction against the same configuration with sidecar execution disabled. Its latency-oriented Planned residency floor4 configuration with TailGuard reduces rollout time by 24.09% from Vanilla and by 2.31% from LengthSort with TailGuard. On public Eurus-2-RL code rollouts, a request-seed-matched run reduces rollout time by 20.56% and dummy run by 89.13% while preserving generated work and mean reward. In a two-worker EP8 composition, AdaFloor adds 7219.0 rank-seconds within active workers to 2730.1 rank-seconds of TLT-style cross-worker slack, increasing total reusable capacity by 3.64×.

This paper makes the following contributions.

  • We identify EP-coupled dummy run as residual rank-time after local completion, distinguish it from independently schedulable idle capacity, and quantify its duration and utilization on a real MoE rollout.

  • We design an ordered planner that constructs guarded epoch batches and then jointly selects KV-safe floors, quorum schedules, and prompt placement to increase predicted release area without changing the fixed epoch layout.

  • We implement progressive EP shrink and bounded rank leases on 16 Ascend devices. End-to-end runs across mathematical reasoning and code workloads validate released rank-time, inference sidecar utility, rollout latency, safety, control overhead, training-signal comparability, and composition with worker-level reuse.

Background

Synchronous MoE RL Rollout

RL post-training alternates rollout, reward computation, and policy updates across distributed resources [16, 38, 46, 47, 51, 60, 65]. Our workload assigns 32 prompts to 16 rollout ranks. Each prompt produces n=16n = 16 sampled responses, so each rank owns two prompts and 32 responses. Let MrM_r be the time when rank rr finishes all local responses. The synchronized step completes at

T=max⁡rMr.(1)T = \max_r M_r. \tag*{(1)}

A few long responses can therefore hold all ranks. Completed requests cannot leave independently because samples from one

policy version enter later RL stages together. This tail is not only a scheduling problem. An MoE rank can keep receiving tokens for experts selected elsewhere after local completion. AdaFloor must therefore change the active EP world while preserving state and collective correctness.

EP-Coupled Dummy Run

EP partitions MoE experts across ranks [7, 9, 24, 48]. Figure 1 shows that each rank is both a token owner and an expert host. A locally finished rank may still receive remote tokens for its resident experts.

Figure 1. EP couples token ownership and expert placement. A rank with no unfinished local responses may still host experts selected by active tokens on other ranks, so it cannot leave a fixed EP group independently.

We call a rank active while it owns at least one unfinished response. It enters dummy run after all local responses finish but before it can leave the active EP group. Reclaiming this interval requires a legal smaller EP group, sufficient survivor memory, and a transition that preserves collective order.

Shrink Floors, Quorums, and KV-Safe Planning

A shrink floor is the number of ranks that remain active after a transition. Our 16-rank runtime supports progressive ladders such as 16→8→4→216 \to8 \to4 \to2. These transitions become available after 8, 12, and 14 ranks complete. For a ladder F=⟨f0,…,fS⟩\mathcal{F} = \langle f_0, \ldots, f_S \rangle, let tst_s be the time when floor fsf_s becomes ready and let TT be the step tail. Its released rank-time is

R(F)=∑s=1S(fs−1−fs)(T−ts).(2)R(\mathcal{F}) = \sum_{s=1}^{S}(f_{s-1} - f_s)(T - t_s). \tag*{(2)}

This objective counts only time after a valid quorum. Rank-local idle that appears before a quorum remains trapped inside the active EP group.

Deeper floors expose more rank-time but concentrate responses, experts, and workspace on fewer survivors. Maximum response length alone does not capture their KV pressure. For prompt pair e=(i,j)e = (i,j), AdaFloor estimates the active token peak as

Ae=max⁡τ(τ⋅#{ℓ∈Li⊎Lj∣ℓ≥τ}),(3)A_e = \max_{\tau}\left(\tau\cdot\#\{\ell\in L_i \uplus L_j \mid\ell\ge\tau\}\right), \tag*{(3)}

where LiL_i is the multiset of sampled response lengths for prompt ii. Let Cf,meffC^{\mathrm{eff}}_{f,m} be the per-rank KV token capacity profiled by vLLM from residual HBM after prompt KV, block allocation, model and expert state, and workspace. Pair ee is feasible only when Ae≤Cf,meffA_e \le C^{\mathrm{eff}}_{f,m}. Section VI-A inflates predicted lengths before applying this constraint.

Backend legality and memory feasibility are separate. AdaFloor accepts only runtime-supported floors, checks survivor KV and workspace per step, and retains full16 as the correctness fallback.

Shrink is progressive, so a floor2 plan traverses 16→8→4→216 \to8 \to4 \to2 and exposes each earlier interval. The planner considers every quorum because a late deep transition can release less area than a shallower one.

Characterizing Patterns Behind Long-Tail Waste

Methodology

We profile one 1305.886 s Qwen3-30B DeepScaleR rollout and record response lengths, rank completion, utilization, KV pressure, and preemption. We replay alternative prompt orders, pairings, ladders, and floors on this trace. Online experiments validate plans built only from history.

Observation 1. Synchronized Tails Dominate Rollout Waste

The node accumulates 14,034.8 active rank-seconds and 6,859.4 dummy-run rank-seconds. Dummy run therefore accounts for 32.83% of total rank-time. Ranks 2, 15, and 6 spend 83.22%, 74.39%, and 71.39% of the cycle in dummy run, while three tail ranks remain active until the barrier. Figure 2 shows long suffixes rather than scattered gaps. Average utilization drops from 14.78% during active execution to 2.04% during dummy run. These intervals can support long leases, but local completion does not create a legal smaller EP world.

Figure 2. Per-rank activity during one 1305.886 s Qwen3-30B rollout cycle on 16 ranks. Dummy-run suffixes account for 32.83% of total rank-time but remain EP-coupled until a valid completion quorum enables shrink.

Insight 1. Long-tail waste is temporally structured, but local completion alone does not make a rank reusable.

Observation 2. Tail Structure is Predictable but Imperfect

Figure 3 compares prompt-level maximum response lengths across adjacent epochs. Most prompts remain stable, making historical prediction useful for epoch construction, but three prompts at 3.2K to 3.7K tokens reach the 16K limit. Such outliers can turn a planned short step into the epoch tail.

Figure 3. Prompt-level maximum response lengths for one adjacent epoch pair. Most prompts remain stable, while three rare jumps reach the 16K response limit and motivate a calibrated step-level tail guard.

Insight 2. Prediction can shape the epoch, but short planned steps need protection from rare underestimation outliers.

Observation 3. KV-Safe Assignment and Release Quality are Pair-Level

Figure 4 shows why nearest-length pairing is insufficient. Pair (p3,p4)(p_3,p_4) has four responses alive at τ=39\tau= 39, giving an active-token peak of 156 above the KV cap of 144. Pair (p0,p3)(p_0,p_3) has only two live responses and a safe peak of 78. KV feasibility therefore depends on response overlap rather than maximum length alone. Among KV-safe assignments, the release-aware matching advances the 4→24 \to2 quorum from 60 to 52 and increases release area from 66 to 82 under the same tail time.

Toy four-rank assignment under a KV cap

Figure 4. Toy four-rank assignment under a KV cap of 144 token units. The complete workload determines pair safety through response overlap. Release-aware matching reaches the 4→2 transition earlier despite larger local skew.

When local balance misses a quorum. MinSkew minimizes local completion gaps, but KV filtering can force short prompts into pairs that finish early without completing the next shrink quorum. Quota-aware matching may accept a larger local gap to advance that quorum. For a matching M\mathcal{M}, let nM(τ)=∑e∈M1[Ae≤τ]n_{\mathcal{M}}(\tau) = \sum_{e \in\mathcal{M}} 1[A_e \le\tau]. A quorum opportunity exists

nMMS(τ)<N−f≤nMQA(τ),τ<Tt.(4)n_{\mathcal{M}_{\mathrm{MS}}}(\tau) < N-f \le n_{\mathcal{M}_{\mathrm{QA}}}(\tau), \qquad\tau< T_t. \tag*{(4)}

Equation (4) is the quorum-gap condition used in the replay analysis. The total benefit is determined by the weighted release area across all stages.

Insight 3. KV-safe local balance can miss early completion quorums. Quota-aware placement targets released rank-time directly.

Observation 4. The Best Floor is Step-Dependent

A deeper floor releases more ranks but reduces the feasible prompt-pair graph and increases survivor memory pressure. Let Gf=(V,Ef)G_f = (V, E_f) be the graph for floor ff, where

Ef={(i,j)∣Ai,j≤Cfeff}.(5)E_f = \{(i,j) \mid A_{i,j} \le C_f^{\mathrm{eff}}\}. \tag*{(5)}

As the effective capacity decreases, KV-heavy edges disappear and the graph may no longer contain a perfect matching. A pair feasible at floor8 can therefore be invalid at floor4 or floor2 even though the prompt set is unchanged.

Fixed-depth stress tests confirm the resulting trade-off. A forced deep floor exposes more theoretical release area on short steps, but it can trigger KV preemption, fail workspace allocation, or require a slower fallback on long steps. A globally shallow floor avoids these failures but leaves safe floor2 and floor4 opportunities unused. In a floor2-enabled epoch2 plan, the first two short steps select floor2, the next two select floor4, and the final 16K step remains at full16. An aggressive fixed floor would either fail on the last steps or require a conservative global cap that wastes the early opportunities.

★ Insight 4. Shrink depth is a step decision. AdaFloor should compare the best KV-safe schedule from every feasible floor and select the largest release area.

Observation 5. Released Rank-Time is Practically Usable

Final Natural and Planned residency policies release 3.6K to 9.6K rank-seconds per epoch. Lease duration, rank-set fragmentation, initialization, and restore deadlines determine how much work this area can support. Our end-to-end evaluation serves up to 3134 Qwen3-8B GSM8K queries per epoch, showing that the windows are not merely accounting slack.

★ Insight 5. Released rank-time becomes a resource only after the runtime decouples ranks from EP and exposes explicit, restore-bounded leases.

Together, these observations motivate guarded tail sorting, KV-safe pair graphs, release-area-aware planning, and restore-bounded leases.

AdaFloor Overview

AdaFloor applies an ordered objective to the candidate layouts produced by sorted batching and bounded repair. Let π\pi denote one such epoch plan and let Πsafe\Pi_{\mathrm{safe}} contain the candidates that satisfy backend legality, KV capacity, survivor memory, and TailGuard constraints. AdaFloor selects

π⋆=lexmin⁡π∈Πsafe(T^epoch(π),−R^(π)).(6)\pi^\star= \underset{\pi\in\Pi_{\mathrm{safe}}}{\operatorname{lexmin}}\left(\widehat{T}_{\mathrm{epoch}}(\pi),-\widehat{R}(\pi)\right). \tag*{(6)}

The lexicographic objective first minimizes predicted rollout time over the safe candidate layouts and then maximizes their predicted release area. With step membership and response caps fixed, the floor planner preserves the selected layout’s modeled tail lengths. This guarantee applies to the examined candidate layouts. Actual rollout time also depends on floor-dependent computation, communication, and transition costs, which are not fully represented by T^epoch\widehat{T}_{\mathrm{epoch}}.

Figure 5 summarizes the two-level planner and runtime. The epoch level predicts prompt tails, forms guarded 32-prompt steps, and repairs only steps that the complete step planner cannot solve. For each retained step, the planner call jointly selects the KV-safe floor, completion quorums, and prompt-to-rank map. It either returns a complete cached plan or reports failure to the repair loop. Validation and final planning use the same solver. One cached plan carries each step’s response cap, logical pairing, physical placement, and survivor schedule from offline planning into execution without recomputation.

Figure 5. AdaFloor overview. Historical traces produce length-sorted candidate steps and calibrated response caps. For each fixed step, the planner jointly selects a KV-safe floor, completion quorum schedule, prompt pairing, and physical rank placement. The runtime switches EP groups at the selected quorums and leases detached ranks until full-world restoration.

The runtime loads the plan before generation. It applies the per-step response cap, monitors rank completion, switches to smaller precomputed or on-demand EP groups at the planned quorums, and continues decoding on survivor ranks. A rank becomes compute-shareable only after it leaves the active group and memory-shareable only after the relevant KV storage is released. All ranks restore full-world execution before reward computation and training resume.

Prediction- and KV-Robust Epoch Batching

AdaFloor first decides which prompts belong to each rollout step. It builds this epoch layout with the complete floor-conditioned planner in the loop, so a length-sorted step is retained only when the planner can also produce its final rank-level shrink plan.

Prompt-Level Tail Prediction and Sorted Steps

For prompt ii in epoch kk, let Li,j(k)L_{i,j}^{(k)} be the generated length of its jj-th sampled response. AdaFloor defines the realized prompt tail as

Mi(k)=max⁡j∈[1,n]Li,j(k).(7)M_i^{(k)} = \max_{j \in[1,n]} L_{i,j}^{(k)}. \tag*{(7)}

Using the maximum matches the synchronized objective. One long sample is enough to delay the prompt’s owning rank. AdaFloor maintains an exponential moving prediction

M^i(k+1)=αM^i(k)+(1−α)Mi(k),α=0.3.(8)\widehat{M}_i^{(k+1)} = \alpha\widehat{M}_i^{(k)} + (1-\alpha) M_i^{(k)}, \qquad\alpha= 0.3. \tag*{(8)}

This setting assigns 30% weight to the historical estimate and 70% to the most recent observation so the predictor follows recent rollout behavior while retaining one epoch of smoothing. The complete response-length multiset is retained separately for KV estimation. The predictor is maintained at dataset-item granularity because prompts sharing a template can still have different reasoning depth. The initial no-shrink epoch provides the first observation, and each completed epoch updates the estimate used to plan the next one.

AdaFloor sorts the epoch prompts by M^i\hat{M}_i and partitions the sorted order into contiguous 32-prompt steps. Let PsortedP_{\mathrm{sorted}} be this order. Step tt receives

Bt={Psorted[r]∣(t−1)B≤r<tB},B=32.(9)B_t = \{P_{\mathrm{sorted}}[r] \mid(t-1)B \le r < tB\}, \qquad B = 32. \tag*{(9)}

Grouping similar predicted tails avoids repeatedly mixing short and long prompts and minimizes the sum of step maxima under the predicted step-time model. Later repair is bounded because arbitrary regrouping could recover KV feasibility while recreating the epoch tails that sorting removes. The floor planner acts only after this primary layout decision.

Calibrated Step-Level Tail Guard

Observation 2 shows why prediction alone is insufficient. AdaFloor measures the one-sided underestimation ratio

ri+=max⁡(1,MiM^i+ϵ),(10)r_i^{+} = \max\left(1,\frac{M_i}{\hat{M}_i+\epsilon}\right), \tag*{(10)}

where ϵ\epsilon avoids division by zero. Ratios below one are clipped because overestimation is conservative for latency and should not reduce the guard. Let Qq(⋅)Q_q(\cdot) denote the empirical qq-quantile. AdaFloor computes ρq\rho_q from the most recent KK epoch transitions and, for step BtB_t, defines St=max⁡i∈BtM^iS_t = \max_{i\in B_t}\hat{M}_i. The response cap is

Ct=min⁡(Lmax⁡,max⁡(Cmin⁡,⌈ρqStQ⌉Q)).(11)C_t = \min\left(L_{\max},\max\left(C_{\min},\left\lceil\frac{\rho_q S_t}{Q}\right\rceil Q\right)\right). \tag*{(11)}

Here CtC_t is the cap applied to step tt and Lmax⁡L_{\max} is the global response limit. The lower bound Cmin⁡C_{\min} avoids aggressive caps on short predictions. The quantile qq supplies a one-sided margin and the recent window of KK transitions lets that margin adapt across epochs. The granularity QQ aligns caps with runtime token blocks. We use q=0.95q = 0.95, K=3K = 3, Cmin⁡=4096C_{\min} = 4096, Q=512Q = 512, and Lmax⁡=16384L_{\max} = 16384. Short planned steps receive bounded caps, while naturally long steps recover the full budget.

Planner Validation and Bounded Repair

For every candidate step, the epoch planner invokes

P(Bt,Ct)→(success,ϕt),(12)\mathcal{P}(B_t,C_t) \to(\mathrm{success},\phi_t), \tag*{(12)}

where ϕt\phi_t contains the selected floor, shrink schedule, logical prompt pairs, physical rank placement, survivor groups, and expected release area. Thus, validation and final planning are the same operation. The epoch stage only decides whether the candidate prompt-to-step grouping can be kept.

If P\mathcal{P} fails under every allowed floor, AdaFloor performs bounded repair with the preceding step and reruns P\mathcal{P} on the affected steps. Starting from the last rejected step, it exchanges Algorithm 1 Epoch construction from historical response traces. Every retained step has a validated floor, quorum schedule, and prompt-to-rank assignment.

Require: Prompt set PP and historical response traces Ensure: Planner validated epoch plan Π\Pi

  1. Update tail predictions and uncertainty calibration

  2. Sort PP and form contiguous candidate steps

  3. Compute guards and validate all steps with P\mathcal{P}

  4. for each rejected step from last to first do Search bounded swaps with the preceding step

  5. Recompute guards and rerun P\mathcal{P} on affected steps

  6. Cache the first valid repaired plans

  7. end for

  8. return Π\Pi

that step’s shortest prompt with the preceding step’s shortest prompt. If validation still fails, it re-sorts the preceding step and advances to the next-shortest prompt in the rejected step. Each swap must lower the rejected step’s KV demand. This bounded search keeps the perturbation local and prioritizes swaps with the smallest change to adjacent step maxima.

Algorithm 1 summarizes epoch construction with planner validation.

A step is not repaired merely because it cannot reach the deepest floor. If the planner returns a safe floor8 or full16 plan, AdaFloor keeps the sorted grouping and caches that shallower result. Repair is therefore a feasibility fallback, not a second release optimizer. The output of this stage is a planning-validated epoch layout with fixed prompt groups, tail guards, and cached per-step plans.

Release-Area-Aware Multi-Floor Planning

This section defines P(Bt,Ct)\mathcal{P}(B_t,C_t). Given one fixed step, the planner jointly chooses the floor, shrink schedule, logical prompt pairs, and physical rank mapping. These decisions cannot be separated because a floor is feasible only if its KV-safe graph admits a matching whose completion order satisfies the required shrink quorums.

Floor-Specific KV-Safe Graphs

For pair e=(i,j)e=(i,j), AdaFloor uses

Me=max⁡(M^i,M^j)(13)M_e=\max(\hat{M}_i,\hat{M}_j) \tag*{(13)}

as its rank completion proxy. To keep KV admission conservative under modest prediction drift, each historical response length is inflated by μ=1.16\mu=1.16 and clipped at CtC_t and Lmax⁡L_{\max}. The resulting lengths are evaluated with Eq. (3) to obtain Aμ,eA_{\mu,e}.

Floor ff under runtime mode mm induces graph Gt,f=(Vt,Et,f)G_{t,f}=(V_t,E_{t,f}), where Vt=BtV_t=B_t and

Et,f={e∣Aμ,e≤Cf,meff}.(14)E_{t,f}=\{e\mid A_{\mu,e}\le C^{\mathrm{eff}}_{f,m}\}. \tag*{(14)}

We use the capacity for each runtime mode defined in Section II-C. Natural mode creates resources on demand and supports floors {2,4,8,16}\{2,4,8,16\}, while Planned residency precreates selected groups and supports {4,8,16}\{4,8,16\}. At floor4, both modes

allocate 13.50 GiB per rank for 32 expert slots. Planned residency adds a calibrated 13.50 GiB reserve for communication state, dispatcher state, and runtime workspace, reducing its KV budget from 280,576 to 133,120 tokens. Budgets for each mode prevent plans that pass offline checks but fail during group warmup.

Candidate Schedules and Quota-Aware Matching

AdaFloor supports any number of shrink stages, as captured by the general release-area objective in Eq. (2). A candidate schedule assigns a pair-completion proxy to each shrink quorum. In our 16-rank implementation, the supported ladders instantiate this objective as

R8(a)=8(Tt−a),R4(a,b)=8(Tt−a)+4(Tt−b),R2(a,b,c)=8(Tt−a)+4(Tt−b)+2(Tt−c).(15)\begin{aligned} R_8(a)&=8(T_t-a),\\ R_4(a,b)&=8(T_t-a)+4(T_t-b),\\ R_2(a,b,c)&=8(T_t-a)+4(T_t-b)+2(T_t-c). \tag*{(15)} \end{aligned}

Candidate transition points are drawn from the distinct pair-completion proxies {Me}\{M_e\}. AdaFloor enumerates schedules in descending release area, so the first feasible schedule for a ladder is optimal within the enumerated space.

For each KV-safe edge e∈Et,fe\in E_{t,f}, binary variable xex_e indicates whether the prompt pair is selected. A valid assignment satisfies

∑e∋ixe=1,∀i∈Vt,(16)\sum_{e\ni i}x_e=1,\qquad\forall i\in V_t, \tag*{(16)}

and, for each planned floor fsf_s reached at completion proxy τs\tau_s,

∑e∣Me≤τsxe≥N−fs.(17)\sum_{e\mid M_e\le\tau_s}x_e\ge N-f_s. \tag*{(17)}

Thus, the formulation permits any positive number of ranks to exit at each stage, provided that the cumulative exits do not exceed the original NN ranks and the resulting floors are runtime-valid. For 16→8→4→216\to8\to4\to2, Eq. (17) yields quorums of 8, 12, and 14 at aa, bb, and cc.

Because schedule enumeration determines the primary objective for released area, the ILP uses this cost only to choose among matchings satisfying the same schedule.

min⁡x∑e∈Et,fcexe,ce=106gap⁡e+103Me+Aμ,e.(18)\min_x\sum_{e\in E_{t,f}}c_ex_e,\qquad c_e=10^6\operatorname{gap}_e+10^3M_e+A_{\mu,e}. \tag*{(18)}

Here gape=∣M^i−M^j∣gape = |\hat{M}_i - \hat{M}_j| favors pairs with similar predicted tails, MeM_e favors earlier pair completion, and Aμ,eA_{\mu,e} favors lower calibrated KV pressure. The coefficients set their relative influence when choosing among matchings that satisfy the same release quotas. KV-safe filtering remains a hard constraint because unsafe edges are excluded from Et,fE_{t,f}. The schedule is fixed before this cost is evaluated, so these preferences do not change its predicted release area or the order in which floors are tested.

Scope of the optimization. For identical feasible candidates, exact quota-aware optimization can retain any admissible MinSkew schedule and therefore cannot reduce the release area calculated from response lengths. This property concerns the fixed-length release model only. It does not guarantee measured latency or runtime rank-time because execution, transition, and forecast costs are separate.

Joint Floor Selection and Physical Materialization

AdaFloor evaluates every candidate floor permitted by the runtime. Each floor contributes its best schedule only when the corresponding graph admits a KV-safe perfect matching with positive release area. A floor2 plan includes the earlier 16→8 and 8→4 stages rather than replacing them. If floor2 is unsafe or releases less area, the planner may select floor4 or floor8. Full16 is the correctness fallback.

Let Rt,f⋆R^\star_{t,f} be the largest feasible schedule area for floor ff in step tt. The planner selects

f^t=arg⁡max⁡f∈FtsafeRt,f⋆.(19)\hat{f}_t = \arg\max_{f \in\mathcal{F}^{\mathrm{safe}}_{t}} R^\star_{t,f}. \tag*{(19)}

Ties prefer the larger floor and therefore avoid a transition that releases no additional area.

The ILP returns logical prompt pairs rather than physical rank identifiers. AdaFloor orders the pairs by MeM_e and maps them to the roles required by the runtime. For floor4, the earliest eight pairs map to donor ranks 0 through 7, the next four to wave ranks 8 through 11, and the longest four to final survivors 12 through 15. For floor2, the next two pairs occupy ranks 12 and 13, and the longest two occupy ranks 14 and 15. This ordering ensures that the physical ranks expected to leave at each stage are also the ranks predicted to complete first.

A deeper floor can lose even when it is feasible because its tighter KV graph may delay an earlier quorum by more than the final stage recovers. Comparing Rt,f⋆R^\star_{t,f} across floors captures this effect. Equal-area plans prefer the larger floor to avoid an unnecessary transition.

Each step has 32 vertices and at most 496 edges. Discrete schedule points and first-feasible termination keep planning off the token path. The five step plans are cached once per epoch, with measured cost reported in Section VIII-F.

Runtime Realization and Implementation

Lossless Expert Availability after Shrink

Shrinking the active EP group requires the survivor ranks to retain lossless access to all experts needed by the remaining tokens. We implement and evaluate five mechanisms. Mode 1 uses resident replicas. Mode 2 uses CPU swapping. Mode 3 uses CPU double buffering. Mode 4 fetches experts from remote NPUs. Mode 5 combines remote NPU and CPU sources. These mechanisms differ in the source and staging path used to materialize missing experts after a survivor set is selected. Our evaluation in Sec. VIII-F shows that Mode 1 achieves the lowest measured rollout time, so AdaFloor adopts it as the default expert-materialization substrate. This choice is separate from the Natural or Planned residency group life-cycle that determines when survivors become known. Replica residency can restrict shrink depth by reducing KV capacity. AdaFloor then selects a higher feasible floor or retains Full16, reducing reclamation rather than requiring a switch to Mode4. AdaFloor restores the full-rank layout before actor training and reuses VERL’s standard weight synchronization before the next rollout. Natural and Planned prepare survivor replicas from the newly loaded weights, which remain fixed throughout that rollout.

Progressive Detach and Group Switching

Before rollout, the driver attaches the cached plan and tail guard to the batch metadata. During generation, ranks report whether they still own unfinished responses. Survivors switch to smaller DP, EP, and communication groups only after observed completions satisfy the required quorum and runtime safety checks pass. Predicted completion alone never triggers rank release. Runtime transitions follow halving stages, such as 16→8→4→2, which use the optimized collective path and avoid slow non-halving state movement.

An exiting rank does not terminate its process. It detaches from the active compute group, clears shrink-specific routing and communication state, and waits in the rollout loop until survivors finish. Non-members still participate in group-creation ordering so that all ranks remain aligned for later restore. This distinction is important. Released ranks stop serving rollout tokens, but their workers remain alive.

Compute, Memory, and Restore Boundaries

Detach releases compute ownership but does not necessarily free the global KV tensor. Request blocks can return to the allocator while the backing cache storage remains resident. Sidecar admission therefore distinguishes compute-shareable ranks from memory-shareable ranks and invokes explicit KV resize or cache-release paths when memory is required. KV admission checks account for actual cache capacity. During generation, insufficient KV blocks trigger scheduler preemption. Affected requests release their blocks and later recompute KV from retained tokens, trading extra computation for bounded KV residency. Sidecars use only released ranks and do not share primary survivors’ KV pools.

Before full-world restore, sidecars must stop, durably merge results, and acknowledge resource release. Missing acknowledgement blocks restore. Reward computation and policy training resume only after restoration completes.

AdaFloor supports two runtime policies. Natural mode lets realized completion determine the survivor set and constructs lower-floor resources on demand. It tolerates prediction drift and preserves more KV headroom, including floor2 support, but pays group-setup cost during execution. Planned residency fixes the planned survivor topology and keeps selected communication and dispatcher resources warm across steps. It reduces transition latency but consumes HBM and can leave planned survivors in dummy run when realized completion order drifts. The offline planner uses the corresponding effective capacities for each policy.

Correctness Invariants

The runtime enforces three invariants. First, exiting ranks must finish their local requests, and every transition requires a valid completion quorum and preserved expert availability on survivors. Second, all ranks observe communication-group creation and restore in a consistent order, including ranks that are no longer active participants. Third, the next RL stage begins only after full-world restore. These invariants separate performance policy from correctness. A rejected shrink leaves execution in the current group.

Sidecar leases begin only after compute detach and required memory release. Admission stops when duration, memory, or restore constraints cannot be met, and no lease may alter the training topology. Q4 measures lease consumption and shared-resource interference.

Implementation

AdaFloor is implemented in VERL 0.6.0 with vLLM 0.11.0, vLLM Ascend 0.11.0rc0, PyTorch NPU, CANN 8.5.0, and HCCL. The offline planner uses SciPy MILP and NetworkX matching [13], [58] and writes plans keyed by dataset-item identity.

Evaluation

We answer five questions on rollout performance, mechanism contributions, KV and tail safety, sidecar reuse, and system overhead.

Experimental Setup

We evaluate AdaFloor on one node with eight dual-die Huawei Ascend 910C packages. Its 16 software-visible devices each have 64 GiB HBM. Eight SIO-connected die pairs communicate through HCCS. The rollout model is a local Qwen3-30B-A3B checkpoint. It is a 30.53B parameter BF16 MoE with 128 experts and top-8 routing. Rollout uses TP1, PP1, DP16, and EP16. The actor uses TP4, PP4, and EP4.

The rollout workload uses the DeepScaleR training split. Each step contains 32 prompts and samples 16 responses per prompt, yielding 512 responses. Maximum prompt and response lengths are 1024 and 16384 tokens. Sampling uses temperature 0.9, top-p 0.9, and top-k 50. Epoch 0 provides the common history for every epoch 1 plan. Each step computes rewards and advantages and performs a GRPO actor update at a 10−610^{-6} learning rate. Updated actor weights are synchronized before the next rollout. Epoch 2 loads the actor checkpoint saved after the five epoch 1 updates and plans from its epoch 1 responses. The measured epochs are therefore updated-policy training trajectories rather than frozen-model replays. For each of three independent seeds, we average the two sequential five-step epochs and report the mean and sample standard deviation across seeds. Every policy uses the same 160-prompt multiset and sampling parameters per epoch.

We compare five policy families. Vanilla uses random batching and all 16 ranks. LengthSort uses historical response tails without shrinking. FixedFloor targets floor4 using KV-constrained MinSkew matching and repairs predicted-infeasible epoch buckets by swapping prompts between neighboring steps.

MinSkew minimizes local completion gaps using the same floor-specific graphs. AdaFloor jointly selects a KV-safe floor and prompt matching to maximize predicted released rank-time. AdaFloor-N and AdaFloor-P denote the Natural and Planned residency lifecycles. FixedFloor, MinSkew, and AdaFloor use the same LengthSort and TailGuard epoch construction. LengthSort with TailGuard is AdaFloor’s non-elastic ablation, retaining guarded epoch construction while keeping all 16 ranks active. This comparison measures the incremental effect of elastic reclamation beyond epoch construction. The LengthSort row omits TailGuard.

Planner-only analysis protocol. We replay the 150 original shrinking steps in Figure 6 using unchanged prompt membership and response lengths. All planners use the same Natural KV capacities and the original response-peak rule. Stepwise-F4 denotes fixed-floor MinSkew without cross-step repair. An unsafe step falls back to full16. This replay uses observed lengths as inputs and is not an online prediction experiment.

Figure 6. End-to-end results across three seeds, each averaging epochs 1 and 2. Bars show means and whiskers show sample standard deviations. (a) Five-step rollout time and reduction from Vanilla. The bracket compares Ada-P floor4 with LengthSort using TailGuard. LS and LS+TG denote LengthSort without and with TailGuard. (b) Rank-time released from coordinated exits to the rollout boundary, stacked by transition stage with grayscale-safe hatching. Ada-N and Ada-P denote Natural and Planned residency. Planned floor2 is infeasible under the evaluated HBM budget.

End-to-End Reclamation and Rollout Performance

Q1. Can AdaFloor reclaim rank-time while preserving rollout performance? Figure 6 reports both outcomes across three seeds. The capacity-oriented Natural floor2 point reduces rollout time by 19.88% relative to Vanilla while generated work differs by 0.27%. Accounting from local completion to the rollout boundary gives Vanilla 29393.9 rank-seconds of dummy run per epoch, or 31.25% of its total rollout rank-time. This is close to the 32.83% representative cycle in Section III-B. The shorter critical path eliminates 18693.6 rank-seconds, while coordinated exits convert another 9568.6 rank-seconds into reusable capacity. Together they reduce dummy run by 28262.2 rank-seconds, or 96.15%, leaving 1131.7 rank-seconds. The latency-oriented Planned residency floor4 point records the shortest path at 4462.1 s and generates 15.400 million response tokens per epoch, compared with 15.370 million for Vanilla. Its 24.09% reduction from Vanilla includes robust epoch construction. LengthSort alone reaches 4891.3 s, while the stronger LengthSort with TailGuard full-world control reaches 4567.5 s. Relative to this controlled baseline, coordinated shrink contributes a further 2.31% reduction. The mean paired reduction is 105.4 s with a two-sided 95% Student tt interval from 26.8 to 184.0 s.

Figure 7. Robust epoch construction for one representative trace. Steps 1 through 5 belong to epoch 1 and steps 6 through 10 belong to epoch 2. Panel (a) reports per-step rollout time for Random, LengthSort, and LengthSort with TailGuard. Panel (b) compares predicted tails and guard caps with realized maxima from guarded and unguarded LengthSort. Markers and line styles remain distinct in grayscale.

The operating points expose a latency-capacity tradeoff. Planned residency floor4 minimizes wall time. Natural floor2 releases 18.21% more area than Natural floor4 at a 4.43% longer critical path and reaches 99.28% of the MinSkew area with a 0.62% shorter path. Across the measured means, all three AdaFloor configurations lie on the latency–released-rank-time frontier alongside MinSkew, offering distinct choices between primary speed and reclaimable capacity.

Mechanism Analysis

Q2. Which mechanisms create rollout speedup and reusable capacity?

a) Robust epoch construction: We first isolate epoch construction by keeping all 16 ranks active. Across three seeds, LengthSort reduces mean rollout time from 5878.3 to 4891.3 s by 16.79% and improves throughput by 20.15%. TailGuard reaches 4567.5 s and 3368.3 tokens per second, further improvements of 6.62% and 7.20%. Across 30 updated-policy steps per method, mean response length, reward, and actor KL differ by at most 0.15%, 0.07%, and 0.14%, and every update has nonzero gradients. Since both policies remain full-world, these comparisons isolate epoch construction from elastic rank release. Only five of 4096 guarded short-step responses reach their caps. A paired three-step trace with fixed assignments and request seeds produces identical responses, masks, and scores for all 1536 samples under full16 and Planned residency floors 4, 4, and 8.

History sorting places 74.0% to 79.0% of prompts in the realized 32-prompt bucket and 99.4% to 99.8% within one bucket. Noncausal sorting by realized tails lowers the sum of step maxima by 13.5% to 15.0%. This gap measures prediction uncertainty rather than attainable speedup and motivates TailGuard and online quorum checks.

b) Sensitivity to forecast error: We scale historical forecasts by 0.7 or 1.3, or assign either factor to each prompt using a deterministic hash (Mixed). Each Natural-F2 window uses seed 505 and five steps of 512 responses, retaining forecast-dependent floors, TailGuard caps, and cross-step repair. All windows complete without OOM or abort.

TABLE II FORECAST SENSITIVITY. PERTURBATION WINDOWS FREEZE THE ACTOR. THE REUSED ×1.0 REFERENCE INCLUDES ACTOR UPDATES.

ForecastTime (ks)Release (k rank-s)RewardTG/P
×1.04.918.280.83522/0
×0.74.452.210.8324265/55
×1.34.9410.350.83751/0
Mixed5.026.970.8336146/16

Table 2. Forecast sensitivity. Perturbation windows freeze the actor. The reused ×1.0 reference includes actor updates.

TG counts responses reaching a TailGuard cap below 16384, excluding global-limit hits. P counts KV preemptions. Time denotes rollout time.

Table II shows continued reclamation under all tested perturbations. Underprediction and mixed errors increase reduced-cap hits and expose forecast-only admission to preemption. For ×1.3, repair exchanges five prompts in each direction between steps 4 and 5 and raises step 4 to F8, enabling completion with zero preemptions. Its larger released area reflects both placement and realized tail duration. These single-window measurements characterize execution sensitivity, while long-term learning requires updated-policy evidence.

c) Elastic floor planning and release-aware matching: With the layout and caps fixed, AdaFloor optimizes release area. Release starts at coordinated exit rather than local completion, so ranks waiting for a quorum are not counted as reusable. Ordinary-workload measurements show similar realized release for MinSkew and AdaFloor. The following replay and controlled experiment isolate the conditions in which their quorum objectives diverge.

d) Record-step opportunities: Under the replay protocol in Sec. VIII-A, ILP matches MinSkew on 142 records and improves calculated release by 0.028–2.267% on eight. Table III reports all ranks, including ties.

e) Controlled realization: We use a frozen actor, 32 real prompts, controlled response lengths, 512 responses, identical work, and the same Planned-F4 KV capacity of 133120 tokens per rank. MinSkew reaches the first shrink quorum at 6000 tokens, while quota-aware matching reaches it at 5200. The calculated release areas are 34400 and 40800 rank-token units. The controlled NPU pair releases 3093.67 and 3777.85 rank-seconds, respectively, while both arms generate 1,710,400 tokens and restore full16. Both complete without preemption, OOM, or abort. Figure 9 visualizes the two measured release waves.

Figure 8. Elastic scheduling for one representative trace with LengthSort and TailGuard fixed. Steps 1 through 5 belong to epoch 1 and steps 6 through 10 belong to epoch 2. Shading encodes rank-time from observed coordinated exits to rollout boundaries, while every cell reports the selected floor and released amount for grayscale use.

PlannerRank 1Rank 2Rank 3
Quota-aware ILP15000
MinSkew14280
Stepwise-F488260

Table III. Ranks by release area calculated from original response lengths on 150 Fig. 6 step records. Ties receive the same rank.

f) Workload sensitivity: Frozen request-seed-matched runs vary one workload dimension from the main setup. With eight responses per prompt and 16 prompts per step, Natural floor2 reduces rollout time by 21.19% and 18.43%, improves work-normalized throughput by 28.59% and 22.59%, and releases 9649.3 and 19637.3 rank-seconds, respectively. At an 8192-token limit, it reduces time by 1.22% and releases 3769.6 rank-seconds. All six runs finish without preemption, OOM, or abort. The smaller gain at 8192 tokens confirms that AdaFloor targets substantial decode tails rather than adding a constant speedup.

On public Eurus-2-RL code rollouts [4], a frozen request-seed-matched run uses the same model and topology with executable test-based rewards. AdaFloor selects floor4 only for step 1. It reduces rollout time from 8851.5 to 7031.6 s by 20.56%, improves work-normalized throughput by 25.73%, reduces dummy run from 22342 to 2429 rank-seconds by 89.13%, and reduces KV preemptions from 94 to 1. Generated tokens differ by 0.12%, mean reward is 0.4854 versus 0.4849, and neither run has an OOM or aborted response.

g) Composition with worker-level reuse: In a two-worker EP8 capacity experiment, we omit TLT’s model-specific draft training and speculative decoding and isolate its worker-level reuse boundary [17]. Each worker receives 16 prompts and runs an EP8 instance for five steps. Worker-level reuse observes only the gap between worker completion times. AdaFloor instead detaches completed ranks while their worker remains active. Cross-worker gaps expose 2730.1 rank-seconds, while AdaFloor adds 7219.0 rank-seconds within active workers. Together they expose 9949.1 rank-seconds, or 3.64× worker-level reuse alone. This distinct intra-EP window can expand the capacity available to a TLT-style drafter.

Figure 9. Controlled quorum-aware release. Shaded areas show measured released rank-time. The quota-aware release wave occurs earlier and exposes 22.12% more rank-time.

KV Feasibility and Tail Safety

Q3. Does adaptive planning preserve KV and tail safety? Table 1 gives the effective per-rank KV budgets used by the final planners. They are measured runtime profiles rather than free planner parameters. Natural mode validates floor2 with 131,072 KV tokens per rank, while Planned residency floor2 is infeasible during group warmup before KV sizing. The planner therefore selects floor2 only in Natural mode and only for the two shortest steps. Intermediate steps use floor4, and the longest step falls back to full16.

FloorNaturalPlanned residency
2131,072Not supported
4280,576133,120
8315,648262,656
16380,800380,800

Table 1. Profiled effective per-rank KV token capacities after model, expert, communication, dispatcher, and workspace reservations. Not supported means that group warmup exhausts HBM before usable KV allocation.

Across 30 Natural floor2 and 30 Planned residency floor4 DeepScaler steps, maximum realized cap use is 93.14% and 88.28%, with minimum headroom of 8994 and 15599 tokens. No step exceeds its effective cap. All 48 validated epochs complete without an OOM or aborted response, and these 60 profiled steps avoid unplanned KV preemption. A 20-step stress run forces floors 2, 4, and 8 through 41 shrink events and 20 full-world restores without a collective timeout, OOM, abort, or HBM growth. These results demonstrate effective admission at high KV occupancy under the evaluated workloads.

Sidecar Rank-Time Reuse

Q4. Can released rank-time support useful sidecar work? We use released ranks to serve Qwen2.5-1.5B, Qwen3-8B, and PanguProMoE-72B-A16B [57] on GSM8K and count queries completed within one rollout epoch. PanguProMoE-72B-A16B requires at least four NPUs, so every run uses floor4 as its minimum. Each model uses its best measured parallel deployment. The independent sidecar workload does not affect DeepScaler prompt assignment or floor selection. Every request has a 4096-token response limit. Without Tail-

Guard, Planned residency floor4 completes 67,272 Qwen2.5-1.5B queries, 3,134 Qwen3-8B queries, or 1,465 Pangu queries per epoch. Their GSM8K accuracies are 76.68%, 99.46%, and 98.08%. With the primary configuration unchanged, enabling sidecars reduces primary post-training throughput by only 2–3% relative to sidecar-disabled execution, while completing the auxiliary queries reported above on the same 16 devices. Table IV shows that TailGuard shortens release windows and query counts while retaining useful low-overhead inference capacity for multi-tenant and latency-insensitive cloud services. Query counts measure completed work before restore, demonstrating that reclaimed capacity supports useful auxiliary execution at modest primary interference.

floor4 policyQ2.5-1.5BQ3-8BPangu-72BAICore Rel.
Natural66,1843,0831,441139.49%
Planned residency67,2723,1341,465128.94%
Natural+TailGuard55,1642,5701,201140.09%
Planned residency +TailGuard21,9751,024478127.44%

Table IV. GSM8K sidecar queries completed during one floor4 rollout epoch. Each row is a separate run and each request has a 4096-token response limit. AICore utilization is normalized to the matching floor4 policy with sidecar execution disabled.

Runtime Mechanisms and System Overheads

Q5. Which expert availability mechanism is most efficient, and what are the offline planning and online control costs?

a) Expert availability: We explored a broad design space and implemented five representative mechanisms that cover resident replication, CPU swapping, double buffering, remote NPU fetch, and combined sources. Table V compares them under the same prompt workload and validated KV headroom. Every cell processes the same number of generated tokens. We assign per-response decode caps of 256, 512, 640, 768, and 896 tokens to rank groups of 8, 4, 2, 1, and 1. Mode 1 is fastest at every floor and remains near 100 s as shrink deepens. Resident replicas avoid expert transfer and staging on the decode critical path, while the alternatives incur growing movement overhead at smaller floors. AdaFloor therefore adopts Mode 1 to minimize survivor expert materialization cost in both Natural and Planned residency group lifecycles. Under residency pressure, AdaFloor raises the floor or retains Full16 rather than automatically switching to remote expert access.

ModeExpert availabilityfloor8floor4floor2
1Resident replica93.6598.12100.58
2CPU swap484.57496.78514.71
3CPU double buffer197.99199.72265.88
4Remote NPU144.91164.35193.43
5Dual source151.44185.70201.58

Table V. Single-step rollout time for five expert availability mechanisms under the same prompt workload and validated KV headroom. Every cell processes the same configured token workload and lower time is better.

b) Planning and control overheads: Figure 10 decomposes offline planning and online control for Planned residency floor4. We plan epoch 1 from epoch 0 and epoch 2 from epoch 1 with five repeated measurements per transition. History update, graph construction, MILP solving, repair, and other planner work average 0.031 s, 1.673 s, 2.815 s, 8.1 μs, and 0.259 s. Other planner work includes dataset mapping, orchestration, validation, and output materialization. The complete planner averages 4.778 s internally and 5.568 s with process

Figure 10. Offline planning and online control costs for Planned residency floor4. Bars report means and error bars span observed minima and maxima across ten planner runs or three control events. The shared logarithmic axis covers microsecond and second scale components.

startup and imports. Compact history construction takes 0.40 to 0.52 s once after each epoch. Including this preparation, offline work is about 0.14% of the 4427.01 s five-step rollout time.

Mean group preparation, shrink, and restore costs are 9.82 s, 1.97 s, and 4.51 s across three online events. Their combined 48.89 s is 1.10% of the same rollout time. Group preparation includes one initial expert prefill of 21.51 s. These costs cover expert preparation and layout transitions, not the complete actor-to-rollout weight synchronization pipeline. Offline planning remains outside the timed rollout and measured online control occupies a small fraction of its critical path. This decomposition also shows why group preparation is performed ahead of likely transitions in Planned residency while Natural mode preserves memory by preparing groups on demand. Five one-second HCCS samples per window report mean traffic of 11.273, 23.015, and 9.053 GB/s during decode, shrink, and restore. Shrink is therefore a brief communication-intensive phase.

Discussion

Policy choice depends on primary-runtime requirements and available reuse opportunities. For latency-sensitive training, Planned residency or the non-elastic LengthSort+TailGuard configuration is appropriate. A higher minimum floor restricts shrinking to shallower groups. For more reclaimable capacity, Natural permits deeper KV-safe floors with less resident preparation. For sidecar goodput, operators should consider lease duration, available memory, startup, restoration, and interference. More released area alone does not guarantee more completed auxiliary work. Matched-request tests confirm generation equivalence for floor4 and floor8. The updated-policy trajectories establish short-horizon training-signal comparability, not final convergence of a complete RL recipe.

a) Platforms and topology: Porting requires backend adaptation and re-profiling legal groups, KV/workspace budgets, expert residency, and transition costs. Multi-node extensions range from node-local EP with inter-node DP, when each replica fits locally, to cross-node EP and hybrid parallelism. For cross-node EP, a practical design could enumerate topology-aware survivor templates and solve placement and quorum planning within each. Template-specific memory and communication constraints would preserve shard dependencies and account for sidecar interference. Restricting templates bounds planning cost but may forgo release opportunities.

b) When quota-aware planning helps: Quota-aware planning helps when local balance delays a release quorum. Released capacity remains distinct from consumed sidecar work and energy savings.

AdaFloor exploits unequal local completion under continued EP coupling. Its generalization requires remaining work and state to fit on fewer devices, coordinated transitions preserving dependencies and collective order, and reuse windows that justify transition costs. Expert replication and KV admission realize these requirements for MoE. Distributed inference and dynamic prefill/decode allocation are potential extensions requiring workload-specific state management. Released ranks can support inference [22, 66, 76, 78], rollout generation, reward scoring, and evaluation [16, 51, 60, 65], or speculative draft generation and training [2, 3, 17, 25, 26]. AdaFloor determines when ranks become reusable, while these systems determine how to consume them.

Related Work

MoE Execution

Sparse MoE models scale capacity through conditional expert activation [5, 7, 9, 20, 24, 48]. Systems improve sharding, placement, load balancing, sparse kernels, and communication [10, 14, 18, 36, 39, 45, 71], while inference-oriented designs prefetch, buffer, or offload experts [8, 19, 21, 68, 69, 70]. Unlike these fixed-world systems, AdaFloor changes the active EP world when completion quorums, expert availability, and survivor KV constraints permit.

RL Post-Training and Rollout Systems

RLHF and reasoning post-training build on preference learning and policy optimization [6, 38, 46, 47, 52, 80]. DeepSpeed-Chat, NeMo-Aligner, HybridFlow, OpenRLHF, ReaLHF, RLFuse, ROLL, and Laminar improve orchestration, placement, and overlap between rollout and training [16, 33, 49, 50, 51, 60, 65, 79]. RollPacker, RhymeRL, and TLT reduce or predict request tails through tail batching, history-based prediction, or speculative generation [11, 15, 17]. AdaFloor instead forms a legal survivor world after local completion, when collectives and expert ownership still bind ranks. These approaches reshape completion profiles, while AdaFloor converts the remaining EP-coupled suffixes into release waves. Our two-worker experiment confirms this composition at TLT’s worker-level reuse boundary.

LLM Serving and Speculative Decoding

Serving systems improve continuous batching, preemption, model parallelism, and resource allocation between prefill and decode [1, 22, 27, 37, 66, 76, 78]. Adapter-aware and application-aware serving systems exploit shared base models and request dependencies [28, 61]. Speculative decoding reduces autoregressive latency through draft models, parallel candidates, or learned token trees [2, 3, 25, 26]. These methods schedule or accelerate independent streams. AdaFloor instead forms legal EP release waves that they can consume.

Elastic Execution and Resource Reuse

GPU schedulers improve utilization through time sharing, elastic allocation, and heterogeneous placement [12, 32, 35, 41, 43, 63, 64, 67]. SpotServe adapts LLM parallelism to preemptible capacity [34]. Colocation systems manage interference and memory handoff between training and inference [54, 59, 73]. These systems primarily resize whole jobs or consume independently usable slack. AdaFloor targets quorum-coupled rank-time whose usability also depends on expert and communication state.

KV Management and Optimization-Based Planning

KV systems reduce fragmentation, storage, transfer, and retrieval cost through paging, virtual memory, eviction, quantization, sparse attention, migration, or disaggregation [22, 23, 30, 31, 40, 42, 44, 55, 56, 62, 72, 74]. Optimization-based systems select allocations and parallel plans under performance, communication, and memory constraints [29, 43, 53, 75, 77]. AdaFloor uses these ideas at prompt-pair granularity, where each candidate floor induces a different KV-safe graph and the objective maximizes staged rank-time released at shrink quorums.

Conclusion

Long-tail MoE RL rollouts create trapped rank-time rather than independent accelerator idleness. AdaFloor reclaims it through prediction-robust batching, KV-safe multi-floor planning, and runtime rank release. Across distinct DeepScaler floor2 operating points, capacity-oriented Natural floor2 reduces dummy run from 29393.9 to 1131.7 rank-seconds per epoch, or 96.15%, and converts 9568.6 rank-seconds into reusable capacity. A sidecar-oriented point serves 3134 Qwen3-8B queries at 99.46% GSM8K accuracy with a 2 to 3% primary throughput cost. Its latency-oriented point is 24.09% faster than Vanilla and 2.31% faster than LengthSort with TailGuard. Results across reasoning and code workloads demonstrate EP rank-time reclamation and practical reuse. Additional rollout speedup depends on the selected operating point.

References

References

  1. [1]A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, A. Tumanov, and R. Ramjee, “Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve,” in USENIX Symposium on Operating Systems Design and Implementation, 2024, pp. 117–134. [Online]. Available: https://www.usenix.org/conference/osdi24/presentation/agrawalarxiv.org/abs/2403.02310
  2. [2]T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao, “Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads,” in International Conference on Machine Learning, 2024, pp. 5209–5235. [Online]. Available: https://proceedings.mlr.press/v235/cai24b.htmlarxiv.org/abs/2401.10774
  3. [3]C. Chen, S. Berant, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper, “Accelerating Large Language Model Decoding with Speculative Sampling,” arXiv preprint arXiv:2302.01318, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2302.01318
  4. [4]G. Cui, L. Yuan, Z. Wang, H. Wang, Y. Zhang, J. Chen, W. Li, B. He, Y. Fan, T. Yu, Q. Xu, W. Chen, J. Yuan, H. Chen, K. Zhang, X. Lv, S. Wang, Y. Yao, X. Han, H. Peng, Y. Cheng, Z. Liu, M. Sun, B. Zhou, and N. Ding, “Process Reinforcement through Implicit Rewards,” arXiv preprint arXiv:2502.01456, 2025. [Online]. Available: https://arxiv.org/abs/2502.01456
  5. [5]D. Dai, C. Deng, Z. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, Z. Xie, Y. K. Li, P. Huang, F. Luo, C. Ruan, Z. Sui, and Y. Liang, “DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models,” in Annual Meeting of the Association for Computational Linguistics, 2024, pp. 1280–1297. [Online]. Available: https://doi.org/10.18653/v1/2024.acl-long.70
  6. [6]DeepSeek-AI, “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,” arXiv preprint arXiv:2501.12948, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2501.12948
  7. [7]N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhин, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, Q. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui, “GLaM: Efficient Scaling of Language Models with Mixture-of-Experts,” in International Conference on Machine Learning, 2022, pp. 5547–5569. [Online]. Available: https://proceedings.mlr.press/v162/du22c.htmlarxiv.org/abs/2112.06905
  8. [8]Z. Fang, Y. Huang, Z. Wang, Y. Lu, W. Chen, Y. Yu, F. Yu, and Z. Zheng, “Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline,” in ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2025, pp. 574–588. [Online]. Available: https://doi.org/10.1145/3676641.3716261
  9. [9]W. Fedus, B. Zoph, and N. Shazeer, “Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity,” J. Mach. Learn. Res., vol. 23, pp. 120:1–120:39, 2022. [Online]. Available: https://jmlr.org/papers/v23/21-0998.htmlarxiv.org/abs/2101.03961
  10. [10]T. Gale, D. Narayanan, C. Young, and M. Zaharia, “MegaBlocks: Efficient Sparse Training with Mixture-of-Experts,” in Conference on Machine Learning and Systems, 2023, pp. [Online]. Available: https://proceedings.mlsys.org/paper_files/paper/2023/hash/5a54f7933768effe78927bccffe40-Abstract-mlsys2023.htmlarxiv.org/abs/2211.15841
  11. [11]W. Gao, Y. Zhao, D. An, T. Wu, L. Cao, S. Xiong, J. Huang, W. Wang, S. Yang, W. Su, J. Wang, L. Qu, B. Zheng, and W. Wang, “RollPacker: Taming Long-Tail Rollouts for RL Post-Training with ‘Tail Batching’,” in USENIX Symposium on Networked Systems Design and Implementation, 2026, pp. 849–866. [Online]. Available: https://www.usenix.org/conference/nsdi26/presentation/gao-wei
  12. [12]J. Gu, M. Chowdhury, K. G. Shin, Y. Zhu, M. Jeon, J. Qian, H. H. Liu, and C. Guo, “Tiresias: A GPU Cluster Manager for Distributed Deep Learning,” in USENIX Symposium on Networked Systems Design and Implementation, 2019, pp. 485–500. [Online]. Available: https://www.usenix.org/conference/nsdi19/presentation/gu
  13. [13]A. A. Hagberg, D. A. Schult, and P. J. Swart, “Exploring Network Structure, Dynamics, and Function Using NetworkX,” in Proceedings of the 7th Python in Science Conference, 2008, pp. 11–15. [Online]. Available: https://conference.scipy.org/proceedings/SciPy2008/paper_2/
  14. [14]and Practice of Parallel Programming, 2022, pp. 120–134. [Online]. Available: https://doi.org/10.1145/3503221.3508418
  15. [15]J. He, T. Li, E. Feng, D. Du, Q. Liu, T. Liu, Y. Xia, and H. Chen, “History Doesn’t Repeat Itself but Rollouts Rhyme: Accelerating Reinforcement Learning with RhymeRL,” in ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2026, pp. 929–945. [Online]. Available: https://doi.org/10.1145/3779212.3790172
  16. [16]J. Hu, X. Wu, W. Wang, Xianyu, D. Zhang, and Y. Cao, “OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework,” arXiv preprint arXiv:2405.11143, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2405.11143
  17. [17]Q. Hu, S. Yang, J. Guo, X. Yao, Y. Lin, Y. Gu, H. Cai, C. Gan, A. Klimovic, and S. Han, “Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter,” in ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2026, pp. 1933–1948. [Online]. Available: https://doi.org/10.1145/3779212.3790231
  18. [18]C. Hwang, W. Cui, Y. Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas, J. Jose, P. Ram, J. Chau, P. Cheng, F. Yang, M. Yang, and Y. Xiong, “Tutel: Adaptive Mixture-of-Experts at Scale,” in Conference on Machine Learning and Systems, 2023. [Online]. Available: https://proceedings.mlsys.org/paper_files/paper/2023/hash/5616d34cf8ff73942cfd5aa922842556-Abstract-mlsys2023.html
  19. [19]R. Hwang, J. Wei, S. Cao, C. Hwang, X. Tang, T. Cao, and M. Yang, “Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference,” in ACM/IEEE International Symposium on Computer Architecture, 2024, pp. 1018–1031. [Online]. Available: https://doi.org/10.1109/ISCA59077.2024.00078
  20. [20]A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de Las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M.-A. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao, T. Gervet, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, “Mixtral of Experts,” arXiv preprint arXiv:2401.04088, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2401.04088
  21. [21]K. Kamahori, T. Tang, Y. Gu, K. Zhu, and B. Kasicki, “Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models,” in International Conference on Learning Representations, 2025. [Online]. Available: https://openreview.net/forum?id=N5fVv6PZGzarxiv.org/abs/2402.07033
  22. [22]W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient Memory Management for Large Language Model Serving with PagedAttention,” in ACM Symposium on Operating Systems Principles, 2023, pp. 611–626. [Online]. Available: https://doi.org/10.1145/3600006.3613165
  23. [23]W. Lee, J. Lee, J. Seo, and J. Sim, “InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management,” in USENIX Symposium on Operating Systems Design and Implementation, 2024, pp. 155–172. [Online]. Available: https://www.usenix.org/conference/osdi24/presentation/leearxiv.org/abs/2406.19707
  24. [24]D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding,” in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/forum?id=qrwe7XHITmYb
  25. [25]Y. Leviathan, M. Kalman, and Y. Matias, “Fast Inference from Transformers via Speculative Decoding,” in International Conference on Machine Learning, 2023, pp. 19274–19286. [Online]. Available: https://proceedings.mlr.press/v202/leviathan23a.htmlarxiv.org/abs/2211.17192
  26. [26]Y. Li, F. Wei, C. Zhang, and H. Zhang, “EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty,” in International Conference on Machine Learning, 2024, pp. 28 935–28 948. [Online]. Available: https://proceedings.mlr.press/v235/li24bt.htmlarxiv.org/abs/2401.15077
  27. [27]Z. Li, L. Zheng, Y. Zhong, W. Liu, Y. Sheng, X. Jin, Y. Huang, Z. Chen, H. Zhang, J. E. Gonzalez, and I. Stoica, “AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving,” in USENIX Symposium on Operating Systems Design and Implementation, 2023, pp. 663–679. [Online]. Available: https://www.usenix.org/conference/osdi23/presentation/li-zhouhanarxiv.org/abs/2302.11665
  28. [28]C. Lin, Z. Han, C. Zhang, Y. Yang, F. Yang, C. Chen, and L. Qiu, “Q-Prot: Efficient Serving of LLM-based Applications with Semantic Variable,” in 18th USENIX Symposium on Operating Systems Design and Implementation. USENIX Association, 2024, pp. 929–945. [Online]. Available: https://www.usenix.org/conference/osdi24/presentation/lin-chaofan
  29. [29]Z. Lin, Y. Miao, Q. Zhang, F. Yang, Y. Zhu, C. Li, S. Maleki, X. Cao, N. Shang, Y. Yang, W. Xu, M. Yang, L. Zhang, and L. Zhou, “nnScaler: Constraint-Guided Parallelization Plan Generation for Deep Learning Training,” in USENIX Symposium on Operating Systems Design and Implementation, 2024, pp. 347–363. [Online]. Available: https://www.usenix.org/conference/osdi24/presentation/lin-zhiqi
  30. [30]Y. Liu, H. Li, Y. Cheng, S. Ray, Y. Huang, Q. Zhang, K. Du, J. Yao, S. Lu, G. Ananthanarayanan, M. Maire, H. Hoffmann, A. Holtzman, and J. Jiang, “CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving,” in ACM SIGCOMM Conference, 2024, pp. 38–56. [Online]. Available: https://doi.org/10.1145/3651890.3672274
  31. [31]Z. Liu, J. Yuan, H. Jin, S. H. Zhong, Z. Xu, V. Braverman, B. Chen, and X. Hu, “KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache,” in International Conference on Machine Learning, 2024, pp. 32 332–32 344. [Online]. Available: https://proceedings.mlr.press/v235/liu24bz.htmlDOI
  32. [32]K. Mahajan, A. Balasubramanian, A. Singhvi, S. Venkataraman, A. Akella, A. Phanishayee, and S. Chawla, “Themis: Fair and Efficient GPU Cluster Scheduling,” in USENIX Symposium on Networked Systems Design and Implementation, 2020, pp. 289–304. [Online]. Available: https://www.usenix.org/conference/nsdi20/presentation/mahajan
  33. [33]Z. Mei, W. Fu, K. Li, G. Wang, H. Zhang, and Y. Wu, “ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation,” in Conference on Machine Learning and Systems, 2025. [Online]. Available: https://openreview.net/forum?id=yLU1zRf95darxiv.org/abs/2406.14088
  34. [34]X. Miao, C. Shi, J. Duan, X. Xi, D. Lin, B. Cui, and Z. Jia, “SpotServe: Serving Generative Large Language Models on Preemptible Instances,” in ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2024, pp. 1112–1127. [Online]. Available: https://doi.org/10.1145/3620665.3640411
  35. [35]D. Narayanan, K. Santhanam, F. Kazhamiaka, A. Phanishayee, and M. Zaharia, “Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads,” in 14th USENIX Symposium on Operating Systems Design and Implementation. USENIX Association, 2020, pp. 481–498. [Online]. Available: https://www.usenix.org/conference/osdi20/presentation/narayanan-deepakarxiv.org/abs/2008.09213
  36. [36]X. Nie, X. Miao, Z. Wang, Z. Yang, J. Xue, L. Ma, G. Cao, and B. Cui, “FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement,” Proc. ACM Manag. Data, vol. 1, no. 1, pp. 110:1–110:19, 2023. [Online]. Available: https://doi.org/10.1145/3588964
  37. [37]H. Oh, K. Kim, J. Kim, S. Kim, J. Lee, D.-S. Chang, and J. Seo, “ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference,” in ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2024, pp. 369–384. [Online]. Available: https://doi.org/10.1145/3620665.3640383
  38. [38]L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions with human feedback,” in Advances in Neural Information Processing Systems, 2022. [Online]. Available: http://papers.nips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.htmlarxiv.org/abs/2203.02155
  39. [39]X. Pan, W. Lin, L. Zhang, S. Shi, Z. Tang, R. Wang, B. Li, and X. Chu, “FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models,” in ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2025, pp. 524–539. [Online]. Available: https://doi.org/10.1145/3669940.3707272
  40. [40]P. Patel, E. Choukse, C. Zhang, A. Shah, Í. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient Generative LLM Inference Using Phase Splitting,” in ACM/IEEE International Symposium on Computer Architecture, 2024, pp. 118–132. [Online]. Available: https://doi.org/10.1109/ISCA59777.2024.00019
  41. [41]Y. Peng, Y. Bao, Y. Chen, C. Wu, and C. Guo, “Optimus: an efficient dynamic resource scheduler for deep learning clusters,” in European Conference on Computer Systems, 2018, pp. 3:1–3:14. [Online]. Available: https://doi.org/10.1145/3190508.3190517
  42. [42]R. Prabhu, A. Nayar, J. Mohan, R. Ramjee, and A. Panwar, “vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention,” in ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2025, pp. 1133–1150. [Online]. Available: https://doi.org/10.1145/3669940.3707256
  43. [43]A. Qiao, S. K. Choe, S. J. Subramanya, W. Neiswanger, Q. Ho, H. Zhang, G. R. Ganger, and E. P. Xing, “Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning,” in USENIX Symposium on Operating Systems Design and Implementation, 2021. [Online]. Available: https://www.usenix.org/conference/osdi21/presentation/qiaoarxiv.org/abs/2008.12260
  44. [44]R. Qin, Z. Li, W. He, J. Cui, F. Ren, M. Zhang, Y. Wu, W. Zheng, and X. Xu, “Mooncake: Trading More Storage for Less Computation - A KVCache-centric Architecture for Serving LLM Chatbot,” in USENIX Conference on File and Storage Technologies, 2025, pp. 155–170. [Online]. Available: https://www.usenix.org/conference/fast25/presentation/qin
  45. [45]S. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He, “DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale,” in International Conference on Machine Learning, 2022, pp. 18 332–18 346. [Online]. Available: https://proceedings.mlr.press/v162/rajbhandari22a.htmlarxiv.org/abs/2201.05596
  46. [46]J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” arXiv preprint arXiv:1707.06347, 2017. [Online]. Available: https://arxiv.org/abs/1707.06347
  47. [47]Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. K. Li, Y. Wu, and D. Guo, “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models,” arXiv preprint arXiv:2402.03300, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2402.03300
  48. [48]N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean, “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer,” in International Conference on Learning Representations, 2017. [Online]. Available: https://openreview.net/forum?id=B1ckMDqlgarxiv.org/abs/1701.06538
  49. [49]G. Shen, Z. Wang, O. Dellalou, J. Zeng, Y. Dong, D. Egert, S. Sun, J. J. Zhang, S. Jain, A. Taghibakhshi, M. S. Ausin, A. Aithal, and O. Kuchaiev, “NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment,” arXiv preprint arXiv:2405.01481, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2405.01481
  50. [50]G. Sheng, Y. Tong, B. Wan, W. Zhang, C. Jia, X. Wu, Y. Wu, X. Li, C. Zhang, Y. Peng, H. Lin, X. Liu, and C. Wu, “Laminar: A Scalable Asynchronous RL Post-Training Framework,” in European Conference on Computer Systems, 2026, pp. 400–422. [Online]. Available: https://doi.org/10.1145/3767295.3803580
  51. [51]G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, Y. Peng, H. Lin, and C. Wu, “HybridFlow: A Flexible and Efficient RLHF Framework,” in European Conference on Computer Systems, 2025, pp. 1279–1297. [Online]. Available: https://doi.org/10.1145/3689031.3696075
  52. [52]N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano, “Learning to summarize with human feedback,” in Advances in Neural Information Processing Systems, 2020. [Online]. Available: https://proceedings.neurips.cc/paper/2020/hash/1f89885d556929e98d3ef9b86448f951-Abstract.htmlarxiv.org/abs/2009.01325
  53. [53]J. Stojkovic, C. Zhang, I. Goiri, J. Torrellas, and E. Choukse, “DynamoLM: Designing LLM Inference Clusters for Performance and Energy Efficiency,” in IEEE International Symposium on High-Performance Computer Architecture, 2025, pp. 1348–1362. [Online]. Available: https://doi.org/10.1109/HPCA61900.2025.00102
  54. [54]F. Strati, X. Ma, and A. Klimovic, “Orion: Interference-aware, Fine-grained GPU Sharing for ML Applications,” in European Conference on Computer Systems, 2024, pp. 1075–1092. [Online]. Available: https://doi.org/10.1145/3627703.3629578
  55. [55]B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y. Li, and W. Lin, “Lumnix: Dynamic Scheduling for Large Language Model Serving,” in USENIX Symposium on Operating Systems Design and Implementation, 2024, pp. 173–191. [Online]. Available: https://www.usenix.org/conference/osdi24/presentation/sun-biaoarxiv.org/abs/2406.03243
  56. [56]J. Tang, Y. Zhao, K. Zhu, G. Xiao, B. Kasikci, and S. Han, “QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference,” in International Conference on Machine Learning, 2024, pp. 47 901–47 911. [Online]. Available: https://proceedings.mlr.press/v235/tang24l.htmlarxiv.org/abs/2406.10774
  57. [57]Y. Tang, X. Li, F. Liu, W. Guo, H. Zhou, Y. Wang, K. Han, X. Yu, J. Li, H. Zang, F. Mi, X. Meng, Z. Liu, H. Chen, B. Zheng, C. Chen, Y. Yan, R. Tang, P. Qin, X. Chen, D. Tao, and Y. Wang, “Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsity,” arXiv preprint arXiv:2505.21411, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2505.21411
  58. [58]P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, K. H. Petersen, L. Larson, C. Carey, I. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors, “SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python,” Nature Methods, vol. 17, pp. 261–272, 2020. [Online]. Available: https://doi.org/10.1038/s41592-019-0686-2
  59. [59]J. Wang, Y. Wang, M. Han, and R. Chen, “Colocating ML Inference and Training with Fast GPU Memory Handover,” in USENIX Annual Technical Conference, 2025, pp. 1657–1675. [Online]. Available: https://www.usenix.org/conference/atc25/presentation/wang-jiali
  60. [60]W. Wang, S. Xiong, G. Chen, W. Gao, S. Guo, Y. He, J. Huang, J. Liu, Z. Li, X. Li, Z. Liu, H. Zhao, D. An, L. Cao, Q. Cao, W. Deng, F. Du, Y. Gu, J. Li, X. Li, M. Liu, Y. Luo, Z. Liu, Y. Wang, P. Wang, T. Wu, Y. Wu, Y. Zhao, S. Zhao, J. Yang, S. Yang, Y. Tan, H. Yi, X. Xu, Y. Yuan, X. Zhang, L. Qu, W. Su, W. Wang, J. Wang, and B. Zheng, “Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library,” arXiv preprint arXiv:2506.06122, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2506.06122
  61. [61]B. Wu, R. Zhu, Z. Zhang, P. Sun, X. Liu, and X. Jin, “dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving,” in 18th USENIX Symposium on Operating Systems Design and Implementation. USENIX Association, 2024, pp. 911–927. [Online]. Available: https://www.usenix.org/conference/osdi24/presentation/wu-bingyang
  62. [62]G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis, “Efficient Streaming Language Models with Attention Sinks,” in International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=NG7sS51zVFarxiv.org/abs/2309.17453
  63. [63]W. Xiao, R. Bhardwaj, R. Ramjee, M. Sivathanu, N. Kwatra, Z. Han, P. Patel, X. Peng, H. Zhao, Q. Zhang, F. Yang, and L. Zhou, “Gradual: Introspective Cluster Scheduling for Deep Learning,” in USENIX Symposium on Operating Systems Design and Implementation, 2018, pp. 595–610. [Online]. Available: https://www.usenix.org/conference/osdi18/presentation/xiao
  64. [64]W. Xiao, S. Ren, Y. Li, Y. Zhang, P. Hou, Z. Li, Y. Feng, W. Lin, and Y. Jia, “AntMan: Dynamic Scaling on GPU Clusters for Deep Learning,” in USENIX Symposium on Operating Systems Design and Implementation, 2020, pp. 533–548. [Online]. Available: https://www.usenix.org/conference/osdi20/presentation/xiao
  65. [65]Z. Yao, R. Y. Aminabadi, O. Ruwase, S. Rajbhandari, X. Wu, A. A. Wan, J. Rasley, M. Zhang, C. Li, C. Holmes, Z. Zhou, M. Wyatt, M. Smith, L. Kurilenko, H. Qin, M. Tanaka, S. Che, S. L. Song, and Y. He, “DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales,” arXiv preprint arXiv:2308.01320, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2308.01320
  66. [66]G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A Distributed Serving System for Transformer-Based Generative Models,” in USENIX Symposium on Operating Systems Design and Implementation, 2022, pp. 521–538. [Online]. Available: https://www.usenix.org/conference/osdi22/presentation/yu
  67. [67]P. Yu and M. Chowdhury, “Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applications,” in Proceedings of Machine Learning and Systems, vol. 2, 2020. [Online]. Available: https://proceedings.mlsys.org/paper_files/paper/2020/hash/d9cd83bc91b8c36a0c7c0fcca59228f2-Abstract.htmlarxiv.org/abs/1902.04610
  68. [68]Y. Yuan, A. Gupta, J. Li, S. Dash, F. Wang, and M. Zhang, “X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms,” in International Conference for High Performance Computing, Networking, Storage and Analysis, 2025, pp. 1315–1331. [Online]. Available: https://doi.org/10.1145/3712285.3759886
  69. [69]S. Yun, K. Kyung, J. Cho, J. Choi, J. Kim, B. Kim, S. Lee, K. Sohn, and J. H. Ahn, “Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batching,” in IEEE/ACM International Symposium on Microarchitecture, 2024, pp. 1429–1443. [Online]. Available: https://doi.org/10.1109/MICRO61859.2024.00105
  70. [70]Y. Zeng, C. Huang, Y. Mei, L. Zhang, T. Su, W. Ye, W. Shi, and S. Wang, “EfficientMoE: Optimizing Mixture-of-Experts Model Training With Adaptive Load Balance,” IEEE Trans. Parallel Distributed Syst., vol. 36, no. 4, pp. 677–688, 2025. [Online]. Available: https://doi.org/10.1109/TPDS.2025.3539297
  71. [71]M. Zhai, J. He, Z. Ma, Z. Zong, R. Zhang, and J. Zhai, “SmartMoE: Efficiently Training Sparsely-Activated Models through Combining Offline and Online Parallelization,” in USENIX Annual Technical Conference, 2023, pp. 961–975. [Online]. Available: https://www.usenix.org/conference/atc23/presentation/zhai
  72. [72]Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. W. Barrett, Z. Wang, and B. Chen, “H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models,” in Advances in Neural Information Processing Systems, 2023. [Online]. Available: http://papers.nips.cc/paper_files/paper/2023/hash/6ceefa7b15572587b78efcceb2827f8-Abstract-Conference.htmlarxiv.org/abs/2306.14048
  73. [73]Y. Zhao, X. Liu, S. Liu, X. Li, Y. Zhu, G. Huang, X. Liu, and X. Jin, “MuxFlow: Efficient and Safe GPU Sharing in Large-Scale Production Deep Learning Clusters,” arXiv preprint arXiv:2303.13803, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2303.13803
  74. [74]Y. Zhao, D. Wu, and J. Wang, “ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching,” in ACM/IEEE International Symposium on Computer Architecture, 2024, pp. 1005–1017. [Online]. Available: https://doi.org/10.1109/ISCA59077.2024.00077
  75. [75]L. Zheng, Z. Li, H. Zhang, Y. Zhuang, Z. Chen, Y. Huang, Y. Wang, Y. Xu, D. Zhuo, E. P. Xing, J. E. Gonzalez, and I. Stoica, “Alpa: Automatic Inter- and Intra-Operator Parallelism for Distributed Deep Learning,” in USENIX Symposium on Operating Systems Design and Implementation, 2022, pp. 559–578. [Online]. Available: https://www.usenix.org/conference/osdi22/presentation/zheng-liamminarxiv.org/abs/2201.12023
  76. [76]L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. W. Barrett, and Y. Sheng, “SGLang: Efficient Execution of Structured Language Model Programs,” in Advances in Neural Information Processing Systems, 2024. [Online]. Available: http://papers.nips.cc/paper_files/paper/2024/hash/724be4472168f31ba1c9ac630f15dec8-Abstract-Conference.htmlarxiv.org/abs/2312.07104
  77. [77]P. Zheng, R. Pan, T. Khan, S. Venkataraman, and A. Akella, “Shockwave: Fair and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learning,” in USENIX Symposium on Networked Systems Design and Implementation, 2023, pp. 703–723. [Online]. Available: https://www.usenix.org/conference/nsdi23/presentation/zhengarxiv.org/abs/2210.00093
  78. [78]Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang, “DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving,” in USENIX Symposium on Operating Systems Design and Implementation, 2024, pp. 193–210. [Online]. Available: https://www.usenix.org/conference/osdi24/presentation/zhong-yinminarxiv.org/abs/2401.09670
  79. [79]Y. Zhong, Z. Zhang, B. Wu, S. Liu, Y. Chen, C. Wan, H. Hu, L. Xia, R. Ming, Y. Zhu, and X. Jin, “Optimization RLHF Training for Large Language Models with Stage Fusion,” in USENIX Symposium on Networked Systems Design and Implementation, 2025, pp. 489–503. [Online]. Available: https://www.usenix.org/conference/nsdi25/presentation/zhongarxiv.org/abs/2409.13221
  80. [80]D. M. Ziegler, L. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. F. Christiano, and G. Irving, “Fine-Tuning Language Models from Human Preferences,” arXiv preprint arXiv:1909.08593, 2019. [Online]. Available: http://arxiv.org/abs/1909.08593

Paper details

vlnc:W_64eukt9q
Version
v2 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 17

downloads · 2

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Codex

View statement
Looking for related papers…
Contents