betaFeedback

WorksComputer Science

LEAP: Efficient LLM Inference Scheduling with Generalizable Length Prediction

Abstract

Length-aware scheduling reduces request latency in LLM serving by prioritizing shorter requests. However, despite their large model sizes, length predictors can suffer degraded ranking performance under domain and task shifts, and adapting them can require costly retraining. Their prediction overhead can further offset the benefits of scheduling. To improve the generalization and efficiency of length-aware scheduling, we propose LEAP, a length prediction system with fast online adaptation and asynchronous CPU execution. LEAP reuses the serving model's prefill hidden states, which contain information about the output length, and extracts this signal through a linear readout. To adapt to substantial domain and task shifts, it updates the readout at test time through recursive least squares, using completed requests as supervision without gradient computation or historical-data replay. Prediction and updates run asynchronously on CPU, while prediction and initial batch ordering overlap the remaining GPU prefill. Across the evaluated cross-domain transfers, LEAP retains strong ranking performance, and online adaptation improves Spearman correlation by 64.7% under a larger domain shift. Serving experiments show that LEAP reduces average per-token latency by 15.9–40.8% compared with the evaluated length predictors, while achieving 1.9% higher throughput with 30,569×30{,}569\times fewer predictor parameters than the strongest evaluated baseline.

Introduction

Length-aware scheduling plays a key role in LLM serving: it prevents long requests from blocking multiple short ones, reducing average response time [9, 11, 18] (Fu et al. 2024). Its effectiveness depends on length predictions that distinguish requests by their remaining work. Yet deployment brings new tasks, prompt styles, and user populations. A predictor that performs well on its source domain may no longer provide useful priorities when these change.

This concern appears in our cross-domain evaluations (§ 4.2). On the same target requests, several predictors lose substantial ranking quality when trained on a different domain. Restoring that quality through repeated offline retraining adds data collection, optimization, and redeployment costs. Moreover, prediction itself also consumes resources: prompt-based methods run separate encoders [18, 9] (Fu et al. 2024), while hidden-state methods reuse the serving model’s representations through neural prediction heads [11, 16, 17, 10]. A GPU-based predictor spends resources that scheduling is intended to use more effectively; a slow CPU predictor can leave the scheduler waiting. A practical design must therefore consider how well predictions transfer, how they can be updated, and what they cost to deliver.

For scheduling, sufficient prediction differs from perfect prediction. Requests can be prioritized effectively even when their exact lengths remain uncertain. This motivates us to favor a lightweight predictor: higher accuracy does not improve serving efficiency when the additional prediction cost outweighs its scheduling benefits. We seek useful length information in features that the serving model already computes, avoiding the cost of a heavier predictor. Prior work uses hidden states for generation control and future-token prediction [5, 1]; hidden-state length predictors more directly demonstrate signals related to output length [11, 17]. This motivates reusing the representation produced during prefill, which is conditioned on both the prompt and the model that will generate the answer. We extract its length signal through a linear readout, whose small parameter set supports fast CPU inference and recursive updates.

To improve the generalization and efficiency of length-aware scheduling, we propose LEAP1, a length prediction system with fast online adaptation and asynchronous CPU execution. Its readout uses the final prompt token’s hidden state from one late layer to estimate output length for decode scheduling. When substantial domain or task shifts degrade the predictions, LEAP adapts the readout through test-time training with recursive least squares (RLS). Each completed request supplies its observed output length as supervision. Starting from the source-trained readout, RLS incorporates this feedback incrementally without backpropagation or replaying completed requests (§ 3.3).

What may be overlooked is the latency introduced by feature transfer, prediction, and score publication. This overhead can offset the benefits of more accurate predictions. To keep predictor computation off the GPU, LEAP runs prediction and updates asynchronously on CPU, allowing GPU decoding to continue using published scores. To reduce prediction delay, it overlaps CPU prediction and initial batch ordering with the remaining GPU prefill. In practice, both can finish before GPU prefill completes, as we verify in § 5.4.

Together, these components support decode scheduling as domains and tasks change. First, LEAP estimates output length from a single prefill hidden state. Second, it adapts the readout using completion feedback. Finally, it publishes predictions asynchronously for scheduling while GPU inference continues. Besides scheduling, our methodology may also benefit other use cases that involve output lengths, such as output length control through intervention and KV cache management.

Our contributions are as follows:

  • Reusing prefill signals for scheduling. We reuse a single prefill hidden state to predict output length without a separate prompt encoder. The linear readout uses square-root target regression to balance tail compression and the separation of long requests (§ 3.2).

  • Adaptation through test-time training. We adapt the source-trained readout to domain and task shifts using completion-supervised RLS updates, without gradient computation or historical-data replay (§ 3.3).

  • Asynchronous execution on CPU. We integrate CPU prediction, online updates, and score publication into a vLLM serving path. Prediction and initial batch ordering overlap the remaining GPU prefill (§ 3.4).

  • Experimental results. The frozen readout retains stronger ranking performance than the evaluated baselines across domain transfers. Online adaptation substantially improves ranking under larger language shifts. Serving experiments demonstrate measurable per-token latency improvements with significantly lower predictor parameter counts.

Motivation: Rethinking Length Prediction for LLM Scheduling

Q1: Does a length predictor generalize to new domains and tasks?

A predictor trained on one dataset may encounter different tasks, prompt styles, and user populations after deployment. These shifts alter the relationship between prompts and output length, so a fixed-distribution test does not establish the robustness of its request rankings when this relationship changes.

Our cross-domain evaluation (Figure 5) shows that frozen predictors can lose substantial ranking quality under domain shifts. Recovering this quality through offline retraining is costly: it requires new data collection, model optimization, and redeployment. A practical design therefore needs two properties: baseline generalization to new domains, and an affordable mechanism to adapt when that generalization proves insufficient. Completed requests naturally provide feedback in the form of observed output lengths, offering a signal for adaptation without additional annotation effort.

Figure 5. Ranking after LMSYS transfer on Qwen3-8B. Crosshatched extensions show RLS changes to LEAP.

Takeaway 1: Look beyond the source domain

Length predictors should retain useful rankings across domains and tasks and support affordable updates when those rankings degrade.

Q2: Does scheduling really need precise length predictions?

With iteration-level batching, LLM serving engines share GPU compute across requests [6]. After prefill builds the KV cache, decoding generates tokens sequentially. When eligible requests exceed the available decode slots, priorities determine which requests advance in that iteration. Output length provides a proxy for decoding work, although batch composition and context length also matter. Numerical length estimates remain useful: countdown priorities subtract generated tokens from the predicted total, and KV cache planning uses estimates of future sequence length.

For request selection, length estimates can support effective priorities even when individual predictions contain errors. Distinguishing a 50-token request from a 500-token request may matter more than refining an estimate from 53 to 50 tokens. The value of further accuracy gains depends on how they change scheduling decisions and what they cost to compute. This motivates a lightweight prediction head that supports useful countdown priorities without excessive prediction overhead. We assess its prediction quality together with the serving benefits and costs it produces.

Takeaway 2: Predict for scheduling

Numerical length estimates support remaining-work priorities, but lower prediction error alone does not establish better serving performance. Further accuracy gains should be evaluated against their scheduling benefits and execution costs.

Q3: Why encode the prompt twice?

Many length predictors run a separate text encoder over the prompt [9, 18] (Fu et al. 2024). For decode scheduling, however, the serving model has already processed that prompt during prefill. Its hidden state reflects both the input text and the model that will generate the answer. This raises the possibility of reading the length signal from the representation already used for generation.

Representation studies support this possibility: low-capacity probes recover entity properties and relations from contextual states (Li et al., 2021), while circuit tracing and Jacobian-lens analyses expose intermediate computations and plans for future output [8, 3].

Prior work also demonstrates length prediction from hidden states [11, 16, 17, 10]. Our design prioritizes obtaining useful scheduling scores with minimal feature processing and predictor computation.

Takeaway 3: Reuse the representation

Prefill representations already contain useful length signals. A separate prompt encoder is unnecessary for post-prefill scheduling.

Q4: Why spend GPU resources to save GPU resources?

Length-aware scheduling aims to make better use of limited GPU resources. Yet a GPU-based length predictor consumes memory and compute to decide how those same resources should be allocated. Its cost can offset the benefit of the scheduling decisions it enables.

While GPU decoding is busy, host CPUs may have spare capacity between tokenization, routing, and bookkeeping tasks. Moving prediction to the CPU can use this capacity, but a slow predictor leaves the scheduler waiting or acting on stale information. The relevant cost therefore includes feature transfer and score publication as well as predictor computation.

Takeaway 4: Make CPU prediction practical

CPU prediction and initial batch ordering can finish before GPU prefill completes § 5.4, keeping predictor computation off the GPU.

Method

Overview

LEAP has three core components:

  1. Linear readout § 3.2: A lightweight head that extracts length signals from a single prefill hidden state using square-root target regression.

  2. Online adaptation § 3.3: Test-time updates via recursive least squares when domain or task shifts degrade prediction quality.

  1. Asynchronous CPU execution (§ 3.4): Prediction and initial batch ordering overlap the remaining GPU prefill, keeping computation off the GPU. Figure 1 illustrates the request flow. When a substantial domain or task shift weakens the frozen readout, LEAP adapts by updating the linear coefficients from completion feedback, without retraining the serving model or modifying feature standardization.

Figure 1. LEAP: CPU-asynchronous prediction, scheduling, and completion-driven adaptation.

Mapped-length linear readout

We reuse hidden states (Q3) to predict lengths for request ordering (Q2). Let hih_i be the final prompt token’s hidden state at a selected late, nonfinal layer, and xix_i its standardized feature. The coordinatewise mean and scale are fitted on source-training features and remain frozen during serving.

We use Ridge regression to predict a transformed length target g(L)g(L) from xx. Given nn source training pairs (xi,yi)(x_i,y_i), where yiy_i is the target length, the base predictor solves

(w0,b0)=arg⁡min⁡w,b∑i=1n(wTxi+b−g(yi))2+λ0∥w∥22(1)(w_0,b_0)=\underset{w,b}{\arg\min}\sum_{i=1}^{n}(w^{\mathsf{T}}x_i+b-g(y_i))^2+\lambda_0\lVert w\rVert_2^2 \tag*{(1)}

Here λ0>0\lambda_0>0 regularizes the weights while the intercept remains unpenalized.

The linear readout z0(x)=w0Tx+b0z_0(x)=w_0^{\mathsf{T}}x+b_0 produces a value on the transformed scale. For an output budget MM, we clip this value to the valid length range and apply the inverse mapping:

L^0(x)=g−1(clip⁡(z0(x),g(1),g(M))).(2)\widehat{L}_0(x)=g^{-1}\left(\operatorname{clip}(z_0(x),g(1),g(M))\right). \tag*{(2)}

At deployment, the readout uses folded coefficients w‾0,b‾0\overline{w}_0,\overline{b}_0 to operate directly on hh, requiring one dot product followed by clipping and the inverse mapping.

Why square-root mapping? The target mapping controls which length errors receive more weight during training. On the original scale, squared loss gives extreme lengths disproportionate influence. A logarithmic mapping compresses the tail, but it also penalizes a fixed token error much less on a long request than on a short one. A square-root mapping weakens this dependence on length. For an error ee small relative to length LL:

(log⁡(L+e)−log⁡L)2≃e2L2(3)\left(\log(L+e)-\log L\right)^2 \simeq\frac{e^2}{L^2} \tag*{(3)}
(L+e−L)2≃e24L(4)\left(\sqrt{L+e}-\sqrt{L}\right)^2 \simeq\frac{e^2}{4L} \tag*{(4)}

Thus, the weight on a fixed absolute error decreases as 1/L21/L^2 under the logarithm and as 1/L1/L under the square root. The square root retains more sensitivity to differences between long requests while still reducing tail dominance.

We use g(L)=Lg(L)=\sqrt{L} as the default compromise between tail compression and long-request separation. The readout is trained on square-root targets, yielding

L^0(x)=[clip⁡(w‾0Tx+b‾0,1,M)]2.(5)\widehat{L}_0(x)=\left[\operatorname{clip}\left(\overline{w}_0^{\mathsf{T}}x+\overline{b}_0,1,\sqrt{M}\right)\right]^2. \tag*{(5)}

Clipping before squaring prevents large negative coordinates from becoming large positive length estimates. Figure 2 illustrates the readout and length conversion for square-root targets. The choice of map reflects the prediction task and constraints. Log targets prioritize short-output fidelity, while square root compresses long-output differences less aggressively and admits a simple inverse. Alternatives such as piecewise or power-law maps could be explored; with cached training features, switching maps requires only refitting the small head.

Frozen linear readout with square-root length mapping

Figure 2. Frozen linear readout with square-root length mapping.

Online adaptation from completion feedback

Source-initialized RLS and asynchronous weight publication

Figure 3. Source-initialized RLS and asynchronous weight publication. Lengths are illustrative.

Incorporating completion feedback. To address domain and task shifts (Q1), we update the readout from completion feedback. Hidden states provide model-conditioned length signals, while stochastic decoding can produce different lengths for the same prompt ([17]). Source-generation labels train the readout on these length statistics; actual completion labels give RLS a mechanism for recalibrating the relationship under the deployed generation policy.

Each completed request supplies its cached prefill feature u=[x;1]∈Rd+1u=[x;1]\in\mathbb{R}^{d+1} and mapped-length label g(L)=Lg(L)=\sqrt{L}. We fit completion feedback while anchoring the coefficients to the deployed source head, keeping source standardization and the target mapping fixed. Recursive least squares (RLS) incorporates new labels incrementally. Starting from the source head θ0=[w0,b0]\theta_0=[w_0,b_0], we initialize the inverse source-feature precision:

A0=UsTUs+λpD,P0=A0−1,(6)A_0=U_s^{\mathsf{T}}U_s+\lambda_pD,\qquad P_0=A_0^{-1}, \tag*{(6)}

where UsU_s contains source-training features, λp>0\lambda_p>0 sets the prior precision, and D=diag⁡(1,…,1,0)D=\operatorname{diag}(1,\ldots,1,0) leaves the intercept unpenalized. For the incorporated completions Ft\mathcal{F}_t, the objective is

θt=arg⁡min⁡θ  (θ−θ0)⊤A0(θ−θ0)+∑j∈Ft(uj⊤θ−g(Lj))2.(7)\theta_t = \underset{\theta}{\arg\min}\;(\theta-\theta_0)^\top A_0(\theta-\theta_0) + \sum_{j\in\mathcal{F}_t}(u_j^\top\theta-g(L_j))^2. \tag*{(7)}

The source-feature precision controls how strongly feedback changes the deployed coefficients. RLS minimizes this objective incrementally; when the source head also satisfies the normal equations defined by A0A_0, it is equivalent to joint source-plus-feedback Ridge refitting.

Incremental updates. Once features and observed lengths have been matched for qq completed requests, they form a batch with feature matrix UU and mapped targets yy. The Woodbury identity gives

V=PU⊤,G=V(Iq+UV⊤)−1,(8)V = PU^\top,\qquad G = V(I_q+UV^\top)^{-1}, \tag*{(8)}
θ+=θ+G(y−Uθ),(9)\theta^+ = \theta+ G(y-U\theta), \tag*{(9)}
P+=P−GV⊤.(10)P^+ = P-GV^\top. \tag*{(10)}

The gain GG maps the new prediction residual to a coefficient correction, while (θ,P)(\theta,P) retains the accumulated information without replaying completed features or computing gradients. Each batch solves a q×qq\times q system and costs O(qd2+q2d+q3)O(qd^2+q^2d+q^3) arithmetic. The inverse precision requires O(d2)O(d^2) CPU state; caching features for NN active requests adds O(Nd)O(Nd) space.

Why the recursive update is exact. Let UfU_f stack the incorporated feedback features and let yfy_f contain their mapped lengths. Expanding Equation (7) gives the normal equations

At=A0+Uf⊤Uf,ct=A0θ0+Uf⊤yf,θt=At−1ct.(11)\begin{aligned} A_t &= A_0+U_f^\top U_f,\\ c_t &= A_0\theta_0+U_f^\top y_f,\qquad\theta_t=A_t^{-1}c_t. \tag*{(11)} \end{aligned}

The regularized source precision is positive definite, so the minimizer is unique. Suppose P=A−1P=A^{-1} and θ=Pc\theta=Pc before a new batch. Adding that batch gives A+=A+U⊤UA^+=A+U^\top U and c+=c+U⊤yc^+=c+U^\top y. For S=Iq+UPU⊤S=I_q+UPU^\top, Woodbury yields

P+=P−PU⊤S−1UP,P+U⊤=G.(12)P^+=P-PU^\top S^{-1}UP,\qquad P^+U^\top=G. \tag*{(12)}

Consequently,

θ+=P+(c+U⊤y)=(P−GUP)c+Gy=θ+G(y−Uθ).(13)\begin{aligned} \theta^+ &= P^+(c+U^\top y)\\ &=(P-GUP)c+Gy=\theta+G(y-U\theta). \tag*{(13)} \end{aligned}

The initialization satisfies these identities, and each batch preserves them. Thus Equations (8)–(10) solve the anchored objective for every incorporated feedback set in exact arithmetic. Batch size controls update timing; the solution for a fixed feedback set is unchanged. Joint refitting on the mapped source labels ysy_s additionally requires A0θ0=U⊤ysA_0\theta_0=U^\top y_s; the anchored guarantee does not require that consistency condition.

CPU execution. A separate CPU thread updates (θ,P)(\theta,P) and hands coefficient snapshots to the prediction worker, which rescores cached live features:

zt(x)=u⊤θt.(14)z_t(x)=u^\top\theta_t. \tag*{(14)}

Equation (2) converts each score back to a token-length estimate. The source head supplies initial predictions; published RLS versions revise them as feedback arrives. This design keeps matrix updates off the initial-readout queue and reuses prefill features without another GPU forward pass. The scheduler continues using published estimates while an update is pending (§ 3.4). The prototype incorporates eight completions per update. Only completed requests supply labels, and pending batches do not change the published predictor. At d=4096d=4096, the inverse precision occupies about 64 MiB in FP32; the small update system is solved in FP64. Asynchrony removes an update dependency from prediction but does not eliminate CPU or memory-bandwidth costs.

Asynchronous prediction and scheduling

To address the resource tradeoff in Q4, we execute prediction asynchronously on the CPU alongside GPU inference.

Figure 4. CPU feature transfer, prediction, and publication overlap with the remaining GPU prefill. Timing is schematic.

At the selected layer, we capture the final prompt token’s hidden state and transfer it to the CPU while the GPU continues prefill. Once the transfer completes, a CPU worker applies the current predictor and publishes the length estimate. After the scheduler consumes the published estimate, subsequent eligible decode decisions use the revised order. The CPU worker can finish prediction and initial batch ordering before GPU prefill completes, as verified by the GPU readiness probe in the prefill-overlap experiment. The engine applies the prepared order at its next eligible decode decision. The feature is captured and transferred only once per request.

At decode step tt, request ii has generated ai(t)a_i(t) tokens and has predicted total length L^i(t)\widehat{L}_i(t). Its remaining-length score is

si(t)=max⁡{1,L^i(t)−ai(t)}.(15)s_i(t)=\max\{1,\widehat{L}_i(t)-a_i(t)\}. \tag*{(15)}

When eligible KV-resident requests exceed decode capacity, LEAP selects the advancing subset by this score; deferred requests retain their KV state. The engine controls admission, prefill, memory allocation, and fairness overrides.

All methods within each comparison share waiting protection: a 128-step guard in the live engine and TIE’s exponential Aging rule for cached replays and the LMSYS load control Zheng et al., [18] (Appendix § 11.§ 2).

Countdown updates the remaining-length score at each decoding step; RLS revises the predicted total length after a batch of completions. The scheduler consumes newly published estimates when available, while countdown continues without waiting for the update thread. Until the first prediction is ready, requests use their output budget as the total-length estimate. If updates lag, existing estimates continue counting down.

Published scores carry request identity and predictor-version metadata. The scheduler accepts only results for matching live requests, preventing stale results from changing unrelated priorities. Score-delivery measurements therefore follow an estimate through publication to scheduler consumption, capturing the delay beyond the readout itself.

Completion feedback is matched by request identity and incorporated in full batches of eight. A final incomplete batch is not fitted, and each incorporated label is the observed served length under the request’s output budget.

Evaluation

Experimental Setup

Datasets and models. We evaluate Qwen3-8B, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.3 on instruction-following requests from Alpaca Taori et al., [15] and conversations from ShareGPT (ShareGPT Team, n.d.) and LMSYS-Chat-1M Zheng et al., [19]. Frozen LMSYS predictors transfer to Alpaca, MATH Hendrycks et al., [4], and non-English Aya Singh et al., [13] requests to test task and language shifts. Methods share request splits, with prompt and conversation-parent overlap excluded between training and test. The first generated output defines each test label.

Baselines. We compare against uncertainty-aware TIE Zheng et al., [18], rank-based LTR (Fu et al., 2024), proxy predictor SSJF Qiu et al., [9], and hidden-state predictors TRAIL Shahout et al., [11] and EGTP Xie et al., [17]. FCFS supplies the no-prediction scheduling control. Initial-quality comparisons use TIE’s bounded expectation and TRAIL’s prefill head; serving comparisons include TRAIL’s arrival predictor, initial and online heads, and probability filter. SSJF uses source-training target normalization, inverted at inference.

Training. The default Qwen readout uses the final prompt token at block 30, square-root targets, and frozen source-training standardization. LEAP fits its head on 2,880 training requests. Baseline fitting retains each author’s objective and checkpoint-selection rule. LEAP and TIE use repeated-generation targets, whereas LTR, SSJF, and EGTP use first-rollout labels. Readout regularization minimizes token MAE on source validation data, without target-domain tuning. Prediction comparisons use one training/generation seed unless noted.

Serving and metrics. Serving runs on one A800 80GB PCIe GPU and eight Xeon Gold 6348 logical CPUs using vLLM 0.8.5.post1 in eager mode. LEAP executes on CPU; deep predictors share the GPU. Full-cost comparisons replay common requests and recorded token trajectories under one scheduling policy, so only the predictor path differs. The engine uses FIFO prefill, 512 resident requests, 64 decode slots, and a 128-step waiting guard; requests without an accepted score fall back to FCFS. Test labels are first-rollout lengths capped at 2,048 tokens. We report MAE and Spearman ρ\rho for prediction (LTR’s ordinal scores have no token MAE), and mean/P90 per-token latency (PTL), time to first token (TTFT), and throughput for serving. Latency starts at planned arrival and includes prediction, publication, queueing, and execution; PTL divides completion latency by actual output tokens.

Prediction Quality and Domain Transfer

Prediction from prefill representations. Table 1 compares initial length estimates across three models and three datasets.

DatasetLEAP (ours)TIETRAILLTRSSJFEGTP
MAE ↓\downarrowρ↑\rho\uparrowMAE ↓\downarrowρ↑\rho\uparrowMAE ↓\downarrowρ↑\rho\uparrowρ↑\rho\uparrowMAE ↓\downarrowρ↑\rho\uparrowMAE ↓\downarrowρ↑\rho\uparrow
Qwen3-8B
Alpaca71.50.955161.80.875114.00.8870.863139.60.847241.90.482
ShareGPT171.60.900316.10.675278.80.7190.733280.40.726428.70.142
LMSYS127.50.932193.70.848213.10.7910.817210.20.811395.50.448
Llama-3.1-8B-Instruct
Alpaca63.00.942130.70.82167.90.9330.86298.10.865202.20.462
ShareGPT145.70.850214.30.672160.20.8080.691196.00.699276.20.101
LMSYS105.00.922177.30.768135.10.8410.782149.60.785287.80.268
Mistral-7B-Instruct-v0.3
Alpaca65.40.931105.50.83381.50.8840.83390.50.852145.20.550
ShareGPT143.90.784193.40.618173.90.6670.640178.30.646243.10.243
LMSYS106.90.896163.50.792139.70.8080.805131.90.808198.90.500

Table 1. Initial length prediction across models and datasets. MAE is in tokens; ρ\rho is Spearman correlation. Bold marks the best value.

Analysis. LEAP has the lowest MAE and highest Spearman ρ\rho in all nine settings, reducing MAE by 7–39% relative to the best baseline in each setting, without a separate prompt encoder.

Answer to Q3: Reusing a single prefill hidden state with a linear readout avoids a second prompt encoder and yields lower MAE and higher Spearman correlation than the evaluated independent-encoder predictors in all nine settings (Table 1).

Static versus dynamic prediction. Refreshing estimates during decoding can lower error, but at a runtime cost. Table 2 compares static countdown (-S) with hidden-state refreshes during decoding (-D). MAE is measured at matched live token positions on 1,000 test requests per domain; Δ\DeltaResp. comes from paired serving runs on 128 held-out requests per domain that share initial estimates and token trajectories, with the full prediction pipeline included.

PredictorShareGPT MAEShareGPT Δ\DeltaResp. (%)LMSYS MAELMSYS Δ\DeltaResp. (%)
TRAIL-S244.67—170.69—
TRAIL-D167.01+12.89108.84+9.16
LEAP-S154.60—111.92—
LEAP-D141.96+9.42103.82+8.37

Table 2. Prediction accuracy and serving cost. Green rows use static countdown (-S); orange rows refresh hidden states during decoding (-D). Δ\DeltaResp. is relative to the static version.

For LEAP, static and dynamic heads use repeated-generation and round-0 remaining-length targets, respectively, so this comparison does not isolate feature refresh alone. RLS parameter adaptation is separate from these decode-time refreshes.

Answer to Q2: Lower prediction error does not necessarily translate into lower serving latency. In our paired controls, dynamic prediction improves MAE but increases mean response time, supporting a lightweight static predictor that preserves useful ordering at low execution cost.

Testing frozen cross-domain generalization. We transfer frozen LMSYS predictors to Alpaca, MATH, and Aya, and measure how much RLS feedback recovers (Figure 5). Alpaca feedback requests are disjoint from its test set; on MATH and Aya, each request is predicted before its label enters RLS.

Analysis. The frozen readout transfers well to Alpaca and MATH, but the language shift to Aya degrades its ranking. On Aya, RLS raises Spearman ρ\rho from 0.498 to 0.820, a 64.7% relative improvement, by updating only the readout, without touching the backbone.

Answer to Q1: Generalization is useful but not uniform: frozen LEAP retains useful rankings on Alpaca and MATH, while completion feedback substantially restores ranking quality on Aya (Figure 5).

Serving on Similar Workloads

Comparing predictors with full deployment costs. Table 3 reports ShareGPT serving with all prediction costs included. Requests arrive at 2.65 requests/s and replay the same output trajectories under every predictor.

MethodMean PTLP90 PTLMean TTFTP90 TTFTTok/s
FCFS250.97359.9655.1074.851094.03
TRAIL107.42189.3267.1890.28944.71
TIE78.48152.3366.5085.221089.84
SSJF76.07154.5062.2780.341108.46
LTR75.62151.9560.4080.171120.88
LEAP (ours)63.58127.4755.2773.431141.93

Table 3. Serving performance on ShareGPT under a shared selector. PTL: ms/token; TTFT: ms. Rows are ordered from higher to lower mean PTL. Our results are bold.

Analysis. LEAP lowers mean PTL by 15.9–40.8% across the evaluated predictors: 15.9% relative to LTR, 19.0% to TIE, and 40.8% to TRAIL. LTR is the strongest baseline here by both mean PTL and throughput; LEAP also raises throughput by 1.9% over LTR, while keeping TTFT close to FCFS. Long requests still wait longer than under FCFS, the usual cost of favoring short work. Results are single runs on a shared host.

Prediction cost under rising load. We replay the same 1,024 held-out LMSYS requests at 20–100 requests/s (Figure 6). All methods share request identities, token trajectories, and one post-prefill scheduling rule, so they differ only in the length estimate and where it is computed. LEAP predicts on CPU, while TIE and LTR run on the serving GPU; client latency includes this prediction cost. Protocol details are in Appendix § 12.§ 2.

Figure 6. LMSYS serving latency versus offered rate under shared post-prefill scheduling. All four methods serve the same 1,024 requests and token trajectories.

Analysis. LEAP has the lowest value in all four panels at every rate, and its mean per-token latency stays at about half that of the better GPU predictor as load rises. All three predictors stay far below FCFS, so length-aware ordering accounts for most of the gain over FCFS. The remaining gap combines LEAP’s more accurate ranking (Table 1) with keeping prediction off the serving GPU; this single-run experiment does not separate the two.

Scheduling under Larger Domain Shifts

Scheduling after domain transfer. We replay Aya and MATH streams in simulation with cached predictions from frozen LMSYS models. All methods share the remaining-length priority, the Aging rule in Equation (16), and the same admission and execution models. Table 4 reports mean response time at relative offered loads 1.2 and 1.5. RLS sees each label only after the simulated completion. Predictor runtime costs are excluded here; § 4.3 covers them.

MethodAya 1.2Aya 1.5MATH 1.2MATH 1.5
SSJF154.83253.2980.30126.83
EGTP136.62223.0996.54157.83
FCFS138.40226.7191.66148.90
TRAIL125.02201.6777.52123.67
LTR123.32197.2875.12118.93
TIE119.56190.0675.05118.34
LEAP (ours)115.18180.5967.39100.94
LEAP+RLS (ours)108.07161.1864.8394.36

Table 4. Simulated mean response time after domain transfer with matched Aging (s; lower is better). Bold: ours.

Analysis. With protection held fixed, latency differences come from ranking alone, and they track the ranking gaps in Figure 5. Frozen LEAP’s advantage over TIE is largest on MATH (up to 14.7%), and all gaps widen at the higher load, where longer queues make ordering matter more. RLS helps most on Aya, where the frozen ranking is weakest. Better ranking does not guarantee better tails: RLS raises P95 on MATH, and LEAP, LEAP+RLS, and TIE all keep a higher P95 than FCFS.

Backbone Scale

Testing backbone scale. We fit the same square-root readout on dense Qwen3 models from 0.6B to 32B, using single-rollout labels and one fixed layer per model (Table 5). Each checkpoint is evaluated on its own outputs.

SizeShareGPT MAEShareGPT ρ\rhoLMSYS MAELMSYS ρ\rho
0.6B151.380.83694.810.910
1.7B206.260.856144.070.921
4B184.840.879124.810.932
8B173.260.901129.140.936
14B160.310.909116.130.936
32B155.300.915116.680.938

Table 5. Length prediction across Qwen3 model sizes.

Analysis. Spearman ρ\rho stays above 0.83 at every size, so the readout ports across checkpoints. Long-output ordering is unreliable in the 0.6B model, so a high aggregate ρ\rho does not ensure uniform quality across length groups. Because outputs differ per checkpoint, this does not isolate the effect of size.

Target-map Ablation

Testing short- and long-output accuracy. Table 6 compares target maps on the same features, averaged over ShareGPT and LMSYS. Short requests have at most 256 output tokens and long requests exceed 1,024; “All” also includes medium requests.

MapShortLongAll
log⁡L\log L50.7 / 0.7907310.4 / 0.5832155.9 / 0.9122
L\sqrt{L}68.5 / 0.7179279.3 / 0.6009149.6 / 0.9161
LL105.4 / 0.4881266.3 / 0.6097162.7 / 0.8984

Table 6. Effect of the target map on length prediction (mean MAE / Spearman ρ\rho).

Analysis. Log gives lower prediction error and better ordering on short requests. Square root compresses long lengths less aggressively, improving error and ordering for long requests while avoiding the larger short-request errors of identity. We therefore use square root by default.

Single-layer versus All-layer Readout

Collecting all layers may improve prediction but adds collection, buffering, and transfer. Table 7 compares block 30 with the concatenation of all hidden states on Qwen3-8B, using the same target, Ridge fit, and splits, averaged over ShareGPT and LMSYS.

InputMean MAEMean Spearman ρ\rho
Block 30149.530.9160
All states144.630.9204

Table 7. Single-layer versus all-layer length prediction on Qwen3-8B.

Analysis. All layers lower MAE by only 3.3% and give little mean-response gain in an idealized replay that ignores transfer cost (Appendix § 12.§ 3). One layer shrinks the FP32 input from 592 to 16 KiB.

Prediction and Deployment Costs

Measuring predictor storage cost. Table 8 reports logical FP32 predictor weights in decimal KB/MB/GB, excluding the serving LLM, activations, cached features, and solver state.

PredictorParametersFP32 weights
TIE185,211,3940.741 GB
LTR125,240,0640.501 GB
SSJF109,597,3130.438 GB
EGTP8,435,73233.74 MB
TRAIL2,118,1848.47 MB
LEAP (ours)4,09716.39 KB

Table 8. Predictor parameter counts and FP32 weight storage.

Analysis. LEAP uses 30,569× fewer predictor parameters than LTR, the strongest serving baseline in Table 3, and 45,207× fewer than TIE. Deployment assessment must also account for feature transfer, score publication, and scheduler consumption, which we measure next.

Testing CPU prediction under a fixed budget. Figure 7 measures score delivery with one prediction core per method

Figure 7. CPU score delivery under matched core budgets (batch eight).

on matched Qwen3-8B/ShareGPT inputs, excluding serving-model execution. Hidden-state paths include feature transfer and pooling (details: Appendix § 12.§ 6).

Analysis. With one core per method, the CPU cost of each score (panel (c)) determines both throughput and delay. Text predictors run a full encoder for every request, so a core delivers only a few scores per second and queued scores wait tens of seconds. Hidden-state baselines skip the encoder but still run million-parameter heads, and EGTP-P must also rebuild logits on CPU, so they saturate between about ten and a few thousand scores per second. LEAP’s readout is a single, 4,097-parameter affine map. One core keeps pace with the offered rate up to 10,000 predictions/s, saturates near 58,000/s, and keeps P95 delay below 3 ms at every rate.

Completing prediction during prefill. Figure 8 measures readiness during Qwen3-8B prefill. A CPU worker reads block-30 features and sets a flag after prediction and initial batch ordering. A GPU probe checks it before final RMSNorm, while prefill work remains. The post-prefill negative control starts CPU work only after prefill returns (protocol: Appendix § 12.§ 7).

Figure 8. Share of trials in which prediction and initial batch order are ready before the final RMSNorm on Qwen3-8B (24 trials per bar).

Analysis. All 144 asynchronous trials were ready, while all 72 controls were unready. Feature transfer, prediction, and initial ordering finish within the remaining prefill at batch sizes 1, 8, and 32 with FP32/FP16 transport. The probe measures readiness, not the remaining time margin; admission and KV allocation remain with the engine.

Answer to Q4: CPU execution sustains higher score-delivery throughput under the tested core budget (Figure 7), and prediction and initial ordering finish before prefill ends in all 144 overlap trials (Figure 8).

Related Work

Prediction with auxiliary models. Zheng et al. [20] use response-length estimates to group requests for scheduling. TIE (Zheng et al. [18]) models length uncertainty with a DeBERTa encoder, SSJF (Qiu et al. [9]) predicts lengths with a proxy model, and LTR (Fu et al. (2024)) learns priorities with a separate ranking model. These methods score requests before prefill but require additional model storage and prompt encoding.

Hidden-state prediction and availability. TRAIL (Shahout et al. [11]) and STAR (Wang et al. [16]) reuse hidden states but retain MLP prediction heads. TRAIL already overlaps CPU prediction with later GPU layers, but its CPU path pools all prompt-token states and evaluates an 8.47 MB MLP. These weights exceed a core’s private L2 cache, so each batch reloads them from the shared L3 cache or memory used by the serving process, and pooling work grows with prompt length. LEAP’s 16.39 KB readout stays in a core’s private cache, and it consumes one hidden state per request.

Feature aggregation costs. EGTP (Xie et al. [17]) pools states using token entropy; ESTP (Ren et al. [10]) adds attention-based semantic weights. Both compute entropy over the full vocabulary, requiring logit projection when logits are unavailable; ESTP also extracts and aggregates attention weights.

Parameter adaptation. Refreshing predictions from new decode states does not itself adapt predictor parameters to a new domain. The MLP heads in TRAIL and STAR are trained with gradient-based optimization; refitting them on completion feedback entails further optimization. Self-supervised test-time training (Sun et al. [14]) likewise updates parameters through a gradient-based auxiliary objective. LEAP combines a single-state linear readout with gradient-free RLS adaptation and asynchronous CPU execution.

Conclusion

LEAP combines cross-domain ranking performance with low-cost prediction and completion-driven updates. The frozen readout generalizes across the evaluated datasets, while RLS restores ranking quality under the larger language shift to Aya. CPU-asynchronous execution turns the compact readout into measured serving benefits. These results connect length prediction to the cost of maintaining useful scheduling decisions as domains and tasks change.

Limitations & Future Work

Tail latency and fairness. Length-aware scheduling prioritizes short requests, which inherently trades mean latency for tail fairness. Introducing waiting-time protections can balance this tradeoff, though methods differ in how strictly they enforce bounds.

Decoding policy. The readout is conditioned only on the prefill representations and does not account for decoding-time choices such as temperature or top-kk parameters. Extensions could condition predictions on these factors.

Adaptation dynamics. RLS updates use uniform weights over all completions and can be extended with forgetting factors or selective weighting for special cases such as capped outputs.

Serving scope. The prototype integrates with single-GPU eager-mode vLLM. Multi-GPU, prefix-cached, and chunked-prefill serving are separate concerns.

AI Usage Disclosure

Generative AI tools, including OpenAI Codex, assisted with method refinement, mathematical exposition, and experimental design. They also assisted with implementation, debugging, experiment execution, data reformatting, and result interpretation. We used generative AI for literature search and summarization, manuscript drafting and language editing, and reference formatting. AI tools also helped create and revise scientific figures and editable graphics. Verification included model and input checks, implementation consistency checks, and comparisons between reported numbers and saved experiment outputs. The authors take responsibility for the final text, methods, figures, code, and reported results.

Readout and Adaptation Details

Base predictor

Source standardization uses coordinatewise training means μ\mu and scales ss: xi=(hi−μ)/sx_i=(h_i-\mu)/s. These statistics remain frozen during deployment and adaptation. They are absorbed into the deployed base-head coefficients; RLS uses the same standardized features with an appended intercept (§ 3.3).

Source precision and recursive solver

The source precision A0A_0 is computed offline from training features only, using the deployed head’s frozen mean and scale. With the prior regularization parameterized as λp=αd\lambda_p=\alpha d, separately from the base-fit λ0\lambda_0, the current instantiation uses α=1\alpha=1, d=4096d=4096, and 2,880 source-training requests. The augmented intercept is not regularized. The deployed coefficients are converted to standardized coordinates exactly: if the folded head is wˉ0⊤h+bˉ0\bar{w}_0^{\top}h+\bar{b}_0, then θ0=[wˉ0⊙s;bˉ0+μ⊤wˉ0]\theta_0=[\bar{w}_0\mathbin{\odot}s;\bar{b}_0+\mu^{\top}\bar{w}_0]. Thus adaptation starts from the same deployed predictor.

The implemented recursion solves the anchored objective in Equation (7), as derived in § 3.3. All incorporated completions have equal weight; there is no sliding window or forgetting factor.

The implementation stores PP in FP32 and coefficients in FP64. It forms VV by an FP32 matrix product, promotes it to FP64 for the small solve and coefficient update, and casts the precision downdate back to FP32. The solve computes SX=V⊤SX=V^{\top}, G=X⊤G=X^{\top}, with S=Iq+UVS=I_q+UV; this and Equation (10) rely on symmetry in exact arithmetic. Finite-precision drift must therefore be checked against a direct anchored solve. The implementation checks finite coefficients and the small system; independent numerical comparisons are reported in Appendix § 10.§ 3.

The current update batch contains q=8q=8 completions, with no feature clipping, target discounting, or source-sample replay. The updater consumes full batches in completion order. Shutdown drains queued updates; a final incomplete batch is not fitted. Each label is the observed served length under the output budget, including capped completions; it does not reveal an uncapped latent length.

Numerical checks

An FP64 synthetic check compares recursive coefficients with an independently solved joint Ridge system after every full eight-sample batch; the maximum absolute coefficient difference is below 10−1210^{-12}. Precision diagnostics on two 4,240-request conversation-transfer streams compare FP32 inverse-precision storage and FP64 small solves against an FP64 RLS reference. Without periodic symmetrization, the maximum token-prediction differences are approximately 0.00024 and 0.00160. These finite-stream checks measure numerical error, not a universal floating-point bound or serving speedup.

Deployment Configuration and Measurement

Execution and feedback protocol

Algorithm 1 specifies the request lifecycle. The worker joins completion labels to cached features by request identity and retains early labels until their features arrive. Only matched completions count toward the update batch. Publications carry identity and version metadata, so results for completed or unrelated requests cannot replace live scores. The prototype integrates with vLLM V0 using a native C++ CPU worker.

Algorithm 1 Completion-driven prediction and scheduling

Notation: CC: live-feature cache; AtA_t: eligible requests; BB: device capacity; θpub\theta_{\mathrm{pub}}: published head. ℓθ(u)\ell_\theta(u) applies Equations (14) and (2).

Require: θ0,P0,g,q\theta_0,P_0,g,q

  1. (θ,P)←(θ0,P0)(\theta,P)\leftarrow(\theta_0,P_0); θpub←θ0\theta_{\mathrm{pub}}\leftarrow\theta_0; C←∅C\leftarrow\varnothing; Q←[]Q\leftarrow[]

  2. on arrival (i)(i): L^i←M\hat{L}_i\leftarrow M

  3. at prefill layer (i,hi)(i,h_i): ASYNCD2H(i,hi)(i,h_i)

  4. prediction worker: feature ready (i,hi)(i,h_i)

  5. ui←STANDARDIZE⁡(hi);1]u_i\leftarrow\operatorname{STANDARDIZE}(h_i);1]; C[i]←uiC[i]\leftarrow u_i

  6. PUBLISHINITIAL(i,ℓθ(ui))(i,\ell_\theta(u_i))

  7. PUBLISHLATEST(i,ℓθpub(ui))(i,\ell_{\theta_{\mathrm{pub}}}(u_i))

  8. prediction worker: matched completion (i,Li)(i,L_i)

  9. Q←Q∥[(C[i],g(Li))]Q\leftarrow Q\mathbin{\Vert}[(C[i],g(L_i))]; ERASE(C,i)(C,i)

  10. if ∣Q∣=q|Q|=q then ENQUEUEUPDATE(Q)(Q); Q←[]Q\leftarrow[]

  11. update thread: batch ready (U,y)(U,y)

  12. (θ,P)←RLS⁡(θ,P,U,y)(\theta,P)\leftarrow\operatorname{RLS}(\theta,P,U,y) Eqs. (8)–(10)

  13. HANDOFFSNAPSHOT(θ)(\theta)

  14. prediction worker: between jobs

  15. if new snapshot then θpub←NEWESTSNAPSHOT⁡()\theta_{\mathrm{pub}}\leftarrow\operatorname{NEWESTSNAPSHOT}()

  16. PUBLISHLIVE(C,θpub)(C,\theta_{\mathrm{pub}}) end if

  17. on decode (At,B)(A_t,B)

  18. L^←CONSUMEVALID⁡(L^)\hat{L}\leftarrow\operatorname{CONSUMEVALID}(\hat{L})

  19. si←max⁡{1,L^i−ai(t)}s_i\leftarrow\max\{1,\hat{L}_i-a_i(t)\}, i∈Ati\in A_t

  20. ADVANCE(GUARDEDSHORTEST⁡(At,s,B))(\operatorname{GUARDEDSHORTEST}(A_t,s,B))

Matched-Aging transfer replay

Table 4 reuses fixed Qwen3-8B output trajectories, taking the first 4,200 eligible requests in the frozen stream order for each of Aya and MATH. The same prefix is used for all methods; requests are not resampled by length. Calibration and the entire queueing process are recomputed for 4,200 requests. The single arrival seed is 42. Offered load ρoff∈{1.2,1.5}\rho_{\mathrm{off}}\in\{1.2,1.5\} scales the arrival rate relative to the capacity reference obtained from the same-request FCFS simulation. All learned predictors share greedy KV admission, the same last-victim rule, a 256-request batch limit, a 380,000-token KV budget, and the same fitted A800 prefill/decode/KV/swap cost model. The exponential Aging rule ranks requests by

s~i(t)=si(t)γWi(t)/τ,(16)\tilde{s}_i(t) = s_i(t)\gamma^{W_i(t)/\tau}, \tag*{(16)}

where si(t)s_i(t) is the remaining-length score in Equation (15), Wi(t)W_i(t) is accumulated queue waiting excluding execution, γ=0.9\gamma= 0.9, and τ=30\tau= 30 s. Lower scores advance first. The rule uses TIE’s aging formula and constants. The TIE control uses its bounded expected length under this common rule; its native CVaR and dynamic-β\beta policy remains a separate reference. LTR produces signed ordinal scores, which require calibration for use as token-length estimates. LTR maps them monotonically to token lengths using isotonic calibration on 360 existing LMSYS validation requests; no target lengths fit this adapter. TRAIL uses the frozen 40-bin initial head, not its online prediction/filter chain. SSJF retains source-training normalization and author tail-512 tokenization; EGTP retains its author prefix-feature extractor and frozen source head. The simulation retains the disclosed method-specific checkpoint training budgets. LEAP and RLS share the same initial source head; every eight completed requests update the latter using only that arm’s completed labels. Different completion orders therefore induce different causal update sequences. Each finite arrival stream is drained to completion. The table includes modeled execution costs, while cached initial scores are assumed available on arrival, so the replay isolates prediction utility; real serving costs are reported separately.

Experimental Protocol and Supporting Results

Shared evaluation protocol

Training uses the method-specific budgets in Appendix § 12.§ 4, with 360 validation and 1,000 test requests per model–dataset setting. Splits exclude normalized-prompt and conversation-parent overlap; the ShareGPT test set covers 983 parents. Test labels are first-rollout lengths capped at 2,048 tokens. Repeated rollouts provide additional labels for the same training request. Readout standardization is fitted on source training data and frozen; regularization minimizes source-validation token MAE over {10−4,10−3,10−2,10−1,1}\{10^{-4}, 10^{-3}, 10^{-2}, 10^{-1}, 1\}.

Figure 5 uses 1,000 LMSYS test requests. The Alpaca comparison uses 1,000 held-out tests after 1,000 separate feedback labels; MATH and Aya use 4,200-request predict-before-feedback streams. The TRAIL results in this comparison use its initial head.

The representative serving panel uses 500 held-out ShareGPT requests and follows § 4.3 with 20,592 KV blocks and no artificial step waits. Requests arrive at 2.64984 requests/s and share recorded output trajectories. A missing candidate score makes the entire unprotected decision fall back to FCFS. Per-length analyses use L<256L < 256, 257≤L≤1,024257 \le L \le1{,}024, and L>1,024L > 1{,}024 for short, medium, and long requests, respectively. Serving points are single-run measurements unless stated otherwise.

Target-map control. Log, square-root, and identity heads share the block-30 features and validation-only regularization selection. Test requests are not resampled by length. Short/long subgroup counts are 196/312 on ShareGPT and 467/188 on LMSYS; medium requests remain in the overall result. Table 6 equally averages metrics computed separately within each dataset and subgroup.

Finite-arrival LMSYS serving control

Figure 6 uses the same frozen first 1,024 held-out LMSYS requests and 544,047 recorded output tokens per point, with seed 0 and offered rates of 20, 40, 60, 80, and 100 requests/s. All four methods are measured at all five rates. The ordered request identities, generation seeds, and token trajectories are shared; only arrival spacing changes with the offered rate.

All methods use 512 resident requests, a 64-request decode cap, 20,592 KV blocks of size 16, a 4,096-token batch limit, and a 0.8 GPU-memory fraction. All plotted predictors publish a frozen initial total-length estimate for common post-prefill ranking, without a per-token countdown. They share wall-clock Aging with γ=0.9\gamma= 0.9 and a 30-second time scale. Protection takes priority; if any unprotected candidate lacks an accepted score, the unprotected choice falls back to FCFS.

LEAP uses the frozen block-30 square-root readout on CPU; FCFS has no prediction head. The source-trained TIE checkpoint supplies a deterministic bounded expectation with a 2,048-token cap. Its encoder was trained with 512-token inputs and executes with a 2,048-token input limit in this control. At every rate, the same adapter splits logical TIE batches into physical forwards of at most eight, preserving logical padding, output order, and the expectation calculation. LTR uses the source-validation isotonic mapping described in Appendix § 11.§ 2, with physical forwards of at most eight. Both predictors execute on serving GPU 0. Client metrics use the planned-arrival origin defined in § 4. Per-token latency is each request’s end-to-end latency divided by its actual output-token count before taking the mean or P90; it is not inter-token latency. Throughput divides actual output tokens by the fully drained client duration. For the last three points, another job used GPU 1 and cores 12–15 while serving kept GPU 0 and cores 0–7.

Input-width control

The all-state input in Table 7 follows Hugging Face’s hidden-state convention: the embedding, raw decoder blocks 1–35, and the normalized final state of block 36. The same original labels, source-only standardization, six validation-selected regularization candidates, and capped round-0 test lengths are used for both inputs.

The input-ablation replay uses a single nonpreemptive server with service time equal to cached output length and initial scores available for free at arrival. A fixed seed (42) supplies common exponential spacings for offered loads 0.5, 0.8, 0.95, 1.1, plus simultaneous arrivals; 500 requests form a nested subset of the 1,000-request cohort. Prefill, batching, KV capacity, feature extraction, transfer, and prediction costs are excluded. For the 1,000-request all-state square-root comparison, P95 response changes range from a 10.36% improvement to an 8.23% increase. Block 30 is the pre-existing default, not selected anew on these tests.

Predictor Training Configurations

We use author implementations or their training cores, retaining the objectives and checkpoint-selection rules described below. These configurations apply to Table 1 and the LMSYS-source transfer results in Figure 5; serving and component controls use the checkpoints specified in their protocols. Optimizer configurations, implementation versions, and training traces are documented with the artifact. TIE uses 20K LMSYS requests; LTR and SSJF use 10K ShareGPT/LMSYS requests per backbone. Other fits use 2,880 requests.

Readout configurations. The single-label Qwen scale control uses blocks 24, 24, 32, 30, 36, and 60 for the 0.6B, 1.7B, 4B, 8B, 14B, and 32B models, respectively. The default Qwen3-8B head has 4,096 inputs, with regularization 1.0 for ShareGPT and 0.1 for LMSYS; Mistral uses block 28. Whole-service resource controls retain their documented input configurations.

Text-based predictors. TIE trains its DeBERTa encoder and two distribution heads from pretrained initialization, following the author’s encoder-freezing schedule. Its 20K LMSYS fits use 20 outputs per prompt. Point predictions use the bounded expectation with a 2,028-token cap; the LMSYS quality results use numerical integration. LTR uses 2,048-token inputs and ListMLE. SSJF uses the BERT REG-L1 recipe with tail-512 tokenization, training the encoder for three epochs and freezing it for three more. Our implementation standardizes targets using source-training statistics and applies the inverse transform at inference; quality results use un-clipped predictions. In the transfer replay, LTR converts ordinal scores to lengths by source-validation isotonic calibration, as specified in Appendix § 11.§ 2.

SSJF target standardization. An early diagnostic compared raw token-length targets with source-training target standardization using 2,880 training, 360 validation, and 1,000 test requests per setting. Both variants retained the author’s best REG-L1 network, tail-512 tokenization, six-epoch training loop, encoder freezing after epoch three, learning-rate schedule, and final-checkpoint rule, with seed 42. Standardization used only the training-set mean and standard deviation; predictions were transformed back to token units before evaluation. Standardization improved MAE and Spearman correlation in all five tested settings, motivating its use in our SSJF implementation; Table 9 shows the two Qwen3-8B settings, using un-clipped predictions against first-rollout test lengths.

BackboneDatasetRaw targetsRaw targetsStandardizedStandardized
MAE ↓\downarrowρ↑\rho\uparrowMAE ↓\downarrowρ↑\rho\uparrow
Qwen3-8BShareGPT779.390.0845308.890.6716
Qwen3-8BLMSYS517.790.1658240.900.7844

Table 9. SSJF target standardization at the early 2,880-request training budget.

Table 9. SSJF target standardization at the early 2,880-request training budget.

No epochs, learning rates, clipping, or mappings were selected on test results.

Hidden-state predictors. The dynamic Qwen control uses block-30 trajectories and 360 validation requests per domain; EWMA gains of 0.01/0.05 for ShareGPT/LMSYS are selected on validation only. A separate eight-setting TRAIL diagnostic uses 1,000 test requests per setting, matched live token positions, and the same remaining-budget clip; it improves MAE over frozen LEAP countdown in six settings. TRAIL includes a 40-bin initial MLP and an online head trained on intermediate generation states, with the original probability filter. Initial-head quality results remain distinct from evaluations of the complete prediction chain. EGTP follows the LP_Bench implementation: entropy-weighted pooling of the first four raw prompt tokens, a two-layer 20-bin MLP, and the 0.95 KL + 0.05 MSE objective. Its training batch sizes are 16 for Qwen/Llama and 256 for Mistral, with 10 training epochs on Qwen/Llama and 200 on Mistral.

CPU pipeline measurement details

The component study assigns one prediction core per method, separate producer, consumer, and sampling cores, and one thread per numerical library. Each plotted cell offers batches of eight requests for ten seconds at rates of 1/10/100/1k/10k/100k per second, with eight queued batches and up to sixty seconds of drain. All paths reuse 64 source-validation prompts and predictors trained on 2,880 requests. Timing starts when raw text or the GPU representation is available and ends at consumer receipt; serving-model execution is excluded.

LEAP transfers a 4,096-coordinate FP32 feature to its FP64 affine routine. TRAIL transfers BF16 prompt states and pools them on CPU. Text predictors include tokenization and encoder execution; TIE uses 10,000 Monte Carlo draws for its capped expectation, excluding CVaR. Both EGTP variants use the author’s four-token prefix: EGTP-P reconstructs logits on CPU, whereas EGTP-L assumes GPU-resident logits and hidden states. EGTP-P’s FP32 projection occupies 2.49 GB beyond prediction-head storage. GPU pooling is evaluated as a separate feature-availability control.

Throughput counts receipts during the supply window; latency includes accepted batches completed during drain. Rejected offers, generator skips, and supply-end backlog remain in the accounting. CPU measurements separate prediction-process and consumer CPU time from producer and sampler work, and exclude initialization. Each cell is one hardware observation; generator-limited cells report delivered throughput without establishing predictor saturation.

The plotted seven pipelines were uniformly remeasured with the same process-isolated producer, receiver, and sampler. Offers are clocked by an integer native loop on the reserved producer core, using the same monotonic clock and arrival schedule. The producer continuously polls on its single reserved core at every offered rate, including the same one-second lead-in before each ten-second supply window. No rate uses a sleep branch. Admission telemetry is buffered during supply and written afterward; actual offer timestamps and skipped arrivals are retained. The same waiting policy applies across the entire offered-rate grid. The input FIFO retains eight queued batches and immediate queue-full rejection. Request transport and result publication use direct pipes, avoiding background queue-feeder threads; result serialization is charged to the same prediction process. The numerical worker, model weights, precision, inputs, offered-rate grid, ten-second supply, sixty-second drain, and absence of TTL remain unchanged. Original and intermediate measurements are preserved separately; none are mixed into this figure.

Generator qualification retains the original criteria: a cell is limited if more than 1% of scheduled batches are skipped or if the P95 attempt lag exceeds one batch interval. Every newly plotted cell passes these criteria. Accepted work, rejected batches, generator skips, supply-end backlog, post-supply completions, all original timestamps, and all warnings remain in the artifact. Prediction workers and the separate receiver/sampler processes inherit the verified resource leases; CPU roles remain 12/13/14/15, with source representations on GPU1 and no serving-model execution.

Prefill-overlap measurement

Throughput characterizes CPU prediction capacity; overlap characterizes score availability. We measure the completion of CPU prediction and initial batch ordering relative to the remaining GPU prefill. On Qwen3-8B, a native CPU worker reads block-30 features, computes scores and a stable batch order, and publishes a ready flag. A GPU probe reads this flag immediately before the final RMSNorm, confirming CPU completion while mandatory prefill work is still pending. Figure 8 reports the counts separately for FP32 and FP16 asynchronous feature transport. Each mode and batch size has 24 executions across three temporal blocks in one A800 session, reusing fixed source-validation prompt groups. The FP32 control captures the same layer’s features but defers transfer and CPU work until prefill returns; its zero readiness is the expected negative control.

References

  1. [1]Chen, J., Liang, Y., and Liu, Z. DFlash: Block diffusion for flash speculative decoding. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pp. 15343–15355. PMLR, 2026. URL https://proceedings.mlr.press/v306/chen26bv.html.arxiv.org/abs/2602.06036
  2. [3]Gurnee, W., Sofroniew, N., Pearce, A., Piotrowski, M., Kauvar, I., Chen, R., Soligo, A., Bogdan, P., Ong, E., Wang, R., Thompson, B., Abrahams, D., Kantamneni, S., Ameisen, E., Batson, J., and Lindsey, J. Verbalizable representations form a global workspace in language models. Transformer Circuits Thread, 2026. URL https://transformer-circuits.pub/2026/workspace/index.html.arxiv.org/abs/2607.15495
  3. [4]Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. URL https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/hash/be83ab3ecd0db773eb2dc1b0a177836a1-Abstract-round2.html.
  4. [5]Hou, Z., Li, X., and You, K. Efficient reasoning via constrained optimization in latent space. arXiv preprint arXiv:2609.34181, 2026. URL https://arxiv.org/abs/2609.34181.
  5. [6]Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611–626. ACM, 2023. doi: 10.1145/3600006.3613165. URL https://doi.org/10.1145/3600006.3613165.
  6. [8]Lindsey, J., Gurnee, W., Ameisen, E., Chen, B., Pearce, A., Turner, N. L., Citro, C., Abrahams, D., Carter, S., Hosmer, B., Marcus, J., Sklar, M., Templeton, A., Bricken, T., McDougall, C., Cunningham, H., Henighan, T., Jermyn, A., Jones, A., Persic, A., Qi, Z., Thompson, T. B., Zimmerman, S., Rivoire, K., Conerly, T., Olah, C., and Batson, J. On the biology of a large language model. Transformer Circuits Thread, 2025. URL https://transformer-circuits.pub/2025/attribution-graphs/biology.html.
  7. [9]Qiu, H., Mao, W., Patke, A., Cui, S., Jha, S., Wang, C., Franke, H., Kalbarczyk, Z. T., Başar, T., and Iyer, R. K. Efficient interactive LLM serving with proxy model-based sequence length prediction. arXiv preprint arXiv:2404.08509, 2024. URL https://arxiv.org/abs/2404.08509.
  8. [10]Ren, F., Wen, S., Guo, L., Tian, Y., Cui, Y., and Chen, X. When entropy is not enough: Reclaiming lost semantics in LLM output length prediction. arXiv preprint arXiv:2608.15592v1, 2026. URL https://arxiv.org/abs/2608.15592v1.
  9. [11]Shahout, R., Malach, E., Liu, C., Jiang, W., Yu, M., and Mitzenmacher, M. Don’t stop me now: Embedding based scheduling for LLMs. arXiv preprint arXiv:2410.01035, 2024. URL https://arxiv.org/abs/2410.01035.
  10. [13]Singh, S., Vargus, F., D’souza, D., Karlsson, B. F., Mahendiran, A., Ko, W.-Y., Shandilya, H., Patel, J., Mataciūnas, D., O’Mahony, L., Zhang, M., Hettiachchi, R., Wilson, J., Machado, M., Moura, L., Krzemiński, D., Fadaei, H., Ergun, I., Okoh, I., Alaagib, A., Mudaunnayake, O., Alyafeai, Z., Chien, V., Ruder, S., Guthikonda, S., Alghamdi, E., Gehrmann, S., Muennighoff, N., Bartolo, M., Kreutzer, J., Üstün, A., Fadaee, M., and Hooker, S. Aya dataset: An open-access collection for multilingual instruction tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11521–11567. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-long.620. URL https://aclanthology.org/2024.acl-long.620/.DOI
  11. [14]Sun, Y., Wang, X., Liu, Z., Miller, J., Efros, A., and Hardt, M. Test-time training with self-supervision for generalization under distribution shifts. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9229–9248. PMLR, 2020. URL https://proceedings.mlr.press/v119/sun20b.html.arxiv.org/abs/1909.13231
  12. [15]Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford Alpaca: An instruction-following LLaMA model. GitHub repository, 2023. URL https://github.com/tatsu-lab/stanford_alpaca.
  13. [16]Wang, Z., Hong, Z., Li, X., Wang, Z., Li, S., Meng, Q., Wang, Q., Huan, C., Gu, R., Zhong, S., and Tian, C. STAR: Decode-phase rescheduling for LLM inference. In Proceedings of the 35th International Symposium on High-Performance Parallel and Distributed Computing, pp. 125–138. ACM, 2026. doi: 10.1145/3806645.3807813. URL https://doi.org/10.1145/3806645.3807813.
  14. [17]Xie, H., Chen, Y., Wang, L., Hu, L., and Wang, D. Predicting LLM output length with entropy-guided representations. arXiv preprint arXiv:2602.11812, 2026. URL https://arxiv.org/abs/2602.11812.
  15. [18]Zheng, H., Zhang, Y., Fu, F., Zhou, X., Luo, H., Zhu, H., Zhu, Y., Wang, H., Yan, X., and Jiang, J. Scheduling LLM inference with uncertainty-aware output length predictions. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pp. 163894–163918. PMLR, 2026. URL https://proceedings.mlr.press/v306/zheng26ae.html.arxiv.org/abs/2604.00499
  16. [19]Zheng, L., Chiang, W.-L., Sheng, Y., Li, T., Zhuang, S., Wu, Z., Zhuang, Y., Li, Z., Lin, Z., Xing, E. P., Gonzalez, J. E., Stoica, I., and Zhang, H. LMSYS-Chat-1M: A large-scale real-world LLM conversation dataset. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=BOFdKxifwto.arxiv.org/abs/2309.11998
  17. [20]Zheng, Z., Ren, X., Xue, F., Luo, Y., Jiang, X., and You, Y. Response length perception and sequence scheduling: An LLM-empowered LLM inference pipeline. arXiv preprint arXiv:2305.13144, 2023. URL https://arxiv.org/abs/2305.13144.

Paper details

vlnc:W_thrcraz6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 7

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Codex

View statement
Looking for related papers…
Contents