betaFeedback

WorksComputer Science

Scaling VLA Serving Under Robotic Execution Constraints

Abstract

Vision-language-action (VLA) models have emerged as a promising paradigm for general-purpose robotic manipulation. In our production setting, hundreds of VLA-powered robots are already operating in real-world environments, creating an immediate need for efficient fleet serving. We present XInfer, a production-deployed VLA serving system designed to reduce the inference infrastructure cost of large-scale robotic deployments. XInfer jointly coordinates heterogeneous VLA computation with the evolving response requirements of robotic execution. By exploiting computational parallelism while managing interference, XInfer improves GPU efficiency while protecting time-critical robotic feedback. We evaluate XInfer across three VLA models using production-derived workloads, achieving 1.26-3.75 times the effective robot capacity (ERC) of state-of-the-art systems under physical-execution constraints. Our implementation is publicly available at https://github.com/X-Square-Robot/xinfer.

Introduction

Vision-language-action (VLA) models are bringing general-purpose robotic manipulation into logistics and manufacturing, where multiple robots can share GPU infrastructure to reduce serving cost. Figure 1 shows representative parcel-handling operations from our production deployment. Recent inference engines improve the efficiency of individual VLA queries [25] (Su 2026; Xu et al. 2026). However, higher inference throughput does not necessarily translate into greater physical productivity, as delayed feedback can disrupt robotic execution. We use effective robot capacity (ERC), the maximum number of concurrent robots a GPU can serve while meeting specified latency and reliability requirements, as the serving objective.

Production parcel handling

Figure 1. Production parcel handling. A carton (top) and a soft parcel (bottom) are reoriented and pushed onto the conveyor. Each row follows the same parcel from left to right.

Shared VLA serving requires exploiting concurrency without compromising time-critical feedback. VLA requests traverse heterogeneous stages with distinct computational demands, creating opportunities to overlap independent work across robots. However, concurrent GPU execution

also introduces interference: admitting a new stage may delay stages already in progress. Since these stages are non-preemptive, prioritizing urgent requests alone cannot prevent deadline violations caused by previously admitted work. Effective fleet serving therefore requires two coordinated decisions: which requests should be prioritized, and which stages can be admitted without jeopardizing the deadlines of current and forthcoming requests.

The cost of this interference is not uniform across requests. Robots continue executing previously predicted actions while awaiting new responses, and the consequences of delayed feedback depend on their ongoing physical execution. As a result, requests with similar computational demands may have substantially different response-time requirements, which evolve throughout task execution. Moreover, response timing influences subsequent request arrivals, coupling scheduling decisions with future workload dynamics. Fleet serving must therefore balance concurrency against interference under changing per-request deadlines, rather than optimize inference throughput or average latency alone.

Prior systems adapt action horizons and request priorities to execution context ([9 Anonymous Authors, n.d.]), coordinate serving for factory productivity ([15]), or overlap encoding and diffusion in lockstep rounds ([1]). Yet effective fleet serving must reconcile the time-varying urgency of robotic feedback with the distinct computational demands of VLA stages. This calls for fine-grained scheduling that matches stage execution to available GPU capacity while controlling interference with time-critical requests.

We present XINFER, a VLA serving system that couples asynchronous execution with fine-grained, execution-aware scheduling. Its dependency-driven engine exposes parallelism across requests and heterogeneous VLA stages, allowing computation to progress as dependencies resolve. The scheduler coordinates these execution opportunities with the time-varying urgency of robotic feedback by decoupling request prioritization from stage admission. At stage boundaries, it combines per-request response-time budgets, remaining computation, and predicted interference to admit ordinary work only when protected requests remain predicted feasible. It further accounts for anticipated earlier-deadline arrivals using request history, balancing immediate concurrency gains against future scheduling demands.

We evaluate VLA-2B, π0.5\pi_{0.5}, and Wall-OSS-0.5 using production-derived closed-loop workloads and controlled experiments. With error-free responses, XINFER achieves ERCs of 18 VLA-2B and 24 π0.5\pi_{0.5} robots, versus 10 and 15 for vLLM-Omni; Robion serves 19 π0.5\pi_{0.5} robots. In a separate VLA-2B evaluation with heterogeneous response-time budgets, all policies share the same backend and execution resources. Under the same real-time feasibility criterion, XINFER achieves an ERC of 18 robots, compared with 15 for Slack-Gate, 14 for FIFO, and 12 for EDF (§6). Mechanism ablations isolate the effects of asynchronous execution and scheduling decisions.

Our contributions are:

  • An asynchronous VLA serving engine that exposes cross-request and cross-stage parallelism by advancing work as its dependencies complete.

  • Execution-aware cross-stage scheduling that coordinates shared GPU computation by separating request priority from stage admission, accounting for interference on current and anticipated urgent requests.

  • Production deployment and empirical evaluation. XINFER is deployed in production. We characterize closed-loop workloads, compare ERC across three VLA backends, and evaluate heterogeneous-budget scheduling and its mechanisms.

Background

Heterogeneous VLA Computation

Vision-language-action (VLA) models map observations and instructions to robot actions. We study models coupling a vision-language backbone to an action expert that generates continuous actions through flow matching or diffusion ([17, 8, 5, 30] (Yu et al. 2026)). As Figure 3(a) shows, the backbone computes conditioning keys and values once; the expert reuses them over KK denoising steps to generate an action chunk of HH future keyframes ([31]). We distinguish three computational stages: ViT encodes visual observations, Prefill performs language-model computation after visual encoding, and Action generates actions through denoising. These stages differ in computation and memory-access patterns. The expert’s weights differ from the backbone’s, and its evolving action state is private to each request.

VLA computation and real-time chunking

Figure 3. VLA computation and real-time chunking. (a) Conditioning keys and values are reused across denoising steps. (b) The frozen prefix aligns the next chunk with committed actions while inference overlaps execution. The illustrated early response satisfies L<dΔ<TEL < d\Delta< T_E, so the next send occurs at tk+TEt_k + T_E.

Closed-Loop Robot Control

Real-time chunking (RTC) ([6]) aligns independently generated chunks by freezing the next chunk’s first dd keyframes to committed actions (Figure 3(b)). For a request issued at tkt_k with keyframe spacing Δ\Delta, the planned handoff is tk+dΔt_k+d\Delta and its budget is dΔd\Delta. A late chunk can cause an action discontinuity if applied without alignment to current execution progress. If the action queue empties, our controller pauses execution until a replacement chunk becomes usable.

Longer commitment permits more inference time but postpones feedback, even when dd adapts to recent latency (§ 3.1). Appendix § 13.§ 1 gives the inpainting and handoff details.

Fleet Workload and Request Contract

A fleet shares inference resources across robots, each running its own control loop. A request carries observations, robot state, and model inputs. Each request jj is associated with a response-time budget bjb_j that expresses its feedback requirement. The budget is supplied as request metadata and may be assigned by the robot controller, a task planner, or a policy at the serving frontend. RTC requests additionally supply the aligned previous action chunk and commitment length dd. The response budget, RTC handoff budget dΔd\Delta, and minimum request period TET_E are distinct quantities: they specify the serving requirement, the planned chunk handoff, and the earliest next send.

Robots continue their current chunks while requests are processed. We use a minimum-period request policy with at most one outstanding request per robot. Let sjs_j be the actual send time of request jj and rjr_j its response arrival time. Abstracting local client-side handling, the next request is sent at

sj+1=max⁡(sj+TE,rj).(1)s_{j+1} = \max(s_j + T_E, r_j). \tag*{(1)}

The next send occurs when the minimum interval has elapsed and the previous response has arrived. Thus, NN robots offer at most N/TEN/T_E requests per unit time; late responses reduce both offered load and action-update frequency.

Figure 4 overlays five streams from deployed robots, including two sessions from one robot. Requests occupy only 45–53% of each dedicated instance’s timeline; combined arrivals show gaps and bursts.

Five separately recorded request streams aligned on a common 2-s axis

Figure 4. Five separately recorded request streams aligned on a common 2-s axis. Top: bars mark requests in flight and ticks mark sends. Bottom: the superimposed in-flight count.

Objective and service metrics. We use effective robot capacity (ERC), denoted N∗N^*, to measure the maximum number of concurrent robots a GPU can serve while meeting specified latency and reliability requirements. Each comparison fixes the model, workload, and acceptance criterion. Response latency measures feedback delay. SLO attainment is the fraction of responses meeting their supplied budgets; on-time goodput counts such responses per unit time. Section 6.1 specifies the acceptance criteria and measurement windows for each experiment.

Motivation

Sharing GPU capacity across independent robots must preserve timely feedback. The workload in Section 2 motivates three requirements: express changing feedback needs, expose independent work, and control concurrent interference.

Feedback Needs Change During Execution

In our delay-injection study, adding 200 ms before action delivery reduces completed parcel output by 29.6%. Separate fixed-count studies observe additional attempts for parcel reorientation and conveyor pushing, both contact operations. These results connect delayed feedback to reduced physical progress; Appendix A gives the protocols and results.

Acceptable delay varies during execution. Contact operations, such as grasping, parcel flipping, and conveyor pushing, or interaction with a changing scene may require frequent updates. Predictable free-space transport may tolerate a longer execution horizon. Increasing the RTC commitment length buys response time but postpones feedback; it cannot replace timely service when rapid adaptation is needed. A fixed priority per robot cannot capture these changes.

These observations call for per-request feedback requirements. Planned motion, execution progress, and application-specific signals, such as gripper events or contact observations, provide cues about how quickly feedback is needed. A budget-selection policy can map this execution context to a response-time budget for each request. The contract in § 2.3 exposes these budgets to the scheduler through a uniform interface.

Closed-loop throughput and goodput. By Equation (1), robot ii’s kkth response latency Li,kL_{i,k} determines send interval τi,k=max⁡(TE,Li,k)\tau_{i,k} = \max(T_E, L_{i,k}). Over robot ii’s requests, define mean cycle overrun hi=E[(Li−TE)+]h_i = \mathbb{E}[(L_i - T_E)^+] and attainment pi=Pr⁡(Li≤bi)p_i = \Pr(L_i \le b_i), where bib_i is the request’s budget and (x)+=max⁡(x,0)(x)^+ = \max(x, 0). For NN continuously active robots with stationary, ergodic request sequences and finite mean latency, long-run throughput XX and on-time goodput JJ satisfy

X=∑i=1N1TE+hi,J=∑i=1NpiTE+hi.(2)X = \sum_{i=1}^{N} \frac{1}{T_E + h_i}, \qquad J = \sum_{i=1}^{N} \frac{p_i}{T_E + h_i}. \tag*{(2)}

These long-run rates follow from one response per mean cycle TE+hiT_E + h_i, with fraction pip_i on time. Delaying wide-budget requests can lengthen request cycles even when responses remain on time. Higher attainment can therefore coexist with lower goodput; scheduling must preserve update frequency as well as deadline attainment. § 7.3 evaluates both effects.

Independent Requests Offer Parallelism

The streams in Figure 4 leave gaps in each dedicated instance yet produce bursts when combined. Sharing can fill these gaps by coordinating independent work. Each request follows ViT, Prefill, and Action dependencies, but CPU preparation can overlap GPU execution, and another request’s ViT or Prefill can progress during Action. Serializing complete requests or advancing stages in common rounds introduces waits that these dependencies do not require.

At 24 robots, enabling the full execution engine on π0.5\pi_{0.5} raises throughput from 54.90 to 131.53 req/s (2.40×, Figure 9). These results motivate exposing work as its dependencies complete.

Figure 9. Cumulative ablation on π0.5\pi_{0.5} at 24 robots. Bars show per-window means, rows retain prior mechanisms, and arrows show incremental gains. Execution-aware concurrency enables intra-stage concurrency. Cross-stage coordination combines Context–Action overlap and independent progression. Full engine: 2.40× throughput and 3.06× P50 speedup over All-Off.

Throughput Alone Does Not Protect Deadlines

Throughput gains need not improve tail latency. In a separate 32-robot π0.5\pi_{0.5} ablation, adding independent progression to multi-stream execution raises QPS by 9.7%, but P99 rises from 467.07 to 560.28 ms (Table 6).

NConfigurationQPSP50P95P99
8All-Off44.2620.0026.9232.46
M44.2519.5124.5030.54
M+C44.2619.8724.8227.34
M+R44.2520.4031.3833.66
M+G44.2719.6720.5421.08
M+C+R+G44.2719.8521.1921.71
16All-Off54.81305.80350.92364.93
M87.98145.05184.75208.85
M+C88.4485.0097.11103.96
M+R87.46152.56193.70224.76
M+G88.44109.63151.40172.86
M+C+R+G88.5139.4661.5373.70
32All-Off53.77581.14672.59691.47
M90.57351.93428.52467.07
M+C100.79327.11352.05362.57
M+R99.33311.91461.74560.28
M+G96.67327.68423.47475.40
M+C+R+G132.24235.18310.03378.06

Table 6. Incremental ablations on π0.5\pi_{0.5} using a multi-stream baseline. M denotes execution-aware concurrency, C CPU–GPU overlap, R independent progression, and G GPU-stage overlap. C, R, and G are enabled individually on top of M. All-Off disables all four mechanisms. QPS is measured in requests/s. P50, P95, and P99 are client response latencies in ms.

A wide-budget request’s Prefill can consume GPU resources needed by a tight-budget Action. Deadline ordering (Liu & Layland, [18]) chooses which queued request runs next, but moving the urgent request to the front does not release resources held by non-preemptive work. Urgency must account for remaining computation and interference, and starting another stage must depend on its effect on that work.

Protecting only requests already present can also be too late: a long-running stage may still be active when an earlier-deadline request arrives. In matched experiments with alternating task phases, arrival-aware admission raises mean SLO attainment by 5.0 and 11.3 percentage points at 16 and 18 robots, respectively (Table 8). Conversely, holding all other work whenever a request becomes urgent can forgo useful overlap.

WorkloadFSNo interference modelStrict gatingNo arrival-aware adm.Entry-only admissionZero-slack promotion
Phases, N=15N = 1596.49+24.29 (3)+7.73 (3)−0.40 (2)+4.87 (3)+0.27 (1)
Phases, N=16N = 1695.57+45.74 (3)+11.16 (3)+5.02 (3)+15.97 (3)+0.19 (1)
Phases, N=18N = 1895.96+45.47 (3)+21.34 (3)+11.31 (3)+12.13 (3)+1.04 (3)
Phases, N=20N = 2094.23+45.68 (3)+30.49 (3)+13.80 (3)+11.07 (3)−0.29 (1)
Bursty arrivals98.28+9.92 (3)+7.54 (3)+0.58 (2)+0.71 (2)+2.26 (3)
Multiple budgets96.01+14.24 (3)+22.29 (3)+1.47 (2)+5.64 (3)−0.25 (2)
Parcel, N=16N = 1694.78+56.62 (3)+18.55 (3)+16.97 (3)+13.77 (3)−0.77 (1)
Parcel, N=18N = 1888.77+51.44 (3)+29.90 (3)+18.05 (3)+11.12 (3)+2.01 (2)

Table 8. GPU component ablations. Strict gating jointly replaces slack stealing and arrival-aware admission. FS: SLO attainment of full FleetServe (mean of the nine rounds in the three campaigns, %). Other columns: gain of full FleetServe over each variant (percentage points, mean of three rounds paired with full FleetServe in the same campaign), with the number of rounds in which full FleetServe is higher. Bursty robot arrivals use the full measurement window.

The scheduler must therefore separate request protection from stage admission. Admission should permit useful overlap while keeping the deadlines of current and anticipated urgent requests predicted feasible. Urgency alone should not stop all other work.

Design

XInfer separates the availability of parallel work from the decision to execute it. An asynchronous engine makes work ready as its dependencies complete. A central scheduler identifies requests that need protection, then admits ordinary work only when the resulting interference preserves predicted deadline feasibility. This division addresses the tension in § 3: exploiting parallelism without delaying urgent action updates.

System Overview

The server consumes per-request response-time budgets and RTC inputs through the request contract in § 2.3. Budget assignment and execution-horizon selection are separate from the scheduler’s request prioritization and stage admission.

Figure 5(a) shows the ViT, Prefill, and Action pipeline, preceded by CPU preparation and followed by output processing. The engine maintains dependencies, ready queues, and bounded execution resources; the central scheduler controls admission at each stage entry. Solid arrows carry data, dashed arrows convey admission decisions, and dotted arrows return scheduler state.

XINFER architecture and illustrative execution

Figure 5. XINFER architecture and illustrative execution. (a) Central scheduling controls admission at the ViT, Prefill, and Action entries. (b) Admission permits concurrent work when A’s response is predicted to meet its deadline.

The policy makes two decisions: which requests need protection § 4.3 and which ready stages may start § 5. Both use request progress and shared resource state; the latter also considers arrival history. Issued GPU units are non-preemptive. § 6 specifies each backend’s execution boundaries.

In Figure 5(b), A has an urgent Action, B a ready Prefill, and C additional vision work; D is preceding ordinary work.

Asynchronous Stage Execution

Advance work by its dependencies. Each request follows ViT, Prefill, and Action dependencies without a common round barrier. Once Prefill supplies its conditioning state, Action can run alongside another request’s ViT or Prefill. CPU preparation and output processing can overlap GPU work from other requests. Admission controls interference before additional GPU work starts.

Separate submission paths. ViT and Prefill share a process; Action runs in another, with independent host submission loops and private execution state. This avoids serializing their host work through one submission loop. CUDA streams support concurrency within the ViT/Prefill process; MPS (NVIDIA, 2026) permits the processes to execute concurrently on the shared GPU. Requests reuse read-only weights and retain private intermediate state. Bounded state slots propagate backpressure when a downstream stage cannot accept work.

Dispatch admitted work. The engine dispatches admitted work at backend-supported execution boundaries. Each execution unit advances on its own completion events, without a common round barrier, and runs to completion. Request state remains private. § 6 maps computational stages to execution units and specifies their dispatch boundaries.

Request Prioritization

The scheduler protects queued and running requests according to their remaining slack and the work that could delay them.

Progress-aware service-time estimation. At server arrival aja_j, request jj with budget bjb_j receives scheduling deadline ℓj=aj+bj−δ\ell_j = a_j + b_j - \delta, where δ\delta reserves estimated time outside the server. Given projected concurrent stage counts nn, a calibrated model Rj(n)R_j(n) estimates remaining service time, covering the current stage’s residual, subsequent stages, and output processing, excluding future queueing. At time tt, the slack available for waiting and additional interference is

σj(n)=ℓj−t−Rj(n).(3)\sigma_j(n) = \ell_j - t - R_j(n). \tag*{(3)}

The predictor tracks each running stage’s progress by accumulating calibrated execution rates as concurrency changes, and refreshes RjR_j at each decision. Appendix § 14.§ 1 specifies the model and calibration; Appendix § 14.§ 2 evaluates remaining-time prediction for running stages.

Blocking-aware promotion. Following non-preemptive response-time analysis [4, 14], let BjB_j be the maximum predicted remaining time of another request’s running ordinary unit, or zero if none exists. When σj<Bj\sigma_j < B_j, waiting for that unit could exhaust jj’s slack, so the scheduler promotes jj. In Figure 6(a), A is promoted because its 10 ms slack is below BA=14B_A = 14 ms. Promotion persists while the request is live. Effective protection is withheld while the request is predicted late even in isolation; it remains eligible for service and can regain protection if the prediction changes.

Scheduling decisions: (a) priority promotion showing $A$: $\ell_A-t=50$ ms, $R_A=40$ ms, slack $\sigma_A=10$, ordinary running units, ViT 4, Prefill 14, Action 8, $B_A=\max(4,14,8)=14$ ms, $\sigma_A<B_A\Rightarrow$ Protect A; (b) shared-slack admission showing Protected A: shared slack = 10 ms, Check Prefill $u$, 6 extra delay and 4 left, $\Delta_A(u)=6\le10$, Admit $u$, Refresh state, Check ViT $v$, 4 left and +2, $\Delta_A(v)=6>4$, Defer $v$; (c) arrival-aware admission showing $w$ passes the current-request check, forecast $e$ has an earlier deadline $\ell_e<\ell_w$, $w$ if admitted now, Forecast, Arrival, Extra delay, Predicted miss $\Rightarrow$ Defer $w$

Figure 6. Scheduling decisions (illustrative predictions). (a) Protect AA using remaining slack. (b) Refresh shared slack after each launch. (c) Defer ww for an earlier-deadline forecast. Panel (b) fixes tt and H={A}H=\{A\}, with additive costs and other checks passing. Dashed outlines mark forecasts; hatching marks excess cost in (b) and extra delay in (c).

RefreshPriority⁡\operatorname{RefreshPriority} returns the effective protected set HH, including queued and running requests, and foreseeable load nfn_f: running units plus protected ready work that available resources can issue. Ready requests are ordered by protected status and then deadline.

SLO-Aware Admission

Protected work starts when resources permit; ordinary candidates must pass current-request and forecast checks to limit interference on urgent work. This adapts slack stealing [16] to concurrent GPU stages, using parallel capacity when the affected deadlines remain predicted feasible.

Current-request protection. Consider an ordinary candidate unit uu at stage ss, not yet included in foreseeable load nfn_f. At a fixed decision time, let n+=nf+esn^+=n_f+e_s add that unit. Its predicted interference on protected request hh and the resulting slack satisfy

Δh(u)=Rh(n+)−Rh(nf),\Delta_h(u)=R_h(n^+)-R_h(n_f),
σh(n+)=σh(nf)−Δh(u).(4)\sigma_h(n^+)=\sigma_h(n_f)-\Delta_h(u). \tag*{(4)}

CURRENTFEASIBLE accepts exactly when Δh(u)≤σh(nf)\Delta_h(u)\le\sigma_h(n_f) for every h∈Hh\in H.

Local property. A passing candidate leaves the current protected set predicted feasible under this decision’s snapshot. Proof. The inequality gives σh(n+)≥0\sigma_h(n^+)\ge0, hence t+Rh(n+)≤ℓht+R_h(n^+)\le\ell_h for every h∈Hh\in H.

A failed check leaves the candidate queued while other ready work is considered.

Coordinate decisions across stages. Arrivals, completions, and periodic ticks trigger Algorithm 1. The driver visits Action, Prefill, and ViT. SCANREADY starts from priority-ordered ready work and yields candidates while the stage has capacity, retaining deferred work for later scans. Each launch updates shared execution state and refreshes protection before the next check. Figure 6(b) shows why: under the illustrative additive predictor, admitting uu consumes 6 ms of A’s 10 ms slack, leaving 4 ms. Candidate vv would add another 6 ms and is deferred. Independent checks against the original snapshot would admit both. Appendix § 14.§ 1 specifies the priority update and stage iteration rules.

Algorithm 1 SLO-aware stage dispatch

Require: State SS: ready queues, running work, arrival history

  1. Record arrivals and completions in SS

  2. for s∈[Action,Prefill,ViT]s\in[\mathrm{Action},\mathrm{Prefill},\mathrm{ViT}] do

  3. (H,nf)←REFRESHPRIORITY(S)\quad(H,n_f)\leftarrow\mathrm{REFRESHPRIORITY}(S)

  4. \quadfor uu in SCANREADY(S,s,H)\mathrm{SCANREADY}(S,s,H) do

  5. \qquadif request⁡(u)∉H\operatorname{request}(u)\notin H then

  6. n+←nf+es\qquad\quad n^+\leftarrow n_f+e_s

  7. \qquad\quadif not CURRENTFEASIBLE(H,n+)\mathrm{CURRENTFEASIBLE}(H,n^+) then

  8. \qquad\qquadcontinue {protect current requests}

  9. \qquad\quadend if

  10. \qquad\quadif not FORECASTFEASIBLE(u,n+)\mathrm{FORECASTFEASIBLE}(u,n^+) then

  11. \qquad\qquadcontinue {protect forecast arrivals}

  12. \qquad\quadend if

  13. \qquadend if

  14. \qquad\mathrm{ISSUE}(u)$; update queues and running state in SS

  15. (H,nf)←REFRESHPRIORITY(S)\qquad(H,n_f)\leftarrow\mathrm{REFRESHPRIORITY}(S)

  16. \quadend for

  17. end for

Arrival-aware admission. In Figure 6(c), ww passes the current-request check but would delay forecast request ee past its earlier deadline. FORECASTFEASIBLE uses recent arrivals grouped by response budget to forecast requests during the candidate’s execution, retaining those whose deadlines precede the candidate’s. Each retained forecast must satisfy δ+f+RV(ne′)≤be\delta+f+R_V(n'_e)\le b_e, where ff reserves front-end time, RVR_V estimates the remaining pipeline from ViT, and ne′n'_e includes the candidate and preceding retained forecasts.

Implementation

Model adapters connect input preparation, kernels, CUDA graphs, and output processing to the runtime. VLA-2B and Wall-OSS-0.5 use MXFP8 ViT/Prefill linear operations [21] and BF16 Action; π0.5\pi_{0.5} uses native FlashRT FP8 with explicit state-text conditioning (Appendix § 13.§ 1). The central scheduler dispatches each stage separately; the three-model backends combine ViT and Prefill as Context. Each Action dispatch runs a complete, non-preemptive denoising trajectory with fixed request membership; central Action dispatches contain one request. Client timestamps measure response latency; server timers and event traces diagnose execution. Runtime controls toggle GPU-stage overlap, cohort barriers, and CPU–GPU overlap (§ 7.4). Appendix § 14.§ 1 records the central predictor’s calibration and constants.

Evaluation

Experimental Setup

Models and serving comparisons. Each serving run uses NVIDIA consumer-grade Blackwell GPUs. The three-model study covers VLA-2B, Wall-OSS-0.5, and π0.5\pi_{0.5}. VLA-2B uses a Qwen3.5-2B backbone and non-public weights. We use requests representative of our deployment setting and keep inputs fixed across serving methods within each model. Each request contains three camera views, a shared language instruction, and a 14-dimensional synthetic robot state. Table 1 summarizes the input shapes and actual prefix lengths. All models encode 868 image patches into 217 visual tokens; text lengths reflect their native tokenizers and templates. Appendix § 13.§ 1 details input preprocessing. For each model, we compare XINFER with integrations of SGLang (Zheng et al., [32]) and vLLM-Omni (Yin et al., [27]), FlashRT (served through its Nexus frontend) (Su, 2026), and a Robion single-GPU reimplementation (Adamopoulos et al., [1]). The latter combines true stage batches, lockstep progression, and calibrated Context SM quotas; Context group gg overlaps Action group g−1g - 1, with a barrier between rounds. Comparisons share the checkpoint, workload, precision, and compute backend. Appendix § 13.§ 3 lists the Robion reproduction settings; Appendix § 13.§ 1 provides the measurement protocol.

InputVLA-2BWall-OSS-0.5π0.5\pi_{0.5}
Camera 1 (pixels)224×352224 \times352196×308196 \times308196×308196 \times308
Cameras 2/3 (pixels)224×320224 \times320196×280196 \times280196×280196 \times280
Patch size (pixels)16×1616 \times1614×1414 \times1414×1414 \times14
Visual tokens217217217
Instruction tokens414140
State tokens7274127
Template/special tokens11611311
Full prefix446445395

Table 1. Input configuration. Token segments sum to the full prefix; action tokens and sequence padding are excluded.

Central-scheduling comparison. All SLO policies share the three-stage VLA-2B backend, model, kernels, and allocated GPU resources. FIFO orders ready requests by arrival. EDF orders each stage queue by deadline with fixed per-stage concurrency limits. Slack-Gate uses deadline queues and defers non-urgent work when another request has less than 40 ms estimated slack. Appendix § 14.§ 4 details baseline configurations and parameter selection. Policy-specific bounds and admission decisions determine actual concurrency. XINFER applies the same prediction and admission rules across workloads. Matched policy comparisons use the same GPU.

Load and metrics. Each emulated robot follows a closed-loop request stream with at most one request in flight and a minimum send interval of 180 ms (Equation (1)). Response latency spans sending a packed request to receiving the complete response; capture, packing, and parsing are excluded. The three-model runs use ten denoising steps and 16 future action points. The RTC commitment length is fixed at d=8d = 8 in both studies; response-time budgets in the central-scheduling replay follow each workload. Both serving studies measure requests over 180 s after 30 s of warmup. We report ERC as the largest validated integer robot population. The three-model comparison requires zero request errors and response P50 below 150 ms in all three windows; the central-scheduling study requires at least 95% SLO attainment. Three-model OPS counts successful requests sent and completed within the window; central-scheduling goodput counts on-time responses to requests sent within it. Both rates divide these counts by 180 s. Appendix § 13.§ 1 details aggregation and measurement-window boundaries; Appendix § 13.§ 2 and Appendix § 14.§ 4 detail ERC validation.

ERC criteria and physical task throughput. We apply the deployment-derived 150 ms threshold across models: 50 ms baseline response plus 100 ms injected delay, near the delay–throughput knee (Appendix § 12.§ 1). For heterogeneous budgets, the 95% attainment target defines a common high-attainment operating point allowing at most 5% deadline misses; goodput captures the accompanying response-rate tradeoff. Figure 2 projects fleet parcel throughput from measured serving latency and physical calibration (Appendix § 12.§ 1).

Figure 2. Projected parcel throughput on one GPU (VLA-2B), combining measured latency with single-robot trials. Guides mark 13k parcels/h; ERC requires response P50<150P50 < 150 ms and zero request errors. Shading reflects trial variation; dashed curves and hatching mark extrapolation.

Serving-System Comparison

XINFER achieves ERCs of 18, 15, and 24 robots across the three models, exceeding all compared systems (Table 2). Compared with Robion’s 11, 10, and 19 robots, these ERC values are 63.6%, 50%, and 26.3% higher.

Serving methodVLA-2BWall-OSS-0.5π0.5\pi_{0.5}
XINFER181524
SGLang10514
vLLM-Omni10815
FlashRT549
Robion111019

Table 2. ERC (robots) under the response-latency criterion.

For each model, all five systems are evaluated at the same robot counts. At low load, throughput follows the aggregate request rate; after saturation, responses delay subsequent sends and latency rises sharply (Figure 7). Appendix § 13.§ 2 details ERC validation.

Figure 7. Serving performance across three models. Latency uses a log scale; dashed lines mark 150 ms. Large markers and guides identify confirmed ERCs for XINFER and Robion.

These complete-system comparisons combine multiple mechanisms; § 7.4 separates their contributions within a common runtime.

Serving Heterogeneous SLOs

Alternating task phases. In our block-stacking tasks, alignment and contact are more sensitive to delayed feedback than transport. Motivated by this observation and the recorded phase patterns (Appendix § 14.§ 3), we alternate each robot’s response budget between 90 and 400 ms every 30 completed requests, with staggered robot phases and the closed-loop trigger in Equation (1). We report SLO attainment, on-time goodput, and mean wide-budget response latency.

XINFER achieves an ERC of 18 robots under the 95% SLO criterion, compared with 15 for Slack-Gate, 14 for FIFO, and 12 for EDF (Figure 8). EDF degrades gracefully under overload: it keeps 88.6–90.8% attainment from 16 to 20 robots, where FIFO falls to about 50%, and exceeds Slack-Gate at 20 robots (88.6 versus 75.9%). Appendix § 14.§ 4 gives detailed results. At 15 robots, both XINFER and Slack-Gate satisfy the 95% overall SLO target. XINFER reduces mean wide-budget latency by 33% and increases goodput from 69.3 to 80.2 responses/s. The closed-loop relation in Equation (2) explains how shorter request cycles can increase goodput alongside improved attainment.

Figure 8. Four workloads with matched resources: (a–c) show three metrics for alternating phases; (d–f) show attainment for the other workloads. Curves show means with min–max whiskers. Large markers denote ERC in (a) and 33% lower mean latency for XINFER versus Slack-Gate at 15 robots in (b). Shading in (d) marks added robots’ measured active periods.

Bursty robot arrivals. To test transient overload, we start with 12 robots and add six robots with 90 ms budgets at 60 s and again at 120 s, each issuing 55 requests. Figure 8(d) shows the resulting attainment over time. Within the designated 60–80 s and 120–140 s windows, XINFER attains 96.6%, versus 95.9% for EDF and 85.1% for Slack-Gate.

Multiple response budgets. To broaden the budget distribution, 15 robots cycle through 80, 120, 160, 250, and 400 ms budgets every 30 responses, with staggered phases. XINFER attains 96.1% overall versus 95.1% for EDF and 92.5% for Slack-Gate, with the per-budget breakdown in Figure 8(e). Appendix § 14.§ 4 reports per-round results and aggregation details.

Parcel handling. The parcel-handling workload derives response-indexed budget sequences from gripper events in 23 episodes recorded in production. Sequences advance with successful responses, yielding 61–63% tight requests and intervals in which all robots carry tight budgets (Appendix § 14.§ 6). Under the 95% SLO criterion, XINFER achieves an ERC of 14 robots, versus 12 for Slack-Gate and 10 for FIFO. EDF has no confirmed ERC within the tested range of 10–20 robots (Figure 8(f)). From 16 to 20 robots, observed SLO attainment gains for XINFER over Slack-Gate span 3.8–11.4 percentage points, while wide-budget responses return 7–39 ms sooner.

Execution Engine Ablations

CPU–GPU overlap in Figure 9 permits host work during GPU execution, but retains serial GPU execution. Execution-aware concurrency enables the selected intrastage multi-stream mode, raising throughput from 59.52 to 104.23 req/s (1.75×). Cross-stage coordination then combines Context–Action overlap with independent progression, reaching 131.53 req/s (1.26×) and reducing P50 from 229.45 to 142.35 ms (1.61× speedup). These increments depend on the enable order.

Closed-loop triggering limits throughput headroom. The 180 ms minimum send interval caps 24 robots at 24/0.18≈133.3324/0.18 \approx133.33 req/s, only 27.9% above the penultimate configuration. The full engine reaches 131.53 req/s. Responses arriving before the next eligible trigger cannot advance the next request (Equation (1)), so latency reductions need not yield proportional QPS gains. Appendix § 13.§ 4 details the controls and protocol.

Scheduler Analysis

Policy controls and component ablations. Figure 10 compares four controls with full XINFER. Removing interference modeling replaces concurrent-stage predictions with isolated durations, lowering mean attainment by 45.5 and 51.4 percentage points for alternating phases and parcel handling at 18 robots. Strict priority gating blocks ordinary work whenever a protected request exists, reducing attainment by 7.5–30.5 points across eight settings. Arrival-aware and stage-wise admission primarily protect tight-budget requests: disabling arrival-aware checks or checking only before ViT weakens this protection as load approaches saturation. The losses grow with the tight-budget share and concentrate within bursts; strict gating instead starves ordinary work. Appendix § 14.§ 3 reports workload-level results and the latency and goodput effects of arrival-aware admission.

Figure 10. Scheduler ablations. Mean SLO attainment drop relative to full XINFER for each scheduler variant. Appendix § 14.§ 3 gives workload settings.

Running-stage prediction. Causal replay of recorded stage transitions at 8/18/20 robots uses only the concurrency history available at each sample. Mean absolute errors for the remaining execution time of running stages are 2.0/2.6/2.8 ms. Appendix § 14.§ 2 details the measurement protocol and remaining-time error distributions.

Scheduling overhead. In chronological CPU replay, a policy update followed by an admission check takes 73/77 μ\mus at P99 with 15/18 robots, below 0.1% of the 90 ms response budget. This measures CPU policy time, not added response latency; GPU execution proceeds asynchronously. Appendix § 14.§ 3 gives the replay protocol.

Related Work

Robot and VLA serving. Prior robot-serving systems use execution context to guide scheduling: Kairos adapts execution horizons, Armory uses action-queue state, and ROSA coordinates placement, batching, and request rates [9] (Anonymous Authors, n.d.; Jiang et al. 2026). Robion overlaps encoding and action diffusion with fixed stage batches and lockstep progression, and selects model queues by their head requests’ SLOs (Adampoulos et al., 2026). XINFER also exploits stage overlap, but advances requests independently and separates request priority from interference-aware stage admission.

Stage-aware inference. Iteration-level scheduling and chunked prefill improve LLM serving ([2] (Yu et al. 2022)), while DistServe, Splitwise, and semi-PD separate prefill and decoding resources ([33, 20, 12]). In VLA serving, encoding and denoising use different model components, and closed-loop control couples response time to subsequent arrivals. Optimized inference runtimes and model-level work reduction ([25] (Su 2026; Su et al. 2026; Xu et al. 2026)) complement our service; RTC maintains action-chunk continuity during ongoing motion ([6]). XINFER coordinates shared execution using per-request response-time budgets.

SLO-aware GPU sharing. Clockwork uses predictable execution to meet request latency targets ([10]). SLOs-Serve allocates work to meet application- and stage-specific SLOs ([7, 13]), while Orion limits interfering kernels and REEF supports controlled kernel preemption ([22, 11]). Our admission tests adapt slack stealing and non-preemptive blocking analysis from real-time scheduling ([16, 4, 14]). They use request progress, calibrated concurrent interference, and arrival history to protect current and anticipated deadlines at VLA stage boundaries. Issued stages remain non-preemptive, and feasibility is predicted rather than established by worst-case timing bounds.

Conclusion

XINFER is a production-deployed VLA serving system that coordinates shared GPU computation with the changing feedback needs of robotic execution. Its asynchronous engine exposes parallelism across requests and stages, while execution-aware scheduling separates request priority from stage admission to control interference on current and anticipated urgent requests. Across three VLA models, XINFER achieves 1.26–3.75× the effective robot capacity of the compared systems under a common response-latency criterion. Under heterogeneous SLOs, it also improves attainment and on-time goodput. These results demonstrate how coordinating parallelism and interference around execution-dependent feedback requirements enables more efficient serving of robot fleets.

Contributors

Fleetserve is a collaborative effort of the X Square Robot team. The full contributor list is given below; * denotes core contributors, † denotes the project lead, and ‡ denotes the corresponding author.

Yue Zhang*, Zicong Ye*, Lufang Chen†, Dance Yang, Mingyang Li, Zhenwei Duan, Michael Zhao, Ziye Deng, Starrick Liu, Ruizhi Chengze, YongHua Chen, SongAo Han, Yu Sun, Ran Yu, Xiaofan Li, Ruyi Gan, Hao Wang‡

AI USAGE DISCLOSURE

The authors led the system and experimental designs. We used OpenAI ChatGPT/Codex and Anthropic Claude to generate and iteratively revise implementation code and experimental, analysis, and plotting scripts based on the authors’ specifications. The tools also assisted with literature review and drafting and revising multiple manuscript sections. The authors determine the final designs and content and take full responsibility for the accuracy and integrity of the paper.

Response Latency and Physical Task Efficiency

Sustained Parcel Handling

Protocol and latency criterion. We use one robot and VLA-2B BF16 (execute_horizon=7), holding inference settings fixed. Each of six added-delay settings has three five-minute trials. Delay is inserted between inference completion and response delivery; baseline response latency is approximately 50 ms. Table 3 distinguishes all handled parcels from those reaching the required task endpoint.

Added delay (ms)Handled parcels/hCompleted parcels/hHandled SD
01968192086.5
251944191243.3
502020196873.3
1001892181266.1
1501552151696.2
20013761352129.4

Table 3. Hourly parcel rates under added response delay. Handled and completed are measured aggregate rates; handled SD is estimated from three trial markers digitized from the source plot. Figure 2 uses handled rates.

Completed output falls 29.6% at 200 ms added delay. Handling throughput drops 6.3% from 50 to 100 ms, then 18.0% from 100 to 150 ms. Selecting 100 ms before this steeper loss, plus the 50 ms baseline, motivates the 150 ms response-P50 criterion (§ 7.1).

Predicted parcel-handling throughput. Figure 2 maps serving latency to fleet parcel throughput for 1–16 robots. Let q(Δ)q(\Delta) interpolate the handled rates in Table 3, and R~s(N)\widetilde{R}_s(N) interpolate mean per-run response P50s across robot counts for system ss (Figure 7). Both interpolations are piecewise linear. Up to 250 ms response latency,

Q^s(N)=Nq(max⁡{0,R~s(N)−50 ms}).(5)\widehat{Q}_s(N) = N q\left(\max\{0,\widetilde{R}_s(N)-50\ \mathrm{ms}\}\right). \tag*{(5)}

Subtracting 50 ms removes baseline latency; faster responses use the zero-injection rate. FlashRT’s measured P50 reaches 505.4 ms at 16 robots. Beyond 250 ms, dashed curves extrapolate per-parcel handling time:

qext(Δ)=3600t200+k(Δ−200),Δ>200.(6)q_{\mathrm{ext}}(\Delta) = \frac{3600}{t_{200}+k(\Delta-200)}, \qquad\Delta> 200. \tag*{(6)}

Here Δ\Delta is added delay in milliseconds, qextq_{\mathrm{ext}} is in parcels/h, and t200=3600/1376=2.6163t_{200}=3600/1376=2.6163 seconds per parcel. The slope k=0.0071353k=0.0071353 seconds per millisecond is fitted to 3600/q(Δ)3600/q(\Delta) at 100, 150, and 200 ms using only physical calibration data. Adjacent-interval slope fits give FlashRT 12,137–13,940 parcels/h at 16 robots, so its comparison with seven FleetServe robots depends on the extrapolation slope.

Uncertainty and scope. Using Table 3’s sample SDs sjs_j, we linearly interpolate variance v(Δ)v(\Delta) within calibration and share N[q(Δ)±v(Δ)]N[q(\Delta)\pm\sqrt{v(\Delta)}], using s02s_0^2 below baseline. Beyond 250 ms, independent sample-SD perturbations of the calibration means qˉ100\bar{q}_{100} and qˉ200\bar{q}_{200} propagate through both the anchor and fitted slope, giving first-order half-width

B(N,Δ)=N∑j∈{100,200}(∂qext∂qˉj)2sj2.(7)B(N,\Delta)=N\sqrt{\sum_{j\in\{100,200\}}\left(\frac{\partial q_{\mathrm{ext}}}{\partial\bar{q}_j}\right)^2s_j^2}. \tag*{(7)}

Serving P50s remain fixed. These bands quantify sensitivity to single-robot variation, not confidence intervals or fleet-level SDs; they exclude digitization, latency, and model-form uncertainty.

The projection transfers BF16, horizon-7 calibration to MXFP8 Context/BF16 Action with RTC d=8d=8, assuming identical, independent robots with fixed task mix and control behavior. It treats P50 as constant delay and omits jitter and shared physical bottlenecks; it is distinct from measured multi-robot task output and ERC.

Manipulation Attempts by Parcel Type

Protocol. We evaluate parcel reorientation (barcode initially down) and conveyor pushing (barcode initially up) with VLA-2B BF16 and execute_horizon=7; baseline model latency is approximately 45 ms. Each task tests the four parcel types in Table 4, with five trials per type at each added delay from 0 to 500 ms in 50 ms increments. A pushing attempt is one complete push; injected delay differs from full response latency.

Added delay (ms)Carton AGray bagCarton BLong carton CMean
06/55/55/55/51.05/1.00
506/55/56/56/51.15/1.00
1005/56/86/57/51.20/1.15
1506/75/87/66/51.20/1.30
2006/66/78/66/51.30/1.20
2508/56/86/55/61.25/1.20
3007/710/69/67/51.65/1.20
35011/88/89/77/71.75/1.50
4007/89/115/55/101.30/1.70
45012/97/115/55/81.45/1.65
5009/88/76/76/101.45/1.60

Table 4. Manipulation attempts (flip/push) by parcel type under added response delay. Type columns give totals over five tasks; Mean gives averages over twenty tasks.

Effect on manipulation effort. Attempt counts vary by parcel type and need not increase monotonically with delay (Figure 11). Mean flip attempts rise from 1.05 without added delay to 1.75 at 350 ms; mean push attempts rise from 1.00 to 1.50–1.70 at 350–500 ms. Table 4 gives the complete counts.

Figure 11. Response delay increases physical effort, with different effects across parcel types. Each point averages five tasks of one type; the black line is the mean of the four types and the band their range. Zero denotes no added delay. Full counts are in Table 4.

Serving Evaluation: Detailed Results

Protocol and Configuration

Measurement and aggregation. Section 6.1 defines the workload and timing boundaries. Warmup uses the target robot population. The 180 ms request period corresponds to six 30 ms model steps in production. Latency statistics include responses completed after the measurement window for requests sent within it. We average per-window P50/P95/P99 and QPS; whiskers show the observed min–max range. Each repeat restarts the client, while the model service may remain resident.

RTC handoff. After freezing the first dd keyframes, RTC [6] guides subsequent keyframes toward the remaining old trajectory with decaying weight and generates the rest freely. Excessive commitment is over-freezing; a response after tk+dΔt_k+d\Delta causes a handoff violation, whereas starvation means available actions run out before replacement. A trigger miss occurs when sj+TEs_j+T_E passes while request jj remains in flight (Equation (1)).

Input preprocessing. VLA-2B and Wall-OSS-0.5 merge image patches in their trained encoders. π0.5\pi_{0.5} instead averages non-overlapping 2×22 \times2 patch blocks after the full ViT, producing 77, 70, and 70 tokens across the three cameras. Equal visual-token counts do not imply equal total FLOPs. Its native FlashRT backend uses per-tensor FP8 E4M3 with BF16 inputs and outputs; Section 5 specifies the other precision settings.

The shared instruction retains its original metadata. Wall-X converts, normalizes, and discretizes the all-zero 14-dimensional state into 20 values, encoded as 72/74 tokens for VLA-2B/Wall-OSS-0.5. π0.5\pi_{0.5} uses identity normalization and its native state-as-text formatter, padding to 32 dimensions and discretizing into 256 bins. This synthetic state is not a calibrated physical-robot state policy. State labels, punctuation, and special markers belong to the template column of Table 1.

Serving interfaces. SGLang and vLLM-Omni retain their queues. FlashRT uses its native single-consumer model session over HTTP with JSON/base64; the other paths use WebSocket with MessagePack. Transport costs are included in response latency. π0.5\pi_{0.5} enables Context-completion notification.

ERC Validation

We apply the criterion in § 7.1 to unrounded window-level values: every window at the reported ERC must pass, and the adjacent larger population must fail in at least one window. For π0.5\pi_{0.5}, XInfer passes at 24 robots with P50s of 133.470–136.592 ms and mean QPS 132.31; all three windows at 25 robots fail (163.104–166.842 ms). VLA-2B Robion at 12 robots and π0.5\pi_{0.5} Robion at 20 robots each pass in only two of three windows, despite passing on mean P50.

Robion Reproduction Settings

Table 5 specifies our single-GPU reproduction of Robion [1].

SettingConfiguration
ScopeSingle-model serving; distributed placement and routing are excluded.
Execution unitsContext and Action each use a sequential CUDA stream with a dedicated CPU submission thread.
Round progressionContext work in round kk overlaps Action work from round k−1k-1; the next round starts after both finish.
Result publicationPublish completed Action results after the round barrier (a reproduction choice); CPU output processing may overlap the next GPU round.
SM allocationAn offline lookup selects the Context SM quota that minimizes full-round CUDA completion time; Action remains unrestricted.
CUDA GraphsCapture supported execution-unit and Context SM configurations before measurement.
CPU preparationKeep input preparation pipelined with GPU execution.

Table 5. Robion configuration used in the serving-system comparison.

Execution Mechanisms

Cumulative protocol. Figure 9 follows the sequence in § 7.4 at 24 robots with fixed inputs, backend, and resources. All-Off and all-enabled each use two windows, while intermediate configurations use one, totaling six displayed windows under Appendix § 13.§ 1. Values average per-window P50 and QPS. Execution-aware concurrency enables intra-stage concurrency with the round barrier retained. Cross-stage coordination combines Context–Action overlap with independent progression, removing that barrier. Gains depend on the enabling order. This study does not re-estimate ERC.

Incremental controls. Table 6 evaluates six configurations at 8, 16, and 32 robots, each with a separate service start and one window. All-Off serializes GPU stages, allows one request per stage, and retains a round barrier. M denotes execution-aware concurrency with the selected intra-stage streams, concurrency, and batching limits. C overlaps CPU preparation and output processing with GPU work. R removes only the round barrier. G overlaps combined ViT–Prefill (Context) with Action on the GPU. R and G are the two components of cross-stage coordination. M+C, M+R, and M+G each add one mechanism to M. All-enabled combines them. Lockstep states share a round-admission rule, and completed results can return without waiting for the barrier. Inputs, weights, precision, kernels, workspaces, KV storage, and captured graphs remain fixed with completion notification enabled.

Throughput and tail latency. At 32 robots, M+R improves QPS and P50 over M while increasing P95/P99. All-enabled remains above the capacity latency threshold (Table 6).

GPU execution timelines. Nine separate profiler runs compare three states per model at N=32N = 32: independent progression with CPU overlap disabled/enabled, and lockstep with CPU overlap enabled. Environments and configurations match within each model; profiler configurations differ from the ablations above. CPU controls toggle preparation/output overlap with GPU work; progression controls only the round barrier, retaining cross-stage overlap. Two host-clock markers align the traces; Table 7 summarizes 61–69 s and Figure 12 shows an excerpt. With CPU overlap

Figure 12. CPU–GPU overlap reduces idle gaps in a separate profiler diagnostic. VLA-2B serves 32 robots with overlap (a) disabled and (b) enabled. The figure shows the 61–61.5 s excerpt; the outlined gap marks an observed GPU idle interval. Table 7 reports statistics for the full profiling window.

ModelStateEitherBothCPU/GPU
VLA-2BInd., off85.4958.360.00
Ind., on99.7279.7356.85
Lock., on90.7925.4511.28
Wall-OSS-0.5Ind., off82.0555.350.00
Ind., on98.7887.7826.88
Lock., on87.5568.9914.49
π0.5\pi_{0.5}Ind., off72.7919.080.00
Ind., on94.7328.8019.60
Lock., on72.4418.264.04

Table 7. Wall-time shares (%) at N=32N = 32 over 61–69 s. Ind./Lock.: independent/lockstep; off/on: CPU overlap. Either/Both: union/intersection of Context/Action kernel intervals. CPU/GPU: kernel overlap with instrumented CPU preparation/output scopes.

disabled, CPU/GPU intersection is zero at display precision (below 0.001% for π0.5\pi_{0.5}). These shares measure activity, not SM occupancy or MFU; profiler throughput is excluded from ERC.

SLO-AWARE SCHEDULING: MODEL AND DETAILED RESULTS

Prediction and Dispatch Model

The central backend schedules non-preemptive units at three boundaries: ViT, post-vision Prefill, and a complete Action denoising trajectory. Its evaluation uses batch size one. The response-latency SLO bjb_j is converted to a server deadline ℓj=aj+bj−δ\ell_j = a_j + b_j - \delta, where aja_j is server arrival time and δ=4\delta= 4 ms reserves estimated time outside the server. The request does not carry its robot-side send timestamp.

Progress-aware remaining-time model. For stages s∈{V,P,A}s \in\{V, P, A\} in pipeline order, define

qs(n)=hs+Wsρs(nˉ),nˉs=min⁡(ns,cs).(8)q_s(\mathbf{n}) = h_s + \frac{W_s}{\rho_s(\bar{\mathbf{n}})}, \qquad\bar{n}_s = \min(n_s,c_s). \tag*{(8)}

WsW_s is calibrated stage work in isolation-time units, ρs\rho_s is a dimensionless rate table, and hsh_s is stage-issue overhead. Every rate lookup is clipped to the calibrated bounds c\mathbf{c}. For a running stage ss, the supplied counts include request jj’s own unit; its rate therefore uses n−j=[n−es]+\mathbf{n}^{-j} = [\mathbf{n} - \mathbf{e}_s]_+, where es\mathbf{e}_s is the unit vector for stage ss and the positive part is componentwise. Waiting and downstream stages use qs(n)q_s(\mathbf{n}) without subtraction. In foreseeable load nf\mathbf{n}_f, protected ready units already included in the scheduler remain in this lookup, even for their own requests. The supplied load approximates future concurrency.

Let nr(u)\mathbf{n}_r(u) be the scheduler-observed running counts at time uu, including request jj’s own unit, and define n−j(u)=[nr(u)−es]+\mathbf{n}^{-j}(u) = [\mathbf{n}_r(u)-\mathbf{e}_s]_+. For a stage issued at τj\tau_j, its modeled compute start is bj,s=τj+hsb_{j,s} = \tau_j + h_s. Completed work and residual time are

w^j,s(t)=∫bj,smax⁡(t,bj,s)ρs(n−j(u)) du,\widehat{w}_{j,s}(t) = \int_{b_{j,s}}^{\max(t,b_{j,s})} \rho_s(\mathbf{n}^{-j}(u))\,du,
r~j,s(t,n)=[bj,s−t]++[Ws−w^j,s(t)]+ρs(n−j).(9)\widetilde{r}_{j,s}(t,\mathbf{n}) = [b_{j,s}-t]_+ + \frac{[W_s-\widehat{w}_{j,s}(t)]_+}{\rho_s(\mathbf{n}^{-j})}. \tag*{(9)}

Here [x]+=max⁡(0,x)[x]_+ = \max(0,x). The integral is a sum over intervals between observed stage transitions. Historical rates determine completed work; the supplied load determines the rate used for the remaining work. The remaining pipeline time is

Rj(n)=hout+rj,s(n)+∑k>sqk(n),(10)R_j(\mathbf{n}) = h_{\mathrm{out}} + r_{j,s}(\mathbf{n}) + \sum_{k>s} q_k(\mathbf{n}), \tag*{(10)}
rj,s(n)={qs(n),waiting,r~j,s(t,n),running.r_{j,s}(\mathbf{n}) = \begin{cases} q_s(\mathbf{n}), & \text{waiting},\\ \tilde{r}_{j,s}(t,\mathbf{n}), & \text{running}. \end{cases}

The term houth_{\mathrm{out}} covers output processing. A completion event advances the request to its next stage. Future stages use the same projected load, without an additional prediction of future queueing delay. The counters describe scheduler in-flight units, including observation delay.

Calibration. The rate table comes from a microbenchmark with RTC d=8d = 8 and B1 Action trajectories. It measures isolated and mixed-stage execution at 167 concurrency combinations, repeated three times per calibration instance. Rates are derived from isolated work divided by measured unit duration. The frozen table incorporates stage corrections and a common work factor of 1.02 fitted during simulator calibration. This yields (WV,WP,WA)=(4.328931,8.907749,12.116308) ms(W_V, W_P, W_A) = (4.328931, 8.907749, 12.116308)\,\mathrm{ms}. The model uses (hV,hP,hA)=(2.1,0.25,1.1) ms(h_V,h_P,h_A) = (2.1, 0.25, 1.1)\,\mathrm{ms} and hout=1.2 msh_{\mathrm{out}} = 1.2\,\mathrm{ms}; hAh_A includes preparation and KV copying. The front-end reserve f=8.5 msf = 8.5\,\mathrm{ms} is added only for forecast arrivals below. These parameters were frozen before the reported final-policy GPU comparisons and are not fitted to their outcomes.

Current-request protection. Algorithm 2 implements the promotion and protection rules of § 4.3, refreshing the effective protected set HH and foreseeable load nfn_f as the promoted set PP grows. Ordinary dispatch must preserve every protected request’s predicted feasibility; these predictions are not worst-case timing bounds or SLO guarantees.

Arrival-aware admission. Recent arrivals are grouped by observed SLO value. For a value seen k>0k > 0 times in the rolling window P=180 msP = 180\,\mathrm{ms}, forecasts start at max⁡(t,alast+P/k)\max(t, a_{\mathrm{last}} + P/k) and repeat every P/kP/k, without robot identities or predefined tight/wide classes. Before issuing ordinary unit uu, FORECASTFEASIBLE (§ 5) checks forecasts during uu’s estimated execution, retaining only those with ℓe<ℓu\ell_e < \ell_u. Its projected load ne′\mathbf{n}'_e adds uu to foreseeable load and accumulates earlier retained ViT forecasts. The front-end reserve is f=8.5 msf = 8.5\,\mathrm{ms}; physical resources still bound concurrency.

Termination of priority refresh. With time, live requests J\mathcal{J}, and execution state fixed, each nonempty update only adds members to PP, so the loop terminates within ∣J∣−∣P0∣+1|\mathcal{J}| - |P_0| + 1 evaluations including the final empty update, where

Algorithm 2 Refresh protected requests and foreseeable load
Require: State S: live requests 𝒥, running units ℒ, promoted set 𝒫, and current time t
1: 𝒫 ← 𝒫 ∩ 𝒥
2: ℒ ← {j ∈ 𝒥 : ℓ_j − t < R_j(0)}
3: repeat
4:   H ← 𝒫 \ ℒ
5:   n_f ← FORESEEABLELOAD(S, H)
6:   Compute B_j for each j ∈ 𝒥 from other requests’ low-priority units
7:   𝒰 ← {j ∈ 𝒥 \ 𝒫 : σ_j(n_f) < B_j}
8:   𝒫 ← 𝒫 ∪ 𝒰
9: until 𝒰 = ∅
10: Save 𝒫 in S; return H, n_f

P0P_0 is the initial promoted set after intersection with J\mathcal{J}, without requiring monotonic calibrated rates.

In Algorithm 1, SCANREADY initially orders each stage’s ready work by protected status, deadline, then request ID. After each launch it updates occupancy and refreshes priority, Action retains its remaining order, while Prefill and ViT rebuild it. Failed checks advance the scan while leaving candidates queued; both checks apply only to ordinary work.

Running-Stage Prediction Accuracy

Protocol and causal replay. We evaluate the remaining execution time of running stages using Equation (9), with one recorded run each at 8, 18, and 20 robots under the alternating 90/400 ms workload. We use the frozen calibration and lookup bounds of Appendix § 14.§ 1. At the first policy observation at least 20 ms after the preceding sample, we capture request state and final foreseeable load, without adding predictor evaluations to the scheduling path. We retain valid running-stage forecasts for requests sent in [30,210] s[30,210]\,\mathrm{s} and track each sampled stage through its observed completion.

Replay integrates scheduler-observed dispatch and retirement transitions strictly before forecast time tt and uses the captured nfn_f. Tests verify that changing future events cannot change a prediction. The artifact retains the pre-run protocol, instrumentation, analysis scripts, and raw traces.

Stage endpoints and aggregation. ViT and Prefill end when completion is observed by the scheduler. Action ends after its RPC and CPU output processing. The stage-endpoint estimate r^j,s(t)\hat{r}_{j,s}(t) equals r~j,s(t,nf)\tilde{r}_{j,s}(t,n_f) from Equation (9), with the fixed houth_{\mathrm{out}} added for Action. The signed prediction error is

ej,s,t=tj,s,endactual−t−r^j,s(t).(11)e_{j,s,t} = t^{\mathrm{actual}}_{j,s,\mathrm{end}} - t - \hat{r}_{j,s}(t). \tag*{(11)}

Positive error denotes underprediction. Primary mean absolute errors (MAEs) weight each forecast equally. Request-balanced summaries give each represented request equal total weight within a robot population or stage, divided equally among its forecasts.

Results. Across all running-stage samples, current-stage MAE is 2.04/2.62/2.79 ms at 8/18/20 robots. With equal request weights within each stage, MAE is 1.91–2.49 ms for ViT and Prefill and 2.15–3.05 ms for Action. Figure 13 shows both error directions.

Figure 13. Remaining-time prediction errors for running stages. Positive error is underprediction. Each represented request has equal total weight within its robot population. Curves include both tails; the horizontal scale is logarithmic outside [−1, 1] ms.

GPU Ablations and Scheduling Cost

Recorded task phases. Eight recorded π0.5\pi_{0.5} block-stacking episodes motivate the alternating phase workload. Marking 1.0 s before to 0.5 s after each gripper switch as critical covers 15–72% of each trace. The evaluation abstracts these phase patterns into controlled 90/400 ms budgets and 30-response phases.

Workloads and measurement. The GPU controls use the workloads in § 7.3, with batch-size-one execution, a 180 ms minimum send interval, and RTC d=8d = 8. Appendix § 14.§ 6 specifies the parcel-handling trace. Response latency spans the client send to receipt of the complete response. We retain requests sent during [30,210)[30, 210) s through completion and divide their on-time response count by 180 s for goodput. Unless stated otherwise, bursty results cover this full window; burst-only results use 60–80 s and 120–140 s. All measured requests complete successfully.

Component controls. Five controls change one mechanism or a specified policy combination while matching the full policy’s execution resources. Removing interference modeling uses isolated-stage durations without changing actual execution. Strict priority gating replaces both slack stealing and arrival-aware admission: ordinary work waits whenever a protected request exists. Removing arrival-aware admission retains current-request protection. Entry-only admission checks before ViT; Prefill and Action advance in priority/deadline order when resources permit. Zero-slack promotion sets the blocking term to zero.

Each of three campaigns rotates policy order across three matched rounds, with a 75 s premeasurement smoke run. In Table 8, full-policy means pool nine rounds, while each difference uses its own campaign’s three paired rounds. Zero-slack promotion has mixed effects.

Arrival-aware admission. A separate experiment isolates arrival-aware admission in six matched rounds with rotated policy order and identical model instances, stage computations, and execution resources. These rounds are not pooled with the component campaigns. Table 9 retains both attainment and the latency and goodput effects: protecting tight-budget requests can increase wider-budget latency and need not increase goodput.

WorkloadOverall attainment (%)Tight attainment (%)Wide mean latency (ms)Goodput (responses/s)
Phases, N=15N = 1597.3 / 97.894.5 / 95.6114 / 14880.4 / 80.4
Phases, N=16N = 1689.2 / 96.678.3 / 93.2200 / 20874.0 / 79.0
Phases, N=18N = 1884.3 / 96.568.7 / 93.0260 / 26268.7 / 78.4
Bursty arrivals97.7 / 97.895.7 / 96.094 / 9566.6 / 66.5
Multiple budgets94.6 / 96.175.0 / 81.9139 / 23677.7 / 74.7

Table 9. Arrival-aware GPU ablation, six-round means. Each cell gives U / T: FleetServe without / with arrival-aware admission under matched execution resources. Tight budgets are 90 ms for alternating phases and bursty arrivals, and 80 ms for multiple budgets. Wide-budget latency is for the 400 ms class. Bursty results use the full measurement window.

CPU decision cost. On a Xeon Gold 6530, we replay queued decision states from 40–45 s of the simulation four times in chronological order, with no serving process running. For 8/15/18 robots, FleetServe’s policy update and admission check (including arrival observation) take 41/63, 52/73, and 55/77 μ\mus (median/P99), respectively; these are CPU policy times, not added response latency.

Scheduling Policy Comparison

Baseline configuration. All policies share allocated execution resources and per-request stage computation. Simulation selects EDF’s smallest per-stage concurrency limits preserving the best SLO attainment under alternating task phases at 15 robots; these limits are fixed for GPU evaluation. Slack-Gate’s 40 ms threshold is selected from 20, 40, and 80 ms jointly with ViT concurrency, using the same workload and bursty arrivals. Among candidates within 1 percentage point of the best attainment, selection favors lower mean latency.

Comparison and aggregation. Each policy uses three rounds with a fixed configuration. FIFO and Slack-Gate rotate within matched rounds; EDF and FleetServe are measured separately. Comparisons use equally weighted round means, with reused measurements counted once. Reference-load checks under unchanged configurations show mean-attainment variation of −1.02 to +1.00 percentage points. Alternating task phases use 1,200 requests per robot; measurement boundaries follow Appendix § 14.§ 3. Table 10 reports per-round attainment and differences in three-round mean goodput.

Table 10. Scheduling policy comparison under matched execution resources. SLO attainment (%) is reported as R1 / R2 / R3. The NN rows use alternating task phases; bursty arrivals use the full [30,210)[30,210) send window. Δ\Deltagoodput is the difference between three-round means (FleetServe −- Slack-Gate; on-time responses/s).

WorkloadFIFOEDFSlack-GateFleetServeΔ\Delta goodput (responses/s)
N=8N = 874.86 / 99.96 / 99.8199.86 / 99.92 / 99.9999.97 / 100.00 / 99.9099.94 / 99.91 / 99.94−0.03
N=10N = 1098.94 / 97.98 / 96.1799.94 / 99.92 / 99.9899.98 / 99.97 / 99.9699.90 / 98.95 / 99.93−0.14
N=12N = 1299.77 / 99.75 / 99.5899.66 / 98.45 / 99.7099.92 / 99.86 / 99.2799.89 / 97.99 / 99.79+1.86
N=14N = 1496.30 / 96.01 / 97.4991.05 / 88.72 / 96.7198.86 / 99.26 / 97.9395.82 / 98.22 / 99.48+5.34
N=15N = 1581.19 / 81.36 / 82.3587.00 / 90.61 / 91.3595.88 / 97.28 / 95.8896.85 / 97.14 / 99.19+10.84
N=16N = 1650.14 / 49.98 / 52.3892.20 / 91.23 / 88.0095.69 / 95.68 / 94.7796.12 / 96.89 / 97.59+10.62
N=18N = 1849.97 / 49.99 / 50.0088.92 / 89.42 / 94.0589.99 / 90.33 / 93.6396.12 / 95.56 / 97.43+14.94
N=20N = 2050.01 / 49.99 / 50.0285.71 / 88.87 / 91.2275.28 / 75.43 / 76.9995.93 / 93.43 / 94.90+21.90
Bursty robot arrivals88.89 / 89.00 / 88.3798.81 / 97.99 / 97.2295.61 / 95.61 / 97.5098.66 / 98.79 / 99.27+4.08
Multiple response budgets, N=15N = 1585.49 / 86.31 / 79.0294.13 / 94.80 / 96.4492.16 / 92.47 / 92.7996.61 / 95.01 / 96.58+12.28

Table 10.

ERC. ERC is the largest tested population with at least 95% attainment in each of three rounds: FleetServe passes at 18 robots but not at 20; 19 was not measured. This criterion does not imply that every smaller population passes.

Temporal and budget aggregation. Figure 8(d) averages attainment in 2 s request-send-time bins across rounds. § 7.3 reports attainment over the 60–80 s and 120–140 s burst windows; Table 10 uses the full [30, 210) s send window. For multiple response budgets (Figure 8(e)), overall attainment aggregates all requests within each round before averaging rounds, rather than weighting budget classes equally.

Parcel handling

Client budgets follow recorded contact timing during parcel picking, flipping, and passing. We use 23 validation episodes, excluding one collection site with constant gripper readings that obscure contact timing. A contact interval spans 1.0 s before to 0.5 s after any gripper switch, and overlapping intervals merge; contact covers 36–78% of each episode (64% on average). Robot ii replays episode i mod 23i \mathbin{\bmod} 23 from a seeded start offset and loops it. Contact intervals carry the 90 ms budget and the remaining time the 400 ms budget. Interval durations convert to response counts at the 180 ms period, because phases advance by successful responses as in alternating task phases. Each robot sends 1,200 requests with the same staggered starts, and each of up to 20 robots replays a different episode. Tight requests make up 61–63% of each scenario, and the fraction of robots in contact varies between 20% and 100%.

FIFO and Slack-Gate are measured at 10–20 robots in three rounds; EDF comes from its separate campaign and FleetServe from a separate run, all with the protocol of Appendix C.4. Repeated runs of the same configuration differ by up to 2.9 percentage points in three-round mean at 16 and 18 robots, so comparisons across runs carry uncertainty of this size. All 941,655 requests sent in the measurement windows succeed. Table 11 reports every round.

Table 11. Parcel handling: overall SLO attainment (%) as R1 / R2 / R3 under matched execution resources.

RobotsFIFOEDFSlack-GateFleetServe
1099.79 / 99.84 / 99.6899.79 / 92.78 / 95.7799.88 / 99.91 / 98.3598.07 / 99.76 / 99.74
1298.79 / 99.74 / 94.0397.48 / 99.60 / 86.8199.26 / 99.70 / 99.8399.81 / 99.26 / 99.66
1497.46 / 96.57 / 94.3990.26 / 89.49 / 93.0295.87 / 95.80 / 93.8995.49 / 99.10 / 98.67
1654.46 / 38.09 / 47.6572.45 / 73.19 / 73.2489.25 / 87.64 / 92.7695.21 / 94.50 / 96.51
1837.65 / 37.82 / 37.6565.32 / 69.96 / 73.4581.46 / 80.64 / 83.5986.68 / 89.43 / 90.97
2036.94 / 37.12 / 36.8450.65 / 52.13 / 55.7956.90 / 55.18 / 58.4062.66 / 66.61 / 68.99

Table 11.

At 16 robots, tight-budget attainment is 56.6% for EDF, 92.6% for FleetServe, and 83.8% for Slack-Gate. From 14 robots upward, FleetServe returns wide-budget responses 7–56 ms sooner than Slack-Gate and increases goodput by 4.1–9.1 requests/s.

References

  1. [1]Adamopoulos, D., Chanpaisit, N., Fakhri, B., and Giannoula, C. Efficient vision-language-action management and serving for robot factories. arXiv preprint arXiv:2609.12075, 2026. URL https://arxiv.org/abs/2609.12075.
  2. [2]Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwattra, N., Gulavani, B. S., Tumanov, A., and Ramjee, R. Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 117–134. USENIX Association, 2024. URL https://www.usenix.org/conference/osdi24/presentation/agrawal.arxiv.org/abs/2403.02310
  3. [4]Audsley, N., Burns, A., Richardson, M., Tindell, K., and Wellings, A. J. Applying new scheduling theory to static priority pre-emptive scheduling. Software Engineering Journal, 8(5):284–292, 1993. doi: 10.1049/sej.1993.0034.DOI
  4. [5]Black, K., Brown, N., Darpinian, J., Dhabalia, K., Driess, D., Esmail, A., Equi, M. R., Finn, C., Fusai, N., Galliker, M. Y., Ghosh, D., Groom, L., Hausman, K., Ichter, B., Jakubczak, S., Jones, T., Ke, L., LeBlanc, D., Levine, S., Li-Bell, A., Mothukuri, M., Nair, S., Pertsch, K., Ren, A. Z., Shi, L. X., Smith, L., Springenberg, J. T., Stachowicz, K., Tanner, J., Vuong, Q., Walke, H., Walling, A., Wang, H., Yu, L., and Zhilinsky, U. π0.5: a vision-language-action model with open-world generalization. In Proceedings of The 9th Conference on Robot Learning, volume 305 of Proceedings of Machine Learning Research, pp. 17–40. PMLR, 2025a. URL https://proceedings.mlr.press/v305/black25a.html.arxiv.org/abs/2504.16054
  5. [6]Black, K., Galliker, M., and Levine, S. Real-time execution of action chunking flow policies. In Advances in Neural Information Processing Systems, volume 38, pp. 33383–33407. Curran Associates, Inc., 2025b. doi: 10.52202/085713-1122.DOI
  6. [7]Chen, S., Jia, Z., Khan, S., Krishnamurthy, A., and Gibbons, P. B. SLOs-Serve: Optimized serving of multi-SLO LLMs. arXiv preprint arXiv:2504.08784, 2025. URL https://arxiv.org/abs/2504.08784.
  7. [8]Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burchfiel, B., and Song, S. Diffusion Policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023. doi: 10.15607/RSS.2023.XIX.026.DOI
  8. [9]Dai, Y., Ananthanarayanan, G., Cox, L., Foukas, X., Radunovic, B., and Netravali, R. Kairos: A scalable serving system for physical AI. arXiv preprint arXiv:2605.11381, 2026. URL https://arxiv.org/abs/2605.11381.
  9. [10]Gujarati, A., Karimi, R., Alzayat, S., Hao, W., Kaufmann, A., Vigfusson, Y., and Mace, J. Serving DNNs like Clockwork: Performance predictability from the bottom up. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pp. 443–462. USENIX Association, 2020. URL https://www.usenix.org/conference/osdi20/presentation/gujarati.arxiv.org/abs/2006.02464
  10. [11]Han, M., Zhang, H., Chen, R., and Chen, H. Microscond-scale preemption for concurrent GPU-accelerated DNN inferences. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 539–558. USENIX Association, 2022. URL https://www.usenix.org/conference/osdi22/presentation/han.
  11. [12]Hong, K., Chen, L., Wang, Z., Li, X., Mao, Q., Ma, J., Xiong, C., Wu, G., Han, B., Dai, G., Liang, Y., and Wang, Y. semi-PD: Towards efficient LLM serving via phase-wise disaggregated computation and unified storage. arXiv preprint arXiv:2504.19867, 2025a. URL https://arxiv.org/abs/2504.19867.
  12. [13]Hong, K., Li, X., Chen, L., Mao, Q., Ning, X., Dai, G., Yan, S., Liang, Y., and Wang, Y. SOLA: Optimizing SLO attainment for large language model serving with state-aware scheduling. In Proceedings of Machine Learning and Systems, volume 7. MLSys, 2025b. URL https://mlsys.org/virtual/2025/poster/3231.
  13. [14]Jeffay, K., Stanat, D. F., and Martel, C. U. On non-preemptive scheduling of periodic and sporadic tasks. In Proceedings of the Twelfth IEEE Real-Time Systems Symposium (RTSS), pp. 129–139, 1991. doi: 10.1109/REAL.1991.160366.
  14. [15]Jiang, W., Clemons, J., O’Flaherty, R., Hadfield, H., Degirmenci, A., Song, S., Narang, Y., and Kozyraki, C. ROSA: A robotics foundation model serving system for robot factories. arXiv preprint arXiv:2607.01088, 2026. URL https://arxiv.org/abs/2607.01088.
  15. [16]Lehoczky, J. P. and Ramos-Thuel, S. An optimal algorithm for scheduling soft-aperiodic tasks in fixed-priority pre-emptive systems. In Proceedings of the IEEE Real-Time Systems Symposium (RTSS), pp. 110–123, 1992. doi: 10.1109/REAL.1992.242671.
  16. [17]Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., Xu, L., Li, B., Wu, H., Han, C. L., Li, X., Hua, M., Jiang, and Le, M. Flow matching for generative modeling. In International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum?id=PqvMRDCJT9t.arxiv.org/abs/2210.02747
  17. [18]Liu, C. L. and Layland, J. W. Scheduling algorithms for multiprogramming in a hard-real-time environment. Journal of the ACM, 20(1):46–61, 1973. doi: 10.1145/321738.321743.DOI
  18. [20]Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., and Bianchini, R. Splitwise: Efficient generative LLM inference using phase splitting. In ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pp. 118–132, 2024. doi: 10.1109/ISCA59077.2024.00019.DOI
  19. [21]Rouhani, B. D., Zhao, R., More, A., Hall, M., Khodamoradi, A., Deng, S., Choudhary, D., Cornea, M., Dellinger, E., Denolf, K., Stosic, D., Elango, V., Golub, M., Heinecke, A., James-Roxby, P., Jani, D., Kolhe, G., Langhammer, M., Li, A., Melnick, L., Mesmakhosroshahi, M., Rodriguez, A., Schulte, M., Shafipour, R., Shao, L., Siu, M., Dubey, P., Micikevicius, P., Naumov, M., Verrilli, C., Wittig, R., Burger, D., and Chung, E. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537, 2023. URL https://arxiv.org/abs/2310.10537.
  20. [22]Strati, F., Ma, X., and Klimovic, A. Orion: Interference-aware, fine-grained GPU sharing for ML applications. In Proceedings of the Nineteenth European Conference on Computer Systems (EuroSys), pp. 1075–1092, 2024. doi: 10.1145/3627703.3629578.DOI
  21. [25]Tan, X., Yang, Y., Ye, P., Zheng, J., Bai, B., Wang, X., Hao, J., and Chen, T. Think twice, act once: Token-aware compression and action reuse for efficient inference in vision-language-action models. arXiv preprint arXiv:2505.21200, 2025. URL https://arxiv.org/abs/2505.21200.
  22. [27]Yin, P., Zhu, J., Gao, H., Zheng, C., Huang, Y., Zhou, T., Yang, R., Liu, W., Chen, W., Guo, C., Deng, D., Mo, Z., Wang, C., Cheng, J., Wang, R., and Liu, H. vLLM-Omni: Fully disaggregated serving for any-to-any multimodal models. arXiv preprint arXiv:2602.02204, 2026. URL https://arxiv.org/abs/2602.02204.
  23. [30]Zhai, A., Liu, B., Fang, B., Cai, C., Ma, E., Yin, E., Wang, H., Zhou, H., Wang, J., Shi, L., Liang, L., Wang, M., Wang, Q., Gan, R., Yu, R., Li, S., Liu, S., Chen, S., Chen, V., and Xu, Z. Igniting vlms toward the embodied space. arXiv preprint arXiv:2509.11766, 2025.arxiv.org/abs/2509.11766
  24. [31]Zhao, T. Z., Kumar, V., Levine, S., and Finn, C. Learning fine-grained bimanual manipulation with low-cost hardware. In Proceedings of Robotics: Science and Systems (RSS), 2023. doi: 10.15607/RSS.2023.XIX.016.DOI
  25. [32]Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozaruk, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, volume 37, pp. 62557–62583. Curran Associates, Inc., 2024. doi: 10.52202/079017-2000.DOI
  26. [33]Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., and Zhang, H. Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 193–210. USENIX Association, 2024. URL https://www.usenix.org/conference/osdi24/presentation/zhong-yinmin.arxiv.org/abs/2401.09670

Paper details

vlnc:W_p7wqrwuv
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 6

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

OpenAI ChatGPT/Codex, Anthropic Claude

View statement
Looking for related papers…
Contents