betaFeedback

WorksComputer Science

Koi: Evidence-Guided, Self-Calibrating Planning for Multi-Cloud, Heterogeneous LLM Inference Fleets

Abstract

Deploying a large language model (LLM) requires choosing the right accelerator, parallelization strategy, engine configuration, and replica count amongst ~140 other variables across clouds, regions, and markets. These decisions are intertwined, and the best cluster-wide plan continually changes as jobs contend for shared capacity on shifting workloads. We present Koi, a self-calibrating cluster planner for heterogeneous, multi-cloud fleets. Koi uses a Recursive Language Model (RLM) hierarchy to propose hypotheses and reason over deployment while deterministic components predict serving behavior, score candidates, and control placement. It represents inference behavior as a fixed causal graph linking decisions (e.g., accelerator, parallelism), runtime mediators (e.g., KV-cache pressure, communication overhead), and outcomes (e.g., latency, throughput). Each deployment strengthens or weakens scoped causal relationships, building a comprehensive playbook for inference that the planner employs to guide future decisions. Job-wise configurations are composed into cluster-wide plans and evaluated against serving objectives, resource constraints, SLO risks, and cost. Across benchmarks, Koi achieves 1.18x greater goodput than the strongest baseline while reducing $/token by 20% in a real heterogeneous cluster. Ablations isolate the contribution of Koi's architecture: a pure language model attains a composite score of 35.3, the same model equipped with Koi's deterministic tools reaches 51.6, and the full Koi planner reaches 70.8.

Introduction

An LLM inference provider serves a changing mix of jobs on a shared accelerator fleet that spans clouds, regions, markets (reserved, on-demand, or spot), and hardware generations. Users specify each job’s model, workload profile (input/output lengths and arrival process), class (online or batch), and serving target (a latency bound for online jobs or a completion deadline for batch jobs). The provider decides how every job runs, including the environment and accelerator, the parallelism and engine configuration, and the number of replicas, and must revise these decisions as jobs arrive and complete and as available capacity changes.

Even for a single job, this choice is consequential. For a profiled 70B dense model on one accelerator family, five feasible tensor- and pipeline-parallel configurations span 4.4× in throughput and 7.5× in cost per token and trade off performance, cost, and utilization differently (Figure 1 and Figure 2). TP4/PP2 across two four-GPU instances serves the workload at 180 tok/s for $0.97, whereas TP8/PP1 on one eight-GPU instance reaches only 162 tok/s for $2.16. The best configuration also depends on workload shape, because prefill-heavy traffic stresses compute while decode-heavy

Figure 1. Parallelism configurations expose conflicting performance, cost, and utilization trade-offs. It used Llama 3 70B on AWS A10G GPUs with 2,048/512 input/output tokens. Each axis is normalized independently.

Figure 2. Parallelism substantially changes inference performance and cost. Done on AWS A10G instances with Llama 3 70B model. The workload has 2,048 input and 512 output tokens per request.

An LLM inference provider serves a changing mix of jobs on a shared accelerator fleet that spans clouds, regions, markets (reserved, on-demand, or spot), and hardware generations. Users specify each job’s model, workload profile traffic stresses memory bandwidth and KV-cache capacity. Neither the workload nor the serving stack is stationary. A job’s realized input/output distribution drifts, and engine and library updates change throughput for the same deployment, so a configuration profiled once does not stay optimal and the planner must keep recalibrating from what it observes.

These choices are also coupled across jobs that compete for the same finite accelerators, quota, and budget. Most schedulers decide one job at a time, giving each arriving job its best-ranked feasible placement, so scarce capacity goes to whichever job arrives first. This is suboptimal whenever an earlier job has a workable second choice that a later job lacks. For example, an online chat job and a batch evaluation job both prefer the cluster’s only free H100 node, but only the batch job can meet its target on A100s. If the batch job arrives first, a greedy scheduler gives it the H100, and the online job must miss its SLO or wait. Even a correct per-job ranking cannot prevent this, because avoiding it requires weighing the opportunity cost of giving a scarce device to one job rather than another, which only a cluster-wide decision can do.

Existing systems address parts of this problem. Serving engines optimize request execution within a given deployment [3, 14, 31]. Multi-cloud orchestrators place workloads across clouds and markets but leave the accelerator type and configuration to the user [18, 30]. Configuration optimizers and heterogeneous serving planners choose parallelism, placement, or GPU types for a single model or a fixed set of models [11, 15, 19, 29]. Request schedulers decide how requests are queued and served on existing deployments [10, 22], and cluster schedulers allocate resources among competing jobs [7, 12, 25], but neither chooses the deployment configuration of an inference job. No existing system jointly chooses deployment configurations and allocates shared capacity for all online and batch inference jobs

across a heterogeneous, multi-cloud fleet, while recalibrating those choices as observed behavior drifts.

Joint planning is hard for two reasons. First, the space is vast. Each job’s deployment sets about 140 configuration knobs (Appendix A), including accelerator, parallelism, engine, quantization, batching, and KV-cache settings, which yields at least 21402^{140} configurations per job. A cluster plan chooses one deployment for every job under shared capacity, so the number of candidate plans grows exponentially with the number of jobs and cannot be enumerated. Second, the planner must predict the performance of never-run deployments for every job and candidate. Analytical models and profile-based simulators [2, 29] rest on fixed assumptions about memory, communication, and engine behavior that break as workloads, hardware, and software change. In our measurements, even a roofline model calibrated with measured peak compute and memory bandwidth errs by up to 4.7×4.7\times. Language models encode broad systems knowledge but not calibrated, deployment-specific quantities. Two frontier models, Claude Opus 4.8 and GPT-5.6 Sol, overpredict throughput by 4×4\times and 9×9\times. A planner therefore cannot trust predictions unless it corrects them from observed behavior.

Search-based methods instead deploy candidate configurations, observe their end-to-end performance, and use it to choose the next candidates. Bayesian optimization fits a statistical model to these observations [5, 8], and LLM-guided evolutionary search asks a language model to propose new candidates from high-scoring ones [6, 17, 27, 28]. These methods have two fundamental limitations. First, because they learn only from end-to-end outcomes, they cannot tell why a candidate performed as it did, for example whether low throughput came from pipeline bubbles or KV-cache pressure, and therefore which fix applies or which other jobs the evidence should inform. Second, every trial is a live deployment that occupies capacity other jobs need and can breach a running job’s SLO. Evolutionary search needs many such trials, and Bayesian optimization restarts for each new workload. LLM-guided evolutionary search pairs a language model with measurement only loosely, because the model sees only a scalar score and measurements never confirm or refute its reasoning. A tighter pairing is difficult because the model’s knowledge is qualitative while telemetry is numeric.

Koi pairs them tightly through a fixed causal graph on which both can operate. The graph links deployment decisions (e.g., accelerator, parallelism), runtime mediators (e.g., KV-cache pressure, communication overhead), and serving outcomes (e.g., latency, throughput). Scoped mechanisms compose its edges into testable hypotheses, which a language model can propose and telemetry can support or contradict. For example, one mechanism states that raising pipeline parallelism lowers throughput through pipeline bubbles for a given model and hardware. Each edge and mechanism carries a confidence and an uncertainty updated from runtime evidence, so confident relationships guide exploitation and uncertain ones identify deployments worth testing. Koi runs no separate trials, since every deployment both serves its job and tests the relationships it exercises, and the plan score values both. Bayesian optimization also updates a model after each trial, but that model maps configurations to end-to-end outcomes for a single workload. Koi instead updates its beliefs about mechanisms, so evidence from one job informs every job within the mechanism’s scope.

Koi turns this evidence into a cluster plan by alternating between language-model proposals and deterministic evaluation. For each job, the language model proposes candidate deployments from the causal graph, the cluster snapshot, the job’s workload, and the matching mechanisms. The graph lets it focus on the decisions that affect the job’s targets, so Koi evaluates a few candidates per job instead of enumerating the space. Deterministic components predict each candidate’s behavior with a profile-based performance model whose error is tracked against observed outcomes, score it against the job’s weighted objectives, and compose candidates into a cluster-wide plan. The plan score balances serving value, the value of testing uncertain relationships, SLO risk under prediction error, and reconfiguration cost, subject to shared capacity. Because candidates for all jobs compete within one plan, this score exposes the opportunity costs that per-job optimization misses. Only deployment evidence updates causal confidence, and the language model only proposes and cannot override measured evidence or cluster constraints.

We evaluate Koi on a real 104-GPU cluster spanning three regions and on simulated 400- and 800-GPU clusters, all heterogeneous, against nine baselines including the approaches above (three on the real cluster). Against the strongest baseline, Koi delivers 1.18×1.18 \times the goodput at 20% lower $/token on the real cluster and 45% more goodput at 31% lower $/token in the 3-hour simulation. Ablations show that these gains come mainly from the structure around the language model rather than from the model itself.

This paper makes the following three contributions.

  • We design and implement Koi, a provider-side control plane that self-calibrates the cluster-wide deployment plan for mixed online and batch jobs across heterogeneous clouds, and release it as open source1.

  • We develop an evidence-guided agentic planning algorithm around a causal DAG that links deployment decisions, runtime mediators, and outcomes. Our RLM [33] hierarchy proposes candidates and scoped hypotheses, deterministic evaluation selects the cluster plan, and telemetry updates the evidence that guides subsequent intervals.

  • We evaluate Koi on heterogeneous benchmarks spanning ten model architectures and nine baselines, where it improves goodput by up to 45% and lowers $/token by up to 31% while admitting 96% of online and 100% of batch jobs over 12 hours.

Background

Where deployment planning sits

LLM serving spans three control layers. Serving engines admit and batch requests [3, 14, 31], routers distribute requests among running replicas [10, 22], and a deployment controller chooses the hardware, parallelism, engine configuration, replica count, and placement within which those layers operate. Koi composes with serving engines and routers while operating at the third layer, where it plans and controls deployment. The separation is operational as well as temporal. Engine and router decisions are made per request and are reversible within milliseconds, whereas a deployment decision commits model weights to devices. Changing one requires loading weights, warming caches, and draining in-flight work, which bounds how many configurations a controller can try in a given period and requires damping against churn.

What determines deployment performance

The deployment objective is not to maximize raw throughput or GPU utilization. Rather, the controller should select the least-cost configuration that satisfies the workload’s service objectives: online workloads must meet TTFT, TPOT, and throughput targets, while batch workloads must complete by their deadlines. Operating closer to saturation can reduce cost, but it also reduces queueing headroom and can sharply degrade tail latency under bursty demand.

The lowest-cost feasible configuration is workload-dependent because different workload characteristics expose different hardware bottlenecks. Input-heavy requests make prefill compute the limiting resource and first manifest as elevated TTFT. Long-context and output-heavy requests increase KV-cache residency and HBM-bandwidth pressure, which can degrade TPOT and reduce the capacity available to concurrent requests. Arrival rate and burstiness determine the replication and queueing headroom required to preserve these objectives. Consequently, a model identifier and mean request rate alone are insufficient to determine the appropriate GPU type, model-parallel configuration, or replica count.

Configuration choices affect end-to-end objectives indirectly through runtime mechanisms. For example, pipeline parallelism can reduce per-GPU memory pressure but introduce pipeline bubbles and stage imbalance; similarly degraded throughput can instead arise from KV-cache pressure or interconnect contention. These mechanisms require different corrective actions. Thus, a controller must use runtime evidence to identify the binding bottleneck and select the least-cost feasible intervention, rather than relying on repeated configuration trials guided only by endpoint metrics. This motivates Koi’s evidence-conditioned control loop.

Related Work

Deployment configuration and placement

Per-job optimizers evaluate alternative serving configurations against workload and SLO objectives. AIConfigurator predicts configurations across parallelism strategies, serving modes, and engine-specific settings, then ranks them by derived latency and throughput [29].

Other systems extend deployment decisions to concurrent jobs. AlpaServe places and parallelizes multiple models to improve serving efficiency under bursty workloads [15]. Prism improves multi-model serving through demand-aware cross-model GPU-memory sharing, model placement, and request arbitration [32]. Heterogeneous serving planners optimize a single model across mixed hardware: Mélange computes a minimum-cost heterogeneous GPU allocation for a given workload and latency target [11], while Helix jointly optimizes model placement and request scheduling over heterogeneous GPUs and network links [19]. Cloud systems operate at the infrastructure layer: SkyPilot places workloads across clouds [30], while SkyServe distributes spot and on-demand replicas across regions and clouds [18].

Feedback-driven and causal optimization

Measurement-driven optimizers refine decisions from observed outcomes. CherryPick uses Bayesian optimization to test cloud configurations and select subsequent trials from observed execution [5]. SCOOT applies Bayesian optimization to inference-engine parameters, using learned constraints to avoid invalid configurations while optimizing SLOs [8]. Cluster schedulers also adapt from runtime feedback: Pollux models how resource allocation and batch size affect training goodput [25]; Cilantro learns online resource-to-performance mappings under application-level objectives [7]; and Tiresias schedules distributed training jobs from observations and prior knowledge of job duration [12].

Causal methods instead use structure to determine which interventions or relationships should generalize. Invariant causal prediction identifies predictors whose relationship with a target remains stable across environments [24], while causal Bayesian optimization selects interventions on a supplied causal graph to optimize a target with fewer experiments [1].

LLM-guided search

FunSearch uses an LLM to modify a target function and an evaluator to score each candidate [27]. AlphaEvolve extends this evolutionary search to larger programs and codebases, retaining evaluated programs to guide subsequent proposals [6]. OpenEvolve maintains a diverse set of high-scoring programs and uses them to guide subsequent mutations [28]. EvoX further co-evolves candidate solutions and the search strategy that selects and varies them, allowing the exploration policy itself to adapt during optimization [17]. These systems evolve executable artifacts by repeatedly proposing and scoring candidate programs.

Problem Formulation

At each nn-minute tick, Koi selects a deployment configuration for every job in the cluster. At tick tt, the cluster contains active jobs At\mathcal{A}_t and pending jobs QtQ_t. Each job ii has a workload profile WiW_i and target SiS_i: a latency bound for online jobs or a completion deadline for batch jobs. Cluster capacity is represented by a resource map RtR_t over environments ee.

A deployment is organized into the following hierarchy: Chain. A chain c=(e,x)c=(e,x) is one model copy deployed in environment ee under one deployment configuration. Rank. A rank (c,n)(c,n) consists of nn homogeneous replicas of the same chain, where nn is its data-parallel size. Ladder. A ladder Li={(ci1,ni1),…,(cik,nik)}L_i=\{(c_i^1,n_i^1),\ldots,(c_i^k,n_i^k)\} is the set of ranks assigned to job ii. Plan. A cluster plan Pt={Li}i∈At∪QtP_t=\{L_i\}_{i\in\mathcal{A}_t\cup Q_t} assigns one ladder to every active and pending job in the cluster. For an active job, Li=Li(t−1)L_i=L_i^{(t-1)} retains its current deployment; for a pending job, Li=∅L_i=\varnothing defers admission.

Each ladder LiL_i produces five outcomes ($/tok, p99 TTFT, p99 TPOT, throughput, and SLO margin) collectively denoted yi(Li)∈R5y_i(L_i)\in\mathbb{R}^5. The first three are minimized and the latter two maximized.

Each job ii specifies objective weights wi=(wi1,…,wi5)w_i=(w_{i1},\ldots,w_{i5}), with wij≥0w_{ij}\ge0 and ∑jwij=1\sum_j w_{ij}=1, over yi(Li)y_i(L_i). These weights encode the user’s relative priorities for that job’s serving objectives. Koi reduces these five objectives to a single scalar using an augmented Tchebycheff scalarization [20]. Let zi∗z_i^* be a reference point for job ii that weakly dominates every outcome the job can achieve, maintained per job and updated from realized outcomes (§5.4), and let σij\sigma_{ij} be a normalizing constant per objective. The normalized gap along objective jj is gij(Li)=(yij(Li)−zij∗)/σijg_{ij}(L_i)=(y_{ij}(L_i)-z_{ij}^*)/\sigma_{ij} if jj is minimized and (zij∗−yij(Li))/σij(z_{ij}^*-y_{ij}(L_i))/\sigma_{ij} if jj is maximized, and the ladder’s exploitation score is

Ji(Li)=−[max⁡j∈[1,5]wijgij(Li)+ρ∑j=15wijgij(Li)].(1)J_i(L_i)=-\left[\max_{j\in[1,5]} w_{ij}g_{ij}(L_i)+\rho\sum_{j=1}^{5}w_{ij}g_{ij}(L_i)\right]. \tag*{(1)}

where ρ>0\rho>0 is an augmentation constant. The max term ensures that no single objective is neglected on non-convex Pareto frontiers; the weighted sum breaks ties among candidates with the same maximum gap. Ji(Li)≤0J_i(L_i)\le0 always, and Ji(Li)=0J_i(L_i)=0 only when every objective the job weights attains its reference.

Let Switch⁡(Li(t−1),Li)\operatorname{Switch}(L_i^{(t-1)},L_i) denote the reconfiguration cost defined in §5.3.4, with Switch⁡(L,L)=0\operatorname{Switch}(L,L)=0 for an unchanged ladder. Koi selects a cluster-wide plan by maximizing aggregate ladder value minus this cost:

max⁡{Li}∑i∈At∪QtJi(Li)−λt∑i∈AtSwitch⁡(Li(t−1),Li)(2)\max_{\{L_i\}} \sum_{i\in A_t\cup Q_t} J_i(L_i) - \lambda_t \sum_{i\in A_t} \operatorname{Switch}\left(L_i^{(t-1)}, L_i\right) \tag*{(2)}

subject to

C1 Resource feasibility. The chains requested across all ladders must fit within the available capacity under RtR_t.

C2 Physical feasibility. Each chain must fit available VRAM; TP must divide the attention and KV head counts; PP must partition the model by layers.

C3 SLO feasibility. Each ladder must satisfy SiS_i under prediction uncertainty, with P(breach)≤τP(\text{breach}) \le\tau.

C4 Swap budget. At most BtB_t active jobs may change ladders during one tick: ∣{i∈At:Li≠Li(t−1)}∣≤Bt\lvert\{i\in A_t:L_i\ne L_i^{(t-1)}\}\rvert\le B_t.

C5 Job policy. Each job must remain within its capacity quota, and higher-priority jobs must take precedence over lower-priority jobs competing for the same resources.

A pending job is deferred when no ladder satisfies C1–C3, or when job priority, quota, or resource contention favors reserving capacity for another job.

Equation (2) defines the decision Koi would make if yi(Li)y_i(L_i) were known for every candidate ladder. In practice, Li↦yi(Li)L_i\mapsto y_i(L_i) is unknown and changes over time. Because evaluating a ladder requires deployment, consuming cluster capacity and risking SLO violations, Koi instead estimates yiy_i from available evidence, deploys, observes the outcome, and updates its estimate for the subsequent tick.

Koi

Overview

Each tick, Koi executes an eight-stage closed loop that updates evidence from the cluster and searches for a better cluster-wide plan (Figure 3). States S0–S3 refresh the planner state by snapshotting jobs, placements, and capacity (S0), collecting per-rank telemetry (S1), comparing observations against stored predictions (S2), and updating the evidence state (S3). Planning begins at S4, where the agent proposes a cluster-wide action for each job (place, reconfigure, retain, or defer). Deterministic evaluation at S5 scores the proposal against job-specific performance, expected evidence, SLO risk, and reconfiguration cost, and validates constraints C1–C5. Invalid or non-improving plans return to S4 with scores and diagnostics. Once the search budget is exhausted, Koi commits the best feasible cluster-wide plan (S6) and closes the loop (S7). This design relies on two assumptions: near-optimal proposal coverage (Assumption 5.3) and convergence under stationary conditions (Assumption 5.4).

End-to-end Koi architecture

Figure 3. End-to-end Koi architecture. Each planning tick snapshots cluster state, observes runtime telemetry, updates causal confidence, proposes deployment candidates, and deterministically evaluates and selects a cluster-wide plan.

Mechanisms and Predictions

To make deployment behavior testable and transferable, Koi represents inference as a directed acyclic graph (DAG), X→V→YX\to V\to Y, where XX contains planner-controlled decisions (hardware, engine, parallelism, quantization, batching, routing), VV contains telemetry-derived runtime mediators (KV-cache utilization, pipeline bubbles, communication latency, memory pressure, SM utilization), and YY contains the serving outcomes defined in §4. Only X→VX\to V and V→YV\to Y edges are permitted, ensuring every modeled effect passes through telemetry; the registered edge set E⊆(X×V)∪(V×Y)E\subseteq(X\times V)\cup(V\times Y) is fixed at initialization.

Each edge e∈Ee\in E maintains a belief pe∼Beta⁡(αe,βe)p_e\sim\operatorname{Beta}(\alpha_e,\beta_e), where αe\alpha_e and βe\beta_e accumulate supporting and contradictory evidence, and ce=αe/(αe+βe)c_e=\alpha_e/(\alpha_e+\beta_e) is its confidence. Because the same confidence can reflect different amounts of evidence (supporting and contradictory deployment behavior), Koi also tracks normalized posterior uncertainty

U(αe,βe)=12αeβe(αe+βe)2(αe+βe+1),U(1,1)=1.U(\alpha_e,\beta_e)=\frac{12\alpha_e\beta_e}{(\alpha_e+\beta_e)^2(\alpha_e+\beta_e+1)},\qquad U(1,1)=1.

Exploration (§5.3.2) prioritizes edges with high UU, where additional deployment evidence is most informative.

Edges compose into scoped hypotheses called mechanisms. A mechanism m=(Em,Sm,ηm,Bm)m=(E_m,S_m,\eta_m,B_m) pairs a connected subgraph Em⊆EE_m\subseteq E with an applicability scope SmS_m over models, hardware, workloads, and conditions, a natural-language statement ηm\eta_m, and Beta state BmB_m. The LLM proposes mechanisms, which are admitted only if they form a valid, non-duplicate X→V→YX\to V\to Y path over registered edges and are initialized with Bm=Beta⁡(1,1)B_m=\operatorname{Beta}(1,1).

Both V^\hat V and Y^\hat Y, the predicted mediator and outcome values, come from Koi’s planner-internal deterministic surrogate. The surrogate builds on AIConfigurator [29]. Given a job’s workload profile and one candidate ladder, it composes operator-level latency profiles with a replay simulator to predict TTFT, TPOT, and throughput point estimates. The remaining mediators and outcomes are derived from those predictions, the job’s targets, and the allocation price carried by RtR_t. When a model has no measured profile, it substitutes the nearest profiled architecture on that hardware and rescales by its measured prefill and decode speed ratios. This surrogate is used during planning to score candidates and is distinct from LLMServingSim 2.0, which is used only as the execution environment for the simulated evaluation 6.

The surrogate is an approximation, so Koi carries prediction uncertainty into planning and rejects ladders with P(breach)>τP(\text{breach})>\tau under C3. A deployed ladder tests every mechanism whose scope it satisfies by comparing the predicted mediator and outcome trajectories, V^\hat V and Y^\hat Y, against the observed VobsV_{\mathrm{obs}} and YobsY_{\mathrm{obs}}. For either trajectory O∈{V,Y}O\in\{V,Y\}, Koi defines the residual at observation kk as rk=Oobs,k−O^kr_k=O_{\mathrm{obs},k}-\hat O_k and accumulates it in a two-sided cumulative sum (CUSUM) [21] detector:

Sk+=max⁡(0,Sk−1++rk−δ)andSk−=min⁡(0,Sk−1−+rk+δ)S_k^+=\max(0,S_{k-1}^++r_k-\delta)\quad\text{and}\quad S_k^-=\min(0,S_{k-1}^-+r_k+\delta)

with slack δ=0.5σ^\delta= 0.5\hat{\sigma}, declaring divergence when Sk+>hS_k^{+} > h or Sk−<−hS_k^{-} < -h for threshold h=4σ^h = 4\hat{\sigma}, where σ^\hat{\sigma} is the empirical residual scale. Accumulating the residual rather than thresholding each sample separates persistent prediction error from isolated deviations.

CUSUM assigns each mechanism evaluation to one of four quadrants: Q1 (Vobs≈V^,Y^obs≈Y^V_{\mathrm{obs}} \approx\hat{V}, \hat{Y}_{\mathrm{obs}} \approx\hat{Y}), Q2 (Vobs≈V^,Y^obs≉Y^V_{\mathrm{obs}} \approx\hat{V}, \hat{Y}_{\mathrm{obs}} \not\approx\hat{Y}), Q3 (Vobs≉V^,Y^obs≈Y^V_{\mathrm{obs}} \not\approx\hat{V}, \hat{Y}_{\mathrm{obs}} \approx\hat{Y}), and Q4 (Vobs≉V^,Y^obs≉Y^V_{\mathrm{obs}} \not\approx\hat{V}, \hat{Y}_{\mathrm{obs}} \not\approx\hat{Y}). Q1 adds supporting evidence, Q4 contradictory evidence, and Q2–Q3 update edges according to their invariant causal prediction (ICP) result [24]. §5.4 uses the Q1 rate to adapt exploration.

Scoring a Plan

Because Koi scores predicted rather than known outcomes, it augments Equation (2) with two terms: risk from prediction error and exploration value from testing uncertain causal relationships. Exploitation, risk, and switching cost sum over jobs; exploration is scored once per plan to avoid double-counting repeated tests. The plan score relies on a corresponding coverage assumption:

Assumption 1. At each tick, the planner generates KPK_P candidate plans. Let Pt⋆P_t^\star be the optimal feasible plan and Φ(⋅)\Phi(\cdot) the plan score. We assume that, with probability at least η\eta, at least one candidate PkP_k has score

Φ(Pk)≥Φ(Pt⋆)−εP,\Phi(P_k) \ge\Phi(P_t^\star) - \varepsilon_P,

where εP\varepsilon_P is the tolerated gap from the optimum.

5.3.1. Exploitation. Exploitation is the per-job score Ji(Li)J_i(L_i) of Equation (1), evaluated on the ladder’s aggregated outcomes. High-confidence causal relationships guide exploitative candidates, while Ji(Li)J_i(L_i) favors those whose predicted outcomes best match the job’s weighted priorities.

5.3.2. Exploration. Exploitation alone favors deployments the surrogate already predicts to be good, so relationships that have never been exercised stay untested. Exploration rewards a plan for testing uncertain edges and mechanisms:

Expl⁡(P)=∑e∈EAe(P)U(αe,βe)+ωm∑m∈MAm(P)U(αm,βm),(3)\operatorname{Expl}(P) = \sum_{e \in E} A_e(P)U(\alpha_e,\beta_e) + \omega_m \sum_{m \in M} A_m(P)U(\alpha_m,\beta_m), \tag*{(3)}

where Ae(P)A_e(P) indicates whether any ladder in PP exercises edge ee, and analogously for mechanisms. UU is the normalized posterior variance from before. Mechanisms, weighted by ωm\omega_m, are scored separately because their scope can fail even when each constituent edge is well supported.

5.3.3. Risk. A surrogate is most likely to be wrong when conditions have shifted, which is also when a plan is most likely to breach an SLO. Koi therefore evaluates each prediction against a band rather than a point. Let rjr_j be the historical residuals for objective jj, observed minus predicted as in §5.2, and σ^j\hat{\sigma}_j their empirical standard deviation. The band is

upperj=y^j+Q0.95(rj)+2ϵσ^j,lowerj=y^j+Q0.05(rj)−2ϵσ^j.(4)\begin{aligned} \mathrm{upper}_j &= \hat{y}_j + Q_{0.95}(r_j) + 2\epsilon\hat{\sigma}_j,\\ \mathrm{lower}_j &= \hat{y}_j + Q_{0.05}(r_j) - 2\epsilon\hat{\sigma}_j. \tag*{(4)} \end{aligned}

where QqQ_q is the empirical qq-quantile. The quantile terms carry the observed error distribution and the 2ϵσ^j2\epsilon\hat{\sigma}_j terms inflate it, with ϵ\epsilon in the role of a Wasserstein ambiguity radius around that distribution [13] rather than as the solution to a distributionally robust program. Koi calibrates ϵ\epsilon from realized coverage in the slow loop (§5.4), widening it when outcomes fall outside the band more often than the target rate and tightening it as calibration improves.

From the band, Koi estimates the probability pibrkp_i^{\mathrm{brk}} that each SLO-constrained objective is violated, and combines them as

pibrk(Li)=1−∏j∈Si(1−pjbrk),(5)p_i^{\mathrm{brk}}(L_i) = 1 - \prod_{j \in S_i} (1 - p_j^{\mathrm{brk}}), \tag*{(5)}

where SiS_i is the set of objectives constrained by SiS_i. Treating those objectives as independent is conservative, since latency violations are positively correlated in practice, so the product understates the joint satisfaction probability and overestimates risk.

5.3.4 Switching Cost. The reconfiguration cost for job ii is decomposed into four components:

Switchi=Ccold+Cparallel+Ckill+Crisk.\mathrm{Switch}_i = C_{\mathrm{cold}} + C_{\mathrm{parallel}} + C_{\mathrm{kill}} + C_{\mathrm{risk}}.

All terms are zero for an unchanged ladder: CcoldC_{\mathrm{cold}} captures loading and warm-up, CparallelC_{\mathrm{parallel}} concurrent execution, CkillC_{\mathrm{kill}} draining and termination, and CriskC_{\mathrm{risk}} transient underperformance. The penalty λt\lambda_t adapts to the observed reconfiguration rate (§5.4) to discourage unnecessary migrations. Separately, the swap budget BtB_t imposes the hard C4 cap on how many active jobs may change ladders in one tick.

5.3.5 Cluster Score. The four terms in Sections 5.3.1–5.3.4 combine into a single plan score:

Φ(P)=∑(i,Li)∈P[Ji(Li)−γpibrk(Li)−λtSwitchi(Li(t−1),Li)]+βtExpl⁡(P).(6)\Phi(P) = \sum_{(i,L_i)\in P} \left[J_i(L_i) - \gamma p_i^{\mathrm{brk}}(L_i) - \lambda_t \mathrm{Switch}_i(L_i^{(t-1)},L_i)\right] + \beta_t \operatorname{Expl}(P). \tag*{(6)}

Koi aims to select the highest-scoring feasible plan,

Pt∈arg⁡max⁡P∈FtΦ(P),(7)P_t \in\arg\max_{P \in\mathcal{F}_t} \Phi(P), \tag*{(7)}

where Ft\mathcal{F}_t is the set of plans satisfying C1–C5. Exhaustive maximization over Ft\mathcal{F}_t is infeasible and the per-tick search is budgeted, so Koi commits the highest-scoring feasible plan it finds rather than the maximizer (§5.5). The exploration weight βt\beta_t and the switching penalty λt\lambda_t are adapted by the slow loop (§5.4); the risk weight γ\gamma is fixed.

Self-Calibration

Koi adapts the quantities that govern planning from how well its predictions have held. Edge and mechanism confidence update every tick. On a slower schedule, Koi adjusts the exploration weight βt\beta_t, the swap budget BtB_t, the switching penalty λt\lambda_t, the robustness radius ϵt\epsilon_t, the ideal point z⋆z^\star, and the CUSUM thresholds, each from accumulated residuals.

Most of these follow one signal. Let qtq_t be the fraction of decided evidence–mechanism pairs over a rolling window of WW ticks that fall in Q1, and with q⋆=1q^\star= 1 let the regret signal be

st=1W∑k=t−W+1tmax⁡(0,q⋆−qk),s_t = \frac{1}{W}\sum_{k=t-W+1}^{t}\max(0,q^\star-q_k),

which increases as predictions become less reliable. Regret is derived from the Q1 rate: a high regret signal indicates poor configurations and motivates continued exploration. The exploration weight is given by

βt+1=clip⁡(βtexp⁡(ηβ(st−s⋆)),βmin⁡,βmax⁡),\beta_{t+1} = \operatorname{clip}\left(\beta_t \exp\left(\eta_\beta(s_t-s^\star)\right),\beta_{\min},\beta_{\max}\right),

with target regret s⋆s^\star and adaptation rate ηβ\eta_\beta. The same signal raises the swap budget BtB_t when predictions are unreliable, so more active jobs may be reconsidered. The switching penalty λt\lambda_t instead follows the observed reconfiguration rate, rising as churn grows to damp thrashing.

The radius ϵt\epsilon_t follows realized coverage, widening when outcomes fall outside the band more often than the target rate and tightening otherwise. The ideal reference tracks the best outcomes a job has achieved,

zi,j,t⋆={max⁡r∈Hi,tyr,j+Δj,j maximized,min⁡r∈Hi,tyr,j−Δj,j minimized,z^\star_{i,j,t} = \begin{cases} \max_{r\in\mathcal{H}_{i,t}} y_{r,j}+\Delta_j, & j\ \text{maximized},\\ \min_{r\in\mathcal{H}_{i,t}} y_{r,j}-\Delta_j, & j\ \text{minimized}, \end{cases}

where Hi,t\mathcal{H}_{i,t} is job ii’s deployment history and Δj\Delta_j a margin that preserves dominance. The self-calibration loop relies on a corresponding convergence assumption:

Assumption 2 (Convergence). If jobs, workloads, available capacity, and serving conditions remain stable for sufficiently long, we assume that Koi converges to a stable operating point. At this point, predictions remain accurate, exploration no longer finds materially better plans, reconfiguration diminishes, and the learned causal and robustness state changes only marginally.

Agentic Planner

The search space for a cluster deployment is vast and coupled. Each chain sets about 140 knobs (Table 7), giving at least 21402^{140} configurations per chain before replica counts and multi-rank ladders are considered. A plan chooses one ladder for every job, so the number of candidate plans grows exponentially with the number of jobs, and the choices are coupled because jobs with different workloads and SLOs compete for the same capacity. Koi therefore never enumerates this space. Budget books and backward DAG queries, described next, narrow it for each job, an LLM proposes a small set of ladders within it, and deterministic code evaluates, scores, and selects among them.

A root planner receives the frozen cluster snapshot and assigns each job a budget book, a soft upper bound on which GPU types and instance pools the job may consider. Budgets can overlap in resources selected, and contention is resolved later by a capacity-constrained solver that assigns at most one ladder per job to the available GPUs. The root planner invokes per-job specialist sub-agents to propose candidate ladders within each job’s budget. Each specialist sub-agent receives the job’s model architecture, workload profile, current deployment, budgeted hardware, and all mechanisms whose scope matches the job. It can query the DAG backward: given a target SLO, the sub-agent traces which XX decisions affect it through which VV mediators, focusing on the decisions that matter (§ 5.2). This division lets the root reason about cluster-wide tradeoffs (which scarce accelerators to reserve, which workloads can share resources) while sub-agents focus on the configuration details of individual models.

These proposals are advisory: Koi first validates each candidate for physical feasibility, then invokes the surrogate to predict and score its performance. The resulting ladders compete for finite capacity under the joint selector, which assigns at most one ladder per job to maximize cluster-wide gain (§ 5.3) while accounting for unserved demand, improvement over active deployments, and switching cost. The root planner may inspect the result and propose additional candidates, but cannot override deterministic scores, capacity checks, or constraints C1–C5.

Implementation

Batch and online inference jobs are executed through two different pathways. For batch jobs, large prompt files are sharded and a custom shard manager coordinates shard assignment and completion for each worker. This ensures that every rank and chain planned by Koi make progress on the same job without conflicts. Online jobs are deployed through the Nvidia Dynamo inference deployment framework. Koi decides on the exact chain configuration and desired number of maximum replicas, while Dynamo autoscales within that maximum replica bound in response to request load and desired SLAs. Per-rank telemetry from all running jobs is continuously streamed from the different worker clusters to a unified cluster state store. At each five-minute tick, Koi reads a snapshot of this store to use for the next plan.

Evaluation

Our evaluation asks three questions. First, how does Koi compare with static, classical, outcome-learning, evolutionary, and language-model planners in goodput, SLO attainment, admission, and cost (§ 6.2)? Second, do joint optimization, evidence accumulation, and health-triggered recovery operate as intended, and how do these effects change with deployment scale and duration (§ 6.3)? Third, what do deterministic planning tools, the causal evidence graph, and cross-tick learning contribute, and how sensitive are the results to the language-model backbone (§ 6.4)?

Experimental Setup

Benchmarks. We evaluate Koi in one real-hardware setting and three simulated settings: a 3-hour deployment on a 104-GPU heterogeneous GCP cluster, 3-hour and 12-hour heterogeneous simulations, and a 12-hour homogeneous-cluster simulation. Each setting defines a benchmark consisting of a shared resource pool and a concurrent mix of online and batch workloads. For each benchmark, Koi and each of the ten baselines independently generate the placements and configurations used to serve the same workload.

Each benchmark mixes 25 online and batch workloads across ten model architectures: Llama-3.2-1B, Llama-3.3-70B, Qwen2.5-Coder-32B, Phi-4, Mixtral-8x7B, Kimi-K2-Instruct, GLM-4.5-Air, Qwen3-Next-80B, DeepSeek-V3 and Qwen3-235B-A22B-Instruct, spanning 1B to 235B parameters and including dense, mixture-of-experts, and frontier models. Online workloads distribution range from short moderation (256/8 tokens, TTFT < 100 ms) to long-context summaries (24,000/1,200 tokens, TTFT < 3s) and decode-heavy agentic tasks (1,400/4,096 tokens). Six batch jobs carry fixed deadlines ranging from 20 to 135 minutes. Request traffic follows Poisson, constant, and gamma arrival processes parameterized from production profiles to each job type. The workloads used in simulation and real cluster are summarized in Figure 7 and Table 6 in the Appendix.

Figure 7. 3-hour simulation benchmark schedule and traffic patterns.

WorkloadModelModeISLOSLSLO / deadline
Free multiturn chatLlama-3.2-1BOnline900300TTFT <700 ms; TPOT <55 ms
Premium chatLlama-3.3-70BOnline1,200500TTFT <600 ms; TPOT <38 ms
Code autocompleteQwen2.5-Coder-32BOnline1,800120TTFT <180 ms; TPOT <45 ms
Safety moderationLlama-3.2-1BOnline2568TTFT <100 ms; TPOT <30 ms
Small-model chatLlama-3.2-1BOnline25664TTFT <300 ms; TPOT <40 ms
Phi-4 long-context chatPhi-4Online9,000220TTFT <1.5 s; TPOT <55 ms
Small-model chatLlama-3.2-1BOnline25664TTFT <300 ms; TPOT <40 ms
Support assistantMixtral-8x7BOnline1,400350TTFT <700 ms; TPOT <50 ms
Nightly evaluationLlama-3.3-70BBatch1,8001,200Completion deadline: 135 min
Phi-4 quantization probePhi-4Batch1,000512Completion deadline: 105 min
Code morning burstQwen2.5-Coder-32BOnline2,20096TTFT <160 ms; TPOT <45 ms
Enterprise chatPhi-4Online1,500420TTFT <500 ms; TPOT <42 ms
Product-launch chat burstMixtral-8x7BOnline850280TTFT <650 ms; TPOT <50 ms
Moderation launch burstLlama-3.2-1BOnline3208TTFT <90 ms; TPOT <30 ms
Agentic decode-heavy reasoningKimi-K2-InstructOnline1,4004,096TTFT <2.5 s; TPOT <45 ms
GLM frontier reasoningGLM-4.5-AirOnline1,8002,500TTFT <2.2 s; TPOT <50 ms
Small-active MoE reasoningQwen3-Next-80BOnline1,200900TTFT <1 s; TPOT <45 ms
Phi-4 calibration probePhi-4Batch1,400512Completion deadline: 85 min
P0 safety replayLlama-3.3-70BBatch2,200700Completion deadline: 60 min
P0 safety reasonerPhi-4Online900160TTFT <350 ms; TPOT <40 ms
Long-context meeting summaryQwen3-Next-80BOnline24,0001,200TTFT <3 s; TPOT <65 ms
Sales-report generationQwen3-Next-80BBatch18,0001,400Completion deadline: 55 min
Legal reviewLlama-3.3-70BOnline3,000700TTFT <1 s; TPOT <45 ms
Held-out frontier evaluationDeepSeek-V3Batch1,8001,600Completion deadline: 35 min
MoE exploration probeQwen3-Next-80BBatch1,000800Completion deadline: 20 min

Table 6. Workload overview of 3-hour simulation benchmark.

Real-Hardware Execution. Each experiment is conducted with three Google Cloud Kubernetes clusters across three distinct regions: us-central1, us-east5 and europe-west2. There are 104 GPUs in total, consisting of a heterogeneous mix of H100-80GB, A100-80GB, A100-40GB, and RTXPRO6000. Due to cost constraints, non-evolutionary algorithm based baselines were omitted from real experiments.

Simulation Setup. For the simulation environment, we used LLMServingSim 2.0 [9], a discrete event simulator (DES) as the execution environment for the simulated clusters. Unlike Koi’s internal surrogate model, LLMServingSim 2.0 does not provide point estimates per placement, but rather simulates request execution and queueing semantics for all requests routed to a deployed configuration. After a plan is selected by Koi, LLMServingSim 2.0 executes the corresponding configuration and provides the realized performance observations that would otherwise come from hardware telemetry. It simulates using per-operator compute profiles across GPU, model, and tensor-parallel configurations, and an ASTRA-SIM [26] backend for collective communication. To support all model–GPU combinations in our benchmark, we augmented LLMServingSim 2.0 with layer-wise profiles for the additional LLM models and GPU types used in the evaluation.

Baselines. Koi is compared against ten schedulers, grouped by the capability each baseline is intended to test.

Non-learning baselines. FCFS+First-Fit admits jobs in arrival order and assigns the least-VRAM feasible placement.

MethodBatch prog. (%)Admit online (%)Admit batch (%)$/token (10−410^{-4}) | TTFT met (%) | TPOT met (%) | Goodput (tok/s) | Overall score
Contextual bandit50.0053.33100.000.4522.2333.662291.0351.05
Rolling EvoX50.0080.00100.000.4428.0835.482348.1554.90
OpenEvolve75.00100.00100.000.3065.6691.413497.4384.44
Koi100.00100.00100.000.2479.0599.114119.5194.50
Δ\Delta vs. best baseline+25.00pp+0.00pp+0.00pp+20.00%+13.39pp+7.70pp+17.79%+11.91%

Table 2. Real-hardware results for the 3-hour, 104-GPU heterogeneous benchmark. Bold marks the best value and <u>underline</u> the second-best.

EDF [16] applies the same placement rule but prioritizes earlier deadlines. Best Fixed uses one static hardware and parallelism configuration per job, selected offline by GPT-5.6-Sol with access to the complete workload trace; at runtime, it reuses these configurations and falls back to VRAM-based placement when the selected configuration is unavailable.

Per-job optimizers. Prism [32] estimates memory and KV-cache pressure to place preconfigured model-parallel groups, reallocating memory at runtime but never choosing GPU type, parallelism, or replica count. To adapt Prism to cluster scheduling, we enumerate policy-compliant GPU-pool and TP/PP placement candidates using workload-derived VRAM and KV-cache demand. Prism then admits waiting jobs by assigning high-KV-demand workloads to the least-pressured feasible pool under global capacity constraints.

Global optimizers without mechanism learning. Rolling CP-SAT [23] solves ILP at each tick over the same candidate catalog and constraints as Koi. It enumerates policy-valid placements using checked-in maximum VRAM requirements and resource-map capacity, then optimizes lexicographically for priority-weighted admission, lower estimated reservation cost, and lower GPU footprint. But it never learns from outcomes. Contextual Thompson sampling [4] maintains a Bayesian linear reward model over public job and candidate-placement features, updating only from terminal deployment outcomes. Its bounded reward combines online TTFT, TPOT, and throughput attainment with weights 0.4, 0.4, and 0.2, respectively; for batch jobs, it combines throughput attainment and deadline completion with weights 0.4 and 0.6.

LLM-guided planners. GPT-only (non-agentic) receives a frozen cluster snapshot, containing the same public job, resource, policy, and model-catalog information available to Koi, and generates a placement plan without tools or cross-tick state. Its prompt requires a complete policy-compliant action for each job, specifying feasible GPU pool and TP/PP/DP configuration while respecting current capacity. Agentic additionally uses a REPL and the same read-only inspection and placement-validation tools available to Koi’s agent.

Evolutionary search. Rolling EvoX [17] evolves a deterministic placement function at each tick, adapting its search strategy when progress stalls. OpenEvolve [28] retains a MAP-Elites database of diverse, high-scoring placement programs across ticks. In each baseline, we used the similar objective function as Koi to make it comparable. EvoX’s each reasoning round takes ≈30\approx30 seconds. Considering reasonable scheduling overhead, we set it to max 8 rounds or 5 minutes whatever returns first. Again, we provided the same information with both EvoX and OpenEvolve baselines that was given to Koi.

For the main performance evaluation, we used the same LLM model (GPT-5.6-Luna) for the policies that uses LLM. Additionally, we did LLM-backbone sensitivity study with different LLM models for Koi. To standardize the comparison, each baseline is given the same cluster snapshots, candidate catalog, and safety checks. Differences are limited to the plan-search policy and, for Koi alone, cross-tick mechanism learning.

Metrics. A single evaluator computes every number from per-run traces. All denominators belong to the workload (offered requests, submitted jobs, full campaign sizes, a common clock TT), so no planner can improve a metric by refusing, dropping, or narrowing work. Stochastic planners report the mean and range over at least three runs.

We collapse results into an overall score with three equally weighted pillars: Quality (TTFT SLO met, TPOT SLO met, deadline met), Coverage (online admission, batch admission, batch progress), and Efficiency (goodput and cost per SLO-token, each normalized to the best in each comparison). The score is their geometric mean; no pillar compensates for another. The problem formulation uses the same instantaneous outcome vector for online and batch jobs (§ 4). The evaluation, however, judges batch jobs by completion progress and deadline attainment, which capture finite-work completion rather than instantaneous serving quality.

SLO met is the fraction of offered online requests meeting both TTFT and TPOT p99 targets, averaged per job so one high-volume tenant cannot dominate. Deadline met is the share of batch jobs finishing before their deadline. Admission is the share of submitted jobs whose ranks materialized without a fatal interruption. Batch progress is request-weighted completion over all submitted jobs. Goodput is SLO-honoring output tokens per second: online tokens passing both latency gates plus the projected tokens of deadline-meeting jobs. Cost per SLO-token divides the fleet capacity cost by total SLO-token output; idle GPUs count as waste, so the cheapest planner produces the most SLO output from the same sunk fleet. Planner LLM cost is reported separately.

End-to-End Performance

End-to-End Performance overview. Table 1 reports the end-to-end results. Over three hours, Koi achieves 2,911 goodput tok/s, 45.24% more than the strongest baseline, while reducing cost per token by 31.23%; it also improves batch progress by 26.84% and TTFT/TPOT SLO attainment by 18.89/14.72%. Over twelve hours, Koi admits 95.65% of online jobs and all batch jobs, improves goodput by 5.72%, and leads TTFT/TPOT attainment by 15.99/11.78%, while batch progress is 1.10 points below EvoX and cost is 0.59% higher. Thus, Koi’s gain does not come from maximizing one workload class: it sustains more admitted work and substantially better online latency while remaining competitive on batch progress and cost.

MethodHrsBatch prog. (%)Admit online (%)Admit batch (%)$/token (10−410^{-4}) | TTFT met (%) | TPOT met (%) | Goodput (tok/s) | Overall score
Non-learning baselines
FCFS + first-fit357.7177.7857.142.953.906.891830.8449.13
FCFS + first-fit1238.6978.2650.006.092.456.271772.0131.46
Best fixed configuration357.7166.6757.142.915.2413.611857.3150.24
Best fixed configuration1224.5469.5757.1429.657.6313.55364.0317.59
EDF + first-fit357.7177.7857.142.944.057.271833.1649.27
EDF + first-fit1238.6278.2650.006.092.366.211771.4931.40
Placement and search baselines
Rolling CP-SAT341.9972.2257.146.693.0015.88807.0331.90
Rolling CP-SAT1214.9978.2664.2940.182.8916.83268.5715.83
Contextual bandit344.4266.6757.145.438.0927.08993.0837.49
Contextual bandit1230.5265.2250.0014.169.5623.06762.2024.89
PRISM342.5577.7857.147.062.805.16764.6629.06
PRISM1215.3573.9164.2957.661.944.69187.1811.14
Rolling EvoX357.7166.6757.142.699.9022.102004.0654.34
Rolling EvoX1250.7165.2264.293.3711.9624.023012.0050.01
OpenEvolve356.7066.6757.144.745.2223.411139.0041.74
OpenEvolve1225.0178.2664.2926.249.1916.14411.2519.84
GPT-Only (non-agentic)352.0477.7871.436.313.9818.98855.6035.28
GPT-Only (non-agentic)1217.5469.5771.4316.863.0217.29640.2521.34
Koi384.5594.4471.431.8528.7941.802910.6470.83
Koi1249.6195.65100.003.3927.9535.803184.1563.13
Δ\Delta vs. best baseline3+26.84pp+16.66pp+0.00pp+31.23%+18.89pp+14.72pp+45.24%+30.35%
Δ\Delta vs. best baseline12−1.10pp+17.39pp+28.57pp−0.59%+15.99pp+11.78pp+5.72%+26.23%

Table 1. Simulated benchmark results for 3-hour (400-GPU) and 12-hour (800-GPU) heterogeneous workloads. Bold marks the best value and <u>underline</u> the second-best; ties are marked for all matching methods.

Design Evaluation

Joint optimization. In the real-cluster run, a Qwen3-235B job waits at tick 18 while ten services are running. The only reclaim candidate would reduce a Phi-4 service from two TP2 RTXPRO6000 replicas to one. This would halve its modeled capacity from 1,572 to 786 tok/s, below its 834 tok/s requirement, while its observed p99 TTFT already exceeds the 500 ms target at 645 ms. A full preemption would be worse, reducing that service’s availability. The joint selector rejects the reclaim as not Pareto-better than keeping Phi-4, defers Qwen for two ticks, and admits it at tick 20 after another batch finishes naturally. The ten-minute delay protects an existing service from a predictable SLO regression without sacrificing Qwen’s eventual admission.

Deployment evidence makes planning more calibrated. In the 12-hour simulation on heterogeneous cluster, the mean causal uncertainty Uˉ\bar{U} falls from a first-quarter median of 0.0215 to 0.00053 in the final quarter, a 41× reduction. The fraction of predictions receiving a learned residual correction rises from 2.3% to 10.8%, while their median absolute log error falls from 0.965 to 0.823. During warm-up, the regret signal raises the exploration weight from 0.49 to 2.0 by tick 23. As evidence accumulates, the median DRO radius tightens from 0.379 to 0.260, then widens again after the late workload shift since the uncertainty increases again with it. Thus, Koi becomes more confident where observations repeat while retaining the ability to increase exploration and robustness when conditions change.

Swap. In the real cluster experiment, a Mixtral service reaches critical health at tick 15 after only one of its two A100-80GB replicas materializes and observed p99 TTFT reaches 2,299 ms against a 700 ms target. The per-job sub-agent proposes one physically launchable TP8 A100 rank, and the deterministic scorer assigns a 25.44 gain over keeping the failed deployment, including a 25.00 critical-rehabilitation credit. The joint selector commits only this swap and keeps the other active jobs unchanged. The request-level TTFT pass rate rises from 2.4% before recovery to 99.9% on the relaunched rank, and the job is healthy by tick 18. This example connects telemetry, candidate generation, deterministic scoring, and cluster-wide commitment in one decision.

Learning and performance improvement. We distinguish fleet scale from elapsed learning time using two comparisons. Over roughly three hours, the 104-GPU real cluster commits 20 deployments and selects eight mechanisms, whereas the 800-GPU simulation commits 80 deployments and selects 17 mechanisms. At a larger cluster scale (800 GPUs for 12 hours vs. 104 GPUs for 3 hours), Koi gathered more evidence, increasing from 407 to 2,272 pieces of evidence, and ran more CUSUM tests, increasing from 18,705 to 101,420. The share of decisive mechanism tests that support their predicted mediator–outcome path rises from 1.9% to 19.1%, and ICP acceptance for the same workload-to-mediator edge rises from 11.0% to 63.3%. These runs show that more concurrent deployments activate learning sooner and that a longer horizon strengthens its evidence.

Evidence changes both allocation and planner state. Figure 4 shows Koi increasing its plan-declared allocation above 90% as jobs arrive while sustaining high served-job TTFT and TPOT attainment. As observed outcomes expose prediction-band misses, Koi widens its DRO radius from 0.15 to 0.191 rather than continuing to trust an under-covering bound. In parallel, the cumulative mechanism set grows to 64 as new workload contexts arrive.

Real-cluster allocation, SLO attainment, and adaptation over three hours

Figure 4. Real-cluster allocation, SLO attainment, and adaptation over three hours. (a) Plan-declared logical GPU allocation for each policy, with Koi’s realized swap marked; (b) equal-job mean TTFT (solid) and TPOT (dashed) attainment among served online jobs; (c) Koi’s calibrated DRO radius; and (d) active/applicable and cumulatively discovered mechanisms.

Balance between exploitation and exploration. Figure 5 shows long stable placement intervals under Koi, with changes concentrated on a subset of degraded services. EvoX and the contextual bandit instead produce dense swap sequences across nearly every online workload, while OpenEvolve and GPT-only perform no explicit swaps and largely preserve their initial choices; OpenEvolve’s additional transitions are failure-driven relaunches. Across the complete trace, Koi issues 201 swaps and 2,076 keeps, compared with 1,464 swaps for EvoX and 1,928 for the bandit. The timeline therefore places Koi between two failure modes: repeatedly perturbing productive deployments and never correcting a poor initial placement.

Online workload scheduling across methods

Figure 5. Online workload scheduling across methods. Per-job GPU assignments for nine workloads over 12 hours under Koi, EvoX, OpenEvolve, the contextual bandit, and GPT-only. Colors denote GPU families; markers indicate placement, swap, and rank relaunch events.

Ablation Study

Koi sub-component ablation. Figure 6 and Table 4 compare progressively more complete planners. Giving the language model deterministic prediction, scoring, and validation tools increases goodput from 856 to 1,827 tok/s, raises TTFT attainment from 3.98% to 25.74%, and reduces cost per token from 6.31 × 10⁻⁴ to 2.95 × 10⁻⁴. The complete system reaches 2,911 tok/s; removing the causal graph reduces goodput to 1,770 tok/s, while freezing online updates reduces it to 1,621 tok/s. Full Koi also completes 84.55% of batch work, versus

Figure 6. Ablations over the 3-hour heterogeneous simulation. Overall score, TTFT and TPOT SLO attainment, and goodput across five planner variants.

VariantBatch progress (%)Admit (%) online / batch$/token (10−410^{-4}) | SLO met (%) TTFT/TPOT | Goodput (tok/s) | Overall score
GPT-only (non agentic)52.0477.78/71.436.313.98/18.98855.6035.28
Opencode SDK (GPT-5.6-Luna)54.7788.89/57.142.9525.74/32.621826.6451.55
Koi without DAG/mechanisms59.4294.44/100.003.0521.66/38.331770.0358.34
Koi without online learning63.0194.44/85.713.3316.49/37.741621.4854.79
Full Koi84.5594.44/71.431.8528.79/41.802910.6470.83
Δ\Delta vs. best baseline+21.54+21.54pp−8.00-8.00pp+37.29%+37.29\%+3.05+3.05pp+59.34%+59.34\%+21.41%+21.41\%

Table 4. Agentic and Koi ablations. All runs use the 3-hour workload horizon on the simulator.

59.42% without the graph and 63.01% without online learning, despite admitting fewer batch jobs than either variant. The result separates two roles: deterministic tools ground each planning attempt, while persistent mechanism evidence improves which valid plans are retained and revisited across ticks.

LLM backbone ablation. The backbone ablation localizes the Cohere–GPT-5.6 Sol difference to a single 7,200-request Llama-70B batch campaign. Both planners admit it at 00:15 and never reconfigure it, but Cohere selects one eight-GPU MI300 TP8 chain and finishes at 01:07 with 82 minutes of deadline slack, whereas Sol selects four TP4 A100 replicas using 16 GPUs and reaches 7,115 requests by the 02:30 deadline, missing by 85. Because batch goodput credits the campaign only when it meets its deadline, this single decision contributes 8.64 million tokens and explains 99.3% of their trace-derived goodput gap; their online contributions differ by only 57,921 tokens. The insight is that deterministic validation bounds the action space but cannot make all valid candidates equally useful: a small number of initial proposals can dominate the final result. This single-run comparison does not isolate accelerator type, parallelism, replication, or batching limits, which change together. Another insight is that the achieved outcomes (TTFT, TPOT etc.) of all the jobs, in all ablations are well within three standard deviations of each other - indicating a model agnostic behavior.

Discussion, Limitations and future work

Koi benefits from a larger fleet as it runs more concurrent jobs, exercising more relationships per tick and covering more scopes. Evidence accumulates faster and applies to a wider set of future jobs. Additional headroom also makes exploration affordable without putting an SLO at risk, and planner cost is charged per tick and amortized over the jobs in it. Utilization rises 24.5% between the first and last quarter of the 12-hour run, but the run is too short to reach the convergence conditions. New decision axes enlarge XX without changing Koi’s per-tick cost, which is bounded by the number of candidates evaluated rather than by the size of the space. Expert parallelism, prefill–decode disaggregation, and engine-level tuning are therefore straightforward to admit. Each requires only that telemetry report at least one mediator through which its effect passes, since an axis with no measured mediator can be scored but not tested. Enumerative and black-box search have no such property and require more samples as the space grows. New accelerators, including AMD and TPU can then be additional environments in RtR_t and need no change to the planner. Training, post-training, and fine-tuning jobs have the same three properties Koi relies on: a configuration space that can be enumerated, runtime behavior the stack already reports, and evaluations too expensive to run exhaustively. Extending the planner to admit them would let one controller allocate a fleet across all of these workloads rather than inference alone.

Conclusion

Heterogeneous LLM inference requires more than a better predictor. A useful controller must allocate scarce devices globally, act safely under prediction error, and decide which expensive deployments should improve its future knowledge. Koi addresses these requirements by attaching falsifiable, mediator-grounded mechanisms to deployment plans; using a sandboxed LLM only to propose candidate plans; and keeping scoring, validation, and side effects deterministic. Telemetry closes a calibrated slow loop, while a stateless execution plane keeps fast serving and recovery independent of the planner. The result is a cluster control architecture that can improve with use without making availability depend on an agent’s correctness.

Acknowledgements

Generative AI tools, specifically ChatGPT, Claude Code and DeepSeek, were used to assist with generating sections of this work, including text, tables, graphs and code.

Deployment Configuration Space

Table 7 groups the configuration knobs that a chain sets into families, with representative knobs for each.

FamilyKnobsRepresentative knobs
Environment[?]market, cloud, region, availability zone, GPU type, instance type, nodes per chain, interconnect
Parallelism[?]TP, PP, DP, EP, SP, CP
Engine and runtime[?]engine and version, runtime image, attention backend, CUDA graphs, torch.compile
Batching and scheduling[?]max sequences, max batched tokens, max model length, chunked prefill and chunk size, scheduling, preemption, and router policies
Memory and KV cache[?]GPU memory utilization, block size, KV-cache dtype, prefix caching, KV offloading, KV transfer method
Quantization[?]weight and activation dtypes, weight quantization method and bits, activation quantization method
Speculative decoding[?]method, draft model, speculative tokens, acceptance threshold
Disaggregation[?]prefill/decode split, prefill and decode worker counts
Total∼\sim140

Table 7. Configuration knobs per chain, grouped by family.

Baseline Implementation Details

This section describes the non-Koi baselines that were implemented. It excludes Koi and its ablations.

Common Store-to-plan contract

Each implemented driver independently reads one frozen Store snapshot at a planning tick: the resource map and the running, waiting, paused, and finished job groups for the evaluated user. It subtracts the footprint of running jobs from the resource map before making new assignments. It then emits exactly one lifecycle action for every visible job: keep for running jobs, place or defer for waiting jobs, defer for paused jobs, and preempt for finished jobs. A placement is serialized as a normal Store plan and is checked against the concrete environment or pool capacity before it is recorded.

CP-SAT, contextual Thompson sampling, Prism, and EvoX each locally construct their candidate set from the same kinds of public facts, rather than reading Koi state. For every waiting job, they read the model, token and SLO fields, priority, and an explicit or benchmark VRAM requirement. They enumerate available resource targets and policy-permitted TP values, choose the smallest PP sufficient for the VRAM requirement with DP fixed to one, and record each candidate’s resource keys, GPU footprint, price, model-weight estimate, and KV-demand estimate. The drivers reject candidates that violate the configuration policy or nested environment/pool capacity. This gives the four methods the same job and resource-map interface while leaving their selection rules independent of Koi’s predictions, mechanisms, and objective.

The contextual Thompson-sampling and EvoX drivers are currently registered in the simulator CLI. The best-fit, CP-SAT, AIConfigurator, and Prism drivers are implemented and unit-tested Store adapters, but still need factory registration before they can be selected through the CLI.

Implemented baseline policies

Best fixed configuration. The static control uses a checked-in mapping from benchmark job identity to hardware target and TP/PP/DP configuration. For each waiting job, it resolves that mapping against the current resource map and greedily places the job only when the required target still has capacity. If a mapping is absent or no longer fits, the implementation uses a VRAM-feasible fallback; otherwise it defers the job. The fallback lets the control produce a valid Store plan when a named benchmark configuration cannot be bound to the current fleet. It must be reported as a fallback, not as a development-selected fixed configuration.

Rolling CP-SAT. CP-SAT receives the static candidates and capacity constraints above. It creates one Boolean variable for each job–candidate pair, constrains each job to at most one selected candidate, and constrains every environment and pool footprint to its available capacity. The solver first maximizes priority-weighted job admission. With that value fixed, it minimizes hourly price, GPU footprint, and a stable candidate-order tie break. It uses one worker, seed zero, and a declared time limit. We use a rolling solve because capacity and the waiting queue change each tick; the re-solve is necessary to make the static optimizer operate on the actual current resource map rather than an offline snapshot.

Contextual Thompson sampling. The contextual-bandit baseline uses a Bayesian linear reward model over 11 public action features: an intercept; job priority and queue depth; average input and output lengths; SLO tightness; candidate GPU count, memory, KV-cache pressure, price, and requested capacity share. Jobs are processed in priority order. For each job, the planner samples a weight vector from its current linear-Gaussian posterior, scores the feasible candidates with that sample, and chooses the highest-scoring candidate before reserving its capacity. After the simulator returns an outcome, the driver applies the corresponding rank-one posterior update. Online reward is the fraction of offered output tokens that meet both latency targets, minus a normalized price penalty; batch reward is zero when the deadline is missed.

We use Thompson sampling rather than LinUCB, epsilon-greedy, a neural bandit, or a tree/forest bandit because the action set is discrete, the context is a small explicit feature vector, and the controller needs to act from few observations. Sampling from the posterior performs probability matching: a candidate is explored when uncertainty gives it a plausible chance of being best, instead of using a fixed random-action rate or a separately tuned UCB bonus. The implementation is deliberately a linear Thompson sampler, not Bayesian optimization and not a claim that it dominates the other bandit families on these workloads. Its default posterior-draw scale is 0.25 and its random seed is zero for reproducibility.

Prism port. The Prism adapter translates a memory-pressure placement rule to the aggregate resource map. It orders jobs by decreasing maximum estimated KV demand and then priority. For each job, it filters out candidates that do not leave room for the model weights or exceed remaining nested capacity, then chooses the target with the least accumulated KV demand divided by residual memory. Price, GPU footprint, and candidate ID are deterministic tie breaks. After selection, the target’s residual memory and KV-demand state are updated before placing the next job.

MethodBatch progress (%)Admit (%) online / batch$/token (10−410^{-4}) | SLO met (%) TTFT/TPOT | Goodput (tok/s) | Overall score
Best Fixed Configuration37.5860.87/42.863.5624.38/31.092147.7750.69
Contextual Bandit1.8943.48/7.1414.0410.32/10.95544.3414.59
Rolling EvoX14.9265.22/14.294.7629.61/37.101606.3940.37
Openevolve14.9265.22/14.296.7620.26/32.461130.7733.18
GPT-Only (non agentic)40.0665.22/57.144.2918.15/33.621783.6149.17
Koi36.6278.26/57.143.5131.37/39.292175.6356.68
Δ\Delta vs. best baseline−3.44-3.44pp+8.11+8.11pp+1.40%+1.40\%+1.76+1.76pp / +2.19+2.19pp+1.30%+1.30\%+11.82%+11.82\%

Table 3. Mixed online/batch homogeneous results. All runs use the 12-hour workload horizon on the simulator.

Base modelBatch progress (%)Admit (%) online / batchPlanner $/tick | Overall score
GPT-5.6 Sol67.3194.44/85.710.7562.12
DeepSeek-V4 Pro48.7694.44/71.430.3050.44
Cohere Command-A67.3794.44/85.710.4468.84
Koi (default, GPT-5.6 Luna)84.5594.44/71.430.05770.83

Table 5. Koi backbone LLM ablation. All runs use the 3-hour workload horizon on the simulator.

This preserves the memory-centric placement idea while using Store-expressible, exclusive shapes and the common safety checks. It is a documented heuristic port, not a claim that the full Prism artifact, including its complete sharing and scheduling runtime, is executed unchanged.

Rolling EvoX. At each tick, EvoX receives the static candidate payload and nested capacities. SkyDiscover evolves a deterministic Python function, compute_placement(payload), that returns one candidate ID or a defer decision for every waiting job. The evaluator rejects malformed or infeasible programs and scores valid outputs by priority-weighted admission, followed by hourly price and GPU-footprint tie breaks. The initial program is a simple priority-first greedy

policy; the best program from one tick seeds the next tick’s search. The winning program is rerun on the frozen payload, its choices are revalidated, and valid choices become Store placement actions.

Figure 8. 3-hour simulation benchmark workload distribution characteristics.

EvoX is the search procedure in this baseline; there is no second optimizer. Candidate construction and feasibility checks stay in the simulator so evolved code cannot invent hardware or bypass capacity rules. The rolling driver adapts to the current queue and resource map, but its EvoX evaluator currently uses static placement facts rather than simulated serving outcomes as its fitness signal. The frozen-snapshot EvoX table uses the same candidate encoding and evaluator without the cross-tick lifecycle loop.

(London, UK). The 3 clusters combine to a fleet of 104 heterogeneous GPUs, consisting of a mix of Nvidia H100-80GB, RTX PRO 6000, A100-80GB and A100-40GB. The H100s and A100s are placed in 8-GPU instances, and the RTX PRO 6000 are placed in both 8-GPU instances and 4-GPU instances.

Real hardware configuration

Each of the real experiments are conducted on 3 Google Kubernetes Engine (GKE) clusters, in the following regions: us-central1 (Iowa, USA), us-east5 (Ohio, USA), europe-west2

References

  1. [1]Virginia Aglietti, Xiaoyu Lu, Andrej Paleyes, and Javier González. 2020. Causal Bayesian Optimization. In Proceedings of the Twenty-Third International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research, Vol. 108). PMLR, Palermo, Sicily, Italy, 3155–3164. https://proceedings.mlr.press/v108/aglietti20a.htmlarxiv.org/abs/2005.11741
  2. [2]Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav S. Gulavani, Ramachandran Ramjee, and Alexey Tumanov. 2024. Vidur: A Large-Scale Simulation Framework for LLM Inference. In Proceedings of Machine Learning and Systems, Vol. 6. 351–366. https://proceedings.mlsys.org/paper_files/paper/2024/hash/b74a8de47d2b3c928360e0a011f48351-Abstract-Conference.htmlarxiv.org/abs/2405.05465
  3. [3]Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, 117–134. https://www.usenix.org/conference/osdi24/presentation/agrawalarxiv.org/abs/2403.02310
  4. [4]Shipra Agrawal and Navin Goyal. 2013. Thompson Sampling for Contextual Bandits with Linear Payoffs. In Proceedings of the 30th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 28). PMLR, 127–135.arxiv.org/abs/1209.3352
  5. [5]Omid Alipourfard, Hongqiang Harry Liu, Jianshu Chen, Shivaram Venkataraman, Minlan Yu, and Ming Zhang. 2017. CherryPick: Adaptively Unearthing the Best Cloud Configurations for Big Data Analytics. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). USENIX Association, 469–482. https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/alipourfard
  6. [6]Matei Balog, Alexander Novikov, Ngan Vu, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, and Pushmeet Kohli. 2025. AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery. Google DeepMind technical report. https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/AlphaEvolve.pdf
  7. [7]Romil Bhardwaj, Kirthevasan Kandasamy, Asim Biswal, Wenshuo Guo, Benjamin Hindman, Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica. 2023. Cilantro: Performance-Aware Resource Allocation for General Objectives via Online Feedback. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). USENIX Association, 623–643. https://www.usenix.org/conference/osdi23/presentation/bhardwaj
  8. [8]Ke Cheng, Zhi Wang, Wen Hu, Tiannuo Yang, Jianguo Li, and Sheng Zhang. 2024. SLO-Oriented Performance Tuning for LLM Inference Engines. arXiv preprint arXiv:2408.04323 (2024).arxiv.org/abs/2408.04323
  9. [9]Jaehong Cho, Hyunmin Choi, Guseul Heo, and Jongse Park. 2026. LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure. In 2026 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). 1–14. doi:10.1109/ISPASS69722.2026.020012
  10. [10]Kanishk Goel, Jayashree Mohan, Nipun Kwatra, Ravi Shreyas Anupindi, and Ramachandran Ramjee. 2026. QoServe: Breaking the Silos of LLM Inference Serving. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 1492–1507. doi:10.1145/3779212.3790206DOI
  11. [11]Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, and Ion Stoica. 2024. Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity. arXiv:2404.14527 [cs.DC] https://arxiv.org/abs/2404.14527
  12. [12]Juncheng Guo, Mosharaf Chowdhury, Kang G. Shin, Yibo Zhu, Myeongjae Jeon, Junjie Qian, Hongqiang Liu, and Chuanxiong Guo. 2019. Tiresias: A GPU Cluster Manager for Distributed Deep Learning. In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19). USENIX Association, 485–500. https://www.usenix.org/conference/nsdi19/presentation/gu
  13. [13]Daniel Kuhn, Soroosh Shafiee, and Wolfram Wiesemann. 2025. Distributionally Robust Optimization. arXiv:2411.02549 [math.OC] doi:10.48550/arXiv.2411.02549DOI
  14. [14]Woosuk Kwok, Zhaohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles. 611–626. doi:10.1145/3600006.3613165
  15. [15]Zhuohan Li, Lianmin Zheng, Yunmin Zhang, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). USENIX Association, 663–679. https://www.usenix.org/conference/osdi23/presentation/li-zhouhan
  16. [16]C. L. Liu and James W. Layland. 1973. Scheduling Algorithms for Multiprogramming in a Hard-Real-Time Environment. J. ACM 20, 1 (1973), 46–61. doi:10.1145/321738.321743DOI
  17. [17]Shu Liu, Shubham Agarwal, Monishwaran Maheswaran, Mert Cemri, Zhifei Li, Qiuyang Mang, Ashwin Naren, Ethan Boneh, Audrey Cheng, Melissa Z. Pan, Alexander Du, Kurt Keutzer, Alexandros G. Dimakis, Koushik Sen, Matei Zaharia, and Ion Stoica. 2026. EvoX: Meta-Evolution for Automated Discovery. arXiv:2602.23413 [cs.LG] doi:10.48550/arXiv.2602.23413DOI
  18. [18]Ziming Mao, Tian Xia, Zhanghao Wu, Wei-Lin Chiang, Tyler Griggs, Romil Bhardwaj, Zhongheng Yang, Scott Shenker, and Ion Stoica. 2025. SkyServe: Serving AI Models across Regions and Clouds with Spot Instances. In Proceedings of the Twentieth European Conference on Computer Systems. 159–175. doi:10.1145/3689031.3717459DOI
  19. [19]Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. 2025. Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1. 586–602. doi:10.1145/3669940.3707215
  20. [20]Kaisa Miettinen. 1999. Nonlinear Multiobjective Optimization. International Series in Operations Research & Management Science, Vol. 12. Kluwer Academic Publishers. doi:10.1007/978-1-4615-5563-6
  21. [21]E. S. Page. 1954. Continuous Inspection Schemes. Biometrika 41, 1–2 (1954), 100–115. doi:10.1093/biomet/41.1-2.100DOI
  22. [22]Archit Patke, Dhemath Reddy, Saurabh Jha, Haoran Qiu, Christian Pinto, Chandra Narayanaswami, Zbigniew Kalbarczyk, and Ravishankar Iyer. 2024. Queue Management for SLO-Oriented Large Language Model Serving. In Proceedings of the 2024 ACM Symposium on Cloud Computing. 18–35. doi:10.1145/3698038.3698253
  23. [23]Laurent Perron, Frédéric Didier, and Steven Gay. 2023. The CP-SAT-LP Solver. In 29th International Conference on Principles and Practice of Constraint Programming (CP 2023) (Leibniz International Proceedings in Informatics (LIPIcs), Vol. 280). Schloss Dagstuhl – Leibniz-Zentrum für Informatik, 3:1–3:2. doi:10.4230/LIPIcs.CP.2023.3DOI
  24. [24]Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. 2016. Causal Inference by using Invariant Prediction: Identification and Confidence Intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 78, 5 (2016), 947–1012. doi:10.1111/rssb.12167
  25. [25]Aurick Qiao, Sang Keun Choe, Suhas Jayaram Subramanya, Willie Neiswanger, Qirong Ho, Hao Zhang, Gregory R. Ganger, and Eric P. Xing. 2021. Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning. In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21). USENIX Association, 1–18. https://www.usenix.org/conference/osdi21/presentation/qiaoarxiv.org/abs/2008.12260
  26. [26]Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, and Tushar Krishna. 2020. ASTRA-SIM: Enabling SW/HW Co-Design Exploration for Distributed DL Training Platforms. In 2020 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). 81–92. doi:10.1109/ISPASS48437.2020.00018DOI
  27. [27]Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. 2024. Mathematical Discoveries from Program Search with Large Language Models. Nature 625 (2024), 468–475. doi:10.1038/s41586-023-06924-6
  28. [28]Asankhaya Sharma. 2025. OpenEvolve: An Open-Source Evolutionary Coding Agent. https://github.com/algorithmicsuperintelligence/openevolve. Open-source evolutionary coding agent.
  29. [29]Tianhao Xu, Yiming Liu, Xianglong Lu, Yijia Zhao, Xuting Zhou, Aichen Feng, Yici Chen, Yi Shen, Qin Zhou, Xumeng Chen, Ilya Sherstyuk, Haorui Li, Rishi Thakkar, Ben Hamm, Yuanzhe Li, Xue Huang, Wenpeng Wu, Anish Shanbhag, Harry Kim, Chuan Chen, and Junjie Lai. 2026. AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving. arXiv preprint arXiv:2601.06288 (2026). doi:10.48550/arXiv.2601.06288DOI
  30. [30]Zongheng Yang, Zhanghao Wu, Michael Luo, Wei-Lin Chiang, Romil Bhardwaj, Woosuk Kwon, Siyuan Zhuang, Frank Sifei Luan, Gautam Mittal, Scott Shenker, and Ion Stoica. 2023. SkyPilot: An Intercloud Broker for Sky Computing. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association, 437–455. https://www.usenix.org/conference/nsdi23/presentation/yang-zongheng
  31. [31]Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, 521–538. https://www.usenix.org/conference/osdi22/presentation/yu
  32. [32]Shan Yu, Yifan Qiao, Mingyuan Ma, Yangmin Li, Shuo Yang, Xinyuan Tong, Yang Wang, Zhiqiang Xie, Yuwei An, Shiyi Cao, Ke Bao, Deepak Vij, Xiaoning Ding, Yichen Wang, Qingda Lu, Zhong Wang, Gao Gao, Harry Xu, Junyi Shu, Jiarong Xing, and Ying Sheng. 2025. Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning. Accepted to OSDI 2026. arXiv:2505.04021 [cs.DC] https://arxiv.org/abs/2505.04021
  33. [33]Alex L. Zhang, Tim Kraska, and Omar Khattab. 2026. Recursive Language Models. arXiv:2512.24601 [cs.AI] https://arxiv.org/abs/2512.24601

Paper details

vlnc:W_vvcaup2p
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

ChatGPT, Claude Code, DeepSeek

View statement
Looking for related papers…
Contents