betaFeedback

WorksComputer Science

Towards Fair Resource Sharing in Agentic LLM Serving

Abstract

As LLM agents increasingly execute complex, multi-step tasks on shared inference infrastructure, resource contention across the end-to-end LLM serving stack causes severe and uneven slowdowns across agents. Existing agentic serving systems primarily optimize throughput and latency, with comparatively limited attention to fairness among concurrent agents. Although fairness has been studied in conventional LLM serving at the token and request levels, these approaches do not fully capture cumulative delays and stateful resource dependencies across successive LLM calls within an agent. Moreover, interference spans request routing, runtime scheduling, and hardware resources, making agent-level fairness challenging to define and optimize across an end-to-end serving stack. This paper presents a cross-layer framework to define agent-level fairness, attribute sources of unfairness, and optimize fairness while preserving throughput. To account for agents with widely varying execution demands, we build a theoretical foundation to define fairness using slowdown relative to standalone execution. Our empirical analysis reveals that routing, runtime scheduling, and hardware contention jointly determine fairness, with performance-oriented policies often leading to prolonged queuing delays. To optimize fairness, we introduce Duru, which monitors each agent's slowdown and uses this signal to guide admission and scheduling decisions across the router and runtime. Duru further alleviates starvation through admission exceptions and integrates slowdown-driven autoscaling to adhere to Service-Level Objectives (SLOs). Our evaluation across various benchmarks and hardware platforms shows that Duru reduces 99th-percentile normalized slowdown by a geometric mean of 44.4% (up to 79.5%) while delivering 91.8-127.4% of the state-of-the-art system's throughput.

Introduction

An LLM agent places a language model in a loop, repeatedly invoking the model and interacting with an external environment through tools such as Linux terminals, databases, and web APIs to solve complex tasks [90]. Agents are increasingly used today in solving real-world problems in software engineering [24, 64, 85, 89], deep research [34, 59, 70, 93], and computer use [8, 23, 30, 88]. To complete a single user task, an agent executes successive LLM calls with growing context, typically on inference infrastructure shared with other agents. Resource contention across the hardware and software layers of this infrastructure can cause some agents to disproportionately delay others, particularly when their computational demands differ. Consequently, ensuring fair resource sharing within LLM serving infrastructure is critical for agentic workloads.

Recent research on agentic LLM serving has primarily focused on improving performance and resource utilization through various techniques including through KV cache reuse, prefix sharing, and program-aware scheduling [28, 33, 43, 60, 81]. While these optimizations improve aggregate efficiency, they can also favor agents with reusable contexts, potentially at the expense of others. Fairness has also been studied in LLM serving: VTC [71] balances token-level service across clients, DLPM [9] incorporates prefix locality into fair scheduling, and FairInference [6] targets per-token latency isolation. More recently, SAGA [22] considered agent-workflow fairness through scheduling based on predicted remaining work and deadlines. However, whether these mechanisms adequately address fairness over an agent’s entire execution remains unclear. This raises a fundamental question: Do existing LLM serving fairness mechanisms work out of the box for agentic workloads?

Extending fairness from individual LLM requests to complete agent executions presents three fundamental challenges. First, agents are stateful: delays accumulate across dependent LLM requests [37, 96], so request-level fairness does not necessarily translate into fairness in end-to-end task completion. Second, agents are heterogeneous: the computational cost of processing tokens varies with context length, inference phase, and resource contention [10, 63, 91], making token counts insufficient to capture the actual service received by each agent. Third, agents are unpredictable: their future execution depends on intermediate decisions, including failures that trigger additional reasoning and retries [37, 96], and the strength of the model-harness combination, making remaining work difficult to estimate with high accuracy, as required by prediction-based approaches such as SAGA [22]. Addressing these challenges requires measuring cumulative agent-level slowdown, accounting for heterogeneous computational costs, and correcting unfairness online without predicting future execution.

In this paper, we present a principled framework for understanding and optimizing fairness in agentic LLM serving that captures cumulative delays and heterogeneous computational costs without requiring predictions of future agent execution. The goals of our framework are threefold. First, we seek to establish a theoretical foundation for fairness in single- and multi-agent workflows that distinguishes unfair resource allocation from differences in task complexity and overall system load. Second, we aim to provide empirical evidence of how hardware contention, runtime scheduling, and routing decisions interact to create unfairness across the end-to-end serving stack. Third, we aim to translate these insights into a coordinated, fairness-aware serving system that alleviates starvation while limiting throughput degradation.

We show how to establish slowdown-based fairness from shared hardware [48, 76] to agentic workloads, capturing cumulative delays across dependent LLM calls and multi-agent workflows. By comparing observed execution against estimated standalone performance and ideal processor sharing, our metric distinguishes unfair resource allocation from intrinsic task complexity and overall system load. We estimate this metric online from observed execution progress, without predicting future agent behavior. Our empirical analysis across the hardware, runtime, and routing layers reveals a fundamental tension: optimizations that improve serving efficiency can inadvertently worsen fairness. Long contexts amplify interference, KV cache retention can starve waiting agents, and admission control can shift delays from GPU execution to router queues.

This work’s objective is to minimize 99th percentile (p99) agent-level unfairness while limiting throughput degradation. To this end, we design Duru, a fairness-aware serving system that coordinates scheduling and admission control across the router and runtime. Using online estimates of agent-level normalized slowdown, Duru prioritizes agents that have fallen behind their fair share, rather than favoring requests solely for their immediate execution efficiency. To balance fairness and throughput, Duru combines slowdown-aware scheduling with capacity-aware resource allocation, preserving KV cache reuse while introducing targeted admission exceptions to alleviate starvation. However, fairness optimization alone cannot resolve performance degradation caused by insufficient capacity. To address this, Duru incorporates SLO-driven autoscaling that adjusts GPU capacity based on agent-level slowdown under varying workloads. Collectively, these mechanisms improve fairness and SLO attainment while maintaining high throughput.

We evaluate Duru using three LLMs (Qwen3-Coder-30B-A3B, Llama-3.3-70B, and DeepSeek-V4-Flash) and three real-world agentic workload traces: Azure [37], TraceLab [96], and AgentX [68]. We use both NVIDIA and AMD GPUs to demonstrate that Duru is

vendor-agnostic. We compare against state-of-the-art production frameworks, including NVIDIA Dynamo [55] and llm-d [41], as well as recent agent-aware scheduling and KV cache management techniques [33, 81]. Our evaluation shows that Duru improves p99 fairness by a geometric mean of 44.4% (up to 79.5%) while maintaining 91.8–127.4% of baseline throughput. In a 48-hour replay of the production trace [37], Duru achieves 97.4% agent-level SLO attainment with comparable throughput while using fewer GPUs than NVIDIA Dynamo. Our key contributions are as follows.

  • Definition. We formulate slowdown-based fairness for single and multi-agent workflows, capturing unfairness relative to ideal resource sharing without predicting future execution.

  • Attribution. We reveal how performance-oriented optimizations across hardware, runtime, and routing layers can amplify unfairness or shift it between layers, leading to agent starvation.

  • Optimization. We design Duru to reduce p99 unfairness via coordinated, slowdown-aware scheduling and admission control with limiting throughput loss, and SLO-driven autoscaling.

Background

LLM Agent

An LLM agent places a language model in a loop to interact with an environment, such as a Linux terminal, database, or web browser, using tool outputs to guide LLM calls. The agent harness [4, 12, 58, 78, 89] orchestrates LLM calls and tool execution, passing results between the model and its environment. In a single-agent system [90], one agent orchestrates this loop, e.g., a coding agent reads files, edits code, and executes tests. Multi-agent systems [24, 64, 86] instead decompose tasks into interdependent subtasks assigned to specialized agents that exchange results and may execute concurrently.

In both settings, an agentic loop alternates between LLM calls and tool execution. As an agent executes successive LLM calls, its context grows with instructions, LLM outputs, and tool results, much of which forms a reusable prefix [37, 91]. Reusing the attention keys and values (KV cache) computed for this prefix avoids redundant prefix computation in subsequent calls [17, 32]. This paper focuses on understanding and optimizing fairness across the end-to-end LLM serving stack.

LLM Serving Stack for Agentic Inference

Figure 1 illustrates the end-to-end LLM serving stack, comprising three layers: router, runtime, and hardware. For each LLM call, the agent harness sends an HTTP request and receives generated tokens. After a tool call, the harness may return with updated context. We describe the three serving layers below.

Figure 1. An end-to-end agentic LLM serving stack.

Router layer. The routing layer is typically the first scheduling stage encountered by an incoming HTTP request, dispatching it to one or more inference workers. Systems such as NVIDIA Dynamo [55], llm-d [41], and SGLang Model Gateway [69] support distributed request routing. A primary objective is to minimize latency while balancing worker utilization against KV cache locality. Load-based routing [55, 74] distributes requests to avoid queuing but may scatter successive calls from the same agent across workers, losing KV cache reuse opportunities. Conversely, cache-aware routing preserves reusable context prefixes at the risk of overloading cache-rich workers. Modern routers [41, 55] combine cache affinity with worker load and may perform admission control or select separate prefill and decode workers. Agent-aware policies [28, 84] further exploit session- and program-level information to coordinate placement across successive LLM calls.

Runtime layer. Within each inference worker, the runtime schedules queued requests and manages KV caches. Inference engines such as vLLM [32] and SGLang [92] employ first-come, first-served (FCFS) scheduling with continuous batching. A primary objective is to maximize throughput while minimizing latency through efficient GPU computation and KV cache utilization. For agentic workloads, retaining KV caches during tool execution avoids redundant prefill but occupies GPU memory while agents are idle. Agent-aware systems [7, 33, 39, 43, 60, 61, 81] coordinate scheduling and KV-cache management using execution dependencies and tool calls.

Hardware layer. LLM serving runs on hardware accelerators such as GPUs [11, 25], TPUs [26, 27], and specialized AI accelerators [36, 50, 73]. Each inference worker runs a model on one or more accelerators, using model parallelism and high-bandwidth interconnects when spanning multiple devices [51, 56, 67]. Within a worker, continuously batched requests contend for compute and memory bandwidth, while agents’ growing contexts and retained KV caches place additional pressure on limited GPU memory (HBM). To address these limitations, systems can offload KV caches to host DRAM or SSD [38, 65], or disaggregate prefill and decode across separate workers to reduce interference [63, 95].

Real-World LLM Serving

LLM serving operates under different degrees of resource sharing and performance isolation. Our survey of LLM providers (Table 1) identifies service tiers that trade off cost, latency, and isolation. At one extreme, dedicated endpoints provide exclusive GPUs to individual tenants [52, 82], while guaranteed-throughput offerings reserve capacity [21] [46]. Priority tiers offer preferential processing and predictable latency on shared infrastructure [3, 20, 45]. In contrast, standard, flex, and batch tiers give providers greater scheduling flexibility, with flex and batch offerings trading higher latency or asynchronous completion for lower cost [1, 3, 19, 83].

TierDescriptionExamples
DedicatedIsolated, single-tenant GPU inference.Together AI, Nebius [52, 82]
ReservedGuaranteed throughput; reserved capacity.Azure, Google [21] [46]
PriorityPreferential service with lower latency.Azure, Google, AWS [3, 20, 45]
StandardShared capacity; throughput varies with load.Google, AWS [3, 20]
FlexDiscounted; higher latency or throttling.Google, AWS [3, 19]
BatchDiscounted asynchronous bulk inference.Together AI, Alibaba [1, 83]

Table 1. LLM serving tiers and resource sharing.

While service tiers differ in their performance guarantees, shared infrastructure exposes requests to interference from other tenants. Serving engines such as vLLM [32] and TensorRT-LLM [54] employ continuous batching to improve GPU utilization, but introduce co-batch interference [32, 57]. Latency-sensitive tiers may limit such interference by reserving capacity or reducing worker load [77], whereas best-effort tiers can favor higher utilization through more aggressive batching. Thus, performance guarantees determine resource sharing and acceptable interference across tiers.

Fairness in Resource Scheduling

Fairness has been extensively studied across shared computing resources. In DRAM and LLCs, fairness mechanisms [16, 42, 48, 49] aim to minimize interference-induced slowdowns while preserving throughput. Operating system schedulers [79, 80] approximate equal or weighted processor sharing through virtual-runtime and deadline-based scheduling. Similarly, network schedulers approximate generalized processor sharing through fair queuing, with flow slowdown used to characterize interference [14, 47, 62, 72]. In LLM serving, VTC [71] tracks per-client service using weighted input and output tokens, while DLPM [9] additionally considers KV cache locality to balance fairness and efficiency. FairInference [6] extends beyond service fairness to provide token-level latency isolation by accounting for GPU computation and KV cache interference.

Fairness for agentic workloads. Unlike single-turn inference, agentic workloads involve dependent LLM calls with varying context lengths and concurrent subagents [7, 29, 37, 96]. Interference and queuing delays accumulate across calls, so token- or request-level fairness does not necessarily ensure agent-level fairness. Recent agent-aware systems exploit program structure to improve scheduling [28, 43, 84], while SAGA [22] addresses workflow-level fairness. However, existing approaches do not jointly measure and optimize agent-level slowdown from routing, runtime scheduling, KV cache management, and hardware contention, motivating fairness coordination across the entire LLM serving stack.

Defining Fairness for Agentic LLM Serving

An absolute latency deadline is ill-suited for assessing fairness across agent sessions, whose execution time depends on task complexity, the model, and the agent harness [29, 53, 89, 91]. Agent workloads range from a few calls to hundreds, spanning seconds to hours [37, 53, 91, 94]. A fixed latency target would therefore conflate intrinsic task complexity with delays imposed by resource sharing. Instead, we employ slowdown, an established fairness metric in shared-resource systems [16, 48], comparing execution time under sharing (including queuing) against standalone execution time. Applying this principle to agentic workloads requires accounting for cumulative delays across dependent LLM calls and fair-share entitlements among concurrent sub-agents. To this end, we establish a unified theoretical formulation for single- and multi-agent workflows relative to ideal processor sharing, enabling online fairness estimation and cross-layer optimization.

Definition 3.1 (Single-agent slowdown). For an agent session ii, let TialoneT_i^{\mathrm{alone}} denote its execution time when served alone, TisharedT_i^{\mathrm{shared}} its execution time under sharing, and TiqueueT_i^{\mathrm{queue}} its cumulative waiting time before and between execution intervals. The slowdown of session ii is

Si=Tishared+TiqueueTialone≥1.(1)S_i = \frac{T_i^{\mathrm{shared}} + T_i^{\mathrm{queue}}}{T_i^{\mathrm{alone}}} \ge1. \tag*{(1)}

We define slowdown in the time domain because GPU requests contend for multiple resources, including compute, memory bandwidth, memory capacity, and interconnect bandwidth. Since these resources have different units, quantifying their combined interference is challenging. Time provides a common measure: contention for any resource manifests as increased latency, whether through queuing delays or slower execution. Thus, slowdown captures the impact of resource sharing regardless of the source of contention.

For multi-agent tasks, we aggregate the root agent rr and spawned agents’ slowdowns, weighted by their shares of standalone work.

Definition 3.2 (Multi-agent slowdown). For a task initiated by root agent rr, let A(r)\mathcal{A}(r) denote the set of rr and all its spawned agents. For each j∈A(r)j \in\mathcal{A}(r), define

wj=Tjalone∑k∈A(r)Tkalone.w_j = \frac{T_j^{\mathrm{alone}}}{\sum_{k \in\mathcal{A}(r)} T_k^{\mathrm{alone}}}.

The task slowdown of rr is

Sr=∑j∈A(r)wjSj=∑j∈A(r)(Tjshared+Tjqueue)∑j∈A(r)Tjalone.(2)S_r = \sum_{j \in\mathcal{A}(r)} w_j S_j = \frac{\sum_{j \in\mathcal{A}(r)} \left(T_j^{\mathrm{shared}} + T_j^{\mathrm{queue}}\right)}{\sum_{j \in\mathcal{A}(r)} T_j^{\mathrm{alone}}}. \tag*{(2)}

The standalone-time weights cancel in the second expression, so task slowdown is the ratio of aggregate inference time under sharing to aggregate standalone inference time.

Normalized slowdown. Absolute slowdown conflates unfair scheduling with heavy load: a job can slow down despite receiving its fair share. We therefore normalize against ideal processor sharing (PS) [62], which divides capacity equally among active jobs.

Definition 3.3 (Normalized slowdown, UiU_i). Let G(t)G(t) be the number of available workers, N(t)N(t) the number of root agents with at least one outstanding (waiting or running) request, and kr(t)k_r(t) the number of agents in root agent rr’s multi-agent session with at least one outstanding request, all at time tt. Let IjI_j be the times agent jj has a request outstanding, excluding tool-execution gaps, so that ∣Ij∣=Tjqueue+Tjshared\lvert I_j\rvert= T_j^{\mathrm{queue}} + T_j^{\mathrm{shared}}. In the PS reference, each multi-agent job receives an equal share G/N(t)G/N(t) of the workers, split equally among its kr(t)k_r(t) outstanding agents, and no agent exceeds one worker, so agent jj associated with root agent rr receives a fraction

ρj(t)=min⁡{1,G(t)/N(t)kr(t)}.\rho_j(t) = \min\left\{1,\frac{G(t)/N(t)}{k_r(t)}\right\}.

of its standalone processing rate. We define the PS-relative slowdown of agent jj and the multi-agent session associated with root agent rr (with wjw_j as in Definition 3.2) as

Uj=1Tjalone∫Ijρj(t) dt,Ur=∑j∈A(r)wjUj.(3)U_j = \frac{1}{T_j^{\mathrm{alone}}}\int_{I_j}\rho_j(t)\,\mathrm{d}t,\qquad U_r = \sum_{j \in\mathcal{A}(r)} w_j U_j. \tag*{(3)}

Thus, UiU_i is the ratio of the service job ii was entitled to under PS to the standalone-equivalent work it completed.

Estimating TialoneT_i^{\mathrm{alone}}. TialoneT_i^{\mathrm{alone}} cannot be directly observed at runtime under resource sharing. We estimate it using offline measurements of isolated prefill and decode latencies under the same model, hardware, and serving configuration. At runtime, we interpolate these measurements to estimate each iteration’s standalone latency and sum them to obtain TialoneT_i^{\mathrm{alone}}. Interpolation is effective because the latency of memory-bound kernels grows approximately linearly with the amount of KV cache data accessed, particularly during decoding and prefill with long cached prefixes [37, 91].

Decode. We model isolated single-iteration latency by KV-cache size and linearly interpolate between measured points.

Prefill. We measure prefill latency across new token counts and cached prefix sizes, constructing a two-dimensional grid. We estimate latency through bilinear interpolation between grid points.

Practicality. Calibration is a one-time cost per deployment configuration, as the resulting latency tables are independent of workloads. Estimating TialoneT_i^{\mathrm{alone}} incrementally at each iteration also eliminates the need to know a request’s output length in advance.

Validation. We validate TialoneT_i^{\mathrm{alone}} by comparing estimated and measured inference times on GPUs using agent traces from Azure’s GitHub Copilot [37], TraceLab [96], and AgentX [68]. On Qwen3-Coder-30B-A3B (RTX PRO 6000), 534 Azure and TraceLab sessions yield mean absolute relative errors of 1.9% and 1.8%, respectively, with maximum errors of 4.1% and 4.2%. On Llama-3.3-70B (2×H100, TP=2), 200 sessions totaling 1.95M output tokens yield mean errors of 0.2%, 0.3%, and 0.1% for Azure, TraceLab, and AgentX, respectively, with maximum errors of 1%, 2.6%, and 0.3%.

Definition 3.4 (Live slowdown). Let Kj(t)K_j(t) be the number of LLM iterations agent jj has completed by time tt, and t^j,k\hat{t}_{j,k} the estimated standalone latency of its kkth iteration, interpolated from its number of new tokens and KV cache size. Its estimated completed work is W^j(t)=∑k=1Kj(t)t^j,k\hat{W}_j(t) = \sum_{k=1}^{K_j(t)} \hat{t}_{j,k}. When W^r(t)=∑j∈A(r)W^j(t)>0\hat{W}_r(t) = \sum_{j \in\mathcal{A}(r)} \hat{W}_j(t) > 0, the live slowdown S^r(t)\hat{S}_r(t) and live normalized slowdown U^r(t)\hat{U}_r(t) of task rr are

1W^r(t)∑j∈A(r)∫Ij∩[0,t]ρj(t′) dt′.(4)\frac{1}{\hat{W}_r(t)} \sum_{j \in\mathcal{A}(r)} \int_{I_j \cap[0,t]} \rho_j(t')\,\mathrm{d}t'. \tag*{(4)}

with ρj≡1\rho_j \equiv1 for S^r(t)\hat{S}_r(t) and ρj\rho_j as in Definition 3.3 for U^r(t)\hat{U}_r(t).

This estimate updates from observed waiting and completed work, without predicting future requests or remaining time.

Intuition. A normalized slowdown of Ui=1U_i = 1 indicates that an agent received its fair share under processor sharing. A value Ui>1U_i > 1 indicates the agent fell behind its fair share, whereas Ui<1U_i < 1 indicates it received more. These values identify disproportionately delayed agents, enabling fairness-aware system design.

Attributing Sources of Unfairness

We posit that agent-level unfairness arises from interacting effects across the hardware, runtime, and router layers of the serving stack (Figure 1). For a given workload, model, and hardware, the cumulative slowdown of agent session ii depends on

Si=f(c,b,ℓ,Λ),(5)S_i = f(\mathbf{c}, b, \boldsymbol{\ell}, \Lambda), \tag*{(5)}

where, for the nn outstanding requests, c=(c1,…,cn)\mathbf{c} = (c_1, \ldots, c_n) denotes their context lengths, b≤nb \le n the number running, ℓ=(ℓ1,…,ℓn)\boldsymbol{\ell} = (\ell_1, \ldots, \ell_n) their KV cache locations (GPU, CPU, or discarded), and Λ\Lambda the prefill–decode (PD) placement. The runtime selects which bb requests run, leaving the rest queued, and shapes ℓ\boldsymbol{\ell} via cache retention and eviction. The router shapes ℓ\boldsymbol{\ell} by steering an agent’s calls toward or away from the worker holding its context, and sets Λ\Lambda: whether and how prefill and decode are disaggregated. At the hardware layer, running requests contend for GPU compute, memory bandwidth, and capacity, so their number and context lengths set how much they slow one another. All four factors evolve throughout an agent’s execution.

Intuitively, the context lengths c\mathbf{c} determine the computation and memory bandwidth required for each inference iteration. Long-context requests increase KV cache accesses, slowing other requests in the same batch, while their larger memory footprint can also limit concurrency. The number of running requests bb determines the demand for GPU resources. Larger batches increase per-iteration computation and, for MoE models, expert-weight traffic, while requests that cannot be scheduled must wait, increasing queuing delay TqueueT_{\mathrm{queue}}. The KV cache locality ℓ\boldsymbol{\ell} determines whether previously computed context can be reused or must be reloaded or recomputed. Runtime cache management and router placement influence this locality, trading reduced computation against GPU memory pressure and potential starvation of waiting agents. Finally, the PD placement Λ\Lambda determines how prefill and decode workloads share GPU resources. Co-locating the two phases introduces interference that slows token generation, while disaggregation reduces this interference but may increase queuing at either phase. Next, we quantify these effects at each serving layer.

Hardware Contention and Unfairness

Finite GPU compute, HBM capacity, memory bandwidth, and interconnect bandwidth create contention among concurrently executing requests. Because co-batched requests advance in lockstep, long-context requests slow the entire batch by increasing compute and memory bandwidth demand for each prefill and decode phase. These delays disproportionately slow short-context requests, whose standalone execution time is smaller. Larger batches further increase resource demand; for MoE models, expert-weight traffic and inter-GPU communication. Consequently, identical requests can experience different slowdowns depending on the resource demands of their co-runners and the load on their assigned workers.

We examine batch size and context length effects on decode latency using equal-context requests within each batch. We focus on decode because agentic workloads exhibit substantial KV cache reuse across LLM calls ([37, 91]), making decode the dominant inference phase when contexts remain resident ([91]). We measure Qwen3-Coder-30B-A3B (Qwen), DeepSeek-V4-Flash (DeepSeek), and Llama-3.3-70B (Llama) on 8 AMD MI300X and 8 NVIDIA H200 GPUs with TP=8 and EP=8 for the MoE models (Figure 2).

Figure 2. Normalized decode latency on 8 MI300X and H200 GPUs (TP=8; EP=8 for MoE): (a) varying batch size at 983,040 total tokens; (b) varying context at batch size 32. Decode latency depends on batch size, context length, and model.

In Figure 2(a), we hold aggregate context fixed at 983,040 tokens while increasing batch size from 8 to 384, reducing the context per request. This isolates the effect of processing more concurrent requests without increasing aggregate KV cache demand. DeepSeek’s decode latency increases by 3.3×, versus 1.5× for Qwen and 1.7–2.3× for Llama. DeepSeek’s increase stems from MoE communication: on MI300X, its all-to-all time rises from 12.9 to 74.7 ms, versus 5.4 to 7.2 ms for Qwen. Conversely, Figure 2(b) fixes batch size at 32 while increasing per-request context from 3,840 to 122,880 tokens, isolating the effect of growing KV cache demand at fixed concurrency. Qwen and Llama latency rises by 2.7–3.8×, while DeepSeek increases by only 1.1× as its compressed attention ([13]) reduces KV cache accesses. These results show that the per-token cost varies with batch size, context length, and model architecture.

Next, we examine how mixing requests with different context lengths leads to agent-level unfairness under hardware contention. Figure 3 compares three workloads with heterogeneous prompt lengths. Each workload is evenly divided into two job groups, with prompt lengths of 96K/96K tokens for uniform, 64K/128K for intermediate, and 16K/176K for wide contexts. All three workloads have the same average prompt length of 96K tokens. We use Qwen with TP=2, FCFS scheduling, and LMCache ([38]), with each job generating 256 tokens per request and waiting 10 seconds between requests. Increasing concurrency from 14 to 18 raises working-set demand from 95% to 122% of GPU KV cache capacity.

Figure 3. Fairness versus KV cache pressure under FCFS with LMCache on H100 GPUs, for (a) uniform (96K/96K), (b) intermediate (64K/128K), and (c) wide (16K/176K) contexts. Percentages show the short-context group’s normalized slowdown relative to the long-context group’s.

Under uniform contexts, both groups experience similar normalized slowdown (UU). However, the short-context group’s UU exceeds the long-context group’s by 4–13% under intermediate and 19–24% under wide contexts, with the gap widening under memory pressure. This disparity arises from two mechanisms. First, without memory pressure, long-context requests increase shared decode iteration time, imposing a larger slowdown on short-context jobs with smaller TaloneT_{\mathrm{alone}}. Second, as memory demand exceeds capacity, runtime engine introduces queuing delays. Because UU normalizes delay by standalone execution time, the same waiting time disproportionately penalizes short-context jobs, amplifying unfairness.

Takeaway #1: Per-token decode cost varies with batch size and context length, with sensitivity differing across models. The mix of co-located requests can disproportionately amplify some requests’ latency, leading to unfairness.

Unfairness From Runtime Scheduling

Inference runtimes such as vLLM [32] and SGLang [92] schedule requests for execution and manage their KV caches in GPU memory. When the working set exceeds GPU HBM capacity, the KV caches of some requests must be evicted, requiring recomputation or reloading from secondary storage (e.g., CPU DRAM) when those requests resume [2, 38, 65]. For agentic workloads, retaining KV cache across tool calls avoids redundant prefill but reserves HBM for idle agents. Figure 4 illustrates this trade-off; assuming each agent’s KV cache occupies nearly all available GPU memory, retaining agent A’s cache can leave agent B waiting across multiple turns, whereas FCFS interleaves their requests at the cost of repeated cache reloads or recomputation. Empirically, KV cache retention reduces median normalized slowdown from 1.4× in FCFS to 1.0× but increases p99 slowdown from 6.6× in FCFS to 10.7×, demonstrating how improving common-case performance can worsen p99 fairness.

Agent timelines under FCFS and KV-cache retention, with normalized slowdown by percentile of agents

Figure 4. Left: Agent timelines under (a) FCFS and (b) KV-cache retention: retaining A’s cache avoids reloads but prolongs B’s waiting. Right: Normalized slowdown for 100 Azure agents at concurrency 18, using Qwen on an RTX PRO 6000.

We further examine two state-of-the-art cache-aware runtime policies: Continuum [33] and CacheWise [81]. Continuum retains KV cache across tool calls using time-to-live estimates, favoring reuse by returning agents but potentially delaying new admissions. CacheWise instead prioritizes requests with low additional KV cache demand and evicts blocks with the most distant future reuse; we evaluate its oracle variant. Both optimize cache efficiency, but neither explicitly accounts for cumulative agent-level slowdown. Figure 5 compares these policies with FCFS over 100-agent concurrency sweeps. As concurrency increases, Continuum reaches 1.5X and 3.5X FCFS’s throughput on Azure and TraceLab, respectively. However, its p99 UU rises from 2.7 to 22.9 on Azure and from 3.5 to 25.5 on TraceLab. Avoiding repeated prefill raises throughput, but retaining some agents’ KV cache can starve others as memory pressure grows. CacheWise achieves lower throughput than Continuum but lower p99 UU. Because UU normalizes delay by standalone time, the same delay inflates a short job’s slowdown more (§4.1); by serving inexpensive requests first, CacheWise avoids many such slowdowns. However, the benefit is partial, as per-request cost does not reflect agent-level cost. Thus, runtime efficiency does not necessarily translate into fair progress across agents.

Figure 5. Throughput and p99 normalized slowdown of FCFS, Continuum [33], and the CacheWise oracle [81] on Azure and TraceLab (one RTX PRO 6000). Concurrency is set per trace to yield comparable memory pressure. Continuum improves throughput but worsens fairness at high concurrency.

Takeaway #2: KV cache retention that improves throughput can also cause agent starvation, worsening p99 fairness.

Unfairness Due to Routing Decisions

The router determines request placement and admission, trading off KV cache locality against worker load. We evaluate three policies: NVIDIA Dynamo’s KV-aware policy [55], which prioritizes cache affinity; llm-d::VTC [40], which adds admission control based on VTC’s token-level fairness [71]; and NVIDIA Dynamo’s ThunderAgent [28], which uses program-level working-set limits to control admission. We replay TraceLab traces [96] on 4× NVIDIA RTX PRO 6000 Blackwell GPUs, each serving Qwen, sweeping concurrency from 32 to 48 agents in steps of 4 until 200 agents complete per configuration. Across this range, aggregate KV cache demand increases from approximately 68% to 96–102% of capacity.

Figure 6 decomposes p99 normalized slowdown across these routing policies. Dynamo::KV-aware preserves cache affinity but lacks admission control, allowing memory pressure and prefix recomputation to slow co-running requests. llm-d::VTC rotates admission to balance token-level service, but KV cache eviction and recomputation increase prefill interference at high concurrency. In contrast, Dynamo::ThunderAgent limits working sets, reducing execution interference but shifting unfairness into admission waiting. At high concurrency, its most delayed agents, particularly short-running ones, mostly wait for GPU admission rather than execution. Thus, admission policies can reduce worker interference while disproportionately delaying agents that have yet to be admitted.

Figure 6. Breakdown of p99 normalized slowdown across TraceLab routing policies into Prefill + other (prompt processing and residual time), Decode + prefill (decoding during concurrent prefill), Decode only, and Waiting (queuing). Admission control trades interference for router waiting time.

We next examine how PD disaggregation policies affect fairness. Figure 7 compares aggregated serving against strict and conditional PD disaggregation [55] on 8× AMD MI300X GPUs1, using Qwen with TP=1, EP=1, and FCFS scheduling. Strict disaggregation sends every prefill to prefill workers, while conditional disaggregation offloads a prefill only when its uncached prompt length exceeds a threshold and otherwise runs it locally on the decode worker. We replay Azure traces with 20 concurrent agents for 60 minutes, evaluating 5P/3D and 4P/4D allocations. With 5P/3D, strict and conditional disaggregation reduce p99 slowdown from 4.9× to 2.9× and 2.6×, respectively. However, the benefit depends on capacity allocation: under 4P/4D, conditional disaggregation achieves 2.8×, whereas strict disaggregation worsens slowdown to 5.9× due to prefill queuing. Even with effective disaggregation, residual interference persists among concurrent prefills or when conditional placement overlaps prefill and decode. Thus, eliminating cross-phase interference alone does not guarantee fairness; the placement policy must also balance phase-specific demand.

Figure 7. Agent slowdown under aggregated serving and strict or conditional PD disaggregation on eight MI300X.

Takeaway #3: Routing policies can shift unfairness from GPU contention to admission waiting, while PD disaggregation can introduce new queuing bottlenecks if capacity is poorly allocated.

Optimizing Fairness using Duru

We now present Duru, a fairness-aware serving system designed to address the sources of unfairness identified in § 4.

Optimization Objective

As demonstrated in Takeaway #2, throughput-oriented optimizations such as KV cache retention can improve efficiency while disproportionately slowing some agents. In this paper, for a given LLM and GPU architecture, we focus on optimizing how agents share serving resources through routing, admission control, and runtime scheduling. To balance fairness and efficiency, we formulate the optimization objective as minimizing p99 agent-level normalized slowdown subject to a throughput constraint:

min⁡π p99({Ur})s.t.X(π)≥(1−ϵ)Xbaseline,(6)\min_{\pi}\ p_{99}(\{U_r\}) \quad\text{s.t.}\quad X(\pi) \ge(1-\epsilon)X_{\mathrm{baseline}}, \tag*{(6)}

where π\pi is the serving policy, UrU_r is the normalized slowdown of the task rooted at agent rr, X(π)X(\pi) is the throughput, and XbaselineX_{\mathrm{baseline}} is the throughput of a performance-optimized baseline. The parameter ϵ\epsilon is the desired throughput tolerance, although Duru aims to minimize throughput degradation rather than guarantee a strict bound.

Design Overview

Figure 8 illustrates the design of Duru, which coordinates fairness tracking, scheduling, admission control, and autoscaling. As demonstrated in Takeaway #3, reducing resource contention at one serving layer can shift unfairness into queuing delays at another. This motivates Duru’s cross-layer design, which uses agent-level slowdown as a common control signal to coordinate router and runtime decisions rather than optimizing each layer independently. The design comprises four key components:

Figure 8. Overview of Duru. The fairness tracker computes each agent’s control score Uˉa\bar{U}_a (§ 5.3). The router and workers order waiting agents and requests by Uˉa\bar{U}_a (§ 5.4), and the router force admits starving agents (§ 5.5). The autoscaler adds GPUs when the p99 of Sˉa\bar{S}_a exceeds StargetS_{\mathrm{target}} (§ 5.6).

  • Online Fairness Tracker. Duru continuously estimates each agent’s normalized slowdown using observed waiting time, estimated TaloneT_{\mathrm{alone}}, and completed inference work (§ 5.3). The score Uˉa\bar{U}_a tracks each agent’s deficit from its fair share.

  • Slowdown-Aware Scheduling. The router and runtime prioritize agents with higher Uˉa\bar{U}_a, coordinating scheduling across both layers (§ 5.4). These decisions prioritize delayed agents while respecting KV cache capacity to preserve throughput.

  • Starvation-Aware Admission. Under memory pressure, strict admission limits can indefinitely delay agents despite slowdown-aware ordering. Duru introduces admission exceptions when an agent’s score exceeds a threshold, temporarily relaxing the router’s working-set budget to guarantee progress (§ 5.5).

  • SLO-Driven Autoscaling. Scheduling and admission redistribute resources to improve relative fairness but cannot eliminate delays caused by insufficient capacity. To meet SLO, Duru adds GPUs when p99 live slowdown exceeds StargetS_{\mathrm{target}} (§ 5.6).

Together, these components reduce disproportionate delays through coordinated scheduling and admission, while autoscaling addresses capacity shortfalls that scheduling alone cannot resolve.

Control Signal

Duru translates the agent-level normalized slowdown metric (§ 3) into an online control signal for fairness-aware scheduling. Its fairness tracker maintains a score Uˉa\bar{U}_a for each agent, which the router and workers use to prioritize execution. A higher score identifies an agent further behind its fair share, allowing Duru to prioritize disproportionately delayed agents. Unlike metrics computed after job completion, this signal is continuously updated using observed execution progress, without requiring knowledge of future requests.

For agents that have begun execution, Uˉa\bar{U}_a is their live normalized slowdown (Definition 3.4), computed from accumulated waiting and execution time relative to estimated standalone work (§ 3). Tracking this signal incurs little overhead: existing serving infrastructure already maintains request-level metadata, such as timestamps and token counts [32, 92], while Duru only requires a few additional per-agent counters for accumulated service entitlement and estimated completed work. In our measurements, this bookkeeping has a negligible 0.3–0.8% time overhead. These counters are updated incrementally, requiring constant-size metadata per agent rather than storing its execution history. For multi-agent jobs, the tracker aggregates the root agent and its sub-agents into a single score.

For newly arriving agents, live slowdown is initially undefined because no inference work has been completed. To initialize their scores, Duru uses lookahead UU, which includes the estimated standalone cost of the request’s next five engine iterations in the denominator. This estimate uses the same latency models (§ 3), enabling newly arriving agents to participate in slowdown-based scheduling.

Slowdown-Based Ordering

Duru prioritizes agents based on their accumulated unfairness, using the control score Uˉa\bar{U}_a to determine scheduling order. Both the router and runtime consider waiting candidates in decreasing Uˉa\bar{U}_a, breaking ties by arrival order. This ordering dynamically adapts to each agent’s execution progress: an agent experiencing waiting accumulates a higher score, increasing its priority over agents receiving more favorable service. This ensures that scheduling decisions account for accumulated agent-level delays rather than relying solely on immediate execution efficiency.

To preserve throughput, Duru combines slowdown-based ordering with capacity-aware scheduling at both layers. At the router, agents are admitted subject to a working-set budget and resume hysteresis, while workers schedule requests according to KV cache capacity and scheduling budgets. At either layer, candidates that cannot fit are skipped, allowing lower-priority candidates to backfill unused capacity. This avoids head-of-line blocking and maintains resource utilization without abandoning fairness priorities. Skipped candidates are reconsidered in subsequent scheduling rounds as their control scores evolve. However, when memory pressure persistently prevents admission, ordering alone cannot ensure timely progress, motivating the admission exceptions discussed next.

Admission Exceptions

Slowdown-based ordering prioritizes delayed agents, but it cannot guarantee their progress when the router’s working-set budget is exhausted. In particular, long-running agents with retained KV caches can occupy the memory, preventing other agents from being admitted even as their slowdown increases. To address this, Duru introduces an admission-exception threshold α\alpha, which determines when an agent’s accumulated unfairness warrants overriding admission limits. When a waiting agent cannot be admitted under the working-set budget and its control score satisfies Uˉa≥α\bar{U}_a \ge\alpha, Duru force-admits it to an eligible worker. This relaxes the working-set budget to allow delayed agents to make progress. As admitted

Algorithm 1 Duru: End-to-End Fairness-Aware Serving

Require:
  Q, A: worker and router waiting queues
  Ū_a, S̄_a: fairness and SLO signals
  B_i, W_j, w_a: KV budget, usage, and agent demand
  G: active GPU workers
  α, S_target: admission and SLO thresholds

1: function SCHEDULEWAITING(Q) ▹ runtime worker
2:   for q ∈ Q in decreasing Ū do
3:     if KV allocation for q succeeds then
4:       schedule q
5:     else
6:       defer q
7:     end if
8:   end for
9: end function
10: function ADMITCANDIDATES(A) ▹ router
11:   for a ∈ A in decreasing Ū do
12:     if ∃j : W_j + w_a ≤ B_j then
13:       admit a to j
14:     else if a is pending and Ū_a ≥ α then
15:       force-admit a
16:     else
17:       defer a
18:     end if
19:   end for
20:   for each worker j with W_j > B_j do
21:     pause smallest-w_a tool-phrase agents
22:     until W_j ≤ B_j
23:   end for
24: end function
25: function AUTOSCALE(G) ▹ autoscaler
26:   if p_99({S̄_a}) ≥ S_target then
27:     add one GPU worker
28:   else if ∄j ∈ G: idle ≥ 300 s then
29:     remove worker j
30:   end if
31: end function

agents execute and others are paused, the system returns toward its normal working-set budget, limiting sustained memory pressure. The threshold α\alpha provides a mechanism to balance tail fairness against the throughput benefits of KV cache reuse. A smaller α\alpha triggers earlier and more frequent interventions, reducing prolonged waiting but potentially increasing KV cache eviction, recomputation, and scheduling overhead. Conversely, a larger α\alpha favors cache retention and throughput by admitting fewer exceptions, at the risk of longer delays for some agents. Overly aggressive admission can worsen both throughput and fairness if the resulting cache churn outweighs the benefits of reducing waiting. Thus, α\alpha tunes fairness intervention against throughput degradation. A natural choice is α=1\alpha= 1, since U=1U = 1 corresponds to receiving the PS fair share §3.3.

SLO-Driven Autoscaling

Optimizing normalized slowdown improves fairness across agents but does not guarantee that agents meet their performance SLOs, particularly when system load exceeds available capacity. Traditional LLM serving SLOs, such as Time-To-First-Token (TTFT) and Time-Between-Tokens (TBT), primarily target human-facing interactions and do not adequately capture progress in autonomous agentic workloads. Instead, we adopt a slowdown-based SLO, following prior studies that express latency targets relative to isolated execution time [35, 75]. This formulation suits heterogeneous agentic tasks because it scales each task’s latency target by its standalone execution time. We therefore define SLO attainment at the agent-task level as completing inference within StargetS_{\mathrm{target}} times its standalone execution time, using Starget=5S_{\mathrm{target}} = 5 in our evaluation. Unlike the SLO, normalized slowdown UU measures whether an agent receives its fair share of available capacity under the prevailing system load. Thus, fairness guides resource sharing at a given capacity, whereas the SLO guides capacity scaling to meet a specified slowdown target.

To improve SLO attainment under varying workloads, Duru introduces SLO-driven autoscaling that adjusts GPU capacity based on observed agent performance rather than resource utilization alone. The autoscaler monitors active agents to identify potential SLO violations and adds a GPU worker when the slowdown is close to exceeding the SLO target. This targets capacity-induced delays that scheduling and admission control alone cannot eliminate. To avoid oscillations and unnecessary provisioning, Duru scales out incrementally, and scales in only after a worker has remained idle for five minutes with no requests waiting in the router. Algorithm 1 summarizes Duru’s end-to-end workflow, integrating slowdown-aware runtime scheduling, starvation-aware admission control, and SLO-driven autoscaling.

Evaluation Methodology

Our evaluation seeks to answer four key questions. (Q1) Does Duru improve p99 fairness without significantly sacrificing throughput? (Q2) How do router- and runtime-level optimizations contribute to end-to-end fairness? (Q3) How does Duru’s fairness–throughput trade-off scale with increasing GPU fleet size? (Q4) Can Duru meet agent-level SLOs through autoscaling under dynamic workloads?

To answer these questions, we evaluate Duru across diverse LLM architectures, real-world agentic workloads, and GPU platforms

from multiple vendors, as summarized in Table 2. We use three models spanning sizes and computational characteristics: Qwen3-Coder-30B-A3B (30B parameters, MoE), Llama-3.3-70B (70B parameters, dense), and DeepSeek-V4-Flash (284B parameters, MoE with compressed attention). This diversity allows us to evaluate fairness under different compute, memory, and KV cache demands. For workloads, we use three real-world production agentic execution traces: Azure GitHub Copilot [37], TraceLab [96], and AgentX [68]. These traces capture complex, multi-turn agentic workflows, including both single-agent and multi-agent execution, representative of contemporary AI serving workloads. Our main evaluation uses NVIDIA H100 and AMD MI300X GPUs.2 Finally, we use DynoSim [15] to evaluate scalability across clusters of 8 to 256 GPUs and study autoscaling under production arrival patterns.

NameDescription
ModelsQwen3-Coder-30B-A3B [66]<br>Llama-3.3-70B [44]<br>DeepSeek-V4-Flash [13]MoE, 30.5B total, 3.3B active<br>Dense, 70B parameters<br>MoE, 284B total, 13B active
WorkloadsAzure [37]<br>TraceLab [96]<br>AgentX [68]Copilot coding-agent sessions<br>Claude Code and Codex sessions<br>SemiAnalysis agentic workload traces
PlatformsNVIDIA H100 SXM<br>AMD MI300X<br>DynoSim [15]8 GPUs, 80 GB HBM3 each<br>8 GPUs, 192 GB HBM3 each<br>NVIDIA Dynamo simulator [55]
SoftwareLMCache 0.5.4 [38]<br>vLLM 0.28.0 [32]CPU DRAM offload, 128GiB per worker<br>LLM serving engine

Table 2. Evaluation setup.

Baselines. We compare Duru against the state-of-the-art scheduling and routing policies at both layers of the serving stack. At the runtime layer, we evaluate FCFS scheduling, Continuum [33], which retains KV cache across tool calls using time-to-live estimates, and CacheWise [81], which prioritizes requests with low KV cache demand and evicts blocks based on future reuse. We use the oracle variant of CacheWise. At the routing layer, we evaluate NVIDIA Dynamo’s KV-aware routing [55], which prioritizes KV cache locality; llm-d::VTC [40], which incorporates token-level fairness into admission control [71]; and ThunderAgent [28], which uses program-level working-set accounting to control admission. NVIDIA Dynamo and llm-d are production-grade serving frameworks deployed in data centers today. By evaluating their routing and admission policies alongside research baselines, we assess Duru against approaches representative of serving infrastructure.

Memory Pressure. To evaluate fairness under resource contention, we configure concurrency to target an aggregate working-set demand of approximately 1.3× the available GPU KV cache capacity. Due to variations in agent context lengths, actual demand ranges from 1.2× to 1.5× capacity across workloads. For each model–workload combination, we maintain identical concurrency across all policies to ensure a fair comparison. Host-memory KV cache offloading using LMCache [38] is enabled for all main comparisons.

Scalability and autoscaling. We use DynoSim [15] to simulate clusters of 8–256 NVIDIA H100 GPUs, each running one TP1 Qwen (FP8) worker. We replay Azure and AgentX traces with inter-request gaps capped at 30 s. For autoscaling, we replay 48 hours of Azure traffic with UTC arrivals and causal request dependencies. To minimize prefill–decode interference, we use disaggregated pools, with 32 GPUs per phase, scaling between 8 and 128 GPUs in total. All policies share the slowdown-driven autoscaler (§ 5.6) with Starget=5S_{\mathrm{target}} = 5 [75], adding one GPU with a 30 s cooldown and removing workers only after 300 s of idleness with no queued requests for that phase. We model worker startup delays of 480 s (cold) and 60 s (warm), with released nodes remaining warm for 600 s; workers serve requests after startup completes. Host-memory KV cache offloading is disabled in simulations due to DynoSim limitations.

Evaluation

Q1: Throughput–fairness trade-off. Figure 9 compares the fairness–throughput trade-off of Duru against different routing and runtime policies across three models, three workloads, and AMD MI300X and NVIDIA H100 GPUs, using α=1\alpha= 1. Because throughput and slowdown vary across models and workloads, we normalize both metrics against Dynamo::ThunderAgent with FCFS for each configuration. The X-axis shows the inverse of relative p99 normalized slowdown (Ubaseline/UpolicyU_{\mathrm{baseline}}/U_{\mathrm{policy}}); higher values indicate better fairness, while the Y-axis shows relative throughput, where higher is better. The dashed lines indicate baseline performance, and the shaded upper-right region represents improvements in both fairness and throughput. Duru achieves more favorable operating points than most competing configurations, pushing the fairness–throughput trade-off outward. Across the 15 model–workload–hardware settings evaluated on MI300X and H100, Duru reduces p99 normalized slowdown and the p99/p50 ratio by geometric means of 44.4% and 46.7%, respectively, while maintaining 91.8–127.4% of baseline throughput. In 14 of these settings, Duru reduces p99 normalized slowdown by 13.8–79.5% and the p99/p50 ratio by 19.2–79.1%. In contrast, other state-of-the-art policies achieve lower throughput, worse fairness, or both.

Figure 9. Throughput versus inverse p99 normalized slowdown, relative to Dynamo::ThunderAgent+FCFS. Key takeaway: Duru improves p99 fairness while largely preserving throughput across diverse models, workloads, and GPU architectures.

The trade-off between fairness and throughput varies across workloads and models. On MI300X, Duru improves fairness for Qwen on Azure and TraceLab with modest throughput reductions, while AgentX benefits in both dimensions. For Llama, fairness and throughput improve across all workloads on both platforms.

DeepSeek also achieves better fairness across all workloads on MI300X while largely preserving throughput, with particularly pronounced fairness gains on TraceLab and AgentX. On H100, Qwen achieves substantial fairness gains on TraceLab and AgentX with small throughput reductions, but exhibits worse fairness on Azure with essentially unchanged throughput.

Azure’s weaker fairness stems from long-prefill interference. Azure has 2.3× as many appended input tokens per output token as TraceLab. In the Qwen H100 run, tail requests queue behind long prefills and decode alongside them, increasing slowdown. Although Duru reduces queuing through slowdown-aware scheduling and admission, it cannot eliminate the prefill interference with concurrent decoding. Consequently, fairness improvements are strongest when waiting dominates the tail, whereas prefill interference can limit the benefit of scheduling alone. These results highlight that improving agent-level fairness requires addressing both unfair resource allocation and the sources of execution interference.

Sensitivity to parameter α\alpha. While Figure 9 uses α=1\alpha= 1, Figure 10 examines how different thresholds affect fairness. The primary Y-axis shows p99 normalized slowdown, while the secondary Y-axis shows throughput and KV cache hit rate to explain the underlying performance changes. At α=∞\alpha= \infty, admission exceptions are disabled, resulting in high tail slowdown despite high cache hit rates. Reducing α\alpha generally improves fairness by admitting delayed agents sooner, but increases competition for GPU memory. When α\alpha becomes too small, KV cache hit rates can drop sharply, increasing recomputation, dropping throughput, and potentially worsening fairness. Importantly, α=1\alpha= 1 achieves consistently strong fairness across the evaluated workloads, suggesting that a fixed threshold suffices without sophisticated dynamic tuning.

Sensitivity analysis of the admission-exception threshold

Figure 10. Sensitivity analysis: effect of admission-exception threshold α\alpha on fairness, throughput, and GPU KV hit rate.

Attributing Duru’s fairness improvements. To understand why Duru improves fairness, Figure 11 decomposes the p99 normalized queuing delay, p99({Tiqueue/Tialone})p_{99}(\{T_i^{\mathrm{queue}}/T_i^{\mathrm{alone}}\}), into router and runtime waiting on AMD MI300X GPUs, normalized to Dynamo::ThunderAgent with FCFS. Across workloads, the dominant source of waiting varies: admission limits delay at the router, while resource contention delays admitted requests at the runtime. Importantly, reductions in p99 queuing delay correspond to improved p99 fairness (Figure 9), but not proportionally, as the tail agents in these two distributions may differ. By prioritizing agents based on their accumulated slowdown across both layers and introducing admission exceptions, Duru mitigates disproportionate waiting rather than merely shifting delays between queues. This underscores the importance of cross-layer fairness mechanisms over single-layer optimizations.

Figure 11. p99 normalized queuing delay relative to ThunderAgent+FCFS (100%) for each model–workload pair.

Q2: Ablation study of cross-layer fairness. To quantify the contribution of each layer’s fairness mechanisms, Figure 12 compares four configurations on Qwen: admission control without slowdown ordering and admission exceptions (Adm. only), runtime-only fairness (Sched), router-only fairness (Router), and their combination (Duru). The bars show the inverse of p99 normalized slowdown, and markers indicate throughput relative to Adm. only. The results show that each mechanism’s effectiveness depends on workload characteristics. On Azure, both runtime and router optimizations contribute to fairness, with their combination achieving the best result. Azure’s longer profiles increase runtime contention, making slowdown-aware scheduling important. In contrast, TraceLab and AgentX contain longer-running sessions whose retained

Figure 12. Duru ablation study on Qwen across three traces: inverse p99 normalized slowdown and throughput change.

KV caches occupy the available working-set budget, concentrating waiting at the router. Runtime scheduling alone provides little benefit in these workloads. However, router-level admission exceptions introduce additional runtime contention, which slowdown-aware scheduling helps mitigate. These results demonstrate that coordinating router admission and runtime scheduling improves fairness by addressing their interacting effects on resource contention.

Q3: Scalability analysis. Figure 13 compares the inverse of p99 normalized slowdown (higher is better) against throughput as GPU fleets scale from 8 to 256 GPUs, with 4 or 8 root agents per GPU. Duru maintains throughput comparable to the strongest baseline, Dynamo::ThunderAgent, while its fairness advantage generally grows with fleet size. This improvement is particularly pronounced under higher concurrency, where Duru reduces p99 normalized slowdown by up to 87.2%. Interestingly, even with constant per-GPU concurrency, fairness can deteriorate as the fleet grows. This occurs because stateful agents tend to remain on workers holding their KV caches, creating persistent load imbalance across workers. Larger fleets increase the likelihood of overloaded workers and disproportionately delayed agents, amplifying tail slowdown. By coordinating fairness-aware admission and scheduling, Duru mitigates these delays while preserving KV cache reuse. This shows that scaling GPU capacity alone does not eliminate unfairness; Duru preserves its fairness–throughput benefits as the fleet grows.

Figure 13. Throughput–fairness trade-off of AgentX. Fleet scales from 8 to 256 GPUs, with 4 or 8 root agents per GPU.

Q4: Effectiveness of autoscaling in meeting SLO. Figure 14 evaluates llm-d, NVIDIA Dynamo, and Duru replaying 48 hours of Azure GitHub Copilot production traces [37]. The figure shows session arrival (top), SLO attainment (middle), and #GPUs (bottom). We use the slowdown-based SLO from §5.6 with Starget=5S_{\mathrm{target}} = 5.3 All policies use the same autoscaler, allowing us to evaluate fairness-aware scheduling and admission on SLO attainment.

Figure 14. Azure agentic trace replay showing Duru achieves improved SLO attainment at lower GPU fleet size.

Duru achieves the highest SLO attainment of 97.4%, compared with 93.4% for NVIDIA Dynamo and 70.2% for llm-d, with nearly identical throughput. Duru achieves this improvement using 76 GPUs on average, compared with 89 GPUs for NVIDIA Dynamo. This benefit arises because fairness-aware scheduling changes which agents are admitted and co-scheduled on GPUs, reducing disproportionate waiting without added capacity. When arrivals spike, startup delays temporarily constrain capacity, causing attainment drops; Duru prioritizes delayed agents and maintains near-constant SLO attainment. Conversely, GPU scale-in lags declining demand because stateful agents retain contexts on workers until they become idle. These results show that Duru effectively combines fairness optimization with autoscaling to achieve high SLO attainment and efficient GPU utilization under production workloads.

Related Work

Fairness in resource scheduling. Prior work on shared-hardware fairness uses application slowdown relative to isolated execution to characterize interference and guide resource allocation [16, 48, 49, 76]. Meanwhile, OS schedulers [79, 80] and network schedulers [14, 62, 72] approximate PS, while dominant resource fairness (DRF) [18] generalizes fair allocation to multiple resources. Building on these principles, Duru adopts slowdown relative to isolated execution and uses PS as its fairness reference (Definition 3.3), extending fairness measurement and optimization to agentic LLM serving.

Fairness in LLM serving. VTC [71] and DLPM [9] balance weighted token service; DLPM additionally exploits prefix locality. FairInference [6] provides per-token latency isolation; FairCache [5] accounts for the compute, memory, and I/O costs in context caching. Token-based metrics miss execution costs (§ 4.1); request-level metrics miss cumulative latency (Figure 11).

Agent-aware serving. Recent systems optimize KV cache retention [33, 81, 87], prefix sharing [7, 60, 61], and program-aware scheduling [28, 39, 43]. SMetric [84] considers session-level service balance, while SAGA [22] introduces workflow-level fairness using predicted remaining work and task deadlines. Unlike prediction-based scheduling, Duru tracks fairness through observed slowdown and requires no knowledge of future. It further coordinates admission and scheduling to address cumulative delays.

Conclusion

Agent-level fairness has received limited attention despite the adoption of shared LLM serving infrastructure. In this work, we defined agent-level fairness using normalized slowdown relative to standalone execution, characterized its sources, and designed Duru to optimize it. Our analysis revealed hardware contention, KV cache retention, and routing decisions caused disproportionate slowdowns, with queuing and starvation dominating tail performance. To address these challenges, Duru coordinated slowdown-aware routing and runtime scheduling, introduced admission exceptions to mitigate starvation, and incorporated autoscaling to improve SLO attainment under varying workloads. Duru improved p99 fairness by a geometric mean of 44.4% (up to 79.5%) while achieving 91.8–127.4% of baseline throughput. It also achieved 97.4% SLO attainment under production arrivals. These results demonstrate that coordinated, cross-layer fairness control substantially reduces agent-level unfairness while preserving serving efficiency.

Acknowledgement

Generative AI tools were used for manuscript editing, formatting, and code development. The authors take full responsibility for the ideas and the accuracy of the results. The authors acknowledge insightful discussions with Vikram Sharma Mailthody, Kriztian Flautner, Mosharaf Chowdhury, and Rakesh Kumar. This work was supported in part by the AMD AI and HPC Cluster Program, AMD’s Funding Academic Research program, funding from Gimlet Labs, and DARPA ML2P Award HR0011-26-9-E190. This work used the Delta advanced computing resource at the National Center for Supercomputing Applications (NCSA) through the National Science Foundation’s Advanced Cyber infrastructure Coordination Ecosystem: Services and Support (ACCESS) program under allocation CIS261278.

References

References

  1. [1]Alibaba Cloud. 2026. Batch inference. https://www.alibabacloud.com/help/en/model-studio/batch-inference. Accessed: October 17, 2026.
  2. [2]Eduardo Alvarez, Jamie Li, and Ronil Prasad. 2026. Five Ways NVIDIA Co-Design Maximizes the Value of Every Byte in the AI Factory. NVIDIA Technical Blog. https://developer.nvidia.com/blog/five-ways-nvidia-co-design-maximizes-the-value-of-every-byte-in-the-ai-factory/
  3. [3]Amazon Web Services. 2026. Service tiers for optimizing performance and cost. https://docs.aws.amazon.com/bedrock/latest/userguide/service-tiers-inference.html. Accessed: October 17, 2026.
  4. [4]Anthropic. 2025. Claude Code. GitHub repository. https://github.com/anthropics/claude-code Accessed: November 6, 2026.
  5. [5]Zhuoyan Bai, Bin Gao, Fei Xu, Hanyu Zhao, Zhi Zhou, and Weng-Fai Wong. 2026. FairCache: Demystifying Cache-Induced Unfairness in Multi-Tenant Large Language Model Serving. IEEE Transactions on Parallel and Distributed Systems 37, 11 (2026), 2390–2406. doi:10.1109/TPDS.2026.3729988
  6. [6]Dey Bali, Soujanya Ponnangali, Yichuan Wang, Natacha Crooks, Scott Shenker, and Matei Zaharia. 2026. Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving. arXiv:2609.18112 [cs.DC]. https://arxiv.org/abs/2609.18112
  7. [7]Zhuohang Bian, Feiyang Wu, Zhuoran Li, Teng Ma, and Youwei Zhuo. 2026. TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications. arXiv:2510.18586 [cs.DC]. https://arxiv.org/abs/2510.18586
  8. [8]Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yining Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Keunho Jang, and Zheng Hui. 2025. Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267). PMLR, 4874–4910. https://proceedings.mlr.press/v267/bonatti25a.htmlarxiv.org/abs/2409.08264
  9. [9]Shiyi Cao, Yichuan Wang, Ziming Mao, Pin-Lun Hsu, Liangsheng Yin, Tian Xia, Dacheng Li, Shu Liu, Yining Zhang, Yang Zhou, Ying Sheng, Joseph Gonzalez, and Ion Stoica. 2025. Locality-Aware Fair Scheduling in LLM Serving. arXiv:2501.14312 [cs.DC]. https://arxiv.org/abs/2501.14312
  10. [10]Chaokun Chang, Yukun Zhou, Kaihua Fu, Dakai An, Tianyu Feng, Hanfeng Lu, Sheng Yao, Pu Guo, Yinghao Yu, Yizhou Shan, Bo Li, Binhang Yuan, and Wei Wang. 2026. From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems. arXiv:2608.15127 [cs.OS] https://arxiv.org/abs/2608.15127
  11. [11]Jack Choquette. 2023. NVIDIA Hopper H100 GPU: Scaling Performance. IEEE Micro 43, 3 (2023), 9–17. doi:10.1109/MM.2023.3256796DOI
  12. [12]DeepSeek-AI. 2026. DeepSeek Harness: Everything is a Plugin. https://github.com/deepseek-ai/deepseek-harness.
  13. [13]DeepSeek-AI. 2026. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348 [cs.CL] https://arxiv.org/abs/2606.19348
  14. [14]Alan Demers, Srinivasan Keshav, and Scott Shenker. 1989. Analysis and Simulation of a Fair Queueing Algorithm. In Proceedings of the Symposium on Communications Architectures & Protocols (SIGCOMM ’89). Association for Computing Machinery, Austin, Texas, USA, 1–12. doi:10.1145/75246.75248DOI
  15. [15]Yongming Ding, Rudy Pei, Hongkuan Zhou, Ryan Olson, Alec Flowers, and Vikram Sharma Mailthody. 2026. DynoSim: Simulating the Pareto Frontier. NVIDIA Technical Blog. https://developer.nvidia.com/blog/dynosim-simulating-the-pareto-frontier Accessed: October 18, 2026.
  16. [16]Eiman Ebrahimi, Chang Joo Lee, Onur Mutlu, and Yale N. Patt. 2010. Fairness via source throttling: a configurable and high-performance fairness substrate for multi-core memory systems. SIGARCH Comput. Archit. News 38, 1 (March 2010), 335–346. doi:10.1145/1735970.1736058DOI
  17. [17]Bin Gao, Zhumin He, Pur Shu Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zhou. 2024. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention. In 2024 USENIX Annual Technical Conference (USENIX ATC ’24). USENIX Association, 111–126. https://www.usenix.org/conference/atc24/presentation/gao-bin-costarxiv.org/abs/2403.19708
  18. [18]Ali Ghodsi, Matei Zaharia, Benjamin Hindman, Andy Konwinski, Scott Shenker, and Ion Stoica. 2011. Dominant Resource Fairness: Fair Allocation of Multiple Resource Types. In 8th USENIX Symposium on Networked Systems Design and Implementation (NSDI). USENIX Association, 323–336. https://www.usenix.org/conference/nsdi11/dominant-resource-fairness-fair-allocation-multiple-resource-typesDOI
  19. [19]Google Cloud. 2026. Flex PayGo. https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/flex-paygo. Accessed: October 17, 2026.
  20. [20]Google Cloud. 2026. Priority PayGo. https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/priority-paygo. Accessed: October 17, 2026.
  21. [21]Google Cloud. 2026. Throughput quota. https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/resources/throughput-quota. Accessed: October 17, 2026.
  22. [22]Dongxian Guo, Jikun Wu, and Siu-Ming Yiu. 2026. SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters. In Proceedings of the 35th International Symposium on High-Performance Parallel and Distributed Computing (HPDC ’26). Association for Computing Machinery. doi:10.1145/3806548.3807598arxiv.org/abs/2605.00528
  23. [23]Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzong Lan, and Dong Yu. 2024. WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 6864–6890. doi:10.18653/v1/2024.acl-long.371DOI
  24. [24]Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2024. MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. In International Conference on Learning Representations, B. Kim, Y. Yue, S. Chaudhuri, K. Fragiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024. 23247–23275. https://proceedings.iclr.cc/paper_files/paper/2024/file/6507b115562bba035f1958ccc87355a-Paper-Conference.pdfarxiv.org/abs/2308.00352
  25. [25]Aaron Jarmusch and Sunita Chandrasekaran. 2026. Microbenchmarking NVIDIA’s Blackwell Architecture: An In-depth Architectural Analysis. In 2026 IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 1026–1036. doi:10.1109/IPDPS65963.2026.00087DOI
  26. [26]Norman P. Jouppi, George Kurian, Sheng Li, Peter C. Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Souvik Subramanian, Andy Swing, Brian Towles, Cliff Young, Xiang Zhou, Zongwei Zhou, and David A. Patterson. 2023. TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings. In Proceedings of the 50th Annual International Symposium on Computer Architecture (ISCA). ACM, 82:1–82:14. doi:10.1145/3579371.3589530
  27. [27]Norman P. Jouppi, Cliff Young, Nishant Patil, David A. Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre luc Cantin, Clifford Chao, Chris Clark, Jeremy Correll, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Daniel Killebrew, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Ormernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon. 2017. In-Datacenter Performance Analysis of a Tensor Processing Unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture (ISCA). ACM, 1–12. doi:10.1145/3079856.3080246
  28. [28]Hao Kang, Ziyang Li, Weili Xu, Xinyu Yang, Yinfang Chen, Junxiong Wang, Beidi Chen, Tushar Krishna, Chenfeng Xu, and Simran Arora. 2026. ThunderAgent: A Fast, Simple, and Program-Aware Agentic Inference System. In Proceedings of the 43rd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 306). PMLR, 55630–55657. https://proceedings.mlr.press/v306/kang26j.html
  29. [29]Yubin Kim, Ken Gu, Chanwoo Park, Chunjong Park, Samuel Schmidgall, A. Ali Heydari, Yao Yan, Zhihan Zhang, Yuchen Zhuang, Yun Liu, Mark Malhotra, Paul Pu Liang, Hae Won Park, Yuzhe Yang, Yuhai Xu, Yilun Du, Shwetak Patel, Tim Althoff, Daniel McDuff, and Xin Liu. 2026. Towards a Science of Scaling Agent Systems. arXiv:2512.08296 [cs.AI] https://arxiv.org/abs/2512.08296
  30. [30]Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. 2024. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 881–905. doi:10.18653/v1/2024.acl-long.50
  31. [31]Jayacharan Kollu, Pedram Alizadeh, and Gilbert Lee. 2025. Understanding RCCL Bandwidth and xGMI Performance on AMD Instinct MI300X. AMD ROCm Blog. https://rocm.blogs.amd.com/software-tools-optimization/mi300x-rccl-xgmi/README.html
  32. [32]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP).arxiv.org/abs/2309.06180
  33. [33]Hanchen Li, Runyuan He, Qiyang Mang, Qizheng Zhang, Huanhui Mao, Xiaokun Chen, Hangrui Zhou, Huanchen Zhang, Alvin Cheung, Joseph Gonzalez, and Ion Stoica. 2026. Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live. arXiv:2511.02230 [cs.AI] https://arxiv.org/abs/2511.02230
  34. [34]Xiaoli Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yongkang Wu, Ji-Rong Wen, Yutao Zhu, and Zhicheng Dou. 2025. WebThinker: Empowering Large Reasoning Models with Deep Research Capability. In Advances in Neural Information Processing Systems, Vol. 38. https://papers.neurips.cc/paper_files/paper/2025/hash/ae03bdef276132fae0899245725635-Abstract-Conference.html
  35. [35]Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yaping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’23). USENIX Association, Boston, MA, 663–679. https://www.usenix.org/conference/osdi23/presentation/li-zhouhanarxiv.org/abs/2302.11665
  36. [36]Sean Lie. 2024. Inside the Cerebras Wafer-Scale Cluster. IEEE Micro 44, 3 (2024), 49–57. doi:10.1109/MM.2024.3386628DOI
  37. [37]Banruo Liu, Haoran Qiu, Iñigo Goiri, Rodrigo Fonseca, Ricardo Bianchini, and Esha Choukse. 2026. Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale. arXiv:2608.00101 [cs.AI] https://arxiv.org/abs/2608.00101
  38. [38]Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Rui Zhang, Kuntai Du, and Junchen Jiang. 2025. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference. arXiv:2510.09665 [cs.LG] https://arxiv.org/abs/2510.09665
  39. [39]Yipeng Liu, Yingqiang Zhang, Feifei Li, and Huanchen Zhang. 2026. Ask the Tool, Don’t Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read It. arXiv:2609.18849 [cs.DC] https://arxiv.org/abs/2609.18849
  40. [40]llm-d Contributors. 2026. Weighted Fair Share Fairness Policy. https://github.com/llm-d/llm-d-router/blob/fe26034d7f05f7892cb4ce8853b9dbd5626cf8ab/pkg/epp/framework/plugins/flowcontrol/fairness/weightedfairshare/README.md Revision fe26034; accessed November 1, 2026.
  41. [41]llm-d Project. 2025. llm-d: A Kubernetes-Native Distributed LLM Inference Framework. https://github.com/llm-d/llm-d. Accessed: 2026-10-21.
  42. [42]Shih-Lien Lu, Ying-Chen Lin, and Chia-Lin Yang. 2015. Improving DRAM latency with dynamic asymmetric subarray. In 2015 48th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). 255–266. doi:10.1145/2830772.2830827
  43. [43]Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang, Yanping Huang, Zhifeng Chen, Joseph E. Gonzalez, and Ion Stoica. 2026. Agentx: An Efficient Serving Engine for LLM Agents as General Programs. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). USENIX Association, 2443–2459. https://www.usenix.org/conference/nsdi26/presentation/luoarxiv.org/abs/2502.13965
  44. [44]Meta. 2024. Llama-3.3-70B-Instruct. Hugging Face model card. https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct
  45. [45]Microsoft. 2026. Enable priority processing for Microsoft Foundry models. https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/priority- [46] Microsoft. 2026. Provisioned throughput for Foundry Models. https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/provisioned-throughput. Accessed: October 17, 2026.
  46. [47]Behnam Montazeri, Yilong Li, Mohammad Alizadeh, and John Ousterhout. 2018. Homa: a receiver-driven low-latency transport protocol using network priorities. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication (Budapest, Hungary) (SIGCOMM ’18). Association for Computing Machinery, New York, NY, USA, 221–235. doi:10.1145/3230543.3230564DOI
  47. [48]Onur Mutlu and Thomas Moscibroda. 2007. Stall-Time Fair Memory Access Scheduling for Chip Multiprocessors. In Proceedings of the 40th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO).
  48. [49]Onur Mutlu and Thomas Moscibroda. 2008. Parallelism-Aware Batch Scheduling: Enhancing both Performance and Fairness of Shared DRAM Systems. In 2008 International Symposium on Computer Architecture. 63–74. doi:10.1109/ISCA.2008.7DOI
  49. [50]Prashant J. Nair, Ramyad Hadidi, Subramani Ganesh, Sangamesh Kodge, Shubhankit Rathore, Neil Thanawala, Nikitha Reddy, Gyanesh Saharia, Vinayak Patankar, Arun Tiruvuri, Nithesh Kurella, and Sudeep Bhoja. 2026. Early Silicon of Raptor: The First 3D-DRAM Accelerator for Generative Inference. In 2026 ACM/IEEE 53rd Annual International Symposium on Computer Architecture (ISCA). IEEE, 2632–2647. doi:10.1109/ISCA66937.2026.00183DOI
  50. [51]Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Ananthanaraman, Dmitri Vainbrand, Pethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’21). doi:10.1145/3458817.3476209DOI
  51. [52]Nebius. 2026. Introducing Dedicated Endpoints and Custom Weights Hub in Nebius Token Factory. https://nebius.com/blog/posts/dedicated-endpoints-and-custom-weights-hub. Accessed: October 17, 2026.
  52. [53]William Nixon, Muxin Tian, Yunjia Zheng, Haryadi S. Gunawi, and Juncheng Yang. 2026. From Requests to Sessions: A Large-Scale Characterization of Human-Driven Agentic Workloads. doi:10.5281/zenodo.23903440 Preprint.DOI
  53. [54]NVIDIA. 2023. TensorRT-LLM: A TensorRT Toolbox for Optimized Large Language Model Inference. GitHub repository. https://github.com/NVIDIA/TensorRT-LLM. Accessed: November 6, 2026.
  54. [55]NVIDIA. 2025. NVIDIA Dynamo: A Datacenter Scale Distributed Inference Serving Framework. https://github.com/ai-dynamo/dynamo.
  55. [56]NVIDIA. 2026. NVIDIA NVLink and NVLink Switch. NVIDIA Data Center Technology. https://www.nvidia.com/en-us/data-center/nvlink/ Accessed: 2026-10-21.
  56. [57]NVIDIA. 2026. Paged Attention, IFB, and Request Scheduling. https://nvidia.github.io/TensorRT-LLM/features/paged-attention-ifb-scheduler.html Accessed: Oct. 16, 2026.
  57. [58]OpenAI. 2025. Codex: A Coding Agent for the Terminal. GitHub repository. https://github.com/openai/codex Accessed: November 6, 2026.
  58. [59]OpenAI. 2025. Introducing Deep Research. OpenAI Blog. https://openai.com/index/introducing-deep-research/
  59. [60]Zaifeng Pan, Ajikkumar Patel, Yipeng Shen, Zhengding Hu, Yue Guan, Wan-Lu Li, Jianhui Qin, Yida Wang, and Yufei Ding. 2025. FlexFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=5Iw1nDtYmT
  60. [61]Zaifeng Pan, Yipeng Shen, Zhengding Hu, Zhuang Wang, Aninda Manocha, Zheng Wang, Zhongkai Yu, Yue Guan, and Yufei Ding. 2026. ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Management. In Proceedings of the 43rd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 306). Tong Zhang, Miroslav Dudík, Martin Jagg i, Alekh Agarwal, Sharon Li, Dale Schuurmans, Jerry Zhu, Felix Berkenkamp, Hanze Dong, and Alberto Bietti (Eds.). PMLR, 95898–95913. https://proceedings.mlr.press/v306/pan26k.htmlarxiv.org/abs/2601.21473
  61. [62]A.K. Parekh and R.G. Gallager. 1993. A generalized processor sharing approach to flow control in integrated services networks: the single-node case. IEEE/ACM Transactions on Networking 1, 3 (1993), 344–357. doi:10.1109/90.234856
  62. [63]Pratyush Patel, Esha Choukse, Chaojie Zhang, Saashka Shah, Iñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). 118–132. doi:10.1109/ISCA59077.2024.00019DOI
  63. [64]Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ChatDev: Communicative Agents for Software Development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 15174–15186. doi:10.18653/v1/2024.acl-long.810
  64. [65]Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST ’25). USENIX Association, 155–170. https://www.usenix.org/conference/fast25/presentation/qin
  65. [66]Qwen Team. 2025. Qwen3 Technical Report. arXiv:2505.09388 [cs.CL] https://arxiv.org/abs/2505.09388
  66. [67]Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Jawan, Jeff Rasley, and Yuxiong He. 2022. DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale. In Proceedings of the 39th International Conference on Machine Learning (ICML) (Proceedings of Machine Learning Research, Vol. 162). 18332–18346. https://proceedings.mlr.press/v162/rajbhandari22a.html
  67. [68]SemiAnalysis. 2026. AgentX Harness: Long-Context, Multi-Turn Agentic Workload Trace Replayer. https://github.com/SemiAnalysis/agentx-harness.
  68. [69]SGLang Project. 2026. SGLang Model Gateway: Cache-Aware Routing and Load Balancing. https://github.com/sgl-project/sglang/tree/main/sgl-model-gateway. Source code. Accessed: 2026-10-21.
  69. [70]Yijia Shao, Yueheng Jiang, Theodore Kanell, Peter Xu, Omar Khatib, and Monica Lam. 2024. Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Association for Computational Linguistics, 6252–6278. doi:10.18653/v1/2024.naacl-long.347DOI
  70. [71]Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E. Gonzalez, and Ion Stoica. 2024. Fairness in Serving Large Language Models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 965–988. https://www.usenix.org/conference/osdi24/presentation/sheng
  71. [72]M. Shreedhar and George Varghese. 1995. Efficient fair queueing using deficit round robin. In Proceedings of the Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication (Cambridge, Massachusetts, USA) (SIGCOMM ’95). Association for Computing Machinery, New York, NY, USA, 231–242. doi:10.1145/217382.217453DOI
  72. [73]Satyam Srivastava, Akhil Arunkumar, Nithesh Kurella, Amrit Panda, Gaurav Jain, Purushotham Kamath, Mark Wutzke, Arun Tiruvur, Mayank Mike Gupta, Ilya Soloveychik, Vamsi Darsi, Malav Dalal, Vinayak Patankar, Sasidhar Udyala, Senthil Duraisamy, Santosh Ramachandran, Raghav Venkatasubramanian, Yuwei Qin, Xin Wang, Jayaprakash Balachandran, Ali Murat Gok, Piotr Wojciechowski, Saliya Ekanayake, Chris Ng, Ranjju Sarma, Shubhankit Rathore, Tristan Trouwenn, Siwei Zhuang, Chris J. Nicol, and Sudeep Bhoja. 2025. Corsair: An In-Memory Computing Chiplet Architecture for Inference-Time Compute Acceleration. IEEE Micro 45, 5 (2025), 30–42. doi:10.1109/MM.2025.3593444DOI
  73. [74]Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, and Yiying Zhang. 2025. Preble: Efficient Distributed Prompt Scheduling for LLM Serving. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025. 37057–37082. https://proceedings.iclr.cc/paper_files/paper/2025/file/5bc342f48de8264779952fac378f96dc-Paper-Conference.pdfarxiv.org/abs/2407.00023
  74. [75]Jovan Stojkovic, Chaojie Zhang, Iñigo Goiri, Josep Torrallas, and Esha Choukse. 2025. DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency. In HPCA. https://www.microsoft.com/en-us/research/publication/dynamollm-designing-llm-inference-clusters-for-performance-and-energy-efficiency/DOI
  75. [76]Lavanya Subramanian, Vivek Seshadri, Arnab Ghosh, Samira Khan, and Onur Mutlu. 2015. The Application Slowdown Model: Quantifying and Controlling the Impact of Inter-Application Interference at Shared Caches and Main Memory. In Proceedings of the 48th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). ACM. doi:10.1145/2830772.2830803DOI
  76. [77]B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y. Li, and W. Lin. 2024. Jumnix: Dynamic Scheduling for Large Language Model Serving. In Proc. 18th USENIX Symp. Operating Systems Design and Implementation (OSDI). https://www.usenix.org/conference/osdi24/presentation/sun-biaoarxiv.org/abs/2406.03243
  77. [78]SWE-agent Team. 2025. mini-SWE-agent: The Minimal AI Software Engineering Agent. GitHub repository. https://github.com/SWE-agent/mini-swe-agent Accessed: November 6, 2026.
  78. [79]The Linux Kernel Community. 2026. CFS Scheduler. The Linux Kernel documentation. https://www.kernel.org/doc/html/latest/scheduler/sched-design-CFS.html Accessed: 2026-10-19.
  79. [80]The Linux Kernel Community. 2026. EEVDF Scheduler. The Linux Kernel documentation. https://www.kernel.org/doc/html/latest/scheduler/sched-eevdf.html Accessed: 2026-10-19.
  80. [81]Shubham Tiwari, Tapan Chugh, Nash Rickert, Simon Peter, Ratul Mahajan, and Haiying Shen. 2026. CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficient Serving LLM Coding Agents. arXiv:2606.16824 [cs.DC] https://arxiv.org/abs/2606.16824
  81. [82]Together AI. 2025. On-demand dedicated endpoints: run inference with unmatched price-performance & control at scale. https://www.together.ai/blog/on-demand-dedicated-endpoints. Accessed: October 17, 2026.
  82. [83]Together AI. 2026. Batch processing: Overview. https://docs.together.ai/docs/inference/batch/overview. Accessed: October 17, 2026.
  83. [84]Jiahao Wang, Kaizhan Lin, Kaixi Zhang, Jimbo Han, Xingda Wei, Sijie Shen, Chenguang Fang, Wenyuan Yu, Rong Chen, and Haibo Chen. 2026. SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling. arXiv:2607.08565 [cs.DC] https://arxiv.org/abs/2607.08565
  84. [85]Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. 2025. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. In International Conference on Learning Representations. https://manthology.org/iclr/2025/wang2025iclr-openhands/arxiv.org/abs/2407.16741
  85. [86]Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. 2024. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversations. In First Conference on Language Modeling. https://openreview.net/forum?id=BAakYHnNKSarxiv.org/abs/2308.08155
  86. [87]Tian Xia, Hanchen Li, Zifei Li, Xiaokun Chen, Hao Kang, Yifan Qiao, Yi Xu, and Ion Stoica. 2026. Idleness is Relative: Exploiting Tool-Call Idle Windows for Offloading in Agentic Systems with MORI. arXiv:2606.00866 [cs.OS] https://arxiv.org/abs/2606.00866
  87. [88]Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. 2024. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. In Advances in Neural Information Processing Systems, Vol. 37. https://papers.nips.cc/paper_files/paper/2024/hash/5d413e48f84f46dc61244b6be550f1cd8f5-Abstract-Datasets_and_Benchmarks_Track.htmlarxiv.org/abs/2404.07972
  88. [89]John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-Agent: Agent-Computer Interfaces Enable Automated Software Engineering. In Neural Information Processing Systems, doi:10.5220/09017-1601arxiv.org/abs/2405.15793
  89. [90]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2210.03629
  90. [91]Yichao Yuan, Ankita Nayak, Souvik Kundu, and Nishil Talati. 2026. Agentic AI Workload Characteristics. In 2026 IEEE International Symposium on Workload Characterization (IISWC). IEEE. https://arxiv.org/pdf/2605.26297
  91. [92]Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christus Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. In Advances in Neural Information Processing Systems (NeurIPS). http://papers.nips.cc/paper_files/paper/2024/hash/724be4472168f31ba1c9ac630f15dec8-Abstract-Conference.htmlarxiv.org/abs/2312.07104
  92. [93]Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lymunshan Ye, Pengrui Lu, and Pengfei Liu. 2025. DeepResearch: Scaling Deep Research via Reinforcement Learning in Real-world Environments. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 414–431. doi:10.18653/v1/2025.emnlp-main.22
  93. [94]Yihao Zheng, Jingzhe Jiang, Dejiang Zhu, Zhiyuan Tan, Yang Tian, Tao Wang, and Minchen Yu. 2026. Semantics, Workflows, and Infrastructure: Understanding Agent Serving at Production Scale. arXiv:2609.34432 https://arxiv.org/abs/2609.34432
  94. [95]Yimin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuan zhe Liu, Xin Jin, and Hao Zhang. 2024. DisServ: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’24). USENIX Association, 193–210. https://www.usenix.org/conference/osdi24/presentation/zhong-yimin
  95. [96]Kan Zhu, Mathew Jacob, Chenxi Ma, Yi Pan, Stephanie Wang, Arvind Krishnamurthy, and Baris Jakicic. 2026. TraceLab: Characterizing Coding Agent Workloads for LLM Serving. arXiv:2606.30560 [cs.LG] https://arxiv.org/abs/2606.30560

Paper details

vlnc:W_3pnehene
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 5

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Generative AI tools

View statement
Looking for related papers…
Contents