WorksComputer Science
LADS: Lease-Aware Delivery Scheduling of Tool Results for Language Model Agents
Abstract
Web agents complete multi-step online tasks through repeated browser actions and model calls. Their returned observations lie on the critical path to subsequent decisions. When concurrent browser actions finish in bursts, observations can accumulate on a constrained return path to a shared inference service, while waiting histories compete for reusable cache state. Conventional delivery rules prioritize short transfers or immediate cache reuse but overlook how sending a result earlier can help a request resume sooner and free cache-protection capacity for other waiting requests. We present LADS (Lease-Aware Delivery Scheduling), a tool-result scheduler that couples cache-aware delivery with bounded prefix leases. Its key insight is that resumption lets extra protection references retire, making lease quota available to protect another waiting history. LADS prioritizes timely handoffs using current lease ownership and an estimated resumption delay, with bounded departures from Cache-first order. Compared with ReAct using first-in, first-out (FIFO) tool-result delivery, LADS reduces mean complete-workflow latency by 44.6% under bursty tool returns and by 42.4% on Web-derived burst replay.
CCS Concepts
• Information systems → Web applications; • Software and its engineering → Scheduling.
Introduction
Web agents advance through observations: a returned page, retrieved passage, or tool response supplies the context for the next model decision. ReAct makes this reasoning–action–observation loop explicit [41]. When concurrent agents share an inference service, completed observations also share a return path. Waiting on that path delays the next model call and every dependent action. We study how to order these completed results to reduce complete-workflow latency.
The difficulty is that delivery order affects both communication waiting and the computation needed to resume an agent. Concurrent tool completions can create a backlog on a constrained return path, while their model histories compete for reusable key–value (KV) cache state. First-in, first-out (FIFO) follows readiness order; Size-first advances short transfers; Cache-first prioritizes results whose associated history prefixes are currently resident in the KV cache. A history observed in cache must still survive committed traffic and engine waiting before reuse. The useful scheduling signal therefore extends beyond result size or an immediate cache hit.
Existing systems offer controls at several points in this process. PagedAttention supplies block-based KV management [14]; Infer- Cept and Continuum manage state across interruptions and multi- turn execution [1, 16]. Application-aware serving uses dependen- cies and program progress to schedule model calls [18, 23]. Tail- aware Release separates readiness from commitment and combines priority with feedback-driven control of committed model work [9]. Before completed tool results are committed for transmission, a further question arises: how can delivery order preserve reusable histories and make bounded prefix-cache protection available to other waiting agents?
Our key insight is that engine-native resumption—handled by vLLM’s own scheduler and block manager—can return protection quota before inference finishes. A bounded prefix lease adds references to keep a resident history available while its observation waits. Once the inference engine acquires that history and allocates new tokens, the extra references can retire. Their logical quota can protect another waiting history while the resumed request continues using its physical cache blocks. Delivery can thus influence who receives protection next. Exploiting this opportunity requires identifying owners with exclusive quota and estimating whether their results can reach allocation before the leases expire.
We present LADS (Lease-Aware Delivery Scheduling), which connects result ordering to bounded prefix protection. Its cache-aware base order favors currently reusable histories. Leases are granted in FIFO order with a fixed maximum lifetime.
The protection budget is based on expected near-term prefix reuse and capped to reserve cache space for model execution. A turnover priority advances a protected result when an estimated timely handoff can return exclusive quota for another waiting history. Each actual promotion is followed by a base-order choice, bounding consecutive interventions. Fully received observations continue through ordinary FIFO model entry; the engine verifies native ownership and positive-token allocation before retiring the extra references.
We evaluate the complete method and its components through controlled bursts and Web-derived burst replay, and additionally validate delivery on captured Web-agent task paths. The two main workloads compare LADS with FIFO, Size-first, and a delivery-side adaptation of Tail-aware Release that works with accounting and feedback. Web-derived replay supplies Web observations within the controlled workload construction. Across five input seeds in the controlled cohort-8 realization, LADS reduces mean complete-workflow latency by 44.6% relative to FIFO. On Web-derived burst replay, the corresponding reduction is 42.4% across three input seeds.
Our main contributions are summarized as follows:
We introduce LADS, a tool-result delivery scheduler that reduces complete-workflow latency for concurrent Web agents by coordinating result ordering with prefix-cache protection.
We identify resumption as an opportunity to return cache-protection quota before inference completes. Bounded prefix leases and an allocation-time ownership check make this quota available to other waiting histories while active requests continue using their cached prefixes.
Across five input seeds in the controlled cohort-8 realization and three input seeds in Web-derived burst replay, LADS reduces mean complete-workflow latency by 44.6% and 42.4%, respectively, relative to first-in, first-out delivery.
Background and Motivation
The tool-result return path
Concurrent Web agents return completed observations through a shared path to a shared inference engine. The sender chooses among ready, uncommitted result frames; committed frames retain their transport order. Complete receipt immediately registers the continuation in FIFO order, leaving batching and execution to the engine. Delivery order affects both resumption delay and whether cached histories survive until reuse.
Each completed tool result is encapsulated in an application-layer result frame containing the result payload and its metadata. Let , , , , , and denote result readiness, full-frame commitment, complete receipt, request registration, first successful positive-token allocation, and inference completion for continuation . Sender waiting spans , while includes receipt processing and engine waiting. The intervals jointly determine how long a cached history must survive before reuse. Complete-workflow latency is measured from workflow arrival to termination and includes all model calls and tool waits. Figure 2 locates the delivery decision and the prefix-survival interval.
Figure 2. Schematic workflow, return path, and prefix protection. (a) Model calls alternate with tool use until termination. (b) LADS selects a ready frame before commitment at ; committed queuing and transfer end at . (c) Lease references run from to verified handoff at or an earlier lease-termination event. The dashed segment is unused lifetime before . Grant and deadline need not coincide with readiness and model-call completion.
Prefix leases and the turnover opportunity
Block-based KV management and multi-turn context caching permit the service to reuse resident history instead of recomputing it [10, 14]. A prefix lease temporarily adds references to already-resident blocks while their continuation waits. It preserves available state rather than reconstructing evicted state. Motivating example. Suppose ready results A and B hold the available leases, while C’s resident history awaits protection. Cache-first selects A’s smaller frame. If B owns enough exclusive quota to protect C, returning B first can make that quota available earlier than sending B after A, provided B reaches allocation before its lease expires. C can then obtain a lease while B continues inference on the same physical blocks.
For lease , let be the set of KV blocks it protects at time , and its expiry time. Let the protection occupancy count the distinct blocks protected across all leases. The admission target governs new lease grants, while the protection commitment accounts for both this target and existing protection. Let be the usable block-pool capacity and the headroom reserved for model allocation. All budgets and block counts are measured in KV blocks:
Shared blocks count once toward . Lowering does not itself revoke existing leases; when , accounts for the protection still outstanding. Let count the blocks protected by lease and by no other lease:
Here equals 1 when its condition holds and 0 otherwise. The sum over counts the leases protecting block ; requiring it to equal 1 selects blocks protected only by lease .
When the engine acquires its own prefix references and successfully allocates positive-token work, the extra lease references can retire. This handoff can return units of logical protection quota, allowing another resident prefix to obtain a lease. The model may still use the same physical blocks. Quota turnover is therefore distinct from freeing GPU memory or completing inference.
LADS Design
Problem definition
We consider concurrent Web agents that share an inference service and a return path for completed tool observations. Bursty returns can create a transmission backlog while waiting histories compete for cache capacity.
At time , let contain ready, uncommitted result frames permitted by the transport window. Given payload sizes, cache hints, and published lease state, the sender selects the next frame for commitment to reduce complete-workflow latency under bounded prefix protection.
Design overview
LADS combines a demand-based admission target, FIFO prefix leases, and bounded departures from Cache-first order (Figure 1). The engine publishes lease ownership, deadlines, and exclusive quota. Under protection pressure, the sender advances an eligible owner whose estimated handoff precedes expiry; verified handoff makes that quota available to FIFO replenishment.
Figure 1. LADS scheduling and quota turnover. A, B, and C label the tool results and associated model histories of three concurrent Web agents. (a) Cache-first chooses among results permitted by the transport window. Under pressure, eligible owners are ranked by , where is exclusive protection quota and is estimated handoff time. Each override is followed by a base choice; committed frames retain their order. FIFO leases use a demand-based budget and fixed lifetime limit. (b) Two cache states show the same cached blocks. After B’s native request owns every leased block and successfully allocates at least one new token, its extra lease references retire. B keeps using its blocks. Returned exclusive quota can fund a FIFO lease on C’s own resident prefix if grant conditions hold.
Cache-first default ordering
For each , the sender knows payload size , a version-bound cached-prefix hint , and a stable identity . Its default selection minimizes
lexicographically, treating invalid hints as zero. The key prioritizes valid nonzero cache hits, then smaller payloads and stable identity. This is the default whenever no usable turnover opportunity exists.
Budget estimation
For each current, noncold pending result , let be its resident reusable-prefix block set. The budget identifies prefixes expected to be reused within the remaining lease horizon and aggregates their distinct blocks. Its admission target adapts to this demand while reserving headroom for model allocation.
Let be the cumulative lease lifetime limit and the time of result ’s first successful lease grant. The remaining horizon is before that grant and thereafter.
The budget estimate for a current FIFO prefix-aware inventory through result . Let denote the frames up to and including , and let count its preceding frames. Let be the known committed-frame backlog in bytes, the framing bytes per result, its payload bytes, and the current link rate in bytes/s. Let and count engine waiting requests (including preempted requests) and running requests, be the engine’s concurrency limit, and the frozen wall-time proxy per 512-token chunk. The estimated time to first successful allocation is
The first term accounts for committed transport work and the FIFO frame prefix through ; the second is a coarse engine-queue delay proxy. This estimate ends at the first successful allocation, before the request’s own missing-input computation.
The near-use set contains current, noncold pending results satisfying , , and . Its distinct-block demand is . With currently free blocks, the safe ceiling and admission target are
The fit condition requires some with . Protection starts at zero. The lease provider evaluates the target every 100 ms and synchronously before a real first offer; periodic evaluation and replenishment share that interval. The near-use set determines aggregate demand; the resulting target is applied to FIFO lease grants. Figure 3 summarizes how the target and returned quota support FIFO whole-prefix admission.
Figure 3. Demand-based protection and FIFO admission. Near-use demand and the safe ceiling set the aggregate target , with a whole-prefix fit check. FIFO order selects recipients; each grant protects a complete resident prefix subject to target and headroom checks. Verified handoff retains the cached blocks under native references and returns exclusive logical quota; expiry also ends lease protection. The return loop supports replenishment. A lower target preserves existing leases, with accounting for outstanding protection.
FIFO prefix leases
Leases apply the admission target to preserve resident prefixes between tool readiness and reuse. At actual tool-result readiness, the engine-side lease provider updates the target and attempts a lease before exposing the sender descriptor. It considers candidates in ready-time, identity, and version order, keeping protection allocation independent of the sender’s priority score. A grant covers the entire reusable prefix returned by the resident-block lookup, with no partial subset chosen for packing.
A grant succeeds only if its additional distinct blocks satisfy and leave at least free blocks after acquiring the references. Empty resident prefixes receive no lease. The deadline is . Repeated observations leave an existing lease unchanged. A later regrant cannot reset or extend the lifetime beyond .
The lease provider retries unprotected, still-pending identities from a causal sender inventory in the same FIFO order. The inventory establishes identity and phase; its proposed ranking and time estimates do not select recipients. Replenishment lets returned quota cover another resident history. Inventory delivery can race with commitment, so replenishment records distinguish grants before and after actual commitment. A lower target preserves existing references and accounts for them through until handoff, expiry, cancellation, or identity invalidation releases them. Target changes neither renew leases nor directly revoke them.
Before allocation, the engine-side lease provider may revoke lease references to preserve headroom. Each revocation removes only that lease’s extra references, preserving native references and references held by other leases.
Quota turnover priority
The lease provider publishes lease identities, phases, deadlines, exclusive counts, occupancy, and a delay proxy combining two observed queues. Using the same engine waiting count , let be the zero-based position in the local FIFO queue awaiting engine submission, ordered by arrival and stable identity; for an upstream request absent from that queue, is its current length. This local queue drain each engine iteration through immediate add request, without an additional admission limit. Using the same frozen time proxy , the published delay in seconds is
Prior development cold-prefill steps provide and 47.155 ms on GPU0 and GPU1, respectively, by normalizing elapsed host step time per 512 scheduled tokens. These card-specific values remain fixed across both Web-agent realizations. The proxy uses queue state and batching-inclusive host time without fitting test latency.
At decision time , the sender adds serialization for a feasible frame:
Here , , and are committed full-frame bytes, framing bytes, and payload bytes; is bytes/s and seconds guards division. These byte counts use the transmitted representation after any common lossless encoding. Thus is seconds and is blocks/s. Counting a partially transmitted frame at full size conservatively overestimates its remaining serialization. Engine scheduling and future arrivals make total handoff delay an estimate.
The override activates at and , where denotes 75.0% committed protection occupancy, or when fresh evidence shows that a feasible pending result was denied protection because . The latter captures whole-prefix packing pressure below the threshold. Cold prefixes and headroom rejections do not trigger it.
Unlike the FIFO inventory estimate , estimates handoff delay if result is sent next. Among valid, identity-matching upstream lease owners with , , and , LADS selects the highest , breaking ties with . The score heuristically prioritizes quota turnover; workflow benefit depends on other requests’ waiting and cache reuse (Figure 4).
Figure 4. Example of earlier delivery creating an earlier protection opportunity. Cache-first sends A’s shorter frame; one eligible override sends B first, followed by A. B’s verified handoff returns enough exclusive quota to protect C, whereas A’s alone cannot. B retains its physical blocks under native references. B’s handoff precedes expiry in both orders; C remains resident, is next in FIFO order, and passes target, whole-prefix, and headroom checks.
Bounded intervention and handoff
A promotion is a committed departure from Cache-first. The override balances quota turnover with the immediate reuse and short transfers favored by the base order. At most one actual departure from Cache-first is permitted consecutively. The following commitment uses Cache-first, after which the override becomes eligible again. A preview that is not committed changes neither this counter nor byte accounting.
Complete receipt binds the observation to its workflow, turn, prefix digest, and payload digest, then immediately registers its request. Receipt and registration do not release protection. The handoff callback runs only after allocation succeeds with positive new-token work and verifies that the native request owns every protected block. It then removes exactly the lease’s extra references. Replenishment can use the resulting logical quota while the resumed model call continues running.
Algorithm 1 LADS at a full-frame commitment opportunity
Require: Feasible set , ownership record Prior-promotion flag
1: if F = ∅ then
2: return without commitment
3: end if
4: b ← arg min_{j∈F} k_j; j ← b
5: if z = 0 and record is valid and protection is pressured then
6: E ← {i ∈ F : valid upstream lease, E_i > 0,
0 < ê_i < ∞, t + T̂_i ≤ d_i}
7: if E ≠ ∅ then
8: j ← arg max_{i∈E} ρ_i, ties by k_i
9: end if
10: end if
11: Commit the complete frame of j in transport order
12: z ← 1[j ≠ b]
Implementation
We implement LADS in Python as a tool-result gateway coupled to a vLLM inference service. In the default evaluations, initial model inputs enter the engine directly, while subsequent tool observations become eligible for delivery when tool execution completes. The gateway chooses among ready observations and transmits complete result frames over a userspace-paced localhost TCP connection. After complete receipt, the continuation enters the engine’s FIFO registration path.
Two workload adapters realize this shared Web-agent execution model. The controlled adapter uses archived tool text and resized histories to construct repeatable multi-turn workloads with configurable return cohorts and context demand; Web-derived burst replay feeds prepared Web observations into this adapter. The Web-agent adapter uses a custom ReAct action loop over a local, read-only product and review catalog. Capture records model-selected task paths; timing replay preserves their model inputs, observations, dependencies, and tool-service durations across delivery policies.
The sender maintains the uncommitted inventory and fixed committed-frame order. vLLM tracks per-block reference counts; allocation acquires references to reusable prefix blocks and keeps them out of the free queue [34]. LADS performs cache lookup, extra-reference acquisition, expiry, revocation, and handoff on the engine thread, aligning lease changes with allocation. These operations use native block references rather than extra model calls, cache reconstruction, or state transfer.
The engine publishes timestamped, version-bound lease records; the sender uses the latest valid record for each budget generation and falls back to Cache-first on invalid evidence. Appendix § 12 specifies freshness, identity, and budget-denial checks.
A decision scans candidates and lease records; lease acquisition and release update the extra reference for each protected block.
Evaluation
We compare complete-workflow latency in controlled bursts, Web-derived burst replay, and captured Web-agent paths, then examine matched component controls, stage timing, and control costs.
Experimental setup
Common platform. All workloads use Qwen2.5-7B-Instruct [38] through vLLM on a host with two NVIDIA RTX PRO 5000 Blackwell GPUs, with at most eight concurrent sequences and a 512-token per-step batching budget. The sender uses a four-frame in-flight window and 1028 application-framing bytes per payload: a 1024-byte fixed metadata header plus a 4-byte length prefix. Initial model inputs enter the engine directly; subsequent completed tool results enter delivery scheduling. Initial inference and all tool waits remain in complete-workflow time. Within each timing comparison, methods use the same model, prefix-cache and batching settings, input realization, and physical card. Appendix § 12 lists exact software versions and hardware details.
Controlled Web-agent realization. This realization models concurrent Web-agent workflows with configurable histories and tool-return patterns. The default setting contains workflows with six model calls each: 16 cold calls and 80 continuations. Inert archived tool text and resized histories create controlled context competition. Histories have 8,192, 12,288, 20,480, or 22,528 tokens; the context limit is 32,768 tokens, the output cap is 64 tokens, and the cache provides 197,040 usable tokens. Tool service is grouped into two cohorts of workflows. Same-round calls in a cohort start before its common 50-millisecond service completes, with a 60-second limit permitting partial release. Completed results contend for a 32 kbit/s userspace-paced localhost return link. The default comparison uses five input seeds.
Web-derived burst replay. Both Web replay workloads use a local, read-only catalog of product and review data from the WebArena shopping database [48], a 32,768-token context limit, and 12,316 configured KV blocks. For this workload, observations from 16 shopping tasks populate the controlled cohort-16 workload shape. Each run retains the controlled realization’s 16-workflow, six-call structure (16 initial calls and 80 tool-result continuations), four history lengths, 64-token output cap, and 32 kbit/s return link. Three input seeds vary workflow arrivals and history-length assignments while preserving each seed’s inputs across methods.
Captured Web-agent workflow replay. This validation records model-selected paths for 16 shopping tasks from the same catalog. Each task asks the agent to inspect three products and their review pages using open and read_reviews over the local catalog. Histories accumulate from actual observations and model outputs. Replay preserves each path’s complete model inputs, observations, dependencies, and independent tool-service durations across methods. The recorded paths contain 204 model calls: 16 initial calls and 188 tool-result continuations. The output cap is 256 tokens with natural stopping. Both methods apply gzip6 lossless compression to tool observations over the same 48 kbit/s return link, with encoding and decoding included in measured workflow time. The comparison uses three matched arrival seeds over the same captured task paths.
Policies and matched controls. All policies select completed results at the same gateway. References are ReAct with ready-order FIFO delivery, Size-first (smallest feasible frame), and our delivery-side adaptation of Tail-Risk-Aware Turn Release Scheduling (Tail-aware Release) [9]. The adaptation retains its ready-frozen full-prompt work estimate plus four times causal expected output, engine feedback, and model-completion ledger retirement. LADS combines Cache-first ordering, bounded FIFO leases, and turnover. Component controls compare Cache-first without leases, Turnover-off with quota-turnover priority disabled, and complete LADS. LADS and Turnover-off share inputs, model entry, the lease lifecycle, the demand-derived budget rule, headroom blocks, and cumulative lease limit seconds; execution feedback can produce different realized trajectories.
Metrics and aggregation. The primary metric is mean complete-workflow latency over all workflows, measured in seconds from workflow arrival to termination. It includes initial inference, tool-service waiting, delivery, engine waiting, and the final model call. Tables average per-run means and nearest-rank P95 values equally. With 16 workflows, nearest-rank P95 selects the largest observation. Each reported percentage is computed within a matched seed and then averaged equally: , where is the number of matched seeds. Positive values indicate lower candidate latency. This differs from a ratio of displayed mean times. Makespan measures whole-batch completion separately from per-workflow latency. Captured-workflow statistics average three matched arrival seeds over the same recorded task paths.
Cache measurements distinguish historical-prefix lookup hints from engine full-prompt cache observations. Controller CPU time is cumulative CPU seconds per run, including budget evaluation. Protection block-time is the time integral of the distinct protected-block count , measured in block-seconds. Serialized control bytes, protection block-time, preemptions, grants, expiries, and handoffs accompany latency. Each method’s control and protection mechanisms are included in its measured cost. Actual allocation, handoff, and expiry events establish what occurred when published state lags execution.
End-to-end workflow performance
Table 1 shows that LADS lowers mean complete-workflow latency in controlled cohort-8 by 44.6% against FIFO, 41.7% against Size-first, and 37.0% against Tail-aware Release across five matched input seeds. At cohort-16, LADS also achieves the lowest mean latency: 119.0 seconds versus 204.6 seconds for FIFO (Table 3).
| Method | Controlled (cohort-8) Mean | Controlled (cohort-8) P95 | Web-derived burst replay Mean | Web-derived burst replay P95 |
| ReAct (FIFO) | 196.6 | 210.4 | 204.8 | 218.1 |
| Size-first | 186.7 | 201.1 | 158.1 | 171.6 |
| Tail-aware Release | 176.5 | 196.7 | 192.3 | 211.4 |
| LADS (ours) | 108.9 | 119.2 | 118.0 | 130.0 |
Table 1. Complete-workflow latency (seconds; lower is better). Controlled cohort-8 results average five input seeds; Web-derived burst replay averages three input seeds. The two workload columns use 16 workflows, 96 calls, and a 32 kbit/s return link. Entries average run-level means and P95s equally. Boldface identifies LADS; underlining marks minima.
| Method | A mean | A P95 | B mean | B P95 |
| ReAct (FIFO) | 196.6 | 210.4 | 204.6 | 218.3 |
| Size-first | 186.7 | 201.1 | 151.6 | 165.1 |
| Tail-aware Release | 176.5 | 196.7 | 203.4 | 223.1 |
| LADS (ours) | 108.9 | 119.2 | 119.0 | 131.7 |
Table 3. Five-seed controlled complete-method comparisons, in complete-workflow seconds. A/B denote cohort-8/cohort-16, with 16 workflows per run. Entries equally average five run-level means or nearest-rank P95 values. Boldface identifies LADS (ours).
In Web-derived burst replay, the corresponding reductions are 42.4%, 25.3%, and 38.6% across three matched seeds, with LADS improving on every reference in every seed. These totals include all workflows, initial calls, and tool waits; Table 1 reports absolute means and P95s.
Captured Web-agent workflow replay
Table 2 evaluates FIFO and LADS on the 16 captured task paths. Across three matched arrival seeds, LADS reduces mean complete-workflow latency by 50.6% (48.2–51.8% within seeds), with lower latency in all 48 paired workflow comparisons. Mean latency falls from 156.6 to 77.3 seconds, including initial inference and all tool waits.
| Method | Mean | P95 |
| ReAct (FIFO) | 156.6 | 187.4 |
| LADS (ours) | 77.3 | 105.1 |
Table 2. Complete-workflow latency for captured Web-agent paths (seconds). Both methods replay the same 16 recorded paths, comprising 204 model calls and 188 continuations, using gzip6 at 48 kbit/s. Entries equally average three matched arrival-seed run statistics. Boldface identifies LADS (ours).
Ablation and mechanism analysis
Figure 5 separates protection (Cache-first versus Turnover-off) from turnover (Turnover-off versus complete LADS). With the budget rule and lease lifecycle held constant, turnover lowers mean cohort-8 latency by 7.6% across five matched inputs, improving four seeds.
Figure 5. Controlled cohort-8 component ablation (16 workflows, five matched input seeds). Bars show equal-weight means; points show individual runs. Cache-first has no leases; Turnover-off shares LADS’s budget and lease lifecycle with quota-turnover priority disabled. Model work includes pre-fill, decode, and repeated execution.
For the captured paths, Figure 6 shows receipt-to-allocation falling from 5.5 seconds to 7.5 milliseconds per continuation and allocation-to-completion from 1.9 to 0.9 seconds. Engine full-prompt cache observations rise from 0.95M to 1.69M cached tokens per run, alongside 51.7M promptations and 95.3 verified handoffs. Table 5 and Table 6 give all intervals and costs from these runs.

Figure 6. Captured Web-agent continuation latency (gzip6, 48 kbit/s). Stage means use the same 188 continuations per run and weight the three matched arrival seeds equally. Receipt-to-allocation includes registration and engine waiting. Initial calls are included in complete-workflow totals.
| Method | Ready–commit (s) | Commit–receipt (s) | Receipt–alloc. (ms) | Alloc.–end (s) |
| ReAct (FIFO) | 4.1 | 1.8 | 5500.3 | 1.9 |
| LADS (ours) | 3.6 | 2.0 | 7.5 | 0.9 |
Table 5. Mean stage latency per continuation in captured Web-agent workflow replay. Units are shown in the column headers. Entries equally average three matched run means over the same 188 continuations per run. Receipt-to-allocation includes registration and engine waiting. Boldface identifies LADS (ours).
| Method | Cached (ktok) | Missing (ktok) | Protect. (kblock-s) | CPU (s) | Budget CPU (s) | Control (MiB) | P/H/E |
| ReAct (FIFO) | 947.9 | 1053.9 | 0 | 8.4 | 0 | 11.6 | 0/0/0 |
| LADS (ours) | 1693.2 | 308.6 | 144.8 | 11.5 | 5.3 | 9.8 | 51.7/95.3/9 |
Table 6. Cache reuse and control costs in captured Web-agent workflow replay. Entries equally average three per-run measurements. Cache input covers all 204 calls; missing input is prompt tokens minus cached tokens. CPU includes state-synchronization checks and owner budget control. P/H/E gives mean per-run promotions, verified handoffs, and expiries. Boldface identifies LADS (ours).
Control overhead
In captured Web-agent replay, LADS uses 11.5 cumulative controller CPU seconds per run versus FIFO’s 8.4, including budget evaluation and state synchronization. It consumes 144,792.6 protection block-seconds per run; all 148,381 added references across the three runs are released. Codec time remains included. Table 4 and Table 6 report the remaining costs; serialized control bytes measure in-process records.
| Method | Mean (s) | Work (ktok) | Protect. (kblock-s) | Actions | CPU (s) |
| Cache-first | 185.2 | 1515.1 | 0 | 0/0/0 | 1.7 |
| Turnover-off | 118.0 | 882.0 | 271.3 | 0/28.6/17.8 | 3.5 |
| LADS (ours) | 108.9 | 764.5 | 274.0 | 21.6/36.4/18.2 | 4.3 |
Table 4. Controlled cohort-8 component ablation over five matched input seeds. Work is actual scheduled model work, including prefill, decode and repeat work, in thousands of tokens; protection is thousands of distinct-block seconds. Actions are promotions/handoffs/expiries, averaged per run. CPU includes the complete controller. Cache-first has no added leases; Turnover-off and LADS share the demand-budget rule and lease lifecycle, differing only in sender turnover. Boldface identifies LADS (ours).
Related Work
Web agents use returned observations to guide subsequent model decisions [41, 48]. Serving such agents concurrently involves scheduling model calls and managing reusable context [1, 23].
Tool use and Web-agent evaluation. ReAct interleaves reasoning and action [41]; Toolformer, ToolLLM, and Gorilla develop tool/API use [25, 28, 29], while LLMCompiler exposes parallel-call dependencies [12]. Interactive and action-generation benchmarks include WebShop, WebArena, Mind2Web, WebLINX, VisualWebArena, and WorkArena [6, 7, 13, 22, 40, 48]; BrowserGym, AgentLab, and AgentBoard provide evaluation and progress measurement [15, 24, 30]. LADS schedules completed observations across workflows while holding agent policy and observation semantics common in matched comparisons.
Inference scheduling and application structure. Orca, Sarathi-Serve, and DistServe coordinate iterations, prefills, and decoding [2, 43, 47]; FastServe and Llumnix preempt or redistribute requests [33, 35]. AlpaServe and the Virtual Token Counter address allocation and fairness [17, 31], while PagedAttention and vAttention manage physical cache memory [14, 26]. SGLang, Parrot, and Agentix exploit program structure or cross-call progress [18, 23, 46]; Sutradhara co-designs tool orchestration and inference to overlap tools with prefill [3], and Preble combines prefix sharing with placement [32]. LADS instead selects which completed result reaches ordinary FIFO registration next.
Context reuse and residency. Prompt Cache, CachedAttention, and ChunkAttention reuse modular, multi-turn, or shared-prefix state [10, 11, 42]; CacheBlend combines cached knowledge segments [39]. CacheGen and Mooncake compress, stream, or pool KV state [19, 27], and Pensieve spans memory tiers [44]. PEEK guides admission and eviction with pending prefixes [37]; H₂O, StreamingLLM, and KIVI select or quantize retained context [21, 36, 45]. LADS couples completed-result order with bounded resident-prefix protection and quota return at handoff.
Interruptions, readiness, and delivery. InferCept manages context around external calls, Continuum uses cache time-to-live, and Ask the Tool, Don’t Guess uses tool progress for retention/reload decisions [1, 16, 20]. Tail-aware Release prioritizes ready turns and controls committed model work [9]; our delivery-side adaptation retains its work estimate, causal output history, engine feedback, and model-completion ledger. SCOOT tunes inference-engine configurations [4], Baton reorganizes batches [5], and Fate accelerates edge mixture-of-experts inference [8]. LADS addresses completed-observation delivery and reuse at handoff.
Discussion
LADS targets shared return paths where sender backlog coincides with resident histories competing for finite protection. The benefit depends on whether an owner reaches handoff within its lease horizon; sustained committed transfers and engine waiting can consume that horizon before useful turnover.
The evaluation covers randomized controlled inputs, Web-derived burst replay, and captured model-selected paths using one model. Web-derived input seeds vary arrivals and history assignments, while captured-path arrival seeds replay the same task paths. Broader tasks and models would test how return-link capacity, reuse, and lease duration shape this operating regime.
Conclusion
LADS is a tool-result delivery scheduler for concurrent Web agents sharing a constrained return path and an inference service. It protects waiting histories with bounded prefix leases and, under protection pressure, prioritizes results whose timely resumption can return protection quota to other waiting histories. Compared with ReAct using FIFO tool-result delivery, LADS reduces mean complete-workflow latency by 44.6% on controlled cohort-8 bursts and 42.4% on Web-derived burst replay, without changing the model or agent policy. In the two burst workloads, LADS lowers mean latency relative to all three delivery baselines on every input seed.
More broadly, LADS makes tool-result delivery part of agent serving: delivery order affects when requests reach inference and whether cached histories remain available for reuse. As Web agents undertake longer tasks with more tool calls on shared inference services, coordinating communication with inference state offers a practical way to improve serving efficiency.
References
Additional evaluation results
Captured-path stage measurements. Table 5 and Table 6 report stage timing, cache work, and control costs for the same three matched arrival seeds in the captured Web-agent workflow replay of § 5.3.
Randomized inputs and replication pairs. Table 7 lists the five input seeds for each controlled cohort setting and the three Web-derived burst input seeds. Table 8 lists the three matched arrival seeds over the same captured task paths. Each group preserves its own seed identity.
| Workload | Seed | FIFO | Size-first | Tail-aware | Release | LADS (ours) |
| Cohort-8 | 1667935512 | 194.9 | 187.5 | 122.4 | 96.9 | 96.9 |
| Cohort-8 | 1461454527 | 194.4 | 175.8 | 182.6 | 109.6 | 109.6 |
| Cohort-8 | 1055731457 | 196.6 | 184.4 | 193.0 | 102.4 | 102.4 |
| Cohort-8 | 1321697771 | 197.9 | 189.1 | 196.3 | 106.2 | 106.2 |
| Cohort-8 | 711104771 | 199.4 | 196.6 | 188.1 | 129.5 | 129.5 |
| Cohort-16 | 1667935512 | 201.0 | 157.8 | 197.1 | 111.6 | 111.6 |
| Cohort-16 | 1461454527 | 200.2 | 143.8 | 190.8 | 110.5 | 110.5 |
| Cohort-16 | 1055731457 | 210.9 | 149.8 | 209.5 | 116.1 | 116.1 |
| Cohort-16 | 1321697771 | 203.3 | 152.1 | 201.2 | 119.6 | 119.6 |
| Cohort-16 | 711104771 | 207.7 | 154.4 | 218.3 | 137.3 | 137.3 |
| Web-derived burst | 732418792 | 200.2 | 159.0 | 196.7 | 112.4 | 112.4 |
| Web-derived burst | 1533424038 | 207.4 | 159.4 | 188.4 | 125.0 | 125.0 |
| Web-derived burst | 55546983 | 206.9 | 155.7 | 191.7 | 116.7 | 116.7 |
Table 7. Per-seed complete-workflow means (seconds). Controlled cohort-8 and cohort-16 comparisons each use five input seeds; Web-derived burst replay uses three input seeds. Each group preserves its own seed identity. Boldface identifies LADS (ours).
| Arrival seed | FIFO | LADS (ours) |
| 172910251 | 159.0 | 76.7 |
| 1147784114 | 159.7 | 76.9 |
| 688237994 | 151.1 | 78.2 |
Table 8. Per-arrival-seed complete-workflow means for captured Web-agent workflow replay (seconds). FIFO and LADS use the same recorded task paths and matched arrivals across three seeds. Boldface identifies LADS (ours).
Budget estimation and admission
§ 3.4 defines the budget estimate and near-use demand. The lease provider uses its current local inventory or the latest accepted, identity-matching sender inventory. The FIFO prefix contains the continuation frames currently carried by the return path. A partly transmitted frame can conservatively remain counted at full size.
Measurement and validity checks
Table 9 lists the software versions and hardware configuration shared by all workloads.
| Component | Configuration |
| Model revision | a09a354 |
| PyTorch | 2.8.0 |
| vLLM | 0.11.0 |
| Transformers | 4.56.2 |
| Host CPU | AMD Ryzen Threadripper PRO 7975WX |
| GPU model | NVIDIA RTX PRO 5000 Blackwell |
| Reported GPU memory | 48,935 MiB per card |
Table 9. Software and hardware configuration shared by all workloads.
Stage latencies. Using the timestamps in § 2.1, we divide each continuation’s ready-to-completion time into sender waiting (), queuing and transfer (), receipt to allocation (, including registration and engine waiting), and resumed inference (). Table 5 and Figure 6 average the same continuations within each run, then weight run means equally. Initial calls enter complete-workflow latency but not these stage means.
LADS budget-context observation uses 0.3 CPU seconds per run, included in total CPU. FIFO/LADS codec CPU is 40.8/40.7 ms per run, recorded separately and included in workflow wall time.
Record validity. The sender accepts ownership records only when workflow/turn versions and prefix/payload digests match and the records are at most one second old. Target, commitment, and budget-denial records must share the same received budget generation.
The sender uses Cache-first when identity, sequence, or timestamp checks fail, including stale or duplicate records, future timestamps, and inconsistent event order. Each completed run verifies that all added lease references have been released.
References
- [1]Reyna Abhyankar, Zijian He, Vikranth Srivatsa, Hao Zhang, and Yiying Zhang. 2024. InferCept: Efficient Intercept Support for Augmented Large Language Model Inference. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). PMLR, 81–95. https://proceedings.mlr.press/v235/abhyankar24a.htmlarxiv.org/abs/2402.01869
- [2]Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, 117–134. https://www.usenix.org/conference/osdi24/presentation/agrawalarxiv.org/abs/2403.02310
- [3]Ashish Biswas, Kanishk Goel, Srivarshinee S, Jayashree Mohan, Alind Khare, Anjal Parayil, Ramachandran Ramjee, and Chetan Bansal. 2026. Sudrathara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inference. Preprint, version 3, April 22, 2026. arXiv:2601.12967 [cs.DC] doi:10.48550/arXiv.2601.12967arxiv.org/abs/2601.12967
- [4]Ke Cheng, Zhi Wang, Wen Hu, Tiannuo Yang, Jianguo Li, and Sheng Zhang. 2025. SCOOT: SLO-Oriented Performance Tuning for LLM Inference Engines. In Proceedings of the ACM Web Conference 2025 (WWW ’25). Association for Computing Machinery, 829–839. doi:10.1145/3696410.3714930DOI
- [5]Peizhuang Cong, Qizhi Chen, Haochen Zhao, and Tong Yang. 2025. Baton: Enhancing Batch-wise Inference Efficiency for Large Language Models via Dynamic Re-batching. In Proceedings of the ACM Web Conference 2025 (WWW ’25). Association for Computing Machinery, 2309–2318. doi:10.1145/3696410.3714950DOI
- [6]Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2Web: Towards a Generalist Agent for the Web. In Advances in Neural Information Processing Systems, Vol. 36. Curran Associates, Inc., 28091–28114. doi:10.52202/075280-1220DOI
- [7]Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. 2024. WorkArena: How Capable are Web Agents at Solving Common Knowledge Work Tasks? . In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). PMLR, 11642–11662. https://proceedings.mlr.press/v235/drouin24a.htmlarxiv.org/abs/2403.07718
- [8]Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gate. In Proceedings of the ACM Web Conference 2026 (WWW ’26). Association for Computing Machinery, 5537–5546. doi:10.1145/3774904.3792527DOI
- [9]Bohao Feng, Jianjiang Li, Haojie Wang, Lin Qiao, Yinghui Li, Yukun Yan, and Jidong Zhai. 2026. Decoupling Readiness from Release for Tail-Aware Scheduling of Agentic LLM Workflows. Preprint, version 1, September 10, 2026. arXiv:2609.10964 [cs.AI] https://arxiv.org/abs/2609.10964v1
- [10]Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. 2024. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention. In 2024 USENIX Annual Technical Conference (USENIX ATC 24). USENIX Association, 111–126. https://www.usenix.org/conference/atc24/presentation/gao-bin-costarxiv.org/abs/2403.19708
- [11]Prompt Cache: Modular Attention Reuse for Low-Latency Inference. In Proceedings of Machine Learning and Systems, Vol. 6. 325–338. https://proceedings.mlsys.org/paper_files/paper/2024/hash/a66caa1703ef3405a4368c3014c1966-Abstract-Conference.html
- [12]Sehoon Kim, Suhong Maoon, Ryan Tabrizi, Nicholas Lee, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami. 2024. An LLM Compiler for Parallel Function Calling. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). PMLR, 24370–24391. https://proceedings.mlr.press/v235/kim24y.htmlarxiv.org/abs/2312.04511
- [13]Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shu yan Zhou, Russ Salakhutdinov, and Daniel Fried. 2024. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Bangkok, Thailand, 881–905. doi:10.18653/v1/2024.acl-long.50
- [14]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23). Association for Computing Machinery. doi:10.1145/3600006.3613165DOI
- [15]Thibault Le Sellier de Chezelles, Maxime Gasse, Alexandre Lacoste, Massimo Caccia, Alexandre Drouin, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omid Shavegan, Lawrence Keunho Jang, Xing Han Lü, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Graham Neubig, Quentin Cappart, Russ Salakhutdinov, and Nicolas Chapados. 2025. The BrowserGym Ecosystem for Web Agent Research. Transactions on Machine Learning Research (2025). https://openreview.net/forum?id=5298fKGmv3
- [16]Hanchen Li, Runyuan He, Qiuyang Mang, Qizheng Zhang, Huan zhi Mao, Xi-aokun Chen, Hangrui Zhou, Huanchen Zhang, Alvin Cheung, Joseph Gonzalez, and Ion Stoica. 2026. Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live. Preprint, version 7, September 8, 2026. arXiv:2511.02230 [cs.OS] doi:10.48550/arXiv.2511.02230
- [17]Zhuohan Li, Lianzhua Chen, Hengruiding Zhou, Yingying Yang, Fan Yang, Chen Chen, and Lili Qiu. 2024. Parrot: Efficient Serving of LLM-based Applications with Semantic Variable. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, 929–945. https://www.usenix.org/conference/osdi24/presentation/lin-chao fanarxiv.org/abs/2302.11665
- [18]Chaofan Lin, Zhenhua Han, Chengruiding Zhou, Yuqing Yang, Fan Yang, Chen Chen, and Lili Qiu. 2024. Parrot: Efficient Serving of LLM-based Applications with Semantic Variable. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, 929–945. https://www.usenix.org/conference/osdi24/presentation/lin-chao fanarxiv.org/abs/2405.19888
- [19]Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Gane sh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. 2024. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving. In Proceedings of the ACM SIGCOMM 2024 Conference (ACM SIGCOMM ’24). Association for Computing Machinery. doi:10.1145/3651890.3672274
- [20]Yipeng Liu, Yingqiang Zhang, Feifei Li, and Huanchen Zhang. 2026. Ask the Tool, Don’t Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read It. Preprint, version 1, September 16, 2026. arXiv:2609.18849 [cs.DC] https://arxiv.org/abs/2609.18849v1
- [21]Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024. KVFI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). PMLR, 32332–32344. https://proceedings.mlr.press/v235/liu24bz.htmlDOI
- [22]Xing Han Lu, Zdeněk Kasner, and Siva Reddy. 2024. WebLLINX: Real-World Website Navigation with Multi-Turn Dialogue. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). PMLR, 33007–33056. https://proceedings.mlr.press/v235/lu24e.htmlarxiv.org/abs/2402.05930
- [23]Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang, Yaping Huan, Zhifeng Chen, Joseph E. Gonzalez, and Ion Stoica. 2026. Agentix: An Efficient Serving Engine for LLM Agents as General Programs. In 23rd USENIX Symposium on Operating Systems Design and Implementation (NSDI 26). USENIX Association, 2443–2459. https://www.usenix.org/conference/nsdi26/presentation/luoarxiv.org/abs/2502.13965
- [24]Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. 2024. AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents. In Advances in Neural Information Processing Systems, Vol. 37. Curran Associates, Inc., 74325–74362. doi:10.52202/079017-2365
- [25]Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2024. Gorilla: Large Language Model Connected with Massive APIs. In Advances in Neural Information Processing Systems, Vol. 37. Curran Associates, Inc., 126544–126565. doi:10.52202/079017-4020DOI
- [26]Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, and Ashish Panwar. 2025. vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1. Association for Computing Machinery, 1133–1150. doi:10.1145/3669940.3707256DOI
- [27]Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation—A KVCache-centric Architecture for Serving LLM Chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25). USENIX Association, 155–170. https://www.usenix.org/conference/fast25/presentation/qin
- [28]Yujia Qin, Shih ao Liang, Ying Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. In International Conference on Learning Representations, Vol. 2024. 9695–9717. https://proceedings.iclr.cc/paper_files/paper/2024/hash/28e50ee5b72e90b50e7196fde8ea260e-Abstract-Conference.htmlarxiv.org/abs/2307.16789
- [29]Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. In Advances in Neural Information Processing Systems, Vol. 36. Curran Associates, Inc., 68539–68551. doi:10.52202/075280-2997DOI
- [30]ServiceNow Research. 2026. AgentLab: An Open-Source Framework for Developing and Evaluating Web Agents. Software repository. Accessed October 17, 2026. https://github.com/ServiceNow/AgentLab
- [31]Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E. Gonzalez, and Ion Stoica. 2024. Fairness in Serving Large Language Models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, 965–98. https://www.usenix.org/conference/osdi24/presentation/shengarxiv.org/abs/2401.00588
- [32]Krivanthra Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, and Yiying Zhang. 2025. Preble: Efficient Distributed Prompt Scheduling for LLM Serving. In International Conference on Learning Representations. 37057–37082. https://proceedings.iclr.cc/paper_files/paper/2025/hash/5bc342f48de8264779952fac378f96dc-Abstract-Conference.html
- [33]Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. 2024. Lumnix: Dynamic Scheduling for Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, 173–191. https://www.usenix.org/conference/osdi24/presentation/sun-biaoarxiv.org/abs/2406.03243
- [34]vLLM Contributors. 2025. Automatic Prefix Caching. vLLM version 0.11.0 design documentation; accessed October 26, 2026. https://docs.vllm.ai/en/v0.11.0/design/prefix_caching.html
- [35]Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. 2026. FastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). USENIX Association, 57–74. https://www.usenix.org/conference/nsdi26/presentation/wu-bingyang
- [36]Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient Streaming Language Models with Attention Sinks. In International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/paper/2024/hash/5e5fd18f863cbe6d8ae392a93fd271c9-Abstract-Conference.htmlarxiv.org/abs/2309.17453
- [37]Bing Xie, Zhipeng Wang, Masahiro Tanaka, and Zhen Zhang. 2026. PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving. Preprint under review, version 2, September 12, 2026. arXiv:2607.02525 [cs.DC] doi:10.48550/arXiv.2607.02525DOI
- [38]An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Chang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yunxiang Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2024. Qwen2.5 Technical Report. Version 2, revised January 3, 2025. arXiv:2412.15115 [cs.CL] doi:10.48550/arXiv.2412.15115DOI
- [39]Jiayi Yao, Hancheng Li, Yuhan Liu, Siddharth Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion. In Proceedings of the Twentieth European Conference on Computer Systems (EuroSys ’25). Association for Computing Machinery, 16 pages. doi:10.1145/3689031.3696098
- [40]Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. In Advances in Neural Information Processing Systems, Vol. 35. Curran Associates, Inc., 20744–20757. doi:10.52202/064831-1508arxiv.org/abs/2207.01206
- [41]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. WebArena: A Realistic Web Environment for Building Autonomous Agents. In International Conference on Learning Representations. https://openreview.net/forum?id=WE_vluYUL-X
- [42]Lu Ye, Ze Tao, Yong Huang, and Yang Li. 2024. ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 11608–11620. doi:10.18653/v1/2024.acl-long.623DOI
- [43]Gyeong-In Yu, Junsung Lee, Goon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, 521–538. https://www.usenix.org/conference/osdi22/presentation/yu
- [44]Lingfan Yu, Jinkun Lin, and Jinyang Li. 2025. Stateful Large Language Model Serving with Pensieve. In Proceedings of the Twentieth European Conference on Computer Systems (EuroSys ’25). Association for Computing Machinery, 15 pages. doi:10.1145/3689031.3696086DOI
- [45]Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. In Advances in Neural Information Processing Systems, Vol. 36. Curran Associates, Inc., 34661–34710. doi:10.52202/072850-1506
- [46]Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. In Advances in Neural Information Processing Systems, Vol. 37. Curran Associates, Inc., 62557–62583. doi:10.52202/079017-2000
- [47]Yimin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanxhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, 193–210. https://www.usenix.org/conference/osdi24/presentation/zhong-yimin
- [48]Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xinyi Chen, Tianyu Oou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. WebArena: A Realistic Web Environment for Building Autonomous Agents. In International Conference on Learning Representations, Vol. 2024. 15585–15606. https://proceedings.iclr.cc/paper_files/paper/2024/hash/4410c0711e9154a7a2d6f9b3816d1ef-Abstract-Conference.htmlarxiv.org/abs/2307.13854




