WorksComputer Science
Aggron: Aggressive Compression of Agentic KV Caches during Head-of-Line Stalls in the Request Queue
Abstract
LLM agents run long, multi-turn programs whose contexts grow at every step, and serving them is limited by KV cache capacity. To alleviate this limitation, systems use context compression techniques, which typically wait until the KV cache crosses a large threshold and then summarize large spans of history at once, either in the background while the agent progresses or on the critical path. These summarizations can be done using the same model running on the same engine, or using the same or a different model running on extra hardware. When running on the same engine, summarization competes with the requests being served for both compute and memory capacity. To maximize the cost-effectiveness of summarization, this paper presents a new serving system called Aggron. Aggron summarizes agentic requests at every turn and selectively summarizes only certain parts of the KV cache. As a result, Aggron retains accuracy and keeps summarization requests lightweight. Moreover, Aggron launches summarization requests from a separate Summarization Queue while the corresponding agentic request whose KV cache we want to compress is stalled in the main request queue due to head-of-line stalls. The Summarization queue utilizes compute and memory capacity left over by the head-of-line stalls in the main queue. Further, to maximize efficiency, the server issues summarizations only when it predicts that they can complete before the corresponding agentic request will finally execute. In our evaluation, we compare Aggron to state-of-the-art asynchronous summarization. At high load, Aggron reduces P99 program latency by –, and raises throughput by –.
Introduction
Large Language Models (LLMs) are increasingly deployed as agents that invoke tools [57, 75] to edit code, run shell commands, query databases, and browse the web to complete multi-step tasks autonomously [5, 47, 73]. Unlike a chat, an agentic task is a long-running program of many LLM calls, each followed by a tool use whose output is appended to the context. Executions of tens to hundreds of LLM calls are common. In production traces [85] that include Claude Code [5] and Codex [47] sessions, the 90th-percentile session issues over 100 model calls (241 for Codex), and its context peaks at 186K tokens (235K for Codex). Even on frontier mixture-of-experts models with compact attention, the result is a large memory footprint—e.g., we compute that such a session holds 36–45 GB of context with Qwen3-235B-A22B [53] and 13–17 GB with DeepSeek-V3’s latent attention [12].
A request’s context is stored in the KV cache [52] in the accelerator’s HBM memory. As the request runs, it accumulates context, which can consume substantial memory. Figure 1 shows how the increase in the context length of a request in tokens (X-axis) increases the KV cache size (Y-axis). The figure depicts lines for multiple models, and shows the KV cache size for 1M tokens (upper right side numbers). The figure also shows, in horizontal lines, the size of the HBM memory in popular machines. We can see that one GLM-4.5 [18] request with a 128K-token context fills 60 GB; two of them would fill 80GB, which is the memory of an H100 GPU.
Figure 1. KV cache size of a request vs. context length (assuming bf16 data types) for different models.
To serve multiple requests simultaneously, servers continuously batch requests based on specific scheduling policies [27, 77, 82] and availability of space in the KV cache. A request is batched only if its context fits in the KV cache. Agentic requests hold so much memory that, within a few requests, the KV cache saturates and the batch size freezes, leaving the GPU’s compute underused while requests wait in queues.
To avoid this problem, KV caches can be periodically summarized to shrink their size. A summarizer uses the record of the agent’s history to read or recompute the KV cache and then summarize it.
Recent work shows that frontier agents that compact their history whenever it reaches 12–18K tokens and restart from the summary end up losing little to no task accuracy [7]. That work, however, reports only final accuracy, not whether compressing and restarting changes the agent’s trajectory and with it the cost to serve.
An open problem is how to simultaneously serve agentic and summarization requests under load on a shared server. In general, there are two approaches: synchronous and asynchronous. In synchronous summarization, the agent harness waits until the context approaches a certain threshold, and then stalls the agent while a summarization request executes. In asynchronous summarization [10, 36], the harness lets the agent continue while a request summarizes the context. The agent runs on the full long context until the summarization completes. With synchronous summarization, the summarization is in the critical path, raising the mean and P99 latency. With asynchronous summarization, the summarization and agentic requests compete for memory and compute capacity. The top and middle rows of Figure 2 show a timeline of synchronous and asynchronous summarization, respectively. In the latter, the agentic execution takes longer due to contention. In both cases, summarization can add significant overhead.
Figure 2. Comparing different summarization strategies.
To maximize the cost-effectiveness of summarization, this paper presents a new serving system called Aggron. Aggron reduces the overhead of summarization through several techniques. First, Aggron summarizes agentic requests at every turn, to limit the work performed per summarization operation, and to opportunistically hide summarization with the agent’s waiting time in the serving queue. In addition, Aggron selectively summarizes only certain parts of the KV cache, namely those that are likely to have less impact on the accuracy. Second, Aggron launches summarization requests from a separate Summarization Queue. A summarization is launched while the corresponding agentic request whose KV cache it wants to compress is stalled in the main request queue due to head-of-line stalls. The Summarization queue adds modest overhead because it utilizes compute and memory capacity left over by the head-of-line stalls in the main queue. Third, to maximize efficiency, the server issues a summarization only when it predicts that it can complete before the corresponding queued agentic request will finally execute. The server has information about waiting agentic requests and unused capacity. When there is no queue to hide behind, the server reverts to synchronous summarization. With Aggron, the resulting timeline of the summarization is shown in the bottom row of Figure 2. We can see that the execution time decreases.
We measure Aggron’s accuracy on real agentic tasks from four benchmarks, served on NVIDIA H100 NVL GPUs, and evaluate its P99 latency and throughput with a simulator calibrated against the same hardware. We use simulations to limit the cost of the many runs needed. Across three models (Qwen3.6-35B-A3B [55], Gemma-4-26B-A4B [62], and GLM-4.7-Flash [18]), Aggron’s accuracy stays within run-to-run noise of uncompressed execution on every benchmark (i.e., within 2.7 pp). Compared to the Slipstream [10] and APEC [36] state-of-the-art asynchronous summarization schemes, Aggron reduces P99 latency by 27% and 31%, reduces mean latency by 11% and 15%, and increases throughput by 1.34× and 1.40×, respectively.
This paper makes the following contributions:
A characterization of agentic serving under KV pressure, showing that requests spend most of their latency waiting for admission, and that selective summarization does not hurt accuracy.
Aggron, the first system to compress agentic requests opportunistically at the server in online, multi-tenant serving.
An evaluation of Aggron’s accuracy, trajectory divergence, and serving performance against state-of-the-art techniques on different models.
Background
LLM inference runs in two phases [64]. Prefill processes the whole prompt in one forward pass and is compute-bound; long prompts are often split into chunks and interleaved with decode work [3]. Decode then generates one token per step and is bound by the bandwidth needed to read the cached state from HBM [52, 58]. Both phases store, for every token and every attention layer, a key and a value vector, the KV cache, which is kept for the request’s lifetime so that later steps need not recompute it.
Agentic requests make this cache large, since every call carries the task’s whole history. In production traces [85], more than half of all calls carry over 100K tokens (8–30 GB of KV; Figure 3).
Figure 3. Distribution of context length per agent call, across our benchmarks (Qwen3.6-35B-A3B) and production traces.
Architectures have been modified to reduce the per-token cost. Grouped-query attention [4] shares keys and values across heads, latent attention [12] compresses them, and linear attention [24] replaces them with a fixed-size recurrent state in some layers. Qwen3.6 [55] combines GQA with linear attention in three of every four layers. These reduce the slope of the footprint but not its growth with context (Figure 1). Even on Qwen3.6, about 40 requests of 64K tokens, a common agentic size, fill an H100’s HBM.
Serving systems batch requests at every forward pass, adding new requests and retiring finished ones (continuous batching [77]).
A waiting request is admitted, in first-come-first-served order, only when enough blocks are free to hold its context; KV of a shared prefix can be reused from a prefix cache instead of being prefilled again [40, 82]. For agentic requests, KV capacity, not compute, decides how many requests run together. We replay the long traces [15, 48, 61, 85] and single-turn traces [45, 50] and measure batch occupancy (Figure 4) at tensor parallelism of 8. Single-turn requests are small, and the batch reaches its limit of 1024. Agentic requests fill the HBM first, and the batch plateaus well below it, at a median of 37 requests on production traces, while the rest wait in the queue until enough KV is free.
Figure 4. Median batch occupancy versus arrival rate. Single-turn requests (Azure) are small, and the batch reaches its 1024 limit. Agentic requests fill the HBM first.
A request’s footprint can also be cut at runtime by compressing its context. Pruning [9, 42, 69, 72, 81] drops tokens, or their cached keys and values, chosen by position or by the attention they have received; it edits the cache in place and needs no recomputation, but evicting a token in the middle of a sequence breaks prefix-cache matches for every block after it. Summarization [10, 23, 37, 84] replaces spans of the context with text generated by the model itself; it costs an LLM call, and the KV of the rewritten context must be prefilled again. Deployed agents summarize synchronously, stopping when the context nears the model’s window.
Opportunities for Compression of Agentic Requests
To understand the impacts and challenges of summarization on agentic requests, we start by running experiments on an H100 GPU with 94 GB of HBM3, using the Qwen3.6-35B-A3B-FP8 [55] model, served by a vLLM [27] server. We draw agentic requests from the Terminal-Bench [44], MCPMark [67], DeepResearch Bench [14], and SWE-bench [21] benchmark suites. We present our main observations that motivate Aggron’s design by answering the following questions.
What is the best way to reduce the memory footprint of a request? Pruning is the cheaper of the two runtime methods, but it is the wrong fit for agents, because it decides what to keep from the attention a token has received so far, and agents return to old context. Figure 5 shows the attention of the last 3K query tokens of a 38-iteration SWE-bench trajectory. Vertical stripes across the whole history are spans, read thousands of tokens and many iterations earlier, that later steps keep attending to. Worse, the history is revisited unpredictably, with keys around 6-8K receiving almost no attention from most later queries, yet two later iterations return to them. An attention-ranked budget would already have evicted exactly those tokens. Offloading them to host memory keeps them reachable, but every decode step then fetches KV over the CPU–GPU interconnect, which has 10-50× less bandwidth than HBM [28, 60].

Figure 5. Attention of the last 3K query tokens of a 38-iteration SWE-bench trajectory (astropy-14309, 20K tokens) over all earlier tokens, at one head of a full-attention layer (layer 15) of Qwen3.6-35B-A3B; log color scale
Summarization decides what to keep from meaning instead, with the model itself as the judge, and it is safe at small budgets. Agents whose prose is capped at 4K tokens lose at most 2.7 pp of uncompressed accuracy, consistent with frontier agents compacted to 16K [7]. We therefore compress by summarization.
Observation 1: Pruning discards context that agents later revisit; summarization keeps what matters.
Are all parts of a request’s memory compressible? Summarizations can compress the entire context or parts of it. To ensure that requests can run indefinitely, existing systems trigger summarizations whenever the context approaches the maximum model length. We experiment with summarization triggers and observe their impact on accuracy. The trigger refers to the context size in token count after which compression through summarization occurs, effectively fixing the maximum context size. Table 1 tabulates the variation in accuracy with the trigger size. SWE-bench consists of coding tasks whose context includes code snippets, among other information. Summarizing raw code results in loss of variable names and assertion statements, as the summary captures the function’s intent but not its semantics. This results in incorrect decisions later in the trajectory, hurting accuracy. The accuracy saturates after 32K as most requests do not exceed a 32K context size.
| Benchmark | 4K | 16K | 32K | 64K | 120K |
| DRB | 41.2% | 42.3% | 38.8% | 42.2% | 42.2% |
| SWE | 58.0% | 60.0% | 65.0% | 63.0% | 64.0% |
Table 1. Accuracy of whole-context summarization at different trigger lengths (Qwen3.6-35B-A3B, 50 DRB and 100 SWE-bench tasks).
On the other hand, the context of requests from DeepResearch Bench (DRB) is populated with results from web searches. These are verbose, and when given a context, summaries effectively keep only relevant information. We thus see a relatively stable accuracy trend on DRB while accuracy drops by up to 6 pp on SWE, despite having an uncompressed context size of 40-50K tokens. The distinction in data type, produced by tools, helps us identify suitable summarization targets. We use the term prose to refer to data that can be compressed without hurting accuracy, and non-prose for incompatible summarization targets. Aggron opts to selectively compress prose data when it gets large, while ensuring non-prose is kept verbatim.
Observation 2: Prose data are suitable targets to summarize, while non-prose data are not suitable for summarization.
What does compressing only prose buy? Compressing only prose shrinks each request less than compressing everything, yet the system still gains. On the coding-heavy production traces, where prose is a small share, the 4K cap lowers KV memory use by 1.35-1.57× (Table 2), and how much that helps depends on load. At low load, nothing queues, and shorter contexts make each step only slightly cheaper (1.02× mean latency). Near the knee (0.05-0.1 req/s), the smaller footprint keeps requests from queuing, and mean latency improves most (1.31-1.33×). At the highest rates, both arms queue; the smaller footprint admits more requests per batch, holding throughput 1.07× higher, and P99 improves 1.17×.
| (req/s) | Throughput | Mean latency | P99 latency | HBM use |
| 0.01 | 1.00× | 1.02× | 1.03× | 1.57× |
| 0.05 | 1.06× | 1.33× | 1.07× | 1.50× |
| 0.1 | 1.07× | 1.31× | 1.08× | 1.35× |
| 0.5 | 1.07× | 1.19× | 1.16× | 1.37× |
| 1 | 1.07× | 1.17× | 1.17× | 1.37× |
Table 2. Gain of the 4K prose cap over vanilla on production traces [85].
With synchronous summarization, if a request’s prose exceeds the cap, the request pauses, the same model summarizes the prose, and the request resumes. The summarization call sits on the critical path, time the request would otherwise spend on its task. In exchange, every later turn runs on a shorter context. The new context can also change the trajectory, so the agent may take more or fewer turns to finish. Figure 6a shows the distribution of the request latency normalized to its corresponding vanilla run. We see that the median effect is near parity. The CDF weights every program equally, so the many short programs, which rarely compact, hold the median at parity, while the long programs that dominate GPU time gain most. Figure 6b shows that from 0.05 req/s up, synchronous summarization raises throughput 1.19-1.45× over vanilla, from 1.05-1.5× larger mean batches. Asynchronous summarization matches it in throughput and gains little in latency. This runs counter to prior findings that asynchronous summarization outperforms synchronous [10, 36], which hold when summarization is a large share of latency, and when the background summary does not compete with other requests for the GPU. The reduced summarization latency is only a fraction of the stalled time it sees every turn, as seen in Figure 6c. It shows that once memory saturates, requests spend over 60% of their end-to-end time queued under every method, due to head-of-line stalls. With end-to-end latency determined by queue time, the summary’s own runtime is a small share of latency under either scheme, so hiding it buys little.

Figure 6. Synchronous and asynchronous summarization at the 4K prose cap on our benchmark traces (SWE-bench, TerminalBench, MCPMark). (a) Distribution of per-request latency normalized to vanilla. (b) Throughput and mean program latency over vanilla. (c) Share of latency spent queuing.
Observation 3: Under load, a summary’s cost is the queue wait it adds, not its own runtime.
Table 3 reports queue statistics under the 4K prose cap. At high loads, the request at the head of the queue cannot be admitted for 75-89% of the run. In about 5% of that blocked time, the head would have fit had it been slightly smaller. Compressing its prose tail, which frees ~ 2.4K tokens, would have admitted it. Although small as a share, this time is costly, because the entire queue stalls behind a single request. At , it amounts to 15 minutes of a 6-hour run with a mean queue of ~ 250 waiting requests. Admission also does not stop at the head. Once it is admitted, the next request becomes the head and is checked in turn. Had every waiting request’s tail been compressed, at least one more request would have been admitted in ~ 46% of the blocked time. This motivates compressing waiting requests before they reach the head of the queue.
| (req/s) | Head blocked (% of run) | Compressed tail admits (% of blocked) | admitted (% of blocked) |
| 0.05 | 8.7% | 4.8% | 12.8% |
| 0.1 | 74.8% | 4.5% | 44.0% |
| 0.2 | 83.0% | 4.9% | 46.0% |
| 0.5 | 88.7% | 5.0% | 46.5% |
| 1 | 86.2% | 5.0% | 46.2% |
Table 3. Queue statistics under the 4K prose cap.
Observation 4: Opportunistically compressing the prose tail end can admit the blocking head and another request.
Aggron: Aggressive Compression
Based on the previous observations, we propose Aggron, an agentic serving system that compresses agentic contexts aggressively and hides the resulting summarization latency inside queuing delay. Aggron is based on four main ideas.
Aggron compresses only the prose regions of a request’s context. Tool output carries structure as content, the identifiers the agent acts on next, while natural-language reasoning compresses freely (Observation 1, Observation 2). Aggron classifies every observation and makes only the prose a candidate for the compressor.
Aggron hides summarization latency in admission wait. Under load, a summary’s cost is the time its request spends waiting (Observation 3). A request returning from a tool call is not running but waiting for readmission, so Aggron issues the summarization into that wait, and the summary is ready when the request is admitted.
Aggron serves summarization requests from a separate queue. Small thresholds produce many summaries, and in the main queue they take batch slots from agentic requests (Observation 3). Aggron’s scheduler draws from the summarization queue only when no further agentic request can be admitted.
Aggron treats summarization as a server-issued mechanism. Compressing the prose tails of waiting requests can admit the blocked head and the request behind it (Observation 4), so Aggron decides when to compress from the queue state, not only from a request’s context length.
Design overview
Figure 7 provides an overview of the Aggron architecture and the path a request takes through it. A request enters the Session Registry ①, which holds the per-session state, namely the frozen summaries, the protected region, and the compressible accumulation . Requests returning from a tool call re-enter through the same point ⑦, where the Context Classifier partitions the new observations into protected and compressible. Prose increments by its token count, calculated from its transcript. A non-prose observation advances the protected region over everything before it, including any prose not yet compressed, and resets to zero; also resets when a summarization fires. The agentic request is forwarded to the queue ②. The session is also handed to the Controller ③, which reads the request’s position in the agentic queue ④ and the admission rate ⑥ from the admission control, and decides if a request’s tail should be compressed. When exceeds a threshold that falls the deeper the request sits in the queue (Section 4.3), and three gates confirm that the summary lands before the request reaches the head, shrinks the tail enough to be worth it, and frees at least one KV block (Section 4.4), the Controller enqueues a summarization for that session on the summarization queue ⑤. Once the summarization completes, the Registry replaces the compressed region of that request’s context in place, in the agentic queue; the request never left the queue and keeps its position. If the summarization has not landed when the request reaches the head, the request holds the head until it does. Admission Control draws from the agentic queue in default order, and from the summarization queue only when the agentic head does not fit in free KV space, and dispatches to the GPU ⑧. A preempted request returns to the head of the agentic queue with its KV discarded ⑨.
Figure 7. Aggron architecture
Identifying compressible context
Classification. Aggron classifies at the granularity of an observation, i.e. a single tool result in the transcript. For each observation, Aggron recovers the call that produced it by its tool_call_id and inspects the tool name and arguments. When the agent reads the same resource again, such as the same file, the new observation replaces the earlier one, so the context holds one copy. Aggron classifies as non-prose the output of three families of tools, (i) file-system and shell inspection (cat, grep, read_file and similar), (ii) database access (execute_sql, schema listings, query plans), and (iii) program and test execution (pytest, python). A SQL result set or a test log, with its pass/fail counts and tracebacks, is structured content in the same way a file read is. The model’s reasoning, the tool calls themselves, and web search and fetch results are prose and remain compressible.
The incompressible region. A non-prose observation protects not only itself but everything before it that has not yet been compressed. When one arrives, the protected region advances to cover the transcript up to and including it, so it stays a contiguous prefix and never separates a call from the result it explains. The price is that prose written before a file read or a test run is never summarized. Compressible prose is therefore exactly the prose that follows the most recent non-prose observation.
Triggering summarization
Counting just the prose. Aggron maintains a per-request counter of the tokens that summarization can actually reduce. Writing for the messages in the context, for a token count, and , , for the head, the frozen summaries, and the protected region, respectively,
is therefore not the context length. It excludes exactly the regions summarization cannot shrink, and counts only the prose tail, i.e. the prose that follows the most recent non-prose observation. A non-prose observation moves everything before it into and so returns to zero; so does a summarization.
Limiting the prose tail. Aggron keeps a hard backstop. When exceeds the default budget , summarization fires on the agent’s critical path, as in any summarization scheme. This upper limit guarantees that summarization does occur and that the compressible tail stays bounded. Firing only there, however, wastes the one resource the workload has in abundance, the queuing time. A request that has returned from a tool call sits in the waiting pool for a readmission delay that it pays whether or not anything runs on its behalf (§ 4.4), and a summarization issued into that window is hidden. Aggron therefore also compacts preemptively while a request waits, at a second and much lower threshold.
Queue depth-dependent thresholding. A request at the front of the queue will be admitted shortly and should be left alone; one at the back has time to spare and can be compacted hard. Aggron makes the threshold a function of position. Let be the request’s age rank among live programs ( oldest), its depth in the queue ( at the back), and its context-size rank ( largest), each normalized to . With a weight on the size term,
where and counts the program’s queue-time summaries since its last 4K-cap compaction, so each further one pulls back toward . Aggron compacts a waiting request when . The threshold runs from just under at the head of the queue down to at the back, so is an aggressiveness dial that the queue itself sets. Deep in the queue, Aggron compresses a tail far smaller than any critical-path scheme would consider, and at the head it does nothing. We set , , tokens, and , which weighs waiting and context size equally. Among requests equally deep in the queue, Aggron compacts the larger contexts first, since they return the most KV to the pool. The decision is taken afresh on every queue visit and at most once per visit, since both and the request’s position change while it waits.
Avoiding re-summarization. When the trigger fires, Aggron compresses the prose tail, appends the result to the context as a frozen segment, and resets to zero. It does not feed the standing summary back into the next compressor’s input. The distinction is a main reason Aggron can compress aggressively without losing accuracy. Under the conventional scheme, the summary is re-summarized at every summarization, so its information is distilled further every time. Freezing passes every token through the compressor exactly once and pays in linear context growth instead, still well below the growth of an uncompressed context. Freezing has a second, systems-side benefit by keeping the KV prefix valid for the whole episode. The conventional scheme rewrites the summary at every summarization and forces a full re-prefill of the context.
Upper-bounding the context. Linear growth reaches the model window eventually. Aggron therefore maintains a reset ceiling , as agent harnesses do [5, 47]. When the assembled context exceeds , Aggron summarizes everything, including the frozen segments, into a single segment and restarts. Aggron sets just below the max-model length to give a margin for the completion budget. The protected set is folded in at a reset, so the reset is the one point at which Aggron does compress non-prose; it is reached rarely, and the alternative is a request that cannot be served at all.
Masking summarization latency
Aggron’s observation is that under load an agentic request spends most of its lifetime not running; in fact, at high loads, 60–79% of end-to-end latency is spent waiting for admission (Figure 6c). Aggron spends it on summarization.
Issuing a summarization request. When a request returns from a tool call and its compressible accumulation exceeds , and passes a gate, Aggron enqueues the summarization immediately, rather than waiting for the agent to reach the next LLM call and discover that it must compact first. The summarization and the queuing delay then overlap. If the summary completes before the request reaches the head of the queue, the stall is fully hidden, and the agent observes none of it. If it does not, the request keeps its position and holds the head until the summary lands. Aggron does not admit it uncompressed, which would discard the summarization’s benefit. Holding the head delays this one request, but the requests behind it wait too, and they spend that time completing their own summarizations. The hideability gate below keeps this case rare.
Gating the summarization queue. Crossing makes a request a candidate, not a summarization. Aggron issues it only when three gates pass.
Gate 1. The summarization is hideable. Aggron predicts the summarization’s latency from the summarizations it has just served. Using the 90th percentile of the last 50 measured issue-to-landing times per summary token, which include their wait in the summarization queue, it scales by this summarization’s summary cap . Normalizing by length matters because summaries vary widely in size, and a single percentile under-predicts the long ones. A summarization that lands late raises the estimate, so later decisions learn from this. To handle cold starts, the first ten summarizations fall back to a model of the summarization’s own cost, (one prefill of the tail plus the decode of an -token summary), plus the backlog of the summarization queue ahead of it, drained at the recent summary admission rate. It estimates the request’s own wait as its position in the queue divided by the recent admission rate, and issues the summarization only if that wait covers the prediction.
Gate 2. The summarization is worth it. The summary is appended to the prompt permanently, so compressing a tail barely larger than its summary gains little. The summary’s size is not known until it is decoded, so Aggron bounds it by the summarizer’s output cap and issues a summary only if . The summary then holds at most of the tail it replaces. We set .
Gate 3. The summarization frees memory. The rewrite must release at least one KV block, since a summarization that shortens the context without returning a block does nothing for admission. Aggron requires , where is the request’s context and the block size. Only requests past their first call are candidates, since a first call has no prose tail yet. A request whose summarization is still in flight is skipped, and each request is decided once per visit to the queue.
Summarization requests occupy their own queue. Summarization requests and agentic requests have different shapes, especially since we compress only the small prose tail. A summarization is a single prefill with a short, capped decode and a small context, while an agentic turn is a moderate prefill with an open-ended decode over a larger context. Figure 4 shows that agentic requests saturate memory before compute, while single-turn requests, which summarization requests are, saturate compute with little memory pressure. However, a single queue lets summarization calls take KV and batch slots from agentic requests, which lengthens queuing, as seen in Figure 6c. Aggron therefore maintains a separate summarization queue. The agentic queue is the default queue. When the request at its head does not fit in free KV space, no agentic request is admitted. This is checked at every batch. That is exactly when the scheduler draws from the summarization queue, admitting summarization work into the KV and batch slots the blocked head cannot use, instead of leaving them idle. This is the point in the cycle at which summarization is most valuable, since the pool is full and a summarization is the only admission that can fit.
Experimental Setup
Hardware Infrastructure. We perform our accuracy analysis on a single NVIDIA H100 NVL GPU with 94 GB of HBM3. The model is served through vLLM with a 131,072-token context window, up to 256 concurrent sequences, and 92% of HBM given to the KV cache, yielding a resident KV pool of roughly 2.49M tokens in 1056-token blocks.
Models. We evaluate three models to separate the method from any single architecture. Qwen3.6-35B-A3B (FP8) [55] is the primary model and carries the headline result; GLM-4.7-Flash [18] and Gemma-4-26B-A4B [62] verify that the effect generalizes. The three span hybrid linear, latent, and grouped-query attention. We report headline serving results on Qwen, the model whose simulator we validate against live serving (Table 4).
| Throughput | P50 | P99 | Mean | |
| 0.05 | +4.3 | −16.3 | −3.0 | −14.5 |
| 0.2 | +1.9 | +2.1 | −3.4 | −0.6 |
Table 4. Simulated vs. live vanilla serving on one H100 (Qwen3.6-35B-A3B; 200 programs at =0.05, 500 at 0.2). Error (%) is (sim − live)/live against the mean of two live runs at the same .
Arms. Aggron compresses a prose tail once it holds between and tokens (Equation (2)), at a point that depends on load. The grow-4k and grow-1k arms fix that point at its two ends. The serving model summarizes the prose, and the summary is appended to the prompt and never rewritten, so the cached prefix survives. Non-prose tool output stays verbatim. An Aggron summary is the same summary issued while the request waits, so only its timing and threshold differ. The summarizer writes about 300–500 tokens at any threshold, keeping 10–20% of the tail it replaces under grow-1k and 5–10% under grow-4k, as one tool result often carries the tail well past its threshold. Accuracy cannot fall as a summary keeps more, so the two arms bound Aggron’s. Vanilla runs without compaction in the full context window. Slipstream and APEC compact the whole context at 60K, past the point where accuracy saturates (Table 1).
Noise. Agentic benchmark scores move by several points across identical runs [6, 20, 59, 79]. We therefore run vanilla twice, report the mean, and take the disagreement between the two runs as the noise band, , where is the number of tasks on which the runs disagree and the number of tasks scored. Each row is scored on the tasks that every run of that model completed.
Trajectory divergence. Summarization can change how an agent solves a task, not just whether it succeeds. Following prior work [38, 46, 71], we compare the ordered sequences of tool calls two runs make on the same task, using Levenshtein distance normalized by the longer sequence, from 0 (identical) to 1 (nothing shared). We complement it with Jaccard overlap of the sets of tools used, which ignores order. Because a rerun already strays, we report the median excess over two vanilla runs, . means summarization changes the trajectory no more than a rerun does.
Applications. We evaluate on four agentic benchmarks, each long enough for summarization to engage, covering code editing, raw shell, web research, and structured tool protocols. SWE-bench Verified [21] (100 stratified instances) covers repository-level bug fixing. The agent edits and tests inside a per-instance container and is scored by the official harness on whether its patch resolves the issue. Terminal-Bench [44] (49 tasks) covers open-ended terminal work driven by the benchmark’s own terminus agent, one container per task, scored by task-specific tests. DeepResearch Bench [14] (50 English queries) covers long-horizon web research against a self-hosted SearXNG instance, identical across arms, with reports graded by the RACE rubric judge. MCPMark [67] (66 tasks) covers tool use over the Model Context Protocol across filesystem and PostgreSQL services, run under the benchmark’s own orchestration agent and scored by its per-task verifiers.
Simulation Infrastructure. The accuracy harness runs agentic tasks sequentially, each a closed loop of model calls and tool executions, so it cannot exhibit the contention under concurrent load that our scheduler resolves. We therefore evaluate the system in a trace-driven discrete-event simulator that replays recorded agentic trajectories under a synthetic arrival process. Replaying recorded trajectories is valid because each arm replays its own trajectories from the accuracy runs, and summarization leaves them within run-to-run variation (§ 6.2).
We follow Vidur [2], calibrating a few constants on one GPU and simulating serving at scale. We build our own simulator because the unit we serve is an agentic program, not an independent request. A program issues its next call only after the previous one finishes and its tool returns, so a slow server delays its later arrivals, and its KV persists across calls, so each call’s prefix hit depends on what was evicted while its tool ran. The simulator replays recorded trajectories (per-call context length, generated tokens, and tool time) through a vLLM-style continuous-batching engine over a paged KV-cache block pool, with prefix caching and CPU KV offload [40], following the same design as recent agent-serving simulators [16, 25, 56]. Each serving run replays 1,000 programs drawn from all four benchmarks, arriving as a Poisson process at rate programs per second with a fixed seed, so every arm sees the same arrivals. Live runs load Aggron into unmodified vLLM 0.24 as a subclass of its scheduler, and the simulator runs the same controller code. Section 6.4 runs the same simulator on four frontier models, T1 (a fine-tuned Qwen3.5-122B [54], TP=8), DeepSeek-V4-Flash [13] (TP=8), GLM-5.3 [19] (TP=16) and Kimi K3 [26] (TP=32), all on H100s. Every model replays long-horizon traces, from T1 on LH-TB [33], DSV4 on Terminal-Bench 2.1 [44], and GLM-5.3 and K3 on Terminal-Bench Science [63]. We fit each model’s step costs to the measured H100 vLLM tables of NVIDIA AIConfiguraror [70] (3.5–9.0% MAPE), and the baselines cap each program’s context at 120K tokens, near the 131K window of our vanilla arm, and at 24K, within the budgets of Belcak et al. [7], summarizing in-band and restarting at the cap.
Again measurements, the simulator predicts decode step time within 7.9% and prefill latency within 9.4% mean absolute error. We
validate the simulator end-to-end on vanilla, replaying the same programs live via vLLM on a single H100 and comparing each run with its simulated twin (Table 4). Every metric lies within 3.4% at . At , the box never queues, so latency tracks host speed. Throughput and P99 lie within 4.3%, while the median and mean miss by up to 16%, as host speed varied by 21% between the two live runs. Aggron extends this validated engine with its controller, running the same controller code as the live scheduler, and Section 6.1 reports its simulated values.
Evaluation
Performance analysis against SOTA
Figure 8a plots the P99 latency and throughput for requests arriving at different rates drawn from a Poisson distribution on the Qwen model. We compare our results to SOTA asynchronous summarization techniques Slipstream [10] and APEC [36]. At high loads (), Aggron sustains 1.44× the throughput of vanilla, 1.40× APEC, and 1.34× Slipstream, and cuts P99 latency by 33%, 31% and 27% respectively. This is because its smaller per-request KV footprint raises batch occupancy from 26% to 39% of the 256-sequence limit, as seen in Figure 8b. It also shows the distribution of context size when an inference is made. This captures the variation in context size over the duration of the run and the summarization achieved across the methods. We see that Aggron has a larger share at smaller sizes, as the summarization of Aggron operates at smaller sizes. Once the thresholds for Slipstream and APEC are reached (60-90K), those methods start compressing, and their distributions catch up with Aggron’s.
Figure 8. (a) P99 latency and throughput on Qwen3.6-35B-A3B. (b) Batch occupancy at different rates, and the distribution of context size per inference call. (c) Latency distribution and (d) latency breakdown at 0.35×, 0.7× and 1.4× the saturation rate (≈ 0.047 req/s). (e) Slowdown against program length, normalized to isolated runs, and per-benchmark P99 latency, at 0.7× the saturation rate. (f) Compression statistics.
To understand the trend in P99, Figure 8c plots the distribution of the request latency. At lower rates, when inference dominates the latency, we get similar latencies across the methods. As the rate increases, the batch size increases by , and inference gets slower. At higher rates, when memory saturates, the queue dominates latency. Figure 8d plots the latency breakdown. Here Aggron’s benefit comes from two factors. First, the improved batch occupancy drains the queue faster, so every request spends less time waiting. Second, agents do not contend for resources with summary requests. We examine this further in § 6.3.
Aggron’s gain grows with program length; we examine this in Figure 8e. In the longest bin, its median slowdown is , against for vanilla; in the middle bin it is against . Slowdown here is measured against isolated vanilla. Each turn a program takes is cheaper to prefill and decode and frees KV sooner, so the benefit accumulates over long programs. The freed memory raises batch occupancy, which shortens the admission queue for every program. Short programs rarely compact, yet their median slowdown still falls from to , because they wait in the same admission queue. Figure 8e also shows the per-benchmark P99 latency. SWE benefits most ( lower P99 than vanilla) as it spends the largest share of its time in the model (59%) and the cap removes the most context (74%). TB and MCPMark spend most of their time in tool execution, which summarization cannot shorten, so they gain only and . DRB is tool-bound as well, and Aggron matches Slipstream there (0.42 h).
From the discussion so far, Aggron’s gain comes from its smaller per-turn context; Figure 8f shows that Aggron issues summarization at smaller thresholds. The smaller size of summarizations produces faster inferences and demands a smaller share of the KV cache. Aggron’s largest summary is still about smaller than the smallest async summarization. At the median, the gap is - in input size and - in prefill time. So Aggron makes about more summarization calls, but each one is small enough to hide in queuing delay. The async systems make few, large ones. Asynchronous methods can also operate at Aggron’s thresholds but with accuracy degradation, as they compress prose and non-prose alike (Table 1, 4K). Even if they compress only prose, an asynchronous compactor lets the agent continue when the summary has not landed, which Figure 6c shows lengthens head-of-line stalls. A synchronous 4K prose cap reaches the same throughput on these traces, since each 4K summary is small next to the queue wait (Figure 6c); Aggron’s queue-time issue keeps that true as thresholds fall below 4K and summaries grow costlier on the frontier models of Section 6.4.
Accuracy comparisons
Table 5 reports accuracy for the three arms across three models and four benchmarks. Accuracy rises in some cells and falls in others, but every difference lies within the run-to-run noise band, so both compressed arms are as accurate as vanilla. Since grow-4k and grow-1k sit at the two ends of Aggron’s threshold range, we expect Aggron’s accuracy to lie between them.
| Model | Benchmark | grow-1k | grow-4k | vanilla | Noise (±) | ||||
| Qwen3.6-35B-A3B | SWE | 67.7 | 66.3 | 68.1 | −1.1 | −1.1 | 6.7 | +0.11 | 0.00 |
| TerminalBench | 55.0 | 57.5 | 50.0 | +5.0 | +7.5 | 12.0 | 0.00 | +0.04 | |
| MCPMark | 47.0 | 42.4 | 39.4 | +7.6 | +3.0 | 7.9 | +0.08 | 0.00 | |
| DeepResearchBench | 42.91 | 43.46 | 43.55 | −0.72 | −0.10 | 1.9 | +0.03 | 0.00 | |
| GLM-4.7-Flash | SWE | 55.0 | 57.5 | 56.2 | −1.2 | +1.2 | 9.2 | +0.07 | +0.04 |
| TerminalBench | 26.7 | 36.7 | 31.7 | −5.0 | +5.0 | 19.6 | +0.01 | 0.00 | |
| MCPMark | 34.9 | 34.9 | 35.7 | −0.8 | −0.8 | 8.2 | −0.04 | −0.04 | |
| DeepResearchBench | 41.16 | 42.27 | 41.12 | +0.04 | +1.15 | 1.9 | +0.05 | +0.16 | |
| Gemma-4-26B-A4B | SWE | 62.7 | 62.7 | 65.3 | −2.7 | −2.7 | 7.4 | 0.00 | 0.00 |
| TerminalBench | 39.4 | 48.5 | 39.4 | 0.0 | +9.1 | 16.8 | −0.02 | 0.00 | |
| MCPMark | 45.5 | 45.5 | 40.2 | +5.3 | +5.3 | 6.6 | 0.00 | 0.00 | |
| DeepResearchBench | 42.35 | 40.63 | 40.55 | +1.80 | +0.08 | 3.0 | +0.07 | +0.11 |
Table 5. Task accuracy and trajectory divergence under compaction (§ 5).
The noise band varies widely across benchmarks, and is widest on Terminal-Bench (+12-20 pp, against +7-9 pp on SWE-bench). The band shrinks with the number of tasks and grows with the number on which two identical runs disagree, and Terminal-Bench is worst on both. It has the fewest tasks (30-40 scored), and many of its tasks are long, open-ended shell sessions whose outcome flips between identical runs. A single early mistake, such as a failed install or a wrong path, can derail the rest of the session. Its differences are therefore the least resolved in the table, not evidence of a larger effect.
Summarization also leaves trajectories within run-to-run variation in 22 of 24 arm-benchmark pairs. For each pair, a sign test counts the tasks on which summarization moves the trajectory further from vanilla than a second vanilla run does, and in 22 pairs that count is not significant once corrected for running 24 tests, and the Jaccard excess is at most 0.08 in every cell. The two exceptions are SWE grow-1k on Qwen () and GLM (+0.07). These agents make - as many tool calls as vanilla because they re-check their state after each summary. The extra calls are insertions, which raise , while the order of tools is unchanged. The largest raw excess, DRB grow-4k on GLM (+0.16) and Gemma (+0.11), is not significant. DRB’s tool names include the domain each search fetches, so citing a different source counts as a substitution, and its null is itself wide.
Ablation studies
We first isolate the queue design, comparing Aggron’s separate summary queue against a shared queue in which summaries wait in the main queue alongside agent calls (Figure 9a). Both fire the same tail-only summaries, so only the summary waits differ. Once summaries fire (), the separate queue raises throughput by 1.9-8.0% and lowers P99 by 3.8-8.1% on Qwen, by 1.1-1.4% and 1.2-1.4% on GLM, and by 0.1-0.8% and 0.4-0.5% on Gemma. Each stalled summary blocks the head for 8-14 s on all three models, but Qwen fits batches of 68-77 requests and finishes in a quarter to a third of the time GLM and Gemma take at batches of 11-19, so the same stalls fill more of its run. The difference lies in summary latency (Figure 9b). In the shared queue, a summary waits behind the agent calls ahead of it. Head-of-line stalls make up 82-98% of its latency; its median latency reaches 40-280 s, and none land before their call reaches the head of the queue, which then waits on it for 8-40% of the run. Aggron admits its summaries almost immediately, so they land in 7-33 s and 93-100% arrive in time. When summaries can be hidden, a separate queuing strategy improves performance.

Figure 9. (a) Throughput and P99 of the separate vs. shared summary queue. (b) P50 summary latency and queuing share of summary latency. (c) Throughput and P99 when the 4K cap’s own summaries move to the separate queue, against keeping them in the main queue. (d) Gate ablations, normalized to the default setting.
Next, we measure the cost of raising summarization requests that cannot be hidden. Since we force the agentic request to stall at the head when its summary is in flight, performance can degrade if the summary does not complete before the agentic request is served. This can arise due to insufficient capacity for summaries to run, stalling the summarization queue. Figure 9c masks the controller and moves the 4K cap’s own summarizations, with no early summaries, from the main queue into the separate one. The head then waits on its own summary for 12-49% of the run, so the separate queue loses 0.2–3.8% throughput and adds 1.3–47% to P99 relative to keeping the cap synchronous. The loss peaks at light load (), where the queue is too short to hide a summary. This is why Aggron raises a summary only when gate 1 predicts it can be hidden. On the frontier models of Section 6.4, gate 1 passes 89–99% of candidate summaries, over 99.5% of them land in time, and the head waits on its own summary for under 0.3% of the run.
Predicting if the summarization can be hidden is the first gate of the controller. The other two gates check if the summarization is fruitful. Gate 2 requires the tail to be at least times the summary cap, and gate 3 requires the summary to free at least one KV block. Figure 9d sweeps both, normalized to the default setting (, 1056-token blocks). Summaries are typically smaller than their tails, so neither gate changes which summaries fire at the default setting. Removing gate 2 () leaves throughput and P99 unchanged, and gate 3 declines at most one check. Tightening gate 2 to drops 63–66% of summaries but moves throughput and P99 by under 0.4%. Gate 3 starts declining only at 4K-token blocks, where one block is as large as the tail. Across 16-token to 4K-token blocks, throughput moves by at most 0.65% and P99 by at most 2.6%, mainly because block size sets allocation granularity for every request.
Figure 10. Aggron on four frontier models (shared trace mix, ). (a) and (b) compare each arm with the cap it runs under. The side summarizer uses one more GPU. Labels under each model give its KV per token per GPU.
Scaling Aggron
Under the 120K cap (Figure 10a), Aggron raises throughput by 5-14% on T1, K3 and GLM-5.3 and cuts P99 latency by 7-13%. Its summaries cut cap restarts from 1,468 to at most 226, and the memory they free admits more requests. The gain grows with KV per token, from +5% on T1 (12 KiB) to +14% on GLM-5.3 (59 KiB). DSV4 (3.4 KiB) fills only 68% of HBM uncompressed, so freed memory buys nothing while summary prefill takes 8% of serving-GPU time (1.6-3.2% elsewhere), and it loses 1.7%. If each summary reuses the cached KV of the prose it compresses and recomputes 15% of it [74], this share falls to 0.3-1.5%, and throughput rises on all four models by 5-16%, with DSV4 at +5.3%, and P99 falls by 5-14%.
Under the 24K cap (Figure 10b), Aggron halves the baseline’s 49,421 restarts, which offsets its summary prefill, and raises throughput by 3-14% and cuts P99 by 4-12%, or 10-17% and 10-15% with tail reuse. Moving every summary to a 35B side model frees the serving GPU but cannot reuse its KV. At 120K it gains 8-18%, only 2-4 points more than tail reuse, with a second GPU. At 24K the side GPU runs 83% busy. It prefills about 13K tokens/s, faster than GLM-5.3 and K3 (+40-41%) but a third of DSV4’s, where it loses 26% as requests wait on their own summary for 81% of the run. Routing each summarization request dynamically to the less loaded of the two is future work. Gains trail § 6.1, where Qwen’s programs fit its 131K window and vanilla never compacts. Here most programs outgrow the 120K cap (median peak 163K), so the baseline compacts as much as Aggron or more.
On four frontier models, Aggron raises throughput by up to 14% over either a 120K or a 24K cap, and by 5-17% on all four when summaries reuse the cached KV of the prose they replace.
Related Work
Context summarization. Most work on agent context summarization either trains the model to summarize better or decides when to summarize. ACON [23] optimizes the summarization guideline from failed trajectories; ReSum [66], MEM1 [83], AgentFold [76] and CompactionRL [31] train agents to fold or rewrite their own history with supervised or reinforcement learning; SelfCompact [30] lets the model choose when to compact, and MemGPT [49] pages context between the window and external memory. These methods summarize the whole trajectory and rewrite it in place, typically near the context limit, and none of them considers the serving system. The Complexity Trap [35] shows that simply masking old observations rivals LLM summarization. The agents in our traces, however, revisit old tool outputs (§ 3), so Aggron keeps observations and compresses the prose around them. Belcak et al. [7] ask how much context an agent needs, and show that frontier agents that compact their full history at budgets as small as 16K tokens and restart from the summary lose little to no accuracy on SWE-bench and Terminal-Bench. Their agent compacts in-band at a fixed threshold and re-prefills the restarted context, and serving cost is measured per token on isolated fixed-length requests. Aggron instead asks how to serve summarization on a loaded, shared server. The server decides when to compress from queue state, summarizes only prose while the request waits for admission, keeps tool outputs verbatim, and freezes each summary into an append-only prefix so the cached KV survives.
KV-cache eviction. Eviction drops the tokens the model is predicted not to need (H2O [81], Scissorhands [42], StreamingLLM [69], SnapKV [32], PyramidInfer [72], and recent work adapts eviction and attention matching to agents [39, 41, 86]). Eviction is lossy for every token, including tool outputs, and its memory savings do not reliably translate into serving throughput [8]. It is orthogonal to Aggron and can be combined with it.
Asynchronous summarization. Several systems move summarization off the agent’s critical path. APEC [36] summarizes a snapshot in the background once the context passes a soft threshold and installs it at the next request; Slipstream [10] runs the compactor beside the agent and validates the summary with a judge; CoMem [80] overlaps a trained memory model with agent inference; Parallel Context Compaction [11] splits one compaction across parallel requests; and sleep-time compute [34] uses user idle time. All of them hide compaction behind a single agent’s own execution and are evaluated on a dedicated server. Under load, their summaries compete with other requests for the same KV and batch slots. Aggron instead hides compaction inside the queueing delay that load creates, and runs it only in capacity the waiting requests cannot use.
Serving schedulers. LLM serving builds on continuous batching [77], paged KV memory [27] and chunked prefill [3]. Schedulers fight head-of-line blocking by preemption [65], learned shortest-job-first ordering [17], or queue reordering and eviction [51]. Agent-aware systems retain or move KV across tool calls. InferCept [1] @page 11/14 starts_mid=0 ends_mid=1
Figure 10 (PDF p. 11). Aggron on four frontier models (shared trace mix, ). (a) and (b) compare each arm with the cap it runs under. The side summarizer uses one more GPU. Labels under each model give its KV per token per GPU.
Scaling Aggron
Under the 120K cap (Figure 10a), Aggron raises throughput by 5-14% on T1, K3 and GLM-5.3 and cuts P99 latency by 7-13%. Its summaries cut cap restarts from 1,468 to at most 226, and the memory they free admits more requests. The gain grows with KV per token, from +5% on T1 (12 KiB) to +14% on GLM-5.3 (59 KiB). DSV4 (3.4 KiB) fills only 68% of HBM uncompressed, so freed memory buys nothing while summary prefill takes 8% of serving-GPU time (1.6-3.2% elsewhere), and it loses 1.7%. If each summary reuses the cached KV of the prose it compresses and recomputes 15% of it [74], this share falls to 0.3-1.5%, and throughput rises on all four models by 5-16%, with DSV4 at +5.3%, and P99 falls by 5-14%.
Under the 24K cap (Figure 10b), Aggron halves the baseline’s 49,421 restarts, which offsets its summary prefill, and raises throughput by 3-14% and cuts P99 by 4-12%, or 10-17% and 10-15% with tail reuse. Moving every summary to a 35B side model frees the serving GPU but cannot reuse its KV. At 120K it gains 8-18%, only 2-4 points more than tail reuse, with a second GPU. At 24K the side GPU runs 83% busy. It prefills about 13K tokens/s, faster than GLM-5.3 and K3 (+40-41%) but a third of DSV4’s, where it loses 26% as requests wait on their own summary for 81% of the run. Routing each summarization request dynamically to the less loaded of the two is future work. Gains trail § 6.1, where Qwen’s programs fit its 131K window and vanilla never compacts. Here most programs outgrow the 120K cap (median peak 163K), so the baseline compacts as much as Aggron or more.
On four frontier models, Aggron raises throughput by up to 14% over either a 120K or a 24K cap, and by 5-17% on all four when summaries reuse the cached KV of the prose they replace.
Related Work
Context summarization. Most work on agent context summarization either trains the model to summarize better or decides when to summarize. ACON [23] optimizes the summarization guideline from failed trajectories; ReSum [66], MEM1 [83], AgentFold [76] and CompactionRL [31] train agents to fold or rewrite their own history with supervised or reinforcement learning; SelfCompact [30] lets the model choose when to compact, and MemGPT [49] pages context between the window and external memory. These methods summarize the whole trajectory and rewrite it in place, typically near the context limit, and none of them considers the serving system. The Complexity Trap [35] shows that simply masking old observations rivals LLM summarization. The agents in our traces, however, revisit old tool outputs (§ 3), so Aggron keeps observations and compresses the prose around them. Belcak et al. [7] ask how much context an agent needs, and show that frontier agents that compact their full history at budgets as small as 16K tokens and restart from the summary lose little to no accuracy on SWE-bench and Terminal-Bench. Their agent compacts in-band at a fixed threshold and re-prefills the restarted context, and serving cost is measured per token on isolated fixed-length requests. Aggron instead asks how to serve summarization on a loaded, shared server. The server decides when to compress from queue state, summarizes only prose while the request waits for admission, keeps tool outputs verbatim, and freezes each summary into an append-only prefix so the cached KV survives.
KV-cache eviction. Eviction drops the tokens the model is predicted not to need (H2O [81], Scissorhands [42], StreamingLLM [69], SnapKV [32], PyramidInfer [72], and recent work adapts eviction and attention matching to agents [39, 41, 86]). Eviction is lossy for every token, including tool outputs, and its memory savings do not reliably translate into serving throughput [8]. It is orthogonal to Aggron and can be combined with it.
Asynchronous summarization. Several systems move summarization off the agent’s critical path. APEC [36] summarizes a snapshot in the background once the context passes a soft threshold and installs it at the next request; Slipstream [10] runs the compactor beside the agent and validates the summary with a judge; CoMem [80] overlaps a trained memory model with agent inference; Parallel Context Compaction [11] splits one compaction across parallel requests; and sleep-time compute [34] uses user idle time. All of them hide compaction behind a single agent’s own execution and are evaluated on a dedicated server. Under load, their summaries compete with other requests for the same KV and batch slots. Aggron instead hides compaction inside the queueing delay that load creates, and runs it only in capacity the waiting requests cannot use.
Serving schedulers. LLM serving builds on continuous batching [77], paged KV memory [27] and chunked prefill [3]. Schedulers fight head-of-line blocking by preemption [65], learned shortest-job-first ordering [17], or queue reordering and eviction [51]. Agent-aware systems retain or move KV across tool calls. InferCept [1] preserves, discards or swaps it; Continuum [29] pins it with a time-to-live; MORI [68] offloads idle programs; and Autellix [43], Pythia [78] and ThunderAgent [22] schedule at the program level. These systems reorder, move or retain KV, but none reduces the KV a waiting request will need. Aggron keeps vLLM’s FCFS order and shrinks the waiting requests themselves.
Conclusions
To maximize the cost-effectiveness of summarization, this paper presented a novel serving system called Aggron. Compared to the Slipstream [10] and APEC [36] state-of-the-art asynchronous summarization schemes, Aggron reduces P99 latency by 27% and 31%, reduces mean latency by 11% and 15%, and increases throughput by and , respectively. Aggron’s accuracy stays within run-to-run noise of uncompressed execution. On four frontier models, it raises throughput by up to 14% over either a 120K or a 24K cap, and by 5-17% on all four when summaries reuse the cached KV of the prose they replace.
References
References
- [1]Reyna Abhyankar, Zijian He, Vikranth Srivatsa, Hao Zhang, and Yiying Zhang. 2024. InferCept: Efficient Intercept Support for Augmented Large Language Model Inference. In Proceedings of the 41st International Conference on Machine Learning (ICML ’24, Vol. 235). PMLR, Vienna, Austria, 81–95. https://proceedings.mlr.press/v235/abhayankar24.html
- [2]Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav S. Gulavani, Ramachandran Ramjee, and Alexey Tumanov. 2024. Vidur: A Large-Scale Simulation Framework for LLM Inference. In Proceedings of Machine Learning and Systems (MLSys ’24, Vol. 6). mlsys.org, Santa Clara, CA, USA, 351–366. https://proceedings.mlsys.org/paper_files/paper/2024/hash/b74a8de47d2b3c928360e0a011f48351-Abstract-Conference.htmlarxiv.org/abs/2405.05465
- [3]Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, and Ramachandran Ramjee. 2023. SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. arXiv:2308.16369arxiv.org/abs/2308.16369
- [4]Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yuriy Zemlyansky, Federico Lebrón, and Sumit Sanghavi. 2023. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP ’23). Association for Computational Linguistics, Stroudsburg, PA, USA, 4895–4901. doi:10.18653/v1/2023.emnlp-main.298
- [5]Anthropic. 2025. Claude Code. https://www.anthropic.com/claude-code. Accessed October 2026.
- [6]Berk Atıl, Sarp Ayıkent, Alexa Chittams, Liseng Fu, Rebecca J. Passonneau, Evan Radcliffe, Guru Rajan Rajagopal, Adam Sloan, Tomasz Trudrej, Ferhan Ture, Zhe Wu, Lixinyu Xu, and Breck Baldwin. 2025. Non-Determinism of “Deterministic” LLM System Settings in Hosted Environments. In Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP ’25). Association for Computational Linguistics, Stroudsburg, PA, USA, 135–148. doi:10.18653/v1/2025.eval4nlp-1.12
- [7]Peter Belcak, Saurav Muralidharan, Jindong Jiang, Yonggan Fu, Wonmin Byeon, Hanrong Ye, Ryan Angilly, Ali Taghibakhshi, Shih-Yang Liu, Siru Ouyang, Greg Heinrich, and Pavlo Molchanov. 2024. You Don’t Need Long Context. arXiv:2611.05306 [cs.AI]
- [8]Alex Chen, Renato Geh, Aditya Grover, Guy Van den Broeck, and Daniel Israel. 2026. The Pitfalls of KV Cache Compression. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL ’26, Vol. 1). Association for Computational Linguistics, Stroudsburg, PA, USA, 41530–41553. doi:10.18653/v1/2026.acl-long.1926DOI
- [9]Yilong Chen, Guoxia Wang, Junyuan Shang, Shiyao Cui, Zhenyu Zhang, Tingwen Liu, Shouhuan Wang, Yu Sun, Dianhai Yu, and Hua Wu. 2024. NACL: A General and Effective KV Cache Eviction Framework for LLM at Inference Time. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL ’24, Vol. 1). Association for Computational Linguistics, Stroudsburg, PA, USA, 7913–7926. doi:10.18653/v1/2024.acl-long.428DOI
- [10]Zhuofu Chen, Rui Pan, Yiwei Dai, and Ravi Netravali. 2026. Slip-stream: Trajectory-Grounded Compaction Validation for Long-Horizon Agents. arXiv:2605.08580 https://arxiv.org/abs/2605.08580
- [11]Musa Cim, Burak Topcu, Chita Das, and Mahmut Taylan Kandemir. 2026. Parallel Context Compaction for Long-Horizon LLM Agent Serving. arXiv:2605.23296 https://arxiv.org/abs/2605.23296
- [12]DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. arXiv:2412.19437arxiv.org/abs/2412.19437
- [13]DeepSeek-AI. 2026. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348 [cs.CL]
- [14]Mingxuan Du, Benfang Xu, Chiwei Zhu, Licheng Zhang, Xiaorui Wang, and Zhendong Mao. 2026. DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents. In International Conference on Learning Representations (ICLR ’26). OpenReview.net, Rio de Janeiro, Brazil, 42414–42448. https://proceedings.iclr.cc/paper_files/paper/2026/hash/465f22be10e07b301c6ed58f0472f704-Abstract-Conference.htmlarxiv.org/abs/2506.11763
- [15]Exgentic. 2026. Agent LLM Traces. https://huggingface.co/datasets/Exgentic/agent-llm-traces. CDLA-Permissive-2.0, accessed 2026-09-11.
- [16]Yicheng Feng, Xin Tan, Yangtao Deng, Yimin Jiang, Yibo Zhu, and Hong Xu. 2026. Frontier: Towards Comprehensive and Accurate LLM Inference Simulation. arXiv:2605.21312 [cs.DC]arxiv.org/abs/2605.21312
- [17]Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, Ion Stoica, and Hao Zhang. 2024. Efficient LLM Scheduling by Learning to Rank. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024) (NeurIPS ’24, Vol. 37). Curran Associates, Inc., Red Hook, NY, USA, 59006–59029. doi:10.5555/20207917-1882arxiv.org/abs/2408.15792
- [18]GLM-4.5 Team. 2025. GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models. arXiv:2508.06471 [cs.CL] https://arxiv.org/abs/2508.06471
- [19]GLM-5 Team. 2026. GLM-5: From Vibe Coding to Agentic Engineering. arXiv:2602.15763 [cs.LG]
- [20]Aayush Gupta. 2026. ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions. arXiv:2601.06112arxiv.org/abs/2601.06112
- [21]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? In The Twelfth International Conference on Learning Representations (ICLR ’24). OpenReview.net, Vienna, Austria, 54107–54157. https://proceedings.iclr.cc/paper_files/paper/2024/hash/edac78c3e300629acfe6be9ca88fb84-Abstract-Conference.html
- [22]Hao Kang, Ziyang Li, Weili Xu, Xinyu Yang, Yinfang Chen, Junxiong Wang, Beidi Chen, Tushar Krishna, Chenfeng Xu, and Simran Arora. 2026. ThunderAgent: A Fast, Simple, and Program-Aware Agentic Inference System. In Proceedings of the 43rd International Conference on Machine Learning (ICML ’26, Vol. 306). PMLR, Seoul, South Korea, 55630–55657. https://proceedings.mlr.press/v306/kang26j.html
- [23]Minki Kang, Wei-Ning Chen, Dongge Han, Huseyin A. Inan, Lukas Wutschitz, Yanzhi Chen, Robert Sim, and Saravan Rajmohan. 2026. ACON: Optimizing Context Compression for Long-Horizon LLM Agents. In Proceedings of the 43rd International Conference on Machine Learning (ICML ’26, Vol. 306). PMLR, Seoul, South Korea, 55321–55375. https://proceedings.mlr.press/v306/kang26b.htmlarxiv.org/abs/2510.00615
- [24]Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. In Proceedings of the 37th International Conference on Machine Learning (ICML ’20, Vol. 119). PMLR, Virtual Event, 5156–5165. https://proceedings.mlr.press/v119/katharopoulos20a.html
- [25]Donghwan Kim, Prakhar Singh, Younghoon Min, Jongryool Kim, Jongseok Park, and Kiwan Maeng. 2026. Characterizing How Complex Agentic AI Systems Handle General Tasks: A Trace-Based Simulation Study. In 2026 IEEE International Symposium on Workload Characterization (IISWC) (IISWC ’26). IEEE, Piscataway, NJ, USA, 72–86. doi:10.1109/IISWC71020.2026.00018DOI
- [26]Kimi Team. 2026. Kimi K2: Open Frontier Intelligence. https://github.com/MoonshotAI/Kimi-K3. Technical report.
- [27]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23). Association for Computing Machinery, New York, NY, USA, 611–626. doi:10.1145/3600006.3613165
- [28]Wonbeom Lee, Junyoung Lee, Junghwan Seo, and Jaewoong Sim. 2024. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) (OSDI ’24). USENIX Association, Berkeley, CA, USA, 155–172. https://www.usenix.org/conference/osdi24/presentation/leearxiv.org/abs/2406.19707
- [29]Hanchen Li, Runyuan He, Qiuyang Mang, Qizheng Zhang, Huanzhi Mao, Xiaokun Chen, Hangrui Zhou, Huancheng Zhang, Alvin Cheung, Joseph Gonzalez, and Ion Stoica. 2026. Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live. In ICLR 2026 Workshop on Lifelong Agents: Learning, Aligning, Evolving. https://openreview.net/forum?id=sVzK0LC9pn
- [30]Tianjian Li, Jingyu Zhang, William Jurayj, Xi Wang, Chuanyang Jin, Mehrdad Farajtabar, Eric Nalisnick, and Daniel Khashabi. 2026. Self-Compacting Language Model Agents. To appear in Advances in Neural Information Processing Systems (NeurIPS ’26). https://neurips.cc/virtual/2026/poster/155897
- [31]Yujiang Li, Zhenyu Hou, Yi Jing, Jie Tang, and Yuxiao Dong. 2026. CompressionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents. arXiv:2607.05378 https://arxiv.org/abs/2607.05378
- [32]Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. SnapKV: LLM Knows What You are Looking for Before Generation. In Advances in Neural Information Processing Systems (NeurIPS ’24, Vol. 37). Curran Associates, Inc., Red Hook, NY, USA, 29247–29702. doi:10.1109/ICML55092/079017-0722
- [33]Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Zhichao Liu, Xiyang Wu, Anhao Li, Chengsong Huang, Ergan Shang, Yue Yu, Ninghao Liu, Lichao Sun, Haotao Mi, and Leowei Liang. 2026. Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading. arXiv:2607.08964 [cs.AI]arxiv.org/abs/2607.08964
- [34]Kevin Lin, Charlie Snell, Yu Wang, Charles Packer, Sarah Wooders, Ion Stoica, and Joseph E. Gonzalez. 2025. Sleep-time Compute: Beyond Inference Scaling at Test-time. arXiv:2504.13171 https://arxiv.org/abs/2504.13171
- [35]Tobias Lindenbauer, Igor Slinko, Ludwig Felder, Egor Bogomolov, and Yaroslav Zharov. 2025. The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management. In Fourth Workshop on Deep Learning for Code (DL4C) at NeurIPS 2025. https://arxiv.org/abs/2508.21433
- [36]Banruo Liu, Haoran Qiu, Íñigo Goiri, Jingran Zhou, Rodrigo Fonseca, Ricardo Bianchini, Fan Lai, and Esha Choukse. 2026. Benchmarking Long-Horizon Agent Context Compaction with CompletionBench. doi:10.20944/preprints202611.0780. v1DOI
- [37]Yinyi Liu, Liangzhi Li, Tong Xiang, Bowen Wang, and Yiming Qian. 2023. TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction. In Findings of the Association for Computational Linguistics: EMNLP 2023 (Findings of EMNLP ’23). Association for Computational Linguistics, Stroudsburg, PA, USA, 9796–9810. doi:10.18653/v1/2023.findings-emnlp.655
- [38]Jiayu Liu, Cheng Qian, Zhaochhen Su, Qing Zong, Shijue Huang, Bingxiang He, and Yi R. (May) Fung. 2026. CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL ’26). Association for Computational Linguistics, Stroudsburg, PA, USA, 12826–12858. doi:10.18653/v1/2026.acl-long.584DOI
- [39]Taowen Tony Liu, Jeffrey T. H. Wong, Can Xiao, Bowen Yang, Hao Mark Chen, and Yiren Zhao. 2026. AgentKV: Phase-Aware KV Eviction for Agentic LLMs. arXiv:2609.14872 https://arxiv.org/abs/2609.14872
- [40]Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Sahel Shen, Rui Zhang, Kuntai Du, and Junchen Jiang. 2025. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference. arXiv:2510.09665 doi:10.48550/arXiv.2510.09665DOI
- [41]Yujian Liu, Jiabao Ji, Li An, Rohit Jain, Gunger Polatkan, Siyu Zhu, and Shiyu Chang. 2026. Practical Online KV Cache Compaction for LLM Agents: An Empirical Study. arXiv:2608.00902 https://arxiv.org/abs/2608.00902
- [42]Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhou Xu, Anastaios Kyrillidis, and Anshumali Shrivastava. 2023. Scissorchands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time. In Advances in Neural Information Processing Systems (NeurIPS ’23, Vol. 36). Curran Associates, Inc., Red Hook, NY, USA, 52342–52364. doi:10.52202/1925-082279
- [43]Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang, Yanping Huang, Zhifeng Chen, Joseph E. Gonzalez, and Ion Stoica. 2025. Autellix: An Efficient Serving Engine for LLM Agents as General Programs. arXiv:2502.13965 https://arxiv.org/abs/2502.13965
- [44]Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, et al. 2026. Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces. In The Fourteenth International Conference on Learning Representations (ICLR ’26). OpenReview.net, Rio de Janeiro, Brazil, 40903–40986. https://proceedings.iclr.cc/paper_files/paper/2026/hash/444a3737adaee10d86ad2ef5f74468e6-Abstract-Conference.html
- [45]Microsoft Azure. 2023. Azure LLM Inference Traces. https://github.com/Azure/AzurePublicDataset/blob/master/AzureLLMInferenceDataset2023.md. Accessed 2026-09-13.
- [46]Mahmoud Mohammadi, Yipeng Li, Jane Lo, and Wendy Yip. 2025. Evaluation and Benchmarking of LLM Agents: A Survey. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’25). ACM, New York, NY, USA, 6129–6139. doi:10.1145/3711896.3736570DOI
- [47]OpenAI. 2025. Codex CLI: A Lightweight Coding Agent for the Terminal. https://github.com/openai/codex. Accessed October 2026.
- [48]openguardrails. 2026. DeepSeek-V4-Flash on Terminal-Bench 2.1: Agent Trajectories. https://huggingface.co/datasets/openguardrails/terminal-bench-2.1-deepseek-v4-flash-trajectories. MIT licence, accessed 2026-09-11.
- [49]Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2023. MemGPT: Towards LLMs as Operating Systems. arXiv:2310.08560 https://arxiv.org/abs/2310.08560
- [50]Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed
- [51]Archit Patke, Dhemath Reddy, Saurabh Jha, Haoran Qiu, Christian Pinto, Chandra Narayanaswami, Zbigniew Kalbarczyk, and Ravishankar Iyer. 2024. Queue Management for SLO-Oriented Large Language Model Serving. In Proceedings of the ACM Symposium on Cloud Computing (SoCC ’24). ACM, New York, NY, USA, 18–35. doi:10.1145/3698038.3698523DOI
- [52]Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently Scaling Transformer Inference. In Proceedings of Machine Learning and Systems (MLSys ’23, Vol. 5). mlsys.org, Miami, FL, USA, 606–624. https://proceedings.mlsys.org/paper_files/paper/2023/hash/c4be71ab8d24cdbf45e3d06dbfca2780-Abstract-mlsys2023.htmlarxiv.org/abs/2211.05102
- [53]Qwen Team. 2025. Qwen Technical Report. arXiv:2505.09388
- [54]Qwen Team. 2026. Qwen3.5: Towards Native Multimodal Agents. https://qwen.ai/blog?id=qwen3.5.
- [55]Qwen Team. 2026. Qwen3.6-35B-A3B: Agentic Coding Power, Now Open to All. https://qwen.ai/blog?id=qwen3.6-35b-a3b
- [56]Rakibul Hasan Rajib, Mengxin Zheng, and Qian Lou. 2026. AgentServeSim: Serving-System Simulation and Policy Search for LLM Agent Programs. arXiv:2606.09613 [cs.CL]
- [57]Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Mham, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023) (NeurIPS ’23, Vol. 36). Curran Associates, Inc., Red Hook, NY, USA, 68539–68551. doi:10.52202/075280-2997DOI
- [58]Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU. In Proceedings of the 40th International Conference on Machine Learning (ICML ’23, Vol. 202). PMLR, Honolulu, HI, USA, 31094–31116. https://proceedings.mlr.press/v202/sheng23a.html
- [59]Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. 2025. The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). Albuquerque, New Mexico, 4195–4206. doi:10.18653/v1/2025.naacl-long.211DOI
- [60]Hanshin Sun, Li-Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. 2025. ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference. In Proceedings of the 42nd International Conference on Machine Learning (ICML ’25, Vol. 267). PMLR, Vancouver, Canada, 57355–57373. https://proceedings.mlr.press/v267/sun25b.html
- [61]TiberiusJunyao. 2026. TI Long-Horizon Terminal-Bench Trajectories. https://huggingface.co/datasets/TiberiusJunyao/T1-Long-Horizon-Terminal-Bench-Trajectories. Accessed 2026-09-11; no longer publicly accessible as of 2026-10-27.
- [62]Gemma Team. 2026. Gemma 4 Technical Report. arXiv:2607.02770 [cs.CL] https://arxiv.org/abs/2607.02770
- [63]The Terminal-Bench Team. 2026. Terminal-Bench Science: Benchmarking AI Agents on Scientific Workflows. https://www.tbench.ai/news/terminal-bench-science-0-1. Version 0.1.DOI
- [64]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems 30 (NeurIPS 2017) (NeurIPS ’17, Vol. 30). Curran Associates, Inc., Red Hook, NY, USA, 5998–6008. https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
- [65]Bingyang Wu, Yumin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yunhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. 2025. Fast Distributed Inference Serving for Large Language Models. arXiv:2305.05920 https://arxiv.org/abs/2305.05920
- [66]Xixi Wu, Kuan Li, Yida Zhao, Liwen Zhang, Litu Ou, Huifeng Yin, Zhongwang Zhang, Xinmiao Yu, Dingchu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, Minhao Cheng, Shuai Wang, Hong Cheng, and Jingren Zhou. 2025. ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarization. arXiv:2509.13313 https://arxiv.org/abs/2509.13313
- [67]Zijian Wu, Yuanyang Liu, Xinyuan Zhang, Lingjun Chen, Fanqing Meng, Lingxiao Du, Yiran Zhao, Fanshi Zhang, Yaoqi Ye, Jiawei Wang, Zirui Wang, Jinjie Ni, Yufan Yang, Arvin Xu, and Michael Qizhe Shi. 2026. MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use. In The Fourteenth International Conference on Learning Representations (ICLR ’26). OpenReview.net, Rio de Janeiro, Brazil, 79852–79895. https://proceedings.iclr.cc/paper_files/paper/2026/hash/8138d211ce8790fdbeeeb9781838a37-Abstract-Conference.htmlarxiv.org/abs/2509.24002
- [68]Tian Xia, Yi Xu, Han-Chen Li, Zhifei Li, Xiaokun Chen, Hao Kang, Yifan Qiao, and Jon Stoica. 2026. Idleness is Relative: Exploiting Tool-Call Idle Windows for Offloading in Agentic Systems with MORI. arXiv:2606.00866 https://arxiv.org/abs/2606.00866
- [69]Guangxiang Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient Streaming Language Models with Attention Sinks. In The Twelfth International Conference on Learning Representations (ICLR 2024) (ICLR ’24). OpenReview.net, Vienna, Austria, 21875–21895. https://proceedings.iclr.cc/paper/2024/hash/5e5df18f863cbe6d8ae392a93fd271c9-Abstract-Conference.html
- [70]Tianhao Xu, Yiming Liu, Xianglong Lu, Yijia Zhao, Xutong Zhou, Aichen Feng, Yiyi Chen, Yi Shen, Qin Zhou, Xumeng Chen, Ilya Sherstyuk, Haorui Li, Rishi Thakkar, Ben Hamm, Yuanzhe Li, Xue Huang, Wengpeng Wu, Anish Shanbhag, Harry Kim, Chuan Chen, and Junjie Lai. 2026. AConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving. arXiv:2601.06288 [cs.DC]
- [71]Abel Yagubyan. 2026. How Consistent Are LLM Agents? Measuring Behavioral Reproducibility in Multi-Step Tool-Calling Pipelines. arXiv:2605.28840
- [72]Dongjie Yang, Xiaodong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. 2024. PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference. In Findings of the Association for Computational Linguistics: ACL 2024 (Findings of ACL ’24). Association for Computational Linguistics, Stroudsburg, PA, USA, 3258–3270. doi:10.18653/v1/2024.findings-acl.195DOI
- [73]John Yang, Carlos E. Jimenez, Alexender Wettig, Karthik Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. In Advances in Neural Information Processing Systems (NeurIPS ’24, Vol. 37). Curran Associates, Inc., Red Hook, NY, USA, 50528–50652. doi:10.52202/079017-1601
- [74]Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion. In Proceedings of the Twentieth European Conference on Computer Systems (Rotterdam, Netherlands) (EuroSys ’25). Association for Computing Machinery, New York, NY, USA, 94–109. doi:10.1145/3689031.3696098DOI
- [75]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations (ICLR ’23). OpenReview.net, Kigali, Rwanda, 33 pages. https://openreview.net/forum?id=WE-vluYUL-Xarxiv.org/abs/2210.03629
- [76]Rui Ye, Zhongwang Zhang, Kuan Li, Huifeng Yin, Zhengwei Tao, Yida Zhao, Liangcai Su, Liwen Zhang, Zile Qiao, Xinyu Wang, Pengjun Xie, Fei Huang, Jingren Zhou, Shiheng Chen, and Yong Jiang. 2026. AgentFold: Long-Horizon Web Agents with Proactive Context Folding. In The Fourteenth International Conference on Learning Representations (ICLR ’26). OpenReview.net, Rio de Janeiro, Brazil, 154754–154776. https://proceedings.iclr.cc/paper_files/paper/2026/hash/fa94824dfe016371e72c1a7c4aaebc-Abstract-Conference.htmlarxiv.org/abs/2510.24699
- [77]Gyeong-In Yu, Jeong Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) (OSDI ’22). USENIX Association, Berkeley, CA, USA, 521–538. https://www.usenix.org/conference/osdi22/presentation/yu
- [78]Shan Yu, Junyi Shu, Yuanjiang Ni, Kun Qian, Xue Li, Yang Wang, Jinyuan Zhang, Ziyi Xu, Shuo Yang, Lingjun Zhu, Ennan Zhai, Qingda Lu, Jiarong Xing, Youyou Lu, Xin Jin, Xuanzhe Liu, and Harry Xu. 2026. Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving. arXiv:2604.25899 [cs.MA] https://arxiv.org/abs/2604.25899
- [79]Jiayi Yuan, Hao Li, Xinsheng Ding, Wenya Xie, Yu-He Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, and Zirui Liu. 2025. Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference. In Advances in Neural Information Processing Systems (NeurIPS ’25, Vol. 38). Curran Associates, Inc., Red Hook, NY, USA, 188048–188077. doi:10.52202/087513-5653arxiv.org/abs/2506.09501
- [80]Yuwei Zhang, Chengyu Dong, Shuowei Jin, Changlong Yu, Hejie Cui, Hongye Jin, Xinyang Zhang, Hamed Bonab, Colin Lockard, Jianshu Chen, Zhenyu Shi, Jingbo Shang, Xian Li, and Bing Yin. 2026. CoMem: Context Management with a Decoupled Long-Context Model. In International Conference on Machine Learning (ICML) (Proceedings of Machine Learning Research, Vol. 306). PMLR, Seoul, South Korea, 155062–155077. https://proceedings.mlr.press/v306/zhang26ai.htmlarxiv.org/abs/2605.30842
- [81]Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Shiyang Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. In Advances in Neural Information Processing Systems 36 (NeurIPS 23, Vol. 36). Curran Associates, Inc., Red Hook, NY, USA, 34661–34710. doi:10.52202/075280-1506DOI
- [82]Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. In Advances in Neural Information Processing Systems (NeurIPS ’24, Vol. 37). Curran Associates, Inc., Red Hook, NY, USA, 62557–62583. doi:10.52202/079017-2000
- [83]Zijian Zhou, Ao Qu, Zhaoxuan Wu, Sunghwan Kim, Alok Prakash, Daniela Rus, Bryan Kian Hsiang Low, and Paul Pu Liang. 2026. MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents. In The Fourteenth International Conference on Learning Representations (ICLR ’26). OpenReview.net, Rio de Janeiro, Brazil, 58413–58438. https://proceedings.iclr.cc/paper_files/paper/2026/hash/5fc8b3bdfbb6917b514df5d3fae4616-Abstract-Conference.html
- [84]Jialiang Zhu, Gongrui Zhang, Xiaolong Ma, Lin Xu, Miaosen Zhang, Ruiqi Yang, Song Wang, Kai Qiu, Zhirong Wu, Qi Dai, Ruichun Ma, Bei Liu, Yifan Yang, Chong Luo, Zhengyuan Yang, Linjie Li, Lijuan Wang, Weizhu Chen, Xin Geng, and Baining Guo. 2026. RE-TRAC: Recursive Trajectory Compression for Deep Search Agents. In Proceedings of the 43rd International Conference on Machine Learning (ICML ’26, Vol. 306). PMLR, Seoul, South Korea, 167986–168007. https://proceedings.mlr.press/v306/zhub26d.htmlarxiv.org/abs/2602.02486
- [85]Kan Zhu, Mathew Jacob, Chenxi Ma, Yi Pan, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikci. 2026. TraceLab: Characterizing Coding Agent Workloads for LLM Serving. arXiv:2606.30560 [cs.DC] https://arxiv.org/abs/2606.30560 Trace release (CC BY 4.0), https://github.com/uw-syf/TraceLab, accessed 2026-09-11.
- [86]Adam Zweiger, Xinghong Fu, Han Guo, and Yoon Kim. 2026. Fast KV Compaction via Attention Matching. In Proceedings of the 43rd International Conference on Machine Learning (ICML ’26, Vol. 306). PMLR, Seoul, South Korea, 169210–169239. https://proceedings.mlr.press/v306/zweiger26a.htmlarxiv.org/abs/2602.16284






