WorksComputer Science
Making Preemption Productive in Long-Output LLM Serving
Abstract
Reasoning LLMs generate long outputs whose key–value (KV) caches occupy GPU memory for minutes. When memory fills, the server preempts requests, and a preempted request can repeatedly lose and recover its state while producing little new output. We call this behavior KV thrashing. In certain cases, existing systems leave more than a fifth of requests stalled for over a minute. Because preemption cannot be avoided without delaying admission, we ask how to make it productive. We present Stint, a serving system that preserves a preempted request's work, bounds how long it waits, and gives it fresh output after it returns. Stint stores fixed-length KV snapshots in a contiguity-first CPU pool with an exact-extent allocator, and moves them with fused K/V gather/scatter kernels and double buffering. Age-based readmission brings stalled requests back, and protection gives them a turn of fresh decoding before another preemption. Remaining-length estimates from decoding hidden states guide victim selection. On olympiad-math replays with Qwen3-32B on H200 GPUs, no request pauses for more than 50 seconds under Stint and p99 time per output token falls by 50.7% relative to recomputation in the same engine. The replay as a whole finishes 10.7% sooner (12.0% higher throughput), because the requests that baselines starve no longer finish last.
Introduction
Reasoning LLMs can solve difficult problems by generating intermediate steps before reaching an answer [24, 34]. This longer reasoning process also puts pressure on GPU memory. As a request generates tokens, the serving system stores their keys and values in a growing KV cache. Longer outputs therefore require both more memory and more time before that memory can be released. Figure 1 illustrates the difference: across the three models, mean outputs on American Invitational Mathematics Examination (AIME) problems range from 1,657 to 13,733 tokens, compared with 314 to 2,173 tokens on the two conversational datasets.
Figure 1. Historical output-length profiles for Llama-3.1-8B, Qwen3-14B and Gemini 3 Pro, separate from our serving experiments. Numbers label means; error bars show 95% confidence intervals.
When GPU memory runs out, the server must preempt a running request to make room. It can save the request’s KV cache in CPU memory or discard the cache and recompute it later. In either case, the request must wait for GPU space and recover its state before generating another token. Preemption thus interrupts generation even while the server continues working on other requests. Figure 2(a) follows GPU KV usage during a replay: preemptions accumulate while the KV pool is nearly full.
Figure 2. KV thrashing with Qwen3-32B on 200 programming requests from LiveCodeBench (LCB) [14], denoted LCB200. GPU KV capacity is reduced and recovery is disabled. (a) GPU KV usage and cumulative preemptions over time. (b) One request’s repeated pauses (orange) and fresh decoding (blue). (c) Median longest generation pause by preemption count across three replays; whiskers show the middle 50%.
More importantly, the same request can be preempted repeatedly. While it waits, other requests keep growing their KV caches and consume the space released by its eviction. Even after it returns to the GPU, it may soon be evicted again. We call repeated eviction and recovery with little fresh output between interruptions KV thrashing. Figure 2(b) shows one request pausing for 21.5 seconds, producing just 28 fresh tokens, then pausing for another 26.3 seconds. Swap thrashing has also been observed in preemptive serving [39]; here we study how it disrupts long decoding. The stakes are high. When we replay 399 olympiad-level math problems with a median output of 17,089 tokens, vLLM with recomputation, LMCache and SGLang with HiCache each leave 21–23% of requests stalled for more than a minute, and about one request in six for more than five minutes. Under recomputation, a single pause takes up to 72% of a request’s lifetime.
Preemption itself cannot be avoided once long decodes oversubscribe GPU memory. Admission control can prevent it, but moves the wait to before the first token: a simple concurrency cap delays first output by several minutes in our replays (§ 6.10). We therefore ask a different question: how can preemption be made productive? A preempted request should keep the work it has already done, return to the GPU within a bounded time, and produce new output once it returns. Each condition poses a challenge. First, the server must choose which request to interrupt. A request with a large KV cache frees more memory when evicted, but may be close to completion; its cache size does not reveal how much work remains. Second, saving and restoring KV must be cheap. CPU memory is larger than GPU memory but still limited. Reserving large chunks wastes space, while moving scattered GPU blocks individually adds transfer overhead. Third, restoration must lead to progress. A saved request can wait indefinitely while running requests keep growing their caches, and even after it returns, another preemption can interrupt it before it produces much output.
Fast transfers alone do not solve this scheduling problem.
We present Stint, a serving system that makes preemption productive by connecting the choice of victim, the storage and transfer of its KV, and its return to useful decoding.
Prediction-guided victim selection. Stint uses hidden states from ongoing decoding to estimate how much output each request has left, and prefers to keep requests that appear close to completion. Sparse online inference refreshes these estimates periodically, limiting their cost while letting them change as the answer develops.
Contiguity-first CPU KV pool. Stint stores KV snapshots in contiguous CPU regions. The design follows a simple observation: GPU KV grows during decoding, but an offloaded snapshot has a fixed length until the request resumes. The allocator can therefore reserve exactly the snapshot’s size, preferring one contiguous extent, without reserving room for future growth.
Bulk KV transfer. Fused K/V kernels gather a request’s scattered GPU blocks into staging buffers and scatter reloaded KV back, so CPU–GPU copies move contiguous ranges. Double buffering lets one copy proceed while the GPU gathers or scatters the next group of blocks.
Age-based protected recovery. An age-based rule gives a stalled request priority to return to the GPU, bounding how long it waits. Once selected, the request is protected from preemption through restoration and a turn of fresh decoding. Reloading KV or recomputing old context does not count as progress; generating new tokens does.
We implement Stint in vLLM and evaluate it on H200 GPUs with Qwen3-32B. On the olympiad math replay (IMO-AnswerBench [22]), Stint raises throughput by 12.0% and lowers 99th-percentile time per output token (TPOT) by 50.7% relative to recomputation in the same engine, across three matched runs. No request pauses for more than 50 seconds, versus 11–14 minutes for the worst requests under every baseline. Stint also outperforms LmCache by 14.2% and SGLang with HiCache by 48.5% in throughput, although the latter comparison also spans different engines. On AIME and LiveCodeBench, p99 TPOT falls by 35–75% while throughput changes by −0.5% to +3.6%.
These gains have costs. Stint pauses most long requests once, for about 30 seconds but never more than 8.4% of the request’s lifetime, and it raises first-token and pooled inter-token latency tails. It also needs more pinned host memory. Benefits concentrate on long-output workloads; on short-output summarization and BurstGPT replays, Stint offers no gain. In component studies, retaining more snapshots and readmitting stalled requests raise throughput, while prediction-guided selection lowers it in exchange for shorter output gaps.
Contributions.
We characterize KV thrashing during long decoding and show that existing systems leave a large fraction of requests stalled for minutes.
We design age-based protected recovery, which bounds how long a preempted request waits and guarantees it fresh output after it returns.
We design a contiguity-first CPU KV pool with an exact-extent allocator, and bulk KV transfer with fused K/V kernels and double buffering, exploiting the fixed length of offloaded snapshots.
We use decoding-time hidden states for prediction-guided victim selection, with sparse online inference to limit serving cost.
We evaluate the complete system against three baselines on five workloads, including where its benefits end and what they cost.
Background
KV Cache Growth During Decoding
Each new token attends to keys and values from earlier tokens. The serving system keeps these tensors in a KV cache so it does not have to compute them again at every step. With full attention, each generated token adds KV state, and the cache remains live until the request finishes. Long outputs therefore increase both the size of the cache and how long it occupies memory.
Table 1 shows the KV payload for several models, including Qwen3-32B and Llama-3.1-8B from our main evaluation. A single 10,000-token context requires about 2.44 GiB for Qwen3-32B and 4.81 GiB for Llama-3.1-405B. Serving many such requests together can exhaust GPU memory even when each request fits individually.
Table 1. BF16 KV payload per request, before sharding or replication across GPUs. Context includes prompt and output tokens. Values exclude allocation padding and metadata. DeepSeek-R1 uses its compressed MLA cache.
For Llama and Qwen, the per-token payload is bytes: layers, KV heads, head dimension , and bytes per element, for both keys and values. These three models have eight KV heads with dimension 128; the number of layers is 32, 64 and 126 for Llama-3.1-8B, Qwen3-32B and Llama-3.1-405B, respectively.
DeepSeek-R1 uses multi-head latent attention (MLA). It stores a compressed vector shared by keys and values, together with a positional key vector. The table counts this compressed representation. MLA reduces the bytes per token, but the cache still grows with context length. The physical memory needed across GPUs also depends on how a serving system shards or replicates these tensors.
KV Cache Management for Serving
PagedAttention [16] divides GPU KV into fixed-size blocks. A request acquires more blocks as it grows, without reserving a large contiguous region for its possible future output. This reduces fragmentation, although the final block can remain partly empty. Prefix caching [47] also allows requests with a common prefix to reuse KV blocks. Systems such as LMCache [19, 43] store and transfer KV across memory tiers and serving workers.
These mechanisms help use available capacity. When capacity is exhausted, the server still needs to choose a request to preempt and a way to recover its state later.
Current Preemption Mechanisms
Preemption releases GPU space by removing a request’s KV blocks. The request can recover through either recomputation or offloading.
Recomputation. The server discards KV and rebuilds missing state from the saved token sequence when the request resumes. Any reusable cached prefix reduces this work. For long contexts, rebuilding still consumes substantial GPU compute: full causal attention has a quadratic cost component. These computations restore old state before the request can produce new output.
Offloading. The server saves KV in CPU memory and reloads it when the request resumes. This preserves computed state but uses CPU capacity and host–device bandwidth. Moving many small GPU blocks can also add dispatch and packing costs. The choice between these mechanisms depends on how much state must be rebuilt or moved, as well as the available compute and bandwidth. Appendix F derives a compute–transfer cost model and explains how context length and model geometry affect this tradeoff.
Recovering KV state is only part of resuming a preempted request. The scheduler must first make enough GPU memory available to reload or rebuild that state. If running requests continue to occupy the memory, the preempted request has to wait. Even after it resumes, it may be preempted again after generating only a few new tokens. The server then repeats the transfer or recomputation, while the request makes little progress. The next section examines how this cycle interrupts long decoding requests.
Challenges with Serving Reasoning Models
KV thrashing involves more than the cost of one transfer or one recomputation. A request can repeatedly lose GPU space, recover its state, and lose that space again. We first examine the resulting interruptions, then the decisions that a serving system must make to reduce them.
Repeated Preemption Interrupts Decoding
Repeated preemption matters because a request can spend time recovering without making much progress. Figure 2(b) follows three consecutive interruptions in one request. It produces only 22 fresh tokens between its third and fourth evictions, then 28 between its fourth and fifth. The fourth and fifth evictions interrupt fresh output for 21.5 and 26.3 seconds, respectively. Recovery repeatedly gives this request only a short opportunity to decode before it must recover again.
Figure 2(c) puts these interruptions in context. The median longest streaming pause is 0.09 seconds for request executions without preemption, 22.1 seconds for those preempted once, and 39.7 seconds for those preempted repeatedly. These groups differ in output length and memory pressure, so the comparison shows an association, not a fixed latency penalty per preemption. Looking at each request’s longest pause exposes interruptions that an average time per output token can hide.
This illustration uses a separate 200-request LiveCodeBench replay (LCB200) with recovery disabled and reduced GPU KV capacity, making repeated preemption easier to observe. It is a motivation experiment, not a comparison of complete systems at their native capacity. The setup and sampling method are given in Appendix § 16.
Memory Relief and Remaining Work Can Conflict
The first decision is which request to interrupt when GPU memory runs out. Current KV size measures how much memory a request occupies; remaining output length estimates how much decoding it has left. These signals need not rank requests in the same order.
For example, a request with a large cache may be close to completion, while a request with a smaller cache may have many tokens left to generate. Preempting the first can free more memory immediately but interrupt a possible completion. Preempting the second preserves that opportunity but may free too little space, requiring more evictions. Shared prefix blocks can further reduce the memory released by an eviction.
The scheduler must therefore balance immediate memory relief against the opportunity to finish a request and release its cache. It knows how much memory each request occupies, but how much output remains must be estimated while the answer is still being generated.
Contiguity vs Fragmentation Tradeoff
After choosing a victim, the server can save its KV in CPU memory to avoid rebuilding it later. Storing the snapshot in a contiguous region supports bulk transfers. Each snapshot has a fixed length, but snapshots from different requests have different sizes and lifetimes. As requests resume and free their CPU storage, gaps appear between the snapshots that remain. A new snapshot may not fit in any gap, even when the pool has enough free space in total.
Large uniform chunks simplify allocation but waste space in partially filled chunks. Dividing snapshots into many small pieces uses capacity more flexibly, but requires the transfer engine to handle more regions. The challenge is to keep snapshots easy to transfer while retaining as much useful KV as possible in limited CPU memory.
Recovery Must Be Efficient and Lead to Progress
Saving a snapshot avoids losing computed state, but returning it to the GPU still takes work. GPU KV is spread across pages, and copying them individually can require many small transfers, as in vLLM v0.9.2’s block-swapping primitive [37]. Existing systems reduce this overhead with larger chunks and pipelines, specialized GPU kernels, or batched copies [19, 23, 28, 45]. The CPU layout and transfer method must work together to make restoration efficient.
Even a fast transfer cannot begin until GPU space is available. While an offloaded request waits, running requests can consume that space. Once the snapshot is restored, another eviction may interrupt decoding after only a few new tokens, as in Figure 2(b). Reducing the cost of each restoration therefore addresses only part of KV thrashing. The scheduler must also give waiting requests a way to return and enough time to use their restored state. Giving one request a longer turn, however, can increase the waiting time of others.
Motivation
A preempted request leaves behind both completed work and unfinished output. Our design aims to preserve what has already been computed while deciding when to spend GPU time on what remains.
Use Remaining Work to Choose Victims
A request close to completion can soon release GPU memory. Its current cache size, however, does not tell us how much output remains. Stint uses hidden states from ongoing decoding to estimate that remaining work. These states contain information from the partially generated answer that a prompt-only estimator cannot use [7]. TRAIL also uses decoding-time embeddings for length-aware scheduling [29].
This allows the estimate to change as the answer develops. Stint refreshes it periodically through sparse online inference, while keeping the predictor’s weights fixed (§ 5.2). The goal is to use information about remaining work without adding prediction cost at every token.
Preserve KV When Space Is Available
After choosing which request to pause, the server must decide what to do with the computation it has already completed. Offloading keeps the resulting KV so that recovery can copy it back; recomputation discards KV and spends GPU compute to rebuild it. For conventional full attention, KV bytes grow linearly with context length, while dense attention adds a quadratic component to recomputation. Long contexts can therefore favor offloading, though CPU capacity, bandwidth, prefix reuse and interference with other requests also affect the cost.
Figure 3 compares these two costs across six attention mechanisms. Each cell divides the time to rebuild a full context by the time to offload and reload its saved state. A value of two, for example, means rebuilding takes twice as long as the transfer round trip in this model.

Figure 3. Modeled cost of rebuilding a full context relative to offloading and reloading its saved state. Red favors offloading; blue favors recomputation. Cell labels show the time ratio, with a shared logarithmic color scale centered at equal cost ().
All plotted cases favor transferring saved state at the assumed rates. Compact state can be cheap to move even when rebuilding it is costly: MLA and CSA store compressed history, while pure KDA saves a fixed-size state. For pure KDA, a longer context adds rebuild work without increasing the snapshot size. Smaller state can also reduce memory pressure, so these ratios alone do not show how often preemption occurs.
To compute these ratios, we give all panels the same feedforward layer at each model width, with an intermediate width four times the hidden width. We assume the GPU can perform 12,500 floating-point operations in the time the CPU–GPU link moves one byte. MLA, KDA and CSA use attention dimensions drawn from DeepSeek-V3 [5], Kimi Linear [15] and DeepSeek-V4 [6], respectively. We keep their head sizes and compression dimensions fixed while varying the hidden width. These are comparisons of individual blocks, not measured serving speedups for the full models. Appendix F gives the dimensions and derivation.
Preserving state also gives us a useful allocation opportunity. GPU KV grows during decoding, but an offloaded snapshot has stopped growing. Its known size lets the CPU pool reserve a few contiguous regions without leaving room for future tokens. Those regions can then support bulk transfers from and back to the paged GPU cache.
The cost model motivates retaining KV; it is not an online decision rule. Stint offloads when CPU snapshot allocation succeeds and recomputes otherwise (§ 5.3.1).
Make Restoration Lead to New Output
Retaining KV can make restoration cheaper, but its value depends on what happens next. If a request waits indefinitely for GPU space, its saved state cannot help it finish. If it returns only to be evicted again after a few tokens, the system repeatedly pays to restore the same accumulated work.
This motivates coupling restoration with an opportunity to generate new output. Stint uses stall age to give a waiting request priority to return, then protects it for a turn of new tokens. The turn starts with state restoration, but only newly generated tokens count toward completing it. This connects the CPU pool and transfer engine to the scheduler: they preserve and restore the state, while recovery scheduling gives the request time to use it (§ 5.3.2).
Design
Stint follows a request from preemption back to decoding: choose a victim, save its KV if possible, then restore it and give it time to produce new output. Prediction guides the first decision; stall age and protection guide the return.
The common goal is useful progress. A fast KV reload is wasted if the request is evicted again before it can use that state. Conversely, a protected request still has to recover its state before decoding. The scheduler, CPU pool and transfer engine therefore work along the same recovery path.
Design Goals and System Overview
Figure 4 shows how the components interact within one replica.

Figure 4. One replica combines prediction-guided victim selection, a contiguity-first CPU KV pool, bulk KV transfer, and age-based protected recovery. Arrows connect prediction, state preservation and return to decoding.
Prediction-guided victim selection. On GPU allocation failure, Stint uses hidden-state estimates of remaining output length to choose a victim, extending vLLM’s first-come-first-served scheduler (§ 5.3).
Contiguity-first CPU KV pool. The KV manager stores fixed-length snapshots in CPU extents, or contiguous ranges of block positions. It prefers one extent and falls back to several, then to recomputation if allocation fails (§ 5.4).
Bulk KV transfer. Fused K/V kernels gather scattered GPU blocks for bulk copying into CPU extents and scatter them back on reload. Double buffering overlaps these kernels with copying (§ 5.5).
Age-based protected recovery. The scheduler selects an overdue stalled request, makes GPU space for it, and protects it through KV restoration and a turn of fresh decoding (§ 5.3.2).
Remaining-Length Prediction During Decoding
To choose which request to preempt, the scheduler needs to know which requests are close to finishing. KV cache size cannot answer this question: two requests with similar caches may have very different amounts of output left. Stint therefore estimates remaining length while the answer is being generated.
Estimating how much output remains. We use hidden states, the internal representations the model already computes during token generation. Prior work suggests that these representations encode output structure [7]. A small predictor combines information from selected layers to estimate what fraction of the answer has been generated. This requires no additional forward pass through the LLM. For example, if a request has generated 1,000 tokens and the predictor estimates that it is halfway finished, about 1,000 tokens remain. More generally, after generating tokens, an estimated completed fraction gives a total length of . Subtracting the tokens already generated gives the estimated remaining length . We smooth successive estimates to reduce fluctuations. The predictor’s weights stay fixed during serving; only its estimates change as the answer develops.
Keeping prediction inexpensive. Hidden states are already available during generation, but collecting them and running a predictor at every token still adds overhead. Stint therefore reuses predictions across decoding steps. It caches the predicted total length and subtracts newly generated tokens to update the remaining length. This sparse online inference normally refreshes predictions every decoding steps, with in our evaluated configuration. A request receives an earlier update when it has no estimate or is approaching its predicted end.
Reducing predictor calls alone would still leave the cost of collecting its inputs. Stint avoids this work by deciding which requests need an update before the LLM’s forward pass. During that pass, it saves only the required token representations from selected layers for those requests. When no update is needed, it skips both feature collection and predictor execution. Thus, most decoding steps reuse an estimate without preparing another prediction.
When several requests need updates, Stint processes them together in one batch on the serving GPU. The small predictor first compresses the selected hidden states into smaller vectors, then estimates progress. Keeping these features on the GPU avoids transferring large hidden-state tensors to the CPU; only the resulting scalar predictions need to be returned.
The interval balances prediction cost against freshness. Longer intervals reduce regular prediction work, while early updates let requests revise estimates they are about to out-grow. § 6.7 measures the serving overhead, and Appendix § 14.§ 2 examines this tradeoff across different intervals.
Supporting different models. Qwen and Llama use their own trained predictors, but both provide the scheduler with the same information: a predicted total output length. This lets us change the predictor without changing the scheduling policy.
These estimates help the scheduler choose which request to interrupt. Completing the recovery cycle also requires deciding when that request should return and giving it time to produce new output. The next section explains how Stint connects these decisions.
Victim Selection and Recovery
When GPU memory fills, the scheduler must decide which request to pause and how that request will resume. Stint uses predicted remaining length to favor keeping requests that are close to completion. It also tracks the progress of preempted requests, giving those that remain stalled priority to return and a protected turn to generate new output.
Prediction-Guided Victim Selection
When memory runs out, Stint favors keeping requests that appear close to completion. It preempts the eligible request with the longest predicted remaining output. For the eligible set , this rule is
This is the shortest-remaining-job-first (SRJF) victim policy: evicting the longest remaining job favors keeping near-completion requests on the GPU. The eligible set contains other running requests, excluding those already selected for the current step and the request protected by recovery. Ties prefer earlier arrivals, then request ID. If any eligible candidate lacks a usable prediction, the whole decision falls back to physical-largest: choose the request that releases the most GPU blocks. If no alternative is eligible, the ordinary request needing memory can itself be preempted.
Remaining work versus memory relief: A request with much output left can have a small current cache; evicting it may free little memory. A nearly finished request can hold a large cache that is expensive to move. SRJF favors possible completions, while physical-largest favors immediate memory relief. § 6.8 compares them with storage, transfers and recovery held fixed.
Preserve KV when CPU space is available. After selecting a victim, the scheduler allocates CPU storage for its snapshot. Success leads to offloading; failure due to capacity, extent or request limits leads to discarding KV and queuing recomputation (§ 4.2). This reacts to GPU allocation failure; it does not predict future offload times or use a per-victim compute–transfer threshold.
Age-Based Protected Recovery
Choosing a victim frees memory for other requests, but leaves the interrupted request waiting. Those other requests continue growing their KV caches, so the waiting request may struggle to regain GPU space. Even after it returns, another eviction can interrupt it before it generates much output. It then pays for recovery again after making little progress: the KV thrashing described in § 3.
Stint connects the return to the GPU with a turn of new output. Age-based readmission gives a stalled request priority to return. Protection then prevents preemption while it restores KV and generates new tokens. The first addresses prolonged waiting; the second lets the request use the state it has recovered. Figure 5 follows one request through this process.

Figure 5. Age-based protected recovery: readmit the oldest eligible stalled request and protect KV restoration and 64 fresh tokens, ending early on completion. Unfinished requests return to normal scheduling.
1. Give the oldest stalled request a turn. The first pre-emption starts a stall timer. Reloading KV or recomputing old context leaves it running; only a new output token clears it. Resetting the timer at restoration would make a repeatedly interrupted request appear newly active even if it had produced no new output. Keeping the timer running lets the scheduler recognize that it is still waiting to make progress.
When no request is protected, the scheduler selects the oldest stall that has reached seconds in the evaluated configuration. Each replica’s scheduler protects one request at a time. Another protected request may still be finishing its turn, so five seconds is an eligibility threshold, not a bound on waiting. Offloaded requests can also reload normally before this threshold when GPU space and scheduling budget permit.
2. Make room and restore its state. Protection starts as soon as the request is selected, before restoration. The scheduler reserves a running-request slot and, if necessary, preempts other requests to free GPU blocks. For this step it uses physical-largest victim selection, even when ordinary eviction uses SRJF. The immediate goal is to obtain enough space for the selected request to return.
The request then reloads its saved CPU snapshot. If its KV was discarded, it reuses any cached prefix and recomputes the missing state. Both paths recover previous work. Protection remains active throughout this work so that the request can proceed to decoding.
3. Protect a turn of new output. Once its state is ready, the request is scheduled first and cannot be preempted until it generates new tokens or finishes. We use in the evaluated configuration. Reloaded KV and recomputed tokens do not count toward this turn: they restore existing context. The first new token clears the stall timer, but protection continues until the request has generated all new tokens since selection.
Other requests can decode alongside it when resources permit. If more space is needed as its KV grows, the scheduler can preempt those other requests. After the protected turn, the request keeps its KV and continues under ordinary scheduling; it is not automatically evicted. A later preemption starts a new stall timer.
The length of the protected turn controls how much new output a request can produce between recovery cycles. A short turn can be followed quickly by another eviction. A longer turn gives the request more progress for the cost of restoring its state, but can make other requests wait longer. § 6.9 evaluates this tradeoff; Appendix § 16.10 gives additional scheduling details.
Recovery therefore needs both efficient state restoration and time to use the restored state. The following sections explain how the CPU pool and transfer engine support the restoration step.
Contiguity-First CPU KV Pool
Recovery can reuse a saved snapshot only while the CPU pool has room to retain it. The pool must therefore support efficient transfers without wasting space that could hold another request’s state.
GPU KV grows with each generated token, so fixed-size blocks let requests acquire space as needed [16]. An offloaded CPU snapshot does not grow: decoding has stopped. Stint can therefore allocate its known size in contiguous CPU ranges, without reserving room for future tokens. After restoration, growth resumes on the GPU; a later offload saves a new snapshot at its current size.
Figure 6 shows why a fixed snapshot size does not by itself solve allocation. Uniform large pages leave unused space inside partially filled chunks: internal fragmentation. A ring that reclaims space in allocation order cannot reuse a released region while an older snapshot blocks reclamation. This creates external fragmentation under that FIFO policy. The ring is an illustrative comparison, not an evaluated baseline or a claim about all ring designs.
Figure 6. Allocation in the same 24-block pool: A, B and C each need six blocks; release B, then request eight blocks for D. (a) Uniform eight-block chunks admit D but retain four padding blocks (internal fragmentation). (b) The FIFO-reclaim ring holds B’s released gap behind A (external fragmentation under this policy), so D waits. (c) Exact extents admit D using 6 + 2 blocks and leave four free. All sizes are illustrative KV-block counts.
Exact-extent allocator. Stint first tries to place the whole snapshot in one contiguous CPU region, called an extent. If no single region is large enough, it combines a few free regions. This keeps a snapshot contiguous when possible while still using gaps left by earlier requests.
The allocator searches for the smallest free extent that fits the snapshot and returns the unused portion to the pool. This best-fit rule preserves larger free regions for other snapshots. If none fits, it takes space from the largest free extents, using at most eight per snapshot in the optimized configuration. Bounding the number of extents limits how many separate copies the transfer engine must issue.
Each allocation reserves the snapshot’s current block count, including its final partially filled KV block. It adds no padding to a fixed-size request slot. If the pool cannot provide a complete allocation within its capacity, extent and request-count limits, the request falls back to recomputation.
Capacity and reuse. After a snapshot is reloaded, its CPU regions can hold other requests. Stint waits until the model step completes before releasing them, then merges neighboring free regions. It does not move live snapshots to compact the pool, so fragmentation can still prevent some allocations.
Each GPU worker has its own pool of pinned CPU memory. The configured budget counts usable KV storage, which differs from the memory reserved by the underlying allocator. In our implementation, an 800 GiB usable pool requires 1,024 GiB of backing memory. This pool-level overhead is separate from the per-snapshot padding that exact extents avoid. Appendix G.7 describes the data layout and memory accounting; Section 6.4 reports measured host-memory use.
Bulk KV Transfer
The CPU pool gives each snapshot a few contiguous regions, but its GPU blocks remain scattered. Copying each block separately would turn a large snapshot into many small transfers. Stint instead gathers the GPU blocks into a temporary GPU buffer and copies them to the CPU in bulk. Reload reverses these steps: copy into the temporary buffer, then scatter the data into the request’s GPU blocks (Figure 7). These staging buffers keep scattered accesses within GPU memory while transfers over the CPU–GPU link use contiguous ranges.

Figure 7. KV reload (one layer; K or V shown): (a) HiCache’s indexed unified virtual addressing (UVA) kernel; (b) Stint’s bulk DMA followed by scatter; (c) double buffering overlaps copying and scattering. Arrows denote readiness and safe reuse, not durations. HiCache’s separate staged-DMA offload path is not shown.
Fused K/V gather/scatter kernels. Staging adds GPU work to gather and scatter the data. To reduce launch overhead, Stint handles K and V together in one kernel invocation. Transfers proceed layer by layer, with the engine grouping CPU regions into chunks that fit the staging buffers. One kernel gathers both K and V into separate contiguous buffers; cudaMemcpyAsync then copies each range to pinned CPU memory using direct memory access (DMA). Reload uses the corresponding fused scatter kernel. K and V share the kernel launch but retain separate DMA copies. Free gaps between extents are never copied.
Double buffering. With one staging pair, gathering the next chunk must wait until the current copy finishes. Reload has the same problem: the next copy must wait until scattering releases the buffers. These operations serialize even though copying and gather/scatter use different GPU resources.
Stint alternates between two staging pairs, each containing a K buffer and a V buffer. During offload, the GPU gathers the next chunk into B while DMA copies A to the CPU. During reload, DMA fills B while the GPU scatters A into the request’s KV blocks. The two stages can therefore work on different chunks at the same time.
Protect data until transfers finish. A staging buffer must not be overwritten while a copy or scatter still reads it. Likewise, attention must wait until incoming KV is ready and outgoing KV is safe to reuse. CUDA events enforce these dependencies. Later layers’ transfers can continue while the model proceeds, but the dependencies can still delay attention. Double buffering overlaps copying with gather/scatter; it does not hide every transfer behind model computation. Appendix G.7 gives buffer sizes and execution details.
Prediction Errors and Limits
Prediction errors change who keeps GPU memory. An overestimate can cause a nearly finished request to be evicted; restoring its accumulated KV may still be expensive. An underestimate can keep a long-running request resident while other requests give up memory. Sparse refresh supplies new information but may not correct either error in time. Physical-largest fallback handles missing or invalid estimates, while protected recovery gives a selected stalled request time to produce fresh output. Valid but inaccurate estimates can still hurt victim selection. We therefore evaluate the cost and effect of prediction separately from the rest of the system, measuring throughput and latency alongside prediction accuracy.
Evaluation
We evaluate whether Stint makes preemption productive in long-output serving. We ask four questions. Does Stint complete long outputs faster, with a shorter tail of slow requests (§ 6.2)? Does it bound how long preempted requests stall (Section 6.3)? Where do its benefits end, and what do they cost (§ 6.4)? Which components contribute (§ 6.5–§ 6.10, summarized in § 6.11)?
Evaluation Setup
Models and workloads. We use three long-output workloads. IMO-AnswerBench contains olympiad-level problems with short verifiable answers [22]; we replay all 399 valid questions from its public release [10]. Its median output is 17,089 tokens, and 213 of 399 outputs exceed 16,384 tokens. AIME1000 replays 1,000 requests drawn from 355 distinct AIME tasks, and LCB880 replays 880 LiveCodeBench programming tasks [14]. The main experiments serve BF16 Qwen3-32B on eight H200 GPUs as four two-GPU replicas (DP4/TP2). Finite Poisson arrivals enter at 4 requests/s across the deployment, with round-robin routing.
Fixed-output replay. Each system receives identical prompts and the output length that the same model produced in a natural-completion run with greedy decoding. Requests use temperature 0 and ignore EOS until that length, so every system performs the same output work; answer quality is not measured. Greedy trajectories still diverge as batch composition changes: in matched runs, only 4 of 1,197 IMO-AnswerBench responses have identical text under recompute and Stint. Length predictions therefore see hidden states from a trajectory that resembles, but need not match, the one that defined the request’s length. We test natural completion separately on 175 LiveCodeBench v6 tasks at 1 request/s on one TP2 replica. Further studies cover Qwen/arXiv summarization, Qwen/BurstGPT, Llama-3.1-8B-Instruct at DP8/TP1, and Cachewise structural replay; settings are in Appendix § 16.
Predictor training. Weights remain fixed during serving. The IMO-AnswerBench predictor is trained on 100 randomly selected questions, and all 399 enter the serving comparison. Restricting the analysis to the 299 questions outside the training set gives the same direction: relative to recompute, p99 TPOT falls by 53.56% and p99 completion time by 4.16% (Appendix § 10.§ 2). AIME1000 and LCB880 also draw on development tasks used for predictor validation or training.
Systems. Recompute uses Stint’s maintained vLLM 0.9.2 build with default FCFS scheduling, no predictor and no CPU KV pool, so it differs from Stint only in the mechanisms this paper adds. Stint uses SRJF victims, K64 inference, exact-extent storage, fused K/V transfers, double buffering and age-5/Q64 recovery. We compare them on all three main workloads. IMO-AnswerBench also compares LMCache [19] with upstream vLLM, and SGLang with HiCache. Both can restore a prefix and rebuild missing state; Stint preserves the complete decode snapshot. The HiCache comparison also includes differences between the SGLang and vLLM engines. LMCache’s runs include the resumed-request metadata repair described in Appendix § 18.§ 5; Appendix § 16 gives baseline settings.
Resources. Recompute and Stint have equal GPU KV capacity. LMCache and HiCache retain their native capacities, so the four-system results compare complete stacks. CPU KV payload budgets are similar, but Stint requires more pinned backing (§ 6.4).
Metrics and repetitions. Throughput counts output tokens from first scheduled arrival through final completion, including drain. TTFT starts at HTTP dispatch; TPOT is each request’s average output-token interval after its first token. ITL pools client gaps between token-bearing chunks across all requests. Each request’s longest gap exposes interruptions hidden by its average. Completion time covers the whole request; arrival-to-first-output also includes admission waiting. Each main comparison uses three matched runs; percentage changes average paired ratios. Figures show every run, with the arithmetic mean and sample SD. Component studies state their own repetition counts.
Long-Output Serving Performance
We first measure how quickly each system completes a fixed amount of output work and how slowly its most affected requests generate tokens. Figure 8 shows throughput and p99 TPOT on all three workloads; Table 2 reports the remaining IMO-AnswerBench metrics, including those that Stint makes worse.
Figure 8. Long-output fixed replays on Qwen3-32B, H200 DP4/TP2, at 4 requests/s. Higher throughput and lower p99 TPOT are better. Dots are three runs; bars and whiskers show mean and sample SD. AIME and LiveCodeBench compare recompute with Stint; IMO-AnswerBench adds LMCache and HiCache. All systems complete the same prescribed output per workload.
| Metric | Recomp. | LMCache | HiCache | Stint |
| Throughput (tok/s) | 4,100 | 4,023 | 3,092 | 4,592 |
| Median TPOT (ms) | 38.2 | 39.5 | 49.7 | 40.9 |
| p99 TPOT (ms) | 104.5 | 107.1 | 93.8 | 51.5 |
| Pooled p99 ITL (ms) | 47.9 | 111.7 | 57.4 | 78.0 |
| Mean TTFT (ms) | 49.4 | 74.7 | 147.4 | 56.7 |
| p99 TTFT (ms) | 67.9 | 535.5 | 567.9 | 246.7 |
| p99 completion (s) | 1,588 | 1,631 | 2,101 | 1,522 |
| Preemptions per run | 510 | 509 | – | 1,376 |
| Host memory (GiB) | 328 | 1,131 | 1,134 | 1,357 |
Table 2. IMO-AnswerBench serving metrics, mean of three matched runs (Qwen3-32B, H200 DP4/TP2, 4 requests/s). Lower is better except throughput. HiCache does not report preemption events. Host memory is the sampled node/cgroup peak, not process RSS.
Throughput and tail TPOT. On IMO-AnswerBench, Stint raises throughput by 11.99%, 14.15% and 48.53% relative to recompute, LMCache and HiCache, and lowers p99 TPOT by 50.69%, 51.86% and 44.96% in every matched run against every baseline. On AIME and LCB, p99 TPOT falls by 34.99% and 74.55% relative to recompute, while throughput changes by −0.45% and +3.64%; AIME’s difference changes sign across runs. Tail TPOT improves on all three workloads, while throughput benefits most from IMO-AnswerBench’s longer outputs. These finite replays measure a fixed amount of work, not sustainable arrival capacity.
What gets worse. Stint preempts 2.7 times as often as recompute, but retains KV in every preemption: it records zero recomputation events. These more frequent interruptions cost typical latency. Median TPOT rises from 38.2 to 40.9 ms, pooled p99 ITL rises from 47.9 to 78.0 ms, and mean TTFT rises by 14.81%, with run-level p99 TTFT of 181–281 ms versus 65–70 ms (Table 2). Section 6.3 shows how these frequent pauses differ from recompute’s rare ones.
Starting and finishing a request. A request can wait long to begin, or finish later even if its average token spacing improves (Figure 9). Stint lowers IMO-AnswerBench’s p99 completion time by 4.16%. The gain is concentrated in the slowest requests. Pairing each request with its recompute run, 74% of the 1,197 requests finish later under Stint, with a median slowdown of 5%; p90 completion time rises by 4%, while the maximum falls by 8%. Stint trades a small delay for most requests against the multi-minute delays of the starved ones. On AIME, mean TTFT falls from 4.69 s to 73 ms; on LCB it stays near 85 ms. AIME and LCB completion-time p99 rises by 1.30% and 2.43%: lower TPOT does not ensure an earlier finish.

Figure 9. Latency tradeoffs in the same runs as Figure 8; lower is better. TTFT starts at HTTP dispatch. Completion time starts at scheduled arrival for AIME and LiveCodeBench, and at HTTP dispatch for IMO-AnswerBench; comparisons are within each workload. Dots show three runs, with mean and sample SD.
Are Stalls Bounded?
KV thrashing starves requests: a request preempted under memory pressure can wait while others keep growing their caches. A low average time per token can hide such a stall. We therefore examine each request’s longest output pause, the largest gap between token-bearing chunks after its first output (Figure 10 and Table 3).
Figure 10. Longest output pause per request in the IMO-AnswerBench replays. Each request contributes one maximum inter-chunk gap, excluding time before its first output. Curves retain the three runs separately. Stint removes the multi-minute stalls while making the typical request’s longest pause larger.
| Recomp. | LMCache | HiCache | Stint | |
| Median (s) | 0.24 | 0.62 | 2.31 | 29.9 |
| p99 (s) | 788 | 813 | 613 | 47.3 |
| Maximum (s) | 808 | 840 | 658 | 49.3 |
| Requests > 60 s | 23.2% | 23.3% | 21.2% | 0% |
| Requests > 300 s | 15.9% | 16.2% | 15.5% | 0% |
| Max. share of lifetime | 72.4% | – | – | 8.4% |
Table 3. Longest output pause per IMO-AnswerBench request, pooled over three runs (1,197 requests per system). Share of lifetime divides the pause by the request’s dispatch-to-completion time.
Baselines leave requests stalled for minutes. Every baseline leaves 21–23% of requests with a pause longer than one minute and about 16% with one longer than five minutes; the longest single pause reaches 11–14 minutes. Under recompute, the 90th-percentile request spends 45% of its lifetime in one pause, and the worst spends 72%. These requests have no path back to the GPU while other requests hold the memory they need.
Stint bounds every stall. No Stint request pauses for more than 49.3 s, and no pause takes more than 8.4% of the request’s lifetime (median 3.1%). Age-based readmission provides this bound. The price is that Stint pauses more requests: 72% of requests have a pause above 10 s, and the median longest pause is about 30 s. These pauses fall on long requests, as prediction-guided selection intends. Dividing requests into output-length terciles, a pause above 10 s affects 30% of the shortest third, 87% of the middle third and 99% of the longest third under Stint, versus 21–31% of each third under recompute.
Stint thus replaces rare, unbounded stalls with frequent, bounded ones. Which is preferable depends on the client. Batch, agentic and evaluation workloads that consume complete answers benefit; an interactive client that streams tokens to a reader sees a pause of about 30 seconds in most long responses. The readmission threshold and protected-turn length tune this tradeoff (§ 6.9). Pooled ITL tells the same story from a different angle: it counts every gap, so Stint’s more frequent pauses raise its p99 even though its worst pauses are far shorter.
Why can throughput improve? Superficially, more preemptions should cost throughput. Stint records 1,344–1,392 swaps and zero recomputations per run, versus 506–514 recomputation preemptions with recompute, yet completes the same output work 12% faster. Saving KV preserves work that would otherwise need rebuilding, while readmission brings stalled requests back into decode batches instead of leaving them idle. Section 6.11 summarizes the component evidence for each mechanism.
Natural Completion and Workload Limits
Natural completion. The tail benefit also appears when requests stop naturally. On 175 LiveCodeBench v6 tasks with a 32,768-token output ceiling, Stint with SRJF delivers 3.46% higher throughput and 68.07% lower p99 TPOT than recompute across three matched runs on one H200 TP2 replica at 1 request/s. Mean arrival-to-completion time falls by 4.16%; median TPOT and p99 TTFT rise by 13.99% and 17.90%. Recompute’s generated programs pass 304 of 525 evaluations, Stint’s physical-largest policy passes 308, and SRJF passes 311; outcomes change in both directions across runs, so this shows neither a quality gain nor a quality loss. Within Stint, SRJF lowers mean completion time by 6.78% relative to physical-largest while reducing throughput by 7.37%. Appendix § 10.§ 1 reports per-run results.
Load sweeps. Figure 11 sweeps the offered rate across four workloads and two models, one run per point. These sweeps predate the repeated comparisons above, and their LMCache curves predate the resume-path repair in Appendix § 18.§ 5; one repaired rerun at 8 requests/s raised LMCache throughput by 7.80% on AIME1000 and 1.71% on LCB880. We therefore use them to locate where Stint helps, not to quantify its margin over LMCache.
Figure 11. Earlier end-to-end fixed-output load sweeps, one run per curve point. Stint curves use Q64; the Llama/AIME star at 8 requests/s uses Q1024 (mean ± population SD, two runs). arXiv and BurstGPT use 2,000-request sweeps. TTFT uses a logarithmic scale. Llama/HiCache on AIME and LCB is unmeasured. LMCache curves predate the resume-path repair.
On Qwen reasoning and coding, Stint’s p99 TPOT advantage appears as soon as preemption begins, at 1 request/s on LCB880 and 2 requests/s on AIME1000, and grows with load: at 4 requests/s its p99 TPOT is 74.81 ms on AIME1000 and 60.31 ms on LCB880, versus 138.09 and 210.10 ms for LMCache. The gains depend on workload. Stint offers no consistent gain on arXiv summarization, whose outputs were capped at 1,024 tokens during collection and leave less decode work over which scheduling can offset its cost. In larger high-rate Qwen/arXiv cohorts, Stint has 19–21% lower throughput than recompute. High-rate BurstGPT has no preemptions in either system and 2.5–3.5% lower throughput with Stint: its added work has no recovery cost to offset.
On Llama/AIME, Stint has lower throughput and higher TPOT than LMCache. Llama has no explicit thinking mode, and many saved AIME outputs repeat text until the generation limit, which may weaken the hidden-state cues for predicting completion; this comparison does not isolate that effect. Long requests also pass repeatedly through short protected turns. Increasing the protection quantum to 1,024 tokens improves throughput with the same predictor (§ 6.9).
Other workloads. Repeated confirmations agree with the sweeps. The Qwen/arXiv, Qwen/BurstGPT and Llama/AIME confirmations do not improve throughput over recompute. On Cachewise’s shared-prefix structural replay, Stint outperforms the measured LMCache and HiCache configurations in throughput and p99 TPOT, but HiCache has lower TTFT. Appendix § 10.§ 3 and Appendix § 10.§ 4 give details.
Host-memory cost. Preserving snapshots requires substantial host memory. On IMO-AnswerBench, Stint’s sampled host-memory peak is 1,357 GiB on average, versus 328 GiB for recompute and 1,131–1,134 GiB for LMCache and HiCache. Most of the difference from the other caches comes from the pinned allocator rounding each 1.5625 GiB per-layer buffer to 2 GiB, which turns an 800 GiB payload into 1,024 GiB of backing (Appendix § 16). This overhead belongs to the current implementation rather than to exact extents, and allocating each buffer at its exact size would remove it.
Contiguity-First CPU KV Pool
We test whether reducing wasted CPU space lets the system retain more snapshots and avoid recomputation. We compare exact extents with uniform 8K-token chunks and a mix of 8K-, 16K- and 32K-token chunks. All three designs store snapshots in contiguous regions, so this comparison tests how well they use the available capacity.
This study uses AIME1000 and the first 800 tasks of the LiveCodeBench replay LCB880, denoted LCB800. Each design serves Qwen3-32B on H200 TP4 with an 800 GiB pool at 4 requests/s, with three runs per configuration (Tables 4 and 5).
Fallbacks count preemptions that use recomputation. Padding divides the unused blocks within snapshot allocations by all their reserved blocks, accumulating both counts across allocations. It excludes the 16-token block rounding shared by all designs.
Exact extents eliminate recorded recomputation fallbacks on AIME1000 and raise throughput by 4.09% over uniform chunks, with one CPU region per offload. On LCB800, exact and uniform designs avoid fallbacks and throughput is nearly unchanged (). Reclaiming padding helps when it retains additional snapshots; once both pools hold them, compact storage adds little throughput. Appendix § 13 reports smaller-pool placement limits.
Table 4. AIME1000 allocator comparison at 800 GiB: mean ± population SD across three runs.
| Allocator | Tok/s | Fallbacks | Padding (%) |
| Uniform 8K | 4603 ± 123 | 521 ± 12 | 68.3 ± 1.6 |
| Mixed sizes | 4615 ± 35 | 1093 ± 82 | 77.8 ± 0.2 |
| Exact extents | 4792 ± 61 | 0 ± 0 | 0 ± 0 |
Table 4.
Table 5. LCB800 allocator comparison at 800 GiB: mean ± population SD across three runs.
| Allocator | Tok/s | Fallbacks | Padding (%) |
| Uniform 8K | 3301 ± 43 | 0 ± 0 | 48.0 ± 1.4 |
| Mixed sizes | 3188 ± 61 | 433 ± 13 | 64.9 ± 1.2 |
| Exact extents | 3293 ± 29 | 0 ± 0 | 0 ± 0 |
Table 5.
Bulk KV Transfer
Retaining a snapshot is useful only if the system can move it efficiently. We compare the time to transfer identical BF16 payloads on H200, including gathering, copying, scattering and completion. Each method has three fresh-process repetitions, with no model computation during the transfer.
Moving the bytes is only part of this work: the system must also prepare the mappings between source and destination blocks. Table 6 therefore reports two timings. “MemCopy” starts after the adapter’s mappings and indices are prepared; “Prepare+MemCopy” includes that preparation. Both include HiCache’s internal descriptor construction.
| Method | MemCopy D2H | MemCopy H2D | Prepare+MemCopy D2H | Prepare+MemCopy H2D |
| HiCache cohort (host A) | ||||
| Stint | 3.680 | 3.231 | 4.243 | 3.961 |
| HiCache layer-first | 3.274 | 3.361 | 8.929 | 9.075 |
| HiCache page-first | 15.793 | 3.360 | 21.948 | 9.086 |
| Local-adapter cohort (host B) | ||||
| Stint | 3.642 | 3.231 | 4.210 | 3.957 |
| Mooncake batch | 4.753 | 4.919 | 27.064 | 27.068 |
| Mooncake fallback | 16.358 | 17.308 | 38.177 | 39.164 |
| SparDA-derived | 16.344 | 4.546 | 52.268 | 6.322 |
Table 6. Table 6. Local KV transfer latency (ms), averaged over three repetitions. Each operation moves 160 MiB of K/V (one layer, 2560 scattered GPU pages). The two cohorts run on different hosts, each with its own Stint control; compare within a cohort. Mooncake adapters are local; SparDA uses 16 CTAs. Standard deviations are in Appendix § 13.
Once mappings are prepared, Stint and HiCache layer-first are close and neither wins both directions: HiCache offloads 11% faster (3.274 versus 3.680 ms), while Stint reloads 4% faster. Both move 46–52 GB/s, so either path nearly saturates the host link. Contiguous extents help mainly by simplifying preparation, since a snapshot needs a few CPU ranges per layer rather than one index per page; with preparation included, Stint takes less than half of HiCache’s time in both directions. Stint is faster than the Mooncake and SparDA-derived adapters in every column of their cohort. Across the full HiCache sweep, Stint wins 72 of 96 large fragmented configurations and 126 of 324 overall, while HiCache often wins at smaller sizes. The Mooncake/SparDA cohort also depends on size and tuning; Appendix § 16 and Appendix § 13 give the full comparisons.
Fused K/V kernels and double buffering. The transfer design reduces two sources of overhead: separate kernel launches for gathering or scattering K and V, and waiting to reuse staging buffers. To examine both, we compare separate and fused kernels with one or two staging pairs, each holding K and V buffers. All four paths use bulk copying. Fusion also batches CPU regions, while double buffering overlaps gather/scatter with DMA. The diagnostic moves 2 GiB across 64 layers between 1,024 scattered GPU blocks and eight CPU regions, gathering on offload and scattering on reload. Using fusion and two staging pairs gives the lowest offload and reload times, about 23.2% and 19.9% below separate K/V kernels with one pair (Table 7). Earlier profiling records 64 gather/scatter launches with fusion versus 1,024 without it, while payload DMA calls remain at 1,024. The gains from fusion and double buffering are not additive.
| K/V kernels | Staging pairs | D2H | H2D |
| Separate | 1 | 60.438 ± 0.121 | 53.065 ± 0.050 |
| Fused | 1 | 48.434 ± 0.136 | 44.170 ± 0.128 |
| Separate | 2 | 47.792 ± 0.079 | 43.140 ± 1.091 |
| Fused | 2 | 46.425 ± 0.022 | 42.503 ± 0.122 |
Table 7. Table 7. K/V kernel and buffering ablation: transfer latency (ms), mean ± population SD of ten operations, not serving runs. D2H offloads KV; H2D reloads it.
Faster local transfers need not translate into higher serving throughput: requests also wait for GPU space and compete for compute. We therefore test the transfer changes during serving as well. The ablation uses 300-request AIME and 200-request LiveCodeBench replays, denoted AIME300 and LCB200. On Qwen3-32B, p99 ITL falls by 4.02% and 8.14%, respectively, in two paired runs per arm (H200 DP2/TP1, 4 requests/s). Throughput changes are small (−0.96% and +0.33%) and vary in sign across seeds; maximum pauses increase in three of four pairs. Faster transfers help recovery, while scheduling determines when requests can use the restored state.
Remaining-Length Prediction
Victim selection needs an ordering by remaining work. We compare length predictions from Stint with DSMN (single-layer hidden states), LTR [9] (OPT-350M), S3 (DistilBERT length buckets) and PiA (target-model LoRA); the latter three use prompts. All use Qwen3-32B generations and validation-selected checkpoints, with no online training. The plotted AIME300/LCB200 populations combine validation and test requests used during development.
Accuracy. Stint ranks AIME lengths best (), although LTR has lower absolute error. On LCB, Stint (, MAE 4,744 tokens) is close to the leading PiA predictor (, MAE 4,660). Hidden states provide useful ordering information, with prompt-time prediction remaining competitive on code. Appendix § 14 gives the complete regression results.
Serving cost. Sparse prediction has a small aggregate cost: enabling K64 reduces throughput by 0.58% on AIME1000 and 0.26% on LCB880 across three Qwen runs at DP4/TP2 and 4 requests/s. We keep physical-largest victims, storage, transfers and recovery fixed, computing predictions without using them. This includes scheduling effects as well as predictor execution.
A separate three-run K1/K64/K128 study on Qwen/LCB880 shows that K64 makes 97.13% fewer batched prediction calls than K1. Serving changes are small and mixed, and call counts do not measure time saved. The interval trades inference cost against decision freshness; Appendix § 14.§ 2 reports the complete sensitivity results.
Prediction-Guided Victim Selection
Producing length estimates has a cost; using them changes which requests keep GPU memory. We separate these effects by enabling prediction in both configurations and changing only victim selection: SRJF uses predicted remaining length, while physical-largest uses the memory an eviction would release.

Figure 12. Output-length predictions on common step-500 survivors: 300 AIME requests (top), and 199 of 200 LiveCodeBench requests (bottom). Stint predicts at step 500; LTR, S3 and PiA use prompts. Kendall’s measures ordering; measures numerical fit. Higher is better for both; 1 is perfect. DSMN results are in Appendix § 14.
In the same three-run controls, SRJF lowers p99 ITL by 13.86% on AIME and 14.57% on LiveCodeBench, but reduces throughput and increases p99 TPOT. Here, the tail of individual output gaps improves at the cost of overall output rate and the tail of per-request average token spacing. Appendix § 14.§ 1 gives the full results.
Prediction-guided victim selection therefore offers a scheduling tradeoff within Stint; these controls do not explain the full system’s throughput gains by prediction alone. Both policies retain the same CPU storage, transfer and recovery mechanisms. We examine recovery next.
Age-Based Protected Recovery
Readmission versus protection. Recovery must bring a request back and give it time to produce output. We compare disabled recovery, age-only readmission and age-5/Q64 protection on 50 LCB requests with Qwen3-32B, H200 TP2 and reduced GPU KV capacity. Each mode uses SRJF/K64 and identical transfers, with one run per mode.
Protection raises throughput and reduces offload cycles relative to age-only readmission (Table 8). Age-only shortens waits but raises offloads from 71 to 494; protection reduces them to 286 and increases throughput by 1.99% relative to age-only.
| Mode | Tok/s | Wait (s) | Offloads | Gap > 10 s |
| Disabled | 475.43 | 45.11 | 71 | 22% |
| Age-only | 484.45 | 5.20 | 494 | 0% |
| Protected(Q=64) | 494.10 | 13.75 | 286 | 68% |
Table 8. Qwen/LCB recovery controls, one run per mode. Wait is p99 eviction-to-next-fresh-output time. Gap > 10 s is the fraction of requests whose longest client chunk gap exceeds 10 s.
Age-only still has the lowest wait p99, and protection leaves more requests with gaps above 10 s: one request’s protected progress makes others wait. All modes record zero repeated evictions without fresh output, so this sample does not establish that protection is needed to prevent zero-progress evictions.
Scheduling dominates the observed recovery wait: time before reload is scheduled accounts for 98.12–99.70% of summed eviction-to-next-token intervals across the three modes. Protection also reduces repeated evictions from 458 to 251, although short decode turns remain. This supports improving readmission alongside transfer speed. The overlapping request intervals are not a wall-clock breakdown of RAM residence or DMA, and this small study does not quantify protection’s role in the full DP4 gains.
Length of the protected turn. Longer protected turns can sharply reduce repeated rescues. On Llama/AIME1000 at DP8/TP1 and 8 requests/s, increasing Q64 to Q1024 lowers mean repeat rescues from 4,622.5 to 37.5 per run. Across two runs per arm, throughput rises 16.68% and p99 TPOT falls 16.30%. The mean run-maximum streaming gap rises from 143.43 to 400.71 s, and Q1024 still trails same-node recompute in throughput and p99 TPOT. This tuned development replay shows both the value of protected progress and its waiting cost. Appendix C.2 retains the full counters and controls.
How recovery affects throughput. Readmission can also keep decode batches fuller. In two Qwen/SRJF comparisons with matched seeds, Q64 raises throughput by 3.93% and 6.78%, while mean decode batches grow from 11.91 to 12.50 and from 12.58 to 13.77 requests. Both pairs have more offloads and no recomputation events, so fewer transfers cannot explain the gains. Larger batches are consistent with the improvement; GPU utilization was not measured. The second pair has higher mean TPOT and p99 ITL, and other victim policies have mixed throughput effects; Appendix C.3 gives the full controls.
Admission Control and Continuing Arrivals
Stint also avoids the long admission wait of a simple concurrency cap. FIFO64 keeps the recompute engine fixed and admits at most 64 dispatched-but-unfinished requests per replica, or 256 across DP4; the rest wait in an external FIFO queue. This is one policy, not a tuned upper bound on admission control.
Across three AIME and LCB runs, Stint has higher throughput and much earlier first output than FIFO64. Mean arrival-to-first-output time is 0.074 s and 0.087 s with Stint, versus 260.63 s and 225.19 s with FIFO64. The cap achieves lower p99 TPOT after admission, while dispatch-based TTFT omits its queue. Both endpoints are needed to locate waiting; Appendix B.1 gives the full comparison.
The tail TPOT benefit persists with continuing arrivals. We replay LCB at 0.25, 0.5 and 1 request/s for 900 seconds per phase, then drain every request. Across three runs, Stint lowers p99 TPOT by 47.10% relative to recompute, with −0.28% throughput and +6.60% p99 completion time. Requests span phase boundaries, precluding independent steady-state capacity estimates. Appendix B.2 reports phase and drain results: the tradeoff between progress and waiting extends beyond a single arrival burst.
Where Do the Gains Come From?
Table 9 collects the controlled comparisons above. Each changes one mechanism inside Stint while holding the others fixed, but on smaller or different workloads than the main comparison. We have not ablated the full system on IMO-AnswerBench, so no row explains its 12% throughput gain by itself.
| Change | Workloads | Runs | Throughput (%) | Other effect |
| Exact extents vs. uniform 8K | AIME1000, LCB800 | 3 | +4.1, −0.3 | AIME fallbacks 521 0 |
| Fused K/V, two pairs vs. legacy | AIME300, LCB200 | 2 | −1.0, +0.3 | p99 ITL −4%, −8% |
| Prediction enabled but unused | AIME1000, LCB880 | 3 | −0.6, −0.3 | – |
| SRJF vs. physical-largest victims | AIME1000, LCB880 | 3 | −7.9, −7.4 | p99 ITL −14%, −15% |
| Protected recovery (Q64) vs. none | LCB50, 2 pairs | 1 | +3.9, +6.8 | mean decode batch +5%, +9% |
| Protected vs. age-only readmission | LCB50 | 1 | +2.0 | offloads 494 286 |
| Protected turn Q1024 vs. Q64 | Llama AIME1000 | 2 | +16.7 | max. gap ×2.8 |
Table 9. Controlled comparisons within Stint. Each row changes one mechanism and holds the others fixed. Throughput changes are relative to the second configuration; paired values follow the workload order. Runs are per arm.
Three patterns emerge. First, retaining more snapshots and readmitting stalled requests raise throughput where they apply: exact extents help when they avoid recomputation fallbacks, and protected recovery keeps decode batches fuller. Second, faster transfers mainly shorten output gaps; they barely change throughput because requests spend most of their recovery time waiting to be scheduled rather than moving data (Section 6.9). Third, prediction-guided selection lowers throughput by 7–8% in exchange for 14–15% shorter p99 ITL. It acts as a latency policy rather than a source of throughput: a deployment that values throughput can choose physical-largest victims while keeping the same storage, transfer and recovery mechanisms, as the natural-completion study also suggests (Section 6.4). An ablation that adds mechanisms one at a time on IMO-AnswerBench would attribute the full-system gain directly; we leave it to future work.
Related Work
We compare how prior systems schedule requests, predict remaining work and manage KV during recovery.
LLM Serving Systems
Orca [44] schedules at each model iteration. Sarathi-Serve [2] and DeepSpeed-FastGen [13] combine prompt chunks with decoding to improve batching. Splitwise [25] and DistServe [48] place prefill and decode on separate GPU pools; Llumnix [35] reschedules requests across instances. Stint focuses on repeated eviction and recovery within a worker.
FastServe [39] uses multilevel feedback queues and service quanta, promotes starved requests, and proactively swaps KV based on estimated next execution time. SuperInfer [45] rotates requests according to SLO-derived urgency and current memory and transfer budgets. TokenFlow [4] uses generated-output buffers and client consumption rates to coordinate admission, preemption and resumption. InferCept [1] pauses requests during external calls and chooses among preserving, discarding and swapping their KV to minimize wasted memory and recomputation. FastSwitch [30] reduces the overhead of preemption-induced context switching in fairness-aware serving, including transfers of fragmented KV blocks. CacheOPT [31] orders preemption victims by latency SLO and estimated remaining time, and chooses between swapping and recomputation. These systems decide how to preempt or how to switch contexts cheaply. Stint adds a guarantee about what happens afterward: stall age bounds how long a preempted request waits, age survives reload or recomputation, and only fresh output counts toward the protected quantum. Our controls evaluate these rules within Stint; we do not benchmark these complete systems.
KV Cache Management Systems
KV Cache Compression. KIVI [21] reduces KV precision, while HO [46] and Scissorhands [20] retain selected tokens. These methods reduce stored state and can change attention results. Stint preserves KV precision and restores the full available state for dense attention, rebuilding discarded state if needed.
Inference on Sparse KV Cache. StreamingLLM [40] retains attention sinks and a sliding window. Quest [36] selects pages by query similarity; InfiniGen [17] predicts important tokens and prefetches their KV. These methods skip or discard entries, so outputs can differ from dense attention. SparDA [8] predicts the next layer’s KV block choices and prefetches them during the current layer’s execution. Our SparDA-derived adapter measures equal-payload transport only, excluding its sparse attention, prediction and overlap with model computation.
Output Length Prediction
Length estimates for scheduling can come from prompts [9, 27] or intermediate model states. TRAIL, from Don’t Stop Me Now [29], feeds target-model embeddings to a lightweight remaining-length classifier and uses its predictions to order requests in a preemptive scheduling policy. Stint uses a multilayer regression head with sparse online inference, and applies its estimates only to choose which running request to evict under memory pressure. Age-based recovery then readmits victims regardless of prediction, which bounds the cost of a wrong estimate. § 6.7 compares predictors on the same recorded populations.
KV Cache Offloading
FlexGen [32] plans GPU–CPU–disk placement for batch inference. LoongServe [38] uses elastic sequence parallelism; Infinite-LLM [18] distributes KV storage. Mooncake [26] separates prefill and decode, using shared CPU/SSD KV storage and cache-aware request placement.
LMCache [19] gathers scattered GPU pages into chunks and pipelines prefix-aware KV storage and loading with execution. It also supports delayed saving of decode KV, disabled in our measured configuration (§ 6.1). Stint also packs and pipelines transfers, but allocates CPU space around each snapshot’s known length.
SuperInfer’s DuplexKV [45] makes a KV block’s layers contiguous and batches transfers on GH200. Stint retains paged GPU KV and uses bounded exact extents with matching offsets in each layer’s CPU buffers. Two staging pairs overlap gather/scatter with copying. Our local transfer measurements do not compare the complete SuperInfer system.
Pie [41] prefetches upcoming layers’ CPU KV during computation. TokenFlow [4] chooses reload versus recomputation by estimated cost and overlaps loading with eviction. Stint offloads on GPU allocation failure when CPU snapshot allocation succeeds, otherwise falling back to recomputation. Protected recovery then gives restored requests time to generate fresh output.
Earlier workshop version. An earlier three-page workshop paper [33] described cascading preemption, hidden-state length prediction, an analytic recomputation/offload tradeoff, and asynchronous layerwise transfers with packing. This paper adds the contiguity-first CPU pool and exact-extent allocator, age-based protected recovery, and sparse predictor execution. It also adds repeated system comparisons, component controls and studies of natural completion, output pauses and workload limits.
Discussion and Limitations
Who benefits. Stint helps when outputs are long enough that requests are preempted while much decoding remains. It then shortens the tail of slow requests and bounds every stall, at the cost of more frequent pauses of about 30 seconds for long requests (Section 6.3). Batch, agentic and evaluation workloads that consume complete answers gain the most. An interactive client that streams tokens to a reader may prefer shorter, more frequent rescues, which a lower readmission threshold or a different protected-turn plan can provide. Short-output workloads gain nothing: without preemption, prediction and scheduling add work but save no recovery.
Scope of the evaluation. We evaluate dense models (Qwen3-32B and Llama-3.1-8B) with up to two GPUs per replica, or four in the allocator study, on vLLM 0.9.2. Reasoning deployments increasingly use mixture-of-experts models with compressed or latent attention. Our cost model suggests that preserving their smaller per-token state remains favorable (§ 4.2), but we have not measured them. Disaggregated serving places decoding on dedicated instances where the same memory pressure arises; combining Stint with prefill–decode disaggregation is future work. The main comparisons use finite arrival bursts at one rate; the earlier load sweeps (Figure 11) and continuing-arrival study (§ 6.10) cover other rates with fewer runs.
Fixed-output replay and predictor exposure. Fixed-output replay gives every system identical work, but it does not measure answer quality, and replayed greedy trajectories diverge from the generation that defined each length (§ 6.1). The predictors were trained or validated on tasks that also appear in the serving workloads. The 299 IMO-AnswerBench questions outside the training set show the same direction of change, and the natural-completion study shows the tail benefit with real termination, but neither is a fully held-out generalization test.
Attribution. Component studies change one mechanism at a time on smaller or different workloads (§ 6.11). Several recovery studies use one or two runs and reduced GPU capacity. An ablation on IMO-AnswerBench that adds mechanisms one at a time would attribute the full-system gain directly.
Baselines. Baselines run their native configurations. The HiCache comparison also spans the SGLang and vLLM engines, and LMCache’s measured configuration includes a resume-path repair (Appendix § 18.§ 5). We compare against FastServe, TokenFlow, SuperInfer, InferCept and CacheOPT only in related work.
Host memory. Retaining snapshots needs host memory in proportion to the preempted KV. In the current implementation, allocator rounding adds 224 GiB to the 800 GiB payload, which roughly matches Stint’s excess host memory over LMCache and HiCache. Each worker also manages its own pool; sharing host capacity across replicas could help when pressure is uneven, at the cost of coordination.
Conclusion
Long decoding makes preemption unavoidable, and existing systems let preempted requests stall for minutes. Stint makes preemption productive: it preserves a preempted request’s work in a contiguity-first CPU pool, moves it back with bulk transfers, bounds how long the request waits, and gives it fresh output after it returns. On olympiad-math replays, this completes the same work faster with a much shorter tail of slow requests, and no request stalls for more than a minute. The price is host memory and more frequent, bounded pauses. Preserving state and guaranteeing time to use it must be designed together.
Guide to the supplementary material. The first sections expand the main serving results and explain the roles of recovery, storage, transfers and prediction. The cost model and exact settings follow. Earlier extensions and historical experiments are kept separate from the current system’s evidence.
| Evidence or detail | Section |
| Natural completion and workload scope | A |
| Admission waiting and continuing arrivals | B |
| Recovery, protected turns and batching | C |
| CPU allocation and KV transfers | D |
| Prediction accuracy, policy and interval | E |
| Recompute–offload cost model | F |
| Evaluation settings and implementation | G |
| Earlier online-training extension | H |
| Historical serving experiments | I |
| Earlier predictor variants | J |
Workloads include American Invitational Mathematics Examination (AIME) math problems, LiveCodeBench (LCB) programming tasks, and IMO-AnswerBench olympiad problems [22]. AIME1000 and LCB880 denote the main 1,000- and 880-request replays. Numeric suffixes give replay request counts, not repetition counts; each smaller study states its own population.
Additional Serving Results
The natural-completion study extends the tail-TPOT result beyond fixed output replay. We then examine predictor training exposure and additional workloads to explain where the serving benefits apply. These comparisons retain their task outcomes, latency tradeoffs and resource costs.
Natural Completion and Coding Quality
Fixed-length replay holds output work constant, but real programs can finish early or reach the generation limit. We therefore serve 175 new LiveCodeBench v6 tasks (LCB175) with natural EOS, then run their program tests. Recompute, physical-largest Stint (P0), and SRJF Stint (S) each run three matched arrival seeds. The setting is Qwen3-32B, one H200 TP2 replica, 1 request/s, a 32,768-new-token ceiling, and 200 GiB CPU KV for P0/S. Both Stint arms use age-5/Q64 recovery; only S runs the frozen predictor.
Against recompute, S reduces p99 TPOT by 68.07%. Its mean completion time from arrival falls 4.16%, and throughput rises 3.46%. Median TPOT instead increases 13.99%, and p99 TTFT increases 17.90%. P0 has higher throughput than S in every block. Compared with P0, S has 7.37% lower throughput and 4.14% higher p99 TPOT, but 6.78% lower mean completion time. Prediction-guided ordering therefore offers a tradeoff in this study; it does not improve every endpoint over the same recovery system with physical-largest victims.
Pass counts vary in both directions (Table 10). Of 67 recompute/SRJF pass–fail changes, 60 occur when neither response hits the output limit. The results therefore establish neither better nor equivalent task quality. They are also not an identical-work comparison: natural output amounts differ across arms.
| Arrival seed | Recompute | P0 | SRJF |
| 30 | 103 | 103 | 98 |
| 31 | 100 | 100 | 107 |
| 32 | 101 | 105 | 106 |
Table 10. Programs passing the offline tests, out of 175 tasks per run. All evaluations have a valid outcome. Repeated seeds do not create new independent tasks.
As an additional descriptive metric, we divide passing responses by full serving time, including time spent on unsuccessful responses. S improves this quantity by 6.62% over recompute on average, but the three changes are −8.35%, +7.96% and +20.26%. It is 5.12% lower than P0 on average. This excludes offline scoring time and is not whole-agent workflow throughput.
Both Stint arms record no preemption before a protected turn ends, but repeated rescues remain common: 955 for P0 and 913 for S across the three runs. Mean selection-time stall ages are 38.94 and 44.16 s; mean rescue-acquisition-to-next-token times are 0.1640 and 0.1653 s. These are separate histograms, not a joined breakdown of RAM waiting and transfer time. The 5-second threshold makes a request eligible for rescue; it does not bound its pause to five seconds.
IMO-AnswerBench Training Exposure
We use all 399 usable IMO-AnswerBench questions to create enough concurrent long decodes for the serving comparison. A fixed random draw (seed 20261003) selects 100 questions for predictor training. Within those 100, an 80/20 split selects the training epoch using validation MAE; a fresh head then refits all 100 for that many epochs. Qwen selects 22 epochs. The head remains frozen during the four-system serving runs. Llama and Hunyuan heads were also trained, but this study reports serving results only for Qwen.
The shared output-length freeze comes from one natural-completion run. It retains all 30 requests that reach the generation limit. Median output length is 17,089 tokens; 213 of 399 requests exceed 16,384. Subsequent system comparisons replay the same 7,279,811 output tokens, so these are performance measurements rather than a new answer-quality comparison.
Table 11 reports the two groups separately within the same three recompute/Stint pairs. Both groups share scheduling and memory pressure. Of the 299, 28 were used in an earlier calibration, so training-unseen does not mean fully held out.
| Group | Mean TTFT | Mean TPOT | p99 TPOT | Mean dispatch-to-completion | p99 dispatch-to-completion |
| Training100 | +7.42% | +1.34% | −36.47% | +3.85% | −3.52% |
| Training-unseen299 | +17.37% | −8.45% | −53.56% | −4.10% | −4.16% |
Table 11. IMO-AnswerBench subgroup changes with Stint relative to recompute. Percentages average three within-block ratios; negative latency changes are better. Both groups share each full 399-request replay.
Both groups have lower p99 TPOT, while their mean completion-time changes differ. Their TTFT and median
Figure 13. Natural-completion LCB175: serving performance and coding outcomes across three arrival seeds. Circles show individual runs; diamonds and whiskers show means and population standard deviations. The same 175 tasks repeat, but generated lengths differ across arms.
TPOT worsen on average. The grouping does not isolate predictor benefit: offloading and recovery also differ between systems. Nor does it measure predictor accuracy or held-out generalization. We report throughput only for the full replay, because the two groups share the same execution resources and duration.
Selected Configurations on Other Workloads
We use the best-supported settings from the separate tuning study, then compare each with unchanged recompute in three additional runs. Each comparison reuses its frozen tasks, output lengths, arrival schedule and routing. These experiments test the selected configuration; they do not isolate every optimization or re-tune the baseline.
_Qwen/arXiv: fewer preemptions do not ensure better tails._ This study uses 2,000 requests at 16 requests/s on H200 DP4/TP2. The selected setting waits for enough space to admit the full prompt, reserves headroom of 64 blocks, and sets the long-prefill threshold to zero. Qwen thinking is disabled when preparing these prompts. Compared with recompute, throughput changes by −0.57%, mean TTFT falls 25.07%, and mean arrival-to-completion falls 16.96%. However, p99 TTFT rises 5.73%, p99 TPOT rises 6.51%, and p99 completion time rises 10.53%. Preemptions fall by 98.60%, yet throughput does not improve. This shows why preemption count alone cannot explain performance.
_Llama/AIME: longer protected turns reduce extreme pauses._ This comparison uses 1,000 requests at 16 requests/s on H200 DP8/TP1, with 54,257 GPU KV blocks per worker. The selected Stint setting uses , age 5 s, a 1,024-token protected turn, early root disabled, and 800 GiB CPU KV. Throughput changes by +0.10%, while the p99 of per-request longest output gaps falls 54.63%. Mean TPOT, p99 TPOT and p99 completion time instead increase 3.89%, 2.74% and 2.62%. The small throughput difference varies in sign across runs; it does not establish either a general gain or equivalence.
_Qwen/BurstGPT: limited benefit without long output pauses._ This comparison uses 2,000 synthetic length-matched requests at 32 requests/s on H200 DP4/TP2. Prediction turns on above 80% KV pressure and off below 60%, with 64-step hysteresis. The selected setting remains 2.04% below recompute throughput, with 5.20% higher mean TPOT and 0.72% higher p99 TPOT. The longest output gaps stay below one second in both systems. This workload leaves little scope for reducing the severe interruptions that motivate Stint.
Cachewise: Shared Prefixes and Short Outputs
Cachewise coding-agent traces contain redacted request text. We therefore replay synthetic tokens that preserve the recorded lengths, known shared prefixes and parent–child dependencies. Each run contains 1,418 calls in 65 sessions and 94 root branches. Roots arrive at 4 branches/s; a child becomes eligible only after its own parent succeeds, with no added think time. Thus, 4 roots/s is not 4 total requests/s, and child release times depend on the serving system’s progress.
Figure 14. Three-run confirmation of the selected configurations. Each row retains its own workload and arrival rate. Points are runs; black ticks and whiskers show means and population standard deviations. First-output delay uses a logarithmic axis; the other axes are linear.
The four systems run three matched blocks on Qwen3-32B/H200 DP4/TP2. They complete the same 151,247 output tokens per run, averaging only 106.66 tokens per call. Stint uses physical-largest victims with prediction and feature capture disabled, retaining age-5/Q64 recovery and 800 GiB CPU KV. The other cache systems use the settings in Appendix § 16.
Against recompute, Stint changes throughput by −0.89%, mean TTFT by +1.14%, and p99 TPOT by +2.59%. It shows no advantage in these metrics. Against LMCache, throughput rises 5.33% and p99 TPOT falls 9.87%. Against HiCache, throughput rises 52.97% and p99 TPOT falls 97.13%, but mean TTFT rises 95.73%. HiCache’s faster first output remains an important counterpoint to its longer later delays in this replay. Mean sampled memory-guard peaks are 330.12, 1,133.69, 1,135.16 and 1,358.39 GiB for recompute, LMCache, HiCache and Stint, respectively. These are node/cgroup peaks, not isolated pool measurements.
All nine recompute/LMCache/Stint runs record zero preemptions; HiCache’s requested counters are unavailable. Zero preemptions do not imply that no CPU lookup or reload occurs. The study provides a low-pressure, shared-prefix comparison, not evidence of the thrashing mechanism, predictor generalization or original coding-task quality. Each run has 1,294 defined TPOT samples; the 124 single-output-token calls are excluded rather than assigned zero TPOT.

Figure 15. Cachewise structural replay, three matched blocks per system. Bars show means with sample standard deviation; points retain individual runs. Eligible-to-completion includes waiting after a parent permits a call, while session makespan also includes the dependency chain.
Admission Waiting and Continued Arrivals
Shorter pauses after a request starts do not necessarily mean a shorter response. Admission control can improve streaming by keeping requests outside the engine. These experiments account for that waiting and then ask whether the tradeoff persists while new requests continue to arrive.
A FIFO Admission Control
We compare recompute, FIFO64 and Stint on AIME1000 and LCB880. Each arm has three matched runs on Qwen3-32B, H200 DP4/TP2, at 4 requests/s. FIFO64 changes only client admission: each replica allows 64 dispatched but unfinished requests. The recompute runtime stays fixed. All arms complete the same inputs and prescribed output lengths.
FIFO64 has lower p99 TPOT than Stint, but its throughput is lower and its mean arrival-to-first-output time is several minutes (Table 12). Measuring TTFT only after dispatch would hide most of this delay. Stint starts output sooner by keeping more requests in flight, at the cost of substantially more host memory and less regular streaming than FIFO64. This compares one FIFO limit; it does not establish that protected recovery beats every admission policy.
Continued Arrivals
The finite main replay eventually stops admitting work. We therefore repeat the three-system comparison while requests arrive for 2,700 seconds: 900 seconds each at 0.25, 0.5 and 1 request/s, followed by a drain. The model, DP4/TP2 topology and storage/recovery settings remain the same. Three matched seeds complete 1,591, 1,577 and 1,583 requests per arm, with no censored requests.
Relative to recompute, Stint reduces p99 TPOT by 47.10% and the longest observed output gap by 80.55%. Whole-run throughput changes by −0.28% on average, with mixed changes across the three pairs. The tradeoff remains. Completion p99, measured from arrival, increases 6.60%; pooled p99 ITL increases 4.83%; and median TPOT increases 1.67%. Smaller extreme pauses do not mean every response finishes sooner.
Stint delivers 1.07% more output per second during the arrival period, but its drain is longer in two of the three runs. At the end of arrivals, Stint/recompute have 413/416, 416/437 and 385/382 unfinished requests. FIFO64 further reduces streaming pauses while adding 55.11–65.40 s of mean admission waiting. That waiting stays in the arrival-based metrics. These are finite, changing-load replays: overlapping cohorts prevent us from treating each phase as an independent steady state or a capacity limit.
| Workload | System | Output tok/s | Mean arrival-to-first-output (s) | p99 TPOT (ms) | Memory guard peak (GiB) |
| AIME1000 | Recompute | 7,741.27 | 4.687 | 118.26 | 417.18 |
| FIFO64 recompute | 6,493.93 | 260.629 | 32.16 | 416.35 | |
| Stint | 7,701.16 | 0.074 | 76.86 | 1,446.90 | |
| LCB880 | Recompute | 5,007.50 | 0.084 | 225.14 | 451.33 |
| FIFO64 recompute | 4,703.59 | 225.188 | 39.99 | 451.01 | |
| Stint | 5,189.30 | 0.087 | 57.16 | 1,481.02 |
Table 12. FIFO admission reduces post-admission TPOT but adds waiting before output begins. Values average three runs. Arrival-to-first-output includes the admission queue; the memory column is the sampled node/cgroup guard, not process RSS or CPU KV capacity.

Figure 16. Three repeated runs with continued arrivals followed by a drain. Each column shows one matched block: delivered output, unfinished requests, and the client admission queue. Output rates use 10-second windows. Earlier phases can leave requests active in later phases.
Additional Recovery Evidence
Age-based readmission and protection address different parts of a request’s return to decoding. The main paper compares their effects on waiting and repeated offload. Here we show the pause distributions, longer protected turns, and batching observations behind that discussion. These studies also show how progress for one request can increase waiting for others. Their configurations and repetition counts remain separate.
Historical Recovery Across Pause Thresholds
For each request, we take its longest observed gap between token-bearing client chunks. This excludes waiting before the first output. Figure 17 plots the fraction of requests whose longest gap exceeds each threshold. It uses every observed maximum, rather than selecting a single latency target.
Figure 17. Longest client-chunk gaps in the six LCB recovery controls. Each curve is one complete run of 50 requests; the x-axis is logarithmic. Protected recovery uses age 5 s and a 64-fresh-token quantum. A lower curve means fewer requests exceed that pause threshold. In these runs, protected recovery reduces the longest pauses while increasing the fraction of requests with moderate pauses.
This earlier reduced-capacity study is separate from the repeated full-deployment experiments. All six runs use Qwen3-32B BF16, one H200 TP2 replica, 11,251 GPU KV blocks per worker, and 100 GiB CPU KV per worker. They replay the same 50 requests and 544,661 output tokens, using replica 0’s offsets from the finite Poisson 16-request/s DP4 schedule with seed 10. K64 prediction and the transfer path stay fixed. Each mode has one observation; the curves describe those requests, not uncertainty across repeated runs.
With SRJF, protected recovery raises the fraction above 10 s from 22% to 68%, while reducing the fraction above 30 s from 16% to zero. Queue-tail and physical-largest show the same direction at these thresholds. Thus, reducing the longest pauses can come with more frequent moderate interruptions. These distributions do not establish a waiting-time bound. The separate age-only comparison in the main paper shows that readmission without protection can have still shorter waits, at the cost of more offloads in that observation.
Earlier Protection-Quantum Controls
Give a recovered request enough time to progress. This earlier H200 study isolates the protected-turn setting; the later three-run selected-configuration confirmation is reported in Appendix A.3. A short protected turn can still be followed by another recovery cycle. To test this, we change the fresh-output quantum from 64 to 1,024 tokens on Llama-3.1-8B/AIME1000. This full-deployment study uses eight H200 GPUs at DP8/TP1, finite Poisson arrivals at 8 requests/s, and an 800 GiB aggregate CPU KV pool. Each Stint arm and a same-node recompute control have two complete runs.
The longer turn reduces mean repeat rescues from 4,622.5 to 37.5 per run. Swap preemptions fall from 5,348.5 to 889.5, and recomputation preemptions from 1,143.5 to 8.5. Throughput rises from to tokens/s (+16.68%), while p99 TPOT falls from to ms (−16.30%). Here, changing the protected turn produces fewer recovery cycles together with higher throughput. The counters do not measure how much GPU time those avoided cycles save.
The cost appears in the longest pauses. The mean run-maximum streaming gap rises from 143.43 to 400.71 s. Q1024 also remains 2.81% below same-node recompute throughput and 2.37% above its p99 TPOT, although its maximum-gap summary is shorter than recompute’s 910.45 s. These controls isolate the protected-turn setting; the later confirmation of the selected configuration appears in Appendix A.3.
The predictor, K64 cadence, transfer path, GPU capacity and 5 s age threshold stay fixed. Runs use the same seed-10 replay, with arm order reversed between repetition blocks. Summaries report arithmetic means and population SD; p99 summaries average run-level p99 values. Tuning and confirmation use the same population, whose 413 generation-capped profiles supply 96.44% of output tokens. This result supports balancing repeated recovery against other requests’ waiting; it does not establish one quantum for all workloads or a waiting-time bound under continued arrivals.
Recovery, Batching and Victim Policies
Throughput can also change through batching. In the seed-10 Qwen/SRJF recovery comparison, Q64 increases throughput by 3.93% over disabled recovery while increasing the recorded mean decode batch from 11.91 to 12.50 requests. A second within-seed pair shows the same direction: throughput rises by 6.78%, the mean batch grows from 12.58 to 13.77, and eviction-wait p99 falls from 77.32 to 28.59 s. Both pairs record more offloads and no recomputation events. Fewer transfers or recorded recomputations therefore do not explain these gains. Larger decode batches are consistent with the improvement, but these measurements do not isolate GPU utilization.
The second pair also has latency costs: offloads rise from 141 to 454, requests with gaps above 10 s increase from 46% to 86%, mean TPOT rises 15.10%, and pooled p99 ITL rises 46.47%. Its p99 TPOT falls 16.58%. These metrics describe different aspects of the request experience; a lower eviction-wait p99 does not mean every latency measure improves.
Recovery also shortens waits with physical-largest victims. The eviction-wait p99 falls from 42.66 to 13.51 s at seed 10, and from 82.68 to 24.50 s at seed 11. Throughput changes by +0.85% and −0.51%, and gaps above 10 s affect more requests in both pairs. Seeds route different subsets to replica 0, and the initial screen and follow-up differ in concurrent lane activity. We compare treatments within each setting, without pooling them as identical-workload repetitions. The unflushed terminal batch sample is excluded from batch summaries. With queue-tail eviction, recovery reduces the mean batch from 13.86 to 12.46 and throughput by 9.90%. The initial screen uses single seed-10 pairs at reduced GPU capacity (11,251 blocks per worker), with 100 GiB CPU KV per worker, using one replica’s offsets from the original finite Poisson 16-request/s DP4 schedule. The scheduler summaries are consistent with a batching explanation; they do not measure GPU utilization or establish its share of the end-to-end gains.
CPU KV Pool and Bulk KV Transfer
The CPU pool and transfer engine support the same recovery path: retain a request’s state, then move it back when the scheduler makes room. We first examine storage under greater CPU pressure, then the broader transfer comparisons. Each study retains its own configuration and repetition count.
Allocation Under Greater CPU Pressure
A smaller CPU pool. A separate AIME1000 study reduces the pool to 200 GiB on H200 TP4 at 4 requests/s. Across three runs, exact extents reduce recomputation fallbacks from to relative to uniform 4K chunks, while throughput rises from to tokens/s. Exact extents use 1.04 regions per offload on average, although some allocations still fail because free space is fragmented.
This pressure study also limits each snapshot to eight regions. Uniform 4K chunks therefore cannot represent snapshots above 32K tokens within that limit. The arms run concurrently on different fixed NUMA/GPU placements, so the throughput change cannot be assigned solely to the allocator. Fewer fallback events show that more state was retained; they do not measure how much GPU recomputation time was saved.
Additional Transfer Comparisons
Table 13 reports the two transfer cohorts of the main paper’s table with their standard deviations, each with its own matched Stint control. Both use CUDA 12.8, but they run on different H200 hosts and use different GPU/NUMA placements. Comparisons stay within a cohort and direction.
| Method | Prepared D2H | Prepared H2D | With preparation D2H | With preparation H2D |
| HiCache cohort: H200 host A, CUDA 12.8 | ||||
| Stint | 3.680 ± 0.001 | 3.231 ± 0.001 | 4.243 ± 0.009 | 3.961 ± 0.007 |
| HiCache layer-first | 3.274 ± 0.002 | 3.361 ± 0.002 | 8.929 ± 0.068 | 9.075 ± 0.086 |
| HiCache page-first | 15.793 ± 0.009 | 3.360 ± 0.001 | 21.948 ± 0.012 | 9.086 ± 0.054 |
| Local-adapter cohort: H200 host B, CUDA 12.8 | ||||
| Stint | 3.642 ± 0.003 | 3.231 ± 0.002 | 4.210 ± 0.009 | 3.957 ± 0.003 |
| Mooncake local batch | 4.753 ± 0.010 | 4.919 ± 0.015 | 27.064 ± 0.776 | 27.068 ± 0.770 |
| Mooncake local fallback | 16.358 ± 0.027 | 17.308 ± 0.075 | 38.177 ± 0.440 | 39.164 ± 0.481 |
| SparDA-derived (16 CTAs) | 16.344 ± 0.070 | 4.546 ± 0.003 | 52.268 ± 0.335 | 6.322 ± 0.015 |
Table 13. Local KV transfer latency in milliseconds (mean ± population SD, three repetitions). Each operation moves one layer of 2,560 pages (160 MiB of K/V), with scattered GPU pages and up to eight CPU regions. The cohorts use separate hosts and separately measured Stint controls.
Results. These measurements show that bulk copying is only part of transfer cost. At 160 MiB, HiCache layer-first has lower D2H latency after mappings are prepared, but Stint is faster when adapter preparation is included. For the 64-layer case (10 GiB), prepared offload takes 202.236 ms in Stint and 204.433 ms in HiCache layer-first. Reload takes 196.935 ms and 209.548 ms, respectively. Across the HiCache matrix, Stint has the lowest mean latency in 72 of 96 large fragmented configurations, but only 126 of 324 overall; these counts weight configurations equally. HiCache often wins at smaller sizes.
In the separate local-adapter cohort, the 64-layer prepared offload means are 201.550 ms for Stint and 218.266 ms for Mooncake local batch. Reload takes 196.806 and 222.676 ms, respectively. Mean sample bandwidth is 53.27/54.56 GB/s for Stint and 50.11/49.32 GB/s for the local batch adapter (D2H/H2D). These values use that cohort’s own control. SparDA tuning also matters: at 256 pages and eight CPU regions, its 32-CTA variant beats Stint on prepared H2D (0.4164 versus 0.4274 ms). These results compare complete transfer paths. They do not isolate the effects of packing and double buffering, measure overlap with model computation, or establish an end-to-end serving speedup.
Additional Predictor Results
The scheduler uses predicted remaining length to order requests. We report the complete accuracy comparison, then the serving effect of using that ordering, and finally the cost–freshness tradeoff of sparse inference. The accuracy comparison uses 300 AIME requests (AIME300) and a 200-request LiveCodeBench population (LCB200); 199 of the latter reach the plotted decoding step.
Complete Prediction Metrics
Regression accuracy. Table 14 compares all methods on the same step-500 survivors: 300 AIME requests and 199 LCB requests. No method dominates both workloads. On AIME300, Stint gives the best ordering (), while LTR has the lowest absolute error. On LCB200, PiA gives the best ordering and lowest absolute error (, MAE 4,660 tokens), closely followed by Stint (, MAE 4,744). The main paper’s predictor scatter plot also exposes the quantized outputs of S3 and the narrow prediction bands of the prompt-only regressors.
| Model | Workload | MAE | RMSE | |||
| Stint | AIME300 | 300 | 3,471 | 4,604 | −0.14 | 0.34 |
| DSMN | AIME300 | 300 | 3,942 | 5,580 | −0.68 | 0.24 |
| LTR | AIME300 | 300 | 3,179 | 3,993 | 0.14 | 0.24 |
| S3 | AIME300 | 300 | 4,459 | 5,833 | −0.83 | 0.21 |
| PiA | AIME300 | 300 | 3,377 | 4,114 | 0.09 | 0.30 |
| Stint | LCB200 | 199 | 4,744 | 7,286 | 0.15 | 0.56 |
| DSMN | LCB200 | 199 | 5,914 | 8,869 | −0.26 | 0.46 |
| LTR | LCB200 | 199 | 5,664 | 7,224 | 0.16 | 0.32 |
| S3 | LCB200 | 199 | 7,333 | 9,365 | −0.41 | 0.37 |
| PiA | LCB200 | 199 | 4,660 | 7,146 | 0.18 | 0.60 |
Table 14. Output-length prediction on the common step-500 populations. Stint and DSMN predict at step 500; the other methods predict at prefill time. These development populations combine validation and test requests.
Ranking quality. The victim rule needs to rank requests by remaining length. We therefore report Kendall’s alongside absolute error: a numerically imperfect estimate can still put two requests in the correct order. On these common step-500 populations, Stint has the best ordering on AIME, while PiA leads on LCB. Decoding-time hidden states provide useful information, but prompt-time methods remain competitive on code generation. These accuracy measurements alone do not establish a serving benefit.
Victim selection. We compare SRJF with physical-largest victims in three matched runs on each workload, AIME1000 and LCB880. Both arms serve Qwen3-32B with prediction enabled. SRJF lowers p99 ITL by 13.86% on AIME and 14.57% on LCB, while throughput falls by 7.88% and 7.41% and p99 TPOT rises by 13.68% and 0.62%, respectively. Thus shorter tail chunk gaps come at a throughput cost in these controls. Favoring nearly finished requests may free less memory immediately, but the comparison does not isolate this mechanism. The full-system gains cannot be attributed to learned victim selection alone.
Sensitivity to the Prediction Interval
Sparse online inference reduces how often the predictor runs, but also changes how fresh its estimates are. We vary on Qwen3-32B/LCB880, using H200 DP4/TP2 at 4 requests/s. Each of three arrival seeds (30–32) has fresh , 64 and 128 runs, for nine observations. All runs complete 880 requests and 9,074,823 output tokens. They keep 45,105 GPU KV blocks per worker, 800 GiB CPU KV, age-5/Q64 recovery, fused K/V kernels and double buffering.
Relative to , changes throughput by −0.12%, mean TPOT by +1.83%, and p99 TPOT by +1.45%, averaging matched ratios. changes them by +0.90%, −0.05%, and +0.53%. Its throughput is higher in all three pairs, but its p99 TPOT changes range from −10.88% to +11.44%. These small, mixed serving effects do not establish one best interval for every workload.
Predictor activity changes much more. records 97.13% fewer batched invocations and 98.41% fewer selected rows than . selects 46.32% fewer rows than , but makes 32.65% more calls because exceptional refreshes bypass the regular interval. The counters are cumulative snapshots taken after the run, summed from the output TP ranks; no corresponding before snapshots were recorded. They support a reduction in recorded predictor activity, not a measured reduction in predictor execution time. Because also changes the estimates used for scheduling, this is a whole-system sensitivity study, not an isolated head-latency benchmark.
Recompute vs. Offload Cost Model
A preempted request can recover by rebuilding missing KV or by reloading saved KV. This simple model compares rebuilding a full context with one offload–reload round trip. It explains the compute–transfer tradeoff; the serving system does not use its crossover to choose a recovery action.
Use the model’s actual dimensions. Let be context length, the number of layers, the hidden width, and the intermediate width of the SwiGLU MLP. Let and be the numbers of query and KV heads, with head dimension . The combined query width is and the combined KV width is . A KV element occupies bytes ( for BF16).
These widths need not equal the hidden width. Qwen3-32B, for example, has but , and its MLP uses . Table 15 gives the dimensions used here.
| Model | KiB/token | |||||
| Qwen3-32B | 64 | 5,120 | 25,600 | 8,192 | 1,024 | 256 |
| Llama-3.1-8B | 32 | 4,096 | 14,336 | 4,096 | 1,024 | 128 |
Table 15. Dense-model dimensions and BF16 KV payload before TP sharding. The payload counts both K and V across all layers.
Bytes moved. The full KV cache occupies bytes: one key and one value per token in each layer. Offloading and later reloading that cache therefore moves
Figure 18. Prediction-interval sensitivity with the current H200 transfer path. Points show the three arrival seeds; diamonds and error bars show means and sample standard deviations. Throughput and p99 TPOT change much less than predictor activity.
This count applies to the full available state. Partial snapshots and reused prefixes require accounting for the state actually transferred.
Work to rebuild KV. Count a multiply–add as two FLOPs. The three SwiGLU projections cost FLOPs per layer. Query and output projections cost ; key and value projections cost . The leading term for the two causal-attention matrix products is . Thus,
The attention term counts useful lower-triangular work. Actual kernels may perform additional work because of tiling and padding. Normalization, rotary embeddings, activation functions and the output vocabulary head are omitted.
Comparing modeled times. Let be effective compute throughput in FLOPs/s and effective host–device bandwidth in bytes/s. With one bandwidth for both transfer directions,
If the two directions have different effective bandwidths,
instead. For the common-bandwidth approximation, their ratio is
Values above one favor the transfer round trip within this model. Writing , the modeled crossover is
When , this approximation favors transfer for positive context lengths. That conclusion depends on the assumed effective rates. Peak hardware specifications alone do not establish those rates.
Relation to the simpler formula. If , , and , then and the ratio reduces to
These assumptions explain the earlier expression. They are not the actual Qwen3-32B dimensions. MHA, GQA and MQA differ in ; the general expression above handles each without assuming .
Compressed and Recurrent Attention
The six-panel comparison in Figure 3 uses hypothetical blocks with a common SwiGLU width and FLOPs/byte. MHA, GQA and MQA use 32 query heads, head width , and 32, 8 and 1 KV heads, respectively. For MLA, KDA and CSA, we instead keep the source models’ attention dimensions fixed while varying . Thus the panels compare specified block designs, not equally capable models or complete hybrid/MoE models.
Let denote one layer’s saved-state bytes and its contribution to rebuilding the full context. Each panel uses : rebuilding from scratch versus one complete offload–reload round trip. Cache vectors use BF16; recurrent and compressor state use FP32 where required by the reference implementations.
MLA: save the latent state. DeepSeek-V3’s MLA [5] stores a shared 512-element KV latent and a 64-element RoPE key per token. It does not need separate expanded keys and values in the saved snapshot. We use 128 query heads, 128-dimensional non-RoPE keys and values, and query rank 1,536. Rebuilding uses expanded prefill: temporary per-head keys and values support attention, while the retained cache remains compressed. Including the projections and leading causal-attention work gives
The compact decode path has a different attention cost; we do not use single-token decode work to estimate a full prefill.
Pure KDA: save the recurrent state. Kimi Delta Attention [15] keeps a recurrent matrix and short-convolution history instead of a growing token cache. We use 32 heads of width 128, convolution width four, gate rank 128, and the published chunked-prefill approximation with chunk size 64. Saving the FP32 matrices and three BF16 convolution buffers costs
The work includes projections, gates, convolutions and the published chunked core, rather than replaying the single-token recurrent kernel. This is a pure KDA block; Kimi Linear also contains MLA layers whose cache grows with context length.
CSA: save all compressed history. DeepSeek-V4-Flash’s CSA [6] compresses every four tokens into one shared key–value entry, then selects at most 512 entries for each query. The snapshot must keep all compressed entries and indexer keys, because the next query may select a different subset. It also keeps a 128-token sliding window and the overlapping compressor state. We use 64 main heads of width 512, 64 indexer heads of width 128, query rank 1,024, and eight output groups of width 1,024. With the reference implementation’s full FP32 compressor buffers,
The final term saves both main and indexer compressor buffers, including slots that a more compact snapshot could omit. Define and . The corresponding work is
The last term scans the compressed history for indexer scores; top- does not remove this work. These counts use the reference BF16 caches, not the production model’s low-precision storage or arithmetic speedups.
Scope. These counts describe unsharded blocks. With tensor parallelism, each worker needs its own state, work and bandwidth accounting. Communication and KV replication prevent a simple division of all costs by the TP degree. Full hybrid and MoE models also need their actual layer mix and expert computation.
The model omits cached-prefix savings, waiting for GPU space, launch and packing costs, overlap, and interference with other requests. It also omits softmax, normalization, nonlinearities, CSA selection/pooling work, and small state metadata. A saved prefix or checkpoint reduces rebuild work. Compute rates can also differ across kernels. The plotted ratio therefore does not predict serving throughput or total interruption time. Those outcomes require measured system comparisons.
Evaluation Settings and Implementation
We use two kinds of serving experiments. Fixed-length replay gives every system the same amount of output work. Natural-completion experiments let responses end normally and check the resulting programs. We keep their results separate because natural completion can change both output length and task outcome.
We use American Invitational Mathematics Examination (AIME) math tasks and LiveCodeBench (LCB) [14] programming tasks. A numeric suffix gives the request count of a study’s replay, not the number of repeated runs.
Current H200 Comparisons
Hardware and common settings. The repeated studies use NVIDIA H200 GPUs with 141 GB each and BF16 model weights and KV. Qwen3-32B uses four two-GPU replicas (DP4/TP2) unless a study states otherwise. Llama-3.1-8B uses eight single-GPU replicas (DP8/TP1). Each node runs one serving deployment at a time. The natural LCB study uses one TP2 replica and reserves the whole node.
All serving systems use eager execution, GPU memory fraction 0.90, a 40,960-token context limit, and 16-token KV pages. Prefix caching and chunked prefill are enabled. The vLLM systems use a 2,048-token batch budget and at most 1,024 sequences. SGLang uses 2,048-token prefill and chunk limits and at most 1,024 running requests. These backend controls have different meanings; matching their values does not make the schedulers identical. Each formal run starts fresh engines with empty caches. Rates refer to the entire deployment, with round-robin routing.
Systems. Recompute uses the maintained vLLM 0.9.2 build with its default scheduler, recomputation-based preemption, and no Stint predictor or CPU pool. FIFO64 adds a client-side FIFO admission queue to this unchanged engine. It allows at most 64 dispatched but unfinished requests per replica, or 256 across DP4. Requests still waiting for dispatch do not occupy these slots; their waiting remains part of arrival-based latency.
Stint uses an 800 GiB aggregate CPU KV payload budget, the exact-extent allocator, bulk KV transfer with fused K/V kernels and double buffering, and age-based protected recovery. Its default recovery age is 5 s and its protected turn is 64 fresh tokens. The SRJF victim policy uses frozen, workload-specific predictor weights with prediction interval . P0 keeps the same storage, transfer and recovery stack but disables prediction and selects physical-largest victims. Exceptions, including P0 on Cachewise, are stated with each study.
The four-system studies also include LMCache 0.3.5 with upstream vLLM 0.9.2, and SGLang v0.5.20 with HiCache. LMCache uses 256-token chunks, partial-prompt saving and an 800 GiB CPU payload budget, with decode saving disabled and no disk or remote backend. Its resumed-request path uses the checked prompt-resume repair. HiCache uses its native page-first layout, kernel transfers and write-through caching, with 797.219 GiB of CPU payload and no disk or remote backend. It may restore a cached prefix and rebuild newer state; neither baseline is described as preserving every request’s latest complete decode snapshot.
Memory accounting. Equal payload budgets do not mean equal host-memory use. For Qwen3-32B DP4/TP2, Stint exposes 100 GiB per worker, divided across 64 layer tensors. The pinned allocator rounds each 1.5625 GiB allocation to 2 GiB, giving a source-derived 1,024 GiB backing requirement for the 800 GiB payload. The extra 224 GiB is outside the KV allocator’s usable space. GPU staging buffers are separate from this host backing.
We also record a sampled memory guard: the larger of node-used memory and the observed cgroup charge at each sample. Its peak includes startup, serving and teardown, and can include file-cache and other charges. It is not process RSS or an isolated measure of the KV pool. Systems retain their native GPU KV capacities. In the IMO-AnswerBench study, these are 721,680 KV tokens per TP2 replica for recompute/Stint, 720,080 for LMCache and 768,480 for HiCache. The comparison therefore includes software and resource differences between complete systems.
Workload Populations and Repetitions
Fixed-length reasoning replay. The 1,000-request AIME replay (AIME1000) repeats 355 distinct development tasks: 178 from the original predictor validation fold and 177 from its test fold, without training-task overlap. The 880-request LiveCodeBench replay (LCB880) contains 880 tasks, including 300 used in predictor training. These are development-exposed serving populations. Each system receives the same saved prompt tokens and output budgets, ignoring EOS until the recorded length is reached. This fixes the amount of output work; it does not evaluate answer quality.
The main recompute/FIFO64/Stint comparison uses three matched arrival seeds (30–32) per workload at 4 requests/s. Each block uses the same inputs, arrivals and routing across systems, with rotated system order. The continuing-arrival study uses three successive 900-second phases at 0.25, 0.5 and 1 request/s, then drains all outstanding requests. The sensitivity study runs fresh configurations on LCB880 for each of the same three seeds. It does not reuse a main-study run as its reference.
Additional workloads. IMO-AnswerBench [10] contributes all 399 usable questions and 7,279,811 prescribed output tokens per run. Four systems run three matched blocks at 4 requests/s, with seeds 50–52. The prompt and common output-length freeze are shared across systems. The predictor is trained on 100 of these questions; the overlap analysis is in Appendix § 10.§ 2. Cachewise [3] instead preserves shared prefixes and parent–child dependencies in 1,418 calls. Appendix A.4 describes why its four root arrivals per second differ from a fixed request rate. In both four-system studies, blocks 0 and 2 run on one H200 node and block 1 on the other; comparisons remain within each block.
The arXiv and BurstGPT tuning confirmations each use 2,000 saved requests. arXiv prompts use the ccdv arXiv summarization dataset. Qwen thinking is disabled during prompt preparation. BurstGPT supplies lengths rather than conversation text, so its prompts are deterministic synthetic tokens. Llama/AIME uses 1,000 saved requests and 17,463,660 output tokens. Its 413 generation-capped profiles contribute 96.44% of replayed output. The three repetitions of each tuning comparison reuse the same frozen input, arrival and routing schedule; they are not new task samples.
Natural completion. The natural-completion study uses 175 new LiveCodeBench v6 tasks (LCB175). Recompute, P0 and SRJF each run three seeds at 1 request/s on one H200 TP2 replica, with 45,105 GPU blocks per worker. P0 and SRJF use 200 GiB of CPU KV. Responses end at natural EOS or the 32,768-new-token limit. Offline program tests use the same sandbox, a 300-second limit per task, and one scoring task at a time. Scoring time is outside serving time.
Metrics
TTFT measures HTTP dispatch to the first token-bearing chunk. We also report arrival-to-first-output time, which includes any admission queue. For requests with at least two output tokens, TPOT is the time between the first and last token-bearing chunks divided by the output count minus one. Single-token responses have undefined TPOT and are excluded from that metric. ITL pools gaps between consecutive token-bearing chunks. It measures received chunks, not exact device-side token timing.
A request’s longest output gap is one maximum per request. Its p99 differs from both pooled ITL p99 and p99 TPOT. Completion latency includes the entire response; its starting point is stated with each result. Finite-replay throughput counts all output tokens from the first scheduled arrival through final completion, including the drain. It is not a measurement of sustainable arrival capacity.
We compute quantiles within each run. Repeated summaries average the run-level values; percentage changes compare systems within each matched block before averaging. The main repeated AIME/LCB and IMO-AnswerBench figures use sample standard deviations, as do the appendix and Cachewise figures. The retained appendix natural-completion and tuning-confirmation figures use population standard deviations. These show run-to-run spread, not confidence intervals. Requests within a shared run are not independent repetitions. Missing counters remain missing. Pre-empted or discarded KV-token instances do not measure GPU work spent recomputing them.
Earlier Load Sweeps
The earlier load curves use the same H200 model topologies but mostly one run per point, with arrival seed 10. Qwen AIME/LCB use 0.5, 1, 2, 4 and 8 requests/s; Llama uses 1, 2, 4, 8, 16 and 32. arXiv uses 1–16 and BurstGPT 2–32 requests/s, each with 2,000 requests. Some systems run on different node allocations. They remain distinct from the repeated studies above. Historical LMCache restoration limits and the two new rate-8 observations are described in Appendix I.5. Earlier A100/H100 experiments are separated in Appendix I.
Motivation Replay
Measurement setup. The motivation replay contains 200 LiveCodeBench requests (LCB200). It serves Qwen3-32B in BF16 on H200 DP4/TP2, with finite Poisson arrivals at 16 requests/s. Stint uses SRJF/K64 offloading with recovery disabled, 11,251 GPU KV blocks per worker (16 tokens per block), and 800 GiB of aggregate CPU KV storage. The reduced GPU capacity exposes memory pressure; the end-to-end comparison uses full capacity. The grouped summary includes all 200 requests from each of three arrival seeds: 443 request executions with no preemption, 39 with one, and 118 with at least two. Seed 10 uses the original allocation; seeds 11 and 12 use its replacement.
For the timeline, we select the median-preemption request among those preempted at least twice in seed 10, breaking ties by request ID. It completes 31,157 output tokens with seven preemptions; the figure shows cycles three through five to make the short decoding turns visible. Panel (a) of the main paper’s KV-thrashing figure follows the same request’s replica from its first admission to its last completion. It overlays GPU KV usage, sampled about every 5.2 seconds, with a cumulative count of all 31 timestamped preemptions on that replica. Usage is the fraction of the reserved GPU KV pool in use; total device memory is largely preallocated. The sampled curve shows how pressure changes during the replay, not the exact occupancy at each allocation failure.
Streaming pauses are gaps between token-bearing chunks, while the orange intervals run from eviction to the next fresh token. They include both waiting and recovery; they are not measurements of pure RAM waiting. These measurements associate preemption with interruptions; they do not separate transfer time from waiting for GPU space or assign a fixed cost to each eviction.
Allocator Controls
We run Qwen3-32B BF16 on one H200 TP4 replica with an 800 GiB aggregate CPU pool. Each design serves AIME1000 and the first 800 tasks of the retained LCB880 replay (LCB800), at finite Poisson 4 requests/s with three schedule seeds (10–12). All arms run eager vLLM 0.9.2 with 104,935 GPU blocks per worker. They use learned SRJF, K32 prediction, asynchronous transfers, and age-5/Q64 recovery. Designs run serially on GPUs 0–3 with the same NUMA0 affinity. The padding statistic describes reserved space across allocations; it is not instantaneous fragmentation or process memory usage.
KV Pool and Transfer Implementation
Each GPU worker preallocates pinned CPU buffers for its KV shard. An extent identifies the same contiguous block range in each layer’s K and V buffers; layers remain in separate arrays. The byte budget and maximum number of offloaded requests are separate limits. Free space in one data-parallel replica cannot serve another. Pool backing, allocator metadata and block-index tables all consume host memory. Metadata and index-table costs are included in sampled host-memory use rather than measured separately.
The scheduler supplies GPU block IDs and CPU offsets to the transfer engine. Staging buffers are shared across requests and sized by the maximum context length. For Qwen3-32B BF16 at TP2 with a 40,960-token limit, one K/V staging pair uses 80 MiB per GPU, and two pairs use 160 MiB. Block-index buffers are separate and reused across layers. Compatible one- or two-block transfers can bypass staging.
Within each layer, offloads are submitted before reloads. Attention waits for both transfer streams so outgoing KV is safe and incoming KV is ready. Empty steps and error cleanup also wait before reusing storage. Reloaded CPU extents remain reserved until the model step completes.
Transfer Measurements
Methodology. We measure host–device transfers on H200 GPUs using BF16 KV, 16-token pages, eight KV heads and head dimension 128. The matrix covers 1–3,304 pages per layer, 1/32/64 layers, and three layouts: contiguous GPU/CPU regions, scattered GPU pages with one CPU region, and the same scattered pages with up to eight CPU regions. Each cell has 20 warmups and 100 measured operations in each of three fresh-process repetitions. We report arithmetic means and population standard deviations across the three repetition means; seeds also vary mappings and measurement order. We time both GPU-to-CPU offload (D2H) and CPU-to-GPU reload (H2D). Prepared timing excludes adapter mapping/index preparation, whereas dispatch-inclusive timing includes it. Both include payload packing, movement, scattering and completion; HiCache’s internal C++ descriptor construction remains timed in both views.
Stint uses asynchronous bulk KV transfer, fused K/V gather/scatter kernels and two staging pairs, allowing gathering or scattering of one chunk to overlap DMA for another. In this geometry the two pairs occupy 320 MiB. The HiCache cohort compares fresh Stint controls with SLGang v0.5.20 layer-first and native page-first paths under CUDA 12.8. The earlier cohort compares Stint with SparDA-derived and Mooncake-derived local adapters under CUDA 12.6/12.8; the batch API is available only in the latter environment. Each comparison keeps all methods and repetitions for a geometry on the same host and GPU. All participating layers are ready before an operation; there is no model computation in these microbenchmarks.
K/V kernels and double-buffering diagnostic. The four-way diagnostic moves 2 GiB across 64 layers, between 1,024 scattered GPU blocks and eight CPU regions. Each configuration measures ten operations. It runs on H200 GPU0 without model computation, using Qwen TP2 rank-local geometry: BF16, four KV heads per rank, head dimension 128, 16-token blocks, and token-major (NHD) storage. A staging pair has one K and one V buffer; the second pair adds 80 MiB of scratch space. Timing runs through transfer completion, including any CPU enqueue gaps and synchronization. The new paths share planner and metadata changes, which are not varied separately.
Serving ablation. We compare the legacy one-pair path with fused K/V kernels and two staging pairs on Qwen3-32B, using a 300-request AIME replay (AIME300) and a 200-request LiveCodeBench replay (LCB200), on H200 DP2/TP1 with finite Poisson arrivals at 4 requests/s. Each arm has two runs, paired by arrival seeds 10 and 11. Both arms use exact extents, learned SRJF, K64, age-5/Q64 recovery, 15,023 GPU blocks per worker, and 800 GiB aggregate CPU KV payload with 1,024 GiB backing. The replicas use GPU0/NUMA0 and GPU4/NUMA1. These settings differ from the transfer diagnostic above. The earlier TP2 four-way serving screen also had mixed latency results and only one run per arm; it is separate from these repetitions.
Prediction and Victim Controls
We compare Stint’s frozen, workload-matched decode-time predictor with four baselines: DSMN, a single-layer hidden-state predictor; LTR [9], an OPT-350M prompt-time regressor; S3, a DistilBERT prompt-time length-bucket classifier; and PiA, which applies LoRA adapters to the target LLM to predict output length at prefll time. We use Qwen3-32B generations from AIME and LiveCodeBench. Training, validation, and test request IDs are disjoint. Checkpoint selection uses only validation data. The scatter plot uses the exact AIME300 and LCB200 development populations used by our system studies; these populations combine validation and test requests and have therefore been exposed to development analysis. No continual or online adaptation is used.
We separate the cost of running the predictor from the effect of using its estimates. On Qwen3-32B BF16 with H200 DP4/TP2, eager execution, 800 GiB aggregate CPU KV capacity and deployment-wide Poisson arrivals at 4 requests/s, we compare prediction disabled with K64 prediction enabled but unused by victim selection. Both controls use physical-largest victim selection, the same GPU KV capacity, optimized transfers, and recovery age 5 s/quantum 64. Across three matched repetitions, enabling prediction changes mean output throughput by −0.58% on AIME1000 and −0.26% on LCB880. These are serving-level comparisons: predictor execution can affect batching and wall-clock recovery timing even though the victim rule is unchanged. They do not measure isolated head latency or the benefit of prediction-guided victims.
Recovery Scheduling Details
The stall timer uses a monotonic clock and starts at the first preemption in a period without new output. Reloads, recomputation, retries and further preemptions leave it running. A previously preempted request remains eligible for protected recovery whether it is offloaded, waiting for recomputation, or already running without new output. Requests never preempted are ineligible. Selection prefers the oldest eligible stall, with arrival time and request ID breaking ties.
Recovery reserves both a running-request slot and GPU KV space. When choosing physical-largest victims, shared prefix blocks still in use do not count as reclaimable memory. A saved CPU snapshot is reloaded; discarded state is rebuilt while reusing and pinning any available cached prefix. An already-running request retains its resident KV. If the selected request cannot fit after reclaiming eligible memory, the implementation reports failure. Completion or cancellation releases protection early.
Ordinary reloads do not require the age threshold or grant protection. After scheduling the running batch, the scheduler attempts them before admitting new work when GPU space and scheduling budget permit, provided the step has no preemptions. The stall timer still clears only when new output is generated. Protected recovery handles requests that remain stalled despite these ordinary opportunities to return.
Recovery Controls
Reduced-capacity Qwen/LCB study. The disabled, age-only and protected-Q64 comparison serves 50 LCB requests on one H200 TP2 replica with Qwen3-32B and SRJF/K64. It uses 11,251 GPU KV blocks per worker and 100 GiB CPU KV per worker. Arrivals retain replica 0’s offsets from the finite Poisson 16-request/s DP4 schedule, seed 10. It measures one replica’s routed requests, not the throughput of a full DP4 deployment.
Protection quantum. The Q64/Q1024 comparison serves Llama-3.1-8B/AIME1000 at DP8/TP1 on eight H200 GPUs, with an 800 GiB CPU pool and finite Poisson arrivals at 8 requests/s. Each arm and the same-node recompute control have two runs. The predictor, K64 cadence, transfer path, GPU capacity and 5 s age threshold stay fixed. Runs use the same seed-10 replay, with arm order reversed between repetition blocks. Summaries report arithmetic means and population SD; p99 summaries average run-level p99 values. Tuning and confirmation use the same population, whose 413 generation-capped profiles supply 96.44% of output tokens. This result supports balancing repeated recovery against other requests’ waiting; it does not establish one quantum for all workloads or a waiting-time bound under continued arrivals.
Historical Online-Training Extension
The main serving design uses a pretrained predictor with fixed weights and updates its length estimates as decoding reveals new hidden states. This earlier H200 study asks whether updating the weights after completed requests helps under workload shift. Its one-replica TP2 setup and development populations are separate from the current repeated DP4 serving comparisons. We report prediction accuracy, isolated update cost and live serving separately.
Accuracy Under Workload Shift
We first test whether completed requests can teach the predictor about a changing workload. The replay gradually shifts from math to coding, or from coding to math. Each direction uses three randomized orders of the same 500 requests: 300 from AIME and 200 from LCB. Table 16 compares a frozen head with one that learns from completed requests. The source-domain share decreases from 100% to 0% in 10-point steps. Each mixed phase contains 40 requests; the pure endpoint sizes are 120/20 requests for AIME-to-LCB and 20/120 for LCB-to-AIME. The initial source-domain head is either frozen or updated with AdamW at learning rate , five passes after every ten completions, batch size 128 and a FIFO window of 64 completed requests. Each request is scored before its completion label is available. This stage replays saved features with serialized completions; it does not run live LLM serving.
| Shift | Scored scope | Frozen MAE | Updated MAE | Change |
| 494 requests/order | −19.05% | |||
| 494 requests/order | −18.55% | |||
| Final target phase | −24.86% | |||
| Final target phase | −33.67% |
Table 16. Causal feature-replay accuracy at decode token 1,000. Entries are means ± population SD over three orders of the same development population. MAE is measured in output tokens; changes compare arm means.
Only 494 requests reach token 1,000 and are eligible for that score. The final LCB phase contains only 19–20 eligible requests per order; the final AIME phase contains 120. Although mean error improves in both directions, the final AIME-to-LCB phase regresses by 8.74% for seed 12. These results support average improvement, not stable accuracy at every point in the shift. The populations include historically used validation tasks; four LCB labels are capped served lengths, and AIME retains completion-selection bias. The study does not establish held-out generalization, later-token accuracy or natural answer quality.
Isolated Training Cost
Updating the head repeatedly reuses a window of completed requests. The packed path transfers that window to the GPU once per update, reuses it across training passes, and reads scalar loss back at the end. We compare it with the original path in six paired cases: both shift directions and seeds 10–12 on one H200. Each case performs 50 updates over 500 requests, keeping the optimizer, five passes per update, batch size and replay window fixed. Timing includes packing and CPU-to-GPU transfer.
Across the six pairs, total training time falls from 6.804 ± 0.142 to 3.983 ± 0.043 seconds, a 41.46% reduction. Mean update time falls from 136.08 to 79.67 ms. Every scored prediction and final weight hash matches between the two paths. These isolated timings exclude live feature capture, completion transport, tensor-parallel weight installation and interference with serving. They are not the total serving overhead.
Live Serving: Three-Seed Comparison
Better predictions and cheaper updates are useful only if they help serving. We therefore compare frozen weights, original updates and packed updates while the LLM serves requests. Both shift directions use three schedule seeds (10–12) per mode, giving 18 observations. Each run serves the same 500 requests and 4,914,049 forced output tokens on Qwen3-32B BF16, one H200 TP2 replica, eager vLLM 0.9.2, and finite Poisson arrivals at 4 requests/s.
All modes retain learned SRJF, K64 prediction, a 5 s recovery threshold, a 64-token protection quantum, a native GPU memory fraction of 0.90 and an 800 GiB aggregate CPU KV payload (1,024 GiB backing). Frozen runs allocate 45,107 GPU KV blocks per worker; training runs allocate 45,037, reflecting training memory within the same GPU budget. Only the predictor head trains, on the TP leader GPU; Qwen weights stay fixed. Completed features are buffered in CPU memory. Every ten completions trigger five training passes over the last 64 requests, using batch size 128 and learning rate . Both TP ranks install the updated weights at an engine-step boundary.
The serving results are mixed. For AIME-to-LCB, packed updates reduce throughput by 4.33% while lowering mean TPOT by 2.47% and p99 TPOT by 12.28% relative to frozen weights (Table 17). For LCB-to-AIME, throughput falls by 1.69%, mean TPOT rises by 0.12%, and p99 TPOT falls by 8.51%. The reverse-direction throughput change varies across seeds: −6.41%, +11.35% and −8.48%. The mean of per-run p99 client TTL changes by +1.15% for AIME-to-LCB and −0.87% for LCB-to-AIME. Thus, lower p99 TPOT does not imply a uniform improvement across serving metrics.
Recorded update cost. Packed training takes 252.57 ms per update on average. The original path takes 303.32 ms, so packing reduces this time by 16.73%. These values average six runs per update mode. Recorded training plus weight installation accounts for 0.424% of packed serving time. This fraction excludes feature capture, archival and synchronization outside those timers. The full-system comparisons include those activities and scheduling changes, so the timer fraction is not the total serving overhead.
The live path captures up to 16 early K64 or exceptional positions per request through token 1,000, versus seven saved anchors in the isolated replay. The longer live update time therefore cannot be attributed solely to serving interference.
Delayed training labels. Training starts after a request finishes and its output length is known. In the packed runs, the first update arrives on average 168.83 s after the first arrival for AIME-to-LCB and 99.28 s for LCB-to-AIME. Only 6.90% and 38.62%, respectively, of recorded early predictions through token 1,000 use updated weights. Long requests thus delay the labels needed to improve predictions for other requests. This is one plausible reason that the accuracy benefit has limited reach during the replay; the measurements do not isolate its contribution to net serving performance.
Prediction positions vary with scheduling, so pooled live prediction error is not the fixed-anchor accuracy metric from the feature replay. These are three orders of one development-exposed population, with forced lengths and finite arrivals. Better prediction accuracy and cheaper updates do not establish a consistent serving benefit here. Online training therefore remains a supplementary extension; the main design uses fixed-weight inference.
Historical Serving Experiments
These earlier experiments provide context for the current H200 evaluation. They use different models, hardware and implementations, so they do not isolate the current predictor, CPU allocator, transfers or protected recovery. The older A100 tables do not report repeated-run uncertainty. They illustrate earlier design tradeoffs, not the effect size of the current H200 system.
The earlier Qwen3-14B results include H100 load curves and an output-length sweep. Their retained per-panel settings are incomplete. We leave them out of the current evidence and use the documented H200 experiments instead.
| Shift | Mode | Output tok/s | Mean TTFT | Mean TPOT | p99 TPOT |
| AIME → LCB | Frozen | 1712.12 ± 40.97 | 120.65 ± 1.21 | 88.80 ± 0.37 | 197.06 ± 7.02 |
| AIME → LCB | Original | 1605.02 ± 63.42 | 124.55 ± 1.18 | 86.85 ± 0.24 | 164.26 ± 3.97 |
| AIME → LCB | Packed | 1638.00 ± 19.44 | 123.39 ± 1.15 | 86.61 ± 0.62 | 172.86 ± 14.31 |
| LCB → AIME | Frozen | 1679.15 ± 97.17 | 115.03 ± 1.67 | 94.15 ± 0.78 | 234.94 ± 10.84 |
| LCB → AIME | Original | 1690.33 ± 9.06 | 116.97 ± 1.21 | 92.34 ± 0.05 | 192.23 ± 16.29 |
| LCB → AIME | Packed | 1650.75 ± 47.99 | 116.31 ± 1.34 | 94.26 ± 1.64 | 214.94 ± 13.04 |
Table 17. Live serving with frozen weights, original updates, or packed updates. Values are mean ± population SD across three runs per direction/mode. Latencies are milliseconds; p99 TPOT averages the per-run p99 values. Repetitions reuse one development population.
The selected older studies below retain their own hardware, workloads and transfer implementations.
Recomputation and Synchronous Offload on A100
The historical A100-40GB testbed lists an AMD EPYC 7643 CPU and 500 GB of system memory; the transfer microbenchmark separately uses an A100-80GB GPU. Table 18 compares QWQ-32B on 100 AIME requests at 2 requests/s and Llama-3.1-8B on 300 synthetic requests with 128 input and 1,024 output tokens. For QWQ-32B, synchronous offload lowers p99 TPOT from 110.80 to 103.13 ms, with throughput of 712.3 versus 702.6 tokens/s. For the shorter Llama workload, recomputation instead reaches 3,885.8 versus 3,161.1 tokens/s. These observations show a workload-dependent tradeoff; they do not establish its individual time costs.
| Model | Mode | TPOT (ms) avg | p50 | p99 | Thpt (tok/s) |
| QWQ-32B | Recompute | 67.28 | 61.73 | 110.80 | 702.6 |
| Swap | 64.78 | 58.14 | 103.13 | 712.3 | |
| Llama-8B | Recompute | 55.93 | 57.42 | 142.91 | 3885.8 |
| Swap | 69.61 | 74.50 | 144.37 | 3161.1 |
Table 18. Recompute vs. synchronous swap on A100. QWQ-32B uses AIME (100 requests, rate=2); Llama-3.1-8B uses random (300 requests, input=128, output=1024).
Historical Transfer Microbenchmark
Table 19 reports block-transfer throughput on Llama-3.1-8B and an A100-80GB GPU; each block contains 16 tokens. Moving more blocks per operation raises observed throughput over much of the tested range, but the trend is not strictly monotonic in either direction. These are measurements of an older transfer path, separate from the current bulk-transfer path with fused K/V kernels and double buffering.
| Batch (blocks) | GPU→CPU | CPU→GPU |
| 1 | 1,068 | 1,103 |
| 4 | 2,625 | 2,590 |
| 16 | 3,681 | 3,623 |
| 64 | 4,112 | 3,490 |
| 256 | 4,237 | 3,496 |
| 512 | 4,252 | 3,496 |
Table 19. KV block swap throughput (blocks/s) on Llama-3.1-8B, A100-80GB. Each block is 16 tokens.
Historical Victim Policies
Table 20 uses Qwen2.5-14B, A100-40GB, and 200 synthetic requests with 128 input and 2,048 output tokens, submitted in offline mode with recomputation-based preemption.
| Policy | # Pretps. | TPOT (ms) avg | p50 | p99 | Thpt (tok/s) |
| LIFO (last) | 448 | 167.8 | 46.2 | 2,618.0 | 269.4 |
| Largest-first | 456 | 105.8 | 45.5 | 990.4 | 240.1 |
| Earliest | 446 | 126.9 | 45.9 | 2,024.7 | 242.5 |
| Random | 479 | 209.7 | 44.7 | 1,199.5 | 242.1 |
Table 20. Victim selection policy comparison (Qwen2.5-14B, A100-40GB, 200 requests). TPOT reported as avg/p50/p99 in ms.
The policies trigger similar preemption counts (446–479) but differ in p99 TPOT. Largest-first has the lowest p99 TPOT among these four policies, while LIFO has the highest throughput. Aggregate counts alone do not establish repeated preemption of particular requests or explain the latency differences.
A separate historical oracle study used Qwen3-32B on 355 AIME development requests selected from completed runs. They contain 3,278,404 forced output tokens. The setting uses one H200/TP1, finite Poisson arrivals at 4 requests/s, a 400 GiB CPU KV pool, eager execution, synchronous layer-wise transfers, a 5 s recovery age and a 64-token protection quantum. Across three paired repetitions (seeds 0, 1, and 2), it compared admission-time oracle SRJF with predictor-on physical-largest selection; both arms retained the same predictor work. The oracle received true lengths earlier than hidden-state predictions, so this historical comparison does not bound the headroom of the current learned policy.
Synchronous Offload Under Sustained Pressure
Table 21 uses QWQ-32B on A100 with 40 synthetic requests, each containing 100 input and 2,000 output tokens, at 0.5 requests/s. Synchronous offload lowers p99 TPOT from 680.7 to 249.5 ms, while throughput changes from 46.6 to 45.9 tokens/s. The offload run has fewer preemption events but more preempted token instances. Those counters do not measure executed recomputation or isolate how much delay came from copying, scheduling or waiting. These controls therefore cannot establish the current protected-recovery benefit.
Two Historical LMCache Corrections
The new four-system studies use the checked prompt-resume path. We selected two earlier rate-8 Qwen AIME/LCB observations for a source and trigger audit. Both exposed the affected resume pattern. We repeat these two points with the repair; this bounded audit does not classify every earlier point. The reruns preserve their workloads and settings: H200 DP4/TP2, BF16, finite Poisson arrivals at 8 requests/s, seed 10, 800 GiB CPU KV and decode saving disabled.
Throughput and p99 TPOT improve, while p99 TTFT and preemption counts increase. The new measurements are not repeated estimates and do not retroactively validate the other historical points. The original 84-point sweep contains 30 observations with preemptions that require source/trigger review before reuse as restoration evidence. A positive preemption count alone does not prove a failed or corrupt restore. The new IMO-AnswerBench and Cachewise comparisons provide the repeated four-system evidence used for the current claims.
Earlier Predictor Experiments
Progressive Refinement on GSM8K
In an earlier controlled GSM8K trace, the predictor’s MAE decreases from 221 at decode step 100 to 154 at step 500 (Figure 19). The prompt-time baselines remain fixed after prefill. This illustration uses a separate trace from the current common-population AIME300/LCB200 comparison and is not evidence that estimates improve monotonically for every request or workload.
Figure 19. Earlier GSM8K trace: prediction errors and accuracy within 200 tokens as decoding advances. Horizontal lines show the prompt-time baselines. Individual steps need not improve monotonically.
Progress Classification
An earlier four-class head predicts progress quartiles: 0–25%, 25–50%, 50–75%, and 75–100%. The retained exploratory results report 85.7% overall accuracy on capped data and 79.3% on uncapped data, with per-class accuracy between 80% and 92% on capped data. These results describe a separate predictor variant; the current serving implementation uses a regression head. Classification accuracy alone does not measure victim-ranking quality or serving benefit, and this
variant is not used as evidence for the main system’s performance. The original manuscript description does not specify repeated-run uncertainty for these exploratory values.
| Mode | TPOT (ms) | Thpt (tok/s) | Pre-empt. | Tokens preempt. | ||
| avg | p50 | p99 | ||||
| Swap | 128.0 | 116.9 | 249.5 | 45.9 | 191 | 126,509 |
| Recompute | 170.5 | 149.9 | 680.7 | 46.6 | 202 | 101,311 |
Table 21. Sustained memory pressure (QWQ-32B, A100, 40 random requests, rate=0.5). TPOT in ms.
| Workload | Observation | Output tok/s | p99 TPOT (ms) | p99 TTFT (ms) | Preemptions |
| AIME1000 | Original | 6,844.34 | 169.02 | 282.79 | 1,732 |
| AIME1000 | Repaired follow-up | 7,378.17 | 159.10 | 483.08 | 1,801 |
| LCB880 | Original | 5,159.31 | 173.51 | 277.42 | 1,103 |
| LCB880 | Repaired follow-up | 5,247.53 | 172.57 | 614.82 | 1,122 |
Table 22. Historical LMCache observations and their repaired follow-ups. Each cell is one run; old-to-new changes do not isolate the repair’s causal effect. Original observations remain available.
References
- [1]Reyna Abhyankar, Zijian He, Vikranth Srivatsa, Hao Zhang, and Yiying Zhang. 2024. InFerCept: Efficient Intercept Support for Augmented Large Language Model Inference. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). 81–95.arxiv.org/abs/2402.01869
- [2]Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 117–134. https://www.usenix.org/conference/osdi24/presentation/agrawalarxiv.org/abs/2403.02310
- [3]CacheWise Project. 2026. CacheWise Coding Traces. GitHub repository. Trace snapshot 181c435a090d328d0dbbee4c8eeb27d32f3abd2. https://github.com/cachewise-project/cachewise-coding-traces
- [4]Junyi Chen, Chuheng Du, Renyuan Liu, Shouchao Yao, Dingtian Yan, Jiang Liao, Shengzhong Liu, Fan Wu, and Guihai Chen. 2026. TokenFlow: Responsive LLM Text Streaming under Request Burst via Preemptive Scheduling. In Proceedings of the 21st European Conference on Computer Systems. ACM, Edinburgh, United Kingdom, 497–513. doi:10.1145/3767295.3769328DOI
- [5]DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. arXiv:2412.19437 doi:10.48550/arXiv.2412.19437DOI
- [6]DeepSeek-AI. 2026. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348 doi:10.48550/arXiv.2606.19348DOI
- [7]Zhichen Dong, Guanhui Zhou, Zhixuan Liu, Chao Yang, and Chaochao Lu. 2025. Emergent Response Planning in LLM. arXiv:2502.06258 [cs]. doi:10.48550/arXiv.2502.06258
- [8]Yaosheng Fu, Guangxuan Xiao, Xin Dong, Song Han, and Oreste Villa. 2026. SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference. arXiv:2606.04511 [cs.CL] doi:10.48550/arXiv.2606.04511DOI
- [9]Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, Ion Stoica, and Hao Zhang. 2024. Efficient LLM Scheduling by Learning to Rank. arXiv:2408.15792 [cs]. doi:10.48550/arXiv.2408.15792arxiv.org/abs/2408.15792
- [10]Google DeepMind. [n. d.]. Superhuman. https://github.com/google-deepmind/superhuman. AnswerBench benchmark data and evaluation code. Accessed October 5, 2026.
- [11]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jau... [12] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Rouyu Zhang, Shirong Ma, Xiao Bi, et al. 2025. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645, 8081 (2025), 633–638. doi:10.1038/s41586-025-09422-z
- [13]Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Zadmadi Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, and Yuxiong He. 2024. DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference. arXiv:2401.08671 doi:10.48550/arXiv.2401.08671DOI
- [14]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. arXiv:2403.07974 https://arxiv.org/abs/2403.07974
- [15]Kimi Team. 2025. Kimi Linear: An Expressive, Efficient Attention Architecture. arXiv:2510.26692 doi:10.48550/arXiv.2510.26692
- [16]Woosook Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with Paged Attention. In Proceedings of the 29th Symposium on Operating Systems Principles. ACM, Koblenz, Germany, 611–626. doi:10.1145/3600006.3613165DOI
- [17]Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024. InfniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 155–172. https://www.usenix.org/conference/osdi24/presentation/leearxiv.org/abs/2406.19707
- [18]Bin Lin, Chen Zhang, Tao Peng, Hanyu Zhao, Wencong Xiao, Minmin Sun, Anmin Liu, Zhipeng Zhang, Lanbo Li, Xiafei Qiu, Shen Li, Zhigang Ji, Tao Xie, Yong Li, and Wei Lin. 2024. Infinite-LLM: Efficient LLM Service for Long Context with Dist Attention and Distributed KVCache. arXiv:2401.02669. http://arxiv.org/abs/2401.02669
- [19]Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Rui Zhang, Kuntai Du, and Junchen Jiang. 2025. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference. arXiv:2510.09665 doi:10.48550/arXiv.2510.09665DOI
- [20]Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2023. Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time. Advances in Neural Information Processing Systems 36 (2023), 52342–52364.arxiv.org/abs/2305.17118
- [21]Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). PMLR, Vienna, Austria, 32332–32344. https://proceedings.mlr.press/v235/liu24bz.htmlDOI
- [22]Thang Luong et al. 2025. Towards Robust Mathematical Reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP). arXiv:2511.01846.arxiv.org/abs/2511.01846
- [23]Mooncake contributors. 2026. Mooncake Transfer Engine: Local Batched Copy Implementation. Source code, revision 68ce23861197. Accessed September 25, 2026. https://github.com/vllm-project/Mooncake/blob/68ce23861197fd3751274cfdf56771d61534b7/mooncake-transfer-engine/src/transport/nvlink_transport/nvlink_transport.cpp
- [24]Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hanneke Haisma, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. arXiv:2501.19393 doi:10.48550/arXiv.2501.19393DOI
- [25]Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, Buenos Aires, Argentina, 118–132. doi:10.1109/ISCA59079.2024.00019
- [26]Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation—A KVCache-centric Architecture for Serving LLM Chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25). USENIX Association, Santa Clara, CA, 155–170. https://www.usenix.org/conference/fast25/presentation/qin
- [27]Haoran Qiu, Weichao Mao, Arichit Patke, Shengkun Cui, Saurabh Jha, Chen Wang, Hubertus Franke, Zbigniew T. Kalbarczyk, Tamer Başar, and Ravishankar K. Iyer. 2024. Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction. arXiv:2404.08509 [cs]. doi:10.48550/arXiv.2404.08509
- [28]SGLang contributors. 2026. HiCache System Design and Optimization. SGLang documentation. Accessed September 25, 2026. https://docs.sglang.io/docs/advanced_features/hicache_design
- [29]Rana Shahout, Eran Malach, Chunwei Liu, Weifan Jiang, Minlan Yu, and Michael Mitzennmacher. 2024. Don’t Stop Me Now: Embedding Based Scheduling for LLMs. arXiv:2410.01035 doi:10.48550/arXiv.2410.01035DOI
- [30]Ao Shen, Zhiyao Li, and Mingyu Gao. 2024. FastSwitch: Optimizing Context Switching Efficiency in Fairness-aware Large Language Model Serving. arXiv:2411.18424 [cs.LG]arxiv.org/abs/2411.18424
- [31]Haiying Shen, Tanmoy Sen, and Masahiro Tanaka. 2025. Mitigating KV Cache Competition to Enhance User Experience in LLM Inference. arXiv:2503.13773 [cs.DC]arxiv.org/abs/2503.13773
- [32]Ying Sheng, Lianmin Zheng, Binzhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202). PMLR, Honolulu, HI, 31094–31116. https://proceedings.mlr.press/v202/sheng23a.html
- [33]Ziji Shi, Chaoyi Ruan, Penghui Qi, Guangxing Huang, Xinyi Wan, Min Lin, and Jialin Li. 2025. Tetris: Efficient and Predictive KV Cache Offloading for Agentic and Reasoning Workloads. 1st Workshop on Systems for Agentic AI (SAA ’25), co-located with SOSP 2025. Three-page workshop paper. https://saa2025.github.io/papers/Tetris%20-%20Efficient%20and%20Predictive%20KV%20Cache%20Offloading%20for%20Agentic%20and%20Reasoning%20Workloads.pdf
- [34]Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. arXiv:2408.03314 doi:10.48550/arXiv.2408.03314
- [35]Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. 2024. Lumnmix: Dynamic Scheduling for Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 173–191. https://www.usenix.org/conference/osdi24/presentation/sun-biaoarxiv.org/abs/2406.03243
- [36]Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). PMLR, Vienna, Austria, 47901–47911. https://proceedings.mlr.press/v235/tang24l.htmlarxiv.org/abs/2406.10774
- [37]vLLM contributors. 2025. vLLM KV Cache Block-Swapping Implementation. Source code, version 0.9.2. Accessed September 25, 2026. https://github.com/vllm-project/vllm/blob/v0.9.2/csrc/cache_kernels.cu
- [38]Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuan zhe Liu, and Xin Jin. 2024. LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles. ACM, Austin TX USA, 640–654. doi:10.1145/3694715.3695948DOI
- [39]Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyu Liu, Yunhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. 2026. FastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). USENIX Association, Renton, WA, 57–74. https://www.usenix.org/conference/nsdi26/presentation/wu-bingyang
- [40]Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient Streaming Language Models with Attention Sinks. In The Twelfth International Conference on Learning Representations. Vienna, Austria. https://openreview.net/forum?id=NG7sS51zVFarxiv.org/abs/2309.17453
- [41]Yi Xu, Ziming Mao, Xiangxi Mo, Shu Liu, and Ion Stoica. 2024. Pie: Pooling CPU Memory for LLM Inference. arXiv:2411.09317 doi:10.48550/arXiv.2411.09317DOI
- [42]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 Technical Report. arXiv:2505.09388 doi:10.48550/arXiv.2505.09388DOI
- [43]Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion. In Proceedings of the Twentieth European Conference on Computer Systems. ACM, Rotterdam Netherlands, 94–109. doi:10.1145/3689031.3696098DOI
- [44]Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soo-jeong Kim, and Byung-Gon Chun. 2022. ORCA: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 521–538. https://www.usenix.org/conference/osdi22/presentation/yu
- [45]Jiahua Yu, Mingtao Hu, Zichao Lin, and Minjia Zhang. 2026. SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchips. In Proceedings of Machine Learning and Systems, Vol. 8. MLSys, Bellevue, WA, 75–93. https://proceedings.mlsys.org/paper_files/paper/2026/hash/07fd64f9316f40193c6a4d87d8afa011-Abstract-Conference.htmlarxiv.org/abs/2601.20309
- [46]Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. 2023. H2O: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems 36 (2023), 34661–34710.arxiv.org/abs/2306.14048
- [47]Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2024. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems 37 (2024), 62557–62583.arxiv.org/abs/2312.07104
- [48]Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanze Liew, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 193–210. https://www.usenix.org/conference/osdi24/presentation/zhong-yinminarxiv.org/abs/2401.09670










