betaFeedback

WorksComputer Science

TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming

Abstract

Hybrid architectures have become mainstream among open-weight language models. Their global-attention and bounded-state layers maintain heterogeneous caches whose demands vary with context length and concurrency. We present TideCache, a serving system on CPU–GPU superchips with three components for allocating GPU memory among caches and managing their residency across GPU and host memory. The Transparent Cache Pool redistributes physical pages among cache components behind fixed virtual addresses, preserving cache layouts and backend interfaces. The Cache Streamer uses computation intervals provided by bounded-state layers to prefetch the next global-attention layer's prefix into GPU slots shared across global-attention layers, reducing GPU residency while overlapping layer-wise transfers with computation. Generated KV remains resident. The Request Scheduler selects resident or streamed execution and triggers capacity adjustments according to request size and queue state. Together, they let HBM released by streaming accommodate other cache components and additional requests. Implemented in SGLang and evaluated on GB300 and GH200 across seven model/platform configurations, TideCache completes up to 2.04x as many agent turns, reduces overall P50 per-token latency by 20–59%, and improves agent P50 TTFT by 1.40–3.05x compared with SGLang with HiCache.

Introduction

Hybrid architectures have become mainstream among open-weight language models. Kimi K3 (Kimi Team, [9]), Ring-2.6 (Li et al., [12]), GLM-5.3-Flash (Z.ai, [35]), Qwen3.8 (Qwen Team, [20]) and GPT-OSS (OpenAI, [16]) keep a global-attention layer only once every few layers and fill the rest with sliding-window attention or recurrent layers such as Mamba, Gated DeltaNet and Kimi Delta Attention (Gu & Dao, 2024; Yang et al., [32]; Kimi Team, [8]). We refer to the latter as bounded-state layers, since their cache memory per request is bounded independently of context length. These models serve everyday chat, coding agents, document analysis and automated data labeling. Such applications differ in context reuse, interaction structure, arrival patterns and concurrency (Lin et al., [13]; Xiang et al., [28]). A coding agent accumulates tool outputs across turns, document analysis retains reference material, and chat or labeling services may process many independent requests. These behaviors create different demands on global-attention KV, window caches and recurrent states; no application label alone determines which component limits serving capacity.

CPU–GPU superchips offer a larger memory hierarchy for these workloads by coupling GPU memory with host memory through high-bandwidth interconnects. The bandwidth of this link has grown by an order of magnitude in five

years (Table 1): from 64 GB/s over PCIe 4.0 on A100 to 900 GB/s over NVLink-C2C on Grace Hopper, and 1.8 TB/s on Vera Rubin (Evans et al., [5]; Aubrey & Stam, [3]; Aubrey, [2]). This allows GPUs to access host-resident data through both direct memory accesses and explicit transfers. Host memory thus provides an additional tier for the different cache components maintained by hybrid models.

Platform (year)CPU–GPU interconnectGB/svs. A100
A100 (2020)PCIe 4.0 ×16641×
H100 (2022)PCIe 5.0 ×161282×
GH200 (2023)NVLink-C2C90014×
GB300 (2025)NVLink-C2C90014×
VR300 (2026)NVLink-C2C1,80028×

Table 1. Bidirectional bandwidth of the CPU–GPU interconnect across NVIDIA generations: one PCIe ×16 slot per GPU, and one NVLink-C2C interface per Grace superchip (shared by the two GPUs of a GB300 superchip).

Existing systems improve hybrid models’ memory utilization through cache allocation (Zheng et al., [38]; Kwon et al., [10]). Static partitioning reserves a fixed budget for each component and can leave one pool underused while another runs out of space (Figure 1a). Dynamic sharing schemes, such as SGLang’s Unified Memory Pool and vLLM’s Hybrid KV Cache Manager, reduce this imbalance (SGLang Team, [23]; vLLM Team, 2026). However, sharing an arena across cache layouts can require changes to indexing and memory access, complicating integration with attention backends and other serving features.

Two opportunities for efficient hybrid-model serving

Figure 1. Two opportunities for efficient hybrid-model serving: sharing cache capacity and exploiting idle decode-time bandwidth.

A complementary line of work uses offloading to reduce GPU memory residency. HiCache [29] preserves reusable contexts in a hierarchical cache and restores them for later execution, reducing repeated prefill computation. However, restored prefixes remain GPU-resident during subsequent decoding. HiSparse [30] reduces active requests’ GPU footprint by fetching selected KV entries during decode, but is specialized for sparse attention. General offloading systems such as FlexGen and Pie [24] (Xu et al. 2024) support larger inference batches through tensor placement and transfer scheduling, but do not leverage the cache access patterns of hybrid models.

To this end, we present TideCache, a hybrid-model serving system with three components: the Transparent Cache Pool, a Cache Streamer, and a Request Scheduler. Our key observation is that bounded-state layers create intervals of computation during which the historical KV of the next global-attention layer is not yet needed. The Cache Streamer exploits idle CPU–GPU bandwidth during decode to prefetch host-resident prefixes into GPU slots shared across global-attention layers (Figure 1b). Overlapping these transfers with computation expands effective global-KV capacity: requests with reusable host-resident prefixes shrink their GPU footprint, while newly generated KV stays resident. To make the released capacity available to other cache components, the Transparent Cache Pool redistributes physical pages behind the reserved virtual addresses of the hybrid subpools. The Transparent Cache Pool transparently supports existing cache layouts and attention backend interfaces, without requiring changes to either. The two memory mechanisms work together: layer-wise KV streaming frees HBM that can hold window caches and recurrent states for additional requests, and a larger decode batch provides more computation with which to overlap subsequent transfers. The Request Scheduler coordinates these mechanisms, selecting resident or streamed execution and triggering sub-

pool capacity adjustments according to request and queue conditions.

We implement TideCache in SGLang and evaluate seven model/platform configurations on GB300 and GH200 superchips without model-specific kernel changes. Across mixtures of Chat and Coding Agent traffic, TideCache completes up to 2.04× as many agent turns, reduces overall P50 per-token latency by 20–59%, and improves agent P50 TTFT by 1.40–3.05× compared with SGLang with HiCache. The Transparent Cache Pool achieves 96.3–113.6% of the turns of the better of two hand-tuned partitions at each fixed load; under a shifting workload on GH200, the full system completes 59% more agent turns and reduces agent P50 TTFT by 78% relative to SGLang with HiCache. At equal batch size, the system adds at most 6.3% to decode-step latency.

This paper makes three contributions:

  • We identify idle partition capacity and idle decode-time CPU–GPU bandwidth as two opportunities in hybrid serving.

  • We design the Transparent Cache Pool, the Cache Streamer and the Request Scheduler, and build them behind fixed virtual addresses in SGLang.

  • We evaluate seven model/platform configurations on two superchip platforms across varying workloads.

Background

Hybrid Model Inference

An autoregressive model runs a prefill phase, which builds per-layer state for the input, and a decode phase, which produces one token per request per step and updates that state. What the state is depends on the layer (Figure 2). A global-attention layer lets every token attend to all earlier positions, so its Key–Value (KV) cache grows with the context. A bounded-state layer keeps a state whose size is independent of the context [17]: a sliding-window layer keeps the KV of the most recent window, and a recurrent layer such as Mamba, Gated DeltaNet or Kimi Delta Attention keeps a fixed-size state updated in place. The model stacks the hybrid block of Figure 2 repeatedly, placing one global-attention layer every few bounded-state layers. We call the data each layer type keeps a cache component: global-KV, window KV and recurrent state; where the distinction does not matter we write state for the two bounded components. For Ring-2.6-1T at TP4 with BF16 caches, a 67K-token context uses about 0.72 GiB of global-KV per GPU, while one recurrent-state slot occupies 35 MiB. The evaluated SGLang configuration budgets five such slots per running request for admission (175 MiB, or 0.17 GiB), independent of context length.

Figure 2. Illustration of a typical hybrid LLM architecture. Global-attention layers (full, DSA or QSA [19]) hold a global-KV that grows with the context; bounded-state layers (sliding window, GDN, KDA) hold a window KV or recurrent state of fixed size per request. All cache components share GPU memory.

Hybrid Cache Management

Serving systems reuse the cached state of a prefix to avoid prefilling it again. In a hybrid model a global-KV hit alone is not enough: a bounded-state layer can resume only from a checkpoint at the matched position, so every component must be consistent at the same boundary. SGLang’s unified radix cache coordinates matching, insertion and eviction of all components under one prefix tree [38, 7], and its memory pool owns the tensors and decides how much GPU memory each component receives, either through a partition fixed at start-up or through a unified pool whose boundary moves at run time [23]. HiCache extends the tree to host memory: evicted prefixes and checkpoints are backed up there and loaded back before a session’s next turn runs [29]. TideCache replaces only the memory-pool layer and inherits the radix cache and HiCache.

Offloading on CPU–GPU Superchips

On Grace superchips the GPU reaches host memory over NVLink-C2C; one GB300 GPU reads pinned host memory at 211 GB/s, and four GPUs reading concurrently reach 182–192 GB/s each. Two ways of using this path differ in what stays on the GPU. Request-level restore copies a session’s prefix back once before the turn runs; the prefix is then resident for the whole turn [29]. Layer-wise loading keeps on the GPU only the layers about to execute and transfers the rest again in every decode step (Xu et al., 2024); it is hidden only if each layer’s copy finishes before that layer starts, which depends on how much compute separates the layers that need the data.

Motivation

Cache Demand Depends on the Workload

Different workloads stress different parts of a hybrid model’s cache. A busy chat or data-labeling service may have many requests active at once, each requiring its own window KV or recurrent state. Coding agents or document analysis may instead retain substantial context, increasing the global-KV footprint per request. These examples describe demand patterns, not fixed application categories. For NN running requests with mean context cˉ\bar{c}, global-KV demand grows with NcˉN\bar{c}, whereas bounded-state demand grows only with NN. A fixed partition can therefore exhaust one component while leaving another underused. Figure 3 illustrates this imbalance using Ring-2.6-1T under SGLang’s default partition. The state pool limits concurrency to 61 requests even when most of global-KV remains unused. Conversely, at 64K tokens per request, global-KV holds only 17 requests while most of the state pool remains unused. A different partition moves these limits but retains the tradeoff: giving more memory to one component leaves less for the others. In a mixed workload, demand shifts with each request source’s arrival rate, so a partition tuned for one mix can become a bottleneck for another. Repartitioning cannot increase the GPU’s total cache budget: when combined demand exceeds HBM capacity, it only changes which requests wait.

Figure 3. Loads that one GPU of Ring-2.6-1T (TP4) can run under SGLang’s default partition (state : global-KV = 0.9). The state pool caps the request count at S=61S = 61; the global-KV pool caps the product of request count and context length at G=1.09MG = 1.09\mathrm{M} tokens.

The CPU–GPU Link Is Idle During Decode

HiCache uses host memory between turns: a session’s prefix is restored in full before its next turn is admitted and stays resident until the turn ends. During decode, which dominates the turn, the host-to-device link carries almost nothing. Its capacity is large: a GB300 GPU reads host memory at 211 GB/s, so one 57 ms decode step of Ring-2.6-1T at the loads of § 5 could move about 12 GB per GPU at that bandwidth. A hybrid model can use this capacity because only its global-attention layers read the prefix. Between two of them, 7 bounded-state layers on Ring-2.6 compute without needing any global-KV, so the next layer’s prefix can be transferred in that interval and the previous layer’s can be discarded. A request served this way needs device memory for a few global-attention layers at a time rather than for all of them. More computation per decode step allows more transfer to overlap. Diverse workloads can create this opportunity: concurrent requests enlarge the decode batch while other requests retain reusable host-resident prefixes. The relevant properties are batch computation and cache demand, rather than application identity.

Design

SGLangvLLMTideCache
TechniqueUnified Memory PoolHybrid KV ManagerTransparent Cache Pool
[38][10]
[37](vLLM Team, 2026)
Dynamic Sharing✓✓✓
Copy-Free Repartitioning✗✓✓
Transparency✗✗✓
Streaming Compatibility✗✗✓

Table 2. Comparison of hybrid cache management solutions.

Design

Motivated by the observations in § 3, we build TideCache, a serving system with three components. The Transparent Cache Pool (TCP, § 4.1) repartitions GPU memory at run time by moving physical pages to the cache component that blocks admission. The Cache Streamer (§ 4.2) serves streamed requests by copying each global-attention layer’s prefix into a shared slot just before that layer runs. The Request Scheduler (§ 4.4) decides whether the first request in SGLang’s waiting queue (the head request) runs with its prefix resident on the GPU or streamed from host memory, and which cache component blocks it if neither fits. All three operate on one memory layout whose defining property is that every cache component keeps fixed virtual addresses: attention backends, the page table, CUDA graphs and HiCache address the cache as they always have, and TideCache only changes which physical pages back an address and which host rows are present at an address during a step.

Figure 4 presents TideCache’s architecture. The Request Scheduler controls admission to the resident and streaming regions, shown by its two labeled arrows, and dispatches admitted requests to the inference engine. The data plane exposes virtual cache layouts, physical GPU memory and host cache. The Transparent Cache Pool manages a shared arena of GPU pages, repartitioning it among cache components as demand changes. For the head request, the scheduler selects resident or streamed execution, or reports the blocking cache component to the Transparent Cache Pool for growth between forward passes. Before each forward pass the Cache Streamer assigns slot rows to the batch’s streamed prefixes and prepares their copies, while HiCache restores resident prefixes and backs up finished turns as before.

TideCache overview showing the Transparent Cache Pool, Request Scheduler, Inference Engine, Cache Streamer, Mixed Workload, Data Plane, and Control Plane

Figure 4. TideCache overview: the Request Scheduler controls resident- and streaming-region admission; the Transparent Cache Pool manages cache backing, and the Cache Streamer feeds shared GPU slots from host memory.

Transparent Cache Pool

As § 3.1 shows, a fixed partition can block requests even when another cache component has unused capacity. The Transparent Cache Pool addresses this imbalance by dynamically repartitioning a shared GPU physical page arena among global-KV, window KV and recurrent state. Each cache component has a subpool: a fixed virtual address range backed by a variable number of physical pages. To grow a subpool, the Transparent Cache Pool maps additional arena pages into its virtual range; to shrink one, it unmaps pages that no active request uses, making them available to another subpool. This adjusts capacity without relocating live data or changing its virtual addresses.

The physical arena is allocated once in 2 MiB pages through the CUDA virtual memory API ([15]). Each subpool reserves more virtual address space than any physical share it will receive, and its tensors span this reservation. The allocator uses only rows whose backing pages are mapped. Rows retain SGLang’s index addressing, so forward passes are independent of physical page locations.

Transparent dynamic sharing. Table 2 contrasts how dynamic sharing affects existing cache interfaces.1 SGLang’s Unified Memory Pool (UMP) allocates global-KV and bounded state from opposite ends of a raw uint8 buffer. It packs all layers’ KV for a page into one page envelope; reclaiming fragmented space can require copying live envelopes and updating address mappings. This page-major layout requires stride-compatible attention backends and logical-to-physical index translation ([23, 37]). In this revision, UMP rejects HiCache and LMCache: host pools are not initialized, and host offloading lacks logical-to-physical slot translation ([22]). vLLM’s Hybrid KV Cache Manager shares uniformly sized pages among cache groups without relocating live blocks ((vLLM Team, 2026; vLLM Team and Inferact, 2026)). KV-transfer connectors must implement SupportsHMA to handle per-group block lists and their asynchronous release ((vLLM Contributors, 2026)).

These designs expose pool-specific layouts or grouping to cache consumers. TCP confines repartitioning to physical page mappings. It preserves per-layer tensors, virtual addresses and cache indexing, enabling copy-free repartitioning without pool-specific adaptation of attention backends or HiCache. The same virtual layout natively supports aliased mappings for layer-wise streaming (§ 4.2), preserving existing cache interfaces.

Move protocol. The Transparent Cache Pool rebalances capacity once per epoch of E=8E = 8 forward passes, with at most one move in flight (Figure 5). It plans the receiver, the amount and the donors, and parks the donor rows by removing them from their allocator’s free list; rows in use are never touched. A step fence waits until every forward pass issued before the park has completed. A worker thread then calls cuMemUnmap on the parked pages and cuMemMap to back the receiver’s lowest parked rows, followed by cuMemSetAccess; these driver calls do not wait for the GPU and touch only memory no batch may use, so forward passes continue. The pool manager polls the worker at each epoch and applies the move, un-parking the receiver’s rows, in the first epoch in which every rank’s worker is done. Each epoch starts with one all-reduce of a busy flag, so all ranks apply the same move before the same forward pass and build identical batches.

One page move across epochs showing plan, park, poll, and apply operations

Figure 5. One page move across epochs: plan and park on the scheduler thread, unmap and map on a worker after the step fence, apply at the next epoch in which every rank is idle.

Policy. The Transparent Cache Pool adopts a failure-driven repartitioning policy to direct capacity to the component blocking admission while limiting cache eviction. When the Request Scheduler cannot admit the head request because a subpool lacks capacity, it asks the Transparent Cache Pool to grow that subpool. This subpool is the receiver; pages come from the following sources, in order of increasing reclamation cost:

  1. Free pages, which are not assigned to any subpool.

  2. Idle capacity in other subpools, reclaimed by unmapping pages from unused rows. These subpools act as donors and shrink as the receiver grows.

  3. Cold cached prefixes in donor subpools, defined as prefixes untouched for τ\tau seconds. Evicting a cold prefix and its checkpoints releases their cache rows.

The Transparent Cache Pool exhausts each source before proceeding to the next. If growth can be funded without eviction, its size is scaled by the waiting-queue length (capped at kk) to avoid admitting only one request per multi-epoch move. If eviction is required, growth is limited to the head request’s needs to avoid discarding more cached data than necessary. Hot cached prefixes are not reclaimed as donors, because discarding a prefix reused after a tool gap can force recomputation. A subpool shrinks only to supply a receiver. Before admission fails, TCP can also grow a subpool whose free space runs low by one small step from idle capacity or cold cache in other subpools, sparing hot prefixes from later eviction.

Cache Streamer

Streaming every request’s context at every decode step can exceed the link’s transfer budget. For example, the GPT-OSS-120B ablation (§ 5.3) sustains a batch of 29 requests with 8K-token contexts on GH200. Their BF16 global-KV occupies 8.76 GB across 18 global-attention layers. Transferring this entire batch at 400 GB/s would take 21.9 ms per step, exceeding the measured 19.7 ms decode step with selective streaming. Layer-wise pipelining alone therefore cannot hide all-batch transfers at this operating point. The Cache Streamer instead streams only a subset of requests selected by the Request Scheduler, keeping the rest resident. This raises three questions: how to reduce only these requests’ physical footprint, how to move their changing KV addresses without breaking the decode graph, and how much transfer can be hidden.

How can selected requests use less GPU memory? The global-KV subpool separates resident and streaming regions while retaining the same per-layer tensor interfaces (Figure 6). The resident region has independent backing pages for every layer, managed by TCP. It holds the prefixes of requests served without streaming and the newly generated KV of all requests. Only historical prefixes already available in host memory are streamed; generated KV is backed up by HiCache at the end of the turn.

Figure 6. The global-KV subpool (KV1, KV2, etc. denote successive layers’ caches): a resident region with its own pages per layer and a streaming region whose LL layer ranges alias M=2M = 2 slots; HiCache restores into the former, streaming feeds the latter.

At start-up, the streaming region’s virtual ranges for LL global-attention layers are mapped onto M=2M = 2 physical slots, with layer ℓ\ell using slot (ℓ−1) mod M(\ell- 1) \bmod M. These aliased mappings let different layers address the same physical pages through their original per-layer tensors. A streamed prefix occupies only M/LM/L of its full-layer global-KV footprint on the GPU. Its rows hold one layer’s prefix at a time; generated tokens remain in separate resident rows. The slot capacity is set at start-up using the bound in Equation (1), and the alias mappings remain fixed during execution. TCP can repartition the resident backing without changing these mappings.

These slots support the double-buffered overlap pipeline in Figure 7. A single H2D stream fills one slot while the GPU reads the other, using both bounded-state computation and global attention to hide transfers. For example, after attention finishes reading KV2 from slot B, the copy stream can overwrite it with KV4 while computation proceeds through KV3 and the intervening bounded-state layers. Realizing this schedule requires coordinating slot reuse and KV availability without interrupting the decode graph.

Double-buffered overlap with two slots

Figure 7. Double-buffered overlap with two slots: reading KV2 releases slot B for KV4’s transfer, which must finish before attention uses KV4. Bounded-state layers (BSL) and global attention provide computation to overlap with H2D.

How can selective KV transfer retain low launch overhead?

The pipeline must transfer only the batch’s streamed prefixes, whose host addresses, destination rows and lengths can change each step. A CUDA graph, however, captures a fixed set of operations and dependencies. The difficulty is bidirectional: a transfer must wait until the previous reader releases its slot, and a later attention operation inside the same decode step must wait for that transfer. Figure 8 compares three designs for selective KV transfer and low launch overhead.

Selective KV transfer designs

Figure 8. Selective KV transfer with low launch overhead: (a) Graph Segmentation transfers selected KV but incurs repeated launches; (b) Fixed-Range Transfer retains low launch overhead but moves extra KV; (c) Graph-Compatible Handshake achieves both. Timelines are schematic.

Graph Segmentation (Figure 8a) divides the decode graph into segments and submits dynamic copies and their dependencies between segment launches. The boundaries let the host submit a copy and its completion record before enqueueing the consumer’s wait. This supports selective KV transfer, but repeated graph launches and host coordination can expose submission bubbles. Fixed-Range Transfer (Figure 8b) captures copies of a fixed host range into the slots alongside computation, retaining low launch overhead through a full decode graph. The copies also move inactive rows and unused capacity, however, wasting link bandwidth and delaying attention when the extra transfers exceed the available overlap.

Our Graph-Compatible Handshake (Figure 8c) retains both properties: low launch overhead and selective KV transfer. Before replay, the Cache Streamer assigns the batch’s streamed prefixes to contiguous slot rows, updates their page-table entries, and coalesces adjacent host rows into source–destination copy runs. This preparation changes metadata only; it adds no KV staging copy. The Cache Streamer submits these runs with cudaMemcpyBatchAsync ([14]) on the H2D stream, copying directly from the host cache into the slots without transferring KV for requests outside the batch.

Two persistent flags coordinate each layer’s use of a slot. On the copy stream, a transfer waits for and clears FREE from the previous reader, copies the selected rows, and publishes READY. Inside the graph, small kernels wait for and clear READY before the layer’s first KV access and publish FREE after its final read. The arrows in Figure 8c illustrate both dependencies: completing KV1’s reads permits KV3’s transfer, whose completion permits KV3’s reads. Each flag is cleared when consumed, preventing a previous step’s signal from satisfying the current wait. The first use of a slot waits for its last reader in the previous step; both slots start free.

The host submits the first transfer into each slot, launches the full decode graph once, and then submits the remaining transfers. Execution-time flag checks let the graph wait for these transfers even though their submission follows its launch. Thus DMA addresses and lengths can vary while graph nodes and flag addresses stay fixed. The host need not wait for individual layers, and a late transfer stalls its consumer until the KV is ready. Each auxiliary kernel uses one warp and replays within the graph, without a separate host launch. This preserves selective-transfer efficiency without fragmenting the decode graph; the combined implementation overhead is evaluated in § 5.3.

How much can streaming expand effective global-KV capacity? For a fixed GPU cache budget, the useful expansion is bounded by how much host-resident KV can be delivered within each decode step. With LL global-attention layers, bb bytes per row and layer, computation time TT and sustained link bandwidth BW\mathrm{BW}, streaming AA rows transfers LAbLAb bytes per step. To hide these transfers, we require LAb≤ηBWTLAb \le\eta\mathrm{BW} T, where η<1\eta< 1 leaves scheduling slack. The resulting hidden bound CC limits admission (§ 4.4). With M=2M = 2 streaming slots, the bound and relative capacity gain are

C=ηBWTLb,r=C(1−M/L)D(1)C = \frac{\eta\mathrm{BW} T}{L b}, \qquad r = \frac{C(1-M/L)}{D} \tag*{(1)}

where DD is the capacity, in rows, of an entirely resident global-KV pool with the same GPU budget. Slots for CC streamed rows consume the equivalent of MC/LMC/L resident rows, leaving a net gain of C(1−M/L)C(1-M/L). Ignoring page rounding, effective capacity therefore reaches (1+r)D(1+r)D, provided the slots fit within the GPU budget. This is an aggregate bandwidth bound; transfers must also meet each layer’s deadline.

In the GPT-OSS-120B ablation (§ 5.3), L=18L = 18, b=2b = 2 KiB, M=2M = 2, and D=147,648D = 147{,}648. The sizing calibration (T=13.16T = 13.16 ms, BW=400\mathrm{BW} = 400 GB/s, η=0.9\eta= 0.9) gives C≈128.5C \approx128.5K and 77% capacity gain. The configured 126,976 streaming rows plus 133,056 resident rows yield 260,032 tokens after page rounding—1.76× the original capacity.

Algorithm 1 Request Admission

Input: rr: (ns,nf,h)(n_s, n_f, h); queue QQ; streamed rows AA; bound CC

  1. if ns=0n_s = 0, then Grow(SS); return

  2. if h≥hmin⁡h \ge h_{\min} and room(GG) - h<Pressure(Q)h < \mathrm{Pressure}(Q) then

  3. if A+h≤CA + h \le C then

  4. Admit(rr, streamed); A←A+hA \gets A + h; return

  5. else if h≤Ch \le C then

  6. Skip(rr); return

  7. end if

  8. end if

  9. if nf+h≤room⁡(G)n_f + h \le\operatorname{room}(G) then

  10. Admit(rr, resident)

  11. else

  12. Grow(GG)

  13. end if

  14. Pressure(Q)=∑q∈Qneed⁡(q)+Nℓˉ/2+λℓˉϕˉ\mathrm{Pressure}(Q) = \sum_{q \in Q} \operatorname{need}(q) + N\bar{\ell}/2 + \lambda\bar{\ell}\bar{\phi}

  15. procedure Grow(XX) {TCP, next epoch}

  16. n←need⁡(X)−free⁡(X)n \gets\operatorname{need}(X) - \operatorname{free}(X)

  17. if n≤idle⁡n \le\operatorname{idle} then n←n⋅min⁡(∣Q∣,k)n \gets n \cdot\min(|Q|, k)

  18. Take(nn) {free, idle, cold}

  19. Park; Fence; Remap; Apply

  20. end procedure

For a fixed model and slot count, more bandwidth or longer computation increases the expansion; a larger per-token KV footprint or original resident capacity reduces the relative gain. We size CC using an offline decode calibration at the expected batch. Extra capacity serves host-resident historical prefixes; newly generated KV remains resident. Higher concurrency follows when global-KV capacity limits admission and the other cache components have room.

Request Scheduler

The Request Scheduler coordinates TCP and the Cache Streamer to admit more requests within the available GPU capacity and transfer budget. For the head request rr, it considers its state slots nsn_s, uncached global-KV rows nfn_f, and host-cached prefix length hh. A resident turn holds nf+hn_f + h resident rows. A streamed turn keeps nfn_f rows resident and uses hh streaming-slot rows, transferring hbhb bytes per layer each step; total streamed rows must stay within CC (§ 4.2). If capacity is insufficient, TCP can grow the blocking subpool using free pages, idle capacity, or cold cache. Algorithm 1 gives the decision order.

A request is streamed only when the resident region is contended: Pressure projects the region’s demand over one turn from the rows the waiting requests need, the decode growth of the NN running requests over a mean output length ℓˉ\bar{\ell}, and the arrivals during that turn (λ\lambda per step, prompt length pˉ\bar{p}, cache-miss fraction ϕ\phi), all count-based running averages of events every rank observes. A request selected for streaming skips this iteration if the slots are full, allowing requests needing other resources to proceed. room(GG) counts the resident region’s free rows and cached rows with a host copy, minus a reserve for in-flight prefills; hmin⁡h_{\min} is the minimum prefix size worth streaming.

Evaluation

Experimental Setup

Implementation. TideCache adds about 9K lines to SGLang, mainly in a memory-pool package. It supports multi-head, latent and sparse-indexed global attention, sliding-window caches and recurrent states, while preserving attention backends and the CUDA graph runner.

Testbed. We use two superchip platforms. The GB300 node has four NVIDIA GB300 GPUs with 288 GB of HBM3e each, connected by NVLink, and two 72-core Grace CPUs with 960 GB of LPDDR5X host memory; each GPU reaches host memory over NVLink-C2C at a measured 211 GB/s host-to-device. The GH200 superchip pairs one 72-core Grace CPU with 480 GB of LPDDR5X and one H100 GPU with 96 GB of HBM3 over NVLink-C2C.

Models. Table 3 lists seven model/platform configurations, covering latent, grouped-query and sparse-indexed global attention over three kinds of recurrent layer and sliding-window attention. Kimi K3 and Qwen3.8-2.4T use their first 48 and 36 layers to fit in HBM, preserving the cache geometry per layer. SGLang’s memory fraction is left at its normal deployment value (0.80–0.95).

ModelHybrid combinationTPPlatform
Kimi K3 (48L)MLA : KDA = 1 : 34GB300
Ring-2.6-1TMLA : LA = 1 : 74GB300
GLM-5.3-FlashDSA : KDA = 11 : 341GB300
Qwen3.8-2.4T (36L)GQA : GDN = 1 : 34GB300
GLM-5.3-FlashDSA : KDA = 11 : 344GH200
Qwen3.8-27BGQA : GDN = 1 : 31GH200
GPT-OSS-120BGQA : SWA = 1 : 11GH200

Table 3. Hybrid layer combinations and hardware configurations of the evaluated models. MLA: multi-head latent attention; DSA: DeepSeek sparse attention; GQA: grouped-query attention; KDA: Kimi Delta Attention; GDN: Gated DeltaNet; LA: Lightning linear attention; SWA: sliding-window attention.

Workloads and metrics. We construct synthetic mixtures from ShareGPT request traces (anon8231489123, 2023) and multi-turn Claude Code traces released by SemiAnalysis (SemiAnalysis, 2026). The former have 896–1152 input tokens and 703 output tokens on average, and arrive as an open-loop Poisson process. The latter have 40K–110K-token contexts and 8–28 turns; closed-loop replay preserves tool gaps and feeds outputs into subsequent turns. Both sources share one first-come-first-served queue, without application labels or priorities supplied to the scheduler. We vary the independent-request arrival rate and the number of concurrent sessions; configuration labels use req/s×sessionsreq/s \times sessions. After warm-up (360 s on GB300; at least 600 s on GH200), the measurement window is 600 s or 900 s for the shifting workload. Turns/s is the number of agent-session turns completed within the window divided by the window duration. P50 per-token latency is the median, over all requests submitted within the window, of end-to-end request latency divided by actual output-token count. P50 TTFT is the median submission-to-first-token latency for session requests submitted within that window. Turns/s and TTFT thus describe agent sessions, while per-token latency covers the full mixture; both latencies include prefill and queueing, and they count the submitted cohort’s completions after the window.

End-to-End Performance

We evaluate performance under memory pressure across seven configurations, then examine adaptation to a workload whose demand changes during execution.

Performance under memory pressure. Figure 9 uses one load point per configuration, where session prefixes exceed the resident global-KV capacity. Without HiCache, repeated prefix computation occupies the GPU and delays subsequent requests. HiCache avoids much of this recomputation, but each restored prefix remains in GPU memory throughout its turn, limiting the sessions that can proceed concurrently. TideCache relieves this capacity constraint by reusing GPU memory across global-attention layers. Relative to SGLang w/ HiCache (Tuned), it completes up to 2.04× as many turns, reduces overall P50 per-token latency by 20–59%, and improves agent P50 TTFT by 1.40–3.05×. TideCache reduces TTFT by relieving memory pressure and shortening the wait to serve session prefixes. Turn gains reach up to 2.04× on GH200 and 1.44× on GB300, consistent with the larger capacity benefit when less HBM remains for global-KV. Section 5.3 examines the effect of baseline partition tuning.

Figure 9. End-to-end performance across seven model/platform configurations. (a) P50 latency / token speedup across all requests; (b) turns/s and (c) P50 TTFT speedups for agent sessions. Both baselines use their best tested split ratios. Values are normalized to SGLang w/ HiCache (Tuned). All panels are higher-is-better. GH200 is on the left, GB300 on the right.

Adaptation to workload changes. Figure 10 runs Qwen3.8-27B on GH200 after a 900 s warm-up. The 900 s measurement comprises three 300 s phases: independent-request arrivals rise from 0.05 to 1.6 req/s and then return to 0.05 req/s. Ten agent workers replay ten sessions before and after the burst; new agent turns pause during the burst while in-flight requests finish. This shifts demand between global-KV and bounded state. The static baselines retain a state : global-KV ratio of 1.0, while both TideCache variants start at 0.6 and adapt online. SGLang, SGLang w/ HiCache, TideCache w/o streaming, and TideCache complete 57, 115, 174, and 183 agent turns, respectively. Relative to HiCache, TideCache completes 59% more turns, reduces overall P50 per-token latency from 63.4 to 39.3 ms (38%), and reduces agent P50 TTFT from 25.70 to 5.68 s (78%).

Figure 10. Qwen3.8-27B on GH200 under a shifting workload, with a burst at 300–600 s. (a) Overall P50 latency / token; (b) turns/s and (c) P50 TTFT for agent sessions. (d–e) Physical GPU capacity allocated to each cache component, including occupied and free space within its pool, alongside running and queueing request counts. Unassigned denotes pages outside both pools.

The timelines expose the difference between fixed and adaptive allocation. HiCache’s pool capacities remain unchanged as the workload shifts. TCP grows TideCache’s bounded-state allocation from about 5.6 to 17.6 GiB during the burst, then returns it to about 6.0 GiB as agent activity resumes. Roughly half of the GPU cache budget changes owners, making capacity available to the component under pressure. The static baselines reach their native 32-request running limit; HiCache’s burst queue peaks at 74 requests, compared with 15 for TideCache. Streaming further expands effective global-KV capacity for session prefixes: relative to the no-streaming variant, it increases completed turns from 174 to 183 and reduces agent P50 TTFT from 8.13 to 5.68 s (30%).

Ablation Study

We examine the Transparent Cache Pool, sensitivity to the split ratio, and the implementation overhead in ablations.

Standalone TCP Performance. Figure 11 isolates TCP by disabling host caching and layer-wise streaming, comparing it with fixed partitions and UMP. Both TCP and UMP dynamically share GPU capacity; we test whether TCP matches UMP’s throughput while preserving existing cache interfaces. UMP’s HiCache integration was incomplete in the evaluated revision, so we include it only in this GPU-only comparison. Each model uses three mixtures, with large/small fixed state : global-KV ratios of approximately 1.3/0.26 for Kimi K3 and 2.8/0.85 for GLM-5.3Flash. TCP starts from one allocation per model and adapts online. Across six points, it achieves 96.3–113.6% of the better fixed partition’s turns/s without per-mixture tuning. It matches UMP on Kimi K3, with 0–3.7% higher turns/s. N/A denotes GLM’s DSA backend, unsupported by UMP revision 25ce8063; TCP supports its existing layout.

Figure 11. Dynamic cache partitioning on Kimi K3 and GLM-5.3-Flash. Turns/s is normalized to the better fixed partition; configurations denote req/s ×\times concurrent sessions.

Sensitivity to Split Ratio. Figure 12 illustrates the base-line tuning used in Figure 9, sweeping the split ratio from 2 to 6 on Kimi K3. SGLang w/ HiCache peaks at ratio 3 with 67 completed turns, the setting used in the main comparison, versus 44–59 at the other tested ratios: allocating too little memory to state limits admission, while allocating too little to global-KV limits resident prefixes. The TideCache reference completes 94 turns, 40% above even the best tested baseline, and has 29.5% lower overall P50 per-token latency at that starting ratio. Thus tuning the baseline’s fixed partition does not close the gap.

Sensitivity to the state : global-KV split ratio on Kimi K3

Figure 12. Sensitivity to the state : global-KV split ratio on Kimi K3. Horizontal lines show TideCache starting from ratio 3.

System Overhead. Figure 13 compares median decode throughput over three runs per batch size, with the same GPU cache budget and SWA : global-KV split. The measurement includes the combined implementation on this path: transfer submission, page-table updates, and CUDA-graph coordination. At batch sizes both systems sustain, TideCache’s decode-step time differs from SGLang w/ HiCache by −1.1% to +6.3%. The added capacity compensates for this cost: effective global-KV capacity rises from 148K to 260K tokens, and the largest sustained batch grows from 16 to 29. Peak decode throughput rises from 1.19K to 1.47K tokens/s, a 24% gain.

Figure 13. Median decode throughput versus batch size on GPT-OSS-120B / GH200 with 8K-token contexts.

Related Work

Hybrid Cache Management. Static partitioning fixes each cache component’s GPU budget at start-up. SGLang’s

Unified Memory Pool instead shares a byte buffer among subpools and compacts live slots by copying (SGLang Team, [23]); vLLM’s Hybrid KV Cache Manager dynamically allocates shared blocks to cache groups (vLLM Team, 2026). Jenga uses LCM-sized parent pages in a two-level allocator for heterogeneous caches (Zhang et al., [36]). These designs enable sharing through software-level cache layouts and mappings. vAttention first backed a contiguous virtual KV cache with physical pages on demand through the CUDA virtual memory API (Prabhu et al., [18]), and Prism remaps physical pages between models sharing a GPU (Yu et al., 2025). TideCache applies the same primitive across the cache components of one hybrid model, adds a region whose pages are shared by all global-attention layers, and drives the remapping from the admission queue.

KV Cache Offloading. FlexGen schedules weights, activations and KV across GPU, CPU and disk (Sheng et al., [24]); ZeRO-Inference streams layer weights from CPU or NVMe to support large inference batches (DeepSpeed Team, [4]). Pie swaps the KV of transformer layers between host and device so that the active set exceeds GPU memory (Xu et al., 2024); InfiniGen and HiSparse fetch only the KV that attention will actually read (Lee et al., [11]; Xie et al., [30]); SuperInfer rotates requests through the GPU and uses both transfer directions to keep latency bounded on superchips (Yu et al., 2026). TideCache exploits hybrid structure: only global-attention layers hold context-proportional data, while bounded-state computation provides time to overlap its transfer. This makes per-step streaming affordable when decode provides sufficient computation.

Conclusion

We presented TideCache: the Transparent Cache Pool transparently repartitions HBM, the Cache Streamer overlaps layer-wise transfers with computation, and the Request Scheduler coordinates admission. Across seven model/platform configurations on GB300 and GH200, it completes up to 2.04×\times as many agent turns, reduces overall P50 per-token latency by 20–59%, and improves agent P50 TTFT by 1.40–3.05×\times over SGLang with HiCache.

AI USAGE DISCLOSURE

We used OpenAI Codex interactively to assist with literature discovery and source checking, manuscript drafting and revision, terminology and presentation choices, and figure design. Codex also assisted in writing and modifying plotting and data-analysis scripts, inspecting recorded experimental results, and checking consistency between figures and reported metrics. This assistance was iterative and guided by the authors, covering multiple manuscript sections and evaluation figures. AI assistance did not extend to system implementation, experiment design or execution, or workload construction. The authors retain full responsibility for the research claims, experimental methodology, reported results, citations, and final manuscript.

References

  1. [2]Aubrey, K. Inside the NVIDIA Vera Rubin Platform: Six New Chips, One AI Supercomputer. NVIDIA Technical Blog, January 2026. URL https://developer.nvidia.com/blog/inside-the-nvidia-rubin-platform-six-new-chips-one-ai-supercomputer/. Published January 5, 2026. Accessed October 24, 2026.
  2. [3]Aubrey, K. and Stam, N. Inside NVIDIA Blackwell Ultra: The Chip Powering the AI Factory Era. NVIDIA Technical Blog, August 2025. URL https://developer.nvidia.com/blog/inside-nvidia-blackwell-ultra-the-chip-powering-the-ai-factory-era/. Published August 22, 2025. Accessed October 24, 2026.
  3. [4]DeepSpeed Team. ZeRO-Inference: Democratizing massive model inference. DeepSpeed Blog, September 2022. URL https://www.deepspeed.ai/2022/09/09/zero-inference.html.
  4. [5]Evans, J., Andersch, M., Sethi, V., Brito, G., and Mehta, V. NVIDIA Grace Hopper Superchip Architecture In-Depth. NVIDIA Technical Blog, November 2022. URL https://developer.nvidia.com/blog/nvidia-grace-hopper-superchip-architecture-in-depth/. Published November 10, 2022. Accessed October 24, 2026.
  5. [7]Huang, Z., Bao, K., Zhang, Y., Ouyang, J., and Pan, S. Unified Radix Cache: One tree for hybrid model prefix caching. LMSYS Org Blog. https://www.lmsys.org/blog/2026-08-11-unified-radix-cache/, August 2026. Accessed: 2026-10-08.
  6. [8]Kimi Team. Kimi Linear: An expressive, efficient attention architecture, 2025. URL https://arxiv.org/abs/2510.26692.
  7. [9]Kimi Team. Kimi K3: Open frontier intelligence, 2026. URL https://arxiv.org/abs/2607.24653.
  8. [10]Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention, 2023. URL https://arxiv.org/abs/2309.06180.
  9. [11]Lee, W., Lee, J., Seo, J., and Sim, J. InfinGen: Efficient generative inference of large language models with dynamic KV cache management, 2024. URL https://arxiv.org/abs/2406.19707.
  10. [12]Li, A., Liu, B., Han, B., Hu, B., Jing, B., Hu, B., Li, B., Chen, C., Tang, C., Tian, C., Huang, C., Zhang, C., Liang, C., Qian, C., Tang, C., Wen, C., Fu, C., Wu, C., Zhang, C., Peng, C., Wang, D., Zhang, D., Zhao, D., Jin, D., Zhu, D., Zhang, D., Yuan, F., Zhao, F., Meng, F., Wu, F., Xu, F., Fang, F., Wang, G., Yang, G., Zhao, H., Wang, H., Zhang, H., Zhang, H., Wang, H., Dai, H., Liu, H., Qian, H., Wu, H., Liu, H., Xu, H., Zhang, H., Liu, H., Zhang, H., Liu, H., Li, H., Ruan, H., Xiong, H., Zheng, H., Tang, H., Guo, J., Li, J., Liu, J., Wang, J., Liu, J., Shi, J., Wei, J., Yang, J., Wang, J., Gao, J., Wang, J., Wu, J., Yang, J., Li, J., Huang, J., Sun, J., Chen, J., Tu, J., Liu, J., Mei, J., Xu, J., Zhou, J., Ou, J., Sipan, J., Fang, J., Zhang, K., Hu, K., Shi, K., Xu, K., Tang, K., Chen, K., Mei, L., Chen, L., Liang, L., Xu, L., Tang, L., Jiang, L., Fu, L., Zhang, L., Shi, L., Ma, L., Liu, L., Li, L., Zheng, L., Liu, L., Yu, L., Li, M., Zhu, M., Li, M., Gao, M., Sun, M., Yin, M., Zhang, M., Fan, M., Xu, N., Tang, P., Jiang, P., Zhao, P., Lin, P., Liu, P., Zuo, Q., Zhao, Q., Cheng, Q., Cao, Q., Bao, Q., Cui, Q., Yang, Q., Shi, Q., Huang, Q., Zhou, Q., Wan, Q., Zhao, R., Zheng, S., Wei, S., Zhang, S., Li, S., Li, S., Zhang, S., Bian, S., Yao, T., Xu, T., Wang, T., Guo, T., Wang, T., Huang, T., Zhao, T., Yang, T., Hong, W., Gu, W., Lu, W., Wu, W., Han, W., Li, W., Shen, W., Fang, W., Tang, W., Shu, X., Shi, X., Yan, X., Zhang, X., Wan, X., Sun, X., Zhao, X., Lu, X., Yang, X., Tang, X., Kong, X., Liu, X., Xu, X., Sun, X., Han, X., Wang, X., Shen, X., Zhang, Y., Hou, Y., Ren, Y., Zhao, Y., Chen, Y., Chen, Y., Cao, Y., Zuo, Y., Chen, Y., Li, Y., Song, Y., Li, Y., Wang, Y., Sun, Y., Xiao, Y., Xu, Y., Liu, Y., Fang, Y., Gao, Y., Yu, Y., Zhang, Y., Zhang, Y., He, Y., Lu, Y., Tian, Y., Li, Y., Fu, Y., Xu, Z., Huan, Z., Zhang, Z., Gui, Z., Huang, Z., Ma, Z., Pan, Z., Qu, Z., Zhu, Z., Fan, Z., Huangfu, Z., Wang, Z., Zhang, Z., Liu, Z., Zhou, Z., Lin, Z., Zeng, Z., Wang, Z., Wang, Z., Liu, Z., Xuan, Z., Cheng, Z., Wen, Z., and Tang, Z. Ling and ring 2.6 technical report: Efficient and instant agentic intelligence at trillion-parameter scale, 2026. URL https://arxiv.org/abs/2606.15079.
  11. [13]Lin, C., Han, Z., Zhang, C., Yang, Y., Yang, F., Chen, C., and Qiu, L. Parrot: Efficient serving of LLM-based applications with semantic variable. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 929–945, 2024. URL https://www.usenix.org/conference/osdi24/presentation/lin-chaofan.arxiv.org/abs/2405.19888
  12. [14]NVIDIA. CUDA runtime API: Memory management, 2026a. URL https://docs.nvidia.com/cuda/c uda-runtime-api/group__CUDART__MEMORY.html. cudaMemcpyBatchAsync; accessed October 30, 2026.
  13. [15]NVIDIA. CUDA driver API: Virtual memory management, 2026b. URL https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__VA.html. Accessed October 30, 2026.
  14. [16]OpenAI. gpt-oss-120b & gpt-oss-20b model card, 2025. URL https://arxiv.org/abs/2508.10925.
  15. [17]Peng, H., Kasai, J., Pappas, N., Yogatama, D., Wu, Z., Kong, L., Schwartz, R., and Smith, N. A. ABC: Attention with bounded-memory control. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), 2022.arxiv.org/abs/2110.02488
  16. [18]Prabhu, R., Nayak, A., Mohan, J., Ramjee, R., and Panwar, A. vAttention: Dynamic memory management for serving LLMs without PagedAttention. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2025.arxiv.org/abs/2405.04437
  17. [19]Qiu, Z., Wang, Z., Li, X., Li, Y., et al. On the design of Qwen3.8-Next architecture: Evaluation, efficiency, and training stability, 2026. URL https://arxiv.org/abs/2608.30320.
  18. [20]Qwen Team. Qwen3.8-Max: A new bar for coding and cowork, August 2026. URL https://qwen.ai/blog?id=qwen3.8.
  19. [22]SGLang Contributors. Unified memory pool compatibility checks. SGLang source code, revision 25ce8063, 2026. URL https://github.com/sgl-project/sglang/blob/25ce8063f7dbe444c4e24c562f31f86268e05a23/python/sglang/srt/arg_groups/kv_cache_hook.py. Accessed 2026-10-31.
  20. [23]SGLang Team. Unified memory pool for hybrid models. SGLang GitHub repository, revision 25ce8063, 2026. URL https://github.com/sgl-project/sglang/blob/25ce8063f7dbe444c4e24c562f31f86268e05a23/python/sglang/srt/mem_cache/unified_memory_pool.py.
  21. [24]Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Fu, D. Y., Xie, Z., Chen, B., Barrett, C., Gonzalez, J. E., Liang, P., Ré, C., Stoica, I., and Zhang, C. Flexgen: High-throughput generative inference of large language models with a single gpu, 2023. URL https://arxiv.org/abs/2303.06865.
  22. [28]Xiang, Y., Li, X., Qian, K., Zhang, Y., Yu, W., Zhai, E., Jin, X., and Zhou, J. ServeGen: Workload characterization and generation of large language model serving in production. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26), pp. 1845–1859, 2026. URL https://www.usenix.org/conference/nsdi26/presentation/xiang-servegen.
  23. [29]Xie, Z. SGLang HiCache: Fast hierarchical KV caching with your favorite storage backends. LMSYS Org Blog, September 2025. URL https://www.lmsys.org/blog/2025-09-10-sglang-hicache/.
  24. [30]Xie, Z., Huang, Z., Huang, T., Xu, Z., Ma, R., and Kozyrakis, C. HiSparse: Scaling sparse-attention decoding with hierarchical KV cache management, 2026. URL https://arxiv.org/abs/2608.07009.
  25. [32]Yang, S., Kautz, J., and Hatamizadeh, A. Gated delta networks: Improving Mamba2 with delta rule, 2025. URL https://arxiv.org/abs/2412.06464.
  26. [35]Z.ai. GLM-5.3-Flash: Frontier intelligence, flash cost. https://z.ai/blog/glm-5.3-flash, 2026. Accessed: 2026-10-08.
  27. [36]Zhang, C., Du, K., Liu, S., Kwon, W., Mo, X., Wang, Y., Liu, X., You, K., Li, Z., Long, M., Zhai, J., Gonzalez, J., and Stoica, I. Jenga: Effective memory management for serving LLM with heterogeneity, 2025. URL https://arxiv.org/abs/2503.18292.
  28. [37]Zhang, Y. and SGLang Team. Unified memory in SGLang: 1.9× as many prefixes retained, 37.9% less time for returning requests. SGLang Blog, 2026. URL https://www.sglang.io/blog/unified-memory. Accessed 2026-10-29.
  29. [38]Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. Sglang: Efficient execution of structured language model programs, 2024. URL https://arxiv.org/abs/2312.07104.

Paper details

vlnc:W_yhtytgf6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 11

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Codex

View statement
Looking for related papers…
Contents