WorksComputer Science
Request-Adaptive Execution for Efficient Hybrid LLM Inference
Abstract
Hybrid large language models (LLMs) have emerged as a promising way to combine the modeling capability of the Attention mechanism with the efficiency of state space models (SSMs). However, most existing hybrid LLMs still inherit a static design philosophy: once the Attention-to-SSM layout is chosen, every request must follow the same hybrid execution. This one-layout-fits-all design is inefficient for real-world serving workloads, where different requests rely on contextual information in different ways. This paper presents Morph, a serving system that co-designs the model architecture and runtime for request-adaptive hybrid LLM inference. Its model architecture pairs Attention and SSM layers within each block and uses prefill-oriented gating to select a request-specific layout that remains fixed throughout decoding. Its serving runtime preserves batching efficiency under divergent layer choices through selective batching and enables correct prefix reuse through hybrid-aware context caching. Experiments on six datasets show that Morph achieves a better quality-efficiency trade-off than existing static hybrid baselines. System-level evaluation further shows that Morph improves the serving throughput by up to 3.87x over baselines while substantially reducing inference latency.
Introduction
Large language models (LLMs) have become the foundation of modern AI services, powering applications such as search [31, 40], coding assistants [20], document analysis [18, 49], and autonomous agents [24, 34]. Most deployed LLMs are based on the Transformer architecture, where the attention mechanism provides strong global-context modeling but incurs quadratic computation and growing KV-cache memory during inference [43, 50, 56]. To reduce this cost, state space models (SSMs), such as Mamba [9, 16], have been introduced as efficient sequence modeling primitives. SSMs summarize prior tokens into compact recurrent states and update them in linear time. However, this compact recurrent state may limit the ability to recall context-dependent information, making SSMs less effective on tasks that require precise long-range context modeling [46].
To combine the complementary strengths of attention mechanisms and SSMs, recent works have proposed hybrid LLMs1, which interleave Attention layers and SSM layers within the same model to balance model quality and inference efficiency. Representative works such as Jamba [22], Mamba-LLM [38], and Nemotron-H [3] have demonstrated a better quality-efficiency trade-off than Attention-only or SSM-only architectures. A core design factor behind the performance of hybrid LLMs is the Attention-to-SSM layout, which specifies not only how many Attention and SSM layers are used, but also where these layers are placed in the model.
Existing hybrid LLMs typically rely on hand-crafted architecture designs, where the Attention-to-SSM layout is selected based on empirical studies. In addition, this layout remains static for all requests once the model is constructed. For example, Jamba adopts an rotation strategy, inserting Attention layers for every SSM layers. As illustrated in Figure 1(a), such static designs assume requests with different characteristics, e.g., summarization, QA, and reasoning, benefit from the same hybrid structure. However, our preliminary experiments in § 2.2 show that this assumption does not always hold in real-world inference workloads. A layout that achieves the best accuracy on a reasoning task such as ARC-Challenge can lead to suboptimal quality on a summarization task such as PubMed. This inconsistency arises because different workloads rely on contextual information in different ways. For example, reasoning tasks may only require selected Attention layers for option comparison, while long-document summarization often depends more heavily on fine-grained global context modeling to identify and preserve salient information. As a result, a static Attention-to-SSM layout can either overuse Attention for simple requests, wasting computation, or underuse Attention for context-sensitive requests, sacrificing output quality.
Key Idea. Our key idea is to make the hybrid LLM request-adaptive: rather than fixing the Attention-to-SSM layout, the model selects an appropriate hybrid layout for each incoming request. As illustrated in Figure 1(b), different requests can activate different Attention and SSM layers according to their context-modeling demands. We realize this idea through algorithm-system co-design in Flex, an inference system that integrates a request-adaptive hybrid model with runtime optimizations for efficient serving. The hybrid model consists of multiple adaptive hybrid blocks, each containing a lightweight gate, paired Attention and SSM branches, and shared components such as the FFN. For each request, the gates select one branch per block, forming a request-specific execution layout that balances modeling capability and inference cost.
Figure 1. (a) Hybrid LLMs use the static Attention-to-SSM layout for all requests. (b) Request-adaptive hybrid layout allows different requests to activate different Attention and SSM execution. Colored lines indicate layer selections.
A key design choice is the gating granularity. A natural token-level gating design would allow each token to independently choose whether to execute the Attention or SSM layer in each hybrid block. However, this design is ill-suited for autoregressive decoding, where newly generated tokens rely on the cached results of previous tokens, i.e., Attention KVs and SSM states, to avoid recomputation. If tokens from the same request choose different layers within the same block, later tokens may not be able to directly reuse the Attention KVs or SSM states produced by earlier tokens, forcing the runtime to recompute previous tokens under the new layer choice. Flex therefore adopts prefill-oriented gating mechanism (§ 4): during prefill, the gate uses all input tokens to make one request-level Attention and SSM choice for each hybrid block, and these choices are fixed throughout decoding. This allows the Attention KVs or SSM states built by previously generated tokens to be reused consistently.
At the system level, request-adaptive layouts introduce two challenges: requests within the same batch may select different branches, and requests sharing a textual prefix may require incompatible layer-wise caches. The serving runtime in Flex addresses these challenges through the following two designs to preserve batching efficiency and ensure correct context reuse.
Selective Batching Engine. To maintain high inference throughput, Flex must efficiently batch multiple requests for parallel execution. However, requests in the same batch may select different branches at each hybrid block, which breaks the traditional batching assumption of conventional serving engines, where all requests in a batch follow the same operator sequence [12, 21, 32]. A naive solution would either serialize or execute block computations separately. The former loses parallelism, while the latter fragments the original batch and prevents request-independent operations, such as normalization and FFN computation, from being executed as a single compact batch [55].
Flex addresses this with selective batching engine (§ 5). The key idea is to keep branch-independent operations on the original packed batch, and split requests only for the branch-dependent Attention and SSM computation. At each block, it partitions requests into compact Attention and SSM sub-batches, executes them with their corresponding kernels, and then merges outputs back into the original batch layout. Since two branches may have different execution costs, Flex further applies load-aware branch coordination by assigning more resources to heavier branch, reducing synchronization stalls before merging.
Hybrid-Aware Context Caching. Real-world LLM serving workloads often contain repeated long prefixes, such as system prompts, document contexts, and multi-turn conversation histories. Recomputing these prefixes for every request wastes substantial prefill computation, especially in long-context scenarios. Context caching [19, 41, 45] addresses this issue by identifying matched textual prefixes and reusing their previously generated caches. Existing techniques typically assume that a matched textual prefix uniquely determines all reusable caches. However, this assumption no longer holds for Flex: even when two requests share the same prefix, they may select different Attention and SSM layers across hybrid blocks, producing different layer-wise caches. Moreover, conventional cache eviction operates at the context level, where evicting a prefix removes all layer-wise caches associated with it. This coarse-grained eviction is inefficient for request-adaptive hybrid execution, because only a subset of layer caches under a matched context may be frequently reused. Evicting the whole context can reduce future cache hit ratios, while retaining it may keep many rarely used layer caches in memory.
To address these limitations, Flex introduces the hybrid-aware context caching (§ 6). Its core mechanism, a two-level radix tree structure, decouples textual-prefix matching from layer-wise cache matching. After identifying a shared textual prefix, it further determines which layer-wise caches are compatible with the current hybrid layout. Built on this structure, Flex further performs priority-based eviction at
Background and Motivation
Background
Attention-based LLMs. Attention-based LLMs rely on multi-head self-attention to capture global token relationships [36]. Given the hidden representations , the Attention layer projects them into queries (), keys (), and values () tensors and computes:
where denotes the head dimension. This dense interaction allows each token to directly aggregate information from previous tokens, providing strong capability for reasoning and long-context understanding. However, the computation of an Attention layer grows quadratically with the prompt length during prefill [43, 50, 54, 56, 57]. During decoding, it needs to maintain KV caches for preceding tokens, introducing increasing memory and computation costs as the context grows.
SSM-based LLMs. State space model (SSM), such as Mamba model [9, 16], provide an efficient alternative by summarizing historical information into compact recurrent states:
where is the SSM state at time , and define the input-dependent state update, maps the state to the output, and is a skip connection. Unlike conventional linear time-invariant SSMs, Mamba makes the state update input-dependent, enabling selective preservation of information from the input sequence [16]. Since an SSM layer compresses historical tokens into a bounded-size state, it avoids the growing Attention KV cache required by Attention-based LLMs and supports efficient long-sequence inference. Mamba-2 further improves the efficiency of selective SSM computation through state-space duality [9]. However, this compact recurrent state may limit the ability to recall context-dependent information, making SSMs less effective on tasks that require precise long-range context modeling [46].
Hybrid LLMs. Hybrid LLMs combine Attention layers and SSM layers within the same model to balance modeling capability and inference efficiency, as shown in Figure 2. A key design factor is the Attention-to-SSM layout, which determines how frequently Attention and SSM layers are used. In Figure 2(d), we evaluate layouts of 1 : 1, 1 : 3, 1 : 6, and 1 : 12, where 1 : layout denotes one Attention layer for every SSM layers. The SSM-only configuration achieves the lowest computation cost but suffers substantial accuracy degradation, whereas the Attention-only configuration incurs much higher cost. Several hybrid layouts achieve higher accuracy than both endpoints with moderate computation cost, demonstrating the advantage of combining Attention and SSM layers. Moreover, the performance varies noticeably across layouts, indicating that the hybrid layout is a critical factor in determining the quality-efficiency trade-off.
Figure 2. Overview and quality-efficiency trade-off of conventional hybrid LLMs. (a)–(c) Illustrations of Attention-based, SSM-based, and conventional hybrid architectures. (d) PIQA accuracy versus model efficiency under different Attention-to-SSM layouts. All models have 3B parameters.
Motivation
The quality-efficiency trade-off of a hybrid LLM is largely determined by its Attention-to-SSM layouts. Existing hybrid LLMs determine this configuration based on empirical studies and keep it static during model serving. For example, Jamba [22] periodically inserts Attention layers into an SSM-dominated backbone, while Nemotron-H [3] fixes a predefined distribution of Attention and SSM layers. Once constructed, all requests are forced to execute the same layout, regardless of their input characteristics or contextual requirements. We find that such static layouts are insufficient for diverse inference workloads: different tasks, and even different requests from the same task, can prefer different uses of Attention and SSM computation.
Observation 1: No single Attention-to-SSM layout can achieve the best performance across different workloads.
We first study whether a single static hybrid layout can achieve a consistently good quality-efficiency trade-off across different workloads. We use a set of static hybrid models from [37], which combine Attention layers with SSM layers under different Attention-to-SSM layouts ranging from to . We evaluate these layouts on a single NVIDIA H100 GPU using three representative datasets: ARC-Challenge for reasoning, PIQA for commonsense QA, and PubMed for long-document summarization. We report task quality together with generation throughput.
As shown in Figure 3, different workloads favor different hybrid layouts. On ARC-Challenge, the hybrid layout achieves the highest accuracy, suggesting that reasoning benefits from sufficient SSM efficiency while retaining selected Attention computation. On PIQA and PubMed, the hybrid layout performs best, reflecting its stronger demand for fine-grained long-context modeling. Meanwhile, increasing the proportion of SSM layers significantly improves throughput, from 23.83 tokens/s under to 87.11 tokens/s under $1:12. These results show that the Attention-to-SSM layout directly affects both task quality and serving efficiency, but no single static layout is consistently optimal across diverse workloads.

Figure 3. Quality and throughput of static hybrid LLMs with different Attention-to-SSM layouts.
Observation 2: Even within the same workload, different requests prefer different Attention-to-SSM layouts.
Even within the same task type, different requests can still prefer different hybrid structures. To show this, we further study ARC-Challenge by partitioning the evaluation set into six disjoint subsets, each containing 100 requests, and evaluate the same four static Attention-to-SSM layouts on
each subset. As shown in Figure 4, the best layout changes across subsets. For example, Set 1 and Set 3 favor more Attention-heavy settings, while Set 4 and Set 6 achieve the best results under the hybrid layout. Set 5 instead performs best with the layout. Even within a single reasoning benchmark, there is no universally optimal static hybrid structure. This result suggests that the preferred Attention-to-SSM layout depends not only on the task category, but also on the specific requests within the same dataset. Therefore, a static hybrid layout is fundamentally limited, motivating a dynamic hybrid architecture that can adapt its structure to each request.

Figure 4. Accuracy of different static Attention-to-SSM layouts on six disjoint ARC-Challenge subsets.
System Overview
Figure 5 presents an overview of Flex, an inference system that combines a request-adaptive hybrid model with an optimized serving runtime through algorithm-system co-design.
Figure 5. System overview of Flex. For clarity, standard components in the model, e.g., normalization, are omitted.
Request-adaptive hybrid model. The right part of Figure 5 shows the proposed hybrid model, which consists of a stack of adaptive hybrid blocks. Each block contains a lightweight gate, paired Attention and SSM branches, and shared components such as FFNs. For each request, the gates select one branch per block, forming a request-specific hybrid layout that balances modeling capability and inference cost.
Serving runtime. The left part of Figure 5 presents the runtime, which addresses two challenges of dynamic hybrid execution: requests within a batch may select different branches, and requests sharing a textual prefix may require incompatible caches. The runtime incorporates two modules. The selective batching engine executes shared operations on the original packed batch and splits requests into Attention and SSM sub-batches for branch-specific computation. Hybrid-aware context caching matches both textual prefixes and layer-selection histories to identify reusable Attention KV caches and SSM states.
Workflow. Given incoming requests, the runtime forms serving batches and identifies candidate cached prefixes. During prefill, the gate at each hybrid block selects one branch per request, and the runtime executes the selected branches while reusing caches compatible with the layer-selection history up to that block. The runtime records these choices and keeps them fixed throughout decoding, grouping requests accordingly and updating their Attention KV caches or SSM states until generation completes.
Prefill-Oriented Gating
This section presents the prefill-oriented gating mechanism of Flex. The goal is to enable request-adaptive Attention and SSM layer selection while preserving efficient autoregressive decoding.
Gating Design
A natural way to enable dynamic hybrid execution is to make layer choices for each token. However, such token-level gating is ill-suited for autoregressive hybrid LLM serving. During decoding, each newly generated token relies on the Attention KV caches or SSM state caches produced by previous tokens to avoid redundant computation. If different tokens from the same request choose different layers within the same block, later tokens may no longer be able to directly reuse the caches produced by earlier tokens. As a result, the runtime must recompute previous tokens under the new layer choice, introducing substantial overhead and complicating decoding.
To avoid these problems, Flex adopts a coarse-grained prefill-oriented gating mechanism in the request-adaptive hybrid model. As shown in Figure 6, this mechanism has two key properties. First, the gate is evaluated only during prefill, where each hybrid block makes one request-level choice between the Attention layer and the SSM layer based on all input tokens. Second, the selected layer choices are fixed after prefill and directly reused throughout decoding, so generated tokens do not trigger additional gating decisions.
Figure 6. Illustration of the prefill-oriented gating mechanism. The activated layers are shown with filled colors, while inactive layers are left unfilled.
This design is reasonable for both system efficiency and hybrid adaptation. From the system perspective, prefill-oriented gating preserves a consistent computation throughout model inference and newly generated tokens can directly reuse the Attention KV caches or SSM state caches produced by previous tokens, avoiding the recomputation overhead. From the model perspective, the input tokens processed during prefill already encode the key characteristics of the request, such as task type, context length, and dependency requirements [30, 52]. Our experimental results further show that such prefill information is sufficient to guide the Attention and SSM layer choices for the whole request, enabling the hybrid model to adapt its computation without repeatedly invoking the gate during decoding.
In each hybrid block, the gate first aggregates the prefill hidden states into a compact request representation, and then feeds it into a lightweight gate network to obtain the probabilities of selecting the Attention and SSM layers. We use top-1 gating and activate the layer with the larger probability. The formal definition of the gating process is provided in the supplementary manuscript.
Hybrid Model Training
The hybrid model in Flex is initialized from pretrained Attention and SSM components [11, 23]. We use structured pruning to align their hidden dimensions [26]. When their depths differ, we select source layers using KL-divergence-based importance scores and pair the selected Attention and SSM layers to form hybrid blocks. We jointly fine-tune the gates and paired branches while freezing the shared embedding, FFN, and normalization modules initialized from the Attention backbone.
To ensure fine-tuning quality, we use a soft routing mode to keep optimization differentiable. For the -th hybrid block, let denote the hidden states from the previous block, and let and denote the candidate Attention and SSM layers in this block, respectively. The gate outputs the probabilities of selecting the Attention and SSM layers. Instead of activating only one layer, the block computes a weighted combination of the two branch outputs:
where and are the gate probabilities for the Attention and SSM layers at the -th hybrid block, and is the output before the following feed-forward network. This soft routing formulation allows gradients to update the gate as well as both candidate layers. During inference, the model switches to hard routing, where each hybrid block activates only the layer with the larger gate probability.
The training objective balances generation quality and inference efficiency. For generation quality, we use the standard supervised fine-tuning loss. To encourage efficient layer selection, we add a lightweight cost-aware regularization term:
where denotes the standard supervised fine-tuning loss for next-token prediction, is the number of hybrid blocks, and and are normalized layer-level cost weights for selecting the Attention and SSM layers, respectively. The co-efficient controls the trade-off between generation quality and inference efficiency. This regularization is used only to bias the gate toward efficient layer choices during training.
Selective Batching Engine
This section introduces the design of selective batching engine in Flex, which supports efficient batching processing of the proposed request-adaptive hybrid model.
Split-and-Merge Execution
Given the request-adaptive layer choices produced by dynamic hybrid execution, the key question is how to execute a mixed batch without sacrificing normal batching efficiency. Flex addresses this through a split-and-merge execution mechanism. As shown in Figure 7, the input tokens from different requests are first packed into a single batch tensor. For example, denotes the -th token of request , and the two requests in the figure form a packed tensor with shape . Flex keeps this packed layout for shared operations, such as gate network and FFN computation, and splits the batch only before the layer-specific Attention or SSM computation. Requests selecting the Attention layer are compacted into an Attention sub-batch, while requests selecting the SSM layer are compacted into an SSM sub-batch. The two sub-batches are then executed by their corresponding kernels. Afterward, Flex merges their outputs back into the original packed batch layout, so subsequent shared operations can still be processed as a regular batch.
Figure 7. Illustration of the proposed split-and-merge mechanism in selective batching engine. We ignore Layer Norm, Add, and other operations for simplicity.
This design confines dynamic layer selection to the Attention and SSM computation inside each hybrid block. Other operations still use the standard packed tensor interface and therefore preserve normal batching efficiency. During prefill, the gate of each hybrid block is executed once to determine whether each request uses the Attention or SSM layer, and the selected layer processes all input tokens of that request. Meanwhile, the corresponding Attention KV or SSM state is cached. After prefill, each request records its selected layer choices across all hybrid blocks. During decoding, the gate is disabled, and Flex directly reuses the recorded layer choices to form Attention and SSM sub-batches at each block. The Attention layer reads and appends the Attention KV cache, while the SSM layer reads and updates the SSM cache.
Load-Aware Branch Coordination
Although selective batching enables the Attention and SSM layers to run independently, simply launching them on normal CUDA streams is insufficient. The two branches have different execution characteristics: the Attention branch is sensitive to context length and KV access, while the SSM branch is mainly dominated by recurrent state updates. Moreover, the number of tokens assigned to each branch changes across blocks and batches. As a result, one branch may finish earlier than the other, forcing the merge operation to wait for the slower branch.
To reduce this synchronization stall, Flex adopts load-aware branch coordination. The key idea is to estimate the relative workload of the Attention and SSM branches and assign more SM resources to the heavier branch. Guided by the computation complexity in Table 1, Flex uses lightweight profiled cost models to predict branch execution costs from simple runtime features, including the number of requests in each branch, input lengths during prefill, and current KV-cache lengths during decoding. Since branch coordination only requires relative costs, the predictor does not need exact latency estimation. In our implementation, its prediction error is within 6.32% for the Attention branch and 5.18% for the SSM branch. The profiling is a one-time effort for each model-GPU pair and can be completed efficiently. Detailed cost models are provided in the supplementary manuscript.
| Prefill | Decoding | |
| Attention Layer | ||
| SSM Layer |
Table 1. Computation complexities of Attention and SSM layers. Here, denotes the input length in prefill, denotes the current cached context length during decoding, denotes hidden dimension, and denotes SSM state dimension.
Based on the predicted costs, Flex partitions the schedulable GPU resources between the Attention and SSM sub-batches at the SM granularity. In our implementation, we use NVIDIA H100 SXM GPUs and empirically set the allocation granularity to SMs, following recent intra-GPU multiplexing practice [4, 13, 55]. Of the 132 SMs available on each GPU, 128 are included in the partitioned resource pool, yielding a compact set of feasible allocations in multiples of 16 SMs. Flex rounds the target allocation to the nearest feasible partition and assigns the remaining schedulable SMs to the other sub-batch. If only one sub-batch is nonempty, it skips partitioning and assigns the entire resource pool to that sub-batch. This compact allocation space enables lightweight online selection while providing sufficient flexibility to balance the two computations. The detailed SM allocation rule is provided in the supplementary manuscript.
The selected SM partition is implemented using NVIDIA GreenContext [8]. Flex creates separate Green Contexts and CUDA streams for the Attention and SSM sub-batches, binding each stream to its assigned SM partition. Since GreenContext only constrains the SM scheduling pool, existing FlashAttention and mamba_ssm kernels can run without modification. The two sub-batches are launched concurrently and synchronized with CUDA events before the merge operation and the next hybrid block. This implementation assigns more compute resources to the heavier sub-batch and reduces the completion-time imbalance between them.
Hybrid-Aware Context Caching
This section presents the hybrid-aware context caching mechanism in Flex, which enables correct prefix reuse when the same textual context may produce different layer-wise Attention KV and SSM state caches.
Context Caching Mechanism
Context caching is widely used to improve LLM serving efficiency by reusing previously computed caches (e.g., Attention KVs and SSM states) for repeated prefixes [19, 41, 45]. For example, RadixAttention [51] organizes historical contexts with a radix tree, where each node represents a textual prefix. When a new request arrives, the runtime searches the radix tree to find the longest matched prefix, reuses the caches associated with the matched prefix, and computes only the unmatched suffix. However, conventional RadixAttention cannot be directly applied to Flex. Its key assumption is that a matched textual prefix uniquely determines all layer-wise caches that can be reused. This assumption holds for static models because the layer type at each position is fixed by the model architecture. For example, in an Attention-only model, each layer always produces a KV cache, while in an SSM-only model, each layer always produces an SSM cache.
In Flex, requests sharing a textual prefix may select different layer layouts depending on their full contexts. As shown in Figure 8, the incoming request matches the cached prefix “You are a helpful assistant.”, but requires Attention KV caches at the third and fourth blocks, where the cached entries are SSM states. Conventional methods treat the matched entry as a whole: it is either reused or rejected. Reusing the cached states despite the layer output mismatch can introduce inconsistent intermediate representations and degrade inference quality. Alternatively, rejecting the incompatible entry results in a cache miss and requires recomputing the shared prefix, losing the opportunity to exploit the textual overlap.
Figure 8. Motivation for hybrid-aware context caching. (a) A conventional radix tree indexes Attention KV caches and SSM states by textual prefixes. (b) The incoming request matches context 1 (CTX1) but selects different layer types, resulting in a cache miss.
To support correct and efficient context reuse under dynamic hybrid execution, Flex implements hybrid-aware context caching using a two-level radix-tree structure. The outer-context radix tree indexes textual prefixes, while an inner-cache radix tree attached to each context node organizes layer-selection paths and their associated Attention KV caches and SSM states. Cache lookup proceeds in two steps. First, Flex locates the longest matching textual prefix in the outer-context radix tree. It then examines the inner-cache radix tree of the matched node and its ancestors to identify compatible cache paths. Compatibility requires matching the sequence of layer choices from the first block through the block being reused. Among these candidates, Flex selects the prefix and cache path with the lowest estimated recomputation cost, considering both prefix length and the number of reusable leading blocks.
As illustrated in Figure 9, the inner-cache radix tree associated with CTX1 contains multiple execution paths. These paths share caches for the first two blocks, then branch according to the branch choices at the third and fourth blocks. For cache, Flex retains input hidden states and terminal SSM states, reconstructing Attention KV caches on demand. For an incoming request matching CTX1, Flex follows the inner-tree path consistent with its branch-selection history and reuses the corresponding caches. At the first route mismatch, the cached hidden states remain valid because all preceding branch choices match. Flex uses these states to recompute the prefix at that block and all subsequent blocks, then extends the inner-cache radix tree with the newly computed caches while retaining existing branches. This organization supports partial cache reuse and preserves alternative execution paths for future requests sharing same textual prefix.
Figure 9. Illustration of the proposed two-level radix-tree structure. The outer-context radix tree indexes contextual prefixes, while each inner-cache radix tree organizes alternative layer-selection paths and their associated Attention KV and SSM state caches, enabling cache reuse along compatible execution paths for a matched prefix.
Priority-based Eviction Policy
Hybrid-aware context caching enables correct cache reuse under dynamic hybrid execution, but also increases the number of cache entries. With a fixed layer layout, conventional RadixAttention associates each textual prefix with one set of layer-wise caches. In Flex, the same context node can maintain multiple cache paths corresponding to different Attention and SSM choices. Evicting an entire context node therefore discards all these paths, potentially removing frequently reused caches together with rarely accessed ones.
Flex instead performs eviction at the granularity of individual layer-wise cache entries. Under GPU memory pressure, it evicts entries from the inner-cache radix trees. An outer-context node is removed only when all its associated caches have been evicted and the node is no longer needed to represent shared textual prefixes. To determine which entries to retain, Flex assigns each cache a retention priority:
where is its recent access frequency, is the recomputation cost saved by reusing it, is its memory footprint, and is a depth factor that favors caches at earlier blocks. Flex evicts entries with the lowest priority first. This policy favors caches that provide greater expected computation savings per unit of memory. The depth factor further prioritizes early-layer caches, which can form shared ancestors of multiple execution paths in an inner-cache radix tree.
Implementation
We implement a prototype of Flex with approximately 6.5K lines of code using Python, PyTorch, HuggingFace Transformers, and CUDA/C++. The construction and fine-tuning of the request-adaptive hybrid model are implemented in PyTorch, while the serving runtime extends a vLLM-style execution loop with selective batching and hybrid-aware context caching. It caches layer hidden states and terminal SSM states, supporting Attention KV reconstruction and recomputation from the first mismatched layer. For concurrent Attention and SSM execution, Flex uses NVIDIA Green-Context to partition SM resources and launches the two sub-batches on separate CUDA streams. It also adopts a unified paging-based memory manager for heterogeneous Attention KV caches and SSM caches, following Jenga [47]. The runtime currently uses a conventional continuous batching scheduler [44], and is compatible with other scheduling policies such as chunked prefill, prefill-decode disaggregation, and latency-aware scheduling [1, 12, 27, 30, 53]. Detailed implementation is provided in the supplementary manuscript.
Evaluation
Evaluation Setup
Platform. We evaluate Flex on a server equipped with one NVIDIA H100 SXM GPU (80GB memory) and Intel Xeon Platinum 8458P CPUs. The server runs Ubuntu 22.04.5 LTS with NVIDIA driver 580.82.07 and CUDA runtime 12.4. We use Python 3.12.3, PyTorch 2.4.1, Transformers 4.46.0 and mamba_ssm 2.2.4.
Models and baselines. We evaluate the proposed hybrid model against both homogeneous and static hybrid LLMs. The homogeneous baselines include Llama-3 [15] as an Attention-only model and Mamba-2 [9] as an SSM-only model. The static hybrid baselines include Zamba [14], Jamba [22], and Nemotron-H [3], whose Attention-to-SSM layouts are fixed after model construction. For serving-system evaluation, we compare Flex with vLLM [21] and SGLang [51], two widely used LLM serving systems, as well as Jenga [47], which further provides memory management support for Attention KV and SSM states. All systems use the same model, workload, and generation settings, with unrelated optimizations such as speculative decoding disabled.
Datasets and workloads. We fine-tune each model on a 10K-example subset of ShareGPT-90K and reuse the resulting checkpoint across all evaluation tasks without task-specific fine-tuning. We evaluate model quality on six datasets covering reasoning, question answering (QA), and summarization. For reasoning, we use ARC-Easy and ARC-Challenge [6]. For QA, we use PIQA [2] and TruthfulQA [25]. For summarization, we use CNN/DailyMail [35] and PubMed [7].
Metrics. For model quality, we report accuracy for ARC-Easy, ARC-Challenge, and PIQA, MC1 accuracy for TruthfulQA, and ROUGE-L for CNN/DM and PubMed. For model efficiency, we estimate decoding TFLOPs per token:
where counts executed linear operations, including projections, FFNs, and the output head. and are the Attention and SSM layers activated by request . is the attended KV length at decoding step , and and are the query-head count and head dimension. For Flex, only the selected branch at each block is counted, together with the shared operations. counts convolution, state update, and readout operations beyond the linear projections [9, 16]. We count each multiply-add as two FLOPs [5, 17], omit normalization, activations, and softmax, and average over evaluated decoding steps. For serving performance, we report request throughput, time-to-first-token (TTFT), time-per-output-token (TPOT), and end-to-end latency.
Other configurations. Fine-tuning uses Adam with a learning rate of , an effective batch size of 64, and a maximum sequence length of 4,096 tokens. We train for 1,000 optimization steps using BF16 precision and fully sharded data parallelism (FSDP), with across all model scales and benchmarks.
Overall Model Performance
Quality-efficiency trade-off. Table 2 compares Flex with homogeneous and static hybrid LLMs across three model scales. Overall, Flex achieves the best results on 13 out of 18 quality metrics while maintaining competitive computation cost. At the small scale, Flex obtains the best results on ARC-Easy, TruthfulQA, and PubMed, and remains close to the best baseline on ARC-Challenge and CNN/DailyMail. At the medium scale, it consistently outperforms all baselines on all six datasets. At the large scale, it achieves the best results on reasoning and QA tasks, while remaining competitive on summarization tasks. More importantly, Flex achieves these quality gains with substantially lower estimated TFLOPs/token: compared with Attention-only Llama models, it reduces computation cost by 40.5%, 33.2%, and 67.2% at the three evaluated scales, respectively. Compared with Nemotron-H, it reduces TFLOPs/token by 28.4%, 28.9%, and 58.7%. Although Mamba and some static hybrid models can be cheaper in certain settings, their quality is often less stable across tasks. These results show that request-adaptive Attention and SSM selection provides a better quality-efficiency trade-off than static hybrid layouts. We further analyze evaluation results across different tasks.
| Models | ARC-E Accuracy | ARC-C Accuracy | PIQA Accuracy | TruthfulQA Accuracy (MC1) | CNN/DM ROUGE-L | PubMed ROUGE-L | Model Efficiency TFLOPs/Token | |||||
| Mamba | 5.3B | 5.3B | – | 64 | 2560 | 67.4 ± 0.11 | 35.8 ± 0.13 | 74.9 ± 0.14 | 33.3 ± 0.07 | 10.9 ± 0.14 | 14.3 ± 0.15 | |
| Llama | 3.2B | 3.2B | 28 | – | 3072 | 73.2 ± 0.08 | 42.4 ± 0.12 | 75.5 ± 0.15 | 42.1 ± 0.33 | 17.1 ± 0.27 | 15.9 ± 0.18 | |
| Zamba | 2.7B | 2.7B | 9 | 45 | 2560 | 69.4 ± 0.24 | 41.9 ± 0.25 | 78.7 ± 0.31 | 41.8 ± 0.17 | 20.5 ± 0.22 | 16.0 ± 0.19 | |
| Jamba | 3.0B | 3.0B | 2 | 26 | 2560 | 69.7 ± 0.15 | 47.3 ± 0.26 | 70.5 ± 0.32 | 48.8 ± 0.16 | 18.9 ± 0.33 | 18.1 ± 0.19 | |
| Nemotron-H | 4.5B | 4.5B | 4 | 24 | 3072 | 74.0 ± 0.24 | 44.6 ± 0.17 | 71.1 ± 0.41 | 49.9 ± 0.33 | 16.8 ± 0.19 | 16.5 ± 0.20 | |
| Ours | 4.4B | 3.2B | 28 | 28 | 2560 | 74.7 ± 0.17 | 45.5 ± 0.26 | 76.6 ± 0.31 | 51.4 ± 0.19 | 20.1 ± 0.25 | 19.4 ± 0.24 | |
| Mamba | 8.2B | 8.2B | 50 | 56 | 4096 | 72.5 ± 0.23 | 50.9 ± 0.19 | 76.6 ± 0.33 | 38.8 ± 0.32 | 16.7 ± 0.24 | 15.3 ± 0.25 | |
| Llama | 13.1B | 13.1B | 55 | – | 4096 | 76.7 ± 0.17 | 54.3 ± 0.17 | 79.5 ± 0.19 | 59.8 ± 0.24 | 19.5 ± 0.10 | 16.4 ± 0.22 | |
| Zamba | 7.4B | 7.4B | 13 | 68 | 3584 | 70.3 ± 0.30 | 45.4 ± 0.17 | 81.1 ± 0.22 | 65.6 ± 0.24 | 20.1 ± 0.32 | 16.2 ± 0.15 | |
| Jamba | 13.0B | 13.0B | 1 | 7 | 4096 | 72.8 ± 0.24 | 52.5 ± 0.22 | 80.3 ± 0.38 | 66.2 ± 0.41 | 19.4 ± 0.15 | 18.3 ± 0.18 | |
| Nemotron-H | 12.3B | 12.3B | 6 | 28 | 5120 | 77.3 ± 0.18 | 56.1 ± 0.17 | 82.2 ± 0.26 | 68.7 ± 0.41 | 18.3 ± 0.39 | 16.8 ± 0.25 | |
| Ours | 14.5B | 6.5B | 32 | 32 | 4096 | 77.8 ± 0.17 | 57.7 ± 0.36 | 83.1 ± 0.28 | 68.9 ± 0.39 | 23.4 ± 0.14 | 21.5 ± 0.13 | |
| Mamba | – | – | – | – | – | – | – | – | – | – | – | – |
| Llama | 70B | 70B | 80 | – | 8192 | 84.3 ± 0.16 | 82.7 ± 0.24 | 84.3 ± 0.33 | 63.7 ± 0.18 | 25.7 ± 0.09 | 23.6 ± 0.11 | |
| Zamba | – | – | – | – | – | – | – | – | – | – | – | – |
| Jamba | 52B | 12B | 4 | 28 | 4096 | 73.7 ± 0.22 | 63.6 ± 0.19 | 83.1 ± 0.31 | 72.1 ± 0.17 | 20.5 ± 0.16 | 19.8 ± 0.11 | |
| Nemotron-H | 56B | 56B | 10 | 54 | 8192 | 85.2 ± 0.23 | 84.8 ± 0.17 | 84.4 ± 0.12 | 73.4 ± 0.26 | 20.1 ± 0.19 | 17.9 ± 0.28 | |
| Ours | 61B | 56B | 32 | 32 | 8192 | 85.6 ± 0.21 | 85.1 ± 0.17 | 85.3 ± 0.25 | 75.2 ± 0.31 | 24.5 ± 0.27 | 22.3 ± 0.21 |
Table 2. Table 2. Comparison of model quality and efficiency across homogeneous, static hybrid, and our proposed request-adaptive hybrid model. and denote the total model parameters and the activated parameters for one request, respectively. and denote the numbers of Attention and SSM layers. For Flex, counts all paired candidate branches, while reports the average activated parameters under request-adaptive layer selection. All multiple-choice tasks use free-form generation with regex-based letter extraction. Chat-template-aware models receive a “letter only” system prompt and reasoning models use no_think mode.
Reasoning tasks. On ARC-Easy and ARC-Challenge, Flex performs particularly well at medium and large scales. For example, at the medium scale, Flex achieves 77.8% accuracy on ARC-Easy and 57.7% on ARC-Challenge, outperforming all static hybrid baselines. At the large scale, it further improves the results to 85.6% and 85.1%, respectively. These results suggest that reasoning tasks benefit from selectively activating Attention in important layers, while still using efficient SSM computation elsewhere. Static hybrid models improve over pure SSM models, but their static attention placement cannot adapt to the varying dependency patterns of different requests.
QA tasks. On PIQA and TruthfulQA, Flex also shows strong performance. At the medium scale, it achieves 83.1% on PIQA and 68.9% on TruthfulQA, outperforming all baselines. At the large scale, it reaches 85.3% on PIQA and 75.2% on TruthfulQA. These results indicate that QA workloads do not always require Attention in every layer, but still need Attention in selected layers to preserve factual and contextual signals. Flex captures this behavior by adapting the Attention and SSM choices to each request instead of relying on a static hybrid layout.
Summarization tasks. Summarization workloads have a stronger dependence on global context modeling. Attention-heavy models such as Llama remain competitive, especially at the large scale. Nevertheless, Flex achieves the best summarization results at the medium scale, reaching 23.4 ROUGE-L on CNN/DailyMail and 21.5 ROUGE-L on PubMed. At the small scale, it also achieves the best PubMed score and remains close to the best CNN/DailyMail result. These results show that summarization benefits from more frequent Attention activation, but does not require all layers to be Attention layers. Flex can allocate Attention to the layers and requests that need global context, while using SSM layers elsewhere to reduce computation.
Routing Effectiveness. We compare three routing strategies using the same fine-tuned checkpoint at each model scale. We collect candidate layouts from adaptive routing on held-out validation data and evaluate each as a fixed layout under matched inference-cost budgets. Within this candidate set, Global Optimal (G-Optimal) selects one layout that maximizes the macro-average of normalized validation scores across datasets, while Dataset Optimal (D-Optimal) selects the best layout separately for each dataset. Both strategies keep their selected layouts fixed during evaluation, whereas Adaptive selects a layout for each request. As shown in Table 3, our adaptive method outperforms both static strategies across all 18 model-dataset combinations. Compared with D-Optimal, it achieves average gains of 4.86 percentage points in accuracy and 2.55 points in ROUGE-L. Although D-Optimal consistently improves over G-Optimal, its remaining gap to Adaptive demonstrates the benefit of adapting layer choices to individual requests within the same dataset.
| Scale | Method | ARC-E | ARC-C | PIQA | TruthfulQA | CNN/DM | PubMed |
| 4.4B | G-Optimal | 68.2 | 37.2 | 70.5 | 45.4 | 16.6 | 15.2 |
| D-Optimal | 71.3 | 40.4 | 73.6 | 47.3 | 18.3 | 16.5 | |
| Adaptive | 74.5 | 45.6 | 76.4 | 51.3 | 20.2 | 19.6 | |
| 14.5B | G-Optimal | 71.8 | 49.5 | 73.7 | 60.3 | 17.6 | 18.8 |
| D-Optimal | 73.4 | 52.6 | 77.2 | 63.5 | 19.2 | 20.6 | |
| Adaptive | 77.9 | 57.7 | 83.2 | 68.8 | 23.4 | 21.6 | |
| 61B | G-Optimal | 74.3 | 72.1 | 75.4 | 69.7 | 20.3 | 19.6 |
| D-Optimal | 79.6 | 76.4 | 80.3 | 72.6 | 21.5 | 20.1 | |
| Adaptive | 85.4 | 85.0 | 85.3 | 75.4 | 24.6 | 22.1 |
Table 3. Comparison of routing methods across model scales.
Overall System Performance
We evaluate the system-level performance of Flex at three model scales on a single H100 GPU, using FP8 quantization for the 61B model. We compare its serving runtime with vLLM, SGLang, and Jenga, with all methods serving the same model checkpoint at each scale. We also include two ablated variants: Flex w/o Bat. disables selective batching (§5), while Flex w/o Con. disables hybrid-aware context reuse (§6). We construct mixed workloads from all datasets
in which requests share article prefixes and append different questions. We vary the number of requests processed per scheduling iteration, using batch sizes of 8, 16, 32, and 64 for the 4.4B model and 4, 8, 16, and 32 for the other two scales. We report request throughput, TTFT, and TPOT.
Figure 10 shows that Flex consistently outperforms all three baselines in throughput, TTFT, and TPOT across the evaluated batch sizes and model scales. It achieves up to 3.79×, 3.87×, and 3.44× the throughput of vLLM for the 4.4B, 14.5B, and 61B models, respectively. Its throughput advantage over vLLM grows with batch size. Larger batches offer more opportunities to group requests taking the same branch, which Flex exploits through selective batching. Both ablated variants also achieve lower throughput than the full system, showing that selective batching and context reuse contribute to serving efficiency.
Figure 10. Figure 10. End-to-end serving performance of Flex under varying batch sizes on a single H100 GPU. Rows correspond to the 4.4B, 14.5B, and 61B models, respectively. Columns report request throughput, TTFT, and TPOT. The 61B model uses FP8 quantization to avoid out-of-memory.
Hybrid-aware context reuse substantially reduces TTFT on the mixed workload by avoiding repeated computation over shared article prefixes. Under dynamic layer choices, matching prefix tokens alone does not establish cache compatibility: the cached states also depend on the execution paths that produced them. Flex uses the two-level radix tree to check both textual prefixes and execution-path compatibility before reusing layer-wise Attention KV caches and SSM states. At the largest tested batch size for each scale, Flex reduces TTFT by an average of 80.9%, 78.9%, and 75.2% compared with vLLM, SGLang, and Jenga, respectively. It also reduces TTFT by 59.1% on average compared with Flex w/o Con., demonstrating the benefit of context reuse.
The TPOT results show the benefit of selective batching during decoding. Each request retains its layer choices after prefill, allowing Flex to group requests taking the same branch into compact Attention and SSM sub-batches. At the same configurations, Flex reduces TPOT by an average of 71.6%, 63.1%, and 56.5% compared with vLLM, SGLang, and Jenga, respectively. Compared with Flex w/o Bat., it reduces TPOT by 43.6% on average, demonstrating that selective batching improves decoding efficiency under request-specific layer choices.
Ablation Study
Effectiveness of selective batching. We further evaluate the selective batching engine by varying the batch size from 2 to 16 on the PubMed dataset. We compare Flex with vLLM and an ablated variant, denoted as Flex w/o Coord., which retains split-and-merge batching but disables load-aware branch coordination for SSM allocation between the Attention and SSM sub-batches. As shown in Figure 11, Flex consistently achieves lower end-to-end (E2E) latency than vLLM across all tested batch sizes and model scales. Its selective batching engine groups requests according to their layer choices at each block, forming compact Attention and SSM sub-batches for execution with the corresponding kernels. Averaged over all batch sizes, Flex reduces E2E latency compared with vLLM by 54.6%, 66.0%, and 59.9% for the 4.4B, 14.5B, and 61B models, respectively. At batch size 16, the corresponding reductions reach 61.9%, 73.2%, and 70.9%, showing a larger latency advantage at the upper end of the tested batch sizes.
Figure 11. Ablation study of the selective batching engine and branch coordination under varying batch sizes. From left to right, the three subfigures report end-to-end latency for the 4.4B, 14.5B, and 61B hybrid models in FLEX, respectively.
Comparing Flex with Flex w/o Coord. isolates the additional benefit of load-aware branch coordination. Although split-and-merge batching groups requests by their selected layers, the Attention and SSM sub-batches can still have imbalanced workloads, and the merge operation must wait for both branches to complete. Flex coordinates SM allocation according to branch workloads to reduce this imbalance. Compared with Flex w/o Coord., the full system reduces E2E latency by 21.7%, 28.2%, and 26.2% on average for the three model scales, respectively. For the 61B model at batch size 16, vLLM requires 33.53s, while Flex w/o Coord. and the full Flex require 13.77s and 9.75s, respectively. Branch coordination thus provides a further 29.2% latency reduction beyond split-and-merge batching alone in this configuration.
Effectiveness of hybrid-aware context caching. We evaluate the proposed hybrid-aware context caching mechanism using a PubMed-based shared-context workload. Each article serves as a shared textual prefix, and multiple requests append different questions after the same article [47]. We vary the number of articles from 2 to 20 under a fixed cache capacity, increasing cache competition while preserving prefix reuse opportunities within each article. For conventional context caching [51], a context is counted as reusable only when all layer-wise caches associated with the matched prefix can be reused. If any Attention KV cache or SSM cache is incompatible with the current request’s layer choices, the whole context is treated as a miss. In contrast, the proposed hybrid-aware context caching indexes layer-wise caches under each matched context, so it can reuse compatible caches even when later layers diverge. We compare the two mechanisms using effective cache hit rate and TTFT.
As shown in Figure 12, the hit rates of both methods decrease as the number of articles increases, because more shared contexts compete for the fixed memory capacity. Nevertheless, hybrid-aware context caching consistently maintains a higher effective hit rate. Across all settings, it achieves an average hit rate of 43.4%, compared with 18.7% for the baseline, improving the hit rate by 24.7 percentage points. When the number of articles reaches 20, conventional caching method retains only a 7% hit rate, whereas hybrid-aware context caching still achieves 24%. This confirms that layer-wise cache indexing captures valid reuse opportunities that conventional prefix-level caching would discard once any layer becomes incompatible.
Figure 12. Effectiveness of hybrid-aware context caching under varying numbers of shared PubMed contexts.
The higher hit rate directly reduces prefill latency. As the number of articles increases, TTFT grows for both methods due to more frequent cache misses and recomputation. However, TTFT grows more slowly with hybrid-aware context caching because it reuses compatible Attention KV caches and SSM state caches instead of recomputing the entire shared context. On average, hybrid-aware context caching reduces TTFT from 10.68s to 5.92s, corresponding to a 44.6% reduction. At 20 articles, it reduces TTFT from 16.18s to 9.36s, a reduction of 42.2%. These results demonstrate that hybrid-aware context caching improves prefill efficiency by enabling fine-grained and correct context reuse under dynamic hybrid execution.
Layer-wise selection. Figure 13 characterizes routing in the 28-block hybrid model in FLEX across three datasets. The heatmaps show the fractions of requests selecting Attention
Figure 13. Routing patterns of the 28-block hybrid model across three datasets. Left: per-block Attention and SSM selection ratios. Right: cumulative request coverage by the most frequent hybrid layouts within each dataset.
and SSM at each block. On ARC-Challenge, Attention selection gradually increases toward deeper blocks. On TruthfulQA, it peaks in the early-to-middle blocks and decreases thereafter. On CNN/DailyMail, Attention selection is highest in the middle blocks and lower toward both ends. These patterns show that the allocation of Attention and SSM computation varies across both block depth and workloads.
The right panel illustrates cumulative request coverage by the top- full-model layouts, ranked by frequency within each dataset. This view complements the per-block statistics by characterizing complete routing decisions across requests. For ARC-Challenge, 500 requests activate 35 distinct layouts, with the six most frequent covering over 90% of requests. Request-level routing therefore combines a small set of frequently selected layouts with a long tail of less common choices.
Related Work
Hybrid LLMs. Recent works reduce the cost of long-context LLM inference by replacing part of the attention computation with efficient sequence modeling modules [10, 28, 33]. SSM-based models, such as Mamba [16] and Mamba-2 [9] maintain compact recurrent states instead of full Attention KV caches. Hybrid LLMs further combine Attention and SSM layers to balance global-context modeling and inference efficiency, with representative examples including Jamba [22], Zamba [14], Hymba [11], Mamba-in-Llama [38], and Nemotron-H [3]. While prior hybrid LLMs demonstrate the promise of Attention and SSM hybridization, they rely on static layouts, whereas Flex enables request-adaptive Attention and SSM layer selection within the same model.
Efficient batching. Efficient batching is central to high-throughput LLM serving. Orca [44] introduces iteration-level scheduling, while vLLM [21] improves batching capacity with PagedAttention. SGLang [51] supports structured LLM program execution, and systems such as Sarathi-Serve [1], Split-wise [27], DistServe [53], LoongServe [39], NanoFlow [55], and MuxWise [4] optimize prefill/decode scheduling or GPU resource coordination. Unlike these systems, Flex supports batches with divergent Attention and SSM layer choices and preserves efficiency through selective hybrid batching.
Context caching. Context caching reduces redundant prefill computation by reusing caches for repeated prefixes [19, 41, 42, 45, 48]. SGLang [51] proposes RadixAttention for prefix-based cache reuse, and works such as Parrot [24], CacheBlend [41], and Mooncake [29] exploit reuse opportunities in structured, RAG, multi-turn, or disaggregated serving scenarios. Unlike prior techniques that bind each matched prefix to a single reusable cache, Flex uses the two-level radix tree to distinguish different layer-wise Attention KV and SSM state caches for the same textual prefix.
Conclusion
This paper presents Flex, an inference system that combines a request-adaptive hybrid model with an optimized serving runtime through algorithm-system co-design. The model uses prefill-oriented gating to select Attention or SSM branches for each request and keeps the resulting layout fixed throughout decoding. The runtime integrates selective batching and hybrid-aware context caching to support efficient mixed-branch execution and correct prefix reuse. Experiments show that the hybrid model achieves a better quality-efficiency trade-off than static hybrid baselines, while the runtime improves serving throughput and reduces inference latency.
References
References
- [1]Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve. In 18th USENIX symposium on operating systems design and implementation (OSDI 24). 117–134.arxiv.org/abs/2403.02310
- [2]Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34. 7432–7439.arxiv.org/abs/1911.11641
- [3]Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich, Aleksander Ficek, Alexis Bjornlin, Ali Taghibakhshi, Amala Sanjay Deshmukh, Ameya Sunil Mahabaleshwarkar, et al. 2025. Nemotron-h: A family of accurate and efficient hybrid mamba-transformer models. arXiv preprint arXiv:2504.03624 (2025).arxiv.org/abs/2504.03624
- [4]Yukang Chen, Weihao Cui, Han Zhao, Ziyi Xu, Xiaoze Fan, Xusheng Chen, Yangjie Zhou, Shixuan Sun, Bingsheng He, and Quan Chen. 2026. Towards High-Goodput LLM Serving with Prefill-decode Multiplexing. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 2030–2047.arxiv.org/abs/2504.14489
- [5]Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of machine learning research 24, 240 (2023), 1–113.arxiv.org/abs/2204.02311
- [6]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 (2018).arxiv.org/abs/1803.05457
- [7]Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). 615–621.arxiv.org/abs/1804.05685
- [8]NVIDIA Corporation. 2025. CUDA Driver API: Green Contexts. https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__GREEN_CONTEXTS.html
- [9]Tri Dao and Albert Gu. 2024. Transformers are SSMs: generalized state space models and efficient algorithms through structured state space duality. In Proceedings of the 41st International Conference on Machine Learning. 10041–10071.
- [10]Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Sriyansh Srinivasan, et al. 2024. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402.19427 (2024).
- [11]Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Mahabalashwarkar, Shih-Yang Liu, Min-Hung Chen, Yoshi Suhara, Yingyan Celine Lin, et al. 2025. Hymba: A hybrid-head architecture for small language models. In International Conference on Learning Representations, Vol. 2025. 97729–97755.arxiv.org/abs/2411.13676
- [12]Kuntai Du, Bowen Wang, Chen Zhang, Yiming Cheng, Qing Lan, Hejian Sang, Yihua Cheng, Jiayi Yao, Xiaoxuan Liu, Yifan Qiao, et al. 2025. Prefillonly: An inference engine for prefill-only workloads in large language model applications. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. 399–414.arxiv.org/abs/2505.07203
- [13]Lei Gao, Chaoyi Jiang, Hossein Entezari Zarch, Daniel Wong, and Murali Annavaram. 2025. DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing. arXiv preprint arXiv:2511.04791 (2025).arxiv.org/abs/2511.04791
- [14]Paolo Glorioso, Quentin Anthony, Yury Tokpanov, Anna Golubeva, Vasudev Shyam, James Whittington, Jonathan Pilault, and Beren Millidge. 2024. The zamba2 suite: Technical report. arXiv preprint arXiv:2411.15242 (2024).arxiv.org/abs/2411.15242
- [15]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024).arxiv.org/abs/2407.21783
- [16]Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 (2023).arxiv.org/abs/2312.00752
- [17]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems. 30016–30030.arxiv.org/abs/2203.15556
- [18]Tomoki Ito and Shun Nakagawa. 2024. Tender document analyzer with the combination of supervised learning and llm-based improver. In Companion Proceedings of the ACM Web Conference 2024. 995–998.DOI
- [19]Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Shufan Liu, Xuanzhe Liu, and Xin Jin. 2025. Ragcache: Efficient knowledge caching for retrieval-augmented generation. ACM Transactions on Computer Systems 44, 1 (2025), 1–27.arxiv.org/abs/2404.12457
- [20]Majeed Kazemitabar, Runlong Ye, Xiaoning Wang, Austin Zachary Henley, Paul Denny, Michelle Craig, and Tovi Grossman. 2024. Codeaid: Evaluating a classroom deployment of an llm-based programming assistant that balances student and educator needs. In Proceedings of the 2024 chi conference on human factors in computing systems. 1–20.
- [21]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles. 611–626.arxiv.org/abs/2309.06180
- [22]Barak Lenz, Opher Lieber, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, et al. 2025. Jamba: Hybrid transformer-mamba language models. In International Conference on Learning Representations, Vol. 2025. 679594–67984.arxiv.org/abs/2403.19887
- [23]Yixing Li, Ruobing Xie, Zhen Yang, Xingwu Sun, Shuaipeng Li, Weidong Han, Zhanhui Kang, Yu Cheng, Chengzhong Xu, Di Wang, et al. 2025. Transmamba: Flexibly switching between transformer and mamba. arXiv preprint arXiv:2503.24067 (2025).
- [24]Chaofan Lin, Zhenhua Han, Chengruong Zhang, Yuqing Yang, Fan Yang, Chen Chen, and Lili Qiu. 2024. Parrot: Efficient serving of LLM-based applications with semantic variable. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 929–945.arxiv.org/abs/2405.19888
- [25]Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers). 3214–3252.arxiv.org/abs/2109.07958
- [26]Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. 2024. Compact language models via pruning and knowledge distillation. Advances in Neural Information Processing Systems 37 (2024), 41076–41102.arxiv.org/abs/2407.14679
- [27]Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 118–132.
- [28]Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, et al. 2023. Rwkv: Reinventing rnns for the transformer era. In Findings of the association for computational linguistics: EMNLP 2023. 14048–14077.arxiv.org/abs/2305.13048
- [29]Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading more storage for less computation—a KVCache-centric architecture for serving LLM chatbot. In 23rd USENIX conference on file and storage technologies (FAST 25). 155–170.
- [30]Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha, Chen Wang, Hubertus Franke, Zbigniew T Kalbarczyk, Tamer Başar, and Ravishankar K Iyer. 2024. Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction. In International Conference on Architectural Support for Programming Languages and Operating Systems.arxiv.org/abs/2404.08509
- [31]Abdur-Rahman Ibrahim Sayyid-Ali, Daanish Uddin Khan, and Naveen Anwar Bhatti. 2026. Are LLM Web Search Engines Sustainable? A Web-Measurement Study of Real-Time Fetching. In Proceedings of the ACM Web Conference 2026. 2083–2093.
- [32]Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. 2024. Lumnix: Dynamic scheduling for large language model serving. In 18th USENIX symposium on operating systems design and implementation (OSDI 24). 173–191.arxiv.org/abs/2406.03243
- [33]Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. 2023. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621 (2023).arxiv.org/abs/2307.08621
- [34]Xin Tan, Yimin Jiang, Yitao Yang, and Hong Xu. 2025. Towards end-to-end optimization of llm-based applications with ayo. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 1302–1316.arxiv.org/abs/2407.00326
- [35]Hugging Face Team. 2024. Hugging Face Dataset cnn dailymail (CNN/DM). https://huggingface.co/datasets/cnn_dailymail
- [36]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).arxiv.org/abs/1706.03762
- [37]Dustin Wang, Rui-Jie Zhu, Steven Abreu, Yong Shan, Taylor Kergan, Yuqi Pan, Yuhong Chou, Zheng Li, Ge Zhang, Wenhao Huang, et al. 2025. A systematic analysis of hybrid linear attention. arXiv preprint arXiv:2507.06457 (2025).arxiv.org/abs/2507.06457
- [38]Junxiong Wang, Daniele Paliotta, Avner May, Alexander M Rush, and Tri Dao. 2024. The mamba in the llama: Distilling and accelerating hybrid models. Advances in Neural Information Processing Systems 37 (2024), 62432–62457.arxiv.org/abs/2408.15237
- [39]Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. Longserve: Efficiently serving long-context large language models with elastic sequence parallelism. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles. 640–654.
- [40]Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. 2024. Search-in-the-chain: Interactively enhancing large language models with search for knowledge-intensive tasks. In Proceedings of the ACM Web Conference 2024. 1362–1373.arxiv.org/abs/2304.14732
- [41]Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. Cacheblend: Fast large language model serving for rag with cached knowledge fusion. In Proceedings of the twentieth European conference on computer systems. 94–109.
- [42]Hanchen He, Zhengqi Gao, Mingyuan Ma, Qinsi Wang, Yuzhe Fu, Ming-Yu Chung, Yueqian Lin, Zhijian Liu, Jianyi Zhang, Danyang Zhuo, et al. 2026. Kvcomm: Online cross-context kv-cache communication for efficient llm-based multi-agent systems. Advances in Neural Information Processing Systems 38 (2026), 17882–17928.
- [43]Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, et al. 2025. Flashinfer: Efficient and customizable attention engine for llm inference serving. Proceedings of Machine Learning and Systems 7 (2025).arxiv.org/abs/2501.01005
- [44]Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for Transformer-Based generative models. In 16th USENIX symposium on operating systems design and implementation (OSDI 22). 521–538.
- [45]Yifan Yu, Gan Nikhil Sarda, Lillian Tsai, Jiaming Shen, Yanqi Zhou, Arvind Krishnamurthy, Fan Lai, Hank Levy, and David Culler. 2025. Ic-cache: Efficient large language model serving via in-context caching. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. 375–398.
- [46]Zhihao Zhan, Jianan Zhao, Zhaocheng Zhu, and Jian Tang. 2026. Overcoming Long Context Limitations of State Space Models via Context Dependent Sparse Attention. Advances in Neural Information Processing Systems 38 (2026), 43752–43779.arxiv.org/abs/2507.00449
- [47]Chen Zhang, Kuntai Du, Shu Liu, Woosuk Kwon, Xiangxi Mo, Yufeng Wang, Xiaoxuan Liu, Kaichao You, Zhuohan Li, Mingsheng Long, et al. 2025. JENGA: Effective memory management for serving LLM with heterogeneity. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. 446–461.arxiv.org/abs/2503.18292
- [48]Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, and Bin Cui. 2025. Pqcache: Product quantization-based kvcache for long context llm inference. Proceedings of the ACM on Management of Data 3, 3 (2025), 1–30.arxiv.org/abs/2407.12820
- [49]Yichi Zhang. 2024. Detecting code comment inconsistencies using llm and program analysis. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. 683–685.DOI
- [50]Zhenyu Zhang, Shiwei Liu, Runjin Chen, Bhavya Kailkhura, Beidi Chen, and Zhangyang Wang. 2024. Q-hitter: A better token oracle for efficient llm inference via sparse-quantized kv cache. Proceedings of Machine Learning and Systems 6 (2024), 381–394.
- [51]Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody H Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2024. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems 37 (2024), 62557–62583.arxiv.org/abs/2312.07104
- [52]Zangwei Zheng, Xiaozhe Ren, Fuzhao Xue, Yang Luo, Xin Jiang, and Yang You. 2023. Response length perception and sequence scheduling: An llm-empowered llm inference pipeline. Advances in Neural Information Processing Systems 36 (2023), 65517–65530.arxiv.org/abs/2305.13144
- [53]Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Zhu, Yibo Zhu, Xuan zhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating prefilling and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 193–210.arxiv.org/abs/2401.09670
- [54]Yuan Zhou, Shaojie Xiang, Lingfan Yu, Zhenyu Song, Charith Mendis, and Yida Wang. 2026. SAS: Sparse Attention Synthesizer for Efficient Language Model Inference. In Proceedings of the 21st European Conference on Computer Systems. 1707–1721.DOI
- [55]Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, et al. 2025. NanoFlow: Towards optimal large language model serving throughput. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25). 749–765.arxiv.org/abs/2408.12757
- [56]Qianchao Zhu, Jiangfei Duan, Chang Chen, Siran Liu, Xiuhong Li, Guanyu Feng, Xin Lv, Xiao Chuanfu, Dahua Lin, and Chao Yang. 2025. Sampleattention: Near-lossless acceleration of long context llm inference with adaptive structured sparse attention. Proceedings of Machine Learning and Systems 7 (2025).arxiv.org/abs/2406.15486
- [57]Jiaxiang Zou, Yonghao Chen, Xingyu Chen, Chenxi Xu, and Xinyu Chen. 2025. AxCore: A Quantization-Aware Approximate GEMM Unit for LLM Inference. In Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture. 839–853.DOI










