WorksComputer Science
SpecSearch: Towards Hardware-Aware Efficient Speculative Decoding
Abstract
Speculative decoding (SD) accelerates large language model (LLM) inference, but its optimal configuration depends jointly on workload characteristics, GPU hardware, and available memory. For example, on the RTX~5090, the state-of-the-art speculative drafter DFlash achieves the throughput of autoregressive (AR) decoding at low concurrency, but only at high concurrency with short outputs. We present SpecSearch, a hardware-aware search engine that jointly optimizes decoding strategy, draft length, and precision across workload–GPU pairs. Its analytical cost model captures memory and compute constraints to identify feasible configurations and narrow the search space for throughput optimization. Guided by this model, calibration measurements are collected from multiple GPUs. Using the measurements, a Jev-based predictor learns to select high-throughput decoding configurations on held-out GPUs without requiring deployment-GPU calibration. The predictor is parameter-efficiently trained on calibration data, while adapting a frozen Qwen3-4B using LoRA adapters and a linear correction layer. Across NVIDIA RTX~5090, H100, L40 and AMD MI300X GPUs, SpecSearch with full calibration achieves an average of the throughput of AR and the throughput of SD, both served by vLLM. The proposed Jev-based predictor closely matches this search performance with throughput improvement over SD with vLLM, while eliminating the need for held-out GPU calibration. Additionally, it retains of the throughput achieved by SpecSearch that utilizes full calibration.
Introduction
Speculative decoding (SD) [1, 2] reduces autoregressive (AR) decoding latency: an inexpensive drafter model proposes several draft tokens, and the target model verifies them in parallel in a single forward pass. Verification makes generation lossless, so the output distribution equals that of the target model. State-of-the-art diffusion-LLM drafters such as DFlash [3] propose a whole block of draft tokens in a single drafter forward pass. Although advances in drafter architectures are rapidly improving speculative decoding efficiency, the achievable speedup remains highly dependent on deployment-time decisions: whether to load a drafter model at all, how many tokens to draft (i.e., the draft length) and at which precision to store the model weights and the key–value (KV) cache. These decisions are particularly critical on memory-constrained GPUs. Model weights and speculation buffers consume a major portion of device memory. Consequently, the memory left for the KV cache determines how many requests the inference engine can serve concurrently. This limitation hinders efficient speculative decoding on memory-constrained GPUs.
Modern inference engines such as vLLM [4] run kernels optimized for the deployment hardware. They further enhance aggregate throughput with continuous batching, which admits waiting requests into the running batch as others finish. Additionally, they reduce memory footprint through paging KV cache. Nonetheless, they leave the algorithmic configuration of SD to the user: the draft length and the decoding precision are manual knobs, typically set once and reused across workloads and hardware. We observe that this practice loses substantial throughput, because the best decoding configuration, including whether to speculate at all, depends jointly on the workload and the hardware. Figure 2 illustrates this for Qwen3-4B with a DFlash drafter on a resource-constrained GPU. In Figure 2(a), with shorter text generation, vLLM SD is 3.50× faster than AR decoding at 2 concurrent requests but only 0.76× as fast at 64, because verification becomes compute-bound. In Figure 2(b), with longer text generation, vLLM SD with an fp8 KV cache is the fastest configuration at every concurrency from 32 to 128, ahead of AR decoding and of vLLM SD with an fp16 KV cache. The concurrency at which speculation stops paying off therefore depends on the output length, the memory budget, and the ridge point of the device, i.e., the ratio of its peak compute throughput to its memory bandwidth.
Figure 2. The best decoding configuration and its memory cost depend on the workload. (a) vLLM SD-to-AR throughput ratio with 256-token outputs and fp16 KV; the dashed line marks the critical batch size predicted by SpecSearch cost model (Section IV). (b) With longer outputs (2,048 tokens), vLLM SD with fp8 KV is fastest from concurrency 32 to 128. (c) With shorter outputs (up to 256 tokens), vLLM SD has 3.5–3.9× throughput at concurrency 1–4, but its advantage over AR disappears by concurrency 64. (d) The drafter use reduces KV cache capacity to 0.73× that of AR at equal KV precision.
We present SpecSearch, which automatically identifies the best-performing decoding configuration for a given GPU and workload in terms of throughput and wall-clock speedup. A decoding configuration comprises a set of deployment policies, including whether to enable speculation, setting a static or dynamic draft length based on effective batch size (concurrency), as well as the precision of model weights and the KV cache. We characterize each workload by the number of concurrent requests (), which we refer to interchangeably as concurrency or batch size, and their generation lengths. SpecSearch has two components:
First, an analytical cost model characterizes the device’s memory budget, memory bandwidth, and peak compute throughput. It determines whether a deployment configuration fits within the available memory and, if feasible, estimates the maximum concurrency () that the KV cache can support without request preemption. It also identifies the critical batch size (), beyond which compute contention diminishes the throughput benefits of speculation.
Second, SpecSearch selects the most profitable decoding configuration for each workload on a given GPU through two complementary approaches. In full calibration mode, SpecSearch uses calibration measurements guided by the cost model to search the policy space and identify the best-performing configuration for each workload. These configurations are then stored in a lookup table before real-time inference deployment. This workflow is illustrated in Figure 1. We further use the measured calibration data to construct a training dataset for fine-tuning a Jev-based policy selector [5]. Alternatively, in learning-based mode, SpecSearch leverages this trained model to predict the optimal decoding configuration for each workload on a given GPU, eliminating the need for prior calibration on held-out GPUs.
Figure 1. Overview of SpecSearch with full calibration: Given the GPU and the workload (1), the proposed cost model discards deployment configurations that exceed the memory budget and determines the critical batch size of the device (2). Calibration on the deployment GPU runs AR and SD where quantization and optimal draft lengths are explored within vLLM (3). SpecSearch explores dynamic draft-length mapping for vLLM that sets from the effective batch size at every decoding step (4). The most profitable configuration for each workload is frozen into a policy lookup table to be reused during real-time inference (5).
Contributions.
Defining a joint optimization space: We provide evidence that the optimal speculative decoding (SD) configuration depends both on the workload and the deployment GPU (Section III). SpecSearch introduces a hardware-aware cost model to narrow the search space for finding performance-optimal decoding policies.
Validation and portability of cost model: SpecSearch’s cost model estimates total GPU memory footprint for LLM inference within 0.1% and 0.2% on the AMD MI300X [6] and NVIDIA RTX 5090 [7] GPUs, respectively. Furthermore, it estimates a maximum serviceable concurrency within 0.5% and 0.6% on the GPUs, respectively. The cost model also correctly predicts whether a workload fits a device budget under all evaluated configurations on the two GPUs.
End-to-end performance gains: SpecSearch with full calibration achieves 2.16× and 1.68× the throughput of AR on RTX 5090 and MI300X, respectively, averaged across several workloads. Furthermore, it achieves 1.44× and 1.21× speedup over baseline SD served by vLLM, respectively. Across four different GPUs, it delivers an average 1.36× speedup over vLLM SD. The Jev-based predictor closely approaches the full calibration SpecSearch performance, retaining 97.36% of its throughput and achieving 1.30× the throughput of vLLM SD.
Background and Related Work
Speculative decoding. Draft-then-verify approach was introduced with an independent draft model [1, 2]. Medusa uses parallel decoding heads with tree attention [8]. EAGLE drafts at the level of target features [9]. EAGLE-3 drafts tokens from fused multi-layer target features [10]. State-of-the-art (SoTA) DFlash drafters propose a block of tokens in a single forward pass as a block-diffusion model conditioned on the target features [3]. All methods deliver lossless generation with respect to the target distribution. We use DFlash drafters for evaluating SD throughput.
Paged KV cache and memory-efficient speculation. vLLM [4] and SGLang [11] are LLM serving frameworks that employ paged KV-cache management to support efficient memory allocation and high-throughput serving. vLLM [4] introduces PagedAttention, which manages KV-cache entries in non-contiguous, fixed-size blocks, reducing memory fragmentation and enabling flexible KV-cache sharing and allocation across requests. Building on that, SpecSearch proposes a cost model that estimates the maximum serviceable concurrency a paged KV cache can support. Prior works on memory-efficient speculative decoding include SpecMemo [12], which optimizes runtime buffer allocation in Medusa [8] to improve single-request throughput and reduce memory footprint. However, it does not account for concurrency under paged KV-cache management or workload-dependent throughput.
Dynamic draft-length selection. vLLM and SGLang expose configurable speculative decoding parameters, including the draft length, which is the number of consecutive tokens proposed by the drafter. vLLM supports user-defined mappings from batch-size ranges to draft lengths. SGLang provides adaptive SD that adjusts draft length based on observed acceptance rates within a workload. However, neither determines an optimal draft length based on both the workload and deployment hardware. SpecSearch automates this process through GPU-specific calibration, selecting draft-length policies based on measured throughput and which precision to use. Thus, SpecSearch performs resource- and workload-aware dynamic draft length selection.
Quantization. Model-weight [13], [14], [15] and KV-cache quantization [16], [17], [18], [19] are established techniques for reducing the memory footprint of LLM inference. SpecSearch incorporates both model-weight precision and KV-cache precision as configurable dimensions of speculative decoding (SD), and quantifies their individual and combined effects on end-to-end performance.
Adapting speculation to workload and hardware. Existing works address complementary aspects of optimizing SD. Sequoia profiles hardware-dependent draft and verification costs to select speculative tree size and depth [20]. MagicDec [21] models the transition from memory-bound to compute-bound execution as batch size increases in self-speculative decoding, and uses this analysis to optimize speculation for long-context workloads. At the serving layer, TurboSpec controls draft lengths under changing load [22]. AdaServe constructs hardware-aware speculative decoding trees for per-request latency objectives [23]. Unlike prior work, SpecSearch addresses a broader deployment-level choice: whether to use AR or SD, which static or dynamic draft length to use, and which precision fits the deployment GPU’s resource constraints. The proposed cost model analyzes workloads and calibrates them on GPUs to select a workload- and GPU-specific decoding policy before serving.
Learning-based deployment optimization. Prior work predicts performance of deep learning workloads across hardware platforms to reduce the cost of hardware profiling. Habitat scales operation-level measurements from a source GPU to estimate training iteration time on other GPUs [24]. NeuSight predicts training and inference latency on GPUs using learned tile-level models constrained by hardware performance bounds [25]. For LLM inference, LLMCompass models hardware architectures and explores operator mappings and schedules to estimate latency and analyze performance–cost tradeoffs [26]. These works provide a foundation for making deployment decisions across GPUs. Building on this
direction, SpecSearch proposes training a policy selector on GPU performance measurements to select optimal decoding configurations. Recent Jev [5] is a lightweight decision model that predicts choices among predefined alternatives, enabling efficient learned decision-making. Unlike prior approaches, SpecSearch fine-tunes a Jev-based policy selector for predicting the optimal decoding configuration across LLM workloads and hardware.
Motivation
We preliminarily evaluate AR and SD using vLLM with FP16 and FP8 KV-cache precisions on an RTX 5090. We find that the best-performing configuration depends jointly on the workload and GPU.
Observation 1: The concurrency at which speculation loses its advantage depends on the workload and GPU. As concurrency increases, decoding shifts from memory-bound to compute-bound execution [27], reducing speculation’s benefit. The transition depends on the GPU’s compute-to-bandwidth ratio, which our cost model estimates through the critical batch size (Table I). SD throughput advantage for 256-token outputs drops from 3.50× at concurrency 2 to 1.21× at 32, then reverses to 0.76× at 64 (Figure 2(a)). We validate this finding on AMD MI300X GPU, where the SD gain is 1.13× at concurrency 32. For 2,048-token outputs, SD remains advantageous beyond concurrency 128 (Figure 2(b)).
Observation 2: KV quantization benefits long outputs. For 4,096-token outputs at concurrency 48, throughput reaches 8,479 tokens/s with SD and fp8 KV, versus 6,813 with SD and fp16 KV. This yields a 1.24× speedup, with nearly identical acceptance lengths (11.8 vs. 11.9). The benefit persists for 2,048-token outputs: fp8 KV improves SD throughput by 1.10–1.16× across concurrency levels 32–128 (Figure 2(b)). For outputs up to 256 tokens, however, the difference is at most 4% (Figure 2(c)). Thus, KV quantization primarily benefits long-output workloads where KV-cache capacity limits concurrency.
Observation 3: Speculation reduces KV-cache capacity. Drafter weights and runtime buffers consume memory otherwise available for KV caching. Under a 15.7 GiB budget on RTX 5090, vLLM SD retains only 0.73× the AR KV-cache capacity at equal KV precision (Figure 2(d)).
These observations motivate a joint search mechanism for optimal decoding policy selection across workloads and GPUs. SpecSearch combines an analytical cost model with device calibration to capture hardware-specific costs at run-time.
| GPU | GB | (TB/s) | (TF/s) | predicted |
| MI300X | 192 | 5.30 | 1,307 | 30.8 |
| RTX 5090 | 32 | 1.79 | 209.5 | 29.3 |
Table 1. Device parameters of SpecSearch cost model: memory capacity, peak theoretical bandwidth , and peak dense FP16 matrix throughput [6, 7]. denotes the GPU roofline knee point.
SpecSearch
SpecSearch searches for an optimal deployment policy for LLM inference that maximizes throughput and speedup on a given GPU. In full calibration mode, SpecSearch uses the cost model to define the policy search space and performs calibration runs to evaluate the policies. In learning-based mode, a fine-tuned Jev-based model infers the optimal policy on held-out GPUs without calibration.
Notation. GPU has nominal memory bandwidth , peak fp16 compute throughput (see Table I), and inference memory budget . The target and drafter have parameters and layers. We denote the number of KV heads by , head dimension by , hidden size by , and vocabulary size by . A policy specifies draft length , target and drafter weight sizes bytes (fp16, fp8, int4), KV element size bytes (fp16, fp8), and execution mode (CUDA/HIP graphs or eager). A workload specifies batch size and generation length . Given prompt length , the per-request context length is .
Cost Model
Memory model. Peak memory is
where the terms represent model weights, KV caches, the context buffer for target layers, verification logits, CUDA/HIP graph memory (zero in eager mode), and runtime/allocator overhead, respectively. Here, is the activation element size in bytes, and is the KV-cache cost per context token. The drafter cache stores keys and values projected from target model hidden states.
Concurrency model. The available KV-cache capacity, measured in tokens, is
where and includes the context buffer and verification logits. Maximum supported concurrency is
A configuration is feasible at required concurrency if .
Throughput model. Following the roofline model [27], we approximate decode-step time as
where is target arithmetic per token. The terms capture weight reads and computation over tokens, respectively. At low concurrency, decoding is memory-bound, and speculation incurs approximately one AR verification step while accepting draft tokens per step, yielding an approximate speedup of . At high concurrency, decoding becomes compute-bound, reducing the approximate speedup to .
Equating the two terms for AR decoding () gives the critical batch size
where is the fraction of peak compute throughput achieved during decoding. Model size cancels, making primarily dependent on the device’s compute-to-bandwidth ratio and target weight precision. We use to estimate the transition from memory-bound to compute-bound decoding.
Joint cost model. Given quality tolerance , we select
Here, contains configurations feasible under the device memory budget. The quality constraint applies only to lossy target-weight and KV precisions; draft length and drafter precision preserve the target distribution. Since the analytical model cannot capture low-level kernel efficiency, the full calibration setting measures on the deployment GPU. On held-out GPUs, our zero-shot approach ranks static candidates using predictions or selection scores derived from source-GPU measurements.
SpecSearch with full calibration
The cost model identifies the set of feasible deployment policies for a workload on a given GPU, which may include varying precision settings for model weights and KV cache, as well as a draft length mapping. SpecSearch performs an offline calibration pass for each feasible policy determined by the cost model on the deployment GPU. Accordingly, it ranks these deployment policies by their mean wall-clock speedup and throughput relative to AR decoding. The best policy (whether SD or AR) is then stored in a lookup table for inference-time selection before deploying the workload.
The effective batch size, defined as the number of scheduled requests per decoding step, varies as requests finish, are admitted, or are preempted in vLLM. Together with KV-cache precision, it determines GPU resource contention and KV-cache pressure. Thus, the optimal draft length also varies with the effective batch size and precision.
Dynamic draft-length search. SpecSearch constructs a dynamic draft length mapping from calibration data, identifying the fastest-performing for each workload and precision. At each decoding step, SpecSearch uses the mapping to dynamically set the corresponding draft length within vLLM. Table VI provides an example of the dynamic draft length mapping derived by SpecSearch. Batches 21–48 use for and , but for . With fp8 KV and , the general trend is , 7, and 3, respectively, for batches 1–20, 21–48, and 49–128.
Learning-Based Policy Predictor
SpecSearch trains a lightweight Jev-based policy predictor [5] to eliminate calibration on held-out GPUs. Given a deployment GPU and workload , the predictor selects the decoding mode, draft length, and KV-cache precision.
Training and validation datasets. The dataset covers eight decoding configurations: AR and SD with , each using fp16 or fp8 KV cache. Each GPU runs 96 configurations across and . We use four distinct GPUs when creating the dataset: two for training, one for validation, and one held out for evaluation. For each training GPU, the highest-throughput configuration per workload serves as the training label.
Model architecture. We adapt Qwen3-4B into an eight-choice Jev-style decision model by training rank-8 LoRA adapters on its attention projections. Each choice represents one of the eight decoding configurations in the training dataset. We additionally train a linear answer-logit correction layer using cross-entropy loss, while keeping the pretrained backbone frozen.
Training objective. Each training example represents one configuration-selection problem for a deployment GPU and workload . Its structured prompt contains the GPU and workload metadata, eight candidate configurations labeled A–H in shuffled order, and the permitted source calibration records. The label is the answer letter assigned to the candidate with the highest measured throughput for that GPU–workload pair. Given the prompt, Qwen produces a final hidden representation , from which we compute the eight answer logits as
where denotes the frozen rows of Qwen’s language-model output head corresponding to tokens A–H, and is a trainable, zero-initialized linear correction layer. Rank-8 LoRA adapters on the query, key, value, and output attention projections modify the hidden representation , while the pretrained Qwen weights remain frozen. We optimize the LoRA parameters and using the classification cross-entropy objective
Thus, training encourages the model to assign the highest answer score to the measured throughput-maximizing configuration. Further details are provided in Appendix B.
Learning-based policy prediction. At deployment time, the trained Jev-based predictor takes a structured text input, termed a decision prompt, describing the deployment GPU, workload, and eight candidate configurations labeled A through H. Rather than generating text, the model scores the corresponding answer-label tokens and selects the configuration with the highest score. This enables policy selection on a held-out GPU without calibration or profiling, leveraging the relationships between workloads, hardware characteristics, and decoding configurations learned during training.
Experimental Setup
GPUs. We evaluate four GPUs: NVIDIA GeForce RTX 5090 (32 GB) [28], AMD Instinct MI300X (192 GB) [29], NVIDIA H100 (80 GB) [30], and NVIDIA L40 (48 GB) [31].
Models and datasets. We use Qwen3-4B [32] target model and its DFlash drafter [3] with block size 16. Prompts are sampled from GSM8K [33], MATH-500 [34], MBPP [35], MT-Bench [36], and HumanEval [37].
Baselines. We compare against AR and default SD (, no adjustments made to default drafting length) served by vLLM. Furthermore, we evaluate against MagicDec 3.
Calibration. When the cost model is disabled, we evaluate eight policies across several GPUs: AR decoding with FP16 or FP8 KV cache, and SD with , each using FP16 or FP8 KV cache. We consider workloads with and . On each GPU, the 12 workloads and three seeds yield $8 \times 12 \times 3 = 288 calibration runs. We additionally calibrate two dynamic draft-length mappings per workload and seed, adding 72 runs. Thus, an exhaustive calibration performs 360 runs per GPU.
Results
In this section, we evaluate both policy selection modes of SpecSearch: full calibration and learning-based policy selection.
In full calibration mode, SpecSearch uses its analytical cost model to refine the policy search space and identify feasible candidate configurations, then selects the best-performing policy through GPU-specific calibration.
The learning-based mode trains a Jev-based predictor to select optimal configurations from calibration data. As a result, the accuracy and portability of the cost model directly affect the quality of full calibration policy selection and the training dataset of the learned predictor.
We first validate the cost model across GPUs by comparing its predictions against measurement-based reference points obtained from actual vLLM executions. We then evaluate the end-to-end performance of both policy selection modes.
Cost model validation across GPUs
Table II validates the proposed cost model, including memory, concurrency, and throughput. We compare the predicted cost by our model against the measured cost when running vLLM for AR and SD with two KV cache precisions. (1) predicts total engine memory within 0.1% on the MI300X and 0.2% on the RTX 5090. (2) predicts maximum concurrency within 0.5% and 0.6%, respectively. Across the tested budgets, the cost model correctly rejects 7 of 20 configuration–budget pairs on the MI300X and 3 of 20 on the RTX 5090. These rejected pairs occur at allocations below the 15.7 GiB budget used for the main evaluation. In other words, the cost model correctly accepts all remaining pairs.
| configuration | MI300X Memory (GB) pred. | MI300X Memory (GB) meas. | MI300X Memory (GB) err. | MI300X pred. | MI300X meas. | MI300X err. | MI300X Starts within allocation (GiB) 9 | MI300X Starts within allocation (GiB) 10 | MI300X Starts within allocation (GiB) 11 | MI300X Starts within allocation (GiB) 12 | MI300X Starts within allocation (GiB) 15.7 | RTX 5090 Memory (GB) pred. | RTX 5090 Memory (GB) meas. | RTX 5090 Memory (GB) err. | RTX 5090 pred. | RTX 5090 meas. | RTX 5090 err. | RTX 5090 Starts within allocation (GiB) 9 | RTX 5090 Starts within allocation (GiB) 10 | RTX 5090 Starts within allocation (GiB) 11 | RTX 5090 Starts within allocation (GiB) 12 | RTX 5090 Starts within allocation (GiB) 15.7 |
| AR, fp16 KV | 17.01 | 17.02 | +0.1% | 11.07 | 11.05 | −0.2% | ×/× | ✓/✓ | ✓/✓ | ✓/✓ | ✓/✓ | 16.92 | 16.93 | +0.1% | 13.24 | 13.22 | −0.2% | ✓/✓ | ✓/✓ | ✓/✓ | ✓/✓ | ✓/✓ |
| AR, fp8 KV | 17.01 | 17.02 | +0.1% | 22.14 | 22.11 | −0.1% | ×/× | ✓/✓ | ✓/✓ | ✓/✓ | ✓/✓ | 16.89 | 16.92 | +0.2% | 26.48 | 26.33 | −0.6% | ✓/✓ | ✓/✓ | ✓/✓ | ✓/✓ | ✓/✓ |
| vLLM SD, fp16 KV | 17.54 | 17.56 | +0.1% | 7.14 | 7.10 | −0.5% | ×/× | ×/× | ×/× | ✓/✓ | ✓/✓ | 17.29 | 17.31 | +0.1% | 8.83 | 8.80 | −0.4% | ×/× | ×/× | ✓/✓ | ✓/✓ | ✓/✓ |
| vLLM SD, fp8 KV | 17.54 | 17.56 | −0.5% | 14.28 | 14.20 | −0.5% | ×/× | ×/× | ✓/✓ | ✓/✓ | ✓/✓ | 17.28 | 17.31 | +0.2% | 17.66 | 17.57 | −0.5% | ×/× | ✓/✓ | ✓/✓ | ✓/✓ | ✓/✓ |
Table II. Validation of the cost model across policies. Predicted (pred.) and measured (meas.) denote cost-model predictions for memory footprint and realized allocation, respectively. Additionally, GPU memory budgets are emulated, ranging from 9 to 15.7 GiB, to determine deployment feasibility. Thus, each cell contains two symbols: the first is the cost model prediction for feasibility and the second is the vLLM deployment outcome. A ✓ means feasible (runs successfully), × means out-of-memory (fails).
The model-dependent per-token KV-cache cost () and drafter weight footprint match measured allocations within 1%, without any device-specific adjustments. Only the constant runtime overheads (, , and ) require a one-time startup measurement on each GPU. These results validate the cost model on MI300X and RTX 5090 after device-specific initialization.
Performance of SpecSearch with full calibration
Throughput and speedup results. Table III and IV report throughput (in k tokens/s) and wall-clock speedup measured by the calibration runs, respectively. The last two columns in each GPU block report the throughput improvements achieved by the policy SpecSearch selects over the fastest AR and vLLM SD configurations, respectively. Each GPU has available 15.7 GiB device budget for LLM inference. Similarly, Table XII and XIII report findings on NVIDIA H100 and L40 GPUs at a 15.7 GiB budget.
The performance gains vary with concurrency, output length, and GPU. At , vLLM SD is already competitive, leaving little room for further improvement (Table III). At higher concurrency, SpecSearch more consistently outperforms
the vLLM SD baseline, particularly for longer outputs. For example, at and , SpecSearch reaches 12.03 k tokens/s on the MI300X and 9.53 k tokens/s on the RTX 5090, versus 8.91 and 5.55 k tokens/s, respectively, for vLLM SD with fp8 KV. The same setup delivers wall-clock speedups of 1.96× and 1.52× over AR with fp16 KV.
Averaged across workloads, SpecSearch achieves throughput improvements of 1.68×, 2.16×, 2.33×, and 1.85× over the best AR configuration on the MI300X, RTX 5090, H100, and L40, respectively. The corresponding speedup values over vLLM SD are 1.21×, 1.44×, 1.28×, and 1.51×. These results demonstrate SpecSearch’s ability to identify the best deployment configuration depending on both the workload and the GPU.
Figure 3 compares the end-to-end throughput of MagicDec and SpecSearch on the same RTX 5090, using Qwen3-4B as the target model and an identical prompt set. The two systems employ different serving optimizations: MagicDec uses sparse-KV self-speculation to compress the KV cache and reduce memory consumption for long-context workloads, with BF16 KV cache, static batching, and no request refill in our evaluation. In contrast, SpecSearch uses the vLLM backend, which supports paged KV caching and continuous batching, and jointly selects the KV-cache precision and speculation policy. These differences in serving mechanisms also affect the set of workloads each system can execute: MagicDec cannot run six decoding configurations due to memory constraints. Among the workloads supported by both systems, MagicDec achieves 0.28–1.26 k tokens/s, corresponding to 0.08–0.13× the throughput of SpecSearch. The results thus compare the end-to-end performance of the respective systems under a common hardware, model, and prompt configuration, rather than isolating the contribution of individual optimizations.
Figure 3. Throughput (in k tokens/s) comparison of MagicDec vs. SpecSearch for Qwen3-4B. Each cell reports MagicDec throughput on RTX5090, while its ratio to SpecSearch is given in parentheses. Gray cells denote out-of-memory (OOM) failures.
Selected policies. SpecSearch calibrates SD across different draft lengths to construct dynamic draft-length mappings. Table V details the policies selected by SpecSearch in Tables III and IV, including their corresponding draft lengths. Tables XIV and XV provide the individual calibration results used to derive draft length mappings. On the MI300X, SpecSearch selects speculation with fp8 KV at every workload, using a dynamic draft-length mapping at one workload and a static draft length at eleven workloads. On the RTX 5090, it selects fp16 KV at six workloads and fp8 KV at six, including the one workload at which AR decoding is selected. Eight RTX 5090 workloads use a static draft length, three use the dynamic mapping, and one uses AR decoding. tbl:6 lists the dynamic draft length mappings constructed by SpecSearch. On H100, five workloads use dynamic mappings and seven use static draft lengths, with fp16 KV selected at four workloads and fp8 at eight. On L40, two workloads use dynamic mappings, nine use static draft lengths, while one uses AR and all select fp8 KV. Target model quantization. At a smaller device budget such as 8 GiB, serving SD may require quantizing the target model. Table VII shares the policies with a quantized target and drafter that calibration explores, demonstrating feasible configurations
| policy | target | draft | KV cache | weights (GiB) | KV tokens | max conc. | ktok/s | wall (s) | speedup |
| AR | fp16 | – | fp16 | 7.56 | |||||
| AR | fp16 | – | fp8 | 7.59 | |||||
| AR | fp8 | – | fp16 | 4.24 | 19,712 | 4.81× | 8.617 | 5.942 | 0.63× |
| AR | fp8 | – | fp8 | 4.24 | 39,568 | 9.66× | 13.661 | 3.748 | 1.00× |
| SD | fp16 | fp16 | fp16 | 8.61 | |||||
| SD | fp16 | fp16 | fp8 | 8.65 | |||||
| SD | fp8 | fp16 | fp16 | 5.36 | |||||
| SD | fp8 | fp16 | fp8 | 5.36 | 5,376 | 1.31× | 5.701 | 8.981 | 0.42× |
| SD | fp16 | fp8 | fp8 | 8.1 | |||||
| SD | fp8 | fp8 | fp8 | 4.83 | 11,808 | 2.88× | 5.898 | 8.681 | 0.43× |
| SD | fp16 | int4 | fp8 | 7.85 | |||||
| SD | fp8 | int4 | fp8 | 4.55 | 6,960 | 1.70× | 6.093 | 8.403 | 0.45× |
Table VII. Quantization explored under an 8 GiB memory budget on the RTX 5090. Shaded rows indicate policies that exceed the device memory budget. Target and draft denote the target and draft model precisions, respectively. .
| \multicolumn{6}{c}{MI300X} | \multicolumn{8}{c}{RTX 5090} | |||||||||||||||||
| vLLM AR fp16 | vLLM AR fp8 | vLLM SD fp16 | vLLM SD fp8 | SpecSearch fp16 | SpecSearch fp8 | best AR | best vLLM SD | AR fp16 | AR fp8 | vLLM SD fp16 | vLLM SD fp8 | SpecSearch fp16 | SpecSearch fp8 | best AR | best SD | |||
| 256 | 8 | 1.88 | 1.90 | 4.26 | 4.55 | 4.32 | 4.69 | 2.47× | 1.03× | 1.02 | 1.01 | 3.91 | 3.89 | 3.91 | 3.84 | 3.83× | 1.00× | |
| 32 | 5.15 | 5.75 | 7.18 | 8.14 | 8.38 | 8.88 | 1.54× | 1.09× | 3.65 | 3.67 | 5.13 | 5.19 | 7.36 | 7.54 | 2.01× | 1.42× | ||
| 64 | 8.21 | 9.34 | 8.45 | 8.93 | 10.88 | 11.91 | 1.28× | 1.33× | 6.37 | 6.65 | 5.44 | 5.53 | 9.46 | 9.31 | 1.42× | 1.71× | ||
| 128 | 11.98 | 13.91 | 9.38 | 10.13 | 14.99 | 16.24 | 1.17× | 1.60× | 10.01 | 10.76 | 5.67 | 5.78 | 10.48 | 10.89 | 1.00× | 1.86× | ||
| 1,024 | 8 | 1.81 | 1.86 | 3.58 | 4.18 | 3.58 | 4.32 | 2.32× | 1.03× | 1.03 | 1.03 | 3.73 | 3.75 | 3.73 | 3.75 | 3.62× | 0.99× | |
| 32 | 3.92 | 4.84 | 6.09 | 6.89 | 6.85 | 7.84 | 1.62× | 1.14× | 3.25 | 3.31 | 4.89 | 4.94 | 7.01 | 7.00 | 2.12× | 1.42× | ||
| 64 | 5.85 | 7.16 | 7.21 | 8.53 | 8.78 | 10.15 | 1.42× | 1.19× | 5.14 | 5.37 | 5.30 | 5.37 | 8.31 | 8.53 | 1.59× | 1.59× | ||
| 128 | 8.32 | 10.76 | 8.14 | 10.13 | 9.83 | 14.77 | 1.37× | 1.46× | 7.97 | 9.04 | 5.29 | 5.77 | 8.48 | 10.29 | 1.14× | 1.78× | ||
| 2,048 | 8 | 1.75 | 1.81 | 3.42 | 4.03 | 3.40 | 4.03 | 2.23× | 1.00× | 1.01 | 1.01 | 3.75 | 3.65 | 3.75 | 3.65 | 3.71× | 1.00× | |
| 32 | 3.14 | 3.93 | 5.50 | 6.47 | 6.05 | 7.17 | 1.82× | 1.11× | 2.67 | 2.77 | 4.91 | 4.81 | 6.12 | 6.34 | 2.29× | 1.29× | ||
| 64 | 4.62 | 5.73 | 6.39 | 7.48 | 7.48 | 9.19 | 1.60× | 1.23× | 3.95 | 4.21 | 5.06 | 5.16 | 7.10 | 7.65 | 1.82× | 1.48× | ||
| 128 | 6.05 | 8.79 | 6.90 | 8.91 | 8.75 | 12.03 | 1.37× | 1.35× | 6.30 | 7.16 | 5.16 | 5.55 | 7.86 | 9.53 | 1.33× | 1.72× | ||
| Average | 1.68× | 1.21× | 2.16× | 1.44× |
Table 3. Throughput (K tokens/s) of decoding policies at 15.7 GiB device budget. Bold: highest mean value per workload. Underlined: the policy selected by SpecSearch. Within each method (AR, vLLM SD, SpecSearch), the darker shading marks FP8 KV in contrast to FP16 KV cache.
| vLLM AR fp16 | vLLM AR fp8 | vLLM SD fp16 | vLLM SD fp8 | SpecSearch fp16 | SpecSearch fp8 | best AR | best vLLM SD | AR fp16 | AR fp8 | Baseline SD fp16 | Baseline SD fp8 | SpecSearch fp16 | SpecSearch fp8 | best AR | best SD | |||
| 256 | 8 | 1.00× | 1.01× | 2.27× | 2.42× | 2.30× | 2.50× | 2.46× | 1.03× | 1.00× | 0.99× | 3.84× | 3.82× | 3.84× | 3.77× | 3.84× | 1.00× | |
| 32 | 1.00× | 1.12× | 1.39× | 1.58× | 1.62× | 1.72× | 1.54× | 1.09× | 1.00× | 1.01× | 1.41× | 1.43× | 2.02× | 2.06× | 2.00× | 1.41× | ||
| 64 | 1.00× | 1.13× | 1.03× | 1.08× | 1.32× | 1.45× | 1.28× | 1.35× | 1.00× | 1.05× | 0.85× | 0.87× | 1.49× | 1.46× | 1.42× | 1.71× | ||
| 128 | 1.00× | 1.16× | 0.78× | 0.85× | 1.25× | 1.36× | 1.17× | 1.60× | 1.00× | 1.08× | 0.57× | 0.58× | 1.05× | 1.09× | 1.00× | 1.87× | ||
| 1,024 | 8 | 1.00× | 1.03× | 1.99× | 2.34× | 1.99× | 2.40× | 2.34× | 1.03× | 1.00× | 1.01× | 3.63× | 3.66× | 3.63× | 3.66× | 3.59× | 0.99× | |
| 32 | 1.00× | 1.23× | 1.57× | 1.76× | 1.76× | 2.01× | 1.63× | 1.14× | 1.00× | 1.02× | 1.53× | 1.53× | 2.14× | 2.13× | 2.10× | 1.40× | ||
| 64 | 1.00× | 1.22× | 1.22× | 1.44× | 1.50× | 1.72× | 1.42× | 1.19× | 1.00× | 1.05× | 1.05× | 1.05× | 1.66× | 1.66× | 1.58× | 1.59× | ||
| 128 | 1.00× | 1.30× | 0.98× | 1.22× | 1.19× | 1.77× | 1.36× | 1.45× | 1.00× | 1.13× | 0.67× | 0.73× | 1.06× | 1.30× | 1.15× | 1.78× | ||
| 2,048 | 8 | 1.00× | 1.06× | 1.98× | 2.32× | 2.01× | 2.32× | 2.19× | 1.00× | 1.00× | 1.01× | 3.77× | 3.61× | 3.77× | 3.61× | 3.73× | 1.00× | |
| 32 | 1.00× | 1.23× | 1.77× | 2.01× | 1.93× | 2.26× | 1.83× | 1.12× | 1.00× | 1.01× | 1.82× | 1.79× | 2.26× | 2.30× | 2.28× | 1.26× | ||
| 64 | 1.00× | 1.25× | 1.38× | 1.60× | 1.61× | 1.96× | 1.57× | 1.22× | 1.00× | 1.06× | 1.30× | 1.28× | 1.83× | 1.92× | 1.81× | 1.47× | ||
| 128 | 1.00× | 1.45× | 1.13× | 1.46× | 1.45× | 1.96× | 1.36× | 1.35× | 1.00× | 1.13× | 0.83× | 0.87× | 1.26× | 1.52× | 1.34× | 1.75× | ||
| Average | 1.68× | 1.21× | 2.16× | 1.44× |
Table 4. Speedup of decoding policies at 15.7 GiB device budget. Bold: highest mean value per workload. Underlined: the policy selected by SpecSearch. Within each method (AR, vLLM SD, SpecSearch), the darker shading marks FP8 KV in contrast to FP16 KV cache.
| MI300X | 256 | , fp8 | , fp8 | , fp8 | , fp8 |
| 1,024 | , fp8 | , fp8 | , fp8 | , fp8 | |
| 2,048 | , fp8 | , fp8 | , fp8 | , fp8 | |
| RTX 5090 | 256 | , fp16 | , fp16 | , fp16 | AR, fp8 |
| 1,024 | , fp16 | , fp16 | , fp16 | , fp8 | |
| 2,048 | , fp16 | , fp8 | , fp8 | , fp8 | |
| H100 | 256 | , fp8 | , fp8 | , fp8 | , fp16 |
| 1,024 | , fp16 | , fp8 | , fp8 | , fp8 | |
| 2,048 | , fp16 | , fp8 | , fp16 | , fp8 | |
| L40 | 256 | , fp8 | , fp8 | , fp8 | AR, fp8 |
| 1,024 | , fp8 | , fp8 | , fp8 | , fp8 | |
| 2,048 | , fp8 | , fp8 | , fp8 | , fp8 | |
| GPU | 1–20 | 21–48 | 49–96 | 97–128 | |
| MI300X | 256 | 7/7 | 7/3 | 3/3 | 3/3 |
| MI300X | 1,024 | 15/15 | 7/3 | 7/3 | 3/3 |
| MI300X | 2,048 | 15/15 | 7/7 | 7/3 | 7/3 |
| RTX 5090 | 256 | 15/15 | 3/3 | 3/3 | 3/3 |
| RTX 5090 | 1,024 | 15/15 | 3/3 | 3/3 | 3/3 |
| RTX 5090 | 2,048 | 15/15 | 7/7 | 3/3 | 3/3 |
| H100 | 256 | 15/15 | 7/7 | 3/3 | 3/3 |
| H100 | 1,024 | 15/15 | 7/7 | 3/3 | 3/3 |
| H100 | 2,048 | 15/15 | 7/7 | 7/7 | 3/3 |
| L40 | 256 | 15/15 | 3/3 | 3/3 | 3/3 |
| L40 | 1,024 | 15/15 | 3/3 | 3/3 | 3/3 |
| L40 | 2,048 | 15/15 | 3/3 | 3/3 | 3/3 |
Table 5. Policy selected by SpecSearch in Table III and IV: AR vs. SD decoding (with draft length ) and KV precision. : dynamic draft-length mapping for SD (Table VI).
under this budget on an RTX 5090. The SD policies that fit 8 GB are slower than AR decoding on this GPU. tbl:XI reports the throughput of configurations with an fp8 target explored by SpecSearch on the MI300X, using a 15.7 GiB budget. Within this setting, SpecSearch averages 2.17× the throughput of AR and 1.03× that of vLLM SD. Relative to the corresponding fp16-target results in tbl:III, target quantization reduces the throughput of AR, vLLM SD, and SpecSearch across all workloads on the MI300X GPU. This is because quantizing the target model perturbs the reference distribution, introducing loss to verification and harming the overall acceptance.
Drafter model quantization. Quantized drafter models leave the output distribution of SD unchanged, maintaining losslessness. Since drafter only determines the draft token quality, a quantized drafter only impacts the draft token acceptance length. tbl:VIII reports this effect, measured on the RTX 5090 at with fp16 target and fp8 KV cache. Quantized fp8 and int4 drafters reduce drafter weight memory by 50% and 74%, while the corresponding acceptance retention stays within 1% of the fp16 drafter. Thus, neither drafter precision changes serving throughput measurably.
Performance of Learning-Based Policy Predictor
We evaluate zero-shot policy selection using a four-fold cross-GPU evaluation. In each fold, one of the four GPUs is held
| batch size | drafter | weights (GiB) | retention | k tok/s | retention | |
| 32 | fp16 | 1.01 | 9.87 | 1.0 | 8.98 | 1.0× |
| 32 | fp8 | 0.51 | 9.90 | 1.0 | 9.16 | 1.0× |
| 32 | int4 | 0.26 | 9.81 | 0.9 | 9.00 | 1.0× |
| 64 | fp16 | 1.01 | 9.81 | 1.0 | 8.63 | 1.0× |
| 64 | fp8 | 0.51 | 10.03 | 1.0 | 9.43 | 1.0× |
| 64 | int4 | 0.26 | 9.76 | 0.9 | 8.97 | 1.0× |
Table VIII. Drafter quantization on the RTX 5090. is the mean acceptance length of draft tokens across the datasets. Retention is the rate at which mean acceptance and throughput are preserved relative to FP16.
out for evaluation, while measurements from the remaining three GPUs are used to train the predictor. We repeat this process four times, using each GPU as the held-out GPU exactly once. Each GPU contributes 12 workloads, resulting in 48 workload-hardware pairs across the four folds. We assess each policy predicted by our Jev-based selector by measuring its throughput on the held-out GPU. These measurements are strictly reserved for evaluation and are excluded from predictor training and model selection. We compare the performance of configurations selected by the Jev-based predictor against the performance of policy search predicted in full calibration mode. Unlike the learned predictor, this SpecSearch mode has access to calibration measurements from the evaluation GPU.
Selected policy performance. tbl:IX evaluates how effectively the trained Jev-based predictor selects decoding configurations on unseen GPUs, comparing it against the baseline vLLM decoding. The Jev-based predictor achieves average throughput ratios of 2.02× over AR and 1.30× over vLLM SD. Selected policies are shared in Appendix B.
Comparison with full calibration. tbl:IX shares the throughput gap that measures the relative performance loss of the inferred policy compared with the best-performing configuration returned by SpecSearch in full calibration mode. A smaller gap indicates more efficient policy selection. Relative to the best measured static configuration, its mean throughput loss is 0.75% on RTX 5090, 1.80% on MI300X, 1.52% on L40, and 5.95% on H100. Averaged equally across the four held-out GPUs, the fine-tuned Jev-based predictor retains 97.36% of the matched full calibration SpecSearch throughput. Jev-based fine-tuning maintains an average throughput gap of 2.50% to the optimal policies discovered during calibration. These results demonstrate effective policy selection without deployment-GPU profiling.
Predictions across varying concurrency. We study how workload concurrency influences the predictor’s choices. The largest benefit of Jev-based prediction occurs at , where the mean throughput gap to the best configuration’s throughput decreases from 10.76% to 1.82%. The Jev-based model selects for all 12 workloads at this concurrency, whereas the frozen Qwen-3-4B decision model selects for all four 256-token workloads. At , both models sets with fp8
| GPU | |||||
| RTX 5090 | MI300X | H100 | L40 | Mean | |
| Jev-based vs. F.C. | 0.75% | 1.80% | 5.95% | 1.52% | 2.50% |
| Jev-based vs. vLLM AR | 2.24× | 1.68× | 2.27× | 1.90× | 2.02× |
| Jev-based vs. vLLM SD | 1.40× | 1.14× | 1.19× | 1.47× | 1.30× |
Table IX. Learned policy selection via Jev-based predictor. First row reports its throughput gap compared to the best policy returned by full calibration (F.C.) SpecSearch. Last two rows are speedup over baseline vLLM runs.
KV cache precision. At , the Jev-based model selects AR with fp8 KV for the 256-token RTX 5090 workload and for the other 11 workloads. These choices reflect the load-dependent behavior observed in Section III: longer drafts remain beneficial at low concurrency, whereas shorter drafts or AR remain faster as concurrency increases.
Prediction errors. The Jev-based predictor selects a configuration within 5% of the best configuration’s throughput for 38 of the 48 workloads. Most of the larger prediction errors occur on H100 at : the predictor selects with fp8 KV for all output lengths from 256 to 2048 tokens. For these workloads, the best-performing configurations use on H100. The resulting throughput gaps range from 13.73% to 21.64%. These results reveal a limitation at moderate concurrency on H100, where the predictor favors shorter drafts than the measured throughput optimum.
Ablations
Memory cost of speculation. Figure 2(c,Figure 2d) and Table X quantify the memory cost of speculation described in Observation 3. At equal KV precision, vLLM SD retains 0.73× the AR KV cache capacity on the RTX 5090 under a 15.7 GiB budget and 0.68× on the MI300X under a 16 GB budget. Despite its smaller KV cache capacity, vLLM SD delivers 3.5–3.9× the AR throughput at concurrency 1–4. This advantage disappears by concurrency 64, where the throughput ratio falls to 0.93–0.97×.
| policy | RTX 5090 capacity (tokens) | RTX 5090 max conc. (queries) | MI300X capacity (tokens) | MI300X max conc. (queries) |
| AR, fp16 KV | 54,800 | 13.4× | 49,632 | 12.1× |
| AR, fp8 KV | 109,168 | 26.7× | 99,264 | 24.2× |
| vLLM SD, , fp16 KV | 40,000 | 9.8× | 33,808 | 8.3× |
| vLLM SD, , fp8 KV | 79,600 | 19.4× | 67,632 | 16.5× |
| Concurrency retention rate | 0.73 | 0.68 |
Table X. Relationship between KV cache capacity, in tokens, and maximum concurrency. The workload has across runs.
Generation length sensitivity. An fp8 KV cache doubles the KV cache capacity of both decoding methods. For short outputs, however, it changes throughput by at most 4%. The
| AR | vLLM SD | SpecSearch | Improvement vs. best AR | Improvement vs. vLLM SD | ||
| 256 | 8 | 0.86 | 3.04 | 2.69 | 3.14× | 0.89× |
| 256 | 32 | 2.97 | 6.42 | 6.27 | 2.11× | 0.98× |
| 256 | 64 | 4.99 | 7.77 | 8.74 | 1.75× | 1.12× |
| 256 | 128 | 7.48 | 8.49 | 9.98 | 1.33× | 1.18× |
| 1,024 | 8 | 0.83 | 2.88 | 2.87 | 3.47× | 1.00× |
| 1,024 | 32 | 2.59 | 5.44 | 5.18 | 2.00× | 0.95× |
| 1,024 | 64 | 3.96 | 6.26 | 6.49 | 1.64× | 1.04× |
| 1,024 | 128 | 5.25 | 6.71 | 7.56 | 1.44× | 1.13× |
| 2,048 | 8 | 0.83 | 2.81 | 2.69 | 3.24× | 0.96× |
| 2,048 | 32 | 2.24 | 5.01 | 5.48 | 2.45× | 1.09× |
| 2,048 | 64 | 3.36 | 5.89 | 5.74 | 1.71× | 0.97× |
| 2,048 | 128 | 3.84 | 6.31 | 6.87 | 1.79× | 1.09× |
| Average | 2.17× | 1.03× |
Table XI. Throughput (K tokens/s) with target model quantization (fp8) on the MI300X. SpecSearch uses a dynamic draft-length mapping with the fp8 target. The improvement columns report SpecSearch throughput relative to the vLLM baseline. Bold: highest mean throughput per workload.
additional capacity becomes useful as outputs grow longer. For AR decoding on the RTX 5090, the fp8-to-fp16 throughput ratio rises from 1.08× at 512 generated tokens to 1.73× at 16,384 tokens (Figure 4). The MI300X exhibits the same trend.
Figure 4. AR decoding: fp8-to-fp16 KV throughput ratio versus generated tokens per request, with concurrency 48.
Limitations
This study focuses on a single target model and a single diffusion-based drafter. Although the proposed cost model is portable across model architectures, extending it to alternative SD methods, such as EAGLE [9], [10] and Medusa [8], requires adapting the terms that characterize drafter-specific costs. The empirical results remain limited to the chosen DFlash drafter.
SpecSearch supports dynamic draft-length policies by deriving them from calibration runs over static draft lengths. However, the learned Jev-based predictor currently selects only static configurations. Incorporating calibration measurements for dynamic draft-length policies into the training dataset could extend the predictor to support dynamic policy selection. We leave this as future work.
The learned Jev-based predictor also exhibits performance gaps on certain workload–GPU combinations, most notably at moderate concurrency on H100, where it favors shorter drafts than the measured throughput optimum. Future work could improve prediction accuracy by expanding the training data across a broader range of GPU architectures and workloads, and by incorporating more informative hardware and workload features into the decision prompts.
Conclusion
Speculative decoding has no universally optimal configuration, as its performance depends on workload characteristics and GPU hardware. For example, on the RTX 5090, its speedup over AR drops from at low concurrency to at high concurrency when generation length shortens. To address this challenge, SpecSearch combines an analytical cost model that narrows the deployment search space with two complementary policy-selection approaches. full calibration SpecSearch evaluates candidate configurations on the deployment GPU, jointly optimizing KV-cache and model-weight precision, draft length, and whether to enable speculation. It achieves average throughput improvements of over AR and over vLLM SD across four GPUs.
Using the calibration dataset, SpecSearch fine-tunes a Jev-based optimal policy predictor. Across four held-out GPUs, it maintains a minimal throughput gap of to the best-performing configurations returned by full calibration SpecSearch. The learned policy predictor achieves the throughput of AR and that of vLLM SD, retaining of the throughput achieved by full calibration SpecSearch.
Together, these approaches enable hardware-aware policy selection across diverse workloads and deployment conditions, improving inference-serving efficiency on resource-constrained GPUs.
References
Appendix A: SpecSearch with full calibration and Results
Cost-model initialization
In Equation (1), the weight term and the per-token KV cost follow from the model architecture. The constant overheads , , and in Equation (2) depend on the GPU and software stack. For MI300X and RTX 5090, we measure them from vLLM’s startup memory breakdown with , interpolating linearly in . This gives of 1.14/1.25 GiB (AR/SD) on the MI300X and 0.35/0.44 GiB on the RTX 5090. We define compute factors on the MI300X and on the RTX 5090 in Eq. (5). Zero-shot policy selection does not use deployment GPU startup measurements or fitted runtime overheads.
Appendix B: SpecSearch with Learning-Based Policy Selectors
Data and features. Each deployment GPU is excluded from the three-GPU source dataset. Each source contributes 96 configuration–workload points: eight static choices at 12 workloads, with throughput aggregated over the available measurement seeds. For IDW, the shallow tree, and TabICL, numerical deployment metadata features encode memory budget, physical memory capacity, submitted request count, and whether the software platform uses ROCm. Workload features encode normalized log concurrency and output cap. Candidate features encode AR versus SD, draft length, and KV precision. The compact model uses only workload and candidate features. Jev receives these settings as text, together with the recorded attention backend, weight precision, and software versions where available.
IDW and shallow tree. IDW predicts log-throughput with weights , where is squared Euclidean distance in the fixed feature representation. It fits no model parameters. The tree uses maximum depth 2 and random seed 0. Training labels are the fastest observed configurations grouped by source GPU and workload.
TabICL. We use the pretrained TabICL regressor with eight estimators, GPU inference, and the trial seed as its random seed. Inputs are the numerical features above and labels are log-throughput. Source measurements condition its predictions. No gradient fine-tuning of the pretrained model is performed.
Compact numerical model. Separate workload and candidate encoders feed a shared scoring head, with hidden width 32 and GELU activations. Training minimizes centered log-throughput mean squared error plus 0.1 times a within-workload soft-ranking cross-entropy term, with temperature 0.1. AdamW uses learning rate 0.003, weight decay 0.01, and seed 0. Source-only validation leaves out each source GPU in turn and compares 300 versus 900 training steps and ridge-correction penalties , using three random 16-point supports on each validation GPU. The final model is refit on all three sources. All four folds select 900 steps. The source-validation correction penalty is 0.1 for RTX 5090 and 1 for the other targets. The zero-shot evaluation uses the source-trained model without a deployment-GPU residual correction.
Jev decision model. Both variants use Qwen3-4B with an eight-choice answer readout. Prompts contain the workload and the deployment settings listed above, together with up to 16 retrieved measurements per source GPU, selected by workload proximity without ranking by throughput. The source-tuned variant trains rank-8 LoRA adapters on attention query, key, value, and output projections (, dropout 0.05), together with a zero-initialized linear correction to the answer logits. Training uses classification cross-entropy, AdamW with learning rate and weight decay 0.01, effective batch size 8, gradient clipping at 1, and seed 0. Execution uses FP32 with SDPA and an 8,192-token prompt limit.
During training on source GPUs, prediction tasks simulate 0, 4, 8, or 16 observations from a source GPU treated as the query device, excluding query–workload outcomes from prediction inputs. These observations come entirely from the source dataset. Evaluation on the deployment GPU uses zero-shot policy selection. One supplied-value reading task accompanies every four prediction tasks; answer order is shuffled consistently with the labels. One of the three source GPUs is reserved for development validation, with training on the other two. Checkpoints at 0, 25, 50, 100, and 200 steps are compared by validation throughput loss, then the chosen duration is used to refit from initialization on all three sources. The selected durations are 100 steps for RTX 5090 and MI300X and 200 for H100 and L40. The frozen variant retains the initial base model and zero correction. Supplied-value reading checks are recorded separately and are not uniformly perfect.
Selected policies. Table XVI reports the saved static choices from zero-shot policy selection for IDW and source-tuned Jev in the format of Table V. Table XVII gives dynamic mappings constructed from their saved candidate scores, using the same batch ranges as Table VI. For each GPU, output cap, KV precision, and concurrency anchor , we maximize the score over ; exact ties prefer smaller . The anchors define ranges 1–20, 21–48, 49–96, and 97–128. AR is excluded from this within-engine mapping; choosing AR or a KV precision remains a separate startup decision.
| Method | GPU | 1–20 | 21–48 | 49–96 | 97–128 | |
| IDW | MI300X | 256 | 7/7 | 7/7 | 7/7 | 3/3 |
| 1,024 | 7/7 | 7/7 | 7/7 | 3/3 | ||
| 2,048 | 7/7 | 7/7 | 7/7 | 3/3 | ||
| IDW | RTX 5090 | 256 | 15/15 | 3/3 | 3/3 | 3/3 |
| 1,024 | 15/15 | 3/3 | 3/3 | 3/3 | ||
| 2,048 | 15/15 | 3/3 | 3/3 | 3/3 | ||
| IDW | H100 | 256 | 15/15 | 3/3 | 3/3 | 3/3 |
| 1,024 | 15/15 | 3/3 | 3/3 | 3/3 | ||
| 2,048 | 15/15 | 3/3 | 3/3 | 3/3 | ||
| IDW | L40 | 256 | 15/15 | 3/3 | 3/3 | 3/3 |
| 1,024 | 15/15 | 7/3 | 3/3 | 3/3 | ||
| 2,048 | 15/15 | 7/7 | 3/3 | 3/3 | ||
| Jev | MI300X | 256 | 15/15 | 3/3 | 3/3 | 3/3 |
| 1,024 | 15/15 | 3/3 | 3/3 | 3/3 | ||
| 2,048 | 15/15 | 3/7 | 3/3 | 3/3 | ||
| Jev | RTX 5090 | 256 | 15/15 | 3/3 | 3/3 | 3/3 |
| 1,024 | 15/15 | 3/3 | 3/3 | 3/3 | ||
| 2,048 | 15/15 | 3/3 | 3/3 | 3/3 | ||
| Jev | H100 | 256 | 15/15 | 3/3 | 3/3 | 3/3 |
| 1,024 | 15/15 | 3/3 | 3/3 | 3/3 | ||
| 2,048 | 15/15 | 3/3 | 3/3 | 3/3 | ||
| Jev | L40 | 256 | 15/15 | 3/3 | 3/3 | 3/3 |
| 1,024 | 15/15 | 3/3 | 3/3 | 3/3 | ||
| 2,048 | 15/15 | 3/7 | 3/3 | 3/3 |
Table XVII. Predicted dynamic draft-length mappings of IDW and source-tuned Jev. Each entry reports for FP16/FP8 KV, respectively, at the indicated effective-batch range.
IDW uses predicted log-throughput; Jev uses the eight-choice answer logits restricted to the three speculative actions at the specified KV precision. Neither construction uses target outcomes. These tables characterize the dynamic mappings implied by the static candidate scores. The performance evaluation is restricted to static configuration selection; it does not measure the throughput of these dynamic mappings.
| H100 vLLM AR fp16 | H100 vLLM AR fp8 | H100 vLLM SD fp16 | H100 vLLM SD fp8 | H100 SpecSearch fp16 | H100 SpecSearch fp8 | H100 Improvement vs. best AR | H100 Improvement vs. best SD | L40 AR fp16 | L40 AR fp8 | L40 vLLM SD fp16 | L40 vLLM SD fp8 | L40 SpecSearch fp16 | L40 SpecSearch fp8 | L40 Improvement vs. best AR | L40 Improvement vs. best SD | ||
| 8 | 256 | 1.20 | 1.20 | 4.19 | 4.16 | 4.19 | 4.18 | 3.48× | 1.00× | 0.61 | 0.62 | 1.85 | 1.76 | 1.85 | 1.82 | 2.94× | 0.99× |
| 32 | 256 | 3.88 | 3.87 | 7.46 | 7.51 | 9.53 | 9.56 | 2.47× | 1.27× | 1.97 | 2.05 | 2.16 | 2.37 | 3.65 | 3.65 | 1.78× | 1.54× |
| 64 | 256 | 6.90 | 6.83 | 8.33 | 8.21 | 12.57 | 12.45 | 1.80× | 1.49× | 3.34 | 3.54 | 2.42 | 2.46 | 4.23 | 4.37 | 1.23× | 1.77× |
| 128 | 256 | 11.57 | 11.74 | 9.15 | 9.03 | 14.20 | 13.96 | 1.21× | 1.55× | 5.06 | 5.44 | 2.57 | 2.61 | 4.88 | 4.93 | 1.00× | 2.09× |
| 8 | 1,024 | 1.19 | 1.20 | 3.92 | 3.91 | 3.92 | 3.91 | 3.26× | 1.00× | 0.59 | 0.61 | 1.78 | 1.77 | 1.78 | 1.79 | 2.95× | 1.01× |
| 32 | 1,024 | 3.33 | 3.33 | 7.07 | 6.83 | 8.53 | 8.22 | 2.47× | 1.16× | 1.73 | 1.83 | 2.17 | 2.20 | 3.43 | 3.44 | 1.88× | 1.56× |
| 64 | 1,024 | 5.35 | 5.40 | 7.78 | 7.48 | 10.60 | 10.44 | 1.93× | 1.34× | 2.60 | 2.86 | 2.31 | 2.50 | 3.92 | 3.94 | 1.38× | 1.57× |
| 128 | 1,024 | 8.90 | 9.24 | 8.20 | 8.27 | 12.43 | 13.75 | 1.49× | 1.66× | 3.96 | 4.66 | 2.28 | 2.60 | 4.00 | 4.73 | 1.01× | 1.82× |
| 8 | 2,048 | 1.16 | 1.18 | 3.84 | 3.74 | 3.84 | 3.74 | 3.25× | 1.00× | 0.58 | 0.59 | 1.70 | 1.82 | 1.70 | 1.82 | 3.06× | 1.00× |
| 32 | 2,048 | 2.75 | 2.74 | 6.81 | 6.52 | 8.07 | 7.68 | 2.80× | 1.13× | 1.45 | 1.55 | 2.12 | 2.22 | 3.17 | 3.33 | 2.16× | 1.50× |
| 64 | 2,048 | 4.14 | 4.24 | 7.35 | 6.97 | 9.03 | 8.58 | 2.13× | 1.23× | 2.04 | 2.29 | 2.26 | 2.30 | 3.55 | 3.55 | 1.55× | 1.54× |
| 128 | 2,048 | 7.13 | 7.42 | 7.83 | 7.65 | 10.49 | 12.00 | 1.62× | 1.53× | 3.31 | 3.67 | 2.26 | 2.55 | 3.61 | 4.51 | 1.23× | 1.77× |
| Average | 2.33× | 1.28× | 1.85× | 1.51× |
Table XII. Throughput (K tokens/s) of decoding policies at 15.7 GiB device budget. Bold: highest mean value per workload. Underlined: the policy selected by SpecSearch. Within each method (AR, vLLM SD, SpecSearch), the darker shading marks fp8 KV in contrast to fp16 KV cache.
| H100 vLLM AR fp16 | H100 vLLM AR fp8 | H100 vLLM SD fp16 | H100 vLLM SD fp8 | H100 SpecSearch fp16 | H100 SpecSearch fp8 | H100 Improvement vs. best AR | H100 Improvement vs. best SD | L40 AR fp16 | L40 AR fp8 | L40 Baseline SD fp16 | L40 Baseline SD fp8 | L40 SpecSearch fp16 | L40 SpecSearch fp8 | L40 Improvement vs. best AR | L40 Improvement vs. best SD | ||
| 8 | 256 | 1.00× | 1.00× | 3.49× | 3.48× | 3.49× | 3.49× | 3.49× | 1.00× | 1.00× | 1.01× | 3.02× | 2.87× | 3.02× | 2.97× | 2.94× | 0.98× |
| 32 | 256 | 1.00× | 1.00× | 1.92× | 1.94× | 2.46× | 2.47× | 2.47× | 1.27× | 1.00× | 1.04× | 1.09× | 1.20× | 1.85× | 1.85× | 1.78× | 1.54× |
| 64 | 256 | 1.00× | 0.99× | 1.21× | 1.19× | 1.83× | 1.81× | 1.81× | 1.49× | 1.00× | 1.06× | 0.73× | 0.74× | 1.27× | 1.31× | 1.23× | 1.78× |
| 128 | 256 | 1.00× | 1.01× | 0.79× | 0.78× | 1.23× | 1.21× | 1.21× | 1.55× | 1.00× | 1.08× | 0.51× | 0.51× | 0.97× | 0.98× | 1.00× | 2.09× |
| 8 | 1,024 | 1.00× | 1.03× | 3.33× | 3.27× | 3.31× | 3.27× | 3.23× | 1.00× | 1.00× | 1.03× | 2.97× | 2.95× | 2.97× | 2.99× | 2.91× | 1.01× |
| 32 | 1,024 | 1.00× | 0.99× | 2.09× | 2.05× | 2.51× | 2.46× | 2.46× | 1.18× | 1.00× | 1.06× | 1.27× | 1.28× | 1.99× | 2.01× | 1.90× | 1.57× |
| 64 | 1,024 | 1.00× | 1.01× | 1.44× | 1.39× | 1.97× | 1.94× | 1.92× | 1.34× | 1.00× | 1.08× | 0.88× | 0.95× | 1.50× | 1.51× | 1.40× | 1.58× |
| 128 | 1,024 | 1.00× | 1.04× | 0.93× | 0.93× | 1.40× | 1.54× | 1.49× | 1.66× | 1.00× | 1.18× | 0.58× | 0.66× | 1.01× | 1.20× | 1.01× | 1.82× |
| 8 | 2,048 | 1.00× | 1.02× | 3.32× | 3.16× | 3.32× | 3.16× | 3.26× | 1.00× | 1.00× | 1.05× | 2.97× | 3.13× | 2.97× | 3.13× | 2.99× | 1.00× |
| 32 | 2,048 | 1.00× | 0.98× | 2.46× | 2.37× | 2.91× | 2.79× | 2.79× | 1.13× | 1.00× | 1.06× | 1.47× | 1.51× | 2.18× | 2.23× | 2.11× | 1.48× |
| 64 | 2,048 | 1.00× | 1.00× | 1.75× | 1.67× | 2.15× | 2.04× | 2.15× | 1.23× | 1.00× | 1.12× | 1.13× | 1.13× | 1.75× | 1.76× | 1.57× | 1.56× |
| 128 | 2,048 | 1.00× | 1.02× | 1.08× | 1.04× | 1.45× | 1.64× | 1.60× | 1.51× | 1.00× | 1.09× | 0.68× | 0.76× | 1.09× | 1.36× | 1.25× | 1.79× |
| Average | 2.32× | 1.28× | 1.84× | 1.52× |
Table XIII. Speedup of decoding policies at 15.7 GiB device budget. Bold: highest mean value per workload. Underlined: the policy selected by SpecSearch. Within each method (AR, vLLM SD, SpecSearch), the darker shading marks fp8 KV in contrast to fp16 KV cache.
| fp16 | fp8 | fp16 | fp8 | fp16 | fp8 | ||
| 8 | 256 | 3.49 | 3.75 | 4.32 | 4.63 | 4.26 | 4.55 |
| 32 | 256 | 8.15 | 8.90 | 8.35 | 9.12 | 7.18 | 8.14 |
| 64 | 256 | 10.69 | 11.30 | 10.09 | 11.28 | 8.45 | 8.93 |
| 128 | 256 | 13.15 | 14.21 | 12.35 | 12.53 | 9.38 | 10.13 |
| 8 | 1,024 | 2.94 | 3.47 | 3.68 | 4.31 | 3.58 | 4.18 |
| 32 | 1,024 | 7.01 | 8.15 | 7.03 | 8.13 | 6.09 | 6.89 |
| 64 | 1,024 | 8.46 | 9.65 | 8.55 | 9.89 | 7.21 | 8.53 |
| 128 | 1,024 | 9.54 | 11.92 | 9.36 | 10.75 | 8.14 | 10.13 |
| 8 | 2,048 | 2.64 | 3.19 | 3.38 | 4.01 | 3.42 | 4.03 |
| 32 | 2,048 | 5.80 | 6.70 | 6.39 | 6.79 | 5.50 | 6.47 |
| 64 | 2,048 | 6.88 | 8.20 | 6.86 | 7.76 | 6.39 | 7.48 |
| 128 | 2,048 | 7.62 | 9.04 | 7.71 | 8.58 | 6.90 | 8.91 |
Table XIV. MI300X throughput (K tokens/s) for every static draft length and KV-cache precision at the 15.7 GiB budget. Bold: highest mean throughput at that workload.
| fp16 | fp8 | fp16 | fp8 | fp16 | fp8 | ||
| 256 | 8 | 1.86× | 2.01× | 2.31× | 2.47× | 2.27× | 2.42× |
| 32 | 1.52× | 1.67× | 1.56× | 1.70× | 1.39× | 1.58× | |
| 64 | 1.34× | 1.41× | 1.26× | 1.41× | 1.03× | 1.08× | |
| 128 | 1.21× | 1.31× | 1.13× | 1.15× | 0.78× | 0.85× | |
| 1,024 | 8 | 1.64× | 1.93× | 2.05× | 2.41× | 1.99× | 2.34× |
| 32 | 1.72× | 2.02× | 1.72× | 1.97× | 1.57× | 1.76× | |
| 64 | 1.42× | 1.60× | 1.43× | 1.64× | 1.22× | 1.44× | |
| 128 | 1.27× | 1.57× | 1.25× | 1.39× | 0.98× | 1.22× | |
| 2,048 | 8 | 1.44× | 1.72× | 1.89× | 2.21× | 1.98× | 2.32× |
| 32 | 1.80× | 1.95× | 1.91× | 1.98× | 1.77× | 2.01× | |
| 64 | 1.39× | 1.62× | 1.40× | 1.53× | 1.38× | 1.60× | |
| 128 | 1.30× | 1.52× | 1.34× | 1.40× | 1.13× | 1.46× |
Table XV. MI300X wall-clock speedup over AR decoding with FP16 KV for every static draft length and KV-cache precision. Bold: highest mean speedup at that workload.
| Method | GPU | |||||
| IDW | MI300X | 256 | , fp8 | , fp8 | , fp8 | , fp8 |
| 1,024 | , fp8 | , fp8 | , fp8 | , fp8 | ||
| 2,048 | , fp8 | , fp8 | , fp8 | , fp8 | ||
| IDW | RTX 5090 | 256 | , fp16 | , fp8 | , fp8 | AR, fp8 |
| 1,024 | , fp8 | , fp8 | , fp8 | , fp8 | ||
| 2,048 | , fp8 | , fp8 | , fp8 | , fp8 | ||
| IDW | H100 | 256 | , fp16 | , fp8 | , fp8 | , fp8 |
| 1,024 | , fp8 | , fp8 | , fp8 | , fp8 | ||
| 2,048 | , fp8 | , fp8 | , fp8 | , fp8 | ||
| IDW | L40 | 256 | , fp16 | , fp8 | , fp8 | , fp8 |
| 1,024 | , fp16 | , fp8 | , fp8 | , fp8 | ||
| 2,048 | , fp8 | , fp8 | , fp8 | , fp8 | ||
| Jev | MI300X | 256 | , fp16 | , fp8 | , fp8 | , fp8 |
| 1,024 | , fp16 | , fp8 | , fp8 | , fp8 | ||
| 2,048 | , fp8 | , fp8 | , fp8 | , fp8 | ||
| Jev | RTX 5090 | 256 | , fp8 | , fp8 | , fp8 | AR, fp8 |
| 1,024 | , fp16 | , fp8 | , fp8 | , fp8 | ||
| 2,048 | , fp8 | , fp8 | , fp8 | , fp8 | ||
| Jev | H100 | 256 | , fp16 | , fp8 | , fp8 | , fp8 |
| 1,024 | , fp8 | , fp8 | , fp8 | , fp8 | ||
| 2,048 | , fp8 | , fp8 | , fp8 | , fp8 | ||
| Jev | L40 | 256 | , fp16 | , fp8 | , fp8 | , fp8 |
| 1,024 | , fp8 | , fp8 | , fp8 | , fp8 | ||
| 2,048 | , fp8 | , fp8 | , fp8 | , fp8 |
Table XVI. Policies selected by IDW and source-tuned Jev at each workload: AR vs. vLLM SD and KV precision.
References
- [1]Y. Leviathan, M. Kalman, and Y. Matias, “Fast Inference from Transformers via Speculative Decoding,” arXiv:2211.17192, 2022.arxiv.org/abs/2211.17192
- [2]C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper, “Accelerating Large Language Model Decoding with Speculative Sampling,” arXiv:2302.01318, 2023.arxiv.org/abs/2302.01318
- [3]J. Chen, Y. Liang, and Z. Liu, “DFlash: Block Diffusion for Flash Speculative Decoding,” arXiv:2602.06036, 2026.arxiv.org/abs/2602.06036
- [4]M. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the 29th Symposium on Operating Systems Principles. ACM, Oct. 2023, pp. 611–626. [Online]. Available: http://dx.doi.org/10.1145/3600006.3613165
- [5]Y. Li and J. Wagle, “LLM-as-Jey: LLMs Are Already Jey-Style Decision Models—When and How to Fine-Tune Them,” arXiv preprint arXiv:2610.02076, 2026.arxiv.org/abs/2610.02076
- [6]Advanced Micro Devices, Inc., “AMD Instinct MI300X Accelerator Data Sheet,” https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/data-sheets/amd-instinct-mi300x-data-sheet.pdf, 2023, accessed: 2026-10-21.
- [7]NVIDIA Corporation, “NVIDIA RTX Blackwell GPU Architecture,” https://images.nvidia.com/aem-dam/Solutions/geforce/blackwell/nvidia-rtx-blackwell-gpu-architecture.pdf, 2025, accessed: 2026-10-21.
- [8]T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao, “Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads,” arXiv:2401.10774, 2024.arxiv.org/abs/2401.10774
- [9]Y. Li, F. Wei, C. Zhang, and H. Zhang, “EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty,” arXiv:2401.15077, 2024.arxiv.org/abs/2401.15077
- [10]——, “EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test,” arXiv:2503.01840, 2025.arxiv.org/abs/2503.01840
- [11]L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y. Sheng, “SGLang: Efficient Execution of Structured Language Model Programs,” in Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2312.07104
- [12]S. Yildirim and D. Chen, “SpecMemo: Speculative Decoding is in Your Pocket,” arXiv:2506.01986, 2025.arxiv.org/abs/2506.01986
- [13]E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers,” arXiv:2210.17323, 2022.arxiv.org/abs/2210.17323
- [14]G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “SmoothQuant: Accurate and efficient post-training quantization for large language models,” in Proceedings of the 40th International Conference on Machine Learning, vol. 202, 2023, pp. 38 087–38 099.arxiv.org/abs/2211.10438
- [15]J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration,” arXiv:2306.00978, 2023.arxiv.org/abs/2306.00978
- [16]Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V. Braverman, B. Chen, and X. Hu, “KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache,” arXiv:2402.02750, 2024.DOI
- [17]C. Hooper, S. Kim, H. Mohammadzaheh, M. W. Mahoney, Y. S. Shao, K. Keutzer, and A. Gholami, “KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization,” arXiv:2401.18079, 2024.arxiv.org/abs/2401.18079
- [18]Y. He, L. Zhang, W. Wu, J. Liu, H. Zhou, and B. Zhuang, “ZipCache: Accurate and efficient KV cache quantization with salient token identification,” in Advances in Neural Information Processing Systems, vol. 37, 2024.arxiv.org/abs/2405.14256
- [19]X. Li, Z. Xing, Y. Li, L. Qu, H.-L. Zhen, Y. Yao, W. Liu, S. J. Pan, and M. Yuan, “KVTuner: Sensitivity-aware layer-wise mixed-precision KV cache quantization for efficient and nearly lossless LLM inference,” in Proceedings of the 42nd International Conference on Machine Learning, vol. 267, 2025, pp. 36 451–36 485.DOI
- [20]Z. Chen, A. May, R. Svirschevski, Y. Huang, M. Ryabinin, Z. Jia, and B. Chen, “Sequoia: Scalable and Robust Speculative Decoding,” in Advances in Neural Information Processing Systems, vol. 37, 2024, pp. 129 531–129 563.arxiv.org/abs/2402.12374
- [21]R. Sadhukhan, J. Chen, Z. Chen, V. Tiwari, R. Lai, J. Shi, I. E.-H. Yen, A. May, T. Chen, and B. Chen, “MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding,” in The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2408.11049
- [22]X. Liu, J. Park, L. Hu, W. Kwon, Z. Li, C. Zhang, K. Du, X. Mo, K. You, A. Cheung, Z. Deng, I. Stoica, and H. Zhang, “TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput,” arXiv:2406.14066, 2024.arxiv.org/abs/2406.14066
- [23]Z. Li, Z. Chen, R. Delacourt, G. Oliaro, Z. Wang, Q. Chen, S. Lin, A. Yang, Z. Zhang, Z. Chen, Y.-H. Lai, X. Cheng, X. Miao, and Z. Jia, “AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding,” in Proceedings of the 21st European Conference on Computer Systems, 2026.arxiv.org/abs/2501.12162
- [24]G. X. Yu, Y. Gao, P. Golikov, and G. Pekhimenko, “Habitat: A Runtime-Based Computational Performance Predictor for Deep Neural Network Training,” in 2021 USENIX Annual Technical Conference (USENIX ATC 21), 2021. [Online]. Available: https://www.usenix.org/conference/atc21/presentation/yu
- [25]S. Lee, A. Phanishayee, and D. Mahajan, “Forecasting GPU Performance for Deep Learning Training and Inference,” in Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2025. [Online]. Available: https://arxiv.org/abs/2407.13853
- [26]H. Zhang, A. Ning, R. Prabhakar, and D. Wentzlaff, “LLMCompass: Enabling Efficient Hardware Design for Large Language Model Inference,” in 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), 2024, pp. 1080–1096.DOI
- [27]S. Williams, A. Waterman, and D. Patterson, “Roofline: an insightful visual performance model for multicore architectures,” Communications of the ACM, vol. 52, no. 4, pp. 65–76, Apr. 2009. [Online]. Available: http://dx.doi.org/10.1145/1498765.1498785
- [28]NVIDIA, “GeForce RTX 5090 Specifications,” https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/, 2025.
- [29]AMD, “AMD Instinct MI300X Accelerators,” https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html, 2023.
- [30]NVIDIA, “NVIDIA H100 Tensor Core GPU Datasheet,” https://resources.nvidia.com/en-us-gpu-resource/h100-datasheet-24306, 2024.
- [31]NVIDIA, “NVIDIA L40 GPU Datasheet,” https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/support-guide/NVIDIA-L40-Datasheet-January-2023.pdf, 2023.
- [32]A. Yang et al., “Qwen3 Technical Report,” arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
- [33]K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training Verifiers to Solve Math Word Problems,” arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
- [34]H. Lightman et al., “Let’s Verify Step by Step,” arXiv:2305.20050, 2023.
- [35]J. Austin et al., “Program Synthesis with Large Language Models,” arXiv:2108.07732, 2021.arxiv.org/abs/2108.07732
- [36]L. Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” arXiv:2306.05685, 2023.arxiv.org/abs/2306.05685
- [37]M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan et al., “Evaluating Large Language Models Trained on Code,” arXiv:2107.03374, 2021.arxiv.org/abs/2107.03374



