betaFeedback

WorksComputer Science

SpecSearch: Towards Hardware-Aware Efficient Speculative Decoding

Abstract

Speculative decoding (SD) accelerates large language model (LLM) inference, but its optimal configuration depends jointly on workload characteristics, GPU hardware, and available memory. For example, on the RTX~5090, the state-of-the-art speculative drafter DFlash achieves 3.50×3.50\times the throughput of autoregressive (AR) decoding at low concurrency, but only 0.76×0.76\times at high concurrency with short outputs. We present SpecSearch, a hardware-aware search engine that jointly optimizes decoding strategy, draft length, and precision across workload–GPU pairs. Its analytical cost model captures memory and compute constraints to identify feasible configurations and narrow the search space for throughput optimization. Guided by this model, calibration measurements are collected from multiple GPUs. Using the measurements, a Jev-based predictor learns to select high-throughput decoding configurations on held-out GPUs without requiring deployment-GPU calibration. The predictor is parameter-efficiently trained on calibration data, while adapting a frozen Qwen3-4B using LoRA adapters and a linear correction layer. Across NVIDIA RTX~5090, H100, L40 and AMD MI300X GPUs, SpecSearch with full calibration achieves an average of 2.01×2.01\times the throughput of AR and 1.36×1.36\times the throughput of SD, both served by vLLM. The proposed Jev-based predictor closely matches this search performance with 1.30×1.30\times throughput improvement over SD with vLLM, while eliminating the need for held-out GPU calibration. Additionally, it retains 97.36%97.36\% of the throughput achieved by SpecSearch that utilizes full calibration.

Introduction

Speculative decoding (SD) [1, 2] reduces autoregressive (AR) decoding latency: an inexpensive drafter model proposes several draft tokens, and the target model verifies them in parallel in a single forward pass. Verification makes generation lossless, so the output distribution equals that of the target model. State-of-the-art diffusion-LLM drafters such as DFlash [3] propose a whole block of draft tokens in a single drafter forward pass. Although advances in drafter architectures are rapidly improving speculative decoding efficiency, the achievable speedup remains highly dependent on deployment-time decisions: whether to load a drafter model at all, how many tokens to draft (i.e., the draft length) and at which precision to store the model weights and the key–value (KV) cache. These decisions are particularly critical on memory-constrained GPUs. Model weights and speculation buffers consume a major portion of device memory. Consequently, the memory left for the KV cache determines how many requests the inference engine can serve concurrently. This limitation hinders efficient speculative decoding on memory-constrained GPUs.

Modern inference engines such as vLLM [4] run kernels optimized for the deployment hardware. They further enhance aggregate throughput with continuous batching, which admits waiting requests into the running batch as others finish. Additionally, they reduce memory footprint through paging KV cache. Nonetheless, they leave the algorithmic configuration of SD to the user: the draft length and the decoding precision are manual knobs, typically set once and reused across workloads and hardware. We observe that this practice loses substantial throughput, because the best decoding configuration, including whether to speculate at all, depends jointly on the workload and the hardware. Figure 2 illustrates this for Qwen3-4B with a DFlash drafter on a resource-constrained GPU. In Figure 2(a), with shorter text generation, vLLM SD is 3.50× faster than AR decoding at 2 concurrent requests but only 0.76× as fast at 64, because verification becomes compute-bound. In Figure 2(b), with longer text generation, vLLM SD with an fp8 KV cache is the fastest configuration at every concurrency from 32 to 128, ahead of AR decoding and of vLLM SD with an fp16 KV cache. The concurrency at which speculation stops paying off therefore depends on the output length, the memory budget, and the ridge point of the device, i.e., the ratio of its peak compute throughput to its memory bandwidth.

Figure 2. The best decoding configuration and its memory cost depend on the workload. (a) vLLM SD-to-AR throughput ratio with 256-token outputs and fp16 KV; the dashed line marks the critical batch size B⋆B^\star predicted by SpecSearch cost model (Section IV). (b) With longer outputs (2,048 tokens), vLLM SD with fp8 KV is fastest from concurrency 32 to 128. (c) With shorter outputs (up to 256 tokens), vLLM SD has 3.5–3.9× throughput at concurrency 1–4, but its advantage over AR disappears by concurrency 64. (d) The drafter use reduces KV cache capacity to 0.73× that of AR at equal KV precision.

We present SpecSearch, which automatically identifies the best-performing decoding configuration for a given GPU and workload in terms of throughput and wall-clock speedup. A decoding configuration comprises a set of deployment policies, including whether to enable speculation, setting a static or dynamic draft length based on effective batch size (concurrency), as well as the precision of model weights and the KV cache. We characterize each workload by the number of concurrent requests (BB), which we refer to interchangeably as concurrency or batch size, and their generation lengths. SpecSearch has two components:

First, an analytical cost model characterizes the device’s memory budget, memory bandwidth, and peak compute throughput. It determines whether a deployment configuration fits within the available memory and, if feasible, estimates the maximum concurrency (BB) that the KV cache can support without request preemption. It also identifies the critical batch size (B∗B^{*}), beyond which compute contention diminishes the throughput benefits of speculation.

Second, SpecSearch selects the most profitable decoding configuration for each workload on a given GPU through two complementary approaches. In full calibration mode, SpecSearch uses calibration measurements guided by the cost model to search the policy space and identify the best-performing configuration for each workload. These configurations are then stored in a lookup table before real-time inference deployment. This workflow is illustrated in Figure 1. We further use the measured calibration data to construct a training dataset for fine-tuning a Jev-based policy selector [5]. Alternatively, in learning-based mode, SpecSearch leverages this trained model to predict the optimal decoding configuration for each workload on a given GPU, eliminating the need for prior calibration on held-out GPUs.

Figure 1. Overview of SpecSearch with full calibration: Given the GPU and the workload (1), the proposed cost model discards deployment configurations that exceed the memory budget and determines the critical batch size B⋆B^\star of the device (2). Calibration on the deployment GPU runs AR and SD where quantization and optimal draft lengths kk are explored within vLLM (3). SpecSearch explores dynamic draft-length mapping for vLLM that sets kk from the effective batch size at every decoding step (4). The most profitable configuration for each workload is frozen into a policy lookup table to be reused during real-time inference (5).

Contributions.

  1. Defining a joint optimization space: We provide evidence that the optimal speculative decoding (SD) configuration depends both on the workload and the deployment GPU (Section III). SpecSearch introduces a hardware-aware cost model to narrow the search space for finding performance-optimal decoding policies.

  1. Validation and portability of cost model: SpecSearch’s cost model estimates total GPU memory footprint for LLM inference within 0.1% and 0.2% on the AMD MI300X [6] and NVIDIA RTX 5090 [7] GPUs, respectively. Furthermore, it estimates a maximum serviceable concurrency within 0.5% and 0.6% on the GPUs, respectively. The cost model also correctly predicts whether a workload fits a device budget under all evaluated configurations on the two GPUs.

  1. End-to-end performance gains: SpecSearch with full calibration achieves 2.16× and 1.68× the throughput of AR on RTX 5090 and MI300X, respectively, averaged across several workloads. Furthermore, it achieves 1.44× and 1.21× speedup over baseline SD served by vLLM, respectively. Across four different GPUs, it delivers an average 1.36× speedup over vLLM SD. The Jev-based predictor closely approaches the full calibration SpecSearch performance, retaining 97.36% of its throughput and achieving 1.30× the throughput of vLLM SD.

Speculative decoding. Draft-then-verify approach was introduced with an independent draft model [1, 2]. Medusa uses parallel decoding heads with tree attention [8]. EAGLE drafts at the level of target features [9]. EAGLE-3 drafts tokens from fused multi-layer target features [10]. State-of-the-art (SoTA) DFlash drafters propose a block of tokens in a single forward pass as a block-diffusion model conditioned on the target features [3]. All methods deliver lossless generation with respect to the target distribution. We use DFlash drafters for evaluating SD throughput.

Paged KV cache and memory-efficient speculation. vLLM [4] and SGLang [11] are LLM serving frameworks that employ paged KV-cache management to support efficient memory allocation and high-throughput serving. vLLM [4] introduces PagedAttention, which manages KV-cache entries in non-contiguous, fixed-size blocks, reducing memory fragmentation and enabling flexible KV-cache sharing and allocation across requests. Building on that, SpecSearch proposes a cost model that estimates the maximum serviceable concurrency a paged KV cache can support. Prior works on memory-efficient speculative decoding include SpecMemo [12], which optimizes runtime buffer allocation in Medusa [8] to improve single-request throughput and reduce memory footprint. However, it does not account for concurrency under paged KV-cache management or workload-dependent throughput.

Dynamic draft-length selection. vLLM and SGLang expose configurable speculative decoding parameters, including the draft length, which is the number of consecutive tokens proposed by the drafter. vLLM supports user-defined mappings from batch-size ranges to draft lengths. SGLang provides adaptive SD that adjusts draft length based on observed acceptance rates within a workload. However, neither determines an optimal draft length based on both the workload and deployment hardware. SpecSearch automates this process through GPU-specific calibration, selecting draft-length policies based on measured throughput and which precision to use. Thus, SpecSearch performs resource- and workload-aware dynamic draft length selection.

Quantization. Model-weight [13], [14], [15] and KV-cache quantization [16], [17], [18], [19] are established techniques for reducing the memory footprint of LLM inference. SpecSearch incorporates both model-weight precision and KV-cache precision as configurable dimensions of speculative decoding (SD), and quantifies their individual and combined effects on end-to-end performance.

Adapting speculation to workload and hardware. Existing works address complementary aspects of optimizing SD. Sequoia profiles hardware-dependent draft and verification costs to select speculative tree size and depth [20]. MagicDec [21] models the transition from memory-bound to compute-bound execution as batch size increases in self-speculative decoding, and uses this analysis to optimize speculation for long-context workloads. At the serving layer, TurboSpec controls draft lengths under changing load [22]. AdaServe constructs hardware-aware speculative decoding trees for per-request latency objectives [23]. Unlike prior work, SpecSearch addresses a broader deployment-level choice: whether to use AR or SD, which static or dynamic draft length to use, and which precision fits the deployment GPU’s resource constraints. The proposed cost model analyzes workloads and calibrates them on GPUs to select a workload- and GPU-specific decoding policy before serving.

Learning-based deployment optimization. Prior work predicts performance of deep learning workloads across hardware platforms to reduce the cost of hardware profiling. Habitat scales operation-level measurements from a source GPU to estimate training iteration time on other GPUs [24]. NeuSight predicts training and inference latency on GPUs using learned tile-level models constrained by hardware performance bounds [25]. For LLM inference, LLMCompass models hardware architectures and explores operator mappings and schedules to estimate latency and analyze performance–cost tradeoffs [26]. These works provide a foundation for making deployment decisions across GPUs. Building on this

direction, SpecSearch proposes training a policy selector on GPU performance measurements to select optimal decoding configurations. Recent Jev [5] is a lightweight decision model that predicts choices among predefined alternatives, enabling efficient learned decision-making. Unlike prior approaches, SpecSearch fine-tunes a Jev-based policy selector for predicting the optimal decoding configuration across LLM workloads and hardware.

Motivation

We preliminarily evaluate AR and SD using vLLM with FP16 and FP8 KV-cache precisions on an RTX 5090. We find that the best-performing configuration depends jointly on the workload and GPU.

Observation 1: The concurrency at which speculation loses its advantage depends on the workload and GPU. As concurrency increases, decoding shifts from memory-bound to compute-bound execution [27], reducing speculation’s benefit. The transition depends on the GPU’s compute-to-bandwidth ratio, which our cost model estimates through the critical batch size B⋆B^\star (Table I). SD throughput advantage for 256-token outputs drops from 3.50× at concurrency 2 to 1.21× at 32, then reverses to 0.76× at 64 (Figure 2(a)). We validate this finding on AMD MI300X GPU, where the SD gain is 1.13× at concurrency 32. For 2,048-token outputs, SD remains advantageous beyond concurrency 128 (Figure 2(b)).

Observation 2: KV quantization benefits long outputs. For 4,096-token outputs at concurrency 48, throughput reaches 8,479 tokens/s with SD and fp8 KV, versus 6,813 with SD and fp16 KV. This yields a 1.24× speedup, with nearly identical acceptance lengths (11.8 vs. 11.9). The benefit persists for 2,048-token outputs: fp8 KV improves SD throughput by 1.10–1.16× across concurrency levels 32–128 (Figure 2(b)). For outputs up to 256 tokens, however, the difference is at most 4% (Figure 2(c)). Thus, KV quantization primarily benefits long-output workloads where KV-cache capacity limits concurrency.

Observation 3: Speculation reduces KV-cache capacity. Drafter weights and runtime buffers consume memory otherwise available for KV caching. Under a 15.7 GiB budget on RTX 5090, vLLM SD retains only 0.73× the AR KV-cache capacity at equal KV precision (Figure 2(d)).

These observations motivate a joint search mechanism for optimal decoding policy selection across workloads and GPUs. SpecSearch combines an analytical cost model with device calibration to capture hardware-specific costs at run-time.

GPUGBβg\beta_g (TB/s)Φg\Phi_g (TF/s)predicted B⋆B^\star
MI300X1925.301,30730.8
RTX 5090321.79209.529.3

Table 1. Device parameters of SpecSearch cost model: memory capacity, peak theoretical bandwidth βg\beta_g, and peak dense FP16 matrix throughput Φg\Phi_g [6, 7]. B⋆B^\star denotes the GPU roofline knee point.

SpecSearch

SpecSearch searches for an optimal deployment policy for LLM inference that maximizes throughput and speedup on a given GPU. In full calibration mode, SpecSearch uses the cost model to define the policy search space and performs calibration runs to evaluate the policies. In learning-based mode, a fine-tuned Jev-based model infers the optimal policy on held-out GPUs without calibration.

Notation. GPU gg has nominal memory bandwidth βg\beta_g, peak fp16 compute throughput Φg\Phi_g (see Table I), and inference memory budget MgM_g. The target and drafter have Pt,PdP_t, P_d parameters and Lt,LdL_t, L_d layers. We denote the number of KV heads by HkvH_{kv}, head dimension by dd, hidden size by hh, and vocabulary size by VV. A policy θ\theta specifies draft length kk, target and drafter weight sizes bwt,bwd∈{2,1,0.5}b_w^t, b_w^d \in\{2, 1, 0.5\} bytes (fp16, fp8, int4), KV element size bkv∈{2,1}b_{kv} \in\{2, 1\} bytes (fp16, fp8), and execution mode (CUDA/HIP graphs or eager). A workload (B,ℓ)(B,\ell) specifies batch size BB and generation length ℓ\ell. Given prompt length S0S_0, the per-request context length is S=S0+ℓS = S_0 + \ell.

Cost Model

Memory model. Peak memory is

Mpeak(θ)=(Ptbwt+Pdbwd)+κSB+nchS0b+2(k+1)VBb+Cgraph+ρg,(1)M_{\mathrm{peak}}(\theta) = (P_t b_w^t + P_d b_w^d) + \kappa S B + n_c h S_0 b + 2(k+1)V B b + C_{\mathrm{graph}} + \rho_g, \tag*{(1)}

where the terms represent model weights, KV caches, the context buffer for ncn_c target layers, verification logits, CUDA/HIP graph memory (zero in eager mode), and runtime/allocator overhead, respectively. Here, bb is the activation element size in bytes, and κ=2(Lt+Ld)Hkvdbkv\kappa= 2(L_t + L_d)H_{kv}d b_{kv} is the KV-cache cost per context token. The drafter cache stores keys and values projected from target model hidden states.

Concurrency model. The available KV-cache capacity, measured in tokens, is

Ntok(θ)=Mg−W(θ)−Cact−Cgraph−ρgκ,(2)N_{\mathrm{tok}}(\theta) = \frac{M_g - W(\theta) - C_{\mathrm{act}} - C_{\mathrm{graph}} - \rho_g}{\kappa}, \tag*{(2)}

where W(θ)=Ptbwt+PdbwdW(\theta) = P_t b_w^t + P_d b_w^d and CactC_{\mathrm{act}} includes the context buffer and verification logits. Maximum supported concurrency is

Cmax⁡(θ,ℓ)=Ntok(θ)S.(3)C_{\max}(\theta,\ell) = \frac{N_{\mathrm{tok}}(\theta)}{S}. \tag*{(3)}

A configuration is feasible at required concurrency CreqC_{\mathrm{req}} if Ntok(θ)≥CreqSN_{\mathrm{tok}}(\theta) \ge C_{\mathrm{req}}S.

Throughput model. Following the roofline model [27], we approximate decode-step time as

tstep≈max⁡(W(θ)βg,B(k+1)ϕΦg),(4)t_{\mathrm{step}} \approx\max\left(\frac{W(\theta)}{\beta_g}, \frac{B(k+1)\phi}{\Phi_g}\right), \tag*{(4)}

where ϕ≈2Pt\phi\approx2P_t is target arithmetic per token. The terms capture weight reads and computation over B(k+1)B(k+1) tokens, respectively. At low concurrency, decoding is memory-bound, and speculation incurs approximately one AR verification step while accepting τ\tau draft tokens per step, yielding an approximate speedup of σˉ\bar{\sigma}. At high concurrency, decoding becomes compute-bound, reducing the approximate speedup to τˉ/(k+1)\bar{\tau}/(k+1).

Equating the two terms for AR decoding (W=PtbwtW = P_t b_w^t) gives the critical batch size

B⋆≈PtbwtβgηΦgϕ=ηbwt2Φgβg,(5)B^\star\approx\frac{P_t b_w^t}{\beta_g}\frac{\eta\Phi_g}{\phi} = \frac{\eta b_w^t}{2}\frac{\Phi_g}{\beta_g}, \tag*{(5)}

where η≤1\eta\le1 is the fraction of peak compute throughput achieved during decoding. Model size cancels, making B⋆B^\star primarily dependent on the device’s compute-to-bandwidth ratio and target weight precision. We use B⋆B^\star to estimate the transition from memory-bound to compute-bound decoding.

Joint cost model. Given quality tolerance ε\varepsilon, we select

θ⋆(B,ℓ)=arg⁡max⁡θ∈Θg Tput⁡g(θ,B,ℓ)\theta^\star(B,\ell) = \underset{\theta\in\Theta_g}{\arg\max}\ \operatorname{Tput}_g(\theta,B,\ell)
s.t.Mpeak(θ)≤Mg,QGap⁡(θ)≤ε.(6)\text{s.t.}\quad M_{\mathrm{peak}}(\theta) \le M_g,\quad\operatorname{QGap}(\theta) \le\varepsilon. \tag*{(6)}

Here, Θg\Theta_g contains configurations feasible under the device memory budget. The quality constraint applies only to lossy target-weight and KV precisions; draft length and drafter precision preserve the target distribution. Since the analytical model cannot capture low-level kernel efficiency, the full calibration setting measures Tput⁡g\operatorname{Tput}_g on the deployment GPU. On held-out GPUs, our zero-shot approach ranks static candidates using predictions or selection scores derived from source-GPU measurements.

SpecSearch with full calibration

The cost model identifies the set Θg\Theta_g of feasible deployment policies for a workload on a given GPU, which may include varying precision settings for model weights and KV cache, as well as a draft length mapping. SpecSearch performs an offline calibration pass for each feasible policy determined by the cost model on the deployment GPU. Accordingly, it ranks these deployment policies by their mean wall-clock speedup and throughput relative to AR decoding. The best policy (whether SD or AR) is then stored in a lookup table for inference-time selection before deploying the workload.

The effective batch size, defined as the number of scheduled requests per decoding step, varies as requests finish, are admitted, or are preempted in vLLM. Together with KV-cache precision, it determines GPU resource contention and KV-cache pressure. Thus, the optimal draft length kk also varies with the effective batch size and precision.

Dynamic draft-length search. SpecSearch constructs a dynamic draft length mapping from calibration data, identifying the fastest-performing kk for each workload and precision. At each decoding step, SpecSearch uses the mapping to dynamically set the corresponding draft length kk within vLLM. Table VI provides an example of the dynamic draft length mapping derived by SpecSearch. Batches 21–48 use k=3k = 3 for ℓ=256\ell= 256 and ℓ=1,024\ell= 1{,}024, but k=7k = 7 for ℓ=2,048\ell= 2{,}048. With fp8 KV and ℓ=2,048\ell= 2{,}048, the general trend is k=15k = 15, 7, and 3, respectively, for batches 1–20, 21–48, and 49–128.

Learning-Based Policy Predictor

SpecSearch trains a lightweight Jev-based policy predictor [5] to eliminate calibration on held-out GPUs. Given a deployment GPU and workload (B,ℓ)(B, \ell), the predictor selects the decoding mode, draft length, and KV-cache precision.

Training and validation datasets. The dataset covers eight decoding configurations: AR and SD with k∈{3,7,15}k \in\{3, 7, 15\}, each using fp16 or fp8 KV cache. Each GPU runs 96 configurations across B∈{8,32,64,128}B \in\{8, 32, 64, 128\} and ℓ∈{256,1024,2048}\ell\in\{256, 1024, 2048\}. We use four distinct GPUs when creating the dataset: two for training, one for validation, and one held out for evaluation. For each training GPU, the highest-throughput configuration per workload serves as the training label.

Model architecture. We adapt Qwen3-4B into an eight-choice Jev-style decision model by training rank-8 LoRA adapters on its attention projections. Each choice represents one of the eight decoding configurations in the training dataset. We additionally train a linear answer-logit correction layer using cross-entropy loss, while keeping the pretrained backbone frozen.

Training objective. Each training example ii represents one configuration-selection problem for a deployment GPU and workload (B,ℓ)(B, \ell). Its structured prompt contains the GPU and workload metadata, eight candidate configurations labeled A–H in shuffled order, and the permitted source calibration records. The label yiy_i is the answer letter assigned to the candidate with the highest measured throughput for that GPU–workload pair. Given the prompt, Qwen produces a final hidden representation hih_i, from which we compute the eight answer logits as

zi=WA:Hhi+Wchi,z_i = W_{\mathrm{A:H}}h_i + W_c h_i,

where WA:HW_{\mathrm{A:H}} denotes the frozen rows of Qwen’s language-model output head corresponding to tokens A–H, and WcW_c is a trainable, zero-initialized linear correction layer. Rank-8 LoRA adapters on the query, key, value, and output attention projections modify the hidden representation hih_i, while the pretrained Qwen weights remain frozen. We optimize the LoRA parameters and WcW_c using the classification cross-entropy objective

L=−1N∑i=1Nlog⁡(softmax⁡(zi)yi).\mathcal{L} = -\frac{1}{N}\sum_{i=1}^{N}\log\left(\operatorname{softmax}(z_i)_{y_i}\right).

Thus, training encourages the model to assign the highest answer score to the measured throughput-maximizing configuration. Further details are provided in Appendix B.

Learning-based policy prediction. At deployment time, the trained Jev-based predictor takes a structured text input, termed a decision prompt, describing the deployment GPU, workload, and eight candidate configurations labeled A through H. Rather than generating text, the model scores the corresponding answer-label tokens and selects the configuration with the highest score. This enables policy selection on a held-out GPU without calibration or profiling, leveraging the relationships between workloads, hardware characteristics, and decoding configurations learned during training.

Experimental Setup

GPUs. We evaluate four GPUs: NVIDIA GeForce RTX 5090 (32 GB) [28], AMD Instinct MI300X (192 GB) [29], NVIDIA H100 (80 GB) [30], and NVIDIA L40 (48 GB) [31].

Models and datasets. We use Qwen3-4B [32] target model and its DFlash drafter [3] with block size 16. Prompts are sampled from GSM8K [33], MATH-500 [34], MBPP [35], MT-Bench [36], and HumanEval [37].

Baselines. We compare against AR and default SD (k=15k = 15, no adjustments made to default drafting length) served by vLLM. Furthermore, we evaluate against MagicDec 3.

Calibration. When the cost model is disabled, we evaluate eight policies across several GPUs: AR decoding with FP16 or FP8 KV cache, and SD with k∈{3,7,15}k \in\{3, 7, 15\}, each using FP16 or FP8 KV cache. We consider workloads with B∈{8,32,64,128}B \in\{8, 32, 64, 128\} and ℓ∈{256,1,024,2,048}\ell\in\{256, 1{,}024, 2{,}048\}. On each GPU, the 12 workloads and three seeds yield $8 \times 12 \times 3 = 288 calibration runs. We additionally calibrate two dynamic draft-length mappings per workload and seed, adding 72 runs. Thus, an exhaustive calibration performs 360 runs per GPU.

Results

In this section, we evaluate both policy selection modes of SpecSearch: full calibration and learning-based policy selection.

In full calibration mode, SpecSearch uses its analytical cost model to refine the policy search space and identify feasible candidate configurations, then selects the best-performing policy through GPU-specific calibration.

The learning-based mode trains a Jev-based predictor to select optimal configurations from calibration data. As a result, the accuracy and portability of the cost model directly affect the quality of full calibration policy selection and the training dataset of the learned predictor.

We first validate the cost model across GPUs by comparing its predictions against measurement-based reference points obtained from actual vLLM executions. We then evaluate the end-to-end performance of both policy selection modes.

Cost model validation across GPUs

Table II validates the proposed cost model, including memory, concurrency, and throughput. We compare the predicted cost by our model against the measured cost when running vLLM for AR and SD with two KV cache precisions. (1) predicts total engine memory within 0.1% on the MI300X and 0.2% on the RTX 5090. (2) predicts maximum concurrency within 0.5% and 0.6%, respectively. Across the tested budgets, the cost model correctly rejects 7 of 20 configuration–budget pairs on the MI300X and 3 of 20 on the RTX 5090. These rejected pairs occur at allocations below the 15.7 GiB budget used for the main evaluation. In other words, the cost model correctly accepts all remaining pairs.

configurationMI300X Memory (GB) pred.MI300X Memory (GB) meas.MI300X Memory (GB) err.MI300X Cmax⁡C_{\max} pred.MI300X Cmax⁡C_{\max} meas.MI300X Cmax⁡C_{\max} err.MI300X Starts within allocation (GiB) 9MI300X Starts within allocation (GiB) 10MI300X Starts within allocation (GiB) 11MI300X Starts within allocation (GiB) 12MI300X Starts within allocation (GiB) 15.7RTX 5090 Memory (GB) pred.RTX 5090 Memory (GB) meas.RTX 5090 Memory (GB) err.RTX 5090 Cmax⁡C_{\max} pred.RTX 5090 Cmax⁡C_{\max} meas.RTX 5090 Cmax⁡C_{\max} err.RTX 5090 Starts within allocation (GiB) 9RTX 5090 Starts within allocation (GiB) 10RTX 5090 Starts within allocation (GiB) 11RTX 5090 Starts within allocation (GiB) 12RTX 5090 Starts within allocation (GiB) 15.7
AR, fp16 KV17.0117.02+0.1%11.0711.05−0.2%×/×✓/✓✓/✓✓/✓✓/✓16.9216.93+0.1%13.2413.22−0.2%✓/✓✓/✓✓/✓✓/✓✓/✓
AR, fp8 KV17.0117.02+0.1%22.1422.11−0.1%×/×✓/✓✓/✓✓/✓✓/✓16.8916.92+0.2%26.4826.33−0.6%✓/✓✓/✓✓/✓✓/✓✓/✓
vLLM SD, fp16 KV17.5417.56+0.1%7.147.10−0.5%×/××/××/×✓/✓✓/✓17.2917.31+0.1%8.838.80−0.4%×/××/×✓/✓✓/✓✓/✓
vLLM SD, fp8 KV17.5417.56−0.5%14.2814.20−0.5%×/××/×✓/✓✓/✓✓/✓17.2817.31+0.2%17.6617.57−0.5%×/×✓/✓✓/✓✓/✓✓/✓

Table II. Validation of the cost model across policies. Predicted (pred.) and measured (meas.) denote cost-model predictions for memory footprint and realized allocation, respectively. Additionally, GPU memory budgets are emulated, ranging from 9 to 15.7 GiB, to determine deployment feasibility. Thus, each cell contains two symbols: the first is the cost model prediction for feasibility and the second is the vLLM deployment outcome. A ✓ means feasible (runs successfully), × means out-of-memory (fails).

The model-dependent per-token KV-cache cost (κ\kappa) and drafter weight footprint match measured allocations within 1%, without any device-specific adjustments. Only the constant runtime overheads (ρg\rho_g, CactC_{\mathrm{act}}, and CgraphC_{\mathrm{graph}}) require a one-time startup measurement on each GPU. These results validate the cost model on MI300X and RTX 5090 after device-specific initialization.

Performance of SpecSearch with full calibration

Throughput and speedup results. Table III and IV report throughput (in k tokens/s) and wall-clock speedup measured by the calibration runs, respectively. The last two columns in each GPU block report the throughput improvements achieved by the policy SpecSearch selects over the fastest AR and vLLM SD configurations, respectively. Each GPU has available 15.7 GiB device budget for LLM inference. Similarly, Table XII and XIII report findings on NVIDIA H100 and L40 GPUs at a 15.7 GiB budget.

The performance gains vary with concurrency, output length, and GPU. At B=8B = 8, vLLM SD is already competitive, leaving little room for further improvement (Table III). At higher concurrency, SpecSearch more consistently outperforms

the vLLM SD baseline, particularly for longer outputs. For example, at B=128B = 128 and ℓ=2,048\ell= 2{,}048, SpecSearch reaches 12.03 k tokens/s on the MI300X and 9.53 k tokens/s on the RTX 5090, versus 8.91 and 5.55 k tokens/s, respectively, for vLLM SD with fp8 KV. The same setup delivers wall-clock speedups of 1.96× and 1.52× over AR with fp16 KV.

Averaged across workloads, SpecSearch achieves throughput improvements of 1.68×, 2.16×, 2.33×, and 1.85× over the best AR configuration on the MI300X, RTX 5090, H100, and L40, respectively. The corresponding speedup values over vLLM SD are 1.21×, 1.44×, 1.28×, and 1.51×. These results demonstrate SpecSearch’s ability to identify the best deployment configuration depending on both the workload and the GPU.

Figure 3 compares the end-to-end throughput of MagicDec and SpecSearch on the same RTX 5090, using Qwen3-4B as the target model and an identical prompt set. The two systems employ different serving optimizations: MagicDec uses sparse-KV self-speculation to compress the KV cache and reduce memory consumption for long-context workloads, with BF16 KV cache, static batching, and no request refill in our evaluation. In contrast, SpecSearch uses the vLLM backend, which supports paged KV caching and continuous batching, and jointly selects the KV-cache precision and speculation policy. These differences in serving mechanisms also affect the set of workloads each system can execute: MagicDec cannot run six decoding configurations due to memory constraints. Among the workloads supported by both systems, MagicDec achieves 0.28–1.26 k tokens/s, corresponding to 0.08–0.13× the throughput of SpecSearch. The results thus compare the end-to-end performance of the respective systems under a common hardware, model, and prompt configuration, rather than isolating the contribution of individual optimizations.

Figure 3. Throughput (in k tokens/s) comparison of MagicDec vs. SpecSearch for Qwen3-4B. Each cell reports MagicDec throughput on RTX5090, while its ratio to SpecSearch is given in parentheses. Gray cells denote out-of-memory (OOM) failures.

Selected policies. SpecSearch calibrates SD across different draft lengths to construct dynamic draft-length mappings. Table V details the policies selected by SpecSearch in Tables III and IV, including their corresponding draft lengths. Tables XIV and XV provide the individual calibration results used to derive draft length mappings. On the MI300X, SpecSearch selects speculation with fp8 KV at every workload, using a dynamic draft-length mapping at one workload and a static draft length at eleven workloads. On the RTX 5090, it selects fp16 KV at six workloads and fp8 KV at six, including the one workload at which AR decoding is selected. Eight RTX 5090 workloads use a static draft length, three use the dynamic mapping, and one uses AR decoding. tbl:6 lists the dynamic draft length mappings constructed by SpecSearch. On H100, five workloads use dynamic mappings and seven use static draft lengths, with fp16 KV selected at four workloads and fp8 at eight. On L40, two workloads use dynamic mappings, nine use static draft lengths, while one uses AR and all select fp8 KV. Target model quantization. At a smaller device budget such as 8 GiB, serving SD may require quantizing the target model. Table VII shares the policies with a quantized target and drafter that calibration explores, demonstrating feasible configurations

policytargetdraftKV cacheweights (GiB)KV tokensmax conc.ktok/swall (s)speedup
ARfp16–fp167.56
ARfp16–fp87.59
ARfp8–fp164.2419,7124.81×8.6175.9420.63×
ARfp8–fp84.2439,5689.66×13.6613.7481.00×
SDfp16fp16fp168.61
SDfp16fp16fp88.65
SDfp8fp16fp165.36
SDfp8fp16fp85.365,3761.31×5.7018.9810.42×
SDfp16fp8fp88.1
SDfp8fp8fp84.8311,8082.88×5.8988.6810.43×
SDfp16int4fp87.85
SDfp8int4fp84.556,9601.70×6.0938.4030.45×

Table VII. Quantization explored under an 8 GiB memory budget on the RTX 5090. Shaded rows indicate policies that exceed the device memory budget. Target and draft denote the target and draft model precisions, respectively. ℓ=4,096\ell= 4,096.

ℓ\ellBB\multicolumn{6}{c}{MI300X}\multicolumn{8}{c}{RTX 5090}
vLLM AR fp16vLLM AR fp8vLLM SD fp16vLLM SD fp8SpecSearch fp16SpecSearch fp8best ARbest vLLM SDAR fp16AR fp8vLLM SD fp16vLLM SD fp8SpecSearch fp16SpecSearch fp8best ARbest SD
25681.881.904.264.554.324.692.47×1.03×1.021.013.913.893.913.843.83×1.00×
325.155.757.188.148.388.881.54×1.09×3.653.675.135.197.367.542.01×1.42×
648.219.348.458.9310.8811.911.28×1.33×6.376.655.445.539.469.311.42×1.71×
12811.9813.919.3810.1314.9916.241.17×1.60×10.0110.765.675.7810.4810.891.00×1.86×
1,02481.811.863.584.183.584.322.32×1.03×1.031.033.733.753.733.753.62×0.99×
323.924.846.096.896.857.841.62×1.14×3.253.314.894.947.017.002.12×1.42×
645.857.167.218.538.7810.151.42×1.19×5.145.375.305.378.318.531.59×1.59×
1288.3210.768.1410.139.8314.771.37×1.46×7.979.045.295.778.4810.291.14×1.78×
2,04881.751.813.424.033.404.032.23×1.00×1.011.013.753.653.753.653.71×1.00×
323.143.935.506.476.057.171.82×1.11×2.672.774.914.816.126.342.29×1.29×
644.625.736.397.487.489.191.60×1.23×3.954.215.065.167.107.651.82×1.48×
1286.058.796.908.918.7512.031.37×1.35×6.307.165.165.557.869.531.33×1.72×
Average1.68×1.21×2.16×1.44×

Table 3. Throughput (K tokens/s) of decoding policies at 15.7 GiB device budget. Bold: highest mean value per workload. Underlined: the policy selected by SpecSearch. Within each method (AR, vLLM SD, SpecSearch), the darker shading marks FP8 KV in contrast to FP16 KV cache.

ℓ\ellBBvLLM AR fp16vLLM AR fp8vLLM SD fp16vLLM SD fp8SpecSearch fp16SpecSearch fp8best ARbest vLLM SDAR fp16AR fp8Baseline SD fp16Baseline SD fp8SpecSearch fp16SpecSearch fp8best ARbest SD
25681.00×1.01×2.27×2.42×2.30×2.50×2.46×1.03×1.00×0.99×3.84×3.82×3.84×3.77×3.84×1.00×
321.00×1.12×1.39×1.58×1.62×1.72×1.54×1.09×1.00×1.01×1.41×1.43×2.02×2.06×2.00×1.41×
641.00×1.13×1.03×1.08×1.32×1.45×1.28×1.35×1.00×1.05×0.85×0.87×1.49×1.46×1.42×1.71×
1281.00×1.16×0.78×0.85×1.25×1.36×1.17×1.60×1.00×1.08×0.57×0.58×1.05×1.09×1.00×1.87×
1,02481.00×1.03×1.99×2.34×1.99×2.40×2.34×1.03×1.00×1.01×3.63×3.66×3.63×3.66×3.59×0.99×
321.00×1.23×1.57×1.76×1.76×2.01×1.63×1.14×1.00×1.02×1.53×1.53×2.14×2.13×2.10×1.40×
641.00×1.22×1.22×1.44×1.50×1.72×1.42×1.19×1.00×1.05×1.05×1.05×1.66×1.66×1.58×1.59×
1281.00×1.30×0.98×1.22×1.19×1.77×1.36×1.45×1.00×1.13×0.67×0.73×1.06×1.30×1.15×1.78×
2,04881.00×1.06×1.98×2.32×2.01×2.32×2.19×1.00×1.00×1.01×3.77×3.61×3.77×3.61×3.73×1.00×
321.00×1.23×1.77×2.01×1.93×2.26×1.83×1.12×1.00×1.01×1.82×1.79×2.26×2.30×2.28×1.26×
641.00×1.25×1.38×1.60×1.61×1.96×1.57×1.22×1.00×1.06×1.30×1.28×1.83×1.92×1.81×1.47×
1281.00×1.45×1.13×1.46×1.45×1.96×1.36×1.35×1.00×1.13×0.83×0.87×1.26×1.52×1.34×1.75×
Average1.68×1.21×2.16×1.44×

Table 4. Speedup of decoding policies at 15.7 GiB device budget. Bold: highest mean value per workload. Underlined: the policy selected by SpecSearch. Within each method (AR, vLLM SD, SpecSearch), the darker shading marks FP8 KV in contrast to FP16 KV cache.

ℓ\ellB=8B=8B=32B=32B=64B=64B=128B=128
MI300X256k=7k=7, fp8k=3k=3, fp8k=3k=3, fp8k=3k=3, fp8
1,024k=7k=7, fp8k=3k=3, fp8k=3k=3, fp8k=3k=3, fp8
2,048k†k^\dagger, fp8k=7k=7, fp8k=7k=7, fp8k=3k=3, fp8
RTX 5090256k=15k=15, fp16k=3k=3, fp16k=3k=3, fp16AR, fp8
1,024k=15k=15, fp16k†k^\dagger, fp16k=3k=3, fp16k=3k=3, fp8
2,048k=15k=15, fp16k†k^\dagger, fp8k†k^\dagger, fp8k=3k=3, fp8
H100256k†k^\dagger, fp8k†k^\dagger, fp8k=3k=3, fp8k=3k=3, fp16
1,024k†k^\dagger, fp16k†k^\dagger, fp8k=3k=3, fp8k=3k=3, fp8
2,048k=15k=15, fp16k†k^\dagger, fp8k=7k=7, fp16k=3k=3, fp8
L40256k†k^\dagger, fp8k=3k=3, fp8k=3k=3, fp8AR, fp8
1,024k†k^\dagger, fp8k=3k=3, fp8k=3k=3, fp8k=3k=3, fp8
2,048k=15k=15, fp8k=3k=3, fp8k=3k=3, fp8k=3k=3, fp8
GPUℓ\ell1–2021–4849–9697–128
MI300X2567/77/33/33/3
MI300X1,02415/157/37/33/3
MI300X2,04815/157/77/37/3
RTX 509025615/153/33/33/3
RTX 50901,02415/153/33/33/3
RTX 50902,04815/157/73/33/3
H10025615/157/73/33/3
H1001,02415/157/73/33/3
H1002,04815/157/77/73/3
L4025615/153/33/33/3
L401,02415/153/33/33/3
L402,04815/153/33/33/3

Table 5. Policy selected by SpecSearch in Table III and IV: AR vs. SD decoding (with draft length kk) and KV precision. k†k^\dagger: dynamic draft-length mapping for SD (Table VI).

under this budget on an RTX 5090. The SD policies that fit 8 GB are slower than AR decoding on this GPU. tbl:XI reports the throughput of configurations with an fp8 target explored by SpecSearch on the MI300X, using a 15.7 GiB budget. Within this setting, SpecSearch averages 2.17× the throughput of AR and 1.03× that of vLLM SD. Relative to the corresponding fp16-target results in tbl:III, target quantization reduces the throughput of AR, vLLM SD, and SpecSearch across all workloads on the MI300X GPU. This is because quantizing the target model perturbs the reference distribution, introducing loss to verification and harming the overall acceptance.

Drafter model quantization. Quantized drafter models leave the output distribution of SD unchanged, maintaining losslessness. Since drafter only determines the draft token quality, a quantized drafter only impacts the draft token acceptance length. tbl:VIII reports this effect, measured on the RTX 5090 at ℓ=2,048\ell=2,048 with fp16 target and fp8 KV cache. Quantized fp8 and int4 drafters reduce drafter weight memory by 50% and 74%, while the corresponding acceptance retention stays within 1% of the fp16 drafter. Thus, neither drafter precision changes serving throughput measurably.

Performance of Learning-Based Policy Predictor

We evaluate zero-shot policy selection using a four-fold cross-GPU evaluation. In each fold, one of the four GPUs is held

batch sizedrafterweights (GiB)τˉ\bar{\tau}retentionk tok/sretention
32fp161.019.871.08.981.0×
32fp80.519.901.09.161.0×
32int40.269.810.99.001.0×
64fp161.019.811.08.631.0×
64fp80.5110.031.09.431.0×
64int40.269.760.98.971.0×

Table VIII. Drafter quantization on the RTX 5090. τˉ\bar{\tau} is the mean acceptance length of draft tokens across the datasets. Retention is the rate at which mean acceptance and throughput are preserved relative to FP16.

out for evaluation, while measurements from the remaining three GPUs are used to train the predictor. We repeat this process four times, using each GPU as the held-out GPU exactly once. Each GPU contributes 12 workloads, resulting in 48 workload-hardware pairs across the four folds. We assess each policy predicted by our Jev-based selector by measuring its throughput on the held-out GPU. These measurements are strictly reserved for evaluation and are excluded from predictor training and model selection. We compare the performance of configurations selected by the Jev-based predictor against the performance of policy search predicted in full calibration mode. Unlike the learned predictor, this SpecSearch mode has access to calibration measurements from the evaluation GPU.

Selected policy performance. tbl:IX evaluates how effectively the trained Jev-based predictor selects decoding configurations on unseen GPUs, comparing it against the baseline vLLM decoding. The Jev-based predictor achieves average throughput ratios of 2.02× over AR and 1.30× over vLLM SD. Selected policies are shared in Appendix B.

Comparison with full calibration. tbl:IX shares the throughput gap that measures the relative performance loss of the inferred policy compared with the best-performing configuration returned by SpecSearch in full calibration mode. A smaller gap indicates more efficient policy selection. Relative to the best measured static configuration, its mean throughput loss is 0.75% on RTX 5090, 1.80% on MI300X, 1.52% on L40, and 5.95% on H100. Averaged equally across the four held-out GPUs, the fine-tuned Jev-based predictor retains 97.36% of the matched full calibration SpecSearch throughput. Jev-based fine-tuning maintains an average throughput gap of 2.50% to the optimal policies discovered during calibration. These results demonstrate effective policy selection without deployment-GPU profiling.

Predictions across varying concurrency. We study how workload concurrency influences the predictor’s choices. The largest benefit of Jev-based prediction occurs at B=8B=8, where the mean throughput gap to the best configuration’s throughput decreases from 10.76% to 1.82%. The Jev-based model selects k=15k=15 for all 12 workloads at this concurrency, whereas the frozen Qwen-3-4B decision model selects k=3k=3 for all four 256-token workloads. At B=64B=64, both models sets k=3k=3 with fp8

GPU
RTX 5090MI300XH100L40Mean
Jev-based vs. F.C.0.75%1.80%5.95%1.52%2.50%
Jev-based vs. vLLM AR2.24×1.68×2.27×1.90×2.02×
Jev-based vs. vLLM SD1.40×1.14×1.19×1.47×1.30×

Table IX. Learned policy selection via Jev-based predictor. First row reports its throughput gap compared to the best policy returned by full calibration (F.C.) SpecSearch. Last two rows are speedup over baseline vLLM runs.

KV cache precision. At B=128B=128, the Jev-based model selects AR with fp8 KV for the 256-token RTX 5090 workload and k=3k=3 for the other 11 workloads. These choices reflect the load-dependent behavior observed in Section III: longer drafts remain beneficial at low concurrency, whereas shorter drafts or AR remain faster as concurrency increases.

Prediction errors. The Jev-based predictor selects a configuration within 5% of the best configuration’s throughput for 38 of the 48 workloads. Most of the larger prediction errors occur on H100 at B=32B=32: the predictor selects k=3k=3 with fp8 KV for all output lengths from 256 to 2048 tokens. For these workloads, the best-performing configurations use k=7k=7 on H100. The resulting throughput gaps range from 13.73% to 21.64%. These results reveal a limitation at moderate concurrency on H100, where the predictor favors shorter drafts than the measured throughput optimum.

Ablations

Memory cost of speculation. Figure 2(c,Figure 2d) and Table X quantify the memory cost of speculation described in Observation 3. At equal KV precision, vLLM SD retains 0.73× the AR KV cache capacity on the RTX 5090 under a 15.7 GiB budget and 0.68× on the MI300X under a 16 GB budget. Despite its smaller KV cache capacity, vLLM SD delivers 3.5–3.9× the AR throughput at concurrency 1–4. This advantage disappears by concurrency 64, where the throughput ratio falls to 0.93–0.97×.

policyRTX 5090 capacity (tokens)RTX 5090 max conc. (queries)MI300X capacity (tokens)MI300X max conc. (queries)
AR, fp16 KV54,80013.4×49,63212.1×
AR, fp8 KV109,16826.7×99,26424.2×
vLLM SD, k=15k=15, fp16 KV40,0009.8×33,8088.3×
vLLM SD, k=15k=15, fp8 KV79,60019.4×67,63216.5×
Concurrency retention rate0.730.68

Table X. Relationship between KV cache capacity, in tokens, and maximum concurrency. The workload has ℓ=4,096\ell=4,096 across runs.

Generation length sensitivity. An fp8 KV cache doubles the KV cache capacity of both decoding methods. For short outputs, however, it changes throughput by at most 4%. The

ℓ\ellBBARvLLM SDSpecSearchImprovement vs. best ARImprovement vs. vLLM SD
25680.863.042.693.14×0.89×
256322.976.426.272.11×0.98×
256644.997.778.741.75×1.12×
2561287.488.499.981.33×1.18×
1,02480.832.882.873.47×1.00×
1,024322.595.445.182.00×0.95×
1,024643.966.266.491.64×1.04×
1,0241285.256.717.561.44×1.13×
2,04880.832.812.693.24×0.96×
2,048322.245.015.482.45×1.09×
2,048643.365.895.741.71×0.97×
2,0481283.846.316.871.79×1.09×
Average2.17×1.03×

Table XI. Throughput (K tokens/s) with target model quantization (fp8) on the MI300X. SpecSearch uses a dynamic draft-length mapping with the fp8 target. The improvement columns report SpecSearch throughput relative to the vLLM baseline. Bold: highest mean throughput per workload.

additional capacity becomes useful as outputs grow longer. For AR decoding on the RTX 5090, the fp8-to-fp16 throughput ratio rises from 1.08× at 512 generated tokens to 1.73× at 16,384 tokens (Figure 4). The MI300X exhibits the same trend.

Figure 4. AR decoding: fp8-to-fp16 KV throughput ratio versus generated tokens per request, with concurrency 48.

Limitations

This study focuses on a single target model and a single diffusion-based drafter. Although the proposed cost model is portable across model architectures, extending it to alternative SD methods, such as EAGLE [9], [10] and Medusa [8], requires adapting the terms that characterize drafter-specific costs. The empirical results remain limited to the chosen DFlash drafter.

SpecSearch supports dynamic draft-length policies by deriving them from calibration runs over static draft lengths. However, the learned Jev-based predictor currently selects only static configurations. Incorporating calibration measurements for dynamic draft-length policies into the training dataset could extend the predictor to support dynamic policy selection. We leave this as future work.

The learned Jev-based predictor also exhibits performance gaps on certain workload–GPU combinations, most notably at moderate concurrency on H100, where it favors shorter drafts than the measured throughput optimum. Future work could improve prediction accuracy by expanding the training data across a broader range of GPU architectures and workloads, and by incorporating more informative hardware and workload features into the decision prompts.

Conclusion

Speculative decoding has no universally optimal configuration, as its performance depends on workload characteristics and GPU hardware. For example, on the RTX 5090, its speedup over AR drops from 3.50×3.50 \times at low concurrency to 0.76×0.76 \times at high concurrency when generation length shortens. To address this challenge, SpecSearch combines an analytical cost model that narrows the deployment search space with two complementary policy-selection approaches. full calibration SpecSearch evaluates candidate configurations on the deployment GPU, jointly optimizing KV-cache and model-weight precision, draft length, and whether to enable speculation. It achieves average throughput improvements of 2.09×2.09 \times over AR and 1.36×1.36 \times over vLLM SD across four GPUs.

Using the calibration dataset, SpecSearch fine-tunes a Jev-based optimal policy predictor. Across four held-out GPUs, it maintains a minimal throughput gap of 2.50%2.50\% to the best-performing configurations returned by full calibration SpecSearch. The learned policy predictor achieves 2.02×2.02 \times the throughput of AR and 1.30×1.30 \times that of vLLM SD, retaining 97.36%97.36\% of the throughput achieved by full calibration SpecSearch.

Together, these approaches enable hardware-aware policy selection across diverse workloads and deployment conditions, improving inference-serving efficiency on resource-constrained GPUs.

References

Appendix A: SpecSearch with full calibration and Results

Cost-model initialization

In Equation (1), the weight term and the per-token KV cost κ\kappa follow from the model architecture. The constant overheads ρg\rho_g, CactC_{\mathrm{act}}, and CgraphC_{\mathrm{graph}} in Equation (2) depend on the GPU and software stack. For MI300X and RTX 5090, we measure them from vLLM’s startup memory breakdown with B∈{1,32}B \in\{1, 32\}, interpolating linearly in BB. This gives ρg\rho_g of 1.14/1.25 GiB (AR/SD) on the MI300X and 0.35/0.44 GiB on the RTX 5090. We define compute factors η=0.125\eta= 0.125 on the MI300X and η=0.25\eta= 0.25 on the RTX 5090 in Eq. (5). Zero-shot policy selection does not use deployment GPU startup measurements or fitted runtime overheads.

Appendix B: SpecSearch with Learning-Based Policy Selectors

Data and features. Each deployment GPU is excluded from the three-GPU source dataset. Each source contributes 96 configuration–workload points: eight static choices at 12 workloads, with throughput aggregated over the available measurement seeds. For IDW, the shallow tree, and TabICL, numerical deployment metadata features encode memory budget, physical memory capacity, submitted request count, and whether the software platform uses ROCm. Workload features encode normalized log concurrency and output cap. Candidate features encode AR versus SD, draft length, and KV precision. The compact model uses only workload and candidate features. Jev receives these settings as text, together with the recorded attention backend, weight precision, and software versions where available.

IDW and shallow tree. IDW predicts log-throughput with weights wi=(di2+0.01)−2w_i = (d_i^2 + 0.01)^{-2}, where di2d_i^2 is squared Euclidean distance in the fixed feature representation. It fits no model parameters. The tree uses maximum depth 2 and random seed 0. Training labels are the fastest observed configurations grouped by source GPU and workload.

TabICL. We use the pretrained TabICL regressor with eight estimators, GPU inference, and the trial seed as its random seed. Inputs are the numerical features above and labels are log-throughput. Source measurements condition its predictions. No gradient fine-tuning of the pretrained model is performed.

Compact numerical model. Separate workload and candidate encoders feed a shared scoring head, with hidden width 32 and GELU activations. Training minimizes centered log-throughput mean squared error plus 0.1 times a within-workload soft-ranking cross-entropy term, with temperature 0.1. AdamW uses learning rate 0.003, weight decay 0.01, and seed 0. Source-only validation leaves out each source GPU in turn and compares 300 versus 900 training steps and ridge-correction penalties α∈{0.1,1,10}\alpha\in\{0.1, 1, 10\}, using three random 16-point supports on each validation GPU. The final model is refit on all three sources. All four folds select 900 steps. The source-validation correction penalty is 0.1 for RTX 5090 and 1 for the other targets. The zero-shot evaluation uses the source-trained model without a deployment-GPU residual correction.

Jev decision model. Both variants use Qwen3-4B with an eight-choice answer readout. Prompts contain the workload and the deployment settings listed above, together with up to 16 retrieved measurements per source GPU, selected by workload proximity without ranking by throughput. The source-tuned variant trains rank-8 LoRA adapters on attention query, key, value, and output projections (α=16\alpha= 16, dropout 0.05), together with a zero-initialized linear correction to the answer logits. Training uses classification cross-entropy, AdamW with learning rate 10−410^{-4} and weight decay 0.01, effective batch size 8, gradient clipping at 1, and seed 0. Execution uses FP32 with SDPA and an 8,192-token prompt limit.

During training on source GPUs, prediction tasks simulate 0, 4, 8, or 16 observations from a source GPU treated as the query device, excluding query–workload outcomes from prediction inputs. These observations come entirely from the source dataset. Evaluation on the deployment GPU uses zero-shot policy selection. One supplied-value reading task accompanies every four prediction tasks; answer order is shuffled consistently with the labels. One of the three source GPUs is reserved for development validation, with training on the other two. Checkpoints at 0, 25, 50, 100, and 200 steps are compared by validation throughput loss, then the chosen duration is used to refit from initialization on all three sources. The selected durations are 100 steps for RTX 5090 and MI300X and 200 for H100 and L40. The frozen variant retains the initial base model and zero correction. Supplied-value reading checks are recorded separately and are not uniformly perfect.

Selected policies. Table XVI reports the saved static choices from zero-shot policy selection for IDW and source-tuned Jev in the format of Table V. Table XVII gives dynamic mappings constructed from their saved candidate scores, using the same batch ranges as Table VI. For each GPU, output cap, KV precision, and concurrency anchor B∈{8,32,64,128}B \in\{8, 32, 64, 128\}, we maximize the score over k∈{3,7,15}k \in\{3, 7, 15\}; exact ties prefer smaller kk. The anchors define ranges 1–20, 21–48, 49–96, and 97–128. AR is excluded from this within-engine mapping; choosing AR or a KV precision remains a separate startup decision.

MethodGPUℓ\ell1–2021–4849–9697–128
IDWMI300X2567/77/77/73/3
1,0247/77/77/73/3
2,0487/77/77/73/3
IDWRTX 509025615/153/33/33/3
1,02415/153/33/33/3
2,04815/153/33/33/3
IDWH10025615/153/33/33/3
1,02415/153/33/33/3
2,04815/153/33/33/3
IDWL4025615/153/33/33/3
1,02415/157/33/33/3
2,04815/157/73/33/3
JevMI300X25615/153/33/33/3
1,02415/153/33/33/3
2,04815/153/73/33/3
JevRTX 509025615/153/33/33/3
1,02415/153/33/33/3
2,04815/153/33/33/3
JevH10025615/153/33/33/3
1,02415/153/33/33/3
2,04815/153/33/33/3
JevL4025615/153/33/33/3
1,02415/153/33/33/3
2,04815/153/73/33/3

Table XVII. Predicted dynamic draft-length mappings of IDW and source-tuned Jev. Each entry reports kk for FP16/FP8 KV, respectively, at the indicated effective-batch range.

IDW uses predicted log-throughput; Jev uses the eight-choice answer logits restricted to the three speculative actions at the specified KV precision. Neither construction uses target outcomes. These tables characterize the dynamic mappings implied by the static candidate scores. The performance evaluation is restricted to static configuration selection; it does not measure the throughput of these dynamic mappings.

ℓ\ellBBH100 vLLM AR fp16H100 vLLM AR fp8H100 vLLM SD fp16H100 vLLM SD fp8H100 SpecSearch fp16H100 SpecSearch fp8H100 Improvement vs. best ARH100 Improvement vs. best SDL40 AR fp16L40 AR fp8L40 vLLM SD fp16L40 vLLM SD fp8L40 SpecSearch fp16L40 SpecSearch fp8L40 Improvement vs. best ARL40 Improvement vs. best SD
82561.201.204.194.164.194.183.48×1.00×0.610.621.851.761.851.822.94×0.99×
322563.883.877.467.519.539.562.47×1.27×1.972.052.162.373.653.651.78×1.54×
642566.906.838.338.2112.5712.451.80×1.49×3.343.542.422.464.234.371.23×1.77×
12825611.5711.749.159.0314.2013.961.21×1.55×5.065.442.572.614.884.931.00×2.09×
81,0241.191.203.923.913.923.913.26×1.00×0.590.611.781.771.781.792.95×1.01×
321,0243.333.337.076.838.538.222.47×1.16×1.731.832.172.203.433.441.88×1.56×
641,0245.355.407.787.4810.6010.441.93×1.34×2.602.862.312.503.923.941.38×1.57×
1281,0248.909.248.208.2712.4313.751.49×1.66×3.964.662.282.604.004.731.01×1.82×
82,0481.161.183.843.743.843.743.25×1.00×0.580.591.701.821.701.823.06×1.00×
322,0482.752.746.816.528.077.682.80×1.13×1.451.552.122.223.173.332.16×1.50×
642,0484.144.247.356.979.038.582.13×1.23×2.042.292.262.303.553.551.55×1.54×
1282,0487.137.427.837.6510.4912.001.62×1.53×3.313.672.262.553.614.511.23×1.77×
Average2.33×1.28×1.85×1.51×

Table XII. Throughput (K tokens/s) of decoding policies at 15.7 GiB device budget. Bold: highest mean value per workload. Underlined: the policy selected by SpecSearch. Within each method (AR, vLLM SD, SpecSearch), the darker shading marks fp8 KV in contrast to fp16 KV cache.

ℓ\ellBBH100 vLLM AR fp16H100 vLLM AR fp8H100 vLLM SD fp16H100 vLLM SD fp8H100 SpecSearch fp16H100 SpecSearch fp8H100 Improvement vs. best ARH100 Improvement vs. best SDL40 AR fp16L40 AR fp8L40 Baseline SD fp16L40 Baseline SD fp8L40 SpecSearch fp16L40 SpecSearch fp8L40 Improvement vs. best ARL40 Improvement vs. best SD
82561.00×1.00×3.49×3.48×3.49×3.49×3.49×1.00×1.00×1.01×3.02×2.87×3.02×2.97×2.94×0.98×
322561.00×1.00×1.92×1.94×2.46×2.47×2.47×1.27×1.00×1.04×1.09×1.20×1.85×1.85×1.78×1.54×
642561.00×0.99×1.21×1.19×1.83×1.81×1.81×1.49×1.00×1.06×0.73×0.74×1.27×1.31×1.23×1.78×
1282561.00×1.01×0.79×0.78×1.23×1.21×1.21×1.55×1.00×1.08×0.51×0.51×0.97×0.98×1.00×2.09×
81,0241.00×1.03×3.33×3.27×3.31×3.27×3.23×1.00×1.00×1.03×2.97×2.95×2.97×2.99×2.91×1.01×
321,0241.00×0.99×2.09×2.05×2.51×2.46×2.46×1.18×1.00×1.06×1.27×1.28×1.99×2.01×1.90×1.57×
641,0241.00×1.01×1.44×1.39×1.97×1.94×1.92×1.34×1.00×1.08×0.88×0.95×1.50×1.51×1.40×1.58×
1281,0241.00×1.04×0.93×0.93×1.40×1.54×1.49×1.66×1.00×1.18×0.58×0.66×1.01×1.20×1.01×1.82×
82,0481.00×1.02×3.32×3.16×3.32×3.16×3.26×1.00×1.00×1.05×2.97×3.13×2.97×3.13×2.99×1.00×
322,0481.00×0.98×2.46×2.37×2.91×2.79×2.79×1.13×1.00×1.06×1.47×1.51×2.18×2.23×2.11×1.48×
642,0481.00×1.00×1.75×1.67×2.15×2.04×2.15×1.23×1.00×1.12×1.13×1.13×1.75×1.76×1.57×1.56×
1282,0481.00×1.02×1.08×1.04×1.45×1.64×1.60×1.51×1.00×1.09×0.68×0.76×1.09×1.36×1.25×1.79×
Average2.32×1.28×1.84×1.52×

Table XIII. Speedup of decoding policies at 15.7 GiB device budget. Bold: highest mean value per workload. Underlined: the policy selected by SpecSearch. Within each method (AR, vLLM SD, SpecSearch), the darker shading marks fp8 KV in contrast to fp16 KV cache.

ℓ\ellBBk=3k=3 fp16k=3k=3 fp8k=7k=7 fp16k=7k=7 fp8k=15k=15 fp16k=15k=15 fp8
82563.493.754.324.634.264.55
322568.158.908.359.127.188.14
6425610.6911.3010.0911.288.458.93
12825613.1514.2112.3512.539.3810.13
81,0242.943.473.684.313.584.18
321,0247.018.157.038.136.096.89
641,0248.469.658.559.897.218.53
1281,0249.5411.929.3610.758.1410.13
82,0482.643.193.384.013.424.03
322,0485.806.706.396.795.506.47
642,0486.888.206.867.766.397.48
1282,0487.629.047.718.586.908.91

Table XIV. MI300X throughput (K tokens/s) for every static draft length and KV-cache precision at the 15.7 GiB budget. Bold: highest mean throughput at that workload.

ℓ\ellBBk=3k=3 fp16k=3k=3 fp8k=7k=7 fp16k=7k=7 fp8k=15k=15 fp16k=15k=15 fp8
25681.86×2.01×2.31×2.47×2.27×2.42×
321.52×1.67×1.56×1.70×1.39×1.58×
641.34×1.41×1.26×1.41×1.03×1.08×
1281.21×1.31×1.13×1.15×0.78×0.85×
1,02481.64×1.93×2.05×2.41×1.99×2.34×
321.72×2.02×1.72×1.97×1.57×1.76×
641.42×1.60×1.43×1.64×1.22×1.44×
1281.27×1.57×1.25×1.39×0.98×1.22×
2,04881.44×1.72×1.89×2.21×1.98×2.32×
321.80×1.95×1.91×1.98×1.77×2.01×
641.39×1.62×1.40×1.53×1.38×1.60×
1281.30×1.52×1.34×1.40×1.13×1.46×

Table XV. MI300X wall-clock speedup over AR decoding with FP16 KV for every static draft length and KV-cache precision. Bold: highest mean speedup at that workload.

MethodGPUℓ\ellB=8B=8B=32B=32B=64B=64B=128B=128
IDWMI300X256k=7k=7, fp8k=7k=7, fp8k=7k=7, fp8k=3k=3, fp8
1,024k=7k=7, fp8k=7k=7, fp8k=7k=7, fp8k=3k=3, fp8
2,048k=7k=7, fp8k=7k=7, fp8k=7k=7, fp8k=3k=3, fp8
IDWRTX 5090256k=15k=15, fp16k=3k=3, fp8k=3k=3, fp8AR, fp8
1,024k=15k=15, fp8k=3k=3, fp8k=3k=3, fp8k=3k=3, fp8
2,048k=15k=15, fp8k=3k=3, fp8k=3k=3, fp8k=3k=3, fp8
IDWH100256k=15k=15, fp16k=3k=3, fp8k=3k=3, fp8k=3k=3, fp8
1,024k=15k=15, fp8k=3k=3, fp8k=3k=3, fp8k=3k=3, fp8
2,048k=15k=15, fp8k=3k=3, fp8k=3k=3, fp8k=3k=3, fp8
IDWL40256k=15k=15, fp16k=3k=3, fp8k=3k=3, fp8k=3k=3, fp8
1,024k=15k=15, fp16k=3k=3, fp8k=3k=3, fp8k=3k=3, fp8
2,048k=15k=15, fp8k=7k=7, fp8k=3k=3, fp8k=3k=3, fp8
JevMI300X256k=15k=15, fp16k=3k=3, fp8k=3k=3, fp8k=3k=3, fp8
1,024k=15k=15, fp16k=3k=3, fp8k=3k=3, fp8k=3k=3, fp8
2,048k=15k=15, fp8k=7k=7, fp8k=3k=3, fp8k=3k=3, fp8
JevRTX 5090256k=15k=15, fp8k=3k=3, fp8k=3k=3, fp8AR, fp8
1,024k=15k=15, fp16k=3k=3, fp8k=3k=3, fp8k=3k=3, fp8
2,048k=15k=15, fp8k=3k=3, fp8k=3k=3, fp8k=3k=3, fp8
JevH100256k=15k=15, fp16k=3k=3, fp8k=3k=3, fp8k=3k=3, fp8
1,024k=15k=15, fp8k=3k=3, fp8k=3k=3, fp8k=3k=3, fp8
2,048k=15k=15, fp8k=3k=3, fp8k=3k=3, fp8k=3k=3, fp8
JevL40256k=15k=15, fp16k=3k=3, fp8k=3k=3, fp8k=3k=3, fp8
1,024k=15k=15, fp8k=3k=3, fp8k=3k=3, fp8k=3k=3, fp8
2,048k=15k=15, fp8k=7k=7, fp8k=3k=3, fp8k=3k=3, fp8

Table XVI. Policies selected by IDW and source-tuned Jev at each workload: AR vs. vLLM SD and KV precision.

References

  1. [1]Y. Leviathan, M. Kalman, and Y. Matias, “Fast Inference from Transformers via Speculative Decoding,” arXiv:2211.17192, 2022.arxiv.org/abs/2211.17192
  2. [2]C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper, “Accelerating Large Language Model Decoding with Speculative Sampling,” arXiv:2302.01318, 2023.arxiv.org/abs/2302.01318
  3. [3]J. Chen, Y. Liang, and Z. Liu, “DFlash: Block Diffusion for Flash Speculative Decoding,” arXiv:2602.06036, 2026.arxiv.org/abs/2602.06036
  4. [4]M. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the 29th Symposium on Operating Systems Principles. ACM, Oct. 2023, pp. 611–626. [Online]. Available: http://dx.doi.org/10.1145/3600006.3613165
  5. [5]Y. Li and J. Wagle, “LLM-as-Jey: LLMs Are Already Jey-Style Decision Models—When and How to Fine-Tune Them,” arXiv preprint arXiv:2610.02076, 2026.arxiv.org/abs/2610.02076
  6. [6]Advanced Micro Devices, Inc., “AMD Instinct MI300X Accelerator Data Sheet,” https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/data-sheets/amd-instinct-mi300x-data-sheet.pdf, 2023, accessed: 2026-10-21.
  7. [7]NVIDIA Corporation, “NVIDIA RTX Blackwell GPU Architecture,” https://images.nvidia.com/aem-dam/Solutions/geforce/blackwell/nvidia-rtx-blackwell-gpu-architecture.pdf, 2025, accessed: 2026-10-21.
  8. [8]T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao, “Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads,” arXiv:2401.10774, 2024.arxiv.org/abs/2401.10774
  9. [9]Y. Li, F. Wei, C. Zhang, and H. Zhang, “EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty,” arXiv:2401.15077, 2024.arxiv.org/abs/2401.15077
  10. [10]——, “EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test,” arXiv:2503.01840, 2025.arxiv.org/abs/2503.01840
  11. [11]L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y. Sheng, “SGLang: Efficient Execution of Structured Language Model Programs,” in Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2312.07104
  12. [12]S. Yildirim and D. Chen, “SpecMemo: Speculative Decoding is in Your Pocket,” arXiv:2506.01986, 2025.arxiv.org/abs/2506.01986
  13. [13]E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers,” arXiv:2210.17323, 2022.arxiv.org/abs/2210.17323
  14. [14]G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “SmoothQuant: Accurate and efficient post-training quantization for large language models,” in Proceedings of the 40th International Conference on Machine Learning, vol. 202, 2023, pp. 38 087–38 099.arxiv.org/abs/2211.10438
  15. [15]J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration,” arXiv:2306.00978, 2023.arxiv.org/abs/2306.00978
  16. [16]Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V. Braverman, B. Chen, and X. Hu, “KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache,” arXiv:2402.02750, 2024.DOI
  17. [17]C. Hooper, S. Kim, H. Mohammadzaheh, M. W. Mahoney, Y. S. Shao, K. Keutzer, and A. Gholami, “KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization,” arXiv:2401.18079, 2024.arxiv.org/abs/2401.18079
  18. [18]Y. He, L. Zhang, W. Wu, J. Liu, H. Zhou, and B. Zhuang, “ZipCache: Accurate and efficient KV cache quantization with salient token identification,” in Advances in Neural Information Processing Systems, vol. 37, 2024.arxiv.org/abs/2405.14256
  19. [19]X. Li, Z. Xing, Y. Li, L. Qu, H.-L. Zhen, Y. Yao, W. Liu, S. J. Pan, and M. Yuan, “KVTuner: Sensitivity-aware layer-wise mixed-precision KV cache quantization for efficient and nearly lossless LLM inference,” in Proceedings of the 42nd International Conference on Machine Learning, vol. 267, 2025, pp. 36 451–36 485.DOI
  20. [20]Z. Chen, A. May, R. Svirschevski, Y. Huang, M. Ryabinin, Z. Jia, and B. Chen, “Sequoia: Scalable and Robust Speculative Decoding,” in Advances in Neural Information Processing Systems, vol. 37, 2024, pp. 129 531–129 563.arxiv.org/abs/2402.12374
  21. [21]R. Sadhukhan, J. Chen, Z. Chen, V. Tiwari, R. Lai, J. Shi, I. E.-H. Yen, A. May, T. Chen, and B. Chen, “MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding,” in The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2408.11049
  22. [22]X. Liu, J. Park, L. Hu, W. Kwon, Z. Li, C. Zhang, K. Du, X. Mo, K. You, A. Cheung, Z. Deng, I. Stoica, and H. Zhang, “TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput,” arXiv:2406.14066, 2024.arxiv.org/abs/2406.14066
  23. [23]Z. Li, Z. Chen, R. Delacourt, G. Oliaro, Z. Wang, Q. Chen, S. Lin, A. Yang, Z. Zhang, Z. Chen, Y.-H. Lai, X. Cheng, X. Miao, and Z. Jia, “AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding,” in Proceedings of the 21st European Conference on Computer Systems, 2026.arxiv.org/abs/2501.12162
  24. [24]G. X. Yu, Y. Gao, P. Golikov, and G. Pekhimenko, “Habitat: A Runtime-Based Computational Performance Predictor for Deep Neural Network Training,” in 2021 USENIX Annual Technical Conference (USENIX ATC 21), 2021. [Online]. Available: https://www.usenix.org/conference/atc21/presentation/yu
  25. [25]S. Lee, A. Phanishayee, and D. Mahajan, “Forecasting GPU Performance for Deep Learning Training and Inference,” in Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2025. [Online]. Available: https://arxiv.org/abs/2407.13853
  26. [26]H. Zhang, A. Ning, R. Prabhakar, and D. Wentzlaff, “LLMCompass: Enabling Efficient Hardware Design for Large Language Model Inference,” in 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), 2024, pp. 1080–1096.DOI
  27. [27]S. Williams, A. Waterman, and D. Patterson, “Roofline: an insightful visual performance model for multicore architectures,” Communications of the ACM, vol. 52, no. 4, pp. 65–76, Apr. 2009. [Online]. Available: http://dx.doi.org/10.1145/1498765.1498785
  28. [28]NVIDIA, “GeForce RTX 5090 Specifications,” https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/, 2025.
  29. [29]AMD, “AMD Instinct MI300X Accelerators,” https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html, 2023.
  30. [30]NVIDIA, “NVIDIA H100 Tensor Core GPU Datasheet,” https://resources.nvidia.com/en-us-gpu-resource/h100-datasheet-24306, 2024.
  31. [31]NVIDIA, “NVIDIA L40 GPU Datasheet,” https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/support-guide/NVIDIA-L40-Datasheet-January-2023.pdf, 2023.
  32. [32]A. Yang et al., “Qwen3 Technical Report,” arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
  33. [33]K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training Verifiers to Solve Math Word Problems,” arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
  34. [34]H. Lightman et al., “Let’s Verify Step by Step,” arXiv:2305.20050, 2023.
  35. [35]J. Austin et al., “Program Synthesis with Large Language Models,” arXiv:2108.07732, 2021.arxiv.org/abs/2108.07732
  36. [36]L. Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” arXiv:2306.05685, 2023.arxiv.org/abs/2306.05685
  37. [37]M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan et al., “Evaluating Large Language Models Trained on Code,” arXiv:2107.03374, 2021.arxiv.org/abs/2107.03374

Paper details

vlnc:W_kjq8pcgf
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents