betaFeedback

WorksComputer Science

Validation-driven Deployment of LLM Inference Platforms for Agentic Workloads

Abstract

Enterprises are moving agentic workloads from closed APIs to their own open-weight model deployments. For this reason, lifecycle-wide validation, from small-scale testing through sizing and pre-production to production, becomes all the more important. Typically, rollouts are evaluated with ad hoc tools whose accuracy is rarely checked. We present an end-to-end evaluation pipeline for agentic LLM serving and empirically ground the design choices that connect its stages. Our experiments show that replaying captured traces reproduces the per-session load of the original agent sessions: at the 95th percentile replay stays within 1.18x of the recorded input tokens while live re-runs reach 4.98x, so a small set of recorded sessions yields a repeatable per-session token load. A code-coverage analysis of vLLM further shows that the evaluated live, replayed, and synthetic workloads are complementary in this configuration: each exercises Python serving paths not reached in the measured runs of the others. In a case study rolling out four large mixture-of-experts (MoE) models on llm-d, experiments show that the target models reached about half the optimal concurrency, and correspondingly about 60% of the throughput, on captured workloads compared with public traces. We open-source the relevant tools and release a trace dataset.

Introduction

Agentic workloads are driving rapid growth in LLM inference demand in enterprises ([19, 4]). Coding assistants, terminal agents, and multi-step workflow automation issue tens to hundreds of model calls per task, each carrying a context that grows turn by turn. At this volume, relying on closed-model APIs becomes expensive ([41]), and it also raises concerns about data governance, vendor lock-in, and control over model versions. As a result, many organizations are moving to serve open-weight models on their own clusters ([12, 25, 43]), built on serving stacks such as vLLM ([21]), SGLang ([49]), and llm-d (llm-d, 2026).

Such an in-house deployment aims to provide an inference service that serves the organization’s agentic workloads at scale: it must support many concurrent users and agents, keep latency predictable as sessions grow, and match the task quality users would get from an external provider. Reaching this goal is not a single step. Every new model, hardware configuration, or serving feature passes through a sequence of stages before it reaches users, and each stage asks a different question. Small-scale testing checks that the model runs in the stack and matches its reference quality. Sizing depends on the shape of agentic sessions (growing prompts, parallel tool calls, sub-agent fan-out, and idle gaps while tools run), so it needs realistic load replayed at rising concurrency. Qualifying a release requires real agents solving real tasks, to show that the platform serves them as well as an external provider does. In production, the question becomes what changed between releases, which requires telling real regressions apart from the run-to-run variance of non-deterministic agents.

Evaluating across these stages faces two gaps. First, the stages are disconnected. Agent evaluation frameworks ([44, 6] (Shi et al. 2026)) measure agent quality but treat the inference endpoint as fixed, while load generators (ai-dynamo, 2026; Chandrasekar et al., 2026; Neural Magic, Inc., 2024) measure serving performance using synthetic prompts or replayed traces. Recent work such as XPerf ([45]) demonstrates that traces from real agentic applications can be collected and reproducibly replayed across LLM serving systems. However, XPerf focuses on benchmarking and profiling serving systems, whereas our goal is to provide a connected evaluation pipeline spanning agent development, deployment, and serving evaluation. Public traces (SemiAnalysis, 2026a; Qin et al., [35] drop the request content, so they cannot be re-issued against the target model. Because these tools rely on different data, what one stage captures cannot be fed directly to the next. Second, the evaluation choices are seldom checked. Every stage depends on which workloads to run, which traces to replay, and which tool to replay them with. Teams deploying models follow similar rollout steps on their own platforms (Baseten Model Performance Team, 2026; Lohmeyer & Horgan, [24]), but make these choices by experience, without measuring whether a workload exercises the serving paths agents use, whether traces represent the target traffic, or whether replay reproduces the recorded sessions. A poor choice gives a misleading answer: sizing a deployment on traces unlike its actual traffic yields capacity estimates that do not hold in production.

We address both gaps. To connect the stages, we design live benchmarking to record every model call in full, so that the runs used to measure quality also produce the traces needed for sizing, and build the pipeline around this shared replayable format. To ground the design, we empirically examine the choices that connect these stages and let the results shape the design. We then apply the pipeline while rolling out large mixture-of-experts (MoE) models on llm-d. We focus on performance and quality; functional API tests, which check that the deployed serving API handles each type of request correctly, serve as a first gate, and security or safety evaluation is out of scope.

Contributions. This paper makes the following contributions: (1) A connected lifecycle evaluation pipeline for agentic LLM serving that covers quality and performance from small-scale testing to production, combining live benchmarking, dependency-aware trace replay, synthetic workload generation, and regression analysis over a single trace format; (2) Empirical grounding of cross-stage design choices: vLLM statement coverage shows that the evaluated live, replayed, and synthetic workloads are complementary in this configuration: each reaches Python serving paths absent from the measured runs of the other families; an analysis of existing trace formats and replay systems identifies what agentic replay requires; and replay reproduces the per-session load of recorded agent sessions: it keeps the 95th-percentile input within 1.18× of the recording versus up to 4.98× for live re-runs, with stable load across repeated runs; (3) Lessons from real-world rollouts of four MoE models on H100 and H200 GPUs, showing how workloads and tools were used across stages. Public traces sufficed for initial configuration, but traces captured on all target models reached roughly half the optimal concurrency and 60% of the throughput, and exposed a routing imbalance before deployment; (4) Open-source tools and data: Workflow-level OpenTelemetry trace replay, per-session error reporting, and synthetic agent workload generation were contributed upstream to Inference Perf (Chandrasekar et al., [11]). We also extended Exgentic with container-native live benchmarking and released a dataset containing 10,056 agent sessions with full request contents.

Workflow Overview

When deploying a new version of a production agentic system, teams must quantify the impact of a broad range of configurations, including modifications to KV-cache mechanisms, hardware configurations (e.g., additional GPUs), model updates, and agent behavior. We first describe the stages a deployment goes through on its way to production and the questions teams need to answer at each stage (§ 2.1). We then derive from these questions the evaluation capabilities required for the development process, and map each capability to the respective component of our framework (§ 2.5).

Stages of Development

We view the path to deployment as a sequence of five lifecycle stages (Figure 1). Each stage is defined by its purpose, a question about the system, and by the evaluation that answers it:

Figure 1. Integrated evaluation along the path to deployment.

Stage 1: Small-scale testing. The model is deployed on a few GPUs and checked in increasing order of cost before any scale-out. Functional API tests come first. They send short, scripted requests through the deployed serving stack and check each response against known expected behavior, such as chat completion, streaming, tool calling, reasoning output, structured output, and clean errors for malformed requests. At this stage integration errors are caught cheaply, before any agent runs. Shape-only traces then drive initial sizing and tuning. Finally, real agents run on representative benchmarks to confirm that the model matches its reference quality on the intended use cases. This stage answers: Does the model and initial configuration work in our stack?

Stage 2: Workload capture. Before the platform can be sized or qualified, we must know what it will serve. Given the intended use cases, we target relevant open-source benchmarks and run them with real agents on the target model, producing a set of representative sessions. These sessions retain the full content, including inputs, outputs, and tool definitions (the trace record, item #6 in Figure 1). Running real agents at this point also surfaces gross quality problems, such as a model that cannot drive a given agent scaffold, long before the release is qualified. This stage answers: What will the platform be asked to serve?

Stage 3: Sizing. Once quality is established, the deployment grows to its target scale using the captured workload.

The required hardware depends on the session shape of agentic traffic rather than on individual requests: prompts that grow turn by turn, sub-agent fan-out, and idle gaps while tools execute. The recorded traces are replayed at increasing load to size the deployment and find where throughput saturates. This stage answers: How does serving scale up, and which configuration is optimal?

Pre-production

Stage 4: Pre-production. This stage is the acceptance gate. The candidate version is deployed end to end on the release images, with real agents, models, and tools, and is served with the real benchmark workloads. Agent success is compared with the previous version as well as external results. Pre-production also checks behavior under contention, such as priority and fairness between agent traffic and interactive users. This stage answers: Does it serve real agents on par with external baselines? A capacity gap found here loops back to sizing.

Production

Stage 5: Production. After deployment, live traffic is itself the benchmark, since no trace fully matches it. Latency and throughput are tracked per tenant at the router and at the engines, and each release is compared with its predecessor. Production usage also keeps evolving: new agents, tools, and user behaviors appear, and a version validated against yesterday’s workload may not hold up against today’s. This stage answers: Is it still right, and what changed? Degradation at this stage feeds back to pre-production and sizing.

Monitoring, Regression, and Analysis

Monitoring, Regression, and Analysis. Analysis and regression tools can be used to identify performance degradations during pre-production validation and production benchmarking. When a metric deviates from a previous release or an external baseline, the root cause must be isolated, whether it originates in the model, agent framework, serving infrastructure, or workload itself, enabling the change to be corrected, rolled back, or intentionally accepted. Because live agent executions are inherently non-deterministic, effective regression analysis must also distinguish true regressions from normal run-to-run variability. This capability answers: If something changed, what explains the observed behavior?

From Development Stages to Evaluation Needs

Answering the questions above requires a set of evaluation capabilities, which we now derive from the stages in Figure 1 and map to the components of our framework.

Live end-to-end benchmarking. Workload capture and pre-production both require running real agents, models, and benchmarks against a deployment, since synthetic prompt loops fail to capture the complexity and diversity of real-world agent interactions. During workload capture, live benchmarking characterizes the intended use cases and checks agent success against publicly available results; during pre-production, it monitors task success, latency, throughput, and resource utilization on the candidate stack in addition to comparing them against external baselines and prior releases (§ 3).

Full-content replayable traces. Collecting and repeatedly executing live workloads is costly and operationally challenging. Live benchmarking therefore records each request as a trace in a replayable format. Each record preserves the full multi-turn messages, tool definitions and calls, request parameters (e.g., sampling settings and reasoning effort), timestamps, token counts, and session identifiers. Inter-request dependencies are not explicit; replay infers them from message content and timing, then derives post-predecessor delays. In contrast, existing traces such as WEKA ([38]) hash away the input and are therefore insufficient for robust sizing. This is what makes faithful replay possible. Production traffic can be recorded in the same format, so once its capture is permitted, production traces can flow back into the workflow without conversion (§ 4.1).1

High-fidelity trace replay. Sizing requires reconstructing and executing the recorded workload against the serving stack at controlled, increasing load, while scheduling requests according to inferred dependencies and delays derived from recorded timestamps. Replaying traces isolates the serving stack from agent and tool variability and lets the same workload be re-run across hardware configurations and releases (§ 4.2).

Synthetic agentic workload generation. Synthetic workloads cover agentic execution patterns that are rare or absent in real traces, such as hierarchical agent spawning, latency-intensive tool calls, and long-running user-agent dialogues, to further facilitate configuration tuning in the sizing stage (Section 5).

Monitoring, analysis, and regression detection. The analysis and regression capability requires tools that track runs across releases, compare them, detect regressions, and explain them. In pre-production, these tools decide whether the candidate passes the acceptance gate by comparing it against prior releases and external baselines; in production, they compare per-release metrics of live traffic and flag drift as usage evolves. We combine rule-based checks with agent-assisted analysis to investigate root causes (§ 6).

While existing approaches for live benchmarking (e.g., Harbor ([40])) and trace replay (e.g., AIPerf ((ai-dynamo, 2026))) address parts of this process, they typically provide only partial coverage of the evaluation lifecycle. In contrast, our framework combines synthetic workloads, live benchmarking, trace replay, and analysis into a unified end-to-end pipeline, as described in the following sections.

Live Benchmarking

Unlike synthetic data generation or trace replay, live benchmarking directly measures end-to-end task quality, the only component of our pipeline to do so, making it the best proxy for the quality of agentic executions.

Hundreds of agentic benchmarks now exist, spanning software engineering (e.g., DeepSWE ([20])), hardware design (e.g., HWE-Bench ([14])), computer and terminal use (e.g., Terminal-Bench ([26])), office and cross-application workflows (e.g., AutomationBench ([39])), and mathematics (e.g., FrontierMath ([17]). Running them individually does not scale: each demands its own setup expertise and operational care. Frameworks such as Harbor ([40]), Inspect AI ([44]), and Exgentic ([6]) address this by consolidating multiple benchmarks behind a common interface, but consolidation alone does not suffice for agentic platform evaluation.

Coverage. The harness must run public benchmarks representative of expected deployment use cases, and allow adding proprietary benchmarks with modest development effort. Because benchmarks, agent scaffolds, and models each shape the traffic reaching the inference endpoints, and use cases may mandate particular models or scaffolds, the harness must also support all relevant combinations across the three axes.

Scale. Evaluation must execute many benchmarks concurrently as agentic tasks can take hours to complete. For example, Long-Horizon-Terminal-Bench 2.0 ((Li et al., 2026)) assigns tasks a three-hour budget and reports that 21% of solved tasks take over two hours. Execution must therefore be robust and consistent in local and cloud environments, handling transient failures and retries.

Trace recording fidelity. Concurrent execution is eventually bounded by the resources each task needs to stand up its benchmark and agent; some environments span multiple containers or virtual machines, consuming many gigabytes of memory and storage per task. To scale further, LLM call traces must be recorded at high fidelity once and then replayed concurrently at much larger scale.

Reporting. Each evaluation must report three distinct levels of measurement: task quality (benchmark success), task-level performance (end-to-end latency, token consumption, and mean per-call prefix-cache reuse, measured as the fraction of input tokens reported as cached), and per-call behavior (latency, output rate, LLM errors).

We address these requirements by extending Exgentic, an open-source framework for capturing, storing, analyzing, and recording agentic sessions, along three lines.

First, we adopt a fully container-native design2 in which an evaluation is represented as the composition of four independent dimensions—benchmark, agent, model, and task. Each benchmark is packaged as a self-contained Open Container Initiative (OCI)-compliant image, so evaluations run directly under Docker Compose or Kubernetes without a dedicated evaluation framework at runtime. The resulting architecture separates experimental methodology from runtime orchestration, making large-scale evaluations easier to reproduce across local and cloud deployments.

Second, every model executes behind a logging gateway that records each LLM call in full fidelity at the HTTP transport level. Inference Perf replays these records directly or as OpenTelemetry (OTel) GenAI spans Section 4. Each run thus stores the full messages, tool calls, latencies, and token usage alongside benchmark scores for later replay and analysis, improving auditability, debugging, and reproducibility.

Finally, a dashboard UI launches evaluations and displays their results, highlighting inference-platform issues such as LLM errors and latency alongside standard benchmark quality metrics § 6.

Trace Replay

Replayable Trace Formats

The traces captured during live benchmarking are consumed directly by replay. They record semantic request content (messages, tool calls and definitions, and request parameters) and serving metadata (token counts and timing). Because full request content is retained, replay can reconstruct shared prefixes and thereby induce prefix reuse, although the traces contain no explicit prefix-sharing or cache signal. Replay infers dependencies from content and timing, while agent hierarchy remains unannotated. Existing formats emphasize either agent behavior, as in SWE-smith (Yang et al., [48] and TraceLab Zhu et al., [50], or workload shape and cache behavior, as in WEKA SemiAnalysis, [38], Mooncake Qin et al., [35, 36], and Baseten AIPerf Contributors, [2]; none retains the structured content needed to re-issue agent requests verbatim.

Exgentic traces fill this gap: each LLM call is an OTel GenAI chat span holding the complete input and output messages, tool definitions, request parameters, token counts, and timestamps, so requests can be re-issued without synthesizing their content. We release 10,056 such sessions (241,674 LLM calls), spanning six benchmarks, five models, and five agent harnesses Exgentic team, [16]. Appendix § 13 compares the formats (Table 6), details the Exgentic schema, and shows an example session (Figure 8).

FormatMsgs.CallsDeps.Hier.Defs.Params.TokensTimingPrefix
AgentX (WEKA)✗✗✓✗✗✓✓✓
SWE-smith✓✓✗✗✗✗✗✗✗
TraceLab✗◐◐✓✓✓✓✓✓
Mooncake✗✗✗✗✗✗✗✓✓
Baseten◐✗◐✗✗✗✓✓✓
Exgentic✓✓◐✓✓✓✓✓✗

Table 6. Information available from each trace format (✓: explicitly recorded; ◐: incomplete or heuristically inferable; ✗: unavailable or not applicable). Msgs.: structured messages (roles and content); Calls: tool invocations; Deps.: execution dependencies between requests; Hier.: hierarchical agent relationships; Defs.: tool definitions; Params.: request parameters (e.g., max tokens, temperature); Tokens: token counts; Timing: timestamps or latencies; Prefix: a cross-request prefix-sharing or cache signal recorded in the trace itself (e.g., per-block prefix hashes or cached-token counts), not merely content from which one could be computed.

Figure 8. An example agent session from the released Exgentic trace dataset (§ 4.1).

Workflow-Level OTel Replay

These full-content traces drive workflow-level replay of agentic patterns involving tools, parallel branches, and multiple steps. Request-level replay reproduces arrival patterns and token lengths, conversation replay preserves sequential history, and graph-capable replay additionally schedules concurrent calls from a dependency graph. We contributed trace replay3 to Inference Perf: it reads agent traces exported in the OpenTelemetry (OTel) GenAI format OpenTelemetry Authors, [31], reconstructs each workflow, and executes it against the target endpoint.

The replay pipeline comprises trace ingestion, inferred dependency-graph construction, dependency-aware execution, and live-output propagation.

Rather than replaying requests from timestamps alone, the framework infers predecessor–successor relationships among LLM invocations. It adds causal edges when a request contains an assistant message matching an earlier span’s output, uses the closest non-overlapping earlier span as a temporal fallback, and applies transitive reduction. Roots run immediately; other calls wait for every predecessor and then observe their derived post-predecessor delay. The scheduler supports sequential chains, parallel branches, and fan-out/fan-in synchronization in the inferred graph.

Replay is also output-aware: live generated text and structured tool calls are substituted into dependent requests, rather than only recorded responses. To preserve tool-call fidelity, a turn with exactly one recorded function available in its tool definitions forces that function; turns with multiple or unmatched recorded functions require a tool call without selecting a specific function. This keeps live generation structurally aligned with the recorded tool-result messages reused downstream, without executing external tools.

Because a failed request can affect or cancel downstream requests, the per-load-condition error report we contributed to Inference Perf records request- and session-level failures (Appendix § 13).

Comparison with other replay systems. Table 7 in Appendix § 13 compares Inference Perf with five other replay systems; the broader evaluation workflow is replay-system agnostic.

CapabilityAIPerfInference PerfGuideLLMvLLMSGLangDynoSim
Recorded timing✓✓✓✓✓✓
Conversation replay✓✓✓✓✓✓
Live output propagation✓✓✓✓✓✗
Dependency-aware scheduling✓✓✓✗✗✓
Parallel branches✓✓✓✗✗✓
Fan-in synchronization✓✓✓✗✗✓
Workflow-level replay✓✓✓✓✓✓
Native OTel input✓✓✓✗✗✗
Live inference target✓✓✓✓✓✗

Table 7. Capabilities of representative trace-replay mechanisms, based on documentation and source code accessed October 21, 2026 (✓: supported; ◐: limited or approximated; ✗: unsupported). Cells report tool-level capability: capabilities in the same column may be provided by different loaders or modes and need not be composable in one replay path. “Timing” means scheduling from recorded timestamps or inter-call gaps; “workflow” means replay beyond independent request rows. DynoSim targets simulated engine cores, whereas the other mechanisms drive live endpoints.

Evaluation

We evaluate how closely replay matches the recorded session’s load and call/tool sequences, and how consistent repeated replays are (metric definitions and setup in Appendix § 13.§ 1).

Fidelity. We compute per-session ratios of replayed to recorded input tokens (RinR_{\mathrm{in}}), output tokens (RoutR_{\mathrm{out}}), and peak context, i.e., maximum input size (RpeakR_{\mathrm{peak}}). We also measure the normalized edit similarity of call-type sequences (workflow fidelity, WF) and the per-turn overlap of invoked tool names (tool-call fidelity, TF).

the workflow shape but sometimes picks a different tool at the same step.

We collect traces with Exgentic using the Claude Code [3] agent harness, by running GLM-5.3 [18] on 40 HWE-Bench tasks and DeepSeek-V4.1-Flash-1017 [15] on 40 CyberGym [46] tasks. We replay each trace with Inference Perf using its default tool-choice forcing policy. As a reference, not a competing method, we re-run the same tasks live in Exgentic; this shows how far a fresh sample of the agent drifts from its recording, the run-to-run variance that replay is meant to remove.

Table 1 presents the results. At the median, replay and live re-runs are similar: 0.74–1.05× for replay and 1.01–1.16× for re-runs. The tails differ sharply. Replay keeps the P95 RinR_{\mathrm{in}} within 1.18× and the P95 RpeakR_{\mathrm{peak}} within 1.30×, whereas re-runs reach 4.98× and 1.85×. The P95 RoutR_{\mathrm{out}} is 1.28–2.04× for replay and 3.84–4.52× for re-runs. Replay thus bounds per-session load, partly by design because it reuses recorded tool outputs and enforces tool selection, demonstrating that propagating live outputs over long sessions does not substantially erode this property compared to live re-runs. Replay also closely matches the recorded call-type and tool-name sequences: WF is 0.92–0.94 for replay vs. 0.68–0.70 for re-runs, and TF is 0.87–0.90 vs. 0.20–0.30, as expected due to tool-call enforcement.

TypeBenchmark (Model)RinR_{\mathrm{in}} Med.RinR_{\mathrm{in}} P95RoutR_{\mathrm{out}} Med.RoutR_{\mathrm{out}} P95RpeakR_{\mathrm{peak}} Med.RpeakR_{\mathrm{peak}} P95
ReplayHWE-Bench (GLM-5.3)0.911.120.741.280.891.14
Re-runHWE-Bench (GLM-5.3)1.014.981.093.841.011.85
ReplayCyberGym (DeepSeek)1.001.181.052.041.021.30
Re-runCyberGym (DeepSeek)1.143.971.164.521.061.76

Table 1. Per-session load ratios relative to the recorded traces (replayed or re-run total ÷ recorded total), median and P95 over 40 sessions per benchmark. Replay: replaying workload in Inference Perf; Re-run: reproducing live benchmark in Exgentic. DeepSeek stands for DeepSeek-V4.1-Flash-1017.

Consistency. We replay one trace per benchmark 50 times and compare the replays with each other. We report the coefficient of variation (CV) of per-replay time per output token (TPOT) and workflow consistency (WC), which is WF averaged over all replay pairs. WC is computed on call types only (WCrole_{\mathrm{role}}) and with each assistant step labeled by the tools it called (WCtool_{\mathrm{tool}}). Per-replay TPOT is token-weighted, TPOTi=∑rdr/∑r(or−1)\mathrm{TPOT}_i = \sum_r d_r / \sum_r (o_r - 1), where drd_r is the decode time of request rr (latency minus time to first token) and oro_r is its output token count, so short calls do not dominate.

Table 2 shows a TPOT CV of 5.2%–10.7%, so replay places a stable load on the server. Replays agree almost completely at the role level. For GLM-5.3 on HWE-Bench, agreement drops once tool choices are included: live sampling keeps

Benchmark (Model)TPOT CVWCrole_{\mathrm{role}}WCtool_{\mathrm{tool}}
HWE-Bench (GLM-5.3)5.2%92%86.8%
CyberGym (DeepSeek)10.7%99.6%99.3%

Table 2. Inter-replay consistency over 50 replays of one trace: CV of per-replay TPOT, and workflow consistency with call-type symbols only (WCrole_{\mathrm{role}}) and with tool-labeled assistant steps (WCtool_{\mathrm{tool}}), averaged over all replay pairs. DeepSeek stands for DeepSeek-V4.1-Flash-1017.

Synthetic Agentic Workload Generation

Trace-driven replay is limited to traces that have already been collected. We therefore contributed synthetic agentic workload generation to Inference Perf,4 which procedurally constructs agentic sessions from a few configurable parameters to stress-test dimensions of agent behavior that are hard to isolate in recorded traces: deep tool-use loops, wide parallel tool calls, recursive sub-agent delegation, long-context accumulation, and context compaction. The generator emits the same ReplayGraph used by trace replay, so synthetic and trace-driven runs share one execution and measurement path. Token counts, loop depth, fan-out, and latencies follow configured distributions, a declarative theme supplies semantically coherent domain content so the model under test responds plausibly, and all stochastic choices derive from a seed for reproducibility (more details in Appendix § 14).

Monitoring, Regression, and Analysis

Our dashboard launches, monitors, and analyzes Exgentic live benchmarking jobs. Users select a benchmark × agent pair, a model, and an inference endpoint, then follow each job’s progress, pass rate, token usage, and cost (Figure 2); each task’s downloadable trace is the same artifact consumed by trace replay (Section 4), so any inspected run can be replayed at scale (Figure 3–Figure 4). To assess deployment changes, regression suites, seeded automatically per model with fast, previously passing tasks, compare two executions down to the tasks that regressed from pass to fail (Figure 5). On request, a deterministic pass computes a run’s factual figures, Claude Code interprets them, and a validator rejects any report whose facts differ from the recomputed values (Figure 6 and Figure 7). Screenshots and further details are given in Appendix § 12.

Monitoring view showing evaluation jobs on the cluster

Figure 2. Monitoring view: every evaluation job on the cluster, with live progress, pass and error rates, LLM errors, throughput, cost, and runtime per task.

Run view showing outcome and token-usage summary, launch parameters, and actions to publish, extend, analyze, or compare the run

Figure 3. Run view: outcome and token-usage summary, launch parameters, and actions to publish, extend, analyze, or compare the run.

Task call view showing model requests, latency, streaming profile, and token counts

Figure 4. Task call view: every model request of a single task as recorded at the inference edge, with time to first token, latency, streaming profile, and token counts. The recorded calls can be downloaded and used for trace replay (Section 4).

Regression view

Figure 5. A model deployment is evaluated against a fixed suite of tasks, and the results are compared with those from a previous run.

Analysis report, performance metrics

Figure 6. Analysis report, performance metrics: per-task distributions of completion time, LLM time, cost, and input and output tokens, and per-call distributions of time to first token and output throughput, each with mean, p50, and p90.

Analysis report, agent trajectories

Figure 7. Analysis report, agent trajectories: per-task distribution of LLM calls and per-turn distributions of input, output, and cached tokens, followed by the agent-written interpretation of these distributions and a trace content analysis that cites specific tasks and turns.

Workload Complementarity via Code Coverage

The evaluation pipeline combines multiple approaches, each intended to capture behaviors the others miss. We quantify this by instrumenting a real vLLM deployment, running each approach as a separate workload, and measuring vLLM source coverage. Workloads that execute python statements no other workload reaches are considered complementary.

Experimental setup

We deploy vLLM ([21]) v0.31.0 serving DeepSeek-V4.1-Flash-1017 on an eight-H200-GPU node, in an image modified to collect Python statement coverage from the live server with coverage.py ([8]). Coverage is attributed to individual runs without redeploying the server. Appendix § 15.§ 1 gives the server command line (Figure 11) and the isolation mechanism.

The vLLM server command line for every run in these experiments

Figure 11. The vLLM server command line for every run in these experiments, wrapped in coverage.py. dspark is the speculative-decoding method named in --speculative-config; we use that spelling throughout rather than a display form. The reachability partition is computed from the modules this command line selects.

The installation has 419,671 measurable Python statements across 2,705 files. Much of this code is unreachable by construction: one vLLM install ships model implementations, hardware backends, and optional features that a given deployment never loads. Absolute coverage is therefore not meaningful on its own; we report each workload’s relative contribution. To make absolute figures interpretable, a static reachability analysis seeded from the modules the server command line selects rules out 180,352 statements (43.0%) this deployment cannot import, and we report coverage against the 239,319 that remain (Appendix § 15, Table 9).

Table 9. Statement coverage by static reachability class, over the union of all 24 runs, including server startup. Reachability is computed on the code as deployed in the serving image, so every measured file is classified. The three unreachable classes act as a control: a statement executed in one would falsify the partition, and no measurement window executes any.

ClassCoveredStatementsCoverage
Reachable here74,562239,31931.16%
Features not enabled066,3350%
Other architectures0109,9260%
Other accelerator04,0910%
Total74,562419,67117.77%

Workloads

We measure the statement coverage of vLLM serving DeepSeek-V4.1-Flash-1017 under five test families: Trace replay is represented by WEKA traces (SemiAnalysis, [37]) replayed with AIPerf and Exgentic agent traces replayed with Inference Perf, kept as two families because the corpora differ in content and in the tool driving them; synthetic agentic workloads are generated with Inference Perf; and live benchmarking runs CyberGym agents with Exgentic. The smoke family is a single non-streaming request, included as a cold-start reference point rather than as a pipeline stage.

Results

Table 3 shows what each family covered and the statements it alone reached. Appendix § 15 gives per-run figures (Table 8), reachable-statement coverage (Table 9), expanded unique coverage (Table 10), and the hard-to-reach scheduling, caching and KV-offload states each workload induced (Table 11).

FamilyRunsCov.Uniq.Reached only by this family
live18,777331Anthropic messages path; aborted KV transfer
weka38,43154streaming tool-call parsing; deferred offload fences
exgentic-traces38,96727sparse-attention short-context branch
smoke15,74820greedy sampling
synthetic168,7955sparse-SWA chunk cut-off
Union249,778437

Table 3. Coverage by test family, request-driven only (server startup excluded). Cov. is the union of the statements the family’s runs executed in their own isolated windows; Uniq. is the statements in that union that no other family reached.

Table 8. Workloads used in the coverage experiments, grouped by test family. All targeted DeepSeek-V4.1-Flash-1017 served by vLLM on eight H200 GPUs with tensor parallelism, expert parallelism, dspark speculative decoding, and GPU-to-CPU KV offloading; Figure 11 gives the exact command line. Duration is measured load time from each run’s own load metadata; the cybergym run has none, so its duration is wall-clock and includes model load and client setup, which is why we report no per-minute rate. Cov. is the statements that run executed while its own coverage window was open, measured in isolation. The union row counts distinct statements, so it is smaller than the column sum. The weka family replays shape-only traces, and the synthetic family runs synthetic agentic workloads (Section 5).

RunFamilyDuration (min)Cov.Workload
smokesmoke0.15,748Single non-streaming chat request (17 prompt tokens, 128 max tokens, temperature 0). Cold-start and single-call baseline.
cybergymlive136.88,777Live CyberGym agent benchmark on the Anthropic messages API—the only workload not using the OpenAI chat API. Large enough KV footprint to fill the CPU offload tier and force eviction.
weka-20m-c8weka20.08,245AIPerf streaming chat replay of WEKA multi-agent traces, concurrency 8; 291 requests.
weka-20m-c24weka20.08,331Same corpus and scenario at concurrency 24; 333 requests, 11 client timeouts (3.3% error rate).
weka-1h-c24weka60.08,332Same corpus at concurrency 24 over one hour; 822 requests, 40 timeouts. Long enough to outlive the metrics-window retention.
exgentic-2m-c10exgentic-traces2.08,738Inference Perf non-streaming replay of Exgentic/agent-llm-traces, 10 concurrent sessions; 49 requests. First coverage window after engine start, so it also captures lazy imports.
exgentic-20m-c10exgentic-traces20.08,678Identical configuration over 20 min; 393 requests, 2 failures.
exgentic-1h-c20exgentic-traces60.08,811Full trace corpus (no subset cap) at concurrency 20; 1,309 requests, 24 failures. Batches large enough to engage the fused expert router and the draft model’s own forward pass.
01-bare-no-toolssynthetic0.37,282Synthetic agentic baseline: 5 sessions, one turn each, no tools advertised, no fan-out.
02-multiturn-no-toolssynthetic0.57,2925 sessions of 3 turns with growing context, still no tools; 98.2% KV hit.
03-single-agent-tool-loopsynthetic2.28,4955 sessions, tool-loop depth 2 over an 8-tool catalog.
04-long-tool-loopsynthetic5.28,543Tool-loop depth 5, exercising argument variety across a longer loop.
05-multiturn-with-toolssynthetic3.07,9784 sessions of 3 turns, tool-loop depth 1, 6-tool catalog.
06-parallel-tool-callssynthetic4.98,516Tool-loop depth 2 emitting 3 parallel tool calls per step; 70.8% KV hit.
07-chat-with-tools-varyingsynthetic7.28,5314 sessions of 4 turns with tool-loop depth drawn uniformly from [0, 3], so depth varies per round.
08-orchestrator-fanoutsynthetic4.98,546Orchestrator spawning two non-recursive children, each running a 3-turn tool loop. De-graded: 3 request failures, 1 of 4 sessions completed, 33 of 60 events cancelled.
09-recursive-fanoutsynthetic2.68,602Recursive fan-out to depth 2 (root, children, grandchildren) over identical 200-token prefixes; 92 requests at 96.3% KV hit.
10-big-catalogsynthetic1.28,534One turn per session but a 200-tool catalog per agent; 8 requests carrying 303,569 prompt tokens at 50.7% KV hit.
11-mixed-everythingsynthetic25.78,67810 sessions blending every dimension: 1–3 turns, 40% fan-out to depth 2, tool-loop depth 0–2, lognormal turn sizes; 373 requests.
12-realistic-scalesynthetic66.18,730Production-shaped: 20 sessions, 30% fan-out to depth 2, tool-loop depth 1–4, the real 487-tool catalog, lognormal inputs averaging 110,000 tokens; 724 requests, 171,529,142 prompt tokens.
13-high-turn-countsynthetic16.78,530Long-conversation tail: 2 sessions of 32 turns, transcript growing each round; 97.8% KV hit.
14-high-tool-loop-depthsynthetic13.18,561Long-tool-loop tail: tool-loop depth 32, i.e. 33 calls within a single turn; 66 requests.
15-high-turns-and-tool-loop-distributionsynthetic85.28,59230 sessions with turns and tool-loop depth both drawn lognormally, giving many short sessions and a few very long ones; 725 requests, 171 of 896 events cancelled.
16-budget-capsynthetic1.78,541Fully recursive fan-out to depth 3 deliberately halted by a budget cap—an expected stop condition, not an error; 40 requests.
Union559.49,77824 runs across 5 families; 5,443 statements covered by every family
FamilyRunsDurationCoverageUniqueWhat only this family reached
live1136.88,777331The only family to use the Anthropic messages API: its request handler, router, protocol models, and the token scanner that drives incremental parsing on that path.
weka3100.08,43154The only streaming family, which shifts tool-call parsing to different partial-token positions. In the KV offload scheduler, the only family to reconcile deferred fences: blocks still in flight to CPU as their requests finish, reconciled when those blocks come back for reuse.
exgentic-traces382.08,96727The only family to enter the short-context branch of sparse attention, taken when a request offers fewer index candidates than the top-k width so the indices are filled directly rather than selected. Also tool-call parser states reached only by the recorded trace mix: parser-engine transitions, the token scanner’s anchor-rescan path, and the tool-argument JSON wrapper’s non-finite-value fallback.
synthetic16240.58,7955Two tool-argument coercion branches, one parser fallback, and the workspace-area cutoff in sparse-SWA chunk planning, reached only by this family’s very long prompts.
smoke10.15,74820Greedy sampling: at temperature 0 the sampling parameters take the zero-temperature branch that every other workload skips. Also the profiling-scope and first-request metrics paths that only a cold single request reaches.
Union24559.49,778437

Table 10. Table 10. Coverage by test family. Coverage is the union of the statements the family’s runs executed in their own isolated windows; unique is the statements in that union which no other family reached. Families are ordered by unique contribution. Across all five families, 9,778 distinct statements were covered, of which 5,443 were reached by every family. Separately, 437 statements were reached by exactly one family; those are what the unique column counts. The remaining statements were reached by more than one family but not all five. Startup is excluded here, unlike Table 9, so these figures are request-driven execution only. Unique coverage is a property of this particular partition into five families: a statement two families reach counts as unique to neither. Statements in the structured-output subsystem are excluded from the unique column, because they are reached at engine initialization rather than by any family’s traffic (Section D.1).

State, and the code that enters itStmtssmokesyntheticexgenticwekaliveWhat the state is (italics), and what it takes to reach it
Queue backpressure: defer a waiting request that already holds KV blocks<br>vllm/v1/core/sched/scheduler.py<br>878–883, 1345<br>req = from_queue.pop_request()<br>self.deferred_waiting.add(req)<br>if self._holds_kv_blocks(req):<br>step_skipped_kv_holding.appendleft(req)7–77––The scheduler cannot start every queued request in one step, and one it must postpone is already holding GPU cache blocks from a previous turn. It has to set that request aside without losing track of the blocks. Reaching it means offering more concurrent sessions than the token budget admits. Only the two generated and replayed agent families sustained enough queued work; the hour-long WEKA replays at concurrency 24 never filled the budget this way.
Prefix-cache hash collision: one block hash mapping to several blocks<br>vllm/v1/core/block_pool.py 68–69, 91–97<br>elif isinstance(blocks, KVCacheBlock):<br>self._cache[key] =<br>{blocks.block_id:<br>blocks, block.block_id: block}<br>if isinstance(blocks, dict):<br>return next(iter(blocks.values()))6–662–Two different cache blocks turn out to have the same content hash, so the prefix index can no longer store one block per key and must keep a set of them. Reaching it means sending many sessions whose prompts share an identical prefix. Recursive agent fan-out over a common system prompt is what produces this.
LRU heap compaction while evicting from the CPU tier<br>vllm/v1/kv_offload/cpu/<br>policies/lru.py 55–70, 168<br>if len(self._heap) < max(64,<br>2 * len(self._evictable): return<br>self._heap = [(self._ranks[k], k)<br>for k in self._evictable]<br>heapq.heapify(self._heap)9–7797The CPU tier is full and must evict. The heap that tracks least-recently-used blocks has accumulated so many stale entries that it is cheaper to rebuild it from scratch than keep sifting it. Reaching it means filling the CPU tier and then continuing to touch blocks already in it. Every sustained family reaches this; only weka drives the recency-update path that rebuilds the heap from the evictable set.
Deferred KV fence reconciled when a block is reused<br>vllm/distributed/kv_transfer/<br>kv_connector/v1/offloading/<br>scheduler.py 654–658, 1971–1973<br>for bid in job_status.<br>deferred_reference_block_ids or ():<br>self._block_id_to_pending_jobs<br>.setdefault(bid,<br>set()).add(job_id)8–––8–A request finished, but its cache blocks were still being copied to CPU. The server cannot free them yet, so it records a marker and settles the copy later, when something else asks for those same blocks. Reaching it means completing responses while transfers are still in flight. Reached only by streaming replay, where responses complete while transfers are still in flight.
Bulk cancellation across the waiting, KV-holding and deferred queues<br>vllm/v1/core/sched/scheduler.py<br>2608–2630<br>if request.status ==<br>RequestStatus.RUNNING:<br>running_requests_to_remove.add(request)<br>self.waiting.remove_requests( <br>waiting_requests_to_remove)6–––66Requests are dropped while still waiting rather than while running, so the scheduler has to pull them out of several different queues at once instead of retiring them normally. Reaching it means clients going away before they are served. Client timeouts under replay and terminated agent sessions both produce it; no generated workload did.
Speculative draft tokens trimmed and padded after scheduling<br>vllm/v1/core/sched/scheduler.py<br>2515–2547, 2917<br>del spec_token_ids[orig_num_spec_tokens:]<br>num_invalid = orig_num_spec_tokens<br>- len(spec_token_ids)<br>spec_token_ids.extend([-1] *<br>num_invalid)10–1010––Speculative decoding guesses several tokens ahead, but the scheduler admitted fewer tokens for this request than the drafter guessed, so the surplus guesses are cut and the gap padded out. Reaching it means batches large enough that drafting and admission disagree. Requires concurrent decode and chunked prefill in the same step.

Table 11. Hard-to-reach scheduling, caching and KV-offload states, and which workloads drove the serving stack into them. These are engine-internal transitions—queue backpressure, prefix-cache collisions, eviction, in-flight transfer reconciliation—rather than API surface, and each requires a particular kind of pressure rather than a particular request format. Each cell counts how many statements of that state’s code the family executed, out of the total in the Stmts column; a dash means the family never entered that code at all. Italics in the last column describe the state itself, and the sentence after says what a workload must do to provoke it. The states are identified by the statements that no single-request run reaches; line numbers are from the deployed image (vLLM v0.31.0). The smoke column is a dash throughout: a correct single request, served end to end, enters none of these paths.

Including server startup, the server-wide cumulative total of the 24 runs is 74,562 of the 239,319 reachable statements (31.16%), dominated by the 69,127 executed while loading the model, initializing KV caches, and warming up kernels. In each run’s own isolated coverage window, the request-driven union is 9,778 statements.

API Surface Coverage. A workload’s value lies in which code it reaches, not how much. Most unique statements reflect API and client choices rather than differences in engine behavior. The live family contributes 331 unique statements largely because it alone exercises the Anthropic Messages API; most of weka’s 54 and smoke’s 20 stem from their unique use of streaming and temperature-0 sampling, respectively. Because statements reached by more than one family are not counted as unique, these totals provide only a lower bound on how much the families differ.

The API surface is nonetheless a real thing to test, not an artifact of our setup: a deployment serves whichever endpoints its clients use, and in our experience (§ 8) a mismatched tool parser can degrade quality on one path while leaving others healthy, while a LiteLLM (BerriAI, [9]) gateway may convert between the OpenAI and Anthropic formats in ways that introduce errors exposed only on specific APIs. A suite covering one endpoint would miss both. The remainder of the column is not API surface at all: the exgentic-traces and synthetic entries involve no endpoint or client setting the other families lacked.

Load-Driven Paths. GPU-to-CPU offloading code was touched by every family, but at very different depths. The single-request smoke baseline reached only 4 statements of the CPU offload worker—worker setup, with no block ever transferred—and none of the eviction policy, whereas live reaches 112 statements of the offload worker and 26 of the eviction policy, and the weka replays 112 and 28 respectively.

Family-Specific Behaviors. Within those shared paths, individual families still reach distinct code. In weka, a request completed before its KV-cache copy to CPU had finished, leaving a deferred fence that must be reconciled when the block is reused; this is the one KV-offload path no other family reaches (Table 11). In live, a session was terminated, causing an abort of the in-flight KV-cache transfer. The exgentic-traces replays were the only family to enter the short-context branch of sparse attention, taken when a request offers fewer index candidates than the top-kk width.

Closing a Coverage Hole with a Synthetic Workload. While GPU-to-CPU offloading was entered by every sustained family, the reverse CPU-to-GPU restore path was executed by none of the 24 runs, exposing a real coverage hole. It requires a working set exceeding the GPU KV pool and revisits spaced far enough apart to evict a prefix first—a conjunction no workload in our set satisfied, but one a synthetic workload sets directly. This covered 124 new statements of the offload and tiering subsystem, demonstrating the value of synthetic generation (Appendix § 15.§ 2).

How much testing is enough?

Figure 12 plots the cumulative union of request-driven statements against cumulative testing time, per family and overall, with each series ordered shortest-run-first so the horizontal axis is the testing budget spent. Saturation is evident. The synthetic family reaches 8,581 statements after 2 cumulative minutes and 8,795 after 240.5: its two longest runs occupy 151 of those minutes and add 22 statements between them. The 1-hour weka run adds 33 statements over the 20-minute one at the same concurrency, and loses 32 others, so tripling the load time leaves coverage essentially unchanged. Long runs still measure sustained performance, but coverage stops distinguishing them early, so a coverage-guided suite would spend its minutes on more workloads rather than longer ones.

Cumulative request-driven statement coverage against cumulative testing time

Figure 12. Cumulative request-driven statement coverage against cumulative testing time. Within each series the runs are ordered shortest-first, so the xx value of a point is the time that series had spent once that run finished; because per-run coverage windows are independent of run order (Section D.1), the endpoint of each curve is the same set whatever order is chosen. The upper curve is all 24 runs, the lower three the multi-run families. The time axis is logarithmic: the first order of magnitude buys most of the coverage, and the last buys almost none. synthetic reaches 8,581 statements after 2 cumulative minutes and 8,795 after 240.5 minutes, so the 151 minutes of its two longest runs add 22 statements. All families together reach 9,406 after 126 minutes and 9,778 after 559; of the 372 statements added in that last factor of four, 331 come from the single 137-minute live run—a different workload rather than more time. The later runs still measure performance, which Table 8 shows varies across them.

Limitations

Similar statement coverage does not show that tests are redundant. Coverage collection is incomplete, since only vLLM’s Python code is measured, not its compiled kernels; a covered line was executed but not shown to be correct, and the approaches differ in what they can check; and the results come from a single model, hardware, vLLM configuration, and selected set of workloads (details in Appendix § 15.§ 3).

Case Study: Bringing Up MoE Models on LLM-d

We now show how the llm-d team (llm-d, 2026) uses the workflow of § 2 and our tools in practice. llm-d is an open-source, Kubernetes-native distributed inference platform that runs on model servers such as vLLM and SGLang. It serves open-weight models at scale on heterogeneous hardware, adding an inference-aware router (the endpoint picker, EPP), KV-cache offloading, prefill/decode (P/D) disaggregation, and wide expert parallelism. Each feature must be tuned per model, model server, and accelerator, and its value depends on the shape of the served workload.

We brought up four large MoE models: GLM-5.3 (GLM-5-Team et al., [18]), Nemotron 3 Ultra (Blakeman et al., [10]), DeepSeek-V4.1-Flash-1017 (DeepSeek-AI et al., [15]),5 and Kimi-K3 (Team et al., [42]), on H100 and H200 GPUs, following the stages of Figure 1, with the sizing stage run twice: first for the model servers, then for the router.

Small-scale testing. Functional API tests confirmed that chat completion, tool calling, and streaming work through the llm-d stack, and sizing arithmetic chose tensor and expert parallelism, GPU count, and KV-cache memory. With no captured workload yet, we tuned against WEKA traces (SemiAnalysis, [38])6 on H100 and NVIDIA Moon-trace traces (NVIDIA, [30]) on H200, replaying both with AIPerf. We replayed traces at eight increasing concurrency levels, restarting the server for a cold cache, and ran each configuration until performance degraded; these sweeps identified the operating point and guided tuning of prefix caching, CPU KV offloading, multi-token prediction (MTP), and P/D disaggregation.

Workload capture. Public traces describe someone else’s workload, and WEKA hashes away message content, so its requests cannot be re-issued against the target model (§ 4.1). We therefore selected benchmarks spanning the model’s potential uses, e.g., Terminal-Bench 2.1 (Merrill et al., [26]),7 HWE-Bench, and SWE-Bench Multimodal (MM) (Yang et al., [47]), which adds image inputs. We ran them with real agents on the deployed model using Exgentic (§ 3), producing full-content traces.

Sizing, step 1: model servers. We replayed the captured traces with the OTel trace replay of Inference Perf (Section 4), and used session duplication with cache-isolating markers to reach high concurrency from a limited set of recorded sessions. Compared with the public traces used for tuning, the captured workloads reached around half the optimal concurrency (the highest load level before performance degraded) for all target models, and per-GPU throughput was correspondingly lower, around 60% of that achieved with the public traces. We attribute this gap mainly to differences in the workloads themselves: completion lengths and session shapes depend on the model that generated the trace, and the public traces were not recorded on the target model, whereas the captured traces were.

Pre-production → sizing, step 2: router. With the model servers sized, we repeated the sizing stage across replicas to configure the router. The router balances a KV-cache prefix-affinity scorer, which keeps a session on the replica holding its KV cache, against a token-load scorer, which avoids overloading that replica. Flow control adds priority and fairness under contention: a latency-sensitive workload with a TTFT target receives higher priority and reserved capacity, while standard and batch traffic are capped at lower saturation ceilings. We first tuned the scorers by replaying WEKA traces with AIPerf at rising load across replica counts, and tuned flow control with fixed prefill-heavy and decode-heavy workloads, accepting a configuration only if the high-priority workload met its TTFT target.

We then replayed the captured agentic workloads at scale with Inference Perf. Terminal-Bench 2.1 sessions all begin with a large identical prompt prefix, and under this workload the router sent most traffic to the first few replicas that had cached that prefix, leaving the others almost idle: prefix affinity dominated because the token-load scorer never compensated for the cache reuse. We fixed the imbalance by switching the token producer feeding the token-load scorer from estimate to accurate token computation (real tokenization). The bug appeared only under a workload with the structure of real agent traffic, which production traffic is also expected to have: every session in these traces begins with the same tool definitions (about 17K tokens), followed by a system prompt of about 26K–27K characters that differs only slightly between sessions. In contrast, the WEKA traces scope prefix hashes to each session, so replayed sessions share no common prefix, and their first substantive requests vary widely (29K–98K tokens, p5–p95).8

Production: workload representativeness. After deployment, we checked whether captured workloads represent what the platform serves, by running an experiment comparing per-request input and output token-length distributions

of typical agentic workload on the GLM-5.3 deployment (a 4-day window of about 2.2M requests; aggregate serving-stack histograms only, with no request content or session data) against WEKA and four traces captured with Exgentic on the same model (HWE-Bench, CyberGym, HANDBOOK, and DeepSWE; Table 4). Each of these traces was closer to the typical agentic workload than WEKA in both input and output lengths, yet none matched it on its own, which supports capturing several use-case benchmarks rather than relying on a single public trace (more details in Appendix § 16).

WorkloadInput tokensOutput tokens
HWE-Bench0.1350.044
CyberGym0.1400.133
HANDBOOK0.1600.164
DeepSWE0.1610.081
WEKA0.2230.212

Table 4. KS distance between per-request token-length distributions of each workload and typical agentic workload on the GLM-5.3 deployment (lower is closer).

Lessons Learned

Each tool closed a gap the previous step left open. Public traces and a load ladder gave a starting configuration but described a different workload; Exgentic captured the intended use cases with full content, the trace format made them replayable, and dependency-aware replay with session duplication turned a limited set of recorded sessions into high-concurrency load. Together they moved two problems, a capacity shortfall and a routing imbalance, from production into sizing and pre-production, where they are cheap to fix. Table 5 summarizes the lessons.

LessonEvidence
Public traces suffice for initial configuration, not for sizing.Tuning on WEKA and Moontrace gave a working configuration, but captured workloads reached ∼0.5× the optimal concurrency and ∼60% of the throughput (Sizing, step 1).
Record and replay on the target model.Replay fidelity is highest when the original model is used, since its responses are closer to the recorded version, whereas equivalent fidelity cannot be expected from traces generated by other models.
No single benchmark matches the typical agentic workload.Every captured trace was closer to the typical agentic workload than WEKA, yet none matched it alone (Table 4).
Router bugs depend on the prefix structure of the workload.The routing imbalance appeared only with a constant shared prompt prefix, which WEKA lacks (Pre-production → sizing, step 2).

Table 5. Lessons from the llm-d bring-up, with the evidence for each.

Related Work

Agentic Platform Benchmarking. Serving providers describe how they evaluate their own platforms mostly in blog posts. Baseten’s day-0 Kimi K3 deployment ([7]) follows stages close to ours: quality checks (Grade School Math 8K (GSM8K) ([13]), the Berkeley Function-Calling Leaderboard (BFCL) ([34]), and the Kimi Vendor Verifier ([28], rerun as a regression check), configuration sweeps over parallelism, batch size, speculation, and routing across vLLM, SGLang, and an in-house engine, and finally scale-out with KV-aware routing. However, it reports no quality scores or end-to-end latency or throughput, and does not specify the workloads behind its sweeps, optimization, or scale validation, so it is unclear whether these stages use distinct representative workloads.

The llm-d project ([5]) runs a controlled prefix-cache routing benchmark on 16 H100s using synthetic multi-tenant traffic with shared prefixes rather than agentic sessions. The llm-d 1.0 announcement ([24]) reports headline agentic coding results and links to a Qwen3-Coder guide and Prism that document the workload, deployment configuration, and concurrency sweep, but its workload is also synthetic (Inference Perf) rather than captured agent traces, and it reports no run-to-run variance. For quality, Moonshot ([28]) and MiniMax ([27]) publish verifiers that third-party hosts run against their deployments, but these are model-specific. None of these efforts drives sizing with replayed agentic traces or links the stages of the release cycle; our functional API tests are model-agnostic and cover the full deployed path, including the gateway.

Trace Replay. Because agent behavior depends on non-deterministic LLM outputs, identical tasks can generate different requests, making serving-system comparisons unfair. XPerf ([45]) addresses this with fine-grained replay of traces captured from real agentic applications, reproduced across serving systems. It presents output token ID enforcement, which is prototyped in vLLM. AgentReplay ([32]) further argues that greedy decoding and length-only replay are insufficient, since token content drives prefix-cache hits and mixture-of-experts routing. It therefore forces the recorded output tokens, and optionally expert selections and tool delays, during otherwise normal decoding. Token-wise forcing requires control inside each engine’s decoding loop and router, which is impractical for Inference Perf: it targets many inference engines (vLLM, SGLang, and others) through their standard serving APIs. We instead make replay output-aware: live outputs are propagated to dependent requests, and recorded tool choices are enforced through the API’s tool-choice parameter, which keeps tool-call behavior aligned with the trace on any compatible engine (Section 4).

Our work is complementary: it connects quality evaluation, workload capture, scalable replay, controlled stress testing, and regression analysis across the deployment lifecycle.

Conclusion

Bringing an agentic serving platform from a first deployment to production requires more than running a single benchmark. Instead, deployment proceeds through a sequence of stages, each asking a different question and needing different data. We presented an end-to-end evaluation pipeline that follows this lifecycle and connects its stages through a single replayable trace format. Live benchmarking with real agents records every model call in full, so the runs that measure quality also produce the traces that drive sizing; the same format can also carry production traces back into the pipeline upon user consent.

Dependency-aware replay preserves the per-session load characteristics of recorded agent sessions: the 95th-percentile input differs by at most 1.18× from the recording, versus up to 4.98× under live re-runs, and vLLM statement coverage shows that the evaluated live, replayed, and synthetic workloads reach complementary Python serving paths in this configuration. In the bring-up of four large MoE models on llm-d, traces captured on the target model reached about half the optimal concurrency, and correspondingly about 60% of the throughput, of public traces, and exposed a routing imbalance before production. Experiments on the GLM-5.3 deployment showed that the captured workloads are closer to typical agentic traffic than public traces.

Large-scale evaluation of synthetic workload generation remains future work. We contributed the replay and synthetic workload tools in Inference Perf, the Exagentic extensions for live benchmarking, and a dataset of 10,056 agent sessions with full request content.

AI USAGE DISCLOSURE

AI is both the subject of this work and a tool used throughout the paper. The platform we evaluate serves large language models, and our experiments exercise GLM-5.3, Nemotron 3 Ultra, Kimi K3, and DeepSeek V4.1 Flash (§ 3–§ 8). The rest of this section describes AI as a tool.

How we worked with AI. Each author worked with AI coding assistants throughout the project, mainly Claude Code and Codex. An assistant typically supported the same author across every phase of the work: surveying existing tools, trace formats, and literature; designing experiments and metrics; implementing the tools; running experiments and analyzing their results; and drafting and reviewing text, tables, and figures, for this paper and for related artifacts such as blog posts. Because the same assistant, often across several sessions, had already worked on a tool or survey, it carried that context into later steps. For example, the comparisons of trace formats and replay mechanisms (Table 6 and Table 7) grew out of AI-assisted surveys carried out while building the tools; the tables were drafted with AI, re-verified against documentation and source code by assistant sessions that had done that survey work, and then checked by the authors. Authors also had assistants consult one another’s sessions for details. Design decisions, metric definitions, and the final text were made or approved by the authors.

The remaining paragraphs describe the specific uses where AI output affects data, results, or claims, and how each is checked.

Data generated by AI. Exgentic itself is conventional software, but the traces it records are executions of AI agents: the released Exgentic traces (§ 4.1) were produced by five agent harnesses, including Claude Code, driving five models to solve benchmark tasks. Two of the benchmarks involve an LLM beyond the agent under test: τ2\tau^2-bench uses an LLM-based user simulator, and BrowseComp-Plus uses an LLM judge that determines the task score. These calls are removed from the released traces, but they shape the agents’ trajectories and, for BrowseComp-Plus, the reported scores. The traces we captured for this paper (HWE-Bench, CyberGym, Terminal-Bench 2.1, SWE-Bench Multimodal, DeepSWE, and HANDBOOK) come from benchmarks scored by deterministic programmatic checks (unit tests or rubric verifiers), without an LLM judge or user simulator. Synthetic workloads (Section 5) are generated procedurally, without a model.

AI inside our tools at runtime. Trace replay (Section 4) is driven by recorded traces, not by an AI agent; however, the responses of the model under test are fed into the downstream requests, so model output becomes part of the replayed workload by design. The dashboard’s automated run analysis (§ 6) uses Claude Code (with Claude Sonnet by default; the model is configurable) to write the interpretive part of each report (verdict, recurring errors, trajectory observations). All factual figures are computed deterministically beforehand, and a validator rejects any report whose facts differ from the precomputed values, so the model can interpret the numbers but not change them.

AI in reported results. The replay-fidelity and consistency metrics (Section 4) are computed by a script generated with Claude Code; the metric definitions were chosen by the authors.

References. AI assistants were used to find candidate references. Every reference was checked by the authors to exist and to support the claim it is cited for.

Responsibility. The authors take full responsibility for the content of this paper, including all text, code, data, and results produced with AI assistance.

Dashboard Details

Launching. The launch form offers only benchmark×\timesagent combinations whose container images are published in the registry, so a user cannot select a pair that would fail to start. A UI launch renders the same Helm chart as the command-line workflow, so dashboard launches are interchangeable with CLI runs. Runs can be restricted to task subsets, extended, retried, or cancelled.

Monitoring and inspection. Each benchmark run appears in the monitoring view as a single row with live progress, pass rate, mean reward, token usage, and cost, alongside the launcher and current cluster load (Figure 2). The run view shows per-task reward and pass/fail status, and each task’s trace records every model call with its prompt, response, tool calls, tokens, and latency. Aggregate usage is broken down into input, output, and cached tokens, latency, time to first token, throughput, and cost (Figure 3–Figure 4).

Regression suites. A suite is a fixed collection of tasks drawn from selected benchmark×\timesagent combinations. For each model, suites are seeded automatically from historical results with small, diverse sets of fast-running tasks that previously passed, giving a low-cost regression check. A suite is immutable, so two of its executions, for example across deployment versions, can be compared directly. When two executions of a suite are compared, each task is categorized as pass, fail, or error in each run, and users can drill down into any disagreement category (Figure 5). Running a suite against a model launches each cell as an ordinary run and records which runs belong to that suite execution.

Automated analysis. On request, a deterministic pass computes completion, latency, token and cost distributions, tool usage, and structured errors; Claude Code then adds interpretive content, such as an overall verdict, recurring errors and anomalies, and trajectory-level observations (Figure 6 and Figure 7). A validator rejects any report whose facts differ from the precomputed values, so the model can interpret the numbers without altering them. A rule-based mode produces the same report without a model. Each analysis runs under a cost budget, and its own model calls, tokens, and cost are recorded and shown alongside the report. Because the report summarizes the distributions and metrics of a run’s traces, it can also serve as a summary of the workload capture stage before those traces are replayed.

Sharing. The dashboard has a public and an internal view. Runs are internal by default and can be published individually; the public view lists only published runs and, for launching, offers only benchmark×\timesagent pairs that have a published run. Reports and comparisons can be exported as standalone HTML or JSON.

Implementation. The dashboard is a FastAPI/htmx web application deployed on OpenShift. It keeps no database: the cluster’s Jobs and the results stored in object storage (IBM COS) serve as the state, with a small per-run metadata file for the launcher, visibility, and label. Analyses and comparisons run as separate, sandboxed Kubernetes Jobs that write their artifacts next to the run’s results.

Trace Replay Details

Table 6 compares the information retained by the trace formats discussed in § 4.1.

Trace formats. WEKA records subagent patterns, timing, token counts, and prefix structure, but hashes message content, so requests cannot be reproduced verbatim. SWE-smith (Yang et al., [48]) captures software-engineering actions and outcomes, while TraceLab (Zhu et al., [50]) normalizes sanitized traces from Claude Code and Codex, omitting original user messages and tool input/output text. Mooncake (Qin et al., [35, 36]) and Baseten (AIPerf Contributors, [2]) prioritize workload shape and cache behavior; Baseten retains a flattened completion prompt, but not structured messages, tool calls, or tool definitions.

Exgentic traces. Each Exgentic session contains one OTel GenAI chat span per LLM call, retaining complete input and output messages as typed text, tool-call, and tool-result parts; tool definitions; request parameters; token counts; finish reasons; and start and end timestamps. Each session also records its task outcome. Shared identifiers link tool calls to their results, including parallel calls. Dependencies between LLM requests are not explicit, but can be inferred from message content and timing; agent hierarchy is likewise not annotated, so subagent delegation may appear only as a tool call. The released traces retain calls from the agent under test, excluding benchmark scaffolding and failed retries, and provide call-level start and end timestamps. Figure 8 shows an example session.

Table 7 compares AIPerf,9 Inference Perf,10 GuideLLM,11 vLLM bench serve,12 SGLang bench serving,13 and NVIDIA DynoSim.14

Format and loader behavior. Basic Mooncake rows preserve timestamps, token counts, and prefix hashes, enabling serving-shape replay with synthesized prompts but not workflow reconstruction. AgentX (WEKA) traces add declared subagent structure; AIPerf, GuideLLM, and DynoSim use it to reconstruct spawn/join topology, with some adapters also inferring flattened branches from prefix hashes and timing. These formats may therefore preserve graph shape without original prompt and response bodies. Exgentic traces instead retain OTel GenAI spans with full request snapshots, recorded responses, tool definitions, request parameters, and timestamps. Inference Perf infers dependencies using message/output matches and temporal fallback, propagates live responses, and preserves recorded shared-prefix, tool-call, and image structure. Its loader accepts span JSON from files, directories, or Hugging Face datasets, and sessions can be duplicated with cache-isolating markers to scale load without artificial prefix sharing.

Replay-system distinctions. Capability labels in Table 7 do not imply equivalent implementations. AIPerf and GuideLLM provide graph replay for WEKA traces, while their other loaders have different semantics. SGLang propagates live responses in sequential agentic-trace mode, whereas timestamped Mooncake replay uses placeholder assistant responses. DynoSim preserves agentic dependency graphs but executes them against simulated rather than live engines. Inference Perf combines OTel ingestion, recorded timing, dependency-aware parallel and fan-in scheduling, live-output propagation, and a live inference target in one path.

Error reporting. When responses are propagated, a request failure affects every downstream request. We contribute to Inference Perf a report for each configured load condition (“stage”), emitted at the end of the run (Figure 9), that lists successful requests and failures grouped by HTTP status or transport error.15 It also counts bad tool-call substitutions, where malformed live tool-call arguments cause the recorded assistant message to be substituted, and reports failed sessions and downstream events cancelled because a predecessor failed.

Per-load-condition error report

Figure 9. Per-load-condition error report emitted at the end of a trace replay run (Section 4): successful requests, bad tool-call substitutions in which the recorded assistant message replaced a live response with malformed tool-call arguments, and failed requests grouped by cause.

Replay Evaluation Details

Workflow fidelity.

WF=1−Lev⁡(Srec,Srep)max⁡(∣Srec∣,∣Srep∣)\mathrm{WF} = 1 - \frac{\operatorname{Lev}(S_{\mathrm{rec}}, S_{\mathrm{rep}})}{\max(\lvert S_{\mathrm{rec}}\rvert, \lvert S_{\mathrm{rep}}\rvert)}

where SS is the session call-type sequence [user,assistant,tool,...]. Each LLM invocation contributes one assistant symbol, followed by one tool symbol for every tool call it generates. A user symbol is emitted when a conversation opens, or after an LLM call that invokes no tools.

Tool-call fidelity.

TF=∣Trec∩Trep∣max⁡(∣Trec∣,∣Trep∣)\mathrm{TF} = \frac{\lvert T_{\mathrm{rec}} \cap T_{\mathrm{rep}}\rvert}{\max(\lvert T_{\mathrm{rec}}\rvert, \lvert T_{\mathrm{rep}}\rvert)}

computed over the multisets of invoked tool names and averaged over the turns in which the recorded or the replayed trace issues at least one tool call. WF and TF are averaged across replayed sessions.

Load ratios. For each session ss with turns ii, we compare the replayed (or re-run) load to the recorded load:

Rin,s=∑iins,irep∑iins,irecR_{\mathrm{in},s} = \frac{\sum_i \mathrm{in}^{\mathrm{rep}}_{s,i}}{\sum_i \mathrm{in}^{\mathrm{rec}}_{s,i}}
Rout,s=∑iouts,irep∑iouts,irecR_{\mathrm{out},s} = \frac{\sum_i \mathrm{out}_{s,i}^{\mathrm{rep}}}{\sum_i \mathrm{out}_{s,i}^{\mathrm{rec}}}
Rpeak,s=max⁡iins,irepmax⁡iins,irecR_{\mathrm{peak},s} = \frac{\max_i \mathrm{in}_{s,i}^{\mathrm{rep}}}{\max_i \mathrm{in}_{s,i}^{\mathrm{rec}}}

Input tokens are the server-reported prompt size, including cached tokens. Output tokens are the server-reported completion count, including reasoning. Each side is aggregated over all of its own turns rather than over paired turns, so turns that a replay drops or adds count toward its load. A ratio of 1× means the session placed the same load on the server as the recording. We report the median and the 95th percentile (P95, linearly interpolated) over the 40 sessions of each benchmark (Table 1).

At the median, replay and live re-runs both stay near 1×. The one exception is replay output on HWE-Bench (0.74×): replayed GLM-5.3 turns generate fewer tokens than the recorded ones, while replayed input and peak context stay at 0.91× and 0.89×. The two modes diverge in the tail. A re-run session can take a different path, e.g., more steps or different tool outputs. These differences compound to up to 4.98× the recorded input, 4.52× the recorded output and 1.85× the recorded peak context. Replay reuses the recorded tool results and constrains tool selection (Section 4), so its P95 input and peak-context ratios stay within 1.12–1.30×. Output is the least constrained quantity under replay, because only the structure of each turn is enforced, not its length. Even so, its P95 ratio of 1.28–2.04× is less than half of the re-run tail.

TPOT CV. CV = σ/μ\sigma/\mu, the sample standard deviation over the mean of per-replay TPOT. TPOT does not depend on how much each replay generates, so it should be stable on a stable server even though live output substitution changes the outputs.

Workflow consistency.

WC=2R(R−1)∑i<j(1−Lev⁡(Si,Sj)max⁡(∣Si∣,∣Sj∣))\mathrm{WC} = \frac{2}{R(R-1)} \sum_{i<j} \left(1-\frac{\operatorname{Lev}(S_i,S_j)}{\max(|S_i|,|S_j|)}\right)

the mean over all pairs of the RR replays, where SiS_i is replay ii’s call-type sequence, built as for WF over its successful LLM calls. WCrole\mathrm{WC}_{\mathrm{role}} uses the call-type symbols only. WCtool\mathrm{WC}_{\mathrm{tool}} also labels each assistant symbol with the names of the tools it called. A score of 100% means every replay followed the same workflow.

Context. When this work began, no other surveyed system publicly supported OTel replay. AIPerf added Weka/DAG and Exgentic replay roughly one and three months later; GuideLLM added comparable capabilities roughly four to six months later. This chronology establishes public availability, not design provenance, and the workflow is replay-system agnostic.

Consistency setup. The two traces come from GLM-5.3 on HWE-Bench task chipsalliance_caliptra-rtl-594 and DeepSeek-V4.1-Flash-1017 on CyberGym task 1237. Each is replayed 50 times with Inference Perf, giving 1,225 replay pairs per trace.

Synthetic Workload Generation Details

This appendix details the synthetic agentic workload generator summarized in Section 5.

Controllable trace patterns. Sessions may consist of multiple conversational turns, repeated tool-calling loops, parallel tool calls, and recursive sub-agent spawning. Context evolves naturally as conversations progress, causing prompt lengths and KV-cache utilization to grow over time. The generator also supports explicit context compaction, where accumulated history is replaced by a shorter summary once a configurable threshold is reached, enabling evaluation of systems under both context growth and context-reset scenarios.

Themes. Prompt sizes, response lengths, tool catalog sizes, and interaction latencies are controlled through statistical distributions, while a theme supplies the content. A theme is a declarative description of a domain (e.g., software engineering, retrieval-augmented generation, or incident response) consisting of a system prompt, a tool catalog with descriptions and parameter schemas, per-tool result templates, an objective template, introductory documents (e.g., an issue report), and domain filler snippets (e.g., code, log, and CI lines). Each session draws a theme from a weighted mix and pins one value per entity category (service, symptom, file, symbol, test), so every turn refers to the same subject: the first user turn presents the objective and an introductory document, tool results are rendered from the tool’s template, and filler pads each turn to its sampled token budget. Token counts therefore follow the configured distributions, while the prompts stay semantically coherent, so the model under test emits plausible tool calls and answers rather than reacting to random text.

Graph construction. The generator emits a ReplayGraph, the same DAG representation used by trace replay, so synthetic workloads exercise the identical scheduling, dependency, output-substitution, and metric-collection paths, keeping comparisons with trace-driven runs meaningful. Each node represents a single LLM invocation. During a tool loop, the model produces one or more tool calls, receives synthetic tool outputs, and iterates until a final answer is emitted. Fan-out patterns are modeled through dispatch_agent-style actions that create parallel child agents, each executing its own workflow before returning results to a parent aggregation step. All stochastic decisions, including branching, tool-loop depth, token counts, and sub-agent creation, are derived from a configuration-controlled random seed, making generated workloads fully deterministic and reproducible. Figure 10 shows an abbreviated synthesized session.

Abbreviated synthetic session showing a software-engineering agent interaction

Figure 10. Abbreviated synthetic session (theme code-change task), configured with tool loop depth=3 which generates 3 tool calls and tool results. Pinned entities (orders-api, apply discount, src/billing/tax.py) recur throughout the interaction.

SYSTEM: You are an autonomous software-engineering agent ... tools=[list_dir,
read_file, grep_code, find_symbol, git_diff, run_tests, run_command, edit_file]

USER: <context> def OrderHandler(cart, ...) ... pytest tests/test_cache_... FAILED
</context> Issue #474 (orders-api) Title: incorrect tax rounding File
"src/billing/tax.py", function apply_discount AttributeError: ’NoneType’ has no
attribute ’amount’

ASSISTANT: grep_code("apply_discount")

TOOL: 78 matches in orders-api src/billing/tax.py:516: total = total - ...

ASSISTANT: find_symbol("apply_discount")

TOOL: apply_discount -> src/billing/tax.py:105

ASSISTANT: ...

TOOL: ...

USER: Final turn. Answer now.

ASSISTANT: <final answer, 80 tokens>
coverage run --rcfile=/etc/coveragerc --data-file=/coverage/.coverage \
  -m vllm.entrypoints.openai.api_server \
  --model deepseek-ai/DeepSeek-V4.1-Flash-1017 \
  --served-model-name deepseek-ai/DeepSeek-V4.1-Flash-1017 \
  --tensor-parallel-size 8 --enable-expert-parallel \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,
      "draft_sample_method":"probabilistic"}' \
  --tokenizer-mode deepseek_v41 --reasoning-parser deepseek_v41 \
  --tool-call-parser deepseek_v41 --enable-auto-tool-choice \
  --mm-processor-cache-type shm --allowed-media-domains deny-all.invalid \
  --enable-prompt-tokens-details --max-model-len 1048576 \
  --gpu-memory-utilization 0.90 --safetensors-load-strategy prefetch \
  --enable-ep-weight-filter \
  --compilation-config '{"max_cudagraph_capture_size":64}' \
  --engram-config '{"cpu_offload":true}' \
  --kv-transfer-config '{"kv_connector":"OffloadingConnector",
      "kv_role":"kv_both","kv_connector_extra_config":{
      "spec_name":"TieringOffloadingSpec","cpu_bytes_to_use":549755813888,
      "blocks_per_chunk":4}}' \
  --block-size 64 --host 0.0.0.0 --port 8000

Workload complementarity: coverage tables

This appendix holds the supporting material for the coverage experiments of § 7: the exact server command line (Figure 11), the static reachability partition and the per-run isolation mechanism (Section D.1), the individual runs and their workloads (Table 8), coverage against the statements this deployment can statically reach (Table 9), what each test family covered and the statements it alone reached (Table 10), the hard-to-reach scheduling, caching and KV-offload states each workload drove the stack into (Table 11), how coverage accumulates as runs are added (Figure 12), and the follow-up experiment that closed the one coverage hole those 24 runs left open (Section D.2).

Server configuration and measurement method

Figure 11 gives the command line verbatim. The static reachability partition below is computed from the modules this command line selects, so it is reproduced here in full.

Static reachability partition. Starting from the modules the command line in Figure 11 selects, we walk import edges with ast, resolving imports and following the lazy registries that map a configuration string to a module. Of the 419,671 measurable statements, 180,352 (43.0%) sit in files this deployment cannot import at all: 109,926 belong to other model architectures, 66,335 to features the command line does not enable, and 4,091 to the served architecture built for a different accelerator. Table 9 reports coverage per class. The unreachable classes act as a control: no run executes any statement in them.

Per-run isolation. To allow measuring coverage per run, without requiring a full redeploy of the image, which can take several minutes per run, we leverage coverage.py’s dynamic contexts with ctrace core. Before a run’s traffic starts, every measured process is switched into a named context; when the run ends, all processes are switched into a unique internal idle context and the data is flushed. The per-run figures are therefore independent of run order.

Two caveats follow from the mechanism. Attribution is per-process at signal-delivery time, so a process may execute a few lines—an idle loop, a log line, the tail of an in-flight request—between the harness ending a run’s traffic and that process handling the switch; those lines are attributed to the run just ended. And a run’s window captures lazy first imports and one-time construction as well as traffic: exgenic-2m-c10 was the first measured window after engine start, so some of its statements are initialization rather than traffic-driven.

Excluding engine initialization from unique coverage. The second caveat distorts unique coverage directly, so we correct for it. The structured-output subsystem (vllm/v1/structured_output/, together with the LazyLoader construction in vllm/utils/import_utils.py) is built when the engine starts, regardless of whether any request asks for a grammar. Its lines appear in whichever run holds the first measured window, and the code involved confirms the reading: the covered statements are module imports, class-body field declarations, backend __init__ bodies, and a request-validation function that returns immediately when a request carries no structured-output parameters. No family in this set sends structured-output requests, and no grammar was compiled. We therefore exclude these 109 statements from the unique column of Table 10; without the exclusion they would be credited to exgentic-traces as 109 of 136 unique statements, implying that trace replay exercises three grammar backends it never invoked. The exclusion is applied uniformly to every family, and affects only this one; the coverage column and all cumulative figures are left untouched.

The three caches, in plain terms. Table 11 touches three different stores, and they are easy to confuse:

  • The GPU KV cache holds the attention keys and values for tokens already processed, in GPU memory, so a model does not recompute them for every new token. It is divided into fixed-size blocks (64 tokens here). Blocks no longer referenced by any request go on a free queue to be handed out again.

  • The prefix cache is an index over those blocks, keyed by a hash of the tokens they contain. When a new request begins with the same tokens as an earlier one—the same system prompt, or the same conversation so far—the server reuses the cached blocks instead of recomputing them. This is why agent workloads matter: every turn resends the whole transcript, so almost all of it is a repeated prefix.

  • The CPU offload tier is a second, larger pool in host RAM (512 GiB here). When the GPU cache fills, blocks are copied out to CPU rather than discarded, so they can be brought back if needed. That tier fills too, and then it must evict: an LRU policy decides which blocks to drop.

A single request exercises almost none of this. It has no earlier prefix to reuse, never fills the GPU cache, so nothing is offloaded and nothing is evicted. The states below are the transitions that only appear once several sessions compete for the same finite cache.

Closing the KV-restore coverage hole with a targeted synthetic workload

The coverage analysis of § 7 left one gap that no workload in Table 8 closed: GPU-to-CPU offloading was entered by every sustained family, but the reverse CPU-to-GPU restore path was never executed. This appendix reports the follow-up experiment that closed it, because it is the clearest demonstration we have of what a parametrically controlled synthetic workload buys that neither replay nor live agents did.

The target. The restore path is entered through submit_load in vllm/v1/kv_offload/cpu/gpu_worker.py, which forwards to load_handler.transfer_async.

Why no earlier workload reached it. A block is promoted back to GPU only if it was evicted from GPU first: lookup() in vllm/v1/kv_offload/tiering/manager.py returns early on a primary cache hit and never consults the CPU tier. Reaching the restore path therefore requires a conjunction of two properties that no recorded corpus in our set happened to satisfy at once: a working set that exceeds the GPU KV pool, so that eviction occurs, and revisits to an evicted prefix that are spaced far enough apart in wall-clock time that the prefix is evicted before it is asked for again. The WEKA and Exgentic replays and the CyberGym live run all filled and evicted from the CPU tier—Table 11 shows them reaching the LRU compaction path—but their traffic never produced the second property.

Workload design. We used the shared_prefix synthetic data generator in Inference Perf, which constructs GG groups of prompts sharing a per-group system prompt of a configured length. Figure 13 gives the configuration.

Workload configuration for closing the KV-restore coverage hole

Figure 13. The workload configuration that closed the KV-restore coverage hole (Inference Perf, shared_prefix generator). The two parameters that matter are the working set, set by num_groups × system_prompt_len to 1.37× the GPU KV pool so that eviction is guaranteed, and the offered rate, held well below measured capacity so that revisits to a prefix are spaced far enough apart for it to be evicted first.

The sizing argument is explicit. The deployment reports a GPU KV pool of 9,580,564 tokens (about 149,696 blocks at --block-size 64). We chose 200 groups × 65,536 tokens = 13,107,200 tokens, or 1.37× the pool: above 1.0×, so eviction is guaranteed, and far enough above that the approximate pool figure read from the server log cannot push the true ratio below 1. The 512 GiB CPU tier is large enough that this working set does not saturate it.

Result. Table 12 reports the outcome on two independent signals. The pod had restarted before this run, so the server counters start from a true zero baseline.

Server metric (CPU →\to GPU)beforeafter
kv_offload.total_bytes_total0108.1 GB
kv_offload.size_count02,816 ops
kv_offload.total_time_total03.33 s
GPU →\to CPU bytes (for reference)0193.8 GB
Coverage, gpu_worker.py
line 891, defcovered at import
line 894, docstringnot covered
line 895, bodycovered, request-driven
File (under vllm/)+stmts
distributed/.../offloading/scheduler.py82
v1/kv_offload/cpu/manager.py25
distributed/.../offloading/worker.py6
distributed/.../offloading/metrics.py3
v1/kv_offload/tiering/manager.py3
distributed/.../offloading/events.py2
v1/kv_offload/cpu/gpu_worker.py1
v1/kv_offload/cpu/policies/lru.py1
v1/kv_offload/tiering/metrics.py1
Offload/tiering subsystem124

Table 12. The KV-restore run. Server metrics are vllm:kv_offload.* counters read from the server before and after the run, from a true zero baseline. Coverage is from the run’s own isolated window: line 895 is the entire executable body of submit_load, and request-driven means it was executed inside that window rather than at import. Line 891 is the def, executed at import by any run that loads the module, which is why it cannot serve as the signal.

Limitations in detail

The coverage results of § 7 highlight the complementary value of different testing approaches by identifying code uniquely exercised by specific workloads. However, given the limitations below, similar statement coverage should not be immediately interpreted as evidence that tests are redundant: two harnesses may execute the same statements while checking different properties of the result.

  • Python only. Only the interpreted parts of vLLM (the API, Scheduler, Sampling, and KV-Offloader) are measured. Parts of vLLM, and specifically the attention and matrix-multiply kernels, are compiled C++, CUDA, or Rust and are not visible.

  • Execution is not correctness. A covered line was executed, not shown to behave correctly; different approaches have different result-checking capabilities (e.g., live checks quality, while other approaches can only identify performance issues or crashes).

  • Single configuration. Results are for one model on one hardware configuration of one vLLM version. We expect the broad ordering of approaches to transfer, but we have not tested this: with the live and smoke families represented by a single run each, the ranking rests on few observations, and the specific statement sets will certainly not transfer.

  • Unique coverage is sensitive to the family partition. A statement reached by two families counts as unique to neither, so adding a workload can only reduce another family’s unique count. The figures in Table 10 therefore describe this set of five families, not the approaches in general.

  • The reachable-statement count is an over-estimate. Because reachability is resolved per file, the 239,319-statement denominator is an over-estimate and 31.16% is correspondingly a floor. A module imported for one symbol counts as reachable even when most of its body cannot run (for example, the vendored CuTeDSL and FlashAttention kernel emitters: 27,096 statements stay reachable though most emit kernels that a sparse-MLA inference server never calls).

Production Workload Representativeness

We experimented with typical agentic workloads on the GLM-5.3 deployment. We compared per-request token lengths against the public WEKA trace and the traces captured before deployment.

Data and confidentiality. Traffic is subject to confidentiality restrictions, so we report only aggregate statistics: the per-request input and output length histograms exported by the serving stack (vLLM Prometheus metrics, 20 buckets from 1 to 10610^{6} tokens), which contain token counts but no request content or session identifiers. The comparison is therefore at bucket resolution. A session-level comparison, covering calls per session, session duration, and arrival patterns, is out of the scope of this paper.

Setup. We use a 4-day traffic window of about 2.2M requests, and bin every candidate workload into the same 20 buckets. The candidates are the public WEKA trace ([37]) (94,298 main-agent requests, with token counts as provided by the dataset) and traces captured with Exgentic on CyberGym, DeepSWE, HWE-Bench, and HANDBOOK ([33]) using the same model (1.6K–16.4K calls each). We report the Kolmogorov–Smirnov (KS) distance between cumulative distributions (0 means identical). A second, non-overlapping 4-day window yields nearly identical workload distributions (KS 0.02–0.06) and the same conclusions.

Results. The typical agentic workload is long-context with short outputs: the median request carries 50K–100K input tokens and produces 200–500 output tokens. As Table 13 and Figure 14 show, WEKA is the furthest from our workload in both input and output lengths. Its input distribution is shifted toward longer prompts (median 100K–200K tokens, with about a third of requests above 200K), and its output distribution has fewer short responses than our workload. The captured traces all fall in the workload’s input range, but their shapes differ: CyberGym produces shorter outputs, HANDBOOK has a long output tail, and DeepSWE concentrates its inputs at 100K–200K. HWE-Bench is the closest overall, with an output distribution that nearly matches the workload’s distribution (KS 0.04, about 5× closer than WEKA). No single benchmark reproduces the typical agentic workload on its own, which supports capturing several use-case benchmarks rather than relying on a single trace.

WorkloadInput tokensOutput tokens
HWE-Bench0.1350.044
CyberGym0.1400.133
HANDBOOK0.1600.164
DeepSWE0.1610.081
WEKA0.2230.212

Table 13. KS distance between each workload and the GLM-5.3 typical agentic workload, per request (lower is closer).

Distribution of per-request input and output token lengths

Figure 14. Distribution of per-request input and output token lengths for the GLM-5.3 typical agentic workload, the public WEKA trace, and the four traces captured on the target model. Each point is the share of requests in one workload histogram bucket, plotted at the bucket’s upper edge.

References

  1. [2]AIPerf Contributors. Baseten trace replay. AIPerf documentation, 2026. https://github.com/ai-dynamo/aiperf/blob/main/docs/tutorials/baseten-trace.md. Accessed October 2026.
  2. [3]Anthropic. Claude Code. https://github.com/anthropics/claude-code, 2025. Accessed October 28, 2026.
  3. [4]Aubakirova, M., Atallah, A., Clark, C., Summerville, J., and Midha, A. State of AI: An empirical 100 trillion token study with OpenRouter. arXiv preprint arXiv:2601.10088, 2026. https://arxiv.org/abs/2601.10088.
  4. [5]Ayoub, M., Harnik, D., Smith, T., Swain, K., Wang, X., Yin, H., and Yan, K. KV-cache wins you can see: From prefix caching in vLLM to distributed scheduling with llm-d. llm-d blog, September 2025. https://llm-d.ai/blog/kvcache-wins-you-can-see. llm-d blog.
  5. [6]Bandel, E., Yehudai, A., Eden, L., Sagron, Y., Perlitz, Y., Venezian, E., Razinkov, N., Ergas, N., Shachor Ifergan, S., Shlomov, S., Jacovi, M., Choshen, L., Ein-Dor, L., Katz, Y., and Shmueli-Schuerer, M. General agent evaluation. arXiv preprint arXiv:2602.22953, 2026. https://arxiv.org/abs/2602.22953. Presented at the ICLR 2026 Workshop on Agents in the Wild.
  6. [7]Baseten Model Performance Team. How to build a Day-0 API for Kimi K3, July 2026. https://www.baseten.co/blog/how-to-build-a-day-zero-api-for-kimi-k3/. Baseten blog.
  7. [8]Batchelder, N. and contributors. Coverage.py: Code coverage measurement for Python, 2026. https://github.com/nedbat/coveragepy.
  8. [9]BerriAI. LiteLLM: Call 100+ LLM APIs in the OpenAI format, 2026. https://github.com/BerriAI/litellm.
  9. [10]Blakeman, A., Thomas, A., Jhunjhunwala, A., Gupta, A., Khattar, A., Rajfer, A., Renduchintala, A., Asif, A., Vavre, A., Miranda, A. F., Bilal, A., Zaman, A., Hotchandani, A., Shukla, A., Bercovich, A., Ficek, A., Gronsky, A., Kondratenko, A., Steiner, A., Ye, A., Bukharin, A., Milesi, A., Taghibakhshi, A., Gatti, A., Liu, A., Kumar, A., Phanishayee, A., Mahabalashwar, A. S., Klein, A., Zuker, A., Geifman, A., Bhiwandivalla, A., Subramaniam, A., Santilli, A., Fulks, A., McHarg, A., Tao, A., Skliar, A., Agrusa, A., Srivastava, A., Verma, A., Shors, A., Warno, A., Llaquet, A.-J. S. I., Mehta, A., Nowaczynski, A., Jain, A., Aithal, A., Poojary, A., Ahamed, A., Mishra, A., Thekkumpate, A. K., Sohrabizadeh, A., Kaur, A., Vem, A., Dattagupta, A., Anandan, B. S., Sadeghi, B., Lanir, B., Schifferer, B., Nushi, B., Kartal, B., Thiede, B., Rouhani, B. D., Deng, B., Schatz, B., Ginsburg, B., Wang, B., Nemire, B., Norick, B., Dang, B., Westphal, B., Yu, B., Khailany, B., Catanzaro, B., del Mundo, C., Aarish, C., Lee, C., Hwang, C., Sakr, C., Wang, C., Truong, C., Cui, C., Cheng, C., Hsieh, C.-P., Zhang, C., Deng, C., Patel, C., Alexiuk, C., Cosgrove, C., Munley, C., Harvey, C., Parisien, C., Shen, C., Li, C., Neale, C., Gao, C., Meurillon, C., Gil, D., Su, D., Zhao, D., Corneil, D., Afrimi, D., Egert, D., Korzekwa, D., Lo, D., Machlab, D., Serebrenik, D., Sorokin, D., Gitman, D., Levy, D., Stosic, D., Mosallanezhad, D., Yu, D., Karamyan, D., Donia, D., Debroy, D., Narayanan, D., O’Kelly, D., Peri, D., Nathawani, D., Wu, D., Rekesh, D., Kakwani, D., Plummer, D., Anh, D., Yu, D., Jiang, D., Kim, D., Poorkay, D., Riach, D., Stosic, D., VanStee, D., Meng, E., Minasayan, E., Lin, E., Long, E. M. P., Sarafin, E., Segal, E., Lantz, E., Evans, E., Ning, E., Chung, E., Harper, E., Pham-Hung, E., Tramel, E., Yang, E., Galinkin, E., Pounds, E., Goncalves, E. G., Briones, E., Wu, E., Bakhturina, E., Tsykunov, E., Dobrowolska, E., Ladhak, F., Memarian, F., Wang, F., Jia, F., Soares, F., Frujeri, F. V., Chen, F., Lin, F., Galko, F., Sun, F., Siino, F., Hou, F., Agam, G. H., Kaplun, G., Bhatt, G., Prasad, G., Kulshrestha, G., Armstrong, G., Shen, G., Borghesi, G., Neskovic, G., Batmaz, G., Lam, G., Mason, G., Pauloski, G., Nalbandyan, G., Chlebus, G., Karch, G., Liu, G.-T., Zhang, G., Huang, G., Maron, H., Qian, H., Elisha, H., Ren, H., Kumar, H. K. S., Hud, H., Nover, H., Hall, H. S., Iso, H., Ngo, H., Hum, H., Sahota, H., Wang, H., Soni, H., Tamoyan, H., Li, H., Chen, H., Li, H., Wang, H., Nguyen, H., Chiles, I., Galil, I., Shahaf, I., Gitman, I., Shovkun, I., Loshchilov, I., Guehring, I., Schen, I., Levy, I., Neeman, I., Moshkov, I., Golan, I., Putterman, I., Choi, J., Slowikowski, J., Kautz, J., Scowcroft, J. P., Casper, J., Mitra, J., Glick, J., Chen, J., Oliver, J., Xu, J., Zhu, J., Song, J., Zhang, J., Jiao, J., Zeng, J., Lou, J., King, J., Zhang, J., Wang, J., Choi, J., Chu, J., Conway, J., Guman, J., Jatko, J., Rausch, J., Kamalu, J., Roberts, J., Greco, J., Mensel, J., Alben, J., Yang, J., Cohen, J., Raiman, J., Jennings, J., Mabry, J., Pierce, J., Daw, J., Vialard, J. V., Yi, J., Parmar, J., Jain, K., Zhu, K., Briski, K., Cheung, K., Luna, K., Willowhawk, K., Wyss, K., Santhanam, K., Shih, K., Kong, K., Nguyen, K., Bhardwaj, K., Sivamani, K. S., Krommydas, K., Puvvada, K. C., Pawelec, K., Anik, K., Keprios, K., Day, K., McAfee, L., Du, L., Derczynski, L., Ding, L., Liu, L., Wu, L., Kadoch, L., Wei, L., Vega, L., Robinson, L., Su, L., Segbroeck, M. W., Mikulski, M. J., de Melo, M. R., Sypula, M., Fathi, M., Sreedhar, M. N., Chandran, M. T., Kilaru, M., Ashkenazi, M., Cuevas, M., Romeijn, M., Chochowski, M., Cai, M., Mozolewski, M., Klíegl, M., Stepniewska-Dziubinska, M., Patelka, M., Machczynski, M., Novikov, M., Ferrato, M., Golub, M., Samadi, M., Corpuz, M., Wang, M., Wu, M., Price, M., Boubdir, M., Schaffer, M., Andersch, M., Boone, M., Gschwind, M., Lightstone, M., Loh, M., Bien, M., Zawalski, M., Gill, M., Martinez, M., Khona, M., Chrzanowski, M., Houston, M., Ma, M., Lee, M., Fawzy, M., Dabbah, M., Shoeybi, M., Patwary, M., Mulepati, N., Nabwani, N., Dhameja, N., Hennouni, N., Hereth, N., Pinckney, N., Algarici, N., Assaf, N., Haber, N., Knight, N., Reamaroon, N., Quak, N., Bhatia, N., Desai, N., Ludwig, N., Tajbakhsh, N., Xu, N., Ailon, N., Juluru, N., Nitin, N., Masad, O., Rybakov, O., Hrinchuk, O., Kuchaiev, O., Weissmann, O., Delaileau, O., Olabiyi, O., Argov, O. U., Puny, O., Tropp, O., Ribalta, P., Bhattacharya, P., Lampropoulos, P., Mannan, P., Shamis, P., Legresley, P., Gibbons, P., Molchanov, P., Morkisz, P., Dykas, P., Jin, P., Aquilanti, P.-Y., Xu, P., Januszewski, P., Laskiewicz, P., Jannaty, P., Gurumurthy, P., Thombre, P. P., Varshney, P., Gundecha, P., Tredak, P., Meng, P., Wan, Q., Mahabadi, R. K., Oberman, R., Garg, R., Sri-Tharan, R., Kandu, R., Sanadhya, R., El-Yaniv, R., Zilberstein, R., Shafipour, R., Macalisang, R., Tian, R., Kovacs, R., Pi, R., Izzo, R., Shahbazyan, R., Garg, R., Puri, R., Neves, R. F., Zhao, R., Borkar, R., Gala, R., Islam, R., Clark, R., Hesse, R., Kirby, R., Waleffe, R., Watve, R., Koren, R., Banner, R., Zhang, R., Hewett, R. J., Prenger, R., Stewart, R., Egashira, R., Mahdavi, S., Paliwal, S., Singh, S., Modi, S., Dave, S., Shinagawa, S., Kriman, S., Bhaskar, S., Lym, S., Kariyappa, S., Satheesh, S., Murari, S. V., Pasumarthi, S., Mishra, S., Muralidharan, S., Hara, S., Narentheran, S., Anandaraj, S., Na, S., Bak, S., Bak, S., Sameni, S., Mard, S., Panev, S., Henneman, S., Poulos, S., Mor, S., Acharya, S., Ghosh, S., Sreenivas, S. T., Mendelson, S., Kotek, S., Wang, S., Aharon, S., Gharghabi, S., Lin, S.-C., Chen, S., Fan, S., Baskaran, S., Gopa, S., Prabhumoye, S., Pachori, S., Toshniwal, S., Ding, S., Krishnamurthy, S., Singh, S., Sun, S., Das, S., Thottakara, S. A., Ithape, S., Majumdar, S., Singhal, S., Singudasu, S. H., Bhuvanapalli, S., Veecham, S., Sergienko, S., Alborghetti, S., Ge, S., Rong, S., Devare, S. D., Rao, S., Barua, S. K., Ha, S., Gai, S., Gunasekar, S., Panguluri, S., Gupta, S., Hinzburh, S., Priyadarshi, S., Akter, S. N., Abramovich, T., Bui, T., Varshney, T., Ter-Hovhannisyan, T., Ene, T.-D., Kong, T., Do, T., Zhang, T., Moore, T., Blankervoort, T., Moon, T., Mitra, T., Balough, T., Grzegorzek, T., Hliwiak, T., Asida, T., Natan, T. B., Keren, T., Ronen, T., Salim, T., Wang, T., Rebedea, T., Konuk, T., Vashishth, T., Karpas, U., De, U., Noorozi, V., Srinivasan, V., Elango, V., Agrawal, V., Cui, V., Korthikanti, V., Mehta, V., Rao, V., Wu, V., Kurin, V., Robison, L., Su, L., Segbroeck, M. W., Mikulski, M. J., Lavrukhin, V., Anisimov, V., Pham, V., Jiang, W., Ahmad, W. U., Ishihara, W., Du, W., Ping, W., Chai, W., Dai, W., Helmholz, W., Jennings, W., Zhu, W., Prazuch, W., Ren, X., Yu, X., Breek, Y., Chen, Y., Yu, Y., Chen, Y., Galron, Y., Karnati, Y., Choi, Y., Meyer, Y., Wu, Y.-F., Zhang, Y., Lin, Y., Geifman, Y., Fu, Y., Kwon, Y., Yao, Y., Guvvala, Y., Huang, Y., Liu, Y., Moshe, Z., Newell, Z., Wang, Z., Li, Z., Zhu, Z., Yang, Z., Liu, Z., Yan, Z., and Wertheimer, Z. A. Nemotron 3 ultra: Open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning, 2026. https://arxiv.org/abs/2606.15007.
  10. [11]Chandrasekar, A., Varghese, S., Kramberger, J., Slabe, B., Wang, C., and Tang, Y. Inference Perf: A benchmarking tool for GenAI inference. Journal of Open Source Software, 11(124):10314, 2026. doi: 10.21105/joss.10314. https://doi.org/10.21105/joss.10314.
  11. [12]Clark, C. The open weight models that matter: June 2026. OpenRouter blog, June 2026. https://openrouter.ai/blog/insights/the-open-weight-models-that-matter-june-2026/.
  12. [13]Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
  13. [14]Cui, F., Hou, H., Luo, Z., Yin, C., and Liang, Y. HWE-Bench: Benchmarking LLM agents on real-world hardware bug repair tasks. arXiv preprint arXiv:2604.14709, 2026. https://arxiv.org/abs/2604.14709.
  14. [15]DeepSeek-AI, Xu, A., Li, B., Lin, B., Xue, B., Xian, B., Xu, B., Wu, B., Zhang, B., Deng, B., Yu, C. C., Jin, C., Lin, C., Dong, C., Wang, C., Feng, C., Lu, C., Zhao, C., Deng, C., Zhang, C., Xu, C., Zhao, C., Shao, C., Wang, C., Zhang, C., Dai, D., Yang, D., Chen, D., Huang, D., Wu, D., Li, D., Li, E., Fu, E., Zhou, F., Zhou, F., Lin, F., Yuan, F., Xia, F., Dai, F., Hao, G., Li, G., Chen, G., Cao, G., Fan, G., Meng, G., Li, G., Zhang, H., Ma, H., Shen, H., Li, H., Yu, H., Zhang, H., Deng, H., Xu, H., Xu, H., Zhong, H., Guo, H., Jiang, H., Li, H., Qin, H., Wen, H., Liang, H., Huang, H., Liu, H., Zhang, H., Luo, H., Yang, H., Xu, H., Yuan, H., Huang, H., Luo, H., Cai, H., Chen, H., Ji, H., Zhang, H., Wang, H., Wu, H., Ding, H., Tang, H., Wang, H., Cao, H., Gao, H., Qu, H., Zeng, H., Yang, J., Jin, J. H., Zhang, J. H., Zou, J. X., Yu, J., Zhou, J., Chen, J., Huang, J., Zhao, J., Tang, J., Zhou, J., Tong, J., Li, J., Zhu, J., Wang, J., Ye, J., Li, J., Xu, J., Ding, J., Lu, J., Hu, J., Yan, J., Zhai, J., Chen, J., Hu, J., Zhou J., Xu, J., Xiang, J., Yun, J., Yuan, J., Cheng, J., Zhu, J., Wang, J., Chen, J., Hu, J., Yu, J., Guo, J., Pei, J., Sun, J., Jiang, J., Qiu, J., Zhou, J., Liu, J., Li, J., Li, J., Song, J., Guo, J., Dong, K., Chen, K., Gao, K., Guan, K., Yuan, K., Hong, K., Xu, K., Zhao, K., Ji, K., Zhang, K., Zhou, K., Yu, K., Zhang, L., Wang, L., Zhang, L., Wang, L., Gao, L., Zhao, L., Xu, L., Guo, L., Luo, L., Fu, L., Deng, L., Wang, L., Zhang, L., Chen, L., Chen, L., Huang, L., Ma, L., Wang, L., Di, M. S., Mei, M., Ye, M., Cui, M., Zhang, M., Zhang, M., Tang, M., Zhang, M., Wei, M., Chen, M., Liu, M., Zhou, M., Xu, M., Yang, M., Wang, M., Chen, M., Shentu, N., Wang, N., Ning, N., Huang, P., Cong, P., Wang, P., Xin, P., Ren, P., Yan, P., Zhang, P., Kang, Q., Tang, Q., Wang, Q., Li, Q., Zhu, Q., Li, Q., Chen, Q., Du, Q., Guo, Q., Xu, R., Ding, R., Hu, R., Tian, R., Yu, R., Zhu, R., Xu, R., Yang, R., Xia, R., Lu, R., Geng, R., Hong, R., Ge, R., Zhang, R., Sun, R., Pan, R., Wang, R., Chen, R., Xu, R., Tian, R., Shen, R., Zhang, R., X., R., Liu, S. H., Lu, S., Zhou, S., Chen, S., Cai, S., Nie, S., Chen, S., Hu, S., Lin, S., Ran, S., Liu, S., Jia, S., Bai, S., Feng, S., Xu, S., Liu, S., Hu, S., Ma, S., Wang, S., Feng, S., Gong, S., Lin, S., Yu, S., Zhou, S., Yang, S., Wang, S., Guo, S., Pan, S., Yu, S., Cao, S., Lin, S., Chen, S., Chen, S., Zhou, S., Ni, T., Yun, T., Jin, T., Pei, T., Ye, T., Lin, T., Ji, T., Cui, T., Yue, T., Yu, T., Xiong, T., Zeng, W., Liu, W., Zhang, W., Xu, W., Zeng, W., Zhao, W., Liu, W., Liang, W., Pang, W., Luo, W., Yao, W., Gao, W., Shao, W., Yang, W., Zhang, W., Wang, W., Huang, W., Yan, W., Zhang, W., Gao, X., He, X., Li, X., Li, X., Wang, X., Zhang, X., Wei, X., Bi, X., Liu, X., Wang, X., Qu, X., Chen, X., Zhang, X., Nie, X., Zou, X., Li, X., Guo, X., Chu, X., Cheng, X., Liu, X., Xie, X., Xu, X., Liu, X., Liu, X., Yu, X., Li, X., Yao, X., Chen, X., Jiang, X., Yang, X., Yang, X., Chen, X., Wang, X., Zhong, X., Su, X., Liu, X., Lin, X., Fan, X., Zhao, X., Fu, X., Yan, Y. C., Jiang, Y. H., Wu, Y. T., M., Y. W., Wang, Y. Z., Gao, Y., Yang, Y., Zhang, Y., Ma, Y., Huang, Y., Li, Y., Li, Y., Meng, Y., Zhao, Y., Sun, Y., Wang, Y., Ye, Y., Yin, Y., Wu, Y., Qian, Y., Tao, Y., Yu, Y., Zhang, Y., Jiang, Y., Wang, Y., Ding, Y., Shi, Y., Peng, Y., Zhai, Y., Wu, Y., Xiong, Y., Wang, Y., He, Y., Zhou, Y., Luo, Y., Zhong, Y., Wang, Y., Wang, Y., Zhang, Y., Chen, Y., Tan, Y., Wei, Y., Ma, Y., Yang, Y., Liu, Y., Cai, Y., Wei, Y., Wang, Y., Yang, Y., Zhuo, Y., Guo, Y., Wu, Y., Wu, Y., Zhang, Y., Bian, Y., Cheng, Y., Ou, Y., Sun, Y., Xu, Y., Sun, Y., Li, Y., Liu, Y., Yao, Y., Han, Y., Wang, Y., Wu, Y., Meng, Y., Zou, Y., Li, Y., Wang, Y., Xiao, Y., Xiong, Y., Chen, Y., Cao, Y., Wang, Y., Chen, Y., Zhang, Y., Lin, Y., Xiao, Y., Gu, Y., Chen, Y., Huang, Y., Luo, Y., You, Y., Chen, Y., Xiang, Y., Liu, Y., Zhou, Y., Zhou, Y., Guo, Y., Huang, Y., Bai, Y., Z., Z. Y., Ni, Z., Wang, Z., Zhao, Z., Ren, Z., Zhao, Z., Sha, Z., Wang, Z., Zhang, Z., Du, Z., Fu, Z., Xu, Z., Xie, Z., Liu, Z., Zhang, Z., Dong, Z., Hao, Z., Wang, Z., Gou, Z., Ma, Z., Li, Z., Shao, Z., Huang, Z., Li, Z., Lu, Z., Huang, Z., Chen, Z., Chen, Z., Pan, Z., Wu, Z., Ren, Z., He, Z., Li, Z., Zhang, Z., Xu, Z., Wang, Z., Gu, Z., Zhu, Z., Zhang, Z., Li, Z., Hou, Z., Lyu, Z., Wang, Z., Xie, Z., Zhang, Z., Gao, Z., Pan, Z., Li, Z., Yao, Z., Chen, Z., Wu, Z., Ling, C., Hou, C., Chen, C., Li, D., Qi, D., Ji, D., Wei, F., Xia, F., Xie, F., Tan, F., Guo, H., Zhai, H., Zhou, H., Tan, H., Li, H., Luo, J., Song, J., Cai, J., Liang, J., Zhou, J., Gao, J., Shao, J., Chen, J., Yang, J., Chen, J., Zhang, J., Zhou, J., Wang, J., Liu, J., Sun, J., Ling, J., Zheng, J., Yang, K., Xu, K., Su, L., Xia, L., Ding, L., Zhuo, L., Ma, L., Zhu, L., Cai, L., Yao, L., Zhang, M. K., Li, M., Lin, M., Wang, M., Zhang, M., Li, M., Wang, M., Yin, M., Han, M., Cao, N., Wang, N., Ma, N., Wang, P., Lin, P., Sun, P., Zhang, P., Ying, Q., Xiang, Q., Wang, Q., Mao, Q., Jiang, Q., Jin, R., Chen, R., Tao, S., Sun, S., Wu, S., Zou, S., Lei, S., Zhang, T., Sun, T., Yin, T., Xiao, W. L., An, W., Li, W., Wang, W., Lin, W., Hou, W., Lin, X., Meng, X., Huang, X., Peng, X., Li, X., Zhang, X., Sun, X., Wang, X., Ye, X., Zhang, X., Zhang, X., Cao, X., Chen, X., Zhou, Y., Xu, Y., Xia, Y., Xu, Y., Shao, Y., Zhang, Y., Ma, Y., Tang, Y., Lou, Y., Chen, Y., Piao, Y., Chen, Y., Xiong, Y., Xuan, Y., Yang, Y., Xu, Y., Zha, Y., Ma, Y., Lin, Y., Yan, Y., Xie, Y., Sheng, Y., Zhu, Y., Zhang, Z., Ju, Z., Lin, Z., Gao, Z., Sun, Z., Yan, Z., Wu, Z., Wang, Z., Qu, Z., Yan, Z., and Wan, Z. DeepSeek-V4.1-Flash: Pushing the limits of KV cache compression, 2026. https://arxiv.org/abs/2609.19969.
  15. [16]Exgentic team. Exgentic agent LLM traces v2: Chat-only execution traces for agent benchmarks, 2026. https://huggingface.co/datasets/Exgentic/agent-llm-traces-v2.
  16. [17]Glazer, E., Erdil, E., Besiroglu, T., Chicharro, D., Chen, E., Gunning, A., Olsson, C. F., Denain, J.-S., Ho, A., de Oliveira Santos, E., Järviniemi, O., Barnett, M., Sandler, R., Vrzala, M., Sevilla, J., Ren, Q., Pratt, E., Levine, L., Barkley, G., Stewart, N., Grechuk, B., Grechuk, T., Enugandlа, S. V., and Wildon, M. Frontiеrmath: A benchmark for evaluating advanced mathematical reasoning in ai, 2025. https://arxiv.org/abs/2411.04872.
  17. [18]GLM-5-Team, Zeng, A., Lv, X., Hou, Z., Du, Z., Zheng, Q., Chen, B., Yin, D., Ge, C., Huang, C., Xie, C., Zhu, C., Yin, C., Wang, C., Pan, G., Zeng, H., Zhang, H., Wang, H., Chen, H., Zhang, J., Jiao, J., Guo, J., Wang, J., Du, J., Wu, J., Wang, K., Li, L., Fan, L., Zhong, L., Liu, M., Zhao, M., Du, P., Dong, Q., Lu, R., Shuang-Li, Cao, S., Liu, S., Jiang, T., Chen, X., Zhang, X., Huang, X., Dong, X., Xu, Y., Wei, Y., An, Y., Niu, Y., Zhu, Y., Wen, Y., Cen, Y., Bai, Y., Qiao, Z., Wang, Z., Wang, Z., Zhu, Z., Liu, Z., Li, Z., Wang, B., Wen, B., Huang, C., Cai, C., Yu, C., Li, C., Hu, C., Zhang, C., Zhang, D., Lin, D., Yang, D., Wang, D., Ai, D., Zhu, E., Yi, F., Chen, F., Wen, G., Sun, H., Zhao, H., Hu, H., Zhang, H., Liu, H., Zhang, H., Peng, H., Tai, H., Zhang, H., Liu, H., Wang, H., Yan, H., Ge, H., Liu, H., Chu, H., Zhao, J., Wang, J., Zhao, J., Ren, J., Wang, J., Zhang, J., Gui, J., Zhao, J., Li, J., An, J., Li, J., Yuan, J., Du, J., Liu, J., Zhi, J., Duan, J., Zhou, K., Wei, K., Wang, K., Luo, K., Zhang, L., Sha, L., Xu, L., Wu, L., Ding, L., Chen, L., Li, M., Lin, N., Ta, P., Zou, Q., Song, R., Yang, R., Tu, S., Yang, S., Wu, S., Zhang, S., Li, S., Li, S., Fan, S., Qin, W., Tian, W., Zhang, W., Yu, W., Liang, W., Kuang, X., Cheng, X., Li, X., Yan, X., Hu, X., Ling, X., Fan, X., Xia, X., Zhang, X., Zhang, X., Pan, X., Zou, X., Zhang, X., Liu, Y., Wu, Y., Li, Y., Wang, Y., Zhu, Y., Tan, Y., Zhou, Y., Pan, Y., Zhang, Y., Su, Y., Geng, Y., Yan, Y., Tan, Y., Bi, Y., Shen, Y., Yang, Y., Li, Y., Liu, Y., Wang, Y., Li, Y., Wu, Y., Zhang, Y., Duan, Y., Zhang, Y., Liu, Z., Jiang, Z., Yan, Z., Zhang, Z., Wei, Z., Chen, Z., Feng, Z., Yao, Z., Chai, Z., Wang, Z., Zhang, Z., Xu, B., Huang, M., Wang, H., Li, J., Dong, Y., and Tang, J. GLM-5: from vibe coding to agentic engineering, 2026. https://arxiv.org/abs/2602.15763.
  18. [19]Goldman Sachs Research. AI agents forecast to boost tech cash flow as usage soars, May 2026. https://www.goldmansachs.com/insights/articles/ai-agents-forecast-to-boost-tech-cash-flow-as-usage-soars.
  19. [20]Huang, W., Lee, C., Tng, L., and Ge, S. DeepSWE: Measuring frontier coding agents on original, Long-Horizon engineering tasks. arXiv preprint arXiv:2607.07946, 2026. https://arxiv.org/abs/2607.07946.
  20. [21]Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles (SOSP), pp. 611–626, 2023. doi: 10.1145/3600006.3613165. https://arxiv.org/abs/2309.06180.
  21. [24]Lohmeyer, M. and Horgan, S. Announcing llm-d 1.0: Open distributed inference for agents on Google Cloud TPUs and GPUs, October 2026. https://cloud.google.com/blog/topics/ai-infrastructure/llm-d-1-0-an-open-source-stack-for-distributed-inference. Google Cloud blog.
  22. [25]Maguire, J. Open-weight AI models expected to capture 41% of enterprise token usage. Techstrong.ai, September 2026. https://techstrong.ai/articles/open-weight-ai-models-expected-to-capture-41-of-enterprise-token-usage/. Reporting an Enterprise Technology Research (ETR) survey of 200 enterprise technology leaders, September 2026.
  23. [26]Merrill, M. A., Shaw, A. G., Carlini, N., Li, B., Raj, H., Bercovich, I., Shi, L., Shin, J. Y., Walshe, T., Buchanan, E. K., Shen, J., Ye, G., Lin, H., Poulos, J., Wang, M., Nezhurina, M., Jitsev, J., Lu, D., Mastromichalakis, O. M., Xu, Z., Chen, Z., Liu, Y., Zhang, R., Chen, L. L., Kashyap, A., Uslu, J.-L., Li, J., Wu, J., Yan, M., Bian, S., Sharma, V., Sun, K., Dillmann, S., Anand, A., Lanpouthakaun, A., Koopah, B., Hu, C., Guha, E., Dreiman, G. H. S., Zhu, J., Krauth, K., Zhong, L., Muenninghoff, N., Amanfu, R., Tan, S., Pimpalgaonkar, S., Aggarwal, T., Lin, X., Lan, X., Zhao, X., Liang, Y., Wang, Y., Wang, Z., Zhou, C., Heineman, D., Liu, H., Trivedi, H., Yang, J., Lin, J., Shetty, M., Yang, M., Omi, N., Raoof, N., Li, S., Zhuo, T. Y., Lin, W., Dai, Y., Wang, Y., Chai, W., Zhou, S., Wahdany, D., She, Z., Hu, J., Dong, Z., Zhu, Y., Cui, S., Saiyed, A., Kolbeinsson, A., Hu, J., Rytting, C. M., Marten, R., Wang, Y., Dimakis, A., Konwinski, A., and Schmidt, L. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces, 2026. https://arxiv.org/abs/2601.11868.
  24. [27]MiniMax. MiniMax-Provider-Verifier, 2025. https://github.com/MiniMax-AI/MiniMax-Provider-Verifier. GitHub repository.
  25. [28]Moonshot AI. Kimi Vendor Verifier, 2026. https://github.com/MoonshotAI/Kimi-vendor-Verifier. GitHub repository.
  26. [30]NVIDIA. Nemotron 3 Ultra recipe: Moontrace agentic and chat replay traces, 2026. https://github.com/ai-dynamo/dynamo/tree/main/recipes/nemotron-3-ultra/perf/traces. NVIDIA Dynamo; trace description at https://docs.nvidia.com/dynamo/dev/recipes/nemotron-3-ultra.
  27. [31]OpenTelemetry Authors. OpenTelemetry Specification, 2026. https://opentelemetry.io/docs/specs/otel/. Accessed: 2026-10-21.
  28. [32]Pan, Z., Wang, M., Wu, C., Hu, Z., Qiang, X., Yu, Z., and Ding, Y. AgentReplay: Token-wise trace replay is essential for fair serving system performance benchmarking, 2026. https://arxiv.org/abs/2609.32283.
  29. [33]Panavas, L., Minus, S., Monton, B., Ray, D., Garre, S., Mehta, S., and Chen, E. Handbook: A benchmark for long-context agentic instruction following, 2026. https://arxiv.org/abs/2607.25398.
  30. [34]Patil, S. G., Mao, H., Yan, F., Ji, C. C.-J., Suresh, V., Stoica, I., and Gonzalez, J. E. The berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models. In Proceedings of the 42nd International Conference on Machine Learning, 2025.
  31. [35]Qin, R., Li, Z., He, W., Zhang, M., Wu, Y., Zheng, W., and Xu, X. Mooncake: A KVCache-centric disaggregated architecture for LLM serving, 2024. https://arxiv.org/abs/2407.00079.
  32. [36]Qin, R., Li, Z., He, W., Cui, J., Ren, F., Zhang, M., Wu, Y., Zheng, W., and Xu, X. Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25), pp. 155–170, Santa Clara, CA, February 2025. USENIX Association. ISBN 978-1-939133-45-8. https://www.usenix.org/conference/fast25/presentation/qin.
  33. [37]SemiAnalysis. SemiAnalysis Claude Code WEKA traces with subagents (cc-traces-weka-with-subagents-060826), 2026a. https://huggingface.co/datasets/semianalysisai/cc-traces-weka-with-subagents-060826. Hugging Face dataset.
  34. [38]SemiAnalysis. SemiAnalysis Claude Code WEKA traces (cc-traces-weka-062126). Hugging Face dataset, 2026b. https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126.
  35. [39]Shepard, D. and Salimans, R. AutomationBench. arXiv preprint arXiv:2604.18934, 2026. https://arxiv.org/abs/2604.18934.
  36. [40]Shi, L., Lin, H., Zhu, Z., Zhou, X., Li, X., Lin, X., Deng, Y., Xu, H., Li, Y., Li, S., Chen, Z., Xing, H., Raj, H., Chen, B., Shi, Q., Dillmann, S., Gao, Y., Khanna, P., Lu, R., Zhou, C. B., Yang, M., Zhang, R., Chai, S., Chang, J., Chen, Y., Chen, X., Dai, Y., Yang, W., Liu, H., Liu, M., Wang, Z., Assadi, A. E., Stroebel, B., Buchanan, E. K., Meng, H., He, J., Yu, L., Shayanfar, R., Lee, Y., Dong, Z., Hart, A. G., Wei, A., Kashyap, A., Khatua, A., Zheng, A. J., Ma, C., Heineman, D., Chen, D., Trinh, H.-A., Fang, H., Zhang, H., Shen, H., Sugiura, I., Sun, J., Gao, J., Lin, J., Li, J., Yang, K., Hsiung, L., Wang, M., Tang, M., Omi, N., Raoof, N., Edwards, N., Guo, O., Mastromichalakis, O. M., Ji, P., Hejman, P., Qi, Q., Lin, Q., Zhuang, R., Yang, R., Zheng, R., Marten, R., Fazliani, S., Hou, S., Jiang, S., Li, S., Yuan, B., Glass, M., Bian, S., Zhuo, T. Y., Wu, T., Tang, T., Zhao, W., Xuan, W., Liang, W., Liu, X., Lan, X., Zhang, X., Zhao, X., Tang, Y., Jiang, Y., Li, Y., Guan, Y., Li, Y., Liu, Y., Tang, Y., Mao, Y., Zhao, Y., Wang, Y., Tang, Y., Tang, Z., Li, Z., Wang, Z., She, Z., Liu, K., Chaabane, I., Tang, Y., Li, X., GNVV S. S. S. N., Zheng, X., Konwinski, A., Li, B., Chen, L. L., Dimakis, A., Carlini, N., Vosoughi, S., Koyejo, S., He, D., Guha, E., Feuer, B., Merrill, M., Schmidt, L., and Shaw, A. Harbor adapters and harbor-index: Infrastructure and a curated meta-dataset for large-scale agentic evaluation, 2026. https://arxiv.org/abs/2609.04298.
  37. [41]Smith, S., Wanner, S., Butler, W., and Diasio, D. Unlocking agentic value: A new investment discipline for the agentic era, June 2026. https://www.ey.com/en_us/insights/ai/agentic-ai-token-costs. EY, Total Cost of Agents series.
  38. [42]Team, K., Bai, T., Bai, Y., Bao, Y., C., M., Cai, J., Cai, X., Cao, P., Cao, Y., Chai, Z., Charles, Y., Che, H. S., Chen, G., Chen, G., Chen, G., Chen, H., Chen, J., Chen, J., Chen, J., Chen, K., Chen, P., Chen, R., Chen, W., Chen, X., Chen, Y., Chen, Y., Chen, Y., Chen, Y., Chen, D., Cheng, Y., Cui, J., Cui, J., Dai, A., Deng, J., Ding, H., Ding, R., Ding, S., Dong, M., Dong, M., Dong, Y., Dong, Y., Du, A., Du, C., Du, D., Du, J., Du, Y., Fan, Y., Feng, J., Feng, Q., Feng, Y., Fu, K., Fu, Q., Gao, F., Gao, H., Gao, J., Gao, T., Gao, W., Geng, S., Gong, J., Gong, L., Gong, S., Gong, X., Gu, Q., Gu, Y., Guan, S., Guo, H., Guo, S., Guo, X., Guo, Z., Hao, B., Hao, W., Hao, X., He, D., He, H., He, L., He, Q., He, W., He, X., He, X., He, Y., He, Y., Hong, C., Hong, T., Hu, H., Hu, J., Hu, R., Hu, W., Hu, Y., Hu, Z., Hua, L., Huang, J., Huang, K., Huang, R., Huang, S., Huang, W., Huang, Y., Huang, Z., Huang, Z., Hui, Y., Jia, C., Jiang, Y., Jiang, Z., Jiang, Z., Jin, W., Jin, X., Jing, Y., Kong, H., Lai, G., Li, L., Li, C., Li, C., Li, C., Li, F., Li, G., Li, H., Li, J., Li, J., Li, L., Li, L., Li, L., Li, W., Li, W., Li, X., Li, Y., Li, Y., Li, Y., Li, Y., Li, Y., Li, Z., Li, Z., Li, Z., Li, Z., Li, Z., Lin, J., Lin, X., Lin, Y., Lin, Z., Lin, Z., Liu, B., Liu, B., Liu, C., Liu, L., Liu, S., Liu, S., Liu, S., Liu, T., Liu, W., Liu, Y., Liu, Y., Liu, Y. Liu, Y., Liu, Z., Liu, Z., Liu, E., Liu, H., Lu, L., Lu, T., Lu, Z., Luo, A., Luo, G., Luo, J., Luo, Y., Lyu, B., Lyu, W., Mao, S., Mei, Y., Men, X., Ni, M., Niu, Y., Pan, S., Peng, S., Qi, Z., Qin, R., Qin, Z., Qin, Z., Qiu, H., Qiu, J., Qiu, J., Qiu, B., Qu, Y., Shang, Z., Shao, Y., Shen, H., Shi, J., Shi, J., Shi, L., Shi, S., Siu, W., Song, P., Song, X., Su, J., Su, Y., Su, Z., Sui, L., Sun, J., Sun, J., Sun, S., Sun, S., Sun, T., Sun, Y., Tai, Y., Tang, C., Tang, H., Tang, S., Tang, Z., Tian, C., Tian, R., Tian, Y., Tu, W., Wang, C., Wang, C., Wang, C., Wang, D., Wang, F., Wang, H., Wang, H., Wang, H., Wang, H., Wang, H., Wang, H., Wang, J., Wang, J., Wang, J., Wang, J., Wang, J., Wang, L., Wang, S., Wang, S., Wang, S., Wang, S., Wang, S., Wang, T., Wang, W., Wang, X., Wang, X., Wang, X., Wang, X., Wang, X., Wang, Y., Wang, Y., Wang, Y., Wang, Y., Wang, Y., Wang, Y., Wang, Z., Wang, Z., Wang, Z., Wang, Z., Wang, Z., Wang, Z., Wang, Z., Wei, C., Wei, M., Wei, S., Wen, Z., Wu, F., Wu, H., Wu, R., Wu, W., Wu, X., Wu, Y., Wu, Y., Wu, Y., Wu, Z., Xian, X., Xiang, C., Xiang, Y., Xiao, B., Xiao, C., Xiao, X., Xie, J., Xie, X., Xie, Y., Xie, Z., Xing, B., Xiong, Y., Xu, B., Xu, B., Xu, J., Xu, J., Xu, J., Xu, L. H., Xu, Q., Xu, S., Xu, S., Xu, T., Xu, T., Xu, W., Xu, X., Xu, Y., Xu, Y., Xu, Y., Xu, Z., Xue, H., Yan, J., Yan, Y., Yang, F., Yang, G., Yang, H., Yang, J., Yang, R., Yang, W., Yang, X., Yang, X., Yang, Y., Yang, Y., Yang, Y., Yang, Y., Yang, Z., Yang, Z., Yang, Z., Yang, Z., Yang, Z., Yang, Z., Yang, Z., Yao, H., Ye, D., Ye, H., Ye, W., Ye, Z., Yin, B., Yin, H., Yin, X., Yu, C., Yu, H., Yu, L., Yu, S., Yu, S., Yu, T., Yuan, E., Yuan, M., Yue, T., Yue, W., Yue, Y., Zha, D., Zhan, H., Zhang, B. H., Zhang, D., Zhang, F., Zhang, H., Zhang, H., Zhang, H., Zhang, J., Zhang, J., Zhang, J., Zhang, K., Zhang, M., Zhang, P., Zhang, Q., Zhang, R., Zhang, R., Zhang, S., Zhang, S., Zhang, X., Zhang, X., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Z., Zhang, Z., Zhao, B., Zhao, C., Zhao, F., Zhao, J., Zhao, J., Zhao, S., Zhao, W., Zhao, X., Zhao, X., Zhao, Y., Zhao, Z., Zheng, H., Zheng, H., Zheng, R., Zheng, S., Zheng, T., Zhong, H., Zhong, L., Zhou, M., Zhou, Q., Zhou, R., Zhou, R., Zhou, X., Zhou, Y., Zhou, Z., Zhu, J., Zhu, L., Zhu, X., Zhu, Y., Zhu, Y., Zhu, Z., Zhuang, C., Zhuang, W., and Zu, X. Kimi k3: Open frontier intelligence, 2026. https://arxiv.org/abs/2607.24653.
  39. [43]Tokenomics Foundation. State of tokenomics: September 2026, September 2026. https://www.tokenomics.com/state-of-tokenomics/. Industry survey of 472 organizations; Release Candidate 1.0.
  40. [44]UK AI Security Institute. Inspect AI: Framework for large language model evaluations, 2024. https://github.com/UKGovernmentBEIS/inspect_ai.DOI
  41. [45]Wang, M., Yue, Y., Li, S., Zhou, Y. E., Wang, C., and Huang, J. Benchmarking LLM serving systems for agentic AI workloads with XPerf, 2026a. https://arxiv.org/abs/2608.20370.
  42. [46]Wang, Z., Shi, T., He, J., Cai, M., Zhang, J., and Song, D. Cybergym: Evaluating AI agents’ real-world cybersecurity capabilities at scale. In The Fourteenth International Conference on Learning Representations, 2026b. https://openreview.net/forum?id=2YvbLQEdYt.arxiv.org/abs/2506.02548
  43. [47]Yang, J., Jimenez, C. E., Zhang, A. L., Lieret, K., Yang, J., Wu, X., Press, O., Muennighoff, N., Synnaeve, G., Narasimhan, K. R., Yang, D., Wang, S., and Press, O. SWE-bench Multimodal: Do AI systems generalize to visual software domains? In The Thirteenth International Conference on Learning Representations, 2025a. https://openreview.net/forum?id=riTiq3i21b.arxiv.org/abs/2410.03859
  44. [48]Yang, J., Lieret, K., Jimenez, C. E., Wettig, A., Khandpur, K., Zhang, Y., Hui, B., Press, O., Schmidt, L., and Yang, D. SWE-smith: Scaling data for software engineering agents. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2025b. https://arxiv.org/abs/2504.21798.
  45. [49]Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. SGLang: Efficient execution of structured language model programs, 2024. https://arxiv.org/abs/2312.07104.
  46. [50]Zhu, K., Jacob, M., Ma, C., Pan, Y., Wang, S., Krishnamurthy, A., and Kasiccki, B. TraceLab: Characterizing coding agent workloads for LLM serving, 2026. https://arxiv.org/abs/2606.30560.

Paper details

vlnc:W_2ktnqz24
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude Code, Codex

View statement
Looking for related papers…
Contents