betaFeedback

WorksComputer Science

ServerBench: How can your coding agents design specialized LLM serving systems?

Abstract

Large language model (LLM) coding agents develop software through experimentation, offering an opportunity to design the serving systems supporting their own operation. Effective serving requires coordinating implementation and operating decisions under session dependencies, latency constraints, and correctness requirements. We introduce ServerBench, a benchmark for agentic design of complete, specialized LLM serving systems. Agents experiment against a common objective, dollars per dollar: latency-qualified token value per dollar of GPU cost. Stateful replay preserves session dependencies, and independent verification checks the submitted system. Across 11 coding-agent models and three deployments, Claude Sonnet 5.5 achieves 2.97 dollars per dollar on Gemma E4B, 4.87×4.87\times the vLLM reference; 22 of 33 runs yield verified systems, and 13 exceed the strongest baseline. Our analysis identifies three findings: high-value designs coordinate capacity improvements with load selection; agents interpret performance feedback through workload-relevant experiments; and reliable delivery requires verification of the execution paths introduced by optimization. Overall, ServerBench supports systematic study of how agents translate service objectives and feedback into effective, verified serving systems for given task specifications.

Code github.com/Infini-AI-Lab/Server-Bench

Blog infini-ai-lab.github.io/Server-Bench

Introduction

Large language model (LLM) coding agents build on code-generation capabilities ([36, 38]), using tools and execution feedback to undertake software development and professional workflows ([31]; [28, 6, 26]. These workflows depend on LLM serving systems, which run models and handle requests [13, 35, 39]. This creates an opportunity for coding agents to design these systems. Service providers need correct model execution and timely responses. We therefore ask: how can coding agents design LLM serving systems while satisfying given deployment and service constraints?

Despite rapid progress in agent-driven systems optimization, existing benchmarks evaluate individual kernels, code patches, or servers under preset test conditions [19, 15, 27, 32, 30], leaving three gaps in evaluating how agents design complete serving systems:

  • Oversimplified workloads. Prior benchmarks issue independent requests or fixed prefill, decoding, and throughput scenarios [32, 30]. With uniform request lengths and fixed batches, a static batch serves nearly as well as a production engine, so techniques that real deployments depend on, such as continuous batching and paged attention [33, 13], bring little benefit. Real applications such as coding agents instead mix short and long requests, extend a shared context over many turns, and fan out into parallel subagents, where prefix reuse, memory management, and prefill–decode interference shape performance [35, 25, 1, 37].

  • Speedup-only objectives. Existing benchmarks score speedup or throughput relative to a baseline [19, 32, 30]. Such objectives ignore latency requirements and GPU cost and fix the offered load, whereas a service provider must decide how much workload to serve while keeping time to first token (TTFT) and time per output token (TPOT) within targets.

  • Correctness-only validation. Validation typically checks outputs against a reference implementation or test suite [19, 27, 30]. A serving system must additionally meet tail-latency targets on the deployed workload and execute that workload in full, with complete prompts, prescribed output lengths, and consistent streaming. Optimizations such as speculative decoding, state reuse, and custom scheduling introduce execution paths that short or isolated tests rarely exercise.

We introduce ServerBench, a benchmark in which coding agents build a complete LLM serving system for a given model, GPU allocation, and workload, and decide how much workload it serves (Figure 2). Starting from a minimal Transformers server, agents plan experiments and revise their systems within a fixed development budget. ServerBench closes the three gaps with three designs. 1. Workload: real coding-agent trajectories. ServerBench replays recorded coding-agent sessions [24] with their request lengths, multi-turn structure, shared prefixes, parallel subagent branches, and inter-turn delays. Requests wait on the responses they depend on, so serving latency governs how much work completes within the measurement window (§ 3.2). 2. Objective: dollars per dollar. We price the input and output tokens of each completed request at fixed API rates, count only requests that meet both TTFT and TPOT targets, and divide by the GPU cost of the measurement window (§ 3.1). For example, USD 3.65 of qualified token value over USD 0.83 of GPU cost scores 4.40. The objective rewards capacity improvements together with a matching load choice: serving more workload raises gross value only while requests still meet their latency targets. 3. Validation: beyond correctness. Latency targets are the p90 TTFT and TPOT of an SGLang reference with its model-specific configuration, frozen before development (§ 3.1). The submitted system is rebuilt in a fresh deployment and replayed on held-out seeds. Independent checks verify complete prompts, output lengths, streaming consistency, generation fidelity to the checkpoint, and held-out answer accuracy; a post-run review inspects code and recorded actions for prohibited shortcuts (§ 3.4). Only verified systems receive a score.

Figure 2. Agents design the server and choose how many sessions it serves at once for a given model, GPU allocation, and workload. Replay preserves shared context and dependencies between requests; final checks assess model correctness and complete workload execution.

We evaluate eleven coding-agent models across three model–hardware deployments. Claude Sonnet 5.5 achieves 2.97 dollars per dollar on Gemma E4B, 4.87× the vLLM serving baseline (§ 4.2). Trajectory analysis identifies three findings. High-value designs coordinate capacity and concurrency. Sonnet 5.5 improves its server before raising concurrency, then limits load when additional sessions worsen TPOT (§ 4.3). Interpreting feedback in workload context guides optimization. A random-text probe leads one agent to reject an optimization; workload replies reveal its benefit and lead another to adopt it (§ 4.4). Verification must match the deployed workload. Agents’ short or isolated checks can miss long-context stalls and cross-request crashes; known faults can persist or recur after edits in submitted systems (§ 4.5).

Our contributions are:

  • ServerBench, a benchmark for designing complete LLM serving systems to maximize service value per GPU dollar (§ 3.1). Agents design the server and set the number of concurrent application sessions.

  • An evaluation protocol that replays coding sessions with shared context and dependent requests, supports experiments designed by agents, and independently verifies the final system (Sections 3.2 to 3.4).

  • A study of eleven coding-agent models across three deployments, using controlled comparisons and trajectory analysis to explain successes and failures in system design, experimental judgment, and verification (Sections 4.2 to 4.6).

Related Work

Inference System Optimization

Orca and vLLM improve request scheduling and memory management ([33, 13]). SGLang and Preble exploit prefix reuse ([35, 25]), while Sarathi-Serve and DistServe mitigate prefill–decode interference through chunked prefill and prefill–decode disaggregation ([1, 37]). AlphaEvolve and AI-Driven Research for Systems (ADRS) use evaluator feedback to generate and refine algorithms and system code ([16, 8]). Vortex enables agents to generate and refine sparse-attention algorithms in a programmable serving system ([7]). ServerBench studies whether agents integrate such optimizations into systems under fixed deployment conditions.

Inference-Performance Benchmarks

MLPerf Inference standardizes workloads, quality targets, and deployment scenarios ([23]). Agent-oriented benchmarks assess correctness and performance: KernelBench tests generated GPU kernels against PyTorch references for correctness and speedup ([19]), ISO-Bench compares vLLM and SGLang optimization patches with expert implementations ([15]), and Φ-Bench spans kernel completion through end-to-end infrastructure optimization ([10]). SWE-Serve tests production features within SGLang ([27]), while InferenceBench evaluates complete servers on prefill, decoding, concurrent throughput, and a balanced scenario using baseline-relative speedup ([32]). ServerBench extends this line of evaluation to response-dependent, stateful sessions and a latency-qualified revenue-to-cost objective, linking implementation and operating choices to the service delivered by complete systems.

ServerBench

ServerBench evaluates agentic design through a service objective, stateful workload replay, iterative experimentation, and final verification. Task configurations appear in Appendices § 6 and § 9.

Task and Service Objective

Each task fixes a model checkpoint, GPU allocation, coding-session workload, and service requirements. Starting from a minimal Transformers server S0S_0, an agent has a time budget BB to develop a serving system. Its submission comprises an implementation SBS_B and session concurrency cBc_B. Agents may redesign execution, batching, memory management, scheduling, and state reuse [33, 13, 35] under the prescribed checkpoint, precision, and inference semantics. Agents prioritize changes to meet the service objective.

Dollars per dollar. The objective, dollars per dollar, measures completed token value meeting both latency targets per dollar of GPU cost. For request ii, client-measured input and output counts xix_i, yiy_i give vi=(pinxi+poutyi)/106v_i = (p_{\mathrm{in}}x_i + p_{\mathrm{out}}y_i)/10^6, with prices in USD per million tokens. Let qi=1q_i = 1 when its finite TTFT and TPOT meet targets τfirst\tau_{\mathrm{first}}, τtoken\tau_{\mathrm{token}}, and zero otherwise. A verified system measured for WW seconds after warmup on dd GPUs, priced at rr dollars per GPU-hour, receives

D(SB,cB)=∑i∈Rqivid r (W/3600)(1)D(S_B, c_B) = \frac{\sum_{i\in\mathcal{R}} q_i v_i}{d\,r\,(W/3600)} \tag*{(1)}

where R\mathcal{R} contains completed requests. Define gross value G=∑i∈Rvi/[d r (W/3600)]G = \sum_{i\in\mathcal{R}} v_i/[d\,r\,(W/3600)] and the value-weighted qualified share f=D/Gf = D/G for G>0G > 0; thus D=GfD = Gf. Input value includes cached prefixes at full price once per logical request. For example, USD 3.65 of qualified value over USD 0.83 of GPU cost yields 4.40.

Latency calibration. Before development, we select a reference operating point using SGLang’s model-specific cookbook configuration [35]. At the reference operating point, repeated replays yield p90 TTFT aja_j and TPOT bjb_j for seed jj. We set τfirst=max⁡jaj\tau_{\mathrm{first}} = \max_j a_j and τtoken=max⁡jbj\tau_{\mathrm{token}} = \max_j b_j, then freeze these request-level targets before agent development. Taking the maxima provides headroom for observed run-to-run variation. Agents choose the concurrency used by their submitted server.

Stateful Workload and Closed-Loop Replay

Coding sessions contain dependent turns, shared context, and parallel subagent branches. Using the AgentX harness [24], we replay response-dependent turns and parallel branches using recorded inter-turn delays, with long global idle periods compressed. Offered concurrency bounds active top-level sessions, whose branches can issue additional requests. Response times thus affect session progress and the work completed within the measurement window.

Privacy-preserving reconstruction replaces prompt and tool contents while retaining request lengths, shared prefixes, and session topology. Tasks fix reconstructed prompts and output lengths. Shared context supports computation reuse [35, 25], while retained state and active requests compete for memory. Agents are responsible for translating these workload characteristics into optimizations while preserving model behavior and executing the workload in full.

Growing context enables reuse, while new sessions add prefill work that competes with decoding [1, 37]. Session progress thus combines prompt processing, generation, and retained state.

Agent Environment and Development Loop

Agents work in an Internet-disabled Harbor sandbox with container or Modal backends [14]. They receive a minimal server, read-only model weights, reference configurations, measurement tools, and the published performance procedure. Task instructions require model fidelity and complete workload execution and prohibit evaluation manipulation. Independent correctness-test details are withheld, and trusted evaluation components remain outside the editable serving implementation.

The tools report latency, throughput, and request value. Agents organize their own experiments, diagnose bottlenecks, and revise the implementation [31]. They budget time for replays, diagnostics, and development. Publishing fixes the implementation and concurrency separately from performance testing; when the allotted time for development expires, the last published version becomes the submission.

Agents choose experiment scale: diagnostics isolate suspected bottlenecks, while complete replays test interacting requests.

They decide which observations warrant further tests, which changes to retain, and when to publish. Versioned submissions connect these decisions to the implementation evaluated after development.

System Verification

After development, ServerBench rebuilds the submission in a fresh deployment and replays sessions sampled with seeds disjoint from development. Independent tests and post-run review check the resulting system (Figure 3).

Figure 3. Stateful replay connects session progress to the agent’s design experiments (a). Final evaluation verifies the published system and measures request value against calibrated latency targets (b).

Workload and output verification. Replay checks cover complete prompts, output lengths, streaming consistency, and session progress. Timed replay disables EOS stopping to generate the prescribed output length; separate EOS probes exercise natural stopping. Reference-scored continuations check generation against the checkpoint. Held-out questions compare answer correctness with a plain implementation of the same model, using the task’s accuracy tolerance to screen for substantial regressions and shortcuts that bypass model computation. Fresh deployments also test behavior beyond development state.

Post-run review. A review agent and human inspection examine submitted code and recorded actions for checkpoint fidelity, inference semantics, and workload execution. Review can identify prohibited shortcuts missed by behavioral tests and overturn automated failures when inspection establishes compliance with task requirements. These checks determine score eligibility (Equation (1)).

Evaluation

Experimental Setup

We evaluate 11 coding-agent models (Table 1): Claude ([3 c,d]), GPT ([18]), Kimi ([12]), GLM ([34]), DeepSeek ([9]), MiMo ([29]), and Qwen ([20]) models. All use maximum reasoning effort on Qwen3.5 9B (2×L40), Gemma-4-E4B-it (Gemma E4B; 1×L40), and Qwen3.8 27B INT4 (2×L40). In Harbor, GPT models use Codex ([18]), and other models use Claude Code ((Anthropic, 2026a)). Each agent–deployment pair allows one 8 hours development run and a 1,800-second final replay, with at most one rerun for a clearly anomalous failed outcome. We report each pair’s strongest verified final submission across the 33 Table 1 entries; development analysis focuses on 30 runs with server execution records.

SystemGemma E4B (1×L40)Qwen3.5 9B (2×L40)Qwen3.8 27B INT4 (2×L40)
Best baseline
vLLM $/$ = 0.61, cc = 8SGLang $/$ = 1.62, cc = 1vLLM $/$ = 1.47, cc = 1
Other baseline
SGLang $/$ = 0.16, cc = 2vLLM $/$ = 1.49, cc = 1SGLang $/$ = 1.42, cc = 4
$/$ @ ccTTFTTPOT$/$ @ ccTTFTTPOT$/$ @ ccTTFTTPOT
Claude Opus 5.52.47 @403.01118.594.40 @162.2528.162.64 @811.4446.40
Claude Sonnet 5.52.97 @322.7583.054.34 @2012.3426.762.01 @1246.5747.35
GPT 6 Astra1.75 @324.7794.193.18 @1215.6221.941.48 @412.9642.83
GPT 6.1 Sol1.59 @244.4372.452.84 @42.7123.670.98 @475.6643.31
Claude Opus 51.15 @243.98109.121.16 @15.5128.090.62 @633.5562.97
GPT 6 Sol0.55 @53.63110.521.11 @113.1920.450.26 @2136.4849.53
Kimi K30.52 @68.6755.320.93 @14.9933.12Not passed
Qwen3.8-Max-09020.87 @142.6159.16Not passedNot passed
MiMo V2.6 Pro0.22 @1043.63108.96Not passedNot passed
GLM 5.3Not passedNot passedNot passed
DeepSeek V4.1 FlashNot passedNot passedNot passed

Table 1. Serving performance across three deployments. Entries pair latency-qualified dollars per dollar with offered session concurrency cc; TTFT (s) and TPOT (ms) are p90 latencies. Shading marks the strongest baseline and largest agent value in each deployment.

Baselines are SGLang v0.5.19 and vLLM v0.28.0 ([35, 13]). SGLang uses each model’s cookbook configuration; reference results average replay seeds at fixed concurrency. Controlled comparisons replay frozen implementations and intermediate versions on common seeds.

End-to-End Results

Table 1 shows that agents can design verified serving systems with higher value than the strongest reference on each deployment. Claude Opus 5.5 leads Qwen3.5 9B and Qwen3.8 27B INT4 with 4.40 and 2.64 dollars per dollar, 2.71× the SGLang reference and 1.80× the vLLM reference, respectively. Claude Sonnet 5.5 achieves 2.97 on Gemma E4B, 4.87× the vLLM reference.

Performance remains uneven across agents. Across 33 runs, 22 yield verified systems and 13 exceed the strongest reference. Six models deliver verified systems on all three deployments, but only three exceed the strongest reference on all three. GPT 6.1 Sol achieves 2.84 on Qwen3.5 9B, while its Qwen3.8 27B INT4 system scores 0.98 against a reference of 1.47. Opus 5 passes verification but trails both Qwen references.

Figure 4 separates gross value from the SLO meet values. In the final replays on Qwen3.5 9B, Sonnet 5.5 produces more gross value than Opus 5.5 (6.42 versus 5.56), but a smaller share qualifies, yielding 4.34 and 4.40. Figure 1 contrasts GPT 6.1 Sol’s lower-cost gains with Claude’s higher serving value. We next analyze agents’ design, feedback, and verification decisions.

API cost and serving value relative to the strongest baseline for Qwen3.5 9B and Gemma E4B

Figure 1. API cost and serving value relative to the strongest baseline. GPT 6.1 Sol offers lower-cost gains; Claude reaches higher value. Costs use OpenRouter rates (Table 3).

AgentQwen 9BGemma E4B
Claude Opus 5.599.69104.25
Claude Sonnet 5.578.6030.72
GPT 6 Astra110.81119.79
GPT 6.1 Sol13.0113.67
Claude Opus 5108.90104.37
GPT 6 Sol50.8835.93
Kimi K3126.9828.71
Qwen3.8-Max-090246.1343.33
MiMo V2.6 Pro6.3210.19
GLM 5.316.8010.79
DeepSeek V4.1 Flash1.642.07

Table 3. Agent API cost estimates (USD) at OpenRouter rates for Figure 1.

Gross value versus value-weighted SLO meet rate for Qwen3.5 9B and Gemma E4B

Figure 4. Gross value GG versus value-weighted SLO meet rate ff. Large markers show final evaluations; open markers compare frozen systems at different loads. Arrows indicate increasing concurrency; dashed curves mark equal D=GfD = Gf.

High-Value Designs Coordinate Capacity and Concurrency

At fixed GPU cost, additional sessions can increase token value but raise TTFT and TPOT through queueing and contention. Agents must match load to serving capacity. Figure 5 relates recorded concurrency patterns to final economic outcomes; runs that raise and hold load more often exceed the strongest reference. Among runs reaching at least twice the strongest reference, 7 of 17 load decisions improve value on subsequent measurement, compared with 2 of 23 for verified runs below the reference score.

Figure 5. Recorded concurrency paths and final outcomes. Colors mark value relative to the strongest baseline; labels count runs. Dots below panels mark first load declarations.

Early end-to-end replay gives agents time to develop capacity while adjusting load. The former group reaches usable replay in a median of 0.90 hours, compared with 2.96 hours for the latter. The optimization inventory and controlled comparisons in Appendix § 10.§ 4 connect these operating decisions to changes in execution, state management, and scheduling. Higher-value systems tend to combine more serving optimizations, including speculative decoding, host-memory state storage, and latency-aware scheduling. Cross-turn reuse is common to all audited systems.

Controlled comparisons illustrate this distinction. At 16 sessions, Opus 5.5’s v1→v2 Qwen redesign raises value by 1.32–1.88× on matched seeds. For frozen Gemma systems, Sonnet

5.5 increases value from 1.30 at two sessions to 2.97 at 32. Sonnet 5.5 also holds four Gemma sessions while revising the engine, then increases load together with further implementation changes. The trajectory illustrates capacity development before increasing load. On Qwen, Opus 5 instead lowers concurrency from four to two to one as it refines the implementation. Kimi likewise reduces load to one after identifying long prefills that interfere with decoding. Load reduction relieves contention; capacity improvements support more simultaneous work.

Agents estimate the economic operating point in different ways. Opus 5.5 uses a workload simulator to guide Qwen load tests, then retains 16 sessions after a 20-session replay yields more gross value but only 64% qualifying value. Sonnet 5.5 on Gemma compares settings on a fixed seed: 36 sessions produce more gross value but incur more TPOT misses, and it submits 32. These decisions match offered load to serving capacity within the latency targets.

Using Feedback to Guide Optimization

Agents use performance measurements and diagnostic tests to retain, revise, or abandon changes. Useful feedback requires testing a change under relevant workload conditions. Costly replays and seed variation make diagnostic tests and workload analysis essential within the development budget.

Figure 6 connects retained design decisions to the tests supporting them. Among runs reaching at least twice the strongest reference, 49% of retained decisions follow improvements on workload-relevant tests, versus 25% in failed runs; decisions without such tests account for 13% and 40%, respectively. In the other verified groups, 39% remain unconfirmed. These outcomes highlight testing and follow-up when deciding which changes to retain.

Retained changes, showing validation of retained design changes and retained changes in representative runs

Figure 6. Retained changes more often show workload-relevant gains in the ≥ 2× group (a); representative trajectories show their follow-up (b). Groups use final value relative to the strongest baseline.

On Gemma, experimental judgment changes whether agents pursue prompt-lookup speculation. Opus 5 accepts a null result from a synthetic random-text probe and omits the optimization. Opus 5.5 investigates its initial null result using 60 replies from replay, repairs its simulator, and measures 2.09–2.34 accepted tokens per step before adopting speculation. At a fixed assumed draft cost, a corrected, length-matched probe selects a smaller draft from synthetic text than from workload replies (Appendix § 11). To test the effect of this workload feedback on the agent’s decision, we resume Opus 5 from the same checkpoint for two hours per arm, with or without feedback from its own replay replies. Only the feedback arm re-adopts speculation; its server delivers 1.13–1.31× the qualified value on two fresh common seeds. Workload feedback improves the agent’s design decision and its submitted server’s value.

Favorable feedback can also lead agents to retain a configuration that performs poorly under other workload conditions. In one Opus 5.5 run on Qwen3 8B 27B INT4, the agent raises concurrency from 10 to 16 and revises scheduling, observing a score increase from 2.20 to 2.99. It recognizes overload and acknowledges that changing seeds complicate comparison, but judges the additional value worth the latency losses. Near the deadline, a scheduling repair remains in warmup, so it returns to the configuration supported by one completed 900-second replay. In the final replay, the same implementation’s qualified share falls from 76.0% to 16.8% and its score to 0.48.

Workload coverage can be checked cheaply. Our CPU scan of the workload generator over 128 additional seeds takes only 7.71 seconds: the development draw lies near the fourth percentile of initial stream availability, while the final draw is near the median. The agent had inspected corpus statistics, yet used an unusually light initial branching condition to validate its final configuration (Appendix § 10.§ 5).

Null and favorable results both require checking what the test exercises. Inspecting test inputs and profiling workload variation helps agents select representative and demanding conditions for costly replay, supporting decisions to retain, revise, or abandon a design.

Verification Must Match the Deployed Workload

Verification checks the request boundaries and state transitions affected by changes to admission, scheduling, and state reuse, including after subsequent edits. Figure 7 shows sparse coverage at submission time: only 5 of the 33 entries have recorded checks of natural stopping on the submitted system under load or long context.

Workload-relevant checks and first related symptom in not-passed runs

Figure 7. Workload-relevant checks on submitted implementations (a). Related symptoms precede submission in 8 of 11 not-passed runs (b); markers distinguish responses and recorded submission times.

Admission policies require checking which requests can make progress. On Qwen3.5 9B, Qwen3.8-Max-0902 tightens admission to protect TPOT, then checks a short prompt and a replay completing 11 requests. Its submitted policy leaves a range of permitted prompt lengths unable to enter execution even after other requests finish. Final evaluation stalls on a 315K-token input in this range. The successful short tests leave the new policy’s prompt-length boundaries unchecked.

State reuse requires checking behavior across requests. On Qwen3.8 27B INT4, Kimi’s replay exposes a prefix-cache exception alongside a numerical error. The agent repairs the numerical path and successfully retests an isolated long prompt with a fresh engine. A separate check of the submitted cache succeeds on the first insertion but crashes when the same prefix is inserted again; the exception also recurs in final evaluation. The isolated test misses prefix reuse across requests.

Opus 5.5 on Gemma explicitly tests the operating conditions made relevant by its design. After adding preemption and cache restoration to coordinate TTFT and TPOT, it reduces cache capacity to force these paths to execute. The stress test exposes invalid outputs from uninitialized cache storage. The defect predates the preemption change. The agent repairs cache access, checks that unused slots filled with NaNs cannot affect valid computation, and reruns forced preemption while comparing resumed outputs with isolated execution. The final system retains these repairs. Reduced-pool replication passes 0/3 tests with the repair reverted and 3/3 after repair.

Reliable delivery requires testing admission boundaries, shared state, and recovery, then rerunning affected checks after repairs.

Design and Workload Ablations

Finding-guided changes improve serving value and correctness (Appendix § 11). On two fresh common seeds under the final 1,800-second protocol, Opus 5.5’s Qwen v1→v6 redesign at c=16c = 16 raises qualified value by 1.35× and 1.15×. On frozen Gemma at c=40c = 40, workload-selected draft length 7 outperforms synthetic-selected length 3 by 3.0% and 3.9%. For workload-matched verification, Opus 5.5’s Gemma cache repair passes all 3 reduced-pool stress tests (§ 4.5).

Figure 8 fixes implementation, concurrency, and sessions. Across deployments, 300-second windows average 1.42–1.70× the 1,800-second scores, versus 1.04–1.21× at 900 seconds. For Opus 5.5 on Qwen, extending the window from 300 to 1,800 seconds lowers the across-seed coefficient of variation from 0.37 to 0.19, yielding more stable performance estimates. The closer agreement at 900 seconds supports using intermediate replays to guide development. Remaining variation motivates combining these measurements with workload profiling and targeted diagnostics to select configurations and workload conditions for longer replay.

Short windows yield higher average value and sampling sensitivity across systems

Figure 8. Short windows yield higher average value (a); sampling sensitivity varies across systems (b). Colored curves average systems within each deployment.

Conclusion

ServerBench evaluates agentic design of complete, specialized LLM serving systems through an economic service objective, stateful closed-loop replay, and independent verification. Results across three deployments show substantial gains alongside uneven delivery. Trajectory analysis and controlled comparisons highlight three priorities: coordinating capacity improvements with operating load, designing experiments around workload characteristics and variation, and verifying the execution paths introduced by system changes.

Limitations

Our evaluation covers three model–hardware deployments and one coding-trace source. Development and final draws share a corpus; reconstructed inputs preserve workload structure without reproducing original coding tasks. Closed-loop replay differs from open-loop production demand, limiting generalization to other architectures, accelerators, and arrival patterns.

The comparisons characterize measured artifacts. Correlations, source inspection, and selected trajectories do not establish causal optimization effects or design success rates. Repeated agent runs and replays are needed to separate agent variability from workload variability. Version notes record reported design changes and can omit implementation details. Behavioral tests and code review have finite coverage, and calibration depends on the reference engine. Fixed-price token value measures deployment efficiency, excluding development and other operating costs, rather than realized profit.

Ethical Considerations

Recorded coding sessions can contain sensitive information, and agent-generated serving code can introduce unsafe behavior. ServerBench replaces original session content during workload reconstruction and executes generated code in isolated environments with restricted network access. Any release of traces or agent trajectories should be reviewed for sensitive information and comply with source licenses. Post-run code and action review supports the identification of unauthorized behavior. Controlled GPU allocations and experiment resource reporting support responsible use. These controls reduce exposure but do not establish that arbitrary generated code is safe.

Acknowledgements

This work was partially supported by Google Research Award, Google ML & System Junior Faculty Award, Amazon Research Award, Fireworks AI, Intel, Li Auto, Moffett AI, Daytona, and CMU CyLab Seed funding. This material is also based upon work supported by the National Science Foundation under Grant Nos. CCF-2504353 and CCF-2247014, and by IARPA. Any opinions, findings, conclusions or recommendations expressed are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Task Calibration and Accounting

Table Table 2 summarizes the frozen deployment settings.

ParameterQwen3.5 9BGemma E4BQwen3.8 27B INT4
Checkpoint source[21][11][22]
GPU allocation2×L402 \times\mathrm{L40}1×L401 \times\mathrm{L40}2×L402 \times\mathrm{L40}
Context limit (tokens)1,048,576131,0721,048,576
Input price (USD / million tokens)0.10440.02000.0898
Output price (USD / million tokens)0.18550.10002.6570
TTFT target (s)4.100282.5269310.2988
TPOT target (ms)23.4974121.705748.3769
GPU cost (USD / GPU-hour)0.820.820.82
SGLang reference concurrency124
vLLM reference concurrency181

Table 2. Deployment parameters, service targets, and reference operating points.

We calibrate SGLang and vLLM by selecting the concurrency that maximizes gross dollars per dollar. Using 5 independent random seeds, we set each latency target to the largest per-replay p90 to decide the SLO standard, and use the mean p90 TTFT, TPOT and dollars per dollar as the baseline. These targets are frozen before agent development and applied to individual requests; agents choose their own operating concurrency. Changes to the model, hardware, workload, or inference policy require recalibration under the revised deployment conditions.

Reference results average 5 independent replay seeds at the operating points in Table 2.

Scores use the measured window duration and allocated GPU cost. Cached input is valued at the full input-token price and counted once per logical request. Value is computed from client-side token counts. Completed requests contribute to the numerator of Equation (1) when both latency measurements are finite and meet their respective targets.

Replay and Reproducibility Record

The replay uses SemiAnalysis’s AgentX [24] semianalysisai/cc-traces-weka-062126 coding sessions, with the corpus revision fixed by the task. It samples session progress under a closed-loop load. Development and final evaluation use disjoint seed sets drawn from the same source corpus.

Workload reconstruction and eligibility. Deterministic reconstruction replaces original prompt and tool contents while preserving request lengths, shared prefixes, and session topology. Replay uses recorded end-to-start delays and session dependencies, including parallel subagent branches. When no requests are in flight, global idle gaps are capped at 10 seconds by moving all pending timed events earlier by the same amount, preserving their relative spacing, order, and dependency constraints. Context eligibility is fixed before development; where necessary, whole traces are filtered to preserve dependencies without allowing candidates to truncate prompts. State reuse must preserve all required states, including recurrent state where applicable.

Development and deployment. The Transformers starter uses a global request lock and has no cross-request prefix cache. Agents may change scheduling, batching, memory management, kernels, parallel execution, and state reuse. Reference configurations support comparison without prescribing the candidate architecture. Harbor supports container or Modal backends; evaluation components remain outside the editable server. Full development tests launch frozen snapshots, while lightweight checks support shorter experiments. Publishing a version and measuring its performance are separate operations. Portability checks test compatibility with the deployment interface. After the final workload replay, protocol and output tests run while the server remains active. Final evaluation uses the frozen submission after the development budget ends.

Reproduction record. Reproduction requires the effective checkpoint and inference configuration, task and corpus revisions, environment snapshot, agent model identifier and scaffold, development budget, submitted artifact, chosen concurrency, and replay seeds. Immutable version records connect intermediate measurements to code; final results are associated with the submitted artifact. The record separates construction runs from repeated replays of each frozen system.

Output Verification Settings

The answer test samples 20 held-out questions and permits an absolute accuracy decrease of at most 0.1 relative to a plain implementation of the same checkpoint. For nn questions and krefk_{\mathrm{ref}} correct reference answers, the candidate must satisfy

kcand≥max⁡{0,kref−⌊0.1n⌋}.(2)k_{\mathrm{cand}} \ge\max\{0, k_{\mathrm{ref}} - \lfloor0.1n \rfloor\}. \tag*{(2)}

The host runs this test on the frozen submission after development.

Replay checks compare native token identifiers with streamed text and client accounting, and use context probes to test required extended-context behavior. Candidates may not count malformed responses as completed work. These checks assess faithful workload execution and token accounting.

Natural stopping is tested with end-of-sequence termination enabled. The replay requests fixed output lengths, but tokens must still arise from the checkpoint’s computation. Suspicious continuations, including repeated tokens and continuations after an end-of-sequence token, are examined using reference scoring. Up to 16 suspect continuations are sampled, with a maximum per-token log-probability regret tolerance of 4.0 under teacher forcing. Timed replay disables end-of-sequence stopping to obtain prescribed output lengths, but fabricated padding is prohibited. Continuations produced by the prescribed model, including repetition, are permitted.

Integrity Review and Agent Configuration

The task prompt fixes model semantics and prohibits reducing work through truncation, approximate computation, precision reduction, answer retrieval, or model substitution. It also prohibits identifying evaluation probes to alter behavior, manipulating token or timing records, and modifying trusted evaluation components. Computed-state reuse that preserves the specified semantics is admissible.

The prompt announces post-run review of code and recorded actions by a separate agent. The review examines the submitted implementation, version changes, commands, and measurement records. Automated checks record reported model identifiers, network destinations, and commands indicating potential undeclared model use. These records support code inspection alongside the final execution checks. Human review complements these automated assessments when evaluating suspected correctness failures and prohibited shortcuts. Only maximum-reasoning-effort configurations appear in the main comparison; full execution identifiers belong in the campaign reproducibility record.

Development Analysis and Controlled Comparisons

Figure labels. Figure labels shorten Qwen3.8-Max-0902, DeepSeek V4.1 Flash, and MiMo V2.6 Pro to Qwen3.8 Max, DeepSeek V4.1, and MiMo V2.6.

Development resources. Agent token consumption sums uncached input, cache reads, cache writes, and output, including reasoning, across a run’s continuation segments. API cost estimates characterize development expenditure; serving costs are accounted for by allocated GPU time.

Agent API Costs

Table 3 reprices recorded token usage at OpenRouter’s advertised standard rates, retrieved on October 13, 2026.1 We sum uncached input, cache reads, cache writes, and output, including reasoning tokens once. Cache writes use the listed 5-minute rate, or the ordinary input rate when no separate write price is listed. GPT prompts remain below the 272K-token pricing threshold. The estimates cover token and cache-operation charges.

Design Changes and Decision Measurements

The reconstruction connects published versions to their implementation, declared concurrency, measurements, and recorded agent actions. Figure 9 distinguishes implementation changes, load-only changes, coordinated changes, and publications changing neither. The first usable end-to-end measurement requires a non-mock, agent-owned server, replay fidelity, and positive measured value. Implementation inspection records mechanisms enabled on the served path, including scheduling, speculation, cross-turn reuse, host-memory state storage, and CUDA graphs. These inventories show that higher-value systems tend to combine a broader set of serving optimizations, particularly speculative decoding, latency-aware scheduling, and host-memory state storage. Cross-turn reuse is present across all audited systems, while CUDA graphs are also widespread. Decision-episode labels record the triggering signal, hypothesis, workload relevance, comparability, action, follow-up, and support from the recorded comparison. The labeling scheme distinguishes supported, partially supported, unsupported, and unstable conclusions using the decision’s tests and follow-up. Episodes cover tested hypotheses, rollback, retention, and load selection; load changes are linked to their preceding signals. Declared concurrency and the largest experimentally tested load are recorded separately, and trajectories use agent-active time.

Figure 9. Development trajectories connect published implementation changes to the optimizations identified in submitted systems. Filled cells mark latency-aware scheduling (T), speculative decoding (S), cross-turn reuse (R), host-memory state storage (K), and CUDA graphs (G); half-filled K cells indicate recurrent-state storage only. Groups use value relative to each deployment’s strongest reference; outcomes show dollars per dollar or the failed check.

Figure 10 gives the individual trajectories summarized in Figure 5.

Individual concurrency trajectories showing declared concurrency, maximum tested load, number of changes, and final outcome

Figure 10. Individual concurrency trajectories underlying Figure 5. Rows show declared concurrency, the maximum tested load, the number of changes, and the final outcome.

Across the 30 plotted trajectories, runs achieving at least twice the strongest reference value reach a usable end-to-end replay in a median of 0.90 hours, compared with 2.05 hours for runs between one and two times the reference and 2.96 hours for verified runs below it (Figure 11). A usable replay executes the workload and yields positive latency-qualified value. Earlier measurement leaves more of the development budget for subsequent design and testing.

Time to first recorded usable replay by run value category

Figure 11. Higher-value runs establish end-to-end replay earlier, leaving more time for system development. Three failed runs never reach a usable replay.

Verification Coverage and Continued Applicability

Verification annotation distinguishes long-context tests, concurrent requests, state reuse, natural stopping, stopping under load or extended contexts, and cold restarts. Coverage records executed checks and their applicability after subsequent edits. The classification distinguishes checks on the submitted tree, earlier checks followed by relevant edits, and absent coverage.

The failure reconstruction links observed symptoms, agent responses, repairs, and later regressions, including symptoms dismissed or left unexamined and faults recurring after repair.

Post-run review applies the task’s prescribed precision and inference semantics to all submitted systems.

Figure 7 records natural-stopping checks under load or long context for 5 of 33 entries, and prior related symptoms for 8 of the 11 not-passed entries.

Controlled Design and Load Comparisons

Frozen systems are replayed on three common seeds selected before the comparison. Primary matched-system comparisons include Opus 5.5 and Opus 5 on Qwen, Sonnet 5.5 and MiMo on Gemma, and each deployment’s SGLang reference. Additional comparisons include Gemma Opus 5.5, Opus 5, and Qwen3.8-Max-0902. Replays retain the task’s workload, targets, and accounting. Repeated replays quantify workload variation for each frozen system.

Table 4 reports fixed-load version comparisons with qualified-value gains across their matched seeds. Each pair measures the complete set of implementation changes between those versions.

VersionsMatched-seed value ratios
Opus 5.5 / Qwen 9B, c=16c = 16
v1 → v21.88, 1.32
Opus 5 / Qwen 9B, c=2c = 2
v3 → v41.90, 1.67, 1.44
Sonnet 5.5 / Gemma E4B, c=32c = 32
v3 → v41.24, 1.02, 1.05

Table 4. Selected qualified-value gains from successive versions at fixed concurrency. Each value corresponds to a scoreable matched-seed pair.

Load comparisons vary concurrency while retaining the implementation and common seeds. Sonnet and MiMo on Gemma use 2, 10, and 32 sessions; Sonnet’s mean qualified value increases from 1.30 to 2.97. Additional curves compare Gemma Opus 5 at 8, 24, and 40 sessions, Gemma Opus 5.5 at 10 and 40, and the two Qwen Opus systems at one, four, and 16. Warmup completion is recorded separately from scored replay. Figure 4 plots these load comparisons alongside final-evaluation measurements.

Observation Windows and Workload Sampling

Figure 8 rescores prefixes of the same replay, holding implementation, concurrency, and sampled sessions fixed. It shows deployment-level mean window ratios in panel (a) and across-seed coefficients of variation in panel (b).

We profile initial stream availability across 128 workload seeds. For the 16-session Opus 5.5 27B run, the development draw lies near the fourth percentile and the final draw near the median (Figure 12). Initial availability measures the independent streams ready to advance when replay begins.

Workload coverage and latency-qualified value in an Opus 5.5 27B development trajectory

Figure 12. Workload coverage and latency-qualified value in an Opus 5.5 27B development trajectory. (a) Initially available streams across 128 additional workload seeds, with development and final draws marked. (b) Qualified and excluded token value for the same system at 16 sessions under the development and final replays.

The development replay qualifies 76.0% of token value and scores 2.99 dollars per dollar. Under the final draw, the qualified share falls to 16.8% and the score to 0.48. Profiling the session distribution therefore helps interpret the favorable development result and identify workload conditions that warrant further testing.

For the current window curves, each seed’s prefix score is normalized by its 1,800-second score; system-level seed medians are then averaged within each deployment. Panel (b) averages system-level across-seed coefficients of variation. The 27B agent submissions each contribute one recorded replay to panel (a); only its two multi-seed references contribute to panel (b). Deployment summaries weight each system equally.

Finding-Guided Comparisons

We report serving improvements and verification checks from interventions suggested by the three findings on agent-built servers. Table 5 lists the serving improvements. Each pair uses fresh common replay seeds, the same deployment, and the 1,800-second final replay followed by independent correctness checks in both arms. All reported serving arms pass verification. Ratios compare qualified value DD, gross value GG, and qualified share f=D/Gf = D/G. Verification repairs are assessed by targeted correctness tests.

ComparisonSeedDD beforeDD afterDD ratioGG ratioff ratio
CapacityD3.5334.7611.3481.1101.214
CapacityE4.2944.9601.1551.0741.076
Draft lengthD2.2402.3081.0301.0301.000
Draft lengthE2.4502.5441.0391.0291.009
Replay feedbackF0.7860.8881.1291.1121.015
Replay feedbackG0.8781.1481.3081.2431.052

Table 5. Selected serving improvements from finding-guided changes on fresh common seeds. Capacity compares Qwen v1 to v6 at c=16c = 16; draft length compares Gemma K=3K = 3 to K=7K = 7 at c=40c = 40; replay feedback compares two resumed Opus 5 Gemma servers. Ratios are after/before; all arms pass verification.

Capacity at Fixed Load

Opus 5.5’s Qwen3.5 9B v1 and v6 run on 2×L40 at c=16c = 16. The v1→v6 bundle improves qualified value on both confirmation seeds, by 1.35× and 1.15×. Gross value rises by 1.11× and 1.07×, while qualified share rises by 1.21× and 1.08×: the gain combines greater throughput and fewer latency losses. The v1→v6 redesign extends the earlier fixed-load v1→v2 comparison in Table 4.

Workload Feedback and Design Decisions

The corrected prompt-lookup simulator matches prompt and continuation lengths in a crossed comparison of real and synthetic text over 131 requests. At draft length K=7K = 7, pooled emitted tokens per step are 2.635 for real prompts and replies, 1.744 for synthetic prompts with real replies, 1.246 for real prompts with synthetic replies, and 1.244 for fully synthetic text. Emitted tokens include accepted draft tokens plus one verification token. With an assumed incremental draft-cost parameter α=0.01\alpha= 0.01, workload replies select K=7K = 7, whereas fully synthetic text selects K=3K = 3. The pooled draft-length selection holds over α≈0.004\alpha\approx0.004–0.047.

Serving tests change only KK on Opus 5.5’s Gemma v7 at c=40c = 40. The workload-selected K=7K = 7 increases qualified value by 3.0% and 3.9% on the two fresh seeds, mainly through gross value.

To test feedback’s effect on agent behavior, we resume Opus 5’s Gemma v2 checkpoint. Two continuations use the same model, task, checkpoint, and two-hour budget, with development-only workload data. The feedback arm receives its own 205 development-replay replies and their prompt-lookup measurements, then re-adopts speculation. Its submitted server raises qualified value by 1.13×1.13 \times and 1.31×1.31 \times on fresh common seeds F and G.

Verification and Repair

Opus 5.5’s Gemma cache repair masks invalid cache loads, initializes cache pools, and uses NaN-safe argmax. The full repair passes the 420-page concurrent check (1/1), all three 330-page checks exercising preemption and recovery (3/3), and the 330-page check with preemption disabled (1/1). The submitted v7 also passes all three targeted checks (3/3). Masking invalid cache loads and zeroing cache pools each passes its targeted attempt; the full repair passes all three repetitions. These checks verify correct outputs while exercising the server’s cache management and recovery paths.

References

  1. [1]Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve. In 18th USENIX symposium on operating systems design and implementation (OSDI 24), pages 117–134, 2024.arxiv.org/abs/2403.02310
  2. [3]Anthropic. Introducing Claude Opus 5, 2026b. https://www.anthropic.com/news/claude-opus-5.
  3. [6]Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. MLE-bench: Evaluating machine learning agents on machine learning engineering. In International Conference on Learning Representations, volume 2025, pages 50466–50494, 2025.arxiv.org/abs/2410.07095
  4. [7]Zhuoming Chen, Xinrui Zhong, Qilong Feng, Ranajoy Sadhukhan, Yang Zhou, Michael Qizhe Shieh, Zihao Jia, and Beidi Chen. Vortex: Efficient and programmable sparse attention serving for AI agents. arXiv preprint arXiv:2606.06453, 2026.arxiv.org/abs/2606.06453
  5. [8]Audrey Cheng, Shu Liu, Melissa Pan, Zhifei Li, Bowen Wang, Alex Krentsel, Tian Xia, Mert Cemri, Jongseok Park, Shuo Yang, et al. Barbarians at the gate: How ai is upending systems research. arXiv preprint arXiv:2510.06189, 2025.arxiv.org/abs/2510.06189
  6. [9]DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the limits of KV cache compression. arXiv preprint arXiv:2609.19969, 2026. https://arxiv.org/abs/2609.19969.
  7. [10]Leilei Ding, Shumin Wang, Yuting Huang, Fanqi Wan, Yinmin Zhang, Qi Han, Yiming Xu, Feiyuan Zhang, Xiaomeng Chu, Guoliang You, et al. ϕ-bench: Can large language models engineer the infrastructure that powers them? arXiv preprint arXiv:2609.10226, 2026.arxiv.org/abs/2609.10226
  8. [11]Google. Gemma-4-E4B-it, 2026. https://huggingface.co/google/gemma-4-E4B-it. Model card.
  9. [12]Kimi Team. Kimi K3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026. https://arxiv.org/abs/2607.24653.
  10. [13]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PageAttention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626, 2023.arxiv.org/abs/2309.06180
  11. [14]Mike Merrill, Alexander Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. In International Conference on Learning Representations, volume 2026, pages 40903–40986, 2026.
  12. [15]Ayush Nangia, Shikhar Mishra, Aman Gokrani, and Paras Chopra. ISO-Bench: Can coding agents optimize real-world inference workloads? arXiv preprint arXiv:2602.19594, 2026.arxiv.org/abs/2602.19594
  13. [16]Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. Alphaevolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131, 2025.arxiv.org/abs/2506.13131
  14. [18]OpenAI. Using GPT-6, 2026b. https://developers.openai.com/api/docs/guides/latest-model. API documentation. Accessed October 11, 2026.
  15. [19]Anne Ouyang, Simon Guo, Simran Arora, Alex L Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini. KernelBench: Can LLMs write efficient GPU kernels? arXiv preprint arXiv:2502.10517, 2025.arxiv.org/abs/2502.10517
  16. [20]Qwen Team. Qwen3.8-Max-0902, 2026a. https://www.qwencloud.com/models/qwen3.8-max-0902. Official model page. Accessed October 11, 2026.
  17. [21]Qwen Team. Qwen3.5-9B, 2026b. https://huggingface.co/Qwen/Qwen3.5-9B. Model card.
  18. [22]Red Hat AI. Qwen3.8-27B-INT4, 2026. https://huggingface.co/RedHatAI/Qwen3.8-27B-INT4. Model card.
  19. [23]Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, et al. MLPerf inference benchmark. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), pages 446–459. IEEE, 2020.
  20. [24]Semi Analysis. AgentX Harness. https://github.com/SemiAnalysisAI/agentx-harness, 2026a. Open-source software. SemiAnalysis. AgentX methodology and datasets, 2026b. https://inferencex.semianalysis.com/agentx. Dataset and methodology. Accessed October 12, 2026.
  21. [25]Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, and Yiying Zhang. Preble: Efficient distributed prompt scheduling for LLM serving. In International conference on learning representations, volume 2025, pages 37057–37082, 2025.arxiv.org/abs/2407.00023
  22. [26]Yiyou Sun, Xinyang Han, Weichen Zhang, Yuanbo Pang, Tianyu Wang, Yuhan Cao, Yixiao Huang, Chris Duruio, Haoyun Zhang, Jeffrey Lin, et al. Agents’ last exam. arXiv preprint arXiv:2606.05405, 2026. https://arxiv.org/abs/2606.05405.
  23. [27]Jennifer Williams, Dave Farris, Jeff Farris, and Jiantao Jiao. SWE-Serve: Benchmarking agentic engineering for production inference serving. arXiv preprint arXiv:2609.26777, 2026.arxiv.org/abs/2609.26777
  24. [28]Yijia Xiao, Runhui Wang, Luyang Kong, Davor Golac, and Wei Wang. CSR-Bench: Benchmarking LLM agents in deployment of computer science research repositories. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 12705–12723, 2025.arxiv.org/abs/2502.06111
  25. [29]Xiaomi MiMo. MiMo-V2.6-Pro, 2026. https://mimo.mi.com/models/en-US/mimo-v2.6-pro. Official model page. Accessed October 11, 2026.
  26. [30]Zhen Xu, Jingyu Liu, Zongze Li, Tahseen Rabbani, and Ce Zhang. SEIS: Self-evolving inference systems. arXiv preprint arXiv:2610.04646, 2026.arxiv.org/abs/2610.04646
  27. [31]John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50562, 2024.arxiv.org/abs/2405.15793
  28. [32]Jehyeok Yeon, Ben Rank, and Maksym Andriushchenko. InferenceBench: A benchmark for open-ended LLM inference optimization by AI agents. arXiv preprint arXiv:2607.20468, 2026.arxiv.org/abs/2607.20468
  29. [33]Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for Transformer-based generative models. In 16th USENIX symposium on operating systems design and implementation (OSDI 22), pages 521–538, 2022.
  30. [34]Z.ai. GLM-5.3, 2026. https://docs.z.ai/guides/llm/glm-5.3. Developer documentation. Accessed October 11, 2026.
  31. [35]Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody H Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. SGLang: Efficient execution of structured language model programs. Advances in neural information processing systems, 37:62557–62583, 2024.arxiv.org/abs/2312.07104
  32. [36]Zihan Zheng, Zerui Cheng, Zeyu Shen, Shang Zhou, Kaiyuan Liu, Hansen He, Dongruixuan Li, Stanley Wei, Hangyi Hao, Jianzhu Yao, Peiyao Sheng, Zixuan Wang, Wenhao Chai, Aleksandra Korolova, Peter Henderson, Sanjeev Arora, Pramod Viswanath, Jingbo Shang, and Saining Xie. LiveCodeBench Pro: How do olympiad medalists judge LLMs in competitive programming? In Advances in Neural Information Processing Systems, volume 38, 2025. https://proceedings.neurips.cc/paper_files/paper/2025/hash/9712b7386cebcd3db7f1a48c2d20edb–Abstract–Datasets_and_Benchmarks_Track.html.arxiv.org/abs/2506.11928
  33. [37]Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX symposium on operating systems design and implementation (OSDI 24), pages 193–210, 2024.arxiv.org/abs/2401.09670
  34. [38]Shang Zhou, Zihan Zheng, Kaiyuan Liu, Zeyu Shen, Zerui Cheng, Zexing Chen, Hansen He, Jianzhu Yao, Huanzhi Mao, Qiuyang Mang, Tianfu Fu, Beichen Li, Dongruixuan Li, Wenhao Chai, Zhuang Liu, Aleksandra Korolova, Peter Henderson, Natasha Jaques, Pramod Viswanath, Saining Xie, and Jingbo Shang. AutoCode: LLMs as problem setters for competitive programming. In International Conference on Learning Representations, 2026a. https://proceedings.iclr.cc/paper_files/paper/2026/hash/1bd78da0966968409e65b5e0b5dbbb57e–Abstract–Conference.html.arxiv.org/abs/2510.12803
  35. [39]Tianbao Zhou, Yi Wang, Yu Zhou, Zirui Liu, Zhiming Wang, Yebo Peng, Yongfu Wang, Yi Zhang, Jinrun Yin, Kemeng Tian, Fangcheng Fu, Tongxuan Liu, Tao Peng, Tong Yang, Bin Cui, Xupeng Miao, and Ke Zhang. Janus: Multi-LLM serving at production scale. In Proceedings of the ACM SIGOPS 32nd Symposium on Operating Systems Principles, 2026b. doi: 10.1145/3830418.3843867.DOI

Paper details

vlnc:W_g3vnug2x
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 13

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents