betaFeedback

WorksComputer Science

When the Cheapest Feasible Router Fails: Auditing Deployment Commitments in Cost-Aware LLM Routing

Abstract

Cost-aware LLM routing often selects the cheapest policy meeting empirical risk and coverage constraints. Yet a router that passes an audit can violate its fixed service contract on new requests, even when candidate policies are constructed independently. We study this deployment-commitment decision with disjoint construction, audit, and test samples. Across 19 RouterBench task families, the cheapest audit-feasible router deploys in 91.82% of opportunities but violates the held-out contract in 32.22% of deployments. We introduce Menu-Aware Routing Commitment (MARC), which qualifies policies using audit evidence and a serving-cost tolerance to retain the apparent winner, deploy a supported alternative, or defer. Its practical policywise variant (MARC-P) reduces observed conditional failure to 0.73% at 85.79% deployment, with higher serving cost; fixed-margin and formal risk-control procedures provide competitive alternatives. Nested audits show how added evidence changes deployable policy sets, and learned semantic routing exhibits the same commitment failure. These results establish deployment commitment as a distinct operational decision connecting router evaluation, service reliability, and inference cost.

Introduction

Cost-aware language-model routing aims to direct each request to an economical model while preserving useful response quality [6, 8, 19, 21]. Selective routers may also decline to answer, trading model-call cost against the fraction of requests served. Service-aware routing and calibrated selection methods increasingly express such decisions through reliability requirements [13, 24] (Woisetschläger et al. 2025). Consider an operator requiring at most r⋆r^{\star} accepted-query loss and at least c⋆c^{\star} coverage. Among evaluated routing configurations, choosing the least expensive policy that meets both measured thresholds is a natural recommendation. Yet the router that looks cheapest and feasible on an audit sample may not satisfy the promised contract on new traffic.

The deployment problem does not end when a router has been constructed. Given a menu of candidate policies specifying model assignments or abstention, an operator must decide which policy has enough evidence to serve traffic under a fixed contract. Cost optimization can favor weaker model choices or lower acceptance, leaving little headroom against the risk ceiling or coverage floor.

Figure 1. MARC overview. Frozen policies are audited against a fixed risk–coverage contract; evidence and cost tolerance determine whether to retain, replace, or defer. Schematic estimates and intervals.

Sampling fluctuations can then make boundary-adjacent policies appear feasible, while selecting the cheapest among them favors optimistic estimates. These effects connect classical model-selection bias and high-confidence policy evaluation [5, 23, 2] (Laufer-Goldshtein et al. 2023) to a concrete routing decision: which policy serves requests, how many requests it answers, and at what inference cost. Even with a policy menu frozen independently of the audit, the final point-estimate feasibility check remains a consequential source of deployment error.

We study this decision under prespecified risk–coverage contracts and disjoint construction, audit, and test partitions. On RouterBench, the cheapest independently audited feasible router deploys in 91.82% of opportunities yet violates its held-out contract in 32.22% of deployments. Even selecting a router before auditing and applying a separate point-estimate gate yields 21.08% conditional failure at 63.00% deployment. The violations have meaningful magnitude: among failed winner deployments, mean severity is 3.15 percentage points. A separately fitted CARROT-style semantic-routing family shows the same deployment vulnerability. Thus the problem persists beyond reused construction data and beyond hand-designed routing policies.

We introduce Menu-Aware Routing Commitment (MARC) to turn independent audit evidence into an operational decision over a frozen policy menu. MARC first identifies policies supported under the fixed contract, then selects a cost-tolerant alternative with stronger service-support margins. It can retain the cheapest apparent winner, replace that winner with a supported policy, or defer deployment. Two evidence constructions instantiate this interface: practical policywise bootstrap adjustments with explicit margins (MARC-P) and a simultaneous-menu correction based on shared resampling (MARC-S). We evaluate their deployment consequences alongside fixed buffers, Learn-then-Test, Pareto Testing, and exact controls.

Our contributions are threefold. First, we characterize how independently audited, cost-selected LLM routers violate fixed risk–coverage contracts, including the responsible constraint and violation magnitude. Second, we develop a cost-tolerant commitment interface that converts audit evidence into an eligible routing menu and an explicit retain/replace/defer action; we quantify the resulting reliability, deployment, and serving-cost tradeoffs against strong fixed-margin and formal baselines. Third, we show how the eligible policy menu and deployment outcomes change with additional audit evidence and establish the same commitment problem with learned semantic routers.

Related Work

Learning and evaluating economical routers. FrugalGPT, Hybrid LLM, RouteLLM, and CARROT study cascades, learned selectors, and prompt-conditioned choices that balance model quality and inference cost [6, 8, 19, 21]. RouterBench, RouterEval, and LLMRouterBench expose heterogeneous model capabilities and costs across tasks [15, 16, 18]. MESS+ and SLARouter further study service-constrained, dynamic, and online routing (Woisetschläger et al., 2025; Woisetschläger et al., 2026). MARC complements this policy-construction literature by studying the downstream choice of which frozen routing configuration to authorize from independent audit evidence under a fixed contract.

Selective prediction and calibrated routing. Selective classification studies error conditional on answering and its associated coverage [7, 10, 12]; Conformal Risk Control supplies guarantees for suitable bounded-loss calibration problems [1]. Routing-specific methods such as RACER, conformal LLM routing, and CP-Router calibrate model sets, gates, or uncertainty-guided model choices [13, 24, 22]. MARC instead evaluates a complete, frozen policy menu against a joint accepted-risk ceiling and coverage floor, then connects service evidence to affordability and the final policy-level deployment action.

Evidence-supported deployment and multiple testing. High-Confidence Policy Improvement established statistical support as a basis for deployment decisions [23]; model-selection bias and adaptive evaluation motivate accounting for choices among candidates [5, 9, 4]. Learn-then-Test and Pareto Testing provide established routes to risk control and optimization under multiple constraints [2, 17], with Holm, Bentkus-type bounds, and simultaneous resampling supplying relevant statistical tools [14, 3, 25] (Westfall and Young 1993). Recent LLM-routing diagnostics additionally distinguish model-pool opportunity from realizable router performance [20]. We place these approaches on a common fixed-contract routing decision problem and measure their consequences for the selected policy, deployment frequency, held-out violations, and serving cost. This makes the commitment decision itself—rather than only an estimated policy score—the object of evaluation.

Fixed-Contract Routing Deployment

A policy π∈P\pi\in\mathcal{P} maps a query ZZ to a model action or abstention. Let Aπ(Z)∈{0,1}A_{\pi}(Z) \in\{0,1\} indicate an answer, and let ℓπ(Z)∈[0,1]\ell_{\pi}(Z) \in[0,1] be the loss on answered requests. Coverage and selective risk are

C(π)=E[Aπ],R(π)=E[Aπℓπ]E[Aπ](1)C(\pi) = \mathbb{E}[A_{\pi}], \qquad R(\pi) = \frac{\mathbb{E}[A_{\pi}\ell_{\pi}]}{\mathbb{E}[A_{\pi}]} \tag*{(1)}

with RR defined at positive coverage. The population service contract is R(π)≤r⋆R(\pi) \le r^\star and C(π)≥c⋆C(\pi) \ge c^\star. We distinguish the policy-level serving-cost objective J(π)J(\pi), computed per incoming query under the benchmark’s abstention convention, from economic cost of refusing to deploy a policy at all. The latter may involve fallback or latency and is not observed as a single number in the traces.

The numerical pair (r⋆,c⋆)(r^\star, c^\star) and the finite menu P\mathcal{P} are fixed independently of the commitment audit. Using audit estimates R^,C^,J^\widehat{R}, \widehat{C}, \widehat{J}, the cheapest apparently feasible winner is

M^={π:R^(π)<r⋆,C^(π)≥c⋆},π^win=arg⁡min⁡π∈M^ J^(π).(2)\widehat{\mathcal{M}} = \{\pi: \widehat{R}(\pi) < r^\star, \widehat{C}(\pi) \ge c^\star\}, \qquad \widehat{\pi}_{\mathrm{win}} = \underset{\pi\in\widehat{\mathcal{M}}}{\arg\min}\ \widehat{J}(\pi). \tag*{(2)}

An empty feasible menu implies deferral. Define the joint population and empirical contract margins

m(π)=min⁡{r⋆−R(π),C(π)−c⋆},m^(π)=min⁡{r⋆−R^(π),C^(π)−c⋆}.(3)m(\pi) = \min\{r^\star- R(\pi), C(\pi) - c^\star\}, \qquad \widehat{m}(\pi) = \min\{r^\star- \widehat{R}(\pi), \widehat{C}(\pi) - c^\star\}. \tag*{(3)}

A point-feasible winner has m^(π)≥0\widehat{m}(\pi) \ge0, not necessarily m(π)≥0m(\pi) \ge0. The problem combines two sources of uncertainty: empirical margins for any fixed policy can be wrong, and selection among many audit-evaluated candidates can exploit such fluctuations. The independent-gate comparator fixes a winner on a separate selection sample before applying a point-feasibility test on audit data. Because its deployed population differs, its outcome is not a causal estimate of selection bias removed; it establishes that an additional point gate remains insufficient evidence in this evaluated setting.

The data chronology is construction →\to freeze routing menu and numerical targets →\to independent commitment audit →\to commit →\to untouched held-out test. A committed policy is evaluated on fresh test samples without changing its thresholds. Let DD indicate commitment and FF a fresh contract violation, Rtest>r⋆R_{\mathrm{test}} > r^\star or Ctest<c⋆C_{\mathrm{test}} < c^\star. We report deployment P(D)P(D), unconditional deploy-and-fail P(D∩F)P(D \cap F), conditional failure P(F∣D)P(F \mid D) (undefined when no deployment), and severity

v=max⁡{Rtest−r⋆,c⋆−Ctest,0}.(4)v = \max\{R_{\mathrm{test}} - r^\star, c^\star- C_{\mathrm{test}}, 0\}. \tag*{(4)}

Observed held-out failure is a measurable experiment outcome, whereas an audit’s simultaneous-support statement is a statistical guarantee subject to its assumptions. Neither implies the other mechanically. Our primary inferential unit is the dataset, not every seed/contract/cost cell, because the latter share underlying examples and model outcomes.

MARC: Evidence-Aware Commitment

MARC is a commitment interface applied after candidate routers have been constructed. It qualifies candidates against a fixed contract, forms an affordable subset of the qualified menu, and deploys a policy with relatively strong evidence or defers. The two versions below differ in evidence construction, not in the deployment decision.

MARC-P: policywise practical evidence. For policy π\pi, bootstrap the shared query-level audit records to form one-sided bounds URα(π)U_R^\alpha(\pi) for accepted-query risk and LCα(π)L_C^\alpha(\pi) for coverage. The eligible menu is

BP={π:r⋆−URα(π)≥δR,LCα(π)−c⋆≥δC}.(5)\mathcal{B}_{\mathrm{P}} = \{\pi: r^\star- U_R^\alpha(\pi) \ge\delta_R, L_C^\alpha(\pi) - c^\star\ge\delta_C\}. \tag*{(5)}

Explicit buffers δR,δC\delta_R, \delta_C are fixed; bootstrap adjustments vary with the audit evidence and policy. Policywise intervals do not, by themselves, validate arbitrary post-selection choice over the menu. Accordingly, MARC-P’s small test failure rates are empirical observations, not an unconditional finite-sample familywise certificate.

MARC-S: menu-level simultaneous evidence. For policy π\pi, let m^(b)(π)\widehat{m}^{(b)}(\pi) be its margin on common query-level bootstrap resample bb. Set

T(b)=max⁡π∈P{m^(π)−m^(b)(π)},T^{(b)}=\max_{\pi\in\mathcal{P}}\{\widehat{m}(\pi)-\widehat{m}^{(b)}(\pi)\},
q1−α=Q1−α{T(b)}b=1B.(6)q_{1-\alpha}=Q_{1-\alpha}\{T^{(b)}\}_{b=1}^{B}. \tag*{(6)}

The common correction yields

BS(δ)={π:m^(π)−q1−α≥δ}.(7)\mathcal{B}_{S}(\delta)=\{\pi:\widehat{m}(\pi)-q_{1-\alpha}\ge\delta\}. \tag*{(7)}

Shared resamples preserve policy dependence; valid simultaneous coverage extends to an audit-selected policy from the frozen menu. Appendix § 12 gives the conditional finite-menu asymptotic result and a proof, not a distribution-free finite-sample guarantee. Positive population coverage, separated active margin components, and bootstrap regularity matter, as do the source’s finite-BB upper empirical-quantile semantics.

Affordable selection and three actions. For either eligible menu B\mathcal{B}, defer if it is empty. Otherwise select among candidates within a frozen relative cost tolerance of the least expensive eligible policy:

Bκ={π∈B:J^(π)≤(1+κ)min⁡ρ∈BJ^(ρ)}.(8)\mathcal{B}_{\kappa}=\{\pi\in\mathcal{B}:\widehat{J}(\pi)\le(1+\kappa)\min_{\rho\in\mathcal{B}}\widehat{J}(\rho)\}. \tag*{(8)}

MARC-P ranks this affordable menu by the sum of risk and coverage bound-adjusted slacks; MARC-S ranks by the simultaneous lower joint margin. Deterministic ties use cost and frozen policy ID. The output is retain if the selected policy equals the cheapest apparent audit winner, replace if it is different, and defer if no qualified policy exists. The output therefore changes the routing policy or withholds deployment, rather than simply testing the cheapest candidate.

The displayed MARC settings are nominal α=.10\alpha=.10, κ=.25\kappa=.25, B=1000B=1000, and explicit δR=δC=.05\delta_R=\delta_C=.05 for MARC-P; the frozen MARC-S menu contains δ∈{0,.02,.05}\delta\in\{0,.02,.05\}. These rules have different validity scopes; implementation details and tie conventions appear in Appendix § 11.

Experimental Design

Primary RouterBench study. We use frozen cached model quality and cost outcomes from RouterBench (Hu et al., [15]), covering 19 task families and deterministic menus of simple/selective policies. For each of 20 split seeds, 20% of the data constructs the policies, 20% provides the independent deployment audit, and 60% is untouched test data. The fixed contracts are all six combinations of r⋆∈{.30,.50,.70}r^\star\in\{.30,.50,.70\} and c⋆∈{.40,.60}c^\star\in\{.40,.60\}; the evaluation includes the frozen binary and bounded correctness channels and separate mean and 90th-percentile tail serving-cost objectives. MARC uses B=1000B=1000 bootstrap replicates; full provenance appears in Appendix § 10.

Learned semantic-routing study. A second fixed-contract experiment uses a subset of LLM-RouterBench (Li et al., [18]): 10 datasets, 12,444 queries, 11 models and a menu of 48 CARROT-style KNN operating policies (Somerstep et al., [21]). Query embeddings inform estimated model suitability; frozen cost preferences and acceptance thresholds define the menu. The split is 40% fitting, 20% policy selection, 20% independent audit and 20% untouched test, repeated for 20 seeds. The experiment covers one learned architecture on a selected ten-dataset subset (Appendix § 17).

Comparator and evaluation scope. We compare the cheapest point-feasible winner; an independently selected winner subsequently audited with a point gate; the full frozen invariant-margin family (δ∈{0,.02,.05,.08,.10,.15}\delta\in\{0, .02, .05, .08, .10, .15\}); MARC-P/S; Hoeffding–Bentkus LTT with Bonferroni and Holm multiplicity; Pareto Testing; and exact-binomial Holm/Bonferroni tests where the score truly is Bernoulli ([2] (Laufer-Goldshtein et al. 2023)). Nominal α=.10\alpha= .10 comparisons use applicable methods on compatible score domains; formal-only α=.05\alpha= .05 and binary exact controls are reported separately. The stricter (r⋆,c⋆)=(.30,.60)(r^\star, c^\star) = (.30, .60) contract is an illustrative member of the original grid, not an endpoint nominated after seeing its outcome.

Sampling and outcome estimands. Each dataset’s repeated seeds, contracts, score settings, and cost objectives are summarized first and dataset means are then weighted equally in primary RouterBench tables. The 19 task families include 12 MMLU-derived subdomains, so these are reporting clusters rather than 19 distinct benchmark sources. We supply dataset-cluster bootstrap intervals and leave-one-dataset-out sensitivity rather than counting 9,120 correlated decision records per method as independent statistical replicates. Learned-router percentages are pooled; conditional metrics are undefined when their denominators are zero. Serving cost is per incoming query with zero policy-level abstention cost, reported among commitments; fallback and commitment-stage deferral are unpriced.

Evidence-size sensitivity. The separate, pre-frozen audit-size study uses a 20% construction sample, a 40% audit reservoir, and a 40% common untouched test. Deterministic query hashes create nested A10⊂A20⊂A40A_{10} \subset A_{20} \subset A_{40} audits of 10%, 20%, and 40% of each full dataset. Policies, contracts, settings, and test data remain fixed; MARC-P is compared with the complete fixed-margin sweep.

Results

Why independent point feasibility can fail

The cheapest audit-feasible policy deploys in 91.82% of RouterBench opportunities, yet 32.22% of those deployments violate the fresh contract (29.63% unconditional deploy-and-fail; Table 1). An independent point-feasibility gate, applied to a candidate selected on separate data, deploys in only 63.00% of cases and still fails in 21.08% of its deployments. Thus an extra validation split reduces exposure but does not turn point feasibility into evidence-backed authorization. The comparison highlights the practical difference between independently checking point feasibility and accounting for uncertainty when authorizing a policy: a separate gate still leaves substantial held-out violations.

ProcedureDeployD&FFail|D
Audit winner91.8229.6332.22
Independent gate63.0012.3821.08
Point margin .0091.8222.2524.40
Fixed margin .0587.726.067.25
Fixed margin .1086.321.431.78
Fixed margin .1582.740.290.38
MARC-P .0585.790.630.73
Pareto Testing84.410.490.60
LTT-Holm78.380.090.12
MARC-S .0271.930.130.21
MARC-S .0568.030.000 observed

Table 1. RouterBench full-grid results (dataset-equal percentages). D&F: deploy-and-fail; Fail|D: dataset-averaged conditional failure, not necessarily D&F/Deploy. MARC suffixes denote buffers δ\delta; guarantee scopes differ. Binary-only exact tests appear in Appendix E.

The winner’s failures are not merely infinitesimal threshold crossings: among violating deployments, the maximum risk/coverage excess averages 3.15 percentage points (median 1.94, p90 7.39). In the full grid, 10.83% of opportunities incur risk-only violations, 16.64% coverage-only, and 2.15% both (Appendix § 15). For MARC-P .05, the corresponding unconditional violation rates are 0.54% risk-only, 0.09% coverage-only, and 0.00% joint, compared with 10.83%, 16.64%, and 2.15% for the audit winner. Coverage misses are especially consequential for selective routing: a policy can improve accepted-query accuracy by abstaining more often yet breach its minimum service-availability commitment. These are observed failure compositions of different deployment decisions, not effects of an isolated algorithm component.

The frozen strict contract (r⋆,c⋆)=(.30,.60)(r^\star, c^\star) = (.30, .60) exhibits the same problem: the winner deploys in 81.32% of cells (652 failures / 1,236 deployments), versus MARC-P’s 65.00% (8/988). MARC-S .02 and LTT-Holm have no observed failures among 600 and 772 deployments, respectively. These are finite held-out counts, not guarantees of zero population failure. The six-contract grid remains the primary evaluation; the strict cell is illustrative.

What evidence changes in the deployment decision

The operating points reveal how both feasibility evidence and policy ranking shape deployment. The cheapest winner deploys in 91.82% of opportunities with 29.63% deploy-and-fail. Zero-margin, cost-tolerant reranking maintains deployment while lowering failure exposure to 22.25%, but its effect depends strongly on the objective: only 29.52% → 29.17% for mean cost versus 29.74% → 15.33% for tail cost (Table 2). A fixed .05 margin yields 87.72% deployment and 6.06% deploy-and-fail; MARC-P .05 yields 85.79% and 0.63%. Reranking matters, but cannot substitute for supported feasibility. The mean-cost result is especially instructive: changing the selected point-feasible policy alone barely changes the failure exposure, whereas adding margin or uncertainty evidence sharply reduces it. For tail cost, changing the ranking itself can have a large effect, so the contributions of eligibility and ranking are not interchangeable across economic objectives.

ObjectiveWinnerFixed .00Δ\DeltaMARC-P .05
Mean cost29.5229.17−0.350.77
Tail cost29.7415.33−14.410.48

Table 2. Deploy-and-fail by optimization objective (percent). Δ\Delta is fixed .00 minus the audit winner, in percentage points.

The action comparison (Table 3) makes the deployment decision tangible. MARC-P .05 replaces the apparent winner in 77.11% of opportunities and defers in 14.21%; its improvement therefore involves authorizing different routing policies, not solely withholding service. Pareto Testing and LTT-Holm also replace many winners, but the methods produce different retain/replace/defer mixes, with corresponding reliability, availability, and serving-cost consequences.

ProcedureRetainReplaceDefer
Audit winner91.820.008.18
LTT-Holm8.6469.7421.62
MARC-P .058.6877.1114.21
MARC-S .028.4663.4628.07
Pareto17.8166.6015.59

Table 3. RouterBench commitment actions (percent of all decision opportunities; dataset-equal).

More stringent evidence rules occupy different operating points. MARC-P has 85.79% deployment and 0.73% conditional failure; Pareto Testing has 84.41% and 0.60%, while LTT-Holm has 78.38% and 0.12%. MARC-S at .02 deploys in 71.93% with 0.21% observed conditional failure. Conservative fixed margins are also competitive. The procedures offer distinct combinations of observed reliability and deployment, with statistical guarantees specific to each construction. MARC-P uses practical policywise bounds; MARC-S uses a simultaneous asymptotic correction; and formal tests supply their corresponding risk-control guarantees. These differences make serving cost and the frequency of withheld commitments essential to comparing operating points.

These reliability gains have a serving-cost consequence (Figure 2). Under mean-cost optimization, the deployed policy costs about 0.000438 trace units per incoming request for the winner versus 0.001071 for MARC-P, using each rule’s committed population. Matching only opportunities where both deploy gives a MARC-P premium of 0.000844 over the winner (3,912 paired decisions; 95% dataset-bootstrap interval [0.000408, 0.001444]). Against fixed margin .10, the matched difference is +0.000070 (interval [−0.000019, 0.000159]); the interval crosses zero. Comparisons with Pareto and LTT-Holm instead yield −0.000243 and −0.000517 on their respective shared deployment populations. These pairwise comparisons have different matching sets and exclude fallback or nonservice costs. Mean and tail serving-cost objectives are also distinct; Appendix § 18 reports objective-specific estimates, uncertainty, and frozen penalty scenarios. The method-specific deployed averages answer what a rule pays on the opportunities it actually commits; common-deployment comparisons instead isolate the cost of the replacement policy on identical opportunities. Neither estimates the economic effect of cases where only one procedure deploys. This distinction matters because an apparent cost reduction may arise from serving a different mix of tasks rather than routing the same tasks more economically.

Mean-cost RouterBench operating points

Figure 2. Mean-cost RouterBench operating points: (a) deployment versus deploy-and-fail; (b) selected serving cost versus deploy-and-fail. Costs are trace units per incoming query among deployed policies. Dashed lines connect frozen parameter sweeps, not interpolated policies; deferral costs are unmeasured.

When additional audit information becomes available

The nested-audit study holds policies, fixed contracts, method settings, and a common test set constant while expanding the audit A10⊂A20⊂A40A_{10} \subset A_{20} \subset A_{40}. It uses a separate 20/40/40 construction/audit-reservoir/test partition and paired comparisons within dataset and seed (Appendix § 16).

As the audit grows from 10% to 40%, MARC-P .05 increases deployment from 83.55% to 86.58% while reducing deploy-and-fail from 1.34% to 0.42% (Figure 3). The paired A40–A10 differences are +3.03 and −0.92 percentage points; mean eligible-set size rises by about 1.49 policies. Dataset-bootstrap intervals are [+1.01, +5.44] and [−1.67, −0.22] points, respectively. The increase in eligible-set size shows how additional audit evidence can make more cost-aware commitments available. The response varies by dataset: nine gain deployment, two lose it, and eight are unchanged between A10 and A40. Paired shifts and supported-menu expansion together characterize the practical effect of collecting more audit data.

Figure 3. Figure 3: Nested-audit comparison at 10%, 20%, and 40%: (a) deployment and (b) deploy-and-fail, with between-dataset intervals. MARC-P and the fixed-margin controls keep their settings unchanged. Paired audit-size effects appear in Appendix § 16.

On opportunities deployed at both audit sizes, MARC-P also selects lower-cost policies at A40 than at A10 (ΔJ=−0.000193\Delta J = -0.000193 in frozen trace units per incoming query; Appendix § 16). This matched comparison isolates a change in policy choice, while the deployment-rate increase captures newly authorized opportunities. The two measures together help distinguish benefits from a larger eligible menu from benefits that merely reflect serving fewer requests.

Fixed margins offer informative alternatives. The .05 buffer has 89.50% deployment and 10.73% deploy-and-fail at A10, versus 87.02% and 3.16% at A40; fixed .15 is comparably conservative to MARC-P at A10. At A40, fixed .10 closely matches MARC-P (86.47% versus 86.58% deployment; 0.44% versus 0.42% deploy-and-fail). MARC-P’s policy-specific uncertainty adjustment responds to additional evidence without changing its preset numerical buffer, offering a practical way to manage commitments across audit-size regimes while preserving competitive fixed-margin choices.

The same deployment issue arises in learned semantic routing

In the learned CARROT-style study, query embeddings inform model-suitability predictions, while 48 frozen policies vary cost preferences and acceptance thresholds. The learned-menu protocol separates fitting, operating-policy selection, commitment auditing, and final testing. Unlike the hand-designed policy families, these routers make text-dependent model choices; the same fixed-contract commitment issue persists when routing policies are learned before the audit.

In the pooled learned-router full grid (Table 4), the winner fails on 334 of 983 commitments (33.98%) at 81.92% deployment. MARC-P .05 fails on 31 of 828 (3.74%) at 69.00% deployment; MARC-S .02, Pareto Testing, and LTT-Holm achieve lower observed failure with fewer deployments. Unlike dataset-equal RouterBench aggregates, these results pool 1,200 learned-router decision opportunities. The strict contract in the same table isolates a more demanding service target without changing the frozen evaluation protocol.

ProcedureFull frozen grid (N=1,200N = 1{,}200) DeployD&FFail|DFail/dep.Illustrative strict contract (N=200N = 200) DeployFail|DFail/dep.
Audit winner81.9227.8333.98334/98367.5041.4856/135
Fixed .0576.839.4212.26113/92262.5020.0025/125
MARC-P .0569.002.583.7431/82845.508.798/91
Pareto Testing58.250.500.866/69925.500 obs.0/51
LTT-Holm51.330.170.322/61621.000 obs.0/42
MARC-S .0256.170.500.896/67423.000 obs.0/46
MARC-S .0553.170.250.473/63820.000 obs.0/40

Table 4. Learned-router commitments (pooled percentages; mean-cost objective). Full grid: N=1,200N = 1{,}200; strict (r⋆,c⋆)=(.30,.60)(r^\star, c^\star) = (.30, .60): N=200N = 200. Fail/dep.: observed failure/deployment counts.

At the strict (.30, .60) contract, MARC-P deploys in 45.50% of opportunities with 8 failures among 91 commitments. MARC-S .02, Pareto, and LTT-Holm observe no failures among 46, 51, and 42 commitments but authorize only 23.00%, 25.50%, and 21.00% of opportunities, respectively. The strict-contract contrast makes the availability cost of stronger evidence requirements visible, alongside the reduction in observed failures.

The pooled average also masks distinct task-level decisions. On ArenaHard, MARC-P authorizes 108 of 120 opportunities with no observed failures, compared with 39 failures among 120 winner deployments. On LiveCodeBench, it authorizes all 120 opportunities with two observed failures, versus 41 for the winner. On tau2, it authorizes 83 of 120 with six failures, compared with 58 failures in the winner’s 114 deployments. HLE and SWE-bench present the opposite operational outcome: MARC-P defers every opportunity in this panel. These contrasts (Appendix § 17) show why both authorization rate and conditional failure matter when a frozen learned-router menu is evaluated across heterogeneous NLP tasks.

Serving cost follows the same practical tradeoff. Among deployed learned-router policies under mean-cost optimization, the winner costs 0.007221 trace units per incoming query and MARC-P .05 costs 0.012464; Pareto and LTT-Holm cost 0.012916 and 0.012046, respectively. These method-specific committed populations differ, so the values describe selected serving cost rather than a matched end-to-end cost comparison. Residual failures are rarer under MARC-P but are not uniformly mild: conditional mean severity is 0.0709, versus 0.0624 for the winner. Detailed per-task counts and severity appear in Appendix § 17.

Discussion and Conclusion

Cost-aware router evaluation requires a deployment decision beyond choosing the cheapest policy that passes empirical thresholds. Across frozen policies and learned semantic routing, independent audit estimates can favor configurations with insufficient service-contract headroom. MARC makes that last decision explicit: evidence-supported eligibility and serving-cost tolerance determine whether the apparent winner is retained, replaced by another router, or deferred.

These choices have measurable consequences. MARC-P typically replaces the cheapest apparent winner rather than only withholding deployments; formal controls and fixed margins offer different combinations of reliability, deployment rate, and serving cost. As audit evidence grows, MARC-P supports more policies and can authorize more deployments with fewer observed violations. These results expose a practical reliability–deployment–cost tradeoff. Evaluating a commitment requires accounting for the reliability of its service claims, the policies it authorizes, the opportunities it defers, and the serving cost of those choices.

Limitations

The results come from cached benchmark outcomes and randomized independent partitions, not live traffic subject to domain shift, model updates, incomplete feedback, or changing service promises. The learned semantic panel covers ten datasets, eleven models and one KNN router family, with sparse deployment on some tasks. The 19 RouterBench task families, not repeated cells within them, are the reporting clusters for descriptive benchmark uncertainty. Because 12 are MMLU-derived subdomains, cluster bootstrap intervals describe variation across the evaluated task panel rather than independent generalization across 19 distinct benchmark sources. MARC-P’s policywise bootstrap is not a generic finite-sample post-selection certificate, and MARC-S’s joint-menu bootstrap justification is conditional and asymptotic under positive coverage and active-constraint regularity. Formal controls have their own null, score-channel and familywise assumptions. The evaluation measures specific frozen menus and cannot assign a monetary value to withheld deployment, fallback, delays, or service-contract violations. Contract thresholds are fixed before auditing.

Ethical Considerations

The study analyzes cached evaluation traces and does not deploy policies to users. Low average selective loss or a formal statistical threshold does not establish suitability for high-stakes decisions, nor does it ensure fairness across user subpopulations. Production deployments require independently justified losses, coverage requirements, monitoring, safe fallback behavior, and re-auditing under distribution shift.

Data, Contracts, and Experimental Chronology

Data scope and inferential unit. RouterBench’s 19 dataset/task families are the reporting clusters used for descriptive benchmark-level uncertainty. In the frozen full-grid experiment, each method has 9,120 decision records: 19 datasets, 20 seeds, six risk–coverage contracts, two cost objectives, and applicable score channels; the binary-only exact-testing panel has a smaller domain. Repetitions from one dataset share cached LLM outcomes and are not treated as 9,120 independent population samples. The learned panel has ten task datasets and its mean-cost comparison has 1,200 frozen decision opportunities over all contracts and seeds. Its pooled numbers are intentionally not pooled with the dataset-equal RouterBench metrics.

Trace costs and abstentions. Model inference cost is inherited from the frozen benchmark trace and assembled per routing policy. A policy that abstains on a request receives the frozen zero policy-level serving-cost convention on that request. The resulting mean-cost objective is therefore per incoming query, not per answered request. A tail-cost objective separately summarizes an upper-quantile cost functional. The commitment-stage act of declining to deploy a policy is a different event; its downstream fallback or nonservice cost was not measured. All economic results are conditional on deployment decisions unless stated otherwise.

Zero accepted queries. A policy accepting no audit queries has undefined conditional-risk estimation and cannot be treated as satisfying the risk constraint. Positive population coverage is separately required for MARC-S’s asymptotic result. For ARC Challenge, inference costs use the per-query token/length aggregation in the evaluation trace.

Evaluation units. A decision opportunity consists of a dataset, split seed, frozen candidate menu, loss channel, cost objective, and fixed risk/coverage contract. The test sample is used solely to assess the selected policy. RouterBench has 19 task families and 20 seeds. Its primary construction/audit/test shares are 20/20/60 percent. LLMRouterBench has 10 tasks and a 40/20/20/20 fit/selection/audit/test chronology, 20 seeds, and 48 deterministic KNN policies. Contract thresholds are enumerated before the audit; no downstream target selection is permitted. Repeated split seeds and contract cells are not independent task replications.

Model and candidate-policy scope. The primary menu uses the frozen policies from the RouterBench input manifest: each construction split selects a least-cost model and a strongest model using construction-only statistics, then defines preset selective/length- and confidence-threshold operating policies. The frozen construction uses three prompt-length quantiles (.25.25, .50.50, .75.75), three confidence thresholds (.55.55, .65.65, .75.75), and four acceptance thresholds (.40.40, .55.55, .70.70, .85.85), alongside predefined policy IDs and tie ordering. These policies are evaluated as an already constructed menu, not tuned on audit or test outcomes. The learned panel instead fixes 48 CARROT-style KNN operating policies using independent fit and selection splits. The accompanying reproducibility package provides the evaluated model pools, filtering rules, construction code, and source manifests.

Frozen settings. RouterBench risk targets are r⋆∈{.30,.50,.70}r^\star\in\{.30,.50,.70\} and minimum coverage targets c⋆∈{.40,.60}c^\star\in\{.40,.60\}. Both bounded correctness and binary correctness channels are included where valid, and the mean-cost and 90th-percentile tail-cost objectives are kept separate for economic inference. Main MARC experiments use B=1000B = 1000, nominal α=.10\alpha= .10, and κ=.25\kappa= .25; MARC-P’s principal explicit risk and coverage margins are .05.05 each and the MARC-S simultaneous buffer is varied on its frozen grid. Bounds and risk estimators are recomputed from accepted-query observations; a zero-acceptance bootstrap replicate is handled conservatively under the frozen canonical rule. Identical resampled query indices are applied to all frozen policies within a replicate.

Construction and reproducibility. Two independent executions of the RouterBench analysis with B=1000B = 1000 yielded identical 200,640 commitment decisions and 319,200 intermediate audit records. For learned routing, deterministic reconstruction from verified traces and embeddings matched 800 reference router diagnostics and 30,438 policy-decision metrics; repeated evaluations agreed. Reported results use these validated records.

Sample size and validity. Formal testing baselines use their canonical Hoeffding–Bentkus p-values and multiple-testing conventions. Exact-binomial controls use Bernoulli outcomes and therefore are not applied to fractional losses. The same nominal α\alpha need not entail the same theoretical guarantee across MARC-P, MARC-S, and formal controls. Zero observed failures indicate a sample result, not a zero population violation probability.

Commitment Procedures and Formal Comparators

Commitment-rule pseudocode.

  1. Freeze inputs: candidate menu, (r⋆,c⋆)(r^\star, c^\star), policy IDs, loss channel, cost objective, method, α\alpha, δ\delta, κ\kappa, and bootstrap seed; none may depend on commitment-audit outcomes.

  1. Audit: calculate each policy’s accepted-query risk, coverage, objective-specific serving cost, and any method-specific bootstrap bounds or valid component p-values. A policy with zero accepted audit queries has undefined/infinite risk and cannot pass the risk requirement.

  1. Qualify: construct the method-specific set B\mathcal{B} using point margins (fixed-buffer control), policywise bootstrap inequalities (MARC-P), simultaneous lower joint margins (MARC-S), or the formal testing/ordered-frontier acceptance rule. The empirical audit winner is selected by minimum feasible cost instead.

  1. Commit: if B=∅\mathcal{B} = \varnothing, defer; otherwise restrict to (1+κ)(1+\kappa) times the cheapest supported audit cost (zero minimum permits only zero-cost policies). Choose maximum method-specific safety score,

then apply the prespecified deterministic cost/policy-ID tie rules. Pareto Testing retains its own cheapest accepted-prefix selection rule.

  1. Evaluate: compare the emitted policy with the audit-feasible cheapest policy (retain versus replace); compute held-out contract failure only after the decision is fixed. Record the entire menu and support set, not only the winner.

The methods use distinct margin scores: MARC-P maximizes the sum of its bound-adjusted risk and coverage slacks; MARC-S maximizes a simultaneous lower joint margin; fixed-margin and LTT controls rank by empirical joint margin within their own eligible and affordable sets. A joint unsafe policy null is handled by the maximum of component risk/noncoverage p-values before applying Holm or Bonferroni across policies. Pareto’s ordering arises on its separate optimization split and stops testing at the first policy failing the strict p<αp < \alpha criterion. The final action space is shared, while eligibility tests and selection rules differ.

MARC-P. The frozen source evaluates one-sided policy-specific bootstrap quantiles of risk and coverage, shifts each point-estimated contract boundary by the resulting uncertainty bound, and enforces explicit buffers δR,δC\delta_R, \delta_C. For MARC-P, the risk-upper quantile uses finite bootstrap replicate risks (and is +∞+\infty if none are finite), while the coverage-lower quantile includes replicates with zero accepted requests; the frozen risk and coverage marginal quantiles use the linear empirical convention. This differs from MARC-S, which maps zero-acceptance replicate joint margins to −∞-\infty and uses the higher convention for its menu-level upper maximum-deviation quantile. If there is no supported policy, MARC-P defers; otherwise the cost-tolerant subset of Eq. (8) is evaluated using the sum of bound-adjusted slacks. The replicate budget and empirical quantile convention are fixed implementation choices.

MARC-S. The source uses one common maximum downward bootstrap deviation across the entire frozen policy family; its empirical upper order statistic (higher convention) is subtracted from every policy’s empirical joint margin. This preserves common-resample policy dependence. Cost-tolerant selection then maximizes the resulting lower joint margin. Neither method retrains or changes policy predictions during auditing.

Independent point gate and fixed margins. The independent gate first constructs a winner on separate selection data, then uses point-estimated risk and coverage from audit data to decide whether to deploy that frozen winner. It is not an uncertainty-corrected feasibility certificate. The fixed-margin family replaces point-feasibility inequalities with Rˉ≤r⋆−δ\bar{R} \le r^\star- \delta and C^≥c⋆+δ\widehat{C} \ge c^\star+ \delta, followed by the same frozen cost tolerance and selection convention where applicable. All .00, .02, .05, .08, .10, .15 buffers are retained in the frozen grid.

LTT, Holm/Bonferroni, and Pareto. Learn-then-Test (LTT) and Pareto Testing are substantial alternatives with different validity premises from MARC-P. The canonical controls apply Hoeffding–Bentkus bounded-loss hypothesis tests [3]; multiplicity is handled by the named Bonferroni or Holm rule [14], and Pareto uses its canonical fixed-sequence stopping convention. Their guarantees depend on the chosen testing families, bounded-loss assumptions, and calibration protocols; observed test failures are reported separately from those guarantees. Exact-binomial controls employ valid Bernoulli testing only on the binary score domain. The primary comparison uses nominal α=.10\alpha= .10; the existing formal-only .05 rows are reported as sensitivity, never mistaken for matched MARC .05 configurations.

Statistical Validity and Its Scope

This section gives an asymptotic justification for the menu-level max-deviation correction when the candidate policies and their parameters are fixed independently of the sample used for the commitment audit.

Let Z1,…,ZnZ_1,\ldots,Z_n denote i.i.d. audit observations. For a fixed policy π\pi, let Aπ(Z)∈{0,1}A_\pi(Z) \in\{0,1\} indicate acceptance and let ℓπ(Z)∈[0,1]\ell_\pi(Z) \in[0,1] denote loss. Define

C(π)=E[Aπ(Z)],N(π)=E[Aπ(Z)ℓπ(Z)],R(π)=N(π)C(π).(9)\begin{aligned} C(\pi) &= \mathbb{E}[A_\pi(Z)],\\ N(\pi) &= \mathbb{E}[A_\pi(Z)\ell_\pi(Z)],\\ R(\pi) &= \frac{N(\pi)}{C(\pi)}. \tag*{(9)} \end{aligned}

Their empirical counterparts are denoted by C^n(π)\widehat{C}_n(\pi), N^n(π)\widehat{N}_n(\pi), and R^n(π)\widehat{R}_n(\pi). The population and empirical joint margins are

m(π)=min⁡{r⋆−R(π),C(π)−c⋆},(10)m(\pi) = \min\{r^\star- R(\pi), C(\pi) - c^\star\}, \tag*{(10)}

and

m^n(π)=min⁡{r⋆−R^n(π),C^n(π)−c⋆}.(11)\widehat{m}_n(\pi) = \min\{r^\star- \widehat{R}_n(\pi), \widehat{C}_n(\pi) - c^\star\}. \tag*{(11)}

Let m^n⋆(π)\widehat{m}^{\star}_n(\pi) denote the corresponding nonparametric-bootstrap quantity, and define

Dn⋆=max⁡π∈P{m^n(π)−m^n⋆(π)}.(12)D_n^\star= \max_{\pi\in\mathcal{P}}\{\widehat{m}_n(\pi) - \widehat{m}^{\star}_n(\pi)\}. \tag*{(12)}

Let q1−α,nmenuq^{\mathrm{menu}}_{1-\alpha,n} be the ideal conditional (1−α)(1-\alpha) bootstrap quantile of Dn⋆D_n^\star, or a Monte Carlo estimate whose number of replicates tends to infinity.

Theorem 1 (Fixed-menu simultaneous validity). Suppose that:

  1. P\mathcal{P} is finite and fixed independently of Z1,…,ZnZ_1,\ldots,Z_n;

  1. the observations are i.i.d., losses are bounded, and

inf⁡π∈PC(π)≥c0>0;(13)\inf_{\pi\in\mathcal{P}} C(\pi) \ge c_0 > 0; \tag*{(13)}
  1. for some η>0\eta> 0, every policy has a locally unique active contract component:

∣{r⋆−R(π)}−{C(π)−c⋆}∣≥η;(14)\left|\{r^\star- R(\pi)\} - \{C(\pi) - c^\star\}\right| \ge\eta; \tag*{(14)}
  1. the limiting distribution of the menu-level maximum is continuous at its (1−α)(1-\alpha) quantile.

Then

lim inf⁡n→∞Pr⁡{m(π)≥m^n(π)−q1−α,nmenu for every π∈P}≥1−α.(15)\liminf_{n\to\infty}\Pr\left\{m(\pi) \ge\widehat{m}_n(\pi) - q^{\mathrm{menu}}_{1-\alpha,n}\ \text{for every }\pi\in\mathcal{P}\right\} \ge1-\alpha. \tag*{(15)}

Proof. For a finite menu, the vector containing N^n(π)\widehat{N}_n(\pi) and C^n(π)\widehat{C}_n(\pi) for all π∈P\pi\in\mathcal{P} satisfies a multivariate central limit theorem, and the ordinary nonparametric bootstrap consistently estimates its joint limiting distribution. Because coverage is bounded away from zero, the map

(N,C)↦NC(N,C) \mapsto\frac{N}{C}

is smooth in a neighborhood of the population values. The active-component separation condition makes the minimum defining m(π)m(\pi) locally equal to one smooth branch.

Write m^n=(m^n(π))π∈P\widehat{\boldsymbol{m}}_n = (\widehat{m}_n(\pi))_{\pi\in\mathcal{P}} and m=(m(π))π∈P\boldsymbol{m} = (m(\pi))_{\pi\in\mathcal{P}}. The multivariate bootstrap delta method therefore yields, jointly over the finite menu,

n(m^n−m)⇝G,(16)\sqrt{n}(\widehat{\boldsymbol{m}}_n-\boldsymbol{m})\rightsquigarrow\boldsymbol{G}, \tag*{(16)}

and, conditionally on the audit sample,

n(m^n∗−m^n)⇝G,(17)\sqrt{n}(\widehat{\boldsymbol{m}}_n^*-\widehat{\boldsymbol{m}}_n)\rightsquigarrow\boldsymbol{G}, \tag*{(17)}

where G\boldsymbol{G} is a centered Gaussian vector [25]. The sign of the correction matters: the one-sided lower bound requires the upper quantile of max⁡π∈P(m^n(π)−m^n∗(π))\max_{\pi\in\mathcal{P}}(\widehat{m}_n(\pi)-\widehat{m}_n^*(\pi)). Since −G-\boldsymbol{G} has the same distribution as G\boldsymbol{G} for a centered Gaussian vector, the conditional distribution of

nmax⁡π∈P{m^n(π)−m^n∗(π)}(18)\sqrt{n}\max_{\pi\in\mathcal{P}}\{\widehat{m}_n(\pi)-\widehat{m}_n^*(\pi)\} \tag*{(18)}

consistently estimates the distribution of

nmax⁡π∈P{m^n(π)−m(π)}.(19)\sqrt{n}\max_{\pi\in\mathcal{P}}\{\widehat{m}_n(\pi)-m(\pi)\}. \tag*{(19)}

Quantile consistency and continuity at the target quantile give the stated simultaneous lower-band result.

Corollary 2 (Data-dependent commitment). Let π^\widehat{\pi} be any data-dependent selector satisfying

π^∈BS(δ)\widehat{\pi}\in\mathcal{B}_{S}(\delta)

whenever that set is nonempty. Under the conditions of Theorem 1,

lim inf⁡n→∞Pr⁡{BS(δ)=∅ or m(π^)≥δ}≥1−α.(20)\liminf_{n\to\infty}\Pr\left\{\mathcal{B}_{S}(\delta)=\varnothing\ \text{or}\ m(\widehat{\pi})\ge\delta\right\}\ge1-\alpha. \tag*{(20)}

Proof. On the simultaneous event in Theorem 1, every member of BS(δ)\mathcal{B}_{S}(\delta) has population joint margin at least δ\delta. The conclusion therefore holds for any subsequent cost- or margin-based selector from that set.

Remark 3 (Calibration-derived menus). The theorem requires the menu and its policy parameters to be fixed independently of the audit sample. In the reported experiments, the policy menu and its parameters are frozen using construction/selection data before the independent commitment audit. The theorem therefore aligns with the sample chronology conditional on this construction. Its remaining distributional and regularity conditions, and the finite Monte Carlo approximation, continue to limit any general claim of certification.

Remark 4 (Tied active constraints). The separation condition excludes policies whose population risk and coverage margins are exactly tied. At such kink points, the minimum map is only directionally differentiable, and the ordinary bootstrap need not be consistent without an additional correction [11].

Remark 5 (Zero accepted observations and bootstrap centering). The positive population coverage requirement is asymptotic. A finite audit or bootstrap replicate may nevertheless accept no examples; the frozen implementation assigns an infeasible/undefined-risk outcome rather than fabricating a zero risk. Under a fixed positive coverage floor, the probability of such a replicate vanishes as the audit sample grows. The theorem concerns a centered common-resample maximum over the fixed menu; separate per-policy marginal quantiles would not imply the stated selection corollary.

Remark 6 (Finite bootstrap budget). The theorem concerns the ideal conditional bootstrap quantile, or a Monte Carlo approximation with an increasing replicate budget. The canonical reported implementation uses B=1000B = 1000 replicates. The frozen B=100B = 100 comparison assesses numerical stabilization, not finite-sample validity.

Complete RouterBench Operating Results

Full-grid dataset-equal deployment, unconditional deploy-and-fail, and conditional failure are given in Table 5. The table retains all frozen fixed-margin operating points, rather than selecting the most favorable test outcome. Figure 4 shows six-contract heterogeneity for representative procedures. Table 6 and Table 7 separate the two cost objectives. Serving costs are not unconditional system costs.

ProcedureDeployD&FFail|D
Audit winner91.8229.6332.22
Independent gate63.0012.3821.08
Fixed .0091.8222.2524.40
Fixed .0289.8914.0415.86
Fixed .0587.726.067.25
Fixed .0886.932.663.32
Fixed .1086.321.431.78
Fixed .1582.740.290.38
MARC-P .0585.790.630.73
MARC-S .0271.930.130.21
MARC-S .0568.030.000 obs.
LTT-Holm78.380.090.12
Pareto84.410.490.60

Table 5. Full-grid RouterBench results (dataset-equal percent; α=.10\alpha= .10 when applicable).

Figure 4. RouterBench operating rates across the six prespecified risk–coverage contracts.

ProcedureDeploy (%)D&F (%)Fail|D (%)Selected cost (trace units/query)
Audit winner91.8229.5232.140.000438
Independent gate62.9812.4321.280.000328
Fixed .0091.8229.1731.780.000448
Fixed .0289.8918.3320.460.000431
Fixed .0587.727.949.360.000592
Fixed .0886.933.334.080.000808
Fixed .1086.321.732.120.001014
Fixed .1582.740.370.470.001455
MARC-P .0087.267.068.360.000549
MARC-P .0286.712.923.460.000734
MARC-P .0585.790.770.890.001071
MARC-S .0075.480.370.470.001124
MARC-S .0271.930.150.240.001125
MARC-S .0568.030.000.000.001130
LTT-Bonferroni78.380.090.120.001392
LTT-Holm78.380.090.120.001391
Pareto Testing84.410.530.660.001297

Table 6. RouterBench mean-cost optimization: full configuration grid (dataset-equal rates across six contracts and applicable score channels). Cost per incoming query, conditional on commitment.

ProcedureDeploy (%)D&F (%)Fail|D (%)Selected cost (trace units/query)
Audit winner91.8229.7432.300.000749
Independent gate63.0312.3220.910.000554
Fixed .0091.8215.3317.010.000833
Fixed .0289.899.7411.250.000763
Fixed .0587.724.195.140.000978
Fixed .0886.932.002.550.001238
Fixed .1086.321.121.430.001449
Fixed .1582.740.200.290.001910
MARC-P .0087.262.963.810.000853
MARC-P .0286.711.361.740.001118
MARC-P .0585.790.480.570.001511
MARC-S .0075.480.200.280.001552
MARC-S .0271.930.110.180.001516
MARC-S .0568.030.000.000.001483
LTT-Bonferroni78.380.090.120.001835
LTT-Holm78.380.090.120.001835
Pareto Testing84.410.460.540.001708

Table 7. RouterBench tail-cost optimization: full configuration grid (dataset-equal rates across six contracts and applicable score channels). Cost per incoming query, conditional on commitment.

Exact-Control Applicability

The exact-binomial rows use only the Bernoulli correctness channel. They are not comparable to all-channel entries in Table 1 as if the sample sizes were identical. Fractional/bounded correctness requires the appropriate bounded-loss tests rather than an exact binomial model. Table 8 compares all applicable methods on this common binary-outcome domain.

ProcedureDeploy (%)D&F (%)Fail|D (%)DeploymentsFailures
Audit winner88.6428.8633.812,021658
Fixed .0585.097.9410.161,940181
MARC-P .0583.070.791.041,89418
MARC-S .0268.820.180.501,5694
MARC-S .0564.690.000.001,4750
LTT-Bonf.75.260.090.131,7162
LTT-Holm75.260.090.131,7162
Pareto Testing81.580.570.951,86013
Exact-Bonf.77.720.090.121,7722
Exact-Holm77.720.090.121,7722

Table 8. Binary-score RouterBench comparison under the mean-cost objective (dataset-equal rates; pooled counts; nominal α=.10\alpha= .10).

Failure Magnitude and Dataset-Level Inference

Estimands and aggregation. For each dataset, unconditional deploy-and-fail is the fraction of decision opportunities committing a policy that violates at least one fresh contract constraint. Conditional failure is evaluated among committed opportunities, then the available dataset-level

conditional proportions are equally weighted. Accordingly, the product of the separately equal-weighted deployment and conditional-failure columns need not equal equal-weighted deploy-and-fail. When a method makes no commitments on a dataset, its conditional failure is missing, not zero. Each cluster bootstrap resamples datasets, rather than treating within-dataset seed/contract cells as independent. Intervals describe dataset heterogeneity, not a finite-sample bound on an unobserved policy’s true risk.

Severity and aggregation. Eq. (4) measures the maximum of positive risk excess and coverage shortfall, assigning zero to nonviolating commitments. Incidence-adjusted severity averages over the stated evaluation population; conditional severity averages over violations only. The latter may increase even when a commitment rule reduces failure exposure. Figure 5 displays severity distributions, and Table 9 separates risk-only, coverage-only, and joint failures. The RouterBench audit winner has conditional mean, median, and p90 severity of 3.15, 1.94, and 7.39 percentage points, respectively. Table 10 reports a prespecified strict contract, and Table 11 presents dataset-level operating rates. Dataset-cluster uncertainty (Table 12) and leave-one-dataset-out sensitivity (Table 13) respect the 19 task-family reporting units.

Figure 5. Distribution of held-out contract-violation severity among failed commitments.

ProcedureRisk onlyCov. onlyBothD&F
Audit winner10.8316.642.1529.63
Independent gate2.859.220.3112.38
Fixed .053.422.520.126.06
MARC-P .050.540.090.000.63
MARC-S .020.110.020.000.13
LTT-Holm0.090.000.000.09
Pareto Testing0.270.220.000.49

Table 9. Mutually exclusive risk-only, coverage-only, and joint held-out contract violations, as a percentage of all RouterBench decision opportunities, dataset-equal. The components sum to deploy-and-fail before display rounding.

ProcedureDeployD&FFail/dep.
Audit winner81.3242.89652/1236
LTT-Holm50.790.000/772
MARC-P .0565.000.538/988
MARC-S .0239.470.000/600
Pareto63.160.264/960

Table 10. RouterBench strict (r⋆,c⋆r^\star, c^\star) = (.30, .60) contract. Dataset-equal rates and observed counts use distinct denominators.

DatasetWin DWin D&FP DP D&F
arc-challenge100.0030.00100.000.00
chinese_zodiac85.8342.5065.830.00
consensus_summary64.5829.1748.330.21
grade-school-math33.334.1733.330.00
hellaswag100.0020.00100.000.00
mbpp90.8347.5066.671.67
mmlu-elementary-mathematics90.0066.2565.000.83
mmlu-high-school-biology100.0030.42100.000.83
mmlu-high-school-macroeconomics100.0032.08100.002.50
mmlu-high-school-psychology100.0025.42100.003.33
mmlu-miscellaneous100.0017.50100.000.00
mmlu-moral-disputes100.0019.5891.670.83
mmlu-moral-scenarios100.0032.08100.000.00
mmlu-nutrition100.0030.83100.000.83
mmlu-philosophy100.0034.1793.330.00
mmlu-prehistory100.0022.08100.000.00
mmlu-professional-law80.0022.9266.670.83
mmlu-professional-psychology100.0026.6799.170.00
winogrande100.0029.58100.000.00

Table 11. Full-grid per-dataset deployment and deploy-and-fail percentages (winner versus MARC-P), each dataset weighted equally in primary aggregates.

ProcedureDeploy (%)D&F (%)Severity (pp)
Audit winner91.82 [83.46, 98.03]29.63 [24.34, 35.53]1.03 [0.70, 1.38]
MARC-P .0585.79 [75.92, 94.08]0.63 [0.26, 1.06]0.01 [0.01, 0.02]
MARC-S .0271.93 [62.89, 80.79]0.13 [0.02, 0.26]0.00 [0.00, 0.01]
MARC-S .0568.03 [58.55, 77.28]0.00 [0.00, 0.00]0.00 [0.00, 0.00]
LTT-Holm78.38 [67.85, 88.20]0.09 [0.00, 0.22]0.00 [0.00, 0.00]
Pareto Testing84.41 [74.85, 92.76]0.49 [0.16, 0.92]0.01 [0.00, 0.02]
Fixed .1086.32 [76.40, 94.78]1.43 [0.81, 2.11]0.04 [0.02, 0.06]

Table 12. RouterBench dataset-bootstrap 95% intervals over 19 task families. Severity is the maximum contract violation, with zero assigned to nonviolations.

Method vs winnerOutcomeLODO minLODO max
MARC-P .05D&F-30.38-26.98
MARC-P .05Deploy-6.37-4.98
Fixed .10D&F-29.54-26.39
Fixed .10Deploy-5.81-4.51
ParetoD&F-30.52-27.07
ParetoDeploy-7.82-6.34
LTT-HolmD&F-30.95-27.50
LTT-HolmDeploy-14.19-12.11
MARC-S .02D&F-30.90-27.45
MARC-S .02Deploy-21.00-18.59

Table 13. Leave-one-dataset-out ranges for paired method-minus-winner differences (percentage points; 19 omissions).

Dataset-level paired effects. Table 13 summarizes the already-frozen leave-one-dataset-out estimator for method-minus-winner effects. For example, MARC-P’s deploy-and-fail reduction relative to the winner remains between −30.38 and −26.98 percentage points across the 19 possible omissions; its corresponding deployment difference ranges from −6.37 to −4.98 points. These are ranges of descriptive benchmark-panel contrasts, not finite-sample confidence bounds or independent replicate tests. Intervals for failure-conditional quantities should be interpreted only when deployment denominators are positive.

Nested Audit-Size Robustness

Paired comparison and uncertainty. The independent audit grows through nested frozen query-ID hash prefixes, keeping the construction and test samples constant for all audit sizes within each seed. We contrast A40 with A10 on each task and split and aggregate the changes at the dataset level; marginal pointwise error bars alone do not characterize this paired change. Frozen outputs include per-dataset paired results, retained/deferred decisions, supported-set counts, and costs on common deployment opportunities. The reported audit-size comparison keeps buffer settings constant across audit sizes.

The pre-frozen robustness design fixes a 20% construction sample, 40% audit reservoir, and 40% common test. A deterministic query-ID hash gives A10⊂A20⊂A40A_{10} \subset A_{20} \subset A_{40}. At each audit size, the same frozen candidate menus and numerical buffers are used. Figure 6 displays the point-margin sweep and MARC-P; Table 14 lists all audit-size endpoints, and Table 15 and Table 16 report paired changes and dataset intervals. The test population is common within the audit-size experiment, but differs from the primary 20/20/60 RouterBench evaluation. Matched-common-deployment cost comparisons from the frozen paired ledger avoid conflating policy selection with changing deployment populations. The existing evidence establishes the direction of the MARC-P cost change but not an overall service optimum without accounting for fallback and deferral. The dataset-specific A10-to-A40 changes are heterogeneous; the average should not be read as a monotonic claim for every task.

Figure 6. Frozen audit-size results for MARC-P and fixed margins. Fixed .15 also gains deployment as audit evidence grows.

ProcedureD10D20D40DF10DF20DF40
MARC-P .0583.5585.8386.581.340.770.42
Fixed 0.0092.7692.1191.0723.6020.0917.60
Fixed 0.0291.4790.0088.7716.9612.599.86
Fixed 0.0589.5087.9487.0210.736.153.16
Fixed 0.0888.0087.1986.806.613.111.17
Fixed 0.1086.6286.2186.474.011.510.44
Fixed 0.1582.1383.0384.411.270.460.07

Table 14. Nested-audit deployment (DD) and deploy-and-fail (DFDF) rates at 10/20/40% audit; fixed method settings.

ProcedureΔ\DeltaDeploy (pp)ΔD&F\Delta D\&F (pp)Δ\DeltaSeverity (pp)Δ\Delta common costΔ\Delta supported-set size
MARC-P .05+3.03-0.92-0.030-0.000193+1.49
Fixed .05-2.48-7.58-0.356+0.000082-0.57
Fixed .10-0.15-3.57-0.147+0.000129-0.65
Fixed .15+2.28-1.21-0.044+0.000198+0.15

Table 15. Nested-audit paired A40–A10 changes (dataset-equal). Serving-cost differences use common deployments.

A40–A10Mean95% CI+ / − /0 tasks
Deployment (pp)+3.03[+1.01, +5.44]9/2/8
Deploy-and-fail (pp)-0.92[-1.67, -0.22]3/10/6
Supported-set size+1.49[+1.04, +1.95]17/2/0

Table 16. MARC-P .05 paired A40–A10 changes over 19 RouterBench datasets. Descriptive dataset-cluster percentile intervals use 20,000 bootstrap resamples of the paired results.

Learned Semantic-Router Evaluation

Learned policy construction. Query embeddings enter the CARROT-style nearest-neighbor quality estimator. Frozen model-cost preferences and acceptance thresholds give 48 routing policies per dataset. These policies are fitted and selected before the independent commitment audit; no

learned parameters are updated using audit or test outcomes.

The learned panel covers 10 datasets, 11 models, and 20 seeds with independent 40/20/20/20 fit/selection/audit/test splits. Query traces, embeddings, and split partitions were verified against reference records; deterministic replay and repeated commitment runs agreed. The evaluation isolates commitment behavior for an independently trained routing family.

Main-text Table 4 reports the full-grid and strict-contract comparisons on pooled decision opportunities with observed failure/deployment counts. Figure 7 shows the heterogeneity that pooled failure rates can hide; Table 17 and Table 18 give per-dataset operating rates, and Table 19 gives actual deployment/failure counts. Conditional failure is undefined when a dataset has zero deployments, rather than imputed to zero. At the strict contract, stronger formal and simultaneous methods observe no failures but often deploy in only about one fifth to one quarter of opportunities. Residual MARC-P failures need not be less severe conditional on failure even when total violation incidence drops.

Figure 7. Learned-router deployment and deploy-and-fail by dataset, using the frozen KNN menu.

DatasetWin DWin DFP DP DF
aime100.0034.1786.6711.67
arenahard100.0032.5090.000.00
gpqa100.0030.0097.503.33
hle24.1720.000.000.00
livecodebench100.0034.17100.001.67
livemathbench100.0025.0080.004.17
mmlupro100.0022.50100.000.00
simpleqa66.676.6766.670.00
swe-bench33.3325.000.000.00
tau295.0048.3369.175.00

Table 17. Learned-router deployment and unconditional deploy-and-fail by dataset (percent; pooled within task).

DatasetWinner DWinner D&FMARC-P DMARC-P D&FLTT DPareto D
aime100.034.286.711.718.348.3
arenahard100.032.590.00.068.377.5
gpqa100.030.097.53.370.073.3
hle24.220.00.00.00.00.8
livecodebench100.034.2100.01.7100.0100.0
livemathbench100.025.080.04.244.260.0
mmlupro100.022.5100.00.0100.0100.0
simpleqa66.76.766.70.066.766.7
swe-bench33.325.00.00.00.00.0
tau295.048.369.25.045.855.8

Table 18. Learned-router deployment (DD) and deploy-and-fail (D&F) by dataset, averaged over 20 seeds and six fixed contracts (percent).

Task (120 opportunities each)WinnerMARC-P .05MARC-S .02LTT-HolmPareto
aime120/41104/1438/222/058/3
arenahard120/39108/098/082/093/0
gpqa120/36117/487/184/088/1
hle29/240/01/10/01/1
livecodebench120/41120/2120/2120/2120/1
livemathbench120/3096/564/053/072/0
mmlupro120/27120/0120/0120/0120/0
simpleqa80/880/080/080/080/0
swe-bench40/300/00/00/00/0
tau2114/5883/666/055/067/0

Table 19. Learned-router deployment/failure counts by dataset, over 20 seeds and six fixed contracts (mean-cost objective). A 0/0 entry indicates no deployment.

Serving-Cost Objectives and Sensitivity

Three economic populations. (i) Policy serving-cost J(π)J(\pi) is computed over incoming queries, including abstention at the frozen zero policy-level cost; (ii) average JJ among committed decision opportunities summarizes the policy that a commitment rule selects, and is sensitive to which opportunities each method chooses to serve; (iii) a whole-system objective additionally requires an application-dependent cost for never deploying a policy, fallback, or delay. The frozen trace identifies the first two but not the third. Matched-deployment comparisons condition on opportunities common to a pair of rules or to two audit-size treatments and are presented as comparators, not complete system valuations. The mean-cost and tail-cost optimization objectives remain separate throughout.

The experiment has separate mean_cost and tail_cost optimization objectives. Summarizing selected policy cost over a mixture of these objectives is not a homogeneous monetary estimand. Figure 2(b) therefore uses an explicitly mean-cost-only RouterBench comparison; Table 20 separates objectives. Costs are conditional on deployment, so methods choosing different cells cannot be ranked by complete end-to-end operational expense without a specified deferral/fallback price. Table 21 and Table 22 report matched and objective-specific cost uncertainty; Table 23 gives the frozen illustrative penalty scenarios. These do not identify an overall optimum without real deferral and fallback costs.

ProcedureMean objectiveTail objective
Audit winner0.0004380.000749
Fixed .050.0005920.000978
Fixed .100.0010140.001449
Fixed .150.0014550.001910
MARC-P .050.0010710.001511
MARC-S .020.0011250.001516
MARC-S .050.0011300.001483
Pareto Testing0.0012970.001708
LTT-Holm0.0013910.001835

Table 20. RouterBench selected-policy cost by optimization objective (trace units per incoming query; dataset-equal, conditional on commitment).

ObjectiveComparison (MARC-P .05 vs.)Shared NNΔJ\Delta J95% cluster interval
MeanWinner3912+0.000844[+0.000408, +0.001444]
MeanFixed .103912+0.000070[-0.000019, +0.000159]
MeanPareto3841-0.000243[-0.000429, -0.000069]
MeanLTT-Holm3574-0.000517[-0.000770, -0.000295]
TailWinner3912+0.001105[+0.000495, +0.002032]
TailFixed .103912+0.000082[-0.000016, +0.000187]
TailPareto3841-0.000224[-0.000461, -0.000009]
TailLTT-Holm3574-0.000571[-0.000903, -0.000314]

Table 21. Matched RouterBench serving-cost differences for MARC-P versus each comparator on their jointly deployed opportunities. Dataset-equal means and 95% dataset-bootstrap intervals; positive ΔJ\Delta J indicates higher MARC-P cost. Comparator rows use distinct matched populations.

ObjectiveMethodCost95% dataset interval
MeanWinner0.000438[0.000185, 0.000752]
MeanMARC-P .050.001071[0.000563, 0.001750]
MeanPareto0.001297[0.000781, 0.001968]
MeanLTT-Holm0.001391[0.000774, 0.002177]
TailWinner0.000749[0.000319, 0.001283]
TailMARC-P .050.001511[0.000749, 0.002620]
TailPareto0.001708[0.000971, 0.002695]
TailLTT-Holm0.001835[0.000993, 0.003000]

Table 22. Selected-policy serving cost by optimization objective, conditional on deployment. Dataset-equal means and descriptive 95% bootstrap intervals over 19 tasks.

Failure penaltyDefer penaltyWinnerMARC-P .05ParetoLTT-Holm
0.0010.00010.0006960.0008920.0010830.001094
0.0010.010.0015060.0022990.0026260.003235
0.10.00010.0299180.0016520.0016040.001181
0.10.010.0307280.0030590.0031470.003322

Table 23. Illustrative mean-cost decision penalties over the fixed 4 × 4 failure/deferral grid. The scenario objective is E[DJtrace+λF1{D∩F}+λD(1−D)]\mathbb{E}[D J_{\mathrm{trace}} + \lambda_F 1\{D \cap F\} + \lambda_D(1-D)]; penalty values are hypothetical.

What matched comparisons identify. In Table 21, a method pair contributes only decisions on which both rules deploy. This isolates the serving-cost difference between their selected policies on the same fixed opportunities; it does not estimate the cost of deferring other opportunities.

Objective-separated 95% dataset-bootstrap intervals are descriptive and not extrapolations to every future traffic distribution. The frozen scenario table reports a transparent linear penalty functional on the mean-cost objective using only the existing (λF,λD)(\lambda_F,\lambda_D) grid, and is not a business-calibrated monetary return or a new parameter search. Scenarios cannot justify a universal optimum without a real operator’s failure, fallback and lost-service values.

Operating-Point and Monte Carlo Sensitivity

Table 5 includes the full prespecified MARC buffer and fixed-margin sweeps. RouterBench MARC-P and MARC-S were recomputed with B=1000B=1000 using the verified fixed implementation. Relative to B=100B=100, MARC-P .05 deployment moved 85.44% to 85.79%; MARC-S .02 moved 71.69% to 71.93%, and MARC-S .05 moved 67.83% to 68.03%. The latter still had zero observed failures. These changes reflect Monte Carlo stabilization at fixed operating settings. Table 24 additionally reports the existing formal-control α=.05\alpha=.05 sensitivity; MARC settings remain unchanged.

Procedureα\alphaDeployD&F
LTT-Holm0.1078.380.09
LTT-Bonf.0.1078.380.09
Pareto0.1084.410.49
LTT-Holm0.0576.750.00
LTT-Bonf.0.0576.750.00
Pareto0.0582.830.37

Table 24. RouterBench formal-control sensitivity (dataset-equal percent) at nominal α∈{.10,.05}\alpha\in\{.10,.05\}.

External Menu-Size Sensitivity

The retain/replace/defer comparison is reported in main-text Table 3. Table 25 provides a complementary xRouteBench menu-size trajectory: deployment is constant while observed held-out failure decreases as the menu grows. Together with the main RouterBench findings, this highlights the role of policy composition, feasibility margins, and cost-driven selection rather than menu cardinality alone.

Menu sizeDeployFail|DD&F
766.679.626.42
1466.677.254.83
2166.675.253.50
2866.672.501.67
3566.670.000.00

Table 25. xRouteBench held-out menu-size sensitivity (percent). Its policy menu and contract differ from the main RouterBench study.

Evaluation scope. Independent-audit results exclude earlier same-split diagnostics. xRouteBench uses a separate frozen policy menu and contract. In every study, numerical service targets are fixed before auditing and held through selection and test evaluation.

References

  1. [1]Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/f3549ef9b5ff520a7e41ff3cc306ab2b-Abstract-Conference.html.
  2. [2]Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Candès, Michael I. Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. The Annals of Applied Statistics, 19(2):1641–1662, 2025. doi: 10.1214/24-AOAS1998.DOI
  3. [3]Vidmantas Bentkus. On hoeffding’s inequalities. The Annals of Probability, 32(2):1650–1673, 2004. doi: 10.1214/009117904000000360.DOI
  4. [4]Richard Berk, Lawrence Brown, Andreas Buja, Kai Zhang, and Linda Zhao. Valid post-selection inference. The Annals of Statistics, 41(2):802–837, 2013. doi: 10.1214/12-AOS1077. URL https://projecteuclid.org/journals/annals-of-statistics/volume-41/issue-2/Valid-post-selection-inference/10.1214/12-AOS1077.full.DOI
  5. [5]Gavin C. Cawley and Nicola L. C. Talbot. On over-fitting in model selection and subsequent selection bias in performance evaluation. Journal of Machine Learning Research, 11(70):2079–2107, 2010. URL https://jmlr.org/papers/v11/cawley10a.html.DOI
  6. [6]Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=cSimKw5p6R.arxiv.org/abs/2305.05176
  7. [7]C. K. Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 16(1):41–46, 1970. doi: 10.1109/TIT.1970.1054406. URL https://ieeexplore.ieee.org/document/1054406.DOI
  8. [8]Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Rühle, Laks V. S. Lakshmanan, and Ahmed Hassan Awadallah. Hybrid LLM: Cost-efficient and quality-aware query routing. In The Twelfth International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/b47d93c991a22ac0b377578af0a1f63a-Abstract-Conference.html.arxiv.org/abs/2404.14618
  9. [9]Cynthia Dwork, Vitaly Feldman, Moritz Hardt, T oniann Pitassi, Omer Reingold, and Aaron Roth. Preserving statistical validity in adaptive data analysis. In Proceedings of the Forty-Seventh Annual ACM Symposium on Theory of Computing, pages 117–126. Association for Computing Machinery, 2015. doi: 10.1145/2746539.2746580. URL https://dl.acm.org/doi/10.1145/2746539.2746580.DOI
  10. [10]Ran El-Yaniv and Yair Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(53):1605–1641, 2010. URL https://jmlr.org/papers/v11/el-yaniv10a.html.DOI
  11. [11]Zheng Fang and Andres Santos. Inference on directionally differentiable functions. The Review of Economic Studies, 86(1):377–412, 2019. doi: 10.1093/restud/rdy049.DOI
  12. [12]Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems, volume 30, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/4a8423d5e91fda00bb7e46540e2b0cf1-Abstract.html.arxiv.org/abs/1705.08500
  13. [13]Sai Hao, Hao Zeng, Hongxin Wei, and Bingyi Jing. RACER: Risk-aware calibrated efficient routing for large language models. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 40499–40524, 2026. URL https://proceedings.mlr.press/v306/hao26c.html.arxiv.org/abs/2603.06616
  14. [14]Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70, 1979. URL https://www.jstor.org/stable/4615733.DOI
  15. [15]Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyansh Kaushubh Upadhyay. RouterBench: A benchmark for multi-LLM routing system. In Agentic Markets Workshop at the 41st International Conference on Machine Learning, 2024. URL https://icml.cc/virtual/2024/39041.arxiv.org/abs/2403.12031
  16. [16]Zhongzhan Huang, Guoming Ling, Yupei Lin, Yandong Chen, Shanshan Zhong, Hefeng Wu, and Liang Lin. RouterEval: A comprehensive benchmark for routing LLMs to explore model-level scaling up in LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 3860–3887, Suzhou, China, November 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.findings-emnlp.208. URL https://aclanthology.org/2025.findings-emnlp.208/.DOI
  17. [17]Bracha Laufer-Goldstein, Adam Fisch, Regina Barzilay, and Tommi Jaakkola. Efficiently controlling multiple risks with pareto testing. In International Conference on Learning Representations, 2023.
  18. [18]Hao Li, Yiqun Zhang, Zhaoyan Guo, Chenxu Wang, Shengji Tang, Qiaosheng Zhang, Yang Chen, Biqing Qi, Peng Ye, Lei Bai, Zhen Wang, and Shuyue Hu. LLMRouterBench: A massive benchmark and unified framework for LLM routing. In Findings of the Association for Computational Linguistics: ACL 2026, pages 37733–37754, San Diego, California, United States, July 2026. Association for Computational Linguistics. doi: 10.18653/v1/2026.findings-acl.1881. URL https://aclanthology.org/2026.findings-acl.1881/.DOI
  19. [19]Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs from preference data. In The Thirteenth International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/5503a7c69d48a2f86fc00b3dc09de686-Abstract-Conference.html.arxiv.org/abs/2406.18665
  20. [20]Ibne Farabi Shihab, Abu Sa-Adat Mohamed Moon-Im Al Ahsan, and Md Najmus Swaqeeb. Opportunity is not realizability: Selection-valid diagnostics for multi-llm routing, 2026. URL https://arxiv.org/abs/2608.08265.
  21. [21]Seamus Somerstep, Felipe Maia Polo, Allysson Flavio Melo de Oliveira, Prattyush Mangal, Mírian Silva, Onkar Bhardwaj, Mikhail Yurochkin, and Subha Maity. CARROT: A cost aware rate optimal router. In The 2nd Workshop on Foundation Models in the Wild at ICLR 2025, 2025. URL https://iclr.cc/virtual/2025/10000657.arxiv.org/abs/2502.03261
  22. [22]Jiayuan Su, Fulin Lin, Zhaopeng Feng, Han Zheng, Teng Wang, Zhenyu Xiao, Xinlong Zhao, Zuozhu Liu, Lu Cheng, and Hongwei Wang. CP-Router: An uncertainty-aware router between LLM and LRM. Proceedings of the AAAI Conference on Artificial Intelligence, 40(39):33065–33073, 2026. doi: 10.1609/aaai.v40i39.40589.DOI
  23. [23]Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High confidence policy improvement. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2380–2388, 2015. URL https://proceedings.mlr.press/v37/thomas15.html.
  24. [24]Iqtেদar Uddin and André Bauer. Conformal LLM routing with distribution-free safety guarantees. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 791–799, San Diego, California, United States, July 2026. Association for Computational Linguistics. doi: 10.18653/v1/2026.acl-srw.70. URL https://aclanthology.org/2026.acl-srw.70/.DOI
  25. [25]Aad W. van der Vaart and Jon A. Wellner. Weak Convergence and Empirical Processes: With Applications to Statistics. Springer, New York, 1996. doi: 10.1007/978-1-4757-2545-2.DOI

Paper details

vlnc:W_426ur3k3
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents