betaFeedback

WorksComputer Science

Circuits of LLM Classifiers

Abstract

Language models make it easy to build classifiers by simply prompting a model with the sample to classify. However, their predictions are neither deterministic nor uniformly reliable across samples. How can we combine predictions from different models to improve reliability? We view LLM classifiers as noisy measurement instruments of an underlying target label. Our goal is to maximize average classification accuracy at a given average query cost. To this end, we introduce circuits that organize LLM classifiers into an adaptive sequence of queries and decisions. We then characterize the optimal circuit, obtain the fundamental limits of achievable accuracy, and provide practical insights on realizing the optimal circuit. Across seven classification tasks spanning safety, health, political science, human preference, agent evaluation, decision-making and reasoning, our method improves the cost-accuracy frontier over routing and cascading baselines.

Introduction

Large language models (LLMs) are increasingly used in prediction and classification tasks as (auto-) raters (Vu et al., 2024), verifiers (Cobbe et al., [16]; Kwok et al., [42]), judges (Zheng et al., [79]; Chiang et al., [15]), and more. A main draw of LLMs as classifiers is their ease of implementation and deployment (Brown et al., [11]; Liu et al., [45]). By simply specifying a new classification task in natural language, we can prompt these systems to assign labels in a range of applications such as content moderation (OpenAI et al., [55]), scientific review (Liu & Shah, 2023), guardrailing (Inan et al., [35]), and qualitative data analysis in social sciences (Halterman & Keith, [28]). Here, a “classifier” is defined by a combination of backend model choice, instruction prompt, grading rubric, decoding parameters (e.g., temperature, chain-of-thought), and more. “Fitting” a classifier then boils down to navigating these choices during inference.

Consider first the task of designing a single LLM classifier. Even after fixing the model and instructions, repeated evaluations of the same prompt can return conflicting labels (Haldar & Hockenmaier, [27]; Wang et al., [73])1. It is therefore useful to view an LLM classifier as a noisy measurement instrument (Brennan, [9]; Kahneman et al., [38]). Naturally, if an instrument is noisy, we should collect more than one measurement with our tool, or measure with multiple (independent) instruments to control error. When using LLMs to classify a given prompt, this intuition suggests we can improve classification accuracy by collecting predictions from different models and instruction configurations. Of course, querying multiple models incurs a cost (e.g., API calls add up). A key challenge is deciding which classifier, if any, to query next as we collect measurements. These decisions create a tradeoff between accuracy and expected inference cost. In this paper, we ask:

How should we combine LLM classifiers to optimize the cost-accuracy trade-off in a given task?

Strategies for combining classifiers, such as bagging [8] and boosting [57], are familiar in machine learning. More recently, this question has received growing attention in work on LLM routing [21, 54], cascading [13, 37, 7], and multi-judge evaluation [67, 66]. These methods show that combining multiple LLM classifiers outperforms any single model.

We refer to any procedure that queries one or more LLM classifiers and combines their outputs into a final decision as a classification circuit, or a circuit for short. The name reflects how these strategies use LLMs. The classifiers act as components, while decision rules determine how their outputs are passed forward, combined, or used to select the next query. Changing either the components or how they are connected produces a different circuit. For example, routing queries a single selected model, cascades query models sequentially in a fixed order, and voting schemes like majority voting query several models in parallel and aggregate their outputs. Different circuits using the same set of classifiers can therefore have different accuracy and cost, as shown in Figure 1.

Figure 1. Classification circuits and their cost–accuracy tradeoffs. Majority voting, routing, and cascades are examples of circuits, differing in how they select and combine LLM classifiers. For any prompt, our method searches over a broad class of circuits and uses a dynamic program to determine which classifier to query next and when to stop. The right panel shows such circuits for the prompt zz: starting from its embedding, the circuit may predict immediately (as in A) or sequentially acquire additional LLM measurements as needed (as in B and C). Across prompts, these decisions produce the cost–accuracy frontier for the BeaverTails safety dataset.

Given the many ways of combining LLM classifiers, which circuit is best? What is the “optimal” circuit when samples come from a known distribution, such as a large safety classification dataset? What properties should we promote across classifiers to improve accuracy, such as independence or decorrelated errors? And what is the best achievable tradeoff between cost and accuracy?

We show that existing ways of combining LLM classifiers are facets of a common decision-theoretic formulation. This formulation maximizes average classification accuracy while minimizing the average cost, and is essentially a Bayesian sequential decision problem [71, 18], closely related to sequential hypothesis testing [14, 52]. The fundamental challenge is to determine the value of an additional query given the observed measurements. For this formulation, an optimal circuit is given by the solution to a dynamic program.

Analyzing this dynamic program offers several insights into combining LLM classifiers. A first insight is that a query is valuable not only for its label, but also for what it reveals about how hard the sample is to classify. When classifiers disagree, the circuit learns to spend more queries on difficult prompts while resolving easier ones quickly. A second insight concerns how accuracy improves with additional query cost. For a fixed sample, asymptotically, the optimal error decreases with increasing cost at a rate determined by the most informative classifier per unit cost. A final, more obvious insight is that the error of any circuit lies between two limits: the Bayes error before any query is made and the Bayes error given the outputs of all classifiers. The latter is the accuracy ceiling for any way of combining a given set of classifiers.

Solving for this dynamic program requires a joint distribution between the ground truth label and instruments for a prompt. In practice, however, the joint distribution is unknown. We show that this optimal circuit can nevertheless be approximated in two steps. First, we fit a single probabilistic model of the joint distribution of the label and classifier outputs using a labeled training set. Then, we solve the dynamic program with the learned model, which results in a prompt-dependent policy that determines which LLM to query and when to stop. In a range of classification tasks, this two-step approach Pareto-dominates routing and cascading baselines in the cost–accuracy tradeoff.

Contributions. Our contributions are as follows:

  • Optimal classification circuits. We introduce classification circuits as a general framework for combining LLM classifiers. Under a cost–accuracy tradeoff, we show that the optimal circuit for a given prompt can be solved exactly through a dynamic program.

  • Properties of classification circuits. We identify the accuracy ceiling for a fixed pool of classifiers, quantify the value of adding complementary classifiers, and show how the distribution of prompt difficulty determines how average accuracy scales with cost.

  • Learning circuits from data. We show how to approximate the optimal circuit from a labeled training set by learning the distributions required by the dynamic program from prompt embeddings and classifier outputs. Across seven classification tasks, circuits improve the cost–accuracy frontier over routing and cascading baselines.

Problem Formulation

Let Z∼PZZ \sim\mathcal{P}_{Z} denote a random variable representing a classification prompt drawn from a prompt space Z\mathcal{Z}. In our applications, a prompt consists of the text to be classified together with task-specific information defining the classification problem. Each prompt zz has a binary target label y(z)∈{0,1}y(z) \in\{0,1\}. To predict y(z)y(z) for a realized prompt Z=zZ = z, we query a subset of measurement instruments from a pool of available instruments denoted by H\mathcal{H}.

Definition 1 (Measurement instrument). A measurement instrument, indexed by i∈Hi \in\mathcal{H}, is characterized by a conditional distribution QX∣Z[i]Q^{[i]}_{X \mid Z} that takes a prompt zz as input and outputs a measurement X(i)∈XiX^{(i)} \in\mathcal{X}_i. For a prompt zz, instrument ii has a query cost ci(z)>0c_i(z) > 0.3

An instrument can correspond to a particular configuration of an LLM. For example, two instruments may differ in the underlying model, instruction prompt or rubric, reasoning configuration, or inference harness. Each such configuration defines a distinct distribution of the form in Definition 1.

We next study how to compose LLMs as measurement instruments into classification circuits that improve prediction accuracy. Informally, a classification circuit for a prompt zz specifies which instruments to query, how subsequent queries depend on the measurements obtained so far, and when to stop before producing a final prediction of y(z)y(z). Because this circuit may vary with the prompt, we use π\pi to denote the circuit policy and π(z)\pi(z) for the circuit instantiated on zz. A few example circuits are given in Figure 2.

Figure 2. Illustration of a few known circuits within the framework of Definition 2.

Definition 2 (Classification circuit). Fix a finite instrument pool H\mathcal{H}, where querying instrument i∈Hi \in\mathcal{H} returns an outcome in Xi\mathcal{X}_i. For t≥0t \ge0, let

Tt={((A1,x1),…,(At,xt)):∅≠Ar⊆H, xr∈{0,1}Ar, Ar∩As=∅ for r≠s}\mathcal{T}_t = \left\{ \left((A_1,x_1),\ldots,(A_t,x_t)\right) : \varnothing\ne A_r \subseteq\mathcal{H},\ x_r \in\{0,1\}^{A_r},\ A_r \cap A_s = \varnothing\ \text{for } r \ne s \right\}

denote the set of all histories of length tt. Here ArA_r is the set of instruments queried in round rr, xrx_r records their responses, and the disjointness constraint encodes that no item in H\mathcal{H} is queried twice. We set T0={∅}\mathcal{T}_0 = \{\varnothing\} and write A(ht)=⋃r≤tArA(h_t) = \bigcup_{r \le t} A_r for the instruments already queried in ht∈Tth_t \in\mathcal{T}_t.

A classification circuit is a pair π=(πquery,πpredict)\pi= (\pi_{\mathrm{query}}, \pi_{\mathrm{predict}}). The query policy πquery\pi_{\mathrm{query}} is a family {πquery,t}t≥0\{\pi_{\mathrm{query},t}\}_{t \ge0} of maps

πquery,t:Z×Tt⟶2H∖A(ht),At+1=πquery,t(z,ht),\pi_{\mathrm{query},t} : \mathcal{Z} \times\mathcal{T}_t \longrightarrow2^{\mathcal{H}\setminus A(h_t)}, \qquad A_{t+1} = \pi_{\mathrm{query},t}(z,h_t),

where 2H∖A(ht)2^{\mathcal{H}\setminus A(h_t)} is the power set of the instruments left unqueried in hth_t. Thus the query policy determines, for a given prompt zz, which instruments are queried in each round: starting from h0=∅h_0 = \varnothing, the process terminates if At+1=∅A_{t+1} = \varnothing, and otherwise queries the instruments in At+1A_{t+1}, observes xt+1∈{0,1}At+1x_{t+1} \in\{0,1\}^{A_{t+1}}, and appends (At+1,xt+1)(A_{t+1},x_{t+1}) to the history hth_t.

Call h=((A1,x1),…,(Aτ,xτ))∈Tτh = ((A_1,x_1),\ldots,(A_\tau,x_\tau)) \in\mathcal{T}_\tau terminal for zz if Ar+1=πquery,r(z,hr)A_{r+1} = \pi_{\mathrm{query},r}(z,h_r) for every 0≤r<τ0 \le r < \tau, where hrh_r is the length-rr prefix of hh, and πquery,τ(z,h)=∅\pi_{\mathrm{query},\tau}(z,h) = \varnothing. Let T⋆(z)\mathcal{T}^\star(z) be the set of such histories considering all possible 0≤τ≤∣H∣0 \le\tau\le|\mathcal{H}|. The decision rule is a map

πpredict:Z×T⋆(z)⟶{0,1},Y^π(z)=πpredict(z,h).\pi_{\mathrm{predict}} : \mathcal{Z} \times\mathcal{T}^\star(z) \longrightarrow\{0,1\}, \qquad\widehat{Y}_\pi(z) = \pi_{\mathrm{predict}}(z,h).

Remark 1. Definition 2 permits repeated independent queries to the same instrument. In particular, items i,j∈Hi,j \in\mathcal{H} may correspond to two independent queries of the same instrument, with XI∣Z[i]=XI∣Z[j]X^{[i]}_{I\mid Z} = X^{[j]}_{I\mid Z} and ci(z)=cj(z)c_i(z) = c_j(z). We assume that ∣H∣<∞|\mathcal{H}| < \infty, so only finitely many queries can be included.

We focus on the binary setting throughout this work, i.e., Xi={0,1}\mathcal{X}_i = \{0,1\} for all i∈Hi \in\mathcal{H}. For a fixed prompt Z=zZ = z, executing a circuit π(z)\pi(z) in Definition 2 produces the collected measurements h=(Ai,xi)i≤τh = (A_i,x_i)_{i \le\tau} for some τ\tau. Let A=⋃i≤τAi⊆HA = \bigcup_{i \le\tau} A_i \subseteq\mathcal{H} denote the subset of instruments queried by π(z)\pi(z) and let X∈{0,1}∣A∣X \in\{0,1\}^{|A|} be the corresponding measurements. Then, the circuit’s prediction on the target label is given by Y^π(z)=πpredict(z,X)\widehat{Y}_\pi(z) = \pi_{\mathrm{predict}}(z,X). Over the prompt distribution, the probability of error of a circuit policy π\pi is

E(π)=Pr⁡(Y^π(Z)≠y(Z))=EZ[EX∣Z[1{Y^π(Z)≠y(Z)}∣Z]].(1)\mathbb{E}(\pi) = \Pr\left(\widehat{Y}_\pi(Z) \ne y(Z)\right) = \mathbb{E}_Z\left[\mathbb{E}_{X\mid Z}\left[\mathbf{1}_{\{\widehat{Y}_\pi(Z)\ne y(Z)\}}\mid Z\right]\right]. \tag*{(1)}

and the expected query cost is given by

C(π)=EZ[EX∣Z[∑i∈Aci(Z)∣Z]](2)C(\pi) = \mathbb{E}_{Z}\left[\mathbb{E}_{X\mid Z}\left[\sum_{i\in A} c_i(Z)\mid Z\right]\right] \tag*{(2)}

where the inner expectation is over the instrument measurements, conditional on a fixed prompt.

Since each LLM query uses fresh randomness, we model measurements as independent conditional on a prompt.

Assumption 1 (Conditional independence of measurements). Conditional on Z=zZ=z, the measurements {X(i)}i∈H\{X^{(i)}\}_{i\in\mathcal{H}} are jointly independent. Thus, for any (xi)i∈H(x_i)_{i\in\mathcal{H}},

Pr⁡(X(i)=xi, ∀i∈H∣Z=z)=∏i∈HQX∣Z[i](xi∣z).\Pr\left(X^{(i)}=x_i,\ \forall i\in\mathcal{H}\mid Z=z\right)=\prod_{i\in\mathcal{H}}Q^{[i]}_{X\mid Z}(x_i\mid z).

At first glance, this may seem to rule out correlated errors observed across language models [39]. The distinction is that independence holds within a prompt, whereas error correlation is measured across prompts. We also validate this empirically in Figure 7.

Figure 7. Instrument answers are independent within a prompt but correlated across prompts. (a) ϕ\phi coefficient of two instruments’ answers within a prompt, averaged over prompts on which both instruments’ answers vary. (b) The same coefficient across prompts, pooling all calls. (c) Total correlation of the eight answers on each prompt (Equation (43)) against the 95th percentile of its permutation null. Red marks a rejection at the 5% level. Within a prompt the coefficients lie between −0.10-0.10 and 0.150.15, while across prompts they reach 0.690.69. Only one of the 20 prompts rejects joint independence, the rate expected by chance.

Optimizing Classification Circuits

Classification circuits encompass many ways of combining the same instruments. We study one particular criterion for choosing among them, the expected cost–accuracy tradeoff, and show how to construct the resulting circuits for any prompt. We present all proofs in Section B.

Our goal is to find a circuit policy π\pi that best trades off classification error against query cost. We control this tradeoff with a parameter λ≥0\lambda\ge0 that weights cost relative to error, and define the objective Lλ(π)=E(π)+λC(π)\mathcal{L}_{\lambda}(\pi) = E(\pi) + \lambda C(\pi). For a given λ\lambda, the optimal policy is

πλ⋆∈arg⁡min⁡π∈Π Lλ(π)(3)\pi^\star_{\lambda}\in\underset{\pi\in\Pi}{\arg\min}\ \mathcal{L}_{\lambda}(\pi) \tag*{(3)}

where Π\Pi denotes the set of all circuit policies described in Definition 2. We first show that the class Π\Pi can be restricted to a smaller class without loss of optimality with respect to this objective.

Proposition 1. Any circuit that queries multiple instruments simultaneously has a sequential counterpart that achieves the same probability of error in Equation (1) at the same cost in Equation (2). Hence an optimal circuit for Equation (3) can always be chosen to query one instrument at a time.

The optimal solution to Equation (3) can hence be viewed as a sequence of decisions. Among all possible sequential circuits, the dynamic program resulting from the Bellman equations [5] gives the optimal solution to Equation (3).

Optimal Circuit Setup

Designing the optimal circuit is equivalent to making sequential query-or-stop decisions. At each step, given the responses observed so far, the policy either selects the next model to query or stops and predicts. The resulting policy defines the optimal circuit. We first introduce the objects that appear in the dynamic program: 1) the state of a circuit, 2) the actions available in each state, and the transitions between states.

State. Fix a prompt zz. At any point during the execution of a circuit, the information gathered so far is the set of (instrument, response) pairs observed,

s={(j,x(j)):j∈M(s)},(4)s = \{(j, x^{(j)}) : j \in\mathcal{M}(s)\}, \tag*{(4)}

where M(s)⊆H\mathcal{M}(s) \subseteq\mathcal{H} is the set of instruments queried so far. We call ss the state, with SS denoting the set of all possible states. ∣S∣=3∣H∣|S| = 3^{|\mathcal{H}|} since we assume that each instrument indexed i∈Hi \in\mathcal{H} can only be queried once.4 s0=∅s_0 = \varnothing denotes the initial state in which nothing has been queried. Querying an instrument i∉M(s)i \notin\mathcal{M}(s) at state ss, and observing xx moves the circuit to the state s∪{(i,x)}s \cup\{(i,x)\}.

Actions. In a state ss, the circuit chooses one of two actions: (a) STOP, and output the Bayes prediction y^π(z)=arg⁡max⁡y∈{0,1}PD(Y=y∣z,s)\hat{y}_{\pi}(z) = \arg\max_{y \in\{0,1\}} P_D(Y = y \mid z,s), where YY denotes the unobserved label y(z)y(z), and PDP_D denotes the probabilistic model used to estimate the posterior belief, or (b) QUERY an instrument i∉M(s)i \notin\mathcal{M}(s) at cost ci(z)c_i(z), observe its response x(i)x^{(i)}, and move to the state s∪{(i,x(i))}s \cup\{(i,x^{(i)})\}. If M(s)=H\mathcal{M}(s) = \mathcal{H}, only STOP is available.

For a fixed prompt zz at state ss, the circuit’s posterior belief on the target label y(z)=1y(z) = 1 is given by,

qz(s)=PD(Y=1∣z,s)(5)q_z(s) = P_D(Y = 1 \mid z,s) \tag*{(5)}

If the circuit stops at ss, the optimal prediction is the more likely label, with Bayes error

ez(s)=min⁡{qz(s),1−qz(s)}.(6)e_z(s) = \min\{q_z(s), 1-q_z(s)\}. \tag*{(6)}

For a given λ≥0\lambda\ge0, let Vλ,z(s)V_{\lambda,z}(s) denote the minimum Lλ(π)L_{\lambda}(\pi) (for prompt zz) at state ss. The Bellman equations [5], together with Assumption 1, characterize Vλ,z(s)V_{\lambda,z}(s) as:

Vλ,z(s)=min⁡{ez(s), min⁡i∉M(s)[λci(z)+∑x∈{0,1}QX∣Z[i](x∣z)Vλ,z(s∪(i,x))]}(7)V_{\lambda,z}(s) = \min\left\{ e_z(s),\ \min_{i \notin\mathcal{M}(s)} \left[\lambda c_i(z) + \sum_{x \in\{0,1\}} Q_{X \mid Z}^{[i]}(x \mid z) V_{\lambda,z}\left(s \cup(i,x)\right)\right] \right\} \tag*{(7)}

The first term in Equation (7) is the terminal error for stopping and predicting the label at state ss. The second term selects the unqueried instrument with the lowest expected loss, paying λci(z)\lambda c_i(z) immediately and continuing from the subsequent state.

Vλ,z(s)V_{\lambda,z}(s) values can be computed exactly by backward induction over nn stages. For a fixed prompt zz and weight λ\lambda, solving Equation (7) yields the optimal circuit policy: 1) which models to query, 2) in what order, and 3) when to stop. This optimal policy is the solution πλ∗\pi^*_{\lambda} in Equation (3).

Generating circuits for unseen prompts: LMCircuit

Notice that the optimal policy is determined entirely by the label posterior at each state qz(s)q_z(s) and the instrument output distributions QX∣Z[i]Q_{X \mid Z}^{[i]} for a given prompt zz. This suggests a two-step approach for solving Equation (7): (i) estimate the probabilities qz(s)=PD(Y=1∣s,z)q_z(s) = P_D(Y = 1 \mid s,z), ∀s∈S\forall s \in S and QX∣Z[i](⋅∣z)Q_{X \mid Z}^{[i]}(\cdot\mid z), for a given zz from a training set and (ii) plug the estimates into the dynamic program solver. The training set contains prompts, their ground truth labels, and measurements from instruments in H\mathcal{H}. To estimate the probabilities, we represent each prompt zz by an embedding ϕ(z)∈Rd\phi(z) \in\mathbb{R}^{d} and fit a model fθf_{\theta} with outputs,

fθ(ϕ(z),s)=(ηθ(ϕ(z),s),νθ(1)(ϕ(z),s),…,νθ(n)(ϕ(z),s)),∀s∈Sf_{\theta}(\phi(z),s) = \left(\eta_{\theta}(\phi(z),s), \nu_{\theta}^{(1)}(\phi(z),s), \ldots, \nu_{\theta}^{(n)}(\phi(z),s)\right), \qquad\forall s \in S

Figure 3. Generating circuits. We first fit a probabilistic model from labeled prompts and instrument measurements (Section 3.2). At inference time, a prompt embedding is passed through this model, and a dynamic program generates a circuit that adaptively queries instruments and decides when to stop (Equation (7)).

where ηθ(ϕ(z),s)≈Pr⁡(Y=1∣ϕ(z),s)\eta_\theta(\phi(z),s) \approx\Pr(Y=1\mid\phi(z),s) is an estimate of the label posterior, while νθ(i)(ϕ(z),s)≈Pr⁡(X(i)=1∣ϕ(z),s)\nu_\theta^{(i)}(\phi(z),s) \approx\Pr(X^{(i)}=1\mid\phi(z),s) is an estimate of the instrument probabilities. We then use the above estimates to compute the dynamic program for any new prompt. Since the distributions do not depend on λ\lambda, the same fitted model generates circuits that operate at different points of the cost–accuracy frontier.

Remark 2 (Zero-cost prediction). At the initial state s0=∅s_0=\varnothing, the LMCircuit can give a prediction on the output label for a prompt zz without querying any instrument, i.e., at zero cost, with Pr⁡(Y=1∣z)≈ηθ(ϕ(z),s0)\Pr(Y=1\mid z)\approx\eta_\theta(\phi(z),s_0), where ηθ(ϕ(z),s0)\eta_\theta(\phi(z),s_0) is an output of fθ(ϕ(z),s0)f_\theta(\phi(z),s_0).

The fitted model fθf_\theta can then run completely on CPU. Hence, it provides a practical and lightweight way to organize how inference is allocated and optimized over LLM classifiers.

Analysis of the Optimal Circuit

This section studies the properties of optimal circuits. For a fixed instrument pool H\mathcal{H}, we first characterize the two endpoints of the error–cost tradeoff: (i) when it is optimal to make no queries, and (ii) what the best possible accuracy is using all the instruments. We then ask how adding one additional instrument to an existing pool improves this maximum accuracy, which yields practical guidance for assembling a pool of instruments. Finally, we derive scaling laws for circuits, i.e., the rate at which the error of the optimal circuit decreases with query cost on a prompt, and how this rate changes once averaged over a population of prompts with varying difficulty. All proofs are provided in Section B of the appendix.

Fundamental Limits of Circuits for a Fixed and Finite Pool

In this section, we consider the two extreme circuits for a given prompt zz and an instrument pool H\mathcal{H}: 1) Zero-cost circuit, where no instrument is queried and the output is predicted based on the prior belief, and 2) Full-cost circuit, where all instruments in H\mathcal{H} are queried.

For a prompt zz, consider the circuit π0(z)\pi_0(z) that stops at state s0=∅s_0=\varnothing, i.e., before any instrument is queried. It then predicts the more likely label according to its prior belief qz(∅)=PD(Y=1∣z)q_z(\varnothing)=P_D(Y=1\mid z), and incurs the Bayes error Eπ0(z)=min⁡y∈{0,1}PD(Y=y∣z)\mathbb{E}_{\pi_0(z)}=\min_{y\in\{0,1\}}P_D(Y=y\mid z).

Proposition 2 (Optimality of the zero-cost circuit). The zero-cost circuit π0(z)\pi_0(z) is optimal for prompt zz if @ and only if λ≥λLB(z):=sup⁡π:Cπ(z)>0[Eπ0(z)−Eπ(z)]+Cπ(z)\lambda\ge\lambda_{\mathrm{LB}}(z) := \sup_{\pi:C_\pi(z)>0} \frac{[E_{\pi_0}(z)-E_\pi(z)]_+}{C_\pi(z)}. Letting cmin⁡(z):=min⁡i∈Hci(z)c_{\min}(z) := \min_{i\in\mathcal{H}} c_i(z), we have that λ≥12cmin⁡(z)\lambda\ge\frac{1}{2c_{\min}(z)} is sufficient for π0\pi_0 to be optimal.

Proposition 2 states that for all λ\lambda exceeding a certain threshold, it is optimal to make no queries. At the other endpoint, λ=0\lambda= 0, the circuit uses all measurements in H\mathcal{H} to maximize accuracy, and its minimum error is the Bayes error after observing the full instrument pool. Let X(H)X^{(\mathcal{H})} denote the measurements obtained by querying every instrument in H\mathcal{H}.

Proposition 3 (Minimum error for full-cost circuit). For a prompt zz and instrument pool H\mathcal{H}, the minimum achievable error is

Emin⁡(z,H)=EX(H)∣Z=z[min⁡y∈{0,1}PD(Y=y∣z,X(H))](8)E_{\min}(z,\mathcal{H}) = \mathbb{E}_{X^{(\mathcal{H})}\mid Z=z}\left[\min_{y\in\{0,1\}} P_D\left(Y=y\mid z,X^{(\mathcal{H})}\right)\right] \tag*{(8)}

As a result, Emin⁡(z,H)E_{\min}(z,\mathcal{H}) is a lower bound on the error of any circuit constructed from H\mathcal{H} for any λ\lambda. This limit allows us to answer how much performance can improve when another instrument becomes available. Denote this instrument by inewi_{\mathrm{new}}, and let the corresponding distribution generating observations be QX∣Z[inew](⋅∣z)Q^{[i_{\mathrm{new}}]}_{X\mid Z}(\cdot\mid z). Then, the new pool is denoted by Hnew=H∪inew\mathcal{H}_{\mathrm{new}} = \mathcal{H} \cup i_{\mathrm{new}}. The next result characterizes the value of adding a new instrument to the existing pool.

Proposition 4 (Gain from adding one extra instrument). Over Z∼PZZ\sim P_Z,

EZ[E(Z,H)−E(Z,Hnew)]≤12I(Y;X(inew)∣X(H),Z)(9)\mathbb{E}_Z[E(Z,\mathcal{H})-E(Z,\mathcal{H}_{\mathrm{new}})] \le\sqrt{\frac{1}{2}I\left(Y;X^{(i_{\mathrm{new}})}\mid X^{(\mathcal{H})},Z\right)} \tag*{(9)}

where the mutual information is computed under the joint distribution of (Y,X(H),X(inew),Z)(Y,X^{(\mathcal{H})},X^{(i_{\mathrm{new}})},Z).

What makes a new instrument useful? Its value is not determined by the instrument’s accuracy but by the information it provides about the target beyond the measurements already available.

Practical significance of Proposition 4. To generate the circuit for any prompt zz, we use a labeled training set to estimate the joint distribution of (Y,X(H))(Y,X^{(\mathcal{H})}), conditional on zz. As the pool size ∣H∣|\mathcal{H}| increases, this joint distribution becomes more difficult to estimate, which can degrade the performance of the resulting dynamic program. At the same time, a larger pool provides more opportunities to obtain complementary measurements, enabling more powerful circuits for a wider range of prompts. Pool construction must therefore balance the additional information supplied by new instruments against the statistical cost of estimating their joint behavior. Proposition 4 suggests a greedy strategy for constructing a task-specific instrument pool. Given a candidate pool of NN instruments, we can initialize H\mathcal{H} with a low-cost instrument and iteratively add the candidate that maximizes the conditional-information bound in Equation (9), until the additional gains are non-significant. The resulting pool can be used in the dynamic program.5 Figure 12 shows that greedy selection by conditional information reaches the full-pool frontier with 2–3 instruments on the four tasks shown, and dominates accuracy-first, cheapest-first, and random ordering methods of building the pool.

Figure 12. Building the instrument pool. We build the instrument pool one instrument at a time according to different strategies: by estimated conditional information given the embedding and the pool (greedy algorithm), by accuracy on the training calls (most accurate first), cheapest first, or in random order (mean of three runs). Every point is our circuit restricted to that pool, using a single fitted model (an earlier version of the model without latent types). The circled point is the circuit with all instruments. Top: frontier area of test accuracy. Middle: test accuracy when every instrument in the pool is queried (λ=0\lambda= 0). Bottom: the upper bound I/2\sqrt{I}/2 of Proposition 4 on the drop in Bayes error from adding each instrument (II its estimated conditional information, in nats), and the drop in the model’s Bayes error E[min⁡{η,1−η}]\mathbb{E}[\min\{\eta, 1-\eta\}] that the instrument’s answers actually produce, both on test.

Scaling Laws for Classification Circuits

In this section, we characterize the resulting relationship between optimal error and expected query cost through its error–cost scaling law. In particular, we study how the error probability decreases with query cost as λ→0\lambda\to0 and consider its implications in a given population of prompts.

For each instrument i∈Hi \in\mathcal{H}, its accuracy on prompt zz is QX∣Z[i](y(z)∣z)Q^{[i]}_{X|Z}(y(z)\mid z), which we denote by ai(z)a_i(z). For the asymptotic analysis, we make the following assumptions: (1) each instrument can be queried arbitrarily many times, (2) every instrument is at least as accurate as random guessing on every prompt, i.e., ai(z)∈[1/2,1)a_i(z) \in[1/2,1) for all i∈Hi \in\mathcal{H} and z∈Zz \in\mathcal{Z}, and (3) the response distribution is symmetric with respect to the ground-truth label: conditional on Hy:y(z)=yH_y : y(z)=y, an independent call to instrument ii returns the correct label yy with probability ai(z)a_i(z). While the optimal query and stopping decisions continue to satisfy the Bellman equation in Equation (7), they can no longer be realized using backward induction in this asymptotic regime.

For a fixed prompt zz, it is useful to view the sequential decision problem as a hypothesis test between H0:y(z)=0H_0 : y(z)=0 and H1:y(z)=1H_1 : y(z)=1. Each collected measurement provides information about which label is correct. A measurement from instrument ii is distributed as Ber⁡(1−ai(z))\operatorname{Ber}(1-a_i(z)) under H0H_0 and Ber⁡(ai(z))\operatorname{Ber}(a_i(z)) under H1H_1. Then, the KL-divergence di(z)=DKL(Ber⁡(ai(z))∥Ber⁡(1−ai(z)))d_i(z)=D_{\mathrm{KL}}(\operatorname{Ber}(a_i(z))\mathbin{\|}\operatorname{Ber}(1-a_i(z))) quantifies the expected information provided by one measurement for distinguishing the two hypotheses. Because instruments differ in cost, we normalize this quantity and define the information rate of instrument ii to be:

Ii(z)≔di(z)ci(z)I_i(z) \coloneqq\frac{d_i(z)}{c_i(z)}

and maximum information rate of the model pool to be:

I(z)≔max⁡i∈HIi(z).I(z) \coloneqq\max_{i\in\mathcal{H}} I_i(z).

Thus I(z)I(z) measures the largest information rate available for classifying prompt zz.

For a fixed prompt zz, let Eλ(z)E_\lambda(z) and Cλ(z)C_\lambda(z) denote the error probability and expected query cost of the optimal circuit for parameter λ\lambda. As λ\lambda approaches zero, the circuit places progressively less weight on query cost and can spend more to reduce its error. For 0<I(z)<∞0 < I(z) < \infty, the optimal error approaches zero, but doing so requires an expected cost that grows without bound. This raises the question of how quickly error decreases as more cost is incurred. The following result shows that the decay is exponential with cost, at rate I(z)I(z).

Theorem 1 (Scaling law for a fixed prompt). For every prompt zz with 0<I(z)<∞0 < I(z) < \infty, we have that

lim⁡λ↓0−log⁡Eλ(z)Cλ(z)=I(z).(10)\lim_{\lambda\downarrow0}\frac{-\log E_\lambda(z)}{C_\lambda(z)} = I(z). \tag*{(10)}

The information rate also determines how the optimal circuit allocates its queries.

How does the optimal circuit allocate its queries? As λ↓0\lambda\downarrow0, the optimal circuit spends nearly all of its expected cost on instruments with the largest information rate, Ii(z)=I(z)I_i(z)=I(z); see Proposition 5.

For a fixed λ\lambda, the optimal next query can still depend on the measurements collected so far and on its cost, so the optimal circuit may query several instruments. However, as λ↓0\lambda\downarrow0, the fraction of expected query cost spent on instruments with Ii(z)<I(z)I_i(z)<I(z) tends to zero.

This result characterizes the optimal circuit for a fixed prompt as λ↓0\lambda\downarrow0. The information rate I(z)I(z) determines the rate of error decay. It then offers a natural notion of prompt “difficulty”: a prompt with smaller I(z)I(z) requires more query cost to attain the same error level.

Now consider a population of prompts Z∼PZZ \sim\mathcal{P}_Z, and let EλE_\lambda and CλC_\lambda denote the average error probability and expected query cost of the optimal circuit. While Theorem 1 establishes exponential error decay for each prompt with I(z)>0I(z)>0, the population may contain prompts on which every instrument performs only slightly better than random guessing. Thus, information rates can be arbitrarily close to zero across the population. The population error–cost tradeoff therefore depends on the distribution of these rates, as characterized by the following theorem.

Theorem 2 (Scaling law across prompts). Assume that, for almost every prompt zz, the prior Bayes error Eπ0(z)E_{\pi_0}(z) is bounded below by a uniform positive constant, query costs are uniformly bounded above, and 0<I(z)<∞0 < I(z) < \infty. Then

lim⁡λ↓0−log⁡EλCλ=(E[1I(Z)])−1.(11)\lim_{\lambda\downarrow0}\frac{-\log E_\lambda}{C_\lambda} = \left(\mathbb{E}\left[\frac{1}{I(Z)}\right]\right)^{-1}. \tag*{(11)}

where the right-hand side is zero if the expectation is infinite.

An important consequence is that exponential error decay for almost every prompt need not imply exponential decay with average cost across the population. To characterize the resulting error–cost tradeoff even further, we can consider how the probability of small information rates decreases as they approach zero.

Corollary 1 (Scaling law across prompts under a tail assumption). Under the conditions of Theorem 2, suppose that, for some α>0\alpha> 0,

Pr⁡(I(Z)≤r)≍rαas r↓0,\Pr(I(Z) \le r) \asymp r^\alpha\qquad\text{as } r \downarrow0,

where r>0r > 0.6 In this case, E[1/I(Z)]\mathbb{E}[1/I(Z)] is finite for α>1\alpha> 1. Then, as λ↓0\lambda\downarrow0:

(i) If α>1\alpha> 1, the error decays exponentially with cost, at the rate given in Theorem 2.

(ii) If α=1\alpha= 1, the error decays at a stretched-exponential rate, −log⁡Eλ≍Cλ-\log E_\lambda\asymp\sqrt{C_\lambda}.

(iii) If 0<α<10 < \alpha< 1, the error decays polynomially, Eλ≍Cλ−α1−αE_\lambda\asymp C_\lambda^{-\frac{\alpha}{1-\alpha}}.

The results point to an obvious limit of inference-time computation: even an optimal circuit cannot compensate for a lack of informative measurements. Improving the instruments on the hardest prompts can therefore change the scaling law itself.

Experiments

Setup. We consider seven tasks: AgentJudgeBench (Verma et al., [68]) for judging whether an AI agent’s planned tool calls are correct, BeaverTails (Ji et al., [36]) for safety classification, BIG-Bench Hard (Suzgun et al., [62]) for reasoning questions, Decision Index (Multimodal Art, [51]) for decision tasks, HealthBench (Arora et al., [4]) to evaluate whether a text satisfies a physician’s rubric, LMArena (Chiang et al., [15]) for classifying human rater preference, and PolitiCause (Garcia-Corral et al., [23]) for detecting causality in political text. For each task, a training set is used to estimate the LMCIRCUIT (as described in § 3.2) required by the dynamic program. We then vary the parameter λ\lambda on a validation set to obtain circuits at different operating points and report test accuracy against average query cost. Within each task, costs are normalized relative to the most expensive instrument. Additional details are in Section C.

We benchmark against (1) Predictive routers, including ROUTELLM (Ong et al., [54]), HYBRIDLLM (Ding et al., [21]), and CARROT (Somerstep et al., [60]); (2) Model cascades, including CASCADE ROUTING (Dekoninck et al., [19]), CONSISTENCY CASCADE (Yue et al., [76]), CASCADED SELECTIVE (Jung et al., [37]), and FRUGALGPT (Chen et al., [13]); (3) Sampling methods, namely ADAPTIVE-CONSISTENCY (Aggarwal et al., [2]); and (4) individual instruments and majority voting.

Results

LMCIRCUIT improves the cost–accuracy frontier. Across the seven tasks, the learned circuits attain higher accuracy than every individual instrument and than the routing and cascade baselines at nearly all budgets (Figure 4). The choice of instruments that the circuit uses depends on the task and is not predictable from cost or accuracy alone. On LMArena, the two mid-cost Gemma models (26B and 31B) are the most informative per unit cost, and the circuit queries almost exclusively these two for most budgets. On BeaverTails, the cheapest model, Gemma 4B, carries the most information per unit cost (Figure 11). Hence, the circuit queries it on most prompts and reserves the expensive models for the highest budgets.

Figure 4. LMCircuit improves the cost-accuracy frontier over routing and cascading baselines. As λ\lambda decreases and additional queries become less costly, the circuits make more calls and also change which instruments they use (see lower panel). The allocations differ substantially across tasks, indicating that the value of an instrument depends on the classification problem and on other measurements available to the circuit.

Figure 11. Instruments with higher information per unit cost enter the circuit earlier. Expected calls per unseen prompt to each instrument as a function of the circuit’s expected cost per item. Instruments are colored by their rank under the estimated information rate r^i\hat r_i in Equation (45), with the corresponding value in bits per unit cost shown in parentheses. Across the three tasks, instruments with larger r^i\hat r_i tend to begin receiving calls at lower expected cost. The highest-rate instrument enters first on every task, including Arena, where Gemma 26B is not the cheapest instrument.

LMCIRCUIT provides calibrated decisions. For a given prompt zz, the distribution estimator outputs an estimate of Pr⁡(Y=1∣z)\Pr(Y = 1 \mid z). Based on its confidence, LMCIRCUIT assesses whether the prompt is easy or hard, selects the instrument path accordingly, and updates its posterior belief after each measurement. As expected, under the same cost-accuracy parameter λ\lambda, harder prompts require more model calls (Figure 10). An advantage of our method is that LMCIRCUIT’s prior alone can yield a label prediction without querying any instrument. Therefore, at the low-cost end of the frontier, the average query cost can fall below that of the cheapest individual instrument. Figure 6 further shows that LMCIRCUIT produces better calibrated decisions on unseen data.

Figure 6. The circuit’s beliefs are calibrated at every budget. For each task, left: Brier score on the test set (lower is better) against expected cost per item; right: reliability diagrams of the circuits at different budgets and of majority voting. The dashed line is perfect calibration. The circuit’s expected calibration error is at most 2.1% at every budget on six tasks, and at most 5.2% on BIG-Bench Hard, whereas majority voting’s is 4.8% to 24.6% and its miscalibration differs in direction by task.

Figure 10. The circuit allocates more queries when the prior prediction and instrument measurements disagree. We present the expected number of calls per test prompt for circuits at normalized budgets 0.03, 0.1, and 0.2 on Arena, BeaverTails and PolitiCAUSE, grouped by prompt type (left), instrument disagreement d(z)d(z) (middle: 0, (0, 1/3], (1/3, 2/3], and (2/3, 1]), and whether the embedding prediction agrees with the majority of the recorded calls (right). Percentages below each group give its share of the test set. Mean calls increase from all right to mostly wrong prompts on every task. The clearest separation occurs when conditioning on agreement with the embedding prediction: at budget 0.2, the circuit makes 1.7–1.9× as many calls when the embedding and instrument majority disagree on Arena (1.84 vs. 1.08), BeaverTails (1.77 vs. 0.95) and PolitiCAUSE (1.85 vs. 1.11).

LMCIRCUIT learns whether more queries help and how reliable each instrument is. For any given prompt, initially, the circuit predicts its label from the prompt embedding alone, without querying any instrument (Remark 2). We call this the embedding prediction. To investigate the impact of LLM calls on top of embedding predictions, we organize unseen prompts by the fraction of recorded calls that return the correct label (Figure 9). On noisy prompts, where this fraction lies between (1/2,1)(1/2, 1), the circuit at budget 1 improves the accuracy of embedding prediction by 5%–44%. On prompts where every call is correct, it matches majority voting (within 1.3% accuracy difference) while using 38%–86% fewer calls.

Figure 9. Additional queries improve accuracy when instrument errors vary across calls. We stratify unseen prompts by r(z)r(z) and report accuracy for four predictors: the embedding prediction ηθ(ϕ(z),s0)\eta_\theta(\phi(z), s_0), which makes no instrument calls; our circuit at normalized budgets of 0.03 and 1; and majority voting over the instruments the circuit can query. Numbers above the bars give the mean number of calls per prompt, and percentages below each group give its share of the test set. Relative to the embedding prediction, the circuit at budget 1 improves accuracy on noisy prompts by 5–44 percentage points. On all right prompts, it comes within 1.3 percentage points of majority voting on four tasks while using 38–86% fewer calls.

The fitted model of LMCIRCUIT is accurate about the instruments. The dynamic program relies on the fitted model’s estimates of how each instrument responds on a given prompt. Figure 8 shows that, without access to test labels, the fitted model predicts 1) each instrument’s accuracy on unseen prompts, 2) the correlations of the instruments’ answers and errors across prompts. Across the seven tasks, the implied correlations between instruments’ answers, and between their errors, are within 4%–17% of the measured values, and the implied accuracy of each instrument is within 1.2%–3.6% of its measured accuracy.

Figure 8. The fitted model predicts each instrument’s accuracy on unseen data without labels and recovers nearly all of the dependence between instruments. Correlations and accuracies implied by the fitted model on unlabeled test prompts (Equation (44)) against those measured from recorded calls and labels. Left: correlation of two instruments’ answers across prompts (one point per pair). Middle: correlation of their errors. Right: accuracy of each instrument. The dashed line is equality.

LMCIRCUIT is easy and fast to implement. Our method is less complex than many baselines (Figure 5), attaining significantly shorter training times. The estimation step uses a simple neural network together with pretrained embeddings of prompts. We do not require token probabilities or training larger models (such as DistilBERT or DeBERTa) for each instrument as in other methods, and the dynamic program is cheap for moderate pools of models. We note, however, that the dynamic program grows exponentially with the number of models. In many applications, this is less restrictive than it may seem, since a small, well-chosen pool of classifiers is often sufficient (see § 4.1 for guidance on model pool selection).

Figure 5. LMCircuit is substantially faster to construct than most routing and cascading baselines. Across all seven tasks, LMCircuit is trained in 1–3 minutes, while some baselines require minutes to hours.

Types of noise in measurements: A measurement instrument is subject to two types of noise: (1) stochastic noise, errors that can be reduced through repeated or correlated measurements, and (2) systematic noise, errors that are inherent to how the instrument was trained. Our results show that the dynamic program automatically adapts to both. It selects highly correlated models to reduce (1) when queried models are moderately accurate (∼\sim75–80%, Figure 13), and less correlated models to reduce (2) when queried models are less accurate (∼\sim63–66%, Figure 14).

Figure 13. Example of generated circuit and instrument’s error correlations on BeaverTails

Figure 14. Example of generated circuit and instrument’s error correlations on LMArena

Conclusion

As LLM classifiers become components of larger systems, a practical question is how inference should be organized and allocated among them to maximize accuracy at the lowest expected cost. We take a decision-theoretic view of this problem, using a lightweight probabilistic model to characterize the available classifiers and a dynamic program to decide what to query and when to stop. We are then able to generate circuits for any prompt while offering control on the error–cost tradeoff. One challenge is to extend this to larger pools of general measurement instruments with potential feedback among them. More broadly, our framework can broaden access to high-quality and automated decision-making for practitioners without frontier-model budgets.

Acknowledgments

This work was supported in part by the National Science Foundation under Grant No: 2506573, CIF-2231707, CIF-2312667, and a Gift by Coefficient Giving. Hadi Khalaf is supported by a Kempner Graduate Research Fellowship which has been made possible by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University. The Flatiron Institute is funded by the Simons Foundation.

Related Work

This work is closely related to the following fields: 1) Sequential information acquisition (decision-theoretic foundations), 2) Routing and cascading methods, 3) LLMs as classifiers, judges, and annotators, 4) Theory of LLM orchestration and test-time compute, and 5) Aggregating Noisy Classifiers.

Sequential information acquisition: Our formulation is based on sequential hypothesis testing. Wald’s sequential probability ratio test [71] established that stopping adaptively, as soon as the accumulated evidence crosses a threshold, requires fewer samples on average than any fixed-sample test with the same error probabilities. Chernoff extended this to the sequential design of experiments [14], where the decision-maker chooses which experiment to run next, and DeGroot cast the problem as Bayesian decision-making in which each experiment is valued by its expected reduction in posterior uncertainty [18]. These fundamental ideas are the principle behind our circuits. Since each LLM query is a costly experiment, the circuit selects the next instrument by the expected improvement in its posterior belief, and it stops once that belief is confident enough that further queries are not worth their cost. The modern literature on active hypothesis testing and controlled sensing [52, 53, 69] develops this framework further, deriving information-theoretic lower bounds on expected sample size under multiple hypotheses and heterogeneous action costs, and showing that simple posterior-driven policies attain the optimal error exponents. These works solve a single testing instance in which the observation model is known in advance. In our setting, each prompt is one such instance, but there are countably many possibilities of prompts, and the observation model of each instrument for any given prompt must be learned from data.

Routing and Cascading Methods: In routing, a prompt is assigned to a single model in a pool of models that is predicted to handle it best. For example, RouteLLM [54] trains binary routers between a strong and a weak model, while HybridLLM [21] trains a classifier to predict whether a cheap model’s answer will be of acceptable quality and routes to the large model only when it is needed. CARROT [60] routes by predicting both expected quality and expected cost per model for a query and minimizing a weighted combination, with a minimax-rate-optimality guarantee for the resulting router. More routing mechanisms [48, 22, 72, 77, 61, 31] can be found in the survey [50]. The circuits proposed in this work generalize routers since a circuit decides after each observation whether the evidence collected so far is sufficient or whether another model should be queried.

In cascading, models are queried one after another in a fixed order, typically from cheapest to most expensive, and the process stops as soon as some confidence or quality criterion is met. FrugalGPT [13] chains LLM APIs from cheap to expensive and uses a learned scorer to decide whether the current answer is reliable enough to stop. Automix [3] has a small model self-verify its own answer and uses this verification signal to decide whether to escalate to a larger model. Yue et al. [76] escalate based on agreement between a weak model’s answers under different reasoning formats, treating disagreement as the trigger for the strong model. Jung et al. [37] use judges of increasing size with a conformal-style confidence threshold that guarantees a target level of agreement with humans. More cascading methods can be found in the survey [50]. Our approach differs from these cascades in two ways. First, a cascade queries models in a fixed order and only decides whether to escalate, whereas a circuit chooses which model to query next based on the prompt and the outputs observed so far. Second, a cascade typically returns the output of the model at which it stops, discarding earlier outputs once they have triggered escalation, whereas a circuit combines all observed outputs into a posterior belief on the label and predicts from that belief.

Among cascading methods, TREACLE Zhang et al., [78] and cascade-routing Dekoninck et al., [19] do not follow a fixed order of models. Using reinforcement learning, TREACLE Zhang et al., [78] learns a policy that selects the next LLM and prompting scheme based on the question and the consistency of responses so far, under a budget shared across a dataset of questions. Our circuits also adapt the next query to past model outputs, but rather than training the policy itself, we learn a probabilistic model of how each instrument responds on a given prompt and compute the optimal query-and-stop policy from it by dynamic programming. The resulting circuit is optimal with respect to the learned model and its error–cost tradeoff can be analyzed, neither of which holds for a policy trained end-to-end by reinforcement learning. Cascade routing Dekoninck et al., [19] selects the next model after each output by estimating, for each set of remaining models, the quality of the best output it would produce net of its cost, and finally returns the output with the highest estimated quality. Our circuits also choose the next model from the full pool after each observation, but differ in two respects. The decision to continue is solved exactly by dynamic programming rather than by a one-shot estimate of what the remaining models would yield, and the prediction combines all observed outputs through the posterior on the label rather than selecting one output.

LLMs as classifiers, judges, and annotators. Prompting turns an LLM into a classifier without training [11, 45], and this has made LLMs default raters in many pipelines. They judge model outputs [79, 47, 74], either as prompted frontier models or as judges fine-tuned for the task [40, 80, 33] (Vu et al. 2024). They provide feedback for alignment [43], act as safety classifiers and guardrails [35, 29, 55], and annotate text in the social sciences and in scientific review [28] (Liu & Shah 2023). A parallel line of work measures the reliability of raters. This includes documenting position and verbosity biases, disagreement with humans, and vulnerability to adversarial inputs [63, 65], as well as low intra-rater reliability, where the same judge returns different labels across runs [27].

Theory of LLM orchestration and test-time compute: A growing body of work analyzes how to allocate inference across LLM calls. For a single model, the test-time scaling literature studies how accuracy improves with the number of LLM calls or the amount of verification [59, 58]. Within this line, Chen et al. [12] shows that accuracy under majority voting can be non-monotone in the number of calls, Huang et al. [32] give sample-complexity bounds for voting and verifier-based selection, and Huang et al. [34] characterize when to stop sampling through Bayesian optimal stopping.

For multiple models, routing has been analyzed as a cost-aware prediction problem [60]. Cascades have been studied from several angles: as a sequence of linear programs [19], as a Pandora’s Box search problem [6], and as a queueing system with congestion [26]. Information-theoretic criteria have also been proposed for choosing which models to include in an ensemble [66]. These rest on older foundations in metareasoning [30], adaptive submodularity [25], and learning to defer [49]. Closest to our analysis, concurrent work by Li et al. [44] studies sequential binary classification over heterogeneous LLMs with known accuracies, costs, and latencies. They show that in the high-confidence limit a policy alternating between two hypothesis-specific specialists is asymptotically optimal. Our asymptotic analysis of agrees with this: under symmetric accuracies the two specialists coincide, and the optimal circuit concentrates its cost on the instrument with the highest information rate per unit cost. These works, however, analyze a single instance with known model parameters and unlimited repeated queries. Our main focus is on the practical problem of a finite pool of instruments whose reliabilities vary across prompts and must be learned from data, for which we solve the exact dynamic program and show that the resulting circuits improve the cost–accuracy frontier in practice.

Aggregating Noisy Classifiers: We draw from work on combining the outputs of multiple noisy classifiers to determine a true label. In the context of LLMs, aggregating the outputs of smaller models can be used to approximate the judgement of frontier models at lower cost [67]. Previous work in crowdsourcing and weak supervision [17, 56] learn the error rate of each individual classifier and aggregate them for the distribution over the true label, assuming independence between the errors of each classifier. Ensemble diversity [20] extends this to a machine learning setting by accounting for the correlation of errors between models, which remains an important hurdle for combining the outputs of large language models [24, 39, 41]. In this spirit, our probabilistic model learns a distribution over their responses and prompt embeddings, enabling the dynamic program to account for correlation between model outputs.

Proofs

Proof of Proposition 1. Consider a circuit that queries a batch (i1,…,ik)(i_1,\ldots,i_k). Replace the batch by the same kk queries in any fixed order, without allowing the intermediate responses to affect subsequent queries. After all kk responses have been observed, the circuit proceeds exactly as the original circuit would after observing the batch. The resulting circuit therefore has the same measurements, the same final prediction, and the same total query cost. Repeating this replacement for every batch gives the desired sequential circuit.

Proof of Proposition 2. Consider the objective function for a fixed prompt zz:

Lλ(π,z)=Eπ(z)+λCπ(z)(12)\mathcal{L}_{\lambda}(\pi,z)=E_{\pi}(z)+\lambda C_{\pi}(z) \tag*{(12)}

Then, the zero-cost circuit is optimal when:

Eπ0(z)≤Eπ(z)+λCπ(z)(13)E_{\pi_0}(z)\leq E_{\pi}(z)+\lambda C_{\pi}(z) \tag*{(13)}

since Cπ0(z)=0C_{\pi_0}(z)=0 for the zero-cost circuit. This, together with λ≥0\lambda\geq0 directly results in proving Proposition 2.

Now consider the expression:

λLB=sup⁡π:Cπ(z)>0[Eπ0(z)−Eπ(z)]+Cπ(z)(14)\lambda_{\mathrm{LB}}=\sup_{\pi:C_{\pi}(z)>0}\frac{\left[E_{\pi_0}(z)-E_{\pi}(z)\right]_+}{C_{\pi}(z)} \tag*{(14)}

The upper bound on λLB\lambda_{\mathrm{LB}} is obtained using the following bounds: Eπ0(z)≤12E_{\pi_0}(z)\leq\frac{1}{2}, Eπ(z)≥0E_{\pi}(z)\geq0, and Cπ(z)≥Cmin⁡(z)C_{\pi}(z)\geq C_{\min}(z).

Proof of Proposition 4. Fix a prompt zz, and write X=X(H)X=X^{(\mathcal{H})} and Xnew=X(inew)X_{\mathrm{new}}=X^{(i_{\mathrm{new}})}. Define

ηz(x)=PD(Y=1∣z,X=x),ηz(x,u)=PD(Y=1∣z,X=x,Xnew=u).\eta_z(x)=P_D(Y=1\mid z,X=x),\qquad\eta_z(x,u)=P_D(Y=1\mid z,X=x,X_{\mathrm{new}}=u).

Since the Bayes accuracy for a binary label is g(q)=max⁡{q,1−q}g(q) = \max\{q, 1-q\}, we have

Emin⁡(z,H)−Emin⁡(z,Hnew)=E[g(ηz(X,Xnew))−g(ηz(X))∣Z=z]≤E[∣ηz(X,Xnew)−ηz(X)∣∣Z=z],E_{\min}(z,\mathcal{H}) - E_{\min}(z,\mathcal{H}_{\mathrm{new}}) = \mathbb{E}\left[g(\eta_z(X,X_{\mathrm{new}})) - g(\eta_z(X)) \mid Z=z\right] \le\mathbb{E}\left[\left|\eta_z(X,X_{\mathrm{new}}) - \eta_z(X)\right| \mid Z=z\right],

where the inequality follows because gg is 1-Lipschitz.

Since YY is binary, conditioning on X=xX=x, the inner expectation equals

TV⁡(PD(Y,Xnew∣z,x),PD(Y∣z,x)PD(Xnew∣z,x)).\operatorname{TV}\left(P_D(Y,X_{\mathrm{new}}\mid z,x),P_D(Y\mid z,x)P_D(X_{\mathrm{new}}\mid z,x)\right).

By Pinsker’s inequality and Jensen’s inequality,

Emin⁡(z,H)−Emin⁡(z,Hnew)≤12I(Y;Xnew∣X,Z=z).E_{\min}(z,\mathcal{H}) - E_{\min}(z,\mathcal{H}_{\mathrm{new}}) \le\sqrt{\frac{1}{2}I(Y;X_{\mathrm{new}}\mid X,Z=z)}.

Averaging over ZZ and applying Jensen’s inequality again,

EZ[Emin⁡(Z,H)−Emin⁡(Z,Hnew)]≤12I(Y;X(inew)∣X(H),Z).\mathbb{E}_Z\left[E_{\min}(Z,\mathcal{H}) - E_{\min}(Z,\mathcal{H}_{\mathrm{new}})\right] \le\sqrt{\frac{1}{2}I(Y;X^{(i_{\mathrm{new}})}\mid X^{(\mathcal{H})},Z)}.

Nonnegativity follows because additional measurements can always be ignored.

Proofs for Fixed-Prompt Scaling

We first show that any circuit achieving error probability ε\varepsilon requires expected cost at least log⁡(1/ε)/I(z)−O(1)\log(1/\varepsilon)/I(z) - O(1), and then construct a sequential likelihood-ratio test that attains this bound up to an additive constant.

Preliminaries. Fix a prompt zz and suppress its dependence in the notation. Write ai:=ai(z)a_i := a_i(z) for an instrument’s accuracy on prompt zz and ci:=ci(z)c_i := c_i(z) for its cost. For the asymptotic analysis, each instrument i∈Hi \in\mathcal{H} may be queried arbitrarily many times, with independent randomness across calls.

Recall that classification reduces to testing H0:y(z)=0H_0:y(z)=0 against H1:y(z)=1H_1:y(z)=1, where a query to instrument ii produces

X∣H1∼Ber⁡(ai),X∣H0∼Ber⁡(1−ai).X\mid H_1\sim\operatorname{Ber}(a_i), \qquad X\mid H_0\sim\operatorname{Ber}(1-a_i).

The log-likelihood ratio contributed by a measurement x∈{0,1}x\in\{0,1\} is

Li(x)=log⁡Pr⁡(X=x∣H1,i)Pr⁡(X=x∣H0,i)=(2x−1)ℓi,ℓi:=log⁡ai1−ai.(15)L_i(x)=\log\frac{\Pr(X=x\mid H_1,i)}{\Pr(X=x\mid H_0,i)}=(2x-1)\ell_i,\qquad\ell_i:=\log\frac{a_i}{1-a_i}. \tag*{(15)}

By our assumption in this analysis, an instrument’s measurement distribution is symmetric under the two hypotheses, so

E1[Li(X)]=di,E0[Li(X)]=−di,\mathbb{E}_1[L_i(X)]=d_i,\qquad\mathbb{E}_0[L_i(X)]=-d_i,

where Ey\mathbb{E}_y denotes expectation under HyH_y and di:=DKL(Ber⁡(ai(z))∥Ber⁡(1−ai(z)))=(2ai−1)ℓid_i:=D_{\mathrm{KL}}(\operatorname{Ber}(a_i(z))\mathbin{\|}\operatorname{Ber}(1-a_i(z)))=(2a_i-1)\ell_i.

A sequential circuit chooses an instrument At∈HA_t\in\mathcal{H} at each step tt, observes its measurement XtX_t, and stops at a stopping time τ\tau. The choice of AtA_t and the decision to continue depend only on the preceding history Ft−1=σ(A1,X1,…,At−1,Xt−1)\mathcal{F}_{t-1}=\sigma(A_1,X_1,\ldots,A_{t-1},X_{t-1}). Let Ni=∑t=1τ1{At=i}N_i=\sum_{t=1}^{\tau}\mathbf{1}\{A_t=i\} denote the number of queries to ii.

Write q=qz(∅)∈(0,1)q=q_z(\varnothing)\in(0,1) for the prior probability of H1H_1 and m=min⁡{q,1−q}m=\min\{q,1-q\}. For a circuit π\pi, let PyπP_y^\pi and Eyπ\mathbb{E}_y^\pi denote probability and expectation under HyH_y, and define its prior-averaged query counts by

N‾iπ=qE1π[Ni]+(1−q)E0π[Ni].\overline{N}_i^\pi=q\mathbb{E}_1^\pi[N_i]+(1-q)\mathbb{E}_0^\pi[N_i].

The error probability and expected cost, conditional on the fixed prompt, are

E(π)=qP1π(Y^=0)+(1−q)P0π(Y^=1),(16)E(\pi)=qP^\pi_1(\widehat{Y}=0)+(1-q)P^\pi_0(\widehat{Y}=1), \tag*{(16)}
C(π)=∑i∈HciN‾iπ.(17)C(\pi)=\sum_{i\in\mathcal{H}}c_i\overline{N}^\pi_i. \tag*{(17)}

For an optimal circuit at price λ\lambda, these are Eλ(z)E_\lambda(z) and Cλ(z)C_\lambda(z), respectively.

Lemma 1 (Information lower bound). Every sequential circuit with finite expected cost satisfies

DKL(Pyπ∥P1−yπ)=∑i∈HdiEyπ[Ni],y∈{0,1},(18)D_{\mathrm{KL}}(P^\pi_y\mathbin\Vert P^\pi_{1-y})=\sum_{i\in\mathcal{H}}d_i\mathbb{E}^\pi_y[N_i],\qquad y\in\{0,1\}, \tag*{(18)}

where PyπP^\pi_y denotes the distribution of the complete terminated history under HyH_y. Consequently,

log⁡m2E(π)≤∑i∈HdiN‾iπ≤IC(π).(19)\log\frac{m}{2E(\pi)}\leq\sum_{i\in\mathcal{H}}d_i\overline{N}^\pi_i\leq IC(\pi). \tag*{(19)}

Proof. The querying and stopping rules depend only on the observed history and are identical under both hypotheses. Their contributions therefore cancel from the likelihood ratio of the terminated history. By the chain rule for relative entropy,

DKL(P1π∥P0π)=E1π[∑t=1τLAt(Xt)]=E1π[∑t=1τdAt]=∑i∈HdiE1π[Ni].D_{\mathrm{KL}}(P^\pi_1\mathbin\Vert P^\pi_0)=\mathbb{E}^\pi_1\left[\sum_{t=1}^{\tau}L_{A_t}(X_t)\right]=\mathbb{E}^\pi_1\left[\sum_{t=1}^{\tau}d_{A_t}\right]=\sum_{i\in\mathcal{H}}d_i\mathbb{E}^\pi_1[N_i].

The second equality follows by conditioning on the history before each query, since E1[LAt(Xt)∣Ft−1]=dAt\mathbb{E}_1[L_{A_t}(X_t)\mid\mathcal{F}_{t-1}]=d_{A_t}. Reversing the hypotheses gives DKL(P0π∥P1π)=∑idiE0π[Ni]D_{\mathrm{KL}}(P^\pi_0\mathbin\Vert P^\pi_1)=\sum_i d_i\mathbb{E}^\pi_0[N_i], proving Equation (18).

To relate the acquired information to classification error, let απ=P1π(Y^=0)\alpha_\pi=P^\pi_1(\widehat{Y}=0) and βπ=P0π(Y^=1)\beta_\pi=P^\pi_0(\widehat{Y}=1). By Equation (16),

E(π)=qαπ+(1−q)βπ≥m(απ+βπ).E(\pi)=q\alpha_\pi+(1-q)\beta_\pi\geq m(\alpha_\pi+\beta_\pi).

The Bretagnolle–Huber inequality [10] gives, in either direction,

DKL(Pyπ∥P1−yπ)≥log⁡12(απ+βπ)≥log⁡m2E(π).D_{\mathrm{KL}}(P^\pi_y\mathbin\Vert P^\pi_{1-y})\geq\log\frac{1}{2(\alpha_\pi+\beta_\pi)}\geq\log\frac{m}{2E(\pi)}.

Averaging these inequalities with weights qq and 1−q1-q, and applying Equation (18), yields

log⁡m2E(π)≤∑idiN‾iπ.\log\frac{m}{2E(\pi)}\leq\sum_i d_i\overline{N}^\pi_i.

Finally, di≤Icid_i\leq I c_i for every instrument, so ∑idiN‾iπ≤I∑iciN‾iπ=IC(π)\sum_i d_i\overline{N}^\pi_i\leq I\sum_i c_i\overline{N}^\pi_i=IC(\pi).

Lemma 2 (Likelihood-ratio test). Suppose 0<I<∞0<I<\infty, and choose i⋆∈arg⁡max⁡iIii^\star\in\arg\max_i I_i. For every B≥0B\geq0, there exists a sequential circuit πB\pi_B using only instrument i⋆i^\star such that

E(πB)≤e−B,IC(πB)≤B+ℓi⋆.(20)E(\pi_B)\leq e^{-B},\qquad IC(\pi_B)\leq B+\ell_{i^\star}. \tag*{(20)}

Proof. Query instrument i⋆i^\star repeatedly, and let

St=log⁡q1−q+∑k=1tLi⋆(Xk)S_t=\log\frac{q}{1-q}+\sum_{k=1}^{t}L_{i^\star}(X_k)

denote the posterior log-odds after tt queries. Write b=log⁡(q/(1−q))b = \log(q/(1-q)), so S0=bS_0 = b.

Consider the stopping rule

τB=inf⁡{t≥0:∣St∣≥B},\tau_B = \inf\{t \ge0 : |S_t| \ge B\},

which predicts H1H_1 if SτB≥0S_{\tau_B} \ge0 and H0H_0 otherwise. The posterior probability of error at any stopping state with log-odds StS_t is

min⁡{Pr⁡(H1∣Ft),Pr⁡(H0∣Ft)}=11+e∣St∣.\min\{\Pr(H_1 \mid\mathcal{F}_t), \Pr(H_0 \mid\mathcal{F}_t)\} = \frac{1}{1+e^{|S_t|}}.

Thus, once we establish that τB\tau_B is finite, the stopping rule immediately gives E(πB)≤e−B\mathbb{E}(\pi_B) \le e^{-B}.

It remains to bound the expected stopping time. If ∣b∣≥B|b| \ge B, the circuit stops without querying, and both inequalities hold.

Suppose ∣b∣<B|b| < B. Each query changes StS_t by ±ℓi⋆\pm\ell_{i_\star}, so for Tn=τB∧nT_n = \tau_B \wedge n, we have that

∣STn∣≤B+ℓi⋆.|S_{T_n}| \le B + \ell_{i_\star}.

Under HyH_y, the process (2y−1)(St−b)−tdi⋆(2y-1)(S_t-b)-t d_{i_\star} is a martingale. Since τB\tau_B is not yet known to have finite expectation, we first apply optional stopping at the bounded stopping time TnT_n. Using ∣STn∣≤B+ℓi⋆|S_{T_n}| \le B+\ell_{i_\star} gives

di⋆EyπB[Tn]=EyπB[(2y−1)(STn−b)]≤B+ℓi⋆−(2y−1)b.d_{i_\star}\mathbb{E}^{\pi_B}_y[T_n] = \mathbb{E}^{\pi_B}_y\left[(2y-1)(S_{T_n}-b)\right] \le B+\ell_{i_\star}-(2y-1)b.

Since Tn↑τBT_n \uparrow\tau_B, monotone convergence gives the same bound for di⋆EyπB[τB]d_{i_\star}\mathbb{E}^{\pi_B}_y[\tau_B]. In particular, τB\tau_B has finite expectation under both hypotheses.

Every query has cost ci⋆c_{i_\star}, and di⋆=Ici⋆d_{i_\star}=I c_{i_\star}. Averaging over the prior therefore yields

IC(πB)=di⋆(qE1πB[τB]+(1−q)E0πB[τB])≤B+ℓi⋆+(1−2q)b≤B+ℓi⋆,I C(\pi_B)=d_{i_\star}\left(q\mathbb{E}^{\pi_B}_1[\tau_B]+(1-q)\mathbb{E}^{\pi_B}_0[\tau_B]\right)\le B+\ell_{i_\star}+(1-2q)b\le B+\ell_{i_\star},

where the last inequality follows because b=log⁡(q/(1−q))b=\log(q/(1-q)) has the same sign as 2q−12q-1.

Since τB\tau_B is finite almost surely and ∣SτB∣≥B|S_{\tau_B}| \ge B, the posterior error bound above gives E(πB)≤e−B\mathbb{E}(\pi_B) \le e^{-B}. □

Together, the two lemmas bound the optimal error–cost tradeoff from both sides. The information lower bound applies to every adaptive circuit, while the likelihood-ratio test achieves the same leading rate using only an instrument maximizing IiI_i.

Lemma 3 (Optimal error and cost for a fixed prompt). Suppose 0<I<∞0 < I < \infty. Let EλE_\lambda and CλC_\lambda denote the error and expected cost of any circuit minimizing E(π)+λC(π)E(\pi)+\lambda C(\pi). As λ↓0\lambda\downarrow0,

Eλ=Θ(λ/I),(21)E_\lambda=\Theta(\lambda/I), \tag*{(21)}
ICλ=log⁡(I/λ)+O(1).(22)I C_\lambda=\log(I/\lambda)+O(1). \tag*{(22)}

Proof. For 0<λ<I0 < \lambda< I, set η=λ/I\eta=\lambda/I and choose B=log⁡(1/η)B=\log(1/\eta) in Lemma 2. By optimality,

Eλ+λCλ≤η(1+log⁡1η+ℓi⋆).(23)E_\lambda+\lambda C_\lambda\le\eta\left(1+\log\frac{1}{\eta}+\ell_{i_\star}\right). \tag*{(23)}

To bound the error separately, combine (23) with the information lower bound (19). Writing xλ=Eλ/ηx_\lambda=E_\lambda/\eta gives

xλ+log⁡m2ηxλ≤1+log⁡1η+ℓi⋆,x_\lambda+\log\frac{m}{2\eta x_\lambda}\le1+\log\frac{1}{\eta}+\ell_{i_\star},

or equivalently,

xλ−log⁡xλ≤1+ℓi⋆+log⁡2m.(24)x_\lambda-\log x_\lambda\le1+\ell_{i^\star}+\log\frac{2}{m}. \tag*{(24)}

For a fixed prompt, the right-hand side is constant. Since x−log⁡xx-\log x diverges as either x↓0x\downarrow0 or x→∞x\to\infty, the sequence xλx_\lambda is bounded above and away from zero. Hence Eλ=Θ(η)=Θ(λ/I)E_\lambda=\Theta(\eta)=\Theta(\lambda/I).

For the cost, Equation (19) and Equation (23) imply

log⁡m2Eλ≤ICλ≤1+log⁡1η+ℓi⋆.\log\frac{m}{2E_\lambda}\le IC_\lambda\le1+\log\frac{1}{\eta}+\ell_{i^\star}.

Since Eλ=Θ(η)E_\lambda=\Theta(\eta), the lower bound equals log⁡(1/η)+O(1)\log(1/\eta)+O(1) and we obtain our result. □\square

Proof of Theorem 1. Restoring the dependence on zz, Lemma 3 gives

−log⁡Eλ(z)=log⁡I(z)λ+O(1)=I(z)Cλ(z)+O(1).-\log E_\lambda(z)=\log\frac{I(z)}{\lambda}+O(1)=I(z)C_\lambda(z)+O(1).

□\square

Allocation of queries. The same information bound also characterizes which instruments an optimal circuit uses. In particular, any expected cost spent on instruments with strictly lower information rates remains bounded as λ↓0\lambda\downarrow0.

Proposition 5 (Asymptotic query allocation). Fix a prompt with 0<I<∞0<I<\infty, and let Nˉiλ\bar{N}_i^\lambda be the expected number of queries to instrument ii under an optimal circuit. Then

∑i:Ii<IciNˉiλ=O(1).\sum_{i:I_i<I}c_i\bar{N}_i^\lambda=O(1).

Consequently,

∑i:Ii<IciNˉiλCλ⟶0as λ↓0.\frac{\sum_{i:I_i<I}c_i\bar{N}_i^\lambda}{C_\lambda}\longrightarrow0\qquad\text{as }\lambda\downarrow0.

Proof. By Equation (19),

∑idiNˉiλ≥log⁡m2Eλ.\sum_i d_i\bar{N}_i^\lambda\ge\log\frac{m}{2E_\lambda}.

Using di=Iicid_i=I_ic_i and Cλ=∑iciNˉiλC_\lambda=\sum_i c_i\bar{N}_i^\lambda, we obtain

∑i(I−Ii)ciNˉiλ=ICλ−∑idiNˉiλ≤ICλ−log⁡m2Eλ=O(1),(25)\sum_i(I-I_i)c_i\bar{N}_i^\lambda=IC_\lambda-\sum_i d_i\bar{N}_i^\lambda\le IC_\lambda-\log\frac{m}{2E_\lambda}=O(1), \tag*{(25)}

where the last equality follows from Lemma 3.

If any instrument has Ii<II_i<I, define Δ=min⁡i:Ii<I(I−Ii)>0\Delta=\min_{i:I_i<I}(I-I_i)>0. Equation (25) then gives

Δ∑i:Ii<IciNˉiλ≤O(1),\Delta\sum_{i:I_i<I}c_i\bar{N}_i^\lambda\le O(1),

so the total expected cost allocated to strictly suboptimal instruments is O(1)O(1). If every instrument attains II, the sum is zero. Finally, Equation (22) implies Cλ→∞C_\lambda\to\infty. Dividing the bounded suboptimal expenditure by the total expected cost proves the result. □\square

Proof of the Population Scaling Law

We now consider the prompt distribution Z∼PZZ \sim P_Z. To extend the fixed-prompt results to a population, we need bounds that hold uniformly across prompts.

Assumption 2 (Uniformity across prompts). There exist constants m0,cmax⁡>0m_0, c_{\max} > 0 such that, for almost every prompt zz,

min⁡{qz(∅),1−qz(∅)}≥m0,min⁡i:Ii(z)=I(z)ci(z)≤cmax⁡.(26)\min\{q_z(\varnothing),1-q_z(\varnothing)\}\ge m_0,\qquad\min_{i:I_i(z)=I(z)}c_i(z)\le c_{\max}. \tag*{(26)}

The first condition ensures that the error probability before querying is bounded away from zero, while the second ensures that at least one instrument with the highest information rate has bounded query cost.

We first establish uniform error bounds and use them to prove the population scaling law. We then obtain the finer error–cost rates under the polynomial tail assumption.

Lemma 4 (Uniform fixed-prompt error bounds). Under Assumption 2, for every fixed r0>0r_0>0, there exist positive constants, independent of zz and λ\lambda, such that, for almost every zz and all sufficiently small λ\lambda,

Eλ(z)≍min⁡{1,λI(z)},0<I(z)≤r0,(27)E_\lambda(z)\asymp\min\left\{1,\frac{\lambda}{I(z)}\right\},\qquad0<I(z)\le r_0, \tag*{(27)}
Eλ(z)=O(λ),I(z)>r0.(28)E_\lambda(z)=O(\lambda),\qquad I(z)>r_0. \tag*{(28)}

Proof. For each prompt, choose an information-rate-maximizing instrument i⋆i^\star with ci⋆(z)≤cmax⁡c_{i^\star}(z)\le c_{\max}. We first bound the overshoot term appearing in Equation (24).

For ai∈[1/2,1)a_i\in[1/2,1), we have

ℓi−di=2(1−ai)log⁡ai1−ai≤2(1−ai)log⁡11−ai≤2e,\ell_i-d_i=2(1-a_i)\log\frac{a_i}{1-a_i}\le2(1-a_i)\log\frac{1}{1-a_i}\le\frac{2}{e},

where the last inequality uses ulog⁡(1/u)≤1/eu\log(1/u)\le1/e for u∈(0,1)u\in(0,1). Since di⋆=I(z)ci⋆(z)d_{i^\star}=I(z)c_{i^\star}(z),

ℓi⋆(z)≤2e+cmax⁡I(z).(29)\ell_{i^\star}(z)\le\frac{2}{e}+c_{\max}I(z). \tag*{(29)}

Suppose first that λ<I(z)≤r0\lambda<I(z)\le r_0. By Equation (24),

xλ−log⁡xλ≤1+2e+cmax⁡r0+log⁡2m0.x_\lambda-\log x_\lambda\le1+\frac{2}{e}+c_{\max}r_0+\log\frac{2}{m_0}.

The right-hand side is uniform over this range. As x−log⁡xx-\log x diverges at both zero and infinity, xλx_\lambda lies in a fixed compact interval inside (0,∞)(0,\infty). Consequently,

Eλ(z)≍λI(z)E_\lambda(z)\asymp\frac{\lambda}{I(z)}

if λ<I(z)≤r0\lambda<I(z)\le r_0.

Now suppose 0<I(z)≤λ0<I(z)\le\lambda. Stopping without querying gives error at most 1/21/2, so optimality implies

λCλ(z)≤12,I(z)Cλ(z)≤12.\lambda C_\lambda(z)\le\frac{1}{2},\qquad I(z)C_\lambda(z)\le\frac{1}{2}.

The information lower bound Equation (19) therefore yields Eλ(z)≥m02eE_\lambda(z)\ge\frac{m_0}{2\sqrt{e}}. Since Eλ(z)≤1/2E_\lambda(z)\le1/2, the error is bounded above and below by positive constants. Combining the two ranges proves Equation (27).

Finally, consider I(z)>r0I(z)>r_0. For sufficiently small λ\lambda, we have λ<I(z)\lambda<I(z). By Equation (24) and Equation (29),

xλ−log⁡xλ≤K0+cmax⁡I(z),K0=1+2e+log⁡2m0.x_\lambda-\log x_\lambda\le K_0+c_{\max}I(z),\qquad K_0=1+\frac{2}{e}+\log\frac{2}{m_0}.

Since x−log⁡x≥x/2x-\log x\ge x/2 for every x>0x>0,

xλ≤2K0+2cmax⁡I(z).x_\lambda\le2K_0+2c_{\max}I(z).

It follows that

Eλ(z)=λxλI(z)≤2λ(K0r0+cmax⁡),E_\lambda(z)=\frac{\lambda x_\lambda}{I(z)}\le2\lambda\left(\frac{K_0}{r_0}+c_{\max}\right),

proving Equation (28).

Proof of Theorem 2. Write Lλ=log⁡(1/λ)L_\lambda=\log(1/\lambda). For each fixed prompt zz with 0<I(z)<∞0<I(z)<\infty, Lemma 4 gives Eλ(z)=Θ(λ)E_\lambda(z)=\Theta(\lambda). Hence

−log⁡Eλ(z)Lλ⟶1.\frac{-\log E_\lambda(z)}{L_\lambda}\longrightarrow1.

Combining this with Theorem 1 yields

Cλ(z)Lλ⟶1I(z).(30)\frac{C_\lambda(z)}{L_\lambda}\longrightarrow\frac{1}{I(z)}. \tag*{(30)}

To average this limit across prompts, we first establish a uniform bound on the expected cost.

For each zz, choose an instrument i⋆i^\star attaining I(z)I(z) with ci⋆(z)≤cmax⁡c_{i^\star}(z)\le c_{\max}. Consider the sequential likelihood-ratio test that repeatedly queries i⋆i^\star and stops when the log-likelihood ratio exits (−B,B)(-B,B). We have established that this test has error probability at most e−Be^{-B} and expected query cost at most

B+ℓi⋆(z)I(z).\frac{B+\ell_{i^\star}(z)}{I(z)}.

Setting B=LλB=L_\lambda and using optimality gives

Eλ(z)+λCλ(z)≤λ+λLλ+ℓi⋆(z)I(z).E_\lambda(z)+\lambda C_\lambda(z)\le\lambda+\lambda\frac{L_\lambda+\ell_{i^\star}(z)}{I(z)}.

By Equation (29), ℓi⋆(z)≤2/e+cmax⁡I(z)\ell_{i^\star}(z)\le2/e+c_{\max}I(z). Consequently,

0≤Cλ(z)Lλ≤(1+2eLλ)1I(z)+1+cmax⁡Lλ.(31)0\le\frac{C_\lambda(z)}{L_\lambda}\le\left(1+\frac{2}{eL_\lambda}\right)\frac{1}{I(z)}+\frac{1+c_{\max}}{L_\lambda}. \tag*{(31)}

If E[1/I(Z)]<∞\mathbb{E}[1/I(Z)]<\infty, the right-hand side provides an integrable bound for sufficiently small λ\lambda. Dominated convergence and Equation (30) therefore yield

lim⁡λ→0CλLλ=E[1I(Z)].\lim_{\lambda\to0}\frac{C_\lambda}{L_\lambda}=\mathbb{E}\left[\frac{1}{I(Z)}\right].

If E[1/I(Z)]=∞\mathbb{E}[1/I(Z)]=\infty, Fatou’s lemma gives

lim inf⁡λ→0CλLλ≥E[1I(Z)]=∞.\liminf_{\lambda\to0}\frac{C_\lambda}{L_\lambda}\ge\mathbb{E}\left[\frac{1}{I(Z)}\right]=\infty.

Thus, in either case,

lim⁡λ↓0CλLλ=E[1I(Z)],(32)\lim_{\lambda\downarrow0}\frac{C_\lambda}{L_\lambda}=\mathbb{E}\left[\frac{1}{I(Z)}\right], \tag*{(32)}

where the limit is interpreted as infinite when the expectation diverges.

We next bound the population error. Since 0<I(Z)<∞0<I(Z)<\infty almost surely, there exist 0<a<b<∞0<a<b<\infty such that

p:=Pr⁡(a≤I(Z)≤b)>0.p:=\Pr(a\le I(Z)\le b)>0.

By Lemma 4, for sufficiently small λ\lambda and some constant k>0k>0,

Eλ≥E[Eλ(Z)1{a≤I(Z)≤b}]≥kpbλ.E_\lambda\ge\mathbb{E}\left[E_\lambda(Z)\mathbf{1}_{\{a\le I(Z)\le b\}}\right]\ge\frac{kp}{b}\lambda.

Therefore,

lim sup⁡λ↓0−log⁡EλLλ≤1.(33)\limsup_{\lambda\downarrow0}\frac{-\log E_\lambda}{L_\lambda}\le1. \tag*{(33)}

Suppose first that E[1/I(Z)]<∞\mathbb{E}[1/I(Z)]<\infty. For any fixed r0>0r_0>0, Lemma 4 gives

Eλ≤KλE[1I(Z)1{I(Z)≤r0}]+K′λ=O(λ),E_\lambda\le K\lambda\mathbb{E}\left[\frac{1}{I(Z)}\mathbf{1}_{\{I(Z)\le r_0\}}\right]+K'\lambda=O(\lambda),

for constants K,K′>0K,K'>0. Together with the lower bound above, this implies Eλ=Θ(λ)E_\lambda=\Theta(\lambda), and hence

−log⁡EλLλ⟶1.\frac{-\log E_\lambda}{L_\lambda}\longrightarrow1.

Combining this with Equation (32), we obtain

lim⁡λ↓0−log⁡EλCλ=(E[1I(Z)])−1.\lim_{\lambda\downarrow0}\frac{-\log E_\lambda}{C_\lambda}=\left(\mathbb{E}\left[\frac{1}{I(Z)}\right]\right)^{-1}.

Finally, if E[1/I(Z)]=∞\mathbb{E}[1/I(Z)]=\infty, Equation (32) gives Cλ/Lλ→∞C_\lambda/L_\lambda\to\infty, while Equation (33) shows that −log⁡Eλ/Lλ-\log E_\lambda/L_\lambda remains bounded. Therefore,

lim⁡λ↓0−log⁡EλCλ=0,\lim_{\lambda\downarrow0}\frac{-\log E_\lambda}{C_\lambda}=0,

completing the proof. □

We now prove Corollary 1. The tail assumption allows us to determine the orders of both the population error and expected cost. We first establish an identity that recovers expected cost from the optimal error.

Lemma 5 (Recovering cost from error). Let Π\Pi be a fixed policy class containing a zero-cost policy with error at most 1/21/2. For each λ>0\lambda>0, let (Eλ,Cλ)(E_\lambda,C_\lambda) denote the error and cost of a policy minimizing E(π)+λC(π)E(\pi)+\lambda C(\pi) over Π\Pi. Then

Cλ=∫λ∞Ess2 ds−Eλλ.(34)C_\lambda=\int_\lambda^\infty\frac{E_s}{s^2}\,ds-\frac{E_\lambda}{\lambda}. \tag*{(34)}

If minimizers are not unique, the identity holds for any measurable selection.

Proof. Let

V(λ)=min⁡π∈Π{E(π)+λC(π)}.V(\lambda)=\min_{\pi\in\Pi}\{\mathbb{E}(\pi)+\lambda C(\pi)\}.

As the infimum of affine functions of λ\lambda, VV is concave and hence differentiable almost everywhere. At every point of differentiability, the cost of any optimal policy equals V′(λ)V'(\lambda). Therefore,

ddλV(λ)λ=λCλ−V(λ)λ2=−Eλλ2.\frac{d}{d\lambda}\frac{V(\lambda)}{\lambda}=\frac{\lambda C_\lambda-V(\lambda)}{\lambda^2}=-\frac{E_\lambda}{\lambda^2}.

The zero-cost policy gives 0≤V(λ)≤1/20\le V(\lambda)\le1/2, so V(λ)/λ→0V(\lambda)/\lambda\to0 as λ→∞\lambda\to\infty. Integrating from λ\lambda to infinity gives

V(λ)λ=∫λ∞Ess2 ds.\frac{V(\lambda)}{\lambda}=\int_\lambda^\infty\frac{E_s}{s^2}\,ds.

Substituting V(λ)=Eλ+λCλV(\lambda)=E_\lambda+\lambda C_\lambda proves Equation (34).

Proof of Corollary 1. Let D=1/I(Z)D=1/I(Z) denote the inverse information rate. The assumed lower tail Pr⁡(I(Z)≤r)≍rα\Pr(I(Z)\le r)\asymp r^\alpha implies

Pr⁡(D>t)≍t−α(t→∞).(35)\Pr(D>t)\asymp t^{-\alpha}\qquad(t\to\infty). \tag*{(35)}

By the tail-integral formula,

E[D]=∫0∞Pr⁡(D>t) dt.\mathbb{E}[D]=\int_0^\infty\Pr(D>t)\,dt.

Thus, E[D]<∞\mathbb{E}[D]<\infty if and only if α>1\alpha>1. In this case, the exponential error-decay rate follows directly from Theorem 2. To obtain the finer scaling for the remaining cases, we first determine the population error.

Choose d0>0d_0>0 sufficiently large that the tail bounds in Equation (35) hold for t≥d0t\ge d_0, and define

Mλ=E[min⁡{1,λD}1{D≥d0}].M_\lambda=\mathbb{E}\left[\min\{1,\lambda D\}\mathbf{1}_{\{D\ge d_0\}}\right].

By Lemma 4, the expected error over prompts with D≥d0D\ge d_0 is of order MλM_\lambda, while prompts with D<d0D<d_0 contribute at most O(λ)O(\lambda). Since

Mλ≥λd0Pr⁡(D≥d0)=Ω(λ)M_\lambda\ge\lambda d_0\Pr(D\ge d_0)=\Omega(\lambda)

for sufficiently small λ\lambda, the latter contribution does not change the asymptotic order. Hence Eλ≍MλE_\lambda\asymp M_\lambda.

For λ<1/d0\lambda<1/d_0, the tail-integral formula gives

Mλ=λd0Pr⁡(D≥d0)+λ∫d01/λPr⁡(D>t) dt≍λ(1+∫d01/λt−α dt).M_\lambda=\lambda d_0\Pr(D\ge d_0)+\lambda\int_{d_0}^{1/\lambda}\Pr(D>t)\,dt\asymp\lambda\left(1+\int_{d_0}^{1/\lambda}t^{-\alpha}\,dt\right).

Evaluating the integral yields

Eλ≍{λ,α>1,λlog⁡(1/λ),α=1,λα,0<α<1.(36)E_\lambda\asymp \begin{cases} \lambda, & \alpha>1,\\ \lambda\log(1/\lambda), & \alpha=1,\\ \lambda^\alpha, & 0<\alpha<1. \end{cases} \tag*{(36)}

We next determine the expected cost. For α>1\alpha>1, Equation (32) already establishes

Cλ∼E[D]log⁡(1/λ).C_\lambda\sim\mathbb{E}[D]\log(1/\lambda).

If α=1\alpha= 1, then Es≍slog⁡(1/s)E_s \asymp s \log(1/s) near zero. Consequently,

∫λ∞Ess2 ds≍∫λs0log⁡(1/s)s ds≍(log⁡(1/λ))2,\int_{\lambda}^{\infty} \frac{E_s}{s^2}\, \mathrm{d}s \asymp\int_{\lambda}^{s_0} \frac{\log(1/s)}{s}\, \mathrm{d}s \asymp\left(\log(1/\lambda)\right)^2,

where s0>0s_0 > 0 is fixed. Since Eλ/λ=Θ(log⁡(1/λ))E_\lambda/\lambda= \Theta(\log(1/\lambda)) is of lower order, Equation (34) gives

Cλ≍(log⁡(1/λ))2.C_\lambda\asymp\left(\log(1/\lambda)\right)^2.

Finally, suppose 0<α<10 < \alpha< 1. Since Es≍sαE_s \asymp s^\alpha, Equation (34) gives

Cλ=O(λα−1).C_\lambda= O(\lambda^{\alpha-1}).

For the matching lower bound, choose constants u,v>0u, v > 0 such that, for sufficiently small λ\lambda,

uλα≤Eλ≤vλα.u\lambda^\alpha\leq E_\lambda\leq v\lambda^\alpha.

Fix a>1a > 1 large enough that uaα>vua^\alpha> v. Optimality of the circuit at price aλa\lambda implies

Eaλ+aλCaλ≤Eλ+aλCλ.E_{a\lambda} + a\lambda C_{a\lambda} \leq E_\lambda+ a\lambda C_\lambda.

Therefore,

Cλ≥Eaλ−Eλaλ≥uaα−vaλα−1.C_\lambda\geq\frac{E_{a\lambda}-E_\lambda}{a\lambda} \geq\frac{ua^\alpha-v}{a}\lambda^{\alpha-1}.

Hence Cλ≍λα−1C_\lambda\asymp\lambda^{\alpha-1}.

Combining the three cases,

Cλ≍{log⁡(1/λ),α>1,(log⁡(1/λ))2,α=1,λα−1,0<α<1.(37)C_\lambda\asymp \begin{cases} \log(1/\lambda), & \alpha> 1, \\ \left(\log(1/\lambda)\right)^2, & \alpha= 1, \\ \lambda^{\alpha-1}, & 0 < \alpha< 1. \end{cases} \tag*{(37)}

It remains to eliminate λ\lambda between Equation (36) and Equation (37).

For α>1\alpha> 1, Theorem 2 gives

lim⁡λ→0−log⁡EλCλ=1E[D].\lim_{\lambda\to0} \frac{-\log E_\lambda}{C_\lambda} = \frac{1}{\mathbb{E}[D]}.

For α=1\alpha= 1,

−log⁡Eλ=log⁡(1/λ)−log⁡log⁡(1/λ)+O(1)≍Cλ.-\log E_\lambda= \log(1/\lambda) - \log\log(1/\lambda) + O(1) \asymp\sqrt{C_\lambda}.

For 0<α<10 < \alpha< 1,

Eλ≍λα≍Cλ−α/(1−α).E_\lambda\asymp\lambda^\alpha\asymp C_\lambda^{-\alpha/(1-\alpha)}.

These establish the three regimes in Corollary 1.

Additional Experimental Details

Data

In this work, we consider the following seven classification datasets. AgentJudgeBench [68]. Each prompt is a user request, the tools available to an AI agent and the tool calls the agent planned, together with one of two criteria: whether the plan uses exactly the right set of tools (tool selection), or whether it covers every part of the request (query coverage). An instrument answers whether the plan fully meets the criterion without seeing the reference plan. We draw 1,000 requests, giving 20,850 prompts. The seven instruments are: Prometheus-2-7B, gpt-oss-20b, gpt-oss-120b, QwQ-32B, Gemini-2.5-Pro, GPT-5.4 and Claude Sonnet 4.5.

BeaverTails [36]. Each prompt is a user request together with a model’s response. An instrument answers whether the response is unsafe under the benchmark’s harm categories.

BIG-Bench Hard [62]. Each prompt is a question with two answer options from one of eight tasks (boolean expressions, causal judgement, formal fallacies, hyperbaton, navigate, snarks, sports understanding and web of lies), with 1,865 questions in total. An instrument answers which option is correct. The 46 instruments are models from 0.36B to 123B parameters.

Decision Index [51]. Each prompt is a request from one of six binary decision tasks: whether clinical-trial reports entail a statement (NLI4CT), a yes/no question about cause and effect (CLadder), which of two limericks is the original (BPoMP), whether Wikipedia introductions support a multi-hop claim (HoVer), whether a tweet is sarcastic (iSarcasmEval) and which of two headline edits is funnier (Humicroedit), with 24,928 questions in total. The 18 instruments are entrants of the Decision Index leaderboard, from 0.4B to 36B parameters.

HealthBench [4]. Each prompt is a conversation between a user and an AI assistant together with one rubric criterion written by physicians, and an instrument answers whether the last response meets the criterion, using the benchmark’s grading instruction. Each prompt was judged by two to five physicians. Therefore, the target is the probability that a randomly chosen physician judges the criterion met. The eight instruments are configurations of three reasoning models: Granite 4.2 3B with reasoning off, low and default, and gpt-oss-20b (low, medium and high effort) and gpt-oss-120b (low and medium effort) [1].

LMArena [15]. Each prompt is a user message together with two responses and an instrument answers which response is more likely to be preferred (A or B). The order in which the two responses are shown is drawn at random for every run of the prompt. We consider 34,952 single-turn English comparisons.

PolitiCAUSE [23]. Each prompt is one sentence from political text, and an instrument answers whether the sentence states a causal relation.

On BeaverTails, HealthBench, LMArena and PolitiCAUSE, we query the instruments ourselves. Each instrument is given the dataset’s official classification instruction, with the model’s default decoding (unless we state otherwise). BeaverTails, LMArena and PolitiCAUSE share seven instruments: Gemma 4 [64] (E2B, E4B, 26B-A4B and 31B, instruction-tuned) and Qwen3 [75] (4B, 8B and 32B). At their defaults, the Gemma models answer directly and the Qwen3 models reason before answering. On AgentJudgeBench, BIG-Bench Hard and Decision Index, we use the answers each benchmark recorded. No call costs were recorded there, so on BIG-Bench Hard and Decision Index a call costs $0.01 per million tokens per billion active parameters, and on AgentJudgeBench we price each call at the model’s API price with the prompt’s input tokens and an assumed output length.

Each dataset is split into five folds, keeping prompts that share a source together. Each fold serves once as the test set, and the remaining prompts are divided into training and validation.

Embedding model. The embedding ϕ(z)\phi(z) is built from Qwen3-Embedding-0.6B (Yang et al., [75]) with last-token pooling, which gives a 1024-dimensional vector per text.

Training the Probabilistic Model

Architecture. The model fθf_\theta assumes that each prompt belongs to one of TT latent types. It first maps the embedding to h(z)=Wϕ(z)∈R64h(z) = W\phi(z) \in\mathbb{R}^{64} with a learned matrix WW, and three networks take h(z)h(z) as input:

  1. a type head, a two-layer network of width 128, gives the probability ρt(z)\rho_t(z) that zz is of type tt;

  2. a measurement head, a two-layer network of width 128, gives the probability μit(z)\mu_{it}(z) that instrument ii returns 1 on a prompt of type tt;

  3. a label head gives the probability ηt(z,s)\eta_t(z,s) that Y=1Y = 1 on a prompt of type tt at state ss. It encodes each observed pair (j,x)∈s(j,x) \in s as g(ej+ex′+et′′)g(e_j + e'_x + e''_t), where eje_j, ex′e'_x and et′′e''_t are learned 64-dimensional embeddings of the instrument, the measurement and the type, and gg is a two-layer network. It sums these vectors over the state and passes [h(z)+et′′,∑(j,x)∈sg(ej+ex′+et′′),∣M(s)∣/n]\left[h(z) + e''_t, \sum_{(j,x)\in s} g(e_j + e'_x + e''_t), |\mathcal{M}(s)|/n\right] through a two-layer network of width 128 and a linear output.

The measurements observed so far update the type by Bayes’ rule,

wt(z,s)∝ρt(z)∏(j,x)∈sμjt(z)x(1−μjt(z))1−x,(38)w_t(z,s) \propto\rho_t(z) \prod_{(j,x)\in s} \mu_{jt}(z)^x \left(1-\mu_{jt}(z)\right)^{1-x}, \tag*{(38)}

and the outputs of fθf_\theta average over the types:

ηθ(ϕ(z),s)=∑twt(z,s)ηt(z,s),νθ(i)(ϕ(z),s)=∑twt(z,s)μit(z).(39)\eta_\theta(\phi(z),s) = \sum_t w_t(z,s)\eta_t(z,s), \qquad\nu_\theta^{(i)}(\phi(z),s) = \sum_t w_t(z,s)\mu_{it}(z). \tag*{(39)}

Loss function. For a prompt zz, let D(z)\mathcal{D}(z) be the multiset of its recorded measurements (i,x)(i,x), and let a1(z)a_1(z) and a0(z)a_0(z) be the numbers of annotations that give label 1 and label 0. For a hard label, ay(z)(z)=1a_{y(z)}(z) = 1 and the other count is zero. On HealthBench, they count the physicians who judged the criterion met and not met. For each prompt zz in a minibatch BB we draw a state szs_z: a number kk uniformly from {0,…,n}\{0,\ldots,n\}, a uniformly random set M(sz)\mathcal{M}(s_z) of kk instruments, and for each i∈M(sz)i \in\mathcal{M}(s_z) one of its recorded measurements on zz, chosen uniformly. With ℓ(p,x)=−xlog⁡p−(1−x)log⁡(1−p)\ell(p,x) = -x \log p - (1-x)\log(1-p) the binary cross-entropy, the label loss is

LY(θ)=∑z∈B[a1(z)ℓ(ηθ(ϕ(z),sz),1)+a0(z)ℓ(ηθ(ϕ(z),sz),0)]∑z∈B(a0(z)+a1(z)).(40)\mathcal{L}_Y(\theta) = \frac{\sum_{z\in B}\left[a_1(z)\ell\left(\eta_\theta(\phi(z),s_z),1\right)+a_0(z)\ell\left(\eta_\theta(\phi(z),s_z),0\right)\right]} {\sum_{z\in B}\left(a_0(z)+a_1(z)\right)}. \tag*{(40)}

For the measurement loss we draw a second state sz′s'_z in the same way and score the prediction for every unqueried instrument against all its recorded measurements,

LX(θ)=1∣B∣∑z∈B1N′(z)∑(i,x)∈D(z),i∉M(sz′)ℓ(νθ(i)(ϕ(z),sz′),x),(41)\mathcal{L}_X(\theta) = \frac{1}{|B|}\sum_{z\in B}\frac{1}{N'(z)}\sum_{\substack{(i,x)\in\mathcal{D}(z),\\ i\notin\mathcal{M}(s'_z)}} \ell\left(\nu_\theta^{(i)}(\phi(z),s'_z),x\right), \tag*{(41)}

where N′(z)N'(z) is the number of terms in the inner sum. We minimize LY+LX\mathcal{L}_Y+\mathcal{L}_X.

Optimization and calibration. We use AdamW (learning rate 2×10−32 \times10^{-3}, weight decay 10−410^{-4}, batch size 256) with a cosine schedule over at most 300 epochs. After each epoch we compute LY\mathcal{L}_Y on a fixed set of validation states, one per validation prompt, keep the epoch where it is lowest, and stop after 30 epochs without improvement. We train one model for each T∈{1,2,4,8}T \in\{1,2,4,8\} and keep the one with the lowest validation loss LY+LX\mathcal{L}_Y + \mathcal{L}_X. We then fit one temperature per head on ten states per validation prompt, η=σ(uη/τη)\eta= \sigma(u_\eta/\tau_\eta) and ν(i)=σ(uν,i/τν)\nu^{(i)} = \sigma(u_{\nu,i}/\tau_\nu) on the logits uu, each chosen to minimize the validation negative log-likelihood. We train five models per fold by changing the random seed.

Generating circuits. We solve Equation (7) with qZ(s)=ηθ(ϕ(z),s)q_Z(s) = \eta_\theta(\phi(z),s) and with νθ(i)(ϕ(z),s)\nu_\theta^{(i)}(\phi(z),s) in place of QX∣Z[i](1∣z)Q_{X|Z}^{[i]}(1 \mid z). On BIG-Bench Hard and Decision Index, which have a large set of instruments, we keep the eight whose per-task cost–accuracy frontier on the training and validation prompts comes closest to that of all instruments. The program uses a cost known before the query. On the datasets we query ourselves, ci(z)c_i(z) is the input tokens of zz times the input price plus the 95th percentile of instrument ii’s output length on training prompts times the output price. On the other three, ci(z)c_i(z) is instrument ii’s mean recorded cost on training prompts. Costs are divided by the mean cost of the most expensive instrument. We solve Equation (7) for λ=0\lambda= 0, for 201 values of λ\lambda spaced geometrically from 10−810^{-8} to 10210^2, and for λ=∞\lambda= \infty (the zero-cost prediction). For each budget, we choose the λ\lambda with the best validation metric among those whose mean validation cost is within the budget and report its test metric. Since some benchmarks have class imbalance, we always optimize for balanced accuracy and report balanced accuracy in our plots.

Full list of baselines

  • Predictive routers: ROUTELLM ([54]), HYBRIDLLM ([21]), and CARROT ([60]).

  • Model cascades: CASCADE ROUTING ([19]), CONSISTENCY CASCADES ([76]), CASCADED SELECTIVE ([37]), and FRUGALGPT ([13]).

  • Sampling methods: ADAPTIVE-CONSISTENCY ([2]).

RouteLLM and Consistency Cascade route between two instruments, and we sweep every pair. Since HYBRIDLLM fine-tunes a separate router for each pair of instruments, we use a single pair: the cheapest instrument and the most accurate of the costlier instruments on the training prompts.

Additional Experimental Results

In this subsection, we provide additional experimental analysis of our learned circuits.

Calibration of the circuit’s beliefs.

Our method returns a predictive distribution over the ground truth label, allowing us to assess whether its confidence is well calibrated. The circuit’s belief at the terminal state sTs_T is p^(z)=ηθ(ϕ(z),sT)\hat{p}(z) = \eta_\theta(\phi(z),s_T). We evaluate these probabilities with the Brier score

Brier⁡(π)=EZEX∣Z[(p^(Z)−Y)2],(42)\operatorname{Brier}(\pi) = \mathbb{E}_Z\mathbb{E}_{X \mid Z}\left[(\hat{p}(Z)-Y)^2\right], \tag*{(42)}

which is small only when the forecasts are both calibrated and sharp. A forecast is calibrated if, among the prompts forecast at probability pp, a fraction pp have label 1. While the dynamic program never optimizes for the Brier metric directly, it attains very competitive calibration across different budgets; see Figure 6. We also report the calibration of majority voting, treating the fraction of the instruments’ calls that give label 1 as its forecast probability.

Joint distribution between instruments

To test the assumption that LLMs are conditionally independent given any prompt (Assumption 1), we queried eight instruments 100 times each on 20 prompts from BeaverTails. For any two instruments, we compute the Pearson correlation ϕ\phi of their answers in two ways: within a prompt, across its 100 calls, averaged over prompts; and across prompts, pooling all 2,000 calls. Under the assumption, the first is zero, while the second can be large because the instruments share the (latent) difficulty of each prompt. To test the joint independence of all eight instruments on a prompt, we use their total correlation

TC⁡(z)=∑iH(X(i)∣Z=z)−H(X(1),…,X(8)∣Z=z),(43)\operatorname{TC}(z) = \sum_i H\left(X^{(i)} \mid Z = z\right) - H\left(X^{(1)}, \ldots, X^{(8)} \mid Z = z\right), \tag*{(43)}

estimated in bits from the 100 calls, which is zero if and only if the answers are jointly independent. We compare it with a permutation null that shuffles each instrument’s calls independently, preserving each instrument’s answer frequencies while removing any dependence between instruments. We present these results in Figure 7.

Given labeled data, the fitted model treats measurements as independent given the embedding and the latent type (Section C.2). Since the type is unobserved, a measurement changes the model’s prediction of the other instruments’ measurements, so the model can represent dependence between instruments that the embedding alone does not explain. We use this joint to estimate correlations on the test prompts, without the true labels, and compare them with those measured from the recorded calls and labels on the same set of prompts, pooling the five test folds. Write νx(i)(ϕ,s)\nu_x^{(i)}(\phi,s) for the model’s probability that instrument ii answers xx at state ss (so ν1(i)=ν1(i)\nu_1^{(i)}=\nu_1^{(i)}), ηx(ϕ,s)\eta_x(\phi,s) for its belief that Y=xY=x at state ss, and ei=1{X(i)≠Y}e_i=1\{X^{(i)}\ne Y\} for the error of instrument ii. Then

Pr⁡(X(i)=x,X(j)=x′∣ϕ)=νx(i)(ϕ,∅)νx′(j)(ϕ,{(i,x)}),Pr⁡(ei=1∣ϕ)=∑xνx(i)(ϕ,∅)[1−ηx(ϕ,{(i,x)})],Pr⁡(ei=ej=1∣ϕ)=∑xPr⁡(X(i)=x,X(j)=x∣ϕ)[1−ηx(ϕ,{(i,x),(j,x)})].(44)\begin{aligned} \Pr\left(X^{(i)}=x,X^{(j)}=x'\mid\phi\right)&=\nu_x^{(i)}(\phi,\varnothing)\nu_{x'}^{(j)}\left(\phi,\{(i,x)\}\right),\\ \Pr(e_i=1\mid\phi)&=\sum_x\nu_x^{(i)}(\phi,\varnothing)\left[1-\eta_x\left(\phi,\{(i,x)\}\right)\right],\\ \Pr(e_i=e_j=1\mid\phi)&=\sum_x\Pr\left(X^{(i)}=x,X^{(j)}=x\mid\phi\right)\left[1-\eta_x\left(\phi,\{(i,x),(j,x)\}\right)\right]. \tag*{(44)} \end{aligned}

where two binary instruments err together only when they give the same answer. Averaging these moments over the test prompts gives the implied correlations, and the implied accuracy of instrument ii is 1−E[Pr⁡(ei=1∣ϕ)]1-\mathbb{E}[\Pr(e_i=1\mid\phi)].

Consider Figure 8. The fitted model recovers nearly all of the dependence between instruments: on six of the seven tasks, the implied correlations of answers and of errors are within 4% of the measured ones on average, and on BIG-Bench Hard, the smallest task, they are 83% to 85% of them. The model also predicts each instrument’s accuracy on unseen prompts with high fidelity: the predicted accuracies are within 1.2 points of the measured ones on average on five tasks, 2.1 points on AgentJudgeBench and 3.6 points on BIG-Bench Hard.

How does the circuit allocate its queries?

We first examine how the value of additional queries depends on the reliability of the instrument pool for a given prompt; see Figure 9. For a given prompt zz, let r(z)r(z) be the proportion of the recorded calls of the instruments the circuit can query that return the label y(z)y(z). A prompt is (1) all right if r(z)=1r(z)=1, (2) noisy if 1/2<r(z)<11/2<r(z)<1, (3) mostly wrong if 0<r(z)≤1/20<r(z)\le1/2, and (4) all wrong if r(z)=0r(z)=0. On HealthBench, prompts on which the physicians disagree are left out. On noisy prompts, correct measurements occur more often, so additional queries provide further evidence for the correct label. On mostly wrong and all wrong prompts, the measurements favor the incorrect label, so additional queries to the same pool tend to reinforce the error.

We next examine how the circuit allocates those measurements across prompts; see Figure 10. Let s(z)s(z) be the fraction of these recorded calls on zz that return 1, and define the instruments’ disagreement by d(z)=1−∣2s(z)−1∣d(z) = 1 - \lvert2s(z) - 1\rvert, so that d(z)=0d(z) = 0 when all calls agree and d(z)=1d(z) = 1 when they split evenly. We also record whether the embedding prediction 1{ηθ(ϕ(z),s0)≥1/2}\mathbf{1}\{\eta_\theta(\phi(z), s_0) \ge1/2\} agrees with the majority of these calls. The dynamic program tends to continue querying when its posterior belief remains near 1/21/2, where an additional measurement can still change the optimal prediction. Agreement between the embedding and subsequent measurements typically moves the belief away from 1/21/2 and shortens the circuit, whereas disagreement induces further queries.

Lastly, § 4.2 shows that, as λ↓0\lambda\downarrow0, the optimal circuit concentrates its queries on instruments with the largest prompt-specific information rate Ii(z)=di(z)/ci(z)I_i(z) = d_i(z)/c_i(z). Since di(z)d_i(z) depends on the unknown label, we construct an analogue using the fitted model that measures the information provided by the first query to instrument ii, beyond the embedding, per unit cost:

r^i=I^icˉi,I^i=Eθ^[h(ηθ(ϕ(Z),s0))−h(ηθ(ϕ(Z),{(i,X(i))}))](45)\hat{r}_i = \frac{\hat{I}_i}{\bar{c}_i}, \qquad\hat{I}_i = \mathbb{E}_{\hat{\theta}}\left[h\left(\eta_\theta(\phi(Z),s_0)\right) - h\left(\eta_\theta\left(\phi(Z),\{(i,X^{(i)})\}\right)\right)\right] \tag*{(45)}

where h(p)=−plog⁡2p−(1−p)log⁡2(1−p)h(p) = -p\log_2 p - (1-p)\log_2(1-p), cˉi\bar{c}_i is the mean cost of instrument ii, and the expectation is taken over validation prompts, their recorded responses, and the fitted models of every fold and seed. Thus I^i\hat{I}_i estimates I(Y;X(i)∣ϕ(Z))I(Y;X^{(i)}\mid\phi(Z)) in bits, and r^i\hat{r}_i measures information per unit cost. Unlike Ii(z)I_i(z), however, r^i\hat{r}_i is averaged over prompts and describes only the value of the first query.

We define the entry point of instrument ii as the smallest expected cost per item at which the circuit makes at least 0.05 expected calls per prompt to ii. The Spearman rank correlation between entry order and r^i\hat{r}_i is 1.00, 0.96, and 0.79 on Arena, BeaverTails, and PolitiCAUSE, respectively. For comparison, the corresponding correlations are 0.75, 0.89, and 0.93 for inverse cost; 0.79, 0.21, and 0.71 for accuracy; and −0.79-0.79, 0.04, and 0.43 for low redundancy, measured as the negative mean error correlation with the remaining instruments.

Circuit Examples

We present two examples of circuits our method builds for individual test prompts at three values of λ\lambda; see Figure 13 and Figure 14. Using our trained probabilistic model, the prompt’s embedding gives the initial belief ηθ(ϕ(z),s0)\eta_\theta(\phi(z), s_0), shown as “Embedding @ pp%”. For each λ\lambda, the circuit either predicts immediately or asks an instrument. Each node is the state reached after that instrument’s answer, where pp% is the updated belief ηθ(ϕ(z),s)\eta_\theta(\phi(z), s) given every answer so far. Instrument names are colored by cost, from green (cheapest) to red. The highlighted path is the one followed with the prompt’s recorded answers. At stopping, the circuit predicts the label with belief above 1/21/2.

References

  1. [1]Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025.arxiv.org/abs/2508.10925
  2. [2]Pranjal Aggarwal, Aman Madaaan, Yiming Yang, et al. Let’s sample step by step: Adaptive-consistency for efficient reasoning and coding with llms. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12375–12396, 2023.arxiv.org/abs/2305.11860
  3. [3]Pranjal Aggarwal, Aman Madaaan, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappagant hu, Yiming Yang, et al. Automix: Automatically mixing language models. Advances in Neural Information Processing Systems, 37:131000–131034, 2024.arxiv.org/abs/2310.12963
  4. [4]Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. Healthbench: Evaluating large language models towards improved human health, 2025.
  5. [5]Richard Bellman. Dynamic programming. Science, 153(3731):34–37, 1966.pubmed.ncbi.nlm.nih.gov/17730601
  6. [6]Alexandre Belloni, Yan Chen, and Yehua Wei. Online pandora’s box for contextual llm cascading, 2026.arxiv.org/abs/2606.07392
  7. [7]Dylan Bouchard. Is escalation worth it? a decision-theoretic characterization of llm cascades, 2026.
  8. [8]Leo Breiman. Bagging predictors. Machine learning, 24(2):123–140, 1996.
  9. [9]Robert L Brennan. Generalizability theory. Educational Measurement: Issues and Practice, 11(4):27–34, 1992.
  10. [10]Jean Bretagnolle and Catherine Huber. Estimation des densités: risque minimax. Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete, 47(2):119–137, 1979.DOI
  11. [11]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877–1901. Curran Associates, Inc., 2020.arxiv.org/abs/2005.14165
  12. [12]Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou. Are more LLM calls all you need? towards the scaling properties of compound AI systems. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a.arxiv.org/abs/2403.02419
  13. [13]Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research, 2024b. ISSN 2835–8856.
  14. [14]Herman Chernoff. Sequential design of experiments. In Breakthroughs in Statistics: Methodology and Distribution, pp. 345–360. Springer, 1959.
  15. [15]Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: an open platform for evaluating llms by human preference. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.arxiv.org/abs/2403.04132
  16. [16]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichi Hiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.arxiv.org/abs/2110.14168
  17. [17]Alexander Philip Dawid and Allan M Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1):20–28, 1979.
  18. [18]Morris H DeGroot. Uncertainty, information, and sequential experiments. The Annals of Mathematical Statistics, 33(2):404–419, 1962.DOI
  19. [19]Jasper Dekoninck, Maximilian Baader, and Martin Vechey. A unified approach to routing and cascading for LLMs. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 12987–13010. PMLR, 13–19 Jul 2025.arxiv.org/abs/2410.10347
  20. [20]Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multiple classifier systems, pp. 1–15. Springer, 2000.DOI
  21. [21]Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks V. S. Lakshmanan, and Ahmed Hassan Awadallah. Hybrid LLM: Cost-efficient and quality-aware query routing. In International Conference on Learning Representations, 2024.arxiv.org/abs/2404.14618
  22. [22]Tao Feng, Yanzhen Shen, and Jiaxuan You. Graphrouter: A graph-based router for llm selections. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (eds.), International Conference on Learning Representations, volume 2025, pp. 26186–26203, 2025.arxiv.org/abs/2410.03834
  23. [23]Paulina Garcia-Corral, Hannah Béchara, Ran Zhang, and Slava Jankin. Politicause: An annotation scheme and corpus for causality in political texts. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 12836–12845, 2024.
  24. [24]Shashwat Goel, Jॉschka Strüber, Ilze Amanda Auzina, Karuna K Chandra, Ponnurangam Kumaraguru, Douwe Kiela, Ameya Prabhu, Matthias Bethge, and Jonas Geiping. Great models think alike and this undermines AI oversight. In Forty-second International Conference on Machine Learning, 2025.arxiv.org/abs/2502.04313
  25. [25]Daniel Golovin and Andreas Krause. Adaptive submodularity: Theory and applications in active learning and stochastic optimization. Journal of Artificial Intelligence Research, 42:427–486, 2011. doi: 10.1613/jair.3278.DOI
  26. [26]Y. Guo, Stefanus Jasin, and C. A. A. Lin. Congestion-aware static LLM cascades: Analysis of a steady-state framework. SSRN preprint 6759238, 2026.DOI
  27. [27]Rajarshi Haldar and Julia Hockenmaier. Rating roulette: Self-inconsistency in LLM-as-a-judge frameworks. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 24986–25004, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-335-7. doi: 10.18653/v1/2025.findings-emnlp.1361.DOI
  28. [28]Andrew Halterman and Katherine A. Keith. Codebook llms: Evaluating llms as measurement tools for political science concepts. Political Analysis, 34(2):188–204, 2025. ISSN 1476-4989. doi: 10.1017/pan.2025.10017.DOI
  29. [29]Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. Advances in neural information processing systems, 37:8093–8131, 2024.arxiv.org/abs/2406.18495
  30. [30]Nicholas Hay, Stuart Russell, David Tolpin, and Solomon Eyal Shim ony. Selecting computations: Theory and applications, 2012.
  31. [31]Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shri yash Kaustubh Upadhyay. Routerbench: A benchmark for multi-LLM routing system. In Agentic Markets Workshop at ICML 2024, 2024.arxiv.org/abs/2403.12031
  32. [32]Baihe Huang, Shanda Li, Tianhao Wu, Yiming Yang, Ameet Talwalkar, Kannan Ramchandran, Michael I. Jordan, and Jiantao Jiao. Sample complexity and representation ability of test-time scaling paradigms. In The Fourteenth International Conference on Learning Representations, 2026a.arxiv.org/abs/2506.05295
  33. [33]Hui Huang, Xingyuan Bu, Hongli Zhou, Yingqi Qu, Jing Liu, Muyun Yang, Bing Xu, and Tiejun Zhao. An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge model is not a general substitute for gpt-4. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 5880–5895, 2025.arxiv.org/abs/2403.02839
  34. [34]Jingkai Huang, Will Ma, and Zhengyuan Zhou. Optimal bayesian stopping for efficient inference of consistent LLM answers. In Forty-third International Conference on Machine Learning, 2026b.arxiv.org/abs/2602.05395
  35. [35]Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023.arxiv.org/abs/2312.06674
  36. [36]Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36:24678–24704, 2023.arxiv.org/abs/2307.04657
  37. [37]Jaehun Jung, Faeze Brahman, and Yejin Choi. Trust or escalate: LLM judges with provable guarantees for human agreement. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2407.18370
  38. [38]Daniel Kahneman, Olivier Sibony, and Cass R Sunstein. Noise: A flaw in human judgment. 2021.
  39. [39]Elliot Myunghoon Kim, Avi Garg, Kenny Peng, and Nikhil Garg. Correlated errors in large language models. In Forty-second International Conference on Machine Learning, 2025.arxiv.org/abs/2506.07962
  40. [40]Seungone Kim, Jay Shin, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Ryan Shin, Sungdong Kim, James Thorne, Minjoon Seo, et al. Prometheus: Inducing fine-grained evaluation capability in language models. In International Conference on Learning Representations, volume 2024, pp. 29927–29962, 2024.arxiv.org/abs/2310.08491
  41. [41]Guneet Kohli. Nine judges, two effective votes: Correlated errors undermine LLM evaluation panels. arXiv preprint arXiv:2605.29800, 2026.arxiv.org/abs/2605.29800
  42. [42]Jacky Kwok, Shulu Li, Pranav Atreya, Yuejiang Liu, Yixing Jiang, Chelsea Finn, Marco Pavone, Ion Stoica, and Azalia Mirhoseini. Llm-as-a-verifier: A general-purpose verification framework. arXiv preprint arXiv:2607.05391, 2026.arxiv.org/abs/2607.05391
  43. [43]Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. RLAIF vs. RLHF: Scaling reinforcement learning from human feedback with AI feedback. In Forty-first International Conference on Machine Learning, 2024.arxiv.org/abs/2309.00267
  44. [44]Guokai Li, Alys Liang, Mo Liu, Murray Lei, Stefanus Jasin, Fenghua Yang, and Preet Baxi. Sequential sampling for binary classification: Two llms are (almost) all you need, 2026.arxiv.org/abs/2604.01086
  45. [45]Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM computing surveys, 55(9):1–35, 2023a.arxiv.org/abs/2107.13586
  46. [47]Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2511–2522. Association for Computational Linguistics, 2023b. doi: 10.18653/v1/2023.emnlp-main.153.DOI
  47. [48]Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient reward-guided ensemble of large language models, 2023. URL https://arxiv.org/abs/2311.08692.
  48. [49]Anqi Mao, Christopher Mohri, Mehryar Mohri, and Yutao Zhong. Two-stage learning to defer with multiple experts. Advances in neural information processing systems, 36:3578–3606, 2023.DOI
  49. [50]Yasmin Moslem and John D. Kelleher. Dynamic model routing and cascading for efficient LLM inference: A survey. arXiv preprint arXiv:2603.04445, 2026.arxiv.org/abs/2603.04445
  50. [51]Multimodal Art. Jev decision index. Hugging Face Spaces, 2026. URL https://huggingface.com/spaces/multimodalart/jev-decision-index. Version 0.3.2, accessed October 13, 2026.
  51. [52]Mohammad Naghshvar and Tara Javidi. Active sequential hypothesis testing. The Annals of Statistics, 41 (6):2703–2738, 2013. ISSN 00905364, 21688966.DOI
  52. [53]Sirin Nitinawarat, George K. Atia, and Venugopal V. Veeravalli. Controlled sensing for multihypothesis testing. IEEE Transactions on Automatic Control, 58(10):2451–2464, 2013. doi: 10.1109/TAC.2013.2261188.DOI
  53. [54]Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs from preference data. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2406.18665
  54. [55]OpenAI, Josh Achiam, Steven Adler, and Sandhini Agarwal et. al. Gpt-4 technical report, 2024.arxiv.org/abs/2303.08774
  55. [56]Alexander J. Ratner, Christopher De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. Advances in neural information processing systems, 29:3567 – 3575, 2016.
  56. [57]Robert E Schapire and Yoav Freund. Boosting: Foundations and algorithms, 2013.
  57. [58]Nishad Singhi, Hritik Bansal, Arian Hosseini, Aditya Grover, Kai-Wei Chang, Marcus Rohrbach, and Anna Rohrbach. When to solve, when to verify: Compute-optimal problem solving and generative verification for llm reasoning, 2025.arxiv.org/abs/2504.01005
  58. [59]Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In International Conference on Learning Representations, 2025.arxiv.org/abs/2408.03314
  59. [60]Seamus Somerstep, Felipe Maia Polo, Allysson Flavio Melo de Oliveira, Prattyush Mangal, Mírian Silva, Onkar Bhardwaj, Mikhail Yurochkin, and Subha Maity. Carrot: A cost aware rate optimal router. arXiv preprint arXiv:2502.03261, 2025.arxiv.org/abs/2502.03261
  60. [61]Wei Song, Zhenya Huang, Cheng Cheng, Weibo Gao, Bihan Xu, GuanHao Zhao, Fei Wang, and Runze Wu. Irt-router: Effective and interpretable multi-llm routing via item response theory, 2025.arxiv.org/abs/2506.01048
  61. [62]Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 13003–13051, 2023.arxiv.org/abs/2210.09261
  62. [63]Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Tang, Alejandro Cuadron, Chenguang Wang, Raluca Popa, and Ion Stoica. Judgebench: A benchmark for evaluating llm-based judges. In International Conference on Learning Representations, volume 2025, pp. 63277–63303, 2025.arxiv.org/abs/2410.12784
  63. [64]Gemma Team, Sherif El Abd, and Vaibhav Aggarwal et. al. Gemma 4 technical report, 2026.arxiv.org/abs/2607.02770
  64. [65]Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. Judging the judges: Evaluating alignment and vulnerabilities in LLMs-as-judges. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics, 2025.arxiv.org/abs/2406.12624
  65. [66]Yigit Turkmen, Baturalp Buyukates, and Melih Bastopcu. Don’t always pick the highest-performing model: An information theoretic view of LLM ensemble selection. In ICML 2026 Statistical Frameworks for Uncertainty in Agentic Systems, 2026.arxiv.org/abs/2602.08003
  66. [67]Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating llm generations with a panel of diverse models. arXiv preprint arXiv:2404.18796, 2024.arxiv.org/abs/2404.18796
  67. [68]Abhigya Verma, Amit Kumar Saha, Seganrasan Subramanian, and Sai Harshitha Aluru. Agentjudgebench: A multi-difficulty benchmark for evaluating llm judges on agentic tool-calling, 2026.arxiv.org/abs/2608.26623
  68. [69]George Vershinin, Asaf Cohen, and Omer Gurewitz. Active sequential hypothesis testing with non-homogeneous costs. In ICASSP 2026 - 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 91–95, 2026. doi: 10.1109/ICASSP55912.2026.11463839.DOI
  69. [71]Abraham Wald. Sequential tests of statistical hypotheses. In Breakthroughs in statistics: Foundations and basic theory, pp. 256–298. Springer, 1992.
  70. [72]Xinyuan Wang, Yanchi Liu, Wei Cheng, Xujiang Zhao, Zhengzhang Chen, Wenchao Yu, Yanjie Fu, and Haifeng Chen. Mixllm: Dynamic routing in mixed large language models, 2025.arxiv.org/abs/2502.18482
  71. [73]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023a.arxiv.org/abs/2203.11171
  72. [74]Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. Pandalm: An automatic evaluation benchmark for LLM instruction tuning optimization. arXiv preprint arXiv:2306.05087, 2023b.arxiv.org/abs/2306.05087
  73. [75]An Yang, Anfeng Li, and Baosong Yang et. al. Qwen3 technical report, 2025.arxiv.org/abs/2505.09388
  74. [76]Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. Large language model cascades with mixture of thought representations for cost-efficient reasoning. In International Conference on Learning Representations, volume 2024, pp. 21691–21728, 2024.arxiv.org/abs/2310.03094
  75. [77]Haozhen Zhang, Tao Feng, and Jiaxuan You. Router-r1: Teaching llms multi-round routing and aggregation via reinforcement learning. arXiv preprint arXiv:2506.09033, 2025.arxiv.org/abs/2506.09033
  76. [78]Xuechen Zhang, Zijian Huang, Ege Onur Taga, Carlee Joe-Wong, Samet Oymak, and Jiasi Chen. Efficient contextual llm cascades through budget-constrained policy learning, 2024.arxiv.org/abs/2404.13082
  77. [79]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.arxiv.org/abs/2306.05685
  78. [80]Lianghui Zhu, Xinggang Wang, and Xinlong Wang. JudgeIm: Fine-tuned large language models are scalable judges. In The Thirteenth International Conference on Learning Representations, 2025.

Paper details

vlnc:W_3xsqnymh
Version
v1 · latest
Published
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents