In this paper
AbstractIntroductionProblem FormulationCost-Aware Single-Model RoutingCost-Aware Ensemble RoutingMethodologyPISA: Model Routing via Soft AssignmentsPISA-E: Prompt-Adaptive Cost-Aware LLM EnsemblingExperimentsExperimental SetupPISA EvaluationPISA-E EvaluationAdaptive vs. Fixed EnsemblingComponent AnalysisAblation of Prompt-Dependent Model RankingRelated WorkConclusionReferencesAdditional Experimental DetailsGaussian-Copula Model for Dependent Ensemble SuccessPISA Additional ExperimentsHypervolume Reference-Point SensitivityPISA-E Additional ExperimentsHypervolume Reference-Point SensitivitySensitivity to Maximum Ensemble CardinalityGreedy Selection vs. Exhaustive Surrogate OptimizationWorksComputer Science
Ensemble Routing for Prompt-Adaptive Cost-Aware LLM Inference
Abstract
Large language models (LLMs) increasingly support Web-based services that must balance response quality and inference cost across heterogeneous requests. Conventional routing selects one model per prompt, while fixed ensembles cannot adapt their composition and cost to individual requests. We introduce PISA, a prompt-adaptive, cost-aware single-model router, and PISA-E, its extension to ensemble routing. At the core of both methods, PISA combines soft cluster memberships with model-specific predictors to estimate individual model capabilities. Building on these estimates, PISA-E jointly selects ensemble composition and cardinality under an explicit cost–quality preference that can be adjusted at inference time without retraining. A tractable ensemble-success surrogate and efficient candidate selection enable this extension without ensemble-specific training. Across four LLM routing benchmarks under in-distribution (ID) and out-of-distribution (OOD) conditions, PISA achieves the highest macro-average cost–performance hypervolume among the evaluated single-model routers in both settings. Meanwhile, PISA-E achieves the highest macro-average task performance among the evaluated ensemble approaches when prioritizing response quality, surpassing the strongest competitors by 2.09 and 2.87 percentage points under ID and OOD conditions, respectively. At selected lower-cost operating points, it also exceeds the strongest ensemble competitors' macro-average task performance while reducing average inference cost by approximately 36% and 25%, respectively. These results demonstrate the effectiveness of prompt-adaptive capability estimation for both single-model routing and selective ensembling.
Introduction
Large language models (LLMs) are increasingly deployed in Web-based services, supporting applications such as search and information discovery, recommendation, and intelligent agents [3, 16]. Their adoption also extends to publicly accessible LLM services, creating new challenges for efficient and scalable inference [22]. These services must handle heterogeneous user requests while balancing response quality and inference cost. When multiple LLMs with different capabilities and pricing structures are available, invoking expensive models indiscriminately wastes resources, whereas consistently selecting inexpensive models can compromise response quality. Moreover, different users and requests may have different cost preferences, making it desirable to adapt inference strategies to both prompt characteristics and inference-cost preferences [31].
LLM routing addresses this challenge by selecting models according to prompt characteristics, expected response quality, and inference cost [25]. Recent approaches extend routing to budget-adaptive selection [43], reasoning-strategy selection [26], and additional deployment constraints [20]. However, selecting a single model cannot exploit the complementary capabilities of multiple LLMs. Ensembling offers an alternative by combining multiple model responses [5], but introduces additional inference costs. Existing ensemble approaches explore adaptive model composition, diversity-aware selection, and budget-constrained optimization [6, 15, 30]. Nevertheless, existing approaches typically address only parts of the problem: some adapt ensemble composition without explicitly optimizing inference cost, while others rely on classification-specific aggregation or sequential execution. Moreover, ensemble cardinality is generally fixed during training or configuration, limiting the ability to adjust the number of invoked models to individual prompts and changing cost preferences at inference time. This motivates ensemble routing, where both the composition and cardinality of the invoked model subset are determined dynamically at inference time according to prompt characteristics and an explicit cost–quality preference.
We introduce PISA, a prompt-adaptive, cost-aware single-model router that estimates the capabilities of individual LLMs, and PISA-E, its extension to prompt-adaptive ensemble routing. PISA-E reuses the individual capability estimates to construct a tractable surrogate for ensemble success, eliminating the need to train separate quality estimators for different model combinations. By combining these estimates with inference cost, PISA-E selects prompt-specific model subsets, jointly determining which models to invoke and how many. Crucially, the cost–quality preference can be adjusted at inference time without additional hyperparameter optimization or retraining, allowing the same framework to support different operating points. The selected responses are combined through capability-weighted semantic similarity to produce the final ensemble response.
We evaluate PISA and PISA-E on four LLM routing benchmarks under in-distribution (ID) and out-of-distribution (OOD) conditions. PISA achieves the highest macro-average hypervolume among the evaluated single-model routers in both settings, ranking first in six of the eight benchmark–setting combinations. PISA-E achieves the highest macro-average task performance among the evaluated ensemble approaches in both settings, improving over the strongest competitors by 2.09 percentage points under ID and 2.87 under OOD conditions when prioritizing response quality. At selected lower-cost operating points, it also exceeds the strongest competitors’ macro-average task performance while reducing average inference cost by approximately 36% and 25%, respectively. Finally, comparisons with fixed-cardinality variants demonstrate the benefits of adapting ensemble size to individual prompts.
Our main contributions are:
We develop PISA, a prompt-adaptive, cost-aware single-model router that estimates individual model capabilities through soft cluster memberships and model-specific predictors, providing the foundation for ensemble routing.
We propose PISA-E, a framework for prompt-adaptive, cost-aware ensemble routing that uses ensemble-level utility estimates derived from individual model capabilities to jointly determine ensemble composition and cardinality at inference time, according to prompt characteristics and inference-cost preferences.
We evaluate both methods across four benchmarks under ID and OOD conditions, demonstrating the highest macro-average single-model routing HV and ensemble task performance among the evaluated approaches in both settings. We also demonstrate operating points with higher task performance at lower cost than the strongest ensemble competitors and quantify the benefits of prompt-dependent ranking and adaptive ensemble cardinality.
Problem Formulation
Let denote a fixed pool of candidate LLMs. Given a prompt , model routing selects one or more models from while trading off response quality and inference cost.
For each model , let indicate whether the model produces a correct answer, with prompt-dependent probability
Since is unknown at inference time, a router operates on an estimate . Let denote the normalized cost of querying model on prompt .
Cost-Aware Single-Model Routing
Standard model routing selects exactly one model for each prompt. We formulate the cost-aware routing decision as
where
Here, controls the trade-off between predicted response quality and inference cost.
Cost-Aware Ensemble Routing
We generalize standard routing by allowing the router to select any non-empty subset , . All models in are queried, and their outputs are combined by a response resolver , which may implement any suitable aggregation strategy. If denotes the output of model , the resulting ensemble output is
Let
and define the corresponding probability of correctness as
As in the single-model setting, is unavailable at inference time; we denote its router-provided estimate by . Let denote the normalized inference cost associated with querying ensemble . Cost-aware ensemble routing then solves
with
Single-model routing is recovered by restricting the feasible ensembles to singletons, . Thus, the two settings differ only in their action spaces: standard routing selects an element of , whereas ensemble routing selects an element of .
Solving this problem requires estimating the expected quality and inference cost of candidate model subsets. Section 3 describes how we instantiate these quantities and perform cost-aware model and ensemble selection.
Methodology
Our approach builds on a shared capability-estimation framework to support both single-model and ensemble routing. PISA estimates the prompt-dependent capabilities of individual LLMs and uses them for cost-aware single-model selection. PISA-E extends this framework to ensemble routing by combining the same capability estimates into an ensemble-success surrogate, allowing model subsets of different compositions and cardinalities to be evaluated under the cost–quality objective introduced in Section 2. Figure 1 illustrates the resulting inference pipeline.

Figure 1. Inference-time overview of PISA and PISA-E. PISA maps a prompt embedding to soft memberships over X-means centroids, estimates each model’s success probability, and selects the model with highest cost-aware utility. PISA-E reuses the capability estimates to rank models by utility using its ensemble cost normalization, and evaluates odd-cardinality ranking prefixes using a strict-majority success surrogate and normalized ensemble cost. In the illustrated five-model example, the candidates are , , and . The selected ensemble is queried in parallel, and self-weighted BERTScore selects the final response using the estimated model capabilities.
PISA: Model Routing via Soft Assignments
The first stage estimates how likely each candidate model is to produce a correct response to a given prompt. PISA learns prompt-dependent capability estimates from historical model outcomes and semantic representations of the input queries. Assume a training set of prompts, where each prompt is represented by an embedding and is associated with a correctness score for each model . At inference time, only the embedding of the incoming prompt is available.
To capture similarities among prompts, we partition the fit-set embeddings using X-means, which determines both the number of clusters and their centroids. PISA represents each query through soft memberships over the learned centroids, allowing its representation to incorporate information from multiple clusters. After -normalizing embeddings and centroids, these memberships are computed from cosine distances as
where denotes the centroid of cluster and controls the concentration of the assignments. This representation allows capability estimation to exploit information from multiple neighboring regions instead of relying on a single hard assignment.
Using the resulting membership vector , PISA learns an independent logistic predictor for each candidate model. Each predictor is trained using the soft memberships of the training prompts and their corresponding binary correctness labels, obtained by thresholding evaluation scores at 0.5. For an incoming prompt, the estimated capability of model is
where is the logistic sigmoid and are the parameters fitted for model using L2-regularized logistic regression. The regularization strength is selected independently for each model through 5-fold cross-validation minimizing log loss over 10 logarithmically spaced values between and . The selected predictor is then refitted on the full training set.
To incorporate inference cost into the routing decision, we estimate each model’s cost from its average observed training-set cost. Let denote the cost of invoking model on training prompt , and define
The normalized cost provides the cost term in Eq. (3). For each incoming prompt and cost preference , PISA selects the model maximizing the corresponding single-model utility in Eq. (2).
The resulting capability estimates also provide the foundation for the ensemble-routing extension described next.
PISA-E: Prompt-Adaptive Cost-Aware LLM Ensembling
While PISA selects a single model, PISA-E extends the routing decision to model subsets, allowing both ensemble composition and cardinality to vary with the prompt and cost preference. Given the capability estimates , PISA-E considers ensembles with , where denotes the maximum ensemble size. Its central challenge is to evaluate the expected utility of candidate subsets before invoking their constituent models.
Directly estimating the performance of every possible ensemble would require modeling the behavior of numerous model combinations. Instead, PISA-E constructs an ensemble-level utility estimate from the individual capabilities already provided by PISA. Specifically, we use the probability that a strict majority of the selected models produce correct responses as a tractable surrogate for comparing candidate ensembles. This criterion captures how the predicted capabilities of the constituent models combine, without requiring ensemble-specific predictors or observing their generated responses.
For an ensemble of size , we model the correctness of each constituent model through a Bernoulli variable with success probability . Assuming conditional independence of these variables given the prompt, the strict-majority success probability is
with the treated as independent. We compute this probability using the standard Poisson-binomial dynamic program. The strict-majority success probability serves as a surrogate for ensemble selection, since the outcome of the final response resolver also depends on relationships among the generated responses. Accordingly, provides the quality term in the ensemble-routing objective independently of the response resolution procedure.
To balance this surrogate with inference cost, PISA-E estimates the cost of an ensemble by summing the average costs of its constituent models. Let the model costs defined in Eq. (11) be ordered as . We normalize the ensemble cost as
The denominator corresponds to the maximum cost of any subset containing at most models and remains constant across candidate ensembles. Combining this normalized cost with the strict-majority surrogate yields the ensemble utility in Eq. (8), allowing subsets of different cardinalities to be compared under the same cost–quality preference.
Since evaluating all feasible subsets would require a combinatorial search, PISA-E uses a greedy candidate-construction procedure. For each prompt, the models are first ranked according to their utilities, using the same cost normalization as for all candidate ensembles:
Here, is obtained from Eq. (13) with , so the cost of each model is normalized by the sum of the largest average model costs. Candidate ensembles are then constructed as progressively larger prefixes of this ranking, considering odd cardinalities up to . Each candidate is evaluated using its strict-majority success probability and normalized inference cost, and the ensemble maximizing the resulting utility is selected. For , both the cost normalization and the selection rule reduce to those of PISA.
Restricting the candidates to odd cardinalities follows directly from the strict-majority surrogate. For any even-cardinality ensemble, removing one constituent model produces an odd-cardinality subset whose strict-majority success probability is at least as high, since the required number of correct models decreases by one. As the subset also incurs lower inference cost, the even-cardinality ensemble cannot achieve higher utility under the surrogate objective. Consequently, considering only odd cardinalities does not exclude an optimal solution to the surrogate-based subset optimization problem. The greedy prefix construction further restricts the search space and does not guarantee the globally optimal subset, but provides an efficient procedure for selecting among candidate ensembles. A comparison with exhaustive surrogate optimization is provided in Appendix E.
Once an ensemble has been selected, PISA-E queries its constituent models in parallel and combines their responses through self-weighted BERTScore [36], a capability-aware extension of minimum Bayes risk (MBR) decoding. The resolver incorporates the estimated capabilities of both candidate and reference responses, weighting their contributions according to their predicted reliability. Let denote the response produced by model . We first normalize the capability estimates over the selected ensemble:
The score assigned to each candidate response is then
The final ensemble response is selected as
Alternative response resolution strategies are described in Appendix A. Algorithm 1 summarizes the complete inference procedure, from prompt-dependent capability estimation to cost-aware ensemble selection.
Algorithm 1 PISA-E: Prompt-Adaptive Cost-Aware Ensembling
Require: query embedding $x$; trade-off parameter $\alpha$; maximum ensemble size $k_{\max}$
Require: fitted PISA; average model costs $\{c_m^{\mathrm{avg}}\}_{m=1}^{M}$
1: for $m\in\mathcal{M}$ do
2: compute $\hat{p}_m(x)$ using PISA
3: compute $\bar{C}(\{m\})$ using Eq. (13)
4: compute $J_{\{m\}}(x;\alpha)$ using Eq. (14)
5: end for
6: sort models as $m_{(1)},\ldots,m_{(M)}$ such that
$J_{\{m_{(1)}\}}(x;\alpha)\ge\cdots\ge J_{\{m_{(M)}\}}(x;\alpha)$
7: best $\leftarrow\varnothing$
8: bestval $\leftarrow-\infty$
9: for $k=1,\ldots,k_{\max}$ with $k\bmod 2=1$ do
10: $E_k\leftarrow\{m_{(1)},\ldots,m_{(k)}\}$
11: compute $\hat{p}^{\mathrm{maj}}_{E_k}(x)$
12: compute $\bar{C}(E_k)$
13: compute $J_{E_k}(x;\alpha)$ using Eq. (8)
14: if $J_{E_k}(x;\alpha)>$ bestval then
15: best $\leftarrow E_k$
16: bestval $\leftarrow J_{E_k}(x;\alpha)$
17: end if
18: end for
19: return best
Experiments
Experimental Setup
Benchmarks. We evaluate PISA and PISA-E on four LLM routing benchmarks: LLMRouterBench Track-2 [19], RouterBench [13], FusionBench [10], and SPROUT [27]. These benchmarks cover heterogeneous candidate model pools and task groups, with performance measured using their respective evaluation scores.
Evaluation Protocol. We consider both ID and OOD evaluations. For ID experiments, we use the released data partitions when available; otherwise, we split the data into 70% development and 30% test, further dividing the development set into 80% training and 20% validation. For OOD experiments, we train exclusively on ID task groups, select configurations using ID validation data, and evaluate on held-out task groups. ID and OOD experiments use separately fitted routers. All experiments are evaluated over five seeds, with fixed data partitions, and all approaches are evaluated using the model responses provided by the benchmarks.
Baselines. We compare PISA against four single-model routing baselines: EmbedLLM [41], Avengers-Pro [37, 38], K-NN, and a Multi-Layer Perceptron (MLP). For PISA-E, we consider five LLM ensembling approaches: DFPE [6], LLM-TOPLA [30], SelectLLM [24], PairRanker [17], and BERTScore-based MBR [36]. PairRanker employs PairRM, the pairwise reward model introduced by LLM-Blender, to rank candidate responses. We evaluate its pretrained checkpoint (PairRanker-PT) and a fine-tuned variant (PairRanker-FT), trained following the original procedure.
We additionally report an oracle single-model selector that uses ground-truth evaluation scores to select the best-performing model for each prompt, providing a reference for the performance attainable through perfect model selection.
Metrics. We report task performance and inference cost, computed from benchmark-specific scores and model invocation costs. Task performance is macro-averaged across task groups. To summarize cost–performance trade-offs, we use normalized dominated hypervolume (HV), multiplied by 100. The reference performance corresponds to a random router, while the reference cost is set to the 40th percentile of the full-pool cost distribution for single-model routing and the 70th percentile for ensemble routing. These reference points are chosen to accommodate all the operating points of the compared methods. Since existing ensemble approaches generally provide a single operating point rather than a cost–performance frontier, we compare them directly in terms of task performance and inference cost.
Implementation. We employ Qwen3-Embedding-8B [39] for creating the prompt embeddings required by the routers. PISA-E considers ensembles of up to eleven models. Hyperparameters and ensemble configurations are selected independently for each benchmark using validation task performance at . The same fitted capability estimators are reused across different cost–performance operating points. Further implementation details, including the HV definition, are provided in Appendix A.
PISA Evaluation
We first evaluate PISA as a single-model router, comparing its cost–performance trade-offs against four established routing baselines. Table 1 reports the HV across four benchmarks under ID and OOD conditions.
| Approach | Track-2 | RouterBench | FusionBench | SPROUT | Macro average |
| ID | |||||
| EmbedLLM-MF | 22.87 (0.62) | 41.28 (0.98) | 41.92 (1.32) | 37.03 (0.36) | 35.78 (0.51) |
| Avengers-Pro | 19.61 (2.81) | 45.46 (0.81) | 40.46 (1.57) | 36.22 (0.96) | 35.44 (0.60) |
| KNN | 19.80 (0.00) | 43.03 (0.00) | 39.93 (0.00) | 34.74 (0.00) | 34.38 (0.00) |
| MLP | 1.79 (1.24) | 27.99 (0.31) | 33.95 (0.89) | 22.34 (1.47) | 21.52 (0.69) |
| PISA | 22.58 (0.42) | 46.15 (0.25) | 42.57 (0.98) | 38.33 (0.49) | 37.41 (0.35) |
| OOD | |||||
| EmbedLLM-MF | 13.60 (1.32) | 27.90 (0.36) | 19.74 (3.46) | 24.08 (0.25) | 21.33 (0.75) |
| Avengers-Pro | 3.48 (3.85) | 28.46 (2.67) | 17.36 (2.06) | 18.96 (1.31) | 17.06 (1.52) |
| KNN | 9.21 (0.00) | 26.94 (0.00) | 19.42 (0.00) | 24.53 (0.00) | 20.02 (0.00) |
| MLP | 2.66 (0.88) | 15.10 (4.55) | 18.19 (2.79) | 14.38 (0.90) | 12.58 (1.01) |
| PISA | 17.18 (1.85) | 29.50 (0.59) | 22.91 (1.21) | 21.81 (0.65) | 22.85 (0.38) |
Table 1. Normalized hypervolume (HV) of single-model routers across ID and OOD benchmarks. HV is multiplied by 100 and uses the p40 full-pool cost reference. Parentheses report sample standard deviations across five seeds.
PISA achieves the highest macro-average HV among the evaluated routers in both ID and OOD settings, reaching 37.41 and 22.85, respectively, compared with 35.78 and 21.33 for EmbedLLM-MF, the strongest competing router by this measure. PISA ranks first in six of the eight benchmark–setting combinations, showing that its
aggregate advantage extends across multiple benchmarks. These results establish PISA as an effective single-model router in its own right, while its capability estimates also provide the foundation for ensemble selection in PISA-E.
PISA-E Evaluation
We evaluate PISA-E against existing LLM ensembling approaches across the four benchmarks. Tables 2 and 3 report task performance and inference cost under ID and OOD conditions, respectively.
PISA-E attains higher macro-average task performance than the strongest competing ensemble approaches while requiring less inference expenditure. On ID benchmarks, PISA-E at achieves 73.78% task performance at a cost of 84 USD/query, compared with 73.18% at 132 USD/query of the best competitor, SelectLLM. This represents a 0.60 percentage-point improvement with approximately 36% lower cost. Under OOD conditions, PISA-E at achieves 60.85% performance at a cost of 292 USD/query, compared with 60.37% at 387 USD/query for DFPE, improving performance by 0.48 percentage points while reducing cost by approximately 25%.
When response quality is prioritized (), PISA-E reaches the highest macro-average task performance among the evaluated deployable approaches in both settings, reaching 75.27% on ID benchmarks and 63.24% on OOD benchmarks. These scores exceed those of the strongest competing ensemble approaches by 2.09 and 2.87 percentage points, respectively. Although some competitors achieve higher performance on individual benchmarks, none matches PISA-E in macro-average task performance across either evaluation setting.
Thus, PISA-E supports different cost–performance preferences by varying the trade-off parameter . On ID benchmarks, decreasing from 1.0 to 0.3 reduces average inference cost by a factor of 21.5, while task performance decreases from 75.27% to 70.75%. On OOD benchmarks, the corresponding cost reduction is approximately 106-fold, with performance decreasing from 63.24% to 50.54%.
Table 4 reports how ensemble cardinality varies with the cost preference , together with the proportion of queries assigned to a single model. At , PISA-E selects a single model for 84.33% of ID queries and 82.41% of OOD queries. Increasing leads to larger ensembles on average, with cardinalities reaching 6.42 and 7.39 at under ID and OOD conditions, respectively. Even when inference cost is not penalized, however, PISA-E selects a single model for 21.56% of ID queries and 9.31% of OOD queries. Ensemble cardinality therefore remains prompt-dependent even when the selection objective prioritizes response quality exclusively.
| Split | Models/query | Singleton (%) | Perf. (%) | Cost (USD/query ) | |
| ID | 0.3 | 1.32(0.05) | 84.33(2.46) | 70.75(0.18) | 13(0.23) |
| ID | 0.6 | 2.00(0.04) | 55.70(2.02) | 72.07(0.42) | 50.8(3.1) |
| ID | 1.0 | 6.42(0.15) | 21.56(1.39) | 75.27(0.27) | 280(8.4) |
| OOD | 0.3 | 1.35(0.01) | 82.41(0.62) | 50.54(0.09) | 8.01(0.25) |
| OOD | 0.6 | 2.41(0.02) | 37.44(1.23) | 55.33(0.15) | 30.1(0.42) |
| OOD | 1.0 | 7.39(0.15) | 9.31(0.09) | 63.24(0.15) | 854(6.8) |
| Both | 0.3 | 1.34(0.03) | 83.37(1.44) | 60.64(0.12) | 10.5(0.13) |
| Both | 0.6 | 2.21(0.02) | 46.57(0.82) | 63.70(0.21) | 40.4(1.6) |
| Both | 1.0 | 6.91(0.14) | 15.43(0.68) | 69.25(0.17) | 567(5.7) |
Table 4. Ensemble size, performance, and cost of PISA-E versus . Singleton (%) denotes queries using one model. “Both” averages all eight scenarios; parentheses give sample standard deviations over five seeds.
| Approach | Track-2 Perf. (%) | Track-2 Cost | RouterBench Perf. (%) | RouterBench Cost | FusionBench Perf. (%) | FusionBench Cost | SPROUT Perf. (%) | SPROUT Cost | Macro average Perf. (%) | Macro average Cost |
| Oracle | 85.93 | 213 | 92.15 | 13.7 | 91.01 | 4.48 | 98.04 | 62.2 | 91.78 | 73.3 |
| Prior ensemble approaches | ||||||||||
| PairRanker-PT | 60.87 | 946 | 67.71 | 87 | 51.03 | 8.05 | 68.54 | 132 | 62.04 | 293 |
| PairRanker-FT | 62.18 | 1200 | 77.62 | 87 | 52.04 | 8.05 | 75.75 | 162 | 66.90 | 365 |
| MBR | 60.75 | 1040 | 70.22 | 87 | 62.40 | 45.6 | 71.31 | 139 | 66.17 | 329 |
| DFPE | 59.95 | 965 | 81.25 | 81.2 | 69.17 | 59.9 | 76.30 | 167 | 71.67 | 318 |
| LLM-TOPLA | 57.93(0.43) | 1050(99) | 81.25(0.1) | 119(33) | 59.44(1.6) | 15.5(8.5) | 73.30(3.18) | 18.4(8.9) | 67.98(0.79) | 301(26) |
| SelectLLM | 61.55 | 309 | 78.93 | 63.5 | 74.62 | 14.7 | 77.63 | 142 | 73.18 | 132 |
| Our approach: PISA-E | ||||||||||
| PISA-E () | 59.86(0.24) | 40(0.8) | 74.53(0.35) | 5.97(0.077) | 71.61(0.56) | 2.18(0.12) | 77.01(0.24) | 3.84(0.095) | 70.75(0.18) | 13(0.23) |
| PISA-E () | 64.64(0.88) | 175(11) | 75.01(0.42) | 12.9(1.5) | 72.71(0.27) | 4.14(0.31) | 75.92(0.49) | 11.5(0.41) | 72.07(0.42) | 50.8(3.1) |
| PISA-E () | 66.18(0.33) | 268(5.1) | 78.64(0.35) | 43.4(3.8) | 73.47(0.27) | 7.17(0.6) | 76.86(0.29) | 17.3(0.48) | 73.78(0.16) | 84(1.6) |
| PISA-E () | 66.91(0.57) | 372(15) | 80.43(0.29) | 69.4(2.4) | 74.95(0.4) | 14.5(1.6) | 77.34(0.33) | 32(0.6) | 74.91(0.19) | 122(4.1) |
| PISA-E () | 68.08(0.23) | 774(31) | 81.26(0.1) | 152(3.5) | 75.31(0.45) | 34.1(1.2) | 76.44(0.83) | 159(1.6) | 75.27(0.27) | 280(8.4) |
Table 2. ID test task performance (%) and inference cost (USD/query ) across four benchmarks and their macro average. Parentheses report sample standard deviations across five seeds where available.
| Approach | Track-2 Perf. (%) | Track-2 Cost | RouterBench Perf. (%) | RouterBench Cost | FusionBench Perf. (%) | FusionBench Cost | SPROUT Perf. (%) | SPROUT Cost | Macro average Perf. (%) | Macro average Cost |
| Oracle | 77.09 | 303 | 87.17 | 12.3 | 88.61 | 3.97 | 96.27 | 50.1 | 87.29 | 92.3 |
| Prior ensemble approaches | ||||||||||
| PairRanker-PT | 56.94 | 1920 | 47.94 | 82.6 | 27.42 | 8.73 | 40.83 | 122 | 43.28 | 534 |
| PairRanker-FT | 50.97 | 1940 | 66.57 | 82.6 | 26.81 | 11.7 | 65.27 | 122 | 52.40 | 539 |
| MBR | 52.96 | 1840 | 50.58 | 82.6 | 29.28 | 41.5 | 47.56 | 135 | 45.10 | 524 |
| DFPE | 52.98 | 1230 | 73.30 | 78.9 | 48.94 | 69.3 | 66.25 | 169 | 60.37 | 387 |
| LLM-TOPLA | 53.55(4.49) | 1240(620) | 73.34(0.31) | 110(27) | 36.06(3.81) | 30.3(1.5) | 61.80(4.06) | 42.4(29) | 56.18(1.27) | 357(150) |
| SelectLLM | 58.42 | 787 | 73.46 | 78.9 | 39.85 | 2.91 | 64.02 | 56.7 | 58.94 | 231 |
| Our approach: PISA-E | ||||||||||
| PISA-E () | 48.46(0.1) | 21.4(0.98) | 56.28 | 3.64 | 38.89(0.47) | 2.63(0.073) | 58.52(0.26) | 4.37(0.054) | 50.54(0.09) | 8.01(0.25) |
| PISA-E () | 55.42(0.13) | 91.4(1.5) | 56.28 | 3.64 | 44.93(0.45) | 6.14(0.02) | 64.69(0.23) | 19.4(0.28) | 55.33(0.15) | 30.1(0.42) |
| PISA-E () | 58.57(0.58) | 375(74) | 56.42(0.15) | 3.69(0.038) | 49.37(0.79) | 11.8(0.23) | 63.03(0.27) | 49(2.5) | 56.85(0.31) | 110(19) |
| PISA-E () | 61.81(0.61) | 996(65) | 66.94(2.06) | 69.8(8.2) | 50.10(0.4) | 16.4(0.34) | 64.56(0.32) | 87.2(1.7) | 60.85(0.65) | 292(188) |
| PISA-E () | 62.65(0.25) | 3100(20) | 72.79(0.6) | 115(15) | 52.16(0.21) | 44.8(0.6) | 65.34(0.26) | 159(1.3) | 63.24(0.15) | 854(6.8) |
Table 3. OOD test task performance (%) and inference cost (USD/query ) on held-out task groups across four benchmarks and their macro average. Parentheses report sample standard deviations across five seeds where available.
Adaptive vs. Fixed Ensembling
We investigate whether prompt-adaptive ensemble cardinality improves cost–performance trade-offs over fixed-cardinality alternatives. To isolate this effect, we compare PISA-E against variants of the same framework constrained to select exactly models per query, with . In particular, corresponds to PISA. All variants use the same capability estimates, ensemble-success surrogate, and resolver, while varying to obtain different operating points.
Table 5 reports the HV achieved by adaptive and fixed-cardinality selection under ID and OOD conditions. PISA-E achieves the highest macro-average HV in both settings, reaching 39.35 on ID benchmarks and 26.30 on OOD benchmarks. It also outperforms the strongest fixed-cardinality alternatives, with improvements of 0.28 HV points under ID conditions and 1.80 points under OOD conditions. These results are particularly noteworthy because the competing configurations are optimized separately for each benchmark, whereas PISA-E determines ensemble cardinality dynamically for individual prompts at inference time. The larger advantage under OOD conditions further highlights the benefits of adaptive cardinality when generalizing to previously unseen task groups.
| Scenario | PISA-E | Fixed | Fixed | Fixed | Fixed | Fixed | Fixed |
| ID | |||||||
| Track-2 | 28.57(0.59) | 27.89(0.49) | 27.80(0.53) | 26.69(0.26) | 24.06(0.38) | 19.90(0.41) | 14.55(0.40) |
| RouterBench | 46.37(0.31) | 46.99(0.25) | 44.71(0.30) | 41.66(0.38) | 38.75(0.37) | 28.95(0.21) | 2.67(0.02) |
| FusionBench | 44.89(0.87) | 42.74(0.98) | 44.72(0.75) | 44.42(0.66) | 43.74(0.64) | 42.90(0.89) | 41.60(0.60) |
| SPROUT | 37.57(0.67) | 38.65(0.49) | 36.13(1.13) | 35.86(0.89) | 34.27(0.92) | 31.83(0.45) | 27.61(0.60) |
| ID macro average | 39.35(0.44) | 39.07(0.37) | 38.34(0.52) | 37.16(0.40) | 35.21(0.42) | 30.89(0.36) | 21.61(0.29) |
| OOD | |||||||
| Track-2 | 22.39(0.34) | 20.54(1.67) | 21.69(0.76) | 23.25(0.32) | 20.35(0.08) | 17.51(0.20) | 13.56(0.39) |
| RouterBench | 26.86(1.29) | 31.09(0.62) | 24.13(0.42) | 19.28(1.40) | 18.27(0.87) | 16.32(1.18) | 5.72(0.15) |
| FusionBench | 30.96(0.22) | 23.03(1.23) | 26.42(0.37) | 27.61(0.28) | 29.16(0.23) | 29.95(0.24) | 30.11(0.20) |
| SPROUT | 24.98(0.44) | 23.32(0.64) | 23.68(0.32) | 23.70(0.53) | 22.59(0.82) | 22.09(0.50) | 17.64(0.36) |
| OOD macro average | 26.30(0.27) | 24.50(0.31) | 23.98(0.29) | 23.46(0.43) | 22.59(0.14) | 21.47(0.17) | 16.76(0.15) |
| All scenarios | |||||||
| Full macro average | 32.82(0.25) | 31.78(0.31) | 31.16(0.35) | 30.31(0.37) | 28.90(0.27) | 26.18(0.21) | 19.18(0.17) |
Table 5. Normalized HV () of adaptive and fixed-cardinality ensemble routing under ID and OOD conditions, using the p70 cost reference. Parentheses report sample standard deviations across five seeds.
The results also reveal differences across benchmarks. While adaptive cardinality improves HV in several cases, fixed-cardinality strategies remain competitive and occasionally outperform PISA-E. In particular, fixed achieves higher HV on RouterBench in both settings, indicating that invoking additional models does not necessarily improve the trade-off for every task distribution. Nevertheless, the aggregate results suggest that ensemble cardinality is better treated as a prompt-dependent routing decision rather than a fixed configuration parameter, allowing additional model invocations to be allocated selectively according to predicted utility and inference-cost preferences.
Component Analysis
We analyze two components of PISA-E: the ensemble-success estimator used to select model subsets and the response resolver used to combine their outputs. For ensemble-success estimation, we compare the default Poisson-binomial (PB) formulation with a Gaussian-copula (GC) alternative that explicitly models dependencies among model correctness events. For response resolution, we consider standard BERTScore-based minimum Bayes risk (MBR) decoding, capability-weighted BERTScore variants, and pretrained or fine-tuned PairRanker. Standard BERTScore selects the response with the highest aggregate semantic similarity to the other ensemble responses. Other-weighted BERTScore weights the reference responses according to their estimated model capabilities, while self-weighted BERTScore additionally weights each candidate response by its own estimated capability. PairRanker instead employs a learned pairwise response-ranking model. Further details on the alternative estimator and resolvers are provided in the appendix.
Ensemble-Success Estimation. We investigate the impact of the ensemble-success estimator by comparing PB and GC across different response resolvers. Table 6 reports task performance at under ID and OOD conditions. GC improves performance for several resolver configurations, particularly other-weighted BERTScore and PairRanker. However, these improvements do not consistently extend to self-weighted BERTScore, for which PB achieves higher macro-average performance in both ID (75.27% versus 75.23%) and OOD (63.24% versus 62.36%) settings. Overall, PB combined with self-weighted BERTScore achieves the highest aggregate performance, indicating that explicitly modeling correctness dependencies does not consistently improve ensemble performance and supporting the simpler PB formulation as the default estimator.
| Selector | Resolver | ID | OOD | Both |
| PB | Self-weighted BERTScore | 75.27(0.27) | 63.24(0.15) | 69.25(0.17) |
| PB | Other-weighted BERTScore | 73.04(0.26) | 59.88(0.26) | 66.46(0.21) |
| PB | BERTScore | 74.35(0.19) | 62.03(0.15) | 68.19(0.06) |
| PB | PairRanker (pretrained) | 68.00(0.48) | 48.68(0.17) | 58.34(0.23) |
| PB | PairRanker (fine-tuned) | 71.65(0.31) | 52.07(0.24) | 61.86(0.19) |
| GC | Self-weighted BERTScore | 75.23(0.13) | 62.36(0.22) | 68.80(0.13) |
| GC | Other-weighted BERTScore | 74.99(0.30) | 60.99(0.35) | 67.99(0.28) |
| GC | BERTScore | 75.18(0.25) | 61.91(0.22) | 68.54(0.18) |
| GC | PairRanker (pretrained) | 74.43(0.34) | 57.07(1.11) | 65.75(0.61) |
| GC | PairRanker (fine-tuned) | 74.95(0.15) | 57.94(0.53) | 66.44(0.30) |
Table 6. Task performance (%) at for different ensemble-success estimators and response resolvers. The ensemble-success estimators are Poisson-binomial (PB) and Gaussian copula (GC). ID and OOD results are macro-averaged across the four benchmarks, while “Both” averages all eight scenarios. Parentheses report sample standard deviations across five seeds.
Ensemble Resolver Quality. We further investigate how the choice of response resolver affects ensemble performance. As shown in Table 6, self-weighted BERTScore achieves the highest aggregate performance among the evaluated resolvers when paired with PB, outperforming standard BERTScore by 0.92 percentage points in ID and 1.21 percentage points in OOD settings. In contrast, PairRanker exhibits greater sensitivity to the ensemble-success estimator, with GC substantially improving performance over PB for both pretrained and fine-tuned variants. Fine-tuning consistently improves PairRanker accuracy over its pretrained counterpart under both estimation methods, although neither variant surpasses self-weighted BERTScore in aggregate performance. These findings support capability-weighted semantic similarity as an effective response resolution strategy and motivate self-weighted BERTScore as the default resolver in PISA-E.
Ablation of Prompt-Dependent Model Ranking
To assess the contribution of prompt-dependent model ranking, we compare PISA-E against a fixed-global baseline that retains adaptive ensemble cardinality but ranks models using their global training-set capabilities instead of prompt-specific estimates. This isolates the effect of prompt-dependent ranking while preserving the cost-aware ensemble selection framework. Table 7 shows that PISA-E improves macro-average HV by 4.96 points under ID conditions and 0.91 points under OOD conditions. It outperforms the fixed-global baseline on all four ID benchmarks and on Track-2 and RouterBench under OOD conditions, while the baseline performs better on FusionBench and SPROUT. Across all eight scenarios, PISA-E achieves a macro-average HV of 32.82 compared with 29.89 for the fixed-global baseline, indicating that prompt-dependent ranking contributes to cost–performance improvements independently of adaptive cardinality.
| Scenario | PISA-E | Fixed global + PB |
| ID | ||
| Track-2 | 28.57(0.59) | 22.86 |
| RouterBench | 46.37(0.31) | 39.12 |
| FusionBench | 44.89(0.87) | 41.26 |
| SPROUT | 37.57(0.67) | 34.31 |
| ID macro average | 39.35(0.44) | 34.39 |
| OOD | ||
| Track-2 | 22.39(0.34) | 14.95 |
| RouterBench | 26.86(1.29) | 24.90 |
| FusionBench | 30.96(0.22) | 34.91 |
| SPROUT | 24.98(0.44) | 26.81 |
| OOD macro average | 26.30(0.27) | 25.39 |
| All scenarios | ||
| Full macro average | 32.82(0.25) | 29.89 |
Table 7. Effect of prompt-dependent model ranking on normalized hypervolume (HV, ×100) under ID and OOD conditions, using the p70 cost reference. PISA-E is compared with a variant that ranks models globally by their task performance on the fit set while retaining Poisson-binomial (PB) ensemble-success estimation.
Related Work
Cost-Aware Model Routing. LLM routing selects a model for each prompt by estimating response quality and accounting for inference cost. EmbedLLM [41] learns latent representations from a model–question correctness matrix. Clustering-based methods instead use similarities among prompts to estimate model performance: UniRoute [18] and Avengers-PRO [37, 38] rely on hard cluster assignments and cluster-level statistics, while ZeroRouter [35] combines clustering with item response theory. PISA differs in using soft cluster memberships and model-specific predictors to estimate prompt-dependent capabilities. Related cost-aware inference strategies include LLM cascades [4, 8] and risk-aware routing [12]. Cascades may escalate requests sequentially; PISA-E instead selects models for parallel execution before generating their responses.
LLM Ensembling and Response Aggregation. LLM ensembling combines responses from multiple models to exploit complementary capabilities. Aggregation methods include majority voting [1, 33], learned ranking and fusion, as in LLM-Blender [17] and URG [23], and similarity-based minimum Bayes risk decoding, including variants that account for uncertainty [7]. Multi-agent debate [9] and Mixture-of-Agents [32] also combine model outputs, but involve interaction across successive rounds or stages. PISA-E does not organize such collaboration: after selecting an ensemble, it queries the models in parallel and resolves their responses using capability-weighted semantic similarity.
Adaptive and Cost-Aware Ensemble Selection. Ensemble selection determines which models to invoke, and whether that choice changes across queries. LLM-TOPLA [30] and DFPE [6] construct ensembles that are subsequently reused across queries, whereas SelectLLM [24] adapts ensemble composition to the input while retaining a fixed cardinality. ThriftLLM [14] jointly adapts composition and cardinality under cost constraints, but addresses classification queries rather than open-ended response generation.
Other adaptive multi-model approaches make different inference decisions. EMS [21] expands the set of invoked models sequentially; LightRouter [40] combines model selection with staged collaboration; and retrieval-based selection and response enhancement are investigated in [29]. Query-specific agentic systems instead configure task plans, agents, or collaboration strategies [11, 28, 34]. PISA-E targets pre-inference selection of a parallel ensemble for response generation. It scores candidate subsets using a strict-majority surrogate constructed from prompt-dependent individual capability estimates, jointly selecting composition and cardinality under an explicit cost–quality preference without training ensemble-specific predictors.
Conclusion
We introduced PISA, a prompt-adaptive, cost-aware single-model router, and PISA-E, its extension to ensemble routing. Both methods use prompt-dependent capability estimates, which PISA-E reuses to jointly select ensemble composition and cardinality without ensemble-specific training. Across four heterogeneous benchmarks, PISA achieves the highest macro-average hypervolume among the evaluated single-model routers under both ID and OOD conditions. PISA-E achieves the highest macro-average task performance among the evaluated ensemble approaches, exceeding the strongest competitors by 2.09 and 2.87 percentage points, respectively, when prioritizing response quality. At selected lower-cost operating points, PISA-E also improves macro-average task performance while reducing average inference cost by approximately 36% under ID and 25% under OOD conditions. Component analyses and ablations support the contributions of capability-aware response resolution, prompt-dependent ranking, and adaptive ensemble cardinality. Together, these results demonstrate that a shared capability-estimation framework supports effective single-model routing and selective ensembling, with cost–quality preferences adjustable at inference time without retraining. For Web-based services such as search, recommendation, and intelligent agents, this flexibility offers a way to adapt inference expenditure to heterogeneous user requests and changing quality–cost preferences.
References
Additional Experimental Details
Benchmarks and evaluation scores. Table 8 summarizes the evaluated benchmarks. We use the original evaluation scores provided by each benchmark, which are all in the range . FusionBench uses binary correctness, SPROUT employs LLM-judge scores, while RouterBench and Track-2 include task-dependent evaluation scores. Task performance is computed by averaging scores within each task group and then taking the macro-average across groups, consistently for all methods. For capability estimation, PISA derives binary success labels from the training scores using and trains model-specific predictors on these labels.
| Benchmark | Prompts | Candidate models |
| LLMRouterBench Track-2 | 12,446 | 13 |
| RouterBench | 34,684 | 11 |
| FusionBench | 6,398 | 40 |
| SPROUT | 44,241 | 13 |
Table 8. Number of prompts and candidate LLMs in each evaluation benchmark.
Data partitions. For OOD evaluation, we construct group level partitions independently of the original benchmark splits. Entire task groups are assigned either to ID, which supplies training and validation data, or to OOD, which is reserved exclusively for testing. All available rows of held-out groups are used for OOD evaluation, irrespective of their original partition. The group assignments are selected without considering model outcomes, targeting approximately 20% of the available prompts for OOD evaluation, subject to a 30% upper bound. We require at least three OOD groups spanning three task categories and domains, and at least three retained ID groups. Among valid partitions, we select those closest to the target proportion. Table 9 reports the resulting assignments.
| Benchmark | ID groups | OOD groups (rows; share) |
| Track-2 | aime, livecodebench, swe-bench, mmlu, simpleqa, arena-hard, tau2 mmlu | livemathbench, gpqa, hle (2,477; 19.9%) |
| RouterBench | mmlu, hellaswag, arc | winogrande, mbpp, grade (9,144; 26.4%) |
| FusionBench | arc_challenge, boolq, commonsense_qa, gsm8k, hellaswag, human_eval, mmlu, natural_qa, openbook_qa, squad, trivia_qa | gpqa, math, mbpp (1,278; 20.0%) |
| SPROUT | ragbench, openhermes, MMLU-Pro | MATH, MuSR, GPQA (11,173; 25.3%) |
Table 9. Task-group partitions used for OOD evaluation. The final column reports the number and percentage of held-out prompts.
Experimental repetitions. The five random seeds are 11, 22, 33, 44, and 55. Thus, reported standard deviations measure run-to-run variability under fixed data partitions.
Baselines. DFPE follows its clustering and representative selection procedure, using global statistics fitted on ID training data when evaluating held-out OOD task groups. LLM-TOPLA constructs and prunes model teams following its original procedure. SelectLLM uses WeightedMaxConf, its best-performing configuration, with a RoBERTa-base multilabel classifier and inverse-square-root confidence weighted voting, following the original article. For PairRanker and MBR, fixed-size ensembles are formed from the top- models ranked by macro-average training performance.
Implementation Details. For PISA, the maximum number of clusters and the soft-assignment parameter are selected using validation data. Cost–performance operating points are obtained by sweeping from 0.10 to 1.00 in increments of 0.05.
Ensemble resolution. We consider several alternatives to the self-weighted BERTScore resolver introduced in § 3. Given an ensemble and its generated responses , standard minimum Bayes risk (MBR) decoding with BERTScore [36] selects the response maximizing its aggregate similarity to the ensemble outputs:
The other-weighted variant incorporates the prompt-dependent capability estimates produced by PISA:
where denotes the normalized capability estimate of the model generating response . Unlike self-weighted BERTScore, other-weighted BERTScore does not additionally weight each candidate by its own estimated capability.
Inference costs. Routing decisions use training-set average model costs, normalized as described in Section 3. Reported inference costs instead use the benchmark-recorded costs of individual responses, summed over all selected models and averaged across test queries. Models with missing responses still contribute their recorded inference costs. Local routing and response-resolution overhead are excluded.
Hypervolume metric. We assess routing cost–performance trade-offs using normalized dominated hypervolume (HV) [42]. Let denote the non-dominated operating points ordered by increasing cost, with reference point . The dominated area is
where only points within the reference bounds contribute. We normalize by
The performance reference is the macro-average expected score of a uniformly random single-model router. The cost reference is defined as a percentile of the per-prompt full-model-pool inference cost distribution, computed separately for each benchmark and evaluation partition. We use the 40th percentile for single-model routing comparisons and the 70th percentile for ensemble-routing comparisons. The higher reference for ensemble routing accommodates the larger inference costs associated with querying multiple models. Within each comparison, the same reference point is used for all methods. Both reference coordinates remain fixed across methods, and HV values are multiplied by 100 when reported.
Gaussian-Copula Model for Dependent Ensemble Success
The Poisson-binomial surrogate assumes conditional independence among model correctness events. To investigate whether modeling their dependencies improves ensemble selection, we consider a Gaussian-copula alternative.
For an ensemble of size , let denote the capability estimate provided by PISA for model . We introduce latent variables , where is a correlation matrix, and define
where denotes the standard normal CDF. This construction preserves the marginal probabilities while introducing dependencies through . The ensemble-success surrogate becomes
When , the formulation reduces to the Poisson-binomial surrogate.
Dependence estimation. We estimate a prompt-independent correlation matrix from training data using pairwise maximum likelihood, following the composite-likelihood approach to multivariate probit estimation [2]. Let denote the observed binary correctness of model on training prompt , using the same threshold as PISA, and let denote its out-of-fold capability estimate. For each model pair , the probability of joint correctness under latent correlation is
where denotes the bivariate standard-normal CDF. The remaining outcome probabilities follow from the marginals: , , and . We estimate each correlation as
The pairwise estimates are assembled into , with unit diagonal, and projected onto the nearest valid correlation matrix to ensure positive semidefiniteness.
Inference. For each candidate ensemble, we extract the corresponding submatrix and approximate (23) using 2,048 Monte Carlo samples from . The estimated success probability is the fraction of samples satisfying the strict-majority condition. This estimate replaces the Poisson-binomial surrogate in the PISA-E utility, leaving candidate construction, cost normalization, and ensemble selection unchanged.
PISA Additional Experiments
Hypervolume Reference-Point Sensitivity
To assess the sensitivity of our results to the hypervolume reference point, we vary both its cost and performance coordinates. As described in Appendix A, the main experiments use fixed reference costs corresponding to the 40th and 70th percentiles of the full-pool cost distribution for single-model and ensemble routing, respectively. These reference points accommodate the operating points of the compared methods, while the reference performance corresponds to a uniformly random router. Although these choices provide a principled basis for comparison, HV values depend on the selected reference point. We therefore examine whether the aggregate advantage of PISA persists under alternative reference points. Figure 2 shows that PISA maintains a positive macro-average HV advantage over all competing single-model routers across the evaluated configurations. While the magnitude of this advantage varies with the reference coordinates, the relative ranking remains unchanged. These results indicate that the superior aggregate cost–performance trade-offs of PISA are not contingent on the specific reference point adopted in the main evaluation.

Figure 2. Sensitivity of PISA’s macro-average HV advantage over competing routers to the performance reference (left) and full-pool cost reference (right). Positive values favor PISA.
PISA-E Additional Experiments
Hypervolume Reference-Point Sensitivity
We further examine whether the advantage of adaptive ensemble cardinality depends on the HV reference point selected for the main evaluation. Figure 3 reports the macro-average HV differences between PISA-E and each fixed-cardinality variant when varying the performance and cost references independently. Across all evaluated configurations, PISA-E maintains a positive HV advantage over every fixed-cardinality alternative. Although the magnitude of these differences varies with the reference coordinates, the aggregate ranking remains unchanged. The consistent advantage across reference configurations shows that the aggregate benefits of adaptive ensemble cardinality persist under different HV reference points.

Figure 3. Sensitivity of PISA-E’s macro-average HV advantage over fixed-cardinality variants to the performance reference (left) and full-pool cost reference (right). Positive values favor PISA-E.
Sensitivity to Maximum Ensemble Cardinality
Table 10 examines the effect of restricting the maximum ensemble cardinality . Under ID conditions, PISA-E achieves a macro-average HV of 39.35 with , remaining competitive when the maximum cardinality is reduced, with HV reaching 39.41 at . Under OOD conditions, allowing larger ensembles provides more substantial benefits, although restricting to seven still yields an HV of 26.07, compared with 26.30 when allowing up to eleven models. Thus, PISA-E retains strong aggregate cost–performance trade-offs under moderate restrictions on ensemble cardinality, while the larger gains observed under OOD conditions highlight the benefits of access to a broader range of ensemble sizes.
| Scenario | ||||||
| ID | ||||||
| Track-2 | 27.89 (0.49) | 28.26 (0.64) | 28.60 (0.60) | 28.58 (0.54) | 28.51 (0.58) | 28.57 (0.59) |
| RouterBench | 46.99 (0.25) | 46.38 (0.26) | 46.25 (0.33) | 46.31 (0.29) | 46.38 (0.28) | 46.37 (0.31) |
| FusionBench | 42.74 (0.98) | 44.40 (0.69) | 44.68 (0.69) | 44.83 (0.72) | 44.90 (1.04) | 44.89 (0.87) |
| SPROUT | 38.65 (0.49) | 37.67 (0.82) | 38.03 (1.21) | 37.91 (0.51) | 37.60 (0.76) | 37.57 (0.67) |
| ID average | 39.07 (0.37) | 39.18 (0.48) | 39.39 (0.50) | 39.41 (0.48) | 39.35 (0.46) | 39.35 (0.44) |
| OOD | ||||||
| Track-2 | 20.54 (1.67) | 21.67 (0.80) | 23.37 (0.38) | 22.51 (0.33) | 22.42 (0.37) | 22.39 (0.34) |
| RouterBench | 31.09 (0.62) | 26.85 (0.84) | 24.77 (0.93) | 27.04 (1.23) | 26.83 (1.17) | 26.86 (1.29) |
| FusionBench | 23.03 (1.23) | 26.49 (0.38) | 27.81 (0.26) | 29.62 (0.33) | 30.59 (0.09) | 30.96 (0.22) |
| SPROUT | 23.52 (0.64) | 24.18 (0.31) | 25.29 (0.38) | 25.11 (0.08) | 25.04 (0.44) | 24.98 (0.44) |
| OOD average | 24.50 (0.31) | 24.80 (0.38) | 25.31 (0.25) | 26.07 (0.64) | 26.22 (0.28) | 26.30 (0.27) |
| All scenarios | ||||||
| Full average | 31.78 (0.31) | 31.99 (0.32) | 32.35 (0.31) | 32.74 (0.32) | 32.78 (0.25) | 32.82 (0.25) |
Table 10. Normalized HV (×100) of PISA-E for different maximum ensemble cardinalities under ID and OOD conditions, using the p70 cost reference. Parentheses report sample standard deviations across five seeds.
Greedy Selection vs. Exhaustive Surrogate Optimization
We evaluate the greedy candidate-selection procedure by comparing it with exhaustive optimization of the ensemble-success surrogate. Both approaches use the same capability estimates, Poisson-binomial surrogate, and self-weighted BERTScore resolver, with . Exhaustive optimization evaluates all feasible model subsets, requiring the assessment of candidates for a pool of models. In contrast, greedy selection considers only odd-cardinality prefixes of the single-model utility ranking, substantially reducing the search space.
Figure 4 compares the cost–performance curves on Track-2 and RouterBench. Greedy selection achieves trade-offs comparable to exhaustive optimization, with closely aligned curves on Track-2 and favorable performance across several intermediate-cost regions on RouterBench. These results indicate that greedy ranking prefixes provide a computationally efficient alternative while maintaining competitive cost–performance trade-offs in the evaluated settings.

Figure 4. Cost–performance trade-offs of greedy candidate selection and exhaustive surrogate optimization on Track-2 and RouterBench, with .
References
- [1]Rui Ai, Yuqi Pan, David Simchi-Levi, Milind Tambe, and Haifeng Xu. 2026. Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Information. arXiv:2510.01499 [cs.LG] doi:10.48550/arXiv.2510.01499
- [2]J. R. Ashford and R. R. Sowden. 1970. Multi-Variate Probit Analysis. Biometries 26, 3 (1970), 535–546. http://www.jstor.org/stable/2529107pubmed.ncbi.nlm.nih.gov/5480663
- [3]Kayhan Behdin, Ata FatahiBaizi, Qingquan Song, Yun Dai, Aman Gupta, Zhipeng Wang, Hejian Sang, Shao Tang, Gregory Dexter, Sirou Zhu, Siyu Zhu, Tejas Dharamsi, Vignesh Kothapalli, Zhoutong Fu, Yihan Cao, Pin-Lun Hsu, Fedor Borisyuk, Natesh S. Pillai, Luke Simon, and Rahul Mazumder. 2025. Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, Saloni Potdar, Lina Rojas-Barahona, and Sebastien Montella (Eds.). Association for Computational Linguistics, Suzhou (China), 1687–1702. doi:10.18653/v1/2025.emnlp-industry.119
- [4]Lingjiao Chen, Matei Zaharia, and James Zou. 2024. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. Transactions on Machine Learning Research (2024). https://openreview.net/forum?id=cSimKw5p6R Featured Certification.
- [5]Zhijun Chen, Xiaodong Lu, Jingzhe Li, Pengpeng Chen, Zhuoran Li, Kai Sun, Yuankai Luo, Qianren Mao, Ming Li, Likang Xiao, Dingqi Yang, Yikun Ban, and Hailong Sun. 2026. Harnessing Multiple Large Language Models: A Survey on LLM Ensemble. In Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26, Diego Calvanese (Ed.). International Joint Conferences on Artificial Intelligence Organization, 7807–7816. doi:10.24963/ijcai.2026/867 Survey Track.DOI
- [6]Seffi Cohen, Nurit Cohen, Niv Goldshlager, Bracha Shapira, and Lior Rokach. 2026. DFPE: A Diverse Fingerprint Ensemble for Enhancing LLM Performance. In Findings of the Association for Computational Linguistics: EACL 2026, Vera Demberg, Kentaro Inui, and Lluis Marquez (Eds.). Association for Computational Linguistics, Rabat, Morocco, 5326–5336. doi:10.18653/v1/2026.findings-eacl.282DOI
- [7]Nico Daheim, Clara Meister, Thomas Möllenhoff, and Iryna Gurevych. 2025. Uncertainty-Aware Decoding with Minimum Bayes Risk. International Conference on Learning Representations 2025 (May 2025), 7212–7234.arxiv.org/abs/2503.05318
- [8]Jasper Dekoninck, Maximilian Baader, and Martin Vacher. 2025. A Unified Approach to Routing and Cascading for LLMs. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267), Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (Eds.). PMLR, 12987–13010. https://proceedings.mlr.press/v267/dekoninck25a.htmlarxiv.org/abs/2410.10347
- [9]Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2024. Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on Machine Learning (Vienna, Austria) (ICML’24). JMLR.org, Article 467, 31 pages.
- [10]Tao Feng, Haozhen Zhang, Zijie Lei, Fengrui Han, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, and Jiaxuan You. 2026. FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data. Transactions on Machine Learning Research (2026). https://openreview.net/forum?id=N951scS3yEarxiv.org/abs/2507.10540
- [11]Aamir Hamid, Bharg Barot, Satvik Racharla, Tim Finnin, Primal Pappachan, and Roberto Yus. 2026. Opti-Q: A Constraint-Based Optimization Framework for Multi-LLM Question Planning. arXiv:2607.22621 [cs.AI] https://arxiv.org/abs/2607.22621
- [12]Sai Hao, Hao Zeng, Hongxin Wei, and Bingyi Jing. 2026. RACER: Risk-Aware Calibrated Efficient Routing for Large Language Models. In Proceedings of the 43rd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 306), Tong Zhang, Miroslav Dudík, Martin Jaggi, Alekh Agarwal, Sharon Li, Dale Schuurmans, Jerry Zhu, Felix Berkenkamp, Hanze Dong, and Alberto Bietti (Eds.). PMLR, 40499–40524. https://proceedings.mlr.press/v306/hao26c.htmlarxiv.org/abs/2603.06616
- [13]Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyans Kaustubh Upadhyay. 2024. RouterBench: A Benchmark for Multi-LLM Routing System. In Agentic Markets Workshop at ICML 2024. https://openreview.net/forum?id=IXVmV8Uxwharxiv.org/abs/2403.12031
- [14]KeKe Huang, Yimin Shi, Dujian Ding, Yifei Li, Yang Fei, Laks Laksmanan, and Xiaokui Xiao. 2025. ThriftLLM: On Cost-Effective Selection of Large Language Models for Classification Queries. Proceedings of the VLDB Endowment 18, 11 (July 2025), 4410–4423. doi:10.14778/3749466.3749702arxiv.org/abs/2501.04901
- [15]Yichong Huang, Xiaocheng Feng, Baohang Li, Yang Xiang, Hui Wang, Ting Liu, and Bing Qin. 2024. Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration. In Advances in Neural Information Processing Systems, Vol. 37. Curran Associates, Inc., 119838–119860. doi:10.5220/079017-3808arxiv.org/abs/2404.12715
- [16]Zijing Hui, Wenhan Lyu, Shusen Wang, Li Chen, and Chu Wang. 2026. Real-Time Trend Prediction via Continually-Aligned LLM Query Generation. In Proceedings of the ACM Web Conference 2026 (Dubai, United Arab Emirates) (WWW ’26). Association for Computing Machinery, New York, NY, USA, 8725–8728. doi:10.1145/3749044.3792950
- [17]Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023. LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 14165–14178. doi:10.18653/v1/2023.acl-long.792DOI
- [18]Wittawat Jitkrittum, Harikrishna Narasimhan, Ankit Singh Rawat, Jeeves Juneja, Congchao Wang, Zifeng Wang, Alec Go, Chen-Yu Lee, Pradeep Shenoy, Rina Panigrahy, Aditya Krishna Menon, and Sanjiv Kumar. 2026. Universal Model Routing for Efficient LLM Inference. International Conference on Learning Representations 2026 (April 2026), 10169–10218.arxiv.org/abs/2502.08773
- [19]Hao Li, Yiqun Zhang, Zhaoyan Guo, Chenxu Wang, Shengji Tang, Qiaosheng Zhang, Yang Chen, Binqing Qi, Peng Ye, Lei Bai, Zhen Wang, and Shuyue Hu. 2026. LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing. In Findings of the Association for Computational Linguistics: ACL 2026, Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens (Eds.). Association for Computational Linguistics, San Diego, California, United States, 37733–37754. doi:10.18653/v1/2026.findings-acl.1881DOI
- [20]Tingting Li, Ziming Zhao, Zhaoxuan Li, Xiaofei Yue, and Jiongchi Yu. 2026. Fair and Carbon-Aware LLM Routing for Web Services. In Proceedings of the ACM Web Conference 2026 (WWW ’26). Association for Computing Machinery, New York, NY, USA, 9563–9571. doi:10.1145/3774904.3793001
- [21]Yiqing Liu, Hantao Yao, Wu Liu, and Yongdong Zhang. 2024. EMS: Multi-Agent Voting via Efficient Majority-Then-Stopping. arXiv:2404.02863 [cs.AI] doi:10.48550/arXiv.2404.02863
- [22]Zhiwuang Liu, Ling Hu, Yunghao Tang, Tongqing Zhou, Fang Liu, and Zhiping Cai. 2026. Exploring and Exploiting Security Vulnerabilities in Self-Hosted LLM Services. In Proceedings of the ACM Web Conference 2026 (Dubai, United Arab Emirates) (WWW ’26). Association for Computing Machinery, New York, NY, USA, 2535–2546. doi:10.1145/3774904.3792097DOI
- [23]Bo Lv, Chen Tang, Yanan Zhang, Xin Liu, Ping Luo, and Yue Yu. 2024. URG: A Unified Ranking and Generation Method for Ensembling Language Models. In Findings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 4421–4434. doi:10.18653/v1/2024.findings-acl.261
- [24]Kaushal Kumar Maurya, K. Aditya Srivatsa, and Ekaterina Kochmar. 2025. Se-lectLLM: Query-Aware Efficient Selection Algorithm for Large Language Models. In Findings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 20847–20863. doi:10.18653/v1/2025.findings-acl.1072
- [25]Yasmin Moslem and John Kelleher. 2026. Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey. Transactions on Machine Learning Research (2026). https://openreview.net/forum?id=ypPgTqVqAm Survey Certification.
- [26]Zhihong Pan, Kai Zhang, Yuze Zhao, and Yupeng Han. 2026. Adaptive Model and Strategy Routing for Cost-Efficient LLM Services. In Proceedings of the ACM Web Conference 2026 (WWW ’26). Association for Computing Machinery, New York, NY, USA, 5568–5578. doi:10.1145/3774904.3792556DOI
- [27]Seamus Somersstep, Felipe Maia Polo, Allysson Flavio Melo de Oliveira, Prattyush Mangal, Mirian Silva, Onkar Bhardwaj, Mikhail Yuorkchin, and Subha Maity. 2025. CARROT: A Cost Aware Rate Optimal Router. arXiv:2502.03261 [stat] doi:10.48550/arXiv.2502.03261DOI
- [28]Jinwei Su, Qizhen Lan, Yinghui Xia, Lifan Sun, Weiyou Tian, Tianyu Shi, and Lewei He. 2026. Difficulty-Aware Agent Orchestration for Query-Specific Multi-Agent Workflows. In Proceedings of the ACM Web Conference 2026 (Dubai, United Arab Emirates) (WWW ’26). Association for Computing Machinery, New York, NY, USA, 2060–2070. doi:10.1145/3774904.3792240DOI
- [29]Shengji Tang, Jianjian Cao, Weihao Lien, Jiale Hong, Bo Zhang, Shuyue Hu, Lei Bai, Tao Chen, Wanli Ouyang, and Peng Ye. 2026. A Scalable Multi-LLM Collaboration System with Retrieval-Based Selection and Exploration-Driven Enhancement. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens (Eds.). Association for Computational Linguistics, San Diego, California, United States, 12679–12697. doi:10.18653/v1/2026.acl-long.578DOI
- [30]Selim Furkan Tekin, Fatih Ilhan, Tiansheng Huang, Sihao Hu, and Ling Liu. 2024. LLM-TOPL: Efficient LLM Ensemble by Maximising Diversity. In Findings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 1951–11966. doi:10.18653/v1/2024.findings-emnlp.698DOI
- [31]Clovis Varangot-Reille, Christophe Bouvard, Mathieu Ciancione, Antoine Gourru, Marion Schaeffer, and François Jacquenet. 2026. Doing more with less: A survey on routing strategies for resource optimisation in large language model-based systems. Journal of Artificial Intelligence Research 86 (2026).
- [32]Junlin Wang, Jue Wang, Ben Athiwatirakun, Ce Zhang, and James Y. Zou. 2025. Mixture-of-Agents Enhances Large Language Model Capabilities. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025. 33944–33963. https://proceedings.iclr.cc/paper_files/paper/2025/file/5434be94e82c54327bb9dcaf7fca52b6-Paper-Conference.pdfarxiv.org/abs/2406.04692
- [33]Xuezhai Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum? id=1PLNIMMrw
- [34]Yitao Xiao, Shaoyong Guo, Guoming Yang, Qingnan Wang, Yinlin Ren, Xuesong Qiu, and Qi Feng. 2026. RouterHGC: Optimized Router for LLM-Based Multi-Agent Systems via Heterogeneous Graph Contrastive Learning. In Findings of the Association for Computational Linguistics: ACL 2026, Maria Liakata, Vivian P. Moreira, Jiaju Zhang, and David Jurgens (Eds.). Association for Computational Linguistics, San Diego, California, United States, 31759–31789. doi:10.18653/v1/2026.findings-acl.1589DOI
- [35]Cheng Yang, Wuyang Zhang, Zhiyuan Ning, Fan Xu, Ziyang Tao, Lu Zhang, Bing Yin, and Yanyong Zhang. 2026. Breaking Model Lock-in: Cost-Efficient Zero-Shot LLM Routing via a Universal Latent Space. Proceedings of the AAAI Conference on Artificial Intelligence 40, 43 (March 2026), 36483–36490. doi:10.1609/aaai.v40i43.40970
- [36]Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations. https://openreview.net/forum?id=SkeHuCV9FDrarxiv.org/abs/1904.09675
- [37]Yiqun Zhang, Hao Li, Jianhao Chen, Hangfan Zhang, Peng Ye, Lei Bai, and Shuyue Hu. 2025. Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routing. In Proceedings of the 2025 7th International Conference on Distributed Artificial Intelligence (DAI ’25). Association for Computing Machinery, New York, NY, USA, 122–129. doi:10.1145/3772429.3772445
- [38]Qingyang Zhang, Hao Li, Chenxi Wong, Linyan Chen, Qiaosheng Zhang, Peng Ye, Shi Feng, Xinrun Wang, Jia Xu, Lei Bai, and Shuyue Hu. 2026. The Avengers: A Routing Recipe for Collective Intelligence in Language Models. Proceedings of the AAAI Conference on Artificial Intelligence 40, 41 (March 2026), 34870–34878. doi:10.1609/aaai.v40i41.40790DOI
- [39]Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. arXiv:2506.05176 [cs.CL] https://arxiv.org/abs/2506.05176
- [40]Yifan Zhang, Xinkui Zhao, Zuixin Wang, Guanjie Cheng, Yueshen Xu, Shuiguang Deng, and Jianwei Yin. 2025. LightRouter: Towards Efficient LLM Collaboration with Minimal Overhead. arXiv:2505.16221 [cs.AI] doi:10.48550/arXiv.2505.16221DOI
- [41]Richard Zhuang, Tianhao Wu, Zhaojin Wen, Andrew Li, Jiantao Jiao, and Kannan Ramchandran. 2025. Embedding Learning Compact Representations of Large Language Models. International Conference on Learning Representations 2025 (May 2025), 76913–76926.arxiv.org/abs/2410.02223
- [42]Eckart Zitzler and Lothar Thiele. 1998. Multiobjective optimization using evolutionary algorithms – A comparative case study. In Parallel Problem Solving from Nature — PPSN V, Agoston E. Eiben, Thomas Bäck, Marc Schoenauer, and Hans-Paul Schwefel (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 292–301.DOI
- [43]Lingkai Zu, Xiyue Peng, and Xin Liu. 2026. BARouter: A Budget-Adaptive Online Large Language Model Router Framework. In Proceedings of the ACM Web Conference 2026 (WWW ’26). Association for Computing Machinery, New York, NY, USA, 7645–7656. doi:10.1145/3774904.3792725DOI