WorksComputer Science
Aspect-wise Pairwise Tournaments for Test-Time Selection
Abstract
Scaling test-time compute has proven highly effective at improving LLM reasoning. A popular method is best-of-n sampling, where a model generates many candidate solutions and one is selected. This motivates the question: how can we select the best candidate? When humans make choices, we find it more natural to compare two options than judge each in isolation. Additionally, when the options become complex, we like to judge the options one aspect at a time, such as comparing two job candidates on experience, communication, etc. In the test-time setting, candidate solutions should be judged in the same way. Here we introduce Aspect-wise Pairwise Tournaments (APT), which verifies by comparison and compares by aspect. Each matchup of a single elimination tournament is decided by a vote over multiple off-the-shelf aspect verifiers, each expressing a pairwise preference along its own aspect. Across five domains and three generator LLMs, APT improves over binary multi-aspect approval, self-consistency and reward model verification, with the largest gains on open-ended tasks. We find that both components matter; neither binary approval over the same aspects nor a generic pairwise judge at a matched budget matches APT. Because aspects are specified in natural language, APT also lets users steer selection toward properties they care about, better aligning with user preferences without needing to regenerate solutions. No component is trained, so APT transfers to new domains by simply rewriting prompts.
Introduction
Best-of- sampling, where, given a task, a model generates solutions and returns only the best one, substantially improves LLM performance without additional training [12, 30]. With current LLMs, generating many candidates is cheap, but it is often hard to identify the “best” one. Many problems arise in the selection process, such as how to define “best” or detect subtle differences in candidates. In this paper, we apply intuitions from human decision-making to develop a verification method.
People rarely rank options by scoring them in isolation. An optometrist never directly asks a patient what their prescription should be. Instead, they show the patient a sequence of lenses side-by-side and ask, “is one or two better?” It’s hard to have a concrete standard for what “good” is, especially if the problem is not in your domain of expertise, as in the optometrist example. On the other hand, comparison draws attention to where two options differ, allowing an easier judgment to be made. The same can be said about LLM verifiers. Current models are not experts in every field, and two solutions might each sound plausible when read alone, but the differences that determine right from wrong come to light when they are read side by side.
Even comparison, though, gets harder as the options become more complex. A hiring committee choosing between two finalists does not ask which one is better overall. It compares them on technical depth, then on communication, then on experience, and only then weighs the results. This allows the committee to develop a definition of “better”, thus making the comparison more tractable. Different conditions also fail in different places, so an LLM verifier that compares along several axes is robust to a mistake on one of them.
Identifying a set of aspects also allows the committee to prioritize those they care most about. For instance, a committee hiring for a client-facing role will count communication more heavily than one hiring for a research role, even when looking at the same two candidates. What counts as “better” depends on each person’s preferences and needs. With a single holistic judgment, there is only one notion of quality. If the axes are written down, a user can add one they care about and decide how much it should be weighted, thereby changing which candidate is selected without generating new ones.
Figure 1. APT improves test-time selection across models and domains. Comparison of Aspect-wise Pairwise Tournaments (APT), best-of- binary approval verification (BoN-MAV), best-of- reward model verification (BoN-RM), and self-consistency using candidate solutions across three generator LLMs on two benchmarks. Left: Accuracy (%) on HMMT competition mathematics. Right: pawn score on Chess Optimality dataset (higher is better). Full results are in Table 1.
Applying these observations on human decision-making to test-time verification, we introduce Aspect-wise Pairwise Tournaments (APT). It verifies by comparison and compares by aspect. APT samples candidate solutions from a generator LLM and runs a single elimination tournament over them. Each matchup is decided by a vote across off-the-shelf aspect verifiers, each stating which of the two candidates it prefers along its own aspect.
We evaluate APT on competition math, a hard multiple-choice benchmark, and two chess reasoning tasks, using three generator LLMs. APT achieves the best result in 14 of 15 generator-domain pairs against self-consistency, reward model verification, and binary multi-aspect approval (§ 4.1). Adding a weighted aspect steers selection toward a user’s preference, selecting concise solutions on the math benchmarks and aggressive moves on Chess Optimality (§ 4.2). Both replacing the aspect vote with a generic comparative judge and binary approval over the same aspects fall short (§ 4.3).
Our contributions are as follows:
Aspect-wise Pairwise Tournaments. A training-free test-time selection method that decides each matchup of a knockout tournament by a vote over aspect-specific pairwise verifiers.
Steering selection toward user preferences. Because aspects are natural-language prompts, users can add and weight aspects at inference time to change what is selected, without regenerating candidates or training verifiers.
Isolating each component. BoN-MAV ([29]), which uses the same aspect verifiers with binary approval, isolates the contribution of pairwise comparison; a pairwise tournament with a generic judge at matched budget isolates the contribution of aspect decomposition. We find that both contributions improve accuracy.
Related Work
Test-time compute scaling. Increasing inference-time computation can improve LLM performance without changing model parameters ([47, 58, 63]). As larger sample pools become more likely to contain a correct answer [4], selecting among candidates becomes increasingly important. Existing approaches include best-of- selection with reward models [48, 12], self-consistency for structured and free-form answers [55, 7], and search over intermediate reasoning steps [60, 30]. Welleck et al. [56] survey these approaches as meta-generation. However, additional computation does not guarantee better selection: majority-vote accuracy can decline on tasks mixing easy and hard queries [6], while verifier false positives can limit the accuracy achievable through resampling (Stroebl et al., 2026). These findings motivate improving the judgments used to select candidates. APT studies aspect-specific pairwise verification as one approach to this problem.
LLM judges and verifiers. LLMs can evaluate responses through prompted judgments [64, 32, 19] or serve as trained generative verifiers for best-of- selection [62]. FLASK and Prometheus make evaluation more specific by decomposing quality into explicit criteria [61, 25]; Constitutional AI similarly uses written principles to guide model-generated feedback [1]. APT adopts criterion-based evaluation through separate aspect verifiers. Nevertheless, verification remains difficult. Self-verification and iterative refinement can improve responses [57, 36], but intrinsic self-correction often fails on reasoning tasks [21], and even strong judges struggle to distinguish objectively correct and incorrect responses on challenging pairs [50]. Pairwise judgments are also sensitive to candidate order, particularly when responses are similar in quality [54, 46], and judges may favor their own generations [43]. We address position bias in § 3.1 and discuss the implications of using same-family generators and verifiers in § 5.
Multi-agent evaluation and verification. Multi-agent LLM systems combine model outputs through interaction, as in debate [14, 28], or through layered aggregation [53]. Debate has also been proposed for scalable oversight [22], with evidence that more capable debaters can help non-expert judges identify truthful answers [24]. Other studies find that simple ensembling explains much of the improvement attributed to interaction: majority voting captures most of the gains from multi-agent debate [10], and sampling followed by voting improves with the number of agents [27]. For evaluation, panels of smaller judges from different model families can outperform a single larger judge [51], although verification remains a common failure point in multi-agent systems [5].
The closest verification framework is Multi-Agent Verification (MAV) [29], which aggregates binary approvals from prompted aspect verifiers and demonstrates the value of scaling the verifier panel. MAV evaluates each candidate independently. APT instead asks each aspect verifier to compare two candidates, then combines their preferences by majority vote within each tournament match. We compare APT to BoN-MAV in § 4.
Pairwise comparison for test-time selection. Pairwise preferences have a long history in statistical ranking [2] and underpin preference-based model training [11, 42] and large-scale model evaluation [9]. For LLM evaluation, pairwise judgments can agree more closely with human preferences than pointwise scores [64, 34]. Pairwise prompting has also proved effective for zero-shot document ranking [45] and NLG evaluation [35], while LLM-Blender trains a pairwise ranker to select among outputs from different models [23].
Several works apply these ideas directly to test-time selection. PairJudge trains a pairwise reward model on PairJudge-432K and selects mathematical solutions through knockout tournaments [33]. Guo et al. [20] train reward reasoning models with reinforcement learning and aggregate pairwise judgments using Elo-based round-robin or knockout selection. Zhou et al. [65] evaluate LLM judges as selectors using round-robin comparisons in both candidate orders. On the theoretical side, Chen et al. [8] show that knockout selection can achieve vanishing failure probability under assumptions including a generator with nonzero success probability and a comparator that performs better than chance. Their repeated direct comparisons are closely related to our PV (no aspects) ablation in § 4.3.
APT combines pairwise tournament selection with prompted multi-aspect verification. Its contribution is this training-free combination and its empirical evaluation, testing whether aspect-specific voting improves comparison over repeated generic judgments, and whether pairwise preferences provide useful selection information beyond binary approval (§ 4.3, Appendix § 6.§ 4).
Tournament aggregation. Tournament rules determine how individual comparisons produce a final selection ([39, 26]). Single elimination requires only matches but depends on seeding ([52]); Copeland’s rule uses a full round-robin and provides a more comparison-intensive reference ([13]). Work on noisy comparisons studies selection when individual judgments can be wrong ([15]), including settings where a comparison’s error persists across repeated queries ([3]). Within each match, APT aggregates aspect-verifier votes. Condorcet’s jury theorem motivates majority aggregation when voters are independent and individually better than chance ([38, 18]), but shared errors among LLM verifiers limit its direct applicability. We empirically compare single-elimination, double-elimination, and Swiss-style tournaments in Appendix § 6.10.
Aspect-wise Pairwise Tournaments (APT)
We consider the standard best-of- setting. Given a question , a generator LLM samples candidate solutions , and a selection procedure returns one of them as the final output. APT selects through a single elimination tournament over the candidates. In each matchup, a set of aspect verifiers compares the two candidates, each along a different aspect such as logical soundness or domain knowledge, and each states which candidate it prefers. The candidate preferred by more verifiers advances, and the last candidate remaining is the final selection.
§ 3.1 defines pairwise preference verification with a single judge. § 3.2 decomposes each comparison across multiple aspect verifiers. § 3.3 describes how the resulting matchup outcomes are combined into a single selection through a single elimination tournament, and § 3.4 assembles the full algorithm.
Pairwise Preference Verification
In pairwise preference verification, a judge is presented with two candidate solutions at once and asked which is better. Formally, given candidates and , a judge produces
where denotes “no preference.” The “no preference” option exists because the two options might both be equally correct or equally incorrect, and forcing a choice in those cases would add noise to the tournament while also possibly increasing token or thinking cost.
Position bias. LLM judges have a tendency to prefer candidates based on their given ordering in the prompt ([64, 54, 46]). We therefore query the judge twice per comparison, once with the candidates presented as and once as , and keep the preference only if both orderings agree. When they disagree, the preference is assigned uniformly at random between the two candidates. We directly measure the position bias of our verifiers in Appendix § 6.8.
Decomposing Comparisons by Aspect
A single comparison asks the judge to weigh every dimension of quality at once, a task that gets harder with more complex candidates, so we instead decompose each comparison into several narrower ones. Following Lifshitz et al. [29], we use aspect verifiers: off-the-shelf LLMs prompted to evaluate a specific property of a solution. Each aspect verifier is specified by a base LLM, the aspect it checks (e.g., logical soundness, domain knowledge), and a verification strategy (e.g., step-by-step analysis, identifying mistakes). In Lifshitz et al. [29], each aspect verifier evaluates one candidate and returns a binary approval. In APT, each aspect verifier instead acts as a pairwise judge restricted to its own aspect, producing (Figure 2).
Figure 2. Multi-aspect pairwise verification for a single matchup. Two candidate solutions are compared by five aspect verifiers. Solution A finds 8 valid arrangements but fails to account for distinct rotations; Solution B correctly multiplies by 7 positions to get 56. Each verifier reads both full solutions and selects the one it judges to be better along its assigned aspect. Four of five verifiers identify Solution A’s counting error and prefer Solution B, while the units check verifier finds no meaningful difference. This illustrates the core advantage of pairwise verification: the rotation counting error in Solution A is subtle enough that it might be missed under independent binary evaluation, but becomes salient when a verifier can directly compare the two counting strategies side by side.
The outcome of each matchup is decided by vote. Writing for the number of verifiers preferring , and analogously,
Abstentions count toward neither side, so a matchup is decided by a plurality of the verifiers that express a preference rather than by a strict majority of . Ties, including the case where every verifier abstains, are broken uniformly at random.
§ 4.3 tests the contribution of aspects by comparing APT against a single generic judge.
Tournament Aggregation
Given candidate solutions sampled from a generator LLM, pairwise preferences must be aggregated into a single selected output. We adopt single elimination tournaments as the aggregation mechanism (Figure 3). Candidates are paired, and the majority vote winner of each matchup advances to the next round. This process repeats until one candidate remains.

Figure 3. Tournament aggregation for candidate selection. Given 8 candidate solutions, a single elimination tournament selects the final output. At each matchup, aspect verifiers compare both solutions and the candidate preferred by the majority advances. The full tournament requires only matchups, comparable in total verifier queries to binary approval methods which require independent evaluations.
Single elimination requires exactly pairwise matchups, making it the most compute-efficient tournament format. This efficiency does make it vulnerable to cases where a strong candidate could be eliminated early if it draws a difficult matchup. However, we find empirically that this is rarely a concern in practice (Appendix § 6.7).
Aspect-wise Pairwise Tournaments
We introduce Aspect-wise Pairwise Tournaments (APT) as a test-time compute algorithm that uses pairwise preference tournaments to aggregate judgements from aspect verifiers. APT consists of three steps:
Generate: Sample candidate solutions from a generator LLM.
Compare: For each matchup in the single elimination bracket, present both candidate solutions to a set of aspect verifiers and determine the winner by majority vote over the verifiers’ preferences.
Select: Return the tournament winner as the final output.
The APT algorithm is detailed in Algorithm 1. Formally, the selected output is:
where the SingleElim function applies a single elimination tournament using pairwise matchups judged by the verifier set .
APT scales test-time compute along two dimensions: the number of candidate solutions and the number of verifiers . Increasing provides more candidate diversity, while increasing improves the reliability of each pairwise comparison. We test these in Appendix § 6.§ 5 and Appendix § 6.6. Generation can be fully parallelized since all candidates are sampled independently. Each round of the single elimination tournament can also be parallelized, as each of the verifiers and matchups are independent from the others.
Experiments
We evaluate APT across five domains using three generator LLMs. We aim to address the following questions: (1) how APT compares to existing methods; (2) whether it can be steered toward user preferences; and (3) what each component, aspects and pairwise preferences, contributes.
Domains. We evaluate on five domains spanning mathematical reasoning, general knowledge, and strategic reasoning:
AIME. Competition mathematics problems from the American Invitational Mathematics Examination (2022-2026 exams), 150 problems total
HMMT. Problems from the Harvard-MIT Mathematics Tournament (February and November 2025, February 2026), 93 problems total.
HLE. A subset of the Humanity’s Last Exam (HLE) benchmark ([44]), filtered to text-only multiple-choice questions for ease of evaluation. Problems span diverse academic domains (biology, medicine, science, etc.). Tested on 300 problems.
Chess Validity. A state-tracking task over chess games. Given a sequence of moves with the final move’s destination square omitted, the model must supply a valid destination square. Problems were drawn from synthetic games with short length. Tested on 300 problems.
Chess Optimality. A strategic reasoning task over positions from grandmaster games. Given the first 13 moves of a game, the model must suggest the best 14th move for White. Unlike the other datasets, this uses continuous scoring via Stockfish (depth 20), with the score reported in pawn units. Tested on 300 problems.
Generator and Verifier Setups. We evaluate three generator models spanning different model families and capability levels: Gemini-3.1-Pro, GPT-5.2, and Qwen3-8B. For each model, we sample candidate solutions per problem. Generation with Gemini-3.1-Pro used Gemini-2.5-Flash as the verifier model while generation with GPT-5.2 and Qwen3-8B used GPT-5-mini as the verifier model. Appendix § 6.§ 2 includes details on verifier setups and generator/verifier prompts.
Baselines. We compare APT against the following test-time compute methods, which all also use candidate solutions:
Self-Consistency: Most common answer among 8 candidate solutions ([55]). Ties broken arbitrarily.
BoN-RM: Answer with highest reward model score, using Skywork-Reward-V2-Llama-3.1-8B ([31]), which is currently the top performing model on RewardBench2 ([37]).
BoN-MAV: Answer with highest number of approvals from binary output aspect verifiers ([29]). Ties are broken uniformly at random among the tied candidates, since we take equal approval counts to indicate equal estimated quality. BoN-MAV uses the same aspect verifiers and the same base verifier models as APT.
Main Results
Table 1 presents the main results across all three generator models and five domains. We report accuracy (%) for AIME, HMMT, HLE, and Chess Validity, and pawn score for Chess Optimality (higher is better for all metrics). APT is compared against self-consistency, reward model verification, and binary MAV, all using candidate solutions.
| Generator LLM | Method | AIME | HMMT | HLE | Chess Val. | Chess Opt. |
| Gemini-3.1-Pro | Self-Consistency | 96.67 | 50.54 | 48.00 | 99.00 | 0.031 |
| BoN-RM | 97.33 | 50.54 | 43.33 | 98.00 | 0.107 | |
| BoN-MAV | 96.67 | 50.54 | 46.33 | 97.33 | 0.056 | |
| APT | 98.00 | 54.84 | 49.67† | 98.67† | 0.166† | |
| GPT-5.2 | Self-Consistency | 66.00 | 26.88 | 12.00 | 52.67 | 1.866 |
| BoN-RM | 68.00 | 30.11 | 11.67 | 58.33 | 1.666 | |
| BoN-MAV | 78.00 | 36.56 | 12.67 | 62.00 | 1.856 | |
| APT | 82.67† | 39.78 | 15.00† | 70.00† | 1.982† | |
| Qwen3-8B | Self-Consistency | 21.33 | 7.53 | 11.33 | 10.67 | 0.586 |
| BoN-RM | 24.00 | 6.45 | 12.00 | 11.00 | 0.577 | |
| BoN-MAV | 28.67 | 7.53 | 11.33 | 10.33 | 0.677 | |
| APT | 31.33 | 10.75 | 14.33 | 15.67 | 1.221 |
Table 1. Main results across models and domains. Performance comparison of four test-time verification methods using candidate solutions across three generator LLMs. Bold indicates the best result among methods within each model group. Accuracy (%) is reported for AIME, HMMT, HLE, and Chess Validity; pawn score is reported for Chess Optimality. Higher is better. †APT exceeds BoN-MAV with a 95% paired-bootstrap interval that excludes zero (Table 5, Appendix § 6.§ 4); Qwen3-8B was not tested.
APT outperforms all other baselines in nearly all model-domain pairs. APT achieves the highest score in 14 out of 15 model-domain pairs. We find that APT yields the largest gains in performance on the open-ended task, Chess Optimality. On these problems, since there is no single correct answer, the combination of multiple aspects and pairwise comparisons force the verifier to make a more well-rounded judgement. These types of judgements are where we would expect the method to transfer (§ 5). For a similar reason, we would expect APT to improve on Humanity’s Last Exam well, since the difficulty of the problems themselves also lead to more difficult verification tasks, where the combination of multiple aspects and pairwise comparison could be more illustrative for verification. However, we also observe that the gains from APT on the HLE dataset is limited by the ability of the generator to create valid solutions (Appendix § 6.9). Improvements from APT are also more limited in already saturated domains, like Gemini-3.1-Pro on AIME or Chess Validity; here the marginal benefit of APT is small because there is limited room to improve. We report additional failure analysis results in Appendix § 6.9.
Weak-to-strong generalization. We also observe that APT enables weak-to-strong generalization, i.e. combining weaker verifiers can improve the outputs of stronger generators. In our setup, aspect verifiers use smaller models from the same family as the generator (GPT-5-mini verifiers for GPT-5.2 generation, Gemini-2.5-Flash verifiers for Gemini-3.1-Pro generation). Despite this capability gap, APT consistently improves solution quality over both self-consistency and reward model baselines. This suggests that even when a weaker verifier may struggle to determine whether a solution from a stronger generator is correct in isolation, it can still reliably identify which of two such solutions is better. This further justifies our use of pairwise comparisons, as these judgments can be made using weaker models, enabling APT to be carried out even on small compute budgets.
Self-improvement We also investigate whether APT can enable self-improvement, where the same base model serves as both the generator and the set of aspect verifiers. We evaluate this setting using two models: Gemini-2.5-Flash and GPT-5-mini, which are both relatively small and cost-effective compared to their larger analogues used in the main experiments.
As shown in Table 2, APT with self-verification consistently achieves the best performance across both models in all evaluated settings. Across both models, pairwise self-verification provides the strongest selection signal on the harder and more diverse benchmarks (such as HMMT, HLE, and Chess Optimality), while self-consistency remains competitive on tasks where accuracy is already high (such as AIME). APT with self-verification is also more accurate than BoN-MAV with self verification, indicating that pairwise preferences might set an easier task for smaller models. These results demonstrate that APT does not require an exceptionally capable generator model to be effective. Even smaller models can improve their own outputs by comparing candidate solutions head-to-head.
| Model | Method | AIME | HMMT | HLE | Chess Val. | Chess Opt. |
| Gemini-2.5-Flash | Self-Consistency | 88.00 | 37.63 | 8.00 | 62.67 | 0.035 |
| Gemini-2.5-Flash | BoN-MAV | 84.67 | 32.26 | 10.33 | 62.00 | 0.047 |
| Gemini-2.5-Flash | APT | 90.00 | 38.71 | 13.33 | 73.00 | 0.086 |
| GPT-5-mini | Self-Consistency | 53.33 | 25.81 | 10.00 | 43.67 | 0.611 |
| GPT-5-mini | BoN-MAV | 57.33 | 29.03 | 10.67 | 49.00 | 0.614 |
| GPT-5-mini | APT | 64.67 | 35.48 | 11.33 | 53.33 | 0.695 |
Table 2. Self-improvement experiments. The same (small) model is used as both generator and verifier.
Steering Selection Toward User Preferences
Because each aspect verifier is simply a prompt, APT lets users decide what “better” means. A user who cares about a property beyond correctness can add an aspect for it, and weight that aspect’s vote according to how much it matters to them. This lets APT select outputs that are aligned with a user’s preferences, without any training.
Setup. We add one new aspect verifier to the existing set: conciseness on AIME and HMMT, and aggressiveness (a preference for attacking, initiative-seeking play) on Chess Optimality. Both use the same base verifier models as our main experiments. Each matchup is decided by a weighted vote, , where existing verifiers keep and the new aspect receives weight , so recovers standard APT. We reuse the candidates from our main experiments. For conciseness we measure the token length of the selected solution. For aggressiveness we measure whether the selected move is forcing (a check or a capture), determined by parsing the move against the board position.
Results. Upweighting the new aspect moves the selected output toward the user’s preference in every setting (Figure 4). Going from to , the average length of the selected solution decreases by 13% (Gemini-3.1-Pro) and 16% (GPT-5.2) on AIME, and by 8% and 14% on HMMT. On Chess Optimality, the
Figure 4. Steering APT toward user-specified preferences. A single new aspect verifier is added to the existing set and its vote is weighted by . Left, Center: a conciseness aspect on AIME and HMMT, measured by the average token length of the selected solution. The dashed line is the average length of the shortest candidate in each pool, the lowest achievable value. Right: an aggressiveness aspect on Chess Optimality, measured by the number of problems where the selected move is forcing (a check or capture). The dotted line is the number of problems whose candidate pool contains at least one forcing move.
| Dataset | APT | PV-1 | PV-5 | APT | PV-1 | PV-5 |
| AIME | 98.00 | 94.67 | 95.33 | 82.67 | 66.67 | 69.33 |
| HMMT | 54.84 | 47.31 | 49.46 | 39.78 | 27.96 | 31.18 |
| HLE | 49.67 | 42.67 | 43.33 | 15.00 | 10.33 | 11.00 |
| Chess Val. | 98.67 | 96.00 | 96.67 | 70.00 | 59.33 | 61.67 |
| Chess Opt. | 0.166 | 0.051 | 0.093 | 1.982 | 1.314 | 1.519 |
caption: The aspect vote is not cosmetic. APT compared against pairwise tournaments whose matchups are decided by a generic “which solution is better?” query instead of by a vote over aspect verifiers, at . PV-1 samples one generic judgment per matchup; PV-5 samples five and aggregates them by the same majority rule as APT, matching APT’s verification budget. Verifiers are Gemini-2.5-Flash and GPT-5-mini respectively. Accuracy (%) except Chess Optimality (pawns).
:::
number of problems where APT selects a forcing move increases by 36% (Gemini-3.1-Pro) and 42% (GPT-5.2). Measured against the best achievable value in each pool, the most concise candidate or the maximum number of available forcing moves, closes 36–79% of the gap on the math tasks and 57–61% on chess.
Importantly, the new aspect adds a preference on top of the existing verifiers rather than replacing them. The existing verifiers continue to vote on correctness, so APT is inclined to select the most concise solution out of the existing set of correct answers. We saw that accuracy on AIME and HMMT decreases by at most three problems in any generator-dataset-weight setting, which shows steering toward conciseness comes at little cost to correctness.
Ablations and Additional Results
We first check that the main results do not depend on the choice of . We then test the two components of APT separately. The comparison against BoN-MAV in Table 1 already isolates the pairwise structure, since BoN-MAV uses the same aspect verifiers and base models; the ablations below isolate the aspect vote, by replacing it with a generic judge and by varying the number of aspect verifiers.
Scaling the number of candidates. Table 1 fixes , leaving open whether the ordering of methods is an artifact of that choice. We re-ran every method at every for both closed-source generators and all five datasets (Figure 5, Appendix § 6.§ 5). All three methods improve with more candidates, though not at the same rate. In every generator-dataset combination except Gemini-3.1-Pro on Chess Validity and GPT-5.2 on Chess Optimality, APT@ is at or above both baselines over the full range of , and in several cases (Gemini-3.1-Pro on HLE, GPT-5.2 on Chess Validity), the gap widens with rather than closing. The results therefore understate the method’s full potential.
Figure 5. Scaling curves for all methods. Accuracy (pawn score for Chess Optimality) against the number of candidate outputs , for Gemini-3.1-Pro (left) and GPT-5.2 (right) across all five datasets. Dashed line is pass@1. All three methods improve with , but APT@ is at or above both baselines over the full range in nearly every combination, and its margin widens with on Chess Optimality in particular.
Aspect voting versus a single generic judge. We construct PV-1, identical to APT with the same tournament setup, the same position-bias handling, and the same base verifier model, except that each matchup is decided by a direct “which solution is better?” query with no aspect-specific instruction. This is the closest analogue in our setup to the comparator of Chen et al. [8]. One such query per matchup uses fewer verifier calls than APT’s verifiers, so we also report a budget-matched variant in which five independent generic judgments are sampled per matchup and aggregated by the same majority rule.
APT outperforms both variants on every generator-dataset pair (Table 3). Most of what the aspect vote contributes therefore comes from the diversity of the criteria rather than from the extra samples. Also, five draws from one generic judge are susceptible to correlated errors while five verifiers along different axes can diversify errors. Thus, we find that both the pairwise structure without aspects (PV-5) and the aspects without the pairwise structure (BoN-MAV) are weaker than APT.
Number of aspect verifiers. Removing verifiers from the full set one at a time and recomputing accuracy for each resulting subset (Figure 6, Appendix § 6.6) shows accuracy increasing with the number of verifiers for both APT and BoN-MAV on all five datasets, so the aspects contribute complementary rather than redundant signal, and performance does not rely on any single aspect. APT is at or above BoN-MAV at nearly every verifier count, so the pairwise advantage is not an artifact of a particular number or subset of aspects.
Figure 6. Verifier-count ablation over all subsets. Accuracy against the number of aspect verifiers for APT (purple) and BoN-MAV (blue), with shaded regions spanning all subsets of each size, for Gemini-3.1-Pro (left) and GPT-5.2 (right). Accuracy increases with for both methods on all five datasets, indicating that the aspects are complementary rather than redundant, and no single verifier accounts for the bulk of the performance. APT is at or above BoN-MAV at nearly every verifier count.
Robustness and signal quality. Three further measurements, reported in full in the appendix, support the design. Sampling five random bracket seedings per problem changes APT’s score by little relative to the gaps in Table 5 (Appendix § 6.7), because most problems admit several correct candidates and a poor draw for one does not affect the rest. The verifier majority prefers the correct solution in 78.4–92.3% of correct-versus-incorrect matchups across domains and models (Appendix § 6.8), and individual verifier preferences are transitive in 89–98% of candidate triples. These explain why the aggregation rule barely matters (Appendix § 6.10) and provides the basis for the use of single-elimination tournaments.
| Comparison | Gemini-3.1-Pro | GPT-5.2 |
| AIME − cons | +1.33 [−0.67, +3.33] | +16.67 [+10.67, +22.67] |
| AIME − BoN-MAV | +1.33 [−0.67, +4.00] | +4.67 [+1.33, +10.67] |
| HMMT − cons | +4.30 [+1.08, +8.61] | +12.90 [+4.30, +21.51] |
| HMMT − BoN-MAV | +4.30 [+0.00, +9.68] | +3.23 [−1.08, +8.60] |
| HLE − cons | +1.67 [+0.33, +4.33] | +3.00 [+1.00, +5.33] |
| HLE − BoN-MAV | +3.33 [+0.66, +6.00] | +2.33 [+0.67, +4.67] |
| Chess Val. − cons | −0.33 [−2.00, +1.33] | +17.33 [+12.67, +22.67] |
| Chess Val. − BoN-MAV | +1.33 [+0.33, +3.00] | +8.00 [+4.33, +12.67] |
| Chess Opt. − cons | +0.13 [+0.03, +0.26] | +0.12 [+0.04, +0.22] |
| Chess Opt. − BoN-MAV | +0.11 [+0.05, +0.28] | +0.13 [+0.04, +0.24] |
Table 5. Paired differences with 95% bootstrap CIs at . Mean paired difference between APT@8 and each baseline, with a 95% interval from a paired bootstrap over problems (10,000 resamples). Accuracy points except Chess Optimality (pawns). Bold marks intervals that strictly exclude zero.
Discussion and Future Work
We have introduced APT, which verifies candidate solutions at test-time in the same way people tend to choose between complex options, by comparing them one aspect at a time. Each matchup of a knockout tournament is decided by a vote across off-the-shelf aspect verifiers, with each verifier stating a pairwise preference along its own aspect. Pairwise comparison and multi-aspect verification each have precedent, but our ablations show the combination is not redundant, as binary approval over the same aspects and a generic pairwise judge at a matched budget both fall short of APT. The method also enables weak-to-strong generalization and self-improvement in weaker models, and requires no training at any stage, so adapting it to a new domain is easy.
We find the gains from APT are not uniform. They are largest on Chess Optimality, the one open-ended task in our suite, where no strict answer key exists, and a verifier has more leeway in determining which candidates are “better”. They are smallest on near-saturated benchmarks, where most candidates are already correct and majority voting is hard to beat. This suggests that pairwise, aspect-level comparison matters most when quality is measured in degrees rather than a binary property that can be checked in isolation.
Decomposing verification into aspects has a second benefit beyond accuracy. It also makes the notion of “better” explicit and editable. Adding a single weighted aspect was enough to steer selection toward more concise math solutions and more aggressive chess moves, with little cost to correctness on the math tasks, and without needing to generate new candidates. A trained judge can only encode a fixed notion of quality, but APT’s can be changed by the user at inference time. Natural extensions include steering along several preferences at once, generating aspects automatically from a user’s description of what they want, and applying steering in domains where preferences vary more widely across users.
Limitations. We only test generators and verifiers from the same model family. The Qwen3-8B setting, which uses GPT-5-mini verifiers, offers a partial cross-family check, but swapping verifier families would test this directly. Most of our evaluation is also on domains with verifiable answers. We did not evaluate summarization, long-form writing, or code generation, though the Chess Optimality results suggest these are where both the accuracy and steering benefits of APT are most likely to matter. Finally, although APT requires no training, the tournament adds inference cost. With candidates, aspect verifiers, problem-prompt length , and mean solution length , APT issues verifications over the problem plus two solutions, since each comparison is queried in both orderings, costing roughly input tokens. APT is therefore up to about four times as expensive as BoN-MAV in input tokens when . It is also slower: BoN-MAV’s verifications run in a single parallel batch, whereas tournaments require sequential rounds.
Appendix
APT Algorithm
Algorithm 1: APT: Aspect-wise Pairwise Tournaments
Input: Question , generator LLM , aspect verifiers , number of candidates
Output: Selected solution
Step 1: Generate candidates
Sample candidate solutions from given
Step 2: Single elimination tournament
; // active candidates
while do
Pair candidates in into matchups
foreach matchup do
// Pairwise preference voting
foreach verifier do
; // query in both orderings; see § 3.1
if then ;
else if then ;
else abstain; // no preference
end
if then
else if then
else
; // tie or all abstain
end
end
end
Step 3: Return winner
the single remaining element of
return
Experimental Setup
Table 4 lists the aspect verifiers used for each of the five domains tested.
| Base Model | Aspect | Strategy | AIME/HMMT | HLE | Chess Val. | Chess Opt. |
| GPT-5-mini / Gemini-2.5-Flash | Unit Conversions | Step-by-Step | • | |||
| GPT-5-mini / Gemini-2.5-Flash | General Correctness | Summarize | • | • | • | |
| GPT-5-mini / Gemini-2.5-Flash | General Correctness | Identify Mistakes | • | • | • | • |
| GPT-5-mini / Gemini-2.5-Flash | Domain Knowledge | Direct | • | • | • | • |
| GPT-5-mini / Gemini-2.5-Flash | Edge Cases | Direct | • | |||
| GPT-5-mini / Gemini-2.5-Flash | Logical Soundness | Step-by-Step | • | • | • | |
| GPT-5-mini / Gemini-2.5-Flash | General Correctness | Diff Comparison | • | |||
| GPT-5-mini / Gemini-2.5-Flash | Factual Accuracy | Step-by-Step | • | • | • |
Table 4. Aspect verifiers used per domain. Each verifier is defined by a base model and an aspect-strategy pair. The specific base model depends on the generator model family (i.e. Gemini verifiers used for Gemini generations). Verifiers marked with • are included in the domain-specific subset for that domain.
The generator and verifier models used throughout were: gemini-3.1-pro-preview ([17]), gemini-2.5-flash ([16]), gpt-5.2-2025-12-11 ([41]), gpt-5-mini-2025-08-07 ([40]), and Qwen3-8B ([59]).
We used domain-specific prompts for the generator LLMs, which are documented in Table 13. Prompts for aspect verifiers consisted of domain-specific system prompts combined with an aspect-specific verification prompt. These are documented in Table 14 and Table 15.
| Domain | Prompt |
| AIME / HMMT | You are a helpful assistant skilled in math problem-solving. ({IF AIME:} Always express your final numerical answer as a three digit integer. If your final numerical answer is less than three digits, fill in leading zeros. For example, if your final numerical answer is ’38’, express your answer as ’038’.) Always end your solution with the final numerical answer in latex, using ’\boxed{<answer>}’. If there is no solution, reply with an empty boxed ’\boxed{}’. Please solve the following math problem step by step. Think carefully about your solution. QUESTION: {problem} Provide your detailed solution below: |
| HLE | Answer the following multiple choice question. Think step by step before answering, and then output the answer in the format of "The answer is (X)", where X is the LETTER of the correct answer. QUESTION: {problem} Think step by step, then answer with EXACTLY "The answer is (X)", where X is the LETTER of the correct answer. Do not include the answer text itself, only the letter. After stating your answer, give a detailed explanation of why this answer is correct. |
| Chess Validity | Given the following (in-progress) chess game, please complete the notation for the last shown move by filling in the destination square. Begin your solution by stating the answer. State your answer EXACTLY in the form "The answer is (X)", where X follows the regex [a-h][1-8], for example "The answer is (e5)." After stating your answer, give a brief explanation of why answer is valid. GAME: {problem} |
| Chess Optimality | You are an expert chess player. Given the following chess position in FEN notation, suggest the MOST OPTIMAL next move for White. Think step by step about the position, threats, and candidate moves before deciding. Then state your suggested move EXACTLY in the form "The answer is (MOVE)", where MOVE is in Standard Algebraic Notation (SAN). Examples: "The answer is (Nf3)", "The answer is (e4)", "The answer is (O-O)", "The answer is (Bxc6)". After stating your move, give a brief explanation of why this move is optimal. POSITION (FEN): {problem} |
Table 13. Generator prompts. The system prompt used to sample candidate solutions for each domain. {problem} is replaced with the problem text.
| Domain | System Prompt + Input Format |
| AIME / HMMT | You are a critical verifier tasked with comparing two mathematical solutions. You will be presented with a question and two proposed solutions. Your job is to determine which solution is better based on the specified criteria. Follow the instructions. QUESTION: {question} SOLUTION A: {solution_a} SOLUTION B: {solution_b} |
| HLE | You are a critical verifier tasked with comparing two multiple-choice answers. You will be presented with a question and two proposed solutions. Your job is to determine which solution is better based on the specified criteria. Follow the instructions. QUESTION: {question} SOLUTION A: {solution_a} SOLUTION B: {solution_b} |
| Chess Validity / Chess Optimality | You are a critical verifier tasked with comparing solutions related to moves in the game of chess. You will be presented with a question and two proposed solutions. Your job is to determine which solution is better based on the specified criteria. Follow the instructions. QUESTION: {question} SOLUTION A: {solution_a} SOLUTION B: {solution_b} |
Table 14. Pairwise verifier system prompts by domain. The system-level prompt prepended to all pairwise verifier queries. {question}, {solution_a}, and {solution_b} are replaced with the problem text and the two candidate solutions.
| Aspect–Strategy | Comparison Instruction |
| Unit Conversions / Step-by-Step | Compare how well each solution handles units in calculations. Which solution has better unit handling? Think out loud. |
| General Correctness / Summarize | Compare the overall quality and clarity of both solutions. Which solution is more comprehensive and well-explained? Think out loud. |
| General Correctness / Identify Mistakes | Compare both solutions for common mistakes and calculation errors. Which solution has fewer mistakes? Think out loud. |
| Domain Knowledge / Direct | Compare how well each solution applies domain knowledge and best practices. Which solution demonstrates better domain expertise? Think out loud. |
| Edge Cases / Direct | Compare how well each solution handles edge cases and boundary conditions. Which solution better addresses potential edge cases? Think out loud. |
| Logical Soundness / Step-by-Step | Compare the logical soundness of both solutions. Which solution follows more logical reasoning? Think out loud. |
| General Correctness / Diff Comparison | Compare the difficulty and sophistication of both solutions. Which solution demonstrates better problem-solving approach? Think out loud. |
| Factual Accuracy / Step-by-Step | Compare the factual accuracy of both solutions step by step. Which solution has more accurate facts? Think out loud. |
Table 15. Pairwise verifier aspect prompts. The aspect-specific instruction appended after the system prompt and input for each verifier. All prompts end with the format instruction: To indicate your final answer, you MUST end your response with ‘FINAL PAIRWISE PREFERENCE: A’ if Solution A is better, ‘FINAL PAIRWISE PREFERENCE: B’ if Solution B is better, or ‘FINAL PAIRWISE PREFERENCE: NO PREFERENCE’ if both solutions are equally good or equally flawed. This suffix is omitted above for brevity.
Chess Optimality Scoring. A proposed move is scored by playing it on the board and evaluating the resulting position with Stockfish at search depth 20. The score is the evaluation in pawn units (centipawns divided by 100) from White’s perspective, so higher values indicate better moves. Positions with a forced mate are assigned . Illegal or unparseable moves, and responses from which no move can be extracted, receive a neutral score of 0.
Additional Qualitative Illustrations
Figure 7 and Figure 8 provide additional examples of how APT selects a final solution based on a set of candidate solutions. Figure 7 illustrates APT on a problem from the 2024 AIME I exam. Figure 8 demonstrates the algorithm on a chess move validity problem. Both figures are designed in the same style as Figure 2 and Figure 3.
Figure 7. APT applied to AIME problem. Top: Pairwise verification for a matchup between an incorrect solution and a correct solution. Both solutions identify as the -symmedian, but Solution A uses the wrong ratio for , squaring , and then wrongly assumes . Three of five verifiers detect these errors and prefer Solution B; the units verifier abstains, and the edge-case verifier is misled into accepting A’s squared ratio. This illustrates how majority vote across diverse aspects provides robustness to individual verifier mistakes. Bottom: The full single elimination tournament over 8 candidates (2 correct, 6 incorrect). Incorrect candidates can advance past other incorrect candidates, but the correct solutions win every matchup against incorrect ones; both reach the final, and one is selected as the output.
Figure 8. APT applied to a chess move validity problem. Top: Pairwise verification for a quarterfinal matchup. Given a partial game, the model must supply the destination square for the final move. Solution A correctly identifies that the rook on h7 can return to the empty h8 square; Solution B proposes h5 but overlooks that Black’s own pawn on h6 blocks the rook’s path. All five verifiers unanimously prefer Solution A. Bottom: The full tournament over 8 candidates, of which only one is correct. Despite 7 of 8 candidates being incorrect, pairwise verification reliably eliminates them.
Testing Significance
Scaling the Number of Candidates
Figure 5 reports accuracy against the number of candidate outputs for APT, BoN-MAV and self-consistency, for both closed-source generators and all five datasets, with pass@1 shown as a reference line. When is not a power of 2, the last candidate gets a first-round bye.
Verifier-Count Ablation
To test whether the aspect verifiers carry complementary signal, we remove verifiers from the full set one at a time and recompute accuracy for every resulting subset, for both APT and BoN-MAV. Figure 6 plots accuracy against the number of verifiers , with the shaded region spanning the subsets of each size.
Bracket Sensitivity
Single elimination commits to an outcome after only matchups, so the bracket assignment could in principle matter a great deal. We sample five independent random seedings per problem and recompute the result in the same way; Table 6 reports the mean across seedings with the observed range.
| Dataset | Gemini-3.1-Pro | GPT-5.2 |
| AIME | 98.13 [97.33, 99.33] | 82.68 [80.67, 84.00] |
| HMMT | 55.26 [53.76, 58.06] | 38.06 [36.56, 40.86] |
| HLE | 49.80 [49.00, 50.33] | 15.33 [14.33, 16.00] |
| Chess Val. | 98.27 [98.00, 99.00] | 70.60 [68.67, 71.33] |
| Chess Opt. | 0.171 [0.133, 0.206] | 2.048 [1.982, 2.104] |
Table 6. Sensitivity to bracket seeding. Mean APT performance at across five independent random bracket seedings per problem, with the range across seedings in brackets. Accuracy (%) except Chess Optimality (pawns).
Variation across seedings is small relative to the differences between methods in Table 5, because most problems admit several correct candidates and a poor seeding for one of them does not remove the rest from the bracket.
Quality of the Pairwise Signal
Three properties of the verifier preferences underpin the method: that a matchup between a correct and an incorrect solution is usually won by the correct one, that preferences are close to transitive, and that they are not dominated by presentation order.
Per-matchup accuracy. Table 7 reports how often the verifier majority prefers the correct solution when a correct and an incorrect candidate meet. Chess Optimality is omitted because it has no binary notion of correctness.
| Dataset | Gemini-3.1-Pro | GPT-5.2 |
| AIME | 89.5 | 85.6 |
| HMMT | 85.2 | 82.9 |
| HLE | 83.7 | 78.4 |
| Chess Val. | 92.3 | 87.9 |
Table 7. Per-matchup verifier accuracy (%). Rate at which the majority vote across aspect verifiers prefers the correct solution in a correct-versus-incorrect matchup, by dataset and generator (verifiers are Gemini-2.5-Flash and GPT-5-mini respectively).
Transitivity. Tournament aggregation implicitly assumes preferences are not badly cyclic. We enumerate all triples of candidate solutions and measure the fraction for which the three pairwise preferences are transitive, separately for each aspect verifier. Table 8 shows that transitivity holds in 89-98% of triples across all aspects and both verifier models. This rate is high, though not perfect, which is consistent with the Copeland ceiling being close to but slightly above single elimination (Appendix § 6.10).
| Aspect | Strategy | Gemini-2.5-Flash | GPT-5-mini |
| General Correctness | Summarize | 95.7 | 94.6 |
| General Correctness | Identify Mistakes | 97.3 | 95.7 |
| Domain Knowledge | Direct | 94.8 | 92.7 |
| General Correctness | Diff Comparison | 95.7 | 93.5 |
| Edge Cases | Direct | 93.2 | 89.2 |
| Unit Conversions | Step-by-Step | 96.4 | 94.3 |
| Logical Soundness | Step-by-Step | 97.5 | 95.4 |
| Factual Accuracy | Step-by-Step | 96.2 | 95.7 |
Table 8. Transitivity of verifier preferences (%). Fraction of candidate triples for which the three pairwise preferences produced by a single aspect verifier are transitive.
Position bias. Table 9 reports the raw rate at which the first-presented candidate (A) is chosen over the second (B), before the order-swapping correction described in § 3.1. Most settings fall below 0.5, meaning the verifiers mildly favour the second candidate rather than the first; the exceptions are Qwen3-8B solutions on HMMT and HLE. Either direction is a bias, which is why we query both orderings and discard order-dependent judgments rather than relying on the bias being small.
| Dataset | Gemini-3.1-Pro | GPT-5.2 | Qwen3-8B |
| AIME | 0.4340 | 0.3215 | 0.4739 |
| HMMT | 0.4919 | 0.3482 | 0.6821 |
| HLE | 0.4802 | 0.3348 | 0.5368 |
| Chess Val. | 0.4926 | 0.3544 | 0.4614 |
| Chess Opt. | 0.4380 | 0.3581 | 0.4564 |
Table 9. First-position selection rate. Fraction of single-ordering comparisons in which the first-presented candidate is preferred, by dataset and by the generator whose solutions are being compared. A value of 0.5 indicates no positional preference.
Failure Analysis
Not every error APT makes is a selection error. We separate generator-limited failures, where no correct candidate was sampled and no verification method could have succeeded, from selection failures, where a correct candidate was present in the pool but was not chosen. Only the latter are attributable to APT.
Table 10 reports the breakdown at . Chess Optimality is omitted as it has no binary correctness criterion.
| Dataset | Generator | APT correct | No correct candidate | Selection failure |
| AIME | Gemini-3.1-Pro | 99.3 | 0.67 | 0.0 |
| AIME | GPT-5.2 | 85.3 | 13.3 | 1.3 |
| HMMT | Gemini-3.1-Pro | 59.1 | 34.4 | 6.5 |
| HMMT | GPT-5.2 | 41.9 | 57.0 | 1.1 |
| HLE | Gemini-3.1-Pro | 52.0 | 34.3 | 13.7 |
| HLE | GPT-5.2 | 15.3 | 72.0 | 12.67 |
| Chess Val. | Gemini-3.1-Pro | 99.7 | 0.3 | 0.0 |
| Chess Val. | GPT-5.2 | 72.3 | 19.0 | 8.7 |
Table 10. Failure breakdown (% of problems). Problems are partitioned into those APT answers correctly, those where the generator produced no correct candidate, and those where a correct candidate existed but APT selected a different one. Only the last column is attributable to the verification method.
Across datasets, the majority of errors are generator-limited: on HMMT with GPT-5.2, for instance, 57% of problems had no correct candidate in the pool at all, against 1.1% where APT had a correct candidate available and passed it over. Inspecting the selection failures qualitatively, two causes recur. The first is a weak margin in the preference vote, where the verifier majority is narrow and falls on the wrong side; these are the matchups where the aspect verifiers disagree with each other. The second is unfavourable bracket placement, where a correct candidate is drawn against another strong candidate in an early round and eliminated before reaching the final. The second cause is bounded by the bracket ablation in Appendix § 6.7, which shows the effect is small in aggregate.
Varying Tournament Structure
In this section, we present preliminary experiments comparing different tournament formats on the chess validity and chess optimality domains. These experiments were conducted using GPT-5.2 (with GPT-5-mini verifiers) and Gemini-3.1-Pro (with Gemini-2.5-Flash verifiers), with candidates and aspect verifiers.
We compare three pairwise tournament formats against the non-pairwise baselines (Self-Consistency@8 and BoN-MAV@8):
Single Elimination. Candidates are paired and the majority-vote winner of each matchup advances. Requires matchups.
Double Elimination. Each candidate must lose twice before being eliminated. Losers drop to a losers’ bracket and may still win the tournament. Requires between and matchups.
Swiss. Candidates are paired in each round based on their current win-loss record. After rounds, the candidate with the best record is selected. Requires matchups.
Tables 11 and 12 present the results on chess validity and chess optimality, respectively.
| Method | GPT-5.2 | Gemini-3.1-Pro |
| Pass@1 | 57.33 | 96.00 |
| Self-Consistency@8 | 52.67 | 99.00 |
| BoN-MAV@8 | 62.00 | 97.33 |
| Single Elimination | 70.00 | 98.67 |
| Double Elimination | 72.33 | 99.00 |
| Swiss | 70.33 | 98.00 |
Table 11. Tournament format comparison on Chess Validity. Accuracy (%) with candidates.
| Method | GPT-5.2 | Gemini-3.1-Pro |
| Pass@1 | 1.378 | 0.018 |
| Self-Consistency@8 | 1.866 | 0.031 |
| BoN-MAV@8 | 1.856 | 0.056 |
| Single Elimination | 1.982 | 0.166 |
| Double Elimination | 1.905 | 0.194 |
| Swiss | 1.940 | 0.173 |
Table 12. Tournament format comparison on Chess Optimality. pawn score (higher is better) with candidates.
On chess validity, all three tournament formats produce a large improvement over the baselines for GPT-5.2. We see performance jumping from 62% (BoN-MAV) to 70-72.33%. With Gemini-3.1-Pro, as mentioned before, gains are limited due to the saturation of the benchmark. Double elimination achieves the highest accuracy on both models, though the margin over single elimination is small (0.33-2.33%). Swiss is slightly better with GPT-5.2 but slightly worse with Gemini-3.1-Pro.
On chess optimality, the pattern is similar. We see that all tournament formats substantially outperform BoN-MAV, particularly for Gemini-3.1-Pro where the jump is from 0.056 to 0.166-0.194 pawns. Double elimination again achieves the highest score for Gemini-3.1-Pro (0.194), while for GPT-5.2 single elimination slightly leads (1.982 vs. 1.905 for double elimination). The differences between tournament formats are small relative to the gap between any tournament format and the non-pairwise baselines.
We also conducted a diagnostic experiment on the GPT-5.2 chess validity results, running all pairwise comparisons for each of the 300 problems (with 5 verifiers per matchup). This full round-robin allows us to compute the Copeland winner (the candidate with the most pairwise matchup wins) which serves as a more comparison-intensive reference method. The Copeland method achieves 75.67%, compared to 70% for single elimination and a pass@8 oracle of 80%. This confirms that single elimination captures most of the signal available in the full pairwise preference matrix. Furthermore, when the majority vote across verifiers compares a correct solution against an incorrect one, it selects the correct solution 87.9% of the time, explaining the strong performance of all tournament formats: with pairwise signal this reliable, the correct solution is unlikely to be eliminated regardless of bracket structure.
These results support our choice of single elimination as the default tournament format in APT. While double elimination offers a small additional margin in some settings, single elimination achieves comparable performance with fewer matchups, making it the more compute-efficient choice.
References
- [1]Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073, 2022.arxiv.org/abs/2212.08073
- [2]Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952.DOI
- [3]Mark Braverman and Elchanan Mossel. Noisy sorting without resampling. In Proceedings of the Nineteenth Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 268–276, 2008.arxiv.org/abs/0707.1051
- [4]Bradley Brown, Jordan Juraysky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024.arxiv.org/abs/2407.21787
- [5]Mert Cemri, Melissa Z. Pan, Shui Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Why do multi-agent LLM systems fail? In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2503.13657
- [6]Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou. Are more LLM calls all you need? towards the scaling properties of compound AI systems. In Advances in Neural Information Processing Systems, 2024.
- [7]Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. Universal self-consistency for large language model generation. arXiv preprint arXiv:2311.17311, 2023.arxiv.org/abs/2311.17311
- [8]Yanxi Chen, Xuchen Pan, Yaliang Li, Bolin Ding, and Jingren Zhou. Provable scaling laws for the test-time compute of large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2411.19477.
- [9]Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, et al. Chatbot arena: An open platform for evaluating LLMs by human preference. In Proceedings of the 41st International Conference on Machine Learning, 2024.
- [10]Hyeong Kyu Choi, Xiaojin Zhu, and Yixuan Li. Debate or vote: Which yields better decisions in multi-agent large language models? In Advances in Neural Information Processing Systems, 2025.
- [11]Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017.
- [12]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
- [14]Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on Machine Learning, 2024.
- [15]Uriel Feige, Prabhakar Raghavan, David Peleg, and Eli Upfal. Computing with noisy information. SIAM Journal on Computing, 23(5):1001–1018, 1994.DOI
- [16]Google DeepMind. Gemini 2.5 flash. https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash, 2025.
- [17]Google DeepMind. Gemini 3.1 pro: A smarter model for your most complex tasks. https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/, 2026.
- [18]Bernard Grofman, Guillermo Owen, and Scott L. Feld. Thirteen theorems in search of the truth. Theory and Decision, 15(3):261–278, 1983.
- [19]Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Hongqiao Liu, Yuanhuo Wang, and Jian Guo. A survey on LLM-as-a-judge. arXiv preprint arXiv:2411.15594, 2024.arxiv.org/abs/2411.15594
- [20]Jiaxin Guo, Zewen Chi, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, and Furu Wei. Reward reasoning models. In Advances in Neural Information Processing Systems, 2025.DOI
- [21]Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.01798
- [22]Geoffrey Irving, Paul Christiano, and Dario Amodei. AI safety via debate. arXiv preprint arXiv:1805.00899, 2018.arxiv.org/abs/1805.00899
- [23]Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-Blender: Ensembling large language models with pairwise ranking and generative fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023.arxiv.org/abs/2306.02561
- [24]Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez. Debating with more persuasive LLMs leads to more truthful answers. In Proceedings of the 41st International Conference on Machine Learning, 2024.arxiv.org/abs/2402.06782
- [25]Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. Prometheus: Inducing fine-grained evaluation capability in language models. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.08491
- [26]Jean-François Laslier. Tournament Solutions and Majority Voting. Springer, 1997.DOI
- [27]Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. More agents is all you need. Transactions on Machine Learning Research, 2024.arxiv.org/abs/2402.05120
- [28]Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.arxiv.org/abs/2305.19118
- [29]Shalev Lifshitz, Sheila A. McIlraith, and Yilun Du. Multi-agent verification: Scaling test-time compute with multiple verifiers. arXiv preprint arXiv:2502.20379, 2025.arxiv.org/abs/2502.20379
- [30]Hunter Lightman, Vineet Kosaraju, Yura Burda, et al. Let’s verify step by step. In International Conference on Learning Representations, 2024.arxiv.org/abs/2305.20050
- [31]Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, Yang Liu, and Yahui Zhou. Skywork-reward-v2: Scaling preference data curation via human-ai synergy. arXiv preprint arXiv:2507.01352, 2025a.arxiv.org/abs/2507.01352
- [32]Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-Eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023.arxiv.org/abs/2303.16634
- [33]Yantao Liu et al. PairJudge RM: Perform best-of-N sampling with knockout tournament. arXiv preprint arXiv:2501.13007, 2025b.arxiv.org/abs/2501.13007
- [34]Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulic, Anna Korhonen, and Nigel Collier. Aligning with human judgement: The role of pairwise preference in large language model evaluators. In Conference on Language Modeling (COLM), 2024.arxiv.org/abs/2403.16950
- [35]Adian Liusie, Potsawee Manakul, and Mark Gales. LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, 2024.
- [36]Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2303.17651
- [37]Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannah Hajishirzi, and Nathan Lambert. RewardBench 2: Advancing reward model evaluation. In *International Conference on Learning Representations*, 2026.arxiv.org/abs/2506.01937
- [38]Marquis de Condorcet. *Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix*. Imprimerie Royale, Paris, 1785.
- [39]Hervé Moulin. Choosing from a tournament. *Social Choice and Welfare*, 3(4):271–291, 1986.DOI
- [40]OpenAI. GPT-5 system card. *arXiv preprint arXiv:2601.03267*, 2025a.
- [41]OpenAI. Update to GPT-5 system card: GPT-5.2. https://openai.com/index/gpt-5-system-card-update-gpt-5-2/, 2025b.
- [42]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. *Advances in Neural Information Processing Systems*, 35, 2022.arxiv.org/abs/2203.02155
- [43]Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations. In *Advances in Neural Information Processing Systems*, 2024.arxiv.org/abs/2404.13076
- [44]Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity’s last exam. *arXiv preprint arXiv:2501.14249*, 2025.
- [45]Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. Large language models are effective text rankers with pairwise ranking prompting. In *Findings of the Association for Computational Linguistics: NAACL 2024*, 2024.arxiv.org/abs/2306.17563
- [46]Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, and Soroush Vosoughi. Judging the judges: A systematic study of position bias in LLM-as-a-judge. In *Proceedings of IJCNLP-AACL 2025*, 2025.
- [47]Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. In *International Conference on Learning Representations*, 2025.arxiv.org/abs/2408.03314
- [48]Nisan Stiennon, Long Ouyang, Jeff Wu, et al. Learning to summarize with human feedback. *Advances in Neural Information Processing Systems*, 33, 2020.arxiv.org/abs/2009.01325
- [50]Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. JudgeBench: A benchmark for evaluating LLM-based judges. In *International Conference on Learning Representations*, 2025.arxiv.org/abs/2410.12784
- [51]Pat Verga, Sebastian Hofstätter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating LLM generations with a panel of diverse models. *arXiv preprint arXiv:2404.18796*, 2024.arxiv.org/abs/2404.18796
- [52]Thuc Vu, Alon Altman, and Yoav Shoham. On the complexity of schedule control problems for knockout tournaments. In *International Conference on Autonomous Agents and Multiagent Systems*, 2009.DOI
- [53]Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities. In *International Conference on Learning Representations*, 2025.arxiv.org/abs/2406.04692
- [54]Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyi Liu, and Zhifang Sui. Large language models are not fair evaluators. In *Proceedings of the 62nd Annual Meeting of the ACL (Volume 1: Long Papers)*, 2024.arxiv.org/abs/2305.17926
- [55]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, 2023.arxiv.org/abs/2203.11171
- [56]Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui. From decoding to meta-generation: Inference-time algorithms for large language models. Transactions on Machine Learning Research, 2024.arxiv.org/abs/2406.16838
- [57]Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification. In Findings of the Association for Computational Linguistics: EMNLP 2023, 2023.arxiv.org/abs/2212.09561
- [58]Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for LLM problem-solving. In International Conference on Learning Representations, 2025.arxiv.org/abs/2408.00724
- [59]An Yang, Binyuan Hui, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
- [60]Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2305.10601
- [61]Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. FLASK: Fine-grained language model evaluation based on alignment skill sets. In International Conference on Learning Representations, 2024.arxiv.org/abs/2307.10928
- [62]Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. In International Conference on Learning Representations, 2025a.arxiv.org/abs/2408.15240
- [63]Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, et al. What, how, where, and how well? a survey on test-time scaling in large language models. arXiv preprint arXiv:2503.24235, 2025b.arxiv.org/abs/2503.24235
- [64]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In NeurIPS Datasets and Benchmarks Track, 2023.arxiv.org/abs/2306.05685
- [65]Yilun Zhou, Austin Xu, Peifeng Wang, Caiming Xiong, and Shafiq Joty. Evaluating judges as evaluators: The JETTS benchmark of LLM-as-judges as test-time scaling evaluators. In Proceedings of the 42nd International Conference on Machine Learning, 2025.arxiv.org/abs/2504.15253






