WorksComputer Science
AGORA: Reliability-Aware Aggregation of LLM Judges on Small Evaluation Sets
Abstract
LLM-as-a-judge is a common substitute for costly human evaluation, and aggregating several judges is the standard remedy for any single judge's bias. Judge pools mix judges of very different reliability, while the human-labeled sets that validate them hold a few hundred items or fewer. In this regime the per-item mean weights weak judges as heavily as strong ones, and confusion-table models such as Dawid-Skene must estimate a per-judge error table from too few items. We propose AGORA (Aggregation Of Reliability-weighted Assessments), an item-response-theory aggregator that places every item on one latent quality scale and describes each judge by a sensitivity parameter and ordered rating thresholds, so judge reliability is learned from all items jointly and without human labels. Across three benchmarks, AGORA matches or exceeds Dawid-Skene on large evaluation sets, and its advantage widens as the set shrinks toward a few hundred items. It remains accurate as pools grow more uneven, a label-free dispersion statistic indicates in advance when trust weighting helps, and its learned trust weights track per-judge human validity. Code is available at https://github.com/JunhyungAhn/llm-as-judge-irt.
Introduction
LLM judges have become a standard tool for evaluating generative systems at scale (Zheng et al., 2023; Liu et al., 2023; Verga et al., 2024). Because individual judges show systematic biases, including length, position, and self-preference effects (Zheng et al., 2023; Wang et al., 2024b), a common remedy is to score each item with several judges and aggregate their scores. Aggregators range from simple averages to reliability-aware annotation models: Dawid–Skene [7] estimates a per-judge error table over the rating categories, MACE (Hovy et al., 2013) a competence-and-spamming profile, and GLAD (Whitehill et al., 2009) continuous judge-ability and item-difficulty parameters. All of them learn reliability from repeated judge–item overlap, so their estimates depend on having enough shared items.
This requirement is strained in LLM-judge meta-evaluation. Human reference sets are costly and small: MT-Bench scores 80 prompts (Zheng et al., 2023), Topical-Chat 360 dialogues (Mehri and Eskenazi, 2020), and SummEval 1,600 summaries (Fabbri et al., 2021). Because judge validity changes with the systems scored [3], the rating range (Peyrard, 2019), and the task or quality dimension [2] (Nimah et al. 2023), each new setting needs a fresh reference set of a few hundred items, where meta-evaluation correlations are already uncertain [4, 9].
Pool composition compounds the problem. Judge reliability varies sharply across models [2], and ensembles routinely combine models of unequal strength (Verga et al., 2024; Wang et al., 2024a), so pools are heterogeneous.1 The per-item mean gives every judge equal weight, so lower-reliability judges shift the aggregate as much as reliable ones. Confusion-table models fail differently: with a -point rating scale each judge carries error-table cells, so a few hundred items leave most cells sparse and determined by the prior. MACE and GLAD use fewer parameters but treat ratings as unordered classes and do not model how each judge uses an ordinal scale. An aggregator for this regime should learn which judges to trust without spending scarce observations on a per-judge error table.
We propose AGORA: Aggregation Of Reliability-weighted Assessments via ItemResponse Theory. From the judge-score matrix (Figure 1), AGORA fits a graded-response model, the item-response-theory model for ordinal ratings ((Samejima, 1969)), with one latent quality per item, one discrimination per judge, and ordered thresholds that capture each judge’s use of the rating scale. The discrimination acts as a reliability weight: judges whose scores track latent quality steeply receive more influence, and a regularized-horseshoe prior shrinks uninformative judges toward zero. Each judge costs parameters rather than cells, and every parameter is informed by all items, so trust estimation remains stable on small sets. Prior work uses item response theory to diagnose LLM judges ([6]); AGORA uses it to aggregate them. Our contributions are as follows:
Figure 1. AGORA on an illustrative pool of three judges rating five items on a 0–4 scale. (1) The input is the judge-score matrix alone. (2) From all items jointly, AGORA fits a graded-response model per judge, drawn as its expected rating against latent quality . The thresholds encode severity (B is lenient, A severe), and the discrimination encodes trust (bars: ); the horseshoe prior shrinks the flat judge C’s trust toward zero. (3) For each item, every rating places in that judge’s threshold interval, with influence set by the judge’s trust. The posterior concentrates where A and B agree, so the EAP estimate ignores C’s outlying 0 (circled). (4) is mapped back to the rating scale. Against a human reference used only for evaluation, the AGORA scores reproduce the human ranking (Spearman ), whereas the per-item mean, shifted by C’s outliers, reaches . Judge parameters and the human reference are illustrative; all other quantities are computed from them.
We propose AGORA, a graded-response aggregator that learns per-judge reliability weights under a sparsity prior and models judge-specific scale use with parameters per judge (§ 3).
We show that AGORA matches or exceeds the strongest baselines on large evaluation sets and that its advantage over Dawid–Skene grows as the set shrinks, exceeding it on 98–100% of 60 random five-judge pools with 50 items (§ 5.1–§ 5.3).
We introduce agreement dispersion, a label-free statistic of the score matrix that tracks AGORA’s gain over the per-item mean (Spearman +0.73 and +0.61) and so indicates, before any human labels are collected, when trust weighting helps (§ 5.3).
We show that the learned discriminations track per-judge human validity, giving a label-free judge diagnostic (§ 7).
Related Work
Aggregating LLM judges. Most LLM-judge ensembles average or majority-vote their members ((Verga et al., 2024; Wang et al., 2024a)), and some weight judges by a learned or heuristic reliability score ((Guha et al., 2024)). Qian et al. (2026) fit a Bradley–Terry model with a per-judge discrimination over pairwise comparisons, SkillAggregation ((Sun et al., 2025)) trains a neural context encoder over binary judge probabilities, CARE decomposes the score matrix into confounder-aware sparse and low-rank parts ((Zhao et al., 2026)), and Balasubramanian et al. [1] model pairwise judge dependence with Ising models. AGORA instead takes the latent-trait route on absolute ordinal scores: one continuous quality per item and a trust-weighted read-out over per-judge discriminations.
Annotation Models. Dawid–Skene, MACE, and related Bayesian annotation models ([7, 8] (Hovy et al. 2013; Paun et al. 2018)) were developed for large crowdsourcing corpora, and GLAD ((Whitehill et al., 2009)) replaces the confusion table with a latent ability–difficulty model but still relies on item redundancy. These models tie on balanced pools of comparably reliable annotators ((Paun et al., 2018)) and need much redundancy to estimate reliability stably ((Zheng et al., 2017)). We make the complementary point on the item axis: when items are few, the confusion-table parameterization fragments the data and loses its margin over a flat mean (§ 5.2), whereas a latent-trait parameterization retains it.
Item Response Theory. Item response theory (IRT) jointly scores items and raters, and its graded-response model (GRM) handles ordinal ratings ((Lord, 1980; Samejima, 1969; Rasch, 1960)). Closest to our work, Choi et al. [6] apply IRT to diagnose LLM-judge reliability. AGORA instead aggregates judges with a discrimination-weighted read-out under a regularized-horseshoe prior ([5] (Piironen and Vehtari 2017)), estimated via the augmentation of Makalic and Schmidt (2016), and adds a score-only diagnostic of when reliability weighting helps (§ 5.3).
The AGORA Aggregator
AGORA treats LLM-judge aggregation as latent-trait estimation. Each evaluated item has a latent quality , and each judge has a discrimination , measuring how sharply it scores a change with latent quality, and ordered thresholds over rating categories , capturing scale use such as leniency or range compression. A judge with large is informative about , a judge with near-zero has little influence, and the thresholds let identical raw scores map to different latent qualities across judges. The model has four parts: the graded-response likelihood (§ 3.1), the discrimination prior (§ 3.2), an expectation-maximization (EM) procedure for maximum-a-posteriori estimation (§ 3.3), and a posterior read-out that maps each item’s latent quality to one score (§ 3.4). All quantities are fit per dimension from the judge-score matrix (Algorithm 1).
Latent-trait model
The latent-trait model ties judges through a shared latent quality for each item and models each judge’s ordinal response as a function of that quality. Let item be scored by judge , producing an ordinal label on a -point scale.2 AGORA assigns each item a latent quality and models judge with Samejima’s graded-response model (Samejima, 1969):
for , where is the logistic function and the thresholds satisfy .
The category probabilities are successive differences of the cumulative curves:
with and . These probabilities define the likelihood used for estimation and the posterior read-out in § 3.4.
We use the standard IRT normalization to fix location and scale, and constrain to avoid per-judge sign flips. With every , a higher latent quality raises every judge’s expected rating, so the latent scale has a fixed orientation. This compact parameterization contrasts with a per-judge confusion table: each judge has GRM parameters (one discrimination and thresholds) rather than cell probabilities. We return to this parameterization and its inference in § 5.2 and App. § 13.
The discrimination prior
The discriminations determine how strongly each judge can influence the read-out, so the prior on them controls how aggressively weak or uninformative judges are shrunk. We use a regularized horseshoe prior [5] (Piironen and Vehtari 2017), with global and local scales estimated from the score matrix rather than fixed judge weights:
where enforces and denotes a standard half-Cauchy distribution on the positive reals. The global scale controls the overall shrinkage of discriminations, the local scales let indiindividual judges retain large values when supported by the likelihood, and the slab scale caps the effective prior variance to prevent extremely large discriminations. Together, these components induce sparsity over judge discriminations, shrinking uninformative judges toward low trust while allowing informative judges to remain influential. This shape suits LLM-judge pools, which mix informative and near-uninformative judges: a log-normal prior, common in Bayesian IRT, places little mass near zero and pulls discriminations toward one typical value, whereas the horseshoe shrinks only the uninformative ones [5] (Piironen and Vehtari 2017). We compare this prior against an unpenalized fit in § 6.
Estimation
Given the likelihood and prior, we estimate item qualities and judge parameters jointly with an EM procedure for maximum-a-posteriori estimation. In the E-step, we compute the posterior distribution on a fixed grid for each item. In the M-step, each judge’s discrimination and thresholds are updated by minimizing the posterior-expected penalized negative log-likelihood
over and the ordered thresholds . The penalty is the Gaussian term induced by the horseshoe prior in Eq. (3). We update the horseshoe scales at their conditional modes using the inverse-gamma augmentation of Makalic and Schmidt (2016); details are in App. § 10.
Aggregate read-out
The read-out converts the fitted model and an item’s judge scores into a single latent-quality estimate. We use the posterior mean, or expected a posteriori (EAP) estimate, computed on the grid :
where is the standard-normal prior discretized on and the probabilities use the fitted judge parameters from § 3.3. We use the posterior mean rather than the posterior mode because it is more stable when item posteriors are diffuse. The role of the discriminations is visible in a standard Fisher-information linearization of the
| Algorithm 1 AGORA |
| Input: score matrix |
| Output: per-item scores on the internal ordinal scale |
| 1: warm-start from an unpenalized GRM |
| 2: for to do |
| 3: E-step: grid posterior per item |
| 4: M-step: per judge , minimize Eq. (4) over , ordered |
| 5: Scale update: refresh horseshoe scales (App. § 10) |
| 6: end for |
| 7: EAP estimate, Eq. (5) |
| 8: expected-score map, Eq. (7) |
| 9: return per-item scores (with trust , thresholds ) |
Table 1.
EAP. Let denote judge ’s observed category re-expressed on the latent scale by inverting its fitted thresholds. Then, locally,
The weight is the graded-response Fisher information that judge contributes at item (App. § 10). Its global factor makes the read-out judge-weighted, and its item-level factor , which measures how informative judge ’s response curve is near , makes it item-sensitive: the same judge receives different effective weights on different items. All AGORA results are computed from the exact EAP (Eq. (5)); the -weighted average (Eq. (6)) is its first-order interpretation, not a separate estimator.
Mapping to the rating scale. The EAP estimate is a latent coordinate, not a benchmark score. We therefore convert it to the benchmark’s rating scale using the fitted model’s expected score, i.e., the multi-judge test characteristic curve (Lord, 1980; Samejima, 1969):
where the category probabilities come from the fitted GRM in Eq. (2). This mapped score is the reported aggregate and the value compared with the human reference. Because the map is monotone in , preserves the item ranking induced by the latent quality; we assess it by rank correlation with the human reference, the standard meta-evaluation target.
| Method | SummEval full | SummEval | Topical-Chat full | Topical-Chat | WMT22 full | WMT22 |
| Mean | 0.521 | 0.526 | 0.672 | 0.668 | 0.416 | 0.402 |
| Median | 0.523 | 0.529 | 0.643 | 0.643 | 0.406 | 0.402 |
| Trimmed Mean | 0.538 | 0.545 | 0.674 | 0.672 | 0.418 | 0.400 |
| Dawid–Skene | 0.555 | 0.529 | 0.695 | 0.678 | 0.423 | 0.399 |
| CARE | 0.544 | 0.534 | 0.688 | 0.662 | 0.418 | 0.405 |
| GLAD | 0.545 | 0.543 | 0.685 | 0.680 | 0.419 | 0.405 |
| MACE | 0.533 | 0.546 | 0.690 | 0.682 | 0.418 | 0.404 |
| AGORA | 0.556 | 0.550 | 0.705 | 0.696 | 0.426 | 0.409 |
Table 1 (PDF p. 5). Main results on the primary five-judge pool. Dimension-averaged Spearman against human references. full: the full evaluation set (SummEval , Topical-Chat , WMT22 ). : mean over 50 random item subsets of 50 items, shared by all methods (paired tests in § 5.1). Bold: best point estimate in each column.
Experimental Setup
We evaluate on three meta-evaluation benchmarks with human ratings. SummEval (Fabbri et al., 2021) contains 1,600 news summaries rated by three experts on four dimensions (coherence, consistency, fluency, relevance) on a 1–5 scale. Topical-Chat (Mehri and Eskenazi, 2020) contains 360 dialogue responses rated by three annotators on six dimensions with 2-, 3-, and 5-point scales. WMT22 en–de (Freitag et al., 2022) contains 19,725 translated segments with professional MQM quality scores. SummEval and Topical-Chat are the small human-referenced sets that motivate our setting, while WMT22 tests AGORA on a large evaluation set.
Each judge is a single model evaluated with a fixed prompt, drawn from a 14-model candidate pool spanning proprietary frontier and open-weight model families (App. § 11.§ 1). Our primary pool has five judges and is heterogeneous by construction: one proprietary frontier judge (claude-sonnet-4-6) and four open-weight judges from distinct model families (Llama-4-Scout, DeepSeek-R1, GLM-4.7-Flash, Nemotron-120B). The pool was fixed by model name before fitting any aggregation method, without consulting benchmark-specific human references, and is held constant across datasets and methods. We test generality beyond this pool on random five-judge pools drawn from the candidate pool: 30 pools on all three benchmarks across evaluation-set sizes, and 60 further pools on SummEval and Topical-Chat (§ 5.3).
We report Spearman against the human reference, averaged across dimensions on multidimensional benchmarks. For each evaluation-set size below the full sample, every method is re-fit on the same 50 random item subsets, and we report the mean over subsets. Small-sample contrasts use a paired bootstrap over per-subset differences, and full-sample contrasts use an item-level paired bootstrap (). We compare against the per-item mean, median, trimmed mean (discarding the highest- and lowest-scoring judge per item), Dawid–Skene, MACE, GLAD, and CARE, a confounder-aware low-rank aggregator; implementation details are in App. § 11.§ 3.
Results
Main Results
AGORA has the highest point estimate in every column of Table 1, on the full sample and at , the smallest size of the subsampling ladder in § 5.2. At full it matches Dawid–Skene on SummEval (0.556 vs. 0.555), exceeds it on Topical-Chat (0.705 vs. 0.695) and WMT22 (0.426 vs. 0.423), and improves on the per-item mean by +0.035, +0.033, and +0.010. It also exceeds MACE, GLAD, and CARE at both sizes on every benchmark (full- margins +0.007 to +0.023); the item-level bootstrap confirms these separations at full , and the paired subset bootstrap confirms the gains on SummEval and Topical-Chat.
Performance as Evaluation Sets Shrink
Figure 2 follows the small-sample regime from down to , where AGORA is the top aggregator at every size on all three benchmarks. Dawid–Skene’s per-judge confusion tables receive less evidence as shrinks: on SummEval it moves from parity with AGORA at full (Table 1) to a 0.021 deficit at , and on WMT22 it falls below the per-item mean. Across the shared subsets, AGORA exceeds the per-item mean on at least 86% of subsets at every size ( against chance) and Dawid–Skene on 78–92%.
Figure 2. Data efficiency on the primary five-judge pool in the small-sample regime (). Dimension-averaged gap to the per-item mean, , as shrinks (larger on the left). AGORA is the top aggregator at every size; Dawid–Skene loses margin as shrinks.
AGORA Gains Grow with Pool Heterogeneity
Trust weighting should help most when a pool’s judges agree unevenly. We measure this with agreement dispersion, , where is judge ’s mean rank correlation with the other judges in the pool; larger means more uneven agreement. Because uses only the score matrix, it is available before any human labels are collected.
Random pools. On 30 random five-judge pools from the candidate pool (Table 2), AGORA exceeds Dawid–Skene at in 93%, 100%, and 83% of pools on SummEval, Topical-Chat, and WMT22 (mean margins +0.017, +0.015, +0.019), and stays at or above the per-item mean at every size. On a second draw of 60 pools at , it exceeds Dawid–Skene in 98% and 100% of pools on SummEval and Topical-Chat.
| Dataset | Mean | Dawid–Skene | AGORA | |
| SummEval | 50 | 0.510 | 0.501 | 0.518 |
| SummEval | 100 | 0.545 | 0.546 | 0.555 |
| SummEval | 200 | 0.542 | 0.545 | 0.547 |
| Topical-Chat | 50 | 0.715 | 0.703 | 0.718 |
| Topical-Chat | 100 | 0.697 | 0.699 | 0.703 |
| Topical-Chat | 200 | 0.703 | 0.708 | 0.709 |
| WMT22 | 50 | 0.442 | 0.423 | 0.442 |
| WMT22 | 100 | 0.402 | 0.393 | 0.403 |
| WMT22 | 200 | 0.428 | 0.427 | 0.429 |
Table 2. Thirty random five-judge pools: dimension-averaged Spearman , averaged over pools and over 15, 12, and 10 item subsets at , 100, and 200. AGORA has the highest average at every size.
Gain and dispersion. On these 60 pools, AGORA’s gain over the per-item mean grows with (Spearman +0.73 on SummEval and +0.61 on Topical-Chat; ), as does its gain over Dawid–Skene on SummEval (+0.53, ). On pools at least as dispersed as the primary pool ( on SummEval, on Topical-Chat; 6 and 15 pools), it exceeds the per-item mean in 100% and 93% of cases. On balanced pools it recovers the per-item mean as a limiting case at no accuracy cost: on the full 14-judge SummEval pool () it obtains 0.577 against the mean’s 0.572 (App. § 11), as expected from the convergence of annotation models on balanced data ((Paun et al., 2018)).
Controlled dispersion sweep. Adding 0–8 uninformative judges with uniformly random ratings to random five- and seven-judge pools extends from its natural range (about 0.3) to about 1.1 (Figure 3). Across this range, AGORA stays within 0.543–0.562 on SummEval and 0.689–0.712 on Topical-Chat, while the per-item mean falls from 0.536 to 0.436 and from 0.703 to 0.612, and Dawid–Skene from 0.533 to 0.469 and from 0.697 to 0.671.
Figure 3. Accuracy at as agreement dispersion increases, over random five- and seven-judge pools with 0–8 uninformative judges (uniformly random ratings) added; points are means within quintiles. AGORA remains stable while the per-item mean and Dawid–Skene decline. Dashed line: of the primary pool.
Ablations and Label-Free Baselines
Fitted trust weights drive the gain. Holding the fitted model fixed, we vary only the read-out’s trust vector (Table 3): the raw per-item mean (R0), the EAP under one pooled discrimination shared by all judges (R1), and the exact EAP with fitted per-judge discriminations (R2). At full the fitted weights supply the gain (R1 to R2: +0.023 on SummEval, +0.033 on Topical-Chat); at the latent parameterization itself adds +0.014 and +0.016, consistent with § 5.2.
| Read-out | SummEval full | SummEval | Topical-Chat full | Topical-Chat |
| R0: per-item mean | 0.521 | 0.526 | 0.672 | 0.668 |
| R1: latent, equal trust | 0.533 | 0.540 | 0.672 | 0.684 |
| R2: latent, trust | 0.556 | 0.550 | 0.705 | 0.696 |
Table 3. Read-out decomposition: the fit is held fixed and only the trust vector changes. At full the fitted-discrimination read-out (R2−R1) supplies most of the gain over the mean; at both the latent read-out and the fitted trust contribute.
The prior adds robustness at no clean-data cost. Replacing the regularized horseshoe with an unpenalized (flat MLE) fit (Table 4), the two match on clean pools (0.554 vs. 0.551 on SummEval, 0.705 vs. 0.697 on Topical-Chat). With injected adversarial judges, each mirroring an existing judge’s scores around the rating midpoint, the horseshoe improves on the flat fit by +0.008 and +0.017. Neither the prior nor the fixed estimation settings are tuned on human labels (App. § 10).
| prior on | clean (full ) SE | clean (full ) TC | adversarial () SE | adversarial () TC |
| flat (MLE) | 0.551 | 0.697 | 0.527 | 0.663 |
| horseshoe | 0.554 | 0.705 | 0.535 | 0.680 |
Table 4. Discrimination-prior ablation (: number of injected adversarial judges). The regularized horseshoe matches the unpenalized (flat MLE) fit on clean pools and improves on it under adversarial corruption.
Learned weights outperform label-free heuristics. AGORA exceeds weighted means based on mean inter-judge correlation, inverse variance, and a spectral meta-learner (Table 5; 0.556 vs. the best heuristic’s 0.550 on SummEval, 0.705 vs. 0.686 on Topical-Chat).
| Mean | wt. | inv-var | spectral | AGORA | |
| SummEval (full) | 0.521 | 0.550 | 0.541 | 0.548 | 0.556 |
| Topical-Chat (full) | 0.672 | 0.686 | 0.674 | 0.683 | 0.705 |
Table 5. AGORA vs. label-free continuous reliability weightings (Spearman , primary pool, full ): mean-correlation (), inverse-variance, and a spectral meta-learner. AGORA is best on both datasets.
Pools with the strongest judge. AGORA also exceeds the label-free single-best judge and a drop-worst average (App. § 14.§ 1). When the strongest candidate judge, gemma-4–31B (App. § 11.§ 2), is added to the primary pool and every method is scored on held-out items, AGORA exceeds the per-item mean and Dawid–Skene in all eight benchmark-by-size settings (Table 6). On a pool of the five most valid judges, it exceeds the best single judge at every size by +0.015 to +0.046, and with gemma-4–31B in every random pool it still exceeds Dawid–Skene at in 97% and 100% of 30 pools on SummEval and Topical-Chat.
| Dataset | Mean | DS | AGORA | gemma | AGORA | |
| SummEval | 50 | 0.541 | 0.549 | 0.569 | 0.572 | 0.588 |
| SummEval | 200 | 0.543 | 0.562 | 0.573 | 0.548 | 0.582 |
| SummEval | full | 0.552 | 0.574 | 0.578 | 0.563 | 0.586 |
| Topical-Chat | 50 | 0.697 | 0.701 | 0.717 | 0.708 | 0.750 |
| Topical-Chat | 200 | 0.705 | 0.722 | 0.732 | 0.704 | 0.745 |
| Topical-Chat | full | 0.696 | 0.711 | 0.720 | 0.695 | 0.735 |
| WMT22 | 50 | 0.394 | 0.399 | 0.401 | 0.383 | 0.428 |
| WMT22 | 200 | 0.427 | 0.426 | 0.437 | 0.431 | 0.473 |
Table 6. Pools containing the strongest judge, gemma-4-31B, scored on held-out items (75%; 10 and 5 item subsets at and 200). Left: the primary pool plus gemma. Right: the five judges with the highest human validity on a separate calibration quarter, against its best member, gemma alone.
Interpreting the Trust Weights
AGORA’s fitted trust weights also serve as a label-free diagnostic of judge quality. Compared with each judge’s post-hoc Spearman correlation with human labels on the full 14-judge pool (Figure 4), the dimension-averaged correlation is 0.65 on SummEval, 0.82 on Topical-Chat, and 0.66 on WMT22. Because is learned without human labels, it gives a practical screen for judge quality before any annotation.
Figure 4. Trust weights track judge validity on the full 14-judge pool. Each point is one judge on one dimension: post-hoc validity (Spearman with human labels) against the fitted trust weight ; is their dimension-averaged Spearman correlation.
Conclusion
AGORA aggregates LLM judges of uneven reliability with a graded-response latent-trait model whose discrimination-weighted read-out replaces per-judge confusion tables. Its compact parameterization keeps reliability estimation data-efficient where confusion tables fragment, so its margin over Dawid–Skene widens as evaluation sets shrink.
decisions.
The learned discriminations also serve as a label-free proxy for judge validity, and the dispersion statistic signals when trust weighting improves on the simple mean.
Limitations
Our experiments cover three meta-evaluation benchmarks spanning summarization, dialogue, and machine translation; further task families and deployment settings are future work. AGORA targets absolute ordinal ratings, the format of these benchmarks; pairwise preference data would call for a Bradley–Terry-style likelihood, and joint modeling across dimensions and dependence-aware likelihoods [1] (Zhao et al. 2026) are natural extensions. AGORA is intended for meta-evaluation of LLM judges, not as a replacement for human judgment in high-stakes decisions.
Estimation Details
We compute the maximum-a-posteriori estimate by EM, treating as latent and as parameters (Algorithm 1). Direct half-Cauchy MAP updates are awkward, so we use the inverse-gamma augmentation of Makalic and Schmidt (2016), which writes each half-Cauchy as a scale mixture with auxiliary variables and yields closed-form conditional modes for the local and global scales. The E-step forms the posterior over on a fixed grid by combining the prior with the likelihood (2) across judges and takes the posterior mean . The M-step minimizes Eq. (4) over and the ordered thresholds (parameterized through a cumulative-softplus map), the penalty being exactly the Gaussian induced by the horseshoe (3). The scale update then refreshes the scales at their conditional modes:
followed by the slab cap . We warm-start from a 6-iteration unpenalized graded-response fit and run iterations, fitting one model per dimension; the auxiliaries are initialized to . The few fixed quantities are held constant across all datasets: slab scale , the discrimination range , a posterior grid of 150 points on , and an L-BFGS-B M-step. As a numerical safeguard, the implementation also checks that correlates positively with the per-item mean and flips its sign otherwise.
Information factorization. The read-out weight in Eq. (6) is the graded-response Fisher information
where is the category probability from Eq. (2), its derivative in , and the item-level information factor.
Sensitivity. These settings are not tuned. Varying each over a wide range, and perturbing the warm-start initialization, leaves the full- accuracy essentially unchanged (Table 7).
| Perturbation | SE | TC | WMT |
| slab | 0.0000 | 0.0001 | 0.0000 |
| grid | 0.0000 | 0.0000 | 0.0000 |
| EM iters | 0.0002 | 0.0000 | 0.0000 |
| warm-start (5 seeds) | 0.0000 | 0.0000 | 0.0000 |
Table 7. Estimation sensitivity: the maximum change in full- Spearman when each fixed setting is varied from its default (, grid 150, ) and when the warm-start is randomly perturbed. Base is 0.556 (SummEval), 0.705 (Topical-Chat), and 0.426 (WMT22); every shift is at most 0.0002.
Judge Pool and Prompts
Candidate Pool and Prompt Templates
We use a diverse set of instruction-tuned LLMs as judges. The 14-model pool comprises three proprietary models (gpt-5, gemini-2.5-pro, claude-sonnet-4-6) and eleven open-weight models (DeepSeek-V3, DeepSeek-R1, Qwen3-Next-80B, Qwen2.5-72B, Llama-4-Scout, gpt-oss-120b, Mixtral-8x22B, Phi-4, gemma-4-31B, GLM-4.7-Flash, Nemotron-120B), spanning a wide range of size, architecture, and training data, a heterogeneous pool of judges. For each dataset and evaluation dimension, every judge scores a single item on the benchmark’s integer scale and returns the score as JSON, decoded at temperature 0. No human labels are used to tune or calibrate the judges; human annotations are used solely for evaluation. The prompts for the three datasets are shown below.
SummEval
You are an expert evaluator assessing {dimension}.
Source Article: {source}
Summary: {summary}
Rate on 1–5 (5=Excellent ... 1=Very Poor).
Output ONLY valid JSON: {"score": N}
{dimension}: coherence, consistency, fluency, relevance. The Source Article line is included only for consistency and relevance.
Topical-Chat
You are an expert evaluator of open-domain dialog responses.
Assess {dimension}.
Dialog history: {context}
Background fact: {fact}
Candidate response: {response}
Rate on integer scale {lo}–{hi} ({anchors}).
Output ONLY valid JSON: {"score": N}
{dimension}: the six USR dimensions (Understandable 0–1; Natural, Maintains Context, Engaging each 1–3; Uses Knowledge 0–1; Overall 1–5); {anchors} gives that scale’s endpoint labels.
WMT22 en–de
You are an expert evaluating machine translation quality.
Source (English): {src}
Translation (German): {mt}
Rate overall quality on 1–5 (5=Excellent …
1=Very Poor).
Output ONLY valid JSON: {"score": N}
Full-Pool Validity Diagnostics
Table 8 reports each judge’s Spearman with the human labels on the full 14-judge pool (dimension-averaged), from which our primary pool is drawn (its composition is given in § 4). Validity is dispersed (0.40 to 0.57 on SummEval). This full pool has lower agreement dispersion than the primary pool on SummEval (=0.21 vs. 0.29); here AGORA obtains 0.577 against the per-item mean’s 0.572, matching the convergence of annotation models on balanced pools ((Paun et al., 2018)).
| Judge | SummEval | Topical-Chat | WMT22 |
| gpt-5 | 0.530 | 0.629 | 0.427 |
| gemini-2.5-pro | 0.550 | 0.655 | 0.421 |
| claude-sonnet-4-6 | 0.531 | 0.700 | 0.417 |
| DeepSeek-V3 | 0.491 | 0.652 | 0.384 |
| Qwen3-Next-80B | 0.495 | 0.567 | 0.358 |
| Llama-4-Scout | 0.422 | 0.440 | 0.312 |
| DeepSeek-R1 | 0.445 | 0.492 | 0.358 |
| gpt-oss-120b | 0.480 | 0.584 | 0.368 |
| Mixtral-8x22B | 0.520 | 0.635 | 0.366 |
| Phi-4 | 0.464 | 0.550 | 0.331 |
| Qwen2.5-72B | 0.439 | 0.603 | 0.337 |
| gemma-4-31B | 0.566 | 0.704 | 0.432 |
| GLM-4.7-Flash | 0.412 | 0.503 | 0.322 |
| Nemotron-120B | 0.402 | 0.500 | 0.347 |
Table 8. Per-judge Spearman vs. human labels on the full pool (dimension-averaged).
Baseline Implementations
All baselines see the same judge-score matrix as AGORA and use no human labels. The per-item mean and median average or take the median of the raw scores across judges. The trimmed mean sorts each item’s judge scores and discards a fraction from each end before averaging; at this removes the single highest- and lowest-scoring judge and averages the remaining three. Dawid–Skene, MACE, and GLAD are the crowd-kit library implementations, run for 10 EM iterations (MACE with one restart; GLAD with five inner M-step iterations). Because these estimators are categorical, each integer rating level is treated as a separate class; we recover a scalar per item as the posterior expected rating . This categorical treatment discards the ordinal structure that AGORA’s graded-response link uses. CARE ((Zhao et al., 2026)) is run at its label-free library default (hyperparameter =1) with the SCS conic solver, taking the score-weighted quality vector as output. Each aggregator returns one scalar per item, standardized before the Spearman comparison.
Additional Evaluation Results
WMT22 under Field-Standard Metrics
We report Spearman throughout for comparability across datasets, but machine-translation evaluation conventionally uses segment-level Kendall and pairwise accuracy ((Freitag et al., 2022)). Table 9 recomputes the full- WMT22 results under all three: AGORA ranks first on every metric, and the per-item mean is last on pairwise accuracy, so the conclusion does not depend on the rank-correlation protocol.
| Method | Spearman | Kendall | pairwise acc. |
| Mean | 0.416 | 0.3418 | 0.619 |
| GLAD | 0.419 | 0.3362 | 0.644 |
| Dawid–Skene | 0.423 | 0.3396 | 0.645 |
| AGORA | 0.426 | 0.3419 | 0.647 |
Table 9. WMT22 (full ) under three agreement metrics. AGORA ranks first on all three; the per-item mean trails on pairwise accuracy.
Parameterization Diagnostic
Figure 5 contrasts the confusion-table and latent-trait parameterizations on a small evaluation set, complementing the small- analysis in § 5.2. A -category confusion matrix gives each judge parameters; for SummEval’s the five-judge pool carries 125, against AGORA’s 25. As shrinks the confusion cells become sparse: 25% have fewer than five observations at full but 86% at =50, so many cells become prior-dominated, making the high-dimensional estimate less data-efficient, which is why Dawid–Skene loses its full- accuracy on SummEval at small while AGORA retains its margin.

Figure 5. Two parameterizations on the same small evaluation set ( items, judges, ). Left: confusion-matrix aggregators give each judge a error table; with few items most cells are prior-dominated, so the estimate becomes less data-efficient. Right: AGORA maps every item onto one latent quality and summarizes each judge with a discrimination and ordered thresholds, each informed by all items; the read-out then weights each judge by , down-weighting unreliable ones.
Additional Analyses
AGORA vs. Single-Best and Drop-Worst Baselines
We first test whether AGORA reduces to selecting the single best judge. We compare AGORA against the label-free single-best judge, the judge with the highest mean correlation to the rest of the pool, selected without consulting human labels (Table 10). AGORA exceeds this label-free single best at every size on all three datasets; on Topical-Chat the margin even grows as shrinks, from +0.005 at full to +0.039 at . AGORA combines the pool rather than collapsing onto one member.
| Dataset | Single best | AGORA |
| SummEval (full) | 0.526 | 0.556 |
| SummEval () | 0.514 | 0.550 |
| Topical-Chat (full) | 0.700 | 0.705 |
| Topical-Chat () | 0.657 | 0.696 |
| WMT22 (full) | 0.417 | 0.426 |
| WMT22 () | 0.396 | 0.409 |
Table 10. AGORA vs. the label-free single-best judge (mean-correlation argmax), Spearman on the primary pool under the same 50-subsample protocol. AGORA leads at every size on all three datasets; on the multi-dimensional benchmarks the margin grows as shrinks.
Drop-worst baseline. A second simple alternative is to drop the single worst judge and average the rest. We form a label-free drop-worst mean (remove the judge with the lowest mean correlation to the pool, then take the per-item mean of the others) and compare it to AGORA. AGORA exceeds the drop-worst mean at every size on both multi-dimensional datasets, by up to +0.016 on SummEval and +0.025 on Topical-Chat: the trust-weighted read-out, which down-weights every judge continuously, outperforms excluding one judge and trusting the rest equally.
References
- [1]Krishnakumar Balasubramanian, Aleksandr Pod-kopaev, and Shiva Prasad Kasiviswanathan. 2026. Dependence-aware label aggregation for LLM-as-a-judge via Ising models. arXiv preprint arXiv:2601.22336.arxiv.org/abs/2601.22336
- [2]Anna Bavaresco, Raffaella Bernardi, Leonardo Berto-lazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzele, Barbara Plank, David Schlangen, Alessandro Sug-lia, Aditya K Surikuchi, Ece Takmaz, and Alberto Testoni. 2025. Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), Short Papers, pages 238–255.
- [3]Manik Bhandari, Pranav Narayan Gour, Atabak Ash-faq, Pengfei Liu, and Graham Neubig. 2020. Re-evaluating evaluation in text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347–9359.arxiv.org/abs/2010.07100
- [4]Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020. With little power comes great responsibility. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9263–9274.arxiv.org/abs/2010.06595
- [5]Carlos M Carvalho, Nicholas G Polson, and James G Scott. 2010. The horseshoe estimator for sparse signals. Biometrika, 97(2):465–480.
- [6]Junhyuk Choi, Sohhyung Park, Chanhee Cho, Hyeonchu Park, and Bugeun Kim. 2026. Diagnosing the reliability of LLM-as-a-judge via item response theory. arXiv preprint arXiv:2602.00521.arxiv.org/abs/2602.00521
- [7]A. P. Dawid and A. M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, 28(1):20–28.
- [8]Gianluca Demartini, Djellel Eddine Difallah, and Philippe Cudré-Mauroux. 2012. ZenCrowd: Leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In Proceedings of the 21st International Conference on World Wide Web (WWW), pages 469–478.
- [9]Daniel Deutsch, Rotem Dror, and Dan Roth. 2021. A statistical analysis of summarization evaluation metrics using resampling methods. Transactions of the Association for Computational Linguistics, 9:1132–1146.arxiv.org/abs/2104.00054



