betaFeedback

WorksComputer Science

AGORA: Reliability-Aware Aggregation of LLM Judges on Small Evaluation Sets

Abstract

LLM-as-a-judge is a common substitute for costly human evaluation, and aggregating several judges is the standard remedy for any single judge's bias. Judge pools mix judges of very different reliability, while the human-labeled sets that validate them hold a few hundred items or fewer. In this regime the per-item mean weights weak judges as heavily as strong ones, and confusion-table models such as Dawid-Skene must estimate a per-judge error table from too few items. We propose AGORA (Aggregation Of Reliability-weighted Assessments), an item-response-theory aggregator that places every item on one latent quality scale and describes each judge by a sensitivity parameter and ordered rating thresholds, so judge reliability is learned from all items jointly and without human labels. Across three benchmarks, AGORA matches or exceeds Dawid-Skene on large evaluation sets, and its advantage widens as the set shrinks toward a few hundred items. It remains accurate as pools grow more uneven, a label-free dispersion statistic indicates in advance when trust weighting helps, and its learned trust weights track per-judge human validity. Code is available at https://github.com/JunhyungAhn/llm-as-judge-irt.

Introduction

LLM judges have become a standard tool for evaluating generative systems at scale (Zheng et al., 2023; Liu et al., 2023; Verga et al., 2024). Because individual judges show systematic biases, including length, position, and self-preference effects (Zheng et al., 2023; Wang et al., 2024b), a common remedy is to score each item with several judges and aggregate their scores. Aggregators range from simple averages to reliability-aware annotation models: Dawid–Skene [7] estimates a per-judge error table over the rating categories, MACE (Hovy et al., 2013) a competence-and-spamming profile, and GLAD (Whitehill et al., 2009) continuous judge-ability and item-difficulty parameters. All of them learn reliability from repeated judge–item overlap, so their estimates depend on having enough shared items.

This requirement is strained in LLM-judge meta-evaluation. Human reference sets are costly and small: MT-Bench scores 80 prompts (Zheng et al., 2023), Topical-Chat 360 dialogues (Mehri and Eskenazi, 2020), and SummEval 1,600 summaries (Fabbri et al., 2021). Because judge validity changes with the systems scored [3], the rating range (Peyrard, 2019), and the task or quality dimension [2] (Nimah et al. 2023), each new setting needs a fresh reference set of a few hundred items, where meta-evaluation correlations are already uncertain [4, 9].

Pool composition compounds the problem. Judge reliability varies sharply across models [2], and ensembles routinely combine models of unequal strength (Verga et al., 2024; Wang et al., 2024a), so pools are heterogeneous.1 The per-item mean gives every judge equal weight, so lower-reliability judges shift the aggregate as much as reliable ones. Confusion-table models fail differently: with a KK-point rating scale each judge carries K2K^2 error-table cells, so a few hundred items leave most cells sparse and determined by the prior. MACE and GLAD use fewer parameters but treat ratings as unordered classes and do not model how each judge uses an ordinal scale. An aggregator for this regime should learn which judges to trust without spending scarce observations on a per-judge error table.

We propose AGORA: Aggregation Of Reliability-weighted Assessments via ItemResponse Theory. From the judge-score matrix (Figure 1), AGORA fits a graded-response model, the item-response-theory model for ordinal ratings ((Samejima, 1969)), with one latent quality per item, one discrimination per judge, and ordered thresholds that capture each judge’s use of the rating scale. The discrimination acts as a reliability weight: judges whose scores track latent quality steeply receive more influence, and a regularized-horseshoe prior shrinks uninformative judges toward zero. Each judge costs KK parameters rather than K2K^2 cells, and every parameter is informed by all items, so trust estimation remains stable on small sets. Prior work uses item response theory to diagnose LLM judges ([6]); AGORA uses it to aggregate them. Our contributions are as follows:

Figure 1. AGORA on an illustrative pool of three judges rating five items on a 0–4 scale. (1) The input is the judge-score matrix alone. (2) From all items jointly, AGORA fits a graded-response model per judge, drawn as its expected rating against latent quality θ\theta. The thresholds βj\beta_j encode severity (B is lenient, A severe), and the discrimination αj\alpha_j encodes trust (bars: αj\alpha_j); the horseshoe prior shrinks the flat judge C’s trust toward zero. (3) For each item, every rating places θ\theta in that judge’s threshold interval, with influence set by the judge’s trust. The posterior concentrates where A and B agree, so the EAP estimate θ^3\hat{\theta}_3 ignores C’s outlying 0 (circled). (4) θ^i\hat{\theta}_i is mapped back to the rating scale. Against a human reference used only for evaluation, the AGORA scores reproduce the human ranking (Spearman ρ=1.00\rho= 1.00), whereas the per-item mean, shifted by C’s outliers, reaches ρ=0.70\rho= 0.70. Judge parameters and the human reference are illustrative; all other quantities are computed from them.

  1. We propose AGORA, a graded-response aggregator that learns per-judge reliability weights under a sparsity prior and models judge-specific scale use with KK parameters per judge (§ 3).

  1. We show that AGORA matches or exceeds the strongest baselines on large evaluation sets and that its advantage over Dawid–Skene grows as the set shrinks, exceeding it on 98–100% of 60 random five-judge pools with 50 items (§ 5.1–§ 5.3).

  1. We introduce agreement dispersion, a label-free statistic of the score matrix that tracks AGORA’s gain over the per-item mean (Spearman +0.73 and +0.61) and so indicates, before any human labels are collected, when trust weighting helps (§ 5.3).

  1. We show that the learned discriminations track per-judge human validity, giving a label-free judge diagnostic (§ 7).

Related Work

Aggregating LLM judges. Most LLM-judge ensembles average or majority-vote their members ((Verga et al., 2024; Wang et al., 2024a)), and some weight judges by a learned or heuristic reliability score ((Guha et al., 2024)). Qian et al. (2026) fit a Bradley–Terry model with a per-judge discrimination over pairwise comparisons, SkillAggregation ((Sun et al., 2025)) trains a neural context encoder over binary judge probabilities, CARE decomposes the score matrix into confounder-aware sparse and low-rank parts ((Zhao et al., 2026)), and Balasubramanian et al. [1] model pairwise judge dependence with Ising models. AGORA instead takes the latent-trait route on absolute ordinal scores: one continuous quality per item and a trust-weighted read-out over per-judge discriminations.

Annotation Models. Dawid–Skene, MACE, and related Bayesian annotation models ([7, 8] (Hovy et al. 2013; Paun et al. 2018)) were developed for large crowdsourcing corpora, and GLAD ((Whitehill et al., 2009)) replaces the confusion table with a latent ability–difficulty model but still relies on item redundancy. These models tie on balanced pools of comparably reliable annotators ((Paun et al., 2018)) and need much redundancy to estimate reliability stably ((Zheng et al., 2017)). We make the complementary point on the item axis: when items are few, the confusion-table parameterization fragments the data and loses its margin over a flat mean (§ 5.2), whereas a latent-trait parameterization retains it.

Item Response Theory. Item response theory (IRT) jointly scores items and raters, and its graded-response model (GRM) handles ordinal ratings ((Lord, 1980; Samejima, 1969; Rasch, 1960)). Closest to our work, Choi et al. [6] apply IRT to diagnose LLM-judge reliability. AGORA instead aggregates judges with a discrimination-weighted read-out under a regularized-horseshoe prior ([5] (Piironen and Vehtari 2017)), estimated via the augmentation of Makalic and Schmidt (2016), and adds a score-only diagnostic of when reliability weighting helps (§ 5.3).

The AGORA Aggregator

AGORA treats LLM-judge aggregation as latent-trait estimation. Each evaluated item ii has a latent quality θi\theta_i, and each judge jj has a discrimination αj\alpha_j, measuring how sharply it scores a change with latent quality, and ordered thresholds βj,k\beta_{j,k} over rating categories kk, capturing scale use such as leniency or range compression. A judge with large αj\alpha_j is informative about θi\theta_i, a judge with near-zero αj\alpha_j has little influence, and the thresholds let identical raw scores map to different latent qualities across judges. The model has four parts: the graded-response likelihood (§ 3.1), the discrimination prior (§ 3.2), an expectation-maximization (EM) procedure for maximum-a-posteriori estimation (§ 3.3), and a posterior read-out that maps each item’s latent quality to one score (§ 3.4). All quantities are fit per dimension from the judge-score matrix (Algorithm 1).

Latent-trait model

The latent-trait model ties judges through a shared latent quality for each item and models each judge’s ordinal response as a function of that quality. Let item i=1,…,Ni = 1,\ldots,N be scored by judge j=1,…,Jj = 1,\ldots,J, producing an ordinal label Yij∈{0,…,Kj−1}Y_{ij} \in\{0,\ldots,K_j-1\} on a KjK_j-point scale.2 AGORA assigns each item a latent quality θi\theta_i and models judge jj with Samejima’s graded-response model (Samejima, 1969):

P(Yij≥k∣θi)=σ(αjθi−βj,k),(1)P(Y_{ij} \ge k \mid\theta_i) = \sigma(\alpha_j\theta_i - \beta_{j,k}), \tag*{(1)}

for k=1,…,K−1k = 1,\ldots,K-1, where σ\sigma is the logistic function and the thresholds satisfy βj,1<⋯<βj,K−1\beta_{j,1} < \cdots< \beta_{j,K-1}.

The category probabilities are successive differences of the cumulative curves:

P(Yij=k∣θi)=σ(αjθi−βj,k)−σ(αjθi−βj,k+1),(2)P(Y_{ij} = k \mid\theta_i) = \sigma(\alpha_j\theta_i - \beta_{j,k}) - \sigma(\alpha_j\theta_i - \beta_{j,k+1}), \tag*{(2)}

with βj,0=−∞\beta_{j,0} = -\infty and βj,K=+∞\beta_{j,K} = +\infty. These probabilities define the likelihood used for estimation and the posterior read-out in § 3.4.

We use the standard IRT normalization θi∼N(0,1)\theta_i \sim\mathcal{N}(0,1) to fix location and scale, and constrain αj>0\alpha_j > 0 to avoid per-judge sign flips. With every αj>0\alpha_j > 0, a higher latent quality raises every judge’s expected rating, so the latent scale has a fixed orientation. This compact parameterization contrasts with a per-judge confusion table: each judge has KK GRM parameters (one discrimination and K−1K-1 thresholds) rather than K2K^2 cell probabilities. We return to this parameterization and its inference in § 5.2 and App. § 13.

The discrimination prior

The discriminations αj\alpha_j determine how strongly each judge can influence the read-out, so the prior on them controls how aggressively weak or uninformative judges are shrunk. We use a regularized horseshoe prior [5] (Piironen and Vehtari 2017), with global and local scales estimated from the score matrix rather than fixed judge weights:

αj∣λj,τ∼N+(0,τ2λ~j2),λ~j2=c2λj2c2+τ2λj2,(3)\alpha_j \mid\lambda_j,\tau\sim\mathcal{N}_{+}(0,\tau^2\tilde{\lambda}_j^2), \qquad\tilde{\lambda}_j^2 = \frac{c^2\lambda_j^2}{c^2+\tau^2\lambda_j^2}, \tag*{(3)}
λj∼C+(0,1),τ∼C+(0,1),\lambda_j \sim\mathcal{C}^{+}(0,1), \qquad\tau\sim\mathcal{C}^{+}(0,1),

where N+\mathcal{N}_{+} enforces αj>0\alpha_j > 0 and C+(0,1)\mathcal{C}^{+}(0,1) denotes a standard half-Cauchy distribution on the positive reals. The global scale τ\tau controls the overall shrinkage of discriminations, the local scales λj\lambda_j let indiindividual judges retain large values when supported by the likelihood, and the slab scale cc caps the effective prior variance to prevent extremely large discriminations. Together, these components induce sparsity over judge discriminations, shrinking uninformative judges toward low trust while allowing informative judges to remain influential. This shape suits LLM-judge pools, which mix informative and near-uninformative judges: a log-normal prior, common in Bayesian IRT, places little mass near zero and pulls discriminations toward one typical value, whereas the horseshoe shrinks only the uninformative ones [5] (Piironen and Vehtari 2017). We compare this prior against an unpenalized fit in § 6.

Estimation

Given the likelihood and prior, we estimate item qualities and judge parameters jointly with an EM procedure for maximum-a-posteriori estimation. In the E-step, we compute the posterior distribution qi(θ)=P(θi=θ∣Yi,:)q_i(\theta) = P(\theta_i = \theta\mid Y_{i,:}) on a fixed grid G\mathcal{G} for each item. In the M-step, each judge’s discrimination and thresholds are updated by minimizing the posterior-expected penalized negative log-likelihood

−∑iEqi(θ)[log⁡P(Yij∣θ;αj,βj)]+12αj2τ2λ~j2(4)-\sum_i \mathbb{E}_{q_i(\theta)}\left[\log P(Y_{ij}\mid\theta;\alpha_j,\beta_j)\right]+\frac{1}{2}\frac{\alpha_j^2}{\tau^2\widetilde{\lambda}_j^2} \tag*{(4)}

over αj\alpha_j and the ordered thresholds βj\beta_j. The penalty is the Gaussian term induced by the horseshoe prior in Eq. (3). We update the horseshoe scales at their conditional modes using the inverse-gamma augmentation of Makalic and Schmidt (2016); details are in App. § 10.

Aggregate read-out

The read-out converts the fitted model and an item’s judge scores into a single latent-quality estimate. We use the posterior mean, or expected a posteriori (EAP) estimate, computed on the grid G\mathcal{G}:

θ^i=E[θi∣Yi,:]=∑θ∈Gθ π(θ)∏jP(Yij∣θ)∑θ∈Gπ(θ)∏jP(Yij∣θ)(5)\hat{\theta}_i=\mathbb{E}[\theta_i\mid Y_{i,:}]=\frac{\sum_{\theta\in\mathcal{G}}\theta\,\pi(\theta)\prod_jP(Y_{ij}\mid\theta)}{\sum_{\theta\in\mathcal{G}}\pi(\theta)\prod_jP(Y_{ij}\mid\theta)} \tag*{(5)}

where π(θ)\pi(\theta) is the standard-normal prior discretized on G\mathcal{G} and the probabilities use the fitted judge parameters from § 3.3. We use the posterior mean rather than the posterior mode because it is more stable when item posteriors are diffuse. The role of the discriminations is visible in a standard Fisher-information linearization of the

Algorithm 1 AGORA
Input: score matrix Y∈{0,…,K−1}N×JY\in\{0,\ldots,K-1\}^{N\times J}
Output: per-item scores s^∈RN\hat{s}\in\mathbb{R}^N on the internal ordinal scale [0,K−1][0,K-1]
1: warm-start (α,β)(\alpha,\beta) from an unpenalized GRM
2: for t=1t=1 to TT do
3: E-step: grid posterior qi(θ)∝π(θ)∏jP(Yij∣θ)q_i(\theta)\propto\pi(\theta)\prod_jP(Y_{ij}\mid\theta) per item
4: M-step: per judge jj, minimize Eq. (4) over αj\alpha_j, ordered βj\beta_j
5: Scale update: refresh horseshoe scales τ,λ\tau,\lambda (App. § 10)
6: end for
7: θ^i←\hat{\theta}_i\leftarrow EAP estimate, Eq. (5)
8: s^i←\hat{s}_i\leftarrow expected-score map, Eq. (7)
9: return per-item scores s^\hat{s} (with trust α\alpha, thresholds β\beta)

Table 1.

EAP. Let Y~ij\widetilde{Y}_{ij} denote judge jj’s observed category re-expressed on the latent scale by inverting its fitted thresholds. Then, locally,

θ^i≈∑jwijY~ij∑jwij,wij=αj2vij(θ^i).(6)\hat{\theta}_i\approx\frac{\sum_j w_{ij}\widetilde{Y}_{ij}}{\sum_j w_{ij}},\qquad w_{ij}=\alpha_j^2v_{ij}(\hat{\theta}_i). \tag*{(6)}

The weight wijw_{ij} is the graded-response Fisher information that judge jj contributes at item ii (App. § 10). Its global factor αj2\alpha_j^2 makes the read-out judge-weighted, and its item-level factor vijv_{ij}, which measures how informative judge jj’s response curve is near θ^i\hat{\theta}_i, makes it item-sensitive: the same judge receives different effective weights on different items. All AGORA results are computed from the exact EAP (Eq. (5)); the αj2\alpha_j^2-weighted average (Eq. (6)) is its first-order interpretation, not a separate estimator.

Mapping to the rating scale. The EAP estimate θ^i\hat{\theta}_i is a latent coordinate, not a benchmark score. We therefore convert it to the benchmark’s rating scale using the fitted model’s expected score, i.e., the multi-judge test characteristic curve (Lord, 1980; Samejima, 1969):

s^i=1J∑j=1J∑k=0K−1k P(Yij=k∣θ^i)(7)\hat{s}_i=\frac{1}{J}\sum_{j=1}^{J}\sum_{k=0}^{K-1}k\,P(Y_{ij}=k\mid\hat{\theta}_i) \tag*{(7)}

where the category probabilities come from the fitted GRM in Eq. (2). This mapped score s^i\hat{s}_i is the reported aggregate and the value compared with the human reference. Because the map is monotone in θ^i\hat{\theta}_i, s^i\hat{s}_i preserves the item ranking induced by the latent quality; we assess it by rank correlation with the human reference, the standard meta-evaluation target.

MethodSummEval fullSummEval N=50N=50Topical-Chat fullTopical-Chat N=50N=50WMT22 fullWMT22 N=50N=50
Mean0.5210.5260.6720.6680.4160.402
Median0.5230.5290.6430.6430.4060.402
Trimmed Mean0.5380.5450.6740.6720.4180.400
Dawid–Skene0.5550.5290.6950.6780.4230.399
CARE0.5440.5340.6880.6620.4180.405
GLAD0.5450.5430.6850.6800.4190.405
MACE0.5330.5460.6900.6820.4180.404
AGORA0.5560.5500.7050.6960.4260.409

Table 1 (PDF p. 5). Main results on the primary five-judge pool. Dimension-averaged Spearman ρ\rho against human references. full: the full evaluation set (SummEval N=1,600N=1{,}600, Topical-Chat N=360N=360, WMT22 N=19,725N=19{,}725). N=50N=50: mean over 50 random item subsets of 50 items, shared by all methods (paired tests in § 5.1). Bold: best point estimate in each column.

Experimental Setup

We evaluate on three meta-evaluation benchmarks with human ratings. SummEval (Fabbri et al., 2021) contains 1,600 news summaries rated by three experts on four dimensions (coherence, consistency, fluency, relevance) on a 1–5 scale. Topical-Chat (Mehri and Eskenazi, 2020) contains 360 dialogue responses rated by three annotators on six dimensions with 2-, 3-, and 5-point scales. WMT22 en–de (Freitag et al., 2022) contains 19,725 translated segments with professional MQM quality scores. SummEval and Topical-Chat are the small human-referenced sets that motivate our setting, while WMT22 tests AGORA on a large evaluation set.

Each judge is a single model evaluated with a fixed prompt, drawn from a 14-model candidate pool spanning proprietary frontier and open-weight model families (App. § 11.§ 1). Our primary pool has five judges and is heterogeneous by construction: one proprietary frontier judge (claude-sonnet-4-6) and four open-weight judges from distinct model families (Llama-4-Scout, DeepSeek-R1, GLM-4.7-Flash, Nemotron-120B). The pool was fixed by model name before fitting any aggregation method, without consulting benchmark-specific human references, and is held constant across datasets and methods. We test generality beyond this pool on random five-judge pools drawn from the candidate pool: 30 pools on all three benchmarks across evaluation-set sizes, and 60 further pools on SummEval and Topical-Chat (§ 5.3).

We report Spearman ρ\rho against the human reference, averaged across dimensions on multidimensional benchmarks. For each evaluation-set size NN below the full sample, every method is re-fit on the same 50 random item subsets, and we report the mean over subsets. Small-sample contrasts use a paired bootstrap over per-subset differences, and full-sample contrasts use an item-level paired bootstrap (n=1000n=1000). We compare against the per-item mean, median, trimmed mean (discarding the highest- and lowest-scoring judge per item), Dawid–Skene, MACE, GLAD, and CARE, a confounder-aware low-rank aggregator; implementation details are in App. § 11.§ 3.

Results

Main Results

AGORA has the highest point estimate in every column of Table 1, on the full sample and at N=50N=50, the smallest size of the subsampling ladder in § 5.2. At full NN it matches Dawid–Skene on SummEval (0.556 vs. 0.555), exceeds it on Topical-Chat (0.705 vs. 0.695) and WMT22 (0.426 vs. 0.423), and improves on the per-item mean by +0.035, +0.033, and +0.010. It also exceeds MACE, GLAD, and CARE at both sizes on every benchmark (full-NN margins +0.007 to +0.023); the item-level bootstrap confirms these separations at full NN, and the paired subset bootstrap confirms the N=50N=50 gains on SummEval and Topical-Chat.

Performance as Evaluation Sets Shrink

Figure 2 follows the small-sample regime from N=400N=400 down to N=50N=50, where AGORA is the top aggregator at every size on all three benchmarks. Dawid–Skene’s per-judge confusion tables receive less evidence as NN shrinks: on SummEval it moves from parity with AGORA at full NN (Table 1) to a 0.021 deficit at N=50N=50, and on WMT22 it falls below the per-item mean. Across the shared subsets, AGORA exceeds the per-item mean on at least 86% of subsets at every size (p<0.01p < 0.01 against chance) and Dawid–Skene on 78–92%.

Figure 2. Data efficiency on the primary five-judge pool in the small-sample regime (N≤400N \le400). Dimension-averaged gap to the per-item mean, Δρ=ρmethod−ρMean\Delta\rho= \rho_{\mathrm{method}} - \rho_{\mathrm{Mean}}, as NN shrinks (larger NN on the left). AGORA is the top aggregator at every size; Dawid–Skene loses margin as NN shrinks.

AGORA Gains Grow with Pool Heterogeneity

Trust weighting should help most when a pool’s judges agree unevenly. We measure this with agreement dispersion, δ=std⁡j(ρˉj)/mean⁡j(ρˉj)\delta= \operatorname{std}_{j}(\bar{\rho}_{j})/\operatorname{mean}_{j}(\bar{\rho}_{j}), where ρˉj\bar{\rho}_{j} is judge jj’s mean rank correlation with the other judges in the pool; larger δ\delta means more uneven agreement. Because δ\delta uses only the score matrix, it is available before any human labels are collected.

Random pools. On 30 random five-judge pools from the candidate pool (Table 2), AGORA exceeds Dawid–Skene at N=50N=50 in 93%, 100%, and 83% of pools on SummEval, Topical-Chat, and WMT22 (mean margins +0.017, +0.015, +0.019), and stays at or above the per-item mean at every size. On a second draw of 60 pools at N=50N=50, it exceeds Dawid–Skene in 98% and 100% of pools on SummEval and Topical-Chat.

DatasetNNMeanDawid–SkeneAGORA
SummEval500.5100.5010.518
SummEval1000.5450.5460.555
SummEval2000.5420.5450.547
Topical-Chat500.7150.7030.718
Topical-Chat1000.6970.6990.703
Topical-Chat2000.7030.7080.709
WMT22500.4420.4230.442
WMT221000.4020.3930.403
WMT222000.4280.4270.429

Table 2. Thirty random five-judge pools: dimension-averaged Spearman ρ\rho, averaged over pools and over 15, 12, and 10 item subsets at N=50N=50, 100, and 200. AGORA has the highest average at every size.

Gain and dispersion. On these 60 pools, AGORA’s gain over the per-item mean grows with δ\delta (Spearman +0.73 on SummEval and +0.61 on Topical-Chat; p<0.001p < 0.001), as does its gain over Dawid–Skene on SummEval (+0.53, p<0.001p < 0.001). On pools at least as dispersed as the primary pool (δ≥0.29\delta\ge0.29 on SummEval, δ≥0.15\delta\ge0.15 on Topical-Chat; 6 and 15 pools), it exceeds the per-item mean in 100% and 93% of cases. On balanced pools it recovers the per-item mean as a limiting case at no accuracy cost: on the full 14-judge SummEval pool (δ=0.21\delta=0.21) it obtains 0.577 against the mean’s 0.572 (App. § 11), as expected from the convergence of annotation models on balanced data ((Paun et al., 2018)).

Controlled dispersion sweep. Adding 0–8 uninformative judges with uniformly random ratings to random five- and seven-judge pools extends δ\delta from its natural range (about 0.3) to about 1.1 (Figure 3). Across this range, AGORA stays within 0.543–0.562 on SummEval and 0.689–0.712 on Topical-Chat, while the per-item mean falls from 0.536 to 0.436 and from 0.703 to 0.612, and Dawid–Skene from 0.533 to 0.469 and from 0.697 to 0.671.

Figure 3. Accuracy at N=50N=50 as agreement dispersion δ\delta increases, over random five- and seven-judge pools with 0–8 uninformative judges (uniformly random ratings) added; points are means within δ\delta quintiles. AGORA remains stable while the per-item mean and Dawid–Skene decline. Dashed line: δ\delta of the primary pool.

Ablations and Label-Free Baselines

Fitted trust weights drive the gain. Holding the fitted model fixed, we vary only the read-out’s trust vector (Table 3): the raw per-item mean (R0), the EAP under one pooled discrimination shared by all judges (R1), and the exact EAP with fitted per-judge discriminations (R2). At full NN the fitted weights supply the gain (R1 to R2: +0.023 on SummEval, +0.033 on Topical-Chat); at N=50N=50 the latent parameterization itself adds +0.014 and +0.016, consistent with § 5.2.

Read-outSummEval fullSummEval N=50N=50Topical-Chat fullTopical-Chat N=50N=50
R0: per-item mean0.5210.5260.6720.668
R1: latent, equal trust0.5330.5400.6720.684
R2: latent, αj2\alpha_j^2 trust0.5560.5500.7050.696

Table 3. Read-out decomposition: the fit is held fixed and only the trust vector changes. At full NN the fitted-discrimination read-out (R2−R1) supplies most of the gain over the mean; at N=50N=50 both the latent read-out and the fitted trust contribute.

The prior adds robustness at no clean-data cost. Replacing the regularized horseshoe with an unpenalized (flat MLE) fit (Table 4), the two match on clean pools (0.554 vs. 0.551 on SummEval, 0.705 vs. 0.697 on Topical-Chat). With m=2m=2 injected adversarial judges, each mirroring an existing judge’s scores around the rating midpoint, the horseshoe improves on the flat fit by +0.008 and +0.017. Neither the prior nor the fixed estimation settings are tuned on human labels (App. § 10).

prior on α\alphaclean (full NN) SEclean (full NN) TCadversarial (m=2m=2) SEadversarial (m=2m=2) TC
flat (MLE)0.5510.6970.5270.663
horseshoe0.5540.7050.5350.680

Table 4. Discrimination-prior ablation (mm: number of injected adversarial judges). The regularized horseshoe matches the unpenalized (flat MLE) fit on clean pools and improves on it under adversarial corruption.

Learned weights outperform label-free heuristics. AGORA exceeds weighted means based on mean inter-judge correlation, inverse variance, and a spectral meta-learner (Table 5; 0.556 vs. the best heuristic’s 0.550 on SummEval, 0.705 vs. 0.686 on Topical-Chat).

Mean ρˉj\bar{\rho}_jwt.inv-varspectralAGORA
SummEval (full)0.5210.5500.5410.5480.556
Topical-Chat (full)0.6720.6860.6740.6830.705

Table 5. AGORA vs. label-free continuous reliability weightings (Spearman ρ\rho, primary pool, full NN): mean-correlation (ρˉj\bar{\rho}_j), inverse-variance, and a spectral meta-learner. AGORA is best on both datasets.

Pools with the strongest judge. AGORA also exceeds the label-free single-best judge and a drop-worst average (App. § 14.§ 1). When the strongest candidate judge, gemma-4–31B (App. § 11.§ 2), is added to the primary pool and every method is scored on held-out items, AGORA exceeds the per-item mean and Dawid–Skene in all eight benchmark-by-size settings (Table 6). On a pool of the five most valid judges, it exceeds the best single judge at every size by +0.015 to +0.046, and with gemma-4–31B in every random pool it still exceeds Dawid–Skene at N=50N=50 in 97% and 100% of 30 pools on SummEval and Topical-Chat.

DatasetNNMeanDSAGORAgemmaAGORA
SummEval500.5410.5490.5690.5720.588
SummEval2000.5430.5620.5730.5480.582
SummEvalfull0.5520.5740.5780.5630.586
Topical-Chat500.6970.7010.7170.7080.750
Topical-Chat2000.7050.7220.7320.7040.745
Topical-Chatfull0.6960.7110.7200.6950.735
WMT22500.3940.3990.4010.3830.428
WMT222000.4270.4260.4370.4310.473

Table 6. Pools containing the strongest judge, gemma-4-31B, scored on held-out items (75%; 10 and 5 item subsets at N=50N=50 and 200). Left: the primary pool plus gemma. Right: the five judges with the highest human validity on a separate calibration quarter, against its best member, gemma alone.

Interpreting the Trust Weights

AGORA’s fitted trust weights αj2\alpha_j^2 also serve as a label-free diagnostic of judge quality. Compared with each judge’s post-hoc Spearman correlation with human labels on the full 14-judge pool (Figure 4), the dimension-averaged correlation is 0.65 on SummEval, 0.82 on Topical-Chat, and 0.66 on WMT22. Because αj2\alpha_j^2 is learned without human labels, it gives a practical screen for judge quality before any annotation.

Figure 4. Trust weights track judge validity on the full 14-judge pool. Each point is one judge on one dimension: post-hoc validity rjr_j (Spearman with human labels) against the fitted trust weight αj2\alpha_j^2; ρˉs\bar{\rho}_s is their dimension-averaged Spearman correlation.

Conclusion

AGORA aggregates LLM judges of uneven reliability with a graded-response latent-trait model whose discrimination-weighted read-out replaces per-judge confusion tables. Its compact parameterization keeps reliability estimation data-efficient where confusion tables fragment, so its margin over Dawid–Skene widens as evaluation sets shrink.

decisions.

The learned discriminations also serve as a label-free proxy for judge validity, and the dispersion statistic δ\delta signals when trust weighting improves on the simple mean.

Limitations

Our experiments cover three meta-evaluation benchmarks spanning summarization, dialogue, and machine translation; further task families and deployment settings are future work. AGORA targets absolute ordinal ratings, the format of these benchmarks; pairwise preference data would call for a Bradley–Terry-style likelihood, and joint modeling across dimensions and dependence-aware likelihoods [1] (Zhao et al. 2026) are natural extensions. AGORA is intended for meta-evaluation of LLM judges, not as a replacement for human judgment in high-stakes decisions.

Estimation Details

We compute the maximum-a-posteriori estimate by EM, treating {θi}\{\theta_i\} as latent and {αj,βj,λj,τ}\{\alpha_j,\beta_j,\lambda_j,\tau\} as parameters (Algorithm 1). Direct half-Cauchy MAP updates are awkward, so we use the inverse-gamma augmentation of Makalic and Schmidt (2016), which writes each half-Cauchy as a scale mixture with auxiliary variables νj,ξ\nu_j,\xi and yields closed-form conditional modes for the local and global scales. The E-step forms the posterior over θi\theta_i on a fixed grid G\mathcal{G} by combining the prior with the likelihood (2) across judges and takes the posterior mean θ^i\hat{\theta}_i. The M-step minimizes Eq. (4) over αj∈[0.01,20]\alpha_j \in[0.01,20] and the ordered thresholds (parameterized through a cumulative-softplus map), the penalty being exactly the Gaussian induced by the horseshoe (3). The scale update then refreshes the scales at their conditional modes:

λj2←12(νj−1+αj2/2τ2),νj←12(1+λj−2),\lambda_j^2 \leftarrow\frac{1}{2}\left(\nu_j^{-1}+\alpha_j^2/2\tau^2\right), \qquad\nu_j \leftarrow\frac{1}{2}\left(1+\lambda_j^{-2}\right),
τ2←ξ−1+∑jαj2/2λj2J/2+1,ξ←12(1+τ−2).\tau^2 \leftarrow\frac{\xi^{-1}+\sum_j\alpha_j^2/2\lambda_j^2}{J/2+1}, \qquad\xi\leftarrow\frac{1}{2}\left(1+\tau^{-2}\right).

followed by the slab cap λ~j2←c2λj2/(c2+τ2λj2)\widetilde{\lambda}_j^2 \leftarrow c^2\lambda_j^2/(c^2+\tau^2\lambda_j^2). We warm-start from a 6-iteration unpenalized graded-response fit and run T=12T=12 iterations, fitting one model per dimension; the auxiliaries are initialized to λj,νj,τ,ξ←1\lambda_j,\nu_j,\tau,\xi\leftarrow1. The few fixed quantities are held constant across all datasets: slab scale c=5c=5, the discrimination range αj∈[0.01,20]\alpha_j \in[0.01,20], a posterior grid G\mathcal{G} of 150 points on [−4.5,4.5][-4.5,4.5], and an L-BFGS-B M-step. As a numerical safeguard, the implementation also checks that θ^\hat{\theta} correlates positively with the per-item mean and flips its sign otherwise.

Information factorization. The read-out weight in Eq. (6) is the graded-response Fisher information

Iij(θ)=∑k[Pijk′(θ)]2Pijk(θ)=αj2vij(θ),I_{ij}(\theta)=\sum_k\frac{\left[P'_{ijk}(\theta)\right]^2}{P_{ijk}(\theta)}=\alpha_j^2v_{ij}(\theta),

where Pijk(θ)≡P(Yij=k∣θi=θ)P_{ijk}(\theta)\equiv P(Y_{ij}=k\mid\theta_i=\theta) is the category probability from Eq. (2), Pijk′P'_{ijk} its derivative in θ\theta, and vij(θ)v_{ij}(\theta) the item-level information factor.

Sensitivity. These settings are not tuned. Varying each over a wide range, and perturbing the warm-start initialization, leaves the full-NN accuracy essentially unchanged (Table 7).

PerturbationSETCWMT
slab c∈{3,8,12}c \in\{3, 8, 12\}0.00000.00010.0000
grid ∈{100,200,250}\in\{100, 200, 250\}0.00000.00000.0000
EM iters ∈{8,16,20}\in\{8, 16, 20\}0.00020.00000.0000
warm-start (5 seeds)0.00000.00000.0000

Table 7. Estimation sensitivity: the maximum change in full-NN Spearman ρ\rho when each fixed setting is varied from its default (c=5c=5, grid 150, T=12T=12) and when the warm-start is randomly perturbed. Base ρ\rho is 0.556 (SummEval), 0.705 (Topical-Chat), and 0.426 (WMT22); every shift is at most 0.0002.

Judge Pool and Prompts

Candidate Pool and Prompt Templates

We use a diverse set of instruction-tuned LLMs as judges. The 14-model pool comprises three proprietary models (gpt-5, gemini-2.5-pro, claude-sonnet-4-6) and eleven open-weight models (DeepSeek-V3, DeepSeek-R1, Qwen3-Next-80B, Qwen2.5-72B, Llama-4-Scout, gpt-oss-120b, Mixtral-8x22B, Phi-4, gemma-4-31B, GLM-4.7-Flash, Nemotron-120B), spanning a wide range of size, architecture, and training data, a heterogeneous pool of judges. For each dataset and evaluation dimension, every judge scores a single item on the benchmark’s integer scale and returns the score as JSON, decoded at temperature 0. No human labels are used to tune or calibrate the judges; human annotations are used solely for evaluation. The prompts for the three datasets are shown below.

SummEval

You are an expert evaluator assessing {dimension}.
Source Article: {source}
Summary: {summary}
Rate on 1–5 (5=Excellent ... 1=Very Poor).
Output ONLY valid JSON: {"score": N}

{dimension}: coherence, consistency, fluency, relevance. The Source Article line is included only for consistency and relevance.

Topical-Chat

You are an expert evaluator of open-domain dialog responses.
Assess {dimension}.
Dialog history: {context}
Background fact: {fact}
Candidate response: {response}
Rate on integer scale {lo}–{hi} ({anchors}).
Output ONLY valid JSON: {"score": N}

{dimension}: the six USR dimensions (Understandable 0–1; Natural, Maintains Context, Engaging each 1–3; Uses Knowledge 0–1; Overall 1–5); {anchors} gives that scale’s endpoint labels.

WMT22 en–de
You are an expert evaluating machine translation quality.
Source (English): {src}
Translation (German): {mt}
Rate overall quality on 1–5 (5=Excellent …
1=Very Poor).
Output ONLY valid JSON: {"score": N}

Full-Pool Validity Diagnostics

Table 8 reports each judge’s Spearman ρ\rho with the human labels on the full 14-judge pool (dimension-averaged), from which our primary pool is drawn (its composition is given in § 4). Validity is dispersed (0.40 to 0.57 on SummEval). This full pool has lower agreement dispersion than the primary pool on SummEval (δ\delta=0.21 vs. 0.29); here AGORA obtains 0.577 against the per-item mean’s 0.572, matching the convergence of annotation models on balanced pools ((Paun et al., 2018)).

JudgeSummEvalTopical-ChatWMT22
gpt-50.5300.6290.427
gemini-2.5-pro0.5500.6550.421
claude-sonnet-4-60.5310.7000.417
DeepSeek-V30.4910.6520.384
Qwen3-Next-80B0.4950.5670.358
Llama-4-Scout0.4220.4400.312
DeepSeek-R10.4450.4920.358
gpt-oss-120b0.4800.5840.368
Mixtral-8x22B0.5200.6350.366
Phi-40.4640.5500.331
Qwen2.5-72B0.4390.6030.337
gemma-4-31B0.5660.7040.432
GLM-4.7-Flash0.4120.5030.322
Nemotron-120B0.4020.5000.347

Table 8. Per-judge Spearman ρ\rho vs. human labels on the full pool (dimension-averaged).

Baseline Implementations

All baselines see the same judge-score matrix as AGORA and use no human labels. The per-item mean and median average or take the median of the raw scores across judges. The trimmed mean sorts each item’s judge scores and discards a fraction ⌊0.2J⌋\lfloor0.2J \rfloor from each end before averaging; at J=5J=5 this removes the single highest- and lowest-scoring judge and averages the remaining three. Dawid–Skene, MACE, and GLAD are the crowd-kit library implementations, run for 10 EM iterations (MACE with one restart; GLAD with five inner M-step iterations). Because these estimators are categorical, each integer rating level is treated as a separate class; we recover a scalar per item as the posterior expected rating ∑kkP(Y=k)\sum_k kP(Y=k). This categorical treatment discards the ordinal structure that AGORA’s graded-response link uses. CARE ((Zhao et al., 2026)) is run at its label-free library default (hyperparameter γ\gamma=1) with the SCS conic solver, taking the score-weighted quality vector as output. Each aggregator returns one scalar per item, standardized before the Spearman comparison.

Additional Evaluation Results

WMT22 under Field-Standard Metrics

We report Spearman ρ\rho throughout for comparability across datasets, but machine-translation evaluation conventionally uses segment-level Kendall τb\tau_b and pairwise accuracy ((Freitag et al., 2022)). Table 9 recomputes the full-NN WMT22 results under all three: AGORA ranks first on every metric, and the per-item mean is last on pairwise accuracy, so the conclusion does not depend on the rank-correlation protocol.

MethodSpearman ρ\rhoKendall τb\tau_bpairwise acc.
Mean0.4160.34180.619
GLAD0.4190.33620.644
Dawid–Skene0.4230.33960.645
AGORA0.4260.34190.647

Table 9. WMT22 (full NN) under three agreement metrics. AGORA ranks first on all three; the per-item mean trails on pairwise accuracy.

Parameterization Diagnostic

Figure 5 contrasts the confusion-table and latent-trait parameterizations on a small evaluation set, complementing the small-NN analysis in § 5.2. A KK-category confusion matrix gives each judge O(K2)\mathcal{O}(K^2) parameters; for SummEval’s K=5K=5 the five-judge pool carries 125, against AGORA’s 25. As NN shrinks the confusion cells become sparse: 25% have fewer than five observations at full NN but 86% at NN=50, so many cells become prior-dominated, making the high-dimensional estimate less data-efficient, which is why Dawid–Skene loses its full-NN accuracy on SummEval at small NN while AGORA retains its margin.

Comparison of confusion-matrix and latent-trait parameterizations

Figure 5. Two parameterizations on the same small evaluation set (N=50N=50 items, J=5J=5 judges, K=5K=5). Left: confusion-matrix aggregators give each judge a K×KK \times K error table; with few items most cells are prior-dominated, so the estimate becomes less data-efficient. Right: AGORA maps every item onto one latent quality θ\theta and summarizes each judge with a discrimination and ordered thresholds, each informed by all NN items; the read-out then weights each judge by αj2\alpha_j^2, down-weighting unreliable ones.

Additional Analyses

AGORA vs. Single-Best and Drop-Worst Baselines

We first test whether AGORA reduces to selecting the single best judge. We compare AGORA against the label-free single-best judge, the judge with the highest mean correlation to the rest of the pool, selected without consulting human labels (Table 10). AGORA exceeds this label-free single best at every size on all three datasets; on Topical-Chat the margin even grows as NN shrinks, from +0.005 at full NN to +0.039 at N=50N=50. AGORA combines the pool rather than collapsing onto one member.

DatasetSingle bestAGORA
SummEval (full)0.5260.556
SummEval (N=50N=50)0.5140.550
Topical-Chat (full)0.7000.705
Topical-Chat (N=50N=50)0.6570.696
WMT22 (full)0.4170.426
WMT22 (N=50N=50)0.3960.409

Table 10. AGORA vs. the label-free single-best judge (mean-correlation argmax), Spearman ρ\rho on the primary pool under the same 50-subsample protocol. AGORA leads at every size on all three datasets; on the multi-dimensional benchmarks the margin grows as NN shrinks.

Drop-worst baseline. A second simple alternative is to drop the single worst judge and average the rest. We form a label-free drop-worst mean (remove the judge with the lowest mean correlation to the pool, then take the per-item mean of the others) and compare it to AGORA. AGORA exceeds the drop-worst mean at every size on both multi-dimensional datasets, by up to +0.016 on SummEval and +0.025 on Topical-Chat: the trust-weighted read-out, which down-weights every judge continuously, outperforms excluding one judge and trusting the rest equally.

References

  1. [1]Krishnakumar Balasubramanian, Aleksandr Pod-kopaev, and Shiva Prasad Kasiviswanathan. 2026. Dependence-aware label aggregation for LLM-as-a-judge via Ising models. arXiv preprint arXiv:2601.22336.arxiv.org/abs/2601.22336
  2. [2]Anna Bavaresco, Raffaella Bernardi, Leonardo Berto-lazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzele, Barbara Plank, David Schlangen, Alessandro Sug-lia, Aditya K Surikuchi, Ece Takmaz, and Alberto Testoni. 2025. Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), Short Papers, pages 238–255.
  3. [3]Manik Bhandari, Pranav Narayan Gour, Atabak Ash-faq, Pengfei Liu, and Graham Neubig. 2020. Re-evaluating evaluation in text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347–9359.arxiv.org/abs/2010.07100
  4. [4]Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020. With little power comes great responsibility. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9263–9274.arxiv.org/abs/2010.06595
  5. [5]Carlos M Carvalho, Nicholas G Polson, and James G Scott. 2010. The horseshoe estimator for sparse signals. Biometrika, 97(2):465–480.
  6. [6]Junhyuk Choi, Sohhyung Park, Chanhee Cho, Hyeonchu Park, and Bugeun Kim. 2026. Diagnosing the reliability of LLM-as-a-judge via item response theory. arXiv preprint arXiv:2602.00521.arxiv.org/abs/2602.00521
  7. [7]A. P. Dawid and A. M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, 28(1):20–28.
  8. [8]Gianluca Demartini, Djellel Eddine Difallah, and Philippe Cudré-Mauroux. 2012. ZenCrowd: Leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In Proceedings of the 21st International Conference on World Wide Web (WWW), pages 469–478.
  9. [9]Daniel Deutsch, Rotem Dror, and Dan Roth. 2021. A statistical analysis of summarization evaluation metrics using resampling methods. Transactions of the Association for Computational Linguistics, 9:1132–1146.arxiv.org/abs/2104.00054

Paper details

vlnc:W_wn9bvuyc
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents