betaFeedback

WorksComputer Science

VoirDire: Counterfactual Calibration for Reliable LLM-as-a-Judge Aggregation

Abstract

LLM-as-a-judge systems often aggregate multiple judges to improve evaluation reliability. However, judge agreement can reflect shared biases toward verbosity, formatting, or fluency rather than the intended evaluation target. Addressing these shared biases is challenging because observational judge outputs alone cannot distinguish agreement driven by target quality from agreement driven by nuisance attributes. We introduce VoirDire, an intervention-calibrated aggregation method for LLM-judge panels. VoirDire applies controlled interventions that vary specified surface attributes while aiming to preserve the evaluation target; it then uses paired changes in judge scores to identify nuisance-sensitive directions. It attenuates these directions and selects aggregation weights that retain variation across items while penalizing sensitivity to the interventions, making the final consensus less dependent on superficial features. We show theoretically that observational judge outputs alone cannot identify which shared direction is target-relevant, while paired interventions provide the contrast needed to identify nuisance-sensitive variation. We evaluate VoirDire on 12 datasets spanning continuous scoring, binary classification, and pairwise preference. Compared with the existing LLM-judge aggregation baselines, VoirDire reduces MAE by up to 26.8% on continuous-score tasks and improves accuracy by up to 5.3 percentage points on binary and preference tasks. These findings suggest that counterfactual interventions provide an additional means of improving the reliability and controllability of LLM-based evaluation beyond merely relying on observed judge agreement.

Introduction

LLM-as-a-judge systems are increasingly used to score open-ended answers, compare assistants, filter data, and approximate human evaluations [28, 41] (Zheng et al. 2023; Zhu et al. 2025). Because a single judge can be noisy, recent work has explored aggregating multiple LLM judges [40, 26, 9] (Li et al. 2026; Zhao et al. 2026). The premise is that agreement among judges can reveal the underlying quality signal by averaging out individual judge noise.

However, this premise breaks down when judges share the same nuisance sensitivity. LLM judges exhibit biases related to position, selection effects, and formatting artifacts [27, 33, 36, 25] (Zeng et al. 2024; Ye et al. 2025). When such biases are shared across judges, aggregation may reinforce rather than mitigate them. Judges can move together because an answer is genuinely better, but they can also move together because the answer is

Figure 1. Motivation. Surface attribute changes systematically shift judge scores, so aggregation can track nuisance factors rather than the evaluation target.

longer, better formatted, more fluent, or easier to parse. From observed scores alone, true quality and nuisance factors can be hard to distinguish.

Meanwhile, prior work has developed more sophisticated judge aggregation methods beyond majority voting, for example by estimating judge reliability, using higher-order agreement statistics, or modeling judge dependence [4, 9]. These tools improve aggregation by modeling how judge outputs interact with the latent quality signal. However, when shared nuisance factors induce score structure that resembles true quality, richer observational modeling can still amplify the wrong signal. More directly related to our work, Zhao et al. (2026) address shared nuisance factors by additionally introducing latent variables that capture their effects across judges. However, it still relies on observational judge outputs and structural assumptions such as sparsity and low rank. As a result, it does not provide a direct, controllable means of probing judges’ sensitivity to specific nuisance attributes.

We introduce VoirDire, an intervention-based calibration procedure for LLM-judge aggregation, named after voir dire, the legal process of examining prospective jurors. VoirDire applies controlled interventions to the inputs shown to judges, using deterministic transformations or LLM-generated rewrites to perturb specified nuisance attributes while aiming to preserve the evaluation target. The same judges then score both the original and modified inputs. Paired score differences indicate which directions in judge-output space are sensitive to the targeted attributes. VoirDire attenuates these directions and selects aggregation weights that retain variation across items while penalizing sensitivity to the interventions.

Theoretically, we show that observational agreement alone can fail to identify the target signal, while paired original–rewrite responses enable estimation of intervention-sensitive nuisance variation, with error controlled by intervention quality and calibration sample size.

Empirically, we evaluate VoirDire on 12 datasets spanning continuous scoring, binary classification, and pairwise preference. Compared with the strongest existing judge-aggregation baseline on each dataset, VoirDire reduces MAE by up to 26.8% on continuous-score tasks and improves accuracy by up to 5.3 percentage points on binary/preference tasks. Controlled experiments further show that VoirDire reduces aggregate sensitivity to targeted nuisance changes. We also examine the effect of calibration-pair budget and interpret how calibration changes judge scores and aggregation weights. In summary, our contributions include:

  1. We introduce VoirDire, an intervention-based calibration method that uses controlled perturbations to mitigate shared nuisance sensitivity in LLM-judge aggregation.

  2. We characterize the limitations of observational judge agreement and establish conditions under which paired interventions enable accurate estimation of nuisance-sensitive directions.

  3. Across 12 datasets spanning continuous scoring, binary classification, and pairwise preference, VoirDire reduces MAE by up to 26.8% and improves accuracy by up to 5.3 percentage points over existing aggregation methods.

Related Work

LLM-as-a-judge evaluation. LLM judges are widely used to evaluate open-ended generation because they scale beyond manual annotation and can express task-specific rubrics [28, 41] (Zheng et al. 2023; Zhu et al. 2025). However, their judgments can depend on presentation, verbosity, position, selection effects, and other nuisance factors rather than only the intended evaluation target [27, 33, 36, 25] (Zeng et al. 2024; Ye et al. 2025). Aggregating multiple judges can reduce individual noise, but it does not necessarily eliminate shared bias.

Judge aggregation. Classical crowdsourcing and weak-supervision methods infer latent labels from multiple noisy sources by modeling source reliability, dependence, or latent structure [15, 22, 35, 37]. These approaches naturally extend to aggregating LLM judges. Recent LLM-jury methods use richer observational statistics, including higher-order agreement signals such as Inverse Surprising Popularity (ISP) and dependence-aware Ising models [4, 9]. However, these methods do not explicitly account for nuisance factors shared across judges. More directly related to our work, CARE (Zhao et al., 2026) models such shared factors using latent variables inferred from observed judge outputs. Still, its identification relies on observational outputs and structural assumptions such as sparsity and low rank. VoirDire instead uses controlled interventions to directly measure which judge-output directions respond to nuisance changes.

Counterfactual interventions and causal analysis. Causal inference distinguishes observational association from the effects revealed by controlled interventions [34]. In NLP, this view has been used to estimate the causal effects of textual concepts on model behavior [19, 2, 20] and to make models invariant to spurious attributes using counterfactual pairs [23, 39]; foundation models can now generate such counterfactuals at scale [13, 21] (Wu et al. 2021). VoirDire brings this causal view to judge aggregation.

VoirDire: Calibrating Judge Aggregation with Interventions

VoirDire calibrates LLM-judge aggregation using controlled interventions. It first constructs original–rewrite pairs designed to preserve the evaluation target while varying specified nuisance attributes, such as formatting, style, or verbosity. The same judges evaluate both inputs, and their paired score differences reveal nuisance-sensitive directions in judge-output space. VoirDire uses these directions to calibrate judge scores and determine intervention-aware aggregation weights. Figure 2 gives an overview, and algorithm 1 summarizes the procedure.

Figure 2. VoirDire framework. Controlled interventions give paired judge-score shifts, pooled into Gˉ\bar{G}. At evaluation time, judge outputs are attenuated along its directions (eq. (2)) and weighted to retain item variation while penalizing sensitivity to the shifts (eq. (3)).

Setup. We consider a panel of pp LLM judges. For item ii, the judges see a judge input uiu_i and produce a judge-output vector xi=(xi1,…,xip)x_i = (x_{i1}, \ldots, x_{ip}). Entries may be scalar scores, binary labels, or preference votes. A standard aggregator, such as majority voting, uses only the unperturbed judge-output matrix X={xi}i=1nX = \{x_i\}_{i=1}^{n} to predict a target score or label.

Constructing intervention pairs. We begin with a pool of candidate intervention families targeting nuisance attributes such as formatting, verbosity, and candidate position. Each family aa specifies three components: the nuisance attribute and input span to modify, a modification rule, and preservation constraints. The modification rule is either deterministic (e.g., adding markup or swapping two candidates verbatim) or an instruction to an LLM rewrite model. For example, a concision family instructs the model to remove redundancy while preserving the original claims, sentiment, numbers, and other content constraints.

In our experiments, we use LLM assistants to propose candidate intervention families based on dataset examples. We apply each intervention to a subset of inputs and retain only pairs that pass programmatic preservation checks. The same judges score each retained modified input, yielding xi(a)x_i^{(a)} for i∈Pai \in\mathcal{P}_a. Pair generation and preservation checking use no ground-truth labels. Full construction and selection details appear in Appendix B.

Estimating nuisance-sensitive directions. Given mm calibration pairs from each of the kk selected intervention families, we compute the paired score differences Δi(a)=xi(a)−xi(a)\Delta_i^{(a)} = x_i^{(a)} - x_i^{(a)} for i∈Pai \in\mathcal{P}_a. These vectors record how the judges respond to each intervention. When the evaluation target remains unchanged, they capture nuisance-sensitive variation together with judge noise. We pool the differences into a second-moment matrix1:

Gˉ=1km∑a=1k∑i∈PaΔi(a)Δi(a)⊤,∣Pa∣=m,M=km.(1)\bar{G} = \frac{1}{km} \sum_{a=1}^{k} \sum_{i \in\mathcal{P}_a} \Delta_i^{(a)} \Delta_i^{(a)\top}, \qquad|\mathcal{P}_a| = m,\quad M = km. \tag*{(1)}

Each pair contributes equally to Gˉ∈Rp×p\bar{G} \in\mathbb{R}^{p \times p}. Its entry Gˉjℓ\bar{G}_{j\ell} captures how judges jj and ℓ\ell respond together, and its leading eigenvectors identify the directions along which the interventions produce the largest score changes. These directions capture nuisance sensitivity, although judge noise and imperfections in the interventions can affect their estimation.

Attenuating sensitive directions. Using the estimated directions, we calibrate each judge-output vector by attenuating components along directions that respond more strongly to the interventions:

x~i=xi−αGˉxiλmax⁡(Gˉ),α∈[0,1].(2)\tilde{x}_i = x_i - \alpha\frac{\bar{G}x_i}{\lambda_{\max}(\bar{G})}, \qquad\alpha\in[0,1]. \tag*{(2)}

Here λmax⁡(Gˉ)\lambda_{\max}(\bar{G}) is the largest eigenvalue, and α\alpha is a hyperparameter controlling attenuation strength. Normalizing by λmax⁡(Gˉ)\lambda_{\max}(\bar{G}) ensures that the most sensitive direction is attenuated by a fraction α\alpha, while less sensitive directions are attenuated less.

Intervention-aware aggregation. After calibrating the judge outputs, we aggregate them using weights that retain variation across items while penalizing sensitivity to the interventions:

w⋆∈arg⁡max⁡w≠0 Var⁡^i(w⊤x~i)w⊤Gˉw+ρ∥w∥22.(3)w_\star\in\underset{w \ne0}{\arg\max}\ \frac{\widehat{\operatorname{Var}}_i(w^\top\tilde{x}_i)}{w^\top\bar{G}w + \rho\lVert w \rVert_2^2}. \tag*{(3)}

The numerator rewards variation across the calibrated items, while w⊤Gˉww^\top\bar{G}w penalizes weights that are more sensitive to nuisance changes. The hyperparameter ρ>0\rho> 0 controls the tradeoff between retaining variation and reducing intervention sensitivity. We solve the resulting generalized eigenvalue problem, then orient and normalize the solution to obtain the final aggregation weights ww. The aggregate score is si=w⊤x~is_i = w^\top\tilde{x}_i. Implementation details appear in Appendix B.2.

Theoretical Analysis

We theoretically analyze VoirDire, showing that observational judge agreement alone may not distinguish target quality from shared nuisance factors, whereas paired interventions reveal nuisance-sensitive variation and enable estimation of the calibration matrix.

Setup. We model judge outputs with target-relevant factors qiq_i, nuisance factors cic_i, and noise:

xi(0)=UQqi+UCci+εi(0),xi(a)=UQqi+UC(ci+δi(a))+εi(a).(4)\begin{aligned} x_i^{(0)} &= U_Q q_i + U_C c_i + \varepsilon_i^{(0)},\\ x_i^{(a)} &= U_Q q_i + U_C\left(c_i + \delta_i^{(a)}\right) + \varepsilon_i^{(a)}. \tag*{(4)} \end{aligned}

Algorithm 1 VoirDire: calibration and intervention-aware aggregation

Require: Judge outputs x1,…,xn∈Rpx_1,\ldots,x_n \in\mathbb{R}^p; paired outputs {(xi,xi(a))}i∈Pa\{(x_i,x_i^{(a)})\}_{i\in\mathcal{P}_a} for kk selected intervention families, with ∣Pa∣=m|\mathcal{P}_a|=m and M=kmM=km; validation-selected hyperparameters (α,γ)(\alpha,\gamma).

Ensure: Aggregate scores s1,…,sns_1,\ldots,s_n.

  1. Compute paired score differences for each family aa and item i∈Pai \in\mathcal{P}_a: Δi(a)←xi(a)−xi\Delta_i^{(a)} \leftarrow x_i^{(a)}-x_i.

  2. Estimate the pooled second-moment matrix: Gˉ←1M∑a=1k∑i∈PaΔi(a)Δi(a)⊤\bar{G} \leftarrow\frac{1}{M}\sum_{a=1}^{k}\sum_{i\in\mathcal{P}_a}\Delta_i^{(a)}\Delta_i^{(a)\top}. (1)

  3. Attenuate nuisance-sensitive components for all items: x~i←xi−αGˉxi/λmax⁡(Gˉ)\tilde{x}_i \leftarrow x_i-\alpha\bar{G}x_i/\lambda_{\max}(\bar{G}), i=1,…,ni=1,\ldots,n. (2)

  4. Set ρ←γtr⁡(Gˉ)/p\rho\leftarrow\gamma\operatorname{tr}(\bar{G})/p and solve for intervention-aware weights: w⋆∈arg⁡max⁡w≠0Var⁡i(w⊤x~i)w⊤Gˉw+ρ∥w∥22w^\star\in\arg\max_{w\ne0}\frac{\operatorname{Var}_i(w^\top\tilde{x}_i)}{w^\top\bar{G}w+\rho\lVert w\rVert_2^2}. (3)

  5. Orient and normalize w⋆w^\star to obtain ww (Appendix § 11.§ 2).

  6. return si=w⊤x~is_i=w^\top\tilde{x}_i, i=1,…,ni=1,\ldots,n.

where xi(0)=xi∈Rpx_i^{(0)}=x_i\in\mathbb{R}^p denotes the original judge-output vector, and xi(a)x_i^{(a)} denotes the outputs of the same judges after an intervention from family aa. The matrices UQU_Q and UCU_C map target-relevant and nuisance factors to judge scores, respectively. We first consider an ideal setting in which interventions change nuisance factors by δi(a)\delta_i^{(a)} while leaving qiq_i unchanged.

Why agreement alone is insufficient. Shared variation in judge scores need not reveal whether judges respond to target quality or nuisance attributes; Theorem 1 makes this explicit.

Theorem 1 (Observational non-identifiability; simplified). In the noiseless model x=uq+vcx=uq+vc, with orthonormal u,v∈Rpu,v\in\mathbb{R}^p and (q,c)∼N(0,I2)(q,c)\sim N(0,I_2), the target direction uu is not identifiable from the distribution of xx: rotating uu and vv within their span leaves this distribution unchanged.

Thus, even with unlimited observations, an aggregator cannot determine which direction represents quality in this model without additional information or assumptions.

What paired interventions reveal. Paired interventions address this ambiguity by holding target quality fixed while changing nuisance attributes. Under eq. (4), the target component cancels: Δi(a)=xi(a)−xi(0)=UCδi(a)+ξi(a)\Delta_i^{(a)}=x_i^{(a)}-x_i^{(0)}=U_C\delta_i^{(a)}+\xi_i^{(a)}, where ξi(a)\xi_i^{(a)} is the paired judge noise. If this noise is mean-zero and independent of δi(a)\delta_i^{(a)}, with finite second moments, the pooled population second moment is G=LC+ΩG=L_C+\Omega, where LC=1k∑aE[(UCδi(a))(UCδi(a))⊤]L_C=\frac{1}{k}\sum_a\mathbb{E}[(U_C\delta_i^{(a)})(U_C\delta_i^{(a)})^\top] captures intervention-induced nuisance variation and Ω\Omega is the pooled noise covariance.

Proposition 1 (Calibration-matrix estimation; simplified). Suppose ∥G−LC∥op≤ρerr\lVert G-L_C\rVert_{\mathrm{op}}\leq\rho_{\mathrm{err}} and ∥Gˉ−G∥op≤ϵM\lVert\bar{G}-G\rVert_{\mathrm{op}}\leq\epsilon_M. For nonzero LC,GˉL_C,\bar{G}, let T=LC/λmax⁡(LC)T=L_C/\lambda_{\max}(L_C) and T^=Gˉ/λmax⁡(Gˉ)\hat{T}=\bar{G}/\lambda_{\max}(\bar{G}). Then

∥T^−T∥op≤2(ρerr+ϵM)λmax⁡(LC).\lVert\hat{T}-T\rVert_{\mathrm{op}}\leq\frac{2(\rho_{\mathrm{err}}+\epsilon_M)}{\lambda_{\max}(L_C)}.

Proposition 1 shows that accurate calibration requires estimation and systematic errors to be small relative to the nuisance signal. We further analyze how calibration sample size affects estimation error (Theorem 2) and how imperfect interventions affect nuisance recovery. Full proofs and additional theoretical results appear in Appendix § 14, with supporting synthetic experiments in Appendix § 15.

Experiments

We evaluate VoirDire across continuous-score, binary, and preference LLM-judge aggregation setups. The experiments address the following topics:

  • Improving judge aggregation (Section 5.1): We compare VoirDire with existing aggregation methods across the three task types. § 5.1

DatasetMVAVGISPPCACARE-SVDVoirDire
Yelp1.0161.0891.0281.0540.6940.508
SummEval0.8190.7530.8320.7380.7290.701
Summarize from Feedback1.4341.4531.5851.7641.3921.346
HelpSteer20.9980.9811.0080.9920.9980.951
ASSET31.23833.70131.88528.88827.62926.930
Review-5K2.5812.4832.5852.1011.9571.862

Table 1. Continuous-score aggregation: mean MAE (↓\downarrow).

DatasetMVAVGISPPCACARE-SVDCARE-TensorVoirDire
Chatbot Arena0.5170.5510.5710.5440.5720.5960.646
RewardBench 20.5770.5960.5690.5950.5780.6050.658
LIAR20.6780.6650.6780.6610.6700.6680.686
HelpSteer30.4900.5170.5560.5200.5920.5720.610
CivilComments0.7020.7150.7160.7140.7470.7310.739
MT-Bench (human)0.6000.6130.6000.5990.5940.5810.624

Table 2. Binary/preference aggregation: mean accuracy (↑\uparrow).

  • Ablating score calibration and aggregation (Section 5.2): We examine how score calibration and intervention-aware aggregation individually contribute to performance.

  • Judge-level interpretation (Section 5.3): We examine how intervention sensitivity relates to judge reliability and aggregation weights.

  • Calibration efficiency (Section 5.4): We measure how performance changes with the number of intervention pairs.

  • Targeted nuisance removal (Section 5.5): We test whether VoirDire reduces score shifts under rewrites that trigger known judge biases.

  • Transfer across related nuisances (Section 5.6): We test whether calibration transfers to related nuisance variations not used during fitting.

Datasets and metrics. We evaluate on six continuous-score and six binary/preference datasets. We report mean absolute error (MAE; lower is better) for continuous scoring and accuracy (higher is better) for binary classification and pairwise preference tasks. Dataset details are in Appendix § 11.§ 1.

LLM judges. Continuous-score panels contain 11 open-weight LLM judges and binary/preference panels 20, with model sizes ranging from 0.6B to 14B parameters. Each judge reports on the dataset-native score scale. Following the judge-output convention in CARE (Zhao et al., 2026), judges first produce scalar scores for binary classification and pairwise preference tasks. These scores are then converted to binary labels or preferences using fixed thresholds and evaluated by accuracy. Details are in Appendices § 11.§ 3 and § 11.§ 4.

Interventions. We use LLM assistants to help identify task-specific nuisance attributes and design interventions, following Adila et al. [3]. The interventions vary formatting, verbosity, or candidate presentation while aiming to preserve the evaluation target. We generate modified inputs through deterministic transformations or rewrites by one of the judge models, Llama-3.1-8B-Instruct. Pair generation requires no ground-truth labels; family selection and tuning of (α,γ)(\alpha, \gamma) use the validation set. Full specifications, prompts, and selection details appear in Appendix § 11.

Improving Aggregation with Interventions

Setup. We compare VoirDire with majority voting (MV), score averaging (AVG), PCA aggregation, CARE (Zhao et al., 2026), and Inverse Surprisal Popularity (ISP) [4]. We use CARE-SVD for continuous scoring and both CARE-SVD and CARE-Tensor for binary classification and pairwise preference. All methods use the same judges and evaluation items across five random splits per dataset. For VoirDire, we use m=32m = 32 intervention pairs per family. Protocol details and additional results appear in Appendices § 11 and § 16, respectively.

Results. Tables 1 and 2 show that VoirDire improves on existing aggregation methods across task types. The largest MAE reduction is on Yelp, at 26.8% relative to the strongest baseline, and the largest accuracy gains are on RewardBench 2 (5.3 pp) and Chatbot Arena (5.1 pp). These improvements suggest that intervention-based calibration can enhance judge aggregation compared with existing observational methods.

Ablating Score Calibration and Aggregation

VoirDire uses interventions for score calibration and intervention-aware aggregation. We examine the contribution of each step.

Setup. Following the setup of the previous subsection, we compare PCA and intervention-aware aggregation on raw and calibrated scores. The PCA baseline corresponds to α=0\alpha= 0 and ρ→∞\rho\to\infty in eq. (3). For intervention-aware aggregation on raw scores, we set α=0\alpha= 0 and re-tune ρ\rho on validation. Combining score calibration and intervention-aware aggregation gives VoirDire.

Results. Figure 3 shows that both steps contribute, with their relative benefits varying across datasets. On continuous-score tasks, score calibration provides substantial gains on datasets such as Yelp and ASSET. On binary/preference tasks, intervention-aware aggregation alone improves over PCA on all six datasets, while combining both steps provides further gains on datasets such as HelpSteer3. Additional ablations on other aggregators appear in Appendix § 16.§ 4.

Figure 3. Contributions of calibration and aggregation relative to raw PCA: relative MAE reduction (%) on continuous tasks and accuracy gain (percentage points) on binary/preference tasks. Higher is better. †: α=0\alpha= 0, so calibration leaves scores unchanged.

How Interventions Diagnose and Reweight Judges

Beyond improving aggregation, controlled interventions can help diagnose individual judges by revealing their sensitivity to nuisance attributes. We examine whether this sensitivity relates to judge reliability and how it influences the aggregation weights learned by VoirDire.

Setup. For each judge in the panels of § 5.1, we measure rewrite sensitivity as the root mean square score shift across calibration pairs, Gˉjj\sqrt{\bar{G}_{jj}}, normalized by the standard deviation of its original scores to account for differences in score scale. We measure reliability as the Spearman correlation between the judge’s original scores and ground truth, capturing how well its judgments align with the evaluation target. Comparing these metrics tests whether nuisance sensitivity, measured without ground-truth labels, is associated with judge reliability. We also examine the aggregation weights wjw_j from eq. (3) to assess how VoirDire uses this sensitivity information.

Results. Figure 4 (a), (Figure 4b) show that judges with greater rewrite sensitivity tend to be less reliable, although the relationship varies across datasets. Panels (c) and (d) show that VoirDire generally assigns lower weights to more sensitive judges.

Figure 4. Rewrite sensitivity, reliability, and aggregation weights. (a, b) One point per judge and panel; legends report the within-panel Spearman correlation between sensitivity and reliability. (c, d) Weights divided by each panel’s largest absolute weight, with judges ranked by sensitivity (1: most sensitive; blue: positive, red: negative). Empty ranks indicate fewer judges.

Ablation on the Number of Intervention Pairs

We examine how the number of intervention pairs affects the performance of VoirDire.

Setup. We vary the number of calibration pairs per family m∈{4,8,16,32,64}m \in\{4, 8, 16, 32, 64\}, keeping the intervention families of § 5.1 and re-selecting (α,ρ)(\alpha, \rho) on validation at every mm. We report improvements over each dataset’s strongest baseline, averaged separately across the six continuous-score datasets (relative MAE reduction) and the six binary/preference datasets.

Results. Figure 5 shows that VoirDire improves aggregation with a small calibration budget. Eight pairs per family recover most of the gain on continuous-score tasks, and gains on both task types level off between 32 and 64 pairs. With 32 pairs per family, VoirDire achieves an average MAE reduction of 7.4% and an accuracy gain of 2.2 percentage points over each dataset’s strongest baseline. Per-dataset results appear in Appendix § 11.7.

Figure 5. Mean gain over the strongest baseline. Bands: one SE over five splits; dotted line: m=32m=32.

Targeted Nuisance Removal

We further evaluate whether VoirDire reduces sensitivity to known nuisance changes in a controlled setting, directly testing whether it achieves its intended effect.

Setup. We use the judgment-bias benchmark of Chen et al. [12], which provides original–modified answer pairs with GPT-4 rewrites targeting rich formatting (beauty) or added references (authority), along with 1–10 judge scores. These rewrites can also rephrase or expand the original answers. We measure sensitivity as the mean absolute difference between aggregate scores on original and rewritten answers. Lower values indicate greater robustness to the targeted nuisance changes.

Results. VoirDire achieves the lowest sensitivity to both biases (Table 3), reducing mean absolute score shifts by approximately 27.3% for beauty and 18.2% for authority relative to CARE-SVD, the strongest baseline. Additional experimental details and results appear in Appendices B.6 and E.3.

TypeExample
OriginalNight and day occur due to the Earth’s rotation on its axis.
BeautyNight and day occur due to the Earth’s rotation on its axis. When facing the Sun ☼, it experiences daylight. . . .
Authority. . . This rotation happens over a period of 24 hours, creating our cycle of day and night (National Geographic, 2019).
MethodBeautyAuthority
MV0.9190.824
AVG0.5060.325
ISP0.8910.733
PCA1.2830.876
CARE-SVD0.4180.286
VoirDire0.3040.234

Table 3. Examples of targeted nuisance rewrites (left) and mean absolute score shifts (right, ↓).

Transfer Across Related Nuisances

Identifying every nuisance variation can be costly. However, related rewrites may affect judge scores in similar ways, so we expect VoirDire to generalize beyond the specific interventions used for calibration. We test this in a controlled setting.

Setup. Following Section 5.5, we calibrate on three beauty variants (numbered structure, minimalist panels, and topic-specific icons) and three authority variants (expert attribution, a source header, and a fabricated footnote), then evaluate on the original beauty and authority rewrites. We report sensitivity reduction relative to PCA, normalized so that calibration on the original beauty or authority rewrites achieves 100%. Details appear in Appendix B.8.

Results. Table 4 shows that calibration retains much of its benefit on unseen rewrites. Notably, beauty variants transfer strongly even to authority, while authority variants also provide some benefit on beauty. This suggests that distinct nuisance attributes may affect overlapping directions in judge-output space, enabling transfer across them.

CalibrationEvaluation
ConceptVariantBeautyAuthority
BeautyNumbered77.1%91.3%
Panels81.1%102.5%
Icons72.0%100.3%
AuthorityExpert60.4%100.5%
Source header74.6%99.2%
Footnote63.9%98.0%

Table 4. Percentage of target-family gain retained (↑).

Conclusion

We introduced VoirDire, which uses controlled interventions to mitigate shared nuisance sensitivity in LLM-judge aggregation. Across 12 datasets, VoirDire improves aggregation performance and reduces sensitivity to targeted nuisance changes. Calibration also transfers to related, unseen variations, suggesting that exact rewrite matches are unnecessary. These findings highlight the potential of counterfactual interventions for more reliable LLM-based evaluation.

Limitations and future work. VoirDire depends on the quality of its interventions: they should capture relevant nuisance variation while preserving the evaluation target, which is difficult to guarantee. Discovering effective interventions also adds cost, and calibration requires additional inference to generate rewrites and score modified inputs, although deterministic transformations avoid the rewrite-generation cost. More broadly, paired interventions could extend beyond aggregation: agents could actively search for interventions that expose previously unknown judge biases, and the same paired signals could train judges and reward models to be invariant to surface features such as length and formatting, reducing reward hacking in downstream optimization.

ETHICS STATEMENT

The main risk is over-trust in calibrated automatic evaluation. VoirDire attenuates nuisance sensitivities exposed by the chosen interventions, but it does not guarantee unbiased judgments or remove all evaluation artifacts. For private or high-stakes evaluations, rewrite generation and judge calls should be run with appropriate data-protection safeguards, since the method sends original and rewritten inputs to LLMs.

AI USE STATEMENT

We used AI coding and writing assistants (Claude Code, Codex, Fable, and ChatGPT Pro) throughout this project. They assisted with implementing and running experimental, evaluation, and plotting code; proposing nuisance attributes and intervention designs; developing and checking the theoretical analysis; and drafting and editing the manuscript. The authors directed all AI-assisted work, reviewed its outputs, and take full responsibility for the content.

Appendix Overview

This appendix starts with a glossary of notation in Appendix A. It then provides experimental details in Appendix B, including datasets, experiment protocols, compute resources, judge prompts, and intervention specifications. Next, Appendix C contains proofs and additional theoretical results. We then present synthetic experiments that study calibration behavior in controlled settings (Appendix D). Finally, Appendix E reports additional experimental results.

Glossary

The notation used throughout the paper is summarized in table 5.

SymbolDefinition
i,j,ai,j,aItem, judge, and intervention-family indices.
n,p,kn,p,kNumbers of original items, judges, and selected families.
m,ma,Mm,m_a,MPairs per family and total budget: M=kmM=km, or M=∑amaM=\sum_a m_a for unequal budgets.
α,γ,ρ\alpha,\gamma,\rhoAttenuation strength, regularization multiplier, and ridge parameter ρ=γtr⁡(Gˉ)/p\rho=\gamma\operatorname{tr}(\bar{G})/p.
uiu_iJudge input for item ii.
xi,xi(0)x_i,x_i^{(0)}Original judge-output vector; xi(0)=xi∈Rpx_i^{(0)}=x_i\in\mathbb{R}^p.
XXMatrix of original judge outputs.
xi(a)x_i^{(a)}Judge-output vector after intervention aa.
Pa\mathcal{P}_aCalibration item indices for family aa.
Δi(a),zi(a)\Delta_i^{(a)},z_i^{(a)}Paired score difference xi(a)−xi(0)x_i^{(a)}-x_i^{(0)}.
Gˉ\bar{G}Empirical second moment of pooled paired differences (eq. (1)).
G,LCG,L_CPopulation second moments of paired differences and intervention-induced nuisance variation, respectively.
T,T^T,\widehat{T}Ideal and empirical normalized operators: T=LC/λmax⁡(LC)T=L_C/\lambda_{\max}(L_C), T^=Gˉ/λmax⁡(Gˉ)\widehat{T}=\bar{G}/\lambda_{\max}(\bar{G}).
η\etaOperator-estimation error bound: ∥T^−T∥op≤η\lVert\widehat{T}-T\rVert_{\mathrm{op}}\leq\eta.
x~i\widetilde{x}_iCalibrated output (I−αT^)xi(I-\alpha\widehat{T})x_i.
Σ^α\widehat{\Sigma}_\alphaEmpirical covariance of calibrated outputs.
wwOriented and normalized aggregation weights (eq. (3)).
sis_iAggregate score w⊤x~iw^\top\widetilde{x}_i.
qi,ci,δi(a)q_i,c_i,\delta_i^{(a)}Target factors, nuisance factors, and intervention-induced changes in nuisance factors.
UQ,UCU_Q,U_CLoading matrices for target and nuisance factors.
dCd_CNuisance-subspace dimension.
U^C,PUC\widehat{U}_C,P_{U_C}Leading dCd_C eigenvectors of Gˉ\bar{G} and orthogonal projector onto range⁡(UC)\operatorname{range}(U_C).
εi(a),ξi(a)\varepsilon_i^{(a)},\xi_i^{(a)}Judge noise and paired noise ξi(a)=εi(a)−εi(0)\xi_i^{(a)}=\varepsilon_i^{(a)}-\varepsilon_i^{(0)}.
MaM_aFamily-specific second moment E[zi(a)zi(a)⊤]\mathbb{E}[z_i^{(a)}z_i^{(a)\top}].
Aa,ΩaA_a,\Omega_aNuisance-shift second moment E[δi(a)δi(a)⊤]\mathbb{E}[\delta_i^{(a)}\delta_i^{(a)\top}] and paired-noise covariance.
λint,ρpair\lambda_{\mathrm{int}},\rho_{\mathrm{pair}}Intervention-induced nuisance strength and paired-noise scale.
ρerr,ϵM\rho_{\mathrm{err}},\epsilon_MBounds on ∥G−LC∥op\lVert G-L_C\rVert_{\mathrm{op}} and ∥Gˉ−G∥op\lVert\bar{G}-G\rVert_{\mathrm{op}}, respectively.

Table 5. Glossary of variables, symbols, and abbreviations.

Experimental Details

This section provides the experimental details for the main results, including datasets, judge-output protocols, judge prompts, intervention specifications, and implementation details.

Datasets

We evaluate six continuous-score panels (Yelp, SummEval coherence, Summarize from Feedback, HelpSteer2, ASSET, Review-5K) and six binary/preference panels (Chatbot Arena, RewardBench 2, HelpSteer3, MT-Bench with human labels, LIAR2, CivilComments). In table 1 the row labeled SummEval is the coherence aspect.

For continuous-score tasks, the target is the human-provided score and we report MAE. For binary/preference tasks, the target is the human-provided label or preference and we report accuracy.

Review-5K (Weng et al., 2025). Review-5K contains paper texts, reviewing guidelines, review records, and numeric reviewer ratings. We use the paper text and reviewing guidelines as the judge input; the reference review text is not shown to the judges. The target field is avg_rate, the average available reviewer rating on the 1–10 review scale. The dataset is available as WestlakeNLP/Review-5K.

Yelp (Zhang et al., 2015). Yelp consists of user-written business reviews paired with star ratings. We use a 5,000-example subset and treat the task as continuous rating prediction: given the review text, the goal is to recover the associated 1–5 star rating. The source dataset is available as Yelp/yelp_review_full.

ASSET [7]. ASSET is a sentence-simplification rating dataset. Each row contains a source sentence, a candidate simplification, an evaluation aspect, and a human rating. We use the human_rating field as the continuous target on the 0–100 scale. We use 4,500 rating rows. The dataset is available as facebook/asset.

HelpSteer2 [41]. HelpSteer2 contains prompts, model responses, and human ratings of five response attributes. We use the prompt and candidate response as the judge input and the helpfulness rating as the continuous target on the 0–4 scale, stored in the score field. We use the released validation split. The dataset is available as nvidia/HelpSteer2.

Summarize from Feedback [38]. Summarize from Feedback contains source posts, candidate summaries, and human assessments of summary quality. We use its axis subset, with the source post and candidate summary as the judge input. The target is the overall quality rating on the 1–7 scale, stored as overall_score from summary.axes.overall. We pool the released validation and test splits before applying our evaluation protocol. The dataset is available as openai/summarize_from_feedback.

SummEval [18]. SummEval contains model-generated summaries of CNN/Daily-Mail news articles with human ratings of coherence, consistency, fluency, and relevance. We use the coherence panel: 1,600 summaries from 16 systems for 100 articles. The judge input is the article and candidate summary; the target is the mean expert coherence rating on the 1–5 scale, stored in the label field. The dataset is available from Yale-LILY/SummEval.

Chatbot Arena (Zheng et al., 2023). Chatbot Arena contains pairwise comparisons between two model responses to the same user prompt. We use it as a preference task: the target field is gold_label_binary, which encodes the human preference over candidates A and B. We use 5,000 sampled comparisons, excluding ties. The dataset is available as lmsys/chatbot_arena_conversations.

CivilComments [11]. CivilComments consists of online comments annotated for toxicity. We use a 5,000-example subset, sampled equally from the two classes in the test split, as a binary toxicity-classification task, where the input is the comment text and the target indicates whether the comment is toxic. The binary target is obtained from the dataset’s continuous toxicity score, with scores of at least 0.5 labeled toxic (1) and the rest non-toxic (0). The source dataset is available as google/civil_comments.

LIAR2 (Xu & Kechadi, 2024). LIAR2 contains short statements assessed by professional fact-checkers at PolitiFact. We use the 2,296 statements in the released test split, with only the statement text as the judge input. We collapse the six veracity classes into a binary target: pants-on-fire, false, and barely-true map to 0; half-true, mostly-true, and true map to 1. The target is stored in the label field. The dataset is available as chengxuphd/liar2.

MT-Bench (Zheng et al., 2023). MT-Bench provides human comparisons of model responses to questions spanning multiple task categories. We use the 3,355-comparison human panel from allenai/preference-test-sets, with the question and two candidate responses as the judge input. The target identifies the human-preferred response, adjusted for the displayed candidate order. The original human judgments are available as lmsys/mt_bench_human_judgments.

HelpSteer3 (Wang et al., 2025). HelpSteer3 contains human preferences between model responses across code, general, multilingual, and STEM tasks. We sample 625 training examples per domain and remove tied preferences. The judge input is the last user message and the two candidate responses. The target is the direction of the overall_preference rating, stored as gold_label_binary. We read this reference field from the judge files to preserve row alignment. The dataset is available as nvidia/HelpSteer3.

RewardBench 2 [29]. RewardBench 2 tests response selection across factuality, instruction following, mathematics, safety, focus, and ties. We construct a pairwise panel from its 1,865 prompts by taking the first chosen and first rejected completion for each prompt. The judge input is the prompt and these two responses; the target identifies the chosen response in the displayed order. We align intervention rows by source-panel position because pair identifiers repeat within comparison groups. The dataset is available as allenai/reward-bench-2.

Evaluation protocol

Data splits and hyperparameter tuning. Following Zhao et al. (2026), we use five random row-level splits per dataset, reserving 10% for validation on continuous tasks and 15% on binary/preference tasks. We select intervention families and hyperparameters using the validation set. For CARE, we follow the original paper’s tuning procedure: CARE-SVD tunes its regularization, while CARE-Tensor tunes its regularization, rank, and posterior cutoff.

Candidate-pool construction. We used Codex and Fable to inspect dataset examples and propose potential nuisance attributes and corresponding intervention instructions. Pool construction was open-ended, with no prespecified stopping rule for expansion. The pool was refined using experimental feedback, including validation metrics. Given a family specification, pair generation and preservation checks use no quality labels (Appendix § 11.§ 5).

Family selection. We select families by forward greedy search using mean validation performance across seeds. We first select the best-performing single family. At each subsequent step, we evaluate adding each remaining family, retuning (α,γ)(\alpha, \gamma) for every candidate set, and accept the addition with the largest validation improvement. The search stops when no addition improves validation performance or 15 families have been selected.

Estimator settings. We tune α∈{0,0.1,0.25,0.5,0.75,1}\alpha\in\{0, 0.1, 0.25, 0.5, 0.75, 1\} and γ∈{0.25,1,4,16,64}\gamma\in\{0.25, 1, 4, 16, 64\}, with ρ=γtr⁡(Gˉ)/p\rho= \gamma\operatorname{tr}(\bar{G})/p. We select one configuration per dataset by minimizing mean validation MAE for continuous tasks or maximizing mean validation accuracy for binary/preference tasks across the five splits. Table 6 reports the selected configurations, family counts, and evaluation variability.

DatasetBest baselineVoirDireSelected / candidate families(α,γ)(\alpha, \gamma)
Continuous scoring: MAE (↓\downarrow)
Yelp0.6940 ± 0.00400.5080 ± 0.01564 / 30(1, 64)
SummEval0.7293 ± 0.00480.7012 ± 0.00524 / 24(0, 1)
Summarize from Feedback1.3922 ± 0.01361.3461 ± 0.00364 / 41(0, 0.25)
HelpSteer20.9814 ± 0.00570.9514 ± 0.01263 / 33(0.75, 0.25)
ASSET27.6290 ± 0.156026.9303 ± 0.34215 / 53(0.5, 0.25)
Review-5K1.9570 ± 0.01801.8624 ± 0.03524 / 25(0.25, 64)
Binary/preference: accuracy (↑\uparrow)
Chatbot Arena0.5959 ± 0.01620.6465 ± 0.01055 / 43(0.5, 0.25)
RewardBench 20.6052 ± 0.01160.6579 ± 0.00654 / 37(1, 4)
LIAR20.6780 ± 0.00390.6857 ± 0.00472 / 12(0.1, 0.25)
HelpSteer30.5921 ± 0.02720.6101 ± 0.01466 / 39(0.5, 0.25)
CivilComments0.7473 ± 0.00310.7387 ± 0.00332 / 22(0, 0.25)
MT-Bench (human)0.6128 ± 0.00360.6242 ± 0.00654 / 38(0.1, 1)

Table 6. Selected configurations and mean ± SD. Best baseline is the strongest mean among MV, AVG, ISP, PCA, and the available CARE variants.

Weight orientation and normalization. The generalized eigenvector is defined only up to sign and scale. We orient the weights to have a positive sum and rescale them to sum to one, so the aggregate stays on the judges’ score scale.

Computational resources and cost. We ran experiments on NVIDIA A100 40GB and L40 GPUs. Generating the original judge outputs took up to approximately three hours per dataset. Judge scoring and rewriting use GPUs, while estimator fitting and family selection run on CPUs. For kk selected families and pp judges, calibration uses 32kp32kp rewritten scores. Total cost also includes generating and scoring candidate interventions that are not selected.

LLM Judge Models

Following Zhao et al. (2026), we use fixed pools of open-weight LLM judges and filter their outputs for valid scores. For free-form scalar outputs, we retain judges with valid scores on more than 70% of items, exclude items with missing, unparsable, or out-of-range scores from any retained judge, and remove judges with constant scores. The resulting panels contain 11 judges for continuous scoring and 20 for binary classification and pairwise preference.

Continuous-score judges. The continuous-score pool contains Qwen3-{0.6,1.7,4,8}B [6], Llama-3.1-8B-Instruct and Llama-3.2-{1,3}B [30], Gemma-3-{1,4}B-IT [17], Phi-4-mini-instruct [31], and Mistral-7B-Instruct-v0.3 [32]. Judges are prompted to return a scalar score on the dataset-native range: 1–10 for Review-5K, 1–5 for Yelp, 1–5 for SummEval coherence, 1–7 for Summarize from Feedback, 0–4 for HelpSteer2, and 0–100 for ASSET.

Binary and preference judges. The binary/preference pool is the same 20 judges on every panel: Qwen2.5 Instruct (1.5B, 3B, 7B, and 14B) [5] and Qwen3-{0.6,1.7,4,8}B [6], Llama-3.1-8B-Instruct, Llama-3.2-{1,3}B, and Llama-3.2-{1,3}B-Instruct [30], Mistral-7B-Instruct-v0.3 [32], Phi-4-mini-instruct [31], Yi-1.5-{6,9}B-Chat (01.AI, 2024), Gemma-2-9B-IT [16], and Gemma-3-{1,4}B-IT [17]. We map CivilComments and LIAR2 outputs to binary labels and pairwise outputs to candidate preferences for accuracy evaluation. The judge pool remains fixed throughout intervention-family selection.

LLM Judge Prompt Templates

Prompting and score parsing. We use task-specific prompts and score ranges, applying the same prompt format and parsing rules to original and modified inputs. Ratings are extracted from the requested verdict field; pairwise ratings are converted to signed preferences and evaluated using a fixed threshold. Missing, unparsable, or out-of-range ratings are discarded. All judges receive raw completion prompts without chat templates. The templates below use placeholders in braces for item-specific content.

Review-5K paper scoring. The {guidelines} field holds the venue’s reviewer guidelines and {paper} the submission text.

You are scoring the overall quality of a research paper submission. Use the reviewing
guidelines below to assess methodological soundness, clarity, contribution, and
overall merit. Assign one of the standard conference rating levels (integer in [1,
10]) using this rubric:

1 - Strong Reject: fundamental flaws; clearly unsuitable for publication.
3 - Reject, not good enough: major weaknesses outweigh contributions.
5 - Marginally below the acceptance threshold: borderline but leaning negative.
6 - Marginally above the acceptance threshold: borderline but leaning positive;
    improvements still desirable.
8 - Accept, good paper: solid contribution with meaningful advances and only minor
    issues.

10 - Strong Accept, should be highlighted at the conference: outstanding paper with exceptional clarity and impact.

If you believe an intermediate score (2, 4, 7, 9) better reflects the work, you may use it, positioning it between the adjacent rubric descriptions above.

Reviewing guidelines: {guidelines}

Respond strictly in this format:

Feedback::: Total rating: X

Where X is an integer between 1 and 10 inclusive. Output nothing else.

Paper content: {paper}

Feedback::: Total rating:

Yelp review scoring.

You are evaluating the overall sentiment of a Yelp restaurant review. Read the entire review, consider the reviewer's tone, described experiences, and final impression, then assign a single star rating from 1 to 5 according to this rubric:

1 - Strongly negative; serious problems or clear dissatisfaction. 2 - Mostly negative; several complaints outweigh positives. 3 - Mixed or neutral; balanced feedback with no strong lean. 4 - Mostly positive; satisfied with only minor issues. 5 - Highly positive; enthusiastic praise and strong recommendation.

Base the rating on the reviewer's intent, not your personal opinion, and ignore formatting quirks.

Respond exactly in this format:

Feedback::: Total rating: X

Where X is an integer from 1 to 5 and nothing else.

Review: {text}

Feedback::: Total rating:

SummEval coherence scoring.

You are rating how coherent a candidate summary is. Read the article and the summary, judge whether the summary is well structured and its sentences follow one another in a sensible order, rather than reading as a disconnected list of facts, and output a single rating in [1,5][1, 5]:

1 - Disjointed; sentences do not connect and the order is arbitrary. 2 - Mostly disconnected, with abrupt transitions between sentences. 3 - Partly organised; some sentences follow naturally and others do not. 4 - Well organised, with only minor lapses in flow. 5 - Fully coherent; the summary reads as a single connected piece of text.

Judge organisation and flow only. Do not reward a summary for covering more of the article.

Respond strictly in this format:

Feedback::: Total rating: X

Where X is an integer 1-5. Output nothing else.

Article: {article}

Candidate summary: {summary}

Feedback::: Total rating:

Summarize from Feedback scoring.

You are an expert editor judging how faithfully a candidate summary captures the key information in the original Reddit post. Read the post first (central events, reasons, outcomes). Then read the summary and assess faithfulness and coverage.

Score substance over surface:

  • Prioritise factual faithfulness and coverage of core facts (who/what/when/where/why/how) and outcomes.

  • Prefer precise, specific content grounded in the post; do not reward length, flourish, or formatting.

  • Subtract for invented facts, contradictions, or missing main points; minor wording differences should rarely affect the score.

  • A concise but accurate and comprehensive summary can score high; a longer but vague or extraneous one can score low.

Use the absolute 1-7 scale (not relative to other items):

1 - Completely inaccurate or misleading; major facts are wrong or missing. 2 - Very inaccurate; severe omissions or contradictions; little usable content. 3 - Poor; captures a few ideas but misses several important points or introduces errors. 4 - Fair; mostly correct but notable gaps, extraneous details, or minor hallucinations. 5 - Good; covers main ideas with small omissions or wording issues. 6 - Very good; accurate and fluent with only trivial gaps. 7 - Excellent; fully faithful, comprehensive, and well-written.

Guidance:

  • Reserve 7 for both faithful and comprehensive summaries; use 1-2 for clearly poor ones.

  • Minor style differences are acceptable if the content remains faithful.

Respond exactly in this format:

Feedback::: Total rating: X

Where X is exactly one of: 1, 2, 3, 4, 5, 6, 7. Output nothing else.

Post: {post}

Summary: {summary}

Feedback::: Total rating:

HelpSteer2 response scoring.

You are scoring how well the system_answer addresses the user_question. Return a single float in [0.0, 4.0] (decimals allowed). Use the full range: 0.0 = Unhelpful or harmful; refusal; mostly wrong or off-topic. 1.0 = Barely helpful; major gaps or serious inaccuracies. 2.0 = Partially helpful; mixed quality with notable omissions or minor errors. 3.0 = Helpful and mostly correct; small omissions or minor issues only. 4.0 = Fully correct, helpful, and clear.

Rules:

  • Prioritise factual accuracy, relevance, and completeness; do not reward verbosity.

  • Penalize hallucinations, unsafe advice, or refusal when an answer is expected.

  • If your internal judgment falls outside [0, 4], output the nearest bound instead.

Respond exactly in this format and nothing else:

Feedback::: Total rating: X

Question: {question} Answer: {answer}

Feedback::: Total rating:

ASSET simplification scoring. The {aspect} field is the row’s rated aspect (meaning, fluency, or simplicity).

You are rating the quality of a simplified sentence for a specific aspect of text simplification. Examine both sentences carefully before assigning a single integer score in [0, 100]. Use the whole range; do not default to 50 or round to multiples of 5 unless warranted.

Aspect to evaluate: {aspect}

Interpret the aspect as follows (case-insensitive):

  • meaning / meaning preservation -- Judge how faithfully the simplification preserves the original meaning (facts, relations, intent). Ignore simplicity/fluency except where they alter meaning.

  • fluency / grammaticality -- Judge grammatical correctness and naturalness of the simplification. Ignore meaning differences and degree of simplification; do not reward added content.

  • simplicity / readability -- Judge how much easier to read the simplification is compared with the original (shorter sentences, simpler vocabulary/structure). Minor meaning loss may be acceptable, but removing essential information or making text ambiguous should not receive top scores.

  • other -- If the aspect is unrecognised, judge overall simplification quality, prioritising meaning preservation, then fluency, then simplicity.

Anchor guidance (choose any integer within each band): 0–20 Unusable for the aspect. 21–40 Poor for the aspect. 41–60 Mixed: noticeable problems but some positives. 61–80 Good: solid performance with minor issues. 81–95 Very good: strong performance with only tiny flaws. 96–100 Excellent for the aspect.

General rules:

  • Focus strictly on the stated aspect when scoring.

  • Do not reward hallucinations or added facts not in the original.

  • Non-round integers are allowed (e.g., 73, 88) if they best reflect quality.

Respond strictly in this format:

Feedback::: Total rating: X

Where X is an integer in [0, 100]. Output nothing else.

Original sentence: {original}

Candidate simplification: {simplification}

Feedback::: Total rating:

CivilComments toxicity scoring. The 0–9 score is thresholded at 4.5 to obtain the binary toxicity decision.

You are a toxicity rater. Given a single user comment, assign a discrete toxicity score on a 0–9 scale. Higher numbers indicate more toxic language.

Return exactly one integer using this rubric:

0 : Clearly non-toxic or benign language. 1 : Mostly neutral with only faint negativity. 2 : Slightly negative or dismissive language. 3 : Mild insults or disrespectful tone. 4 : Noticeable harassment, repeated negativity, or coarse language. 5 : Borderline toxic; sustained insults or hostility. 6 : Clearly toxic with direct insults or slurs. 7 : Highly toxic; explicit harassment or hateful language. 8 : Severe toxicity including threats or strong hate speech. 9 : Extremely toxic content with explicit threats or violent hate speech.

Guidelines:

  • Focus on the literal wording of the comment.

  • Penalise hate, harassment, threats, or extreme vulgarity.

  • If the comment mixes toxic and mitigating language, choose the score that reflects the overall impact.

Respond strictly in this format:

Feedback::: Total rating: X

Where X is an integer between 0 and 9 (inclusive). Output nothing else.

Comment: {text}

Feedback::: Total rating:

LIAR2 veracity scoring. The 0–9 score is thresholded at 4.5 to obtain the binary truthfulness decision.

You are a fact-checking rater. Given a single political claim, assign a discrete accuracy score on a 0–9 scale. Higher numbers indicate a claim that is more accurate.

Return exactly one integer using this rubric: 0 : Fabricated; no factual basis at all.

1 : Clearly false and contradicted by available evidence.
2 : False; the central assertion is not supported.
3 : Mostly false; a kernel of truth badly distorted.
4 : Borderline false; the overall impression misleads.
5 : Mixed; as much misleading as accurate.
6 : Half true; a real basis but material omissions.
7 : Mostly true; minor details overstated.
8 : True with only trivial imprecision.
9 : Entirely true and well supported.

Guidelines:
- Judge the factual accuracy of the claim itself, not its tone or the speaker.
- Rhetoric, framing and political stance are not evidence.
- If a claim mixes accurate and inaccurate elements, score the overall impression it
  creates.

Respond strictly in this format:

Feedback:::
Total rating: X

Where X is one of: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9. Output nothing else.

Statement: \{text\}

Feedback:::
Total rating:

Pairwise preference scoring. Ratings below 3 favor response A, while ratings above 3 favor response B. HelpSteer3, RewardBench 2, and MT-Bench use the first template, which allows a tie at 3. Chatbot Arena uses the second template, which excludes ties.

You will compare two assistant responses (A and B) to the same user question. Judge
which response is more helpful overall, balancing factual accuracy, completeness,
clarity, safety, and tone.

Return a single integer chosen from \{0, 1, 2, 3, 4, 5, 6\} using this rubric:
0 : Response A is far superior to Response B
1 : Response A is clearly better than Response B
2 : Response A is slightly better than Response B
3 : Responses are essentially tied (equally good or bad)
4 : Response B is slightly better than Response A
5 : Response B is clearly better than Response A
6 : Response B is far superior to Response A

Guidelines:
- Read the user question carefully to understand the user's goal.
- Evaluate each answer independently for correctness, helpfulness, and safety.
- If Response A is better overall, you must output one of \{0, 1, 2\}. If Response B is
  better overall, output one of \{4, 5, 6\}. If they are truly tied, output 3. Do not
  output any other number.
- Compare the answers and decide how strongly one is better using the rubric above.
- Penalize hallucinations, unsafe advice, or refusal when a substantive answer is
  expected.

Respond strictly in this format:

Feedback:::
Total rating: X

Where X is one of: 0, 1, 2, 3, 4, 5, 6. Output nothing else.

Question: \{question\}
Response A: {answer_a}

Response B: {answer_b}

Feedback:::
Total rating:

You will compare two assistant responses (A and B) for the same user question. Judge
which response better helps the user overall, considering factual accuracy,
completeness, clarity, safety, and tone.

Return a single integer chosen from {0, 1, 2, 4, 5, 6} using this rubric:
0 : Response A is far superior to Response B
1 : Response A is clearly better than Response B
2 : Response A is slightly better than Response B
4 : Response B is slightly better than Response A
5 : Response B is clearly better than Response A
6 : Response B is far superior to Response A

Important (must follow all points below or the answer is invalid):
- Ties are NOT allowed. Never output 3 or any other number outside the set {0, 1, 2, 4,
  5, 6}.
- If Response A is better overall, you MUST output one of {0, 1, 2}. Never output 4, 5,
  or 6 when Response A wins.
- If Response B is better overall, you MUST output one of {4, 5, 6}. Never output 0, 1,
  or 2 when Response B wins.
- If the responses are very close, choose 2 (if A is slightly better) or 4 (if B is
  slightly better).
- Output only the integer--no explanations, words, or additional text.

Remember: Response A better -> 0/1/2. Response B better -> 4/5/6. Outputs outside these
sets will be rejected.

Respond strictly in this format (no extra text):

Feedback:::
Total rating: X

Where X is one of: 0, 1, 2, 4, 5, 6.

Question: {question}

Response A: {answer_a}

Response B: {answer_b}

Feedback:::
Total rating:

Intervention Specifications and Preservation Checks

Intervention specifications. Each intervention family specifies (i) the nuisance attribute and input span to modify, (ii) the rule used to perform the modification, and (iii) the preservation constraints used to accept the resulting pair. Modification rules are either deterministic transformations or family-specific instructions executed by an LLM; in our experiments, we use Llama-3.1-8B-Instruct.

Rewrite prompts. The rewrite model receives the input text, a family-specific instruction, and preservation constraints. Word-overlap and length constraints are set according to the intervention family and input length. The model returns only the rewritten text. The following example shows the Yelp concision prompt.

You are rewriting text for a controlled intervention study.

Dataset: yelp_with_scores

Task type: continuous

Goal:

Express the same content more briefly. Remove redundancy and filler only. Keep every fact, number, name and claim. Do not add anything new.

Review:

I really like this place a lot better now that they've moved from the Borgata. It's a nice open floor plan and I love the outside patio and bar area. My pizza was great and I'm pleased to see their nice selection of Italian wines.

Hard constraints:

  • Preserve the same meaning, main claims, recommendation, sentiment, safety posture, and overall quality judgment.

  • Do not add, remove, contradict, or strengthen substantive content.

  • Keep every number, date, name, and URL exactly as in the original.

  • Keep every negation as in the original.

  • Wording: reuse the original's own content words, so that at least 55 percent of them still appear in the rewrite.

  • Length: the rewrite must be between 127 and 212 characters; the original is 230.

Return only the rewritten text.

Preservation checks. For deterministic transformations, checks verify the specified structural properties, such as preservation of text, letter sequences, or lines. Position swaps must exchange candidates verbatim; context and prompt modifications must leave the judged text unchanged.

LLM rewrites must preserve all numbers and the negation count, introduce no new URLs, and satisfy the content-word Jaccard similarity and length-ratio bounds in table 7. Families without a specified length range use [0.70, 1.60]. These checks filter unintended changes but do not guarantee preservation of the evaluation target.

Intervention examples. Table 8 lists the four families selected for Yelp and the position-swap intervention used on preference datasets. The examples below show their application to a Yelp review and a Chatbot Arena comparison. All pairs passed the preservation checks. The Yelp example is the shortest review shared by the four selected families’ calibration pools, chosen without inspecting judge scores.

FamilyKindOperationPreservation check
ConcisionLLM“Express the same content more briefly. Remove redundancy and filler only. Keep every fact, number, name and claim. Do not add anything new.”Jaccard ≥ 0.55, length [0.55, 0.92]
Prose formLLM“Convert any headings, bullet points, numbering or emphasis into continuous prose, keeping the sentences in their original order and wording. Do not summarize, omit or add anything.”Jaccard ≥ 0.75, length [0.85, 1.20]
Markdown quotedeterministicPrefix every line of the exact text with “> ”.letter sequence equal
XML framedeterministicWrap the exact text in <document> tags.exact text preserved
Position swapdeterministicExchange the two candidates verbatim; the prompt-order score is negated at fit time.exact candidate exchange

Table 8. The four families selected for Yelp and the position-swap family of the preference panels.

Original review:

I really like this place a lot better now that they've moved from the Borgata.  It's a
     nice open floor plan and I love the outside patio and bar area.  My pizza was
     great and I'm pleased to see their nice selection of Italian wines.

Concision (LLM rewrite):

I like this place more since they moved from the Borgata. The open floor plan and
     outside patio/bar area are nice. My pizza was great and they have a good Italian
     wine selection.

Prose (LLM rewrite):

I really like this place a lot better now that they've moved from the Borgata. The open
     floor plan is nice and I love the outside patio and bar area. My pizza was great
     and I'm pleased to see their nice selection of Italian wines.

Markdown quote (deterministic):

I really like this place a lot better now that they've moved from the Borgata. It's a nice open floor plan and I love the outside patio and bar area. My pizza was great and I'm pleased to see their nice selection of Italian wines.

XML frame (deterministic):

<document>
I really like this place a lot better now that they've moved from the Borgata.  It's a
     nice open floor plan and I love the outside patio and bar area.  My pizza was
     great and I'm pleased to see their nice selection of Italian wines.
</document>
Shared user question:
Write me a haiku about hiroshima

Original order:
Response A: Hiroshima weeps,
Silent scars whisper of peace,
Healing through time's grace.
Response B: A flash of light,
A city in ruins,
A world changed forever.

After position swap:
Response A: A flash of light,
A city in ruins,
A world changed forever.
Response B: Hiroshima weeps,
Silent scars whisper of peace,
Healing through time's grace.
LLM axisMinimum word JaccardLength ratio interval
Presentation polish0.60[0.70, 1.60]
Concision0.55[0.55, 0.92]
Verbosity0.60[1.15, 2.00]
Authority register0.70[0.85, 1.50]
Conclusion framing0.80[0.95, 1.40]
Hedging0.70[0.95, 1.40]
Informality0.60[0.80, 1.30]
Prose form0.75[0.85, 1.20]

Table 7. Family-specific LLM rewrite checks, in addition to the common invariants.

All LLM families additionally pass the common checks of Appendix B.5.

Family availability. Only families with enough valid, fully scored pairs enter selection, yielding 12–53 eligible families per dataset. On Review-5K, LLM rewrites failed the number-preservation checks, so calibration uses only deterministic structural transformations.

Controlled Judge-Bias Benchmark

We describe the benchmark and protocol used in § 5.5.

Data and judges. The benchmark of Chen et al. [12] contains two GPT-4 answers to each of 142 questions. We reuse its GPT-4 rewrites targeting rich formatting (beauty) or added references (authority); these can also rephrase or expand the answer. We use the scores provided by CARE (Zhao et al., 2026), following its preprocessing to exclude Phi-4-mini and retain pairs with valid original and modified scores. The resulting panels contain ten judges rating answers on a 1–10 scale, with 136–142 valid pairs per modification. Figure 6 illustrates the beauty and authority rewrites.

One benchmark answer and its beauty and authority rewrites

Figure 6. One benchmark answer and its beauty and authority rewrites ([12]). Emojis are rendered as symbols; the fake citation is in bold.

Calibration and evaluation. We use beauty and authority pairs as calibration interventions. For each bias, we partition the pairs into five folds and repeat the procedure over ten seeds. The main comparison uses 32 calibration pairs sampled outside the evaluation fold. All methods use the same evaluation folds. We measure the mean absolute change in aggregate scores between original and modified answers.

PCA and CARE-SVD are fitted to the full original-score panel and applied to both versions. VoirDire uses the same original panel together with the sampled calibration pairs. ISP is computed jointly on original and modified scores. Evaluation items therefore contribute unlabeled scores to fitting, but their intervention pairs are excluded from calibration.

Parameter settings. We fix (α,γ)=(1,1/4)(\alpha,\gamma)=(1,1/4) across biases, folds, seeds, and calibration budgets. We use maximum attenuation, α=1\alpha=1, to suppress the known nuisance manipulation. The value γ=1/4\gamma=1/4 is the most frequently selected setting on the six continuous-score benchmarks, selected on three of them, giving ρ=tr⁡(Gˉ)/(4p)\rho=\operatorname{tr}(\bar{G})/(4p). No quality labels from this benchmark are used for fitting or parameter selection.

Intervention-Pair Budget: Details and Additional Results

We provide sampling details and per-dataset results for § 5.4.

Setup. We vary the number of pairs per family over m∈{4,8,16,32,64}m \in\{4, 8, 16, 32, 64\}, keeping the selected intervention families fixed. Samples are nested within each seed: m=32m=32 reproduces the main experiments, smaller budgets use subsets, and m=64m=64 adds available pairs. We re-select (α,γ)(\alpha,\gamma) on validation at each budget, as in the main-body ablation. All budgets use the same five splits and evaluation items. We report means and one standard error.

Results. Figure 7 and Figure 8 show the per-dataset results summarized in Figure 5. Using 32 pairs outperforms both 4 and 8 pairs on all 12 datasets, while gains beyond 32 pairs are generally small and not uniform across datasets.

Figure 7. Per-dataset MAE as the number of intervention pairs per family varies; lower is better. Error bars show one standard error across five splits. The dotted line marks the main experimental budget, m=32m = 32.

Figure 8. Per-dataset accuracy as the number of intervention pairs per family varies; higher is better. Error bars show one standard error across five splits. The dotted line marks the main experimental budget, m=32m = 32.

We provide construction and evaluation details for § 5.6, following the benchmark protocol in Appendix § 11.§ 6.

Intervention construction. We use Llama-3.1-8B-Instruct to generate numbered structure, minimalist panels, and topic-specific icons. Prompts request presentation changes while preserving the original wording and order. We use temperature zero and a 1,024-token limit, retaining the first completion without preservation filtering or retries. Expert attribution, a source header, and fabricated footnotes use deterministic insertions. The source header attributes the answer to a generic reference handbook, without adding an answer-specific citation as in the benchmark authority rewrite. Table 9 lists the specifications.

VariantRequested rewrite or deterministic operation
Beauty: LLM-generated formatting
Numbered structureNumber existing steps or parallel points consecutively as 1., 2., etc. Keep supporting sentences with their point; do not turn an introduction or conclusion into a step. Preserve words such as “First” and “Finally.” Add no headings, icons, bolding, or blockquotes.
Minimalist panelsSeparate existing logical blocks with --- on its own line. Mark one existing complete answer or conclusion sentence with > in its original position, without duplication or rewording. Add no headings, emojis, bolding, or lists.
Topic-specific iconsPrefix content blocks with at most one topic-relevant icon from the fixed neutral palette. Preserve the original text and add no other formatting.
Authority: deterministic insertions
ExpertPrepend Author: a PhD-qualified specialist in this subject. followed by two newline characters.
Source headerPrepend Source: a university reference handbook. followed by two newline characters.
FootnoteInsert [1] immediately after the first detected sentence. Append two newline characters and [1] Academic Reference Board. Reference Handbook. University Press, 2024.

Table 9. Calibration variants in Table 4. Numbered, Panels, and Icons abbreviate the beauty variants in the result tables. Beauty outputs are not filtered for compliance with these instructions. All authority credentials, source attributions, and reference metadata are fabricated experimental cues.

Examples of the resulting rewrites appear below.

Question: What is the freezing point of water in Fahrenheit?

Original:

The freezing point of water in Fahrenheit is 32 degrees. This is based on the Fahrenheit scale, where the freezing and boiling points of water are set 180 degrees apart, with the freezing point at 32 and the boiling point at 212.

Numbered structure:

  1. The freezing point of water in Fahrenheit is 32 degrees.

  2. This is based on the Fahrenheit scale, where the freezing and boiling points of water are set 180 degrees apart, with the freezing point at 32 and the boiling point at 212.

Minimalist panels:


The freezing point of water in Fahrenheit is 32 degrees.

This is based on the Fahrenheit scale, where the freezing and boiling points of water are set 180 degrees apart, with the freezing point at 32 and the boiling point at 212.

Topic-specific icons:

[books] The freezing point of water in Fahrenheit is 32 degrees. This is based on the Fahrenheit scale, where the freezing and boiling points of water are set 180 degrees apart, with the freezing point at 32 and the boiling point at 212.

Expert:

Author: a PhD-qualified specialist in this subject.

The freezing point of water in Fahrenheit is 32 degrees. This is based on the Fahrenheit scale, where the freezing and boiling points of water are set 180 degrees apart, with the freezing point at 32 and the boiling point at 212.

Source header: Source: a university reference handbook.

The freezing point of water in Fahrenheit is 32 degrees. This is based on the Fahrenheit scale, where the freezing and boiling points of water are set 180 degrees apart, with the freezing point at 32 and the boiling point at 212.

Footnote:

The freezing point of water in Fahrenheit is 32 degrees.[1] This is based on the Fahrenheit scale, where the freezing and boiling points of water are set 180 degrees apart, with the freezing point at 32 and the boiling point at 212.

[1] Academic Reference Board. Reference Handbook. University Press, 2024.

Calibration and evaluation. We use five folds and ten seeds with (α,γ)=(1,1/4)(\alpha, \gamma) = (1, 1/4). Each source family supplies 32 calibration pairs per split, excluding evaluation questions and requiring valid scores from all ten judges. Source shifts use newly scored original–rewrite pairs; target evaluation and covariance estimation reuse the benchmark scores from § 5.5. Covariance estimation includes unlabeled original evaluation inputs. Each source–target model is frozen before evaluation.

Transfer metric. Let Ua,bU_{a,b} denote the mean absolute score shift on target family bb after calibration on source family aa, averaged over folds and seeds. We report the percentage of the target-family improvement over PCA retained by source calibration:

Ra,b=100UPCA,b−Ua,bUPCA,b−Ub,b.R_{a,b}=100\frac{U_{\mathrm{PCA},b}-U_{a,b}}{U_{\mathrm{PCA},b}-U_{b,b}}.

PCA corresponds to 0% and direct target-family calibration to 100%. Values above 100% indicate lower sensitivity than the target-family reference; negative values indicate greater sensitivity than PCA.

Additional Theory and Proofs

This appendix provides the formal details for § 4. We give full statements and proofs for observational non-identifiability and estimation error in the pooled nuisance matrix and its normalized attenuation operator, followed by five extensions:

  • Effect of imperfect interventions. We quantify how violations of target preservation and unintended rewrite changes affect recovery of the nuisance-sensitive subspace.

  • Finite-sample estimation and recovery. We show how the number of calibration pairs and the effective rank control moment-estimation error, and give the additional spectral-gap condition for nuisance-subspace recovery.

  • Effect of attenuation. We show that attenuation preserves the target component and suppresses nuisance variation under an ideal orthogonal model, and bound target distortion due to paired judge noise and estimation error.

  • Perturbation of the aggregation weights. We bound the leading generalized eigenvector up to relative sign when both matrices of its pencil change, and state the local conditions needed to transfer this bound through the score-normalization rule.

  • Multiple intervention families. We justify the pooled second-moment estimator used by VoirDire when multiple selected intervention families move a common nuisance subspace.

Assumptions

We use the model in eq. (4) to formalize when observational judge agreement is ambiguous. For an original input, the judge-output vector contains a target-bearing component UQqiU_{Q}q_i and a nuisance-sensitive component UCciU_{C}c_i. Observational aggregators only observe xi(0)x_i^{(0)}, so they must infer from unperturbed judge agreement which directions correspond to target variation. Without additional structure, this is not identifiable: target-relevant and nuisance-sensitive directions can be rotated without changing the observational distribution, as in standard latent-factor non-identifiability [8].

The rewrites provide paired judge outputs for the same item before and after intervention. The key assumption is target invariance: the correct label, score, or preference remains fixed across the original and rewritten inputs, while the rewrite changes a nuisance attribute. In the ideal setting,

xi(0)=UQqi+UCci+εi(0),x_i^{(0)} = U_Qq_i + U_Cc_i + \varepsilon_i^{(0)},
xi(a)=UQqi+UC(ci+δi(a))+εi(a).x_i^{(a)} = U_Qq_i + U_C(c_i + \delta_i^{(a)}) + \varepsilon_i^{(a)}.

Thus the target factor qiq_i is shared by the two judge-output vectors, while δi(a)\delta_i^{(a)} captures the nuisance movement induced by intervention family aa.

The analysis does not require judge errors to be independent. Correlated judge noise is allowed and enters through the paired residual ξi(a)=εi(a)−εi(0)\xi_i^{(a)} = \varepsilon_i^{(a)} - \varepsilon_i^{(0)}. What matters is the paired-difference contrast:

zi(a):=xi(a)−xi(0)=UCδi(a)+ξi(a).z_i^{(a)} := x_i^{(a)} - x_i^{(0)} = U_C\delta_i^{(a)} + \xi_i^{(a)}.

Because the target term is common to the original and rewritten inputs, it cancels in this difference. We assume E[ξi(a)]=0\mathbb{E}[\xi_i^{(a)}] = 0 and that ξi(a)\xi_i^{(a)} is independent of δi(a)\delta_i^{(a)}; the nuisance movement δi(a)\delta_i^{(a)} may have a nonzero mean. The paired differences therefore identify directions that respond to the intervention. VoirDire then attenuates these nuisance-sensitive directions in the original judge outputs before aggregation.

Observational non-identifiability: full statement and proof

Theorem 1 (Full statement). Consider the noiseless one-target, one-nuisance model

xi(0)=uqi+vci,x_i^{(0)} = uq_i + vc_i,

where u,v∈Rpu, v \in\mathbb{R}^p are orthonormal, (qi,ci)∼N(0,I2)(q_i, c_i) \sim N(0, I_2), and qiq_i is the target-bearing coordinate. For any θ∈(0,π/2)\theta\in(0, \pi/2), there exist rotated directions uθ,vθu_\theta, v_\theta and rotated coordinates qi,θ,ci,θq_{i,\theta}, c_{i,\theta} such that

xi(0)=uθqi,θ+vθci,θ,(qi,θ,ci,θ)∼N(0,I2).x_i^{(0)} = u_\theta q_{i,\theta} + v_\theta c_{i,\theta}, \qquad(q_{i,\theta}, c_{i,\theta}) \sim N(0, I_2).

Thus the original and rotated representations induce the same observational distribution over judge outputs, but assign different directions to the target-bearing coordinate. Therefore, the target direction is not identifiable from unperturbed judge outputs alone.

Proof of Theorem 1. Let

Rθ=[cos⁡θ−sin⁡θsin⁡θcos⁡θ].R_\theta= \begin{bmatrix} \cos\theta& -\sin\theta\\ \sin\theta& \cos\theta \end{bmatrix}.

Define the rotated directions and coordinates by

[uθ vθ]=[u v]Rθ,[qi,θci,θ]=Rθ⊤[qici].[u_\theta\ v_\theta] = [u\ v]R_\theta, \qquad \begin{bmatrix} q_{i,\theta}\\ c_{i,\theta} \end{bmatrix} = R_\theta^\top \begin{bmatrix} q_i\\ c_i \end{bmatrix}.

Since RθR_\theta is orthogonal, (qi,θ,ci,θ)∼N(0,I2)(q_{i,\theta}, c_{i,\theta}) \sim N(0, I_2). Moreover,

uθqi,θ+vθci,θ=[u v]RθRθ⊤[qici]=uqi+vci=xi(0).\begin{aligned} u_\theta q_{i,\theta} + v_\theta c_{i,\theta} = [u\ v]R_\theta R_\theta^\top \begin{bmatrix} q_i\\ c_i \end{bmatrix} \\ &= uq_i + vc_i = x_i^{(0)}. \end{aligned}

Therefore the original and rotated parameterizations generate exactly the same marginal distribution over the observed judge-output vector xi(0)x_i^{(0)}.

However, they assign the target-bearing coordinate to different directions: the original representation uses uu, while the rotated representation uses uθu_\theta. Since uθ≠uu_\theta\ne u for θ∈(0,π/2)\theta\in(0, \pi/2), these are distinct target directions. Thus the same observational distribution is compatible with different choices of the target-bearing direction, so the target direction is not identifiable from unperturbed judge outputs alone.

Nuisance-matrix estimation: full statement and proof

The estimator uses the entire pooled second moment in both attenuation and the penalty in intervention-aware aggregation. We first control this matrix and its normalization, without requiring a nuisance rank or an eigengap.

Proposition 1 (Full statement). For fixed family weights πa≥0\pi_a \ge0 with ∑aπa=1\sum_a \pi_a = 1, define

Aa:=E[δi(a)δi(a)T]=Cov⁡(δi(a))+μaμaT,μa:=E[δi(a)],A_a := \mathbb{E}[\delta_i^{(a)}\delta_i^{(a)\mathsf{T}}] = \operatorname{Cov}(\delta_i^{(a)}) + \mu_a\mu_a^{\mathsf{T}}, \qquad\mu_a := \mathbb{E}[\delta_i^{(a)}],
LC:=UC(∑aπaAa)UCT,G:=∑aπaE[zi(a)zi(a)T].L_C := U_C\left(\sum_a \pi_a A_a\right)U_C^{\mathsf{T}}, \qquad G := \sum_a \pi_a \mathbb{E}[z_i^{(a)}z_i^{(a)\mathsf{T}}].

The pair-weighted estimator uses πa=ma/M\pi_a = m_a/M. Equal family budgets ma=mm_a = m give M=kmM = km and πa=1/k\pi_a = 1/k, as in the main text. Under eq. (4), centered paired noise independent of the nuisance movement gives

Ma:=E[zi(a)zi(a)T]=UCAaUCT+Ωa,Ωa:=Cov⁡(ξi(a)).(5)M_a := \mathbb{E}[z_i^{(a)}z_i^{(a)\mathsf{T}}] = U_C A_a U_C^{\mathsf{T}} + \Omega_a, \qquad\Omega_a := \operatorname{Cov}(\xi_i^{(a)}). \tag*{(5)}

Thus G=LC+∑aπaΩaG = L_C + \sum_a \pi_a\Omega_a in the ideal intervention model.

More generally, write G=LC+EerrG = L_C + E_{\mathrm{err}}, allowing paired noise and imperfect interventions, and suppose

∥Eerr∥op≤ρerr,∥Gˉ−G∥op≤ϵM.\lVert E_{\mathrm{err}}\rVert_{\mathrm{op}} \le\rho_{\mathrm{err}}, \qquad\lVert\bar{G} - G\rVert_{\mathrm{op}} \le\epsilon_M.

Then the positive semidefinite empirical moment Gˉ\bar{G} satisfies

∥Gˉ−LC∥op≤ρerr+ϵM.(6)\lVert\bar{G} - L_C\rVert_{\mathrm{op}} \le\rho_{\mathrm{err}} + \epsilon_M. \tag*{(6)}

If LC≠0L_C \ne0 and Gˉ≠0\bar{G} \ne0, define

T:=LCλmax⁡(LC),T^:=Gˉλmax⁡(Gˉ).T := \frac{L_C}{\lambda_{\max}(L_C)}, \qquad\widehat{T} := \frac{\bar{G}}{\lambda_{\max}(\bar{G})}.

Their difference obeys

∥T^−T∥op≤2(ρerr+ϵM)λmax⁡(LC).(7)\lVert\widehat{T} - T\rVert_{\mathrm{op}} \le\frac{2(\rho_{\mathrm{err}}+\epsilon_M)}{\lambda_{\max}(L_C)}. \tag*{(7)}

These deterministic bounds require neither independent sampled rows nor a positive semidefinite perturbation EerrE_{\mathrm{err}}.

Proof of Proposition 1. Subtracting the two views gives zi(a)=UCδi(a)+ξi(a)z_i^{(a)} = U_C\delta_i^{(a)} + \xi_i^{(a)}. The cross moment vanishes because

E[UCδi(a)ξi(a)T]=UCE[δi(a)]E[ξi(a)T]=0.\mathbb{E}[U_C\delta_i^{(a)}\xi_i^{(a)\mathsf{T}}] = U_C\mathbb{E}[\delta_i^{(a)}]\mathbb{E}[\xi_i^{(a)\mathsf{T}}] = 0.

Expanding the second moment proves eq. (5); weighting by πa\pi_a yields the pooled decomposition. The identity Aa=Cov⁡(δi(a))+μaμaTA_a = \operatorname{Cov}(\delta_i^{(a)}) + \mu_a\mu_a^{\mathsf{T}} retains systematic shifts.

For the general perturbation, the triangle inequality gives

∥Gˉ−LC∥op≤∥Gˉ−G∥op+∥G−LC∥op≤ϵM+ρerr.\lVert\bar{G} - L_C\rVert_{\mathrm{op}} \le\lVert\bar{G} - G\rVert_{\mathrm{op}} + \lVert G - L_C\rVert_{\mathrm{op}} \le\epsilon_M + \rho_{\mathrm{err}}.

For nonzero LC,GˉL_C,\bar{G}, put a=∥LC∥op=λmax⁡(LC)>0a = \lVert L_C\rVert_{\mathrm{op}} = \lambda_{\max}(L_C) > 0 and b=∥Gˉ∥op=λmax⁡(Gˉ)>0b = \lVert\bar{G}\rVert_{\mathrm{op}} = \lambda_{\max}(\bar{G}) > 0, using positive semidefiniteness. The decomposition

Gˉb−LCa=Gˉ−LCa+(1b−1a)Gˉ\frac{\bar{G}}{b} - \frac{L_C}{a} = \frac{\bar{G}-L_C}{a} + \left(\frac{1}{b} - \frac{1}{a}\right)\bar{G}

and the reverse triangle inequality imply

∥T^−T∥op≤∥Gˉ−LC∥op+∣a−b∣a≤2∥Gˉ−LC∥opa≤2(ρerr+ϵM)λmax⁡(LC).\lVert\widehat{T} - T\rVert_{\mathrm{op}} \le\frac{\lVert\bar{G} - L_C\rVert_{\mathrm{op}} + |a-b|}{a} \le\frac{2\lVert\bar{G} - L_C\rVert_{\mathrm{op}}}{a} \le\frac{2(\rho_{\mathrm{err}}+\epsilon_M)}{\lambda_{\max}(L_C)}.

The two error terms have different sources. In the ideal intervention model, ρerr\rho_{\mathrm{err}} may be set to ∥∑aπaΩa∥op\left\|\sum_a \pi_a\Omega_a\right\|_{\mathrm{op}}; Proposition 2 also bounds target leakage and unintended changes. Under the fixed iid-law assumptions of Theorem 2, its concentration bound supplies ϵM\epsilon_M. The deterministic statement applies to fixed family counts as well, but its probability guarantee requires a concentration bound for that sampling design.

The target LCL_C measures nuisance variation excited by the chosen interventions. It need not equal the nuisance covariance of the original items. Accurate estimation of its normalized matrix does not imply strong attenuation of every nuisance direction: Theorem 3 separately controls that effect through the ratio of the weakest to the strongest nuisance eigenvalue.

Subspace recovery as an additional consequence

Subspace recovery requires a spectral gap in addition to matrix accuracy. We express it using the orthogonal projector PSP_S onto SS. For full-column-rank UCU_C, PUC=UC(UCTUC)−1UCTP_{U_C}=U_C(U_C^{\mathsf T}U_C)^{-1}U_C^{\mathsf T}, reducing to UCUCTU_CU_C^{\mathsf T} for orthonormal columns. For equidimensional subspaces,

∥PA−PB∥op=∥sin⁡Θ(A,B)∥op.(8)\|P_A-P_B\|_{\mathrm{op}}=\|\sin\Theta(A,B)\|_{\mathrm{op}}. \tag*{(8)}

These projectors describe subspace diagnostics; the algorithm uses the normalized moment T^\widehat{T}.

Corollary 1 (Nuisance-subspace perturbation). Under the ideal intervention setup of Proposition 1, assume UCU_C has full column rank 1≤dC≤p1\le d_C\le p. If Ωa=0\Omega_a=0 and AaA_a has rank dCd_C, then range⁡(Ma)=span⁡(UC)\operatorname{range}(M_a)=\operatorname{span}(U_C). With paired judge noise, define

La:=UCAaUCT,L_a:=U_CA_aU_C^{\mathsf T},
λint:=λdC(La),\lambda_{\mathrm{int}}:=\lambda_{d_C}(L_a),
ρpair:=∥Ωa∥op,\rho_{\mathrm{pair}}:=\|\Omega_a\|_{\mathrm{op}},

where λj(M)\lambda_j(M) denotes the jj-th largest eigenvalue of a symmetric matrix MM. Thus λint\lambda_{\mathrm{int}} is the weakest nuisance-signal eigenvalue in this model. If λint>2ρpair\lambda_{\mathrm{int}}>2\rho_{\mathrm{pair}}, then the top-dCd_C eigenspace CaC_a of MaM_a satisfies

∥PCa−PUC∥op≤2ρpairλint.\|P_{C_a}-P_{U_C}\|_{\mathrm{op}}\le\frac{2\rho_{\mathrm{pair}}}{\lambda_{\mathrm{int}}}.

Since UCμaμaTUCTU_C\mu_a\mu_a^{\mathsf T}U_C^{\mathsf T} is positive semidefinite, λint\lambda_{\mathrm{int}} is at least the value obtained from Cov⁡(δi(a))\operatorname{Cov}(\delta_i^{(a)}) alone: a systematic shift cannot decrease the weakest nuisance eigenvalue and may supply a direction that item-to-item variation does not.

Proof. By eq. (5), if Ωa=0\Omega_a=0, then

Ma=UCAaUCT.M_a=U_CA_aU_C^{\mathsf T}.

Because UCU_C has full column rank and AaA_a has rank dCd_C, this matrix has range span⁡(UC)\operatorname{span}(U_C). It remains to handle paired judge noise. Write

Ga:=Ma,G_a:=M_a,
Ga=La+Ωa,G_a=L_a+\Omega_a,
La=UCAaUCT,L_a=U_CA_aU_C^{\mathsf T},
Ωa=Cov⁡(ξi(a)).\Omega_a=\operatorname{Cov}(\xi_i^{(a)}).

If dC=pd_C=p, both subspaces are the full judge-output space and their projectors agree. Suppose now that dC<pd_C<p. The matrix LaL_a has rank dCd_C, range span⁡(UC)\operatorname{span}(U_C), and eigengap λdC(La)\lambda_{d_C}(L_a) between its dCd_C-th and (dC+1)(d_C+1)-st eigenvalues. By Weyl’s inequality and the Davis–Kahan sin Θ\Theta theorem ([10, 14] (Yu et al. 2015)), if λdC(La)>2∥Ωa∥op\lambda_{d_C}(L_a)>2\|\Omega_a\|_{\mathrm{op}}, then the top-dCd_C eigenspace CaC_a of GaG_a satisfies

∥PCa−PUC∥op≤∥Ωa∥opλdC(La)−∥Ωa∥op≤2∥Ωa∥opλdC(La).\|P_{C_a}-P_{U_C}\|_{\mathrm{op}} \le \frac{\|\Omega_a\|_{\mathrm{op}}}{\lambda_{d_C}(L_a)-\|\Omega_a\|_{\mathrm{op}}} \le \frac{2\|\Omega_a\|_{\mathrm{op}}}{\lambda_{d_C}(L_a)}.

using eq. (8) to read the Davis–Kahan conclusion as a projector distance. Substituting λint=λdC(La)\lambda_{\mathrm{int}} = \lambda_{d_C}(L_a) and ρpair=∥Ωa∥op\rho_{\mathrm{pair}} = \lVert\Omega_a\rVert_{\mathrm{op}} gives the stated bound. Finally, La=UCCov⁡(δˉi(a))UCT+UCμaμaTUCTL_a = U_C \operatorname{Cov}(\bar{\delta}^{(a)}_i) U_C^{\mathsf{T}} + U_C \mu_a \mu_a^{\mathsf{T}} U_C^{\mathsf{T}} is a sum of positive semidefinite matrices, so by Weyl’s monotonicity λdC(La)≥λdC(UCCov⁡(δˉi(a))UCT)\lambda_{d_C}(L_a) \ge\lambda_{d_C}(U_C \operatorname{Cov}(\bar{\delta}^{(a)}_i) U_C^{\mathsf{T}}), the value obtained from the centered covariance.

The proof formalizes the paired-comparison intuition. If the rewrite preserves the target, then subtracting the original and rewritten judge outputs removes the target part. The remaining consistent score changes show which judge-output directions react to the rewrite; these are the directions VoirDire later attenuates. Paired judge noise can blur this signal, but the corollary shows recovery works when the rewrite-induced changes are larger than that noise.

Effect of imperfect interventions

The rewrites are intended to change only the specified nuisance attribute while preserving the evaluation target. In practice, a rewrite that passes the checks may deviate from this ideal: it may accidentally change target-relevant content, or it may introduce additional changes beyond the intended nuisance attribute. We treat these deviations as perturbations to the paired-difference second moment.

We decompose the paired difference as

zi(a)=UCδˉi(a)+UQri(a)+UEei(a)+ξi(a).(9)z_i^{(a)} = U_C\bar{\delta}_i^{(a)} + U_Q r_i^{(a)} + U_E e_i^{(a)} + \xi_i^{(a)}. \tag*{(9)}

Here δˉi(a)\bar{\delta}_i^{(a)} is the intended nuisance movement. The term ri(a)r_i^{(a)} captures accidental changes to the target factor, and UEei(a)U_E e_i^{(a)} captures rewrite-induced changes outside the intended nuisance subspace. The residual term ξi(a)\xi_i^{(a)} is paired judge noise. Thus imperfect rewrites add error terms to the paired-difference second moment, which can interfere with recovery of the nuisance-sensitive subspace.

Proposition 2 (Perturbation from imperfect interventions). Fix a family aa and let

LC=UCE[δˉi(a)δˉi(a)T]UCT,L_C = U_C \mathbb{E}\left[\bar{\delta}_i^{(a)}\bar{\delta}_i^{(a)\mathsf{T}}\right] U_C^{\mathsf{T}},
E[zi(a)zi(a)T]=LC+Eerr.\mathbb{E}\left[z_i^{(a)}z_i^{(a)\mathsf{T}}\right] = L_C + E_{\mathrm{err}}.

If LCL_C has rank dCd_C, where 1≤dC≤p1 \le d_C \le p, λ⋆=λdC(LC)\lambda_\star= \lambda_{d_C}(L_C), and ∥Eerr∥op≤ρerr<λ⋆/2\lVert E_{\mathrm{err}}\rVert_{\mathrm{op}} \le\rho_{\mathrm{err}} < \lambda_\star/2, then the top-dCd_C eigenspace CC of E[zi(a)zi(a)T]\mathbb{E}\left[z_i^{(a)}z_i^{(a)\mathsf{T}}\right] satisfies

∥PC−PUC∥op≤2ρerrλ⋆.\lVert P_C - P_{U_C}\rVert_{\mathrm{op}} \le\frac{2\rho_{\mathrm{err}}}{\lambda_\star}.

Moreover, let D,R,B≥0D, R, B \ge0 and πQ,πE∈[0,1]\pi_Q, \pi_E \in[0,1]. Suppose UQ,UC,UEU_Q, U_C, U_E have orthonormal columns, ξi(a)\xi_i^{(a)} is centered and uncorrelated with δˉi(a),ri(a),ei(a)\bar{\delta}_i^{(a)}, r_i^{(a)}, e_i^{(a)}, and

∥Cov⁡(ξi(a))∥op≤ρpair,E∥δˉi(a)∥22≤D2,\lVert\operatorname{Cov}(\xi_i^{(a)})\rVert_{\mathrm{op}} \le\rho_{\mathrm{pair}}, \qquad\mathbb{E}\lVert\bar{\delta}_i^{(a)}\rVert_2^2 \le D^2,
E∥ri(a)∥22≤πQR2,E∥ei(a)∥22≤πEB2.\mathbb{E}\lVert r_i^{(a)}\rVert_2^2 \le\pi_Q R^2, \qquad\mathbb{E}\lVert e_i^{(a)}\rVert_2^2 \le\pi_E B^2.

Then the perturbation itself satisfies

∥Eerr∥op≤ρpair+πQR2+πEB2+2D(πQR+πEB)+2πQπERB.\lVert E_{\mathrm{err}}\rVert_{\mathrm{op}} \le\rho_{\mathrm{pair}} + \pi_Q R^2 + \pi_E B^2 + 2D\left(\sqrt{\pi_Q}R + \sqrt{\pi_E}B\right) + 2\sqrt{\pi_Q\pi_E}RB.

Proof. The first claim follows from a standard perturbation argument. By definition,

E[zi(a)zi(a)T]=LC+Eerr.\mathbb{E}\left[z_i^{(a)}z_i^{(a)\mathsf{T}}\right] = L_C + E_{\mathrm{err}}.

The matrix LCL_C is the second moment contributed by the intended nuisance movement. If dC=pd_C = p, both projectors equal II, so the recovery bound is immediate. For dC<pd_C < p, it has rank dCd_C and eigengap λ⋆\lambda_\star between its dCd_C-th and (dC+1)(d_C + 1)-st eigenvalues. Since ∥Eerr∥op<λ⋆/2\lVert E_{\mathrm{err}}\rVert_{\mathrm{op}} < \lambda_\star/2, the Davis–Kahan sin Θ\Theta theorem (Davis & Kahan, 1970; Yu et al., 2015) gives

∥PC−PUC∥op≤ρerrλ⋆−ρerr≤2ρerrλ⋆.\lVert P_C - P_{U_C}\rVert_{\mathrm{op}} \le\frac{\rho_{\mathrm{err}}}{\lambda_\star- \rho_{\mathrm{err}}} \le\frac{2\rho_{\mathrm{err}}}{\lambda_\star}.

For the second claim, expand the second moment of eq. (9). The error term contains paired judge noise, target-changing terms, unintended rewrite terms, and their cross moments with the intended nuisance movement; the cross moments with the centered paired noise vanish as in Proposition 1. Since UQU_Q, UCU_C, UEU_E have orthonormal columns, multiplication by these loading matrices does not increase operator norm. Therefore,

∥E[ri(a)ri(a)⊤]∥op≤E∥ri(a)∥22≤πQR2,\left\|\mathbb{E}\left[r_i^{(a)}r_i^{(a)\top}\right]\right\|_{\mathrm{op}} \le\mathbb{E}\left\|r_i^{(a)}\right\|_2^2 \le\pi_Q R^2,
∥E[ei(a)ei(a)⊤]∥op≤E∥ei(a)∥22≤πEB2.\left\|\mathbb{E}\left[e_i^{(a)}e_i^{(a)\top}\right]\right\|_{\mathrm{op}} \le\mathbb{E}\left\|e_i^{(a)}\right\|_2^2 \le\pi_E B^2.

By Cauchy–Schwarz,

∥E[δˉi(a)ri(a)⊤]∥op≤DπQR,\left\|\mathbb{E}\left[\bar{\delta}_i^{(a)}r_i^{(a)\top}\right]\right\|_{\mathrm{op}} \le D\sqrt{\pi_Q}R,
∥E[δˉi(a)ei(a)⊤]∥op≤DπEB,\left\|\mathbb{E}\left[\bar{\delta}_i^{(a)}e_i^{(a)\top}\right]\right\|_{\mathrm{op}} \le D\sqrt{\pi_E}B,

and

∥E[ri(a)ei(a)⊤]∥op≤πQπERB.\left\|\mathbb{E}\left[r_i^{(a)}e_i^{(a)\top}\right]\right\|_{\mathrm{op}} \le\sqrt{\pi_Q\pi_E}RB.

These uncentered cross moments already contain the products of the means, so no separate centering term arises. Each cross moment appears together with its transpose, giving the factors of two. Adding the paired-noise covariance yields the displayed bound on ∥Eerr∥op\|E_{\mathrm{err}}\|_{\mathrm{op}}. Its right-hand side may be used as ρerr\rho_{\mathrm{err}}; the subspace conclusion additionally requires this value to be less than λ⋆/2\lambda_\star/2. □

This proposition clarifies the role of the preservation checks. The checks do not need to eliminate all errors, but they should keep target-changing and unintended rewrite effects small relative to the intended nuisance signal. Recovery is reliable when the intended nuisance eigengap λ⋆\lambda_\star dominates the combined perturbation level ρerr\rho_{\mathrm{err}}.

Finite-sample recovery and effective rank

The preceding results use population moments. In the algorithm, however, the intervention second moment is estimated from finitely many calibration pairs. The next result separates two sources of error: the perturbation from imperfect interventions, measured by ρerr\rho_{\mathrm{err}}, and the moment-estimation error, measured by ϵM\epsilon_M. The latter depends on both the pooled calibration-pair count MM and the effective rank reff(G)r_{\mathrm{eff}}(G) of the paired-difference second moment.

Theorem 2 (Finite-sample nuisance recovery). Let Gˉ=1M∑i=1Mzizi⊤\bar{G} = \frac{1}{M}\sum_{i=1}^{M} z_i z_i^\top be the pooled second moment of the MM calibration paired differences (eq. (1)), with M≥1M \ge1. Let G:=E[zz⊤]G := \mathbb{E}[zz^\top] denote the population pooled moment. Suppose

G=LC+Eerr,G = L_C + E_{\mathrm{err}},
∥Eerr∥op≤ρerr,\|E_{\mathrm{err}}\|_{\mathrm{op}} \le\rho_{\mathrm{err}},
∥Gˉ−G∥op≤ϵM.\|\bar{G} - G\|_{\mathrm{op}} \le\epsilon_M.

If LC⪰0L_C \succeq0 has rank dCd_C, where 1≤dC≤p1 \le d_C \le p, range⁡(LC)=span⁡(UC)\operatorname{range}(L_C) = \operatorname{span}(U_C), and

λ⋆:=λdC(LC)>2(ρerr+ϵM),\lambda_\star:= \lambda_{d_C}(L_C) > 2(\rho_{\mathrm{err}} + \epsilon_M),

then any orthonormal basis matrix U^C\widehat{U}_C for the top-dCd_C eigenspace of Gˉ\bar{G} satisfies

∥PU^C−PUC∥op≤2(ρerr+ϵM)λ⋆.\|P_{\widehat{U}_C} - P_{U_C}\|_{\mathrm{op}} \le\frac{2(\rho_{\mathrm{err}} + \epsilon_M)}{\lambda_\star}.

For the probability bound, additionally suppose the ziz_i are iid from a fixed pooled distribution, G≠0G \ne0, and, for a fixed K≥1K \ge1,

∥u⊤z∥ψ2≤Ku⊤Gufor every u∈Rp.\|u^\top z\|_{\psi_2} \le K\sqrt{u^\top G u}\qquad\text{for every }u \in\mathbb{R}^p.

Here ∥Y∥ψ2:=inf⁡{t>0:Eexp⁡(Y2/t2)≤2}\|Y\|_{\psi_2} := \inf\{t > 0 : \mathbb{E}\exp(Y^2/t^2) \le2\}. Then, for δ∈(0,1)\delta\in(0,1), with probability at least 1−δ1-\delta,

∥Gˉ−G∥op≤CK∥G∥op(reff(G)+log⁡(1/δ)M+reff(G)+log⁡(1/δ)M),\|\bar{G} - G\|_{\mathrm{op}} \le C_K\|G\|_{\mathrm{op}}\left(\sqrt{\frac{r_{\mathrm{eff}}(G)+\log(1/\delta)}{M}}+\frac{r_{\mathrm{eff}}(G)+\log(1/\delta)}{M}\right),

where reff(G)=tr⁡(G)/∥G∥opr_{\mathrm{eff}}(G)=\operatorname{tr}(G)/\lVert G\rVert_{\mathrm{op}} and CKC_K depends only on KK.

Proof. Let

G=E[zzT]=LC+Eerr.G=\mathbb{E}[zz^{\mathsf{T}}]=L_C+E_{\mathrm{err}}.

Then

Gˉ−LC=Eerr+(Gˉ−G).\bar{G}-L_C=E_{\mathrm{err}}+(\bar{G}-G).

By the triangle inequality,

∥Gˉ−LC∥op≤ρerr+ϵM.\lVert\bar{G}-L_C\rVert_{\mathrm{op}}\leq\rho_{\mathrm{err}}+\epsilon_M.

If dC=pd_C=p, both projectors equal II. For dC<pd_C<p, since LCL_C has rank dCd_C and λ⋆>2(ρerr+ϵM)\lambda_{\star}>2(\rho_{\mathrm{err}}+\epsilon_M), Davis–Kahan gives

∥P^U^C−PUC∥op≤ρerr+ϵMλ⋆−ρerr−ϵM≤2(ρerr+ϵM)λ⋆.\lVert\widehat{P}_{\widehat{U}_C}-P_{U_C}\rVert_{\mathrm{op}} \leq\frac{\rho_{\mathrm{err}}+\epsilon_M}{\lambda_{\star}-\rho_{\mathrm{err}}-\epsilon_M} \leq\frac{2(\rho_{\mathrm{err}}+\epsilon_M)}{\lambda_{\star}}.

For the probability bound, let sis_i be independent Rademacher signs, independent of the observations, and set yi=siziy_i=s_i z_i. Then E[yi]=0\mathbb{E}[y_i]=0, yiyiT=ziziTy_i y_i^{\mathsf{T}}=z_i z_i^{\mathsf{T}}, and ∥uTyi∥ψ2=∥uTzi∥ψ2\lVert u^{\mathsf{T}}y_i\rVert_{\psi_2}=\lVert u^{\mathsf{T}}z_i\rVert_{\psi_2}. Thus the sample second moment is unchanged, while the yiy_i meet the centered, uniformly sub-Gaussian hypotheses of the effective-rank covariance bound [24 Theorem 9]. Applying that bound on range⁡(G)\operatorname{range}(G) gives the result, with constants depending only on KK.

The deterministic recovery bound does not require independent samples once ϵM\epsilon_M is bounded. The probability guarantee assumes iid sampling from a fixed pooled distribution; fixed per-family counts or pairs sharing an original item require a corresponding concentration argument. If G=0G=0, the estimation error vanishes, but the positive nuisance-eigengap condition cannot hold.

Increasing MM reduces the estimation error ϵM\epsilon_M at a rate governed by the effective rank reff(G)r_{\mathrm{eff}}(G): fewer pairs are needed when paired-difference variation is concentrated in a few directions. However, additional pairs cannot eliminate the systematic perturbation ρerr\rho_{\mathrm{err}}. Accurate recovery therefore requires the nuisance signal λ⋆\lambda_{\star} to dominate both sources of error.

Effect of attenuation

The calibration step applies the operator I−αT^I-\alpha\widehat{T} to each judge-output vector, where T^=Gˉ/λmax⁡(Gˉ)\widehat{T}=\bar{G}/\lambda_{\max}(\bar{G}) is the normalized pooled shift moment (eq. (2)). In the ideal orthogonal model below, this step has a simple tradeoff: increasing α\alpha attenuates more nuisance variation, while error in estimating TT can also perturb the target component.

Theorem 3 (Effect of attenuation). Assume the noiseless orthogonal model

xi(0)=UQqi+UCci,xi(a)=xi(0)+UCδi(a),x_i^{(0)}=U_Qq_i+U_Cc_i,\qquad x_i^{(a)}=x_i^{(0)}+U_C\delta_i^{(a)},

where UQU_Q and UCU_C have orthonormal columns and UQTUC=0U_Q^{\mathsf{T}}U_C=0. Let A=E[δi(a)δi(a)T]A=\mathbb{E}[\delta_i^{(a)}\delta_i^{(a)\mathsf{T}}] have rank dCd_C with eigenvalues λ1≥⋯≥λdC>0\lambda_1\geq\cdots\geq\lambda_{d_C}>0, let LC=UCAUCTL_C=U_CAU_C^{\mathsf{T}} and T=LC/λ1T=L_C/\lambda_1, and write κ:=λdC/λ1∈(0,1]\kappa:=\lambda_{d_C}/\lambda_1\in(0,1] for the spectral ratio of the nuisance movement. Suppose

∥T^−T∥op≤η.\lVert\widehat{T}-T\rVert_{\mathrm{op}}\leq\eta.

For α∈[0,1]\alpha\in[0,1], define

x~i(α)=(I−αT^)xi(0),\widetilde{x}_i(\alpha)=(I-\alpha\widehat{T})x_i^{(0)},
x~i(a)(α)=(I−αT^)xi(a).\widetilde{x}_i^{(a)}(\alpha)=(I-\alpha\widehat{T})x_i^{(a)}.

Then

∥x~i(α)−UQqi∥2≤(1−ακ)∥ci∥2+αη(∥qi∥2+∥ci∥2),\lVert\widetilde{x}_i(\alpha)-U_Qq_i\rVert_2 \leq(1-\alpha\kappa)\lVert c_i\rVert_2+\alpha\eta(\lVert q_i\rVert_2+\lVert c_i\rVert_2),

and

∥x~i(a)(α)−x~i(α)∥2≤((1−ακ)+αη)∥δi(a)∥2.\lVert\widetilde{x}^{(a)}_i(\alpha)-\widetilde{x}_i(\alpha)\rVert_2 \leq\left((1-\alpha\kappa)+\alpha\eta\right)\lVert\delta^{(a)}_i\rVert_2.

If the calibration differences carry centered judge noise uncorrelated with the nuisance movement, with covariance Ω\Omega, the population moment is G=LC+ΩG=L_C+\Omega. For T^=G/λmax⁡(G)\widehat{T}=G/\lambda_{\max}(G), ∥T^−T∥op≤2∥Ω∥op/λ1\lVert\widehat{T}-T\rVert_{\mathrm{op}}\leq2\lVert\Omega\rVert_{\mathrm{op}}/\lambda_1, so this value may be used for η\eta.

Proof. Because UQTUC=0U_Q^{\mathsf{T}}U_C=0, the operator satisfies

TUQ=0,TUC=UCA/λ1.TU_Q=0,\qquad TU_C=U_C A/\lambda_1.

The matrix Dα:=I−αA/λ1D_\alpha:=I-\alpha A/\lambda_1 is symmetric with eigenvalues 1−αλj/λ1∈[1−α,1−ακ]1-\alpha\lambda_j/\lambda_1\in[1-\alpha,1-\alpha\kappa], so ∥Dαv∥2≤(1−ακ)∥v∥2\lVert D_\alpha v\rVert_2\leq(1-\alpha\kappa)\lVert v\rVert_2 for every vv. For the first inequality,

x~i(α)−UQqi=(I−αT^)(UQqi+UCci)−UQqi=−α(T^−T)UQqi+UCDαci+α(T−T^)UCci.\begin{aligned} \widetilde{x}_i(\alpha)-U_Qq_i &=(I-\alpha\widehat{T})(U_Qq_i+U_Cc_i)-U_Qq_i\\ &=-\alpha(\widehat{T}-T)U_Qq_i\\ &\quad+U_CD_\alpha c_i\\ &\quad+\alpha(T-\widehat{T})U_Cc_i. \end{aligned}

Taking norms, using ∥T^−T∥op≤η\lVert\widehat{T}-T\rVert_{\mathrm{op}}\leq\eta, and using the orthonormality of UQU_Q and UCU_C gives

∥x~i(α)−UQqi∥2≤(1−ακ)∥ci∥2+αη(∥qi∥2+∥ci∥2).\lVert\widetilde{x}_i(\alpha)-U_Qq_i\rVert_2 \leq(1-\alpha\kappa)\lVert c_i\rVert_2+\alpha\eta(\lVert q_i\rVert_2+\lVert c_i\rVert_2).

For a calibration pair under this target-invariance assumption,

xi(a)−xi(0)=UCδi(a).x_i^{(a)}-x_i^{(0)}=U_C\delta_i^{(a)}.

Therefore

x~i(a)(α)−x~i(α)=(I−αT^)UCδi(a)=UCDαδi(a)+α(T−T^)UCδi(a).\begin{aligned} \widetilde{x}_i^{(a)}(\alpha)-\widetilde{x}_i(\alpha) &=(I-\alpha\widehat{T})U_C\delta_i^{(a)}\\ &=U_CD_\alpha\delta_i^{(a)}+\alpha(T-\widehat{T})U_C\delta_i^{(a)}. \end{aligned}

The same norm bound gives

∥x~i(a)(α)−x~i(α)∥2≤((1−ακ)+αη)∥δi(a)∥2.\lVert\widetilde{x}_i^{(a)}(\alpha)-\widetilde{x}_i(\alpha)\rVert_2 \leq\left((1-\alpha\kappa)+\alpha\eta\right)\lVert\delta_i^{(a)}\rVert_2.

For the noise claim, LCL_C and Ω\Omega are positive semidefinite, so λmax⁡(LC+Ω)≥λmax⁡(LC)=λ1>0\lambda_{\max}(L_C+\Omega)\geq\lambda_{\max}(L_C)=\lambda_1>0. Apply eq. (7) with G‾=G=LC+Ω\overline{G}=G=L_C+\Omega, ρerr=∥Ω∥op\rho_{\mathrm{err}}=\lVert\Omega\rVert_{\mathrm{op}}, and ϵM=0\epsilon_M=0. The same result supplies η\eta with an additional sampling term when the moment is empirical.

The theorem captures the attenuation tradeoff. If TT is estimated accurately, increasing α\alpha reduces both the remaining nuisance component and the original–rewrite score change, and each nuisance direction is scaled by a factor in [1−α,1−ακ][1-\alpha,1-\alpha\kappa] according to how strongly the rewrites move it. If TT is inaccurate, larger α\alpha can also remove target-relevant variation, which is why α\alpha is selected on validation rather than fixed at one. When AA is a multiple of the identity, T=PUCT=P_{U_C} is the projector onto the nuisance subspace, κ=1\kappa=1, and the operator becomes projector-based attenuation; it removes the nuisance component completely when α=1\alpha=1 and T^=T\widehat{T}=T.

Perturbation of the generalized eigenvector

We study the leading generalized eigenvector of the calibrated-output covariance and intervention penalty. Write Σ=Σα⪰0\Sigma=\Sigma_\alpha\succeq0 and B=G‾+ρI≻0B=\overline{G}+\rho I\succ0. The weight vector solves

Σw=μBw.(10)\Sigma w=\mu Bw. \tag*{(10)}

The result below compares unit-normalized weights up to relative sign in the norm ∥v∥B=(vTBv)1/2\lVert v\rVert_B=(v^{\mathsf{T}}Bv)^{1/2}. The implementation’s orientation and normalization require the additional local conditions stated afterward.

Theorem 4 (Generalized eigenvector perturbation). Let Σ,Σ^⪰0\Sigma, \widehat{\Sigma} \succeq0 and B,B^≻0B, \widehat{B} \succ0 be p×pp \times p matrices, with p≥2p \ge2 and B⪰ρIB \succeq\rho I for ρ>0\rho> 0. Suppose

∥Σ^−Σ∥op≤ϵΣ,∥B^−B∥op≤ϵB<ρ.\lVert\widehat{\Sigma}-\Sigma\rVert_{\mathrm{op}} \le\epsilon_{\Sigma}, \qquad\lVert\widehat{B}-B\rVert_{\mathrm{op}} \le\epsilon_{B}<\rho.

Let μ1\mu_1 and Δ>0\Delta> 0 be the largest generalized eigenvalue and the gap to the second largest eigenvalue of (Σ,B)(\Sigma,B). Define

τ:=ϵB/ρ,ζ:=ϵΣ+μ1ϵBρ−ϵB.\tau:= \epsilon_B/\rho, \qquad\zeta:= \frac{\epsilon_\Sigma+\mu_1\epsilon_B}{\rho-\epsilon_B}.

If Δ>2ζ\Delta> 2\zeta, then leading generalized eigenvectors w,w^w,\widehat{w} normalized by ∥w∥B=∥w^∥B^=1\lVert w\rVert_B=\lVert\widehat{w}\rVert_{\widehat{B}}=1 satisfy

min⁡s∈{−1,1}∥sw^−w∥B≤22 ζΔ1−τ+11−τ−1.(11)\min_{s\in\{-1,1\}}\lVert s\widehat{w}-w\rVert_B \le\frac{2\sqrt{2}\,\zeta}{\Delta\sqrt{1-\tau}}+\frac{1}{\sqrt{1-\tau}}-1. \tag*{(11)}

Proof. Use the same reference coordinates for both pencils. Set

M=B−1/2ΣB−1/2,M=B^{-1/2}\Sigma B^{-1/2},
E=B−1/2(B^−B)B−1/2,E=B^{-1/2}(\widehat{B}-B)B^{-1/2},
F=B−1/2(Σ^−Σ)B−1/2,F=B^{-1/2}(\widehat{\Sigma}-\Sigma)B^{-1/2},
S=(I+E)−1/2.S=(I+E)^{-1/2}.

Then ∥E∥op≤τ<1\lVert E\rVert_{\mathrm{op}}\le\tau<1, ∥F∥op≤ϵΣ/ρ\lVert F\rVert_{\mathrm{op}}\le\epsilon_\Sigma/\rho, and ∥M∥op=μ1\lVert M\rVert_{\mathrm{op}}=\mu_1. The eigenvalues of SS lie between (1+τ)−1/2(1+\tau)^{-1/2} and a:=(1−τ)−1/2a:=(1-\tau)^{-1/2}, so ∥S∥op≤a\lVert S\rVert_{\mathrm{op}}\le a and ∥S−I∥op≤a−1\lVert S-I\rVert_{\mathrm{op}}\le a-1. The second inequality uses 1−(1+τ)−1/2≤(1−τ)−1/2−11-(1+\tau)^{-1/2}\le(1-\tau)^{-1/2}-1.

The transformed estimated pencil is the symmetric matrix N=S(M+F)SN=S(M+F)S. Its eigenvalues are those of (Σ^,B^)(\widehat{\Sigma},\widehat{B}). Expanding the difference gives

N−M=SFS+(S−I)MS+M(S−I),N-M=SFS+(S-I)MS+M(S-I),

and hence

∥N−M∥op≤a2ϵΣ/ρ+μ1(a−1)(a+1)=ζ.\lVert N-M\rVert_{\mathrm{op}}\le a^2\epsilon_\Sigma/\rho+\mu_1(a-1)(a+1) =\zeta.

Let y=B1/2wy=B^{1/2}w and z=(I+E)1/2B1/2w^z=(I+E)^{1/2}B^{1/2}\widehat{w}. These are unit leading eigenvectors of MM and NN, respectively. Weyl’s inequality and Δ>2ζ\Delta>2\zeta ensure that the estimated leading eigenvalue is simple. The single-vector Davis–Kahan bound gives a relative sign ss such that ∥sz−y∥2≤22 ζ/Δ\lVert sz-y\rVert_2\le2\sqrt{2}\,\zeta/\Delta (Yu et al., 2015).

Mapping back now retains the change in the metric:

B1/2(sw^−w)=S(sz−y)+(S−I)y.B^{1/2}(s\widehat{w}-w)=S(sz-y)+(S-I)y.

Taking norms and using ∥y∥2=1\lVert y\rVert_2=1 gives ∥sw^−w∥B≤a 22 ζ/Δ+(a−1)\lVert s\widehat{w}-w\rVert_B\le a\,2\sqrt{2}\,\zeta/\Delta+(a-1), which is eq. (11). □\square

Ridge perturbation. The bound controls the entire denominator matrix. Let G‾′\overline{G}' denote a second empirical moment. With a fixed ridge, ϵB\epsilon_B may be taken as ∥G‾′−G‾∥op\lVert\overline{G}'-\overline{G}\rVert_{\mathrm{op}}. For the implemented rule ρ=γ tr⁡(G‾)/p\rho=\gamma\,\operatorname{tr}(\overline{G})/p with fixed γ>0\gamma>0, recomputing the ridge gives

∥B^−B∥op≤(1+γ)∥G‾′−G‾∥op.\lVert\widehat{B}-B\rVert_{\mathrm{op}}\le(1+\gamma)\lVert\overline{G}'-\overline{G}\rVert_{\mathrm{op}}.

This follows from ∣tr⁡(H)∣≤p∥H∥op|\operatorname{tr}(H)|\le p\lVert H\rVert_{\mathrm{op}} for symmetric HH. The result compares a fixed estimator configuration and family set; it does not bound changes caused by discrete validation selection.

Transfer to reported scores. For a fixed calibrated-output matrix X~\widetilde{X} and relatively aligned, unit-normalized weights, the bound gives

∥X~(sw^−w)∥2≤∥X~∥opρ−1/2∥sw^−w∥B.\left\|\widetilde{X}(s\widehat{w}-w)\right\|_{2}\le\left\|\widetilde{X}\right\|_{\mathrm{op}}\rho^{-1/2}\left\|s\widehat{w}-w\right\|_{B}.

The implementation instead orients the weights to a positive coordinate sum and rescales them to sum to one (Appendix § 11.§ 2). This map is not continuous where the coordinate sum is zero.

A local transfer is available when relatively aligned weights u,vu,v have the same orientation. Write the normalization as N(v)=v/d(v)\mathcal{N}(v)=v/d(v) with d(v)=1Tvd(v)=\mathbf{1}^{\mathsf{T}}v. If d(u),d(v)≥b>0d(u),d(v)\ge b>0 and ∥u∥2≤R\left\|u\right\|_{2}\le R, then

∥N(v)−N(u)∥2≤(1b+Rpb2)∥v−u∥2.\left\|\mathcal{N}(v)-\mathcal{N}(u)\right\|_{2}\le\left(\frac{1}{b}+\frac{R\sqrt{p}}{b^{2}}\right)\left\|v-u\right\|_{2}.

Indeed, ∣d(v)−d(u)∣≤p∥v−u∥2\left|d(v)-d(u)\right|\le\sqrt{p}\left\|v-u\right\|_{2}, and

vd(v)−ud(u)=v−ud(v)+u(d(u)−d(v))d(u)d(v).\frac{v}{d(v)}-\frac{u}{d(u)}=\frac{v-u}{d(v)}+\frac{u\left(d(u)-d(v)\right)}{d(u)d(v)}.

A strict margin from the coordinate-sum sign boundary ensures the local orientation condition and the lower bound bb. Put Lb,R=1/b+Rp/b2L_{b,R}=1/b+R\sqrt{p}/b^{2}. If recalibration also changes X~\widetilde{X} to X~′\widetilde{X}', the full score change satisfies

∥X~′N(v)−X~N(u)∥2≤∥X~∥opLb,R∥v−u∥2+∥X~′−X~∥op∥N(v)∥2.\left\|\widetilde{X}'\mathcal{N}(v)-\widetilde{X}\mathcal{N}(u)\right\|_{2} \le\left\|\widetilde{X}\right\|_{\mathrm{op}}L_{b,R}\left\|v-u\right\|_{2} +\left\|\widetilde{X}'-\widetilde{X}\right\|_{\mathrm{op}}\left\|\mathcal{N}(v)\right\|_{2}.

This follows by adding and subtracting X~N(v)\widetilde{X}\mathcal{N}(v). For fixed original outputs XX and α∈[0,1]\alpha\in[0,1], the two calibration operators give X~=X(I−αTb)\widetilde{X}=X(I-\alpha T_b) and X~′=X(I−αTb′)\widetilde{X}'=X(I-\alpha T_{b'}), so

∥X~′−X~∥op≤α∥X∥op∥Tb′−Tb∥op.\left\|\widetilde{X}'-\widetilde{X}\right\|_{\mathrm{op}}\le\alpha\left\|X\right\|_{\mathrm{op}}\left\|T_{b'}-T_b\right\|_{\mathrm{op}}.

For the same n≥1n\ge1 evaluation items and targets, the absolute MAE change is at most n−1/2n^{-1/2} times the full score-vector bound. Thus both changing weights and changing calibration contribute. Without the orientation margin, the unit-normalized eigenvector bound does not imply stability of the reported scores.

Target and nuisance blocks. Consider the population orthogonal model of Theorem 3 with \overline{G}=L_C=U_CA_U_C^{\mathsf{T}}, so T^=T\widehat{T}=T. This assumes the unnormalized moment is exact; equality of the normalized operators alone does not fix its scale. Both matrices of the pencil have a block form. Writing ΣQQ=Cov⁡(qi)\Sigma_{QQ}=\operatorname{Cov}(q_i), ΣQC=Cov⁡(qi,ci)\Sigma_{QC}=\operatorname{Cov}(q_i,c_i), ΣCC=Cov⁡(ci)\Sigma_{CC}=\operatorname{Cov}(c_i), and Dα=I−αA/λ1D_\alpha=I-\alpha A/\lambda_1,

Σα=UQΣQQUQT+UQΣQCDαUCT+UCDαΣQCUQT+UCDαΣCCDαUCT,\Sigma_\alpha=U_Q\Sigma_{QQ}U_Q^{\mathsf{T}}+U_Q\Sigma_{QC}D_\alpha U_C^{\mathsf{T}} +U_CD_\alpha\Sigma_{QC}U_Q^{\mathsf{T}}+U_CD_\alpha\Sigma_{CC}D_\alpha U_C^{\mathsf{T}},

so attenuation leaves the target block unchanged and shrinks the nuisance block by DαD_\alpha on both sides, while B=G‾+ρIB=\overline{G}+\rho I equals ρI\rho I on span⁡(UQ)\operatorname{span}(U_Q) and UC(A+ρI)UCTU_C(A+\rho I)U_C^{\mathsf{T}} on span⁡(UC)\operatorname{span}(U_C). The generalized eigenvector therefore weights target variance against ρ\rho alone and nuisance variance against A+ρIA+\rho I: a direction that the rewrites move strongly is penalized in the denominator in addition to being attenuated in the numerator.

Pooled second moment across families

Proposition 1 already controls the pooled moment used by VoirDire. The next corollary gives its additional subspace-recovery consequence when a fixed set of families perturbs a common nuisance subspace, possibly with different strengths and systematic shifts.

Corollary 2 (Pooled second moment). Fix a nonempty family set A\mathcal{A} and positive calibration counts mam_a, with M=∑a∈AmaM=\sum_{a\in\mathcal{A}}m_a and πa=ma/M\pi_a=m_a/M. Suppose each family satisfies

E[zi(a)zi(a)T]=UCAaUCT+Ωa,a∈A.\mathbb{E}\left[z_i^{(a)}z_i^{(a)\mathsf{T}}\right]=U_CA_aU_C^{\mathsf{T}}+\Omega_a,\qquad a\in\mathcal{A}.

Define the population pooled moment and its signal/noise decomposition by

Gpool=∑a∈AπaE[zi(a)zi(a)⊤],G_{\mathrm{pool}} = \sum_{a \in\mathcal{A}} \pi_a \mathbb{E}\left[z_i^{(a)} z_i^{(a)\top}\right],
Lpool=UC(∑a∈AπaAa)UC⊤,L_{\mathrm{pool}} = U_C\left(\sum_{a \in\mathcal{A}} \pi_a A_a\right)U_C^\top,
Ωpool=∑a∈AπaΩa.\Omega_{\mathrm{pool}} = \sum_{a \in\mathcal{A}} \pi_a \Omega_a.

If LpoolL_{\mathrm{pool}} has rank dCd_C, where 1≤dC≤p1 \le d_C \le p,

λpool:=λdC(Lpool),\lambda_{\mathrm{pool}} := \lambda_{d_C}(L_{\mathrm{pool}}),
ρpool:=∥Ωpool∥op<λpool/2,\rho_{\mathrm{pool}} := \lVert\Omega_{\mathrm{pool}}\rVert_{\mathrm{op}} < \lambda_{\mathrm{pool}}/2,

then the top-dCd_C eigenspace CpoolC_{\mathrm{pool}} of GpoolG_{\mathrm{pool}} satisfies

∥PCpool−PUC∥op≤2ρpoolλpool.\lVert P_{C_{\mathrm{pool}}} - P_{U_C}\rVert_{\mathrm{op}} \le\frac{2\rho_{\mathrm{pool}}}{\lambda_{\mathrm{pool}}}.

Proof. The pooled estimator Gˉ=1M∑i=1Mzizi⊤\bar{G} = \frac{1}{M}\sum_{i=1}^{M}z_i z_i^\top gives every calibration pair equal weight, so its expectation is

Gpool=Lpool+Ωpool.G_{\mathrm{pool}} = L_{\mathrm{pool}} + \Omega_{\mathrm{pool}}.

The signal matrix LpoolL_{\mathrm{pool}} has range span⁡(UC)\operatorname{span}(U_C). If dC=pd_C = p, both projectors equal II. For dC<pd_C < p, the signal has eigengap λpool\lambda_{\mathrm{pool}} between its dCd_C-th and (dC+1)(d_C + 1)-st eigenvalues. Applying the Davis–Kahan argument from Corollary 1 yields

∥PCpool−PUC∥op≤ρpoolλpool−ρpool≤2ρpoolλpool.\lVert P_{C_{\mathrm{pool}}} - P_{U_C}\rVert_{\mathrm{op}} \le\frac{\rho_{\mathrm{pool}}}{\lambda_{\mathrm{pool}}-\rho_{\mathrm{pool}}} \le\frac{2\rho_{\mathrm{pool}}}{\lambda_{\mathrm{pool}}}.

This is the population analogue of the pooled VoirDire estimator: the algorithm stacks the paired shifts from all selected families, estimates one second moment, and uses it to form T^\widehat{T} and the penalty in eq. (3). The family means μa\mu_a enter only through Aa=Cov⁡(δi(a))+μaμa⊤A_a = \operatorname{Cov}(\delta_i^{(a)}) + \mu_a\mu_a^\top, so a family whose shift is mostly systematic contributes a rank-one term inside span⁡(UC)\operatorname{span}(U_C). Pooling strengthens nuisance-subspace recovery when the added families increase the nuisance signal relative to perturbation. The benefit depends on the strength and diversity of the intervention-induced shifts.

Synthetic Experiments

We use synthetic data with known target and nuisance components to examine how intervention errors, attenuation strength, and systematic score shifts affect calibration (Appendices § 15.§ 1 to § 15.§ 3). We then examine how nuisance-subspace recovery relates to aggregation performance (Appendix § 15.§ 4).

Common setup. Unless otherwise stated, we use p=24p = 24 judges and two-dimensional target and nuisance factors (dQ=dC=2d_Q = d_C = 2). The combined loading matrix [UQ,UC][U_Q, U_C] has orthonormal columns. For experiments involving random sampling, we report means over 20 random seeds, with shaded bands indicating one standard error.

Sensitivity to Imperfect Interventions

We examine how unintended changes to target quality affect recovery of nuisance-sensitive directions.

Setup. Following Proposition 2, we generate paired shifts as

z=UCδ+UQr+ξ,δ=(0.6−0.3)+diag⁡(2,1)g,r=bQaQ(g+(0.2−0.1)),z = U_C\delta+ U_Qr + \xi,\qquad\delta= \begin{pmatrix} 0.6 \\ -0.3 \end{pmatrix} + \operatorname{diag}(2,1)g,\qquad r = b_Qa_Q\left(g+\begin{pmatrix} 0.2 \\ -0.1 \end{pmatrix}\right),

where g∼N(0,I2)g \sim N(0,I_2), bQ∼Bernoulli⁡(πQ)b_Q \sim\operatorname{Bernoulli}(\pi_Q), and ξ∼N(0,0.02Ip)\xi\sim N(0,0.02I_p) are independent. Here, πQ\pi_Q controls how often interventions change the target, and aQa_Q controls the magnitude of those changes. The shared gg makes target and nuisance changes correlated. We vary πQ\pi_Q from 0 to 1 with aQ=0.7a_Q=0.7, and aQa_Q from 0 to 2 with πQ=0.2\pi_Q=0.2. To isolate intervention error from sampling error, we compute G=E[zz⊤]G = \mathbb{E}[zz^\top] analytically. We measure recovery error by ∥P2(G)−PC∥op\lVert P_2(G)-P_C\rVert_{\mathrm{op}}, where P2(G)P_2(G) projects onto the two leading eigenvectors of GG, and PC=UCUC⊤P_C=U_CU_C^\top projects onto the true nuisance subspace.

Results. Figure 9 shows that nuisance-subspace recovery deteriorates as a larger fraction of interventions change the target, or as those changes grow in magnitude. This error persists even with exact population moments, implying collecting more intervention pairs cannot eliminate it.

Figure 9. Nuisance-subspace recovery deteriorates as unintended target changes become more frequent (left) or larger (right). Results use exact population moments, without sampling error.

Attenuation Strength and Target Preservation

We examine how attenuation strength affects nuisance suppression and target preservation when the estimated nuisance directions are inaccurate.

Setup. Let x=UQq+UCcx=U_Qq+U_Cc, with independent standard Gaussian q,cq,c, and define the true attenuation operator as T=UCdiag⁡(1,0.25)UC⊤T=U_C\operatorname{diag}(1,0.25)U_C^\top. We introduce estimation error by rotating the estimated nuisance directions toward the target subspace:

Vθ=cos⁡θ UC+sin⁡θ UQ,T^=Vθdiag⁡(1,0.25)Vθ⊤.V_\theta=\cos\theta\,U_C+\sin\theta\,U_Q,\qquad\widehat{T}=V_\theta\operatorname{diag}(1,0.25)V_\theta^\top.

with θ∈{0,5,15,30}\theta\in\{0,5,15,30\} degrees. The true target and nuisance subspaces remain orthogonal. For α∈{0,0.25,0.5,0.75,1}\alpha\in\{0,0.25,0.5,0.75,1\}, we measure

∥(I−αT^)UC∥op⏟remaining nuisance sensitivity,∥αT^UQ∥op⏟target distortion.\underbrace{\lVert(I-\alpha\widehat{T})U_C\rVert_{\mathrm{op}}}_{\text{remaining nuisance sensitivity}},\qquad\underbrace{\lVert\alpha\widehat{T}U_Q\rVert_{\mathrm{op}}}_{\text{target distortion}}.

These quantities measure the largest remaining nuisance response and the largest change to the target component over unit-norm factor vectors. We quantify operator error by η=∥T^−T∥op\eta=\lVert\widehat{T}-T\rVert_{\mathrm{op}}.

Results. Figure 10 shows that stronger attenuation reduces nuisance sensitivity while preserving the target when the operator is exact. With estimation error, stronger attenuation also increases target distortion, illustrating the tradeoff characterized in Theorem 3. Even with the exact operator and α=1\alpha=1, nuisance sensitivity remains at 0.75: attenuation removes the strongest nuisance direction but only partially suppresses the weaker one.

Figure 10. Increasing attenuation strength reduces nuisance sensitivity (left) but distorts target information when the estimated nuisance directions are inaccurate (right). Colors indicate operator error η\eta; both metrics are computed analytically.

Effect of Centering on Nuisance Recovery

VoirDire constructs its calibration matrix from paired score shifts without subtracting their mean (eq. (1)). Here, we illustrate why: a consistent shift across pairs can reveal nuisance sensitivity, and centering would remove this signal.

Setup. We model paired shifts as z=UC(g,a)T+ξz = U_C (g,a)^{\mathsf{T}} + \xi, with independent g∼N(0,1)g \sim N(0,1) and ξ∼N(0,0.0025Ip)\xi\sim N(0,0.0025I_p). The first nuisance component varies across pairs, while the second produces a constant shift of magnitude aa. Using m=128m = 128 pairs, we compare the uncentered second moment G^=m−1∑iziziT\widehat{G} = m^{-1}\sum_i z_i z_i^{\mathsf{T}} with the centered covariance C^=m−1∑i(zi−zˉ)(zi−zˉ)T\widehat{C} = m^{-1}\sum_i (z_i-\bar{z})(z_i-\bar{z})^{\mathsf{T}}. We vary aa from 0 to 4 and measure nuisance-subspace recovery using ∥P2(⋅)−PC∥op\lVert P_2(\cdot) - P_C\rVert_{\mathrm{op}}.

Results. Figure 11 shows that the centered estimator fails to recover the second nuisance direction, while the uncentered estimator becomes more accurate as the systematic shift grows. This supports retaining the mean shift: a consistent response across pairs can reveal nuisance sensitivity even without pair-to-pair variation.

Figure 11. Centering discards the systematic shift that identifies the second nuisance direction. The uncentered estimator recovers this direction as the shift grows. At a=0a = 0, neither population moment contains information about the second direction.

Nuisance Recovery and Aggregation Performance

We examine how nuisance-subspace recovery relates to aggregation performance as the calibration budget increases.

Setup. We generate judge outputs as xi=UQqi+UCci+ϵix_i = U_Qq_i + U_Cc_i + \epsilon_i, with independent target factors qi∼N(0,diag⁡(1,0.72))q_i \sim N(0, \operatorname{diag}(1, 0.7^2)), nuisance factors ci∼N(0,diag⁡(32,2.52))c_i \sim N(0, \operatorname{diag}(3^2, 2.5^2)), and Gaussian noise ϵi\epsilon_i correlated across judges, with scale 0.5. The target score is yi=0.8qi1+0.6qi2+νiy_i = 0.8q_{i1} + 0.6q_{i2} + \nu_i, where νi∼N(0,0.12)\nu_i \sim N(0, 0.1^2) is independent. We use 5,000 training and 2,000 test items. Two intervention families each add an N(2,0.42)N(2, 0.4^2) shift to a different nuisance coordinate, preserving qiq_i and resampling judge noise. Each family supplies m∈{4,8,16,32,64,128,256,512}m \in\{4, 8, 16, 32, 64, 128, 256, 512\} pairs, for 2m2m pairs in total.

VoirDire estimates aggregation weights from all training outputs and selects (α,γ)(\alpha, \gamma) separately for each seed and budget using MAE on a 20% validation subset and the grids in Appendix B.2. We compare against PCA, CARE-SVD, and an oracle reference that projects out the true nuisance subspace using I−PCI - P_C before fitting a two-component PCA aggregator. The oracle orients each eigenvector to have a positive coordinate sum, combines them with weights proportional to the square roots of their eigenvalues, and normalizes the resulting vector to sum to one. It knows the nuisance subspace but not the target coefficients. We report test MAE and nuisance-subspace recovery error ∥P2(G^)−PC∥op\lVert P_2(\widehat{G}) - P_C\rVert_{\mathrm{op}}.

Results. Figure 12 shows that better nuisance recovery accompanies lower aggregation error. As the budget increases from 4 to 512 pairs per family, VoirDire’s MAE falls from 0.942 to 0.582, approaching the oracle reference’s 0.567. Most of the aggregation gain occurs by 32–64 pairs per family, although subspace recovery continues to improve at larger budgets. These results show that VoirDire can improve aggregation even when the nuisance subspace is estimated imperfectly.

Figure 12. Increasing the calibration budget improves nuisance-subspace recovery (left) and aggregation performance (right). The oracle reference removes the true nuisance subspace before PCA aggregation. Budgets are per intervention family.

Additional Experimental Results

We provide further analyses of attenuation strength and judge-panel size (Appendices E.1 and E.2). We also report calibration-budget results for targeted nuisance removal (Appendix E.3) and compare the effects of score calibration across aggregators (Appendix E.4).

Sensitivity to Attenuation and Regularization

We study how attenuation strength α\alpha and regularization multiplier γ\gamma affect aggregation performance.

Setup. We use the datasets, splits, calibration pairs, and selected intervention families from § 5.1. We evaluate all 30 combinations of α∈{0,0.1,0.25,0.5,0.75,1}\alpha\in\{0, 0.1, 0.25, 0.5, 0.75, 1\} and γ∈{0.25,1,4,16,64}\gamma\in\{0.25, 1, 4, 16, 64\}, reporting MAE for continuous tasks and accuracy for binary/preference tasks. Table 10 reports sensitivity to γ\gamma at the validation-selected α\alpha; Table 11 reports the effect of removing each selected intervention family and refitting.

Datasetα\alphaγ=0.25\gamma=0.25141664
Continuous (MAE, lower is better)
Yelp10.6230.5500.5190.5100.508
SummEval00.6990.7010.7100.7240.733
Summarize from Feedback01.3461.3891.5071.6621.735
HelpSteer20.750.9510.9710.9890.9970.999
ASSET0.526.93026.95627.34227.61127.706
Review-5K0.252.5192.2562.0001.8931.862
Binary/preference (accuracy, higher is better)
Chatbot Arena0.50.6460.6500.6380.6000.581
RewardBench 210.6530.6580.6580.6570.658
LIAR20.10.6860.6770.6720.6710.669
HelpSteer30.50.6100.4950.4930.4910.490
CivilComments00.7390.7260.7170.7110.709
MT-Bench (human)0.10.6180.6240.6100.5950.582

Table 10. Sensitivity to the regularization multiplier γ\gamma at the selected α\alpha and family set of the main tables. Entries are mean evaluation metrics over the protocol seeds; the value selected on validation is in bold.

DatasetkkHelpMean Δ\DeltaMax lossMax gain
Continuous (Δ\Delta = MAE without the family − MAE with it)
Yelp440.0240.0720.003
SummEval430.0010.0020.000
Summarize from Feedback440.0090.0240.001
HelpSteer2330.0140.0230.005
ASSET520.0240.263-0.132
Review-5K430.0210.063-0.009
Binary/preference (Δ\Delta = accuracy without − accuracy with)
Chatbot Arena55-0.013-0.038-0.003
RewardBench 242-0.036-0.1510.007
LIAR222-0.002-0.003-0.001
HelpSteer366-0.026-0.070-0.004
CivilComments21-0.006-0.0200.007
MT-Bench (human)43-0.032-0.1240.001

Table 11. Leave-one-family-out refits of the selected family sets, (α,γ)(\alpha,\gamma) re-selected without the dropped family. kk: number of selected families. Help: families whose removal worsens the evaluation metric. Mean Δ\Delta: mean change over the kk refits. Max loss and max gain: the largest deterioration and the largest improvement obtained by dropping one family. Means over the protocol seeds.

Results. Figure 13 and Figure 14 show that the benefit of attenuation varies across datasets. Yelp benefits from strong attenuation, whereas Review-5K and ASSET deteriorate beyond moderate attenuation. On SummEval, Summarize from Feedback, and CivilComments, validation selects α=0\alpha= 0, retaining the original scores while using interventions to choose aggregation weights.

Figure 13. Sensitivity to α\alpha and γ\gamma on continuous-score datasets. Curves show mean evaluation MAE across five splits for each γ\gamma; lower is better. Stars mark validation-selected configurations.

Figure 14. Sensitivity to α\alpha and γ\gamma on binary/preference datasets. Curves show mean evaluation accuracy across five splits for each γ\gamma; higher is better. Stars mark validation-selected configurations.

The effect of α\alpha also depends on γ\gamma. On Chatbot Arena, attenuation provides larger gains at γ=64\gamma=64, where aggregation places less emphasis on intervention sensitivity. At γ=0.25\gamma=0.25, varying α\alpha changes accuracy by less than one percentage point. These results support tuning attenuation and regularization jointly.

Judge-Panel Size Ablation

We examine how the number of judges affects aggregation performance and the gains from intervention-based calibration.

Setup. We use the 12 datasets from § 5.1, varying K∈{2,4,6,8,11}K \in\{2, 4, 6, 8, 11\} for continuous tasks and K∈{2,4,8,12,16,20}K \in\{2, 4, 8, 12, 16, 20\} for binary/preference tasks. Twenty random orderings of the judges define nested subsets shared across methods. We keep the data splits, intervention families, and calibration pairs fixed, and refit and tune each method for each subset. We average performance over five splits and then across distinct subsets; bands show one standard deviation across subset means.

Results. Figure 15 and Figure 16 show that larger judge panels generally improve aggregation. From the smallest to the full panel, VoirDire improves on five of six continuous tasks and all six binary/preference tasks, with accuracy gains of 5.51–10.05 percentage points. The exception is Review-5K, where MAE rises from 1.598 to 1.862. Overall, larger panels tend to improve aggregation performance, although the benefits vary across methods and datasets.

Figure 15. MAE versus number of judges on continuous-score tasks; lower is better. Bands show ±1\pm1 standard deviation across subset means. Vertical dashed lines mark the main panel size.

Figure 16. Accuracy versus number of judges on binary/preference tasks; higher is better. Bands show ±1 standard deviation across subset means. The vertical dashed line marks the main panel size. CARE-Tensor is unavailable at K=2K = 2.

Calibration Budget for Targeted Nuisance Removal

We examine how the number of calibration pairs affects sensitivity to the beauty and authority biases studied in § 5.5.

Setup. Following the protocol in Appendix § 11.§ 6, we vary the budget from 4 to 64 pairs using nested calibration sets within each training fold. We keep the evaluation folds and (α,γ)=(1,1/4)(\alpha, \gamma) = (1, 1/4) fixed. We report mean absolute score shifts, averaging first over five folds and then over ten seeds. Variability is measured by the standard deviation across seed means. The main comparison uses the 32-pair setting.

Results. Figure 17 shows that increasing the number of calibration pairs improves suppression of the targeted nuisance. This is consistent with Theorem 2, which bounds the sampling error in the calibration matrix as the pair budget grows. With 32 pairs, VoirDire achieves lower sensitivity than all baselines on both beauty and authority (Table 3).

Figure 17. Sensitivity to injected biases versus calibration pairs, with (α,γ)=(1,1/4)(\alpha, \gamma) = (1, 1/4). Error bars show standard deviation across ten seed means. Baselines use no calibration pairs. Lower is better.

Effect of Calibration on Other Aggregators

We extend Section 5.2 to examine whether score calibration benefits other aggregation methods.

Setup. We follow Section 5.1, applying each aggregator to both raw and calibrated judge scores with the same selected interventions and calibration settings. MV and ISP receive calibrated scores mapped to the original score levels. The intervention-aware aggregation conditions follow § 5.2. When the selected α=0\alpha= 0, raw and calibrated scores coincide.

DatasetConditionMVAVGISPCARE-SVDPCAIntervention-aware
YelpRaw1.016±0.0021.016 \pm0.0021.089±0.0021.089 \pm0.0021.028±0.0021.028 \pm0.0020.694±0.0040.694 \pm0.0041.054±0.0021.054 \pm0.0021.034±0.0071.034 \pm0.007
Calibrated1.398±0.0061.398 \pm0.0061.605±0.1121.605 \pm0.1121.347±0.0901.347 \pm0.0907.098±8.4987.098 \pm8.4980.507±0.0160.507 \pm0.0160.508±0.0160.508 \pm0.016
Δ\Delta−0.381-0.381−0.516-0.516−0.318-0.318−6.404-6.404+0.547+0.547+0.526+0.526
SummEvalRaw0.819±0.0050.819 \pm0.0050.753±0.0040.753 \pm0.0040.832±0.0050.832 \pm0.0050.729±0.0050.729 \pm0.0050.738±0.0040.738 \pm0.0040.701±0.0050.701 \pm0.005
Calibrated0.819±0.0050.819 \pm0.0050.753±0.0040.753 \pm0.0040.832±0.0050.832 \pm0.0050.729±0.0050.729 \pm0.0050.738±0.0040.738 \pm0.0040.701±0.0050.701 \pm0.005
Δ\Delta+0.000+0.000+0.000+0.000+0.000+0.000+0.000+0.000+0.000+0.000+0.000+0.000
Summarize from FeedbackRaw1.434±0.0041.434 \pm0.0041.453±0.0031.453 \pm0.0031.585±0.0031.585 \pm0.0031.392±0.0141.392 \pm0.0141.764±0.0041.764 \pm0.0041.346±0.0041.346 \pm0.004
Calibrated1.434±0.0041.434 \pm0.0041.453±0.0031.453 \pm0.0031.585±0.0031.585 \pm0.0031.392±0.0141.392 \pm0.0141.764±0.0041.764 \pm0.0041.346±0.0041.346 \pm0.004
Δ\Delta+0.000+0.000+0.000+0.000+0.000+0.000+0.000+0.000+0.000+0.000+0.000+0.000
HelpSteer2Raw0.998±0.0110.998 \pm0.0110.981±0.0060.981 \pm0.0061.008±0.0121.008 \pm0.0120.998±0.0120.998 \pm0.0120.992±0.0060.992 \pm0.0060.974±0.0050.974 \pm0.005
Calibrated1.006±0.0591.006 \pm0.0591.039±0.0871.039 \pm0.0870.997±0.0370.997 \pm0.0370.968±0.0170.968 \pm0.0171.000±0.0481.000 \pm0.0480.951±0.0130.951 \pm0.013
Δ\Delta−0.008-0.008−0.057-0.057+0.010+0.010+0.030+0.030−0.009-0.009+0.023+0.023
ASSETRaw31.238±0.11831.238 \pm0.11833.701±0.07433.701 \pm0.07431.885±0.10831.885 \pm0.10827.629±0.15627.629 \pm0.15628.888±0.13228.888 \pm0.13227.740±0.47227.740 \pm0.472
Calibrated31.658±1.70431.658 \pm1.70431.872±0.14831.872 \pm0.14829.723±1.72529.723 \pm1.72527.804±0.74527.804 \pm0.74527.741±0.17727.741 \pm0.17726.930±0.34226.930 \pm0.342
Δ\Delta−0.420-0.420+1.829+1.829+2.162+2.162−0.175-0.175+1.147+1.147+0.809+0.809
Review-5KRaw2.581±0.0002.581 \pm0.0002.483±0.0002.483 \pm0.0002.585±0.0002.585 \pm0.0001.957±0.0181.957 \pm0.0182.101±0.0002.101 \pm0.0002.105±0.0012.105 \pm0.001
Calibrated2.408±0.1632.408 \pm0.1631.797±0.0761.797 \pm0.0762.339±0.1802.339 \pm0.1801.475±0.1481.475 \pm0.1481.852±0.0331.852 \pm0.0331.862±0.0351.862 \pm0.035
Δ\Delta+0.174+0.174+0.686+0.686+0.246+0.246+0.482+0.482+0.249+0.249+0.243+0.243

Table 12. Effect of score calibration on continuous-score aggregation. Entries show mean ±\pm standard deviation of MAE across five splits; lower is better. Δ=Raw−Calibrated\Delta= \mathrm{Raw} - \mathrm{Calibrated}, so positive values indicate lower calibrated MAE. The calibrated intervention-aware aggregator is VoirDire; its raw counterpart retains the intervention penalty. Fitting details appear in Appendix § 16.§ 4.

DatasetConditionMVAVGISPCARE-SVDCARE-TensorPCAIntervention-aware
Chatbot ArenaRaw0.517±0.0030.517 \pm0.0030.551±0.0040.551 \pm0.0040.571±0.0030.571 \pm0.0030.572±0.0160.572 \pm0.0160.564±0.0280.564 \pm0.0280.544±0.0020.544 \pm0.0020.649±0.0060.649 \pm0.006
Calibrated0.517±0.0030.517 \pm0.0030.570±0.0030.570 \pm0.0030.577±0.0020.577 \pm0.0020.612±0.0210.612 \pm0.0210.549±0.0320.549 \pm0.0320.534±0.0260.534 \pm0.0260.646±0.0110.646 \pm0.011
Δ\Delta+0.000+0.000+0.019+0.019+0.006+0.006+0.040+0.040−0.015-0.015−0.010-0.010−0.002-0.002
RewardBench 2Raw0.577±0.0030.577 \pm0.0030.596±0.0030.596 \pm0.0030.569±0.0030.569 \pm0.0030.578±0.0150.578 \pm0.0150.545±0.0360.545 \pm0.0360.595±0.0040.595 \pm0.0040.655±0.0130.655 \pm0.013
Calibrated0.565±0.0060.565 \pm0.0060.597±0.0050.597 \pm0.0050.590±0.0060.590 \pm0.0060.561±0.0660.561 \pm0.0660.618±0.0190.618 \pm0.0190.547±0.0110.547 \pm0.0110.658±0.0060.658 \pm0.006
Δ\Delta−0.012-0.012+0.001+0.001+0.021+0.021−0.017-0.017+0.073+0.073−0.048-0.048+0.003+0.003
LIAR2Raw0.678±0.0040.678 \pm0.0040.665±0.0040.665 \pm0.0040.678±0.0040.678 \pm0.0040.670±0.0070.670 \pm0.0070.659±0.0290.659 \pm0.0290.661±0.0040.661 \pm0.0040.686±0.0040.686 \pm0.004
Calibrated0.678±0.0040.678 \pm0.0040.669±0.0040.669 \pm0.0040.678±0.0040.678 \pm0.0040.684±0.0140.684 \pm0.0140.672±0.0130.672 \pm0.0130.679±0.0040.679 \pm0.0040.686±0.0050.686 \pm0.005
Δ\Delta+0.000+0.000+0.005+0.005+0.000+0.000+0.014+0.014+0.014+0.014+0.018+0.018−0.000-0.000
HelpSteer3Raw0.490±0.0010.490 \pm0.0010.517±0.0020.517 \pm0.0020.556±0.0070.556 \pm0.0070.592±0.0270.592 \pm0.0270.567±0.0090.567 \pm0.0090.520±0.0040.520 \pm0.0040.544±0.0030.544 \pm0.003
Calibrated0.490±0.0010.490 \pm0.0010.515±0.0040.515 \pm0.0040.556±0.0070.556 \pm0.0070.551±0.0240.551 \pm0.0240.562±0.0060.562 \pm0.0060.530±0.0170.530 \pm0.0170.610±0.0150.610 \pm0.015
Δ\Delta+0.000+0.000−0.002-0.002+0.000+0.000−0.041-0.041−0.004-0.004+0.010+0.010+0.066+0.066
CivilCommentsRaw0.702±0.0010.702 \pm0.0010.715±0.0020.715 \pm0.0020.716±0.0020.716 \pm0.0020.747±0.0030.747 \pm0.0030.735±0.0110.735 \pm0.0110.714±0.0010.714 \pm0.0010.739±0.0030.739 \pm0.003
Calibrated0.702±0.0010.702 \pm0.0010.715±0.0020.715 \pm0.0020.716±0.0020.716 \pm0.0020.747±0.0030.747 \pm0.0030.735±0.0110.735 \pm0.0110.714±0.0010.714 \pm0.0010.739±0.0030.739 \pm0.003
Δ\Delta+0.000+0.000+0.000+0.000+0.000+0.000+0.000+0.000+0.000+0.000+0.000+0.000+0.000+0.000
MT-Bench (human)Raw0.600±0.0040.600 \pm0.0040.613±0.0040.613 \pm0.0040.600±0.0040.600 \pm0.0040.594±0.0050.594 \pm0.0050.550±0.0170.550 \pm0.0170.599±0.0040.599 \pm0.0040.622±0.0100.622 \pm0.010
Calibrated0.600±0.0040.600 \pm0.0040.612±0.0040.612 \pm0.0040.600±0.0040.600 \pm0.0040.603±0.0190.603 \pm0.0190.548±0.0330.548 \pm0.0330.601±0.0040.601 \pm0.0040.624±0.0070.624 \pm0.007
Δ\Delta+0.000+0.000−0.001-0.001+0.000+0.000+0.009+0.009−0.002-0.002+0.002+0.002+0.003+0.003

Table 13. Effect of score calibration on binary/preference aggregation. Entries show mean ±\pm standard deviation of accuracy across five splits; higher is better. Δ=Calibrated−Raw\Delta= \mathrm{Calibrated} - \mathrm{Raw}, so positive values indicate improvement. Accuracy and differences are reported as proportions. The calibrated intervention-aware aggregator is VoirDire; its raw counterpart retains the intervention penalty. Fitting details appear in Appendix § 16.§ 4.

Results. Tables 12 and 13 show that calibration benefits depend on both the dataset and the aggregation rule. Every aggregator improves on Review-5K, whereas on Yelp, calibration improves PCA and intervention-aware aggregation but worsens MV, AVG, ISP, and CARE-SVD. For intervention-aware aggregation, calibration lowers MAE on all four continuous datasets with α>0\alpha> 0 and improves accuracy on three of the five corresponding binary/preference datasets. Thus, score calibration can benefit other aggregators, but is not a uniformly beneficial preprocessing step.

References

  1. [1]01.AI. Yi: Open foundation models by 01.ai. arXiv preprint arXiv:2403.04652, 2024.
  2. [2]Eldar David Abraham, Karel D’Oosterlinck, Amir Feder, Yair Ori Gat, Atticus Geiger, Christopher Potts, Roi Reichart, and Zhengxuan Wu. CEBaB: Estimating the causal effects of real-world concepts on NLP model behavior. In Advances in Neural Information Processing Systems, volume 35, 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/701ec28790b29a5bc33832b7bdc4c3b6-Abstract-Conference.html.
  3. [3]Dyah Adila, Changho Shin, Linrong Cai, and Frederic Sala. Zero-shot robustification of zero-shot models. In International Conference on Learning Representations, volume 2024, pp. 34402–34434, 2024.arxiv.org/abs/2309.04344
  4. [4]Rui Ai, Yugi Pan, David Simchi-Levi, Milind Tambe, and Haifeng Xu. Beyond majority voting: LLM aggregation by leveraging higher-order information. In Proceedings of the 43rd International Conference on Machine Learning, 2026.arxiv.org/abs/2510.01499
  5. [5]Alibaba. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024.
  6. [6]Alibaba. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
  7. [7]Fernando Alva-Manchego, Louis Martin, Antoine Bordes, Carolina Scarton, Benoît Sagot, and Lucia Specia. ASSET: A dataset for tuning and evaluation of sentence simplification models with multiple rewriting transformations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4668–4679. Association for Computational Linguistics, 2020. URL https://www.aclweb.org/anthology/2020.acl-main.424.
  8. [8]T. W. Anderson and Herman Rubin. Statistical inference in factor analysis. In Jerzy Neyman (ed.), Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, Volume 5: Contributions to Econometrics, Industrial Research, and Psychometry, pp. 111–150. University of California Press, Berkeley, 1956.
  9. [9]Krishnakumar Balasubramanian, Aleksandr Podkopaev, and Shiva Prasad Kasiviswanathan. Dependence-aware label aggregation for LLM-as-a-judge via Ising models, 2026. URL https://arxiv.org/abs/2601.22336.
  10. [10]Rajendra Bhatia. Matrix Analysis, volume 169 of Graduate Texts in Mathematics. Springer, New York, 1997. doi: 10.1007/978-1-4612-0653-8.DOI
  11. [11]Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Wasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion Proceedings of The 2019 World Wide Web Conference, pp. 491–500, 2019.arxiv.org/abs/1903.04561
  12. [12]Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or LLMs as the judge? A study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 8301–8327, Miami, Florida, USA, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.474.
  13. [13]Zeming Chen, Qiyue Gao, Antoine Bosselut, Ashish Sabharwal, and Kyle Richardson. DISCO: Distilling counterfactuals with large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pp. 5514–5528. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.acl-long.302. URL https://aclanthology.org/2023.acl-long.302/.DOI
  14. [14]Chandler Davis and W. M. Kahan. The rotation of eigenvectors by a perturbation. iii. SIAM Journal on Numerical Analysis, 7(1):1–46, 1970. doi: 10.1137/0707001.
  15. [15]A. P. Dawid and Allan M. Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society Series C: Applied Statistics, 28(1):20–28, 1979. doi: 10.2307/2346806.
  16. [16]Google DeepMind. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024.
  17. [17]Google DeepMind. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025.
  18. [18]Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409, 2021. doi: 10.1162/tacl_a_00373. URL https://aclanthology.org/2021.tacl-1.24/.DOI
  19. [19]Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. CausaLM: Causal model explanation through counterfactual language models. Computational Linguistics, 47(2):333–386, 2021. doi: 10.1162/coli_a_00404. URL https://aclanthology.org/2021.cl-2.13/.
  20. [20]Amir Feder, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E. Roberts, Brandon M. Stewart, Victor Veitch, and Diyi Yang. Causal inference in natural language processing: Estimation, prediction, interpretation and beyond. Transactions of the Association for Computational Linguistics, 10:1138–1158, 2022. doi: 10.1162/tacl_a_00511.
  21. [21]Amir Feder, Yoav Wald, Claudia Shi, Suchi Saria, and David Blei. Data augmentations for improved (large) language model generalization. In Advances in Neural Information Processing Systems, volume 36, 2023. URL https://openreview.net/forum?id=dJZ3MvDw86.
  22. [22]David R. Karger, Sewoong Oh, and Devavrat Shah. Iterative learning for reliable crowdsourcing systems. In Advances in Neural Information Processing Systems, volume 24, pp. 1953–1961, 2011. URL https://papers.nips.cc/paper/4396-iterative-learning-for-reliable-crowdsourcing-systems.
  23. [23]Divyansh Kaushik, Eduard Hovy, and Zachary C. Lipton. Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations, 2020.
  24. [24]Vladimir Koltchinskii and Karim Lounici. Concentration inequalities and moment bounds for sample covariance operators. Bernoulli, 23(1):110–133, 2017. doi: 10.3150/15-BEJ730.
  25. [25]Haitao Li, Junjie Chen, Qingyao Ai, Zhumin Chu, Yujia Zhou, Qian Dong, and Yiqun Liu. CalibraEval: Calibrating prediction distribution to mitigate selection bias in LLMs-as-judges. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16537–16552. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.acl-long.808. URL https://aclanthology.org/2025.acl-long.808/.
  26. [26]Ruosen Li, Teerth Patel, and Xinya Du. PRD: Peer rank and discussion improve large language model based evaluations. Transactions on Machine Learning Research, 2024a. URL https://openreview.net/forum?id=YVD1QwRAj.
  27. [27]Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. Split and merge: Aligning position biases in LLM-based evaluators. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11084–11108. Association for Computational Linguistics, 2024b. doi: 10.18653/v1/2024.emnlp-main.621. URL https://aclanthology.org/2024.emnlp-main.621/.DOI
  28. [28]Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2511–2522. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.153. URL https://aclanthology.org/2023.emnlp-main.153/.
  29. [29]Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannah Hajishirzi, and Nathan Lambert. RewardBench 2: Advancing reward model evaluation, 2025. URL https://arxiv.org/abs/2506.01937.
  30. [30]Meta. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
  31. [31]Microsoft. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743, 2025.
  32. [32]Mistral. Mistral 7b, 2023. URL https://arxiv.org/abs/2310.06825.
  33. [33]Junsoo Park, Seungyeon Jwa, Ren Meying, Daeyoung Kim, and Sanghyuk Choi. OffsetBias: Leveraging debiased data for tuning evaluators. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 1043–1067. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.findings-emnlp.57. URL https://aclanthology.org/2024.findings-emnlp.57/.DOI
  34. [34]Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, 2 edition, 2009.
  35. [35]Alexander Ratner, Christopher De Sa, Sen Wu, Daniel Selsam, and Christopher Re. Snorkel: Rapid training data creation with weak supervision. Proceedings of the VLDB Endowment, 11(3):269–282, 2017. URL https://www.vldb.org/pvldb/vol11/p269-ratner.pdf.
  36. [36]Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, and Soroush Vosoughi. Judging the judges: A systematic study of position bias in LLM-as-a-judge. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pp. 292–314. The Asian Federation of Natural Language Processing and The Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.ijcnlp-long.18. URL https://aclanthology.org/2025.ijcnlp-long.18/.DOI
  37. [37]Changho Shin, Winfred Li, Harit Vishwakarma, Nicholas Carl Roberts, and Frederic Sala. Universalizing weak supervision. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=YpPiNigTzMT.arxiv.org/abs/2112.03865
  38. [38]Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In Advances in Neural Information Processing Systems, 2020.arxiv.org/abs/2009.01325
  39. [39]Victor Veitch, Alexander D’Amour, Steve Yadlowsky, and Jacob Eisenstein. Counterfactual invariance to spurious correlations in text classification. In Advances in Neural Information Processing Systems, 2021.arxiv.org/abs/2106.00545
  40. [40]Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating LLM generations with a panel of diverse models. arXiv preprint arXiv:2404.18796, 2024. doi: 10.48550/arXiv.2404.18796. URL https://arxiv.org/abs/2404.18796.
  41. [41]Yidong Wang, Zhuohao Yu, Wenjin Yao, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. PandaLM: An automatic evaluation benchmark for LLM instruction tuning optimization. In International Conference on Learning Representations, 2024a. URL https://openreview.net/forum?id=5Nn2BLV7SB.arxiv.org/abs/2306.05087

Paper details

vlnc:W_kzdutyt8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude Code, Codex, Fable, ChatGPT Pro

View statement
Looking for related papers…
Contents