WorksComputer Science
VoirDire: Counterfactual Calibration for Reliable LLM-as-a-Judge Aggregation
Abstract
LLM-as-a-judge systems often aggregate multiple judges to improve evaluation reliability. However, judge agreement can reflect shared biases toward verbosity, formatting, or fluency rather than the intended evaluation target. Addressing these shared biases is challenging because observational judge outputs alone cannot distinguish agreement driven by target quality from agreement driven by nuisance attributes. We introduce VoirDire, an intervention-calibrated aggregation method for LLM-judge panels. VoirDire applies controlled interventions that vary specified surface attributes while aiming to preserve the evaluation target; it then uses paired changes in judge scores to identify nuisance-sensitive directions. It attenuates these directions and selects aggregation weights that retain variation across items while penalizing sensitivity to the interventions, making the final consensus less dependent on superficial features. We show theoretically that observational judge outputs alone cannot identify which shared direction is target-relevant, while paired interventions provide the contrast needed to identify nuisance-sensitive variation. We evaluate VoirDire on 12 datasets spanning continuous scoring, binary classification, and pairwise preference. Compared with the existing LLM-judge aggregation baselines, VoirDire reduces MAE by up to 26.8% on continuous-score tasks and improves accuracy by up to 5.3 percentage points on binary and preference tasks. These findings suggest that counterfactual interventions provide an additional means of improving the reliability and controllability of LLM-based evaluation beyond merely relying on observed judge agreement.
Introduction
LLM-as-a-judge systems are increasingly used to score open-ended answers, compare assistants, filter data, and approximate human evaluations [28, 41] (Zheng et al. 2023; Zhu et al. 2025). Because a single judge can be noisy, recent work has explored aggregating multiple LLM judges [40, 26, 9] (Li et al. 2026; Zhao et al. 2026). The premise is that agreement among judges can reveal the underlying quality signal by averaging out individual judge noise.
However, this premise breaks down when judges share the same nuisance sensitivity. LLM judges exhibit biases related to position, selection effects, and formatting artifacts [27, 33, 36, 25] (Zeng et al. 2024; Ye et al. 2025). When such biases are shared across judges, aggregation may reinforce rather than mitigate them. Judges can move together because an answer is genuinely better, but they can also move together because the answer is
Figure 1. Motivation. Surface attribute changes systematically shift judge scores, so aggregation can track nuisance factors rather than the evaluation target.
longer, better formatted, more fluent, or easier to parse. From observed scores alone, true quality and nuisance factors can be hard to distinguish.
Meanwhile, prior work has developed more sophisticated judge aggregation methods beyond majority voting, for example by estimating judge reliability, using higher-order agreement statistics, or modeling judge dependence [4, 9]. These tools improve aggregation by modeling how judge outputs interact with the latent quality signal. However, when shared nuisance factors induce score structure that resembles true quality, richer observational modeling can still amplify the wrong signal. More directly related to our work, Zhao et al. (2026) address shared nuisance factors by additionally introducing latent variables that capture their effects across judges. However, it still relies on observational judge outputs and structural assumptions such as sparsity and low rank. As a result, it does not provide a direct, controllable means of probing judges’ sensitivity to specific nuisance attributes.
We introduce VoirDire, an intervention-based calibration procedure for LLM-judge aggregation, named after voir dire, the legal process of examining prospective jurors. VoirDire applies controlled interventions to the inputs shown to judges, using deterministic transformations or LLM-generated rewrites to perturb specified nuisance attributes while aiming to preserve the evaluation target. The same judges then score both the original and modified inputs. Paired score differences indicate which directions in judge-output space are sensitive to the targeted attributes. VoirDire attenuates these directions and selects aggregation weights that retain variation across items while penalizing sensitivity to the interventions.
Theoretically, we show that observational agreement alone can fail to identify the target signal, while paired original–rewrite responses enable estimation of intervention-sensitive nuisance variation, with error controlled by intervention quality and calibration sample size.
Empirically, we evaluate VoirDire on 12 datasets spanning continuous scoring, binary classification, and pairwise preference. Compared with the strongest existing judge-aggregation baseline on each dataset, VoirDire reduces MAE by up to 26.8% on continuous-score tasks and improves accuracy by up to 5.3 percentage points on binary/preference tasks. Controlled experiments further show that VoirDire reduces aggregate sensitivity to targeted nuisance changes. We also examine the effect of calibration-pair budget and interpret how calibration changes judge scores and aggregation weights. In summary, our contributions include:
We introduce VoirDire, an intervention-based calibration method that uses controlled perturbations to mitigate shared nuisance sensitivity in LLM-judge aggregation.
We characterize the limitations of observational judge agreement and establish conditions under which paired interventions enable accurate estimation of nuisance-sensitive directions.
Across 12 datasets spanning continuous scoring, binary classification, and pairwise preference, VoirDire reduces MAE by up to 26.8% and improves accuracy by up to 5.3 percentage points over existing aggregation methods.
Related Work
LLM-as-a-judge evaluation. LLM judges are widely used to evaluate open-ended generation because they scale beyond manual annotation and can express task-specific rubrics [28, 41] (Zheng et al. 2023; Zhu et al. 2025). However, their judgments can depend on presentation, verbosity, position, selection effects, and other nuisance factors rather than only the intended evaluation target [27, 33, 36, 25] (Zeng et al. 2024; Ye et al. 2025). Aggregating multiple judges can reduce individual noise, but it does not necessarily eliminate shared bias.
Judge aggregation. Classical crowdsourcing and weak-supervision methods infer latent labels from multiple noisy sources by modeling source reliability, dependence, or latent structure [15, 22, 35, 37]. These approaches naturally extend to aggregating LLM judges. Recent LLM-jury methods use richer observational statistics, including higher-order agreement signals such as Inverse Surprising Popularity (ISP) and dependence-aware Ising models [4, 9]. However, these methods do not explicitly account for nuisance factors shared across judges. More directly related to our work, CARE (Zhao et al., 2026) models such shared factors using latent variables inferred from observed judge outputs. Still, its identification relies on observational outputs and structural assumptions such as sparsity and low rank. VoirDire instead uses controlled interventions to directly measure which judge-output directions respond to nuisance changes.
Counterfactual interventions and causal analysis. Causal inference distinguishes observational association from the effects revealed by controlled interventions [34]. In NLP, this view has been used to estimate the causal effects of textual concepts on model behavior [19, 2, 20] and to make models invariant to spurious attributes using counterfactual pairs [23, 39]; foundation models can now generate such counterfactuals at scale [13, 21] (Wu et al. 2021). VoirDire brings this causal view to judge aggregation.
VoirDire: Calibrating Judge Aggregation with Interventions
VoirDire calibrates LLM-judge aggregation using controlled interventions. It first constructs original–rewrite pairs designed to preserve the evaluation target while varying specified nuisance attributes, such as formatting, style, or verbosity. The same judges evaluate both inputs, and their paired score differences reveal nuisance-sensitive directions in judge-output space. VoirDire uses these directions to calibrate judge scores and determine intervention-aware aggregation weights. Figure 2 gives an overview, and algorithm 1 summarizes the procedure.
Figure 2. VoirDire framework. Controlled interventions give paired judge-score shifts, pooled into . At evaluation time, judge outputs are attenuated along its directions (eq. (2)) and weighted to retain item variation while penalizing sensitivity to the shifts (eq. (3)).
Setup. We consider a panel of LLM judges. For item , the judges see a judge input and produce a judge-output vector . Entries may be scalar scores, binary labels, or preference votes. A standard aggregator, such as majority voting, uses only the unperturbed judge-output matrix to predict a target score or label.
Constructing intervention pairs. We begin with a pool of candidate intervention families targeting nuisance attributes such as formatting, verbosity, and candidate position. Each family specifies three components: the nuisance attribute and input span to modify, a modification rule, and preservation constraints. The modification rule is either deterministic (e.g., adding markup or swapping two candidates verbatim) or an instruction to an LLM rewrite model. For example, a concision family instructs the model to remove redundancy while preserving the original claims, sentiment, numbers, and other content constraints.
In our experiments, we use LLM assistants to propose candidate intervention families based on dataset examples. We apply each intervention to a subset of inputs and retain only pairs that pass programmatic preservation checks. The same judges score each retained modified input, yielding for . Pair generation and preservation checking use no ground-truth labels. Full construction and selection details appear in Appendix B.
Estimating nuisance-sensitive directions. Given calibration pairs from each of the selected intervention families, we compute the paired score differences for . These vectors record how the judges respond to each intervention. When the evaluation target remains unchanged, they capture nuisance-sensitive variation together with judge noise. We pool the differences into a second-moment matrix1:
Each pair contributes equally to . Its entry captures how judges and respond together, and its leading eigenvectors identify the directions along which the interventions produce the largest score changes. These directions capture nuisance sensitivity, although judge noise and imperfections in the interventions can affect their estimation.
Attenuating sensitive directions. Using the estimated directions, we calibrate each judge-output vector by attenuating components along directions that respond more strongly to the interventions:
Here is the largest eigenvalue, and is a hyperparameter controlling attenuation strength. Normalizing by ensures that the most sensitive direction is attenuated by a fraction , while less sensitive directions are attenuated less.
Intervention-aware aggregation. After calibrating the judge outputs, we aggregate them using weights that retain variation across items while penalizing sensitivity to the interventions:
The numerator rewards variation across the calibrated items, while penalizes weights that are more sensitive to nuisance changes. The hyperparameter controls the tradeoff between retaining variation and reducing intervention sensitivity. We solve the resulting generalized eigenvalue problem, then orient and normalize the solution to obtain the final aggregation weights . The aggregate score is . Implementation details appear in Appendix B.2.
Theoretical Analysis
We theoretically analyze VoirDire, showing that observational judge agreement alone may not distinguish target quality from shared nuisance factors, whereas paired interventions reveal nuisance-sensitive variation and enable estimation of the calibration matrix.
Setup. We model judge outputs with target-relevant factors , nuisance factors , and noise:
Algorithm 1 VoirDire: calibration and intervention-aware aggregation
Require: Judge outputs ; paired outputs for selected intervention families, with and ; validation-selected hyperparameters .
Ensure: Aggregate scores .
Compute paired score differences for each family and item : .
Estimate the pooled second-moment matrix: . (1)
Attenuate nuisance-sensitive components for all items: , . (2)
Set and solve for intervention-aware weights: . (3)
return , .
where denotes the original judge-output vector, and denotes the outputs of the same judges after an intervention from family . The matrices and map target-relevant and nuisance factors to judge scores, respectively. We first consider an ideal setting in which interventions change nuisance factors by while leaving unchanged.
Why agreement alone is insufficient. Shared variation in judge scores need not reveal whether judges respond to target quality or nuisance attributes; Theorem 1 makes this explicit.
Theorem 1 (Observational non-identifiability; simplified). In the noiseless model , with orthonormal and , the target direction is not identifiable from the distribution of : rotating and within their span leaves this distribution unchanged.
Thus, even with unlimited observations, an aggregator cannot determine which direction represents quality in this model without additional information or assumptions.
What paired interventions reveal. Paired interventions address this ambiguity by holding target quality fixed while changing nuisance attributes. Under eq. (4), the target component cancels: , where is the paired judge noise. If this noise is mean-zero and independent of , with finite second moments, the pooled population second moment is , where captures intervention-induced nuisance variation and is the pooled noise covariance.
Proposition 1 (Calibration-matrix estimation; simplified). Suppose and . For nonzero , let and . Then
Proposition 1 shows that accurate calibration requires estimation and systematic errors to be small relative to the nuisance signal. We further analyze how calibration sample size affects estimation error (Theorem 2) and how imperfect interventions affect nuisance recovery. Full proofs and additional theoretical results appear in Appendix § 14, with supporting synthetic experiments in Appendix § 15.
Experiments
We evaluate VoirDire across continuous-score, binary, and preference LLM-judge aggregation setups. The experiments address the following topics:
Improving judge aggregation (Section 5.1): We compare VoirDire with existing aggregation methods across the three task types. § 5.1
| Dataset | MV | AVG | ISP | PCA | CARE-SVD | VoirDire |
| Yelp | 1.016 | 1.089 | 1.028 | 1.054 | 0.694 | 0.508 |
| SummEval | 0.819 | 0.753 | 0.832 | 0.738 | 0.729 | 0.701 |
| Summarize from Feedback | 1.434 | 1.453 | 1.585 | 1.764 | 1.392 | 1.346 |
| HelpSteer2 | 0.998 | 0.981 | 1.008 | 0.992 | 0.998 | 0.951 |
| ASSET | 31.238 | 33.701 | 31.885 | 28.888 | 27.629 | 26.930 |
| Review-5K | 2.581 | 2.483 | 2.585 | 2.101 | 1.957 | 1.862 |
Table 1. Continuous-score aggregation: mean MAE ().
| Dataset | MV | AVG | ISP | PCA | CARE-SVD | CARE-Tensor | VoirDire |
| Chatbot Arena | 0.517 | 0.551 | 0.571 | 0.544 | 0.572 | 0.596 | 0.646 |
| RewardBench 2 | 0.577 | 0.596 | 0.569 | 0.595 | 0.578 | 0.605 | 0.658 |
| LIAR2 | 0.678 | 0.665 | 0.678 | 0.661 | 0.670 | 0.668 | 0.686 |
| HelpSteer3 | 0.490 | 0.517 | 0.556 | 0.520 | 0.592 | 0.572 | 0.610 |
| CivilComments | 0.702 | 0.715 | 0.716 | 0.714 | 0.747 | 0.731 | 0.739 |
| MT-Bench (human) | 0.600 | 0.613 | 0.600 | 0.599 | 0.594 | 0.581 | 0.624 |
Table 2. Binary/preference aggregation: mean accuracy ().
Ablating score calibration and aggregation (Section 5.2): We examine how score calibration and intervention-aware aggregation individually contribute to performance.
Judge-level interpretation (Section 5.3): We examine how intervention sensitivity relates to judge reliability and aggregation weights.
Calibration efficiency (Section 5.4): We measure how performance changes with the number of intervention pairs.
Targeted nuisance removal (Section 5.5): We test whether VoirDire reduces score shifts under rewrites that trigger known judge biases.
Transfer across related nuisances (Section 5.6): We test whether calibration transfers to related nuisance variations not used during fitting.
Datasets and metrics. We evaluate on six continuous-score and six binary/preference datasets. We report mean absolute error (MAE; lower is better) for continuous scoring and accuracy (higher is better) for binary classification and pairwise preference tasks. Dataset details are in Appendix § 11.§ 1.
LLM judges. Continuous-score panels contain 11 open-weight LLM judges and binary/preference panels 20, with model sizes ranging from 0.6B to 14B parameters. Each judge reports on the dataset-native score scale. Following the judge-output convention in CARE (Zhao et al., 2026), judges first produce scalar scores for binary classification and pairwise preference tasks. These scores are then converted to binary labels or preferences using fixed thresholds and evaluated by accuracy. Details are in Appendices § 11.§ 3 and § 11.§ 4.
Interventions. We use LLM assistants to help identify task-specific nuisance attributes and design interventions, following Adila et al. [3]. The interventions vary formatting, verbosity, or candidate presentation while aiming to preserve the evaluation target. We generate modified inputs through deterministic transformations or rewrites by one of the judge models, Llama-3.1-8B-Instruct. Pair generation requires no ground-truth labels; family selection and tuning of use the validation set. Full specifications, prompts, and selection details appear in Appendix § 11.
Improving Aggregation with Interventions
Setup. We compare VoirDire with majority voting (MV), score averaging (AVG), PCA aggregation, CARE (Zhao et al., 2026), and Inverse Surprisal Popularity (ISP) [4]. We use CARE-SVD for continuous scoring and both CARE-SVD and CARE-Tensor for binary classification and pairwise preference. All methods use the same judges and evaluation items across five random splits per dataset. For VoirDire, we use intervention pairs per family. Protocol details and additional results appear in Appendices § 11 and § 16, respectively.
Results. Tables 1 and 2 show that VoirDire improves on existing aggregation methods across task types. The largest MAE reduction is on Yelp, at 26.8% relative to the strongest baseline, and the largest accuracy gains are on RewardBench 2 (5.3 pp) and Chatbot Arena (5.1 pp). These improvements suggest that intervention-based calibration can enhance judge aggregation compared with existing observational methods.
Ablating Score Calibration and Aggregation
VoirDire uses interventions for score calibration and intervention-aware aggregation. We examine the contribution of each step.
Setup. Following the setup of the previous subsection, we compare PCA and intervention-aware aggregation on raw and calibrated scores. The PCA baseline corresponds to and in eq. (3). For intervention-aware aggregation on raw scores, we set and re-tune on validation. Combining score calibration and intervention-aware aggregation gives VoirDire.
Results. Figure 3 shows that both steps contribute, with their relative benefits varying across datasets. On continuous-score tasks, score calibration provides substantial gains on datasets such as Yelp and ASSET. On binary/preference tasks, intervention-aware aggregation alone improves over PCA on all six datasets, while combining both steps provides further gains on datasets such as HelpSteer3. Additional ablations on other aggregators appear in Appendix § 16.§ 4.
Figure 3. Contributions of calibration and aggregation relative to raw PCA: relative MAE reduction (%) on continuous tasks and accuracy gain (percentage points) on binary/preference tasks. Higher is better. †: , so calibration leaves scores unchanged.
How Interventions Diagnose and Reweight Judges
Beyond improving aggregation, controlled interventions can help diagnose individual judges by revealing their sensitivity to nuisance attributes. We examine whether this sensitivity relates to judge reliability and how it influences the aggregation weights learned by VoirDire.
Setup. For each judge in the panels of § 5.1, we measure rewrite sensitivity as the root mean square score shift across calibration pairs, , normalized by the standard deviation of its original scores to account for differences in score scale. We measure reliability as the Spearman correlation between the judge’s original scores and ground truth, capturing how well its judgments align with the evaluation target. Comparing these metrics tests whether nuisance sensitivity, measured without ground-truth labels, is associated with judge reliability. We also examine the aggregation weights from eq. (3) to assess how VoirDire uses this sensitivity information.
Results. Figure 4 (a), (Figure 4b) show that judges with greater rewrite sensitivity tend to be less reliable, although the relationship varies across datasets. Panels (c) and (d) show that VoirDire generally assigns lower weights to more sensitive judges.
Figure 4. Rewrite sensitivity, reliability, and aggregation weights. (a, b) One point per judge and panel; legends report the within-panel Spearman correlation between sensitivity and reliability. (c, d) Weights divided by each panel’s largest absolute weight, with judges ranked by sensitivity (1: most sensitive; blue: positive, red: negative). Empty ranks indicate fewer judges.
Ablation on the Number of Intervention Pairs
We examine how the number of intervention pairs affects the performance of VoirDire.
Setup. We vary the number of calibration pairs per family , keeping the intervention families of § 5.1 and re-selecting on validation at every . We report improvements over each dataset’s strongest baseline, averaged separately across the six continuous-score datasets (relative MAE reduction) and the six binary/preference datasets.
Results. Figure 5 shows that VoirDire improves aggregation with a small calibration budget. Eight pairs per family recover most of the gain on continuous-score tasks, and gains on both task types level off between 32 and 64 pairs. With 32 pairs per family, VoirDire achieves an average MAE reduction of 7.4% and an accuracy gain of 2.2 percentage points over each dataset’s strongest baseline. Per-dataset results appear in Appendix § 11.7.
Figure 5. Mean gain over the strongest baseline. Bands: one SE over five splits; dotted line: .
Targeted Nuisance Removal
We further evaluate whether VoirDire reduces sensitivity to known nuisance changes in a controlled setting, directly testing whether it achieves its intended effect.
Setup. We use the judgment-bias benchmark of Chen et al. [12], which provides original–modified answer pairs with GPT-4 rewrites targeting rich formatting (beauty) or added references (authority), along with 1–10 judge scores. These rewrites can also rephrase or expand the original answers. We measure sensitivity as the mean absolute difference between aggregate scores on original and rewritten answers. Lower values indicate greater robustness to the targeted nuisance changes.
Results. VoirDire achieves the lowest sensitivity to both biases (Table 3), reducing mean absolute score shifts by approximately 27.3% for beauty and 18.2% for authority relative to CARE-SVD, the strongest baseline. Additional experimental details and results appear in Appendices B.6 and E.3.
| Type | Example | |
| Original | Night and day occur due to the Earth’s rotation on its axis. | |
| Beauty | Night and day occur due to the Earth’s rotation on its axis. When facing the Sun ☼, it experiences daylight. . . . | |
| Authority | . . . This rotation happens over a period of 24 hours, creating our cycle of day and night (National Geographic, 2019). | |
| Method | Beauty | Authority |
| MV | 0.919 | 0.824 |
| AVG | 0.506 | 0.325 |
| ISP | 0.891 | 0.733 |
| PCA | 1.283 | 0.876 |
| CARE-SVD | 0.418 | 0.286 |
| VoirDire | 0.304 | 0.234 |
Table 3. Examples of targeted nuisance rewrites (left) and mean absolute score shifts (right, ↓).
Transfer Across Related Nuisances
Identifying every nuisance variation can be costly. However, related rewrites may affect judge scores in similar ways, so we expect VoirDire to generalize beyond the specific interventions used for calibration. We test this in a controlled setting.
Setup. Following Section 5.5, we calibrate on three beauty variants (numbered structure, minimalist panels, and topic-specific icons) and three authority variants (expert attribution, a source header, and a fabricated footnote), then evaluate on the original beauty and authority rewrites. We report sensitivity reduction relative to PCA, normalized so that calibration on the original beauty or authority rewrites achieves 100%. Details appear in Appendix B.8.
Results. Table 4 shows that calibration retains much of its benefit on unseen rewrites. Notably, beauty variants transfer strongly even to authority, while authority variants also provide some benefit on beauty. This suggests that distinct nuisance attributes may affect overlapping directions in judge-output space, enabling transfer across them.
| Calibration | Evaluation | ||
| Concept | Variant | Beauty | Authority |
| Beauty | Numbered | 77.1% | 91.3% |
| Panels | 81.1% | 102.5% | |
| Icons | 72.0% | 100.3% | |
| Authority | Expert | 60.4% | 100.5% |
| Source header | 74.6% | 99.2% | |
| Footnote | 63.9% | 98.0% |
Table 4. Percentage of target-family gain retained (↑).
Conclusion
We introduced VoirDire, which uses controlled interventions to mitigate shared nuisance sensitivity in LLM-judge aggregation. Across 12 datasets, VoirDire improves aggregation performance and reduces sensitivity to targeted nuisance changes. Calibration also transfers to related, unseen variations, suggesting that exact rewrite matches are unnecessary. These findings highlight the potential of counterfactual interventions for more reliable LLM-based evaluation.
Limitations and future work. VoirDire depends on the quality of its interventions: they should capture relevant nuisance variation while preserving the evaluation target, which is difficult to guarantee. Discovering effective interventions also adds cost, and calibration requires additional inference to generate rewrites and score modified inputs, although deterministic transformations avoid the rewrite-generation cost. More broadly, paired interventions could extend beyond aggregation: agents could actively search for interventions that expose previously unknown judge biases, and the same paired signals could train judges and reward models to be invariant to surface features such as length and formatting, reducing reward hacking in downstream optimization.
ETHICS STATEMENT
The main risk is over-trust in calibrated automatic evaluation. VoirDire attenuates nuisance sensitivities exposed by the chosen interventions, but it does not guarantee unbiased judgments or remove all evaluation artifacts. For private or high-stakes evaluations, rewrite generation and judge calls should be run with appropriate data-protection safeguards, since the method sends original and rewritten inputs to LLMs.
AI USE STATEMENT
We used AI coding and writing assistants (Claude Code, Codex, Fable, and ChatGPT Pro) throughout this project. They assisted with implementing and running experimental, evaluation, and plotting code; proposing nuisance attributes and intervention designs; developing and checking the theoretical analysis; and drafting and editing the manuscript. The authors directed all AI-assisted work, reviewed its outputs, and take full responsibility for the content.
Appendix Overview
This appendix starts with a glossary of notation in Appendix A. It then provides experimental details in Appendix B, including datasets, experiment protocols, compute resources, judge prompts, and intervention specifications. Next, Appendix C contains proofs and additional theoretical results. We then present synthetic experiments that study calibration behavior in controlled settings (Appendix D). Finally, Appendix E reports additional experimental results.
Glossary
The notation used throughout the paper is summarized in table 5.
| Symbol | Definition |
| Item, judge, and intervention-family indices. | |
| Numbers of original items, judges, and selected families. | |
| Pairs per family and total budget: , or for unequal budgets. | |
| Attenuation strength, regularization multiplier, and ridge parameter . | |
| Judge input for item . | |
| Original judge-output vector; . | |
| Matrix of original judge outputs. | |
| Judge-output vector after intervention . | |
| Calibration item indices for family . | |
| Paired score difference . | |
| Empirical second moment of pooled paired differences (eq. (1)). | |
| Population second moments of paired differences and intervention-induced nuisance variation, respectively. | |
| Ideal and empirical normalized operators: , . | |
| Operator-estimation error bound: . | |
| Calibrated output . | |
| Empirical covariance of calibrated outputs. | |
| Oriented and normalized aggregation weights (eq. (3)). | |
| Aggregate score . | |
| Target factors, nuisance factors, and intervention-induced changes in nuisance factors. | |
| Loading matrices for target and nuisance factors. | |
| Nuisance-subspace dimension. | |
| Leading eigenvectors of and orthogonal projector onto . | |
| Judge noise and paired noise . | |
| Family-specific second moment . | |
| Nuisance-shift second moment and paired-noise covariance. | |
| Intervention-induced nuisance strength and paired-noise scale. | |
| Bounds on and , respectively. |
Table 5. Glossary of variables, symbols, and abbreviations.
Experimental Details
This section provides the experimental details for the main results, including datasets, judge-output protocols, judge prompts, intervention specifications, and implementation details.
Datasets
We evaluate six continuous-score panels (Yelp, SummEval coherence, Summarize from Feedback, HelpSteer2, ASSET, Review-5K) and six binary/preference panels (Chatbot Arena, RewardBench 2, HelpSteer3, MT-Bench with human labels, LIAR2, CivilComments). In table 1 the row labeled SummEval is the coherence aspect.
For continuous-score tasks, the target is the human-provided score and we report MAE. For binary/preference tasks, the target is the human-provided label or preference and we report accuracy.
Review-5K (Weng et al., 2025). Review-5K contains paper texts, reviewing guidelines, review records, and numeric reviewer ratings. We use the paper text and reviewing guidelines as the judge input; the reference review text is not shown to the judges. The target field is avg_rate, the average available reviewer rating on the 1–10 review scale. The dataset is available as WestlakeNLP/Review-5K.
Yelp (Zhang et al., 2015). Yelp consists of user-written business reviews paired with star ratings. We use a 5,000-example subset and treat the task as continuous rating prediction: given the review text, the goal is to recover the associated 1–5 star rating. The source dataset is available as Yelp/yelp_review_full.
ASSET [7]. ASSET is a sentence-simplification rating dataset. Each row contains a source sentence, a candidate simplification, an evaluation aspect, and a human rating. We use the human_rating field as the continuous target on the 0–100 scale. We use 4,500 rating rows. The dataset is available as facebook/asset.
HelpSteer2 [41]. HelpSteer2 contains prompts, model responses, and human ratings of five response attributes. We use the prompt and candidate response as the judge input and the helpfulness rating as the continuous target on the 0–4 scale, stored in the score field. We use the released validation split. The dataset is available as nvidia/HelpSteer2.
Summarize from Feedback [38]. Summarize from Feedback contains source posts, candidate summaries, and human assessments of summary quality. We use its axis subset, with the source post and candidate summary as the judge input. The target is the overall quality rating on the 1–7 scale, stored as overall_score from summary.axes.overall. We pool the released validation and test splits before applying our evaluation protocol. The dataset is available as openai/summarize_from_feedback.
SummEval [18]. SummEval contains model-generated summaries of CNN/Daily-Mail news articles with human ratings of coherence, consistency, fluency, and relevance. We use the coherence panel: 1,600 summaries from 16 systems for 100 articles. The judge input is the article and candidate summary; the target is the mean expert coherence rating on the 1–5 scale, stored in the label field. The dataset is available from Yale-LILY/SummEval.
Chatbot Arena (Zheng et al., 2023). Chatbot Arena contains pairwise comparisons between two model responses to the same user prompt. We use it as a preference task: the target field is gold_label_binary, which encodes the human preference over candidates A and B. We use 5,000 sampled comparisons, excluding ties. The dataset is available as lmsys/chatbot_arena_conversations.
CivilComments [11]. CivilComments consists of online comments annotated for toxicity. We use a 5,000-example subset, sampled equally from the two classes in the test split, as a binary toxicity-classification task, where the input is the comment text and the target indicates whether the comment is toxic. The binary target is obtained from the dataset’s continuous toxicity score, with scores of at least 0.5 labeled toxic (1) and the rest non-toxic (0). The source dataset is available as google/civil_comments.
LIAR2 (Xu & Kechadi, 2024). LIAR2 contains short statements assessed by professional fact-checkers at PolitiFact. We use the 2,296 statements in the released test split, with only the statement text as the judge input. We collapse the six veracity classes into a binary target: pants-on-fire, false, and barely-true map to 0; half-true, mostly-true, and true map to 1. The target is stored in the label field. The dataset is available as chengxuphd/liar2.
MT-Bench (Zheng et al., 2023). MT-Bench provides human comparisons of model responses to questions spanning multiple task categories. We use the 3,355-comparison human panel from allenai/preference-test-sets, with the question and two candidate responses as the judge input. The target identifies the human-preferred response, adjusted for the displayed candidate order. The original human judgments are available as lmsys/mt_bench_human_judgments.
HelpSteer3 (Wang et al., 2025). HelpSteer3 contains human preferences between model responses across code, general, multilingual, and STEM tasks. We sample 625 training examples per domain and remove tied preferences. The judge input is the last user message and the two candidate responses. The target is the direction of the overall_preference rating, stored as gold_label_binary. We read this reference field from the judge files to preserve row alignment. The dataset is available as nvidia/HelpSteer3.
RewardBench 2 [29]. RewardBench 2 tests response selection across factuality, instruction following, mathematics, safety, focus, and ties. We construct a pairwise panel from its 1,865 prompts by taking the first chosen and first rejected completion for each prompt. The judge input is the prompt and these two responses; the target identifies the chosen response in the displayed order. We align intervention rows by source-panel position because pair identifiers repeat within comparison groups. The dataset is available as allenai/reward-bench-2.
Evaluation protocol
Data splits and hyperparameter tuning. Following Zhao et al. (2026), we use five random row-level splits per dataset, reserving 10% for validation on continuous tasks and 15% on binary/preference tasks. We select intervention families and hyperparameters using the validation set. For CARE, we follow the original paper’s tuning procedure: CARE-SVD tunes its regularization, while CARE-Tensor tunes its regularization, rank, and posterior cutoff.
Candidate-pool construction. We used Codex and Fable to inspect dataset examples and propose potential nuisance attributes and corresponding intervention instructions. Pool construction was open-ended, with no prespecified stopping rule for expansion. The pool was refined using experimental feedback, including validation metrics. Given a family specification, pair generation and preservation checks use no quality labels (Appendix § 11.§ 5).
Family selection. We select families by forward greedy search using mean validation performance across seeds. We first select the best-performing single family. At each subsequent step, we evaluate adding each remaining family, retuning for every candidate set, and accept the addition with the largest validation improvement. The search stops when no addition improves validation performance or 15 families have been selected.
Estimator settings. We tune and , with . We select one configuration per dataset by minimizing mean validation MAE for continuous tasks or maximizing mean validation accuracy for binary/preference tasks across the five splits. Table 6 reports the selected configurations, family counts, and evaluation variability.
| Dataset | Best baseline | VoirDire | Selected / candidate families | |
| Continuous scoring: MAE () | ||||
| Yelp | 0.6940 ± 0.0040 | 0.5080 ± 0.0156 | 4 / 30 | (1, 64) |
| SummEval | 0.7293 ± 0.0048 | 0.7012 ± 0.0052 | 4 / 24 | (0, 1) |
| Summarize from Feedback | 1.3922 ± 0.0136 | 1.3461 ± 0.0036 | 4 / 41 | (0, 0.25) |
| HelpSteer2 | 0.9814 ± 0.0057 | 0.9514 ± 0.0126 | 3 / 33 | (0.75, 0.25) |
| ASSET | 27.6290 ± 0.1560 | 26.9303 ± 0.3421 | 5 / 53 | (0.5, 0.25) |
| Review-5K | 1.9570 ± 0.0180 | 1.8624 ± 0.0352 | 4 / 25 | (0.25, 64) |
| Binary/preference: accuracy () | ||||
| Chatbot Arena | 0.5959 ± 0.0162 | 0.6465 ± 0.0105 | 5 / 43 | (0.5, 0.25) |
| RewardBench 2 | 0.6052 ± 0.0116 | 0.6579 ± 0.0065 | 4 / 37 | (1, 4) |
| LIAR2 | 0.6780 ± 0.0039 | 0.6857 ± 0.0047 | 2 / 12 | (0.1, 0.25) |
| HelpSteer3 | 0.5921 ± 0.0272 | 0.6101 ± 0.0146 | 6 / 39 | (0.5, 0.25) |
| CivilComments | 0.7473 ± 0.0031 | 0.7387 ± 0.0033 | 2 / 22 | (0, 0.25) |
| MT-Bench (human) | 0.6128 ± 0.0036 | 0.6242 ± 0.0065 | 4 / 38 | (0.1, 1) |
Table 6. Selected configurations and mean ± SD. Best baseline is the strongest mean among MV, AVG, ISP, PCA, and the available CARE variants.
Weight orientation and normalization. The generalized eigenvector is defined only up to sign and scale. We orient the weights to have a positive sum and rescale them to sum to one, so the aggregate stays on the judges’ score scale.
Computational resources and cost. We ran experiments on NVIDIA A100 40GB and L40 GPUs. Generating the original judge outputs took up to approximately three hours per dataset. Judge scoring and rewriting use GPUs, while estimator fitting and family selection run on CPUs. For selected families and judges, calibration uses rewritten scores. Total cost also includes generating and scoring candidate interventions that are not selected.
LLM Judge Models
Following Zhao et al. (2026), we use fixed pools of open-weight LLM judges and filter their outputs for valid scores. For free-form scalar outputs, we retain judges with valid scores on more than 70% of items, exclude items with missing, unparsable, or out-of-range scores from any retained judge, and remove judges with constant scores. The resulting panels contain 11 judges for continuous scoring and 20 for binary classification and pairwise preference.
Continuous-score judges. The continuous-score pool contains Qwen3-{0.6,1.7,4,8}B [6], Llama-3.1-8B-Instruct and Llama-3.2-{1,3}B [30], Gemma-3-{1,4}B-IT [17], Phi-4-mini-instruct [31], and Mistral-7B-Instruct-v0.3 [32]. Judges are prompted to return a scalar score on the dataset-native range: 1–10 for Review-5K, 1–5 for Yelp, 1–5 for SummEval coherence, 1–7 for Summarize from Feedback, 0–4 for HelpSteer2, and 0–100 for ASSET.
Binary and preference judges. The binary/preference pool is the same 20 judges on every panel: Qwen2.5 Instruct (1.5B, 3B, 7B, and 14B) [5] and Qwen3-{0.6,1.7,4,8}B [6], Llama-3.1-8B-Instruct, Llama-3.2-{1,3}B, and Llama-3.2-{1,3}B-Instruct [30], Mistral-7B-Instruct-v0.3 [32], Phi-4-mini-instruct [31], Yi-1.5-{6,9}B-Chat (01.AI, 2024), Gemma-2-9B-IT [16], and Gemma-3-{1,4}B-IT [17]. We map CivilComments and LIAR2 outputs to binary labels and pairwise outputs to candidate preferences for accuracy evaluation. The judge pool remains fixed throughout intervention-family selection.
LLM Judge Prompt Templates
Prompting and score parsing. We use task-specific prompts and score ranges, applying the same prompt format and parsing rules to original and modified inputs. Ratings are extracted from the requested verdict field; pairwise ratings are converted to signed preferences and evaluated using a fixed threshold. Missing, unparsable, or out-of-range ratings are discarded. All judges receive raw completion prompts without chat templates. The templates below use placeholders in braces for item-specific content.
Review-5K paper scoring. The {guidelines} field holds the venue’s reviewer guidelines and {paper} the submission text.
You are scoring the overall quality of a research paper submission. Use the reviewing
guidelines below to assess methodological soundness, clarity, contribution, and
overall merit. Assign one of the standard conference rating levels (integer in [1,
10]) using this rubric:
1 - Strong Reject: fundamental flaws; clearly unsuitable for publication.
3 - Reject, not good enough: major weaknesses outweigh contributions.
5 - Marginally below the acceptance threshold: borderline but leaning negative.
6 - Marginally above the acceptance threshold: borderline but leaning positive;
improvements still desirable.
8 - Accept, good paper: solid contribution with meaningful advances and only minor
issues.
10 - Strong Accept, should be highlighted at the conference: outstanding paper with exceptional clarity and impact.
If you believe an intermediate score (2, 4, 7, 9) better reflects the work, you may use it, positioning it between the adjacent rubric descriptions above.
Reviewing guidelines: {guidelines}
Respond strictly in this format:
Feedback::: Total rating: X
Where X is an integer between 1 and 10 inclusive. Output nothing else.
Paper content: {paper}
Feedback::: Total rating:
Yelp review scoring.
You are evaluating the overall sentiment of a Yelp restaurant review. Read the entire review, consider the reviewer's tone, described experiences, and final impression, then assign a single star rating from 1 to 5 according to this rubric:
1 - Strongly negative; serious problems or clear dissatisfaction. 2 - Mostly negative; several complaints outweigh positives. 3 - Mixed or neutral; balanced feedback with no strong lean. 4 - Mostly positive; satisfied with only minor issues. 5 - Highly positive; enthusiastic praise and strong recommendation.
Base the rating on the reviewer's intent, not your personal opinion, and ignore formatting quirks.
Respond exactly in this format:
Feedback::: Total rating: X
Where X is an integer from 1 to 5 and nothing else.
Review: {text}
Feedback::: Total rating:
SummEval coherence scoring.
You are rating how coherent a candidate summary is. Read the article and the summary, judge whether the summary is well structured and its sentences follow one another in a sensible order, rather than reading as a disconnected list of facts, and output a single rating in :
1 - Disjointed; sentences do not connect and the order is arbitrary. 2 - Mostly disconnected, with abrupt transitions between sentences. 3 - Partly organised; some sentences follow naturally and others do not. 4 - Well organised, with only minor lapses in flow. 5 - Fully coherent; the summary reads as a single connected piece of text.
Judge organisation and flow only. Do not reward a summary for covering more of the article.
Respond strictly in this format:
Feedback::: Total rating: X
Where X is an integer 1-5. Output nothing else.
Article: {article}
Candidate summary: {summary}
Feedback::: Total rating:
Summarize from Feedback scoring.
You are an expert editor judging how faithfully a candidate summary captures the key information in the original Reddit post. Read the post first (central events, reasons, outcomes). Then read the summary and assess faithfulness and coverage.
Score substance over surface:
Prioritise factual faithfulness and coverage of core facts (who/what/when/where/why/how) and outcomes.
Prefer precise, specific content grounded in the post; do not reward length, flourish, or formatting.
Subtract for invented facts, contradictions, or missing main points; minor wording differences should rarely affect the score.
A concise but accurate and comprehensive summary can score high; a longer but vague or extraneous one can score low.
Use the absolute 1-7 scale (not relative to other items):
1 - Completely inaccurate or misleading; major facts are wrong or missing. 2 - Very inaccurate; severe omissions or contradictions; little usable content. 3 - Poor; captures a few ideas but misses several important points or introduces errors. 4 - Fair; mostly correct but notable gaps, extraneous details, or minor hallucinations. 5 - Good; covers main ideas with small omissions or wording issues. 6 - Very good; accurate and fluent with only trivial gaps. 7 - Excellent; fully faithful, comprehensive, and well-written.
Guidance:
Reserve 7 for both faithful and comprehensive summaries; use 1-2 for clearly poor ones.
Minor style differences are acceptable if the content remains faithful.
Respond exactly in this format:
Feedback::: Total rating: X
Where X is exactly one of: 1, 2, 3, 4, 5, 6, 7. Output nothing else.
Post: {post}
Summary: {summary}
Feedback::: Total rating:
HelpSteer2 response scoring.
You are scoring how well the system_answer addresses the user_question. Return a single float in [0.0, 4.0] (decimals allowed). Use the full range: 0.0 = Unhelpful or harmful; refusal; mostly wrong or off-topic. 1.0 = Barely helpful; major gaps or serious inaccuracies. 2.0 = Partially helpful; mixed quality with notable omissions or minor errors. 3.0 = Helpful and mostly correct; small omissions or minor issues only. 4.0 = Fully correct, helpful, and clear.
Rules:
Prioritise factual accuracy, relevance, and completeness; do not reward verbosity.
Penalize hallucinations, unsafe advice, or refusal when an answer is expected.
If your internal judgment falls outside [0, 4], output the nearest bound instead.
Respond exactly in this format and nothing else:
Feedback::: Total rating: X
Question: {question} Answer: {answer}
Feedback::: Total rating:
ASSET simplification scoring. The {aspect} field is the row’s rated aspect (meaning, fluency, or simplicity).
You are rating the quality of a simplified sentence for a specific aspect of text simplification. Examine both sentences carefully before assigning a single integer score in [0, 100]. Use the whole range; do not default to 50 or round to multiples of 5 unless warranted.
Aspect to evaluate: {aspect}
Interpret the aspect as follows (case-insensitive):
meaning / meaning preservation -- Judge how faithfully the simplification preserves the original meaning (facts, relations, intent). Ignore simplicity/fluency except where they alter meaning.
fluency / grammaticality -- Judge grammatical correctness and naturalness of the simplification. Ignore meaning differences and degree of simplification; do not reward added content.
simplicity / readability -- Judge how much easier to read the simplification is compared with the original (shorter sentences, simpler vocabulary/structure). Minor meaning loss may be acceptable, but removing essential information or making text ambiguous should not receive top scores.
other -- If the aspect is unrecognised, judge overall simplification quality, prioritising meaning preservation, then fluency, then simplicity.
Anchor guidance (choose any integer within each band): 0–20 Unusable for the aspect. 21–40 Poor for the aspect. 41–60 Mixed: noticeable problems but some positives. 61–80 Good: solid performance with minor issues. 81–95 Very good: strong performance with only tiny flaws. 96–100 Excellent for the aspect.
General rules:
Focus strictly on the stated aspect when scoring.
Do not reward hallucinations or added facts not in the original.
Non-round integers are allowed (e.g., 73, 88) if they best reflect quality.
Respond strictly in this format:
Feedback::: Total rating: X
Where X is an integer in [0, 100]. Output nothing else.
Original sentence: {original}
Candidate simplification: {simplification}
Feedback::: Total rating:
CivilComments toxicity scoring. The 0–9 score is thresholded at 4.5 to obtain the binary toxicity decision.
You are a toxicity rater. Given a single user comment, assign a discrete toxicity score on a 0–9 scale. Higher numbers indicate more toxic language.
Return exactly one integer using this rubric:
0 : Clearly non-toxic or benign language. 1 : Mostly neutral with only faint negativity. 2 : Slightly negative or dismissive language. 3 : Mild insults or disrespectful tone. 4 : Noticeable harassment, repeated negativity, or coarse language. 5 : Borderline toxic; sustained insults or hostility. 6 : Clearly toxic with direct insults or slurs. 7 : Highly toxic; explicit harassment or hateful language. 8 : Severe toxicity including threats or strong hate speech. 9 : Extremely toxic content with explicit threats or violent hate speech.
Guidelines:
Focus on the literal wording of the comment.
Penalise hate, harassment, threats, or extreme vulgarity.
If the comment mixes toxic and mitigating language, choose the score that reflects the overall impact.
Respond strictly in this format:
Feedback::: Total rating: X
Where X is an integer between 0 and 9 (inclusive). Output nothing else.
Comment: {text}
Feedback::: Total rating:
LIAR2 veracity scoring. The 0–9 score is thresholded at 4.5 to obtain the binary truthfulness decision.
You are a fact-checking rater. Given a single political claim, assign a discrete accuracy score on a 0–9 scale. Higher numbers indicate a claim that is more accurate.
Return exactly one integer using this rubric: 0 : Fabricated; no factual basis at all.
1 : Clearly false and contradicted by available evidence.
2 : False; the central assertion is not supported.
3 : Mostly false; a kernel of truth badly distorted.
4 : Borderline false; the overall impression misleads.
5 : Mixed; as much misleading as accurate.
6 : Half true; a real basis but material omissions.
7 : Mostly true; minor details overstated.
8 : True with only trivial imprecision.
9 : Entirely true and well supported.
Guidelines:
- Judge the factual accuracy of the claim itself, not its tone or the speaker.
- Rhetoric, framing and political stance are not evidence.
- If a claim mixes accurate and inaccurate elements, score the overall impression it
creates.
Respond strictly in this format:
Feedback:::
Total rating: X
Where X is one of: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9. Output nothing else.
Statement: \{text\}
Feedback:::
Total rating:
Pairwise preference scoring. Ratings below 3 favor response A, while ratings above 3 favor response B. HelpSteer3, RewardBench 2, and MT-Bench use the first template, which allows a tie at 3. Chatbot Arena uses the second template, which excludes ties.
You will compare two assistant responses (A and B) to the same user question. Judge
which response is more helpful overall, balancing factual accuracy, completeness,
clarity, safety, and tone.
Return a single integer chosen from \{0, 1, 2, 3, 4, 5, 6\} using this rubric:
0 : Response A is far superior to Response B
1 : Response A is clearly better than Response B
2 : Response A is slightly better than Response B
3 : Responses are essentially tied (equally good or bad)
4 : Response B is slightly better than Response A
5 : Response B is clearly better than Response A
6 : Response B is far superior to Response A
Guidelines:
- Read the user question carefully to understand the user's goal.
- Evaluate each answer independently for correctness, helpfulness, and safety.
- If Response A is better overall, you must output one of \{0, 1, 2\}. If Response B is
better overall, output one of \{4, 5, 6\}. If they are truly tied, output 3. Do not
output any other number.
- Compare the answers and decide how strongly one is better using the rubric above.
- Penalize hallucinations, unsafe advice, or refusal when a substantive answer is
expected.
Respond strictly in this format:
Feedback:::
Total rating: X
Where X is one of: 0, 1, 2, 3, 4, 5, 6. Output nothing else.
Question: \{question\}
Response A: {answer_a}
Response B: {answer_b}
Feedback:::
Total rating:
You will compare two assistant responses (A and B) for the same user question. Judge
which response better helps the user overall, considering factual accuracy,
completeness, clarity, safety, and tone.
Return a single integer chosen from {0, 1, 2, 4, 5, 6} using this rubric:
0 : Response A is far superior to Response B
1 : Response A is clearly better than Response B
2 : Response A is slightly better than Response B
4 : Response B is slightly better than Response A
5 : Response B is clearly better than Response A
6 : Response B is far superior to Response A
Important (must follow all points below or the answer is invalid):
- Ties are NOT allowed. Never output 3 or any other number outside the set {0, 1, 2, 4,
5, 6}.
- If Response A is better overall, you MUST output one of {0, 1, 2}. Never output 4, 5,
or 6 when Response A wins.
- If Response B is better overall, you MUST output one of {4, 5, 6}. Never output 0, 1,
or 2 when Response B wins.
- If the responses are very close, choose 2 (if A is slightly better) or 4 (if B is
slightly better).
- Output only the integer--no explanations, words, or additional text.
Remember: Response A better -> 0/1/2. Response B better -> 4/5/6. Outputs outside these
sets will be rejected.
Respond strictly in this format (no extra text):
Feedback:::
Total rating: X
Where X is one of: 0, 1, 2, 4, 5, 6.
Question: {question}
Response A: {answer_a}
Response B: {answer_b}
Feedback:::
Total rating:
Intervention Specifications and Preservation Checks
Intervention specifications. Each intervention family specifies (i) the nuisance attribute and input span to modify, (ii) the rule used to perform the modification, and (iii) the preservation constraints used to accept the resulting pair. Modification rules are either deterministic transformations or family-specific instructions executed by an LLM; in our experiments, we use Llama-3.1-8B-Instruct.
Rewrite prompts. The rewrite model receives the input text, a family-specific instruction, and preservation constraints. Word-overlap and length constraints are set according to the intervention family and input length. The model returns only the rewritten text. The following example shows the Yelp concision prompt.
You are rewriting text for a controlled intervention study.
Dataset: yelp_with_scores
Task type: continuous
Goal:
Express the same content more briefly. Remove redundancy and filler only. Keep every fact, number, name and claim. Do not add anything new.
Review:
I really like this place a lot better now that they've moved from the Borgata. It's a nice open floor plan and I love the outside patio and bar area. My pizza was great and I'm pleased to see their nice selection of Italian wines.
Hard constraints:
Preserve the same meaning, main claims, recommendation, sentiment, safety posture, and overall quality judgment.
Do not add, remove, contradict, or strengthen substantive content.
Keep every number, date, name, and URL exactly as in the original.
Keep every negation as in the original.
Wording: reuse the original's own content words, so that at least 55 percent of them still appear in the rewrite.
Length: the rewrite must be between 127 and 212 characters; the original is 230.
Return only the rewritten text.
Preservation checks. For deterministic transformations, checks verify the specified structural properties, such as preservation of text, letter sequences, or lines. Position swaps must exchange candidates verbatim; context and prompt modifications must leave the judged text unchanged.
LLM rewrites must preserve all numbers and the negation count, introduce no new URLs, and satisfy the content-word Jaccard similarity and length-ratio bounds in table 7. Families without a specified length range use [0.70, 1.60]. These checks filter unintended changes but do not guarantee preservation of the evaluation target.
Intervention examples. Table 8 lists the four families selected for Yelp and the position-swap intervention used on preference datasets. The examples below show their application to a Yelp review and a Chatbot Arena comparison. All pairs passed the preservation checks. The Yelp example is the shortest review shared by the four selected families’ calibration pools, chosen without inspecting judge scores.
| Family | Kind | Operation | Preservation check |
| Concision | LLM | “Express the same content more briefly. Remove redundancy and filler only. Keep every fact, number, name and claim. Do not add anything new.” | Jaccard ≥ 0.55, length [0.55, 0.92] |
| Prose form | LLM | “Convert any headings, bullet points, numbering or emphasis into continuous prose, keeping the sentences in their original order and wording. Do not summarize, omit or add anything.” | Jaccard ≥ 0.75, length [0.85, 1.20] |
| Markdown quote | deterministic | Prefix every line of the exact text with “> ”. | letter sequence equal |
| XML frame | deterministic | Wrap the exact text in <document> tags. | exact text preserved |
| Position swap | deterministic | Exchange the two candidates verbatim; the prompt-order score is negated at fit time. | exact candidate exchange |
Table 8. The four families selected for Yelp and the position-swap family of the preference panels.
Original review:
I really like this place a lot better now that they've moved from the Borgata. It's a
nice open floor plan and I love the outside patio and bar area. My pizza was
great and I'm pleased to see their nice selection of Italian wines.
Concision (LLM rewrite):
I like this place more since they moved from the Borgata. The open floor plan and
outside patio/bar area are nice. My pizza was great and they have a good Italian
wine selection.
Prose (LLM rewrite):
I really like this place a lot better now that they've moved from the Borgata. The open
floor plan is nice and I love the outside patio and bar area. My pizza was great
and I'm pleased to see their nice selection of Italian wines.
Markdown quote (deterministic):
I really like this place a lot better now that they've moved from the Borgata. It's a nice open floor plan and I love the outside patio and bar area. My pizza was great and I'm pleased to see their nice selection of Italian wines.
XML frame (deterministic):
<document>
I really like this place a lot better now that they've moved from the Borgata. It's a
nice open floor plan and I love the outside patio and bar area. My pizza was
great and I'm pleased to see their nice selection of Italian wines.
</document>
Shared user question:
Write me a haiku about hiroshima
Original order:
Response A: Hiroshima weeps,
Silent scars whisper of peace,
Healing through time's grace.
Response B: A flash of light,
A city in ruins,
A world changed forever.
After position swap:
Response A: A flash of light,
A city in ruins,
A world changed forever.
Response B: Hiroshima weeps,
Silent scars whisper of peace,
Healing through time's grace.
| LLM axis | Minimum word Jaccard | Length ratio interval |
| Presentation polish | 0.60 | [0.70, 1.60] |
| Concision | 0.55 | [0.55, 0.92] |
| Verbosity | 0.60 | [1.15, 2.00] |
| Authority register | 0.70 | [0.85, 1.50] |
| Conclusion framing | 0.80 | [0.95, 1.40] |
| Hedging | 0.70 | [0.95, 1.40] |
| Informality | 0.60 | [0.80, 1.30] |
| Prose form | 0.75 | [0.85, 1.20] |
Table 7. Family-specific LLM rewrite checks, in addition to the common invariants.
All LLM families additionally pass the common checks of Appendix B.5.
Family availability. Only families with enough valid, fully scored pairs enter selection, yielding 12–53 eligible families per dataset. On Review-5K, LLM rewrites failed the number-preservation checks, so calibration uses only deterministic structural transformations.
Controlled Judge-Bias Benchmark
We describe the benchmark and protocol used in § 5.5.
Data and judges. The benchmark of Chen et al. [12] contains two GPT-4 answers to each of 142 questions. We reuse its GPT-4 rewrites targeting rich formatting (beauty) or added references (authority); these can also rephrase or expand the answer. We use the scores provided by CARE (Zhao et al., 2026), following its preprocessing to exclude Phi-4-mini and retain pairs with valid original and modified scores. The resulting panels contain ten judges rating answers on a 1–10 scale, with 136–142 valid pairs per modification. Figure 6 illustrates the beauty and authority rewrites.

Figure 6. One benchmark answer and its beauty and authority rewrites ([12]). Emojis are rendered as symbols; the fake citation is in bold.
Calibration and evaluation. We use beauty and authority pairs as calibration interventions. For each bias, we partition the pairs into five folds and repeat the procedure over ten seeds. The main comparison uses 32 calibration pairs sampled outside the evaluation fold. All methods use the same evaluation folds. We measure the mean absolute change in aggregate scores between original and modified answers.
PCA and CARE-SVD are fitted to the full original-score panel and applied to both versions. VoirDire uses the same original panel together with the sampled calibration pairs. ISP is computed jointly on original and modified scores. Evaluation items therefore contribute unlabeled scores to fitting, but their intervention pairs are excluded from calibration.
Parameter settings. We fix across biases, folds, seeds, and calibration budgets. We use maximum attenuation, , to suppress the known nuisance manipulation. The value is the most frequently selected setting on the six continuous-score benchmarks, selected on three of them, giving . No quality labels from this benchmark are used for fitting or parameter selection.
Intervention-Pair Budget: Details and Additional Results
We provide sampling details and per-dataset results for § 5.4.
Setup. We vary the number of pairs per family over , keeping the selected intervention families fixed. Samples are nested within each seed: reproduces the main experiments, smaller budgets use subsets, and adds available pairs. We re-select on validation at each budget, as in the main-body ablation. All budgets use the same five splits and evaluation items. We report means and one standard error.
Results. Figure 7 and Figure 8 show the per-dataset results summarized in Figure 5. Using 32 pairs outperforms both 4 and 8 pairs on all 12 datasets, while gains beyond 32 pairs are generally small and not uniform across datasets.
Figure 7. Per-dataset MAE as the number of intervention pairs per family varies; lower is better. Error bars show one standard error across five splits. The dotted line marks the main experimental budget, .
Figure 8. Per-dataset accuracy as the number of intervention pairs per family varies; higher is better. Error bars show one standard error across five splits. The dotted line marks the main experimental budget, .
Related-Nuisance Transfer Details
We provide construction and evaluation details for § 5.6, following the benchmark protocol in Appendix § 11.§ 6.
Intervention construction. We use Llama-3.1-8B-Instruct to generate numbered structure, minimalist panels, and topic-specific icons. Prompts request presentation changes while preserving the original wording and order. We use temperature zero and a 1,024-token limit, retaining the first completion without preservation filtering or retries. Expert attribution, a source header, and fabricated footnotes use deterministic insertions. The source header attributes the answer to a generic reference handbook, without adding an answer-specific citation as in the benchmark authority rewrite. Table 9 lists the specifications.
| Variant | Requested rewrite or deterministic operation |
| Beauty: LLM-generated formatting | |
| Numbered structure | Number existing steps or parallel points consecutively as 1., 2., etc. Keep supporting sentences with their point; do not turn an introduction or conclusion into a step. Preserve words such as “First” and “Finally.” Add no headings, icons, bolding, or blockquotes. |
| Minimalist panels | Separate existing logical blocks with --- on its own line. Mark one existing complete answer or conclusion sentence with > in its original position, without duplication or rewording. Add no headings, emojis, bolding, or lists. |
| Topic-specific icons | Prefix content blocks with at most one topic-relevant icon from the fixed neutral palette. Preserve the original text and add no other formatting. |
| Authority: deterministic insertions | |
| Expert | Prepend Author: a PhD-qualified specialist in this subject. followed by two newline characters. |
| Source header | Prepend Source: a university reference handbook. followed by two newline characters. |
| Footnote | Insert [1] immediately after the first detected sentence. Append two newline characters and [1] Academic Reference Board. Reference Handbook. University Press, 2024. |
Table 9. Calibration variants in Table 4. Numbered, Panels, and Icons abbreviate the beauty variants in the result tables. Beauty outputs are not filtered for compliance with these instructions. All authority credentials, source attributions, and reference metadata are fabricated experimental cues.
Examples of the resulting rewrites appear below.
Question: What is the freezing point of water in Fahrenheit?
Original:
The freezing point of water in Fahrenheit is 32 degrees. This is based on the Fahrenheit scale, where the freezing and boiling points of water are set 180 degrees apart, with the freezing point at 32 and the boiling point at 212.
Numbered structure:
The freezing point of water in Fahrenheit is 32 degrees.
This is based on the Fahrenheit scale, where the freezing and boiling points of water are set 180 degrees apart, with the freezing point at 32 and the boiling point at 212.
Minimalist panels:
The freezing point of water in Fahrenheit is 32 degrees.
This is based on the Fahrenheit scale, where the freezing and boiling points of water are set 180 degrees apart, with the freezing point at 32 and the boiling point at 212.
Topic-specific icons:
[books] The freezing point of water in Fahrenheit is 32 degrees. This is based on the Fahrenheit scale, where the freezing and boiling points of water are set 180 degrees apart, with the freezing point at 32 and the boiling point at 212.
Expert:
Author: a PhD-qualified specialist in this subject.
The freezing point of water in Fahrenheit is 32 degrees. This is based on the Fahrenheit scale, where the freezing and boiling points of water are set 180 degrees apart, with the freezing point at 32 and the boiling point at 212.
Source header: Source: a university reference handbook.
The freezing point of water in Fahrenheit is 32 degrees. This is based on the Fahrenheit scale, where the freezing and boiling points of water are set 180 degrees apart, with the freezing point at 32 and the boiling point at 212.
Footnote:
The freezing point of water in Fahrenheit is 32 degrees.[1] This is based on the Fahrenheit scale, where the freezing and boiling points of water are set 180 degrees apart, with the freezing point at 32 and the boiling point at 212.
[1] Academic Reference Board. Reference Handbook. University Press, 2024.
Calibration and evaluation. We use five folds and ten seeds with . Each source family supplies 32 calibration pairs per split, excluding evaluation questions and requiring valid scores from all ten judges. Source shifts use newly scored original–rewrite pairs; target evaluation and covariance estimation reuse the benchmark scores from § 5.5. Covariance estimation includes unlabeled original evaluation inputs. Each source–target model is frozen before evaluation.
Transfer metric. Let denote the mean absolute score shift on target family after calibration on source family , averaged over folds and seeds. We report the percentage of the target-family improvement over PCA retained by source calibration:
PCA corresponds to 0% and direct target-family calibration to 100%. Values above 100% indicate lower sensitivity than the target-family reference; negative values indicate greater sensitivity than PCA.
Additional Theory and Proofs
This appendix provides the formal details for § 4. We give full statements and proofs for observational non-identifiability and estimation error in the pooled nuisance matrix and its normalized attenuation operator, followed by five extensions:
Effect of imperfect interventions. We quantify how violations of target preservation and unintended rewrite changes affect recovery of the nuisance-sensitive subspace.
Finite-sample estimation and recovery. We show how the number of calibration pairs and the effective rank control moment-estimation error, and give the additional spectral-gap condition for nuisance-subspace recovery.
Effect of attenuation. We show that attenuation preserves the target component and suppresses nuisance variation under an ideal orthogonal model, and bound target distortion due to paired judge noise and estimation error.
Perturbation of the aggregation weights. We bound the leading generalized eigenvector up to relative sign when both matrices of its pencil change, and state the local conditions needed to transfer this bound through the score-normalization rule.
Multiple intervention families. We justify the pooled second-moment estimator used by VoirDire when multiple selected intervention families move a common nuisance subspace.
Assumptions
We use the model in eq. (4) to formalize when observational judge agreement is ambiguous. For an original input, the judge-output vector contains a target-bearing component and a nuisance-sensitive component . Observational aggregators only observe , so they must infer from unperturbed judge agreement which directions correspond to target variation. Without additional structure, this is not identifiable: target-relevant and nuisance-sensitive directions can be rotated without changing the observational distribution, as in standard latent-factor non-identifiability [8].
The rewrites provide paired judge outputs for the same item before and after intervention. The key assumption is target invariance: the correct label, score, or preference remains fixed across the original and rewritten inputs, while the rewrite changes a nuisance attribute. In the ideal setting,
Thus the target factor is shared by the two judge-output vectors, while captures the nuisance movement induced by intervention family .
The analysis does not require judge errors to be independent. Correlated judge noise is allowed and enters through the paired residual . What matters is the paired-difference contrast:
Because the target term is common to the original and rewritten inputs, it cancels in this difference. We assume and that is independent of ; the nuisance movement may have a nonzero mean. The paired differences therefore identify directions that respond to the intervention. VoirDire then attenuates these nuisance-sensitive directions in the original judge outputs before aggregation.
Observational non-identifiability: full statement and proof
Theorem 1 (Full statement). Consider the noiseless one-target, one-nuisance model
where are orthonormal, , and is the target-bearing coordinate. For any , there exist rotated directions and rotated coordinates such that
Thus the original and rotated representations induce the same observational distribution over judge outputs, but assign different directions to the target-bearing coordinate. Therefore, the target direction is not identifiable from unperturbed judge outputs alone.
Proof of Theorem 1. Let
Define the rotated directions and coordinates by
Since is orthogonal, . Moreover,
Therefore the original and rotated parameterizations generate exactly the same marginal distribution over the observed judge-output vector .
However, they assign the target-bearing coordinate to different directions: the original representation uses , while the rotated representation uses . Since for , these are distinct target directions. Thus the same observational distribution is compatible with different choices of the target-bearing direction, so the target direction is not identifiable from unperturbed judge outputs alone.
Nuisance-matrix estimation: full statement and proof
The estimator uses the entire pooled second moment in both attenuation and the penalty in intervention-aware aggregation. We first control this matrix and its normalization, without requiring a nuisance rank or an eigengap.
Proposition 1 (Full statement). For fixed family weights with , define
The pair-weighted estimator uses . Equal family budgets give and , as in the main text. Under eq. (4), centered paired noise independent of the nuisance movement gives
Thus in the ideal intervention model.
More generally, write , allowing paired noise and imperfect interventions, and suppose
Then the positive semidefinite empirical moment satisfies
If and , define
Their difference obeys
These deterministic bounds require neither independent sampled rows nor a positive semidefinite perturbation .
Proof of Proposition 1. Subtracting the two views gives . The cross moment vanishes because
Expanding the second moment proves eq. (5); weighting by yields the pooled decomposition. The identity retains systematic shifts.
For the general perturbation, the triangle inequality gives
For nonzero , put and , using positive semidefiniteness. The decomposition
and the reverse triangle inequality imply
The two error terms have different sources. In the ideal intervention model, may be set to ; Proposition 2 also bounds target leakage and unintended changes. Under the fixed iid-law assumptions of Theorem 2, its concentration bound supplies . The deterministic statement applies to fixed family counts as well, but its probability guarantee requires a concentration bound for that sampling design.
The target measures nuisance variation excited by the chosen interventions. It need not equal the nuisance covariance of the original items. Accurate estimation of its normalized matrix does not imply strong attenuation of every nuisance direction: Theorem 3 separately controls that effect through the ratio of the weakest to the strongest nuisance eigenvalue.
Subspace recovery as an additional consequence
Subspace recovery requires a spectral gap in addition to matrix accuracy. We express it using the orthogonal projector onto . For full-column-rank , , reducing to for orthonormal columns. For equidimensional subspaces,
These projectors describe subspace diagnostics; the algorithm uses the normalized moment .
Corollary 1 (Nuisance-subspace perturbation). Under the ideal intervention setup of Proposition 1, assume has full column rank . If and has rank , then . With paired judge noise, define
where denotes the -th largest eigenvalue of a symmetric matrix . Thus is the weakest nuisance-signal eigenvalue in this model. If , then the top- eigenspace of satisfies
Since is positive semidefinite, is at least the value obtained from alone: a systematic shift cannot decrease the weakest nuisance eigenvalue and may supply a direction that item-to-item variation does not.
Proof. By eq. (5), if , then
Because has full column rank and has rank , this matrix has range . It remains to handle paired judge noise. Write
If , both subspaces are the full judge-output space and their projectors agree. Suppose now that . The matrix has rank , range , and eigengap between its -th and -st eigenvalues. By Weyl’s inequality and the Davis–Kahan sin theorem ([10, 14] (Yu et al. 2015)), if , then the top- eigenspace of satisfies
using eq. (8) to read the Davis–Kahan conclusion as a projector distance. Substituting and gives the stated bound. Finally, is a sum of positive semidefinite matrices, so by Weyl’s monotonicity , the value obtained from the centered covariance.
The proof formalizes the paired-comparison intuition. If the rewrite preserves the target, then subtracting the original and rewritten judge outputs removes the target part. The remaining consistent score changes show which judge-output directions react to the rewrite; these are the directions VoirDire later attenuates. Paired judge noise can blur this signal, but the corollary shows recovery works when the rewrite-induced changes are larger than that noise.
Effect of imperfect interventions
The rewrites are intended to change only the specified nuisance attribute while preserving the evaluation target. In practice, a rewrite that passes the checks may deviate from this ideal: it may accidentally change target-relevant content, or it may introduce additional changes beyond the intended nuisance attribute. We treat these deviations as perturbations to the paired-difference second moment.
We decompose the paired difference as
Here is the intended nuisance movement. The term captures accidental changes to the target factor, and captures rewrite-induced changes outside the intended nuisance subspace. The residual term is paired judge noise. Thus imperfect rewrites add error terms to the paired-difference second moment, which can interfere with recovery of the nuisance-sensitive subspace.
Proposition 2 (Perturbation from imperfect interventions). Fix a family and let
If has rank , where , , and , then the top- eigenspace of satisfies
Moreover, let and . Suppose have orthonormal columns, is centered and uncorrelated with , and
Then the perturbation itself satisfies
Proof. The first claim follows from a standard perturbation argument. By definition,
The matrix is the second moment contributed by the intended nuisance movement. If , both projectors equal , so the recovery bound is immediate. For , it has rank and eigengap between its -th and -st eigenvalues. Since , the Davis–Kahan sin theorem (Davis & Kahan, 1970; Yu et al., 2015) gives
For the second claim, expand the second moment of eq. (9). The error term contains paired judge noise, target-changing terms, unintended rewrite terms, and their cross moments with the intended nuisance movement; the cross moments with the centered paired noise vanish as in Proposition 1. Since , , have orthonormal columns, multiplication by these loading matrices does not increase operator norm. Therefore,
By Cauchy–Schwarz,
and
These uncentered cross moments already contain the products of the means, so no separate centering term arises. Each cross moment appears together with its transpose, giving the factors of two. Adding the paired-noise covariance yields the displayed bound on . Its right-hand side may be used as ; the subspace conclusion additionally requires this value to be less than . □
This proposition clarifies the role of the preservation checks. The checks do not need to eliminate all errors, but they should keep target-changing and unintended rewrite effects small relative to the intended nuisance signal. Recovery is reliable when the intended nuisance eigengap dominates the combined perturbation level .
Finite-sample recovery and effective rank
The preceding results use population moments. In the algorithm, however, the intervention second moment is estimated from finitely many calibration pairs. The next result separates two sources of error: the perturbation from imperfect interventions, measured by , and the moment-estimation error, measured by . The latter depends on both the pooled calibration-pair count and the effective rank of the paired-difference second moment.
Theorem 2 (Finite-sample nuisance recovery). Let be the pooled second moment of the calibration paired differences (eq. (1)), with . Let denote the population pooled moment. Suppose
If has rank , where , , and
then any orthonormal basis matrix for the top- eigenspace of satisfies
For the probability bound, additionally suppose the are iid from a fixed pooled distribution, , and, for a fixed ,
Here . Then, for , with probability at least ,
where and depends only on .
Proof. Let
Then
By the triangle inequality,
If , both projectors equal . For , since has rank and , Davis–Kahan gives
For the probability bound, let be independent Rademacher signs, independent of the observations, and set . Then , , and . Thus the sample second moment is unchanged, while the meet the centered, uniformly sub-Gaussian hypotheses of the effective-rank covariance bound [24 Theorem 9]. Applying that bound on gives the result, with constants depending only on .
The deterministic recovery bound does not require independent samples once is bounded. The probability guarantee assumes iid sampling from a fixed pooled distribution; fixed per-family counts or pairs sharing an original item require a corresponding concentration argument. If , the estimation error vanishes, but the positive nuisance-eigengap condition cannot hold.
Increasing reduces the estimation error at a rate governed by the effective rank : fewer pairs are needed when paired-difference variation is concentrated in a few directions. However, additional pairs cannot eliminate the systematic perturbation . Accurate recovery therefore requires the nuisance signal to dominate both sources of error.
Effect of attenuation
The calibration step applies the operator to each judge-output vector, where is the normalized pooled shift moment (eq. (2)). In the ideal orthogonal model below, this step has a simple tradeoff: increasing attenuates more nuisance variation, while error in estimating can also perturb the target component.
Theorem 3 (Effect of attenuation). Assume the noiseless orthogonal model
where and have orthonormal columns and . Let have rank with eigenvalues , let and , and write for the spectral ratio of the nuisance movement. Suppose
For , define
Then
and
If the calibration differences carry centered judge noise uncorrelated with the nuisance movement, with covariance , the population moment is . For , , so this value may be used for .
Proof. Because , the operator satisfies
The matrix is symmetric with eigenvalues , so for every . For the first inequality,
Taking norms, using , and using the orthonormality of and gives
For a calibration pair under this target-invariance assumption,
Therefore
The same norm bound gives
For the noise claim, and are positive semidefinite, so . Apply eq. (7) with , , and . The same result supplies with an additional sampling term when the moment is empirical.
The theorem captures the attenuation tradeoff. If is estimated accurately, increasing reduces both the remaining nuisance component and the original–rewrite score change, and each nuisance direction is scaled by a factor in according to how strongly the rewrites move it. If is inaccurate, larger can also remove target-relevant variation, which is why is selected on validation rather than fixed at one. When is a multiple of the identity, is the projector onto the nuisance subspace, , and the operator becomes projector-based attenuation; it removes the nuisance component completely when and .
Perturbation of the generalized eigenvector
We study the leading generalized eigenvector of the calibrated-output covariance and intervention penalty. Write and . The weight vector solves
The result below compares unit-normalized weights up to relative sign in the norm . The implementation’s orientation and normalization require the additional local conditions stated afterward.
Theorem 4 (Generalized eigenvector perturbation). Let and be matrices, with and for . Suppose
Let and be the largest generalized eigenvalue and the gap to the second largest eigenvalue of . Define
If , then leading generalized eigenvectors normalized by satisfy
Proof. Use the same reference coordinates for both pencils. Set
Then , , and . The eigenvalues of lie between and , so and . The second inequality uses .
The transformed estimated pencil is the symmetric matrix . Its eigenvalues are those of . Expanding the difference gives
and hence
Let and . These are unit leading eigenvectors of and , respectively. Weyl’s inequality and ensure that the estimated leading eigenvalue is simple. The single-vector Davis–Kahan bound gives a relative sign such that (Yu et al., 2015).
Mapping back now retains the change in the metric:
Taking norms and using gives , which is eq. (11).
Ridge perturbation. The bound controls the entire denominator matrix. Let denote a second empirical moment. With a fixed ridge, may be taken as . For the implemented rule with fixed , recomputing the ridge gives
This follows from for symmetric . The result compares a fixed estimator configuration and family set; it does not bound changes caused by discrete validation selection.
Transfer to reported scores. For a fixed calibrated-output matrix and relatively aligned, unit-normalized weights, the bound gives
The implementation instead orients the weights to a positive coordinate sum and rescales them to sum to one (Appendix § 11.§ 2). This map is not continuous where the coordinate sum is zero.
A local transfer is available when relatively aligned weights have the same orientation. Write the normalization as with . If and , then
Indeed, , and
A strict margin from the coordinate-sum sign boundary ensures the local orientation condition and the lower bound . Put . If recalibration also changes to , the full score change satisfies
This follows by adding and subtracting . For fixed original outputs and , the two calibration operators give and , so
For the same evaluation items and targets, the absolute MAE change is at most times the full score-vector bound. Thus both changing weights and changing calibration contribute. Without the orientation margin, the unit-normalized eigenvector bound does not imply stability of the reported scores.
Target and nuisance blocks. Consider the population orthogonal model of Theorem 3 with \overline{G}=L_C=U_CA_U_C^{\mathsf{T}}, so . This assumes the unnormalized moment is exact; equality of the normalized operators alone does not fix its scale. Both matrices of the pencil have a block form. Writing , , , and ,
so attenuation leaves the target block unchanged and shrinks the nuisance block by on both sides, while equals on and on . The generalized eigenvector therefore weights target variance against alone and nuisance variance against : a direction that the rewrites move strongly is penalized in the denominator in addition to being attenuated in the numerator.
Pooled second moment across families
Proposition 1 already controls the pooled moment used by VoirDire. The next corollary gives its additional subspace-recovery consequence when a fixed set of families perturbs a common nuisance subspace, possibly with different strengths and systematic shifts.
Corollary 2 (Pooled second moment). Fix a nonempty family set and positive calibration counts , with and . Suppose each family satisfies
Define the population pooled moment and its signal/noise decomposition by
If has rank , where ,
then the top- eigenspace of satisfies
Proof. The pooled estimator gives every calibration pair equal weight, so its expectation is
The signal matrix has range . If , both projectors equal . For , the signal has eigengap between its -th and -st eigenvalues. Applying the Davis–Kahan argument from Corollary 1 yields
This is the population analogue of the pooled VoirDire estimator: the algorithm stacks the paired shifts from all selected families, estimates one second moment, and uses it to form and the penalty in eq. (3). The family means enter only through , so a family whose shift is mostly systematic contributes a rank-one term inside . Pooling strengthens nuisance-subspace recovery when the added families increase the nuisance signal relative to perturbation. The benefit depends on the strength and diversity of the intervention-induced shifts.
Synthetic Experiments
We use synthetic data with known target and nuisance components to examine how intervention errors, attenuation strength, and systematic score shifts affect calibration (Appendices § 15.§ 1 to § 15.§ 3). We then examine how nuisance-subspace recovery relates to aggregation performance (Appendix § 15.§ 4).
Common setup. Unless otherwise stated, we use judges and two-dimensional target and nuisance factors (). The combined loading matrix has orthonormal columns. For experiments involving random sampling, we report means over 20 random seeds, with shaded bands indicating one standard error.
Sensitivity to Imperfect Interventions
We examine how unintended changes to target quality affect recovery of nuisance-sensitive directions.
Setup. Following Proposition 2, we generate paired shifts as
where , , and are independent. Here, controls how often interventions change the target, and controls the magnitude of those changes. The shared makes target and nuisance changes correlated. We vary from 0 to 1 with , and from 0 to 2 with . To isolate intervention error from sampling error, we compute analytically. We measure recovery error by , where projects onto the two leading eigenvectors of , and projects onto the true nuisance subspace.
Results. Figure 9 shows that nuisance-subspace recovery deteriorates as a larger fraction of interventions change the target, or as those changes grow in magnitude. This error persists even with exact population moments, implying collecting more intervention pairs cannot eliminate it.
Figure 9. Nuisance-subspace recovery deteriorates as unintended target changes become more frequent (left) or larger (right). Results use exact population moments, without sampling error.
Attenuation Strength and Target Preservation
We examine how attenuation strength affects nuisance suppression and target preservation when the estimated nuisance directions are inaccurate.
Setup. Let , with independent standard Gaussian , and define the true attenuation operator as . We introduce estimation error by rotating the estimated nuisance directions toward the target subspace:
with degrees. The true target and nuisance subspaces remain orthogonal. For , we measure
These quantities measure the largest remaining nuisance response and the largest change to the target component over unit-norm factor vectors. We quantify operator error by .
Results. Figure 10 shows that stronger attenuation reduces nuisance sensitivity while preserving the target when the operator is exact. With estimation error, stronger attenuation also increases target distortion, illustrating the tradeoff characterized in Theorem 3. Even with the exact operator and , nuisance sensitivity remains at 0.75: attenuation removes the strongest nuisance direction but only partially suppresses the weaker one.
Figure 10. Increasing attenuation strength reduces nuisance sensitivity (left) but distorts target information when the estimated nuisance directions are inaccurate (right). Colors indicate operator error ; both metrics are computed analytically.
Effect of Centering on Nuisance Recovery
VoirDire constructs its calibration matrix from paired score shifts without subtracting their mean (eq. (1)). Here, we illustrate why: a consistent shift across pairs can reveal nuisance sensitivity, and centering would remove this signal.
Setup. We model paired shifts as , with independent and . The first nuisance component varies across pairs, while the second produces a constant shift of magnitude . Using pairs, we compare the uncentered second moment with the centered covariance . We vary from 0 to 4 and measure nuisance-subspace recovery using .
Results. Figure 11 shows that the centered estimator fails to recover the second nuisance direction, while the uncentered estimator becomes more accurate as the systematic shift grows. This supports retaining the mean shift: a consistent response across pairs can reveal nuisance sensitivity even without pair-to-pair variation.
Figure 11. Centering discards the systematic shift that identifies the second nuisance direction. The uncentered estimator recovers this direction as the shift grows. At , neither population moment contains information about the second direction.
Nuisance Recovery and Aggregation Performance
We examine how nuisance-subspace recovery relates to aggregation performance as the calibration budget increases.
Setup. We generate judge outputs as , with independent target factors , nuisance factors , and Gaussian noise correlated across judges, with scale 0.5. The target score is , where is independent. We use 5,000 training and 2,000 test items. Two intervention families each add an shift to a different nuisance coordinate, preserving and resampling judge noise. Each family supplies pairs, for pairs in total.
VoirDire estimates aggregation weights from all training outputs and selects separately for each seed and budget using MAE on a 20% validation subset and the grids in Appendix B.2. We compare against PCA, CARE-SVD, and an oracle reference that projects out the true nuisance subspace using before fitting a two-component PCA aggregator. The oracle orients each eigenvector to have a positive coordinate sum, combines them with weights proportional to the square roots of their eigenvalues, and normalizes the resulting vector to sum to one. It knows the nuisance subspace but not the target coefficients. We report test MAE and nuisance-subspace recovery error .
Results. Figure 12 shows that better nuisance recovery accompanies lower aggregation error. As the budget increases from 4 to 512 pairs per family, VoirDire’s MAE falls from 0.942 to 0.582, approaching the oracle reference’s 0.567. Most of the aggregation gain occurs by 32–64 pairs per family, although subspace recovery continues to improve at larger budgets. These results show that VoirDire can improve aggregation even when the nuisance subspace is estimated imperfectly.
Figure 12. Increasing the calibration budget improves nuisance-subspace recovery (left) and aggregation performance (right). The oracle reference removes the true nuisance subspace before PCA aggregation. Budgets are per intervention family.
Additional Experimental Results
We provide further analyses of attenuation strength and judge-panel size (Appendices E.1 and E.2). We also report calibration-budget results for targeted nuisance removal (Appendix E.3) and compare the effects of score calibration across aggregators (Appendix E.4).
Sensitivity to Attenuation and Regularization
We study how attenuation strength and regularization multiplier affect aggregation performance.
Setup. We use the datasets, splits, calibration pairs, and selected intervention families from § 5.1. We evaluate all 30 combinations of and , reporting MAE for continuous tasks and accuracy for binary/preference tasks. Table 10 reports sensitivity to at the validation-selected ; Table 11 reports the effect of removing each selected intervention family and refitting.
| Dataset | 1 | 4 | 16 | 64 | ||
| Continuous (MAE, lower is better) | ||||||
| Yelp | 1 | 0.623 | 0.550 | 0.519 | 0.510 | 0.508 |
| SummEval | 0 | 0.699 | 0.701 | 0.710 | 0.724 | 0.733 |
| Summarize from Feedback | 0 | 1.346 | 1.389 | 1.507 | 1.662 | 1.735 |
| HelpSteer2 | 0.75 | 0.951 | 0.971 | 0.989 | 0.997 | 0.999 |
| ASSET | 0.5 | 26.930 | 26.956 | 27.342 | 27.611 | 27.706 |
| Review-5K | 0.25 | 2.519 | 2.256 | 2.000 | 1.893 | 1.862 |
| Binary/preference (accuracy, higher is better) | ||||||
| Chatbot Arena | 0.5 | 0.646 | 0.650 | 0.638 | 0.600 | 0.581 |
| RewardBench 2 | 1 | 0.653 | 0.658 | 0.658 | 0.657 | 0.658 |
| LIAR2 | 0.1 | 0.686 | 0.677 | 0.672 | 0.671 | 0.669 |
| HelpSteer3 | 0.5 | 0.610 | 0.495 | 0.493 | 0.491 | 0.490 |
| CivilComments | 0 | 0.739 | 0.726 | 0.717 | 0.711 | 0.709 |
| MT-Bench (human) | 0.1 | 0.618 | 0.624 | 0.610 | 0.595 | 0.582 |
Table 10. Sensitivity to the regularization multiplier at the selected and family set of the main tables. Entries are mean evaluation metrics over the protocol seeds; the value selected on validation is in bold.
| Dataset | Help | Mean | Max loss | Max gain | |
| Continuous ( = MAE without the family − MAE with it) | |||||
| Yelp | 4 | 4 | 0.024 | 0.072 | 0.003 |
| SummEval | 4 | 3 | 0.001 | 0.002 | 0.000 |
| Summarize from Feedback | 4 | 4 | 0.009 | 0.024 | 0.001 |
| HelpSteer2 | 3 | 3 | 0.014 | 0.023 | 0.005 |
| ASSET | 5 | 2 | 0.024 | 0.263 | -0.132 |
| Review-5K | 4 | 3 | 0.021 | 0.063 | -0.009 |
| Binary/preference ( = accuracy without − accuracy with) | |||||
| Chatbot Arena | 5 | 5 | -0.013 | -0.038 | -0.003 |
| RewardBench 2 | 4 | 2 | -0.036 | -0.151 | 0.007 |
| LIAR2 | 2 | 2 | -0.002 | -0.003 | -0.001 |
| HelpSteer3 | 6 | 6 | -0.026 | -0.070 | -0.004 |
| CivilComments | 2 | 1 | -0.006 | -0.020 | 0.007 |
| MT-Bench (human) | 4 | 3 | -0.032 | -0.124 | 0.001 |
Table 11. Leave-one-family-out refits of the selected family sets, re-selected without the dropped family. : number of selected families. Help: families whose removal worsens the evaluation metric. Mean : mean change over the refits. Max loss and max gain: the largest deterioration and the largest improvement obtained by dropping one family. Means over the protocol seeds.
Results. Figure 13 and Figure 14 show that the benefit of attenuation varies across datasets. Yelp benefits from strong attenuation, whereas Review-5K and ASSET deteriorate beyond moderate attenuation. On SummEval, Summarize from Feedback, and CivilComments, validation selects , retaining the original scores while using interventions to choose aggregation weights.
Figure 13. Sensitivity to and on continuous-score datasets. Curves show mean evaluation MAE across five splits for each ; lower is better. Stars mark validation-selected configurations.
Figure 14. Sensitivity to and on binary/preference datasets. Curves show mean evaluation accuracy across five splits for each ; higher is better. Stars mark validation-selected configurations.
The effect of also depends on . On Chatbot Arena, attenuation provides larger gains at , where aggregation places less emphasis on intervention sensitivity. At , varying changes accuracy by less than one percentage point. These results support tuning attenuation and regularization jointly.
Judge-Panel Size Ablation
We examine how the number of judges affects aggregation performance and the gains from intervention-based calibration.
Setup. We use the 12 datasets from § 5.1, varying for continuous tasks and for binary/preference tasks. Twenty random orderings of the judges define nested subsets shared across methods. We keep the data splits, intervention families, and calibration pairs fixed, and refit and tune each method for each subset. We average performance over five splits and then across distinct subsets; bands show one standard deviation across subset means.
Results. Figure 15 and Figure 16 show that larger judge panels generally improve aggregation. From the smallest to the full panel, VoirDire improves on five of six continuous tasks and all six binary/preference tasks, with accuracy gains of 5.51–10.05 percentage points. The exception is Review-5K, where MAE rises from 1.598 to 1.862. Overall, larger panels tend to improve aggregation performance, although the benefits vary across methods and datasets.
Figure 15. MAE versus number of judges on continuous-score tasks; lower is better. Bands show standard deviation across subset means. Vertical dashed lines mark the main panel size.
Figure 16. Accuracy versus number of judges on binary/preference tasks; higher is better. Bands show ±1 standard deviation across subset means. The vertical dashed line marks the main panel size. CARE-Tensor is unavailable at .
Calibration Budget for Targeted Nuisance Removal
We examine how the number of calibration pairs affects sensitivity to the beauty and authority biases studied in § 5.5.
Setup. Following the protocol in Appendix § 11.§ 6, we vary the budget from 4 to 64 pairs using nested calibration sets within each training fold. We keep the evaluation folds and fixed. We report mean absolute score shifts, averaging first over five folds and then over ten seeds. Variability is measured by the standard deviation across seed means. The main comparison uses the 32-pair setting.
Results. Figure 17 shows that increasing the number of calibration pairs improves suppression of the targeted nuisance. This is consistent with Theorem 2, which bounds the sampling error in the calibration matrix as the pair budget grows. With 32 pairs, VoirDire achieves lower sensitivity than all baselines on both beauty and authority (Table 3).
Figure 17. Sensitivity to injected biases versus calibration pairs, with . Error bars show standard deviation across ten seed means. Baselines use no calibration pairs. Lower is better.
Effect of Calibration on Other Aggregators
We extend Section 5.2 to examine whether score calibration benefits other aggregation methods.
Setup. We follow Section 5.1, applying each aggregator to both raw and calibrated judge scores with the same selected interventions and calibration settings. MV and ISP receive calibrated scores mapped to the original score levels. The intervention-aware aggregation conditions follow § 5.2. When the selected , raw and calibrated scores coincide.
| Dataset | Condition | MV | AVG | ISP | CARE-SVD | PCA | Intervention-aware |
| Yelp | Raw | ||||||
| Calibrated | |||||||
| SummEval | Raw | ||||||
| Calibrated | |||||||
| Summarize from Feedback | Raw | ||||||
| Calibrated | |||||||
| HelpSteer2 | Raw | ||||||
| Calibrated | |||||||
| ASSET | Raw | ||||||
| Calibrated | |||||||
| Review-5K | Raw | ||||||
| Calibrated | |||||||
Table 12. Effect of score calibration on continuous-score aggregation. Entries show mean standard deviation of MAE across five splits; lower is better. , so positive values indicate lower calibrated MAE. The calibrated intervention-aware aggregator is VoirDire; its raw counterpart retains the intervention penalty. Fitting details appear in Appendix § 16.§ 4.
| Dataset | Condition | MV | AVG | ISP | CARE-SVD | CARE-Tensor | PCA | Intervention-aware |
| Chatbot Arena | Raw | |||||||
| Calibrated | ||||||||
| RewardBench 2 | Raw | |||||||
| Calibrated | ||||||||
| LIAR2 | Raw | |||||||
| Calibrated | ||||||||
| HelpSteer3 | Raw | |||||||
| Calibrated | ||||||||
| CivilComments | Raw | |||||||
| Calibrated | ||||||||
| MT-Bench (human) | Raw | |||||||
| Calibrated | ||||||||
Table 13. Effect of score calibration on binary/preference aggregation. Entries show mean standard deviation of accuracy across five splits; higher is better. , so positive values indicate improvement. Accuracy and differences are reported as proportions. The calibrated intervention-aware aggregator is VoirDire; its raw counterpart retains the intervention penalty. Fitting details appear in Appendix § 16.§ 4.
Results. Tables 12 and 13 show that calibration benefits depend on both the dataset and the aggregation rule. Every aggregator improves on Review-5K, whereas on Yelp, calibration improves PCA and intervention-aware aggregation but worsens MV, AVG, ISP, and CARE-SVD. For intervention-aware aggregation, calibration lowers MAE on all four continuous datasets with and improves accuracy on three of the five corresponding binary/preference datasets. Thus, score calibration can benefit other aggregators, but is not a uniformly beneficial preprocessing step.
References
- [1]01.AI. Yi: Open foundation models by 01.ai. arXiv preprint arXiv:2403.04652, 2024.
- [2]Eldar David Abraham, Karel D’Oosterlinck, Amir Feder, Yair Ori Gat, Atticus Geiger, Christopher Potts, Roi Reichart, and Zhengxuan Wu. CEBaB: Estimating the causal effects of real-world concepts on NLP model behavior. In Advances in Neural Information Processing Systems, volume 35, 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/701ec28790b29a5bc33832b7bdc4c3b6-Abstract-Conference.html.
- [3]Dyah Adila, Changho Shin, Linrong Cai, and Frederic Sala. Zero-shot robustification of zero-shot models. In International Conference on Learning Representations, volume 2024, pp. 34402–34434, 2024.arxiv.org/abs/2309.04344
- [4]Rui Ai, Yugi Pan, David Simchi-Levi, Milind Tambe, and Haifeng Xu. Beyond majority voting: LLM aggregation by leveraging higher-order information. In Proceedings of the 43rd International Conference on Machine Learning, 2026.arxiv.org/abs/2510.01499
- [5]Alibaba. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024.
- [6]Alibaba. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
- [7]Fernando Alva-Manchego, Louis Martin, Antoine Bordes, Carolina Scarton, Benoît Sagot, and Lucia Specia. ASSET: A dataset for tuning and evaluation of sentence simplification models with multiple rewriting transformations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4668–4679. Association for Computational Linguistics, 2020. URL https://www.aclweb.org/anthology/2020.acl-main.424.
- [8]T. W. Anderson and Herman Rubin. Statistical inference in factor analysis. In Jerzy Neyman (ed.), Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, Volume 5: Contributions to Econometrics, Industrial Research, and Psychometry, pp. 111–150. University of California Press, Berkeley, 1956.
- [9]Krishnakumar Balasubramanian, Aleksandr Podkopaev, and Shiva Prasad Kasiviswanathan. Dependence-aware label aggregation for LLM-as-a-judge via Ising models, 2026. URL https://arxiv.org/abs/2601.22336.
- [10]Rajendra Bhatia. Matrix Analysis, volume 169 of Graduate Texts in Mathematics. Springer, New York, 1997. doi: 10.1007/978-1-4612-0653-8.DOI
- [11]Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Wasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion Proceedings of The 2019 World Wide Web Conference, pp. 491–500, 2019.arxiv.org/abs/1903.04561
- [12]Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or LLMs as the judge? A study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 8301–8327, Miami, Florida, USA, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.474.
- [13]Zeming Chen, Qiyue Gao, Antoine Bosselut, Ashish Sabharwal, and Kyle Richardson. DISCO: Distilling counterfactuals with large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pp. 5514–5528. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.acl-long.302. URL https://aclanthology.org/2023.acl-long.302/.DOI
- [14]Chandler Davis and W. M. Kahan. The rotation of eigenvectors by a perturbation. iii. SIAM Journal on Numerical Analysis, 7(1):1–46, 1970. doi: 10.1137/0707001.
- [15]A. P. Dawid and Allan M. Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society Series C: Applied Statistics, 28(1):20–28, 1979. doi: 10.2307/2346806.
- [16]Google DeepMind. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024.
- [17]Google DeepMind. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025.
- [18]Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409, 2021. doi: 10.1162/tacl_a_00373. URL https://aclanthology.org/2021.tacl-1.24/.DOI
- [19]Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. CausaLM: Causal model explanation through counterfactual language models. Computational Linguistics, 47(2):333–386, 2021. doi: 10.1162/coli_a_00404. URL https://aclanthology.org/2021.cl-2.13/.
- [20]Amir Feder, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E. Roberts, Brandon M. Stewart, Victor Veitch, and Diyi Yang. Causal inference in natural language processing: Estimation, prediction, interpretation and beyond. Transactions of the Association for Computational Linguistics, 10:1138–1158, 2022. doi: 10.1162/tacl_a_00511.
- [21]Amir Feder, Yoav Wald, Claudia Shi, Suchi Saria, and David Blei. Data augmentations for improved (large) language model generalization. In Advances in Neural Information Processing Systems, volume 36, 2023. URL https://openreview.net/forum?id=dJZ3MvDw86.
- [22]David R. Karger, Sewoong Oh, and Devavrat Shah. Iterative learning for reliable crowdsourcing systems. In Advances in Neural Information Processing Systems, volume 24, pp. 1953–1961, 2011. URL https://papers.nips.cc/paper/4396-iterative-learning-for-reliable-crowdsourcing-systems.
- [23]Divyansh Kaushik, Eduard Hovy, and Zachary C. Lipton. Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations, 2020.
- [24]Vladimir Koltchinskii and Karim Lounici. Concentration inequalities and moment bounds for sample covariance operators. Bernoulli, 23(1):110–133, 2017. doi: 10.3150/15-BEJ730.
- [25]Haitao Li, Junjie Chen, Qingyao Ai, Zhumin Chu, Yujia Zhou, Qian Dong, and Yiqun Liu. CalibraEval: Calibrating prediction distribution to mitigate selection bias in LLMs-as-judges. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16537–16552. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.acl-long.808. URL https://aclanthology.org/2025.acl-long.808/.
- [26]Ruosen Li, Teerth Patel, and Xinya Du. PRD: Peer rank and discussion improve large language model based evaluations. Transactions on Machine Learning Research, 2024a. URL https://openreview.net/forum?id=YVD1QwRAj.
- [27]Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. Split and merge: Aligning position biases in LLM-based evaluators. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11084–11108. Association for Computational Linguistics, 2024b. doi: 10.18653/v1/2024.emnlp-main.621. URL https://aclanthology.org/2024.emnlp-main.621/.DOI
- [28]Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2511–2522. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.153. URL https://aclanthology.org/2023.emnlp-main.153/.
- [29]Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannah Hajishirzi, and Nathan Lambert. RewardBench 2: Advancing reward model evaluation, 2025. URL https://arxiv.org/abs/2506.01937.
- [30]Meta. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
- [31]Microsoft. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743, 2025.
- [32]Mistral. Mistral 7b, 2023. URL https://arxiv.org/abs/2310.06825.
- [33]Junsoo Park, Seungyeon Jwa, Ren Meying, Daeyoung Kim, and Sanghyuk Choi. OffsetBias: Leveraging debiased data for tuning evaluators. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 1043–1067. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.findings-emnlp.57. URL https://aclanthology.org/2024.findings-emnlp.57/.DOI
- [34]Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, 2 edition, 2009.
- [35]Alexander Ratner, Christopher De Sa, Sen Wu, Daniel Selsam, and Christopher Re. Snorkel: Rapid training data creation with weak supervision. Proceedings of the VLDB Endowment, 11(3):269–282, 2017. URL https://www.vldb.org/pvldb/vol11/p269-ratner.pdf.
- [36]Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, and Soroush Vosoughi. Judging the judges: A systematic study of position bias in LLM-as-a-judge. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pp. 292–314. The Asian Federation of Natural Language Processing and The Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.ijcnlp-long.18. URL https://aclanthology.org/2025.ijcnlp-long.18/.DOI
- [37]Changho Shin, Winfred Li, Harit Vishwakarma, Nicholas Carl Roberts, and Frederic Sala. Universalizing weak supervision. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=YpPiNigTzMT.arxiv.org/abs/2112.03865
- [38]Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In Advances in Neural Information Processing Systems, 2020.arxiv.org/abs/2009.01325
- [39]Victor Veitch, Alexander D’Amour, Steve Yadlowsky, and Jacob Eisenstein. Counterfactual invariance to spurious correlations in text classification. In Advances in Neural Information Processing Systems, 2021.arxiv.org/abs/2106.00545
- [40]Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating LLM generations with a panel of diverse models. arXiv preprint arXiv:2404.18796, 2024. doi: 10.48550/arXiv.2404.18796. URL https://arxiv.org/abs/2404.18796.
- [41]Yidong Wang, Zhuohao Yu, Wenjin Yao, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. PandaLM: An automatic evaluation benchmark for LLM instruction tuning optimization. In International Conference on Learning Representations, 2024a. URL https://openreview.net/forum?id=5Nn2BLV7SB.arxiv.org/abs/2306.05087















