betaFeedback

WorksComputer Science

Evaluation Contracts and Evaluation Leverage: Sliced-Gap Readings in Structured Prediction Can Move Without Model Change

Abstract

NLP evaluation reports routinely cite sliced performance differences, most prominently the intra-inter gap in document-level relation extraction, as capability evidence. We show that such gap readings are joint properties of models and evaluation contracts: scoring function, gold version, threshold, output format, and prompt language. With predictions frozen, a method ordering reverses under a gold-standard exchange (+0.0363, CI excluding zero; tie-level after), movement localizing where gold re-partitions: renames are gauge transformations (Delta=0; re-annotation +0.862). We formalize this as an evaluation leverage account centered on the rank-flip condition. Constructively, the audited evaluation reading (AER) is a contract-marginalized, tie-declared, seed-prescribed, gate-filtered procedure whose tie declarations cover every contract-induced flip on the declared WMT19 grids (14/14; held-out 73% vs. union-of-ties 89%), matching the best single metric contract at epsilon=0.10 equivalence. Evidence spans DocRE, HacRED, a UD probe, and released gold revisions; an alternative-reference-set replication extends it beyond our testbeds. Failed predictions retained as boundaries; a 5% violation gate flags unverifiable settings and bounds survivor-selection bias. We release EvalLever, a preregistered workflow with an amendment trail; on LLM-as-judge grids, the audited ensemble stabilizes every unstable pair (13-15 to zero; generic ensemble shrinkage: the preregistered correlation criterion failed, the companion stabilization criterion passed).

Introduction

Evaluation reports steer NLP, yet their readings are increasingly distrusted: rankings disagree across

benchmarks, results fail to replicate under seed resampling, and selective reporting skews the literature. Treated separately, these ailments share a root cause: for a widely used family of readings (sliced performance differences, the sliced-gap readings of the title), each reading is a function of evaluation contracts that are rarely frozen, reported, or audited. Each mechanism has been observed piecemeal: gold revisions reorder rankings [57], metric reuse distorts readouts [45], prompt formatting shifts scores [52], label-error audits shift reported performance [38, 39], but as unrelated anomalies. To our knowledge, such choices have not been formalized as a contract object with measurable coordinates, a derived invariance, a registered flip condition, or tie-aware readings covering contract-induced flips, the steps this paper takes; frozen-prediction re-scoring under released golds has precedents [40, 22, 1] that we extend, not replace. Propositions 2–3 and § 6–§ 7 respond in turn.

Consider two published methods for document-level relation extraction (DocRE) (Yao et al., 2019) (an ATLOP-style encoder (Zhou et al., 2021) and the PU-metric-learning model P3M) on the same 498-document DocRED development subset (189,556 frozen candidate pairs). Under the original DocRED gold standard (gD), the intra–inter gap of the ATLOP-style variant exceeds P3M’s by +0.0309 (CI [+0.0167, +0.0455], excluding zero). Switch only the gold standard, to the revised Re-DocRED projection (gR), published precisely to fix false negatives, and the ordering reverses (−0.0053, CI [−0.0190, +0.0086]); the flip magnitude +0.0363 (CI [+0.0204, +0.0529]) excludes zero. Neither model changed a single prediction; the arm-level movement replicates on a held-out revised population (§ 3.3). A claim of the form “our model narrows the intra–inter gap” is thus not guaranteed to be a property of the models: it is jointly a property of the models and a contract the claim never states. Figure 1 previews both readings with paired CIs: the same comparison reads as an edge, a tie, or a reversal depending on the contract alone.

Figure 1. One claim, two readings (doc-paired bootstrap 95% CIs; whiskers crossing or touching zero declare a tie; shading: ±\pm.0039 floor (sign-symmetrized), § 6). Point-estimate flip magnitude +0.0363 [+0.0204, +0.0529]. (a) Reconstructed pair Γ\Gamma(ATLOP-PE8)−Γ-\Gamma(P3M): under the released gold gDg_D (light) the interval excludes zero; under the re-annotated gRg_R (dark) it crosses zero: the edge survives only at tie level. (b) Published claim F1(DREEAM)−-F1(ATLOP): the gDg_D reading is tie-level (interval touching zero) while the gRg_R interval excludes zero on the opposite side: the ordering reverses without any model change. Unlike point-estimate rescoring, intervals separate edges from reversals.

We contribute (i) a formal account of evaluation leverage (four propositions with explicit assumptions and tested boundaries, all projections of one reading ρ(M,c)\rho(M,c)), stating when contract changes move readings and rankings, and when they provably cannot: rename-only revisions are gauge transformations whose arm is zero by derivation (§ 2); (ii) a demonstration across task families: with predictions frozen, a reconstructed ordering reverses under the gold standard at the point-estimate level, arms recur across the remaining coordinates (flips only where arms are long and signs contrastive), with a violation surface for prompted predictors; negative results retained (§ 3–§ 6); (iii) a construction: the audited evaluation reading (AER: contract marginalization, tie declarations, a measurement prescription, and a validity gate), validated on official human anchors, matching the best single metric contract at ϵ=0.10\epsilon=0.10 equivalence on the WMT19 grids (a point-level edge on de-en only), tie declarations covering every flip on the declared grids (73% held out; § 4); and (iv) a prescription: freeze, state, and audit the contract, with seed prescriptions, evidence tiers, and EvalLever1 (§ 6–§ 7). Together they form a normative baseline (freeze, report with ties, gate) at checklist cost (§ 7) on any task satisfying the applicability criterion. Where the data contradict a preregistered prediction we say so. Scope: deeply validated on DocRE, HacRED, and a UD probe; boundary results (§ 5) mark where it fails.

Related Work

Gold revision and evaluation bias. Gold revisions reorder rankings: Re-DocRED Tan et al., [57], TACREV/Re-TACRED Alt et al., [1]; Stoica et al. [55], label-error re-annotation Nahum et al., [38]; Neuhaus and Hein [39]; Gema et al. [22], and re-scoring under corrected labels Northcutt et al., [40]. ReCOGS’s rename-invariance (Wu et al., 2023) and guideline-centered annotation Ruggeri et al., [50] presage our gauge principle. We complement: holding predictions fixed, we measure the leverage of switching among released golds, rather than producing a corrected dataset. Closest in spirit, Opitz [41] and Chen and Tanaka-Ishii [9] diagnose under-specified evaluation; we quantify how much a reading moves across the legal alternatives.

Variance, seeds, and significance. Seed variance and significance protocols address training-side variance Bouthillier et al., [4]; Dror et al. [18]; Card et al. [6]; conformal evaluation attaches per-reading coverage (Zerva and Martins, 2024). We find variance structured by the randomness level of its effect path (§ 3.3).

Rank stability. Rank reversal under benchmark perturbation, prompt formatting, and answer extraction is the empirical backdrop Bouthillier et al., [4]; Dodge et al. [17]; Mizrahi et al. [35]; Sclar et al. [52]. Closest to our constructive layer, Ceron et al. [7] gate readouts on CI indifference bands; AER instead registers and marginalizes over contracts, with ties and prescribed seeds.

LLM-as-judge and prompted evaluators. LLM judges correlate with humans [31, 59] (Zheng et al. 2023) while positional and self-preference biases persist [60] (Wataoka et al. 2024); none audits the judge-side axes our framework treats as coordinates (§ 4). Concurrent work [8, 42] and Gilda and Gilda [23] likewise call scores perishable; we operationalize the apparatus: contract object, invariance, frozen-prediction audits, constructive reading.

Metric-choice sensitivity and this account. Metric-choice sensitivity extends beyond NLP [36, 16, 33, 27, 54, 26]; the WMT metrics tasks evaluate evaluators annually [28, 44]. Each examines one instrument facet; none holds predictions fixed while switching among officially released alternatives with arms, registered flip checks, and a derived invariance (Table 12, appendix).

Line of workPred. frozenLegal goldDer. inv.Tie aware
Metrics tasks (2023)✓✗✗✗
Label-error re-scoring (2021)✓✗✗✗
Gold re-scoring (2025)✓✓✗✗
Gold revision: TACREV (2020)✓✓✗✗
Gold revision: Re-TACRED (2021)✓✓✗✗
Input perturbation (2024)✗✗✗✗
Metric resampling (2021)✓✗✗✗
This paper✓✓✓✓

Table 12. What each evaluation-audit line of work leaves unmeasured. Pred. frozen = scoring over fixed outputs; legal gold = switching among officially released gold-side alternatives only (scorer-side and format-side alternatives are tracked under Pred. frozen; a gold revision qualifies when it is a human-expert re-annotation under a written protocol, officially released—TACREV does; Re-TACRED additionally revises the guideline (41→40 inventory), counted as legal with that caveat; model-in-the-loop relabelings and ad-hoc corrections do not, §3.1); derived inv. = a proven invariance separating normative from substantive changes; tie aware = ties as first-class ranking output. Pred. frozen covers ss, gg, τ\tau; ff, ℓ\ell regenerate outputs by construction (§ 2.1).

An Evaluation Leverage Account

(notation summary: Appendix Table 9.)

SymbolMeaning
c=(s,g,τ,f,ℓ)c=(s,g,\tau,f,\ell)contract: scoring, gold, threshold, format, language
ρ(M,c), Γ\rho(M,c),\ \Gammareal-valued reading; the sliced gap (intra − inter) is its primary instance
arm ∣ΔΓ∣|\Delta\Gamma|realized movement under a contract switch
AA,BA_{A,B}error-set agreement between models (Jaccard; § 2.1)
arm⁡D\operatorname{arm}_{D}disagreement-referenced leverage over D(g,g′)D(g,g') (§ 2.1)
flipsign change of Γ(A,c)−Γ(B,c)\Gamma(A,c)-\Gamma(B,c) across contracts
violation, 5% gateoutput failing structural grounding; prereg. verifiability bound
V1–V4randomness levels of the effect path
n∗n^{*}prescribed seed count (sign determination)
gD / gRoriginal / revised DocRED gold versions
ESS / ASSearliest_mention / any_shared_sentence grouping

Table 9. Notation used throughout the paper.

The four propositions are projections of one reading ρ(M,c)\rho(M, c): Proposition 1 fixes cc and varies MM; Proposition 2 evaluates the same reading at two contracts and asks when model orderings survive; Proposition 3 decomposes the variance of ρ\rho across randomization levels; Proposition 4 separates its geometry from output validity. Proposition 2, with the rename invariance of § 2.1, is the account’s center. Status: the rename invariance (§ 2.1) is derived, and Propositions 1–3 each carry a derived formal core, Lemma A (the disagreement-set upper bound), Lemma B (the no-flip margin guarantee), and Theorem C (the variance ordering under nested randomization), whose assumptions and residual slack the empirical sections test; Proposition 4 remains observational.

Formalization

An evaluation contract c=(s,g,τ,f,ℓ)c = (s, g, \tau, f, \ell) specifies a scoring function ss, a gold-standard version gg (officially released alternates, not ad-hoc corrections), a decision-threshold policy τ\tau, an output format ff, and a prompt language ℓ\ell. The evaluation reading ρ(M,c)\rho(M, c) is any real-valued quantity an evaluator reports as capability evidence; the sliced gap Γ(M,c)=s(intra∣M,c)−s(inter∣M,c)\Gamma(M, c) = s(\text{intra} \mid M, c) - s(\text{inter} \mid M, c) (neither a property of the model alone nor of the contract alone, but of their interaction) is the primary instance studied. Propositions are stated on ρ\rho; empirics instantiate it at Γ\Gamma (other instances in § 3.4) unless noted. Throughout: arm =∣Δρ∣= \lvert\Delta\rho\rvert (movement under a contract change); flip (rank reversal); violation (output failing structural grounding; notation: Table 9, appendix). Predictions are fixed for s,g,τs, g, \tau; f,ℓf, \ell regenerate outputs; the realizing parser is a de-facto sixth axis folded into ff and audited. Fixed coordinates (κ\kappa). Beyond cc, readings depend on fixed coordinates κ\kappa (aggregation level, pool composition, anchor set, evaluated population); the intra/inter grouping rule rides on ss, declared switchable (H-G1, Table 10). All experiments freeze κ\kappa and disclose it with the registry; the axis list is not claimed exhaustive. Operation levels. A contract switch replacing one contract with another is axis-aligned when the two differ in exactly one coordinate; the legal contract pair (c1,c2)(c_1, c_2) is the object over which arms and the flip condition (§ 2.3) are quantified; all switches below are axis-aligned unless stated.

Rename invariance (gauge). Statement. A gold revision that only renames the label inventory, g′=r∘gg' = r \circ g for a bijection rr on labels, applied by convention to both sides of every pair (gold and predictions relabeled by the same rr; the conjugate action), cannot move a reading whose scorer factors through the joint (prediction, gold) partition symmetrically (accuracy and full-inventory micro-/macro-F1 do; subset-restricted variants that exclude a distinguished class are name-anchored and satisfy the premise only when rr preserves the inventory). Proof. The rename applies one common label permutation to both indices of the joint-partition cell counts, which symmetric statistics cannot see; hence ρ(M,(s,g′,τ,…))=ρ(M,(s,g,τ,…))\rho(M, (s, g', \tau, \ldots)) = \rho(M, (s, g, \tau, \ldots)) exactly (full proof: Appendix B). A gold-side-only relabeling (predictions still matched by their unchanged string identity) would move accuracy; the synthetic check below returns Δ=0.000000\Delta= 0.000000 under the implemented convention, confirming the double-sided application. Consequences. Only re-partitioning revisions can move a reading; scorers referencing label identity outside the gauge (name-anchored class weights, soft string matching) do not factor through the partition invariantly and can move under renames alone; psychometrics’ counterpart is facet-level invariance (generalizability theory; measurement invariance Meredith, [34]). A synthetic six-class check confirms both halves (accuracy/micro-/macro-F1 0.000000 exactly under a pure rename; −0.003 for a name-anchored weight vector), as do the UD slices (renames Δ=0.000\Delta= 0.000, re-annotation +0.862) and the exact character/word null (B2, § 5).

Three quantities circulate under the name “arm”: the error-set agreement (Jaccard similarity of error sets, a between-model quantity), the disagreement-referenced leverage arm⁡D=∣Y^∩D∣/∣D∣\operatorname{arm}_{D} = |\hat{Y} \cap D|/|D| over the two golds’ disagreement set (model-vs-gold; the two need not agree, B1), and the realized shift ∣Δρ∣|\Delta\rho|, an ex-post outcome. Henceforth an unqualified “arm” denotes ∣Δρ∣|\Delta\rho| (Table 13, appendix).

SettingDensityArm ∣ΔΓ∣|\Delta\Gamma|CI95
X20.2040.022[+0.008, +0.036]
ATLOP-PE80.0480.063[+0.044, +0.082]
ATLOP-full0.0470.043[+0.023, +0.063]
DREEAM0.0270.001[−0.023, +0.024]
P3M0.0800.026[+0.006, +0.047]
DocuNet0.0420.016[−0.036, +0.005]
KD-DocRE0.0300.013[−0.011, +0.037]
Mistral-7B ft0.0940.002[−0.003, +0.008]
Mistral-7B json0.0160.000[−0.009, +0.009]
Qwen-14B json0.0290.006[−0.002, +0.015]
Qwen-7B ft0.0420.015[+0.006, +0.024]
Qwen-7B json—0.017(reg.)

Table 13. Gold-swap leverage arms for the twelve generator settings (Proposition 1): prediction density and the realized gold-swap shift; CIs are 95% document-bootstrap on the signed ΔΓ\Delta\Gamma. Density–arm Spearman +0.64 across the eleven measurable-density settings (+0.68 within encoders; descriptive pp). (reg.) = registered-only arm, released without a bootstrap CI.

The instruments need two testbed properties: legal contract pairs (same-data changes any evaluator could equally have chosen) and a dense disagreement set between alternatives; structured prediction supplies both, single-gold classification neither (B1, § 5). The applicability criterion (sufficiency tested, necessity not): retrieval and MT satisfy both; code generation the contract-pair criterion.

Generator-Dependent Leverage

Statement. With data and the switched axis fixed, only the predictor varying, the realized shift ∣Δρ∣|\Delta\rho| grows with prediction–disagreement overlap; the claim is made at family resolution (fine-tuned encoders vs. prompted models, plus two within-family slices), with no monotone curve claimed (tested across the full contract registry: § 3.2).

Formal core (Lemma A: disagreement-set upper bound). For affine count aggregates with 1-Lipschitz outer map (accuracy; micro-/macro-F1 are ratio functionals and escape the premise3, Appendix § 11), ∣ρ(M,c1)−ρ(M,c2)∣≤B∣D(g1,g2)∣|\rho(M,c_1)-\rho(M,c_2)| \le B|\mathcal{D}(g_1,g_2)| deterministically; ∣D(g1,g2)∣/N|\mathcal{D}(g_1,g_2)|/N for accuracy (B=1/NB=1/N). A model-dependent refinement:

∣Δρ∣≤2arm⁡D∣D(g1,g2)∣/N|\Delta\rho| \le2\operatorname{arm}_{D}|\mathcal{D}(g_1,g_2)|/N

coupling the bound to prediction–disagreement overlap (the quantity Proposition 1 varies). The bound is monotone in ∣D∣|\mathcal{D}|, the realized shift need not be (sign cancellation; Appendix § 11). Empirical surplus. The family separation below is an ordering consistent with the envelope’s slack structure (accuracy instance), not an ordering of the bound itself.

Assumptions. Prediction-fixing levers (s,q,τ)(s,q,\tau) leave outputs unchanged; f,ℓf,\ell regenerate outputs and are treated separately.

Evidence and boundary. § 3.2 (Table 13, appendix): the family-level separation and two within-family slices are the supported content; open-ended generation is untested scope.

Rank-Flip Condition

Statement. For models AA, BB and legal contracts c1,c2c_1,c_2, a rank flip occurs when sign⁡(ρ(A,c1)−ρ(B,c1))≠sign⁡(ρ(A,c2)−ρ(B,c2))\operatorname{sign}(\rho(A,c_1)-\rho(B,c_1)) \ne\operatorname{sign}(\rho(A,c_2)-\rho(B,c_2)). Assumptions. Arms and margins are computed on the same frozen predictions over legal contract pairs, under the document-paired resampling of §4.2. A flip requires comparable margins (the necessary condition, Lemma B below) and is empirically associated with long arms (low error-set agreement AA,BA_{A,B}): a far-apart pair does not reorder even with nearly disjoint errors. Flips concentrate among statistically tied pairs: a dominant system shows level-arms without reversals (§4.4), predicting where rankings move and where they cannot (human agreement similarly collapses on near-tie systems [24]); all checks and Holm survivors: § 3.2.

Formal core (Lemma B: no-flip margin guarantee). Whenever ∣δc1∣>arm⁡(A)+arm⁡(B)|\delta_{c_1}| > \operatorname{arm}(A) + \operatorname{arm}(B), the signs of δc1\delta_{c_1} and δc2\delta_{c_2} coincide: no flip is possible (triangle inequality; proof: Appendix § 11). Empirical surplus. Flips occur only where the margin is within the combined arms. AER’s protection is two-layered and the regions differ: tie declarations (CI-based, resample layer; empirically covering 14/14 registered flips) and the arm/margin audit (point layer, this Lemma’s region); the former does not imply the latter.

Evidence. § 3.2: thirteen flips among 57 checks, concentrated where the arm is long and signs contrastive; a leave-one-out check shows the condition carries flip-location information.

Proposition 3: Variance–Level Ordering and a Seed Prescription

Statement. The across-seed SD of a contrast is ordered by the level of randomness entering its effect path: V1 (scoring changes) ≤\leq V2 (decoding changes) ≤\leq V3 (training sources) ≤\leq V4 (double differences), strictness left to the data. (A discretized generalizability-style decomposition [15, 5]; the four-benchmark test and the prescription it induces: § 3.3.)

Formal core (Theorem C: variance ordering under nested randomization). Nested effect paths with independent mean-zero increments give Var⁡(ρk)=∑i≤kVar⁡(εi)\operatorname{Var}(\rho_k) = \sum_{i \leq k} \operatorname{Var}(\varepsilon_i) by the law of total variance, hence V1≤⋯≤V4V_1 \leq\cdots\leq V_4 (proof: Appendix § 11). The theorem covers one contrast; ordering contrasts of different benchmarks additionally assumes comparable per-level increment magnitudes (tested alongside nesting and independence). Empirical content. The theorem fixes the assumptions the data must test: nesting and increment independence; the de-duplication cascade below is exactly the independence failure, and where the registered criterion fails.

Evidence. § 3.3 (Figure 7, appendix), with load-bearing caveats: the points are not independent (the registered ρ>0.5\rho> 0.5 fails the strictest de-duplication), and H3’s benchmarks were substituted by amendment 6.

Figure 7. Across-seed SD of 15 contrast points from four benchmarks, by random-source level (V1–V4). Lines: per-level medians; higher levels introduce more randomness into the reading—the ordering Proposition 3 formalizes. The pooled association is ρ=0.656\rho= 0.656, p=0.0045p = 0.0045, reported as secondary because the points are not independent; under the strictest de-duplication ρ=0.359\rho= 0.359 (p=0.27p = 0.27, n=6n = 6).

Corollary (seed prescription). On the convergence curve of a HacRED decoding-side double difference (V4, 15 seeds; § 3.3), the registered conservative track determines the sign at n∗=8n^*=8, the exact track at n∗=5n^*=5 (Figure 6, appendix), a preliminary, contrast-specific count (per-contrast MDE analysis [6]), conditional on equal-σ\sigma imputation (10/15 seeds; pooled ϕ^=0.34\hat{\phi}=0.34, validated to <3%< 3\%) but insensitive to ϕ^∈{0,…,0.80}\hat{\phi} \in\{0,\ldots,0.80\} (n∗=8n^*=8 throughout).

Figure 6. Seed prescription for the HacRED decoding-side double-difference contrast: fraction of nn-seed subsets whose pooled 95% CI excludes zero. The preregistered 95% criterion is met at n∗=8n^* = 8 under the conservative track (equal-σ\sigma imputation for 10/15 seeds); the exact document-bootstrap track gives n∗=5n^* = 5.

Proposition 4: Violations Are Orthogonal to Leverage

Statement. Contract-violation rates (outputs failing structural grounding) and leverage arms are independent dimensions of evaluation validity (§ 3.3; LLM-gate boundary: § 5).

Assumptions. Independence is assumed; the evidence is observational decoupling.

Evidence and boundary. § 3.3: format changes move violations by >10>10 points and arms by ±0.003\pm0.003; violations are graded; two independent-evidence settings come from gate-flagged cells, whose violation rates limit generalization. Observational decoupling under an assumption, not a theorem.

The three error sources (leverage bias, variance, violations) are validated separately below (anchor single-annotated; every segment-level claim carries that caveat). The decomposition is additive by construction with the cross-term vanishing empirically (r57); zz-standardization removes per-contract location shifts by construction, so it captures variance, not bias: at segment level human–instrument disagreement accounts for effectively all error; at the system level, where rankings live, contract switches move a system’s error by 0.0130.013–0.0430.043 (33–26%26\% of between-system variance), never dominating it; judge-side parse failures correlate with systematic bias (+0.33+0.33, fifteen cells): the selection bias the gate removes before marginalization.

Empirical Validation

Setup

Benchmarks and preregistration. All experiments follow a preregistration document frozen before any training or download (audit trail and verification records in the footnote 1 release); the released bundle ships with sha256 manifests. DocRED (Yao et al., 2019) contributes a 498-document dev subset with 189,556 frozen candidate pairs; its dev split carries 57.7% cross-sentence gold links (58.2% over the 498-doc retained subset), the slice we study. Two gold versions, gD and gR (revised projection of Tan et al., [57]), instantiate the gold-revision lever (registry: Table 4, appendix); HacRED (Cheng et al., [12]) (Chinese, 15 seeds) has one gold, its lever a decoding interaction; BC5CDR (Li et al., [29]) and RAMS (Ebner et al., [19]) are boundary benchmarks.

Bench.Cand.Grouping ruleGold
DocRED189,556 pairs (498 docs)earliest_mention: intra iff both mentions fall in one sentence (earliest-mention coreference)gDg_D, gRg_R
HacRED15 seedsGold segmentation; one intra iff both entities share a sentence
BC5CDR500/500/500Mention-pair expansion of concept-level CID gold; title = one sentenceofficial
RAMS5-sentence windowsIntra iff the argument is in the trigger’s sentenceofficial

Table 4. Contract registry: the grouping rule and gold versions behind every intra–inter reading in the paper.

Predictors. Table 8 lists the seven families; X2 is an instrument, a controllable prediction source, not a contributed system.

FamilyConfigurationSeedsSource
X2BERT-base, two sparsity levels5internal
ATLOP-PE8 / -fullprotocol pair (PE8 drops calib. shard)3+3retrained
DREEAM, DocuNet, KDsame PE8, F1 .577–.6132–3retrained
P3MPU metric learning, shared scoring5retrained
LLMs7/14/32B + Mistral + Phi-4, 4-shot1/cellreleased

Table 8. Seven generator families; all predictions fixed and released, no retraining for the contract comparisons. ATLOP variants from Zhou et al. (2021), P3M from Wang et al. [60], LLMs from Qwen Team [48]; Jiang et al. [25]; DREEAM, DocuNet, KD-DocRE retrained from official code (Ma et al. [32]; Chen et al. [11]; Tan et al. [56]). LLM prompt configuration, the post-review arms (TTM-RE, DocuNet-L; Table 16, appendix), and per-model values are in the supplement.

Statistics. All intervals: document-paired bootstrap. The three preregistered comparisons (Holm(3)) all reject at α=0.05\alpha=0.05: C1, the paired-unit D-score on historical single-output models (.036 [.021, .050]; p=.0006p=.0006); C2, the P3M gold-swap arm (scoring-dependent, MDE .44; p=.0024p=.0024); C3, the HacRED decoding-side contrast (sign unstable, negative in 13/15 seeds; p=.011p=.011).

Leverage and Rank Flips

Four questions structure this section (one per proposition; the flip registry is exploratory throughout; the confirmatory layer is C1–C3, above); four checks rule out slice selection, grouping, format, and population; three probes carry the axes beyond DocRE (arms: the twelve-setting pool; census: the seventeen LLM settings).

Generator-dependent arms (Proposition 1). Across twelve generator settings4, within-model format changes move arms by at most ±0.003\pm0.003, family-level separation tracks prediction–disagreement overlap rather than capability (7B→\to14B lowers both), and density tracks the arm as a tendency, not a law, with two inversions precluding monotonicity (density–arm Spearman +0.64+0.64; Table 13, appendix).

A training-investment confound remains (bounded by three checks; supplement).

Rank flips (Proposition 2). Among 57 contract-pair checks over twenty-one encoder-family pairs, thirteen flips occur, eleven CI-excluding (threshold ten, gold one, the headline flip of § 1, plus two CI-including; scoring none; Figure 2; full registry: Table 14, appendix); the mass concentrates on the threshold axis; all ten threshold-axis flips involve the fixed-τ=0.5\tau=0.5 vs. decoded-threshold contrast, a policy choice some systems optimize internally.

Figure 2. The thirteen registry flips (Proposition 2): flip magnitude with CI95, by axis; filled markers survive Holm (10 of the 11 CI-excluding). Flips concentrate on the threshold axis and among near-tied pairs.

PairDim.Δ\DeltaCI95≠0\ne0Holm
Doc|KDthr-0.259[−0.296, −0.221]✓✓
DRE|Docthr0.238[+0.200, +0.276]✓✓
PE8|Docthr0.191[+0.162, +0.219]✓✓
AT|Docthr0.190[+0.162, +0.218]✓✓
X2|Docthr0.096[+0.081, +0.111]✓✓
AT|KDthr-0.069[−0.086, −0.052]✓✓
PE8|KDthr-0.069[−0.086, −0.051]✓✓
AT|DREthr-0.048[−0.064, −0.032]✓✓
PE8|DREthr-0.048[−0.064, −0.032]✓✓
PE8|P3Mgold0.036[+0.020, +0.053]✓✓
DRE|KDthr-0.021[−0.035, −0.007]✓—
X2|DREgold0.021[−0.005, +0.048]——
DRE|KDgold-0.012[−0.026, +0.002]——

Table 14. Rank-flip registry: all thirteen contract-pair flips over the 57 testable checks (Proposition 2). Pairs abbreviated: PE8 = ATLOP-PE8, AT = ATLOP-full, DRE = DREEAM, Doc = DocuNet, KD = KD-DocRE. Dim.: thr = threshold (fixed vs. decoded), gold = gold version, score = scoring function. Holm: survives the family correction (34 of 57 tests survive).

The pattern matches the leverage condition (longest threshold arms: KD-DocRE, DREEAM, ATLOP variants; calibration raises DocuNet’s gap while collapsing KD-DocRE’s) and modestly exceeds redescription: a single-threshold classifier on arm length reaches leave-one-out accuracy 0.860 vs. majority 0.772 (p<0.002p < 0.002). Under Holm 34 of 57 survive, as do 10 of the 11 CI-excluding flips; Kendall τ∈[−0.333,+1.000]\tau\in[-0.333, +1.000] across the registry’s 15 contract dimensions (supplement).

Auditing published orderings. The same machinery re-scores three published claims under both golds (Table 1; r56, gold-axis recomputation v2): one of three reverses under the revised gold, and the audit separates contract from population effects.

ClaimgDgRVerdict
DREEAM>ATLOP+.0061−.0215reversed
KD-DocRE>DocuNet+.121+.044holds, arm :3
TTM-RE>KD-DocRE−.048+.292opposite

Table 1. Three published orderings re-scored under both golds (frozen predictions from retrained checkpoints; paired reading differences instantiated at Γ\Gamma, the F1-based scorer at τ=0.5\tau=0.5, where the published F1 claims live; CI95; reversed = gD tie, gR CI-excluding; claims from Ma et al., [32]; Tan et al., [56]; Gao et al., [21]). Shading: reversed by the gold alone.

Variance Levels, Violations, and Further Axes

Variance-level ordering (Proposition 3). The pooled 15-point level-vs-SD Spearman is rs=0.656r_s = 0.656 (one-sided permutation p=0.0045p = 0.0045), with monotone level medians across V1–V4 (Figure 7). The points are not independent, however, and nine are calibrator-derived, so the pooled test is secondary: de-duplication gives rs=0.588r_s = 0.588 (p=0.017p = 0.017), rs=0.493r_s = 0.493 (p=0.044p = 0.044), and rs=0.359r_s = 0.359 (p=0.27p = 0.27, six non-calibration points); the registered rs>0.5r_s > 0.5 pass does not survive the strictest. Per benchmark the criterion is unmeet­able by construction (RAMS ρ=0.5\rho= 0.5; BC5CDR minimum attainable ρ=0.17\rho= 0.17 at n=3n = 3): a failed registered sub-hypothesis. Within HacRED the 15-seed double difference shows the prescribed convergence (pooled-CI exclusion reaches 0.991 at n=8n = 8, Figure 6, appendix; exact track n∗=5n^{*} = 5), while the X2 source contrast stays sign-undetermined at n=10n = 10. The subset machinery replicates on the X2 family’s second contrast (criterion at n=5n=5 for ϕ=1.0\phi=1.0; undetermined for ϕ=0.25\phi=0.25 at all n≤10n \le10; r40).

Per-seed sign stability (preregistered H1). The preregistration predicted sign consistency below 0.8 at 15 seeds; the observed 0.867 (13/15 negative) contradicts the clause in the point estimate, though not significantly (P(X≥13∣p=0.8)=0.40P(X \ge13 \mid p=0.8) = 0.40; diagnostics: supplement).

Violation surfaces under LLMs (Proposition 4). Over the instruction-tuned grid (two models ×\times two formats, HacRED, NF4 quantization, greedy; extension adds 32B), three patterns matter. (i) The failure surface is a (model ×\times instruction language ×\times format) object: same-dataset controls give opposite answers for the two models: Qwen-7B barely moves between prompt languages, Mistral-7B swings 8.1×\times between formats (Table 5); the inventory channel is dataset-borne (1.7% English-DocRED vs. 13.8% English-instruction HacRED, same model). A doc-clustered decomposition attributes incidence to relation frequency (odds ratio [OR] 0.43) and model identity, not format (OR 1.02).

ModelPrompt–dataFRate%CI953%5%10%
Q14BDJ4.57[3.4, 6.0]—✓✓
P4H-zhJ7.36[6.0, 9.0]——✓
M7BDF7.05[5.3, 8.9]——✓
Q32BH-zhJ21.94[19.8, 24.2]———
M7BH-zhF10.91[9.4, 12.7]———
Q7BDJ11.40[9.4, 13.4]———
Q7BDF11.64[8.8, 14.6]———
M7BH-enF12.93[11.9, 14.0]———
Q7B†H-enJ15.74[14.4, 17.1]———
M7BH-enJ18.77[17.3, 20.4]———
M7BDJ18.84[15.3, 22.9]———
Q7BH-enJ19.76[17.9, 21.7]———
Q7BH-zhF21.76[19.7, 24.0]———
Q7BH-zhJ22.15[20.2, 24.1]———
Q7BH-enF23.01[20.8, 25.4]———
Q14BH-zhJ28.87[26.4, 31.5]———
M7BH-zhJ88.71[83.0, 93.9]———

Table 5. The preregistered 5% violation gate over all seventeen instruction-tuned settings (preregistered cells, one post-review BF16 cell, and the scale/family extension). Q7B/M7B/Q14B = Qwen-7B / Mistral-7B / Qwen-14B; prompt–data combination: D = English DocRED, H-zh / H-en = Chinese / English prompts on HacRED; format J/F = json / freetext. The last three columns give the gate verdict at 3% / 5% / 10% thresholds. † = post-review BF16 cell (amendment 66), outside the preregistered gate family; its CI follows the r24 protocol (amendment 75 extension). Decoding: greedy, max 768 new tokens, top-pp=1.0. Post-review stop-sequence diagnostic (amendment 82): constraining M7B in the collapsing cell to halt at the few-shot separator (zero truncations, 1,500/1,500) re-reads it at 54.97%—roughly a third of the collapse is unstopped-truncation artifact, the residual parse-failure-dominated (1,245 of 2,507 events); the character-vs-word violation-rate gap reads +0.0003 with L2 difference 0.000. Rows sorted by rate. P4 = Phi-4 (14B-class second family, capability-sensitive band per amendment 85); Q32B = Qwen-2.5-32B (scale point; both new constraint arms are byte-identical to their unconstrained pairs—the few-shot separator pattern never occurs at these scales).

Mistral-7B collapses in exactly one cell (Chinese json), re-read at 55.0% under a stop-sequence constraint: the violation reading itself is decoding-contract-relative. The two failure channels separate: collapse cells are parse-failure-dominated, while the six HacRED-side Qwen cells carry 55.9–78.1% of violation mass in the label-inventory channel: survivor-set arms therefore compare parseable populations. Selection on parseability is itself a contract effect: survivor-set arms are conditional on survival, fractions per cell: supplement. (ii) Violations are a (model ×\times contract) interaction: format relaxation rescues the collapsing cell, language switching roughly halves it, Qwen-7B unchanged (Table 5). (iii) Violations and leverage move independently: the lowest-violation setting carries a near-zero arm; 7B→\to14B lowers violations (11.4%→\to4.6%) and the arm (+0.0174 →\to +0.0063); the Chinese counterpart rises (Δ\Delta+6.7pt, CI [+3.8, +9.7]), wholly in out-of-inventory names (14.4%→\to22.6%): the larger renames the code, the smaller slips morphologically (supplement).

The 5% gate, applied uniformly. The gate marks a setting above 5% unverifiable: the conventional Type-I rate (verdicts at 3%/10% alongside, Table 5); by the per-pair counting bound (micro-accuracy), survivor-set selection bias is at most v∣asurv−aviol∣≤vv \lvert a_{\mathrm{surv}} - a_{\mathrm{viol}} \rvert\le v. Of seventeen settings one passes by point estimate; none passes a CI gate (Table 5); the extreme cell maps 192 of 1,701 triples, unreliable under every threshold column. Gated settings therefore supply descriptive evidence only (open markers in Figure 3), not confirmatory; smoke runs do not extrapolate.

Figure 3. Contract-violation rate vs. gold-swap arm for the five instruction-tuned settings with a gold pair (English DocRED prompts; bubble size ∝\propto mapped triples); open markers: flagged unverifiable by the preregistered 5% gate (four of five; the passing cell reads 4.57%). The four Chinese HacRED cells have no gold pair (rates: Figure 5, Table 5).

Figure 5. Violation surface (Proposition 4): contract-violation rates (%) over seven instruction-tuned settings—six Chinese HacRED cells plus the English-DocRED cell of Qwen-14B-json (D), the one setting passing the gate (its Chinese value: 28.9, Table 5); the dashed line is the preregistered 5% gate (full seventeen-setting census: Table 5).

Scale and family extension (preregistered bands). The preregistered extension (amendment 85) fixed two bands (15–35% = Qwen-profile replication, below 10% = capability-sensitive); both occur across families rather than scales (32B Qwen scale-robust, same-scale Phi-4 at 7.4%, family ×\times dataset-bound; Table 5); quantization replicates, every quantized cell above the gate.

The grouping rule itself (registered H-G1). The intra/inter split is itself a contract coordinate: recomputing every run under any_shared_sentence (ASS) vs. frozen earliest_mention (ESS) moves the readings of all eight generators with stored pair-level logits (every CI excluding zero, arms exceeding the gold-swap arm for all three public retrains; H-G1 PASS; Table 15). The direction is uniform (ASS shrinks the gap in every run), so no pair flips: a long arm with non-contrastive signs moves levels, not rankings (X2/P3M lack pair-level scores).

GeneratorRunsArmCI95Arm(gold)
ATLOP-PE860.030[0.020, 0.041]0.063
ATLOP-full60.028[0.017, 0.039]0.043
DREEAM30.021[0.009, 0.033]0.001
DocuNet20.029[0.018, 0.040]0.016
KD-DocRE20.020[0.008, 0.033]0.013

Table 15. The grouping-rule axis (registered H-G1, amendment 60; contract gRg_{\mathrm{R}}, each generator’s native decode): leverage arm ∣Γ(ASS)−Γ(ESS)∣|\Gamma(\mathrm{ASS})-\Gamma(\mathrm{ESS})| with document-bootstrap 95% CIs. ASS = any_shared_sentence, ESS = earliest_mention. Direction is uniform (ASS shrinks the gap in every run listed here and in all three post-review grouping levers of Table 16 (.018/.021/.022), so no pair flips). X2/P3M lack stored pair-level scores. Seeds per generator (3/3/3/2/2): two to three training seeds evaluated per seed pair; HacRED has no grouping axis.

Replication on an unseen population (gold axis). The gold-swap protocol replicates on a second same-source population: the 500 Re-DocRED test-revised documents (schema-identical to the dev release). Five families have retained checkpoints: the three dev-significant levers replicate (DocuNet-L 0.025 → 0.041, TTM-RE same-signed); near-zero arms stay near zero; a published-configuration RoBERTa-large ATLOP enters as a near-zero lever (0.021, CI [−0.048, +0.005]): the DREEAM profile at large scale, across five provenance tiers.

Slices and robustness. Robustness slices (relation frequency, document length, sentence distance) show the gap positive in all 91 slice tables across seven arms; population slices leave the sign fixed while contract axes move it, and UD label-revision slices localize movement to the re-annotated class (Table 19, appendix): slice choice does not carry the movement.

Armaccv2_{v2}accv1→v2_{v1\to v2}Δ\Delta [CI]Seeds
English EWT
C stanza.962.919+.043 [+ .040, +.046]1
A perc..928.887+.041 [+ .038, +.044]15+
B (BERT).927.890+.037 [+ .034, +.040]15+
RL probe.931.893+.038 [+ .035, +.041]15+
B′ (BERT).892.921−.030 [−.032, −.027]15−
A′ perc..886.917−.030 [−.033, −.028]15−
Chinese GSD
B (RoBERTa).910.900+.010 [+ .008, +.012]15+
A perc..889.880+.009 [+ .007, +.011]15+
B′ (RoBERTa).899.913−.014 [−.016, −.012]15−
A′ perc..886.900−.014 [−.016, −.012]15−

Table 19. UD cross-version gold axis (separately preregistered; judgments frozen before any arm ran). Identical text, two released gold versions: English EWT r1.2 vs. r2.18 test (2,070 of the official 2,077 sentences; 316 documents; zero tokenization drift) and Chinese GSD r1.3 vs. r2.0 dev (500 sentences, sentence-level bootstrap); the older gold is projected by the documented v1→v2 migration (no fitting; coverage 98.5%/100%). A/B = v2-contract training, A′/B′ = migrated-v1 contract (perceptron or frozen-feature probe, 15 seeds each); C = off-the-shelf stanza parser (checkpoint-conditional). RL = RoBERTa-large probe (355M; scale reinforcement—Δ\Delta sits between the base probes B and A). Δ=acc⁡(gv2)−acc⁡(gv1→v2)\Delta= \operatorname{acc}(g_{v2}) - \operatorname{acc}(g_{v1\to v2}), paired document bootstrap (10,000, seed 20260926). Training contract reverses the sign (60/60 seeds); rankings flip in 9/10 (EN), 6/6 (ZH). Flips are counted between training-contract arm families (A/B vs. A′/B′)—a within-regime object, not between released systems. Slices: renames Δ=0.000\Delta= 0.000 (731 tokens), re-annotated +0.862 (1,139), unchanged 0.000 (22,728); C-family LAS 0.881 → 0.724.

Cross-Task Replication

Six further task families replicate the audit under the same frozen-prediction discipline: retrieval on TREC DL 2019 ([14]) (r49), MT on WMT19 ([3]) (r50), QA on SQuAD v2 ([49]) (r51), code (r52), and the released-gold audits (r59/N1–N2); observational breadth for Props. 1–2, outside the preregistered ledger (readings and flip counts per family: Table 18, appendix).

Family / axisReadingArm or flips
Retrieval: gold rubricnDCG@10+0.137–0.158 (5 runs)
(graded → binarized)MAP@100, bpref0 (by construction)
MT de-en: scorerBLEU→chrF3/120 pairs flip
(16 frozen systems)BLEU→BERTScore10/120 flip, τ=0.83\tau=0.83
MT: formatBLEU 13a→none≤5.3\le5.3 BLEU
QA: gold populationF10.189 → 0.378
(v2.0 → v1.1-style)EM0.096 → 0.192
QA: scorerEM↔F1≈0.10\approx0.10; 1 pair flips
QA: thresholdEM vs. F1≥0.5\ge0.5≈0.12\approx0.12
Code: gold testspass@1HE 0.835 → 0.768
(base → EvalPlus)pass@1MBPP 0.780 → 0.648
(3 coder models)rankingno flip; Qwen leads both golds
Code: formatfenced vs. rawup to 0.84 of pass@1
Code: decodinggreedy vs. sampled≤0.018\le0.018 (n.s.)

Table 18. Cross-task replication summary (post-review, r49–r52, r59/N1–N3): frozen predictions, one contract switch at a time. Retrieval: five frozen runs, TREC DL 2019 (43 topics); zero flips in 66 gold×scorer pairs. MT: WMT19 (16/15 systems); newstest 2021–23, eleven into-English grids of 8–21 systems, consecutive-scorer flip rates 5.0% (tokenizer), 7.7% (chrF), 14.1% (chrF→BERTScore), full matrix 5.0–14.8%; 22 released En→De submissions under five human-written alternative reference sets (AR, AR.p, WMT.p, HQ(R), HQ(P); 21 with usable output) flip 12–24/210 pairs per scorer (r59/N3, E1). QA: three extractive models, SQuAD v2 dev. Code: three coder models, HumanEval/MBPP [10, 2] vs. EvalPlus [30], frozen samples n=5n=5–10. RE: 45 released TACRED predictions, TACREV/Re-TACRED gold gauges (subset-matched re-scoring −13.7 to −21.6, N1b). MCQA: five frozen models from three families, original vs. Redux-corrected labels (r59/N2). UD: two released UDPipe parsers, +0.045/+0.082 LAS under the v1→v2 gauge switch (r59/E3); AER-vs-single TOST and flip coverage (r59/E2); gauge-breaking scorer −0.003 under pure rename (r59/E5).

Three readings carry the section. First, gold-coordinate switches move readings wherever they are legal (Table 18; MAP/bpref exactly invariant under rubric binarization, r58), and the two TACRED revisions move the leaderboard leaders themselves ([1, 55]): over 45 released systems, 88/990 pairs flip between TACRED and its same-guideline re-annotation TACREV, CI-tie coverage 88/88 (r64). Second, the MT scorer and tokenizer axes persist ([47]) across newstest 2021–23 and the released alternative reference sets ([20]); ours replicates it (r59/E1). Third, flips concentrate where margins are near-tied (Table 18; 44 of the 88 TACRED flips sit inside the ±0.5pp slice) and far-apart pairs mostly shift levels; the output-format contract on code is larger still (up to 0.84 of pass@1). AER transfers to these grids as declared: retrieval (four contracts) and code show zero contract-induced flips, the QA EM↔F1 flip is ensemble-covered (r59/N4).

From Falsification to Construction: The Audited Evaluation Reading

The audited evaluation reading (AER) is the constructive response to the falsifications above, with four components: (i) marginalization: report the registered contract-set average ρˉ(M)=Ec∈CZc(ρ(M,c))\bar{\rho}(M)=\mathbb{E}_{c\in C}\mathcal{Z}_c(\rho(M,c)), Zc\mathcal{Z}_c the within-contract standardization over systems; (ii) tolerance: a pair is tied when a segment-bootstrap CI of the paired difference contains zero; orderings are claimed only for CI-excluding pairs (touching counts as tied); (iii) prescription: the seed and contract sample sizes needed for a target precision (§ 2.4); (iv) gate: readings failing the validity gate are excluded (§ 3.3; tooling in § 6). The leverage law applies to AER itself: leave-one-member-out moves its reading in [0.667, 0.700] (de-en) and [0.752, 0.771] (zh-en; there the chrF+BERTScore pair, 0.771, dominates the registered ensemble, 0.752; r55): member selection is itself a contract choice; AER claims are relative to the declared, frozen member set.

On the WMT19 de-en and zh-en grids (r60 canonical re-freeze; contract set {BLEU-13a ([43]; [47]), chrF ([46]), BERTScore ((Zhang et al., 2020))}; R=2000R=2000 segment resamples, ties re-declared at R=2000R=2000; anchor = official raw human scores.5 The ensembled reading agrees with the anchor at point τb\tau_b 0.700 on de-en (single contracts 0.683/0.667/0.650) and 0.752 on zh-en (singles 0.733/0.752/0.771), while carrying tie declarations. Equivalence to the best single contract holds at ϵ=0.10\epsilon=0.10 on both grids (TOST, [51]; Table 2 carries Δ\Delta, all intervals, and the full grid). The R=200R=200 freeze recorded de-en ϵ=0.05\epsilon=0.05 as not passing (r59, boundary-exact; r60). The tolerance component covers every flip the falsification found (Figure 4; de-en 11/11, zh-en 3/3); no single contract covers all 14 (9–10; McNemar n.s.).

Figure 4. Registered-flip coverage on the WMT19 grids (r60 canonical): the audited ensemble covers every registered flip on both grids; no single contract does.

AER audit (R=2000R=2000, r60)de-enzh-en
point τb\tau_b (CI95).700.752
best single (post hoc).683.771
Δ\Delta vs. best+.017−.019
TOST ϵ=.05\epsilon=.05 / .10pass / passpass / pass
flips covered by ties11/113/3

Table 2. Canonical AER validation on WMT19 (16/15 released systems). CIs: point τb\tau_b CI95 .700 [.617, .717] / .752 [.733, .771]; Δ\Delta CI90 +.017 [−.050, +.017] / −.019 [−.019, +.019] (de-en lower limit at the ϵ=0.05\epsilon=0.05 band edge). Ties re-declared at R=2000R=2000; the flip set is identical to R=200R=200 (pair-for-pair, both grids). Best singles: BLEU-13a (de-en), BERTScore (zh-en); shading marks the audit’s headline readings.

The comparison is conservative: the best single is selected post hoc and changes across grids,

still (up to 0.84 of pass@1). AER transfers to these grids as declared: retrieval (four contracts) and code show zero contract-induced flips, the QA EM↔F1 flip is ensemble-covered (r59/N4). so an evaluator who cannot know the right contract ex ante gets its level plus full flip coverage. The anchor is itself a coordinate: the raw-vs-zz anchor choice moves AER by −0.050 (de-en) and −0.038 (zh-en), with the margins to the best single contract preserved (r61/E14). Held out (newstest2021; 180 contract-induced flips): AER ensemble covers 73%, median-of-zz 73%, best single 69%; the union-of-ties rule (declare tied if any flipping contract does) reaches 89% but with 72–91% false-tie rate (pairs tied under one contract while CI-excluding under another), a coverage–precision trade-off the marginalization choice accepts (r64/E-A3). The ϵ\epsilon bands exceed the anchor’s resampling noise (jackknife SD .010/.005, splithalf floor .033/.026), scoping coverage to the declared grid and member set (r64).

Extension to LLM-as-Judge: Three Judges on a Contract Grid

The audit extends to LLM-as-judge, whose identity, rubric, format, and language are contract

coordinates: three open-weight judges (Qwen2.5-7B, Llama-3.1-8B, Mistral-7B; three families) score 2,000 WMT19 de-en translations from 16 frozen systems under rubric, format, and language contrasts, each cell frozen (r54; anchor: official raw human scores).

Three patterns replicate (Table 3; r54): Mistral-7B shows the largest verbosity sensitivity, and system-level correlation coexists with pair-level unreliability.

PatternReading
(i) Judge leveragejudge swap moves rankings: τb\tau_b .683–.717 vs. anchor; zz-gap .055 (rubric .067, format .047); pair-level κ\kappa .05–.06
(ii) Judge variancemonotone noise ladders (single-sample SD above five-vote residual); candidate-order flips .47–.63 of verdicts
(iii) AER suppressionunstable pairs 13–15 →\to 0 (singles 15/14/13); generic shrinkage: median-of-zz .717 = best single; ensemble .700, CI [.70, .70] (τb\tau_b discreteness); H-J4 fails, H-J3 met

Table 3. Judge-grid patterns (three judges, three families; 2,000 WMT19 de-en translations, 16 frozen systems; § 4).

Boundary Conditions and Negative Results

Fiveboundary findings and two failed preregistered hypotheses map where the claims hold and stop (Table 17, appendix); H-X1 registered a cross-task correlation between arm coefficient and rank volatility (expected ρ≥0.7\rho\ge0.7, observed −0.963, p=0.0021p = 0.0021). The findings span SST-2 (Socher et al., [53]), NER (Tjong Kim Sang and De Meulder, [58]), XNLI (Conneau et al., [13]), plus four DocRE-family benchmarks.

BSettingResultImport
B1SST-2, NERr=−0.963r=-0.963 pooled (6 pts., descr.); per-task −1.0/−0.26 (n=3n=3)Construct differs
B2HacRED, 4 settingsΔ=0\Delta=0 exactly, 4 replicationsLayers identical
B3X2, n≤10n\le10Sign undetermined at every nnArm bounds seeds
B4XNLI, strong modelsAcc. near 0.99; no flipsMargin clause
B5All LLM settingsSmoke vs. full both waysFull-run gate

Table 17. Boundary conditions and negative results (retained under the preregistration protocol; shifts carry doc-level bootstrap CIs, 10k resamples; interpretations abbreviated, full text and data in the supplementary repository).

One pattern stands out: B1 is a failure of naive transfer: the arm needs the disagreement sets of a gold revision, which single-gold benchmarks lack (NER’s dual gold yields a rescuable fraction far below DocRED’s revision density, r36); B3 is a seed-budget bound (sign undetermined at every n≤10n \le10, §4.3). Two mechanism checks: the X2 gold-swap direction is robust across scoring functions (magnitude varies four-fold, r40); removing entity markers collapses part of the HacRED gap (−0.006-0.006 to −0.015-0.015 vs. +0.05+0.05 to +0.09+0.09): an input-side convention carries part of the reading. B4 localizes the phenomenon: leverage needs models that err differently and score comparably, violated by strong models on easy benchmarks; with B2, B5, H1, H-X1, these scope Propositions 1–4. One inference rule: a flip under frozen predictions shows a single-contract reading cannot be presumed portable without an arm/margin audit; stability within the tested family does not certify a model-intrinsic property.

The Audit Protocol and the EvalLever Toolkit

Fault-injection and positive control. Five fault classes over 675 scenarios: 100% detection, zero false positives (Wilson 1.15%; 330 legal variants), 98.19% key localization (preregistered ≥99%\ge99\% failed). A seeded oracle: clearing 1% of intra pairs moves the reading −0.004-0.004 [−0.006-0.006, −0.002-0.002], monotone at every level; the sham injection reproduces frozen counts bitwise.

Official-scorer reproduction gates. Adapters enter only after exactly reproducing official evaluation (digit-matched; supplement).

Human verification with programmatic ground truth. A stratified 50-violation sample: nonblind author detection 50/50, classification 48/50 (Wilson does not); held-out Chinese-arm violations 20/20; blind annotator detection 50/50, categories 14/21 (κ=0.561\kappa= 0.561, CI [0.38, 0.77]), 29/29 program-truth anomalies, label-level divergence; taxonomy percentages are single-coder descriptions.

The audit auditing itself. One ledger reconciliation exposed a parser bug, corrected by re-auditing preserved raw predictions (supplement).

EvalLever. We release the stack as EvalLever (footnote 1): register, aer, audit, seed (contract set, marginal reading with ties, violation scan, minimum seeds).

A Preregistered Evaluation Paradigm

The workflow (<10<10 min/experiment, CPU-only): register the contract set before running (evallever register); report the marginal reading with ties declared, not ranked, admitting multiple winners (evallever aer); gate and prescribe: violation rate, unverifiable cells, required seed count (evallever audit/seed); amendments logged, negatives retained. Three evidence tiers (Table 6): T1 preregistered and audit-passed, T2 multi-seed unaudited, T3 single seed; the regress stops at preregistered artifacts and anchors, not another model judge.

TierDefinition
T1Preregistered and frozen contract, seed prescription met, fault-injected audit passed.
T2Fully reported, multiple seeds, not audited.
T3Single contract, single seed—reference only.

Table 6. Evidence tiers for reading-side claims (§ 7).

Against its own standard (Table 7): 147 logged amendments (hypothesis-bearing ones disclosed at point of use), each with a verification record; every negative result retained; four defective launches halted and re-derived.

Discussion and Conclusion

Conclusion. (i) Readings are joint model×\timescontract properties: a point-estimate reversal under a gold switch (frozen; tie-level after), localizing on re-partitioned gold. (ii) Leverage separates families several-fold (descriptive; not monotone). (iii) A seed prescription from the variance-level ordering (dedup-sensitive, imputation-conditional); (iv) violations orthogonal to leverage (observational). Freeze, state, and audit contracts: movement CI-excluding, reversal tie-level, stated not smoothed. The audited evaluation reading (AER), validated on MT and extended to LLM-as-judge (§ 4.1), transfers to paradigms where the evaluator is itself a language model.

Limitations

Scope limits: thin pool (Zhang et al. 2025 absent); five LLMs, one decode per cell; n∗n^* from one curve; one verifier; DocRE dev-side; released predictions recompute every arm. Ethics: public artifacts only (footnote 1); no personal data; verification: § 6.

Replication Details

Provenance and robustness notes. (i) Subset construction. The 498-document development subset is the title-matched intersection of DocRED dev (998) and Re-DocRED dev (500); two documents without a same-name match are dropped, none silently (release DATASET_CARD). (ii) What the gold swap adds. On this subset the revised standard adds 11,084 relations absent from the original (6,074 →\rightarrow 17,098) and drops 60; 70.8% of the added relations are cross-sentence versus 58.2% of retained ones (mean sentence distance 2.37 vs. 1.90)—the added mass sits where the gap lives, the entity-level counterpart of the gold-swap arms. (iii) Composition versus rates. A Kitagawa decomposition of the intra–inter gap (hit-rate estimand, fixed τ=0.5\tau=0.5; X2 and HacRED, 15 runs, residuals < 10−1510^{-15}) places 0.9% (X2-D) and 9.2% (HacRED) of the gap on the composition term, and −17.8% for the X2 revised-gold arm—the gap is a within-stratum rate difference, and the revised gold’s added mass is itself more cross-sentence (full table in the released bundle). (iv) Truncation. 11.3% of HacRED candidates exceed the 512-token encoding window (51/1500 documents); restricting to fully visible candidates moves the gap readings by < 0.0005. (v) Chinese label-lexicon mechanism. The rise sits wholly in the relation-name component (14.4%→22.6%) and shifts from action-form slips (23%→4%) to role-noun paraphrase (0.28 → 0.54 per document); the dominant out-of-inventory relation is the unannotated birth-date one, instruction language leaves rate and signature intact, and a controlled narrowing test rules out inventory width. Claim-level evidence tiers and registration timing are consolidated in Table 10.

ClaimEvidence tierRegistered
Rename invariance (§ 2.1)derived; UD slices preregisteredinitial
C1–C3 main comparisonsT1initial (Holm)
P1 family separationderived envelope (Lemma A, accuracy-premise: arm ≤B∣D∣\le B|D|, i.e., ∣D∣/N|D|/N) + T2 slack ordering (multi-seed, not audited)initial
P2 flip registryregistered checks (exploratory status, § 3.2) + Lemma B guaranteeinitial
P3 variance orderingderived under independent assumptions (Theorem C); (am. 6) T1-conditional magnitudes (registered criterion fails strictest dedup)initial; H3 amended
Seed n∗=8n^{*}=8prescription T1-conditional (single curve)initial
P4 violation decouplingobservational, assumption statedinitial
H-G1 groupingT1am. 60
H1 / H-X1registered, failed in point estimate / reversedinitial
Scale/family bandsT1 (bands cover most of am. 85 feasible range)
Post-review generatorsT3 per generator (single checkpoint)am. 61–66

Table 10. Claims × evidence tier × registration timing (tiers per Table 6; read row-wise—the tier qualifies the claim, not the converse). Rows map one-to-one to entries in the released amendment ledger, whose verification records accompany each row.

RequirementStatusWhere
Freeze contract before runsmet147 logged amendments (prereg doc)
Retain negative resultsmet5 boundary findings; H1/H-X1 failed; 14-of-15 gate flags
Halt defective launchesmet4 launches halted, re-derived from raw inputs
Human verificationpartial (point est. only; Wilson fails)50 stratified violation checks
Value integritymet26/26 paper values matched to frozen files
Registered, executed post-reviewloggedconstrained decoding (am. 19b; Table 5)
Promised, supersededloggedfour-task meta-analysis (am. 16-3): B1’s construct finding makes the merge invalid

Table 7: The paper audited against its own preregistration standard (§ 7).

Proofs of Formal Results

Rename invariance (gauge; § 2.1). A partition-symmetric reading has the form ρ=f((n(p,g))p)\rho= f\left((n(p,g))_p\right) where n(p,g)n(p,g) are the joint (prediction, gold) cell counts. By convention the rename is applied to both sides of every pair at once (the conjugate action): the cell counts undergo one common permutation of the label index, applied to the prediction-side and gold-side indices together, and ff is invariant under such simultaneous relabeling. For g′=r∘gg' = r \circ g, the map n(p,g)↦n(rp,r(g))n(p,g) \mapsto n(rp,r(g)) permutes the label index of each cell vector with one and the same permutation; ff returns the same value on permuted arguments, hence ρ(M,(s,g′,τ,…))=ρ(M,(s,g,τ,…))\rho(M,(s,g',\tau,\ldots)) = \rho(M,(s,g,\tau,\ldots)). Accuracy, micro-F1, and macro-F1 are all functions of the label-count vector through symmetric statistics (sums and per-label ratios), so they satisfy the premise. A gold-side-only relabeling (predictions still matched by unchanged string identity) is not an instance of the premise and would move accuracy; the synthetic check’s exact-zero Δ\Delta confirms the implementation applies the double-sided convention.

Lemma A (disagreement-set upper bound; § 2.2). Let the reading be an affine count functional ρ=Φ(∑pϕ(p;yp,gp))\rho= \Phi\left(\sum_p \phi(p;y_p,g_p)\right) with per-pair contribution differences bounded by BB and Φ\Phi 1-Lipschitz (accuracy: B=1/NB = 1/N). Under a gold switch g1→g2g_1 \to g_2, a pair pp changes its contribution only if g1(p)≠g2(p)g_1(p) \ne g_2(p), i.e., p∈D(g1,g2)p \in D(g_1,g_2). Hence

∣ρ(M,c1)−ρ(M,c2)∣≤∑p∈D∣Δϕp∣≤B∣D∣.\left|\rho(M,c_1)-\rho(M,c_2)\right| \le\sum_{p\in D} |\Delta\phi_p| \le B|D|.

with B=1/NB = 1/N for accuracy. Ratio aggregates such as micro- and macro-F1 admit no per-pair decomposition as instantiated here (their denominators move under re-partitioning; micro-F1 is pooled over the scored classes—in plain single-label multiclass it coincides with accuracy), so they escape the lemma’s premise; for them a bound holds only with count- or class-count-dependent constants (for micro-F1, (2/γ)∣D∣/N(2/\gamma)|D|/N whenever $2TP + FP + FN \ge \gamma N$ for a constant γ>0\gamma> 0; for macro-F1, O(∣D∣/L)O(|D|/L) when the disagreement set concentrates in rare labels), stated as a remark rather than a lemma. The bound is monotone in ∣D∣|D|; the realized shift need not be, because per-pair changes carry signs and can cancel (the two empirical inversions of § 3.2 are instances).

Lemma B (no-flip margin guarantee; § 2.3). Write δc=ρ(A,c)−ρ(B,c)\delta_c = \rho(A,c) - \rho(B,c) and arm⁡(M)=∣ρ(M,c1)−ρ(M,c2)∣\operatorname{arm}(M) = |\rho(M,c_1)-\rho(M,c_2)|. Then

∣δc2−δc1∣≤arm⁡(A)+arm⁡(B),|\delta_{c_2}-\delta_{c_1}| \le\operatorname{arm}(A)+\operatorname{arm}(B),

(apply the triangle inequality to ρ(A,c2)−ρ(A,c1)\rho(A,c_2)-\rho(A,c_1) and ρ(B,c2)−ρ(B,c1)\rho(B,c_2)-\rho(B,c_1)). If ∣δc1∣>arm⁡(A)+arm⁡(B)|\delta_{c_1}| > \operatorname{arm}(A)+\operatorname{arm}(B) then δc1\delta_{c_1} and δc2\delta_{c_2} have the same sign: no flip.

Theorem C (variance ordering under nested randomization; § 2.4). Let level VkV_k induce reading ρk=μ+∑i≤kεi\rho_k = \mu+ \sum_{i\le k}\varepsilon_i with independent, mean-zero increments εi\varepsilon_i (nesting: level k+1k+1 adds a new source on top of level kk’s). By the law of total variance (here plain additivity of independent increments),

Var⁡(ρk)=∑i≤kVar⁡(εi),\operatorname{Var}(\rho_k) = \sum_{i\le k}\operatorname{Var}(\varepsilon_i),

which is nondecreasing in kk, giving Var⁡(ρ1)≤Var⁡(ρ2)≤Var⁡(ρ3)≤Var⁡(ρ4)\operatorname{Var}(\rho_1) \le\operatorname{Var}(\rho_2) \le\operatorname{Var}(\rho_3) \le\operatorname{Var}(\rho_4). The empirical content the data must test is the two assumptions: path nesting and increment independence; the deduplication cascade (§ 3.3) is the failure of the latter.

Statistical Procedures

Table 11 gives resamples, pairing unit, seed, and multiplicity correction per result family. Document-level readings use a paired document bootstrap, recomputing both compared systems on each resampled multiset, so each percentile 95% interval carries the within-document dependence of the structured instances; this is the inputs-variable, systems-fixed scheme of Deutsch et al. [16], read conditionally on the deployed system pool—the reading the frozen-prediction claims require; generalization to newly trained systems is not claimed (segment readings pair segments analogously, Table 11, Unit). Comparisons are paired by document (segment where noted), with a Holm correction for the three main comparisons, family-survival reporting for the 57-check matrix (Table 14), and exact McNemar for categorical agreement. A comparison is tie-level when the two-sided 95% interval of the paired difference contains zero; AER validity reports TOST at ϵ∈{0.05,0.10}\epsilon\in\{0.05, 0.10\} (90% interval held inside the band)—the significant-and-meaningful pairing of Bouthillier et al. [4], applied to contract readings. Seeds are fixed per family (Table 11, Seed); the retraining-side prescription is n∗=8n^*=8 (Figure 6).

FamilyResamplesUnitSeed
Main arms/flips (C1–C3)10kdoc0926
57-check reg.10kdoc0926
AER valid. (r60)2000segment7
Value exps (r55)200/200seg/sys7/11
Claims audit (r56)10kdoc0926/+1
Error decomp (r57)—exact—
Judge grid (r54)1k/200seg/sys7/5
Null bounds (r58)20ktop/prob11/12

Table 11. Statistical procedures by result family (reviewer consensus C4): resample counts, pairing unit, RNG seed. Correction: Holm (3) for the main arms, Holm sensitivity for the 57-check registry, none elsewhere. All bootstrap intervals are percentile 95%.

Complementary Results

This section collects cross-task replication grids, post-review generator arms (Table 16), UD cross-version arms, and boundary conditions.

ArmGold [CI]Grouping [CI]Thr.Dens.
DocuNet-L.025 [.003, .049].021 [.009, .033].024.039
TTM-RE s74.038 [.012, .064].018 [.007, .029]–.070
TTM-RE s2.046 [.019, .073].022 [.010, .035]–.037
Qwen NF4.006 [−.005, .017]––.032
BF16 greedy−.001 [−.016, .013]––.039
T=0.7T=0.7 (×5\times5).007–.015–––

Table 16. Post-review generator and contract-axis arms, self-contained (the registered 57-check matrix is not renumbered); DocuNet-large single-checkpoint (hardware sensitivity, amendment 77; diagnostic in the bundle). DocuNet-large = published configuration retrained on a 24 GB cloud GPU (amendment 61). TTM-RE = two checkpoints (amendments 62/66): the official v0.2 release (inference only) and an independent from-base retrain; deviations disclosed in the bundle. BF16/NF4 = Qwen-7B same-grid greedy pair (quantization axis); T=0.7T=0.7 = five temperature-sampling rounds vs. the greedy anchor (decoding-stochasticity axis). Bootstrap: 10,000 doc resamples, seed 20260926. “–” = not applicable: the LLM grid yields name-level triples without pair-level structure, so grouping and threshold arms are undefined there.

References

  1. [1]Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020. Tacred revisited: A thorough evaluation of the TACRED relation extraction task. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1558–1569. doi: 10.18653/v1/2020.acl-main.142.DOI
  2. [2]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
  3. [3]Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Chris Federmann, Mark Fishel, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Philipp Koehn, Tomáš Kocmi, Pavel Macháček, Christoph Monz, Masaaki Morishita, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Marco Turchi, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (wmt19). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5046–5067. doi:10.18653/v1/P19-1426.
  4. [4]Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Naz Sepah, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Dmitriy Serdyuk, Tal Arbel, Chris Pal, Gaël Varoquaux, and Pascal Vincent. 2021. Accounting for variance in machine learning benchmarks. In Proceedings of Machine Learning and Systems.arxiv.org/abs/2103.03098
  5. [5]Robert L. Brennan. 2001. Generalizability Theory. Springer, New York.
  6. [6]Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020. With little power comes great responsibility. Transactions of the Association for Computational Linguistics, 8:743–762.arxiv.org/abs/2010.06595
  7. [7]Tanise Ceron, Neele Falk, Ana Barić, Dmitry Nikolaev, and Sebastian Padó. 2024. Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMs. Transactions of the Association for Computational Linguistics, 12:1378–1400. doi:10.1162/tacl_a_00710.DOI
  8. [8]Cheng Chang, Yining Mao, and Peng Qi. 2026. Padmé: Preference alignment data synthesis for meta-evaluation of LM agent evaluators. arXiv preprint arXiv:2609.36086.arxiv.org/abs/2609.36086
  9. [9]Haoyang Chen and Kumiko Tanaka-Ishii. 2026. Understanding benchmark language under weakened formal semantics. Transactions of the Association for Computational Linguistics, 14:2012–2032. doi:10.1162/tacl.a.789.DOI
  10. [10]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, et al. 2021a. Evaluating large language models trained on code. In arXiv preprint arXiv:2107.03374.arxiv.org/abs/2107.03374
  11. [11]Xiang Chen, Xin Xie, Shuming Deng, Ningyu Zhang, and Huajun Chen. 2021b. Document-level relation extraction as semantic segmentation. In Proceedings of the 30th International Joint Conference on Artificial Intelligence, pages 4645–4651. doi:10.24963/ijcai.2021/551.DOI
  12. [12]Qiao Cheng, Juntao Liu, Xiaoye Qu, Jin Zhao, Jiaqiang Liang, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, and Yanghua Xiao. 2021. HacRED: A large-scale relation extraction dataset toward hard cases in practical applications. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2819–2831. doi:10.18653/v1/2021.findings-acl.249.DOI
  13. [13]Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. doi:10.18653/v1/D18-1269.DOI
  14. [14]Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2020. Overview of the trec 2019 deep learning track. In Text REtrieval Conference (TREC 2019). NIST Special Publication 1266.arxiv.org/abs/2003.07820
  15. [15]Lee J. Cronbach, Goldine C. Gleser, Hursch Nanda, and Nageswari Rajaratnam. 1972. The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. John Wiley & Sons, New York.
  16. [16]Daniel Deutsch, Rotem Dror, and Dan Roth. 2021. A statistical analysis of summarization evaluation metrics using resampling methods. Transactions of the Association for Computational Linguistics, 9:1132–1145.arxiv.org/abs/2104.00054
  17. [17]Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2185–2194, Hong Kong, China. Association for Computational Linguistics.arxiv.org/abs/1909.03004
  18. [18]Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. doi:10.18653/v1/p18-1128.DOI
  19. [19]Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins, and Benjamin Van Durme. 2020. Multi-sentence argument linking. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. doi:10.18653/v1/2020.acl-main.678.
  20. [20]Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 61–71. Association for Computational Linguistics. doi:10.18653/v1/2020.emnlp-main.5.DOI
  21. [21]Chufan Gao, Xuan Wang, and Jimeng Sun. 2024. TTM-RE: Memory-augmented document-level relation extraction. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 443–458. doi:10.18653/v1/2024.acl-long.26.DOI
  22. [22]Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, and Pasquale Minervini. 2025. Are we done with MMLU? In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics, pages 5069–5096.
  23. [23]Sankalp Gilda and Shlok Gilda. 2026. Position: Evaluation scores are perishable knowledge claims. In Proceedings of the 4th Workshop on Generative AI and NLP (GEM). doi:10.18653/v1/2026.gem-main.80.DOI
  24. [24]Jena D. Hwang, Varsha Kishore, Amanpreet Singh, Dany Haddad, Aakanksha Naik, Malachi Hamada, Jonathan Bragg, Mike D’Arcy, Daniel S. Weld, Lucy Lu Wang, Doug Downey, and Sergey Feldman. 2026. Deep research, shallow evaluation: A case study in meta-evaluation for long-form QA benchmarks. arXiv:2603.06942.arxiv.org/abs/2603.06942
  25. [25]Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825.arxiv.org/abs/2310.06825
  26. [26]Pengcheng Jiang, Jiacheng Lin, Zifeng Wang, Jimeng Sun, and Jiawei Han. 2024. GenRES: Rethinking evaluation for generative relation extraction in the era of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2820–2837. doi:10.18653/v1/2024.naacl-long.155.DOI
  27. [27]Masamune Kobayashi, Masato Mita, and Mamoru Komachi. 2024. Revisiting meta-evaluation for grammatical error correction. Transactions of the Association for Computational Linguistics, 12:837–855. doi:10.1162/tacl_a_00676.DOI
  28. [28]Tom Kocmi, Vilém Zouhar, Chris Federmann, and Matt Post. 2023. Results of the wmt2023 metrics shared task: Metrics including MQM and SPA fitseg and metrics based on GEMBA. In Proceedings of the Eighth Conference on Machine Translation (WMT), pages 210–234. doi:10.18653/v1/2023.wmt-1.15.
  29. [29]Jiao Li, Yueping Sun, Robin J. Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Thomas C. Wiegers, and Zhiyong Lu. 2016. BioCreative V CDR task corpus: a resource for chemical disease relation extraction. Database, 2016:baw068. doi:10.1093/database/baw068.
  30. [30]Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023a. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems 36, pages 21558–21572.arxiv.org/abs/2305.01210
  31. [31]Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023b. G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522. doi:10.18653/v1/2023.emnlp-main.153.
  32. [32]Youmi Ma, An Wang, and Naoaki Okazaki. 2023. DREAM: Guiding attention with evidence for improving document-level relation extraction. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. doi:10.18653/v1/2023.eacl-main.145.DOI
  33. [33]Benjamin Marie, Atsushi Fujita, and Raphael Rubino. 2021. Scientific credibility of machine translation research: A meta-evaluation of 769 papers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7297–7306, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.arxiv.org/abs/2106.15195
  34. [34]William Meredith. 1993. Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58(4):525–543.DOI
  35. [35]Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? a call for multi-prompt LLM evaluation. Transactions of the Association for Computational Linguistics, 12:933–949.
  36. [36]Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. 2020. A metric learning reality check. In European Conference on Computer Vision (ECCV).arxiv.org/abs/2003.08505
  37. [38]Omer Nahum, Nitay Calderon, Orgad Keller, Idan Spektor, and Roi Reichart. 2024. Are LLMs better than reported? Detecting label errors and mitigating their effect on model performance. arXiv:2410.18889.arxiv.org/abs/2410.18889
  38. [39]Yannic Neuhaus and Matthias Hein. 2025. Re-POPE: Impact of annotation errors on the POPE benchmark. arXiv:2504.15707.arxiv.org/abs/2504.15707
  39. [40]Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. 2021. Pervasive label errors in test sets destabilize machine learning benchmarks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks.arxiv.org/abs/2103.14749
  40. [41]Juri Opitz. 2024. A closer look at classification evaluation metrics and a critical reflection of common evaluation practice. Transactions of the Association for Computational Linguistics, 12:820–836. doi:10.1162/tacl_a_00675.DOI
  41. [42]Egor Pakhomov and Erik Nijkamp. 2026. Book-keeping, composition, or unreachable gold? Reading MemoryAgentBench’s conflict-resolution scores against a frozen last-write resolver. arXiv:2610.09193.arxiv.org/abs/2610.09193
  42. [43]Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. doi:10.3115/1073083.1073135.DOI
  43. [44]Peter Polák, Sara Papi, Luisa Bentivogli, and Ondřej Bojar. 2025. Better late than never: Meta-evaluation of latency metrics for simultaneous speech-to-text translation. arXiv preprint arXiv:2509.17349.arxiv.org/abs/2509.17349
  44. [45]José Pombal, Nuno M. Guerreiro, Ricardo Rei, and André F. T. Martins. 2025. Adding chocolate to MINT: Mitigating metric interference in machine translation. Transactions of the Association for Computational Linguistics, 13:1319–1339. doi:10.1162/tacl.a.37.DOI
  45. [46]Maja Popović. 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395. doi:10.3115/v1/W15-3049.DOI
  46. [47]Matt Post. 2018. A call for clarity in reporting bleu scores. In Proceedings of the Third Conference on Machine Translation (WMT), pages 186–191. doi:10.18653/v1/W18-6319.DOI
  47. [48]Qwen Team. 2025. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115.
  48. [49]Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 784–789. doi:10.18653/v1/P18-2124.DOI
  49. [50]Federico Ruggeri, Eleonora Misino, Arianna Muti, Katerina Korre, Paolo Torroni, and Alberto Barrón-Cedeño. 2026. Let guidelines guide you: A prescriptive guideline-centered data annotation methodology. Transactions of the Association for Computational Linguistics, 14:2337–2356. doi:10.1162/tacl.a.804.DOI
  50. [51]Donald J. Schuir­mann. 1987. A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. Journal of Pharmacokinetics and Biopharmaceutics, 15:657–680.pubmed.ncbi.nlm.nih.gov/3450848
  51. [52]Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models’ sensitivity to spurious features in prompt design. In The Twelfth International Conference on Learning Representations.
  52. [53]Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642. doi:10.3115/v1/d13-1170.DOI
  53. [54]Elias Stengel-Eskin and Benjamin Van Durme. 2023. Calibrated interpretation: Confidence estimation in semantic parsing. Transactions of the Association for Computational Linguistics, 11:1213–1231. doi:10.1162/tacl_a_00598.
  54. [55]George Stoica, Emmanouil Antonios Platanios, and Barnabás Póczos. 2021. Re-TACRED: Addressing shortcomings of the TACRED dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13843–13850.arxiv.org/abs/2104.08398
  55. [56]Qingyu Tan, Tiwei He, Lidong Bing, and Zhouhan Lin. 2022a. Document-level relation extraction with adaptive focal loss and knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2022. doi:10.18653/v1/2022.findings-acl.132.DOI
  56. [57]Qingyu Tan, Lu Xu, Lidong Bing, Hwee Tou Ng, and Sharifah Mahani Aljunied. 2022b. Revisiting DocRED: Addressing the false negative problem in relation extraction. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. doi:10.18653/v1/2022.emnlp-main.580.DOI
  57. [58]Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning (CoNLL-2003). doi:10.3115/1119176.1119195.DOI
  58. [59]Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. Replacing judges with juries: Evaluating llm generations with a panel of diverse models. arXiv preprint arXiv:2404.18796.arxiv.org/abs/2404.18796
  59. [60]Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024a. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. doi:10.18653/v1/2024.acl-long.754.

Paper details

vlnc:W_xswp95ar
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 11

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents