Introduction
Evaluating the performance of classification models is a core step in any empirical machine learning study (Japkowicz and Shah, 2011). Many classifiers output scores in [ 0 , 1 ] [0,1] [ 0 , 1 ] , often interpreted as estimated class probabilities; we collect such scores in a matrix P ^ ∈ [ 0 , 1 ] n × m \hat{P} \in[0,1]^{n \times m} P ^ ∈ [ 0 , 1 ] n × m before a decision rule is applied. This output carries distinct kinds of information. First, binary decisions obtained using a specified decision rule determine threshold-dependent discrimination measures such as F 1 F_1 F 1 , precision, and recall; the relative ordering of scores alone does not determine these measures. Second, the magnitudes of probability estimates matter for probabilistic predictive quality: calibration concerns whether a forecast such as P ^ i , j = 0.8 \hat{P}_{i,j}=0.8 P ^ i , j = 0.8 corresponds to an event frequency near 80%, while proper scoring rules such as Brier and log-loss assess overall probabilistic quality, including calibration and resolution rather than calibration alone. A third, distinct question concerns ranking : row-axis nDCG evaluates the positions of relevant labels among the labels of each example, while column-axis AUC evaluates the ordering of positive versus negative examples for a fixed class. All three evaluation paradigms are active in most empirical study of multilabel or probability-producing classifiers, yet they have developed largely in isolation, with separate literatures, separate intuitions, and no common algebraic language.
A unified algebraic framework for discrimination was recently proposed by Prati (2027). It represents actual and predicted labels as binary indicator matrices Y , Y ^ ∈ { 0 , 1 } n × m Y,\hat{Y}\in\{0,1\}^{n\times m} Y , Y ^ ∈ { 0 , 1 } n × m and defines cell-level true/false positive/negative counts via element-wise Boolean conjunction and negation; soft labels and predictions are accommodated by replacing the Boolean connectives with triangular norms (t -norms), which reduce to the Boolean case on binary matrices. Three aggregation operators, global (Σ 1 \Sigma_1 Σ 1 ), column-wise (Σ m \Sigma_m Σ m ), and row-wise (Σ n \Sigma_n Σ n ), corresponding to micro, macro, and exemplar averaging, then generate entire families of measures for multiclass, multilabel and related classification problems from any single binary formula. The framework characterises skew-invariant measures (functions of recall and specificity only), proves that micro-averaging equals denominator-weighted macro-averaging, and identifies the product t -norm as the unique marginally consistent one (which implies the confusion-matrix partition property). However, calibration measures (Brier score, log-loss, Murphy decomposition) and nDCG fall outside the framework of Prati (2027): they operate on the raw probability matrix P ^ \hat{P} P ^ rather than a thresholded indicator matrix, and their cell values are not expressible as Boolean or t -norm outputs. AUC is reachable there only as an integral over thresholds of the coincidence cells.
This paper extends the indicator-matrix framework by replacing its coincidence cell operator (the Boolean conjunction and its t -norm generalisation) with two further operator classes, yielding a three-class taxonomy:
Coincidence operators (Boolean conjunction, generalised by t -norms): the cell operator of Prati (2027), quantifying Boolean coincidence of actual and predicted binary labels, or their soft overlap according to a chosen t -norm; for soft entries, positivity of both inputs alone need not imply a positive t -norm value. These recover the Discrimination Framework of Prati (2027) with no modification.
Deviation operators (ϕ ( y − y ^ ) \phi(y-\hat{y}) ϕ ( y − y ^ ) for convex ϕ \phi ϕ ): measuring the magnitude of the gap between actual and predicted soft labels. This yields the Calibration Framework , covering calibration.
Positional and rank-dependent operators : row-positional gains Y i j w ( rank i ( j ) ) Y_{ij}w(\operatorname{rank}_i(j)) Y ij w ( rank i ( j )) , with non-increasing w w w , give the nDCG family; column-dependent cells compare positive and negative examples of a class to give AUC. A fixed column-rank weight yields an affine identity for aggregated AUC under distinct scores, but does not generally
reproduce an individual AUC concordance cell. The row and column ranking axes produce complementary subfamilies of the Ranking Framework .
Within this taxonomy we establish the following results:
For the Calibration Framework, the squared-error deviation operator D ( y , y ^ ) = ( y − y ^ ) 2 D(y,\hat{y}) = (y-\hat{y})^2 D ( y , y ^ ) = ( y − y ^ ) 2 induces the Brier score family. Applying the three aggregation operators yields micro-, macro-, and exemplar-averaged Brier scores; we prove that all three collapse to the same scalar, because the Brier score is a sum of per-cell terms with a fixed denominator (the same holds for any per-cell deviation, but not for the ratio measures of the Discrimination Framework or for normalised nDCG). We derive a matrix-language version of the Murphy (Murphy (1973)) reliability–resolution–uncertainty decomposition, B S j = U N C j − R E S j + R E L j \mathrm{BS}_j = \mathrm{UNC}_j - \mathrm{RES}_j + \mathrm{REL}_j BS j = UNC j − RES j + REL j , via the tower property of conditional expectations. The same identity is also stated for a calibration map estimated by isotonic regression, which needs no bins (Proposition 28 ).
We characterise proper scoring rules within the Calibration Framework via the Savage representation (Savage (1971); Gneiting and Raftery (2007)), showing that a symmetric deviation operator ϕ ( y − y ^ ) \phi(y-\hat{y}) ϕ ( y − y ^ ) is proper exactly when its weight ϕ ′ ( t ) / t \phi'(t)/t ϕ ′ ( t ) / t is symmetric about 1 2 \frac{1}{2} 2 1 (convexity of ϕ \phi ϕ alone is not sufficient); the proper ones include squared error and nonquadratic losses, while log-loss satisfies the same weight condition in an extended-valued sense. For skew-sensitivity, we prove that a Brier-type class-indexed measure of the form M = g ( { π j , α j , β j } j = 1 m ) M = g(\{\pi_j,\alpha_j,\beta_j\}_{j=1}^{m}) M = g ({ π j , α j , β j } j = 1 m ) is skew-invariant if and only if it depends on { α j , β j } j = 1 m \{\alpha_j,\beta_j\}_{j=1}^{m} { α j , β j } j = 1 m alone, where α j = E [ ( 1 − Y ^ i j ) 2 ∣ Y i j = 1 ] \alpha_j = \mathbb{E}[(1-\hat{Y}_{ij})^2 \mid Y_{ij}=1] α j = E [( 1 − Y ^ ij ) 2 ∣ Y ij = 1 ] and β j = E [ Y ^ i j 2 ∣ Y i j = 0 ] \beta_j = \mathbb{E}[\hat{Y}_{ij}^{2} \mid Y_{ij}=0] β j = E [ Y ^ ij 2 ∣ Y ij = 0 ] are the class-conditional squared errors, analogous to the complements of recall and specificity in the hard-prediction limit (and, in the binary case, the two components of the stratified Brier score (Wallace and Dahabreh (2012, 2014)). General calibration measures are skew-invariant exactly when they depend only on the within-class score distributions (Theorem 47 (a)).
For the Ranking Framework, the row-dependent positional operator D w ( Y i , j , Y ^ i , ⋅ ) = Y i , j ⋅ w ( rank i ( j ) ) D_w(Y_{i,j},\hat{Y}_{i,\cdot}) = Y_{i,j}\cdot w(\operatorname{rank}_i(j)) D w ( Y i , j , Y ^ i , ⋅ ) = Y i , j ⋅ w ( rank i ( j )) induces the nDCG family. We show that raw DCG collapses across aggregation schemes (micro = macro = exemplar) while normalised nDCG does not, due to heterogeneous IDCG denominators. We propose a per-class nDCG summary within this matrix formalism, with the conditional-mean interpretation: n D C G j c l a s s = 1 w ( 1 ) E [ w ( rank i ( j ) ) ∣ Y i j = 1 ] \mathrm{nDCG}^{\mathrm{class}}_j = \frac{1}{w(1)}\mathbb{E}[w(\operatorname{rank}_i(j)) \mid Y_{ij}=1] nDCG j class = w ( 1 ) 1 E [ w ( rank i ( j )) ∣ Y ij = 1 ] . The column-dependent AUC family instead uses the concordance cell A i j = Y i j f ( i , j ) A_{ij} = Y_{ij}f(i,j) A ij = Y ij f ( i , j ) , where f ( i , j ) f(i,j) f ( i , j ) is the fraction of negative examples for class j j j scored below example i i i (with half credit for ties). Unlike a fixed positional cell Y i j w ( rank i ( j ) ) Y_{ij}w(\operatorname{rank}_i(j)) Y ij w ( rank i ( j )) , its value also depends on the negative-label set. At the per-class aggregate level, AUC is the Wilcoxon–Mann–Whitney statistic (Hanley and McNeil (1982)) and, for distinct column scores, an affine function of the mean positive-example rank (Proposition 73 ). This construction yields the per-example A U C i e x \mathrm{AUC}^{\mathrm{ex}}_i AUC i ex summary and makes the per-class skew-invariance transparent. A row–column duality theorem unifies the two subfamilies: both n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class and A U C j \mathrm{AUC}_j AUC j are conditional means over positive examples of rank-based quality measures, with a fixed row-rank discount for nDCG and a label-dependent column-concordance fraction for AUC (whose aggregate is an affine function of mean column rank).
Finally, as a cross-framework synthesis, we show that the three skew-invariant per-class bases, namely recall/specificity (Discrimination Framework), ( α j , β j ) (\alpha_j,\beta_j) ( α j , β j ) (Calibration Framework), and n D C G j c l a s s / A U C j \mathrm{nDCG}^{\mathrm{class}}_j/\mathrm{AUC}_j nDCG j class / AUC j (Ranking Framework), all share the same algebraic structure: they are class-conditional means (over the positives for recall, α j \alpha_j α j , per-class nDCG and AUC; over the negatives for specificity and β j \beta_j β j ). This single template organises the skew-invariance proofs for all three frameworks, under reweightings that preserve the conditional distribution of the averaged cell quality (for per-class nDCG this includes the within-row ranks of the positives, not only each label’s score distribution), and reveals that pooled (micro) aggregates are skew-sensitive in general (for the ratio measures of the Discrimination Framework and for micro-AUC because they weight classes by prevalence-dependent denominators; for the Brier score because each per-class term is itself prevalence-dependent). We further argue that the row/column dependency split is specific to the Ranking Framework: the natural ways of introducing rank dependence into the Discrimination or Calibration Frameworks either trivialise or recover the positional operator.
The measures of the Calibration and Ranking Frameworks, together with diagnostics that report the covariances behind the gaps between the micro, class and exemplar summaries, are implemented in the experimental subpackage aggmetrics.scores of the aggmetrics package (https://github.com/rcprati/aggmetrics), available from version 0.3.0.
This paper is organised as follows. Section 2 reviews the framework of Prati (2027). Section 3 reviews related work. Section 4 introduces the three-class cell operator taxonomy and the framework bridge theorem connecting the hard-prediction limits of the Discrimination and Calibration Frameworks; the hard-prediction limit of the Ranking Framework is treated in Section 6. Section 5 develops the Calibration Framework in full. Section 6 develops the Ranking Framework, including both the nDCG and AUC subfamilies and the duality theorem. Section 7 synthesises the three frameworks. Section 8 presents empirical illustrations. Section 9 concludes.
Background: The Indicator-Matrix Framework
This section provides a self-contained review of the base framework of Prati (2027), which covers discrimination-based evaluation. Readers familiar with that paper may skip ahead to Section 4.
Indicator Matrices
Consider a dataset of n n n examples, each associated with a subset of m m m labels. The actual and predicted label assignments are represented as binary matrices.
Definition 1 (Indicator matrices). The actual label matrix Y ∈ { 0 , 1 } n × m \mathbf{Y} \in\{0,1\}^{n \times m} Y ∈ { 0 , 1 } n × m has Y i , j = 1 Y_{i,j}=1 Y i , j = 1 iff example e i e_i e i belongs to class or label l j l_j l j . The predicted label matrix Y ^ ∈ { 0 , 1 } n × m \hat{\mathbf{Y}} \in\{0,1\}^{n \times m} Y ^ ∈ { 0 , 1 } n × m has Y ^ i , j = 1 \hat{Y}_{i,j}=1 Y ^ i , j = 1 iff the classifier assigns label l j l_j l j to e i e_i e i .
In multiclass problems, each row of Y \mathbf{Y} Y and Y ^ \hat{\mathbf{Y}} Y ^ is a one-hot vector: exactly one entry equals 1. In multilabel problems, any number of entries per row may equal 1. The binary case is the special case m = 2 m=2 m = 2 .
A binary task can be encoded by one indicator column for a designated positive class or by two complementary one-hot columns; the encodings carry the same label information, but scoring the positive class alone differs from pooling or macro-averaging both columns (Appendix § 11 ).
A soft classifier outputs a probability matrix P ^ ∈ [ 0 , 1 ] n × m \hat{P} \in[0,1]^{n \times m} P ^ ∈ [ 0 , 1 ] n × m ; a decision rule converts it into the binary Y ^ \hat{Y} Y ^ of Definition 1: argmax in the multiclass case and thresholding at an arbitrary θ \theta θ in the multilabel case. When misclassification costs are available, these rules can be adjusted accordingly, following the cost-sensitive principles of Elkan (2001). When the t t t -norm extension is applied to soft values, the entries of Y ^ \hat{Y} Y ^ in Definition 4 may represent scores in [ 0 , 1 ] [0,1] [ 0 , 1 ] rather than hard predictions. The Calibration and Ranking Frameworks instead work explicitly with P ^ \hat{P} P ^ ; a hard prediction matrix is obtained only when a decision rule is required.
Aggregation Operators
Definition 2 (Aggregation operators). Let A ∈ R n × m A \in\mathbb{R}^{n \times m} A ∈ R n × m be any real-valued matrix. Three aggregation operators are defined:
Σ 1 ( A ) = ∑ i = 1 n ∑ j = 1 m A i , j ∈ R ( global / micro ) , \Sigma_1(A) = \sum_{i=1}^{n} \sum_{j=1}^{m} A_{i,j} \in\mathbb{R} \qquad(\text{global / micro}), Σ 1 ( A ) = i = 1 ∑ n j = 1 ∑ m A i , j ∈ R ( global / micro ) ,
[ Σ m ( A ) ] j = ∑ i = 1 n A i , j ∈ R ( column-wise / per-class ) , [\Sigma_m(A)]_j = \sum_{i=1}^{n} A_{i,j} \in\mathbb{R} \qquad(\text{column-wise / per-class}), [ Σ m ( A ) ] j = i = 1 ∑ n A i , j ∈ R ( column-wise / per-class ) ,
[ Σ n ( A ) ] i = ∑ j = 1 m A i , j ∈ R ( row-wise / per-example ) . [\Sigma_n(A)]_i = \sum_{j=1}^{m} A_{i,j} \in\mathbb{R} \qquad(\text{row-wise / per-example}). [ Σ n ( A ) ] i = j = 1 ∑ m A i , j ∈ R ( row-wise / per-example ) .
In particular, Σ m ( A ) ∈ R m \Sigma_m(A) \in\mathbb{R}^{m} Σ m ( A ) ∈ R m and Σ n ( A ) ∈ R n \Sigma_n(A) \in\mathbb{R}^{n} Σ n ( A ) ∈ R n ; their indexed components are scalars.
These operators correspond to the three most common averaging schemes in machine learning libraries (e.g., average= ’micro’, ’macro’, and ’samples’ in scikit-learn): Σ 1 \Sigma_1 Σ 1 pools all counts globally; Σ m \Sigma_m Σ m produces a per-class vector from which an unweighted (macro) or prevalence-weighted average is computed; Σ n \Sigma_n Σ n produces a per-example vector whose unweighted mean is the exemplar average.
Coincidence Cell Operators and Confusion Matrices
In Prati (2027), the four confusion matrices are defined by element-wise Boolean operations (conjunction and negation) on the binary indicator matrices. Soft labels or soft predictions are handled by an extension in which the Boolean connectives are replaced by their fuzzy counterparts, triangular norms, with the Boolean definitions recovered exactly on binary matrices. Since this paper allows soft entries throughout, we state the general t t t -norm form directly.
Definition 3 (t t t -norm). A triangular norm (Klement et al. (2000)) is a function T : [ 0 , 1 ] 2 → [ 0 , 1 ] T : [0,1]^2 \to[0,1] T : [ 0 , 1 ] 2 → [ 0 , 1 ] satisfying commutativity, associativity, monotonicity, and the boundary condition T ( a , 1 ) = a T(a,1) = a T ( a , 1 ) = a for all a ∈ [ 0 , 1 ] a \in[0,1] a ∈ [ 0 , 1 ] . The three standard t t t -norms are the Gödel (minimum) T M ( a , b ) = min ( a , b ) T_M(a,b) = \min(a,b) T M ( a , b ) = min ( a , b ) , the product T P ( a , b ) = a ⋅ b T_P(a,b) = a \cdot b T P ( a , b ) = a ⋅ b , and the Łukasiewicz T L ( a , b ) = max ( 0 , a + b − 1 ) T_L(a,b) = \max(0,a+b-1) T L ( a , b ) = max ( 0 , a + b − 1 ) .
Definition 4 (Confusion cell matrices). Given Y , Y ^ ∈ [ 0 , 1 ] n × m \mathbf{Y},\hat{\mathbf{Y}} \in[0,1]^{n \times m} Y , Y ^ ∈ [ 0 , 1 ] n × m , a t -norm T T T , and the standard complement N ( a ) = 1 − a N(a)=1-a N ( a ) = 1 − a , the four confusion cell matrices are:
T P i , j = T ( Y i , j , Y ^ i , j ) , F P i , j = T ( N ( Y i , j ) , Y ^ i , j ) , F N i , j = T ( Y i , j , N ( Y ^ i , j ) ) , T N i , j = T ( N ( Y i , j ) , N ( Y ^ i , j ) ) . \begin{aligned}
TP_{i,j} &= T(Y_{i,j},\hat{Y}_{i,j}), & FP_{i,j} &= T(N(Y_{i,j}),\hat{Y}_{i,j}),\\
FN_{i,j} &= T(Y_{i,j},N(\hat{Y}_{i,j})), & TN_{i,j} &= T(N(Y_{i,j}),N(\hat{Y}_{i,j})).
\end{aligned} T P i , j F N i , j = T ( Y i , j , Y ^ i , j ) , = T ( Y i , j , N ( Y ^ i , j )) , F P i , j T N i , j = T ( N ( Y i , j ) , Y ^ i , j ) , = T ( N ( Y i , j ) , N ( Y ^ i , j )) .
For binary Y , Y ^ ∈ { 0 , 1 } n × m \mathbf{Y},\hat{\mathbf{Y}} \in\{0,1\}^{n \times m} Y , Y ^ ∈ { 0 , 1 } n × m , all three t -norms reduce to the Boolean AND, so T P , F P , F N , T N ∈ { 0 , 1 } n × m TP,FP,FN,TN \in\{0,1\}^{n \times m} T P , F P , F N , T N ∈ { 0 , 1 } n × m are binary indicator matrices whose entries equal 1 exactly where the corresponding outcome holds for example e i e_i e i and label l j l_j l j .
Applying the aggregation operators to these matrices yields the familiar scalar and per-class confusion counts: t p = Σ 1 ( T P ) tp=\Sigma_1(TP) tp = Σ 1 ( T P ) , t p j = [ Σ m ( T P ) ] j tp_j=[\Sigma_m(TP)]_j t p j = [ Σ m ( T P ) ] j , and so on, from which any binary measure (precision, recall, F 1 F1 F 1 , MCC (Matthews, 1975), ...) can be computed. The framework is thus a measure generator : given one binary formula for M ( t p , f p , f n , t n ) M(tp,fp,fn,tn) M ( tp , f p , f n , t n ) and the three aggregation operators, a family of multiclass and multilabel measures follows automatically.
Example 1 (Running example). Throughout the paper we use n = 4 n=4 n = 4 examples and m = 3 m=3 m = 3 labels A , B , C A,B,C A , B , C , with
Y = ( 1 1 0 1 0 1 0 1 0 0 1 0 ) , P ^ = ( .7 .6 .1 .6 .1 .4 .8 .2 .7 .4 .3 .2 ) . \mathbf{Y}=
\begin{pmatrix}
1&1&0\\
1&0&1\\
0&1&0\\
0&1&0
\end{pmatrix},
\qquad
\hat{\mathbf{P}}=
\begin{pmatrix}
.7&.6&.1\\
.6&.1&.4\\
.8&.2&.7\\
.4&.3&.2
\end{pmatrix}. Y = 1 1 0 0 1 0 1 1 0 1 0 0 , P ^ = .7 .6 .8 .4 .6 .1 .2 .3 .1 .4 .7 .2 . Scores are pairwise distinct within every row and every column. The classifier scores A A A highly for every example, including the two for which A A A is irrelevant; it separates B B B from its single negative example, but ranks B B B low within examples. Thresholding at θ = 0.5 \theta=0.5 θ = 0.5 gives Y ^ i j = 1 [ P ^ i j > 0.5 ] \hat{Y}_{ij}=1[\hat{P}_{ij}>0.5] Y ^ ij = 1 [ P ^ ij > 0.5 ] and the per-class counts ( t p j , f p j , f n j , t n j ) = ( 2 , 1 , 0 , 1 ) (tp_j,fp_j,fn_j,tn_j)=(2,1,0,1) ( t p j , f p j , f n j , t n j ) = ( 2 , 1 , 0 , 1 ) , ( 1 , 0 , 2 , 1 ) (1,0,2,1) ( 1 , 0 , 2 , 1 ) and ( 0 , 1 , 1 , 2 ) (0,1,1,2) ( 0 , 1 , 1 , 2 ) for A A A , B B B and C C C , so that ( t p , f p , f n , t n ) = ( 3 , 2 , 3 , 4 ) (tp,fp,fn,tn)=(3,2,3,4) ( tp , f p , f n , t n ) = ( 3 , 2 , 3 , 4 ) . Micro-precision is 3 / 5 3/5 3/5 and macro-precision is ( 2 / 3 + 1 + 0 ) / 3 = 5 / 9 (2/3+1+0)/3=5/9 ( 2/3 + 1 + 0 ) /3 = 5/9 : the per-class denominators t p j + f p j = ( 3 , 1 , 1 ) tp_j+fp_j=(3,1,1) t p j + f p j = ( 3 , 1 , 1 ) covary with the per-class precisions (Theorem 5). Appendix A lists every intermediate value of the running example, and the script experiments/running_example.py reproduces them.
Theoretical Results of the Base Framework
The following results from Prati (2027) are recalled for use in this paper. For rational measures, all denominators appearing below are assumed nonzero; positive denominators are assumed whenever they are used as averaging weights.
Theorem 5 (Micro = denominator-weighted macro for aggregation-decomposable measures). Let M M M be an aggregation-decomposable binary measure, as in Definition 7 of Prati (2027): for the count vector v = ( t p , f p , f n , t n ) T \mathbf{v}=(tp,fp,fn,tn)^{\mathsf{T}} v = ( tp , f p , f n , t n ) T , suppose
M ( v ) = ⟨ a , v ⟩ ⟨ b , v ⟩ , a , b ∈ R 4 . M(\mathbf{v})=\frac{\langle\mathbf{a},\mathbf{v}\rangle}{\langle\mathbf{b},\mathbf{v}\rangle},\qquad\mathbf{a},\mathbf{b}\in\mathbb{R}^{4}. M ( v ) = ⟨ b , v ⟩ ⟨ a , v ⟩ , a , b ∈ R 4 .
Let v j \mathbf{v}_j v j be the per-class count vectors obtained using Σ m \Sigma_m Σ m , and assume den j = ⟨ b , v j ⟩ > 0 \operatorname{den}_j=\langle\mathbf{b},\mathbf{v}_j\rangle>0 den j = ⟨ b , v j ⟩ > 0 for every included class. Then
M m i c r o = ∑ j = 1 m den j M j ∑ j = 1 m den j , M j = M ( v j ) . M^{\mathrm{micro}}=\frac{\sum_{j=1}^{m}\operatorname{den}_j M_j}{\sum_{j=1}^{m}\operatorname{den}_j},\qquad M_j=M(\mathbf{v}_j). M micro = ∑ j = 1 m den j ∑ j = 1 m den j M j , M j = M ( v j ) .
For fixed local denominators, this weighted mean agrees with the unweighted macro-mean for every independently variable vector ( M 1 , … , M m ) (M_1,\ldots,M_m) ( M 1 , … , M m ) if and only if all denominators are equal. For a given observed vector of scores, the exact equality condition is Cov j ( den j , M j ) = 0 \operatorname{Cov}_j(\operatorname{den}_j,M_j)=0 Cov j ( den j , M j ) = 0 , with covariance taken uniformly over the m m m classes. In general, the identity does not hold for measures whose numerators or denominators are not additive in the count vectors.
Appendix B gives a non-additive ratio (squared recall) for which the identity fails.
Definition 6 (Marginal consistency). A t t t -norm T T T , paired with the standard complement N ( x ) = 1 − x N(x)=1-x N ( x ) = 1 − x , is marginally consistent if, for all a , b ∈ [ 0 , 1 ] a,b\in[0,1] a , b ∈ [ 0 , 1 ] ,
T ( a , b ) + T ( a , 1 − b ) = a , T ( a , b ) + T ( 1 − a , b ) = b . T(a,b)+T(a,1-b)=a,\qquad T(a,b)+T(1-a,b)=b. T ( a , b ) + T ( a , 1 − b ) = a , T ( a , b ) + T ( 1 − a , b ) = b . Equivalently, both marginal identities T P i , j + F N i , j = Y i , j \mathbf{T}\mathbf{P}_{i,j}+\mathbf{F}\mathbf{N}_{i,j}=\mathbf{Y}_{i,j} T P i , j + F N i , j = Y i , j and T P i , j + F P i , j = Y ^ i , j \mathbf{T}\mathbf{P}_{i,j}+\mathbf{F}\mathbf{P}_{i,j}=\widehat{\mathbf{Y}}_{i,j} T P i , j + F P i , j = Y i , j must hold for every possible pair of soft cell values, and not only for the cells of one fixed data set.
Theorem 7 (Marginal consistency characterises the product t t t -norm). A t t t -norm T T T is marginally consistent in the sense of Definition 6 if and only if T = T P T=T_P T = T P .
Corollary 8 (Partition property). Under the product t t t -norm, the four confusion cell matrices partition the unit measure entrywise,
T P i , j + F P i , j + F N i , j + T N i , j = 1 \mathbf{T}\mathbf{P}_{i,j}+\mathbf{F}\mathbf{P}_{i,j}+\mathbf{F}\mathbf{N}_{i,j}+\mathbf{T}\mathbf{N}_{i,j}=1 T P i , j + F P i , j + F N i , j + T N i , j = 1 for all ( i , j ) (i,j) ( i , j ) . Consequently, the aggregated four counts sum to n m nm nm for an n × m n\times m n × m matrix, and to n n n in a single-column binary encoding. This is a consequence of marginal consistency (sum the two equations of Definition 6 for Y i , j \mathbf{Y}_{i,j} Y i , j and N ( Y i , j ) = 1 − Y i , j N(\mathbf{Y}_{i,j})=1-\mathbf{Y}_{i,j} N ( Y i , j ) = 1 − Y i , j ), not an independent characterisation of T P T_P T P : the weaker single-equation partition property alone does not pin down T P T_P T P among all t t t -norms without further argument, so it should not be read as restating Theorem 7's uniqueness claim in a different form.
Definition 6 and Theorem 7 are Definition 8 and Theorem 3 of Prati (2027); Corollary 8 is that paper's Corollary 3. Together they provide a justification for using T P T_P T P when evaluating probabilistic classifiers.
Theorem 9 (Skew-invariance characterisation). Define the domain D = { ( t p , f p , f n , t n ) ∈ R ≥ 0 4 : t p + f n > 0 , t n + f p > 0 } \mathcal{D}=\{(tp,fp,fn,tn)\in\mathbb{R}_{\geq0}^{4}: tp+fn>0,\ tn+fp>0\} D = {( tp , f p , f n , t n ) ∈ R ≥ 0 4 : tp + f n > 0 , t n + f p > 0 } . A binary measure M : D → R M:\mathcal{D}\to\mathbb{R} M : D → R is skew-invariant, meaning
M ( λ t p , μ f p , λ f n , μ t n ) = M ( t p , f p , f n , t n ) M(\lambda tp,\mu fp,\lambda fn,\mu tn)=M(tp,fp,fn,tn) M ( λ tp , μ f p , λ f n , μ t n ) = M ( tp , f p , f n , t n ) for all λ , μ > 0 \lambda,\mu>0 λ , μ > 0 and all ( t p , f p , f n , t n ) ∈ D (tp,fp,fn,tn)\in\mathcal{D} ( tp , f p , f n , t n ) ∈ D , if and only if M M M is a function of recall rec = t p / ( t p + f n ) \operatorname{rec}=tp/(tp+fn) rec = tp / ( tp + f n ) and specificity spec = t n / ( t n + f p ) \operatorname{spec}=tn/(tn+fp) spec = t n / ( t n + f p ) alone. The positive factors λ \lambda λ and μ \mu μ independently rescale the positive and negative class counts; the domain restriction excludes undefined class-conditional rates.
Corollary 10 (Macro- and micro-averaging under skew). Let M = g ( r e c , s p e c ) M = g(\mathrm{rec},\mathrm{spec}) M = g ( rec , spec ) be a skew-invariant binary measure (Theorem 9 ) evaluated class-wise on a collection of classes with positive actual-positive and actual-negative counts. Assume that the classifier’s class-conditional behaviour is held fixed under the prevalence shifts being considered.
If the per-class rates r e c j \mathrm{rec}_j rec j and s p e c j \mathrm{spec}_j spec j remain unchanged under the shifts (as they do for each binary subproblem in the multilabel setting, and under the corresponding condition on one-vs-rest error rates in multiclass problems), then M m a c r o M^{\mathrm{macro}} M macro is skew-invariant when the set of included classes is fixed.
The pooled measure is
M m i c r o = g ( ∑ j ω j r e c j , ∑ j ω ~ j s p e c j ) , ω j = t p j + f n j ∑ k ( t p k + f n k ) , ω ~ j = f p j + t n j ∑ k ( f p k + t n k ) . M^{\mathrm{micro}} = g\left(\sum_j \omega_j \mathrm{rec}_j, \sum_j \widetilde{\omega}_j \mathrm{spec}_j\right), \qquad\omega_j = \frac{tp_j + fn_j}{\sum_k (tp_k + fn_k)}, \qquad\widetilde{\omega}_j = \frac{fp_j + tn_j}{\sum_k (fp_k + tn_k)}. M micro = g ( j ∑ ω j rec j , j ∑ ω j spec j ) , ω j = ∑ k ( t p k + f n k ) t p j + f n j , ω j = ∑ k ( f p k + t n k ) f p j + t n j . Hence equality of the per-class rates within each argument on which g g g depends is a sufficient condition for micro skew-invariance. It is not necessary for a general function g g g : changes in the two pooled arguments can cancel, and the weights may be coupled by the classification setting. In general, micro skew-invariance holds exactly when the displayed composite expression is constant over the admissible prevalence shifts.
Appendix B shows such a cancellation for micro balanced accuracy, with unequal class-specific rates and a constant pooled value.
Proposition 11 (Exemplar-averaging is a mixture over label configurations). Partition the examples into types τ \tau τ by their true label configuration, with population share ρ τ \rho_\tau ρ τ . For a per-example measure M i M_i M i (computed via Σ n \Sigma_n Σ n ) whose conditional mean given the type, M ‾ τ = E [ M i ∣ τ ] \overline{M}_\tau= \mathbb{E}[M_i \mid\tau] M τ = E [ M i ∣ τ ] , is fixed by the classifier’s conditional behaviour,
M e x e m p l a r = ∑ τ ρ τ M ‾ τ . M^{\mathrm{exemplar}} = \sum_\tau\rho_\tau\overline{M}_\tau. M exemplar = τ ∑ ρ τ M τ . Hence M e x e m p l a r M^{\mathrm{exemplar}} M exemplar is invariant under all changes in the proportions over a fixed set of admissible types if and only if the values M ‾ τ \overline{M}_\tau M τ are equal across those types; in general it varies with the population shares ρ \rho ρ even when every M ‾ τ \overline{M}_\tau M τ is held fixed. For example, with M = r e c a l l M = \mathrm{recall} M = recall , a type with true labels { A , B } \{\mathrm{A}, \mathrm{B}\} { A , B } and prediction { A } \{\mathrm{A}\} { A } has M ‾ τ = 0.5 \overline{M}_\tau= 0.5 M τ = 0.5 , while a type with true label { A } \{\mathrm{A}\} { A } and the same prediction { A } \{\mathrm{A}\} { A } has M ‾ τ = 1 \overline{M}_\tau= 1 M τ = 1 ; equal shares give M e x e m p l a r = 0.75 M^{\mathrm{exemplar}} = 0.75 M exemplar = 0.75 , while shares 0.8 and 0.2 give 0.60, although neither type’s own predictions change.
Corollary 10 is Prati’s own Corollary 5; Proposition 11 is that paper’s Proposition 7. Binary skew-invariance alone does not guarantee macro- or exemplar-invariance in every setting; both require the additional condition stated above.
Among the standard binary measures on D D D , recall and specificity are skew-invariant, while precision, F 1 F_1 F 1 , accuracy, and MCC are not. Balanced accuracy and G-mean, as functions of recall and specificity, are skew-invariant in the binary case but exhibit residual sensitivity in multiclass settings due to cross-class contamination of the false-positive counts.
The Micro-Collapse Result for Multiclass Discrimination
A fifth result from the same source is included for completeness, since it explains a well-known practitioner observation left otherwise unaddressed by the results above.
Theorem 12 (Micro-collapse in multiclass). For a multiclass problem (one-hot Y Y Y and Y ^ \hat{Y} Y ^ ), micro-precision, micro-recall, and micro-F 1 F_1 F 1 are all equal to overall accuracy:
P m i c r o = R m i c r o = F 1 m i c r o = Σ 1 ( T P ) n . P^{\mathrm{micro}} = R^{\mathrm{micro}} = F_1^{\mathrm{micro}} = \frac{\Sigma_1(\mathbf{T}\mathbf{P})}{n}. P micro = R micro = F 1 micro = n Σ 1 ( TP ) . This collapse is a consequence of the one-hot constraint: each example contributes exactly one actual positive (to its true class) and one predicted positive (to its predicted class); it is a true positive when they coincide, and a false negative and a false positive otherwise, so the global counts satisfy p p o s = t p o s = n ppos = tpos = n pp os = tp os = n and t p = tp = tp = correct predictions. The result explains the well-known practitioner observation that micro-averaged F 1 F_1 F 1 in multiclass settings equals accuracy, which holds without qualification under the indicator-matrix encoding.
Boundary of the Base Framework
Prati (2027) identify a group of measures as lying at or beyond the boundary of the indicator-matrix framework: these are measures that operate on the raw probability matrix P ^ \hat{P} P ^ and whose cell values cannot be expressed as t t t -norm outputs. The base framework reaches AUC as a threshold integral of the coincidence cells, but not as a measure on the raw score matrix. The two main families are the following.
Proper scoring rules (Brier score, log-loss): B S m i c r o = 1 n m Σ 1 ( ( Y − P ^ ) ∘ 2 ) \mathrm{BS}^{\mathrm{micro}} = \frac{1}{nm}\Sigma_1((Y-\hat{P}) \circ2) BS micro = nm 1 Σ 1 (( Y − P ^ ) ∘ 2 ) , where ∘ 2 \circ2 ∘ 2 denotes element-wise squaring. These rules assess overall probabilistic predictive quality through the magnitude of prediction errors; they are not pure calibration diagnostics, since the Brier score also reflects uncertainty and resolution.
Ranking measures (nDCG, AUC) evaluate the ordering of the predicted scores P ^ i , ⋅ \hat{P}_{i,\cdot} P ^ i , ⋅ or P ^ ⋅ , j \hat{P}_{\cdot,j} P ^ ⋅ , j rather than their absolute values, so their cell values depend on the entire predicted score vector, not on the individual entry P ^ i , j \hat{P}_{i,j} P ^ i , j .
Prati (2027) suggest these measures form a direction for future extension. This paper carries out that extension by introducing two further cell operator classes (deviation operators for calibration and positional operators for ranking) and showing that all three paradigms fit within a common algebraic structure governed by the same three aggregation operators Σ 1 \Sigma_1 Σ 1 , Σ m \Sigma_m Σ m , Σ n \Sigma_n Σ n .
Related Work
Research on classification performance evaluation can be grouped into six broad threads: catalogues of binary measures and their properties; unified algebraic or decision-theoretic frameworks; calibration and proper scoring rules; ranking evaluation; skew-sensitivity and class imbalance; and multilabel-specific evaluation. We position our contribution with respect to each.
Evaluation Surveys and Catalogues
The literature on binary classification measures is extensive. Sokolova and Lapalme (2009) analyse twenty-four measures used in binary, multiclass, multilabel and hierarchical classification and build a taxonomy of their invariance under changes of the confusion matrix, but do so measure-by-measure rather than through a generative mechanism. Ferri et al. (2009) conduct an experimental comparison showing that rankings of classifiers are highly sensitive to measure choice. Prati et al. (2011) survey graphical evaluation methods and their scalar counterparts. Japkowicz and Shah (2011) provide a comprehensive textbook treatment of evaluation methodology, covering significance testing, protocol design, and practical considerations that complement the algebraic approach taken here. These works provide taxonomies, but they analyse measures individually and do not offer a mechanism for automatically generating multiclass or multilabel extensions.
Unified Algebraic and Decision-Theoretic Frameworks
The closest work in spirit to ours is Hernández-Orallo et al. (2012), who propose a unified view of performance metrics as expected classification losses under threshold variation. Their framework operates on continuous score-based classifiers and connects thresholds to cost curves and the ROC framework. It covers the binary and cost-sensitive cases within a decision-theoretic foundation, whereas Prati (2027)’s framework operates at the level of the confusion matrix and provides an explicit algebraic mechanism for generating multiclass and multilabel measures from any binary formula. The two frameworks are complementary: the approach of Hernández-Orallo et al. gives a decision-theoretic view of discrimination, while the framework of Prati (2027) provides structural results on averaging schemes.
More recently, Shirdel et al. (2026) analyse binary confusion-matrix metrics through an underlying reward matrix , showing that each metric embeds an implicit cost–benefit trade-off that reduces to a single break-even ratio between resources invested and value gained. Their framework makes the decision-theoretic assumptions behind discrimination measures explicit and supports the design of new metrics tailored to a given cost structure; the indicator-matrix account addresses the complementary, generative question of extending binary formulae to multiclass and multilabel settings, which the present paper carries beyond discrimination.
Our present work extends Prati’s framework to calibration and ranking, which are adjacent to but outside both of the above frameworks. Flach (2003) provides a geometric treatment of ROC space that complements the algebraic view: his isometric analysis explains which classifier properties are preserved under cost variations, while our column-dependent operator formulation explains why AUC is structurally dual to nDCG. Provost and Fawcett (1997) use ROC analysis to compare classifiers when class and cost distributions are imprecise, introducing the ROC convex hull as a robust selection method; this concern with distribution shift connects to the skew-sensitivity analysis of § 5.7 .
Calibration and Proper Scoring Rules
The Brier score (Brier, [2 ] ) is the foundational calibration measure; Murphy (1973) gave the vector partition into uncertainty, reliability, and resolution components that is now standard (refining earlier partitions by Sanders and by Murphy himself), which we reformulate in matrix language (§ 5.4 ). Winkler and Murphy (1968) already define a proper scoring rule as one under which an assessor maximises the expected score if and only if the reported probabilities equal their true judgments, and give the Brier, logarithmic and spherical rules as examples; their attributes of predictions are validity and bias , not yet the reliability–resolution vocabulary. The theoretical foundations of proper scoring rules are given by Savage (1971) and the comprehensive treatment of Gneiting and Raftery (2007), whose Savage representation theorem we embed into the cell operator framework (Theorem 40 ). Kull and Flach (2015) give a four-term proper-score decomposition into adjustment loss, post-adjustment calibration loss, grouping loss, and irreducible loss. Their distinction between score adjustment and calibration is complementary to, but is not identical to, the three-term Murphy decomposition used here. Our contribution is to express column- and row-level Murphy identities in matrix aggregation notation using Σ m \Sigma_m Σ m and Σ n \Sigma_n Σ n . Bröcker (2009) shows that the reliability–resolution decomposition extends from the Brier score to every proper score; our matrix-language decomposition is stated for squared error, and Bröcker’s result delimits how far it can be generalised within the proper subfamily. Closest to our binning-free variant, Dimitriadis et al. (2021) estimate the calibration map by isotonic regression of the outcomes on the forecasts (the CORP approach, after the pool-adjacent-violators algorithm of Ayer et al., 1955) and decompose a proper score into miscalibration, discrimination and uncertainty as score differences. The calibration–resolution form of the decomposition for a single target is due to Pohle (2020), and Fissler et al. (2022) present it as a user guide for the consistent scoring functions used in regression and classification, with CORP reliability diagrams. Proposition 28 is the squared-error case in our notation, with its relation to the level-set and binned terms; the estimator and its statistical properties are theirs, and what we add is the matrix reading for the columns and rows of a multilabel matrix, together with the class-conditional pair ( α j , β j ) (\alpha_j,\beta_j) ( α j , β j ) .
Outcome-conditioned decompositions also have a long history in forecast verification. Murphy and Winkler (1987) factorise the joint distribution of forecasts and observations as likelihood times base rate, conditioning on the observation rather than on the forecast; Yates (1982) decomposes the Brier score into, among other terms, the covariance between forecast and outcome and a minimum conditional variance; and Merkle and Hartman (2018) review the Murphy and Yates decompositions and extend them to weighted scores. The pair ( α j , β j ) (\alpha_j,\beta_j) ( α j , β j ) belongs to the same outcome-conditioned family but is a different summary: writing Y ^ \hat{Y} Y ^ for the forecast, α j = Var ( Y ^ ∣ Y = 1 ) + ( 1 − E [ Y ^ ∣ Y = 1 ] ) 2 \alpha_j=\operatorname{Var}(\hat{Y}\mid Y=1)+(1-\mathbb{E}[\hat{Y}\mid Y=1])^2 α j = Var ( Y ^ ∣ Y = 1 ) + ( 1 − E [ Y ^ ∣ Y = 1 ] ) 2 and β j = Var ( Y ^ ∣ Y = 0 ) + E [ Y ^ ∣ Y = 0 ] 2 \beta_j=\operatorname{Var}(\hat{Y}\mid Y=0)+\mathbb{E}[\hat{Y}\mid Y=0]^2 β j = Var ( Y ^ ∣ Y = 0 ) + E [ Y ^ ∣ Y = 0 ] 2 , that is, within each outcome class a dispersion term plus a squared bias. Unlike Yates’ covariance, equal for a binary outcome to π j ( 1 − π j ) ( E [ Y ^ ∣ Y = 1 ] − E [ Y ^ ∣ Y = 0 ] ) \pi_j(1-\pi_j)(\mathbb{E}[\hat{Y}\mid Y=1]-\mathbb{E}[\hat{Y}\mid Y=0]) π j ( 1 − π j ) ( E [ Y ^ ∣ Y = 1 ] − E [ Y ^ ∣ Y = 0 ]) , uses only first moments. What the Calibration Framework adds to this family is the skew-invariance characterisation (Theorem 47 ), not a new decomposition.
The practical importance of calibration for neural networks is demonstrated by Guo et al. (2017), who show that modern deep classifiers are systematically overconfident and that temperature scaling (a single-parameter post-hoc method) substantially improves calibration. Post-hoc calibration methods, including Platt scaling (Platt, 2000) for SVMs and isotonic regression for decision trees (Zadrozny and Elkan, 2002), convert raw classifier scores into calibrated probabilities; the Calibration Framework provides the evaluation language for assessing the quality of such conversions at the cell level. Dal Pozzolo et al. (2015) document empirically the applied counterpart of Theorem 47 and Corollary 50 : undersampling the majority class to correct for imbalance, a common practice before training, systematically distorts predicted probabilities away from the true posterior and requires an explicit post-hoc correction to restore calibration. This is a prior-shift effect of the same kind as the prevalence dependence that the Calibration Framework’s skew analysis expresses through ( α j , β j ) (\alpha_j,\beta_j) ( α j , β j ) , although theirs concerns the training-set prior and not the evaluation prevalence. Wallace and Dahabreh (2012, 2014) approach the same problem from the evaluation side: they show that, under imbalance, class probability estimates systematically underestimate minority-class probabilities even when overall calibration looks good, and propose the stratified Brier score (the Brier score computed separately over positive and negative examples, motivated by analogy with the sensitivity/specificity decomposition of accuracy) together with a remedy that bags estimators calibrated over balanced bootstrap samples. In the binary case, the two outcome-stratified Brier components coincide with ( α j , β j ) (\alpha_j,\beta_j) ( α j , β j ) of Definition 44 ; the pair is not being identified with one undifferentiated scalar score. The Calibration Framework makes the analogy exact (Corollary 52 ), embeds the pair in the matrix formalism, and characterises it as the skew-invariant basis (Theorem 47 ). Class-weighted variants of the Brier score also appear in risk prediction: Gerds et al. (2008) weight cases and controls differently, in the form 1 n ∑ i { w 1 Y i ( 1 − r ^ i ) 2 + w 0 ( 1 − Y i ) r ^ i 2 } \frac{1}{n}\sum_i\{w_1Y_i(1-\hat r_i)^2+w_0(1-Y_i)\hat r_i^2\} n 1 ∑ i { w 1 Y i ( 1 − r ^ i ) 2 + w 0 ( 1 − Y i ) r ^ i 2 } discussed by Zhu et al. (2025), who contrast it with their own decision-theoretic weighting. In the present framework such a score is w 1 π j α j + w 0 ( 1 − π j ) β j w_1\pi_j\alpha_j+w_0(1-\pi_j)\beta_j w 1 π j α j + w 0 ( 1 − π j ) β j , and Remark 51 records when it is skew-invariant. Widmann et al. (2019) propose kernel-based calibration tests for multi-class classifiers that assess the full distributional relationship between predicted probabilities and true labels, going beyond the mean squared error captured by the Brier score; we identify this distributional calibration direction as an extension of the present framework.
DeGroot and Fienberg (1983) give an early unified treatment of forecaster evaluation that connects proper scoring rules to information theory and decision-making; Niculescu-Mizil and Caruana (2005) show empirically how different classifier families differ in their calibration behaviour, providing practical context for the theoretical results of § 5 .
Ranking Evaluation
Gain-based ranking evaluation traces to Järvelin and Kekäläinen (2002), who developed cumulative and discounted cumulative gain as alternatives to set-based precision-at-k k k in information retrieval. We use the now-standard logarithmic discount w ( r ) = 1 / log 2 ( r + 1 ) w(r)=1/\log_2(r+1) w ( r ) = 1/ log 2 ( r + 1 ) in a row-positional construction, without attributing every later normalisation convention to their original formulation. The three aggregation operators generate per-class, per-example, and global versions (§ 6.3 ). Manning et al. (2008) survey the IR evaluation literature in which nDCG plays a central role. Wang et al. (2013) analyse nDCG-type measures through the notion of consistent distinguishability : under the authors’ asymptotic sampling and ranking assumptions, standard nDCG with a logarithmic discount converges to 1 as the number of ranked items grows even though it retains consistent distinguishability; whether a discount has this property depends on its decay rate; their results concern the choice of w w w , whereas ours concern the aggregation structure for a fixed w w w .
The AUC and its connection to the Wilcoxon–Mann–Whitney (WMW) statistic is established by Wilcoxon (1945), Mann and Whitney (1947), and Hanley and McNeil (1982).
We exploit this equivalence to define the label-dependent concordance cells whose per-class mean is AUC (§ 6.9 ). An individual AUC cell also depends on which examples are negative and need not equal a fixed function of the column rank alone. Rather, the fixed-weight column-rank operator (Definition 71 ) yields an affine representation of the aggregate AUC through mean positive rank when scores are distinct (Proposition 73 ). The column-dependent fixed-weight rank operator with a general weight v v v is closely related to the two-sample linear rank statistics W n , m ϕ = ∑ i ϕ ( Rank ( X i ) / ( N + 1 ) ) W_{n,m}^{\phi} = \sum_i \phi(\operatorname{Rank}(X_i)/(N+1)) W n , m ϕ = ∑ i ϕ ( Rank ( X i ) / ( N + 1 )) of Clémençon et al. (2021), who show that AUC, DCG, the p p p -norm push and local AUC are all instances of this family, indexed by the score-generating function ϕ \phi ϕ : their rank is computed within the pooled positive-and-negative sample, exactly as rank j ( i ) \operatorname{rank}_j(i) rank j ( i ) ranks example i i i within the whole column Y ˙ ⋅ , j \dot{Y}_{\cdot,j} Y ˙ ⋅ , j here, and the linear rank choice ϕ ( u ) = u \phi(u)=u ϕ ( u ) = u gives the WMW/AUC relation at the aggregate level, after the appropriate count-dependent affine normalisation (for distinct scores). Their contribution and ours are complementary: they establish concentration inequalities and generalisation bounds for empirical maximisers of this family within a single bipartite-ranking population, with no matrix structure or multiclass/multilabel aggregation; the present paper instead embeds the same rank-statistic idea into the indicator-matrix architecture, where it becomes only one of two axes: the row-dependent nDCG family (§ 6.2 ) ranks labels within one example rather than ranking examples in a pooled positive–negative population, an axis not addressed by their bipartite two-sample formulation. The row–column duality of Theorem 74 formalises this comparison in the present matrix framework. Hand and Till (2001) extend AUC to multi-class problems via pairwise class comparisons; Fawcett (2006) provides a comprehensive practical introduction.
Cortes and Mohri (2004) prove that AUC optimisation and error rate minimisation can favour different classifiers, which is consistent with the conclusion of § 6.12 that AUC is not a strictly proper scoring rule and measures a different property from calibration (their result concerns optimisation, not properness). Hand (2009) criticises AUC on decision-theoretic grounds (interpreted as an expected minimum loss, it applies cost distributions that depend on the classifier being evaluated) and proposes the H-measure as a coherent alternative. That critique is orthogonal to ours: the column-dependent view characterises the algebraic structure and skew-invariance of AUC independently of its decision-theoretic interpretation. Fürnkranz et al. (2008) use label ranking (ordering the labels of each example, with a calibrated threshold) for multilabel classification, the row-axis ranking setting related to the nDCG family and to the per-example measures of Definition 72 . Dembczyński et al. (2012) characterise the Bayes-optimal multilabel classifier under different loss functions, including ranking losses; their framework and ours are complementary in that they identify which classifier to train for a given loss, while we analyse the structure of the loss itself.
Skew-Sensitivity and Class Imbalance
The foundational cost-sensitive view of class imbalance is given by Elkan (2001), who shows that cost-sensitive classification reduces to reweighted probability estimation. Luque et al. (2019) quantify, metric by metric, how class imbalance distorts confusion-matrix measures, and Brzezinski et al. (2018) analyse the same behaviour through gradient and isometric visualisations in the spirit of Flach (2003). These studies document empirically the phenomenon. that the algebraic view explains structurally. Prati (2027) establish the algebraic characterisation of skew-invariance (Theorem 9), which we extend to calibration (Theorem 47 ) and ranking (Propositions 68 and 77). A unifying observation of this paper is that the skew-invariant per-class bases across all three frameworks, namely recall/specificity (Discrimination Framework), ( α j , β j ) (\alpha_j, \beta_j) ( α j , β j ) (Calibration Framework), and n D C G j c l a s s / A U C j \mathrm{nDCG}^{\mathrm{class}}_j/\mathrm{AUC}_j nDCG j class / AUC j (Ranking Framework), share a class-conditional mean template over the appropriate outcome: positive examples for recall, α j \alpha_j α j , per-class nDCG and AUC, and negative examples for specificity and β j \beta_j β j . § 7 states the hypotheses in full.
Skew-invariance itself, as Prati (2027) note, is related to but distinct from the constant baseline property of Gösgens et al. (2021), also discussed by Hand et al. (2024): constant baseline requires a measure to score a random assignment consistently regardless of how many objects are assigned to each class, constraining behaviour at one reference point, whereas skew-invariance constrains behaviour over the entire space of classifier configurations. This distinction transfers unchanged to the Calibration and Ranking Frameworks: a constant-baseline-style axiom for the Brier score or for nDCG/AUC would fix their value only under a random or degenerate predictor, which is a different requirement from the class-conditional invariance that ( α j , β j ) (\alpha_j, \beta_j) ( α j , β j ) and n D C G j c l a s s / A U C j \mathrm{nDCG}^{\mathrm{class}}_j/\mathrm{AUC}_j nDCG j class / AUC j satisfy throughout this paper (neither property implies the other). A closely related axiomatic tradition exists in the quantification literature, whose object is estimating aggregate class prevalence robustly under prior shift: Sebastiani (2020) develops an axiomatic account of what makes an evaluation measure appropriate for quantification, in the same spirit as Gösgens et al. (2021) for classification but specific to the prior-shift setting that motivates skew-invariance here. Quantification and skew-invariant classifier evaluation address the same underlying phenomenon (distributional change in class or label prevalence) from complementary directions: quantification estimates the prevalence itself, while the skew-invariant measures of this paper are constructed to be unaffected by it.
Multilabel and Multiclass Evaluation
The indicator-matrix representation of multilabel problems goes back to Tsoumakas and Katakis (2007) and Zhang and Zhou (2014), who catalogue example-based and label-based measures including Hamming loss, exemplar precision, and exemplar recall. Godbole and Sarawagi (2004) extend precision, recall, and F 1 F_1 F 1 to multilabel using set-intersection semantics, which the framework of Prati (2027) subsumes as a special case of the product t t t -norm applied to binary matrices. Our extension preserves the indicator-matrix setting across multilabel, multiclass, and binary data, subject to each measure’s stated domain and tie conventions. In particular, normalised DCG needs a positive ideal denominator, and per-class AUC needs both positive and negative examples (or an explicit degenerate-case convention); the formulas should not be read as universally defined without such conditions. Wu and Zhou (2017) also unify multilabel performance measures, relating eleven of them through label-wise and instance-wise margins and proving that maximising these margins optimises the corresponding measures (they assume tie-free predictions, the same regime in which our positional results hold). Their unification targets the learning side (which margins a training algorithm should optimise), whereas the indicator-matrix account targets the algebraic structure of the measures themselves; the two views are complementary. Koyejo et al. (2014) characterise which multilabel losses admit consistent optimisation, connecting our structural analysis to the learning-theoretic perspective.
The ordinal extension via quadratic weighted kappa builds on Cohen (1960) and the ordinal classification framework of Frank and Hall (2001); we treat QWK as a hybrid of the Calibration Framework (deviation numerator) and auxiliary column-marginal quantities, in direct parallel to Prati’s treatment of Cohen’s κ \kappa κ .
A Taxonomy of Cell Operators
The indicator-matrix framework of Prati (2027) rests on a specific choice of how to compare actual label Y i , j Y_{i,j} Y i , j and predicted label Y ^ i , j \hat{Y}_{i,j} Y ^ i , j at each matrix cell. In the Discrimination Framework, this comparison is a conjunction (the Boolean AND on binary matrices, a t t t -norm in the soft case), measuring the degree to which both values coincide. The resulting cells T P \mathit{TP} TP , F P \mathit{FP} FP , F N \mathit{FN} FN , T N \mathit{TN} TN measure discrimination, that is, whether the classifier identifies the correct label. This section shows that replacing this coincidence operator with a different cell operator yields two further evaluation paradigms, each with its own interpretation and its own family of measures.
Definition 13 (Cell operator). A cell operator is a function that assigns to each cell ( i , j ) (i,j) ( i , j ) of the label matrices a real value comparing the actual label Y i , j ∈ [ 0 , 1 ] Y_{i,j} \in[0,1] Y i , j ∈ [ 0 , 1 ] with the prediction. A cell operator is pointwise if this value depends only on the single pair of entries at ( i , j ) (i,j) ( i , j ) , i.e. D : [ 0 , 1 ] × [ 0 , 1 ] → R D : [0,1] \times[0,1] \to\mathbb{R} D : [ 0 , 1 ] × [ 0 , 1 ] → R with [ D ] i , j = D ( Y i , j , Y ^ i , j ) [D]_{i,j} = D(Y_{i,j},\hat{Y}_{i,j}) [ D ] i , j = D ( Y i , j , Y ^ i , j ) ; it is row-dependent if the value at ( i , j ) (i,j) ( i , j ) may depend on the full predicted row Y ^ i , ⋅ \hat{Y}_{i,\cdot} Y ^ i , ⋅ , i.e. D r o w : [ 0 , 1 ] × [ 0 , 1 ] m → R D_{\mathrm{row}} : [0,1] \times[0,1]^m \to\mathbb{R} D row : [ 0 , 1 ] × [ 0 , 1 ] m → R with [ D ] i , j = D r o w ( Y i , j , Y ^ i , ⋅ ) [D]_{i,j} = D_{\mathrm{row}}(Y_{i,j},\hat{Y}_{i,\cdot}) [ D ] i , j = D row ( Y i , j , Y ^ i , ⋅ ) . A cell operator is column-dependent if it may also use the actual labels in the full column. In general it has the form Φ c o l : { 1 , … , n } × [ 0 , 1 ] n × [ 0 , 1 ] n → R \Phi_{\mathrm{col}} : \{1,\ldots,n\} \times[0,1]^n \times[0,1]^n \to\mathbb{R} Φ col : { 1 , … , n } × [ 0 , 1 ] n × [ 0 , 1 ] n → R , with [ D ] i , j = Φ c o l ( i , Y ⋅ , j , Y ^ ⋅ , j ) [D]_{i,j} = \Phi_{\mathrm{col}}(i,Y_{\cdot,j},\hat{Y}_{\cdot,j}) [ D ] i , j = Φ col ( i , Y ⋅ , j , Y ^ ⋅ , j ) . The restricted form D c o l ( Y i , j , Y ^ ⋅ , j ) D_{\mathrm{col}}(Y_{i,j},\hat{Y}_{\cdot,j}) D col ( Y i , j , Y ^ ⋅ , j ) is included, but is insufficient for concordance-based AUC cells, which also depend on which other examples are negative. In every case the cell matrix induced by D D D is
D ( Y , Y ^ ) ∈ R n × m . D(Y,\hat{Y}) \in\mathbb{R}^{n \times m}. D ( Y , Y ^ ) ∈ R n × m .
Applying the aggregation operators Σ 1 \Sigma_1 Σ 1 , Σ m \Sigma_m Σ m , Σ n \Sigma_n Σ n to D ( Y , Y ^ ) D(Y,\hat{Y}) D ( Y , Y ^ ) yields scalar, per-class, and per-example summaries of the evaluation criterion encoded by D D D . Coincidence and deviation operators (§ 4.1 –§ 4.2 ) are pointwise; positional operators (§ 4.3 ) are row- or column-dependent.
We identify three qualitatively distinct classes of cell operator, each giving rise to a different evaluation framework. Each class generalises one elementary comparison between an actual and a predicted value: conjunction (do they coincide?), deviation (how far apart are they?), or position (where does the prediction rank the label?). Each elementary comparison anchors one evaluation paradigm, namely discrimination, calibration, and ranking, respectively. Figure 1 summarises the three classes; the remainder of the paper develops the Calibration and Ranking Frameworks in detail.
Figure 1. Taxonomy of the three cell operator classes. Each panel shows the operator form, its dependency structure (illustrated by the highlighted cell in the 3 × 3 3 \times3 3 × 3 matrix icon), canonical evaluation measures, proper scoring rule status, skew-invariant basis, and micro = macro collapse behaviour. Framework 3 (Positional) splits into a row-dependent subfamily (nDCG, rank within a row) and a column-dependent subfamily (AUC, rank within a column), unified by the row–column duality theorem (§ 6.1 ). T T T denotes a triangular norm; ϕ \phi ϕ is a convex symmetric function with ϕ ( 0 ) = 0 \phi(0) = 0 ϕ ( 0 ) = 0 ; w ( ⋅ ) w(\cdot) w ( ⋅ ) denotes a non-increasing discount weight for row nDCG, whereas the column AUC mean-rank construction admits the increasing rank cost v ( r ) = r v(r) = r v ( r ) = r . The cross under “strictly proper” in the Positional panel covers both subfamilies: raw positional scores are never strictly proper. For non-increasing discounts and pairwise distinct true marginals, truthful reporting maximises their expected raw score among competing reports with distinct entries (Proposition 65 ). Under the min-rank convention, allowing tied reports can invalidate weak properness; constant discounts are an exception, since every report then has the same expected raw score. Normalised nDCG and AUC need not be proper (Remark 66, Proposition 79 ).
Coincidence Operators (Discrimination Framework)
A coincidence operator measures the degree to which Y i , j Y_{i,j} Y i , j and Y ^ i , j \hat{Y}_{i,j} Y ^ i , j are simultaneously large. In the binary case it is the Boolean AND of Prati (2027); for soft values the canonical family is the t t t -norms adopted there for soft labels: T P ( y , y ^ ) = y y ^ T_P(y,\hat{y}) = y\hat{y} T P ( y , y ^ ) = y y ^ , T M ( y , y ^ ) = min ( y , y ^ ) T_M(y,\hat{y}) = \min(y,\hat{y}) T M ( y , y ^ ) = min ( y , y ^ ) , and T L ( y , y ^ ) = max ( 0 , y + y ^ − 1 ) T_L(y,\hat{y}) = \max(0,y+\hat{y}-1) T L ( y , y ^ ) = max ( 0 , y + y ^ − 1 ) . For binary Y , Y ^ Y,\hat{Y} Y , Y ^ , all three recover the Boolean AND, giving the crisp confusion-matrix cells T P TP T P , F P FP F P , F N FN F N , T N TN T N of the Discrimination Framework. For soft Y , Y ^ ∈ [ 0 , 1 ] Y,\hat{Y} \in[0,1] Y , Y ^ ∈ [ 0 , 1 ] , the product t t t -norm is singled out by marginal consistency (the characterisation theorem of Prati 2027, restated as Theorem 7 ): it is the unique t t t -norm for which T P i , j + F N i , j = Y i , j TP_{i,j} + FN_{i,j} = Y_{i,j} T P i , j + F N i , j = Y i , j and T P i , j + F P i , j = Y ^ i , j TP_{i,j} + FP_{i,j} = \hat{Y}_{i,j} T P i , j + F P i , j = Y ^ i , j , and it then satisfies the partition property T P i , j + F P i , j + F N i , j + T N i , j = 1 TP_{i,j} + FP_{i,j} + FN_{i,j} + TN_{i,j} = 1 T P i , j + F P i , j + F N i , j + T N i , j = 1 (Corollary 8 ).
Coincidence operators are designed to reward correct hard decisions, and the measures they generate (precision, recall, F 1 F_1 F 1 , MCC, …) are appropriate when only the final class assignment matters. They do not reward confidence once the predictions are thresholded: a classifier that predicts Y ^ i , j = 0.51 \hat{Y}_{i,j}=0.51 Y ^ i , j = 0.51 and one that predicts Y ^ i , j = 0.99 \hat{Y}_{i,j}=0.99 Y ^ i , j = 0.99 receive identical credit when Y i , j = 1 Y_{i,j}=1 Y i , j = 1 and the threshold is 0.5. (Applied to the soft predictions themselves, every t t t -norm gives T ( 1 , y ^ ) = y ^ T(1,\hat{y})=\hat{y} T ( 1 , y ^ ) = y ^ , so the true-positive credit does grow with y ^ \hat{y} y ^ , but it measures coincidence, not calibration.)
Deviation Operators (Calibration Framework)
A deviation operator measures the magnitude of the gap between Y i , j Y_{i,j} Y i , j and Y ^ i , j \hat{Y}_{i,j} Y ^ i , j . The natural requirement is that D ( y , y ^ ) = 0 D(y,\hat{y})=0 D ( y , y ^ ) = 0 when y = y ^ y=\hat{y} y = y ^ and D ( y , y ^ ) > 0 D(y,\hat{y})>0 D ( y , y ^ ) > 0 otherwise, with larger deviations incurring larger penalties. The prototypical family is parametrised by a convex, symmetric function ϕ : R → R ≥ 0 \phi:\mathbb{R}\to\mathbb{R}_{\ge0} ϕ : R → R ≥ 0 with ϕ ( 0 ) = 0 \phi(0)=0 ϕ ( 0 ) = 0 and ϕ ( t ) > 0 \phi(t)>0 ϕ ( t ) > 0 for t ≠ 0 t\ne0 t = 0 :
D ϕ ( y , y ^ ) = ϕ ( y − y ^ ) . (1) D_{\phi}(y,\hat{y})=\phi(y-\hat{y}). \tag*{(1)} D ϕ ( y , y ^ ) = ϕ ( y − y ^ ) . (1)
The two most important special cases are:
ϕ ( t ) = t 2 \phi(t)=t^2 ϕ ( t ) = t 2 : the squared-error operator D ( y , y ^ ) = ( y − y ^ ) 2 D(y,\hat{y})=(y-\hat{y})^2 D ( y , y ^ ) = ( y − y ^ ) 2 , inducing the Brier score family (§ 5 ).
ϕ ( t ) = ∣ t ∣ \phi(t)=|t| ϕ ( t ) = ∣ t ∣ : the absolute-error operator D ( y , y ^ ) = ∣ y − y ^ ∣ D(y,\hat{y})=|y-\hat{y}| D ( y , y ^ ) = ∣ y − y ^ ∣ , inducing a mean absolute calibration error family.
Unlike thresholded coincidence operators, deviation operators reward confident correct predictions: a prediction of 0.99 for a true positive is penalised far less than a prediction of 0.51. This is accuracy together with sharpness at the cell level; calibration proper, whether predicted probabilities match frequencies, is measured by the reliability term of § 5 . The Savage representation theorem (§ 5.6 ) characterises which members of this family yield proper scoring rules: a symmetric deviation operator is proper exactly when its weight ϕ ′ ( t ) / t \phi'(t)/t ϕ ′ ( t ) / t is symmetric about 1 2 \frac{1}{2} 2 1 (Corollary 41 (c)). Squared error is the simplest such operator, and the only power-law one; it is not the only one. Log-loss is an extended-valued variant of the deviation construction: for binary outcomes, it corresponds to ϕ ( t ) = − log ( 1 − ∣ t ∣ ) \phi(t)=-\log(1-|t|) ϕ ( t ) = − log ( 1 − ∣ t ∣ ) , with ϕ ( ± 1 ) = + ∞ \phi(\pm1)=+\infty ϕ ( ± 1 ) = + ∞ , and therefore lies outside the finite-valued family defined above.
Positional Operators (Ranking Framework)
A positional operator weights the actual label Y i , j Y_{i,j} Y i , j by a function of the rank that the prediction assigns to the entry ( i , j ) (i,j) ( i , j ) within a chosen slice of Y ^ \hat{Y} Y ^ . The slice may be a row or a column , and this choice is itself a degree of freedom, unique to this operator class, which gives rise to two dual subfamilies.
The row-dependent subfamily (nDCG) ranks label j j j among the other labels of example i i i ’s prediction row:
D w ( Y i , j , Y ^ i , . ) = Y i , j ⋅ w ( rank i ( j ) ) , rank i ( j ) = ∣ { j ′ : Y ^ i , j ′ > Y ^ i , j } ∣ + 1 , D_w(Y_{i,j},\hat{Y}_{i,.})=Y_{i,j}\cdot w(\operatorname{rank}_i(j)), \qquad\operatorname{rank}_i(j)=\left|\{j':\hat{Y}_{i,j'}>\hat{Y}_{i,j}\}\right|+1, D w ( Y i , j , Y ^ i , . ) = Y i , j ⋅ w ( rank i ( j )) , rank i ( j ) = { j ′ : Y ^ i , j ′ > Y ^ i , j } + 1 ,
with w : N → R ≥ 0 w:\mathbb{N}\to\mathbb{R}_{\ge0} w : N → R ≥ 0 non-increasing; the canonical nDCG weight is w ( r ) = 1 / log 2 ( r + 1 ) w(r)=1/\log_2(r+1) w ( r ) = 1/ log 2 ( r + 1 ) .
The column-dependent subfamily (AUC) ranks example i i i among the other examples of class j j j ’s prediction column:
D v ( Y i , j , Y ^ i , j ) = Y i , j ⋅ v ( rank j ( i ) ) , rank j ( i ) = ∣ { i ′ : Y ^ i ′ , j > Y ^ i , j } ∣ + 1 , D_v(Y_{i,j}, \hat{Y}_{i,j}) = Y_{i,j} \cdot v(\operatorname{rank}_j(i)), \qquad\operatorname{rank}_j(i) = \lvert\{i' : \hat{Y}_{i',j} > \hat{Y}_{i,j}\}\rvert+ 1, D v ( Y i , j , Y ^ i , j ) = Y i , j ⋅ v ( rank j ( i )) , rank j ( i ) = ∣{ i ′ : Y ^ i ′ , j > Y ^ i , j }∣ + 1 ,
with a fixed nonnegative rank function v : N → R ≥ 0 v : \mathbb{N} \to\mathbb{R}_{\ge0} v : N → R ≥ 0 . Unlike a direct gain weight, v v v need not be non-increasing: the mean-rank construction of AUC takes the increasing rank cost v ( r ) = r v(r) = r v ( r ) = r (Proposition 73).
The fixed-rank template applies directly to the row DCG gain cells Y i j w ( rank i ( j ) ) Y_{ij}w(\operatorname{rank}_i(j)) Y ij w ( rank i ( j )) and to the column mean-rank cost cells Y i j v ( rank j ( i ) ) Y_{ij}v(\operatorname{rank}_j(i)) Y ij v ( rank j ( i )) . These use a fixed, exogenous rank function, but they must not be identified cell by cell with AUC’s concordance cells A i j = Y i j f ( i , j ) A_{ij} = Y_{ij}f(i,j) A ij = Y ij f ( i , j ) : the latter also depend on which other examples are negative (Equation (50)). The common geometric distinction is the axis of ranking (row Y ^ i , ⋅ \hat{Y}_{i,\cdot} Y ^ i , ⋅ versus column Y ^ ⋅ , j \hat{Y}_{\cdot,j} Y ^ ⋅ , j ). The row DCG aggregate is normalised by its ideal DCG to give nDCG. For columns with both positive and negative examples and distinct scores, the mean positive column rank is instead related to per-class AUC by an affine Wilcoxon–Mann–Whitney transformation of the aggregate (Section 6.8, Proposition 73), not by an identity of the underlying cells. The normalisations act differently: nDCG divides by an ideal DCG, whereas AUC applies a decreasing affine transformation of mean rank whose coefficients depend on class counts (Proposition 73). This produces skew-invariant per-class nDCG and AUC, but normalisation alone does not guarantee skew-invariance of micro- or exemplar-level summaries (Corollary 70; Proposition 77).
The AUC concordance fraction f ( i , j ) f(i,j) f ( i , j ) is not a fixed function of an individual example’s column rank: it counts outranked negative examples and therefore depends on the other labels in column j j j . Only the resulting per-class mean admits the affine rank representation under the stated assumptions. The duality of § 6.1 (Theorem 74 ) records the shared endpoint: both n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class and A U C j \mathrm{AUC}_j AUC j are conditional means over positive examples.
Positional operators differ from the first two classes: the cell value depends on the full predicted slice , that is, the entire row (row-dependent) or column (column-dependent), not only on the single entry Y ^ i , j \hat{Y}_{i,j} Y ^ i , j . For concordance-based AUC cells, the negative-label membership within the column is also needed. The cell matrix is therefore not pointwise in the sense of Definition 13 . The aggregation operators Σ m \Sigma_m Σ m and Σ n \Sigma_n Σ n still apply to the resulting cell matrix, but the interpretation changes: correcting a single prediction Y ^ i , j \hat{Y}_{i,j} Y ^ i , j may alter the rank of every other entry in its slice, changing multiple cell values at once. The full development of both subfamilies (the DCG and AUC measure families, their collapse behaviour, skew-invariance, and the duality theorem) is deferred to § 6.1 .
Example 2 (Running example: four cell matrices). The pair ( Y , P ^ ) (\mathbf{Y}, \hat{\mathbf{P}}) ( Y , P ^ ) of Example 1 produces one cell matrix per operator class. With w 2 = 1 / log 2 3 ≈ 0.631 w_2 = 1/\log_2 3 \approx0.631 w 2 = 1/ log 2 3 ≈ 0.631 and w 3 = 1 / 2 w_3 = 1/2 w 3 = 1/2 , the product t t t -norm, the squared deviation, the row-positional gain Y i j w ( rank i ( j ) ) Y_{ij}w(\operatorname{rank}_i(j)) Y ij w ( rank i ( j )) and the column concordance fraction give
( .7 .6 0 .6 0 .4 0 0 0 0 0 0 ) , ( .09 .16 .01 .16 .01 .36 .64 .64 .49 .16 .49 .04 ) , ( 1 w 2 0 1 0 w 2 0 w 3 0 0 w 2 0 ) , ( 1 / 2 1 0 1 / 2 0 2 / 3 0 1 1 0 1 0 ) . \begin{pmatrix}
.7 & .6 & 0 \\
.6 & 0 & .4 \\
0 & 0 & 0 \\
0 & 0 & 0
\end{pmatrix},
\quad
\begin{pmatrix}
.09 & .16 & .01 \\
.16 & .01 & .36 \\
.64 & .64 & .49 \\
.16 & .49 & .04
\end{pmatrix},
\quad
\begin{pmatrix}
1 & w_2 & 0 \\
1 & 0 & w_2 \\
0 & w_3 & 0 \\
0 & w_2 & 0
\end{pmatrix},
\quad
\begin{pmatrix}
1/2 & 1 & 0 \\
1/2 & 0 & 2/3 \\
0 & 1 & 1 \\
0 & 1 & 0
\end{pmatrix}. .7 .6 0 0 .6 0 0 0 0 .4 0 0 , .09 .16 .64 .16 .16 .01 .64 .49 .01 .36 .49 .04 , 1 1 0 0 w 2 0 w 3 w 2 0 w 2 0 0 , 1/2 1/2 0 0 1 0 1 1 0 2/3 1 0 .
The first two depend on ( Y i j , P ^ i j ) (Y_{ij}, \hat{P}_{ij}) ( Y ij , P ^ ij ) alone. The positional gain of cell ( 1 , B ) (1, B) ( 1 , B ) needs the whole first row (B B B is ranked second there), and the concordance cell ( 2 , C ) (2, C) ( 2 , C ) needs the whole column C C C and its negatives: the score 0.4 exceeds 0.1 and 0.2 but not 0.7.
Framework Bridge and Axis Duality
A complete comparison of the three operator classes is deferred to Table 4 in § 7 , once all frameworks have been fully developed. Two structural results are noted here, as they motivate the design of § 5 and § 6.1 .
The first is a bridge between the Discrimination and Calibration Frameworks: although the two frameworks use different cell operators in general, they coincide exactly for binary hard predictions.
Theorem 14 (Framework bridge). Let Y , Y ^ ∈ { 0 , 1 } n × m Y, \hat{Y} \in\{0,1\}^{n \times m} Y , Y ^ ∈ { 0 , 1 } n × m be binary indicator matrices. Under the product t t t -norm T P T_{P} T P for the Discrimination Framework and the squared-error operator for the Calibration Framework:
(a) ( Y i , j − Y ^ i , j ) 2 = F P i , j + F N i , j (Y_{i,j} - \hat{Y}_{i,j})^{2} = FP_{i,j} + FN_{i,j} ( Y i , j − Y ^ i , j ) 2 = F P i , j + F N i , j for every ( i , j ) (i,j) ( i , j ) .
(b) B S m i c r o = H a m m i n g L o s s ( Y , Y ^ ) \mathrm{BS}^{\mathrm{micro}} = \mathrm{HammingLoss}(Y, \hat{Y}) BS micro = HammingLoss ( Y , Y ^ ) .
(c) B S m i c r o + L a b e l A c c u r a c y ( Y , Y ^ ) = 1 \mathrm{BS}^{\mathrm{micro}} + \mathrm{LabelAccuracy}(Y, \hat{Y}) = 1 BS micro + LabelAccuracy ( Y , Y ^ ) = 1 .
Proof See § 5 , Theorem 18 .
Theorem 14 establishes that the squared-error cell operator is the natural soft extension of the binary error operator Y ⊕ Y ^ Y \oplus\hat{Y} Y ⊕ Y ^ used for Hamming loss. The two frameworks diverge precisely in the soft setting: when Y ^ i , j ∈ ( 0 , 1 ) \hat{Y}_{i,j} \in(0,1) Y ^ i , j ∈ ( 0 , 1 ) , deviation operators penalise wrong or hesitant confidence, while thresholded coincidence operators reward only the sign of the prediction relative to a threshold. This divergence is the reason both frameworks are needed: a classifier that is highly discriminative (high F 1 F_1 F 1 ) may still be poorly calibrated (high Brier score), and vice versa.
The Calibration Framework: Deviation Operators
The Discrimination Framework evaluates whether a classifier correctly identifies which class or label applies to each example. Its cell operators (Boolean conjunction, t t t -norms) ask whether the actual and predicted labels coincide . This section introduces a complementary framework that evaluates calibration : whether the predicted soft probabilities faithfully represent the true label frequencies. Calibration matters whenever the raw probability P ^ i , j \widehat{P}_{i,j} P i , j is used downstream (for ranking, risk assessment, or decision-making under uncertainty), and not only the hard decision derived from it. Miscalibration is widespread in modern classifiers, including deep neural networks (Guo et al., 2017), and is typically corrected by post-hoc methods such as Platt scaling (Platt, 2000) or isotonic regression (Zadrozny and Elkan, 2002); these methods are easiest to analyse when calibration is formalised as a matrix-level operation, as we do here.
The Calibration Cell Operator
Replace the conjunction-based cell operators of the Discrimination Framework with a single squared-deviation operator.
Definition 16 (Calibration cell operator). For Y i , j Y_{i,j} Y i , j , Y ^ i , j ∈ [ 0 , 1 ] \widehat{Y}_{i,j} \in[0,1] Y i , j ∈ [ 0 , 1 ] , define the calibration entry
C a l i , j = ( Y i , j − Y ^ i , j ) 2 (2) \mathrm{Cal}_{i,j} = \left(Y_{i,j} - \widehat{Y}_{i,j}\right)^2 \tag*{(2)} Cal i , j = ( Y i , j − Y i , j ) 2 (2)
The calibration matrix is C a l = ( Y − Y ^ ) ∘ 2 \mathbf{Cal} = \left(\mathbf{Y} - \widehat{\mathbf{Y}}\right)^{\circ2} Cal = ( Y − Y ) ∘ 2 , where ∘ 2 \circ2 ∘ 2 denotes element-wise squaring (Hadamard square).
Applying the three aggregation operators Σ 1 \Sigma_1 Σ 1 , Σ m \Sigma_m Σ m , Σ n \Sigma_n Σ n to C a l \mathbf{Cal} Cal yields a family of Brier-score variants.
Definition 17 (Brier score family).
B S m i c r o = 1 n m Σ 1 ( C a l ) = 1 n m ∑ i , j ( Y i , j − Y ^ i , j ) 2 (3) \mathrm{BS}^{\mathrm{micro}} = \frac{1}{nm}\Sigma_1(\mathbf{Cal}) = \frac{1}{nm}\sum_{i,j}\left(Y_{i,j} - \widehat{Y}_{i,j}\right)^2 \tag*{(3)} BS micro = nm 1 Σ 1 ( Cal ) = nm 1 i , j ∑ ( Y i , j − Y i , j ) 2 (3) B S j c l a s s = 1 n [ Σ m ( C a l ) ] j = 1 n ∑ i ( Y i , j − Y ^ i , j ) 2 , j ∈ [ 1 , m ] (4) \mathrm{BS}^{\mathrm{class}}_j = \frac{1}{n}\left[\Sigma_m(\mathbf{Cal})\right]_j = \frac{1}{n}\sum_i\left(Y_{i,j} - \widehat{Y}_{i,j}\right)^2,\quad j \in[1,m] \tag*{(4)} BS j class = n 1 [ Σ m ( Cal ) ] j = n 1 i ∑ ( Y i , j − Y i , j ) 2 , j ∈ [ 1 , m ] (4) B S i e x a m p l e = 1 m [ Σ n ( C a l ) ] i = 1 m ∑ j ( Y i , j − Y ^ i , j ) 2 , i ∈ [ 1 , n ] (5) \mathrm{BS}^{\mathrm{example}}_i = \frac{1}{m}\left[\Sigma_n(\mathbf{Cal})\right]_i = \frac{1}{m}\sum_j\left(Y_{i,j} - \widehat{Y}_{i,j}\right)^2,\quad i \in[1,n] \tag*{(5)} BS i example = m 1 [ Σ n ( Cal ) ] i = m 1 j ∑ ( Y i , j − Y i , j ) 2 , i ∈ [ 1 , n ] (5)
The macro-Brier score is the unweighted average of the per-class values, B S m a c r o = 1 m ∑ j B S j c l a s s \mathrm{BS}^{\mathrm{macro}} = \frac{1}{m}\sum_j \mathrm{BS}^{\mathrm{class}}_j BS macro = m 1 ∑ j BS j class , and the exemplar-Brier score is the unweighted average of the per-example values, B S e x e m p l a r = 1 n ∑ i B S i e x a m p l e \mathrm{BS}^{\mathrm{exemplar}} = \frac{1}{n}\sum_i \mathrm{BS}^{\mathrm{example}}_i BS exemplar = n 1 ∑ i BS i example .
Equation (3) recovers, up to the normalisation by m m m , the standard multiclass Brier score [2 ] used in the multilabel literature when Y \mathbf{Y} Y is binary and Y ^ = P ^ \widehat{\mathbf{Y}} = \widehat{\mathbf{P}} Y = P is a soft probability matrix. Equation (4) gives a per-class Brier score useful for diagnosing which labels are poorly calibrated. Equation (5) gives a per-example Brier score measuring calibration quality for individual examples’ label vectors.
Connection to the Discrimination Framework: The Framework Bridge
The following theorem establishes an exact algebraic connection between the Calibration Framework and the Hamming-loss / label-accuracy measures of the Discrimination Framework.
Theorem 18 (Framework bridge). Let Y , Y ^ ∈ { 0 , 1 } n × m Y,\hat{Y} \in\{0,1\}^{n\times m} Y , Y ^ ∈ { 0 , 1 } n × m be binary indicator matrices. Then:
(a) C a l i , j = F P i , j + F N i , j \mathrm{Cal}_{i,j} = \mathrm{FP}_{i,j} + \mathrm{FN}_{i,j} Cal i , j = FP i , j + FN i , j for every ( i , j ) (i,j) ( i , j ) , i.e. C a l = F P + F N \mathbf{Cal} = \mathbf{FP} + \mathbf{FN} Cal = FP + FN (entry-wise).
(b) B S m i c r o = HammingLoss ( Y , Y ^ ) \mathrm{BS}^{\mathrm{micro}} = \operatorname{HammingLoss}(Y,\hat{Y}) BS micro = HammingLoss ( Y , Y ^ ) .
(c) B S m i c r o + LabelAccuracy ( Y , Y ^ ) = 1 \mathrm{BS}^{\mathrm{micro}} + \operatorname{LabelAccuracy}(Y,\hat{Y}) = 1 BS micro + LabelAccuracy ( Y , Y ^ ) = 1 .
Proof (a) For binary a , b ∈ { 0 , 1 } a,b \in\{0,1\} a , b ∈ { 0 , 1 } : ( a − b ) 2 = 1 (a-b)^2=1 ( a − b ) 2 = 1 iff a ≠ b a\ne b a = b , and ( a − b ) 2 = 0 (a-b)^2=0 ( a − b ) 2 = 0 iff a = b a=b a = b . Thus ( a − b ) 2 = a ⊕ b (a-b)^2=a\oplus b ( a − b ) 2 = a ⊕ b (XOR). In the indicator matrix language, Y i , j ⊕ Y ^ i , j = 1 Y_{i,j}\oplus\hat{Y}_{i,j}=1 Y i , j ⊕ Y ^ i , j = 1 iff exactly one of the two is 1, which corresponds to either F P i , j = 1 \mathrm{FP}_{i,j}=1 FP i , j = 1 (predicted positive, actually negative) or F N i , j = 1 \mathrm{FN}_{i,j}=1 FN i , j = 1 (predicted negative, actually positive); both cases are mutually exclusive by definition. Hence C a l i , j = F P i , j + F N i , j \mathrm{Cal}_{i,j}=\mathrm{FP}_{i,j}+\mathrm{FN}_{i,j} Cal i , j = FP i , j + FN i , j .
(b) By (a) and the definition of Hamming loss of Prati (2027),
B S m i c r o = 1 n m Σ 1 ( C a l ) = 1 n m Σ 1 ( F P + F N ) = 1 n m Σ 1 ( Y ⊕ Y ^ ) = HammingLoss . \mathrm{BS}^{\mathrm{micro}}=\frac{1}{nm}\Sigma_1(\mathbf{Cal})=\frac{1}{nm}\Sigma_1(\mathbf{FP}+\mathbf{FN})=\frac{1}{nm}\Sigma_1(Y\oplus\hat{Y})=\operatorname{HammingLoss}. BS micro = nm 1 Σ 1 ( Cal ) = nm 1 Σ 1 ( FP + FN ) = nm 1 Σ 1 ( Y ⊕ Y ^ ) = HammingLoss .
(c) For binary indicator matrices, LabelAccuracy = ( Σ 1 ( T P ) + Σ 1 ( T N ) ) / ( n m ) \operatorname{LabelAccuracy}=(\Sigma_1(\mathbf{TP})+\Sigma_1(\mathbf{TN}))/(nm) LabelAccuracy = ( Σ 1 ( TP ) + Σ 1 ( TN )) / ( nm ) . Since the four matrices T P \mathbf{TP} TP , F P \mathbf{FP} FP , F N \mathbf{FN} FN , T N \mathbf{TN} TN partition every entry (for binary matrices this is immediate; in general it is Corollary 8 ), Σ 1 ( T P ) + Σ 1 ( F P ) + Σ 1 ( F N ) + Σ 1 ( T N ) = n m \Sigma_1(\mathbf{TP})+\Sigma_1(\mathbf{FP})+\Sigma_1(\mathbf{FN})+\Sigma_1(\mathbf{TN})=nm Σ 1 ( TP ) + Σ 1 ( FP ) + Σ 1 ( FN ) + Σ 1 ( TN ) = nm , so LabelAccuracy = 1 − ( Σ 1 ( F P ) + Σ 1 ( F N ) ) / ( n m ) = 1 − B S m i c r o \operatorname{LabelAccuracy}=1-(\Sigma_1(\mathbf{FP})+\Sigma_1(\mathbf{FN}))/(nm)=1-\mathrm{BS}^{\mathrm{micro}} LabelAccuracy = 1 − ( Σ 1 ( FP ) + Σ 1 ( FN )) / ( nm ) = 1 − BS micro .
Remark 19. Theorem 18 (c) is the calibration analogue of the trivial identity AccuracyRate+ErrorRate = 1. It shows that for binary hard predictions, the Brier score carries no information beyond Hamming loss. The two frameworks diverge precisely when predictions are soft (Y ^ i , j ∈ ( 0 , 1 ) \hat{Y}_{i,j}\in(0,1) Y ^ i , j ∈ ( 0 , 1 ) ): in that regime ( Y i , j − Y ^ i , j ) 2 (Y_{i,j}-\hat{Y}_{i,j})^2 ( Y i , j − Y ^ i , j ) 2 no longer equals Y i , j ⊕ Y ^ i , j Y_{i,j}\oplus\hat{Y}_{i,j} Y i , j ⊕ Y ^ i , j , and the calibration matrix C a l \mathbf{Cal} Cal captures confidence quality that the Boolean operators of the Discrimination Framework discard.
Remark 82 in Appendix G extends the deviation view to ordinal labels and quadratic weighted kappa.
Micro, Macro, and Exemplar Brier: A Collapse
The Calibration Framework differs from the Discrimination Framework in the behaviour of the three aggregation operators on the resulting scalar measures.
Theorem 20 (Micro-macro-exemplar collapse for Brier).
B S m i c r o = B S m a c r o = B S e x e m p l a r . \mathrm{BS}^{\mathrm{micro}}=\mathrm{BS}^{\mathrm{macro}}=\mathrm{BS}^{\mathrm{exemplar}}. BS micro = BS macro = BS exemplar .
Proof. By commutativity of finite summation:
B S m a c r o = 1 m ∑ j 1 n ∑ i ( Y i , j − Y ^ i , j ) 2 = 1 n m ∑ i , j ( Y i , j − Y ^ i , j ) 2 = B S m i c r o . \mathrm{BS}^{\mathrm{macro}} = \frac{1}{m}\sum_j \frac{1}{n}\sum_i (Y_{i,j}-\hat{Y}_{i,j})^2 = \frac{1}{nm}\sum_{i,j}(Y_{i,j}-\hat{Y}_{i,j})^2 = \mathrm{BS}^{\mathrm{micro}}. BS macro = m 1 j ∑ n 1 i ∑ ( Y i , j − Y ^ i , j ) 2 = nm 1 i , j ∑ ( Y i , j − Y ^ i , j ) 2 = BS micro .
The argument for exemplar is identical, swapping the order of summation over i i i and j j j .
This collapse follows from the additivity of the aggregation operators applied to a fixed matrix of per-cell losses, together with uniform normalisation: the Brier score is a sum of per-cell squared deviations divided by the fixed constant n m nm nm , so the order of aggregation is irrelevant. The squared-deviation cell operator itself need not be linear.
Murphy Decomposition via Column Aggregation
The Murphy decomposition ((Winkler and Murphy, 1968; Murphy, 1973)) expresses the Brier score for a single binary variable as the sum of three interpretable terms: uncertainty (inherent variability of the labels), reliability (calibration error), and resolution (discriminative power of the forecasts). We re-derive this decomposition within the matrix framework, connecting each term to column-wise aggregation Σ m \Sigma_m Σ m .
Definition 22 (Column-level Murphy terms). For class j ∈ [ 1 , m ] j \in[1,m] j ∈ [ 1 , m ] , let y ˉ j = [ Σ m ( Y ) ] j / n \bar{y}_j=[\Sigma_m(\mathbf{Y})]_j/n y ˉ j = [ Σ m ( Y ) ] j / n denote the empirical label frequency for class j j j , and let
f j ( p ) = E [ Y i , j ∣ Y ^ i , j = p ] (6) f_j(p)=\mathbb{E}[Y_{i,j}\mid\hat{Y}_{i,j}=p] \tag*{(6)} f j ( p ) = E [ Y i , j ∣ Y ^ i , j = p ] (6)
be the calibration function for class j j j : the true conditional probability that label j j j is relevant, given that the classifier predicts probability p p p . In the finite-sample setting, f j f_j f j is defined empirically as f j ( p ) = n p − 1 ∑ i : Y ^ i , j = p Y i , j f_j(p)=n_p^{-1}\sum_{i:\hat{Y}_{i,j}=p}Y_{i,j} f j ( p ) = n p − 1 ∑ i : Y ^ i , j = p Y i , j , where n p = ∣ { i : Y ^ i , j = p } ∣ n_p=\lvert\{i:\hat{Y}_{i,j}=p\}\rvert n p = ∣{ i : Y ^ i , j = p }∣ .
Define:
U N C j = y ˉ j ( 1 − y ˉ j ) (7) \mathrm{UNC}_j=\bar{y}_j(1-\bar{y}_j) \tag*{(7)} UNC j = y ˉ j ( 1 − y ˉ j ) (7)
R E S j = 1 n ∑ i = 1 n ( f j ( Y ^ i , j ) − y ˉ j ) 2 (8) \mathrm{RES}_j=\frac{1}{n}\sum_{i=1}^{n}\left(f_j(\hat{Y}_{i,j})-\bar{y}_j\right)^2 \tag*{(8)} RES j = n 1 i = 1 ∑ n ( f j ( Y ^ i , j ) − y ˉ j ) 2 (8)
R E L j = 1 n ∑ i = 1 n ( Y ^ i , j − f j ( Y ^ i , j ) ) 2 (9) \mathrm{REL}_j=\frac{1}{n}\sum_{i=1}^{n}\left(\hat{Y}_{i,j}-f_j(\hat{Y}_{i,j})\right)^2 \tag*{(9)} REL j = n 1 i = 1 ∑ n ( Y ^ i , j − f j ( Y ^ i , j ) ) 2 (9)
U N C j \mathrm{UNC}_j UNC j is the Bernoulli variance of class j j j ’s label distribution: it is maximised when y ˉ j = 0.5 \bar{y}_j=0.5 y ˉ j = 0.5 (maximally uncertain label) and zero for a label that is always present or always absent. R E S j \mathrm{RES}_j RES j measures how much the calibration function f j ( Y ^ i , j ) f_j(\hat{Y}_{i,j}) f j ( Y ^ i , j ) varies across examples. It measures variability of the calibration map and ignores its direction: when f j f_j f j is increasing, a classifier with high resolution assigns systematically higher predicted probabilities to positive examples than to negative ones; a classifier with Y ^ = 1 − Y \hat{Y}=1-Y Y ^ = 1 − Y has maximal R E S j = U N C j \mathrm{RES}_j=\mathrm{UNC}_j RES j = UNC j (and R E L j = 1 \mathrm{REL}_j=1 REL j = 1 ). R E L j \mathrm{REL}_j REL j measures the mean-squared gap between the predicted probability Y ^ i , j \hat{Y}_{i,j} Y ^ i , j and the calibration function f j ( Y ^ i , j ) f_j(\hat{Y}_{i,j}) f j ( Y ^ i , j ) ; a perfectly calibrated classifier satisfies Y ^ i , j = f j ( Y ^ i , j ) \hat{Y}_{i,j}=f_j(\hat{Y}_{i,j}) Y ^ i , j = f j ( Y ^ i , j ) for all i i i , giving R E L j = 0 \mathrm{REL}_j=0 REL j = 0 .
Lemma 23 (Group-mean residuals are orthogonal to group-level functions). Fix j j j and suppress the subscript. For any function h h h of the predicted value alone,
∑ i = 1 n h ( Y ^ i ) ( Y i − f ( Y ^ i ) ) = 0. (10) \sum_{i=1}^{n}h(\hat{Y}_i)\left(Y_i-f(\hat{Y}_i)\right)=0. \tag*{(10)} i = 1 ∑ n h ( Y ^ i ) ( Y i − f ( Y ^ i ) ) = 0. (10)
Proof Group summands by unique predicted value v ∈ range ( Y ^ ⋅ , j ) v\in\operatorname{range}(\hat{Y}_{\cdot,j}) v ∈ range ( Y ^ ⋅ , j ) , letting I v = { i : Y ^ i , j = v } I_v=\{i:\hat{Y}_{i,j}=v\} I v = { i : Y ^ i , j = v } :
∑ i ∈ I v h ( v ) ( Y i − f ( v ) ) = h ( v ) ∑ i ∈ I v ( Y i − f ( v ) ) = 0 , \sum_{i\in I_v}h(v)\left(Y_i-f(v)\right)=h(v)\sum_{i\in I_v}\left(Y_i-f(v)\right)=0, i ∈ I v ∑ h ( v ) ( Y i − f ( v ) ) = h ( v ) i ∈ I v ∑ ( Y i − f ( v ) ) = 0 ,
since by definition of f f f , ∑ i ∈ I v Y i = ∣ I v ∣ ⋅ f ( v ) \sum_{i\in I_v}Y_i=\lvert I_v\rvert\cdot f(v) ∑ i ∈ I v Y i = ∣ I v ∣ ⋅ f ( v ) , so ∑ i ∈ I v ( Y i − f ( v ) ) = 0 \sum_{i\in I_v}\left(Y_i-f(v)\right)=0 ∑ i ∈ I v ( Y i − f ( v ) ) = 0 regardless of h ( v ) h(v) h ( v ) . Summing over all v v v gives Eq. (10) .
The Murphy decomposition below uses this orthogonality of group-mean residuals to every function of the group index twice, once against h ( v ) = f ( v ) − v h(v)=f(v)-v h ( v ) = f ( v ) − v and once against h ( v ) = f ( v ) − y ˉ h(v)=f(v)-\bar{y} h ( v ) = f ( v ) − y ˉ .
Theorem 24 (Column-level Murphy decomposition). For every class j ∈ [ 1 , m ] j\in[1,m] j ∈ [ 1 , m ] , for binary Y i , j ∈ { 0 , 1 } Y_{i,j}\in\{0,1\} Y i , j ∈ { 0 , 1 } and soft predictions Y ^ i , j ∈ [ 0 , 1 ] \hat{Y}_{i,j}\in[0,1] Y ^ i , j ∈ [ 0 , 1 ] , with f j f_j f j as in Definition 22:
B S j c l a s s = U N C j − R E S j + R E L j . (11) \mathrm{BS}^{\mathrm{class}}_j=\mathrm{UNC}_j-\mathrm{RES}_j+\mathrm{REL}_j. \tag*{(11)} BS j class = UNC j − RES j + REL j . (11)
Proof Fix j j j and suppress the subscript. Add and subtract f ( Y ^ i ) f(\hat{Y}_i) f ( Y ^ i ) inside the squared loss:
( Y i − Y ^ i ) 2 = ( Y i − f ( Y ^ i ) ) 2 ⏟ irreducible residual + ( f ( Y ^ i ) − Y ^ i ) 2 ⏟ calibration error + 2 ( Y i − f ( Y ^ i ) ) ( f ( Y ^ i ) − Y ^ i ) ⏟ cross term . (12) (Y_i-\hat{Y}_i)^2=\underbrace{(Y_i-f(\hat{Y}_i))^2}_{\text{irreducible residual}}+\underbrace{(f(\hat{Y}_i)-\hat{Y}_i)^2}_{\text{calibration error}}+2\underbrace{(Y_i-f(\hat{Y}_i))(f(\hat{Y}_i)-\hat{Y}_i)}_{\text{cross term}}. \tag*{(12)} ( Y i − Y ^ i ) 2 = irreducible residual ( Y i − f ( Y ^ i ) ) 2 + calibration error ( f ( Y ^ i ) − Y ^ i ) 2 + 2 cross term ( Y i − f ( Y ^ i )) ( f ( Y ^ i ) − Y ^ i ) . (12)
The cross term vanishes: applying Lemma 23 with h ( v ) = f ( v ) − v h(v)=f(v)-v h ( v ) = f ( v ) − v gives ∑ i ( f ( Y ^ i ) − Y ^ i ) ( Y i − f ( Y ^ i ) ) = 0 \sum_i(f(\widehat{Y}_i)-\widehat{Y}_i)(Y_i-f(\widehat{Y}_i))=0 ∑ i ( f ( Y i ) − Y i ) ( Y i − f ( Y i )) = 0 . Averaging (12) over i i i therefore gives
B S = 1 n ∑ i ( Y i − f ( Y ^ i ) ) 2 + R E L . \mathrm{BS}=\frac{1}{n}\sum_i(Y_i-f(\widehat{Y}_i))^2+\mathrm{REL}. BS = n 1 i ∑ ( Y i − f ( Y i ) ) 2 + REL .
It remains to show that the first term equals U N C − R E S \mathrm{UNC}-\mathrm{RES} UNC − RES . Apply the finite-sample law of total variance to Y Y Y and f ( Y ^ ) f(\widehat{Y}) f ( Y ) :
1 n ∑ i ( Y i − y ˉ ) 2 ⏟ U N C = 1 n ∑ i ( f ( Y ^ i ) − y ˉ ) 2 ⏟ R E S + 1 n ∑ i ( Y i − f ( Y ^ i ) ) 2 ⏟ A . (13) \underbrace{\frac{1}{n}\sum_i(Y_i-\bar{y})^2}_{\mathrm{UNC}}
=
\underbrace{\frac{1}{n}\sum_i(f(\widehat{Y}_i)-\bar{y})^2}_{\mathrm{RES}}
+
\underbrace{\frac{1}{n}\sum_i(Y_i-f(\widehat{Y}_i))^2}_{A}. \tag*{(13)} UNC n 1 i ∑ ( Y i − y ˉ ) 2 = RES n 1 i ∑ ( f ( Y i ) − y ˉ ) 2 + A n 1 i ∑ ( Y i − f ( Y i ) ) 2 . (13)
The left side is U N C = y ˉ ( 1 − y ˉ ) \mathrm{UNC}=\bar{y}(1-\bar{y}) UNC = y ˉ ( 1 − y ˉ ) for binary Y Y Y . The finite-sample law of total variance holds exactly:
1 n ∑ i ( Y i − y ˉ ) 2 = 1 n ∑ i ( f ( Y ^ i ) − y ˉ ) 2 + 1 n ∑ i ( Y i − f ( Y ^ i ) ) 2 , \frac{1}{n}\sum_i(Y_i-\bar{y})^2
=
\frac{1}{n}\sum_i(f(\widehat{Y}_i)-\bar{y})^2
+
\frac{1}{n}\sum_i(Y_i-f(\widehat{Y}_i))^2, n 1 i ∑ ( Y i − y ˉ ) 2 = n 1 i ∑ ( f ( Y i ) − y ˉ ) 2 + n 1 i ∑ ( Y i − f ( Y i ) ) 2 ,
which follows from the identity ( Y i − y ˉ ) 2 = ( f ( Y ^ i ) − y ˉ ) 2 + ( Y i − f ( Y ^ i ) ) 2 + 2 ( f ( Y ^ i ) − y ˉ ) ( Y i − f ( Y ^ i ) ) (Y_i-\bar{y})^2=(f(\widehat{Y}_i)-\bar{y})^2+(Y_i-f(\widehat{Y}_i))^2+2(f(\widehat{Y}_i)-\bar{y})(Y_i-f(\widehat{Y}_i)) ( Y i − y ˉ ) 2 = ( f ( Y i ) − y ˉ ) 2 + ( Y i − f ( Y i ) ) 2 + 2 ( f ( Y i ) − y ˉ ) ( Y i − f ( Y i )) together with Lemma 23 applied once more, now with h ( v ) = f ( v ) − y ˉ h(v)=f(v)-\bar{y} h ( v ) = f ( v ) − y ˉ : the cross term ∑ i ( f ( Y ^ i ) − y ˉ ) ( Y i − f ( Y ^ i ) ) \sum_i(f(\widehat{Y}_i)-\bar{y})(Y_i-f(\widehat{Y}_i)) ∑ i ( f ( Y i ) − y ˉ ) ( Y i − f ( Y i )) vanishes by the same argument. Hence A = U N C − R E S A=\mathrm{UNC}-\mathrm{RES} A = UNC − RES , and
B S = A + R E L = ( U N C − R E S ) + R E L . \mathrm{BS}=A+\mathrm{REL}=(\mathrm{UNC}-\mathrm{RES})+\mathrm{REL}. BS = A + REL = ( UNC − RES ) + REL .
Remark 25 (Relation to binned reliability diagrams). In practice, the population calibration function f j ( p ) f_j(p) f j ( p ) cannot be determined exactly from a finite sample. If all observed predictions for class j j j are distinct, each empirical level set I v = { i : Y ^ i , j = v } \mathcal{I}_v=\{i:\widehat{Y}_{i,j}=v\} I v = { i : Y i , j = v } contains exactly one example for every observed value v v v . In that case f j ( Y ^ i , j ) = Y i , j f_j(\widehat{Y}_{i,j})=Y_{i,j} f j ( Y i , j ) = Y i , j , R E L j = B S j \mathrm{REL}_j=\mathrm{BS}_j REL j = BS j and R E S j = U N C j \mathrm{RES}_j=\mathrm{UNC}_j RES j = UNC j , so the unbinned empirical decomposition is trivial. A continuous prediction scale does not by itself ensure distinct observations. For example, with Y ^ ⋅ , j = ( 0.2 , 0.5 , 0.5 , 0.8 ) \widehat{Y}_{\cdot,j}=(0.2,0.5,0.5,0.8) Y ⋅ , j = ( 0.2 , 0.5 , 0.5 , 0.8 ) and Y ⋅ , j = ( 0 , 0 , 1 , 1 ) Y_{\cdot,j}=(0,0,1,1) Y ⋅ , j = ( 0 , 0 , 1 , 1 ) , the two examples predicted at 0.5 0.5 0.5 form the same level set, giving f j ( 0.5 ) = 0.5 f_j(0.5)=0.5 f j ( 0.5 ) = 0.5 even though their individual labels are 0 and 1. The standard approach is to group predictions into K K K bins G 1 , … , G K G_1,\ldots,G_K G 1 , … , G K (e.g. equal-width or equal-frequency) and approximate f j ( p ) ≈ y ˉ j , k = n j , k − 1 ∑ i ∈ G k Y i , j f_j(p)\approx\bar{y}_{j,k}=n_{j,k}^{-1}\sum_{i\in G_k}Y_{i,j} f j ( p ) ≈ y ˉ j , k = n j , k − 1 ∑ i ∈ G k Y i , j for p ∈ G k p\in G_k p ∈ G k . Here and in the binned sums below, bin-level means are evaluated only for nonempty bins (n j , k > 0 n_{j,k}>0 n j , k > 0 ); an empty bin has no defined sample mean and contributes no observations. Instead of splicing this approximation into the formulae of Definition 22, we use the binwise orthogonality argument underlying Theorem 24, keeping the bin identities distinct, to obtain an exact decomposition for the discretised predictor obtained by replacing every Y ^ i , j \widehat{Y}_{i,j} Y i , j with its own bin mean p ˉ j , κ ( i ) \bar{p}_{j,\kappa(i)} p ˉ j , κ ( i ) , where κ ( i ) \kappa(i) κ ( i ) denotes the bin containing example i i i . A binning-free alternative is to estimate f j f_j f j by isotonic regression (Proposition 28), which does not degenerate when the scores are distinct.
Proposition 26 (Exact binned Murphy decomposition). Let B S ^ j = 1 n ∑ i ( Y i , j − p ˉ j , κ ( i ) ) 2 \widehat{\mathrm{BS}}_j = \frac{1}{n}\sum_i (Y_{i,j}-\bar{p}_{j,\kappa(i)})^2 BS j = n 1 ∑ i ( Y i , j − p ˉ j , κ ( i ) ) 2 denote the Brier score of the discretised predictor. Then, exactly and with no residual term,
B S ^ j = U N C j − 1 n ∑ k n j , k ( y ˉ j , k − y ˉ j ) 2 + 1 n ∑ k n j , k ( y ˉ j , k − p ˉ j , k ) 2 (14) \widehat{\mathrm{BS}}_j = \mathrm{UNC}_j - \frac{1}{n}\sum_k n_{j,k}(\bar{y}_{j,k}-\bar{y}_j)^2 + \frac{1}{n}\sum_k n_{j,k}(\bar{y}_{j,k}-\bar{p}_{j,k})^2 \tag*{(14)} BS j = UNC j − n 1 k ∑ n j , k ( y ˉ j , k − y ˉ j ) 2 + n 1 k ∑ n j , k ( y ˉ j , k − p ˉ j , k ) 2 (14) where the second term is R E S j \mathrm{RES}_j RES j and the third term is R E L j \mathrm{REL}_j REL j .
Proof. Fix j j j and abbreviate a k = y ˉ j , k a_k=\bar{y}_{j,k} a k = y ˉ j , k and b k = p ˉ j , k b_k=\bar{p}_{j,k} b k = p ˉ j , k for each nonempty bin G k G_k G k . Since ∑ i ∈ G k ( Y i , j − a k ) = 0 \sum_{i\in G_k}(Y_{i,j}-a_k)=0 ∑ i ∈ G k ( Y i , j − a k ) = 0 and Y i , j Y_{i,j} Y i , j is binary, expansion within each bin gives
∑ i ∈ G k ( Y i , j − b k ) 2 = n j , k a k ( 1 − a k ) + n j , k ( a k − b k ) 2 . \sum_{i\in G_k}(Y_{i,j}-b_k)^2=n_{j,k}a_k(1-a_k)+n_{j,k}(a_k-b_k)^2. i ∈ G k ∑ ( Y i , j − b k ) 2 = n j , k a k ( 1 − a k ) + n j , k ( a k − b k ) 2 .
Moreover, ∑ k n j , k a k = n y ˉ j \sum_k n_{j,k}a_k=n\bar{y}_j ∑ k n j , k a k = n y ˉ j , and hence
∑ k n j , k a k ( 1 − a k ) = n y ˉ j ( 1 − y ˉ j ) − ∑ k n j , k ( a k − y ˉ j ) 2 . \sum_k n_{j,k}a_k(1-a_k)=n\bar{y}_j(1-\bar{y}_j)-\sum_k n_{j,k}(a_k-\bar{y}_j)^2. k ∑ n j , k a k ( 1 − a k ) = n y ˉ j ( 1 − y ˉ j ) − k ∑ n j , k ( a k − y ˉ j ) 2 .
Summing the first identity over the bins, inserting the second, and dividing by n n n proves (14). The argument groups observations by bin identity , not by unique discretised prediction value. It therefore remains valid even if two different bins have the same mean prediction b k b_k b k ; no identification of those bins is required.
Proposition 28 (Isotonic Murphy decomposition). Fix a class j j j and write p i = Y ^ i , j p_i = \hat{Y}_{i,j} p i = Y ^ i , j , y i = Y i , j y_i = Y_{i,j} y i = Y i , j , y ˉ = y ˉ j \bar{y} = \bar{y}_j y ˉ = y ˉ j . Let f ^ j \hat{f}_j f ^ j be the least-squares non-decreasing regression of y y y on p p p (the pool-adjacent-violators fit, computed on the level sets of p p p ), and put
RES j i s o = 1 n ∑ i ( f ^ j ( p i ) − y ˉ ) 2 , REL j i s o = BS j c l a s s − 1 n ∑ i ( y i − f ^ j ( p i ) ) 2 . \operatorname{RES}^{\mathrm{iso}}_j = \frac{1}{n}\sum_i(\hat{f}_j(p_i)-\bar{y})^2,\qquad\operatorname{REL}^{\mathrm{iso}}_j = \operatorname{BS}^{\mathrm{class}}_j-\frac{1}{n}\sum_i(y_i-\hat{f}_j(p_i))^2. RES j iso = n 1 i ∑ ( f ^ j ( p i ) − y ˉ ) 2 , REL j iso = BS j class − n 1 i ∑ ( y i − f ^ j ( p i ) ) 2 . Then:
(a) BS j c l a s s = UNC j − RES j i s o + REL j i s o \operatorname{BS}^{\mathrm{class}}_j = \operatorname{UNC}_j-\operatorname{RES}^{\mathrm{iso}}_j+\operatorname{REL}^{\mathrm{iso}}_j BS j class = UNC j − RES j iso + REL j iso exactly, and RES j i s o = UNC j − 1 n ∑ i ( y i − f ^ j ( p i ) ) 2 \operatorname{RES}^{\mathrm{iso}}_j = \operatorname{UNC}_j-\frac{1}{n}\sum_i(y_i-\hat{f}_j(p_i))^2 RES j iso = UNC j − n 1 ∑ i ( y i − f ^ j ( p i ) ) 2 .
(b) 0 ≤ REL j i s o ≤ REL j 0 \le\operatorname{REL}^{\mathrm{iso}}_j \le\operatorname{REL}_j 0 ≤ REL j iso ≤ REL j and 0 ≤ RES j i s o ≤ RES j 0 \le\operatorname{RES}^{\mathrm{iso}}_j \le\operatorname{RES}_j 0 ≤ RES j iso ≤ RES j , where REL j \operatorname{REL}_j REL j and RES j \operatorname{RES}_j RES j are the level-set terms of Theorem 24 .
(c) REL j i s o = 1 n ∑ i ( p i − f ^ j ( p i ) ) 2 + 2 n ∑ i ( y i − f ^ j ( p i ) ) ( f ^ j ( p i ) − p i ) \operatorname{REL}^{\mathrm{iso}}_j = \frac{1}{n}\sum_i(p_i-\hat{f}_j(p_i))^2+\frac{2}{n}\sum_i(y_i-\hat{f}_j(p_i))(\hat{f}_j(p_i)-p_i) REL j iso = n 1 ∑ i ( p i − f ^ j ( p i ) ) 2 + n 2 ∑ i ( y i − f ^ j ( p i )) ( f ^ j ( p i ) − p i ) , and the second term need not vanish, so REL j i s o \operatorname{REL}^{\mathrm{iso}}_j REL j iso is in general not the mean squared distance between the scores and the fitted calibration map.
Proof. The fit f ^ j \hat{f}_j f ^ j is constant on blocks of adjacent level sets, and its value on a block is the mean of y y y over that block. Hence ∑ i ( y i − f ^ j ( p i ) ) g ( p i ) = 0 \sum_i(y_i-\hat{f}_j(p_i))g(p_i)=0 ∑ i ( y i − f ^ j ( p i )) g ( p i ) = 0 for every g g g that is constant on the blocks, in particular for g = f ^ j − y ˉ g=\hat{f}_j-\bar{y} g = f ^ j − y ˉ . Expanding y − f ^ j = ( y − y ˉ ) − ( f ^ j − y ˉ ) y-\hat{f}_j=(y-\bar{y})-(\hat{f}_j-\bar{y}) y − f ^ j = ( y − y ˉ ) − ( f ^ j − y ˉ ) and using this orthogonality gives 1 n ∑ i ( y i − f ^ j ( p i ) ) 2 = UNC j − RES j i s o \frac{1}{n}\sum_i(y_i-\hat{f}_j(p_i))^2=\operatorname{UNC}_j-\operatorname{RES}^{\mathrm{iso}}_j n 1 ∑ i ( y i − f ^ j ( p i ) ) 2 = UNC j − RES j iso , which is the second claim of (a); the first is the definition of REL j i s o \operatorname{REL}^{\mathrm{iso}}_j REL j iso . For (b), the identity map is a non-decreasing function of p p p , so the isotonic fit cannot have a larger squared error than p p p itself, which gives REL j i s o ≥ 0 \operatorname{REL}^{\mathrm{iso}}_j \ge0 REL j iso ≥ 0 . The level-set mean f j f_j f j of Theorem 24 minimises the squared error over all functions of p p p , so it is no worse than f ^ j \hat{f}_j f ^ j ; the same orthogonality for f j f_j f j gives RES j = UNC j − BS ( f j ) \operatorname{RES}_j=\operatorname{UNC}_j-\operatorname{BS}(f_j) RES j = UNC j − BS ( f j ) and REL j = BS j c l a s s − BS ( f j ) \operatorname{REL}_j=\operatorname{BS}^{\mathrm{class}}_j-\operatorname{BS}(f_j) REL j = BS j class − BS ( f j ) , from which both inequalities follow. (c) is the expansion of ( y − p ) 2 = ( ( y − f ^ j ) + ( f ^ j − p ) ) 2 (y-p)^2=((y-\hat{f}_j)+(\hat{f}_j-p))^2 ( y − p ) 2 = (( y − f ^ j ) + ( f ^ j − p ) ) 2 .
Column- and Row-Level Murphy Decompositions
Theorem 24 decomposes the per-class Brier score via Σ m \Sigma_m Σ m . The same decomposition can be applied along the row axis using Σ n \Sigma_n Σ n , yielding a qualitatively different calibration diagnosis.
Remark 29 (When to use the isotonic terms). When all scores of a column are distinct, the level-set terms degenerate (REL j = BS j c l a s s \operatorname{REL}_j=\operatorname{BS}^{\mathrm{class}}_j REL j = BS j class , RES j = UNC j \operatorname{RES}_j=\operatorname{UNC}_j RES j = UNC j ; Remark 25), whereas REL j i s o \operatorname{REL}^{\mathrm{iso}}_j REL j iso and RES j i s o \operatorname{RES}^{\mathrm{iso}}_j RES j iso are generally informative and need neither bins nor a bin count. The price is the monotonicity assumption: f ^ j \hat{f}_j f ^ j is the best non-decreasing calibration map, so a score that is miscalibrated in a non-monotone way has part of that error absorbed into RES j i s o \operatorname{RES}^{\mathrm{iso}}_j RES j iso (Proposition 28 (b)). The estimator is the one used by the CORP approach of Dimitriadis et al. (2021), for which they establish its statistical properties; Proposition 28 states the squared-error case in the notation of this paper, for the columns and, with the same proof, the rows of a multilabel matrix. Like the level-set terms, the isotonic terms are not skew-invariant in general, since f ^ j \hat{f}_j f ^ j depends on the prevalence of the class; replicating the positives of a class changes both of them in a numerical check, and we do not state a sharper result. The pair ( α j , β j ) (\alpha_j,\beta_j) ( α j , β j ) remains the skew-invariant calibration summary.
Definition 30 (Row-level Murphy terms). For example i ∈ [ 1 , n ] i \in[1,n] i ∈ [ 1 , n ] , let Y ˉ i = [ Σ n ( Y ) ] i / m \bar{Y}_i = [\Sigma_n(\mathbf{Y})]_i/m Y ˉ i = [ Σ n ( Y ) ] i / m denote the empirical label density of example i i i (fraction of relevant labels), and let
g i ( p ) = E [ Y i , j ∣ Y ^ i , j = p ] (15) g_i(p) = \mathbb{E}\left[Y_{i,j} \mid\hat{Y}_{i,j}=p\right] \tag*{(15)} g i ( p ) = E [ Y i , j ∣ Y ^ i , j = p ] (15) be the per-example calibration function: the conditional probability that a label is relevant for example i i i given that the classifier predicts p p p for that label . In the finite-sample setting, g i ( p ) = m p − 1 ∑ j : Y ^ i , j = p Y i , j g_i(p)=m_p^{-1}\sum_{j:\hat{Y}_{i,j}=p}Y_{i,j} g i ( p ) = m p − 1 ∑ j : Y ^ i , j = p Y i , j , where m p = ∣ { j : Y ^ i , j = p } ∣ m_p=\left|\{j:\hat{Y}_{i,j}=p\}\right| m p = { j : Y ^ i , j = p } .
Define:
UNC i e x = Y ˉ i ( 1 − Y ˉ i ) (16) \operatorname{UNC}^{\mathrm{ex}}_i=\bar{Y}_i(1-\bar{Y}_i) \tag*{(16)} UNC i ex = Y ˉ i ( 1 − Y ˉ i ) (16) RES i e x = 1 m ∑ j = 1 m ( g i ( Y ^ i , j ) − Y ˉ i ) 2 (17) \operatorname{RES}^{\mathrm{ex}}_i=\frac{1}{m}\sum_{j=1}^{m}\left(g_i(\hat{Y}_{i,j})-\bar{Y}_i\right)^2 \tag*{(17)} RES i ex = m 1 j = 1 ∑ m ( g i ( Y ^ i , j ) − Y ˉ i ) 2 (17) REL i e x = 1 m ∑ j = 1 m ( Y ^ i , j − g i ( Y ^ i , j ) ) 2 (18) \operatorname{REL}^{\mathrm{ex}}_i=\frac{1}{m}\sum_{j=1}^{m}\left(\hat{Y}_{i,j}-g_i(\hat{Y}_{i,j})\right)^2 \tag*{(18)} REL i ex = m 1 j = 1 ∑ m ( Y ^ i , j − g i ( Y ^ i , j ) ) 2 (18)
Theorem 31 (Row-level Murphy decomposition). For every example i ∈ [ 1 , n ] i \in[1,n] i ∈ [ 1 , n ] , for binary Y i , j ∈ { 0 , 1 } Y_{i,j} \in\{0,1\} Y i , j ∈ { 0 , 1 } and soft predictions Y ^ i , j ∈ [ 0 , 1 ] \hat{Y}_{i,j} \in[0,1] Y ^ i , j ∈ [ 0 , 1 ] , with g i g_i g i as in Definition 30 :
BS i e x a m p l e = UNC i e x − RES i e x + REL i e x . (19) \operatorname{BS}^{\mathrm{example}}_i=\operatorname{UNC}^{\mathrm{ex}}_i-\operatorname{RES}^{\mathrm{ex}}_i+\operatorname{REL}^{\mathrm{ex}}_i. \tag*{(19)} BS i example = UNC i ex − RES i ex + REL i ex . (19)
Proof. The proof is structurally identical to that of Theorem 24, with the roles of n n n (examples) and m m m (labels) exchanged. Fix i i i and suppress its subscript. Add and subtract g ( Y ^ i , j ) g(\hat{Y}_{i,j}) g ( Y ^ i , j ) in the squared loss, group by unique predicted values within the m m m labels of example i i i , and apply the same two steps: (1) the cross term vanishes because g ( p ) g(p) g ( p ) is the conditional mean of Y i , j Y_{i,j} Y i , j over labels j j j with Y ^ i , j = p \hat{Y}_{i,j}=p Y ^ i , j = p ; (2) the finite-sample law of total variance gives
1 m ∑ j ( Y i , j − g ( Y ^ i , j ) ) 2 = UNC i e x − RES i e x . \frac{1}{m}\sum_j\left(Y_{i,j}-g(\hat{Y}_{i,j})\right)^2=\operatorname{UNC}^{\mathrm{ex}}_i-\operatorname{RES}^{\mathrm{ex}}_i. m 1 j ∑ ( Y i , j − g ( Y ^ i , j ) ) 2 = UNC i ex − RES i ex .
The row axis also admits a class-conditional decomposition of BS i e x a m p l e \operatorname{BS}^{\mathrm{example}}_i BS i example analogous to Proposition 45, obtained by conditioning on the label value Y i , j Y_{i,j} Y i , j instead of the class index: the row-axis counterpart of the class prevalence π j \pi_j π j is k i / m k_i/m k i / m , example i i i ’s own label density.
Definition 32 (Row-axis calibration errors). For example i i i with label cardinality k i = [ Σ n ( Y ) ] i = ∑ j Y i , j k_i=[\Sigma_n(\mathbf{Y})]_i=\sum_jY_{i,j} k i = [ Σ n ( Y ) ] i = ∑ j Y i , j , and 0 < k i < m 0<k_i<m 0 < k i < m , define
α i = 1 k i ∑ j : Y i , j = 1 ( 1 − Y ^ i , j ) 2 = E [ ( 1 − Y ^ i , j ) 2 ∣ Y i , j = 1 ] (20) \alpha_i=\frac{1}{k_i}\sum_{j:Y_{i,j}=1}(1-\hat{Y}_{i,j})^2=\mathbb{E}\left[(1-\hat{Y}_{i,j})^2\mid Y_{i,j}=1\right] \tag*{(20)} α i = k i 1 j : Y i , j = 1 ∑ ( 1 − Y ^ i , j ) 2 = E [ ( 1 − Y ^ i , j ) 2 ∣ Y i , j = 1 ] (20) β i = 1 m − k i ∑ j : Y i , j = 0 Y ^ i , j 2 = E [ Y ^ i , j 2 ∣ Y i , j = 0 ] . (21) \beta_i=\frac{1}{m-k_i}\sum_{j:Y_{i,j}=0}\hat{Y}_{i,j}^2=\mathbb{E}\left[\hat{Y}_{i,j}^2\mid Y_{i,j}=0\right]. \tag*{(21)} β i = m − k i 1 j : Y i , j = 0 ∑ Y ^ i , j 2 = E [ Y ^ i , j 2 ∣ Y i , j = 0 ] . (21) (For the boundary cases k i ∈ { 0 , m } k_i \in\{0,m\} k i ∈ { 0 , m } , the empty conditioning class contributes no term to (22) below.)
Proposition 33 (Prevalence decomposition of per-example Brier). For binary Y i , j ∈ { 0 , 1 } Y_{i,j}\in\{0,1\} Y i , j ∈ { 0 , 1 } :
BS i e x a m p l e = k i m α i + m − k i m β i . (22) \operatorname{BS}^{\mathrm{example}}_i=\frac{k_i}{m}\alpha_i+\frac{m-k_i}{m}\beta_i. \tag*{(22)} BS i example = m k i α i + m m − k i β i . (22)
caption: The two Murphy decompositions induced by the column and row aggregation operators. Both partition the same per-cell Brier score, but condition on different axes. Column-axis UNC is the class-prevalence Bernoulli variance; row-axis UNC is the example label-density Bernoulli variance. The two are generally unequal (see Proposition 35 ).
:::
Proof Identical to the proof of Proposition 45, with the roles of n n n (examples) and m m m (labels) exchanged. Split B S i e x a m p l e = 1 m ∑ j ( Y i , j − Y ^ i , j ) 2 \mathrm{BS}^{\mathrm{example}}_i = \frac{1}{m}\sum_j(Y_{i,j}-\hat{Y}_{i,j})^2 BS i example = m 1 ∑ j ( Y i , j − Y ^ i , j ) 2 by the value of Y i , j Y_{i,j} Y i , j :
B S i e x a m p l e = 1 m ∑ j : Y i , j = 1 ( 1 − Y ^ i , j ) 2 + 1 m ∑ j : Y i , j = 0 Y ^ i , j 2 = k i m α i + m − k i m β i . \mathrm{BS}^{\mathrm{example}}_i = \frac{1}{m}\sum_{j:Y_{i,j}=1}(1-\hat{Y}_{i,j})^2+\frac{1}{m}\sum_{j:Y_{i,j}=0}\hat{Y}_{i,j}^{2}=\frac{k_i}{m}\alpha_i+\frac{m-k_i}{m}\beta_i. BS i example = m 1 j : Y i , j = 1 ∑ ( 1 − Y ^ i , j ) 2 + m 1 j : Y i , j = 0 ∑ Y ^ i , j 2 = m k i α i + m m − k i β i .
Proposition 35 (Uncertainty terms are generally unequal). In general,
1 m ∑ j = 1 m U N C j ≠ 1 n ∑ i = 1 n U N C i e x . \frac{1}{m}\sum_{j=1}^{m}\mathrm{UNC}_j \ne\frac{1}{n}\sum_{i=1}^{n}\mathrm{UNC}^{\mathrm{ex}}_i. m 1 j = 1 ∑ m UNC j = n 1 i = 1 ∑ n UNC i ex . A sufficient condition for equality is that the label matrix Y \mathbf{Y} Y is doubly balanced: all class frequencies are equal (y ˉ j = c \bar{y}_j=c y ˉ j = c for all j j j ) and all example label densities are equal (Y ˉ i = c \bar{Y}_i=c Y ˉ i = c for all i i i ), for the same constant c c c . The condition is not necessary.
Proof Write f ( x ) = x ( 1 − x ) f(x)=x(1-x) f ( x ) = x ( 1 − x ) . The column-axis macro-UNC is 1 m ∑ j f ( y ˉ j ) \frac{1}{m}\sum_j f(\bar{y}_j) m 1 ∑ j f ( y ˉ j ) and the row-axis exemplar-UNC is 1 n ∑ i f ( Y ˉ i ) \frac{1}{n}\sum_i f(\bar{Y}_i) n 1 ∑ i f ( Y ˉ i ) . If Y Y Y is doubly balanced with common constant c c c , both sides equal f ( c ) f(c) f ( c ) , which proves sufficiency.
To see that the two sides differ in general, take m = 1 m=1 m = 1 and Y = ( 1 , 0 , 0 ) T Y=(1,0,0)^{\mathsf{T}} Y = ( 1 , 0 , 0 ) T : each row mean Y ˉ i ∈ { 0 , 1 } \bar{Y}_i \in\{0,1\} Y ˉ i ∈ { 0 , 1 } gives f ( Y ˉ i ) = 0 f(\bar{Y}_i)=0 f ( Y ˉ i ) = 0 , so the exemplar side vanishes, while the class side is f ( 1 / 3 ) = 2 / 9 > 0 f(1/3)=2/9>0 f ( 1/3 ) = 2/9 > 0 .
For non-necessity, consider
Y = ( 1 1 0 0 0 1 0 1 0 ) Y=\begin{pmatrix} 1 & 1 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \end{pmatrix} Y = 1 0 0 1 0 1 0 1 0
for which the column means are ( 1 / 3 , 2 / 3 , 1 / 3 ) (1/3, 2/3, 1/3) ( 1/3 , 2/3 , 1/3 ) and the row means are ( 2 / 3 , 1 / 3 , 1 / 3 ) (2/3, 1/3, 1/3) ( 2/3 , 1/3 , 1/3 ) ; both sides average f f f to 2 / 9 2/9 2/9 , yet Y Y Y is not doubly balanced.
A caveat applies to the row decomposition as defined: within one example there are only m m m labels, and when the observed predicted values are pairwise distinct, the empirical calibration function g i g_i g i is degenerate (g i ( Y ^ i , j ) = Y i , j g_i(\hat{Y}_{i,j})=Y_{i,j} g i ( Y ^ i , j ) = Y i , j ), which gives R E L i e x = B S i e x a m p l e \mathrm{REL}^{\mathrm{ex}}_i=\mathrm{BS}^{\mathrm{example}}_i REL i ex = BS i example and R E S i e x = U N C i e x \mathrm{RES}^{\mathrm{ex}}_i=\mathrm{UNC}^{\mathrm{ex}}_i RES i ex = UNC i ex . Informative row-level terms need discretised scores (the row analogue of Proposition 26), an isotonic calibration map (Proposition 28, whose proof applies to a row, although with only m m m labels the fit is coarse), or the ( α i , β i ) (\alpha_i,\beta_i) ( α i , β i ) split of Proposition 33, which is what Experiment 6 uses. With that understanding, a model can be well-calibrated per class (low R E L j \mathrm{REL}_j REL j ) while being poorly calibrated per example (high R E L i e x \mathrm{REL}^{\mathrm{ex}}_i REL i ex ), because the former asks whether the marginal label frequencies match predicted probabilities, while the latter asks whether, within each example, the predicted values match the frequency of relevant labels among the labels that share them (a statement about the example’s own m m m labels, not about the joint law of the label vector). This distinction is invisible to the scalar Brier score and unavailable in the Discrimination Framework.
Proper Scoring Rules and the Distinction from the Discrimination Framework
This subsection concerns properness : a proper scoring rule rewards honest probability estimates and is minimised in expectation only when the predicted probability equals the true label probability. Discrimination Framework measures lack this property; Calibration Framework measures can satisfy it. We characterise which cell operators yield proper scoring rules.
Definition 37 (Proper scoring rule induced by a cell operator). A cell operator D : [ 0 , 1 ] × [ 0 , 1 ] → R D : [0,1] \times[0,1] \to\mathbb{R} D : [ 0 , 1 ] × [ 0 , 1 ] → R induces a proper scoring rule if, for every p ∈ [ 0 , 1 ] p \in[0,1] p ∈ [ 0 , 1 ] and every p ^ ∈ [ 0 , 1 ] \hat{p} \in[0,1] p ^ ∈ [ 0 , 1 ] :
p D ( 1 , p ) + ( 1 − p ) D ( 0 , p ) ≤ p D ( 1 , p ^ ) + ( 1 − p ) D ( 0 , p ^ ) . pD(1,p) + (1-p)D(0,p) \le pD(1,\hat{p}) + (1-p)D(0,\hat{p}). p D ( 1 , p ) + ( 1 − p ) D ( 0 , p ) ≤ p D ( 1 , p ^ ) + ( 1 − p ) D ( 0 , p ^ ) . That is, the expected loss under the true Bernoulli distribution Ber ( p ) \operatorname{Ber}(p) Ber ( p ) is minimised when the predicted value equals p p p . The rule is strictly proper if equality holds only when p ^ = p \hat{p} = p p ^ = p .
Proposition 38 (t-norm operators are not proper). For any t-norm T T T , in particular for the three standard ones T ∈ { T M , T P , T L } T \in\{T_M,T_P,T_L\} T ∈ { T M , T P , T L } , the cell operator D ( y , y ^ ) = T ( y , y ^ ) D(y,\hat{y}) = T(y,\hat{y}) D ( y , y ^ ) = T ( y , y ^ ) does not induce a proper scoring rule.
Proof. Every t t t -norm has T ( 1 , q ) = q T(1,q) = q T ( 1 , q ) = q and T ( 0 , q ) = 0 T(0,q) = 0 T ( 0 , q ) = 0 for q ∈ [ 0 , 1 ] q \in[0,1] q ∈ [ 0 , 1 ] . Consequently, when Y ∼ Ber ( p ) Y \sim\operatorname{Ber}(p) Y ∼ Ber ( p ) , its expected loss as specified in Definition 37 is
L p ( q ) = p T ( 1 , q ) + ( 1 − p ) T ( 0 , q ) = p q . L_p(q) = pT(1,q) + (1-p)T(0,q) = pq. L p ( q ) = pT ( 1 , q ) + ( 1 − p ) T ( 0 , q ) = pq . For any p ∈ ( 0 , 1 ) p \in(0,1) p ∈ ( 0 , 1 ) , the truthful report q = p q = p q = p incurs loss p 2 > 0 p^2 > 0 p 2 > 0 , whereas the admissible report q = 0 q = 0 q = 0 incurs loss 0 0 0 . Thus truth-telling does not minimise the expected loss, and D = T D = T D = T is not proper. The argument applies to every t t t -norm, including T M T_M T M , T P T_P T P , and T L T_L T L , because their binary-outcome sections coincide.
Theorem 39 (Brier score is strictly proper). The calibration cell operator D ( y , y ^ ) = ( y − y ^ ) 2 D(y,\hat{y}) = (y-\hat{y})^2 D ( y , y ^ ) = ( y − y ^ ) 2 induces a strictly proper scoring rule: for any p ∈ ( 0 , 1 ) p \in(0,1) p ∈ ( 0 , 1 ) and any p ^ ∈ [ 0 , 1 ] \hat{p} \in[0,1] p ^ ∈ [ 0 , 1 ] , letting L ( p ^ ) : = p ( 1 − p ^ ) 2 + ( 1 − p ) p ^ 2 L(\hat{p}) := p(1-\hat{p})^2 + (1-p)\hat{p}^2 L ( p ^ ) := p ( 1 − p ^ ) 2 + ( 1 − p ) p ^ 2 denote the expected loss,
L ( p ^ ) ≥ L ( p ) , with equality if and only if p ^ = p . L(\hat{p}) \ge L(p), \qquad\text{with equality if and only if } \hat{p} = p. L ( p ^ ) ≥ L ( p ) , with equality if and only if p ^ = p . Proof. Completing the square directly, for every p ^ \hat{p} p ^ ,
L ( p ^ ) = p ( 1 − p ^ ) 2 + ( 1 − p ) p ^ 2 = ( p ^ − p ) 2 + p ( 1 − p ) , L(\hat{p}) = p(1-\hat{p})^2 + (1-p)\hat{p}^2 = (\hat{p}-p)^2 + p(1-p), L ( p ^ ) = p ( 1 − p ^ ) 2 + ( 1 − p ) p ^ 2 = ( p ^ − p ) 2 + p ( 1 − p ) , an identity verified by expanding both sides. Since L ( p ) = p ( 1 − p ) L(p) = p(1-p) L ( p ) = p ( 1 − p ) , this gives L ( p ^ ) − L ( p ) = ( p ^ − p ) 2 ≥ 0 L(\hat{p}) - L(p) = (\hat{p}-p)^2 \ge0 L ( p ^ ) − L ( p ) = ( p ^ − p ) 2 ≥ 0 , with equality iff p ^ = p \hat{p} = p p ^ = p : the expected loss decomposes exactly into a squared-bias term and the irreducible variance p ( 1 − p ) p(1-p) p ( 1 − p ) . The same decomposition reappears, one level down, in the Murphy decomposition of Theorem 24 . (The usual calculus argument is an immediate corollary of this identity: L ′ ( p ^ ) = 2 ( p ^ − p ) L'(\hat{p}) = 2(\hat{p}-p) L ′ ( p ^ ) = 2 ( p ^ − p ) vanishes uniquely at p ^ = p \hat{p} = p p ^ = p , and L ′ ′ ( p ^ ) = 2 > 0 L''(\hat{p}) = 2 > 0 L ′′ ( p ^ ) = 2 > 0 confirms it is a minimum.)
Theorem 40 (Savage representation in the matrix framework (Savage, 1971; Gneiting and Raftery, 2007; Kull and Flach, 2015). A cell operator D D D whose two sections D ( 1 , ⋅ ) D(1,\cdot) D ( 1 , ⋅ ) and D ( 0 , ⋅ ) D(0,\cdot) D ( 0 , ⋅ ) are continuous on [ 0 , 1 ] [0,1] [ 0 , 1 ] induces a proper scoring rule if and only if there exists a
concave function G : [ 0 , 1 ] → R G : [0,1] \to\mathbb{R} G : [ 0 , 1 ] → R , differentiable on [ 0 , 1 ] [0,1] [ 0 , 1 ] (one-sidedly at the endpoints), such that, for y ∈ { 0 , 1 } y \in\{0,1\} y ∈ { 0 , 1 } ,
D ( y , y ^ ) = G ( y ^ ) + ( y − y ^ ) G ′ ( y ^ ) . (23) D(y,\hat{y}) = G(\hat{y}) + (y-\hat{y})G'(\hat{y}). \tag*{(23)} D ( y , y ^ ) = G ( y ^ ) + ( y − y ^ ) G ′ ( y ^ ) . (23)
The rule is strictly proper iff G G G is strictly concave. Only the values of D D D at y ∈ { 0 , 1 } y \in\{0,1\} y ∈ { 0 , 1 } enter the expected loss, so the representation is a statement about those two sections; nothing is asserted about y ∈ ( 0 , 1 ) y \in(0,1) y ∈ ( 0 , 1 ) .
Proof sketch This is the representation theorem of Savage (1971) for binary outcomes, stated in the cell-operator notation of this paper. Savage and Gneiting and Raftery (2007) state the representation for positively-oriented scores (maximised at the truth) with a convex generator; Definition 37 is stated for losses (minimised at the truth), so the generator here is the negative of theirs and is concave .
For sufficiency, substitute (23) into the expected loss: E Y ∼ Ber ( p ) [ D ( Y , p ^ ) ] = G ( p ^ ) + ( p − p ^ ) G ′ ( p ^ ) \mathbb{E}_{Y \sim\operatorname{Ber}(p)}[D(Y,\hat{p})] = G(\hat{p}) + (p-\hat{p})G'(\hat{p}) E Y ∼ Ber ( p ) [ D ( Y , p ^ )] = G ( p ^ ) + ( p − p ^ ) G ′ ( p ^ ) , and the gradient inequality for concave G G G gives G ( p ^ ) + ( p − p ^ ) G ′ ( p ^ ) ≥ G ( p ) G(\hat{p}) + (p-\hat{p})G'(\hat{p}) \ge G(p) G ( p ^ ) + ( p − p ^ ) G ′ ( p ^ ) ≥ G ( p ) , with the minimum attained at p ^ = p \hat{p}=p p ^ = p .
For necessity, we use the Bayes risk and its supporting slopes. Write ℓ 1 = D ( 1 , ⋅ ) \ell_1 = D(1,\cdot) ℓ 1 = D ( 1 , ⋅ ) , ℓ 0 = D ( 0 , ⋅ ) \ell_0 = D(0,\cdot) ℓ 0 = D ( 0 , ⋅ ) and L p ( x ) = p ℓ 1 ( x ) + ( 1 − p ) ℓ 0 ( x ) L_p(x) = p\ell_1(x) + (1-p)\ell_0(x) L p ( x ) = p ℓ 1 ( x ) + ( 1 − p ) ℓ 0 ( x ) , which is affine in p p p . Properness says G ( p ) : = L p ( p ) = min x ∈ [ 0 , 1 ] L p ( x ) G(p) := L_p(p) = \min_{x \in[0,1]} L_p(x) G ( p ) := L p ( p ) = min x ∈ [ 0 , 1 ] L p ( x ) . Put d ( x ) = ℓ 1 ( x ) − ℓ 0 ( x ) d(x) = \ell_1(x)-\ell_0(x) d ( x ) = ℓ 1 ( x ) − ℓ 0 ( x ) , a continuous function; since L p ′ ′ ( x ) = G ( x ) + ( p ′ − x ) d ( x ) L_{p'}'(x) = G(x) + (p'-x)d(x) L p ′ ′ ( x ) = G ( x ) + ( p ′ − x ) d ( x ) , properness gives the supergradient inequality
G ( p ′ ) ≤ G ( x ) + ( p ′ − x ) d ( x ) for all p ′ , x ∈ [ 0 , 1 ] . (24) G(p') \le G(x) + (p'-x)d(x) \quad\text{for all } p',x \in[0,1]. \tag*{(24)} G ( p ′ ) ≤ G ( x ) + ( p ′ − x ) d ( x ) for all p ′ , x ∈ [ 0 , 1 ] . (24)
The function G G G is concave: for r = λ p + ( 1 − λ ) p ′ r = \lambda p + (1-\lambda)p' r = λ p + ( 1 − λ ) p ′ , affinity in p p p gives G ( r ) = λ L p ( r ) + ( 1 − λ ) L p ′ ( r ) ≥ λ G ( p ) + ( 1 − λ ) G ( p ′ ) G(r) = \lambda L_p(r) + (1-\lambda)L_{p'}(r) \ge\lambda G(p) + (1-\lambda)G(p') G ( r ) = λ L p ( r ) + ( 1 − λ ) L p ′ ( r ) ≥ λ G ( p ) + ( 1 − λ ) G ( p ′ ) . It is also differentiable: applying (24) to the pairs ( y , x ) (y,x) ( y , x ) and ( x , y ) (x,y) ( x , y ) gives ( y − x ) d ( y ) ≤ G ( y ) − G ( x ) ≤ ( y − x ) d ( x ) (y-x)d(y) \le G(y)-G(x) \le(y-x)d(x) ( y − x ) d ( y ) ≤ G ( y ) − G ( x ) ≤ ( y − x ) d ( x ) , so the slope of G G G between x x x and y y y lies between d ( y ) d(y) d ( y ) and d ( x ) d(x) d ( x ) , and differs from d ( x ) d(x) d ( x ) by at most ∣ d ( y ) − d ( x ) ∣ \lvert d(y)-d(x)\rvert ∣ d ( y ) − d ( x )∣ , which tends to 0 as y → x y \to x y → x by the continuity of d d d . (Only the continuity of the two sections is used.) Finally, solving G = p ℓ 1 + ( 1 − p ) ℓ 0 G = p\ell_1 + (1-p)\ell_0 G = p ℓ 1 + ( 1 − p ) ℓ 0 at p = x p=x p = x together with G ′ = ℓ 1 − ℓ 0 G' = \ell_1-\ell_0 G ′ = ℓ 1 − ℓ 0 gives ℓ 1 = G + ( 1 − x ) G ′ \ell_1 = G + (1-x)G' ℓ 1 = G + ( 1 − x ) G ′ and ℓ 0 = G − x G ′ \ell_0 = G-xG' ℓ 0 = G − x G ′ , which is (23) . For the strict case, if the rule is strictly proper then (24) is strict for p ′ ≠ x p' \ne x p ′ = x , so for p ≠ p ′ p \ne p' p = p ′ and r r r strictly between them, G ( r ) = λ L p ( r ) + ( 1 − λ ) L p ′ ( r ) > λ G ( p ) + ( 1 − λ ) G ( p ′ ) G(r) = \lambda L_p(r) + (1-\lambda)L_{p'}(r) > \lambda G(p) + (1-\lambda)G(p') G ( r ) = λ L p ( r ) + ( 1 − λ ) L p ′ ( r ) > λ G ( p ) + ( 1 − λ ) G ( p ′ ) : G G G is strictly concave; conversely a strictly concave G G G makes the tangent inequality strict. Both directions, with the strict versions, are checked in Lean (§ 9 ).
Corollary 41 (Special cases of the Savage representation). (a) The strictly concave generator G ( q ) = q ( 1 − q ) G(q) = q(1-q) G ( q ) = q ( 1 − q ) yields the Brier score D ( y , q ) = ( y − q ) 2 D(y,q) = (y-q)^2 D ( y , q ) = ( y − q ) 2 for binary y y y .
(b) The strictly concave binary-entropy generator G ( q ) = − q log q − ( 1 − q ) log ( 1 − q ) G(q) = -q\log q -(1-q)\log(1-q) G ( q ) = − q log q − ( 1 − q ) log ( 1 − q ) gives, for 0 < q < 1 0<q<1 0 < q < 1 , the binary log-loss
D ( y , q ) = − y log q − ( 1 − y ) log ( 1 − q ) = − log ( 1 − ∣ y − q ∣ ) , y ∈ { 0 , 1 } . D(y,q) = -y\log q -(1-y)\log(1-q) = -\log(1-\lvert y-q\rvert), \qquad y \in\{0,1\}. D ( y , q ) = − y log q − ( 1 − y ) log ( 1 − q ) = − log ( 1 − ∣ y − q ∣) , y ∈ { 0 , 1 } . The loss extends to q = 0 , 1 q=0,1 q = 0 , 1 by the standard endpoint conventions, including + ∞ +\infty + ∞ for a confident wrong prediction. Because its two sections are not finite and continuous on [ 0 , 1 ] [0,1] [ 0 , 1 ] , it is an extended-valued example, not a literal instance of the hypotheses of Theorem 40 .
(c) For symmetric deviation losses, suppose D ϕ ( y , q ) = ϕ ( y − q ) D_{\phi}(y,q)=\phi(y-q) D ϕ ( y , q ) = ϕ ( y − q ) , with ϕ \phi ϕ even, convex and finite-valued and continuous on [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] , and differentiable on ( 0 , 1 ) (0,1) ( 0 , 1 ) . Put w ( t ) = ϕ ′ ( t ) / t w(t)=\phi'(t)/t w ( t ) = ϕ ′ ( t ) / t for 0 < t < 1 0<t<1 0 < t < 1 . Then D ϕ D_{\phi} D ϕ is proper if and only if
w ( t ) = w ( 1 − t ) ( 0 < t < 1 ) . w(t)=w(1-t) \qquad(0<t<1). w ( t ) = w ( 1 − t ) ( 0 < t < 1 ) .
It is strictly proper whenever, in addition, w ( t ) > 0 w(t)>0 w ( t ) > 0 on ( 0 , 1 ) (0,1) ( 0 , 1 ) . In particular, squared error is the unique proper member of the convex power-loss family ϕ ( t ) = ∣ t ∣ s \phi(t)=|t|^{s} ϕ ( t ) = ∣ t ∣ s (s ≥ 1 s\ge1 s ≥ 1 ), but not the unique proper symmetric deviation loss.
(d) For any t t t -norm T T T , D ( y , q ) = T ( y , q ) D(y,q)=T(y,q) D ( y , q ) = T ( y , q ) has no concave Savage representation of the form (23) under the assumptions of Theorem 40 . Proper probability elicitation is a property of the proper-scoring subset of Calibration Framework operators, not of every operator in that framework. The Discrimination and Calibration Frameworks thus serve complementary evaluation purposes, without asserting that their full operator classes are disjoint or that all calibration losses are proper.
Proof. (a) Since G ′ ( q ) = 1 − 2 q G'(q)=1-2q G ′ ( q ) = 1 − 2 q , the Savage expression is
G ( q ) + ( y − q ) G ′ ( q ) = q ( 1 − q ) + ( y − q ) ( 1 − 2 q ) = y − 2 y q + q 2 = ( y − q ) 2 , G(q)+(y-q)G'(q)=q(1-q)+(y-q)(1-2q)=y-2yq+q^{2}=(y-q)^{2}, G ( q ) + ( y − q ) G ′ ( q ) = q ( 1 − q ) + ( y − q ) ( 1 − 2 q ) = y − 2 y q + q 2 = ( y − q ) 2 ,
using y 2 = y y^{2}=y y 2 = y for y ∈ { 0 , 1 } y\in\{0,1\} y ∈ { 0 , 1 } . Also G ′ ′ = − 2 < 0 G''=-2<0 G ′′ = − 2 < 0 .
(b) For 0 < q < 1 0<q<1 0 < q < 1 , G ′ ( q ) = log ( ( 1 − q ) / q ) G'(q)=\log((1-q)/q) G ′ ( q ) = log (( 1 − q ) / q ) , and substitution into (23) gives D ( 1 , q ) = − log q D(1,q)=-\log q D ( 1 , q ) = − log q and D ( 0 , q ) = − log ( 1 − q ) D(0,q)=-\log(1-q) D ( 0 , q ) = − log ( 1 − q ) . For binary y y y , these are exactly − log ( 1 − ∣ y − q ∣ ) -\log(1-|y-q|) − log ( 1 − ∣ y − q ∣ ) . The corresponding even convex deviation function is ϕ ( t ) = − log ( 1 − ∣ t ∣ ) \phi(t)=-\log(1-|t|) ϕ ( t ) = − log ( 1 − ∣ t ∣ ) for ∣ t ∣ < 1 |t|<1 ∣ t ∣ < 1 ; it has a kink at t = 0 t=0 t = 0 and is unbounded as ∣ t ∣ ↑ 1 |t|\uparrow1 ∣ t ∣ ↑ 1 . Thus it belongs to the extended-valued deviation family, whereas the finite-section Savage equivalence is invoked only on the interior.
(c) Evenness gives ϕ ( − q ) = ϕ ( q ) \phi(-q)=\phi(q) ϕ ( − q ) = ϕ ( q ) , so at a true Bernoulli probability p p p the expected loss from report q q q is
L p ( q ) = p ϕ ( 1 − q ) + ( 1 − p ) ϕ ( q ) . L_p(q)=p\phi(1-q)+(1-p)\phi(q). L p ( q ) = pϕ ( 1 − q ) + ( 1 − p ) ϕ ( q ) .
For 0 < q < 1 0<q<1 0 < q < 1 its derivative is
L p ′ ( q ) = − p ( 1 − q ) w ( 1 − q ) + ( 1 − p ) q w ( q ) . L'_p(q)=-p(1-q)w(1-q)+(1-p)qw(q). L p ′ ( q ) = − p ( 1 − q ) w ( 1 − q ) + ( 1 − p ) q w ( q ) .
If the rule is proper, then for each 0 < p < 1 0<p<1 0 < p < 1 the interior minimiser q = p q=p q = p satisfies L p ′ ( p ) = 0 L'_p(p)=0 L p ′ ( p ) = 0 . Hence p ( 1 − p ) [ w ( p ) − w ( 1 − p ) ] = 0 p(1-p)[w(p)-w(1-p)]=0 p ( 1 − p ) [ w ( p ) − w ( 1 − p )] = 0 , or w ( p ) = w ( 1 − p ) w(p)=w(1-p) w ( p ) = w ( 1 − p ) . Conversely, if w w w is symmetric, then
L p ′ ( q ) = w ( q ) ( q − p ) . L'_p(q)=w(q)(q-p). L p ′ ( q ) = w ( q ) ( q − p ) .
For an even convex ϕ \phi ϕ , ϕ ′ ( q ) ≥ 0 \phi'(q)\ge0 ϕ ′ ( q ) ≥ 0 on ( 0 , 1 ) (0,1) ( 0 , 1 ) , hence w ( q ) ≥ 0 w(q)\ge0 w ( q ) ≥ 0 . The expected loss therefore decreases (weakly) toward q = p q=p q = p and increases (weakly) thereafter. Continuity on the closed interval supplies the endpoint cases p , q ∈ { 0 , 1 } p,q\in\{0,1\} p , q ∈ { 0 , 1 } . This proves properness; if w ( q ) > 0 w(q)>0 w ( q ) > 0 on ( 0 , 1 ) (0,1) ( 0 , 1 ) the monotonicity is strict away from p p p , giving strict properness.
The examples of part (c), including the power-loss statement, are verified in Appendix § 12 .
(d) Any concave Savage representation of the finite-valued sections would imply properness by Theorem 40 . Proposition 38 shows that T ( y , q ) T(y,q) T ( y , q ) is not proper, giving the claimed exclusion.
Skew-Sensitivity of Calibration Measures
Prati (2027) characterise which Discrimination Framework measures are immune to class imbalance: a measure is skew-invariant if and only if it is a function of recall and specificity alone. This section establishes the calibration analogue. For Brier-type class-indexed measures, the relevant skew-invariant summaries are the class-conditional squared errors; for general measures, Theorem 47 (a) instead requires the full within-class score distributions.
Setup
Fix a class j j j and suppress the subscript. Let n + = ∑ i 1 ( Y i , j ) n^{+} = \sum_i \mathbf{1}(Y_{i,j}) n + = ∑ i 1 ( Y i , j ) and n − = n − n + n^{-} = n - n^{+} n − = n − n + denote the counts of positive and negative examples for class j j j , and let π = n + / n \pi= n^{+}/n π = n + / n be the class prevalence. The prevalence π \pi π coincides with the empirical label frequency y ˉ j \bar{y}_j y ˉ j of Definition 22 ; we write π \pi π in this section to emphasise its role as the prevalence parameter that skew transformations act upon. Throughout this subsection, the conditional means are defined for classes with both positive and negative observations: n j + > 0 , n j − > 0 n_j^{+} > 0, n_j^{-} > 0 n j + > 0 , n j − > 0 , or equivalently 0 < π j < 1 0 < \pi_j < 1 0 < π j < 1 .
Definition 44 (Class-conditional calibration errors). For class j j j , define the positive calibration error and negative calibration error as
α j = 1 n j + ∑ i : Y i , j = 1 ( 1 − Y ^ i , j ) 2 = E [ ( 1 − Y ^ i , j ) 2 ∣ Y i , j = 1 ] , (25) \alpha_j = \frac{1}{n_j^{+}} \sum_{i:Y_{i,j}=1} (1-\hat{Y}_{i,j})^2 = \mathbb{E}\left[(1-\hat{Y}_{i,j})^2 \mid Y_{i,j}=1\right], \tag*{(25)} α j = n j + 1 i : Y i , j = 1 ∑ ( 1 − Y ^ i , j ) 2 = E [ ( 1 − Y ^ i , j ) 2 ∣ Y i , j = 1 ] , (25) β j = 1 n j − ∑ i : Y i , j = 0 Y ^ i , j 2 = E [ Y ^ i , j 2 ∣ Y i , j = 0 ] . (26) \beta_j = \frac{1}{n_j^{-}} \sum_{i:Y_{i,j}=0} \hat{Y}_{i,j}^{2} = \mathbb{E}\left[\hat{Y}_{i,j}^{2} \mid Y_{i,j}=0\right]. \tag*{(26)} β j = n j − 1 i : Y i , j = 0 ∑ Y ^ i , j 2 = E [ Y ^ i , j 2 ∣ Y i , j = 0 ] . (26)
α j \alpha_j α j measures how far predicted probabilities are from 1 among the positive examples; a classifier that assigns probability 1 to every positive example (Y ^ i , j = 1 \hat{Y}_{i,j}=1 Y ^ i , j = 1 for all i i i with Y i , j = 1 Y_{i,j}=1 Y i , j = 1 ) has α j = 0 \alpha_j = 0 α j = 0 . Symmetrically, β j \beta_j β j measures how far predictions are from 0 among the negative examples. Both are defined as conditional means and are therefore independent of the class prevalence π j \pi_j π j .
In the binary case, α j \alpha_j α j and β j \beta_j β j are the Brier score computed separately over the positive and over the negative examples (Brier+ and Brier−), i.e., the two components of the stratified Brier score proposed by Wallace and Dahabreh (2012, 2014), who introduced it to expose the systematic underestimation of minority-class probabilities that the overall Brier score can mask under imbalance. They motivate the split by analogy with decomposing accuracy into sensitivity and specificity; within the present framework this analogy becomes exact: for hard predictions the pair reduces to the class-conditional error rates of the Discrimination Framework (Corollary 52 ), it characterises the skew-invariant Brier-type class-indexed measures that factor through ( π j , α j , β j ) (\pi_j,\alpha_j,\beta_j) ( π j , α j , β j ) (Theorem 47 (b)), and it extends to m m m classes and to multilabel data through the aggregation operators under the per-class skew convention.
The following proposition decomposes B S j c l a s s \mathrm{BS}^{\mathrm{class}}_j BS j class into α j \alpha_j α j and β j \beta_j β j .
Proposition 45 (Prevalence decomposition of per-class Brier). For binary Y i , j ∈ { 0 , 1 } Y_{i,j} \in\{0,1\} Y i , j ∈ { 0 , 1 } :
B S j c l a s s = π j α j + ( 1 − π j ) β j . (27) \mathrm{BS}^{\mathrm{class}}_j = \pi_j \alpha_j + (1-\pi_j)\beta_j. \tag*{(27)} BS j class = π j α j + ( 1 − π j ) β j . (27)
Proof. Split the sum in B S j c l a s s = 1 n ∑ i ( Y i , j − Y ^ i , j ) 2 \mathrm{BS}^{\mathrm{class}}_j = \frac{1}{n}\sum_i(Y_{i,j}-\widehat{Y}_{i,j})^2 BS j class = n 1 ∑ i ( Y i , j − Y i , j ) 2 by the value of Y i , j Y_{i,j} Y i , j :
B S j c l a s s = 1 n ∑ i : Y i , j = 1 ( 1 − Y ^ i , j ) 2 + 1 n ∑ i : Y i , j = 0 Y ^ i , j 2 = n j + n α j + n j − n β j = π j α j + ( 1 − π j ) β j . (28) \mathrm{BS}^{\mathrm{class}}_j = \frac{1}{n}\sum_{i:Y_{i,j}=1}(1-\widehat{Y}_{i,j})^2+\frac{1}{n}\sum_{i:Y_{i,j}=0}\widehat{Y}_{i,j}^2=\frac{n_j^+}{n}\alpha_j+\frac{n_j^-}{n}\beta_j=\pi_j\alpha_j+(1-\pi_j)\beta_j. \tag*{(28)} BS j class = n 1 i : Y i , j = 1 ∑ ( 1 − Y i , j ) 2 + n 1 i : Y i , j = 0 ∑ Y i , j 2 = n n j + α j + n n j − β j = π j α j + ( 1 − π j ) β j . (28)
Example 3 (Running example: Brier collapse and the conditional errors). Averaging the squared-deviation cells of Example 2 by columns gives B S j c l a s s = ( 21 / 80 , 13 / 40 , 9 / 40 ) \mathrm{BS}^{\mathrm{class}}_j=(21/80,13/40,9/40) BS j class = ( 21/80 , 13/40 , 9/40 ) , and by rows B S i e x a m p l e = ( 13 / 150 , 53 / 300 , 59 / 100 , 23 / 100 ) \mathrm{BS}^{\mathrm{example}}_i=(13/150,53/300,59/100,23/100) BS i example = ( 13/150 , 53/300 , 59/100 , 23/100 ) ; both average to the micro Brier score 13 / 48 13/48 13/48 (Theorem 20 ). The class-conditional errors are α = ( 1 / 8 , 43 / 100 , 9 / 25 ) \alpha=(1/8,43/100,9/25) α = ( 1/8 , 43/100 , 9/25 ) and β = ( 2 / 5 , 1 / 100 , 9 / 50 ) \beta=(2/5,1/100,9/50) β = ( 2/5 , 1/100 , 9/50 ) : the classifier is overconfident on the negatives of A and underconfident on the positives of B. For A, π A = 1 / 2 \pi_A=1/2 π A = 1/2 and B S A = 1 2 ⋅ 1 8 + 1 2 ⋅ 2 5 = 21 80 \mathrm{BS}_A=\frac{1}{2}\cdot\frac{1}{8}+\frac{1}{2}\cdot\frac{2}{5}=\frac{21}{80} BS A = 2 1 ⋅ 8 1 + 2 1 ⋅ 5 2 = 80 21 (Proposition 45 ). Two companion predictors separate calibration from ranking. Squaring every score preserves every row and column order, so all ranking measures of § 6.1 are unchanged, yet the Brier score rises to 14039 / 40000 ≈ 0.351 14039/40000\approx0.351 14039/40000 ≈ 0.351 . Conversely, the predictor with rows ( .8 , .7 , .2 ) (.8,.7,.2) ( .8 , .7 , .2 ) , ( .9 , .4 , .1 ) (.9,.4,.1) ( .9 , .4 , .1 ) , ( .4 , .2 , .3 ) (.4,.2,.3) ( .4 , .2 , .3 ) , ( .2 , .1 , .6 ) (.2,.1,.6) ( .2 , .1 , .6 ) has the same Brier score 13 / 48 13/48 13/48 but macro-AUC 4 / 9 4/9 4/9 instead of 13 / 18 13/18 13/18 .
(28) expresses B S j \mathrm{BS}_j BS j as a convex mixture of α j \alpha_j α j and β j \beta_j β j with prevalence weights. If α j ≠ β j \alpha_j\ne\beta_j α j = β j , varying π j \pi_j π j over ( 0 , 1 ) (0,1) ( 0 , 1 ) makes B S j \mathrm{BS}_j BS j range over the open interval with endpoints α j \alpha_j α j and β j \beta_j β j ; if α j = β j \alpha_j=\beta_j α j = β j , the value is constant. This mixture is the main source of skew-sensitivity. The identity itself is elementary (it is the law of total expectation over the label) and is stated only to make this mechanism explicit; the contribution of this section is the characterisation below.
Even a perfectly calibrated forecaster has non-zero α j \alpha_j α j and β j \beta_j β j : if it issues the same prediction p p p for every unit and p p p equals the true base rate, then α j = ( 1 − p ) 2 \alpha_j=(1-p)^2 α j = ( 1 − p ) 2 and β j = p 2 \beta_j=p^2 β j = p 2 . The pair therefore mixes calibration with sharpness, and is best read as a pair of class-conditional squared errors.
Skew-invariance
Following Prati (2027), we define skew-invariance by asking how a measure changes when the counts of positive and negative examples are independently rescaled while their within-class prediction distributions are held fixed.
Definition 46 (Skew-invariant calibration measure). A calibration measure M M M is skew-invariant if, for any class j j j and any λ , μ > 0 \lambda,\mu>0 λ , μ > 0 , replacing n j + n_j^+ n j + positive examples with λ n j + \lambda n_j^+ λ n j + copies and n j − n_j^- n j − negative examples with μ n j − \mu n_j^- μ n j − copies (while keeping the within-class distributions of Y ^ i , j \widehat{Y}_{i,j} Y i , j unchanged) leaves M M M unchanged. It is skew-sensitive otherwise.
Replicating whole examples also changes the columns of the other classes. As in the base paper, skew-invariance is therefore understood at the level of the per-class summaries: a measure is evaluated on the data { ( n j + , n j − , F j + , F j − ) } j = 1 m \{(n_j^+,n_j^-,F_j^+,F_j^-)\}_{j=1}^m {( n j + , n j − , F j + , F j − ) } j = 1 m , and the transformation changes the two counts of one class, all the other quantities being held fixed. A factor λ \lambda λ or μ \mu μ that is not an integer is read as the corresponding weighting of the empirical distribution; for integer factors this is literal replication, and every pair of data sets with the same { F j ± } \{F_j^{\pm}\} { F j ± } is then connected by a chain of such replications (pass through a common multiple of the counts). The theorem below holds on this representation, and is proved in Lean in the following model: for one class, the predicted scores of the positive and of the negative examples are two finite multisets, and the transformation replicates them k k k - and l l l -fold; exemplar-indexed measures are outside it (Remark 49).
Theorem 47 (Characterisation of skew-invariant calibration measures). Let F j + F_j^+ F j + and F j − F_j^- F j − denote the within-class empirical distributions of the predicted scores Y ^ i , j \hat{Y}_{i,j} Y ^ i , j over the positive and negative examples of class j j j , respectively. Both statements use the per-class skew transformations of Definition 46, not arbitrary replications of entire multilabel rows.
(a) A calibration measure M M M is skew-invariant if and only if it can be written as a function of { F j + , F j − } j = 1 m \{F_j^+,F_j^-\}_{j=1}^m { F j + , F j − } j = 1 m alone.
(b) Restrict attention to Brier-type class-indexed measures: functions g ( { π j , α j , β j } j ) g(\{\pi_j,\alpha_j,\beta_j\}_j) g ({ π j , α j , β j } j ) of the class prevalences and of the squared-error class-conditional errors (the quantities through which the per-class values B S j c l a s s \mathrm{BS}^{\mathrm{class}}_j BS j class depend on the data, by the mixture identity (27)). Measures built from another deviation operator have their own conditional errors α j ϕ , β j ϕ \alpha_j^\phi,\beta_j^\phi α j ϕ , β j ϕ and obey the same statement with them. This includes B S j c l a s s \mathrm{BS}^{\mathrm{class}}_j BS j class itself, B S m a c r o \mathrm{BS}^{\mathrm{macro}} BS macro , and (by Theorem 20) B S m i c r o \mathrm{BS}^{\mathrm{micro}} BS micro . Within this class, M M M is skew-invariant if and only if it is a function of { α j , β j } j = 1 m \{\alpha_j,\beta_j\}_{j=1}^m { α j , β j } j = 1 m alone. (Exemplar-indexed measures such as B S i e x a m p l e \mathrm{BS}^{\mathrm{example}}_i BS i example are outside this scope; see Remark 49.)
Proof sketch. (a) A skew transformation changes only the counts ( n j + , n j − ) (n_j^+,n_j^-) ( n j + , n j − ) and leaves every F j ± F_j^\pm F j ± fixed, and two data sets with the same { F j + , F j − } \{F_j^+,F_j^-\} { F j + , F j − } are linked by a chain of such transformations; invariance is therefore exactly dependence on { F j + , F j − } \{F_j^+,F_j^-\} { F j + , F j − } . (b) α j \alpha_j α j and β j \beta_j β j are means under F j ± F_j^\pm F j ± ; conversely, rescaling sweeps π j \pi_j π j over ( 0 , 1 ) (0,1) ( 0 , 1 ) with ( α j , β j ) (\alpha_j,\beta_j) ( α j , β j ) fixed. The full proof is in Appendix D.
Remark 48 (Scope of the characterisation). Part (a) shows that the full invariant of the skew relation is the pair of within-class score distributions, not the pair ( α j , β j ) (\alpha_j,\beta_j) ( α j , β j ) : measures such as the within-class variance of predicted scores are skew-invariant without being functions of ( α j , β j ) (\alpha_j,\beta_j) ( α j , β j ) . Within the restricted class of Brier-type measures that factor through the conditional mean squared errors, as specified in Theorem 47(b), the pair ( α j , β j ) (\alpha_j,\beta_j) ( α j , β j ) is sufficient to characterise skew-invariant measures. It is not a complete invariant for arbitrary functionals of squared-error cells.
Remark 49 (Exemplar-indexed measures are outside part (b)’s scope). Part (b) concerns class-indexed measures only. Exemplar-indexed quantities such as B S i e x a m p l e \mathrm{BS}^{\mathrm{example}}_i BS i example (Equation 5) or the row-level Murphy terms of Definition 30 depend on how errors are distributed within each example’s label vector, which the per-class summaries { π j , α j , β j } j \{\pi_j,\alpha_j,\beta_j\}_j { π j , α j , β j } j do not determine; moreover, a per-class skew transformation duplicates or removes whole examples rather than acting on a fixed one. A skew-invariance theory for exemplar-indexed measures is not developed here.
Corollary 50 (Skew-sensitivity of B S c l a s s \mathrm{BS}^{\mathrm{class}} BS class ). Under the per-class skew transformations of Definition 46, B S j c l a s s \mathrm{BS}^{\mathrm{class}}_j BS j class is skew-sensitive precisely when α j ≠ β j \alpha_j \ne\beta_j α j = β j . If this holds for at least one class, the class-averaged B S m a c r o \mathrm{BS}^{\mathrm{macro}} BS macro is skew-sensitive, and so are the numerically identical B S m i c r o \mathrm{BS}^{\mathrm{micro}} BS micro and B S e x e m p l a r \mathrm{BS}^{\mathrm{exemplar}} BS exemplar by Theorem 20. This statement concerns independent per-class changes of prevalence; it does not assert sensitivity under every joint replication of entire multilabel examples.
Proof By Proposition 45, B S j = π j α j + ( 1 − π j ) β j \mathrm{BS}_j=\pi_j\alpha_j+(1-\pi_j)\beta_j BS j = π j α j + ( 1 − π j ) β j depends explicitly on π j \pi_j π j , and hence on the class counts n j + n_j^+ n j + and n j − n_j^- n j − . It is invariant to their rescaling only if α j = β j \alpha_j=\beta_j α j = β j , i.e., the classifier has identical calibration error on positive and negative examples. Skew-sensitivity of B S m a c r o \mathrm{BS}^{\mathrm{macro}} BS macro follows since it is an unweighted average of the (skew-sensitive) B S j \mathrm{BS}_j BS j ; skew-sensitivity of B S m i c r o \mathrm{BS}^{\mathrm{micro}} BS micro and of B S e x e m p l a r \mathrm{BS}^{\mathrm{exemplar}} BS exemplar then follows from Theorem 20, since both equal B S m a c r o \mathrm{BS}^{\mathrm{macro}} BS macro as scalars for every dataset, skewed or not. ■
The scalar Brier score is therefore skew-sensitive. The next corollary relates ( α j , β j ) (\alpha_j,\beta_j) ( α j , β j ) to the skew-invariant quantities already established for the Discrimination Framework; the connection is exact in the hard-prediction limit.
Corollary 52 (Connection to the Discrimination Framework for hard predictions). For binary hard predictions Y ^ i , j ∈ { 0 , 1 } \hat{Y}_{i,j}\in\{0,1\} Y ^ i , j ∈ { 0 , 1 } :
α j = 1 − recall j , \alpha_j = 1-\operatorname{recall}_j, α j = 1 − recall j , β j = 1 − specificity j . \beta_j = 1-\operatorname{specificity}_j. β j = 1 − specificity j .
Consequently, B S j c l a s s = π j ( 1 − recall j ) + ( 1 − π j ) ( 1 − specificity j ) \mathrm{BS}^{\mathrm{class}}_j=\pi_j(1-\operatorname{recall}_j)+(1-\pi_j)(1-\operatorname{specificity}_j) BS j class = π j ( 1 − recall j ) + ( 1 − π j ) ( 1 − specificity j ) , which is the prevalence-weighted error rate, and the skew-invariant calibration measures ( α j , β j ) (\alpha_j,\beta_j) ( α j , β j ) are exactly the class-conditional error rates of the Discrimination Framework.
Proof For binary Y ^ i , j ∈ { 0 , 1 } \hat{Y}_{i,j}\in\{0,1\} Y ^ i , j ∈ { 0 , 1 } , ( 1 − Y ^ i , j ) 2 = 1 − Y ^ i , j (1-\hat{Y}_{i,j})^2=1-\hat{Y}_{i,j} ( 1 − Y ^ i , j ) 2 = 1 − Y ^ i , j , so α j = E [ 1 − Y ^ i , j ∣ Y i , j = 1 ] = 1 − E [ Y ^ i , j ∣ Y i , j = 1 ] = 1 − recall j \alpha_j=\mathbb{E}[1-\hat{Y}_{i,j}\mid Y_{i,j}=1]=1-\mathbb{E}[\hat{Y}_{i,j}\mid Y_{i,j}=1]=1-\operatorname{recall}_j α j = E [ 1 − Y ^ i , j ∣ Y i , j = 1 ] = 1 − E [ Y ^ i , j ∣ Y i , j = 1 ] = 1 − recall j . Similarly, Y ^ i , j 2 = Y ^ i , j \hat{Y}_{i,j}^2=\hat{Y}_{i,j} Y ^ i , j 2 = Y ^ i , j for binary values, so β j = E [ Y ^ i , j ∣ Y i , j = 0 ] = FPR j = 1 − specificity j \beta_j=\mathbb{E}[\hat{Y}_{i,j}\mid Y_{i,j}=0]=\operatorname{FPR}_j=1-\operatorname{specificity}_j β j = E [ Y ^ i , j ∣ Y i , j = 0 ] = FPR j = 1 − specificity j . ■
Given the skew-sensitivity of B S j \mathrm{BS}_j BS j itself (Corollary 50), the following corollary examines, term by term, whether the individual terms of the Murphy decomposition inherit the same prevalence dependence.
Corollary 53 (Skew-sensitivity of Murphy terms). For binary Y i , j ∈ { 0 , 1 } Y_{i,j} \in\{0,1\} Y i , j ∈ { 0 , 1 } , of the three Murphy terms:
(a) U N C j = π j ( 1 − π j ) \mathrm{UNC}_j = \pi_j(1-\pi_j) UNC j = π j ( 1 − π j ) is skew-sensitive : it depends only on π j \pi_j π j and carries no information about the classifier.
(b) R E S j \mathrm{RES}_j RES j is skew-sensitive if and only if the class-conditional score distributions differ, F j + ≠ F j − F_j^{+} \ne F_j^{-} F j + = F j − . If F j + = F j − F_j^{+} = F_j^{-} F j + = F j − , then R E S j = 0 \mathrm{RES}_j = 0 RES j = 0 at every prevalence and is skew-invariant. In the other case, R E S j \mathrm{RES}_j RES j varies with π j \pi_j π j according to the Bayes formula of the proof.
(c) R E L j = E [ ( Y ^ i , j − f j ( Y ^ i , j ) ) 2 ] \mathrm{REL}_j = \mathbb{E}[(\hat{Y}_{i,j} - f_j(\hat{Y}_{i,j}))^2] REL j = E [( Y ^ i , j − f j ( Y ^ i , j ) ) 2 ] is skew-sensitive in general, because f j f_j f j changes with π j \pi_j π j . Perfect calibration (R E L j = 0 \mathrm{REL}_j = 0 REL j = 0 ) is not preserved either: it holds at two different prevalences if and only if every predicted value is attained by one class alone and equals 1 (positives only) or 0 (negatives only). In particular, if some predicted value is attained by both positive and negative examples, then R E L j = 0 \mathrm{REL}_j = 0 REL j = 0 at one prevalence implies R E L j > 0 \mathrm{REL}_j > 0 REL j > 0 at every other.
The class-conditional pair ( α j , β j ) (\alpha_j,\beta_j) ( α j , β j ) is invariant under the per-class skew transformation, whereas the individual Murphy terms are not invariant in general. Degenerate exceptions exist: R E S j ≡ 0 \mathrm{RES}_j \equiv0 RES j ≡ 0 when F j + = F j − F_j^{+} = F_j^{-} F j + = F j − (part (b)), and, when every predicted value is attained by one class alone, R E L j = π j α j + ( 1 − π j ) β j \mathrm{REL}_j = \pi_j\alpha_j + (1-\pi_j)\beta_j REL j = π j α j + ( 1 − π j ) β j , which is invariant if α j = β j \alpha_j = \beta_j α j = β j .
Proof sketch. (a) is immediate. (b) By Bayes’ theorem, R E S j = π j 2 ( 1 − π j ) 2 ∑ p ( a p − b p ) 2 / ( π j a p + ( 1 − π j ) b p ) \mathrm{RES}_j = \pi_j^2(1-\pi_j)^2 \sum_p (a_p - b_p)^2/(\pi_j a_p + (1-\pi_j)b_p) RES j = π j 2 ( 1 − π j ) 2 ∑ p ( a p − b p ) 2 / ( π j a p + ( 1 − π j ) b p ) , where a p a_p a p and b p b_p b p are the probabilities of the predicted value p p p under F j + F_j^{+} F j + and F j − F_j^{-} F j − ; it vanishes identically iff F j + = F j − F_j^{+} = F_j^{-} F j + = F j − , and is otherwise positive yet bounded by π j ( 1 − π j ) \pi_j(1-\pi_j) π j ( 1 − π j ) . (c) For a value attained by both classes, the calibration map is strictly increasing in the replication factor. The full proof, with the concrete instance R E L j = 0 → 1 / 45 \mathrm{REL}_j = 0 \to1/45 REL j = 0 → 1/45 , is in Appendix D.
The Ranking Framework: Positional Operators
The first two frameworks evaluate a classifier’s ability to identify which labels are correct (discrimination) and how accurately its confidence values match empirical frequencies (calibration). Both rest on pointwise cell operators: the cell value D ( Y i , j , Y ^ i , j ) D(Y_{i,j},\hat{Y}_{i,j}) D ( Y i , j , Y ^ i , j ) depends only on the pair at position ( i , j ) (i,j) ( i , j ) . Ranking evaluation breaks this assumption in two orthogonal ways. First: how useful is it to predict label j j j at rank 1 for example i i i vs. rank 5? The answer depends on the full row Y ^ i , ⋅ \hat{Y}_{i,\cdot} Y ^ i , ⋅ , not just on Y ^ i , j \hat{Y}_{i,j} Y ^ i , j , which gives rise to row-dependent positional operators and the nDCG family. Second: how highly does the classifier rank example i i i among all examples for class j j j ? This depends on the full column Y ^ ⋅ , j \hat{Y}_{\cdot,j} Y ^ ⋅ , j , giving rise to column-dependent positional operators and the AUC family. This section develops both families within a unified operator framework, establishes the duality between them, and shows that the two families share the same structural properties (collapse behaviour, skew-invariance, and failure of strict properness), derived from the same matrix machinery.
Discount Cell Operators and the DCG Matrix
Definition 55 (Rank function). For example i i i and a predicted score vector Y ^ i , ⋅ ∈ [ 0 , 1 ] m \hat{Y}_{i,\cdot} \in[0,1]^m Y ^ i , ⋅ ∈ [ 0 , 1 ] m , the rank of label j j j is
rank i ( j ) = ∣ { j ′ : Y ^ i , j ′ > Y ^ i , j } ∣ + 1 , (29) \operatorname{rank}_{i}(j) = \left|\{j' : \hat{Y}_{i,j'} > \hat{Y}_{i,j}\}\right| + 1, \tag*{(29)} rank i ( j ) = { j ′ : Y ^ i , j ′ > Y ^ i , j } + 1 , (29) i.e. the position of label j j j in the descending ordering of Y ^ i , ⋅ \hat{Y}_{i,\cdot} Y ^ i , ⋅ . Under this convention tied scores share the same rank, namely the smallest position among them (min-rank, or “competition”, ranking). Labels predicted most confidently receive rank 1.
Definition 56 (Positional operator and DCG cell matrix). A positional weight function is a non-increasing map w : N → R ≥ 0 w : \mathbb{N} \to\mathbb{R}_{\ge0} w : N → R ≥ 0 with w ( 1 ) > 0 w(1) > 0 w ( 1 ) > 0 . The positional cell operator with weight w w w is
D w ( Y i , j , Y ^ i , ⋅ ) = Y i , j ⋅ w ( rank i ( j ) ) . (30) D_w(Y_{i,j}, \hat{Y}_{i,\cdot}) = Y_{i,j} \cdot w(\operatorname{rank}_{i}(j)). \tag*{(30)} D w ( Y i , j , Y ^ i , ⋅ ) = Y i , j ⋅ w ( rank i ( j )) . (30) The DCG cell matrix D w ∈ R n × m D_w \in\mathbb{R}^{n \times m} D w ∈ R n × m is defined entry-wise by [ D w ] i , j = D w ( Y i , j , Y ^ i , ⋅ ) [D_w]_{i,j} = D_w(Y_{i,j}, \hat{Y}_{i,\cdot}) [ D w ] i , j = D w ( Y i , j , Y ^ i , ⋅ ) .
Two canonical weight functions define the families developed in this section, both fixed a priori. For the row-dependent case (nDCG), w ( r ) = 1 / log 2 ( r + 1 ) w(r) = 1/\log_{2}(r + 1) w ( r ) = 1/ log 2 ( r + 1 ) : labels ranked first receive full credit w ( 1 ) = 1 w(1) = 1 w ( 1 ) = 1 , with credit decaying logarithmically so that w ( 2 ) ≈ 0.63 w(2) \approx0.63 w ( 2 ) ≈ 0.63 and w ( r ) → 0 w(r) \to0 w ( r ) → 0 as r → ∞ r \to\infty r → ∞ . For the column-dependent case, the fixed rank function can instead act as a cost: the AUC construction uses the increasing rank cost v ( r ) = r v(r) = r v ( r ) = r to obtain the mean rank of positive examples, followed by a decreasing affine transformation (§ 6.9 , Proposition 73). In both cases, cells with Y i , j = 0 Y_{i,j} = 0 Y i , j = 0 contribute zero regardless of rank: the positional weight rewards relevant items ranked high and gives a cell nothing for an irrelevant label, so a false alarm (an irrelevant label ranked high) is penalised only indirectly, by pushing relevant labels down the ranking.
The positional operator is row-dependent : changing Y ^ i , j ′ \hat{Y}_{i,j'} Y ^ i , j ′ for any j ′ ≠ j j' \ne j j ′ = j can alter rank i ( j ) \operatorname{rank}_{i}(j) rank i ( j ) and therefore [ D w ] i , j [D_w]_{i,j} [ D w ] i , j . This violates the pointwise structure of Definition 13 and has structural consequences developed in Sections 6.2 and 6.3.
The DCG Family via the Three Aggregation Operators
Applying the three aggregation operators Σ 1 \Sigma_1 Σ 1 , Σ m \Sigma_m Σ m , Σ n \Sigma_n Σ n to D w D_w D w yields three families of measures; they are summarised in Table 2 .
Table 2. DCG family via the three aggregation operators. “Novel” indicates measures not previously expressed within the matrix framework. The standard nDCG of the IR literature (Järvelin and Kekäläinen, 2002) corresponds to the exemplar axis.
Definition 57 (Raw DCG family). For the DCG cell matrix D w \mathbf{D}_w D w with weight w w w :
D C G m i c r o = 1 n m Σ 1 ( D w ) = 1 n m ∑ i = 1 n ∑ j = 1 m Y i , j w ( rank i ( j ) ) , (31) \mathrm{DCG}^{\mathrm{micro}} = \frac{1}{nm}\Sigma_{1}(\mathbf{D}_w) = \frac{1}{nm}\sum_{i=1}^{n}\sum_{j=1}^{m}Y_{i,j}\,w(\operatorname{rank}_{i}(j)), \tag*{(31)} DCG micro = nm 1 Σ 1 ( D w ) = nm 1 i = 1 ∑ n j = 1 ∑ m Y i , j w ( rank i ( j )) , (31) D C G j c l a s s = 1 n [ Σ m ( D w ) ] j = 1 n ∑ i = 1 n Y i , j w ( rank i ( j ) ) , (32) \mathrm{DCG}^{\mathrm{class}}_{j} = \frac{1}{n}\left[\Sigma_{m}(\mathbf{D}_w)\right]_{j} = \frac{1}{n}\sum_{i=1}^{n}Y_{i,j}\,w(\operatorname{rank}_{i}(j)), \tag*{(32)} DCG j class = n 1 [ Σ m ( D w ) ] j = n 1 i = 1 ∑ n Y i , j w ( rank i ( j )) , (32) D C G i e x = 1 m [ Σ n ( D w ) ] i = 1 m ∑ j = 1 m Y i , j w ( rank i ( j ) ) . (33) \mathrm{DCG}^{\mathrm{ex}}_{i} = \frac{1}{m}\left[\Sigma_{n}(\mathbf{D}_w)\right]_{i} = \frac{1}{m}\sum_{j=1}^{m}Y_{i,j}\,w(\operatorname{rank}_{i}(j)). \tag*{(33)} DCG i ex = m 1 [ Σ n ( D w ) ] i = m 1 j = 1 ∑ m Y i , j w ( rank i ( j )) . (33)
D C G j c l a s s \mathrm{DCG}^{\mathrm{class}}_{j} DCG j class is a per-label ranking measure novel to this framework: it asks how highly the classifier ranks label j j j across all examples in which j j j is relevant. D C G i e x \mathrm{DCG}^{\mathrm{ex}}_{i} DCG i ex is the standard per-example DCG of the IR literature scaled by 1 / m 1/m 1/ m (Järvelin and Kekäläinen, 2002), here derived as the row-wise aggregation of the cell matrix.
Definition 58 (Ideal DCG and normalisation). The ideal DCG for example i i i is achieved when all k i = Σ 1 ( Y i , ⋅ ) k_i = \Sigma_{1}(Y_{i,\cdot}) k i = Σ 1 ( Y i , ⋅ ) relevant labels are ranked first:
I D C G i = ∑ r = 1 k i w ( r ) , (34) \mathrm{IDCG}_{i} = \sum_{r=1}^{k_i} w(r), \tag*{(34)} IDCG i = r = 1 ∑ k i w ( r ) , (34) and for class j j j the ideal is achieved when every positive example ranks label j j j at position 1:
I D C G j c l a s s = π j ⋅ w ( 1 ) , (35) \mathrm{IDCG}^{\mathrm{class}}_{j} = \pi_j \cdot w(1), \tag*{(35)} IDCG j class = π j ⋅ w ( 1 ) , (35) where π j = n j + / n \pi_j = n_j^{+}/n π j = n j + / n with n j + = ∑ i Y i , j n_j^{+} = \sum_i Y_{i,j} n j + = ∑ i Y i , j . The ratios below are defined whenever their denominators are positive: k i ≥ 1 k_i \ge1 k i ≥ 1 for n D C G i \mathrm{nDCG}_i nDCG i , n j + ≥ 1 n_j^{+} \ge1 n j + ≥ 1 (equivalently π j > 0 \pi_j > 0 π j > 0 ) for n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class , and ∑ i I D C G i > 0 \sum_i \mathrm{IDCG}_i > 0 ∑ i IDCG i > 0 for m i c r o \mathrm{micro} micro -n D C G \mathrm{nDCG} nDCG . Zero-ideal instances or classes require an explicit exclusion or zero-score convention rather than an unqualified division. The normalised measures are:
n D C G i = D C G i e x I D C G i / m = ∑ j Y i , j w ( rank i ( j ) ) I D C G i , (36) \mathrm{nDCG}_{i} = \frac{\mathrm{DCG}^{\mathrm{ex}}_{i}}{\mathrm{IDCG}_{i}/m} = \frac{\sum_j Y_{i,j}w(\operatorname{rank}_{i}(j))}{\mathrm{IDCG}_{i}}, \tag*{(36)} nDCG i = IDCG i / m DCG i ex = IDCG i ∑ j Y i , j w ( rank i ( j )) , (36) n D C G j c l a s s = D C G j c l a s s I D C G j c l a s s = 1 π j w ( 1 ) ⋅ 1 n ∑ i Y i , j w ( rank i ( j ) ) . (37) \mathrm{nDCG}^{\mathrm{class}}_{j} = \frac{\mathrm{DCG}^{\mathrm{class}}_{j}}{\mathrm{IDCG}^{\mathrm{class}}_{j}} = \frac{1}{\pi_j w(1)}\cdot\frac{1}{n}\sum_i Y_{i,j}w(\operatorname{rank}_{i}(j)). \tag*{(37)} nDCG j class = IDCG j class DCG j class = π j w ( 1 ) 1 ⋅ n 1 i ∑ Y i , j w ( rank i ( j )) . (37) The micro-nDCG normalises the global DCG by the global ideal:
n D C G m i c r o = D C G m i c r o I D C G ‾ = ∑ i , j Y i , j w ( rank i ( j ) ) ∑ i I D C G i , (38) \mathrm{nDCG}^{\mathrm{micro}} = \frac{\mathrm{DCG}^{\mathrm{micro}}}{\overline{\mathrm{IDCG}}} = \frac{\sum_{i,j}Y_{i,j}w(\operatorname{rank}_{i}(j))}{\sum_i\mathrm{IDCG}_i}, \tag*{(38)} nDCG micro = IDCG DCG micro = ∑ i IDCG i ∑ i , j Y i , j w ( rank i ( j )) , (38) where I D C G ‾ = 1 n m ∑ i I D C G i \overline{\mathrm{IDCG}} = \frac{1}{nm}\sum_i\mathrm{IDCG}_i IDCG = nm 1 ∑ i IDCG i is the average per-example ideal DCG.
on each positive cell. The realised DCG can then exceed I D C G i \mathrm{IDCG}_i IDCG i , which is computed for a strict ordering in which the k i k_i k i positives occupy positions 1 , … , k i 1,\ldots,k_i 1 , … , k i , and n D C G i \mathrm{nDCG}_i nDCG i can exceed 1. Throughout we therefore assume that the predicted scores within each row (and, for the AUC family of § 6.9 , within each column) are distinct (for example, almost surely under a joint score distribution assigning probability zero to exact ties), so that n D C G ∈ [ 0 , 1 ] \mathrm{nDCG} \in[0,1] nDCG ∈ [ 0 , 1 ] for all aggregations. When exact ties do occur in practice, they can be resolved by any fixed tie-breaking permutation applied before ranks are computed, which restores the standard range.
Collapse for Raw DCG, Non-Collapse for nDCG
The main structural difference between the Ranking Framework and the Discrimination and Calibration Frameworks lies in the behaviour of micro- and macro-averaging.
Theorem 60 (Raw DCG collapses; nDCG does not). Assume that the ideal denominators appearing below are positive. Put D i j = Y i j w ( rank i ( j ) ) D_{ij}=Y_{ij}w(\operatorname{rank}_i(j)) D ij = Y ij w ( rank i ( j )) , S i = ∑ j D i j S_i=\sum_j D_{ij} S i = ∑ j D ij , b i = I D C G i b_i=\mathrm{IDCG}_i b i = IDCG i , r i = S i / b i = n D C G i r_i=S_i/b_i=\mathrm{nDCG}_i r i = S i / b i = nDCG i , T j = ∑ i D i j T_j=\sum_i D_{ij} T j = ∑ i D ij , N j = n j + w ( 1 ) N_j=n_j^{+}w(1) N j = n j + w ( 1 ) and B = ∑ i b i B=\sum_i b_i B = ∑ i b i .
(a) The three raw DCG aggregations are equal as scalars:
D C G m i c r o = 1 m ∑ j D C G j c l a s s = 1 n ∑ i D C G i e x . (39) \mathrm{DCG}^{\mathrm{micro}}=\frac{1}{m}\sum_j\mathrm{DCG}^{\mathrm{class}}_j=\frac{1}{n}\sum_i\mathrm{DCG}^{\mathrm{ex}}_i. \tag*{(39)} DCG micro = m 1 j ∑ DCG j class = n 1 i ∑ DCG i ex . (39) (b) The micro-nDCG, a global ratio, is a b i b_i b i -weighted mean of the r i r_i r i , whereas the exemplar mean of nDCG is unweighted. They coincide if and only if
Cov ^ ( I D C G i , n D C G i ) = 0. (40) \widehat{\operatorname{Cov}}(\mathrm{IDCG}_i,\mathrm{nDCG}_i)=0. \tag*{(40)} Cov ( IDCG i , nDCG i ) = 0. (40) In particular, equality holds when all k i k_i k i are equal.
(c) The micro-nDCG and the class mean of nDCG coincide if and only if
∑ j T j ( 1 B − 1 m N j ) = 0. (41) \sum_j T_j\left(\frac{1}{B}-\frac{1}{mN_j}\right)=0. \tag*{(41)} j ∑ T j ( B 1 − m N j 1 ) = 0. (41) For arbitrary class-cell sums T j T_j T j , this is an identity if and only if m N j = B mN_j=B m N j = B for every j j j . These coefficients cannot all vanish when some example has at least two relevant labels and w ( 2 ) < w ( 1 ) w(2)<w(1) w ( 2 ) < w ( 1 ) .
(d) The class mean and the exemplar mean of n D C G nDCG n D C G coincide if and only if
∑ i , j D i j ( 1 m N j − 1 n b i ) = 0. (42) \sum_{i,j} D_{ij}\left(\frac{1}{mN_j}-\frac{1}{nb_i}\right)=0. \tag*{(42)} i , j ∑ D ij ( m N j 1 − n b i 1 ) = 0. (42)
For arbitrary cell values supported on the relevant positions (D i j = 0 D_{ij}=0 D ij = 0 when Y i j = 0 Y_{ij}=0 Y ij = 0 ), this equality holds identically if and only if m N j = n b i mN_j=nb_i m N j = n b i at every relevant cell. These coefficient equalities cannot all hold when some example has at least two relevant labels and w ( 2 ) < w ( 1 ) w(2)<w(1) w ( 2 ) < w ( 1 ) , assuming every class and every example has a relevant entry.
Consequently, the three normalised summaries
n D C G m i c r o , 1 m ∑ j n D C G j c l a s s , 1 n ∑ i n D C G i (43) nDCG^{\mathrm{micro}},\qquad\frac{1}{m}\sum_j nDCG_j^{\mathrm{class}},\qquad\frac{1}{n}\sum_i nDCG_i \tag*{(43)} n D C G micro , m 1 j ∑ n D C G j class , n 1 i ∑ n D C G i (43)
need not coincide; their equality on a particular data set is governed by (b)–(d), not ruled out universally.
Proof sketch. (a) is finite-sum rearrangement, as for the Brier score (Theorem 20 ). Parts (b)–(d) compare weighted and unweighted means of the same cells: (b) is the covariance identity for weighted means, and (c)–(d) reduce equality for all freely chosen cells to coefficient conditions that contradict ∑ i IDCG i < w ( 1 ) ∑ i k i \sum_i \operatorname{IDCG}_i < w(1)\sum_i k_i ∑ i IDCG i < w ( 1 ) ∑ i k i when some k i ≥ 2 k_i \ge2 k i ≥ 2 and w ( 2 ) < w ( 1 ) w(2)<w(1) w ( 2 ) < w ( 1 ) . The full proof is in Appendix E.
Per-Class nDCG as a Novel Measure
The class-level measure n D C G j c l a s s nDCG_j^{\mathrm{class}} n D C G j class (Definition 58 , equation (37) ) does not appear in the standard IR literature, which focuses on the exemplar axis. It has the following interpretation.
Proposition 62 (Per-class nDCG as conditional mean). Let I I I be uniformly distributed over the n n n observed examples, and assume n j + > 0 n_j^+>0 n j + > 0 . The expectation below is the resulting empirical conditional average.
n D C G j c l a s s = 1 w ( 1 ) E [ w ( rank i ( j ) ) ∣ Y i , j = 1 ] . (44) nDCG_j^{\mathrm{class}}=\frac{1}{w(1)}\mathbb{E}\left[w(\operatorname{rank}_i(j))\mid Y_{i,j}=1\right]. \tag*{(44)} n D C G j class = w ( 1 ) 1 E [ w ( rank i ( j )) ∣ Y i , j = 1 ] . (44)
Proof From (36):
nDCG j c l a s s = 1 π j w ( 1 ) ⋅ 1 n ∑ i Y i j w ( rank i ( j ) ) = 1 w ( 1 ) ⋅ 1 n j + ∑ i : Y i j = 1 w ( rank i ( j ) ) = 1 w ( 1 ) E [ w ( rank i ( j ) ) ∣ Y i j = 1 ] . \operatorname{nDCG}^{\mathrm{class}}_j=\frac{1}{\pi_j w(1)}\cdot\frac{1}{n}\sum_i Y_{ij}w(\operatorname{rank}_i(j))=\frac{1}{w(1)}\cdot\frac{1}{n_j^+}\sum_{i:Y_{ij}=1}w(\operatorname{rank}_i(j))=\frac{1}{w(1)}\mathbb{E}[w(\operatorname{rank}_i(j))\mid Y_{ij}=1]. nDCG j class = π j w ( 1 ) 1 ⋅ n 1 i ∑ Y ij w ( rank i ( j )) = w ( 1 ) 1 ⋅ n j + 1 i : Y ij = 1 ∑ w ( rank i ( j )) = w ( 1 ) 1 E [ w ( rank i ( j )) ∣ Y ij = 1 ] .
The measure asks: on average, how highly does the classifier rank label j j j for examples where j j j is truly relevant? A classifier that always ranks label j j j first among its relevant examples achieves nDCG j c l a s s = w ( 1 ) / w ( 1 ) = 1 \operatorname{nDCG}^{\mathrm{class}}_j=w(1)/w(1)=1 nDCG j class = w ( 1 ) / w ( 1 ) = 1 , but this cannot hold for all labels at once when examples have several relevant labels, since only one label per example is first under a strict (tie-broken) ordering: with the strict-order oracle used in the experiments (relevant labels of an example ordered uniformly at random, averaged over 20 draws), the illustrative macro values are 0.81 on Emotions and 0.61 on Yeast (§ 8 ). The values depend on the specified tie-breaking convention. In contrast, if all truly relevant labels receive the same highest score and irrelevant labels are scored lower, the min-rank convention assigns every relevant label rank 1, giving per-class nDCG 1 for each represented class; this is not the strict-order oracle used for the reported values. A uniformly random ordering of the m m m labels gives 1 m ∑ r = 1 m w ( r ) / w ( 1 ) \frac{1}{m}\sum_{r=1}^{m}w(r)/w(1) m 1 ∑ r = 1 m w ( r ) / w ( 1 ) , which for the nDCG weight is 0.55 for m = 6 m=6 m = 6 and 0.40 for m = 14 m=14 m = 14 and decreases slowly with m m m (0.12 for m = 1000 m=1000 m = 1000 ). The measure is analogous to recall (E [ Y ^ i j ∣ Y i j = 1 ] \mathbb{E}[\hat{Y}_{ij}\mid Y_{ij}=1] E [ Y ^ ij ∣ Y ij = 1 ] for binary predictions) in the Discrimination Framework.
Prati (2027) establish (in their proposition on ordinal AUC as a column-wise aggregation) that the ordinal AUC across multiple classes can be expressed as a column-wise aggregation of per-class binary AUCs. An analogous structure holds here: DCG m i c r o = 1 m ∑ j DCG j c l a s s \operatorname{DCG}^{\mathrm{micro}}=\frac{1}{m}\sum_j\operatorname{DCG}^{\mathrm{class}}_j DCG micro = m 1 ∑ j DCG j class (Theorem 60 (a)) expresses the global DCG as the unweighted mean of per-class DCG values, another column-wise decomposition. This is not an assertion that micro-AUC equals macro-AUC: the latter are generally different because micro-AUC weights the per-class values by their positive-example counts.
Connection to the Discrimination Framework for Hard Predictions
For binary hard predictions Y ^ i j ∈ { 0 , 1 } \hat{Y}_{ij}\in\{0,1\} Y ^ ij ∈ { 0 , 1 } , let k i + = ∑ j Y ^ i , j k_i^+=\sum_j\hat{Y}_{i,j} k i + = ∑ j Y ^ i , j be the number of predicted positives. Under the min-rank convention of Definition 55 , ties are not broken:
rank i ( j ) = ∣ { j ′ : Y ^ i , j ′ > Y ^ i , j } ∣ + 1 = { 1 if Y ^ i , j = 1 , k i + + 1 if Y ^ i , j = 0. (45) \operatorname{rank}_i(j)=\left|\{j':\hat{Y}_{i,j'}>\hat{Y}_{i,j}\}\right|+1=
\begin{cases}
1 & \text{if }\hat{Y}_{i,j}=1,\\
k_i^++1 & \text{if }\hat{Y}_{i,j}=0.
\end{cases} \tag*{(45)} rank i ( j ) = { j ′ : Y ^ i , j ′ > Y ^ i , j } + 1 = { 1 k i + + 1 if Y ^ i , j = 1 , if Y ^ i , j = 0. (45)
A different, explicitly tie-broken ordering can instead put the predicted positives at positions r i , j ∈ { 1 , … , k i + } r_{i,j}\in\{1,\ldots,k_i^+\} r i , j ∈ { 1 , … , k i + } and the predicted negatives at positions k i + + s i , j k_i^++s_{i,j} k i + + s i , j , where s i , j ∈ { 1 , … , m − k i + } s_{i,j}\in\{1,\ldots,m-k_i^+\} s i , j ∈ { 1 , … , m − k i + } . These are tie-broken positions, not the min-ranks in Equation (45) ; all statements below specify which convention they use.
Proposition 63 (Hard-prediction limit). For binary hard predictions Y ^ i j ∈ { 0 , 1 } \hat{Y}_{ij}\in\{0,1\} Y ^ ij ∈ { 0 , 1 } with exactly one predicted label per example (k i + = 1 k_i^+=1 k i + = 1 ), let j ∗ = arg max j Y ^ i , j j^*=\operatorname*{arg\,max}_j\hat{Y}_{i,j} j ∗ = arg max j Y ^ i , j and let k i = ∑ j Y i , j k_i=\sum_jY_{i,j} k i = ∑ j Y i , j be the number of relevant labels of example i i i . Under the min-rank convention of Definition 55 , rank i ( j ∗ ) = 1 \operatorname{rank}_i(j^*)=1 rank i ( j ∗ ) = 1 and every other label has rank 2, so
DCG i e x = Y i , j ∗ w ( 1 ) m + w ( 2 ) m ( k i − Y i , j ∗ ) . (46) \operatorname{DCG}^{\mathrm{ex}}_i=\frac{Y_{i,j^*}w(1)}{m}+\frac{w(2)}{m}\left(k_i-Y_{i,j^*}\right). \tag*{(46)} DCG i ex = m Y i , j ∗ w ( 1 ) + m w ( 2 ) ( k i − Y i , j ∗ ) . (46)
The correction term R i R_i R i is the credit earned by the relevant labels other than j ∗ j^* j ∗ . In particular, a wrong top prediction does not earn zero: it earns w ( 2 ) k i / m w(2)k_i/m w ( 2 ) k i / m , which vanishes only if w ( 2 ) = 0 w(2)=0 w ( 2 ) = 0 . (Under another tie-breaking among the predicted negatives their ranks lie in { 2 , … , m } \{2,\ldots,m\} { 2 , … , m } , and R i R_i R i is positive for a strictly positive weight if and only if another relevant label exists, k i > Y i , j ∗ k_i>Y_{i,j^*} k i > Y i , j ∗ .) Consequently:
(a) In the multiclass case (one-hot Y \mathbf{Y} Y ), k i = 1 k_i=1 k i = 1 , and the one-hot constraint forces Y i , j = 0 Y_{i,j}=0 Y i , j = 0 for every j j j other than the true class c i c_i c i , which need not be j ∗ j^* j ∗ . If the prediction is correct, R i = 0 R_i=0 R i = 0 and D C G i e x = w ( 1 ) / m \mathrm{DCG}^{\mathrm{ex}}_i=w(1)/m DCG i ex = w ( 1 ) / m ; if it is wrong, Y i , j ∗ = 0 Y_{i,j^*}=0 Y i , j ∗ = 0 and D C G i e x = w ( 2 ) / m \mathrm{DCG}^{\mathrm{ex}}_i=w(2)/m DCG i ex = w ( 2 ) / m . The exemplar average is therefore
1 n ∑ i D C G i e x = Precision@1 ⋅ w ( 1 ) + ( 1 − Precision@1 ) ⋅ w ( 2 ) m , \frac{1}{n}\sum_i \mathrm{DCG}^{\mathrm{ex}}_i
=
\frac{\operatorname{Precision@1}\cdot w(1)+(1-\operatorname{Precision@1})\cdot w(2)}{m}, n 1 i ∑ DCG i ex = m Precision@1 ⋅ w ( 1 ) + ( 1 − Precision@1 ) ⋅ w ( 2 ) ,
an affine function of Precision@1 \operatorname{Precision@1} Precision@1 that reduces to Precision@1 ⋅ w ( 1 ) / m \operatorname{Precision@1}\cdot w(1)/m Precision@1 ⋅ w ( 1 ) / m if and only if w ( 2 ) = 0 w(2)=0 w ( 2 ) = 0 (or Precision@1 = 1 \operatorname{Precision@1}=1 Precision@1 = 1 ). For the canonical weight w ( r ) = 1 / log 2 ( r + 1 ) w(r)=1/\log_2(r+1) w ( r ) = 1/ log 2 ( r + 1 ) , w ( 2 ) ≈ 0.63 w ( 1 ) w(2)\approx0.63w(1) w ( 2 ) ≈ 0.63 w ( 1 ) , so wrong top predictions retain a large share of the credit.
(b) In the general multilabel case, since R i ≥ 0 R_i\geq0 R i ≥ 0 , D C G i e x ≥ Y i , j ∗ w ( 1 ) / m \mathrm{DCG}^{\mathrm{ex}}_i\geq Y_{i,j^*}w(1)/m DCG i ex ≥ Y i , j ∗ w ( 1 ) / m , hence
1 n ∑ i D C G i e x ≥ Precision@1 ⋅ w ( 1 ) / m , \frac{1}{n}\sum_i \mathrm{DCG}^{\mathrm{ex}}_i\geq\operatorname{Precision@1}\cdot w(1)/m, n 1 i ∑ DCG i ex ≥ Precision@1 ⋅ w ( 1 ) / m ,
with equality iff R i = 0 R_i=0 R i = 0 for every example, i.e. (for w ( 2 ) > 0 w(2)>0 w ( 2 ) > 0 and k i ≥ 1 k_i\geq1 k i ≥ 1 ) iff every example’s top-predicted label is its only relevant one. Precision@1 \operatorname{Precision@1} Precision@1 is therefore a lower bound on exemplar DCG, not an exact reduction.
Proof For k i + = 1 k_i^+=1 k i + = 1 the single predicted label j ∗ j^* j ∗ has score 1 and every other label score 0, so under the min-rank convention rank i ( j ∗ ) = 1 \operatorname{rank}_i(j^*)=1 rank i ( j ∗ ) = 1 and rank i ( j ) = 2 \operatorname{rank}_i(j)=2 rank i ( j ) = 2 for j ≠ j ∗ j\neq j^* j = j ∗ . Splitting the sum defining D C G i e x \mathrm{DCG}^{\mathrm{ex}}_i DCG i ex (Equation (33) ) into the j = j ∗ j=j^* j = j ∗ term and the remaining terms gives (45), with R i = w ( 2 ) m ∑ j ≠ j ∗ Y i , j = w ( 2 ) m ( k i − Y i , j ∗ ) ≥ 0 R_i=\frac{w(2)}{m}\sum_{j\neq j^*}Y_{i,j}=\frac{w(2)}{m}(k_i-Y_{i,j^*})\geq0 R i = m w ( 2 ) ∑ j = j ∗ Y i , j = m w ( 2 ) ( k i − Y i , j ∗ ) ≥ 0 because w ≥ 0 w\geq0 w ≥ 0 . Part (a): for one-hot Y Y Y with true class c i c_i c i we have k i = 1 k_i=1 k i = 1 and Y i , j ∗ = 1 [ c i = j ∗ ] Y_{i,j^*}=\mathbf{1}[c_i=j^*] Y i , j ∗ = 1 [ c i = j ∗ ] , so D C G i e x = ( w ( 1 ) 1 [ c i = j ∗ ] + w ( 2 ) 1 [ c i ≠ j ∗ ] ) / m \mathrm{DCG}^{\mathrm{ex}}_i=(w(1)\mathbf{1}[c_i=j^*]+w(2)\mathbf{1}[c_i\neq j^*])/m DCG i ex = ( w ( 1 ) 1 [ c i = j ∗ ] + w ( 2 ) 1 [ c i = j ∗ ]) / m ; averaging over i i i , with Precision@1 = 1 n ∑ i 1 [ c i = j ∗ ] \operatorname{Precision@1}=\frac{1}{n}\sum_i\mathbf{1}[c_i=j^*] Precision@1 = n 1 ∑ i 1 [ c i = j ∗ ] , gives the stated formula, whose difference from Precision@1 ⋅ w ( 1 ) / m \operatorname{Precision@1}\cdot w(1)/m Precision@1 ⋅ w ( 1 ) / m is ( 1 − Precision@1 ) w ( 2 ) / m (1-\operatorname{Precision@1})w(2)/m ( 1 − Precision@1 ) w ( 2 ) / m . Part (b) follows from R i ≥ 0 R_i\geq0 R i ≥ 0 by averaging over i i i .
For the canonical nDCG weight w ( r ) = 1 / log 2 ( r + 1 ) w(r) = 1/\log_{2}(r+1) w ( r ) = 1/ log 2 ( r + 1 ) , which is strictly positive at every rank, R i > 0 R_i > 0 R i > 0 whenever a relevant label falls outside the top-k k k predictions, so only the inequality of part (b) survives in general, in the multiclass case as well (Proposition 63(a): under min-rank a wrong top prediction still earns w ( 2 ) / m w(2)/m w ( 2 ) / m ; under the strict tie-break used here it earns w ( 1 + s ) / m w(1+s)/m w ( 1 + s ) / m , where s ∈ { 1 , … , m − 1 } s \in\{1,\ldots,m-1\} s ∈ { 1 , … , m − 1 } is the position of the true label among the predicted negatives). If w ( r ) = 0 w(r) = 0 w ( r ) = 0 for every r > k r > k r > k , the residual term R i R_i R i vanishes, but the remaining top-k k k sum is generally weighted:
D C G i e x = 1 m ∑ l = 1 k Y i , j l w ( l ) . \mathrm{DCG}^{\mathrm{ex}}_i = \frac{1}{m}\sum_{l=1}^{k}Y_{i,j_l}w(l). DCG i ex = m 1 l = 1 ∑ k Y i , j l w ( l ) .
The unweighted standard Precision@k identity requires the indicator discount w ( r ) = 1 [ r ≤ k ] w(r) = \mathbf{1}[r \le k] w ( r ) = 1 [ r ≤ k ] (under the stated strict top-k k k tie-breaking convention), for which
D C G i e x = 1 m ∑ l = 1 k Y i , j l = k m P r e c i s i o n i @ k . \mathrm{DCG}^{\mathrm{ex}}_i = \frac{1}{m}\sum_{l=1}^{k}Y_{i,j_l} = \frac{k}{m}\mathrm{Precision}_i@k. DCG i ex = m 1 l = 1 ∑ k Y i , j l = m k Precision i @ k .
If instead w ( r ) = c w(r) = c w ( r ) = c for 1 ≤ r ≤ k 1 \le r \le k 1 ≤ r ≤ k and w ( r ) = 0 w(r) = 0 w ( r ) = 0 for r > k r > k r > k , the right-hand side is ( c k / m ) P r e c i s i o n i @ k (ck/m)\mathrm{Precision}_i@k ( c k / m ) Precision i @ k . For instance, at m = 3 m = 3 m = 3 , k = 2 k = 2 k = 2 , w = ( 1 , 1 / 2 , 0 ) w = (1,1/2,0) w = ( 1 , 1/2 , 0 ) , with only the second-ranked label relevant, the weighted DCG is 1 / 6 1/6 1/6 whereas ( k / m ) P r e c i s i o n i @ k = 1 / 3 (k/m)\mathrm{Precision}_i@k = 1/3 ( k / m ) Precision i @ k = 1/3 . Thus compact support removes the tail, while constant top-k k k weights are needed for the usual unweighted identity; the multiclass assumption alone does not imply it.
More generally, for soft predictions, a rank-truncated positional operator with w k ( r ) = w ( r ) 1 [ r ≤ k ] w_k(r) = w(r)\mathbf{1}[r \le k] w k ( r ) = w ( r ) 1 [ r ≤ k ] yields a rank-weighted Precision@k quantity; the untruncated DCG with a strictly positive tail does not equal a top-k k k metric. This connects the Ranking Framework to the standard precision–recall-at-k family used in IR evaluation (Manning et al., 2008).
Positional Operators Are Not Strictly Proper
Proposition 65 (Positional operators are not strictly proper). For m ≥ 2 m \ge2 m ≥ 2 , no positional operator D w D_w D w with non-increasing w w w induces a strictly proper scoring rule. Among reports with pairwise distinct entries (the standing assumption of Remark 59), truthful reporting, with distinct p i , j p_{i,j} p i , j , attains the maximum expected score (the rule is proper in the weak sense), but it is never the unique maximiser. Without the distinctness restriction, weak properness need not hold: under the min-rank convention a constant report puts every label at rank 1 and earns w ( 1 ) ∑ j p i , j w(1)\sum_j p_{i,j} w ( 1 ) ∑ j p i , j . If w ( 2 ) < w ( 1 ) w(2) < w(1) w ( 2 ) < w ( 1 ) and a lower-ranked label has positive marginal probability, this may strictly exceed the truthful score; a constant weight is an exception.
Proof For the positional operator, the total expected score for example i i i under report Y ^ i , ⋅ \widehat{Y}_{i,\cdot} Y i , ⋅ is
E [ ∑ j D w ( Y i , j , Y ^ i , ⋅ ) ] = ∑ j p i , j ⋅ w ( rank i ( j ) ) , (47) \mathbb{E}\left[\sum_j D_w(Y_{i,j},\widehat{Y}_{i,\cdot})\right] = \sum_j p_{i,j}\cdot w(\operatorname{rank}_i(j)), \tag*{(47)} E [ j ∑ D w ( Y i , j , Y i , ⋅ ) ] = j ∑ p i , j ⋅ w ( rank i ( j )) , (47)
where p i , j = P ( Y i , j = 1 ) p_{i,j} = P(Y_{i,j} = 1) p i , j = P ( Y i , j = 1 ) and the ranks are determined by the reported vector. Since w w w is non-increasing, the rearrangement inequality shows that (46) is maximised by the rankings that sort labels in decreasing order of p i , j p_{i,j} p i , j (not necessarily uniquely so if w w w itself has ties), among reports whose entries are pairwise distinct, so that the rank is a permutation. Reporting Y ^ i , ⋅ = p i \hat{\mathbf{Y}}_{i,\cdot}=\mathbf{p}_i Y ^ i , ⋅ = p i induces such an ordering, so the truthful report attains the maximum among them and the rule is proper there. If tied entries are allowed and ranked by the min-rank rule, a constant report assigns rank 1 to every label and earns w ( 1 ) ∑ j p i , j w(1)\sum_j p_{i,j} w ( 1 ) ∑ j p i , j . For instance, with m = 2 m=2 m = 2 , p i = ( 0.7 , 0.3 ) \mathbf{p}_i=(0.7,0.3) p i = ( 0.7 , 0.3 ) and w ( r ) = 1 / log 2 ( r + 1 ) w(r)=1/\log_2(r+1) w ( r ) = 1/ log 2 ( r + 1 ) , the tied report earns 1, whereas the truthful strict ordering earns 0.7 + 0.3 / log 2 3 ≈ 0.889 0.7+0.3/\log_2 3\approx0.889 0.7 + 0.3/ log 2 3 ≈ 0.889 ; hence weak properness fails if such tied reports are admitted.
Strict properness fails: the score depends on the report only through the induced ordering, so any report that preserves the ordering of p i \mathbf{p}_i p i (for instance, any strictly increasing transformation of the probabilities, such as x ↦ x / 2 x\mapsto x/2 x ↦ x /2 or x ↦ x 10 x\mapsto x^{10} x ↦ x 10 ) attains the same expected score. For m ≥ 2 m\ge2 m ≥ 2 these maximisers form an uncountable set, so the truth is never the unique maximiser, and the score cannot distinguish calibrated forecasts from arbitrary order-preserving distortions of them. ■
Remark 66 (Normalised measures are not proper in general). Proposition 65 concerns the raw positional operator. The normalised n D C G i = D C G i / I D C G i \mathrm{nDCG}_i=\mathrm{DCG}_i/\mathrm{IDCG}_i nDCG i = DCG i / IDCG i divides by a quantity depending on the random number k i k_i k i of relevant labels; consequently, truthful probability reporting need not maximise expected nDCG. To avoid any ambiguity from tied true probabilities, consider m = 3 m=3 m = 3 and the label outcomes ( 1 , 0 , 0 ) (1,0,0) ( 1 , 0 , 0 ) , ( 0 , 1 , 1 ) (0,1,1) ( 0 , 1 , 1 ) and ( 0 , 1 , 0 ) (0,1,0) ( 0 , 1 , 0 ) with respective probabilities 0.45 0.45 0.45 , 0.54 0.54 0.54 and 0.01 0.01 0.01 . The true marginal vector is p i = ( 0.45 , 0.55 , 0.54 ) \mathbf{p}_i=(0.45,0.55,0.54) p i = ( 0.45 , 0.55 , 0.54 ) , whose entries are pairwise distinct and whose truthful ranking is 2 ≻ 3 ≻ 1 2\succ3\succ1 2 ≻ 3 ≻ 1 . For w ( r ) = 1 / log 2 ( r + 1 ) w(r)=1/\log_2(r+1) w ( r ) = 1/ log 2 ( r + 1 ) , the truthful report has expected nDCG 0.45 w ( 3 ) + 0.54 + 0.01 = 0.775 0.45\,w(3)+0.54+0.01=0.775 0.45 w ( 3 ) + 0.54 + 0.01 = 0.775 , whereas a report inducing 1 ≻ 2 ≻ 3 1\succ2\succ3 1 ≻ 2 ≻ 3 has expectation
0.45 + 0.54 w ( 2 ) + w ( 3 ) w ( 1 ) + w ( 2 ) + 0.01 w ( 2 ) ≈ 0.830760 > 0.775. 0.45+0.54\frac{w(2)+w(3)}{w(1)+w(2)}+0.01\,w(2)\approx0.830760>0.775. 0.45 + 0.54 w ( 1 ) + w ( 2 ) w ( 2 ) + w ( 3 ) + 0.01 w ( 2 ) ≈ 0.830760 > 0.775.
Thus the failure of properness holds even when neither the truthful report nor the competing report has tied scores. For A U C j \mathrm{AUC}_j AUC j , Proposition 79 gives properness under independent labels (and for at most three examples under the stated conditions).
Remark 67 (Properness and evaluation purpose). The failure of strict properness is not a defect of positional operators; it reflects the fact that ranking evaluation asks a different question than probabilistic calibration (Cortes and Mohri, 2004). A positional score is entirely insensitive to order-preserving miscalibration: a classifier that reports p i 10 \mathbf{p}_i^{10} p i 10 (elementwise) scores identically to one that reports p i \mathbf{p}_i p i . Conversely, a ranking rule that places every relevant label ahead of every irrelevant one (using distinct ranks) achieves the ideal per-example DCG but may be arbitrarily miscalibrated. The three frameworks therefore target three distinct classifier behaviours, none of which subsumes the others.
Skew-Sensitivity of Ranking Measures
We now characterise the skew-sensitivity of the DCG family. The analysis differs from § 5.7 because the positional operator acts on the label ranking within an example , not on per-class example counts. Skewing class j j j (rescaling n j + n_j^+ n j + vs. n j − n_j^- n j − ) adds or removes examples, but does not alter the rank of label j j j for any existing example. Two skew models are used in this section. The nDCG results (Propositions 68 and 69, Corollary 70) replicate whole examples, which leaves every example’s row rank unchanged. The AUC results of Proposition 77(b)–(c) use the per-class-summary convention of § 5.7 : only the counts of one class change and the summaries of the other classes are held fixed, because literally replicating examples also changes the positives and the composition of the negatives of the other classes, so that their A U C l \mathrm{AUC}_l AUC l would change. A U C j \mathrm{AUC}_j AUC j and n D C G j c l a s s \mathrm{nDCG}_j^{\mathrm{class}} nDCG j class themselves are invariant under literal replication. For exemplar averages in this subsection, we adopt the explicit convention n D C G i = 0 \mathrm{nDCG}_i=0 nDCG i = 0 when k i = 0 k_i=0 k i = 0 , retaining such rows in the sample counts; dropping these rows instead defines a different average and requires recalculating its group counts. Per-class nDCG is reported only for n j + > 0 n_j^+>0 n j + > 0 , while AUC requires both n j + > 0 n_j^+>0 n j + > 0 and n j − > 0 n_j^->0 n j − > 0 . These conventions distinguish undefined normalisations from the implementation’s zero-score convention for empty-label examples.
Proposition 68 (Per-class nDCG is skew-invariant). n D C G j c l a s s \mathrm{nDCG}_j^{\mathrm{class}} nDCG j class , as defined in (37) , is skew-invariant: under independent rescaling of n j + n_j^+ n j + and n j − n_j^- n j − , it remains unchanged.
Proof By Proposition 62,
n D C G j c l a s s = 1 w ( 1 ) E [ w ( rank i ( j ) ) ∣ Y i j = 1 ] . \mathrm{nDCG}_j^{\mathrm{class}}=\frac{1}{w(1)}\mathbb{E}[w(\operatorname{rank}_i(j))\mid Y_{ij}=1]. nDCG j class = w ( 1 ) 1 E [ w ( rank i ( j )) ∣ Y ij = 1 ] .
This is a conditional mean over the population of positive examples for class j j j . Since rank i ( j ) \operatorname{rank}_i(j) rank i ( j ) depends only on the predicted score vector Y ^ i , ⋅ \hat{\mathbf{Y}}_{i,\cdot} Y ^ i , ⋅ of example i i i (not on the class prevalence π j \pi_j π j or the overall sample counts), adding duplicate copies of positive or negative examples (the skew operation) does not change the within-class distribution of rank i ( j ) \operatorname{rank}_i(j) rank i ( j ) , and hence does not change the conditional mean.
Proposition 69 (Exemplar nDCG is skew-sensitive). Fix a class j j j with n j + > 0 n_j^+>0 n j + > 0 and n j − > 0 n_j^->0 n j − > 0 , and let n D C G ‾ + \overline{\mathrm{nDCG}}^+ nDCG + and n D C G ‾ − \overline{\mathrm{nDCG}}^- nDCG − be the mean of n D C G i \mathrm{nDCG}_i nDCG i over the positive examples (Y i j = 1 Y_{ij}=1 Y ij = 1 ) and over the negative examples (Y i j = 0 Y_{ij}=0 Y ij = 0 ) of class j j j , respectively. Replicating the positive examples λ \lambda λ -fold (λ > 0 \lambda>0 λ > 0 ; for noninteger λ \lambda λ , interpreted as uniform weighting) changes the exemplar average 1 n ∑ i n D C G i \frac{1}{n}\sum_i\mathrm{nDCG}_i n 1 ∑ i nDCG i for some λ \lambda λ if and only if n D C G ‾ + ≠ n D C G ‾ − \overline{\mathrm{nDCG}}^+\ne\overline{\mathrm{nDCG}}^- nDCG + = nDCG − ; that is, the exemplar average is skew-sensitive exactly when the two groups differ in mean per-example nDCG.
Proof After replicating the n j + n_j^+ n j + positive examples λ \lambda λ -fold, the exemplar average is
λ n j + n D C G ‾ + + n j − n D C G ‾ − λ n j + + n j − , \frac{\lambda n_j^+\overline{\mathrm{nDCG}}^+ + n_j^-\overline{\mathrm{nDCG}}^-}{\lambda n_j^+ + n_j^-}, λ n j + + n j − λ n j + nDCG + + n j − nDCG − ,
a weighted mean of the two group means whose weight on n D C G ‾ + \overline{\mathrm{nDCG}}^{+} nDCG + is strictly increasing in λ \lambda λ . For two values λ ≠ μ \lambda\ne\mu λ = μ , cross-multiplying, the two cross-products differ by n j + n j − ( λ − μ ) ( n D C G ‾ + − n D C G ‾ − ) n_j^{+}n_j^{-}(\lambda-\mu)(\overline{\mathrm{nDCG}}^{+}-\overline{\mathrm{nDCG}}^{-}) n j + n j − ( λ − μ ) ( nDCG + − nDCG − ) , so the two averages coincide if and only if the group means do. ■
The condition is a knife-edge one. A difference in the number of relevant labels k i k_i k i between the two groups, which may also change I D C G i \mathrm{IDCG}_i IDCG i , is one way in which the group means can differ, but it is neither necessary nor sufficient, and equal k i k_i k i for all examples does not make the exemplar average skew-invariant. For a concrete example with the standard logarithmic discount and three labels, suppose the sole relevant label of a class-j j j positive example is ranked first, giving n D C G i = 1 \mathrm{nDCG}_i=1 nDCG i = 1 , while the sole relevant label of a class-j j j negative example is ranked third, giving n D C G i = w ( 3 ) / w ( 1 ) = 1 / 2 \mathrm{nDCG}_i=w(3)/w(1)=1/2 nDCG i = w ( 3 ) / w ( 1 ) = 1/2 . The exemplar mean changes from 3 / 4 3/4 3/4 to 5 / 6 5/6 5/6 when the positive example is duplicated. By contrast, n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class conditions on the positive examples only and has no such dependence (Proposition 68 ).
Corollary 70 (Skew behaviour of the DCG family). Fix a class j j j and replicate its positive examples λ \lambda λ -fold (λ > 0 \lambda>0 λ > 0 ). Write ( ⋅ ) + (\cdot)^{+} ( ⋅ ) + and ( ⋅ ) − (\cdot)^{-} ( ⋅ ) − for means over the positive and the negative examples of class j j j , and D C G i = ∑ j ′ Y i j ′ w ( rank i ( j ′ ) ) \mathrm{DCG}_i=\sum_{j'}Y_{ij'}w(\operatorname{rank}_i(j')) DCG i = ∑ j ′ Y i j ′ w ( rank i ( j ′ )) . Assume n j + > 0 n_j^{+}>0 n j + > 0 and n j − > 0 n_j^{-}>0 n j − > 0 , and that the global micro-nDCG denominator is positive. The six aggregations of the DCG family behave as follows; the value changes with λ \lambda λ for some λ \lambda λ if and only if the stated condition fails. In the last row, put G + = { i : Y i j = 1 } G^{+}=\{i:Y_{ij}=1\} G + = { i : Y ij = 1 } , G − = { i : Y i j = 0 } G^{-}=\{i:Y_{ij}=0\} G − = { i : Y ij = 0 } , and set A ± = ∑ i ∈ G ± D C G i A^{\pm}=\sum_{i\in G^{\pm}}\mathrm{DCG}_i A ± = ∑ i ∈ G ± DCG i , B ± = ∑ i ∈ G ± I D C G i B^{\pm}=\sum_{i\in G^{\pm}}\mathrm{IDCG}_i B ± = ∑ i ∈ G ± IDCG i . The cross-product condition does not require B − > 0 B^{-}>0 B − > 0 ; if both group ideal sums are positive, it is equivalent to A + / B + = A − / B − A^{+}/B^{+}=A^{-}/B^{-} A + / B + = A − / B − .
Table 70.
Thus n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class is invariant for every admissible non-increasing weight without an additional data-dependent equality. The other aggregations obey the exact conditions in the table; such conditions may hold automatically for particular weights. For instance, when w ( r ) = 1 w(r)=1 w ( r ) = 1 and every included example has k i > 0 k_i>0 k i > 0 , both exemplar-mean and micro-nDCG are identically 1 1 1 under skew. Hence no unqualified uniqueness claim is intended. The per-class profile { n D C G j c l a s s } j = 1 m \{\mathrm{nDCG}^{\mathrm{class}}_j\}_{j=1}^{m} { nDCG j class } j = 1 m plays, in the Ranking Framework, the role that recall and specificity play in the Discrimination Framework and that ( α j , β j ) (\alpha_j,\beta_j) ( α j , β j ) plays in the Calibration Framework. Proof. The first row is Proposition 68. In the second, Y i j w ( rank i ( j ) ) Y_{ij}w(\operatorname{rank}_i(j)) Y ij w ( rank i ( j )) vanishes on the negative examples, so D C G j c l a s s \mathrm{DCG}^{\mathrm{class}}_j DCG j class is the two-group average of the proof of Proposition 69 with negative-group mean 0 and positive-group mean w ( 1 ) n D C G j c l a s s w(1)\,\mathrm{nDCG}^{\mathrm{class}}_j w ( 1 ) nDCG j class ; the average is independent of λ \lambda λ if and only if the two group means coincide, i.e. if and only if n D C G j c l a s s = 0 \mathrm{nDCG}^{\mathrm{class}}_j=0 nDCG j class = 0 . Rows three and four are the same two-group argument applied to the per-example quantities D C G i \mathrm{DCG}_i DCG i and n D C G i \mathrm{nDCG}_i nDCG i . In row three the two reported raw summaries coincide exactly, not merely up to a factor: D C G m i c r o = ( 1 / n ) ∑ i D C G i e x = ( 1 / ( n m ) ) ∑ i D C G i \mathrm{DCG}^{\mathrm{micro}}=(1/n)\sum_i\mathrm{DCG}^{\mathrm{ex}}_i=(1/(nm))\sum_i\mathrm{DCG}_i DCG micro = ( 1/ n ) ∑ i DCG i ex = ( 1/ ( nm )) ∑ i DCG i , by Theorem 60(a). For the last row, the replicated micro-nDCG is ( λ A + + A − ) / ( λ B + + B − ) (\lambda A^+ + A^-)/(\lambda B^+ + B^-) ( λ A + + A − ) / ( λ B + + B − ) with A ± = ∑ ± D C G i A^\pm=\sum_\pm\mathrm{DCG}_i A ± = ∑ ± DCG i and B ± = ∑ ± I D C G i B^\pm=\sum_\pm\mathrm{IDCG}_i B ± = ∑ ± IDCG i , where B + > 0 B^+>0 B + > 0 but B − B^- B − may vanish; for λ ≠ μ \lambda\ne\mu λ = μ the cross-products ( λ A + + A − ) ( μ B + + B − ) (\lambda A^++A^-)(\mu B^++B^-) ( λ A + + A − ) ( μ B + + B − ) and ( μ A + + A − ) ( λ B + + B − ) (\mu A^++A^-)(\lambda B^++B^-) ( μ A + + A − ) ( λ B + + B − ) differ by ( λ − μ ) ( A + B − − A − B + ) (\lambda-\mu)(A^+B^- - A^-B^+) ( λ − μ ) ( A + B − − A − B + ) , which vanishes if and only if A + B − = A − B + A^+B^-=A^-B^+ A + B − = A − B + ; if B + B − > 0 B^+B^->0 B + B − > 0 , this is equivalently A + / B + = A − / B − A^+/B^+=A^-/B^- A + / B + = A − / B − .
Column-Dependent Operators and the AUC Family
The row-dependent positional operator rewards label j j j for being ranked highly within example i i i ’s score vector . The dual operator asks whether example i i i is ranked highly within class j j j ’s score column . This column-ranking perspective underlies AUC-based multilabel evaluation (Flach, 2003).
Definition 71 (Column rank and column-dependent positional operator). The column rank of example i i i for class j j j is
rank j ( i ) = ∣ { i ′ : Y ^ i ′ , j > Y ^ i , j } ∣ + 1 , (48) \operatorname{rank}_j(i)=\left|\{i' : \widehat{Y}_{i',j}>\widehat{Y}_{i,j}\}\right|+1, \tag*{(48)} rank j ( i ) = { i ′ : Y i ′ , j > Y i , j } + 1 , (48) the position of Y ^ i , j \widehat{Y}_{i,j} Y i , j in the descending ordering of column j j j of Y ^ \widehat{Y} Y . A column-dependent positional operator with weight function v v v is
D v ( Y i , j , Y ^ ⋅ , j ) = Y i , j ⋅ v ( rank j ( i ) ) . (49) D_v(Y_{i,j},\widehat{Y}_{\cdot,j})=Y_{i,j}\cdot v(\operatorname{rank}_j(i)). \tag*{(49)} D v ( Y i , j , Y ⋅ , j ) = Y i , j ⋅ v ( rank j ( i )) . (49) The structural difference from the row-dependent case is the axis of dependency: D v D_v D v depends on the full predicted column Y ^ ⋅ , j \widehat{Y}_{\cdot,j} Y ⋅ , j rather than the full predicted row Y ^ i , ⋅ \widehat{Y}_{i,\cdot} Y i , ⋅ .
AUC via the Wilcoxon–Mann–Whitney statistic
The canonical column-dependent measure is AUC. For class j j j , define the concordance fraction of positive example i i i as the fraction of negative examples it outranks:
f ( i , j ) = 1 n j − ∑ i ′ : Y i ′ , j = 0 ( 1 [ Y ^ i , j > Y ^ i ′ , j ] + 1 2 1 [ Y ^ i , j = Y ^ i ′ , j ] ) . (50) f(i,j)=\frac{1}{n_j^-}\sum_{i':Y_{i',j}=0}\left(\mathbf{1}[\widehat{Y}_{i,j}>\widehat{Y}_{i',j}]+\frac{1}{2}\mathbf{1}[\widehat{Y}_{i,j}=\widehat{Y}_{i',j}]\right). \tag*{(50)} f ( i , j ) = n j − 1 i ′ : Y i ′ , j = 0 ∑ ( 1 [ Y i , j > Y i ′ , j ] + 2 1 1 [ Y i , j = Y i ′ , j ] ) . (50)
This definition requires n j − > 0 n_j^->0 n j − > 0 ; the corresponding class AUC additionally requires n j + > 0 n_j^+>0 n j + > 0 . The AUC cell matrix A ∈ [ 0 , 1 ] n × m A\in[0,1]^{n\times m} A ∈ [ 0 , 1 ] n × m is
A i , j = Y i , j ⋅ f ( i , j ) , (51) A_{i,j}=Y_{i,j}\cdot f(i,j), \tag*{(51)} A i , j = Y i , j ⋅ f ( i , j ) , (51)
so A i , j = 0 A_{i,j}=0 A i , j = 0 whenever Y i , j = 0 Y_{i,j}=0 Y i , j = 0 .
At the per-class aggregate level, Proposition 73 shows that AUC is an affine, decreasing transformation of the mean positive-example column rank obtained from the fixed-rank operator of Definition 71 with v ( r ) = r v(r)=r v ( r ) = r . This aggregate identity does not equate the individual concordance cells Y i j f ( i , j ) Y_{ij}f(i,j) Y ij f ( i , j ) with fixed rank-weighted cells Y i j v ( rank j ( i ) ) Y_{ij}v(\operatorname{rank}_{j}(i)) Y ij v ( rank j ( i )) : f ( i , j ) f(i,j) f ( i , j ) also depends on which other examples are negative. Thus the analogy with normalised DCG holds at the aggregate level, while the AUC cell construction retains its label-dependent concordance fractions.
Definition 72 (AUC family via the three aggregation operators). The “micro-AUC” defined below aggregates within-class concordance values with positive-class-count weights. It is not the AUC of a single ROC curve obtained by pooling all class–example scores, which also involves cross-class positive–negative comparisons (Remark 81 ). The three aggregations of A A A yield:
AUC j = [ ∑ m ( A ) ] j n π j = 1 n j + ∑ i : Y i j = 1 f ( i , j ) = E [ f ( i , j ) ∣ Y i j = 1 ] , (52) \operatorname{AUC}_{j}=\frac{\left[\sum_{m}(A)\right]_{j}}{n\pi_{j}}=\frac{1}{n_{j}^{+}}\sum_{i:Y_{ij}=1}f(i,j)=\mathbb{E}[f(i,j)\mid Y_{ij}=1], \tag*{(52)} AUC j = n π j [ ∑ m ( A ) ] j = n j + 1 i : Y ij = 1 ∑ f ( i , j ) = E [ f ( i , j ) ∣ Y ij = 1 ] , (52) AUC m i c r o = ∑ 1 ( A ) ∑ 1 ( Y ) = ∑ j n j + AUC j ∑ j n j + , (53) \operatorname{AUC}^{\mathrm{micro}}=\frac{\sum_{1}(A)}{\sum_{1}(Y)}=\frac{\sum_{j}n_{j}^{+}\operatorname{AUC}_{j}}{\sum_{j}n_{j}^{+}}, \tag*{(53)} AUC micro = ∑ 1 ( Y ) ∑ 1 ( A ) = ∑ j n j + ∑ j n j + AUC j , (53) AUC i e x = [ ∑ n ( A ) ] i k i = 1 k i ∑ j : Y i j = 1 f ( i , j ) , (54) \operatorname{AUC}_{i}^{\mathrm{ex}}=\frac{\left[\sum_{n}(A)\right]_{i}}{k_{i}}=\frac{1}{k_{i}}\sum_{j:Y_{ij}=1}f(i,j), \tag*{(54)} AUC i ex = k i [ ∑ n ( A ) ] i = k i 1 j : Y ij = 1 ∑ f ( i , j ) , (54) *where k i = ∑ j Y i j k_{i}=\sum_{j}Y_{ij} k i = ∑ j Y ij . The per-example ratio is defined for k i > 0 k_{i}>0 k i > 0 ; empty-label examples require an explicit omission or zero-score convention. AUC j \operatorname{AUC}_{j} AUC j is the standard per-class AUC (Fawcett, 2006); AUC m i c r o \operatorname{AUC}^{\mathrm{micro}} AUC micro is the prevalence-weighted average of per-class AUCs1 ; AUC i e x \operatorname{AUC}_{i}^{\mathrm{ex}} AUC i ex is a per-example AUC (the average concordance fraction of example i i i across its relevant labels), which does not appear as a named measure in the existing literature. It compares example i i i with the negatives of each relevant class (a column-axis quantity), and is not the per-example label-AUC of the row axis, which equals one minus the multilabel ranking loss (see the grid below).*
(52) recovers the Wilcoxon–Mann–Whitney statistic: AUC j = P ( Y ^ i + , j > Y ^ i − , j ) + 1 2 P ( Y ^ i + , j = Y ^ i − , j ) \operatorname{AUC}_{j}=P(\hat{Y}_{i^{+},j}>\hat{Y}_{i^{-},j})+\frac{1}{2}P(\hat{Y}_{i^{+},j}=\hat{Y}_{i^{-},j}) AUC j = P ( Y ^ i + , j > Y ^ i − , j ) + 2 1 P ( Y ^ i + , j = Y ^ i − , j ) for independently sampled positive i + i^{+} i + and negative i − i^{-} i − (the second term gives half credit to ties as in Equation (50) ), which equals the area under the ROC curve (Wilcoxon, 1945; Mann and Whitney, 1947; Hanley and McNeil, 1982).
The next proposition identifies the exact aggregate-level rank relationship. It does not assert a cell-by-cell identity or that the affine AUC normalisation is literally an IDCG ratio.
Proposition 73 (AUC as a normalised mean rank). Fix a class j j j with n j + n_j^+ n j + positive and n j − n_j^- n j − negative examples, n = n j + + n j − n = n_j^+ + n_j^- n = n j + + n j − , and let
M j = 1 n j + ∑ i : Y i j = 1 rank j ( i ) (55) M_j = \frac{1}{n_j^+} \sum_{i:Y_{ij}=1} \operatorname{rank}_j(i) \tag*{(55)} M j = n j + 1 i : Y ij = 1 ∑ rank j ( i ) (55) be the mean descending column rank (48) of the positive examples (the raw aggregate of the fixed-weight column operator D v D_v D v with v ( r ) = r v(r) = r v ( r ) = r ). Assuming distinct scores within column j j j ,
AUC j = n − 1 2 ( n j + − 1 ) − M j n j − , (56) \operatorname{AUC}_j = \frac{n - \frac{1}{2}(n_j^+ - 1) - M_j}{n_j^-}, \tag*{(56)} AUC j = n j − n − 2 1 ( n j + − 1 ) − M j , (56) an affine, strictly decreasing function of M j M_j M j whose coefficients depend only on the counts ( n , n j + , n j − ) (n, n_j^+, n_j^-) ( n , n j + , n j − ) . Hence M j M_j M j is skew-sensitive in general (replicating negatives can change the positive ranks and thus M j M_j M j ), while AUC j \operatorname{AUC}_j AUC j is its skew-invariant normalisation, the affine map (56) removing the prevalence dependence.
Proof. A positive example i i i outranks n − rank j ( i ) n - \operatorname{rank}_j(i) n − rank j ( i ) of the other examples. Summing over positives, ∑ i : Y i j = 1 ( n − rank j ( i ) ) = n n j + − n j + M j \sum_{i:Y_{ij}=1}(n - \operatorname{rank}_j(i)) = n n_j^+ - n_j^+ M_j ∑ i : Y ij = 1 ( n − rank j ( i )) = n n j + − n j + M j counts all pairs a positive outranks; of these, ( n j + 2 ) = 1 2 n j + ( n j + − 1 ) \binom{n_j^+}{2} = \frac{1}{2}n_j^+(n_j^+ - 1) ( 2 n j + ) = 2 1 n j + ( n j + − 1 ) join two positives, leaving n n j + − n j + M j − 1 2 n j + ( n j + − 1 ) n n_j^+ - n_j^+ M_j - \frac{1}{2}n_j^+(n_j^+ - 1) n n j + − n j + M j − 2 1 n j + ( n j + − 1 ) positive–negative concordant pairs. Dividing by n j + n j − n_j^+n_j^- n j + n j − gives (56) ; strict decrease in M j M_j M j and the count-only coefficients are immediate. With ties, the same identity holds for mid-ranks and the 1 2 \frac{1}{2} 2 1 credit of (50) , the standard Mann–Whitney correction; with the min-rank column rank of Definition 71 it does not. The formally verified statement assumes distinct scores.
Row–Column Duality: nDCG and AUC
The two families share a common algebraic form, differing only in the axis of the rank.
Theorem 74 (Row–column duality). The per-class measures of the nDCG and AUC families are both conditional expectations over positive examples evaluating rank quality along orthogonal axes:
nDCG j c l a s s = 1 w ( 1 ) E [ w ( rank i ( j ) ) ∣ Y i j = 1 ] , (57) \operatorname{nDCG}^{\mathrm{class}}_j = \frac{1}{w(1)} \mathbb{E}\left[w(\operatorname{rank}_i(j)) \mid Y_{ij}=1\right], \tag*{(57)} nDCG j class = w ( 1 ) 1 E [ w ( rank i ( j )) ∣ Y ij = 1 ] , (57) AUC j = E [ f ( i , j ) ∣ Y i j = 1 ] . (58) \operatorname{AUC}_j = \mathbb{E}\left[f(i,j) \mid Y_{ij}=1\right]. \tag*{(58)} AUC j = E [ f ( i , j ) ∣ Y ij = 1 ] . (58) In (57) , rank i ( j ) \operatorname{rank}_i(j) rank i ( j ) is the row rank: position of label j j j in the descending ordering of Y ^ i , ⋅ \hat{Y}_{i,\cdot} Y ^ i , ⋅ .
In (58) , f ( i , j ) f(i,j) f ( i , j ) is the concordance fraction, the share of negative examples that example i i i outranks. It is not by itself a function of the column rank rank j ( i ) \operatorname{rank}_j(i) rank j ( i ) , because the examples below i i i include positives as well as negatives; but its mean over the positives is, by Proposition 73 , the affine normalisation of the mean column rank. Both measures are therefore conditional means, over the positive examples, of a rank-based quality: for nDCG a fixed function of the row rank, for AUC the concordance fraction, whose average over the positives is an affine function of the mean column rank.
Proof. (57) is Proposition 62. (58) follows from Definition 72, equation (51). Both are conditional means over positive cells. For nDCG, the summand is a fixed discount of the row rank; for AUC, it is a positive–negative concordance fraction depending on the negative-score distribution. Only after aggregation over positives is the latter an affine function of the mean column rank (Proposition 73).
Example 4 (Running example: the two ranking axes disagree). In Example 1, A is ranked first in every row, so n D C G A c l a s s = 1 \mathrm{nDCG}^{\mathrm{class}}_{A}=1 nDCG A class = 1 , yet A U C A = 1 / 2 \mathrm{AUC}_{A}=1/2 AUC A = 1/2 : its positives ( 0.7 , 0.6 ) (0.7,0.6) ( 0.7 , 0.6 ) are not separated from its negatives ( 0.8 , 0.4 ) (0.8,0.4) ( 0.8 , 0.4 ) . For B it is the other way round: A U C B = 1 \mathrm{AUC}_{B}=1 AUC B = 1 , since its positives ( 0.6 , 0.2 , 0.3 ) (0.6,0.2,0.3) ( 0.6 , 0.2 , 0.3 ) all exceed its negative 0.1 0.1 0.1 , but B is never first in a row and n D C G B c l a s s ≈ 0.587 \mathrm{nDCG}^{\mathrm{class}}_{B}\approx0.587 nDCG B class ≈ 0.587 . Both are conditional means over the positives (Theorem 74), of a row-rank and of a column-rank quality respectively, and here they order the labels in opposite ways ( n D C G C c l a s s ≈ 0.631 , A U C C = 2 / 3 ) (\mathrm{nDCG}^{\mathrm{class}}_{C}\approx0.631,\ \mathrm{AUC}_{C}=2/3) ( nDCG C class ≈ 0.631 , AUC C = 2/3 ) . The hard top-1 prediction is A for every example, so Precision@1 = 1 / 2 =1/2 = 1/2 (Proposition 63).
Remark 75 (Two complementary ranking questions). Theorem 74 makes explicit the distinct questions the two families answer. n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_{j} nDCG j class asks: for a randomly chosen positive example for label j j j , how highly does the classifier rank j j j among all labels for that example? A U C j \mathrm{AUC}_{j} AUC j asks: for a randomly chosen positive example for class j j j , what share of the negative examples for class j j j does the classifier rank below it? The two questions are independent: a classifier may rank each example’s positive labels highly (high nDCG) while failing to separate positive from negative examples globally (low AUC), or vice versa. A complete ranking evaluation requires both.
The rest of this subsection discusses the weight w w w itself and states no new formal results. The weight w w w is an exogenous parameter of the positional operator, and different choices recover the standard ranking measures from the single template Y i , j ⋅ w ( rank ) Y_{i,j}\cdot w(\operatorname{rank}) Y i , j ⋅ w ( rank ) , in the same way that one t t t -norm generates precision, recall and F 1 F_1 F 1 on the discrimination axis.
Table 4.
Two structural facts hold across the whole family. First, skew-invariance is independent of w w w : for any non-increasing w w w , Theorem 74 gives n D C G j c l a s s = 1 w ( 1 ) E [ w ( rank i ( j ) ) ∣ Y i j = 1 ] \mathrm{nDCG}^{\mathrm{class}}_{j}=\frac{1}{w(1)}\mathbb{E}[w(\operatorname{rank}_{i}(j))\mid Y_{ij}=1] nDCG j class = w ( 1 ) 1 E [ w ( rank i ( j )) ∣ Y ij = 1 ] , a conditional mean over positives and hence skew-invariant; the weight sets the discount profile but not the invariance. Second, no choice of non-increasing w w w makes the operator strictly proper (Proposition 65 holds for every non-increasing w w w ); the weight w ( r ) = r w(r)=r w ( r ) = r is a cost to be minimised rather than a gain, and AUC’s non-strict properness is Proposition 79. The weight thus indexes named measures without altering the structural profile of the class. The geometric weight γ r − 1 \gamma^{r-1} γ r − 1 carries a probabilistic user-persistence semantics (the user inspects rank r r r with probability γ r − 1 \gamma^{r-1} γ r − 1 ) and a generating-function form ∑ r ≥ 1 w ( r ) x r − 1 = ( 1 − γ x ) − 1 \sum_{r\ge1}w(r)x^{r-1}=(1-\gamma x)^{-1} ∑ r ≥ 1 w ( r ) x r − 1 = ( 1 − γ x ) − 1 , and interpolates between top-heavy (γ → 0 \gamma\to0 γ → 0 ) and uniform (γ → 1 \gamma\to1 γ → 1 ) discounts through a single parameter.
The exogenous/effective distinction of Section 6.8 is not specific to the column axis. On the row axis, expected reciprocal rank (Chapelle et al., 2009) uses an effective weight whose discount at position r r r depends on the relevance of the higher-ranked labels (a cascade user model), in the same way as the AUC weight depends on which lower-ranked examples are negative. Expected reciprocal rank is therefore to nDCG what AUC is to the raw mean rank: an endogenous, label-dependent specialisation of the fixed-weight operator. The exogenous-versus-effective dichotomy thus runs along both axes, and in both cases the effective weight is a normalisation of, or a data-dependent modulation of, the underlying fixed-weight positional operator.
Crossing the two axes (row: rank labels within an example; column: rank examples within a label) with the two canonical exogenous weights (linear w ( r ) = r w(r)=r w ( r ) = r and top-heavy w ( r ) = 1 / log 2 ( r + 1 ) w(r)=1/\log_{2}(r+1) w ( r ) = 1/ log 2 ( r + 1 ) ) places the named measures on a 2 × 2 2 \times2 2 × 2 grid with one unfilled cell:
Table 1.
The two standard measures sit on the anti-diagonal: nDCG is the row axis under a top-heavy weight, and AUC (via Proposition 73 ) is the column axis under a linear weight, so they differ in both coordinates. The cells on the main diagonal are the duals with the weights swapped. The row/linear cell corresponds to per-example label-pair ranking (one minus the multilabel ranking loss under matching conventions), which is distinct from the column-based A U C i e x \mathrm{AUC}_{i}^{\mathrm{ex}} AUC i ex of Definition 72 (Schapire and Singer, 2000). For a row with 0 < k i < m 0 < k_i < m 0 < k i < m and distinct score ranks, this is the affine-normalised linear rank sum 1 − R L i = 1 − ( ∑ j : Y i j = 1 rank i ( j ) − k i ( k i + 1 ) / 2 ) / ( k i ( m − k i ) ) 1-\mathrm{RL}_i = 1-\left(\sum_{j:Y_{ij}=1}\operatorname{rank}_i(j)-k_i(k_i+1)/2\right)/(k_i(m-k_i)) 1 − RL i = 1 − ( ∑ j : Y ij = 1 rank i ( j ) − k i ( k i + 1 ) /2 ) / ( k i ( m − k i )) ; the raw cost v ( r ) = r v(r)=r v ( r ) = r alone is not numerically equal to 1 − R L i 1-\mathrm{RL}_i 1 − RL i . The column/top-heavy cell, column-nDCG , is not, to our knowledge, a standard measure of multilabel evaluation (in bipartite ranking, the DCG of a class column is a member of the linear rank-statistics family of Clémençon et al. (2021)): it asks whether the truly positive examples of a label sit at the top of that label’s ranked list, under a logarithmic discount. It is the top-heavy alternative to AUC. Because AUC uses a linear rank weight, it has no additional top-heavy discount beyond positive–negative pair ordering (although swapping a positive and a negative does change AUC), whereas column-nDCG (or its rank-biased variant w ( r ) = γ r − 1 w(r)=\gamma^{r-1} w ( r ) = γ r − 1 ) rewards concentrating them at the top, the quantity that matters when the top-k k k examples of a label are retrieved for downstream action. Close relatives exist in retrieval and extreme classification (per-label average precision, precision@k); the point here is structural: the operator grid exposes column-nDCG as the axis-swapped dual of nDCG rather than as an ad hoc metric.
Structural Properties of the AUC Family
Theorem 76 (Collapse and non-collapse for AUC). *Assume n j + > 0 n_j^{+}>0 n j + > 0 and n j − > 0 n_j^{-}>0 n j − > 0 for every class j j j , and k i > 0 k_i>0 k i > 0 for every example i i i , so that every exemplar AUC entering the mean over all n n n examples is defined. Write Cov ^ j \widehat{\operatorname{Cov}}_j Cov j and Cov ^ i \widehat{\operatorname{Cov}}_i Cov i for the uniform empirical covariances over the m m m classes and over the n n n examples, respectively. Put H = ∑ j n j + = ∑ i k i > 0 , H=\sum_j n_j^{+}=\sum_i k_i>0, H = ∑ j n j + = ∑ i k i > 0 ,
A i j = Y i j f ( i , j ) A_{ij}=Y_{ij}f(i,j) A ij = Y ij f ( i , j ) , and denote the unweighted macro- and exemplar-means by A U C ‾ c l a s s = m − 1 ∑ j A U C j \overline{\mathrm{AUC}}^{\mathrm{class}}=m^{-1}\sum_j\mathrm{AUC}_j AUC class = m − 1 ∑ j AUC j and A U C ‾ e x = n − 1 ∑ i A U C i e x \overline{\mathrm{AUC}}^{\mathrm{ex}}=n^{-1}\sum_i\mathrm{AUC}^{\mathrm{ex}}_i AUC ex = n − 1 ∑ i AUC i ex .
(a) The raw cell aggregate satisfies
Σ 1 ( A ) n m = 1 m ∑ j π j A U C j = 1 n ∑ i k i m A U C i e x . (59) \frac{\Sigma_1(A)}{nm}=\frac{1}{m}\sum_j\pi_j\mathrm{AUC}_j=\frac{1}{n}\sum_i\frac{k_i}{m}\mathrm{AUC}^{\mathrm{ex}}_i. \tag*{(59)} nm Σ 1 ( A ) = m 1 j ∑ π j AUC j = n 1 i ∑ m k i AUC i ex . (59)
(b) A U C m i c r o = A U C ‾ c l a s s \mathrm{AUC}^{\mathrm{micro}}=\overline{\mathrm{AUC}}^{\mathrm{class}} AUC micro = AUC class if and only if Cov ^ j ( n j + , A U C j ) = 0 \widehat{\operatorname{Cov}}_j(n_j^+,\mathrm{AUC}_j)=0 Cov j ( n j + , AUC j ) = 0 .
(c) A U C m i c r o = A U C ‾ e x \mathrm{AUC}^{\mathrm{micro}}=\overline{\mathrm{AUC}}^{\mathrm{ex}} AUC micro = AUC ex if and only if Cov ^ i ( k i , A U C i e x ) = 0 \widehat{\operatorname{Cov}}_i(k_i,\mathrm{AUC}^{\mathrm{ex}}_i)=0 Cov i ( k i , AUC i ex ) = 0 .
(d) A U C ‾ c l a s s = A U C ‾ e x \overline{\mathrm{AUC}}^{\mathrm{class}}=\overline{\mathrm{AUC}}^{\mathrm{ex}} AUC class = AUC ex if and only if
∑ i , j : Y i j = 1 A i j ( 1 m n j + − 1 n k i ) = 0. (60) \sum_{i,j:Y_{ij}=1} A_{ij}\left(\frac{1}{mn_j^+}-\frac{1}{n k_i}\right)=0. \tag*{(60)} i , j : Y ij = 1 ∑ A ij ( m n j + 1 − n k i 1 ) = 0. (60)
As an identity for every real cell matrix supported on the positive cells of a fixed label matrix Y Y Y , this equality holds if and only if m n j + = n k i mn_j^+=nk_i m n j + = n k i at every positive cell ( i , j ) (i,j) ( i , j ) . This stronger, universal statement is about supported cell arrays; it does not claim that arbitrary such arrays can be generated by AUC concordances.
Thus the three normalised summaries need not coincide; their equality is governed by the stated conditions, rather than being excluded in all datasets.
Proof sketch (a) sums the concordance cells A i j A_{ij} A ij by rows and by columns. (b) and (c) are the covariance identity for the weights n j + n_j^+ n j + and k i k_i k i . (d) expands both means as linear forms in the positive cells; equality for every supported array forces each coefficient to vanish. The full proof is in Appendix E.
Example 5 (Running example: collapse and non-collapse). The positional cells of Example 2 give the same raw DCG, 0.366, whether averaged over all cells, by labels or by examples (Theorem 60 (a)); the normalised summaries differ: micro-nDCG 0.835, class mean 0.739, exemplar mean 0.783. Likewise the raw AUC aggregate is 7 / 18 7/18 7/18 in every order of summation, while macro-AUC is 13 / 18 13/18 13/18 , micro-AUC 7 / 9 7/9 7/9 and the exemplar mean 5 / 6 5/6 5/6 (Theorem 76 ). Micro and macro AUC differ because the class sizes n j + = ( 2 , 3 , 1 ) n_j^+=(2,3,1) n j + = ( 2 , 3 , 1 ) covary with A U C j = ( 1 / 2 , 1 , 2 / 3 ) \mathrm{AUC}_j=(1/2,1,2/3) AUC j = ( 1/2 , 1 , 2/3 ) .
Proposition 77 (Per-class and macro-AUC are skew-invariant; micro-AUC may be skew-sensitive). Under independent ( λ , μ ) (\lambda,\mu) ( λ , μ ) rescaling of positive and negative example counts for class j j j :
(a) A U C j = P ( Y ^ i + , j > Y ^ i − , j ) + 1 2 P ( Y ^ i + , j = Y ^ i − , j ) \mathrm{AUC}_j=P(\widehat{Y}_{i^+,j}>\widehat{Y}_{i^-,j})+\frac{1}{2}P(\widehat{Y}_{i^+,j}=\widehat{Y}_{i^-,j}) AUC j = P ( Y i + , j > Y i − , j ) + 2 1 P ( Y i + , j = Y i − , j ) is skew-invariant (with the same half-credit tie convention as Definition 72 ).
(b) M a c r o - A U C = 1 m ∑ j A U C j \mathrm{Macro\text{-}AUC}=\frac{1}{m}\sum_j\mathrm{AUC}_j Macro - AUC = m 1 ∑ j AUC j is skew-invariant.
(c) Micro-AUC = ∑ l n l + A U C l ∑ l n l + = \frac{\sum_l n_l^+ \mathrm{AUC}_l}{\sum_l n_l^+} = ∑ l n l + ∑ l n l + AUC l is skew-sensitive in general: scaling the positive examples of class j j j by λ \lambda λ changes it (for some λ \lambda λ ) if and only if A U C j \mathrm{AUC}_j AUC j differs from the micro-AUC itself. If the A U C l \mathrm{AUC}_l AUC l are not all equal, such a class exists; if they are all equal, micro-AUC is invariant.
Proof (a) Replicating positive examples λ \lambda λ -fold leaves their score distribution unchanged; replicating negatives μ \mu μ -fold leaves theirs unchanged. Hence both the strict-concordance probability and the tie probability are invariant; their half-credit combination, A U C j \mathrm{AUC}_j AUC j , is invariant. (b) Average of invariant terms. (c) Scaling the positives of class j j j by λ \lambda λ replaces n j + n_j^+ n j + by λ n j + \lambda n_j^+ λ n j + and leaves every A U C l \mathrm{AUC}_l AUC l unchanged, by (a). With S = ∑ l n l + A U C l S = \sum_l n_l^+\mathrm{AUC}_l S = ∑ l n l + AUC l and N = ∑ l n l + N = \sum_l n_l^+ N = ∑ l n l + , micro-AUC becomes S + ( λ − 1 ) n j + A U C j N + ( λ − 1 ) n j + \frac{S+(\lambda-1)n_j^+\mathrm{AUC}_j}{N+(\lambda-1)n_j^+} N + ( λ − 1 ) n j + S + ( λ − 1 ) n j + AUC j . For λ ≠ μ \lambda\ne\mu λ = μ , cross-multiplying, the two cross-products differ by n j + ( λ − μ ) ( A U C j N − S ) n_j^+(\lambda-\mu)(\mathrm{AUC}_jN-S) n j + ( λ − μ ) ( AUC j N − S ) , which vanishes if and only if A U C j = S / N \mathrm{AUC}_j = S/N AUC j = S / N , the micro-AUC. If A U C j = S / N \mathrm{AUC}_j = S/N AUC j = S / N for every j j j , the A U C j \mathrm{AUC}_j AUC j are all equal; otherwise some class has A U C j ≠ S / N \mathrm{AUC}_j \ne S/N AUC j = S / N .
Proposition 77 settles AUC’s skew-invariance case by case. The same pattern has now appeared in all three frameworks, and the next remark gives it a single explanation.
Properness (§ 5.6 ) is a property independent of skew-invariance; the next proposition settles it for AUC.
Proposition 79 (AUC is non-strict; truthful ranking may fail under dependence). Fix n ≥ 2 n \ge2 n ≥ 2 examples of one class, with Bernoulli labels Y 1 , … , Y n Y_1,\ldots,Y_n Y 1 , … , Y n and marginal probabilities p i = P ( Y i = 1 ) p_i = P(Y_i=1) p i = P ( Y i = 1 ) . Reports q ∈ [ 0 , 1 ] n q \in[0,1]^n q ∈ [ 0 , 1 ] n are evaluated by pairwise AUC with half credit for ties. For each label realisation, adopt the conventional A U C ( q , Y ) = 0 \mathrm{AUC}(q,Y)=0 AUC ( q , Y ) = 0 whenever n + n − = 0 n^+n^-=0 n + n − = 0 ; otherwise use the ordinary concordance fraction. Then:
(a) Under any joint law of the labels, the expected AUC cannot be uniquely maximised by a report: a distinct report with the same order and ties has the same value.
(b) If the labels of different examples are independent, ordering scores by the truthful marginals p i p_i p i maximises expected AUC (ties between equal marginals are allowed). Hence AUC is not strictly proper even in this setting.
(c) For n = 2 n = 2 n = 2 or n = 3 n = 3 n = 3 , the truthful ordering maximises expected AUC for every joint law, without independence.
(d) For every n ≥ 4 n \ge4 n ≥ 4 , there exists a joint label distribution with distinct marginals for which a non-truthful ordering has strictly greater expected AUC. Thus truthful maximality is not guaranteed under arbitrary dependence.
Proof (a) AUC depends on q q q only through the strict comparisons and ties between reported scores, all preserved by a strictly increasing transformation g : [ 0 , 1 ] → [ 0 , 1 ] g : [0,1] \to[0,1] g : [ 0 , 1 ] → [ 0 , 1 ] . For any report q ≠ 0 q \ne0 q = 0 , take g ( t ) = t / 2 g(t) = t/2 g ( t ) = t /2 , so g ( q ) ≠ q g(q) \ne q g ( q ) = q ; if q = 0 q = 0 q = 0 , take g ( t ) = ( 1 + t ) / 2 g(t) = (1+t)/2 g ( t ) = ( 1 + t ) /2 . In either case the transformed report is different but has exactly the same AUC for every label vector, and hence the same expectation.
(b) Put Y i c = 1 − Y i Y_i^c = 1 - Y_i Y i c = 1 − Y i and, for i ≠ i ′ i \ne i' i = i ′ , define
b i i ′ = E [ Y i Y i ′ c n + n − 1 [ n + n − > 0 ] ] . b_{ii'} = \mathbb{E}\left[\frac{Y_iY_{i'}^c}{n^+n^-}\mathbf{1}[n^+n^- > 0]\right]. b i i ′ = E [ n + n − Y i Y i ′ c 1 [ n + n − > 0 ] ] .
A report ordering q i > q i ′ q_i > q_{i'} q i > q i ′ receives b i i ′ b_{ii'} b i i ′ from this ordered pair; a tie receives ( b i i ′ + b i ′ i ) / 2 (b_{ii'} + b_{i'i})/2 ( b i i ′ + b i ′ i ) /2 . Under label independence, conditioning on Y i = 1 Y_i = 1 Y i = 1 , Y i ′ = 0 Y_{i'} = 0 Y i ′ = 0 leaves the other labels independent of this pair. On this event, n + = 1 + R + n^+ = 1 + R^+ n + = 1 + R + and n − = 1 + R − n^- = 1 + R^- n − = 1 + R − , where R + , R − R^+, R^- R + , R − count the labels of the remaining n − 2 n - 2 n − 2 examples. Hence
b i i ′ = p i ( 1 − p i ′ ) K i i ′ , b i ′ i = p i ′ ( 1 − p i ) K i i ′ , K i i ′ = E [ 1 ( 1 + R + ) ( 1 + R − ) ] > 0. b_{ii'} = p_i(1-p_{i'})K_{ii'}, \qquad b_{i'i} = p_{i'}(1-p_i)K_{ii'}, \qquad K_{ii'} = \mathbb{E}\left[\frac{1}{(1+R^+)(1+R^-)}\right] > 0. b i i ′ = p i ( 1 − p i ′ ) K i i ′ , b i ′ i = p i ′ ( 1 − p i ) K i i ′ , K i i ′ = E [ ( 1 + R + ) ( 1 + R − ) 1 ] > 0.
The same K i i ′ K_{ii'} K i i ′ occurs in both orientations, so
b i i ′ − b i ′ i = K i i ′ ( p i − p i ′ ) . b_{ii'} - b_{i'i} = K_{ii'}(p_i-p_{i'}). b i i ′ − b i ′ i = K i i ′ ( p i − p i ′ ) .
Every pair is therefore optimally ordered by decreasing p i p_i p i , with ties harmless where p i = p i ′ p_i = p_{i'} p i = p i ′ . The truthful report q = p q = p q = p achieves all these pairwise optima simultaneously. Parts (c) and (d) are proved in Appendix F.
The next two remarks relate this column-dependent construction to the earlier threshold-integral account of AUC in the Discrimination Framework and state how far the relation extends.
Remark 80 (Threshold-integral and column-dependent views are equivalent). Prati (2027) derive AUC within the Discrimination Framework via the threshold-integral view: at each threshold θ \theta θ , the thresholded predictions produce cell matrices T P ( θ ) \mathrm{TP}(\theta) TP ( θ ) , F P ( θ ) \mathrm{FP}(\theta) FP ( θ ) , etc., and AUC integrates the ( F P R ( θ ) , T P R ( θ ) ) (\mathrm{FPR}(\theta), \mathrm{TPR}(\theta)) ( FPR ( θ ) , TPR ( θ )) curve. The column-dependent view introduced here is numerically equivalent when the ROC curve uses the matching half-credit (mid-rank/trapezoidal) convention for tied scores (both then give the same A U C j \mathrm{AUC}_j AUC j ), but it exposes different structure. The threshold view embeds AUC in the Discrimination Framework as a functional of coincidence-operator cell matrices parameterised by θ \theta θ . The column-dependent view embeds it in the Ranking Framework as a conditional expectation, making the duality with nDCG visible (Theorem 74 ) and yielding a uniform proof of skew-invariance (Proposition 77) that generalises across all three frameworks.
Cross-Framework Synthesis
Sections 5 and 6 developed Frameworks 2 and 3 in full. This section draws the three frameworks together, comparing their theoretical properties and identifying the common algebraic template that underlies all three.
Properties of the Three Classes
Table 4 summarises the theoretical properties of the three cell operator classes established in § 4 –§ 6.1 .
Table 3. Structural properties of the three frameworks. For Ranking, nDCG and AUC are row-dependent and column-dependent subfamilies with raw aggregation collapse but generally different normalised summaries. Properness depends on the functional and its assumptions: normalised nDCG need not be proper, while AUC is never strictly proper and may fail truthful maximality under dependent labels. The skew-invariant bases use the appropriate conditioning strata, not only the positive examples.
Several properties deserve comment.
First, coincidence operators have D ( y , y ) ≠ 0 D(y,y) \ne0 D ( y , y ) = 0 in general: under the product t t t -norm, T P ( y , y ) = y 2 T_P(y,y)=y^2 T P ( y , y ) = y 2 , which equals 1 only when y = 1 y=1 y = 1 (certain positive) and 0 only when y = 0 y=0 y = 0 (certain negative). For a soft label y = 0.7 y=0.7 y = 0.7 with equally soft prediction y ^ = 0.7 \hat{y}=0.7 y ^ = 0.7 , the TP cell value is 0.49, not 1. This is appropriate for discrimination: the classifier correctly predicted the soft label, but there is still a 0.3 probability that the label is absent, so a full TP credit would overstate certainty. Deviation operators, by contrast, give D ( 0.7 , 0.7 ) = 0 D(0.7,0.7)=0 D ( 0.7 , 0.7 ) = 0 : a prediction that exactly matches the soft label incurs zero calibration error.
Second, the micro = macro = exemplar collapse survives normalisation only in the Calibration Framework. For coincidence operators, micro- and macro-averaging differ in general when per-class denominators are unequal (they agree for every numerator only if the denominators are equal; Corollary 1 of Prati 2027). For deviation operators, all three aggregation schemes give the same scalar Brier score (Theorem 20 ), because the Brier score is a linear aggregation over cells with a fixed denominator n m nm nm . For positional operators the raw aggregates collapse as well, since they are sums of cells (Theorem 60 (a)); what breaks the collapse is normalisation by heterogeneous denominators, the ideal DCG of each example or class, so that normalised nDCG can differ across the three schemes (Theorem 60 (b)–(d)).
Third, properness selects a sub-family of the deviation operators within the Calibration Framework: among symmetric deviation operators, properness holds exactly when the weight ϕ ′ ( t ) / t \phi'(t)/t ϕ ′ ( t ) / t is symmetric about 1 2 \frac{1}{2} 2 1 (Corollary 41 (c)); squared error is the simplest member and the only power-law one. Other choices such as ϕ ( t ) = ∣ t ∣ \phi(t)=|t| ϕ ( t ) = ∣ t ∣ (MAE) are structurally within the Calibration Framework and still satisfy the micro = macro = exemplar collapse, but are not proper: their minimiser is the conditional median rather than the true probability (Remark 43 ), and the Murphy decomposition does not apply. Proper alternatives such as log-loss admit the Savage representation (Theorem 40 ); log-loss is itself a deviation operator, with ϕ ( t ) = − log ( 1 − ∣ t ∣ ) \phi(t)=-\log(1-|t|) ϕ ( t ) = − log ( 1 − ∣ t ∣ ) .
Table 4 (PDF p. 59). Properties of the three cell operator classes. Operator types: Discrimination = coincidence (Boolean conjunction, generalised by t t t -norms); Calibration = deviation ( ϕ ( y − y ^ ) ) (\phi(y-\hat{y})) ( ϕ ( y − y ^ )) ; Ranking = positional ( Y ⋅ w ( rank ) ) (Y \cdot w(\operatorname{rank})) ( Y ⋅ w ( rank )) . * A positional operator is a gain, not a loss: D = Y i , j w ( rank ) D=Y_{i,j}w(\operatorname{rank}) D = Y i , j w ( rank ) vanishes only when Y i , j = 0 Y_{i,j}=0 Y i , j = 0 (irrelevant label) or when the weight does, and is largest, D = w ( 1 ) Y i , j D=w(1)Y_{i,j} D = w ( 1 ) Y i , j , when the label is ranked first. † Raw positional operators are proper among reports with distinct entries (the truthful report maximises the expected score) but never strictly proper: any order-preserving distortion of the probabilities attains the same score (Proposition 65 ). With ties ranked by the min-rank rule, raw positional operators need not be proper (non-constant discounts admit counterexamples, while a constant weight makes every report a maximiser); normalised nDCG is not proper in general (Remark 66). ‡ For the symmetric deviation operators ϕ ( y − y ^ ) \phi(y-\hat{y}) ϕ ( y − y ^ ) with ϕ \phi ϕ even, convex, finite-valued and continuous on [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] and differentiable on ( 0 , 1 ) (0,1) ( 0 , 1 ) , properness holds exactly when ϕ ′ ( t ) / t \phi'(t)/t ϕ ′ ( t ) / t is symmetric about 1 2 \frac{1}{2} 2 1 ; strictly positive weights additionally guarantee strict properness (Corollary 41 (c)). Examples include squared error and the nonquadratic loss of Appendix § 12 ; log-loss satisfies the weight condition only in the extended-valued sense of Corollary 41 (b). Convexity alone is insufficient: ϕ ( t ) = ∣ t ∣ \phi(t)=|t| ϕ ( t ) = ∣ t ∣ (MAE) yields a non-proper operator whose minimiser is the conditional median rather than the conditional mean (Remark 43 ). § Per-class n D C G j c l a s s \mathrm{nDCG}_j^{\mathrm{class}} nDCG j class and per-class A U C j \mathrm{AUC}_j AUC j are skew-invariant (Propositions 68 and 77); exemplar and micro summaries are not in general.
Fourth, under class-indexed skew transformations that preserve the relevant conditional cell-quality distributions , skew-invariance holds for the per-class building blocks in all three frameworks and for their unweighted means; pooled and exemplar aggregates are generally skew-sensitive. For per-class AUC, preservation of the positive- and negative-conditional score distributions suffices. For per-class nDCG, preservation of each label’s own conditional score distribution alone is insufficient: the conditional distribution of the within-row rank (or at least its expected discount) among the positives must also be preserved. In the Discrimination and Calibration Frameworks, skew-invariant measures exist at the per-class and macro levels: unweighted means of skew-invariant per-class quantities remain skewinvariant, e.g. macro-recall and the unweighted means of α j \alpha_j α j and β j \beta_j β j (note that macro-F 1 F_1 F 1 is not skew-invariant, since F 1 F_1 F 1 depends on precision, and the individual Murphy terms are not skew-invariant in general either, by Corollary 53 , up to the degenerate exceptions listed there, such as R E S j = 0 \mathrm{RES}_j = 0 RES j = 0 when F j + = F j − F_j^+ = F_j^- F j + = F j − ). In the Ranking Framework, n D C G j c l a s s \mathrm{nDCG}_j^{\mathrm{class}} nDCG j class (Proposition 68 ) and A U C j \mathrm{AUC}_j AUC j (Proposition 77) are skew-invariant, and so are their unweighted means (macro-AUC under the per-class convention of § 5.7 ; the multiclass caveat of Corollary 10 applies to the per-class rates). Replicating or removing entire multilabel rows is a different operation: it may change the positive- or negative-conditional score distributions of other classes, so the class-indexed invariance statement need not hold for every affected class. Exemplar and micro positional aggregations reintroduce label-prevalence weights and are therefore not skew-invariant in general, as with prevalence-weighted discrimination summaries; Brier averages can instead change because the individual classwise losses depend on prevalence.
A Unified Skew-Invariant Template
Across the three frameworks, the skew-invariant per-class measures share a common algebraic template: each is a conditional mean of a cell-level quality measure given the relevant positive or negative class outcome, with the distribution of that conditional cell-quality quantity held fixed.
In the Discrimination Framework, recall j = E [ Y ^ i j ∣ Y i j = 1 ] \operatorname{recall}_j = \mathbb{E}[\widehat{Y}_{ij} \mid Y_{ij} = 1] recall j = E [ Y ij ∣ Y ij = 1 ] and specificity j = E [ 1 − Y ^ i j ∣ Y i j = 0 ] \operatorname{specificity}_j = \mathbb{E}[1 - \widehat{Y}_{ij} \mid Y_{ij} = 0] specificity j = E [ 1 − Y ij ∣ Y ij = 0 ] form the skew-invariant basis.
In the Calibration Framework, α j = E [ ( 1 − Y ^ i j ) 2 ∣ Y i j = 1 ] \alpha_j = \mathbb{E}[(1 - \widehat{Y}_{ij})^2 \mid Y_{ij} = 1] α j = E [( 1 − Y ij ) 2 ∣ Y ij = 1 ] (miss-calibration of positives) and β j = E [ Y ^ i j 2 ∣ Y i j = 0 ] \beta_j = \mathbb{E}[\widehat{Y}_{ij}^2 \mid Y_{ij} = 0] β j = E [ Y ij 2 ∣ Y ij = 0 ] (false-alarm calibration) form the per-class analogue (Theorem 47 ).
In the Ranking Framework, n D C G j c l a s s = 1 w ( 1 ) E [ w ( rank i ( j ) ) ∣ Y i j = 1 ] \mathrm{nDCG}_j^{\mathrm{class}} = \frac{1}{w(1)}\mathbb{E}[w(\operatorname{rank}_i(j)) \mid Y_{ij} = 1] nDCG j class = w ( 1 ) 1 E [ w ( rank i ( j )) ∣ Y ij = 1 ] and A U C j = E [ f ( i , j ) ∣ Y i j = 1 ] \mathrm{AUC}_j = \mathbb{E}[f(i,j) \mid Y_{ij} = 1] AUC j = E [ f ( i , j ) ∣ Y ij = 1 ] , with f ( i , j ) f(i,j) f ( i , j ) the share of negatives of class j j j that example i i i outranks, are conditional means of rank-based quality over positive examples (Propositions 68 and 77).
Conditioning on the appropriate outcome (Y i j = 1 Y_{ij} = 1 Y ij = 1 or Y i j = 0 Y_{ij} = 0 Y ij = 0 ) removes explicit weighting by the class prevalence π j = P ( Y i j = 1 ) \pi_j = P(Y_{ij} = 1) π j = P ( Y ij = 1 ) ; invariance additionally requires the relevant conditional cell-quality distribution to remain unchanged under the skew transformation. Any measure that can be expressed as E [ q ∣ Y i j = 1 ] \mathbb{E}[q \mid Y_{ij} = 1] E [ q ∣ Y ij = 1 ] (or E [ q ∣ Y i j = 0 ] \mathbb{E}[q \mid Y_{ij} = 0] E [ q ∣ Y ij = 0 ] ), where q q q depends on the rest of the data only through the within-class score distributions, is skew-invariant under independent rescaling of positive and negative counts. The condition on q q q is needed: the raw mean column rank of the positives is a conditional mean over the positives but depends on the number of negatives, and is skew-sensitive. Likewise, for per-class nDCG a change in the joint row-score configuration can alter rank i ( j ) \operatorname{rank}_i(j) rank i ( j ) even when the positive- and negative-conditional distributions of each column’s individual scores are unchanged; preservation of F j + F_j^+ F j + and F j − F_j^- F j − alone is therefore not the relevant invariance hypothesis. The three frameworks share this structure because in each of them the per-class building blocks are cell values conditioned on the class label: coincidence operators give TP (conditioned on Y i j = 1 Y_{ij} = 1 Y ij = 1 ) and TN (on Y i j = 0 Y_{ij} = 0 Y ij = 0 ); deviation operators give the calibration errors α j \alpha_j α j (on Y i j = 1 Y_{ij} = 1 Y ij = 1 ) and β j \beta_j β j (on Y i j = 0 Y_{ij}=0 Y ij = 0 ); positional operators give a weighted gain that is non-zero only on the positives. The template is therefore a pair of class-conditional means, over the positives and over the negatives, with the ranking measures using only the positive side.
Pooled aggregates reintroduce skew sensitivity, though not always for the same reason. For the ratio measures of the Discrimination Framework and for micro-AUC, the pooled value is a weighted mean of per-class values with prevalence-dependent weights. For the Brier score, micro = macro, so the pooled value is the plain average of the per-class scores, each of which is itself skew-sensitive (B S j = π j α j + ( 1 − π j ) β j \mathrm{BS}_j=\pi_j\alpha_j+(1-\pi_j)\beta_j BS j = π j α j + ( 1 − π j ) β j ). The common remedy across all three paradigms is to aggregate the per-class skew-invariant building blocks (r e c j \mathrm{rec}_j rec j , s p e c j \mathrm{spec}_j spec j , α j \alpha_j α j , β j \beta_j β j , A U C j \mathrm{AUC}_j AUC j , n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class ) and then average, ensuring that each class contributes equally regardless of its frequency; for the Brier score this means averaging α j \alpha_j α j and β j \beta_j β j , not B S j \mathrm{BS}_j BS j .
Example 6 (Running example: a change of prevalence). Double the positive examples of A in Example 1 under the per-class convention, so that π A \pi_A π A rises from 1 / 2 1/2 1/2 to 2 / 3 2/3 2/3 and nothing else changes:
Table 5.
Every class-conditional mean stays fixed; every summary that weights by prevalence moves. Duplicating the two examples in which A A A is relevant, as whole rows, is a different operation: it also changes the other classes (A U C C \mathrm{AUC}_C AUC C from 2 / 3 2/3 2/3 to 3 / 4 3/4 3/4 , n D C G B c l a s s \mathrm{nDCG}^{\mathrm{class}}_B nDCG B class from 0.587 to 0.598), the coupling between classes seen in Experiment 3.
Experiments
The seven experiments below illustrate the core theoretical claims of the paper. Each uses at least one of the multilabel benchmark datasets described in § 8.1 , and each is designed to produce a qualitative demonstration rather than a performance benchmark: the goal is to show that the structural properties derived in § 5 –§ 7 hold empirically on real data and are not artefacts of the algebraic framework alone.
The first four experiments share the Emotions dataset, whose six human-interpretable labels make individual classifier behaviour easy to inspect directly. Experiments 1–3 establish, in turn, that the three frameworks disagree on classifier ranking, that they differ in whether their own aggregation levels collapse to one scalar, and that the skew-invariant per-class measures stay flat under a prevalence sweep while the corresponding skew-sensitive aggregates do not. Experiment 4 then shows that an order-preserving distortion of a classifier’s own probabilities is invisible to every ranking measure while moving its Brier score and its MAE in opposite directions. The remaining three experiments mostly use Yeast (Experiment 6 uses it for panel (a) and Emotions for panel (b)), whose fourteen labels support a finer-grained dissection: Experiments 5 and 6 decompose the Brier score along the class axis and the example axis respectively, and Experiment 7 closes the section by comparing the two Ranking Framework measures directly on the same labels, showing where they disagree.
Experimental Setup
We use two multilabel benchmark datasets drawn from the MULAN collection (Tsoumakas et al., 2011), retrieved via OpenML (Vanschoren et al., 2014). The Emotions dataset (Trohidis et al., 2008) consists of 593 music clips described by 72 spectral features; each clip is annotated with one or more of six emotional categories (amazed-surprised, happy-pleased, relaxing-calm, quiet-still, sad-lonely, angry-aggressive), giving a label-set density of approximately 0.31. The Yeast dataset (Elisseeff and Weston, 2002) consists of 2417 yeast genes described by 103 expression-profile features; each gene is annotated with one or more of 14 functional classes, with label-set density approximately 0.30. Emotions provides a small, human-interpretable label space suitable for visualising individual class behaviour; Yeast provides a richer label space (m = 14 m = 14 m = 14 ) that yields a wider range of per-example label cardinalities. Each dataset is split 70/30 into training and test sets using a fixed random seed.
Five classifiers are trained with the binary relevance strategy (Tsoumakas and Katakis, 2007) (one binary sub-classifier per label): Logistic Regression (LR ), Random Forest (RF ), Gradient Boosted Trees (GBT ), an uncalibrated Multi-Layer Perceptron (MLP (uncal.) ), and the same MLP with sigmoid Platt scaling applied via cross-validation (MLP (cal.) ). The last pair is deliberately included to isolate the effect of post-hoc calibration while keeping the base classifier constant. All feature vectors are min-max normalised to [ 0 , 1 ] [0, 1] [ 0 , 1 ] . Hard binary predictions are obtained by thresholding soft probabilities at 0.5. Ranks are computed with ties broken by label index (NumPy’s argsort), and AUC counts tied positive–negative pairs as 1 2 \frac{1}{2} 2 1 (scikit-learn). Ties arise mainly in the random forest, whose scores tie within 36 of the 178 Emotions test rows and 524 of the 726 Yeast test rows. GBT repeats values within a few columns (1 of 6 labels in Emotions, 6 of 14 in Yeast), and LR and the two MLPs have no ties. The tie rule therefore matters for RF and, in macro-AUC on Yeast, for GBT, which falls from 0.7050 to 0.6966 if ties count 0 and so changes places with the uncalibrated MLP (0.6993); for the other classifiers the conventions agree (Experiment 1 and the files tie_sensitivity_emotions.csv and tie_sensitivity_yeast.csv in the repository).
Framework Disagreement in Classifier Ranking
The first experiment asks whether the choice of evaluation framework alters the relative ranking of classifiers. We compute four scalar summary measures on the Emotions test set: macro-F 1 F_1 F 1 (Discrimination Framework), 1 − B S m a c r o 1 - \mathrm{BS}^{\mathrm{macro}} 1 − BS macro (Calibration Framework, inverted so that higher is better), macro-n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class (Ranking Framework, row-dependent), and macro-AUC (Ranking Framework, column-dependent). Figure 2 shows the resulting scores and rank heatmap.
Figure 2. Experiment 1 (Emotions). (a) Scores across frameworks for five classifiers. (b) Rank heatmap (green = 1st, red = last). No two metrics produce the same classifier ordering.
The heatmap reveals that no two metrics produce the same classifier ordering. Kendall’s τ \tau τ values between the four metric-induced rankings are all strictly less than 1, confirming pairwise disagreement. MLP (cal.) achieves the best Brier score, as expected, since Platt scaling directly optimises calibration, and is also first in F 1 F_1 F 1 and macro-AUC, but it is second in nDCG (0.7339 against 0.7343 for the uncalibrated MLP). Below the top the orderings diverge strongly: RF is second in Brier but fourth in macro-AUC and, with the implementation’s tie rule, last in nDCG; GBT is last in Brier but third in macro-AUC; LR is last in F 1 F_1 F 1 and in macro-AUC but third in nDCG. This disagreement is expected: the three frameworks measure distinct classifier behaviours (coincidence, deviation, and positional quality, § 7 ), and a classifier that is good in one dimension need not be good in another; the theorems explain why the measures differ, not which classifier each one favours. Two cautions apply. The random forest’s probabilities contain ties (a tied pair in 36 of the 178 Emotions test rows, and repeated values in every RF column), and the position of RF in nDCG depends on how they are broken: its macro-nDCGc l a s s ^{\mathrm{class}} class is 0.7158 with the implementation’s rule, 0.7211 under the min-rank convention of Definition 55 (where GBT, not RF, is last) and 0.7172 under random tie-breaking; counting ties as 0 instead of 1 2 \frac{1}{2} 2 1 in the concordance fraction lowers its macro-AUC from 0.8367 to 0.8319. And the margins in this single split are small (nDCG 0.7339 against 0.7343), so only the disagreement among the four orderings, not the individual positions, should be read from the figure. Practitioners who report only one metric risk drawing conclusions that would be reversed by an equally legitimate choice from a different framework.
Calibration Collapse vs. Ranking Non-Collapse
The collapse theorem and its ranking-side failure make a joint prediction, tested here on the Emotions dataset: B S m i c r o = B S m a c r o = B S e x e m p l a r \mathrm{BS}^{\mathrm{micro}} = \mathrm{BS}^{\mathrm{macro}} = \mathrm{BS}^{\mathrm{exemplar}} BS micro = BS macro = BS exemplar (Theorem 20 ) while normalised nDCG does not collapse (Theorem 60 ). Figure 3 confirms both.
Figure 3. Experiment 2 (Emotions). (a) Brier score aggregations (micro, macro, exemplar) are numerically identical, confirming the collapse theorem. (b) nDCG aggregations: macro-class nDCG is well separated from the other two (≈ 0.72 \approx0.72 ≈ 0.72 against ≈ 0.88 \approx0.88 ≈ 0.88 –0.90 0.90 0.90 ) and micro and exemplar differ slightly, confirming non-collapse for normalised ranking measures.
Panel (a) shows the three Brier aggregations for all five classifiers. For each classifier, the three bars are numerically identical to at least four decimal places, with max absolute difference below 10 − 14 10^{-14} 1 0 − 14 , within floating-point rounding. Panel (b) shows the three nDCG aggregations (micro, macro-class, exemplar-mean). Macro-class nDCG is well separated from the other two for every classifier (≈ 0.72 \approx0.72 ≈ 0.72 against ≈ 0.88 \approx0.88 ≈ 0.88 –0.90 0.90 0.90 ), while micro and exemplar nDCG differ by about 0.01. Macro-class nDCG is consistently lower than exemplar nDCG, largely because the class ideal I D C G j c l a s s = π j w ( 1 ) \mathrm{IDCG}^{\mathrm{class}}_j = \pi_j w(1) IDCG j class = π j w ( 1 ) cannot be attained for all labels at once: an oracle that scores every label by its truth, with ties among the relevant labels of an example broken uniformly at random (mean over 20 draws), reaches a macro n D C G c l a s s nDCG^{\mathrm{class}} n D C G class of only 0.81 on this test set. Under the min-rank convention of Definition 55 the same tied oracle would score 1 on every represented class (§ 6.5 ). The splitting of the ranking aggregations comes from the non-uniform IDCG normalisation, not from class imbalance alone, as shown in Theorem 60 (b)–(d).
The contrast between the two panels illustrates the practical consequence of the collapse theorem: for calibration measures, choosing between micro, macro, and exemplar aggregation is a matter of convention. For ranking measures, the choice changes the value and should be reported explicitly.
Skew-Invariance Under Prevalence Variation
Propositions 68 and 77 establish that n D C G j c l a s s nDCG^{\mathrm{class}}_j n D C G j class and A U C j AUC_j A U C j are skew-invariant, while Corollary 50 shows that B S m a c r o BS^{\mathrm{macro}} B S macro is not and Proposition 77(c) that micro-AUC is not in general.
Figure 4 tests these claims by varying the prevalence of the first Emotions label (amazed-surprised, π = 0.281 \pi= 0.281 π = 0.281 in the original test set) via subsampling of the negative examples, using logistic regression (LR) trained once on the unchanged training set. Only negatives are removed, so the achievable prevalence starts at the natural value (here 0.281): the three lowest of the ten targets 0.10, . . . , 0.80 all give π = 0.281 \pi= 0.281 π = 0.281 , and the sweep covers 0.28 to 0.81. At each target prevalence, the procedure is repeated ten times with different random subsamples; solid curves show means and shaded bands show ± 1 \pm1 ± 1 standard deviation.
Figure 4. Experiment 3 (Emotions). Varying the prevalence of the first label from its natural 0.28 up to 0.81 (by removing negatives). (a) Per-class n D C G j c l a s s nDCG_{j}^{\mathrm{class}} n D C G j class is exactly constant in this subsampling design, whereas empirical per-class A U C j AUC_j A U C j remains approximately stable, with finite-sample variation (left axis), while the raw mean rank M 0 M_0 M 0 (right axis) drifts strongly; A U C 0 AUC_0 A U C 0 is its affine normalisation (Proposition 73 ). (b) Solid: micro-AUC and macro-BS under row subsampling, which couples the classes. Dashed: per-class-summary convention (micro-AUC by scaling the positives of class 0, macro-BS by the mixture formula, the other classes held fixed). Shaded bands: ± 1 \pm1 ± 1 s.d. over 10 subsamples.
Panel (a) shows A U C 0 AUC_0 A U C 0 (green) and n D C G 0 c l a s s nDCG^{\mathrm{class}}_0 n D C G 0 class (blue), both trained on the unchanged training set and evaluated on the skewed cal test. A U C 0 \mathrm{AUC}_0 AUC 0 varies by 0.025 over the sweep, with a standard deviation up to 0.054 at π = 0.8 \pi= 0.8 π = 0.8 . Superimposed on a second axis is the raw mean rank M 0 M_0 M 0 of the positive examples (purple, right axis of panel (a)), which is the unnormalised fixed-weight column operator of Proposition 73 . It drifts strongly with prevalence (as negatives are removed, the positives rise in rank), yet its affine Wilcoxon–Mann–Whitney normalisation, A U C 0 \mathrm{AUC}_0 AUC 0 , is approximately stable across this finite-subsample sweep (with the nonzero variation and standard deviation reported above). Under ideal per-class reweightings that preserve the positive and negative conditional score distributions, AUC is exactly skew-invariant; finite negative subsampling only approximates that invariance. This distinction parallels the contrast between normalised and raw DCG. Ranking itself is not intrinsically prevalence-free. We verified the identity A U C 0 = ( n − ( n 0 + − 1 ) / 2 − M 0 ) / n 0 − \mathrm{AUC}_0 = (n - (n_0^+ - 1)/2 - M_0)/n_0^- AUC 0 = ( n − ( n 0 + − 1 ) /2 − M 0 ) / n 0 − numerically at every subsample; the maximum deviation over all trials was 2.2 × 10 − 16 2.2 \times10^{-16} 2.2 × 1 0 − 16 , at the level of floating-point round-off. The same raw-to-normalised relationship holds on the row axis: the flat n D C G 0 c l a s s \mathrm{nDCG}^{\mathrm{class}}_0 nDCG 0 class curve is the ideal-DCG normalisation of a raw DCG that itself scales as 1 / n 1/n 1/ n with prevalence, the normalisation cancelling the n n n , since n D C G j c l a s s = 1 w ( 1 ) E [ w ( rank i ( j ) ) ∣ Y i j = 1 ] \mathrm{nDCG}^{\mathrm{class}}_j = \frac{1}{w(1)} \mathbb{E}[w(\operatorname{rank}_i(j)) \mid Y_{ij} = 1] nDCG j class = w ( 1 ) 1 E [ w ( rank i ( j )) ∣ Y ij = 1 ] does not depend on n n n . Panel (a) thus exhibits both positional axes obeying the same pattern: a fixed-weight operator followed by a normalisation that removes the prevalence dependence. Panel (b) shows micro-AUC (left axis) and macro-BS (right axis) under two schemes. The solid curves are the row subsampling just described: micro-AUC falls from 0.814 to 0.783 (with a small initial rise near π = 0.33 \pi= 0.33 π = 0.33 ) and macro-BS rises from 0.142 to about 0.152, peaking near π = 0.72 \pi= 0.72 π = 0.72 . These curves are confounded: removing negative rows of class 0 also removes positives and changes the negatives of the other classes, so the other classes’ summaries move too (in a separate diagnostic with 100 subsamples at π ≈ 0.8 \pi\approx0.8 π ≈ 0.8 , the numbers of positives of the other five classes fall from (55, 84, 34, 50, 60) to (22, 11, 3, 5, 33) and their AUCs change, for instance 0.63 → \to → 0.55 and 0.91 → \to → 0.84). The dashed curves use the per-class-summary convention of Definition 46, in which only the counts of class 0 change and the other classes’ summaries are held at their full-test-set values. Scaling the negatives leaves micro-AUC exactly unchanged, since n 0 + n_0^+ n 0 + and every A U C l \mathrm{AUC}_l AUC l are untouched; scaling the positives moves it towards A U C 0 = 0.83 \mathrm{AUC}_0 = 0.83 AUC 0 = 0.83 , from 0.814 to 0.825 over the same prevalence range (Proposition 77(c)); and macro-BS follows the mixture B S 0 = π α 0 + ( 1 − π ) β 0 \mathrm{BS}_0 = \pi\alpha_0 + (1 - \pi)\beta_0 BS 0 = π α 0 + ( 1 − π ) β 0 of Proposition 45 with the other classes fixed, rising from 0.142 to 0.164. The rise of B S 0 \mathrm{BS}_0 BS 0 is not explained by U N C 0 = π ( 1 − π ) \mathrm{UNC}_0 = \pi(1 - \pi) UNC 0 = π ( 1 − π ) , which peaks at π = 0.5 \pi= 0.5 π = 0.5 : it is the weight of the mixture moving to α 0 \alpha_0 α 0 , consistent with α 0 > β 0 \alpha_0 > \beta_0 α 0 > β 0 (0.32 against 0.07 on the full test set). The falling micro-AUC of the subsampling curve is therefore a side effect of the coupling between classes, not the effect described by Proposition 77(c). This experiment illustrates the mixture mechanism for the Brier score and the stability of ( A U C j , n D C G j c l a s s ) (\mathrm{AUC}_j,\mathrm{nDCG}^{\mathrm{class}}_j) ( AUC j , nDCG j class ) under subsampling, and shows that row subsampling couples the classes.
Properness, or the Limits of Order-Based Evaluation
Proposition 65 establishes weak properness for the raw positional operator among distinct-entry reports when the true marginals are also distinct. With tied reports scored by min-rank, weak properness can fail, although constant discounts are an exception. Remark 66 and Proposition 79 show that normalised nDCG and AUC are not universally proper. Nevertheless, any strictly order-preserving distortion of predicted probabilities leaves their re alised rankings, and hence their nDCG and AUC values, unchanged; this demonstrates non-strictness, not universal properness. Theorem 39 states the opposite for the Brier score, and Remark 43 that MAE, despite belonging to the same structural family, is not proper at all. This experiment shows all three claims on the same real predictions, using the LR classifier’s predicted probabilities P ^ \hat{P} P ^ on the Emotions test set, and is the last of the Emotions-based experiments.
For a sweep of exponents γ ∈ [ 0.2 , 5 ] \gamma\in[0.2, 5] γ ∈ [ 0.2 , 5 ] , every entry is distorted as P ^ ↦ P ^ γ \hat{P} \mapsto\hat{P}^{\gamma} P ^ ↦ P ^ γ . Since x ↦ x γ x \mapsto x^{\gamma} x ↦ x γ is strictly increasing on [ 0 , 1 ] [0, 1] [ 0 , 1 ] for every γ > 0 \gamma> 0 γ > 0 , this transformation preserves the rank of every entry within any row and within any column, so it is an order-preserving distortion of the kind Propositions 65 and 79 describe; for every 0 < p < 1 0 < p < 1 0 < p < 1 , exponents 0 < γ < 1 0 < \gamma< 1 0 < γ < 1 raise the score (p γ > p p^{\gamma} > p p γ > p ), whereas γ > 1 \gamma> 1 γ > 1 lower it (p γ < p p^{\gamma} < p p γ < p ). This one-sided power map does not generally move all scores toward 0.5 or sharpen them toward both endpoints. The choice γ = 1 \gamma= 1 γ = 1 recovers the untouched, classifier-reported probabilities.
Figure 5 confirms the three claims. Panel (a): macro-AUC and macro-n D C G c l a s s nDCG^{\mathrm{class}} n D C G class are constant across the entire sweep to machine precision (a spread of exactly 0.00 over 25 values of γ \gamma γ ): rank-based measures cannot distinguish the untouched classifier from any monotonic distortion of it, whether overconfident or not. Panel (b): the macro-Brier score traces an asymmetric curve with its minimum at γ = 0.93 \gamma= 0.93 γ = 0.93 , value 0.1413, indistinguishable in practice from the untouched γ = 1 \gamma= 1 γ = 1 value of 0.1416. This is consistent with Theorem 39 given that LR’s raw output is, as expected, close to but not perfectly calibrated. Panel (c): macro-MAE behaves differently. Its minimum sits at γ = 2.16 \gamma= 2.16 γ = 2.16 , value 0.2618, a substantial improvement over the untouched value of 0.2895 at γ = 1 \gamma= 1 γ = 1 . MAE favours the downwardshifted scores produced by this particular power transformation, rather than necessarily sharper or more extreme probabilities. This is qualitatively compatible with the conditional-median target of MAE (Remark 43 ), but the experiment does not establish that the raw scores are the true conditional probabilities. On a single real dataset, the three panels show that ranking measures cannot see calibration at all, the Brier score rewards honesty, and structural membership in the Calibration Framework (MAE satisfies the same aggregation collapse as Brier) is not sufficient for properness.
Figure 5. Experiment 4 (Emotions, classifier LR). Predicted probabilities distorted as P ^ → P ^ γ \widehat{P} \to\widehat{P}^{\gamma} P → P γ , an order-preserving transformation for every γ > 0 \gamma> 0 γ > 0 . (a) macro-AUC and macro-nDCGc l a s s ^{\mathrm{class}} class are exactly flat (spread 0.00) across the entire sweep. (b) Macro-Brier score is minimised near γ = 1 \gamma= 1 γ = 1 (the untouched probabilities), consistent with a strictly proper scoring rule applied to nearly calibrated scores. (c) Macro-MAE is minimised at γ = 2.16 \gamma= 2.16 γ = 2.16 , well away from γ = 1 \gamma= 1 γ = 1 , consistent with Remark 43 : MAE is not a proper scoring rule, and here it favours the downward-shifted scores of the power map (P ^ γ < P ^ \widehat{P}^{\gamma} < \widehat{P} P γ < P for γ > 1 \gamma> 1 γ > 1 ).
Per-Class Murphy Decomposition
The next two experiments move to Yeast for a finer-grained diagnosis, starting with the column axis: Theorem 24 decomposes the per-class Brier score exactly as B S j = U N C j − R E S j + R E L j \mathrm{BS}_{j} = \mathrm{UNC}_{j} - \mathrm{RES}_{j} + \mathrm{REL}_{j} BS j = UNC j − RES j + REL j when f j f_{j} f j is evaluated at its exact argument; in practice R E L j \mathrm{REL}_{j} REL j and R E S j \mathrm{RES}_{j} RES j are computed from 10 equal-width calibration bins, so the figure displays the binned decomposition of Proposition 26 (B S ^ j = U N C j − R E S ^ j + R E L ^ j \widehat{\mathrm{BS}}_{j} = \mathrm{UNC}_{j} - \widehat{\mathrm{RES}}_{j} + \widehat{\mathrm{REL}}_{j} BS j = UNC j − RES j + REL j , exactly, for the discretised predictor) rather than Theorem 24 ’s pointwise version, which degenerates to R E S j = U N C j \mathrm{RES}_{j} = \mathrm{UNC}_{j} RES j = UNC j , R E L j = B S j \mathrm{REL}_{j} = \mathrm{BS}_{j} REL j = BS j on continuous scores with no repeated values. Figure 6 displays this binned decomposition for all five classifiers across the 14 Yeast labels.
Figure 6. Experiment 5 (Yeast, 14 labels, 5 classifiers). Per-class binned Murphy decomposition B S ^ j = U N C j − R E S ^ j + R E L ^ j \widehat{\mathrm{BS}}_j = \mathrm{UNC}_j - \widehat{\mathrm{RES}}_j + \widehat{\mathrm{REL}}_j BS j = UNC j − RES j + REL j . Top row: R E L ^ j \widehat{\mathrm{REL}}_j REL j (miscalibration, Reds) and R E S ^ j \widehat{\mathrm{RES}}_j RES j (resolution, Greens) heatmaps. Bottom: U N C j \mathrm{UNC}_j UNC j bar chart, identical across classifiers (data property only).
The figure has three panels. The top-left heatmap shows R E L ^ j \widehat{\mathrm{REL}}_{j} REL j (binned reliability, lower is better) for each classifier–label pair; darker red indicates higher miscalibration. The top-right heatmap shows R E S ^ j \widehat{\mathrm{RES}}_{j} RES j (binned resolution, higher is better); darker green indicates more discriminative predictions. The bottom bar chart shows U N C j = π j ( 1 − π j ) \mathrm{UNC}_{j} = \pi_{j}(1 - \pi_{j}) UNC j = π j ( 1 − π j ) , which is identical across all classifiers by definition (it is a property of the data, not the model) and is annotated accordingly.
Several patterns are consistent with theory. First, MLP (cal.) shows lower R E L j \mathrm{REL}_{j} REL j than MLP (uncal.) on 13 of the 14 labels, consistent with Platt scaling reducing miscalibration as measured by the REL term while leaving U N C j \mathrm{UNC}_j UNC j unchanged. Second, GBT and RF exhibit higher R E S j \mathrm{RES}_j RES j than LR on 13 of the 14 labels, reflecting their stronger discriminative power. Third, R E S j \mathrm{RES}_j RES j is strongly associated with U N C j \mathrm{UNC}_j UNC j across labels (Spearman 0.87–0.91 for every classifier), as expected since R E S j ≤ U N C j \mathrm{RES}_j \le\mathrm{UNC}_j RES j ≤ UNC j ; R E L j \mathrm{REL}_j REL j shows no consistent association with U N C j \mathrm{UNC}_j UNC j (Spearman between −0.52 and 0.71 depending on the classifier). The dependence of RES on prevalence is consistent with Corollary 53 (b); a comparison across labels with different prevalences is, however, not a controlled test of that corollary; such a test would vary the prevalence of a single label, as Experiment 3 does for the Brier mixture, and is not carried out for R E S j \mathrm{RES}_j RES j here.
The binned decomposition depends on the number of bins and, on continuous scores, replaces each score by a bin mean. tbl:5 repeats it with the isotonic calibration map of Proposition 28 , which needs no bins, on the same classifiers and labels. The identity B S j = U N C j − R E S j i s o + R E L j i s o \mathrm{BS}_j = \mathrm{UNC}_j - \mathrm{RES}^{\mathrm{iso}}_j + \mathrm{REL}^{\mathrm{iso}}_j BS j = UNC j − RES j iso + REL j iso holds to rounding error on all 70 classifier–label pairs, with B S j \mathrm{BS}_j BS j the Brier score of the original scores, not of a discretised predictor. The two resolution terms are almost the same ranking of the labels (Spearman 0.99–1.00 for every classifier), and the isotonic one is somewhat larger on average. The reliability terms differ more: the isotonic R E L j i s o \mathrm{REL}^{\mathrm{iso}}_j REL j iso is larger than the binned one for every classifier, because the binned predictor absorbs the within-bin variation of the scores (Remark 27 ), and the rank correlation across labels falls to 0.22–0.92 depending on the classifier. Two of the observations above change with the estimator. MLP (cal.) has lower reliability error than MLP (uncal.) on 12 of the 14 labels instead of 13, and GBT has higher resolution than LR on 10 of the 14 labels instead
caption: tbl:5 : Experiment 5 (Yeast), binned (10 equal-width bins) and isotonic Murphy terms, averaged over the 14 labels. The isotonic terms satisfy B S j = U N C j − R E S j i s o + R E L j i s o \mathrm{BS}_{j} = \mathrm{UNC}_{j} - \mathrm{RES}_{j}^{\mathrm{iso}} + \mathrm{REL}_{j}^{\mathrm{iso}} BS j = UNC j − RES j iso + REL j iso for the original scores.
:::
of 13 (RF, 13 of 14 in both). The mean reliability error of RF and GBT is close under both estimators and swaps order (0.0039 and 0.0040 binned, 0.0067 and 0.0063 isotonic), so we draw no conclusion about their order. The two main findings, that Platt scaling lowers the reliability error and that the tree ensembles resolve more than the logistic regression, hold under both estimators. The correlation of R E S j i s o \mathrm{RES}_{j}^{\mathrm{iso}} RES j iso with U N C j \mathrm{UNC}_{j} UNC j across labels is 0.88–0.92, as for the binned terms, and that of R E L j i s o \mathrm{REL}_{j}^{\mathrm{iso}} REL j iso with U N C j \mathrm{UNC}_{j} UNC j is positive (0.38–0.87), whereas it had no consistent sign for the binned terms.
Per-Exemplar Brier Decomposition
Section 5.5 introduces the row-axis counterpart, which decomposes the per-exemplar Brier score B S i e x a m p l e = ( k i / m ) α i + ( ( m − k i ) / m ) β i \mathrm{BS}_{i}^{\mathrm{example}} = (k_{i}/m)\,\alpha_{i} + ((m-k_{i})/m)\,\beta_{i} BS i example = ( k i / m ) α i + (( m − k i ) / m ) β i where α i \alpha_{i} α i is the miss-calibration on example i i i ’s positive labels and β i \beta_{i} β i is the false-alarm calibration on its negative labels (Proposition 33). Figure 7 provides two complementary views of this decomposition.
Figure 7. Experiment 6. (a) Mean α i \alpha_i α i (solid) and β i \beta_i β i (dashed) by label cardinality k i k_i k i on Yeast; only bins with ≥ 15 \ge15 ≥ 15 examples are shown. (b) Empirical CDFs of B S i e x a m p l e \mathrm{BS}_i^{\mathrm{example}} BS i example on Emotions; dotted verticals mark each classifier’s mean. The means are close (0.131–0.142) but the CDFs differ.
Panel (a) plots α i \alpha_{i} α i and β i \beta_{i} β i by label cardinality on Yeast, whose m = 14 m = 14 m = 14 labels yield cardinalities k i k_{i} k i from 2 to 8 with adequate support (only bins with ≥ 15 \ge15 ≥ 15 examples are shown; the sizes are 121, 33, 321, 52, 127, 31, 24 for k i = 2 , … , 8 k_{i} = 2, \ldots, 8 k i = 2 , … , 8 ). For each classifier, solid lines trace mean α i \alpha_{i} α i and dashed lines trace mean β i \beta_{i} β i as a function of k i k_{i} k i . Two patterns emerge. First, β i \beta_{i} β i (false-alarm calibration) is low and broadly decreases with k i k_{i} k i for all classifiers, from about 0.13–0.15 at k i = 2 k_{i} = 2 k i = 2 to about 0.05 at k i = 8 k_{i} = 8 k i = 8 : examples with many relevant labels have fewer negative labels, and the classifier assigns them low probability. Second, α i \alpha_{i} α i (miss-calibration on positives) is higher than β i \beta_{i} β i at every cardinality and is not monotone in k i k_{i} k i : for LR it is 0.38 at k i = 2 k_{i} = 2 k i = 2 , falls to 0.21 at k i = 4 k_{i} = 4 k i = 4 , and rises again to 0.41 at k i = 8 k_{i} = 8 k i = 8 . The two classifiers that differ only in post-hoc calibration, MLP (cal.) and MLP (uncal.), differ mainly in β i \beta_{i} β i : Platt scaling lowers β i \beta_{i} β i at every cardinality (for example 0.134 against 0.147 at k i = 2 k_{i} = 2 k i = 2 ), while α i \alpha_{i} α i is slightly higher after calibration at six of the seven cardinalities (for example 0.379 against 0.363 at k i = 2 k_{i} = 2 k i = 2 ), so the effect of calibration at the exemplar level is a trade between the positive and the negative side.
Panel (b) shows empirical CDFs of B S i e x a m p l e \mathrm{BS}_{i}^{\mathrm{example}} BS i example on Emotions. The collapse theorem (Theorem 20 ) guarantees that 1 n ∑ i B S i e x a m p l e \frac{1}{n}\sum_{i}\mathrm{BS}_{i}^{\mathrm{example}} n 1 ∑ i BS i example is the same for all aggregation axes. Panel (b) tests whether this equality of means conceals distributional heterogeneity. The five empirical CDFs of B S i e x a m p l e \mathrm{BS}_i^{\mathrm{example}} BS i example are shown with vertical dotted lines marking the classifiers’ mean values.
The means are close across classifiers (0.131 to 0.142; the collapse theorem concerns the aggregations of a single classifier, not differences between classifiers), but the CDFs differ. GBT has both the most mass near zero (5% quantile 0.008, against 0.040 for LR and RF) and the heaviest tail (95% quantile 0.33, maximum 0.54); RF has the smallest maximum (0.38) and MLP (cal.) the smallest median (0.113, against 0.134 for LR). This divergence is invisible to any scalar Brier-score summary and can only be revealed by examining the row-level distribution of B S i e x a m p l e \mathrm{BS}_i^{\mathrm{example}} BS i example . The experiment therefore provides empirical support for the claim in Remark 36 : the row-level view, through the per-example errors ( α i , β i ) (\alpha_i,\beta_i) ( α i , β i ) and the distribution of B S i e x a m p l e \mathrm{BS}_i^{\mathrm{example}} BS i example , exposes per-example calibration structure that the per-class summaries and the scalar Brier score discard.
Per-Class nDCG as a Novel Measure
Proposition 62 and Theorem 74 establish that n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class and A U C j \mathrm{AUC}_j AUC j are both skew-invariant conditional means over positive examples, but along orthogonal axes: n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class evaluates the row rank of label j j j within each positive example’s predicted label vector, while A U C j \mathrm{AUC}_j AUC j evaluates the column rank of positive examples above negatives for class j j j . This closing experiment asks whether the two measures agree in practice on the same Yeast labels just decomposed two ways above, and if not, what drives their disagreement.
Figure 8 uses the Yeast dataset (m = 14 m = 14 m = 14 labels). For all five classifiers and all 14 labels we compute A U C j \mathrm{AUC}_j AUC j and n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class .
Figure 8. Experiment 7 (Yeast, m = 14 m = 14 m = 14 ). (a) Scatter of label-mean A U C j \mathrm{AUC}_j AUC j vs. n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class , annotated with label names; light dots show per-classifier spread. Most labels fall below the diagonal y = x y = x y = x ; Class12 and Class13, the two most prevalent labels, are exceptions (n D C G j c l a s s > A U C j \mathrm{nDCG}^{\mathrm{class}}_j > \mathrm{AUC}_j nDCG j class > AUC j ). (b) Dumbbell plot for RF: circle = A U C j \mathrm{AUC}_j AUC j (column-rank), diamond = n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class (row-rank), sorted by mean A U C j \mathrm{AUC}_j AUC j ; line length = row-rank deficit.
Panel (a) shows the scatter of per-label means (black dots, annotated with label names) against the diagonal y = x y = x y = x , with light per-classifier dots in the background showing within-label spread across classifiers. Twelve of the fourteen label means fall below the diagonal (A U C j > n D C G j c l a s s \mathrm{AUC}_j > \mathrm{nDCG}^{\mathrm{class}}_j AUC j > nDCG j class , mean gap 0.32 over these twelve labels and 0.26 over all fourteen), and two, Class12 and Class13, sit above it. These two are by far the most prevalent labels in the test set (π = 0.76 \pi= 0.76 π = 0.76 and 0.75 0.75 0.75 , against at most 0.45 0.45 0.45 for every other label). Across the 14 labels, n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class is almost a monotone function of prevalence (Spearman ρ = 0.97 \rho= 0.97 ρ = 0.97 ), whereas A U C j \mathrm{AUC}_j AUC j is not (ρ = 0.22 \rho= 0.22 ρ = 0.22 ), so the gap A U C j − n D C G j c l a s s \mathrm{AUC}_j - \mathrm{nDCG}^{\mathrm{class}}_j AUC j − nDCG j class decreases with prevalence (ρ = − 0.78 \rho= -0.78 ρ = − 0.78 ). At the other end, the four least prevalent labels (Class14, Class9, Class10, Class11, π ≤ 0.14 \pi\le0.14 π ≤ 0.14 ) have n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class between 0.27 0.27 0.27 and 0.30 0.30 0.30 , below the 0.40 0.40 0.40 that a uniformly random ordering of 14 labels would give (§ 6.5 ; the ceiling for an oracle that scores every label by its truth, with random tie-breaking among relevant labels as above, is 0.61 0.61 0.61 on Yeast, itself increasing with prevalence, Spearman 0.49 0.49 0.49 , and dividing n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class by its oracle value leaves the Spearman correlation with prevalence at 0.97 0.97 0.97 ), while their A U C j \mathrm{AUC}_j AUC j ranges from 0.57 0.57 0.57 to 0.71 0.71 0.71 (Class14: 0.71 0.71 0.71 and 0.27 0.27 0.27 ). Two labels with similar A U C j \mathrm{AUC}_j AUC j can therefore have very different n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class (Class2 and Class8, both ≈ 0.69 \approx0.69 ≈ 0.69 , have 0.47 0.47 0.47 and 0.35 0.35 0.35 ; Class13 and Class11, 0.65 0.65 0.65 and 0.60 0.60 0.60 , have 0.73 0.73 0.73 and 0.30 0.30 0.30 ).
Panel (b) shows a dumbbell plot for RF: each label has a circle (A U C j \mathrm{AUC}_j AUC j , column-rank) and a diamond (n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class , row-rank) connected by a line, listed by increasing A U C j \mathrm{AUC}_j AUC j averaged over the classifiers (top to bottom), so the two panels are directly comparable. The line length is the row-rank deficit A U C j − n D C G j c l a s s \mathrm{AUC}_j - \mathrm{nDCG}^{\mathrm{class}}_j AUC j − nDCG j class , which for RF is between 0.25 0.25 0.25 and 0.47 0.47 0.47 for twelve labels, largest for Class14 (0.47 0.47 0.47 ) and Class7 (0.42 0.42 0.42 ), and negative for Class12 (− 0.15 -0.15 − 0.15 ) and Class13 (− 0.09 -0.09 − 0.09 ). The ordering of labels by A U C j \mathrm{AUC}_j AUC j and by n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class differs: Class9 is last by A U C j \mathrm{AUC}_j AUC j (0.57 0.57 0.57 ) and also among the last by nDCG, but Class12 and Class13, which are 11th and 12th of 14 by A U C j \mathrm{AUC}_j AUC j , are first and second by n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class .
This is not in conflict with the skew-invariance of n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class (Proposition 68 ), which concerns a fixed classifier whose scores are unchanged when the examples of one class are replicated; the experiment compares different labels. The row rank of label j j j is decided by the competition with the other labels’ scores in the same example, and a classifier fitted to the data produces scores whose scale follows each label’s base rate: a frequent label tends to receive high scores and often tops the row, a rare one receives low scores and rarely does, whatever its separation of positives from negatives. Two interventions test this explanation directly.
The first intervention is a prior-only predictor: give label j j j the same score π j \pi_j π j (the training prevalence) for every example. Its A U C j \mathrm{AUC}_j AUC j is 0.5 0.5 0.5 by construction, so it carries the scale of the scores and nothing else. Its n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class equals one plus the number of more prevalent labels, put through the weight, so it is a monotone function of prevalence (Spearman ρ = 1.00 \rho= 1.00 ρ = 1.00 ) and averages to 0.40 0.40 0.40 , the value for a random ordering. It already tracks the classifiers: across labels, the Spearman correlation between the two is 0.97 0.97 0.97 (Pearson 0.89 0.89 0.89 , R 2 = 0.79 R^2 = 0.79 R 2 = 0.79 ), the mean absolute difference is $0.08, and the classifiers add on average only 0.04 0.04 0.04 to the scale-only baseline (0.44 0.44 0.44 against 0.40 0.40 0.40 ).
The second intervention is a column-wise normalisation: replace each column of scores by its mid-rank percentile. This order-preserving, tie-preserving transform leaves every A U C j \mathrm{AUC}_j AUC j exactly unchanged (the largest change in the run is 0 0 0 ). It places all labels on a common percentile scale and yields identical empirical percentile distributions when their scores have no ties, or tie groups of the same sizes at the same rank positions; with differing patterns of ties, mid-ranks need not produce identical distributions. The Spearman correlation of n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class with prevalence falls from 0.97 0.97 0.97 to − 0.23 -0.23 − 0.23 , and its correlation with A U C j \mathrm{AUC}_j AUC j rises from 0.30 0.30 0.30 to 0.86 0.86 0.86 (Spearman; Pearson 0.19 0.19 0.19 to 0.69 0.69 0.69 ). The normalisation is applied to each classifier and averaged over 20 random tie-breaks between labels. Both results are what the explanation predicts, and a fixed decision rule (a normalised correlation with prevalence below 0.3 in absolute value) was set before running them. Caveats: one data set, one split and five classifiers; Class14 has about a dozen test positives, so its values are noisy.
It follows that, across labels, n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class mixes the quality of the label’s score with the scale of its scores relative to those of the other labels, which is largely set by the base rate; A U C j \mathrm{AUC}_j AUC j compares only the scores of one label across examples, so it is not affected by what the other labels receive. The two measures are therefore complementary, and n D C G j c l a s s \mathrm{nDCG}^{\mathrm{class}}_j nDCG j class should not be read as a prevalence-free measure of ranking quality when comparing different labels; a complete ranking evaluation should report both.
Conclusion
This paper extended the indicator-matrix framework of Prati (2027) from discrimination to calibration and ranking, covering the three major paradigms of probabilistic multilabel evaluation within a single algebraic language.
Replacing the t t t -norm cell operator with a deviation operator ϕ ( y − y ^ ) \phi(y-\hat{y}) ϕ ( y − y ^ ) or a positional operator Y i j ⋅ w ( rank ) Y_{ij} \cdot w(\operatorname{rank}) Y ij ⋅ w ( rank ) preserves the three-aggregation architecture of the base framework (micro, macro, and exemplar) while producing qualitatively different collapse behaviour and skew-sensitivity profiles. The Calibration Framework is the only operator class whose measures collapse to the same scalar under all three aggregation schemes even after normalisation; the Discrimination Framework’s ratio measures need not collapse, and in the Ranking Framework the raw DCG and AUC aggregates collapse while the normalised nDCG and AUC scalars need not, because of heterogeneous denominators (the ideal DCG, the class sizes); equality holds only under the covariance conditions of the collapse theorems. The Calibration Framework is also the only class that supports strictly proper scoring rules, with the Savage representation providing the exact characterisation; raw positional operators can be weakly proper under the stated ranking assumptions, but normalised nDCG and AUC are not universally proper; none is strictly proper.
These structural differences have practical consequences. Reporting B S m a c r o \mathrm{BS}^{\mathrm{macro}} BS macro on an imbalanced dataset mixes calibration quality with class prevalence in a way that recall and specificity do not; the remedy is to report the class-conditional pair ( α j , β j ) (\alpha_j,\beta_j) ( α j , β j ) , which plays the role in the Calibration Framework that recall and specificity play in the Discrimination Framework and that per-class AUC and nDCG play in the Ranking Framework. The seven experiments confirmed that these theoretical distinctions are empirically visible and non-trivial: the classifier orderings induced by the three frameworks differ (for instance RF is second in Brier score but fourth in AUC); pooled aggregates drift under prevalence changes while per-class measures remain stable (macro-BS follows the mixture of α j \alpha_j α j and β j \beta_j β j , and row subsampling couples the classes); per-class nDCG tracks label prevalence across labels, which AUC does not; the scalar Brier score can conceal substantial distributional heterogeneity at the per-exemplar level; and an order-preserving distortion of a classifier’s probabilities leaves every ranking measure exactly unchanged while raising its Brier score (beyond a small neighbourhood of the original probabilities) and, in the opposite direction, lowering its MAE, which makes the properness distinction concrete.
The skew-invariant per-class building blocks in all three frameworks share the same algebraic form: each is a class-conditional expectation E [ q ∣ Y i j = c ] E[q \mid Y_{ij}=c] E [ q ∣ Y ij = c ] (over the positives for recall, per-class AUC and nDCG, and α j \alpha_j α j ; over the negatives for specificity and β j \beta_j β j ). This template explains why recall, specificity, per-class AUC, nDCGj c l a s s {}^{\mathrm{class}}_j j class , and ( α j , β j ) (\alpha_j,\beta_j) ( α j , β j ) are skew-invariant under per-class reweightings that preserve the conditional distribution of the cell quality q q q , and why their micro-averaged counterparts are not in general (§ 7 states the exact hypotheses). Skew-invariance is thus not a special property of AUC or of any particular measure, and aggregations that reintroduce prevalence as a weight are skew-sensitive in general.
Most results of Sections 5 and 6 have additionally been mechanically checked in the Lean 4 proof assistant against Mathlib, as part of the same formalization project verifying the base framework of Prati (2027). Each of the twelve theorems, propositions and corollaries of Section 6 is covered, in the form and under the conventions listed in the supplementary verification report: scores are taken distinct (so ranks are permutations), except in the hard-prediction results, which use min-rank; cell matrices range over arbitrary real matrices where the paper restricts them to the relevant cells (the nDCG impossibility argument for relevant cells only is on paper, as is the condition of Theorem 76 (d) in its relevant-cell form); replication is modelled by weights, or by integer multiples of per-class score multisets for the skew characterisation; and expected scores are taken over finite label scenarios, which the bridge to an arbitrary probability space shows to be no restriction for binary labels. The properness of AUC (Proposition 79 ) is checked under the paper’s half-credit convention for ties: the non-uniqueness, the maximality for independent labels and, for two and three examples, under any dependence (in both cases also when marginals tie), and the counterexample for every number n ≥ 4 n \ge4 n ≥ 4 of examples, with its exact difference; only the final perturbation to pairwise distinct marginals in that counterexample is on paper. The hard-prediction limit of Proposition 63 is covered in full, including the remark on a general number of predicted positives. In Section 5, the checked statements are the Brier collapse and strict properness, both Murphy decompositions, the prevalence decomposition and its corollaries, Theorem 47 in a per-class model of score multisets under integer replication, Corollary 53 (the exact form of the resolution term, the exact condition for perfect calibration at two prevalences, and the degenerate exception for REL j \operatorname{REL}_j REL j ), the Savage representation (both directions, for continuous sections), the characterisation of the proper symmetric deviation operators (Corollary 41(c): the equivalence with a symmetric weight and strictness for a positive weight, with ϕ \phi ϕ differentiable and the weight’s nonnegativity assumed rather than derived from convexity), and Remark 51. Not formalised are, in that section, the framework bridge (Theorem 18), Proposition 35, Proposition 38, the row prevalence decomposition (Proposition 33), parts (a), (b) and (d) of Corollary 41 together with the uniqueness of the squared error among power losses and the MAE item, the binned and isotonic Murphy decompositions and the QWK remark; and, throughout, the remarks other than those named above and all experiments. The formalization and the scripts and data of all experiments are available at https://github.com/rcprati/discrimination-calibration-ranking.
The Calibration and Ranking Frameworks are implemented in the aggmetrics.scores subpackage of aggmetrics (https://github.com/rcprati/aggmetrics): the Brier score and other deviation operators, the class-conditional errors, the level-set, binned and isotonic Murphy decompositions, the DCG and AUC families with explicit tie conventions, and diagnostics that report the covariance behind each gap and the pairs of labels that nDCG and AUC order in opposite ways. The subpackage reproduces the running example of this paper and is tested against scikit-learn where the conventions coincide.
Several directions remain open for future work. First, the framework currently handles binary labels (Y i j ∈ { 0 , 1 } Y_{ij} \in\{0,1\} Y ij ∈ { 0 , 1 } ) and soft probability outputs Y ^ i j ∈ [ 0 , 1 ] \hat{Y}_{ij} \in[0,1] Y ^ ij ∈ [ 0 , 1 ] ; extending it to ordinal or real-valued labels would require a generalised notion of the calibration function f j f_j f j and of the rank function. Remark 82 sketches the connection to quadratic weighted kappa for ordinal classes, but a full treatment is left for future work. Second, the Murphy decomposition derived here applies to each label independently; capturing cross-label calibration, that is, whether the joint probability vector ( Y ^ i , ⋅ ) (\hat{Y}_{i,\cdot}) ( Y ^ i , ⋅ ) matches the joint label distribution, would require multivariate proper scoring rules (Gneiting and Raftery, 2007) and a corresponding matrix-level decomposition. Third, the per-exemplar Brier decomposition (Section 5.5) and per-exemplar AUC (Definition 72) are novel measures whose properties (consistency, sample complexity, finite-sample behaviour) deserve dedicated study. Fourth, this paper focuses on the evaluation side; connecting the framework to learning , specifically to loss functions used during training, would clarify which training objectives correspond to minimising measures in each framework and what guarantees they carry for calibration, ranking, and discrimination simultaneously.
The Running Example in Full
All values below are produced by experiments/running_example.py (exact fractions where the value is rational; the nDCG family involves log 2 3 \log_2 3 log 2 3 and is given to three decimals). Labels are ordered ( A , B , C ) (A,B,C) ( A , B , C ) and examples 1–4, as in Example 1.
At threshold 0.5 0.5 0.5 the hard predictions, by rows, are ( 1 , 1 , 0 ) (1,1,0) ( 1 , 1 , 0 ) , ( 1 , 0 , 0 ) (1,0,0) ( 1 , 0 , 0 ) , ( 1 , 0 , 1 ) (1,0,1) ( 1 , 0 , 1 ) , ( 0 , 0 , 0 ) (0,0,0) ( 0 , 0 , 0 ) . The per-class counts ( t p j , f p j , f n j , t n j ) (tp_j,fp_j,fn_j,tn_j) ( t p j , f p j , f n j , t n j ) are ( 2 , 1 , 0 , 1 ) (2,1,0,1) ( 2 , 1 , 0 , 1 ) , ( 1 , 0 , 2 , 1 ) (1,0,2,1) ( 1 , 0 , 2 , 1 ) , ( 0 , 1 , 1 , 2 ) (0,1,1,2) ( 0 , 1 , 1 , 2 ) ; precision is ( 2 3 , 1 , 0 ) (\frac{2}{3},1,0) ( 3 2 , 1 , 0 ) (micro 3 5 \frac{3}{5} 5 3 , macro 5 9 \frac{5}{9} 9 5 ), and recall is ( 1 , 1 3 , 0 ) (1,\frac{1}{3},0) ( 1 , 3 1 , 0 ) (micro 1 2 \frac{1}{2} 2 1 , macro 4 9 \frac{4}{9} 9 4 ).
For calibration, the squared-deviation cells are as in Example 2, with B S j c l a s s = ( 21 80 , 13 40 , 9 40 ) \mathrm{BS}^{\mathrm{class}}_j=(\frac{21}{80},\frac{13}{40},\frac{9}{40}) BS j class = ( 80 21 , 40 13 , 40 9 ) , B S i e x a m p l e = ( 13 150 , 53 300 , 59 100 , 23 100 ) \mathrm{BS}^{\mathrm{example}}_i=(\frac{13}{150},\frac{53}{300},\frac{59}{100},\frac{23}{100}) BS i example = ( 150 13 , 300 53 , 100 59 , 100 23 ) , micro = macro = exemplar = 13 48 \frac{13}{48} 48 13 ; α = ( 1 8 , 43 100 , 9 25 ) \alpha=(\frac{1}{8},\frac{43}{100},\frac{9}{25}) α = ( 8 1 , 100 43 , 25 9 ) , β = ( 2 5 , 1 100 , 9 50 ) \beta=(\frac{2}{5},\frac{1}{100},\frac{9}{50}) β = ( 5 2 , 100 1 , 50 9 ) , π = ( 1 2 , 3 4 , 1 4 ) \pi=(\frac{1}{2},\frac{3}{4},\frac{1}{4}) π = ( 2 1 , 4 3 , 4 1 ) . The squared scores P ^ i j 2 \hat{P}_{ij}^{2} P ^ ij 2 give Brier 14039 40000 \frac{14039}{40000} 40000 14039 with all ranks unchanged. The same-Brier companion has Brier 13 48 \frac{13}{48} 48 13 , A U C j = ( 1 , 1 3 , 0 ) \mathrm{AUC}_j=(1,\frac{1}{3},0) AUC j = ( 1 , 3 1 , 0 ) , macro-AUC 4 9 \frac{4}{9} 9 4 .
The row ranks, by rows, are ( 1 , 2 , 3 ) , ( 1 , 3 , 2 ) , ( 1 , 3 , 2 ) , ( 1 , 2 , 3 ) (1,2,3),(1,3,2),(1,3,2),(1,2,3) ( 1 , 2 , 3 ) , ( 1 , 3 , 2 ) , ( 1 , 3 , 2 ) , ( 1 , 2 , 3 ) , with ideal masses I D C G i = ( 1 + w 2 , 1 + w 2 , 1 , 1 ) \mathrm{IDCG}_i=(1+w_2,1+w_2,1,1) IDCG i = ( 1 + w 2 , 1 + w 2 , 1 , 1 ) , where w 2 = 1 / log 2 3 w_2=1/\log_2 3 w 2 = 1/ log 2 3 . For raw DCG, micro = class mean = exemplar mean = $0.366$; further, n D C G j c l a s s = ( 1 , 0.587 , 0.631 ) \mathrm{nDCG}^{\mathrm{class}}_j=(1,0.587,0.631) nDCG j class = ( 1 , 0.587 , 0.631 ) (mean 0.739 0.739 0.739 ); n D C G i = ( 1 , 1 , 0.5 , 0.631 ) \mathrm{nDCG}_i=(1,1,0.5,0.631) nDCG i = ( 1 , 1 , 0.5 , 0.631 ) (mean $0.783); micro-nDCG 0.835 0.835 0.835 . The top-1 label is A A A for every example, and Precision@1 = 1 2 =\frac{1}{2} = 2 1 .
On the column axis, the concordance cells are as in Example 2, with A U C j = ( 1 2 , 1 , 2 3 ) \mathrm{AUC}_j=(\frac{1}{2},1,\frac{2}{3}) AUC j = ( 2 1 , 1 , 3 2 ) , macro 13 18 \frac{13}{18} 18 13 , micro 7 9 \frac{7}{9} 9 7 ; A U C i e x = ( 3 4 , 7 12 , 1 , 1 ) \mathrm{AUC}^{\mathrm{ex}}_i=(\frac{3}{4},\frac{7}{12},1,1) AUC i ex = ( 4 3 , 12 7 , 1 , 1 ) , mean 5 6 \frac{5}{6} 6 5 ; raw aggregate 7 18 \frac{7}{18} 18 7 ; pooled ROC-AUC over all twelve cells 43 72 \frac{43}{72} 72 43 .
Under the per-class convention with the positives of A A A doubled, the values change as follows: π A 1 2 → 2 3 \pi_A\ \frac{1}{2}\to\frac{2}{3} π A 2 1 → 3 2 , p r e c i s i o n A 2 3 → 4 5 \mathrm{precision}_A\ \frac{2}{3}\to\frac{4}{5} precision A 3 2 → 5 4 , B S A 21 80 → 13 60 \mathrm{BS}_A\ \frac{21}{80}\to\frac{13}{60} BS A 80 21 → 60 13 , micro-AUC 7 9 → 17 24 \frac{7}{9}\to\frac{17}{24} 9 7 → 24 17 ; recall, specificity, ( α A , β A ) (\alpha_A,\beta_A) ( α A , β A ) , A U C A \mathrm{AUC}_A AUC A , n D C G A c l a s s \mathrm{nDCG}^{\mathrm{class}}_A nDCG A class and macro-AUC are unchanged. With whole rows 1 and 2 duplicated, n D C G j c l a s s = ( 1 , 0.598 , 0.631 ) \mathrm{nDCG}^{\mathrm{class}}_j=(1,0.598,0.631) nDCG j class = ( 1 , 0.598 , 0.631 ) , micro-nDCG 0.898 0.898 0.898 , exemplar mean 0.855 0.855 0.855 , A U C j = ( 1 2 , 1 , 1 4 ) \mathrm{AUC}_j=(\frac{1}{2},1,\frac{1}{4}) AUC j = ( 2 1 , 1 , 4 1 ) , micro-AUC 3 4 \frac{3}{4} 4 3 .
Worked Examples for Section 2
A binary classification task can be represented either by one indicator column for a designated positive class or by two complementary one-hot columns. Although these encodings contain equivalent label information, evaluating the positive class alone is different from pooling or macro-averaging both columns. For example, with true classes (Dog, Dog, Cat) and predicted classes (Dog, Cat, Cat), taking Dog as positive yields
P r e c i s i o n D o g = 1 , P r e c i s i o n m i c r o = 2 3 , P r e c i s i o n m a c r o = 3 4 \mathrm{Precision}_{\mathrm{Dog}}=1,\qquad\mathrm{Precision}^{\mathrm{micro}}=\frac{2}{3},\qquad\mathrm{Precision}^{\mathrm{macro}}=\frac{3}{4} Precision Dog = 1 , Precision micro = 3 2 , Precision macro = 4 3
when micro and macro are computed using the two-column encoding. The difference concerns the choice of aggregation, not an inconsistency between the encodings; see Definitions 1 and 2 .
The next example shows why Theorem 5 needs additivity. Consider M ( v ) = ( R e c a l l ( v ) ) 2 = t p 2 ( t p + f n ) 2 M(v)=(\mathrm{Recall}(v))^2=\frac{tp^2}{(tp+fn)^2} M ( v ) = ( Recall ( v ) ) 2 = ( tp + f n ) 2 t p 2 . For two labels with count vectors v 1 = ( 1 , 0 , 1 , 2 ) T v_1=(1,0,1,2)^{\mathsf{T}} v 1 = ( 1 , 0 , 1 , 2 ) T and v 2 = ( 3 , 0 , 1 , 0 ) T v_2=(3,0,1,0)^{\mathsf{T}} v 2 = ( 3 , 0 , 1 , 0 ) T , the per-label scores are M 1 = 1 / 4 M_1=1/4 M 1 = 1/4 and M 2 = 9 / 16 M_2=9/16 M 2 = 9/16 , with denominators d e n 1 = 4 \mathrm{den}_1=4 den 1 = 4 and d e n 2 = 16 \mathrm{den}_2=16 den 2 = 16 . Pooling first and denominator-weighted averaging give, respectively,
M m i c r o = ( 1 + 3 ( 1 + 1 ) + ( 3 + 1 ) ) 2 = 4 9 , M^{\mathrm{micro}}=\left(\frac{1+3}{(1+1)+(3+1)}\right)^2=\frac{4}{9}, M micro = ( ( 1 + 1 ) + ( 3 + 1 ) 1 + 3 ) 2 = 9 4 ,
∑ j = 1 2 d e n j M j ∑ j = 1 2 d e n j = 4 ( 1 / 4 ) + 16 ( 9 / 16 ) 4 + 16 = 1 2 ≠ 4 9 . \frac{\sum_{j=1}^{2}\mathrm{den}_jM_j}{\sum_{j=1}^{2}\mathrm{den}_j}=\frac{4(1/4)+16(9/16)}{4+16}=\frac{1}{2}\ne\frac{4}{9}. ∑ j = 1 2 den j ∑ j = 1 2 den j M j = 4 + 16 4 ( 1/4 ) + 16 ( 9/16 ) = 2 1 = 9 4 .
The squared numerator and denominator are not additive in the count vectors, so this measure lies outside the hypothesis of Theorem 5 ; merely expressing a measure as a ratio is insufficient.
The last example illustrates the cancellation in Corollary 10 . Take two mutually exclusive actual labels with prevalences p p p and 1 − p 1-p 1 − p , where 0 < p < 1 0 < p < 1 0 < p < 1 , but allow multilabel predictions, which need not be one-hot. Keep the class-conditional rates fixed at rec 1 = spec 1 = 0.9 \operatorname{rec}_1 = \operatorname{spec}_1 = 0.9 rec 1 = spec 1 = 0.9 and rec 2 = spec 2 = 0.1 \operatorname{rec}_2 = \operatorname{spec}_2 = 0.1 rec 2 = spec 2 = 0.1 . The positive-class weights are ( p , 1 − p ) (p,1-p) ( p , 1 − p ) and the negative-class weights are ( 1 − p , p ) (1-p,p) ( 1 − p , p ) , so
rec m i c r o = 0.9 p + 0.1 ( 1 − p ) = 0.1 + 0.8 p , \operatorname{rec}_{\mathrm{micro}} = 0.9p + 0.1(1-p) = 0.1 + 0.8p, rec micro = 0.9 p + 0.1 ( 1 − p ) = 0.1 + 0.8 p ,
spec m i c r o = 0.9 ( 1 − p ) + 0.1 p = 0.9 − 0.8 p , \operatorname{spec}_{\mathrm{micro}} = 0.9(1-p) + 0.1p = 0.9 - 0.8p, spec micro = 0.9 ( 1 − p ) + 0.1 p = 0.9 − 0.8 p ,
BA m i c r o = 1 2 ( rec m i c r o + spec m i c r o ) = 1 2 . \operatorname{BA}_{\mathrm{micro}} = \frac{1}{2}\left(\operatorname{rec}_{\mathrm{micro}} + \operatorname{spec}_{\mathrm{micro}}\right) = \frac{1}{2}. BA micro = 2 1 ( rec micro + spec micro ) = 2 1 .
For p = 1 / 4 p = 1/4 p = 1/4 the pooled ( rec , spec ) (\operatorname{rec},\operatorname{spec}) ( rec , spec ) pair is ( 0.3 , 0.7 ) (0.3,0.7) ( 0.3 , 0.7 ) , whereas for p = 3 / 4 p = 3/4 p = 3/4 it is ( 0.7 , 0.3 ) (0.7,0.3) ( 0.7 , 0.3 ) ; balanced accuracy remains 0.5. Thus unequal class-specific rates do not by themselves rule out micro skew-invariance.
Examples for Corollary 41(c)
The standard squared loss has w ( t ) ≡ 2 w(t) \equiv2 w ( t ) ≡ 2 . There also exist nonquadratic strictly proper examples: choosing
w ( t ) = 1 + ( t − 1 2 ) 2 w(t) = 1 + \left(t - \frac{1}{2}\right)^2 w ( t ) = 1 + ( t − 2 1 ) 2
gives
ϕ ( t ) = 5 8 ∣ t ∣ 2 − 1 3 ∣ t ∣ 3 + 1 4 ∣ t ∣ 4 ( ∣ t ∣ ≤ 1 ) . \phi(t) = \frac{5}{8}|t|^2 - \frac{1}{3}|t|^3 + \frac{1}{4}|t|^4 \qquad(|t| \le1). ϕ ( t ) = 8 5 ∣ t ∣ 2 − 3 1 ∣ t ∣ 3 + 4 1 ∣ t ∣ 4 ( ∣ t ∣ ≤ 1 ) .
Here ϕ ′ ( t ) = t w ( t ) \phi'(t) = tw(t) ϕ ′ ( t ) = tw ( t ) for t > 0 t > 0 t > 0 , and ϕ ′ ′ ( t ) = 3 ( t − 1 3 ) 2 + 11 12 > 0 \phi''(t) = 3\left(t - \frac{1}{3}\right)^2 + \frac{11}{12} > 0 ϕ ′′ ( t ) = 3 ( t − 3 1 ) 2 + 12 11 > 0 there, so ϕ \phi ϕ is even and convex, while w > 0 w > 0 w > 0 is symmetric. It is not proportional to t 2 t^2 t 2 . The extended-valued log-loss example of Corollary 41 (b) has w ( t ) = 1 / [ t ( 1 − t ) ] w(t) = 1/[t(1-t)] w ( t ) = 1/ [ t ( 1 − t )] , which is also symmetric and strictly positive, although its ϕ \phi ϕ lies outside the finite-valued assumptions of (c). Finally, for the convex power losses ϕ ( t ) = ∣ t ∣ s \phi(t) = |t|^s ϕ ( t ) = ∣ t ∣ s with s ≥ 1 s \ge1 s ≥ 1 , w ( t ) = s t s − 2 w(t) = st^{s-2} w ( t ) = s t s − 2 is symmetric about 1 / 2 1/2 1/2 for every 0 < t < 1 0 < t < 1 0 < t < 1 only when s = 2 s = 2 s = 2 . Thus MAE, with w ( t ) = 1 / t w(t) = 1/t w ( t ) = 1/ t , and quartic error, with w ( t ) = 4 t 2 w(t) = 4t^2 w ( t ) = 4 t 2 , are not proper; for the quartic case the first-order condition is satisfied only at p = 1 / 2 p = 1/2 p = 1/2 .
Proofs for Section 5
Proof of Theorem 47 .
(a) (⇐ \Leftarrow ⇐ ) By Definition 46, a skew transformation rescales the subpopulation sizes while keeping the within-class distributions of Y ^ i , j \hat{Y}_{i,j} Y ^ i , j unchanged; any function of { F j + , F j − } \{F_j^+,F_j^-\} { F j + , F j − } is therefore invariant.
(a) (⇒ \Rightarrow ⇒ ) By the conventions above, a skew transformation changes only the counts ( n j + , n j − ) (n_j^+,n_j^-) ( n j + , n j − ) and leaves every F j ± F_j^\pm F j ± fixed, so it preserves { F j + , F j − } \{F_j^+,F_j^-\} { F j + , F j − } ; conversely, two data sets with the same { F j + , F j − } \{F_j^+,F_j^-\} { F j + , F j − } and different counts are related by a finite chain of skew transformations (class by class, through a common multiple of the counts). The relation “related by a chain of skew transformations” is therefore exactly “having the same { F j + , F j − } \{F_j^+,F_j^-\} { F j + , F j − } ”. A skew-invariant M M M is constant on each equivalence class of this relation, hence a function of the class representative { F j + , F j − } \{F_j^+,F_j^-\} { F j + , F j − } . (b) (⇐ \Leftarrow ⇐ ) α j \alpha_j α j and β j \beta_j β j are the means of ( 1 − y ^ ) 2 (1-\hat{y})^2 ( 1 − y ^ ) 2 under F j + F_j^+ F j + and of y ^ 2 \hat{y}^2 y ^ 2 under F j − F_j^- F j − , hence functions of { F j + , F j − } \{F_j^+,F_j^-\} { F j + , F j − } ; invariance follows from part (a).
(b) (⇒ \Rightarrow ⇒ ) A Calibration Framework measure is, by hypothesis, a function M = g ( { π j , α j , β j } ) M=g(\{\pi_j,\alpha_j,\beta_j\}) M = g ({ π j , α j , β j }) . Under rescaling ( λ , μ ) (\lambda,\mu) ( λ , μ ) , the effective prevalence becomes π j ′ = λ n j + / ( λ n j + + μ n j − ) \pi'_j=\lambda n_j^+/(\lambda n_j^+ + \mu n_j^-) π j ′ = λ n j + / ( λ n j + + μ n j − ) while ( α j , β j ) (\alpha_j,\beta_j) ( α j , β j ) is held fixed; as λ / μ \lambda/\mu λ / μ ranges over ( 0 , ∞ ) (0,\infty) ( 0 , ∞ ) , π j ′ \pi'_j π j ′ sweeps the whole interval ( 0 , 1 ) (0,1) ( 0 , 1 ) . Skew-invariance forces g g g to be constant in each π j \pi_j π j argument, class by class, on the set of attained values of ( π j , α j , β j ) (\pi_j,\alpha_j,\beta_j) ( π j , α j , β j ) (nothing is asserted outside what data can realise), leaving a function of { α j , β j } \{\alpha_j,\beta_j\} { α j , β j } alone. In particular, by Proposition 45 ,
BS j ( π j ′ ) = π j ′ α j + ( 1 − π j ′ ) β j (61) \operatorname{BS}_j(\pi'_j)=\pi'_j\alpha_j+(1-\pi'_j)\beta_j \tag*{(61)} BS j ( π j ′ ) = π j ′ α j + ( 1 − π j ′ ) β j (61)
sweeps the open interval between α j \alpha_j α j and β j \beta_j β j whenever α j ≠ β j \alpha_j\ne\beta_j α j = β j , which is why the per-class Brier score itself is skew-sensitive (Corollary 50 ).
Proof of Corollary 53
(a) Immediate from the formula π j ( 1 − π j ) \pi_j(1-\pi_j) π j ( 1 − π j ) .
(b) By Bayes’ theorem,
f j ( p ) = P ( Y ^ i , j = p ∣ Y i , j = 1 ) π j P ( Y ^ i , j = p ∣ Y i , j = 1 ) π j + P ( Y ^ i , j = p ∣ Y i , j = 0 ) ( 1 − π j ) . f_j(p)=\frac{P(\hat{Y}_{i,j}=p\mid Y_{i,j}=1)\pi_j}{P(\hat{Y}_{i,j}=p\mid Y_{i,j}=1)\pi_j+P(\hat{Y}_{i,j}=p\mid Y_{i,j}=0)(1-\pi_j)}. f j ( p ) = P ( Y ^ i , j = p ∣ Y i , j = 1 ) π j + P ( Y ^ i , j = p ∣ Y i , j = 0 ) ( 1 − π j ) P ( Y ^ i , j = p ∣ Y i , j = 1 ) π j .
Let a p a_p a p and b p b_p b p be the probabilities of the predicted value p p p under the within-class score distributions F j + F_j^+ F j + and F j − F_j^- F j − , so that the marginal probability of p p p is P ( p ) = π j a p + ( 1 − π j ) b p P(p)=\pi_ja_p+(1-\pi_j)b_p P ( p ) = π j a p + ( 1 − π j ) b p and f j ( p ) = π j a p / P ( p ) f_j(p)=\pi_ja_p/P(p) f j ( p ) = π j a p / P ( p ) . Since f j ( p ) − π j = π j ( 1 − π j ) ( a p − b p ) / P ( p ) f_j(p)-\pi_j=\pi_j(1-\pi_j)(a_p-b_p)/P(p) f j ( p ) − π j = π j ( 1 − π j ) ( a p − b p ) / P ( p ) ,
RES j = ∑ p P ( p ) ( f j ( p ) − π j ) 2 = π j 2 ( 1 − π j ) 2 ∑ p ( a p − b p ) 2 π j a p + ( 1 − π j ) b p , \operatorname{RES}_j=\sum_p P(p)\left(f_j(p)-\pi_j\right)^2=\pi_j^2(1-\pi_j)^2\sum_p\frac{(a_p-b_p)^2}{\pi_ja_p+(1-\pi_j)b_p}, RES j = p ∑ P ( p ) ( f j ( p ) − π j ) 2 = π j 2 ( 1 − π j ) 2 p ∑ π j a p + ( 1 − π j ) b p ( a p − b p ) 2 ,
the sum running over the values p p p with P ( p ) > 0 P(p)>0 P ( p ) > 0 . Hence RES j = 0 \operatorname{RES}_j=0 RES j = 0 at every prevalence if and only if a p = b p a_p=b_p a p = b p for every p p p , equivalently F j + = F j − F_j^+=F_j^- F j + = F j − . This is the no-separation case, in which positive and negative examples have identical score distributions; no claim about a likelihood ratio is needed (and a p / b p a_p/b_p a p / b p may be undefined where b p = 0 b_p=0 b p = 0 ). Otherwise RES j > 0 \operatorname{RES}_j>0 RES j > 0 for every π j ∈ ( 0 , 1 ) \pi_j\in(0,1) π j ∈ ( 0 , 1 ) , while RES j ≤ Var ( Y i , j ) = π j ( 1 − π j ) → 0 \operatorname{RES}_j\le\operatorname{Var}(Y_{i,j})=\pi_j(1-\pi_j)\to0 RES j ≤ Var ( Y i , j ) = π j ( 1 − π j ) → 0 as π j → 0 \pi_j\to0 π j → 0 or 1 1 1 ; so RES j \operatorname{RES}_j RES j is not constant in π j \pi_j π j and is skew-sensitive.
(c) The calibration map is a conditional frequency, so it is a function of the class balance: for a predicted value p p p attained by both classes (a p , b p > 0 a_p,b_p>0 a p , b p > 0 ), replicating the positive examples λ \lambda λ -fold gives f j ( λ ) ( p ) = λ π j a p / ( λ π j a p + ( 1 − π j ) b p ) f_j^{(\lambda)}(p)=\lambda\pi_ja_p/(\lambda\pi_ja_p+(1-\pi_j)b_p) f j ( λ ) ( p ) = λ π j a p / ( λ π j a p + ( 1 − π j ) b p ) , strictly increasing in λ \lambda λ . If f j ( 1 ) ( p ) = p f_j^{(1)}(p)=p f j ( 1 ) ( p ) = p , then f j ( λ ) ( p ) ≠ p f_j^{(\lambda)}(p)\ne p f j ( λ ) ( p ) = p for λ ≠ 1 \lambda\ne1 λ = 1 , so some term of REL j \operatorname{REL}_j REL j becomes positive. For a value p p p attained by positives only, f j ( p ) = 1 f_j(p)=1 f j ( p ) = 1 for every λ \lambda λ ; by negatives only, f j ( p ) = 0 f_j(p)=0 f j ( p ) = 0 . Hence calibration at two different prevalences forces every predicted value to be 1 1 1 (positive-only) or 0 0 0 (negative-only); conversely, if every value is of this kind, f j ( p ) = p f_j(p)=p f j ( p ) = p at every prevalence, so REL j = 0 \operatorname{REL}_j=0 REL j = 0 throughout. For a concrete instance, let three examples have score 2 / 3 2/3 2/3 , two of them positive, and three have score 1 / 3 1/3 1/3 , one of them positive. Then f j ( Y ^ ) = Y ^ f_j(\hat{Y})=\hat{Y} f j ( Y ^ ) = Y ^ and REL j = 0 \operatorname{REL}_j=0 REL j = 0 . Doubling the positives gives f j ( 2 / 3 ) = 4 / 5 f_j(2/3)=4/5 f j ( 2/3 ) = 4/5 and f j ( 1 / 3 ) = 1 / 2 f_j(1/3)=1/2 f j ( 1/3 ) = 1/2 , and REL j = 1 9 [ 5 ( 2 3 − 4 5 ) 2 + 4 ( 1 3 − 1 2 ) 2 ] = 1 / 45 \operatorname{REL}_j=\frac{1}{9}\left[5\left(\frac{2}{3}-\frac{4}{5}\right)^2+4\left(\frac{1}{3}-\frac{1}{2}\right)^2\right]=1/45 REL j = 9 1 [ 5 ( 3 2 − 5 4 ) 2 + 4 ( 3 1 − 2 1 ) 2 ] = 1/45 . Calibration is a property of the classifier together with the class balance of the data it is evaluated on, which is why the skew-invariant summary is ( α j , β j ) (\alpha_j,\beta_j) ( α j , β j ) and not a reliability term.
Proofs for Section 6
Proof of Theorem 60
(a) For raw DCG, by finite-sum rearrangement,
D C G m i c r o = 1 n m ∑ i , j D i j = 1 m ∑ j ( 1 n ∑ i D i j ) = 1 m ∑ j D C G j c l a s s , \mathrm{DCG}^{\mathrm{micro}}=\frac{1}{nm}\sum_{i,j}D_{ij}=\frac{1}{m}\sum_j\left(\frac{1}{n}\sum_iD_{ij}\right)=\frac{1}{m}\sum_j\mathrm{DCG}^{\mathrm{class}}_j, DCG micro = nm 1 i , j ∑ D ij = m 1 j ∑ ( n 1 i ∑ D ij ) = m 1 j ∑ DCG j class ,
D C G m i c r o = 1 n ∑ i ( 1 m ∑ j D i j ) = 1 n ∑ i D C G i e x . \mathrm{DCG}^{\mathrm{micro}}=\frac{1}{n}\sum_i\left(\frac{1}{m}\sum_jD_{ij}\right)=\frac{1}{n}\sum_i\mathrm{DCG}^{\mathrm{ex}}_i. DCG micro = n 1 i ∑ ( m 1 j ∑ D ij ) = n 1 i ∑ DCG i ex .
This is the same finite-sum argument as for Brier (Theorem 20).
(b) We compare micro and exemplar nDCG. Since S i = b i r i S_i=b_i r_i S i = b i r i , the difference between the weighted and unweighted averages is
n D C G m i c r o − 1 n ∑ i r i = ∑ i b i r i ∑ i b i − 1 n ∑ i r i = Cov ( b , r ) b ˉ , b ˉ = 1 n ∑ i b i > 0. \mathrm{nDCG}^{\mathrm{micro}}-\frac{1}{n}\sum_i r_i=\frac{\sum_i b_i r_i}{\sum_i b_i}-\frac{1}{n}\sum_i r_i=\frac{\operatorname{Cov}(b,r)}{\bar b},\qquad\bar b=\frac{1}{n}\sum_i b_i>0. nDCG micro − n 1 i ∑ r i = ∑ i b i ∑ i b i r i − n 1 i ∑ r i = b ˉ Cov ( b , r ) , b ˉ = n 1 i ∑ b i > 0.
Thus equality is equivalent to (40) . If the k i k_i k i are equal, so are the b i = ∑ r = 1 k i w ( r ) b_i=\sum_{r=1}^{k_i}w(r) b i = ∑ r = 1 k i w ( r ) , and the covariance vanishes. Constant r i r_i r i or other zero-covariance configurations also yield equality.
(c) For micro versus class-mean nDCG, by the definitions,
n D C G m i c r o = ∑ j T j B , 1 m ∑ j n D C G j c l a s s = 1 m ∑ j T j N j . \mathrm{nDCG}^{\mathrm{micro}}=\frac{\sum_j T_j}{B},\qquad\frac{1}{m}\sum_j\mathrm{nDCG}^{\mathrm{class}}_j=\frac{1}{m}\sum_j\frac{T_j}{N_j}. nDCG micro = B ∑ j T j , m 1 j ∑ nDCG j class = m 1 j ∑ N j T j .
Their difference is the left-hand side of (41) . If it vanishes for all freely chosen T j T_j T j , each coefficient 1 / B − 1 / ( m N j ) 1/B-1/(mN_j) 1/ B − 1/ ( m N j ) must vanish, so m N j = B mN_j=B m N j = B for all j j j ; the converse is immediate. Summing over j j j then gives ∑ j N j = B \sum_jN_j=B ∑ j N j = B . Yet
∑ j N j = w ( 1 ) ∑ i k i , B = ∑ i ∑ r = 1 k i w ( r ) ≤ w ( 1 ) ∑ i k i , \sum_jN_j=w(1)\sum_i k_i,\qquad B=\sum_i\sum_{r=1}^{k_i}w(r)\leq w(1)\sum_i k_i, j ∑ N j = w ( 1 ) i ∑ k i , B = i ∑ r = 1 ∑ k i w ( r ) ≤ w ( 1 ) i ∑ k i ,
with strict inequality when some k i ≥ 2 k_i\geq2 k i ≥ 2 and w ( 2 ) < w ( 1 ) w(2)<w(1) w ( 2 ) < w ( 1 ) . This is a contradiction. The coefficient argument concerns the linear extension to freely chosen class sums; the actual DCG sums come from rankings and need not be freely variable.
(d) The class mean and the exemplar mean are ∑ i , j D i j / ( m N j ) \sum_{i,j}D_{ij}/(mN_j) ∑ i , j D ij / ( m N j ) and ∑ i , j D i j / ( n b i ) \sum_{i,j}D_{ij}/(nb_i) ∑ i , j D ij / ( n b i ) , respectively. Their difference gives (42) . As D i j = 0 D_{ij}=0 D ij = 0 whenever Y i j = 0 Y_{ij}=0 Y ij = 0 , the identity holds for every freely chosen relevant-cell matrix if and only if the coefficients vanish at every relevant ( i , j ) (i,j) ( i , j ) : m N j = n b i mN_j=nb_i m N j = n b i . Substituting N j = n j + w ( 1 ) N_j=n_j^+w(1) N j = n j + w ( 1 ) gives
1 n j + = m w ( 1 ) n b i ( Y i j = 1 ) . \frac{1}{n_j^+}=\frac{mw(1)}{nb_i}\qquad(Y_{ij}=1). n j + 1 = n b i m w ( 1 ) ( Y ij = 1 ) .
Summing over relevant cells in two ways, with every class nonempty, yields
m = ∑ j ∑ i : Y i j = 1 1 n j + = m n ∑ i k i w ( 1 ) b i , hence ∑ i ( k i w ( 1 ) b i − 1 ) = 0. m=\sum_j\sum_{i:Y_{ij}=1}\frac{1}{n_j^+}=\frac{m}{n}\sum_i\frac{k_iw(1)}{b_i},\qquad\text{hence}\qquad\sum_i\left(\frac{k_iw(1)}{b_i}-1\right)=0. m = j ∑ i : Y ij = 1 ∑ n j + 1 = n m i ∑ b i k i w ( 1 ) , hence i ∑ ( b i k i w ( 1 ) − 1 ) = 0.
Each summand is nonnegative because b i ≤ k i w ( 1 ) b_i \le k_iw(1) b i ≤ k i w ( 1 ) , and one is strictly positive if k i ≥ 2 k_i \ge2 k i ≥ 2 and w ( 2 ) < w ( 1 ) w(2) < w(1) w ( 2 ) < w ( 1 ) . This contradiction establishes the last claim. As in (c), the universal-coefficient statement is for admissibly supported cell values, not an assumption that arbitrary such values can all arise from rank vectors.
For a concrete non-collapse witness, take the usual w ( r ) = 1 / log 2 ( r + 1 ) w(r) = 1/\log_2(r+1) w ( r ) = 1/ log 2 ( r + 1 ) and consider
Y = ( 0 0 1 1 1 0 ) , Y ^ = ( .9 .6 .2 .9 .6 .2 ) . Y = \begin{pmatrix} 0 & 0 & 1 \\ 1 & 1 & 0 \end{pmatrix}, \qquad\hat{Y} = \begin{pmatrix} .9 & .6 & .2 \\ .9 & .6 & .2 \end{pmatrix}. Y = ( 0 1 0 1 1 0 ) , Y ^ = ( .9 .9 .6 .6 .2 .2 ) .
Here the two rows have strict ranks ( 1 , 2 , 3 ) (1,2,3) ( 1 , 2 , 3 ) . Direct evaluation gives n D C G m i c r o ≈ 0.809953 \mathrm{nDCG}^{\mathrm{micro}} \approx0.809953 nDCG micro ≈ 0.809953 , 1 3 ∑ j n D C G j c l a s s ≈ 0.710310 \frac{1}{3}\sum_j \mathrm{nDCG}^{\mathrm{class}}_j \approx0.710310 3 1 ∑ j nDCG j class ≈ 0.710310 , and 1 2 ∑ i n D C G i = 0.750000 \frac{1}{2}\sum_i \mathrm{nDCG}_i = 0.750000 2 1 ∑ i nDCG i = 0.750000 . Thus the absence of a general collapse also occurs for rank-generated cells, not just free cell matrices.
Proof of Theorem 76
(a) For the raw collapse, by Definition 72 , Σ 1 ( A ) = ∑ i , j A i j \Sigma_1(A) = \sum_{i,j} A_{ij} Σ 1 ( A ) = ∑ i , j A ij , while ∑ i A i j = n j + A U C j \sum_i A_{ij} = n_j^+ \mathrm{AUC}_j ∑ i A ij = n j + AUC j and ∑ j A i j = k i A U C i e x \sum_j A_{ij} = k_i\mathrm{AUC}^{\mathrm{ex}}_i ∑ j A ij = k i AUC i ex . Divide the two ways of summing by n m nm nm and use π j = n j + / n \pi_j = n_j^+/n π j = n j + / n to obtain (59) .
(b) For micro versus macro, by (53) ,
A U C m i c r o = ∑ j n j + A U C j ∑ j n j + , A U C ‾ c l a s s = 1 m ∑ j A U C j . \mathrm{AUC}^{\mathrm{micro}} = \frac{\sum_j n_j^+\mathrm{AUC}_j}{\sum_j n_j^+}, \qquad\overline{\mathrm{AUC}}^{\mathrm{class}} = \frac{1}{m}\sum_j \mathrm{AUC}_j. AUC micro = ∑ j n j + ∑ j n j + AUC j , AUC class = m 1 j ∑ AUC j .
Writing n + ‾ = m − 1 ∑ j n j + > 0 \overline{n^+} = m^{-1}\sum_j n_j^+ > 0 n + = m − 1 ∑ j n j + > 0 , their difference equals Cov ^ j ( n j + , A U C j ) / n + ‾ \widehat{\operatorname{Cov}}_j(n_j^+,\mathrm{AUC}_j)/\overline{n^+} Cov j ( n j + , AUC j ) / n + . It vanishes exactly when the indicated covariance is zero; equal positive-class sizes are one sufficient case.
(c) For micro versus exemplar mean, similarly,
A U C m i c r o = ∑ i k i A U C i e x ∑ i k i , A U C ‾ e x = 1 n ∑ i A U C i e x . \mathrm{AUC}^{\mathrm{micro}} = \frac{\sum_i k_i\mathrm{AUC}^{\mathrm{ex}}_i}{\sum_i k_i}, \qquad\overline{\mathrm{AUC}}^{\mathrm{ex}} = \frac{1}{n}\sum_i \mathrm{AUC}^{\mathrm{ex}}_i. AUC micro = ∑ i k i ∑ i k i AUC i ex , AUC ex = n 1 i ∑ AUC i ex .
With k ‾ = n − 1 ∑ i k i > 0 \overline{k} = n^{-1}\sum_i k_i > 0 k = n − 1 ∑ i k i > 0 , their difference is Cov ^ i ( k i , A U C i e x ) / k ‾ \widehat{\operatorname{Cov}}_i(k_i,\mathrm{AUC}^{\mathrm{ex}}_i)/\overline{k} Cov i ( k i , AUC i ex ) / k . In particular, equal row cardinalities suffice for equality.
(d) For macro versus exemplar mean, expanding the definitions gives
A U C ‾ c l a s s − A U C ‾ e x = ∑ i , j : Y i j = 1 A i j ( 1 m n j + − 1 n k i ) . (62) \overline{\mathrm{AUC}}^{\mathrm{class}} - \overline{\mathrm{AUC}}^{\mathrm{ex}} = \sum_{i,j:Y_{ij}=1} A_{ij}\left(\frac{1}{mn_j^+} - \frac{1}{nk_i}\right). \tag*{(62)} AUC class − AUC ex = i , j : Y ij = 1 ∑ A ij ( m n j + 1 − n k i 1 ) . (62)
proving (59). If the expression vanishes for every supported real cell array, choose a single nonzero positive cell at a time: each coefficient must vanish, yielding m n j + = n k i mn_j^+ = nk_i m n j + = n k i on that support. The converse follows by substitution. Unlike the analogous nDCG restriction, these support conditions can hold on unbalanced data: with n = m = 4 n=m=4 n = m = 4 and positive-label sets { a , b } , { a , b } , { c } , { d } \{a,b\}, \{a,b\}, \{c\}, \{d\} { a , b } , { a , b } , { c } , { d } , the row and column counts are both ( 2 , 2 , 1 , 1 ) (2,2,1,1) ( 2 , 2 , 1 , 1 ) , and the condition holds on every relevant cell.
Proof of Proposition 79(c)–(d)
The pair weights b i i ′ b_{ii^{\prime}} b i i ′ are those defined in the proof of Proposition 79(b); a report ordering q i > q i ′ q_i > q_{i^{\prime}} q i > q i ′ receives b i i ′ b_{ii^{\prime}} b i i ′ from the ordered pair.
(c) We treat small samples without independence. For n = 2 n = 2 n = 2 , on Y i = 1 , Y i ′ = 0 Y_i = 1, Y_{i^{\prime}} = 0 Y i = 1 , Y i ′ = 0 , the denominator is n + n − = 1 n^{+}n^{-} = 1 n + n − = 1 . Thus b i i ′ − b i ′ i = p i − p i ′ b_{ii^{\prime}} - b_{i^{\prime}i} = p_i - p_{i^{\prime}} b i i ′ − b i ′ i = p i − p i ′ , without independence. For n = 3 n = 3 n = 3 , the remaining label z ∈ { 0 , 1 } z \in\{0,1\} z ∈ { 0 , 1 } gives
n + n − = ( 1 + z ) ( 2 − z ) = 2 n^{+}n^{-} = (1 + z)(2 - z) = 2 n + n − = ( 1 + z ) ( 2 − z ) = 2
under either orientation of the pair. Therefore b i i ′ − b i ′ i = 1 2 ( p i − p i ′ ) b_{ii^{\prime}} - b_{i^{\prime}i} = \frac{1}{2}(p_i - p_{i^{\prime}}) b i i ′ − b i ′ i = 2 1 ( p i − p i ′ ) under any joint law. Sorting by p i p_i p i again maximises each pair contribution.
(d) We give a dependent counterexample and then extend it. For n = 4 n = 4 n = 4 , consider the five label vectors and probability weights below:
Table 6.
Their marginal probabilities are p = ( 8 , 5 , 7 , 6 ) / 11 p = (8,5,7,6)/11 p = ( 8 , 5 , 7 , 6 ) /11 , all distinct. Directly summing the conditional AUC values gives
E [ AUC ( 1 ≻ 3 ≻ 4 ≻ 2 , Y ) ] = 41 66 , E [ AUC ( 1 ≻ 4 ≻ 3 ≻ 2 , Y ) ] = 83 132 . \mathbb{E}[\operatorname{AUC}(1 \succ3 \succ4 \succ2,Y)] = \frac{41}{66}, \qquad\mathbb{E}[\operatorname{AUC}(1 \succ4 \succ3 \succ2,Y)] = \frac{83}{132}. E [ AUC ( 1 ≻ 3 ≻ 4 ≻ 2 , Y )] = 66 41 , E [ AUC ( 1 ≻ 4 ≻ 3 ≻ 2 , Y )] = 132 83 .
The second ordering exceeds the truthful first ordering by 1 / 132 > 0 1/132 > 0 1/132 > 0 . For any n = 4 + k n = 4 + k n = 4 + k with k ≥ 0 k \ge0 k ≥ 0 , append k k k examples that are positive with certainty and rank them above the original four in both reports. Only the relative order of indices 3, 4 differs, and the expected AUC for the truthful ordering minus the alternative is
D k = 1 11 ( − 1 3 ( k + 1 ) + 5 2 ( k + 2 ) − 3 k + 3 ) = − 5 k 2 + 4 k + 3 66 ( k + 1 ) ( k + 2 ) ( k + 3 ) < 0. \begin{aligned}
D_k &= \frac{1}{11}\left(-\frac{1}{3(k+1)}+\frac{5}{2(k+2)}-\frac{3}{k+3}\right) \\
&= -\frac{5k^2+4k+3}{66(k+1)(k+2)(k+3)} < 0.
\end{aligned} D k = 11 1 ( − 3 ( k + 1 ) 1 + 2 ( k + 2 ) 5 − k + 3 3 ) = − 66 ( k + 1 ) ( k + 2 ) ( k + 3 ) 5 k 2 + 4 k + 3 < 0.
Finally, perturb the added certain-positive labels to independent Bernoulli variables with distinct probabilities 1 − ε l 1-\varepsilon_l 1 − ε l sufficiently close to 1 (and different from the original four marginals). By continuity of the finite-scenario expected AUC, the strict inequality persists, now with all marginals distinct. This establishes the claim for every n ≥ 4 n \ge4 n ≥ 4 .
Ordinal Labels and Quadratic Weighted Kappa
Applying the squared-error deviation to these ordinal indices gives the normalised per-example disagreement
D o r d ( Y i , ⋅ , Y ^ i , ⋅ ) = ( y i − y ^ i ) 2 ( K − 1 ) 2 = ∑ k = 0 K − 1 ∑ l = 0 K − 1 ( k − l ) 2 ( K − 1 ) 2 Y i k Y ^ i l , (63) D_{\mathrm{ord}}(Y_{i,\cdot},\widehat{Y}_{i,\cdot})=\frac{(y_i-\widehat{y}_i)^2}{(K-1)^2}=\sum_{k=0}^{K-1}\sum_{l=0}^{K-1}\frac{(k-l)^2}{(K-1)^2}Y_{ik}\widehat{Y}_{il}, \tag*{(63)} D ord ( Y i , ⋅ , Y i , ⋅ ) = ( K − 1 ) 2 ( y i − y i ) 2 = k = 0 ∑ K − 1 l = 0 ∑ K − 1 ( K − 1 ) 2 ( k − l ) 2 Y ik Y i l , (63)
where the second equality follows from the one-hot encoding. The average 1 n ∑ i D o r d \frac{1}{n}\sum_i D_{\mathrm{ord}} n 1 ∑ i D ord is exactly the observed weighted disagreement in the Q W K QWK Q W K numerator.
Full Q W K QWK Q W K takes the form
κ w = 1 − 1 n ∑ i D o r d ∑ k = 0 K − 1 ∑ l = 0 K − 1 ( k − l ) 2 ( K − 1 ) 2 ⋅ n k n ⋅ m l n , (64) \kappa_w=1-\frac{\frac{1}{n}\sum_i D_{\mathrm{ord}}}{\sum_{k=0}^{K-1}\sum_{l=0}^{K-1}\frac{(k-l)^2}{(K-1)^2}\cdot\frac{n_k}{n}\cdot\frac{m_l}{n}}, \tag*{(64)} κ w = 1 − ∑ k = 0 K − 1 ∑ l = 0 K − 1 ( K − 1 ) 2 ( k − l ) 2 ⋅ n n k ⋅ n m l n 1 ∑ i D ord , (64)
where n k = ∑ i Y i k n_k=\sum_iY_{ik} n k = ∑ i Y ik and m l = ∑ i Y ^ i l m_l=\sum_i\widehat{Y}_{il} m l = ∑ i Y i l are the column sums of the one-hot matrices. The expression is defined as written only when the expected weighted disagreement E w E_w E w , namely the denominator in (64) , is strictly positive. When E w = 0 E_w=0 E w = 0 , a separate convention would be needed to assign a value to Q W K QWK Q W K .
The structure of Q W K QWK Q W K is analogous to the treatment of Cohen’s κ \kappa κ in Prati (2027). In both cases the observed term is a cell-operator aggregation (a coincidence aggregation for κ \kappa κ , an ordinal deviation aggregation for κ w \kappa_w κ w ), whereas the expected-under-independence term is computed from products of column marginals of Y Y Y and Y ^ \widehat{Y} Y . As noted in Prati (2027) for κ \kappa κ , these marginals can be calculated as column sums of the indicator matrices and tracked as auxiliary quantities alongside the standard cell aggregations. Q W K QWK Q W K is generally skew-sensitive: changes in class prevalence can alter the observed weighted disagreement and its expected-under-independence normaliser, so invariance under prevalence shifts is not guaranteed.