betaFeedback

WorksComputer Science

Discrimination, Calibration, and Ranking: A Unified Matrix Framework

Abstract

Discrimination, calibration, and ranking are the three pillars of classifier evaluation, yet no single algebraic language spans all three. A recent indicator-matrix account unifies discrimination measures (precision, recall, F1, Matthews correlation) across binary, multiclass, multilabel and related settings, using Boolean cell operations on indicator label matrices, generalised to soft labels by t-norms. Calibration measures (Brier score, log-loss) and ranking measures (nDCG, AUC) fall outside it. We close the gap with two further classes of cell operators: deviation operators phi(y - yhat), which give a Calibration Framework, and rank- and slice-dependent operators (positional gains for DCG, column concordance cells for AUC), which give a Ranking Framework. Both inherit the micro/macro/exemplar aggregation architecture. For calibration, the three Brier aggregations collapse to one scalar; the Murphy decomposition holds in matrix form (unbinned, binned and isotonic); and, within the Brier-type class-indexed family, a measure is skew-invariant if and only if it depends only on the class-conditional errors alpha_j and beta_j, the calibration analogues of recall and specificity. For ranking, raw DCG collapses across aggregations while nDCG does not; per-class nDCG and per-class AUC are related by a row-column duality; micro-AUC differs from pooled ROC-AUC; raw positional scores are never strictly proper, and nDCG and AUC need not be proper. The skew-invariant per-class building blocks share one class-conditional expectation template. Experiments on two multilabel benchmarks illustrate the distinctions. Many of the results are machine-checked in Lean 4; the proofs, the experiments and the aggmetrics.scores package are public: https://github.com/rcprati/discrimination-calibration-ranking

Introduction

Evaluating the performance of classification models is a core step in any empirical machine learning study (Japkowicz and Shah, 2011). Many classifiers output scores in [0,1][0,1], often interpreted as estimated class probabilities; we collect such scores in a matrix P^∈[0,1]n×m\hat{P} \in[0,1]^{n \times m} before a decision rule is applied. This output carries distinct kinds of information. First, binary decisions obtained using a specified decision rule determine threshold-dependent discrimination measures such as F1F_1, precision, and recall; the relative ordering of scores alone does not determine these measures. Second, the magnitudes of probability estimates matter for probabilistic predictive quality: calibration concerns whether a forecast such as P^i,j=0.8\hat{P}_{i,j}=0.8 corresponds to an event frequency near 80%, while proper scoring rules such as Brier and log-loss assess overall probabilistic quality, including calibration and resolution rather than calibration alone. A third, distinct question concerns ranking: row-axis nDCG evaluates the positions of relevant labels among the labels of each example, while column-axis AUC evaluates the ordering of positive versus negative examples for a fixed class. All three evaluation paradigms are active in most empirical study of multilabel or probability-producing classifiers, yet they have developed largely in isolation, with separate literatures, separate intuitions, and no common algebraic language.

A unified algebraic framework for discrimination was recently proposed by Prati (2027). It represents actual and predicted labels as binary indicator matrices Y,Y^∈{0,1}n×mY,\hat{Y}\in\{0,1\}^{n\times m} and defines cell-level true/false positive/negative counts via element-wise Boolean conjunction and negation; soft labels and predictions are accommodated by replacing the Boolean connectives with triangular norms (t-norms), which reduce to the Boolean case on binary matrices. Three aggregation operators, global (Σ1\Sigma_1), column-wise (Σm\Sigma_m), and row-wise (Σn\Sigma_n), corresponding to micro, macro, and exemplar averaging, then generate entire families of measures for multiclass, multilabel and related classification problems from any single binary formula. The framework characterises skew-invariant measures (functions of recall and specificity only), proves that micro-averaging equals denominator-weighted macro-averaging, and identifies the product t-norm as the unique marginally consistent one (which implies the confusion-matrix partition property). However, calibration measures (Brier score, log-loss, Murphy decomposition) and nDCG fall outside the framework of Prati (2027): they operate on the raw probability matrix P^\hat{P} rather than a thresholded indicator matrix, and their cell values are not expressible as Boolean or t-norm outputs. AUC is reachable there only as an integral over thresholds of the coincidence cells.

This paper extends the indicator-matrix framework by replacing its coincidence cell operator (the Boolean conjunction and its t-norm generalisation) with two further operator classes, yielding a three-class taxonomy:

  1. Coincidence operators (Boolean conjunction, generalised by t-norms): the cell operator of Prati (2027), quantifying Boolean coincidence of actual and predicted binary labels, or their soft overlap according to a chosen t-norm; for soft entries, positivity of both inputs alone need not imply a positive t-norm value. These recover the Discrimination Framework of Prati (2027) with no modification.

  1. Deviation operators (ϕ(y−y^)\phi(y-\hat{y}) for convex ϕ\phi): measuring the magnitude of the gap between actual and predicted soft labels. This yields the Calibration Framework, covering calibration.

  1. Positional and rank-dependent operators: row-positional gains Yijw(rank⁡i(j))Y_{ij}w(\operatorname{rank}_i(j)), with non-increasing ww, give the nDCG family; column-dependent cells compare positive and negative examples of a class to give AUC. A fixed column-rank weight yields an affine identity for aggregated AUC under distinct scores, but does not generally

reproduce an individual AUC concordance cell. The row and column ranking axes produce complementary subfamilies of the Ranking Framework.

Within this taxonomy we establish the following results:

  • For the Calibration Framework, the squared-error deviation operator D(y,y^)=(y−y^)2D(y,\hat{y}) = (y-\hat{y})^2 induces the Brier score family. Applying the three aggregation operators yields micro-, macro-, and exemplar-averaged Brier scores; we prove that all three collapse to the same scalar, because the Brier score is a sum of per-cell terms with a fixed denominator (the same holds for any per-cell deviation, but not for the ratio measures of the Discrimination Framework or for normalised nDCG). We derive a matrix-language version of the Murphy (Murphy (1973)) reliability–resolution–uncertainty decomposition, BSj=UNCj−RESj+RELj\mathrm{BS}_j = \mathrm{UNC}_j - \mathrm{RES}_j + \mathrm{REL}_j, via the tower property of conditional expectations. The same identity is also stated for a calibration map estimated by isotonic regression, which needs no bins (Proposition 28).

  • We characterise proper scoring rules within the Calibration Framework via the Savage representation (Savage (1971); Gneiting and Raftery (2007)), showing that a symmetric deviation operator ϕ(y−y^)\phi(y-\hat{y}) is proper exactly when its weight ϕ′(t)/t\phi'(t)/t is symmetric about 12\frac{1}{2} (convexity of ϕ\phi alone is not sufficient); the proper ones include squared error and nonquadratic losses, while log-loss satisfies the same weight condition in an extended-valued sense. For skew-sensitivity, we prove that a Brier-type class-indexed measure of the form M=g({πj,αj,βj}j=1m)M = g(\{\pi_j,\alpha_j,\beta_j\}_{j=1}^{m}) is skew-invariant if and only if it depends on {αj,βj}j=1m\{\alpha_j,\beta_j\}_{j=1}^{m} alone, where αj=E[(1−Y^ij)2∣Yij=1]\alpha_j = \mathbb{E}[(1-\hat{Y}_{ij})^2 \mid Y_{ij}=1] and βj=E[Y^ij2∣Yij=0]\beta_j = \mathbb{E}[\hat{Y}_{ij}^{2} \mid Y_{ij}=0] are the class-conditional squared errors, analogous to the complements of recall and specificity in the hard-prediction limit (and, in the binary case, the two components of the stratified Brier score (Wallace and Dahabreh (2012, 2014)). General calibration measures are skew-invariant exactly when they depend only on the within-class score distributions (Theorem 47(a)).

  • For the Ranking Framework, the row-dependent positional operator Dw(Yi,j,Y^i,⋅)=Yi,j⋅w(rank⁡i(j))D_w(Y_{i,j},\hat{Y}_{i,\cdot}) = Y_{i,j}\cdot w(\operatorname{rank}_i(j)) induces the nDCG family. We show that raw DCG collapses across aggregation schemes (micro = macro = exemplar) while normalised nDCG does not, due to heterogeneous IDCG denominators. We propose a per-class nDCG summary within this matrix formalism, with the conditional-mean interpretation: nDCGjclass=1w(1)E[w(rank⁡i(j))∣Yij=1]\mathrm{nDCG}^{\mathrm{class}}_j = \frac{1}{w(1)}\mathbb{E}[w(\operatorname{rank}_i(j)) \mid Y_{ij}=1]. The column-dependent AUC family instead uses the concordance cell Aij=Yijf(i,j)A_{ij} = Y_{ij}f(i,j), where f(i,j)f(i,j) is the fraction of negative examples for class jj scored below example ii (with half credit for ties). Unlike a fixed positional cell Yijw(rank⁡i(j))Y_{ij}w(\operatorname{rank}_i(j)), its value also depends on the negative-label set. At the per-class aggregate level, AUC is the Wilcoxon–Mann–Whitney statistic (Hanley and McNeil (1982)) and, for distinct column scores, an affine function of the mean positive-example rank (Proposition 73). This construction yields the per-example AUCiex\mathrm{AUC}^{\mathrm{ex}}_i summary and makes the per-class skew-invariance transparent. A row–column duality theorem unifies the two subfamilies: both nDCGjclass\mathrm{nDCG}^{\mathrm{class}}_j and AUCj\mathrm{AUC}_j are conditional means over positive examples of rank-based quality measures, with a fixed row-rank discount for nDCG and a label-dependent column-concordance fraction for AUC (whose aggregate is an affine function of mean column rank).

  • Finally, as a cross-framework synthesis, we show that the three skew-invariant per-class bases, namely recall/specificity (Discrimination Framework), (αj,βj)(\alpha_j,\beta_j) (Calibration Framework), and nDCGjclass/AUCj\mathrm{nDCG}^{\mathrm{class}}_j/\mathrm{AUC}_j (Ranking Framework), all share the same algebraic structure: they are class-conditional means (over the positives for recall, αj\alpha_j, per-class nDCG and AUC; over the negatives for specificity and βj\beta_j). This single template organises the skew-invariance proofs for all three frameworks, under reweightings that preserve the conditional distribution of the averaged cell quality (for per-class nDCG this includes the within-row ranks of the positives, not only each label’s score distribution), and reveals that pooled (micro) aggregates are skew-sensitive in general (for the ratio measures of the Discrimination Framework and for micro-AUC because they weight classes by prevalence-dependent denominators; for the Brier score because each per-class term is itself prevalence-dependent). We further argue that the row/column dependency split is specific to the Ranking Framework: the natural ways of introducing rank dependence into the Discrimination or Calibration Frameworks either trivialise or recover the positional operator.

The measures of the Calibration and Ranking Frameworks, together with diagnostics that report the covariances behind the gaps between the micro, class and exemplar summaries, are implemented in the experimental subpackage aggmetrics.scores of the aggmetrics package (https://github.com/rcprati/aggmetrics), available from version 0.3.0.

This paper is organised as follows. Section 2 reviews the framework of Prati (2027). Section 3 reviews related work. Section 4 introduces the three-class cell operator taxonomy and the framework bridge theorem connecting the hard-prediction limits of the Discrimination and Calibration Frameworks; the hard-prediction limit of the Ranking Framework is treated in Section 6. Section 5 develops the Calibration Framework in full. Section 6 develops the Ranking Framework, including both the nDCG and AUC subfamilies and the duality theorem. Section 7 synthesises the three frameworks. Section 8 presents empirical illustrations. Section 9 concludes.

Background: The Indicator-Matrix Framework

This section provides a self-contained review of the base framework of Prati (2027), which covers discrimination-based evaluation. Readers familiar with that paper may skip ahead to Section 4.

Indicator Matrices

Consider a dataset of nn examples, each associated with a subset of mm labels. The actual and predicted label assignments are represented as binary matrices.

Definition 1 (Indicator matrices). The actual label matrix Y∈{0,1}n×m\mathbf{Y} \in\{0,1\}^{n \times m} has Yi,j=1Y_{i,j}=1 iff example eie_i belongs to class or label ljl_j. The predicted label matrix Y^∈{0,1}n×m\hat{\mathbf{Y}} \in\{0,1\}^{n \times m} has Y^i,j=1\hat{Y}_{i,j}=1 iff the classifier assigns label ljl_j to eie_i.

In multiclass problems, each row of Y\mathbf{Y} and Y^\hat{\mathbf{Y}} is a one-hot vector: exactly one entry equals 1. In multilabel problems, any number of entries per row may equal 1. The binary case is the special case m=2m=2.

A binary task can be encoded by one indicator column for a designated positive class or by two complementary one-hot columns; the encodings carry the same label information, but scoring the positive class alone differs from pooling or macro-averaging both columns (Appendix § 11).

A soft classifier outputs a probability matrix P^∈[0,1]n×m\hat{P} \in[0,1]^{n \times m}; a decision rule converts it into the binary Y^\hat{Y} of Definition 1: argmax in the multiclass case and thresholding at an arbitrary θ\theta in the multilabel case. When misclassification costs are available, these rules can be adjusted accordingly, following the cost-sensitive principles of Elkan (2001). When the tt-norm extension is applied to soft values, the entries of Y^\hat{Y} in Definition 4 may represent scores in [0,1][0,1] rather than hard predictions. The Calibration and Ranking Frameworks instead work explicitly with P^\hat{P}; a hard prediction matrix is obtained only when a decision rule is required.

Aggregation Operators

Definition 2 (Aggregation operators). Let A∈Rn×mA \in\mathbb{R}^{n \times m} be any real-valued matrix. Three aggregation operators are defined:

Σ1(A)=∑i=1n∑j=1mAi,j∈R(global / micro),\Sigma_1(A) = \sum_{i=1}^{n} \sum_{j=1}^{m} A_{i,j} \in\mathbb{R} \qquad(\text{global / micro}),
[Σm(A)]j=∑i=1nAi,j∈R(column-wise / per-class),[\Sigma_m(A)]_j = \sum_{i=1}^{n} A_{i,j} \in\mathbb{R} \qquad(\text{column-wise / per-class}),
[Σn(A)]i=∑j=1mAi,j∈R(row-wise / per-example).[\Sigma_n(A)]_i = \sum_{j=1}^{m} A_{i,j} \in\mathbb{R} \qquad(\text{row-wise / per-example}).

In particular, Σm(A)∈Rm\Sigma_m(A) \in\mathbb{R}^{m} and Σn(A)∈Rn\Sigma_n(A) \in\mathbb{R}^{n}; their indexed components are scalars.

These operators correspond to the three most common averaging schemes in machine learning libraries (e.g., average= ’micro’, ’macro’, and ’samples’ in scikit-learn): Σ1\Sigma_1 pools all counts globally; Σm\Sigma_m produces a per-class vector from which an unweighted (macro) or prevalence-weighted average is computed; Σn\Sigma_n produces a per-example vector whose unweighted mean is the exemplar average.

Coincidence Cell Operators and Confusion Matrices

In Prati (2027), the four confusion matrices are defined by element-wise Boolean operations (conjunction and negation) on the binary indicator matrices. Soft labels or soft predictions are handled by an extension in which the Boolean connectives are replaced by their fuzzy counterparts, triangular norms, with the Boolean definitions recovered exactly on binary matrices. Since this paper allows soft entries throughout, we state the general tt-norm form directly.

Definition 3 (tt-norm). A triangular norm (Klement et al. (2000)) is a function T:[0,1]2→[0,1]T : [0,1]^2 \to[0,1] satisfying commutativity, associativity, monotonicity, and the boundary condition T(a,1)=aT(a,1) = a for all a∈[0,1]a \in[0,1]. The three standard tt-norms are the Gödel (minimum) TM(a,b)=min⁡(a,b)T_M(a,b) = \min(a,b), the product TP(a,b)=a⋅bT_P(a,b) = a \cdot b, and the Łukasiewicz TL(a,b)=max⁡(0,a+b−1)T_L(a,b) = \max(0,a+b-1).

Definition 4 (Confusion cell matrices). Given Y,Y^∈[0,1]n×m\mathbf{Y},\hat{\mathbf{Y}} \in[0,1]^{n \times m}, a t-norm TT, and the standard complement N(a)=1−aN(a)=1-a, the four confusion cell matrices are:

TPi,j=T(Yi,j,Y^i,j),FPi,j=T(N(Yi,j),Y^i,j),FNi,j=T(Yi,j,N(Y^i,j)),TNi,j=T(N(Yi,j),N(Y^i,j)).\begin{aligned} TP_{i,j} &= T(Y_{i,j},\hat{Y}_{i,j}), & FP_{i,j} &= T(N(Y_{i,j}),\hat{Y}_{i,j}),\\ FN_{i,j} &= T(Y_{i,j},N(\hat{Y}_{i,j})), & TN_{i,j} &= T(N(Y_{i,j}),N(\hat{Y}_{i,j})). \end{aligned}

For binary Y,Y^∈{0,1}n×m\mathbf{Y},\hat{\mathbf{Y}} \in\{0,1\}^{n \times m}, all three t-norms reduce to the Boolean AND, so TP,FP,FN,TN∈{0,1}n×mTP,FP,FN,TN \in\{0,1\}^{n \times m} are binary indicator matrices whose entries equal 1 exactly where the corresponding outcome holds for example eie_i and label ljl_j.

References

  1. [2]Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1–3, 1950. doi: 10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2. URL https://journals.ametsoc.org/view/journals/mwre/78/1/1520-0493_1950_078_0001_vofeit_2_0_co_2.xml.doi.org/10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2

Paper details

vlnc:W_s3f48g4h
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents