betaFeedback

WorksComputer Science

aggmetrics: Aggregating Classification Performance Measures for Machine Learning

Abstract

aggmetrics is a Python package that extends any binary performance measure expressed in the four confusion-matrix counts to multiclass, multilabel, ordinal, cost-sensitive, soft-label and cross-validation settings, by substituting one of four aggregation operators for what libraries usually expose as an average switch. The package also stores, for every measure, the structural properties that decide which identities apply to it, can infer those properties from a user-supplied formula, and uses them to warn when a measure is used where it does not behave as commonly assumed. Its test suite encodes the results of the companion theory as executable tests. The package is BSD-3 licensed, depends only on NumPy, and is available at https://github.com/rcprati/aggmetrics; documentation at https://aggmetrics.readthedocs.io.

Introduction

Reporting a classifier’s performance involves two choices that are made separately in practice: which measure to use, and how to aggregate it over classes, examples or cross-validation folds. In scikit-learn (Pedregosa et al., [5]) the second choice is an average argument attached to a fixed list of functions, so a combination outside that list (a macro-averaged Matthews correlation coefficient, an exemplar-averaged Jaccard index) has to be written by hand. More importantly, a function returns a float whether or not the evaluation setting suits the measure. Two examples reproducible with the package illustrate the problem.

Floors. Under one-hot encoding of an mm-class problem, each misclassified example still contributes m−2m - 2 true negatives. On a 10-class problem, a classifier that is wrong on every example scores 0.800 on label accuracy and −0.111 on MCC (micro-averaged), rather than 0 and −1. Measures that depend on the true negatives at tp=0tp = 0 cannot fall below a floor that grows with mm.

Micro-averaging. Micro-averaged precision, recall and F1F_1 are all equal to accuracy on single-label data, so reporting the three together conveys one number. And for measures such as balanced accuracy or MCC, the pooled micro value can lie outside the range of the per-class values, so no choice of weights reproduces it from a macro average.

aggmetrics addresses both issues. It implements the construction of Prati [6], in which micro-, macro-, exemplar- and fold-pooled averaging are four instances of one operation, and it records the properties of each measure that determine such behaviour, so they can be queried and checked.

Design

Construction. Actual and predicted labels are held as indicator matrices YY, Y^∈{0,1}n×m\hat{Y} \in\{0,1\}^{n \times m} (real-valued for soft labels; for ordinal data with mm classes, the n×(m−1)n \times(m-1) matrix of cumulative thresholds I(yi>j)\mathbb{I}(y_i > j)). Four cell matrices TP=Y∧Y^TP = Y \wedge\hat{Y}, and analogously FPFP, FNFN, TNTN, are reduced by an operator: Σ1\Sigma_1 over all entries (micro), Σm\Sigma_m over columns (macro), Σn\Sigma_n over rows (exemplar), and Σk\Sigma_k across cross-validation splits. Substituting the reduced counts into a binary measure M(tp,fp,fn,tn)M(tp, fp, fn, tn) gives its extension. The 21 registered measures and four operators therefore give 84 combinations through one code path, with scikit-learn’s ’micro’, ’macro’, ’weighted’ and ’samples’ as a subset. Constructors accept class-index vectors, binary matrices, probability matrices and ordinal classes; the cells are cached, so a report over many measures costs one pass.

Structural properties. Each measure stores whether it is aggregation-decomposable (M=⟨a,v⟩/⟨b,v⟩M = \langle a,v\rangle/\langle b,v\rangle with v=(tp,fp,fn,tn)v = (tp,fp,fn,tn), which is the condition under which micro equals denominator-weighted macro), whether it is skew-invariant, whether its value at tp=0tp = 0 depends on tntn, and its direction. Bayes decision thresholds and attainable floors follow in closed form. Properties are data, not documentation:

get_measure(’mcc’).decomposable       # False
get_measure(’recall’).bayes_threshold() # 0.0: maximised by predicting all positive
get_measure(’mcc’).attainable_floor(10) # -0.111

Inference from a formula. register_measure accepts any callable of (tp,fp,fn,tn)(tp,fp,fn,tn) and derives the properties above: decomposability by solving a homogeneous linear system for (a,b)(a,b) over sampled count vectors and validating it on held-out vectors, and the remaining properties by evaluation. The new measure is then usable under every operator.

def prevalence_threshold(tp, fp, fn, tn):
    r, s = tp / (tp + fn), tn / (tn + fp)
    return (np.sqrt(r * (1 - s)) + s - 1) / (r + s - 1)

m = register_measure(’prevalence_threshold’, prevalence_threshold)
m.decomposable       # False (inferred)
m.skew_invariant     # True
im.compute(’prevalence_threshold’, ’exemplar’)

Because the 21 built-in measures have analytically derived properties, inference can be checked against them: verify_inference() agrees on all 21, and the recovered coefficients match the analytic ones to within 6.1×10−156.1 \times10^{-15}, about 28 times the double-precision machine epsilon (ε=2.2×10−16\varepsilon= 2.2 \times10^{-16}). The check found two errors in our own hand annotations (the error rate and the false positive rate both retain a tntn dependence at tp=0tp = 0, so both flatter a classifier that is wrong everywhere).

Diagnostics. diagnose(im) returns a list of Finding objects, each with a code, a severity, a message and the triggering values. The checks cover: pooled micro value outside the per-class range (outside_hull); tntn-dependent measures on one-hot data (inflated_floor); micro-collapse on single-label data (micro_collapse); undefined per-class entries and the size of the convention effect (undefined_entries); measures whose optimum is a trivial or non-Bayes rule (degenerate_optimum, not_erm_consistent); one-vs-rest prior drift (ovr_skew_drift); and evaluation suites that exceed the m(m−1)m(m-1) degrees of freedom of an mm-class confusion matrix (redundant_suite). With emit_warnings=True findings are raised as Python warnings, so an evaluation script can fail in continuous integration on a flawed protocol.

Further functionality. Four conventions for 0/0 entries (exclude, zero, one, nan) are an explicit parameter; zero reproduces scikit-learn’s zero_division=0. Soft ground truth uses t-norms in place of conjunction; marginal_defect shows that only the product t-norm preserves the marginals. Cost matrices recover MAE and MSE for Cjk=∣j−k∣C_{jk} = \lvert j-k\rvert and (j−k)2(j-k)^2. Observation masks restrict every operator to observed entries, covering partial labels and abstention. pareto_dominance(im_a, im_b) decides whether one model beats another on all monotone measures, and make_scorer yields scorers usable in cross_val_score and GridSearchCV.

Example

import numpy as np
from aggmetrics import IndicatorMatrix, diagnose

y    = np.arange(500) % 10
yhat = (y + 1) % 10                 # wrong on every example
im = IndicatorMatrix(y, yhat, task=’multiclass’)

im.accuracy()                       # 0.000
im.compute(’label_accuracy’, ’micro’)  # 0.800
im.compute(’mcc’, ’micro’)           # -0.111
for f in diagnose(im):
    print(f)
# [warning] inflated_floor: ’balanced_accuracy’ cannot fall below 0.4444 on this
#            10-class one-hot problem even if every prediction is wrong ...
# [note]    micro_collapse: micro-precision, micro-recall and micro-F1 are all
#            equal to accuracy ...

In a model-selection loop, the scorer interface applies the same measures:

from aggmetrics import make_scorer
from sklearn.model_selection import cross_val_score
cross_val_score(clf, X, y, cv=5, scoring=make_scorer(’mcc’, ’macro’, task=’multiclass’))

Comparison with existing software

scikit-learn and torchmetrics. scikit-learn ([5]) and torchmetrics ([1]) provide accurate, widely used implementations of a fixed catalogue of measures with an averaging argument. aggmetrics does not replace them for production metric computation (torchmetrics targets GPU, streaming and distributed accumulation, which aggmetrics does not); it offers the full measure–operator grid, explicit undefined-value conventions, and the diagnostic layer. Its scorers interoperate with scikit-learn; for F1F_1 on a random multilabel problem the four scikit-learn averages are reproduced to machine precision (with convention=’zero’ for the exemplar average). imbalanced-learn ([4]) adds measures for imbalanced data, such as the geometric mean and index balanced accuracy, again as a fixed list.

PyCM and yardstick. PyCM ([2]) derives a large set of per-class and overall statistics from a confusion matrix and is the closest Python package in breadth of measures; it takes a confusion matrix as input, so multilabel and soft-label settings are not representable, and it does not offer user-defined measures with derived properties. The R package yardstick ([3]) supports macro, weighted macro and micro estimators for multiclass metrics. To our knowledge none of these packages attaches structural properties to measures or warns about floor effects, but we have not benchmarked their behaviour on the cases above beyond reading their documentation.

Project status

Licence and dependencies. BSD 3-Clause. The core requires only NumPy (≥1.20\ge1.20); scikit-learn and pandas are optional (scorers and tabular reports). Pure Python, no compilation, no proprietary dependencies; Python ≥3.9\ge3.9. About 1800 lines of source.

Testing. The suite has 209 tests and runs in about three seconds. Each theoretical result of the companion paper is an executable test (e.g. micro = denominator-weighted macro for the decomposable measures, uniqueness of the product t-norm, micro-collapse, Bayes thresholds, attainable floors, recovery of MAE/MSE from cost matrices), so a change that breaks a theorem breaks a test. Line and branch coverage are both 100%, and continuous integration (GitHub Actions) fails if they drop. It runs the tests, the doctests and the worked example on Linux, macOS and Windows with Python 3.9–3.12.

Documentation and maintenance. The repository contains a README with a feature tour, a runnable examples/quickstart.py, docstrings on all public functions, and a machine-readable guide for AI coding assistants (.agents/skills/). Version 0.1.0 is the version under review. It is archived on Zenodo (10.5281/zenodo.23535051), and the documentation, with a gallery of example notebooks, is at https://aggmetrics.readthedocs.io. A CITATION.cff is provided.

Conclusion

aggmetrics makes the aggregation operator an explicit parameter for every measure, keeps the properties that determine which identities hold alongside the measures, and uses them to flag misleading evaluations. Planned work includes sparse backends for large label spaces, a streaming accumulator, and an R implementation.

Acknowledgments

This research was partially supported by CNPq and FAPESP, Brazil.

References

  1. [1]Nicki Skafte Detlefsen, Jiri Borovec, Justus Schock, Ananya Harsh Jha, Teddy Koker, Luca Di Liello, Daniel Stancl, Changsheng Quan, Maxim Grechkin, and William Falcon. TorchMetrics — measuring reproducibility in PyTorch. Journal of Open Source Software, 7(70):4101, 2022.
  2. [2]Sepand Haghighi, Masoomeh Jasemi, Shaahin Hessabi, and Alireza Zolanyari. PyCM: Multiclass confusion matrix library in Python. Journal of Open Source Software, 3(25):729, 2018.DOI
  3. [3]Max Kuhn, Davis Vaughan, and Emil Hvitfeldt. yardstick: Tidy Characterizations of Model Performance, 2024. URL https://yardstick.tidymodels.org. R package.
  4. [4]Guillaume Lemaître, Fernando Nogueira, and Christos K. Aridas. Imbalanced-learn: A Python toolbox to tackle the curse of imbalanced datasets in machine learning. Journal of Machine Learning Research, 18(17):1–5, 2017.arxiv.org/abs/1609.06570
  5. [5]Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12(85):2825–2830, 2011.arxiv.org/abs/1201.0490
  6. [6]Ronaldo C. Prati. A unified algebraic framework for classification performance evaluation. Information Fusion, 138:104767, 2027. doi: 10.1016/j.inffus.2026.104767.DOI

Paper details

vlnc:W_75vfajf3
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents