betaFeedback

WorksMathematics

Surrogate Losses for Fairness-Aware Classification: Fisher Consistency and Calibration

Abstract

Fairness-aware classification seeks to balance predictive accuracy and fairness across demographic groups. Although surrogate losses facilitate the optimization of fairness-aware objectives, it remains unclear when minimizing these losses yields optimal decisions under the original fairness-aware risk. This paper investigates the statistical validity of surrogate-based fairness-aware classification under demographic parity and equal opportunity, considering both attribute-blind and attribute-aware settings. We introduce fairness-aware Fisher consistency and characterize it through conditions on the derivatives of the surrogate losses. The characterization shows that consistency depends not only on the individual loss functions but also on their interaction within the fairness-regularized objective. We further establish a calibration inequality that relates the excess fairness-aware risk to the excess surrogate risk, providing a quantitative connection between surrogate optimization and the original classification objective. Numerical experiments illustrate how deviations from the consistency conditions affect fairness-aware excess risk and how the regularization parameter influences the well-posedness of surrogate optimization. An application to consumer credit default prediction demonstrates that appropriately calibrated fairness regularization can substantially reduce demographic disparities while maintaining predictive performance.

Introduction

Statistical classification models have become increasingly prevalent in high-stakes decision-making applications, including hiring screening [19], criminal risk assessment [1], and healthcare resource allocation [18].

While these models often achieve strong predictive performance, a growing body of empirical evidence [5] has revealed that they may systematically disadvantage certain demographic groups defined by sensitive attributes such as gender, race, or ethnicity. Such disparities can lead to unequal treatment across groups and may perpetuate or even amplify existing social inequalities when algorithmic decisions influence access to employment opportunities, judicial outcomes, or healthcare services. Consequently, developing classification models that balance predictive accuracy and fairness has become an important objective in the modern statistical learning community.

To address fairness concerns, prior work has proposed a range of group fairness notions for classification that formalize equity requirements across groups. Prominent examples include Demographic Parity (DP) [7], which requires equal selection rates across groups and Equal Opportunity (EO) [10], which mandates equal true positive rates across groups. These definitions provide measures of algorithmic fairness and are widely used to regulate classifier behavior and mitigate discriminatory outcomes.

Based on these fairness notions, researchers have developed a variety of fairness-aware classification approaches by incorporating fairness violations directly into the learning objective. A common approach [17, 3] is to formulate fairness-aware classification as a fairness-regularized risk minimization problem (See Eq. (6)), balancing predictive accuracy against fairness violations through a trade-off parameter. Because both the classification error and fairness violations are typically defined through indicator functions over discrete events (See Eq. (3)), the corresponding optimization problem is generally non-convex and discontinuous.

From an optimization perspective, existing approaches to solve the fairness-regularized risk minimization problem fall into two methodological streams. The first stream attempts to directly optimize the original fairness-aware problem based on the 00–11 loss and exact fairness violations, resulting in a discrete optimization problem. To handle the discrete optimization, mixed-integer programming approaches [23, 29] are adopted for fairness-aware classification. Although the first methodological stream targets the original optimization problem, it typically suffers from substantial computational complexity and limited scalability on large-scale datasets [6].

The second stream takes a different approach by replacing the discontinuous components with convex and smooth surrogate functions. In this paradigm, the 00–11 loss is substituted with surrogate losses such as the logistic, hinge, or exponential loss, while fairness violations are also approximated using surrogate functions [3]. To fix notation, let LL and Lˉ\bar{L} denote the surrogate losses for the accuracy and fairness terms, with risks RL(f)\mathcal{R}_{L}(f) and RLˉ(f)\mathcal{R}_{\bar{L}}(f). Surrogate-based methods then minimize

RfairL,Lˉ(f)≔RL(f)−λ⋅RLˉ(f),\mathcal{R}^{L,\bar{L}}_{\mathrm{fair}}(f) \coloneqq\mathcal{R}_{L}(f) - \lambda\cdot\mathcal{R}_{\bar{L}}(f),

where λ≥0\lambda\ge0 is the trade-off parameter. We formalize these definitions in § 2.4 and denote RL(f)\mathcal{R}_{L}(f), RLˉ(f)\mathcal{R}_{\bar{L}}(f) and RfairL,Lˉ(f)\mathcal{R}^{L,\bar{L}}_{\mathrm{fair}}(f) as LL-risk, Lˉ\bar{L}-risk and (L,Lˉ)(L,\bar{L})-risk with respect to ff respectively. This optimization problem can be efficiently optimized using standard gradient-based algorithms. As a result, surrogate-based methods are computationally scalable and have become the dominant approach for fairness-aware classification [30, 11].

The widespread adoption of surrogate-based methods naturally raises a fundamental question: when does minimizing a surrogate objective lead to a statistically valid solution for the original learning problem? In standard classification, this question is addressed through two foundational theoretical concepts. The first is Fisher consistency, which examines whether minimizing the surrogate LL-risk recovers the Bayes-optimal classifier for the underlying 0–1 loss [16] (Yang, Huang, Huang & Chang 2021; Yang, Guo & Chang 2021). The second is classification calibration, which establishes quantitative relationships between the excess LL-risk and the excess risk through calibrated inequalities [2]. However, much less is known about whether these guarantees continue to hold in fairness-aware classification, where recent finite-sample and distribution-free results focus on the fairness-accuracy trade-off [12] (Li et al. 2023) rather than on the consistency and calibration of surrogate fairness-aware risks. Unlike standard classification, fairness-aware objectives simultaneously balance predictive accuracy and fairness violation through a fairness-regularized risk formulation. This raises two fundamental questions:

Q1. (Fairness-Aware Fisher Consistency): Under what conditions does the population minimizer of a (L,Lˉ)(L,\bar{L})-risk induce the Bayes-optimal classifier for the corresponding fairness-aware risk?

Q2. (Fairness-Aware Calibration): Can we establish a calibration inequality that relates the excess (L,Lˉ)(L,\bar{L})-risk to the excess fairness-aware risk?

This paper addresses these questions by developing a statistical learning theory for surrogate-based fairness-aware classification. To answer the first question, we introduce the notion of Fairness-Aware Fisher Consistency (FAFC). We begin with a pointwise analysis of the fairness-aware conditional risk and identify a fairness-aware discriminant that determines the Bayes-optimal classifier for the original fairness-aware risk. We then define FAFC for the fairness-aware surrogate loss pointwise. At every pair of conditional probabilities at which the fairness-aware discriminant does not vanish, the smallest conditional risk achievable by a wrong-sign prediction is strictly larger than the optimal conditional risk. We then establish necessary and sufficient conditions under which LfairL_{\mathrm{fair}} satisfies FAFC. To answer the second question, we construct a fairness-aware calibration function by minimizing the pointwise wrong-sign excess (L,Lˉ)(L,\bar{L})-risk over level sets of the fairness-aware discriminant. By taking its convex envelope, we obtain a calibrated inequality that controls the excess fairness-aware risk in terms of the excess (L,Lˉ)(L,\bar{L})-risk. Consequently, under the proposed FAFC conditions, convergence of the excess (L,Lˉ)(L,\bar{L})-risk to zero implies convergence of the excess fairness-aware risk to zero, thereby providing a formal justification for surrogate-based fairness-aware classification.

Our main contributions are summarized as follows:

  • A pointwise framework and FAFC characterization. We derive the fairness-aware conditional risk for DP and EO in both the attribute-blind and attribute-aware settings, and identify a fairness-aware discriminant that determines the Bayes-optimal classifier. Based on it, we introduce FAFC for surrogate loss pairs (L,Lˉ)(L,\bar{L}), and give necessary and sufficient conditions for FAFC in terms of the derivatives of the partial losses at the origin, which can be checked directly from the loss pair.

  • A calibration inequality for fairness-aware risks. We construct a fairness-aware calibration function by minimizing the pointwise wrong-sign excess (L,Lˉ)(L,\bar{L})-risk over level sets of the discriminant, and show that its convex envelope yields an inequality bounding the excess fairness-aware risk by the excess (L,Lˉ)(L,\bar{L})-risk. We further show that this envelope is invertible if and only if the pair is FAFC, so that FAFC is exactly the condition under which surrogate risk convergence implies convergence of the fairness-aware risk.

  • Closed-form guarantees for widely used surrogates. We verify FAFC and derive explicit calibration functions and excess-risk bounds for fairness-aware logistic, hinge, and exponential losses, covering DP and EO in both feature settings, and we propose a general surrogate loss-based algorithm that can be used with any FAFC-

compliant pair. Simulations and a credit-default application illustrate the accuracy–fairness trade-off.

Preliminaries

We begin by introducing the notation and basic theory of cost-sensitive classification, since the fairness-aware objectives studied in this paper can be naturally represented and analyzed within the cost-sensitive learning framework [17].

Cost-Sensitive Classification and Surrogate Loss

Let (X,Y)(X,Y) be a random pair drawn from a distribution DD on X×{−1,1}\mathcal{X} \times\{-1,1\}, where X\mathcal{X} is the feature space and Y∈{−1,1}Y \in\{-1,1\} is the ground-truth label. Let F\mathcal{F} denote the set of all measurable functions f:X→Rf:\mathcal{X} \to\mathbb{R}; each ff induces a classifier through the decision rule Y^=sign⁡(f(X))\widehat{Y} = \operatorname{sign}(f(X)), with the convention sign⁡(0)=−1\operatorname{sign}(0) = -1.

Definition 1 (Cost-Sensitive Risk and Bayes-Optimal Classifier). For a classifier f∈Ff \in\mathcal{F} and a distribution DD, the cost-sensitive risk with cost parameter α∈(0,1)\alpha\in(0,1) is defined as

Rα(f;D)≔(1−α)⋅P(Y^=−1,Y=1)+α⋅P(Y^=1,Y=−1),(1)\mathcal{R}_{\alpha}(f;D) \coloneqq(1-\alpha) \cdot\mathbb{P}(\widehat{Y}=-1,Y=1) + \alpha\cdot\mathbb{P}(\widehat{Y}=1,Y=-1), \tag*{(1)}

where Y^=sign⁡(f(X))\widehat{Y} = \operatorname{sign}(f(X)) is the prediction label induced by ff. Then Bayes-optimal classifier is:

γ(x)∈arg⁡min⁡f∈FRα(f;D),(2)\gamma(x) \in\arg\min_{f \in\mathcal{F}} \mathcal{R}_{\alpha}(f;D), \tag*{(2)}

where γ(x)=η(x)−α\gamma(x) = \eta(x) - \alpha and η(x)≔P(Y=1∣X=x)\eta(x) \coloneqq\mathbb{P}(Y=1 \mid X=x) is the posterior probability of the positive class. The cost parameter α\alpha controls the relative penalty on false negatives versus false positives; when α=0.5\alpha= 0.5, Rα(f;D)\mathcal{R}_{\alpha}(f;D) reduces to the misclassification error rate P(Y^≠Y)\mathbb{P}(\widehat{Y} \ne Y).

Equivalently, the risk can be written as the expectation of a pointwise cost-sensitive 0–1 loss [21]. Let us define

Uα(y,t)≔(1−α)⋅1{y=1}1{t≤0}+α⋅1{y=−1}1{t>0},(3)U_{\alpha}(y,t) \coloneqq(1-\alpha)\cdot1\{y=1\}1\{t\le0\}+\alpha\cdot1\{y=-1\}1\{t>0\}, \tag*{(3)}

then Rα(f;D)=E(X,Y)∼D[Uα(Y,f(X))]\mathcal{R}_{\alpha}(f;D)=\mathbb{E}_{(X,Y)\sim D}[U_{\alpha}(Y,f(X))]. Here 1{⋅}1\{\cdot\} denotes the indicator function.

The 0–1 loss UαU_{\alpha} is discontinuous and non-convex, so direct minimization of Rα(⋅;D)\mathcal{R}_{\alpha}(\cdot;D) from finite data is computationally intractable. The standard remedy in statistical learning is to replace UαU_{\alpha} by a smooth, convex surrogate loss L:{−1,1}×R→[0,∞)L:\{-1,1\}\times\mathbb{R}\to[0,\infty). Any surrogate loss admits the partial-loss decomposition

L(y,t)≔1{y=1}L1(t)+1{y=−1}L−1(t),(4)L(y,t)\coloneqq1\{y=1\}L_1(t)+1\{y=-1\}L_{-1}(t), \tag*{(4)}

where L1(t)≔L(1,t)L_1(t)\coloneqq L(1,t) and L−1(t)≔L(−1,t)L_{-1}(t)\coloneqq L(-1,t) denote the partial losses for the positive and negative labels, respectively. Replacing UαU_{\alpha} by LL in Eq. (1) yields the LL-risk

RL(f;D)≔E(X,Y)∼D[L(Y,f(X))].\mathcal{R}_L(f;D)\coloneqq\mathbb{E}_{(X,Y)\sim D}[L(Y,f(X))].

We denote the minimal risk by Rα∗≔inf⁡f∈FRα(f;D)\mathcal{R}_{\alpha}^{*}\coloneqq\inf_{f\in\mathcal{F}}\mathcal{R}_{\alpha}(f;D) and the minimal LL-risk by RL∗≔inf⁡f∈FRL(f;D)\mathcal{R}_{L}^{*}\coloneqq\inf_{f\in\mathcal{F}}\mathcal{R}_{L}(f;D).

To validate that minimizing the LL-risk yields a statistically valid solution for the original optimization problem, we require two theoretical guarantees: (1) a qualitative asymptotic guarantee that the surrogate recovers the Bayes-optimal classifier, and (2) a quantitative bound that links the excess LL-risk to the excess risk. We formalize these two properties in the remainder of this subsection.

Fisher Consistency

Fisher consistency is the fundamental asymptotic guarantee for surrogate optimization: the population minimizer of the LL-risk must recover the Bayes-optimal classifier of the original risk [8, 16].

Definition 2 (Fisher Consistency for Cost-Sensitive Classification). A surrogate loss LL is α\alpha-Fisher consistent for cost-sensitive classification with cost α∈(0,1)\alpha\in(0,1) if, for every distribution DD and every minimizer f∗∈arg min⁡f∈FRL(f;D)f^* \in\operatorname*{arg\,min}_{f\in\mathcal{F}} \mathcal{R}_L(f;D), its sign matches that of the cost-sensitive Bayes-optimal classifier:

sign⁡(f∗(x))=sign⁡(γ(x))for PX-a.e. x∈X with γ(x)≠0,\operatorname{sign}(f^*(x)) = \operatorname{sign}(\gamma(x)) \quad\text{for } P_X\text{-a.e. } x \in\mathcal{X} \text{ with } \gamma(x) \ne0,

where γ(x)=η(x)−α\gamma(x) = \eta(x) - \alpha is the cost-sensitive discriminant in Eq. (2).

Satisfying Fisher consistency guarantees that, given the population distribution, minimizing the surrogate loss yields the same classification decisions as the Bayes-optimal classifier almost surely. However, this guarantee is stated at the level of population minimizers and is not directly checkable from the loss. The following result of Scott [21] shows that, for convex partial losses differentiable at 00, Fisher consistency at cost α\alpha reduces to a local condition on the derivatives of L1L_1 and L−1L_{-1} at the origin, which can be read off from the loss alone.

Lemma 3. Let LL be a surrogate loss with convex partial losses L1,L−1L_1, L_{-1} differentiable at 00, and let α∈(0,1)\alpha\in(0,1). Then LL is Fisher consistent at cost α\alpha if and only if

L1′(0)<0,L−1′(0)>0,αL1′(0)+(1−α)L−1′(0)=0.L_1'(0) < 0,\qquad L_{-1}'(0) > 0,\qquad\alpha L_1'(0) + (1-\alpha)L_{-1}'(0) = 0.

Classification Calibration

Beyond the asymptotic guarantee of Fisher consistency, classification calibration provides a quantitative bound relating the excess LL-risk to the excess risk through a calibration function [2, 21].

Lemma 4 (Classification Calibration for Cost-Sensitive Classification). Let LL be a non-negative surrogate loss and α∈(0,1)\alpha\in(0,1).

  1. There exists a convex, non-decreasing calibration function ψL,α:[0,∞)→[0,∞)\psi_{L,\alpha} : [0,\infty) \to[0,\infty) with ψL,α(0)=0\psi_{L,\alpha}(0) = 0 such that, for every measurable classifier f∈Ff \in\mathcal{F} and every distribution DD,

ψL,α(Rα(f;D)−Rα∗)≤RL(f;D)−RL∗.\psi_{L,\alpha}\left(\mathcal{R}_{\alpha}(f;D)-\mathcal{R}_{\alpha}^{*}\right) \le\mathcal{R}_{L}(f;D)-\mathcal{R}_{L}^{*}.
  1. The calibration function ψL,α\psi_{L,\alpha} is invertible if and only if LL is Fisher consistent in the sense of Definition 2.

Fairness-Aware Classification

Fairness-aware classification extends the cost-sensitive setting of § 2.1 by adding a binary sensitive attribute A∈{−1,1}A \in\{-1,1\} (e.g., gender or race) alongside the target label YY. We consider both the attribute-aware setting, in which AA is available at training and prediction time, and the attribute-blind setting, in which AA is available at training time only [17]. In either setting, the goal is to learn a classifier f∈Ff \in\mathcal{F} that balances accuracy with respect to YY against group fairness with respect to AA.

Group fairness is formalized as a statistical parity condition on the predicted label Y^=sign⁡(f(X))\widehat{Y} = \operatorname{sign}(f(X)) across the two groups. We focus on two of the most widely adopted criteria, Demographic Parity (DP) and Equal Opportunity (EO).

References

  1. [1]Angwin, J., Larson, J., Mattu, S. & Kirchner, L. (2022), ‘Machine bias’, Ethics of Data and Analytics pp. 254–264.DOI
  2. [2]Bartlett, P. L., Jordan, M. I. & McAuliffe, J. D. (2006), ‘Convexity, classification, and risk bounds’, Journal of the American Statistical Association 101(473), 138–156.DOI
  3. [3]Bechavod, Y. & Ligett, K. (2018), ‘Penalizing unfairness in binary classification’.arxiv.org/abs/1707.00044
  4. [4]Calders, T. & Verwer, S. (2010), ‘Three naive bayes approaches for discrimination-free classification’, Data mining and knowledge discovery 21(2), 277–292.DOI
  5. [5]Caton, S. & Haas, C. (2024), ‘Fairness in machine learning: A survey’, ACM Computing Surveys 56(7), 1–38.DOI
  6. [6]Donini, M., Oneto, L., Ben-David, S., Shawe-Taylor, J. & Pontil, M. (2018), Empirical risk minimization under fairness constraints, in ‘Proceedings of the 32nd International Conference on Neural Information Processing Systems’, NIPS’18, Curran Associates Inc., Red Hook, NY, USA, p. 2796–2806.arxiv.org/abs/1802.08626
  7. [7]Dwork, C., Hardt, M., Pitassi, T., Reingold, O. & Zemel, R. (2012), Fairness through awareness, in ‘Proceedings of the 3rd Innovations in Theoretical Computer Science Conference’, ITCS ’12, Association for Computing Machinery, New York, NY, USA, p. 214–226.
  8. [8]Elkan, C. (2001), The foundations of cost-sensitive learning, in ‘Proceedings of the 17th International Joint Conference on Artificial Intelligence - Volume 2’, IJCAI’01, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, p. 973–978.
  9. [9]Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C. & Venkatasubramanian, S. (2015), Certifying and removing disparate impact, in ‘Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining’, KDD ’15, Association for Computing Machinery, New York, NY, USA, p. 259–268.arxiv.org/abs/1412.3756
  10. [10]Hardt, M., Price, E. & Srebro, N. (2016), ‘Equality of opportunity in supervised learning, in ‘Proceedings of the 30th International Conference on Neural Information Processing Systems’, NIPS’16, Curran Associates Inc., Red Hook, NY, USA, p. 3323–3331.arxiv.org/abs/1610.02413
  11. [11]He, X. & Li, Y. (2025), ‘Fairness in machine learning: A review for statisticians’, Journal of the American Statistical Association 120(552), 2834–2851.DOI
  12. [12]Hou, X. & Zhang, L. (2024), ‘Finite-sample and distribution-free fair classification: Optimal trade-off between excess risk and fairness, and the cost of group-blindness’.arxiv.org/abs/2410.16477
  13. [13]Kamiran, F. & Calders, T. (2009), Classifying without discriminating, in ‘2009 2nd International Conference on Computer, Control and Communication’, pp. 1–6.DOI
  14. [14]Kleinberg, J., Mullainathan, S. & Raghavan, M. (2017), Inherent Trade-Offs in the Fair Determination of Risk Scores, in C. H. Papadimitriou, ed., ‘8th Innovations in Theoretical Computer Science Conference (ITCS 2017)’, Vol. 67 of Leibniz International Proceedings in Informatics (LIPIcs), Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl, Germany, pp. 43:1–43:23.arxiv.org/abs/1609.05807
  15. [16]Lin, Y. (2004), ‘A note on margin-based loss functions in classification’, Statistics & Probability Letters 68(1), 73–82.DOI
  16. [17]Menon, A. K. & Williamson, R. C. (2018), The cost of fairness in binary classification, in S. A. Friedler & C. Wilson, eds, ‘Proceedings of the 1st Conference on Fairness, Accountability and Transparency’, Vol. 81 of Proceedings of Machine Learning Research, PMLR, pp. 107–118.
  17. [18]Obermeyer, Z., Powers, B., Vogeli, C. & Mullainathan, S. (2019), ‘Dissecting racial bias in an algorithm used to manage the health of populations’, Science 366(6464), 447–453.
  18. [19]Raghavan, M., Barocas, S., Kleinberg, J. & Levy, K. (2020), Mitigating bias in algorith-mic hiring: Evaluating claims and practices, in ‘Proceedings of the 2020 conference on fairness, accountability, and transparency’, pp. 469–481.
  19. [20]Rockafellar, R. T. (1970), Convex Analysis, number 28 in ‘Princeton Mathematical Series’, Princeton University Press, Princeton, NJ.
  20. [21]Scott, C. (2012), ‘Calibrated asymmetric surrogate losses’, Electronic Journal of Statistics 6, 958–992.DOI
  21. [22]Shen, X., Wong, Y. & Kankanhalli, M. (2023), ‘Fair representation: Guaranteeing approxi-mate multiple group fairness for unknown tasks’, IEEE Transactions on Pattern Analysis and Machine Intelligence 45(1), 525–538.DOI
  22. [23]Ustun, B. & Rudin, C. (2016), ‘Supersparse linear integer models for optimized medical scoring systems’, Machine Learning 102(3), 349–391.DOI
  23. [24]Verbraken, T., Bravo, C., Weber, R. & Baesens, B. (2014), ‘Development and application of consumer credit scoring models using profit-based classification measures’, European Journal of Operational Research 238(2), 505–513.
  24. [28]Yang, Y., Huang, Y. & Chang, X. (2026), Bayes-optimal fair classification with multiple sensitive features, in ‘Proceedings of the AAAI Conference on Artificial Intelligence’, Vol. 40, pp. 27693–27701.
  25. [29]Yang, Y., Wu, Y., Chang, X., Li, M. & Tan, Y. (2025), ‘Toward a fairness-aware scoring system for algorithmic decision-making’, Production and Operations Management p. 10591478251318918.arxiv.org/abs/2109.10053
  26. [30]Zafar, M. B., Valera, I., Gomez-Rodriguez, M. & Gummadi, K. P. (2019), ‘Fairness constraints: A flexible approach for fair classification’, Journal of Machine Learning Research 20(75), 1–42.
  27. [31]Zeng, X., Jiang, K., Cheng, G. & Dobriban, E. (2026), ‘Bayes-optimal fair classification with linear disparity constraints via pre-, in-, and post-processing’, Journal of Machine Learning Research 27(65), 1–87.

Paper details

vlnc:W_7mbh57hz
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents