WorksComputer Science
Overcoming the Resolution Limit: Significance-Aware Regularization for Intersectional Fairness
Abstract
Intersectional fairness requires evaluating models across fine-grained subgroups. However, as group definitions become more granular, the sample size per subgroup shrinks, causing fairness estimates to become increasingly unstable. Existing in-processing methods are certainty-blind: they penalize observed disparities without distinguishing statistically certified unfairness from sampling noise. This leads to fairness overfitting, where models sacrifice utility to "fix" spurious disparities in sparse regimes. To address this, we introduce the significant-excess metric family, a continuous, uncertainty-aware fairness score that measures only the portion of a subgroup's disparity exceeding what sampling variation can explain at a specific confidence level. Building on this metric, we propose SAFER (Significance-Aware Fairness Regularizer), a differentiable in-processing method that gates fairness penalties using a soft-count likelihood-ratio evidence statistic. SAFER satisfies a provable safety property: when every subgroup disparity lies well below the gate threshold, the unconstrained optimum is nearly stationary for SAFER. At that optimum, the back-propagated contribution of its significance-gated component is exponentially small, and the additional fairness contribution comes from a small ungated term with a fixed weight. Mechanism ablations across three benchmark datasets and purpose-built stress variants show that removing any component of SAFER (threshold smoothing, the significance gate, or a dense-tail term) produces the failure the theory predicts on at least one stress variant. In sparse intersectional settings, SAFER achieves a better fairness-utility Pareto frontier than the baseline methods, and it preserves utility when certified unfairness is absent.
Introduction
Intersectional fairness [8, 16] requires evaluating models across conjunctions of protected attributes, but as subgroups become more granular, their sample sizes inevitably shrink. In these sparse regimes, fairness estimates become noise-dominated, and intersectional fairness reaches a resolution limit: the point at which the statistical uncertainty exceeds the disparity being measured [38, 14, 13]. Indeed, even widely adopted heuristics such as the four-fifths rule lack statistical grounding and can both miss and fabricate disparities [35]. Ignoring estimation uncertainty leads to fairness overfitting, where the model memorizes group noise in the training set rather than genuine bias [28]. Existing in-processing methods are certainty-blind: they penalize observed disparities without distinguishing certified unfairness from sampling noise, causing fairness overfitting in the sparse intersectional regime. Conversely, audit tools such as [13] incorporate uncertainty awareness, but only as a post-hoc binary accept/reject decision rather than a continuous metric or a differentiable training signal. If fairness regularization is to behave sensibly in sparse intersectional regimes, the optimization criterion itself must be able to distinguish well-supported disparities from noisy ones.
40th Conference on Neural Information Processing Systems (NeurIPS 2026). Whether gating on significance helps depends on how many training subgroups lie above the gate threshold. If none do, the gated fairness contributions are suppressed. If some do, the gate selects which disparities receive greater weight. If all do, the gate is effectively inactive.
In this context, this paper makes three main contributions:
Uncertainty-aware fairness measurement (Section 3). We introduce the significant-excess metric family, which measures only the portion of each subgroup’s disparity that exceeds what sampling variation can explain at a given confidence level. The construction applies to any parity criterion expressible as a continuously differentiable function of multinomial cell probabilities, including Statistical Parity [11], Disparate Impact [12], Equalized Odds and Equal Opportunity [17], and yields a continuous, uncertainty-aware fairness score that is zero when no violation is statistically certified.
Significance-gated optimization (Section 4). We propose SAFER (Significance-Aware Fairness Regularizer), a differentiable in-processing method that gates fairness penalties using a soft-count likelihood-ratio evidence statistic. The gate attenuates subgroup penalties according to their evidence of disparity; statistical certification comes from the held-out audit. SAFER satisfies a provable safety property: when every disparity lies well below the gate threshold, the optimum of unconstrained empirical risk minimization (ERM) is nearly stationary for SAFER. At that optimum, the back-propagated contribution of its significance-gated component is exponentially small, and the additional fairness contribution comes from a small ungated term with a fixed weight.
Regime characterization and empirical validation (Section 5 and Appendix F.3). We identify three operating regimes (safety, selective gating, gate always open), defined by which training subgroups cross the gate’s threshold, and test each through mechanism ablations on purpose-built stress variants and a controlled benchmark.
The claims are regime-conditional: SAFER gives a better Pareto frontier than the baselines where certified disparities exist and preserves the utility of the unconstrained model where they do not, and removing any component of SAFER produces the failure the theory predicts on at least one stress variant.
Related Work
Intersectional and subgroup fairness. Equality of Opportunity [17] and related parity criteria are the standard building blocks for group fairness, but models can satisfy coarse constraints yet violate fairness on structured intersections [25]. Differential fairness treats intersectionality as a first-class object [23]; worst-case comparisons formalize the combinatorial structure of multi-attribute subgroups [15]; and multicalibration [19] provides simultaneous calibration guarantees over rich subgroup families. Molina and Loiseau [32] bound intersectional violations via marginal constraints; Himmerleich et al. [20] formalize the tension between group granularity and statistical power. Our work targets the same setting but a different bottleneck: once many intersectional groups are audited simultaneously, their disparities become heterogeneous in statistical reliability, and this reliability must enter both evaluation and optimization.
Uncertainty-aware fairness. SAFT [13] is the work closest to ours: it tests whether each subgroup’s confidence interval excludes parity, applying multiplicity corrections across the subgroup family. This makes subgroup size and estimator variance part of the fairness decision; certainty-blind metrics [22] ignore both. We build on SAFT in two directions. First, we replace binary per-subgroup flags with a continuous significant-excess score that preserves severity information beyond the accept/reject decision, yielding a single metric for ranking models (Section 3). Second, this statistical logic motivates a differentiable evidence gate that modulates subgroup penalties during training. SAFER keeps SAFT’s intervals for the audit and trains with a soft-count likelihood-ratio statistic, because a plug-in Wald standard error is unstable on small soft counts and a posterior-sampling test is too expensive inside a training loop (Section 4.2). More broadly, a growing body of work grounds fairness evaluation in statistical inference: Cherian and Candes [7] provide finite-sample valid tests for auditing; Tang et al. [37] replace Wald statistics with empirical-likelihood tests; and Lo et al. [30] advocate classical hypothesis testing for AI fairness. On the evaluation side, Barrainkua et al. [3] show that fairness conclusions are unstable under resampling, and Rosenblatt and Witter [36] benchmark how uncertainty can reverse fairness rankings. Kuzucu et al. [27] treat predictive uncertainty itself as a fairness criterion, whereas we use statistical uncertainty of the fairness estimate to gate optimization. Besse et al. [5] were among the first to propose confidence intervals for testing disparate impact, which replaced point estimates with interval-based decisions.
Fairness-aware optimization. A large literature mitigates unfairness during or after training. Post-processing methods adjust a fixed classifier’s thresholds [17], but cannot reshape the learned representation and become impractical when the number of intersectional subgroups is combinatorially large. Among in-processing approaches, Exponentiated Gradient [1] solves a sequence of weighted learning problems, Adversarial Debiasing [40] and Prejudice Remover [24] embed fairness pressure through an adversary or a dependence penalty, and differentiable regularizers such as Bechavod and Ligett [4] and FAIRRET [6] penalize group discrepancy via smooth surrogates. Recent intersectional-specific optimizers include Lee et al. [29], who regularize over multitype attributes, and Nemeček et al. [33], who enforce exact intersectional constraints via mixed-integer programming. Both remain certainty-blind. Distributionally robust optimization (DRO) [18] and minimax group fairness [9] minimize worst-case subgroup loss, while FairDRO [34] casts group fairness as classwise robust optimization. All these methods are certainty-blind: once a subgroup disparity is observed, it contributes to the objective regardless of whether it is statistically distinguishable from sampling noise. SAFER differs by using a likelihood-ratio evidence gate to modulate disparity penalties during training, while retaining the held-out audit for statistical certification.
Measuring Intersectional Fairness
We introduce the significant-excess metric family to address the resolution limit identified in § 1. To this end, these metrics are designed to answer two questions simultaneously: does a subgroup exhibit a non-zero disparity compared to the rest of the population, and is that disparity statistically significant enough to be distinguished from sampling noise?
Setup. Let be a family of protected subgroups, including intersectional groups formed by conjunctions of sensitive attributes. For each subgroup , we compare the behavior of a model on and its complement . Both are sets of individuals: collects those with the subgroup’s attribute values and the rest of the population. We write for a generic fairness functional expressible as a continuously differentiable function of multinomial cell probabilities. We write for its empirical estimate on a test sample of size . For each subgroup disparity we require a confidence interval of level , expressible by a lower and upper bound , which quantifies the uncertainty of the estimation. We assume .
Definition 1 (Significant excess). For a subgroup and significance level , we define the significant excess of fairness criterion as
The significant excess is the distance from zero to the interval. When the interval contains zero, the disparity is not statistically certified and the contribution is exactly zero. When the interval excludes zero, measures the portion of the observed disparity that remains after subtracting what sampling noise can plausibly explain.
Basic properties. The significant-excess score satisfies:
Zero iff no certified unfairness under the interval rule. if and only if the interval for contains zero.
Scale preservation. is expressed in the same units as .
Uncertainty awareness. The score is nonincreasing when the interval expands by inclusion.
Continuity. The score is continuous in the interval endpoints.
Finally, we refer to the Mean Significant Measure Excess as the mean aggregation over the nonempty audited family of subgroups with defined estimates and intervals:
Let us consider two fairness metrics as an example, the Statistical Parity (SP) gap and the Equal Opportunity (EO) gap:
For SP, the relevant cells are predicted positive/negative outcomes conditioned on and . For EO, the same construction is applied after conditioning on the positive-label population . Both can be written as continuously differentiable functions of multinomial cell probabilities. Given a confidence level , we can define the Mean Significant Parity Excess (MSPE) by applying Equations (1) and (2) with , and the Mean Significant Equal-Opportunity Excess (MSEE) with .
We obtain the confidence intervals with the SAFT framework [13], because its test adapts to both large and small subgroups.
SAFER: A Differentiable Surrogate for Significant-Excess Metrics
(§ 3) measures only the statistically certified portion of each subgroup’s disparity. It is not directly trainable, because it depends on hard predictions, which are piecewise constant in , and on an absolute value and a positive part, which are not differentiable at zero. SAFER replaces these with differentiable approximations and is built around two ideas. First, a soft-count likelihood-ratio test (LRT) provides a differentiable significance statistic that requires no explicit variance estimation. This matters in the sparse intersectional cells, where Wald-type statistics become unreliable (§4.2). Second, a significance gate built from this statistic yields a provable safety property: when every subgroup disparity lies well below the gate threshold, the ERM optimum is nearly stationary for SAFER. At that optimum, the back-propagated contribution of the significance-gated component is exponentially small, and the additional fairness contribution is proportional to the weight of a small ungated term (§4.3). Proofs are in Appendix § 17.
Threshold-Smoothed Soft Counts
Let be the model score. For each subgroup write for the conditioning population: for Statistical Parity, and for Equal Opportunity, matching § 3. Define three population positive rates, relative to the hard binary predictions, the raw predicted probabilities and the smoothed sigmoid:
where is the logistic sigmoid. The corresponding disparity is for each variant . For example in this notation, is exactly the population functional of § 3. Furthermore, we use hats (e.g., ) to denote the empirical analogues computed from sample soft counts. The decision threshold is the Bayes rule when the scores estimate class probabilities and both error types cost the same. It is a property of the deployed classifier and is fixed before training. The construction applies unchanged to any other fixed threshold by replacing with ; Appendix § 13.§ 1 reports SAFER retrained and audited at thresholds from 0.1 to 0.9.
In particular, replacing with yields a differentiable approximation whose quality is controlled linearly by .
Theorem 2. If, for some , the conditional score distributions on and admit densities bounded by on , then .
A natural alternative, using the raw probability directly (), does not work:
Proposition 3 (Raw soft counts are not hard-decision consistent). For every , there exist continuous bounded-density score distributions such that , and no constant rescaling of matches uniformly.
Two groups sitting just across the boundary from opposite sides produce a hard gap near 1 but a raw-probability gap near 0; two groups both well above the boundary produce the reverse. No single correction can fix both regimes, so raw soft counts can rank models differently from the hard-decision audit. Threshold smoothing also localizes gradients to examples near the decision boundary, where predictions can flip (Theorem 7 in Appendix § 17).
A Differentiable LRT for Subgroup Disparities
Given threshold-smoothed soft counts, we need a differentiable evidence statistic to select subgroup disparities for training. An approach would be the Bernoulli-form Wald statistic , where is the two-proportion standard-error proxy of property (ii), but this has two problems in the sparse intersectional regime. First, is numerically unstable when soft counts are small. The second problem matters more for optimization: the Wald loss can decrease by inflating the standard-error estimate rather than reducing the disparity, since both numerator and denominator depend on .
SAFER instead uses the Bernoulli-form two-sample deviance statistic computed on threshold-smoothed soft counts. For subgroup define the soft counts and . Let and define analogous quantities for . Write , , and for the pooled null estimate. The likelihood-ratio statistic for the soft-parity null is
For SP, these are soft counts of predicted positive/negative across and ; for EO, the same construction restricts to the positive-label population.
Three properties make the soft-count LRT a suitable significance statistic for intersectional fairness training:
(i) No variance estimation. is a closed-form function of the soft counts: it never estimates or differentiates through . This removes the gradient path through which the Wald loss can be lowered by inflating the standard error.
(ii) Boundary stability. In sparse cells the Bernoulli-form variance proxy is unstable: with only a few effective observations, threshold-smoothed rates can approach zero or one, the variance estimate collapses toward zero, and grows without bound, so a few tiny, saturated subgroups can dominate the objective. is computed from the Bernoulli log-likelihood itself and contains no division by a plug-in variance estimate that can vanish, so it avoids this boundary singularity.
(iii) Asymptotic equivalence. For interior rates and vanishing disparities, a second-order expansion gives , so the two statistics use the same reference cutoff . For a fixed predictor evaluated on independent observations under soft parity, , , and the Bernoulli reference cutoff is asymptotically conservative (Appendix § 17.§ 6).
During training the gate is an optimization device; statistical certification is provided by the held-out hard-count audit of § 3.
LRT-based threshold scale. To express the significance threshold in the raw-disparity units of when the disparity is nonzero, we define an effective standard-error proxy from the empirical disparity and the deviance:
This conversion puts the dimensionless evidence statistic on the disparity scale, so that the training threshold carries the units of . Under the local conditions of property (iii), , recovering the Bernoulli-form normalization. Before numerical clamping, the threshold therefore scales with for SP and for EO. It thus shrinks as the inverse square root of the sample size when both comparison groups grow in proportion. The implementation regularizes the ratio and clamps it for numerical stability.
The trained objective. SAFER penalizes excess disparity above the LRT-based training threshold, with the threshold detached from the gradient computation:
where and denotes stop-gradient (the enclosed expression is treated as a constant during back-propagation). Detachment ensures that the optimizer can only reduce the loss by shrinking , not by inflating the threshold. The threshold still updates between optimizer steps as changes, so it remains adaptive: it does not provide a gradient path. Within one update the gated term therefore gives the optimizer no first-order incentive to raise the threshold; across updates the threshold follows the predictions, which every term of the objective changes. Appendix J.9 gives a scalar example and a diagnostic over training runs, in which the mean threshold shows no upward drift. A canonical version of Eq. (11), in which the surrogate and the metric share the population Wald threshold , approximates (Eq. (A.8)) pointwise with error at most (Theorem 8 in Appendix J).
Safety Property and Operating Regimes
The softplus gate in separates subgroups above the gate threshold from those below it. Let
Since the stop-gradient (sg) freezes the threshold for differentiation, . Throughout, of an expression containing sg denotes the back-propagated fairness contribution with the detached quantity held fixed; Proposition 4 bounds it. The chain rule and give
When (disparity below the gate threshold), the prefactor decays exponentially as , suppressing the fairness gradient on that subgroup.
Dense tail. Under strong regularization, active disparities may stay close to the hinge boundary where the softplus gate attenuates the gradient substantially: by Eq. (12) the gradient is multiplied by , which equals at the gate boundary and is already below at for , so under strong regularization the optimization can stall before an active disparity is removed. To prevent stalling, SAFER adds a small ungated term:
where . This term carries no gate multiplier, so it provides a gradient even when the softplus is saturated. Since locally expresses the disparity in units of the Bernoulli-form normalization, the tail weights each subgroup by its statistical support (Appendix J.7). Because the term is ungated, the safety bound below includes an additional term proportional to , while the gated part of the gradient keeps its exponential bound; with the objective is gated only. We use the fixed value in all runs of the full method. When active subgroups dominate, the tail is negligible relative to .
Proposition 4 (Safety at the unconstrained optimum). If for every , and the score and statistic gradients are bounded near , then
When every subgroup lies below the gate boundary by a margin large relative to , the ERM optimum is nearly stationary for SAFER: the gated fairness contribution there is exponentially small, and the dense-tail residual (Eq. (13)) enters the full objective multiplied by .
Three operating regimes. The gate threshold determines which subgroups contribute to the fairness gradient. Write for the active set of training subgroups: a subgroup is active when , which, up to the smoothing constant and the numerical clamps, means , with at . At a given checkpoint , the active set places a dataset in one of three regimes:
(i) Safety (): when all subgroups at the ERM point lie well below the boundary relative to , the gated fairness contribution is exponentially small, and the unconstrained optimum is nearly stationary for SAFER up to the dense-tail residual.
(ii) Selective gating (, active and inactive subgroups coexist): the gate concentrates model capacity on the disparities for which the data provide statistical support.
(iii) Gate always open (, all subgroups above the boundary): when all lie well above it relative to , the gate is effectively inactive, and SAFER approaches a certainty-blind penalty.
The regime is read off the active set at a checkpoint; when we characterize a dataset we use the checkpoint of the unconstrained model, and the regime can change during training. Whether a subgroup is active depends on its disparity, on its outcome rates, and on the counts on both sides of the comparison (Appendix § 17.8), so the boundary between regimes is set by the evidence in each subgroup. Appendix § 13.§ 3 states the hypothesis these regimes suggest and tests it on a controlled benchmark.
Full objective. For fairness criterion , SAFER minimizes
where is binary cross-entropy and controls the fairness–utility tradeoff.
Extensions. The construction extends to any fairness functional differentiable in multinomial cell probabilities: substituting SP gives MSPE directly, Equalized Odds adds an FPR-parity hinge, and Disparate Impact uses the log-DI metric with the same SP parity null (Appendix § 16).
Experiments
The experiments serve two purposes. First, we compare SAFER on the Pareto frontier against the strongest available in-processing fairness baselines on three datasets spanning sample sizes and disparity regimes. Second, we use constructed stress variants of the datasets (low training budgets, deeper intersection levels and small validation budgets) to probe the three mechanisms predicted by § 4. Each variant targets a setting in which one SAFER component should be needed, so removing that component in an ablation should have a measurable effect. The full codebase is available at https://github.com/Ambr.../Safer.
Datasets and Stress Variants. Public benchmarks with multiple protected attributes and non-trivial intersectional structures are scarce. The main figures use the three most established ones (Adult Income [26], COMPAS [2] and German Credit [21]). Table 1 summarizes their stress variants, each of which targets one of the mechanisms of § 4. The table also reports ACS Coverage [10] and Taiwan Credit [39], which add a public-coverage task with larger samples and a credit-risk task with many intersections. Unless a train-test split or intersection level is stated explicitly, dataset names below refer to the default variants in Table 1. The default COMPAS variant uses the more severe 20/80 train-test split and is reserved for mechanism ablations; the headline COMPAS comparison instead uses a balanced 50/50 split for a reasonable training budget.
| Dataset | Level | Split | Targeted mechanism | |||
| Adult Income | 4 | 10/90 | 4,522 | 228 | 19.9 / 4.9 | threshold smoothing: high intersectionality, low budget |
| COMPAS | 3 | 20/80 | 993 | 35 | 28.4 / 14.2 | dense tail: small training set, strong regularization |
| German Credit | 3 | 50/50 | 500 | 32 | 15.6 / 10.9 | significance gate: safety regime |
| ACS Coverage | 3 | 50/50 | 25,000 | 71 | 352.1 / 103.5 | higher-sample ACS benchmark |
| Taiwan Credit | 4 | 50/50 | 15,000 | 285 | 52.6 / 11.6 | high-intersectionality credit risk |
Table 1. Datasets, stress variants, and the mechanism each variant probes. is the number of audited intersectional subgroups at the indicated level; is the mean training-set subgroup size; is the mean positive-label subgroup size relevant to EO. All values are approximate averages over splits.
Adult probes the threshold-smoothing mechanism (almost every intersection has few samples). COMPAS probes the dense-tail mechanism under strong regularization (small training set, moderate disparity). German probes the safety regime (almost no SP/EO violation is statistically certifiable at under ERM in our observed splits). ACS Coverage has larger effective subgroup sizes, while Taiwan Credit tests whether the comparison extends to a larger credit-risk task with many audited intersections but a relatively small positive-label budget. Appendix § 8 details preprocessing, protected attributes, subgroup construction, and split-level audit statistics.
Protocol and Metrics Each method and is evaluated over at least 10 stratified shuffle splits, with at least 3 random initialization seeds per split (xref}Appendix B gives the counts per experiment). All headline neural methods share the same MLP backbone, prediction-loss schedule, and held-out validation-only model selection. We report at on hard test predictions (Eq. (5)) against held-out log-loss, with standard error (SE) bars over the runs across splits and seeds. MSEE and MSPE use unadjusted per-subgroup intervals; reported violation counts use Benjamini–Hochberg (BH)-corrected subgroup -values. Hyperparameter grids, BCE-guarded checkpointing, and clustered-SE checks are in Appendix § 9.
Pareto Comparison Against Baselines
We compare SAFER against ERM, EO-Penalty and two FAIRRET [6] regularizers. EO-Penalty is the certainty-blind squared-disparity regularizer () and is the closest direct comparator to the SAFER significant-excess hinge. FAIRRET Norm 1 and FAIRRET Norm 2 are ratio-normalized differentiable fairness regularizers. EO-Penalty and both FAIRRET variants operate on raw soft probabilities rather than threshold-smoothed predictions, which makes them direct empirical tests of the inconsistency claim of Proposition 3. We also report thresholded summaries for Exponentiated Gradient [1], Prejudice Remover [24], and Adversarial Debiasing [40]. Complete per-dataset details are deferred to Appendix § 11, because these methods do not all share the matched neural in-training backbone used in the headline Pareto plots.
Table 2 summarizes the thresholded fairness–utility comparison across all five datasets and all baselines. For each dataset and method, the entry is the best mean test log loss among the swept hyperparameters whose test MSEE is at least 50% below the ERM MSEE on the same dataset. A * indicates that no swept hyperparameter achieved that fairness threshold, so the method did not produce a qualifying tradeoff under this criterion. Appendix § 11 reports the complete per-dataset tables, including the lowest-MSEE point found for each ineligible * case.
Figure 1 shows the EO Pareto frontiers for Adult, COMPAS, and German. Across panels, SAFER preserves near-ERM log-loss while crossing the threshold and reaches lower MSEE than EO-Penalty or FAIRRET without comparable log-loss collapse. Both observations are consistent with the alignment argument of Theorem 2: optimizing a threshold-smoothed surrogate that approximates the hard-decision metric with an error linear in is more effective at reducing that metric than optimizing a raw-soft-count surrogate, especially in regions where the soft-count and hard-decision quantities can disagree by an order-one amount (Proposition 3). SAFER’s frontier matches or improves on that of a baseline that optimizes the Wald statistic directly (Appendix § 13.§ 2). The per-dataset threshold summaries, including the minimum-MSEE points for methods marked ineligible in Table 2, are reported in Appendix § 11.
(a)

(b)

(c) Held-out log-loss versus MSEE Pareto frontiers under Equal Opportunity. The vertical axis is the mean test log-loss over runs across splits and seeds and the horizontal axis is MSEE on hard test predictions; lower is better on both. Markers are the values of the fairness weight along each method’s sweep, annotated and connected in increasing order, with error bars of standard error in both coordinates; the star is ERM (). The methods are SAFER, EO-Penalty, FAIRRET Norm 1 and FAIRRET Norm 2. Adult Income and German Credit use the default variants in Table 1; COMPAS uses the balanced 50/50 variant. The dashed vertical line marks the ERM MSEE, which is the eligibility threshold used in Table 2; outlined markers indicate the selected qualifying points reported in that table.
Figure 1.
| Adult Income | COMPAS | German Credit | Taiwan | ACS | |
| SAFER | 0.3829 | 0.6252 | 0.5349 | 0.4479 | 0.5198 |
| EO-Penalty | 0.4106 | 0.6339 | 0.5358 | 0.4527 | 0.5223 |
| FAIRRET Norm 1 | 0.4272 | 0.6559 | 0.5409 | 0.4668 | 0.5240 |
| FAIRRET Norm 2 | 0.4618 | 0.6366 | 0.5408 | 0.4624 | 0.5314 |
| AdvDebiasing | * | * | * | * | * |
| ExponGrad | * | * | * | 0.6256 | 0.6535 |
| PrejudiceRemover | * | * | * | 0.6216 | * |
Table 2. Thresholded fairness–utility summary across datasets and baselines. Each entry is the best mean test log loss among swept hyperparameters with MSEE at least 50% below ERM MSEE; * means no qualifying hyperparameter was found.
The corresponding SP Pareto plots are in Appendix § 15. Under SP the effective per-subgroup sample sizes are substantially larger than EO’s , so more subgroups are certified already at ERM and the gate contributes less. SAFER nevertheless reaches lower log-loss at matched MSPE than the raw-probability penalty on all three datasets, and the difference is largest on Adult. We attribute this to the threshold-smoothing effect of Theorem 2: with the gate close to open, the hinge approaches a smoothed-disparity penalty, and SAFER’s remaining advantage comes from targeting hard-decision disparities.
Ablation study
Section 4 predicted three mechanisms: threshold smoothing (Theorems 2, 7), the gate threshold (Proposition 4), and the dense tail (Eq. (12)). Each picks out one stress variant in which it is predicted to be empirically necessary. We test all three with single-knob ablations against the SAFER backbone (NoThreshSmooth, NoSelectivity, NoDenseTail); details are in Appendix § 14.
Threshold smoothing. On Adult and COMPAS, NoThreshSmooth is Pareto-dominated across the mid-MSEE range; at matched on Adult, removing threshold smoothing costs roughly 0.14 nats of log-loss. This is the empirical counterpart of Theorems 2 and 7: raw probabilities target a soft-count quantity that is biased relative to the hard-decision metric and whose gradient is not concentrated where decisions can flip.
Dense-tail term. On COMPAS, NoDenseTail plateaus at from while SAFER reaches at (about ten standard errors apart). On Adult, NoDenseTail similarly fails to reach the low- region of SAFER’s frontier: it remains near ERM () for almost the entire sweep and moves toward smaller only at the largest tested , where it is Pareto-dominated by SAFER (Figure 2, left). The mechanism is Eq. (12): as the optimizer drives disparities toward the gate threshold, drops below and the hinge becomes a vanishing fraction of the gradient; the tail’s gradient carries no factor and lets optimization continue. On German Credit, NoDenseTail tracks SAFER, because the significance gate already keeps the hinge inactive.
Figure 2. Single-knob ablations of SAFER on the default Adult Income, COMPAS, and German Credit variants from Table 1. The panels show Adult Income (left), COMPAS (center) and German Credit (right). Axes, markers and error bars are as in Figure 1. The arms are SAFER and its three single-knob removals: NoDenseTail (), NoSelectivity (gate threshold set to zero) and NoThreshSmooth (raw probabilities instead of threshold-smoothed counts).
Gate threshold. On German Credit, almost no EO violation is statistically certified at in our sweeps; SAFER preserves ERM-level log-loss () up to , while NoSelectivity climbs to log-loss by . This is the failure mode that Proposition 4 predicts when the optimizer can act on threshold-smoothed but uncertified signal.
Cross-Regime Summary. Each removal produces the failure that Section 4 predicts. Without threshold smoothing, a Pareto gap opens on Adult and COMPAS, as Theorems 2 and 7 predict. Without the dense tail, the frontier stops short of low on the same two datasets, because of the hinge suppression in Eq. (12). Without the gate threshold, the log-loss rises on German Credit, as Proposition 4 predicts. NoSelectivity has no effect on COMPAS, and on Adult it tracks SAFER up to and loses utility at larger (Appendix § 14.§ 4). The ablation map is in Appendix § 14.§ 5.
Conclusion
Intersectional fairness auditing encounters a resolution limit: as subgroup definitions grow finer, the statistical uncertainty of each disparity estimate comes to exceed the disparity itself. We showed that ignoring this limit, as existing in-processing methods do, leads to fairness overfitting, where the optimizer spends model capacity fitting sampling noise rather than genuine disparities.
To address this, we introduced the significant-excess metric family, which continuously measures only the portion of each subgroup’s disparity that exceeds what sampling variation can explain at a given confidence level, and SAFER, a differentiable regularizer that gates fairness penalties on a soft-count likelihood-ratio evidence statistic. SAFER satisfies a provable safety property: when every subgroup disparity lies well below the gate threshold, the ERM optimum is nearly stationary for SAFER. At that optimum, the gated fairness contribution is exponentially small, and the contribution of the small ungated term is proportional to its weight.
Experiments across several benchmarks and purpose-built stress variants support a regime-conditional claim: SAFER achieves a better Pareto frontier than the baselines where certified disparities exist and preserves the utility of the unconstrained model where they do not, and removing any component of SAFER produces the failure the theory predicts on at least one stress variant.
Limitations and future work. A zero significant excess means that no excess disparity is certifiable at the chosen confidence level and sample size; a real disparity in a small subgroup can remain uncertified, and SAFER then withholds the gradient that would reduce it (Appendix § 12). The current evaluation is limited to tabular benchmarks with at most a few hundred intersectional subgroups. Scaling to high-cardinality attribute spaces and to settings where subgroup membership is uncertain or partially observed remains open. A natural extension is to include multiplicity corrections in the training-time significance test, so that SAFER’s gate also controls the family-wise error rate.
Acknowledgments and Disclosure of Funding
This work is conducted as part of the Horizon Europe project PRE-ACT (Prediction of Radiotherapy side effects using explainable AI for patient communication and treatment modification). It is supported by the European Commission through the Horizon Europe Program (Grant Agreement number 101057746), by the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract number 22 00058, and by the UK government (Innovate UK) under application number 10061955. The authors are employed by Intesa Sanpaolo. The authors declare no other competing interests.
Datasets and stress variants
All experiments use the same preprocessing code as the released scripts. Protected attributes are removed from the feature matrix and kept only for subgroup construction. Subgroup masks include all conjunctions up to the specified intersection level, with groups of size less than two omitted in the corresponding split.
Adult Income. We use the OpenML version of the UCI Adult Income dataset [26]. The label is income > $50K. The protected attributes are race, sex, age binned into , and native country binarized as US/non-US. Non-protected numeric and categorical features are retained, with categorical features one-hot encoded. The stress variant in the main paper uses intersection level 4 and a 10/90 train/test split. After dropping rows with missing values, the processed data contain 45,222 examples, 55 non-protected features, and a positive-label rate of 24.8%. Over 10 stratified splits used for a dataset audit, this dataset variant has 4,522 training examples, training-set subgroup masks and test-set subgroup masks. This is a budget of only training examples per audited subgroup, and only positive-label examples per subgroup for EO. For the ERM model fit, the test-set EO audit has EO-defined subgroups and certifies significant EO violations at after BH correction.
COMPAS. We use the OpenML COMPAS recidivism dataset [2]. The label is two-year recidivism. The protected attributes are race (African-American/Caucasian in the OpenML version), sex, and age category (, , ). The processed data contain 4,966 examples, 5 non-protected features, and a positive-label rate of 50.0%. The headline Pareto comparison uses intersection level 3 and a 50/50 train/test split. Over ten stratified splits this gives 2,483 training examples and exactly 35 audited subgroup masks in both train and test, a budget of 70.9 training examples and 35.5 positive-label examples per audited subgroup. We run an ERM test-set EO audit with 10 splits that certifies significant EO violations at after BH correction. The mechanism ablation uses the more severe 20/80 split: 993 training examples, the same 35 subgroup masks, and only 28.4 training examples and 14.2 positives per audited subgroup. Under this stress split, ERM certifies significant EO violations, making it the high-regularization stress test for the dense-tail component.
German Credit. We use the OpenML version of the Statlog German Credit dataset [21]. The label is good credit. The protected attributes are sex derived from personal_status, age binned into , and foreign-worker status. The processed data contain 1,000 examples, 53 non-protected features, and a positive-label rate of 70.0%. The stress variant uses intersection level 3 and a 50/50 train/test split. Over ten stratified splits this gives 500 training examples, 32.1 1.4 training-set subgroup masks, and 31.1 0.3 test-set subgroup masks, a budget of 15.6 0.7 training examples and 10.9 0.5 positive-label examples per audited subgroup. For the ERM model fit on these same splits, the test-set EO audit has 31.1 0.3 EO-defined subgroups and certifies only 0.3 0.9 significant EO violations at after BH correction, with 0.0007 0.0018. This is therefore a safety-regime test rather than a fairness-improvement test.
ACS Coverage. We use the ACS Public Coverage task from the Folktables benchmark [10], which predicts public health insurance coverage from American Community Survey records. The protected attributes are race, sex, and age. The reported variant uses intersection level 3 and a 50/50 train/test split. The data contains 50,000 examples, so each split has 25,000 training examples. Across the dataset audit there are exactly 71 training-set and test-set subgroup masks, all EO-defined. This corresponds to a budget of 352.1 training examples and 103.5 positive-label examples per audited subgroup ( 29.4% positives). Because the subgroup budget is much larger than in Adult, COMPAS, or German, this variant serves as a higher-sample ACS coverage benchmark rather than a sparse-cell stress test.
Taiwan Credit. We use the UCI Default of Credit Card Clients dataset [39], which predicts credit card default. The protected attributes are sex, education, marriage, and age. The headline variant uses intersection level 4 and a 50/50 train/test split. The processed data contain 30,000 examples, so each split has 15,000 training examples. Over ten stratified splits this gives 285.4 2.8 training-set subgroup masks and 285.9 2.3 test-set subgroup masks. This corresponds to a budget of 52.6 0.5 training examples and 11.6 0.1 positive-label examples per audited subgroup; 245.8 subgroups are EO-defined on average. This variant therefore provides a high-intersectionality credit-risk benchmark with a larger total training set than Adult but still a relatively small positive-label budget per audited subgroup.
Training details and hyperparameters
Model and optimizer. The headline neural methods use the same MLP classifier: hidden layers (64, 32), ReLU activations, dropout 0.2, Adam optimizer (lr = , weight decay ), batch size 256, gradient clipping at norm 1, and at most 200 epochs. Prediction loss is binary cross-entropy. Fairness statistics are computed from full-training-set soft counts each epoch rather than mini-batches, because subgroup-level LRT and significant-excess quantities are unstable on mini-batch counts.
Splits. The main Adult L4 10/90 and COMPAS L3 50/50 Pareto comparisons use 100 stratified shuffle splits and 3 random seeds per split. The mechanism ablation uses 10 splits 3 seeds on Adult L4 10/90, 100 splits 3 seeds on COMPAS L3 20/80, and 30 splits 3 seeds on German L3 50/50. Within the training portion of every split, 10% is held out for model selection. The held-out test partition is never used for hyperparameter or checkpoint selection.
Checkpointing. SAFER and its single-knob ablations use the joint BCE-guarded training configuration. The selector first restricts attention to checkpoints whose validation BCE is within a relative tolerance of the best validation BCE, and then selects the checkpoint with the smallest guarded score, using the full-training-set fairness loss for the fairness term. This prevents high runs from selecting collapsed models while still allowing fairness improvements when their validation-BCE cost is small. ERM selects by validation BCE alone. The headline EO-Penalty and FAIRRET runs use their standard joint-training selectors unless otherwise stated.
SAFER hyperparameters. The main SAFER configuration is the unweighted subgroup average (use_size_weight=false) with , threshold smoothing temperature , softplus temperature , absolute-value smoothing , detached LRT-based , and LRT dense-tail weight . The NoThresholdSmooth ablation sets to null; NoSelectivity sets the hinge threshold to zero; NoDenseTail sets .
Baselines. ERM is the unconstrained MLP. EO-Penalty is BCE plus an unweighted squared EO-disparity penalty. FAIRRET Norm 1 and FAIRRET Norm 2 use the ratio-normalized differentiable fairness regularizers of FAIRRET [6], with the EO variant conditioning on positive labels. We also swept ExponGrad [1] with TPR-parity constraints, Prejudice Remover, and adversarial debiasing. where compatible. These additional baselines are included in the thresholded summary, but they do not all share the same neural in-training backbone as the headline Pareto comparisons.
| Experiment | Method | Grid |
| Adult L4 10/90 main | SAFER | 0.05, 0.1, 0.2, 0.3, 0.5, 1, 2, 5, 10, 20, 30, 50, 80, 100 |
| Adult L4 10/90 main | EO-Penalty | 10, 20, 50, 100, 200, 500, 1000, 2000, 4000 |
| Adult L4 10/90 main | FAIRRET Norm 2 | 0.01, 0.1, 0.5, 1, 2, 5, 10, 20, 50, 100, 500, 1000 |
| Adult L4 10/90 ablation | SAFER / NoThresh / NoTail | 5, 10, 20, 50, 100, 200, 500, 1000 |
| Adult L4 10/90 ablation | NoSelectivity | 0.5, 1, 2, 5, 10, 20, 50, 100 |
| COMPAS L3 50/50 main | SAFER | 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10, 100 |
| COMPAS L3 50/50 main | EO-Penalty | 0.1, 0.5, 1, 2, 5, 10, 20, 30, 50, 80, 100, 200, 500, 1000 |
| COMPAS L3 50/50 main | FAIRRET Norm 1 | 0.001, 0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 1, 10 |
| COMPAS L3 50/50 main | FAIRRET Norm 2 | 0.005, 0.01, 0.02, 0.03, 0.06, 0.1, 0.15, 0.2, 0.3, 0.5, 1, 10, 100 |
| COMPAS L3 20/80 ablation | SAFER / NoThresh / NoTail | 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10, 20, 100 |
| COMPAS L3 20/80 ablation | NoSelectivity | 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 1, 2, 5, 10, 100 |
| German L3 50/50 ablation | SAFER / NoThresh / NoTail | 1, 2, 5, 10, 20, 50, 100 |
| German L3 50/50 ablation | NoSelectivity | 0.01, 1, 2, 3, 4, 5, 10, 20, 50, 100 |
Table A.1. Regularization grids used in the Results section. Main Pareto comparisons use 100 splits × 3 seeds; Adult ablations use 10 splits × 3 seeds.
| Experiment | Method | jobs | Split × seed | runs per job | Estimated wall time |
| Adult L4 10/90 main | SAFER | 14 | 100 × 3 | 3–6 h | |
| Adult L4 10/90 main | EO-Penalty | 9 | 100 × 3 | 1–1.5 h | |
| Adult L4 10/90 main | FAIRRET Norm 1 / Norm 2 | 12–13 | 100 × 3 | 20–45 min | |
| COMPAS L3 50/50 main | SAFER | 9 | 100 × 3 | 20–40 min | |
| COMPAS L3 50/50 main | EO-Penalty | 14 | 100 × 3 | 15–30 min | |
| COMPAS L3 50/50 main | FAIRRET Norm 1 / Norm 2 | 9–13 | 100 × 3 | 5–10 min | |
| German L3 50/50 main | SAFER | 7 | 100 × 3 | 20–30 min | |
| German L3 50/50 main | EO-Penalty | 14 | 100 × 3 | 10–20 min | |
| German L3 50/50 main | FAIRRET Norm 1 / Norm 2 | 9–13 | 100 × 3 | under 10 min | |
| Adult L4 10/90 ablation | SAFER-family arm | 8 | 30 × 3 | 35–55 min per arm | |
| COMPAS L3 20/80 ablation | SAFER-family arm | 10–11 | 100 × 3 | 20–40 min per arm | |
| German L3 50/50 ablation | SAFER-family arm | 7–10 | 30 × 3 | 5–15 min per arm |
Table A.2. Estimated CPU wall-clock time for the full sweeps on the 10-core, 32 GB Apple M1 Max machine with eight joblib workers. SAFER-ablation variants (NOTHRESHSMOOTH, NOSELECTIVITY, and NODENSETAIL) have the same resource profile as SAFER up to small constant-factor changes.
Uncertainty reporting. Figures report means and ±1 standard error over all combinations of split and seed. Because three seeds share the same data split, these observations are not fully independent. We therefore treat the plotted SE as descriptive and verified that the method ordering is unchanged when seeds are averaged within split before computing SE.
Computational resources
All reported neural sweeps were run on CPU workers. The machine used for the final local runs was an Apple MacBook Pro with an Apple M1 Max processor (10 CPU cores: 8 performance and 2 efficiency cores) and 32 GB of unified memory, running macOS/Darwin on arm64. GPU execution was disabled in the experiment scripts. Independent jobs were parallelized across up to 8 CPU worker processes. Each worker trains one combination of method and over its assigned sequence of splits and seeds. Peak memory was dominated by the Python/PyTorch worker processes and the dataset matrices; on this 32 GB machine the reported CPU sweeps ran without memory pressure when running all eight processes.
Table A.2 gives wall-clock estimates for reproducing the full neural sweeps. These estimates exclude plotting and table aggregation, which take seconds to minutes, and should be interpreted as order-of-magnitude wall-clock requirements for this CPU workstation rather than hardware-independent constants.
Per-dataset thresholded baseline summaries
The following tables expand Table 2 with per-dataset details. For eligible rows, we report the best mean test log loss among swept hyperparameters whose MSEE is at least 50% below the ERM MSEE on that dataset. For ineligible rows, corresponding to * entries in the main summary table, we report the minimum-MSEE point found by the sweep.
| Method | Parameter | Eligible | Log loss | AUC | MSEE |
| SAFER | yes | 0.3829 | 0.8818 | 0.0097 | |
| EO-Penalty | yes | 0.4106 | 0.8624 | 0.0093 | |
| FAIRRET Norm 1 | yes | 0.4272 | 0.8561 | 0.0097 | |
| FAIRRET Norm 2 | yes | 0.4618 | 0.8456 | 0.0085 | |
| AdvDebiasing | no | 0.4770 | 0.8208 | 0.0157 | |
| ExponGrad | no | 0.5936 | 0.7373 | 0.0335 | |
| PrejudiceRemover | no | 1.1022 | 0.4367 | 0.0165 |
Table A.3. Adult Income: Best mean test log loss with MSEE at least 50% below ERM MSEE. (ERM MSEE=0.0201, threshold=0.0101; ineligible rows show the minimum-MSEE point).
| Method | Parameter | Eligible | Log loss | AUC | MSEE |
| SAFER | yes | 0.6252 | 0.7148 | 0.0510 | |
| EO-Penalty | yes | 0.6339 | 0.7171 | 0.0661 | |
| FAIRRET Norm 1 | yes | 0.6559 | 0.6975 | 0.0614 | |
| FAIRRET Norm 2 | yes | 0.6366 | 0.7117 | 0.0628 | |
| AdvDebiasing | no | 0.7381 | 0.6522 | 0.1029 | |
| ExponGrad | no | 0.7211 | 0.5291 | 0.1259 | |
| PrejudiceRemover | no | 0.6595 | 0.7066 | 0.0939 |
Table A.4. COMPAS: Best mean test log loss with MSEE at least 50% below ERM (ERM MSEE=0.1292, threshold=0.0646; ineligible rows show the minimum-MSEE point).
| Method | Parameter | Eligible | Log loss | AUC | MSEE |
| SAFER | yes | 0.5349 | 0.7659 | 0.0006 | |
| EO-Penalty | yes | 0.5358 | 0.7554 | 0.0006 | |
| FAIRRET Norm 1 | yes | 0.5409 | 0.7536 | 0.0006 | |
| FAIRRET Norm 2 | yes | 0.5408 | 0.7544 | 0.0006 | |
| AdvDebiasing | no | 1.3272 | 0.7247 | 0.0027 | |
| ExponGrad | no | 0.5664 | 0.7371 | 0.0018 | |
| PrejudiceRemover | no | 0.8915 | 0.6380 | 0.0104 |
Table A.5. German Credit: Best mean test log loss with MSEE at least 50% below ERM (ERM MSEE=0.0013, threshold=0.0007; ineligible rows show the minimum-MSEE point).
| Method | Parameter | Eligible | Log loss | AUC | MSEE |
| SAFER | yes | 0.4479 | 0.7552 | 0.0021 | |
| EO-Penalty | yes | 0.4527 | 0.7531 | 0.0025 | |
| FAIRRET Norm 1 | yes | 0.4668 | 0.7442 | 0.0014 | |
| FAIRRET Norm 2 | yes | 0.4624 | 0.7455 | 0.0021 | |
| AdvDebiasing | no | 0.4569 | 0.7546 | 0.0032 | |
| ExponGrad | yes | 0.6256 | 0.6096 | 0.0027 | |
| PrejudiceRemover | yes | 0.6216 | 0.6093 | 0.0017 |
Table A.6. Taiwan: Best mean test log loss with MSEE at least 50% below ERM (ERM MSEE=0.0060, threshold=0.0030; ineligible rows show the minimum-MSEE point).
Subgroups below the certification boundary
A zero significant excess for a subgroup states only that the available sample cannot certify an excess at the chosen confidence level. The level sets the trade between reacting to sampling noise and leaving low-power disparities uncorrected; a larger protects smaller groups at a utility cost. The size sweep of Appendix § 13.§ 3 makes the dependence concrete: the same planted gap of is certified in about 9% of runs at and in about 95% at . We therefore recommend reporting MSEE together with raw subgroup disparities, confidence intervals, subgroup counts and a power or minimum-detectable-effect analysis, and treating an uncertified subgroup as unresolved; possible responses are collecting more data, pooling groups coarsely or hierarchically, or policy constraints that do not rely on statistical significance alone. Where harm is suspected in a subgroup too small to certify, the appropriate evidence is of a different kind: the structure of the decision procedure, the features it uses, the causal path from the protected attribute to the outcome, and documented practice. A complete audit combines statistical outcome testing with this process evidence, and the certification boundary marks where the hand-over occurs. Medical-imaging benchmarks whose small subgroups carry genuine disparities, such as Harvard-FairVision30k [31], are a natural next test of SAFER outside the tabular setting.
| Method | Parameter | Eligible | Log loss | AUC | MSEE |
| SAFER | yes | 0.5198 ± 0.0003 | 0.7472 ± 0.0006 | 0.0236 ± 0.0009 | |
| EO-Penalty | yes | 0.5223 ± 0.0004 | 0.7427 ± 0.0007 | 0.0383 ± 0.0009 | |
| fairret-norm1 | yes | 0.5240 ± 0.0004 | 0.7401 ± 0.0006 | 0.0357 ± 0.0011 | |
| fairret-norm2 | yes | 0.5314 ± 0.0004 | 0.7320 ± 0.0007 | 0.0218 ± 0.0009 | |
| AdvDebiasing | no | 0.5299 ± 0.0111 | 0.7450 ± 0.0030 | 0.0605 ± 0.0025 | |
| ExponGrad | yes | 0.6535 ± 0.0134 | 0.5623 ± 0.0078 | 0.0257 ± 0.0029 | |
| PrejudiceRemover | no | 0.6196 ± 0.0019 | 0.6401 ± 0.0018 | 0.0464 ± 0.0013 |
Table A.7. ACS: Best mean test log loss with MSEE at least 50% below ERM (ERM MSEE=0.0904, threshold=0.0452; ineligible rows show the minimum-MSEE point).
Additional experiments
Decision-threshold sensitivity
The threshold does not enter the prediction loss, which trains probabilistic scores; it defines the deployed classifier and therefore both its accuracy and its hard-decision fairness audit. We use because, when scores estimate class probabilities and false positives and false negatives cost the same, it is the Bayes-optimal decision threshold. Since enters SAFER’s objective, it is fixed before training: by a deployment requirement such as a cost matrix or a regulatory decision rule, or, when no policy value exists, on validation data with a preliminary ERM model, and then held fixed while all methods are trained and compared. To test the dependence on this default we retrained SAFER for every on the three datasets of the main Pareto comparison, five seeds per threshold, and audited each model at its own threshold. Table A.8 reports the test log-loss, the paired difference against the ERM audit at the same threshold (negative values are a fairness improvement), and the accuracy. Log-loss is computed from probabilistic scores, so for a fixed model it does not depend on ; its variation across columns measures how changes the retrained model through the fairness term. Log-loss is stable across on Adult and German and more sensitive on COMPAS. The fairness benefit persists across operationally meaningful thresholds: Adult has non-positive throughout, COMPAS improves for between 0.2 and 0.9, and the changes on German are negligible because little certified disparity is available to reduce. Extreme thresholds produce nearly constant classifiers with trivially low MSEE (accuracy 0.500 on COMPAS for , 0.700 on German for ), so those columns are not useful operating points. This is a sensitivity analysis; is never tuned on the test set.
A baseline that optimizes the Wald statistic
Wald-Stat penalizes the standardized Wald statistic directly:
where is the disparity computed from the same threshold-smoothed counts as SAFER and its two-proportion plug-in standard error (§ 4.2). Both stay in the computation graph, so gradients flow through the statistic, including the route that lowers the penalty by changing the estimated standard error. Wald-Stat uses the same critical value, threshold smoothing, temperatures, dense-tail weight, architecture, optimizer and checkpointing protocol as SAFER; it tests whether optimizing Wald significance directly yields a stronger fairness–utility frontier than SAFER’s LRT-based construction. Because the two objectives live on different scales, we sweep independently for each and match operating points by mean test log-loss after examining both sweeps; Table A.9 summarizes the matched pairs. Entries are SAFER/Wald-Stat means ± standard deviations over independent runs over splits and seeds, 10 for Adult and COMPAS and 5 for German, rounded to three decimals; several differences are small relative to this variability, so we read the frontier rather than individual rows. For the following table, the ERM reference values in the caption come from these same runs, a lighter protocol than the 100 splits with three seeds of the main sweeps, so they differ from the ERM values of the main tables.
| 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | ||
| Adult | log-loss | .336 ± .002 | .337 ± .002 | .337 ± .003 | .336 ± .002 | .336 ± .002 | .336 ± .002 | .337 ± .002 | .338 ± .003 | .337 ± .003 |
| MSEE | −.008 ± .009 | −.013 ± .005 | −.001 ± .009 | −.007 ± .006 | −.009 ± .006 | −.008 ± .003 | −.014 ± .004 | −.010 ± .005 | −.008 ± .005 | |
| accuracy | .696 ± .004 | .778 ± .003 | .822 ± .003 | .839 ± .002 | .843 ± .002 | .837 ± .002 | .817 ± .002 | .785 ± .004 | .762 ± .002 | |
| COMPAS | log-loss | .615 ± .008 | .615 ± .007 | .623 ± .006 | .633 ± .005 | .640 ± .008 | .643 ± .007 | .628 ± .005 | .620 ± .005 | .615 ± .005 |
| MSEE | .000 ± .000 | −.005 ± .004 | −.089 ± .019 | −.131 ± .021 | −.084 ± .018 | −.069 ± .018 | −.078 ± .012 | −.051 ± .015 | −.011 ± .005 | |
| accuracy | .500 ± .000 | .500 ± .000 | .500 ± .000 | .535 ± .040 | .631 ± .016 | .531 ± .024 | .510 ± .006 | .505 ± .003 | .502 ± .001 | |
| German | log-loss | .531 ± .035 | .531 ± .035 | .532 ± .035 | .532 ± .035 | .531 ± .035 | .531 ± .035 | .531 ± .035 | .532 ± .035 | .531 ± .035 |
| MSEE | .0000 ± .0000 | .0000 ± .0000 | .0000 ± .0000 | −.0005 ± .0013 | +.0002 ± .0014 | −.0005 ± .0007 | −.0014 ± .0018 | −.0041 ± .0078 | −.0006 ± .0027 | |
| accuracy | .700 ± .000 | .700 ± .000 | .702 ± .004 | .722 ± .018 | .731 ± .019 | .737 ± .014 | .673 ± .109 | .590 ± .162 | .478 ± .103 |
Table A.8. SAFER retrained and audited at decision thresholds from 0.1 to 0.9: test log-loss, paired MSEE against ERM at the same threshold, and accuracy (mean ± standard deviation over five seeds).
| log-loss S/W | MSEE S/W | ||
| Adult | (1, 0.0001) | .337 ± .003/.338 ± .004 | .028 ± .013/.040 ± .013 |
| (5, 0.015) | .344 ± .004/.344 ± .005 | .028 ± .015/.035 ± .013 | |
| (15, 0.5) | .372 ± .005/.372 ± .005 | .025 ± .011/.025 ± .014 | |
| (30, 2) | .387 ± .003/.391 ± .003 | .005 ± .010/.002 ± .007 | |
| (50, 3) | .397 ± .003/.397 ± .004 | .0001 ± .0002/.0003 ± .0009 | |
| COMPAS | (0.1, 0.001) | .615 ± .009/.614 ± .009 | .086 ± .026/.087 ± .025 |
| (0.5, 0.005) | .625 ± .010/.621 ± .008 | .058 ± .013/.067 ± .017 | |
| (1, 0.015) | .647 ± .012/.648 ± .007 | .041 ± .015/.052 ± .019 | |
| (3, 0.05) | .671 ± .008/.669 ± .006 | .036 ± .026/.046 ± .030 | |
| (20, 1) | .690 ± .003/.690 ± .003 | .040 ± .048/.038 ± .050 | |
| German | (0.25, 0.0005) | .531 ± .035/.531 ± .035 | .006 ± .005/.006 ± .005 |
| (1, 0.01) | .532 ± .035/.531 ± .035 | .007 ± .005/.008 ± .006 | |
| (15, 0.02) | .532 ± .032/.532 ± .035 | .006 ± .006/.007 ± .006 | |
| (20, 0.1) | .533 ± .032/.533 ± .034 | .004 ± .004/.005 ± .005 | |
| (40, 3) | .540 ± .033/.541 ± .030 | .0005 ± .0007/.0015 ± .0021 |
Table A.9. SAFER versus Wald-Stat at matched test log-loss. Each row pairs a SAFER and a Wald-Stat fairness weight with similar mean test log-loss; entries are SAFER/Wald-Stat, mean ± standard deviation over runs. ERM log-loss/MSEE: Adult 0.334/0.041, COMPAS 0.613 ± 0.009/0.127 ± 0.027, German 0.531 ± 0.035/0.006 ± 0.005.
Diagnostics on Adult, stratified by the positive-label count , explain the gap. Across the five matched points, the median of is 1.00 for . For , saturation drives the plug-in standard error toward zero and the ratio’s 90th percentile is 37 to 43, a roughly fortyfold inflation in the most extreme cases. Among about 1,491 pairs of subgroup and seed per point, Wald-Stat flags 41 to 57 disparities that the LRT does not, and never the reverse. These saturated groups are 2.7% of the total but absorb 99% of Wald-Stat’s penalty and 86% of its final-epoch gradient over the smallest subgroups (), against 43% for SAFER. At the near-ERM Adult point, the residual certified excess of the groups with is 0.0029 for SAFER and 0.0214 for Wald-Stat, so SAFER preserves capacity for the better-supported groups. SAFER also has lower mean MSEE near ERM on Adult and at the intermediate COMPAS and German points, with gaps that are modest relative to the variability across splits and seeds; under strong regularization both methods converge near zero MSEE at a greater utility cost.
A controlled benchmark for the selective-gating regime
The operating regimes of § 4.3 suggest a hypothesis for the gate threshold: when both active and inactive subgroups are present, the gate gives SAFER its largest advantage over NoSelectivity, because it withholds penalties from unsupported disparities while keeping those with support. The benchmark below tests it by separating statistical evidence from disparity magnitude. It has one large reference cell and 34 probe cells. The generator is calibrated so that its population decision rule has a true-positive rate of 0.85 on the reference cell and 0.70 on every unfair probe, a planted reference gap of ; null probes have zero gap. Only the positive-label count of the unfair probes varies, and the reference count is fixed and large. Group identity is hidden from the predictor and used only by the fairness loss and the audit, which keep the -versus- comparison of the main text. We summarize each evidence stratum by the certification statistic computed on the hard predictions of the final ERM model on the training split, the same statistic as in the size sweep below (Table A.10). The null and strong-evidence stratum means lie on opposite sides of the gate in every one of the 10 runs, so the benchmark is in the selective-gating regime; cell by cell the picture is noisier, with at least one null cell certified in 6 runs and one strong-evidence cell uncertified in 1 run.
| probe stratum | cells | planted gap | mean | certified fraction | |
| null | 10 | 0 | 105 | ||
| weak evidence | 12 | 6 | |||
| boundary | 8 | 34 | |||
| strong evidence | 4 | 204 |
Table A.10. Evidence strata of the probe cells for the final ERM model: cells per stratum, planted gap, mean positive-label count in the training split, mean certification statistic on hard training predictions, and fraction of cells certified (); per-run stratum means, reported as mean standard deviation over 10 runs.
Generator and protocol. Every cell draws labels . One signal feature is distributed as for positives and for negatives, where is the planted true-positive rate of the cell and is the standard normal distribution function; eight pure-noise features are appended. The reference cell has 45,000 examples and planted rate 0.85, which pins the decision threshold of a group-blind classifier so that each cell recovers its planted rate and the complement of every probe has a true-positive rate close to 0.85. A probe cell with design size has examples, so about positives fall in the training split; the design sizes are 10, 10, 20, 20, 40, 40, 120, 120, 400, 400 for the ten null probes (planted rate 0.85), 5, 5, 6, 6, 7, 7, 8, 8, 5, 6, 7, 8 for the twelve weak-evidence probes, 20, 30, 35, 35, 40, 40, 45, 45 for the eight boundary probes and 150, 150, 300, 300 for the four strong-evidence probes (planted rate 0.70). Each of the 10 runs draws a fresh dataset of 54,832 examples, a fresh split and a fresh initialization; with a run all arms share the initialization and the dropout stream, so comparisons are paired. Half of the data is held out as test; of the rest, 90% trains the model and 10% is set aside for validation and not used. The model is an MLP with hidden layers (64, 32) and dropout 0.2, trained full-batch for 120 epochs with Adam (learning rate , weight decay ); the final model is evaluated, without early stopping or checkpoint selection, so that traces the whole fairness–utility frontier. SAFER uses the values of the main experiments (, , , , ); NoSelectivity removes only the gate and EO-Penalty adds times the squared EO gap to the prediction loss. The grid is chosen separately for each arm, from near-ERM to a near-constant predictor. Every cell is scored on its training and its test members with hard predictions at 0.5; a cell is skipped on a split where it or its complement has fewer than two positives, which affects only the weak-evidence stratum (10 to 12 of its 12 cells scored per run). Stratum values are cell averages, reported as mean standard deviation over the 10 runs. The generator, the harness and the per-run results are released with the code.
We compare SAFER with NoSelectivity (NS), which removes only the gate, and with EO-Penalty (EO), over independent sweeps. Table A.12 reports the complete sweeps: for every arm and , the test log-loss and the mean residual certified excess in each evidence stratum. Points whose mean test log-loss reaches 0.5 approach the log-loss 0.693 of a constant predictor, and their near-zero excess comes from degenerate predictions. From the remaining part of the frontiers we chose three approximately matched mean test-log-loss levels, reported in Table A.11 with the fairness weight and test log-loss of each arm and the mean residual certified excess in each evidence stratum, in percentage points, as means standard deviations over 10 runs. The methods overlap at the near-ERM level A. At B and C, SAFER has the lowest mean excess in the boundary and strong-evidence strata; in original units, SAFER’s C values in these two strata are 0.0079 and 0.0227, against 0.0184 and 0.0465 for NS and 0.0108 and 0.0429 for EO. SAFER’s null-stratum mean at C is higher and the standard deviations overlap, so the evidence is directional, with no statistically significant pairwise differences at matched utility. The full sweeps show the same pattern: EO’s strong-stratum excess stays near 4 percentage points from to while its log-loss rises from 0.47 to 0.63, and NS becomes unstable above , with large run-to-run variation of the log-loss from on. The strong-evidence gain is an indirect effect of the whole objective: the gate is already active for those probes, and withholding unsupported gradients preserves optimization capacity for well-supported ones. Table A.13 reports the raw gap on training and on test members for the non-degenerate points. On the weak-evidence stratum, relative to ERM (training gap 0.185, test gap 0.200), EO at lowers the training gap by 0.090 but the test gap by only 0.037, whereas SAFER at , at similar log-loss, lowers them by 0.098 and 0.071: the certainty-blind penalty fits training-set noise in the sparse cells. An independent size sweep (Table A.14) locates the transition for this planted gap: the certified fraction rises from at design size to at , where the mean statistic equals 1.96, and to at ; the value is specific to this gap and reference size. On the real datasets, Adult and COMPAS were built as candidates for this regime; NoSelectivity tracks SAFER across the sweep on COMPAS and up to on Adult, where it then loses utility (Appendix § 14.§ 4).
| set | arm | test log-loss | null | weak | boundary | strong | |
| – | ERM | 0 | .356 .004 | .89 1.03 | 3.49 2.69 | 2.01 .74 | 7.42 1.30 |
| A | SAFER | 2 | .362 .003 | 1.09 .77 | 3.35 2.13 | 1.51 .56 | 6.35 1.16 |
| A | NS | .5 | .361 .004 | 1.09 .76 | 3.63 1.90 | 1.54 .63 | 6.58 1.01 |
| A | EO | 5 | .367 .004 | 1.07 .72 | 3.14 2.16 | 1.54 .56 | 6.47 1.07 |
| B | SAFER | 5 | .381 .006 | 1.14 1.06 | 2.88 1.88 | 1.25 .73 | 4.90 1.28 |
| B | NS | 1 | .379 .008 | 1.23 .92 | 3.30 2.48 | 1.49 .81 | 5.63 1.05 |
| B | EO | 10 | .381 .006 | 1.17 .86 | 3.25 2.24 | 1.44 .54 | 5.98 1.08 |
| C | SAFER | 15 | .457 .031 | 1.41 .50 | 2.51 1.62 | .79 .76 | 2.27 1.29 |
| C | NS | 1.25 | .467 .066 | 1.06 .84 | 3.32 2.24 | 1.84 1.26 | 4.65 1.29 |
| C | EO | 50 | .466 .011 | 1.04 .91 | 2.61 1.73 | 1.08 .83 | 4.29 .96 |
Table A.11. Matched-utility comparison on the controlled benchmark: fairness weight , test log-loss, and mean residual certified excess per evidence stratum in percentage points (mean standard deviation over 10 runs; lower is better).
Ablation study details
The arm definitions and detailed numbers for the mechanism ablations of § 5.2 are given below.
Arm definitions
The four arms are evaluated on the three stress variants of Table 1 under the protocols of § 5:
SAFER: full method, all three components active.
NoThreshSmooth: disabled (raw probabilities feed the soft counts).
NoSelectivity: the threshold term is removed from the softplus argument, leaving alone.
NoDenseTail: .
Apart from the toggled knob, every other element of the loss, optimizer, and protocol is identical across arms.
| arm | test log-loss | null | weak | boundary | strong | |
| ERM | 0 | |||||
| SAFER | 0.5 | |||||
| SAFER | 1 | |||||
| SAFER | 1.5 | |||||
| SAFER | 2 | |||||
| SAFER | 2.5 | |||||
| SAFER | 5 | |||||
| SAFER | 10 | |||||
| SAFER | 15 | |||||
| SAFER | 20 | |||||
| SAFER | 30 | † | ||||
| SAFER | 40 | † | ||||
| SAFER | 50 | † | ||||
| SAFER | 70 | † | ||||
| SAFER | 100 | † | ||||
| SAFER | 150 | † | ||||
| SAFER | 200 | † | ||||
| SAFER | 300 | † | ||||
| SAFER | 500 | † | ||||
| NS | 0.25 | |||||
| NS | 0.5 | |||||
| NS | 0.75 | |||||
| NS | 1 | |||||
| NS | 1.25 | |||||
| NS | 1.5 | † | ||||
| NS | 1.75 | † | ||||
| NS | 2 | † | ||||
| NS | 2.25 | † | ||||
| NS | 2.5 | † | ||||
| NS | 2.75 | † | ||||
| NS | 3 | † | ||||
| NS | 3.5 | † | ||||
| NS | 3.75 | † | ||||
| NS | 4 | † | ||||
| NS | 4.5 | † | ||||
| NS | 5 | † | ||||
| NS | 5.4 | † | ||||
| EO | 5 | |||||
| EO | 10 | |||||
| EO | 20 | |||||
| EO | 50 | |||||
| EO | 60 | |||||
| EO | 200 | † | ||||
| EO | 600 | † | ||||
| EO | 1000 | † | ||||
| EO | 1500 | † | ||||
| EO | 2000 | † | ||||
| EO | 2500 | † | ||||
| EO | 5000 | † | ||||
| EO | 7500 | † | ||||
| EO | 10000 | † |
Table A.12. Complete sweeps on the controlled benchmark: test log-loss and mean residual certified excess per evidence stratum in percentage points (mean standard deviation over 10 runs; lower is better). NS is NoSelectivity and EO is EO-Penalty. Rows marked † have mean test log-loss at or above 0.5: these models approach the constant predictor (log-loss 0.693), and their small excess values come from degenerate predictions rather than from fairer ones. The matched levels of Table A.11 are drawn from the unmarked rows.
| arm | test log-loss | \multicolumn{4}{c}{test } | \multicolumn{4}{c}{train } | |||||||
| null | weak | boundary | strong | null | weak | boundary | strong | |||
| ERM | 0 | .356 ± .004 | .048 ± .015 | .200 ± .027 | .126 ± .027 | .137 ± .013 | .054 ± .009 | .185 ± .031 | .124 ± .022 | .134 ± .017 |
| SAFER | 0.5 | .357 ± .003 | .050 ± .015 | .201 ± .028 | .124 ± .030 | .131 ± .012 | .050 ± .011 | .175 ± .026 | .111 ± .021 | .125 ± .015 |
| SAFER | 1 | .358 ± .003 | .050 ± .014 | .187 ± .027 | .122 ± .029 | .129 ± .014 | .046 ± .011 | .170 ± .027 | .100 ± .021 | .116 ± .015 |
| SAFER | 1.5 | .360 ± .003 | .049 ± .014 | .182 ± .025 | .118 ± .028 | .128 ± .013 | .045 ± .009 | .169 ± .027 | .095 ± .019 | .110 ± .014 |
| SAFER | 2 | .362 ± .003 | .047 ± .013 | .185 ± .024 | .117 ± .027 | .124 ± .012 | .043 ± .009 | .162 ± .026 | .092 ± .016 | .106 ± .012 |
| SAFER | 2.5 | .365 ± .004 | .045 ± .011 | .182 ± .025 | .115 ± .026 | .121 ± .013 | .040 ± .008 | .155 ± .020 | .087 ± .013 | .103 ± .012 |
| SAFER | 5 | .381 ± .006 | .041 ± .009 | .170 ± .023 | .105 ± .033 | .105 ± .015 | .038 ± .009 | .139 ± .018 | .071 ± .007 | .085 ± .014 |
| SAFER | 10 | .423 ± .018 | .033 ± .007 | .143 ± .027 | .084 ± .026 | .077 ± .018 | .031 ± .007 | .099 ± .016 | .054 ± .010 | .053 ± .008 |
| SAFER | 15 | .457 ± .031 | .035 ± .013 | .129 ± .031 | .080 ± .034 | .068 ± .022 | .027 ± .006 | .087 ± .025 | .044 ± .015 | .036 ± .007 |
| SAFER | 20 | .487 ± .037 | .041 ± .023 | .129 ± .047 | .076 ± .038 | .065 ± .027 | .031 ± .014 | .095 ± .035 | .041 ± .016 | .034 ± .011 |
| NS | 0.25 | .357 ± .004 | .050 ± .015 | .194 ± .033 | .123 ± .026 | .129 ± .011 | .046 ± .014 | .147 ± .027 | .111 ± .025 | .125 ± .014 |
| NS | 0.5 | .361 ± .004 | .048 ± .014 | .191 ± .031 | .119 ± .028 | .127 ± .010 | .040 ± .010 | .128 ± .021 | .102 ± .024 | .117 ± .014 |
| NS | 0.75 | .366 ± .005 | .044 ± .010 | .183 ± .028 | .115 ± .028 | .121 ± .013 | .038 ± .009 | .121 ± .019 | .094 ± .024 | .113 ± .015 |
| NS | 1 | .379 ± .008 | .044 ± .011 | .171 ± .027 | .113 ± .030 | .114 ± .011 | .035 ± .011 | .106 ± .014 | .080 ± .015 | .105 ± .018 |
| NS | 1.25 | .467 ± .066 | .053 ± .018 | .172 ± .039 | .120 ± .041 | .106 ± .017 | .037 ± .013 | .094 ± .017 | .080 ± .023 | .102 ± .029 |
| EO | 5 | .367 ± .004 | .048 ± .013 | .182 ± .027 | .119 ± .030 | .125 ± .011 | .045 ± .012 | .127 ± .021 | .099 ± .018 | .118 ± .016 |
| EO | 10 | .381 ± .006 | .046 ± .011 | .182 ± .026 | .114 ± .027 | .119 ± .012 | .042 ± .009 | .108 ± .015 | .090 ± .015 | .109 ± .017 |
| EO | 20 | .409 ± .009 | .043 ± .011 | .171 ± .021 | .108 ± .029 | .108 ± .010 | .039 ± .008 | .102 ± .010 | .074 ± .012 | .099 ± .018 |
| EO | 50 | .466 ± .011 | .040 ± .011 | .163 ± .024 | .099 ± .030 | .098 ± .010 | .036 ± .009 | .095 ± .013 | .064 ± .012 | .088 ± .022 |
| EO | 60 | .479 ± .010 | .041 ± .011 | .157 ± .020 | .099 ± .030 | .097 ± .011 | .036 ± .011 | .096 ± .014 | .064 ± .012 | .087 ± .021 |
Table A.13. Raw EO gap on test and on training members, mean over the cells of each evidence stratum, for the non-degenerate points of the sweeps (test log-loss below 0.5; mean ± standard deviation over 10 runs). Train and test values come from the same model. A training gap that falls much faster than the test gap on the sparse strata (weak, boundary) indicates fitting of training-set noise.
| 5 | 10 | 15 | 20 | 30 | 50 | 80 | 120 | 150 | |
| mean | 1.03 | 1.29 | 1.55 | 1.49 | 1.96 | 2.15 | 2.99 | 3.57 | 4.11 |
| certified fraction | .09 ± .09 | .23 ± .16 | .33 ± .18 | .27 ± .18 | .52 ± .15 | .58 ± .24 | .85 ± .18 | .95 ± .08 | .97 ± .07 |
| cells (all runs) | 58 | 60 | 60 | 60 | 60 | 60 | 60 | 60 | 60 |
Table A.14. Size sweep: ERM models on a variant of the benchmark with one reference cell (, true-positive rate 0.85) and six probe cells per design size , all with planted gap . For each , the mean certification statistic over the probe cells and runs, computed on hard training predictions of the final ERM model, and the fraction of probe cells certified () as mean ± standard deviation over 10 runs. At , two of the 60 cells had fewer than two positives and were skipped.
Per-mechanism detailed numbers
Threshold smoothing. On Adult L4 10/90 (10 splits 3 seeds), reaching requires at log-loss for NoThreshSmooth, versus at log-loss for SAFER. At matched fairness, the difference is roughly 0.14 nats. On COMPAS L3 2080, the gap is 0.01–0.02 nats at matched MSE through the mid-range (Figure 2, center).
Dense tail. On COMPAS L3 2080, at the four arms are within the SE bands of one another. From onward, NoDenseTail plateaus at ; the gap to SAFER widens monotonically and reaches roughly ten standard errors at (SAFER 0.018 ± 0.001 versus NoDenseTail 0.060 ± 0.004). On Adult L4 1090 the truncation has a different shape but the predicted form: NoDenseTail stays close to ERM MSE () for almost every , and only at the largest tested does it move toward smaller MSE, where it is Pareto-dominated by SAFER. On German L3 50/50 NoDenseTail tracks SAFER, because the significance gate already keeps the hinge inactive (Proposition 4).
Gate threshold (NoSelectivity on German). SAFER preserves ERM-level log-loss () up to and degrades only mildly (0.59 at ). NoSelectivity climbs from log-loss 0.53 at through 0.60 at and reaches 0.69 by ; the error bars on its MSE become very large in this range.
NoThreshSmooth on German: no departure from ERM
NoThreshSmooth on German exhibits a different and benign failure mode from NoSelectivity: the raw soft-count gradient is too small to move the model on this small dataset, so the optimizer never leaves ERM regardless of . This is consistent with Proposition 3: when subgroup distributions are tightly clustered well above (or below) the decision boundary, raw probabilities and hard decisions disagree but the threshold-smoothing step that would expose the hard-decision gap is absent. This safe behavior is therefore a side effect of the small gradient and is not guaranteed by the method.
Selective-gating: what did not appear
The hypothesis for the selective-gating regime, stated in Appendix F.3, is that the gate threshold gives SAFER an advantage over an ungated alternative when active and inactive subgroups coexist. The Adult L4 1090 and COMPAS L3 2080 results in Figure 2 (left and center) do not exhibit this behavior: NoSelectivity tracks SAFER up to on Adult, where it then loses utility, and across the sweep on COMPAS. We interpret this as evidence that the gate’s empirical role is asymmetric on these two datasets: it suppresses harm in the safety regime, while the evidence for its selective-regime benefit is directional and comes from the controlled benchmark of Appendix F.3, where the evidence strata are isolated.
Cross-regime ablation map
Table A.15: Empirical effect of removing each component on each stress variant. “≈0” indicates indistinguishable from full SAFER within ±SE. “↓↓” indicates Pareto-dominated by SAFER across the regularization sweep. “↓” indicates a Pareto gap that opens at high . “harm” indicates that the ablation degrades utility without any certified-fairness benefit. “n/a” indicates the ablation was not run rigorously at the matched protocol.
| Ablation | Adult L4 1090 | COMPAS L3 2080 | German L3 5050 |
| NoThreshSmooth (Theorem 2) | ↓↓ | ↓↓ | ≈0 (no traction) |
| NoDenseTail ((12)) | ↓ at high | ↓ at high | ≈0 |
| NoSelectivity (Proposition 4) | ≈0, harm at high | ≈0 | harm |
Table A.15.
The Eq. (12) mechanism predicts that NoDenseTail under-reaches SAFER at high on certified-violation datasets, and we observe this on both Adult L4 1090 and COMPAS L3 2080. The shape of the truncation differs: COMPAS shows a clean plateau at from onward, while Adult shows NoDenseTail stuck near ERM MSEE for almost the entire sweep, escaping only at the largest tested and remaining Pareto-dominated by SAFER (Figure 2, left). Both shapes are consistent with the same hinge-suppression argument; a more controlled comparison of where each dataset’s subgroup statistics sit relative to the gate boundary would tighten the quantitative match and is left to future work.
Statistical Parity results
The main text focuses on EO because its positive-label conditioning makes the sparse-cell issue most severe. SP has larger effective subgroup sizes because it conditions on all examples in a subgroup. Figure A.1 shows the Adult, COMPAS and German SP Pareto frontiers. The penalty baseline, SP-Penalty, is the certainty-blind squared-disparity regularizer applied to the SP gap. Results are consistent with the EO interpretation in § 5.
Other instantiations
Equalized Odds
Section 4 shows that the SAFER framework extends beyond the single-component SP/EO case to multi-component criteria, by summing componentwise hinges. We demonstrate this concretely for
Figure A.1. Pareto frontiers for Adult, COMPAS and German Credit under Statistical Parity.
Equalized Odds [17], which requires simultaneous parity of the true-positive rate (TPR) and the false-positive rate (FPR) across subgroups:
The TPR component is the Equal Opportunity criterion already treated in the main paper; FPR parity is the complementary condition on negative-label examples.
FPR soft counts. Section 4 defines TPR soft counts by restricting to positive-label examples. The FPR component requires the analogous restriction to negative-label examples:
These induce a differentiable FPR estimate and, via the identical two-sample proportion construction, a differentiable FPR disparity and LRT statistic where is the log-likelihood ratio for the null of equal FPR between and .
Combined SAFER objective. The Equalized Odds variant penalizes both components independently:
Each component uses the same direct significant-excess form as Eq. (11), but with soft counts restricted to the corresponding label stratum. The significance gate and the safety argument apply componentwise: a subgroup whose TPR and FPR components both lie below the gate threshold has only the exponentially small below-threshold gradient described by Eq. (12), plus any dense-tail residual.
Effective sample sizes. For EqOdds, the binding constraint is the smaller of the two label-conditioned counts:
Whenever (the typical regime for sparse intersectional subgroups), the positive-label count is the binding term; when negative labels are rarer, the FPR component is binding instead.
Metric. Specializing Definition 1 to the two components yields the combined certified-excess metric:
where each component is the significant excess of Definition 1 applied to the respective label stratum. This metric is zero if and only if no subgroup has a certified violation in either the TPR or FPR component, inheriting property 1 of the significant-excess family.
Disparate Impact instantiation
Disparate Impact (DI) [12] measures the ratio of positive-prediction rates across subgroups:
We use DI in its parity formulation, which tests whether , rather than the thresholded 0.8-rule used in legal practice. The parity null is equivalent to , which is exactly the null of Statistical Parity. Consequently, test statistic and the two-sample LRT statistic for DI parity are identical to the one already defined for SP. Therefore, the SAFER training objective for DI is the same as the SP variant of Eq. (16).
Although the training objective is shared with SP, the significant-excess metric can be expressed in log-ratio units, by defining the log-DI functional . The significant-excess is derived by following Definition 1 given a confidence interval from a statistical test on .
Proofs for Section 4
Throughout this appendix, we write , where is the logistic sigmoid, and .
Basic sigmoid bounds
Lemma 5 (Sigmoid–step difference). Let be the logistic sigmoid and . For every and ,
Proof. If , the difference is . If , it is . At , both sides are .
Lemma 6 (Integral and derivative bounds). For every ,
Proof. The integral follows by the change of variables :
For the derivative bound, use . If , then . Symmetry gives the case .
Proof of Theorem 2
Let be the conditional distribution of on the conditioning population , and define
By Lemma 5,
Let be the interval on which, by assumption, has a density bounded by , so that on . We split the integral at . On , substituting , extending the range of integration to the real line and using Lemma 6,
Outside we have , so , and the mass of outside is at most one; hence
Adding the two parts,
The same bound holds for . Since , the triangle inequality gives:
The remainder is also linear in : since for , we have , so the whole bound is at most . Outside the score distribution is arbitrary; in particular it may have atoms at 0 and 1. □
Proof of Proposition 3
Fix . Choose and let concentrate smoothly in a small interval around , while concentrates smoothly in a small interval around . Then
so
for sufficiently narrow intervals. The distributions may be chosen with continuous densities bounded by a finite constant.
To see that no constant rescaling works uniformly, compare this construction with a second one in which both groups are well above the threshold, with mean scores differing by . Then while . Matching the first construction as requires a rescaling constant of order , which then makes the rescaled raw disparity in the second construction diverge instead of matching zero. Hence a single constant cannot align raw soft counts with hard-decision disparities across both regimes. □
Gradient localization at the decision boundary
The following result, deferred from § 4.1, shows that threshold smoothing localizes the fairness gradient to examples near the decision boundary .
Theorem 7 (Gradient localization). For SP and EO,
for a bounded subgroup-membership weight . As , the gradient concentrates on .
Proof. For ,
where is the appropriate conditioning-population weight. The corresponding complement term has the same form with a possibly different weight. Therefore, for a bounded subgroup-membership weight ,
using Lemma 6. Finally, is an approximate identity that converges weakly to a Dirac mass at , yielding the distributional limit.
Canonical surrogate approximation
The main text (Eq. (11)) uses an LRT-based threshold. Here we state and prove a metric-alignment bound for a canonical version in which the surrogate and the metric share the population Wald threshold . The bound shows that the surrogate shape tracks , defined below. This population comparison does not bound the discrepancy between the implemented LRT-based training loss and the SAFT audit metric.
Let , where is evaluated with population cell probabilities. Define
Theorem 8 (Canonical surrogate approximation). Under Theorem 2’s density assumption, with the same and , for every ,
Proof. Fix a subgroup , write , and define
The elementary inequalities
and the Lipschitz property of imply
Since , Theorem 2 gives
Averaging this bound over proves Eq. (A.10).
Asymptotic equivalence of the soft-count LRT and the Wald statistic
Write for the soft count of observation , so that and are the means of the over and , is the pooled mean, and , . With the Bernoulli divergence, Eq. (9) reads
Set and , so that and . Since for bounded away from 0 and 1,
The plug-in Wald variance of property (ii) satisfies , because and differ from by . Under the null and under local alternatives, where disparities are of the order of their standard error, in probability; this is the range in which the gate operates. There , and for nonzero Eq. (10) gives .
Null law of the soft-count statistic. Fix and the predictor independently of the sample. Assume independent observations within and across groups, identically distributed within each group, with a common soft mean under the null and . For fractional counts the Bernoulli form overstates the variance: in each group , and the central limit theorem applied to in the expansion above gives with
The sampling standard error of the soft mean contrast is ; the Bernoulli-form denominator is a proxy for it. For , the reference gate opens under soft parity with asymptotic probability . The same factor scales , preserving the local equivalence of the two statistics. For , division by a consistent estimate of would give asymptotic nominal size under these same fixed-predictor, soft-null assumptions; SAFER retains the Bernoulli reference cutoff.
For exact hard indicators , the corresponding deviance instead has Wilks’ null limit when the common hard positive rate is interior. To relate this to soft counts as , assume that neither conditional score distribution has mass at and that the limiting hard rates are interior. Bounded convergence then gives in each group, and hence along models satisfying soft parity.
At fixed , equal hard positive rates need not give equal soft means, so the null of the gate is soft parity, and certification of hard-decision parity is the task of the held-out hard-count audit of § 3. The fixed-predictor result above applies when the statistic is evaluated on data independent of the fit. During training the sample is also used to fit , and there the gate is an optimization device that uses the audit’s reference cutoff.
Gradient identities for Section 4.3
Recall from § 4.3 that , where denotes stop-gradient. Since the stop-gradient makes the threshold a constant for differentiation, . By the chain rule and ,
Taking norms gives Eq. (12). If , then ; if , then .
The dense-tail contribution for subgroup is . Differentiating gives . This expression contains no factor , because the LRT statistic is computed from the soft counts directly and does not pass through the significant-excess softplus.
Size of the suppression. With , the multiplier equals at the gate boundary , at , and at . The suppression below the boundary is the intended safety mechanism, but it also acts on active subgroups that strong regularization pushes back toward the boundary, so the high-regularization frontier can stall; the dense tail supplies a weak gradient path that does not contain this multiplier.
Why . Three candidates have the required descent direction. An ungated raw-disparity term would be certainty-blind and would penalize noisy disparities regardless of their statistical support. The statistic itself would place the tail on a squared evidence scale and produce much steeper precision-dependent gradients. The intermediate choice remains on the same standardized evidence scale as the gate and satisfies near the gate boundary (property (iii) in Section 4.2), so it retains the uncertainty weighting without introducing a new estimator or threshold; its strength is governed by the single weight .
Effect on the safety guarantee. Because the tail is ungated, Eq. (15) carries an additive residual proportional to , whose contribution to the complete objective scales as ; only the gated component retains the exponentially small gradient guarantee below the gate boundary. Setting removes the residual and recovers the gated-only objective. All full-SAFER runs use the fixed value ; the NoDenseTail ablation sets it to zero. The proposition bounds the back-propagated fairness contribution at one point. How far training moves from the ERM optimum depends also on the curvature of the prediction loss there: for a scalar quadratic prediction loss with curvature , a constant residual gradient of size shifts the minimizer by . The hypothesis is a margin condition: an empty active set alone gives only the multiplier bound , and the exponential bound needs large relative to .
Operating regimes: the gate boundary
The regimes describe the state of the gate at a checkpoint. The implemented boundary is ; in the regular unclamped case with nonzero disparity the active side is . Softplus makes the transition between the gate states continuous through the multiplier of Eq. (12), whose values at are given above and give a concrete meaning to “well below” and “well above” the boundary. There is no universal sample-size boundary. For EO the local approximation
shows that certification depends jointly on the disparity magnitude, the outcome rates, and the positive-label counts on both sides; for SP the total subgroup counts take their place.
Stop-gradient and the threshold across updates
Write for the smoothed disparity and for the threshold. At iteration , SAFER differentiates , so that
The threshold is held fixed during that backward pass only. After the parameters are updated, is recomputed from the new predictions and can increase or decrease.
A scalar example. Let , and , and absorb the positive softplus factor and the learning rate into an effective step of ; the example sets the gate factor aside and shows only the direction of the two gradient routes. With stop-gradient, the direct term differentiates only , giving ; on the next forward pass the threshold is recomputed and changes from to $1.8. Without stop-gradient, the derivative of the excess is , so descent instead gives : the disparity increases from to $1.1, but the threshold rises faster, from to $2.2, and the excess decreases. Detachment removes this route to a lower penalty.
Threshold drift in practice. We logged the unweighted mean of over the subgroups eligible for the EO loss (at least two positive-label examples in both and : 156 subgroups on Adult and 35 on COMPAS) together with the complete fairness loss. Table A.16 reports means standard deviations over 10 Adult and 5 COMPAS runs; “final” is epoch 199 on Adult and each seed’s early-stopped epoch on COMPAS. Near initialization the scores are approximately constant, so disparities and the fairness loss are small, and both grow as the prediction loss learns predictive structure. The mean threshold moves from 0.281 to 0.269 on Adult and from 0.128 to 0.110 on COMPAS. These are averages over subgroups, and individual thresholds can rise; the guarantee itself concerns one update, in which the gated term supplies no gradient through .
| dataset (runs) | mean : initial / final | fairness loss: initial / final |
| Adult (10) | .281 .032 / .269 .014 | .0015 .0007 / .0037 .0011 |
| COMPAS (5) | .128 .021 / .110 .003 | .0034 .0045 / .0205 .0023 |
Table A.16. Mean gate threshold over eligible subgroups and complete fairness loss at the start and end of training (mean standard deviation over runs).
Proof of Proposition 4
At , assume for all . By Eq. (12),
The bounded-gradient regularity condition in Proposition 4 allows the finite average of to be absorbed into the constant. Summing over subgroups gives Eq. (14). Adding the dense-tail gradients and applying the triangle inequality gives Eq. (15). □
References
- [1]Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A reductions approach to fair classification. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80 of PMLR, pages 60–69, 2018.arxiv.org/abs/1803.02453
- [2]Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. In Ethics of data and analytics, pages 254–264. Auerbach Publications, 2022.DOI
- [3]Ainhize Barrainkua, Paula Gordaliza, Jose A. Lozano, and Novi Quadrianto. Uncertainty matters: Stable conclusions under unstable assessment of fairness results. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 1198–1206. PMLR, 2024.
- [4]Yahav Bechavod and Katrina Ligett. Penalizing unfairness in binary classification. arXiv preprint arXiv:1707.00044, 2017.arxiv.org/abs/1707.00044
- [5]Philippe Besse, Eustasio del Barrio, Paula Gordaliza, and Jean-Michel Loubes. Confidence intervals for testing disparate impact in fair learning. arXiv preprint arXiv:1807.06362, 2018.arxiv.org/abs/1807.06362
- [6]Maarten Buyl, Marybeth Defrance, and Tij De Bie. fairret: a framework for differentiable fairness regularization terms. In Proceedings of the 12th International Conference on Learning Representations (ICLR), 2024.
- [7]John J. Cherian and Emmanuel J. Candès. Statistical inference for fairness auditing. Journal of Machine Learning Research, 25(149):1–49, 2024.arxiv.org/abs/2305.03712
- [8]Kimberle Crenshaw. Mapping the margins: Intersectionality, identity politics, and violence against women of color. Stanford Law Review, 43(6):1241–1299, 1991.
- [9]Emily Diana, Wesley Gill, Michael Kearns, Krishnaram Kenthapadi, and Aaron Roth. Minimax group fairness: Algorithms and experiments. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (AIES), pages 66–76, 2021.arxiv.org/abs/2011.03108
- [10]Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems, 34, 2021.arxiv.org/abs/2108.04884
- [11]Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, ITCS ’12, page 214–226, New York, NY, USA, 2012. Association for Computing Machinery.
- [12]Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. Certifying and removing disparate impact. In Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 259–268, 2015.arxiv.org/abs/1412.3756
- [13]Antonio Ferrara, Francesco Cozzi, Alan Perotti, André Panisson, and Francesco Bonchi. Size-adaptive hypothesis testing for fairness. In Advances in Neural Information Processing Systems (NeurIPS) 2025, 2025.arxiv.org/abs/2506.10586
- [14]James R Foulds, Rashidul Islam, Kamrun Naher Keya, and Shimei Pan. An intersectional definition of fairness. In 2020 IEEE 36th international conference on data engineering (ICDE), pages 1918–1921. IEEE, 2020.
- [15]Avijit Ghosh, Lea Genuit, and Mary Reagan. Characterizing intersectional group fairness with worst-case comparisons. In Proceedings of the 2nd Workshop on Diversity in Artificial Intelligence (AIDBEI), volume 142 of PMLR, pages 22–34, 2021.arxiv.org/abs/2101.01673
- [16]Usman Gohar and Lu Cheng. A survey on intersectional fairness in machine learning: Notions, mitigation, and challenges. In Edith Elkind, editor, Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pages 6619–6627. International Joint Conferences on Artificial Intelligence Organization, 2023. Survey Track.DOI
- [17]Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 29, pages 3315–3323, 2016.arxiv.org/abs/1610.02413
- [18]Tatsunori B. Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographics in repeated loss minimization. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80 of PMLR, pages 1929–1938, 2018.arxiv.org/abs/1806.08010
- [19]Úrsula Hébert-Johnson, Michael P. Kim, Omer Reingold, and Guy N. Rothblum. Multicalibration: Calibration for the (Computationally-Identifiable) masses. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80 of PMLR, pages 1939–1948, 2018.
- [20]Johannes Himmelreich, Arbie Hsu, Ellen Veomett, and Kristian Lum. The intersectionality problem for algorithmic fairness. In Miriam Rateike, Awa Dieng, Jamelle Watson-Daniels, Ferdinando Fioretto, and Golnoosh Farnadi, editors, Proceedings of the Algorithmic Fairness Through the Lens of Metrics and Evaluation, volume 279 of Proceedings of Machine Learning Research, pages 68–95. PMLR, 2025.arxiv.org/abs/2411.02569
- [21]Hans Hofmann. Statlog (German Credit Data). UCI Machine Learning Repository, 1994.
- [22]Max Hort, Zhenpeng Chen, Jie M. Zhang, Mark Harman, and Federica Sarro. Bias mitigation for machine learning classifiers: A comprehensive survey. ACM Journal on Responsible Computing, 1(2):1–52, 2024.arxiv.org/abs/2207.07068
- [23]Rashidul Islam, Kamrun Naher Keya, Shimei Pan, Anand D. Sarwate, and James R. Foulds. Differential fairness: An intersectional framework for fair AI. Entropy, 25(4):660, 2023.DOI
- [24]Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. Fairness-aware classifier with prejudice remover regularizer. In Machine Learning and Knowledge Discovery in Databases (ECML PKDD), volume 7524 of Lecture Notes in Computer Science, pages 35–50. Springer, 2012.DOI
- [25]Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80 of PMLR, pages 2564–2572, 2018.arxiv.org/abs/1711.05144
- [26]Ron Kohavi. Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, pages 202–207, 1996.
- [27]Selim Kuzucu, Jiaee Cheong, Hatice Gunes, and Sinan Kalkan. Uncertainty as a fairness measure. J. Artif. Int. Res., 81, 2025.DOI
- [28]Firas Laakom, Haobo Chen, Jürgen Schmidhuber, and Yuheng Bu. Fairness overfitting in machine learning: An information-theoretic perspective. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 32078–32115. PMLR, 2025.arxiv.org/abs/2506.07861
- [29]Ho Ming Lee, Katrien Antonio, Benjamin Avanzi, Lorenzo Marchi, and Rui Zhou. Machine learning with multitype protected attributes: Intersectional fairness through regularisation. arXiv preprint arXiv:2509.08163, 2025.arxiv.org/abs/2509.08163
- [30]Victor Lo, Sayan Datta, and Youssouf Salami. Bringing practical statistical science to AI and predictive model fairness testing. AI and Ethics, 5(3):2149–2164, 2025.
- [31]Yan Luo, Muhammad Osama Khan, Yu Tian, Min Shi, Zehao Dou, Tobias Elze, Yi Fang, and Mengyu Wang. Fair Vision: Equitable deep learning for eye disease screening via fair identity scaling. arXiv preprint arXiv:2310.02492, 2023.
- [32]Mathieu Molina and Patrick Loiseau. Bounding and approximating intersectional fairness through marginal fairness. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc.arxiv.org/abs/2206.05828
- [33]Jiří Němeček, Mark Kozdoba, Ilia Kryvoviáz, Tomáš Pevný, and Jakub Mareček. Intersectional fairness via mixed-integer optimization. arXiv preprint arXiv:2601.19595, 2026.arxiv.org/abs/2601.19595
- [34]Taeеon Park, Sangwon Jung, Sanghyuk Chun, and Taesup Moon. FairDRO: Group fairness regularization via classwise robust optimization. Neural Networks, 182:106891, 2025.DOI
- [35]Manish Raghavan and Pauline T. Kim. Limitations of the “four-fifths rule” and statistical parity tests for measuring fairness. Geo. L. Tech. Rev., 8:93, 2024.
- [36]Lucas Rosenblatt and R. Teal Witter. Fairlyuncertain: A comprehensive benchmark of uncertainty in algorithmic fairness. arXiv preprint arXiv:2410.02005, 2024.arxiv.org/abs/2410.02005
- [37]Jie Tang, Chuanlong Xie, Xianli Zeng, and Lixing Zhu. Empirical likelihood-based fairness auditing: Distribution-free certification and flagging. arXiv preprint arXiv:2601.20269, 2026.arxiv.org/abs/2601.20269
- [38]Ke Yang, Joshua R. Loftus, and Julia Stoyanovich. Causal intersectionality and fair ranking. In Symposium on Foundations of Responsible Computing, 2021.arxiv.org/abs/2006.08688
- [39]I-Cheng Yeh and Che-hui Lien. The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert systems with applications, 36(2):2473–2480, 2009.DOI
- [40]Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society (AIES), pages 335–340, 2018.


