betaFeedback

WorksMathematics

Hidden Tails: Certifying Tail-Risk Claims under Selective Labels

Abstract

Historical decisions determine which outcomes are observed, potentially concealing a predictor's largest errors. Even unlimited data from the same observation process can therefore leave population tail risk unresolved. Here we derive sharp bounds on conditional value-at-risk, the average loss in a specified upper tail, for binary outcomes and bounded losses. These bounds span exactly the risks compatible with the observations and a stated limit on how outcome odds differ between observed and hidden cases at the same covariate values. Under stated regularity conditions, calibrated statistical certificates cover these ranges simultaneously across candidate predictors, supporting risk limits and model comparisons while separating statistical uncertainty from persistent ambiguity. For a fixed predictor and loss threshold, we derive a minimum-cost allocation for reducing excess-loss ambiguity by prioritizing hidden outcomes according to ambiguity removed per unit cost. This allocation treats audited population contributions as known and retains the original sensitivity bounds separately for audited and still-hidden outcomes. In controlled masking experiments with known loan repayments, recovering about 16% of hidden labels removes a median 61% of estimated fixed-threshold ambiguity, compared with 16% for random recovery. These results connect population tail-risk claims to the available evidence and identify where additional outcomes can sharpen their assessment.

Introduction

Population tail-risk claims guide decisions about whether a predictor meets an acceptable loss limit and whether one predictor has lower tail risk than another. Yet historical decisions determine which outcomes are available for evaluation [1, 2, 3], and the missing cases can carry the largest prediction losses. For example, a hospital assessing a model for 30-day adverse events may lack event status for patients who receive subsequent care outside its network; record linkage can later recover those labels. Conditional value-at-risk (CVaR) averages the upper tail of prediction loss [4, 5, 6]. Certifying this quantity for all target patients therefore requires an explicit account of hidden outcomes.

The obstacle persists even if the observed distribution is known exactly. Suppose labels are observed for 25% of a population with constant covariates, with equal proportions of zeros and ones among observed outcomes. Under zero–one loss and CVaR at α=.5\alpha= .5, the constant predictors f=0f = 0 and g=1g = 1 have identical observed loss distributions. If every hidden label is zero, their population CVaRs are (.25,1)(.25, 1); if every hidden label is one, they are (1,.25)(1, .25). Both worlds produce the same records and have positive observation probability. This illustration allows unrestricted hidden outcomes and shows why historical records alone cannot certify an ordering Appendix § 12.§ 2.

We make the assumptions needed for certification explicit through a finite, externally specified sensitivity level Γ\Gamma, which limits how much outcome odds can differ between hidden and observed cases at the same covariates. The compatible population laws yield a sharp CVaR interval: every value in it is attained under the observations and the restriction. Its width measures ambiguity that persists even with unlimited historical data. Figure 1 depicts this range for one fixed predictor; the additional enlargement accounts for estimating its endpoints from a finite sample.

Figure 1. Risk ambiguity and statistical uncertainty for one fixed predictor. (a) The same observed law is compatible with hidden-outcome laws giving population CVaR 0.36 and 0.54. (b) The dark segment is the sharp population interval; pale extensions account for estimating its endpoints from a finite sample. Colors link the laws to the endpoints. Geometry is schematic. Appendix § 12.§ 2 gives the numerical construction.

Our goal is to connect three questions: which risks are compatible with the evidence, how precisely their bounds can be estimated, and which additional labels most reduce unresolved ambiguity. An upper bound below a chosen loss limit certifies that criterion; separated intervals establish a predictor ordering. When these rules leave a decision unresolved, targeted outcome collection can reduce the underlying information gap.

Contributions.

  1. Sharp population identification. For binary outcomes and bounded losses, we derive attainable CVaR endpoints and show that every intermediate risk is compatible. For each endpoint, one hidden-outcome law is extremal at every loss threshold, reducing its calculation to a scalar convex minimization (§ 3).

  1. Uniform finite-sample certification. Paired, locally robust endpoint estimates give simultaneous bounds across candidate predictors at fixed sensitivity and tail levels. The bounds expose the costs of assessing a smaller tail, weaker observation overlap, and auxiliary-estimation error, and support comparisons and predictor selection using the same data (§ 4).

  1. Minimum-cost revelation. Under a specified information model, ranking tail identification value per unit cost gives the exact optimal revelation allocation for a fixed predictor and threshold. A zero-threshold bound also controls optimized interval width (§ 5).

Revelation recovers existing outcomes for a fixed target population. Our information model treats audited contributions as known and retains the original sensitivity bounds separately for audited and still-hidden groups.

Synthetic experiments separate persistent identification width from decreasing endpoint error (Figure 2). Historical LendingClub data illustrate changes in population assessment, while controlled masking of known outcomes evaluates coverage and comparisons (Figure 3). Fixed and sequential audits examine how acquisition choices change estimated ambiguity (Figure 4).

Figure 2. Statistical uncertainty contracts around an identification floor. (a) Mean oracle-calibrated outer width and mean exact population width; shading shows their difference. (b) Cross-fitted endpoint RMSE with learned nuisances; the faint line is an n−1/2n^{-1/2} guide. Each sample size uses 60 simulated populations and 80 datasets per population. The oracle coverage target is the entire sharp interval. Appendix § 15.§ 2 gives the design and complete results.

Figure 3. Observation and sensitivity change population assessment. (a) Accepted-only CVaR (hollow points) and estimated applicant sensitivity endpoints (blue segments) at α=.95\alpha= .95, Γ=2\Gamma= 2, under the regression extension of Appendix § 15.§ 3. Segments precede statistical enlargement. (b) Coverage of full-outcome reference CVaR under controlled masking of mature issued loans, pooled across three predictors and 20 shared masks at α=.95\alpha= .95. The guide marks the masking mechanism’s sensitivity Γ0=3\Gamma_0 = 3; fold enlargement is empirical. Appendix § 15.§ 4 provides the complete results.

Figure 4. Targeted audits reduce estimated ambiguity at equal within-protocol budgets. (a) Median remaining fixed-threshold gap, normalized within each mask by its initial estimated gap. (b) Median estimated retained-sensitivity width for the predictor refitted after each audit batch. Both use boosting, α=.95\alpha= .95, Γ=3\Gamma= 3, unit hidden-label cost, and 20 masks. Lines join eight recorded batches. Appendix § 15.§ 6 specifies both protocols. Budgets are median fractions of the initially hidden cases in each eligible pool: the original pool in (a) and the 80% training pool in (b).

Positioning. This work connects selective-label evaluation [3, 7, 8] and partial identification [9, 10, 11, 12, 13] with CVaR optimization [14, 15, 16] and orthogonal estimation [17, 18, 19]. Appendix § 16.§ 3 develops these connections. We first make the target risk and the assumptions linking observed and hidden outcomes precise.

Tail Risk under Selective Labels

Tail-risk certification requires a target population and loss, a criterion for large errors, and a restriction on hidden outcomes. CVaR measures average loss within a chosen upper tail, making the severity of the largest prediction errors part of the assessment. The observation model specifies what historical records identify; the sensitivity model bounds how observed and hidden outcome probabilities can differ. Together, these choices define the population risks that our certificates must cover.

Target population and observation process

Let F\mathcal{F} be the candidate predictor class and PP the full-data distribution of (X,S,Y)(X,S,Y), with covariate marginal PXP_X. We evaluate f∈Ff \in\mathcal{F} using loss ℓ(f(X),Y)\ell(f(X),Y), where Y∈{0,1}Y \in\{0,1\}. In decision settings, YY denotes Y(a∗)Y(a^*) under one specified action held fixed across the target population and predictor comparisons. The indicator S∈{0,1}S \in\{0,1\} equals one when the outcome is observed. We observe nn independent copies of O=(X,S,Yobs)O=(X,S,Y^{\mathrm{obs}}), where Yobs=SYY^{\mathrm{obs}}=SY. The observed law PobsP_{\mathrm{obs}} identifies PXP_X and

π(x)=P(S=1∣X=x),p1(x)=P(Y=1∣X=x,S=1),\pi(x)=P(S=1\mid X=x), \qquad p_1(x)=P(Y=1\mid X=x,S=1),

while p0(x)=P(Y=1∣X=x,S=0)p_0(x)=P(Y=1\mid X=x,S=0) is hidden wherever π(x)<1\pi(x)<1. Here π\pi is the historical observation probability and p1p_1 the outcome probability among observed cases; both are auxiliary to the predictor ff being assessed.

Bounded loss gives a common risk scale and supports finite-sample error control. Population overlap makes the observed-outcome distribution available as a reference throughout the target covariate population. We formalize these conditions as follows.

Assumption 1 (Bounded loss and population overlap). For every f∈Ff \in\mathcal{F} and y∈{0,1}y \in\{0,1\}, the map x↦ℓ(f(x),y)x \mapsto\ell(f(x),y) is measurable and takes values in [0,1][0,1], and 0<πmin⁡≤π(X)≤10<\pi_{\min}\leq\pi(X)\leq1 almost surely.

Technical regularity and target-label conventions are in Appendix § 12.§ 1. Write z+=max⁡{z,0}z_+=\max\{z,0\} and Lf=ℓ(f(X),Y)L_f=\ell(f(X),Y). At level α∈(0,1)\alpha\in(0,1), β=1−α\beta=1-\alpha is the upper-tail fraction. The threshold η\eta is a candidate loss cutoff, and (Lf−η)+(L_f-\eta)_+ is the excess above it. At α=.95\alpha=.95, CVaR averages the worst 5% of population losses, allowing fractional mass at a tied cutoff [16]:

ρα,P(f)=CVaR⁡α,P(Lf)=inf⁡η∈[0,1]{η+β−1EP[(Lf−η)+]}.(1)\rho_{\alpha,P}(f)=\operatorname{CVaR}_{\alpha,P}(L_f)=\inf_{\eta\in[0,1]}\left\{\eta+\beta^{-1}\mathbb{E}_P[(L_f-\eta)_+]\right\}. \tag*{(1)}

The expectation includes both observed and hidden cases.

Sensitivity model and certification objective

Outcome-dependent observation can change label prevalence even at fixed covariates. We use the observed outcome probability as a reference and express the allowed departure of hidden outcomes through a multiplicative bound on their odds. For a finite, externally specified Γ≥1\Gamma\geq1, we impose the hidden-to-observed outcome-odds restriction [13 Section 7.2]:

p0(x)∈[p‾(x),p‾(x)]=[q−(p1(x)),q+(p1(x))],π(x)<1,p_0(x)\in[\underline{p}(x),\overline{p}(x)]=[q_-(p_1(x)),q_+(p_1(x))], \qquad\pi(x)<1,
q−(p)=pp+Γ(1−p),q+(p)=Γp1−p+Γp.(2)q_-(p)=\frac{p}{p+\Gamma(1-p)}, \qquad q_+(p)=\frac{\Gamma p}{1-p+\Gamma p}. \tag*{(2)}

For interior probabilities, hidden-to-observed outcome odds lie between Γ−1\Gamma^{-1} and Γ\Gamma; the formulas also cover p1∈{0,1}p_1\in\{0,1\}. At Γ=1\Gamma=1, p0(x)=p1(x)p_0(x)=p_1(x): hidden and observed outcomes have the same conditional distribution given XX, called conditional ignorability. For example, p1(x)=.5p_1(x)=.5 and Γ=2\Gamma=2 allow p0(x)∈[1/3,2/3]p_0(x)\in[1/3,2/3]. Larger Γ\Gamma permits greater departure. The chosen Γ\Gamma defines a declared sensitivity scenario, supported by domain knowledge or additional outcome information (Appendix § 16.§ 1).

Let PΓ(Pobs)\mathcal{P}_{\Gamma}(P_{\mathrm{obs}}) contain all full-data laws with observed marginal PobsP_{\mathrm{obs}} and hidden regression satisfying this restriction. Their risk endpoints are

ρ‾α,Γ(f)=inf⁡P∈PΓρα,P(f),ρ‾α,Γ(f)=sup⁡P∈PΓρα,P(f).(3)\underline{\rho}_{\alpha,\Gamma}(f)=\inf_{P\in\mathcal{P}_{\Gamma}}\rho_{\alpha,P}(f),\qquad\overline{\rho}_{\alpha,\Gamma}(f)=\sup_{P\in\mathcal{P}_{\Gamma}}\rho_{\alpha,P}(f). \tag*{(3)}

For fixed α\alpha and Γ\Gamma, our statistical objective is an interval covering every compatible risk simultaneously for every f∈Ff\in\mathcal{F}, with probability at least 1−δtot1-\delta_{\mathrm{tot}} over the observed sample. Here δtot∈(0,1)\delta_{\mathrm{tot}}\in(0,1) is the total failure budget; § 4 accounts for sample fluctuation and auxiliary-estimation error within this budget. Thus Γ\Gamma specifies the admissible hidden outcomes, while δtot\delta_{\mathrm{tot}} specifies the statistical guarantee.

We next derive these population risk endpoints from the observed law and show that every value between them is attainable.

Sharp Population CVaR Bounds

For binary outcomes, both possible prediction losses can be computed from f(x)f(x), even when the actual label is hidden. Sensitivity bounds constrain the probability assigned to these two losses. This reduces population identification to two scalar convex minimizations.

Fix a predictor ff and a cutoff η∈[0,1]\eta\in[0,1]. Write

af,ηy(x)=(ℓ(f(x),y)−η)+,df,η(x)=af,η1(x)−af,η0(x).a^{y}_{f,\eta}(x)=\bigl(\ell(f(x),y)-\eta\bigr)_{+},\qquad d_{f,\eta}(x)=a^{1}_{f,\eta}(x)-a^{0}_{f,\eta}(x).

Here aya^{y} is the excess loss if the label is yy, and dd is its change when the label changes from zero to one. A hidden outcome probability qq gives expected excess loss (1−q)a0+qa1=a0+qd(1-q)a^{0}+qa^{1}=a^{0}+qd. With d+=(df,η)+d_{+}=(d_{f,\eta})_{+} and d−=(−df,η)+d_{-}=(-d_{f,\eta})_{+}, its largest and smallest allowed values are

bf,ηU=af,η0+p‾Γd+−p‾Γd−,b^{U}_{f,\eta}=a^{0}_{f,\eta}+\overline{p}_{\Gamma}d_{+}-\underline{p}_{\Gamma}d_{-},
bf,ηL=af,η0+p‾Γd+−p‾Γd−.(4)b^{L}_{f,\eta}=a^{0}_{f,\eta}+\underline{p}_{\Gamma}d_{+}-\overline{p}_{\Gamma}d_{-}. \tag*{(4)}

All quantities are evaluated at the same covariate value. The upper bound assigns the greatest allowed probability to the label with larger loss; the lower bound reverses the assignment. Averaging the historically observed and hidden contributions gives

mf,ησ=Eobs[af,ηYobs(X)+(1−S)bf,ησ(X)],σ∈{L,U}.m^{\sigma}_{f,\eta}=\mathbb{E}_{\mathrm{obs}}\left[a^{Y_{\mathrm{obs}}}_{f,\eta}(X)+(1-S)b^{\sigma}_{f,\eta}(X)\right],\qquad\sigma\in\{L,U\}.

where Eobs\mathbb{E}_{\mathrm{obs}} integrates over the observed-data law. Adding the cutoff term forms the CVaR objectives

Lf,η=η+β−1mf,ηL,Uf,η=η+β−1mf,ηU.L_{f,\eta}=\eta+\beta^{-1}m^{L}_{f,\eta},\qquad U_{f,\eta}=\eta+\beta^{-1}m^{U}_{f,\eta}.

Dependence on Γ\Gamma is implicit.

References

  1. [1]Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, and Ziad Obermeyer. Prediction policy problems. American Economic Review, 105(5):491–495, 2015. doi: 10.1257/aer.p20151023.DOI
  2. [2]Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. Human decisions and machine predictions. The Quarterly Journal of Economics, 133(1):237–293, 2018. doi: 10.1093/qje/qjx032.DOI
  3. [3]Himabindu Lakkaraju, Jon Kleinberg, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. The selective labels problem: Evaluating algorithmic predictions in the presence of unobservables. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 275–284, 2017. doi: 10.1145/3097983.3098066.DOI
  4. [4]Carlo Acerbi and Dirk Tasche. Expected shortfall: A natural coherent alternative to value at risk. Economic Notes, 31(2):379–388, 2002. doi: 10.1111/1468-0300.00091.DOI
  5. [5]Philippe Artzner, Freddy Delbaen, Jean-Marc Eber, and David Heath. Coherent measures of risk. Mathematical Finance, 9(3):203–228, 1999. doi: 10.1111/1467-9965.00068.DOI
  6. [6]R. Tyrrell Rockafellar and Stanislav Uryasev. Optimization of conditional value-at-risk. Journal of Risk, 2(3):21–41, 2000. doi: 10.21314/JOR.2000.038.DOI
  7. [7]Jian Chen, Zehao Li, and Xiaojie Mao. Learning with selectively labeled data from multiple decision-makers. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 8480–8519. PMLR, 2025. URL https://proceedings.mlr.press/v267/chen25al.html.
  8. [8]Amanda Coston, Ashesh Rambachan, and Alexandra Chouldechova. Characterizing fairness over the set of good models under selective labels. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2144–2155. PMLR, 2021. URL https://proceedings.mlr.press/v139/coston21a.html.arxiv.org/abs/2101.00352
  9. [9]Charles F. Manski. Anatomy of the selection problem. Journal of Human Resources, 24(3):343–360, 1989. doi: 10.2307/145818.DOI
  10. [10]Charles F. Manski. Partial Identification of Probability Distributions. Springer, 2003.DOI
  11. [11]Daniel O. Scharfstein, Andrea Rotnitzky, and James M. Robins. Adjusting for nonignorable drop-out using semiparametric nonresponse models. Journal of the American Statistical Association, 94(448):1096–1120, 1999. doi: 10.1080/01621459.1999.10473862.DOI
  12. [12]Zhiqiang Tan. A distributional approach for causal inference using propensity scores. Journal of the American Statistical Association, 101(476):1619–1637, 2006. doi: 10.1198/016214506000000023.DOI
  13. [13]Qingyuan Zhao, Dylan S. Small, and Bhaswar B. Bhattacharya. Sensitivity analysis for inverse probability weighting estimators via the percentile bootstrap. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 81(4):735–761, 2019. doi: 10.1111/rssb.12327.DOI
  14. [14]Erick Delage and Yinyu Ye. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations Research, 58(3):595–612, 2010. doi: 10.1287/opre.1090.0741.DOI
  15. [15]John C. Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics, 49(3):1378–1406, 2021. doi: 10.1214/20-AOS2004.DOI
  16. [16]R. Tyrrell Rockafellar and Stanislav Uryasev. Conditional value-at-risk for general loss distributions. Journal of Banking & Finance, 26(7):1443–1471, 2002. doi: 10.1016/S0378-4266(02)00271-6.
  17. [17]Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1–C68, 2018. doi: 10.1111/ectj.12097.DOI
  18. [18]Nathan Kallus, Xiaojie Mao, and Masatoshi Uehara. Localized debiased machine learning: Efficient inference on quantile treatment effects and beyond. Journal of Machine Learning Research, 25(16):1–59, 2024. URL https://jmlr.org/papers/v25/23-0661.html.
  19. [19]James M. Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of regression coefficients when some regressors are not always observed. Journal of the American Statistical Association, 89(427):846–866, 1994. doi: 10.1080/01621459.1994.10476818.DOI
  20. [20]George B. Dantzig. Discrete-variable extremum problems. Operations Research, 5(2):266–288, 1957. doi: 10.1287/opre.5.2.266.DOI
  21. [21]Nathan George. All Lending Club loan data. Kaggle dataset, https://www.kaggle.com/datasets/wordsforthewise/lending-club, 2019. License: CC0 Public Domain.
  22. [22]LendingClub Corporation. Amendment no. 3 to form S-1 registration statement, April 2012. URL https://www.sec.gov/Archives/edgar/data/1409970/00011931251251868/d306143ds1a.htm. Filed April 5, 2012; minimum credit criteria.
  23. [23]Peter L. Bartlett and Shahar Mendelson. Rademacher and Gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3:463–482, 2002. URL https://jmlr.org/papers/v3/bartlett02a.html.DOI
  24. [24]Aad W. van der Vaart and Jon A. Wellner. Weak Convergence and Empirical Processes. Springer, 1996.
  25. [25]Andreas Maurer. A vector-contraction inequality for Rademacher complexities. In Algorithmic Learning Theory, volume 9925 of Lecture Notes in Computer Science, pages 3–17. Springer, 2016.arxiv.org/abs/1605.00251
  26. [26]Bin Yu. Assouad, Fano, and Le Cam. In David Pollard, Erik Torgersen, and Grace L. Yang, editors, Festschrift for Lucien Le Cam, pages 423–435. Springer, 1997. doi: 10.1007/978-1-4612-1880-7_29.
  27. [27]Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011. URL https://jmlr.org/papers/v12/pedregosa11a.html.arxiv.org/abs/1201.0490
  28. [28]LendingClub. Loan and rejected-application data dictionary. URL https://resources.lendingclub.com/LCDataDictionary.xlsx. LoanStats and RejectStats worksheets; accessed November 7, 2026.
  29. [29]Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794. ACM, 2016. doi: 10.1145/2939672.2939785.DOI
  30. [30]Paul R. Rosenbaum. Observational Studies. Springer, 2nd edition, 2002.
  31. [31]Roderick J. A. Little and Donald B. Rubin. Statistical Analysis with Missing Data. Wiley, 3rd edition, 2019.
  32. [32]Charles F. Manski. Nonparametric bounds on treatment effects. American Economic Review, 80(2):319–323, 1990.
  33. [33]Jose Blanchet and Karthyek Murthy. Quantifying distributional model risk via optimal transport. Mathematics of Operations Research, 44(2):565–600, 2019. doi: 10.1287/moor.2018.0936.DOI
  34. [34]Peyman Mohajerin Esfahani and Daniel Kuhn. Data-driven distributionally robust optimization using the Wasserstein metric: Performance guarantees and tractable reformulations. Mathematical Programming, 171:115–166, 2018. doi: 10.1007/s10107-017-1172-1.DOI
  35. [35]David A. Cohn, Zoubin Ghahramani, and Michael I. Jordan. Active learning with statistical models. Journal of Artificial Intelligence Research, 4:129–145, 1996. URL https://arxiv.org/abs/cs/9603104.

Paper details

vlnc:W_gnt67qmv
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents