Introduction
Population tail-risk claims guide decisions about whether a predictor meets an acceptable loss limit and whether one predictor has lower tail risk than another. Yet historical decisions determine which outcomes are available for evaluation [1 , 2 , 3 ] , and the missing cases can carry the largest prediction losses. For example, a hospital assessing a model for 30-day adverse events may lack event status for patients who receive subsequent care outside its network; record linkage can later recover those labels. Conditional value-at-risk (CVaR) averages the upper tail of prediction loss [4 , 5 , 6 ] . Certifying this quantity for all target patients therefore requires an explicit account of hidden outcomes.
The obstacle persists even if the observed distribution is known exactly. Suppose labels are observed for 25% of a population with constant covariates, with equal proportions of zeros and ones among observed outcomes. Under zero–one loss and CVaR at α = .5 \alpha= .5 α = .5 , the constant predictors f = 0 f = 0 f = 0 and g = 1 g = 1 g = 1 have identical observed loss distributions. If every hidden label is zero, their population CVaRs are ( .25 , 1 ) (.25, 1) ( .25 , 1 ) ; if every hidden label is one, they are ( 1 , .25 ) (1, .25) ( 1 , .25 ) . Both worlds produce the same records and have positive observation probability. This illustration allows unrestricted hidden outcomes and shows why historical records alone cannot certify an ordering Appendix § 12 .§ 2 .
We make the assumptions needed for certification explicit through a finite, externally specified sensitivity level Γ \Gamma Γ , which limits how much outcome odds can differ between hidden and observed cases at the same covariates. The compatible population laws yield a sharp CVaR interval: every value in it is attained under the observations and the restriction. Its width measures ambiguity that persists even with unlimited historical data. Figure 1 depicts this range for one fixed predictor; the additional enlargement accounts for estimating its endpoints from a finite sample.
Figure 1. Risk ambiguity and statistical uncertainty for one fixed predictor. (a) The same observed law is compatible with hidden-outcome laws giving population CVaR 0.36 and 0.54. (b) The dark segment is the sharp population interval; pale extensions account for estimating its endpoints from a finite sample. Colors link the laws to the endpoints. Geometry is schematic. Appendix § 12 .§ 2 gives the numerical construction.
Our goal is to connect three questions: which risks are compatible with the evidence, how precisely their bounds can be estimated, and which additional labels most reduce unresolved ambiguity. An upper bound below a chosen loss limit certifies that criterion; separated intervals establish a predictor ordering. When these rules leave a decision unresolved, targeted outcome collection can reduce the underlying information gap.
Contributions.
Sharp population identification. For binary outcomes and bounded losses, we derive attainable CVaR endpoints and show that every intermediate risk is compatible. For each endpoint, one hidden-outcome law is extremal at every loss threshold, reducing its calculation to a scalar convex minimization (§ 3 ).
Uniform finite-sample certification. Paired, locally robust endpoint estimates give simultaneous bounds across candidate predictors at fixed sensitivity and tail levels. The bounds expose the costs of assessing a smaller tail, weaker observation overlap, and auxiliary-estimation error, and support comparisons and predictor selection using the same data (§ 4 ).
Minimum-cost revelation. Under a specified information model, ranking tail identification value per unit cost gives the exact optimal revelation allocation for a fixed predictor and threshold. A zero-threshold bound also controls optimized interval width (§ 5 ).
Revelation recovers existing outcomes for a fixed target population. Our information model treats audited contributions as known and retains the original sensitivity bounds separately for audited and still-hidden groups.
Synthetic experiments separate persistent identification width from decreasing endpoint error (Figure 2 ). Historical LendingClub data illustrate changes in population assessment, while controlled masking of known outcomes evaluates coverage and comparisons (Figure 3 ). Fixed and sequential audits examine how acquisition choices change estimated ambiguity (Figure 4 ).
Figure 2. Statistical uncertainty contracts around an identification floor. (a) Mean oracle-calibrated outer width and mean exact population width; shading shows their difference. (b) Cross-fitted endpoint RMSE with learned nuisances; the faint line is an n − 1 / 2 n^{-1/2} n − 1/2 guide. Each sample size uses 60 simulated populations and 80 datasets per population. The oracle coverage target is the entire sharp interval. Appendix § 15 .§ 2 gives the design and complete results.
Figure 3. Observation and sensitivity change population assessment. (a) Accepted-only CVaR (hollow points) and estimated applicant sensitivity endpoints (blue segments) at α = .95 \alpha= .95 α = .95 , Γ = 2 \Gamma= 2 Γ = 2 , under the regression extension of Appendix § 15 .§ 3 . Segments precede statistical enlargement. (b) Coverage of full-outcome reference CVaR under controlled masking of mature issued loans, pooled across three predictors and 20 shared masks at α = .95 \alpha= .95 α = .95 . The guide marks the masking mechanism’s sensitivity Γ 0 = 3 \Gamma_0 = 3 Γ 0 = 3 ; fold enlargement is empirical. Appendix § 15 .§ 4 provides the complete results.
Figure 4. Targeted audits reduce estimated ambiguity at equal within-protocol budgets. (a) Median remaining fixed-threshold gap, normalized within each mask by its initial estimated gap. (b) Median estimated retained-sensitivity width for the predictor refitted after each audit batch. Both use boosting, α = .95 \alpha= .95 α = .95 , Γ = 3 \Gamma= 3 Γ = 3 , unit hidden-label cost, and 20 masks. Lines join eight recorded batches. Appendix § 15 .§ 6 specifies both protocols. Budgets are median fractions of the initially hidden cases in each eligible pool: the original pool in (a) and the 80% training pool in (b).
Positioning. This work connects selective-label evaluation [3 , 7 , 8 ] and partial identification [9 , 10 , 11 , 12 , 13 ] with CVaR optimization [14 , 15 , 16 ] and orthogonal estimation [17 , 18 , 19 ] . Appendix § 16 .§ 3 develops these connections. We first make the target risk and the assumptions linking observed and hidden outcomes precise.
Tail Risk under Selective Labels
Tail-risk certification requires a target population and loss, a criterion for large errors, and a restriction on hidden outcomes. CVaR measures average loss within a chosen upper tail, making the severity of the largest prediction errors part of the assessment. The observation model specifies what historical records identify; the sensitivity model bounds how observed and hidden outcome probabilities can differ. Together, these choices define the population risks that our certificates must cover.
Target population and observation process
Let F \mathcal{F} F be the candidate predictor class and P P P the full-data distribution of ( X , S , Y ) (X,S,Y) ( X , S , Y ) , with covariate marginal P X P_X P X . We evaluate f ∈ F f \in\mathcal{F} f ∈ F using loss ℓ ( f ( X ) , Y ) \ell(f(X),Y) ℓ ( f ( X ) , Y ) , where Y ∈ { 0 , 1 } Y \in\{0,1\} Y ∈ { 0 , 1 } . In decision settings, Y Y Y denotes Y ( a ∗ ) Y(a^*) Y ( a ∗ ) under one specified action held fixed across the target population and predictor comparisons. The indicator S ∈ { 0 , 1 } S \in\{0,1\} S ∈ { 0 , 1 } equals one when the outcome is observed. We observe n n n independent copies of O = ( X , S , Y o b s ) O=(X,S,Y^{\mathrm{obs}}) O = ( X , S , Y obs ) , where Y o b s = S Y Y^{\mathrm{obs}}=SY Y obs = S Y . The observed law P o b s P_{\mathrm{obs}} P obs identifies P X P_X P X and
π ( x ) = P ( S = 1 ∣ X = x ) , p 1 ( x ) = P ( Y = 1 ∣ X = x , S = 1 ) , \pi(x)=P(S=1\mid X=x), \qquad p_1(x)=P(Y=1\mid X=x,S=1), π ( x ) = P ( S = 1 ∣ X = x ) , p 1 ( x ) = P ( Y = 1 ∣ X = x , S = 1 ) ,
while p 0 ( x ) = P ( Y = 1 ∣ X = x , S = 0 ) p_0(x)=P(Y=1\mid X=x,S=0) p 0 ( x ) = P ( Y = 1 ∣ X = x , S = 0 ) is hidden wherever π ( x ) < 1 \pi(x)<1 π ( x ) < 1 . Here π \pi π is the historical observation probability and p 1 p_1 p 1 the outcome probability among observed cases; both are auxiliary to the predictor f f f being assessed.
Bounded loss gives a common risk scale and supports finite-sample error control. Population overlap makes the observed-outcome distribution available as a reference throughout the target covariate population. We formalize these conditions as follows.
Assumption 1 (Bounded loss and population overlap). For every f ∈ F f \in\mathcal{F} f ∈ F and y ∈ { 0 , 1 } y \in\{0,1\} y ∈ { 0 , 1 } , the map x ↦ ℓ ( f ( x ) , y ) x \mapsto\ell(f(x),y) x ↦ ℓ ( f ( x ) , y ) is measurable and takes values in [ 0 , 1 ] [0,1] [ 0 , 1 ] , and 0 < π min ≤ π ( X ) ≤ 1 0<\pi_{\min}\leq\pi(X)\leq1 0 < π m i n ≤ π ( X ) ≤ 1 almost surely.
Technical regularity and target-label conventions are in Appendix § 12 .§ 1 . Write z + = max { z , 0 } z_+=\max\{z,0\} z + = max { z , 0 } and L f = ℓ ( f ( X ) , Y ) L_f=\ell(f(X),Y) L f = ℓ ( f ( X ) , Y ) . At level α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) , β = 1 − α \beta=1-\alpha β = 1 − α is the upper-tail fraction. The threshold η \eta η is a candidate loss cutoff, and ( L f − η ) + (L_f-\eta)_+ ( L f − η ) + is the excess above it. At α = .95 \alpha=.95 α = .95 , CVaR averages the worst 5% of population losses, allowing fractional mass at a tied cutoff [16 ] :
ρ α , P ( f ) = CVaR α , P ( L f ) = inf η ∈ [ 0 , 1 ] { η + β − 1 E P [ ( L f − η ) + ] } . (1) \rho_{\alpha,P}(f)=\operatorname{CVaR}_{\alpha,P}(L_f)=\inf_{\eta\in[0,1]}\left\{\eta+\beta^{-1}\mathbb{E}_P[(L_f-\eta)_+]\right\}. \tag*{(1)} ρ α , P ( f ) = CVaR α , P ( L f ) = η ∈ [ 0 , 1 ] inf { η + β − 1 E P [( L f − η ) + ] } . (1)
The expectation includes both observed and hidden cases.
Sensitivity model and certification objective
Outcome-dependent observation can change label prevalence even at fixed covariates. We use the observed outcome probability as a reference and express the allowed departure of hidden outcomes through a multiplicative bound on their odds. For a finite, externally specified Γ ≥ 1 \Gamma\geq1 Γ ≥ 1 , we impose the hidden-to-observed outcome-odds restriction [13 Section 7.2] :
p 0 ( x ) ∈ [ p ‾ ( x ) , p ‾ ( x ) ] = [ q − ( p 1 ( x ) ) , q + ( p 1 ( x ) ) ] , π ( x ) < 1 , p_0(x)\in[\underline{p}(x),\overline{p}(x)]=[q_-(p_1(x)),q_+(p_1(x))], \qquad\pi(x)<1, p 0 ( x ) ∈ [ p ( x ) , p ( x )] = [ q − ( p 1 ( x )) , q + ( p 1 ( x ))] , π ( x ) < 1 ,
q − ( p ) = p p + Γ ( 1 − p ) , q + ( p ) = Γ p 1 − p + Γ p . (2) q_-(p)=\frac{p}{p+\Gamma(1-p)}, \qquad q_+(p)=\frac{\Gamma p}{1-p+\Gamma p}. \tag*{(2)} q − ( p ) = p + Γ ( 1 − p ) p , q + ( p ) = 1 − p + Γ p Γ p . (2)
For interior probabilities, hidden-to-observed outcome odds lie between Γ − 1 \Gamma^{-1} Γ − 1 and Γ \Gamma Γ ; the formulas also cover p 1 ∈ { 0 , 1 } p_1\in\{0,1\} p 1 ∈ { 0 , 1 } . At Γ = 1 \Gamma=1 Γ = 1 , p 0 ( x ) = p 1 ( x ) p_0(x)=p_1(x) p 0 ( x ) = p 1 ( x ) : hidden and observed outcomes have the same conditional distribution given X X X , called conditional ignorability. For example, p 1 ( x ) = .5 p_1(x)=.5 p 1 ( x ) = .5 and Γ = 2 \Gamma=2 Γ = 2 allow p 0 ( x ) ∈ [ 1 / 3 , 2 / 3 ] p_0(x)\in[1/3,2/3] p 0 ( x ) ∈ [ 1/3 , 2/3 ] . Larger Γ \Gamma Γ permits greater departure. The chosen Γ \Gamma Γ defines a declared sensitivity scenario, supported by domain knowledge or additional outcome information (Appendix § 16 .§ 1 ).
Let P Γ ( P o b s ) \mathcal{P}_{\Gamma}(P_{\mathrm{obs}}) P Γ ( P obs ) contain all full-data laws with observed marginal P o b s P_{\mathrm{obs}} P obs and hidden regression satisfying this restriction. Their risk endpoints are
ρ ‾ α , Γ ( f ) = inf P ∈ P Γ ρ α , P ( f ) , ρ ‾ α , Γ ( f ) = sup P ∈ P Γ ρ α , P ( f ) . (3) \underline{\rho}_{\alpha,\Gamma}(f)=\inf_{P\in\mathcal{P}_{\Gamma}}\rho_{\alpha,P}(f),\qquad\overline{\rho}_{\alpha,\Gamma}(f)=\sup_{P\in\mathcal{P}_{\Gamma}}\rho_{\alpha,P}(f). \tag*{(3)} ρ α , Γ ( f ) = P ∈ P Γ inf ρ α , P ( f ) , ρ α , Γ ( f ) = P ∈ P Γ sup ρ α , P ( f ) . (3)
For fixed α \alpha α and Γ \Gamma Γ , our statistical objective is an interval covering every compatible risk simultaneously for every f ∈ F f\in\mathcal{F} f ∈ F , with probability at least 1 − δ t o t 1-\delta_{\mathrm{tot}} 1 − δ tot over the observed sample. Here δ t o t ∈ ( 0 , 1 ) \delta_{\mathrm{tot}}\in(0,1) δ tot ∈ ( 0 , 1 ) is the total failure budget; § 4 accounts for sample fluctuation and auxiliary-estimation error within this budget. Thus Γ \Gamma Γ specifies the admissible hidden outcomes, while δ t o t \delta_{\mathrm{tot}} δ tot specifies the statistical guarantee.
We next derive these population risk endpoints from the observed law and show that every value between them is attainable.
Sharp Population CVaR Bounds
For binary outcomes, both possible prediction losses can be computed from f ( x ) f(x) f ( x ) , even when the actual label is hidden. Sensitivity bounds constrain the probability assigned to these two losses. This reduces population identification to two scalar convex minimizations.
Fix a predictor f f f and a cutoff η ∈ [ 0 , 1 ] \eta\in[0,1] η ∈ [ 0 , 1 ] . Write
a f , η y ( x ) = ( ℓ ( f ( x ) , y ) − η ) + , d f , η ( x ) = a f , η 1 ( x ) − a f , η 0 ( x ) . a^{y}_{f,\eta}(x)=\bigl(\ell(f(x),y)-\eta\bigr)_{+},\qquad d_{f,\eta}(x)=a^{1}_{f,\eta}(x)-a^{0}_{f,\eta}(x). a f , η y ( x ) = ( ℓ ( f ( x ) , y ) − η ) + , d f , η ( x ) = a f , η 1 ( x ) − a f , η 0 ( x ) .
Here a y a^{y} a y is the excess loss if the label is y y y , and d d d is its change when the label changes from zero to one. A hidden outcome probability q q q gives expected excess loss ( 1 − q ) a 0 + q a 1 = a 0 + q d (1-q)a^{0}+qa^{1}=a^{0}+qd ( 1 − q ) a 0 + q a 1 = a 0 + q d . With d + = ( d f , η ) + d_{+}=(d_{f,\eta})_{+} d + = ( d f , η ) + and d − = ( − d f , η ) + d_{-}=(-d_{f,\eta})_{+} d − = ( − d f , η ) + , its largest and smallest allowed values are
b f , η U = a f , η 0 + p ‾ Γ d + − p ‾ Γ d − , b^{U}_{f,\eta}=a^{0}_{f,\eta}+\overline{p}_{\Gamma}d_{+}-\underline{p}_{\Gamma}d_{-}, b f , η U = a f , η 0 + p Γ d + − p Γ d − ,
b f , η L = a f , η 0 + p ‾ Γ d + − p ‾ Γ d − . (4) b^{L}_{f,\eta}=a^{0}_{f,\eta}+\underline{p}_{\Gamma}d_{+}-\overline{p}_{\Gamma}d_{-}. \tag*{(4)} b f , η L = a f , η 0 + p Γ d + − p Γ d − . (4)
All quantities are evaluated at the same covariate value. The upper bound assigns the greatest allowed probability to the label with larger loss; the lower bound reverses the assignment. Averaging the historically observed and hidden contributions gives
m f , η σ = E o b s [ a f , η Y o b s ( X ) + ( 1 − S ) b f , η σ ( X ) ] , σ ∈ { L , U } . m^{\sigma}_{f,\eta}=\mathbb{E}_{\mathrm{obs}}\left[a^{Y_{\mathrm{obs}}}_{f,\eta}(X)+(1-S)b^{\sigma}_{f,\eta}(X)\right],\qquad\sigma\in\{L,U\}. m f , η σ = E obs [ a f , η Y obs ( X ) + ( 1 − S ) b f , η σ ( X ) ] , σ ∈ { L , U } .
where E o b s \mathbb{E}_{\mathrm{obs}} E obs integrates over the observed-data law. Adding the cutoff term forms the CVaR objectives
L f , η = η + β − 1 m f , η L , U f , η = η + β − 1 m f , η U . L_{f,\eta}=\eta+\beta^{-1}m^{L}_{f,\eta},\qquad U_{f,\eta}=\eta+\beta^{-1}m^{U}_{f,\eta}. L f , η = η + β − 1 m f , η L , U f , η = η + β − 1 m f , η U .
Dependence on Γ \Gamma Γ is implicit.
Theorem 1 (Sharp population CVaR interval). Under Assumption 1, for every f ∈ F f\in\mathcal{F} f ∈ F and finite Γ ≥ 1 , \Gamma\ge1, Γ ≥ 1 ,
ρ ‾ α , Γ ( f ) = min η ∈ [ 0 , 1 ] L f , η , ρ ‾ α , Γ ( f ) = min η ∈ [ 0 , 1 ] U f , η . \underline{\rho}_{\alpha,\Gamma}(f)=\min_{\eta\in[0,1]}L_{f,\eta},\qquad\overline{\rho}_{\alpha,\Gamma}(f)=\min_{\eta\in[0,1]}U_{f,\eta}. ρ α , Γ ( f ) = η ∈ [ 0 , 1 ] min L f , η , ρ α , Γ ( f ) = η ∈ [ 0 , 1 ] min U f , η . Both endpoints and every intermediate value are attained by compatible full-data laws. Consequently, the set of risks { ρ α , P ( f ) : P ∈ P Γ ( P o b s ) } \{\rho_{\alpha,P}(f):P\in\mathcal{P}_{\Gamma}(P_{\mathrm{obs}})\} { ρ α , P ( f ) : P ∈ P Γ ( P obs )} is exactly [ ρ ‾ α , Γ ( f ) , ρ ‾ α , Γ ( f ) ] [\underline{\rho}_{\alpha,\Gamma}(f),\overline{\rho}_{\alpha,\Gamma}(f)] [ ρ α , Γ ( f ) , ρ α , Γ ( f )] .
The key mechanism is that thresholding preserves the ordering of the two label-specific losses. For each endpoint, the same hidden-outcome distribution is therefore extremal at every cutoff, including the optimizing cutoff. Appendices § 12 .§ 3 –§ 12 .§ 4 give the construction and sharpness proof, including discrete losses and ties.
The identification width
I Γ ( f ) = ρ ‾ α , Γ ( f ) − ρ ‾ α , Γ ( f ) I_{\Gamma}(f)=\overline{\rho}_{\alpha,\Gamma}(f)-\underline{\rho}_{\alpha,\Gamma}(f) I Γ ( f ) = ρ α , Γ ( f ) − ρ α , Γ ( f )
is the unresolved range of population risk even when the observed distribution is known exactly. Sharpness means that narrowing this range requires additional information or stronger assumptions. Increasing Γ \Gamma Γ expands the compatible class and can only widen the interval; at Γ = 1 \Gamma=1 Γ = 1 , the population risk is identified under conditional ignorability.
For a chosen acceptable risk level r r r , ρ ‾ α , Γ ( f ) ≤ r \overline{\rho}_{\alpha,\Gamma}(f)\le r ρ α , Γ ( f ) ≤ r certifies that every compatible risk meets the limit. If ρ ‾ α , Γ ( f ) < ρ ‾ α , Γ ( g ) \overline{\rho}_{\alpha,\Gamma}(f)<\underline{\rho}_{\alpha,\Gamma}(g) ρ α , Γ ( f ) < ρ α , Γ ( g ) , the ordering holds under every compatible law. Precision and risk level are distinct: a narrow interval can still lie above the acceptable limit. § 4 adds sampling uncertainty to these decisions.
Estimating and Using Tail-Risk Certificates
The observed distribution must be estimated from a finite sample. We control endpoint error uniformly across the candidate class so that the same certificate event supports comparison and selection after examining the data.
Cross-fitted endpoint estimation
The auxiliary functions p = p 1 p = p_1 p = p 1 and π \pi π describe the outcome among historically observed cases and the probability of historical observation. They are fitted to estimate the risk of f f f . Substituting an estimated p p p into the sensitivity bounds introduces estimation bias. We correct its leading effect using observed outcome residuals.
Write b ˙ σ = ∂ b σ / ∂ p \dot{b}^{\sigma} = \partial b^{\sigma}/\partial p b ˙ σ = ∂ b σ / ∂ p , for σ ∈ { L , U } \sigma\in\{L,U\} σ ∈ { L , U } . The locally robust score combines observed excess loss, the sensitivity-bounded hidden contribution, and the residual correction:
ϕ f , η σ ( O ; p , π ) = S a f , η y o b s ( X ) + ( 1 − S ) b f , η σ ( X ; p ( X ) ) + S 1 − π ( X ) π ( X ) b ˙ f , η σ ( X ; p ( X ) ) { Y o b s − p ( X ) } . (5) \phi^{\sigma}_{f,\eta}(O;p,\pi) = Sa^{\mathrm{yobs}}_{f,\eta}(X) + (1-S)b^{\sigma}_{f,\eta}(X;p(X))
+ S\frac{1-\pi(X)}{\pi(X)}\dot{b}^{\sigma}_{f,\eta}(X;p(X))\{Y^{\mathrm{obs}}-p(X)\}. \tag*{(5)} ϕ f , η σ ( O ; p , π ) = S a f , η yobs ( X ) + ( 1 − S ) b f , η σ ( X ; p ( X )) + S π ( X ) 1 − π ( X ) b ˙ f , η σ ( X ; p ( X )) { Y obs − p ( X )} . (5)
At the true auxiliary functions, its expectation equals m f , η σ m^{\sigma}_{f,\eta} m f , η σ . The correction leaves quadratic and product error terms, quantified in § 4.2 and Appendix § 13 .§ 1 .
Split the sample into K K K balanced folds. Fit p , π p,\pi p , π outside each evaluation fold, then clip their predictions to [ 0 , 1 ] [0,1] [ 0 , 1 ] and [ π min / 2 , 1 ] [\pi_{\min}/2,1] [ π m i n /2 , 1 ] , respectively. Thus each score uses auxiliary fits trained without its evaluation observations. Let n min n_{\min} n m i n be the smallest fold size and k ( i ) k(i) k ( i ) the fold containing observation i i i . Define
ρ ^ σ ( f ) = inf η ∈ [ 0 , 1 ] { η + 1 n β ∑ i = 1 n ϕ f , η σ ( O i ; p ^ ( − k ( i ) ) , π ^ ( − k ( i ) ) ) } . (6) \widehat{\rho}^{\sigma}(f)=\inf_{\eta\in[0,1]}\left\{\eta+\frac{1}{n\beta}\sum_{i=1}^{n}\phi^{\sigma}_{f,\eta}\left(O_i;\widehat{p}^{(-k(i))},\widehat{\pi}^{(-k(i))}\right)\right\}. \tag*{(6)} ρ σ ( f ) = η ∈ [ 0 , 1 ] inf { η + n β 1 i = 1 ∑ n ϕ f , η σ ( O i ; p ( − k ( i )) , π ( − k ( i )) ) } . (6)
The endpoints share auxiliary fits and minimize their cutoffs separately. For each fixed f f f and realized auxiliary fits, exact cutoff minimization checks 0 0 0 , 1 1 1 , and ℓ ( f ( X i ) , y ) \ell(f(X_i),y) ℓ ( f ( X i ) , y ) for every i = 1 , … , n i=1,\ldots,n i = 1 , … , n and y ∈ { 0 , 1 } y\in\{0,1\} y ∈ { 0 , 1 } (Proposition B.5 ).
Uniform certificates and decision guarantees
To turn these estimates into certificates, we control the errors of both endpoints uniformly over the candidate class. The bound combines sampling fluctuation with auxiliary-estimation error.
Write ρ L = ρ α , Γ \rho^L=\rho_{\alpha,\Gamma} ρ L = ρ α , Γ and ρ U = ρ ‾ α , Γ \rho^U=\overline{\rho}_{\alpha,\Gamma} ρ U = ρ α , Γ .
Theorem 2 (Uniform statistical certification). Fix α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) , 1 ≤ Γ < ∞ 1\leq\Gamma<\infty 1 ≤ Γ < ∞ , δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , and δ ν ∈ [ 0 , 1 − δ ) \delta_{\nu}\in[0,1-\delta) δ ν ∈ [ 0 , 1 − δ ) . Assume the bounded-loss, overlap, and sampling conditions of § 2 and the cross-fitting and measurability conventions of Appendix § 13 . Suppose that, with probability at least 1 − δ ν 1-\delta_{\nu} 1 − δ ν , the clipped auxiliary fits satisfy jointly
max k ∥ p ^ ( − k ) − p ∥ L 2 ( P X ) ≤ r p , max k ∥ π ^ ( − k ) − π ∥ L 2 ( P X ) ≤ r π . \max_k\left\|\widehat{p}^{(-k)}-p\right\|_{L^2(P_X)}\leq r_p,\qquad
\max_k\left\|\widehat{\pi}^{(-k)}-\pi\right\|_{L^2(P_X)}\leq r_{\pi}. k max p ( − k ) − p L 2 ( P X ) ≤ r p , k max π ( − k ) − π L 2 ( P X ) ≤ r π . For the calibrated radius r n ( δ ) r_n(\delta) r n ( δ ) in (B.15), with probability at least 1 − δ − δ ν 1-\delta-\delta_{\nu} 1 − δ − δ ν ,
sup f ∈ F max σ ∈ { L , U } ∣ ρ ^ σ ( f ) − ρ σ ( f ) ∣ ≤ r n ( δ ) . (7) \sup_{f\in\mathcal{F}}\max_{\sigma\in\{L,U\}}\left|\widehat{\rho}^{\sigma}(f)-\rho^{\sigma}(f)\right|\leq r_n(\delta). \tag*{(7)} f ∈ F sup σ ∈ { L , U } max ∣ ρ σ ( f ) − ρ σ ( f ) ∣ ≤ r n ( δ ) . (7)
Allocate δ + δ ν ≤ δ t o t \delta+\delta_{\nu}\leq\delta_{\mathrm{tot}} δ + δ ν ≤ δ tot to obtain the confidence level specified in § 2 . Here δ \delta δ covers sample fluctuation and δ ν \delta_{\nu} δ ν covers failure of the joint auxiliary-error bounds. The L 2 ( P X ) L^2(P_X) L 2 ( P X ) bounds measure prediction error across the target covariate distribution, including regions with few observed outcomes. Computing the calibrated radius requires valid bounds on these errors and on score complexity (Appendix § 13 .§ 3 ).
The radius separates sample fluctuation from auxiliary-estimation bias. Smaller tail mass β \beta β amplifies both; weak observation overlap and larger sensitivity levels also enlarge the bound. The remaining bias contains a squared outcome-regression error r p 2 r_p^2 r p 2 and the product r p r π r_p r_{\pi} r p r π . At Γ = 1 \Gamma=1 Γ = 1 , the squared-error coefficient vanishes. For Γ > 1 \Gamma>1 Γ > 1 , accurate outcome regression remains important even when the observation model is known. Cross-fitting supplies the conditional independence used for concentration; the auxiliary-error bounds are established separately.
Outer certificates and decisions. Write r n = r n ( δ ) r_n = r_n(\delta) r n = r n ( δ ) . On the same event, simultaneously for all predictors,
[ ρ α , Γ L ( f ) , ρ α , Γ U ( f ) ] ⊆ C ^ n , Γ ( f ) : = [ max { 0 , ρ ^ L ( f ) − r n } , min { 1 , ρ ^ U ( f ) + r n } ] . (8) [\rho_{\alpha,\Gamma}^{L}(f), \rho_{\alpha,\Gamma}^{U}(f)] \subseteq\widehat{C}_{n,\Gamma}(f) := [\max\{0,\widehat{\rho}^{L}(f)-r_n\},\min\{1,\widehat{\rho}^{U}(f)+r_n\}]. \tag*{(8)} [ ρ α , Γ L ( f ) , ρ α , Γ U ( f )] ⊆ C n , Γ ( f ) := [ max { 0 , ρ L ( f ) − r n } , min { 1 , ρ U ( f ) + r n }] . (8)
The certificate’s width lies between I Γ ( f ) I_\Gamma(f) I Γ ( f ) and I Γ ( f ) + 4 r n I_\Gamma(f)+4r_n I Γ ( f ) + 4 r n . Its upper limit can certify a chosen risk limit. If ρ ^ U ( f ) + r n < ρ ^ L ( g ) − r n \widehat{\rho}^{U}(f)+r_n < \widehat{\rho}^{L}(g)-r_n ρ U ( f ) + r n < ρ L ( g ) − r n , then f f f has lower CVaR than g g g under every compatible law. Minimizing ρ ^ U \widehat{\rho}^{U} ρ U targets the smallest compatible upper risk in F \mathcal{F} F ; Appendix § 13 .§ 4 gives a 2 r n 2r_n 2 r n selection-error bound with an additional term for optimization tolerance.
These decisions retain their guarantee after predictor selection within F \mathcal{F} F . As sampling error shrinks, the identification width remains. Additional labels can reduce that remaining ambiguity.
Resolving Tail Ambiguity through Revelation
We next choose which hidden outcomes to reveal to reduce ambiguity about population risk. We begin with a fixed predictor and a common loss cutoff for the lower and upper CVaR objectives. Under the information model below, their gap decomposes across the population, making ambiguity reduction per unit cost the basis for a minimum-cost allocation.
Tail identification value
Fix f f f , Γ \Gamma Γ , and a cutoff η ∈ [ 0 , 1 ] \eta\in[0,1] η ∈ [ 0 , 1 ] . Throughout this section, expectations are over the target covariate distribution P X P_X P X . An audit reveals existing outcomes: its indicator A ≤ 1 − S A \le1-S A ≤ 1 − S and intensity λ ( x ) = P ( A = 1 ∣ X = x , S = 0 ) \lambda(x)=P(A=1 \mid X=x,S=0) λ ( x ) = P ( A = 1 ∣ X = x , S = 0 ) specify the share of originally hidden outcomes recovered at x x x .
We hold the historical observed law and the audited population contribution fixed; the remaining hidden outcomes retain the specified sensitivity bounds. Formally, let p A ( x ) = P ( Y = 1 ∣ X = x , A = 1 ) p_A(x)=P(Y=1 \mid X=x,A=1) p A ( x ) = P ( Y = 1 ∣ X = x , A = 1 ) and q ( x ) = P ( Y = 1 ∣ X = x , S = 0 , A = 0 ) q(x)=P(Y=1 \mid X=x,S=0,A=0) q ( x ) = P ( Y = 1 ∣ X = x , S = 0 , A = 0 ) . We impose the original interval [ p ‾ Γ ( x ) , p ‾ Γ ( x ) ] [\underline{p}_{\Gamma}(x),\overline{p}_{\Gamma}(x)] [ p Γ ( x ) , p Γ ( x )] separately on these audited and still-hidden outcome probabilities wherever their component has positive mass; q q q is otherwise free within it. This defines the retained-sensitivity information model (Appendix § 14 .§ 1 ). Identification treats the audited population contribution as known; finite audited samples require estimation of that contribution.
The retained lower and upper objectives, L f , η ( λ ) L_{f,\eta}(\lambda) L f , η ( λ ) and U f , η ( λ ) U_{f,\eta}(\lambda) U f , η ( λ ) , share the known contribution and differ only on the remaining hidden mass (Appendix § 14 .§ 2 ). Write s Γ = p ‾ Γ − p ‾ Γ s_\Gamma=\overline{p}_{\Gamma}-\underline{p}_{\Gamma} s Γ = p Γ − p Γ . Their gap is
τ f , η ( x ) : = 1 − π ( x ) β s Γ ( x ) ∣ d f , η ( x ) ∣ , \tau_{f,\eta}(x) := \frac{1-\pi(x)}{\beta}s_\Gamma(x)|d_{f,\eta}(x)|, τ f , η ( x ) := β 1 − π ( x ) s Γ ( x ) ∣ d f , η ( x ) ∣ ,
W f , η ( λ ) : = U f , η ( λ ) − L f , η ( λ ) = E [ ( 1 − λ ) τ f , η ] = W f , η ( 0 ) − E [ λ τ f , η ] . (9) W_{f,\eta}(\lambda) := U_{f,\eta}(\lambda)-L_{f,\eta}(\lambda)
= \mathbb{E}[(1-\lambda)\tau_{f,\eta}]
= W_{f,\eta}(0)-\mathbb{E}[\lambda\tau_{f,\eta}]. \tag*{(9)} W f , η ( λ ) := U f , η ( λ ) − L f , η ( λ ) = E [( 1 − λ ) τ f , η ] = W f , η ( 0 ) − E [ λ τ f , η ] . (9)
The tail identification value (TIV) density τ f , η \tau_{f,\eta} τ f , η combines three quantities: the population share with hidden outcomes, the allowed range of their outcome probability, and the change in excess loss between the two labels. It measures ambiguity per unit target-population mass. Revealing a fraction λ \lambda λ removes exactly E [ λ τ f , η ] \mathbb{E}[\lambda\tau_{f,\eta}] E [ λ τ f , η ] , a value calculable before seeing the audited labels. When both labels produce the same excess loss, this value is zero even if the outcome is uncertain.
Minimum-cost revelation and certificate precision
Let c ( x ) ≥ 0 c(x)\geq0 c ( x ) ≥ 0 be an integrable cost density per unit target-population mass, so allocation λ \lambda λ costs E [ c λ ] \mathbb{E}[c\lambda] E [ c λ ] . If one already-hidden label costs κ ( x ) \kappa(x) κ ( x ) , then c = ( 1 − π ) κ c=(1-\pi)\kappa c = ( 1 − π ) κ . For a target gap ε ≥ 0 \varepsilon\geq0 ε ≥ 0 , the minimum cost is
Cert f , η ( ε ) : = inf 0 ≤ λ ≤ 1 { E [ c λ ] : W f , η ( λ ) ≤ ε } . (10) \operatorname{Cert}_{f,\eta}(\varepsilon):=\inf_{0\leq\lambda\leq1}\{\mathbb{E}[c\lambda]:W_{f,\eta}(\lambda)\leq\varepsilon\}. \tag*{(10)} Cert f , η ( ε ) := 0 ≤ λ ≤ 1 inf { E [ c λ ] : W f , η ( λ ) ≤ ε } . (10)
Theorem 3 (Minimum-cost fixed-threshold revelation). Under the retained-sensitivity model, this infimum is attained. An optimum reveals zero-cost positive-TIV locations first and, on c > 0 c>0 c > 0 , satisfies
λ ⋆ ( x ) = { 1 , τ f , η ( x ) / c ( x ) > t , 0 , τ f , η ( x ) / c ( x ) < t , \lambda^\star(x)=
\begin{cases}
1, & \tau_{f,\eta}(x)/c(x)>t,\\
0, & \tau_{f,\eta}(x)/c(x)<t,
\end{cases} λ ⋆ ( x ) = { 1 , 0 , τ f , η ( x ) / c ( x ) > t , τ f , η ( x ) / c ( x ) < t , with fractional revelation on ties, for some t ∈ [ 0 , ∞ ] t\in[0,\infty] t ∈ [ 0 , ∞ ] .
Thus acquisition follows decreasing ambiguity reduction per unit cost until the target is reached, using a partial terminal tie group when needed. This is the fractional-knapsack rule [20 ] . Appendices § 14 .§ 3 and § 14 .§ 5 give the proof, boundary cases, and finite-cell computation. Among already-hidden candidates with positive label cost, its operational score is
τ f , η ( x ) c ( x ) = s Γ ( x ) ∣ d f , η ( x ) ∣ β κ ( x ) . \frac{\tau_{f,\eta}(x)}{c(x)}=\frac{s_{\Gamma}(x)|d_{f,\eta}(x)|}{\beta\kappa(x)}. c ( x ) τ f , η ( x ) = β κ ( x ) s Γ ( x ) ∣ d f , η ( x ) ∣ .
The hidden-population fraction cancels because the candidate is already known to be hidden.
We now connect the fixed-cutoff gap to the width of the optimized CVaR interval. The retained endpoints minimize their cutoffs separately, giving
I Γ , λ r e t ( f ) : = min η ∈ [ 0 , 1 ] U f , η ( λ ) − min η ∈ [ 0 , 1 ] L f , η ( λ ) ≤ W f , 0 ( λ ) . (11) I^{\mathrm{ret}}_{\Gamma,\lambda}(f):=\min_{\eta\in[0,1]}U_{f,\eta}(\lambda)-\min_{\eta\in[0,1]}L_{f,\eta}(\lambda)\le W_{f,0}(\lambda). \tag*{(11)} I Γ , λ ret ( f ) := η ∈ [ 0 , 1 ] min U f , η ( λ ) − η ∈ [ 0 , 1 ] min L f , η ( λ ) ≤ W f , 0 ( λ ) . (11)
Theorem 3 at η = 0 \eta=0 η = 0 therefore minimizes cost for the sufficient requirement W f , 0 ( λ ) ≤ ε W_{f,0}(\lambda)\le\varepsilon W f , 0 ( λ ) ≤ ε , which guarantees optimized width at most ε \varepsilon ε . Appendix § 14 .§ 4 proves this bound and formulates direct optimization of the optimized interval width.
For sequential acquisition, we use TIV as a ranking rule that adapts as labels accumulate. After each batch, the procedure retrains the predictor and refreshes TIV at the upper-endpoint minimizing cutoff of the retained plug-in objective. Its historical outcome fit remains fixed, preserving the original sensitivity bounds. Audited labels enter through their observed losses. Candidates are ranked by estimated s Γ ∣ d f , η ∣ / ( β κ ) s_{\Gamma}|d_{f,\eta}|/(\beta\kappa) s Γ ∣ d f , η ∣/ ( β κ ) , a batch is revealed, and the procedure repeats. Figure 4 evaluates the fixed and sequential protocols; Appendix § 15 .§ 6 specifies fitting and evaluation splits.
Empirical Evaluation
We evaluate three questions: how accurately endpoints can be estimated, how selective observation changes model assessment, and which additional labels reduce unresolved ambiguity. Simulated populations provide exact identification targets. Historical applications test the consequences of changing the evaluation population. Controlled masking of known outcomes permits direct checks of estimated risks and audit performance. Appendix § 15 provides the complete methods, results, and figure provenance.
Identification versus statistical uncertainty
We separate statistical uncertainty from population ambiguity in finite-cell populations with Brier loss ℓ ( f ( x ) , y ) = ( f ( x ) − y ) 2 \ell(f(x), y) = (f(x) - y)^2 ℓ ( f ( x ) , y ) = ( f ( x ) − y ) 2 , Γ = 2 \Gamma= 2 Γ = 2 , and α = .75 \alpha= .75 α = .75 . An oracle certificate uses the true outcome probability p 1 p_1 p 1 among historically observed cases and a finite-cell Hoeffding radius. On the same sampled datasets, two-fold cross-fitting learns the nuisance functions; root-mean-square error (RMSE) assesses the resulting endpoint estimates. Each sample size uses 60 simulated populations and 80 datasets per population.
As n n n increases from 10 3 10^3 1 0 3 to 10 5 10^5 1 0 5 , mean oracle outer width falls from .614 to .147, while mean exact population width remains .0949 (Figure 2 a). The 95% oracle certificate contains the entire sharp interval in every replication; its simultaneous protection across thresholds is conservative. Upper- and lower-endpoint RMSE decrease from .0361 and .0425 to .0027 and .0023 (panel b). Larger samples sharpen estimation of the interval, while the original observation process leaves its identification width intact. The learned-nuisance experiment measures endpoint accuracy; coverage for learned fits requires the calibration conditions of Theorem 2 .
Historical applicant-level sensitivity analysis
From the LendingClub data [21 ] , we construct a 2012 cohort of 390,644 applications, including 53,367 mature issued loans with known repayment outcomes (Appendix § 15 .§ 3 ). The applicant target is default under loan issuance; this outcome is unavailable for rejected applicants. The cohort includes 180,903 applications with recorded credit-score values below 660, the 2012 FICO minimum [22 ] , and no issued loans in that group. We therefore specify an issued-loan outcome regression extended to all recorded applicant covariates. Its predictions supply the reference probabilities in (5) , together with the observed-residual correction. The historical results are residual-adjusted sensitivity estimates under this regression extension. Appendix § 15 .§ 3 documents unsupported regions and differences in how source fields were measured.
We compare logistic regression, default-weighted logistic regression, and gradient boosting with five-fold cross-fitting. Default examples receive weight 2 in the weighted model. Standard classification objectives represent existing prediction pipelines whose population tail risks must be assessed. Brier loss is bounded and makes both possible label losses explicit. We assess tail levels α ∈ { .90 , .95 } \alpha\in\{.90, .95\} α ∈ { .90 , .95 } .
At α = .95 \alpha= .95 α = .95 , accepted-only CVaR favors the default-weighted model (.661). At Γ = 1 \Gamma= 1 Γ = 1 , its estimated applicant risk is .813, compared with .733 for boosting and .805 for logistic regression: two of three pairwise orderings reverse. Figure 3 a gives the intervals at Γ = 2 \Gamma= 2 Γ = 2 ; Appendix § 15 .§ 3 reports complete sensitivity curves and approximate breakdown values.
Validation with known repayment outcomes
To check estimated intervals and model comparisons against known risks, we impose 20 outcome-dependent masks on the mature issued loans, with sensitivity Γ 0 = 3 \Gamma_0 = 3 Γ 0 = 3 and approximately 45% of labels observed. This study targets the issued-loan reference population. Predictors are trained on observed labels using five-fold cross-fitting, and their full-outcome CVaRs provide reference risks.
At Γ 0 \Gamma_0 Γ 0 , plug-in intervals—the estimated endpoint intervals before statistical widening—contain reference CVaR in 11–14 of 20 masks per predictor and tail level. Enlargement using variability across five fold-specific objectives raises coverage to 19–20 (Figure 3 b). Coverage here targets the full-outcome reference CVaR of each fitted predictor. Fold enlargement provides an empirical calibration diagnostic. Plug-in coverage is zero at every tested Γ ≤ 2 \Gamma\le2 Γ ≤ 2 .
Three unordered predictor pairs in each of 20 masks give 60 comparisons per tail level. At Γ = 3 \Gamma= 3 Γ = 3 , α = .95 \alpha= .95 α = .95 , strict interval separation resolves 37 pairs correctly and leaves 23 unresolved. At Γ = 1 \Gamma= 1 Γ = 1 , the rule declares all 120 orderings across both tail levels and makes 16 errors. Allowing the masking sensitivity changes both the inferred ordering and whether separation supports a decision. Appendix § 15 .§ 4 gives the mechanism, enlargement rule, and complete results.
Fixed and sequential audits
We next evaluate how targeted revelation reduces the remaining ambiguity. In 120 synthetic finite-cell populations with heterogeneous costs, TIV per cost matches the exact fractional-knapsack frontier for fixed predictor and threshold (Appendix § 15 .§ 5 ). We then audit the masked issued-loan population using boosting, α = .95 \alpha= .95 α = .95 , Γ = 3 \Gamma= 3 Γ = 3 , and unit cost per hidden label.
The fixed protocol freezes the initial predictor, historical nuisance estimates, and upper-endpoint threshold. Newly audited outcomes become known contributions, while the original sensitivity bounds remain on unaudited outcomes. The sequential protocol retrains the predictor and reoptimizes thresholds after each batch; retained evaluation keeps the historical outcome fit fixed, and a reserved split measures predictive loss. The lowest-propensity baseline selects cases with the lowest estimated label availability; it uses historical availability in the fixed protocol and refitted current known-label availability in the sequential protocol.
At the recorded budget of approximately 16%, TIV removes a median 61% of the initial estimated fixed-threshold gap, compared with 16% for random acquisition (Figure 4 a). This concentration of removable ambiguity explains the gain from targeting. For Brier loss, predictions near .5 have similar losses under either label, so uncertainty sampling can favor cases with small excess-loss contrasts and little identification value.
With sequential retraining, median estimated retained width falls from .097 to .035 under TIV (panel b). This width describes the current predictor; it can increase when retraining changes the assessed losses, as the other trajectories illustrate (Table 5 ). Terminal median held-out CVaR is .8541 for TIV and .8682 for random, while mean Brier loss is .1362 and .1347, respectively (Table 6 ). Relative to random acquisition, these runs show improved tail performance and precision alongside a small mean-loss trade-off.
Table 5. Complete fixed and sequential audit comparison at the nominal 16% budget. Fixed reductions are medians of within-mask percentage reductions; estimated sequential widths are medians across the 20 runs. Integer rounding gives actual median budget fractions of 16.013% of the fixed initial hidden pool and 16.018% of the sequential initial hidden training pool.
Table 6. The constant forecast supplies an exact predictor reference. Precision and predictive loss after eight equal-budget audit rounds. Policy rows are medians across 20 masks; the paired row is the median of the within-mask TIV-minus-random differences. Estimated width uses the training pool; mean Brier loss and CVaR use the same reserved 20% split.
The constant forecast f = 1 / 2 f = 1/2 f = 1/2 has loss 1 / 4 1/4 1/4 under either label. Its mean Brier loss and every CVaR therefore equal 1 / 4 1/4 1/4 , while identification width and TIV are zero. Its tail loss is below that of the reported fitted models. Table 6 shows the competing mean-loss advantage of the audited boosting models, whose mean Brier losses are near .135. The reference separates average prediction quality, extreme-tail loss, and certainty about that loss. Appendix § 15 .§ 6 gives all policies and both protocols.
Limitations
The sensitivity range must be justified for the target population and covariate resolution. Statistical certificates additionally require overlap, controlled score complexity, and valid nuisance-error bounds. Small tail mass and scarce observed labels can make the bounds conservative. For Γ > 1 \Gamma> 1 Γ > 1 , outcome-regression error contributes quadratically even with a correct observation model. Choosing Γ \Gamma Γ from data or examining multiple sensitivity levels requires corresponding joint error control (Appendix § 16 .§ 1 ).
The historical applicant analysis depends on regression extrapolation into unsupported regions and on differently measured source fields. Controlled masking checks known issued-loan outcomes. Repayment under an untaken lending action requires a causal or experimental model beyond passive recovery of existing outcomes.
The revelation optimum fixes the predictor and threshold and assumes componentwise retained sensitivity bounds. Known outcome-independent audit randomization can identify more, and direct minimization of the reoptimized interval can favor a different allocation. Finite audits must estimate their newly known contributions. Learned acquisition scores, retraining, and data-dependent stopping require uncertainty control across the resulting audit history; the sequential trajectories provide empirical performance evidence. Extensions to richer outcomes, unbounded losses, and other application domains require additional modeling and validation.
Conclusion
Selective observation leaves a range of population tail risks compatible with the available outcomes. Sharp CVaR endpoints characterize that range, and calibrated uniform outer certificates support risk limits, model comparisons, and conservative selection within a candidate class. Under retained sensitivity, TIV quantifies the ambiguity removed by revelation for a fixed predictor and threshold. Ranking this value per cost gives the optimal population allocation for that objective. These outputs connect a specified risk decision to the observations and assumptions required to support it, and identify where further outcome collection can improve precision.
Acknowledgments and Disclosure of Funding
The author declares no competing interests.
References
Guide to the appendices
Appendix A gives compatible laws, examples, and identification proofs; Appendix B gives score bias, uniform estimation, calibration, and the learning lower bound; Appendix C gives retained-information revelation, its cost frontier, and computation. Appendix D records scientific protocols and complete experimental results; Appendix E treats sensitivity calibration, extensions, and related approaches. Appendix F collects additional mechanism, learning, and revelation diagnostics. Implementation provenance is available in the public repository described in Appendix D.7.
Throughout, β = 1 − α \beta= 1 - \alpha β = 1 − α , Γ < ∞ \Gamma< \infty Γ < ∞ , and the target population and outcome are fixed. I Γ ( f ) I_{\Gamma}(f) I Γ ( f ) is identification width; C ^ n , Γ ( f ) \widehat{C}_{n,\Gamma}(f) C n , Γ ( f ) is a statistical outer certificate. After revelation, W f , η ( λ ) W_{f,\eta}(\lambda) W f , η ( λ ) is the gap at a common threshold, whereas I Γ , λ r e t ( f ) I_{\Gamma,\lambda}^{\mathrm{ret}}(f) I Γ , λ ret ( f ) optimizes the two endpoints separately. Auxiliary results and equations use appendix-local numbering.
Population Identification and Sensitivity
Compatible full-data laws and sensitivity bounds
The target outcome and population are fixed across predictors and compatible laws. In a decision problem, Y = Y ( a ⋆ ) Y = Y(a^\star) Y = Y ( a ⋆ ) is the outcome under a specified action (e.g., default under loan issuance), and S S S records its availability [1 , 2 , 3 ] . Using an observed outcome requires consistency with that action and outcome definition. Comparing prediction rules holds a ⋆ a^\star a ⋆ and P X P_X P X fixed; changing the action, recovering an outcome under an untaken action, or transferring to a new population requires an additional causal or transport argument. The sensitivity model supplies neither exchangeability nor causal identification. Passive revelation recovers an already-realized target outcome, as in record linkage or controlled masking. Thus mature issued loans supply the masking target, whereas repayment under issuance remains a potential outcome for rejected applicants in the historical analysis.
Let X \mathcal{X} X be standard Borel, and let P o b s P_{\mathrm{obs}} P obs be the law of O = ( X , S , Y o b s ) O = (X,S,Y^{\mathrm{obs}}) O = ( X , S , Y obs ) with Y o b s = S Y Y^{\mathrm{obs}} = SY Y obs = S Y . Use measurable versions
π ( x ) = P o b s ( S = 1 ∣ X = x ) , p 1 ( x ) = P o b s ( Y o b s = 1 ∣ X = x , S = 1 ) , \pi(x) = P_{\mathrm{obs}}(S = 1 \mid X = x), \qquad p_1(x) = P_{\mathrm{obs}}(Y^{\mathrm{obs}} = 1 \mid X = x,S = 1), π ( x ) = P obs ( S = 1 ∣ X = x ) , p 1 ( x ) = P obs ( Y obs = 1 ∣ X = x , S = 1 ) ,
and losses z f y ( x ) = ℓ ( f ( x ) , y ) ∈ [ 0 , 1 ] z_f^y(x) = \ell(f(x),y) \in[0,1] z f y ( x ) = ℓ ( f ( x ) , y ) ∈ [ 0 , 1 ] . Assumption 1 gives π ≥ π min > 0 \pi\ge\pi_{\min} > 0 π ≥ π m i n > 0 , so π d P X \pi\,dP_X π d P X and P X P_X P X have the same null sets and p 1 p_1 p 1 is identified P X P_X P X -almost surely. All unqualified almost-sure statements use P X P_X P X . Write H = { π < 1 } H = \{\pi< 1\} H = { π < 1 } . For a measurable q : X → [ 0 , 1 ] q:\mathcal{X}\to[0,1] q : X → [ 0 , 1 ] and its Bernoulli kernel K q K_q K q , define
P q ( d x , { 1 } , d y ) = P X ( d x ) π ( x ) K p 1 ( x , d y ) , P q ( d x , { 0 } , d y ) = P X ( d x ) [ 1 − π ( x ) ] K q ( x , d y ) . (12) \begin{aligned}
P^q(dx,\{1\},dy) &= P_X(dx)\pi(x)K_{p_1}(x,dy),\\
P^q(dx,\{0\},dy) &= P_X(dx)[1-\pi(x)]K_q(x,dy). \tag*{(12)}
\end{aligned} P q ( d x , { 1 } , d y ) P q ( d x , { 0 } , d y ) = P X ( d x ) π ( x ) K p 1 ( x , d y ) , = P X ( d x ) [ 1 − π ( x )] K q ( x , d y ) . (12)
Set q = p 1 q=p_1 q = p 1 on H c H^c H c , where its value carries zero hidden mass.
Lemma A.1 (Kernel representation of compatible laws). Every full-data law inducing P o b s P_{\mathrm{obs}} P obs has the form (A.1), with q ( x ) = P ( Y = 1 ∣ X = x , S = 0 ) q(x)=P(Y=1\mid X=x,S=0) q ( x ) = P ( Y = 1 ∣ X = x , S = 0 ) on H H H . Conversely, every measurable q : X → [ 0 , 1 ] q:\mathcal{X}\to[0,1] q : X → [ 0 , 1 ] in (A.1) induces P o b s P_{\mathrm{obs}} P obs . With q = p 1 q=p_1 q = p 1 on H c H^c H c , these kernels identify the compatible laws up to P X P_X P X -null sets.
Proof. Standard Borel disintegration fixes the ( X , S ) (X,S) ( X , S ) law and selected-label kernel, leaving one binary probability on S = 0 S=0 S = 0 . Conversely, the two displayed measures sum to one and have the prescribed pushforward under ( x , s , y ) ↦ ( x , s , s y ) (x,s,y)\mapsto(x,s,sy) ( x , s , y ) ↦ ( x , s , sy ) . Equal full-data laws have equal hidden kernels ( 1 − π ) d P X (1-\pi)dP_X ( 1 − π ) d P X -almost surely, equivalently P X P_X P X -almost surely on H H H ; the canonical assignment settles H c H^c H c .
For finite Γ ≥ 1 \Gamma\ge1 Γ ≥ 1 , the odds restriction is equivalent, at interior p 1 p_1 p 1 , to the bounds p ‾ Γ = q − ( p 1 ; Γ ) \underline{p}_{\Gamma}=q_-(p_1;\Gamma) p Γ = q − ( p 1 ; Γ ) and p ‾ Γ = q + ( p 1 ; Γ ) \overline{p}_{\Gamma}=q_+(p_1;\Gamma) p Γ = q + ( p 1 ; Γ ) , where
q − ( p ; Γ ) = p p + Γ ( 1 − p ) , q + ( p ; Γ ) = Γ p 1 − p + Γ p . (13) q_-(p;\Gamma)=\frac{p}{p+\Gamma(1-p)}, \qquad q_+(p;\Gamma)=\frac{\Gamma p}{1-p+\Gamma p}. \tag*{(13)} q − ( p ; Γ ) = p + Γ ( 1 − p ) p , q + ( p ; Γ ) = 1 − p + Γ p Γ p . (13)
Their denominators are at least one. Thus p 1 = 0 p_1=0 p 1 = 0 forces q = 0 q=0 q = 0 and p 1 = 1 p_1=1 p 1 = 1 forces q = 1 q=1 q = 1 at every finite Γ \Gamma Γ ; these substantive boundary restrictions merit scrutiny when apparent certainty comes from a fitted model. For interior p 1 p_1 p 1 , the endpoints tend to zero and one as Γ → ∞ \Gamma\to\infty Γ → ∞ . Let Q Γ \mathcal{Q}_{\Gamma} Q Γ contain measurable kernels within these bounds on H H H , canonically assigned on H c H^c H c . Then
P Γ ( P o b s ) = { P q : q ∈ Q Γ } . (14) \mathcal{P}_{\Gamma}(P_{\mathrm{obs}})=\{P^q:q\in\mathcal{Q}_{\Gamma}\}. \tag*{(14)} P Γ ( P obs ) = { P q : q ∈ Q Γ } . (14)
It is nonempty since p ‾ Γ ≤ p 1 ≤ p ‾ Γ \underline{p}_{\Gamma}\le p_1\le\overline{p}_{\Gamma} p Γ ≤ p 1 ≤ p Γ ; at Γ = 1 \Gamma=1 Γ = 1 it is the single law satisfying Y ⊥ S ∣ X Y\perp S\mid X Y ⊥ S ∣ X .
For β = 1 − α ∈ ( 0 , 1 ) \beta=1-\alpha\in(0,1) β = 1 − α ∈ ( 0 , 1 ) , the CVaR representation [16 ] reduces to
ρ α , P ( f ) = inf η ∈ R { η + β − 1 E P ( L f − η ) + } = min η ∈ [ 0 , 1 ] { η + β − 1 E P ( L f − η ) + } . (15) \rho_{\alpha,P}(f)=\inf_{\eta\in\mathbb{R}}\left\{\eta+\beta^{-1}\mathbb{E}_{P}(L_f-\eta)_+\right\}=\min_{\eta\in[0,1]}\left\{\eta+\beta^{-1}\mathbb{E}_{P}(L_f-\eta)_+\right\}. \tag*{(15)} ρ α , P ( f ) = η ∈ R inf { η + β − 1 E P ( L f − η ) + } = η ∈ [ 0 , 1 ] min { η + β − 1 E P ( L f − η ) + } . (15)
For η < 0 \eta<0 η < 0 , the objective is β − 1 E P L f + ( 1 − β − 1 ) η \beta^{-1}\mathbb{E}_{P}L_f+(1-\beta^{-1})\eta β − 1 E P L f + ( 1 − β − 1 ) η and exceeds its value at zero; for η > 1 \eta>1 η > 1 , it equals η > 1 \eta>1 η > 1 . Continuity on [ 0 , 1 ] [0,1] [ 0 , 1 ] gives attainment, including distributions with atoms.
Observational equivalence and motivating constructions
Theorem A.2 (Observational equivalence with reversed CVaR rankings). For every α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) , there exist an observed law satisfying strict overlap, two predictors f , g f,g f , g with identical selected-loss distributions, and two full-data laws P ⋆ , Q ⋆ P^\star,Q^\star P ⋆ , Q ⋆ inducing that observed law such that
ρ α , P ⋆ ( f ) < ρ α , P ⋆ ( g ) , ρ α , Q ⋆ ( f ) > ρ α , Q ⋆ ( g ) . \rho_{\alpha,P^\star}(f)<\rho_{\alpha,P^\star}(g),\qquad\rho_{\alpha,Q^\star}(f)>\rho_{\alpha,Q^\star}(g). ρ α , P ⋆ ( f ) < ρ α , P ⋆ ( g ) , ρ α , Q ⋆ ( f ) > ρ α , Q ⋆ ( g ) . Proof. Take constant X X X , 0 < π < min { 1 , 2 β } 0<\pi<\min\{1,2\beta\} 0 < π < min { 1 , 2 β } , S ∼ Bernoulli ( π ) S\sim\operatorname{Bernoulli}(\pi) S ∼ Bernoulli ( π ) , and Y ∣ S = 1 ∼ Bernoulli ( 1 / 2 ) Y\mid S=1\sim\operatorname{Bernoulli}(1/2) Y ∣ S = 1 ∼ Bernoulli ( 1/2 ) . Under zero-one loss, the constant predictors f = 0 , g = 1 f=0,g=1 f = 0 , g = 1 have the same selected-loss law. Set Y = 0 Y=0 Y = 0 on S = 0 S=0 S = 0 under P ⋆ P^\star P ⋆ and Y = 1 Y=1 Y = 1 there under Q ⋆ Q^\star Q ⋆ . Both yield observed masses π / 2 , π / 2 , 1 − π \pi/2,\pi/2,1-\pi π /2 , π /2 , 1 − π on ( S , Y o b s ) = ( 1 , 1 ) , ( 1 , 0 ) , ( 0 , 0 ) (S,Y^{\mathrm{obs}})=(1,1),(1,0),(0,0) ( S , Y obs ) = ( 1 , 1 ) , ( 1 , 0 ) , ( 0 , 0 ) and satisfy overlap with π min = π \pi_{\min}=\pi π m i n = π .
For B ∼ Bernoulli ( r ) B\sim\operatorname{Bernoulli}(r) B ∼ Bernoulli ( r ) , the threshold objective is r / β + η ( 1 − r / β ) r/\beta+\eta(1-r/\beta) r / β + η ( 1 − r / β ) on [ 0 , 1 ] [0,1] [ 0 , 1 ] , hence
CVaR α ( B ) = min { r / β , 1 } . (16) \operatorname{CVaR}_{\alpha}(B)=\min\{r/\beta,1\}. \tag*{(16)} CVaR α ( B ) = min { r / β , 1 } . (16) Under P ⋆ P^\star P ⋆ , the loss probabilities are π / 2 \pi/2 π /2 and 1 − π / 2 1-\pi/2 1 − π /2 , so ρ α , P ⋆ ( f ) = π / ( 2 β ) < min { ( 1 − π / 2 ) / β , 1 } = ρ α , P ⋆ ( g ) \rho_{\alpha,P^\star}(f)=\pi/(2\beta)<\min\{(1-\pi/2)/\beta,1\}=\rho_{\alpha,P^\star}(g) ρ α , P ⋆ ( f ) = π / ( 2 β ) < min {( 1 − π /2 ) / β , 1 } = ρ α , P ⋆ ( g ) . Under Q ⋆ Q^\star Q ⋆ they exchange.
This construction concerns the unrestricted compatible class: its hidden probabilities zero and one lie outside the finite-Γ \Gamma Γ interval [ 1 / ( 1 + Γ ) , Γ / ( 1 + Γ ) ] [1/(1+\Gamma),\Gamma/(1+\Gamma)] [ 1/ ( 1 + Γ ) , Γ/ ( 1 + Γ )] when p 1 = 1 / 2 p_1=1/2 p 1 = 1/2 .
Figure 1 . For squared loss, f = 1 / 5 f=1/5 f = 1/5 , π = 1 / 2 \pi=1/2 π = 1/2 , p 1 = 1 / 3 p_1=1/3 p 1 = 1/3 , Γ = 2 \Gamma=2 Γ = 2 , and α = 1 / 2 \alpha=1/2 α = 1/2 , the loss is 1 / 25 + ( 3 / 5 ) B 1/25+(3/5)B 1/25 + ( 3/5 ) B with B ∼ Bernoulli ( 1 / 6 + q / 2 ) B\sim\operatorname{Bernoulli}(1/6+q/2) B ∼ Bernoulli ( 1/6 + q /2 ) and q ∈ [ 1 / 5 , 1 / 2 ] q\in[1/5,1/2] q ∈ [ 1/5 , 1/2 ] . Translation and positive scaling of (A.4), followed by (A.5), give
ρ 1 / 2 , P q ( f ) = 1 25 + 6 5 ( 1 6 + q 2 ) , (17) \rho_{1/2,P^q}(f)=\frac{1}{25}+\frac{6}{5}\left(\frac{1}{6}+\frac{q}{2}\right), \tag*{(17)} ρ 1/2 , P q ( f ) = 25 1 + 5 6 ( 6 1 + 2 q ) , (17)
ρ 1 / 2 , P 1 / 5 ( f ) = 9 25 = 0.36 , ρ 1 / 2 , P 1 / 2 ( f ) = 27 50 = 0.54. \rho_{1/2,P^{1/5}}(f)=\frac{9}{25}=0.36,\qquad\rho_{1/2,P^{1/2}}(f)=\frac{27}{50}=0.54. ρ 1/2 , P 1/5 ( f ) = 25 9 = 0.36 , ρ 1/2 , P 1/2 ( f ) = 50 27 = 0.54.
These endpoints describe the risk of one fixed predictor under the same observed law.
Conditional excess-loss envelopes and common extremal laws
Write a f , η y = ( z f y − η ) + a^y_{f,\eta}=(z^y_f-\eta)_+ a f , η y = ( z f y − η ) + , d f , η = a f , η y − a f , η 0 d_{f,\eta}=a^y_{f,\eta}-a^0_{f,\eta} d f , η = a f , η y − a f , η 0 , d + = ( d f , η ) + d_+=(d_{f,\eta})_+ d + = ( d f , η ) + , and d − = ( − d f , η ) + d_-=(-d_{f,\eta})_+ d − = ( − d f , η ) + . The hidden conditional excess loss is a 0 + q d a^0+qd a 0 + q d , with envelopes
b f , η U = a f , η 0 + p ‾ Γ d + − p ‾ Γ d − , b^U_{f,\eta}=a^0_{f,\eta}+\overline{p}_{\Gamma}d_+-\underline{p}_{\Gamma}d_-, b f , η U = a f , η 0 + p Γ d + − p Γ d − ,
b f , η L = a f , η 0 + p ‾ Γ d + − p ‾ Γ d − . (18) b^L_{f,\eta}=a^0_{f,\eta}+\underline{p}_{\Gamma}d_+-\overline{p}_{\Gamma}d_-. \tag*{(18)} b f , η L = a f , η 0 + p Γ d + − p Γ d − . (18)
The observable objectives are
m f , η σ = E o b s [ S a f , η Y o b s ( X ) + ( 1 − S ) b f , η σ ( X ) ] , σ ∈ { L , U } , m^\sigma_{f,\eta}=\mathbb{E}_{\mathrm{obs}}\left[Sa^{Y^{\mathrm{obs}}}_{f,\eta}(X)+(1-S)b^\sigma_{f,\eta}(X)\right],\qquad\sigma\in\{L,U\}, m f , η σ = E obs [ S a f , η Y obs ( X ) + ( 1 − S ) b f , η σ ( X ) ] , σ ∈ { L , U } ,
L f , η = η + β − 1 m f , η L , U f , η = η + β − 1 m f , η U . (19) L_{f,\eta}=\eta+\beta^{-1}m^L_{f,\eta},\qquad U_{f,\eta}=\eta+\beta^{-1}m^U_{f,\eta}. \tag*{(19)} L f , η = η + β − 1 m f , η L , U f , η = η + β − 1 m f , η U . (19)
Lemma A.3 (Common extremal kernels and threshold regularity). On H H H , define
q f U ( x ) = p ˉ Γ ( x ) 1 { z f 1 ( x ) ≥ z f 0 ( x ) } + p Γ ( x ) 1 { z f 1 ( x ) < z f 0 ( x ) } , q_f^U(x)=\bar{p}_\Gamma(x)1\{z_f^1(x)\ge z_f^0(x)\}+p_\Gamma(x)1\{z_f^1(x)<z_f^0(x)\}, q f U ( x ) = p ˉ Γ ( x ) 1 { z f 1 ( x ) ≥ z f 0 ( x )} + p Γ ( x ) 1 { z f 1 ( x ) < z f 0 ( x )} , q f L ( x ) = p Γ ( x ) 1 { z f 1 ( x ) ≥ z f 0 ( x ) } + p ˉ Γ ( x ) 1 { z f 1 ( x ) < z f 0 ( x ) } , (20) q_f^L(x)=p_\Gamma(x)1\{z_f^1(x)\ge z_f^0(x)\}+\bar{p}_\Gamma(x)1\{z_f^1(x)<z_f^0(x)\}, \tag*{(20)} q f L ( x ) = p Γ ( x ) 1 { z f 1 ( x ) ≥ z f 0 ( x )} + p ˉ Γ ( x ) 1 { z f 1 ( x ) < z f 0 ( x )} , (20) and set both kernels to p 1 p_1 p 1 on H c H^c H c . These kernels belong to Q Γ \mathcal{Q}_\Gamma Q Γ . For every q ∈ Q Γ q\in\mathcal{Q}_\Gamma q ∈ Q Γ and every η ∈ [ 0 , 1 ] \eta\in[0,1] η ∈ [ 0 , 1 ] ,
m f , η L ≤ E P q [ ( L f − η ) + ] ≤ m f , η U , E P q f g [ ( L f − η ) + ] = m f , η σ . (21) m_{f,\eta}^L\le\mathbb{E}_{P_q}[(L_f-\eta)_+]\le m_{f,\eta}^U,\qquad\mathbb{E}_{P_{q_f^g}}[(L_f-\eta)_+]=m_{f,\eta}^\sigma. \tag*{(21)} m f , η L ≤ E P q [( L f − η ) + ] ≤ m f , η U , E P q f g [( L f − η ) + ] = m f , η σ . (21) The objectives L f , η , U f , η L_{f,\eta},U_{f,\eta} L f , η , U f , η are convex and ( 1 + β − 1 ) (1+\beta^{-1}) ( 1 + β − 1 ) -Lipschitz in η \eta η , and attain their minima on [ 0 , 1 ] [0,1] [ 0 , 1 ] .
Proof. The kernels are measurable and satisfy the bounds. Since z ↦ ( z − η ) + z\mapsto(z-\eta)_+ z ↦ ( z − η ) + is nondecreasing, thresholding preserves the ordering of z f 1 , z f 0 z_f^1,z_f^0 z f 1 , z f 0 ; it can create a tie but cannot reverse the sign. Thus the same kernels extremize the affine expression a 0 + q d a^0+qd a 0 + q d at every threshold, including ties. Disintegration gives
E P q ( L f − η ) + = E o b s [ S a f , η Y o b s ( X ) ] + E P X [ ( 1 − π ) ( a f , η 0 + q d f , η ) ] , (22) \mathbb{E}_{P_q}(L_f-\eta)_+=\mathbb{E}_{\mathrm{obs}}[S a_{f,\eta}^{Y^{\mathrm{obs}}}(X)]
+\mathbb{E}_{P_X}[(1-\pi)(a_{f,\eta}^0+qd_{f,\eta})], \tag*{(22)} E P q ( L f − η ) + = E obs [ S a f , η Y obs ( X )] + E P X [( 1 − π ) ( a f , η 0 + q d f , η )] , (22)
proving (A.10), with zero contribution from H c H^c H c . Each endpoint objective is therefore the CVaR objective of one fixed law. Convexity and the Lipschitz bound follow from the convex, 1-Lipschitz hinge; compactness gives attainment. □
Sharpness, endpoint attainment, and interval filling
Proof of Theorem 1. Put G f ( q , η ) = η + β − 1 E P q ( L f − η ) + G_f(q,\eta)=\eta+\beta^{-1}\mathbb{E}_{P_q}(L_f-\eta)_+ G f ( q , η ) = η + β − 1 E P q ( L f − η ) + . Lemma A.3 gives L f , η = G f ( q f L , η ) ≤ G f ( q , η ) ≤ G f ( q f U , η ) = U f , η L_{f,\eta}=G_f(q_f^L,\eta)\le G_f(q,\eta)\le G_f(q_f^U,\eta)=U_{f,\eta} L f , η = G f ( q f L , η ) ≤ G f ( q , η ) ≤ G f ( q f U , η ) = U f , η simultaneously for all thresholds. Minimizing these inequalities and choosing the respective common kernels proves
ρ ‾ α , Γ ( f ) = min η ∈ [ 0 , 1 ] L f , η , ρ ‾ α , Γ ( f ) = min η ∈ [ 0 , 1 ] U f , η . (23) \underline{\rho}_{\alpha,\Gamma}(f)=\min_{\eta\in[0,1]}L_{f,\eta},\qquad
\overline{\rho}_{\alpha,\Gamma}(f)=\min_{\eta\in[0,1]}U_{f,\eta}. \tag*{(23)} ρ α , Γ ( f ) = η ∈ [ 0 , 1 ] min L f , η , ρ α , Γ ( f ) = η ∈ [ 0 , 1 ] min U f , η . (23)
Both threshold minima and both optimized endpoints are attained. In particular, the upper equality uses one kernel attaining the supremum at every threshold, without an abstract minimax interchange.
For functions with finite infima on a common domain,
∣ inf η F ( η ) − inf η G ( η ) ∣ ≤ sup η ∣ F ( η ) − G ( η ) ∣ . (24) \left|\inf_\eta F(\eta)-\inf_\eta G(\eta)\right|\le\sup_\eta|F(\eta)-G(\eta)|. \tag*{(24)} η inf F ( η ) − η inf G ( η ) ≤ η sup ∣ F ( η ) − G ( η ) ∣. (24)
because each function is bounded below by the other minus their uniform distance. Equation (A.11) and ∣ d f , η ∣ ≤ 1 |d_{f,\eta}|\le1 ∣ d f , η ∣ ≤ 1 therefore imply
∣ ρ α , P q ( f ) − ρ α , P q ~ ( f ) ∣ ≤ sup η ∣ G f ( q , η ) − G f ( q ~ , η ) ∣ ≤ β − 1 E [ ( 1 − π ) ∣ q − q ~ ∣ ] . |\rho_{\alpha,P_q}(f)-\rho_{\alpha,P_{\widetilde q}}(f)|
\le\sup_\eta|G_f(q,\eta)-G_f(\widetilde q,\eta)|
\le\beta^{-1}\mathbb{E}[(1-\pi)|q-\widetilde q|]. ∣ ρ α , P q ( f ) − ρ α , P q ( f ) ∣ ≤ η sup ∣ G f ( q , η ) − G f ( q , η ) ∣ ≤ β − 1 E [( 1 − π ) ∣ q − q ∣ ] .
The mixture q t = ( 1 − t ) q f L + t q f U q_t=(1-t)q_f^L+tq_f^U q t = ( 1 − t ) q f L + t q f U , t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ] , remains measurable and compatible, and satisfies
∣ ρ α , P q t ( f ) − ρ α , P q s ( f ) ∣ ≤ ∣ t − s ∣ β E [ ( 1 − π ) ∣ q f U − q f L ∣ ] ≤ ∣ t − s ∣ β . (25) |\rho_{\alpha,P_{q_t}}(f)-\rho_{\alpha,P_{q_s}}(f)|
\le\frac{|t-s|}{\beta}\mathbb{E}[(1-\pi)|q_f^U-q_f^L|]
\le\frac{|t-s|}{\beta}. \tag*{(25)} ∣ ρ α , P q t ( f ) − ρ α , P q s ( f ) ∣ ≤ β ∣ t − s ∣ E [( 1 − π ) ∣ q f U − q f L ∣ ] ≤ β ∣ t − s ∣ . (25)
This continuous path joins the two attained endpoints. The intermediate value theorem, including the singleton case, yields
{ ρ α , P ( f ) : P ∈ P Γ ( P o b s ) } = [ ρ ‾ α , Γ ( f ) , ρ ‾ α , Γ ( f ) ] . (26) \{\rho_{\alpha,P}(f):P\in\mathcal{P}_\Gamma(P_{\mathrm{obs}})\}
=[\underline{\rho}_{\alpha,\Gamma}(f),\overline{\rho}_{\alpha,\Gamma}(f)]. \tag*{(26)} { ρ α , P ( f ) : P ∈ P Γ ( P obs )} = [ ρ α , Γ ( f ) , ρ α , Γ ( f )] . (26)
□
Sensitivity analysis and population decision criteria
Proposition A.4 (Nested classes and continuous endpoint curves). If 1 ≤ Γ 1 ≤ Γ 2 < ∞ 1 \le\Gamma_1 \le\Gamma_2 < \infty 1 ≤ Γ 1 ≤ Γ 2 < ∞ , then
P Γ 1 ( P o b s ) ⊆ P Γ 2 ( P o b s ) . \mathcal{P}_{\Gamma_1}(\mathcal{P}_{\mathrm{obs}}) \subseteq\mathcal{P}_{\Gamma_2}(\mathcal{P}_{\mathrm{obs}}). P Γ 1 ( P obs ) ⊆ P Γ 2 ( P obs ) . For each f f f , the lower endpoint is nonincreasing in Γ \Gamma Γ , the upper endpoint is nondecreasing, and the identification width I Γ ( f ) = ρ ‾ α , Γ ( f ) − ρ ‾ α , Γ ( f ) I_\Gamma(f) = \overline{\rho}_{\alpha,\Gamma}(f) - \underline{\rho}_{\alpha,\Gamma}(f) I Γ ( f ) = ρ α , Γ ( f ) − ρ α , Γ ( f ) is nondecreasing. The interval is a singleton at Γ = 1 \Gamma= 1 Γ = 1 . Moreover, for σ ∈ { L , U } \sigma\in\{L, U\} σ ∈ { L , U } , writing ρ α , Γ σ = ρ ‾ α , Γ \rho^\sigma_{\alpha,\Gamma} = \underline{\rho}_{\alpha,\Gamma} ρ α , Γ σ = ρ α , Γ and ρ α , Γ U = ρ ‾ α , Γ \rho^U_{\alpha,\Gamma} = \overline{\rho}_{\alpha,\Gamma} ρ α , Γ U = ρ α , Γ ,
∣ ρ α , Γ 2 σ ( f ) − ρ α , Γ 1 σ ( f ) ∣ ≤ E P X ( 1 − π ) 4 β ∣ Γ 2 − Γ 1 ∣ . (27) \left|\rho^\sigma_{\alpha,\Gamma_2}(f) - \rho^\sigma_{\alpha,\Gamma_1}(f)\right| \le\frac{E_{P_X}(1-\pi)}{4\beta}\left|\Gamma_2-\Gamma_1\right|. \tag*{(27)} ρ α , Γ 2 σ ( f ) − ρ α , Γ 1 σ ( f ) ≤ 4 β E P X ( 1 − π ) ∣ Γ 2 − Γ 1 ∣ . (27) In particular, both endpoint curves are continuous at every finite sensitivity level.
Proof. For p ∈ [ 0 , 1 ] p \in[0,1] p ∈ [ 0 , 1 ] , ∂ Γ q − = − p ( 1 − p ) / { p + Γ ( 1 − p ) } 2 ∈ [ − 1 / 4 , 0 ] \partial_\Gamma q_- = -p(1-p)/\{p+\Gamma(1-p)\}^2 \in[-1/4,0] ∂ Γ q − = − p ( 1 − p ) / { p + Γ ( 1 − p ) } 2 ∈ [ − 1/4 , 0 ] and ∂ Γ q + = p ( 1 − p ) / { 1 − p + Γ p } 2 ∈ [ 0 , 1 / 4 ] \partial_\Gamma q_+ = p(1-p)/\{1-p+\Gamma p\}^2 \in[0,1/4] ∂ Γ q + = p ( 1 − p ) / { 1 − p + Γ p } 2 ∈ [ 0 , 1/4 ] . Their signs prove nesting and endpoint monotonicity; at one both maps equal p p p . Their magnitude bounds and the Γ \Gamma Γ -independent loss ordering in (A.9) bound the difference of the extremal kernels by ∣ Γ 2 − Γ 1 ∣ / 4 |\Gamma_2-\Gamma_1|/4 ∣ Γ 2 − Γ 1 ∣/4 . Apply (A.11) and (A.13) to obtain (A.16). □
Corollary A.5 (Population risk budgets, comparisons, and selection). For any risk budget r r r ,
ρ α , P ( f ) ≤ r for every P ∈ P Γ ( P o b s ) ⟺ ρ ‾ α , Γ ( f ) ≤ r . (28) \rho_{\alpha,P}(f) \le r \quad\text{for every } P \in\mathcal{P}_{\Gamma}(\mathcal{P}_{\mathrm{obs}}) \quad\Longleftrightarrow\quad\overline{\rho}_{\alpha,\Gamma}(f) \le r. \tag*{(28)} ρ α , P ( f ) ≤ r for every P ∈ P Γ ( P obs ) ⟺ ρ α , Γ ( f ) ≤ r . (28) For two predictors, the sufficient separation condition
ρ ‾ α , Γ ( f ) < ρ ‾ α , Γ ( g ) (29) \overline{\rho}_{\alpha,\Gamma}(f) < \underline{\rho}_{\alpha,\Gamma}(g) \tag*{(29)} ρ α , Γ ( f ) < ρ α , Γ ( g ) (29) implies ρ α , P ( f ) < ρ α , P ( g ) \rho_{\alpha,P}(f) < \rho_{\alpha,P}(g) ρ α , P ( f ) < ρ α , P ( g ) under every compatible law. If ρ ‾ α , Γ ( f ε ) ≤ inf g ∈ F ρ ‾ α , Γ ( g ) + ε \overline{\rho}_{\alpha,\Gamma}(f_\varepsilon) \le\inf_{g\in\mathcal{F}}\overline{\rho}_{\alpha,\Gamma}(g) + \varepsilon ρ α , Γ ( f ε ) ≤ inf g ∈ F ρ α , Γ ( g ) + ε , then every such law satisfies ρ α , P ( f ε ) ≤ inf g ∈ F ρ ‾ α , Γ ( g ) + ε \rho_{\alpha,P}(f_\varepsilon) \le\inf_{g\in\mathcal{F}}\overline{\rho}_{\alpha,\Gamma}(g) + \varepsilon ρ α , P ( f ε ) ≤ inf g ∈ F ρ α , Γ ( g ) + ε . A nonempty finite class permits ε = 0 \varepsilon= 0 ε = 0 .
Proof. Use the attained upper endpoint for (A.17), the chain ρ P ( f ) ≤ ρ ‾ ( f ) < ρ ‾ ( g ) ≤ ρ P ( g ) \rho_P(f) \le\overline{\rho}(f) < \underline{\rho}(g) \le\rho_P(g) ρ P ( f ) ≤ ρ ( f ) < ρ ( g ) ≤ ρ P ( g ) for comparison, and ρ P ( f ε ) ≤ ρ ‾ ( f ε ) \rho_P(f_\varepsilon) \le\overline{\rho}(f_\varepsilon) ρ P ( f ε ) ≤ ρ ( f ε ) for selection. □
If ρ ‾ ( f ) > r \underline{\rho}(f) > r ρ ( f ) > r , every law violates the budget; if ρ ‾ ( f ) ≤ r < ρ ‾ ( f ) \underline{\rho}(f) \le r < \overline{\rho}(f) ρ ( f ) ≤ r < ρ ( f ) , the two extremal laws respectively satisfy and violate it. Thus sharpness covers all budget and equality cases.
Common-law comparisons. Define D f , g ( Γ ) = ρ ‾ α , Γ ( f ) − ρ ‾ α , Γ ( g ) D_{f,g}(\Gamma) = \overline{\rho}_{\alpha,\Gamma}(f) - \underline{\rho}_{\alpha,\Gamma}(g) D f , g ( Γ ) = ρ α , Γ ( f ) − ρ α , Γ ( g ) . Marginal extrema can use different hidden kernels, whereas the direct contrast is
Δ Γ ( f , g ) = sup P ∈ P Γ ( P o b s ) { ρ α , P ( f ) − ρ α , P ( g ) } ≤ D f , g ( Γ ) . (30) \Delta_\Gamma(f,g) = \sup_{P\in\mathcal{P}_{\Gamma}(\mathcal{P}_{\mathrm{obs}})}\{\rho_{\alpha,P}(f)-\rho_{\alpha,P}(g)\} \le D_{f,g}(\Gamma). \tag*{(30)} Δ Γ ( f , g ) = P ∈ P Γ ( P obs ) sup { ρ α , P ( f ) − ρ α , P ( g )} ≤ D f , g ( Γ ) . (30)
Consequently, D f , g < 0 D_{f,g} < 0 D f , g < 0 proves common-law dominance, but overlapping marginal intervals leave its sign unresolved by this criterion. Determining Δ Γ \Delta_\Gamma Δ Γ requires joint optimization under one hidden kernel. The uniform event of Appendix B.4 makes the marginal comparison and selection rules valid simultaneously over the stipulated predictor class using the same observations.
First breakdown of an interval-certified comparison. Set Γ b r e a k ( f , g ) = inf { Γ ≥ 1 : D f , g ( Γ ) ≥ 0 } \Gamma_{\mathrm{break}}(f,g) = \inf\{\Gamma\ge1 : D_{f,g}(\Gamma) \ge0\} Γ break ( f , g ) = inf { Γ ≥ 1 : D f , g ( Γ ) ≥ 0 } , with inf ∅ = ∞ \inf\varnothing= \infty inf ∅ = ∞ . The gap is continuous and nondecreasing. If D f , g ( 1 ) ≥ 0 D_{f,g}(1) \ge0 D f , g ( 1 ) ≥ 0 , breakdown is one and no strict interval certificate holds. If D f , g ( 1 ) < 0 D_{f,g}(1) < 0 D f , g ( 1 ) < 0 , breakdown exceeds one; when finite, continuity forces D f , g ( Γ b r e a k ) = 0 D_{f,g}(\Gamma_{\mathrm{break}}) = 0 D f , g ( Γ break ) = 0 and strict separation holds exactly on [ 1 , Γ b r e a k ) [1,\Gamma_{\mathrm{break}}) [ 1 , Γ break ) . A nonzero gap at that boundary would persist across it, contradicting first failure. Infinite breakdown means separation at every finite level. Differences between two nondecreasing upper endpoints need not be monotone, so the upper-endpoint minimizer may change independently of these pairwise breakdown values.
Statistical Certificates and Learning Complexity
This appendix proves the statistical results for a fixed tail level α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) and a fixed finite sensitivity level Γ ≥ 1 \Gamma\ge1 Γ ≥ 1 . Write β = 1 − α \beta= 1-\alpha β = 1 − α , p = p 1 p=p_1 p = p 1 , and P = P o b s P=P_{\mathrm{obs}} P = P obs ; thus P g P g P g denotes integration of an observed-data function g g g against the common observed law. For brevity, set
ρ L ( f ) = ρ ‾ α , Γ ( f ) , ρ U ( f ) = ρ ‾ α , Γ ( f ) , I Γ ( f ) = ρ U ( f ) − ρ L ( f ) . \rho^{L}(f)=\underline{\rho}_{\alpha,\Gamma}(f),\qquad\rho^{U}(f)=\overline{\rho}_{\alpha,\Gamma}(f),\qquad I_{\Gamma}(f)=\rho^{U}(f)-\rho^{L}(f). ρ L ( f ) = ρ α , Γ ( f ) , ρ U ( f ) = ρ α , Γ ( f ) , I Γ ( f ) = ρ U ( f ) − ρ L ( f ) .
The population characterization in Appendix Appendix § 12 gives ρ σ ( f ) = inf η ∈ [ 0 , 1 ] { η + β − 1 m f , η σ } \rho^{\sigma}(f)=\inf_{\eta\in[0,1]}\{\eta+\beta^{-1}m^{\sigma}_{f,\eta}\} ρ σ ( f ) = inf η ∈ [ 0 , 1 ] { η + β − 1 m f , η σ } for σ ∈ { L , U } \sigma\in\{L,U\} σ ∈ { L , U } . All L 2 L_{2} L 2 norms below are with respect to P X P_{X} P X .
For uniform empirical-process statements, assume the paired maps x ↦ ( ℓ ( f ( x ) , 0 ) , ℓ ( f ( x ) , 1 ) ) x\mapsto(\ell(f(x),0),\ell(f(x),1)) x ↦ ( ℓ ( f ( x ) , 0 ) , ℓ ( f ( x ) , 1 )) , f ∈ F f\in\mathcal{F} f ∈ F , form a pointwise-separable class of measurable functions with the envelope supplied by Assumption 1. A countable pointwise-dense subclass and rational thresholds including 0, 1 determine the suprema below, since the score transformations are continuous. Measurable, pointwise-separable versions of the fitted score classes suffice as well.
Score regularity and exact nuisance-bias expansion
Recall a f , η y ( x ) = ( ℓ ( f ( x ) , y ) − η ) + a^{y}_{f,\eta}(x)=(\ell(f(x),y)-\eta)_{+} a f , η y ( x ) = ( ℓ ( f ( x ) , y ) − η ) + and d f , η ( x ) = a f , η 1 ( x ) − a f , η 0 ( x ) d_{f,\eta}(x)=a^{1}_{f,\eta}(x)-a^{0}_{f,\eta}(x) d f , η ( x ) = a f , η 1 ( x ) − a f , η 0 ( x ) . For a real number d d d , write d + = max ( d , 0 ) d_{+}=\max(d,0) d + = max ( d , 0 ) and d − = max ( − d , 0 ) d_{-}=\max(-d,0) d − = max ( − d , 0 ) .
H p U ( d ) = q + ( p ) d + − q − ( p ) d − , H p L ( d ) = q − ( p ) d + − q + ( p ) d − , (31) H^{U}_{p}(d)=q_{+}(p)d_{+}-q_{-}(p)d_{-},\qquad H^{L}_{p}(d)=q_{-}(p)d_{+}-q_{+}(p)d_{-}, \tag*{(31)} H p U ( d ) = q + ( p ) d + − q − ( p ) d − , H p L ( d ) = q − ( p ) d + − q + ( p ) d − , (31)
G p U ( d ) = q + ′ ( p ) d + − q − ′ ( p ) d − , G p L ( d ) = q − ′ ( p ) d + − q + ′ ( p ) d − . (32) G^{U}_{p}(d)=q'_{+}(p)d_{+}-q'_{-}(p)d_{-},\qquad G^{L}_{p}(d)=q'_{-}(p)d_{+}-q'_{+}(p)d_{-}. \tag*{(32)} G p U ( d ) = q + ′ ( p ) d + − q − ′ ( p ) d − , G p L ( d ) = q − ′ ( p ) d + − q + ′ ( p ) d − . (32)
Consequently, b f , η σ ( x ; p ) = a f , η 0 ( x ) + H p σ ( d f , η ( x ) ) b^{\sigma}_{f,\eta}(x;p)=a^{0}_{f,\eta}(x)+H^{\sigma}_{p}(d_{f,\eta}(x)) b f , η σ ( x ; p ) = a f , η 0 ( x ) + H p σ ( d f , η ( x )) and b ˙ f , η σ ( x ; p ) = G p σ ( d f , η ( x ) ) \dot{b}^{\sigma}_{f,\eta}(x;p)=G^{\sigma}_{p}(d_{f,\eta}(x)) b ˙ f , η σ ( x ; p ) = G p σ ( d f , η ( x )) .
Lemma B.1 (Regularity of the endpoint maps). For p ∈ [ 0 , 1 ] p\in[0,1] p ∈ [ 0 , 1 ] , the derivatives of the sensitivity endpoints satisfy
q + ′ ( p ) = Γ { 1 + ( Γ − 1 ) p } 2 , q − ′ ( p ) = Γ { Γ − ( Γ − 1 ) p } 2 , (33) q'_{+}(p)=\frac{\Gamma}{\{1+(\Gamma-1)p\}^{2}},\qquad q'_{-}(p)=\frac{\Gamma}{\{\Gamma-(\Gamma-1)p\}^{2}}, \tag*{(33)} q + ′ ( p ) = { 1 + ( Γ − 1 ) p } 2 Γ , q − ′ ( p ) = { Γ − ( Γ − 1 ) p } 2 Γ , (33) q + ′ ′ ( p ) = − 2 Γ ( Γ − 1 ) { 1 + ( Γ − 1 ) p } 3 , q − ′ ′ ( p ) = 2 Γ ( Γ − 1 ) { Γ − ( Γ − 1 ) p } 3 . (34) q''_{+}(p)=-\frac{2\Gamma(\Gamma-1)}{\{1+(\Gamma-1)p\}^{3}},\qquad q''_{-}(p)=\frac{2\Gamma(\Gamma-1)}{\{\Gamma-(\Gamma-1)p\}^{3}}. \tag*{(34)} q + ′′ ( p ) = − { 1 + ( Γ − 1 ) p } 3 2Γ ( Γ − 1 ) , q − ′′ ( p ) = { Γ − ( Γ − 1 ) p } 3 2Γ ( Γ − 1 ) . (34) In particular, Γ − 1 ≤ q ± ′ ( p ) ≤ Γ \Gamma^{-1}\le q'_{\pm}(p)\le\Gamma Γ − 1 ≤ q ± ′ ( p ) ≤ Γ and ∣ q ± ′ ′ ( p ) ∣ ≤ 2 Γ ( Γ − 1 ) \lvert q''_{\pm}(p)\rvert\le2\Gamma(\Gamma-1) ∣ q ± ′′ ( p )∣ ≤ 2Γ ( Γ − 1 ) . Uniformly in ( f , η , x , p , σ ) (f,\eta,x,p,\sigma) ( f , η , x , p , σ ) ,
0 ≤ b f , η σ ( x ; p ) ≤ 1 , ∣ b ˙ f , η σ ( x ; p ) ∣ ≤ Γ , 0\le b^{\sigma}_{f,\eta}(x;p)\le1,\qquad\lvert\dot{b}^{\sigma}_{f,\eta}(x;p)\rvert\le\Gamma, 0 ≤ b f , η σ ( x ; p ) ≤ 1 , ∣ b ˙ f , η σ ( x ; p )∣ ≤ Γ , ∣ ∂ p 2 b f , η σ ( x ; p ) ∣ ≤ 2 Γ ( Γ − 1 ) . (35) \lvert\partial^{2}_{p}b^{\sigma}_{f,\eta}(x;p)\rvert\le2\Gamma(\Gamma-1). \tag*{(35)} ∣ ∂ p 2 b f , η σ ( x ; p )∣ ≤ 2Γ ( Γ − 1 ) . (35) Moreover, H p σ H^{\sigma}_{p} H p σ is 1-Lipschitz and G p σ G^{\sigma}_{p} G p σ is Γ \Gamma Γ -Lipschitz on the real line; both vanish at zero.
Proof. Differentiating q + ( p ) = Γ p / { 1 + ( Γ − 1 ) p } q_{+}(p)=\Gamma p/\{1+(\Gamma-1)p\} q + ( p ) = Γ p / { 1 + ( Γ − 1 ) p } and q − ( p ) = p / { Γ − ( Γ − 1 ) p } q_{-}(p)=p/\{\Gamma-(\Gamma-1)p\} q − ( p ) = p / { Γ − ( Γ − 1 ) p } gives (B.3)–(B.4). The denominators before taking powers lie in [ 1 , Γ ] [1,\Gamma] [ 1 , Γ ] , giving the derivative bounds. The value a 0 + H p σ ( a 1 − a 0 ) a^{0}+H^{\sigma}_{p}(a^{1}-a^{0}) a 0 + H p σ ( a 1 − a 0 ) is a convex combination of a 0 , a 1 ∈ [ 0 , 1 ] a^{0},a^{1}\in[0,1] a 0 , a 1 ∈ [ 0 , 1 ] ; its derivatives are bounded by Γ ∣ a 1 − a 0 ∣ \Gamma\lvert a^{1}-a^{0}\rvert Γ ∣ a 1 − a 0 ∣ and 2 Γ ( Γ − 1 ) ∣ a 1 − a 0 ∣ 2\Gamma(\Gamma-1)\lvert a^{1}-a^{0}\rvert 2Γ ( Γ − 1 ) ∣ a 1 − a 0 ∣ , respectively. This proves (B.5). On either half-line the slopes of H p σ H^{\sigma}_{p} H p σ and G p σ G^{\sigma}_{p} G p σ lie in [ 0 , 1 ] [0,1] [ 0 , 1 ] and [ Γ − 1 , Γ ] [\Gamma^{-1},\Gamma] [ Γ − 1 , Γ ] , respectively. Both maps are continuous at zero, so these slope bounds also prove the Lipschitz bounds across loss ties. □ \square □
For candidate nuisance functions p c : X → [ 0 , 1 ] p_{c}:\mathcal{X}\to[0,1] p c : X → [ 0 , 1 ] and π c : X → [ π min / 2 , 1 ] \pi_{c}:\mathcal{X}\to[\pi_{\min}/2,1] π c : X → [ π m i n /2 , 1 ] , the score is
ϕ f , η σ ( O ; p c , π c ) = S a f , η Y o b s ( X ) + ( 1 − S ) b f , η σ ( X ; p c ( X ) ) + S 1 − π c ( X ) π c ( X ) b ˙ f , η σ ( X ; p c ( X ) ) { Y o b s − p c ( X ) } . (36) \phi^{\sigma}_{f,\eta}(O;p_{c},\pi_{c})=S a^{Y_{\mathrm{obs}}}_{f,\eta}(X)+(1-S)b^{\sigma}_{f,\eta}(X;p_{c}(X))
+S\frac{1-\pi_{c}(X)}{\pi_{c}(X)}\dot{b}^{\sigma}_{f,\eta}(X;p_{c}(X))\{Y_{\mathrm{obs}}-p_{c}(X)\}. \tag*{(36)} ϕ f , η σ ( O ; p c , π c ) = S a f , η Y obs ( X ) + ( 1 − S ) b f , η σ ( X ; p c ( X )) + S π c ( X ) 1 − π c ( X ) b ˙ f , η σ ( X ; p c ( X )) { Y obs − p c ( X )} . (36)
The sum of its first two terms belongs to [ 0 , 1 ] [0,1] [ 0 , 1 ] , whereas its correction has absolute value at most 2 Γ / π min 2\Gamma/\pi_{\min} 2Γ/ π m i n . Therefore
− 2 Γ π min ≤ ϕ f , η σ ( O ; p c , π c ) ≤ 1 + 2 Γ π min , (37) -\frac{2\Gamma}{\pi_{\min}}\le\phi^{\sigma}_{f,\eta}(O;p_{c},\pi_{c})\le1+\frac{2\Gamma}{\pi_{\min}}, \tag*{(37)} − π m i n 2Γ ≤ ϕ f , η σ ( O ; p c , π c ) ≤ 1 + π m i n 2Γ , (37)
∣ ϕ f , η σ ( O ; p c , π c ) ∣ ≤ B : = 1 + 2 Γ π min . \lvert\phi^{\sigma}_{f,\eta}(O;p_{c},\pi_{c})\rvert\le B:=1+\frac{2\Gamma}{\pi_{\min}}. ∣ ϕ f , η σ ( O ; p c , π c )∣ ≤ B := 1 + π m i n 2Γ .
The score can be negative; the concentration argument below explicitly allows signed classes.
Lemma B.2 (Exact conditional nuisance bias). Suppress ( f , η , σ , X ) (f,\eta,\sigma,X) ( f , η , σ , X ) inside b b b and its derivatives. With e = p c − p e=p_c-p e = p c − p , the conditional difference between the score expectation and the target excess-loss integrand is exactly
( 1 − π ) { b ( p c ) − b ( p ) − b ˙ ( p c ) e } + π c − π π c b ˙ ( p c ) e . (38) (1-\pi)\{b(p_c)-b(p)-\dot{b}(p_c)e\}+\frac{\pi_c-\pi}{\pi_c}\dot{b}(p_c)e. \tag*{(38)} ( 1 − π ) { b ( p c ) − b ( p ) − b ˙ ( p c ) e } + π c π c − π b ˙ ( p c ) e . (38) Taking P X P_X P X expectations of (B.8) gives P ϕ f , η σ ( ⋅ ; p c , π c ) − m f , η σ P\phi^\sigma_{f,\eta}(\mathord{\cdot};p_c,\pi_c)-m^\sigma_{f,\eta} P ϕ f , η σ ( ⋅ ; p c , π c ) − m f , η σ , and
∣ P ϕ f , η σ ( ⋅ ; p c , π c ) − m f , η σ ∣ ≤ Γ ( Γ − 1 ) ∥ p c − p ∥ 2 2 + 2 Γ π min ∥ p c − p ∥ 2 ∥ π c − π ∥ 2 . (39) \left|P\phi^\sigma_{f,\eta}(\mathord{\cdot};p_c,\pi_c)-m^\sigma_{f,\eta}\right|
\leq\Gamma(\Gamma-1)\lVert p_c-p\rVert_2^2+\frac{2\Gamma}{\pi_{\min}}\lVert p_c-p\rVert_2\lVert\pi_c-\pi\rVert_2. \tag*{(39)} P ϕ f , η σ ( ⋅ ; p c , π c ) − m f , η σ ≤ Γ ( Γ − 1 ) ∥ p c − p ∥ 2 2 + π m i n 2Γ ∥ p c − p ∥ 2 ∥ π c − π ∥ 2 . (39) Proof. Conditional on X X X , the observed excess-loss contribution has mean π { ( 1 − p ) a 0 + p a 1 } \pi\{(1-p)a^0+pa^1\} π {( 1 − p ) a 0 + p a 1 } and E [ S ( Y o b s − p c ) ∣ X ] = π ( p − p c ) \mathbb{E}[S(Y^{\mathrm{obs}}-p_c)\mid X]=\pi(p-p_c) E [ S ( Y obs − p c ) ∣ X ] = π ( p − p c ) . Subtracting the target integrand π { ( 1 − p ) a 0 + p a 1 } + ( 1 − π ) b ( p ) \pi\{(1-p)a^0+pa^1\}+(1-\pi)b(p) π {( 1 − p ) a 0 + p a 1 } + ( 1 − π ) b ( p ) leaves
( 1 − π ) { b ( p c ) − b ( p ) } + π ( 1 − π c ) π c b ˙ ( p c ) ( p − p c ) . (1-\pi)\{b(p_c)-b(p)\}+\frac{\pi(1-\pi_c)}{\pi_c}\dot{b}(p_c)(p-p_c). ( 1 − π ) { b ( p c ) − b ( p )} + π c π ( 1 − π c ) b ˙ ( p c ) ( p − p c ) . The identity π ( 1 − π c ) / π c = ( 1 − π ) + ( π − π c ) / π c \pi(1-\pi_c)/\pi_c=(1-\pi)+(\pi-\pi_c)/\pi_c π ( 1 − π c ) / π c = ( 1 − π ) + ( π − π c ) / π c reduces this expression to (B.8). Taylor’s theorem, expanded about p c p_c p c along the segment joining p c p_c p c and p p p , and Lemma B.1 give
∣ b ( p c ) − b ( p ) − b ˙ ( p c ) ( p c − p ) ∣ ≤ Γ ( Γ − 1 ) ( p c − p ) 2 . \left|b(p_c)-b(p)-\dot{b}(p_c)(p_c-p)\right|\leq\Gamma(\Gamma-1)(p_c-p)^2. b ( p c ) − b ( p ) − b ˙ ( p c ) ( p c − p ) ≤ Γ ( Γ − 1 ) ( p c − p ) 2 . Use 0 ≤ 1 − π ≤ 1 0\leq1-\pi\leq1 0 ≤ 1 − π ≤ 1 for the first term, and ∣ b ˙ ( p c ) / π c ∣ ≤ 2 Γ / π min |\dot{b}(p_c)/\pi_c|\leq2\Gamma/\pi_{\min} ∣ b ˙ ( p c ) / π c ∣ ≤ 2Γ/ π m i n followed by Cauchy–Schwarz for the second term. This yields (B.9) uniformly over all score indices.
The score is unbiased at the true nuisances and has O ( t 2 ) O(t^2) O ( t 2 ) bias along admissible nuisance paths with L 2 L_2 L 2 perturbations of order t t t . The sign of d f , η ( x ) d_{f,\eta}(x) d f , η ( x ) is independent of the nuisance argument, so endpoint selection adds no remainder. At Γ = 1 \Gamma=1 Γ = 1 the curvature term vanishes; for Γ > 1 \Gamma>1 Γ > 1 , knowing π \pi π alone generally leaves quadratic error in p p p . Thus the result is an orthogonal expansion, not generic double robustness. Its overlap premise and clipping ranges are substantive: clipping cannot supply labels or identification where overlap fails.
Let I 1 , … , I K I_1,\ldots,I_K I 1 , … , I K be a deterministic partition of the observation indices, with n k = ∣ I k ∣ ≥ 1 n_k=|I_k|\geq1 n k = ∣ I k ∣ ≥ 1 , n min = min k n k n_{\min}=\min_k n_k n m i n = min k n k , and w k = n k / n w_k=n_k/n w k = n k / n . A random partition independent of the observations is also allowed by conditioning on that partition. Write D − k D_{-k} D − k for the training observations outside I k I_k I k and P n , k P_{n,k} P n , k for the empirical measure on I k I_k I k . The clipped fits p ^ ( − k ) \widehat{p}^{(-k)} p ( − k ) , π ^ ( − k ) \widehat{\pi}^{(-k)} π ( − k ) are measurable functions of D − k D_{-k} D − k . Any independent fitting randomization can be included in this conditioning. Define
ϕ f , η σ , ( k ) = ϕ f , η σ ( ⋅ ; p ^ ( − k ) , π ^ ( − k ) ) , Φ k σ = { ϕ f , η σ , ( k ) : f ∈ F , η ∈ [ 0 , 1 ] } . \phi^{\sigma,(k)}_{f,\eta}=\phi^\sigma_{f,\eta}(\mathord{\cdot};\widehat{p}^{(-k)},\widehat{\pi}^{(-k)}),\qquad
\Phi_k^\sigma=\{\phi^{\sigma,(k)}_{f,\eta}:f\in\mathcal{F},\ \eta\in[0,1]\}. ϕ f , η σ , ( k ) = ϕ f , η σ ( ⋅ ; p ( − k ) , π ( − k ) ) , Φ k σ = { ϕ f , η σ , ( k ) : f ∈ F , η ∈ [ 0 , 1 ]} .
The empirical Rademacher complexities used here are
R ^ a b s σ , k = E ϵ [ sup f , η ∣ 1 n k ∑ i ∈ I k ϵ i ϕ f , η σ , ( k ) ( O i ) ∣ | D n ] , \widehat{\mathcal{R}}_{\mathrm{abs}}^{\sigma,k}
=\mathbb{E}_{\epsilon}\left[\sup_{f,\eta}\left|\frac{1}{n_k}\sum_{i\in I_k}\epsilon_i\phi^{\sigma,(k)}_{f,\eta}(O_i)\right|\middle|D_n\right], R abs σ , k = E ϵ [ f , η sup n k 1 i ∈ I k ∑ ϵ i ϕ f , η σ , ( k ) ( O i ) D n ] ,
R ^ n , a b s σ = ∑ k = 1 K w k R ^ a b s σ , k . (40) \widehat{\mathcal{R}}_{n,\mathrm{abs}}^\sigma
=\sum_{k=1}^{K}w_k\widehat{\mathcal{R}}_{\mathrm{abs}}^{\sigma,k}. \tag*{(40)} R n , abs σ = k = 1 ∑ K w k R abs σ , k . (40)
Here D n D_n D n includes the observations, partition, and fitting seeds. The expectation averages only independent uniform signs in { − 1 , 1 } \{-1,1\} { − 1 , 1 } ; the fitted functions and evaluation observations are fixed.
Lemma B.3 (Empirical-complexity bound for a signed class). Let Z 1 , … , Z m Z_1,\ldots,Z_m Z 1 , … , Z m be independent with law Q Q Q and empirical measure Q m Q_m Q m , and let H \mathcal{H} H be a fixed pointwise-separable class with ∣ h ∣ ≤ B |h|\leq B ∣ h ∣ ≤ B . For 0 < ξ < 1 0<\xi<1 0 < ξ < 1 , with probability at least 1 − ξ 1-\xi 1 − ξ ,
sup h ∈ H ∣ ( Q m − Q ) h ∣ ≤ 2 R ^ m , a b s ( H ) + 6 B log ( 2 / ξ ) 2 m , (41) \sup_{h\in\mathcal{H}}|(Q_m-Q)h|\leq2\widehat{\mathcal{R}}_{m,\mathrm{abs}}(\mathcal{H})+6B\sqrt{\frac{\log(2/\xi)}{2m}}, \tag*{(41)} h ∈ H sup ∣ ( Q m − Q ) h ∣ ≤ 2 R m , abs ( H ) + 6 B 2 m log ( 2/ ξ ) , (41) where the empirical complexity uses the absolute supremum as in (B.10).
Proof. Put Z = sup h ∣ ( Q m − Q ) h ∣ Z = \sup_h \lvert(Q_m-Q)h\rvert Z = sup h ∣( Q m − Q ) h ∣ and R = R ^ m , a b s ( H ) R = \widehat{\mathfrak{R}}_{m,\mathrm{abs}}(\mathcal{H}) R = R m , abs ( H ) . Symmetrization gives E Z ≤ 2 E R \mathbb{E}Z \le2\mathbb{E}R E Z ≤ 2 E R [23 , 24 ] . Changing one observation changes each of Z , R Z,R Z , R by at most 2 B / m 2B/m 2 B / m . With t = 2 B log ( 2 / ξ ) / ( 2 m ) t = 2B\sqrt{\log(2/\xi)/(2m)} t = 2 B log ( 2/ ξ ) / ( 2 m ) , McDiarmid’s inequality bounds each of P ( Z > E Z + t ) \mathbb{P}(Z > \mathbb{E}Z+t) P ( Z > E Z + t ) and P ( E R > R + t ) \mathbb{P}(\mathbb{E}R > R+t) P ( E R > R + t ) by ξ / 2 \xi/2 ξ /2 . Thus Z ≤ 2 R + 3 t Z \le2R+3t Z ≤ 2 R + 3 t with probability at least 1 − ξ 1-\xi 1 − ξ , proving (B.11) for signed classes with ∣ h ∣ ≤ B \lvert h\rvert\le B ∣ h ∣ ≤ B .
Conditional on D − k D_{-k} D − k , the observations in I k I_k I k are independent with law P P P , and Φ σ \Phi^\sigma Φ σ is fixed with envelope B B B . Apply Lemma B.3 with m = n k m=n_k m = n k and ξ = δ / ( 2 K ) \xi=\delta/(2K) ξ = δ / ( 2 K ) . Its conditional failure probability is at most δ / ( 2 K ) \delta/(2K) δ / ( 2 K ) , so the same bound holds unconditionally. A union bound over the 2 K 2K 2 K fold–endpoint pairs yields an event E e m p \mathcal{E}_{\mathrm{emp}} E emp of probability at least 1 − δ 1-\delta 1 − δ on which
sup f , η ∣ ∑ k = 1 K w k ( P n , k − P ) ϕ f , η σ , ( k ) ∣ ≤ 2 R ^ n , a b s σ + 6 B ∑ k = 1 K w k log ( 4 K / δ ) 2 n k ≤ 2 R ^ n , a b s σ + 6 B log ( 4 K / δ ) 2 n min . (42) \sup_{f,\eta}\left\lvert\sum_{k=1}^{K}w_k(P_{n,k}-P)\phi^{\sigma,(k)}_{f,\eta}\right\rvert
\le2\widehat{\mathfrak{R}}^{\sigma}_{n,\mathrm{abs}}+6B\sum_{k=1}^{K}w_k\sqrt{\frac{\log(4K/\delta)}{2n_k}}
\le2\widehat{\mathfrak{R}}^{\sigma}_{n,\mathrm{abs}}+6B\sqrt{\frac{\log(4K/\delta)}{2n_{\min}}}. \tag*{(42)} f , η sup k = 1 ∑ K w k ( P n , k − P ) ϕ f , η σ , ( k ) ≤ 2 R n , abs σ + 6 B k = 1 ∑ K w k 2 n k log ( 4 K / δ ) ≤ 2 R n , abs σ + 6 B 2 n m i n log ( 4 K / δ ) . (42)
for both endpoints. Here P ϕ f , η σ , ( k ) P\phi^{\sigma,(k)}_{f,\eta} P ϕ f , η σ , ( k ) integrates the fitted function against a fresh observation. The training sets overlap; no independence between folds or between their success events is used. In particular, the proof never conditions simultaneously on all fitted nuisances, which could destroy the needed held-out independence.
Define the cross-fitted empirical objectives and their infima by
Q ^ f σ ( η ) = η + 1 β ∑ k = 1 K w k P n , k ϕ f , η σ , ( k ) , \widehat{Q}^{\sigma}_{f}(\eta)=\eta+\frac{1}{\beta}\sum_{k=1}^{K}w_kP_{n,k}\phi^{\sigma,(k)}_{f,\eta}, Q f σ ( η ) = η + β 1 k = 1 ∑ K w k P n , k ϕ f , η σ , ( k ) ,
ρ ^ σ ( f ) = inf η ∈ [ 0 , 1 ] Q ^ f σ ( η ) . (43) \widehat{\rho}^{\sigma}(f)=\inf_{\eta\in[0,1]}\widehat{Q}^{\sigma}_{f}(\eta). \tag*{(43)} ρ σ ( f ) = η ∈ [ 0 , 1 ] inf Q f σ ( η ) . (43)
Proof of Theorem 2. Under the theorem’s assumptions and the measurability convention above, let E ν \mathcal{E}_{\nu} E ν be the simultaneous nuisance event
max k ∥ p ^ ( − k ) − p ∥ 2 ≤ r p , max k ∥ π ^ ( − k ) − π ∥ 2 ≤ r π , (44) \max_k\left\lVert\widehat{p}^{(-k)}-p\right\rVert_2\le r_p,\qquad
\max_k\left\lVert\widehat{\pi}^{(-k)}-\pi\right\rVert_2\le r_\pi, \tag*{(44)} k max p ( − k ) − p 2 ≤ r p , k max π ( − k ) − π 2 ≤ r π , (44)
with probability at least 1 − δ ν 1-\delta_\nu 1 − δ ν . For δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) and δ ν ∈ [ 0 , 1 − δ ) \delta_\nu\in[0,1-\delta) δ ν ∈ [ 0 , 1 − δ ) , the explicit calibrated radius is
r n ( δ ) = 1 β [ 2 max σ ∈ { L , U } R ^ n , a b s σ + 6 B log ( 4 K / δ ) 2 n min + Γ ( Γ − 1 ) r p 2 + 2 Γ π min r p r π ] . (45) r_n(\delta)=\frac{1}{\beta}\left[2\max_{\sigma\in\{L,U\}}\widehat{\mathfrak{R}}^\sigma_{n,\mathrm{abs}}+6B\sqrt{\frac{\log(4K/\delta)}{2n_{\min}}}+\Gamma(\Gamma-1)r_p^2+\frac{2\Gamma}{\pi_{\min}}r_pr_\pi\right]. \tag*{(45)} r n ( δ ) = β 1 2 σ ∈ { L , U } max R n , abs σ + 6 B 2 n m i n log ( 4 K / δ ) + Γ ( Γ − 1 ) r p 2 + π m i n 2Γ r p r π . (45)
On E ν \mathcal{E}_{\nu} E ν , Lemma B.2, applied to each realized pair of clipped fits, implies
sup f , η , σ ∣ ∑ k = 1 K w k P ϕ f , η σ , ( k ) − m f , η σ ∣ ≤ Γ ( Γ − 1 ) r p 2 + 2 Γ π min r p r π . \sup_{f,\eta,\sigma}\left\lvert\sum_{k=1}^{K}w_kP\phi^{\sigma,(k)}_{f,\eta}-m^\sigma_{f,\eta}\right\rvert
\le\Gamma(\Gamma-1)r_p^2+\frac{2\Gamma}{\pi_{\min}}r_pr_\pi. f , η , σ sup k = 1 ∑ K w k P ϕ f , η σ , ( k ) − m f , η σ ≤ Γ ( Γ − 1 ) r p 2 + π m i n 2Γ r p r π .
The intersection E = E ν ∩ E e m p \mathcal{E}=\mathcal{E}_{\nu}\cap\mathcal{E}_{\mathrm{emp}} E = E ν ∩ E emp has probability at least 1 − δ − δ ν 1-\delta-\delta_\nu 1 − δ − δ ν by the union bound. On this event, (B.12) and the last display give
sup f , η , σ ∣ ∑ k = 1 K w k P n , k ϕ f , η σ , ( k ) − m f , η σ ∣ ≤ β r n ( δ ) . \sup_{f,\eta,\sigma}\left\lvert\sum_{k=1}^{K}w_kP_{n,k}\phi^{\sigma,(k)}_{f,\eta}-m^\sigma_{f,\eta}\right\rvert\le\beta r_n(\delta). f , η , σ sup k = 1 ∑ K w k P n , k ϕ f , η σ , ( k ) − m f , η σ ≤ β r n ( δ ) .
Applying the infimum stability inequality (A.13) to Q ^ f σ \widehat{Q}^\sigma_f Q f σ and its population counterpart yields, on the same event of probability at least 1 − δ − δ ν 1-\delta-\delta_\nu 1 − δ − δ ν ,
sup f ∈ F max σ ∈ { L , U } ∣ ρ ^ σ ( f ) − ρ σ ( f ) ∣ ≤ r n ( δ ) . (46) \sup_{f\in\mathcal{F}}\max_{\sigma\in\{L,U\}}\left\lvert\widehat{\rho}^{\sigma}(f)-\rho^{\sigma}(f)\right\rvert\le r_n(\delta). \tag*{(46)} f ∈ F sup σ ∈ { L , U } max ∣ ρ σ ( f ) − ρ σ ( f ) ∣ ≤ r n ( δ ) . (46)
This requires neither convexity of the signed empirical objective nor a unique threshold or loss density there.
For a total confidence level 1 − δ t o t 1-\delta_{\mathrm{tot}} 1 − δ tot , allocate δ + δ ν ≤ δ t o t \delta+\delta_{\nu}\le\delta_{\mathrm{tot}} δ + δ ν ≤ δ tot . The empirical complexity in (B.15) is an expectation over all Rademacher signs, conditional on the realized data. A Monte Carlo estimate is not automatically an upper bound for this expectation. Any random complexity or nuisance-error bound used instead must hold simultaneously on a specified validity event, whose failure probability is included in the confidence allocation.
We next bound the score complexity. For a sample x 1 , … , x m x_1,\ldots,x_m x 1 , … , x m , let R ^ m , a b s ( V ) \widehat{\mathfrak{R}}_{m,\mathrm{abs}}(\mathcal{V}) R m , abs ( V ) denote its absolute empirical complexity. Define
L y = { x ↦ ℓ ( f ( x ) , y ) : f ∈ F } , A y = { x ↦ ( ℓ ( f ( x ) , y ) − η ) + : f ∈ F , η ∈ [ 0 , 1 ] } . \mathcal{L}_y=\{x\mapsto\ell(f(x),y):f\in\mathcal{F}\},\qquad\mathcal{A}_y=\{x\mapsto(\ell(f(x),y)-\eta)_+:f\in\mathcal{F},\ \eta\in[0,1]\}. L y = { x ↦ ℓ ( f ( x ) , y ) : f ∈ F } , A y = { x ↦ ( ℓ ( f ( x ) , y ) − η ) + : f ∈ F , η ∈ [ 0 , 1 ]} . $
Define the threshold-augmented loss-class complexity by
C n = ∑ k = 1 K w k { R ^ n k , a b s ( L 0 ) + R ^ n k , a b s ( L 1 ) + n k − 1 / 2 } , (47) \mathfrak{C}_n=\sum_{k=1}^{K}w_k\left\{\widehat{\mathfrak{R}}_{n_k,\mathrm{abs}}(\mathcal{L}_0)+\widehat{\mathfrak{R}}_{n_k,\mathrm{abs}}(\mathcal{L}_1)+n_k^{-1/2}\right\}, \tag*{(47)} C n = k = 1 ∑ K w k { R n k , abs ( L 0 ) + R n k , abs ( L 1 ) + n k − 1/2 } , (47)
where each term uses the covariates in its evaluation fold. For fixed fitted nuisances and o = ( x , s , y o b s ) o=(x,s,y^{\mathrm{obs}}) o = ( x , s , y obs ) , put p c = p ^ ( − k ) ( x ) p_c=\widehat{p}^{(-k)}(x) p c = p ( − k ) ( x ) and w = ( 1 − π ^ ( − k ) ( x ) ) / π ^ ( − k ) ( x ) w=(1-\widehat{\pi}^{(-k)}(x))/\widehat{\pi}^{(-k)}(x) w = ( 1 − π ( − k ) ( x )) / π ( − k ) ( x ) . The score is Ψ o σ ( a f , η 0 ( x ) , a f , η 1 ( x ) ) \Psi_o^\sigma(a^0_{f,\eta}(x),a^1_{f,\eta}(x)) Ψ o σ ( a f , η 0 ( x ) , a f , η 1 ( x )) , where
Ψ o σ ( u 0 , u 1 ) = s { ( 1 − y o b s ) u 0 + y o b s u 1 } + ( 1 − s ) { u 0 + H p c σ ( u 1 − u 0 ) } + s w G p c σ ( u 1 − u 0 ) ( y o b s − p c ) . (48) \begin{aligned}
\Psi_o^\sigma(u_0,u_1)
&=s\{(1-y^{\mathrm{obs}})u_0+y^{\mathrm{obs}}u_1\}\\
&\quad+(1-s)\{u_0+H^\sigma_{p_c}(u_1-u_0)\}\\
&\quad+swG^\sigma_{p_c}(u_1-u_0)(y^{\mathrm{obs}}-p_c). \tag*{(48)}
\end{aligned} Ψ o σ ( u 0 , u 1 ) = s {( 1 − y obs ) u 0 + y obs u 1 } + ( 1 − s ) { u 0 + H p c σ ( u 1 − u 0 )} + s w G p c σ ( u 1 − u 0 ) ( y obs − p c ) . (48)
It vanishes at ( 0 , 0 ) (0,0) ( 0 , 0 ) . Lemma B.1, w ≤ 2 / π min w\le2/\pi_{\min} w ≤ 2/ π m i n , and ∣ Δ u 1 − Δ u 0 ∣ ≤ 2 ∥ Δ u ∥ ∞ |\Delta u_1-\Delta u_0|\le2\|\Delta u\|_\infty ∣Δ u 1 − Δ u 0 ∣ ≤ 2∥Δ u ∥ ∞ show that it is L ∗ = 3 + 4 Γ / π min L_*=3+4\Gamma/\pi_{\min} L ∗ = 3 + 4Γ/ π m i n -Lipschitz in the maximum norm, and hence also in Euclidean norm.
The vector-contraction inequality [25 ] therefore gives, for a universal constant C C C ,
R ^ k , a b s σ ≤ C L ∗ { R ^ n k , a b s ( L 0 ) + R ^ n k , a b s ( L 1 ) + n k − 1 / 2 } . (49) \widehat{\mathfrak{R}}^\sigma_{k,\mathrm{abs}}
\le CL_*\left\{\widehat{\mathfrak{R}}_{n_k,\mathrm{abs}}(\mathcal{L}_0)+\widehat{\mathfrak{R}}_{n_k,\mathrm{abs}}(\mathcal{L}_1)+n_k^{-1/2}\right\}. \tag*{(49)} R k , abs σ ≤ C L ∗ { R n k , abs ( L 0 ) + R n k , abs ( L 1 ) + n k − 1/2 } . (49)
To handle the absolute supremum, use that the score class contains zero at η = 1 \eta=1 η = 1 : the absolute supremum is at most the sum of the positive and negative suprema, whose expectations agree by sign symmetry. Vector contraction on the positive supremum, followed by separating the two coordinate suprema, bounds it by a universal multiple of L ∗ ∑ y R ^ n k , a b s ( A y ) L_*\sum_y\widehat{\mathfrak{R}}_{n_k,\mathrm{abs}}(\mathcal{A}_y) L ∗ ∑ y R n k , abs ( A y ) . Apply the same argument to the 2 \sqrt{2} 2 -Lipschitz hinge map ( z , t ) ↦ ( z − t ) + (z,t)\mapsto(z-t)_+ ( z , t ) ↦ ( z − t ) + , adjoining zero to its input class if needed. Its coordinate complexities are those of L y \mathcal{L}_y L y and the constant threshold class; the latter is m − 1 E ε ∣ ∑ i ε i ∣ ≤ m − 1 / 2 m^{-1}\mathbb{E}_{\varepsilon}|\sum_i\varepsilon_i|\le m^{-1/2} m − 1 E ε ∣ ∑ i ε i ∣ ≤ m − 1/2 . This proves (B.19). All contraction maps are continuous endpoint values or derivatives, including at ties; no discontinuous selector class is introduced.
If L y \mathcal{L}_y L y has VC-subgraph dimension at most d ≥ 1 d\ge1 d ≥ 1 , then log N ( ϵ , L y , L 2 ( Q ) ) ≤ C 0 d log ( C 1 / ϵ ) \log N(\epsilon,\mathcal{L}_y,L_2(Q))\le C_0d\log(C_1/\epsilon) log N ( ϵ , L y , L 2 ( Q )) ≤ C 0 d log ( C 1 / ϵ ) for 0 < ϵ < 1 0<\epsilon<1 0 < ϵ < 1 and every probability measure Q Q Q . After adjoining zero and the negative class, the standard entropy-to-complexity bound [23 , 24 ] gives, uniformly over samples,
R ^ m , a b s ( L y ) ≤ C ′ d + 1 m . (50) \widehat{\mathfrak{R}}_{m,\mathrm{abs}}(\mathcal{L}_y)\le C'\sqrt{\frac{d+1}{m}}. \tag*{(50)} R m , abs ( L y ) ≤ C ′ m d + 1 . (50)
It therefore applies to the fixed fitted nuisances in each conditional argument.
For fixed K K K and balanced folds, n min ≥ c K n n_{\min}\ge c_K n n m i n ≥ c K n for a positive constant c K c_K c K once all folds are nonempty. Equations (B.19)–(B.20) give
1 β [ 2 max σ R ^ n , a b s σ + 6 B log ( 4 K / δ ) 2 n min ] (51) \frac{1}{\beta}\left[2\max_\sigma\widehat{\mathfrak{R}}^\sigma_{n,\mathrm{abs}}+6B\sqrt{\frac{\log(4K/\delta)}{2n_{\min}}}\right] \tag*{(51)} β 1 2 σ max R n , abs σ + 6 B 2 n m i n log ( 4 K / δ ) (51)
≤ C Γ , π min , K β d + log ( 1 / δ ) n . (52) \le\frac{C_{\Gamma,\pi_{\min},K}}{\beta}\sqrt{\frac{d+\log(1/\delta)}{n}}. \tag*{(52)} ≤ β C Γ , π m i n , K n d + log ( 1/ δ ) . (52)
For fixed Γ \Gamma Γ , π min \pi_{\min} π m i n , β \beta β , nuisance rates o ( n − 1 / 4 ) o(n^{-1/4}) o ( n − 1/4 ) make the nuisance contribution in (B.15) o ( n − 1 / 2 ) o(n^{-1/2}) o ( n − 1/2 ) . Joint high-probability nuisance bounds remain necessary; cross-fitting does not establish them. All uniformity is over f , η f,\eta f , η at fixed α , Γ \alpha,\Gamma α , Γ .
Example B.4 (Conditional sparse-nuisance specialization). Suppose sparse logistic nuisance fits in D D D covariates satisfy simultaneous L 2 ( P X ) L_2(P_X) L 2 ( P X ) error bounds of orders
r p ≲ s p log ( e D ) n s e l , r π ≲ s π log ( e D ) n t r , r_p \lesssim\sqrt{\frac{s_p\log(eD)}{n_{\mathrm{sel}}}}, \qquad r_\pi\lesssim\sqrt{\frac{s_\pi\log(eD)}{n_{\mathrm{tr}}}}, r p ≲ n sel s p log ( eD ) , r π ≲ n tr s π log ( eD ) , where s p , s π s_p,s_\pi s p , s π are sparsities, n t r n_{\mathrm{tr}} n tr is the smallest training-fold size, and n s e l > 0 n_{\mathrm{sel}}>0 n sel > 0 is a simultaneous lower bound for the number of revealed training labels in every fold. Substitution in the proved remainder gives, up to the constants in these assumed rates,
1 β { Γ ( Γ − 1 ) s p log ( e D ) n s e l + 2 Γ π min s p s π log ( e D ) n s e l n t r } . \frac{1}{\beta}\left\{\Gamma(\Gamma-1)\frac{s_p\log(eD)}{n_{\mathrm{sel}}}+\frac{2\Gamma}{\pi_{\min}}\frac{\sqrt{s_ps_\pi}\log(eD)}{\sqrt{n_{\mathrm{sel}}n_{\mathrm{tr}}}}\right\}. β 1 { Γ ( Γ − 1 ) n sel s p log ( eD ) + π m i n 2Γ n sel n tr s p s π log ( eD ) } . For a fixed number of balanced folds with nonempty training sets, n t r ≍ n n_{\mathrm{tr}}\asymp n n tr ≍ n . These substitutions require the stated model, design, tuning, and confidence conditions. A selected-population bound transfers to P X P_X P X because, for every measurable e e e ,
∥ e ∥ L 2 ( P X ) 2 ≤ P ( S = 1 ) π min ∥ e ∥ L 2 ( P X ∣ S = 1 ) 2 . (53) \lVert e\rVert^2_{L_2(P_X)}\le\frac{P(S=1)}{\pi_{\min}}\lVert e\rVert^2_{L_2(P_{X\mid S=1})}. \tag*{(53)} ∥ e ∥ L 2 ( P X ) 2 ≤ π m i n P ( S = 1 ) ∥ e ∥ L 2 ( P X ∣ S = 1 ) 2 . (53) Indeed, E [ π ( X ) e ( X ) 2 ] = P ( S = 1 ) E [ e ( X ) 2 ∣ S = 1 ] \mathbb{E}[\pi(X)e(X)^2]=P(S=1)\mathbb{E}[e(X)^2\mid S=1] E [ π ( X ) e ( X ) 2 ] = P ( S = 1 ) E [ e ( X ) 2 ∣ S = 1 ] and π ( X ) ≥ π min \pi(X)\ge\pi_{\min} π ( X ) ≥ π m i n . Learning p p p from revealed outcomes must therefore control target-population error.
Proposition B.5 (Finite threshold search for the fitted objectives). For a fixed f f f and realized nuisance fits, each objective in (B.13) attains its minimum on
T n ( f ) = { 0 , 1 } ∪ { ℓ ( f ( X i ) , y ) : 1 ≤ i ≤ n , y ∈ { 0 , 1 } } . T_n(f)=\{0,1\}\cup\{\ell(f(X_i),y):1\le i\le n,\ y\in\{0,1\}\}. T n ( f ) = { 0 , 1 } ∪ { ℓ ( f ( X i ) , y ) : 1 ≤ i ≤ n , y ∈ { 0 , 1 }} . Proof. Thresholding preserves the label-loss ordering at each X i X_i X i . Thus H ρ σ ( d f , η ) H^\sigma_{\rho}(d_{f,\eta}) H ρ σ ( d f , η ) and G ρ σ ( d f , η ) G^\sigma_{\rho}(d_{f,\eta}) G ρ σ ( d f , η ) are fixed coefficients times d f , η d_{f,\eta} d f , η throughout [ 0 , 1 ] [0,1] [ 0 , 1 ] ; at d f , η = 0 d_{f,\eta}=0 d f , η = 0 both representations vanish. Each score is a fixed linear combination of the two label-specific hinges, making the objective continuous and affine between successive knots of T n ( f ) T_n(f) T n ( f ) . Each such interval has an endpoint minimizer, even if the objective is constant there. Signed correction coefficients may prevent convexity but preserve this result.
An exact implementation can sort the at most 2 n + 2 2n+2 2 n + 2 breakpoints once per predictor. Write the fitted score as c 0 i ( ℓ ( f ( X i ) , 0 ) − η ) + + c 1 i ( ℓ ( f ( X i ) , 1 ) − η ) + c^i_{0}(\ell(f(X_i),0)-\eta)_+ + c^i_{1}(\ell(f(X_i),1)-\eta)_+ c 0 i ( ℓ ( f ( X i ) , 0 ) − η ) + + c 1 i ( ℓ ( f ( X i ) , 1 ) − η ) + , with coefficients fixed across η \eta η . Starting from the objective at zero, sweep the sorted breakpoints: the objective is updated by its current slope times the distance to the next breakpoint, and the slope increases by c y i / ( n β ) c^i_{y}/(n\beta) c y i / ( n β ) when the corresponding hinge vanishes. Combine all coefficients at a tied breakpoint and initialize the right slope using only losses above zero. Checking all knot values gives both global minima in O ( n log n ) O(n\log n) O ( n log n ) time and O ( n ) O(n) O ( n ) storage, after the fitted probabilities and label-specific losses have been evaluated.
Outer certificates, comparisons, and predictor selection
Proof of the decision guarantees in § 4.2 . Work on the joint event (B.16) of probability at least 1 − δ − δ ν 1-\delta-\delta_\nu 1 − δ − δ ν , and use the certificate (8) . Since 0 ≤ ρ L ( f ) ≤ ρ U ( f ) ≤ 1 0\le\rho^L(f)\le\rho^U(f)\le1 0 ≤ ρ L ( f ) ≤ ρ U ( f ) ≤ 1 , clipping preserves containment of the sharp interval and hence every compatible CVaR. Containment gives the lower width bound; clipping decreases width, while
ρ ^ U ( f ) − ρ ^ L ( f ) + 2 r n ≤ ρ U ( f ) − ρ L ( f ) + 4 r n , \widehat{\rho}^{U}(f)-\widehat{\rho}^{L}(f)+2r_n\le\rho^{U}(f)-\rho^{L}(f)+4r_n, ρ U ( f ) − ρ L ( f ) + 2 r n ≤ ρ U ( f ) − ρ L ( f ) + 4 r n ,
proving the upper bound. If ρ ^ U ( f ) + r n < ρ ^ L ( g ) − r n \widehat{\rho}^{U}(f)+r_n<\widehat{\rho}^{L}(g)-r_n ρ U ( f ) + r n < ρ L ( g ) − r n , then for every compatible law,
ρ α , P ⋆ ( f ) ≤ ρ U ( f ) ≤ ρ ^ U ( f ) + r n ≤ ρ ^ L ( g ) − r n ≤ ρ L ( g ) ≤ ρ α , P ⋆ ( g ) . \begin{aligned}
\rho_{\alpha,P^\star}(f)\le\rho^{U}(f)\le\widehat{\rho}^{U}(f)+r_n \\
&\le\widehat{\rho}^{L}(g)-r_n\le\rho^{L}(g)\le\rho_{\alpha,P^\star}(g).
\end{aligned} ρ α , P ⋆ ( f ) ≤ ρ U ( f ) ≤ ρ U ( f ) + r n ≤ ρ L ( g ) − r n ≤ ρ L ( g ) ≤ ρ α , P ⋆ ( g ) .
Finally, any measurable ϵ o p t \epsilon_{\mathrm{opt}} ϵ opt -approximate minimizer f ^ ∈ F \widehat{f}\in\mathcal{F} f ∈ F of ρ P ^ \rho^{\widehat{P}} ρ P satisfies
ρ U ( f ^ ) ≤ inf f ρ P ^ ( f ) + ϵ o p t + r n ≤ inf f ρ U ( f ) + ϵ o p t + 2 r n . \rho^{U}(\widehat{f})\le\inf_f\rho^{\widehat{P}}(f)+\epsilon_{\mathrm{opt}}+r_n
\le\inf_f\rho^{U}(f)+\epsilon_{\mathrm{opt}}+2r_n. ρ U ( f ) ≤ f inf ρ P ( f ) + ϵ opt + r n ≤ f inf ρ U ( f ) + ϵ opt + 2 r n .
The last inequality takes infima of the pointwise endpoint bound, so no exact minimizer is needed. Uniformity permits all predictors in these claims to be chosen using the same data.
The selection benchmark is the smallest compatible upper endpoint in the class. For every compatible P ⋆ P^\star P ⋆ , the same argument gives ρ α , P ⋆ ( f ^ ) ≤ inf f ρ U ( f ) + 2 r n + ϵ o p t \rho_{\alpha,P^\star}(\widehat{f})\le\inf_f\rho^U(f)+2r_n+\epsilon_{\mathrm{opt}} ρ α , P ⋆ ( f ) ≤ inf f ρ U ( f ) + 2 r n + ϵ opt . Clipped endpoint errors are at most 2 r n 2r_n 2 r n on the same event. Convergence in probability follows if r n → 0 r_n\to0 r n → 0 in probability and δ n + δ ν , n → 0 \delta_n+\delta_{\nu,n}\to0 δ n + δ ν , n → 0 ; a fixed failure probability remains fixed.
Oracle calibration for finite-cell experiments
For the oracle coverage experiment, suppose p = p 1 p=p_1 p = p 1 is known and define
h f , η σ ( O ) = S a f , η Y o b s ( X ) + ( 1 − S ) b f , η σ ( X ; p ( X ) ) , h^\sigma_{f,\eta}(O)=S a^{Y^{\mathrm{obs}}}_{f,\eta}(X)+(1-S)b^\sigma_{f,\eta}(X;p(X)), h f , η σ ( O ) = S a f , η Y obs ( X ) + ( 1 − S ) b f , η σ ( X ; p ( X )) ,
ρ ^ o r σ ( f ) = inf η ∈ [ 0 , 1 ] { η + β − 1 P n h f , η σ } . (54) \widehat{\rho}^{\sigma}_{\mathrm{or}}(f)=\inf_{\eta\in[0,1]}\{\eta+\beta^{-1}P_nh^\sigma_{f,\eta}\}. \tag*{(54)} ρ or σ ( f ) = η ∈ [ 0 , 1 ] inf { η + β − 1 P n h f , η σ } . (54)
This score lies in [ 0 , 1 ] [0,1] [ 0 , 1 ] and has mean m f , η σ m^\sigma_{f,\eta} m f , η σ ; estimating π \pi π is unnecessary for this oracle construction.
Proposition B.6 (Finite-cell oracle certificate). Fix a predictor f f f independently of the evaluation observations. Suppose P X P_X P X is supported on J J J covariate values x 1 , … , x J x_1,\ldots,x_J x 1 , … , x J . Let
T ( f ) = { 0 , 1 } ∪ { ℓ ( f ( x j ) , y ) : 1 ≤ j ≤ J , y ∈ { 0 , 1 } } , M = ∣ T ( f ) ∣ ≤ 2 J + 2. T(f)=\{0,1\}\cup\{\ell(f(x_j),y):1\le j\le J,\ y\in\{0,1\}\},\qquad M=|T(f)|\le2J+2. T ( f ) = { 0 , 1 } ∪ { ℓ ( f ( x j ) , y ) : 1 ≤ j ≤ J , y ∈ { 0 , 1 }} , M = ∣ T ( f ) ∣ ≤ 2 J + 2.
With probability at least 1 − δ 1-\delta 1 − δ ,
max σ ∈ { L , U } ∣ ρ ^ o r σ ( f ) − ρ σ ( f ) ∣ ≤ r o r ( δ ) : = 1 β log ( 4 M / δ ) 2 n . (55) \max_{\sigma\in\{L,U\}}\left|\widehat{\rho}^{\sigma}_{\mathrm{or}}(f)-\rho^\sigma(f)\right|
\le r_{\mathrm{or}}(\delta):=\frac{1}{\beta}\sqrt{\frac{\log(4M/\delta)}{2n}}. \tag*{(55)} σ ∈ { L , U } max ∣ ρ or σ ( f ) − ρ σ ( f ) ∣ ≤ r or ( δ ) := β 1 2 n log ( 4 M / δ ) . (55)
Thus the clipped interval obtained from (8) using these oracle estimators and r o r r_{\mathrm{or}} r or contains the entire sharp population interval with probability at least 1 − δ 1-\delta 1 − δ .
Proof. By the piecewise-affine argument of Proposition B.5 , both h f , η σ ( O ) h^\sigma_{f,\eta}(O) h f , η σ ( O ) and its expectation have knots only in T ( f ) T(f) T ( f ) . The absolute empirical-minus-population difference is maximal at a knot:
sup η ∈ [ 0 , 1 ] ∣ ( P n − P ) h f , η σ ∣ = max η ∈ T ( f ) ∣ ( P n − P ) h f , η σ ∣ . \sup_{\eta\in[0,1]}\left|(P_n-P)h^\sigma_{f,\eta}\right|
=\max_{\eta\in T(f)}\left|(P_n-P)h^\sigma_{f,\eta}\right|. η ∈ [ 0 , 1 ] sup ( P n − P ) h f , η σ = η ∈ T ( f ) max ( P n − P ) h f , η σ .
Hoeffding’s inequality and a union bound over the 2 M 2M 2 M endpoint–threshold pairs bound the failure probability at tolerance u u u by 4 M e − 2 n u 2 4M e^{-2nu^2} 4 M e − 2 n u 2 . Taking u = log ( 4 M / δ ) / ( 2 n ) u=\sqrt{\log(4M/\delta)/(2n)} u = log ( 4 M / δ ) / ( 2 n ) and applying (A.13) proves (B.24); the containment argument in Appendix § 13 .§ 4 gives whole-interval coverage.
The population-support set T ( f ) T(f) T ( f ) includes losses at cells absent from a particular sample; observed knots T n ( f ) T_n(f) T n ( f ) suffice for empirical minimization but cannot replace T ( f ) T(f) T ( f ) in this concentration argument. Conditioning covers a predictor trained on independent data. Selection using evaluation data requires a uniform bound: for a fixed finite family F 0 \mathcal{F}_0 F 0 , replace M M M by ∑ f ∈ F 0 ∣ T ( f ) ∣ \sum_{f\in\mathcal{F}_0}|T(f)| ∑ f ∈ F 0 ∣ T ( f ) ∣ . Known p p p supplies the oracle experiment’s premise; coverage for learned-nuisance estimators instead requires Theorem 2’s controls.
Revealed-tail learning lower bound
We now examine the difficulty of learning a predictor in the Γ = 1 \Gamma=1 Γ = 1 submodel, where population risks are identified. The lower-bound construction uses equal tail-cell masses and constant observation probability.
Theorem B.7 (Revealed-tail observation complexity). Fix α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) , β = 1 − α \beta= 1-\alpha β = 1 − α , π 0 ∈ ( 0 , 1 ] \pi_0 \in(0,1] π 0 ∈ ( 0 , 1 ] , and n ≥ 1 n \ge1 n ≥ 1 . For every integer 1 ≤ d ≤ n π 0 β 1 \le d \le n\pi_0\beta 1 ≤ d ≤ n π 0 β , there exist a binary classification class F \mathcal{F} F of VC dimension d d d and a family P \mathcal{P} P of full-data laws with Γ = 1 \Gamma= 1 Γ = 1 and π ( X ) = π 0 \pi(X) = \pi_0 π ( X ) = π 0 such that
inf f ^ sup P ⋆ ∈ P E ( P o b s ⋆ ) ⊗ n [ ρ α , P ⋆ ( f ^ ) − inf f ∈ F ρ α , P ⋆ ( f ) ] ≥ 1 16 d n π 0 β . (56) \inf_{\widehat{f}}\sup_{P^\star\in\mathcal{P}}\mathbb{E}_{(P^\star_{\mathrm{obs}})^{\otimes n}}\left[\rho_{\alpha,P^\star}(\widehat{f})-\inf_{f\in\mathcal{F}}\rho_{\alpha,P^\star}(f)\right]\ge\frac{1}{16}\sqrt{\frac{d}{n\pi_0\beta}}. \tag*{(56)} f inf P ⋆ ∈ P sup E ( P obs ⋆ ) ⊗ n [ ρ α , P ⋆ ( f ) − f ∈ F inf ρ α , P ⋆ ( f ) ] ≥ 16 1 n π 0 β d . (56) The infimum ranges over measurable learners taking values in F \mathcal{F} F , including randomized learners.
Proof. We use a direct binary-testing form of the hypercube argument [26 ] . Fix π 0 ∈ ( 0 , 1 ] \pi_0 \in(0,1] π 0 ∈ ( 0 , 1 ] , n ≥ 1 n \ge1 n ≥ 1 , and an integer 1 ≤ d ≤ n π 0 β 1 \le d \le n\pi_0\beta 1 ≤ d ≤ n π 0 β . Put
γ = 1 8 d n π 0 β ≤ 1 8 . (57) \gamma=\frac{1}{8}\sqrt{\frac{d}{n\pi_0\beta}}\le\frac{1}{8}. \tag*{(57)} γ = 8 1 n π 0 β d ≤ 8 1 . (57)
Let X = { ⋆ , 1 , … , d } \mathcal{X}=\{\star,1,\ldots,d\} X = { ⋆ , 1 , … , d } with P X ( ⋆ ) = 1 − β P_X(\star)=1-\beta P X ( ⋆ ) = 1 − β and P X ( j ) = β / d P_X(j)=\beta/d P X ( j ) = β / d . For v ∈ { − 1 , 1 } d v\in\{-1,1\}^{d} v ∈ { − 1 , 1 } d , define f v ( ⋆ ) = 0 f_v(\star)=0 f v ( ⋆ ) = 0 and f v ( j ) = ( 1 + v j ) / 2 f_v(j)=(1+v_j)/2 f v ( j ) = ( 1 + v j ) /2 , and set F = { f v : v ∈ { − 1 , 1 } d } \mathcal{F}=\{f_v:v\in\{-1,1\}^{d}\} F = { f v : v ∈ { − 1 , 1 } d } . The class shatters { 1 , … , d } \{1,\ldots,d\} { 1 , … , d } , and cannot shatter any larger set because its value at ⋆ \star ⋆ is fixed. Its VC dimension is exactly d d d . Use ordinary zero-one loss ℓ ( f ( x ) , y ) = 1 { f ( x ) ≠ y } \ell(f(x),y)=1\{f(x)\ne y\} ℓ ( f ( x ) , y ) = 1 { f ( x ) = y } .
For each θ ∈ { − 1 , 1 } d \theta\in\{-1,1\}^{d} θ ∈ { − 1 , 1 } d , construct a full-data law P θ P_\theta P θ with Y = 0 Y=0 Y = 0 at ⋆ \star ⋆ and
P θ ( Y = 1 ∣ X = j ) = 1 2 + γ θ j , j = 1 , … , d . P_\theta(Y=1\mid X=j)=\frac{1}{2}+\gamma\theta_j,\qquad j=1,\ldots,d. P θ ( Y = 1 ∣ X = j ) = 2 1 + γ θ j , j = 1 , … , d .
Let S ∼ Bernoulli ( π 0 ) S\sim\operatorname{Bernoulli}(\pi_0) S ∼ Bernoulli ( π 0 ) independently of ( X , Y ) (X,Y) ( X , Y ) , and observe ( X , S , S Y ) (X,S,SY) ( X , S , S Y ) . These laws satisfy overlap with π ( X ) = π 0 \pi(X)=\pi_0 π ( X ) = π 0 and conditional ignorability, hence belong to the Γ = 1 \Gamma=1 Γ = 1 model. All losses vanish at ⋆ \star ⋆ . The conditional error within { 1 , … , d } \{1,\ldots,d\} { 1 , … , d } is
R θ ( f v ) = 1 2 − γ + 2 γ d ∑ j = 1 d 1 { v j ≠ θ j } . (58) R_\theta(f_v)=\frac{1}{2}-\gamma+\frac{2\gamma}{d}\sum_{j=1}^{d}1\{v_j\ne\theta_j\}. \tag*{(58)} R θ ( f v ) = 2 1 − γ + d 2 γ j = 1 ∑ d 1 { v j = θ j } . (58)
The full-population loss is Bernoulli with parameter q θ , v = β R θ ( f v ) ≤ β q_{\theta,v}=\beta R_\theta(f_v)\le\beta q θ , v = β R θ ( f v ) ≤ β ; the Bernoulli identity (A.5) gives
ρ α , P θ ( f v ) = R θ ( f v ) , \rho_{\alpha,P_\theta}(f_v)=R_\theta(f_v), ρ α , P θ ( f v ) = R θ ( f v ) ,
ρ α , P θ ( f v ) − inf f ∈ F ρ α , P θ ( f ) = 2 γ d ∑ j = 1 d 1 { v j ≠ θ j } . (59) \rho_{\alpha,P_\theta}(f_v)-\inf_{f\in\mathcal{F}}\rho_{\alpha,P_\theta}(f)=\frac{2\gamma}{d}\sum_{j=1}^{d}1\{v_j\ne\theta_j\}. \tag*{(59)} ρ α , P θ ( f v ) − f ∈ F inf ρ α , P θ ( f ) = d 2 γ j = 1 ∑ d 1 { v j = θ j } . (59)
The minimizing predictor is f θ f_\theta f θ .
Let Q θ = P θ , o b s ⊗ n Q_\theta=P_{\theta,\mathrm{obs}}^{\otimes n} Q θ = P θ , obs ⊗ n and let θ ( j ) \theta^{(j)} θ ( j ) be obtained by reversing the j j j th coordinate. For one observation, the two distributions differ only on { X = j , S = 1 } \{X=j,S=1\} { X = j , S = 1 } , which has probability β π 0 / d \beta\pi_0/d β π 0 / d under both laws. Conditioning on ( X , S ) (X,S) ( X , S ) and using the chain rule for relative entropy therefore gives
D K L ( Q θ ∥ Q θ ( j ) ) = n β π 0 d D K L ( Bernoulli ( 1 / 2 + γ ) ∥ Bernoulli ( 1 / 2 − γ ) ) ≤ 16 n β π 0 γ 2 d = 1 4 . (60) D_{\mathrm{KL}}(Q_\theta\Vert Q_{\theta^{(j)}})=\frac{n\beta\pi_0}{d}D_{\mathrm{KL}}\bigl(\operatorname{Bernoulli}(1/2+\gamma)\Vert\operatorname{Bernoulli}(1/2-\gamma)\bigr)
\le\frac{16n\beta\pi_0\gamma^2}{d}=\frac{1}{4}. \tag*{(60)} D KL ( Q θ ∥ Q θ ( j ) ) = d n β π 0 D KL ( Bernoulli ( 1/2 + γ ) ∥ Bernoulli ( 1/2 − γ ) ) ≤ d 16 n β π 0 γ 2 = 4 1 . (60)
Here the symmetric Bernoulli divergences both equal 2 γ log { ( 1 + 2 γ ) / ( 1 − 2 γ ) } ≤ 16 γ 2 2\gamma\log\{(1+2\gamma)/(1-2\gamma)\}\le16\gamma^2 2 γ log {( 1 + 2 γ ) / ( 1 − 2 γ )} ≤ 16 γ 2 for γ ≤ 1 / 4 \gamma\le1/4 γ ≤ 1/4 , and independence supplies the factor n n n . Pinsker’s inequality gives
TV ( Q θ , Q θ ( j ) ) ≤ D K L ( Q θ ∥ Q θ ( j ) ) / 2 ≤ 1 2 2 ≤ 1 2 . (61) \operatorname{TV}(Q_\theta,Q_{\theta^{(j)}})\le\sqrt{D_{\mathrm{KL}}(Q_\theta\Vert Q_{\theta^{(j)}})/2}\le\frac{1}{2\sqrt{2}}\le\frac{1}{2}. \tag*{(61)} TV ( Q θ , Q θ ( j ) ) ≤ D KL ( Q θ ∥ Q θ ( j ) ) /2 ≤ 2 2 1 ≤ 2 1 . (61)
Consider any measurable learner taking values in F \mathcal{F} F , written f ^ = f v ^ \widehat{f}=f_{\widehat{v}} f = f v . For a fixed coordinate j j j , pair each θ \theta θ with θ j = 1 \theta_j=1 θ j = 1 with θ ( j ) \theta^{(j)} θ ( j ) , and write A j = { v ^ j = 1 } A_j=\{\widehat{v}_j=1\} A j = { v j = 1 } . The sum of the two testing errors satisfies
Q θ ( v ^ j ≠ θ j ) + Q θ ( j ) ( v ^ j ≠ θ ( j ) ) = Q θ ( A j c ) + Q θ ( j ) ( A j ) ≥ 1 − TV ( Q θ , Q θ ( j ) ) ≥ 1 2 . Q_\theta(\widehat{v}_j\ne\theta_j)+Q_{\theta^{(j)}}(\widehat{v}_j\ne\theta^{(j)})=Q_\theta(A_j^c)+Q_{\theta^{(j)}}(A_j)\ge1-\operatorname{TV}(Q_\theta,Q_{\theta^{(j)}})\ge\frac{1}{2}. Q θ ( v j = θ j ) + Q θ ( j ) ( v j = θ ( j ) ) = Q θ ( A j c ) + Q θ ( j ) ( A j ) ≥ 1 − TV ( Q θ , Q θ ( j ) ) ≥ 2 1 .
There are 2 d − 1 2^{d-1} 2 d − 1 such pairs. Averaging over all 2 d 2^d 2 d vertices gives 2 − d ∑ θ Q θ ( v ˉ j ≠ θ j ) ≥ 1 / 4 2^{-d}\sum_{\theta} Q_\theta(\bar{v}_j \ne\theta_j) \ge1/4 2 − d ∑ θ Q θ ( v ˉ j = θ j ) ≥ 1/4 . Sum this inequality over j j j and use (B.28) to obtain
1 2 d ∑ θ E Q θ [ ρ α , P θ ( f ^ ) − inf f ∈ F ρ α , P θ ( f ) ] ≥ 2 γ d d 4 = 1 16 d n π 0 β . \frac{1}{2^d}\sum_{\theta}\mathbb{E}_{Q_\theta}\left[\rho_{\alpha,P_\theta}(\hat{f})-\inf_{f\in\mathcal{F}}\rho_{\alpha,P_\theta}(f)\right]\ge\frac{2\gamma}{d}\frac{d}{4}=\frac{1}{16}\sqrt{\frac{d}{n\pi_0\beta}}. 2 d 1 θ ∑ E Q θ [ ρ α , P θ ( f ^ ) − f ∈ F inf ρ α , P θ ( f ) ] ≥ d 2 γ 4 d = 16 1 n π 0 β d .
The supremum over θ \theta θ is at least this average, for every learner. Taking the infimum over learners proves Theorem B.7 . Randomized learners satisfy the same bound: append their independent random seed to each experiment, which leaves the neighboring total variation unchanged, and apply the identical testing argument. □ \square □
Only the expected n π 0 β n\pi_0\beta n π 0 β revealed observations in the tail stratum distinguish the vertices. This is an expected excess-CVaR lower bound for F \mathcal{F} F -valued learners, already with fully identified risks at Γ = 1 \Gamma=1 Γ = 1 .
Revelation Theory and Algorithms
Fix f f f , α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) , β = 1 − α \beta=1-\alpha β = 1 − α , and finite Γ ≥ 1 \Gamma\ge1 Γ ≥ 1 . Use the historical π \pi π , p 1 p_1 p 1 , sensitivity bounds [ p ‾ Γ , p ‾ Γ ] [\underline{p}_{\Gamma},\overline{p}_{\Gamma}] [ p Γ , p Γ ] , and s Γ = p ‾ Γ − p ‾ Γ s_{\Gamma}=\overline{p}_{\Gamma}-\underline{p}_{\Gamma} s Γ = p Γ − p Γ . Write μ = P X \mu=P_X μ = P X and E = E μ \mathbb{E}=\mathbb{E}_{\mu} E = E μ . All kernels and policies are measurable; conditional statements apply on components of positive population mass.
Retained-information model and compatible laws
Additional revelation A ∈ { 0 , 1 } A\in\{0,1\} A ∈ { 0 , 1 } satisfies A ≤ 1 − S A\le1-S A ≤ 1 − S , with intensity λ ( x ) = P ( A = 1 ∣ X = x , S = 0 ) ∈ [ 0 , 1 ] \lambda(x)=P(A=1\mid X=x,S=0)\in[0,1] λ ( x ) = P ( A = 1 ∣ X = x , S = 0 ) ∈ [ 0 , 1 ] . The audited population law has known p A ( x ) = P ( Y = 1 ∣ X = x , A = 1 ) p_A(x)=P(Y=1\mid X=x,A=1) p A ( x ) = P ( Y = 1 ∣ X = x , A = 1 ) satisfying
p ‾ Γ ≤ p A ≤ p ‾ Γ on { ( 1 − π ) λ > 0 } . (62) \underline{p}_{\Gamma}\le p_A\le\overline{p}_{\Gamma}\quad\text{on }\{(1-\pi)\lambda>0\}. \tag*{(62)} p Γ ≤ p A ≤ p Γ on {( 1 − π ) λ > 0 } . (62)
Extend p A p_A p A within these bounds on its zero-mass region; this has no effect on risks.
Definition C.1 (Retained-sensitivity compatibility). The class P Γ , λ \mathcal{P}_{\Gamma,\lambda} P Γ , λ consists of laws of ( X , S , A , Y ) (X,S,A,Y) ( X , S , A , Y ) with X X X -marginal μ \mu μ and conditional components
The residual kernel ranges freely over q ∈ [ p ‾ Γ , p ‾ Γ ] q\in[\underline{p}_{\Gamma},\overline{p}_{\Gamma}] q ∈ [ p Γ , p Γ ] ; ( S , A ) = ( 1 , 1 ) (S,A)=(1,1) ( S , A ) = ( 1 , 1 ) has zero mass.
Lemma C.2 (Historical compatibility). Every admissible q q q defines a retained law whose marginal belongs to P Γ ( P o b s ) \mathcal{P}_{\Gamma}(P_{\mathrm{obs}}) P Γ ( P obs ) , with original hidden kernel
p 0 = λ p A + ( 1 − λ ) q on { π < 1 } . (63) p_0=\lambda p_A+(1-\lambda)q\quad\text{on }\{\pi<1\}. \tag*{(63)} p 0 = λ p A + ( 1 − λ ) q on { π < 1 } . (63)
The retained class is nonempty.
Proof. Apply the kernel construction of Appendix § 12 .§ 1 to the table. Its masses sum to one and its selected component is unchanged. Conditional total probability gives (C.2); convexity of the sensitivity interval proves compatibility. Taking q = p ‾ Γ q=\underline{p}_{\Gamma} q = p Γ proves nonemptiness. □ \square □
At λ = 0 \lambda=0 λ = 0 the historical class is recovered; at λ = 1 \lambda=1 λ = 1 no residual kernel remains. Membership requires the audited and residual bounds separately: a bound on their mixture p 0 p_0 p 0 alone does not imply either component bound.
Known randomization A ⊥ Y ∣ ( X , S = 0 ) A\perp Y\mid(X,S=0) A ⊥ Y ∣ ( X , S = 0 ) supplies extra information: p A = p 0 p_A=p_0 p A = p 0 on the audited support, and q = p 0 q=p_0 q = p 0 wherever residual mass remains. Thus the population audited law identifies p 0 p_0 p 0 wherever originally hidden cases have positive audit probability. The retained class instead keeps its residual kernel free within the displayed bounds. Finite audits additionally require estimation and uncertainty control for the audited contribution; a positive population audit probability alone supplies no finite-sample precision guarantee.
Post-revelation endpoints and tail identification value
Use a f , η y , d f , η , b f , η σ a^y_{f,\eta},d_{f,\eta},b^\sigma_{f,\eta} a f , η y , d f , η , b f , η σ from Section 3, write s Γ = p ˉ Γ − p Γ s_\Gamma=\bar p_\Gamma-p_\Gamma s Γ = p ˉ Γ − p Γ , and set w λ = ( 1 − π ) ( 1 − λ ) w_\lambda=(1-\pi)(1-\lambda) w λ = ( 1 − π ) ( 1 − λ ) . The known contribution and a residual kernel’s threshold objective are
K f , η ( λ ) = E [ π ( a f , η 0 + p 1 d f , η ) + ( 1 − π ) λ ( a f , η 0 + p A d f , η ) ] , (64) K_{f,\eta}(\lambda)=\mathbb{E}\left[\pi(a^0_{f,\eta}+p_1d_{f,\eta})+(1-\pi)\lambda(a^0_{f,\eta}+p_A d_{f,\eta})\right], \tag*{(64)} K f , η ( λ ) = E [ π ( a f , η 0 + p 1 d f , η ) + ( 1 − π ) λ ( a f , η 0 + p A d f , η ) ] , (64)
G q ( η ) = η + β − 1 { K f , η ( λ ) + E [ w λ ( a f , η 0 + q d f , η ) ] } . (65) G_q(\eta)=\eta+\beta^{-1}\left\{K_{f,\eta}(\lambda)+\mathbb{E}\left[w_\lambda(a^0_{f,\eta}+qd_{f,\eta})\right]\right\}. \tag*{(65)} G q ( η ) = η + β − 1 { K f , η ( λ ) + E [ w λ ( a f , η 0 + q d f , η ) ] } . (65)
Conditioning on the table and optimizing the affine residual term give
L f , η ( λ ) = η + β − 1 { K f , η ( λ ) + E [ w λ b f , η L ] } , L_{f,\eta}(\lambda)=\eta+\beta^{-1}\left\{K_{f,\eta}(\lambda)+\mathbb{E}\left[w_\lambda b^L_{f,\eta}\right]\right\}, L f , η ( λ ) = η + β − 1 { K f , η ( λ ) + E [ w λ b f , η L ] } ,
U f , η ( λ ) = η + β − 1 { K f , η ( λ ) + E [ w λ b f , η U ] } . (66) U_{f,\eta}(\lambda)=\eta+\beta^{-1}\left\{K_{f,\eta}(\lambda)+\mathbb{E}\left[w_\lambda b^U_{f,\eta}\right]\right\}. \tag*{(66)} U f , η ( λ ) = η + β − 1 { K f , η ( λ ) + E [ w λ b f , η U ] } . (66)
Proposition C.3 (Sharp retained-sensitivity interval). The attainable population risks are exactly
{ ρ α , Q ( f ) : Q ∈ P Γ , λ r e t } = [ min η ∈ [ 0 , 1 ] L f , η ( λ ) , min η ∈ [ 0 , 1 ] U f , η ( λ ) ] . \{\rho_{\alpha,Q}(f):Q\in\mathcal{P}^{\mathrm{ret}}_{\Gamma,\lambda}\}
=
\left[
\min_{\eta\in[0,1]}L_{f,\eta}(\lambda),
\min_{\eta\in[0,1]}U_{f,\eta}(\lambda)
\right]. { ρ α , Q ( f ) : Q ∈ P Γ , λ ret } = [ η ∈ [ 0 , 1 ] min L f , η ( λ ) , η ∈ [ 0 , 1 ] min U f , η ( λ ) ] . Both threshold minima and endpoint laws are attained.
Proof. On H = { x : π ( x ) < 1 } H=\{x:\pi(x)<1\} H = { x : π ( x ) < 1 } , the common kernels q f L , q f U q_f^L,q_f^U q f L , q f U of Appendix § 12 .§ 3 satisfy a f , η 0 + q f L d f , η = b f , η L a^0_{f,\eta}+q^L_f d_{f,\eta}=b^L_{f,\eta} a f , η 0 + q f L d f , η = b f , η L for every η ∈ [ 0 , 1 ] \eta\in[0,1] η ∈ [ 0 , 1 ] . On H c H^c H c , the residual weight w λ = ( 1 − π ) ( 1 − λ ) w_\lambda=(1-\pi)(1-\lambda) w λ = ( 1 − π ) ( 1 − λ ) is zero. Thus L f , η ( λ ) ≤ G q ( η ) ≤ U f , η ( λ ) L_{f,\eta}(\lambda)\le G_q(\eta)\le U_{f,\eta}(\lambda) L f , η ( λ ) ≤ G q ( η ) ≤ U f , η ( λ ) for every admissible q q q , with equality at the corresponding common kernel. Taking minima proves the endpoint claims; continuity of each hinge objective on [ 0 , 1 ] [0,1] [ 0 , 1 ] gives attainment. For q t = ( 1 − t ) q f L + t q f U q_t=(1-t)q_f^L+tq_f^U q t = ( 1 − t ) q f L + t q f U ,
sup η ∣ G q t ( η ) − G q s ( η ) ∣ ≤ ∣ t − s ∣ β − 1 E [ w λ s Γ ] . \sup_\eta\left|G_{q_t}(\eta)-G_{q_s}(\eta)\right|
\le|t-s|\beta^{-1}\mathbb{E}[w_\lambda s_\Gamma]. η sup ∣ G q t ( η ) − G q s ( η ) ∣ ≤ ∣ t − s ∣ β − 1 E [ w λ s Γ ] . The infimum inequality (A.13) makes the optimized risk continuous in t t t . This admissible path joins the endpoints and fills their interval.
Proposition C.4 (Exact revelation value). With
τ f , η = ( 1 − π ) s Γ ∣ d f , η ∣ / β , (67) \tau_{f,\eta}=(1-\pi)s_\Gamma|d_{f,\eta}|/\beta, \tag*{(67)} τ f , η = ( 1 − π ) s Γ ∣ d f , η ∣/ β , (67) W f , η ( λ ) : = U f , η ( λ ) − L f , η ( λ ) = E [ ( 1 − λ ) τ f , η ] = W f , η ( 0 ) − E [ λ τ f , η ] , (68) W_{f,\eta}(\lambda):=U_{f,\eta}(\lambda)-L_{f,\eta}(\lambda)
=\mathbb{E}[(1-\lambda)\tau_{f,\eta}]
=W_{f,\eta}(0)-\mathbb{E}[\lambda\tau_{f,\eta}], \tag*{(68)} W f , η ( λ ) := U f , η ( λ ) − L f , η ( λ ) = E [( 1 − λ ) τ f , η ] = W f , η ( 0 ) − E [ λ τ f , η ] , (68) every bounded measurable h h h and feasible t t t satisfy
W f , η ( λ + t h ) − W f , η ( λ ) = − t E [ h τ f , η ] . (69) W_{f,\eta}(\lambda+th)-W_{f,\eta}(\lambda)
=-t\mathbb{E}[h\tau_{f,\eta}]. \tag*{(69)} W f , η ( λ + t h ) − W f , η ( λ ) = − t E [ h τ f , η ] . (69) Proof. Subtract (C.5): K K K cancels and b U − b L = s Γ ∣ d ∣ b^U-b^L=s_\Gamma|d| b U − b L = s Γ ∣ d ∣ . Substitution gives the perturbation identity, including one-sided feasible directions. Since 0 ≤ τ ≤ β − 1 0\le\tau\le\beta^{-1} 0 ≤ τ ≤ β − 1 , all terms are integrable. The fixed-threshold reduction is independent of p A p_A p A , even when that law varies with the policy.
Minimum-cost precision and the exact frontier
Fix f , η , Γ f,\eta,\Gamma f , η , Γ , abbreviate τ = τ f , η \tau=\tau_{f,\eta} τ = τ f , η , and let c ≥ 0 c\ge0 c ≥ 0 be measurable with E [ c ] < ∞ \mathbb{E}[c]<\infty E [ c ] < ∞ . Set T = E [ τ ] = W f , η ( 0 ) T=\mathbb{E}[\tau]=W_{f,\eta}(0) T = E [ τ ] = W f , η ( 0 ) , a = ( T − ϵ ) + a=(T-\epsilon)_+ a = ( T − ϵ ) + for ϵ ≥ 0 \epsilon\ge0 ϵ ≥ 0 , and F = E [ τ 1 { c = 0 } ] F=\mathbb{E}[\tau\mathbf{1}_{\{c=0\}}] F = E [ τ 1 { c = 0 } ] . By (C.7), the precision constraint is E [ τ λ ] ≥ a \mathbb{E}[\tau\lambda]\ge a E [ τ λ ] ≥ a .
Proof of Theorem 3. We establish attainment and the primal–dual identity
Cert f , η ( ϵ ) = min 0 ≤ λ ≤ 1 { E [ c λ ] : E [ τ λ ] ≥ a } = sup u ≥ 0 { u a − E [ ( u τ − c ) + ] } . (70) \operatorname{Cert}_{f,\eta}(\epsilon)
=
\min_{0\le\lambda\le1}\{\mathbb{E}[c\lambda]:\mathbb{E}[\tau\lambda]\ge a\}
=
\sup_{u\ge0}\{ua-\mathbb{E}[(u\tau-c)_+]\}. \tag*{(70)} Cert f , η ( ϵ ) = 0 ≤ λ ≤ 1 min { E [ c λ ] : E [ τ λ ] ≥ a } = u ≥ 0 sup { u a − E [( uτ − c ) + ]} . (70)
Pointwise, ( c − u τ ) λ ≥ − ( u τ − c ) + (c-u\tau)\lambda\ge-(u\tau-c)_+ ( c − uτ ) λ ≥ − ( uτ − c ) + , so every feasible policy obeys
E [ c λ ] ≥ u E [ τ λ ] − E [ ( u τ − c ) + ] ≥ u a − E [ ( u τ − c ) + ] . (71) \mathbb{E}[c\lambda]\ge u\mathbb{E}[\tau\lambda]-\mathbb{E}[(u\tau-c)_+]\ge ua-\mathbb{E}[(u\tau-c)_+]. \tag*{(71)} E [ c λ ] ≥ u E [ τ λ ] − E [( uτ − c ) + ] ≥ u a − E [( uτ − c ) + ] . (71)
The following four cases attain this lower bound, or its limit.
No reduction (a = 0 a=0 a = 0 ). Take λ = 0 \lambda=0 λ = 0 , u = 0 u=0 u = 0 , giving zero cost; this includes T = 0 T=0 T = 0 .
Free reduction (0 < a ≤ F 0<a\le F 0 < a ≤ F ). Take λ = ( a / F ) 1 { c = 0 , τ > 0 } \lambda=(a/F)\mathbf{1}_{\{c=0,\tau>0\}} λ = ( a / F ) 1 { c = 0 , τ > 0 } and u = 0 u=0 u = 0 . The reduction is exactly a a a at zero cost, including when the free component has atoms.
Interior reduction (F < a < T F < a < T F < a < T ). On D = { c > 0 , τ > 0 } D = \{c > 0, \tau> 0\} D = { c > 0 , τ > 0 } , use the finite measure ν ( B ) = ∫ B τ d μ \nu(B) = \int_B \tau\,\mathrm{d}\mu ν ( B ) = ∫ B τ d μ and r = c / τ r = c/\tau r = c / τ . Put b = a − F ∈ ( 0 , ν ( D ) ) b = a - F \in(0, \nu(D)) b = a − F ∈ ( 0 , ν ( D )) and choose
u ∗ = inf { v ≥ 0 : ν ( D ∩ { r ≤ v } ) ≥ b } . (72) u_* = \inf\{v \ge0 : \nu(D \cap\{r \le v\}) \ge b\}. \tag*{(72)} u ∗ = inf { v ≥ 0 : ν ( D ∩ { r ≤ v }) ≥ b } . (72)
Because r ∈ ( 0 , ∞ ) r \in(0,\infty) r ∈ ( 0 , ∞ ) ν \nu ν -almost everywhere, 0 < u ∗ < ∞ 0 < u_* < \infty 0 < u ∗ < ∞ . Continuity of a finite measure along monotone sets gives H : = ν ( D ∩ { r < u ∗ } ) ≤ b ≤ H + J H := \nu(D \cap\{r < u_*\}) \le b \le H + J H := ν ( D ∩ { r < u ∗ }) ≤ b ≤ H + J , where J : = ν ( D ∩ { r = u ∗ } ) J := \nu(D \cap\{r = u_*\}) J := ν ( D ∩ { r = u ∗ }) . Set θ = ( b − H ) / J \theta= (b - H)/J θ = ( b − H ) / J if J > 0 J > 0 J > 0 and θ = 0 \theta= 0 θ = 0 otherwise (then H = b H = b H = b ). The policy
λ ∗ = 1 { c = 0 , τ > 0 } + 1 D ∩ { r < u ∗ } + θ 1 D ∩ { r = u ∗ } (73) \lambda_* = \mathbf{1}_{\{c=0,\tau>0\}} + \mathbf{1}_{D\cap\{r<u_*\}} + \theta\mathbf{1}_{D\cap\{r=u_*\}} \tag*{(73)} λ ∗ = 1 { c = 0 , τ > 0 } + 1 D ∩ { r < u ∗ } + θ 1 D ∩ { r = u ∗ } (73)
achieves reduction F + H + θ J = a F + H + \theta J = a F + H + θ J = a and equality in the pointwise dual bound. Its cost equals u ∗ a − E [ ( u ∗ τ − c ) + ] u_*a - \mathbb{E}[(u_*\tau- c)_+] u ∗ a − E [( u ∗ τ − c ) + ] . It ranks by τ / c \tau/c τ / c with threshold 1 / u ∗ 1/u_* 1/ u ∗ ; fractional ties handle atoms without an atomlessness assumption.
Full reduction (a = T > F a = T > F a = T > F ). Feasibility forces E [ τ ( 1 − λ ) ] = 0 \mathbb{E}[\tau(1-\lambda)] = 0 E [ τ ( 1 − λ )] = 0 , hence λ = 1 \lambda= 1 λ = 1 almost surely on { τ > 0 } \{\tau> 0\} { τ > 0 } . The minimum cost is E [ c 1 { τ > 0 } ] \mathbb{E}[c\mathbf{1}_{\{\tau>0\}}] E [ c 1 { τ > 0 } ] , attained by revealing exactly that set, while
u T − E [ ( u τ − c ) + ] = E [ min { u τ , c } ] ↑ E [ c 1 { τ > 0 } ] uT - \mathbb{E}[(u\tau-c)_+] = \mathbb{E}[\min\{u\tau,c\}] \mathrel{\uparrow} \mathbb{E}[c\mathbf{1}_{\{\tau>0\}}] u T − E [( uτ − c ) + ] = E [ min { uτ , c }] ↑ E [ c 1 { τ > 0 } ]
by monotone convergence. Thus the dual supremum agrees even without a finite maximizer. Its shortfall is E [ ( c − u τ ) + 1 { τ > 0 } ] \mathbb{E}[(c-u\tau)_+\mathbf{1}_{\{\tau>0\}}] E [( c − uτ ) + 1 { τ > 0 } ] , so a finite optimizer exists exactly when c ≤ u τ c \le u\tau c ≤ uτ almost surely on { τ > 0 } \{\tau> 0\} { τ > 0 } for some finite u u u . Full and free reduction correspond to value-density thresholds 0 0 0 and ∞ \infty ∞ . Zero-value locations can always be left unselected; if F = T F = T F = T the whole frontier is free. These cases exhaust 0 ≤ a ≤ T 0 \le a \le T 0 ≤ a ≤ T .
From threshold precision to optimized CVaR precision
Denote the endpoints in Proposition C.3 by ρ α , Γ , λ r e t ( f ) \rho^{\mathrm{ret}}_{\alpha,\Gamma,\lambda}(f) ρ α , Γ , λ ret ( f ) and ρ ‾ α , Γ , λ r e t ( f ) \overline{\rho}^{\mathrm{ret}}_{\alpha,\Gamma,\lambda}(f) ρ α , Γ , λ ret ( f ) , and their difference by I Γ , λ r e t ( f ) I^{\mathrm{ret}}_{\Gamma,\lambda}(f) I Γ , λ ret ( f ) .
Proposition C.5 (Controlling the optimized CVaR interval). For the fixed predictor,
0 ≤ I Γ , λ r e t ( f ) ≤ sup η ∈ [ 0 , 1 ] W f , η ( λ ) = W f , 0 ( λ ) . (74) 0 \le I^{\mathrm{ret}}_{\Gamma,\lambda}(f) \le\sup_{\eta\in[0,1]} W_{f,\eta}(\lambda) = W_{f,0}(\lambda). \tag*{(74)} 0 ≤ I Γ , λ ret ( f ) ≤ η ∈ [ 0 , 1 ] sup W f , η ( λ ) = W f , 0 ( λ ) . (74) Theorem 3 at η = 0 \eta= 0 η = 0 therefore minimizes cost for the sufficient condition W f , 0 ( λ ) ≤ ϵ W_{f,0}(\lambda) \le\epsilon W f , 0 ( λ ) ≤ ϵ .
Proof. If η L \eta_L η L minimizes L L L , then min η U − min η L ≤ U f , η L − L f , η L = W f , η L \min_\eta U - \min_\eta L \le U_{f,\eta_L} - L_{f,\eta_L} = W_{f,\eta_L} min η U − min η L ≤ U f , η L − L f , η L = W f , η L ; nonnegativity follows from ordered endpoints. For v = max y ℓ ( f ( x ) , y ) v = \max_y \ell(f(x), y) v = max y ℓ ( f ( x ) , y ) and w = min y ℓ ( f ( x ) , y ) w = \min_y \ell(f(x), y) w = min y ℓ ( f ( x ) , y ) ,
∣ d f , η ( x ) ∣ = ( v − η ) + − ( w − η ) + = ( v − max { w , η } ) + . |d_{f,\eta}(x)| = (v-\eta)_+ - (w-\eta)_+ = (v-\max\{w,\eta\})_+. ∣ d f , η ( x ) ∣ = ( v − η ) + − ( w − η ) + = ( v − max { w , η } ) + . This decreases in η \eta η and equals v − w v-w v − w at zero. Integration against ( 1 − λ ) ( 1 − π ) s Γ / β ≥ 0 (1-\lambda)(1-\pi)s_\Gamma/\beta\ge0 ( 1 − λ ) ( 1 − π ) s Γ / β ≥ 0 proves the supremum identity.
Direct optimized-width design. Fix an admissible set Λ a d m \Lambda_{\mathrm{adm}} Λ adm of measurable revelation intensities and a specified family of population audit laws { P λ : λ ∈ Λ a d m } \{P^\lambda: \lambda\in\Lambda_{\mathrm{adm}}\} { P λ : λ ∈ Λ adm } sharing one ( X , S , Y ) (X,S,Y) ( X , S , Y ) -marginal. Each P λ P^\lambda P λ has intensity λ \lambda λ and satisfies Definition C.1. Set p A λ ( x ) = P λ ( Y = 1 ∣ X = x , A = 1 ) p_A^\lambda(x) = P^\lambda(Y = 1 \mid X = x, A = 1) p A λ ( x ) = P λ ( Y = 1 ∣ X = x , A = 1 ) , with an extension within the sensitivity bounds on zero-mass regions. Write I Γ , λ ( f ; p A λ ) I_{\Gamma,\lambda}(f;p_A^\lambda) I Γ , λ ( f ; p A λ ) for the width obtained using this kernel in (C.3), retaining the original free residual bounds. For the cost density c c c above and ϵ ≥ 0 \epsilon\ge0 ϵ ≥ 0 , define
Cost f C V a R ( ϵ ) : = inf λ ∈ Λ a d m { E [ c λ ] : I Γ , λ r e t ( f ; p A λ ) ≤ ϵ } . (75) \operatorname{Cost}^{\mathrm{CVaR}}_f(\epsilon) := \inf_{\lambda\in\Lambda_{\mathrm{adm}}}\left\{\mathbb{E}[c\lambda] : I^{\mathrm{ret}}_{\Gamma,\lambda}(f;p_A^\lambda) \le\epsilon\right\}. \tag*{(75)} Cost f CVaR ( ϵ ) := λ ∈ Λ adm inf { E [ c λ ] : I Γ , λ ret ( f ; p A λ ) ≤ ϵ } . (75)
with inf ∅ = + ∞ \inf\varnothing= +\infty inf ∅ = + ∞ . The audited contribution can affect the separate threshold minima even though it cancels from W f , η ( λ ) W_{f,\eta}(\lambda) W f , η ( λ ) .
Strictness of the sufficient bound. Take constant X X X , zero-one loss, f = 0 f = 0 f = 0 , π = p 1 = p A = 1 / 2 \pi= p_1 = p_A = 1/2 π = p 1 = p A = 1/2 , λ = 1 / 2 \lambda= 1/2 λ = 1/2 , Γ = 2 \Gamma= 2 Γ = 2 , and β = 1 / 4 \beta= 1/4 β = 1/4 . The original bounds are [ 1 / 3 , 2 / 3 ] [1/3,2/3] [ 1/3 , 2/3 ] . Under the retained lower and upper endpoint laws, the probabilities r L r_L r L and r U r_U r U of unit loss L f = 1 L_f = 1 L f = 1 are
r L = 1 4 + 1 8 + 1 12 = 11 24 , r U = 1 4 + 1 8 + 1 6 = 13 24 . r_L = \frac{1}{4} + \frac{1}{8} + \frac{1}{12} = \frac{11}{24},
\qquad
r_U = \frac{1}{4} + \frac{1}{8} + \frac{1}{6} = \frac{13}{24}. r L = 4 1 + 8 1 + 12 1 = 24 11 , r U = 4 1 + 8 1 + 6 1 = 24 13 .
Both exceed β \beta β , so (A.5) gives optimized endpoints one and I Γ , λ r e t ( f ) = 0 I^{\mathrm{ret}}_{\Gamma,\lambda}(f) = 0 I Γ , λ ret ( f ) = 0 , although W f , 0 ( λ ) = { ( 1 / 2 ) ( 1 / 2 ) / ( 1 / 4 ) } { ( 2 / 3 − 1 / 3 ) } = 1 / 3 W_{f,0}(\lambda) = \{(1/2)(1/2)/(1/4)\}\{(2/3-1/3)\} = 1/3 W f , 0 ( λ ) = {( 1/2 ) ( 1/2 ) / ( 1/4 )} {( 2/3 − 1/3 )} = 1/3 . Thus the sufficient criterion can require revelation when optimized risk is already identified. Predictor reoptimization and estimated acquisition values require their own statistical or sequential analysis.
Finite-cell computation and operational cost units
For cells j = 1 , … , K j=1,\ldots,K j = 1 , … , K with masses μ j > 0 \mu_j>0 μ j > 0 , write τ j = τ f , η ( j ) \tau_j=\tau_{f,\eta}(j) τ j = τ f , η ( j ) , c j = c ( j ) c_j=c(j) c j = c ( j ) , and set v j = μ j τ j v_j=\mu_j\tau_j v j = μ j τ j , w j = μ j c j w_j=\mu_jc_j w j = μ j c j , T = ∑ j v j T=\sum_j v_j T = ∑ j v j , a = ( T − ϵ ) + a=(T-\epsilon)_+ a = ( T − ϵ ) + , and F = ∑ j : c j = 0 v j F=\sum_{j:c_j=0}v_j F = ∑ j : c j = 0 v j . The exact fractional algorithm is:
If a = 0 a=0 a = 0 , reveal nothing. If 0 < a ≤ F 0<a\le F 0 < a ≤ F , reveal fraction a / F a/F a / F of the zero-cost, positive-value component; both cases cost zero.
Otherwise reveal all useful free cells. Order the positive-cost, positive-value cells by decreasing τ j / c j \tau_j/c_j τ j / c j ; set R k = ∑ j ≤ k v j R_k=\sum_{j\le k}v_j R k = ∑ j ≤ k v j , C k = ∑ j ≤ k w j C_k=\sum_{j\le k}w_j C k = ∑ j ≤ k w j , and R 0 = C 0 = 0 R_0=C_0=0 R 0 = C 0 = 0 .
For b = a − F b=a-F b = a − F , let k = min { j : R j ≥ b } k=\min\{j:R_j\ge b\} k = min { j : R j ≥ b } . Reveal the cells before k k k , fraction θ = ( b − R k − 1 ) / v k \theta=(b-R_{k-1})/v_k θ = ( b − R k − 1 ) / v k of cell k k k , and none after it. Then
Cert f , η ( ϵ ) = C k − 1 + θ w k = C k − 1 + b − R k − 1 τ k / c k . (76) \operatorname{Cert}_{f,\eta}(\epsilon)=C_{k-1}+\theta w_k=C_{k-1}+\frac{b-R_{k-1}}{\tau_k/c_k}. \tag*{(76)} Cert f , η ( ϵ ) = C k − 1 + θ w k = C k − 1 + τ k / c k b − R k − 1 . (76)
Tied ratios may be ordered arbitrarily; full reduction gives θ = 1 \theta=1 θ = 1 in the last useful cell. If there are no positive-cost useful cells, the first step covers every feasible target. This is Theorem 3 ’s policy, so the frontier is piecewise linear. For fixed cell policies and audited laws, the piecewise-affine argument of Proposition B.5 optimizes both retained objectives over { 0 , 1 } ∪ { ℓ ( f ( j ) , y ) : 1 ≤ j ≤ K , y ∈ { 0 , 1 } } \{0,1\}\cup\{\ell(f(j),y):1\le j\le K,\ y\in\{0,1\}\} { 0 , 1 } ∪ { ℓ ( f ( j ) , y ) : 1 ≤ j ≤ K , y ∈ { 0 , 1 }} .
If a hidden label costs κ ( x ) ≥ 0 \kappa(x)\ge0 κ ( x ) ≥ 0 , the target-population cost density and operational value density are
c ( x ) = ( 1 − π ( x ) ) κ ( x ) , E [ c λ ] = E [ ( 1 − π ) λ κ ] , (77) c(x)=(1-\pi(x))\kappa(x),\qquad\mathbb{E}[c\lambda]=\mathbb{E}[(1-\pi)\lambda\kappa], \tag*{(77)} c ( x ) = ( 1 − π ( x )) κ ( x ) , E [ c λ ] = E [( 1 − π ) λκ ] , (77)
τ f , η ( x ) c ( x ) = s Γ ( x ) ∣ d f , η ( x ) ∣ β κ ( x ) when ( 1 − π ) κ > 0. (78) \frac{\tau_{f,\eta}(x)}{c(x)}=\frac{s_{\Gamma}(x)|d_{f,\eta}(x)|}{\beta\kappa(x)}
\qquad\text{when }(1-\pi)\kappa>0. \tag*{(78)} c ( x ) τ f , η ( x ) = β κ ( x ) s Γ ( x ) ∣ d f , η ( x ) ∣ when ( 1 − π ) κ > 0. (78)
The hidden-mass factor cancels; unit label cost ranks already-hidden cases by s Γ ∣ d ∣ / β s_{\Gamma}|d|/\beta s Γ ∣ d ∣/ β . Cell masses likewise cancel from v j / w j v_j/w_j v j / w j . The algorithm allocates divisible population mass. Whole-individual batches add integer constraints and sampling variation. Sequential retraining changes losses and acquisition values; confidence guarantees across adaptive rounds additionally require control of newly revealed contributions, acquisition history, and stopping.
Experimental Protocols and Results
This appendix specifies the experiments in § 6 : finite-cell identification and estimation, historical applicant assessment, controlled masking, and targeted revelation. All results use saved outputs; Appendix § 15 .§ 7 identifies the accompanying provenance.
Evaluation targets, interval construction, and aggregation
Write β = 1 − α \beta=1-\alpha β = 1 − α , ρ L = ρ α , Γ \rho^L=\rho_{\alpha,\Gamma} ρ L = ρ α , Γ , ρ U = ρ ‾ α , Γ \rho^U=\overline{\rho}_{\alpha,\Gamma} ρ U = ρ α , Γ , and I Γ = ρ U − ρ L I_{\Gamma}=\rho^U-\rho^L I Γ = ρ U − ρ L . Endpoints minimize their respective threshold objectives. Finite-support minima are found exactly at 0 0 0 , 1 1 1 , and all label-specific loss breakpoints, including when estimated hinge coefficients are signed. Sampling and lending studies use Brier loss ( f − y ) 2 (f-y)^2 ( f − y ) 2 ; population sweeps and the synthetic frontier use absolute loss.
Table 1.
For fixed f , η f,\eta f , η , the revelation target is W f , η ( λ ) = E [ ( 1 − λ ) τ f , η ] W_{f,\eta}(\lambda)=\mathbb{E}[(1-\lambda)\tau_{f,\eta}] W f , η ( λ ) = E [( 1 − λ ) τ f , η ] , where τ f , η = ( 1 − π ) s Γ ∣ d f , η ∣ / β \tau_{f,\eta}=(1-\pi)s_{\Gamma}|d_{f,\eta}|/\beta τ f , η = ( 1 − π ) s Γ ∣ d f , η ∣/ β . It differs from the optimized width, whose endpoints are minimized separately Appendix § 14 .§ 4 . Oracle containment, masking-reference coverage, and sequential plug-in precision have different calibration premises. Fixed-audit reductions normalize within each run before aggregation; sequential widths aggregate the recorded widths. Ratios of separately aggregated values need not equal aggregated relative reductions.
Finite-cell identification and learned-endpoint accuracy
Generator and estimators. Each of 60 simulated populations has covariate support X 0 \mathcal{X}_0 X 0 consisting of 30 cells, with Dirichlet( 1 , … , 1 ) (1,\ldots,1) ( 1 , … , 1 ) masses, independent π ∼ U [ .2 , .95 ] \pi\sim U[.2,.95] π ∼ U [ .2 , .95 ] and p 1 ∼ U [ .05 , .95 ] p_1 \sim U[.05,.95] p 1 ∼ U [ .05 , .95 ] , and p 0 ∣ p 1 ∼ U [ q − ( p 1 ) , q + ( p 1 ) ] p_0 \mid p_1 \sim U[q_-(p_1),q_+(p_1)] p 0 ∣ p 1 ∼ U [ q − ( p 1 ) , q + ( p 1 )] at Γ = 2 \Gamma=2 Γ = 2 .
Fix f = clip [ 0 , 1 ] ( p 1 + N ( 0 , .15 2 ) ) f=\operatorname{clip}_{[0,1]}(p_1+N(0,.15^2)) f = clip [ 0 , 1 ] ( p 1 + N ( 0 , .1 5 2 )) within each world, use Brier loss and α = .75 \alpha=.75 α = .75 , and compute exact endpoints and generating-law CVaR on T f = { 0 , 1 } ∪ { ( f ( x ) ) 2 , ( 1 − f ( x ) ) 2 : x ∈ X 0 } T_f=\{0,1\}\cup\{(f(x))^2,(1-f(x))^2:x\in\mathcal{X}_0\} T f = { 0 , 1 } ∪ {( f ( x ) ) 2 , ( 1 − f ( x ) ) 2 : x ∈ X 0 } . For each n ∈ { 1000 , 3000 , 10000 , 30000 , 100000 } n\in\{1000,3000,10000,30000,100000\} n ∈ { 1000 , 3000 , 10000 , 30000 , 100000 } , draw 80 samples per world (4,800 replications). The oracle uses known p 1 p_1 p 1 and the bounded score of Appendix § 13 .§ 5 ; its endpoints are expanded by
r o r = 1 β log ( 4 ∣ T f ∣ / δ ) 2 n , δ = .05 , r_{\mathrm{or}}=\frac{1}{\beta}\sqrt{\frac{\log(4|T_f|/\delta)}{2n}},\qquad\delta=.05, r or = β 1 2 n log ( 4∣ T f ∣/ δ ) , δ = .05 ,
then clipped to [ 0 , 1 ] [0,1] [ 0 , 1 ] . Proposition B.6 gives whole-interval containment with probability at least .95 for this fixed predictor. Saved generating-law CVaR coverage also equals one at every reported sample size.
On the same records, randomly assign each observation to one of two folds. In a given training cell, let N 11 N_{11} N 11 count records with ( S , Y ) = ( 1 , 1 ) (S,Y)=(1,1) ( S , Y ) = ( 1 , 1 ) , let N 10 N_{10} N 10 count records with ( S , Y ) = ( 1 , 0 ) (S,Y)=(1,0) ( S , Y ) = ( 1 , 0 ) , and let N N N be the total number of training records in that cell. Estimate
p ^ 1 = N 11 + 1 / 2 N 11 + N 10 + 1 , π ^ = N 11 + N 10 N , \widehat{p}_1=\frac{N_{11}+1/2}{N_{11}+N_{10}+1},\qquad\widehat{\pi}=\frac{N_{11}+N_{10}}{N}, p 1 = N 11 + N 10 + 1 N 11 + 1/2 , π = N N 11 + N 10 ,
clipped to [ .01 , .99 ] [.01,.99] [ .01 , .99 ] and [ .1 , 1 ] [.1,1] [ .1 , 1 ] , respectively; any empty training cell receives propensity 1 / 2 1/2 1/2 . Evaluate the locally robust score using the other fold’s fits. The resulting RMSEs measure endpoint accuracy. Learned-nuisance coverage requires a separate calibration argument.
Observed-tail scaling. For the d = 16 d=16 d = 16 tail-cell construction of Appendix § 13 .§ 6 , use 240 replications at each of 54 settings: n ∈ { 600 , 900 , 1400 , 2200 , 3500 , 5500 } n\in\{600,900,1400,2200,3500,5500\} n ∈ { 600 , 900 , 1400 , 2200 , 3500 , 5500 } , β ∈ { .10 , .15 , .25 } \beta\in\{.10,.15,.25\} β ∈ { .10 , .15 , .25 } , and π 0 ∈ { .20 , .35 , .55 } \pi_0\in\{.20,.35,.55\} π 0 ∈ { .20 , .35 , .55 } . Set γ = min { .20 , .16 d / ( n β π 0 ) } \gamma=\min\{.20,.16\sqrt{d/(n\beta\pi_0)}\} γ = min { .20 , .16 d / ( n β π 0 ) } and recover each cell’s sign by majority vote among revealed tail labels, assigning ties the positive sign. The metric is 2 γ 2\gamma 2 γ times the incorrectly recovered fraction, equal to excess CVaR for this construction. Its least-squares guide against 1 / n π 0 β 1/\sqrt{n\pi_0\beta} 1/ n π 0 β has slope .5030 .5030 .5030 , intercept − .0018 -.0018 − .0018 , and R 2 ≃ .998 R^2\simeq.998 R 2 ≃ .998 ; this illustration does not establish a general matching upper bound for Appendix § 13 .§ 6 . Population sweeps. The five-region absolute-loss design has region masses μ j = P ( X = j ) \mu_j = P(X = j) μ j = P ( X = j ) and full-data outcome probabilities p f u l l , j = P ( Y = 1 ∣ X = j ) p_{\mathrm{full},j} = P(Y = 1 \mid X = j) p full , j = P ( Y = 1 ∣ X = j ) , for j = 1 , … , 5 j = 1, \ldots, 5 j = 1 , … , 5 . It fixes
Table 1 (PDF p. 30). Repeated-sampling decomposition over 4,800 replications per sample size. Widths and oracle radii are means; coverage is the proportion of oracle outer intervals containing both exact endpoints. RMSE uses two-fold learned-nuisance estimates on those same sampled records.
μ = ( .45 , .15 , .15 , .10 , .15 ) , p f u l l = ( .04 , .75 , .10 , .40 , .50 ) , \mu= (.45, .15, .15, .10, .15), \quad p_{\mathrm{full}} = (.04, .75, .10, .40, .50), μ = ( .45 , .15 , .15 , .10 , .15 ) , p full = ( .04 , .75 , .10 , .40 , .50 ) ,
s 1 = ( .80 , .03 , .80 , .80 , .80 ) , s 0 = ( .80 , .60 , .80 , .80 , .80 ) , s_1 = (.80, .03, .80, .80, .80), \quad s_0 = (.80, .60, .80, .80, .80), s 1 = ( .80 , .03 , .80 , .80 , .80 ) , s 0 = ( .80 , .60 , .80 , .80 , .80 ) ,
f = ( .05 , .25 , .10 , .40 , .50 ) , s y = P ( S = 1 ∣ X , Y = y ) . f = (.05, .25, .10, .40, .50), \quad s_y = P(S = 1 \mid X, Y = y). f = ( .05 , .25 , .10 , .40 , .50 ) , s y = P ( S = 1 ∣ X , Y = y ) .
Exact finite-support optimization evaluates 60 tail levels on [ .55 , .95 ] [.55, .95] [ .55 , .95 ] at Γ = 50 \Gamma= 50 Γ = 50 , and 60 common historical observation probabilities on [ .05 , 1 ] [.05, 1] [ .05 , 1 ] at α = .85 \alpha= .85 α = .85 , Γ = 50 \Gamma= 50 Γ = 50 , holding the original p 1 p_1 p 1 fixed in the latter sweep. An archived sweep uses 60 log-spaced Γ ∈ [ 1 , 100 ] \Gamma\in[1, 100] Γ ∈ [ 1 , 100 ] at α = .85 \alpha= .85 α = .85 ; values below the generating sensitivity shift do not claim generating-law coverage.
Historical LendingClub applicant assessment
Cohort and shared preprocessing. From the public 2007–2018 accepted/rejected files [21 ] , retain loans issued and applications rejected in 2012. Mature issued outcomes are charged-off/default (Y = 1 Y = 1 Y = 1 ) or fully paid (Y = 0 Y = 0 Y = 0 ), including corresponding former-credit-policy statuses; exclude nonmature outcomes. The saved cohort has 390,644 applications: 53,367 issued loans (8,644 defaults) and 337,277 rejections. Repayment under issuance is unobserved for rejected applications.
Features are recorded amount, debt-to-income ratio (DTI), FICO credit-score interval midpoint or recorded risk score, employment years, state, three-digit ZIP prefix, and normalized purpose; exclude accepted-only pricing/repayment fields, policy code, and calendar month. Within each training sample, median-impute numeric fields with missingness indicators, most-frequent-impute categorical fields, and one-hot encode categories with minimum frequency 25. Standardize numeric fields for logistic regression. Preprocessing, logistic regression, and stratified splitting use scikit-learn [27 ] .
Eligibility, measurement, and extrapolation. The 2012 FICO minimum was 660 [22 ] . Source checks identify 180,903 below-660 applications (46.31%) and 23,097 missing standardized scores, with no issued loans in either group. All 53,367 issued loans lie among the remaining 186,644 records at or above 660. This structural restriction does not make FICO eligibility sufficient for joint covariate support or other underwriting requirements.
Recorded fields also differ [28 ] : rejected applications record FICO, whereas issued loans use an origination-score interval midpoint, without common coarsening; issued loan_amnt can reflect credit-department reductions, whereas rejected Amount Requested is the requested total. Issued purpose categories and rejected free-text titles are regex-normalized; dates refer to funding month and application date, respectively. Thus these are not identically measured predecision covariates.
In training fold − k -k − k , fit the issued-outcome regression r ^ − k \hat r_{-k} r ^ − k using the boosting settings below, evaluate every held-out record, and specify
r i = clip [ .01 , .99 ] r ^ − k ( i ) ( X i ) , q i ∈ [ q − ( r i ) , q + ( r i ) ] . (79) r_i = \operatorname{clip}_{[.01,.99]} \hat r_{-k(i)}(X_i), \qquad q_i \in[q_{-}(r_i), q_{+}(r_i)]. \tag*{(79)} r i = clip [ .01 , .99 ] r ^ − k ( i ) ( X i ) , q i ∈ [ q − ( r i ) , q + ( r i )] . (79)
On unsupported recorded covariates, r i r_i r i is an extrapolated reference, not an identified selected-label probability. Objectives use [5 ] with p = r i p = r_i p = r i , clipped fitted propensity, and the selected-observation residual correction. These residual-adjusted model-based estimates do not satisfy the population-overlap premise of Theorem 2 on the full cohort.
Figure 5. Observation scarcity and sensitivity shape tail-risk precision. (a) Recorded simulations of the Appendix B.6 learning construction, with a least-squares guide against ( n π 0 β ) − 1 / 2 (n\pi_0\beta)^{-1/2} ( n π 0 β ) − 1/2 . Here n π 0 β n\pi_0\beta n π 0 β is the expected observed-tail sample count. (b–c) Exact identification widths for the fixed five-region design as the tail level or historical observation probability varies. Width can vary nonmonotonically with α \alpha α . Lines join recorded evaluations.
Models, cross-fitting, and optimization. Candidates use standard classification objectives: ℓ 2 \ell_2 ℓ 2 logistic regression (C = 1 C = 1 C = 1 , L-BFGS, 1000-iteration limit), default-weighted logistic regression (default class weight 2), and XGBoost [29 ] (450 trees, depth 5, learning rate .04, minimum child weight 10, row/column subsampling .85, ℓ 2 \ell_2 ℓ 2 penalty 2, histogram trees, recorded CUDA configuration). Both nuisance classifiers use the boosting settings with separately fitted preprocessing and distinct seeds: p 1 p_1 p 1 uses revealed outcomes and π \pi π predicts S S S from all training covariates. Shuffled five-fold cross-fitting is stratified by S S S ; clip p ^ 1 \widehat{p}_1 p 1 to [ .01 , .99 ] [.01, .99] [ .01 , .99 ] and π ^ \widehat{\pi} π to [ .02 , .98 ] [.02, .98] [ .02 , .98 ] . Evaluate Brier loss at α ∈ { .90 , .95 } \alpha\in\{.90, .95\} α ∈ { .90 , .95 } and Γ ∈ { 1 , 1.25 , 1.5 , 2 , 3 , 5 , 8 , 12 } \Gamma\in\{1, 1.25, 1.5, 2, 3, 5, 8, 12\} Γ ∈ { 1 , 1.25 , 1.5 , 2 , 3 , 5 , 8 , 12 } .
The corrected objective can be nonconvex because hinge coefficients can be signed. Enumerate 0, 1, and both label-specific losses, f ( X i ) 2 f(X_i)^2 f ( X i ) 2 and ( 1 − f ( X i ) ) 2 (1-f(X_i))^2 ( 1 − f ( X i ) ) 2 , at every observed covariate X i X_i X i , including records whose actual outcomes are hidden. Deduplicate and sort increasingly; among exactly equal computed minima, choose the smallest threshold, without numerical tie tolerance. Clip endpoints to [ 0 , 1 ] [0,1] [ 0 , 1 ] ; if they cross, replace both by their midpoint. These clipping and crossing conventions are computational operations and do not supply statistical coverage.
Fold enlargement and diagnostics. At each endpoint’s selected threshold, compute five fold-specific objectives and divide their sample standard deviation by 5 \sqrt{5} 5 to obtain an empirical standard error. Let t .975 , 4 t_{.975,4} t .975 , 4 be the .975 quantile of a Student-t t t distribution with four degrees of freedom. Subtract this multiplier times the lower-endpoint standard error from the raw lower endpoint and add the corresponding upper-endpoint quantity to the raw upper endpoint, before clipping or crossing repair. Then clip both limits to [ 0 , 1 ] [0,1] [ 0 , 1 ] and order them by minimum and maximum. This empirical diagnostic uses dependent fold-specific quantities because their training samples overlap; calibrated coverage requires the conditions and radius of Theorem 2 .
Historical propensity AUC is .9944 and Brier score .0201; raw propensities are below .02 for 75.26% and above .98 for 1.80% of applicants, and inverse clipped propensities are at most 50, indicating substantial extrapolation. Subgroups reuse full-cohort cross-fitted predictions and require at least max { 250 , ⌊ .002 N ⌋ } \max\{250,\lfloor.002N\rfloor\} max { 250 , ⌊ .002 N ⌋} applications and 50 issued loans. At Γ = 2 \Gamma=2 Γ = 2 , α = .95 \alpha=.95 α = .95 , boosting width is .080 for DTI above 30% (2.7% issued) and .032 for FICO 700–739 (30.3% issued).
Table 2. Historical applicants: accepted-only CVaR and model-based applicant-level sensitivity intervals. Values other than the exact constant reference are estimates. At Γ = 1 \Gamma=1 Γ = 1 the reference model fixes the hidden probabilities. Default-weighted denotes logistic regression with default class weight 2.
For f ≡ 1 / 2 f \equiv1/2 f ≡ 1/2 , both label-specific Brier losses equal 1 / 4 1/4 1/4 . Mean loss and every CVaR endpoint therefore equal 1 / 4 1/4 1/4 , and d f , η = T I V = 0 d_{f,\eta}=\mathrm{TIV}=0 d f , η = TIV = 0 for every threshold. This row provides an exact predictor reference; the acquisition policies compare audits for a fixed or sequentially refitted boosting predictor.
Off-grid breakdown. For an accepted-only preferred f f f and comparator g g g , set G ( Γ ) = ρ ^ Γ L ( g ) − ρ ^ Γ L ( f ) G(\Gamma)=\widehat{\rho}^{L}_{\Gamma}(g)-\widehat{\rho}^{L}_{\Gamma}(f) G ( Γ ) = ρ Γ L ( g ) − ρ Γ L ( f ) . At the first grid value Γ + \Gamma_{+} Γ + with G ( Γ + ) ≤ 0 G(\Gamma_{+})\leq0 G ( Γ + ) ≤ 0 , if its predecessor Γ − \Gamma_{-} Γ − has positive gap, interpolate:
log Γ ^ b r = log Γ − + G ( Γ − ) G ( Γ − ) − G ( Γ + ) ( log Γ + − log Γ − ) . \log\widehat{\Gamma}_{\mathrm{br}}=\log\Gamma_{-}+\frac{G(\Gamma_{-})}{G(\Gamma_{-})-G(\Gamma_{+})}\left(\log\Gamma_{+}-\log\Gamma_{-}\right). log Γ br = log Γ − + G ( Γ − ) − G ( Γ + ) G ( Γ − ) ( log Γ + − log Γ − ) .
These values interpolate the recorded grid. Boosting versus logistic gives approximate breakdown 2.24 at α = .95 \alpha=.95 α = .95 , with first failed grid value 3, and 1.40 at α = .90 \alpha=.90 α = .90 , with first failed value 1.5. Default-weighted versus logistic gives approximately 1.01 at α = .90 \alpha=.90 α = .90 , with first failed value 1.25. The other three accepted-only orderings fail already at Γ = 1 \Gamma=1 Γ = 1 . A comparison with no crossing through 12 is recorded as right-censored above the grid.
Controlled masking: empirical coverage and model comparisons
To evaluate coverage against known full-outcome risks, we use the 53,367 mature issued loans as the target population and independently draw 20 masks from
logit P ( S = 1 ∣ X , Y ) = c + g ( X ) − Y log 3 , \operatorname{logit}P(S=1\mid X,Y)=c+g(X)-Y\log3, logit P ( S = 1 ∣ X , Y ) = c + g ( X ) − Y log 3 ,
g ( X ) = clip [ − 3 , 3 ] { .65 z ( s c o r e ) − .35 z ( D T I ) g(X)=\operatorname{clip}_{[-3,3]}\left\{.65z(\mathrm{score})-.35z(\mathrm{DTI})\right. g ( X ) = clip [ − 3 , 3 ] { .65 z ( score ) − .35 z ( DTI )
+ .15 z ( log ( 1 + a m o u n t ) ) } . \left.{}+.15z(\log(1+\mathrm{amount}))\right\}. + .15 z ( log ( 1 + amount )) } .
Each z z z standardizes the median-imputed pseudo-population feature. Solve for expected revealed share .45 (c = − .0423744 c=-.0423744 c = − .0423744 ). Bayes’ rule gives odds ( p 0 ) / odds ( p 1 ) = 3 \operatorname{odds}(p_{0})/\operatorname{odds}(p_{1})=3 odds ( p 0 ) / odds ( p 1 ) = 3 , hence Γ 0 = 3 \Gamma_{0}=3 Γ 0 = 3 ; boundary outcome probabilities coincide across hidden/revealed groups as in Appendix § 12 .§ 1 . This validates artificial masking of mature issued loans, not outcomes for historically rejected applicants.
All three predictors share shuffled five-fold splits stratified by imposed S S S ; fitting, clipping, threshold search, and fold enlargement follow Appendix § 15 .§ 3 . Reference CVaR uses every known outcome with that predictor’s cross-fitted predictions. Table 3 counts intervals containing this full-outcome reference risk. Table 4 checks strict separation of plug-in intervals against reference risk orderings.
Table 3. Controlled masking at Γ = Γ 0 = 3 \Gamma= \Gamma_0 = 3 Γ = Γ 0 = 3 . The middle columns count intervals covering the full pseudo-population reference CVaR over 20 masks; the last two columns are mean widths. Fold enlargement follows the empirical diagnostic in Appendix § 15 .§ 3 .
Table 4. Model comparisons under controlled masking. Strict separation of plug-in intervals declares an ordering, whose correctness is checked against known-outcome reference CVaRs. Counts give resolved pairs, incorrect declared orderings, and unresolved pairs. Each tail level includes three unordered predictor pairs in each of 20 masks.
Figure 6. Complete applicant-level sensitivity analysis. All three predictors at a l p h a = .90 \\alpha= .90 a l p ha = .90 and .95 .95 .95 over eight recorded sensitivity levels. Blue boundaries and shading show estimated sensitivity endpoints under the applicant regression extension of Appendix § 15 .§ 3 . Gray dashed lines show accepted-only CVaR. Horizontal axes are logarithmic and vertical limits are shared. Default-weighted denotes default-weighted logistic regression.
Three unordered predictor pairs per mask and 20 masks give 60 comparisons at each tail level, or 120 across both levels.
Figure 7 retains the complete assumed-G a m m a \\Gamma G amma grid; plug-in coverage is zero at every tested G a m m a l e 2 \\Gamma\\le2 G amma l e 2 . Main-figure pooling averages 20 mask-level means over three predictors, giving 60 equally weighted predictor–mask rows from 20 independent masks. Fold enlargement is an empirical calibration diagnostic; the class-wide calibrated statement is Theorem 2 .
Figure 7. Empirical coverage by predictor and tail level. Each proportion uses 20 controlled masks. Blue circles show fold-enlarged intervals and hollow gray squares plug-in intervals; the vertical reference marks the masking mechanism’s sensitivity Γ 0 = 3 \Gamma_0 = 3 Γ 0 = 3 . Predictors share the same 20 masks, so the predictor-specific proportions are dependent. Default-weighted denotes default-weighted logistic regression.
Synthetic fixed-threshold cost–precision frontier
We next evaluate the fixed-threshold cost–precision frontier of Theorem 3 in synthetic populations with known outcome probabilities and acquisition costs.
Generator. Use 120 worlds of 120 cells, α = .85 \alpha= .85 α = .85 , Γ = 12 \Gamma= 12 Γ = 12 , absolute loss, f = p 1 f = p_1 f = p 1 , and η = .5 \eta= .5 η = .5 . Masses are Dirichlet( .75 , … , .75 ) (.75,\ldots,.75) ( .75 , … , .75 ) . Independently draw z j , e j , e j o u t , e j i n ∼ N ( 0 , 1 ) z_j,e_j,e_j^{\mathrm{out}},e_j^{\mathrm{in}} \sim N(0,1) z j , e j , e j out , e j in ∼ N ( 0 , 1 ) across cells/noise types. Hidden-tail cells satisfy z j + .35 e j z_j + .35e_j z j + .35 e j at or above its empirical .65 .65 .65 quantile (NumPy default linear interpolation). Outside/inside these cells, respectively, set
p 1 j = sigmoid ( − 1.35 + .75 z j + .45 e j o u t ) , p 1 j = sigmoid ( − 2.05 + .35 z j + .30 e j i n ) , p_{1j} = \operatorname{sigmoid}(-1.35 + .75z_j + .45e_j^{\mathrm{out}}), \qquad p_{1j} = \operatorname{sigmoid}(-2.05 + .35z_j + .30e_j^{\mathrm{in}}), p 1 j = sigmoid ( − 1.35 + .75 z j + .45 e j out ) , p 1 j = sigmoid ( − 2.05 + .35 z j + .30 e j in ) ,
clipped to [ .04 , .72 ] [.04,.72] [ .04 , .72 ] , and draw π j ∼ U [ .62 , .95 ] / U [ .02 , .16 ] \pi_j \sim U[.62,.95]/U[.02,.16] π j ∼ U [ .62 , .95 ] / U [ .02 , .16 ] . Add to the log-odds of p 1 j p_{1j} p 1 j a shift N ( 0 , .18 2 ) / U [ .8 log 10 , log 10 ] N(0,.18^2)/U[.8\log10,\log10] N ( 0 , .1 8 2 ) / U [ .8 log 10 , log 10 ] to obtain p 0 j p_{0j} p 0 j , clipped to [ .02 , .98 ] [.02,.98] [ .02 , .98 ] . Costs are c j = U j + B j V j c_j = U_j + B_jV_j c j = U j + B j V j with U j ∼ U [ .45 , 1.75 ] U_j \sim U[.45,1.75] U j ∼ U [ .45 , 1.75 ] , B j ∼ Bernoulli ( .20 ) B_j \sim\operatorname{Bernoulli}(.20) B j ∼ Bernoulli ( .20 ) , and V j ∼ U [ .75 , 1.75 ] V_j \sim U[.75,1.75] V j ∼ U [ .75 , 1.75 ] . Primitive draws are mutually independent and independent of masses; hidden-cell membership induces the specified dependence of π j \pi_j π j and log-odds shifts on the normal scores.
The generated p 0 j p_{0j} p 0 j specifies the hidden-outcome distribution in cell j j j . The fixed-threshold frontier follows from W f , η ( λ ) = ∑ j μ j ( 1 − λ j ) τ f , η ( j ) W_{f,\eta}(\lambda) = \sum_j \mu_j(1-\lambda_j)\tau_{f,\eta}(j) W f , η ( λ ) = ∑ j μ j ( 1 − λ j ) τ f , η ( j ) and cost ∑ j μ j c j λ j \sum_j \mu_jc_j\lambda_j ∑ j μ j c j λ j . The frontier calculation uses p 1 , π , f , η , Γ , μ , c p_1,\pi,f,\eta,\Gamma,\mu,c p 1 , π , f , η , Γ , μ , c ; the generated p 0 p_0 p 0 does not enter these formulas.
Allocations and equivalent baselines. Here c j c_j c j is population cost density: allocations cost ∑ j μ j c j λ j \sum_j\mu_jc_j\lambda_j ∑ j μ j c j λ j , with full cost ∑ j μ j c j \sum_j\mu_jc_j ∑ j μ j c j . Appendix § 14 .§ 5 ’s fractional-knapsack solution uses useful zero-cost cells first, then decreasing τ j / c j \tau_j/c_j τ j / c j on positive-cost/value cells (this generator has positive costs). A fractional final cell implements 51 budgets from zero to full cost in .02 .02 .02 increments. TIV/cost and the exact frontier share this ordering and all saved coordinates.
Seven baselines rank by raw independent U [ 0 , 1 ] U[0,1] U [ 0 , 1 ] random score, cell mass, negative revelation probability, selected-label expected absolute loss, p 1 ( 1 − p 1 ) p_1(1-p_1) p 1 ( 1 − p 1 ) , sensitivity width, or absolute tail-excess contrast ∣ d f , η ∣ |d_{f,\eta}| ∣ d f , η ∣ . Only TIV/frontier divide their scores by cost. Each world’s random scores are independent of generator draws and reused across budgets. Decreasing NumPy default sorting (argsort(score)[::-1]) imposes no stable secondary tie rule, including clipping-induced ties. For v j = p 1 j ( 1 − p 1 j ) v_j = p_{1j}(1-p_{1j}) v j = p 1 j ( 1 − p 1 j ) , predicted loss is 2 v j 2v_j 2 v j , predictive uncertainty is v j v_j v j , and
s Γ ( p 1 j ) = ( Γ 2 − 1 ) v j Γ + ( Γ − 1 ) 2 v j . (80) s_\Gamma(p_{1j}) = \frac{(\Gamma^2-1)v_j}{\Gamma+ (\Gamma-1)^2v_j}. \tag*{(80)} s Γ ( p 1 j ) = Γ + ( Γ − 1 ) 2 v j ( Γ 2 − 1 ) v j . (80)
The last expression increases strictly in v j v_j v j for Γ > 1 \Gamma> 1 Γ > 1 , so these three baselines share a mathematical weak ordering and their saved curves coincide.
Normalize remaining ambiguity by its initial value within each world before taking medians/IQR. Cost-to-target is the first recorded budget reaching 25 % 25\% 25% of initial ambiguity, divided by TIV’s corresponding cost in that world; there is no interpolation between budgets. Normalization requires positive initial ambiguity, full cost, and TIV target cost. This measures divisible population revelation at fixed f , η , Γ f,\eta,\Gamma f , η , Γ (Theorem 3 ).
Figure 8. The exact revelation frontier and the cost of alternative policies. Across 120 finite-cell worlds, (a) median remaining fixed-threshold ambiguity, normalized within each world; blue shading is the TIV interquartile range and open markers identify the coincident exact frontier. (b) All seven baselines: median cost of reaching 25% of the initial gap relative to TIV in the same world, with interquartile bars. Costs use the recorded 2%-spaced grid.
Fixed and sequential audits of known repayments
Both protocols use boosting, α = .95 \alpha= .95 α = .95 , Γ = 3 \Gamma= 3 Γ = 3 , the 20 controlled masks, and unit cost per audited label. Let N h i d d e n , i n i t i a l f i x N_{\mathrm{hidden,initial}}^{\mathrm{fix}} N hidden , initial fix count initially hidden cases in the fixed protocol’s original pool, and let N h i d d e n , i n i t i a l s e q N_{\mathrm{hidden,initial}}^{\mathrm{seq}} N hidden , initial seq count initially hidden cases in the sequential protocol’s 80% training pool. Each protocol acquires eight batches of ⌈ .02 N h i d d e n , i n i t i a l p r o t o c o l ⌉ \lceil.02N_{\mathrm{hidden,initial}}^{\mathrm{protocol}}\rceil ⌈ .02 N hidden , initial protocol ⌉ labels, capped by the remaining eligible pool. Its budget fraction is the cumulative acquired count divided by that same initial hidden count. Among candidates already known to be hidden, operational TIV is τ ^ / ( 1 − π ^ ) = s ^ Γ ∣ d f , η ∣ / β \hat{\tau}/(1-\hat{\pi}) = \hat{s}_{\Gamma}|d_{f,\eta}|/\beta τ ^ / ( 1 − π ^ ) = s ^ Γ ∣ d f , η ∣/ β , corresponding to c = 1 − π c = 1-\pi c = 1 − π in Appendix § 14 .§ 5 .
Table 1 (PDF p. 37).
The fixed protocol takes its upper threshold from the empirical upper-endpoint objective in (6) , built from the score (5) , using the initial masking predictions of Appendix § 15 .§ 4 and the breakpoint/tie rule of Appendix § 15 .§ 3 . It freezes those predictor and nuisance predictions, as well as this threshold, for every acquisition and fixed-gap calculation.
Retained objective. Historical p 1 p_1 p 1 is fitted on initially revealed training outcomes, clipped as in Appendix § 15 .§ 3 , and frozen. For training size n n n , let A i t ≤ 1 − S i A_{it} \le1-S_i A i t ≤ 1 − S i indicate audit by round t t t , f t f_t f t be the refitted predictor, and p ^ i \widehat{p}_i p i the frozen fit. The retained plug-in objective is
Q ^ t σ ( η ) = η + 1 n β ∑ i = 1 n [ S i a f t , η Y i ( X i ) + A i t a f t , η Y i ( X i ) + ( 1 − S i − A i t ) b f t , η σ ( X i ; p ^ i ) ] , σ ∈ { L , U } . (81) \widehat{Q}_t^\sigma(\eta) = \eta+ \frac{1}{n\beta}\sum_{i=1}^{n}\left[S_i a_{f_t,\eta}^{Y_i}(X_i) + A_{it}a_{f_t,\eta}^{Y_i}(X_i) + (1-S_i-A_{it})b_{f_t,\eta}^{\sigma}(X_i;\widehat{p}_i)\right], \qquad\sigma\in\{L,U\}. \tag*{(81)} Q t σ ( η ) = η + n β 1 i = 1 ∑ n [ S i a f t , η Y i ( X i ) + A i t a f t , η Y i ( X i ) + ( 1 − S i − A i t ) b f t , η σ ( X i ; p i ) ] , σ ∈ { L , U } . (81)
Known outcomes enter directly; still-hidden outcomes retain historical sensitivity envelopes. Each endpoint uses breakpoint minimization, the threshold tie rule, clipping, and midpoint collapse of Appendix § 15 .§ 3 . At full revelation A i t = 1 − S i A_{it}=1-S_i A i t = 1 − S i , both objectives equal η + ( n β ) − 1 ∑ i a f t , η Y i ( X i ) \eta+(n\beta)^{-1}\sum_i a_{f_t,\eta}^{Y_i}(X_i) η + ( n β ) − 1 ∑ i a f t , η Y i ( X i ) and both endpoints equal full-training-data empirical CVaR; a regression test verifies this identity. These training-prediction widths and held-out predictive risks are descriptive trajectories, without fold enlargement or adaptive confidence guarantees.
Acquisition rules. Among still-hidden candidates, let p ^ k n o w n \widehat{p}_{\mathrm{known}} p known denote the fitted outcome probability from currently known labels, and let s ^ Γ = q + ( p ^ 1 ) − q − ( p ^ 1 ) \widehat{s}_{\Gamma}=q_{+}(\widehat{p}_1)-q_{-}(\widehat{p}_1) s Γ = q + ( p 1 ) − q − ( p 1 ) be the sensitivity width from the frozen historical outcome fit. In the sequential protocol, let η ^ U \widehat{\eta}_U η U be the current upper-endpoint minimizing threshold of (D.3). The six acquisition scores are TIV per label cost s ^ Γ ∣ d f , η ^ U ∣ / ( β ^ κ ) \widehat{s}_{\Gamma}|d_{f,\widehat{\eta}_U}|/(\widehat{\beta}\kappa) s Γ ∣ d f , η U ∣/ ( β κ ) , with κ = 1 \kappa=1 κ = 1 ; independent uniform random scores; predictive uncertainty 1 − 2 ∣ f − .5 ∣ 1-2|f-.5| 1 − 2∣ f − .5∣ ; negative fitted probability of currently having a known label; predicted Brier loss ( 1 − p ^ k n o w n ) f 2 + p ^ k n o w n ( 1 − f ) 2 (1-\widehat{p}_{\mathrm{known}})f^2+\widehat{p}_{\mathrm{known}}(1-f)^2 ( 1 − p known ) f 2 + p known ( 1 − f ) 2 ; and historical sensitivity width s ^ Γ \widehat{s}_{\Gamma} s Γ . The fixed protocol uses its frozen initial predictor, threshold, and historical fits. Sequential TIV uses the current predictor and upper threshold with the frozen historical sensitivity bounds. The propensity baseline refits the indicator of a currently known label on all training covariates, with probabilities clipped to [ .02 , .98 ] [.02,.98] [ .02 , .98 ] . Predicted loss refits the outcome model on currently known outcomes and uses its raw probabilities. These models share the boosting settings above, with their respective targets, available labels, and seeds. Refreshed acquisition fits guide label choice; retained evaluation continues to use the historical outcome fit in (D.3).
In all 20 paired terminal runs, TIV has lower estimated width and held-out CVaR than random, while random has lower mean Brier loss. The median paired differences are −.0332 in width, −.0138 in CVaR, and +.0014 in mean Brier loss (Table 6 ). The exact constant reference has lower tail loss and higher mean loss than both policy medians. Figure 9 shows how the estimated width and held-out tail risk evolve as each policy changes the training labels and predictor.
Figure 9. Sequential precision and predictive outcomes across all six acquisition rules. Lines are medians over 20 masks at α = .95 \alpha= .95 α = .95 , Γ = 3 \Gamma= 3 Γ = 3 ; light bands are interquartile ranges for TIV, random, and lowest-propensity acquisition. (a) Estimated retained-sensitivity width for the current predictor. (b) Its CVaR on the reserved 20% split. Predicted-loss, predictive-uncertainty, and sensitivity-width curves are plotted separately and nearly overlap. Lines join recorded rounds. Table 6 gives the exact constant-forecast reference. The horizontal coordinate is the median fraction of initially hidden training-pool labels acquired.
Recorded implementation inventory
Code, saved results, and reproduction instructions are available at https://github.com/E-zClap/hidden-tails . The repository’s manuscript folder contains reproducibility/README.md and reproducibility/implementation_inventory.tex, which record implementation entry points, configurations, seeds, draw order, saved-output mappings, and runtime/hash logs. The sequential results in Figure 4 b, Figure 9 , and Table 5 –Table 6 use the retained plug-in objective (D.3); the inventory identifies its runner and configuration and records residual-corrected trajectories separately. It also documents the cohort verification: all 53,367 reconstructed issued records match all 12 shared columns in their original order, and all 20 archived mask diagnostics match.
Sensitivity Calibration and Extensions
Choosing and calibrating the sensitivity range
For interior probabilities, γ = log Γ \gamma= \log\Gamma γ = log Γ bounds the hidden–selected log-odds discrepancy:
∣ logit p 0 ( x ) − logit p 1 ( x ) ∣ ≤ γ , p 0 ( x ) ∈ [ expit { logit p 1 ( x ) − γ } , expit { logit p 1 ( x ) + γ } ] . (82) \left| \operatorname{logit} p_0(x) - \operatorname{logit} p_1(x) \right| \le\gamma,
\qquad
p_0(x) \in\left[ \operatorname{expit}\{\operatorname{logit} p_1(x) - \gamma\},\operatorname{expit}\{\operatorname{logit} p_1(x) + \gamma\} \right]. \tag*{(82)} ∣ logit p 0 ( x ) − logit p 1 ( x ) ∣ ≤ γ , p 0 ( x ) ∈ [ expit { logit p 1 ( x ) − γ } , expit { logit p 1 ( x ) + γ } ] . (82)
Overlap identifies the reference p 1 p_1 p 1 ; substantive knowledge or additional outcomes inform the allowed discrepancy, as in sensitivity analysis for nonignorable observation [11 , 12 , 13 , 30 ] .
Probability-scale elicitation. For a reference probability p ∈ ( 0 , 1 ) p \in(0,1) p ∈ ( 0 , 1 ) and elicited hidden range [ l , u ] ⊂ ( 0 , 1 ) [l,u] \subset(0,1) [ l , u ] ⊂ ( 0 , 1 ) , the smallest symmetric odds bound containing that range is
Γ e l i c ( p , [ l , u ] ) = max { 1 , odds ( u ) odds ( p ) , odds ( p ) odds ( l ) } , odds ( v ) = v 1 − v . (83) \Gamma_{\mathrm{elic}}(p,[l,u]) = \max\left\{1,\frac{\operatorname{odds}(u)}{\operatorname{odds}(p)},\frac{\operatorname{odds}(p)}{\operatorname{odds}(l)}\right\},
\qquad
\operatorname{odds}(v) = \frac{v}{1-v}. \tag*{(83)} Γ elic ( p , [ l , u ]) = max { 1 , odds ( p ) odds ( u ) , odds ( l ) odds ( p ) } , odds ( v ) = 1 − v v . (83)
For p = .10 p = .10 p = .10 and [ l , u ] = [ .05 , .20 ] [l,u] = [.05,.20] [ l , u ] = [ .05 , .20 ] , this gives 9 / 4 9/4 9/4 . A common Γ \Gamma Γ must cover the whole target covariate support; uncertainty in p p p can be included by maximizing over its confidence set.
Pilot calibration and covariate resolution. Declare a finite-cell model X = g ∈ G X = g \in\mathcal{G} X = g ∈ G , and write p s g = Pr ( Y = 1 ∣ X = g , S = s ) p_{sg} = \Pr(Y = 1 \mid X = g, S = s) p s g = Pr ( Y = 1 ∣ X = g , S = s ) in cells with originally hidden mass. A pilot identifies p 0 g p_{0g} p 0 g only when its design identifies that original hidden distribution: conditional random sampling suffices, while unequal probabilities require adjustment. An arbitrarily selected audited subset instead identifies its own law p A p_A p A (Appendix § 14 .§ 1 ). Suppose confidence sets satisfy Pr { p s g ∈ C s g \Pr\{p_{sg} \in C_{sg} Pr { p s g ∈ C s g for all relevant ( s , g ) } ≥ 1 − δ c a l (s,g)\} \ge1 - \delta_{\mathrm{cal}} ( s , g )} ≥ 1 − δ cal . For sets contained in ( 0 , 1 ) (0,1) ( 0 , 1 ) , define
Γ c a l = exp { max g ∈ G sup u ∈ C 0 g , v ∈ C 1 g ∣ logit u − logit v ∣ } . (84) \Gamma_{\mathrm{cal}} = \exp\left\{\max_{g \in\mathcal{G}}\sup_{u \in C_{0g},\,v \in C_{1g}} \left|\operatorname{logit} u - \operatorname{logit} v\right|\right\}. \tag*{(84)} Γ cal = exp { g ∈ G max u ∈ C 0 g , v ∈ C 1 g sup ∣ logit u − logit v ∣ } . (84)
On the simultaneous coverage event this bounds every cell’s true discrepancy. The supremum can be infinite as admissible probabilities approach incompatible boundaries. Clipping confidence sets to an interior interval preserves coverage only under a justified probability restriction or a confidence construction already supplying it. The finite-Γ \Gamma Γ restrictions at p 1 = 0 , 1 p_1 = 0,1 p 1 = 0 , 1 are specified in Appendix § 12 .§ 1 .
Calibration in E.3 controls the model with X = g X = g X = g . Cell averages do not control discrepancies at finer covariate resolution: either analyze with these cells or justify explicit within-cell homogeneity of the relevant probabilities. A certificate using finer covariates needs control at that finer resolution.
Joint error control. With an independent pilot, condition on it and apply the certificate at the chosen finite Γ c a l \Gamma_{\mathrm{cal}} Γ cal with valid conditional failure probability δ c e r t \delta_{\mathrm{cert}} δ cert , including nuisance estimation. A union bound gives total failure probability at most δ c a l + δ c e r t \delta_{\mathrm{cal}}+\delta_{\mathrm{cert}} δ cal + δ cert under the two procedures’ assumptions. Reusing labels for calibration and revelation requires joint control of the sensitivity choice, audited contribution, and endpoint estimates; their dependence must enter the analysis. For a prespecified finite sensitivity grid, allocate error across its points, each already controlling the stipulated predictor class. A continuum requires uniform control in sensitivity. Let Γ max \Gamma_{\max} Γ m a x be the largest sensitivity level. under consideration. Population nesting implies that risk-limit or strict-separation claims valid at Γ max \Gamma_{\max} Γ m a x hold throughout [ 1 , Γ max ] [1,\Gamma_{\max}] [ 1 , Γ m a x ] . Compare the calibrated range with the breakdown definition in Appendix § 12 .§ 5 ; Appendix § 15 .§ 3 estimates breakdown numerically by interpolation on a log-sensitivity grid.
Extensions of the outcome model
Binary outcomes provide two label-specific losses whose ordering supplies one extremal hidden kernel for every threshold. A multiclass extension needs a sensitivity set for the hidden class-probability vector and a new extremality and threshold-optimization argument. Continuous outcomes require conditional-distribution sensitivity sets; unbounded losses additionally require tail conditions for finite statistical control. Population overlap and nuisance-error requirements remain those of Appendix § 13 ; the historical extension and masking calibration are specified in Appendices § 15 .§ 3 and § 15 .§ 4 .
Connections to related approaches
Selective-label evaluation concerns outcomes observed through historical decisions [3 , 7 , 8 ] . Missing-data analysis and partial identification organize the resulting uncertainty through compatible laws [9 , 10 , 31 , 32 ] ; here every point of the tail-risk interval has such a law. The CVaR representation accommodates discrete losses [16 ] . As in distributionally robust learning [14 , 15 , 33 , 34 ] , the upper endpoint is a worst-case risk; the lower endpoint records the remaining identification information. Orthogonalization [17 , 18 , 19 ] and cross-fitting [17 , 18 ] control nuisance bias, and Rademacher complexity and contraction control uniform fluctuation [23 , 24 , 25 ] .
Active learning links queries to a learning objective [35 ] ; TIV links acquisition to the precision of a specified risk claim. Its exact allocation theorem fixes f f f , η \eta η , Γ \Gamma Γ . The sequential study (Appendix § 15 .§ 6 ) refreshes acquisition values after retraining while keeping the historical outcome regression fixed in the retained objective. Auxiliary fits to current known-label status or outcomes serve the corresponding acquisition baselines. The study evaluates interval trajectories and held-out predictive risk. Inference across adaptive rounds or at data-dependent stopping requires control of the acquisition history, newly revealed contributions, and stopping rule, beyond the fixed-sample uniform theorem.
Broader impacts
By exposing population risks hidden by selective observation, this framework can help decision makers identify when available evidence is insufficient and direct audits toward informative outcomes. In lending or healthcare, however, a sensitivity range that excludes the true hidden outcome law, inaccurate nuisance estimates, or extrapolation into unsupported regions can produce misleading assurances and reinforce unequal access to credit or care. Even a valid population certificate does not establish fairness across demographic groups. Targeted collection may disproportionately burden particular groups and can expose sensitive financial or health information. Audit design should therefore consider subgroup coverage and acquisition burdens, use appropriate consent or authorization, and limit data collection, access, and retention. Certificates must be interpreted within the specified population, outcome, loss, and assumptions; they do not by themselves justify deployment or replace causal, fairness, privacy, and domain-specific assessments.
Additional Recorded Experiments
This appendix collects saved experiments that complement the studies in Appendix § 15 : simple observation mechanisms, finite-candidate learning, additional diagnostics for the synthetic revelation frontier, and a finite-population illustration constructed from diabetes data. They answer questions about how the objectives behave under controlled conditions. Their population quantities and empirical trajectories carry no additional finite-sample confidence guarantee. No new simulations or model fits were run when preparing these tables.
Observation mechanisms, candidate learning, and fixed audits
Low observed risk can hide high population risk. The two-region example has masses ( .82 , .18 ) (.82, .18) ( .82 , .18 ) , deterministic outcomes ( 0 , 1 ) (0, 1) ( 0 , 1 ) , and observation probabilities ( .95 , .04 ) (.95, .04) ( .95 , .04 ) . Use absolute loss and α = .85 \alpha= .85 α = .85 . For the constant-zero predictor, the saved observed-label CVaR is 0.0611, whereas population CVaR is 1.0000. The region-specific predictor ( 0 , 1 ) (0, 1) ( 0 , 1 ) has zero loss under both populations. Thus this example demonstrates a large discrepancy between observed and population risk; it does not exhibit a pairwise ranking reversal. The saved envelope calculation uses Γ = 50 \Gamma= 50 Γ = 50 and clips boundary reference probabilities to [ 10 − 8 , 1 − 10 − 8 ] [10^{-8}, 1 - 10^{-8}] [ 1 0 − 8 , 1 − 1 0 − 8 ] , producing negligible nonzero envelope values for the zero-loss reference.
Finite-candidate selection with known auxiliary functions. Use the five-region population specified under “Population sweeps” in Appendix § 15 .§ 2 . Outcomes are sampled from its specified Bernoulli probabilities; observation is sampled conditional on the outcome. The adverse-label observation probability in region B B B is .03, compared with .60 for the other label, giving an odds-ratio sensitivity shift of 48.5. Fix Γ = 50 \Gamma= 50 Γ = 50 , α = .85 \alpha= .85 α = .85 , absolute loss, and
F 0 = { ( .05 , b , .10 , .40 , .50 ) : b ∈ { .05 , .25 , .65 , .85 } } . \mathcal{F}_0 = \{(.05, b, .10, .40, .50) : b \in\{.05, .25, .65, .85\}\}. F 0 = {( .05 , b , .10 , .40 , .50 ) : b ∈ { .05 , .25 , .65 , .85 }} .
Each of 250 independent training samples contains 1,400 records. Four rules minimize, over F 0 \mathcal{F}_0 F 0 , empirical full-label CVaR, empirical observed-label CVaR, normalized inverse-propensity-weighted observed-label CVaR, or the empirical upper sensitivity objective. The propensity rule uses the known marginal π ( X ) \pi(X) π ( X ) ; the upper objective uses known p 1 p_1 p 1 and combines observed excess loss with the bounded hidden contribution. It does not estimate auxiliary models or add the locally robust residual correction. All cutoff searches use exact loss breakpoints. The full-label rule is an oracle benchmark unavailable to a learner with selective labels.
Table 7 evaluates each chosen predictor under the exact population law. Each rule chooses the same candidate in every recorded replication, so the table does not attach a sampling-error bar to its repeated population evaluation. The difference between observed-label and inverse-propensity learning, on one hand, and upper-envelope learning on the other illustrates the consequence of outcome-dependent observation in this particular candidate class. It is not a universal comparison between learning algorithms.
Table 7. Finite-candidate selection over 250 samples of size 1,400. Values average exact population evaluations of the chosen predictor; each rule selected the same candidate in every replication. The upper endpoint is a population sensitivity envelope, without statistical enlargement.
A single fixed-predictor audit pool. A further sample of 7,000 records from the same population supplies one audit pool. Fix the population observed-label-preferred predictor f = ( .05 , .25 , .10 , .40 , .50 ) f = (.05, .25, .10, .40, .50) f = ( .05 , .25 , .10 , .40 , .50 ) and the lower-envelope minimizing cutoff η = .5 \eta= .5 η = .5 . Only originally hidden outcomes can be acquired, at unit cost. The five static rankings are per-hidden-label TIV s ∣ d f , η ∣ / β s_{\lvert df,\eta\rvert}/\beta s ∣ df , η ∣ / β , independent uniform random scores, region mass, uncertainty − ∣ f − .5 ∣ -\lvert f - .5\rvert − ∣ f − .5 ∣ , and expected absolute loss at the midpoint ( q − + q + ) / 2 (q_- + q_+)/2 ( q − + q + ) /2 of the sensitivity interval. The implementation sorts each score in decreasing order using NumPy’s default argsort; it does not impose a separate stable tie rule. The 16 recorded budgets range from zero to 1,800 labels in increments of 120. The predictor and rankings remain fixed throughout.
Revealed outcomes enter the known empirical contribution. The remaining hidden outcomes retain the original sensitivity bounds. Table 8 separately reports the reduction of the fixed-cutoff gap and of the empirical interval whose endpoints minimize their cutoffs independently. Both reductions are normalized by their respective initial value in this one pool. The two objectives need not favor the same allocation at every budget: random revelation removes slightly more optimized width at 600 labels, although TIV removes more fixed-cutoff ambiguity. Prediction uncertainty is also distinct from label-sensitive tail ambiguity: the region with f = .5 f=.5 f = .5 has identical absolute loss under both labels, hence zero TIV at every cutoff. These are mechanism observations from one realized pool, without replication-based inference or a claim of adaptive learning optimality.
Table 8. One synthetic pool of 7,000 records, fixed predictor, and unit cost per hidden label. Reductions are normalized by that pool’s initial gap or initial optimized width. These two targets need not rank policies identically; at 600 labels random revelation removes slightly more optimized width than TIV.
Additional diagnostics for the synthetic frontier
Use the same 120 synthetic worlds, each with 120 cells, described in Appendix § 15 .§ 5 . The loss is absolute, α = .85 \alpha=.85 α = .85 , and Γ = 12 \Gamma=12 Γ = 12 . The additional saved outputs compare three specified forecasts: p 1 p_1 p 1 , the upper sensitivity probability q + ( p 1 ) q_+(p_1) q + ( p 1 ) , and p f u l l = π p 1 + ( 1 − π ) p 0 p_{\mathrm{full}}=\pi p_1+(1-\pi)p_0 p full = π p 1 + ( 1 − π ) p 0 . The CSV names for these rows are “Observed-selected,” “Certificate,” and “Oracle.” In this experiment they are formulas fixed by the generator, not the results of minimizing empirical risk or an upper endpoint over a candidate class. The full-label forecast uses information unavailable from the observed law. Table 9 reports mean population evaluations over the 120 worlds, alongside the single finite population introduced below. In particular, the q + ( p 1 ) q_+(p_1) q + ( p 1 ) row is not evidence that the risk-minimizing forecast has been found.
Table 9. Three specified reference forecasts, evaluated with absolute loss and α = .85 \alpha=.85 α = .85 . The CSV role names “Observed-selected,” “Certificate,” and “Oracle” identify these formulas; they do not denote optimization over a candidate class. Synthetic entries use Γ = 12 \Gamma=12 Γ = 12 ; diabetes entries use its oracle-chosen sensitivity level.
From a common cutoff to separately minimized endpoints. For each world, fix f = p 1 f=p_1 f = p 1 and acquire along the TIV-per-cost ordering at η = .5 \eta=.5 η = .5 . Cost is ∑ j μ j c j λ j \sum_j \mu_j c_j\lambda_j ∑ j μ j c j λ j , with a fractional terminal cell; budget fractions divide by the full population cost ∑ j μ j c j \sum_j \mu_j c_j ∑ j μ j c j . The known audited contribution is evaluated with the generative p 0 p_0 p 0 , while the remaining outcomes retain their sensitivity bounds. This isolates the retained population objective from estimation of the newly audited contribution.
The saved diagnostic evaluates 21 cost fractions from zero to one. Its endpoint search uses the grid G = { 0 , 1 / 60 , … , 1 } \mathcal{G}=\{0,1/60,\ldots,1\} G = { 0 , 1/60 , … , 1 } , giving
Γ Γ , λ G ( f ) = min η ∈ G U f , η ( λ ) − min η ∈ G L f , η ( λ ) . \Gamma_{\Gamma,\lambda}^{\mathcal{G}}(f)=\min_{\eta\in\mathcal{G}}U_{f,\eta}(\lambda)-\min_{\eta\in\mathcal{G}}L_{f,\eta}(\lambda). Γ Γ , λ G ( f ) = η ∈ G min U f , η ( λ ) − η ∈ G min L f , η ( λ ) .
This recorded grid-minimized width is distinct from the exact breakpoint-minimized width in Appendix § 14 .§ 4 . The companion bound maximizes the common-cutoff gap on the same grid. Because binary absolute-loss contrasts satisfy ∣ d f , η ∣ ≤ ∣ d f , 0 ∣ |d_{f,\eta}|\leq|d_{f,0}| ∣ d f , η ∣ ≤ ∣ d f , 0 ∣ and 0 ∈ G 0\in\mathcal{G} 0 ∈ G , that maximum equals W f , 0 ( λ ) W_{f,0}(\lambda) W f , 0 ( λ ) . All 2,520 saved world–budget rows satisfy 0 ≤ Γ Γ , λ G ( f ) ≤ W f , 0 ( λ ) 0\leq\Gamma_{\Gamma,\lambda}^{\mathcal{G}}(f)\leq W_{f,0}(\lambda) 0 ≤ Γ Γ , λ G ( f ) ≤ W f , 0 ( λ ) to numerical precision. Table 10 gives selected budgets. This check illustrates the sufficient-width connection; it neither makes the .5-cutoff allocation optimal for optimized width nor turns the finite grid into exact threshold minimization. The gap bound can exceed one because it is not itself a bounded-loss CVaR.
Table 10. Width connection along the fixed-η = .5 \eta= .5 η = .5 TIV-per-cost allocation. Each endpoint is minimized on the recorded 61-point cutoff grid. Synthetic columns are separate medians across 120 worlds; diabetes has one finite population. The sufficient bound is W f , 0 W_{f,0} W f , 0 , which is not itself a CVaR risk and can exceed one.
A diabetes-derived finite-population illustration
Measured outcomes and imposed observation. The construction uses all 442 records in the diabetes dataset distributed with scikit-learn [27 ] . It binarizes the measured disease-progression outcome at its empirical 75th percentile, assigning Y = 1 Y = 1 Y = 1 at or above that cutoff. A standardized logistic regression, fitted to all these outcomes with a 5,000-iteration limit and the recorded software defaults, supplies scores r i r_i r i . Fourteen empirical-quantile score cells define the finite covariate population. The score is used to construct strata, not evaluated as a held-out predictive model. In particular, the construction has access to outcomes subsequently called hidden.
For each record, impose the observation probability
s i = clip [ .03 , .95 ] { .85 − .25 r i − .55 Y i − .20 Y i r i + ϵ i } , ϵ i ∼ N ( 0 , .01 2 ) . s_i = \operatorname{clip}_{[.03,.95]}\{.85-.25r_i-.55Y_i-.20Y_ir_i+\epsilon_i\}, \qquad\epsilon_i \sim N(0,.01^2). s i = clip [ .03 , .95 ] { .85 − .25 r i − .55 Y i − .20 Y i r i + ϵ i } , ϵ i ∼ N ( 0 , .0 1 2 ) .
The saved construction integrates these probabilities; it does not draw a new binary observation mask. For a cell j j j with n j n_j n j records, set
μ j = n j 442 , π j = 1 n j ∑ i ∈ j s i , p 1 j = ∑ i ∈ j Y i s i ∑ i ∈ j s i , p 0 j = ∑ i ∈ j Y i ( 1 − s i ) ∑ i ∈ j ( 1 − s i ) . \mu_j=\frac{n_j}{442}, \qquad\pi_j=\frac{1}{n_j}\sum_{i\in j}s_i, \qquad p_{1j}=\frac{\sum_{i\in j}Y_i s_i}{\sum_{i\in j}s_i}, \qquad p_{0j}=\frac{\sum_{i\in j}Y_i(1-s_i)}{\sum_{i\in j}(1-s_i)}. μ j = 442 n j , π j = n j 1 i ∈ j ∑ s i , p 1 j = ∑ i ∈ j s i ∑ i ∈ j Y i s i , p 0 j = ∑ i ∈ j ( 1 − s i ) ∑ i ∈ j Y i ( 1 − s i ) .
Before evaluation, the implementation clips p 1 j p_{1j} p 1 j and p 0 j p_{0j} p 0 j to [ 10 − 4 , 1 − 10 − 4 ] [10^{-4},1-10^{-4}] [ 1 0 − 4 , 1 − 1 0 − 4 ] and uses the resulting Bernoulli mixture p f u l l , j = π j p 1 j + ( 1 − π j ) p 0 j p_{\mathrm{full},j}=\pi_jp_{1j}+(1-\pi_j)p_{0j} p full , j = π j p 1 j + ( 1 − π j ) p 0 j . Consequently, the reported full risks refer to this smoothed finite-cell law. The measured progression outcomes and covariates define the construction; the observation channel and acquisition costs are imposed. The population observation share is 0.6325.
Sensitivity, loss, and costs. Use absolute loss and α = .85 \alpha=.85 α = .85 . The sensitivity level is chosen with oracle knowledge of both cell kernels:
Γ = max { 2 , 1.10 exp ( max j ∣ logit p 0 j − logit p 1 j ∣ ) } = 72.08 (rounded). \Gamma=\max\left\{2,1.10\exp\left(\max_j\left|\operatorname{logit}p_{0j}-\operatorname{logit}p_{1j}\right|\right)\right\}=72.08 \quad\text{(rounded).} Γ = max { 2 , 1.10 exp ( j max ∣ logit p 0 j − logit p 1 j ∣ ) } = 72.08 (rounded).
This construction ensures compatibility for the known cell law. It is not a calibration procedure justified using observed labels alone, and it supplies no simultaneous or data-dependent-Γ \Gamma Γ confidence statement. With zero-based cell indices, the imposed population cost density is c j = .65 + 1.15 r ˉ j + .201 { j m o d 3 = 0 } c_j=.65+1.15\bar r_j+.201\{j\bmod3=0\} c j = .65 + 1.15 r ˉ j + .201 { j mod 3 = 0 } , where r ˉ j \bar r_j r ˉ j is the cell’s mean fitted score.
The three reference forecasts and their population risk evaluations follow Appendix § 17 .§ 2 . For the acquisition comparison fix f = p 1 f=p_1 f = p 1 and η = .5 \eta=.5 η = .5 , use divisible cell allocations, and evaluate 51 fractions of full cost in increments of .02. The acquisition rules are the seven baselines and TIV-per-cost/frontier ordering described in Appendix § 15 .§ 5 . Random acquisition averages the remaining gap over 100 independent uniform-random cell orderings. The 21-budget width diagnostic uses the same 61-point cutoff grid as the synthetic study.
Table 11 reports the first recorded cost fraction at which each curve reaches 25% of its initial fixed-cutoff gap. For random acquisition, this is the crossing of the mean gap curve, rather than the mean of 100 individual crossing costs. TIV and the exact fractional-knapsack frontier coincide by their common ordering. The data illustrate how observation and costs shape a retained population certificate when all relevant quantities are known; they do not establish clinical effectiveness, generalization to a new patient population, or finite-sample coverage. They also do not convert passive revelation into identification of outcomes under untaken actions.
Table 11. Diabetes-derived population: first hit on the 2%-spaced full-cost grid. Random is the mean remaining gap over 100 random cell orderings, so its entry is the first crossing of that mean curve, not the mean of 100 crossing costs. Acquisition is divisible at the cell level.
Saved-output provenance and reproduction order
The numerical sources are the experiment1, experiment2, and experiment3 CSVs, and the experiment6 and experiment7_semisynthetic CSVs under experiments/results/. The mechanism examples use the full-sequence random stream with master seed 20260503. The current frontier and diabetes CSVs instead match run_summary_main_figure.json, generated by running experiments/run_tail_observability_experiments.py with --main-figure. The full-run summary’s older frontier output uses a different random-stream position. The archived run_summary_semisynthetic.json instead records an earlier 569-record malignancy example, with malignancy-rate fields; it is not the 442-record diabetes construction in this appendix. Neither archived output is substituted for the current CSVs. Reproductions should use separate output copies or follow the documented run order so that one entry point does not silently overwrite another run’s saved results.
The standard-library extraction script figures/scripts/prepare_additional_experiments.py regenerates these tables from the saved CSVs. Its --check mode verifies the generated tables, source checksums, agreement with the matching run summary, repeated-sample aggregation, TIV/frontier equality, and the recorded width inequalities. figures/supplement_additional_provenance.json records those checks and the source-to-entry-point mapping. The corrected lending sequential experiment, original-cohort verification, and finite-sample endpoint study remain documented in Appendix D and the accompanying reproduction material.