Introduction
As machine learning systems are used in increasingly consequential settings, the question is not only how to optimize a model, but also what kind of performance guarantee should generalize. Classical statistical learning theory is organized around a simple template: each example incurs a fixed pointwise loss, and performance is measured by averaging this loss over the data distribution. Algorithm-dependent information-theoretic (IT) generalization bounds provide a complementary view of this template: MI [41 ] , CMI [34 ] , eCMI [21 , 39 ] , and prediction-based fCMI [19 ] relate generalization to how much the learned hypothesis, prediction, or loss reveals about the selected training sample. Because these bounds depend on the learning algorithm rather than only on a uniform hypothesis-class complexity, they can capture stability and implicit regularization; recent supersample refinements further sharpen this picture [39 ] .
Many modern evaluation criteria, however, are not averages of fixed per-example losses. In reliable prediction and uncertainty quantification, multicalibration asks whether predictive probabilities remain meaningful across many subpopulations and score regions, not only on average [20 , 14 , 3 ] . In risk-sensitive learning, optimized-certainty-equivalent risks and conditional value-at-risk (CVaR) emphasize tail behavior rather than mean loss alone [2 , 32 , 23 ] . In robust learning, distributionally robust optimization (DRO) reevaluates performance under adversarial reweightings of the empirical loss distribution [33 , 1 , 6 ] ; related transport-based criteria depend on alignments of empirical distributions [4 ] . These criteria are central when average loss can look benign while subgroup, tail, or worst-case failures remain severe.
What makes these objectives hard for existing IT analysis is not merely that they are non-pointwise. Recent work has already extended IT generalization beyond pointwise averages to fixed higher-order losses, including pairwise and triplet objectives [5 ] ; however, our setting is different. Here the empirical metric is set-input : it depends on the entire batch of predictions and labels. It is optimization-valued : after seeing the batch, the metric selects an auxiliary object by solving a downstream optimization problem, such as the most violated subgroup witness, a tail threshold, or an adversarial reweighting. Because this auxiliary choice is data-dependent, a one-point replacement can change not only one observation but also the optimizer that defines the metric. For multicalibration, the worst witness on the training batch need not coincide with the worst witness on the validation batch; for CVaR, the optimizing threshold can shift in the same way. This helps explain why existing analyses are family-specific, e.g., for ECE [11 ] , OCE/CVaR [23 ] , or robust optimization [1 ] .
We develop a unified information-theoretic analysis of set-input optimization-valued objectives in the supersample setting (§ 3.1 ). From each pair, a selector independently chooses one observation for training with equal probability, while the other is held out for validation. The central object is the selector gap (Eq. (6) ), defined as the optimized value on the validation sample minus that on the training sample. This gap directly compares the values obtained after re-optimizing the auxiliary variable separately on the two samples, without fixing a common optimizer. Complementing the selector vector exchanges the two samples and reverses the sign of the gap; thus, the selector gap is antisymmetric. This antisymmetry lets us control the expected optimized validation–training gap through two quantities (Theorem 1): the fCMI between the learned supersample prediction table and the selector vector, which measures algorithmic dependence on the selector vector, and an objective-dependent sub-Gaussian parameter for the selector gap under the uniform selector law, which reflects the stability or geometry of the optimized objective. To obtain a complete population–training bound, we combine this result with an objective-specific fixed-predictor bound on the population–validation gap. The validation–training bound accounts for auxiliary optimization without an explicit entropy, cardinality, Rademacher, or covering-number term for the auxiliary class.
We give two sufficient conditions for the selector gap to be sub-Gaussian under the uniform selector law. The first is a one-point stability certificate: if the objective changes little when one example is replaced, then the selector gap concentrates by bounded differences; this route also yields a single-index refinement that decomposes the validation–training gap into one-point replacement increments and one CMI term per index (§ 3.2 ; Theorem 2). The second is a convex L 2 L_2 L 2 -Lipschitz certificate, which captures sharper geometry when the objective is a convex function of a coordinatewise score vector (§ 3.3 ). Together, these certificates give concrete bounds for multicalibration, CVaR, and Cressie–Read DRO, including the L 2 L_2 L 2 -geometric improvement for DRO (Corollaries 1–6). We also give population–training decompositions using fixed-predictor MC, CVaR, and DRO comparisons (§ 3.4 ) and experiments illustrating computable local-information diagnostics (§ 5 ).
Preliminaries
We fix notation used throughout. We use uppercase letters for random variables (RVs) and lowercase letters for their realizations. For a ∈ N a \in\mathbb{N} a ∈ N , write [ a ] : = { 1 , … , a } [a] := \{1,\ldots,a\} [ a ] := { 1 , … , a } . Let X \mathcal{X} X be the input space, Y \mathcal{Y} Y the label space, and Z = X × Y \mathcal{Z} = \mathcal{X} \times\mathcal{Y} Z = X × Y . Let V \mathcal{V} V be the prediction space, and let D \mathcal{D} D be an unknown distribution on Z \mathcal{Z} Z . We denote mutual information (MI) and CMI by I ( ⋅ ; ⋅ ) I(\cdot;\cdot) I ( ⋅ ; ⋅ ) and I ( ⋅ ; ⋅ ∣ ⋅ ) I(\cdot;\cdot\mid\cdot) I ( ⋅ ; ⋅ ∣ ⋅ ) , respectively. All expectations are over the random variables under discussion.
IT generalization bound based on eCMI
For ordinary pointwise losses, we use the standard supersample construction [34 , 21 ] . Let Z ~ ∈ Z n × 2 \widetilde{Z} \in\mathcal{Z}^{n \times2} Z ∈ Z n × 2 be an array of 2 n 2n 2 n i.i.d. draws from D \mathcal{D} D , with entries Z ~ i , j = ( X ~ i , j , Y ~ i , j ) \widetilde{Z}_{i,j} = (\widetilde{X}_{i,j}, \widetilde{Y}_{i,j}) Z i , j = ( X i , j , Y i , j ) (i ∈ [ n ] i \in[n] i ∈ [ n ] , j ∈ { 0 , 1 } j \in\{0,1\} j ∈ { 0 , 1 } ). We denote the i i i -th row by Z ~ i = ( Z ~ i , 0 , Z ~ i , 1 ) \widetilde{Z}_i = (\widetilde{Z}_{i,0}, \widetilde{Z}_{i,1}) Z i = ( Z i , 0 , Z i , 1 ) . Let the selector vector U = ( U 1 , … , U n ) ∼ Unif ( { 0 , 1 } n ) U = (U_1,\ldots,U_n) \sim\operatorname{Unif}(\{0,1\}^n) U = ( U 1 , … , U n ) ∼ Unif ({ 0 , 1 } n ) be independent of Z ~ \widetilde{Z} Z . Each component U i U_i U i is called a selector; write U ‾ i = 1 − U i \overline{U}_i = 1 - U_i U i = 1 − U i . For a deterministic selector vector u ∈ { 0 , 1 } n u \in\{0,1\}^n u ∈ { 0 , 1 } n , we similarly write u ‾ i = 1 − u i \overline{u}_i = 1 - u_i u i = 1 − u i . The selected sample Z ~ U : = ( Z ~ i , U i ) i = 1 n \widetilde{Z}_U := (\widetilde{Z}_{i,U_i})_{i=1}^n Z U := ( Z i , U i ) i = 1 n is used as the training sample, while the complementary sample Z ~ U ‾ : = ( Z ~ i , U ‾ i ) i = 1 n \widetilde{Z}_{\overline{U}} := (\widetilde{Z}_{i,\overline{U}_i})_{i=1}^n Z U := ( Z i , U i ) i = 1 n is used as the held-out validation (or test) sample. Write S : = Z ~ U S := \widetilde{Z}_U S := Z U , S ′ : = Z ~ U ‾ S' := \widetilde{Z}_{\overline{U}} S ′ := Z U . We use n n n -subscripts for batch size. A learning algorithm receives S S S and independent randomness R ∈ R R \in\mathcal{R} R ∈ R , outputs W = A ( S , R ) W = A(S,R) W = A ( S , R ) , and returns a predictor f W ∈ F : X → V f_W \in\mathcal{F}:\mathcal{X}\to\mathcal{V} f W ∈ F : X → V . We evaluate f W f_W f W on all 2 n 2n 2 n supersample inputs and write f W ( X ~ ) : = ( f W ( X ~ i , j ) ) i ∈ [ n ] , j ∈ { 0 , 1 } ∈ V n × 2 f_W(\widetilde{X}):=(f_W(\widetilde{X}_{i,j}))_{i\in[n],\,j\in\{0,1\}}\in\mathcal{V}^{n\times2} f W ( X ) := ( f W ( X i , j ) ) i ∈ [ n ] , j ∈ { 0 , 1 } ∈ V n × 2 .
For comparison with the standard setting, let ℓ : V × Y → [ 0 , 1 ] \ell:\mathcal{V}\times\mathcal{Y}\to[0,1] ℓ : V × Y → [ 0 , 1 ] be a bounded pointwise loss function. For a predictor f f f and a sample s = ( ( x i , y i ) ) i = 1 n s=((x_i,y_i))_{i=1}^n s = (( x i , y i ) ) i = 1 n , define the population risk L ( f ) : = E Z ∼ D ℓ ( f ( X ) , Y ) L(f):=\mathbb{E}_{Z\sim D}\ell(f(X),Y) L ( f ) := E Z ∼ D ℓ ( f ( X ) , Y ) and the empirical risk L ^ ( f ; s ) : = n − 1 ∑ i = 1 n ℓ ( f ( x i ) , y i ) \widehat{L}(f;s):=n^{-1}\sum_{i=1}^n\ell(f(x_i),y_i) L ( f ; s ) := n − 1 ∑ i = 1 n ℓ ( f ( x i ) , y i ) . When no confusion is possible, write the loss value ℓ ( f , z ) \ell(f,z) ℓ ( f , z ) as shorthand for ℓ ( f ( x ) , y ) \ell(f(x),y) ℓ ( f ( x ) , y ) with z = ( x , y ) z=(x,y) z = ( x , y ) , and write the i i i -th supersample loss row as ℓ ( f W , Z ~ i ) : = ( ℓ ( f W , Z ~ i , 0 ) , ℓ ( f W , Z ~ i , 1 ) ) \ell(f_W,\widetilde{Z}_i):=(\ell(f_W,\widetilde{Z}_{i,0}),\ell(f_W,\widetilde{Z}_{i,1})) ℓ ( f W , Z i ) := ( ℓ ( f W , Z i , 0 ) , ℓ ( f W , Z i , 1 )) . The standard evaluated CMI (eCMI) and functional CMI (fCMI) generalization bound [19 , 21 ] gives
∣ E [ L ^ ( f W ; S ′ ) − L ^ ( f W ; S ) ] ∣ ≤ 1 n ∑ i = 1 n 2 I ( ℓ ( f W , Z ~ i ) ; U i ∣ Z ~ ) ≤ 2 n I ( f W ( X ~ ) ; U ∣ Z ~ ) . (1) \left|\mathbb{E}\left[\widehat{L}(f_W;S')-\widehat{L}(f_W;S)\right]\right|
\leq\frac{1}{n}\sum_{i=1}^{n}\sqrt{2I(\ell(f_W,\widetilde{Z}_i);U_i\mid\widetilde{Z})}
\leq\sqrt{\frac{2}{n}I(f_W(\widetilde{X});U\mid\widetilde{Z})}. \tag*{(1)} E [ L ( f W ; S ′ ) − L ( f W ; S ) ] ≤ n 1 i = 1 ∑ n 2 I ( ℓ ( f W , Z i ) ; U i ∣ Z ) ≤ n 2 I ( f W ( X ) ; U ∣ Z ) . (1)
After fixing f W f_W f W , flipping one selector swaps only the corresponding training and validation loss values. The next subsection considers batch-level objectives without this row-wise structure.
Motivating examples beyond ordinary losses
Beyond the pointwise empirical risks of § 2.1 , we consider three criteria. They become important precisely when mean performance is not sufficiently expressive: multicalibration [20 , 14 , 3 ] captures subgroup and score-conditional reliability, CVaR [32 , 23 ] captures tail-sensitive risk, and distributionally robust optimization (DRO) [30 , 33 ] captures worst-case performance under adversarial reweighting. Throughout, S = ( ( X i , Y i ) ) i = 1 n ∈ Z n S=((X_i,Y_i))_{i=1}^n\in\mathcal{Z}^n S = (( X i , Y i ) ) i = 1 n ∈ Z n and Z = ( X , Y ) ∼ D Z=(X,Y)\sim D Z = ( X , Y ) ∼ D .
Multicalibration. Multicalibration (MC) controls calibration over subpopulations and score-space tests [20 , 14 , 3 ] . Here Y = { 0 , 1 } \mathcal{Y}=\{0,1\} Y = { 0 , 1 } and V = [ 0 , 1 ] \mathcal{V}=[0,1] V = [ 0 , 1 ] . Let Π M C : = C × H × { ± 1 } \Pi_{\mathrm{MC}}:=\mathcal{C}\times\mathcal{H}\times\{\pm1\} Π MC := C × H × { ± 1 } , where C ⊆ { c : X → [ 0 , 1 ] } \mathcal{C}\subseteq\{c:\mathcal{X}\to[0,1]\} C ⊆ { c : X → [ 0 , 1 ]} , H ⊆ { h : [ 0 , 1 ] → [ − 1 , 1 ] } \mathcal{H}\subseteq\{h:[0,1]\to[-1,1]\} H ⊆ { h : [ 0 , 1 ] → [ − 1 , 1 ]} , and write π = ( c , h , η ) \pi=(c,h,\eta) π = ( c , h , η ) . The population and empirical values are
Φ M C ( f W ; D ) : = sup ( c , h , η ) ∈ Π M C η E [ c ( X ) h ( f W ( X ) ) ( Y − f W ( X ) ) ] , \Phi_{\mathrm{MC}}(f_W;D):=\sup_{(c,h,\eta)\in\Pi_{\mathrm{MC}}}\eta\,\mathbb{E}[c(X)h(f_W(X))(Y-f_W(X))], Φ MC ( f W ; D ) := ( c , h , η ) ∈ Π MC sup η E [ c ( X ) h ( f W ( X )) ( Y − f W ( X ))] ,
Φ M C ( f W ; S ) : = sup ( c , h , η ) ∈ Π M C η 1 n ∑ i = 1 n c ( X i ) h ( f W ( X i ) ) ( Y i − f W ( X i ) ) . (2) \Phi_{\mathrm{MC}}(f_W;S):=\sup_{(c,h,\eta)\in\Pi_{\mathrm{MC}}\!}\eta\,\frac{1}{n}\sum_{i=1}^{n}c(X_i)h(f_W(X_i))(Y_i-f_W(X_i)). \tag*{(2)} Φ MC ( f W ; S ) := ( c , h , η ) ∈ Π MC sup η n 1 i = 1 ∑ n c ( X i ) h ( f W ( X i )) ( Y i − f W ( X i )) . (2)
The empirical value selects the subgroup–score witness and sign after seeing the batch, so the selected witness may change when the evaluation batch changes.
OCE risks and CVaR. OCE risks are tail-sensitive criteria [2 , 32 , 23 ] ; for bounded loss, Φ O C E , ψ ( f W ; D ) = inf λ { λ + E ψ ( ℓ ( f W ( X ) , Y ) − λ ) } \Phi_{\mathrm{OCE},\psi}(f_W;D)=\inf_{\lambda}\{\lambda+\mathbb{E}\psi(\ell(f_W(X),Y)-\lambda)\} Φ OCE , ψ ( f W ; D ) = inf λ { λ + E ψ ( ℓ ( f W ( X ) , Y ) − λ )} . For CVaR at level α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) , write α ˉ : = 1 − α \bar{\alpha}:=1-\alpha α ˉ := 1 − α ; this is the case ψ α ( t ) = α ˉ − 1 [ t ] + \psi_\alpha(t)=\bar{\alpha}^{-1}[t]_+ ψ α ( t ) = α ˉ − 1 [ t ] + . The population and empirical CVaR values are
Φ C V a R , α ( f W ; D ) : = inf τ ∈ [ 0 , 1 ] { τ + 1 α ˉ E ( ℓ ( f W ( X ) , Y ) − τ ) + } , \Phi_{\mathrm{CVaR},\alpha}(f_W;D):=\inf_{\tau\in[0,1]}\left\{\tau+\frac{1}{\bar{\alpha}}\mathbb{E}(\ell(f_W(X),Y)-\tau)_+\right\}, Φ CVaR , α ( f W ; D ) := τ ∈ [ 0 , 1 ] inf { τ + α ˉ 1 E ( ℓ ( f W ( X ) , Y ) − τ ) + } ,
Φ C V a R , α ( f W ; S ) : = inf τ ∈ [ 0 , 1 ] { τ + 1 α ˉ n ∑ i = 1 n ( ℓ ( f W , Z i ) − τ ) + } . (3) \Phi_{\mathrm{CVaR},\alpha}(f_W;S):=\inf_{\tau\in[0,1]}\left\{\tau+\frac{1}{\bar{\alpha}n}\sum_{i=1}^{n}(\ell(f_W,Z_i)-\tau)_+\right\}. \tag*{(3)} Φ CVaR , α ( f W ; S ) := τ ∈ [ 0 , 1 ] inf { τ + α ˉ n 1 i = 1 ∑ n ( ℓ ( f W , Z i ) − τ ) + } . (3)
The empirical CVaR threshold is chosen from the observed loss distribution, so the objective is not an average of a fixed per-example loss. More general OCE examples are in Appendix § 9 .§ 2 .
Cressie–Read DRO. DRO allows adversarial reweighting within a divergence ball [6 ] . We use the Cressie–Read f f f -divergence generator f c ( t ) = ( t c − c t + c − 1 ) / { c ( c − 1 ) } f_c(t)=(t^c-ct+c-1)/\{c(c-1)\} f c ( t ) = ( t c − c t + c − 1 ) / { c ( c − 1 )} , with c = 2 c=2 c = 2 giving Pearson χ 2 \chi^2 χ 2 and limits c → 1 , 0 c\to1,0 c → 1 , 0 giving KL and empirical-likelihood divergences. In what follows, we restrict to c > 1 c>1 c > 1 and write c ∗ : = c / ( c − 1 ) c^*:=c/(c-1) c ∗ := c / ( c − 1 ) . Given a radius ϵ ≥ 0 \epsilon\geq0 ϵ ≥ 0 , and density ratio ω = d Q / d D \omega=dQ/dD ω = d Q / d D ,
Φ D R O , c , ϵ ( f W ; D ) : = sup ω ≥ 0 { E D [ ω ( Z ) ℓ ( f W ( X ) , Y ) ] : E D ω = 1 , E D f c ( ω ) ≤ ϵ } . \Phi_{\mathrm{DRO},c,\epsilon}(f_W;D):=\sup_{\omega\geq0}\left\{\mathbb{E}_D[\omega(Z)\ell(f_W(X),Y)]:\mathbb{E}_D\omega=1,\ \mathbb{E}_D f_c(\omega)\leq\epsilon\right\}. Φ DRO , c , ϵ ( f W ; D ) := ω ≥ 0 sup { E D [ ω ( Z ) ℓ ( f W ( X ) , Y )] : E D ω = 1 , E D f c ( ω ) ≤ ϵ } .
On a finite evaluation batch, the ambiguity set becomes
Φ D R O , c , ϵ ( f W ; S ) : = sup q ∈ R + n { 1 n ∑ i = 1 n q i ℓ ( f W ( X i ) , Y i ) : 1 n ∑ i = 1 n q i = 1 , 1 n ∑ i = 1 n q i c ≤ 1 + c ( c − 1 ) ϵ } . (4) \Phi_{\mathrm{DRO},c,\epsilon}(f_W;S):=\sup_{q\in\mathbb{R}_+^n}\left\{\frac{1}{n}\sum_{i=1}^{n}q_i\ell(f_W(X_i),Y_i):\frac{1}{n}\sum_{i=1}^{n}q_i=1,\ \frac{1}{n}\sum_{i=1}^{n}q_i^c\leq1+c(c-1)\epsilon\right\}. \tag*{(4)} Φ DRO , c , ϵ ( f W ; S ) := q ∈ R + n sup { n 1 i = 1 ∑ n q i ℓ ( f W ( X i ) , Y i ) : n 1 i = 1 ∑ n q i = 1 , n 1 i = 1 ∑ n q i c ≤ 1 + c ( c − 1 ) ϵ } . (4)
Appendix A.3 shows the density-ratio and dual forms used later.
For notational clarity, we write Φ ⋆ , n ( f ; S ) ≡ Φ ⋆ ( f ; S ) \Phi_{\star,n}(f;S) \equiv\Phi_{\star}(f;S) Φ ⋆ , n ( f ; S ) ≡ Φ ⋆ ( f ; S ) when emphasizing that an objective is empirical, and, for fixed objective parameters, Δ ⋆ : = Φ ⋆ , n ( f W ; S ′ ) − Φ ⋆ , n ( f W ; S ) \Delta_{\star} := \Phi_{\star,n}(f_W;S')-\Phi_{\star,n}(f_W;S) Δ ⋆ := Φ ⋆ , n ( f W ; S ′ ) − Φ ⋆ , n ( f W ; S ) ; for example, Δ D R O = Φ D R O , c , ϵ , n ( f W ; S ′ ) − Φ D R O , c , ϵ , n ( f W ; S ) \Delta_{\mathrm{DRO}}=\Phi_{\mathrm{DRO},c,\epsilon,n}(f_W;S')-\Phi_{\mathrm{DRO},c,\epsilon,n}(f_W;S) Δ DRO = Φ DRO , c , ϵ , n ( f W ; S ′ ) − Φ DRO , c , ϵ , n ( f W ; S ) .
Why ordinary eCMI does not directly apply. For ordinary losses, once a predictor f f f is fixed, every row of the supersample table is evaluated by the same pointwise loss ℓ ( f , ⋅ ) \ell(f,\cdot) ℓ ( f , ⋅ ) . Flipping one selector swaps a single training example with its validation counterpart, and only that row changes. MC does not have this row-wise form: its value first looks at the whole batch and then selects a witness π ^ = ( c ^ , h ^ , η ^ ) ∈ Π M C \hat{\pi}=(\hat{c},\hat{h},\hat{\eta})\in\Pi_{\mathrm{MC}} π ^ = ( c ^ , h ^ , η ^ ) ∈ Π MC . In the supersample notation, let π ^ t r \hat{\pi}_{\mathrm{tr}} π ^ tr and π ^ v a l \hat{\pi}_{\mathrm{val}} π ^ val be witnesses selected for Φ M C , n ( f W ; Z ~ U ) \Phi_{\mathrm{MC},n}(f_W;\widetilde{Z}_U) Φ MC , n ( f W ; Z U ) and Φ M C , n ( f W ; Z ~ U ‾ ) \Phi_{\mathrm{MC},n}(f_W;\widetilde{Z}_{\overline{U}}) Φ MC , n ( f W ; Z U ) , respectively. A naive eCMI argument could freeze π ^ t r \hat{\pi}_{\mathrm{tr}} π ^ tr and compare the same signed MC average on the training and validation batches. The target gap, however, is Φ M C , n ( f W ; Z ~ U ‾ ) − Φ M C , n ( f W ; Z ~ U ) \Phi_{\mathrm{MC},n}(f_W;\widetilde{Z}_{\overline{U}})-\Phi_{\mathrm{MC},n}(f_W;\widetilde{Z}_U) Φ MC , n ( f W ; Z U ) − Φ MC , n ( f W ; Z U ) , where the validation batch may choose the different witness π ^ v a l \hat{\pi}_{\mathrm{val}} π ^ val . Bounding this mismatch by ranging uniformly over all witnesses gives a correct but uniform-convergence-type argument over Π M C \Pi_{\mathrm{MC}} Π MC , rather than the signed algorithm-dependent cancellation exploited by eCMI. The same issue for CVaR thresholds and DRO weights motivates this framework.
Main Results
Sections 3.1–3.3 control the optimized validation–training term Δ Φ : = Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) \Delta_{\Phi}:=\Phi_n(f_W;S')-\Phi_n(f_W;S) Δ Φ := Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) . Section 3.1 proves the fCMI bound using an objective-dependent sub-Gaussian parameter. Sections 3.2–3.3 bound this parameter using stability and convex L 2 L_2 L 2 -Lipschitz continuity. Throughout these subsections, write I f C M I : = I ( f W ( X ~ ) ; U ∣ Z ~ ) I_{\mathrm{fCMI}}:=I(f_W(\widetilde{X});U\mid\widetilde{Z}) I fCMI := I ( f W ( X ) ; U ∣ Z ) . Section 3.4 then adds objective-specific population–validation comparisons to obtain population–training statements.
Information-theoretic bound for set-input optimization-valued objectives
The examples in Section 2.2 reveal two structural departures from ordinary pointwise losses. First, the empirical criterion is set-input : it is a function of the whole batch of predictions and labels, rather than a sum of fixed row-wise losses. Second, it is optimization-valued : after the batch is observed, an auxiliary object is chosen by solving a batch-level optimization problem. We call objectives with both features set-input optimization-valued objectives.
Formally, let Π \Pi Π be a nonempty auxiliary space. A candidate π ∈ Π \pi\in\Pi π ∈ Π specifies a measurable score ρ π : V × Z → R \rho_{\pi}:\mathcal{V}\times\mathcal{Z}\to\mathbb{R} ρ π : V × Z → R and a post-aggregation map ψ π : R → R \psi_{\pi}:\mathbb{R}\to\mathbb{R} ψ π : R → R . The score input is ( v , z ) = ( f w ( x ) , ( x , y ) ) (v,z)=(f_w(x),(x,y)) ( v , z ) = ( f w ( x ) , ( x , y )) , so the objective value on a batch is determined by the pairs ( f w ( x i ) , z i ) (f_w(x_i),z_i) ( f w ( x i ) , z i ) . Given a predictor f w ∈ F f_w\in\mathcal{F} f w ∈ F and a sample s = ( ( x i , y i ) ) i = 1 n ∈ Z n s=((x_i,y_i))_{i=1}^{n}\in\mathcal{Z}^{n} s = (( x i , y i ) ) i = 1 n ∈ Z n , define
Φ n ( f w ; s ) : = Opt π ∈ Π ψ π ( 1 n ∑ i = 1 n ρ π ( f w ( x i ) , z i ) ) , Opt ∈ { sup , inf } . (5) \Phi_n(f_w;s):=\operatorname{Opt}_{\pi\in\Pi}\psi_{\pi}\left(\frac{1}{n}\sum_{i=1}^{n}\rho_{\pi}(f_w(x_i),z_i)\right),\qquad\operatorname{Opt}\in\{\sup,\inf\}. \tag*{(5)} Φ n ( f w ; s ) := Opt π ∈ Π ψ π ( n 1 i = 1 ∑ n ρ π ( f w ( x i ) , z i ) ) , Opt ∈ { sup , inf } . (5)
The population target is Φ ( f w ; D ) : = Opt π ∈ Π ψ π ( E Z ∼ D ρ π ( f w ( X ) , Z ) ) \Phi(f_w;D):=\operatorname{Opt}_{\pi\in\Pi}\psi_{\pi}\left(\mathbb{E}_{Z\sim D}\rho_{\pi}(f_w(X),Z)\right) Φ ( f w ; D ) := Opt π ∈ Π ψ π ( E Z ∼ D ρ π ( f w ( X ) , Z ) ) . Throughout this section, optimized values are measurable and standard measurability caveats for uncountable Π \Pi Π are in Appendix E. MC uses π = ( c , h , η ) \pi=(c,h,\eta) π = ( c , h , η ) , CVaR uses π = τ \pi=\tau π = τ , and Cressie–Read DRO enters Eq. (5) through the dual form in Appendix A.4; its sharper L 2 L_2 L 2 certificate later uses the primal reweighting form. Conditional on ( f W ( X ~ ) , Z ~ ) (f_W(\widetilde{X}),\widetilde{Z}) ( f W ( X ) , Z ) , all values { Φ n ( f W ; Z ~ u ) : u ∈ { 0 , 1 } n } \{\Phi_n(f_W;\widetilde{Z}_u):u\in\{0,1\}^{n}\} { Φ n ( f W ; Z u ) : u ∈ { 0 , 1 } n } are fixed. Thus the analysis controls the optimized mismatch directly.
Given a fixed predictor f w f_w f w , a supersample z ~ \widetilde{z} z , and a selector vector u u u , define the selector gap
Γ f w , z ~ ( u ) : = Φ n ( f w ; z ~ u ‾ ) − Φ n ( f w ; z ~ u ) , u ∈ { 0 , 1 } n . (6) \Gamma_{f_w,\widetilde{z}}(u):=\Phi_n(f_w;\widetilde{z}_{\overline{u}})-\Phi_n(f_w;\widetilde{z}_u),\qquad u\in\{0,1\}^{n}. \tag*{(6)} Γ f w , z ( u ) := Φ n ( f w ; z u ) − Φ n ( f w ; z u ) , u ∈ { 0 , 1 } n . (6)
The selector gap is antisymmetric: replacing u u u by u ‾ \overline{u} u exchanges the two samples and reverses the sign of the gap. Pairing every selector vector with its complement under the uniform selector law gives E U ∼ Unif ( { 0 , 1 } n ) Γ f w , z ~ ( U ) = 0 \mathbb{E}_{U\sim\operatorname{Unif}(\{0,1\}^{n})}\Gamma_{f_w,\widetilde{z}}(U)=0 E U ∼ Unif ({ 0 , 1 } n ) Γ f w , z ( U ) = 0 ; see Lemma 4 for the proof. We impose the following sub-Gaussian control on this selector gap.
Assumption 1 (Sub-Gaussianity). There is a deterministic constant L Γ , n < ∞ L_{\Gamma,n}<\infty L Γ , n < ∞ , uniform over fixed predictors f w f_w f w and supersample realizations z ~ \widetilde{z} z , such that, for Q U = Unif ( { 0 , 1 } n ) Q_U=\operatorname{Unif}(\{0,1\}^{n}) Q U = Unif ({ 0 , 1 } n ) ,
log E Q U exp { λ Γ f w , z ~ ( U ) } ≤ λ 2 L Γ , n 2 2 , ∀ λ ∈ R . \log\mathbb{E}_{Q_U}\exp\{\lambda\Gamma_{f_w,\widetilde{z}}(U)\}\leq\frac{\lambda^{2}L_{\Gamma,n}^{2}}{2},\qquad\forall\lambda\in\mathbb{R}. log E Q U exp { λ Γ f w , z ( U )} ≤ 2 λ 2 L Γ , n 2 , ∀ λ ∈ R .
Assumption 1 is fixed-predictor, fixed-supersample concentration under Q U Q_U Q U ; it summarizes objective geometry, not the actual conditional selector law P U ∣ f W ( X ~ ) , Z ~ P_{U\mid f_W(\widetilde{X}),\widetilde{Z}} P U ∣ f W ( X ) , Z , which is charged through I f C M I I_{\mathrm{fCMI}} I fCMI . Assumption 1 is verified below through either the stability certificate of § 3.2 or the convex L 2 L_2 L 2 -Lipschitz certificate of § 3.3 ; Table 1 summarizes the resulting bounds on L Γ , n L_{\Gamma,n} L Γ , n .
Theorem 1 (Validation–Training bound). Under Assumption 1, we have
∣ E Δ Φ ∣ ≤ L Γ , n 2 I f C M I , (7) \lvert\mathbb{E}\Delta_{\Phi}\rvert\le L_{\Gamma,n}\sqrt{2I_{\mathrm{fCMI}}}, \tag*{(7)} ∣ E Δ Φ ∣ ≤ L Γ , n 2 I fCMI , (7) E ∣ Δ Φ ∣ ≤ L Γ , n 2 ( I f C M I + log 2 ) . (8) \mathbb{E}\lvert\Delta_{\Phi}\rvert\le L_{\Gamma,n}\sqrt{2\left(I_{\mathrm{fCMI}}+\log2\right)}. \tag*{(8)} E ∣ Δ Φ ∣ ≤ L Γ , n 2 ( I fCMI + log 2 ) . (8)
Interpretation. Theorem 1 factorizes the bound on the optimized validation–training gap into a learning-dependent term and an objective-dependent term. To see this, condition on G = ( f W ( X ~ ) , Z ~ ) G=(f_W(\widetilde{X}),\widetilde{Z}) G = ( f W ( X ) , Z ) . Conditional on G G G , all optimized values { Φ n ( f W ; Z ~ u ) : u ∈ { 0 , 1 } n } \{\Phi_n(f_W;\widetilde{Z}_u):u\in\{0,1\}^n\} { Φ n ( f W ; Z u ) : u ∈ { 0 , 1 } n } are fixed, and the auxiliary optimizer has already been absorbed into the deterministic function u ↦ Γ f W , Z ~ ( u ) u\mapsto\Gamma_{f_W,\widetilde{Z}}(u) u ↦ Γ f W , Z ( u ) . The Donsker–Varadhan step therefore compares the actual conditional selector law P U ∣ G P_{U\mid G} P U ∣ G with the uniform reference law Q U Q_U Q U , paying only D ( P U ∣ G ∥ Q U ) D(P_{U\mid G}\Vert Q_U) D ( P U ∣ G ∥ Q U ) . Averaging over G G G gives I ( f W ( X ~ ) ; U ∣ Z ~ ) = I f C M I I(f_W(\widetilde{X});U\mid\widetilde{Z})=I_{\mathrm{fCMI}} I ( f W ( X ) ; U ∣ Z ) = I fCMI . Thus I f C M I I_{\mathrm{fCMI}} I fCMI measures how much the learned supersample prediction table depends on which half of each pair was used for training.
The other factor, L Γ , n L_{\Gamma,n} L Γ , n , captures the geometry of the optimized objective under the reference selector law. Auxiliary optimization is not free; its effect enters through the moment-generating function in Assumption 1. What is avoided in the validation–training term is an explicit uniform-convergence penalty over Π \Pi Π , such as an entropy, cardinality, covering-number, or Rademacher complexity term. This is the desired behavior for the validation–training term: if the learned predictor is independent of the selector vector, then ∣ E Δ Φ ∣ = 0 \lvert\mathbb{E}\Delta_{\Phi}\rvert=0 ∣ E Δ Φ ∣ = 0 by symmetry, regardless of the size of Π \Pi Π . Eq. (7) reflects this cancellation, since the right-hand side vanishes when I f C M I = 0 I_{\mathrm{fCMI}}=0 I fCMI = 0 .
§ 3.2 –§ 3.3 are devoted to certifying L Γ , n L_{\Gamma,n} L Γ , n . Different certificates expose different structural properties of the same set-input objective. The stability route (§ 3.2 ) controls coordinate-wise replacement sensitivity, corresponding to an L ∞ L_{\infty} L ∞ -type view of the optimizer. The convex L 2 L_2 L 2 -Lipschitz route (§ 3.3 ) instead uses Euclidean geometry of the score or weight vector, which is sharper when the optimizer spreads mass across the batch. Consequently, the same algorithmic quantity I f C M I I_{\mathrm{fCMI}} I fCMI pairs with different sub-Gaussian parameters depending on the objective: MC is controlled by bounded linear witnesses, CVaR reflects the tail level α \alpha α , and Cressie–Read DRO distinguishes the L ∞ L_{\infty} L ∞ stability scale from the sharper L 2 L_2 L 2 reweighting scale. Table 1 summarizes these choices.
Stability certificate and single-index refinement
Our first certificate is uniform stability, a sufficient condition for Assumption 1:
Assumption 2 (Uniform stability). There exists a possibly n n n -dependent constant β n > 0 \beta_n>0 β n > 0 such that, for every fixed predictor f W f_W f W and every pair of samples s , s ′ ∈ Z n s,s'\in\mathcal{Z}^n s , s ′ ∈ Z n that differ in at most one coordinate,
∣ Φ n ( f W ; s ) − Φ n ( f W ; s ′ ) ∣ ≤ β n . \lvert\Phi_n(f_W;s)-\Phi_n(f_W;s')\rvert\le\beta_n. ∣ Φ n ( f W ; s ) − Φ n ( f W ; s ′ )∣ ≤ β n .
Lemma 1 (Stability certificate). Under Assumption 2, Assumption 1 holds with L Γ , n ≤ β n n L_{\Gamma,n}\le\beta_n\sqrt{n} L Γ , n ≤ β n n and
∣ E Δ Φ ∣ ≤ β n 2 n I f C M I . (9) \lvert\mathbb{E}\Delta_{\Phi}\rvert\le\beta_n\sqrt{2nI_{\mathrm{fCMI}}}. \tag*{(9)} ∣ E Δ Φ ∣ ≤ β n 2 n I fCMI . (9)
In the standard eCMI bound in Eq. (1) , the information term is row-wise. For set-input objectives, the following theorem combines the fCMI bound using the full vector U U U with a single-index bound.
Theorem 2 (Single-index stability bound). Define, for each t ∈ [ n ] t\in[n] t ∈ [ n ] and b ∈ { 0 , 1 } b\in\{0,1\} b ∈ { 0 , 1 } , S ( t , b ) : = ( Z ~ 1 , U 1 , … , Z ~ t − 1 , U t − 1 , Z ~ t , b , Z ~ t + 1 , U t + 1 , … , Z ~ n , U n ) S^{(t,b)}:=(\widetilde{Z}_1,U_1,\ldots,\widetilde{Z}_{t-1},U_{t-1},\widetilde{Z}_{t,b},\widetilde{Z}_{t+1},U_{t+1},\ldots,\widetilde{Z}_n,U_n) S ( t , b ) := ( Z 1 , U 1 , … , Z t − 1 , U t − 1 , Z t , b , Z t + 1 , U t + 1 , … , Z n , U n ) . Set D t : = Φ n ( f W ; S ( t , 0 ) ) − Φ n ( f W ; S ( t , 1 ) ) D_t:=\Phi_n(f_W;S^{(t,0)})-\Phi_n(f_W;S^{(t,1)}) D t := Φ n ( f W ; S ( t , 0 ) ) − Φ n ( f W ; S ( t , 1 ) ) and ϵ t : = 2 U t − 1 \epsilon_t:=2U_t-1 ϵ t := 2 U t − 1 . If Assumption 2 holds, then ∣ D t ∣ ≤ β n \lvert D_t\rvert\le\beta_n ∣ D t ∣ ≤ β n , each D t D_t D t is measurable with respect to ( f W ( X ~ ) , Z ~ , U − t ) (f_W(\widetilde{X}),\widetilde{Z},U_{-t}) ( f W ( X ) , Z , U − t ) , and
Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) = ∑ t = 1 n ϵ t D t . (10) \Phi_n(f_W;S')-\Phi_n(f_W;S)=\sum_{t=1}^{n}\epsilon_tD_t. \tag*{(10)} Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) = t = 1 ∑ n ϵ t D t . (10) For the local pieces, write I l o c , t : = I ( D t ; U t ∣ Z ~ ) I_{\mathrm{loc},t}:=I(D_t;U_t\mid\widetilde{Z}) I loc , t := I ( D t ; U t ∣ Z ) . Then
∣ E Δ Φ ∣ ≤ β n min { 2 n I f C M I , ∑ t = 1 n 2 I l o c , t } . (11) \lvert\mathbb{E}\Delta_{\Phi}\rvert\le\beta_n\min\left\{\sqrt{2nI_{\mathrm{fCMI}}},\sum_{t=1}^{n}\sqrt{2I_{\mathrm{loc},t}}\right\}. \tag*{(11)} ∣ E Δ Φ ∣ ≤ β n min { 2 n I fCMI , t = 1 ∑ n 2 I loc , t } . (11)
Table 1. Upper bounds on L Γ , n L_{\Gamma,n} L Γ , n under stability and convex L 2 L_2 L 2 -Lipschitz certificates.
Eq. (11) uses the sharper of two routes. The global term, I f C M I I_{\mathrm{fCMI}} I fCMI , uses information from the full prediction table; VC or growth-function controls apply only after the table is binary or otherwise finite-valued, with the binary case given by Lemma 9. The local term, I l o c , t I_{\mathrm{loc},t} I loc , t , uses scalar information for the pairs ( D t , U t ) (D_t,U_t) ( D t , U t ) and is often easier to estimate directly. For pointwise empirical risks, the single-index decomposition recovers the row-wise structure underlying standard eCMI bounds and their fast-rate refinements; see Appendices C and § 15 .§ 4 .
We now apply the stability framework to the MC, CVaR, and DRO objectives introduced in § 2.2 . In each case, the task is to evaluate the corresponding stability coefficient. The global information I f C M I I_{\mathrm{fCMI}} I fCMI is common across objectives, while the local information is objective-specific: for ⋆ ∈ { M C , C V a R , D R O } \star\in\{\mathrm{MC},\mathrm{CVaR},\mathrm{DRO}\} ⋆ ∈ { MC , CVaR , DRO } , write I l o c , t ⋆ : = I ( D t ⋆ ; U t ∣ Z ~ ) I_{\mathrm{loc},t}^{\star}:=I(D_t^{\star};U_t\mid\widetilde{Z}) I loc , t ⋆ := I ( D t ⋆ ; U t ∣ Z ) .
Corollary 1 (Multicalibration). For the MC objective (2) , Assumption 2 2 2 holds with β n = β M C , n : = 2 / n \beta_n=\beta_{\mathrm{MC},n}:=2/n β n = β MC , n := 2/ n . Writing D t M C D_t^{\mathrm{MC}} D t MC for the local increment in Theorem 2 specialized to MC, Theorem 2 gives
∣ E Δ M C ∣ ≤ 2 n min { 2 n I f C M I , ∑ t = 1 n 2 I l o c , t M C } . (12) |\mathbb{E}\Delta_{\mathrm{MC}}| \le\frac{2}{n}\min\left\{\sqrt{2nI_{\mathrm{fCMI}}},\sum_{t=1}^{n}\sqrt{2I_{\mathrm{loc},t}^{\mathrm{MC}}}\right\}. \tag*{(12)} ∣ E Δ MC ∣ ≤ n 2 min { 2 n I fCMI , t = 1 ∑ n 2 I loc , t MC } . (12)
We next show that I f C M I I_{\mathrm{fCMI}} I fCMI can be controlled by the number of distinct prediction tables. For MC, since V \mathcal{V} V is real-valued, we focus on the pseudo-dimension [27 ] , Pdim ( F ) \operatorname{Pdim}(\mathcal{F}) Pdim ( F ) .
Corollary 2 (Binned MC with real-valued scores). Let I 1 , … , I B I_1,\ldots,I_B I 1 , … , I B be a fixed partition of [ 0 , 1 ] [0,1] [ 0 , 1 ] , take the MC score-test class H B : = { v ↦ 1 { v ∈ I b } : b ∈ [ B ] } \mathcal{H}_B:=\{v\mapsto\mathbf{1}\{v\in I_b\}:b\in[B]\} H B := { v ↦ 1 { v ∈ I b } : b ∈ [ B ]} , and let Φ M C , B \Phi_{\mathrm{MC},B} Φ MC , B be the corresponding binned objective from Appendix § 16 . Assume that the pseudo-dimension Pdim ( F ) = d 0 < ∞ \operatorname{Pdim}(\mathcal{F})=d_0<\infty Pdim ( F ) = d 0 < ∞ . Then, with Δ M C , B : = Φ M C , B ( f W ; S ′ ) − Φ M C , B ( f W ; S ) \Delta_{\mathrm{MC},B}:=\Phi_{\mathrm{MC},B}(f_W;S')-\Phi_{\mathrm{MC},B}(f_W;S) Δ MC , B := Φ MC , B ( f W ; S ′ ) − Φ MC , B ( f W ; S ) ,
∣ E Δ M C , B ∣ ≲ d 0 log { e n ( B + n ) / d 0 } n + 1 n . (13) |\mathbb{E}\Delta_{\mathrm{MC},B}|\lesssim\sqrt{\frac{d_0\log\{en(B+n)/d_0\}}{n}}+\frac{1}{n}. \tag*{(13)} ∣ E Δ MC , B ∣ ≲ n d 0 log { e n ( B + n ) / d 0 } + n 1 . (13)
The same stability argument applies to CVaR and DRO. To keep the main text concise, we report the stability coefficient β n \beta_n β n and the fCMI/local-information bounds; complete statements and proofs are in Appendix § 13 . The stability/local-information bounds in Corollaries 1, 3, and 4 require no complexity assumption. VC/growth rates are used only after the prediction table is binary or otherwise finite-valued; Lemma 9 gives the binary case, and real-valued binned scores require the discretized pseudo-dimension control of Corollary 2 and Appendix § 16 .
Corollary 3 (CVaR). For the CVaR objective (3) , Assumption 2 2 2 holds with β n = β C V a R , n : = ( α ˉ n ) − 1 \beta_n=\beta_{\mathrm{CVaR},n}:=(\bar{\alpha}n)^{-1} β n = β CVaR , n := ( α ˉ n ) − 1 . Writing D t C V a R D_t^{\mathrm{CVaR}} D t CVaR for the corresponding local increment, Theorem 2 gives
∣ E Δ C V a R ∣ ≤ 1 α ˉ n min { 2 n I f C M I , ∑ t = 1 n 2 I l o c , t C V a R } . (14) |\mathbb{E}\Delta_{\mathrm{CVaR}}|\le\frac{1}{\bar{\alpha}n}\min\left\{\sqrt{2nI_{\mathrm{fCMI}}},\sum_{t=1}^{n}\sqrt{2I_{\mathrm{loc},t}^{\mathrm{CVaR}}}\right\}. \tag*{(14)} ∣ E Δ CVaR ∣ ≤ α ˉ n 1 min { 2 n I fCMI , t = 1 ∑ n 2 I loc , t CVaR } . (14)
Corollary 4 (CR DRO). Let c > 1 c>1 c > 1 , c ∗ = c / ( c − 1 ) c^*=c/(c-1) c ∗ = c / ( c − 1 ) , and κ c , ϵ : = { 1 + c ( c − 1 ) ϵ } 1 / c \kappa_{c,\epsilon}:=\{1+c(c-1)\epsilon\}^{1/c} κ c , ϵ := { 1 + c ( c − 1 ) ϵ } 1/ c . For the empirical CR DRO objective (4) , Assumption 2 2 2 holds with β n = β D R O , n : = κ c , ϵ n 1 / c − 1 \beta_n=\beta_{\mathrm{DRO},n}:=\kappa_{c,\epsilon}n^{1/c-1} β n = β DRO , n := κ c , ϵ n 1/ c − 1 . Writing D t D R O D_t^{\mathrm{DRO}} D t DRO for the corresponding local increment, Theorem 2 gives
∣ E Δ D R O ∣ ≤ κ c , ϵ n 1 / c − 1 min { 2 n I f C M I , ∑ t = 1 n 2 I l o c , t D R O } . (15) |\mathbb{E}\Delta_{\mathrm{DRO}}|\le\kappa_{c,\epsilon}n^{1/c-1}\min\left\{\sqrt{2nI_{\mathrm{fCMI}}},\sum_{t=1}^{n}\sqrt{2I_{\mathrm{loc},t}^{\mathrm{DRO}}}\right\}. \tag*{(15)} ∣ E Δ DRO ∣ ≤ κ c , ϵ n 1/ c − 1 min { 2 n I fCMI , t = 1 ∑ n 2 I loc , t DRO } . (15)
For binary prediction models from a class of VC dimension d d d , Lemma 9 gives I f C M I ≲ d log ( e n / d ) I_{\mathrm{fCMI}}\lesssim d\log(en/d) I fCMI ≲ d log ( e n / d ) . In that discrete-table case, the global route scales as O ( α ˉ − 1 d log ( e n / d ) / n ) O(\bar{\alpha}^{-1}\sqrt{d\log(en/d)/n}) O ( α ˉ − 1 d log ( e n / d ) / n ) for CVaR and O ( κ c , ϵ n 1 / c − 1 / 2 d log ( e n / d ) ) \mathcal{O}(\kappa_{c,\epsilon}n^{1/c-1/2}\sqrt{d\log(en/d)}) O ( κ c , ϵ n 1/ c − 1/2 d log ( e n / d ) ) for stability-based DRO. This stability-based DRO bound is generally suboptimal: Duchi and Namkoong [6 ] showed that the relevant Cressie–Read plug-in rate scales as n − 1 / ( c ∗ ∨ 2 ) n^{-1/(c^*\vee2)} n − 1/ ( c ∗ ∨ 2 ) . The stability route is governed by the coordinate-wise ℓ ∞ \ell_\infty ℓ ∞ sensitivity of the adversarial weights, whereas the sharper CR DRO bound is governed by their ℓ 2 \ell_2 ℓ 2 geometry. This leads to the convex L 2 L_2 L 2 -Lipschitz certificate in § 3.3 .
Convex L 2 L_2 L 2 -Lipschitz certificate
The stability route is sharp when a one-point replacement has small effect, but it can be too crude for objectives whose optimizer spreads mass across the whole batch. We therefore introduce a geometric certificate based on convex L 2 L_2 L 2 -concentration.
Assumption 3 (Convex L 2 L_2 L 2 -Lipschitz score-vector assumption). For every fixed predictor f w f_w f w and sample s = ( z 1 , … , z n ) s=(z_1,\ldots,z_n) s = ( z 1 , … , z n ) , the empirical objective in Eq. (5) can be written as Φ n ( f w ; s ) = ψ ~ ( R f w , n ( s ) ) \Phi_n(f_w;s)=\widetilde{\psi}(R_{f_w,n}(s)) Φ n ( f w ; s ) = ψ ( R f w , n ( s )) , where R f w , n : Z n → [ 0 , 1 ] n R_{f_w,n}:\mathcal{Z}^n\to[0,1]^n R f w , n : Z n → [ 0 , 1 ] n is coordinate-wise, meaning that its i i i -th coordinate depends only on z i z_i z i , and ψ ~ : [ 0 , 1 ] n → R \widetilde{\psi}:[0,1]^n\to\mathbb{R} ψ : [ 0 , 1 ] n → R is convex. Moreover, ψ ~ \widetilde{\psi} ψ is L ψ , n L_{\psi,n} L ψ , n -Lipschitz with respect to the Euclidean norm, meaning that for any r , r ′ ∈ [ 0 , 1 ] n r,r'\in[0,1]^n r , r ′ ∈ [ 0 , 1 ] n , ∣ ψ ~ ( r ) − ψ ~ ( r ′ ) ∣ ≤ L ψ , n ∥ r − r ′ ∥ 2 |\widetilde{\psi}(r)-\widetilde{\psi}(r')|\le L_{\psi,n}\|r-r'\|_2 ∣ ψ ( r ) − ψ ( r ′ ) ∣ ≤ L ψ , n ∥ r − r ′ ∥ 2 .
Under this convex L 2 L_2 L 2 -Lipschitz assumption, Assumption 1 follows directly. Here the relevant scale is 2 I f C M I \sqrt{2I_{\mathrm{fCMI}}} 2 I fCMI , whereas the stability route uses 2 n I f C M I \sqrt{2nI_{\mathrm{fCMI}}} 2 n I fCMI .
Lemma 2 (Convex L 2 L_2 L 2 -Lipschitz certificate). Under Assumption 3, Assumption 1 holds with L Γ , n ≤ 2 L ψ , n L_{\Gamma,n}\le2L_{\psi,n} L Γ , n ≤ 2 L ψ , n and ∣ E Δ Φ ∣ ≤ 2 L ψ , n 2 I f C M I |\mathbb{E}\Delta_\Phi|\le2L_{\psi,n}\sqrt{2I_{\mathrm{fCMI}}} ∣ E Δ Φ ∣ ≤ 2 L ψ , n 2 I fCMI .
For MC, the stability bound in Corollary 1 is sharper, but the same geometry is transparent: MC is a supremum of linear witness averages, and has a convex extension obtained by interpolating the witness scores; Lemma 6 gives the MC entry in Table 1.
Corollary 5 (CVaR via the convex L 2 L_2 L 2 certificate). Using the risk-envelope form of Appendix § 9 .§ 2 , the CVaR objective (3) satisfies Assumption 3 with L ψ , n = ( α ˉ n ) − 1 / 2 L_{\psi,n}=(\bar{\alpha}n)^{-1/2} L ψ , n = ( α ˉ n ) − 1/2 . Combining this with Corollary 3, we have
Γ n C V a R ≤ min { α ˉ − 1 n − 1 / 2 , 2 ( α ˉ n ) − 1 / 2 } , (16) \Gamma^{\mathrm{CVaR}}_n\le\min\{\bar{\alpha}^{-1}n^{-1/2},\,2(\bar{\alpha}n)^{-1/2}\}, \tag*{(16)} Γ n CVaR ≤ min { α ˉ − 1 n − 1/2 , 2 ( α ˉ n ) − 1/2 } , (16)
and
∣ E Δ C V a R ∣ ≤ min { α ˉ − 1 n − 1 / 2 , 2 ( α ˉ n ) − 1 / 2 } 2 I f C M I . |\mathbb{E}\Delta_{\mathrm{CVaR}}|\le\min\{\bar{\alpha}^{-1}n^{-1/2},\,2(\bar{\alpha}n)^{-1/2}\}\sqrt{2I_{\mathrm{fCMI}}}. ∣ E Δ CVaR ∣ ≤ min { α ˉ − 1 n − 1/2 , 2 ( α ˉ n ) − 1/2 } 2 I fCMI .
Thus, depending on the tail level, the convex L 2 L_2 L 2 route may improve the constant relative to the stability route. More importantly, for CR DRO it improves the rate itself.
Corollary 6 (CR DRO). Let c > 1 c>1 c > 1 , c ∗ = c / ( c − 1 ) c^*=c/(c-1) c ∗ = c / ( c − 1 ) , and κ c , ϵ : = { 1 + c ( c − 1 ) ϵ } 1 / c \kappa_{c,\epsilon}:=\{1+c(c-1)\epsilon\}^{1/c} κ c , ϵ := { 1 + c ( c − 1 ) ϵ } 1/ c . For the empirical CR DRO objective (4) , Assumption 3 holds with L ψ , n D R O = κ c , ϵ n − 1 / ( c ∗ ∨ 2 ) L_{\psi,n}^{\mathrm{DRO}}=\kappa_{c,\epsilon}n^{-1/(c^*\vee2)} L ψ , n DRO = κ c , ϵ n − 1/ ( c ∗ ∨ 2 ) . Consequently,
∣ E Δ D R O ∣ ≤ 2 κ c , ϵ n − 1 / ( c ∗ ∨ 2 ) 2 I f C M I . (17) |\mathbb{E}\Delta_{\mathrm{DRO}}|\le2\kappa_{c,\epsilon}n^{-1/(c^*\vee2)}\sqrt{2I_{\mathrm{fCMI}}}. \tag*{(17)} ∣ E Δ DRO ∣ ≤ 2 κ c , ϵ n − 1/ ( c ∗ ∨ 2 ) 2 I fCMI . (17)
Compared with the stability result in Corollary 4, the convex L 2 L_2 L 2 certificate replaces coordinate-wise ℓ ∞ \ell_\infty ℓ ∞ sensitivity by the Euclidean geometry of the adversarial weight vector. In particular, for binary prediction tables generated by a class of VC dimension d d d , Lemma 9 gives I f C M I ≲ d log ( e n / d ) I_{\mathrm{fCMI}}\lesssim d\log(en/d) I fCMI ≲ d log ( e n / d ) , and hence ∣ E Δ D R O ∣ = O ( κ c , ϵ n − 1 / ( c ∗ ∨ 2 ) d log ( e n / d ) ) |\mathbb{E}\Delta_{\mathrm{DRO}}|=\mathcal{O}(\kappa_{c,\epsilon}n^{-1/(c^*\vee2)}\sqrt{d\log(en/d)}) ∣ E Δ DRO ∣ = O ( κ c , ϵ n − 1/ ( c ∗ ∨ 2 ) d log ( e n / d ) ) . Thus the bound on L Γ , n L_{\Gamma,n} L Γ , n recovers the exponent of Duchi and Namkoong [6 ] , up to logarithmic factors, while the additional fCMI factor accounts for algorithmic dependence on the training selector vector.
Population–training generalization bound
We connect the preceding validation–training bounds to the population target through
Φ ( f W ; D ) − Φ n ( f W ; S ) = { Φ ( f W ; D ) − Φ n ( f W ; S ′ ) } + { Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) } . (18) \Phi(f_W;\mathcal{D})-\Phi_n(f_W;S)=\{\Phi(f_W;\mathcal{D})-\Phi_n(f_W;S')\}+\{\Phi_n(f_W;S')-\Phi_n(f_W;S)\}. \tag*{(18)} Φ ( f W ; D ) − Φ n ( f W ; S ) = { Φ ( f W ; D ) − Φ n ( f W ; S ′ )} + { Φ n ( f W ; S ′ ) − Φ n ( f W ; S )} . (18)
The second term is controlled by Theorem 1. Here each row contains two i.i.d. draws and U U U only orders them, so ( S , S ′ ) ∼ D n ⊗ D n (S,S')\sim\mathcal{D}^n\otimes\mathcal{D}^n ( S , S ′ ) ∼ D n ⊗ D n ; hence S ′ ⊥ ( S , R ) S'\perp(S,R) S ′ ⊥ ( S , R ) and S ′ ⊥ f W = A ( S , R ) S'\perp f_W=A(S,R) S ′ ⊥ f W = A ( S , R ) . The first term is a fixed-predictor population–validation comparison. This split separates the fCMI-controlled validation–training term from the objective-specific approximation of the population metric by an independent batch, possibly involving empirical-process terms over witnesses, thresholds, or weights.
Absolute comparisons. Given S ′ = ( Z i ′ ) i = 1 n S' = (Z'_i)_{i=1}^n S ′ = ( Z i ′ ) i = 1 n , the fixed-predictor term is controlled by the corresponding empirical process. In the average-type form of Eq. (5) , write μ π ( f ) : = E Z ∼ D ρ π ( f ( X ) , Z ) \mu_\pi(f) := \mathbb{E}_{Z \sim D}\rho_\pi(f(X), Z) μ π ( f ) := E Z ∼ D ρ π ( f ( X ) , Z ) and μ ^ π , n ( f ; S ′ ) : = n − 1 ∑ i = 1 n ρ π ( f ( X i ′ ) , Z i ′ ) \hat{\mu}_{\pi,n}(f; S') := n^{-1}\sum_{i=1}^n \rho_\pi(f(X'_i), Z'_i) μ ^ π , n ( f ; S ′ ) := n − 1 ∑ i = 1 n ρ π ( f ( X i ′ ) , Z i ′ ) . Then
∣ Φ ( f W ; D ) − Φ n ( f W ; S ′ ) ∣ ≤ sup π ∈ Π ∣ ψ π ( μ π ( f W ) ) − ψ π ( μ ^ π , n ( f W ; S ′ ) ) ∣ . \left|\Phi(f_W;D) - \Phi_n(f_W;S')\right| \le\sup_{\pi\in\Pi}\left|\psi_\pi\left(\mu_\pi(f_W)\right) - \psi_\pi\left(\hat{\mu}_{\pi,n}(f_W;S')\right)\right|. ∣ Φ ( f W ; D ) − Φ n ( f W ; S ′ ) ∣ ≤ π ∈ Π sup ∣ ψ π ( μ π ( f W ) ) − ψ π ( μ ^ π , n ( f W ; S ′ ) ) ∣ .
This term is controlled by objective-specific fixed-predictor uniform-convergence or plug-in arguments rather than by MI/CMI. For fixed f f f , write U C n Φ ( f ) : = E S ′ ∼ D n ∣ Φ ( f ; D ) − Φ n ( f ; S ′ ) ∣ \mathrm{UC}^{\Phi}_n(f) := \mathbb{E}_{S' \sim D^n}\left|\Phi(f;D) - \Phi_n(f;S')\right| UC n Φ ( f ) := E S ′ ∼ D n ∣ Φ ( f ; D ) − Φ n ( f ; S ′ ) ∣ . For MC fixed-predictor comparison, put C ± : = { η c : c ∈ C , η ∈ { ± 1 } } C_{\pm} := \{\eta c : c \in C, \eta\in\{\pm1\}\} C ± := { η c : c ∈ C , η ∈ { ± 1 }} . Given S = ( Z i ) i = 1 n = ( ( X i , Y i ) ) i = 1 n S = (Z_i)_{i=1}^n = ((X_i,Y_i))_{i=1}^n S = ( Z i ) i = 1 n = (( X i , Y i ) ) i = 1 n , for any class G \mathcal{G} G define R ^ S ( G ) : = E σ [ sup g ∈ G n − 1 ∑ i = 1 n σ i g ( Z i ) ] \hat{\mathcal{R}}_S(\mathcal{G}) := \mathbb{E}_{\sigma}\left[\sup_{g \in\mathcal{G}} n^{-1}\sum_{i=1}^n \sigma_i g(Z_i)\right] R ^ S ( G ) := E σ [ sup g ∈ G n − 1 ∑ i = 1 n σ i g ( Z i ) ] and R D , n ( G ) : = E S ∼ D n R ^ S ( G ) \mathcal{R}_{D,n}(\mathcal{G}) := \mathbb{E}_{S \sim D^n}\hat{\mathcal{R}}_S(\mathcal{G}) R D , n ( G ) := E S ∼ D n R ^ S ( G ) . Write N ( δ , H , ∥ ⋅ ∥ ∞ ) N(\delta,\mathcal{H},\|\cdot\|_{\infty}) N ( δ , H , ∥ ⋅ ∥ ∞ ) for the score-test sup-norm covering number.
Corollary 7 (Multicalibration). For the MC objective (2) ,
E [ U C n M C ( f W ) ] ≲ R D , n ( C ± ) + inf δ ≥ 0 { δ + ∫ δ 1 log N ( ε , H , ∥ ⋅ ∥ ∞ ) n d ε } , \mathbb{E}\left[\mathrm{UC}^{\mathrm{MC}}_n(f_W)\right] \lesssim\mathcal{R}_{D,n}(C_{\pm}) + \inf_{\delta\ge0}\left\{\delta+ \int_{\delta}^{1}\sqrt{\frac{\log N(\varepsilon,\mathcal{H},\|\cdot\|_{\infty})}{n}}\,\mathrm{d}\varepsilon\right\}, E [ UC n MC ( f W ) ] ≲ R D , n ( C ± ) + δ ≥ 0 inf { δ + ∫ δ 1 n log N ( ε , H , ∥ ⋅ ∥ ∞ ) d ε } ,
E ∣ Φ M C ( f W ; D ) − Φ M C , n ( f W ; S ) ∣ ≤ E [ U C n M C ( f W ) ] + 2 2 ( I F C M I + log 2 ) n . (19) \mathbb{E}\left|\Phi_{\mathrm{MC}}(f_W;D) - \Phi_{\mathrm{MC},n}(f_W;S)\right| \le\mathbb{E}\left[\mathrm{UC}^{\mathrm{MC}}_n(f_W)\right] + 2\sqrt{\frac{2(I_{\mathrm{FCMI}}+\log2)}{n}}. \tag*{(19)} E ∣ Φ MC ( f W ; D ) − Φ MC , n ( f W ; S ) ∣ ≤ E [ UC n MC ( f W ) ] + 2 n 2 ( I FCMI + log 2 ) . (19)
For finite C C C and H \mathcal{H} H , including standard binned MC, E [ U C n M C ( f W ) ] = O ( log ( 2 ∣ C ∣ ∣ H ∣ ) / n ) \mathbb{E}[\mathrm{UC}^{\mathrm{MC}}_n(f_W)] = O\left(\sqrt{\log(2|C||\mathcal{H}|)/n}\right) E [ UC n MC ( f W )] = O ( log ( 2∣ C ∣∣ H ∣ ) / n ) . When H = Lip 1 ( [ 0 , 1 ] , [ − 1 , 1 ] ) \mathcal{H} = \operatorname{Lip}_1([0,1],[-1,1]) H = Lip 1 ([ 0 , 1 ] , [ − 1 , 1 ]) , Appendix § 13 .§ 6 gives a direct entropy-integral proof of E [ U C n M C ( f W ) ] ≲ n − 1 / 2 + log ( 2 ∣ C ∣ ) / n \mathbb{E}[\mathrm{UC}^{\mathrm{MC}}_n(f_W)] \lesssim n^{-1/2} + \sqrt{\log(2|C|)/n} E [ UC n MC ( f W )] ≲ n − 1/2 + log ( 2∣ C ∣ ) / n for finite C C C . Combining this with Corollary 2, we have a corresponding sample-complexity of E ∣ Φ M C ( f W ; D ) − Φ M C , n ( f W ; S ) ∣ \mathbb{E}|\Phi_{\mathrm{MC}}(f_W;D) - \Phi_{\mathrm{MC},n}(f_W;S)| E ∣ Φ MC ( f W ; D ) − Φ MC , n ( f W ; S ) ∣ for the binned MC.
One-sided comparisons. For supremum-type objectives witten with a common feasible set and convex aggregation maps, Jensen’s inequality gives Φ ( f ; D ) ≤ E S ′ ∼ D n [ Φ n ( f ; S ′ ) ∣ f ] \Phi(f;D) \le\mathbb{E}_{S' \sim D^n}[\Phi_n(f;S') \mid f] Φ ( f ; D ) ≤ E S ′ ∼ D n [ Φ n ( f ; S ′ ) ∣ f ] . Thus an upper bound on the expected population–training gap follows from the validation–training term alone. MC satisfies this comparison; combining it with Theorem 2 gives
E Φ M C ( f W ; D ) ≤ E Φ M C , n ( f W ; S ) + 2 n min { 2 n I F C M I , ∑ t = 1 n 2 I l o c , t M C } . (20) \mathbb{E}\Phi_{\mathrm{MC}}(f_W;D) \le\mathbb{E}\Phi_{\mathrm{MC},n}(f_W;S) + \frac{2}{n}\min\left\{\sqrt{2nI_{\mathrm{FCMI}}},\sum_{t=1}^{n}\sqrt{2I^{\mathrm{MC}}_{\mathrm{loc},t}}\right\}. \tag*{(20)} E Φ MC ( f W ; D ) ≤ E Φ MC , n ( f W ; S ) + n 2 min { 2 n I FCMI , t = 1 ∑ n 2 I loc , t MC } . (20)
Corollaries 8 and 9 in Appendix § 13 .§ 4 give the fixed-predictor plug-in terms E [ U C n C V a R ( f W ) ] = O ( n − 1 / 2 ) \mathbb{E}[\mathrm{UC}^{\mathrm{CVaR}}_n(f_W)] = O(n^{-1/2}) E [ UC n CVaR ( f W )] = O ( n − 1/2 ) for CVaR and E [ U C n D R O ( f W ) ] = O ~ ( n − 1 / ( c ∗ ∨ 2 ) ) \mathbb{E}[\mathrm{UC}^{\mathrm{DRO}}_n(f_W)] = \widetilde{O}(n^{-1/(c^*\vee2)}) E [ UC n DRO ( f W )] = O ( n − 1/ ( c ∗ ∨ 2 ) ) for CR DRO. Both are applied after conditioning on f W f_W f W , so no additional uniform entropy over F \mathcal{F} F is paid here. Appendix § 13 .9 compares the full population–training bounds with existing results in Table 2 .
Table 2. Full-bound comparison with specialized results.
Related Work
Information-theoretic generalization bounds relate expected generalization to the information that a learning algorithm reveals about the selected sample [41 , 34 , 21 , 19 ] . Supersample and evaluated-CMI analyses use the symmetry under u ↦ u ˉ u \mapsto\bar{u} u ↦ u ˉ for ordinary pointwise losses, where fixing the predictor also fixes the row-wise loss table. Refined single-index bounds further localize this information to individual selectors [39 ] . Our setting differs because the empirical criterion first observes the whole batch and then solves an auxiliary optimization problem. Theorem 2 extends the single-index viewpoint by replacing fixed per-example losses with the increments of the optimized objective along the one-point replacement path. Existing work [39 ] suggests that shifted-Rademacher arguments can yield fast-rate improvements for standard eCMI bounds. In the set-input regime, however, the telescoping identity (10) does not generally lead to an analogous shifted decomposition. Appendix § 15 discusses Bernstein bounds and compares I F C M I I_{\mathrm{FCMI}} I FCMI and I l o c I_{\mathrm{loc}} I loc .
Optimization-valued criteria also arise in machine learning in a different form: objectives in which the optimization is attached to a single input rather than to an evaluation set. Examples include predict-then-optimize, decision-focused learning, and data-driven optimization [40 , 24 , 7 , 22 ] . PAC-Bayesian guarantees for learned optimization algorithms have also recently appeared [35 ] . These objectives are different from the set-input objectives studied here: under appropriate assumptions, their optimized value can be treated as a per-example loss. See Proposition 1.
Risk-sensitive generalization bounds have been developed for CVaR and related OCE criteria [23 ] ; the classical optimization formulation of CVaR goes back to Rockafellar et al. [32 ] . Information-theoretic analyses of calibration have also studied the statistical bias of binning ECE [11 ] . Our framework instead treats MC, CVaR, and Cressie–Read DRO as set-input optimized objectives: once Assumption 1 has been verified, the validation–training term follows by applying the standard fCMI inequality to the centered difference of the separately optimized values.
Unlike set-input objectives, single-input optimization metrics solve an optimization problem for each data point. In the single-input regime, the downstream optimization is solved row by row . Predict-then-optimize and other instance-wise learning-to-optimize problems fit this template: for each input row, the model predicts instance-specific quantities and a separate downstream optimization problem is solved using only that row [7 , 40 , 24 , 22 ] . Once the predictor is fixed, the optimized value behaves like an ordinary per-example loss and admits standard IT analysis; see Appendix C.
Numerical Bound Evaluation
We numerically evaluate the proposed bounds and examine their usefulness.2
Linear logistic regression on Flights200K
We apply linear logistic regression to Flights200K [36 ] to numerically evaluate the MC bound in Corollary 1. Experiments for the CVaR and CR DRO bounds in Corollaries 5 and 6 under the same settings are reported in Appendices H.2 and H.3, respectively. We train an L 2 L_2 L 2 -regularized linear logistic regression model and add independent Gaussian noise to the fitted parameters, including the intercept, as W = θ ^ ( S ) + σ ξ W = \hat{\theta}(S) + \sigma\xi W = θ ^ ( S ) + σ ξ , with ξ ∼ N ( 0 , I 8 ) \xi\sim\mathcal{N}(0, I_8) ξ ∼ N ( 0 , I 8 ) and σ ∈ { 0.05 , 0.10 } \sigma\in\{0.05, 0.10\} σ ∈ { 0.05 , 0.10 } . For each sampled pair index, we retrain under both values of the selector while holding the other selectors fixed. The resulting Gaussian parameter distributions allow the conditional mutual information between the parameters and the selector to be evaluated by one-dimensional integration. Combining this with data processing gives an estimate of the single-index bound in Eq. (12) . We compare it with the absolute value of the average validation–training MC difference. Appendix H.1 gives the logistic-regression settings, subgroup/bin construction, and Monte Carlo estimation procedure.
Figure 1 shows the mean and one sample standard deviation from independent Monte Carlo estimates. For both noise levels, the gap and bound estimates decrease with the sample size, and the bound is approximately linear on the log–log scale. The bound estimates exceed the gap estimates in every setting and remain conservative, but fall below the trivial bound of one for larger samples.
Figure 1. MC gap (dashed) and Gaussian bound (solid) on Flights200K: σ = 0.05 \sigma= 0.05 σ = 0.05 (left), 0.10 (right). Points and error bars show the mean ± \pm ± one sample standard deviation of independent estimates.
Large-scale image classification on CelebA
We next evaluate the bounds for a larger model on image data, using ViT on CelebA for MC and CVaR (α = 0.2 \alpha= 0.2 α = 0.2 ). The MC protocol follows Hansen et al. [17 ] ; Appendix § 18 gives the settings. As in § 5.1 , we limit computation by Monte Carlo sampling over supersamples, data-pair indices, and training randomness. We sample ten pairs per supersample and repeat training seven times for each choice within a pair, holding the other selectors fixed. Unlike the Gaussian-perturbed logistic model, the conditional distribution of ViT’s trained parameters is not explicit, so the Gaussian information formula is unavailable. Following the nearest-neighbor estimation approach of Futami and Fujisawa [11 ] , we instead estimate the mutual information between local increments and their corresponding selectors from repeated training. This information additionally conditions on the other selectors. Finite-sample information estimates can be biased, and their aggregation need not yield an upper bound; we therefore call the result a proxy rather than a bound.
Figure 2 compares this estimate with the observed absolute validation–training gap. For both objectives, the estimate is much larger than the observed gap throughout the range of sample sizes.
Figure 2. Observed absolute validation–training gap and estimated local-information proxy on CelebA/ViT: MC (left) and CVaR (α = 0.2 \alpha= 0.2 α = 0.2 ; right). Shaded bands show variation across supersample seeds. The proxy remains above the observed gap in both panels.
Discussion. In logistic regression, Gaussian perturbations make the conditional parameter distributions explicit, allowing the information upper bound to be evaluated accurately by one-dimensional integration. For ViT, nearest-neighbor information estimation from repeated training gives a proxy much larger than the observed gap. Evaluating the proposed bounds for larger models therefore requires more accurate and computationally efficient information estimators.
Conclusion and Limitations
We introduced a unified IT framework for set-input optimization-valued objectives , a class that includes MC, CVaR, and Cressie–Read DRO. The main message is a separation: fCMI isolates algorithmic dependence on the supersample selector vector, while the objective-dependent sub-Gaussian parameter reflects the geometry of the optimized objective. We bound this parameter through one-point stability, which also yields a single-index refinement, and through convex L 2 L_2 L 2 -Lipschitz geometry, which gives the sharper DRO rate. Our theory has several limitations. First, our theory isolates statistical generalization and does not analyze the optimization error of the outer learning problem. Although the single-index refinement isolates one selector at a time, it can still depend on the full prediction table rather than a single row. The centering argument uses equal-size training and validation samples; the finite-class restriction in the smaller-validation-set extension instead comes from a union bound and can be relaxed under suitable conditions (Appendix § 15 .§ 6 ). Future work includes sharper bounds on the sub-Gaussian parameter, tighter fixed-predictor comparisons, fast-rate refinements, and optimization-aware analyses for other batch-level criteria.
Acknowledgments and Disclosure of Funding
FF was supported by JST BOOST Program Grant Number JPMJBY24G8. FF and MF were supported by JSPS KAKENHI Grant Number 26K02974, Japan. MF was supported by JSPS KAKENHI Grant Number JP25K21286, Japan. This work was supported by Japan Science and Technology Agency (JST) as part of Adopting Sustainable Partnerships for Innovative Research Ecosystem (ASPIRE), Grant Number JPMJAP25B1.
Notation
Table 3.
Throughout the Appendix, we write D ( P ∥ Q ) D(P\|Q) D ( P ∥ Q ) for the Kullback–Leibler divergence between two probability measures P P P and Q Q Q .
Detailed explanation about examples of set-input metrics
This appendix records the reliability- and risk-oriented objectives used in the paper. We keep only the definitions and background needed for the later proofs. Throughout, Z = ( X , Y ) ∼ D Z=(X,Y)\sim\mathcal{D} Z = ( X , Y ) ∼ D , S = ( Z i ) i = 1 n S=(Z_i)_{i=1}^n S = ( Z i ) i = 1 n , and V i = f W ( X i ) V_i=f_W(X_i) V i = f W ( X i ) .
Multicalibration
Calibration asks that a score predictor f W : X → [ 0 , 1 ] f_W:\mathcal{X}\to[0,1] f W : X → [ 0 , 1 ] be interpretable as a probability, ideally satisfying E [ Y ∣ f W ( X ) ] = f W ( X ) \mathbb{E}[Y\mid f_W(X)]=f_W(X) E [ Y ∣ f W ( X )] = f W ( X ) almost surely [9 ] . Multicalibration strengthens this requirement by testing calibration simultaneously over many subgroups and score witnesses [20 , 14 , 3 ] . This is useful when reliability on overlapping or computationally identifiable population slices matters, for example in fairness-aware risk scoring and downstream decision support.
For a subgroup class C ⊆ { c : X → [ 0 , 1 ] } \mathcal{C} \subseteq\{c : \mathcal{X} \to[0,1]\} C ⊆ { c : X → [ 0 , 1 ]} and a score-test class H ⊆ { h : [ 0 , 1 ] → [ − 1 , 1 ] } \mathcal{H} \subseteq\{h : [0,1] \to[-1,1]\} H ⊆ { h : [ 0 , 1 ] → [ − 1 , 1 ]} , set Π M C : = C × H × { ± 1 } \Pi_{\mathrm{MC}} := \mathcal{C} \times\mathcal{H} \times\{\pm1\} Π MC := C × H × { ± 1 } . The population and empirical multicalibration metrics are
Φ M C ( f w ; D ) : = sup ( c , h , η ) ∈ Π M C η E [ c ( X ) h ( f w ( X ) ) ( Y − f w ( X ) ) ] . \Phi_{\mathrm{MC}}(f_w; \mathcal{D}) := \sup_{(c,h,\eta)\in\Pi_{\mathrm{MC}}} \eta\,\mathbb{E}[c(X)h(f_w(X))(Y-f_w(X))]. Φ MC ( f w ; D ) := ( c , h , η ) ∈ Π MC sup η E [ c ( X ) h ( f w ( X )) ( Y − f w ( X ))] .
Φ M C ( f w ; S ) : = sup ( c , h , η ) ∈ Π M C η n ∑ i = 1 n c ( X i ) h ( V i ) ( Y i − V i ) . \Phi_{\mathrm{MC}}(f_w; S) := \sup_{(c,h,\eta)\in\Pi_{\mathrm{MC}}} \frac{\eta}{n}\sum_{i=1}^{n}c(X_i)h(V_i)(Y_i-V_i). Φ MC ( f w ; S ) := ( c , h , η ) ∈ Π MC sup n η i = 1 ∑ n c ( X i ) h ( V i ) ( Y i − V i ) .
The empirical optimizer is the witness triple ( c , h , η ) (c,h,\eta) ( c , h , η ) chosen after observing the whole batch.
Typical subgroup classes are indicators C G = { x ↦ 1 { x ∈ G } : G ∈ G } \mathcal{C}_{\mathcal{G}}=\{x\mapsto\mathbf{1}\{x\in G\}:G\in\mathcal{G}\} C G = { x ↦ 1 { x ∈ G } : G ∈ G } , where G \mathcal{G} G may be a finite protected-group collection, a VC or low-degree class, or an oracle-efficient benchmark family [20 , 14 , 29 ] . Common witness classes include bin indicators v ↦ 1 { v ∈ I j } v\mapsto\mathbf{1}\{v\in I_j\} v ↦ 1 { v ∈ I j } , bounded measurable witnesses, Lipschitz witnesses, and RKHS unit balls; these choices recover bin-style, smooth, or kernel calibration distances [8 , 3 , 18 ] . Algorithmically, multicalibration has been studied through iterative post-processing, weak-agnostic-learning reductions, boosting-style procedures, and game-dynamics or privacy-based approaches [20 , 14 , 13 , 29 , 16 ] .
OCE risks and CVaR
Optimized-certainty-equivalent (OCE) risks model tail sensitivity or disutility-sensitive preferences [2 , 32 ] . They are used when a small mean loss may still hide rare but severe failures, for example in finance, medical decision support, safe learning, and risk-sensitive reinforcement learning [23 , 38 ] .
Let L w : = ℓ ( f w ( X ) , Y ) ∈ [ 0 , 1 ] L_w := \ell(f_w(X),Y)\in[0,1] L w := ℓ ( f w ( X ) , Y ) ∈ [ 0 , 1 ] and let ψ : R → R ∪ { ∞ } \psi:\mathbb{R}\to\mathbb{R}\cup\{\infty\} ψ : R → R ∪ { ∞ } be a proper closed convex disutility. The population and empirical OCE risks are
Φ O C E , ψ ( f w ; D ) : = inf η ∈ R { η + E ψ ( L w − η ) } . \Phi_{\mathrm{OCE},\psi}(f_w;\mathcal{D}) := \inf_{\eta\in\mathbb{R}}\left\{\eta+\mathbb{E}\,\psi(L_w-\eta)\right\}. Φ OCE , ψ ( f w ; D ) := η ∈ R inf { η + E ψ ( L w − η ) } .
Φ O C E , ψ ( f w ; S ) : = inf η ∈ R { η + 1 n ∑ i = 1 n ψ ( ℓ ( f w , Z i ) − η ) } . \Phi_{\mathrm{OCE},\psi}(f_w;S) := \inf_{\eta\in\mathbb{R}}\left\{\eta+\frac{1}{n}\sum_{i=1}^{n}\psi(\ell(f_w,Z_i)-\eta)\right\}. Φ OCE , ψ ( f w ; S ) := η ∈ R inf { η + n 1 i = 1 ∑ n ψ ( ℓ ( f w , Z i ) − η ) } .
Thus the threshold η \eta η is selected after seeing the batch of losses.
For CVaR at level α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) , write α ˉ : = 1 − α \bar{\alpha}:=1-\alpha α ˉ := 1 − α ; the disutility is ψ α ( t ) = α ˉ − 1 [ t ] + \psi_\alpha(t)=\bar{\alpha}^{-1}[t]_+ ψ α ( t ) = α ˉ − 1 [ t ] + . Hence
Φ C V a R , α ( f w ; D ) : = inf τ ∈ [ 0 , 1 ] { τ + 1 α ˉ E ( L w − τ ) + } . \Phi_{\mathrm{CVaR},\alpha}(f_w;\mathcal{D}) := \inf_{\tau\in[0,1]}\left\{\tau+\frac{1}{\bar{\alpha}}\mathbb{E}(L_w-\tau)_+\right\}. Φ CVaR , α ( f w ; D ) := τ ∈ [ 0 , 1 ] inf { τ + α ˉ 1 E ( L w − τ ) + } .
Φ C V a R , α ( f w ; S ) : = inf τ ∈ [ 0 , 1 ] { τ + 1 n α ˉ ∑ i = 1 n ( ℓ ( f w , Z i ) − τ ) + } . \Phi_{\mathrm{CVaR},\alpha}(f_w;S) := \inf_{\tau\in[0,1]}\left\{\tau+\frac{1}{n\bar{\alpha}}\sum_{i=1}^{n}(\ell(f_w,Z_i)-\tau)_+\right\}. Φ CVaR , α ( f w ; S ) := τ ∈ [ 0 , 1 ] inf { τ + n α ˉ 1 i = 1 ∑ n ( ℓ ( f w , Z i ) − τ ) + } .
This is the usual upper-tail risk. If the loss distribution has no atom at an optimizing quantile, it equals the conditional mean over the worst α ˉ \bar{\alpha} α ˉ tail; with atoms, the risk-envelope formula below is the unambiguous definition. Empirical OCE objectives and their stochastic approximations have been studied in supervised and reinforcement-learning settings [23 , 38 ] .
When ψ \psi ψ is proper, closed, convex, nondecreasing, and normalized by ψ ( 0 ) = 0 \psi(0)=0 ψ ( 0 ) = 0 , the duality condition (satisfied by the CVaR disutility and by the bounded OCEs used here) holds and it gives the dual representation
Φ O C E , ψ ( f w ; D ) = sup { E [ q ( Z ) L w ] − E [ ψ ∗ ( q ( Z ) ) ] : q ( Z ) ≥ 0 a.s. , E q ( Z ) = 1 } . \Phi_{\mathrm{OCE},\psi}(f_w;\mathcal{D})=\sup\left\{\mathbb{E}[q(Z)L_w]-\mathbb{E}[\psi^*(q(Z))]:q(Z)\geq0\text{ a.s.},\ \mathbb{E}q(Z)=1\right\}. Φ OCE , ψ ( f w ; D ) = sup { E [ q ( Z ) L w ] − E [ ψ ∗ ( q ( Z ))] : q ( Z ) ≥ 0 a.s. , E q ( Z ) = 1 } .
Here the nonnegativity of q q q follows from the monotonicity of ψ \psi ψ , equivalently from the effective domain of ψ ∗ \psi^* ψ ∗ . For CVaR this reduces to the risk-envelope form
Φ C V a R , α ( f w ; D ) = sup { E [ q ( Z ) L w ] : 0 ≤ q ( Z ) ≤ α ˉ − 1 a.s. , E q ( Z ) = 1 } . \Phi_{\mathrm{CVaR},\alpha}(f_w;\mathcal{D})=\sup\left\{\mathbb{E}[q(Z)L_w]:0\leq q(Z)\leq\bar{\alpha}^{-1}\text{ a.s.},\ \mathbb{E}q(Z)=1\right\}. Φ CVaR , α ( f w ; D ) = sup { E [ q ( Z ) L w ] : 0 ≤ q ( Z ) ≤ α ˉ − 1 a.s. , E q ( Z ) = 1 } .
Φ C V a R , α ( f w ; S ) = sup { 1 n ∑ i = 1 n q i ℓ ( f w , Z i ) : 0 ≤ q i ≤ α ˉ − 1 , 1 n ∑ i = 1 n q i = 1 } . \Phi_{\mathrm{CVaR},\alpha}(f_w;S)=\sup\left\{\frac{1}{n}\sum_{i=1}^{n}q_i\ell(f_w,Z_i):0\leq q_i\leq\bar{\alpha}^{-1},\ \frac{1}{n}\sum_{i=1}^{n}q_i=1\right\}. Φ CVaR , α ( f w ; S ) = sup { n 1 i = 1 ∑ n q i ℓ ( f w , Z i ) : 0 ≤ q i ≤ α ˉ − 1 , n 1 i = 1 ∑ n q i = 1 } .
The risk-envelope form is used to expose convex L 2 L_2 L 2 geometry and for common-feasible-set comparisons; the primal threshold form remains the average-form set-input representation used for CVaR.
f f f -divergence DRO and the Cressie–Read family
For a nominal law P = Law ( Z ) P = \operatorname{Law}(Z) P = Law ( Z ) and an alternative law Q ≪ P Q \ll P Q ≪ P , write ω = d Q / d P \omega= dQ/dP ω = d Q / d P . An f f f -divergence ball is specified by D f ( Q ∥ P ) = E P f ( ω ( Z ) ) D_f(Q\mathbin\Vert P) = \mathbb{E}_P f(\omega(Z)) D f ( Q ∥ P ) = E P f ( ω ( Z )) and E P ω ( Z ) = 1 \mathbb{E}_P\omega(Z) = 1 E P ω ( Z ) = 1 . The robust risk is
Φ f - D R O , ϵ ( f w ; D ) : = sup ω ≥ 0 { E D [ ω ( Z ) ℓ ( f w ( X ) , Y ) ] : E D ω ( Z ) = 1 , E D f ( ω ( Z ) ) ≤ ϵ } . \Phi_{f\text{-}\mathrm{DRO},\epsilon}(f_w;\mathcal{D}) := \sup_{\omega\ge0}\left\{\mathbb{E}_{\mathcal{D}}\left[\omega(Z)\ell(f_w(X),Y)\right] : \mathbb{E}_{\mathcal{D}}\omega(Z)=1,\ \mathbb{E}_{\mathcal{D}}f(\omega(Z)) \le\epsilon\right\}. Φ f - DRO , ϵ ( f w ; D ) := ω ≥ 0 sup { E D [ ω ( Z ) ℓ ( f w ( X ) , Y ) ] : E D ω ( Z ) = 1 , E D f ( ω ( Z )) ≤ ϵ } .
For the Cressie–Read generator f c ( t ) = ( t c − c t + c − 1 ) / { c ( c − 1 ) } f_c(t) = (t^c-ct+c-1)/\{c(c-1)\} f c ( t ) = ( t c − c t + c − 1 ) / { c ( c − 1 )} , c ∈ R ∖ { 0 , 1 } c \in\mathbb{R}\setminus\{0,1\} c ∈ R ∖ { 0 , 1 } , the case c = 2 c=2 c = 2 is Pearson χ 2 \chi^2 χ 2 , while the limits c → 1 c\to1 c → 1 and c → 0 c\to0 c → 0 recover KL and empirical-likelihood divergences. For c > 1 c>1 c > 1 , the constraints E f c ( ω ) ≤ ϵ \mathbb{E}f_c(\omega)\le\epsilon E f c ( ω ) ≤ ϵ and E ω = 1 \mathbb{E}\omega=1 E ω = 1 are equivalent to E ω ( Z ) c ≤ 1 + c ( c − 1 ) ϵ \mathbb{E}\omega(Z)^c\le1+c(c-1)\epsilon E ω ( Z ) c ≤ 1 + c ( c − 1 ) ϵ .
For the empirical law P ^ S = n − 1 ∑ i = 1 n δ Z i \widehat{P}_S=n^{-1}\sum_{i=1}^n\delta_{Z_i} P S = n − 1 ∑ i = 1 n δ Z i , the density ratio is a weight vector q = ( q 1 , … , q n ) q=(q_1,\ldots,q_n) q = ( q 1 , … , q n ) with n − 1 ∑ i q i = 1 n^{-1}\sum_iq_i=1 n − 1 ∑ i q i = 1 . The empirical Cressie–Read robust value is
Φ D R O , c , ϵ ( f w ; S ) = sup q ≥ 0 { 1 n ∑ i = 1 n q i ℓ ( f w ( X i ) , Y i ) : 1 n ∑ i = 1 n q i = 1 , 1 n ∑ i = 1 n q i c ≤ 1 + c ( c − 1 ) ϵ } . (21) \Phi_{\mathrm{DRO},c,\epsilon}(f_w;S)=\sup_{q\ge0}\left\{\frac{1}{n}\sum_{i=1}^n q_i\ell(f_w(X_i),Y_i):\frac{1}{n}\sum_{i=1}^nq_i=1,\ \frac{1}{n}\sum_{i=1}^nq_i^c\le1+c(c-1)\epsilon\right\}. \tag*{(21)} Φ DRO , c , ϵ ( f w ; S ) = q ≥ 0 sup { n 1 i = 1 ∑ n q i ℓ ( f w ( X i ) , Y i ) : n 1 i = 1 ∑ n q i = 1 , n 1 i = 1 ∑ n q i c ≤ 1 + c ( c − 1 ) ϵ } . (21)
The auxiliary optimizer is therefore an adversarial reweighting chosen after the empirical loss vector is observed.
Cressie–Read DRO as an average-form objective
The primal reweighting form above is useful for Euclidean Lipschitz bounds. The following standard finite-dimensional Cressie–Read dual form places the same objective in the average-form class (5) ; compare the Cressie–Read plug-in formulation of Duchi and Namkoong [6 ] .
Lemma 3 (Cressie–Read dual average form). Let c > 1 c>1 c > 1 , c ∗ = c / ( c − 1 ) c^*=c/(c-1) c ∗ = c / ( c − 1 ) , and
Q n , c , ϵ : = { q ∈ R + n : 1 n ∑ i = 1 n q i = 1 , 1 n ∑ i = 1 n q i c ≤ κ c , ϵ c } , κ c , ϵ = { 1 + c ( c − 1 ) ϵ } 1 / c . \mathcal{Q}_{n,c,\epsilon}:=\left\{q\in\mathbb{R}_+^n:\frac{1}{n}\sum_{i=1}^nq_i=1,\ \frac{1}{n}\sum_{i=1}^nq_i^c\le\kappa_{c,\epsilon}^c\right\},\qquad\kappa_{c,\epsilon}=\{1+c(c-1)\epsilon\}^{1/c}. Q n , c , ϵ := { q ∈ R + n : n 1 i = 1 ∑ n q i = 1 , n 1 i = 1 ∑ n q i c ≤ κ c , ϵ c } , κ c , ϵ = { 1 + c ( c − 1 ) ϵ } 1/ c . For bounded losses r i = ℓ ( f ( X i ) , Y i ) r_i=\ell(f(X_i),Y_i) r i = ℓ ( f ( X i ) , Y i ) ,
sup q ∈ Q n , c , ϵ 1 n ∑ i = 1 n q i r i = inf η ∈ R { η + κ c , ϵ ( 1 n ∑ i = 1 n ( r i − η ) + c ∗ ) 1 / c ∗ } . \sup_{q\in\mathcal{Q}_{n,c,\epsilon}}\frac{1}{n}\sum_{i=1}^nq_ir_i
=\inf_{\eta\in\mathbb{R}}\left\{\eta+\kappa_{c,\epsilon}\left(\frac{1}{n}\sum_{i=1}^n(r_i-\eta)_+^{c^*}\right)^{1/c^*}\right\}. q ∈ Q n , c , ϵ sup n 1 i = 1 ∑ n q i r i = η ∈ R inf ⎩ ⎨ ⎧ η + κ c , ϵ ( n 1 i = 1 ∑ n ( r i − η ) + c ∗ ) 1/ c ∗ ⎭ ⎬ ⎫ .
See Duchi and Namkoong [6 ] for its derivation.
Consequently, Φ D R O , c , ϵ , n ( f ; S ) \Phi_{\mathrm{DRO},c,\epsilon,n}(f;S) Φ DRO , c , ϵ , n ( f ; S ) has the average-form representation
Opt η ∈ R ψ η ( 1 n ∑ i = 1 n ρ η ( f ( X i ) , Z i ) ) , Opt = inf , \operatorname{Opt}_{\eta\in\mathbb{R}}\psi_\eta\left(\frac{1}{n}\sum_{i=1}^n\rho_\eta(f(X_i),Z_i)\right),\qquad\operatorname{Opt}=\inf, Opt η ∈ R ψ η ( n 1 i = 1 ∑ n ρ η ( f ( X i ) , Z i ) ) , Opt = inf ,
with ρ η ( v , z ) = ( ℓ ( v , z ) − η ) + c ∗ \rho_\eta(v,z)=(\ell(v,z)-\eta)_+^{c^*} ρ η ( v , z ) = ( ℓ ( v , z ) − η ) + c ∗ and ψ η ( m ) = η + κ c , ϵ m 1 / c ∗ \psi_\eta(m)=\eta+\kappa_{c,\epsilon}m^{1/c^*} ψ η ( m ) = η + κ c , ϵ m 1/ c ∗ .
Auxiliary inequalities for supremum and infimum objectives
We collect two elementary inequalities used repeatedly in the paper. For arbitrary families ( a π ) π ∈ Π (a_\pi)_{\pi\in\Pi} ( a π ) π ∈ Π and ( b π ) π ∈ Π (b_\pi)_{\pi\in\Pi} ( b π ) π ∈ Π ,
∣ sup π ∈ Π a π − sup π ∈ Π b π ∣ ≤ sup π ∈ Π ∣ a π − b π ∣ . (22) \left|\sup_{\pi\in\Pi}a_\pi-\sup_{\pi\in\Pi}b_\pi\right|\le\sup_{\pi\in\Pi}|a_\pi-b_\pi|. \tag*{(22)} π ∈ Π sup a π − π ∈ Π sup b π ≤ π ∈ Π sup ∣ a π − b π ∣. (22)
∣ inf π ∈ Π a π − inf π ∈ Π b π ∣ ≤ sup π ∈ Π ∣ a π − b π ∣ . (23) \left|\inf_{\pi\in\Pi}a_\pi-\inf_{\pi\in\Pi}b_\pi\right|\le\sup_{\pi\in\Pi}|a_\pi-b_\pi|. \tag*{(23)} π ∈ Π inf a π − π ∈ Π inf b π ≤ π ∈ Π sup ∣ a π − b π ∣. (23)
These give the absolute-gap population–validation comparison for both supremum and infimum objectives. For one-sided bounds, fix a predictor f f f and suppose the population and empirical inner objectives share a common feasible set Π ( f ) \Pi(f) Π ( f ) . If ψ π ( f , D ) ≤ E [ ψ π , n ( f , S ′ ) ∣ f ] \psi_\pi(f,\mathcal{D})\le\mathbb{E}[\psi_{\pi,n}(f,S')\mid f] ψ π ( f , D ) ≤ E [ ψ π , n ( f , S ′ ) ∣ f ] for all π ∈ Π ( f ) \pi\in\Pi(f) π ∈ Π ( f ) , then
sup π ∈ Π ( f ) ψ π ( f , D ) ≤ E [ sup π ∈ Π ( f ) ψ π , n ( f , S ′ ) ∣ f ] . \sup_{\pi\in\Pi(f)}\psi_\pi(f,\mathcal{D})\le\mathbb{E}\left[\sup_{\pi\in\Pi(f)}\psi_{\pi,n}(f,S')\mid f\right]. π ∈ Π ( f ) sup ψ π ( f , D ) ≤ E [ π ∈ Π ( f ) sup ψ π , n ( f , S ′ ) ∣ f ] .
Thus supremum-type objectives naturally yield upper bounds on Φ ( f ; D ) − Φ ( f ; S ) \Phi(f;\mathcal{D})-\Phi(f;S) Φ ( f ; D ) − Φ ( f ; S ) .
This appendix records the simple reduction mentioned in § 2.2 . In the single-input regime, the downstream optimization is solved row by row . Predict-then-optimize and other instance-wise learning-to-optimize problems fit this template: for each input row, the model predicts instance-specific quantities and a separate downstream optimization problem is solved using only that row [7 , 40 , 24 , 22 ] . Once the predictor is fixed, the optimized value behaves exactly like an ordinary per-example loss.
Let ϕ 1 : X × Y × V × Π → R \phi_1 : \mathcal{X} \times\mathcal{Y} \times\mathcal{V} \times\Pi\to\mathbb{R} ϕ 1 : X × Y × V × Π → R be a measurable objective, and let Π 1 ( x , y , v ) ⊆ Π \Pi_1(x,y,v) \subseteq\Pi Π 1 ( x , y , v ) ⊆ Π be the feasible set for one example. For a predictor f w f_w f w and one data point ( X , Y ) (X,Y) ( X , Y ) , define the optimized single-input loss by
ϕ 1 f w ( X , Y ) : = Opt π ∈ Π 1 ( X , Y , f w ( X ) ) ϕ 1 ( X , Y , f w ( X ) ; π ) , (24) \phi_1^{f_w}(X,Y) := \operatorname{Opt}_{\pi\in\Pi_1(X,Y,f_w(X))} \phi_1(X,Y,f_w(X);\pi), \tag*{(24)} ϕ 1 f w ( X , Y ) := Opt π ∈ Π 1 ( X , Y , f w ( X )) ϕ 1 ( X , Y , f w ( X ) ; π ) , (24)
where Opt ∈ { sup , inf } \operatorname{Opt} \in\{\sup,\inf\} Opt ∈ { sup , inf } . Throughout this appendix, the optimized value is understood to be a finite measurable version whenever it is used; in particular, the relevant feasible sets are assumed to be nonempty.
The corresponding empirical objective is the ordinary average
R ^ opt ( f w ; S ) : = 1 n ∑ i = 1 n ϕ 1 f w ( X i , Y i ) . \widehat{R}^{\operatorname{opt}}(f_w;S) := \frac{1}{n}\sum_{i=1}^{n}\phi_1^{f_w}(X_i,Y_i). R opt ( f w ; S ) := n 1 i = 1 ∑ n ϕ 1 f w ( X i , Y i ) .
The important point is that once f W f_W f W is fixed, the optimized quantity ϕ 1 f W ( X ~ i , j , Y ~ i , j ) \phi_1^{f_W}(\widetilde{X}_{i,j},\widetilde{Y}_{i,j}) ϕ 1 f W ( X i , j , Y i , j ) is just an ordinary supersample loss. Hence, under boundedness or sub-Gaussian assumptions on ϕ 1 f W \phi_1^{f_W} ϕ 1 f W , the standard eCMI bound (1) applies exactly as in ordinary supervised learning. Although the optimized loss may depend on x x x as well as y y y , the eCMI argument only uses the induced bounded supersample loss table.
Proposition 1 (Single-input optimization-valued losses reduce to ordinary eCMI). Assume that, for every w w w in the support of W W W , the feasible set Π 1 ( x , y , f w ( x ) ) \Pi_1(x,y,f_w(x)) Π 1 ( x , y , f w ( x )) is nonempty and the optimized per-example value ϕ 1 f w ( x , y ) \phi_1^{f_w}(x,y) ϕ 1 f w ( x , y ) from (24) admits a finite measurable version lying in [ 0 , 1 ] [0,1] [ 0 , 1 ] . Set ϕ i , j : = ϕ 1 f W ( X ~ i , j , Y ~ i , j ) \phi_{i,j} := \phi_1^{f_W}(\widetilde{X}_{i,j},\widetilde{Y}_{i,j}) ϕ i , j := ϕ 1 f W ( X i , j , Y i , j ) , i ∈ [ n ] i \in[n] i ∈ [ n ] , j ∈ { 0 , 1 } j \in\{0,1\} j ∈ { 0 , 1 } , and write V ~ W : = ( f W ( X ~ i , j ) ) i ∈ [ n ] , j ∈ { 0 , 1 } \widetilde{V}_W := (f_W(\widetilde{X}_{i,j}))_{i\in[n],j\in\{0,1\}} V W := ( f W ( X i , j ) ) i ∈ [ n ] , j ∈ { 0 , 1 } for the supersample prediction array. Then the row-wise eCMI bound
∣ E [ R ^ opt ( f W ; S ′ ) − R ^ opt ( f W ; S ) ] ∣ ≤ 1 n ∑ i = 1 n 2 I ( ( ϕ i , 0 , ϕ i , 1 ) ; U i ∣ Z ~ ) (25) \left|\mathbb{E}\left[\widehat{R}^{\operatorname{opt}}(f_W;S')-\widehat{R}^{\operatorname{opt}}(f_W;S)\right]\right|
\leq\frac{1}{n}\sum_{i=1}^{n}\sqrt{2I((\phi_{i,0},\phi_{i,1});U_i\mid\widetilde{Z})} \tag*{(25)} E [ R opt ( f W ; S ′ ) − R opt ( f W ; S ) ] ≤ n 1 i = 1 ∑ n 2 I (( ϕ i , 0 , ϕ i , 1 ) ; U i ∣ Z ) (25) holds. In addition, the following bound uses the CMI between the loss table and the full vector U U U , given Z ~ \widetilde{Z} Z , and hence also gives the corresponding fCMI bound:
∣ E [ R ^ opt ( f W ; S ′ ) − R ^ opt ( f W ; S ) ] ∣ ≤ 2 I ( ( ϕ i , j ) i , j ; U ∣ Z ~ ) n ≤ 2 I ( V ~ W ; U ∣ Z ~ ) n . (26) \left|\mathbb{E}\left[\widehat{R}^{\operatorname{opt}}(f_W;S')-\widehat{R}^{\operatorname{opt}}(f_W;S)\right]\right|
\leq\sqrt{\frac{2I((\phi_{i,j})_{i,j};U\mid\widetilde{Z})}{n}}
\leq\sqrt{\frac{2I(\widetilde{V}_W;U\mid\widetilde{Z})}{n}}. \tag*{(26)} E [ R opt ( f W ; S ′ ) − R opt ( f W ; S ) ] ≤ n 2 I (( ϕ i , j ) i , j ; U ∣ Z ) ≤ n 2 I ( V W ; U ∣ Z ) . (26) If W W W is independent of ( Z ~ , U ) (\widetilde{Z},U) ( Z , U ) , in particular if f w f_w f w is fixed independently of the supersample and selector vector, then E [ R ^ opt ( f W ; S ′ ) − R ^ opt ( f W ; S ) ] = 0 \mathbb{E}[\widehat{R}^{\operatorname{opt}}(f_W;S')-\widehat{R}^{\operatorname{opt}}(f_W;S)]=0 E [ R opt ( f W ; S ′ ) − R opt ( f W ; S )] = 0 .
Proof. Because the optimization in (24) is solved separately for each example, the array Φ : = ( ϕ i , j ) i ∈ [ n ] , j ∈ { 0 , 1 } \Phi:= (\phi_{i,j})_{i\in[n],j\in\{0,1\}} Φ := ( ϕ i , j ) i ∈ [ n ] , j ∈ { 0 , 1 } is an ordinary bounded supersample loss table. Therefore
R ^ opt ( f W ; S ) = 1 n ∑ i = 1 n ϕ i , U i , R ^ opt ( f W ; S ′ ) = 1 n ∑ i = 1 n ϕ i , 1 − U i . \widehat{R}^{\operatorname{opt}}(f_W;S)=\frac{1}{n}\sum_{i=1}^{n}\phi_{i,U_i},\qquad
\widehat{R}^{\operatorname{opt}}(f_W;S')=\frac{1}{n}\sum_{i=1}^{n}\phi_{i,1-U_i}. R opt ( f W ; S ) = n 1 i = 1 ∑ n ϕ i , U i , R opt ( f W ; S ′ ) = n 1 i = 1 ∑ n ϕ i , 1 − U i .
Applying the standard row-wise eCMI inequality (1) to this bounded loss table, and applying the same argument to the negative gap, proves (25).
It remains to prove the second bound. Fix a deterministic loss table ϕ = ( ϕ i , j ) i ∈ [ n ] , j ∈ { 0 , 1 } ∈ [ 0 , 1 ] n × 2 \phi=(\phi_{i,j})_{i\in[n],j\in\{0,1\}}\in[0,1]^{n\times2} ϕ = ( ϕ i , j ) i ∈ [ n ] , j ∈ { 0 , 1 } ∈ [ 0 , 1 ] n × 2 , and define
Γ ϕ ( u ) : = 1 n ∑ i = 1 n ϕ i , 1 − u i − 1 n ∑ i = 1 n ϕ i , u i . \Gamma_{\phi}(u):=\frac{1}{n}\sum_{i=1}^{n}\phi_{i,1-u_i}-\frac{1}{n}\sum_{i=1}^{n}\phi_{i,u_i}. Γ ϕ ( u ) := n 1 i = 1 ∑ n ϕ i , 1 − u i − n 1 i = 1 ∑ n ϕ i , u i .
@ p a g e 17 / 54 s t a r t s m i d = 0 e n d s m i d = 1 @page 17/54 starts_mid=0 ends_mid=1 @ p a g e 17/54 s t a r t s m i d = 0 e n d s m i d = 1
This appendix records the simple reduction mentioned in § 2.2 . In the single-input regime, the downstream optimization is solved row by row . Predict-then-optimize and other instance-wise learning-to-optimize problems fit this template: for each input row, the model predicts instance-specific quantities and a separate downstream optimization problem is solved using only that row [7 , 40 , 24 , 22 ] . Once the predictor is fixed, the optimized value behaves exactly like an ordinary per-example loss.
Let ϕ 1 : X × Y × V × Π → R \phi_1 : \mathcal{X} \times\mathcal{Y} \times\mathcal{V} \times\Pi\to\mathbb{R} ϕ 1 : X × Y × V × Π → R be a measurable objective, and let Π 1 ( x , y , v ) ⊆ Π \Pi_1(x,y,v) \subseteq\Pi Π 1 ( x , y , v ) ⊆ Π be the feasible set for one example. For a predictor f w f_w f w and one data point ( X , Y ) (X,Y) ( X , Y ) , define the optimized single-input loss by
ϕ 1 f w ( X , Y ) : = Opt π ∈ Π 1 ( X , Y , f w ( X ) ) ϕ 1 ( X , Y , f w ( X ) ; π ) , (27) \phi_1^{f_w}(X,Y) := \operatorname{Opt}_{\pi\in\Pi_1(X,Y,f_w(X))} \phi_1(X,Y,f_w(X);\pi), \tag*{(27)} ϕ 1 f w ( X , Y ) := Opt π ∈ Π 1 ( X , Y , f w ( X )) ϕ 1 ( X , Y , f w ( X ) ; π ) , (27)
where Opt ∈ { sup , inf } \operatorname{Opt} \in\{\sup,\inf\} Opt ∈ { sup , inf } . Throughout this appendix, the optimized value is understood to be a finite measurable version whenever it is used; in particular, the relevant feasible sets are assumed to be nonempty.
The corresponding empirical objective is the ordinary average
R ^ opt ( f w ; S ) : = 1 n ∑ i = 1 n ϕ 1 f w ( X i , Y i ) . \widehat{R}^{\operatorname{opt}}(f_w;S) := \frac{1}{n}\sum_{i=1}^{n}\phi_1^{f_w}(X_i,Y_i). R opt ( f w ; S ) := n 1 i = 1 ∑ n ϕ 1 f w ( X i , Y i ) .
The important point is that once f W f_W f W is fixed, the optimized quantity ϕ 1 f W ( X ~ i , j , Y ~ i , j ) \phi_1^{f_W}(\widetilde{X}_{i,j},\widetilde{Y}_{i,j}) ϕ 1 f W ( X i , j , Y i , j ) is just an ordinary supersample loss. Hence, under boundedness or sub-Gaussian assumptions on ϕ 1 f W \phi_1^{f_W} ϕ 1 f W , the standard eCMI bound (1) applies exactly as in ordinary supervised learning. Although the optimized loss may depend on x x x as well as y y y , the eCMI argument only uses the induced bounded supersample loss table.
Proposition 1 (Single-input optimization-valued losses reduce to ordinary eCMI). Assume that, for every w w w in the support of W W W , the feasible set Π 1 ( x , y , f w ( x ) ) \Pi_1(x,y,f_w(x)) Π 1 ( x , y , f w ( x )) is nonempty and the optimized per-example value ϕ 1 f w ( x , y ) \phi_1^{f_w}(x,y) ϕ 1 f w ( x , y ) from (24) admits a finite measurable version lying in [ 0 , 1 ] [0,1] [ 0 , 1 ] . Set ϕ i , j : = ϕ 1 f W ( X ~ i , j , Y ~ i , j ) \phi_{i,j} := \phi_1^{f_W}(\widetilde{X}_{i,j},\widetilde{Y}_{i,j}) ϕ i , j := ϕ 1 f W ( X i , j , Y i , j ) , i ∈ [ n ] i \in[n] i ∈ [ n ] , j ∈ { 0 , 1 } j \in\{0,1\} j ∈ { 0 , 1 } , and write V ~ W : = ( f W ( X ~ i , j ) ) i ∈ [ n ] , j ∈ { 0 , 1 } \widetilde{V}_W := (f_W(\widetilde{X}_{i,j}))_{i\in[n],j\in\{0,1\}} V W := ( f W ( X i , j ) ) i ∈ [ n ] , j ∈ { 0 , 1 } for the supersample prediction array. Then the row-wise eCMI bound
∣ E [ R ^ opt ( f W ; S ′ ) − R ^ opt ( f W ; S ) ] ∣ ≤ 1 n ∑ i = 1 n 2 I ( ( ϕ i , 0 , ϕ i , 1 ) ; U i ∣ Z ~ ) (28) \left|\mathbb{E}\left[\widehat{R}^{\operatorname{opt}}(f_W;S')-\widehat{R}^{\operatorname{opt}}(f_W;S)\right]\right|
\leq\frac{1}{n}\sum_{i=1}^{n}\sqrt{2I((\phi_{i,0},\phi_{i,1});U_i\mid\widetilde{Z})} \tag*{(28)} E [ R opt ( f W ; S ′ ) − R opt ( f W ; S ) ] ≤ n 1 i = 1 ∑ n 2 I (( ϕ i , 0 , ϕ i , 1 ) ; U i ∣ Z ) (28) holds. In addition, the following bound uses the CMI between the loss table and the full vector U U U , given Z ~ \widetilde{Z} Z , and hence also gives the corresponding fCMI bound:
∣ E [ R ^ opt ( f W ; S ′ ) − R ^ opt ( f W ; S ) ] ∣ ≤ 2 I ( ( ϕ i , j ) i , j ; U ∣ Z ~ ) n ≤ 2 I ( V ~ W ; U ∣ Z ~ ) n . (29) \left|\mathbb{E}\left[\widehat{R}^{\operatorname{opt}}(f_W;S')-\widehat{R}^{\operatorname{opt}}(f_W;S)\right]\right|
\leq\sqrt{\frac{2I((\phi_{i,j})_{i,j};U\mid\widetilde{Z})}{n}}
\leq\sqrt{\frac{2I(\widetilde{V}_W;U\mid\widetilde{Z})}{n}}. \tag*{(29)} E [ R opt ( f W ; S ′ ) − R opt ( f W ; S ) ] ≤ n 2 I (( ϕ i , j ) i , j ; U ∣ Z ) ≤ n 2 I ( V W ; U ∣ Z ) . (29) If W W W is independent of ( Z ~ , U ) (\widetilde{Z},U) ( Z , U ) , in particular if f w f_w f w is fixed independently of the supersample and selector vector, then E [ R ^ opt ( f W ; S ′ ) − R ^ opt ( f W ; S ) ] = 0 \mathbb{E}[\widehat{R}^{\operatorname{opt}}(f_W;S')-\widehat{R}^{\operatorname{opt}}(f_W;S)]=0 E [ R opt ( f W ; S ′ ) − R opt ( f W ; S )] = 0 .
Proof. Because the optimization in (24) is solved separately for each example, the array Φ : = ( ϕ i , j ) i ∈ [ n ] , j ∈ { 0 , 1 } \Phi:= (\phi_{i,j})_{i\in[n],j\in\{0,1\}} Φ := ( ϕ i , j ) i ∈ [ n ] , j ∈ { 0 , 1 } is an ordinary bounded supersample loss table. Therefore
R ^ opt ( f W ; S ) = 1 n ∑ i = 1 n ϕ i , U i , R ^ opt ( f W ; S ′ ) = 1 n ∑ i = 1 n ϕ i , 1 − U i . \widehat{R}^{\operatorname{opt}}(f_W;S)=\frac{1}{n}\sum_{i=1}^{n}\phi_{i,U_i},\qquad
\widehat{R}^{\operatorname{opt}}(f_W;S')=\frac{1}{n}\sum_{i=1}^{n}\phi_{i,1-U_i}. R opt ( f W ; S ) = n 1 i = 1 ∑ n ϕ i , U i , R opt ( f W ; S ′ ) = n 1 i = 1 ∑ n ϕ i , 1 − U i .
Applying the standard row-wise eCMI inequality (1) to this bounded loss table, and applying the same argument to the negative gap, proves (25).
It remains to prove the second bound. Fix a deterministic loss table ϕ = ( ϕ i , j ) i ∈ [ n ] , j ∈ { 0 , 1 } ∈ [ 0 , 1 ] n × 2 \phi=(\phi_{i,j})_{i\in[n],j\in\{0,1\}}\in[0,1]^{n\times2} ϕ = ( ϕ i , j ) i ∈ [ n ] , j ∈ { 0 , 1 } ∈ [ 0 , 1 ] n × 2 , and define
Γ ϕ ( u ) : = 1 n ∑ i = 1 n ϕ i , 1 − u i − 1 n ∑ i = 1 n ϕ i , u i . \Gamma_{\phi}(u):=\frac{1}{n}\sum_{i=1}^{n}\phi_{i,1-u_i}-\frac{1}{n}\sum_{i=1}^{n}\phi_{i,u_i}. Γ ϕ ( u ) := n 1 i = 1 ∑ n ϕ i , 1 − u i − n 1 i = 1 ∑ n ϕ i , u i .
Writing d i : = ϕ i , 1 − ϕ i , 0 d_i := \phi_{i,1} - \phi_{i,0} d i := ϕ i , 1 − ϕ i , 0 , we have Γ ϕ ( u ) = 1 n ∑ i = 1 n ( 1 − 2 u i ) d i \Gamma_\phi(u) = \frac{1}{n}\sum_{i=1}^{n}(1-2u_i)d_i Γ ϕ ( u ) = n 1 ∑ i = 1 n ( 1 − 2 u i ) d i . Under the uniform selector law Q U = Unif ( { 0 , 1 } n ) Q_U = \operatorname{Unif}(\{0,1\}^n) Q U = Unif ({ 0 , 1 } n ) , the random variables ( 1 − 2 U i ) d i / n (1-2U_i)d_i/n ( 1 − 2 U i ) d i / n , i ∈ [ n ] i \in[n] i ∈ [ n ] , are independent and mean-zero. Hence Hoeffding’s lemma gives, for every λ ∈ R \lambda\in\mathbb{R} λ ∈ R ,
log E Q U exp { λ Γ ϕ ( U ) } ≤ λ 2 2 n 2 ∑ i = 1 n d i 2 ≤ λ 2 2 n , \log\mathbb{E}_{Q_U}\exp\{\lambda\Gamma_\phi(U)\} \leq\frac{\lambda^2}{2n^2}\sum_{i=1}^{n}d_i^2 \leq\frac{\lambda^2}{2n}, log E Q U exp { λ Γ ϕ ( U )} ≤ 2 n 2 λ 2 i = 1 ∑ n d i 2 ≤ 2 n λ 2 ,
where the last inequality uses ϕ i , j ∈ [ 0 , 1 ] \phi_{i,j} \in[0,1] ϕ i , j ∈ [ 0 , 1 ] , so that ∣ d i ∣ ≤ 1 |d_i| \leq1 ∣ d i ∣ ≤ 1 .
Now apply the conditional change-of-measure argument with reference law P Φ ∣ Z ~ Q U P_{\Phi\mid\widetilde{Z}}Q_U P Φ ∣ Z Q U . For every λ > 0 \lambda> 0 λ > 0 , E Γ Φ ( U ) ≤ 1 λ I ( Φ ; U ∣ Z ~ ) + λ 2 n \mathbb{E}\Gamma_\Phi(U) \leq\frac{1}{\lambda}I(\Phi;U\mid\widetilde{Z})+\frac{\lambda}{2n} E Γ Φ ( U ) ≤ λ 1 I ( Φ ; U ∣ Z ) + 2 n λ . Optimizing over λ > 0 \lambda>0 λ > 0 yields E Γ Φ ( U ) ≤ 2 I ( Φ ; U ∣ Z ~ ) n \mathbb{E}\Gamma_\Phi(U) \leq\sqrt{\frac{2I(\Phi;U\mid\widetilde{Z})}{n}} E Γ Φ ( U ) ≤ n 2 I ( Φ ; U ∣ Z ) . Applying the same argument to − Γ Φ ( U ) -\Gamma_\Phi(U) − Γ Φ ( U ) gives ∣ E Γ Φ ( U ) ∣ ≤ 2 I ( Φ ; U ∣ Z ~ ) n |\mathbb{E}\Gamma_\Phi(U)| \leq\sqrt{\frac{2I(\Phi;U\mid\widetilde{Z})}{n}} ∣ E Γ Φ ( U ) ∣ ≤ n 2 I ( Φ ; U ∣ Z ) . Since Γ Φ ( U ) = R ^ o p t ( f W ; S ′ ) − R ^ o p t ( f W ; S ) \Gamma_\Phi(U) = \widehat{R}^{\mathrm{opt}}(f_W;S')-\widehat{R}^{\mathrm{opt}}(f_W;S) Γ Φ ( U ) = R opt ( f W ; S ′ ) − R opt ( f W ; S ) , this proves the first inequality in (26).
For the second inequality in (26), observe that conditional on Z ~ \widetilde{Z} Z , the optimized loss table Φ \Phi Φ is a measurable function of the supersample prediction array V ~ W \widetilde{V}_W V W . Indeed,
ϕ i , j = Opt π ∈ Π 1 ( X ~ i , j , Y ~ i , j , f W ( X ~ i , j ) ) ϕ 1 ( X ~ i , j , Y ~ i , j , f W ( X ~ i , j ) ; π ) . \phi_{i,j}=\operatorname{Opt}_{\pi\in\Pi_1}\left(\widetilde{X}_{i,j},\widetilde{Y}_{i,j},f_W(\widetilde{X}_{i,j})\right)\phi_1\left(\widetilde{X}_{i,j},\widetilde{Y}_{i,j},f_W(\widetilde{X}_{i,j});\pi\right). ϕ i , j = Opt π ∈ Π 1 ( X i , j , Y i , j , f W ( X i , j ) ) ϕ 1 ( X i , j , Y i , j , f W ( X i , j ) ; π ) .
Therefore conditional data processing gives I ( Φ ; U ∣ Z ~ ) ≤ I ( V ~ W ; U ∣ Z ~ ) I(\Phi;U\mid\widetilde{Z})\leq I(\widetilde{V}_W;U\mid\widetilde{Z}) I ( Φ ; U ∣ Z ) ≤ I ( V W ; U ∣ Z ) , which proves the fCMI bound.
Finally, suppose that W W W is independent of ( Z ~ , U ) (\widetilde{Z},U) ( Z , U ) . Then, conditional on ( W , Z ~ ) (W,\widetilde{Z}) ( W , Z ) , the optimized loss table Φ \Phi Φ is fixed and U U U is uniform. Hence E [ Γ Φ ( U ) ∣ W , Z ~ ] = 0 \mathbb{E}[\Gamma_\Phi(U)\mid W,\widetilde{Z}]=0 E [ Γ Φ ( U ) ∣ W , Z ] = 0 . Taking expectation gives E [ R ^ o p t ( f W ; S ′ ) − R ^ o p t ( f W ; S ) ] = 0 \mathbb{E}[\widehat{R}^{\mathrm{opt}}(f_W;S')-\widehat{R}^{\mathrm{opt}}(f_W;S)]=0 E [ R opt ( f W ; S ′ ) − R opt ( f W ; S )] = 0 .
Specialization to pointwise losses. Take Φ n = L ^ \Phi_n=\widehat{L} Φ n = L and ϕ i , j = ℓ ( f W , Z ~ i , j ) ∈ [ 0 , 1 ] \phi_{i,j}=\ell(f_W,\widetilde{Z}_{i,j})\in[0,1] ϕ i , j = ℓ ( f W , Z i , j ) ∈ [ 0 , 1 ] in the preceding proof. Its fixed-table MGF bound gives L Γ , n ≤ n − 1 / 2 L_{\Gamma,n}\leq n^{-1/2} L Γ , n ≤ n − 1/2 , so Eq. (7) gives ∣ E Δ Φ ∣ ≤ 2 I f C M I / n |\mathbb{E}\Delta_\Phi|\leq\sqrt{2I_{\mathrm{fCMI}}/n} ∣ E Δ Φ ∣ ≤ 2 I fCMI / n , with the constant in Eq. (1) . Replacing one observation changes the empirical risk by at most β n = 1 / n \beta_n=1/n β n = 1/ n , and the other rows cancel in D t = ( ϕ t , 0 − ϕ t , 1 ) / n D_t=(\phi_{t,0}-\phi_{t,1})/n D t = ( ϕ t , 0 − ϕ t , 1 ) / n . Conditional data processing therefore gives I l o c , t ≤ I ( ( ϕ t , 0 , ϕ t , 1 ) ; U t ∣ Z ~ ) I_{\mathrm{loc},t}\leq I((\phi_{t,0},\phi_{t,1});U_t\mid\widetilde{Z}) I loc , t ≤ I (( ϕ t , 0 , ϕ t , 1 ) ; U t ∣ Z ) . Substitution into Eq. (11) yields the row-wise eCMI bound in Eq. (1) .
Proofs for Section 3
Proofs for Section 3.1: fCMI bound
Lemma 4 (Centering by complementary splits). Fix a predictor f f f and a supersample realization z ~ \widetilde{z} z . Define
Γ f , z ~ ( u ) : = Φ n ( f ; z ~ u ) − Φ n ( f ; z ~ u ‾ ) , u ∈ { 0 , 1 } n . \Gamma_{f,\widetilde{z}}(u):=\Phi_n(f;\widetilde{z}_u)-\Phi_n(f;\widetilde{z}_{\overline{u}}),\qquad u\in\{0,1\}^n. Γ f , z ( u ) := Φ n ( f ; z u ) − Φ n ( f ; z u ) , u ∈ { 0 , 1 } n . Then
Γ f , z ~ ( u ‾ ) = − Γ f , z ~ ( u ) for all u ∈ { 0 , 1 } n . (30) \Gamma_{f,\widetilde{z}}(\overline{u})=-\Gamma_{f,\widetilde{z}}(u)\qquad\text{for all }u\in\{0,1\}^n. \tag*{(30)} Γ f , z ( u ) = − Γ f , z ( u ) for all u ∈ { 0 , 1 } n . (30) Consequently, if U ∼ Q U = Unif ( { 0 , 1 } n ) U\sim Q_U=\operatorname{Unif}(\{0,1\}^n) U ∼ Q U = Unif ({ 0 , 1 } n ) , then
E Q U Γ f , z ~ ( U ) = 2 − n ∑ u ∈ { 0 , 1 } n Γ f , z ~ ( u ) \mathbb{E}_{Q_U}\Gamma_{f,\widetilde{z}}(U)=2^{-n}\sum_{u\in\{0,1\}^n}\Gamma_{f,\widetilde{z}}(u) E Q U Γ f , z ( U ) = 2 − n u ∈ { 0 , 1 } n ∑ Γ f , z ( u ) = 2 − n − 1 ∑ u ∈ { 0 , 1 } n { Γ f , z ~ ( u ) + Γ f , z ~ ( u ‾ ) } = 0. (31) =2^{-n-1}\sum_{u\in\{0,1\}^n}\{\Gamma_{f,\widetilde{z}}(u)+\Gamma_{f,\widetilde{z}}(\overline{u})\}=0. \tag*{(31)} = 2 − n − 1 u ∈ { 0 , 1 } n ∑ { Γ f , z ( u ) + Γ f , z ( u )} = 0. (31)
Proof. First, the sign flip follows by substituting u ‾ \overline{u} u into the definition of the gap. Since u ‾ ‾ = u \overline{\overline{u}}=u u = u ,
Γ f , z ~ ( u ‾ ) = Φ n ( f ; z ~ u ‾ ) − Φ n ( f ; z ~ u ) = Φ n ( f ; z ~ u ‾ ) − Φ n ( f ; z ~ u ) = − Γ f , z ~ ( u ) , \Gamma_{f,\widetilde{z}}(\overline{u})=\Phi_n(f;\widetilde{z}_{\overline{u}})-\Phi_n(f;\widetilde{z}_{u})
=\Phi_n(f;\widetilde{z}_{\overline{u}})-\Phi_n(f;\widetilde{z}_{u})=-\Gamma_{f,\widetilde{z}}(u), Γ f , z ( u ) = Φ n ( f ; z u ) − Φ n ( f ; z u ) = Φ n ( f ; z u ) − Φ n ( f ; z u ) = − Γ f , z ( u ) ,
This proves (30) .
We now prove the conditional centering identity by pairing each u u u with u ˉ \bar{u} u ˉ . Since U U U is uniform on { 0 , 1 } n \{0,1\}^{n} { 0 , 1 } n ,
E Q U Γ f , z ~ ( U ) = 2 − n ∑ u ∈ { 0 , 1 } n Γ f , z ~ ( u ) . \mathbb{E}_{Q_U}\Gamma_{f,\widetilde{z}}(U)=2^{-n}\sum_{u\in\{0,1\}^{n}}\Gamma_{f,\widetilde{z}}(u). E Q U Γ f , z ( U ) = 2 − n u ∈ { 0 , 1 } n ∑ Γ f , z ( u ) .
The complement map u ↦ u ˉ u\mapsto\bar{u} u ↦ u ˉ is a bijection on { 0 , 1 } n \{0,1\}^{n} { 0 , 1 } n . Hence the same finite average can be written as the average over complementary pairs:
2 − n ∑ u ∈ { 0 , 1 } n Γ f , z ~ ( u ) = 2 − n − 1 ∑ u ∈ { 0 , 1 } n { Γ f , z ~ ( u ) + Γ f , z ~ ( u ˉ ) } . 2^{-n}\sum_{u\in\{0,1\}^{n}}\Gamma_{f,\widetilde{z}}(u)=2^{-n-1}\sum_{u\in\{0,1\}^{n}}\{\Gamma_{f,\widetilde{z}}(u)+\Gamma_{f,\widetilde{z}}(\bar{u})\}. 2 − n u ∈ { 0 , 1 } n ∑ Γ f , z ( u ) = 2 − n − 1 u ∈ { 0 , 1 } n ∑ { Γ f , z ( u ) + Γ f , z ( u ˉ )} .
By the sign-flip identity (30) , the term inside braces is zero for every u u u . This proves (31) .
Remark 1 (Pairing when n = 2 n=2 n = 2 ). When n = 2 n=2 n = 2 , the selector vector takes the four values 00 , 01 , 10 , 11 00,01,10,11 00 , 01 , 10 , 11 . These form the two complementary pairs 00 ↔ 11 00\leftrightarrow11 00 ↔ 11 and 01 ↔ 10 01\leftrightarrow10 01 ↔ 10 . For each fixed ( f , z ~ ) (f,\widetilde{z}) ( f , z ) , complementing the selector vector swaps the two selected batches in the difference, so Γ f , z ~ ( 11 ) = − Γ f , z ~ ( 00 ) \Gamma_{f,\widetilde{z}}(11)=-\Gamma_{f,\widetilde{z}}(00) Γ f , z ( 11 ) = − Γ f , z ( 00 ) , Γ f , z ~ ( 10 ) = − Γ f , z ~ ( 01 ) \Gamma_{f,\widetilde{z}}(10)=-\Gamma_{f,\widetilde{z}}(01) Γ f , z ( 10 ) = − Γ f , z ( 01 ) . Thus the average over the four selector vectors cancels pairwise. The proof of Lemma 4 is exactly the same argument on { 0 , 1 } n \{0,1\}^{n} { 0 , 1 } n .
Proof of Theorem 1. Let us consider the sigma-algebra G : = σ ( f W ( X ~ ) , Z ~ ) \mathcal{G}:=\sigma(f_W(\widetilde{X}),\widetilde{Z}) G := σ ( f W ( X ) , Z ) . Conditioning on G \mathcal{G} G fixes the prediction table and the supersample. Hence, for every selector vector u u u , the optimized empirical value Φ n ( f W ; Z ~ u ) \Phi_n(f_W;\widetilde{Z}_u) Φ n ( f W ; Z u ) is fixed and the map u ↦ Γ f W , Z ~ ( u ) = Φ n ( f W ; Z ~ u ) − Φ n ( f W ; Z ~ u ˉ ) u\mapsto\Gamma_{f_W,\widetilde{Z}}(u)=\Phi_n(f_W;\widetilde{Z}_u)-\Phi_n(f_W;\widetilde{Z}_{\bar{u}}) u ↦ Γ f W , Z ( u ) = Φ n ( f W ; Z u ) − Φ n ( f W ; Z u ˉ ) is deterministic. Lemma 4 gives E Q U Γ f W , Z ~ ( U ) = 0 \mathbb{E}_{Q_U}\Gamma_{f_W,\widetilde{Z}}(U)=0 E Q U Γ f W , Z ( U ) = 0 under the reference selector law Q U = Unif ( { 0 , 1 } n ) Q_U=\operatorname{Unif}(\{0,1\}^{n}) Q U = Unif ({ 0 , 1 } n ) .
For a realized value of G \mathcal{G} G , let P U ∣ G P_{U\mid\mathcal{G}} P U ∣ G be a regular conditional law of U U U given G \mathcal{G} G . Since Q U Q_U Q U has full support on the finite space of selector vectors, P U ∣ G ≪ Q U P_{U\mid\mathcal{G}}\ll Q_U P U ∣ G ≪ Q U . Also U U U is independent of Z ~ \widetilde{Z} Z , so Q U = P U ∣ Z ~ Q_U=P_{U\mid\widetilde{Z}} Q U = P U ∣ Z almost surely. Therefore
E D ( P U ∣ G ∥ Q U ) = E D ( P U ∣ f W ( X ~ ) , Z ~ ∥ P U ∣ Z ~ ) = I ( U ; f W ( X ~ ) ∣ Z ~ ) . (32) \mathbb{E}D(P_{U\mid\mathcal{G}}\|Q_U)=\mathbb{E}D(P_{U\mid f_W(\widetilde{X}),\widetilde{Z}}\|P_{U\mid\widetilde{Z}})=I(U;f_W(\widetilde{X})\mid\widetilde{Z}). \tag*{(32)} E D ( P U ∣ G ∥ Q U ) = E D ( P U ∣ f W ( X ) , Z ∥ P U ∣ Z ) = I ( U ; f W ( X ) ∣ Z ) . (32)
If this quantity is infinite, the theorem is trivial, so assume it is finite.
Next, we upper bound the generalization gap. By the Donsker–Varadhan variational inequality, for every λ > 0 \lambda>0 λ > 0 and every realized G \mathcal{G} G ,
E P U ∣ G Γ f W , Z ~ ( U ) ≤ 1 λ D ( P U ∣ G ∥ Q U ) + 1 λ log E Q U e λ Γ f W , Z ~ ( U ) ≤ 1 λ D ( P U ∣ G ∥ Q U ) + λ L Γ , n 2 2 , \begin{aligned}
\mathbb{E}_{P_{U\mid\mathcal{G}}}\Gamma_{f_W,\widetilde{Z}}(U)\leq\frac{1}{\lambda}D(P_{U\mid\mathcal{G}}\|Q_U)+\frac{1}{\lambda}\log\mathbb{E}_{Q_U}e^{\lambda\Gamma_{f_W,\widetilde{Z}}(U)} \\
&\leq\frac{1}{\lambda}D(P_{U\mid\mathcal{G}}\|Q_U)+\frac{\lambda L_{\Gamma,n}^{2}}{2},
\end{aligned} E P U ∣ G Γ f W , Z ( U ) ≤ λ 1 D ( P U ∣ G ∥ Q U ) + λ 1 log E Q U e λ Γ f W , Z ( U ) ≤ λ 1 D ( P U ∣ G ∥ Q U ) + 2 λ L Γ , n 2 ,
where the last step uses Assumption 1. Optimizing over λ > 0 \lambda>0 λ > 0 gives E P U ∣ G Γ f W , Z ~ ( U ) ≤ L Γ , n 2 D ( P U ∣ G ∥ Q U ) \mathbb{E}_{P_{U\mid\mathcal{G}}}\Gamma_{f_W,\widetilde{Z}}(U)\leq L_{\Gamma,n}\sqrt{2D(P_{U\mid\mathcal{G}}\|Q_U)} E P U ∣ G Γ f W , Z ( U ) ≤ L Γ , n 2 D ( P U ∣ G ∥ Q U ) . Applying the same argument to − Γ f W , Z ~ -\Gamma_{f_W,\widetilde{Z}} − Γ f W , Z gives the corresponding lower bound, because Assumption 1 holds for all λ ∈ R \lambda\in\mathbb{R} λ ∈ R . Thus ∣ E P U ∣ G Γ f W , Z ~ ( U ) ∣ ≤ L Γ , n 2 D ( P U ∣ G ∥ Q U ) \left|\mathbb{E}_{P_{U\mid\mathcal{G}}}\Gamma_{f_W,\widetilde{Z}}(U)\right|\leq L_{\Gamma,n}\sqrt{2D(P_{U\mid\mathcal{G}}\|Q_U)} E P U ∣ G Γ f W , Z ( U ) ≤ L Γ , n 2 D ( P U ∣ G ∥ Q U ) . Taking expectations and applying Jensen’s inequality together with (32) ,
∣ E Γ f W , Z ~ ( U ) ∣ ≤ L Γ , n 2 I ( f W ( X ~ ) ; U ∣ Z ~ ) . \left|\mathbb{E}\Gamma_{f_W,\widetilde{Z}}(U)\right|\leq L_{\Gamma,n}\sqrt{2I(f_W(\widetilde{X});U\mid\widetilde{Z})}. E Γ f W , Z ( U ) ≤ L Γ , n 2 I ( f W ( X ) ; U ∣ Z ) .
Since S = Z ~ U S=\widetilde{Z}_U S = Z U and S ′ = Z ~ U ˉ S'=\widetilde{Z}_{\bar{U}} S ′ = Z U ˉ , this is (7) .
Next, we derive the absolute version bound. For λ > 0 \lambda>0 λ > 0 , e λ ∣ x ∣ ≤ e λ x + e − λ x e^{\lambda|x|}\leq e^{\lambda x}+e^{-\lambda x} e λ ∣ x ∣ ≤ e λ x + e − λ x . Using Assumption 1 for both signs gives, conditionally on G \mathcal{G} G ,
log E Q U e λ ∣ Γ f W , Z ~ ( U ) ∣ ≤ log 2 + λ 2 L Γ , n 2 2 . \log\mathbb{E}_{Q_U}e^{\lambda|\Gamma_{f_W,\widetilde{Z}}(U)|}\leq\log2+\frac{\lambda^{2}L_{\Gamma,n}^{2}}{2}. log E Q U e λ ∣ Γ f W , Z ( U ) ∣ ≤ log 2 + 2 λ 2 L Γ , n 2 .
Donsker–Varadhan applied to ∣ Γ ∣ |\Gamma| ∣Γ∣ yields
E P U ∣ G ∣ Γ f W , Z ~ ( U ) ∣ ≤ D ( P U ∣ G ∥ Q U ) + log 2 λ + λ L Γ , n 2 2 . \mathbb{E}_{P_{U\mid\mathcal{G}}}|\Gamma_{f_W,\widetilde{Z}}(U)|\leq\frac{D(P_{U\mid\mathcal{G}}\|Q_U)+\log2}{\lambda}+\frac{\lambda L_{\Gamma,n}^{2}}{2}. E P U ∣ G ∣ Γ f W , Z ( U ) ∣ ≤ λ D ( P U ∣ G ∥ Q U ) + log 2 + 2 λ L Γ , n 2 .
Optimizing in λ > 0 \lambda>0 λ > 0 , averaging over G \mathcal{G} G , and applying Jensen’s inequality gives
E ∣ Γ f W , Z ~ ( U ) ∣ ≤ L Γ , n 2 { I ( f W ( X ~ ) ; U ∣ Z ~ ) + log 2 } , \mathbb{E}|\Gamma_{f_W,\widetilde{Z}}(U)|\leq L_{\Gamma,n}\sqrt{2\{I(f_W(\widetilde{X});U\mid\widetilde{Z})+\log2\}}, E ∣ Γ f W , Z ( U ) ∣ ≤ L Γ , n 2 { I ( f W ( X ) ; U ∣ Z ) + log 2 } ,
which is (8) .
A sub-Gaussian parameter conditional on the prediction table and supersample. Write F = f W ( X ~ ) F=f_W(\widetilde{X}) F = f W ( X ) , G = ( F , Z ~ ) G=(F,\widetilde{Z}) G = ( F , Z ) , and let Γ g \Gamma_g Γ g denote the gap as a function of the selector vector for a fixed value g g g of G G G . Suppose that L ( g ) ≥ 0 L(g)\ge0 L ( g ) ≥ 0 is measurable, E L ( G ) 2 < ∞ \mathbb{E}L(G)^2<\infty E L ( G ) 2 < ∞ , and, for every such g g g ,
log E u ∼ Q U exp { λ Γ g ( u ) } ≤ λ 2 L ( g ) 2 / 2 ( λ ∈ R ) . \log\mathbb{E}_{u\sim Q_U}\exp\{\lambda\Gamma_g(u)\}\le\lambda^2L(g)^2/2\qquad(\lambda\in\mathbb{R}). log E u ∼ Q U exp { λ Γ g ( u )} ≤ λ 2 L ( g ) 2 /2 ( λ ∈ R ) .
Put K ( g ) = D ( P U ∣ G = g ∥ Q U ) K(g)=D(P_{U\mid G=g}\|Q_U) K ( g ) = D ( P U ∣ G = g ∥ Q U ) . Repeating the conditional Donsker–Varadhan step of the preceding proof for both signs gives K ( g ) / λ + λ L ( g ) 2 / 2 K(g)/\lambda+\lambda L(g)^2/2 K ( g ) / λ + λ L ( g ) 2 /2 as an upper bound on ∣ E [ Δ Φ ∣ G = g ] ∣ |\mathbb{E}[\Delta_\Phi\mid G=g]| ∣ E [ Δ Φ ∣ G = g ] ∣ . Minimizing over λ > 0 \lambda>0 λ > 0 gives ∣ E [ Δ Φ ∣ G = g ] ∣ ≤ L ( g ) 2 K ( g ) |\mathbb{E}[\Delta_\Phi\mid G=g]|\le L(g)\sqrt{2K(g)} ∣ E [ Δ Φ ∣ G = g ] ∣ ≤ L ( g ) 2 K ( g ) . Taking expectations and using Cauchy–Schwarz and Eq. (32) yields
∣ E Δ Φ ∣ ≤ E [ L ( G ) 2 K ( G ) ] ≤ 2 E [ L ( G ) 2 ] I f C M I . |\mathbb{E}\Delta_\Phi|\le\mathbb{E}\left[L(G)\sqrt{2K(G)}\right]\le\sqrt{2\mathbb{E}[L(G)^2]}\,I_{\mathrm{fCMI}}. ∣ E Δ Φ ∣ ≤ E [ L ( G ) 2 K ( G ) ] ≤ 2 E [ L ( G ) 2 ] I fCMI .
The MGF condition concerns the entire reference distribution of the selector vector with g g g fixed. A value chosen from the observed selector vector does not by itself certify this condition.
Bounding fCMI for a given learner. For any data-independent prior distribution P 0 P_0 P 0 on the weights,
I f C M I ≤ I ( W ; U ∣ Z ~ ) ≤ I ( W ; S ) ≤ E S D ( P W ∣ S ∥ P 0 ) . I_{\mathrm{fCMI}}\le I(W;U\mid\widetilde{Z})\le I(W;S)\le\mathbb{E}_S D(P_{W\mid S}\|P_0). I fCMI ≤ I ( W ; U ∣ Z ) ≤ I ( W ; S ) ≤ E S D ( P W ∣ S ∥ P 0 ) .
The first inequality follows by conditional data processing since F F F is determined by ( W , Z ~ ) (W,\widetilde{Z}) ( W , Z ) . The second follows from I ( W ; S ) = I ( W ; Z ~ ) + I ( W ; U ∣ Z ~ ) I(W;S)=I(W;\widetilde{Z})+I(W;U\mid\widetilde{Z}) I ( W ; S ) = I ( W ; Z ) + I ( W ; U ∣ Z ) : S S S is determined by ( Z ~ , U ) (\widetilde{Z},U) ( Z , U ) , and W = A ( S , R ) W=A(S,R) W = A ( S , R ) uses independent randomness R R R . The last inequality follows from E S D ( P W ∣ S ∥ P 0 ) = I ( W ; S ) + D ( P W ∥ P 0 ) \mathbb{E}_S D(P_{W\mid S}\|P_0)=I(W;S)+D(P_W\|P_0) E S D ( P W ∣ S ∥ P 0 ) = I ( W ; S ) + D ( P W ∥ P 0 ) . This KL comparison controls the information in the predictor; complement symmetry is still needed to center the difference between separately optimized objectives. The expected posterior–prior KL thus bounds the information term in Theorem 1; the result remains an expectation bound. Even if I ( W ; S ) I(W;S) I ( W ; S ) is infinite, I f C M I ≤ H ( U ) = n log 2 I_{\mathrm{fCMI}}\le H(U)=n\log2 I fCMI ≤ H ( U ) = n log 2 .
For size-k k k compression, Steinke and Zakynthinou [34 ] give an O ( k log n ) O(k\log n) O ( k log n ) CMI bound, as well as bounds based on differential privacy and stability. For SGD, Neu et al. [28 ] use perturbation analysis involving gradient variance, trajectory-wise local smoothness, and final-output sensitivity. For SGLD, Futami and Fujisawa [10 ] bound I ( W T ; S ) I(W_T;S) I ( W T ; S ) through KL divergences between output laws and obtain iteration-independent bounds under smoothness and dissipativity. Bounds based on Gaussian perturbations or PAC-Bayes KL generally require randomized or noise-injected learners. These weight-based bounds complement, rather than replace, prediction-fCMI and VC/pseudo-dimension bounds.
Relation to PAC-Bayes. Avoiding an explicit complexity term for Π \Pi Π relies on complement symmetry u ↦ u ˉ u\mapsto\bar{u} u ↦ u ˉ , not on change of measure alone. Under the uniform selector law, the difference between the separately re-optimized training and validation values is exactly centered. A PAC-Bayes argument with a data-independent prior on the weights does not by itself give this centering, while a posterior on ( W , π ) (W,\pi) ( W , π ) would generally also pay for the auxiliary variable through its joint posterior–prior KL divergence. A corresponding PAC-Bayes treatment would require a supersample- or split-conditioned prior preserving this symmetry. Grunwald et al. [15 ] derive CMI-like PAC-Bayes bounds using priors conditioned on training and ghost samples, providing a starting point for such a treatment.
Proofs for Section 3.2: stability route
Proof of Lemma 1. Fix a predictor f W f_W f W and a supersample realization z ~ \widetilde{z} z . Write h ( u ) : = Φ n ( f W ; z ~ u ) h(u):=\Phi_n(f_W;\widetilde{z}_u) h ( u ) := Φ n ( f W ; z u ) , Γ ( u ) : = h ( u ˉ ) − h ( u ) \Gamma(u):=h(\bar{u})-h(u) Γ ( u ) := h ( u ˉ ) − h ( u ) . Let u ⊕ i u^{\oplus i} u ⊕ i denote the selector vector obtained by flipping only the i i i -th coordinate. The selected samples z ~ u \widetilde{z}_u z u and z ~ u ⊕ i \widetilde{z}_{u^{\oplus i}} z u ⊕ i differ in exactly one coordinate. By Assumption 2,
∣ h ( u ) − h ( u ⊕ i ) ∣ ≤ β n . |h(u)-h(u^{\oplus i})|\le\beta_n. ∣ h ( u ) − h ( u ⊕ i ) ∣ ≤ β n .
The complementary selected samples z ~ u ˉ \widetilde{z}_{\bar{u}} z u ˉ and z ~ u ⊕ i ‾ \widetilde{z}_{\overline{u^{\oplus i}}} z u ⊕ i also differ in exactly one coordinate, so
∣ h ( u ˉ ) − h ( u ⊕ i ‾ ) ∣ ≤ β n . |h(\bar{u})-h(\overline{u^{\oplus i}})|\le\beta_n. ∣ h ( u ˉ ) − h ( u ⊕ i ) ∣ ≤ β n .
Combining the two displays gives
∣ Γ ( u ) − Γ ( u ⊕ i ) ∣ = ∣ h ( u ˉ ) − h ( u ) − h ( u ⊕ i ‾ ) + h ( u ⊕ i ) ∣ ≤ ∣ h ( u ˉ ) − h ( u ⊕ i ‾ ) ∣ + ∣ h ( u ) − h ( u ⊕ i ) ∣ ≤ 2 β n . \begin{aligned}
|\Gamma(u)-\Gamma(u^{\oplus i})|
&=|h(\bar{u})-h(u)-h(\overline{u^{\oplus i}})+h(u^{\oplus i})|\\
&\le|h(\bar{u})-h(\overline{u^{\oplus i}})|+|h(u)-h(u^{\oplus i})|
\le2\beta_n.
\end{aligned} ∣Γ ( u ) − Γ ( u ⊕ i ) ∣ = ∣ h ( u ˉ ) − h ( u ) − h ( u ⊕ i ) + h ( u ⊕ i ) ∣ ≤ ∣ h ( u ˉ ) − h ( u ⊕ i ) ∣ + ∣ h ( u ) − h ( u ⊕ i ) ∣ ≤ 2 β n .
Thus Γ ( U ) \Gamma(U) Γ ( U ) , as a function of independent selectors under Q U Q_U Q U , has coordinate bounded differences at most 2 β n 2\beta_n 2 β n . The bounded differences inequality gives, for all λ ∈ R \lambda\in\mathbb{R} λ ∈ R ,
log E Q U exp { λ ( Γ ( U ) − E Q U Γ ( U ) ) } ≤ λ 2 8 ∑ i = 1 n ( 2 β n ) 2 = λ 2 n β n 2 2 . (33) \log\mathbb{E}_{Q_U}\exp\{\lambda(\Gamma(U)-\mathbb{E}_{Q_U}\Gamma(U))\}\leq\frac{\lambda^2}{8}\sum_{i=1}^{n}(2\beta_n)^2=\frac{\lambda^2n\beta_n^2}{2}. \tag*{(33)} log E Q U exp { λ ( Γ ( U ) − E Q U Γ ( U ))} ≤ 8 λ 2 i = 1 ∑ n ( 2 β n ) 2 = 2 λ 2 n β n 2 . (33)
Moreover, Lemma 4 gives the reference centering E Q U Γ ( U ) = 0 \mathbb{E}_{Q_U}\Gamma(U)=0 E Q U Γ ( U ) = 0 . Combining this centering with (33) , one may take L Γ , n = β n n L_{\Gamma,n}=\beta_n\sqrt{n} L Γ , n = β n n in Assumption 1. Substituting this value into Theorem 1 gives (9) .
Proof of Theorem 2. We first derive the upper bound of ∣ D t ∣ |D_t| ∣ D t ∣ . For each t t t , the samples S ( t , 0 ) S^{(t,0)} S ( t , 0 ) and S ( t , 1 ) S^{(t,1)} S ( t , 1 ) differ only in coordinate t t t . Assumption 2 therefore gives
∣ D t ∣ = ∣ Φ n ( f W ; S ( t , 0 ) ) − Φ n ( f W ; S ( t , 1 ) ) ∣ ≤ β n . |D_t|=\left|\Phi_n(f_W;S^{(t,0)})-\Phi_n(f_W;S^{(t,1)})\right|\leq\beta_n. ∣ D t ∣ = Φ n ( f W ; S ( t , 0 ) ) − Φ n ( f W ; S ( t , 1 ) ) ≤ β n .
The definition of S ( t , b ) S^{(t,b)} S ( t , b ) uses only Z ~ \widetilde{Z} Z , U − t U_{-t} U − t , and the predictions of f W f_W f W on the supersample. Hence D t D_t D t is measurable with respect to ( f W ( X ~ ) , Z ~ , U − t ) (f_W(\widetilde{X}),\widetilde{Z},U_{-t}) ( f W ( X ) , Z , U − t ) .
Define the mixed path
S ( 0 ) : = S ′ = Z ~ U ‾ , S ( t ) : = ( Z ~ 1 , U 1 , … , Z ~ t , U t , Z ~ t + 1 , 1 − U t + 1 , … , Z ~ n , 1 − U n ) , t = 1 , … , n . S^{(0)}:=S'=\widetilde{Z}_{\overline{U}},\qquad S^{(t)}:=\left(\widetilde{Z}_{1,U_1},\ldots,\widetilde{Z}_{t,U_t},\widetilde{Z}_{t+1,1-U_{t+1}},\ldots,\widetilde{Z}_{n,1-U_n}\right),\quad t=1,\ldots,n. S ( 0 ) := S ′ = Z U , S ( t ) := ( Z 1 , U 1 , … , Z t , U t , Z t + 1 , 1 − U t + 1 , … , Z n , 1 − U n ) , t = 1 , … , n .
Then S ( n ) = S = Z ~ U S^{(n)}=S=\widetilde{Z}_U S ( n ) = S = Z U , while S ( t − 1 ) = S ( t , 1 − U t ) S^{(t-1)}=S^{(t,1-U_t)} S ( t − 1 ) = S ( t , 1 − U t ) and S ( t ) = S ( t , U t ) S^{(t)}=S^{(t,U_t)} S ( t ) = S ( t , U t ) . Therefore
Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) = ∑ t = 1 n { Φ n ( f W ; S ( t − 1 ) ) − Φ n ( f W ; S ( t ) ) } = ∑ t = 1 n { Φ n ( f W ; S ( t , 1 − U t ) ) − Φ n ( f W ; S ( t , U t ) ) } . \begin{aligned}
\Phi_n(f_W;S')-\Phi_n(f_W;S)=\sum_{t=1}^{n}\left\{\Phi_n(f_W;S^{(t-1)})-\Phi_n(f_W;S^{(t)})\right\} \\
&=\sum_{t=1}^{n}\left\{\Phi_n(f_W;S^{(t,1-U_t)})-\Phi_n(f_W;S^{(t,U_t)})\right\}.
\end{aligned} Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) = t = 1 ∑ n { Φ n ( f W ; S ( t − 1 ) ) − Φ n ( f W ; S ( t ) ) } = t = 1 ∑ n { Φ n ( f W ; S ( t , 1 − U t ) ) − Φ n ( f W ; S ( t , U t ) ) } .
If U t = 1 U_t=1 U t = 1 , the t t t -th summand is Φ n ( f W ; S ( t , 0 ) ) − Φ n ( f W ; S ( t , 1 ) ) = D t \Phi_n(f_W;S^{(t,0)})-\Phi_n(f_W;S^{(t,1)})=D_t Φ n ( f W ; S ( t , 0 ) ) − Φ n ( f W ; S ( t , 1 ) ) = D t . If U t = 0 U_t=0 U t = 0 , the same summand is Φ n ( f W ; S ( t , 1 ) ) − Φ n ( f W ; S ( t , 0 ) ) = − D t \Phi_n(f_W;S^{(t,1)})-\Phi_n(f_W;S^{(t,0)})=-D_t Φ n ( f W ; S ( t , 1 ) ) − Φ n ( f W ; S ( t , 0 ) ) = − D t . Hence the summand is always ε t D t \varepsilon_tD_t ε t D t , proving (10) .
Let Q t = Unif ( { 0 , 1 } ) Q_t=\operatorname{Unif}(\{0,1\}) Q t = Unif ({ 0 , 1 }) . Although D t D_t D t may be statistically dependent on U t U_t U t through the learned predictor, the reference MGF holds D t D_t D t fixed and averages over an independent selector drawn from Q t Q_t Q t . Conditional on ( D t , Z ~ ) (D_t,\widetilde{Z}) ( D t , Z ) , the scalar map u t ↦ ( 2 u t − 1 ) D t u_t\mapsto(2u_t-1)D_t u t ↦ ( 2 u t − 1 ) D t satisfies
E Q t exp { λ ( 2 U t − 1 ) D t } = 1 2 e λ D t + 1 2 e − λ D t ≤ exp { λ 2 D t 2 2 } ≤ exp { λ 2 β n 2 2 } . \mathbb{E}_{Q_t}\exp\{\lambda(2U_t-1)D_t\}=\frac{1}{2}e^{\lambda D_t}+\frac{1}{2}e^{-\lambda D_t}\leq\exp\left\{\frac{\lambda^2D_t^2}{2}\right\}\leq\exp\left\{\frac{\lambda^2\beta_n^2}{2}\right\}. E Q t exp { λ ( 2 U t − 1 ) D t } = 2 1 e λ D t + 2 1 e − λ D t ≤ exp { 2 λ 2 D t 2 } ≤ exp { 2 λ 2 β n 2 } .
Applying the conditional Donsker–Varadhan variational inequality to P U t ∣ D t , Z ~ P_{U_t\mid D_t,\widetilde{Z}} P U t ∣ D t , Z relative to Q t Q_t Q t , averaging the conditional KL divergence to get I ( D t ; U t ∣ Z ~ ) I(D_t;U_t\mid\widetilde{Z}) I ( D t ; U t ∣ Z ) , optimizing over λ > 0 \lambda>0 λ > 0 , and repeating the same argument with the negative sign yields
∣ E [ ε t D t ] ∣ ≤ β n 2 I ( D t ; U t ∣ Z ~ ) . (34) \left|\mathbb{E}[\varepsilon_tD_t]\right|\leq\beta_n\sqrt{2I(D_t;U_t\mid\widetilde{Z})}. \tag*{(34)} ∣ E [ ε t D t ] ∣ ≤ β n 2 I ( D t ; U t ∣ Z ) . (34)
Summing (34) over t t t and using (10) proves the single-index term in (11) .
Proof of Corollary 1: MC via stability. Fix a predictor f f f , and let two samples s , s ′ ∈ Z n s,s'\in\mathcal{Z}^n s , s ′ ∈ Z n differ in at most one coordinate, say coordinate i i i . For π = ( c , h , η ) ∈ Π M C \pi=(c,h,\eta)\in\Pi_{\mathrm{MC}} π = ( c , h , η ) ∈ Π MC , define
g π ( f , z ) : = η c ( x ) h ( f ( x ) ) ( y − f ( x ) ) . g_\pi(f,z):=\eta c(x)h(f(x))(y-f(x)). g π ( f , z ) := η c ( x ) h ( f ( x )) ( y − f ( x )) .
Since c ( x ) ∈ [ 0 , 1 ] c(x)\in[0,1] c ( x ) ∈ [ 0 , 1 ] , h ( f ( x ) ) ∈ [ − 1 , 1 ] h(f(x))\in[-1,1] h ( f ( x )) ∈ [ − 1 , 1 ] , η ∈ { ± 1 } \eta\in\{\pm1\} η ∈ { ± 1 } , and y − f ( x ) ∈ [ − 1 , 1 ] y-f(x)\in[-1,1] y − f ( x ) ∈ [ − 1 , 1 ] , we have ∣ g π ( f , z ) ∣ ≤ 1 |g_\pi(f,z)|\leq1 ∣ g π ( f , z ) ∣ ≤ 1 . Hence, for every π \pi π ,
∣ 1 n ∑ k = 1 n g π ( f , z k ) − 1 n ∑ k = 1 n g π ( f , z k ′ ) ∣ = 1 n ∣ g π ( f , z i ) − g π ( f , z i ′ ) ∣ ≤ 2 n . \left|\frac{1}{n}\sum_{k=1}^{n}g_\pi(f,z_k)-\frac{1}{n}\sum_{k=1}^{n}g_\pi(f,z'_k)\right|=\frac{1}{n}\left|g_\pi(f,z_i)-g_\pi(f,z'_i)\right|\leq\frac{2}{n}. n 1 k = 1 ∑ n g π ( f , z k ) − n 1 k = 1 ∑ n g π ( f , z k ′ ) = n 1 ∣ g π ( f , z i ) − g π ( f , z i ′ ) ∣ ≤ n 2 .
Using (22) ,
∣ Φ M C , n ( f ; s ) − Φ M C , n ( f ; s ′ ) ∣ ≤ sup π ∈ Π M C ∣ 1 n ∑ k = 1 n g π ( f , z k ) − 1 n ∑ k = 1 n g π ( f , z k ′ ) ∣ ≤ 2 n . \left|\Phi_{\mathrm{MC},n}(f;s)-\Phi_{\mathrm{MC},n}(f;s^{\prime})\right|\le\sup_{\pi\in\Pi_{\mathrm{MC}}}\left|\frac{1}{n}\sum_{k=1}^{n}g_{\pi}(f,z_k)-\frac{1}{n}\sum_{k=1}^{n}g_{\pi}(f,z_k^{\prime})\right|\le\frac{2}{n}. ∣ Φ MC , n ( f ; s ) − Φ MC , n ( f ; s ′ ) ∣ ≤ π ∈ Π MC sup n 1 k = 1 ∑ n g π ( f , z k ) − n 1 k = 1 ∑ n g π ( f , z k ′ ) ≤ n 2 .
Thus Assumption 2 holds with β n = 2 / n \beta_n=2/n β n = 2/ n . The global bound follows from Lemma 1, and the single-index term follows from Theorem 2; together these give (12) .
Proof of Corollary 3: CVaR via stability. Let α ˉ : = 1 − α \bar{\alpha}:=1-\alpha α ˉ := 1 − α . Fix a predictor f f f , and write L i = ℓ ( f ( x i ) , y i ) ∈ [ 0 , 1 ] L_i=\ell(f(x_i),y_i)\in[0,1] L i = ℓ ( f ( x i ) , y i ) ∈ [ 0 , 1 ] . For a threshold τ ∈ [ 0 , 1 ] \tau\in[0,1] τ ∈ [ 0 , 1 ] , define
G τ ( L 1 , … , L n ) : = τ + 1 α ˉ n ∑ i = 1 n ( L i − τ ) + . G_{\tau}(L_1,\ldots,L_n):=\tau+\frac{1}{\bar{\alpha}n}\sum_{i=1}^{n}(L_i-\tau)_+. G τ ( L 1 , … , L n ) := τ + α ˉ n 1 i = 1 ∑ n ( L i − τ ) + .
If two samples differ in at most one coordinate, then the corresponding loss vectors differ in at most one coordinate. Since ( ⋅ ) + (\cdot)_+ ( ⋅ ) + is 1-Lipschitz and the loss is bounded in [ 0 , 1 ] [0,1] [ 0 , 1 ] ,
∣ G τ ( L ) − G τ ( L ′ ) ∣ ≤ 1 α ˉ n . \left|G_{\tau}(L)-G_{\tau}(L^{\prime})\right|\le\frac{1}{\bar{\alpha}n}. ∣ G τ ( L ) − G τ ( L ′ ) ∣ ≤ α ˉ n 1 .
Taking infima over τ \tau τ and using (23) ,
∣ Φ C V a R , α , n ( f ; s ) − Φ C V a R , α , n ( f ; s ′ ) ∣ ≤ sup τ ∈ [ 0 , 1 ] ∣ G τ ( L ) − G τ ( L ′ ) ∣ ≤ 1 α ˉ n . \left|\Phi_{\mathrm{CVaR},\alpha,n}(f;s)-\Phi_{\mathrm{CVaR},\alpha,n}(f;s^{\prime})\right|\le\sup_{\tau\in[0,1]}\left|G_{\tau}(L)-G_{\tau}(L^{\prime})\right|\le\frac{1}{\bar{\alpha}n}. ∣ Φ CVaR , α , n ( f ; s ) − Φ CVaR , α , n ( f ; s ′ ) ∣ ≤ τ ∈ [ 0 , 1 ] sup ∣ G τ ( L ) − G τ ( L ′ ) ∣ ≤ α ˉ n 1 .
Thus Assumption 2 holds with β n = ( α ˉ n ) − 1 \beta_n=(\bar{\alpha}n)^{-1} β n = ( α ˉ n ) − 1 . The global bound follows from Lemma 1, while the local term follows from Theorem 2. This proves (14) , (58) , and the complete statement (59) below.
Proof of Corollary 4: suboptimal Cressie–Read DRO via stability. Fix a predictor f f f , and write the loss vector as r = ( r 1 , … , r n ) r=(r_1,\ldots,r_n) r = ( r 1 , … , r n ) , where r i = ℓ ( f ( x i ) , y i ) ∈ [ 0 , 1 ] r_i=\ell(f(x_i),y_i)\in[0,1] r i = ℓ ( f ( x i ) , y i ) ∈ [ 0 , 1 ] . The empirical DRO value is
ψ n D R O ( r ) = sup q ∈ Q n , c , ϵ 1 n q T r . \psi_n^{\mathrm{DRO}}(r)=\sup_{q\in\mathcal{Q}_{n,c,\epsilon}}\frac{1}{n}q^{\mathsf{T}}r. ψ n DRO ( r ) = q ∈ Q n , c , ϵ sup n 1 q T r .
If two samples differ in at most one coordinate, the corresponding loss vectors r , r ′ r,r^{\prime} r , r ′ differ in at most one coordinate, say i i i . For any q ∈ Q n , c , ϵ q\in\mathcal{Q}_{n,c,\epsilon} q ∈ Q n , c , ϵ ,
∣ 1 n q T r − 1 n q T r ′ ∣ = 1 n q i ∣ r i − r i ′ ∣ ≤ 1 n ∥ q ∥ ∞ . \left|\frac{1}{n}q^{\mathsf{T}}r-\frac{1}{n}q^{\mathsf{T}}r^{\prime}\right|=\frac{1}{n}q_i|r_i-r_i^{\prime}|\le\frac{1}{n}\|q\|_{\infty}. n 1 q T r − n 1 q T r ′ = n 1 q i ∣ r i − r i ′ ∣ ≤ n 1 ∥ q ∥ ∞ .
Using (22) ,
∣ ψ n D R O ( r ) − ψ n D R O ( r ′ ) ∣ ≤ 1 n sup q ∈ Q n , c , ϵ ∥ q ∥ ∞ . (35) \left|\psi_n^{\mathrm{DRO}}(r)-\psi_n^{\mathrm{DRO}}(r^{\prime})\right|\le\frac{1}{n}\sup_{q\in\mathcal{Q}_{n,c,\epsilon}}\|q\|_{\infty}. \tag*{(35)} ψ n DRO ( r ) − ψ n DRO ( r ′ ) ≤ n 1 q ∈ Q n , c , ϵ sup ∥ q ∥ ∞ . (35)
For any feasible q q q , the Cressie–Read constraint (Eq. (21) ) gives
1 n ∑ j = 1 n q j c ≤ κ c , ϵ c . \frac{1}{n}\sum_{j=1}^{n}q_j^c\le\kappa_{c,\epsilon}^{c}. n 1 j = 1 ∑ n q j c ≤ κ c , ϵ c .
Therefore q j c ≤ n κ c , ϵ c q_j^c\le n\kappa_{c,\epsilon}^{c} q j c ≤ n κ c , ϵ c for every j j j , and hence ∥ q ∥ ∞ ≤ κ c , ϵ n 1 / c \|q\|_{\infty}\le\kappa_{c,\epsilon}n^{1/c} ∥ q ∥ ∞ ≤ κ c , ϵ n 1/ c . Substituting this bound into (35) gives
∣ ψ n D R O ( r ) − ψ n D R O ( r ′ ) ∣ ≤ κ c , ϵ n 1 / c − 1 = κ c , ϵ n 1 / c n . \left|\psi_n^{\mathrm{DRO}}(r)-\psi_n^{\mathrm{DRO}}(r^{\prime})\right|\le\kappa_{c,\epsilon}n^{1/c-1}=\frac{\kappa_{c,\epsilon}n^{1/c}}{n}. ψ n DRO ( r ) − ψ n DRO ( r ′ ) ≤ κ c , ϵ n 1/ c − 1 = n κ c , ϵ n 1/ c .
Thus Assumption 2 holds with β n s t a b D R O , n ≤ κ c , ϵ n 1 / c − 1 \beta_{n}^{\mathrm{stab}}{}_{\mathrm{DRO},n}\le\kappa_{c,\epsilon}n^{1/c-1} β n stab DRO , n ≤ κ c , ϵ n 1/ c − 1 , proving (60) . Lemma 1 gives
Γ L , n D R O , s t a b ≤ κ c , ϵ n 1 / c − 1 n = κ c , ϵ n 1 / c − 1 / 2 . \Gamma_{L,n}^{\mathrm{DRO,stab}}\le\kappa_{c,\epsilon}n^{1/c-1}\sqrt{n}=\kappa_{c,\epsilon}n^{1/c-1/2}. Γ L , n DRO , stab ≤ κ c , ϵ n 1/ c − 1 n = κ c , ϵ n 1/ c − 1/2 .
This proves (61) . The global bound follows from Lemma 1 with β n = κ c , ϵ n 1 / c − 1 \beta_n=\kappa_{c,\epsilon}n^{1/c-1} β n = κ c , ϵ n 1/ c − 1 , while the local term follows from Theorem 2. This gives (15) and the complete statement (62) below. The exponent 1 / c − 1 / 2 1/c-1/2 1/ c − 1/2 comes from the coordinate-wise ℓ ∞ \ell_{\infty} ℓ ∞ sensitivity of the adversarial weights; § 3.3 replaces this by an ℓ 2 \ell_2 ℓ 2 -geometric certificate.
Proofs for Section 3.3: convex L 2 L_2 L 2 route
Evaluation of the following moment generating function is the key in our analysis. This result is adapted from Wainwright [37 ] . For the completeness, we show its proof here;
Lemma 5 (Convex L 2 L_2 L 2 -Lipschitz MGF concentration). Let X 1 , … , X n X_1,\ldots,X_n X 1 , … , X n be independent random variables taking values in [ 0 , 1 ] [0,1] [ 0 , 1 ] , and let ψ : [ 0 , 1 ] n → R \psi:[0,1]^n\to\mathbb{R} ψ : [ 0 , 1 ] n → R be convex and L L L -Lipschitz with respect to the Euclidean norm. Then, for every λ ∈ R \lambda\in\mathbb{R} λ ∈ R ,
log E exp { λ ( ψ ( X ) − E ψ ( X ) ) } ≤ λ 2 L 2 2 . \log\mathbb{E}\exp\{\lambda(\psi(X)-\mathbb{E}\psi(X))\}\leq\frac{\lambda^2L^2}{2}. log E exp { λ ( ψ ( X ) − E ψ ( X ))} ≤ 2 λ 2 L 2 .
Proof. Let P P P be the law of X = ( X 1 , … , X n ) X=(X_1,\ldots,X_n) X = ( X 1 , … , X n ) . We first recall the asymmetric coupling cost used in Wainwright [37 ] . For probability measures P , Q P,Q P , Q on a product space, let Γ ( P , Q ) \Gamma(P,Q) Γ ( P , Q ) denote the set of all couplings M M M of random vectors X ∼ P X\sim P X ∼ P and Y ∼ Q Y\sim Q Y ∼ Q . Define
C ( Q , P ) : = inf M ∈ Γ ( P , Q ) { ∫ ∑ i = 1 n M { Y i ≠ x i ∣ X = x } 2 d P ( x ) } 1 / 2 . C(Q,P):=\inf_{M\in\Gamma(P,Q)}\left\{\int\sum_{i=1}^{n}M\{Y_i\ne x_i\mid X=x\}^{2}\,dP(x)\right\}^{1/2}. C ( Q , P ) := M ∈ Γ ( P , Q ) inf { ∫ i = 1 ∑ n M { Y i = x i ∣ X = x } 2 d P ( x ) } 1/2 .
Equivalently, reversing the roles of P P P and Q Q Q ,
C ( P , Q ) : = inf M ∈ Γ ( P , Q ) { ∫ ∑ i = 1 n M { X i ≠ y i ∣ Y = y } 2 d Q ( y ) } 1 / 2 . C(P,Q):=\inf_{M\in\Gamma(P,Q)}\left\{\int\sum_{i=1}^{n}M\{X_i\ne y_i\mid Y=y\}^{2}\,dQ(y)\right\}^{1/2}. C ( P , Q ) := M ∈ Γ ( P , Q ) inf { ∫ i = 1 ∑ n M { X i = y i ∣ Y = y } 2 d Q ( y ) } 1/2 .
For product measures P P P , Samson’s asymmetric inequality (Wainwright [37 ] ) gives
max { C ( Q , P ) , C ( P , Q ) } ≤ 2 D ( Q ∥ P ) \max\{C(Q,P),C(P,Q)\}\leq\sqrt{2D(Q\mathbin{\|}P)} max { C ( Q , P ) , C ( P , Q )} ≤ 2 D ( Q ∥ P )
for all Q ≪ P Q\ll P Q ≪ P .
We next derive the variational inequality needed below. Let Q ≪ P Q\ll P Q ≪ P , and let M ∈ Γ ( P , Q ) M\in\Gamma(P,Q) M ∈ Γ ( P , Q ) . First suppose that h : [ 0 , 1 ] n → R h:[0,1]^n\to\mathbb{R} h : [ 0 , 1 ] n → R is convex and L L L -Lipschitz with respect to the Euclidean norm. At differentiability points, convexity gives
h ( y ) − h ( x ) ≤ ⟨ ∇ h ( y ) , y − x ⟩ ≤ ∑ j = 1 n ∣ ∂ j h ( y ) ∣ 1 { x j ≠ y j } , h(y)-h(x)\leq\langle\nabla h(y),y-x\rangle\leq\sum_{j=1}^{n}|\partial_jh(y)|1\{x_j\ne y_j\}, h ( y ) − h ( x ) ≤ ⟨ ∇ h ( y ) , y − x ⟩ ≤ j = 1 ∑ n ∣ ∂ j h ( y ) ∣1 { x j = y j } ,
where we used x j , y j ∈ [ 0 , 1 ] x_j,y_j\in[0,1] x j , y j ∈ [ 0 , 1 ] . Hence, with b j ( y ) : = M { X j ≠ y j ∣ Y = y } b_j(y):=M\{X_j\ne y_j\mid Y=y\} b j ( y ) := M { X j = y j ∣ Y = y } , we have
E Q h − E P h ≤ ∫ ∑ j = 1 n ∣ ∂ j h ( y ) ∣ b j ( y ) d Q ( y ) ≤ L ∫ ( ∑ j = 1 n b j ( y ) 2 ) 1 / 2 d Q ( y ) ≤ L { ∫ ∑ j = 1 n b j ( y ) 2 d Q ( y ) } 1 / 2 . \begin{aligned}
\mathbb{E}_Qh-\mathbb{E}_Ph\leq\int\sum_{j=1}^{n}|\partial_jh(y)|b_j(y)\,dQ(y) \\
&\leq L\int\left(\sum_{j=1}^{n}b_j(y)^2\right)^{1/2}\,dQ(y) \\
&\leq L\left\{\int\sum_{j=1}^{n}b_j(y)^2\,dQ(y)\right\}^{1/2}.
\end{aligned} E Q h − E P h ≤ ∫ j = 1 ∑ n ∣ ∂ j h ( y ) ∣ b j ( y ) d Q ( y ) ≤ L ∫ ( j = 1 ∑ n b j ( y ) 2 ) 1/2 d Q ( y ) ≤ L { ∫ j = 1 ∑ n b j ( y ) 2 d Q ( y ) } 1/2 .
Taking the infimum over M ∈ Γ ( P , Q ) M\in\Gamma(P,Q) M ∈ Γ ( P , Q ) yields E Q h − E P h ≤ L C ( P , Q ) ≤ L 2 D ( Q ∥ P ) \mathbb{E}_Qh-\mathbb{E}_Ph\leq LC(P,Q)\leq L\sqrt{2D(Q\mathbin{\|}P)} E Q h − E P h ≤ L C ( P , Q ) ≤ L 2 D ( Q ∥ P ) .
If h h h is concave and L L L -Lipschitz, the same argument is applied with the gradient evaluated at x x x . Indeed, concavity gives
h ( y ) − h ( x ) ≤ ⟨ ∇ h ( x ) , y − x ⟩ ≤ ∑ j = 1 n ∣ ∂ j h ( x ) ∣ 1 { x j ≠ y j } . h(y)-h(x)\leq\langle\nabla h(x),y-x\rangle\leq\sum_{j=1}^{n}|\partial_jh(x)|1\{x_j\ne y_j\}. h ( y ) − h ( x ) ≤ ⟨ ∇ h ( x ) , y − x ⟩ ≤ j = 1 ∑ n ∣ ∂ j h ( x ) ∣1 { x j = y j } .
Thus, with a j ( x ) : = M { Y j ≠ x j ∣ X = x } a_j(x):=M\{Y_j\ne x_j\mid X=x\} a j ( x ) := M { Y j = x j ∣ X = x } , the same Cauchy–Schwarz and Jensen steps give E Q h − E P h ≤ L C ( Q , P ) ≤ L 2 D ( Q ∥ P ) \mathbb{E}_Qh-\mathbb{E}_Ph\leq LC(Q,P)\leq L\sqrt{2D(Q\mathbin{\|}P)} E Q h − E P h ≤ L C ( Q , P ) ≤ L 2 D ( Q ∥ P ) . The non-differentiable case follows by the standard a.e. differentiability of Lipschitz convex functions, or by a routine approximation argument. Therefore, for every h h h that is either convex or concave and L L L -Lipschitz,
E Q h − E P h ≤ L 2 D ( Q ∥ P ) for all Q ≪ P . (36) \mathbb{E}_Qh-\mathbb{E}_Ph\leq L\sqrt{2D(Q\mathbin{\|}P)}\qquad\text{for all }Q\ll P. \tag*{(36)} E Q h − E P h ≤ L 2 D ( Q ∥ P ) for all Q ≪ P . (36)
This is the variational form of the convex/concave concentration argument in Wainwright [37 ] .
We now prove the moment generating function bound. Let h h h be either convex or concave and L L L -Lipschitz, and fix λ > 0 \lambda> 0 λ > 0 . Define the tilted measure
d Q λ d P ( x ) = exp { λ ( h ( x ) − E P h ) } E P exp { λ ( h ( X ) − E P h ) } . \frac{dQ_{\lambda}}{dP}(x)=\frac{\exp\{\lambda(h(x)-\mathbb{E}_{P}h)\}}{\mathbb{E}_{P}\exp\{\lambda(h(X)-\mathbb{E}_{P}h)\}}. d P d Q λ ( x ) = E P exp { λ ( h ( X ) − E P h )} exp { λ ( h ( x ) − E P h )} .
Set Λ h ( λ ) : = log E P e λ ( h ( X ) − E P h ) \Lambda_h(\lambda):=\log\mathbb{E}_{P}e^{\lambda(h(X)-\mathbb{E}_{P}h)} Λ h ( λ ) := log E P e λ ( h ( X ) − E P h ) . Then D ( Q λ ∥ P ) = λ ( E Q λ h − E P h ) − Λ h ( λ ) D(Q_{\lambda}\|P)=\lambda(\mathbb{E}_{Q_{\lambda}}h-\mathbb{E}_{P}h)-\Lambda_h(\lambda) D ( Q λ ∥ P ) = λ ( E Q λ h − E P h ) − Λ h ( λ ) , and hence, by (36) ,
Λ h ( λ ) = λ ( E Q λ h − E P h ) − D ( Q λ ∥ P ) ≤ λ L 2 D ( Q λ ∥ P ) − D ( Q λ ∥ P ) ≤ sup r ≥ 0 { λ L 2 r − r } = λ 2 L 2 2 . \begin{aligned}
\Lambda_h(\lambda)&=\lambda(\mathbb{E}_{Q_{\lambda}}h-\mathbb{E}_{P}h)-D(Q_{\lambda}\|P)\\
&\leq\lambda L\sqrt{2D(Q_{\lambda}\|P)}-D(Q_{\lambda}\|P)\\
&\leq\sup_{r\geq0}\{\lambda L\sqrt{2r}-r\}=\frac{\lambda^2L^2}{2}.
\end{aligned} Λ h ( λ ) = λ ( E Q λ h − E P h ) − D ( Q λ ∥ P ) ≤ λ L 2 D ( Q λ ∥ P ) − D ( Q λ ∥ P ) ≤ r ≥ 0 sup { λ L 2 r − r } = 2 λ 2 L 2 .
Thus, log E P e λ ( h ( X ) − E P h ) ≤ λ 2 L 2 2 \log\mathbb{E}_{P}e^{\lambda(h(X)-\mathbb{E}_{P}h)}\leq\frac{\lambda^2L^2}{2} log E P e λ ( h ( X ) − E P h ) ≤ 2 λ 2 L 2 , λ > 0 \lambda>0 λ > 0 , for every convex or concave L L L -Lipschitz function h h h .
Finally take h = ψ h=\psi h = ψ . Since ψ \psi ψ is convex and L L L -Lipschitz, the preceding display gives log E P e λ ( ψ ( X ) − E P ψ ) ≤ λ 2 L 2 2 \log\mathbb{E}_{P}e^{\lambda(\psi(X)-\mathbb{E}_{P}\psi)}\leq\frac{\lambda^2L^2}{2} log E P e λ ( ψ ( X ) − E P ψ ) ≤ 2 λ 2 L 2 , λ > 0 \lambda>0 λ > 0 . For λ < 0 \lambda<0 λ < 0 , put μ = − λ > 0 \mu=-\lambda>0 μ = − λ > 0 and h = − ψ h=-\psi h = − ψ . Then h h h is concave and L L L -Lipschitz, and μ ( h − E P h ) = λ ( ψ − E P ψ ) \mu(h-\mathbb{E}_{P}h)=\lambda(\psi-\mathbb{E}_{P}\psi) μ ( h − E P h ) = λ ( ψ − E P ψ ) . Applying the positive-μ \mu μ bound to h = − ψ h=-\psi h = − ψ gives log E P e λ ( ψ ( X ) − E P ψ ) ≤ λ 2 L 2 2 \log\mathbb{E}_{P}e^{\lambda(\psi(X)-\mathbb{E}_{P}\psi)}\leq\frac{\lambda^2L^2}{2} log E P e λ ( ψ ( X ) − E P ψ ) ≤ 2 λ 2 L 2 . The case λ = 0 \lambda=0 λ = 0 is trivial, so the bound holds for all λ ∈ R \lambda\in\mathbb{R} λ ∈ R .
Proof of Lemma 2. Fix a predictor f w f_w f w and a supersample realization z ~ \widetilde{z} z . Write ψ n \psi_n ψ n for the convex aggregation map ψ ~ \widetilde{\psi} ψ in Assumption 3. For each selector vector u ∈ { 0 , 1 } n u\in\{0,1\}^n u ∈ { 0 , 1 } n , define the selected score vector
R u : = R f w , n ( z ~ u ) ∈ [ 0 , 1 ] n . R_u:=R_{f_w,n}(\widetilde{z}_u)\in[0,1]^n. R u := R f w , n ( z u ) ∈ [ 0 , 1 ] n .
By Assumption 3,
Φ n ( f w ; z ~ u ) = ψ n ( R u ) , Φ n ( f w ; z ~ u ‾ ) = ψ n ( R u ‾ ) . \Phi_n(f_w;\widetilde{z}_u)=\psi_n(R_u),\qquad\Phi_n(f_w;\widetilde{z}_{\overline{u}})=\psi_n(R_{\overline{u}}). Φ n ( f w ; z u ) = ψ n ( R u ) , Φ n ( f w ; z u ) = ψ n ( R u ) .
The selector gap is therefore
Γ f w , z ~ ( u ) = ψ n ( R u ‾ ) − ψ n ( R u ) . (37) \Gamma_{f_w,\widetilde{z}}(u)=\psi_n(R_{\overline{u}})-\psi_n(R_u). \tag*{(37)} Γ f w , z ( u ) = ψ n ( R u ) − ψ n ( R u ) . (37)
The selector vector is not being relaxed. The only point is that, after fixing ( f w , z ~ ) (f_w,\widetilde{z}) ( f w , z ) , the random vector R U R_U R U is a vector of independent bounded coordinates under the reference selector law. Indeed, since R f w , n R_{f_w,n} R f w , n is coordinatewise, its i i i -th coordinate can be written as a measurable function of the i i i -th data point alone. Thus, for fixed ( f w , z ~ ) (f_w,\widetilde{z}) ( f w , z ) , the coordinate ( R U ) i (R_U)_i ( R U ) i depends only on the selector U i U_i U i . Because U 1 , … , U n U_1,\ldots,U_n U 1 , … , U n are independent under Q U Q_U Q U , the coordinates of R U R_U R U are independent and take values in [ 0 , 1 ] [0,1] [ 0 , 1 ] . The same reasoning applies to R U ‾ R_{\overline{U}} R U , since its i i i -th coordinate depends only on U ‾ i \overline{U}_i U i as well. Consequently the convex L 2 L_2 L 2 -Lipschitz concentration inequality in Lemma 5 applies to both ψ n ( R U ) \psi_n(R_U) ψ n ( R U ) and ψ n ( R U ‾ ) \psi_n(R_{\overline{U}}) ψ n ( R U ) .
The gap is antisymmetric by definition: Γ f w , z ~ ( u ‾ ) = − Γ f w , z ~ ( u ) \Gamma_{f_w,\widetilde{z}}(\overline{u})=-\Gamma_{f_w,\widetilde{z}}(u) Γ f w , z ( u ) = − Γ f w , z ( u ) . Lemma 4 therefore gives E Q U Γ f w , z ~ ( U ) = 0 \mathbb{E}_{Q_U}\Gamma_{f_w,\widetilde{z}}(U)=0 E Q U Γ f w , z ( U ) = 0 . Let μ : = E Q U ψ n ( R U ) \mu:=\mathbb{E}_{Q_U}\psi_n(R_U) μ := E Q U ψ n ( R U ) . The complement map is a bijection on { 0 , 1 } n \{0,1\}^n { 0 , 1 } n , so
E Q U ψ n ( R U ‾ ) = 2 − n ∑ u ∈ { 0 , 1 } n ψ n ( R u ‾ ) = 2 − n ∑ v ∈ { 0 , 1 } n ψ n ( R v ) = μ , (38) \mathbb{E}_{Q_U}\psi_n(R_{\overline{U}})=2^{-n}\sum_{u\in\{0,1\}^n}\psi_n(R_{\overline{u}})=2^{-n}\sum_{v\in\{0,1\}^n}\psi_n(R_v)=\mu, \tag*{(38)} E Q U ψ n ( R U ) = 2 − n u ∈ { 0 , 1 } n ∑ ψ n ( R u ) = 2 − n v ∈ { 0 , 1 } n ∑ ψ n ( R v ) = μ , (38)
where the middle equality uses the change of variables v = u ‾ v=\overline{u} v = u . Combining (37) and (38) ,
Γ f w , z ~ ( U ) = { ψ n ( R U ‾ ) − μ } − { ψ n ( R U ) − μ } . \Gamma_{f_w,\widetilde{z}}(U)=\{\psi_n(R_{\overline{U}})-\mu\}-\{\psi_n(R_U)-\mu\}. Γ f w , z ( U ) = { ψ n ( R U ) − μ } − { ψ n ( R U ) − μ } .
By using Cauchy–Schwarz inequality,
E Q U e λ Γ f w , z ~ ( U ) ≤ ( E Q U e 2 λ ( ψ n ( R U ‾ ) − μ ) ) 1 / 2 ( E Q U e − 2 λ ( ψ n ( R U ) − μ ) ) 1 / 2 . (39) \mathbb{E}_{Q_U}e^{\lambda\Gamma_{f_w,\widetilde{z}}(U)}
\leq\left(\mathbb{E}_{Q_U}e^{2\lambda(\psi_n(R_{\overline{U}})-\mu)}\right)^{1/2}
\left(\mathbb{E}_{Q_U}e^{-2\lambda(\psi_n(R_U)-\mu)}\right)^{1/2}. \tag*{(39)} E Q U e λ Γ f w , z ( U ) ≤ ( E Q U e 2 λ ( ψ n ( R U ) − μ ) ) 1/2 ( E Q U e − 2 λ ( ψ n ( R U ) − μ ) ) 1/2 . (39)
Since R U R_U R U has independent [ 0 , 1 ] [0,1] [ 0 , 1 ] -valued coordinates and ψ n \psi_n ψ n is convex and L ψ , n L_{\psi,n} L ψ , n -Lipschitz, Lemma 5 gives
E Q U e − 2 λ ( ψ n ( R U ) − μ ) ≤ exp { 2 λ 2 L ψ , n 2 } . (40) \mathbb{E}_{Q_U}e^{-2\lambda(\psi_n(R_U)-\mu)} \le\exp\{2\lambda^2L_{\psi,n}^2\}. \tag*{(40)} E Q U e − 2 λ ( ψ n ( R U ) − μ ) ≤ exp { 2 λ 2 L ψ , n 2 } . (40)
The same inequality applied to R U ‾ \overline{R_U} R U gives
E Q U e 2 λ ( ψ n ( R U ‾ ) − μ ) ≤ exp { 2 λ 2 L ψ , n 2 } . (41) \mathbb{E}_{Q_U}e^{2\lambda(\psi_n(\overline{R_U})-\mu)} \le\exp\{2\lambda^2L_{\psi,n}^2\}. \tag*{(41)} E Q U e 2 λ ( ψ n ( R U ) − μ ) ≤ exp { 2 λ 2 L ψ , n 2 } . (41)
Substituting (40) and (41) into (39) , we obtain
E Q U e λ Γ f , z ~ ( U ) ≤ exp { 2 λ 2 L ψ , n 2 } = exp { λ 2 ( 2 L ψ , n ) 2 2 } . \mathbb{E}_{Q_U}e^{\lambda\Gamma_{f,\widetilde{z}}(U)} \le\exp\{2\lambda^2L_{\psi,n}^2\} = \exp\left\{\frac{\lambda^2(2L_{\psi,n})^2}{2}\right\}. E Q U e λ Γ f , z ( U ) ≤ exp { 2 λ 2 L ψ , n 2 } = exp { 2 λ 2 ( 2 L ψ , n ) 2 } .
Thus Assumption 1 holds with L Γ , n ≤ 2 L ψ , n L_{\Gamma,n} \le2L_{\psi,n} L Γ , n ≤ 2 L ψ , n .
Lemma 6 (A convex extension of the empirical MC objective). For the MC objective (2) , the selector gap satisfies Assumption 1 with L Γ , n ≤ 4 / n L_{\Gamma,n} \le4/\sqrt{n} L Γ , n ≤ 4/ n by the following convex extension.
Proof. Fix a predictor f f f and a supersample realization z ~ \widetilde{z} z . For each witness π = ( c , h , η ) ∈ Π M C \pi=(c,h,\eta)\in\Pi_{\mathrm{MC}} π = ( c , h , η ) ∈ Π MC , define
a π , i , j : = η c ( x ~ i , j ) h ( f ( x ~ i , j ) ) ( y ~ i , j − f ( x ~ i , j ) ) , i ∈ [ n ] , j ∈ { 0 , 1 } . a_{\pi,i,j}:=\eta c(\widetilde{x}_{i,j})h(f(\widetilde{x}_{i,j}))(\widetilde{y}_{i,j}-f(\widetilde{x}_{i,j})),\qquad i\in[n],\ j\in\{0,1\}. a π , i , j := η c ( x i , j ) h ( f ( x i , j )) ( y i , j − f ( x i , j )) , i ∈ [ n ] , j ∈ { 0 , 1 } .
Since c ∈ [ 0 , 1 ] c\in[0,1] c ∈ [ 0 , 1 ] , h h h takes values in [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] , η ∈ { ± 1 } \eta\in\{\pm1\} η ∈ { ± 1 } , and y ~ − f ( x ~ ) ∈ [ − 1 , 1 ] \widetilde{y}-f(\widetilde{x})\in[-1,1] y − f ( x ) ∈ [ − 1 , 1 ] , we have ∣ a π , i , j ∣ ≤ 1 |a_{\pi,i,j}|\le1 ∣ a π , i , j ∣ ≤ 1 . For θ ∈ [ 0 , 1 ] n \theta\in[0,1]^n θ ∈ [ 0 , 1 ] n , we interpolate the two witness scores in each row and then take the supremum over witnesses:
H ( θ ) : = sup π ∈ Π M C 1 n ∑ i = 1 n { ( 1 − θ i ) a π , i , 0 + θ i a π , i , 1 } . H(\theta):=\sup_{\pi\in\Pi_{\mathrm{MC}}\!}\frac{1}{n}\sum_{i=1}^{n}\{(1-\theta_i)a_{\pi,i,0}+\theta_i a_{\pi,i,1}\}. H ( θ ) := π ∈ Π MC sup n 1 i = 1 ∑ n {( 1 − θ i ) a π , i , 0 + θ i a π , i , 1 } .
This interpolation keeps the witness optimization intact. For each fixed π \pi π , the displayed average is affine in θ \theta θ , so H H H , as a supremum of affine functions, is convex. Moreover, for any θ , θ ′ ∈ [ 0 , 1 ] n \theta,\theta'\in[0,1]^n θ , θ ′ ∈ [ 0 , 1 ] n , the supremum-difference inequality gives
∣ H ( θ ) − H ( θ ′ ) ∣ ≤ sup π ∈ Π M C 1 n ∣ ∑ i = 1 n ( θ i − θ i ′ ) ( a π , i , 1 − a π , i , 0 ) ∣ ≤ 1 n sup π ∈ Π M C ( ∑ i = 1 n ( a π , i , 1 − a π , i , 0 ) 2 ) 1 / 2 ∥ θ − θ ′ ∥ 2 ≤ 2 n ∥ θ − θ ′ ∥ 2 . \begin{aligned}
|H(\theta)-H(\theta')|
&\le\sup_{\pi\in\Pi_{\mathrm{MC}}}\frac{1}{n}\left|\sum_{i=1}^{n}(\theta_i-\theta_i')(a_{\pi,i,1}-a_{\pi,i,0})\right|\\
&\le\frac{1}{n}\sup_{\pi\in\Pi_{\mathrm{MC}}}\left(\sum_{i=1}^{n}(a_{\pi,i,1}-a_{\pi,i,0})^2\right)^{1/2}\|\theta-\theta'\|_2\\
&\le\frac{2}{\sqrt{n}}\|\theta-\theta'\|_2.
\end{aligned} ∣ H ( θ ) − H ( θ ′ ) ∣ ≤ π ∈ Π MC sup n 1 i = 1 ∑ n ( θ i − θ i ′ ) ( a π , i , 1 − a π , i , 0 ) ≤ n 1 π ∈ Π MC sup ( i = 1 ∑ n ( a π , i , 1 − a π , i , 0 ) 2 ) 1/2 ∥ θ − θ ′ ∥ 2 ≤ n 2 ∥ θ − θ ′ ∥ 2 .
For binary selector vectors, H ( u ) = Φ M C , n ( f ; z ~ u ) H(u)=\Phi_{\mathrm{MC},n}(f;\widetilde{z}_u) H ( u ) = Φ MC , n ( f ; z u ) .
Under the reference law Q U Q_U Q U , the vector U U U has independent [ 0 , 1 ] [0,1] [ 0 , 1 ] -valued coordinates, and H H H is convex and 2 / n 2/\sqrt{n} 2/ n -Lipschitz. Therefore Lemma 5 applies to both H ( U ) H(U) H ( U ) and H ( U ‾ ) H(\overline{U}) H ( U ) . Since the selector gap is Γ f , z ~ ( U ) = H ( U ‾ ) − H ( U ) \Gamma_{f,\widetilde{z}}(U)=H(\overline{U})-H(U) Γ f , z ( U ) = H ( U ) − H ( U ) and H ( U ) H(U) H ( U ) and H ( U ‾ ) H(\overline{U}) H ( U ) have the same reference mean, the Cauchy–Schwarz MGF argument used in Lemma 2 gives log E Q U e λ Γ f , z ~ ( U ) ≤ λ 2 ( 4 / n ) 2 2 \log\mathbb{E}_{Q_U}e^{\lambda\Gamma_{f,\widetilde{z}}(U)}\le\frac{\lambda^2(4/\sqrt{n})^2}{2} log E Q U e λ Γ f , z ( U ) ≤ 2 λ 2 ( 4/ n ) 2 . Thus L Γ , n ≤ 4 / n L_{\Gamma,n}\le4/\sqrt{n} L Γ , n ≤ 4/ n .
The stability certificate gives the sharper MC value 2 / n 2/\sqrt{n} 2/ n , but the interpolation argument explains the convex-geometry entry in Table 1.
Proof of Corollary 5: CVaR via convex L 2 L_2 L 2 geometry. For a fixed predictor f f f , let R f , n ( s ) ∈ [ 0 , 1 ] n R_{f,n}(s)\in[0,1]^n R f , n ( s ) ∈ [ 0 , 1 ] n be the loss vector with coordinates ( R f , n ( s ) ) i = ℓ ( f ( x i ) , y i ) (R_{f,n}(s))_i=\ell(f(x_i),y_i) ( R f , n ( s ) ) i = ℓ ( f ( x i ) , y i ) . This map is coordinatewise, and the empirical CVaR value is ψ n C V a R ( R f , n ( s ) ) \psi_n^{\mathrm{CVaR}}(R_{f,n}(s)) ψ n CVaR ( R f , n ( s )) for
ψ n C V a R ( r ) = sup q ∈ Q α , n 1 n q ⊤ r , r ∈ [ 0 , 1 ] n , \psi_n^{\mathrm{CVaR}}(r)=\sup_{q\in Q_{\alpha,n}}\frac{1}{n}q^\top r,\qquad r\in[0,1]^n, ψ n CVaR ( r ) = q ∈ Q α , n sup n 1 q ⊤ r , r ∈ [ 0 , 1 ] n ,
where
Q α , n = { q ∈ R + n : 1 n ∑ i = 1 n q i = 1 , 0 ≤ q i ≤ α ‾ − 1 } . Q_{\alpha,n}=\left\{q\in\mathbb{R}_+^n:\frac{1}{n}\sum_{i=1}^{n}q_i=1,\ 0\le q_i\le\overline{\alpha}^{-1}\right\}. Q α , n = { q ∈ R + n : n 1 i = 1 ∑ n q i = 1 , 0 ≤ q i ≤ α − 1 } .
Because ψ n C V a R \psi_n^{\mathrm{CVaR}} ψ n CVaR is a supremum of affine functions of r r r , it is convex. The set Q α , n Q_{\alpha,n} Q α , n is compact and nonempty, so for every r r r a maximizer exists. Choose q ⋆ ( r ) ∈ arg max q ∈ Q α , n n − 1 q T r q^\star(r) \in\operatorname{arg\,max}_{q\in Q_{\alpha,n}} n^{-1}q^\mathsf{T}r q ⋆ ( r ) ∈ arg max q ∈ Q α , n n − 1 q T r . For any r , r ′ ∈ [ 0 , 1 ] n r,r^\prime\in[0,1]^n r , r ′ ∈ [ 0 , 1 ] n ,
ψ n C V a R ( r ) − ψ n C V a R ( r ′ ) = 1 n q ⋆ ( r ) T r − sup q ∈ Q α , n 1 n q T r ′ ≤ 1 n q ⋆ ( r ) T r − 1 n q ⋆ ( r ) T r ′ = 1 n q ⋆ ( r ) T ( r − r ′ ) ≤ 1 n ∥ q ⋆ ( r ) ∥ 2 ∥ r − r ′ ∥ 2 . \begin{aligned}
\psi_n^{\mathrm{CVaR}}(r)-\psi_n^{\mathrm{CVaR}}(r^\prime)
&=\frac{1}{n}q^\star(r)^\mathsf{T}r-\sup_{q\in Q_{\alpha,n}}\frac{1}{n}q^\mathsf{T}r^\prime\\
&\le\frac{1}{n}q^\star(r)^\mathsf{T}r-\frac{1}{n}q^\star(r)^\mathsf{T}r^\prime
=\frac{1}{n}q^\star(r)^\mathsf{T}(r-r^\prime)\\
&\le\frac{1}{n}\lVert q^\star(r)\rVert_2\lVert r-r^\prime\rVert_2.
\end{aligned} ψ n CVaR ( r ) − ψ n CVaR ( r ′ ) = n 1 q ⋆ ( r ) T r − q ∈ Q α , n sup n 1 q T r ′ ≤ n 1 q ⋆ ( r ) T r − n 1 q ⋆ ( r ) T r ′ = n 1 q ⋆ ( r ) T ( r − r ′ ) ≤ n 1 ∥ q ⋆ ( r ) ∥ 2 ∥ r − r ′ ∥ 2 .
For any q ∈ Q α , n q\in Q_{\alpha,n} q ∈ Q α , n , since q i ≤ α ˉ − 1 q_i\le\bar{\alpha}^{-1} q i ≤ α ˉ − 1 ,
q i 2 ≤ α ˉ − 1 q i , ∑ i = 1 n q i 2 ≤ α ˉ − 1 ∑ i = 1 n q i = n α ˉ . q_i^2\le\bar{\alpha}^{-1}q_i,\qquad\sum_{i=1}^{n}q_i^2\le\bar{\alpha}^{-1}\sum_{i=1}^{n}q_i=\frac{n}{\bar{\alpha}}. q i 2 ≤ α ˉ − 1 q i , i = 1 ∑ n q i 2 ≤ α ˉ − 1 i = 1 ∑ n q i = α ˉ n .
Thus ∥ q ∥ 2 ≤ n / α ˉ \lVert q\rVert_2\le\sqrt{n/\bar{\alpha}} ∥ q ∥ 2 ≤ n / α ˉ , and
ψ n C V a R ( r ) − ψ n C V a R ( r ′ ) ≤ 1 α ˉ n ∥ r − r ′ ∥ 2 . \psi_n^{\mathrm{CVaR}}(r)-\psi_n^{\mathrm{CVaR}}(r^\prime)\le\frac{1}{\sqrt{\bar{\alpha}n}}\lVert r-r^\prime\rVert_2. ψ n CVaR ( r ) − ψ n CVaR ( r ′ ) ≤ α ˉ n 1 ∥ r − r ′ ∥ 2 .
Interchanging r r r and r ′ r^\prime r ′ gives the reverse difference, so ψ n C V a R \psi_n^{\mathrm{CVaR}} ψ n CVaR is ( α ˉ n ) − 1 / 2 (\bar{\alpha}n)^{-1/2} ( α ˉ n ) − 1/2 -Lipschitz. Lemma 2 gives
Γ n C V a R , L 2 ≤ 2 ( α ˉ n ) − 1 / 2 . (42) \Gamma_{n}^{\mathrm{CVaR},L2}\le2(\bar{\alpha}n)^{-1/2}. \tag*{(42)} Γ n CVaR , L 2 ≤ 2 ( α ˉ n ) − 1/2 . (42)
This proves (42) . Combining this value with (58) gives (16) . □ \square □
Proof of Corollary 6. For a fixed predictor f f f , let R f , n ( s ) ∈ [ 0 , 1 ] n R_{f,n}(s)\in[0,1]^n R f , n ( s ) ∈ [ 0 , 1 ] n be the loss vector with coordinates ( R f , n ( s ) ) i = ℓ ( f ( x i ) , y i ) (R_{f,n}(s))_i=\ell(f(x_i),y_i) ( R f , n ( s ) ) i = ℓ ( f ( x i ) , y i ) . This map is coordinatewise, and the empirical Cressie–Read DRO value is ψ n D R O ( R f , n ( s ) ) \psi_n^{\mathrm{DRO}}(R_{f,n}(s)) ψ n DRO ( R f , n ( s )) , where
ψ n D R O ( r ) = sup q ∈ Q n , c , ϵ 1 n q T r , r ∈ [ 0 , 1 ] n . \psi_n^{\mathrm{DRO}}(r)=\sup_{q\in Q_{n,c,\epsilon}}\frac{1}{n}q^\mathsf{T}r,\qquad r\in[0,1]^n. ψ n DRO ( r ) = q ∈ Q n , c , ϵ sup n 1 q T r , r ∈ [ 0 , 1 ] n .
It is a supremum of affine functions of r r r , and hence is convex. The feasible set Q n , c , ϵ Q_{n,c,\epsilon} Q n , c , ϵ is closed, bounded, and nonempty, so a maximizer exists for every r r r . Choose q ⋆ ( r ) ∈ arg max q ∈ Q n , c , ϵ n − 1 q T r q^\star(r)\in\operatorname{arg\,max}_{q\in Q_{n,c,\epsilon}}n^{-1}q^\mathsf{T}r q ⋆ ( r ) ∈ arg max q ∈ Q n , c , ϵ n − 1 q T r . For any r , r ′ ∈ [ 0 , 1 ] n r,r^\prime\in[0,1]^n r , r ′ ∈ [ 0 , 1 ] n ,
ψ n D R O ( r ) − ψ n D R O ( r ′ ) = 1 n q ⋆ ( r ) T r − sup q ∈ Q n , c , ϵ 1 n q T r ′ ≤ 1 n q ⋆ ( r ) T r − 1 n q ⋆ ( r ) T r ′ = 1 n q ⋆ ( r ) T ( r − r ′ ) ≤ 1 n ∥ q ⋆ ( r ) ∥ 2 ∥ r − r ′ ∥ 2 . \begin{aligned}
\psi_n^{\mathrm{DRO}}(r)-\psi_n^{\mathrm{DRO}}(r^\prime)
&=\frac{1}{n}q^\star(r)^\mathsf{T}r-\sup_{q\in Q_{n,c,\epsilon}}\frac{1}{n}q^\mathsf{T}r^\prime\\
&\le\frac{1}{n}q^\star(r)^\mathsf{T}r-\frac{1}{n}q^\star(r)^\mathsf{T}r^\prime
=\frac{1}{n}q^\star(r)^\mathsf{T}(r-r^\prime)\\
&\le\frac{1}{n}\lVert q^\star(r)\rVert_2\lVert r-r^\prime\rVert_2.
\end{aligned} ψ n DRO ( r ) − ψ n DRO ( r ′ ) = n 1 q ⋆ ( r ) T r − q ∈ Q n , c , ϵ sup n 1 q T r ′ ≤ n 1 q ⋆ ( r ) T r − n 1 q ⋆ ( r ) T r ′ = n 1 q ⋆ ( r ) T ( r − r ′ ) ≤ n 1 ∥ q ⋆ ( r ) ∥ 2 ∥ r − r ′ ∥ 2 .
Taking the supremum over feasible q q q and interchanging r r r and r ′ r^\prime r ′ ,
∣ ψ n D R O ( r ) − ψ n D R O ( r ′ ) ∣ ≤ 1 n sup q ∈ Q n , c , ϵ ∥ q ∥ 2 ∥ r − r ′ ∥ 2 . (43) \left|\psi_n^{\mathrm{DRO}}(r)-\psi_n^{\mathrm{DRO}}(r^\prime)\right|
\le\frac{1}{n}\sup_{q\in Q_{n,c,\epsilon}}\lVert q\rVert_2\lVert r-r^\prime\rVert_2. \tag*{(43)} ψ n DRO ( r ) − ψ n DRO ( r ′ ) ≤ n 1 q ∈ Q n , c , ϵ sup ∥ q ∥ 2 ∥ r − r ′ ∥ 2 . (43)
It remains to evaluate sup q ∥ q ∥ 2 \sup_q\lVert q\rVert_2 sup q ∥ q ∥ 2 . The Cressie–Read constraint gives
1 n ∑ i = 1 n q i c ≤ κ c , ϵ c , so ∥ q ∥ c = ( ∑ i = 1 n q i c ) 1 / c ≤ κ c , ϵ n 1 / c . \frac{1}{n}\sum_{i=1}^{n}q_i^c\le\kappa_{c,\epsilon}^c,\qquad
\text{so}\qquad
\lVert q\rVert_c=\left(\sum_{i=1}^{n}q_i^c\right)^{1/c}\le\kappa_{c,\epsilon}n^{1/c}. n 1 i = 1 ∑ n q i c ≤ κ c , ϵ c , so ∥ q ∥ c = ( i = 1 ∑ n q i c ) 1/ c ≤ κ c , ϵ n 1/ c .
If c ≥ 2 c\ge2 c ≥ 2 , the norm comparison ∥ q ∥ 2 ≤ n 1 / 2 − 1 / c ∥ q ∥ c \lVert q\rVert_2\le n^{1/2-1/c}\lVert q\rVert_c ∥ q ∥ 2 ≤ n 1/2 − 1/ c ∥ q ∥ c yields
∥ q ∥ 2 ≤ n 1 / 2 − 1 / c κ c , ϵ n 1 / c = κ c , ϵ n 1 / 2 . \lVert q\rVert_2\le n^{1/2-1/c}\kappa_{c,\epsilon}n^{1/c}=\kappa_{c,\epsilon}n^{1/2}. ∥ q ∥ 2 ≤ n 1/2 − 1/ c κ c , ϵ n 1/ c = κ c , ϵ n 1/2 .
If 1 < c < 2 1<c<2 1 < c < 2 , then ∥ q ∥ 2 ≤ ∥ q ∥ c \lVert q\rVert_2\le\lVert q\rVert_c ∥ q ∥ 2 ≤ ∥ q ∥ c , and hence
∥ q ∥ 2 ≤ κ c , ϵ n 1 / c . \lVert q\rVert_2\le\kappa_{c,\epsilon}n^{1/c}. ∥ q ∥ 2 ≤ κ c , ϵ n 1/ c .
Therefore
1 n sup q ∈ Q n , c , ϵ ∥ q ∥ 2 ≤ κ c , ϵ { n − 1 / 2 , c ≥ 2 , n 1 / c − 1 , 1 < c < 2 = κ c , ϵ { n − 1 / 2 , c ≥ 2 , n − 1 / c ∗ , 1 < c < 2 = κ c , ϵ n − 1 / ( c ∗ ∨ 2 ) . (44) \frac{1}{n}\sup_{q\in Q_{n,c,\epsilon}}\lVert q\rVert_2 \le\kappa_{c,\epsilon}
\begin{cases}
n^{-1/2}, & c\ge2,\\
n^{1/c-1}, & 1<c<2
\end{cases}
= \kappa_{c,\epsilon}
\begin{cases}
n^{-1/2}, & c\ge2,\\
n^{-1/c^*}, & 1<c<2
\end{cases}
= \kappa_{c,\epsilon}n^{-1/(c^*\vee2)}. \tag*{(44)} n 1 q ∈ Q n , c , ϵ sup ∥ q ∥ 2 ≤ κ c , ϵ { n − 1/2 , n 1/ c − 1 , c ≥ 2 , 1 < c < 2 = κ c , ϵ { n − 1/2 , n − 1/ c ∗ , c ≥ 2 , 1 < c < 2 = κ c , ϵ n − 1/ ( c ∗ ∨ 2 ) . (44)
Combining this with (43) proves (44) . Lemma 2 gives
L Γ , n D R O , L 2 ≤ 2 L ψ , n D R O ≤ 2 κ c , ϵ n − 1 / ( c ∗ ∨ 2 ) . (45) L_{\Gamma,n}^{\mathrm{DRO},L_2}\le2L_{\psi,n}^{\mathrm{DRO}}\le2\kappa_{c,\epsilon}n^{-1/(c^*\vee2)}. \tag*{(45)} L Γ , n DRO , L 2 ≤ 2 L ψ , n DRO ≤ 2 κ c , ϵ n − 1/ ( c ∗ ∨ 2 ) . (45)
This proves (45) . Finally, Theorem 1 gives
∣ E Δ D R O ∣ ≤ 2 κ c , ϵ n − 1 / ( c ∗ ∨ 2 ) 2 I ( f W ( X ~ ) ; U ∣ Z ~ ) , \lvert E\Delta_{\mathrm{DRO}}\rvert\le2\kappa_{c,\epsilon}n^{-1/(c^*\vee2)}\sqrt{2I(f_W(\widetilde{X});U\mid\widetilde{Z})}, ∣ E Δ DRO ∣ ≤ 2 κ c , ϵ n − 1/ ( c ∗ ∨ 2 ) 2 I ( f W ( X ) ; U ∣ Z ) ,
which is (17) .
Proofs for Section 3.4: fixed-predictor terms
Corollary 8 (Fixed-predictor CVaR term). Assume 0 ≤ ℓ ( f ( X ) , Y ) ≤ 1 0\le\ell(f(X),Y)\le1 0 ≤ ℓ ( f ( X ) , Y ) ≤ 1 for all predictors and data points. For CVaR, we have
sup f E S ′ ∼ D n ∣ Φ C V a R , α ( f ; D ) − Φ C V a R , α , n ( f ; S ′ ) ∣ ≤ 1 α π 2 n . (46) \sup_f \mathbb{E}_{S'\sim D^n}\left\lvert\Phi_{\mathrm{CVaR},\alpha}(f;D)-\Phi_{\mathrm{CVaR},\alpha,n}(f;S')\right\rvert\le\frac{1}{\alpha}\sqrt{\frac{\pi}{2n}}. \tag*{(46)} f sup E S ′ ∼ D n ∣ Φ CVaR , α ( f ; D ) − Φ CVaR , α , n ( f ; S ′ ) ∣ ≤ α 1 2 n π . (46) Therefore, with L Γ , n C V a R L_{\Gamma,n}^{\mathrm{CVaR}} L Γ , n CVaR chosen as in (16) ,
E ∣ Φ C V a R , α ( f W ; D ) − Φ C V a R , α , n ( f W ; S ′ ) ∣ ≤ 1 α π 2 n + L Γ , n C V a R 2 { I ( f W ( X ~ ) ; U ∣ Z ~ ) + log 2 } . (47) \mathbb{E}\left\lvert\Phi_{\mathrm{CVaR},\alpha}(f_W;D)-\Phi_{\mathrm{CVaR},\alpha,n}(f_W;S')\right\rvert\le\frac{1}{\alpha}\sqrt{\frac{\pi}{2n}}+L_{\Gamma,n}^{\mathrm{CVaR}}\sqrt{2\{I(f_W(\widetilde{X});U\mid\widetilde{Z})+\log2\}}. \tag*{(47)} E ∣ Φ CVaR , α ( f W ; D ) − Φ CVaR , α , n ( f W ; S ′ ) ∣ ≤ α 1 2 n π + L Γ , n CVaR 2 { I ( f W ( X ) ; U ∣ Z ) + log 2 } . (47)
Corollary 9 (Duchi–Namkoong fixed-predictor Cressie–Read DRO term). Let c > 1 c>1 c > 1 , c ∗ = c / ( c − 1 ) c^*=c/(c-1) c ∗ = c / ( c − 1 ) , r c : = 1 / ( c ∗ ∨ 2 ) r_c:=1/(c^*\vee2) r c := 1/ ( c ∗ ∨ 2 ) , and let ϵ > 0 \epsilon>0 ϵ > 0 . Define
κ c , ϵ : = { 1 + c ( c − 1 ) ϵ } 1 / c , K c , ϵ : = κ c , ϵ 2 ( κ c , ϵ κ c , ϵ − 1 ∨ 2 ) . \kappa_{c,\epsilon}:=\{1+c(c-1)\epsilon\}^{1/c},\qquad K_{c,\epsilon}:=\kappa_{c,\epsilon}^2\left(\frac{\kappa_{c,\epsilon}}{\kappa_{c,\epsilon}-1}\vee2\right). κ c , ϵ := { 1 + c ( c − 1 ) ϵ } 1/ c , K c , ϵ := κ c , ϵ 2 ( κ c , ϵ − 1 κ c , ϵ ∨ 2 ) . Assume 0 ≤ ℓ ( f ( X ) , Y ) ≤ 1 0\le\ell(f(X),Y)\le1 0 ≤ ℓ ( f ( X ) , Y ) ≤ 1 for all predictors and data points, n ≥ c ∨ 3 n\ge c\vee3 n ≥ c ∨ 3 , and the fixed-parameter conditions of Duchi and Namkoong [6 ] hold uniformly for the scalar loss laws induced by the predictors under consideration. Then
E f W , S ′ ∣ Φ D R O , c , ϵ ( f W ; D ) − Φ D R O , c , ϵ , n ( f W ; S ′ ) ∣ ≤ O ~ ( K c , ϵ n − r c ) . (48) \mathbb{E}_{f_W,S'}\left\lvert\Phi_{\mathrm{DRO},c,\epsilon}(f_W;D)-\Phi_{\mathrm{DRO},c,\epsilon,n}(f_W;S')\right\rvert\le\widetilde{O}(K_{c,\epsilon}n^{-r_c}). \tag*{(48)} E f W , S ′ ∣ Φ DRO , c , ϵ ( f W ; D ) − Φ DRO , c , ϵ , n ( f W ; S ′ ) ∣ ≤ O ( K c , ϵ n − r c ) . (48) Consequently, combining this fixed-predictor term with Corollary 6 gives
E ∣ Φ D R O , c , ϵ ( f W ; D ) − Φ D R O , c , ϵ , n ( f W ; S ′ ) ∣ ≤ O ~ ( K c , ϵ n − r c ) + 2 κ c , ϵ n − r c 2 { I ( f W ( X ~ ) ; U ∣ Z ~ ) + log 2 } . (49) \mathbb{E}\left\lvert\Phi_{\mathrm{DRO},c,\epsilon}(f_W;D)-\Phi_{\mathrm{DRO},c,\epsilon,n}(f_W;S')\right\rvert\le\widetilde{O}(K_{c,\epsilon}n^{-r_c})
+2\kappa_{c,\epsilon}n^{-r_c}\sqrt{2\{I(f_W(\widetilde{X});U\mid\widetilde{Z})+\log2\}}. \tag*{(49)} E ∣ Φ DRO , c , ϵ ( f W ; D ) − Φ DRO , c , ϵ , n ( f W ; S ′ ) ∣ ≤ O ( K c , ϵ n − r c ) + 2 κ c , ϵ n − r c 2 { I ( f W ( X ) ; U ∣ Z ) + log 2 } . (49) Here O ~ \widetilde{O} O hides only logarithmic factors in n n n and universal numerical constants, not the displayed K c , ϵ K_{c,\epsilon} K c , ϵ dependence. At ϵ = 0 \epsilon=0 ϵ = 0 , the ambiguity set collapses to the ordinary mean, and the first term can be replaced by the standard bounded-mean comparison.
Proof of Corollary 8: fixed-predictor CVaR term. Fix a predictor f f f , and let L = ℓ ( f ( X ) , Y ) ∈ [ 0 , 1 ] L=\ell(f(X),Y)\in[0,1] L = ℓ ( f ( X ) , Y ) ∈ [ 0 , 1 ] and L i = ℓ ( f ( X i ′ ) , Y i ′ ) L_i=\ell(f(X'_i),Y'_i) L i = ℓ ( f ( X i ′ ) , Y i ′ ) . For τ ∈ [ 0 , 1 ] \tau\in[0,1] τ ∈ [ 0 , 1 ] , define
G ( τ ) : = τ + 1 α E ( L − τ ) + , G(\tau):=\tau+\frac{1}{\alpha}\mathbb{E}(L-\tau)_+, G ( τ ) := τ + α 1 E ( L − τ ) + ,
G n ( τ ) : = τ + 1 α n ∑ i = 1 n ( L i − τ ) + . G_n(\tau):=\tau+\frac{1}{\alpha n}\sum_{i=1}^{n}(L_i-\tau)_+. G n ( τ ) := τ + α n 1 i = 1 ∑ n ( L i − τ ) + .
Using (23) ,
∣ Φ C V a R , α ( f ; D ) − Φ C V a R , α , n ( f ; S ′ ) ∣ ≤ sup τ ∈ [ 0 , 1 ] ∣ G ( τ ) − G n ( τ ) ∣ = 1 α ˉ sup τ ∈ [ 0 , 1 ] ∣ E ( L − τ ) + − 1 n ∑ i = 1 n ( L i − τ ) + ∣ . \begin{aligned}
\left|\Phi_{\mathrm{CVaR},\alpha}(f;D)-\Phi_{\mathrm{CVaR},\alpha,n}(f;S')\right|
\le\sup_{\tau\in[0,1]}\left|G(\tau)-G_n(\tau)\right| \\
&= \frac{1}{\bar{\alpha}}\sup_{\tau\in[0,1]}\left|\mathbb{E}(L-\tau)_+ - \frac{1}{n}\sum_{i=1}^{n}(L_i-\tau)_+\right|.
\end{aligned} ∣ Φ CVaR , α ( f ; D ) − Φ CVaR , α , n ( f ; S ′ ) ∣ ≤ τ ∈ [ 0 , 1 ] sup ∣ G ( τ ) − G n ( τ ) ∣ = α ˉ 1 τ ∈ [ 0 , 1 ] sup E ( L − τ ) + − n 1 i = 1 ∑ n ( L i − τ ) + .
The identity ( x − τ ) + = ∫ τ 1 1 { x > s } d s (x-\tau)_+ = \int_{\tau}^{1}\mathbf{1}\{x>s\}\,ds ( x − τ ) + = ∫ τ 1 1 { x > s } d s for x ∈ [ 0 , 1 ] x\in[0,1] x ∈ [ 0 , 1 ] gives
sup τ ∈ [ 0 , 1 ] ∣ E ( L − τ ) + − 1 n ∑ i = 1 n ( L i − τ ) + ∣ ≤ ∫ 0 1 sup s ∈ [ 0 , 1 ] ∣ Pr ( L > s ) − 1 n ∑ i = 1 n 1 { L i > s } ∣ d s = sup s ∈ [ 0 , 1 ] ∣ Pr ( L > s ) − 1 n ∑ i = 1 n 1 { L i > s } ∣ . \begin{aligned}
\sup_{\tau\in[0,1]}\left|\mathbb{E}(L-\tau)_+ - \frac{1}{n}\sum_{i=1}^{n}(L_i-\tau)_+\right| \\
&\le\int_{0}^{1}\sup_{s\in[0,1]}\left|\Pr(L>s)-\frac{1}{n}\sum_{i=1}^{n}\mathbf{1}\{L_i>s\}\right|\,ds \\
&= \sup_{s\in[0,1]}\left|\Pr(L>s)-\frac{1}{n}\sum_{i=1}^{n}\mathbf{1}\{L_i>s\}\right|.
\end{aligned} τ ∈ [ 0 , 1 ] sup E ( L − τ ) + − n 1 i = 1 ∑ n ( L i − τ ) + ≤ ∫ 0 1 s ∈ [ 0 , 1 ] sup Pr ( L > s ) − n 1 i = 1 ∑ n 1 { L i > s } d s = s ∈ [ 0 , 1 ] sup Pr ( L > s ) − n 1 i = 1 ∑ n 1 { L i > s } .
Let M n : = sup s ∈ [ 0 , 1 ] ∣ Pr ( L > s ) − n − 1 ∑ i = 1 n 1 { L i > s } ∣ M_n := \sup_{s\in[0,1]}\left|\Pr(L>s)-n^{-1}\sum_{i=1}^{n}\mathbf{1}\{L_i>s\}\right| M n := sup s ∈ [ 0 , 1 ] Pr ( L > s ) − n − 1 ∑ i = 1 n 1 { L i > s } . The Dvoretzky–Kiefer–Wolfowitz–Massart inequality gives Pr ( M n > t ) ≤ 2 exp ( − 2 n t 2 ) \Pr(M_n>t)\le2\exp(-2nt^2) Pr ( M n > t ) ≤ 2 exp ( − 2 n t 2 ) for all t > 0 t>0 t > 0 [25 ] . Therefore
E M n = ∫ 0 ∞ Pr ( M n > t ) d t ≤ ∫ 0 ∞ 2 e − 2 n t 2 d t = π 2 n . \mathbb{E}M_n=\int_{0}^{\infty}\Pr(M_n>t)\,dt\le\int_{0}^{\infty}2e^{-2nt^2}\,dt=\sqrt{\frac{\pi}{2n}}. E M n = ∫ 0 ∞ Pr ( M n > t ) d t ≤ ∫ 0 ∞ 2 e − 2 n t 2 d t = 2 n π .
This bound is distribution-free over all scalar laws supported on [ 0 , 1 ] [0,1] [ 0 , 1 ] . Hence it holds uniformly in the fixed predictor f f f , and it also applies after conditioning on the learned predictor f W f_W f W . Multiplying by α ˉ − 1 \bar{\alpha}^{-1} α ˉ − 1 proves (46) .
The total bound (47) follows from (18) , the triangle inequality, (46) , and the absolute validation–training bound (8) with L T , n C V a R L^{\mathrm{CVaR}}_{T,n} L T , n CVaR as in (16) .
Proof of Corollary 9: fixed-predictor Cressie–Read DRO term. Let f W f_W f W be the learned predictor and condition on a realization f W = f f_W=f f W = f . The scalar loss L f ( Z ) = ℓ ( f ( X ) , Y ) L_f(Z)=\ell(f(X),Y) L f ( Z ) = ℓ ( f ( X ) , Y ) is then fixed and takes values in [ 0 , 1 ] [0,1] [ 0 , 1 ] . Write X f : = ∣ Φ D R O , c , ϵ ( f ; D ) − Φ D R O , c , ϵ , n ( f ; S ′ ) ∣ X_f:=\left|\Phi_{\mathrm{DRO},c,\epsilon}(f;D)-\Phi_{\mathrm{DRO},c,\epsilon,n}(f;S')\right| X f := ∣ Φ DRO , c , ϵ ( f ; D ) − Φ DRO , c , ϵ , n ( f ; S ′ ) ∣ . For f c ( t ) = ( t c − c t + c − 1 ) / ( c ( c − 1 ) ) f_c(t)=(t^c-ct+c-1)/(c(c-1)) f c ( t ) = ( t c − c t + c − 1 ) / ( c ( c − 1 )) , Duchi and Namkoong [6 ] gives, under the stated fixed-parameter conditions, for every t > 0 t>0 t > 0 ,
X f ≤ 10 K c , ϵ n − r c ( 1 c + t + 2 log n ) (50) X_f\le10K_{c,\epsilon}n^{-r_c}\left(\frac{1}{c}+\sqrt{t}+2\log n\right) \tag*{(50)} X f ≤ 10 K c , ϵ n − r c ( c 1 + t + 2 log n ) (50)
with probability at least 1 − 2 e − t 1-2e^{-t} 1 − 2 e − t . The constants are uniform over the scalar loss laws considered, so the same display holds after conditioning on any realization of f W f_W f W .
Integrating the conditional tail bound yields the expectation bound. Indeed, with B c , ϵ , n = 10 K c , ϵ n − r c B_{c,\epsilon,n}=10K_{c,\epsilon}n^{-r_c} B c , ϵ , n = 10 K c , ϵ n − r c and a c , n = c − 1 + 2 log n a_{c,n}=c^{-1}+2\log n a c , n = c − 1 + 2 log n , (50) implies
Pr { X f > B c , ϵ , n ( a c , n + s ) ∣ f W = f } ≤ 2 e − s 2 , s ≥ 0. \Pr\{X_f>B_{c,\epsilon,n}(a_{c,n}+s)\mid f_W=f\}\le2e^{-s^2},\qquad s\ge0. Pr { X f > B c , ϵ , n ( a c , n + s ) ∣ f W = f } ≤ 2 e − s 2 , s ≥ 0.
Therefore
E [ X f ∣ f W = f ] ≤ B c , ϵ , n a c , n + B c , ϵ , n ∫ 0 ∞ min { 1 , 2 e − s 2 } d s ≤ C K c , ϵ n − r c ( 1 + log n ) , \begin{aligned}
\mathbb{E}[X_f\mid f_W=f]\le B_{c,\epsilon,n}a_{c,n}+B_{c,\epsilon,n}\int_{0}^{\infty}\min\{1,2e^{-s^2}\}\,ds \\
&\le CK_{c,\epsilon}n^{-r_c}(1+\log n),
\end{aligned} E [ X f ∣ f W = f ] ≤ B c , ϵ , n a c , n + B c , ϵ , n ∫ 0 ∞ min { 1 , 2 e − s 2 } d s ≤ C K c , ϵ n − r c ( 1 + log n ) ,
for a universal numerical constant C C C . Averaging over f W f_W f W proves (48) . The boundary case ϵ = 0 \epsilon=0 ϵ = 0 is handled separately by the ordinary bounded-mean comparison, since the ambiguity set is the singleton containing the nominal law.
This argument uses only the fixed-predictor theorem after conditioning on f W f_W f W . It does not use the separate uniform-in-class covering-number bound of Duchi and Namkoong [6 ] ; the remaining randomness is the independent validation sample S ′ S' S ′ , while the algorithm-dependent part is controlled by the validation–training fCMI term.
Using (18) and the triangle inequality,
E ∣ Φ D R O , c , ϵ ( f W ; D ) − Φ D R O , c , ϵ , n ( f W ; S ) ∣ ≤ E ∣ Φ D R O , c , ϵ ( f W ; D ) − Φ D R O , c , ϵ , n ( f W ; S ′ ) ∣ + E ∣ Φ D R O , c , ϵ , n ( f W ; S ′ ) − Φ D R O , c , ϵ , n ( f W ; S ) ∣ . \mathbb{E}\left|\Phi_{\mathrm{DRO},c,\epsilon}(f_W;\mathcal{D})-\Phi_{\mathrm{DRO},c,\epsilon,n}(f_W;S)\right| \leq\mathbb{E}\left|\Phi_{\mathrm{DRO},c,\epsilon}(f_W;\mathcal{D})-\Phi_{\mathrm{DRO},c,\epsilon,n}(f_W;S')\right| + \mathbb{E}\left|\Phi_{\mathrm{DRO},c,\epsilon,n}(f_W;S')-\Phi_{\mathrm{DRO},c,\epsilon,n}(f_W;S)\right|. E ∣ Φ DRO , c , ϵ ( f W ; D ) − Φ DRO , c , ϵ , n ( f W ; S ) ∣ ≤ E ∣ Φ DRO , c , ϵ ( f W ; D ) − Φ DRO , c , ϵ , n ( f W ; S ′ ) ∣ + E ∣ Φ DRO , c , ϵ , n ( f W ; S ′ ) − Φ DRO , c , ϵ , n ( f W ; S ) ∣ .
The first term is bounded by (48) . Applying Theorem 1, Eq. (8) , with L Γ , n D R O , L 2 = 2 κ c , ϵ n − r c L_{\Gamma,n}^{\mathrm{DRO},L_2}=2\kappa_{c,\epsilon}n^{-r_c} L Γ , n DRO , L 2 = 2 κ c , ϵ n − r c bounds the second term and gives (49) .
Lemma 7 (Fixed-predictor MC empirical-process bound). For every fixed predictor f : X → [ 0 , 1 ] f:\mathcal{X}\to[0,1] f : X → [ 0 , 1 ] , define U C n M C ( f ) : = E S ′ ∼ D n ∣ Φ M C ( f ; D ) − Φ M C , n ( f ; S ′ ) ∣ \mathrm{UC}^{\mathrm{MC}}_n(f):=\mathbb{E}_{S'\sim\mathcal{D}^n}\left|\Phi_{\mathrm{MC}}(f;\mathcal{D})-\Phi_{\mathrm{MC},n}(f;S')\right| UC n MC ( f ) := E S ′ ∼ D n ∣ Φ MC ( f ; D ) − Φ MC , n ( f ; S ′ ) ∣ . Let C ± : = { η c : c ∈ C , η ∈ { ± 1 } } \mathcal{C}_{\pm}:=\{\eta c:c\in\mathcal{C},\ \eta\in\{\pm1\}\} C ± := { η c : c ∈ C , η ∈ { ± 1 }} and define the fixed-predictor MC product class
G f : = { ( x , y ) ↦ a ( x ) h ( f ( x ) ) ( y − f ( x ) ) : a ∈ C ± , h ∈ H } . \mathcal{G}_f:=\{(x,y)\mapsto a(x)h(f(x))(y-f(x)):a\in\mathcal{C}_{\pm},\ h\in\mathcal{H}\}. G f := {( x , y ) ↦ a ( x ) h ( f ( x )) ( y − f ( x )) : a ∈ C ± , h ∈ H } . Here R D , n ( G f ) \mathfrak{R}_{D,n}(\mathcal{G}_f) R D , n ( G f ) is the usual Rademacher average on Z Z Z -valued samples, while R D , n ( C ± ) \mathfrak{R}_{D,n}(\mathcal{C}_{\pm}) R D , n ( C ± ) is evaluated under the X X X -marginal of D \mathcal{D} D . Then
U C n M C ( f ) ≤ 2 R D , n ( G f ) . (51) \mathrm{UC}^{\mathrm{MC}}_n(f)\leq2\mathfrak{R}_{D,n}(\mathcal{G}_f). \tag*{(51)} UC n MC ( f ) ≤ 2 R D , n ( G f ) . (51) Moreover, the product structure gives the sufficient separable bound
R D , n ( G f ) ≲ R D , n ( C ± ) + inf δ ∈ [ 0 , 1 ] { δ + ∫ δ 1 log N ( ε , H , ∥ ⋅ ∥ ∞ ) n d ε } . (52) \mathfrak{R}_{D,n}(\mathcal{G}_f)\lesssim\mathfrak{R}_{D,n}(\mathcal{C}_{\pm})+\inf_{\delta\in[0,1]}\left\{\delta+\int_{\delta}^{1}\sqrt{\frac{\log N(\varepsilon,\mathcal{H},\|\cdot\|_{\infty})}{n}}\,\mathrm{d}\varepsilon\right\}. \tag*{(52)} R D , n ( G f ) ≲ R D , n ( C ± ) + δ ∈ [ 0 , 1 ] inf { δ + ∫ δ 1 n log N ( ε , H , ∥ ⋅ ∥ ∞ ) d ε } . (52) Consequently,
U C n M C ( f ) ≲ R D , n ( C ± ) + inf δ ∈ [ 0 , 1 ] { δ + ∫ δ 1 log N ( ε , H , ∥ ⋅ ∥ ∞ ) n d ε } . (53) \mathrm{UC}^{\mathrm{MC}}_n(f)\lesssim\mathfrak{R}_{D,n}(\mathcal{C}_{\pm})+\inf_{\delta\in[0,1]}\left\{\delta+\int_{\delta}^{1}\sqrt{\frac{\log N(\varepsilon,\mathcal{H},\|\cdot\|_{\infty})}{n}}\,\mathrm{d}\varepsilon\right\}. \tag*{(53)} UC n MC ( f ) ≲ R D , n ( C ± ) + δ ∈ [ 0 , 1 ] inf { δ + ∫ δ 1 n log N ( ε , H , ∥ ⋅ ∥ ∞ ) d ε } . (53)
Proof. We first reduce the fixed-predictor population–validation discrepancy to an empirical-process term. For a = η c ∈ C ± a=\eta c\in\mathcal{C}_{\pm} a = η c ∈ C ± and h ∈ H h\in\mathcal{H} h ∈ H , set g a , h a ( x , y ) = a ( x ) h ( f ( x ) ) ( y − f ( x ) ) g^a_{a,h}(x,y)=a(x)h(f(x))(y-f(x)) g a , h a ( x , y ) = a ( x ) h ( f ( x )) ( y − f ( x )) . Since ∣ a ∣ ≤ 1 |a|\leq1 ∣ a ∣ ≤ 1 , ∣ h ∣ ≤ 1 |h|\leq1 ∣ h ∣ ≤ 1 , and y − f ( x ) ∈ [ − 1 , 1 ] y-f(x)\in[-1,1] y − f ( x ) ∈ [ − 1 , 1 ] , every g a , h a g^a_{a,h} g a , h a is bounded by one. The supremum-difference inequality gives
∣ Φ M C ( f ; D ) − Φ M C , n ( f ; S ′ ) ∣ ≤ sup g ∈ G f ∣ P g − P n g ∣ , \left|\Phi_{\mathrm{MC}}(f;\mathcal{D})-\Phi_{\mathrm{MC},n}(f;S')\right|\leq\sup_{g\in\mathcal{G}_f}|Pg-P_ng|, ∣ Φ MC ( f ; D ) − Φ MC , n ( f ; S ′ ) ∣ ≤ g ∈ G f sup ∣ P g − P n g ∣ ,
where P = D P=\mathcal{D} P = D and P n P_n P n is the empirical law of S ′ S' S ′ . Standard symmetrization, as in Wainwright [37 ] , yields U C n M C ( f ) ≤ 2 R D , n ( G f ) \mathrm{UC}^{\mathrm{MC}}_n(f)\leq2\mathfrak{R}_{D,n}(\mathcal{G}_f) UC n MC ( f ) ≤ 2 R D , n ( G f ) , which proves (51) .
It remains to prove the separable product-class bound. Fix a sample S = ( ( X i , Y i ) ) i = 1 n S=((X_i,Y_i))_{i=1}^n S = (( X i , Y i ) ) i = 1 n , r i : = Y i − f ( X i ) r_i:=Y_i-f(X_i) r i := Y i − f ( X i ) , ∣ r i ∣ ≤ 1 |r_i|\leq1 ∣ r i ∣ ≤ 1 . Conditioning on this sample, write
R ^ S ( G f ) = E σ sup a ∈ C ± , h ∈ H 1 n ∑ i = 1 n σ i a ( X i ) h ( f ( X i ) ) r i , \widehat{\mathfrak{R}}_S(\mathcal{G}_f)=\mathbb{E}_{\sigma}\sup_{a\in\mathcal{C}_{\pm},\,h\in\mathcal{H}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i a(X_i)h(f(X_i))r_i, R S ( G f ) = E σ a ∈ C ± , h ∈ H sup n 1 i = 1 ∑ n σ i a ( X i ) h ( f ( X i )) r i ,
and let
R ^ X ( C ± ) = E σ sup a ∈ C ± 1 n ∑ i = 1 n σ i a ( X i ) \widehat{\mathfrak{R}}_X(\mathcal{C}_{\pm})=\mathbb{E}_{\sigma}\sup_{a\in\mathcal{C}_{\pm}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i a(X_i) R X ( C ± ) = E σ a ∈ C ± sup n 1 i = 1 ∑ n σ i a ( X i )
be the empirical Rademacher average of the subgroup class on the observed X X X -sample.
We approximate the score witness h h h by a dyadic chain in ∥ ⋅ ∥ ∞ \|\cdot\|_{\infty} ∥ ⋅ ∥ ∞ . If one of the covering numbers below is infinite, the right-hand side is infinite and there is nothing to prove. Thus assume finite covers at the relevant scales. Fix δ ∈ ( 0 , 1 ] \delta\in(0,1] δ ∈ ( 0 , 1 ] , set ε k : = 2 − k \varepsilon_k:=2^{-k} ε k := 2 − k , ε 0 : = 1 \varepsilon_0:=1 ε 0 := 1 , and choose m m m such that ε m ≤ δ < 2 ε m \varepsilon_m\leq\delta<2\varepsilon_m ε m ≤ δ < 2 ε m . For each k = 1 , … , m k=1,\ldots,m k = 1 , … , m , let H ~ k \widetilde{\mathcal{H}}_k H k be a minimal ε k \varepsilon_k ε k -net of H \mathcal{H} H in ∥ ⋅ ∥ ∞ \|\cdot\|_{\infty} ∥ ⋅ ∥ ∞ , and write N k : = ∣ H ~ k ∣ = N ( ε k , H , ∥ ⋅ ∥ ∞ ) N_k:=|\widetilde{\mathcal{H}}_k|=N(\varepsilon_k,\mathcal{H},\|\cdot\|_{\infty}) N k := ∣ H k ∣ = N ( ε k , H , ∥ ⋅ ∥ ∞ ) . For every h ∈ H h\in\mathcal{H} h ∈ H , choose one net point h k ( h ) ∈ H ~ k h_k(h)\in\widetilde{\mathcal{H}}_k h k ( h ) ∈ H k satisfying ∥ h − h k ( h ) ∥ ∞ ≤ ε k \|h-h_k(h)\|_{\infty}\leq\varepsilon_k ∥ h − h k ( h ) ∥ ∞ ≤ ε k , and set h 0 ( h ) ≡ 0 h_0(h)\equiv0 h 0 ( h ) ≡ 0 . Then
h = ∑ k = 1 m { h k ( h ) − h k − 1 ( h ) } + { h − h m ( h ) } . h=\sum_{k=1}^{m}\{h_k(h)-h_{k-1}(h)\}+\{h-h_m(h)\}. h = k = 1 ∑ m { h k ( h ) − h k − 1 ( h )} + { h − h m ( h )} .
For each scale k k k , define the finite set of actual increment functions D k : = { h k ( h ) − h k − 1 ( h ) : h ∈ H } \mathcal{D}_k := \{h_k(h)-h_{k-1}(h):h\in\mathcal{H}\} D k := { h k ( h ) − h k − 1 ( h ) : h ∈ H } . Each d ∈ D k d\in\mathcal{D}_k d ∈ D k is a function on the score space [ 0 , 1 ] [0,1] [ 0 , 1 ] , because both h k ( h ) h_k(h) h k ( h ) and h k − 1 ( h ) h_{k-1}(h) h k − 1 ( h ) are score-witness functions. Hence d ( f ( X i ) ) d(f(X_i)) d ( f ( X i )) is well-defined. The word “actual” means that we include only pairs of net points that arise from the same h ∈ H h\in\mathcal{H} h ∈ H under the chosen nearest-net assignments.
The cardinality of D k \mathcal{D}_k D k is controlled by the covering numbers. For k = 1 k=1 k = 1 , an increment is determined only by the fine net point h 1 ( h ) h_1(h) h 1 ( h ) , since h 0 ( h ) ≡ 0 h_0(h)\equiv0 h 0 ( h ) ≡ 0 , so ∣ D 1 ∣ ≤ N 1 |\mathcal{D}_1|\leq N_1 ∣ D 1 ∣ ≤ N 1 . For k ≥ 2 k\geq2 k ≥ 2 , an increment is determined by a pair of net points, h k ( h ) ∈ H ~ k h_k(h)\in\widetilde{\mathcal{H}}_k h k ( h ) ∈ H k , h k − 1 ( h ) ∈ H ~ k − 1 h_{k-1}(h)\in\widetilde{\mathcal{H}}_{k-1} h k − 1 ( h ) ∈ H k − 1 , and hence ∣ D k ∣ ≤ N k N k − 1 |\mathcal{D}_k|\leq N_kN_{k-1} ∣ D k ∣ ≤ N k N k − 1 . Since ε k ≤ ε k − 1 \varepsilon_k\leq\varepsilon_{k-1} ε k ≤ ε k − 1 , covering numbers are monotone and N k − 1 ≤ N k N_{k-1}\leq N_k N k − 1 ≤ N k . Therefore, with M k : = ∣ D k ∣ M_k:=|\mathcal{D}_k| M k := ∣ D k ∣ , log M k ≲ log N k = log N ( ε k , H , ∥ ⋅ ∥ ∞ ) \log M_k\lesssim\log N_k=\log N(\varepsilon_k,\mathcal{H},\|\cdot\|_\infty) log M k ≲ log N k = log N ( ε k , H , ∥ ⋅ ∥ ∞ ) for every k k k , up to universal numerical constants.
We now decompose the empirical Rademacher average into the residual term and the dyadic increment terms. For each h ∈ H h\in\mathcal{H} h ∈ H , write d k h : = h k ( h ) − h k − 1 ( h ) d_k^h:=h_k(h)-h_{k-1}(h) d k h := h k ( h ) − h k − 1 ( h ) , e m h : = h − h m ( h ) e_m^h:=h-h_m(h) e m h := h − h m ( h ) . Then, for each sample point,
h ( f ( X i ) ) r i = ∑ k = 1 m d k h ( f ( X i ) ) r i + e m h ( f ( X i ) ) r i . h(f(X_i))r_i=\sum_{k=1}^{m}d_k^h(f(X_i))r_i+e_m^h(f(X_i))r_i. h ( f ( X i )) r i = k = 1 ∑ m d k h ( f ( X i )) r i + e m h ( f ( X i )) r i .
Substituting this decomposition into R ^ S ( G f ) \widehat{\mathfrak{R}}_S(\mathcal{G}_f) R S ( G f ) and using subadditivity of the supremum gives
R ^ S ( G f ) ≤ E σ sup a ∈ C ± , h ∈ H ∣ 1 n ∑ i = 1 n σ i a ( X i ) e m h ( f ( X i ) ) r i ∣ + ∑ k = 1 m E σ sup a ∈ C ± , h ∈ H 1 n ∑ i = 1 n σ i a ( X i ) d k h ( f ( X i ) ) r i . \widehat{\mathfrak{R}}_S(\mathcal{G}_f)\leq\mathbb{E}_{\sigma}\sup_{a\in\mathcal{C}_{\pm},\,h\in\mathcal{H}}\left|\frac{1}{n}\sum_{i=1}^{n}\sigma_i a(X_i)e_m^h(f(X_i))r_i\right|+\sum_{k=1}^{m}\mathbb{E}_{\sigma}\sup_{a\in\mathcal{C}_{\pm},\,h\in\mathcal{H}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i a(X_i)d_k^h(f(X_i))r_i. R S ( G f ) ≤ E σ a ∈ C ± , h ∈ H sup n 1 i = 1 ∑ n σ i a ( X i ) e m h ( f ( X i )) r i + k = 1 ∑ m E σ a ∈ C ± , h ∈ H sup n 1 i = 1 ∑ n σ i a ( X i ) d k h ( f ( X i )) r i .
Indeed, this is the elementary inequality
sup a , h { R a , h + ∑ k = 1 m I k , a , h } ≤ sup a , h ∣ R a , h ∣ + ∑ k = 1 m sup a , h I k , a , h . \sup_{a,h}\left\{R_{a,h}+\sum_{k=1}^{m}I_{k,a,h}\right\}\leq\sup_{a,h}|R_{a,h}|+\sum_{k=1}^{m}\sup_{a,h}I_{k,a,h}. a , h sup { R a , h + k = 1 ∑ m I k , a , h } ≤ a , h sup ∣ R a , h ∣ + k = 1 ∑ m a , h sup I k , a , h .
The residual satisfies ∥ e m h ∥ ∞ ≤ ε m ≤ δ \|e_m^h\|_\infty\leq\varepsilon_m\leq\delta ∥ e m h ∥ ∞ ≤ ε m ≤ δ , and therefore
E σ sup a , h ∣ 1 n ∑ i = 1 n σ i a ( X i ) e m h ( f ( X i ) ) r i ∣ ≤ δ , \mathbb{E}_{\sigma}\sup_{a,h}\left|\frac{1}{n}\sum_{i=1}^{n}\sigma_i a(X_i)e_m^h(f(X_i))r_i\right|\leq\delta, E σ a , h sup n 1 i = 1 ∑ n σ i a ( X i ) e m h ( f ( X i )) r i ≤ δ ,
because ∣ a ( X i ) ∣ ≤ 1 |a(X_i)|\leq1 ∣ a ( X i ) ∣ ≤ 1 and ∣ r i ∣ ≤ 1 |r_i|\leq1 ∣ r i ∣ ≤ 1 .
For the k k k -th increment, since d k h ∈ D k d_k^h\in\mathcal{D}_k d k h ∈ D k ,
sup a , h 1 n ∑ i = 1 n σ i a ( X i ) d k h ( f ( X i ) ) r i ≤ max d ∈ D k sup a ∈ C ± 1 n ∑ i = 1 n σ i a ( X i ) d ( f ( X i ) ) r i . \sup_{a,h}\frac{1}{n}\sum_{i=1}^{n}\sigma_i a(X_i)d_k^h(f(X_i))r_i\leq\max_{d\in\mathcal{D}_k}\sup_{a\in\mathcal{C}_{\pm}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i a(X_i)d(f(X_i))r_i. a , h sup n 1 i = 1 ∑ n σ i a ( X i ) d k h ( f ( X i )) r i ≤ d ∈ D k max a ∈ C ± sup n 1 i = 1 ∑ n σ i a ( X i ) d ( f ( X i )) r i .
Define the sample-dependent multiplier vector b i d : = d ( f ( X i ) ) r i b_i^d:=d(f(X_i))r_i b i d := d ( f ( X i )) r i , d ∈ D k d\in\mathcal{D}_k d ∈ D k , i ∈ [ n ] i\in[n] i ∈ [ n ] , and set
T k : = E σ max d ∈ D k sup a ∈ C ± 1 n ∑ i = 1 n σ i a ( X i ) b i d . T_k:=\mathbb{E}_{\sigma}\max_{d\in\mathcal{D}_k}\sup_{a\in\mathcal{C}_{\pm}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i a(X_i)b_i^d. T k := E σ d ∈ D k max a ∈ C ± sup n 1 i = 1 ∑ n σ i a ( X i ) b i d .
Combining the preceding displays gives R ^ S ( G f ) ≤ δ + ∑ k = 1 m T k \widehat{\mathfrak{R}}_S(\mathcal{G}_f)\leq\delta+\sum_{k=1}^{m}T_k R S ( G f ) ≤ δ + ∑ k = 1 m T k . Thus it remains to bound each T k T_k T k .
The size of the multiplier vector is controlled by the scale of the corresponding witness increment. If d = d k h = h k ( h ) − h k − 1 ( h ) d=d_k^h=h_k(h)-h_{k-1}(h) d = d k h = h k ( h ) − h k − 1 ( h ) , then for k ≥ 2 k\geq2 k ≥ 2 ,
∥ d ∥ ∞ ≤ ∥ h k ( h ) − h ∥ ∞ + ∥ h − h k − 1 ( h ) ∥ ∞ ≤ ε k + ε k − 1 ≤ 2 ε k − 1 . \|d\|_\infty\leq\|h_k(h)-h\|_\infty+\|h-h_{k-1}(h)\|_\infty\leq\varepsilon_k+\varepsilon_{k-1}\leq2\varepsilon_{k-1}. ∥ d ∥ ∞ ≤ ∥ h k ( h ) − h ∥ ∞ + ∥ h − h k − 1 ( h ) ∥ ∞ ≤ ε k + ε k − 1 ≤ 2 ε k − 1 .
For k = 1 k=1 k = 1 , since h 0 ( h ) ≡ 0 h_0(h)\equiv0 h 0 ( h ) ≡ 0 and ∣ h 1 ( h ) ∣ ≤ 1 = ε 0 |h_1(h)|\leq1=\varepsilon_0 ∣ h 1 ( h ) ∣ ≤ 1 = ε 0 , the same bound holds up to a universal constant. Hence, with B k : = sup d ∈ D k max i ∈ [ n ] ∣ b i d ∣ B_k:=\sup_{d\in\mathcal{D}_k}\max_{i\in[n]}|b_i^d| B k := sup d ∈ D k max i ∈ [ n ] ∣ b i d ∣ , we have B k ≲ ε k − 1 B_k\lesssim\varepsilon_{k-1} B k ≲ ε k − 1 .
Fix k k k and first consider a single increment d ∈ D k d\in\mathcal{D}_k d ∈ D k . Define Z d ( σ ) : = sup a ∈ C ± 1 n ∑ i = 1 n σ i a ( X i ) b i d Z_d(\sigma):=\sup_{a\in\mathcal{C}_{\pm}}\frac{1}{n}\sum_{i=1}^{n}\sigma_i a(X_i)b_i^d Z d ( σ ) := sup a ∈ C ± n 1 ∑ i = 1 n σ i a ( X i ) b i d . Because C ± \mathcal{C}_{\pm} C ± is symmetric, this supremum is the same as the corresponding absolute supremum. For fixed d d d , the coordinatewise maps ϕ i ( t ) : = b i d t \phi_i(t) := b_i^d t ϕ i ( t ) := b i d t satisfy ϕ i ( 0 ) = 0 \phi_i(0) = 0 ϕ i ( 0 ) = 0 and ∣ ϕ i ( t ) − ϕ i ( t ′ ) ∣ ≤ ∣ b i d ∣ ∣ t − t ′ ∣ ≤ B k ∣ t − t ′ ∣ |\phi_i(t) - \phi_i(t')| \le|b_i^d| |t - t'| \le B_k|t - t'| ∣ ϕ i ( t ) − ϕ i ( t ′ ) ∣ ≤ ∣ b i d ∣∣ t − t ′ ∣ ≤ B k ∣ t − t ′ ∣ . The Rademacher contraction inequality gives
E σ Z d ( σ ) ≤ E σ sup a ∈ C ± ∣ 1 n ∑ i = 1 n σ i b i d a ( X i ) ∣ ≤ C c t r B k E σ sup a ∈ C ± ∣ 1 n ∑ i = 1 n σ i a ( X i ) ∣ = C c t r B k R ^ X ( C ± ) , \begin{aligned}
\mathbb{E}_{\sigma} Z_d(\sigma) \le\mathbb{E}_{\sigma}\sup_{a\in C_{\pm}}\left|\frac{1}{n}\sum_{i=1}^{n}\sigma_i b_i^d a(X_i)\right| \\
&\le C_{\mathrm{ctr}}B_k\mathbb{E}_{\sigma}\sup_{a\in C_{\pm}}\left|\frac{1}{n}\sum_{i=1}^{n}\sigma_i a(X_i)\right| \\
&= C_{\mathrm{ctr}}B_k\widehat{\mathfrak R}_X(C_{\pm}),
\end{aligned} E σ Z d ( σ ) ≤ E σ a ∈ C ± sup n 1 i = 1 ∑ n σ i b i d a ( X i ) ≤ C ctr B k E σ a ∈ C ± sup n 1 i = 1 ∑ n σ i a ( X i ) = C ctr B k R X ( C ± ) ,
where C c t r C_{\mathrm{ctr}} C ctr is a universal contraction constant; the last equality again uses the symmetry of C ± C_{\pm} C ± .
We next take the maximum over the finite set D k D_k D k . For fixed d d d , changing one Rademacher sign σ j \sigma_j σ j changes Z d ( σ ) Z_d(\sigma) Z d ( σ ) by at most 2 ∣ b j d ∣ n ≤ 2 B k n \frac{2|b_j^d|}{n}\le\frac{2B_k}{n} n 2∣ b j d ∣ ≤ n 2 B k . Hence the bounded-differences inequality gives, for every λ ∈ R \lambda\in\mathbb{R} λ ∈ R ,
E σ exp { λ ( Z d ( σ ) − E σ Z d ( σ ) ) } ≤ exp { λ 2 B k 2 2 n } . \mathbb{E}_{\sigma}\exp\left\{\lambda\left(Z_d(\sigma)-\mathbb{E}_{\sigma}Z_d(\sigma)\right)\right\}\le\exp\left\{\frac{\lambda^2B_k^2}{2n}\right\}. E σ exp { λ ( Z d ( σ ) − E σ Z d ( σ ) ) } ≤ exp { 2 n λ 2 B k 2 } .
Therefore, for every λ > 0 \lambda>0 λ > 0 ,
E σ max d ∈ D k { Z d ( σ ) − E σ Z d ( σ ) } ≤ 1 λ log ∑ d ∈ D k E σ exp { λ ( Z d ( σ ) − E σ Z d ( σ ) ) } ≤ log M k λ + λ B k 2 2 n . \begin{aligned}
\mathbb{E}_{\sigma}\max_{d\in D_k}\left\{Z_d(\sigma)-\mathbb{E}_{\sigma}Z_d(\sigma)\right\}\le\frac{1}{\lambda}\log\sum_{d\in D_k}\mathbb{E}_{\sigma}\exp\left\{\lambda\left(Z_d(\sigma)-\mathbb{E}_{\sigma}Z_d(\sigma)\right)\right\} \\
&\le\frac{\log M_k}{\lambda}+\frac{\lambda B_k^2}{2n}.
\end{aligned} E σ d ∈ D k max { Z d ( σ ) − E σ Z d ( σ ) } ≤ λ 1 log d ∈ D k ∑ E σ exp { λ ( Z d ( σ ) − E σ Z d ( σ ) ) } ≤ λ log M k + 2 n λ B k 2 .
Optimizing over λ > 0 \lambda>0 λ > 0 yields
E σ max d ∈ D k { Z d ( σ ) − E σ Z d ( σ ) } ≤ B k 2 log M k n , \mathbb{E}_{\sigma}\max_{d\in D_k}\left\{Z_d(\sigma)-\mathbb{E}_{\sigma}Z_d(\sigma)\right\}\le B_k\sqrt{\frac{2\log M_k}{n}}, E σ d ∈ D k max { Z d ( σ ) − E σ Z d ( σ ) } ≤ B k n 2 log M k ,
with the convention that this term is zero when M k = 1 M_k=1 M k = 1 . Consequently,
T k = E σ max d ∈ D k Z d ( σ ) ≤ max d ∈ D k E σ Z d ( σ ) + E σ max d ∈ D k { Z d ( σ ) − E σ Z d ( σ ) } ≤ C c t r B k R ^ X ( C ± ) + B k 2 log M k n . \begin{aligned}
T_k=\mathbb{E}_{\sigma}\max_{d\in D_k}Z_d(\sigma) \\
&\le\max_{d\in D_k}\mathbb{E}_{\sigma}Z_d(\sigma)+\mathbb{E}_{\sigma}\max_{d\in D_k}\left\{Z_d(\sigma)-\mathbb{E}_{\sigma}Z_d(\sigma)\right\} \\
&\le C_{\mathrm{ctr}}B_k\widehat{\mathfrak R}_X(C_{\pm})+B_k\sqrt{\frac{2\log M_k}{n}}.
\end{aligned} T k = E σ d ∈ D k max Z d ( σ ) ≤ d ∈ D k max E σ Z d ( σ ) + E σ d ∈ D k max { Z d ( σ ) − E σ Z d ( σ ) } ≤ C ctr B k R X ( C ± ) + B k n 2 log M k .
Using B k ≲ ϵ k − 1 B_k\lesssim\epsilon_{k-1} B k ≲ ϵ k − 1 and the cardinality bound on M k M_k M k ,
T k ≲ ϵ k − 1 R ^ X ( C ± ) + ϵ k − 1 log N ( ϵ k , H , ∥ ⋅ ∥ ∞ ) n . T_k\lesssim\epsilon_{k-1}\widehat{\mathfrak R}_X(C_{\pm})+\epsilon_{k-1}\sqrt{\frac{\log N(\epsilon_k,\mathcal{H},\|\cdot\|_{\infty})}{n}}. T k ≲ ϵ k − 1 R X ( C ± ) + ϵ k − 1 n log N ( ϵ k , H , ∥ ⋅ ∥ ∞ ) .
Summing over k = 1 , … , m k=1,\ldots,m k = 1 , … , m gives
R ^ S ( G f ) ≲ δ + ( ∑ k = 1 m ϵ k − 1 ) R ^ X ( C ± ) + ∑ k = 1 m ϵ k − 1 log N ( ϵ k , H , ∥ ⋅ ∥ ∞ ) n . \widehat{\mathfrak R}_S(\mathcal{G}_f)\lesssim\delta+\left(\sum_{k=1}^{m}\epsilon_{k-1}\right)\widehat{\mathfrak R}_X(C_{\pm})+\sum_{k=1}^{m}\epsilon_{k-1}\sqrt{\frac{\log N(\epsilon_k,\mathcal{H},\|\cdot\|_{\infty})}{n}}. R S ( G f ) ≲ δ + ( k = 1 ∑ m ϵ k − 1 ) R X ( C ± ) + k = 1 ∑ m ϵ k − 1 n log N ( ϵ k , H , ∥ ⋅ ∥ ∞ ) .
The first sum is bounded by a universal constant, because ∑ k = 1 m ϵ k − 1 = ∑ k = 1 m 2 − ( k − 1 ) ≤ 2 \sum_{k=1}^{m}\epsilon_{k-1}=\sum_{k=1}^{m}2^{-(k-1)}\le2 ∑ k = 1 m ϵ k − 1 = ∑ k = 1 m 2 − ( k − 1 ) ≤ 2 .
It remains to compare the dyadic entropy sum with the Dudley integral. Set F ( ϵ ) : = log N ( ϵ , H , ∥ ⋅ ∥ ∞ ) n F(\epsilon):=\sqrt{\frac{\log N(\epsilon,\mathcal{H},\|\cdot\|_{\infty})}{n}} F ( ϵ ) := n l o g N ( ϵ , H , ∥ ⋅ ∥ ∞ ) . Because covering numbers are nonincreasing in the radius, F F F is nonincreasing as a function of ϵ \epsilon ϵ . For each k k k , ϵ k − 1 F ( ϵ k ) = 4 ( ϵ k − ϵ k + 1 ) F ( ϵ k ) \epsilon_{k-1}F(\epsilon_k)=4(\epsilon_k-\epsilon_{k+1})F(\epsilon_k) ϵ k − 1 F ( ϵ k ) = 4 ( ϵ k − ϵ k + 1 ) F ( ϵ k ) . Moreover, for ϵ ∈ [ ϵ k + 1 , ϵ k ] \epsilon\in[\epsilon_{k+1},\epsilon_k] ϵ ∈ [ ϵ k + 1 , ϵ k ] , we have F ( ϵ ) ≥ F ( ϵ k ) F(\epsilon)\ge F(\epsilon_k) F ( ϵ ) ≥ F ( ϵ k ) . Hence ϵ k − 1 F ( ϵ k ) ≤ 4 ∫ ϵ k + 1 ϵ k F ( ϵ ) d ϵ \epsilon_{k-1}F(\epsilon_k)\le4\int_{\epsilon_{k+1}}^{\epsilon_k}F(\epsilon)\,d\epsilon ϵ k − 1 F ( ϵ k ) ≤ 4 ∫ ϵ k + 1 ϵ k F ( ϵ ) d ϵ . Summing over k = 1 , … , m k=1,\ldots,m k = 1 , … , m gives ∑ k = 1 m ϵ k − 1 F ( ϵ k ) ≲ ∫ ϵ m + 1 1 F ( ϵ ) d ϵ \sum_{k=1}^{m}\epsilon_{k-1}F(\epsilon_k)\lesssim\int_{\epsilon_{m+1}}^{1}F(\epsilon)\,d\epsilon ∑ k = 1 m ϵ k − 1 F ( ϵ k ) ≲ ∫ ϵ m + 1 1 F ( ϵ ) d ϵ . By the choice of m m m , ϵ m ≤ δ < 2 ϵ m \epsilon_m\le\delta<2\epsilon_m ϵ m ≤ δ < 2 ϵ m , and hence ϵ m + 1 = ϵ m / 2 ≥ δ / 4 \epsilon_{m+1}=\epsilon_m/2\ge\delta/4 ϵ m + 1 = ϵ m /2 ≥ δ /4 . Therefore ∫ ϵ m + 1 1 F ( ϵ ) d ϵ ≤ ∫ δ / 4 1 F ( ϵ ) d ϵ \int_{\epsilon_{m+1}}^{1}F(\epsilon)\,d\epsilon\le\int_{\delta/4}^{1}F(\epsilon)\,d\epsilon ∫ ϵ m + 1 1 F ( ϵ ) d ϵ ≤ ∫ δ /4 1 F ( ϵ ) d ϵ . Combining the preceding displays,
R ^ S ( G f ) ≲ R ^ X ( C ± ) + δ + ∫ δ / 4 1 log N ( ϵ , H , ∥ ⋅ ∥ ∞ ) n d ϵ . \widehat{\mathfrak R}_S(\mathcal{G}_f)\lesssim\widehat{\mathfrak R}_X(C_{\pm})+\delta+\int_{\delta/4}^{1}\sqrt{\frac{\log N(\epsilon,\mathcal{H},\|\cdot\|_{\infty})}{n}}\,d\epsilon. R S ( G f ) ≲ R X ( C ± ) + δ + ∫ δ /4 1 n log N ( ϵ , H , ∥ ⋅ ∥ ∞ ) d ϵ .
Renaming the lower limit and absorbing universal constants into ≲ \lesssim ≲ , we obtain
R ^ S ( G f ) ≲ R ^ X ( C ± ) + inf δ ∈ [ 0 , 1 ] { δ + ∫ δ 1 log N ( ε , H , ∥ ⋅ ∥ ∞ ) n d ε } . \widehat{\mathfrak{R}}_{\mathcal{S}}(\mathcal{G}_f) \lesssim\widehat{\mathfrak{R}}_{\mathcal{X}}(\mathcal{C}_{\pm})+\inf_{\delta\in[0,1]}\left\{\delta+\int_{\delta}^{1}\sqrt{\frac{\log N(\varepsilon,\mathcal{H},\lVert\cdot\rVert_{\infty})}{n}}\,d\varepsilon\right\}. R S ( G f ) ≲ R X ( C ± ) + δ ∈ [ 0 , 1 ] inf { δ + ∫ δ 1 n log N ( ε , H , ∥ ⋅ ∥ ∞ ) d ε } .
For large truncation levels, the same display follows from the trivial bound R ^ S ( G f ) ≤ 1 \widehat{\mathfrak{R}}_{\mathcal{S}}(\mathcal{G}_f)\leq1 R S ( G f ) ≤ 1 after adjusting the universal constant, so the infimum may be taken over the full interval [ 0 , 1 ] [0,1] [ 0 , 1 ] .
Finally, taking expectation over S ∼ D n \mathcal{S}\sim\mathcal{D}^{n} S ∼ D n proves (52) . Combining this with (51) proves (53) . □ \square □
Convergence of smooth CE
Lemma 8 (Lipschitz-witness refinement for finite subgroup classes). Assume H = Lip 1 ( [ 0 , 1 ] , [ − 1 , 1 ] ) \mathcal{H}=\operatorname{Lip}_{1}([0,1],[-1,1]) H = Lip 1 ([ 0 , 1 ] , [ − 1 , 1 ]) and C \mathcal{C} C is finite. For every fixed predictor f : X → [ 0 , 1 ] f:\mathcal{X}\to[0,1] f : X → [ 0 , 1 ] ,
UC n M C ( f ) ≲ 1 n + log ( 2 ∣ C ∣ ) n . (54) \operatorname{UC}_{n}^{\mathrm{MC}}(f)\lesssim\frac{1}{\sqrt{n}}+\sqrt{\frac{\log(2|\mathcal{C}|)}{n}}. \tag*{(54)} UC n MC ( f ) ≲ n 1 + n log ( 2∣ C ∣ ) . (54) The hidden constant is universal.
Proof. We give a direct entropy proof, avoiding any dependence on a finite discretization of the Lipschitz witness class. Lemma 7 gives
UC n M C ( f ) ≲ R D , n ( C ± ) + inf δ ≥ 0 { δ + ∫ δ 1 log N ( ε , Lip 1 ( [ 0 , 1 ] , [ − 1 , 1 ] ) , ∥ ⋅ ∥ ∞ ) n d ε } . \operatorname{UC}_{n}^{\mathrm{MC}}(f)\lesssim\mathfrak{R}_{D,n}(\mathcal{C}_{\pm})+\inf_{\delta\geq0}\left\{\delta+\int_{\delta}^{1}\sqrt{\frac{\log N(\varepsilon,\operatorname{Lip}_{1}([0,1],[-1,1]),\lVert\cdot\rVert_{\infty})}{n}}\,d\varepsilon\right\}. UC n MC ( f ) ≲ R D , n ( C ± ) + δ ≥ 0 inf { δ + ∫ δ 1 n log N ( ε , Lip 1 ([ 0 , 1 ] , [ − 1 , 1 ]) , ∥ ⋅ ∥ ∞ ) d ε } .
Because ∣ C ± ∣ ≤ 2 ∣ C ∣ |\mathcal{C}_{\pm}|\leq2|\mathcal{C}| ∣ C ± ∣ ≤ 2∣ C ∣ and all functions in C ± \mathcal{C}_{\pm} C ± are bounded by one, Massart’s finite-class lemma gives
R D , n ( C ± ) ≲ log ( 2 ∣ C ∣ ) n . \mathfrak{R}_{D,n}(\mathcal{C}_{\pm})\lesssim\sqrt{\frac{\log(2|\mathcal{C}|)}{n}}. R D , n ( C ± ) ≲ n log ( 2∣ C ∣ ) .
It remains to evaluate the entropy integral. A standard grid construction gives
log N ( ε , Lip 1 ( [ 0 , 1 ] , [ − 1 , 1 ] ) , ∥ ⋅ ∥ ∞ ) ≲ 1 ε log 1 ε , 0 < ε ≤ 1. \log N(\varepsilon,\operatorname{Lip}_{1}([0,1],[-1,1]),\lVert\cdot\rVert_{\infty})\lesssim\frac{1}{\varepsilon}\log\frac{1}{\varepsilon},\qquad0<\varepsilon\leq1. log N ( ε , Lip 1 ([ 0 , 1 ] , [ − 1 , 1 ]) , ∥ ⋅ ∥ ∞ ) ≲ ε 1 log ε 1 , 0 < ε ≤ 1.
Indeed, take a mesh of spacing proportional to ε \varepsilon ε , quantize the function values on the mesh at resolution proportional to ε \varepsilon ε , and extend between mesh points using the Lipschitz constraint. Therefore
∫ 0 1 log N ( ε , Lip 1 ( [ 0 , 1 ] , [ − 1 , 1 ] ) , ∥ ⋅ ∥ ∞ ) n d ε ≲ 1 n ∫ 0 1 log ( 1 / ε ) ε d ε ≲ 1 n . \int_{0}^{1}\sqrt{\frac{\log N(\varepsilon,\operatorname{Lip}_{1}([0,1],[-1,1]),\lVert\cdot\rVert_{\infty})}{n}}\,d\varepsilon\lesssim\frac{1}{\sqrt{n}}\int_{0}^{1}\sqrt{\frac{\log(1/\varepsilon)}{\varepsilon}}\,d\varepsilon\lesssim\frac{1}{\sqrt{n}}. ∫ 0 1 n log N ( ε , Lip 1 ([ 0 , 1 ] , [ − 1 , 1 ]) , ∥ ⋅ ∥ ∞ ) d ε ≲ n 1 ∫ 0 1 ε log ( 1/ ε ) d ε ≲ n 1 .
Substituting these two estimates into Lemma 7 proves (54) . □ \square □
This order matches the results in Futami and Nitanda [12 ] .
Proof of Corollary 7: MC population–training bound. Conditioning on the learned predictor and applying Lemma 7 gives
E ∣ Φ M C ( f W ; D ) − Φ M C , n ( f W ; S ′ ) ∣ ≤ E UC n M C ( f W ) . \mathbb{E}\left|\Phi_{\mathrm{MC}}(f_W;\mathcal{D})-\Phi_{\mathrm{MC},n}(f_W;\mathcal{S}')\right|\leq\mathbb{E}\operatorname{UC}_{n}^{\mathrm{MC}}(f_W). E ∣ Φ MC ( f W ; D ) − Φ MC , n ( f W ; S ′ ) ∣ ≤ E UC n MC ( f W ) .
Now decompose
Φ M C ( f W ; D ) − Φ M C , n ( f W ; S ) = { Φ M C ( f W ; D ) − Φ M C , n ( f W ; S ′ ) } + { Φ M C , n ( f W ; S ′ ) − Φ M C , n ( f W ; S ) } . \Phi_{\mathrm{MC}}(f_W;\mathcal{D})-\Phi_{\mathrm{MC},n}(f_W;\mathcal{S})=\left\{\Phi_{\mathrm{MC}}(f_W;\mathcal{D})-\Phi_{\mathrm{MC},n}(f_W;\mathcal{S}')\right\}+\left\{\Phi_{\mathrm{MC},n}(f_W;\mathcal{S}')-\Phi_{\mathrm{MC},n}(f_W;\mathcal{S})\right\}. Φ MC ( f W ; D ) − Φ MC , n ( f W ; S ) = { Φ MC ( f W ; D ) − Φ MC , n ( f W ; S ′ ) } + { Φ MC , n ( f W ; S ′ ) − Φ MC , n ( f W ; S ) } .
Taking absolute values and expectations, the first term is bounded by E UC n M C ( f W ) \mathbb{E}\operatorname{UC}_{n}^{\mathrm{MC}}(f_W) E UC n MC ( f W ) . The second term is bounded by the bound on E ∣ Δ Φ ∣ \mathbb{E}|\Delta\Phi| E ∣ΔΦ∣ in (8) with L Γ , n ≤ 2 / n L_{\Gamma,n}\leq2/\sqrt{n} L Γ , n ≤ 2/ n . This gives
E ∣ Φ M C ( f W ; D ) − Φ M C , n ( f W ; S ) ∣ ≤ E UC n M C ( f W ) + 2 n 2 { I ( f W ( X ~ ) ; U ∣ Z ~ ) + log 2 } , \mathbb{E}\left|\Phi_{\mathrm{MC}}(f_W;\mathcal{D})-\Phi_{\mathrm{MC},n}(f_W;\mathcal{S})\right|\leq\mathbb{E}\operatorname{UC}_{n}^{\mathrm{MC}}(f_W)+\frac{2}{\sqrt{n}}\sqrt{2\{I(f_W(\widetilde{X});U\mid\widetilde{Z})+\log2\}}, E ∣ Φ MC ( f W ; D ) − Φ MC , n ( f W ; S ) ∣ ≤ E UC n MC ( f W ) + n 2 2 { I ( f W ( X ) ; U ∣ Z ) + log 2 } ,
which is (19) .
Finally, we prove the one-sided comparison. Since MC is a supremum over witnesses with unbiased empirical witness averages, Jensen’s inequality gives
Φ M C ( f ; D ) ≤ E [ Φ M C , n ( f ; S ′ ) ∣ f ] \Phi_{\mathrm{MC}}(f;\mathcal{D})\leq\mathbb{E}[\Phi_{\mathrm{MC},n}(f;\mathcal{S}')\mid f] Φ MC ( f ; D ) ≤ E [ Φ MC , n ( f ; S ′ ) ∣ f ]
for every fixed f f f . Applying this with f = f W f=f_W f = f W and subtracting E Φ M C , n ( f W ; S ) \mathbb{E}\Phi_{\mathrm{MC},n}(f_W;S) E Φ MC , n ( f W ; S ) , we obtain
E Φ M C ( f W ; D ) − E Φ M C , n ( f W ; S ) ≤ E [ Φ M C , n ( f W ; S ′ ) − Φ M C , n ( f W ; S ) ] . (55) \mathbb{E}\Phi_{\mathrm{MC}}(f_W;D)-\mathbb{E}\Phi_{\mathrm{MC},n}(f_W;S)\leq\mathbb{E}\left[\Phi_{\mathrm{MC},n}(f_W;S')-\Phi_{\mathrm{MC},n}(f_W;S)\right]. \tag*{(55)} E Φ MC ( f W ; D ) − E Φ MC , n ( f W ; S ) ≤ E [ Φ MC , n ( f W ; S ′ ) − Φ MC , n ( f W ; S ) ] . (55)
Let D t M C D_t^{\mathrm{MC}} D t MC denote the MC increment and set I l o c , t M C : = I ( D t M C ; U t ∣ Z ~ ) I_{\mathrm{loc},t}^{\mathrm{MC}}:=I(D_t^{\mathrm{MC}};U_t\mid\widetilde{Z}) I loc , t MC := I ( D t MC ; U t ∣ Z ) . Since Corollary 1 gives ∣ D t M C ∣ ≤ 2 / n \lvert D_t^{\mathrm{MC}}\rvert\leq2/n ∣ D t MC ∣ ≤ 2/ n , Theorem 2 yields
E [ Φ M C , n ( f W ; S ′ ) − Φ M C , n ( f W ; S ) ] ≤ 2 n min { 2 n I f C M I , ∑ t = 1 n 2 I l o c , t M C } . (56) \mathbb{E}\left[\Phi_{\mathrm{MC},n}(f_W;S')-\Phi_{\mathrm{MC},n}(f_W;S)\right]\leq\frac{2}{n}\min\left\{\sqrt{2nI_{\mathrm{fCMI}}},\sum_{t=1}^{n}\sqrt{2I_{\mathrm{loc},t}^{\mathrm{MC}}}\right\}. \tag*{(56)} E [ Φ MC , n ( f W ; S ′ ) − Φ MC , n ( f W ; S ) ] ≤ n 2 min { 2 n I fCMI , t = 1 ∑ n 2 I loc , t MC } . (56)
Combining (55) and (56) proves (20) .
One-sided comparisons for common-feasible-set supremum objectives
The one-sided argument used for multicalibration is a general Jensen argument. It is not applied to the primal infimum form of CVaR; for CVaR and CR DRO we use dual common-feasible-set forms or the absolute plug-in comparisons above. Let
Φ ( f ; D ) = sup π ∈ Π E Z ∼ D g f , π ( Z ) , Φ n ( f ; S ′ ) = sup π ∈ Π 1 n ∑ i = 1 n g f , π ( Z i ′ ) , \Phi(f;D)=\sup_{\pi\in\Pi}\mathbb{E}_{Z\sim D}g_{f,\pi}(Z),\qquad\Phi_n(f;S')=\sup_{\pi\in\Pi}\frac{1}{n}\sum_{i=1}^{n}g_{f,\pi}(Z'_i), Φ ( f ; D ) = π ∈ Π sup E Z ∼ D g f , π ( Z ) , Φ n ( f ; S ′ ) = π ∈ Π sup n 1 i = 1 ∑ n g f , π ( Z i ′ ) ,
where S ′ ∼ D n S'\sim D^n S ′ ∼ D n . Write P n ′ g f , π : = n − 1 ∑ i = 1 n g f , π ( Z i ′ ) P'_{n}g_{f,\pi}:=n^{-1}\sum_{i=1}^{n}g_{f,\pi}(Z'_i) P n ′ g f , π := n − 1 ∑ i = 1 n g f , π ( Z i ′ ) . Then
Φ ( f ; D ) = sup π ∈ Π E S ′ ∼ D n P n ′ g f , π ≤ E S ′ ∼ D n sup π ∈ Π P n ′ g f , π = E S ′ ∼ D n Φ n ( f ; S ′ ) . \Phi(f;D)=\sup_{\pi\in\Pi}\mathbb{E}_{S'\sim D^n}P'_ng_{f,\pi}\leq\mathbb{E}_{S'\sim D^n}\sup_{\pi\in\Pi}P'_ng_{f,\pi}=\mathbb{E}_{S'\sim D^n}\Phi_n(f;S'). Φ ( f ; D ) = π ∈ Π sup E S ′ ∼ D n P n ′ g f , π ≤ E S ′ ∼ D n π ∈ Π sup P n ′ g f , π = E S ′ ∼ D n Φ n ( f ; S ′ ) .
Consequently, for any learned predictor f W f_W f W ,
E Φ ( f W ; D ) − E Φ n ( f W ; S ) ≤ E { Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) } . (57) \mathbb{E}\Phi(f_W;D)-\mathbb{E}\Phi_n(f_W;S)\leq\mathbb{E}\left\{\Phi_n(f_W;S')-\Phi_n(f_W;S)\right\}. \tag*{(57)} E Φ ( f W ; D ) − E Φ n ( f W ; S ) ≤ E { Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) } . (57)
Combining (57) with either the global bound in Theorem 1 or the single-index refinement yields the corresponding signed population–training upper bounds. This applies directly to multicalibration and to any dual CVaR/DRO formulation written with a common feasible supremum class. This Jensen comparison is not applied to the primal infimum form of CVaR; for the main CVaR statement we use the absolute plug-in comparison in Corollary 8. When the finite-sample DRO constraint itself is empirical, the absolute plug-in comparisons in Corollaries 8 and 9 are the statements used in the main text.
Growth-function control of global fCMI
The VC-rate statements in the main text use the following standard growth-function observation.
Lemma 9 (VC growth-function bound for global fCMI). Assume V = { 0 , 1 } V=\{0,1\} V = { 0 , 1 } and that the learning algorithm outputs predictors in a binary class F ⊆ { 0 , 1 } X \mathcal{F}\subseteq\{0,1\}^{\mathcal{X}} F ⊆ { 0 , 1 } X with VC dimension d < ∞ d<\infty d < ∞ . Let d n = d ∧ 2 n d_n=d\wedge2n d n = d ∧ 2 n . Then
I ( f W ( X ~ ) ; U ∣ Z ~ ) ≤ H ( f W ( X ~ ) ∣ Z ~ ) ≤ log Π F ( 2 n ) ≤ d n log ( 2 e n d n ) , I(f_W(\widetilde{X});U\mid\widetilde{Z})\leq H(f_W(\widetilde{X})\mid\widetilde{Z})\leq\log\Pi_{\mathcal{F}}(2n)\leq d_n\log\left(\frac{2en}{d_n}\right), I ( f W ( X ) ; U ∣ Z ) ≤ H ( f W ( X ) ∣ Z ) ≤ log Π F ( 2 n ) ≤ d n log ( d n 2 e n ) ,
where Π F ( m ) \Pi_{\mathcal{F}}(m) Π F ( m ) is the growth function of F \mathcal{F} F . Consequently, if d ≤ 2 n d\leq2n d ≤ 2 n , then I ( f W ( X ~ ) ; U ∣ Z ~ ) ≲ d log ( e n / d ) I(f_W(\widetilde{X});U\mid\widetilde{Z})\lesssim d\log(en/d) I ( f W ( X ) ; U ∣ Z ) ≲ d log ( e n / d ) .
Proof. Condition on a supersample realization Z ~ = z ~ \widetilde{Z}=\widetilde{z} Z = z . The prediction table f W ( X ~ ) f_W(\widetilde{X}) f W ( X ) is one of the binary labelings of the 2 n 2n 2 n supersample inputs induced by F \mathcal{F} F . Hence its conditional support size is at most Π F ( 2 n ) \Pi_{\mathcal{F}}(2n) Π F ( 2 n ) , regardless of the learning algorithm and its internal randomness. Therefore I ( f W ( X ~ ) ; U ∣ Z ~ = z ~ ) ≤ H ( f W ( X ~ ) ∣ Z ~ = z ~ ) ≤ log Π F ( 2 n ) I(f_W(\widetilde{X});U\mid\widetilde{Z}=\widetilde{z})\leq H(f_W(\widetilde{X})\mid\widetilde{Z}=\widetilde{z})\leq\log\Pi_{\mathcal{F}}(2n) I ( f W ( X ) ; U ∣ Z = z ) ≤ H ( f W ( X ) ∣ Z = z ) ≤ log Π F ( 2 n ) . Averaging over Z ~ \widetilde{Z} Z gives the first two inequalities. The trivial bound gives Π F ( 2 n ) ≤ 2 2 n \Pi_{\mathcal{F}}(2n)\leq2^{2n} Π F ( 2 n ) ≤ 2 2 n , while Sauer–Shelah gives Π F ( 2 n ) ≤ ( 2 e n / d ) d \Pi_{\mathcal{F}}(2n)\leq(2en/d)^d Π F ( 2 n ) ≤ ( 2 e n / d ) d when 1 ≤ d ≤ 2 n 1\leq d\leq2n 1 ≤ d ≤ 2 n . These two cases are summarized by the displayed bound with d n = d ∧ 2 n d_n=d\wedge2n d n = d ∧ 2 n .
Harutyunyan et al. [19 ] show that
I ( f W ( X ~ ) ; U ∣ Z ~ = z ~ ) ≤ max { ( d + 1 ) log 2 , d log ( 2 e n / d ) } . I(f_W(\widetilde{X});U\mid\widetilde{Z}=\widetilde{z})\leq\max\{(d+1)\log2,d\log(2en/d)\}. I ( f W ( X ) ; U ∣ Z = z ) ≤ max {( d + 1 ) log 2 , d log ( 2 e n / d )} .
Complete statements
Corollary 10 (CVaR via stability). Assume 0 ≤ ℓ ( f w ( X ) , Y ) ≤ 1 0 \le\ell(f_w(X),Y) \le1 0 ≤ ℓ ( f w ( X ) , Y ) ≤ 1 for all predictors and data points, and write α ˉ : = 1 − α \bar{\alpha} := 1-\alpha α ˉ := 1 − α . For the CVaR objective (3) , Assumption 2 holds with β n = β C V a R , n : = ( α ˉ n ) − 1 \beta_n = \beta_{\mathrm{CVaR},n} := (\bar{\alpha}n)^{-1} β n = β CVaR , n := ( α ˉ n ) − 1 . Hence
L Γ , n C V a R , s t a b ≤ 1 α ˉ n , (58) L^{\mathrm{CVaR,stab}}_{\Gamma,n} \le\frac{1}{\bar{\alpha}\sqrt{n}}, \tag*{(58)} L Γ , n CVaR , stab ≤ α ˉ n 1 , (58) and, writing D t C V a R D_t^{\mathrm{CVaR}} D t CVaR for the local increment in Theorem 2 specialized to CVaR, set I l o c , t C V a R : = I ( D t C V a R ; U t ∣ Z ~ ) I^{\mathrm{CVaR}}_{\mathrm{loc},t} := I(D_t^{\mathrm{CVaR}};U_t \mid\widetilde{Z}) I loc , t CVaR := I ( D t CVaR ; U t ∣ Z ) . Then
∣ E Δ C V a R ∣ ≤ 1 α ˉ n min { 2 n I f C M I , ∑ t = 1 n 2 I l o c , t C V a R } . (59) |\mathbb{E}\Delta_{\mathrm{CVaR}}| \le\frac{1}{\bar{\alpha}n}\min\left\{\sqrt{2n I_{\mathrm{fCMI}}},\sum_{t=1}^{n}\sqrt{2I^{\mathrm{CVaR}}_{\mathrm{loc},t}}\right\}. \tag*{(59)} ∣ E Δ CVaR ∣ ≤ α ˉ n 1 min { 2 n I fCMI , t = 1 ∑ n 2 I loc , t CVaR } . (59)
Corollary 11 (Suboptimal Cressie–Read DRO bound via stability). Assume 0 ≤ ℓ ( f w ( X ) , Y ) ≤ 1 0 \le\ell(f_w(X),Y) \le1 0 ≤ ℓ ( f w ( X ) , Y ) ≤ 1 for all predictors and data points. Let c > 1 c>1 c > 1 , c ∗ = c / ( c − 1 ) c^*=c/(c-1) c ∗ = c / ( c − 1 ) , and κ c , ϵ : = { 1 + c ( c − 1 ) ϵ } 1 / c \kappa_{c,\epsilon}:=\{1+c(c-1)\epsilon\}^{1/c} κ c , ϵ := { 1 + c ( c − 1 ) ϵ } 1/ c . For the empirical Cressie–Read DRO objective (4) , Assumption 2 holds with
β D R O , n s t a b ≤ κ c , ϵ n n 1 / c − 1 . (60) \beta^{\mathrm{stab}}_{\mathrm{DRO},n} \le\kappa_{c,\epsilon_n}n^{1/c-1}. \tag*{(60)} β DRO , n stab ≤ κ c , ϵ n n 1/ c − 1 . (60) Consequently,
L Γ , n D R O , s t a b ≤ κ c , ϵ n n 1 / c − 1 / 2 , (61) L^{\mathrm{DRO,stab}}_{\Gamma,n} \le\kappa_{c,\epsilon_n}n^{1/c-1/2}, \tag*{(61)} L Γ , n DRO , stab ≤ κ c , ϵ n n 1/ c − 1/2 , (61) and, writing D t D R O D_t^{\mathrm{DRO}} D t DRO for the local increment in Theorem 2 specialized to DRO, set I l o c , t D R O : = I ( D t D R O ; U t ∣ Z ~ ) I^{\mathrm{DRO}}_{\mathrm{loc},t} := I(D_t^{\mathrm{DRO}};U_t \mid\widetilde{Z}) I loc , t DRO := I ( D t DRO ; U t ∣ Z ) . Then
∣ E Δ D R O ∣ ≤ κ c , ϵ n n 1 / c − 1 min { 2 n I f C M I , ∑ t = 1 n 2 I l o c , t D R O } . (62) |\mathbb{E}\Delta_{\mathrm{DRO}}| \le\kappa_{c,\epsilon_n}n^{1/c-1}\min\left\{\sqrt{2n I_{\mathrm{fCMI}}},\sum_{t=1}^{n}\sqrt{2I^{\mathrm{DRO}}_{\mathrm{loc},t}}\right\}. \tag*{(62)} ∣ E Δ DRO ∣ ≤ κ c , ϵ n n 1/ c − 1 min { 2 n I fCMI , t = 1 ∑ n 2 I loc , t DRO } . (62)
Comparison with existing bounds
Table 2 compares the full population–training bounds, including both fixed-predictor and learner-dependent terms, with specialized existing results. Here I E C E I_{\mathrm{ECE}} I ECE is the ECE-specific loss-CMI, and δ \delta δ is the failure probability in the CVaR guarantee. In the CVaR row, R D , n \mathfrak{R}_{D,n} R D , n is evaluated on the loss class { ℓ ( f , ⋅ ) } f ∈ F \{\ell(f,\cdot)\}_{f\in\mathcal{F}} { ℓ ( f , ⋅ ) } f ∈ F . The CR DRO constant κ c , ϵ \kappa_{c,\epsilon} κ c , ϵ is specified in Corollary 9.
Multicalibration and binning ECE. For fixed bins J 1 , … , J B J_1,\ldots,J_B J 1 , … , J B , choosing C = { 1 } \mathcal{C}=\{1\} C = { 1 } and H E C E = { h a ( v ) = ∑ b = 1 B a b 1 { v ∈ J b } : a ∈ { − 1 , + 1 } B } \mathcal{H}_{\mathrm{ECE}}=\{h_a(v)=\sum_{b=1}^{B}a_b\mathbf{1}\{v\in J_b\}:a\in\{-1,+1\}^{B}\} H ECE = { h a ( v ) = ∑ b = 1 B a b 1 { v ∈ J b } : a ∈ { − 1 , + 1 } B } makes our MC objective equal binning ECE. The bound of Futami and Fujisawa [11 ] combines this information quantity with bin complexity; our bound in Table 2 separates fixed-predictor and learner-dependent terms and removes explicit B B B from the latter. Since the information quantities differ, neither bound uniformly dominates; for fixed B B B and O ( 1 ) O(1) O ( 1 ) information, both scale as n − 1 / 2 n^{-1/2} n − 1/2 . The comparison with smooth calibration by Futami and Nitanda [12 ] concerns only the fixed-predictor term in Appendix D.6.
CVaR and tail mass. Lee et al. [23 ] give a high-probability excess-risk bound for the empirical OCE minimizer, whereas our full bound in Table 2 controls the expected population–training gap for an arbitrary learner. Our improvement to α ˉ − 1 / 2 \bar{\alpha}^{-1/2} α ˉ − 1/2 applies only to the validation–training term; the fixed-predictor term retains α ˉ − 1 \bar{\alpha}^{-1} α ˉ − 1 . The α ˉ − 1 / 2 \bar{\alpha}^{-1/2} α ˉ − 1/2 dependence is worst-case tight for the convex-L 2 L_2 L 2 certificate: ignoring integer rounding, a feasible vector assigning weight 1 / α ˉ 1/\bar{\alpha} 1/ α ˉ to n α ˉ n\bar{\alpha} n α ˉ tail observations has ∥ q ∥ 2 2 = n / α ˉ \lVert q\rVert_2^2=n/\bar{\alpha} ∥ q ∥ 2 2 = n / α ˉ . This concerns the certificate, not a minimax lower bound for all CVaR estimators. The validation–training term vanishes when I f C M I + log 2 = o ( n α ˉ ) I_{\mathrm{fCMI}}+\log2=o(n\bar{\alpha}) I fCMI + log 2 = o ( n α ˉ ) ; the full bound (47) additionally requires α ˉ n → ∞ \bar{\alpha}\sqrt{n}\to\infty α ˉ n → ∞ . The PAC-Bayes CVaR bound of Mahmoudi et al. [26 ] has leading error O ~ ( C V a R ^ K / ( n α ˉ ) + K / ( n α ˉ ) ) \widetilde{O}\left(\sqrt{\widehat{\mathrm{CVaR}}\,K/(n\bar{\alpha})+K/(n\bar{\alpha})}\right) O ( CVaR K / ( n α ˉ ) + K / ( n α ˉ ) ) , where C V a R ^ \widehat{\mathrm{CVaR}} CVaR is the empirical CVaR and K K K contains the posterior–prior KL and confidence terms. Its worst-case order has the same effective-tail scaling as our convex-L 2 L_2 L 2 validation–training term. We do not claim that our full bound improves the existing high-probability guarantees.
CR DRO. The first term in our bound is the fixed-predictor population–validation plug-in error from Duchi and Namkoong [6 ] . The second is the validation–training error due to the learned predictor’s adaptivity. The convex L 2 L_2 L 2 geometry of the adversarial weights yields the same n − 1 / ( c ∗ ∨ 2 ) n^{-1/(c^*\vee2)} n − 1/ ( c ∗ ∨ 2 ) exponent, with an additional fCMI factor; if I f C M I = O ( 1 ) I_{\mathrm{fCMI}}=O(1) I fCMI = O ( 1 ) , both terms retain that exponent. In Corollaries 6 and 9, the exponent is r c = ( c − 1 ) / c r_c=(c-1)/c r c = ( c − 1 ) / c for 1 < c < 2 1<c<2 1 < c < 2 and r c = 1 / 2 r_c=1/2 r c = 1/2 for c ≥ 2 c\ge2 c ≥ 2 . The limit r c → 0 r_c\to0 r c → 0 as c ↓ 1 c\downarrow1 c ↓ 1 does not give a rate for exact KL-DRO. For fixed c > 1 c>1 c > 1 , increasing the positive radius ϵ \epsilon ϵ increases the coefficient κ c , ϵ \kappa_{c,\epsilon} κ c , ϵ but leaves r c r_c r c unchanged; the fixed-predictor constant K c , ϵ K_{c,\epsilon} K c , ϵ need not be monotone. Equal numerical radii for different c c c do not specify equal ambiguity sets. Thus, unification means this decomposition and recovery of known rates, not uniform improvement.
Discussion about measurability of optimized set-input values
This paper defines set-input objectives in a general form, but our concrete targets are the MC, CVaR, and DRO metrics. For these objectives, measurability has already been studied in the existing literature. On the other hand, when considering more general set-input objectives, one needs to account for measurability issues as follows.
The main text works with finite real-valued optimized objectives. We make the measurability convention explicit here, because the proof of Theorem 1 conditions on the entire supersample prediction table and then evaluates the optimized objective for all selector vectors.
If the auxiliary set Π \Pi Π is finite or countable and each row map z ↦ ρ π ( f ( x ) , z ) z\mapsto\rho_\pi(f(x),z) z ↦ ρ π ( f ( x ) , z ) is measurable, then all empirical and population optimized values are measurable. For uncountable auxiliary classes, we use the following separable-version convention. For each fixed predictor f f f , there is a countable subset Π 0 ( f ) ⊆ Π \Pi_0(f)\subseteq\Pi Π 0 ( f ) ⊆ Π , independent of the empirical realization, such that for every probability law Q Q Q used in the argument, the population law, empirical laws, and mixed empirical laws arising in the supersample interpolation,
Opt π ∈ Π ψ π ( ∫ ρ π ( f ( x ) , z ) d Q ( z ) ) = Opt π ∈ Π 0 ( f ) ψ π ( ∫ ρ π ( f ( x ) , z ) d Q ( z ) ) , \operatorname{Opt}_{\pi\in\Pi}\psi_\pi\left(\int\rho_\pi(f(x),z)\,dQ(z)\right)=\operatorname{Opt}_{\pi\in\Pi_0(f)}\psi_\pi\left(\int\rho_\pi(f(x),z)\,dQ(z)\right), Opt π ∈ Π ψ π ( ∫ ρ π ( f ( x ) , z ) d Q ( z ) ) = Opt π ∈ Π 0 ( f ) ψ π ( ∫ ρ π ( f ( x ) , z ) d Q ( z ) ) ,
where Opt ∈ { sup , inf } \operatorname{Opt}\in\{\sup,\inf\} Opt ∈ { sup , inf } . The optimized value is then a countable supremum or infimum of measurable maps. When the predictor is random, the convention is applied after conditioning on the realized predictor or, equivalently, to a measurable separable version of the optimized value. This is sufficient for the conditional maps u ↦ Φ n ( f W , Z ~ u ) u\mapsto\Phi_n(f_W,\widetilde{Z}_u) u ↦ Φ n ( f W , Z u ) and u ↦ Γ f W , Z ~ ( u ) u\mapsto\Gamma_{f_W,\widetilde{Z}}(u) u ↦ Γ f W , Z ( u ) used in the proofs.
The convention covers finite auditor classes used for binned miscalibration. For compact parametric or smooth witness classes, it follows from a countable dense subclass whenever the displayed integral and the post-aggregation map are continuous in the auxiliary parameter on the relevant feasible set. If such a separable version is not available, the statements should be read with outer expectations or essential suprema/infima; we do not use that extra notation in the paper.
Additional Discussion
This appendix collects refinements and alternative comparisons that are not used in the main theorem. They are included to clarify the scope of Theorem 1, the relation to single-index fast-rate bounds for ordinary losses, and variants in which the validation set is replaced by a smaller subset. Throughout this appendix we use the notation of § 3 : the local increments are the random variables D t D_t D t from Theorem 2, and under uniform stability they satisfy ∣ D t ∣ ≤ β n \lvert D_t\rvert\le\beta_n ∣ D t ∣ ≤ β n .
The intuition behind the population–validation term
Another distinction concerns the role of the population–validation discrepancy. For ordinary losses, the difference between population and validation evaluation is expressed through a mean, and hence there is no optimization mismatch. In our setting, however, the object of interest is an optimized criterion, so the population objective and the finite validation set objective can differ significantly. We therefore control the population–validation set difference as a “bias” term by uniform convergence, since learning is not involved in that comparison. By contrast, the validation–training set difference involves the learned predictor and is controlled by IT generalization analysis. This decomposition prevents the objective-specific population–validation cost from being misattributed to the IT-controlled validation–training term.
Conditioning and information relations for local increments
The main single-index bound uses the unconditioned single-coordinate quantities I ( D t ; U t ∣ Z ~ ) I(D_t;U_t \mid\widetilde{Z}) I ( D t ; U t ∣ Z ) . This subsection records how these quantities relate to the more strongly conditioned leave-one-out quantities that sometimes arise in diagnostics. Write F : = f W ( X ~ ) F := f_W(\widetilde{X}) F := f W ( X ) . Throughout this subsection, the selectors U 1 , … , U n U_1,\ldots,U_n U 1 , … , U n are assumed to be conditionally independent given Z ~ \widetilde{Z} Z .
Lemma 10 (Unconditioned and leave-one-out local information). Fix t ∈ [ n ] t \in[n] t ∈ [ n ] . Let D t D_t D t be a local increment satisfying D t ∈ σ ( F , Z ~ , U − t ) D_t \in\sigma(F,\widetilde{Z},U_{-t}) D t ∈ σ ( F , Z , U − t ) . Here σ ( F , Z ~ , U − t ) \sigma(F,\widetilde{Z},U_{-t}) σ ( F , Z , U − t ) denotes the sigma-algebra generated by F F F , Z ~ \widetilde{Z} Z , and U − t U_{-t} U − t , so D t ∈ σ ( F , Z ~ , U − t ) D_t \in\sigma(F,\widetilde{Z},U_{-t}) D t ∈ σ ( F , Z , U − t ) means that D t D_t D t is measurable with respect to these variables.
Then
I ( D t ; U t ∣ Z ~ ) ≤ I ( D t ; U t ∣ Z ~ , U − t ) , (63) I(D_t;U_t \mid\widetilde{Z}) \le I(D_t;U_t \mid\widetilde{Z},U_{-t}), \tag*{(63)} I ( D t ; U t ∣ Z ) ≤ I ( D t ; U t ∣ Z , U − t ) , (63) I ( D t ; U t ∣ Z ~ , U − t ) ≤ I ( F ; U t ∣ Z ~ , U − t ) . (64) I(D_t;U_t \mid\widetilde{Z},U_{-t}) \le I(F;U_t \mid\widetilde{Z},U_{-t}). \tag*{(64)} I ( D t ; U t ∣ Z , U − t ) ≤ I ( F ; U t ∣ Z , U − t ) . (64)
Proof. For (63) , use the entropy form of mutual information. Since U t ⊥ ⊥ U − t ∣ Z ~ U_t \perp\!\!\!\perp U_{-t} \mid\widetilde{Z} U t ⊥ ⊥ U − t ∣ Z , we have H ( U t ∣ Z ~ , U − t ) = H ( U t ∣ Z ~ ) H(U_t \mid\widetilde{Z},U_{-t}) = H(U_t \mid\widetilde{Z}) H ( U t ∣ Z , U − t ) = H ( U t ∣ Z ) . Therefore
I ( D t ; U t ∣ Z ~ , U − t ) − I ( D t ; U t ∣ Z ~ ) = H ( U t ∣ Z ~ , U − t ) − H ( U t ∣ D t , Z ~ , U − t ) − H ( U t ∣ Z ~ ) + H ( U t ∣ D t , Z ~ ) = H ( U t ∣ D t , Z ~ ) − H ( U t ∣ D t , Z ~ , U − t ) = I ( U t ; U − t ∣ D t , Z ~ ) ≥ 0. \begin{aligned}
I(D_t;U_t \mid\widetilde{Z},U_{-t}) - I(D_t;U_t \mid\widetilde{Z})
&= H(U_t \mid\widetilde{Z},U_{-t}) - H(U_t \mid D_t,\widetilde{Z},U_{-t}) - H(U_t \mid\widetilde{Z}) + H(U_t \mid D_t,\widetilde{Z}) \\
&= H(U_t \mid D_t,\widetilde{Z}) - H(U_t \mid D_t,\widetilde{Z},U_{-t}) \\
&= I(U_t;U_{-t} \mid D_t,\widetilde{Z}) \ge0.
\end{aligned} I ( D t ; U t ∣ Z , U − t ) − I ( D t ; U t ∣ Z ) = H ( U t ∣ Z , U − t ) − H ( U t ∣ D t , Z , U − t ) − H ( U t ∣ Z ) + H ( U t ∣ D t , Z ) = H ( U t ∣ D t , Z ) − H ( U t ∣ D t , Z , U − t ) = I ( U t ; U − t ∣ D t , Z ) ≥ 0.
This proves (63) .
For (64) , condition on ( Z ~ , U − t ) (\widetilde{Z},U_{-t}) ( Z , U − t ) . Under this conditioning, the assumption D t ∈ σ ( F , Z ~ , U − t ) D_t \in\sigma(F,\widetilde{Z},U_{-t}) D t ∈ σ ( F , Z , U − t ) says that D t D_t D t is a measurable function of F F F . Hence, conditionally on ( Z ~ , U − t ) (\widetilde{Z},U_{-t}) ( Z , U − t ) , the variables form the Markov chain U t ⟶ F ⟶ D t U_t \longrightarrow F \longrightarrow D_t U t ⟶ F ⟶ D t . The conditional data-processing inequality gives I ( D t ; U t ∣ Z ~ , U − t ) ≤ I ( F ; U t ∣ Z ~ , U − t ) I(D_t;U_t \mid\widetilde{Z},U_{-t}) \le I(F;U_t \mid\widetilde{Z},U_{-t}) I ( D t ; U t ∣ Z , U − t ) ≤ I ( F ; U t ∣ Z , U − t ) , as claimed. □
Lemma 11 (Information about U U U and U t U_t U t in the predictions). Let F F F be any random element jointly distributed with ( Z ~ , U ) (\widetilde{Z},U) ( Z , U ) . Assume that the selectors U 1 , … , U n U_1,\ldots,U_n U 1 , … , U n are conditionally independent given Z ~ \widetilde{Z} Z . Then
∑ t = 1 n I ( F ; U t ∣ Z ~ ) ≤ I ( F ; U ∣ Z ~ ) ≤ ∑ t = 1 n I ( F ; U t ∣ Z ~ , U − t ) . \sum_{t=1}^{n} I(F;U_t \mid\widetilde{Z}) \le I(F;U \mid\widetilde{Z}) \le\sum_{t=1}^{n} I(F;U_t \mid\widetilde{Z},U_{-t}). t = 1 ∑ n I ( F ; U t ∣ Z ) ≤ I ( F ; U ∣ Z ) ≤ t = 1 ∑ n I ( F ; U t ∣ Z , U − t ) .
Proof. For the first inequality, use the chain rule with U < t : = ( U 1 , … , U t − 1 ) U_{<t} := (U_1,\ldots,U_{t-1}) U < t := ( U 1 , … , U t − 1 ) : I ( F ; U ∣ Z ~ ) = ∑ t = 1 n I ( F ; U t ∣ Z ~ , U < t ) I(F;U \mid\widetilde{Z}) = \sum_{t=1}^{n} I(F;U_t \mid\widetilde{Z},U_{<t}) I ( F ; U ∣ Z ) = ∑ t = 1 n I ( F ; U t ∣ Z , U < t ) . Since U t ⊥ ⊥ U < t ∣ Z ~ U_t \perp\!\!\!\perp U_{<t} \mid\widetilde{Z} U t ⊥ ⊥ U < t ∣ Z , H ( U t ∣ Z ~ , U < t ) = H ( U t ∣ Z ~ ) H(U_t \mid\widetilde{Z},U_{<t}) = H(U_t \mid\widetilde{Z}) H ( U t ∣ Z , U < t ) = H ( U t ∣ Z ) . Thus
I ( F ; U t ∣ Z ~ , U < t ) = H ( U t ∣ Z ~ ) − H ( U t ∣ F , Z ~ , U < t ) ≥ H ( U t ∣ Z ~ ) − H ( U t ∣ F , Z ~ ) = I ( F ; U t ∣ Z ~ ) , \begin{aligned}
I(F;U_t \mid\widetilde{Z},U_{<t})
&= H(U_t \mid\widetilde{Z}) - H(U_t \mid F,\widetilde{Z},U_{<t}) \\
&\ge H(U_t \mid\widetilde{Z}) - H(U_t \mid F,\widetilde{Z}) \\
&= I(F;U_t \mid\widetilde{Z}),
\end{aligned} I ( F ; U t ∣ Z , U < t ) = H ( U t ∣ Z ) − H ( U t ∣ F , Z , U < t ) ≥ H ( U t ∣ Z ) − H ( U t ∣ F , Z ) = I ( F ; U t ∣ Z ) ,
where the inequality is the monotonicity of conditional entropy under additional conditioning. Summing over t t t proves the first inequality.
For the second inequality, expand both sides in entropy form. Conditional independence gives
∑ t = 1 n I ( F ; U t ∣ Z ~ , U − t ) = ∑ t = 1 n H ( U t ∣ Z ~ ) − ∑ t = 1 n H ( U t ∣ F , Z ~ , U − t ) . \sum_{t=1}^{n} I(F;U_t \mid\widetilde{Z},U_{-t})=\sum_{t=1}^{n} H(U_t \mid\widetilde{Z})-\sum_{t=1}^{n} H(U_t \mid F,\widetilde{Z},U_{-t}). t = 1 ∑ n I ( F ; U t ∣ Z , U − t ) = t = 1 ∑ n H ( U t ∣ Z ) − t = 1 ∑ n H ( U t ∣ F , Z , U − t ) .
whereas
I ( F ; U ∣ Z ~ ) = ∑ t = 1 n H ( U t ∣ Z ~ ) − H ( U ∣ F , Z ~ ) . I(F;U \mid\widetilde{Z})=\sum_{t=1}^{n} H(U_t \mid\widetilde{Z})-H(U \mid F,\widetilde{Z}). I ( F ; U ∣ Z ) = t = 1 ∑ n H ( U t ∣ Z ) − H ( U ∣ F , Z ) .
Therefore
∑ t = 1 n I ( F ; U t ∣ Z ~ , U − t ) − I ( F ; U ∣ Z ~ ) = H ( U ∣ F , Z ~ ) − ∑ t = 1 n H ( U t ∣ F , Z ~ , U − t ) . \begin{aligned}
\sum_{t=1}^{n} I(F;U_t \mid\widetilde{Z},U_{-t})-I(F;U \mid\widetilde{Z}) \\
&=H(U \mid F,\widetilde{Z})-\sum_{t=1}^{n} H(U_t \mid F,\widetilde{Z},U_{-t}).
\end{aligned} t = 1 ∑ n I ( F ; U t ∣ Z , U − t ) − I ( F ; U ∣ Z ) = H ( U ∣ F , Z ) − t = 1 ∑ n H ( U t ∣ F , Z , U − t ) .
With U > t : = ( U t + 1 , … , U n ) U_{>t}:=(U_{t+1},\ldots,U_n) U > t := ( U t + 1 , … , U n ) , the last display equals ∑ t = 1 n I ( U t ; U < t ∣ F , Z ~ , U > t ) \sum_{t=1}^{n} I(U_t;U_{<t}\mid F,\widetilde{Z},U_{>t}) ∑ t = 1 n I ( U t ; U < t ∣ F , Z , U > t ) , which is the conditional dual total correlation of U U U given ( F , Z ~ ) (F,\widetilde{Z}) ( F , Z ) . It is nonnegative, proving the second inequality.
The first lemma explains why the unconditioned local information used in the main theorem is no larger than the leave-one-out conditional diagnostic: I ( D t ; U t ∣ Z ~ ) ≤ I ( D t ; U t ∣ Z ~ , U − t ) I(D_t;U_t\mid\widetilde{Z})\le I(D_t;U_t\mid\widetilde{Z},U_{-t}) I ( D t ; U t ∣ Z ) ≤ I ( D t ; U t ∣ Z , U − t ) . Moreover, under the mixed-path measurability condition D t ∈ σ ( F , Z ~ , U − t ) D_t\in\sigma(F,\widetilde{Z},U_{-t}) D t ∈ σ ( F , Z , U − t ) , this leave-one-out diagnostic is controlled by the corresponding leave-one-out single-coordinate information of the full prediction table: I ( D t ; U t ∣ Z ~ , U − t ) ≤ I ( F ; U t ∣ Z ~ , U − t ) I(D_t;U_t\mid\widetilde{Z},U_{-t})\le I(F;U_t\mid\widetilde{Z},U_{-t}) I ( D t ; U t ∣ Z , U − t ) ≤ I ( F ; U t ∣ Z , U − t ) . However, these inequalities do not identify the unconditioned local term with the unconditioned single-coordinate table information I ( F ; U t ∣ Z ~ ) I(F;U_t\mid\widetilde{Z}) I ( F ; U t ∣ Z ) , nor do they imply that the sum of local terms is bounded by the CMI for the full vector U U U , namely I ( F ; U ∣ Z ~ ) I(F;U\mid\widetilde{Z}) I ( F ; U ∣ Z ) .
Remark 2 (Conditioning on the supersample alone). There is no general ordering between I ( D t ; U t ∣ Z ~ ) I(D_t;U_t\mid\widetilde{Z}) I ( D t ; U t ∣ Z ) and I ( F ; U t ∣ Z ~ ) I(F;U_t\mid\widetilde{Z}) I ( F ; U t ∣ Z ) under the mixed-path definition. The obstruction is that D t D_t D t is a function of ( F , Z ~ , U − t ) (F,\widetilde{Z},U_{-t}) ( F , Z , U − t ) , not necessarily of ( F , Z ~ ) (F,\widetilde{Z}) ( F , Z ) alone. Thus, after conditioning only on Z ~ \widetilde{Z} Z , the Markov chain U t → F → D t U_t\to F\to D_t U t → F → D t need not hold. Consequently, the preceding inequalities do not imply a general bound of the form ∑ t = 1 n I ( D t ; U t ∣ Z ~ ) ≤ I ( F ; U ∣ Z ~ ) \sum_{t=1}^{n} I(D_t;U_t\mid\widetilde{Z})\le I(F;U\mid\widetilde{Z}) ∑ t = 1 n I ( D t ; U t ∣ Z ) ≤ I ( F ; U ∣ Z ) . The global fCMI I ( F ; U ∣ Z ~ ) I(F;U\mid\widetilde{Z}) I ( F ; U ∣ Z ) remains the natural quantity for VC and growth-function arguments, whereas the one-dimensional local terms are mainly useful as estimable diagnostics unless additional locality structure is imposed on the increments.
Bernstein-type local refinements
Theorem 2 uses a Hoeffding-type bound for the one-dimensional variables ε t D t \varepsilon_tD_t ε t D t . Since ∣ D t ∣ ≤ β n \lvert D_t\rvert\le\beta_n ∣ D t ∣ ≤ β n , a Bernstein-type comparison is also available. The comparison below is intentionally kept as a diagnostic statement: it does not by itself imply a fast rate, and in the present conditional Rademacher setting it is not the sharpest possible local bound. Its role is to identify the additional variance control that would be needed for a Bernstein-style route to be useful.
For each t ∈ [ n ] t\in[n] t ∈ [ n ] , define I t : = I ( D t ; U t ∣ Z ~ ) I_t:=I(D_t;U_t\mid\widetilde{Z}) I t := I ( D t ; U t ∣ Z ) , v ˉ t : = E [ D t 2 ] \bar{v}_t:=\mathbb{E}[D_t^2] v ˉ t := E [ D t 2 ] .
Proposition 2 (Bernstein-type local comparison). Under Assumption 2, for every t ∈ [ n ] t\in[n] t ∈ [ n ] and every 0 < λ < 3 / β n 0<\lambda<3/\beta_n 0 < λ < 3/ β n ,
∣ E [ ε t D t ] ∣ ≤ I t λ + λ v ˉ t 2 ( 1 − λ β n / 3 ) . (65) \left|\mathbb{E}[\varepsilon_tD_t]\right|\le\frac{I_t}{\lambda}+\frac{\lambda\bar{v}_t}{2(1-\lambda\beta_n/3)}. \tag*{(65)} ∣ E [ ε t D t ] ∣ ≤ λ I t + 2 ( 1 − λ β n /3 ) λ v ˉ t . (65)
Consequently,
∣ E [ ε t D t ] ∣ ≤ 2 v ˉ t I t + β n 3 I t . (66) \left|\mathbb{E}[\varepsilon_tD_t]\right|\le\sqrt{2\bar{v}_tI_t}+\frac{\beta_n}{3}I_t. \tag*{(66)} ∣ E [ ε t D t ] ∣ ≤ 2 v ˉ t I t + 3 β n I t . (66)
Therefore,
∣ E [ Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) ] ∣ ≤ ∑ t = 1 n ( 2 ν ˉ t I t + β n 3 I t ) . (67) \left|\mathbb{E}\left[\Phi_n(f_W;S')-\Phi_n(f_W;S)\right]\right| \le\sum_{t=1}^{n}\left(\sqrt{2\bar{\nu}_t I_t}+\frac{\beta_n}{3}I_t\right). \tag*{(67)} ∣ E [ Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) ] ∣ ≤ t = 1 ∑ n ( 2 ν ˉ t I t + 3 β n I t ) . (67)
Proof. Fix t t t . Conditionally on Z ~ \widetilde{Z} Z , use as reference measure P D t ∣ Z ~ ⊗ Q t P_{D_t\mid\widetilde{Z}}\otimes Q_t P D t ∣ Z ⊗ Q t , Q t = Unif ( { 0 , 1 } ) Q_t=\operatorname{Unif}(\{0,1\}) Q t = Unif ({ 0 , 1 }) . Equivalently, under the reference measure, D t D_t D t keeps its actual conditional marginal distribution, while U t U_t U t is resampled independently from Q t Q_t Q t . Thus, conditionally on ( D t , Z ~ ) (D_t,\widetilde{Z}) ( D t , Z ) , the variable ε t D t \varepsilon_tD_t ε t D t is centered, has conditional second moment D t 2 D_t^2 D t 2 , and is bounded in absolute value by β n \beta_n β n . The Bernstein MGF bound for a centered bounded variable gives
log E Q t [ exp { λ ε t D t } ∣ D t , Z ~ ] ≤ λ 2 D t 2 2 ( 1 − λ β n / 3 ) ( 0 < λ < 3 / β n ) . \log\mathbb{E}_{Q_t}\left[\exp\{\lambda\varepsilon_tD_t\}\mid D_t,\widetilde{Z}\right]\le\frac{\lambda^2D_t^2}{2(1-\lambda\beta_n/3)}\qquad(0<\lambda<3/\beta_n). log E Q t [ exp { λ ε t D t } ∣ D t , Z ] ≤ 2 ( 1 − λ β n /3 ) λ 2 D t 2 ( 0 < λ < 3/ β n ) .
Applying the conditional Donsker–Varadhan variational inequality to P U t ∣ D t , Z ~ P_{U_t\mid D_t,\widetilde{Z}} P U t ∣ D t , Z relative to Q t Q_t Q t , and then averaging over ( D t , Z ~ ) (D_t,\widetilde{Z}) ( D t , Z ) , yields λ E [ ε t D t ] ≤ I t + λ 2 ν ˉ t 2 ( 1 − λ β n / 3 ) \lambda\mathbb{E}[\varepsilon_tD_t]\le I_t+\frac{\lambda^2\bar{\nu}_t}{2(1-\lambda\beta_n/3)} λ E [ ε t D t ] ≤ I t + 2 ( 1 − λ β n /3 ) λ 2 ν ˉ t . The same argument applied to − ε t D t -\varepsilon_tD_t − ε t D t gives the reverse inequality, and hence the absolute-value bound in (65) . Dividing by λ \lambda λ proves the unoptimized bound.
The standard sub-gamma optimization inequality
inf 0 < λ < 3 / β n { a λ + λ v 2 ( 1 − λ β n / 3 ) } ≤ 2 a v + β n 3 a \inf_{0<\lambda<3/\beta_n}\left\{\frac{a}{\lambda}+\frac{\lambda v}{2(1-\lambda\beta_n/3)}\right\}\le\sqrt{2av}+\frac{\beta_n}{3}a 0 < λ < 3/ β n inf { λ a + 2 ( 1 − λ β n /3 ) λ v } ≤ 2 a v + 3 β n a
with a = I t a=I_t a = I t and v = ν ˉ t v=\bar{\nu}_t v = ν ˉ t proves (66) . Finally, summing over t t t and using the telescoping identity (10) proves (67) . □
Here, the Bernstein-type bound is not intended to improve the main bound. Rather, it shows that any Bernstein-style fast-rate conclusion would need additional control of the local variance proxy ν ˉ t \bar{\nu}_t ν ˉ t .
When the objective is nonnegative, the variance proxy can be related to mixed-sample objective values.
Proposition 3 (Positive-objective variance proxy). Assume Φ n ( f W ; T ) ≥ 0 \Phi_n(f_W;T)\ge0 Φ n ( f W ; T ) ≥ 0 almost surely for every batch T T T . Then, for every t ∈ [ n ] t\in[n] t ∈ [ n ] ,
D t 2 ≤ β n ∣ D t ∣ ≤ β n ( Φ n ( f W ; S ( t , 0 ) ) + Φ n ( f W ; S ( t , 1 ) ) ) . (68) D_t^2\le\beta_n|D_t|\le\beta_n\left(\Phi_n(f_W;S^{(t,0)})+\Phi_n(f_W;S^{(t,1)})\right). \tag*{(68)} D t 2 ≤ β n ∣ D t ∣ ≤ β n ( Φ n ( f W ; S ( t , 0 ) ) + Φ n ( f W ; S ( t , 1 ) ) ) . (68) Consequently, with M t : = 1 2 E [ Φ n ( f W ; S ( t , 0 ) ) + Φ n ( f W ; S ( t , 1 ) ) ] M_t:=\frac{1}{2}\mathbb{E}\left[\Phi_n(f_W;S^{(t,0)})+\Phi_n(f_W;S^{(t,1)})\right] M t := 2 1 E [ Φ n ( f W ; S ( t , 0 ) ) + Φ n ( f W ; S ( t , 1 ) ) ] , one has
ν ˉ t ≤ min { β n 2 , 2 β n M t } . \bar{\nu}_t\le\min\{\beta_n^2,2\beta_nM_t\}. ν ˉ t ≤ min { β n 2 , 2 β n M t } .
Proof. The stability assumption gives ∣ D t ∣ ≤ β n |D_t|\le\beta_n ∣ D t ∣ ≤ β n , so D t 2 ≤ β n ∣ D t ∣ D_t^2\le\beta_n|D_t| D t 2 ≤ β n ∣ D t ∣ . Since the two mixed-sample values are nonnegative,
∣ D t ∣ = ∣ Φ n ( f W ; S ( t , 0 ) ) − Φ n ( f W ; S ( t , 1 ) ) ∣ ≤ Φ n ( f W ; S ( t , 0 ) ) + Φ n ( f W ; S ( t , 1 ) ) . |D_t|=\left|\Phi_n(f_W;S^{(t,0)})-\Phi_n(f_W;S^{(t,1)})\right|\le\Phi_n(f_W;S^{(t,0)})+\Phi_n(f_W;S^{(t,1)}). ∣ D t ∣ = Φ n ( f W ; S ( t , 0 ) ) − Φ n ( f W ; S ( t , 1 ) ) ≤ Φ n ( f W ; S ( t , 0 ) ) + Φ n ( f W ; S ( t , 1 ) ) .
This proves (68) . Taking expectations gives ν ˉ t ≤ 2 β n M t \bar{\nu}_t\le2\beta_nM_t ν ˉ t ≤ 2 β n M t , while ∣ D t ∣ ≤ β n |D_t|\le\beta_n ∣ D t ∣ ≤ β n also gives ν ˉ t ≤ β n 2 \bar{\nu}_t\le\beta_n^2 ν ˉ t ≤ β n 2 . □
Corollary 12 (Explicit conservative Bernstein constants for CVaR and CR DRO). Assume the base loss satisfies 0 ≤ ℓ ( f w ( X ) , Y ) ≤ 1 0\le\ell(f_w(X),Y)\le1 0 ≤ ℓ ( f w ( X ) , Y ) ≤ 1 .
For CVaR , β n = ( α ˉ n ) − 1 \beta_n=(\bar{\alpha}n)^{-1} β n = ( α ˉ n ) − 1 , and therefore
∣ E [ Φ C V a R , α , n ( f W ; S ′ ) − Φ C V a R , α , n ( f W ; S ) ] ∣ ≤ 1 α ˉ n ∑ t = 1 n ( 2 I t + 1 3 I t ) , \left|\mathbb{E}\left[\Phi_{\mathrm{CVaR},\alpha,n}(f_W;S')-\Phi_{\mathrm{CVaR},\alpha,n}(f_W;S)\right]\right|\le\frac{1}{\bar{\alpha}n}\sum_{t=1}^{n}\left(\sqrt{2I_t}+\frac{1}{3}I_t\right), ∣ E [ Φ CVaR , α , n ( f W ; S ′ ) − Φ CVaR , α , n ( f W ; S ) ] ∣ ≤ α ˉ n 1 t = 1 ∑ n ( 2 I t + 3 1 I t ) , using the crude bound ν ˉ t ≤ β n 2 \bar{\nu}_t\le\beta_n^2 ν ˉ t ≤ β n 2 .
For CR DRO under the stability route , β n ≤ κ c , ϵ n 1 / c − 1 \beta_n\le\kappa_{c,\epsilon}n^{1/c-1} β n ≤ κ c , ϵ n 1/ c − 1 , and hence
∣ E Δ D R O ∣ ≤ κ c , ϵ n 1 / c − 1 ∑ t = 1 n ( 2 I t + 1 3 I t ) . |\mathbb{E}\Delta_{\mathrm{DRO}}|\le\kappa_{c,\epsilon}n^{1/c-1}\sum_{t=1}^{n}\left(\sqrt{2I_t}+\frac{1}{3}I_t\right). ∣ E Δ DRO ∣ ≤ κ c , ϵ n 1/ c − 1 t = 1 ∑ n ( 2 I t + 3 1 I t ) .
Here I t = I ( D t ; U t ∣ Z ~ ) I_t = I(D_t; U_t \mid\widetilde{Z}) I t = I ( D t ; U t ∣ Z ) , with D t D_t D t specialized to the corresponding objective.
Proof. For both objectives, the stated values of β n \beta_n β n are those proved in the stability route. The crude variance bound ν ˉ t ≤ β n 2 \bar{\nu}_t \le\beta_n^2 ν ˉ t ≤ β n 2 follows from ∣ D t ∣ ≤ β n |D_t| \le\beta_n ∣ D t ∣ ≤ β n . Substituting this bound into Proposition 2 gives the two displays.
The preceding corollary shows that a Bernstein-type comparison is available, but it also makes clear why this comparison does not automatically produce a fast rate. A useful Bernstein-style refinement would require an additional self-bounding relation for the local variance proxy.
Proposition 4 (Bernstein closure under a mixed-sample self-bounding assumption). Assume Φ n ≥ 0 \Phi_n \ge0 Φ n ≥ 0 , and suppose there exist constants a , b ≥ 0 a,b \ge0 a , b ≥ 0 such that for every t ∈ [ n ] t \in[n] t ∈ [ n ] ,
M t ≤ a E [ Φ n ( f W ; S ) ] + b . (69) M_t \le a\mathbb{E}[\Phi_n(f_W;S)] + b. \tag*{(69)} M t ≤ a E [ Φ n ( f W ; S )] + b . (69) Then
∣ E [ Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) ] ∣ ≤ ∑ t = 1 n ( 2 min { β n 2 , 2 β n ( a E [ Φ n ( f W ; S ) ] + b ) } I t + β n 3 I t ) . \left|\mathbb{E}\left[\Phi_n(f_W;S')-\Phi_n(f_W;S)\right]\right|
\le\sum_{t=1}^{n}\left(\sqrt{2\min\left\{\beta_n^2,2\beta_n\left(a\mathbb{E}[\Phi_n(f_W;S)]+b\right)\right\}I_t}+\frac{\beta_n}{3}I_t\right). ∣ E [ Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) ] ∣ ≤ t = 1 ∑ n ( 2 min { β n 2 , 2 β n ( a E [ Φ n ( f W ; S )] + b ) } I t + 3 β n I t ) . In particular, if b = 0 b=0 b = 0 and a E [ Φ n ( f W ; S ) ] ≪ β n a\mathbb{E}[\Phi_n(f_W;S)] \ll\beta_n a E [ Φ n ( f W ; S )] ≪ β n , then the leading variance term behaves like
∑ t = 1 n 4 β n a E [ Φ n ( f W ; S ) ] I t , \sum_{t=1}^{n}\sqrt{4\beta_n a\mathbb{E}[\Phi_n(f_W;S)]I_t}, t = 1 ∑ n 4 β n a E [ Φ n ( f W ; S )] I t , up to the residual linear term β n 3 ∑ t = 1 n I t \frac{\beta_n}{3}\sum_{t=1}^{n}I_t 3 β n ∑ t = 1 n I t . This is only a local variance-sensitive, fast-rate-shaped dependence on E [ Φ n ( f W ; S ) ] \mathbb{E}[\Phi_n(f_W;S)] E [ Φ n ( f W ; S )] ; it should not be interpreted as an ordinary-loss fast rate without further aggregation or row-local variance structure.
Proof. Combine (69) with Proposition 3, and then substitute the resulting variance proxy into Proposition 2.
A shifted-sign attempt and why it does not close in general
For the pointwise special case Φ n ( f ; S ) = L ^ ( f ; S ) \Phi_n(f;S)=\widehat{L}(f;S) Φ n ( f ; S ) = L ( f ; S ) , the increments in Theorem 2 reduce to D t = 1 { ℓ ( f W , Z ~ t , 0 ) − ℓ ( f W , Z ~ t , 1 ) } D_t = \mathbbm{1}\{\ell(f_W,\widetilde{Z}_{t,0})-\ell(f_W,\widetilde{Z}_{t,1})\} D t = 1 { ℓ ( f W , Z t , 0 ) − ℓ ( f W , Z t , 1 )} , and the single-index bound recovers the standard row-wise eCMI bound, as shown in Appendix C. In this case, the training and validation risks are averages of loss values from the same loss table. This row-wise structure also permits the shifted-Rademacher fast-rate analysis of Wang and Mao [39 ] , yielding bounds linear in the information terms when the training risk is zero. This refinement uses the empirical-risk structure in addition to the single-index decomposition; it does not follow from the square-root bound in Theorem 2 alone. For general set-input optimization-valued objectives, the additional structure need not hold, as explained below.
For ordinary row-separable losses, fast-rate refinements based on shifted Rademacher variables exploit an additional row-wise representation of the empirical loss table [39 ] . This representation is not available from the mixed-path increments D t D_t D t alone for a general optimized set-input objective. To see the obstruction, fix C > 0 C>0 C > 0 and define Δ C : = Φ n ( f W ; S ′ ) − ( 1 + C ) Φ n ( f W ; S ) \Delta_C := \Phi_n(f_W;S')-(1+C)\Phi_n(f_W;S) Δ C := Φ n ( f W ; S ′ ) − ( 1 + C ) Φ n ( f W ; S ) . Using the telescoping identity (10) ,
Δ C = ∑ t = 1 n ϵ t D t − C Φ n ( f W ; S ) . \Delta_C = \sum_{t=1}^{n}\epsilon_t D_t-C\Phi_n(f_W;S). Δ C = t = 1 ∑ n ϵ t D t − C Φ n ( f W ; S ) .
For any shift parameter η ∈ R \eta\in\mathbb{R} η ∈ R , this can be rewritten exactly as
Δ C = ∑ t = 1 n ( ϵ t − η ) D t + ( η ∑ t = 1 n D t − C Φ n ( f W ; S ) ) . \Delta_C = \sum_{t=1}^{n}(\epsilon_t-\eta)D_t+\left(\eta\sum_{t=1}^{n}D_t-C\Phi_n(f_W;S)\right). Δ C = t = 1 ∑ n ( ϵ t − η ) D t + ( η t = 1 ∑ n D t − C Φ n ( f W ; S ) ) .
The first term has the desired shifted-sign form. The difficulty is the remainder R n , C : = η ∑ t = 1 n D t − C Φ n ( f W ; S ) R_{n,C}:=\eta\sum_{t=1}^{n}D_t-C\Phi_n(f_W;S) R n , C := η ∑ t = 1 n D t − C Φ n ( f W ; S ) . For ordinary pointwise empirical risks, shifted-Rademacher proofs use the fact that, after the algorithmic output is fixed, the empirical training risk is an average of row-wise losses from the same loss-pair table. That extra row-wise structure allows the shifted correlation to be tied directly to the empirical risk.
For a general optimized set-input objective, however, the batch value Φ n ( f W ; S ) \Phi_n(f_W;S) Φ n ( f W ; S ) is not determined by the sum of the local replacement increments D t D_t D t . The optimizer defining Φ n \Phi_n Φ n may change after a single replacement, and D t D_t D t records only a mixed-path difference between two optimized batch values. Hence there is no fixed choice of η \eta η for which the remainder R η , C R_{\eta,C} R η , C vanishes, or is controlled by Φ n ( f W ; S ) \Phi_n(f_W; S) Φ n ( f W ; S ) , under the assumptions used in the main theorem. This is the obstruction to importing the ordinary shifted-Rademacher fast-rate argument directly. The point is not that no shifted argument can ever be designed for a special objective; rather, the telescoping identity alone does not provide the row-wise empirical-risk representation required by the ordinary-loss proof.
This subsection is included only as a comparison route and is not used in the main theorem. The main analysis controls the optimized validation–training gap directly and therefore avoids an explicit complexity term for the optimizer class. A different, more classical route is possible for supremum-type objectives whose inner values are averages of row-wise nonnegative losses. This route first pays a uniform-convergence cost over Π \Pi Π , then freezes the optimizer selected on the training sample and applies an ordinary-loss bound to the resulting augmented algorithmic output.
Assume that
Φ n ( f W ; S ) = sup π ∈ Π ψ n π ( f W ; S ) , ψ n π ( f W ; S ) : = 1 n ∑ i = 1 n ℓ π ( f W , Z i ) , (70) \Phi_n(f_W; S) = \sup_{\pi\in\Pi}\psi_n^\pi(f_W; S), \qquad\psi_n^\pi(f_W; S) := \frac{1}{n}\sum_{i=1}^{n}\ell_\pi(f_W, Z_i), \tag*{(70)} Φ n ( f W ; S ) = π ∈ Π sup ψ n π ( f W ; S ) , ψ n π ( f W ; S ) := n 1 i = 1 ∑ n ℓ π ( f W , Z i ) , (70)
where each row-wise loss takes values in [ 0 , 1 ] [0,1] [ 0 , 1 ] . Assume that a measurable training-side maximizer exists, and write
π ^ = π ^ ( W , S ) ∈ arg max π ∈ Π ψ n π ( f W ; S ) . \widehat{\pi} = \widehat{\pi}(W,S) \in\arg\max_{\pi\in\Pi}\psi_n^\pi(f_W;S). π = π ( W , S ) ∈ arg π ∈ Π max ψ n π ( f W ; S ) .
For readability, define the validation–training uniform-convergence modulus A Π ( f W ; S ′ , S ) : = sup π ∈ Π ∣ ψ n π ( f W ; S ′ ) − ψ n π ( f W ; S ) ∣ A_\Pi(f_W; S', S) := \sup_{\pi\in\Pi}\lvert\psi_n^\pi(f_W; S')-\psi_n^\pi(f_W;S)\rvert A Π ( f W ; S ′ , S ) := sup π ∈ Π ∣ ψ n π ( f W ; S ′ ) − ψ n π ( f W ; S )∣ . Also define
U C n Π , v t ( f W ) : = E [ A Π ( f W ; S ′ , S ) ] . \mathrm{UC}^{\Pi,\mathrm{vt}}_n(f_W) := \mathbb{E}\left[A_\Pi(f_W; S',S)\right]. UC n Π , vt ( f W ) := E [ A Π ( f W ; S ′ , S ) ] .
First, the direct uniform-convergence comparison gives the pointwise bound
∣ Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) ∣ ≤ A Π ( f W ; S ′ , S ) . \left\lvert\Phi_n(f_W; S')-\Phi_n(f_W; S)\right\rvert\le A_\Pi(f_W; S',S). ∣ Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) ∣ ≤ A Π ( f W ; S ′ , S ) .
Consequently,
∣ E [ Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) ] ∣ ≤ E ∣ Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) ∣ ≤ U C n Π , v t ( f W ) . (71) \left\lvert\mathbb{E}\left[\Phi_n(f_W; S')-\Phi_n(f_W; S)\right]\right\rvert\le\mathbb{E}\left\lvert\Phi_n(f_W; S')-\Phi_n(f_W; S)\right\rvert\le\mathrm{UC}^{\Pi,\mathrm{vt}}_n(f_W). \tag*{(71)} ∣ E [ Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) ] ∣ ≤ E ∣ Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) ∣ ≤ UC n Π , vt ( f W ) . (71)
Indeed, for every π \pi π , ψ n π ( f W ; S ′ ) ≤ ψ n π ( f W ; S ) + A Π ( f W ; S ′ , S ) \psi_n^\pi(f_W; S') \le\psi_n^\pi(f_W; S)+A_\Pi(f_W; S',S) ψ n π ( f W ; S ′ ) ≤ ψ n π ( f W ; S ) + A Π ( f W ; S ′ , S ) , and taking the supremum over π \pi π gives Φ n ( f W ; S ′ ) ≤ Φ n ( f W ; S ) + A Π ( f W ; S ′ , S ) \Phi_n(f_W; S')\le\Phi_n(f_W; S)+A_\Pi(f_W; S',S) Φ n ( f W ; S ′ ) ≤ Φ n ( f W ; S ) + A Π ( f W ; S ′ , S ) . The reverse inequality follows by swapping S S S and S ′ S' S ′ .
The following looser decomposition is nevertheless useful for comparison with ordinary-loss fast-rate bounds, because it isolates the validation–training gap of the optimizer selected on the training sample.
Proposition 5 (Freezing the training-side optimizer after paying uniform convergence). Under (67),
Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) ≤ 2 A Π ( f W ; S ′ , S ) \Phi_n(f_W; S')-\Phi_n(f_W; S) \le2A_\Pi(f_W; S',S) Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) ≤ 2 A Π ( f W ; S ′ , S ) + ( ψ n π ^ ( f W ; S ′ ) − ψ n π ^ ( f W ; S ) ) . (72) \qquad+\left(\psi_n^{\widehat{\pi}}(f_W; S')-\psi_n^{\widehat{\pi}}(f_W; S)\right). \tag*{(72)} + ( ψ n π ( f W ; S ′ ) − ψ n π ( f W ; S ) ) . (72) Consequently,
E [ Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) ] ≤ 2 U C n Π , v t ( f W ) + E [ ψ n π ^ ( f W ; S ′ ) − ψ n π ^ ( f W ; S ) ] . (73) \mathbb{E}\left[\Phi_n(f_W; S')-\Phi_n(f_W; S)\right] \le2\mathrm{UC}^{\Pi,\mathrm{vt}}_n(f_W)+\mathbb{E}\left[\psi_n^{\widehat{\pi}}(f_W; S')-\psi_n^{\widehat{\pi}}(f_W; S)\right]. \tag*{(73)} E [ Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) ] ≤ 2 UC n Π , vt ( f W ) + E [ ψ n π ( f W ; S ′ ) − ψ n π ( f W ; S ) ] . (73)
Proof. Since Φ n ( f W ; S ) = ψ n π ^ ( f W ; S ) \Phi_n(f_W; S)=\psi_n^{\widehat{\pi}}(f_W; S) Φ n ( f W ; S ) = ψ n π ( f W ; S ) ,
Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) = ( sup π ∈ Π ψ n π ( f W ; S ′ ) − ψ n π ^ ( f W ; S ′ ) ) \Phi_n(f_W; S')-\Phi_n(f_W; S)=\left(\sup_{\pi\in\Pi}\psi_n^\pi(f_W; S')-\psi_n^{\widehat{\pi}}(f_W; S')\right) Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) = ( π ∈ Π sup ψ n π ( f W ; S ′ ) − ψ n π ( f W ; S ′ ) )
+ ( ψ n π ^ ( f W ; S ′ ) − ψ n π ^ ( f W ; S ) ) . \qquad+\left(\psi_n^{\widehat{\pi}}(f_W; S')-\psi_n^{\widehat{\pi}}(f_W; S)\right). + ( ψ n π ( f W ; S ′ ) − ψ n π ( f W ; S ) ) .
For the first bracket, for every π ∈ Π \pi\in\Pi π ∈ Π ,
ψ n π ( f W ; S ′ ) − ψ n π ^ ( f W ; S ′ ) = ( ψ n π ( f W ; S ′ ) − ψ n π ( f W ; S ) ) + ( ψ n π ( f W ; S ) − ψ n π ^ ( f W ; S ) ) + ( ψ n π ^ ( f W ; S ) − ψ n π ^ ( f W ; S ′ ) ) ≤ A Π ( f W ; S ′ , S ) + 0 + A Π ( f W ; S ′ , S ) . \psi_n^\pi(f_W;S')-\psi_n^{\hat{\pi}}(f_W;S')=\left(\psi_n^\pi(f_W;S')-\psi_n^\pi(f_W;S)\right)
+\left(\psi_n^\pi(f_W;S)-\psi_n^{\hat{\pi}}(f_W;S)\right)
+\left(\psi_n^{\hat{\pi}}(f_W;S)-\psi_n^{\hat{\pi}}(f_W;S')\right)
\le A_\Pi(f_W;S',S)+0+A_\Pi(f_W;S',S). ψ n π ( f W ; S ′ ) − ψ n π ^ ( f W ; S ′ ) = ( ψ n π ( f W ; S ′ ) − ψ n π ( f W ; S ) ) + ( ψ n π ( f W ; S ) − ψ n π ^ ( f W ; S ) ) + ( ψ n π ^ ( f W ; S ) − ψ n π ^ ( f W ; S ′ ) ) ≤ A Π ( f W ; S ′ , S ) + 0 + A Π ( f W ; S ′ , S ) .
where the middle term is nonpositive by the optimality of π ^ \hat{\pi} π ^ on S S S . Taking the supremum over π \pi π proves (72) , and taking expectations gives (73) .
To interpret the remaining term as an ordinary validation–training gap, augment the algorithmic output by the selected optimizer: A ⋆ : = ( W , π ^ ) A^\star:= (W,\hat{\pi}) A ⋆ := ( W , π ^ ) . Define the ordinary frozen loss
ℓ ⋆ ( A ⋆ , z ) : = ℓ π ^ ( f W , z ) . \ell^\star(A^\star,z):=\ell_{\hat{\pi}}(f_W,z). ℓ ⋆ ( A ⋆ , z ) := ℓ π ^ ( f W , z ) .
Then
ψ n π ^ ( f W ; S ) = 1 n ∑ i = 1 n ℓ ⋆ ( A ⋆ , Z ~ i , U i ) , ψ n π ^ ( f W ; S ′ ) = 1 n ∑ i = 1 n ℓ ⋆ ( A ⋆ , Z ~ i , U ‾ i ) . \psi_n^{\hat{\pi}}(f_W;S)=\frac{1}{n}\sum_{i=1}^{n}\ell^\star(A^\star,\widetilde{Z}_i,U_i),\qquad
\psi_n^{\hat{\pi}}(f_W;S')=\frac{1}{n}\sum_{i=1}^{n}\ell^\star(A^\star,\widetilde{Z}_i,\overline{U}_i). ψ n π ^ ( f W ; S ) = n 1 i = 1 ∑ n ℓ ⋆ ( A ⋆ , Z i , U i ) , ψ n π ^ ( f W ; S ′ ) = n 1 i = 1 ∑ n ℓ ⋆ ( A ⋆ , Z i , U i ) .
Thus the second term in (73) is the validation–training gap of the ordinary loss ℓ ⋆ \ell^\star ℓ ⋆ for the augmented output A ⋆ A^\star A ⋆ .
Corollary 13 (Uniform-convergence-plus-ordinary-fast-rate route). Assume (70) . Let
L n ⋆ : = E [ 1 n ∑ i = 1 n ℓ ⋆ ( A ⋆ , Z ~ i , U i ) ] = E [ Φ n ( f W ; S ) ] , L_n^\star:=\mathbb{E}\left[\frac{1}{n}\sum_{i=1}^{n}\ell^\star(A^\star,\widetilde{Z}_i,U_i)\right]=\mathbb{E}[\Phi_n(f_W;S)], L n ⋆ := E [ n 1 i = 1 ∑ n ℓ ⋆ ( A ⋆ , Z i , U i ) ] = E [ Φ n ( f W ; S )] , L μ ⋆ : = E [ 1 n ∑ i = 1 n ℓ ⋆ ( A ⋆ , Z ~ i , U ‾ i ) ] = E [ ψ n π ^ ( f W ; S ′ ) ] . L_\mu^\star:=\mathbb{E}\left[\frac{1}{n}\sum_{i=1}^{n}\ell^\star(A^\star,\widetilde{Z}_i,\overline{U}_i)\right]=\mathbb{E}[\psi_n^{\hat{\pi}}(f_W;S')]. L μ ⋆ := E [ n 1 i = 1 ∑ n ℓ ⋆ ( A ⋆ , Z i , U i ) ] = E [ ψ n π ^ ( f W ; S ′ )] . Let L t ⋆ , + L_t^{\star,+} L t ⋆ , + denote the single-index loss-pair projection associated with the ordinary loss ℓ ⋆ \ell^\star ℓ ⋆ in the sense of Wang and Mao [39 ] , and set I t ⋆ : = I ( L t ⋆ , + ; U t ∣ Z ~ ) I_t^\star:=I(L_t^{\star,+};U_t\mid\widetilde{Z}) I t ⋆ := I ( L t ⋆ , + ; U t ∣ Z ) under the conditional supersample convention used in this paper. Suppose that the ordinary-loss fast-rate theorem applies to the augmented ordinary-loss problem ( A ⋆ , ℓ ⋆ ) (A^\star,\ell^\star) ( A ⋆ , ℓ ⋆ ) , giving
L μ ⋆ − L n ⋆ ≤ 4 n ∑ t = 1 n I t ⋆ + 4 L n ⋆ ⋅ 1 n ∑ t = 1 n I t ⋆ . (74) L_\mu^\star-L_n^\star\le\frac{4}{n}\sum_{t=1}^{n}I_t^\star+4\sqrt{L_n^\star\cdot\frac{1}{n}\sum_{t=1}^{n}I_t^\star}. \tag*{(74)} L μ ⋆ − L n ⋆ ≤ n 4 t = 1 ∑ n I t ⋆ + 4 L n ⋆ ⋅ n 1 t = 1 ∑ n I t ⋆ . (74) Then
E [ Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) ] ≤ 2 U C n Π , v t ( f W ) + 4 n ∑ t = 1 n I t ⋆ + 4 E [ Φ n ( f W ; S ) ] ⋅ 1 n ∑ t = 1 n I t ⋆ . (75) \mathbb{E}\left[\Phi_n(f_W;S')-\Phi_n(f_W;S)\right]\le2\,\mathrm{UC}_n^{\Pi,\mathrm{vt}}(f_W)+\frac{4}{n}\sum_{t=1}^{n}I_t^\star
+4\sqrt{\mathbb{E}[\Phi_n(f_W;S)]\cdot\frac{1}{n}\sum_{t=1}^{n}I_t^\star}. \tag*{(75)} E [ Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) ] ≤ 2 UC n Π , vt ( f W ) + n 4 t = 1 ∑ n I t ⋆ + 4 E [ Φ n ( f W ; S )] ⋅ n 1 t = 1 ∑ n I t ⋆ . (75) Proof. By Proposition 5, it remains to bound the ordinary validation–training gap of the frozen loss ℓ ⋆ \ell^\star ℓ ⋆ , namely L μ ⋆ − L n ⋆ L_\mu^\star-L_n^\star L μ ⋆ − L n ⋆ . Applying (74) and using L n ⋆ = E [ Φ n ( f W ; S ) ] L_n^\star=\mathbb{E}[\Phi_n(f_W;S)] L n ⋆ = E [ Φ n ( f W ; S )] gives (75) .
The bound (75) is included only as a comparison with the ordinary-loss fast-rate route after the training-side optimizer has been frozen. It should not be interpreted as an improvement over the direct uniform-convergence inequality (71) . The main theorem follows a different route: it controls the optimized validation–training gap directly by Theorem 1, avoiding an explicit uniform-convergence complexity term for the optimizer class Π \Pi Π .
A smaller validation subset
In some applications it is useful to compare training performance to a smaller validation subset rather than to the full holdout sample. The following expectation-level result applies to finite supremum-type objectives.
Assume that, for every sample T T T of size m m m ,
Φ m ( f W ; T ) : = sup π ∈ Π 1 m ∑ z ∈ T g π ( f W , z ) , 0 ≤ g π ( f W , z ) ≤ B , (76) \Phi_m(f_W;T) := \sup_{\pi\in\Pi} \frac{1}{m}\sum_{z\in T} g_\pi(f_W,z), \qquad0 \le g_\pi(f_W,z) \le B, \tag*{(76)} Φ m ( f W ; T ) := π ∈ Π sup m 1 z ∈ T ∑ g π ( f W , z ) , 0 ≤ g π ( f W , z ) ≤ B , (76)
where 1 ≤ ∣ Π ∣ < ∞ 1 \le|\Pi| < \infty 1 ≤ ∣Π∣ < ∞ . Let S ′ = ( Z 1 ′ , … , Z n ′ ) S' = (Z'_1,\ldots,Z'_n) S ′ = ( Z 1 ′ , … , Z n ′ ) , and let J ⊂ [ n ] J \subset[n] J ⊂ [ n ] be sampled uniformly without replacement among all subsets of size n ′ n' n ′ , where 1 ≤ n ′ < n 1 \le n' < n 1 ≤ n ′ < n , independently of all other randomness. Write S J ′ = ( Z j ′ ) j ∈ J S'_J = (Z'_j)_{j\in J} S J ′ = ( Z j ′ ) j ∈ J .
Theorem 3 (Small validation subset bound). Under (76) ,
E [ Φ n ′ ( f W ; S J ′ ) − Φ n ( f W ; S ) ] ≤ E [ Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) ] + 2 B ( n − n ′ + 1 ) ( log ∣ Π ∣ + 1 ) 2 n n ′ . (77) \mathbb{E}\left[\Phi_{n'}(f_W;S'_J)-\Phi_n(f_W;S)\right]
\le\mathbb{E}\left[\Phi_n(f_W;S')-\Phi_n(f_W;S)\right]
+2B\sqrt{\frac{(n-n'+1)(\log|\Pi|+1)}{2nn'}}. \tag*{(77)} E [ Φ n ′ ( f W ; S J ′ ) − Φ n ( f W ; S ) ] ≤ E [ Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) ] + 2 B 2 n n ′ ( n − n ′ + 1 ) ( log ∣Π∣ + 1 ) . (77) If E [ Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) ] ≤ IT n ( f W ) \mathbb{E}[\Phi_n(f_W;S')-\Phi_n(f_W;S)] \le\operatorname{IT}_n(f_W) E [ Φ n ( f W ; S ′ ) − Φ n ( f W ; S )] ≤ IT n ( f W ) , then
E [ Φ n ′ ( f W ; S J ′ ) − Φ n ( f W ; S ) ] ≤ IT n ( f W ) + 2 B ( n − n ′ + 1 ) ( log ∣ Π ∣ + 1 ) 2 n n ′ . (78) \mathbb{E}\left[\Phi_{n'}(f_W;S'_J)-\Phi_n(f_W;S)\right]
\le\operatorname{IT}_n(f_W)+2B\sqrt{\frac{(n-n'+1)(\log|\Pi|+1)}{2nn'}}. \tag*{(78)} E [ Φ n ′ ( f W ; S J ′ ) − Φ n ( f W ; S ) ] ≤ IT n ( f W ) + 2 B 2 n n ′ ( n − n ′ + 1 ) ( log ∣Π∣ + 1 ) . (78)
Proof. For each π ∈ Π \pi\in\Pi π ∈ Π , define
P n g π : = 1 n ∑ i = 1 n g π ( f W , Z i ′ ) , P J g π : = 1 n ′ ∑ j ∈ J g π ( f W , Z j ′ ) . P_n g_\pi:= \frac{1}{n}\sum_{i=1}^{n}g_\pi(f_W,Z'_i), \qquad P_Jg_\pi:= \frac{1}{n'}\sum_{j\in J}g_\pi(f_W,Z'_j). P n g π := n 1 i = 1 ∑ n g π ( f W , Z i ′ ) , P J g π := n ′ 1 j ∈ J ∑ g π ( f W , Z j ′ ) .
Then
Φ n ( f W ; S ′ ) = sup π ∈ Π P n g π , Φ n ′ ( f W ; S J ′ ) = sup π ∈ Π P J g π . \Phi_n(f_W;S')=\sup_{\pi\in\Pi}P_ng_\pi,\qquad\Phi_{n'}(f_W;S'_J)=\sup_{\pi\in\Pi}P_Jg_\pi. Φ n ( f W ; S ′ ) = π ∈ Π sup P n g π , Φ n ′ ( f W ; S J ′ ) = π ∈ Π sup P J g π .
Therefore,
Φ n ′ ( f W ; S J ′ ) − Φ n ( f W ; S ′ ) = sup π ∈ Π P J g π − sup π ∈ Π P n g π ≤ sup π ∈ Π ( P J − P n ) g π . (79) \Phi_{n'}(f_W;S'_J)-\Phi_n(f_W;S')
=\sup_{\pi\in\Pi}P_Jg_\pi-\sup_{\pi\in\Pi}P_ng_\pi
\le\sup_{\pi\in\Pi}(P_J-P_n)g_\pi. \tag*{(79)} Φ n ′ ( f W ; S J ′ ) − Φ n ( f W ; S ′ ) = π ∈ Π sup P J g π − π ∈ Π sup P n g π ≤ π ∈ Π sup ( P J − P n ) g π . (79)
Adding and subtracting Φ n ( f W ; S ′ ) \Phi_n(f_W;S') Φ n ( f W ; S ′ ) , taking expectations, and using (79) gives
E [ Φ n ′ ( f W ; S J ′ ) − Φ n ( f W ; S ) ] ≤ E [ Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) ] + E [ sup π ∈ Π ( P J − P n ) g π ] . (80) \mathbb{E}\left[\Phi_{n'}(f_W;S'_J)-\Phi_n(f_W;S)\right]
\le\mathbb{E}\left[\Phi_n(f_W;S')-\Phi_n(f_W;S)\right]
+\mathbb{E}\left[\sup_{\pi\in\Pi}(P_J-P_n)g_\pi\right]. \tag*{(80)} E [ Φ n ′ ( f W ; S J ′ ) − Φ n ( f W ; S ) ] ≤ E [ Φ n ( f W ; S ′ ) − Φ n ( f W ; S ) ] + E [ π ∈ Π sup ( P J − P n ) g π ] . (80)
It remains to bound the last term. Conditional on S ′ S' S ′ and f W f_W f W , for each fixed π \pi π , the values g π ( f W , Z 1 ′ ) , … , g π ( f W , Z n ′ ) g_\pi(f_W,Z'_1),\ldots,g_\pi(f_W,Z'_n) g π ( f W , Z 1 ′ ) , … , g π ( f W , Z n ′ ) form a finite population contained in [ 0 , B ] [0,B] [ 0 , B ] , and P J g π P_Jg_\pi P J g π is the average of a uniform sample of size n ′ n' n ′ drawn without replacement from this population. Hence the Hoeffding inequality gives, for every ϵ > 0 \epsilon>0 ϵ > 0 ,
Pr J ( ( P J − P n ) g π ≥ ϵ ∣ S ′ , f W ) ≤ exp ( − 2 n n ′ ϵ 2 ( n − n ′ + 1 ) B 2 ) . \Pr_J\left((P_J-P_n)g_\pi\ge\epsilon\mid S',f_W\right)
\le\exp\left(-\frac{2nn'\epsilon^2}{(n-n'+1)B^2}\right). J Pr ( ( P J − P n ) g π ≥ ϵ ∣ S ′ , f W ) ≤ exp ( − ( n − n ′ + 1 ) B 2 2 n n ′ ϵ 2 ) .
By a union bound over Π \Pi Π ,
Pr J ( sup π ∈ Π ( P J − P n ) g π ≥ ϵ ∣ S ′ , f W ) ≤ ∣ Π ∣ exp ( − 2 n n ′ ϵ 2 ( n − n ′ + 1 ) B 2 ) . \Pr_J\left(\sup_{\pi\in\Pi}(P_J-P_n)g_\pi\ge\epsilon\mid S',f_W\right)
\le|\Pi|\exp\left(-\frac{2nn'\epsilon^2}{(n-n'+1)B^2}\right). J Pr ( π ∈ Π sup ( P J − P n ) g π ≥ ϵ ∣ S ′ , f W ) ≤ ∣Π∣ exp ( − ( n − n ′ + 1 ) B 2 2 n n ′ ϵ 2 ) .
Let Y : = sup π ∈ Π ( P J − P n ) g π Y:=\sup_{\pi\in\Pi}(P_J-P_n)g_\pi Y := sup π ∈ Π ( P J − P n ) g π , c : = 2 n n ′ ( n − n ′ + 1 ) B 2 c:=\frac{2nn'}{(n-n'+1)B^2} c := ( n − n ′ + 1 ) B 2 2 n n ′ . Since Y ≤ Y + Y\le Y_+ Y ≤ Y + , integrating the tail yields
E [ Y ∣ S ′ , f W ] ≤ ∫ 0 ∞ min { 1 , ∣ Π ∣ e − c ϵ 2 } d ϵ ≤ 2 log ∣ Π ∣ + 1 c . \mathbb{E}[Y\mid S',f_W]\le\int_0^\infty\min\{1,|\Pi|e^{-c\epsilon^2}\}\,d\epsilon
\le2\sqrt{\frac{\log|\Pi|+1}{c}}. E [ Y ∣ S ′ , f W ] ≤ ∫ 0 ∞ min { 1 , ∣Π∣ e − c ϵ 2 } d ϵ ≤ 2 c log ∣Π∣ + 1 .
Thus
E [ sup π ∈ Π ( P J − P n ) g π ∣ S ′ , f W ] ≤ 2 B ( n − n ′ + 1 ) ( log ∣ Π ∣ + 1 ) 2 n n ′ . \mathbb{E}\left[\sup_{\pi\in\Pi}(P_J-P_n)g_\pi\mid S',f_W\right]
\le2B\sqrt{\frac{(n-n'+1)(\log|\Pi|+1)}{2nn'}}. E [ π ∈ Π sup ( P J − P n ) g π ∣ S ′ , f W ] ≤ 2 B 2 n n ′ ( n − n ′ + 1 ) ( log ∣Π∣ + 1 ) .
Taking expectation over S ′ S' S ′ and f W f_W f W , and substituting this bound into (80) , proves (77) . The IT bound (78) follows by replacing the full validation–training term by IT n ( f W ) \operatorname{IT}_n(f_W) IT n ( f W ) .
The additional term satisfies n − n ′ + 1 n n ′ = 1 n ′ − 1 n + 1 n n ′ \frac{n-n'+1}{nn'}=\frac{1}{n'}-\frac{1}{n}+\frac{1}{nn'} n n ′ n − n ′ + 1 = n ′ 1 − n 1 + n n ′ 1 . Since n ′ < n n'<n n ′ < n , this is within a universal constant factor of 1 n ′ − 1 n \frac{1}{n'}-\frac{1}{n} n ′ 1 − n 1 . Thus the correction term is of order B ( 1 n ′ − 1 n ) ( log ∣ Π ∣ + 1 ) B\sqrt{\left(\frac{1}{n'}-\frac{1}{n}\right)}(\log|\Pi|+1) B ( n ′ 1 − n 1 ) ( log ∣Π∣ + 1 ) up to constants. For moderate ∣ Π ∣ |\Pi| ∣Π∣ , this can be substantially smaller than a crude deterministic subset-size comparison when n ′ n' n ′ is not too small.
The balanced supersample removes explicit Π \Pi Π -complexity from the validation–training bound, whereas the current asymmetric-ratio extension reintroduces such complexity. Finiteness of Π \Pi Π in this extension is not fundamental: replacing the union bound by uniform convergence can replace log ∣ Π ∣ \log|\Pi| log ∣Π∣ , under suitable boundedness and measurability, by covering entropy, Rademacher complexity, or another empirical-process complexity.
Pseudo-dimension control for binned multicalibration
We give a scale-sensitive control for binned multicalibration witnesses. There are two different maps from prediction values to discretized objects in this appendix, and they play different roles.
First, q B q_B q B is the binning map that is part of the binned MC objective itself. It tells us which score bin a prediction belongs to. Second, Q δ Q_\delta Q δ is an auxiliary quantizer introduced only for the proof. It will be used to replace the real-valued prediction inside the residual Y − f ( x ) Y-f(x) Y − f ( x ) by a finite-valued approximation.
Throughout the deterministic statements below, f f f denotes a fixed predictor. After these deterministic bounds are established, we apply them to the random learned predictor f W f_W f W .
Let 0 = τ 0 < τ 1 < ⋯ < τ B = 1 0=\tau_0<\tau_1<\cdots<\tau_B=1 0 = τ 0 < τ 1 < ⋯ < τ B = 1 be fixed bin endpoints, and set I 1 = [ τ 0 , τ 1 ] I_1=[\tau_0,\tau_1] I 1 = [ τ 0 , τ 1 ] , I b = ( τ b − 1 , τ b ] I_b=(\tau_{b-1},\tau_b] I b = ( τ b − 1 , τ b ] (b = 2 , … , B b=2,\ldots,B b = 2 , … , B ). Define the corresponding bin index map q B : [ 0 , 1 ] → [ B ] q_B:[0,1]\to[B] q B : [ 0 , 1 ] → [ B ] by q B ( t ) = b ⟺ t ∈ I b q_B(t)=b\Longleftrightarrow t\in I_b q B ( t ) = b ⟺ t ∈ I b . Equivalently, q B ( t ) = 1 + ∑ k = 1 B − 1 1 { t > τ k } q_B(t)=1+\sum_{k=1}^{B-1}\mathbf{1}\{t>\tau_k\} q B ( t ) = 1 + ∑ k = 1 B − 1 1 { t > τ k } .
In the notation of the general MC objective, the binned score witnesses are h b ( v ) = 1 { q B ( v ) = b } h_b(v)=\mathbf{1}\{q_B(v)=b\} h b ( v ) = 1 { q B ( v ) = b } , b ∈ [ B ] b\in[B] b ∈ [ B ] . Hence, for a dataset S = { ( X j , Y j ) } j = 1 n S=\{(X_j,Y_j)\}_{j=1}^n S = {( X j , Y j ) } j = 1 n , the binned multicalibration objective is
Φ M C , B ( f ; S ) = sup c ∈ C , b ∈ [ B ] , η ∈ { ± 1 } η 1 n ∑ j = 1 n c ( X j ) 1 { q B ( f ( X j ) ) = b } ( Y j − f ( X j ) ) . \Phi_{\mathrm{MC},B}(f;S)=\sup_{c\in\mathcal{C},\,b\in[B],\,\eta\in\{\pm1\}}\eta\frac{1}{n}\sum_{j=1}^n c(X_j)\mathbf{1}\{q_B(f(X_j))=b\}(Y_j-f(X_j)). Φ MC , B ( f ; S ) = c ∈ C , b ∈ [ B ] , η ∈ { ± 1 } sup η n 1 j = 1 ∑ n c ( X j ) 1 { q B ( f ( X j )) = b } ( Y j − f ( X j )) .
Assume throughout this appendix that Y ∈ [ 0 , 1 ] Y\in[0,1] Y ∈ [ 0 , 1 ] , f ( x ) ∈ [ 0 , 1 ] f(x)\in[0,1] f ( x ) ∈ [ 0 , 1 ] , and 0 ≤ c ( x ) ≤ 1 0\le c(x)\le1 0 ≤ c ( x ) ≤ 1 for all c ∈ C c\in\mathcal{C} c ∈ C .
The same prediction value f ( X j ) f(X_j) f ( X j ) appears in two different places in this objective: 1 { q B ( f ( X j ) ) = b } \mathbf{1}\{q_B(f(X_j))=b\} 1 { q B ( f ( X j )) = b } and Y j − f ( X j ) Y_j-f(X_j) Y j − f ( X j ) . These two occurrences cannot be discretized in the same way. The first one is the score-witness part h b ( f ( X j ) ) h_b(f(X_j)) h b ( f ( X j )) , namely the bin indicator. It is discontinuous as a function of f ( X j ) f(X_j) f ( X j ) , so replacing the prediction by a nearby rounded value can change the bin whenever the prediction is close to a bin boundary. Without a margin assumption, this can happen for many sample points and does not give a deterministic small approximation error.
By contrast, the second occurrence is inside the residual Y j − f ( X j ) Y_j-f(X_j) Y j − f ( X j ) . This part is Lipschitz in f ( X j ) f(X_j) f ( X j ) : if f ( X j ) f(X_j) f ( X j ) is replaced by a δ \delta δ -accurate quantized value, the summand changes by at most δ \delta δ . Therefore we keep the bin index exactly, using q B q_B q B , and quantize only the residual value, using the auxiliary map Q δ Q_\delta Q δ .
Let Z ~ ∈ Z n × 2 \widetilde{Z}\in\mathcal{Z}^{n\times2} Z ∈ Z n × 2 be the supersample, with entries Z ~ i , j = ( X ~ i , j , Y ~ i , j ) \widetilde{Z}_{i,j}=(\widetilde{X}_{i,j},\widetilde{Y}_{i,j}) Z i , j = ( X i , j , Y i , j ) , i ∈ [ n ] i\in[n] i ∈ [ n ] , j ∈ { 0 , 1 } j\in\{0,1\} j ∈ { 0 , 1 } . For a selector vector v ∈ { 0 , 1 } n v\in\{0,1\}^n v ∈ { 0 , 1 } n , write Z v = { Z ~ i , v i } i = 1 n Z_v=\{\widetilde{Z}_{i,v_i}\}_{i=1}^n Z v = { Z i , v i } i = 1 n , Z ~ v = { Z ~ i , 1 − v i } i = 1 n \widetilde{Z}_v=\{\widetilde{Z}_{i,1-v_i}\}_{i=1}^n Z v = { Z i , 1 − v i } i = 1 n . In particular, S U = Z ~ U S_U=\widetilde{Z}_U S U = Z U , S U ‾ = Z ~ U ‾ S_{\overline{U}}=\widetilde{Z}_{\overline{U}} S U = Z U .
We now introduce the auxiliary discretization. Its purpose is not to approximate the binning map. Rather, it turns the real-valued residual prediction f ( X ~ i , j ) f(\widetilde{X}_{i,j}) f ( X i , j ) into a finite-valued table, so that the learned supersample table has a pseudo-dimension entropy bound. The interval-threshold form below is used only in the pseudo-dimension counting step: it ensures that Q δ ( f ( x ) ) Q_\delta(f(x)) Q δ ( f ( x )) is determined by finitely many threshold comparisons of f ( x ) f(x) f ( x ) .
We use a fixed monotone interval quantizer Q δ : [ 0 , 1 ] → [ 0 , 1 ] Q_\delta:[0,1]\to[0,1] Q δ : [ 0 , 1 ] → [ 0 , 1 ] . More precisely, there are thresholds 0 = γ 0 < γ 1 < ⋯ < γ M δ < γ M δ + 1 = 1 0=\gamma_0<\gamma_1<\cdots<\gamma_{M_\delta}<\gamma_{M_\delta+1}=1 0 = γ 0 < γ 1 < ⋯ < γ M δ < γ M δ + 1 = 1 and grid values g 0 , … , g M δ ∈ [ 0 , 1 ] g_0,\ldots,g_{M_\delta}\in[0,1] g 0 , … , g M δ ∈ [ 0 , 1 ] such that M δ + 1 ≤ ⌈ 1 δ ⌉ + 1 M_\delta+1\le\left\lceil\frac{1}{\delta}\right\rceil+1 M δ + 1 ≤ ⌈ δ 1 ⌉ + 1 and Q δ ( t ) = g r Q_\delta(t)=g_r Q δ ( t ) = g r for t ∈ J r t\in J_r t ∈ J r , where J 0 = [ γ 0 , γ 1 ] J_0=[\gamma_0,\gamma_1] J 0 = [ γ 0 , γ 1 ] , J r = ( γ r , γ r + 1 ] J_r=(\gamma_r,\gamma_{r+1}] J r = ( γ r , γ r + 1 ] (r = 1 , … , M δ r=1,\ldots,M_\delta r = 1 , … , M δ ). We assume ∣ Q δ ( t ) − t ∣ ≤ δ |Q_\delta(t)-t|\le\delta ∣ Q δ ( t ) − t ∣ ≤ δ for all t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ] .
With Q δ Q_{\delta} Q δ defined, the reason for not using q B ( Q δ ( f ( x ) ) ) q_B(Q_{\delta}(f(x))) q B ( Q δ ( f ( x ))) is now explicit: the map q B q_B q B is discontinuous, while the approximation guarantee for Q δ Q_{\delta} Q δ is only metric. A prediction f ( x ) f(x) f ( x ) and its quantized value Q δ ( f ( x ) ) Q_{\delta}(f(x)) Q δ ( f ( x )) can lie on different sides of a bin boundary. Thus Q δ Q_{\delta} Q δ is used only in the residual, not inside the bin indicator.
Remark 3 (A concrete quantizer). One convenient choice is the nearest uniform-grid quantizer. Let M = ⌈ 1 / δ ⌉ M = \lceil1/\delta\rceil M = ⌈ 1/ δ ⌉ , g r = r / M g_r = r/M g r = r / M , r = 0 , … , M r = 0,\ldots,M r = 0 , … , M . Define the midpoints γ r = ( g r − 1 + g r ) / 2 \gamma_r = (g_{r-1}+g_r)/2 γ r = ( g r − 1 + g r ) /2 , r = 1 , … , M r = 1,\ldots,M r = 1 , … , M . Then set
Q δ ( t ) = { g 0 , t ∈ [ 0 , γ 1 ] , g r , t ∈ ( γ r , γ r + 1 ] , r = 1 , … , M − 1 , g M , t ∈ ( γ M , 1 ] . Q_{\delta}(t)=
\begin{cases}
g_0, & t \in[0,\gamma_1],\\
g_r, & t \in(\gamma_r,\gamma_{r+1}],\quad r=1,\ldots,M-1,\\
g_M, & t \in(\gamma_M,1].
\end{cases} Q δ ( t ) = ⎩ ⎨ ⎧ g 0 , g r , g M , t ∈ [ 0 , γ 1 ] , t ∈ ( γ r , γ r + 1 ] , r = 1 , … , M − 1 , t ∈ ( γ M , 1 ] .
This quantizer has M + 1 ≤ ⌈ 1 / δ ⌉ + 1 M+1 \le\lceil1/\delta\rceil+1 M + 1 ≤ ⌈ 1/ δ ⌉ + 1 output values and M ≤ ⌈ 1 / δ ⌉ M \le\lceil1/\delta\rceil M ≤ ⌈ 1/ δ ⌉ internal thresholds. Moreover, ∣ Q δ ( t ) − t ∣ ≤ 1 / ( 2 M ) ≤ δ |Q_{\delta}(t)-t| \le1/(2M) \le\delta ∣ Q δ ( t ) − t ∣ ≤ 1/ ( 2 M ) ≤ δ for every t ∈ [ 0 , 1 ] t \in[0,1] t ∈ [ 0 , 1 ] .
For fixed supersample inputs X ~ \widetilde{X} X , define the discretized supersample prediction table
A δ , B ( f , X ~ ) = ( ( a i , j b i n ( f ) , a i , j v a l ( f ) ) ) i ∈ [ n ] , j ∈ { 0 , 1 } , A_{\delta,B}(f,\widetilde{X})=\left(\left(a^{\mathrm{bin}}_{i,j}(f),a^{\mathrm{val}}_{i,j}(f)\right)\right)_{i\in[n],\,j\in\{0,1\}}, A δ , B ( f , X ) = ( ( a i , j bin ( f ) , a i , j val ( f ) ) ) i ∈ [ n ] , j ∈ { 0 , 1 } ,
where
a i , j b i n ( f ) = q B ( f ( X ~ i , j ) ) , a i , j v a l ( f ) = Q δ ( f ( X ~ i , j ) ) . a^{\mathrm{bin}}_{i,j}(f)=q_B(f(\widetilde{X}_{i,j})),\qquad a^{\mathrm{val}}_{i,j}(f)=Q_{\delta}(f(\widetilde{X}_{i,j})). a i , j bin ( f ) = q B ( f ( X i , j )) , a i , j val ( f ) = Q δ ( f ( X i , j )) .
Thus each supersample input stores two pieces of information: exact bin information and quantized residual-value information. The first component is used in the bin indicator, and the second component is used inside the residual. When f = f W f=f_W f = f W , we write simply A δ , B = A δ , B ( f W , X ~ ) A_{\delta,B}=A_{\delta,B}(f_W,\widetilde{X}) A δ , B = A δ , B ( f W , X ) .
For a generic table a a a , write its entries as a i , j b i n ∈ [ B ] a^{\mathrm{bin}}_{i,j}\in[B] a i , j bin ∈ [ B ] , a i , j v a l ∈ [ 0 , 1 ] a^{\mathrm{val}}_{i,j}\in[0,1] a i , j val ∈ [ 0 , 1 ] . The table a a a need not correspond to an actual predictor; the following display defines a finite-table surrogate objective. For a selector vector v ∈ { 0 , 1 } n v\in\{0,1\}^n v ∈ { 0 , 1 } n , the selected batch z ~ v \widetilde{z}_v z v uses the entries ( i , v i ) (i,v_i) ( i , v i ) . Define
Φ M C , B δ ( a ; z ~ v ) = sup c ∈ C , b ∈ [ B ] , η ∈ { ± 1 } η 1 n ∑ i = 1 n c ( X ~ i , v i ) 1 { a i , v i b i n = b } ( Y ~ i , v i − a i , v i v a l ) . \Phi^{\delta}_{\mathrm{MC},B}(a;\widetilde{z}_v)
=
\sup_{c\in\mathcal{C},\,b\in[B],\,\eta\in\{\pm1\}}
\eta\frac{1}{n}\sum_{i=1}^{n}c(\widetilde{X}_{i,v_i})\mathbf{1}\{a^{\mathrm{bin}}_{i,v_i}=b\}\left(\widetilde{Y}_{i,v_i}-a^{\mathrm{val}}_{i,v_i}\right). Φ MC , B δ ( a ; z v ) = c ∈ C , b ∈ [ B ] , η ∈ { ± 1 } sup η n 1 i = 1 ∑ n c ( X i , v i ) 1 { a i , v i bin = b } ( Y i , v i − a i , v i val ) .
In words, this is the original binned MC objective with the two occurrences of the prediction treated differently: the score-witness part uses the exact stored bin a b i n a^{\mathrm{bin}} a bin , while the residual uses the quantized stored value a v a l a^{\mathrm{val}} a val .
Lemma 12 (Binned MC discretization). For every predictor f f f , every supersample realization z ~ \widetilde{z} z , and every selector vector v ∈ { 0 , 1 } n v\in\{0,1\}^n v ∈ { 0 , 1 } n ,
∣ Φ M C , B ( f ; z ~ v ) − Φ M C , B δ ( A δ , B ( f , X ~ ) ; z ~ v ) ∣ ≤ δ . \left|\Phi_{\mathrm{MC},B}(f;\widetilde{z}_v)-\Phi^{\delta}_{\mathrm{MC},B}\left(A_{\delta,B}(f,\widetilde{X});\widetilde{z}_v\right)\right|\le\delta. Φ MC , B ( f ; z v ) − Φ MC , B δ ( A δ , B ( f , X ) ; z v ) ≤ δ . Consequently, for the learned predictor f W f_W f W ,
∣ E [ Φ M C , B ( f W ; S ~ U ) − Φ M C , B ( f W ; S U ) ] ∣ ≤ 2 δ + 2 2 I ( A δ , B ; U ∣ Z ~ ) n . \left|\mathbb{E}\left[\Phi_{\mathrm{MC},B}(f_W;\widetilde{S}_U)-\Phi_{\mathrm{MC},B}(f_W;S_U)\right]\right|
\le2\delta+2\sqrt{\frac{2I(A_{\delta,B};U\mid\widetilde{Z})}{n}}. E [ Φ MC , B ( f W ; S U ) − Φ MC , B ( f W ; S U ) ] ≤ 2 δ + 2 n 2 I ( A δ , B ; U ∣ Z ) .
Proof. Fix f f f , z ~ \widetilde{z} z , and v v v , and set a = A δ , B ( f , X ~ ) a=A_{\delta,B}(f,\widetilde{X}) a = A δ , B ( f , X ) . For a fixed witness ( c , b , η ) (c,b,\eta) ( c , b , η ) , the original witness average and the surrogate witness average have the same bin indicator, because a i , v i b i n = q B ( f ( X ~ i , v i ) ) a^{\mathrm{bin}}_{i,v_i}=q_B(f(\widetilde{X}_{i,v_i})) a i , v i bin = q B ( f ( X i , v i )) . They differ only through the replacement of f ( X ~ i , v i ) f(\widetilde{X}_{i,v_i}) f ( X i , v i ) by a i , v i v a l = Q δ ( f ( X ~ i , v i ) ) a^{\mathrm{val}}_{i,v_i}=Q_{\delta}(f(\widetilde{X}_{i,v_i})) a i , v i val = Q δ ( f ( X i , v i )) inside the residual. Hence
∣ η 1 n ∑ i = 1 n c ( X ~ i , v i ) 1 { a i , v i b i n = b } { f ( X ~ i , v i ) − a i , v i v a l } ∣ ≤ 1 n ∑ i = 1 n c ( X ~ i , v i ) 1 { a i , v i b i n = b } ∣ f ( X ~ i , v i ) − Q δ ( f ( X ~ i , v i ) ) ∣ ≤ δ . \begin{aligned}
\left|
\eta\frac{1}{n}\sum_{i=1}^{n}c(\widetilde{X}_{i,v_i})\mathbf{1}\{a^{\mathrm{bin}}_{i,v_i}=b\}
\left\{f(\widetilde{X}_{i,v_i})-a^{\mathrm{val}}_{i,v_i}\right\}
\right| \\
&\le\frac{1}{n}\sum_{i=1}^{n}c(\widetilde{X}_{i,v_i})\mathbf{1}\{a^{\mathrm{bin}}_{i,v_i}=b\}
\left|f(\widetilde{X}_{i,v_i})-Q_{\delta}(f(\widetilde{X}_{i,v_i}))\right| \\
&\le\delta.
\end{aligned} η n 1 i = 1 ∑ n c ( X i , v i ) 1 { a i , v i bin = b } { f ( X i , v i ) − a i , v i val } ≤ n 1 i = 1 ∑ n c ( X i , v i ) 1 { a i , v i bin = b } f ( X i , v i ) − Q δ ( f ( X i , v i )) ≤ δ .
The last inequality uses 0 ≤ c ≤ 1 0 \le c \le1 0 ≤ c ≤ 1 , the fact that the bin indicator is bounded by one, and the quantization guarantee. Taking the supremum over the same witness class on both sides and using the elementary supremum-difference inequality gives
∣ Φ M C , B ( f ; z ~ V ) − Φ M C , B δ ( A δ , B ( f , X ~ ) ; z ~ V ) ∣ ≤ δ . \left|\Phi_{\mathrm{MC},B}(f;\widetilde{z}_{V})-\Phi_{\mathrm{MC},B}^{\delta}(A_{\delta,B}(f,\widetilde{X});\widetilde{z}_{V})\right| \le\delta. Φ MC , B ( f ; z V ) − Φ MC , B δ ( A δ , B ( f , X ) ; z V ) ≤ δ .
Applying this approximation once to S U S_U S U and once to S U ‾ S_{\overline{U}} S U , with f = f W f=f_W f = f W , gives ∣ G M C , B ( f W ) − G M C , B δ ( A δ , B ) ∣ ≤ 2 δ \left|G_{\mathrm{MC},B}(f_W)-G_{\mathrm{MC},B}^{\delta}(A_{\delta,B})\right|\le2\delta G MC , B ( f W ) − G MC , B δ ( A δ , B ) ≤ 2 δ , where G M C , B ( f W ) = Φ M C , B ( f W ; S U ‾ ) − Φ M C , B ( f W ; S U ) G_{\mathrm{MC},B}(f_W)=\Phi_{\mathrm{MC},B}(f_W;S_{\overline{U}})-\Phi_{\mathrm{MC},B}(f_W;S_U) G MC , B ( f W ) = Φ MC , B ( f W ; S U ) − Φ MC , B ( f W ; S U ) and G M C , B δ ( A δ , B ) = Φ M C , B δ ( A δ , B ; S U ‾ ) − Φ M C , B δ ( A δ , B ; S U ) G_{\mathrm{MC},B}^{\delta}(A_{\delta,B})=\Phi_{\mathrm{MC},B}^{\delta}(A_{\delta,B};S_{\overline{U}})-\Phi_{\mathrm{MC},B}^{\delta}(A_{\delta,B};S_U) G MC , B δ ( A δ , B ) = Φ MC , B δ ( A δ , B ; S U ) − Φ MC , B δ ( A δ , B ; S U ) . Therefore, ∣ E G M C , B ( f W ) ∣ ≤ 2 δ + ∣ E G M C , B δ ( A δ , B ) ∣ \left|\mathbb{E}G_{\mathrm{MC},B}(f_W)\right|\le2\delta+\left|\mathbb{E}G_{\mathrm{MC},B}^{\delta}(A_{\delta,B})\right| ∣ E G MC , B ( f W ) ∣ ≤ 2 δ + E G MC , B δ ( A δ , B ) .
It remains to control the expectation of the discretized gap. Fix Z ~ = z ~ \widetilde{Z}=\widetilde{z} Z = z and a table A δ , B = a A_{\delta,B}=a A δ , B = a . Define
G a ( u ) = Φ M C , B δ ( a ; z ~ u ‾ ) − Φ M C , B δ ( a ; z ~ u ) , u ∈ { 0 , 1 } n . G_a(u)=\Phi_{\mathrm{MC},B}^{\delta}(a;\widetilde{z}_{\overline{u}})-\Phi_{\mathrm{MC},B}^{\delta}(a;\widetilde{z}_u),\qquad u\in\{0,1\}^{n}. G a ( u ) = Φ MC , B δ ( a ; z u ) − Φ MC , B δ ( a ; z u ) , u ∈ { 0 , 1 } n .
Under the reference law Q U = Unif ( { 0 , 1 } n ) Q_U=\operatorname{Unif}(\{0,1\}^{n}) Q U = Unif ({ 0 , 1 } n ) , the complement map u ↦ u ‾ u\mapsto\overline{u} u ↦ u is measure preserving and G a ( u ‾ ) = − G a ( u ) G_a(\overline{u})=-G_a(u) G a ( u ) = − G a ( u ) . Thus E Q U G a ( U ) = 0 \mathbb{E}_{Q_U}G_a(U)=0 E Q U G a ( U ) = 0 .
We next show that G a ( U ) G_a(U) G a ( U ) is sub-Gaussian under Q U Q_U Q U . If two selector vectors u u u and u ⊕ i u^{\oplus i} u ⊕ i differ only in coordinate i i i , then the selected batch changes in one coordinate. For the surrogate objective, every witness summand is bounded in absolute value by one: ∣ c ( X ~ i , j ) 1 { a i , j b i n = b } ( Y ~ i , j − a i , j v a l ) ∣ ≤ 1 \left|c(\widetilde{X}_{i,j})1\{a^{\mathrm{bin}}_{i,j}=b\}(\widetilde{Y}_{i,j}-a^{\mathrm{val}}_{i,j})\right|\le1 c ( X i , j ) 1 { a i , j bin = b } ( Y i , j − a i , j val ) ≤ 1 . Therefore one coordinate replacement changes Φ M C , B δ ( a ; ⋅ ) \Phi_{\mathrm{MC},B}^{\delta}(a;\cdot) Φ MC , B δ ( a ; ⋅ ) by at most 2 / n 2/n 2/ n . The complementary batch also changes in one coordinate, so ∣ G a ( u ) − G a ( u ⊕ i ) ∣ ≤ 4 / n \left|G_a(u)-G_a(u^{\oplus i})\right|\le4/n G a ( u ) − G a ( u ⊕ i ) ≤ 4/ n . By the bounded-differences Hoeffding lemma,
log E Q U exp { λ G a ( U ) } ≤ λ 2 8 ∑ i = 1 n ( 4 n ) 2 = 2 λ 2 n = λ 2 ( 2 / n ) 2 2 \log\mathbb{E}_{Q_U}\exp\{\lambda G_a(U)\}\le\frac{\lambda^2}{8}\sum_{i=1}^{n}\left(\frac{4}{n}\right)^2=\frac{2\lambda^2}{n}=\frac{\lambda^2(2/\sqrt{n})^2}{2} log E Q U exp { λ G a ( U )} ≤ 8 λ 2 i = 1 ∑ n ( n 4 ) 2 = n 2 λ 2 = 2 λ 2 ( 2/ n ) 2
for all λ ∈ R \lambda\in\mathbb{R} λ ∈ R .
Applying the conditional change-of-measure inequality to the actual conditional law P U ∣ A δ , B , Z ~ P_{U\mid A_{\delta,B},\widetilde{Z}} P U ∣ A δ , B , Z relative to Q U Q_U Q U , and then averaging over ( A δ , B , Z ~ ) (A_{\delta,B},\widetilde{Z}) ( A δ , B , Z ) , gives
∣ E G M C , B δ ( A δ , B ) ∣ ≤ 2 n 2 E D ( P U ∣ A δ , B , Z ~ ∥ Q U ) . \left|\mathbb{E}G_{\mathrm{MC},B}^{\delta}(A_{\delta,B})\right|\le\frac{2}{\sqrt{n}}\sqrt{2\mathbb{E}D\left(P_{U\mid A_{\delta,B},\widetilde{Z}}\middle\|Q_U\right)}. E G MC , B δ ( A δ , B ) ≤ n 2 2 E D ( P U ∣ A δ , B , Z Q U ) .
Since U U U is independent of Z ~ \widetilde{Z} Z , the reference law Q U Q_U Q U is P U ∣ Z ~ P_{U\mid\widetilde{Z}} P U ∣ Z . Therefore E D ( P U ∣ A δ , B , Z ~ ∥ Q U ) = I ( A δ , B ; U ∣ Z ~ ) \mathbb{E}D\left(P_{U\mid A_{\delta,B},\widetilde{Z}}\middle\|Q_U\right)=I(A_{\delta,B};U\mid\widetilde{Z}) E D ( P U ∣ A δ , B , Z Q U ) = I ( A δ , B ; U ∣ Z ) . Consequently,
∣ E G M C , B δ ( A δ , B ) ∣ ≤ 2 2 I ( A δ , B ; U ∣ Z ~ ) n . \left|\mathbb{E}G_{\mathrm{MC},B}^{\delta}(A_{\delta,B})\right|\le2\sqrt{\frac{2I(A_{\delta,B};U\mid\widetilde{Z})}{n}}. E G MC , B δ ( A δ , B ) ≤ 2 n 2 I ( A δ , B ; U ∣ Z ) .
Combining this with the 2 δ 2\delta 2 δ discretization error proves the claim.
Lemma 13 (Multi-threshold growth bound for pseudo-dimension). Let F ⊆ [ 0 , 1 ] X \mathcal{F}\subseteq[0,1]^{\mathcal{X}} F ⊆ [ 0 , 1 ] X have pseudo-dimension d 0 d_0 d 0 . Fix input points x 1 , … , x m x_1,\ldots,x_m x 1 , … , x m and thresholds a 1 , … , a T a_1,\ldots,a_T a 1 , … , a T . The number of joint threshold tables ( 1 { f ( x i ) > a k } ) i ∈ [ m ] , k ∈ [ T ] \left(1\{f(x_i)>a_k\}\right)_{i\in[m],k\in[T]} ( 1 { f ( x i ) > a k } ) i ∈ [ m ] , k ∈ [ T ] , f ∈ F f\in\mathcal{F} f ∈ F , is at most
( e m T d 0 d 0 ) \binom{\frac{emT}{d_0}}{d_0} ( d 0 d 0 e m T ) when 1 ≤ d 0 ≤ m T 1\le d_0\le mT 1 ≤ d 0 ≤ m T , with the trivial bound 2 m T 2^{mT} 2 m T covering the remaining cases.
Consequently, suppose f W ∈ F f_W\in\mathcal{F} f W ∈ F almost surely, and let T δ , B = ∣ { τ 1 , … , τ B − 1 } ∪ { γ 1 , … , γ M δ } ∣ T_{\delta,B}=\left|\{\tau_1,\ldots,\tau_{B-1}\}\cup\{\gamma_1,\ldots,\gamma_{M_\delta}\}\right| T δ , B = ∣ { τ 1 , … , τ B − 1 } ∪ { γ 1 , … , γ M δ } ∣ be the number of distinct thresholds needed to determine both q B ( f ( x ) ) q_B(f(x)) q B ( f ( x )) and Q δ ( f ( x ) ) Q_\delta(f(x)) Q δ ( f ( x )) . Then, with d ⋆ = d 0 ∧ ( 2 n T δ , B ) d_\star=d_0\wedge(2nT_{\delta,B}) d ⋆ = d 0 ∧ ( 2 n T δ , B ) , we have
H ( A δ , B ( f W , X ~ ) ∣ Z ~ ) ≤ d ⋆ log ( 2 e n T δ , B d ⋆ ) H\left(A_{\delta,B}(f_W,\widetilde{X})\mid\widetilde{Z}\right)\le d_\star\log\left(\frac{2enT_{\delta,B}}{d_\star}\right) H ( A δ , B ( f W , X ) ∣ Z ) ≤ d ⋆ log ( d ⋆ 2 e n T δ , B ) whenever d ⋆ ≥ 1 d_\star\ge1 d ⋆ ≥ 1 . If d ⋆ = 0 d_\star=0 d ⋆ = 0 , the entropy is zero. In particular, in the nontrivial regime 1 ≤ d 0 ≤ 2 n T δ , B 1\le d_0\le2nT_{\delta,B} 1 ≤ d 0 ≤ 2 n T δ , B ,
H ( A δ , B ( f W , X ~ ) ∣ Z ~ ) ≤ d 0 log ( 2 e n T δ , B d 0 ) . H\left(A_{\delta,B}(f_W,\widetilde{X})\mid\widetilde{Z}\right)\le d_0\log\left(\frac{2enT_{\delta,B}}{d_0}\right). H ( A δ , B ( f W , X ) ∣ Z ) ≤ d 0 log ( d 0 2 e n T δ , B ) .
Proof. The pseudo-dimension of F \mathcal{F} F is the VC dimension of the subgraph threshold class S F = { ( x , a ) ↦ 1 { f ( x ) > a } : f ∈ F } \mathcal{S}_{\mathcal{F}}=\{(x,a)\mapsto1\{f(x)>a\}:f\in\mathcal{F}\} S F = {( x , a ) ↦ 1 { f ( x ) > a } : f ∈ F } . The joint threshold table on ( x i , a k ) (x_i,a_k) ( x i , a k ) is exactly a binary labeling of the m T mT m T augmented points ( x i , a k ) ∈ X × [ 0 , 1 ] (x_i,a_k) \in\mathcal{X} \times[0,1] ( x i , a k ) ∈ X × [ 0 , 1 ] , i ∈ [ m ] i \in[m] i ∈ [ m ] , k ∈ [ T ] k \in[T] k ∈ [ T ] , by the class S F S_{\mathcal{F}} S F . Sauer–Shelah therefore gives at most
( e m T d 0 ) d 0 \left(\frac{emT}{d_0}\right)^{d_0} ( d 0 e m T ) d 0
possible tables when 1 ≤ d 0 ≤ m T 1 \le d_0 \le mT 1 ≤ d 0 ≤ m T . If d 0 > m T d_0 > mT d 0 > m T , the trivial bound 2 m T 2^{mT} 2 m T applies; this is captured by replacing d 0 d_0 d 0 by d 0 ∧ m T d_0 \wedge mT d 0 ∧ m T .
Now fix a supersample realization Z ~ = z ~ \widetilde{Z}=\widetilde{z} Z = z . The table A δ , B ( f , X ~ ) = ( q B ( f ( X ~ i , j ) ) , Q δ ( f ( X ~ i , j ) ) ) i , j A_{\delta,B}(f,\widetilde{X})=\left(q_B(f(\widetilde{X}_{i,j})),Q_\delta(f(\widetilde{X}_{i,j}))\right)_{i,j} A δ , B ( f , X ) = ( q B ( f ( X i , j )) , Q δ ( f ( X i , j )) ) i , j is determined by threshold comparisons of f f f at the 2 n 2n 2 n supersample inputs X ~ i , j \widetilde{X}_{i,j} X i , j , i ∈ [ n ] i \in[n] i ∈ [ n ] , j ∈ { 0 , 1 } j \in\{0,1\} j ∈ { 0 , 1 } , against the union of two finite threshold sets: { τ 1 , … , τ B − 1 } \{\tau_1,\ldots,\tau_{B-1}\} { τ 1 , … , τ B − 1 } and { γ 1 , … , γ M δ } \{\gamma_1,\ldots,\gamma_{M_\delta}\} { γ 1 , … , γ M δ } . The first set determines the exact bin index q B ( f ( x ) ) q_B(f(x)) q B ( f ( x )) . The second set determines the interval cell of f ( x ) f(x) f ( x ) under Q δ Q_\delta Q δ , and therefore determines the quantized value Q δ ( f ( x ) ) Q_\delta(f(x)) Q δ ( f ( x )) . Hence the number of distinct thresholds needed is
T δ , B = ∣ { τ 1 , … , τ B − 1 } ∪ { γ 1 , … , γ M δ } ∣ ≤ ( B − 1 ) + M δ . T_{\delta,B}=\left|\{\tau_1,\ldots,\tau_{B-1}\}\cup\{\gamma_1,\ldots,\gamma_{M_\delta}\}\right|\le(B-1)+M_\delta. T δ , B = ∣ { τ 1 , … , τ B − 1 } ∪ { γ 1 , … , γ M δ } ∣ ≤ ( B − 1 ) + M δ .
Since A δ , B ( f , X ~ ) A_{\delta,B}(f,\widetilde{X}) A δ , B ( f , X ) is a function of these threshold comparison tables, the number of possible values of A δ , B ( f , X ~ ) A_{\delta,B}(f,\widetilde{X}) A δ , B ( f , X ) , as f f f ranges over F \mathcal{F} F , is at most the Sauer–Shelah growth bound with m = 2 n m=2n m = 2 n , T = T δ , B T=T_{\delta,B} T = T δ , B . Therefore, conditionally on Z ~ \widetilde{Z} Z ,
H ( A δ , B ( f W , X ~ ) ∣ Z ~ = z ~ ) ≤ d ⋆ log ( 2 e n T δ , B d ⋆ ) , H\left(A_{\delta,B}(f_W,\widetilde{X})\mid\widetilde{Z}=\widetilde{z}\right)\le d_\star\log\left(\frac{2enT_{\delta,B}}{d_\star}\right), H ( A δ , B ( f W , X ) ∣ Z = z ) ≤ d ⋆ log ( d ⋆ 2 e n T δ , B ) ,
with the zero-entropy convention when d ⋆ = 0 d_\star=0 d ⋆ = 0 . Averaging over Z ~ \widetilde{Z} Z proves the entropy bound. The displayed nontrivial regime follows by taking d ⋆ = d 0 d_\star=d_0 d ⋆ = d 0 .
Proof of Corollary 2. Choose δ = 1 n \delta=\frac{1}{n} δ = n 1 . Use any interval quantizer satisfying the conditions above; for example, the uniform-grid quantizer in the preceding remark. By construction, the quantizer has M δ + 1 M_\delta+1 M δ + 1 output values and
M δ + 1 ≤ ⌈ 1 δ ⌉ + 1. M_\delta+1\le\left\lceil\frac{1}{\delta}\right\rceil+1. M δ + 1 ≤ ⌈ δ 1 ⌉ + 1.
Therefore
M δ ≤ ⌈ 1 δ ⌉ . M_\delta\le\left\lceil\frac{1}{\delta}\right\rceil. M δ ≤ ⌈ δ 1 ⌉ .
With δ = 1 / n \delta=1/n δ = 1/ n , this becomes M δ ≤ ⌈ n ⌉ = n M_\delta\le\lceil n\rceil=n M δ ≤ ⌈ n ⌉ = n .
The table A δ , B A_{\delta,B} A δ , B is determined by two kinds of thresholds. The exact bin q B ( f ( x ) ) q_B(f(x)) q B ( f ( x )) is determined by the B − 1 B-1 B − 1 bin boundaries τ 1 , … , τ B − 1 \tau_1,\ldots,\tau_{B-1} τ 1 , … , τ B − 1 , while the quantized residual value Q δ ( f ( x ) ) Q_\delta(f(x)) Q δ ( f ( x )) is determined by the M δ M_\delta M δ quantizer thresholds γ 1 , … , γ M δ \gamma_1,\ldots,\gamma_{M_\delta} γ 1 , … , γ M δ . Hence the total number of distinct thresholds is
T δ , B = ∣ { τ 1 , … , τ B − 1 } ∪ { γ 1 , … , γ M δ } ∣ ≤ ( B − 1 ) + M δ ≤ ( B − 1 ) + n ≤ B + n . T_{\delta,B}=\left|\{\tau_1,\ldots,\tau_{B-1}\}\cup\{\gamma_1,\ldots,\gamma_{M_\delta}\}\right|\le(B-1)+M_\delta\le(B-1)+n\le B+n. T δ , B = ∣ { τ 1 , … , τ B − 1 } ∪ { γ 1 , … , γ M δ } ∣ ≤ ( B − 1 ) + M δ ≤ ( B − 1 ) + n ≤ B + n .
If some bin thresholds and quantizer thresholds coincide, the union is even smaller. The important point is that the threshold count is additive, not multiplicative: one needs the bin-boundary comparisons and the quantizer-cell comparisons, not all pairs of bins and quantizer values.
Since mutual information is bounded by conditional entropy, Lemma 13 gives I ( A δ , B ; U ∣ Z ~ ) ≤ H ( A δ , B ∣ Z ~ ) I(A_{\delta,B};U\mid\widetilde{Z})\le H(A_{\delta,B}\mid\widetilde{Z}) I ( A δ , B ; U ∣ Z ) ≤ H ( A δ , B ∣ Z ) . Using the threshold count T δ , B ≤ B + n T_{\delta,B}\le B+n T δ , B ≤ B + n , we obtain, in the usual nontrivial regime,
I ( A δ , B ; U ∣ Z ~ ) ≤ H ( A δ , B ∣ Z ~ ) ≲ d 0 log ( e n ( B + n ) d 0 ) . I(A_{\delta,B};U\mid\widetilde{Z})\le H(A_{\delta,B}\mid\widetilde{Z})\lesssim d_0\log\left(\frac{en(B+n)}{d_0}\right). I ( A δ , B ; U ∣ Z ) ≤ H ( A δ , B ∣ Z ) ≲ d 0 log ( d 0 e n ( B + n ) ) .
More precisely, one may replace d 0 d_0 d 0 by d ⋆ = d 0 ∧ O ( n ( B + n ) ) d_\star=d_0\wedge O(n(B+n)) d ⋆ = d 0 ∧ O ( n ( B + n )) inside the entropy bound. The displayed expression is the form used in the main text.
Substituting this entropy bound and δ = 1 / n \delta=1/n δ = 1/ n into Lemma 12 yields
∣ E { Φ M C , B ( f W ; S U ‾ ) − Φ M C , B ( f W ; S U ) } ∣ ≲ 1 n + d 0 log { e n ( B + n ) / d 0 } n . \left|\mathbb{E}\{\Phi_{\mathrm{MC},B}(f_W;S_{\overline{U}})-\Phi_{\mathrm{MC},B}(f_W;S_U)\}\right|\lesssim\frac{1}{n}+\sqrt{\frac{d_0\log\{en(B+n)/d_0\}}{n}}. E { Φ MC , B ( f W ; S U ) − Φ MC , B ( f W ; S U )} ≲ n 1 + n d 0 log { e n ( B + n ) / d 0 } .
Since S ‾ U = S ′ \overline{S}_U=S' S U = S ′ , S U = S S_U=S S U = S , and Φ M C , B , n ≡ Φ M C , B \Phi_{\mathrm{MC},B,n}\equiv\Phi_{\mathrm{MC},B} Φ MC , B , n ≡ Φ MC , B on empirical batches, this is exactly Eq. (13) .
Why the threshold count is additive
The bound T δ , B = ∣ { τ 1 , … , τ B − 1 } ∪ { γ 1 , … , γ M δ } ∣ ≤ ( B − 1 ) + M δ T_{\delta,B}=\left|\{\tau_1,\ldots,\tau_{B-1}\}\cup\{\gamma_1,\ldots,\gamma_{M_\delta}\}\right|\le(B-1)+M_\delta T δ , B = ∣ { τ 1 , … , τ B − 1 } ∪ { γ 1 , … , γ M δ } ∣ ≤ ( B − 1 ) + M δ is an additive threshold count, not a multiplicative one. The reason is that both the bin index and the quantized residual value are determined from the same scalar prediction value t = f ( x ) ∈ [ 0 , 1 ] t=f(x)\in[0,1] t = f ( x ) ∈ [ 0 , 1 ] . The bin index q B ( t ) q_B(t) q B ( t ) is determined by comparing t t t with the B − 1 B-1 B − 1 bin boundaries τ 1 , … , τ B − 1 \tau_1,\ldots,\tau_{B-1} τ 1 , … , τ B − 1 , while the quantized value Q δ ( t ) Q_\delta(t) Q δ ( t ) is determined by comparing t t t with the M δ M_\delta M δ quantizer thresholds γ 1 , … , γ M δ \gamma_1,\ldots,\gamma_{M_\delta} γ 1 , … , γ M δ . Thus the pair ( q B ( t ) , Q δ ( t ) ) (q_B(t),Q_\delta(t)) ( q B ( t ) , Q δ ( t )) is determined by the union of these two one-dimensional threshold sets.
Equivalently, the bin partition and the quantizer partition are two partitions of the same interval [ 0 , 1 ] [0,1] [ 0 , 1 ] . Their common refinement is obtained by placing all boundary points on the same line. If the total number of distinct boundary points is T δ , B T_{\delta,B} T δ , B , then this common refinement has at most T δ , B + 1 T_{\delta,B}+1 T δ , B + 1 cells. Hence the number of relevant one-dimensional comparisons is controlled by T δ , B ≤ ( B − 1 ) + M δ T_{\delta,B}\le(B-1)+M_\delta T δ , B ≤ ( B − 1 ) + M δ , rather than by the product B ( M δ + 1 ) B(M_\delta+1) B ( M δ + 1 ) .
Experimental settings and additional results
This appendix gives the experimental settings and additional results for logistic regression on Flights200K.
Flights200K: settings and estimation
Settings. We use n = 2 9 , 2 10 , … , 2 15 n = 2^9, 2^{10}, \ldots, 2^{15} n = 2 9 , 2 10 , … , 2 15 and σ ∈ { 0.05 , 0.10 } \sigma\in\{0.05, 0.10\} σ ∈ { 0.05 , 0.10 } . The linear logistic regression model is fitted with the newton-cholesky solver and regularization parameter C = 1 C = 1 C = 1 in scikit-learn. For each sample configuration, we newly split the data into a 20% held-out sample, a 10% reference sample S r e f S_{\mathrm{ref}} S ref , and a 70% candidate pool. The nine groups are defined using the medians of the first three features in the reference sample; the five score bins have equal width. Within each configuration, the same group definitions are used for training, validation, and both selector branches.
We estimate the generalization gap by Monte Carlo as follows. Specifically, we average the validation–training MC differences over sample configurations and Gaussian perturbations, then take the absolute value. For one estimate, we generate 32 sample configurations, indexed by j = 1 , … , J j = 1, \ldots, J j = 1 , … , J with J = 32 J = 32 J = 32 . Each configuration consists of a reference sample, a supersample of n n n pairs drawn without replacement from its candidate pool, and a selector vector. These determine the training sample S j S_j S j and the complementary validation sample S j ′ S'_j S j ′ . For each j j j , we fit the model on S j S_j S j and generate 16 independent Gaussian perturbations, indexed by r = 1 , … , R r = 1, \ldots, R r = 1 , … , R with R = 16 R = 16 R = 16 . The perturbed parameters are W j r = θ ^ ( S j ) + σ ξ j r W_{jr} = \widehat{\theta}(S_j) + \sigma\xi_{jr} W j r = θ ( S j ) + σ ξ j r , where ξ j r ∼ N ( 0 , I 8 ) \xi_{jr} \sim N(0,I_8) ξ j r ∼ N ( 0 , I 8 ) are independent, as in § 5.1 . We evaluate each perturbed predictor on the same S j S_j S j and S j ′ S'_j S j ′ and average the resulting J R JR J R signed differences, giving
L ^ n = ∣ 1 J R ∑ j = 1 J ∑ r = 1 R { Φ M C ( f W j r ; S j ′ ) − Φ M C ( f W j r ; S j ) } ∣ . (81) \widehat{L}_n = \left| \frac{1}{JR} \sum_{j=1}^{J} \sum_{r=1}^{R} \left\{ \Phi_{\mathrm{MC}}(f_{W_{jr}};S'_j) - \Phi_{\mathrm{MC}}(f_{W_{jr}};S_j) \right\} \right|. \tag*{(81)} L n = J R 1 j = 1 ∑ J r = 1 ∑ R { Φ MC ( f W j r ; S j ′ ) − Φ MC ( f W j r ; S j ) } . (81)
The average inside the absolute value approximates the expectation over the sample configuration and Gaussian perturbation in Eq. (12) .
Conditional mutual information. Let D t M C D_t^{\mathrm{MC}} D t MC denote the increment D t D_t D t in Theorem 2 specialized to the MC objective. To upper-bound the local information in Eq. (12) , we evaluate the conditional mutual information between the learned parameter W W W and the selector U t U_t U t . Write C t = ( Z ~ , U − t , S r e f ) C_t = (\widetilde{Z}, U_{-t}, S_{\mathrm{ref}}) C t = ( Z , U − t , S ref ) and fix C t = c C_t = c C t = c . Let μ 0 ( c ) \mu_0(c) μ 0 ( c ) and μ 1 ( c ) \mu_1(c) μ 1 ( c ) be the fitted parameters obtained by retraining with U t = 0 U_t = 0 U t = 0 and U t = 1 U_t = 1 U t = 1 , respectively. The Gaussian perturbation model gives W ∣ ( U t = u , C t = c ) ∼ N ( μ u ( c ) , σ 2 I 8 ) W \mid(U_t = u, C_t = c) \sim N(\mu_u(c), \sigma^2 I_8) W ∣ ( U t = u , C t = c ) ∼ N ( μ u ( c ) , σ 2 I 8 ) . Let p 0 , p 1 p_0, p_1 p 0 , p 1 be these Gaussian densities and put d t ( c ) = ∥ μ 1 ( c ) − μ 0 ( c ) ∥ 2 / σ d_t(c) = \lVert\mu_1(c) - \mu_0(c) \rVert_2/\sigma d t ( c ) = ∥ μ 1 ( c ) − μ 0 ( c ) ∥ 2 / σ . The selector vector is generated independently of ( Z ~ , S r e f ) (\widetilde{Z}, S_{\mathrm{ref}}) ( Z , S ref ) , and its coordinates are independent and uniform on { 0 , 1 } \{0,1\} { 0 , 1 } . Thus U t U_t U t remains uniform given C t = c C_t = c C t = c , and the conditional density of W W W is ( p 0 + p 1 ) / 2 (p_0 + p_1)/2 ( p 0 + p 1 ) /2 . The definition of mutual information gives
I ( W ; U t ∣ C t = c ) = log 2 − 1 2 ∑ u = 0 1 E W ∼ p u log ( 1 + p 1 − u ( W ) p u ( W ) ) . I(W;U_t \mid C_t = c) = \log2 - \frac{1}{2} \sum_{u=0}^{1} \mathbb{E}_{W\sim p_u} \log\left(1 + \frac{p_{1-u}(W)}{p_u(W)}\right). I ( W ; U t ∣ C t = c ) = log 2 − 2 1 u = 0 ∑ 1 E W ∼ p u log ( 1 + p u ( W ) p 1 − u ( W ) ) .
Conditioning on U t = u U_t = u U t = u and C t = c C_t = c C t = c , both means are fixed vectors and W = μ u ( c ) + σ ξ W = \mu_u(c) + \sigma\xi W = μ u ( c ) + σ ξ with ξ ∼ N ( 0 , I 8 ) \xi\sim N(0,I_8) ξ ∼ N ( 0 , I 8 ) . Substituting the Gaussian densities yields
log p u ( W ) p 1 − u ( W ) = d t ( c ) 2 2 + { μ u ( c ) − μ 1 − u ( c ) } T ξ σ . \log\frac{p_u(W)}{p_{1-u}(W)} = \frac{d_t(c)^2}{2} + \frac{\{\mu_u(c)-\mu_{1-u}(c)\}^{\mathsf{T}}\xi}{\sigma}. log p 1 − u ( W ) p u ( W ) = 2 d t ( c ) 2 + σ { μ u ( c ) − μ 1 − u ( c ) } T ξ .
The second term is Gaussian with mean zero and variance d t ( c ) 2 d_t(c)^2 d t ( c ) 2 . For a standard normal random variable Z Z Z , the expectation in the mutual information is therefore
E W ∼ p u log ( 1 + p 1 − u ( W ) p u ( W ) ) = E Z ∼ N ( 0 , 1 ) log ( 1 + e − d t ( c ) 2 / 2 − d t ( c ) Z ) . \mathbb{E}_{W\sim p_u} \log\left(1 + \frac{p_{1-u}(W)}{p_u(W)}\right) = \mathbb{E}_{Z\sim N(0,1)} \log\left(1 + e^{-d_t(c)^2/2-d_t(c)Z}\right). E W ∼ p u log ( 1 + p u ( W ) p 1 − u ( W ) ) = E Z ∼ N ( 0 , 1 ) log ( 1 + e − d t ( c ) 2 /2 − d t ( c ) Z ) .
The right-hand side does not depend on u u u , so averaging over the two branches gives
I ( W ; U t ∣ C t = c ) = G ( d t ( c ) ) , I(W;U_t \mid C_t = c) = G(d_t(c)), I ( W ; U t ∣ C t = c ) = G ( d t ( c )) ,
G ( d ) = log 2 − E Z ∼ N ( 0 , 1 ) log ( 1 + e − d 2 / 2 − d Z ) . (82) G(d) = \log2 - \mathbb{E}_{Z\sim N(0,1)} \log\left(1 + e^{-d^2/2-dZ}\right). \tag*{(82)} G ( d ) = log 2 − E Z ∼ N ( 0 , 1 ) log ( 1 + e − d 2 /2 − d Z ) . (82)
When the means coincide, the log-density ratio is zero and the same expression gives G ( 0 ) = 0 G(0)=0 G ( 0 ) = 0 . We use natural logarithms and evaluate this one-dimensional integral by 80-point Gauss–Hermite quadrature.
Independence of the selectors allows U − t U_{-t} U − t and S r e f S_{\mathrm{ref}} S ref to be added to the conditioning, as in Eq. (63) . For fixed C t C_t C t , the two mixed-path samples in Theorem 2 and the group definitions are fixed, so D t M C D_t^{\mathrm{MC}} D t MC is a function of the predictions F = f W ( X ~ ) F=f_W(\widetilde{X}) F = f W ( X ) . Eq. (64) and data processing give
I l o c , t M C ≤ I ( D t M C ; U t ∣ C t ) ≤ I ( F ; U t ∣ C t ) ≤ I ( W ; U t ∣ C t ) = E C t [ G ( d t ( C t ) ) ] . (83) \begin{aligned}
I_{\mathrm{loc},t}^{\mathrm{MC}} &\le I(D_t^{\mathrm{MC}};U_t\mid C_t)\le I(F;U_t\mid C_t)\\
&\le I(W;U_t\mid C_t)=\mathbb{E}_{C_t}[G(d_t(C_t))]. \tag*{(83)}
\end{aligned} I loc , t MC ≤ I ( D t MC ; U t ∣ C t ) ≤ I ( F ; U t ∣ C t ) ≤ I ( W ; U t ∣ C t ) = E C t [ G ( d t ( C t ))] . (83)
For Δ M C = Φ M C ( f W ; S ′ ) − Φ M C ( f W ; S ) \Delta_{\mathrm{MC}}=\Phi_{\mathrm{MC}}(f_W;S')-\Phi_{\mathrm{MC}}(f_W;S) Δ MC = Φ MC ( f W ; S ′ ) − Φ MC ( f W ; S ) , the single-index term in Eq. (12) therefore gives
∣ E Δ M C ∣ ≤ B G , n = 2 n ∑ t = 1 n 2 E C t [ G ( d t ( C t ) ) ] . (84) \lvert\mathbb{E}\Delta_{\mathrm{MC}}\rvert\le B_{G,n}=\frac{2}{n}\sum_{t=1}^{n}\sqrt{2\mathbb{E}_{C_t}[G(d_t(C_t))]}. \tag*{(84)} ∣ E Δ MC ∣ ≤ B G , n = n 2 t = 1 ∑ n 2 E C t [ G ( d t ( C t ))] . (84)
Estimating the bound. Here t t t indexes one of the n n n pairs in the supersample. For each estimation run, we uniformly sample 16 indices without replacement from { 1 , … , n } \{1,\ldots,n\} { 1 , … , n } , independently of the data, and denote this set by T m T_m T m , with m = 16 m=16 m = 16 . For these indices, we evaluate the mutual information using the same 32 configurations as above. Let Z ~ j \widetilde{Z}_j Z j , U ( j ) U^{(j)} U ( j ) , and S r e f , j S_{\mathrm{ref},j} S ref , j denote the supersample, selector vector, and reference sample in configuration j j j . For each t ∈ T m t\in T_m t ∈ T m , fix C t , j = ( Z ~ j , U − t ( j ) , S r e f , j ) C_{t,j}=(\widetilde{Z}_j,U_{-t}^{(j)},S_{\mathrm{ref},j}) C t , j = ( Z j , U − t ( j ) , S ref , j ) , change only the t t t -th selector to zero or one, and fit the two models. The distance between μ 0 ( C t , j ) \mu_0(C_{t,j}) μ 0 ( C t , j ) and μ 1 ( C t , j ) \mu_1(C_{t,j}) μ 1 ( C t , j ) determines G ( d t ( C t , j ) ) G(d_t(C_{t,j})) G ( d t ( C t , j )) through Eq. (H.2). Although the index t t t is held fixed across configurations, the data pair occupying that position is drawn anew in each configuration.
For each t t t , we average G ( d t ( C t , j ) ) G(d_t(C_{t,j})) G ( d t ( C t , j )) over the 32 configurations to estimate the expectation over C t C_t C t in Eq. (H.3). We substitute this average into Eq. (H.4) before taking the square root, and approximate the average over all n n n indices by the average over the 16 sampled indices. One estimation run thus gives
B ^ G , n = 2 m ∑ t ∈ T m 2 J ∑ j = 1 J G ( d t ( C t , j ) ) , J = 32. (85) \widehat{B}_{G,n}=\frac{2}{m}\sum_{t\in T_m}\sqrt{\frac{2}{J}\sum_{j=1}^{J}G(d_t(C_{t,j}))},\qquad J=32. \tag*{(85)} B G , n = m 2 t ∈ T m ∑ J 2 j = 1 ∑ J G ( d t ( C t , j )) , J = 32. (85)
Means and standard deviations of the estimates. For each sample size and noise level, we independently repeat the entire procedure 20 times, obtaining 20 gap estimates from Eq. (H.1) and 20 bound estimates from Eq. (H.5). In each run, we generate new sample configurations and selector vectors, 16 perturbations for each fitted model, and 16 pair indices. Each point in Figure 1 is the mean of the corresponding 20 estimates. Error bars show this mean plus or minus one sample standard deviation, computed with 19 degrees of freedom. They describe the variation of the finite Monte Carlo estimate when the 32-configuration estimation procedure is repeated. For the bound, Eq. (H.2) integrates over the Gaussian noise. Its estimate still varies because regenerating the samples, selector vectors, reference samples, and pair indices changes the retrained parameter centers.
CVaR
Using the training, sample-generation, Gaussian-perturbation, and repetition settings above, we next examine the generalization gap and upper bound for CVaR, which evaluates the tail of the loss distribution. The training objective is unchanged, and evaluation uses the Brier loss ℓ ( f W ( x ) , y ) = ( f W ( x ) − y ) 2 ∈ [ 0 , 1 ] \ell(f_W(x),y)=(f_W(x)-y)^2\in[0,1] ℓ ( f W ( x ) , y ) = ( f W ( x ) − y ) 2 ∈ [ 0 , 1 ] . We use α ∈ { 0.2 , 0.5 , 0.8 , 0.95 } \alpha\in\{0.2,0.5,0.8,0.95\} α ∈ { 0.2 , 0.5 , 0.8 , 0.95 } , corresponding to CVaR over the largest 80%, 50%, 20%, and 5% of the loss distribution, respectively. Within each repetition, all levels use the same training and validation samples and perturbed models. The gap is estimated by replacing MC with CVaR in Eq. (H.1).
Computing CVaR. With the perturbed predictor f W f_W f W fixed, we compute the n n n Brier loss values separately on the training and validation samples. For an evaluation sample s = ( ( x i , y i ) ) i = 1 n s=((x_i,y_i))_{i=1}^{n} s = (( x i , y i ) ) i = 1 n , let r i = ℓ ( f W ( x i ) , y i ) r_i=\ell(f_W(x_i),y_i) r i = ℓ ( f W ( x i ) , y i ) and sort these values in descending order as r ( 1 ) ≥ ⋯ ≥ r ( n ) r_{(1)}\ge\cdots\ge r_{(n)} r ( 1 ) ≥ ⋯ ≥ r ( n ) . With α ˉ = 1 − α \bar{\alpha}=1-\alpha α ˉ = 1 − α as in the main text, put a = n α ˉ a = n\bar{\alpha} a = n α ˉ , k = ⌊ a ⌋ k = \lfloor a\rfloor k = ⌊ a ⌋ , and δ = a − k \delta= a-k δ = a − k . Eq. (3) can be rewritten as [31 ]
Φ C V a R , α ( f W ; s ) = ∑ i = 1 k r ( i ) + δ r ( k + 1 ) a . \Phi_{\mathrm{CVaR},\alpha}(f_W;s)=\frac{\sum_{i=1}^{k}r^{(i)}+\delta r^{(k+1)}}{a}. Φ CVaR , α ( f W ; s ) = a ∑ i = 1 k r ( i ) + δ r ( k + 1 ) .
The second term is zero when δ = 0 \delta=0 δ = 0 , and the sum is zero when k = 0 k=0 k = 0 .
For the upper bound, we apply Corollary 5 for loss values in [ 0 , 1 ] [0,1] [ 0 , 1 ] and take the smaller of the stability and convex L 2 L_2 L 2 -Lipschitz bounds for the loss vector. To upper-estimate I f C M I I_{\mathrm{fCMI}} I fCMI , we use the conditional mutual-information estimates obtained above. These information estimates are shared across all values of α \alpha α ; the level dependence of the bound comes from the coefficient in Corollary 5.
As for MC, Figure 3 shows decreasing gap and bound estimates as the sample size increases, at both noise levels. Increasing α \alpha α reduces the tail fraction α ˉ \bar{\alpha} α ˉ and concentrates evaluation on fewer, larger loss values. The upper bound increases with α \alpha α , and more samples are needed before it falls below one. However, the estimated generalization gap does not increase consistently with α \alpha α . As in the MC experiment, the bound estimates exceed the gap estimates and remain conservative.
Figure 3. CVaR generalization-gap and upper-bound estimates from Corollary 5, with σ = 0.05 \sigma=0.05 σ = 0.05 (left) and σ = 0.10 \sigma=0.10 σ = 0.10 (right). Dashed lines with filled markers show gaps; solid lines with open markers show bounds. Colors identify α \alpha α . Points and error bars show the mean and one sample standard deviation over 20 independent repetitions. The horizontal dotted line is one.
For CVaR, as for MC, the bounds at σ = 0.10 \sigma=0.10 σ = 0.10 are smaller than those at σ = 0.05 \sigma=0.05 σ = 0.05 . This is consistent with the Gaussian information above being determined by the distance between the two fitted parameter vectors relative to the noise level. In contrast, the CVaR gap is larger at σ = 0.10 \sigma=0.10 σ = 0.10 in many settings. Noise changes both the information and the prediction loss, so its effects on the bound and the generalization gap can be distinguished.
CR DRO
Using the same experimental settings, we evaluate CR DRO on the perturbed models and Brier loss used in Appendix § 17 .§ 2 . For the shape comparison, we fix ϵ = 0.1 \epsilon=0.1 ϵ = 0.1 and use c ∈ { 1.25 , 1.5 , 2 , 4 } c\in\{1.25,1.5,2,4\} c ∈ { 1.25 , 1.5 , 2 , 4 } ; for the radius comparison, we fix c = 2 c=2 c = 2 and use ϵ ∈ { 0 , 0.01 , 0.1 , 1 } \epsilon\in\{0,0.01,0.1,1\} ϵ ∈ { 0 , 0.01 , 0.1 , 1 } . Within each repetition, samples and models are shared, and we vary the evaluation-time reweighting constraint to examine the sensitivity of the gap and bound.
We use the same generator f c ( t ) = t c − c t + c − 1 c ( c − 1 ) f_c(t)=\frac{t^c-ct+c-1}{c(c-1)} f c ( t ) = c ( c − 1 ) t c − c t + c − 1 as in the main text. Let P ^ s \widehat{P}_s P s be the empirical distribution of the evaluation sample s s s and Q Q Q the reweighted distribution. We maximize the mean loss subject to D c ( Q ∥ P ^ s ) = E P ^ s f c ( d Q / d P ^ s ) ≤ ϵ D_c(Q\|\widehat{P}_s)=\mathbb{E}_{\widehat{P}_s}f_c(\mathrm{d}Q/\mathrm{d}\widehat{P}_s)\leq\epsilon D c ( Q ∥ P s ) = E P s f c ( d Q / d P s ) ≤ ϵ [6 ] . The limit as c ↓ 1 c\downarrow1 c ↓ 1 is the KL divergence D K L ( Q ∥ P ^ s ) D_{\mathrm{KL}}(Q\|\widehat{P}_s) D KL ( Q ∥ P s ) , whereas c = 2 c=2 c = 2 corresponds to one half of the Pearson χ 2 \chi^2 χ 2 divergence. Under this convention, radius ϵ \epsilon ϵ corresponds to radius 2 ϵ 2\epsilon 2 ϵ for the Pearson χ 2 \chi^2 χ 2 divergence without the factor 1 / 2 1/2 1/2 . The values c = 1.25 , 1.5 c = 1.25, 1.5 c = 1.25 , 1.5 belong to the Cressie–Read family with 1 < c < 2 1 < c < 2 1 < c < 2 , while c = 4 c = 4 c = 4 gives a fourth-order density-ratio term. Equal numerical radii do not define the same feasible reweightings when c c c changes.
Since the loss values lie in [ 0 , 1 ] [0, 1] [ 0 , 1 ] and empirical CR DRO with c > 1 c > 1 c > 1 is convex and L 2 L_2 L 2 -Lipschitz in the loss vector, we use Eq. (17) in Corollary 6 for positive radii. At ϵ = 0 \epsilon= 0 ϵ = 0 , the objective is the empirical mean, so we substitute the one-point sensitivity β n = 1 / n \beta_n = 1/n β n = 1/ n into the term of Theorem 2 that uses information about the full selector vector. We estimate the gap by replacing MC with CR DRO in Eq. (H.1), and use the same information estimates as in Appendix § 17 .§ 2 for all ( c , ϵ ) (c, \epsilon) ( c , ϵ ) .
Computing CR DRO. We evaluate CR DRO using the method of Duchi and Namkoong [6 ] , as follows. As for CVaR, write the loss values on the evaluation sample s s s as r 1 , … , r n r_1, \ldots, r_n r 1 , … , r n , and let c > 1 c > 1 c > 1 . For a fixed threshold η < max i r i \eta< \max_i r_i η < max i r i , the density-ratio weights have the analytic form [6 ]
q i ( η ) = n [ r i − η ] + 1 / ( c − 1 ) ∑ j = 1 n [ r j − η ] + 1 / ( c − 1 ) . q_i(\eta) = \frac{n[r_i-\eta]_+^{1/(c-1)}}{\sum_{j=1}^{n}[r_j-\eta]_+^{1/(c-1)}}. q i ( η ) = ∑ j = 1 n [ r j − η ] + 1/ ( c − 1 ) n [ r i − η ] + 1/ ( c − 1 ) .
These weights satisfy n − 1 ∑ i q i ( η ) = 1 n^{-1}\sum_i q_i(\eta) = 1 n − 1 ∑ i q i ( η ) = 1 ; they assign zero weight to loss values at or below the threshold and larger weights to larger loss values.
Let Q η Q_\eta Q η assign probability q i ( η ) / n q_i(\eta)/n q i ( η ) / n to each sample point; its density ratio with respect to P ^ s \widehat{P}_s P s is q i ( η ) q_i(\eta) q i ( η ) . For a positive radius, when the worst-case value is below the maximum loss, the threshold η ⋆ \eta^\star η ⋆ satisfies
D c ( Q η ⋆ ∥ P ^ s ) = 1 n ∑ i = 1 n f c ( q i ( η ⋆ ) ) = n − 1 ∑ i = 1 n q i ( η ⋆ ) c − 1 c ( c − 1 ) = ϵ . D_c(Q_{\eta^\star}\Vert\widehat{P}_s) = \frac{1}{n}\sum_{i=1}^{n} f_c(q_i(\eta^\star)) = \frac{n^{-1}\sum_{i=1}^{n}q_i(\eta^\star)^c-1}{c(c-1)} = \epsilon. D c ( Q η ⋆ ∥ P s ) = n 1 i = 1 ∑ n f c ( q i ( η ⋆ )) = c ( c − 1 ) n − 1 ∑ i = 1 n q i ( η ⋆ ) c − 1 = ϵ .
The divergence D c ( Q η ∥ P ^ s ) D_c(Q_\eta\Vert\widehat{P}_s) D c ( Q η ∥ P s ) is nondecreasing in η \eta η . Thus, if D c ( Q η ∥ P ^ s ) < ϵ D_c(Q_\eta\Vert\widehat{P}_s) < \epsilon D c ( Q η ∥ P s ) < ϵ , we narrow the search to larger thresholds; if D c ( Q η ∥ P ^ s ) > ϵ D_c(Q_\eta\Vert\widehat{P}_s) > \epsilon D c ( Q η ∥ P s ) > ϵ , we narrow it to smaller thresholds. We find η ⋆ \eta^\star η ⋆ by bisecting the threshold interval at each candidate value η \eta η and retaining the half containing a solution to the equality above. Finally, we compute Φ D R O , c , ϵ ( f W ; s ) = 1 n ∑ i = 1 n q i ( η ⋆ ) r i \Phi_{\mathrm{DRO},c,\epsilon}(f_W; s) = \frac{1}{n}\sum_{i=1}^{n}q_i(\eta^\star)r_i Φ DRO , c , ϵ ( f W ; s ) = n 1 ∑ i = 1 n q i ( η ⋆ ) r i separately on the training and validation samples. At ϵ = 0 \epsilon= 0 ϵ = 0 , we use the mean loss; if reweighting only the observations attaining the maximum loss satisfies the divergence constraint, we use that maximum.
In Corollary 6, the coefficient excluding the information factor scales as n − ( c − 1 ) / c n^{-(c-1)/c} n − ( c − 1 ) / c for $1 < c < 2$ and as n − 1 / 2 n^{-1/2} n − 1/2 for c ≥ 2 c \ge2 c ≥ 2 . In Figure 4 , the bounds for 1 < c < 2 1 < c < 2 1 < c < 2 are larger and decrease more slowly as $c approaches one. For c = 1.25 c = 1.25 c = 1.25 and σ = 0.05 \sigma= 0.05 σ = 0.05 , the bound remains above one even at the largest sample size. The cases c = 2 , 4 c = 2, 4 c = 2 , 4 share the same exponent, and their difference comes from the coefficient { 1 + c ( c − 1 ) ϵ } 1 / c \{1 + c(c - 1)\epsilon\}^{1/c} { 1 + c ( c − 1 ) ϵ } 1/ c ; here, the bound at c = 4 c = 4 c = 4 is slightly larger than that at c = 2 c = 2 c = 2 . The sample-size dependence of the estimated information also contributes to that of the bound.
Figure 4. CR DRO generalization-gap and upper-bound estimates from Eq. (17) in Corollary 6, at radius ϵ = 0.1 \epsilon= 0.1 ϵ = 0.1 . Colors identify c c c . Panels, line styles, markers, and error bars follow Figure 3 .
In contrast, the estimated gap curves remain close as c c c varies, with differences small relative to their sample standard deviations. In this setting, changing the divergence shape has a more pronounced effect on the bound’s coefficient than on the observed gap.
Increasing ϵ \epsilon ϵ enlarges the feasible reweighting set, allowing larger loss values to receive greater weight in both training and validation evaluation. In Figure 5 , the mean gap estimate increases with the radius. For positive radii, the bound increases through κ 2 , ϵ = 1 + 2 ϵ \kappa_{2,\epsilon} = \sqrt{1 + 2\epsilon} κ 2 , ϵ = 1 + 2 ϵ while retaining the same explicit exponent of n n n . The empirical mean at zero radius uses the coefficient from Theorem 2, so the difference between the zero- and positive-radius bounds also includes a change in the bound being used. As for CVaR, the upper bounds are smaller at σ = 0.10 \sigma= 0.10 σ = 0.10 than at σ = 0.05 \sigma= 0.05 σ = 0.05 , while the gaps are larger in many settings.
Figure 5. CR DRO radius comparison at c = 2 c = 2 c = 2 . Positive-radius bounds use Eq. (17) in Corollary 6; the bound at ϵ = 0 \epsilon= 0 ϵ = 0 uses the full-selector information term of Theorem 2 with β n = 1 / n \beta_n = 1/n β n = 1/ n . Panels and plotting conventions follow Figure 3 .
CelebA/ViT: settings and estimation
This appendix describes the data, retraining protocol, information estimation, and aggregation used in Figure 2 .
Data, model, and evaluation objectives
Dataset and groups. We predict the Blond_Hair attribute in CelebA, with Y = 1 Y = 1 Y = 1 denoting blond hair and Y = 0 Y = 0 Y = 0 denoting its absence. The multicalibration group class contains 18 overlapping groups: Male, Female, Arched_Eyebrows, Bangs, Big_Lips, Chubby, Double_Chin, Eyeglasses, High_Cheekbones, Mouth_Slightly_Open, Oval_Face, Pale_Skin, Receding_Hairline, Smiling, Straight_Hair, Wavy_Hair, Wearing_Hat, and Young. The first two groups select examples with Male equal to 1 and 0, respectively; each remaining group selects examples whose named attribute is 1. These are single-attribute groups. The dataset wrapper removes groups whose sizes are at most 0.005 of the full dataset; all 18 groups are retained in this experiment. The group collection is fixed throughout evaluation, with group indices reindexed within each evaluation sample.
Model and training settings. We use a binary ViT classifier with images resized to 224 × 224. We initialize it from the pretrained google/vit-base-patch16-224-in21k checkpoint. Each classifier is trained for 10 epochs with cross-entropy loss and Adam, using batch size 64, learning rate 10 − 4 10^{-4} 1 0 − 4 , and weight decay 0.01. We use the final-epoch checkpoint.
Evaluated objectives. We evaluate multicalibration (MC) and CVaR. For MC, positive-class probabilities p ^ i \hat{p}_i p ^ i are converted into two-class probabilities ( 1 − p ^ i , p ^ i ) (1-\hat{p}_i,\hat{p}_i) ( 1 − p ^ i , p ^ i ) , with one-hot labels. Each class probability is discretized into B = 10 B=10 B = 10 equal-width bins on [ 0 , 1 ] [0,1] [ 0 , 1 ] , with a probability of 1 included in the last bin. The evaluation routine forms a joint bin index r = ( r 0 , r 1 ) ∈ { 0 , … , B − 1 } 2 r=(r_0,r_1)\in\{0,\ldots,B-1\}^2 r = ( r 0 , r 1 ) ∈ { 0 , … , B − 1 } 2 from the two discretized probabilities and computes calibration errors for joint-bin–group–class triples. The reported statistic weights each occupied cell’s absolute conditional calibration error by its empirical cell fraction:
Φ ^ M C ( f ; A ) = max r ∈ { 0 , … , B − 1 } 2 , g ∈ G , k ∈ { 0 , 1 } c e l l ( r , g ) o c c u p i e d N r , g ( A ) ∣ A ∣ ∣ c a l e r r ^ r , g , k ( f ; A ) ∣ , \widehat{\Phi}_{\mathrm{MC}}(f;A)=\max_{\substack{r\in\{0,\ldots,B-1\}^2,\ g\in G,\ k\in\{0,1\}\\ \mathrm{cell}\ (r,g)\ \mathrm{occupied}}}\frac{N_{r,g}(A)}{|A|}\left|\widehat{\mathrm{calerr}}_{r,g,k}(f;A)\right|, Φ MC ( f ; A ) = r ∈ { 0 , … , B − 1 } 2 , g ∈ G , k ∈ { 0 , 1 } cell ( r , g ) occupied max ∣ A ∣ N r , g ( A ) calerr r , g , k ( f ; A ) ,
where N r , g ( A ) N_{r,g}(A) N r , g ( A ) is the number of examples in the corresponding joint-bin–group cell of evaluation sample A A A . The signed conditional error c a l e r r ^ r , g , k \widehat{\mathrm{calerr}}_{r,g,k} calerr r , g , k is the mean of 1 { y i = k } − p i , k \mathbf{1}\{y_i=k\}-p_{i,k} 1 { y i = k } − p i , k within that cell, where p i , 0 = 1 − p ^ i p_{i,0}=1-\hat{p}_i p i , 0 = 1 − p ^ i and p i , 1 = p ^ i p_{i,1}=\hat{p}_i p i , 1 = p ^ i . The implementation also records an unweighted worst-cell diagnostic; Figure 2 uses the moment-weighted statistic above. For CVaR, we apply the empirical objective in Eq. (3) to the absolute calibration losses ℓ i = ∣ p ^ i − y i ∣ \ell_i=|\hat{p}_i-y_i| ℓ i = ∣ p ^ i − y i ∣ , with α = 0.2 \alpha=0.2 α = 0.2 and α ˉ = 1 − α \bar{\alpha}=1-\alpha α ˉ = 1 − α .
Implementation. We build on the public empirical multicalibration codebase of Hansen et al. [17 ] , released under the MIT license. We extend it with paired supersamples, branch retraining, mixed-path evaluation, local increments, and mutual-information estimation for the two objectives.
Supersample construction and branch retraining
Candidate pool and sample sizes. We first split the dataset into training, validation, and test parts with fractions 0.4, 0.4, and 0.2, respectively. The test split is held out from retraining and local information estimation. The union of the training and validation indices forms the supersample candidate pool, whose size N max N_{\max} N m a x is approximately 80% of the full dataset, up to integer rounding. For each sample-size percentage p p p , we select p % p\% p % of this pool, rounding down to an even total N N N , with a minimum total of 20.
The reported sample sizes are:
Table 1 (PDF p. 52).
Each branch training sample and its complementary validation sample contain n = N / 2 n=N/2 n = N /2 examples. The horizontal axis of Figure 2 uses the total supersample size N = 2 n N=2n N = 2 n .
Supersamples and selected pairs. For each sample size and both objectives, we use J = 3 J=3 J = 3 supersample configurations, generated with seeds 10, 15, and 20. Let j ∈ [ J ] j\in[J] j ∈ [ J ] index these configurations. For each configuration j j j , we sample N N N candidate indices without replacement and randomly permute them into an n × 2 n\times2 n × 2 paired array Z ~ ( j ) \widetilde{Z}^{(j)} Z ( j ) . A selector vector U ( j ) U^{(j)} U ( j ) , with independent uniform binary components, chooses one sample from each pair for training and the other for validation. We uniformly sample a subset T j ⊂ [ n ] T_j\subset[n] T j ⊂ [ n ] of m = 10 m=10 m = 10 pair indices for local information estimation.
Two branches and seven training seeds. For each selected pair t ∈ T j t\in T_j t ∈ T j , we hold U t ( j ) U_t^{(j)} U t ( j ) fixed and consider both choices b ∈ { 0 , 1 } b\in\{0,1\} b ∈ { 0 , 1 } for U t ( j ) U_t^{(j)} U t ( j ) . Write U ( j ; t , b ) U^{(j;t,b)} U ( j ; t , b ) for the resulting selector vector. Its branch training and validation samples are
S t r ( j ) ( t , b ) = ( Z ~ i , U i ( j ; t , b ) ( j ) ) i = 1 n , S v a l ( j ) ( t , b ) = ( Z ~ i , 1 − U i ( j ; t , b ) ( j ) ) i = 1 n . S_{\mathrm{tr}}^{(j)}(t,b)=\left(\widetilde{Z}_{i,U_i^{(j;t,b)}}^{(j)}\right)_{i=1}^{n},
\qquad
S_{\mathrm{val}}^{(j)}(t,b)=\left(\widetilde{Z}_{i,1-U_i^{(j;t,b)}}^{(j)}\right)_{i=1}^{n}. S tr ( j ) ( t , b ) = ( Z i , U i ( j ; t , b ) ( j ) ) i = 1 n , S val ( j ) ( t , b ) = ( Z i , 1 − U i ( j ; t , b ) ( j ) ) i = 1 n .
For each branch, we train a fresh classifier using each of the R = 7 R=7 R = 7 model seeds s ∈ { 0 , 1 , … , 6 } s\in\{0,1,\ldots,6\} s ∈ { 0 , 1 , … , 6 } , which initialize the Python, NumPy, and PyTorch random states. Denote the resulting predictor by f j , t , b , s f_{j,t,b,s} f j , t , b , s . The complementary branch sample is not used for checkpoint selection. Thus, each selected pair yields 2 R = 14 2R = 14 2 R = 14 branch-training runs, and the ten selected pairs give m × 2 × R = 140 m \times2 \times R = 140 m × 2 × R = 140 runs per configuration and sample size. The 14 predictors for each pair produce the 14 increment–selector observations used in its information estimate, as described next.
Local increments and nearest-neighbor information estimation
Mixed-path evaluation. For each selected pair t t t , define the two mixed-path samples
S m i x ( j ) ( t , a ) = ( Z ~ 1 , U ( j ) ( j ) , … , Z ~ t − 1 , U t − 1 ( j ) ( j ) , Z ~ t , a ( j ) , Z ~ t + 1 , 1 − U t + 1 ( j ) ( j ) , … , Z ~ n , 1 − U n ( j ) ( j ) ) , a ∈ { 0 , 1 } . S_{\mathrm{mix}}^{(j)}(t,a)=\left(\widetilde{Z}_{1,U^{(j)}}^{(j)},\ldots,\widetilde{Z}_{t-1,U_{t-1}^{(j)}}^{(j)},\widetilde{Z}_{t,a}^{(j)},\widetilde{Z}_{t+1,1-U_{t+1}^{(j)}}^{(j)},\ldots,\widetilde{Z}_{n,1-U_n^{(j)}}^{(j)}\right),\quad a\in\{0,1\}. S mix ( j ) ( t , a ) = ( Z 1 , U ( j ) ( j ) , … , Z t − 1 , U t − 1 ( j ) ( j ) , Z t , a ( j ) , Z t + 1 , 1 − U t + 1 ( j ) ( j ) , … , Z n , 1 − U n ( j ) ( j ) ) , a ∈ { 0 , 1 } .
These are the empirical counterparts of S ( t , a ) S(t,a) S ( t , a ) in Theorem 2. For each branch-trained predictor, we compute the signed increment
D ^ j , t , b , s = Φ ^ ( f j , t , b , s ; S m i x ( j ) ( t , 0 ) ) − Φ ^ ( f j , t , b , s ; S m i x ( j ) ( t , 1 ) ) . \widehat{D}_{j,t,b,s}=\widehat{\Phi}\left(f_{j,t,b,s};S_{\mathrm{mix}}^{(j)}(t,0)\right)-\widehat{\Phi}\left(f_{j,t,b,s};S_{\mathrm{mix}}^{(j)}(t,1)\right). D j , t , b , s = Φ ( f j , t , b , s ; S mix ( j ) ( t , 0 ) ) − Φ ( f j , t , b , s ; S mix ( j ) ( t , 1 ) ) .
The predictor is fixed across these two evaluations, while the auxiliary optimization defining Φ ^ \widehat{\Phi} Φ is performed separately on each evaluation sample. The mixed-path samples used for this increment differ from the branch training and validation samples used to measure the observed gap.
Information estimate from 14 observations. For each fixed configuration–pair context ( j , t ) (j,t) ( j , t ) , we use
{ ( D ^ j , t , b , s , b ) : b ∈ { 0 , 1 } , s ∈ { 0 , 1 , … , 6 } } \left\{\left(\widehat{D}_{j,t,b,s},b\right):b\in\{0,1\},\ s\in\{0,1,\ldots,6\}\right\} { ( D j , t , b , s , b ) : b ∈ { 0 , 1 } , s ∈ { 0 , 1 , … , 6 } }
to estimate the mutual information between the continuous signed increment and the discrete selector. There are 14 increment–selector observations: seven from each selector value. The supersample and U t ( j ) U_t^{(j)} U t ( j ) remain fixed within this estimate; the seven model seeds vary the learner’s randomness. Following the nearest-neighbor approach of Futami and Fujisawa [11 ] , we use a continuous–discrete k k k -nearest-neighbor MI estimator with k = 1 k=1 k = 1 . The implementation uses mutual_info_classif with a continuous increment feature, a discrete selector label, and random_state=0. If the available samples are insufficient to estimate dependence, for example if only one selector value is present, the implementation returns zero. MI estimates are truncated below at zero before taking any square root. We denote this nonnegative context-specific estimate by I ^ j , t L O O \widehat{I}_{j,t}^{\mathrm{LOO}} I j , t LOO , reflecting the leave-one-out conditioning discussed in Appendix § 15 .§ 2 . No absolute value is applied to D ^ j , t , b , s \widehat{D}_{j,t,b,s} D j , t , b , s before MI estimation. At each sample size, each objective therefore has J m = 30 Jm=30 J m = 30 context-specific MI estimates, each based on 14 observations.
Aggregation of the quantities in Figure 2
Observed absolute validation–training gap. For each trained predictor, define the signed branch-level gap
g j , t , b , s = Φ ^ ( f j , t , b , s ; S v a l ( j ) ( t , b ) ) − Φ ^ ( f j , t , b , s ; S t r ( j ) ( t , b ) ) . g_{j,t,b,s}=\widehat{\Phi}\left(f_{j,t,b,s};S_{\mathrm{val}}^{(j)}(t,b)\right)-\widehat{\Phi}\left(f_{j,t,b,s};S_{\mathrm{tr}}^{(j)}(t,b)\right). g j , t , b , s = Φ ( f j , t , b , s ; S val ( j ) ( t , b ) ) − Φ ( f j , t , b , s ; S tr ( j ) ( t , b ) ) .
We first take the absolute value of each branch-level gap and then average over model seeds, both selector values, and the ten selected pairs:
G Φ , j = 1 2 R m ∑ t ∈ T j ∑ b = 0 1 ∑ s = 0 R − 1 ∣ g j , t , b , s ∣ , G ^ Φ = 1 J ∑ j = 1 J G Φ , j . G_{\Phi,j}=\frac{1}{2Rm}\sum_{t\in T_j}\sum_{b=0}^{1}\sum_{s=0}^{R-1}\left|g_{j,t,b,s}\right|,\qquad\widehat{G}_{\Phi}=\frac{1}{J}\sum_{j=1}^{J}G_{\Phi,j}. G Φ , j = 2 R m 1 t ∈ T j ∑ b = 0 ∑ 1 s = 0 ∑ R − 1 ∣ g j , t , b , s ∣ , G Φ = J 1 j = 1 ∑ J G Φ , j .
The gap curve plots G ^ Φ \widehat{G}_{\Phi} G Φ . Thus, the absolute value precedes all averaging. Theorem 2 instead concerns the absolute value of the expected signed gap; the plotted mean absolute gap is a diagnostic summary.
Local-information proxy. For each context, we first transform its MI estimate into β Φ 2 I ^ j , t L O O \beta_{\Phi}\sqrt{2\widehat{I}_{j,t}^{\mathrm{LOO}}} β Φ 2 I j , t LOO , where β Φ = 2 \beta_{\Phi}=2 β Φ = 2 for MC and β Φ = α − 1 = 1.25 \beta_{\Phi}=\alpha^{-1}=1.25 β Φ = α − 1 = 1.25 for CvAR with α = 0.2 \alpha=0.2 α = 0.2 . We then average over the ten selected pairs within each configuration, followed by an average over configurations:
P Φ , j = β Φ m ∑ t ∈ T j 2 I ^ j , t L O O , B ^ Φ = 1 J ∑ j = 1 J P Φ , j . P_{\Phi,j}=\frac{\beta_{\Phi}}{m}\sum_{t\in T_j}\sqrt{2\widehat{I}_{j,t}^{\mathrm{LOO}}},\qquad\widehat{B}_{\Phi}=\frac{1}{J}\sum_{j=1}^{J}P_{\Phi,j}. P Φ , j = m β Φ t ∈ T j ∑ 2 I j , t LOO , B Φ = J 1 j = 1 ∑ J P Φ , j .
The proxy curve plots B ^ Φ \widehat{B}_{\Phi} B Φ . The square root is therefore taken separately for each context before either average.
Shaded bands and interpretation. Only the proxy curve has a shaded band. At each sample size, its width is determined by one standard deviation across the three supersample-level proxies, computed with divisor J J J :
s Φ = { 1 J ∑ j = 1 J ( P Φ , j − B ^ Φ ) 2 } 1 / 2 , J = 3. s_{\Phi} = \left\{ \frac{1}{J} \sum_{j=1}^{J} \left(P_{\Phi,j} - \widehat{B}_{\Phi}\right)^2 \right\}^{1/2}, \qquad J = 3. s Φ = { J 1 j = 1 ∑ J ( P Φ , j − B Φ ) 2 } 1/2 , J = 3.
The lower and upper band endpoints are max { 10 − 12 , B ^ Φ − s Φ } \max\{10^{-12}, \widehat{B}_{\Phi} - s_{\Phi}\} max { 1 0 − 12 , B Φ − s Φ } and B ^ Φ + s Φ \widehat{B}_{\Phi} + s_{\Phi} B Φ + s Φ , respectively. The mean and standard deviation are computed on the original scale before displaying the curves on a logarithmic vertical axis. The seven model seeds are used within each MI estimate; the band uses supersample configurations as its replication units. As discussed in Appendix § 15 .§ 2 , population information comparisons do not certify this average of context-specific square roots as an upper bound. Finite-sample nearest-neighbor MI estimates can also be biased. We therefore interpret the reported information quantity as a diagnostic proxy.