Introduction
Sample average approximation (SAA) replaces an unknown expectation by its empirical counterpart and then solves the resulting optimization problem. It is model-free, com putationally transparent, and supported by a mature theory of consistency, asymptotic distributions, and finite-sample guarantees; see, for example, Shapiro [31 ] and the monograph treatment in Shapiro et al. [33 ] . A more demanding question is whether SAA is statistically optimal for the particular optimization instance and uncertainty model at hand . Standard worst-case rates do not answer this question. The attainable regret can vary sharply with the local geometry of the feasible region, and the optimality of the empirical mean can additionally depend on how distributional perturbations interact with the tails of the data.
We study these issues in the canonical linear stochastic optimization problem
max w ∈ Ω w T μ , (1) \max_{w \in\Omega} \mathbf{w}^{\mathsf{T}}\boldsymbol{\mu}, \tag*{(1)} w ∈ Ω max w T μ , (1)
where Ω ⊂ R d \Omega\subset\mathbb{R}^{d} Ω ⊂ R d is compact and convex and μ \boldsymbol{\mu} μ is unknown. Its optimal value is the support function
f ∗ ( z ) = sup w ∈ Ω w T z , f_{*}(z) = \sup_{w \in\Omega} \mathbf{w}^{\mathsf{T}}z, f ∗ ( z ) = w ∈ Ω sup w T z ,
and we write w ∗ ( z ) \mathbf{w}_{*}(z) w ∗ ( z ) for a selected maximizer. Given n n n independent observations, x 1 , … , x n \boldsymbol{x}_{1}, \ldots, \boldsymbol{x}_{n} x 1 , … , x n , with mean μ \boldsymbol{\mu} μ , ordinary SAA is the plug-in policy w ∗ ( x ˉ ) \mathbf{w}_{*}(\bar{\boldsymbol{x}}) w ∗ ( x ˉ ) where x ˉ \bar{\boldsymbol{x}} x ˉ is the sample mean. We evaluate a data-dependent policy through decision regret and compare it with all measurable sample-based policies.
Our analysis considers two local statistical experiments. We first study Gaussian observations with covariance Σ \Sigma Σ and local mean perturbations μ + t / n \boldsymbol{\mu} + \boldsymbol{t}/\sqrt{n} μ + t / n . We then allow the entire data-generating distribution to vary in a Kullback–Leibler neighborhood of a reference distribution P P P ,
Q n , M K L ( P ) = { Q ≪ P : E Q ∥ x ∥ 2 < ∞ , D K L ( Q ∥ P ) ≤ M 2 2 n } . \mathcal{Q}_{n,M}^{\mathrm{KL}}(P) = \left\{Q \ll P : \mathbb{E}_{Q}\lVert x\rVert_{2} < \infty,\ D_{\mathrm{KL}}(Q\mathbin{\|}P) \le\frac{M^{2}}{2n}\right\}. Q n , M KL ( P ) = { Q ≪ P : E Q ∥ x ∥ 2 < ∞ , D KL ( Q ∥ P ) ≤ 2 n M 2 } .
The second experiment is locally distributionally robust: it permits changes in the mean, covariance, and tails, subject only to a shrinking entropy budget. In both experiments, the central deterministic object is the local support-function remainder
r μ ( h ) = f ∗ ( μ + h ) − f ∗ ( μ ) − w ∗ ( μ ) T h . r_{\boldsymbol{\mu}}(h) = f_{*}(\boldsymbol{\mu} + h) - f_{*}(\boldsymbol{\mu}) - \mathbf{w}_{*}(\boldsymbol{\mu})^{\mathsf{T}}h. r μ ( h ) = f ∗ ( μ + h ) − f ∗ ( μ ) − w ∗ ( μ ) T h .
Statistical uncertainty determines the scale and directions in which μ \boldsymbol{\mu} μ can be learned; the local growth of r μ r_{\boldsymbol{\mu}} r μ converts that uncertainty into regret.
This viewpoint yields both an optimality result and a limitation of SAA. In the Gaussian experiment, SAA adapts to the local geometry and attains the local minimax rate: the regret is of order n − 1 n^{-1} n − 1 at ordinary smooth instances and n − 1 / 2 n^{-1/2} n − 1/2 at nonsmooth instances, and degenerate curvature can produce faster rates such as n − 2 n^{-2} n − 2 or non-power rates. On KL neighborhoods of radius 1 / n 1/n 1/ n , the exact local minimax benchmark is governed by the same support-function geometry, and a coordinatewise truncated version of SAA attains the minimax rate. Ordinary SAA, however, need not attain even the rate: rare but KL-admissible tail perturbations can dominate its expected regret when the local geometric rate is faster than the available tail control for the empirical mean. Thus there is no universal answer to whether SAA is optimal. It can be sharply optimal, merely rate-optimal, or strictly suboptimal, depending jointly on local geometry and distributional tails.
Main contributions
More specifically, our results can be summarized as follows.
General geometry and exact Gaussian local risk. If
r μ ( ε t ) = a ( ε ) ψ ( t ) + o ( a ( ε ) ) r_{\mu}(\varepsilon t) = a(\varepsilon)\psi(t) + o(a(\varepsilon)) r μ ( εt ) = a ( ε ) ψ ( t ) + o ( a ( ε ))
uniformly on bounded sets, the Gaussian local minimax regret, normalized by a ( n − 1 / 2 ) a(n^{-1/2}) a ( n − 1/2 ) , converges to the Bayes Jensen-gap functional V M , Σ ( ψ ) V_{M,\Sigma}(\psi) V M , Σ ( ψ ) in (26) . When ψ \psi ψ is nonaffine on the local parameter ball, this constant is positive and SAA attains the rate. This covers nonsmooth, quadratic, degenerate, and non-power geometry in a common decision-theoretic representation. One fixed strongly convex feasible set exhibits rates n − 1 / 2 n^{-1/2} n − 1/2 , n − 1 n^{-1} n − 1 , and n − 2 n^{-2} n − 2 at different centers; a separate compact convex set gives 1 / ( n log n ) 1/(\sqrt{n}\log n) 1/ ( n log n ) . Finite-sample upper and lower bounds in terms of a symmetric local optimization modulus dispense with a local expansion altogether.
Exact local minimax risk on KL neighborhoods. For the distributional neighborhood Q n , M K L ( P ) Q^{\mathrm{KL}}_{n,M}(P) Q n , M KL ( P ) , we establish the exact local minimax limit
R n , M K L ( P ) a ( n − 1 / 2 ) ⟶ V M , I d ( ψ ∘ Σ 1 / 2 ) , \frac{R^{\mathrm{KL}}_{n,M}(P)}{a(n^{-1/2})} \longrightarrow V_{M,\mathrm{Id}}\left(\psi\circ\Sigma^{1/2}\right), a ( n − 1/2 ) R n , M KL ( P ) ⟶ V M , Id ( ψ ∘ Σ 1/2 ) ,
where V M , I d V_{M,\mathrm{Id}} V M , Id is given by (26) and Σ \Sigma Σ is the covariance matrix of x x x under P P P , under a local exponential-moment condition; the matching lower bound requires only a finite second moment and nonsingular covariance. The rule attaining the exact constant uses knowledge of P P P and M M M ; the truncated mean introduced below does not, and attains the rate. The same limit arises in a matched Gaussian experiment with local alternatives μ + Σ 1 / 2 u / n \mu+\Sigma^{1/2}u/\sqrt{n} μ + Σ 1/2 u / n , so the KL neighborhood is asymptotically summarized by the covariance ellipsoid of the reference distribution while the loss is summarized by the local geometry ψ \psi ψ . Whenever the limiting local modulus is nonaffine, a coordinatewise truncated empirical mean, followed by optimization, is rate-minimax. At smooth instances, its large-local-ball constant reduces to tr ( Σ ∇ 2 f ∗ ( μ ) ) / 2 \operatorname{tr}(\Sigma\nabla^{2}f_{*}(\mu))/2 tr ( Σ ∇ 2 f ∗ ( μ )) /2 , which is also the local minimax constant. In this sense, local distributional robustness need not change first-order efficiency, although it changes the class of distributions against which an estimator must perform uniformly.
When ordinary SAA is suboptimal. We compare ordinary SAA with the truncated plug-in policy uniformly over the KL neighborhood. A geometry-weighted tail criterion supplies sufficient conditions for rate-optimality and for equality of the leading maximal regrets of ordinary and truncated SAA. For power-type geometry a ( ε ) = ε α a(\varepsilon)=\varepsilon^\alpha a ( ε ) = ε α and a reference distribution satisfying an exponential-power moment condition E P exp ( c ∥ x ∥ 2 ζ ) < ∞ \mathbb{E}_{P}\exp(c\lVert x\rVert_{2}^{\zeta})<\infty E P exp ( c ∥ x ∥ 2 ζ ) < ∞ , ordinary SAA has maximal regret bounded by
O ( n − α / 2 + n − ζ ) . O(n^{-\alpha/2}+n^{-\zeta}). O ( n − α /2 + n − ζ ) .
It is rate-optimal when α ≤ 2 ζ \alpha\leq2\zeta α ≤ 2 ζ , and it has the same leading maximal-regret constant as truncated SAA when α < 2 ζ \alpha<2\zeta α < 2 ζ . Matching examples establish rate sharpness over the stated class. In particular, for Laplace tails (ζ = 1 \zeta=1 ζ = 1 ) and quartic local geometry (α = 4 \alpha=4 α = 4 ), the local minimax and truncated-SAA risks are of order n − 2 n^{-2} n − 2 , whereas ordinary SAA has worst-case regret of order n − 1 n^{-1} n − 1 . The failure is not visible from the Gaussian weak limit of the sample mean; it arises from lack of uniform integrability under rare distributional perturbations.
Smooth efficiency as a special case. At a C 2 C^{2} C 2 instance, a ( ε ) = ε 2 a(\varepsilon)=\varepsilon^{2} a ( ε ) = ε 2 and ψ ( t ) = t T H μ t / 2 \psi(t)=t^{\mathsf{T}}H_{\mu}t/2 ψ ( t ) = t T H μ t /2 , where H μ = ∇ 2 f ∗ ( μ ) H_{\mu}=\nabla^{2}f_{*}(\mu) H μ = ∇ 2 f ∗ ( μ ) . The expanding-ball Gaussian minimax constant is tr ( Σ H μ ) / 2 \operatorname{tr}(\Sigma H_{\mu})/2 tr ( Σ H μ ) /2 , attained by SAA without knowledge of Σ \Sigma Σ . Under the boundary regularity conditions in § 4 , this equals
1 2 ∥ μ ∥ 2 2 ∑ i = 1 d − 1 u i T Σ u i κ i ( μ ) . \frac{1}{2\lVert\mu\rVert_{2}^{2}}\sum_{i=1}^{d-1}\frac{\mathbf{u}_{i}^{\mathsf{T}}\Sigma\mathbf{u}_{i}}{\kappa_{i}(\mu)}. 2 ∥ μ ∥ 2 2 1 i = 1 ∑ d − 1 κ i ( μ ) u i T Σ u i .
where u i \mathbf{u}_{i} u i and κ i ( μ ) \kappa_{i}(\mu) κ i ( μ ) are the principal tangent directions and curvatures. Thus tangent noise is weighted by inverse curvature. Truncated SAA has the same smooth efficiency constant on KL neighborhoods after the local radius tends to infinity.
Relation to existing literature
Classical SAA asymptotics. The statistical theory of SAA establishes consistency, asymptotic distributions of optimal values and solutions, and sample-size guarantees under increasingly general stochastic-programming models; representative references include Shapiro [31 , 32 ] ; Shapiro et al. [33 ] . Our focus differs in two respects. First, the target is regret rather than estimation error of the optimizer or optimal value. Second, we compare SAA with the local minimax risk over all measurable policies. This asks an efficiency question rather than only deriving the sampling distribution of SAA. The support-function structure of (1) makes it possible to identify sharp constants, characterize nonquadratic rates, and distinguish weak convergence from convergence of expected regret under distributional perturbations.
Local minimax and instance-optimal stochastic optimization. The closest conceptual antecedent is Duchi and Ruan (2021), who develop a Hájek–Le Cam-type local minimax theory for stochastic convex optimization and relate statistical difficulty to tilt stability, together with asymptotically optimal online algorithms. Our linear setting permits a different level of geometric resolution. We obtain an exact Bayes–minimax representation in terms of the support-function remainder, sharp constants expressed through principal curvatures, finite-sample modulus bounds, and a theory of degenerate and non-power rates. We also extend the local analysis from a Gaussian mean experiment to population-centered KL neighborhoods and identify when the empirical-mean implementation ceases to attain the benchmark. The smooth trace constant serves as the classical benchmark; the additional conclusions are the exact fixed-radius Jensen-gap limit for general local support growth, the corresponding full-KL minimax limit, and a separation of regret rates caused by rare tails.
A recent and complementary contribution is Jiang and Pananjady [20 ] , who study finite-sample instance optimality for unconstrained smooth strongly convex stochastic optimization with additive and multiplicative oracle noise. They show that SAA and averaged stochastic approximation can be substantially suboptimal at realistic sample sizes and construct an instance-optimal variance-reduced method. Their loss, oracle model, and finite-sample objective differ from ours. Together, the two perspectives emphasize that the optimality of SAA is experiment-, loss-, and instance-dependent.
Distributionally robust optimization and shrinking divergence neighborhoods. KL and more general divergence neighborhoods have been used both to quantify model sensitivity and to construct statistically calibrated robust procedures. Lam [21 ] derives infinitesimal expansions for worst-case performance over shrinking KL neighborhoods. Lam [22 ] and Duchi et al. [7 ] connect empirical divergence-based DRO with empirical likelihood, asymptotically calibrated confidence bounds, and variance regularization, while He and Lam [17 ] develop higher-order expansions and Bartlett corrections for divergence-based DRO. From a large-deviations perspective, Van Parys et al. [37 ] show that relative-entropy DRO is optimal for controlling out-of-sample disappointment. Wasserstein and other data-driven ambiguity sets, which are centered at the empirical distribution, are studied in Mohajerin Esfahani and Kuhn [29 ] ; Bertsimas et al. [3 ] ; Gao and Kleywegt [12 ] . Complementary work asks whether robust or optimistic reformulations can improve the out-of-sample behavior of SAA in particular model classes; see Anderson and Philpott [2 ] and Gotoh et al. [13 ] .
Our KL neighborhood is centered at a fixed reference distribution rather than the empirical distribution, and our target is different. We characterize the minimax decision regret over all measurable policies, not the value of a particular robust optimization formulation or the coverage of an uncertainty set. The radius 1 / n 1/n 1/ n generates a local asymptotic experiment in which covariance and support geometry determine the minimax benchmark. The comparison between ordinary and truncated SAA then exposes an additional phenomenon: two procedures can have the same Gaussian weak limit but different maximal expected regret because KL-admissible rare events affect uniform integrability. This is the mechanism behind the suboptimality of ordinary SAA at sufficiently fast geometric rates.
Predict-then-optimize and decision-focused learning. For linear downstream optimization, the smart predict-then-optimize framework of Elmachtoub and Grigas [9 ] evaluates predictions through induced decision regret rather than prediction error. Generalization analyses such as El Balghiti et al. [8 ] highlight the role of margins around parameter values that lead to nonunique decisions and introduce geometric strength conditions on the feasible set. Our ambiguity set plays a related geometric role, but our question is different: we fix a local statistical experiment and characterize its minimax regret, including exact constants and the effect of distributional tails, rather than bounding generalization error over prediction classes.
Nonsmooth functional and norm estimation. A separate literature studies minimax estimation of nonsmooth functionals and norms in Gaussian models. The connection is direct: for p ⋆ = p / ( p − 1 ) p^\star= p/(p-1) p ⋆ = p / ( p − 1 ) and Ω = B p ⋆ d = { w : ∥ w ∥ p ⋆ ≤ 1 } \Omega= B_{p^\star}^d = \{w : \|w\|_{p^\star} \le1\} Ω = B p ⋆ d = { w : ∥ w ∥ p ⋆ ≤ 1 } , we have f ⋆ ( μ ) = ∥ μ ∥ p f_\star(\mu) = \|\mu\|_p f ⋆ ( μ ) = ∥ μ ∥ p , so estimating an ℓ p \ell_p ℓ p -norm is estimating the optimal value of the same stochastic optimization problem. Relevant work includes Lepski et al. [27 ] on L r L_r L r -norms in nonparametric Gaussian models, Cai and Low [4 ] on the ℓ 1 \ell_1 ℓ 1 functional in the Gaussian sequence model, and the extensions to general L r L_r L r -norms (Han et al., [16 ] ) and to ℓ γ \ell_\gamma ℓ γ -functionals over sparse normal means (Collier et al., [5 ] ).
Our target is different: we evaluate a data-dependent decision through regret rather than estimating f ⋆ ( μ ) f_\star(\mu) f ⋆ ( μ ) or w ⋆ ( μ ) w_\star(\mu) w ⋆ ( μ ) . Regret removes the affine first-order component of the value functional, so the local remainder becomes the governing object, and value estimation and decision regret can have different local orders. At a differentiable nonzero point, ∥ μ ∥ p \|\mu\|_p ∥ μ ∥ p is a regular scalar functional with root-n n n estimation error, whereas decision regret is typically second order. At degenerate points the contrast is larger: for p > 2 p > 2 p > 2 and μ = r e 1 \mu= r e_1 μ = r e 1 , the value remains first-order regular in the radial direction, while the transverse decision regret has order n − p / 2 n^{-p/2} n − p /2 . These faster-order instances are also where tail stabilization can become necessary under KL robustness; see Examples 4.10 and 6.5.
Empirical optimization, model structure, and ETO/IEO. A closely related line of work led by Lam asks when empirical optimization can be statistically improved. Lam [24 ] proves, under suitable smoothness with respect to the data-generating distribution, that the first-order optimality-gap behavior of alternative data-driven formulations is second-order stochastically dominated by empirical optimization. Our smooth, nondegenerate results are consistent with this efficiency principle: ordinary SAA is sharply optimal in the quadratic regime of the Gaussian experiment. Our KL counterexamples delineate a different regime. When the leading quadratic curvature vanishes and a faster regret rate is attainable, rare tail perturbations can make the raw empirical mean suboptimal even though a truncated empirical optimizer remains rate-minimax.
Other work identifies mechanisms that improve on SAA through additional structure. Lam and Wang [24 ] show that first-order improvements can arise when side information is correctly specified and geometrically effective. The comparison of SAA with estimate-then-optimize (ETO) and integrated-estimation-optimization (IEO) is similar in spirit: Elmachtoub et al. [11 ] show that under correct parametric specification ETO asymptotically dominates IEO in the sense of stochastic dominance of regret and, when the comparison applies, can also dominate SAA, whereas under misspecification the ordering can reverse. Elmachtoub et al. [10 ] isolate the impact of misspecification, and Lan et al. [25 ] study local misspecification through contiguity and derive an explicit bias–variance tradeoff among SAA, ETO, and IEO. In these works the gain comes from model structure that reduces effective estimation variance. The mechanism in our KL results is different: tail stabilization can improve maximal expected regret without altering the Gaussian local information. Consequently, no unconditional ranking among data-driven procedures should be expected; which procedure is preferable depends on whether model structure or tail behavior is the binding constraint.
Organization
§ 2 contrasts the global Gaussian-width benchmark with a local quadratic-growth bound. § 3 develops the sharp Gaussian local minimax theory, including the Bayes representation, smooth efficiency bound, general local scales, and a finite-sample optimization modulus. § 4 translates these statistical results into boundary curvature, ambiguity, and contact geometry, and illustrates the resulting range of rates. § 5 first introduces a matched ellipsoidal Gaussian benchmark, then establishes the exact local minimax risk over shrinking KL neighborhoods and quantifies the price of robustness. § 6 compares ordinary and truncated SAA and gives sharp conditions for optimality or suboptimality. § 7 illustrates the Gaussian rates and the geometry–tail transition numerically. § 8 concludes. All proofs, intermediate technical results, and detailed calculations are collected in the Online Supplement. Table 2 in § 6 summarizes the benchmarks and procedure-specific risks used throughout.
Table 2. Statistical benchmarks and procedure-specific maximal risks used in the paper.
Global Complexity and Local Geometric Fast Rates
Problem setup
Let Ω ⊂ R d \Omega\subset\mathbb{R}^{d} Ω ⊂ R d be a compact convex set containing 0 \mathbf{0} 0 , and define its support function by
f ∗ ( z ) : = max w ∈ Ω w ⊤ z , z ∈ R d . (2) f_{*}(z) := \max_{w \in\Omega} w^{\top}z,\qquad z \in\mathbb{R}^{d}. \tag*{(2)} f ∗ ( z ) := w ∈ Ω max w ⊤ z , z ∈ R d . (2)
For each z z z , let
W ∗ ( z ) : = arg max w ∈ Ω w ⊤ z , \mathcal{W}_{*}(z) := \arg\max_{w \in\Omega} w^{\top}z, W ∗ ( z ) := arg w ∈ Ω max w ⊤ z ,
The measurable maximum theorem gives a Borel measurable selection; fix one and denote it by w ∗ ( z ) ∈ W ∗ ( z ) w_{*}(z) \in\mathcal{W}_{*}(z) w ∗ ( z ) ∈ W ∗ ( z ) [1 Theorem 18.19] . The ambiguity set is
A : = { z ∈ R d : W ∗ ( z ) is not a singleton } . (3) A := \{z \in\mathbb{R}^{d}: \mathcal{W}_{*}(z) \text{ is not a singleton}\}. \tag*{(3)} A := { z ∈ R d : W ∗ ( z ) is not a singleton } . (3)
For a true objective vector μ \mu μ and a feasible decision w ∈ Ω w \in\Omega w ∈ Ω , define the regret
R μ ( w ) : = f ∗ ( μ ) − w ⊤ μ . (4) R_{\mu}(w) := f_{*}(\mu) - w^{\top}\mu. \tag*{(4)} R μ ( w ) := f ∗ ( μ ) − w ⊤ μ . (4)
We suppress the subscript μ \mu μ when the underlying instance is clear. Unless stated otherwise, d d d , Ω \Omega Ω , and μ \mu μ are fixed as n → ∞ n \to\infty n → ∞ , and
R Ω : = sup w ∈ Ω ∥ w ∥ 2 < ∞ . R_{\Omega} := \sup_{w \in\Omega} \lVert w\rVert_{2} < \infty. R Ω := w ∈ Ω sup ∥ w ∥ 2 < ∞.
We observe
x 1 , … , x n ∼ i . i . d . N ( μ , Σ ) , Σ ∈ C : = { Γ = Γ ⊤ : λ I d ⪯ Γ ⪯ λ ‾ I d } , (5) x_{1},\ldots,x_{n} \mathrel{\stackrel{\mathrm{i.i.d.}}{\sim}} \mathcal{N}(\mu,\Sigma),\qquad\Sigma\in\mathcal{C} := \{\Gamma= \Gamma^{\top}: \lambda I_{d} \preceq\Gamma\preceq\overline{\lambda}I_{d}\}, \tag*{(5)} x 1 , … , x n ∼ i.i.d. N ( μ , Σ ) , Σ ∈ C := { Γ = Γ ⊤ : λ I d ⪯ Γ ⪯ λ I d } , (5)
where 0 < λ ≤ λ ‾ < ∞ 0 < \lambda\leq\overline{\lambda} < \infty 0 < λ ≤ λ < ∞ are fixed constants. The covariance matrix Σ \Sigma Σ is a fixed element of C \mathcal{C} C that enters only through the benchmarks and the constants below: the SAA policy defined next does not use Σ \Sigma Σ or the constants λ , λ ‾ \lambda,\overline{\lambda} λ , λ . Writing
x ˉ : = 1 n ∑ i = 1 n x i , \bar{x} := \frac{1}{n}\sum_{i=1}^{n}x_{i}, x ˉ := n 1 i = 1 ∑ n x i ,
sample average approximation (SAA) is the plug-in policy
w ^ S A A = w ∗ ( x ˉ ) . \widehat{w}_{\mathrm{SAA}} = w_{*}(\bar{x}). w SAA = w ∗ ( x ˉ ) .
For a compact set D ⊂ R d D \subset\mathbb{R}^{d} D ⊂ R d , let
G ( D ) : = E [ sup w ∈ D w ⊤ G ] , G ∼ N ( 0 , I d ) , (6) \mathcal{G}(D) := \mathbb{E}\left[\sup_{w \in D}w^{\top}G\right],\qquad G \sim\mathcal{N}(0,I_{d}), \tag*{(6)} G ( D ) := E [ w ∈ D sup w ⊤ G ] , G ∼ N ( 0 , I d ) , (6)
be its Gaussian width.
A global benchmark governed by Gaussian width
We first consider performance uniformly over all μ ∈ R d \mu\in\mathbb{R}^d μ ∈ R d . For a fixed covariance matrix Σ \Sigma Σ , define the global minimax regret
R n Σ ( Ω ) : = inf w ^ n sup μ ∈ R d E μ , Σ R μ ( w ^ n ) , (7) \mathcal{R}^{\Sigma}_{n}(\Omega) := \inf_{\widehat{w}_n}\sup_{\mu\in\mathbb{R}^d} \mathbb{E}_{\mu,\Sigma}R_\mu(\widehat{w}_n), \tag*{(7)} R n Σ ( Ω ) := w n inf μ ∈ R d sup E μ , Σ R μ ( w n ) , (7)
where the infimum is over measurable maps w ^ n : ( R d ) n → Ω \widehat{w}_n : (\mathbb{R}^d)^n \to\Omega w n : ( R d ) n → Ω .
Proposition 2.1 (Global minimax regret). *For every positive definite Σ \Sigma Σ and every n ≥ 1 n \ge1 n ≥ 1 ,
( 1 − 1 2 ) G ( Σ 1 / 2 Ω ) n ≤ R n Σ ( Ω ) ≤ sup μ ∈ R d E μ , Σ R μ ( w ∗ ( x ˉ ) ) ≤ 2 G ( Σ 1 / 2 Ω ) n . (8) \left(1-\frac{1}{\sqrt{2}}\right)\frac{\mathcal{G}(\Sigma^{1/2}\Omega)}{\sqrt{n}} \le\mathcal{R}^{\Sigma}_{n}(\Omega) \le\sup_{\mu\in\mathbb{R}^d}\mathbb{E}_{\mu,\Sigma}R_\mu(w_*(\bar{x})) \le2\frac{\mathcal{G}(\Sigma^{1/2}\Omega)}{\sqrt{n}}. \tag*{(8)} ( 1 − 2 1 ) n G ( Σ 1/2 Ω ) ≤ R n Σ ( Ω ) ≤ μ ∈ R d sup E μ , Σ R μ ( w ∗ ( x ˉ )) ≤ 2 n G ( Σ 1/2 Ω ) . (8)
The global benchmark depends on the geometry of the entire feasible set through G ( Σ 1 / 2 Ω ) \mathcal{G}(\Sigma^{1/2}\Omega) G ( Σ 1/2 Ω ) . Moreover, w ∗ ( x ˉ ) w_*(\bar{x}) w ∗ ( x ˉ ) attains the global minimax rate without using Σ \Sigma Σ . For a sequence of problems, global minimax consistency is possible precisely, up to universal constants, when G ( Σ 1 / 2 Ω ) = o ( n ) \mathcal{G}(\Sigma^{1/2}\Omega) = o(\sqrt{n}) G ( Σ 1/2 Ω ) = o ( n ) . Under the fixed-dimensional asymptotics used in the remainder of the paper, this condition holds automatically; the role of Proposition 2.1 is to provide a global baseline against which the instance-specific local rates can be compared.
A local fast rate from quadratic growth
The global rate does not use the location of μ \mu μ . Faster rates become possible when the optimizer is locally stable. Let ∥ ⋅ ∥ Ω \|\cdot\|_\Omega ∥ ⋅ ∥ Ω and ∥ ⋅ ∥ K \|\cdot\|_K ∥ ⋅ ∥ K be a pair of dual norms, and define
dist ( z , A ) : = inf a ∈ A ∥ z − a ∥ K . (9) \operatorname{dist}(z,A) := \inf_{a \in A}\|z-a\|_K. \tag*{(9)} dist ( z , A ) := a ∈ A inf ∥ z − a ∥ K . (9)
A natural choice is available when Ω = − Ω \Omega=-\Omega Ω = − Ω and 0 ∈ int Ω \mathbf{0} \in\operatorname{int}\Omega 0 ∈ int Ω : then
∥ w ∥ Ω : = inf { a > 0 : w ∈ a Ω } , K : = { z : f ∗ ( z ) ≤ 1 } , ∥ z ∥ K = f ∗ ( z ) . (10) \|w\|_\Omega:= \inf\{a>0:w\in a\Omega\}, \qquad K:=\{z:f_*(z)\le1\}, \qquad\|z\|_K=f_*(z). \tag*{(10)} ∥ w ∥ Ω := inf { a > 0 : w ∈ a Ω } , K := { z : f ∗ ( z ) ≤ 1 } , ∥ z ∥ K = f ∗ ( z ) . (10)
Assume that, for some c Ω > 0 c_\Omega>0 c Ω > 0 ,
z T ( w ∗ ( z ) − w ) ≥ c Ω dist ( z , A ) ∥ w − w ∗ ( z ) ∥ Ω 2 , z ∈ R d , w ∈ Ω . (11) z^\mathsf{T}(w_*(z)-w) \ge c_\Omega\operatorname{dist}(z,A)\|w-w_*(z)\|_\Omega^2,\qquad z\in\mathbb{R}^d,\quad w\in\Omega. \tag*{(11)} z T ( w ∗ ( z ) − w ) ≥ c Ω dist ( z , A ) ∥ w − w ∗ ( z ) ∥ Ω 2 , z ∈ R d , w ∈ Ω. (11)
This is a quadratic-growth condition for the exposed support point. It quantifies how rapidly the objective deteriorates as the decision moves away from the optimizer, with the strength of the inequality vanishing as z z z approaches the ambiguity set.
Proposition 2.2 (A covariance-dependent local fast-rate bound). *Suppose (11) holds at z = μ z=\mu z = μ and dist ( μ , A ) > 0 \operatorname{dist}(\mu,A)>0 dist ( μ , A ) > 0 . Then
E μ , Σ R μ ( w ∗ ( x ˉ ) ) ≤ E ∥ Σ 1 / 2 G ∥ K 2 c Ω n dist ( μ , A ) , G ∼ N ( 0 , I d ) . (12) \mathbb{E}_{\mu,\Sigma}R_\mu(w_*(\bar{x})) \le\frac{\mathbb{E}\|\Sigma^{1/2}G\|_K^2}{c_\Omega n\operatorname{dist}(\mu,A)},\qquad G\sim\mathcal{N}(\mathbf{0},I_d). \tag*{(12)} E μ , Σ R μ ( w ∗ ( x ˉ )) ≤ c Ω n dist ( μ , A ) E ∥ Σ 1/2 G ∥ K 2 , G ∼ N ( 0 , I d ) . (12)
If, in addition, the polar-norm convention (10) is in force, then
G ( Σ 1 / 2 Ω ) 2 ≤ E ∥ Σ 1 / 2 G ∥ K 2 ≤ ( 1 + π 2 ) G ( Σ 1 / 2 Ω ) 2 , (13) \mathcal{G}(\Sigma^{1/2}\Omega)^2 \leq\mathbb{E}\|\Sigma^{1/2}G\|_{K}^{2} \leq\left(1+\frac{\pi}{2}\right)\mathcal{G}(\Sigma^{1/2}\Omega)^2, \tag*{(13)} G ( Σ 1/2 Ω ) 2 ≤ E ∥ Σ 1/2 G ∥ K 2 ≤ ( 1 + 2 π ) G ( Σ 1/2 Ω ) 2 , (13)
and consequently
E μ , Σ R μ ( w ∗ ( x ˉ ) ) ≤ 1 + π / 2 c Ω G ( Σ 1 / 2 Ω ) 2 n dist ( μ , A ) ≤ ( 1 + π / 2 ) c ‾ c Ω G ( Ω ) 2 n dist ( μ , A ) . (14) \mathbb{E}_{\mu,\Sigma}R_{\mu}(w_*(\bar{x})) \leq\frac{1+\pi/2}{c_{\Omega}}\frac{\mathcal{G}(\Sigma^{1/2}\Omega)^2}{n\operatorname{dist}(\mu,A)} \leq\frac{(1+\pi/2)\overline{c}}{c_{\Omega}}\frac{\mathcal{G}(\Omega)^2}{n\operatorname{dist}(\mu,A)}. \tag*{(14)} E μ , Σ R μ ( w ∗ ( x ˉ )) ≤ c Ω 1 + π /2 n dist ( μ , A ) G ( Σ 1/2 Ω ) 2 ≤ c Ω ( 1 + π /2 ) c n dist ( μ , A ) G ( Ω ) 2 . (14)
Remark 2.3 (Midpoint strong convexity). If Ω = − Ω \Omega=-\Omega Ω = − Ω , 0 ∈ int Ω 0\in\operatorname{int}\Omega 0 ∈ int Ω , and Ω \Omega Ω is c Ω c_{\Omega} c Ω -strongly convex in the midpoint sense relative to ∥ ⋅ ∥ Ω \|\cdot\|_{\Omega} ∥ ⋅ ∥ Ω (so that ( u + v ) / 2 + s ∈ Ω (u+v)/2+s\in\Omega ( u + v ) /2 + s ∈ Ω whenever ∥ s ∥ Ω ≤ c Ω 2 ∥ u − v ∥ Ω 2 \|s\|_{\Omega}\leq\frac{c_{\Omega}}{2}\|u-v\|_{\Omega}^{2} ∥ s ∥ Ω ≤ 2 c Ω ∥ u − v ∥ Ω 2 ), then A = { 0 } A=\{0\} A = { 0 } and (11) holds with dist ( z , A ) = ∥ z ∥ K \operatorname{dist}(z,A)=\|z\|_{K} dist ( z , A ) = ∥ z ∥ K .
Proposition 2.1 shows that worst-case global regret is of order n − 1 / 2 n^{-1/2} n − 1/2 and is determined by the width of the entire feasible set. Proposition 2.2 shows how local stability can square the estimation error and improve the pointwise rate to n − 1 n^{-1} n − 1 . Its coefficient still combines a global quantity, G ( Ω ) \mathcal{G}(\Omega) G ( Ω ) , with the local margin dist ( μ , A ) \operatorname{dist}(\mu,A) dist ( μ , A ) , so it is a benchmark rather than a sharp characterization. The next section replaces this mixed bound by a local minimax theory based on the full support-function remainder.
Local Minimax Risk under Gaussian Sampling
Local experiments and minimax risks
In the original statistical problem, a policy has the form
w ^ n = w ^ n ( x 1 , … , x n ) , w ^ n : ( R d ) n → Ω . (15) \widehat{w}_n=\widehat{w}_n(x_1,\ldots,x_n),\qquad\widehat{w}_n:(\mathbb{R}^{d})^n\to\Omega. \tag*{(15)} w n = w n ( x 1 , … , x n ) , w n : ( R d ) n → Ω. (15)
Fix μ ∈ R d \mu\in\mathbb{R}^{d} μ ∈ R d and M > 0 M>0 M > 0 to evaluate local performance around μ \mu μ . For the lower-bound argument, define
R n , M Σ ( μ ) = inf w ^ sup t ∈ B 2 ( M ) E μ n , t , Σ [ f ∗ ( μ n , t ) − w ^ T μ n , t ] . (16) R_{n,M}^{\Sigma}(\mu)=\inf_{\widehat{w}}\sup_{t\in B_2(M)}\mathbb{E}_{\mu_{n,t},\Sigma}\left[f_*(\mu_{n,t})-\widehat{w}^{\mathsf{T}}\mu_{n,t}\right]. \tag*{(16)} R n , M Σ ( μ ) = w inf t ∈ B 2 ( M ) sup E μ n , t , Σ [ f ∗ ( μ n , t ) − w T μ n , t ] . (16)
where μ n , t = μ + t / n \mu_{n,t}=\mu+t/\sqrt{n} μ n , t = μ + t / n and B 2 ( M ) = { t ∈ R d : ∥ t ∥ 2 ≤ M } B_2(M)=\{t\in\mathbb{R}^{d}:\|t\|_2\leq M\} B 2 ( M ) = { t ∈ R d : ∥ t ∥ 2 ≤ M } . This is a pointwise local asymptotic minimax benchmark at the fixed reference center μ \mu μ ; we refer to optimality in this sense as instance optimality . As is standard in local asymptotic theory, a rule in R n , M Σ ( μ ) R_{n,M}^{\Sigma}(\mu) R n , M Σ ( μ ) may be indexed by the benchmark center μ \mu μ and the fixed covariance Σ \Sigma Σ , but not by the unknown local perturbation t t t . This convention gives the benchmark procedure more information than ordinary SAA, which is the sample-only rule w ∗ ( x ˉ ) w_*(\bar{x}) w ∗ ( x ˉ ) that uses neither the center nor Σ \Sigma Σ . In expressions evaluated under P μ n , t , Σ \mathbb{P}_{\mu_{n,t},\Sigma} P μ n , t , Σ , the notation R ( w ) R(w) R ( w ) means f ∗ ( μ n , t ) − w T μ n , t f_*(\mu_{n,t})-w^{\mathsf{T}}\mu_{n,t} f ∗ ( μ n , t ) − w T μ n , t . For every rule of the form (15) ,
sup t ∈ B 2 ( M ) E μ n , t , Σ R ( w ^ n ) ≥ R n , M Σ ( μ ) . (17) \sup_{t\in B_2(M)}\mathbb{E}_{\mu_{n,t},\Sigma}R(\widehat{w}_n)\geq R_{n,M}^{\Sigma}(\mu). \tag*{(17)} t ∈ B 2 ( M ) sup E μ n , t , Σ R ( w n ) ≥ R n , M Σ ( μ ) . (17)
Define the local remainder by
r μ ( u ) = f ∗ ( μ + u ) − f ∗ ( μ ) − w ∗ ( μ ) T u . (18) r_{\mu}(u) = f_{*}(\mu+ u) - f_{*}(\mu) - w_{*}(\mu)^{\mathsf{T}}u. \tag*{(18)} r μ ( u ) = f ∗ ( μ + u ) − f ∗ ( μ ) − w ∗ ( μ ) T u . (18)
This is nonnegative and convex, even when f ∗ f_{*} f ∗ is not differentiable.
At an ambiguous instance, the displayed remainder depends on the selected element of W ∗ ( μ ) W_{*}(\mu) W ∗ ( μ ) , but all statistical quantities below do not. If w 0 , w 1 ∈ W ∗ ( μ ) w_{0}, w_{1} \in W_{*}(\mu) w 0 , w 1 ∈ W ∗ ( μ ) and r μ ( j ) r_{\mu}^{(j)} r μ ( j ) is defined using w j w_{j} w j , then
r μ ( 1 ) ( h ) = r μ ( 0 ) ( h ) − ( w 1 − w 0 ) T h . r_{\mu}^{(1)}(h) = r_{\mu}^{(0)}(h) - (w_{1} - w_{0})^{\mathsf{T}}h. r μ ( 1 ) ( h ) = r μ ( 0 ) ( h ) − ( w 1 − w 0 ) T h .
Thus the two versions differ only by a linear functional. Such a term cancels from the Bayes Jensen gaps defining R n , M Σ R_{n,M}^{\Sigma} R n , M Σ and V M , Σ V_{M,\Sigma} V M , Σ and from the symmetric modulus D μ D_{\mu} D μ . The minimax risks, rates, and constants are therefore invariant to the measurable selection.
The next subsection gives an exact decision-theoretic representation of R n , M Σ ( μ ) R_{n,M}^{\Sigma}(\mu) R n , M Σ ( μ ) .
A Bayes representation of local regret
The key simplification is that the optimization problem enters the statistical decision problem only through the convex remainder r μ r_{\mu} r μ . We now present an identity that makes this reduction exact.
Let P ( B 2 ( M ) ) \mathcal{P}(B_{2}(M)) P ( B 2 ( M )) be the set of probability measures supported on B 2 ( M ) B_{2}(M) B 2 ( M ) . For each π ∈ P ( B 2 ( M ) ) \pi\in\mathcal{P}(B_{2}(M)) π ∈ P ( B 2 ( M )) , consider the following hierarchical model
T ∼ π , x 1 , … , x n ∣ T ∼ i . i . d . N ( μ n , T , Σ ) . (19) T \sim\pi,\qquad x_{1},\ldots,x_{n} \mid T \mathrel{\mathop{\sim}^{\mathrm{i.i.d.}}} \mathcal{N}(\mu_{n,T},\Sigma). \tag*{(19)} T ∼ π , x 1 , … , x n ∣ T ∼ i.i.d. N ( μ n , T , Σ ) . (19)
Thus π \pi π is a prior on the local perturbation T T T and induces a prior on the true mean μ n , T \mu_{n,T} μ n , T . Under this model, denote Z = n ( x ˉ − μ n , T ) Z = \sqrt{n}(\bar{x} - \mu_{n,T}) Z = n ( x ˉ − μ n , T ) , Y = n ( x ˉ − μ ) = T + Z Y = \sqrt{n}(\bar{x} - \mu) = T + Z Y = n ( x ˉ − μ ) = T + Z , and m π , Σ ( Y ) = E π , Σ [ T ∣ Y ] m_{\pi,\Sigma}(Y) = \mathbb{E}_{\pi,\Sigma}[T \mid Y] m π , Σ ( Y ) = E π , Σ [ T ∣ Y ] . It is clear that Z ∼ N ( 0 , Σ ) Z \sim\mathcal{N}(0,\Sigma) Z ∼ N ( 0 , Σ ) and is independent of T T T .
Lemma 3.1 (Local Bayes–minimax identity). Under the hierarchical model (19) ,
R n , M Σ ( μ ) = sup π ∈ P ( B 2 ( M ) ) inf w ^ E π , Σ [ f ∗ ( μ n , T ) − w ^ ( x 1 , … , x n ) T μ n , T ] R_{n,M}^{\Sigma}(\mu) = \sup_{\pi\in\mathcal{P}(B_{2}(M))}\ \inf_{\widehat{w}}\ \mathbb{E}_{\pi,\Sigma}\left[f_{*}(\mu_{n,T}) - \widehat{w}(x_{1},\ldots,x_{n})^{\mathsf{T}}\mu_{n,T}\right] R n , M Σ ( μ ) = π ∈ P ( B 2 ( M )) sup w inf E π , Σ [ f ∗ ( μ n , T ) − w ( x 1 , … , x n ) T μ n , T ] = sup π ∈ P ( B 2 ( M ) ) E π , Σ [ r μ ( T n ) − r μ ( m π , Σ ( Y ) n ) ] . (20) = \sup_{\pi\in\mathcal{P}(B_{2}(M))}\ \mathbb{E}_{\pi,\Sigma}\left[r_{\mu}\left(\frac{T}{\sqrt{n}}\right) - r_{\mu}\left(\frac{m_{\pi,\Sigma}(Y)}{\sqrt{n}}\right)\right]. \tag*{(20)} = π ∈ P ( B 2 ( M )) sup E π , Σ [ r μ ( n T ) − r μ ( n m π , Σ ( Y ) ) ] . (20) For each fixed π \pi π , the inner infimum is attained by the Bayes policy
w ^ π B ( x 1 , … , x n ) = w ∗ ( μ + m π , Σ ( Y ) n ) . (21) \widehat{w}_{\pi}^{\mathrm{B}}(x_{1},\ldots,x_{n}) = w_{*}\left(\mu+ \frac{m_{\pi,\Sigma}(Y)}{\sqrt{n}}\right). \tag*{(21)} w π B ( x 1 , … , x n ) = w ∗ ( μ + n m π , Σ ( Y ) ) . (21)
The identity is a Jensen-gap representation. The prior determines how difficult it is to infer the local perturbation T T T from the Gaussian shift Y = T + Z Y = T + Z Y = T + Z , while r μ r_{\mu} r μ converts the remaining posterior uncertainty into regret. In particular, the rate is not fixed by the n − 1 / 2 n^{-1/2} n − 1/2 estimation scale alone; it depends on how rapidly the remainder grows at the reference instance.
Smooth instances and the sharp quadratic constant
At a smooth instance the remainder is locally quadratic. The Hessian of the support function therefore acts as the loss matrix in the Gaussian shift experiment.
Theorem 3.2 (Local asymptotic minimax regret). Suppose that f ∗ f_* f ∗ is twice continuously differentiable on a neighborhood of the true value μ ≠ 0 \boldsymbol{\mu} \ne0 μ = 0 , and write H μ = ∇ 2 f ∗ ( μ ) H_{\mu} = \nabla^2 f_*(\boldsymbol{\mu}) H μ = ∇ 2 f ∗ ( μ ) . Keep d d d , Ω \Omega Ω , and μ \boldsymbol{\mu} μ fixed as n → ∞ n \to\infty n → ∞ . For every fixed M > 0 M > 0 M > 0 and Σ ∈ C \Sigma\in\mathcal{C} Σ ∈ C ,
lim n → ∞ n R n , M Σ ( μ ) = 1 2 B M , Σ ( H μ ) , (22) \lim_{n \to\infty} n R_{n,M}^{\Sigma}(\boldsymbol{\mu}) = \frac{1}{2} B_{M,\Sigma}(H_{\mu}), \tag*{(22)} n → ∞ lim n R n , M Σ ( μ ) = 2 1 B M , Σ ( H μ ) , (22) where B M , Σ ( H μ ) : = sup π ∈ P ( B 2 ( M ) ) E π , Σ [ ( T − m π , Σ ( Y ) ) ⊤ H μ ( T − m π , Σ ( Y ) ) ] B_{M,\Sigma}(H_{\mu}) := \sup_{\pi\in\mathcal{P}(B_2(M))} \mathbb{E}_{\pi,\Sigma}\left[(T-m_{\pi,\Sigma}(Y))^\top H_{\mu}(T-m_{\pi,\Sigma}(Y))\right] B M , Σ ( H μ ) := sup π ∈ P ( B 2 ( M )) E π , Σ [ ( T − m π , Σ ( Y ) ) ⊤ H μ ( T − m π , Σ ( Y )) ] . Moreover,
lim M → ∞ lim n → ∞ n R n , M Σ ( μ ) = 1 2 tr ( Σ H μ ) . (23) \lim_{M \to\infty}\lim_{n \to\infty} n R_{n,M}^{\Sigma}(\boldsymbol{\mu}) = \frac{1}{2}\operatorname{tr}(\Sigma H_{\mu}). \tag*{(23)} M → ∞ lim n → ∞ lim n R n , M Σ ( μ ) = 2 1 tr ( Σ H μ ) . (23) The SAA policy w ∗ ( x ˉ ) \boldsymbol{w}_*(\bar{x}) w ∗ ( x ˉ ) attains this local asymptotic minimax bound in that, for every fixed M M M ,
lim n → ∞ sup Σ ∈ C sup t ∈ B 2 ( M ) ∣ n E μ n , t , Σ R ( w ∗ ( x ˉ ) ) − 1 2 tr ( Σ H μ ) ∣ = 0. (24) \lim_{n \to\infty}\sup_{\Sigma\in\mathcal{C}}\sup_{t \in B_2(M)}\left|n\mathbb{E}_{\mu n,t,\Sigma}R(\boldsymbol{w}_*(\bar{x}))-\frac{1}{2}\operatorname{tr}(\Sigma H_{\mu})\right|=0. \tag*{(24)} n → ∞ lim Σ ∈ C sup t ∈ B 2 ( M ) sup n E μ n , t , Σ R ( w ∗ ( x ˉ )) − 2 1 tr ( Σ H μ ) = 0. (24)
For a fixed M M M , the minimax constant B M , Σ ( H μ ) / 2 B_{M,\Sigma}(H_{\mu})/2 B M , Σ ( H μ ) /2 can be smaller than the plug-in constant tr ( Σ H μ ) / 2 \operatorname{tr}(\Sigma H_{\mu})/2 tr ( Σ H μ ) /2 . Sharp optimality of SAA is therefore understood in the standard local asymptotic minimax sense ([15 , 26 , 36 ] ): first take n → ∞ n \to\infty n → ∞ on a fixed local ball and then let the local radius M → ∞ M \to\infty M → ∞ . At a fixed radius, procedures that exploit the prior information ∥ t ∥ 2 ≤ M \lVert t\rVert_2 \le M ∥ t ∥ 2 ≤ M in the spirit of Stein-type shrinkage ([35 , 19 ] ) can have a strictly smaller constant, and shrinkage-type corrections to SAA are known to matter in high-dimensional, small-sample regimes ([34 , 14 ] ); such gains vanish as M → ∞ M \to\infty M → ∞ and do not affect the fixed-dimensional rate or the sharp constant established here. The theorem also shows that SAA attains the sharp constant without using Σ \Sigma Σ , uniformly over Σ ∈ C \Sigma\in\mathcal{C} Σ ∈ C .
General local growth and arbitrary rates
Quadratic behavior as described in the previous subsection is only one possibility. More generally, suppose that for some a : ( 0 , ϵ 0 ) → ( 0 , ∞ ) a:(0,\epsilon_0)\to(0,\infty) a : ( 0 , ϵ 0 ) → ( 0 , ∞ ) with a ( ϵ ) → 0 a(\epsilon)\to0 a ( ϵ ) → 0 as ϵ ↓ 0 \epsilon\downarrow0 ϵ ↓ 0 , and some finite continuous function ψ : R d → R \psi:\mathbb{R}^d\to\mathbb{R} ψ : R d → R , for every t ∈ R d t\in\mathbb{R}^d t ∈ R d ,
lim ϵ ↓ 0 r μ ( ϵ t ) a ( ϵ ) = ψ ( t ) . (25) \lim_{\epsilon\downarrow0}\frac{r_{\mu}(\epsilon t)}{a(\epsilon)}=\psi(t). \tag*{(25)} ϵ ↓ 0 lim a ( ϵ ) r μ ( ϵ t ) = ψ ( t ) . (25)
ψ \psi ψ is automatically convex if (25) holds true. Since each r μ ( ϵ ⋅ ) / a ( ϵ ) r_{\mu}(\epsilon\mathbin{\cdot})/a(\epsilon) r μ ( ϵ ⋅ ) / a ( ϵ ) is convex and ψ \psi ψ is finite and continuous, pointwise convergence in (25) is automatically uniform on bounded sets. Let
V M , Σ ( ψ ) : = sup π ∈ P ( B 2 ( M ) ) E π , Σ [ ψ ( T ) − ψ ( m π , Σ ( Y ) ) ] . (26) V_{M,\Sigma}(\psi):=\sup_{\pi\in\mathcal{P}(B_2(M))}\mathbb{E}_{\pi,\Sigma}\left[\psi(T)-\psi(m_{\pi,\Sigma}(Y))\right]. \tag*{(26)} V M , Σ ( ψ ) := π ∈ P ( B 2 ( M )) sup E π , Σ [ ψ ( T ) − ψ ( m π , Σ ( Y )) ] . (26)
Note that V M , Σ ( ψ ) V_{M,\Sigma}(\psi) V M , Σ ( ψ ) is finite and nonnegative. It is strictly positive if and only if ψ \psi ψ is not affine on B 2 ( M ) B_2(M) B 2 ( M ) .
Theorem 3.3 (Local minimax regret under a general scaling). Suppose that (25) holds. Then
lim n → ∞ R n , M Σ ( μ ) a ( n − 1 / 2 ) = V M , Σ ( ψ ) , (27) \lim_{n\to\infty}\frac{R^\Sigma_{n,M}(\mu)}{a(n^{-1/2})}=V_{M,\Sigma}(\psi), \tag*{(27)} n → ∞ lim a ( n − 1/2 ) R n , M Σ ( μ ) = V M , Σ ( ψ ) , (27) Moreover, whenever V M , Σ ( ψ ) > 0 V_{M,\Sigma}(\psi)>0 V M , Σ ( ψ ) > 0 , the SAA policy w ∗ ( x ˉ ) w_*(\bar{x}) w ∗ ( x ˉ ) attains the local minimax rate: there are constants 0 < c ≤ C < ∞ 0<c\le C<\infty 0 < c ≤ C < ∞ such that, for all sufficiently large n n n ,
c a ( n − 1 / 2 ) ≤ R n , M Σ ( μ ) ≤ sup t ∈ B 2 ( M ) E μ n , t , Σ R ( w ∗ ( x ˉ ) ) ≤ C a ( n − 1 / 2 ) . (28) ca(n^{-1/2})\le R^\Sigma_{n,M}(\mu)\le\sup_{t\in B_2(M)}\mathbb{E}_{\mu_{n,t,\Sigma}}R(w_*(\bar{x}))\le Ca(n^{-1/2}). \tag*{(28)} c a ( n − 1/2 ) ≤ R n , M Σ ( μ ) ≤ t ∈ B 2 ( M ) sup E μ n , t , Σ R ( w ∗ ( x ˉ )) ≤ C a ( n − 1/2 ) . (28) The constants may depend on the fixed d , Ω , μ , M , λ ‾ , λ ‾ d,\Omega,\mu,M,\underline{\lambda},\overline{\lambda} d , Ω , μ , M , λ , λ and can be chosen uniformly over Σ ∈ C \Sigma\in\mathcal{C} Σ ∈ C .
Quantitatively, the proof yields
∣ R n , M Σ ( μ ) a ( n − 1 / 2 ) − V M , Σ ( ψ ) ∣ ≤ 2 sup t ∈ B 2 ( M ) ∣ r μ ( t / n ) a ( n − 1 / 2 ) − ψ ( t ) ∣ . (29) \left|\frac{R^\Sigma_{n,M}(\mu)}{a(n^{-1/2})}-V_{M,\Sigma}(\psi)\right|
\le2\sup_{t\in B_2(M)}\left|\frac{r_\mu(t/\sqrt{n})}{a(n^{-1/2})}-\psi(t)\right|. \tag*{(29)} a ( n − 1/2 ) R n , M Σ ( μ ) − V M , Σ ( ψ ) ≤ 2 t ∈ B 2 ( M ) sup a ( n − 1/2 ) r μ ( t / n ) − ψ ( t ) . (29) When ψ \psi ψ is not affine on B 2 ( M ) B_2(M) B 2 ( M ) , R n , M Σ ( μ ) ≍ a ( n − 1 / 2 ) R^\Sigma_{n,M}(\mu)\asymp a(n^{-1/2}) R n , M Σ ( μ ) ≍ a ( n − 1/2 ) ; otherwise R n , M Σ ( μ ) = o ( a ( n − 1 / 2 ) ) R^\Sigma_{n,M}(\mu)=o(a(n^{-1/2})) R n , M Σ ( μ ) = o ( a ( n − 1/2 )) .
The theorem gives a direct scale transformation:
estimation uncertainty at scale n − 1 / 2 n^{-1/2} n − 1/2 ⟼ \longmapsto ⟼ decision regret at scale a ( n − 1 / 2 ) a(n^{-1/2}) a ( n − 1/2 ) .
It also separates exact minimax characterization from algorithmic attainment: V M , Σ ( ψ ) V_{M,\Sigma}(\psi) V M , Σ ( ψ ) is the benchmark, while ordinary SAA is guaranteed to attain its rate whenever the limiting convex modulus is nonaffine.
A finite-sample local optimization modulus
The asymptotic expansion is not needed to compare the statistical difficulty of two instances. Define, for any compact convex Ω \Omega Ω , any μ ∈ R d \mu\in\mathbb{R}^d μ ∈ R d , and ε , M > 0 \varepsilon,M>0 ε , M > 0 , the symmetric local modulus
D μ ( ε , M ) : = sup t ∈ B 2 ( M ) { f ∗ ( μ + ε t ) + f ∗ ( μ − ε t ) 2 − f ∗ ( μ ) } = 1 2 sup t ∈ B 2 ( M ) { r μ ( ε t ) + r μ ( − ε t ) } . (30) \begin{aligned}
D_\mu(\varepsilon,M):=\sup_{t\in B_2(M)}
\left\{\frac{f_*(\mu+\varepsilon t)+f_*(\mu-\varepsilon t)}{2}-f_*(\mu)\right\} \\
&=\frac{1}{2}\sup_{t\in B_2(M)}\{r_\mu(\varepsilon t)+r_\mu(-\varepsilon t)\}. \tag*{(30)}
\end{aligned} D μ ( ε , M ) := t ∈ B 2 ( M ) sup { 2 f ∗ ( μ + εt ) + f ∗ ( μ − εt ) − f ∗ ( μ ) } = 2 1 t ∈ B 2 ( M ) sup { r μ ( εt ) + r μ ( − εt )} . (30)
Then D μ ( ε , M ) ≥ 0 D_\mu(\varepsilon,M)\ge0 D μ ( ε , M ) ≥ 0 .
Proposition 3.4 (A finite-sample bound without a local expansion). For every integer n ≥ 1 n\ge1 n ≥ 1 ,
2 Φ ( − M λ min ( Σ ) ) D μ ( n − 1 / 2 , M ) ≤ R n , M Σ ( μ ) ≤ 2 D μ ( n − 1 / 2 , M ) , (31) 2\Phi\left(-\frac{M}{\sqrt{\lambda_{\min}(\Sigma)}}\right)D_\mu(n^{-1/2},M)
\le R^\Sigma_{n,M}(\mu)\le2D_\mu(n^{-1/2},M), \tag*{(31)} 2Φ ( − λ m i n ( Σ ) M ) D μ ( n − 1/2 , M ) ≤ R n , M Σ ( μ ) ≤ 2 D μ ( n − 1/2 , M ) , (31) where Φ \Phi Φ is the standard normal distribution function.
Note that Proposition 3.4 does not assume differentiability of f ∗ f_\ast f ∗ or (25) . It shows that, up to constants depending only on the fixed local radius and spectral bounds,
R n , M Σ ( μ ) ≍ D μ ( n − 1 / 2 , M ) . R_{n,M}^{\Sigma}(\mu) \asymp D_\mu(n^{-1/2},M). R n , M Σ ( μ ) ≍ D μ ( n − 1/2 , M ) .
This is the most assumption-light form of the geometry-to-regret principle: inject statistical uncertainty at scale n − 1 / 2 n^{-1/2} n − 1/2 and evaluate the symmetric growth of the support function at that scale.
Geometry-to-Regret Correspondence
Nonsmooth faces and first-order rates
When the exposed face W ∗ ( μ ) W_\ast(\mu) W ∗ ( μ ) is not a singleton, the first-order directional derivative of the support function is already nonlinear. This yields the slow local regime. More specifically, for any compact convex Ω \Omega Ω , let
ψ 1 ( t ) : = sup w ∈ W ∗ ( μ ) ( w − w ∗ ( μ ) ) ⊤ t . (32) \psi_1(\mathbf{t}) := \sup_{\mathbf{w}\in W_\ast(\mu)}(\mathbf{w}-\mathbf{w}_\ast(\mu))^\top\mathbf{t}. \tag*{(32)} ψ 1 ( t ) := w ∈ W ∗ ( μ ) sup ( w − w ∗ ( μ ) ) ⊤ t . (32)
Corollary 4.1 (First-order limit for a compact convex set). Condition (25) always holds with a ( ε ) = ε a(\varepsilon)=\varepsilon a ( ε ) = ε and ψ = ψ 1 \psi=\psi_1 ψ = ψ 1 . For every M > 0 M>0 M > 0 , V M , Σ ( ψ 1 ) > 0 V_{M,\Sigma}(\psi_1)>0 V M , Σ ( ψ 1 ) > 0 if and only if W ∗ ( μ ) W_\ast(\mu) W ∗ ( μ ) contains more than one point.
Together with Theorem 3.3 , this immediately implies that
lim n → ∞ n R n , M Σ ( μ ) = V M , Σ ( ψ 1 ) > 0 , (33) \lim_{n\to\infty}\sqrt{n}R_{n,M}^{\Sigma}(\mu)=V_{M,\Sigma}(\psi_1)>0, \tag*{(33)} n → ∞ lim n R n , M Σ ( μ ) = V M , Σ ( ψ 1 ) > 0 , (33)
when W ∗ ( μ ) W_\ast(\mu) W ∗ ( μ ) is not a singleton. Otherwise ψ 1 ≡ 0 \psi_1\equiv0 ψ 1 ≡ 0 , and faster rates can be achieved, i.e.,
R n , M Σ ( μ ) = o ( n − 1 / 2 ) . R_{n,M}^{\Sigma}(\mu)=o(n^{-1/2}). R n , M Σ ( μ ) = o ( n − 1/2 ) .
Hence decision ambiguity is precisely the source of a positive n − 1 / 2 n^{-1/2} n − 1/2 local constant. A unique optimizer eliminates the first-order term, but it does not determine the next rate: the remainder may be quadratic, higher order, or non-power.
Distance to ambiguity and normalized local shape
Positive homogeneity separates radial distance from directional shape. Suppose that Ω \Omega Ω is not a singleton, so that 0 ∈ A 0\in A 0 ∈ A and dist ( μ , A ) ≤ ∥ μ ∥ K < ∞ \operatorname{dist}(\mu,A)\le\lVert\mu\rVert_K<\infty dist ( μ , A ) ≤ ∥ μ ∥ K < ∞ . Recall that ∥ ⋅ ∥ K \lVert\cdot\rVert_K ∥ ⋅ ∥ K is a fixed norm used to define dist ( ⋅ , A ) \operatorname{dist}(\cdot,A) dist ( ⋅ , A ) . Since A A A is a cone that need not be closed, dist ( μ , A ) > 0 \operatorname{dist}(\mu,A)>0 dist ( μ , A ) > 0 requires more than differentiability of f ∗ f_\ast f ∗ at μ \mu μ ; it holds whenever μ ≠ 0 \mu\ne0 μ = 0 and f ∗ f_\ast f ∗ is C 2 C^2 C 2 on a neighborhood of μ \mu μ , because then A A A does not meet that neighborhood. For such μ \mu μ , let
ν μ : = μ dist ( μ , A ) , H ν μ : = ∇ 2 f ∗ ( ν μ ) . (34) \nu_\mu:=\frac{\mu}{\operatorname{dist}(\mu,A)},\qquad H_{\nu_\mu}:=\nabla^2 f_\ast(\nu_\mu). \tag*{(34)} ν μ := dist ( μ , A ) μ , H ν μ := ∇ 2 f ∗ ( ν μ ) . (34)
The rescaling ensures that dist ( ν μ , A ) = 1 \operatorname{dist}(\nu_\mu,A)=1 dist ( ν μ , A ) = 1 and H μ = dist ( μ , A ) − 1 H ν μ H_\mu=\operatorname{dist}(\mu,A)^{-1}H_{\nu_\mu} H μ = dist ( μ , A ) − 1 H ν μ so that
lim n → ∞ n R n , M Σ ( μ ) = B M , Σ ( H ν μ ) 2 dist ( μ , A ) . (35) \lim_{n\to\infty} nR^{\Sigma}_{n,M}(\mu)=\frac{\mathcal{B}_{M,\Sigma}(H_{\nu_\mu})}{2\operatorname{dist}(\mu,A)}. \tag*{(35)} n → ∞ lim n R n , M Σ ( μ ) = 2 dist ( μ , A ) B M , Σ ( H ν μ ) . (35)
This implies that
Corollary 4.2 (The sharp constant and the distance to the ambiguity set). For any μ ≠ 0 \mu\ne0 μ = 0 such that f ∗ f_\ast f ∗ is C 2 C^2 C 2 on a neighborhood of μ \mu μ ,
lim M → ∞ lim n → ∞ n R n , M Σ ( μ ) = tr ( Σ H ν μ ) 2 dist ( μ , A ) = [ G ( Σ 1 / 2 Ω ) ] 2 dist ( μ , A ) χ Ω , Σ ( μ ) , (36) \lim_{M\to\infty}\lim_{n\to\infty} nR^{\Sigma}_{n,M}(\mu)=\frac{\operatorname{tr}(\Sigma H_{\nu_\mu})}{2\operatorname{dist}(\mu,A)}=\frac{[\mathcal{G}(\Sigma^{1/2}\Omega)]^2}{\operatorname{dist}(\mu,A)}\chi_{\Omega,\Sigma}(\mu), \tag*{(36)} M → ∞ lim n → ∞ lim n R n , M Σ ( μ ) = 2 dist ( μ , A ) tr ( Σ H ν μ ) = dist ( μ , A ) [ G ( Σ 1/2 Ω ) ] 2 χ Ω , Σ ( μ ) , (36) where χ Ω , Σ ( μ ) : = tr ( Σ H ν μ ) / ( 2 [ G ( Σ 1 / 2 Ω ) ] 2 ) \chi_{\Omega,\Sigma}(\mu):=\operatorname{tr}(\Sigma H_{\nu_\mu})/(2[\mathcal{G}(\Sigma^{1/2}\Omega)]^2) χ Ω , Σ ( μ ) := tr ( Σ H ν μ ) / ( 2 [ G ( Σ 1/2 Ω ) ] 2 ) .
The distance dist ( μ , A ) \operatorname{dist}(\mu,A) dist ( μ , A ) therefore controls only one component of difficulty. The normalized Hessian H ν μ H_{\nu_\mu} H ν μ retains the directional geometry, and two instances at the same distance from ambiguity can have different rates when one has degenerate curvature.
At a smooth point with tr ( Σ H μ ) > 0 \operatorname{tr}(\Sigma H_\mu)>0 tr ( Σ H μ ) > 0 , the power n − 1 n^{-1} n − 1 is locally optimal. The fast rate in (14) is sharp up to constant factors precisely when
tr ( Σ ∇ 2 f ∗ ( μ ) ) ≍ [ G ( Σ 1 / 2 Ω ) ] 2 dist ( μ , A ) , (37) \operatorname{tr}(\Sigma\nabla^2 f_\ast(\mu))\asymp\frac{[\mathcal{G}(\Sigma^{1/2}\Omega)]^2}{\operatorname{dist}(\mu,A)}, \tag*{(37)} tr ( Σ ∇ 2 f ∗ ( μ )) ≍ dist ( μ , A ) [ G ( Σ 1/2 Ω ) ] 2 , (37)
or equivalently, χ Ω , Σ ( μ ) ≍ 1 \chi_{\Omega,\Sigma}(\mu)\asymp1 χ Ω , Σ ( μ ) ≍ 1 , with the strong-convexity constant in (14) controlled.
For example, for Ω = B 2 d \Omega=B^d_2 Ω = B 2 d with d ≥ 2 d\ge2 d ≥ 2 ,
f ∗ ( μ ) = ∥ μ ∥ 2 , tr ( Σ ∇ 2 f ∗ ( μ ) ) = tr ( Σ ) − μ ⊤ Σ μ / ∥ μ ∥ 2 2 ∥ μ ∥ 2 , dist ( μ , A ) = ∥ μ ∥ 2 , f_\ast(\mu)=\lVert\mu\rVert_2,\qquad\operatorname{tr}(\Sigma\nabla^2f_\ast(\mu))=\frac{\operatorname{tr}(\Sigma)-\mu^\top\Sigma\mu/\lVert\mu\rVert_2^2}{\lVert\mu\rVert_2},\qquad\operatorname{dist}(\mu,A)=\lVert\mu\rVert_2, f ∗ ( μ ) = ∥ μ ∥ 2 , tr ( Σ ∇ 2 f ∗ ( μ )) = ∥ μ ∥ 2 tr ( Σ ) − μ ⊤ Σ μ / ∥ μ ∥ 2 2 , dist ( μ , A ) = ∥ μ ∥ 2 ,
and [ G ( Σ 1 / 2 B 2 d ) ] 2 ≍ d [\mathcal{G}(\Sigma^{1/2}B^d_2)]^2\asymp d [ G ( Σ 1/2 B 2 d ) ] 2 ≍ d under the fixed spectral bounds. Hence χ Ω , Σ ( μ ) ≍ 1 \chi_{\Omega,\Sigma}(\mu)\asymp1 χ Ω , Σ ( μ ) ≍ 1 and the fast-rate bound matches the local minimax scale. In contrast, consider the strongly convex ellipse
Ω L = { w ∈ R 2 : w 1 2 / L 2 + w 2 2 ≤ 1 } , L ≥ 1 , μ = e 1 . \Omega_L=\left\{w\in\mathbb{R}^2:w_1^2/L^2+w_2^2\le1\right\},\qquad L\ge1,\qquad\mu=e_1. Ω L = { w ∈ R 2 : w 1 2 / L 2 + w 2 2 ≤ 1 } , L ≥ 1 , μ = e 1 .
Its support function is f ∗ ( μ ) = ( L 2 μ 1 2 + μ 2 2 ) 1 / 2 f_\ast(\mu)=(L^2\mu_1^2+\mu_2^2)^{1/2} f ∗ ( μ ) = ( L 2 μ 1 2 + μ 2 2 ) 1/2 , so A = { 0 } A=\{0\} A = { 0 } and
dist ( μ , A ) = L , [ G ( Σ 1 / 2 Ω L ) ] 2 ≍ L 2 + 1 , tr ( Σ ∇ 2 f ∗ ( μ ) ) = Σ 22 / L . \operatorname{dist}(\mu,A)=L,\qquad[\mathcal{G}(\Sigma^{1/2}\Omega_L)]^2\asymp L^2+1,\qquad\operatorname{tr}(\Sigma\nabla^2f_\ast(\mu))=\Sigma_{22}/L. dist ( μ , A ) = L , [ G ( Σ 1/2 Ω L ) ] 2 ≍ L 2 + 1 , tr ( Σ ∇ 2 f ∗ ( μ )) = Σ 22 / L .
As a linear image of the Euclidean ball, this ellipse satisfies (11) with c Ω L = 1 / 2 c_{\Omega_L}=1/2 c Ω L = 1/2 in its own norm, uniformly in L L L . Under the fixed spectral bounds, Σ 22 ≍ 1 \Sigma_{22}\asymp1 Σ 22 ≍ 1 , so that
χ Ω L , Σ ( e 1 ) ≍ 1 L 2 ⟶ 0 ( L → ∞ ) . \chi_{\Omega_L,\Sigma}(e_1)\asymp\frac{1}{L^2}\longrightarrow0\qquad(L\to\infty). χ Ω L , Σ ( e 1 ) ≍ L 2 1 ⟶ 0 ( L → ∞ ) .
Thus (37) fails. The bound (14) is of order L / n L/n L / n , whereas the sharp local constant is tr ( Σ H μ ) / ( 2 n ) = Σ 22 / ( 2 L n ) \operatorname{tr}(\Sigma H_\mu)/(2n)=\Sigma_{22}/(2Ln) tr ( Σ H μ ) / ( 2 n ) = Σ 22 / ( 2 L n ) ; the bound overshoots by a factor of order L 2 L^2 L 2 , although the quadratic-growth constant does not deteriorate. A bound that combines the global width G ( Ω ) \mathcal{G}(\Omega) G ( Ω ) with the local margin dist ( μ , A ) \operatorname{dist}(\mu,A) dist ( μ , A ) therefore cannot be sharp in general.
Smooth boundaries and inverse curvature
The identity between the tangential Hessian of a support function and the inverse Weingarten map is classical; see Howard and Hug [18 ] . The next result gives its local form and applies it to the statistical constant in Theorem 3.2 .
Let d ≥ 2 d \ge2 d ≥ 2 , let Ω \Omega Ω be compact and convex with nonempty interior, and fix μ ≠ 0 \mu\ne0 μ = 0 with a unique maximizer w ∗ ( μ ) w_{\ast}(\mu) w ∗ ( μ ) . Denote n μ : = μ / ∥ μ ∥ 2 n_{\mu} := \mu/\lVert\mu\rVert_{2} n μ := μ / ∥ μ ∥ 2 as the outward unit normal to the supporting hyperplane selected by μ \mu μ , T μ : = n μ ⊥ T_{\mu} := n_{\mu}^{\perp} T μ := n μ ⊥ as the linear subspace parallel to that hyperplane, and U μ U_{\mu} U μ as an orthonormal basis matrix of T μ T_{\mu} T μ , with U μ T U μ = I d − 1 U_{\mu}^{\mathsf{T}}U_{\mu} = I_{d-1} U μ T U μ = I d − 1 and range ( U μ ) = T μ \operatorname{range}(U_{\mu}) = T_{\mu} range ( U μ ) = T μ . When ∂ Ω \partial\Omega ∂ Ω is a C 2 C^{2} C 2 hypersurface near w ∗ ( μ ) w_{\ast}(\mu) w ∗ ( μ ) , T μ T_{\mu} T μ is its tangent space at w ∗ ( μ ) w_{\ast}(\mu) w ∗ ( μ ) ; in this case, let N N N be the outward unit normal map on the boundary neighborhood and define
S μ : = U μ T D N ( w ∗ ( μ ) ) U μ , κ i ( μ ) : = λ i ( S μ ) , i = 1 , … , d − 1. (38) \mathcal{S}_{\mu} := U_{\mu}^{\mathsf{T}}DN(w_{\ast}(\mu))U_{\mu}, \qquad\kappa_{i}(\mu) := \lambda_{i}(\mathcal{S}_{\mu}), \quad i = 1,\ldots,d-1. \tag*{(38)} S μ := U μ T D N ( w ∗ ( μ )) U μ , κ i ( μ ) := λ i ( S μ ) , i = 1 , … , d − 1. (38)
Here, for any differentiable vector-valued function F : R m → R k F : \mathbb{R}^{m} \to\mathbb{R}^{k} F : R m → R k , the derivative D F ( x ) DF(x) D F ( x ) is defined as a linear map with
F ( x + h ) = F ( x ) + D F ( x ) [ h ] + o ( ∥ h ∥ 2 ) . F(x+h) = F(x) + DF(x)[h] + o(\lVert h\rVert_{2}). F ( x + h ) = F ( x ) + D F ( x ) [ h ] + o (∥ h ∥ 2 ) .
For a C 2 C^{2} C 2 boundary, N N N is C 1 C^{1} C 1 , so its differential exists; it is taken along boundary curves and acts on tangent vectors. We use the sign convention under which a Euclidean sphere of radius R R R has κ i = 1 / R \kappa_{i} = 1/R κ i = 1/ R .
Corollary 4.3 (Boundary principal curvatures and the sharp constant). Assume that ∂ Ω \partial\Omega ∂ Ω is a C 2 C^{2} C 2 hypersurface on a neighborhood of w ∗ ( μ ) w_{\ast}(\mu) w ∗ ( μ ) and that S μ ≻ 0 \mathcal{S}_{\mu} \succ0 S μ ≻ 0 , equivalently κ i ( μ ) > 0 \kappa_{i}(\mu) > 0 κ i ( μ ) > 0 for every i i i . Then f ∗ f_{\ast} f ∗ is C 2 C^{2} C 2 on a neighborhood of μ \mu μ , and
H μ = 1 ∥ μ ∥ 2 2 U μ S μ − 1 U μ T . (39) H_{\mu} = \frac{1}{\lVert\mu\rVert_{2}^{2}}U_{\mu}\mathcal{S}_{\mu}^{-1}U_{\mu}^{\mathsf{T}}. \tag*{(39)} H μ = ∥ μ ∥ 2 2 1 U μ S μ − 1 U μ T . (39) In particular, rank ( H μ ) = d − 1 \operatorname{rank}(H_{\mu}) = d-1 rank ( H μ ) = d − 1 and ker ( H μ ) = span { μ } \operatorname{ker}(H_{\mu}) = \operatorname{span}\{\mu\} ker ( H μ ) = span { μ } . Choose the columns u 1 , … , u d − 1 \boldsymbol{u}_{1},\ldots,\boldsymbol{u}_{d-1} u 1 , … , u d − 1 of U μ U_{\mu} U μ along orthonormal principal directions. The sharp local asymptotic minimax constant is
lim M → ∞ lim n → ∞ n R n , M Σ ( μ ) = 1 2 ∥ μ ∥ 2 2 ∑ i = 1 d − 1 u i T Σ u i κ i ( μ ) \lim_{M\to\infty}\lim_{n\to\infty} nR^{\Sigma}_{n,M}(\mu) = \frac{1}{2\lVert\mu\rVert_{2}^{2}}\sum_{i=1}^{d-1}\frac{\boldsymbol{u}_{i}^{\mathsf{T}}\Sigma\boldsymbol{u}_{i}}{\kappa_{i}(\mu)} M → ∞ lim n → ∞ lim n R n , M Σ ( μ ) = 2 ∥ μ ∥ 2 2 1 i = 1 ∑ d − 1 κ i ( μ ) u i T Σ u i = dist ( n μ , A ) 2 dist ( μ , A ) ∑ i = 1 d − 1 u i T Σ u i κ i ( μ ) . (40) = \frac{\operatorname{dist}(n_{\mu},A)}{2\operatorname{dist}(\mu,A)}\sum_{i=1}^{d-1}\frac{\boldsymbol{u}_{i}^{\mathsf{T}}\Sigma\boldsymbol{u}_{i}}{\kappa_{i}(\mu)}. \tag*{(40)} = 2 dist ( μ , A ) dist ( n μ , A ) i = 1 ∑ d − 1 κ i ( μ ) u i T Σ u i . (40) The second equality uses any fixed norm ∥ ⋅ ∥ K \lVert\cdot\rVert_{K} ∥ ⋅ ∥ K defining dist ( ⋅ , A ) \operatorname{dist}(\cdot,A) dist ( ⋅ , A ) .
The formula has a direct interpretation. Noise normal to the boundary does not change the exposed decision to first order. Tangential noise does, and its contribution is divided by the corresponding principal curvature. Flatter boundary directions are therefore statistically harder.
Boundary contact and rate exponents
The curvature formula extends to nonsmooth or degenerate boundaries through the order of contact with the supporting hyperplane. First, we introduce the following local parametrization of the boundary ∂ Ω \partial\Omega ∂ Ω .
Proposition 4.5 (Local boundary parametrization and growth conditions). Suppose that Ω \Omega Ω has a unique supporting hyperplane at w ∗ ( μ ) w_*(\boldsymbol{\mu}) w ∗ ( μ ) . Then there exist ρ > 0 \rho> 0 ρ > 0 and a continuous convex function ϱ μ \varrho_{\boldsymbol{\mu}} ϱ μ , defined on a neighborhood of { z : ∥ z ∥ 2 ≤ ρ } \{z : \lVert z\rVert_2 \le\rho\} { z : ∥ z ∥ 2 ≤ ρ } , such that ϱ μ ( 0 ) = 0 \varrho_{\boldsymbol{\mu}}(0) = 0 ϱ μ ( 0 ) = 0 , 0 ≤ ϱ μ ( z ) < ρ 0 \le\varrho_{\boldsymbol{\mu}}(z) < \rho 0 ≤ ϱ μ ( z ) < ρ , and
w ∗ ( μ ) + U μ z − n μ b ∈ Ω ⟺ ϱ μ ( z ) ≤ b , ∥ z ∥ 2 ≤ ρ , ∣ b ∣ ≤ ρ . (41) w_*(\boldsymbol{\mu}) + U_{\boldsymbol{\mu}}z - n_{\boldsymbol{\mu}}b \in\Omega\quad\Longleftrightarrow\quad\varrho_{\boldsymbol{\mu}}(z) \le b,\qquad\lVert z\rVert_2 \le\rho,\quad|b| \le\rho. \tag*{(41)} w ∗ ( μ ) + U μ z − n μ b ∈ Ω ⟺ ϱ μ ( z ) ≤ b , ∥ z ∥ 2 ≤ ρ , ∣ b ∣ ≤ ρ . (41) Here ϱ μ ( z ) \varrho_{\boldsymbol{\mu}}(z) ϱ μ ( z ) is differentiable at 0 0 0 , with ∇ ϱ μ ( 0 ) = 0 \nabla\varrho_{\boldsymbol{\mu}}(0) = 0 ∇ ϱ μ ( 0 ) = 0 and ϱ μ ( z ) = o ( ∥ z ∥ 2 ) \varrho_{\boldsymbol{\mu}}(z) = o(\lVert z\rVert_2) ϱ μ ( z ) = o (∥ z ∥ 2 ) as z → 0 z \to0 z → 0 . The following conditions give lower and upper bounds for ϱ μ \varrho_{\boldsymbol{\mu}} ϱ μ .
(i) Lower bound. Suppose that, for some β > 0 \beta> 0 β > 0 and κ > 0 \kappa> 0 κ > 0 ,
n μ T ( w ∗ ( μ ) − w ) ≥ κ 1 + β ∥ U μ T ( w − w ∗ ( μ ) ) ∥ 2 1 + β , w ∈ ∂ Ω near w ∗ ( μ ) . n_{\boldsymbol{\mu}}^{\mathsf T}(w_*(\boldsymbol{\mu}) - w) \ge\frac{\kappa}{1+\beta}\lVert U_{\boldsymbol{\mu}}^{\mathsf T}(w-w_*(\boldsymbol{\mu}))\rVert_2^{1+\beta},\qquad w \in\partial\Omega\text{ near }w_*(\boldsymbol{\mu}). n μ T ( w ∗ ( μ ) − w ) ≥ 1 + β κ ∥ U μ T ( w − w ∗ ( μ )) ∥ 2 1 + β , w ∈ ∂ Ω near w ∗ ( μ ) . Then, after decreasing ρ \rho ρ if necessary,
ϱ μ ( z ) ≥ κ 1 + β ∥ z ∥ 2 1 + β , ∥ z ∥ 2 ≤ ρ . \varrho_{\boldsymbol{\mu}}(z) \ge\frac{\kappa}{1+\beta}\lVert z\rVert_2^{1+\beta},\qquad\lVert z\rVert_2 \le\rho. ϱ μ ( z ) ≥ 1 + β κ ∥ z ∥ 2 1 + β , ∥ z ∥ 2 ≤ ρ . (ii) Upper bound. If ∂ Ω \partial\Omega ∂ Ω is a C 1 , β C^{1,\beta} C 1 , β hypersurface near w ∗ ( μ ) w_*(\boldsymbol{\mu}) w ∗ ( μ ) , where 0 < β ≤ 1 0 < \beta\le1 0 < β ≤ 1 , then ϱ μ ∈ C 1 , β \varrho_{\boldsymbol{\mu}} \in C^{1,\beta} ϱ μ ∈ C 1 , β locally and, after decreasing ρ \rho ρ if necessary, there exists κ ˉ > 0 \bar{\kappa} > 0 κ ˉ > 0 such that
0 ≤ ϱ μ ( z ) ≤ κ ˉ 1 + β ∥ z ∥ 2 1 + β , ∥ z ∥ 2 ≤ ρ . 0 \le\varrho_{\boldsymbol{\mu}}(z) \le\frac{\bar{\kappa}}{1+\beta}\lVert z\rVert_2^{1+\beta},\qquad\lVert z\rVert_2 \le\rho. 0 ≤ ϱ μ ( z ) ≤ 1 + β κ ˉ ∥ z ∥ 2 1 + β , ∥ z ∥ 2 ≤ ρ .
With this local parametrization, two-sided power growth of the boundary profile ϱ μ \varrho_{\boldsymbol{\mu}} ϱ μ determines the order of the local minimax risk and the maximal expected regret of SAA.
Theorem 4.6 (Two-sided boundary contact determines the minimax rate). Suppose that Ω \Omega Ω admits the local representation (41) of Proposition 4.5 , with a continuous convex ϱ μ \varrho_{\boldsymbol{\mu}} ϱ μ . Assume that, for some β > 0 \beta> 0 β > 0 and constants 0 < κ ≤ κ ˉ < ∞ 0 < \kappa\le\bar{\kappa} < \infty 0 < κ ≤ κ ˉ < ∞ ,
κ 1 + β ∥ z ∥ 2 1 + β ≤ ϱ μ ( z ) ≤ κ ˉ 1 + β ∥ z ∥ 2 1 + β , ∥ z ∥ 2 ≤ ρ . \frac{\kappa}{1+\beta}\lVert z\rVert_2^{1+\beta} \le\varrho_{\boldsymbol{\mu}}(z) \le\frac{\bar{\kappa}}{1+\beta}\lVert z\rVert_2^{1+\beta},\qquad\lVert z\rVert_2 \le\rho. 1 + β κ ∥ z ∥ 2 1 + β ≤ ϱ μ ( z ) ≤ 1 + β κ ˉ ∥ z ∥ 2 1 + β , ∥ z ∥ 2 ≤ ρ .
Then, for every fixed M > 0 M > 0 M > 0 ,
R n , M Σ ( μ ) ≲ sup ∥ t ∥ 2 ≤ M E μ , n , t , Σ R ( w ∗ ( x ˉ ) ) ≲ ∥ μ ∥ 2 − 1 / β n − ( 1 + β ) / ( 2 β ) . R^\Sigma_{n,M}(\mu) \lesssim\sup_{\lVert t\rVert_2\le M} \mathbb{E}_{\mu,n,t,\Sigma}R(w^*(\bar{x})) \lesssim\lVert\mu\rVert_2^{-1/\beta}n^{-(1+\beta)/(2\beta)}. R n , M Σ ( μ ) ≲ ∥ t ∥ 2 ≤ M sup E μ , n , t , Σ R ( w ∗ ( x ˉ )) ≲ ∥ μ ∥ 2 − 1/ β n − ( 1 + β ) / ( 2 β ) .
with implied constants that may be chosen uniformly over Σ ∈ C \Sigma\in\mathcal{C} Σ ∈ C . If, in addition, ϱ μ ( δ z ) / δ 1 + β \varrho_\mu(\delta z)/\delta^{1+\beta} ϱ μ ( δ z ) / δ 1 + β converges locally uniformly to a homogeneous convex function ϱ 0 \varrho_0 ϱ 0 , then the local expansion in Theorem 3.3 holds with
a ( ε ) = ε ( 1 + β ) / β , ψ ( t ) = ∥ μ ∥ 2 − 1 / β ϱ 0 ∗ ( U μ ⊤ t ) , a(\varepsilon)=\varepsilon^{(1+\beta)/\beta}, \qquad\psi(t)=\lVert\mu\rVert_2^{-1/\beta}\varrho_0^*(U_\mu^\top t), a ( ε ) = ε ( 1 + β ) / β , ψ ( t ) = ∥ μ ∥ 2 − 1/ β ϱ 0 ∗ ( U μ ⊤ t ) ,
where ϱ 0 ∗ ( y ) : = sup z ∈ R d − 1 { y ⊤ z − ϱ 0 ( z ) } \varrho_0^*(y):=\sup_{z\in\mathbb{R}^{d-1}}\{y^\top z-\varrho_0(z)\} ϱ 0 ∗ ( y ) := sup z ∈ R d − 1 { y ⊤ z − ϱ 0 ( z )} is the convex conjugate.
A more detailed statement is given in Theorem D.2 of the Online Supplement. The exponent provides a useful dictionary. Quadratic boundary growth corresponds to β = 1 \beta=1 β = 1 and the n − 1 n^{-1} n − 1 regime. When 0 < β < 1 0<\beta<1 0 < β < 1 , the boundary is sharper than quadratic and the support remainder has order greater than two, yielding rates faster than n − 1 n^{-1} n − 1 such as n − 2 n^{-2} n − 2 . When β > 1 \beta>1 β > 1 , the boundary is flatter and the rate lies between n − 1 / 2 n^{-1/2} n − 1/2 and n − 1 n^{-1} n − 1 .
Remark 4.7 (Sufficient conditions for quadratic boundary growth). Under the geometric setup above, either Euclidean strong convexity together with a local C 1 , 1 C^{1,1} C 1 , 1 boundary, or a local C 2 C^2 C 2 boundary with S μ ≻ 0 S_\mu\succ0 S μ ≻ 0 , gives the two-sided boundary growth condition with β = 1 \beta=1 β = 1 ; precise statements and proofs are given in Lemma D.3 of the Online Supplement. Theorem 4.6 then gives an n − 1 n^{-1} n − 1 local minimax rate, attained by SAA. In the C 2 C^2 C 2 case, the quadratic profile also recovers the inverse-curvature formula in Corollary 4.3. This complements Remark 4.4 : positive curvature yields local quadratic contact, while the earlier remark derives positive curvature from the quadratic-growth condition (11) .
Examples
The examples below show why neither global strong convexity nor distance to the ambiguity set alone determines the local rate. Detailed calculations are given in Section E of the Online Supplement.
Example 4.8 (Rectangular constraints). Let Ω = [ 0 , 1 ] d \Omega=[0,1]^d Ω = [ 0 , 1 ] d , so f ∗ ( μ ) = ∑ j = 1 d ( μ j ) + f_*(\mu)=\sum_{j=1}^d(\mu_j)_+ f ∗ ( μ ) = ∑ j = 1 d ( μ j ) + . Denote S 0 = { j ∈ [ d ] : μ j = 0 } S_0=\{j\in[d]:\mu_j=0\} S 0 = { j ∈ [ d ] : μ j = 0 } and s = ∥ μ ∥ 0 = d − ∣ S 0 ∣ s=\lVert\mu\rVert_0=d-|S_0| s = ∥ μ ∥ 0 = d − ∣ S 0 ∣ .
For j ∈ [ d ] j\in[d] j ∈ [ d ] , if μ j = 0 \mu_j=0 μ j = 0 , then the optimal w ∗ , j ∗ w^*_{*,j} w ∗ , j ∗ can take any value in [ 0 , 1 ] [0,1] [ 0 , 1 ] , whereas if μ j ≠ 0 \mu_j\ne0 μ j = 0 , w ∗ , j ∗ = I { μ j > 0 } w^*_{*,j}=\mathbb{I}\{\mu_j>0\} w ∗ , j ∗ = I { μ j > 0 } is unique. Hence the ambiguity set is A = ⋃ j = 1 d { μ : μ j = 0 } A=\bigcup_{j=1}^d\{\mu:\mu_j=0\} A = ⋃ j = 1 d { μ : μ j = 0 } , and μ ∈ A \mu\in A μ ∈ A exactly when d > s d>s d > s .
For fixed M > 0 M>0 M > 0 and sufficiently small ε > 0 \varepsilon>0 ε > 0 , if μ j ≠ 0 \mu_j\ne0 μ j = 0 , μ j + ε t j \mu_j+\varepsilon t_j μ j + ε t j has the same sign as μ j \mu_j μ j , uniformly over t ∈ B 2 ( M ) t\in B_2(M) t ∈ B 2 ( M ) . Hence r μ ( ε t ) = ε ψ ( t ) r_\mu(\varepsilon t)=\varepsilon\psi(t) r μ ( εt ) = ε ψ ( t ) , where
ψ ( t ) = ∑ j ∈ S 0 [ ( t j ) + − w ∗ , j ∗ ( μ ) t j ] . \psi(t)=\sum_{j\in S_0}\left[(t_j)_+-w^*_{*,j}(\mu)t_j\right]. ψ ( t ) = j ∈ S 0 ∑ [ ( t j ) + − w ∗ , j ∗ ( μ ) t j ] .
If s < d s<d s < d , μ \mu μ is in the ambiguity set, and Corollary 4.1 gives V M , Σ ( ψ ) > 0 V_{M,\Sigma}(\psi)>0 V M , Σ ( ψ ) > 0 because ψ \psi ψ is nonaffine on B 2 ( M ) B_2(M) B 2 ( M ) . By Theorem 3.3, R n , M Σ ( μ ) ≍ n − 1 / 2 R^\Sigma_{n,M}(\mu)\asymp n^{-1/2} R n , M Σ ( μ ) ≍ n − 1/2 , and SAA attains this local minimax rate.
If s = d s=d s = d , the entire local ball B 2 ( M ) B_2(M) B 2 ( M ) shares the same optimal policy for sufficiently large n n n , so R n , M Σ ( μ ) = 0 R^\Sigma_{n,M}(\mu)=0 R n , M Σ ( μ ) = 0 . The maximal expected regret of SAA on B 2 ( M ) B_2(M) B 2 ( M ) decays exponentially. Calculations and details are provided in Section E.1 of the Online Supplement.
Example 4.9 (One strongly convex set, three local rates). For Ω = B 4 / 3 2 : = { w ∈ R 2 : ∥ w ∥ 4 / 3 ≤ 1 } \Omega=B^2_{4/3}:=\{w\in\mathbb{R}^2:\lVert w\rVert_{4/3}\le1\} Ω = B 4/3 2 := { w ∈ R 2 : ∥ w ∥ 4/3 ≤ 1 } , f ⋆ ( μ ) = ∥ μ ∥ 4 f_\star(\mu)=\lVert\mu\rVert_4 f ⋆ ( μ ) = ∥ μ ∥ 4 , and strict convexity gives A = { 0 } A=\{0\} A = { 0 } . Fix M > 0 M>0 M > 0 and compare the local rates at three centers.
For μ = 0 \mu=0 μ = 0 , any w ∈ Ω w\in\Omega w ∈ Ω is optimal. Corollary 4.1 and Theorem 3.3 therefore give
R n , M Σ ( 0 ) ≍ n − 1 / 2 . R^\Sigma_{n,M}(0)\asymp n^{-1/2}. R n , M Σ ( 0 ) ≍ n − 1/2 . For μ = 2 − 1 / 4 ( 1 , 1 ) T \mu=2^{-1/4}(1,1)^\mathsf{T} μ = 2 − 1/4 ( 1 , 1 ) T , direct calculation gives
H μ = 3 2 2 ( 1 − 1 − 1 1 ) . H_\mu=\frac{3}{2\sqrt{2}}\begin{pmatrix}1&-1\\-1&1\end{pmatrix}. H μ = 2 2 3 ( 1 − 1 − 1 1 ) . Hence by Theorem 3.2, R n , M Σ ( 2 − 1 / 4 ( 1 , 1 ) T ) ≍ n − 1 R^\Sigma_{n,M}(2^{-1/4}(1,1)^\mathsf{T})\asymp n^{-1} R n , M Σ ( 2 − 1/4 ( 1 , 1 ) T ) ≍ n − 1 , where the sharp constant when M → ∞ M\to\infty M → ∞ is
lim M → ∞ lim n → ∞ n R n , M Σ ( 2 − 1 / 4 ( 1 , 1 ) T ) = 1 2 tr ( Σ H μ ) = 3 4 2 ( Σ 11 + Σ 22 − 2 Σ 12 ) . \lim_{M\to\infty}\lim_{n\to\infty}nR^\Sigma_{n,M}(2^{-1/4}(1,1)^\mathsf{T})=\frac{1}{2}\operatorname{tr}(\Sigma H_\mu)=\frac{3}{4\sqrt{2}}(\Sigma_{11}+\Sigma_{22}-2\Sigma_{12}). M → ∞ lim n → ∞ lim n R n , M Σ ( 2 − 1/4 ( 1 , 1 ) T ) = 2 1 tr ( Σ H μ ) = 4 2 3 ( Σ 11 + Σ 22 − 2 Σ 12 ) . The same n − 1 n^{-1} n − 1 rate applies to any μ = ( μ 1 , μ 2 ) T \mu=(\mu_1,\mu_2)^\mathsf{T} μ = ( μ 1 , μ 2 ) T with μ 1 μ 2 ≠ 0 \mu_1\mu_2\ne0 μ 1 μ 2 = 0 .
For μ = e 1 : = ( 1 , 0 ) T \mu=e_1:=(1,0)^\mathsf{T} μ = e 1 := ( 1 , 0 ) T , note that H e 1 = 0 H_{e_1}=0 H e 1 = 0 . Uniformly for bounded t t t ,
f ⋆ ( e 1 + ε t ) = 1 + ε t 1 + ε 4 t 2 4 4 + o ( ε 4 ) . f_\star(e_1+\varepsilon t)=1+\varepsilon t_1+\frac{\varepsilon^4t_2^4}{4}+o(\varepsilon^4). f ⋆ ( e 1 + εt ) = 1 + ε t 1 + 4 ε 4 t 2 4 + o ( ε 4 ) . Theorem 3.3 therefore applies with a ( ε ) = ε 4 a(\varepsilon)=\varepsilon^4 a ( ε ) = ε 4 and ψ ( t ) = t 2 4 / 4 \psi(t)=t_2^4/4 ψ ( t ) = t 2 4 /4 , yielding R n , M Σ ( e 1 ) ≍ n − 2 R^\Sigma_{n,M}(e_1)\asymp n^{-2} R n , M Σ ( e 1 ) ≍ n − 2 . More generally, at any nonzero point on either coordinate axis, the same n − 2 n^{-2} n − 2 rate holds.
Under the polar-norm convention (10) , dist ( μ , A ) = ∥ μ ∥ 4 \operatorname{dist}(\mu,A)=\lVert\mu\rVert_4 dist ( μ , A ) = ∥ μ ∥ 4 , so the two displayed nonzero centers both have distance 1 from A A A , yet have different local minimax rates. Theorem 3.3 shows that SAA attains all three rates; at the smooth centers with nonzero Hessian, Theorem 3.2 gives sharp attainment in the expanding-neighborhood limit. More details are given in Section E.2 of the Online Supplement.
Example 4.10 (Optimal value versus optimal decision on ℓ p ⋆ \ell_{p^\star} ℓ p ⋆ balls). Let d ≥ 2 d\ge2 d ≥ 2 , 1 < p < ∞ 1<p<\infty 1 < p < ∞ , and let p ⋆ = p / ( p − 1 ) p^\star=p/(p-1) p ⋆ = p / ( p − 1 ) . For Ω = B p ⋆ d = { w ∈ R d : ∥ w ∥ p ⋆ ≤ 1 } \Omega=B^d_{p^\star}=\{w\in\mathbb{R}^d:\lVert w\rVert_{p^\star}\le1\} Ω = B p ⋆ d = { w ∈ R d : ∥ w ∥ p ⋆ ≤ 1 } ,
f ⋆ ( μ ) = ∥ μ ∥ p , w ⋆ , j ( μ ) = sgn ( μ j ) ∣ μ j ∣ p − 1 ∥ μ ∥ p p − 1 ( μ ≠ 0 ) . f_\star(\mu)=\lVert\mu\rVert_p,\qquad w_{\star,j}(\mu)=\frac{\operatorname{sgn}(\mu_j)|\mu_j|^{p-1}}{\lVert\mu\rVert_p^{p-1}}\quad(\mu\ne0). f ⋆ ( μ ) = ∥ μ ∥ p , w ⋆ , j ( μ ) = ∥ μ ∥ p p − 1 sgn ( μ j ) ∣ μ j ∣ p − 1 ( μ = 0 ) . For μ = μ 1 e 1 \mu=\mu_1e_1 μ = μ 1 e 1 , μ 1 > 0 \mu_1>0 μ 1 > 0 , the unique optimizer is w ⋆ ( μ ) = e 1 w_\star(\mu)=e_1 w ⋆ ( μ ) = e 1 and, uniformly for bounded t t t ,
f ⋆ ( μ + ε t ) = μ 1 + ε t 1 + ε p p μ 1 p − 1 ∑ j = 2 d ∣ t j ∣ p + o ( ε p ) . f_\star(\mu+\varepsilon t)=\mu_1+\varepsilon t_1+\frac{\varepsilon^p}{p\mu_1^{p-1}}\sum_{j=2}^{d}|t_j|^p+o(\varepsilon^p). f ⋆ ( μ + εt ) = μ 1 + ε t 1 + p μ 1 p − 1 ε p j = 2 ∑ d ∣ t j ∣ p + o ( ε p ) .
Estimating the optimal value still involves the first-order radial term ε t 1 \varepsilon t_1 ε t 1 , whereas decision regret subtracts that affine term and gives a ( ε ) = ε p a(\varepsilon) = \varepsilon^p a ( ε ) = ε p and ψ ( t ) = ( p μ p − 1 ) − 1 ∑ j = 2 d ∣ t j ∣ p \psi(t) = (p\mu^{p-1})^{-1}\sum_{j=2}^d |t_j|^p ψ ( t ) = ( p μ p − 1 ) − 1 ∑ j = 2 d ∣ t j ∣ p . Since d ≥ 2 d \ge2 d ≥ 2 , ψ ( t ) \psi(t) ψ ( t ) is nonaffine on every B 2 ( M ) B_2(M) B 2 ( M ) , M > 0 M > 0 M > 0 . Theorem 3.3 yields, for fixed M > 0 M > 0 M > 0 ,
R n , M Σ ( μ ) ≲ sup t ∈ B 2 ( M ) E μ , n , t , Σ R ( w ∗ ( x ˉ ) ) ≲ n − p / 2 . R^\Sigma_{n,M}(\mu) \lesssim\sup_{t \in B_2(M)} \mathbb{E}_{\mu,n,t,\Sigma} R(w_*(\bar{x})) \lesssim n^{-p/2}. R n , M Σ ( μ ) ≲ t ∈ B 2 ( M ) sup E μ , n , t , Σ R ( w ∗ ( x ˉ )) ≲ n − p /2 .
The case p = 4 p = 4 p = 4 recovers the axis-point rate in Example 4.9 .
Thus, in the present fixed-dimensional Gaussian experiment, the same problem can have a first-order n − 1 / 2 n^{-1/2} n − 1/2 value-estimation scale (under expected absolute loss), but much faster optimal-decision regret. Detailed calculations are given in Section E.3 of the Online Supplement.
Example 4.11 (A non-power rate). Choose 0 < δ < e − 2 0 < \delta< e^{-2} 0 < δ < e − 2 and define, for ∣ s ∣ ≤ δ |s| \le\delta ∣ s ∣ ≤ δ ,
g ( 0 ) = 0 , g ( s ) = ∣ s ∣ log ( 1 / ∣ s ∣ ) ( s ≠ 0 ) . g(0) = 0,\qquad g(s) = \frac{|s|}{\log(1/|s|)}\qquad(s \ne0). g ( 0 ) = 0 , g ( s ) = log ( 1/∣ s ∣ ) ∣ s ∣ ( s = 0 ) . Then g g g is even, convex, and C 1 C^1 C 1 , with g ′ ( 0 ) = 0 g'(0) = 0 g ′ ( 0 ) = 0 . Define
Ω = conv ( { 0 } ∪ { ( 1 + g ( s ) − s g ′ ( s ) , g ′ ( s ) ) : ∣ s ∣ ≤ δ } ) . \Omega= \operatorname{conv}\left(\{0\} \cup\{(1 + g(s) - sg'(s), g'(s)) : |s| \le\delta\}\right). Ω = conv ( { 0 } ∪ {( 1 + g ( s ) − s g ′ ( s ) , g ′ ( s )) : ∣ s ∣ ≤ δ } ) . For μ = e 1 \mu= e_1 μ = e 1 , there exists a unique optimizer w ∗ ( e 1 ) = e 1 w_*(e_1) = e_1 w ∗ ( e 1 ) = e 1 . For bounded t t t and sufficiently small ε > 0 \varepsilon> 0 ε > 0 , direct calculation gives
r e 1 ( ε t ) = ( 1 + ε t 1 ) g ( ε t 2 1 + ε t 1 ) . r_{e_1}(\varepsilon t) = (1 + \varepsilon t_1)g\left(\frac{\varepsilon t_2}{1 + \varepsilon t_1}\right). r e 1 ( εt ) = ( 1 + ε t 1 ) g ( 1 + ε t 1 ε t 2 ) . For every fixed M > 0 M > 0 M > 0 ,
lim ε ↓ 0 sup t ∈ B 2 ( M ) ∣ r e 1 ( ε t ) ε / log ( 1 / ε ) − ∣ t 2 ∣ ∣ = 0 , \lim_{\varepsilon\downarrow0}\sup_{t \in B_2(M)}\left|\frac{r_{e_1}(\varepsilon t)}{\varepsilon/\log(1/\varepsilon)} - |t_2|\right| = 0, ε ↓ 0 lim t ∈ B 2 ( M ) sup ε / log ( 1/ ε ) r e 1 ( εt ) − ∣ t 2 ∣ = 0 , Hence a ( ε ) = ε / log ( 1 / ε ) a(\varepsilon) = \varepsilon/\log(1/\varepsilon) a ( ε ) = ε / log ( 1/ ε ) , and ψ ( t ) = ∣ t 2 ∣ \psi(t) = |t_2| ψ ( t ) = ∣ t 2 ∣ is not affine. Theorem 3.3 gives
R n , M Σ ( e 1 ) ≲ 1 n log n . R^\Sigma_{n,M}(e_1) \lesssim\frac{1}{\sqrt{n}\log n}. R n , M Σ ( e 1 ) ≲ n log n 1 . attained in rate by SAA. This rate is o ( n − 1 / 2 ) o(n^{-1/2}) o ( n − 1/2 ) but slower than n − α / 2 n^{-\alpha/2} n − α /2 for every α > 1 \alpha> 1 α > 1 . Thus uniqueness and differentiability do not force a power expansion, and a theory restricted to integer-order Taylor terms would miss legitimate local regimes. More details are given in Section E.4 of the Online Supplement.
Table 1. Representative local geometries and nominal minimax rates.
Local Minimax Risk on KL Neighborhoods
The robust local experiment
For the following mean parametrization, assume that, for some η > 0 \eta> 0 η > 0 ,
E P x = μ , Cov P ( x ) = Σ ∈ C , \mathbb{E}_{P}x = \mu,\qquad\operatorname{Cov}_{P}(x) = \Sigma\in\mathcal{C}, E P x = μ , Cov P ( x ) = Σ ∈ C ,
E P e θ ⊤ ( x − μ ) < ∞ , ∥ θ ∥ 2 < η . (42) \mathbb{E}_{P}e^{\theta^{\top}(x-\mu)} < \infty,\qquad\lVert\theta\rVert_{2} < \eta. \tag*{(42)} E P e θ ⊤ ( x − μ ) < ∞ , ∥ θ ∥ 2 < η . (42)
Define
Λ P ( θ ) : = log E P e θ ⊤ ( x − μ ) , ∥ θ ∥ 2 < η . (43) \Lambda_{P}(\theta) := \log\mathbb{E}_{P}e^{\theta^{\top}(x-\mu)},\qquad\lVert\theta\rVert_{2} < \eta. \tag*{(43)} Λ P ( θ ) := log E P e θ ⊤ ( x − μ ) , ∥ θ ∥ 2 < η . (43)
The dimension d d d and the reference distribution P P P are fixed.
Throughout, μ = E P x \mu= \mathbb{E}_{P}x μ = E P x and Σ = Cov P ( x ) ≻ 0 \Sigma= \operatorname{Cov}_{P}(x) \succ0 Σ = Cov P ( x ) ≻ 0 , and P P P satisfies (42) except where a result is explicitly stated under a finite second moment only. For any probability measure Q Q Q with E Q ∥ x ∥ 2 < ∞ \mathbb{E}_{Q}\lVert x\rVert_{2} < \infty E Q ∥ x ∥ 2 < ∞ , write
μ Q : = E Q x , R Q ( w ) : = f ∗ ( μ Q ) − w ⊤ μ Q , w ∈ Ω . \mu_{Q} := \mathbb{E}_{Q}x,\qquad R_{Q}(w) := f_{*}(\mu_{Q}) - w^{\top}\mu_{Q},\qquad w \in\Omega. μ Q := E Q x , R Q ( w ) := f ∗ ( μ Q ) − w ⊤ μ Q , w ∈ Ω.
The observations x 1 , … , x n x_{1},\ldots,x_{n} x 1 , … , x n are i.i.d. with law Q Q Q . The covariance assumption Σ ∈ C \Sigma\in\mathcal{C} Σ ∈ C is only imposed on the reference law P P P .
A matched Gaussian benchmark
To isolate the statistical price of allowing full distributional perturbations, it is useful to compare the KL neighborhood with a Gaussian experiment having the same local mean geometry. For fixed Σ ≻ 0 \Sigma\succ0 Σ ≻ 0 , define
R n , M ℓ , Σ ( μ ) : = inf w ^ n sup u ∈ B 2 ( M ) E μ + Σ 1 / 2 u / n , Σ [ f ∗ ( μ + Σ 1 / 2 u n ) − w ^ n ⊤ ( μ + Σ 1 / 2 u n ) ] . (44) R_{n,M}^{\ell,\Sigma}(\mu) := \inf_{\widehat{w}_{n}}\ \sup_{u\in B_{2}(M)}\ \mathbb{E}_{\mu+\Sigma^{1/2}u/\sqrt{n},\Sigma}\left[f_{*}\left(\mu+\frac{\Sigma^{1/2}u}{\sqrt{n}}\right)-\widehat{w}_{n}^{\top}\left(\mu+\frac{\Sigma^{1/2}u}{\sqrt{n}}\right)\right]. \tag*{(44)} R n , M ℓ , Σ ( μ ) := w n inf u ∈ B 2 ( M ) sup E μ + Σ 1/2 u / n , Σ [ f ∗ ( μ + n Σ 1/2 u ) − w n ⊤ ( μ + n Σ 1/2 u ) ] . (44)
The local parameter set is the same covariance ellipsoid that will emerge from the KL constraint.
Corollary 5.1 (Matched ellipsoidal Gaussian experiment). Under the local expansion (25) , for every fixed M > 0 M > 0 M > 0 ,
lim n → ∞ R n , M e l l , Σ ( μ ) a ( n − 1 / 2 ) = V M , I d ( ψ ∘ Σ 1 / 2 ) . (45) \lim_{n \to\infty} \frac{R_{n,M}^{\mathrm{ell},\Sigma}(\mu)}{a(n^{-1/2})} = V_{M,\mathrm{Id}}(\psi\circ\Sigma^{1/2}). \tag*{(45)} n → ∞ lim a ( n − 1/2 ) R n , M ell , Σ ( μ ) = V M , Id ( ψ ∘ Σ 1/2 ) . (45) If ψ \psi ψ is nonaffine on Σ 1 / 2 B 2 ( M ) \Sigma^{1/2} B_2(M) Σ 1/2 B 2 ( M ) , ordinary SAA is rate-optimal for this Gaussian benchmark.
If f ∗ f_* f ∗ is C 2 C^2 C 2 near μ \mu μ , then
lim n → ∞ n R n , M e l l , Σ ( μ ) = 1 2 B M , I d ( Σ 1 / 2 H μ Σ 1 / 2 ) , \lim_{n \to\infty} nR_{n,M}^{\mathrm{ell},\Sigma}(\mu) = \frac{1}{2}\mathcal{B}_{M,\mathrm{Id}}\left(\Sigma^{1/2} H_\mu\Sigma^{1/2}\right), n → ∞ lim n R n , M ell , Σ ( μ ) = 2 1 B M , Id ( Σ 1/2 H μ Σ 1/2 ) , and the expanding-neighborhood constant is tr ( Σ H μ ) / 2 \operatorname{tr}(\Sigma H_\mu)/2 tr ( Σ H μ ) /2 .
KL neighborhoods and a truncated mean
For M > 0 M > 0 M > 0 , define
Q n , M K L ( P ) : = { Q ≪ P : E Q ∥ x ∥ 2 < ∞ , D K L ( Q ∥ P ) ≤ M 2 2 n } , \mathcal{Q}_{n,M}^{\mathrm{KL}}(P) := \left\{Q \ll P : \mathbb{E}_Q\lVert x\rVert_2 < \infty,\ D_{\mathrm{KL}}(Q\mathbin{\|}P) \le\frac{M^2}{2n}\right\}, Q n , M KL ( P ) := { Q ≪ P : E Q ∥ x ∥ 2 < ∞ , D KL ( Q ∥ P ) ≤ 2 n M 2 } ,
R n , M K L ( P ) : = inf w ^ n sup Q ∈ Q n , M K L ( P ) E Q ⊗ n R Q ( w ^ n ) . (46) \mathcal{R}_{n,M}^{\mathrm{KL}}(P) := \inf_{\widehat{w}_n} \sup_{Q \in\mathcal{Q}_{n,M}^{\mathrm{KL}}(P)} \mathbb{E}_{Q^{\otimes n}} R_Q(\widehat{w}_n). \tag*{(46)} R n , M KL ( P ) := w n inf Q ∈ Q n , M KL ( P ) sup E Q ⊗ n R Q ( w n ) . (46)
The infimum is over measurable maps ( R d ) n → Ω (\mathbb{R}^d)^n \to\Omega ( R d ) n → Ω . As in the preceding local benchmarks, a rule chosen for a fixed ( P , n , M ) (P,n,M) ( P , n , M ) may depend on that reference model, but not on the unknown Q Q Q . The common truncated plug-in policy defined below uses neither P P P nor μ \mu μ nor Σ \Sigma Σ .
Define the coordinatewise truncated observations and their mean by
τ n = log 2 ( n + 1 ) , x i , j ( n ) = sgn ( x i , j ) min { ∣ x i , j ∣ , τ n } , \tau_n = \log^2(n+1), \qquad x_{i,j}^{(n)} = \operatorname{sgn}(x_{i,j})\min\{|x_{i,j}|,\tau_n\}, τ n = log 2 ( n + 1 ) , x i , j ( n ) = sgn ( x i , j ) min { ∣ x i , j ∣ , τ n } ,
μ ~ n = 1 n ∑ i = 1 n x i ( n ) , Z n , Q = n ( μ ~ n − μ Q ) . (47) \widetilde{\mu}_n = \frac{1}{n}\sum_{i=1}^{n} x_i^{(n)}, \qquad Z_{n,Q} = \sqrt{n}(\widetilde{\mu}_n-\mu_Q). \tag*{(47)} μ n = n 1 i = 1 ∑ n x i ( n ) , Z n , Q = n ( μ n − μ Q ) . (47)
The truncation threshold τ n \tau_n τ n is deterministic and does not use P P P , μ \mu μ , or Σ \Sigma Σ . Uniform mean-ellipsoid bounds, moment bounds, and a Gaussian approximation for Z n , Q Z_{n,Q} Z n , Q are given in the Online Supplement.
The threshold is applied in a fixed coordinate system and fixed units. The rule is neither affine-equivariant nor invariant to changes of origin, although the asymptotic results hold for every fixed such choice.
Exact local minimax risk
Theorem 5.2 (Exact local minimax limit on a KL neighborhood). Assume (25) and let P P P satisfy
E P ∥ x ∥ 2 2 < ∞ , μ = E P x , Σ = Cov P ( x ) ≻ 0. \mathbb{E}_{P}\lVert x\rVert_{2}^{2}<\infty,\qquad\mu=\mathbb{E}_{P}x,\qquad\Sigma=\operatorname{Cov}_{P}(x)\succ0. E P ∥ x ∥ 2 2 < ∞ , μ = E P x , Σ = Cov P ( x ) ≻ 0. Fix M > 0 M>0 M > 0 .
(i) Lower bound.
lim inf n → ∞ R n , M K L ( P ) a ( n − 1 / 2 ) ≥ V M , I d ( ψ ∘ Σ 1 / 2 ) . (48) \liminf_{n\to\infty}\frac{\mathcal{R}^{\mathrm{KL}}_{n,M}(P)}{a(n^{-1/2})}\ge V_{M,\mathrm{Id}}(\psi\circ\Sigma^{1/2}). \tag*{(48)} n → ∞ lim inf a ( n − 1/2 ) R n , M KL ( P ) ≥ V M , Id ( ψ ∘ Σ 1/2 ) . (48) (ii) Exact limit. If, in addition, (42) holds, then the limit exists and
lim n → ∞ R n , M K L ( P ) a ( n − 1 / 2 ) = V M , I d ( ψ ∘ Σ 1 / 2 ) . (49) \lim_{n\to\infty}\frac{\mathcal{R}^{\mathrm{KL}}_{n,M}(P)}{a(n^{-1/2})}=V_{M,\mathrm{Id}}(\psi\circ\Sigma^{1/2}). \tag*{(49)} n → ∞ lim a ( n − 1/2 ) R n , M KL ( P ) = V M , Id ( ψ ∘ Σ 1/2 ) . (49) (iii) Rate-minimaxity of truncated SAA. Under the conditions of (ii), if V M , I d ( ψ ∘ Σ 1 / 2 ) > 0 V_{M,\mathrm{Id}}(\psi\circ\Sigma^{1/2})>0 V M , Id ( ψ ∘ Σ 1/2 ) > 0 , then the policy w ∗ ( μ ~ n ) \mathbf{w}_{*}(\widetilde{\mu}_{n}) w ∗ ( μ n ) is rate-minimax:
c a ( n − 1 / 2 ) ≤ R n , M K L ( P ) ≤ sup Q ∈ Q n , M K L ( P ) E Q ⊗ n R Q ( w ∗ ( μ ~ n ) ) ≤ C a ( n − 1 / 2 ) , (50) ca(n^{-1/2})\le\mathcal{R}^{\mathrm{KL}}_{n,M}(P)\le\sup_{Q\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)}\mathbb{E}_{Q^{\otimes n}}R_{Q}(\mathbf{w}_{*}(\widetilde{\mu}_{n}))\le Ca(n^{-1/2}), \tag*{(50)} c a ( n − 1/2 ) ≤ R n , M KL ( P ) ≤ Q ∈ Q n , M KL ( P ) sup E Q ⊗ n R Q ( w ∗ ( μ n )) ≤ C a ( n − 1/2 ) , (50) where constants 0 < c ≤ C < ∞ 0<c\le C<\infty 0 < c ≤ C < ∞ may depend on P P P , M M M and the fixed geometry.
The exact limit in (49) is identical to the matched Gaussian limit in (45) . Thus, to first order, the full KL neighborhood and the Gaussian experiment over the covariance ellipsoid have the same information bound. The lower bound (i) is information-theoretic and needs only a finite second moment. The exponential-moment condition is used in (ii) and (iii). The upper bound in (ii) uses deterministic policies based on the truncated mean, obtained by averaging out auxiliary randomization; these policies depend on P P P , M M M , and the local geometry and achieve arbitrarily small excess over the minimax constant in asymptotic normalized maximal regret. The truncated plug-in rule w ∗ ( μ ~ n ) \mathbf{w}_{*}(\widetilde{\mu}_{n}) w ∗ ( μ n ) , which does not use P P P , attains the rate in (iii) uniformly over the full KL neighborhood and, by Theorem 6.1 below, its leading constant is C M p l u g ( ψ , Σ ) ≥ V M , I d ( ψ ∘ Σ 1 / 2 ) C^{\mathrm{plug}}_{M}(\psi,\Sigma)\ge V_{M,\mathrm{Id}}(\psi\circ\Sigma^{1/2}) C M plug ( ψ , Σ ) ≥ V M , Id ( ψ ∘ Σ 1/2 ) .
The local price of robustness
Corollary 5.3 (Smooth constants and the geometric consequences). Under (42) , if f ∗ f_{*} f ∗ is C 2 C^{2} C 2 on a neighborhood of μ \mu μ , then
lim n → ∞ n R n , M K L ( P ) = 1 2 B M , I d ( Σ 1 / 2 H μ Σ 1 / 2 ) (51) \lim_{n\to\infty}n\mathcal{R}^{\mathrm{KL}}_{n,M}(P)=\frac{1}{2}\mathcal{B}_{M,\mathrm{Id}}(\Sigma^{1/2}H_{\mu}\Sigma^{1/2}) \tag*{(51)} n → ∞ lim n R n , M KL ( P ) = 2 1 B M , Id ( Σ 1/2 H μ Σ 1/2 ) (51) lim M → ∞ lim n → ∞ n R n , M K L ( P ) = 1 2 tr ( Σ H μ ) , (52) \lim_{M\to\infty}\lim_{n\to\infty}n\mathcal{R}^{\mathrm{KL}}_{n,M}(P)=\frac{1}{2}\operatorname{tr}(\Sigma H_{\mu}), \tag*{(52)} M → ∞ lim n → ∞ lim n R n , M KL ( P ) = 2 1 tr ( Σ H μ ) , (52)
lim n → ∞ sup Q ∈ Q n , M K L ( P ) ∣ n E Q ⊗ n R Q ( w ∗ ( μ ~ n ) ) − 1 2 tr ( Σ H μ ) ∣ = 0. (53) \lim_{n\to\infty}\sup_{Q\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)}\left|n\mathbb{E}_{Q^{\otimes n}}R_Q(w_*(\widetilde{\mu}_n))-\frac{1}{2}\operatorname{tr}(\Sigma H_\mu)\right|=0. \tag*{(53)} n → ∞ lim Q ∈ Q n , M KL ( P ) sup n E Q ⊗ n R Q ( w ∗ ( μ n )) − 2 1 tr ( Σ H μ ) = 0. (53)
Thus w ∗ ( μ ~ n ) w_*(\widetilde{\mu}_n) w ∗ ( μ n ) attains the sharp constant after M → ∞ M\to\infty M → ∞ . The geometric identities for H μ H_\mu H μ in Corollaries 4.2 and 4.3 remain valid under their respective assumptions.
There are three distinct notions of a robustness cost. First, relative to the matched ellipsoidal Gaussian benchmark, the KL neighborhood is first-order costless: it has the same exact fixed-M M M minimax limit. Second, comparison with the Euclidean-ball Gaussian benchmark can produce a different fixed-M M M constant because the two experiments use different local parameter sets. Third, at smooth instances and after M → ∞ M\to\infty M → ∞ , all of these formulations have the same constant tr ( Σ H μ ) / 2 \operatorname{tr}(\Sigma H_\mu)/2 tr ( Σ H μ ) /2 . Local robustness is therefore asymptotically free at the ordinary smooth efficiency scale, even though a tail-stable implementation may be needed to attain the optimal rate or sharp constant uniformly over the KL neighborhood.
Corollary 5.4 (Boundary contact rates on KL neighborhoods). Under the two-sided local boundary condition of Theorem 4.6 , and under the moment conditions of Theorem 5.2 ,
R n , M K L ( P ) ≍ sup Q ∈ Q n , M K L ( P ) E Q ⊗ n R Q ( w ∗ ( μ ~ n ) ) ≲ n − ( 1 + β ) / ( 2 β ) . \mathcal{R}^{\mathrm{KL}}_{n,M}(P)\asymp\sup_{Q\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)}\mathbb{E}_{Q^{\otimes n}}R_Q(w_*(\widetilde{\mu}_n))\lesssim n^{-(1+\beta)/(2\beta)}. R n , M KL ( P ) ≍ Q ∈ Q n , M KL ( P ) sup E Q ⊗ n R Q ( w ∗ ( μ n )) ≲ n − ( 1 + β ) / ( 2 β ) . The lower bound needs only a finite second moment, while the matching truncated-SAA upper bound uses the local exponential-moment condition.
Limits of SAA under Distributional Robustness
Keep P P P , d d d , Ω \Omega Ω , M M M fixed and define
W n , M m e a n ( P ) = sup Q ∈ Q n , M K L ( P ) E Q ⊗ n R Q ( w ∗ ( x ˉ ) ) , W^{\mathrm{mean}}_{n,M}(P)=\sup_{Q\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)}\mathbb{E}_{Q^{\otimes n}}R_Q(w_*(\bar{x})), W n , M mean ( P ) = Q ∈ Q n , M KL ( P ) sup E Q ⊗ n R Q ( w ∗ ( x ˉ )) ,
W n , M t r ( P ) = sup Q ∈ Q n , M K L ( P ) E Q ⊗ n R Q ( w ∗ ( μ ~ n ) ) , Δ n = x ˉ − μ ~ n . W^{\mathrm{tr}}_{n,M}(P)=\sup_{Q\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)}\mathbb{E}_{Q^{\otimes n}}R_Q(w_*(\widetilde{\mu}_n)),\qquad\Delta_n=\bar{x}-\widetilde{\mu}_n. W n , M tr ( P ) = Q ∈ Q n , M KL ( P ) sup E Q ⊗ n R Q ( w ∗ ( μ n )) , Δ n = x ˉ − μ n .
The truncated and ordinary means have the same sequential Gaussian limit under local alternatives. However, their expected regrets need not agree, because the faster geometric regimes amplify rare tail events, which may alter the rate of maximal expected regret at faster geometric scales and its leading constant even at smooth quadratic instances; see Example 6.5.
Leading plug-in risks
Let ψ \psi ψ be the limit in (25) . As a finite convex function, ψ \psi ψ is locally Lipschitz and differentiable Lebesgue-almost everywhere. For Z ∼ N ( 0 , Σ ) Z\sim N(0,\Sigma) Z ∼ N ( 0 , Σ ) with Σ ≻ 0 \Sigma\succ0 Σ ≻ 0 , the vector
t + Z t + Z t + Z has a density, so ∇ ψ ( t + Z ) \nabla\psi(t + Z) ∇ ψ ( t + Z ) is almost surely well defined, and we define
C M p l u g ( ψ , Σ ) : = sup ∥ Σ − 1 / 2 t ∥ 2 ≤ M E [ ψ ( t ) − ψ ( t + Z ) + ∇ ψ ( t + Z ) T Z ] . (54) C_M^{\mathrm{plug}}(\psi,\Sigma) := \sup_{\lVert\Sigma^{-1/2}t\rVert_2\le M} \mathbb{E}\left[\psi(t)-\psi(t+Z)+\nabla\psi(t+Z)^\mathsf{T}Z\right]. \tag*{(54)} C M plug ( ψ , Σ ) := ∥ Σ − 1/2 t ∥ 2 ≤ M sup E [ ψ ( t ) − ψ ( t + Z ) + ∇ ψ ( t + Z ) T Z ] . (54)
Theorem 6.1 (Asymptotic comparison of maximal plug-in regrets). Assume (42) and (25) , and fix M > 0 M>0 M > 0 such that V M , I d ( ψ ∘ Σ 1 / 2 ) > 0 V_{M,\mathrm{Id}}(\psi\circ\Sigma^{1/2})>0 V M , Id ( ψ ∘ Σ 1/2 ) > 0 . Then 0 < C M p l u g ( ψ , Σ ) < ∞ 0<C_M^{\mathrm{plug}}(\psi,\Sigma)<\infty 0 < C M plug ( ψ , Σ ) < ∞ , and
lim n → ∞ W n , M t r ( P ) a ( n − 1 / 2 ) = C M p l u g ( ψ , Σ ) ≤ lim inf n → ∞ W n , M m e a n ( P ) a ( n − 1 / 2 ) . (55) \lim_{n\to\infty}\frac{W_{n,M}^{\mathrm{tr}}(P)}{a(n^{-1/2})}=C_M^{\mathrm{plug}}(\psi,\Sigma)\le\liminf_{n\to\infty}\frac{W_{n,M}^{\mathrm{mean}}(P)}{a(n^{-1/2})}. \tag*{(55)} n → ∞ lim a ( n − 1/2 ) W n , M tr ( P ) = C M plug ( ψ , Σ ) ≤ n → ∞ lim inf a ( n − 1/2 ) W n , M mean ( P ) . (55) In particular, Theorem 6.1 implies that
lim sup n → ∞ W n , M t r ( P ) − W n , M m e a n ( P ) a ( n − 1 / 2 ) ≤ 0 , lim sup n → ∞ W n , M t r ( P ) W n , M m e a n ( P ) ≤ 1. \limsup_{n\to\infty}\frac{W_{n,M}^{\mathrm{tr}}(P)-W_{n,M}^{\mathrm{mean}}(P)}{a(n^{-1/2})}\le0,\qquad\limsup_{n\to\infty}\frac{W_{n,M}^{\mathrm{tr}}(P)}{W_{n,M}^{\mathrm{mean}}(P)}\le1. n → ∞ lim sup a ( n − 1/2 ) W n , M tr ( P ) − W n , M mean ( P ) ≤ 0 , n → ∞ lim sup W n , M mean ( P ) W n , M tr ( P ) ≤ 1. Strict improvement of the rate is exhibited in Example 6.5.
The constant C M p l u g C_M^{\mathrm{plug}} C M plug is the maximal asymptotic Gaussian plug-in risk instead of the minimax constant V M , I d V_{M,\mathrm{Id}} V M , Id . The theorem says that truncation never worsens the leading maximal plug-in regret in that ordinary SAA cannot have a smaller first-order maximal risk. The remaining issue is whether the ordinary mean has enough tail control to match the truncated rule.
A geometry-weighted tail criterion
Proposition 6.2 (A geometry-weighted tail criterion for the ordinary mean). Under the assumptions of Theorem 6.1 , there is C < ∞ C<\infty C < ∞ such that, for all sufficiently large n n n ,
W n , M m e a n ( P ) ≤ C a ( n − 1 / 2 ) + 1 2 sup Q ∈ Q n , M K L ( P ) E Q ⊗ n r μ ( 4 Δ n ) ≤ C { a ( n − 1 / 2 ) + n − 1 } , (56) W_{n,M}^{\mathrm{mean}}(P)\le Ca(n^{-1/2})+\frac{1}{2}\sup_{Q\in\mathcal{Q}_{n,M}^{\mathrm{KL}}(P)}\mathbb{E}_{Q^{\otimes n}}r_\mu(4\Delta_n)\le C\{a(n^{-1/2})+n^{-1}\}, \tag*{(56)} W n , M mean ( P ) ≤ C a ( n − 1/2 ) + 2 1 Q ∈ Q n , M KL ( P ) sup E Q ⊗ n r μ ( 4 Δ n ) ≤ C { a ( n − 1/2 ) + n − 1 } , (56) where the constants may depend on the fixed P P P , M M M , d d d , Ω \Omega Ω and the local expansion.
The criterion is directional: it measures the truncation error through r μ r_\mu r μ rather than through a generic norm. The conditions below are sufficient. This distinction matters because only the directions to which the optimizer is locally sensitive contribute to regret.
If
sup Q ∈ Q n , M K L ( P ) E Q ⊗ n r μ ( 4 Δ n ) = O ( a ( n − 1 / 2 ) ) . (57) \sup_{Q\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)} \mathbb{E}_{Q\otimes n}r_\mu(4\Delta_n)=O(a(n^{-1/2})). \tag*{(57)} Q ∈ Q n , M KL ( P ) sup E Q ⊗ n r μ ( 4 Δ n ) = O ( a ( n − 1/2 )) . (57)
then Proposition 6.2 implies that
W n , M m e a n ( P ) ≍ W n , M t r ( P ) ≍ R n , M K L ( P ) ≍ a ( n − 1 / 2 ) . W^{\mathrm{mean}}_{n,M}(P)\asymp W^{\mathrm{tr}}_{n,M}(P)\asymp\mathcal{R}^{\mathrm{KL}}_{n,M}(P)\asymp a(n^{-1/2}). W n , M mean ( P ) ≍ W n , M tr ( P ) ≍ R n , M KL ( P ) ≍ a ( n − 1/2 ) .
The stronger condition
sup Q ∈ Q n , M K L ( P ) E Q ⊗ n r μ ( 4 Δ n ) = o ( a ( n − 1 / 2 ) ) (58) \sup_{Q\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)} \mathbb{E}_{Q\otimes n}r_\mu(4\Delta_n)=o(a(n^{-1/2})) \tag*{(58)} Q ∈ Q n , M KL ( P ) sup E Q ⊗ n r μ ( 4 Δ n ) = o ( a ( n − 1/2 )) (58)
on the other hand gives
lim n → ∞ W n , M m e a n ( P ) a ( n − 1 / 2 ) = lim n → ∞ W n , M t r ( P ) a ( n − 1 / 2 ) = C M p l u g ( ψ , Σ ) . (59) \lim_{n\to\infty}\frac{W^{\mathrm{mean}}_{n,M}(P)}{a(n^{-1/2})}
=
\lim_{n\to\infty}\frac{W^{\mathrm{tr}}_{n,M}(P)}{a(n^{-1/2})}
=
C_M^{\mathrm{plug}}(\psi,\Sigma). \tag*{(59)} n → ∞ lim a ( n − 1/2 ) W n , M mean ( P ) = n → ∞ lim a ( n − 1/2 ) W n , M tr ( P ) = C M plug ( ψ , Σ ) . (59)
In particular, lim inf n → ∞ n a ( n − 1 / 2 ) > 0 \liminf_{n\to\infty}na(n^{-1/2})>0 lim inf n → ∞ na ( n − 1/2 ) > 0 suffices for rate-optimality, and n a ( n − 1 / 2 ) → ∞ na(n^{-1/2})\to\infty na ( n − 1/2 ) → ∞ suffices for (59) .
A phase boundary for power geometry and tails
Corollary 6.3 (Local order and reference tail strength). Under the assumptions of Theorem 6.1 , suppose a ( ϵ ) = ϵ α a(\epsilon)=\epsilon^\alpha a ( ϵ ) = ϵ α for some α ≥ 1 \alpha\ge1 α ≥ 1 , and that for some ζ ≥ 1 \zeta\ge1 ζ ≥ 1 and c > 0 c>0 c > 0 ,
E P e c ∥ x ∥ 2 ζ < ∞ . (60) \mathbb{E}_{P}e^{c\lVert x\rVert_2^\zeta}<\infty. \tag*{(60)} E P e c ∥ x ∥ 2 ζ < ∞. (60) Then
sup Q ∈ Q n , M K L ( P ) E Q ⊗ n ∥ Δ n ∥ 2 ζ = O ( n − ζ ) , \sup_{Q\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)}\mathbb{E}_{Q\otimes n}\lVert\Delta_n\rVert_2^\zeta=O(n^{-\zeta}), Q ∈ Q n , M KL ( P ) sup E Q ⊗ n ∥ Δ n ∥ 2 ζ = O ( n − ζ ) , and
W n , M m e a n ( P ) ≤ C ( n − α / 2 + n − ζ ) . (61) W^{\mathrm{mean}}_{n,M}(P)\le C(n^{-\alpha/2}+n^{-\zeta}). \tag*{(61)} W n , M mean ( P ) ≤ C ( n − α /2 + n − ζ ) . (61) This corollary indicates: if α ≤ 2 ζ \alpha\le2\zeta α ≤ 2 ζ , the ordinary-mean policy is rate-minimax:
W n , M m e a n ( P ) ≍ W n , M t r ( P ) ≍ R n , M K L ( P ) ≍ n − α / 2 . W^{\mathrm{mean}}_{n,M}(P)\asymp W^{\mathrm{tr}}_{n,M}(P)\asymp\mathcal{R}^{\mathrm{KL}}_{n,M}(P)\asymp n^{-\alpha/2}. W n , M mean ( P ) ≍ W n , M tr ( P ) ≍ R n , M KL ( P ) ≍ n − α /2 . If α < 2 ζ \alpha<2\zeta α < 2 ζ , it also has the same asymptotic maximal regret constant as the truncated policy:
n α / 2 W n , M m e a n ( P ) → C M p l u g ( ψ , Σ ) . n^{\alpha/2}W^{\mathrm{mean}}_{n,M}(P)\to C_M^{\mathrm{plug}}(\psi,\Sigma). n α /2 W n , M mean ( P ) → C M plug ( ψ , Σ ) . For α > 2 ζ \alpha>2\zeta α > 2 ζ , rate-optimality is not guaranteed. The exponent ζ \zeta ζ describes an exponential moment of the reference distribution P P P .
Table 3. Geometry-tail regimes for ordinary SAA on shrinking KL neighborhoods.
The boundary α = 2 ζ \alpha= 2\zeta α = 2 ζ separates geometry that is slow enough for the raw sample mean from geometry that is fast enough to expose rare tail contamination. More precisely, ordinary and truncated SAA have the same leading maximal-regret constant when α < 2 ζ \alpha< 2\zeta α < 2 ζ ; ordinary SAA remains rate-minimax but can have strictly larger leading maximal regret when α = 2 ζ \alpha= 2\zeta α = 2 ζ ; and ordinary SAA can be rate-suboptimal when α > 2 ζ \alpha> 2\zeta α > 2 ζ . In the regular quadratic case α = 2 \alpha= 2 α = 2 , any subexponential reference law with ζ ≥ 1 \zeta\ge1 ζ ≥ 1 lies on the rate-optimal side. Rate suboptimality becomes possible only when the local optimization problem is easier than the tail behavior of the sample mean.
Corollary 6.4 (A smooth upper bound allowing degenerate curvature). Assume (42) , and suppose f ∗ f_{*} f ∗ is C 2 C^{2} C 2 near μ \mu μ . For each fixed M > 0 M > 0 M > 0 ,
sup Q ∈ Q n , M K L ( P ) E Q ⊗ n R Q ( w ∗ ( x ˉ ) ) = O ( n − 1 ) . (62) \sup_{Q \in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)} \mathbb{E}_{Q^{\otimes n}} R_Q(w_{*}(\bar{x})) = O(n^{-1}). \tag*{(62)} Q ∈ Q n , M KL ( P ) sup E Q ⊗ n R Q ( w ∗ ( x ˉ )) = O ( n − 1 ) . (62) If H μ ≠ 0 H_{\mu} \ne0 H μ = 0 , then the sample-mean policy is rate-minimax on this KL neighborhood.
This corollary clarifies the role of degeneracy. Twice differentiability alone gives an O ( n − 1 ) O(n^{-1}) O ( n − 1 ) robust bound for ordinary SAA. If the Hessian is nonzero, this matches the minimax rate. If the Hessian vanishes and a faster rate is attainable, ordinary SAA may remain stuck at n − 1 n^{-1} n − 1 .
A sharp separation example
The mechanism can be seen from a back-of-the-envelope calculation (heuristic, not used in the proofs). Suppose P P P has tails of order e − ∥ x ∥ ζ e^{-\lVert x\rVert^{\zeta}} e − ∥ x ∥ ζ in some coordinate. A perturbation Q ∈ Q n , M K L ( P ) Q \in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P) Q ∈ Q n , M KL ( P ) may move mass q q q to distance of order s s s at KL cost of about q s ζ q s^{\zeta} q s ζ , so q ≈ n − 1 s − ζ q \approx n^{-1}s^{-\zeta} q ≈ n − 1 s − ζ is admissible. Taking s ≍ n s \asymp n s ≍ n gives q ≈ n − 1 − ζ q \approx n^{-1-\zeta} q ≈ n − 1 − ζ . This perturbation changes μ Q \mu_Q μ Q only by O ( n − 1 − ζ ⋅ n ) = O ( n − ζ ) O(n^{-1-\zeta}\cdot n) = O(n^{-\zeta}) O ( n − 1 − ζ ⋅ n ) = O ( n − ζ ) , which is negligible on the n − 1 / 2 n^{-1/2} n − 1/2 scale, but with probability about n q ≈ n − ζ nq \approx n^{-\zeta} n q ≈ n − ζ one observation lands in the perturbed region and displaces x ˉ \bar{x} x ˉ by a constant amount. The resulting regret is of order one, so the expected regret of ordinary SAA carries a term of order n − ζ n^{-\zeta} n − ζ , which dominates n − α / 2 n^{-\alpha/2} n − α /2 when α > 2 ζ \alpha> 2\zeta α > 2 ζ . Truncation at τ n = log 2 ( n + 1 ) \tau_n = \log^2(n+1) τ n = log 2 ( n + 1 ) caps the displacement caused by any single observation at τ n / n \tau_n/n τ n / n and removes this term.
Example 6.5 (Power geometry with exponential-power tails). Fix ζ ≥ 1 \zeta\ge1 ζ ≥ 1 and α > 1 \alpha> 1 α > 1 (the restriction α > 1 \alpha> 1 α > 1 ensures that the dual exponent α / ( α − 1 ) \alpha/(\alpha-1) α / ( α − 1 ) is finite). Let d = 2 d=2 d = 2 and let P P P be the law of x = ( x 1 , x 2 ) \boldsymbol{x}=(x_1,x_2) x = ( x 1 , x 2 ) with independent coordinates, P { x 1 = 1 / 2 } = P { x 1 = 3 / 2 } = 1 / 2 \mathbb{P}\{x_1=1/2\}=\mathbb{P}\{x_1=3/2\}=1/2 P { x 1 = 1/2 } = P { x 1 = 3/2 } = 1/2 and x 2 x_2 x 2 having density proportional to e − ∣ z ∣ ζ e^{-|z|^\zeta} e − ∣ z ∣ ζ (this is G.16 in the Online Supplement). Then P P P has mean e 1 e_1 e 1 and satisfies (42) and (60) . Take the feasible set Ω = B α / ( α − 1 ) 2 \Omega=B_{\alpha/(\alpha-1)}^2 Ω = B α / ( α − 1 ) 2 , the unit ℓ α / ( α − 1 ) \ell_{\alpha/(\alpha-1)} ℓ α / ( α − 1 ) -ball in R 2 \mathbb{R}^2 R 2 , so that the local support remainder at e 1 e_1 e 1 has order ε α \varepsilon^\alpha ε α . Then
R n , M K L ( P ) ≍ W n , M t r ( P ) ≲ n − α / 2 , W n , M m e a n ( P ) ≲ n − α / 2 + n − ζ . \mathcal{R}_{n,M}^{\mathrm{KL}}(P)\asymp W_{n,M}^{\mathrm{tr}}(P)\lesssim n^{-\alpha/2},\qquad W_{n,M}^{\mathrm{mean}}(P)\lesssim n^{-\alpha/2}+n^{-\zeta}. R n , M KL ( P ) ≍ W n , M tr ( P ) ≲ n − α /2 , W n , M mean ( P ) ≲ n − α /2 + n − ζ .
Hence the phase boundary in Corollary 6.3 is sharp in rate. In particular, with Laplace tails (ζ = 1 \zeta=1 ζ = 1 ) and quartic geometry (α = 4 \alpha=4 α = 4 ),
W n , M m e a n ( P ) ≲ n − 1 , W n , M t r ( P ) ≳ R n , M K L ( P ) ≳ n − 2 . W_{n,M}^{\mathrm{mean}}(P)\lesssim n^{-1},\qquad W_{n,M}^{\mathrm{tr}}(P)\gtrsim\mathcal{R}_{n,M}^{\mathrm{KL}}(P)\gtrsim n^{-2}. W n , M mean ( P ) ≲ n − 1 , W n , M tr ( P ) ≳ R n , M KL ( P ) ≳ n − 2 .
Ordinary SAA is then strictly suboptimal in rate despite sharing the same Gaussian weak limit as the truncated procedure. At the boundary ( ζ , α ) = ( 1 , 2 ) (\zeta,\alpha)=(1,2) ( ζ , α ) = ( 1 , 2 ) , the same construction with Ω = B 2 2 \Omega=B_2^2 Ω = B 2 2 gives, for every fixed M > 0 M>0 M > 0 ,
lim inf n → ∞ n W n , M m e a n ( P ) ≥ 1 + M 2 4 ( 1 − 2 − 1 / 2 ) > 1 = lim n → ∞ n W n , M t r ( P ) . \liminf_{n\to\infty} nW_{n,M}^{\mathrm{mean}}(P)\ge1+\frac{M^2}{4}(1-2^{-1/2})>1=\lim_{n\to\infty} nW_{n,M}^{\mathrm{tr}}(P). n → ∞ lim inf n W n , M mean ( P ) ≥ 1 + 4 M 2 ( 1 − 2 − 1/2 ) > 1 = n → ∞ lim n W n , M tr ( P ) .
Thus equality of the leading maximal regrets can fail even at a smooth quadratic instance. The construction and proofs are given in Section G.5 of the Online Supplement.
Remark 6.6 (Scope of the separation). The separation concerns expected regret. By (G.7) and Lemma F.1 of the Online Supplement, the ordinary and truncated means are asymptotically equivalent at the n − 1 / 2 n^{-1/2} n − 1/2 scale uniformly over the KL neighborhood and have the same centered Gaussian limit; the gap arises only because regret is evaluated in expectation uniformly over KL-admissible laws, and rare observations of order-n n n magnitude contribute O ( 1 ) O(1) O ( 1 ) regret with probability of order n − ζ n^{-\zeta} n − ζ .
Numerical Experiments
We illustrate two consequences of the local minimax theory. First, for the Gaussian experiment, the SAA procedure can attain different regret rates as the local geometry changes, even on a fixed feasible set. Second, under distributional perturbations, this geometry determines whether rare observations change the performance of ordinary SAA. Throughout, the regret is R μ ( w ) = f ∗ ( μ ) − w ⊤ μ R_\mu(w)=f_*(\mu)-w^\top\mu R μ ( w ) = f ∗ ( μ ) − w ⊤ μ . We report expected regret along the specified Gaussian local alternatives or under the specified distributions Q n Q_n Q n . We compare these pointwise estimates with the minimax rates and constants derived above. Error bars are pointwise 95% Monte Carlo intervals.
For the Gaussian experiments, we generate the exact sufficient statistic x ˉ = μ + ( t + Z ) / n \bar{x} = \mu+ (t + Z)/\sqrt{n} x ˉ = μ + ( t + Z ) / n , with Z ∼ N ( 0 , I 2 ) Z \sim\mathcal{N}(0,I_2) Z ∼ N ( 0 , I 2 ) , using 10 5 10^5 1 0 5 independent replications per sample size. For the Laplace experiments, we generate all n n n observations in each replication, using 100 n 100n 100 n replications for M = 4 M = 4 M = 4 and 800 n 800n 800 n for M = 1 M = 1 M = 1 . Each reported value is the unweighted average of the realized regrets. The procedures use common observations at a given n n n ; random streams are independent across sample sizes. Monte Carlo standard errors use the sample standard deviation divided by the square root of the replication count; ratio errors account for pairing. Fitted slopes regress log estimated regret on log n n n over the final four sample sizes. The reported standard errors measure Monte Carlo uncertainty in the slopes fitted over these four sample sizes.
One feasible set, three local rates
Consider the feasible set in Example 4.9 ,
Ω = B 4 / 3 2 = { w ∈ R 2 : ∥ w ∥ 4 / 3 ≤ 1 } , f ∗ ( z ) = ∥ z ∥ 4 , w ∗ , j ( z ) = z j 3 ∥ z ∥ 4 3 ( z ≠ 0 ) , (63) \Omega= B^{2}_{4/3} = \{w \in\mathbb{R}^{2} : \lVert w\rVert_{4/3} \le1\}, \qquad f_{\ast}(z) = \lVert z\rVert_{4}, \qquad w_{\ast,j}(z) = \frac{z_{j}^{3}}{\lVert z\rVert_{4}^{3}} \quad(z \ne0), \tag*{(63)} Ω = B 4/3 2 = { w ∈ R 2 : ∥ w ∥ 4/3 ≤ 1 } , f ∗ ( z ) = ∥ z ∥ 4 , w ∗ , j ( z ) = ∥ z ∥ 4 3 z j 3 ( z = 0 ) , (63)
with w ∗ ( 0 ) = 0 w_{\ast}(0) = 0 w ∗ ( 0 ) = 0 . We fix Σ = I 2 \Sigma= I_{2} Σ = I 2 and sample x i ∼ i . i . d . N ( μ n , t , I 2 ) x_i \mathrel{\overset{\mathrm{i.i.d.}}{\sim}} \mathcal{N}(\mu_{n,t}, I_{2}) x i ∼ i.i.d. N ( μ n , t , I 2 ) , where μ n , t = μ + t / n \mu_{n,t} = \mu+ t/\sqrt{n} μ n , t = μ + t / n . Table 4 specifies the centers and directions, all with ∥ t ∥ 2 = 1 \lVert t\rVert_{2} = 1 ∥ t ∥ 2 = 1 . The quantity estimated is
Table 4. Predicted and fitted slopes for SAA expected regret along Gaussian local alternatives. Fitted slopes use n = 2 9 , 2 10 , 2 11 , 2 12 n = 2^{9}, 2^{10}, 2^{11}, 2^{12} n = 2 9 , 2 10 , 2 11 , 2 12 ; parentheses give their Monte Carlo standard errors.
E μ n , t , I 2 R μ n , t ( w ∗ ( x ˉ ) ) . \mathbb{E}_{\mu_{n,t},I_{2}} R_{\mu_{n,t}}(w_{\ast}(\bar{x})). E μ n , t , I 2 R μ n , t ( w ∗ ( x ˉ )) .
Example 4.9 gives linear, quadratic, and quartic local support expansions at these centers. Theorem 3.3 therefore yields the respective minimax orders n − 1 / 2 n^{-1/2} n − 1/2 , n − 1 n^{-1} n − 1 , and n − 2 n^{-2} n − 2 , all attained in rate by SAA. We examine these predictions over n = 2 5 , 2 6 , … , 2 12 n = 2^{5}, 2^{6}, \ldots, 2^{12} n = 2 5 , 2 6 , … , 2 12 .
Figure 1 shows the three distinct rates. The fitted slopes in Table 4 are close to − 1 / 2 -1/2 − 1/2 , − 1 -1 − 1 , and − 2 -2 − 2 , while the normalized curves become nearly horizontal at the larger sample sizes. Thus changing the center, with the feasible set and covariance held fixed, changes the statistical difficulty of the decision problem.
Figure 1. SAA expected regret along the Gaussian local alternatives in Table 4 . Panel (a) shows the three rates. Panel (b) normalizes by n \sqrt{n} n , n n n , and n 2 n^2 n 2 at the origin μ = 0 \mu= 0 μ = 0 , diagonal μ = 2 − 1 / 4 ( 1 , 1 ) T \mu= 2^{-1/4}(1,1)^{\mathsf{T}} μ = 2 − 1/4 ( 1 , 1 ) T , and axis μ = e 1 \mu= e_1 μ = e 1 , respectively. The reference level 3 / ( 2 2 ) 3/(2\sqrt{2}) 3/ ( 2 2 ) is the sharp constant for diagonal center μ = 2 − 1 / 4 ( 1 , 1 ) T \mu= 2^{-1/4}(1,1)^{\mathsf{T}} μ = 2 − 1/4 ( 1 , 1 ) T .
For the diagonal center μ = 2 − 1 / 4 ( 1 , 1 ) T \mu= 2^{-1/4}(1,1)^{\mathsf{T}} μ = 2 − 1/4 ( 1 , 1 ) T , the quadratic constant is also sharp in the local asymptotic minimax sense of Theorem 3.2 . The Hessian calculated in Section E.2 of the Online Supplement gives
H μ = 3 2 2 ( 1 − 1 − 1 1 ) , lim n → ∞ n E μ n , t , I 2 R μ n , t ( w ∗ ( x ˉ ) ) = 1 2 tr ( H μ ) = 3 2 2 ≈ 1.0607. H_{\mu} = \frac{3}{2\sqrt{2}}\begin{pmatrix}1 & -1 \\ -1 & 1\end{pmatrix}, \qquad\lim_{n\to\infty} n \mathbb{E}_{\mu_n,t,I_2} R_{\mu_n,t}\bigl(w_*(\bar{x})\bigr) = \frac{1}{2}\operatorname{tr}(H_{\mu}) = \frac{3}{2\sqrt{2}} \approx1.0607. H μ = 2 2 3 ( 1 − 1 − 1 1 ) , n → ∞ lim n E μ n , t , I 2 R μ n , t ( w ∗ ( x ˉ ) ) = 2 1 tr ( H μ ) = 2 2 3 ≈ 1.0607.
This equals the minimax constant. At n = 2 12 n = 2^{12} n = 2 12 , the normalized Monte Carlo estimate is approximately 1.0604, with standard error 4.7083 × 10 − 3 4.7083 \times10^{-3} 4.7083 × 1 0 − 3 . The other two centers illustrate the rate conclusion of Theorem 3.3 .
Local geometry and the effect of Laplace tails
We next examine how local geometry changes the effect of rare observations on expected regret. Consider the family in Example 6.5 with Laplace tails (ζ = 1 \zeta= 1 ζ = 1 ) and feasible sets Ω = B α / ( α − 1 ) 2 \Omega= B^2_{\alpha/(\alpha-1)} Ω = B α / ( α − 1 ) 2 , where α ∈ { 3 / 2 , 2 , 4 } \alpha\in\{3/2, 2, 4\} α ∈ { 3/2 , 2 , 4 } .
Under P P P , the coordinates are independent, P { x 1 = 1 / 2 } = P { x 1 = 3 / 2 } = 1 / 2 P\{x_1 = 1/2\} = P\{x_1 = 3/2\} = 1/2 P { x 1 = 1/2 } = P { x 1 = 3/2 } = 1/2 , and x 2 ∼ Laplace ( 0 , 1 ) x_2 \sim\operatorname{Laplace}(0,1) x 2 ∼ Laplace ( 0 , 1 ) . Thus μ P = e 1 \mu_P = e_1 μ P = e 1 and Σ = diag ( 1 / 4 , 2 ) \Sigma= \operatorname{diag}(1/4,2) Σ = diag ( 1/4 , 2 ) . For a fixed M > 0 M > 0 M > 0 , take the perturbation constructed in Section G.5 of the Online Supplement,
Q n = ( 1 − ε n ) P + ε n P ( ⋅ ∣ B n ) , Q_n = (1-\varepsilon_n)P + \varepsilon_n P(\mathord{\cdot}\mid B_n), Q n = ( 1 − ε n ) P + ε n P ( ⋅ ∣ B n ) ,
where ε n = M 2 / ( 4 n 2 ) \varepsilon_n = M^2/(4n^2) ε n = M 2 / ( 4 n 2 ) , and B n = { n ≤ ∣ x 2 ∣ ≤ n + 1 } B_n = \{n \leq|x_2| \leq n + 1\} B n = { n ≤ ∣ x 2 ∣ ≤ n + 1 } . Symmetry preserves the mean μ Q n = e 1 \boldsymbol{\mu}_{Q_n} = e_1 μ Q n = e 1 , and Q n ∈ Q n , M K L ( P ) Q_n \in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P) Q n ∈ Q n , M KL ( P ) for large n n n .
We compare the expected regrets of ordinary SAA, w ∗ ( x ˉ ) w_*(\bar{x}) w ∗ ( x ˉ ) , and truncated SAA, w ∗ ( μ ~ n ) w_*(\tilde{\mu}_n) w ∗ ( μ ~ n ) , based on n n n independent observations from Q n Q_n Q n . Here μ ~ n \tilde{\mu}_n μ ~ n is the coordinatewise truncated mean defined in (47) , with τ n = log 2 ( n + 1 ) \tau_n = \log^2(n+1) τ n = log 2 ( n + 1 ) . Theorem 5.2 gives the local minimax order n − α / 2 n^{-\alpha/2} n − α /2 , attained by truncated SAA. For ordinary SAA, the additional contribution of rare observations gives maximal expected regret of order n − α / 2 + n − 1 n^{-\alpha/2} + n^{-1} n − α /2 + n − 1 , as established in Example 6.5. The comparison therefore changes at α = 2 \alpha= 2 α = 2 : rare observations contribute a lower-order term when α = 3 / 2 \alpha= 3/2 α = 3/2 , change the leading constant when α = 2 \alpha= 2 α = 2 , and determine the regret rate when α = 4 \alpha= 4 α = 4 .
At α = 2 \alpha= 2 α = 2 , Corollary 5.3 gives the truncated-SAA constant tr ( Σ H e 1 ) / 2 = 1 \operatorname{tr}(\Sigma H e_1)/2 = 1 tr ( Σ H e 1 ) /2 = 1 . Along the present sequence Q n Q_n Q n , the construction in Section G.5 gives
n E Q n ⊗ n R Q n ( w ∗ ( μ ~ n ) ) → 1 , n E Q n ⊗ n R Q n ( w ∗ ( x ˉ ) ) → 1 + M 2 4 ( 1 − 2 − 1 / 2 ) . n\mathbb{E}_{Q_n^{\otimes n}}R_{Q_n}\left(w_*(\tilde{\mu}_n)\right) \to1,\qquad n\mathbb{E}_{Q_n^{\otimes n}}R_{Q_n}\left(w_*(\bar{x})\right) \to1 + \frac{M^2}{4}\left(1 - 2^{-1/2}\right). n E Q n ⊗ n R Q n ( w ∗ ( μ ~ n ) ) → 1 , n E Q n ⊗ n R Q n ( w ∗ ( x ˉ ) ) → 1 + 4 M 2 ( 1 − 2 − 1/2 ) .
For M = 4 M = 4 M = 4 , the ordinary-to-truncated expected-regret ratio tends to 1 1 1 when α = 3 / 2 \alpha= 3/2 α = 3/2 , tends to 1 + 4 ( 1 − 2 − 1 / 2 ) ≈ 2.1716 1 + 4(1 - 2^{-1/2}) \approx2.1716 1 + 4 ( 1 − 2 − 1/2 ) ≈ 2.1716 when α = 2 \alpha= 2 α = 2 , and grows at order n n n when α = 4 \alpha= 4 α = 4 . These three behaviors isolate equality of leading regret, a constant loss, and a rate loss, respectively.
We first set M = 4 M = 4 M = 4 and use the eight sample sizes n = 2 5 , 2 6 , … , 2 12 n = 2^5, 2^6, \ldots, 2^{12} n = 2 5 , 2 6 , … , 2 12 . Figure 2 displays the expected regrets and their ratios across the three regimes. The fitted slopes in Table 5 reflect the transition from a common rate below and at the threshold to different rates above it. The table also reports a quartic comparison at M = 1 M = 1 M = 1 on the same eight-point grid.
Table 5. Predicted and fitted slopes for expected regret under Q n Q_n Q n . Fits use n = 2 9 , 2 10 , 2 11 , 2 12 n = 2^9, 2^{10}, 2^{11}, 2^{12} n = 2 9 , 2 10 , 2 11 , 2 12 for both radii; the figures also include n = 2 5 , 2 6 , 2 7 , 2 8 n = 2^5, 2^6, 2^7, 2^8 n = 2 5 , 2 6 , 2 7 , 2 8 . Parentheses give Monte Carlo standard errors. The pointwise rates match the maximal-regret orders in Example 6.5.
Figure 2. Ordinary and truncated SAA under Q n Q_n Q n with M = 4 M = 4 M = 4 . The upper panels show expected regret on a common scale; the lower panels show the ordinary-to-truncated ratio. The reference level 1 marks equal expected regret and is the limiting ratio for α = 3 / 2 \alpha= 3/2 α = 3/2 . For α = 2 \alpha= 2 α = 2 , the additional reference is 1 + 4 ( 1 − 2 − 1 / 2 ) ≈ 2.1716 1 + 4(1 - 2^{-1/2}) \approx2.1716 1 + 4 ( 1 − 2 − 1/2 ) ≈ 2.1716 . For α = 4 \alpha= 4 α = 4 , the ratio grows at order n n n . The ratio scale is linear for α = 3 / 2 , 2 \alpha= 3/2, 2 α = 3/2 , 2 and logarithmic for α = 4 \alpha= 4 α = 4 .
At M = 4 M = 4 M = 4 , n = 2 12 n = 2^{12} n = 2 12 , the estimated expected-regret ratio for α = 3 / 2 \alpha= 3/2 α = 3/2 is approximately 1.2215 1.2215 1.2215 . At larger sample sizes, the estimated ratio declines toward its theoretical limit of
For α = 2 \alpha= 2 α = 2 , the ordinary and truncated estimates multiplied by n n n are approximately 2.2144 and 0.9999, with Monte Carlo standard errors 5.9606 × 10 − 2 5.9606 \times10^{-2} 5.9606 × 1 0 − 2 and 2.2062 × 10 − 3 2.2062 \times10^{-3} 2.2062 × 1 0 − 3 . Their separation persists even though both fitted slopes are close to − 1 -1 − 1 . For α = 4 \alpha= 4 α = 4 , the different slopes display the rate loss of ordinary SAA. For α = 3 / 2 \alpha= 3/2 α = 3/2 , the ordinary-SAA slope − 0.8086 -0.8086 − 0.8086 remains below its limit − 3 / 4 -3/4 − 3/4 : the rare-event term has order n − 1 n^{-1} n − 1 and need not be negligible on this grid.
Table 6 records the quartic endpoint estimates for both radii. At n = 2 12 n = 2^{12} n = 2 12 , the estimated expected-regret ratio is approximately 47.3352 for M = 1 M = 1 M = 1 and 764.6639 for M = 4 M = 4 M = 4 , with Monte Carlo standard errors 3.2736 and 37.6256, respectively.
Table 6. Quartic expected-regret estimates from direct simulation at Q n Q_n Q n . Parentheses give Monte Carlo standard errors; the last column is the ratio of the two estimates.
The results illustrate how distributional tails and local support geometry jointly affect the performance of ordinary SAA. Under the same Laplace perturbation, ordinary SAA matches the leading asymptotic expected regret of truncated SAA when α = 3 / 2 \alpha= 3/2 α = 3/2 , incurs a constant loss when α = 2 \alpha= 2 α = 2 , and becomes suboptimal in rate when α = 4 \alpha= 4 α = 4 .
Discussion
The results yield a unified but qualified view of SAA. In the nominal Gaussian mean experiment, the support-function remainder completely characterizes the instance-specific statistical difficulty, and SAA automatically attains the resulting rate. At smooth points it is sharp in the standard local asymptotic minimax limit. At nonsmooth or degenerate points it adapts to first-order, higher-order, and non-power geometry without knowing the relevant exponent.
Distributional robustness changes the implementation question more than the geometric benchmark. A KL ball of radius 1 / n 1/n 1/ n generates the same n − 1 / 2 n^{-1/2} n − 1/2 local information scale, transformed by the covariance ellipsoid of the reference distribution. Truncated SAA attains the corresponding minimax rate. Ordinary SAA can fail because weak convergence does not control expected regret uniformly over KL-admissible alternatives. The failure is therefore not a contradiction to Gaussian efficiency: it is a lack of uniform integrability caused by rare tail perturbations.
The distinction suggests three meanings of the price of robustness. Robustness can be asymptotically free in rate; it can change a fixed-radius local constant by changing the shape of the local parameter set; and it can require a tail-stable estimator even when the information bound itself is unchanged. The last phenomenon is most visible at instances with degenerate curvature, where the optimization problem permits a rate faster than n − 1 n^{-1} n − 1 . These statements compare procedures that do not use P P P with a benchmark that may depend on the reference law P P P and the radius M M M ; “robustness is free” is thus a statement about the attainable rate and smooth constant relative to a P P P -informed benchmark.
The present theory is deliberately local and fixed-dimensional. It assumes a compact convex feasible set, a linear objective whose distributional dependence enters through the mean, pointwise local asymptotics around a fixed reference instance, and one-sided neighborhoods D K L ( Q ∥ P ) D_{\mathrm{KL}}(Q\Vert P) D KL ( Q ∥ P ) . The KL lower bound requires a finite second moment and nonsingular covariance, whereas the general matching upper bound uses a local exponential-moment condition. The reference law P P P is fixed when defining the robust benchmark. These restrictions isolate the geometry-tail mechanism but leave important questions of growing dimension, nonlinear objectives, data-dependent reference models, and other notions of distributional proximity open.
Several extensions are natural. Other ϕ \phi ϕ -divergence neighborhoods should lead to related local ellipsoidal experiments with divergence-specific scaling. Wasserstein neighborhoods would interact differently with the geometry because they constrain transport rather than likelihood ratios. It would also be useful to develop adaptive, coordinate-free truncation, allow d d d to grow with n n n , and extend the analysis to nonlinear stochastic programs. Finally, combining the present geometry with parametric ETO/IEO models would separate gains from structural dimension reduction from gains due to tail stabilization.
Supplement to
Geometry, Local Minimax Risk, and the Limits of Sample Average Approximation in Linear Stochastic Optimization
This supplement contains all proofs, intermediate technical results, and detailed example calculations for the main paper. Section and equation references without an appendix letter refer to the main paper.
Proof roadmap. The Gaussian Bayes identity reduces minimax regret to a support-function Jensen gap (Section C.2). For the KL limit, bounded-score submodels yield the lower bound under a finite second moment. Under the exponential-moment condition, entropy bounds confine the local means to the covariance ellipsoid, while clipping gives uniform moment bounds and a Gaussian approximation (Lemma F.1 ). The upper bound transfers a nearly optimal Gaussian rule after restricting its actions so that its loss is uniformly of order a ( n − 1 / 2 ) a(n^{-1/2}) a ( n − 1/2 ) (Section F.3). Finally, a polynomial envelope and the clipped moment bounds give uniform integrability of truncated-SAA regret; ordinary SAA shares the weak limit but may fail this expectation bound (Section G.1).
Proofs for the Global and Local Benchmark Bounds
Proof of Proposition 2.1
Proof. Let w μ : = w ∗ ( μ ) w_\mu:= w_*(\mu) w μ := w ∗ ( μ ) and w x ˉ : = w ∗ ( x ˉ ) w_{\bar{x}} := w_*(\bar{x}) w x ˉ := w ∗ ( x ˉ ) . Optimality of w x ˉ w_{\bar{x}} w x ˉ at x ˉ \bar{x} x ˉ gives x ˉ T ( w μ − w x ˉ ) ≤ 0 \bar{x}^{\mathsf T}(w_\mu-w_{\bar{x}}) \le0 x ˉ T ( w μ − w x ˉ ) ≤ 0 . Hence
R μ ( w x ˉ ) = μ T ( w μ − w x ˉ ) ≤ ( x ˉ − μ ) T ( w x ˉ − w μ ) ≤ f ∗ ( x ˉ − μ ) + f ∗ ( μ − x ˉ ) . R_\mu(w_{\bar{x}}) = \mu^{\mathsf T}(w_\mu-w_{\bar{x}})
\le(\bar{x}-\mu)^{\mathsf T}(w_{\bar{x}}-w_\mu)
\le f_*(\bar{x}-\mu)+f_*(\mu-\bar{x}). R μ ( w x ˉ ) = μ T ( w μ − w x ˉ ) ≤ ( x ˉ − μ ) T ( w x ˉ − w μ ) ≤ f ∗ ( x ˉ − μ ) + f ∗ ( μ − x ˉ ) .
Since n ( x ˉ − μ ) ∼ N ( 0 , Σ ) \sqrt{n}(\bar{x}-\mu) \sim\mathcal{N}(0,\Sigma) n ( x ˉ − μ ) ∼ N ( 0 , Σ ) and this distribution is symmetric,
E μ , Σ R μ ( w ∗ ( x ˉ ) ) ≤ 2 n E f ∗ ( Σ 1 / 2 G ) = 2 G ( Σ 1 / 2 Ω ) n . \mathbb{E}_{\mu,\Sigma}R_\mu(w_*(\bar{x})) \le\frac{2}{\sqrt{n}}\mathbb{E}f_*(\Sigma^{1/2}\mathcal{G}) = 2\frac{\mathcal{G}(\Sigma^{1/2}\Omega)}{\sqrt{n}}. E μ , Σ R μ ( w ∗ ( x ˉ )) ≤ n 2 E f ∗ ( Σ 1/2 G ) = 2 n G ( Σ 1/2 Ω ) .
This proves the upper bound in (8) .
For the lower bound, place the Gaussian prior
μ ∼ N ( 0 , Σ n ) . \mu\sim\mathcal{N}\left(0,\frac{\Sigma}{n}\right). μ ∼ N ( 0 , n Σ ) .
Under this prior, the posterior mean based on the full sample is m = E [ μ ∣ x 1 , … , x n ] = x ˉ / 2 m=\mathbb{E}[\mu\mid x_1,\ldots,x_n]=\bar{x}/2 m = E [ μ ∣ x 1 , … , x n ] = x ˉ /2 . Conditional on the data, a Bayes action maximizes w T m w^{\mathsf T}m w T m over Ω \Omega Ω . Therefore the minimum Bayes regret is
inf w ^ n E R μ ( w ^ n ) = E f ∗ ( μ ) − E f ∗ ( m ) . \inf_{\widehat{w}_n} ER_\mu(\widehat{w}_n) = \mathbb{E}f_*(\mu) - \mathbb{E}f_*(m). w n inf E R μ ( w n ) = E f ∗ ( μ ) − E f ∗ ( m ) .
Now
μ ∼ N ( 0 , Σ n ) , m ∼ N ( 0 , Σ 2 n ) , \mu\sim\mathcal{N}\left(0,\frac{\Sigma}{n}\right), \qquad m \sim\mathcal{N}\left(0,\frac{\Sigma}{2n}\right), μ ∼ N ( 0 , n Σ ) , m ∼ N ( 0 , 2 n Σ ) ,
so positive homogeneity of f ∗ f_* f ∗ yields
inf w ^ n E R μ ( w ^ n ) = ( 1 − 1 2 ) G ( Σ 1 / 2 Ω ) n . \inf_{\widehat{w}_n} ER_\mu(\widehat{w}_n) = \left(1-\frac{1}{\sqrt{2}}\right)\frac{\mathcal{G}(\Sigma^{1/2}\Omega)}{\sqrt{n}}. w n inf E R μ ( w n ) = ( 1 − 2 1 ) n G ( Σ 1/2 Ω ) .
The minimax regret dominates every Bayes regret, proving the first inequality in (8) .
We also record a comparison of Gaussian widths, used in Proposition 2.2. For independent standard Gaussian vectors G , G ′ \mathbf{G},\mathbf{G}' G , G ′ , set
Y = Σ 1 / 2 G + ( λ ‾ I d − Σ ) 1 / 2 G ′ . Y=\Sigma^{1/2}\mathbf{G}+(\overline{\lambda}I_d-\Sigma)^{1/2}\mathbf{G}'. Y = Σ 1/2 G + ( λ I d − Σ ) 1/2 G ′ .
Then Y ∼ N ( 0 , λ ‾ I d ) Y\sim\mathcal{N}(0,\overline{\lambda}I_d) Y ∼ N ( 0 , λ I d ) and E [ Y ∣ G ] = Σ 1 / 2 G \mathbb{E}[Y\mid\mathbf{G}]=\Sigma^{1/2}\mathbf{G} E [ Y ∣ G ] = Σ 1/2 G . Convexity of f ∗ f_* f ∗ and conditional Jensen’s inequality give
G ( Σ 1 / 2 Ω ) = E f ∗ ( Σ 1 / 2 G ) ≤ E f ∗ ( Y ) = λ ‾ G ( Ω ) . (64) \mathcal{G}(\Sigma^{1/2}\Omega)=\mathbb{E}f_*(\Sigma^{1/2}\mathbf{G})\leq\mathbb{E}f_*(Y)=\sqrt{\overline{\lambda}}\mathcal{G}(\Omega). \tag*{(64)} G ( Σ 1/2 Ω ) = E f ∗ ( Σ 1/2 G ) ≤ E f ∗ ( Y ) = λ G ( Ω ) . (64)
Equality is attained at Σ = λ ‾ I d \Sigma=\overline{\lambda}I_d Σ = λ I d .
Proof of Proposition 2.2
Proof. Optimality of w ∗ ( x ˉ ) \mathbf{w}_*(\bar{x}) w ∗ ( x ˉ ) at x ˉ \bar{x} x ˉ gives
R μ ( w ∗ ( x ˉ ) ) = μ T ( w ∗ ( μ ) − w ∗ ( x ˉ ) ) ≤ ( x ˉ − μ ) T ( w ∗ ( x ˉ ) − w ∗ ( μ ) ) ≤ ∥ x ˉ − μ ∥ K ∥ w ∗ ( x ˉ ) − w ∗ ( μ ) ∥ Ω . R_\mu(\mathbf{w}_*(\bar{x}))=\mu^\mathsf{T}(\mathbf{w}_*(\mu)-\mathbf{w}_*(\bar{x}))
\leq(\bar{x}-\mu)^\mathsf{T}(\mathbf{w}_*(\bar{x})-\mathbf{w}_*(\mu))
\leq\|\bar{x}-\mu\|_K\|\mathbf{w}_*(\bar{x})-\mathbf{w}_*(\mu)\|_\Omega. R μ ( w ∗ ( x ˉ )) = μ T ( w ∗ ( μ ) − w ∗ ( x ˉ )) ≤ ( x ˉ − μ ) T ( w ∗ ( x ˉ ) − w ∗ ( μ )) ≤ ∥ x ˉ − μ ∥ K ∥ w ∗ ( x ˉ ) − w ∗ ( μ ) ∥ Ω .
On the other hand, (11) at μ \mu μ gives
R μ ( w ∗ ( x ˉ ) ) ≥ c Ω dist ( μ , A ) ∥ w ∗ ( x ˉ ) − w ∗ ( μ ) ∥ Ω 2 . R_\mu(\mathbf{w}_*(\bar{x}))\geq c_\Omega\operatorname{dist}(\mu,A)\|\mathbf{w}_*(\bar{x})-\mathbf{w}_*(\mu)\|_\Omega^2. R μ ( w ∗ ( x ˉ )) ≥ c Ω dist ( μ , A ) ∥ w ∗ ( x ˉ ) − w ∗ ( μ ) ∥ Ω 2 .
Combining the two inequalities yields
R μ ( w ∗ ( x ˉ ) ) ≤ ∥ x ˉ − μ ∥ K 2 c Ω dist ( μ , A ) . R_\mu(\mathbf{w}_*(\bar{x}))\leq\frac{\|\bar{x}-\mu\|_K^2}{c_\Omega\operatorname{dist}(\mu,A)}. R μ ( w ∗ ( x ˉ )) ≤ c Ω dist ( μ , A ) ∥ x ˉ − μ ∥ K 2 .
Since x ˉ − μ = d Σ 1 / 2 G / n \bar{x}-\mu\overset{d}{=}\Sigma^{1/2}\mathbf{G}/\sqrt{n} x ˉ − μ = d Σ 1/2 G / n , taking expectations proves (12) .
Under (10) , ∥ y ∥ K = f ∗ ( y ) \|y\|_K=f_*(y) ∥ y ∥ K = f ∗ ( y ) and hence
E ∥ Σ 1 / 2 G ∥ K = G ( Σ 1 / 2 Ω ) . \mathbb{E}\|\Sigma^{1/2}\mathbf{G}\|_K=\mathcal{G}(\Sigma^{1/2}\Omega). E ∥ Σ 1/2 G ∥ K = G ( Σ 1/2 Ω ) .
Jensen’s inequality gives the first inequality in (13) . Moreover, g ↦ f ∗ ( Σ 1 / 2 g ) g \mapsto f_*(\Sigma^{1/2}g) g ↦ f ∗ ( Σ 1/2 g ) is Lipschitz with squared Lipschitz constant at most sup w ∈ Ω w ⊤ Σ w \sup_{w\in\Omega} w^\top\Sigma w sup w ∈ Ω w ⊤ Σ w . Gaussian Poincare inequality therefore gives
Var ( f ∗ ( Σ 1 / 2 G ) ) ≤ sup w ∈ Ω w ⊤ Σ w . \operatorname{Var}(f_*(\Sigma^{1/2}G)) \le\sup_{w\in\Omega} w^\top\Sigma w. Var ( f ∗ ( Σ 1/2 G )) ≤ w ∈ Ω sup w ⊤ Σ w .
Because Ω = − Ω \Omega=-\Omega Ω = − Ω ,
f ∗ ( Σ 1 / 2 G ) ≥ ∣ w ⊤ Σ 1 / 2 G ∣ f_*(\Sigma^{1/2}G)\geq\lvert w^\top\Sigma^{1/2}G\rvert f ∗ ( Σ 1/2 G ) ≥ ∣ w ⊤ Σ 1/2 G ∣
for every w ∈ Ω w\in\Omega w ∈ Ω . Therefore
G ( Σ 1 / 2 Ω ) ≥ 2 π sup w ∈ Ω w ⊤ Σ w . \mathcal{G}(\Sigma^{1/2}\Omega)\geq\sqrt{\frac{2}{\pi}}\sup_{w\in\Omega}\sqrt{w^\top\Sigma w}. G ( Σ 1/2 Ω ) ≥ π 2 w ∈ Ω sup w ⊤ Σ w .
The variance decomposition now gives
E ∥ Σ 1 / 2 G ∥ K 2 = G ( Σ 1 / 2 Ω ) 2 + Var ( f ∗ ( Σ 1 / 2 G ) ) ≤ ( 1 + π 2 ) G ( Σ 1 / 2 Ω ) 2 . \begin{aligned}
\mathbb{E}\lVert\Sigma^{1/2}G\rVert_K^2
&=\mathcal{G}(\Sigma^{1/2}\Omega)^2+\operatorname{Var}(f_*(\Sigma^{1/2}G))\\
&\leq\left(1+\frac{\pi}{2}\right)\mathcal{G}(\Sigma^{1/2}\Omega)^2.
\end{aligned} E ∥ Σ 1/2 G ∥ K 2 = G ( Σ 1/2 Ω ) 2 + Var ( f ∗ ( Σ 1/2 G )) ≤ ( 1 + 2 π ) G ( Σ 1/2 Ω ) 2 .
which proves (13) . Finally, (A.1) implies the second inequality in (14) .
Strong convexity implies quadratic growth
Suppose Ω = − Ω \Omega=-\Omega Ω = − Ω , 0 ∈ int Ω 0\in\operatorname{int}\Omega 0 ∈ int Ω , and Ω \Omega Ω is c Ω c_\Omega c Ω -strongly convex in its gauge in the midpoint sense. For z ≠ 0 z\neq0 z = 0 , choose v z ∈ Ω v_z\in\Omega v z ∈ Ω satisfying z ⊤ v z = ∥ z ∥ K z^\top v_z=\lVert z\rVert_K z ⊤ v z = ∥ z ∥ K , and apply strong convexity with u = w ∗ ( z ) u=w_*(z) u = w ∗ ( z ) and an arbitrary v = w ∈ Ω v=w\in\Omega v = w ∈ Ω . Taking s = ( c Ω / 2 ) ∥ w − w ∗ ( z ) ∥ Ω 2 v z s=(c_\Omega/2)\lVert w-w_*(z)\rVert_{\Omega}^2v_z s = ( c Ω /2 ) ∥ w − w ∗ ( z ) ∥ Ω 2 v z gives ( w ∗ ( z ) + w ) / 2 + s ∈ Ω (w_*(z)+w)/2+s\in\Omega ( w ∗ ( z ) + w ) /2 + s ∈ Ω . Optimality of w ∗ ( z ) w_*(z) w ∗ ( z ) gives
z ⊤ ( w ∗ ( z ) − w ) ≥ c Ω ∥ z ∥ K ∥ w − w ∗ ( z ) ∥ Ω 2 . z^\top(w_*(z)-w)\geq c_\Omega\lVert z\rVert_K\lVert w-w_*(z)\rVert_{\Omega}^2. z ⊤ ( w ∗ ( z ) − w ) ≥ c Ω ∥ z ∥ K ∥ w − w ∗ ( z ) ∥ Ω 2 .
In particular, every nonzero z z z has a unique optimizer, so A = { 0 } A=\{0\} A = { 0 } and dist ( z , A ) = ∥ z ∥ K \operatorname{dist}(z,A)=\lVert z\rVert_K dist ( z , A ) = ∥ z ∥ K . This proves Remark 2.3.
Decision-Theoretic Tools and Proofs for the Gaussian Experiment
A compact minimax theorem
Lemma B.1 (Minimax identity on compact local sets). Fix n n n , M > 0 M>0 M > 0 , and μ ∈ R d \mu\in\mathbb{R}^d μ ∈ R d . Let Θ \Theta Θ be a nonempty compact subset of B 2 ( M ) × C B_2(M)\times\mathcal{C} B 2 ( M ) × C , and let W W W be a nonempty compact convex subset of Ω \Omega Ω . With samples ( x 1 , … , x n ) (x_1,\ldots,x_n) ( x 1 , … , x n ) drawn from the joint distribution
P t , Σ ( n ) = N ( μ + t n , Σ ) ⊗ n , ( t , Σ ) ∈ Θ , P_{t,\Sigma}^{(n)}=\mathcal{N}\left(\mu+\frac{t}{\sqrt{n}},\Sigma\right)^{\otimes n},\qquad(t,\Sigma)\in\Theta, P t , Σ ( n ) = N ( μ + n t , Σ ) ⊗ n , ( t , Σ ) ∈ Θ ,
denote the loss as
ℓ n ( t , w ) = f ∗ ( μ + t n ) − w ⊤ ( μ + t n ) , w ∈ W . \ell_n(t,w)=f_*\left(\mu+\frac{t}{\sqrt{n}}\right)-w^\top\left(\mu+\frac{t}{\sqrt{n}}\right),\qquad w\in\mathcal{W}. ℓ n ( t , w ) = f ∗ ( μ + n t ) − w ⊤ ( μ + n t ) , w ∈ W .
Then
inf w ^ = w ^ ( x 1 , … , x n ) : ( R d ) n → W w ^ m e a s u r a b l e sup ( t , Σ ) ∈ Θ E μ n , t , Σ ℓ n ( t , w ^ ) = sup Π ∈ P ( Θ ) inf w ^ = w ^ ( x 1 , … , x n ) : ( R d ) n → W w ^ m e a s u r a b l e E Π ℓ n ( T , w ^ ) , (65) \inf_{\substack{\widehat{w}=\widehat{w}(x_1,\ldots,x_n):(\mathbb{R}^d)^n\to\mathcal{W}\\ \widehat{w}\ \mathrm{measurable}}\ } \sup_{(t,\Sigma)\in\Theta}\mathbb{E}_{\mu_{n,t,\Sigma}}\ell_n(t,\widehat{w})
=
\sup_{\Pi\in\mathcal{P}(\Theta)}\inf_{\substack{\widehat{w}=\widehat{w}(x_1,\ldots,x_n):(\mathbb{R}^d)^n\to\mathcal{W}\\ \widehat{w}\ \mathrm{measurable}}\ }\mathbb{E}_{\Pi}\ell_n(T,\widehat{w}), \tag*{(65)} w = w ( x 1 , … , x n ) : ( R d ) n → W w measurable inf ( t , Σ ) ∈ Θ sup E μ n , t , Σ ℓ n ( t , w ) = Π ∈ P ( Θ ) sup w = w ( x 1 , … , x n ) : ( R d ) n → W w measurable inf E Π ℓ n ( T , w ) , (65)
where P ( Θ ) \mathcal{P}(\Theta) P ( Θ ) denotes the Borel probability measures on Θ \Theta Θ , and E Π \mathbb{E}_{\Pi} E Π uses ( T , Σ ) ∼ Π (T,\Sigma)\sim\Pi ( T , Σ ) ∼ Π followed by the sample law P T , Σ ( n ) P_{T,\Sigma}^{(n)} P T , Σ ( n ) .
Moreover, if for every Π ∈ P ( Θ ) \Pi\in\mathcal{P}(\Theta) Π ∈ P ( Θ ) there exists a Bayes rule for decision space Ω \Omega Ω taking values in W \mathcal{W} W , then
inf w ^ = w ^ ( x 1 , … , x n ) : ( R d ) n → W w ^ m e a s u r a b l e sup ( t , Σ ) ∈ Θ E μ n , t , Σ ℓ n ( t , w ^ ) = inf w ^ = w ^ ( x 1 , … , x n ) : ( R d ) n → Ω w ^ m e a s u r a b l e sup ( t , Σ ) ∈ Θ E μ n , t , Σ ℓ n ( t , w ^ ) . \inf_{\substack{\widehat{w}=\widehat{w}(x_1,\ldots,x_n):(\mathbb{R}^d)^n\to\mathcal{W}\\ \widehat{w}\ \mathrm{measurable}}\ }\sup_{(t,\Sigma)\in\Theta}\mathbb{E}_{\mu_{n,t,\Sigma}}\ell_n(t,\widehat{w})
=
\inf_{\substack{\widehat{w}=\widehat{w}(x_1,\ldots,x_n):(\mathbb{R}^d)^n\to\Omega\\ \widehat{w}\ \mathrm{measurable}}\ }\sup_{(t,\Sigma)\in\Theta}\mathbb{E}_{\mu_{n,t,\Sigma}}\ell_n(t,\widehat{w}). w = w ( x 1 , … , x n ) : ( R d ) n → W w measurable inf ( t , Σ ) ∈ Θ sup E μ n , t , Σ ℓ n ( t , w ) = w = w ( x 1 , … , x n ) : ( R d ) n → Ω w measurable inf ( t , Σ ) ∈ Θ sup E μ n , t , Σ ℓ n ( t , w ) .
Proof. Let D n r ( W ) D_n^r(\mathcal{W}) D n r ( W ) be the class of all Markov kernels from ( R d ) n (\mathbb{R}^d)^n ( R d ) n to W \mathcal{W} W , and define
ρ δ ( t , Σ ) = ∫ ( R d ) n ∫ W ℓ n ( t , w ) δ ( d w ∣ x 1 , … , x n ) P t , Σ ( n ) ( d x 1 ⋯ d x n ) . \rho_\delta(t,\Sigma)=\int_{(\mathbb{R}^d)^n}\int_{\mathcal{W}}\ell_n(t,w)\,\delta(dw\mid x_1,\ldots,x_n)P_{t,\Sigma}^{(n)}(dx_1\cdots dx_n). ρ δ ( t , Σ ) = ∫ ( R d ) n ∫ W ℓ n ( t , w ) δ ( d w ∣ x 1 , … , x n ) P t , Σ ( n ) ( d x 1 ⋯ d x n ) .
We apply Liese and Miescke [28 ] with parameter space Θ \Theta Θ , decision space W \mathcal{W} W , and decision class D n r ( W ) D_n^r(\mathcal{W}) D n r ( W ) . Both spaces are compact metric spaces with their Borel σ \sigma σ -algebras. Since Ω \Omega Ω is compact, f ∗ f_* f ∗ is R Ω R_\Omega R Ω -Lipschitz continuous. Thus ( ( t , Σ ) , w ) ↦ ℓ n ( t , w ) ((t,\Sigma),w)\mapsto\ell_n(t,w) (( t , Σ ) , w ) ↦ ℓ n ( t , w ) is jointly continuous, and
0 ≤ ℓ n ( t , w ) ≤ 2 R Ω ( ∥ μ ∥ 2 + M n ) , ( ( t , Σ ) , w ) ∈ Θ × W . 0\leq\ell_n(t,w)\leq2R_\Omega\left(\lVert\mu\rVert_2+\frac{M}{\sqrt{n}}\right),\qquad((t,\Sigma),w)\in\Theta\times\mathcal{W}. 0 ≤ ℓ n ( t , w ) ≤ 2 R Ω ( ∥ μ ∥ 2 + n M ) , (( t , Σ ) , w ) ∈ Θ × W .
The sample space is Euclidean space ( R d ) n (\mathbb{R}^d)^n ( R d ) n with its Borel σ \sigma σ -algebra, and every P t , Σ ( n ) P_{t,\Sigma}^{(n)} P t , Σ ( n ) has a strictly positive Lebesgue density. For ( t , Σ ) , ( s , Γ ) ∈ B 2 ( M ) × C (t,\Sigma),(s,\Gamma)\in B_2(M)\times\mathcal{C} ( t , Σ ) , ( s , Γ ) ∈ B 2 ( M ) × C , the Gaussian KL formula gives
D K L ( P t , Σ ( n ) ∥ P s , Γ ( n ) ) = n 2 [ tr ( Γ − 1 Σ ) − d − log det ( Γ − 1 Σ ) ] + 1 2 ( t − s ) ⊤ Γ − 1 ( t − s ) , D_{\mathrm{KL}}\left(P_{t,\Sigma}^{(n)}\middle\|P_{s,\Gamma}^{(n)}\right)=\frac{n}{2}\left[\operatorname{tr}(\Gamma^{-1}\Sigma)-d-\log\det(\Gamma^{-1}\Sigma)\right]+\frac{1}{2}(t-s)^\top\Gamma^{-1}(t-s), D KL ( P t , Σ ( n ) P s , Γ ( n ) ) = 2 n [ tr ( Γ − 1 Σ ) − d − log det ( Γ − 1 Σ ) ] + 2 1 ( t − s ) ⊤ Γ − 1 ( t − s ) ,
and therefore, by Pinsker’s inequality,
∥ P t , Σ ( n ) − P s , Γ ( n ) ∥ T V ≤ 1 2 D K L ( P t , Σ ( n ) ∥ P s , Γ ( n ) ) → 0 as ( t , Σ ) → ( s , Γ ) . \left\lVert P_{t,\Sigma}^{(n)}-P_{s,\Gamma}^{(n)}\right\rVert_{\mathrm{TV}}\leq\sqrt{\frac{1}{2}D_{\mathrm{KL}}\left(P_{t,\Sigma}^{(n)}\middle\|P_{s,\Gamma}^{(n)}\right)}\to0\quad\text{as }(t,\Sigma)\to(s,\Gamma). P t , Σ ( n ) − P s , Γ ( n ) TV ≤ 2 1 D KL ( P t , Σ ( n ) P s , Γ ( n ) ) → 0 as ( t , Σ ) → ( s , Γ ) .
Here n n n is fixed, and matrix inversion and the log determinant are continuous on C \mathcal{C} C , since λ > 0 \lambda>0 λ > 0 . This verifies condition (A7) of Liese and Miescke [28 ] , also after restriction to Θ \Theta Θ . In particular, ( t , Σ ) ↦ P t , Σ ( n ) ( B ) (t,\Sigma)\mapsto P_{t,\Sigma}^{(n)}(B) ( t , Σ ) ↦ P t , Σ ( n ) ( B ) is Borel measurable for every Borel sample event B B B . For fixed Σ \Sigma Σ , the same calculation reduces to
∥ P t , Σ ( n ) − P s , Σ ( n ) ∥ T V ≤ 1 2 ∥ Σ − 1 / 2 ( t − s ) ∥ 2 ≤ ∥ t − s ∥ 2 2 λ . \left\lVert P_{t,\Sigma}^{(n)}-P_{s,\Sigma}^{(n)}\right\rVert_{\mathrm{TV}}\leq\frac{1}{2}\lVert\Sigma^{-1/2}(t-s)\rVert_2\leq\frac{\lVert t-s\rVert_2}{2\sqrt{\lambda}}. P t , Σ ( n ) − P s , Σ ( n ) TV ≤ 2 1 ∥ Σ − 1/2 ( t − s ) ∥ 2 ≤ 2 λ ∥ t − s ∥ 2 .
The class D n r ( W ) \mathcal{D}^{\mathrm{r}}_{n}(\mathcal{W}) D n r ( W ) is nonempty, since it contains every constant rule in W \mathcal{W} W , and convex: a δ 1 + ( 1 − a ) δ 2 ∈ D n r ( W ) a\delta_1+(1-a)\delta_2\in\mathcal{D}^{\mathrm{r}}_{n}(\mathcal{W}) a δ 1 + ( 1 − a ) δ 2 ∈ D n r ( W ) for a ∈ [ 0 , 1 ] a\in[0,1] a ∈ [ 0 , 1 ] and δ 1 , δ 2 ∈ D n r ( W ) \delta_1,\delta_2\in\mathcal{D}^{\mathrm{r}}_{n}(\mathcal{W}) δ 1 , δ 2 ∈ D n r ( W ) . In the weak convergence of Liese and Miescke [28 ] , δ k ⇒ δ \delta_k\Rightarrow\delta δ k ⇒ δ means
E μ n , t , Σ ∫ W h ( w ) δ k ( d w ∣ x 1 , … , x n ) → E μ n , t , Σ ∫ W h ( w ) δ ( d w ∣ x 1 , … , x n ) \mathbb{E}_{\mu_{n,t,\Sigma}}\int_{\mathcal{W}}h(w)\,\delta_k(\mathrm{d}w\mid x_1,\ldots,x_n)\to\mathbb{E}_{\mu_{n,t,\Sigma}}\int_{\mathcal{W}}h(w)\,\delta(\mathrm{d}w\mid x_1,\ldots,x_n) E μ n , t , Σ ∫ W h ( w ) δ k ( d w ∣ x 1 , … , x n ) → E μ n , t , Σ ∫ W h ( w ) δ ( d w ∣ x 1 , … , x n )
for every ( t , Σ ) ∈ Θ (t,\Sigma)\in\Theta ( t , Σ ) ∈ Θ and every bounded continuous h : W → R h:\mathcal{W}\to\mathbb{R} h : W → R , where δ \delta δ is a Markov kernel into the decision space W \mathcal{W} W . The closure condition in Liese and Miescke [28 ] is relative to the ambient space of Markov kernels into W \mathcal{W} W . Our decision class D n r ( W ) \mathcal{D}^{\mathrm{r}}_{n}(\mathcal{W}) D n r ( W ) is that entire ambient class, so it is automatically weakly closed: whenever a convergent net or sequence has a Markov-kernel limit in the topology of Liese and Miescke [28 ] , that limit again belongs to D n r ( W ) \mathcal{D}^{\mathrm{r}}_{n}(\mathcal{W}) D n r ( W ) . All assumptions of the statistical minimax theorem are therefore satisfied, and it gives
inf δ ∈ D n r ( W ) sup ( t , Σ ) ∈ Θ ρ δ ( t , Σ ) = sup Π ∈ P ( Θ ) inf δ ∈ D n r ( W ) ∫ Θ ρ δ ( t , Σ ) Π ( d t , d Σ ) . (66) \inf_{\delta\in\mathcal{D}^{\mathrm{r}}_{n}(\mathcal{W})}\sup_{(t,\Sigma)\in\Theta}\rho_\delta(t,\Sigma)=\sup_{\Pi\in\mathcal{P}(\Theta)}\inf_{\delta\in\mathcal{D}^{\mathrm{r}}_{n}(\mathcal{W})}\int_{\Theta}\rho_\delta(t,\Sigma)\,\Pi(\mathrm{d}t,\mathrm{d}\Sigma). \tag*{(66)} δ ∈ D n r ( W ) inf ( t , Σ ) ∈ Θ sup ρ δ ( t , Σ ) = Π ∈ P ( Θ ) sup δ ∈ D n r ( W ) inf ∫ Θ ρ δ ( t , Σ ) Π ( d t , d Σ ) . (66)
For δ ∈ D n r ( W ) \delta\in\mathcal{D}^{\mathrm{r}}_{n}(\mathcal{W}) δ ∈ D n r ( W ) , define its conditional barycenter
w ‾ δ ( x 1 , … , x n ) = ∫ W w δ ( d w ∣ x 1 , … , x n ) ∈ W . \overline{w}_\delta(x_1,\ldots,x_n)=\int_{\mathcal{W}}w\,\delta(\mathrm{d}w\mid x_1,\ldots,x_n)\in\mathcal{W}. w δ ( x 1 , … , x n ) = ∫ W w δ ( d w ∣ x 1 , … , x n ) ∈ W .
This is measurable, and compactness and convexity of W \mathcal{W} W give feasibility. Since ℓ n \ell_n ℓ n is affine in w w w ,
ℓ n ( t , w ‾ δ ) = ∫ W ℓ n ( t , w ) δ ( d w ∣ x 1 , … , x n ) . \ell_n(t,\overline{w}_\delta)=\int_{\mathcal{W}}\ell_n(t,w)\,\delta(\mathrm{d}w\mid x_1,\ldots,x_n). ℓ n ( t , w δ ) = ∫ W ℓ n ( t , w ) δ ( d w ∣ x 1 , … , x n ) .
Taking expectation gives, simultaneously for every $(t,\Sigma)\in\Theta,
ρ δ ( t , Σ ) = E μ n , t , Σ ℓ n ( t , w ‾ δ ) . \rho_\delta(t,\Sigma)=\mathbb{E}_{\mu_{n,t,\Sigma}}\ell_n(t,\overline{w}_\delta). ρ δ ( t , Σ ) = E μ n , t , Σ ℓ n ( t , w δ ) .
Thus every randomized policy has exactly the same risk function as a deterministic policy, which gives
inf δ ∈ D n r ( W ) sup ( t , Σ ) ∈ Θ ρ δ ( t , Σ ) = inf δ ∈ D n r ( W ) sup ( t , Σ ) ∈ Θ E μ n , t , Σ ℓ n ( t , w ‾ δ ) ≥ inf w ^ : ( R d ) n → W w ^ m e a s u r a b l e sup ( t , Σ ) ∈ Θ E μ n , t , Σ ℓ n ( t , w ^ ) . (67) \begin{aligned}
\inf_{\delta\in\mathcal{D}^{\mathrm{r}}_{n}(\mathcal{W})}\sup_{(t,\Sigma)\in\Theta}\rho_\delta(t,\Sigma)
&=\inf_{\delta\in\mathcal{D}^{\mathrm{r}}_{n}(\mathcal{W})}\sup_{(t,\Sigma)\in\Theta}\mathbb{E}_{\mu_{n,t,\Sigma}}\ell_n(t,\overline{w}_\delta)\\
&\ge\inf_{\substack{\widehat{w}:(\mathbb{R}^{d})^n\to\mathcal{W}\\ \widehat{w}\ \mathrm{measurable}}}\sup_{(t,\Sigma)\in\Theta}\mathbb{E}_{\mu_{n,t,\Sigma}}\ell_n(t,\widehat{w}). \tag*{(67)}
\end{aligned} δ ∈ D n r ( W ) inf ( t , Σ ) ∈ Θ sup ρ δ ( t , Σ ) = δ ∈ D n r ( W ) inf ( t , Σ ) ∈ Θ sup E μ n , t , Σ ℓ n ( t , w δ ) ≥ w : ( R d ) n → W w measurable inf ( t , Σ ) ∈ Θ sup E μ n , t , Σ ℓ n ( t , w ) . (67)
and for any Π ∈ P ( Θ ) \Pi\in\mathcal{P}(\Theta) Π ∈ P ( Θ ) ,
inf δ ∈ D n r ( W ) ∫ Θ ρ δ ( t , Σ ) Π ( d t , d Σ ) = inf δ ∈ D n r ( W ) ∫ Θ E μ n , t , Σ ℓ n ( t , w ‾ δ ) Π ( d t , d Σ ) ≥ inf w ^ : ( R d ) n → W w ^ m e a s u r a b l e E Π ℓ n ( T , w ^ ) . (68) \begin{aligned}
\inf_{\delta\in\mathcal{D}^{\mathrm{r}}_{n}(\mathcal{W})}\int_{\Theta}\rho_\delta(t,\Sigma)\,\Pi(\mathrm{d}t,\mathrm{d}\Sigma)
&=\inf_{\delta\in\mathcal{D}^{\mathrm{r}}_{n}(\mathcal{W})}\int_{\Theta}\mathbb{E}_{\mu_{n,t,\Sigma}}\ell_n(t,\overline{w}_\delta)\,\Pi(\mathrm{d}t,\mathrm{d}\Sigma)\\
&\ge\inf_{\substack{\widehat{w}:(\mathbb{R}^{d})^n\to\mathcal{W}\\ \widehat{w}\ \mathrm{measurable}}}\mathbb{E}_{\Pi}\ell_n(T,\widehat{w}). \tag*{(68)}
\end{aligned} δ ∈ D n r ( W ) inf ∫ Θ ρ δ ( t , Σ ) Π ( d t , d Σ ) = δ ∈ D n r ( W ) inf ∫ Θ E μ n , t , Σ ℓ n ( t , w δ ) Π ( d t , d Σ ) ≥ w : ( R d ) n → W w measurable inf E Π ℓ n ( T , w ) . (68)
Conversely, for every measurable w ^ : ( R d ) n → W \hat{w} : (\mathbb{R}^{d})^{n} \to\mathcal{W} w ^ : ( R d ) n → W , define
δ w ( B ∣ x 1 , … , x n ) : = I { w ^ ( x 1 , … , x n ) ∈ B } , B ∈ B ( W ) . \delta_{w}(B \mid x_{1},\ldots,x_{n}) := \mathbb{I}\{\hat{w}(x_{1},\ldots,x_{n}) \in B\}, \qquad B \in\mathcal{B}(\mathcal{W}). δ w ( B ∣ x 1 , … , x n ) := I { w ^ ( x 1 , … , x n ) ∈ B } , B ∈ B ( W ) .
For every sample, δ w ( ⋅ ∣ x 1 , … , x n ) \delta_{w}(\cdot\mid x_{1},\ldots,x_{n}) δ w ( ⋅ ∣ x 1 , … , x n ) is the Dirac probability measure at w ^ ( x 1 , … , x n ) \hat{w}(x_{1},\ldots,x_{n}) w ^ ( x 1 , … , x n ) ; for every Borel B B B , its value is measurable in the sample because w ^ \hat{w} w ^ is measurable. Hence δ w ∈ D n r ( W ) \delta_{w} \in\mathcal{D}_{n}^{r}(\mathcal{W}) δ w ∈ D n r ( W ) , and
∫ W ℓ n ( t , w ) δ w ( d w ∣ x 1 , … , x n ) = ℓ n ( t , w ^ ( x 1 , … , x n ) ) . \int_{\mathcal{W}} \ell_{n}(t,w)\,\delta_{w}(\mathrm{d}w \mid x_{1},\ldots,x_{n}) = \ell_{n}(t,\hat{w}(x_{1},\ldots,x_{n})). ∫ W ℓ n ( t , w ) δ w ( d w ∣ x 1 , … , x n ) = ℓ n ( t , w ^ ( x 1 , … , x n )) .
which implies ρ δ w ( t , Σ ) = E μ n , t , Σ ℓ n ( t , w ^ ) \rho_{\delta_{w}}(t,\Sigma) = \mathbb{E}_{\mu_{n,t,\Sigma}}\ell_{n}(t,\hat{w}) ρ δ w ( t , Σ ) = E μ n , t , Σ ℓ n ( t , w ^ ) for ( t , Σ ) ∈ Θ (t,\Sigma) \in\Theta ( t , Σ ) ∈ Θ . Therefore,
inf δ ∈ D n r ( W ) sup ( t , Σ ) ∈ Θ ρ δ ( t , Σ ) ≤ inf δ w ∈ D n r ( W ) sup ( t , Σ ) ∈ Θ ρ δ w ( t , Σ ) = inf w ^ : ( R d ) n → W w ^ m e a s u r a b l e sup ( t , Σ ) ∈ Θ E μ n , t , Σ ℓ n ( t , w ^ ) . (69) \begin{aligned}
\inf_{\delta\in\mathcal{D}_{n}^{r}(\mathcal{W})}\sup_{(t,\Sigma)\in\Theta}\rho_{\delta}(t,\Sigma)
&\leq\inf_{\delta_{w}\in\mathcal{D}_{n}^{r}(\mathcal{W})}\sup_{(t,\Sigma)\in\Theta}\rho_{\delta_{w}}(t,\Sigma) \\
&= \inf_{\substack{\hat{w}:(\mathbb{R}^{d})^{n}\to\mathcal{W}\\ \hat{w}\ \mathrm{measurable}}}\sup_{(t,\Sigma)\in\Theta}\mathbb{E}_{\mu_{n,t,\Sigma}}\ell_{n}(t,\hat{w}). \tag*{(69)}
\end{aligned} δ ∈ D n r ( W ) inf ( t , Σ ) ∈ Θ sup ρ δ ( t , Σ ) ≤ δ w ∈ D n r ( W ) inf ( t , Σ ) ∈ Θ sup ρ δ w ( t , Σ ) = w ^ : ( R d ) n → W w ^ measurable inf ( t , Σ ) ∈ Θ sup E μ n , t , Σ ℓ n ( t , w ^ ) . (69)
and for any Π ∈ P ( Θ ) \Pi\in\mathcal{P}(\Theta) Π ∈ P ( Θ ) ,
\begin{aligned}
\inf_{\delta\in\mathcal{D}_{n}^{r}(\mathcal{W})}\int_{\Theta}\rho_{\delta}(t,\Sigma)\,\Pi(\mathrm{d}t,\mathrm{d}\Sigma)
&\leq\inf_{\delta_{w}\in\mathcal{D}_{n}^{r}(\mathcal{W})}\int_{\Theta}\rho_{\delta_{w}}(t,\Sigma)\,\Pi(\mathrm{d}t,\mathrm{d}\Sigma) \\
&= \inf_{\substack{\hat{w}:(\mathbb{R}^{d})^{n}\to\mathcal{W}\\ \hat{w}\ \mathrm{measurable}}\mathbb{E}_{\Pi}\ell_{n}(T,\hat{w}).
\end{aligned}} \tag*{(70)}\end{aligned}
Combining (B.3) and (B.5),
inf δ ∈ D n r ( W ) sup ( t , Σ ) ∈ Θ ρ δ ( t , Σ ) = inf w ^ : ( R d ) n → W w ^ m e a s u r a b l e sup ( t , Σ ) ∈ Θ E μ n , t , Σ ℓ n ( t , w ^ ) . \inf_{\delta\in\mathcal{D}_{n}^{r}(\mathcal{W})}\sup_{(t,\Sigma)\in\Theta}\rho_{\delta}(t,\Sigma)
=
\inf_{\substack{\hat{w}:(\mathbb{R}^{d})^{n}\to\mathcal{W}\\ \hat{w}\ \mathrm{measurable}}\ }\sup_{(t,\Sigma)\in\Theta}\mathbb{E}_{\mu_{n,t,\Sigma}}\ell_{n}(t,\hat{w}). δ ∈ D n r ( W ) inf ( t , Σ ) ∈ Θ sup ρ δ ( t , Σ ) = w ^ : ( R d ) n → W w ^ measurable inf ( t , Σ ) ∈ Θ sup E μ n , t , Σ ℓ n ( t , w ^ ) .
Similarly, for any Π ∈ P ( Θ ) \Pi\in\mathcal{P}(\Theta) Π ∈ P ( Θ ) , (B.4) and (B.6) give
inf δ ∈ D n r ( W ) ∫ Θ ρ δ ( t , Σ ) Π ( d t , d Σ ) = inf w ^ : ( R d ) n → W w ^ m e a s u r a b l e E Π ℓ n ( T , w ^ ) . \inf_{\delta\in\mathcal{D}_{n}^{r}(\mathcal{W})}\int_{\Theta}\rho_{\delta}(t,\Sigma)\,\Pi(\mathrm{d}t,\mathrm{d}\Sigma)
=
\inf_{\substack{\hat{w}:(\mathbb{R}^{d})^{n}\to\mathcal{W}\\ \hat{w}\ \mathrm{measurable}}\ }\mathbb{E}_{\Pi}\ell_{n}(T,\hat{w}). δ ∈ D n r ( W ) inf ∫ Θ ρ δ ( t , Σ ) Π ( d t , d Σ ) = w ^ : ( R d ) n → W w ^ measurable inf E Π ℓ n ( T , w ^ ) .
Combining (B.2) with the preceding two equations proves (B.1). The barycenter step is the affine-loss equality case of Liese and Miescke [28 ] , equation (3.5).
Finally, under the Bayes-rule assumption in the last assertion, W ⊆ Ω \mathcal{W} \subseteq\Omega W ⊆ Ω and Bayes optimality imply, for every Π ∈ P ( Θ ) \Pi\in\mathcal{P}(\Theta) Π ∈ P ( Θ ) ,
inf δ ∈ D n r ( W ) ∫ Θ ρ δ ( t , Σ ) Π ( d t , d Σ ) = inf δ ∈ D n r ( Ω ) ∫ Θ ρ δ ( t , Σ ) Π ( d t , d Σ ) . \inf_{\delta\in\mathcal{D}_{n}^{r}(\mathcal{W})}\int_{\Theta}\rho_{\delta}(t,\Sigma)\,\Pi(\mathrm{d}t,\mathrm{d}\Sigma)
=
\inf_{\delta\in\mathcal{D}_{n}^{r}(\Omega)}\int_{\Theta}\rho_{\delta}(t,\Sigma)\,\Pi(\mathrm{d}t,\mathrm{d}\Sigma). δ ∈ D n r ( W ) inf ∫ Θ ρ δ ( t , Σ ) Π ( d t , d Σ ) = δ ∈ D n r ( Ω ) inf ∫ Θ ρ δ ( t , Σ ) Π ( d t , d Σ ) .
Taking the supremum over Π \Pi Π and applying (B.1) to both decision spaces proves the assertion.
Proof of Lemma 3.1
Proof. Conditional on any value of T T T , Z ∼ N ( 0 , Σ ) Z \sim\mathcal{N}(0,\Sigma) Z ∼ N ( 0 , Σ ) . This conditional distribution does not depend on T T T , so Z ⊥ T Z \perp T Z ⊥ T under the joint law. Since x ˉ \bar{x} x ˉ is a sufficient statistic, Y Y Y contains all sample information about T T T relevant to the Bayes decision.
Fix π \pi π . Conditional on Y Y Y , the posterior expected regret of a policy w ∈ Ω w \in\Omega w ∈ Ω is
E π , Σ [ f ∗ ( μ n , T ) − w T μ n , T ∣ Y ] = E π , Σ [ f ∗ ( μ n , T ) ∣ Y ] − w T ( μ + m π , Σ ( Y ) n ) . \mathbb{E}_{\pi,\Sigma}\left[f_*(\mu_{n,T})-w^{\mathsf{T}}\mu_{n,T}\mid Y\right]
=
\mathbb{E}_{\pi,\Sigma}\left[f_*(\mu_{n,T})\mid Y\right]
-
w^{\mathsf{T}}\left(\mu+\frac{m_{\pi,\Sigma}(Y)}{\sqrt{n}}\right). E π , Σ [ f ∗ ( μ n , T ) − w T μ n , T ∣ Y ] = E π , Σ [ f ∗ ( μ n , T ) ∣ Y ] − w T ( μ + n m π , Σ ( Y ) ) .
The first term does not depend on w w w . Maximizing the second term over Ω \Omega Ω proves (21) . Substituting that policy and taking expectation gives the minimum Bayes risk
E π , Σ [ f ∗ ( μ + T n ) − f ∗ ( μ + m π , Σ ( Y ) n ) ] . (71) \mathbb{E}_{\pi,\Sigma}\left[f_*\left(\mu+\frac{T}{\sqrt{n}}\right)-f_*\left(\mu+\frac{m_{\pi,\Sigma}(Y)}{\sqrt{n}}\right)\right]. \tag*{(71)} E π , Σ [ f ∗ ( μ + n T ) − f ∗ ( μ + n m π , Σ ( Y ) ) ] . (71)
By the tower property, E π , Σ m π , Σ ( Y ) = E π , Σ T \mathbb{E}_{\pi,\Sigma}m_{\pi,\Sigma}(Y)=\mathbb{E}_{\pi,\Sigma}T E π , Σ m π , Σ ( Y ) = E π , Σ T . Hence the constant and affine terms in (18) cancel in (B.7). This proves the second equality in (20) .
It remains to prove the first equality. Apply Lemma B.1 with
Θ = B 2 ( M ) × { Σ } , W = Ω . \Theta=B_2(M)\times\{\Sigma\},\qquad\mathcal{W}=\Omega. Θ = B 2 ( M ) × { Σ } , W = Ω.
These are nonempty compact sets, and Ω \Omega Ω is convex. Retain D n r ( W ) D_n^{r}(\mathcal{W}) D n r ( W ) and ρ δ ( t , Σ ) \rho_\delta(t,\Sigma) ρ δ ( t , Σ ) from the proof of Lemma B.1. Identifying a prior on Θ \Theta Θ with its marginal π \pi π on B 2 ( M ) B_2(M) B 2 ( M ) gives
R n , M Σ ( μ ) = sup π ∈ P ( B 2 ( M ) ) inf δ ∈ D n r ( Ω ) ∫ B 2 ( M ) ρ δ ( t , Σ ) π ( d t ) = sup π ∈ P ( B 2 ( M ) ) inf w ^ E π , Σ ℓ n ( T , w ^ ) . (72) R_{n,M}^{\Sigma}(\mu)
=
\sup_{\pi\in\mathcal{P}(B_2(M))}
\inf_{\delta\in D_n^{r}(\Omega)}
\int_{B_2(M)}\rho_\delta(t,\Sigma)\,\pi(dt)
=
\sup_{\pi\in\mathcal{P}(B_2(M))}
\inf_{\widehat{w}}
\mathbb{E}_{\pi,\Sigma}\ell_n(T,\widehat{w}). \tag*{(72)} R n , M Σ ( μ ) = π ∈ P ( B 2 ( M )) sup δ ∈ D n r ( Ω ) inf ∫ B 2 ( M ) ρ δ ( t , Σ ) π ( d t ) = π ∈ P ( B 2 ( M )) sup w inf E π , Σ ℓ n ( T , w ) . (72)
The last infimum is over the deterministic sample-only policies in (16) ; its value is the Bayes risk computed in (B.7). This proves the first equality in (20) .
A restricted decision set
Lemma B.2 (Restricted decision set). Fix n n n , L > 0 L>0 L > 0 , and μ ∈ R d \mu\in\mathbb{R}^d μ ∈ R d , and let b L ( ε ) : = sup t ∈ B 2 ( L ) r μ ( ε t ) b_L(\varepsilon):=\sup_{t\in B_2(L)}r_\mu(\varepsilon t) b L ( ε ) := sup t ∈ B 2 ( L ) r μ ( εt ) . Define
Ω n , L : = conv ( ⋃ u ∈ B 2 ( L ) arg max w ∈ Ω ( μ + u / n ) T w ) . \Omega_{n,L}:=\operatorname{conv}\left(\bigcup_{u\in B_2(L)}\operatorname*{arg\,max}_{w\in\Omega}(\mu+u/\sqrt{n})^{\mathsf{T}}w\right). Ω n , L := conv u ∈ B 2 ( L ) ⋃ w ∈ Ω arg max ( μ + u / n ) T w . Then Ω n , L \Omega_{n,L} Ω n , L is a nonempty compact convex subset of Ω \Omega Ω , and
0 ≤ ℓ n ( t , w ) ≤ 2 b 3 L ( n − 1 / 2 ) , t ∈ B 2 ( L ) , w ∈ Ω n , L , (73) 0\leq\ell_n(t,w)\leq2b_{3L}(n^{-1/2}),\qquad t\in B_2(L),\ w\in\Omega_{n,L}, \tag*{(73)} 0 ≤ ℓ n ( t , w ) ≤ 2 b 3 L ( n − 1/2 ) , t ∈ B 2 ( L ) , w ∈ Ω n , L , (73) where ℓ n ( t , w ) = f ∗ ( μ + t / n ) − w T ( μ + t / n ) \ell_n(t,w)=f_*(\mu+t/\sqrt{n})-w^{\mathsf{T}}(\mu+t/\sqrt{n}) ℓ n ( t , w ) = f ∗ ( μ + t / n ) − w T ( μ + t / n ) . Moreover, for every prior π ∈ P ( B 2 ( M ) ) \pi\in\mathcal{P}(B_2(M)) π ∈ P ( B 2 ( M )) with M ≤ L M\leq L M ≤ L and every Σ \Sigma Σ , the Bayes rule (21) takes values in Ω n , L \Omega_{n,L} Ω n , L .
Proof. The graph of the local maximizers is
{ ( u , w ) ∈ B 2 ( L ) × Ω : ( μ + u / n ) T w = f ∗ ( μ + u / n ) } . \{(u,w)\in B_2(L)\times\Omega:(\mu+u/\sqrt{n})^{\mathsf T}w=f_*(\mu+u/\sqrt{n})\}. {( u , w ) ∈ B 2 ( L ) × Ω : ( μ + u / n ) T w = f ∗ ( μ + u / n )} .
Continuity of ( μ + u / n ) T w (\mu+u/\sqrt{n})^{\mathsf T}w ( μ + u / n ) T w and f ∗ f_* f ∗ makes this graph closed in the compact space B 2 ( L ) × Ω B_2(L)\times\Omega B 2 ( L ) × Ω , hence compact. Its image under the continuous coordinate projection ( u , w ) ↦ w (u,w)\mapsto w ( u , w ) ↦ w is
⋃ u ∈ B 2 ( L ) a r g m a x w ∈ Ω ( μ + u / n ) T w , \bigcup_{u\in B_2(L)}\mathop{\rm arg\,max}_{w\in\Omega}(\mu+u/\sqrt{n})^{\mathsf T}w, u ∈ B 2 ( L ) ⋃ arg max w ∈ Ω ( μ + u / n ) T w ,
which is therefore compact and nonempty. In finite dimensions, its convex hull is compact: by Carathéodory’s theorem, every point of the convex hull is a convex combination of at most d + 1 d+1 d + 1 points, and these combinations form the continuous image of a compact product with the closed probability simplex. Since Ω \Omega Ω is convex, Ω n , L ⊆ Ω \Omega_{n,L}\subseteq\Omega Ω n , L ⊆ Ω . Hence Ω n , L \Omega_{n,L} Ω n , L is a nonempty compact convex set.
For t , u ∈ B 2 ( L ) t,u\in B_2(L) t , u ∈ B 2 ( L ) and w ∈ a r g m a x Ω ( μ + u / n ) T w w\in\mathop{\rm arg\,max}_{\Omega}(\mu+u/\sqrt{n})^{\mathsf T}w w ∈ arg max Ω ( μ + u / n ) T w , the subgradient inequality for r μ r_\mu r μ gives
ℓ n ( t , w ) = r μ ( t / n ) − r μ ( u / n ) + ( w − w ∗ ( μ ) ) T ( u − t ) / n ≤ r μ ( t / n ) + r μ ( ( 2 u − t ) / n ) − 2 r μ ( u / n ) ≤ 2 b 3 L ( n − 1 / 2 ) . \begin{aligned}
\ell_n(t,w)=r_\mu(t/\sqrt{n})-r_\mu(u/\sqrt{n})+(w-w_*(\mu))^{\mathsf T}(u-t)/\sqrt{n} \\
&\le r_\mu(t/\sqrt{n})+r_\mu((2u-t)/\sqrt{n})-2r_\mu(u/\sqrt{n})\le2b_{3L}(n^{-1/2}).
\end{aligned} ℓ n ( t , w ) = r μ ( t / n ) − r μ ( u / n ) + ( w − w ∗ ( μ ) ) T ( u − t ) / n ≤ r μ ( t / n ) + r μ (( 2 u − t ) / n ) − 2 r μ ( u / n ) ≤ 2 b 3 L ( n − 1/2 ) .
Since ℓ n ( t , ⋅ ) \ell_n(t,\cdot) ℓ n ( t , ⋅ ) is affine, the same bound holds on the convex hull Ω n , L \Omega_{n,L} Ω n , L , and ℓ n ≥ 0 \ell_n\geq0 ℓ n ≥ 0 on Ω \Omega Ω . Finally, for π ∈ P ( B 2 ( M ) ) \pi\in\mathcal{P}(B_2(M)) π ∈ P ( B 2 ( M )) , conditional Jensen’s inequality gives ∥ m π , Σ ( Y ) ∥ 2 ≤ E π , Σ [ ∥ T ∥ 2 ∣ Y ] ≤ M ≤ L \|m_{\pi,\Sigma}(Y)\|_2\leq\mathbb{E}_{\pi,\Sigma}[\|T\|_2\mid Y]\leq M\leq L ∥ m π , Σ ( Y ) ∥ 2 ≤ E π , Σ [ ∥ T ∥ 2 ∣ Y ] ≤ M ≤ L , so w ∗ ( μ + m π , Σ ( Y ) / n ) ∈ a r g m a x Ω ( μ + u / n ) T w w_*(\mu+m_{\pi,\Sigma}(Y)/\sqrt{n})\in\mathop{\rm arg\,max}_{\Omega}(\mu+u/\sqrt{n})^{\mathsf T}w w ∗ ( μ + m π , Σ ( Y ) / n ) ∈ arg max Ω ( μ + u / n ) T w with u = m π , Σ ( Y ) ∈ B 2 ( L ) u=m_{\pi,\Sigma}(Y)\in B_2(L) u = m π , Σ ( Y ) ∈ B 2 ( L ) .
Proof of Theorem 3.2
Proof. Convexity gives H μ ⪰ 0 H_\mu\succeq0 H μ ⪰ 0 . As argued before, w ∗ ( μ ) = ∇ f ∗ ( μ ) w_*(\mu)=\nabla f_*(\mu) w ∗ ( μ ) = ∇ f ∗ ( μ ) , and Taylor expansion yields
ϵ n , M : = sup t ∈ B 2 ( M ) ∣ n r μ ( t / n ) − 1 2 t T H μ t ∣ = sup t ∈ B 2 ( M ) ∣ n f ∗ ( μ + t / n ) − n f ∗ ( μ ) − n w ∗ ( μ ) T t − t T H μ t / 2 ∣ ≤ M 2 2 sup ∥ h ∥ 2 ≤ M / n ∥ ∇ 2 f ∗ ( μ + h ) − H μ ∥ o p → 0 as n → ∞ . \begin{aligned}
\epsilon_{n,M}:=\sup_{t\in B_2(M)}\left|nr_\mu(t/\sqrt{n})-\frac{1}{2}t^{\mathsf T}H_\mu t\right| \\
&=\sup_{t\in B_2(M)}\left|nf_*(\mu+t/\sqrt{n})-nf_*(\mu)-\sqrt{n}w_*(\mu)^{\mathsf T}t-t^{\mathsf T}H_\mu t/2\right| \\
&\leq\frac{M^2}{2}\sup_{\|h\|_2\leq M/\sqrt{n}}\|\nabla^2f_*(\mu+h)-H_\mu\|_{\mathrm{op}}\to0\qquad\text{as }n\to\infty.
\end{aligned} ϵ n , M := t ∈ B 2 ( M ) sup n r μ ( t / n ) − 2 1 t T H μ t = t ∈ B 2 ( M ) sup n f ∗ ( μ + t / n ) − n f ∗ ( μ ) − n w ∗ ( μ ) T t − t T H μ t /2 ≤ 2 M 2 ∥ h ∥ 2 ≤ M / n sup ∥ ∇ 2 f ∗ ( μ + h ) − H μ ∥ op → 0 as n → ∞.
Both T T T and m π , Σ ( Y ) m_{\pi,\Sigma}(Y) m π , Σ ( Y ) lie in B 2 ( M ) B_2(M) B 2 ( M ) , which yields ∣ n r μ ( T / n ) − 1 2 T T H μ T ∣ ≤ ϵ n , M \left|nr_\mu(T/\sqrt{n})-\frac{1}{2}T^{\mathsf T}H_\mu T\right|\leq\epsilon_{n,M} n r μ ( T / n ) − 2 1 T T H μ T ≤ ϵ n , M and ∣ n r μ ( m π , Σ ( Y ) / n ) − 1 2 m π , Σ ( Y ) T H μ m π , Σ ( Y ) ∣ ≤ ϵ n , M \left|nr_\mu(m_{\pi,\Sigma}(Y)/\sqrt{n})-\frac{1}{2}m_{\pi,\Sigma}(Y)^{\mathsf T}H_\mu m_{\pi,\Sigma}(Y)\right|\leq\epsilon_{n,M} n r μ ( m π , Σ ( Y ) / n ) − 2 1 m π , Σ ( Y ) T H μ m π , Σ ( Y ) ≤ ϵ n , M . Therefore, by taking expectation,
∣ n E π , Σ [ r μ ( T n ) − r μ ( m π , Σ ( Y ) n ) ] − 1 2 E π , Σ [ T T H μ T − m π , Σ ( Y ) T H μ m π , Σ ( Y ) ] ∣ \left|n\mathbb{E}_{\pi,\Sigma}\left[r_\mu\left(\frac{T}{\sqrt{n}}\right)-r_\mu\left(\frac{m_{\pi,\Sigma}(Y)}{\sqrt{n}}\right)\right]-\frac{1}{2}\mathbb{E}_{\pi,\Sigma}\left[T^{\mathsf T}H_\mu T-m_{\pi,\Sigma}(Y)^{\mathsf T}H_\mu m_{\pi,\Sigma}(Y)\right]\right| n E π , Σ [ r μ ( n T ) − r μ ( n m π , Σ ( Y ) ) ] − 2 1 E π , Σ [ T T H μ T − m π , Σ ( Y ) T H μ m π , Σ ( Y ) ]
≤ 2 ϵ n , M → 0 as n → ∞ . \leq2\epsilon_{n,M}\to0\qquad\text{as }n\to\infty. ≤ 2 ϵ n , M → 0 as n → ∞.
Furthermore, by taking conditional expectation, it is noteworthy that
E π , Σ [ T ⊤ H μ T − m π , Σ ( Y ) ⊤ H μ m π , Σ ( Y ) ] = E π , Σ [ ( T − m π , Σ ( Y ) ) ⊤ H μ ( T − m π , Σ ( Y ) ) ] . \mathbb{E}_{\pi,\Sigma}\left[\boldsymbol{T}^{\top}H_{\mu}\boldsymbol{T}-m_{\pi,\Sigma}(\boldsymbol{Y})^{\top}H_{\mu}m_{\pi,\Sigma}(\boldsymbol{Y})\right]=\mathbb{E}_{\pi,\Sigma}\left[(\boldsymbol{T}-m_{\pi,\Sigma}(\boldsymbol{Y}))^{\top}H_{\mu}(\boldsymbol{T}-m_{\pi,\Sigma}(\boldsymbol{Y}))\right]. E π , Σ [ T ⊤ H μ T − m π , Σ ( Y ) ⊤ H μ m π , Σ ( Y ) ] = E π , Σ [ ( T − m π , Σ ( Y ) ) ⊤ H μ ( T − m π , Σ ( Y )) ] .
Combining with equation (20) proves (22) .
We next prove lim M → ∞ B M , Σ ( H μ ) = tr ( Σ H μ ) \lim_{M\to\infty}B_{M,\Sigma}(H_{\mu})=\operatorname{tr}(\Sigma H_{\mu}) lim M → ∞ B M , Σ ( H μ ) = tr ( Σ H μ ) . Note that Y = T + Z \boldsymbol{Y}=\boldsymbol{T}+\boldsymbol{Z} Y = T + Z , and thus
B M , Σ ( H μ ) = sup π ∈ P ( B 2 ( M ) ) E π , Σ [ ( T − m π , Σ ( Y ) ) ⊤ H μ ( T − m π , Σ ( Y ) ) ] = sup π ∈ P ( B 2 ( M ) ) { E π , Σ [ ( T − Y ) ⊤ H μ ( T − Y ) ] − E π , Σ [ ( m π , Σ ( Y ) − Y ) ⊤ H μ ( m π , Σ ( Y ) − Y ) ] } ≤ E π , Σ [ Z ⊤ H μ Z ] = tr ( Σ H μ ) , (74) \begin{aligned}
B_{M,\Sigma}(H_{\mu})&=\sup_{\pi\in\mathcal{P}(B_{2}(M))}\mathbb{E}_{\pi,\Sigma}\left[(\boldsymbol{T}-m_{\pi,\Sigma}(\boldsymbol{Y}))^{\top}H_{\mu}(\boldsymbol{T}-m_{\pi,\Sigma}(\boldsymbol{Y}))\right]\\
&=\sup_{\pi\in\mathcal{P}(B_{2}(M))}\left\{\mathbb{E}_{\pi,\Sigma}\left[(\boldsymbol{T}-\boldsymbol{Y})^{\top}H_{\mu}(\boldsymbol{T}-\boldsymbol{Y})\right]\right.\\
&\qquad\left.-\mathbb{E}_{\pi,\Sigma}\left[(m_{\pi,\Sigma}(\boldsymbol{Y})-\boldsymbol{Y})^{\top}H_{\mu}(m_{\pi,\Sigma}(\boldsymbol{Y})-\boldsymbol{Y})\right]\right\}\\
&\leq\mathbb{E}_{\pi,\Sigma}\left[\boldsymbol{Z}^{\top}H_{\mu}\boldsymbol{Z}\right]=\operatorname{tr}(\Sigma H_{\mu}), \tag*{(74)}
\end{aligned} B M , Σ ( H μ ) = π ∈ P ( B 2 ( M )) sup E π , Σ [ ( T − m π , Σ ( Y ) ) ⊤ H μ ( T − m π , Σ ( Y )) ] = π ∈ P ( B 2 ( M )) sup { E π , Σ [ ( T − Y ) ⊤ H μ ( T − Y ) ] − E π , Σ [ ( m π , Σ ( Y ) − Y ) ⊤ H μ ( m π , Σ ( Y ) − Y ) ] } ≤ E π , Σ [ Z ⊤ H μ Z ] = tr ( Σ H μ ) , (74)
The first equality follows because the cross term vanishes conditionally on Y \boldsymbol{Y} Y :
E π , Σ [ ( T − m π , Σ ( Y ) ) ⊤ H μ ( m π , Σ ( Y ) − Y ) ] = E π , Σ [ E π , Σ [ T − m π , Σ ( Y ) ∣ Y ] ⊤ H μ ( m π , Σ ( Y ) − Y ) ] = 0. \begin{aligned}
\mathbb{E}_{\pi,\Sigma}\left[(\boldsymbol{T}-m_{\pi,\Sigma}(\boldsymbol{Y}))^{\top}H_{\mu}(m_{\pi,\Sigma}(\boldsymbol{Y})-\boldsymbol{Y})\right]
&=\mathbb{E}_{\pi,\Sigma}\left[\mathbb{E}_{\pi,\Sigma}[\boldsymbol{T}-m_{\pi,\Sigma}(\boldsymbol{Y})\mid\boldsymbol{Y}]^{\top}H_{\mu}(m_{\pi,\Sigma}(\boldsymbol{Y})-\boldsymbol{Y})\right]\\
&=0.
\end{aligned} E π , Σ [ ( T − m π , Σ ( Y ) ) ⊤ H μ ( m π , Σ ( Y ) − Y ) ] = E π , Σ [ E π , Σ [ T − m π , Σ ( Y ) ∣ Y ] ⊤ H μ ( m π , Σ ( Y ) − Y ) ] = 0.
For the reverse inequality, first consider the untruncated prior T ∼ π τ = N ( 0 , τ 2 Σ ) \boldsymbol{T}\sim\pi_{\tau}=\mathcal{N}(0,\tau^{2}\Sigma) T ∼ π τ = N ( 0 , τ 2 Σ ) . The posterior distribution is T ∣ Y ∼ N ( τ 2 ( 1 + τ 2 ) − 1 Y , τ 2 ( 1 + τ 2 ) − 1 Σ ) \boldsymbol{T}\mid\boldsymbol{Y}\sim\mathcal{N}(\tau^{2}(1+\tau^{2})^{-1}\boldsymbol{Y},\tau^{2}(1+\tau^{2})^{-1}\Sigma) T ∣ Y ∼ N ( τ 2 ( 1 + τ 2 ) − 1 Y , τ 2 ( 1 + τ 2 ) − 1 Σ ) , so
B ( π τ ) : = E π τ , Σ [ ( T − m π τ , Σ ( Y ) ) ⊤ H μ ( T − m π τ , Σ ( Y ) ) ] = E π τ , Σ [ E [ ( T − m π τ , Σ ( Y ) ) ⊤ H μ ( T − m π τ , Σ ( Y ) ) ∣ Y ] ] = E π τ , Σ [ tr ( H μ Cov ( T ∣ Y ) ) ] = τ 2 1 + τ 2 tr ( Σ H μ ) . \begin{aligned}
B(\pi_{\tau})&:=\mathbb{E}_{\pi_{\tau},\Sigma}\left[(\boldsymbol{T}-m_{\pi_{\tau},\Sigma}(\boldsymbol{Y}))^{\top}H_{\mu}(\boldsymbol{T}-m_{\pi_{\tau},\Sigma}(\boldsymbol{Y}))\right]\\
&=\mathbb{E}_{\pi_{\tau},\Sigma}\left[\mathbb{E}\left[(\boldsymbol{T}-m_{\pi_{\tau},\Sigma}(\boldsymbol{Y}))^{\top}H_{\mu}(\boldsymbol{T}-m_{\pi_{\tau},\Sigma}(\boldsymbol{Y}))\mid\boldsymbol{Y}\right]\right]\\
&=\mathbb{E}_{\pi_{\tau},\Sigma}\left[\operatorname{tr}(H_{\mu}\operatorname{Cov}(\boldsymbol{T}\mid\boldsymbol{Y}))\right]\\
&=\frac{\tau^{2}}{1+\tau^{2}}\operatorname{tr}(\Sigma H_{\mu}).
\end{aligned} B ( π τ ) := E π τ , Σ [ ( T − m π τ , Σ ( Y ) ) ⊤ H μ ( T − m π τ , Σ ( Y )) ] = E π τ , Σ [ E [ ( T − m π τ , Σ ( Y ) ) ⊤ H μ ( T − m π τ , Σ ( Y )) ∣ Y ] ] = E π τ , Σ [ tr ( H μ Cov ( T ∣ Y )) ] = 1 + τ 2 τ 2 tr ( Σ H μ ) .
This prior is not supported on B 2 ( M ) B_{2}(M) B 2 ( M ) , so it cannot be used directly for the local lower bound. Let π τ , M \pi_{\tau,M} π τ , M be the conditional law of π τ = N ( 0 , τ 2 Σ ) \pi_{\tau}=\mathcal{N}(0,\tau^{2}\Sigma) π τ = N ( 0 , τ 2 Σ ) given ∥ T ∥ 2 ≤ M \lVert\boldsymbol{T}\rVert_{2}\leq M ∥ T ∥ 2 ≤ M , p τ , M = P π τ { ∥ T ∥ 2 ≤ M } p_{\tau,M}=\mathbb{P}_{\pi_{\tau}}\{\lVert\boldsymbol{T}\rVert_{2}\leq M\} p τ , M = P π τ {∥ T ∥ 2 ≤ M } , and write B ( π τ , M ) : = E π τ , M , Σ [ ( T − m π τ , M , Σ ( Y ) ) ⊤ H μ ( T − m π τ , M , Σ ( Y ) ) ] B(\pi_{\tau,M}):=\mathbb{E}_{\pi_{\tau,M},\Sigma}\left[(\boldsymbol{T}-m_{\pi_{\tau,M},\Sigma}(\boldsymbol{Y}))^{\top}H_{\mu}(\boldsymbol{T}-m_{\pi_{\tau,M},\Sigma}(\boldsymbol{Y}))\right] B ( π τ , M ) := E π τ , M , Σ [ ( T − m π τ , M , Σ ( Y ) ) ⊤ H μ ( T − m π τ , M , Σ ( Y )) ] . By the minimality of B B B , we have
B ( π τ ) ≤ E π τ , Σ [ ( T − m π τ , M , Σ ( Y ) ) ⊤ H μ ( T − m π τ , M , Σ ( Y ) ) ] = p τ , M E π τ , Σ [ ( T − m π τ , M , Σ ( Y ) ) ⊤ H μ ( T − m π τ , M , Σ ( Y ) ) ∣ ∥ T ∥ 2 ≤ M ] + E π τ , Σ [ ( T − m π τ , M , Σ ( Y ) ) ⊤ H μ ( T − m π τ , M , Σ ( Y ) ) I { ∥ T ∥ 2 > M } ] . \begin{aligned}
B(\pi_{\tau})&\leq\mathbb{E}_{\pi_{\tau},\Sigma}\left[(\boldsymbol{T}-m_{\pi_{\tau,M},\Sigma}(\boldsymbol{Y}))^{\top}H_{\mu}(\boldsymbol{T}-m_{\pi_{\tau,M},\Sigma}(\boldsymbol{Y}))\right]\\
&=p_{\tau,M}\mathbb{E}_{\pi_{\tau},\Sigma}\left[(\boldsymbol{T}-m_{\pi_{\tau,M},\Sigma}(\boldsymbol{Y}))^{\top}H_{\mu}(\boldsymbol{T}-m_{\pi_{\tau,M},\Sigma}(\boldsymbol{Y}))\mid\lVert\boldsymbol{T}\rVert_{2}\leq M\right]\\
&\quad+\mathbb{E}_{\pi_{\tau},\Sigma}\left[(\boldsymbol{T}-m_{\pi_{\tau,M},\Sigma}(\boldsymbol{Y}))^{\top}H_{\mu}(\boldsymbol{T}-m_{\pi_{\tau,M},\Sigma}(\boldsymbol{Y}))\mathbb{I}\{\lVert\boldsymbol{T}\rVert_{2}>M\}\right].
\end{aligned} B ( π τ ) ≤ E π τ , Σ [ ( T − m π τ , M , Σ ( Y ) ) ⊤ H μ ( T − m π τ , M , Σ ( Y )) ] = p τ , M E π τ , Σ [ ( T − m π τ , M , Σ ( Y ) ) ⊤ H μ ( T − m π τ , M , Σ ( Y )) ∣ ∥ T ∥ 2 ≤ M ] + E π τ , Σ [ ( T − m π τ , M , Σ ( Y ) ) ⊤ H μ ( T − m π τ , M , Σ ( Y )) I {∥ T ∥ 2 > M } ] .
Here, E π τ , Σ [ ( T − m π τ , M , Σ ( Y ) ) ⊤ H μ ( T − m π τ , M , Σ ( Y ) ) ∣ ∥ T ∥ 2 ≤ M ] = B ( π τ , M ) \mathbb{E}_{\pi_{\tau},\Sigma}\left[(\boldsymbol{T}-m_{\pi_{\tau,M},\Sigma}(\boldsymbol{Y}))^{\top}H_{\mu}(\boldsymbol{T}-m_{\pi_{\tau,M},\Sigma}(\boldsymbol{Y}))\mid\lVert\boldsymbol{T}\rVert_{2}\leq M\right]=B(\pi_{\tau,M}) E π τ , Σ [ ( T − m π τ , M , Σ ( Y ) ) ⊤ H μ ( T − m π τ , M , Σ ( Y )) ∣ ∥ T ∥ 2 ≤ M ] = B ( π τ , M ) . Since the posterior expectation ∥ m π τ , M , Σ ( Y ) ∥ 2 ≤ M \lVert m_{\pi_{\tau,M},\Sigma}(\boldsymbol{Y})\rVert_{2}\leq M ∥ m π τ , M , Σ ( Y ) ∥ 2 ≤ M ,
∥ T − m π τ , M , Σ ( Y ) ∥ 2 ≤ ∥ T ∥ 2 + M ≤ 2 ∥ T ∥ 2 \lVert\boldsymbol{T}-m_{\pi_{\tau,M},\Sigma}(\boldsymbol{Y})\rVert_{2}\leq\lVert\boldsymbol{T}\rVert_{2}+M\leq2\lVert\boldsymbol{T}\rVert_{2} ∥ T − m π τ , M , Σ ( Y ) ∥ 2 ≤ ∥ T ∥ 2 + M ≤ 2 ∥ T ∥ 2
on { ∥ T ∥ 2 > M } \{\lVert\boldsymbol{T}\rVert_{2}>M\} {∥ T ∥ 2 > M } , which yields
( T − m π τ , M , Σ ( Y ) ) ⊤ H μ ( T − m π τ , M , Σ ( Y ) ) ≤ ∥ H μ ∥ op ∥ T − m π τ , M , Σ ( Y ) ∥ 2 2 ≤ 4 ∥ H μ ∥ op ∥ T ∥ 2 2 . (\boldsymbol{T}-m_{\pi_{\tau,M},\Sigma}(\boldsymbol{Y}))^{\top}H_{\mu}(\boldsymbol{T}-m_{\pi_{\tau,M},\Sigma}(\boldsymbol{Y}))\leq\lVert H_{\mu}\rVert_{\operatorname{op}}\lVert\boldsymbol{T}-m_{\pi_{\tau,M},\Sigma}(\boldsymbol{Y})\rVert_{2}^{2}\leq4\lVert H_{\mu}\rVert_{\operatorname{op}}\lVert\boldsymbol{T}\rVert_{2}^{2}. ( T − m π τ , M , Σ ( Y ) ) ⊤ H μ ( T − m π τ , M , Σ ( Y )) ≤ ∥ H μ ∥ op ∥ T − m π τ , M , Σ ( Y ) ∥ 2 2 ≤ 4 ∥ H μ ∥ op ∥ T ∥ 2 2 .
Therefore,
B ( π τ ) ≤ p τ , M B ( π τ , M ) + 4 ∥ H μ ∥ op E π τ , Σ [ ∥ T ∥ 2 2 I { ∥ T ∥ 2 > M } ] . B(\pi_\tau) \le p_{\tau,M}B(\pi_{\tau,M}) + 4\lVert H_\mu\rVert_{\operatorname{op}}\mathbb{E}_{\pi_\tau,\Sigma}\left[\lVert T\rVert_2^2 I\{\lVert T\rVert_2 > M\}\right]. B ( π τ ) ≤ p τ , M B ( π τ , M ) + 4 ∥ H μ ∥ op E π τ , Σ [ ∥ T ∥ 2 2 I {∥ T ∥ 2 > M } ] .
With τ \tau τ fixed, as M → ∞ M \to\infty M → ∞ , we have p τ , M → 1 p_{\tau,M} \to1 p τ , M → 1 and E π τ , Σ [ ∥ T ∥ 2 2 I { ∥ T ∥ 2 > M } ] → 0 \mathbb{E}_{\pi_\tau,\Sigma}\left[\lVert T\rVert_2^2 I\{\lVert T\rVert_2 > M\}\right] \to0 E π τ , Σ [ ∥ T ∥ 2 2 I {∥ T ∥ 2 > M } ] → 0 , which gives
lim inf M → ∞ B ( π τ , M ) ≥ B ( π τ ) = τ 2 1 + τ 2 tr ( Σ H μ ) . \liminf_{M\to\infty} B(\pi_{\tau,M}) \ge B(\pi_\tau) = \frac{\tau^2}{1+\tau^2}\operatorname{tr}(\Sigma H_\mu). M → ∞ lim inf B ( π τ , M ) ≥ B ( π τ ) = 1 + τ 2 τ 2 tr ( Σ H μ ) .
Combined with B M , Σ ( H μ ) ≥ B ( π τ , M ) B_{M,\Sigma}(H_\mu) \ge B(\pi_{\tau,M}) B M , Σ ( H μ ) ≥ B ( π τ , M ) , we have lim inf M → ∞ B M , Σ ( H μ ) ≥ τ 2 1 + τ 2 tr ( Σ H μ ) \liminf_{M\to\infty} B_{M,\Sigma}(H_\mu) \ge\frac{\tau^2}{1+\tau^2}\operatorname{tr}(\Sigma H_\mu) lim inf M → ∞ B M , Σ ( H μ ) ≥ 1 + τ 2 τ 2 tr ( Σ H μ ) . Letting τ → ∞ \tau\to\infty τ → ∞ gives
lim inf M → ∞ B M , Σ ( H μ ) ≥ tr ( Σ H μ ) . (75) \liminf_{M\to\infty} B_{M,\Sigma}(H_\mu) \ge\operatorname{tr}(\Sigma H_\mu). \tag*{(75)} M → ∞ lim inf B M , Σ ( H μ ) ≥ tr ( Σ H μ ) . (75)
Combining (B.10) and (B.11), we prove (23) .
Finally, fix M > 0 M > 0 M > 0 and represent the sample mean as x ˉ = μ n , t + Z / n \bar{x} = \mu_{n,t} + Z/\sqrt{n} x ˉ = μ n , t + Z / n , where Z ∼ N ( 0 , Σ ) Z \sim\mathcal{N}(0,\Sigma) Z ∼ N ( 0 , Σ ) has the same distribution for every t ∈ B 2 ( M ) t \in B_2(M) t ∈ B 2 ( M ) . Expectations and probabilities without a subscript below are with respect to Z Z Z , and the regret is evaluated at the true mean μ n , t \mu_{n,t} μ n , t . Choose r > 0 r > 0 r > 0 such that f ∗ f_* f ∗ is twice continuously differentiable on the closed ball { z : ∥ z − μ ∥ 2 ≤ r } \{z : \lVert z-\mu\rVert_2 \le r\} { z : ∥ z − μ ∥ 2 ≤ r } , and choose a finite constant C C C satisfying
sup ∥ z − μ ∥ 2 ≤ r ∥ ∇ 2 f ∗ ( z ) ∥ op ≤ C . (76) \sup_{\lVert z-\mu\rVert_2\le r}\lVert\nabla^2 f_*(z)\rVert_{\operatorname{op}} \le C. \tag*{(76)} ∥ z − μ ∥ 2 ≤ r sup ∥ ∇ 2 f ∗ ( z ) ∥ op ≤ C . (76)
Fix a noise truncation radius L > 0 L > 0 L > 0 . For sufficiently large n n n , M / n ≤ r / 2 M/\sqrt{n} \le r/2 M / n ≤ r /2 and ( M + L ) / n ≤ r (M+L)/\sqrt{n} \le r ( M + L ) / n ≤ r . Thus ∥ Z ∥ 2 ≤ L \lVert Z\rVert_2 \le L ∥ Z ∥ 2 ≤ L implies ∥ x ˉ − μ ∥ 2 ≤ r \lVert\bar{x}-\mu\rVert_2 \le r ∥ x ˉ − μ ∥ 2 ≤ r , uniformly over t ∈ B 2 ( M ) t \in B_2(M) t ∈ B 2 ( M ) . Since E [ Z ⊤ H μ Z ] = tr ( Σ H μ ) \mathbb{E}[Z^\top H_\mu Z] = \operatorname{tr}(\Sigma H_\mu) E [ Z ⊤ H μ Z ] = tr ( Σ H μ ) , we first split the expected difference over three disjoint events:
∣ n E μ n , t , Σ R ( w ∗ ( x ˉ ) ) − 1 2 tr ( Σ H μ ) ∣ ≤ I 1 , n ( t ) + I 2 , n ( t ) + I 3 , n ( t ) , \left|n\mathbb{E}_{\mu_{n,t},\Sigma}R(w_*(\bar{x}))-\frac{1}{2}\operatorname{tr}(\Sigma H_\mu)\right| \le I_{1,n}(t)+I_{2,n}(t)+I_{3,n}(t), n E μ n , t , Σ R ( w ∗ ( x ˉ )) − 2 1 tr ( Σ H μ ) ≤ I 1 , n ( t ) + I 2 , n ( t ) + I 3 , n ( t ) ,
where, with M M M and L L L fixed,
I 1 , n ( t ) : = E [ ∣ n R ( w ∗ ( x ˉ ) ) − 1 2 Z ⊤ H μ Z ∣ I { ∥ Z ∥ 2 ≤ L } ] , I_{1,n}(t) := \mathbb{E}\left[\left|nR(w_*(\bar{x}))-\frac{1}{2}Z^\top H_\mu Z\right|I\{\lVert Z\rVert_2\le L\}\right], I 1 , n ( t ) := E [ n R ( w ∗ ( x ˉ )) − 2 1 Z ⊤ H μ Z I {∥ Z ∥ 2 ≤ L } ] ,
I 2 , n ( t ) : = E [ ∣ n R ( w ∗ ( x ˉ ) ) − 1 2 Z ⊤ H μ Z ∣ I { ∥ Z ∥ 2 > L , ∥ x ˉ − μ ∥ 2 ≤ r } ] , I_{2,n}(t) := \mathbb{E}\left[\left|nR(w_*(\bar{x}))-\frac{1}{2}Z^\top H_\mu Z\right|I\{\lVert Z\rVert_2>L,\ \lVert\bar{x}-\mu\rVert_2\le r\}\right], I 2 , n ( t ) := E [ n R ( w ∗ ( x ˉ )) − 2 1 Z ⊤ H μ Z I {∥ Z ∥ 2 > L , ∥ x ˉ − μ ∥ 2 ≤ r } ] ,
I 3 , n ( t ) : = E [ ∣ n R ( w ∗ ( x ˉ ) ) − 1 2 Z ⊤ H μ Z ∣ I { ∥ x ˉ − μ ∥ 2 > r } ] . I_{3,n}(t) := \mathbb{E}\left[\left|nR(w_*(\bar{x}))-\frac{1}{2}Z^\top H_\mu Z\right|I\{\lVert\bar{x}-\mu\rVert_2>r\}\right]. I 3 , n ( t ) := E [ n R ( w ∗ ( x ˉ )) − 2 1 Z ⊤ H μ Z I {∥ x ˉ − μ ∥ 2 > r } ] .
For the first term, on { ∥ Z ∥ 2 ≤ L } \{\lVert Z\rVert_2\le L\} {∥ Z ∥ 2 ≤ L } , both μ n , t \mu_{n,t} μ n , t and x ˉ \bar{x} x ˉ lie in the smooth ball { z : ∥ z − μ ∥ 2 ≤ r } \{z : \lVert z-\mu\rVert_2\le r\} { z : ∥ z − μ ∥ 2 ≤ r } . Therefore,
n R ( w ∗ ( x ˉ ) ) = n ( f ∗ ( μ n , t ) − w ∗ ( x ˉ ) ⊤ μ n , t ) = n [ f ∗ ( μ n , t ) − f ∗ ( x ˉ ) − ∇ f ∗ ( x ˉ ) ⊤ ( μ n , t − x ˉ ) ] = ∫ 0 1 ( 1 − s ) Z ⊤ ∇ 2 f ∗ ( μ + t + ( 1 − s ) Z n ) Z d s . (77) \begin{aligned}
nR(w_*(\bar{x})) &= n\left(f_*(\mu_{n,t})-w_*(\bar{x})^\top\mu_{n,t}\right) \\
&= n\left[f_*(\mu_{n,t})-f_*(\bar{x})-\nabla f_*(\bar{x})^\top(\mu_{n,t}-\bar{x})\right] \\
&= \int_0^1 (1-s)Z^\top\nabla^2 f_*\left(\mu+\frac{t+(1-s)Z}{\sqrt{n}}\right)Z\,\mathrm{d}s. \tag*{(77)}
\end{aligned} n R ( w ∗ ( x ˉ )) = n ( f ∗ ( μ n , t ) − w ∗ ( x ˉ ) ⊤ μ n , t ) = n [ f ∗ ( μ n , t ) − f ∗ ( x ˉ ) − ∇ f ∗ ( x ˉ ) ⊤ ( μ n , t − x ˉ ) ] = ∫ 0 1 ( 1 − s ) Z ⊤ ∇ 2 f ∗ ( μ + n t + ( 1 − s ) Z ) Z d s . (77)
Consequently, on { ∥ Z ∥ 2 ≤ L } \{\|Z\|_2 \le L\} { ∥ Z ∥ 2 ≤ L } ,
∣ n R ( w ∗ ( x ˉ ) ) − 1 2 Z T H μ Z ∣ = ∣ ∫ 0 1 ( 1 − s ) Z T [ ∇ 2 f ∗ ( μ + t + ( 1 − s ) Z n ) − H μ ] Z d s ∣ ≤ ∥ Z ∥ 2 2 ∫ 0 1 ( 1 − s ) ∥ ∇ 2 f ∗ ( μ + t + ( 1 − s ) Z n ) − H μ ∥ o p d s ≤ L 2 sup ∥ z − μ ∥ 2 ≤ ( M + L ) / n ∥ ∇ 2 f ∗ ( z ) − H μ ∥ o p ⋅ ∫ 0 1 ( 1 − s ) d s = L 2 2 sup ∥ z − μ ∥ 2 ≤ ( M + L ) / n ∥ ∇ 2 f ∗ ( z ) − H μ ∥ o p , \begin{aligned}
\left|nR(w_\ast(\bar{x}))-\frac{1}{2}Z^{\mathsf{T}}H_\mu Z\right|=\left|\int_0^1(1-s)Z^{\mathsf{T}}\left[\nabla^2 f_\ast\left(\mu+\frac{t+(1-s)Z}{\sqrt{n}}\right)-H_\mu\right]Z\,\mathrm{d}s\right| \\
&\le\|Z\|_2^2\int_0^1(1-s)\left\|\nabla^2 f_\ast\left(\mu+\frac{t+(1-s)Z}{\sqrt{n}}\right)-H_\mu\right\|_{\mathrm{op}}\,\mathrm{d}s \\
&\le L^2\sup_{\|z-\mu\|_2\le(M+L)/\sqrt{n}}\left\|\nabla^2 f_\ast(z)-H_\mu\right\|_{\mathrm{op}}\cdot\int_0^1(1-s)\,\mathrm{d}s \\
&=\frac{L^2}{2}\sup_{\|z-\mu\|_2\le(M+L)/\sqrt{n}}\left\|\nabla^2 f_\ast(z)-H_\mu\right\|_{\mathrm{op}},
\end{aligned} n R ( w ∗ ( x ˉ )) − 2 1 Z T H μ Z = ∫ 0 1 ( 1 − s ) Z T [ ∇ 2 f ∗ ( μ + n t + ( 1 − s ) Z ) − H μ ] Z d s ≤ ∥ Z ∥ 2 2 ∫ 0 1 ( 1 − s ) ∇ 2 f ∗ ( μ + n t + ( 1 − s ) Z ) − H μ op d s ≤ L 2 ∥ z − μ ∥ 2 ≤ ( M + L ) / n sup ∇ 2 f ∗ ( z ) − H μ op ⋅ ∫ 0 1 ( 1 − s ) d s = 2 L 2 ∥ z − μ ∥ 2 ≤ ( M + L ) / n sup ∇ 2 f ∗ ( z ) − H μ op ,
which immediately gives
sup t ∈ B 2 ( M ) I 1 , n ( t ) ≤ L 2 2 sup ∥ z − μ ∥ 2 ≤ ( M + L ) / n ∥ ∇ 2 f ∗ ( z ) − H μ ∥ o p ⟶ 0 \sup_{t\in B_2(M)}I_{1,n}(t)\le\frac{L^2}{2}\sup_{\|z-\mu\|_2\le(M+L)/\sqrt{n}}\left\|\nabla^2 f_\ast(z)-H_\mu\right\|_{\mathrm{op}}\longrightarrow0 t ∈ B 2 ( M ) sup I 1 , n ( t ) ≤ 2 L 2 ∥ z − μ ∥ 2 ≤ ( M + L ) / n sup ∇ 2 f ∗ ( z ) − H μ op ⟶ 0
as n → ∞ n\to\infty n → ∞ for each fixed L L L , by continuity of the Hessian.
For the second term, formula (B.13) remains valid whenever ∥ x ˉ − μ ∥ 2 ≤ r \|\bar{x}-\mu\|_2\le r ∥ x ˉ − μ ∥ 2 ≤ r , since ∥ μ n , t − μ ∥ 2 ≤ M / n ≤ r / 2 \|\mu_{n,t}-\mu\|_2\le M/\sqrt{n}\le r/2 ∥ μ n , t − μ ∥ 2 ≤ M / n ≤ r /2 . Therefore, (B.12) implies
∣ n R ( w ∗ ( x ˉ ) ) ∣ = ∣ ∫ 0 1 ( 1 − s ) Z T ∇ 2 f ∗ ( μ + t + ( 1 − s ) Z n ) Z d s ∣ ≤ C ∥ Z ∥ 2 2 / 2. \left|nR(w_\ast(\bar{x}))\right|=\left|\int_0^1(1-s)Z^{\mathsf{T}}\nabla^2 f_\ast\left(\mu+\frac{t+(1-s)Z}{\sqrt{n}}\right)Z\,\mathrm{d}s\right|\le C\|Z\|_2^2/2. ∣ n R ( w ∗ ( x ˉ )) ∣ = ∫ 0 1 ( 1 − s ) Z T ∇ 2 f ∗ ( μ + n t + ( 1 − s ) Z ) Z d s ≤ C ∥ Z ∥ 2 2 /2.
Also, 0 ≤ Z T H μ Z / 2 ≤ C ∥ Z ∥ 2 2 / 2 0\le Z^{\mathsf{T}}H_\mu Z/2\le C\|Z\|_2^2/2 0 ≤ Z T H μ Z /2 ≤ C ∥ Z ∥ 2 2 /2 . Therefore, for all sufficiently large n n n ,
sup t ∈ B 2 ( M ) I 2 , n ( t ) ≤ C E [ ∥ Z ∥ 2 2 { ∥ Z ∥ 2 > L } ] . \sup_{t\in B_2(M)}I_{2,n}(t)\le C\mathbb{E}\left[\|Z\|_2^2\{\|Z\|_2>L\}\right]. t ∈ B 2 ( M ) sup I 2 , n ( t ) ≤ C E [ ∥ Z ∥ 2 2 { ∥ Z ∥ 2 > L } ] .
This bound tends to zero as L → ∞ L\to\infty L → ∞ , since E ∥ Z ∥ 2 2 = tr ( Σ ) ≤ d λ ˉ < ∞ \mathbb{E}\|Z\|_2^2=\operatorname{tr}(\Sigma)\le d\bar{\lambda}<\infty E ∥ Z ∥ 2 2 = tr ( Σ ) ≤ d λ ˉ < ∞ .
For the third term, ∥ x ˉ − μ ∥ 2 > r \|\bar{x}-\mu\|_2>r ∥ x ˉ − μ ∥ 2 > r implies ∥ Z ∥ 2 > r n / 2 \|Z\|_2>r\sqrt{n}/2 ∥ Z ∥ 2 > r n /2 , uniformly over t ∈ B 2 ( M ) t\in B_2(M) t ∈ B 2 ( M ) . By Chernoff’s inequality and E e ∥ Z ∥ 2 2 / ( 4 λ ˉ ) ≤ 2 d / 2 \mathbb{E}e^{\|Z\|_2^2/(4\bar{\lambda})}\le2^{d/2} E e ∥ Z ∥ 2 2 / ( 4 λ ˉ ) ≤ 2 d /2 ,
sup t ∈ B 2 ( M ) P { ∥ x ˉ − μ ∥ 2 > r } ≤ P { ∥ Z ∥ 2 > r n / 2 } ≤ e − r 2 n / ( 16 λ ˉ ) E e ∥ Z ∥ 2 2 / ( 4 λ ˉ ) ≤ 2 d / 2 e − r 2 n / ( 16 λ ˉ ) . \sup_{t\in B_2(M)}\mathbb{P}\{\|\bar{x}-\mu\|_2>r\}\le\mathbb{P}\{\|Z\|_2>r\sqrt{n}/2\}\le e^{-r^2n/(16\bar{\lambda})}\mathbb{E}e^{\|Z\|_2^2/(4\bar{\lambda})}\le2^{d/2}e^{-r^2n/(16\bar{\lambda})}. t ∈ B 2 ( M ) sup P { ∥ x ˉ − μ ∥ 2 > r } ≤ P { ∥ Z ∥ 2 > r n /2 } ≤ e − r 2 n / ( 16 λ ˉ ) E e ∥ Z ∥ 2 2 / ( 4 λ ˉ ) ≤ 2 d /2 e − r 2 n / ( 16 λ ˉ ) .
Recall R Ω = sup w ∈ Ω ∥ w ∥ 2 < ∞ R_\Omega=\sup_{w\in\Omega}\|w\|_2<\infty R Ω = sup w ∈ Ω ∥ w ∥ 2 < ∞ . We have
0 ≤ R ( w ∗ ( x ˉ ) ) = ( w ∗ ( μ n , t ) − w ∗ ( x ˉ ) ) T μ n , t ≤ 2 R Ω ( ∥ μ ∥ 2 + M n ) . 0\le R(w_\ast(\bar{x}))=(w_\ast(\mu_{n,t})-w_\ast(\bar{x}))^{\mathsf{T}}\mu_{n,t}\le2R_\Omega\left(\|\mu\|_2+\frac{M}{\sqrt{n}}\right). 0 ≤ R ( w ∗ ( x ˉ )) = ( w ∗ ( μ n , t ) − w ∗ ( x ˉ ) ) T μ n , t ≤ 2 R Ω ( ∥ μ ∥ 2 + n M ) .
Noting that 0 ≤ Z T H μ Z / 2 ≤ C ∥ Z ∥ 2 2 / 2 0\le Z^{\mathsf{T}}H_\mu Z/2\le C\|Z\|_2^2/2 0 ≤ Z T H μ Z /2 ≤ C ∥ Z ∥ 2 2 /2 , we obtain
sup t ∈ B 2 ( M ) I 3 , n ( t ) ≤ 2 n R Ω ( ∥ μ ∥ 2 + M n ) 2 d / 2 e − r 2 n / ( 16 λ ˉ ) + C 2 E [ ∥ Z ∥ 2 2 { ∥ Z ∥ 2 > r n / 2 } ] ⟶ 0. \sup_{t\in B_2(M)}I_{3,n}(t)\le2nR_\Omega\left(\|\mu\|_2+\frac{M}{\sqrt{n}}\right)2^{d/2}e^{-r^2n/(16\bar{\lambda})}
+\frac{C}{2}\mathbb{E}\left[\|Z\|_2^2\{\|Z\|_2>r\sqrt{n}/2\}\right]\longrightarrow0. t ∈ B 2 ( M ) sup I 3 , n ( t ) ≤ 2 n R Ω ( ∥ μ ∥ 2 + n M ) 2 d /2 e − r 2 n / ( 16 λ ˉ ) + 2 C E [ ∥ Z ∥ 2 2 { ∥ Z ∥ 2 > r n /2 } ] ⟶ 0.
as n → ∞ n\to\infty n → ∞ , since $\mathbb{E}|Z|_2^2=\operatorname{tr}(\Sigma)\le d\bar{\lambda}<\infty.
Combining the three bounds and first letting n → ∞ n \to\infty n → ∞ gives
lim sup n → ∞ sup t ∈ B 2 ( M ) ∣ n E μ n , t , Σ R ( w ∗ ( x ˉ ) ) − 1 2 tr ( Σ H μ ) ∣ ≤ C E [ ∥ Z ∥ 2 2 I { ∥ Z ∥ 2 > L } ] . \limsup_{n\to\infty}\sup_{t\in B_2(M)}\left|n\mathbb{E}_{\mu_n,t,\Sigma}R(w_*(\bar{x}))-\frac{1}{2}\operatorname{tr}(\Sigma H_\mu)\right|\le C\mathbb{E}\left[\lVert Z\rVert_2^2\mathbb{I}\{\lVert Z\rVert_2>L\}\right]. n → ∞ lim sup t ∈ B 2 ( M ) sup n E μ n , t , Σ R ( w ∗ ( x ˉ )) − 2 1 tr ( Σ H μ ) ≤ C E [ ∥ Z ∥ 2 2 I {∥ Z ∥ 2 > L } ] .
Letting L → ∞ L\to\infty L → ∞ proves (24) for each fixed Σ \Sigma Σ .
The expansion error ϵ n , M \epsilon_{n,M} ϵ n , M does not depend on Σ \Sigma Σ , so
sup Σ ∈ C ∣ n R n , M Σ ( μ ) − 1 2 B M , Σ ( H μ ) ∣ ≤ 2 ϵ n , M → 0. \sup_{\Sigma\in\mathcal{C}}\left|nR^\Sigma_{n,M}(\mu)-\frac{1}{2}\mathcal{B}_{M,\Sigma}(H_\mu)\right|\le2\epsilon_{n,M}\to0. Σ ∈ C sup n R n , M Σ ( μ ) − 2 1 B M , Σ ( H μ ) ≤ 2 ϵ n , M → 0.
It remains to make the plug-in limit uniform over Σ ∈ C \Sigma\in\mathcal{C} Σ ∈ C . Keep the same r r r and Hessian bound as above; neither depends on Σ \Sigma Σ . In I 1 , n ( t ) I_{1,n}(t) I 1 , n ( t ) , I 2 , n ( t ) I_{2,n}(t) I 2 , n ( t ) , and I 3 , n ( t ) I_{3,n}(t) I 3 , n ( t ) , the expectation is now evaluated under Z = Σ 1 / 2 G Z=\Sigma^{1/2}G Z = Σ 1/2 G , where G ∼ N ( 0 , I d ) G\sim\mathcal{N}(0,I_d) G ∼ N ( 0 , I d ) ; the dependence on Σ \Sigma Σ remains implicit in this notation. For each fixed L L L and all sufficiently large n n n , the first bound gives
sup Σ ∈ C sup t ∈ B 2 ( M ) I 1 , n ( t ) ≤ L 2 2 sup ∥ z − μ ∥ 2 ≤ ( M + L ) / n ∥ ∇ 2 f ∗ ( z ) − H μ ∥ o p → 0. \sup_{\Sigma\in\mathcal{C}}\sup_{t\in B_2(M)} I_{1,n}(t)\le\frac{L^2}{2}\sup_{\lVert z-\mu\rVert_2\le(M+L)/\sqrt{n}}\left\lVert\nabla^2f_*(z)-H_\mu\right\rVert_{\mathrm{op}}\to0. Σ ∈ C sup t ∈ B 2 ( M ) sup I 1 , n ( t ) ≤ 2 L 2 ∥ z − μ ∥ 2 ≤ ( M + L ) / n sup ∇ 2 f ∗ ( z ) − H μ op → 0.
For the second bound, Σ ⪯ λ ‾ I d \Sigma\preceq\overline{\lambda}I_d Σ ⪯ λ I d implies, pointwise in G G G , ∥ Z ∥ 2 2 = G ⊤ Σ G ≤ λ ‾ ∥ G ∥ 2 2 \lVert Z\rVert_2^2=G^\top\Sigma G\le\overline{\lambda}\lVert G\rVert_2^2 ∥ Z ∥ 2 2 = G ⊤ Σ G ≤ λ ∥ G ∥ 2 2 , implying { ∥ Z ∥ 2 > L } ⊆ { ∥ G ∥ 2 > L / λ ‾ } \{\lVert Z\rVert_2>L\}\subseteq\{\lVert G\rVert_2>L/\sqrt{\overline{\lambda}}\} {∥ Z ∥ 2 > L } ⊆ {∥ G ∥ 2 > L / λ } . Consequently,
sup Σ ∈ C sup t ∈ B 2 ( M ) I 2 , n ( t ) ≤ C sup Σ ∈ C E [ ∥ Z ∥ 2 2 I { ∥ Z ∥ 2 > L } ] ≤ C λ ‾ E [ ∥ G ∥ 2 2 I { ∥ G ∥ 2 > L / λ ‾ } ] . \begin{aligned}
\sup_{\Sigma\in\mathcal{C}}\sup_{t\in B_2(M)} I_{2,n}(t)\le C\sup_{\Sigma\in\mathcal{C}}\mathbb{E}\left[\lVert Z\rVert_2^2\mathbb{I}\{\lVert Z\rVert_2>L\}\right] \\
&\le C\overline{\lambda}\mathbb{E}\left[\lVert G\rVert_2^2\mathbb{I}\{\lVert G\rVert_2>L/\sqrt{\overline{\lambda}}\}\right].
\end{aligned} Σ ∈ C sup t ∈ B 2 ( M ) sup I 2 , n ( t ) ≤ C Σ ∈ C sup E [ ∥ Z ∥ 2 2 I {∥ Z ∥ 2 > L } ] ≤ C λ E [ ∥ G ∥ 2 2 I {∥ G ∥ 2 > L / λ } ] .
The right-hand side tends to zero as L → ∞ L\to\infty L → ∞ by DCT, since E ∥ G ∥ 2 2 = d < ∞ \mathbb{E}\lVert G\rVert_2^2=d<\infty E ∥ G ∥ 2 2 = d < ∞ .
For the third bound, if M / n ≤ r / 2 M/\sqrt{n}\le r/2 M / n ≤ r /2 , then
{ ∥ x ˉ − μ ∥ 2 > r } ⊆ { ∥ Z ∥ 2 > r n / 2 } ⊆ { ∥ G ∥ 2 > r n / ( 2 λ ‾ ) } . \{\lVert\bar{x}-\mu\rVert_2>r\}\subseteq\{\lVert Z\rVert_2>r\sqrt{n}/2\}\subseteq\{\lVert G\rVert_2>r\sqrt{n}/(2\sqrt{\overline{\lambda}})\}. {∥ x ˉ − μ ∥ 2 > r } ⊆ {∥ Z ∥ 2 > r n /2 } ⊆ {∥ G ∥ 2 > r n / ( 2 λ )} .
The preceding exponential probability bound and the same pointwise comparison therefore yield
sup Σ ∈ C sup t ∈ B 2 ( M ) I 3 , n ( t ) ≤ 2 n R Ω ( ∥ μ ∥ 2 + M n ) 2 d / 2 e − r 2 n / ( 16 λ ‾ ) + C λ ‾ 2 E [ ∥ G ∥ 2 2 I { ∥ G ∥ 2 > r n / ( 2 λ ‾ ) } ] → 0. \begin{aligned}
\sup_{\Sigma\in\mathcal{C}}\sup_{t\in B_2(M)} I_{3,n}(t)\le2nR_\Omega\left(\lVert\mu\rVert_2+\frac{M}{\sqrt{n}}\right)^{2d/2}e^{-r^2n/(16\overline{\lambda})} \\
&+\frac{C\overline{\lambda}}{2}\mathbb{E}\left[\lVert G\rVert_2^2\mathbb{I}\{\lVert G\rVert_2>r\sqrt{n}/(2\sqrt{\overline{\lambda}})\}\right]\to0.
\end{aligned} Σ ∈ C sup t ∈ B 2 ( M ) sup I 3 , n ( t ) ≤ 2 n R Ω ( ∥ μ ∥ 2 + n M ) 2 d /2 e − r 2 n / ( 16 λ ) + 2 C λ E [ ∥ G ∥ 2 2 I {∥ G ∥ 2 > r n / ( 2 λ )} ] → 0.
The first term tends to zero because exponential decay dominates its polynomial factor, and the second tends to zero by DCT. All three bounds hold for the same sufficiently large n n n , independently of t t t and Σ \Sigma Σ . Taking the supremum in the three-term decomposition and then letting n → ∞ n\to\infty n → ∞ , with M M M and L L L fixed, gives
lim sup n → ∞ sup Σ ∈ C sup t ∈ B 2 ( M ) ∣ n E μ n , t , Σ R ( w ∗ ( x ˉ ) ) − 1 2 tr ( Σ H μ ) ∣ ≤ C λ ‾ E [ ∥ G ∥ 2 2 I { ∥ G ∥ 2 > L / λ ‾ } ] . \limsup_{n\to\infty}\sup_{\Sigma\in\mathcal{C}}\sup_{t\in B_2(M)}\left|n\mathbb{E}_{\mu_n,t,\Sigma}R(w_*(\bar{x}))-\frac{1}{2}\operatorname{tr}(\Sigma H_\mu)\right|\le C\overline{\lambda}\mathbb{E}\left[\lVert G\rVert_2^2\mathbb{I}\{\lVert G\rVert_2>L/\sqrt{\overline{\lambda}}\}\right]. n → ∞ lim sup Σ ∈ C sup t ∈ B 2 ( M ) sup n E μ n , t , Σ R ( w ∗ ( x ˉ )) − 2 1 tr ( Σ H μ ) ≤ C λ E [ ∥ G ∥ 2 2 I {∥ G ∥ 2 > L / λ } ] .
Letting L → ∞ L\to\infty L → ∞ proves (24) , uniformly over Σ ∈ C \Sigma\in\mathcal{C} Σ ∈ C .
Proofs for General Local Scales and the Optimization Modulus
From pointwise to uniform convergence
Proposition C.1. The pointwise convergence (25) implies convergence uniformly over t ∈ B 2 ( M ) t \in B_2(M) t ∈ B 2 ( M ) :
lim ε ↓ 0 sup t ∈ B 2 ( M ) ∣ r μ ( ε t ) a ( ε ) − ψ ( t ) ∣ = 0. (78) \lim_{\varepsilon\downarrow0}\ \sup_{t\in B_2(M)}\left|\frac{r_\mu(\varepsilon t)}{a(\varepsilon)}-\psi(t)\right|=0. \tag*{(78)} ε ↓ 0 lim t ∈ B 2 ( M ) sup a ( ε ) r μ ( εt ) − ψ ( t ) = 0. (78)
Proof. Indeed, r μ r_\mu r μ is nonnegative and convex, with r μ ( 0 ) = 0 r_\mu(0)=0 r μ ( 0 ) = 0 , and a ( ε ) > 0 a(\varepsilon)>0 a ( ε ) > 0 . Passing to pointwise limits gives ψ ( t ) ≥ 0 \psi(t)\geq0 ψ ( t ) ≥ 0 , ψ ( 0 ) = 0 \psi(0)=0 ψ ( 0 ) = 0 , and
ψ ( λ s + ( 1 − λ ) t ) ≤ λ ψ ( s ) + ( 1 − λ ) ψ ( t ) , s , t ∈ R d , λ ∈ [ 0 , 1 ] . \psi(\lambda s+(1-\lambda)t)\leq\lambda\psi(s)+(1-\lambda)\psi(t),\qquad s,t\in\mathbb{R}^d,\quad\lambda\in[0,1]. ψ ( λ s + ( 1 − λ ) t ) ≤ λ ψ ( s ) + ( 1 − λ ) ψ ( t ) , s , t ∈ R d , λ ∈ [ 0 , 1 ] .
We next prove uniform convergence on B 2 ( M ) B_2(M) B 2 ( M ) for a fixed M > 0 M>0 M > 0 .
Let V M = { − ( M + 1 ) , M + 1 } d \mathcal{V}_M=\{-(M+1),M+1\}^d V M = { − ( M + 1 ) , M + 1 } d be the vertices of [ − ( M + 1 ) , M + 1 ] d [-(M+1),M+1]^d [ − ( M + 1 ) , M + 1 ] d , and set C M = 1 + sup z ∈ V M ψ ( z ) < ∞ C_M=1+\sup_{z\in\mathcal{V}_M}\psi(z)<\infty C M = 1 + sup z ∈ V M ψ ( z ) < ∞ . Since V M \mathcal{V}_M V M is finite, pointwise convergence gives 0 < ε M < ε 0 0<\varepsilon_M<\varepsilon_0 0 < ε M < ε 0 such that
sup z ∈ V M r μ ( ε z ) a ( ε ) ≤ C M , 0 < ε < ε M . \sup_{z\in\mathcal{V}_M}\frac{r_\mu(\varepsilon z)}{a(\varepsilon)}\leq C_M,\qquad0<\varepsilon<\varepsilon_M. z ∈ V M sup a ( ε ) r μ ( ε z ) ≤ C M , 0 < ε < ε M .
For every t ∈ B 2 ( M + 1 ) ⊆ conv ( V M ) t\in B_2(M+1)\subseteq\operatorname{conv}(\mathcal{V}_M) t ∈ B 2 ( M + 1 ) ⊆ conv ( V M ) , convexity therefore yields the uniform bound
0 ≤ r μ ( ε t ) a ( ε ) ≤ sup z ∈ V M r μ ( ε z ) a ( ε ) ≤ C M , 0 < ε < ε M . 0\leq\frac{r_\mu(\varepsilon t)}{a(\varepsilon)}\leq\sup_{z\in\mathcal{V}_M}\frac{r_\mu(\varepsilon z)}{a(\varepsilon)}\leq C_M,\qquad0<\varepsilon<\varepsilon_M. 0 ≤ a ( ε ) r μ ( εt ) ≤ z ∈ V M sup a ( ε ) r μ ( ε z ) ≤ C M , 0 < ε < ε M .
For distinct s , t ∈ B 2 ( M ) s,t\in B_2(M) s , t ∈ B 2 ( M ) , write
h = ∥ t − s ∥ 2 , z = t + t − s h ∈ B 2 ( M + 1 ) , t = 1 1 + h s + h 1 + h z . h=\lVert t-s\rVert_2,\qquad z=t+\frac{t-s}{h}\in B_2(M+1),\qquad t=\frac{1}{1+h}s+\frac{h}{1+h}z. h = ∥ t − s ∥ 2 , z = t + h t − s ∈ B 2 ( M + 1 ) , t = 1 + h 1 s + 1 + h h z .
Applying convexity to this combination gives
r μ ( ε t ) − r μ ( ε s ) a ( ε ) ≤ h 1 + h r μ ( ε z ) − r μ ( ε s ) a ( ε ) ≤ h 1 + h C M ≤ C M ∥ t − s ∥ 2 . \frac{r_\mu(\varepsilon t)-r_\mu(\varepsilon s)}{a(\varepsilon)}
\leq\frac{h}{1+h}\frac{r_\mu(\varepsilon z)-r_\mu(\varepsilon s)}{a(\varepsilon)}
\leq\frac{h}{1+h}C_M\leq C_M\lVert t-s\rVert_2. a ( ε ) r μ ( εt ) − r μ ( ε s ) ≤ 1 + h h a ( ε ) r μ ( ε z ) − r μ ( ε s ) ≤ 1 + h h C M ≤ C M ∥ t − s ∥ 2 .
Interchanging s s s and t t t proves, for all s , t ∈ B 2 ( M ) s,t\in B_2(M) s , t ∈ B 2 ( M ) and 0 < ε < ε M 0<\varepsilon<\varepsilon_M 0 < ε < ε M ,
∣ r μ ( ε t ) a ( ε ) − r μ ( ε s ) a ( ε ) ∣ ≤ C M ∥ t − s ∥ 2 . \left|\frac{r_\mu(\varepsilon t)}{a(\varepsilon)}-\frac{r_\mu(\varepsilon s)}{a(\varepsilon)}\right|\leq C_M\lVert t-s\rVert_2. a ( ε ) r μ ( εt ) − a ( ε ) r μ ( ε s ) ≤ C M ∥ t − s ∥ 2 .
Taking pointwise limits also gives
∣ ψ ( t ) − ψ ( s ) ∣ ≤ C M ∥ t − s ∥ 2 , s , t ∈ B 2 ( M ) . |\psi(t)-\psi(s)|\leq C_M\lVert t-s\rVert_2,\qquad s,t\in B_2(M). ∣ ψ ( t ) − ψ ( s ) ∣ ≤ C M ∥ t − s ∥ 2 , s , t ∈ B 2 ( M ) .
Thus ψ \psi ψ is locally Lipschitz, hence continuous.
Finally, fix δ > 0 \delta>0 δ > 0 . By compactness, there are finitely many points t 1 , … , t N ∈ B 2 ( M ) t_1,\ldots,t_N\in B_2(M) t 1 , … , t N ∈ B 2 ( M ) satisfying
sup t ∈ B 2 ( M ) min 1 ≤ j ≤ N ∥ t − t j ∥ 2 ≤ δ . \sup_{t\in B_2(M)}\min_{1\leq j\leq N}\lVert t-t_j\rVert_2\leq\delta. t ∈ B 2 ( M ) sup 1 ≤ j ≤ N min ∥ t − t j ∥ 2 ≤ δ .
For any t ∈ B 2 ( M ) t \in B_{2}(M) t ∈ B 2 ( M ) , choose j j j with ∥ t − t j ∥ 2 ≤ δ \lVert t-t_j\rVert_{2} \le\delta ∥ t − t j ∥ 2 ≤ δ . The two Lipschitz bounds imply
∣ r μ ( ε t ) a ( ε ) − ψ ( t ) ∣ ≤ ∣ r μ ( ε t ) − r μ ( ε t j ) a ( ε ) ∣ + ∣ r μ ( ε t j ) a ( ε ) − ψ ( t j ) ∣ + ∣ ψ ( t j ) − ψ ( t ) ∣ \left\lvert\frac{r_{\mu}(\varepsilon t)}{a(\varepsilon)}-\psi(t)\right\rvert\le\left\lvert\frac{r_{\mu}(\varepsilon t)-r_{\mu}(\varepsilon t_j)}{a(\varepsilon)}\right\rvert+ \left\lvert\frac{r_{\mu}(\varepsilon t_j)}{a(\varepsilon)}-\psi(t_j)\right\rvert+ \lvert\psi(t_j)-\psi(t)\rvert a ( ε ) r μ ( εt ) − ψ ( t ) ≤ a ( ε ) r μ ( εt ) − r μ ( ε t j ) + a ( ε ) r μ ( ε t j ) − ψ ( t j ) + ∣ ψ ( t j ) − ψ ( t )∣
≤ 2 C M δ + sup 1 ≤ j ≤ N ∣ r μ ( ε t j ) a ( ε ) − ψ ( t j ) ∣ . \le2C_M\delta+\sup_{1\le j\le N}\left\lvert\frac{r_{\mu}(\varepsilon t_j)}{a(\varepsilon)}-\psi(t_j)\right\rvert. ≤ 2 C M δ + 1 ≤ j ≤ N sup a ( ε ) r μ ( ε t j ) − ψ ( t j ) .
For this fixed finite set of points, the maximum tends to zero as ε ↓ 0 \varepsilon\downarrow0 ε ↓ 0 . Hence
lim sup ε ↓ 0 sup t ∈ B 2 ( M ) ∣ r μ ( ε t ) a ( ε ) − ψ ( t ) ∣ ≤ 2 C M δ , δ > 0. \limsup_{\varepsilon\downarrow0}\sup_{t\in B_2(M)}\left\lvert\frac{r_{\mu}(\varepsilon t)}{a(\varepsilon)}-\psi(t)\right\rvert\le2C_M\delta,\qquad\delta>0. ε ↓ 0 lim sup t ∈ B 2 ( M ) sup a ( ε ) r μ ( εt ) − ψ ( t ) ≤ 2 C M δ , δ > 0.
Letting δ ↓ 0 \delta\downarrow0 δ ↓ 0 proves (C.1). Thus finiteness of all the pointwise limits suffices. □ \square □
Proof of Theorem 3.3
Proof. For every π ∈ P ( B 2 ( M ) ) \pi\in\mathcal{P}(B_2(M)) π ∈ P ( B 2 ( M )) , both T T T and m π , Σ ( Y ) m_{\pi,\Sigma}(Y) m π , Σ ( Y ) belong to B 2 ( M ) B_2(M) B 2 ( M ) almost surely; for the latter, ∥ E π , Σ [ T ∣ Y ] ∥ 2 ≤ E π , Σ [ ∥ T ∥ 2 ∣ Y ] ≤ M \lVert\mathbb{E}_{\pi,\Sigma}[T\mid Y]\rVert_2\le\mathbb{E}_{\pi,\Sigma}[\lVert T\rVert_2\mid Y]\le M ∥ E π , Σ [ T ∣ Y ] ∥ 2 ≤ E π , Σ [∥ T ∥ 2 ∣ Y ] ≤ M . Consequently,
∣ E π , Σ [ r μ ( T / n ) − r μ ( m π , Σ ( Y ) / n ) a ( n − 1 / 2 ) ] − E π , Σ [ ψ ( T ) − ψ ( m π , Σ ( Y ) ) ] ∣ \left\lvert\mathbb{E}_{\pi,\Sigma}\left[\frac{r_{\mu}(T/\sqrt{n})-r_{\mu}(m_{\pi,\Sigma}(Y)/\sqrt{n})}{a(n^{-1/2})}\right]-\mathbb{E}_{\pi,\Sigma}\left[\psi(T)-\psi(m_{\pi,\Sigma}(Y))\right]\right\rvert E π , Σ [ a ( n − 1/2 ) r μ ( T / n ) − r μ ( m π , Σ ( Y ) / n ) ] − E π , Σ [ ψ ( T ) − ψ ( m π , Σ ( Y )) ]
≤ 2 sup t ∈ B 2 ( M ) ∣ r μ ( t / n ) a ( n − 1 / 2 ) − ψ ( t ) ∣ . \le2\sup_{t\in B_2(M)}\left\lvert\frac{r_{\mu}(t/\sqrt{n})}{a(n^{-1/2})}-\psi(t)\right\rvert. ≤ 2 t ∈ B 2 ( M ) sup a ( n − 1/2 ) r μ ( t / n ) − ψ ( t ) .
The bound is independent of π \pi π . Taking suprema over π \pi π and using (20) proves (29) ; assumption (C.1) then proves (27) .
It remains to determine when V M , Σ ( ψ ) V_{M,\Sigma}(\psi) V M , Σ ( ψ ) is positive. For every prior π ∈ P ( B 2 ( M ) ) \pi\in\mathcal{P}(B_2(M)) π ∈ P ( B 2 ( M )) , the tower property and conditional Jensen’s inequality give
E π , Σ [ ψ ( T ) − ψ ( m π , Σ ( Y ) ) ] = E π , Σ [ E π , Σ [ ψ ( T ) ∣ Y ] − ψ ( E π , Σ [ T ∣ Y ] ) ] ≥ 0. \mathbb{E}_{\pi,\Sigma}[\psi(T)-\psi(m_{\pi,\Sigma}(Y))]=\mathbb{E}_{\pi,\Sigma}\left[\mathbb{E}_{\pi,\Sigma}[\psi(T)\mid Y]-\psi(\mathbb{E}_{\pi,\Sigma}[T\mid Y])\right]\ge0. E π , Σ [ ψ ( T ) − ψ ( m π , Σ ( Y ))] = E π , Σ [ E π , Σ [ ψ ( T ) ∣ Y ] − ψ ( E π , Σ [ T ∣ Y ]) ] ≥ 0.
Since ψ \psi ψ is continuous and both arguments belong to the compact ball B 2 ( M ) B_2(M) B 2 ( M ) , we also have
0 ≤ V M , Σ ( ψ ) ≤ sup t ∈ B 2 ( M ) ψ ( t ) − min t ∈ B 2 ( M ) ψ ( t ) < ∞ . 0\le V_{M,\Sigma}(\psi)\le\sup_{t\in B_2(M)}\psi(t)-\min_{t\in B_2(M)}\psi(t)<\infty. 0 ≤ V M , Σ ( ψ ) ≤ t ∈ B 2 ( M ) sup ψ ( t ) − t ∈ B 2 ( M ) min ψ ( t ) < ∞.
If ψ \psi ψ is affine on this ball, write ψ ( t ) = b + c T t \psi(t)=b+c^{\mathsf T}t ψ ( t ) = b + c T t there. For every such prior,
E π , Σ [ ψ ( T ) − ψ ( m π , Σ ( Y ) ) ] = c T ( E π , Σ [ T ] − E π , Σ [ E π , Σ [ T ∣ Y ] ] ) = 0 , \mathbb{E}_{\pi,\Sigma}[\psi(T)-\psi(m_{\pi,\Sigma}(Y))]=c^{\mathsf T}\left(\mathbb{E}_{\pi,\Sigma}[T]-\mathbb{E}_{\pi,\Sigma}[\mathbb{E}_{\pi,\Sigma}[T\mid Y]]\right)=0, E π , Σ [ ψ ( T ) − ψ ( m π , Σ ( Y ))] = c T ( E π , Σ [ T ] − E π , Σ [ E π , Σ [ T ∣ Y ]] ) = 0 ,
so V M , Σ ( ψ ) = 0 V_{M,\Sigma}(\psi)=0 V M , Σ ( ψ ) = 0 .
Conversely, suppose that ψ \psi ψ is not affine on B 2 ( M ) B_2(M) B 2 ( M ) . There exist t 1 , t 2 ∈ B 2 ( M ) t_1,t_2\in B_2(M) t 1 , t 2 ∈ B 2 ( M ) such that
Δ : = ψ ( t 1 ) + ψ ( t 2 ) 2 − ψ ( t 1 + t 2 2 ) > 0. \Delta:=\frac{\psi(t_1)+\psi(t_2)}{2}-\psi\left(\frac{t_1+t_2}{2}\right)>0. Δ := 2 ψ ( t 1 ) + ψ ( t 2 ) − ψ ( 2 t 1 + t 2 ) > 0.
Indeed, if midpoint equality held for every pair, ψ \psi ψ is affine on every segment in the ball, hence affine on the ball. Choose the two-point prior π \pi π with P π { T = t 1 } = P π { T = t 2 } = 1 / 2 \mathbb{P}_{\pi}\{\mathcal{T}=t_1\}=\mathbb{P}_{\pi}\{\mathcal{T}=t_2\}=1/2 P π { T = t 1 } = P π { T = t 2 } = 1/2 . In the experiment Y = T + Z \mathbf{Y}=\mathbf{T}+\mathbf{Z} Y = T + Z , Bayes’ formula gives the posterior probability
q ( y ) : = P π , Σ { T = t 1 ∣ Y = y } = exp ( − ∥ Σ − 1 / 2 ( y − t 1 ) ∥ 2 2 / 2 ) exp ( − ∥ Σ − 1 / 2 ( y − t 1 ) ∥ 2 2 / 2 ) + exp ( − ∥ Σ − 1 / 2 ( y − t 2 ) ∥ 2 2 / 2 ) . q(y):=\mathbb{P}_{\pi,\Sigma}\{\mathbf{T}=t_1\mid\mathbf{Y}=y\}=\frac{\exp\left(-\lVert\Sigma^{-1/2}(y-t_1)\rVert_2^2/2\right)}{\exp\left(-\lVert\Sigma^{-1/2}(y-t_1)\rVert_2^2/2\right)+\exp\left(-\lVert\Sigma^{-1/2}(y-t_2)\rVert_2^2/2\right)}. q ( y ) := P π , Σ { T = t 1 ∣ Y = y } = exp ( − ∥ Σ − 1/2 ( y − t 1 ) ∥ 2 2 /2 ) + exp ( − ∥ Σ − 1/2 ( y − t 2 ) ∥ 2 2 /2 ) exp ( − ∥ Σ − 1/2 ( y − t 1 ) ∥ 2 2 /2 ) .
Both likelihoods are strictly positive, so 0 < q ( y ) < 1 0<q(y)<1 0 < q ( y ) < 1 for every y ∈ R d y\in\mathbb{R}^d y ∈ R d , and
m π , Σ ( y ) = q ( y ) t 1 + ( 1 − q ( y ) ) t 2 , m_{\pi,\Sigma}(y)=q(y)t_1+(1-q(y))t_2, m π , Σ ( y ) = q ( y ) t 1 + ( 1 − q ( y )) t 2 ,
E π , Σ [ ψ ( T ) ∣ Y = y ] = q ( y ) ψ ( t 1 ) + ( 1 − q ( y ) ) ψ ( t 2 ) . \mathbb{E}_{\pi,\Sigma}[\psi(\mathbf{T})\mid\mathbf{Y}=y]=q(y)\psi(t_1)+(1-q(y))\psi(t_2). E π , Σ [ ψ ( T ) ∣ Y = y ] = q ( y ) ψ ( t 1 ) + ( 1 − q ( y )) ψ ( t 2 ) .
To see explicitly why the conditional Jensen gap is positive, define
g ( q ) : = q ψ ( t 1 ) + ( 1 − q ) ψ ( t 2 ) − ψ ( q t 1 + ( 1 − q ) t 2 ) , 0 ≤ q ≤ 1. g(q):=q\psi(t_1)+(1-q)\psi(t_2)-\psi\left(qt_1+(1-q)t_2\right),\qquad0\leq q\leq1. g ( q ) := q ψ ( t 1 ) + ( 1 − q ) ψ ( t 2 ) − ψ ( q t 1 + ( 1 − q ) t 2 ) , 0 ≤ q ≤ 1.
Convexity of ψ \psi ψ makes g g g concave, with g ( 0 ) = g ( 1 ) = 0 g(0)=g(1)=0 g ( 0 ) = g ( 1 ) = 0 and g ( 1 / 2 ) = Δ g(1/2)=\Delta g ( 1/2 ) = Δ . Applying concavity on [ 0 , 1 / 2 ] [0,1/2] [ 0 , 1/2 ] and [ 1 / 2 , 1 ] [1/2,1] [ 1/2 , 1 ] , respectively, yields
g ( q ) ≥ { 2 q Δ , 0 ≤ q ≤ 1 / 2 , 2 ( 1 − q ) Δ , 1 / 2 ≤ q ≤ 1. g(q)\geq
\begin{cases}
2q\Delta, & 0\leq q\leq1/2,\\
2(1-q)\Delta, & 1/2\leq q\leq1.
\end{cases} g ( q ) ≥ { 2 q Δ , 2 ( 1 − q ) Δ , 0 ≤ q ≤ 1/2 , 1/2 ≤ q ≤ 1.
Consequently, this particular prior satisfies
V M , Σ ( ψ ) ≥ E π , Σ [ ψ ( T ) − ψ ( m π , Σ ( Y ) ) ] = E π , Σ [ g ( q ( Y ) ) ] ≥ 2 Δ E π , Σ [ min { q ( Y ) , 1 − q ( Y ) } ] > 0. V_{M,\Sigma}(\psi)\geq\mathbb{E}_{\pi,\Sigma}\left[\psi(\mathbf{T})-\psi\left(m_{\pi,\Sigma}(\mathbf{Y})\right)\right]
=\mathbb{E}_{\pi,\Sigma}[g(q(\mathbf{Y}))]
\geq2\Delta\mathbb{E}_{\pi,\Sigma}\left[\min\{q(\mathbf{Y}),1-q(\mathbf{Y})\}\right]>0. V M , Σ ( ψ ) ≥ E π , Σ [ ψ ( T ) − ψ ( m π , Σ ( Y ) ) ] = E π , Σ [ g ( q ( Y ))] ≥ 2Δ E π , Σ [ min { q ( Y ) , 1 − q ( Y )} ] > 0.
For the equally weighted two-point prior, the posterior testing error is
E π , Σ min { q ( Y ) , 1 − q ( Y ) } = Φ ( − 1 2 ∥ Σ − 1 / 2 ( t 1 − t 2 ) ∥ 2 ) ≥ Φ ( − M λ ) > 0. \mathbb{E}_{\pi,\Sigma}\min\{q(\mathbf{Y}),1-q(\mathbf{Y})\}
=\Phi\left(-\frac{1}{2}\lVert\Sigma^{-1/2}(t_1-t_2)\rVert_2\right)
\geq\Phi\left(-\frac{M}{\sqrt{\lambda}}\right)>0. E π , Σ min { q ( Y ) , 1 − q ( Y )} = Φ ( − 2 1 ∥ Σ − 1/2 ( t 1 − t 2 ) ∥ 2 ) ≥ Φ ( − λ M ) > 0.
Indeed, the log likelihood ratio is ( t 1 − t 2 ) T Σ − 1 ( Y − ( t 1 + t 2 ) / 2 ) (t_1-t_2)^{\mathsf{T}}\Sigma^{-1}(\mathbf{Y}-(t_1+t_2)/2) ( t 1 − t 2 ) T Σ − 1 ( Y − ( t 1 + t 2 ) /2 ) ; under the two hypotheses it has means ± D 2 / 2 \pm D^2/2 ± D 2 /2 and variance D 2 D^2 D 2 , where D = ∥ Σ − 1 / 2 ( t 1 − t 2 ) ∥ 2 ≤ 2 M / λ D=\lVert\Sigma^{-1/2}(t_1-t_2)\rVert_2\leq2M/\sqrt{\lambda} D = ∥ Σ − 1/2 ( t 1 − t 2 ) ∥ 2 ≤ 2 M / λ . Thus the positive lower bound for V M , Σ ( ψ ) V_{M,\Sigma}(\psi) V M , Σ ( ψ ) is uniform over C \mathcal{C} C , where Φ \Phi Φ denotes the standard normal distribution function. Finally, (27) gives R n , M Σ ( μ ) ≍ a ( n − 1 / 2 ) R_{n,M}^{\Sigma}(\mu)\asymp a(n^{-1/2}) R n , M Σ ( μ ) ≍ a ( n − 1/2 ) when V M , Σ ( ψ ) > 0 V_{M,\Sigma}(\psi)>0 V M , Σ ( ψ ) > 0 , and R n , M Σ ( μ ) = o ( a ( n − 1 / 2 ) ) R_{n,M}^{\Sigma}(\mu)=o(a(n^{-1/2})) R n , M Σ ( μ ) = o ( a ( n − 1/2 )) when V M , Σ ( ψ ) = 0 V_{M,\Sigma}(\psi)=0 V M , Σ ( ψ ) = 0 .
To prove the plug-in bound, fix M > 0 M>0 M > 0 with V M , Σ ( ψ ) > 0 V_{M,\Sigma}(\psi)>0 V M , Σ ( ψ ) > 0 . We first derive a growth bound for the remainder that will control unbounded Gaussian perturbations. Recall b M ( ε ) : = sup t ∈ B 2 ( M ) r μ ( ε t ) b_M(\varepsilon):=\sup_{t\in B_2(M)}r_\mu(\varepsilon t) b M ( ε ) := sup t ∈ B 2 ( M ) r μ ( εt ) for ε ≥ 0 \varepsilon\geq0 ε ≥ 0 . Since r μ r_\mu r μ is nonnegative and convex and vanishes at zero, b M b_M b M is nonnegative, convex, and nondecreasing, with b M ( 0 ) = 0 b_M(0)=0 b M ( 0 ) = 0 . f ∗ f_* f ∗ is R Ω R_\Omega R Ω -Lipschitz, where R Ω = sup w ∈ Ω ∥ w ∥ 2 R_\Omega=\sup_{w\in\Omega}\lVert w\rVert_2 R Ω = sup w ∈ Ω ∥ w ∥ 2 , so
0 ≤ b M ( ε ) ≤ 2 R Ω M ε , ε ≥ 0. (79) 0\leq b_M(\varepsilon)\leq2R_\Omega M\varepsilon,\qquad\varepsilon\geq0. \tag*{(79)} 0 ≤ b M ( ε ) ≤ 2 R Ω M ε , ε ≥ 0. (79)
(C.1) implies ψ ≥ 0 \psi\ge0 ψ ≥ 0 and ψ ( 0 ) = 0 \psi(0)=0 ψ ( 0 ) = 0 . Since V M , Σ ( ψ ) > 0 V_{M,\Sigma}(\psi)>0 V M , Σ ( ψ ) > 0 , sup t ∈ B 2 ( M ) ψ ( t ) > 0 \sup_{t\in B_2(M)}\psi(t)>0 sup t ∈ B 2 ( M ) ψ ( t ) > 0 . (C.1) gives
∣ b M ( ε ) a ( ε ) − sup t ∈ B 2 ( M ) ψ ( t ) ∣ ≤ sup t ∈ B 2 ( M ) ∣ r μ ( ε t ) a ( ε ) − ψ ( t ) ∣ → 0. (80) \left|\frac{b_M(\varepsilon)}{a(\varepsilon)}-\sup_{t\in B_2(M)}\psi(t)\right|\le\sup_{t\in B_2(M)}\left|\frac{r_\mu(\varepsilon t)}{a(\varepsilon)}-\psi(t)\right|\to0. \tag*{(80)} a ( ε ) b M ( ε ) − t ∈ B 2 ( M ) sup ψ ( t ) ≤ t ∈ B 2 ( M ) sup a ( ε ) r μ ( εt ) − ψ ( t ) → 0. (80)
Applying the same argument to b M ( 2 ε ) b_M(2\varepsilon) b M ( 2 ε ) gives
b M ( 2 ε ) b M ( ε ) → sup t ∈ B 2 ( 2 M ) ψ ( t ) sup t ∈ B 2 ( M ) ψ ( t ) < ∞ . (81) \frac{b_M(2\varepsilon)}{b_M(\varepsilon)}\to\frac{\sup_{t\in B_2(2M)}\psi(t)}{\sup_{t\in B_2(M)}\psi(t)}<\infty. \tag*{(81)} b M ( ε ) b M ( 2 ε ) → sup t ∈ B 2 ( M ) ψ ( t ) sup t ∈ B 2 ( 2 M ) ψ ( t ) < ∞. (81)
In particular, b M ( ε ) > 0 b_M(\varepsilon)>0 b M ( ε ) > 0 for small positive ε \varepsilon ε , hence for every ε > 0 \varepsilon>0 ε > 0 by monotonicity. Choose ε 1 > 0 \varepsilon_1>0 ε 1 > 0 sufficiently small that the ratio b M ( 2 ε ) / b M ( ε ) b_M(2\varepsilon)/b_M(\varepsilon) b M ( 2 ε ) / b M ( ε ) is bounded for 0 < ε ≤ ε 1 0<\varepsilon\le\varepsilon_1 0 < ε ≤ ε 1 . For ε ≥ ε 1 \varepsilon\ge\varepsilon_1 ε ≥ ε 1 , convexity and (C.2) imply
b M ( ε ) ≥ ε ε 1 b M ( ε 1 ) , b M ( 2 ε ) b M ( ε ) ≤ 4 R Ω M ε 1 b M ( ε 1 ) . b_M(\varepsilon)\ge\frac{\varepsilon}{\varepsilon_1}b_M(\varepsilon_1),\qquad\frac{b_M(2\varepsilon)}{b_M(\varepsilon)}\le\frac{4R_\Omega M\varepsilon_1}{b_M(\varepsilon_1)}. b M ( ε ) ≥ ε 1 ε b M ( ε 1 ) , b M ( ε ) b M ( 2 ε ) ≤ b M ( ε 1 ) 4 R Ω M ε 1 .
Consequently, a finite constant C b ≥ 2 C_b\ge2 C b ≥ 2 can be chosen so that b M ( 2 ε ) ≤ C b b M ( ε ) b_M(2\varepsilon)\le C_b b_M(\varepsilon) b M ( 2 ε ) ≤ C b b M ( ε ) for every ε > 0 \varepsilon>0 ε > 0 . Set c b = log 2 C b c_b=\log_2 C_b c b = log 2 C b . For λ ≥ 1 \lambda\ge1 λ ≥ 1 , take k = ⌈ log 2 λ ⌉ k=\lceil\log_2\lambda\rceil k = ⌈ log 2 λ ⌉ and iterate this inequality to obtain
b M ( λ ε ) ≤ b M ( 2 k ε ) ≤ C b k b M ( ε ) ≤ C b λ c b b M ( ε ) , ε > 0. (82) b_M(\lambda\varepsilon)\le b_M(2^k\varepsilon)\le C_b^k b_M(\varepsilon)\le C_b\lambda^{c_b}b_M(\varepsilon),\qquad\varepsilon>0. \tag*{(82)} b M ( λ ε ) ≤ b M ( 2 k ε ) ≤ C b k b M ( ε ) ≤ C b λ c b b M ( ε ) , ε > 0. (82)
Under the true mean μ n , t = μ + t / n \mu_{n,t}=\mu+t/\sqrt{n} μ n , t = μ + t / n , write x ˉ = μ + ( t + Z ) / n \bar{x}=\mu+(t+Z)/\sqrt{n} x ˉ = μ + ( t + Z ) / n with Z ∼ N ( 0 , Σ ) Z\sim\mathcal{N}(0,\Sigma) Z ∼ N ( 0 , Σ ) . Expanding the remainder and using f ∗ ( x ˉ ) = w ∗ ( x ˉ ) ⊤ x ˉ f_*(\bar{x})=w_*(\bar{x})^\top\bar{x} f ∗ ( x ˉ ) = w ∗ ( x ˉ ) ⊤ x ˉ gives
R ( w ∗ ( x ˉ ) ) = r μ ( t / n ) − r μ ( ( t + Z ) / n ) + ( w ∗ ( x ˉ ) − w ∗ ( μ ) ) ⊤ Z / n . R(w_*(\bar{x}))=r_\mu(t/\sqrt{n})-r_\mu((t+Z)/\sqrt{n})+(w_*(\bar{x})-w_*(\mu))^\top Z/\sqrt{n}. R ( w ∗ ( x ˉ )) = r μ ( t / n ) − r μ (( t + Z ) / n ) + ( w ∗ ( x ˉ ) − w ∗ ( μ ) ) ⊤ Z / n .
The maximizer w ∗ ( x ˉ ) w_*(\bar{x}) w ∗ ( x ˉ ) is feasible at x ˉ + Z / n \bar{x}+Z/\sqrt{n} x ˉ + Z / n , so
r μ ( ( t + 2 Z ) / n ) ≥ r μ ( ( t + Z ) / n ) + ( w ∗ ( x ˉ ) − w ∗ ( μ ) ) ⊤ Z / n . r_\mu((t+2Z)/\sqrt{n})\ge r_\mu((t+Z)/\sqrt{n})+(w_*(\bar{x})-w_*(\mu))^\top Z/\sqrt{n}. r μ (( t + 2 Z ) / n ) ≥ r μ (( t + Z ) / n ) + ( w ∗ ( x ˉ ) − w ∗ ( μ ) ) ⊤ Z / n .
Combining these two formulas and using r μ ≥ 0 r_\mu\ge0 r μ ≥ 0 yields
0 ≤ R ( w ∗ ( x ˉ ) ) ≤ r μ ( t / n ) + r μ ( ( t + 2 Z ) / n ) − 2 r μ ( ( t + Z ) / n ) ≤ r μ ( t / n ) + r μ ( ( t + 2 Z ) / n ) . (83) \begin{aligned}
0\le R(w_*(\bar{x}))&\le r_\mu(t/\sqrt{n})+r_\mu((t+2Z)/\sqrt{n})-2r_\mu((t+Z)/\sqrt{n})\\
&\le r_\mu(t/\sqrt{n})+r_\mu((t+2Z)/\sqrt{n}). \tag*{(83)}
\end{aligned} 0 ≤ R ( w ∗ ( x ˉ )) ≤ r μ ( t / n ) + r μ (( t + 2 Z ) / n ) − 2 r μ (( t + Z ) / n ) ≤ r μ ( t / n ) + r μ (( t + 2 Z ) / n ) . (83)
For t ∈ B 2 ( M ) t\in B_2(M) t ∈ B 2 ( M ) , we have ∥ t + 2 Z ∥ 2 ≤ M + 2 ∥ Z ∥ 2 \lVert t+2Z\rVert_2\le M+2\lVert Z\rVert_2 ∥ t + 2 Z ∥ 2 ≤ M + 2 ∥ Z ∥ 2 . The definition of b M b_M b M and (C.5) give
r μ ( ( t + 2 Z ) / n ) ≤ b M ( ( 1 + 2 ∥ Z ∥ 2 M ) / n ) ≤ C b ( 1 + 2 ∥ Z ∥ 2 M ) c b b M ( n − 1 / 2 ) . r_\mu((t+2Z)/\sqrt{n})\le b_M\left(\left(1+\frac{2\lVert Z\rVert_2}{M}\right)/\sqrt{n}\right)\le C_b\left(1+\frac{2\lVert Z\rVert_2}{M}\right)^{c_b}b_M(n^{-1/2}). r μ (( t + 2 Z ) / n ) ≤ b M ( ( 1 + M 2 ∥ Z ∥ 2 ) / n ) ≤ C b ( 1 + M 2 ∥ Z ∥ 2 ) c b b M ( n − 1/2 ) .
Gaussian random vectors have finite moments of every fixed order. Taking expectations in (C.6), we obtain
sup t ∈ B 2 ( M ) E μ n , t , Σ R ( w ∗ ( x ˉ ) ) ≤ b M ( n − 1 / 2 ) [ 1 + C b E ( 1 + 2 ∥ Z ∥ 2 M ) c b ] . \sup_{t\in B_2(M)}\mathbb{E}_{\mu_{n,t},\Sigma}R(w_*(\bar{x}))\le b_M(n^{-1/2})\left[1+C_b\mathbb{E}\left(1+\frac{2\lVert Z\rVert_2}{M}\right)^{c_b}\right]. t ∈ B 2 ( M ) sup E μ n , t , Σ R ( w ∗ ( x ˉ )) ≤ b M ( n − 1/2 ) [ 1 + C b E ( 1 + M 2 ∥ Z ∥ 2 ) c b ] .
By (C.3), for sufficiently large n n n , b M ( n − 1 / 2 ) ≤ 2 a ( n − 1 / 2 ) sup t ∈ B 2 ( M ) ψ ( t ) b_M(n^{-1/2}) \le2a(n^{-1/2})\sup_{t\in B_2(M)}\psi(t) b M ( n − 1/2 ) ≤ 2 a ( n − 1/2 ) sup t ∈ B 2 ( M ) ψ ( t ) . Therefore, taking C M = 2 sup t ∈ B 2 ( M ) ψ ( t ) [ 1 + C b E ( 1 + 2 ∥ Z ∥ 2 / M ) c b ] < ∞ C_M = 2\sup_{t\in B_2(M)}\psi(t)\left[1+C_b\mathbb{E}\left(1+2\lVert Z\rVert_2/M\right)^{c_b}\right] < \infty C M = 2 sup t ∈ B 2 ( M ) ψ ( t ) [ 1 + C b E ( 1 + 2 ∥ Z ∥ 2 / M ) c b ] < ∞ , we have
sup t ∈ B 2 ( M ) E μ n , t , Σ R ( w ∗ ( x ˉ ) ) ≤ C M a ( n − 1 / 2 ) . \sup_{t\in B_2(M)} \mathbb{E}_{\mu_{n,t},\Sigma}R(w_*(\bar{x})) \le C_Ma(n^{-1/2}). t ∈ B 2 ( M ) sup E μ n , t , Σ R ( w ∗ ( x ˉ )) ≤ C M a ( n − 1/2 ) .
Finally, (27) gives the lower bound with c M = V M , Σ ( ψ ) / 2 c_M=V_{M,\Sigma}(\psi)/2 c M = V M , Σ ( ψ ) /2 for large n n n , and the definition of R n , M Σ R^\Sigma_{n,M} R n , M Σ gives the middle inequality in (28) .
The right side of (29) is independent of Σ \Sigma Σ , proving the asserted uniform convergence. Writing Z = Σ 1 / 2 G Z=\Sigma^{1/2}G Z = Σ 1/2 G with G ∼ N ( 0 , I d ) G\sim\mathcal{N}(0,I_d) G ∼ N ( 0 , I d ) gives
sup Σ ∈ C E ( 1 + 2 ∥ Z ∥ 2 / M ) c b ≤ E ( 1 + 2 λ ‾ ∥ G ∥ 2 / M ) c b < ∞ . \sup_{\Sigma\in\mathcal{C}}\mathbb{E}\left(1+2\lVert Z\rVert_2/M\right)^{c_b}
\le\mathbb{E}\left(1+2\sqrt{\overline{\lambda}}\lVert G\rVert_2/M\right)^{c_b}<\infty. Σ ∈ C sup E ( 1 + 2 ∥ Z ∥ 2 / M ) c b ≤ E ( 1 + 2 λ ∥ G ∥ 2 / M ) c b < ∞.
Together with the uniform positive lower bound just proved, this permits c M , C M c_M,C_M c M , C M in (28) to be chosen uniformly over Σ ∈ C \Sigma\in\mathcal{C} Σ ∈ C .
Proof of Corollary 4.1
Proof. For each fixed t t t and ε > 0 \varepsilon>0 ε > 0 , ∀ w ∈ W ∗ ( μ ) \forall w\in W_*(\mu) ∀ w ∈ W ∗ ( μ ) , optimality gives
f ∗ ( μ + ε t ) − f ∗ ( μ ) ≥ ( μ + ε t ) ⊤ w − μ ⊤ w = ε t ⊤ w , f_*(\mu+\varepsilon t)-f_*(\mu)\ge(\mu+\varepsilon t)^\top w-\mu^\top w=\varepsilon t^\top w, f ∗ ( μ + εt ) − f ∗ ( μ ) ≥ ( μ + εt ) ⊤ w − μ ⊤ w = ε t ⊤ w ,
and thus
sup w ∈ W ∗ ( μ ) w ⊤ t ≤ f ∗ ( μ + ε t ) − f ∗ ( μ ) ε ≤ w ∗ ( μ + ε t ) ⊤ t . (84) \sup_{w\in W_*(\mu)}w^\top t\le\frac{f_*(\mu+\varepsilon t)-f_*(\mu)}{\varepsilon}\le w_*(\mu+\varepsilon t)^\top t. \tag*{(84)} w ∈ W ∗ ( μ ) sup w ⊤ t ≤ ε f ∗ ( μ + εt ) − f ∗ ( μ ) ≤ w ∗ ( μ + εt ) ⊤ t . (84)
Take any sequence ε k ↓ 0 \varepsilon_k\downarrow0 ε k ↓ 0 . By compactness of Ω \Omega Ω , there is a subsequence, still indexed by k k k , such that
w ∗ ( μ + ε k t ) ⟶ w ‾ ∈ Ω . w_*(\mu+\varepsilon_k t)\longrightarrow\overline{w}\in\Omega. w ∗ ( μ + ε k t ) ⟶ w ∈ Ω.
Optimality at μ + ε k t \mu+\varepsilon_k t μ + ε k t gives
μ ⊤ w ∗ ( μ + ε k t ) = f ∗ ( μ + ε k t ) − ε k t ⊤ w ∗ ( μ + ε k t ) ⟶ f ∗ ( μ ) , \mu^\top w_*(\mu+\varepsilon_k t)=f_*(\mu+\varepsilon_k t)-\varepsilon_k t^\top w_*(\mu+\varepsilon_k t)\longrightarrow f_*(\mu), μ ⊤ w ∗ ( μ + ε k t ) = f ∗ ( μ + ε k t ) − ε k t ⊤ w ∗ ( μ + ε k t ) ⟶ f ∗ ( μ ) ,
where the first term converges by continuity of f ∗ f_* f ∗ and the second term tends to zero because Ω \Omega Ω is bounded. Thus μ ⊤ w ‾ = f ∗ ( μ ) \mu^\top\overline{w}=f_*(\mu) μ ⊤ w = f ∗ ( μ ) , so w ‾ ∈ W ∗ ( μ ) \overline{w}\in W_*(\mu) w ∈ W ∗ ( μ ) . This argument applies to every convergent subsequence of the perturbed maximizers. In particular,
lim sup ε ↓ 0 w ∗ ( μ + ε t ) ⊤ t ≤ sup w ∈ W ∗ ( μ ) w ⊤ t . \limsup_{\varepsilon\downarrow0}w_*(\mu+\varepsilon t)^\top t\le\sup_{w\in W_*(\mu)}w^\top t. ε ↓ 0 lim sup w ∗ ( μ + εt ) ⊤ t ≤ w ∈ W ∗ ( μ ) sup w ⊤ t .
Indeed, choose a sequence realizing the limsup and extract a convergent subsequence of its maximizers; its limit belongs to W ∗ ( μ ) W_*(\mu) W ∗ ( μ ) . Combining this inequality with (C.7), we have
sup w ∈ W ∗ ( μ ) w ⊤ t ≤ lim inf ε ↓ 0 f ∗ ( μ + ε t ) − f ∗ ( μ ) ε ≤ lim sup ε ↓ 0 f ∗ ( μ + ε t ) − f ∗ ( μ ) ε ≤ lim sup ε ↓ 0 w ∗ ( μ + ε t ) ⊤ t ≤ sup w ∈ W ∗ ( μ ) w ⊤ t , \begin{aligned}
\sup_{w\in W_*(\mu)}w^\top t
\le\liminf_{\varepsilon\downarrow0}\frac{f_*(\mu+\varepsilon t)-f_*(\mu)}{\varepsilon}
\le\limsup_{\varepsilon\downarrow0}\frac{f_*(\mu+\varepsilon t)-f_*(\mu)}{\varepsilon} \\
&\le\limsup_{\varepsilon\downarrow0}w_*(\mu+\varepsilon t)^\top t
\le\sup_{w\in W_*(\mu)}w^\top t,
\end{aligned} w ∈ W ∗ ( μ ) sup w ⊤ t ≤ ε ↓ 0 lim inf ε f ∗ ( μ + εt ) − f ∗ ( μ ) ≤ ε ↓ 0 lim sup ε f ∗ ( μ + εt ) − f ∗ ( μ ) ≤ ε ↓ 0 lim sup w ∗ ( μ + εt ) ⊤ t ≤ w ∈ W ∗ ( μ ) sup w ⊤ t ,
and thus
lim ε ↓ 0 f ∗ ( μ + ε t ) − f ∗ ( μ ) ε = sup w ∈ W ∗ ( μ ) w T t . \lim_{\varepsilon\downarrow0}\frac{f_*(\mu+\varepsilon t)-f_*(\mu)}{\varepsilon}
=
\sup_{w\in W_*(\mu)} w^{\mathsf T}t. ε ↓ 0 lim ε f ∗ ( μ + εt ) − f ∗ ( μ ) = w ∈ W ∗ ( μ ) sup w T t .
Subtracting w ∗ ( μ ) T t w_*(\mu)^{\mathsf T}t w ∗ ( μ ) T t proves
lim ε ↓ 0 r μ ( ε t ) ε = lim ε ↓ 0 f ∗ ( μ + ε t ) − f ∗ ( μ ) ε − w ∗ ( μ ) T t = sup w ∈ W ∗ ( μ ) w T t − w ∗ ( μ ) T t = ψ 1 ( t ) . \lim_{\varepsilon\downarrow0}\frac{r_\mu(\varepsilon t)}{\varepsilon}
=
\lim_{\varepsilon\downarrow0}\frac{f_*(\mu+\varepsilon t)-f_*(\mu)}{\varepsilon}
-w_*(\mu)^{\mathsf T}t
=
\sup_{w\in W_*(\mu)}w^{\mathsf T}t-w_*(\mu)^{\mathsf T}t
=\psi_1(t). ε ↓ 0 lim ε r μ ( εt ) = ε ↓ 0 lim ε f ∗ ( μ + εt ) − f ∗ ( μ ) − w ∗ ( μ ) T t = w ∈ W ∗ ( μ ) sup w T t − w ∗ ( μ ) T t = ψ 1 ( t ) .
Proposition C.1 makes this convergence uniform on each B 2 ( M ) B_2(M) B 2 ( M ) , and Theorem 3.3 gives (33) . If W ∗ ( μ ) W_*(\mu) W ∗ ( μ ) is a singleton, then ψ 1 ≡ 0 \psi_1\equiv0 ψ 1 ≡ 0 . Otherwise, choosing w ∈ W ∗ ( μ ) ∖ { w ∗ ( μ ) } w\in W_*(\mu)\setminus\{w_*(\mu)\} w ∈ W ∗ ( μ ) ∖ { w ∗ ( μ )} gives ψ 1 ( w − w ∗ ( μ ) ) > 0 \psi_1(w-w_*(\mu))>0 ψ 1 ( w − w ∗ ( μ )) > 0 . Since ψ 1 \psi_1 ψ 1 is nonnegative, vanishes at zero, and is positively homogeneous, it cannot be affine on any B 2 ( M ) B_2(M) B 2 ( M ) with M > 0 M>0 M > 0 . The positivity assertion follows from the theorem.
Proof of Proposition 3.4
Proof. By the convexity of f ∗ f_* f ∗ ,
f ∗ ( μ + ε t ) + f ∗ ( μ − ε t ) 2 ≥ f ∗ ( μ ) , \frac{f_*(\mu+\varepsilon t)+f_*(\mu-\varepsilon t)}{2}\ge f_*(\mu), 2 f ∗ ( μ + εt ) + f ∗ ( μ − εt ) ≥ f ∗ ( μ ) ,
which yields D μ ( ε , M ) ≥ 0 D_\mu(\varepsilon,M)\ge0 D μ ( ε , M ) ≥ 0 . By continuity of f ∗ f_* f ∗ , since B 2 ( M ) B_2(M) B 2 ( M ) is compact, a maximizer exists. Since r μ ≥ 0 r_\mu\ge0 r μ ≥ 0 ,
0 ≤ r μ ( ε t ) ≤ r μ ( ε t ) + r μ ( − ε t ) ≤ 2 D μ ( ε , M ) , t ∈ B 2 ( M ) . 0\le r_\mu(\varepsilon t)\le r_\mu(\varepsilon t)+r_\mu(-\varepsilon t)\le2D_\mu(\varepsilon,M),\qquad t\in B_2(M). 0 ≤ r μ ( εt ) ≤ r μ ( εt ) + r μ ( − εt ) ≤ 2 D μ ( ε , M ) , t ∈ B 2 ( M ) .
Lemma 3.1 gives
R n , M Σ ( μ ) ≤ sup π ∈ P ( B 2 ( M ) ) E π , Σ r μ ( T / n ) ≤ sup t ∈ B 2 ( M ) r μ ( t / n ) ≤ 2 D μ ( n − 1 / 2 , M ) . R_{n,M}^{\Sigma}(\mu)
\le
\sup_{\pi\in\mathcal{P}(B_2(M))}\mathbb{E}_{\pi,\Sigma}r_\mu(T/\sqrt{n})
\le
\sup_{t\in B_2(M)}r_\mu(t/\sqrt{n})
\le
2D_\mu(n^{-1/2},M). R n , M Σ ( μ ) ≤ π ∈ P ( B 2 ( M )) sup E π , Σ r μ ( T / n ) ≤ t ∈ B 2 ( M ) sup r μ ( t / n ) ≤ 2 D μ ( n − 1/2 , M ) .
This proves the upper bound.
For the lower bound, suppose D μ ( n − 1 / 2 , M ) > 0 D_\mu(n^{-1/2},M)>0 D μ ( n − 1/2 , M ) > 0 and choose a maximizer t 0 ∈ B 2 ( M ) t_0\in B_2(M) t 0 ∈ B 2 ( M ) in (30) . Then t 0 ≠ 0 t_0\ne0 t 0 = 0 . Use the prior
P π { T = t 0 } = P π { T = − t 0 } = 1 2 , Y = T + Z , q ( Y ) : = P π , Σ { T = t 0 ∣ Y } . \mathbb{P}_\pi\{T=t_0\}=\mathbb{P}_\pi\{T=-t_0\}=\frac{1}{2},
\qquad
Y=T+Z,
\qquad
q(Y):=\mathbb{P}_{\pi,\Sigma}\{T=t_0\mid Y\}. P π { T = t 0 } = P π { T = − t 0 } = 2 1 , Y = T + Z , q ( Y ) := P π , Σ { T = t 0 ∣ Y } .
For q ∈ [ 0 , 1 ] q\in[0,1] q ∈ [ 0 , 1 ] , set
g ( q ) : = q r μ ( t 0 / n ) + ( 1 − q ) r μ ( − t 0 / n ) − r μ ( ( 2 q − 1 ) t 0 / n ) , g(q):=qr_\mu(t_0/\sqrt{n})+(1-q)r_\mu(-t_0/\sqrt{n})-r_\mu((2q-1)t_0/\sqrt{n}), g ( q ) := q r μ ( t 0 / n ) + ( 1 − q ) r μ ( − t 0 / n ) − r μ (( 2 q − 1 ) t 0 / n ) ,
g ( 0 ) = g ( 1 ) = 0 , g ( 1 / 2 ) = D μ ( n − 1 / 2 , M ) . g(0)=g(1)=0,\qquad g(1/2)=D_\mu(n^{-1/2},M). g ( 0 ) = g ( 1 ) = 0 , g ( 1/2 ) = D μ ( n − 1/2 , M ) .
Convexity of r μ r_\mu r μ makes g g g concave. Applying concavity on [ 0 , 1 / 2 ] [0,1/2] [ 0 , 1/2 ] and [ 1 / 2 , 1 ] [1/2,1] [ 1/2 , 1 ] gives
g ( q ) ≥ 2 min { q , 1 − q } D μ ( n − 1 / 2 , M ) . g(q)\ge2\min\{q,1-q\}D_\mu(n^{-1/2},M). g ( q ) ≥ 2 min { q , 1 − q } D μ ( n − 1/2 , M ) .
The likelihood ratio between N ( t 0 , Σ ) \mathcal{N}(t_0,\Sigma) N ( t 0 , Σ ) and N ( − t 0 , Σ ) \mathcal{N}(-t_0,\Sigma) N ( − t 0 , Σ ) is exp ( 2 t 0 T Σ − 1 Y ) \exp(2t_0^{\mathsf{T}}\Sigma^{-1}Y) exp ( 2 t 0 T Σ − 1 Y ) . Thus the Bayes-optimal test chooses t 0 t_0 t 0 when t 0 T Σ − 1 Y > 0 t_0^{\mathsf{T}}\Sigma^{-1}Y>0 t 0 T Σ − 1 Y > 0 , and
E π , Σ min { q ( Y ) , 1 − q ( Y ) } = 1 2 P π , Σ { t 0 T Σ − 1 Y ≤ 0 ∣ T = t 0 } + 1 2 P π , Σ { t 0 T Σ − 1 Y > 0 ∣ T = − t 0 } = Φ ( − ∥ Σ − 1 / 2 t 0 ∥ 2 ) ≥ Φ ( − M / λ min ( Σ ) ) . \mathbb{E}_{\pi,\Sigma}\min\{q(Y),1-q(Y)\}=\frac{1}{2}\mathbb{P}_{\pi,\Sigma}\{t_0^{\mathsf{T}}\Sigma^{-1}Y\leq0\mid T=t_0\}+\frac{1}{2}\mathbb{P}_{\pi,\Sigma}\{t_0^{\mathsf{T}}\Sigma^{-1}Y>0\mid T=-t_0\}
=\Phi(-\|\Sigma^{-1/2}t_0\|_2)\geq\Phi(-M/\sqrt{\lambda_{\min}(\Sigma)}). E π , Σ min { q ( Y ) , 1 − q ( Y )} = 2 1 P π , Σ { t 0 T Σ − 1 Y ≤ 0 ∣ T = t 0 } + 2 1 P π , Σ { t 0 T Σ − 1 Y > 0 ∣ T = − t 0 } = Φ ( − ∥ Σ − 1/2 t 0 ∥ 2 ) ≥ Φ ( − M / λ m i n ( Σ ) ) .
Since
m π , Σ ( Y ) = ( 2 q ( Y ) − 1 ) t 0 m_{\pi,\Sigma}(Y)=(2q(Y)-1)t_0 m π , Σ ( Y ) = ( 2 q ( Y ) − 1 ) t 0
and
E T , Σ [ r μ ( T / n ) ∣ Y ] = q ( Y ) r μ ( t 0 / n ) + ( 1 − q ( Y ) ) r μ ( − t 0 / n ) , \mathbb{E}_{T,\Sigma}[r_\mu(T/\sqrt{n})\mid Y]=q(Y)r_\mu(t_0/\sqrt{n})+(1-q(Y))r_\mu(-t_0/\sqrt{n}), E T , Σ [ r μ ( T / n ) ∣ Y ] = q ( Y ) r μ ( t 0 / n ) + ( 1 − q ( Y )) r μ ( − t 0 / n ) ,
Lemma 3.1 yields
R n , M Σ ( μ ) ≥ E π , Σ [ r μ ( T / n ) − r μ ( m π , Σ ( Y ) / n ) ] = E π , Σ g ( q ( Y ) ) ≥ 2 Φ ( − M / λ min ( Σ ) ) D μ ( n − 1 / 2 , M ) . R^\Sigma_{n,M}(\mu)\geq\mathbb{E}_{\pi,\Sigma}\left[r_\mu(T/\sqrt{n})-r_\mu(m_{\pi,\Sigma}(Y)/\sqrt{n})\right]
=\mathbb{E}_{\pi,\Sigma}g(q(Y))\geq2\Phi(-M/\sqrt{\lambda_{\min}(\Sigma)})D_\mu(n^{-1/2},M). R n , M Σ ( μ ) ≥ E π , Σ [ r μ ( T / n ) − r μ ( m π , Σ ( Y ) / n ) ] = E π , Σ g ( q ( Y )) ≥ 2Φ ( − M / λ m i n ( Σ ) ) D μ ( n − 1/2 , M ) .
If D μ ( n − 1 / 2 , M ) = 0 D_\mu(n^{-1/2},M)=0 D μ ( n − 1/2 , M ) = 0 , the result follows directly from nonnegativity of regret.
Since λ min ( Σ ) ≥ λ \lambda_{\min}(\Sigma)\geq\lambda λ m i n ( Σ ) ≥ λ , the two bounds give, for every Σ ∈ C \Sigma\in\mathcal{C} Σ ∈ C ,
2 Φ ( − M / λ ) D μ ( n − 1 / 2 , M ) ≤ R n , M Σ ( μ ) ≤ 2 D μ ( n − 1 / 2 , M ) , 2\Phi(-M/\sqrt{\lambda})D_\mu(n^{-1/2},M)\leq R^\Sigma_{n,M}(\mu)\leq2D_\mu(n^{-1/2},M), 2Φ ( − M / λ ) D μ ( n − 1/2 , M ) ≤ R n , M Σ ( μ ) ≤ 2 D μ ( n − 1/2 , M ) ,
so that the comparison factor between the two sides is finite, independent of n n n and Σ \Sigma Σ , for fixed M M M and λ > 0 \lambda>0 λ > 0 . The inequalities also apply when both risks vanish.
Distance normalization for general scales
Corollary C.2 (Distance normalization under a general local scale). Suppose that Ω \Omega Ω contains at least two points, ∥ ⋅ ∥ K \|\cdot\|_K ∥ ⋅ ∥ K is a fixed norm, and dist ( μ , A ) > 0 \operatorname{dist}(\mu,A)>0 dist ( μ , A ) > 0 . Use ν μ = μ / dist ( μ , A ) \nu_\mu=\mu/\operatorname{dist}(\mu,A) ν μ = μ / dist ( μ , A ) as in (34) . For every ε , M > 0 \varepsilon,M>0 ε , M > 0 ,
r μ ( ε t ) = dist ( μ , A ) r ν μ ( ε dist ( μ , A ) t ) , r_\mu(\varepsilon t)=\operatorname{dist}(\mu,A)r_{\nu_\mu}\left(\frac{\varepsilon}{\operatorname{dist}(\mu,A)}t\right), r μ ( εt ) = dist ( μ , A ) r ν μ ( dist ( μ , A ) ε t ) ,
D μ ( ε , M ) = dist ( μ , A ) D ν μ ( ε dist ( μ , A ) , M ) . (85) D_\mu(\varepsilon,M)=\operatorname{dist}(\mu,A)D_{\nu_\mu}\left(\frac{\varepsilon}{\operatorname{dist}(\mu,A)},M\right). \tag*{(85)} D μ ( ε , M ) = dist ( μ , A ) D ν μ ( dist ( μ , A ) ε , M ) . (85)
In particular, the finite-sample bounds become
2 Φ ( − M / λ min ( Σ ) ) dist ( μ , A ) D ν μ ( 1 n dist ( μ , A ) , M ) ≤ R n , M Σ ( μ ) ≤ 2 dist ( μ , A ) D ν μ ( 1 n dist ( μ , A ) , M ) . (86) 2\Phi(-M/\sqrt{\lambda_{\min}(\Sigma)})\operatorname{dist}(\mu,A)D_{\nu_\mu}\left(\frac{1}{\sqrt{n}\operatorname{dist}(\mu,A)},M\right)\leq R^\Sigma_{n,M}(\mu)
\leq2\operatorname{dist}(\mu,A)D_{\nu_\mu}\left(\frac{1}{\sqrt{n}\operatorname{dist}(\mu,A)},M\right). \tag*{(86)} 2Φ ( − M / λ m i n ( Σ ) ) dist ( μ , A ) D ν μ ( n dist ( μ , A ) 1 , M ) ≤ R n , M Σ ( μ ) ≤ 2 dist ( μ , A ) D ν μ ( n dist ( μ , A ) 1 , M ) . (86)
Suppose, in addition, that there exist a ( ε ) > 0 a(\varepsilon)>0 a ( ε ) > 0 with a ( ε ) → 0 a(\varepsilon)\to0 a ( ε ) → 0 and a finite continuous convex ψ ν μ : R d → R \psi_{\nu_\mu}:\mathbb{R}^{d}\to\mathbb{R} ψ ν μ : R d → R such that, for every fixed M > 0 M>0 M > 0 ,
sup t ∈ B 2 ( M ) ∣ r ν μ ( ε t ) a ( ε ) − ψ ν μ ( t ) ∣ → 0 as ε ↓ 0. (87) \sup_{t\in B_{2}(M)}\left|\frac{r_{\nu_\mu}(\varepsilon t)}{a(\varepsilon)}-\psi_{\nu_\mu}(t)\right|\to0\qquad\text{as }\varepsilon\downarrow0. \tag*{(87)} t ∈ B 2 ( M ) sup a ( ε ) r ν μ ( εt ) − ψ ν μ ( t ) → 0 as ε ↓ 0. (87)
For fixed d , Ω , μ , M d,\Omega,\mu,M d , Ω , μ , M ,
lim n → ∞ R n , M Σ ( μ ) dist ( μ , A ) a ( ( n dist ( μ , A ) ) − 1 ) = V M , Σ ( ψ ν μ ) . (88) \lim_{n\to\infty}\frac{R^{\Sigma}_{n,M}(\mu)}{\operatorname{dist}(\mu,A)a\left((\sqrt{n}\operatorname{dist}(\mu,A))^{-1}\right)}=V_{M,\Sigma}(\psi_{\nu_\mu}). \tag*{(88)} n → ∞ lim dist ( μ , A ) a ( ( n dist ( μ , A ) ) − 1 ) R n , M Σ ( μ ) = V M , Σ ( ψ ν μ ) . (88)
If ψ ν μ \psi_{\nu_\mu} ψ ν μ is nonaffine on B 2 ( M ) B_{2}(M) B 2 ( M ) , then V M , Σ ( ψ ν μ ) > 0 V_{M,\Sigma}(\psi_{\nu_\mu})>0 V M , Σ ( ψ ν μ ) > 0 , and the plug-in policy attains the corresponding rate. In particular, for a ( ε ) = ε α a(\varepsilon)=\varepsilon^\alpha a ( ε ) = ε α ,
lim n → ∞ n α / 2 R n , M Σ ( μ ) = dist ( μ , A ) 1 − α V M , Σ ( ψ ν μ ) . (89) \lim_{n\to\infty}n^{\alpha/2}R^{\Sigma}_{n,M}(\mu)=\operatorname{dist}(\mu,A)^{1-\alpha}V_{M,\Sigma}(\psi_{\nu_\mu}). \tag*{(89)} n → ∞ lim n α /2 R n , M Σ ( μ ) = dist ( μ , A ) 1 − α V M , Σ ( ψ ν μ ) . (89)
Proof. Positive homogeneity gives dist ( ν μ , A ) = 1 \operatorname{dist}(\nu_\mu,A)=1 dist ( ν μ , A ) = 1 and w ∗ ( ν μ ) = w ∗ ( μ ) \mathbf{w}_{*}(\nu_\mu)=\mathbf{w}_{*}(\mu) w ∗ ( ν μ ) = w ∗ ( μ ) . Hence
r μ ( ε t ) = dist ( μ , A ) [ f ∗ ( ν μ + ε t dist ( μ , A ) ) − f ∗ ( ν μ ) − ε w ∗ ( ν μ ) T t dist ( μ , A ) ] = dist ( μ , A ) r ν μ ( ε t / dist ( μ , A ) ) . \begin{aligned}
r_{\mu}(\varepsilon t)
&=\operatorname{dist}(\mu,A)\left[f_{*}\left(\nu_\mu+\frac{\varepsilon t}{\operatorname{dist}(\mu,A)}\right)-f_{*}(\nu_\mu)-\frac{\varepsilon\mathbf{w}_{*}(\nu_\mu)^{\mathsf T}t}{\operatorname{dist}(\mu,A)}\right]\\
&=\operatorname{dist}(\mu,A)r_{\nu_\mu}\left(\varepsilon t/\operatorname{dist}(\mu,A)\right).
\end{aligned} r μ ( εt ) = dist ( μ , A ) [ f ∗ ( ν μ + dist ( μ , A ) εt ) − f ∗ ( ν μ ) − dist ( μ , A ) ε w ∗ ( ν μ ) T t ] = dist ( μ , A ) r ν μ ( εt / dist ( μ , A ) ) .
Taking symmetric differences proves (C.8), and (31) yields (C.9).
By (C.8) and (C.10), for every fixed M > 0 M>0 M > 0 ,
sup t ∈ B 2 ( M ) ∣ r μ ( ε t ) dist ( μ , A ) a ( ε / dist ( μ , A ) ) − ψ ν μ ( t ) ∣ → 0. \sup_{t\in B_{2}(M)}\left|\frac{r_{\mu}(\varepsilon t)}{\operatorname{dist}(\mu,A)a(\varepsilon/\operatorname{dist}(\mu,A))}-\psi_{\nu_\mu}(t)\right|\to0. t ∈ B 2 ( M ) sup dist ( μ , A ) a ( ε / dist ( μ , A )) r μ ( εt ) − ψ ν μ ( t ) → 0.
Apply Theorem 3.3 with scale ε ↦ dist ( μ , A ) a ( ε / dist ( μ , A ) ) \varepsilon\mapsto\operatorname{dist}(\mu,A)a(\varepsilon/\operatorname{dist}(\mu,A)) ε ↦ dist ( μ , A ) a ( ε / dist ( μ , A )) to obtain (C.11), positivity, and attainment of the rate by the plug-in policy. For a ( ε ) = ε α a(\varepsilon)=\varepsilon^\alpha a ( ε ) = ε α , dist ( μ , A ) a ( ( n dist ( μ , A ) ) − 1 ) = dist ( μ , A ) 1 − α n − α / 2 \operatorname{dist}(\mu,A)a((\sqrt{n}\operatorname{dist}(\mu,A))^{-1})=\operatorname{dist}(\mu,A)^{1-\alpha}n^{-\alpha/2} dist ( μ , A ) a (( n dist ( μ , A ) ) − 1 ) = dist ( μ , A ) 1 − α n − α /2 , which proves (C.12).
Remark C.3 (What the distance does and does not determine)
Let C K : = sup ∥ t ∥ 2 ≤ 1 ∥ t ∥ K C_{K}:=\sup_{\lVert t\rVert_{2}\leq1}\lVert t\rVert_{K} C K := sup ∥ t ∥ 2 ≤ 1 ∥ t ∥ K . The distance is 1-Lipschitz in its defining norm, so
sup t ∈ B 2 ( M ) ∣ dist ( μ n , t , A ) − dist ( μ , A ) ∣ ≤ C K M n . \sup_{t\in B_{2}(M)}\left|\operatorname{dist}(\mu_{n,t},A)-\operatorname{dist}(\mu,A)\right|\leq\frac{C_{K}M}{\sqrt{n}}. t ∈ B 2 ( M ) sup ∣ dist ( μ n , t , A ) − dist ( μ , A ) ∣ ≤ n C K M .
For fixed dist ( μ , A ) > 0 \operatorname{dist}(\mu,A)>0 dist ( μ , A ) > 0 , the relative variation tends to zero, but the normalized curvature in (36) and the limiting function in (C.11) still depend on the direction of μ \mu μ and on Ω \Omega Ω .
Without a local expansion, (C.9) provides the following criterion for each fixed M > 0 M>0 M > 0 :
lim inf ε ↓ 0 D ν μ ( ε , M ) ε 2 > 0 ⟹ lim inf n → ∞ n dist ( μ , A ) R n , M Σ ( μ ) > 0. \liminf_{\varepsilon\downarrow0}\frac{D_{\nu_\mu}(\varepsilon,M)}{\varepsilon^{2}}>0\quad\Longrightarrow\quad\liminf_{n\to\infty}n\operatorname{dist}(\mu,A)R^{\Sigma}_{n,M}(\mu)>0. ε ↓ 0 lim inf ε 2 D ν μ ( ε , M ) > 0 ⟹ n → ∞ lim inf n dist ( μ , A ) R n , M Σ ( μ ) > 0.
If this ratio tends to zero, then R n , M Σ ( μ ) = o ( 1 / [ n dist ( μ , A ) ] ) R^{\Sigma}_{n,M}(\mu)=o(1/[n\operatorname{dist}(\mu,A)]) R n , M Σ ( μ ) = o ( 1/ [ n dist ( μ , A )]) . If its liminf is positive and its limsup is finite, then R n , M Σ ( μ ) ≍ 1 / [ n dist ( μ , A ) ] R^{\Sigma}_{n,M}(\mu)\asymp1/[n\operatorname{dist}(\mu,A)] R n , M Σ ( μ ) ≍ 1/ [ n dist ( μ , A )] for fixed $d,\Omega,\mu,M.
When dist ( μ , A ) = 0 \operatorname{dist}(\mu,A)=0 dist ( μ , A ) = 0 , reciprocal-distance formulas do not apply. Corollary 4.1 instead gives
lim n → ∞ n R n , M Σ ( μ ) = V M , Σ ( ψ 1 ) , V M , Σ ( ψ 1 ) > 0 ⟺ μ ∈ A , M > 0. \lim_{n\to\infty}\sqrt{n}R^\Sigma_{n,M}(\mu)=V_{M,\Sigma}(\psi_1),\qquad V_{M,\Sigma}(\psi_1)>0\Longleftrightarrow\mu\in A,\quad M>0. n → ∞ lim n R n , M Σ ( μ ) = V M , Σ ( ψ 1 ) , V M , Σ ( ψ 1 ) > 0 ⟺ μ ∈ A , M > 0.
In particular, μ ∈ A ‾ ∖ A \mu\in\overline{A}\setminus A μ ∈ A ∖ A implies R n , M Σ ( μ ) = o ( n − 1 / 2 ) R^\Sigma_{n,M}(\mu)=o(n^{-1/2}) R n , M Σ ( μ ) = o ( n − 1/2 ) .
The lower spectral bound is substantive. With singular covariance, tr ( Σ H μ ) \operatorname{tr}(\Sigma H_\mu) tr ( Σ H μ ) can vanish even when H μ ≠ 0 H_\mu\ne0 H μ = 0 , and the full-support Gaussian testing argument for positive V M , Σ V_{M,\Sigma} V M , Σ can fail.
Finite-sample plug-in bounds
Lemma C.4 (A finite-sample plug-in bound through symmetric differences). Let Ω \Omega Ω be compact and convex, let μ ∈ R d \mu\in\mathbb{R}^d μ ∈ R d , and let M > 0 M>0 M > 0 . For G ∼ N ( 0 , I d ) G\sim\mathcal{N}(0,I_d) G ∼ N ( 0 , I d ) , every n ≥ 2 n\ge2 n ≥ 2 , and every Σ ∈ C \Sigma\in\mathcal{C} Σ ∈ C ,
sup t ∈ B 2 ( M ) E μ , n , t , Σ R ( w ∗ ( x ˉ ) ) ≤ 2 E D μ ( n − 1 / 2 , M + 2 ∥ Σ 1 / 2 G ∥ 2 ) \sup_{t\in B_2(M)}\mathbb{E}_{\mu,n,t,\Sigma}R(w_*(\bar{x}))\le2\mathbb{E}D_\mu(n^{-1/2},M+2\lVert\Sigma^{1/2}G\rVert_2) t ∈ B 2 ( M ) sup E μ , n , t , Σ R ( w ∗ ( x ˉ )) ≤ 2 E D μ ( n − 1/2 , M + 2 ∥ Σ 1/2 G ∥ 2 ) ≤ 2 E D μ ( n − 1 / 2 , M + 2 λ ∥ G ∥ 2 ) . (90) \le2\mathbb{E}D_\mu(n^{-1/2},M+2\sqrt{\lambda}\lVert G\rVert_2). \tag*{(90)} ≤ 2 E D μ ( n − 1/2 , M + 2 λ ∥ G ∥ 2 ) . (90) The last expression also bounds the supremum over Σ ∈ C \Sigma\in\mathcal{C} Σ ∈ C on the left.
Proof. Denote Z = Σ 1 / 2 G Z=\Sigma^{1/2}G Z = Σ 1/2 G , so x ˉ = μ + ( t + Z ) / n \bar{x}=\mu+(t+Z)/\sqrt{n} x ˉ = μ + ( t + Z ) / n . Optimality at x ˉ \bar{x} x ˉ gives
R ( w ∗ ( x ˉ ) ) = r μ ( t / n ) − r μ ( ( t + Z ) / n ) + ( w ∗ ( x ˉ ) − w ∗ ( μ ) ) ⊤ Z / n , R(w_*(\bar{x}))=r_\mu(t/\sqrt{n})-r_\mu((t+Z)/\sqrt{n})+(w_*(\bar{x})-w_*(\mu))^\top Z/\sqrt{n}, R ( w ∗ ( x ˉ )) = r μ ( t / n ) − r μ (( t + Z ) / n ) + ( w ∗ ( x ˉ ) − w ∗ ( μ ) ) ⊤ Z / n ,
and the subgradient inequality for r μ r_\mu r μ gives
r μ ( ( t + 2 Z ) / n ) ≥ r μ ( ( t + Z ) / n ) + ( w ∗ ( x ˉ ) − w ∗ ( μ ) ) ⊤ Z / n . r_\mu((t+2Z)/\sqrt{n})\ge r_\mu((t+Z)/\sqrt{n})+(w_*(\bar{x})-w_*(\mu))^\top Z/\sqrt{n}. r μ (( t + 2 Z ) / n ) ≥ r μ (( t + Z ) / n ) + ( w ∗ ( x ˉ ) − w ∗ ( μ ) ) ⊤ Z / n .
Consequently,
R ( w ∗ ( x ˉ ) ) ≤ r μ ( t / n ) + r μ ( ( t + 2 Z ) / n ) − 2 r μ ( ( t + Z ) / n ) . (91) R(w_*(\bar{x}))\le r_\mu(t/\sqrt{n})+r_\mu((t+2Z)/\sqrt{n})-2r_\mu((t+Z)/\sqrt{n}). \tag*{(91)} R ( w ∗ ( x ˉ )) ≤ r μ ( t / n ) + r μ (( t + 2 Z ) / n ) − 2 r μ (( t + Z ) / n ) . (91)
By Jensen’s inequality, convexity of r μ r_\mu r μ and E Z = 0 \mathbb{E}Z=0 E Z = 0 yield
E r μ ( ( t + Z ) / n ) ≥ r μ ( t / n ) , r μ ( t / n ) ≥ 0. \mathbb{E}r_\mu((t+Z)/\sqrt{n})\ge r_\mu(t/\sqrt{n}),\qquad r_\mu(t/\sqrt{n})\ge0. E r μ (( t + Z ) / n ) ≥ r μ ( t / n ) , r μ ( t / n ) ≥ 0.
Taking expectations in (C.14) therefore gives
E μ , n , t , Σ R ( w ∗ ( x ˉ ) ) ≤ E r μ ( ( t + 2 Z ) / n ) − r μ ( t / n ) ≤ E r μ ( ( t + 2 Z ) / n ) \mathbb{E}_{\mu,n,t,\Sigma}R(w_*(\bar{x}))\le\mathbb{E}r_\mu((t+2Z)/\sqrt{n})-r_\mu(t/\sqrt{n})\le\mathbb{E}r_\mu((t+2Z)/\sqrt{n}) E μ , n , t , Σ R ( w ∗ ( x ˉ )) ≤ E r μ (( t + 2 Z ) / n ) − r μ ( t / n ) ≤ E r μ (( t + 2 Z ) / n )
≤ 2 E D μ ( n − 1 / 2 , M + 2 ∥ Z ∥ 2 ) \le2\mathbb{E}D_\mu(n^{-1/2},M+2\lVert Z\rVert_2) ≤ 2 E D μ ( n − 1/2 , M + 2 ∥ Z ∥ 2 )
for t ∈ B 2 ( M ) t\in B_2(M) t ∈ B 2 ( M ) . The last inequality uses 0 ≤ r μ ( ε u ) ≤ 2 D μ ( ε , L ) 0\le r_\mu(\varepsilon u)\le2D_\mu(\varepsilon,L) 0 ≤ r μ ( ε u ) ≤ 2 D μ ( ε , L ) for u ∈ B 2 ( L ) u\in B_2(L) u ∈ B 2 ( L ) , as proved in Proposition 3.4. All expectations are finite because r μ ( h ) ≤ 2 R Ω ∥ h ∥ 2 r_\mu(h)\le2R_\Omega\lVert h\rVert_2 r μ ( h ) ≤ 2 R Ω ∥ h ∥ 2 and, by Lipschitz continuity of f ∗ f_* f ∗ ,
0 ≤ D μ ( ε , L ) ≤ R Ω ε L . 0\le D_\mu(\varepsilon,L)\le R_\Omega\varepsilon L. 0 ≤ D μ ( ε , L ) ≤ R Ω ε L .
Finally, D μ ( ε , L ) D_\mu(\varepsilon,L) D μ ( ε , L ) is nondecreasing in L L L , and ∥ Σ 1 / 2 G ∥ 2 ≤ λ ∥ G ∥ 2 \lVert\Sigma^{1/2}G\rVert_2\le\sqrt{\lambda}\lVert G\rVert_2 ∥ Σ 1/2 G ∥ 2 ≤ λ ∥ G ∥ 2 . Taking suprema over t t t proves (C.13) and its uniform-covariance assertion.
Lemma C.5 (Rate minimaxity under a local doubling condition). Retain the assumptions of Lemma C.4 . Suppose that, for some ε 0 > 0 \varepsilon_0 > 0 ε 0 > 0 and Γ ≥ 2 \Gamma\ge2 Γ ≥ 2 ,
0 < D μ ( ε , M ) , D μ ( 2 ε , M ) ≤ Γ D μ ( ε , M ) , 0 < ε ≤ ε 0 . (92) 0 < D_\mu(\varepsilon, M), \qquad D_\mu(2\varepsilon, M) \le\Gamma D_\mu(\varepsilon, M), \qquad0 < \varepsilon\le\varepsilon_0. \tag*{(92)} 0 < D μ ( ε , M ) , D μ ( 2 ε , M ) ≤ Γ D μ ( ε , M ) , 0 < ε ≤ ε 0 . (92) Then
lim sup n → ∞ sup t ∈ B 2 ( M ) ∑ Σ ∈ C E μ n , t , Σ R ( w ∗ ( x ˉ ) ) D μ ( n − 1 / 2 , M ) ≤ 2 Γ E ( 1 + 2 λ ∥ G ∥ 2 M ) log 2 Γ < ∞ . (93) \limsup_{n\to\infty} \frac{ \sup_{t\in B_2(M)} \sum_{\Sigma\in\mathcal{C}} \mathbb{E}_{\mu_{n,t,\Sigma} }R(w_*(\bar{x}))}{D_\mu(n^{-1/2},M)}
\le2\Gamma\mathbb{E}\left(1+\frac{2\sqrt{\lambda}\lVert G\rVert_2}{M}\right)^{\log_2\Gamma}<\infty. \tag*{(93)} n → ∞ lim sup D μ ( n − 1/2 , M ) sup t ∈ B 2 ( M ) ∑ Σ ∈ C E μ n , t , Σ R ( w ∗ ( x ˉ )) ≤ 2Γ E ( 1 + M 2 λ ∥ G ∥ 2 ) l o g 2 Γ < ∞. (93) In particular, as n → ∞ n\to\infty n → ∞ with M M M fixed,
R n , M Σ ( μ ) ≍ D μ ( n − 1 / 2 , M ) ≍ sup t ∈ B 2 ( M ) E μ n , t , Σ R ( w ∗ ( x ˉ ) ) . (94) R_{n,M}^{\Sigma}(\mu)\asymp D_\mu(n^{-1/2},M)\asymp\sup_{t\in B_2(M)}\mathbb{E}_{\mu_{n,t,\Sigma} }R(w_*(\bar{x})). \tag*{(94)} R n , M Σ ( μ ) ≍ D μ ( n − 1/2 , M ) ≍ t ∈ B 2 ( M ) sup E μ n , t , Σ R ( w ∗ ( x ˉ )) . (94) The comparison constants are uniform over Σ ∈ C \Sigma\in\mathcal{C} Σ ∈ C .
Proof. Changing variables in (30) gives
D μ ( ε , L ) = D μ ( ε L / M , M ) , ε , L > 0. D_\mu(\varepsilon,L)=D_\mu(\varepsilon L/M,M), \qquad\varepsilon,L>0. D μ ( ε , L ) = D μ ( ε L / M , M ) , ε , L > 0.
The function ε ↦ D μ ( ε , M ) \varepsilon\mapsto D_\mu(\varepsilon,M) ε ↦ D μ ( ε , M ) is convex and nondecreasing, and vanishes at zero: for every fixed t t t , the symmetric difference is an even convex function of ε \varepsilon ε , and taking a supremum preserves these properties.
For k ≥ 1 k\ge1 k ≥ 1 and k ε ≤ ε 0 k\varepsilon\le\varepsilon_0 k ε ≤ ε 0 , applying (C.15) at ε , 2 ε , … , 2 ⌈ log 2 k ⌉ − 1 ε \varepsilon,2\varepsilon,\ldots,2^{\lceil\log_2 k\rceil-1}\varepsilon ε , 2 ε , … , 2 ⌈ l o g 2 k ⌉ − 1 ε gives
D μ ( k ε , M ) ≤ Γ ⌈ log 2 k ⌉ D μ ( ε , M ) ≤ Γ k log 2 Γ D μ ( ε , M ) . D_\mu(k\varepsilon,M)\le\Gamma^{\lceil\log_2 k\rceil}D_\mu(\varepsilon,M)\le\Gamma k^{\log_2\Gamma}D_\mu(\varepsilon,M). D μ ( k ε , M ) ≤ Γ ⌈ l o g 2 k ⌉ D μ ( ε , M ) ≤ Γ k l o g 2 Γ D μ ( ε , M ) .
Let
H ( G ) : = 1 + 2 λ ∥ G ∥ 2 M . H(G):=1+\frac{2\sqrt{\lambda}\lVert G\rVert_2}{M}. H ( G ) := 1 + M 2 λ ∥ G ∥ 2 .
Applying this estimate on { ε H ( G ) ≤ ε 0 } \{\varepsilon H(G)\le\varepsilon_0\} { ε H ( G ) ≤ ε 0 } and the global bound D μ ( ε , L ) ≤ R Ω ε L D_\mu(\varepsilon,L)\le R_\Omega\varepsilon L D μ ( ε , L ) ≤ R Ω ε L on its complement gives
E D μ ( ε , M + 2 λ ∥ G ∥ 2 ) ≤ Γ D μ ( ε , M ) E [ H ( G ) log 2 Γ ] + R Ω M ε E [ H ( G ) I { ε H ( G ) > ε 0 } ] . \mathbb{E}D_\mu(\varepsilon,M+2\sqrt{\lambda}\lVert G\rVert_2)
\le\Gamma D_\mu(\varepsilon,M)\mathbb{E}[H(G)^{\log_2\Gamma}]
+R_\Omega M\varepsilon\mathbb{E}[H(G)\mathbb{I}\{\varepsilon H(G)>\varepsilon_0\}]. E D μ ( ε , M + 2 λ ∥ G ∥ 2 ) ≤ Γ D μ ( ε , M ) E [ H ( G ) l o g 2 Γ ] + R Ω M ε E [ H ( G ) I { ε H ( G ) > ε 0 }] .
To compare the last term with D μ ( ε , M ) D_\mu(\varepsilon,M) D μ ( ε , M ) , take c ε = ⌊ log 2 ( ε 0 / ε ) ⌋ c_\varepsilon=\lfloor\log_2(\varepsilon_0/\varepsilon)\rfloor c ε = ⌊ log 2 ( ε 0 / ε )⌋ for 0 < ε ≤ ε 0 / 2 0<\varepsilon\le\varepsilon_0/2 0 < ε ≤ ε 0 /2 . Then ε 0 / 2 < 2 c ε ε ≤ ε 0 \varepsilon_0/2<2^{c_\varepsilon}\varepsilon\le\varepsilon_0 ε 0 /2 < 2 c ε ε ≤ ε 0 , and the same doubling condition gives
D μ ( ε , M ) ≥ Γ − c ε D μ ( 2 c ε ε , M ) ≥ D μ ( ε 0 / 2 , M ) ( ε / ε 0 ) log 2 Γ > 0. D_\mu(\varepsilon,M)\ge\Gamma^{-c_\varepsilon}D_\mu(2^{c_\varepsilon}\varepsilon,M)\ge D_\mu(\varepsilon_0/2,M)(\varepsilon/\varepsilon_0)^{\log_2\Gamma}>0. D μ ( ε , M ) ≥ Γ − c ε D μ ( 2 c ε ε , M ) ≥ D μ ( ε 0 /2 , M ) ( ε / ε 0 ) l o g 2 Γ > 0.
For these ε \varepsilon ε , the exceptional event implies ∥ G ∥ 2 > M ε 0 / ( 4 λ ε ) \lVert G\rVert_2>M\varepsilon_0/(4\sqrt{\lambda}\varepsilon) ∥ G ∥ 2 > M ε 0 / ( 4 λ ε ) . Cauchy–Schwarz and the Gaussian exponential-moment bound imply
E [ ( 1 + 2 λ ∥ G ∥ 2 M ) I { ε ( 1 + 2 λ ∥ G ∥ 2 M ) > ε 0 } ] ≤ E [ ( 1 + 2 λ ∥ G ∥ 2 M ) 2 ] 1 / 2 2 d / 4 exp ( − M 2 ε 0 2 128 λ ε 2 ) . \mathbb{E}\left[\left(1+\frac{2\sqrt{\lambda}\lVert G\rVert_2}{M}\right)\mathbb{I}\left\{\varepsilon\left(1+\frac{2\sqrt{\lambda}\lVert G\rVert_2}{M}\right)>\varepsilon_0\right\}\right]
\le\mathbb{E}\left[\left(1+\frac{2\sqrt{\lambda}\lVert G\rVert_2}{M}\right)^2\right]^{1/2}2^{d/4}\exp\left(-\frac{M^2\varepsilon_0^2}{128\lambda\varepsilon^2}\right). E [ ( 1 + M 2 λ ∥ G ∥ 2 ) I { ε ( 1 + M 2 λ ∥ G ∥ 2 ) > ε 0 } ] ≤ E ( 1 + M 2 λ ∥ G ∥ 2 ) 2 1/2 2 d /4 exp ( − 128 λ ε 2 M 2 ε 0 2 ) .
After division by D μ ( ε , M ) D_{\mu}(\varepsilon, M) D μ ( ε , M ) , the exceptional contribution is bounded by
C ε 1 − log 2 Γ exp ( − M 2 ε 0 2 128 λ ε 2 ) → 0. C\varepsilon^{1-\log_{2}\Gamma}\exp\left(-\frac{M^{2}\varepsilon_{0}^{2}}{128\lambda\varepsilon^{2}}\right)\to0. C ε 1 − l o g 2 Γ exp ( − 128 λ ε 2 M 2 ε 0 2 ) → 0.
Lemma C.4 now proves (C.16). Finally, Proposition 3.4 gives the uniform positive lower factor 2 Φ ( − M / λ ) 2\Phi(-M/\sqrt{\lambda}) 2Φ ( − M / λ ) , while the minimax definition compares R n , M Σ ( μ ) R^{\Sigma}_{n,M}(\mu) R n , M Σ ( μ ) with the risk of the same sample-only policy. These inequalities prove (C.17).
Convex-Geometric Proofs
Proof of Corollary 4.2
Proof. For every b > 0 b>0 b > 0 and x ∈ R d x\in\mathbb{R}^{d} x ∈ R d ,
f ∗ ( b x ) = b f ∗ ( x ) , arg max w ∈ Ω w T ( b x ) = arg max w ∈ Ω w T x . f_{*}(bx)=bf_{*}(x),\qquad\underset{w\in\Omega}{\arg\max}\ w^{\mathsf{T}}(bx)=\underset{w\in\Omega}{\arg\max}\ w^{\mathsf{T}}x. f ∗ ( b x ) = b f ∗ ( x ) , w ∈ Ω arg max w T ( b x ) = w ∈ Ω arg max w T x .
It follows that b A = A bA=A b A = A . Also 0 ∈ A 0\in A 0 ∈ A , since Ω \Omega Ω is not a singleton. Therefore
dist ( b x , A ) = inf y ∈ A ∥ b x − y ∥ K = b inf y ∈ A ∥ x − y ∥ K = b dist ( x , A ) . (95) \operatorname{dist}(bx,A)=\inf_{y\in A}\lVert bx-y\rVert_{K}=b\inf_{y\in A}\lVert x-y\rVert_{K}=b\operatorname{dist}(x,A). \tag*{(95)} dist ( b x , A ) = y ∈ A inf ∥ b x − y ∥ K = b y ∈ A inf ∥ x − y ∥ K = b dist ( x , A ) . (95)
We first verify ∂ f ∗ ( x ) = arg max w ∈ Ω w T x \partial f_{*}(x)=\underset{w\in\Omega}{\arg\max}\ w^{\mathsf{T}}x ∂ f ∗ ( x ) = w ∈ Ω arg max w T x . If w ∈ Ω w\in\Omega w ∈ Ω maximizes w T x w^{\mathsf{T}}x w T x , then
f ∗ ( y ) ≥ w T y = f ∗ ( x ) + w T ( y − x ) ∀ y ∈ R d , f_{*}(y)\ge w^{\mathsf{T}}y=f_{*}(x)+w^{\mathsf{T}}(y-x)\qquad\forall y\in\mathbb{R}^{d}, f ∗ ( y ) ≥ w T y = f ∗ ( x ) + w T ( y − x ) ∀ y ∈ R d ,
so w ∈ ∂ f ∗ ( x ) w\in\partial f_{*}(x) w ∈ ∂ f ∗ ( x ) . Conversely, if w ∈ ∂ f ∗ ( x ) w\in\partial f_{*}(x) w ∈ ∂ f ∗ ( x ) , substituting y = 0 y=0 y = 0 and y = 2 x y=2x y = 2 x into the subgradient inequality yields
0 ≥ f ∗ ( x ) − w T x , 2 f ∗ ( x ) ≥ f ∗ ( x ) + w T x . 0\ge f_{*}(x)-w^{\mathsf{T}}x,\qquad2f_{*}(x)\ge f_{*}(x)+w^{\mathsf{T}}x. 0 ≥ f ∗ ( x ) − w T x , 2 f ∗ ( x ) ≥ f ∗ ( x ) + w T x .
Hence w T x = f ∗ ( x ) w^{\mathsf{T}}x=f_{*}(x) w T x = f ∗ ( x ) , and the same inequality gives
w T y ≤ f ∗ ( y ) = sup v ∈ Ω v T y ∀ y ∈ R d . w^{\mathsf{T}}y\le f_{*}(y)=\sup_{v\in\Omega}v^{\mathsf{T}}y\qquad\forall y\in\mathbb{R}^{d}. w T y ≤ f ∗ ( y ) = v ∈ Ω sup v T y ∀ y ∈ R d .
Since Ω \Omega Ω is closed and convex, by the separating hyperplane theorem, if w ∉ Ω w\notin\Omega w ∈ / Ω , there exists y ∈ R d y\in\mathbb{R}^{d} y ∈ R d , such that w T y > sup v ∈ Ω v T y w^{\mathsf{T}}y>\sup_{v\in\Omega}v^{\mathsf{T}}y w T y > sup v ∈ Ω v T y . Hence w ∈ Ω w\in\Omega w ∈ Ω , and ∂ f ∗ ( x ) = arg max w ∈ Ω w T x \partial f_{*}(x)=\underset{w\in\Omega}{\arg\max}\ w^{\mathsf{T}}x ∂ f ∗ ( x ) = w ∈ Ω arg max w T x .
A finite convex function on R d \mathbb{R}^{d} R d is differentiable at x x x if and only if its subdifferential is a singleton. Therefore
f ∗ is differentiable at x ⟺ ∂ f ∗ ( x ) = { ∇ f ∗ ( x ) } ⟺ x ∉ A . f_{*}\text{ is differentiable at }x\ \Longleftrightarrow\ \partial f_{*}(x)=\{\nabla f_{*}(x)\}\ \Longleftrightarrow\ x\notin A. f ∗ is differentiable at x ⟺ ∂ f ∗ ( x ) = { ∇ f ∗ ( x )} ⟺ x ∈ / A .
Differentiability throughout a neighborhood of μ \mu μ gives r > 0 r>0 r > 0 such that
{ x : ∥ x − μ ∥ 2 < r } ∩ A = ∅ . \{x:\lVert x-\mu\rVert_{2}<r\}\cap A=\varnothing. { x : ∥ x − μ ∥ 2 < r } ∩ A = ∅ .
By equivalence of norms, there is c > 0 c > 0 c > 0 such that ∥ v ∥ K ≥ c ∥ v ∥ 2 \lVert v\rVert_K \ge c\lVert v\rVert_2 ∥ v ∥ K ≥ c ∥ v ∥ 2 for all v v v , whence
dist ( μ , A ) = inf a ∈ A ∥ μ − a ∥ K ≥ c inf a ∈ A ∥ μ − a ∥ 2 ≥ c r > 0. \operatorname{dist}(\mu,A)=\inf_{a\in A}\lVert\mu-a\rVert_K\ge c\inf_{a\in A}\lVert\mu-a\rVert_2\ge cr>0. dist ( μ , A ) = a ∈ A inf ∥ μ − a ∥ K ≥ c a ∈ A inf ∥ μ − a ∥ 2 ≥ cr > 0.
Also dist ( μ , A ) ≤ ∥ μ ∥ K < ∞ \operatorname{dist}(\mu,A)\le\lVert\mu\rVert_K<\infty dist ( μ , A ) ≤ ∥ μ ∥ K < ∞ since 0 ∈ A 0\in A 0 ∈ A . Thus ν μ \nu_\mu ν μ is well-defined, and (D.1) gives
dist ( ν μ , A ) = dist ( μ , A ) dist ( μ , A ) = 1. \operatorname{dist}(\nu_\mu,A)=\frac{\operatorname{dist}(\mu,A)}{\operatorname{dist}(\mu,A)}=1. dist ( ν μ , A ) = dist ( μ , A ) dist ( μ , A ) = 1.
Let U U U be an open neighborhood of μ \mu μ on which f ∗ ∈ C 2 f_*\in C^2 f ∗ ∈ C 2 . For each fixed b > 0 b>0 b > 0 , the identity f ∗ ( y ) = b f ∗ ( y / b ) f_*(y)=bf_*(y/b) f ∗ ( y ) = b f ∗ ( y / b ) for y ∈ b U y\in bU y ∈ b U implies f ∗ ∈ C 2 ( b U ) f_*\in C^2(bU) f ∗ ∈ C 2 ( b U ) as well. Differentiating f ∗ ( b x ) = b f ∗ ( x ) f_*(bx)=bf_*(x) f ∗ ( b x ) = b f ∗ ( x ) with respect to x ∈ U x\in U x ∈ U , with b b b fixed, the chain rule gives
b ∇ f ∗ ( b x ) = ∇ x [ f ∗ ( b x ) ] = ∇ x [ b f ∗ ( x ) ] = b ∇ f ∗ ( x ) , b\nabla f_*(bx)=\nabla_x[f_*(bx)]=\nabla_x[bf_*(x)]=b\nabla f_*(x), b ∇ f ∗ ( b x ) = ∇ x [ f ∗ ( b x )] = ∇ x [ b f ∗ ( x )] = b ∇ f ∗ ( x ) ,
b 2 ∇ 2 f ∗ ( b x ) = ∇ x 2 [ f ∗ ( b x ) ] = ∇ x 2 [ b f ∗ ( x ) ] = b ∇ 2 f ∗ ( x ) . b^2\nabla^2f_*(bx)=\nabla_x^2[f_*(bx)]=\nabla_x^2[bf_*(x)]=b\nabla^2f_*(x). b 2 ∇ 2 f ∗ ( b x ) = ∇ x 2 [ f ∗ ( b x )] = ∇ x 2 [ b f ∗ ( x )] = b ∇ 2 f ∗ ( x ) .
Dividing by b b b and b 2 b^2 b 2 , respectively, yields
∇ f ∗ ( b x ) = ∇ f ∗ ( x ) , ∇ 2 f ∗ ( b x ) = b − 1 ∇ 2 f ∗ ( x ) . \nabla f_*(bx)=\nabla f_*(x),\qquad\nabla^2f_*(bx)=b^{-1}\nabla^2f_*(x). ∇ f ∗ ( b x ) = ∇ f ∗ ( x ) , ∇ 2 f ∗ ( b x ) = b − 1 ∇ 2 f ∗ ( x ) .
Taking x = μ x=\mu x = μ and b = 1 / dist ( μ , A ) b=1/\operatorname{dist}(\mu,A) b = 1/ dist ( μ , A ) gives H ν μ = dist ( μ , A ) H μ H_{\nu_\mu}=\operatorname{dist}(\mu,A)H_\mu H ν μ = dist ( μ , A ) H μ . Since B M , Σ ( b H ) = b B M , Σ ( H ) \mathcal{B}_{M,\Sigma}(bH)=b\mathcal{B}_{M,\Sigma}(H) B M , Σ ( b H ) = b B M , Σ ( H ) for b ≥ 0 b\ge0 b ≥ 0 , Theorem 3.2 proves (35) and (36) .
Proof of Corollary 4.3
Proof. Since μ ≠ 0 \mu\ne0 μ = 0 , if w ∗ ( μ ) ∈ int Ω w_*(\mu)\in\operatorname{int}\Omega w ∗ ( μ ) ∈ int Ω , then w ∗ ( μ ) + t n μ ∈ Ω w_*(\mu)+tn_\mu\in\Omega w ∗ ( μ ) + t n μ ∈ Ω for some t > 0 t>0 t > 0 , and μ ⊤ ( w ∗ ( μ ) + t n μ ) = f ∗ ( μ ) + t ∥ μ ∥ 2 > f ∗ ( μ ) \mu^\top(w_*(\mu)+tn_\mu)=f_*(\mu)+t\lVert\mu\rVert_2>f_*(\mu) μ ⊤ ( w ∗ ( μ ) + t n μ ) = f ∗ ( μ ) + t ∥ μ ∥ 2 > f ∗ ( μ ) , which contradicts the definition of w ∗ ( μ ) w_*(\mu) w ∗ ( μ ) . Hence w ∗ ( μ ) ∈ ∂ Ω w_*(\mu)\in\partial\Omega w ∗ ( μ ) ∈ ∂ Ω .
By optimality of w ∗ ( μ ) w_*(\mu) w ∗ ( μ ) , μ ⊤ ( w − w ∗ ( μ ) ) ≤ 0 \mu^\top(w-w_*(\mu))\le0 μ ⊤ ( w − w ∗ ( μ )) ≤ 0 for all w ∈ Ω w\in\Omega w ∈ Ω . Thus the support hyperplane of Ω \Omega Ω at w ∗ ( μ ) w_*(\mu) w ∗ ( μ ) is { w : μ ⊤ w = f ∗ ( μ ) } \{w:\mu^\top w=f_*(\mu)\} { w : μ ⊤ w = f ∗ ( μ )} , with orthogonal unit normal vector N ( w ∗ ( μ ) ) = n μ N(w_*(\mu))=n_\mu N ( w ∗ ( μ )) = n μ .
To justify the local graph representation, the C 2 C^2 C 2 hypersurface assumption provides an open neighborhood W W W of w ∗ ( μ ) w_*(\mu) w ∗ ( μ ) and a defining function F ∈ C 2 ( W ) F\in C^2(W) F ∈ C 2 ( W ) such that
∂ Ω ∩ W = { w ∈ W : F ( w ) = 0 } , ∇ F ( w ∗ ( μ ) ) ≠ 0. \partial\Omega\cap W=\{w\in W:F(w)=0\},\qquad\nabla F(w_*(\mu))\ne0. ∂ Ω ∩ W = { w ∈ W : F ( w ) = 0 } , ∇ F ( w ∗ ( μ )) = 0.
Since this gradient is normal to the boundary, multiplying F F F by a nonzero constant allows us to assume ∇ F ( w ∗ ( μ ) ) = n μ \nabla F(w_*(\mu))=n_\mu ∇ F ( w ∗ ( μ )) = n μ . In tangent and normal coordinates, define
G ( z , a ) : = F ( w ∗ ( μ ) + U μ z + n μ a ) , ( z , a ) ∈ R d − 1 × R near ( 0 , 0 ) . G(z,a):=F\left(w_*(\mu)+U_\mu z+n_\mu a\right),\qquad(z,a)\in\mathbb{R}^{d-1}\times\mathbb{R}\text{ near }(0,0). G ( z , a ) := F ( w ∗ ( μ ) + U μ z + n μ a ) , ( z , a ) ∈ R d − 1 × R near ( 0 , 0 ) .
The chain rule gives
G ( 0 , 0 ) = 0 , G(0,0)=0, G ( 0 , 0 ) = 0 ,
∂ a G ( 0 , 0 ) = ∇ F ( w ∗ ( μ ) ) ⊤ n μ = 1 , \partial_aG(0,0)=\nabla F(w_*(\mu))^\top n_\mu=1, ∂ a G ( 0 , 0 ) = ∇ F ( w ∗ ( μ ) ) ⊤ n μ = 1 ,
D z G ( 0 , 0 ) = ∇ F ( w ∗ ( μ ) ) ⊤ U μ = 0 ⊤ . D_zG(0,0)=\nabla F(w_*(\mu))^\top U_\mu=0^\top. D z G ( 0 , 0 ) = ∇ F ( w ∗ ( μ ) ) ⊤ U μ = 0 ⊤ .
By the implicit function theorem, there are neighborhoods V ⊂ R d − 1 V \subset\mathbb{R}^{d-1} V ⊂ R d − 1 of 0 0 0 and I ⊂ R I \subset\mathbb{R} I ⊂ R of 0 0 0 , and ϱ μ ∈ C 2 ( V ) \varrho_\mu\in C^2(V) ϱ μ ∈ C 2 ( V ) with ϱ μ ( 0 ) = 0 \varrho_\mu(0)=0 ϱ μ ( 0 ) = 0 , such that
G ( z , a ) = 0 ⟺ a = − ϱ μ ( z ) , ( z , a ) ∈ V × I . G(z,a)=0 \quad\Longleftrightarrow\quad a=-\varrho_\mu(z),\qquad(z,a)\in V\times I. G ( z , a ) = 0 ⟺ a = − ϱ μ ( z ) , ( z , a ) ∈ V × I .
Differentiating G ( z , − ϱ μ ( z ) ) = 0 G(z,-\varrho_\mu(z))=0 G ( z , − ϱ μ ( z )) = 0 at z = 0 z=0 z = 0 gives
D z G ( 0 , 0 ) − ∂ a G ( 0 , 0 ) D ϱ μ ( 0 ) = 0 T , D ϱ μ ( 0 ) = 0 T . D_zG(0,0)-\partial_aG(0,0)D\varrho_\mu(0)=\mathbf{0}^{\mathsf{T}},\qquad D\varrho_\mu(0)=\mathbf{0}^{\mathsf{T}}. D z G ( 0 , 0 ) − ∂ a G ( 0 , 0 ) D ϱ μ ( 0 ) = 0 T , D ϱ μ ( 0 ) = 0 T .
Consequently, in this neighborhood the boundary is parametrized by
w ( z ) = w ∗ ( μ ) + U μ z − n μ ϱ μ ( z ) ∈ ∂ Ω (96) w(z)=w_*(\mu)+U_\mu z-n_\mu\varrho_\mu(z)\in\partial\Omega \tag*{(96)} w ( z ) = w ∗ ( μ ) + U μ z − n μ ϱ μ ( z ) ∈ ∂ Ω (96)
where ϱ μ ∈ C 2 \varrho_\mu\in C^2 ϱ μ ∈ C 2 near 0 0 0 , ϱ μ ( 0 ) = 0 \varrho_\mu(0)=0 ϱ μ ( 0 ) = 0 , and ∇ ϱ μ ( 0 ) = 0 \nabla\varrho_\mu(0)=0 ∇ ϱ μ ( 0 ) = 0 . For every a ∈ R d − 1 \mathbf{a}\in\mathbb{R}^{d-1} a ∈ R d − 1 , by (D.2), the tangent vector takes the form
D w ( z ) [ a ] = U μ a − n μ ∇ ϱ μ ( z ) T a . Dw(z)[\mathbf{a}]=U_\mu\mathbf{a}-n_\mu\nabla\varrho_\mu(z)^{\mathsf{T}}\mathbf{a}. D w ( z ) [ a ] = U μ a − n μ ∇ ϱ μ ( z ) T a .
Note that ( n μ + U μ ∇ ϱ μ ( z ) ) T D w ( z ) [ a ] = 0 (n_\mu+U_\mu\nabla\varrho_\mu(z))^{\mathsf{T}}Dw(z)[\mathbf{a}]=0 ( n μ + U μ ∇ ϱ μ ( z ) ) T D w ( z ) [ a ] = 0 , and therefore the unit normal vector is
N ( w ( z ) ) = n μ + U μ ∇ ϱ μ ( z ) 1 + ∥ ∇ ϱ μ ( z ) ∥ 2 2 . N(w(z))=\frac{n_\mu+U_\mu\nabla\varrho_\mu(z)}{\sqrt{1+\lVert\nabla\varrho_\mu(z)\rVert_2^2}}. N ( w ( z )) = 1 + ∥ ∇ ϱ μ ( z ) ∥ 2 2 n μ + U μ ∇ ϱ μ ( z ) .
Again, by (D.2), z = U μ T ( w ( z ) − w ∗ ( μ ) ) z=U_\mu^{\mathsf{T}}(w(z)-w_*(\mu)) z = U μ T ( w ( z ) − w ∗ ( μ )) , which gives N ∈ C 1 N\in C^1 N ∈ C 1 on the neighborhood, since ϱ μ ∈ C 2 \varrho_\mu\in C^2 ϱ μ ∈ C 2 , and
D N ( w ∗ ( μ ) ) [ U μ a ] = U μ ∇ 2 ϱ μ ( 0 ) a = U μ S μ a , a ∈ R d − 1 DN(w_*(\mu))[U_\mu\mathbf{a}]=U_\mu\nabla^2\varrho_\mu(0)\mathbf{a}=U_\mu S_\mu\mathbf{a},\qquad\mathbf{a}\in\mathbb{R}^{d-1} D N ( w ∗ ( μ )) [ U μ a ] = U μ ∇ 2 ϱ μ ( 0 ) a = U μ S μ a , a ∈ R d − 1
on T n μ S d − 1 = n μ ⊥ = T μ T_{n_\mu}S^{d-1}=n_\mu^\perp=T_\mu T n μ S d − 1 = n μ ⊥ = T μ , the tangent space of S d − 1 S^{d-1} S d − 1 at n μ n_\mu n μ . Thus
D N ( w ∗ ( μ ) ) : T w ∗ ( μ ) ∂ Ω ⟶ T n μ S d − 1 DN(w_*(\mu)):T_{w_*(\mu)}\partial\Omega\longrightarrow T_{n_\mu}S^{d-1} D N ( w ∗ ( μ )) : T w ∗ ( μ ) ∂ Ω ⟶ T n μ S d − 1
is a linear isomorphism, since its matrix in the orthonormal basis U μ U_\mu U μ is S μ ≻ 0 \mathcal{S}_\mu\succ0 S μ ≻ 0 . Therefore, by the C 1 C^1 C 1 inverse function theorem, N N N has a C 1 C^1 C 1 local inverse N − 1 N^{-1} N − 1 , defined on a neighborhood of n μ n_\mu n μ in S d − 1 S^{d-1} S d − 1 and taking values in a neighborhood of w ∗ ( μ ) w_*(\mu) w ∗ ( μ ) in ∂ Ω \partial\Omega ∂ Ω .
To identify this inverse with the optimizer, let x k → μ x_k\to\mu x k → μ and w k ∈ a r g m a x w ∈ Ω x k T w w_k\in\mathop{\rm arg\,max}_{w\in\Omega}x_k^{\mathsf{T}}w w k ∈ arg max w ∈ Ω x k T w . Compactness and continuity imply, for every convergent subsequence,
w k j → w ∈ Ω ⟹ μ T w = lim j → ∞ x k j T w k j = lim j → ∞ f ∗ ( x k j ) = f ∗ ( μ ) w_{k_j}\to w\in\Omega\quad\Longrightarrow\quad\mu^{\mathsf{T}}w=\lim_{j\to\infty}x_{k_j}^{\mathsf{T}}w_{k_j}=\lim_{j\to\infty}f_*(x_{k_j})=f_*(\mu) w k j → w ∈ Ω ⟹ μ T w = j → ∞ lim x k j T w k j = j → ∞ lim f ∗ ( x k j ) = f ∗ ( μ )
⟹ w = w ∗ ( μ ) . \Longrightarrow\quad w=w_*(\mu). ⟹ w = w ∗ ( μ ) .
Consequently,
lim x → μ sup w ∈ a r g m a x v ∈ Ω x T v ∥ w − w ∗ ( μ ) ∥ 2 = 0. \lim_{x\to\mu}\sup_{w\in\mathop{\rm arg\,max}_{v\in\Omega}x^{\mathsf{T}}v}\lVert w-w_*(\mu)\rVert_2=0. x → μ lim w ∈ arg max v ∈ Ω x T v sup ∥ w − w ∗ ( μ ) ∥ 2 = 0.
All maximizers thus lie in the boundary neighborhood on which N N N is injective for x x x sufficiently close to μ \mu μ . At each such maximizer, N ( w ) = x / ∥ x ∥ 2 N(w)=x/\lVert x\rVert_2 N ( w ) = x / ∥ x ∥ 2 ; local injectivity yields
arg max w ∈ Ω x T w = { N − 1 ( x ∥ x ∥ 2 ) } , N ( w ∗ ( x ) ) = x ∥ x ∥ 2 . (97) \underset{w\in\Omega}{\arg\max}\,x^{\mathsf T}w=\left\{N^{-1}\left(\frac{x}{\lVert x\rVert_2}\right)\right\},\qquad N(w_*(x))=\frac{x}{\lVert x\rVert_2}. \tag*{(97)} w ∈ Ω arg max x T w = { N − 1 ( ∥ x ∥ 2 x ) } , N ( w ∗ ( x )) = ∥ x ∥ 2 x . (97)
Here the objective neighborhood is also chosen so that x / ∥ x ∥ 2 x/\lVert x\rVert_2 x / ∥ x ∥ 2 belongs to the domain of N − 1 N^{-1} N − 1 . The support-function subdifferential identity gives
∂ f ∗ ( x ) = { w ∗ ( x ) } , ∇ f ∗ ( x ) = w ∗ ( x ) = N − 1 ( x ∥ x ∥ 2 ) , \partial f_*(x)=\{w_*(x)\},\qquad\nabla f_*(x)=w_*(x)=N^{-1}\left(\frac{x}{\lVert x\rVert_2}\right), ∂ f ∗ ( x ) = { w ∗ ( x )} , ∇ f ∗ ( x ) = w ∗ ( x ) = N − 1 ( ∥ x ∥ 2 x ) ,
and therefore w ∗ ∈ C 1 ⟹ ∇ f ∗ ∈ C 1 ⟹ f ∗ ∈ C 2 w_*\in C^1\Longrightarrow\nabla f_*\in C^1\Longrightarrow f_*\in C^2 w ∗ ∈ C 1 ⟹ ∇ f ∗ ∈ C 1 ⟹ f ∗ ∈ C 2 .
Since w ∗ = ∇ f ∗ w_*=\nabla f_* w ∗ = ∇ f ∗ locally, for every v ∈ R d v\in\mathbb{R}^d v ∈ R d ,
D w ∗ ( μ ) [ v ] = D ( ∇ f ∗ ) ( μ ) [ v ] = H μ v = d d t w ∗ ( μ + t v ) ∣ t = 0 . Dw_*(\mu)[v]=D(\nabla f_*)(\mu)[v]=H_\mu v=\left.\frac{\mathrm{d}}{\mathrm{d}t}w_*(\mu+tv)\right|_{t=0}. D w ∗ ( μ ) [ v ] = D ( ∇ f ∗ ) ( μ ) [ v ] = H μ v = d t d w ∗ ( μ + t v ) t = 0 .
The curve t ↦ w ∗ ( μ + t v ) t\mapsto w_*(\mu+tv) t ↦ w ∗ ( μ + t v ) lies in ∂ Ω \partial\Omega ∂ Ω for sufficiently small ∣ t ∣ |t| ∣ t ∣ , so H μ v ∈ T μ H_\mu v\in T_\mu H μ v ∈ T μ . Also, the columns of ( U μ , n μ ) (U_\mu,n_\mu) ( U μ , n μ ) form an orthonormal basis of R d \mathbb{R}^d R d , i.e., U μ U μ T + n μ n μ T = I d U_\mu U_\mu^{\mathsf T}+n_\mu n_\mu^{\mathsf T}=I_d U μ U μ T + n μ n μ T = I d . Hence U μ U μ T H μ v = H μ v U_\mu U_\mu^{\mathsf T}H_\mu v=H_\mu v U μ U μ T H μ v = H μ v .
To differentiate the right-hand side of (D.3), first compute
d d t μ + t v ∥ μ + t v ∥ 2 ∣ t = 0 = v ∥ μ ∥ 2 − μ ( μ T v ) ∥ μ ∥ 2 3 = 1 ∥ μ ∥ 2 ( I d − n μ n μ T ) v = 1 ∥ μ ∥ 2 U μ U μ T v . \left.\frac{\mathrm{d}}{\mathrm{d}t}\frac{\mu+tv}{\lVert\mu+tv\rVert_2}\right|_{t=0}
=\frac{v}{\lVert\mu\rVert_2}-\frac{\mu(\mu^{\mathsf T}v)}{\lVert\mu\rVert_2^3}
=\frac{1}{\lVert\mu\rVert_2}(I_d-n_\mu n_\mu^{\mathsf T})v
=\frac{1}{\lVert\mu\rVert_2}U_\mu U_\mu^{\mathsf T}v. d t d ∥ μ + t v ∥ 2 μ + t v t = 0 = ∥ μ ∥ 2 v − ∥ μ ∥ 2 3 μ ( μ T v ) = ∥ μ ∥ 2 1 ( I d − n μ n μ T ) v = ∥ μ ∥ 2 1 U μ U μ T v .
For the left-hand side, the chain rule gives
d d t N ( w ∗ ( μ + t v ) ) ∣ t = 0 = D N ( w ∗ ( μ ) ) [ D w ∗ ( μ ) [ v ] ] = D N ( w ∗ ( μ ) ) [ H μ v ] . \left.\frac{\mathrm{d}}{\mathrm{d}t}N(w_*(\mu+tv))\right|_{t=0}
=DN(w_*(\mu))[Dw_*(\mu)[v]]
=DN(w_*(\mu))[H_\mu v]. d t d N ( w ∗ ( μ + t v )) t = 0 = D N ( w ∗ ( μ )) [ D w ∗ ( μ ) [ v ]] = D N ( w ∗ ( μ )) [ H μ v ] .
Equating these derivatives yields
D N ( w ∗ ( μ ) ) [ H μ v ] = 1 ∥ μ ∥ 2 U μ U μ T v . DN(w_*(\mu))[H_\mu v]=\frac{1}{\lVert\mu\rVert_2}U_\mu U_\mu^{\mathsf T}v. D N ( w ∗ ( μ )) [ H μ v ] = ∥ μ ∥ 2 1 U μ U μ T v .
Substitute H μ v = U μ ( U μ T H μ v ) H_\mu v=U_\mu(U_\mu^{\mathsf T}H_\mu v) H μ v = U μ ( U μ T H μ v ) and multiply by U μ T U_\mu^{\mathsf T} U μ T . By (38) ,
S μ U μ T H μ v = U μ T D N ( w ∗ ( μ ) ) U μ ( U μ T H μ v ) = 1 ∥ μ ∥ 2 U μ T U μ U μ T v = 1 ∥ μ ∥ 2 U μ T v . \mathcal{S}_\mu U_\mu^{\mathsf T}H_\mu v
=U_\mu^{\mathsf T}DN(w_*(\mu))U_\mu(U_\mu^{\mathsf T}H_\mu v)
=\frac{1}{\lVert\mu\rVert_2}U_\mu^{\mathsf T}U_\mu U_\mu^{\mathsf T}v
=\frac{1}{\lVert\mu\rVert_2}U_\mu^{\mathsf T}v. S μ U μ T H μ v = U μ T D N ( w ∗ ( μ )) U μ ( U μ T H μ v ) = ∥ μ ∥ 2 1 U μ T U μ U μ T v = ∥ μ ∥ 2 1 U μ T v .
Since S μ ≻ 0 \mathcal{S}_\mu\succ0 S μ ≻ 0 , multiplication by its inverse gives U μ T H μ v = 1 ∥ μ ∥ 2 S μ − 1 U μ T v U_\mu^{\mathsf T}H_\mu v=\frac{1}{\lVert\mu\rVert_2}\mathcal{S}_\mu^{-1}U_\mu^{\mathsf T}v U μ T H μ v = ∥ μ ∥ 2 1 S μ − 1 U μ T v , and thus
H μ v = U μ U μ T H μ v = 1 ∥ μ ∥ 2 U μ S μ − 1 U μ T v . H_\mu v=U_\mu U_\mu^{\mathsf T}H_\mu v
=\frac{1}{\lVert\mu\rVert_2}U_\mu\mathcal{S}_\mu^{-1}U_\mu^{\mathsf T}v. H μ v = U μ U μ T H μ v = ∥ μ ∥ 2 1 U μ S μ − 1 U μ T v .
As this holds for every v ∈ R d v\in\mathbb{R}^d v ∈ R d ,
H μ = 1 ∥ μ ∥ 2 U μ S μ − 1 U μ T . H_\mu=\frac{1}{\lVert\mu\rVert_2}U_\mu\mathcal{S}_\mu^{-1}U_\mu^{\mathsf T}. H μ = ∥ μ ∥ 2 1 U μ S μ − 1 U μ T .
Moreover,
v ⊤ H μ v = ( U μ ⊤ v ) ⊤ S μ − 1 ( U μ ⊤ v ) ∥ μ ∥ 2 ≥ 0. v^{\top}H_{\mu}v=\frac{(U_{\mu}^{\top}v)^{\top}\mathcal{S}_{\mu}^{-1}(U_{\mu}^{\top}v)}{\lVert\mu\rVert_{2}}\ge0. v ⊤ H μ v = ∥ μ ∥ 2 ( U μ ⊤ v ) ⊤ S μ − 1 ( U μ ⊤ v ) ≥ 0.
Positive definiteness of S μ − 1 \mathcal{S}_{\mu}^{-1} S μ − 1 gives
v ⊤ H μ v = 0 ⟺ U μ ⊤ v = 0 ⟺ v ∈ T μ ⊥ = span { μ } . v^{\top}H_{\mu}v=0\Longleftrightarrow U_{\mu}^{\top}v=0\Longleftrightarrow v\in T_{\mu}^{\perp}=\operatorname{span}\{\mu\}. v ⊤ H μ v = 0 ⟺ U μ ⊤ v = 0 ⟺ v ∈ T μ ⊥ = span { μ } .
Thus
ker ( H μ ) = span { μ } , rank ( H μ ) = d − dim ker ( H μ ) = d − 1 , \ker(H_{\mu})=\operatorname{span}\{\mu\},\qquad\operatorname{rank}(H_{\mu})=d-\dim\ker(H_{\mu})=d-1, ker ( H μ ) = span { μ } , rank ( H μ ) = d − dim ker ( H μ ) = d − 1 ,
since μ ≠ 0 \mu\ne0 μ = 0 . This proves (39) and the rank and kernel assertions.
In the principal basis, we have
tr ( Σ H μ ) = tr ( S μ − 1 U μ ⊤ Σ U μ ) ∥ μ ∥ 2 = 1 ∥ μ ∥ 2 ∑ i = 1 d − 1 u i ⊤ Σ u i κ i ( μ ) . \operatorname{tr}(\Sigma H_{\mu})=\frac{\operatorname{tr}(\mathcal{S}_{\mu}^{-1}U_{\mu}^{\top}\Sigma U_{\mu})}{\lVert\mu\rVert_{2}}=\frac{1}{\lVert\mu\rVert_{2}}\sum_{i=1}^{d-1}\frac{u_i^{\top}\Sigma u_i}{\kappa_i(\mu)}. tr ( Σ H μ ) = ∥ μ ∥ 2 tr ( S μ − 1 U μ ⊤ Σ U μ ) = ∥ μ ∥ 2 1 i = 1 ∑ d − 1 κ i ( μ ) u i ⊤ Σ u i .
The established local C 2 C^2 C 2 smoothness permits the application of Theorem 3.2 , yielding
lim M → ∞ lim n → ∞ n R n , M Σ ( μ ) = 1 2 tr ( Σ H μ ) = 1 2 ∥ μ ∥ 2 ∑ i = 1 d − 1 u i ⊤ Σ u i κ i ( μ ) . \lim_{M\to\infty}\lim_{n\to\infty}nR_{n,M}^{\Sigma}(\mu)=\frac{1}{2}\operatorname{tr}(\Sigma H_{\mu})=\frac{1}{2\lVert\mu\rVert_{2}}\sum_{i=1}^{d-1}\frac{u_i^{\top}\Sigma u_i}{\kappa_i(\mu)}. M → ∞ lim n → ∞ lim n R n , M Σ ( μ ) = 2 1 tr ( Σ H μ ) = 2 ∥ μ ∥ 2 1 i = 1 ∑ d − 1 κ i ( μ ) u i ⊤ Σ u i .
Corollary 4.2 also gives dist ( μ , A ) > 0 \operatorname{dist}(\mu,A)>0 dist ( μ , A ) > 0 . Since μ = ∥ μ ∥ 2 n μ \mu=\lVert\mu\rVert_{2}n_{\mu} μ = ∥ μ ∥ 2 n μ , (D.1) implies
dist ( μ , A ) = dist ( ∥ μ ∥ 2 n μ , A ) = ∥ μ ∥ 2 dist ( n μ , A ) . \operatorname{dist}(\mu,A)=\operatorname{dist}(\lVert\mu\rVert_{2}n_{\mu},A)=\lVert\mu\rVert_{2}\operatorname{dist}(n_{\mu},A). dist ( μ , A ) = dist (∥ μ ∥ 2 n μ , A ) = ∥ μ ∥ 2 dist ( n μ , A ) .
Substituting into the minimax constant gives
lim M → ∞ lim n → ∞ n R n , M Σ ( μ ) = dist ( n μ , A ) 2 dist ( μ , A ) ∑ i = 1 d − 1 u i ⊤ Σ u i κ i ( μ ) , \lim_{M\to\infty}\lim_{n\to\infty}nR_{n,M}^{\Sigma}(\mu)=\frac{\operatorname{dist}(n_{\mu},A)}{2\operatorname{dist}(\mu,A)}\sum_{i=1}^{d-1}\frac{u_i^{\top}\Sigma u_i}{\kappa_i(\mu)}, M → ∞ lim n → ∞ lim n R n , M Σ ( μ ) = 2 dist ( μ , A ) dist ( n μ , A ) i = 1 ∑ d − 1 κ i ( μ ) u i ⊤ Σ u i ,
which proves (40) .
Quadratic growth implies positive curvature
Lemma D.1. Let d ≥ 2 d\ge2 d ≥ 2 , let Ω \Omega Ω be compact and convex with nonempty interior, and fix μ ≠ 0 \mu\ne0 μ = 0 with dist ( μ , A ) > 0 \operatorname{dist}(\mu,A)>0 dist ( μ , A ) > 0 . Assume that ∂ Ω \partial\Omega ∂ Ω is a C 2 C^2 C 2 hypersurface near w ∗ ( μ ) w_*(\mu) w ∗ ( μ ) and that (11) holds at z = μ z=\mu z = μ . Then S μ ≻ 0 \mathcal{S}_{\mu}\succ0 S μ ≻ 0 , so Corollary 4.3 applies without a separate positive-curvature assumption.
Proof. Since μ ∉ A \mu\notin A μ ∈ / A , the maximizer w ∗ ( μ ) w_*(\mu) w ∗ ( μ ) is unique. The local parametrization (D.2) and the identity D N ( w ∗ ( μ ) ) [ U μ a ] = U μ ∇ 2 ϱ μ ( 0 ) a DN(w_*(\mu))[U_{\mu}a]=U_{\mu}\nabla^2\varrho_{\mu}(0)a D N ( w ∗ ( μ )) [ U μ a ] = U μ ∇ 2 ϱ μ ( 0 ) a were derived in the proof of Corollary 4.3 using only the C 2 C^2 C 2 property of ∂ Ω \partial\Omega ∂ Ω . In particular S μ = ∇ 2 ϱ μ ( 0 ) \mathcal{S}_{\mu}=\nabla^2\varrho_{\mu}(0) S μ = ∇ 2 ϱ μ ( 0 ) is symmetric.
For each a ∈ R d − 1 a\in\mathbb{R}^{d-1} a ∈ R d − 1 and sufficiently small t > 0 t>0 t > 0 , (11) with z = μ z=\mu z = μ and w = w ( t a ) w=w(ta) w = w ( t a ) gives
∥ μ ∥ 2 ϱ μ ( t a ) = μ ⊤ ( w ∗ ( μ ) − w ( t a ) ) ≥ c Ω dist ( μ , A ) ∥ t U μ a − n μ ϱ μ ( t a ) ∥ Ω 2 . \lVert\mu\rVert_{2}\varrho_{\mu}(ta)=\mu^{\top}(w_*(\mu)-w(ta))\ge c_{\Omega}\operatorname{dist}(\mu,A)\lVert tU_{\mu}a-n_{\mu}\varrho_{\mu}(ta)\rVert_{\Omega}^{2}. ∥ μ ∥ 2 ϱ μ ( t a ) = μ ⊤ ( w ∗ ( μ ) − w ( t a )) ≥ c Ω dist ( μ , A ) ∥ t U μ a − n μ ϱ μ ( t a ) ∥ Ω 2 .
Since ϱ μ ∈ C 2 \varrho_\mu\in C^2 ϱ μ ∈ C 2 , ϱ μ ( 0 ) = 0 \varrho_\mu(0)=0 ϱ μ ( 0 ) = 0 , and ∇ ϱ μ ( 0 ) = 0 \nabla\varrho_\mu(0)=0 ∇ ϱ μ ( 0 ) = 0 ,
ϱ μ ( t a ) = t 2 2 a ⊤ S μ a + o ( t 2 ) , ϱ μ ( t a ) t → 0. \varrho_\mu(ta)=\frac{t^2}{2}a^\top\mathcal{S}_\mu a+o(t^2),\qquad\frac{\varrho_\mu(ta)}{t}\to0. ϱ μ ( t a ) = 2 t 2 a ⊤ S μ a + o ( t 2 ) , t ϱ μ ( t a ) → 0.
Dividing the inequality by t 2 t^2 t 2 and letting t ↓ 0 t\downarrow0 t ↓ 0 yields
∥ μ ∥ 2 2 2 a ⊤ S μ a ≥ c Ω dist ( μ , A ) lim t ↓ 0 ∥ U μ a − n μ ϱ μ ( t a ) t ∥ Ω 2 = c Ω dist ( μ , A ) ∥ U μ a ∥ Ω 2 . \frac{\lVert\mu\rVert_2^2}{2}a^\top\mathcal{S}_\mu a\ge c_\Omega\operatorname{dist}(\mu,A)\lim_{t\downarrow0}\left\lVert U_\mu a-n_\mu\frac{\varrho_\mu(ta)}{t}\right\rVert_\Omega^2=c_\Omega\operatorname{dist}(\mu,A)\lVert U_\mu a\rVert_\Omega^2. 2 ∥ μ ∥ 2 2 a ⊤ S μ a ≥ c Ω dist ( μ , A ) t ↓ 0 lim U μ a − n μ t ϱ μ ( t a ) Ω 2 = c Ω dist ( μ , A ) ∥ U μ a ∥ Ω 2 .
Consequently,
min 1 ≤ i ≤ d − 1 κ i ( μ ) = min ∥ a ∥ 2 = 1 a ⊤ S μ a ≥ 2 c Ω dist ( μ , A ) ∥ μ ∥ 2 2 min ∥ a ∥ 2 = 1 ∥ U μ a ∥ Ω 2 > 0. \min_{1\le i\le d-1}\kappa_i(\mu)=\min_{\lVert a\rVert_2=1}a^\top\mathcal{S}_\mu a\ge\frac{2c_\Omega\operatorname{dist}(\mu,A)}{\lVert\mu\rVert_2^2}\min_{\lVert a\rVert_2=1}\lVert U_\mu a\rVert_\Omega^2>0. 1 ≤ i ≤ d − 1 min κ i ( μ ) = ∥ a ∥ 2 = 1 min a ⊤ S μ a ≥ ∥ μ ∥ 2 2 2 c Ω dist ( μ , A ) ∥ a ∥ 2 = 1 min ∥ U μ a ∥ Ω 2 > 0.
The final minimum is positive because U μ U_\mu U μ is injective and { a : ∥ a ∥ 2 = 1 } \{a:\lVert a\rVert_2=1\} { a : ∥ a ∥ 2 = 1 } is compact.
Technical boundary-growth theorem
The following result supplies the explicit lower bound, matching upper rate, and homogeneous-profile conclusion summarized in Theorem 4.6 .
Theorem D.2 (Boundary growth and distance-based lower bounds without C 2 C^2 C 2 ). Let d ≥ 2 d\ge2 d ≥ 2 , let Ω \Omega Ω be compact and convex, and fix μ ≠ 0 \mu\ne0 μ = 0 with dist ( μ , A ) > 0 \operatorname{dist}(\mu,A)>0 dist ( μ , A ) > 0 , where the distance is computed in the fixed norm ∥ ⋅ ∥ K \lVert\cdot\rVert_K ∥ ⋅ ∥ K . Retain n μ = μ / ∥ μ ∥ 2 n_\mu=\mu/\lVert\mu\rVert_2 n μ = μ / ∥ μ ∥ 2 and the orthonormal tangent basis U μ U_\mu U μ used in Corollary 4.3. Suppose there exist ρ > 0 \rho>0 ρ > 0 , κ ˉ > 0 \bar{\kappa}>0 κ ˉ > 0 , β > 0 \beta>0 β > 0 , and a boundary profile ϱ μ \varrho_\mu ϱ μ on { z ∈ R d − 1 : ∥ z ∥ 2 ≤ ρ } \{z\in\mathbb{R}^{d-1}:\lVert z\rVert_2\le\rho\} { z ∈ R d − 1 : ∥ z ∥ 2 ≤ ρ } such that
w ∗ ( μ ) + U μ z − n μ ϱ μ ( z ) ∈ Ω , (98) w_*(\mu)+U_\mu z-n_\mu\varrho_\mu(z)\in\Omega, \tag*{(98)} w ∗ ( μ ) + U μ z − n μ ϱ μ ( z ) ∈ Ω , (98) where ϱ μ ( 0 ) = 0 \varrho_\mu(0)=0 ϱ μ ( 0 ) = 0 , and 0 ≤ ϱ μ ( z ) ≤ κ ˉ 1 + β ∥ z ∥ 2 1 + β 0\le\varrho_\mu(z)\le\frac{\bar{\kappa}}{1+\beta}\lVert z\rVert_2^{1+\beta} 0 ≤ ϱ μ ( z ) ≤ 1 + β κ ˉ ∥ z ∥ 2 1 + β for ∥ z ∥ 2 ≤ ρ \lVert z\rVert_2\le\rho ∥ z ∥ 2 ≤ ρ . This uses the same parametrization as (D.2), but does not require ϱ μ \varrho_\mu ϱ μ nor f ∗ f_* f ∗ to be twice differentiable. For any M > 0 M>0 M > 0 , set h = min { 1 , M } h=\min\{1,M\} h = min { 1 , M } . Whenever
h n ≤ κ ˉ ∥ μ ∥ 2 β , (99) \frac{h}{\sqrt{n}}\le\bar{\kappa}\lVert\mu\rVert_2^\beta, \tag*{(99)} n h ≤ κ ˉ ∥ μ ∥ 2 β , (99) for every unit v ∈ R d − 1 v\in\mathbb{R}^{d-1} v ∈ R d − 1 , the local minimax risk satisfies
R n , M Σ ( μ ) ≥ 2 Φ ( − h v ⊤ U μ ⊤ Σ − 1 U μ v ) β 1 + β ( h 1 + β κ ˉ ∥ μ ∥ 2 n ( 1 + β ) / 2 ) 1 / β R^\Sigma_{n,M}(\mu)\ge2\Phi\left(-h\sqrt{v^\top U_\mu^\top\Sigma^{-1}U_\mu v}\right)\frac{\beta}{1+\beta}\left(\frac{h^{1+\beta}}{\bar{\kappa}\lVert\mu\rVert_2n^{(1+\beta)/2}}\right)^{1/\beta} R n , M Σ ( μ ) ≥ 2Φ ( − h v ⊤ U μ ⊤ Σ − 1 U μ v ) 1 + β β ( κ ˉ ∥ μ ∥ 2 n ( 1 + β ) /2 h 1 + β ) 1/ β = 2 Φ ( − h v ⊤ U μ ⊤ Σ − 1 U μ v ) β 1 + β h ( 1 + β ) / β n − ( 1 + β ) / ( 2 β ) ( dist ( n μ , A ) κ ˉ dist ( μ , A ) ) 1 / β . (100) =2\Phi\left(-h\sqrt{v^\top U_\mu^\top\Sigma^{-1}U_\mu v}\right)\frac{\beta}{1+\beta}h^{(1+\beta)/\beta}n^{-(1+\beta)/(2\beta)}\left(\frac{\operatorname{dist}(n_\mu,A)}{\bar{\kappa}\operatorname{dist}(\mu,A)}\right)^{1/\beta}. \tag*{(100)} = 2Φ ( − h v ⊤ U μ ⊤ Σ − 1 U μ v ) 1 + β β h ( 1 + β ) / β n − ( 1 + β ) / ( 2 β ) ( κ ˉ dist ( μ , A ) dist ( n μ , A ) ) 1/ β . (100) For the matching upper bound, consider μ ≠ 0 \mu\ne0 μ = 0 with a unique maximizer w ∗ ( μ ) w_*(\mu) w ∗ ( μ ) . Suppose that, for some ρ > 0 \rho>0 ρ > 0 , the continuous convex boundary profile ϱ μ \varrho_\mu ϱ μ is defined on a neighborhood of { z ∈ R d − 1 : ∥ z ∥ 2 ≤ ρ } \{z\in\mathbb{R}^{d-1}:\lVert z\rVert_2\le\rho\} { z ∈ R d − 1 : ∥ z ∥ 2 ≤ ρ } , satisfies ϱ μ ( z ) < ρ \varrho_\mu(z)<\rho ϱ μ ( z ) < ρ there, and describes the local feasible set by
w ∗ ( μ ) + U μ z − n μ b ∈ Ω ⟺ ϱ μ ( z ) ≤ b , ∥ z ∥ 2 ≤ ρ , ∣ b ∣ ≤ ρ . (101) w_*(\mu)+U_\mu z-n_\mu b\in\Omega\Longleftrightarrow\varrho_\mu(z)\le b,\qquad\lVert z\rVert_2\le\rho,\quad|b|\le\rho. \tag*{(101)} w ∗ ( μ ) + U μ z − n μ b ∈ Ω ⟺ ϱ μ ( z ) ≤ b , ∥ z ∥ 2 ≤ ρ , ∣ b ∣ ≤ ρ . (101)
Assume that, for β > 0 \beta> 0 β > 0 and 0 < κ ≤ κ ˉ < ∞ 0 < \kappa\le\bar{\kappa} < \infty 0 < κ ≤ κ ˉ < ∞ ,
κ 1 + β ∥ z ∥ 2 1 + β ≤ ϱ μ ( z ) ≤ κ ˉ 1 + β ∥ z ∥ 2 1 + β , ∥ z ∥ 2 ≤ ρ . (102) \frac{\kappa}{1+\beta}\lVert z\rVert_2^{1+\beta} \le\varrho_\mu(z) \le\frac{\bar{\kappa}}{1+\beta}\lVert z\rVert_2^{1+\beta}, \qquad\lVert z\rVert_2 \le\rho. \tag*{(102)} 1 + β κ ∥ z ∥ 2 1 + β ≤ ϱ μ ( z ) ≤ 1 + β κ ˉ ∥ z ∥ 2 1 + β , ∥ z ∥ 2 ≤ ρ . (102)
For every fixed M > 0 M > 0 M > 0 , there are 0 < c M ≤ C M < ∞ 0 < c_M \le C_M < \infty 0 < c M ≤ C M < ∞ such that, for all sufficiently large n n n and all Σ ∈ C \Sigma\in\mathcal{C} Σ ∈ C ,
c M ∥ μ ∥ 2 − 1 / β n − ( 1 + β ) / ( 2 β ) ≤ R n , M Σ ( μ ) ≤ sup t ∈ B 2 ( M ) E μ n , t , Σ R ( w ∗ ( x ˉ ) ) ≤ C M ∥ μ ∥ 2 − 1 / β n − ( 1 + β ) / ( 2 β ) . (103) c_M\lVert\mu\rVert_2^{-1/\beta}n^{-(1+\beta)/(2\beta)} \le R_{n,M}^{\Sigma}(\mu) \le\sup_{t\in B_2(M)} \mathbb{E}_{\mu n,t,\Sigma}R(w_*(\bar{x})) \le C_M\lVert\mu\rVert_2^{-1/\beta}n^{-(1+\beta)/(2\beta)}. \tag*{(103)} c M ∥ μ ∥ 2 − 1/ β n − ( 1 + β ) / ( 2 β ) ≤ R n , M Σ ( μ ) ≤ t ∈ B 2 ( M ) sup E μ n , t , Σ R ( w ∗ ( x ˉ )) ≤ C M ∥ μ ∥ 2 − 1/ β n − ( 1 + β ) / ( 2 β ) . (103)
The constants can depend on d d d , M M M , β \beta β , κ \kappa κ , κ ˉ \bar{\kappa} κ ˉ , λ \lambda λ , λ ˉ \bar{\lambda} λ ˉ .
If, in addition, a function ϱ 0 : R d − 1 → R \varrho_0:\mathbb{R}^{d-1}\to\mathbb{R} ϱ 0 : R d − 1 → R satisfies
lim δ ↓ 0 sup ∥ z ∥ 2 ≤ L ∣ ϱ μ ( δ z ) δ 1 + β − ϱ 0 ( z ) ∣ = 0 , L > 0 , (104) \lim_{\delta\downarrow0}\sup_{\lVert z\rVert_2\le L}\left|\frac{\varrho_\mu(\delta z)}{\delta^{1+\beta}}-\varrho_0(z)\right|=0,\qquad L>0, \tag*{(104)} δ ↓ 0 lim ∥ z ∥ 2 ≤ L sup δ 1 + β ϱ μ ( δ z ) − ϱ 0 ( z ) = 0 , L > 0 , (104)
then ϱ 0 \varrho_0 ϱ 0 is finite, convex, and positively homogeneous of degree 1 + β 1+\beta 1 + β . Define its convex conjugate by
ϱ 0 ∗ ( y ) : = sup z ∈ R d − 1 { y ⊤ z − ϱ 0 ( z ) } . \varrho_0^*(y):=\sup_{z\in\mathbb{R}^{d-1}}\{y^\top z-\varrho_0(z)\}. ϱ 0 ∗ ( y ) := z ∈ R d − 1 sup { y ⊤ z − ϱ 0 ( z )} .
The expansion (C.1) holds with
a ( ϵ ) = ϵ ( 1 + β ) / β , ψ ( t ) = ∥ μ ∥ 2 − 1 / β ϱ 0 ∗ ( U μ ⊤ t ) . (105) a(\epsilon)=\epsilon^{(1+\beta)/\beta},\qquad\psi(t)=\lVert\mu\rVert_2^{-1/\beta}\varrho_0^*(U_\mu^\top t). \tag*{(105)} a ( ϵ ) = ϵ ( 1 + β ) / β , ψ ( t ) = ∥ μ ∥ 2 − 1/ β ϱ 0 ∗ ( U μ ⊤ t ) . (105)
Proof. Fix any v ∈ R d − 1 \boldsymbol{v}\in\mathbb{R}^{d-1} v ∈ R d − 1 with ∥ v ∥ 2 = 1 \lVert\boldsymbol{v}\rVert_2=1 ∥ v ∥ 2 = 1 and take t = h U μ v ∈ B 2 ( M ) t=hU_\mu\boldsymbol{v}\in B_2(M) t = h U μ v ∈ B 2 ( M ) . For 0 ≤ b ≤ ρ 0\le b\le\rho 0 ≤ b ≤ ρ , w ∗ ( μ ) + b U μ v − n μ ϱ μ ( b v ) ∈ Ω \boldsymbol{w}_*(\mu)+bU_\mu\boldsymbol{v}-\boldsymbol{n}_\mu\varrho_\mu(b\boldsymbol{v})\in\Omega w ∗ ( μ ) + b U μ v − n μ ϱ μ ( b v ) ∈ Ω . Since μ ⊤ U μ v = n μ ⊤ U μ v = 0 \mu^\top U_\mu\boldsymbol{v}=\boldsymbol{n}_\mu^\top U_\mu\boldsymbol{v}=0 μ ⊤ U μ v = n μ ⊤ U μ v = 0 , and U μ ⊤ U μ = I d − 1 U_\mu^\top U_\mu=I_{d-1} U μ ⊤ U μ = I d − 1 , the support-function definition gives
r μ ( t / n ) = sup w ∈ Ω ( μ + t / n ) ⊤ ( w − w ∗ ( μ ) ) ≥ ( μ + h U μ v / n ) ⊤ ( b U μ v − n μ ϱ μ ( b v ) ) = h b n − ∥ μ ∥ 2 ϱ μ ( b v ) ≥ h b n − κ ˉ ∥ μ ∥ 2 1 + β b 1 + β . \begin{aligned}
r_\mu(t/\sqrt{n})=\sup_{\boldsymbol{w}\in\Omega}(\mu+t/\sqrt{n})^\top(\boldsymbol{w}-\boldsymbol{w}_*(\mu)) \\
&\ge(\mu+hU_\mu\boldsymbol{v}/\sqrt{n})^\top(bU_\mu\boldsymbol{v}-\boldsymbol{n}_\mu\varrho_\mu(b\boldsymbol{v})) \\
&=\frac{hb}{\sqrt{n}}-\lVert\mu\rVert_2\varrho_\mu(b\boldsymbol{v}) \\
&\ge\frac{hb}{\sqrt{n}}-\frac{\bar{\kappa}\lVert\mu\rVert_2}{1+\beta}b^{1+\beta}.
\end{aligned} r μ ( t / n ) = w ∈ Ω sup ( μ + t / n ) ⊤ ( w − w ∗ ( μ )) ≥ ( μ + h U μ v / n ) ⊤ ( b U μ v − n μ ϱ μ ( b v )) = n hb − ∥ μ ∥ 2 ϱ μ ( b v ) ≥ n hb − 1 + β κ ˉ ∥ μ ∥ 2 b 1 + β .
The same calculation with − b v -b\boldsymbol{v} − b v gives the identical bound for r μ ( − t / n ) r_\mu(-t/\sqrt{n}) r μ ( − t / n ) . The concave function b ↦ h b n − κ ˉ ∥ μ ∥ 2 1 + β b 1 + β b\mapsto\frac{hb}{\sqrt{n}}-\frac{\bar{\kappa}\lVert\mu\rVert_2}{1+\beta}b^{1+\beta} b ↦ n hb − 1 + β κ ˉ ∥ μ ∥ 2 b 1 + β on the last line is maximized over b ≥ 0 b\ge0 b ≥ 0 at
b = ( h κ ˉ ∥ μ ∥ 2 n ) 1 / β . b=\left(\frac{h}{\bar{\kappa}\lVert\mu\rVert_2\sqrt{n}}\right)^{1/\beta}. b = ( κ ˉ ∥ μ ∥ 2 n h ) 1/ β .
By (D.5), this value belongs to [ 0 , ρ ] [0,\rho] [ 0 , ρ ] . Substitution yields
min { r μ ( t / n ) , r μ ( − t / n ) } ≥ β 1 + β ( h n ) ( 1 + β ) / β ( κ ˉ ∥ μ ∥ 2 ) − 1 / β . (106) \min\{r_\mu(t/\sqrt{n}),r_\mu(-t/\sqrt{n})\}\ge\frac{\beta}{1+\beta}\left(\frac{h}{\sqrt{n}}\right)^{(1+\beta)/\beta}(\bar{\kappa}\lVert\mu\rVert_2)^{-1/\beta}. \tag*{(106)} min { r μ ( t / n ) , r μ ( − t / n )} ≥ 1 + β β ( n h ) ( 1 + β ) / β ( κ ˉ ∥ μ ∥ 2 ) − 1/ β . (106)
Use the equal-probability prior on T = t T=t T = t and T = − t T=-t T = − t . The same binary-prior argument in the proof of Proposition 3.4 gives, for this particular t t t ,
R n , M Σ ( μ ) ≥ 2 Φ ( − ∥ Σ − 1 / 2 t ∥ 2 ) { f ∗ ( μ + t / n ) + f ∗ ( μ − t / n ) 2 − f ∗ ( μ ) } = Φ ( − h v T U μ T Σ − 1 U μ v ) { r μ ( t / n ) + r μ ( − t / n ) } . \begin{aligned}
R_{n,M}^{\Sigma}(\mu) \ge2\Phi(-\|\Sigma^{-1/2}t\|_2)\left\{\frac{f_*(\mu+t/\sqrt{n})+f_*(\mu-t/\sqrt{n})}{2}-f_*(\mu)\right\} \\
&= \Phi\left(-h\sqrt{\mathbf{v}^{\mathsf{T}}U_\mu^{\mathsf{T}}\Sigma^{-1}U_\mu\mathbf{v}}\right)\{r_\mu(t/\sqrt{n})+r_\mu(-t/\sqrt{n})\}.
\end{aligned} R n , M Σ ( μ ) ≥ 2Φ ( − ∥ Σ − 1/2 t ∥ 2 ) { 2 f ∗ ( μ + t / n ) + f ∗ ( μ − t / n ) − f ∗ ( μ ) } = Φ ( − h v T U μ T Σ − 1 U μ v ) { r μ ( t / n ) + r μ ( − t / n )} .
Combining with (D.12) and ∥ μ ∥ 2 − 1 = dist ( n μ , A ) / dist ( μ , A ) \|\mu\|_2^{-1}=\operatorname{dist}(n_\mu,A)/\operatorname{dist}(\mu,A) ∥ μ ∥ 2 − 1 = dist ( n μ , A ) / dist ( μ , A ) proves (D.6).
We next prove the two-sided bounds and the conjugate limit. Uniqueness at μ \mu μ and compactness localize every perturbed maximizer. Indeed, if h k → 0 h_k\to0 h k → 0 and w k ∈ arg max w ∈ Ω ( μ + h k ) T w w_k\in\operatorname*{arg\,max}_{w\in\Omega}(\mu+h_k)^{\mathsf{T}}w w k ∈ arg max w ∈ Ω ( μ + h k ) T w , every convergent subsequence satisfies
μ T lim k w k = lim k { f ∗ ( μ + h k ) − h k T w k } = f ∗ ( μ ) , lim k w k = w ∗ ( μ ) . \mu^{\mathsf{T}}\lim_{k}w_k=\lim_{k}\{f_*(\mu+h_k)-h_k^{\mathsf{T}}w_k\}=f_*(\mu),\qquad\lim_{k}w_k=w_*(\mu). μ T k lim w k = k lim { f ∗ ( μ + h k ) − h k T w k } = f ∗ ( μ ) , k lim w k = w ∗ ( μ ) .
Hence all maximizers lie in the interior of the coordinate cylinder in (D.7) for sufficiently small h h h . Since f ∗ ( μ ) = μ T w ∗ ( μ ) f_*(\mu)=\mu^{\mathsf{T}}w_*(\mu) f ∗ ( μ ) = μ T w ∗ ( μ ) , the definition of the remainder gives
r μ ( h ) = f ∗ ( μ + h ) − f ∗ ( μ ) − w ∗ ( μ ) T h = max w ∈ Ω ( μ + h ) T ( w − w ∗ ( μ ) ) = max ∥ z ∥ 2 ≤ ρ , ∣ b ∣ ≤ ρ ϱ μ ( z ) ≤ b ( μ + h ) T ( U μ z − n μ b ) . \begin{aligned}
r_\mu(h)=f_*(\mu+h)-f_*(\mu)-w_*(\mu)^{\mathsf{T}}h \\
&=\max_{w\in\Omega}(\mu+h)^{\mathsf{T}}(w-w_*(\mu)) \\
&=\max_{\substack{\|z\|_2\le\rho,\ |b|\le\rho\\ \varrho_\mu(z)\le b}}(\mu+h)^{\mathsf{T}}(U_\mu z-n_\mu b).
\end{aligned} r μ ( h ) = f ∗ ( μ + h ) − f ∗ ( μ ) − w ∗ ( μ ) T h = w ∈ Ω max ( μ + h ) T ( w − w ∗ ( μ )) = ∥ z ∥ 2 ≤ ρ , ∣ b ∣ ≤ ρ ϱ μ ( z ) ≤ b max ( μ + h ) T ( U μ z − n μ b ) .
The last equality follows from the localization of the maximizers and (D.7). Using U μ T μ = 0 U_\mu^{\mathsf{T}}\mu=0 U μ T μ = 0 and μ T n μ = ∥ μ ∥ 2 \mu^{\mathsf{T}}n_\mu=\|\mu\|_2 μ T n μ = ∥ μ ∥ 2 , we obtain
( μ + h ) T ( U μ z − n μ b ) = ( U μ T h ) T z − ( ∥ μ ∥ 2 + n μ T h ) b . (\mu+h)^{\mathsf{T}}(U_\mu z-n_\mu b)=(U_\mu^{\mathsf{T}}h)^{\mathsf{T}}z-(\|\mu\|_2+n_\mu^{\mathsf{T}}h)b. ( μ + h ) T ( U μ z − n μ b ) = ( U μ T h ) T z − ( ∥ μ ∥ 2 + n μ T h ) b .
For ∥ h ∥ 2 < ∥ μ ∥ 2 \|h\|_2<\|\mu\|_2 ∥ h ∥ 2 < ∥ μ ∥ 2 ,
∥ μ ∥ 2 + n μ T h ≥ ∥ μ ∥ 2 − ∥ h ∥ 2 > 0. \|\mu\|_2+n_\mu^{\mathsf{T}}h\ge\|\mu\|_2-\|h\|_2>0. ∥ μ ∥ 2 + n μ T h ≥ ∥ μ ∥ 2 − ∥ h ∥ 2 > 0.
Moreover, (D.8) and the assumption ϱ μ ( z ) < ρ \varrho_\mu(z)<\rho ϱ μ ( z ) < ρ give 0 ≤ ϱ μ ( z ) < ρ 0\le\varrho_\mu(z)<\rho 0 ≤ ϱ μ ( z ) < ρ for ∥ z ∥ 2 ≤ ρ \|z\|_2\le\rho ∥ z ∥ 2 ≤ ρ . Thus, for each fixed z z z , the feasible values of b b b form [ ϱ μ ( z ) , ρ ] [\varrho_\mu(z),\rho] [ ϱ μ ( z ) , ρ ] , and the objective is strictly decreasing in b b b . Consequently,
max b ∈ [ ϱ μ ( z ) , ρ ] { ( U μ T h ) T z − ( ∥ μ ∥ 2 + n μ T h ) b } = ( U μ T h ) T z − ( ∥ μ ∥ 2 + n μ T h ) ϱ μ ( z ) . \max_{b\in[\varrho_\mu(z),\rho]}\{(U_\mu^{\mathsf{T}}h)^{\mathsf{T}}z-(\|\mu\|_2+n_\mu^{\mathsf{T}}h)b\}
=(U_\mu^{\mathsf{T}}h)^{\mathsf{T}}z-(\|\mu\|_2+n_\mu^{\mathsf{T}}h)\varrho_\mu(z). b ∈ [ ϱ μ ( z ) , ρ ] max {( U μ T h ) T z − ( ∥ μ ∥ 2 + n μ T h ) b } = ( U μ T h ) T z − ( ∥ μ ∥ 2 + n μ T h ) ϱ μ ( z ) .
Maximizing over z z z therefore gives
r μ ( h ) = max ∥ z ∥ 2 ≤ ρ { ( U μ T h ) T z − ( ∥ μ ∥ 2 + n μ T h ) ϱ μ ( z ) } . (107) r_\mu(h)=\max_{\|z\|_2\le\rho}\{(U_\mu^{\mathsf{T}}h)^{\mathsf{T}}z-(\|\mu\|_2+n_\mu^{\mathsf{T}}h)\varrho_\mu(z)\}. \tag*{(107)} r μ ( h ) = ∥ z ∥ 2 ≤ ρ max {( U μ T h ) T z − ( ∥ μ ∥ 2 + n μ T h ) ϱ μ ( z )} . (107)
For c > 0 c>0 c > 0 , direct maximization gives
sup z ∈ R d − 1 { y T z − c 1 + β ∥ z ∥ 2 1 + β } = β 1 + β c − 1 / β ∥ y ∥ 2 ( 1 + β ) / β . \sup_{z\in\mathbb{R}^{d-1}}\left\{y^{\mathsf{T}}z-\frac{c}{1+\beta}\|z\|_2^{1+\beta}\right\}
=\frac{\beta}{1+\beta}c^{-1/\beta}\|y\|_2^{(1+\beta)/\beta}. z ∈ R d − 1 sup { y T z − 1 + β c ∥ z ∥ 2 1 + β } = 1 + β β c − 1/ β ∥ y ∥ 2 ( 1 + β ) / β .
For y ≠ 0 \boldsymbol{y} \ne0 y = 0 , the maximizing vector has direction y \boldsymbol{y} y and norm ( ∥ y ∥ 2 / c ) 1 / β (\|\boldsymbol{y}\|_2/c)^{1/\beta} ( ∥ y ∥ 2 / c ) 1/ β ; for y = 0 \boldsymbol{y}=0 y = 0 , it is zero. Since ∥ μ ∥ 2 + n μ T h > 0 \|\boldsymbol{\mu}\|_2+\boldsymbol{n}_{\mu}^{\mathsf{T}}\boldsymbol{h}>0 ∥ μ ∥ 2 + n μ T h > 0 , the lower bound on ϱ μ \varrho_{\mu} ϱ μ in (D.8) gives
r μ ( h ) ≤ max ∥ z ∥ 2 ≤ ρ { ( U μ T h ) T z − κ ( ∥ μ ∥ 2 + n μ T h ) 1 + β ∥ z ∥ 2 1 + β } ≤ sup z ∈ R d − 1 { ( U μ T h ) T z − κ ( ∥ μ ∥ 2 + n μ T h ) 1 + β ∥ z ∥ 2 1 + β } = β 1 + β ∥ U μ T h ∥ 2 ( 1 + β ) / β [ κ ( ∥ μ ∥ 2 + n μ T h ) ] 1 / β . r_{\mu}(\boldsymbol{h}) \le\max_{\|\boldsymbol{z}\|_2\le\rho}\left\{(\boldsymbol{U}_{\mu}^{\mathsf{T}}\boldsymbol{h})^{\mathsf{T}}\boldsymbol{z}-\frac{\kappa(\|\boldsymbol{\mu}\|_2+\boldsymbol{n}_{\mu}^{\mathsf{T}}\boldsymbol{h})}{1+\beta}\|\boldsymbol{z}\|_2^{1+\beta}\right\}
\le\sup_{\boldsymbol{z}\in\mathbb{R}^{d-1}}\left\{(\boldsymbol{U}_{\mu}^{\mathsf{T}}\boldsymbol{h})^{\mathsf{T}}\boldsymbol{z}-\frac{\kappa(\|\boldsymbol{\mu}\|_2+\boldsymbol{n}_{\mu}^{\mathsf{T}}\boldsymbol{h})}{1+\beta}\|\boldsymbol{z}\|_2^{1+\beta}\right\}
= \frac{\beta}{1+\beta}\frac{\|\boldsymbol{U}_{\mu}^{\mathsf{T}}\boldsymbol{h}\|_2^{(1+\beta)/\beta}}{[\kappa(\|\boldsymbol{\mu}\|_2+\boldsymbol{n}_{\mu}^{\mathsf{T}}\boldsymbol{h})]^{1/\beta}}. r μ ( h ) ≤ ∥ z ∥ 2 ≤ ρ max { ( U μ T h ) T z − 1 + β κ ( ∥ μ ∥ 2 + n μ T h ) ∥ z ∥ 2 1 + β } ≤ z ∈ R d − 1 sup { ( U μ T h ) T z − 1 + β κ ( ∥ μ ∥ 2 + n μ T h ) ∥ z ∥ 2 1 + β } = 1 + β β [ κ ( ∥ μ ∥ 2 + n μ T h ) ] 1/ β ∥ U μ T h ∥ 2 ( 1 + β ) / β .
Here the domain is enlarged only after replacing ϱ μ \varrho_{\mu} ϱ μ by its lower bound on B 2 d − 1 ( ρ ) : = { z ∈ R d − 1 : ∥ z ∥ 2 ≤ ρ } B_2^{d-1}(\rho):=\{\boldsymbol{z}\in\mathbb{R}^{d-1}:\|\boldsymbol{z}\|_2\le\rho\} B 2 d − 1 ( ρ ) := { z ∈ R d − 1 : ∥ z ∥ 2 ≤ ρ } .
Conversely, the upper bound on ϱ μ \varrho_{\mu} ϱ μ gives
r μ ( h ) ≥ max ∥ z ∥ 2 ≤ ρ { ( U μ T h ) T z − κ ˉ ( ∥ μ ∥ 2 + n μ T h ) 1 + β ∥ z ∥ 2 1 + β } . r_{\mu}(\boldsymbol{h}) \ge\max_{\|\boldsymbol{z}\|_2\le\rho}\left\{(\boldsymbol{U}_{\mu}^{\mathsf{T}}\boldsymbol{h})^{\mathsf{T}}\boldsymbol{z}-\frac{\bar{\kappa}(\|\boldsymbol{\mu}\|_2+\boldsymbol{n}_{\mu}^{\mathsf{T}}\boldsymbol{h})}{1+\beta}\|\boldsymbol{z}\|_2^{1+\beta}\right\}. r μ ( h ) ≥ ∥ z ∥ 2 ≤ ρ max { ( U μ T h ) T z − 1 + β κ ˉ ( ∥ μ ∥ 2 + n μ T h ) ∥ z ∥ 2 1 + β } .
To evaluate this maximum, the maximizer over R d − 1 \mathbb{R}^{d-1} R d − 1 must belong to B 2 d − 1 ( ρ ) B_2^{d-1}(\rho) B 2 d − 1 ( ρ ) . If U μ T h = 0 \boldsymbol{U}_{\mu}^{\mathsf{T}}\boldsymbol{h}=0 U μ T h = 0 , it is 0 \boldsymbol{0} 0 . Otherwise, its norm is
( ∥ U μ T h ∥ 2 κ ˉ ( ∥ μ ∥ 2 + n μ T h ) ) 1 / β ≤ ( 2 ∥ h ∥ 2 κ ˉ ∥ μ ∥ 2 ) 1 / β ⟶ 0 , h → 0 , \left(\frac{\|\boldsymbol{U}_{\mu}^{\mathsf{T}}\boldsymbol{h}\|_2}{\bar{\kappa}(\|\boldsymbol{\mu}\|_2+\boldsymbol{n}_{\mu}^{\mathsf{T}}\boldsymbol{h})}\right)^{1/\beta}
\le\left(\frac{2\|\boldsymbol{h}\|_2}{\bar{\kappa}\|\boldsymbol{\mu}\|_2}\right)^{1/\beta}\longrightarrow0,\qquad\boldsymbol{h}\to0, ( κ ˉ ( ∥ μ ∥ 2 + n μ T h ) ∥ U μ T h ∥ 2 ) 1/ β ≤ ( κ ˉ ∥ μ ∥ 2 2∥ h ∥ 2 ) 1/ β ⟶ 0 , h → 0 ,
where the inequality holds for ∥ h ∥ 2 ≤ ∥ μ ∥ 2 / 2 \|\boldsymbol{h}\|_2\le\|\boldsymbol{\mu}\|_2/2 ∥ h ∥ 2 ≤ ∥ μ ∥ 2 /2 . Thus, for sufficiently small h \boldsymbol{h} h , the maximum over B 2 d − 1 ( ρ ) B_2^{d-1}(\rho) B 2 d − 1 ( ρ ) equals the supremum over R d − 1 \mathbb{R}^{d-1} R d − 1 , and
r μ ( h ) ≥ β 1 + β ∥ U μ T h ∥ 2 ( 1 + β ) / β [ κ ˉ ( ∥ μ ∥ 2 + n μ T h ) ] 1 / β . r_{\mu}(\boldsymbol{h}) \ge\frac{\beta}{1+\beta}\frac{\|\boldsymbol{U}_{\mu}^{\mathsf{T}}\boldsymbol{h}\|_2^{(1+\beta)/\beta}}{[\bar{\kappa}(\|\boldsymbol{\mu}\|_2+\boldsymbol{n}_{\mu}^{\mathsf{T}}\boldsymbol{h})]^{1/\beta}}. r μ ( h ) ≥ 1 + β β [ κ ˉ ( ∥ μ ∥ 2 + n μ T h ) ] 1/ β ∥ U μ T h ∥ 2 ( 1 + β ) / β .
Combining the two inequalities gives, for all sufficiently small ∥ h ∥ 2 \|\boldsymbol{h}\|_2 ∥ h ∥ 2 ,
β 1 + β ∥ U μ T h ∥ 2 ( 1 + β ) / β [ κ ˉ ( ∥ μ ∥ 2 + n μ T h ) ] 1 / β ≤ r μ ( h ) ≤ β 1 + β ∥ U μ T h ∥ 2 ( 1 + β ) / β [ κ ( ∥ μ ∥ 2 + n μ T h ) ] 1 / β . (108) \frac{\beta}{1+\beta}\frac{\|\boldsymbol{U}_{\mu}^{\mathsf{T}}\boldsymbol{h}\|_2^{(1+\beta)/\beta}}{[\bar{\kappa}(\|\boldsymbol{\mu}\|_2+\boldsymbol{n}_{\mu}^{\mathsf{T}}\boldsymbol{h})]^{1/\beta}}
\le r_{\mu}(\boldsymbol{h})\le
\frac{\beta}{1+\beta}\frac{\|\boldsymbol{U}_{\mu}^{\mathsf{T}}\boldsymbol{h}\|_2^{(1+\beta)/\beta}}{[\kappa(\|\boldsymbol{\mu}\|_2+\boldsymbol{n}_{\mu}^{\mathsf{T}}\boldsymbol{h})]^{1/\beta}}. \tag*{(108)} 1 + β β [ κ ˉ ( ∥ μ ∥ 2 + n μ T h ) ] 1/ β ∥ U μ T h ∥ 2 ( 1 + β ) / β ≤ r μ ( h ) ≤ 1 + β β [ κ ( ∥ μ ∥ 2 + n μ T h ) ] 1/ β ∥ U μ T h ∥ 2 ( 1 + β ) / β . (108)
For fixed M > 0 M>0 M > 0 , apply (D.14) to h = ε t \boldsymbol{h}=\varepsilon\boldsymbol{t} h = ε t . For the lower bound choose t = M U μ v \boldsymbol{t}=M\boldsymbol{U}_{\mu}\boldsymbol{v} t = M U μ v with ∥ v ∥ 2 = 1 \|\boldsymbol{v}\|_2=1 ∥ v ∥ 2 = 1 , so n μ T t = 0 \boldsymbol{n}_{\mu}^{\mathsf{T}}\boldsymbol{t}=0 n μ T t = 0 . For the upper bound use ∥ U μ T t ∥ 2 ≤ M \|\boldsymbol{U}_{\mu}^{\mathsf{T}}\boldsymbol{t}\|_2\le M ∥ U μ T t ∥ 2 ≤ M and ∥ μ ∥ 2 ± ε n μ T t ≥ ∥ μ ∥ 2 / 2 \|\boldsymbol{\mu}\|_2\pm\varepsilon\boldsymbol{n}_{\mu}^{\mathsf{T}}\boldsymbol{t}\ge\|\boldsymbol{\mu}\|_2/2 ∥ μ ∥ 2 ± ε n μ T t ≥ ∥ μ ∥ 2 /2 . The definition of D μ D_{\mu} D μ then gives, for all sufficiently small ε > 0 \varepsilon>0 ε > 0 ,
β 1 + β ( κ ˉ ∥ μ ∥ 2 ) − 1 / β ( ε M ) ( 1 + β ) / β ≤ D μ ( ε , M ) ≤ β 1 + β ( κ ∥ μ ∥ 2 / 2 ) − 1 / β ( ε M ) ( 1 + β ) / β . \frac{\beta}{1+\beta}(\bar{\kappa}\|\boldsymbol{\mu}\|_2)^{-1/\beta}(\varepsilon M)^{(1+\beta)/\beta}
\le D_{\mu}(\varepsilon,M)\le
\frac{\beta}{1+\beta}(\kappa\|\boldsymbol{\mu}\|_2/2)^{-1/\beta}(\varepsilon M)^{(1+\beta)/\beta}. 1 + β β ( κ ˉ ∥ μ ∥ 2 ) − 1/ β ( εM ) ( 1 + β ) / β ≤ D μ ( ε , M ) ≤ 1 + β β ( κ ∥ μ ∥ 2 /2 ) − 1/ β ( εM ) ( 1 + β ) / β .
Consequently,
D μ ( ε , M ) > 0 , D μ ( 2 ε , M ) D μ ( ε , M ) ≤ 2 ( 1 + β ) / β ( 2 κ ˉ κ ) 1 / β . D_{\mu}(\varepsilon,M)>0,\qquad
\frac{D_{\mu}(2\varepsilon,M)}{D_{\mu}(\varepsilon,M)}
\le2^{(1+\beta)/\beta}\left(\frac{2\bar{\kappa}}{\kappa}\right)^{1/\beta}. D μ ( ε , M ) > 0 , D μ ( ε , M ) D μ ( 2 ε , M ) ≤ 2 ( 1 + β ) / β ( κ 2 κ ˉ ) 1/ β .
Lemma C.5 therefore proves (D.9). The constant in (C.16) depends only on this doubling factor, d d d , M M M , λ ˉ \bar{\lambda} λ ˉ ; the lower bound in Proposition 3.4 is uniform over C \mathcal{C} C .
Under (D.10), convexity and (D.8) pass to ϱ 0 \varrho_0 ϱ 0 . For b > 0 b>0 b > 0 ,
ϱ 0 ( b z ) = lim δ ↓ 0 ϱ μ ( δ b z ) δ 1 + β = b 1 + β ϱ 0 ( z ) , κ 1 + β ∥ z ∥ 2 1 + β ≤ ϱ 0 ( z ) ≤ κ ˉ 1 + β ∥ z ∥ 2 1 + β . \varrho_0(bz)=\lim_{\delta\downarrow0}\frac{\varrho_\mu(\delta bz)}{\delta^{1+\beta}}=b^{1+\beta}\varrho_0(z),\qquad\frac{\kappa}{1+\beta}\lVert z\rVert_2^{1+\beta}\leq\varrho_0(z)\leq\frac{\bar{\kappa}}{1+\beta}\lVert z\rVert_2^{1+\beta}. ϱ 0 ( b z ) = δ ↓ 0 lim δ 1 + β ϱ μ ( δ b z ) = b 1 + β ϱ 0 ( z ) , 1 + β κ ∥ z ∥ 2 1 + β ≤ ϱ 0 ( z ) ≤ 1 + β κ ˉ ∥ z ∥ 2 1 + β .
In (D.13), substitute h = ε t h=\varepsilon t h = εt and rescale the tangent coordinate by ε 1 / β \varepsilon^{1/\beta} ε 1/ β . Then
r μ ( ε t ) ε ( 1 + β ) / β = max ∥ z ∥ 2 ≤ ρ ε − 1 / β { ( U μ ⊤ t ) ⊤ z − ( ∥ μ ∥ 2 + ε n μ ⊤ t ) ϱ μ ( ε 1 / β z ) ε ( 1 + β ) / β } . \frac{r_\mu(\varepsilon t)}{\varepsilon^{(1+\beta)/\beta}}=\max_{\lVert z\rVert_2\leq\rho\varepsilon^{-1/\beta}}\left\{(U_\mu^\top t)^\top z-\left(\lVert\mu\rVert_2+\varepsilon n_\mu^\top t\right)\frac{\varrho_\mu(\varepsilon^{1/\beta}z)}{\varepsilon^{(1+\beta)/\beta}}\right\}. ε ( 1 + β ) / β r μ ( εt ) = ∥ z ∥ 2 ≤ ρ ε − 1/ β max { ( U μ ⊤ t ) ⊤ z − ( ∥ μ ∥ 2 + ε n μ ⊤ t ) ε ( 1 + β ) / β ϱ μ ( ε 1/ β z ) } .
The value at z = 0 z=0 z = 0 is zero. Every nonzero maximizer, uniformly for ∥ t ∥ 2 ≤ M \lVert t\rVert_2\leq M ∥ t ∥ 2 ≤ M and small ε \varepsilon ε , therefore satisfies
0 ≤ M ∥ z ∥ 2 − κ ∥ μ ∥ 2 2 ( 1 + β ) ∥ z ∥ 2 1 + β , ∥ z ∥ 2 ≤ ( 2 ( 1 + β ) M κ ∥ μ ∥ 2 ) 1 / β . 0\leq M\lVert z\rVert_2-\frac{\kappa\lVert\mu\rVert_2}{2(1+\beta)}\lVert z\rVert_2^{1+\beta},\qquad\lVert z\rVert_2\leq\left(\frac{2(1+\beta)M}{\kappa\lVert\mu\rVert_2}\right)^{1/\beta}. 0 ≤ M ∥ z ∥ 2 − 2 ( 1 + β ) κ ∥ μ ∥ 2 ∥ z ∥ 2 1 + β , ∥ z ∥ 2 ≤ ( κ ∥ μ ∥ 2 2 ( 1 + β ) M ) 1/ β .
The same bound contains the maximizers of the limiting objective. Uniform convergence in (D.10) on this fixed ball consequently gives, uniformly for t ∈ B 2 ( M ) t\in B_2(M) t ∈ B 2 ( M ) ,
r μ ( ε t ) ε ( 1 + β ) / β ⟶ sup z { ( U μ ⊤ t ) ⊤ z − ∥ μ ∥ 2 ϱ 0 ( z ) } = ∥ μ ∥ 2 ϱ 0 ∗ ( U μ ⊤ t ∥ μ ∥ 2 ) . \frac{r_\mu(\varepsilon t)}{\varepsilon^{(1+\beta)/\beta}}\longrightarrow\sup_z\left\{(U_\mu^\top t)^\top z-\lVert\mu\rVert_2\varrho_0(z)\right\}=\lVert\mu\rVert_2\varrho_0^*\left(\frac{U_\mu^\top t}{\lVert\mu\rVert_2}\right). ε ( 1 + β ) / β r μ ( εt ) ⟶ z sup { ( U μ ⊤ t ) ⊤ z − ∥ μ ∥ 2 ϱ 0 ( z ) } = ∥ μ ∥ 2 ϱ 0 ∗ ( ∥ μ ∥ 2 U μ ⊤ t ) .
Rescaling z z z in the definition of ϱ 0 ∗ \varrho_0^* ϱ 0 ∗ gives ϱ 0 ∗ ( b y ) = b ( 1 + β ) / β ϱ 0 ∗ ( y ) \varrho_0^*(by)=b^{(1+\beta)/\beta}\varrho_0^*(y) ϱ 0 ∗ ( b y ) = b ( 1 + β ) / β ϱ 0 ∗ ( y ) , proving (D.11). The two-sided bounds imply that ψ \psi ψ is nonaffine on every B 2 ( M ) B_2(M) B 2 ( M ) , M > 0 M>0 M > 0 . Theorem 3.3 and the distance homogeneity identity (D.1) complete the proof.
Proof of Proposition 4.5
Proof. First, we claim that w ∗ ( μ ) − b 0 n μ ∈ int Ω w_*(\mu)-b_0n_\mu\in\operatorname{int}\Omega w ∗ ( μ ) − b 0 n μ ∈ int Ω for some b 0 > 0 b_0>0 b 0 > 0 . Indeed, otherwise there would be b k ↓ 0 b_k\downarrow0 b k ↓ 0 , such that w ∗ ( μ ) − b k n μ ∉ int Ω w_*(\mu)-b_kn_\mu\notin\operatorname{int}\Omega w ∗ ( μ ) − b k n μ ∈ / int Ω for all k k k . By hyperplane separation theorem, for each k k k , there exists a unit vector v k v_k v k satisfying
v k ⊤ ( w − w ∗ ( μ ) + b k n μ ) ≤ 0 ∀ w ∈ Ω , (109) v_k^\top(w-w_*(\mu)+b_kn_\mu)\leq0\qquad\forall w\in\Omega, \tag*{(109)} v k ⊤ ( w − w ∗ ( μ ) + b k n μ ) ≤ 0 ∀ w ∈ Ω , (109)
and in particular, letting w = w ∗ ( μ ) w=w_*(\mu) w = w ∗ ( μ ) gives v k ⊤ n μ ≤ 0 v_k^\top n_\mu\leq0 v k ⊤ n μ ≤ 0 . Suppose v v v is a limit point of v k v_k v k . Then v v v is also a unit vector, and letting k → ∞ k\to\infty k → ∞ in (D.15) gives ∀ w ∈ Ω \forall w\in\Omega ∀ w ∈ Ω , v ⊤ ( w − w ∗ ( μ ) ) ≤ 0 v^\top(w-w_*(\mu))\leq0 v ⊤ ( w − w ∗ ( μ )) ≤ 0 and v ⊤ n μ ≤ 0 v^\top n_\mu\leq0 v ⊤ n μ ≤ 0 , which implies that v v v is a unit supporting normal at w ∗ ( μ ) w_*(\mu) w ∗ ( μ ) distinct from n μ n_\mu n μ , a contradiction to the uniqueness of supporting hyperplane.
For each z ∈ { U μ ⊤ ( w − w ∗ ( μ ) ) : w ∈ Ω } ⊆ R d − 1 z\in\{U_\mu^\top(w-w_*(\mu)):w\in\Omega\}\subseteq\mathbb{R}^{d-1} z ∈ { U μ ⊤ ( w − w ∗ ( μ )) : w ∈ Ω } ⊆ R d − 1 , define
ϱ μ ( z ) = min { b : w ∗ ( μ ) + U μ z − n μ b ∈ Ω } . (110) \varrho_\mu(z)=\min\{b:w_*(\mu)+U_\mu z-n_\mu b\in\Omega\}. \tag*{(110)} ϱ μ ( z ) = min { b : w ∗ ( μ ) + U μ z − n μ b ∈ Ω } . (110)
To verify that the minimum exists, fix such a z z z and choose w ∈ Ω w\in\Omega w ∈ Ω with U μ ⊤ ( w − w ∗ ( μ ) ) = z U_\mu^\top(w-w_*(\mu))=z U μ ⊤ ( w − w ∗ ( μ )) = z . Since U μ U μ ⊤ + n μ n μ ⊤ = I d U_\mu U_\mu^\top+n_\mu n_\mu^\top=I_d U μ U μ ⊤ + n μ n μ ⊤ = I d ,
w = w ∗ ( μ ) + U μ z − n μ [ n μ ⊤ ( w ∗ ( μ ) − w ) ] . w=w_*(\mu)+U_\mu z-n_\mu\left[n_\mu^\top(w_*(\mu)-w)\right]. w = w ∗ ( μ ) + U μ z − n μ [ n μ ⊤ ( w ∗ ( μ ) − w ) ] .
so b = n μ ⊤ ( w ∗ ( μ ) − w ) b = n_\mu^\top(w_\ast(\mu) - w) b = n μ ⊤ ( w ∗ ( μ ) − w ) is feasible. Moreover, every feasible b b b satisfies
0 ≥ μ ⊤ ( w − w ∗ ( μ ) ) = μ ⊤ ( U μ z − n μ b ) = − ∥ μ ∥ 2 b , 0 \ge\mu^\top(w - w_\ast(\mu)) = \mu^\top(U_\mu z - n_\mu b) = -\lVert\mu\rVert_2 b, 0 ≥ μ ⊤ ( w − w ∗ ( μ )) = μ ⊤ ( U μ z − n μ b ) = − ∥ μ ∥ 2 b ,
and hence b ≥ 0 b \ge0 b ≥ 0 . The feasible set is therefore nonempty and bounded below, with a finite infimum
b ∗ : = inf { b : w ∗ ( μ ) + U μ z − n μ b ∈ Ω } ≥ 0. b_\ast:= \inf\{b : w_\ast(\mu) + U_\mu z - n_\mu b \in\Omega\} \ge0. b ∗ := inf { b : w ∗ ( μ ) + U μ z − n μ b ∈ Ω } ≥ 0.
Choose feasible b j → b ∗ b_j \to b_\ast b j → b ∗ . Since Ω \Omega Ω is closed,
w ∗ ( μ ) + U μ z − n μ b j ⟶ w ∗ ( μ ) + U μ z − n μ b ∗ ∈ Ω . w_\ast(\mu) + U_\mu z - n_\mu b_j \longrightarrow w_\ast(\mu) + U_\mu z - n_\mu b_\ast\in\Omega. w ∗ ( μ ) + U μ z − n μ b j ⟶ w ∗ ( μ ) + U μ z − n μ b ∗ ∈ Ω.
Thus b ∗ b_\ast b ∗ is feasible, so the minimum is attained and ϱ μ ( z ) = b ∗ \varrho_\mu(z) = b_\ast ϱ μ ( z ) = b ∗ .
To prove convexity, take z 1 , z 2 z_1,z_2 z 1 , z 2 in the domain of ϱ μ \varrho_\mu ϱ μ and t ∈ [ 0 , 1 ] t \in[0,1] t ∈ [ 0 , 1 ] . Attainment of the minima gives
w i = w ∗ ( μ ) + U μ z i − n μ ϱ μ ( z i ) ∈ Ω , i = 1 , 2. w_i = w_\ast(\mu) + U_\mu z_i - n_\mu\varrho_\mu(z_i) \in\Omega,\qquad i = 1,2. w i = w ∗ ( μ ) + U μ z i − n μ ϱ μ ( z i ) ∈ Ω , i = 1 , 2.
By convexity of Ω \Omega Ω ,
( 1 − t ) w 1 + t w 2 = w ∗ ( μ ) + U μ ( ( 1 − t ) z 1 + t z 2 ) − n μ ( ( 1 − t ) ϱ μ ( z 1 ) + t ϱ μ ( z 2 ) ) ∈ Ω . (1-t)w_1 + tw_2 = w_\ast(\mu) + U_\mu((1-t)z_1 + tz_2) - n_\mu((1-t)\varrho_\mu(z_1) + t\varrho_\mu(z_2)) \in\Omega. ( 1 − t ) w 1 + t w 2 = w ∗ ( μ ) + U μ (( 1 − t ) z 1 + t z 2 ) − n μ (( 1 − t ) ϱ μ ( z 1 ) + t ϱ μ ( z 2 )) ∈ Ω.
By definition of ϱ μ \varrho_\mu ϱ μ (Equation (D.16)),
ϱ μ ( ( 1 − t ) z 1 + t z 2 ) ≤ ( 1 − t ) ϱ μ ( z 1 ) + t ϱ μ ( z 2 ) , \varrho_\mu((1-t)z_1 + tz_2) \le(1-t)\varrho_\mu(z_1) + t\varrho_\mu(z_2), ϱ μ (( 1 − t ) z 1 + t z 2 ) ≤ ( 1 − t ) ϱ μ ( z 1 ) + t ϱ μ ( z 2 ) ,
which proves convexity.
The interior point w ∗ ( μ ) − b 0 n μ w_\ast(\mu) - b_0n_\mu w ∗ ( μ ) − b 0 n μ gives some ρ > 0 \rho> 0 ρ > 0 such that
w ∗ ( μ ) + U μ z − b 0 n μ ∈ Ω , 0 ≤ ϱ μ ( z ) ≤ b 0 , ∥ z ∥ 2 ≤ ρ . w_\ast(\mu) + U_\mu z - b_0n_\mu\in\Omega,\qquad0 \le\varrho_\mu(z) \le b_0,\qquad\lVert z\rVert_2 \le\rho. w ∗ ( μ ) + U μ z − b 0 n μ ∈ Ω , 0 ≤ ϱ μ ( z ) ≤ b 0 , ∥ z ∥ 2 ≤ ρ .
Hence 0 0 0 is in the interior of the domain of ϱ μ \varrho_\mu ϱ μ . A finite convex function is continuous on the interior of its domain, so ϱ μ \varrho_\mu ϱ μ is continuous near 0 0 0 . Since w ∗ ( μ ) ∈ Ω w_\ast(\mu) \in\Omega w ∗ ( μ ) ∈ Ω , b = 0 b = 0 b = 0 is feasible at z = 0 z = 0 z = 0 ; together with nonnegativity, this gives ϱ μ ( 0 ) = 0 \varrho_\mu(0) = 0 ϱ μ ( 0 ) = 0 .
For every a ∈ ∂ ϱ μ ( 0 ) a \in\partial\varrho_\mu(0) a ∈ ∂ ϱ μ ( 0 ) , the subgradient inequality gives ϱ μ ( z ) ≥ ϱ μ ( 0 ) + a ⊤ z = a ⊤ z \varrho_\mu(z) \ge\varrho_\mu(0) + a^\top z = a^\top z ϱ μ ( z ) ≥ ϱ μ ( 0 ) + a ⊤ z = a ⊤ z throughout the domain. For any w ∈ Ω w \in\Omega w ∈ Ω , write z = U μ ⊤ ( w − w ∗ ( μ ) ) z = U_\mu^\top(w - w_\ast(\mu)) z = U μ ⊤ ( w − w ∗ ( μ )) , b = n μ ⊤ ( w ∗ ( μ ) − w ) b = n_\mu^\top(w_\ast(\mu) - w) b = n μ ⊤ ( w ∗ ( μ ) − w ) , and w − w ∗ ( μ ) = U μ z − n μ b w - w_\ast(\mu) = U_\mu z - n_\mu b w − w ∗ ( μ ) = U μ z − n μ b . Here b b b is feasible, so b ≥ ϱ μ ( z ) ≥ a ⊤ z b \ge\varrho_\mu(z) \ge a^\top z b ≥ ϱ μ ( z ) ≥ a ⊤ z . Consequently,
( U μ a + n μ ) ⊤ ( w − w ∗ ( μ ) ) = a ⊤ z − b ≤ 0 , ∀ w ∈ Ω . (U_\mu a + n_\mu)^\top(w - w_\ast(\mu)) = a^\top z - b \le0,\qquad\forall w \in\Omega. ( U μ a + n μ ) ⊤ ( w − w ∗ ( μ )) = a ⊤ z − b ≤ 0 , ∀ w ∈ Ω.
Thus U μ a + n μ U_\mu a+n_\mu U μ a + n μ is a supporting normal at w ∗ ( μ ) w_\ast(\mu) w ∗ ( μ ) . Uniqueness of the supporting hyperplane implies U μ a + n μ = λ n μ U_\mu a + n_\mu= \lambda n_\mu U μ a + n μ = λ n μ for some λ > 0 \lambda> 0 λ > 0 . Taking inner products with n μ n_\mu n μ gives λ = 1 \lambda= 1 λ = 1 , and then U μ a = 0 U_\mu a = 0 U μ a = 0 implies a = 0 a = 0 a = 0 . Conversely, ϱ μ ( z ) ≥ 0 = ϱ μ ( 0 ) \varrho_\mu(z) \ge0 = \varrho_\mu(0) ϱ μ ( z ) ≥ 0 = ϱ μ ( 0 ) shows that 0 ∈ ∂ ϱ μ ( 0 ) 0 \in\partial\varrho_\mu(0) 0 ∈ ∂ ϱ μ ( 0 ) . Therefore ∂ ϱ μ ( 0 ) = { 0 } \partial\varrho_\mu(0) = \{0\} ∂ ϱ μ ( 0 ) = { 0 } . A convex function is differentiable at an interior point of its domain if its subdifferential there is a singleton, so
∇ ϱ μ ( 0 ) = 0 , ϱ μ ( z ) = ϱ μ ( 0 ) + ∇ ϱ μ ( 0 ) ⊤ z + o ( ∥ z ∥ 2 ) = o ( ∥ z ∥ 2 ) . \nabla\varrho_\mu(0) = 0,\qquad\varrho_\mu(z) = \varrho_\mu(0) + \nabla\varrho_\mu(0)^\top z + o(\lVert z\rVert_2) = o(\lVert z\rVert_2). ∇ ϱ μ ( 0 ) = 0 , ϱ μ ( z ) = ϱ μ ( 0 ) + ∇ ϱ μ ( 0 ) ⊤ z + o (∥ z ∥ 2 ) = o (∥ z ∥ 2 ) .
Since w ∗ ( μ ) − b 0 n μ ∈ int Ω w_{\ast}(\mu)-b_{0}n_{\mu}\in\operatorname{int}\Omega w ∗ ( μ ) − b 0 n μ ∈ int Ω , for sufficiently small ρ ∈ ( 0 , b 0 ) \rho\in(0,b_{0}) ρ ∈ ( 0 , b 0 ) we have
w ∗ ( μ ) + U μ z − b 0 n μ ∈ Ω , 0 ≤ ϱ μ ( z ) < ρ , ∥ z ∥ 2 ≤ ρ . w_{\ast}(\mu)+U_{\mu}z-b_{0}n_{\mu}\in\Omega,\qquad0\leq\varrho_{\mu}(z)<\rho,\qquad\lVert z\rVert_{2}\leq\rho. w ∗ ( μ ) + U μ z − b 0 n μ ∈ Ω , 0 ≤ ϱ μ ( z ) < ρ , ∥ z ∥ 2 ≤ ρ .
For each fixed z z z with ∥ z ∥ 2 ≤ ρ \lVert z\rVert_{2}\leq\rho ∥ z ∥ 2 ≤ ρ , convexity of Ω \Omega Ω implies that the feasible b b b form an interval. Its minimum is ϱ μ ( z ) \varrho_{\mu}(z) ϱ μ ( z ) , and it contains b 0 > ρ b_{0}>\rho b 0 > ρ . Thus every b ∈ [ ϱ μ ( z ) , ρ ] b\in[\varrho_{\mu}(z),\rho] b ∈ [ ϱ μ ( z ) , ρ ] is feasible, while every b < ϱ μ ( z ) b<\varrho_{\mu}(z) b < ϱ μ ( z ) is infeasible. Since ϱ μ ( z ) ≥ 0 \varrho_{\mu}(z)\geq0 ϱ μ ( z ) ≥ 0 , the feasible heights within ∣ b ∣ ≤ ρ \lvert b\rvert\leq\rho ∣ b ∣ ≤ ρ are exactly [ ϱ μ ( z ) , ρ ] [\varrho_{\mu}(z),\rho] [ ϱ μ ( z ) , ρ ] , proving (41) .
For (i), let β > 0 \beta>0 β > 0 .
For every boundary point w = w ∗ ( μ ) + U μ z − n μ ϱ μ ( z ) w=w_{\ast}(\mu)+U_{\mu}z-n_{\mu}\varrho_{\mu}(z) w = w ∗ ( μ ) + U μ z − n μ ϱ μ ( z ) in this neighborhood,
n μ T ( w ∗ ( μ ) − w ) = ϱ μ ( z ) , U μ T ( w − w ∗ ( μ ) ) = z . n_{\mu}^{\mathsf{T}}(w_{\ast}(\mu)-w)=\varrho_{\mu}(z),\qquad U_{\mu}^{\mathsf{T}}(w-w_{\ast}(\mu))=z. n μ T ( w ∗ ( μ ) − w ) = ϱ μ ( z ) , U μ T ( w − w ∗ ( μ )) = z .
Consequently, the local support-distance growth condition
n μ T ( w ∗ ( μ ) − w ) ≥ κ 1 + β ∥ U μ T ( w − w ∗ ( μ ) ) ∥ 2 1 + β , w ∈ ∂ Ω near w ∗ ( μ ) , (111) n_{\mu}^{\mathsf{T}}(w_{\ast}(\mu)-w)\geq\frac{\kappa}{1+\beta}\lVert U_{\mu}^{\mathsf{T}}(w-w_{\ast}(\mu))\rVert_{2}^{1+\beta},\qquad w\in\partial\Omega\text{ near }w_{\ast}(\mu), \tag*{(111)} n μ T ( w ∗ ( μ ) − w ) ≥ 1 + β κ ∥ U μ T ( w − w ∗ ( μ )) ∥ 2 1 + β , w ∈ ∂ Ω near w ∗ ( μ ) , (111)
is equivalent to ϱ μ ( z ) ≥ κ 1 + β ∥ z ∥ 2 1 + β \varrho_{\mu}(z)\geq\frac{\kappa}{1+\beta}\lVert z\rVert_{2}^{1+\beta} ϱ μ ( z ) ≥ 1 + β κ ∥ z ∥ 2 1 + β on a sufficiently small coordinate ball. The stronger condition
n μ T ( w ∗ ( μ ) − w ) ≥ κ 1 + β ∥ w − w ∗ ( μ ) ∥ 2 1 + β n_{\mu}^{\mathsf{T}}(w_{\ast}(\mu)-w)\geq\frac{\kappa}{1+\beta}\lVert w-w_{\ast}(\mu)\rVert_{2}^{1+\beta} n μ T ( w ∗ ( μ ) − w ) ≥ 1 + β κ ∥ w − w ∗ ( μ ) ∥ 2 1 + β
also suffices, since ∥ U μ T ( w − w ∗ ( μ ) ) ∥ 2 ≤ ∥ w − w ∗ ( μ ) ∥ 2 \lVert U_{\mu}^{\mathsf{T}}(w-w_{\ast}(\mu))\rVert_{2}\leq\lVert w-w_{\ast}(\mu)\rVert_{2} ∥ U μ T ( w − w ∗ ( μ )) ∥ 2 ≤ ∥ w − w ∗ ( μ ) ∥ 2 .
For (ii), assume 0 < β ≤ 1 0<\beta\leq1 0 < β ≤ 1 and that ∂ Ω \partial\Omega ∂ Ω is locally C 1 , β C^{1,\beta} C 1 , β . Following the same implicit-function argument as in the proof of Corollary 4.3, with a C 1 , β C^{1,\beta} C 1 , β defining function F F F , we obtain the parametrization similar to (D.2):
w ( z ) = w ∗ ( μ ) + U μ z − n μ ϱ μ ( z ) ∈ ∂ Ω , w(z)=w_{\ast}(\mu)+U_{\mu}z-n_{\mu}\varrho_{\mu}(z)\in\partial\Omega, w ( z ) = w ∗ ( μ ) + U μ z − n μ ϱ μ ( z ) ∈ ∂ Ω ,
with ϱ μ ∈ C 1 \varrho_{\mu}\in C^{1} ϱ μ ∈ C 1 , ϱ μ ( 0 ) = 0 \varrho_{\mu}(0)=0 ϱ μ ( 0 ) = 0 , and ∇ ϱ μ ( 0 ) = 0 \nabla\varrho_{\mu}(0)=0 ∇ ϱ μ ( 0 ) = 0 . Using G G G as defined there, differentiation gives
D ϱ μ ( z ) = D z G ( z , − ϱ μ ( z ) ) ∂ a G ( z , − ϱ μ ( z ) ) . D\varrho_{\mu}(z)=\frac{D_{z}G(z,-\varrho_{\mu}(z))}{\partial_{a}G(z,-\varrho_{\mu}(z))}. D ϱ μ ( z ) = ∂ a G ( z , − ϱ μ ( z )) D z G ( z , − ϱ μ ( z )) .
On a sufficiently small neighborhood, the denominator is bounded away from zero. Since D G DG D G is locally β \beta β -Hölder continuous and z ↦ ( z , − ϱ μ ( z ) ) z\mapsto(z,-\varrho_{\mu}(z)) z ↦ ( z , − ϱ μ ( z )) is locally Lipschitz, this formula gives ϱ μ ∈ C 1 , β \varrho_{\mu}\in C^{1,\beta} ϱ μ ∈ C 1 , β . Choose ρ > 0 \rho>0 ρ > 0 small enough that ϱ μ \varrho_{\mu} ϱ μ is defined and belongs to C 1 , β C^{1,\beta} C 1 , β on a neighborhood of { z ∈ R d − 1 : ∥ z ∥ 2 ≤ ρ } \{z\in\mathbb{R}^{d-1}:\lVert z\rVert_{2}\leq\rho\} { z ∈ R d − 1 : ∥ z ∥ 2 ≤ ρ } , and choose κ ˉ > 0 \bar{\kappa}>0 κ ˉ > 0 such that
∥ ∇ ϱ μ ( z ) − ∇ ϱ μ ( z ′ ) ∥ 2 ≤ κ ˉ ∥ z − z ′ ∥ 2 β , ∥ z ∥ 2 , ∥ z ′ ∥ 2 ≤ ρ . \lVert\nabla\varrho_{\mu}(z)-\nabla\varrho_{\mu}(z')\rVert_{2}\leq\bar{\kappa}\lVert z-z'\rVert_{2}^{\beta},\qquad\lVert z\rVert_{2},\lVert z'\rVert_{2}\leq\rho. ∥ ∇ ϱ μ ( z ) − ∇ ϱ μ ( z ′ ) ∥ 2 ≤ κ ˉ ∥ z − z ′ ∥ 2 β , ∥ z ∥ 2 , ∥ z ′ ∥ 2 ≤ ρ .
Optimality at μ \mu μ gives
− ∥ μ ∥ 2 ϱ μ ( z ) = μ T ( w ( z ) − w ∗ ( μ ) ) ≤ 0 , -\lVert\mu\rVert_{2}\varrho_{\mu}(z)=\mu^{\mathsf{T}}(w(z)-w_{\ast}(\mu))\leq0, − ∥ μ ∥ 2 ϱ μ ( z ) = μ T ( w ( z ) − w ∗ ( μ )) ≤ 0 ,
so ϱ μ ( z ) ≥ 0 \varrho_{\mu}(z) \ge0 ϱ μ ( z ) ≥ 0 . For ∥ z ∥ 2 ≤ ρ \lVert z\rVert_{2} \le\rho ∥ z ∥ 2 ≤ ρ , integration along the segment from 0 0 0 to z z z yields
ϱ μ ( z ) = ∫ 0 1 ( ∇ ϱ μ ( s z ) − ∇ ϱ μ ( 0 ) ) T z d s ≤ ∫ 0 1 κ ˉ s β ∥ z ∥ 2 1 + β d s = κ ˉ 1 + β ∥ z ∥ 2 1 + β . \begin{aligned}
\varrho_{\mu}(z)
&= \int_{0}^{1} \left(\nabla\varrho_{\mu}(sz)-\nabla\varrho_{\mu}(0)\right)^{\mathsf{T}}z\,\mathrm{d}s \\
&\le\int_{0}^{1} \bar{\kappa}s^{\beta}\lVert z\rVert_{2}^{1+\beta}\,\mathrm{d}s
= \frac{\bar{\kappa}}{1+\beta}\lVert z\rVert_{2}^{1+\beta}.
\end{aligned} ϱ μ ( z ) = ∫ 0 1 ( ∇ ϱ μ ( sz ) − ∇ ϱ μ ( 0 ) ) T z d s ≤ ∫ 0 1 κ ˉ s β ∥ z ∥ 2 1 + β d s = 1 + β κ ˉ ∥ z ∥ 2 1 + β .
Thus (D.4) and its growth bound hold with exponent 1 + β 1+\beta 1 + β .
Sufficient conditions for quadratic boundary growth
Lemma D.3 (Quadratic boundary growth from convexity and curvature). Let d ≥ 2 d \ge2 d ≥ 2 , let Ω \Omega Ω be compact and convex with nonempty interior, and fix μ ≠ 0 \boldsymbol{\mu} \ne0 μ = 0 with a unique optimizer w ∗ ( μ ) \boldsymbol{w}_{\ast}(\boldsymbol{\mu}) w ∗ ( μ ) . Retain n μ \boldsymbol{n}_{\mu} n μ and U μ U_{\mu} U μ from Proposition 4.5. Each of the following conditions implies the local representation (41) and, for some ρ > 0 \rho> 0 ρ > 0 and 0 < κ ≤ κ ˉ < ∞ 0 < \kappa\le\bar{\kappa} < \infty 0 < κ ≤ κ ˉ < ∞ ,
κ 2 ∥ z ∥ 2 2 ≤ ϱ μ ( z ) ≤ κ ˉ 2 ∥ z ∥ 2 2 , ∥ z ∥ 2 ≤ ρ . \frac{\kappa}{2}\lVert z\rVert_{2}^{2} \le\varrho_{\mu}(z) \le\frac{\bar{\kappa}}{2}\lVert z\rVert_{2}^{2}, \qquad\lVert z\rVert_{2} \le\rho. 2 κ ∥ z ∥ 2 2 ≤ ϱ μ ( z ) ≤ 2 κ ˉ ∥ z ∥ 2 2 , ∥ z ∥ 2 ≤ ρ . (i) ∂ Ω \partial\Omega ∂ Ω is C 1 , 1 C^{1,1} C 1 , 1 near w ∗ ( μ ) \boldsymbol{w}_{\ast}(\boldsymbol{\mu}) w ∗ ( μ ) , and Ω \Omega Ω is strongly convex, i.e., for some $c > 0,
( 1 − s ) w + s w ′ + c s ( 1 − s ) ∥ w − w ′ ∥ 2 2 B 2 ( 1 ) ⊆ Ω , w , w ′ ∈ Ω , s ∈ [ 0 , 1 ] . (1-s)w+sw' + cs(1-s)\lVert w-w'\rVert_{2}^{2}B_{2}(1) \subseteq\Omega,\qquad w,w' \in\Omega,\quad s\in[0,1]. ( 1 − s ) w + s w ′ + cs ( 1 − s ) ∥ w − w ′ ∥ 2 2 B 2 ( 1 ) ⊆ Ω , w , w ′ ∈ Ω , s ∈ [ 0 , 1 ] . In this case, one may take κ = 2 c \kappa=2c κ = 2 c .
(ii) ∂ Ω \partial\Omega ∂ Ω is C 2 C^{2} C 2 near w ∗ ( μ ) \boldsymbol{w}_{\ast}(\boldsymbol{\mu}) w ∗ ( μ ) and S μ ≻ 0 \mathcal{S}_{\mu}\succ0 S μ ≻ 0 .
In case (ii), ϱ μ ( z ) = 1 2 z T S μ z + o ( ∥ z ∥ 2 2 ) \varrho_{\mu}(z)=\frac{1}{2}z^{\mathsf{T}}\mathcal{S}_{\mu}z+o(\lVert z\rVert_{2}^{2}) ϱ μ ( z ) = 2 1 z T S μ z + o (∥ z ∥ 2 2 ) . (D.10) holds with β = 1 \beta=1 β = 1 and ϱ 0 ( z ) = z T S μ z / 2 \varrho_{0}(z)=z^{\mathsf{T}}\mathcal{S}_{\mu}z/2 ϱ 0 ( z ) = z T S μ z /2 . Its convex conjugate and the resulting support-remainder function in Theorem 4.6 are
ϱ 0 ∗ ( y ) = 1 2 y T S μ − 1 y , ψ ( t ) = 1 2 ∥ μ ∥ 2 2 t T U μ S μ − 1 U μ T t = 1 2 t T H μ t . \varrho_{0}^{\ast}(y)=\frac{1}{2}y^{\mathsf{T}}\mathcal{S}_{\mu}^{-1}y,\qquad
\psi(t)=\frac{1}{2\lVert\mu\rVert_{2}^{2}}t^{\mathsf{T}}U_{\mu}\mathcal{S}_{\mu}^{-1}U_{\mu}^{\mathsf{T}}t=\frac{1}{2}t^{\mathsf{T}}H_{\mu}t. ϱ 0 ∗ ( y ) = 2 1 y T S μ − 1 y , ψ ( t ) = 2 ∥ μ ∥ 2 2 1 t T U μ S μ − 1 U μ T t = 2 1 t T H μ t . Proof. In either case, ∂ Ω \partial\Omega ∂ Ω is locally C 1 C^{1} C 1 , so the supporting hyperplane at w ∗ ( μ ) \boldsymbol{w}_{\ast}(\boldsymbol{\mu}) w ∗ ( μ ) is unique and Proposition 4.5 supplies the local boundary profile.
For (i), fix w ′ ∈ Ω w'\in\Omega w ′ ∈ Ω and s ∈ ( 0 , 1 ) s\in(0,1) s ∈ ( 0 , 1 ) . Apply the strong-convexity condition with w = w ∗ ( μ ) w=\boldsymbol{w}_{\ast}(\boldsymbol{\mu}) w = w ∗ ( μ ) . Since ∥ n μ ∥ 2 = 1 \lVert n_{\mu}\rVert_{2}=1 ∥ n μ ∥ 2 = 1 , the point on the resulting ball in the direction n μ n_{\mu} n μ satisfies
v : = ( 1 − s ) w ∗ ( μ ) + s w ′ + c s ( 1 − s ) ∥ w ′ − w ∗ ( μ ) ∥ 2 2 n μ ∈ Ω . v := (1-s)\boldsymbol{w}_{\ast}(\boldsymbol{\mu})+sw'+cs(1-s)\lVert w'-\boldsymbol{w}_{\ast}(\boldsymbol{\mu})\rVert_{2}^{2}n_{\mu}\in\Omega. v := ( 1 − s ) w ∗ ( μ ) + s w ′ + cs ( 1 − s ) ∥ w ′ − w ∗ ( μ ) ∥ 2 2 n μ ∈ Ω. Optimality of w ∗ ( μ ) \boldsymbol{w}_{\ast}(\boldsymbol{\mu}) w ∗ ( μ ) gives μ T ( v − w ∗ ( μ ) ) ≤ 0 \boldsymbol{\mu}^{\mathsf{T}}(v-\boldsymbol{w}_{\ast}(\boldsymbol{\mu}))\le0 μ T ( v − w ∗ ( μ )) ≤ 0 . Dividing by ∥ μ ∥ 2 \lVert\mu\rVert_{2} ∥ μ ∥ 2 and using n μ T n μ = 1 n_{\mu}^{\mathsf{T}}n_{\mu}=1 n μ T n μ = 1 , we obtain
0 ≥ n μ T ( v − w ∗ ( μ ) ) = s n μ T ( w ′ − w ∗ ( μ ) ) + c s ( 1 − s ) ∥ w ′ − w ∗ ( μ ) ∥ 2 2 . \begin{aligned}
0&\ge n_{\mu}^{\mathsf{T}}(v-\boldsymbol{w}_{\ast}(\boldsymbol{\mu}))\\
&=sn_{\mu}^{\mathsf{T}}(w'-\boldsymbol{w}_{\ast}(\boldsymbol{\mu}))+cs(1-s)\lVert w'-\boldsymbol{w}_{\ast}(\boldsymbol{\mu})\rVert_{2}^{2}.
\end{aligned} 0 ≥ n μ T ( v − w ∗ ( μ )) = s n μ T ( w ′ − w ∗ ( μ )) + cs ( 1 − s ) ∥ w ′ − w ∗ ( μ ) ∥ 2 2 . Divide by s s s and let s ↓ 0 s\downarrow0 s ↓ 0 . Then
n μ T ( w ∗ ( μ ) − w ′ ) ≥ c ∥ w ′ − w ∗ ( μ ) ∥ 2 2 , ϱ μ ( z ) ≥ c ( ∥ z ∥ 2 2 + ϱ μ ( z ) ) ≥ c ∥ z ∥ 2 2 . n_{\mu}^{\mathsf{T}}(\boldsymbol{w}_{\ast}(\boldsymbol{\mu})-w')\ge c\lVert w'-\boldsymbol{w}_{\ast}(\boldsymbol{\mu})\rVert_{2}^{2},\qquad
\varrho_{\mu}(z)\ge c\bigl(\lVert z\rVert_{2}^{2}+\varrho_{\mu}(z)\bigr)\ge c\lVert z\rVert_{2}^{2}. n μ T ( w ∗ ( μ ) − w ′ ) ≥ c ∥ w ′ − w ∗ ( μ ) ∥ 2 2 , ϱ μ ( z ) ≥ c ( ∥ z ∥ 2 2 + ϱ μ ( z ) ) ≥ c ∥ z ∥ 2 2 .
This proves the lower bound with κ = 2 c \kappa= 2c κ = 2 c . Proposition 4.5 (ii), applied with β = 1 \beta= 1 β = 1 , supplies the upper bound from the local C 1 , 1 C^{1,1} C 1 , 1 regularity.
For (ii), the calculation in the proof of Corollary 4.3 gives ϱ μ ∈ C 2 \varrho_\mu\in C^2 ϱ μ ∈ C 2 locally and ∇ 2 ϱ μ ( 0 ) = S μ \nabla^2\varrho_\mu(0) = \mathcal{S}_\mu ∇ 2 ϱ μ ( 0 ) = S μ . Since ϱ μ ( 0 ) = 0 \varrho_\mu(0) = 0 ϱ μ ( 0 ) = 0 and ∇ ϱ μ ( 0 ) = 0 \nabla\varrho_\mu(0) = 0 ∇ ϱ μ ( 0 ) = 0 , Taylor expansion gives
ϱ μ ( z ) = 1 2 z T S μ z + o ( ∥ z ∥ 2 2 ) . \varrho_\mu(z) = \frac{1}{2}z^{\mathsf{T}}\mathcal{S}_\mu z + o(\lVert z\rVert_2^2). ϱ μ ( z ) = 2 1 z T S μ z + o (∥ z ∥ 2 2 ) .
The eigenvalues of S μ \mathcal{S}_\mu S μ are positive and finite, so (D.8) holds with β = 1 \beta= 1 β = 1 and suitable κ , κ ˉ > 0 \kappa,\bar{\kappa} > 0 κ , κ ˉ > 0 . The Taylor remainder also implies, for every L > 0 L > 0 L > 0 ,
sup ∥ z ∥ 2 ≤ L ∣ ϱ μ ( δ z ) δ 2 − 1 2 z T S μ z ∣ ⟶ 0 , δ ↓ 0 , \sup_{\lVert z\rVert_2 \le L}\left|\frac{\varrho_\mu(\delta z)}{\delta^2} - \frac{1}{2}z^{\mathsf{T}}\mathcal{S}_\mu z\right| \longrightarrow0,\qquad\delta\downarrow0, ∥ z ∥ 2 ≤ L sup δ 2 ϱ μ ( δ z ) − 2 1 z T S μ z ⟶ 0 , δ ↓ 0 ,
which proves (D.10). Completing the square gives
y T z − 1 2 z T S μ z = 1 2 y T S μ − 1 y − 1 2 ( z − S μ − 1 y ) T S μ ( z − S μ − 1 y ) . y^{\mathsf{T}}z-\frac{1}{2}z^{\mathsf{T}}\mathcal{S}_\mu z
=\frac{1}{2}y^{\mathsf{T}}\mathcal{S}_\mu^{-1}y
-\frac{1}{2}(z-\mathcal{S}_\mu^{-1}y)^{\mathsf{T}}\mathcal{S}_\mu(z-\mathcal{S}_\mu^{-1}y). y T z − 2 1 z T S μ z = 2 1 y T S μ − 1 y − 2 1 ( z − S μ − 1 y ) T S μ ( z − S μ − 1 y ) .
The maximum over z z z is attained at S μ − 1 y \mathcal{S}_\mu^{-1}y S μ − 1 y , yielding the stated formula for ϱ 0 ∗ \varrho_0^* ϱ 0 ∗ . Substitution into (D.11) and the identity (39) give
ψ ( t ) = 1 2 ∥ μ ∥ 2 t T U μ S μ − 1 U μ T t = 1 2 t T H μ t . \psi(t)=\frac{1}{2\lVert\mu\rVert_2}t^{\mathsf{T}}U_\mu\mathcal{S}_\mu^{-1}U_\mu^{\mathsf{T}}t
=\frac{1}{2}t^{\mathsf{T}}H_\mu t. ψ ( t ) = 2 ∥ μ ∥ 2 1 t T U μ S μ − 1 U μ T t = 2 1 t T H μ t .
Thus the conjugate limit recovers the principal-curvature formula in the smooth case. □ \square □
Detailed Calculations for the Gaussian Examples
Rectangular constraints: details for Example 4.8
For the rectangle Ω = [ 0 , 1 ] d \Omega= [0,1]^d Ω = [ 0 , 1 ] d , we have f ∗ ( μ ) = ∑ j = 1 d ( μ j ) + f_*(\mu)=\sum_{j=1}^d(\mu_j)_+ f ∗ ( μ ) = ∑ j = 1 d ( μ j ) + . Write S 0 = { j ∈ [ d ] : μ j = 0 } S_0=\{j\in[d]:\mu_j=0\} S 0 = { j ∈ [ d ] : μ j = 0 } and s = ∥ μ ∥ 0 = d − ∣ S 0 ∣ s=\lVert\mu\rVert_0=d-\lvert S_0\rvert s = ∥ μ ∥ 0 = d − ∣ S 0 ∣ , so that s s s is the number of nonzero coordinates. For every fixed M M M , when ε \varepsilon ε is sufficiently small, ε M < min j ∉ S 0 ∣ μ j ∣ \varepsilon M < \min_{j\notin S_0}\lvert\mu_j\rvert εM < min j ∈ / S 0 ∣ μ j ∣ , with the minimum of the empty set interpreted as + ∞ +\infty + ∞ , every t ∈ B 2 ( M ) t\in B_2(M) t ∈ B 2 ( M ) satisfies
∣ ε t j ∣ ≤ ε ∥ t ∥ 2 ≤ ε M < min ℓ ∉ S 0 ∣ μ ℓ ∣ ≤ ∣ μ j ∣ , j ∉ S 0 . \lvert\varepsilon t_j\rvert\le\varepsilon\lVert t\rVert_2\le\varepsilon M<\min_{\ell\notin S_0}\lvert\mu_\ell\rvert\le\lvert\mu_j\rvert,\qquad j\notin S_0. ∣ ε t j ∣ ≤ ε ∥ t ∥ 2 ≤ εM < ℓ ∈ / S 0 min ∣ μ ℓ ∣ ≤ ∣ μ j ∣ , j ∈ / S 0 .
Thus μ j + ε t j \mu_j+\varepsilon t_j μ j + ε t j has the same sign as μ j \mu_j μ j for every nonzero coordinate, uniformly over t ∈ B 2 ( M ) t\in B_2(M) t ∈ B 2 ( M ) . Coordinatewise optimality gives w ∗ , j ( μ ) = 1 w_{*,j}(\mu)=1 w ∗ , j ( μ ) = 1 when μ j > 0 \mu_j>0 μ j > 0 and w ∗ , j ( μ ) = 0 w_{*,j}(\mu)=0 w ∗ , j ( μ ) = 0 when μ j < 0 \mu_j<0 μ j < 0 ; when μ j = 0 \mu_j=0 μ j = 0 , the selected maximizer has w ∗ , j ( μ ) ∈ [ 0 , 1 ] w_{*,j}(\mu)\in[0,1] w ∗ , j ( μ ) ∈ [ 0 , 1 ] . Expanding the definition of r μ r_\mu r μ yields
r μ ( ε t ) = f ∗ ( μ + ε t ) − f ∗ ( μ ) − ε w ∗ ( μ ) T t = ∑ j = 1 d [ ( μ j + ε t j ) + − ( μ j ) + − ε w ∗ , j ( μ ) t j ] . r_\mu(\varepsilon t)=f_*(\mu+\varepsilon t)-f_*(\mu)-\varepsilon w_*(\mu)^{\mathsf{T}}t
=\sum_{j=1}^{d}\left[(\mu_j+\varepsilon t_j)_+-(\mu_j)_+-\varepsilon w_{*,j}(\mu)t_j\right]. r μ ( εt ) = f ∗ ( μ + εt ) − f ∗ ( μ ) − ε w ∗ ( μ ) T t = j = 1 ∑ d [ ( μ j + ε t j ) + − ( μ j ) + − ε w ∗ , j ( μ ) t j ] .
When ε \varepsilon ε is sufficiently small, the coordinate contributions are
( μ j + ε t j ) + − ( μ j ) + − ε w ∗ , j ( μ ) t j = { ( μ j + ε t j ) − μ j − ε t j = 0 , μ j > 0 , 0 − 0 − 0 = 0 , μ j < 0 , ε ( t j ) + − ε w ∗ , j ( μ ) t j , μ j = 0. (\mu_j+\varepsilon t_j)_+-(\mu_j)_+-\varepsilon w_{*,j}(\boldsymbol{\mu})t_j=
\begin{cases}
(\mu_j+\varepsilon t_j)-\mu_j-\varepsilon t_j=0, & \mu_j>0,\\
0-0-0=0, & \mu_j<0,\\
\varepsilon(t_j)_+-\varepsilon w_{*,j}(\boldsymbol{\mu})t_j, & \mu_j=0.
\end{cases} ( μ j + ε t j ) + − ( μ j ) + − ε w ∗ , j ( μ ) t j = ⎩ ⎨ ⎧ ( μ j + ε t j ) − μ j − ε t j = 0 , 0 − 0 − 0 = 0 , ε ( t j ) + − ε w ∗ , j ( μ ) t j , μ j > 0 , μ j < 0 , μ j = 0.
where ( ε t j ) + = ε ( t j ) + (\varepsilon t_j)_+=\varepsilon(t_j)_+ ( ε t j ) + = ε ( t j ) + since ε > 0 \varepsilon>0 ε > 0 . All nonzero coordinates therefore contribute zero, leaving exactly
r μ ( ε t ) = ε ∑ j ∈ S 0 [ ( t j ) + − w ∗ , j ( μ ) t j ] , t ∈ B 2 ( M ) . (112) r_{\boldsymbol{\mu}}(\varepsilon t)=\varepsilon\sum_{j\in S_0}[(t_j)_+-w_{*,j}(\boldsymbol{\mu})t_j],\qquad t\in B_2(M). \tag*{(112)} r μ ( εt ) = ε j ∈ S 0 ∑ [( t j ) + − w ∗ , j ( μ ) t j ] , t ∈ B 2 ( M ) . (112)
This is an exact identity on the whole local ball. If μ = 0 \boldsymbol{\mu}=0 μ = 0 , then S 0 = [ d ] S_0=[d] S 0 = [ d ] and the minimum above is + ∞ +\infty + ∞ , so the identity holds for every ε > 0 \varepsilon>0 ε > 0 . As in Example 4.8, write ψ ( t ) = ∑ j ∈ S 0 [ ( t j ) + − w ∗ , j ( μ ) t j ] \psi(t)=\sum_{j\in S_0}[(t_j)_+-w_{*,j}(\boldsymbol{\mu})t_j] ψ ( t ) = ∑ j ∈ S 0 [( t j ) + − w ∗ , j ( μ ) t j ] , so that r μ ( ε t ) = ε ψ ( t ) r_{\boldsymbol{\mu}}(\varepsilon t)=\varepsilon\psi(t) r μ ( εt ) = ε ψ ( t ) on the local ball. Suppose that M / n < min j ∉ S 0 ∣ μ j ∣ M/\sqrt{n}<\min_{j\notin S_0}|\mu_j| M / n < min j ∈ / S 0 ∣ μ j ∣ . By (20) , the local minimax risk is
R n , M Σ ( μ ) = sup π ∈ P ( B 2 ( M ) ) E π , Σ [ r μ ( T n ) − r μ ( m π , Σ ( Y ) n ) ] . R^\Sigma_{n,M}(\boldsymbol{\mu})=\sup_{\pi\in\mathcal{P}(B_2(M))}\mathbb{E}_{\pi,\Sigma}\left[r_{\boldsymbol{\mu}}\left(\frac{T}{\sqrt{n}}\right)-r_{\boldsymbol{\mu}}\left(\frac{m_{\pi,\Sigma}(Y)}{\sqrt{n}}\right)\right]. R n , M Σ ( μ ) = π ∈ P ( B 2 ( M )) sup E π , Σ [ r μ ( n T ) − r μ ( n m π , Σ ( Y ) ) ] .
Here T ∼ π T\sim\pi T ∼ π , Y = T + Z Y=T+Z Y = T + Z with independent Z ∼ N ( 0 , Σ ) Z\sim\mathcal{N}(0,\Sigma) Z ∼ N ( 0 , Σ ) , and m π , Σ ( Y ) = E π , Σ [ T ∣ Y ] m_{\pi,\Sigma}(Y)=\mathbb{E}_{\pi,\Sigma}[T\mid Y] m π , Σ ( Y ) = E π , Σ [ T ∣ Y ] . For every such prior, T ∈ B 2 ( M ) T\in B_2(M) T ∈ B 2 ( M ) almost surely, and
∥ m π , Σ ( Y ) ∥ 2 ≤ E π , Σ [ ∥ T ∥ 2 ∣ Y ] ≤ M . \|m_{\pi,\Sigma}(Y)\|_2\leq\mathbb{E}_{\pi,\Sigma}[\|T\|_2\mid Y]\leq M. ∥ m π , Σ ( Y ) ∥ 2 ≤ E π , Σ [ ∥ T ∥ 2 ∣ Y ] ≤ M .
Applying (E.1) at scale n − 1 / 2 n^{-1/2} n − 1/2 gives
r μ ( t n ) = 1 n ψ ( t ) , t ∈ B 2 ( M ) . r_{\boldsymbol{\mu}}\left(\frac{t}{\sqrt{n}}\right)=\frac{1}{\sqrt{n}}\psi(t),\qquad t\in B_2(M). r μ ( n t ) = n 1 ψ ( t ) , t ∈ B 2 ( M ) .
This formula therefore applies to both T T T and m π , Σ ( Y ) m_{\pi,\Sigma}(Y) m π , Σ ( Y ) . Substituting it into their expected remainder difference yields
E π , Σ [ r μ ( T n ) − r μ ( m π , Σ ( Y ) n ) ] = 1 n E π , Σ [ ψ ( T ) − ψ ( m π , Σ ( Y ) ) ] = 1 n E π , Σ [ ∑ j ∈ S 0 ( ( T j ) + − ( m π , Σ , j ( Y ) ) + ) ] − 1 n E π , Σ [ ∑ j ∈ S 0 w ∗ , j ( μ ) ( T j − m π , Σ , j ( Y ) ) ] . \begin{aligned}
\mathbb{E}_{\pi,\Sigma}\left[r_{\boldsymbol{\mu}}\left(\frac{T}{\sqrt{n}}\right)-r_{\boldsymbol{\mu}}\left(\frac{m_{\pi,\Sigma}(Y)}{\sqrt{n}}\right)\right]
=\frac{1}{\sqrt{n}}\mathbb{E}_{\pi,\Sigma}[\psi(T)-\psi(m_{\pi,\Sigma}(Y))] \\
&=\frac{1}{\sqrt{n}}\mathbb{E}_{\pi,\Sigma}\left[\sum_{j\in S_0}\left((T_j)_+-(m_{\pi,\Sigma,j}(Y))_+\right)\right]
-\frac{1}{\sqrt{n}}\mathbb{E}_{\pi,\Sigma}\left[\sum_{j\in S_0}w_{*,j}(\boldsymbol{\mu})\left(T_j-m_{\pi,\Sigma,j}(Y)\right)\right].
\end{aligned} E π , Σ [ r μ ( n T ) − r μ ( n m π , Σ ( Y ) ) ] = n 1 E π , Σ [ ψ ( T ) − ψ ( m π , Σ ( Y ))] = n 1 E π , Σ j ∈ S 0 ∑ ( ( T j ) + − ( m π , Σ , j ( Y ) ) + ) − n 1 E π , Σ j ∈ S 0 ∑ w ∗ , j ( μ ) ( T j − m π , Σ , j ( Y ) ) .
The last expectation vanishes by the tower property: for every j j j ,
E π , Σ [ m π , Σ , j ( Y ) ] = E π , Σ [ E π , Σ [ T j ∣ Y ] ] = E π , Σ [ T j ] , \mathbb{E}_{\pi,\Sigma}[m_{\pi,\Sigma,j}(Y)]
=\mathbb{E}_{\pi,\Sigma}[\mathbb{E}_{\pi,\Sigma}[T_j\mid Y]]
=\mathbb{E}_{\pi,\Sigma}[T_j], E π , Σ [ m π , Σ , j ( Y )] = E π , Σ [ E π , Σ [ T j ∣ Y ]] = E π , Σ [ T j ] ,
and the coefficients w ∗ , j ( μ ) w_{*,j}(\boldsymbol{\mu}) w ∗ , j ( μ ) are fixed. Therefore, by definition of V M , Σ V_{M,\Sigma} V M , Σ ,
R n , M Σ ( μ ) = 1 n sup π ∈ P ( B 2 ( M ) ) E π , Σ [ ψ ( T ) − ψ ( m π , Σ ( Y ) ) ] = 1 n V M , Σ ( ψ ) . (113) R^\Sigma_{n,M}(\boldsymbol{\mu})=\frac{1}{\sqrt{n}}\sup_{\pi\in\mathcal{P}(B_2(M))}\mathbb{E}_{\pi,\Sigma}[\psi(T)-\psi(m_{\pi,\Sigma}(Y))]
=\frac{1}{\sqrt{n}}V_{M,\Sigma}(\psi). \tag*{(113)} R n , M Σ ( μ ) = n 1 π ∈ P ( B 2 ( M )) sup E π , Σ [ ψ ( T ) − ψ ( m π , Σ ( Y ))] = n 1 V M , Σ ( ψ ) . (113)
This equality is exact whenever the sign condition above holds. If s < d s<d s < d , then for every fixed M > 0 M>0 M > 0 and all sufficiently large n n n ,
c min { M d − s , d − s } n ≤ R n , M 2 ( μ ) ≤ C min { M d − s , d − s } n . (114) c\frac{\min\{M\sqrt{d-s},d-s\}}{\sqrt{n}}\le R_{n,M}^{2}(\mu)\le C\frac{\min\{M\sqrt{d-s},d-s\}}{\sqrt{n}}. \tag*{(114)} c n min { M d − s , d − s } ≤ R n , M 2 ( μ ) ≤ C n min { M d − s , d − s } . (114)
where the constants can be chosen as c = Φ ( − 1 ) min { 1 , λ } c=\Phi(-1)\min\{1,\sqrt{\lambda}\} c = Φ ( − 1 ) min { 1 , λ } and C = max { 1 , λ } C=\max\{1,\sqrt{\lambda}\} C = max { 1 , λ } . Here Φ \Phi Φ and ϕ \phi ϕ denote the standard normal distribution function and density. We prove the lower bound by evaluating one prior in V M , Σ ( ψ ) V_{M,\Sigma}(\psi) V M , Σ ( ψ ) , and the upper bound using the sample-only plug-in policy.
For the lower bound, first note that V M , Σ ( ψ ) ≥ V M , λ I d ( ψ ) V_{M,\Sigma}(\psi)\ge V_{M,\lambda I_d}(\psi) V M , Σ ( ψ ) ≥ V M , λ I d ( ψ ) because Σ ⪰ λ I d \Sigma\succeq\lambda I_d Σ ⪰ λ I d . Indeed, write Y Σ = Y 1 + Z ′ Y_\Sigma=Y_1+Z' Y Σ = Y 1 + Z ′ with Y 1 = T + Z 1 Y_1=T+Z_1 Y 1 = T + Z 1 , Z 1 ∼ N ( 0 , λ I d ) Z_1\sim\mathcal{N}(0,\lambda I_d) Z 1 ∼ N ( 0 , λ I d ) , and Z ′ ∼ N ( 0 , Σ − λ I d ) Z'\sim\mathcal{N}(0,\Sigma-\lambda I_d) Z ′ ∼ N ( 0 , Σ − λ I d ) independent of ( T , Z 1 ) (T,Z_1) ( T , Z 1 ) . Since T → Y 1 → Y Σ T\to Y_1\to Y_\Sigma T → Y 1 → Y Σ is a Markov chain, m π , Σ ( Y Σ ) = E [ m π , λ I d ( Y 1 ) ∣ Y Σ ] m_{\pi,\Sigma}(Y_\Sigma)=\mathbb{E}[m_{\pi,\lambda I_d}(Y_1)\mid Y_\Sigma] m π , Σ ( Y Σ ) = E [ m π , λ I d ( Y 1 ) ∣ Y Σ ] , and Jensen’s inequality for the convex function ψ \psi ψ gives E ψ ( m π , Σ ( Y Σ ) ) ≤ E ψ ( m π , λ I d ( Y 1 ) ) \mathbb{E}\psi(m_{\pi,\Sigma}(Y_\Sigma))\le\mathbb{E}\psi(m_{\pi,\lambda I_d}(Y_1)) E ψ ( m π , Σ ( Y Σ )) ≤ E ψ ( m π , λ I d ( Y 1 )) . Hence the Bayes gap under Σ \Sigma Σ is at least that under λ I d \lambda I_d λ I d for every prior π \pi π , and taking suprema over π \pi π proves the claim. Evaluate the latter constant in the experiment Y = T + Z Y=T+Z Y = T + Z with Z ∼ N ( 0 , λ I d ) Z\sim\mathcal{N}(0,\lambda I_d) Z ∼ N ( 0 , λ I d ) . Set
h = min { λ , M d − s } , P π , λ I d { T j = h } = P π , λ I d { T j = − h } = 1 2 , j ∈ S 0 , h=\min\left\{\sqrt{\lambda},\frac{M}{\sqrt{d-s}}\right\},\qquad\mathbb{P}_{\pi,\lambda I_d}\{T_j=h\}=\mathbb{P}_{\pi,\lambda I_d}\{T_j=-h\}=\frac{1}{2},\qquad j\in S_0, h = min { λ , d − s M } , P π , λ I d { T j = h } = P π , λ I d { T j = − h } = 2 1 , j ∈ S 0 ,
with these coordinates independent and T j = 0 T_j=0 T j = 0 for j ∉ S 0 j\notin S_0 j ∈ / S 0 . Every point in the support satisfies
∥ T ∥ 2 2 = ( d − s ) h 2 ≤ M 2 , \lVert T\rVert_2^2=(d-s)h^2\le M^2, ∥ T ∥ 2 2 = ( d − s ) h 2 ≤ M 2 ,
so π ∈ P ( B 2 ( M ) ) \pi\in\mathcal{P}(B_2(M)) π ∈ P ( B 2 ( M )) . When Y = T + Z Y=T+Z Y = T + Z , independence across coordinates implies that, for j ∈ S 0 j\in S_0 j ∈ S 0 , the posterior of T j T_j T j depends only on Y j Y_j Y j . Bayes’ formula gives
P π , λ I d { T j = h ∣ Y } P π , λ I d { T j = − h ∣ Y } = ϕ ( ( Y j − h ) / λ ) ϕ ( ( Y j + h ) / λ ) = e 2 h Y j / λ , \frac{\mathbb{P}_{\pi,\lambda I_d}\{T_j=h\mid Y\}}{\mathbb{P}_{\pi,\lambda I_d}\{T_j=-h\mid Y\}}=\frac{\phi((Y_j-h)/\sqrt{\lambda})}{\phi((Y_j+h)/\sqrt{\lambda})}=e^{2hY_j/\lambda}, P π , λ I d { T j = − h ∣ Y } P π , λ I d { T j = h ∣ Y } = ϕ (( Y j + h ) / λ ) ϕ (( Y j − h ) / λ ) = e 2 h Y j / λ ,
m π , λ I d , j ( Y ) = h e h Y j / λ − e − h Y j / λ e h Y j / λ + e − h Y j / λ . m_{\pi,\lambda I_d,j}(Y)=h\frac{e^{hY_j/\lambda}-e^{-hY_j/\lambda}}{e^{hY_j/\lambda}+e^{-hY_j/\lambda}}. m π , λ I d , j ( Y ) = h e h Y j / λ + e − h Y j / λ e h Y j / λ − e − h Y j / λ .
In particular, m π , λ I d , j ( Y ) m_{\pi,\lambda I_d,j}(Y) m π , λ I d , j ( Y ) has the same sign as Y j Y_j Y j , and the tower property yields
E π , λ I d [ ( m π , λ I d , j ( Y ) ) + ] = E π , λ I d [ m π , λ I d , j ( Y ) I { Y j > 0 } ] = E π , λ I d [ T j I { Y j > 0 } ] . \mathbb{E}_{\pi,\lambda I_d}[(m_{\pi,\lambda I_d,j}(Y))_+]=\mathbb{E}_{\pi,\lambda I_d}[m_{\pi,\lambda I_d,j}(Y)\mathbb{I}\{Y_j>0\}]=\mathbb{E}_{\pi,\lambda I_d}[T_j\mathbb{I}\{Y_j>0\}]. E π , λ I d [( m π , λ I d , j ( Y ) ) + ] = E π , λ I d [ m π , λ I d , j ( Y ) I { Y j > 0 }] = E π , λ I d [ T j I { Y j > 0 }] .
Since Y j ∣ ( T j = ± h ) ∼ N ( ± h , λ ) Y_j\mid(T_j=\pm h)\sim\mathcal{N}(\pm h,\lambda) Y j ∣ ( T j = ± h ) ∼ N ( ± h , λ ) , the expected gap in coordinate j j j is exactly
E π , λ I d [ ( T j ) + − ( m π , λ I d , j ( Y ) ) + ] = E π , λ I d [ ( T j ) + − T j I { Y j > 0 } ] = h P π , λ I d { T j = h , Y j ≤ 0 } + h P π , λ I d { T j = − h , Y j > 0 } = h 2 Φ ( − h / λ ) + h 2 Φ ( − h / λ ) = h Φ ( − h / λ ) . \begin{aligned}
\mathbb{E}_{\pi,\lambda I_d}[(T_j)_+-(m_{\pi,\lambda I_d,j}(Y))_+]=\mathbb{E}_{\pi,\lambda I_d}[(T_j)_+-T_j\mathbb{I}\{Y_j>0\}] \\
&=h\mathbb{P}_{\pi,\lambda I_d}\{T_j=h,Y_j\le0\}+h\mathbb{P}_{\pi,\lambda I_d}\{T_j=-h,Y_j>0\} \\
&=\frac{h}{2}\Phi(-h/\sqrt{\lambda})+\frac{h}{2}\Phi(-h/\sqrt{\lambda})=h\Phi(-h/\sqrt{\lambda}).
\end{aligned} E π , λ I d [( T j ) + − ( m π , λ I d , j ( Y ) ) + ] = E π , λ I d [( T j ) + − T j I { Y j > 0 }] = h P π , λ I d { T j = h , Y j ≤ 0 } + h P π , λ I d { T j = − h , Y j > 0 } = 2 h Φ ( − h / λ ) + 2 h Φ ( − h / λ ) = h Φ ( − h / λ ) .
Summing over S 0 S_0 S 0 , using the cancellation of the linear terms established above and h ≤ λ h\le\sqrt{\lambda} h ≤ λ , we obtain
V M , Σ ( ψ ) ≥ E π , λ I d [ ψ ( T ) − ψ ( m π , λ I d ( Y ) ) ] = ( d − s ) h Φ ( − h / λ ) ≥ Φ ( − 1 ) min { M d − s , λ ( d − s ) } . \begin{aligned}
V_{M,\Sigma}(\psi)\ge\mathbb{E}_{\pi,\lambda I_d}[\psi(T)-\psi(m_{\pi,\lambda I_d}(Y))] \\
&=(d-s)h\Phi(-h/\sqrt{\lambda}) \\
&\ge\Phi(-1)\min\{M\sqrt{d-s},\sqrt{\lambda}(d-s)\}.
\end{aligned} V M , Σ ( ψ ) ≥ E π , λ I d [ ψ ( T ) − ψ ( m π , λ I d ( Y ))] = ( d − s ) h Φ ( − h / λ ) ≥ Φ ( − 1 ) min { M d − s , λ ( d − s )} .
Since min { M d − s , λ ( d − s ) } ≥ min { 1 , λ } min { M d − s , d − s } \min\{M\sqrt{d-s},\sqrt{\lambda}(d-s)\}\ge\min\{1,\sqrt{\lambda}\}\min\{M\sqrt{d-s},d-s\} min { M d − s , λ ( d − s )} ≥ min { 1 , λ } min { M d − s , d − s } , together with (E.2), this proves the lower bound in (E.3).
For the upper bound, use the plug-in policy w ^ = w ∗ ( x ˉ ) \hat{w}=w_*(\bar{x}) w ^ = w ∗ ( x ˉ ) , whose coordinates are w ^ j = I { x ˉ j > 0 } \hat{w}_j=\mathbb{I}\{\bar{x}_j>0\} w ^ j = I { x ˉ j > 0 } . At a true mean μ n , t \boldsymbol{\mu}_{n,t} μ n , t , choosing the wrong sign in coordinate j j j costs ∣ μ n , t , j ∣ |\mu_{n,t,j}| ∣ μ n , t , j ∣ and occurs with probability Φ ( − n ∣ μ n , t , j ∣ / Σ j j ) \Phi(-\sqrt{n}|\mu_{n,t,j}|/\sqrt{\Sigma_{jj}}) Φ ( − n ∣ μ n , t , j ∣/ Σ j j ) . Thus
E μ n , t , Σ R ( w ^ ) = ∑ j = 1 d ∣ μ n , t , j ∣ Φ ( − n ∣ μ n , t , j ∣ / Σ j j ) = 1 n ∑ j ∈ S 0 ∣ t j ∣ Φ ( − ∣ t j ∣ / Σ j j ) + ∑ j ∉ S 0 ∣ μ j + t j n ∣ Φ ( − n Σ j j ∣ μ j + t j n ∣ ) . \begin{aligned}
\mathbb{E}_{\boldsymbol{\mu}_{n,t},\Sigma}R(\hat{w})=\sum_{j=1}^{d}|\mu_{n,t,j}|\Phi\left(-\sqrt{n}|\mu_{n,t,j}|/\sqrt{\Sigma_{jj}}\right) \\
&=\frac{1}{\sqrt{n}}\sum_{j\in S_0}|t_j|\Phi\left(-|t_j|/\sqrt{\Sigma_{jj}}\right)+\sum_{j\notin S_0}\left|\mu_j+\frac{t_j}{\sqrt{n}}\right|\Phi\left(-\frac{\sqrt{n}}{\sqrt{\Sigma_{jj}}}\left|\mu_j+\frac{t_j}{\sqrt{n}}\right|\right).
\end{aligned} E μ n , t , Σ R ( w ^ ) = j = 1 ∑ d ∣ μ n , t , j ∣Φ ( − n ∣ μ n , t , j ∣/ Σ j j ) = n 1 j ∈ S 0 ∑ ∣ t j ∣Φ ( − ∣ t j ∣/ Σ j j ) + j ∈ / S 0 ∑ μ j + n t j Φ ( − Σ j j n μ j + n t j ) .
For the first sum, Cauchy–Schwarz gives
∑ j ∈ S 0 ∣ t j ∣ Φ ( − ∣ t j ∣ / Σ j j ) ≤ 1 2 ∑ j ∈ S 0 ∣ t j ∣ ≤ 1 2 d − s ∥ t ∥ 2 ≤ 1 2 M d − s . \sum_{j\in S_0}|t_j|\Phi\left(-|t_j|/\sqrt{\Sigma_{jj}}\right)\le\frac{1}{2}\sum_{j\in S_0}|t_j|\le\frac{1}{2}\sqrt{d-s}\lVert t\rVert_2\le\frac{1}{2}M\sqrt{d-s}. j ∈ S 0 ∑ ∣ t j ∣Φ ( − ∣ t j ∣/ Σ j j ) ≤ 2 1 j ∈ S 0 ∑ ∣ t j ∣ ≤ 2 1 d − s ∥ t ∥ 2 ≤ 2 1 M d − s .
Also, for every x ≥ 0 x\ge0 x ≥ 0 , by the Mills’ ratio,
x Φ ( − x ) ≤ ϕ ( x ) ≤ 1 2 π < 1 2 . x\Phi(-x)\le\phi(x)\le\frac{1}{\sqrt{2\pi}}<\frac{1}{2}. x Φ ( − x ) ≤ ϕ ( x ) ≤ 2 π 1 < 2 1 .
In particular, ∣ t j ∣ Φ ( − ∣ t j ∣ / Σ j j ) ≤ Σ j j / 2 π ≤ λ / 2 |t_j|\Phi(-|t_j|/\sqrt{\Sigma_{jj}})\le\sqrt{\Sigma_{jj}}/\sqrt{2\pi}\le\sqrt{\lambda}/2 ∣ t j ∣Φ ( − ∣ t j ∣/ Σ j j ) ≤ Σ j j / 2 π ≤ λ /2 . There are d − s d-s d − s terms, so these two bounds imply
sup t ∈ B 2 ( M ) ∑ j ∈ S 0 ∣ t j ∣ Φ ( − ∣ t j ∣ / Σ j j ) ≤ 1 2 min { M d − s , λ ( d − s ) } . \sup_{t\in B_2(M)}\sum_{j\in S_0}|t_j|\Phi\left(-|t_j|/\sqrt{\Sigma_{jj}}\right)\le\frac{1}{2}\min\{M\sqrt{d-s},\sqrt{\lambda}(d-s)\}. t ∈ B 2 ( M ) sup j ∈ S 0 ∑ ∣ t j ∣Φ ( − ∣ t j ∣/ Σ j j ) ≤ 2 1 min { M d − s , λ ( d − s )} .
If s > 0 s>0 s > 0 , write δ = min j ∉ S 0 ∣ μ j ∣ > 0 \delta=\min_{j\notin S_0}|\mu_j|>0 δ = min j ∈ / S 0 ∣ μ j ∣ > 0 . Whenever M / n ≤ δ / 2 M/\sqrt{n}\le\delta/2 M / n ≤ δ /2 , every t ∈ B 2 ( M ) t\in B_2(M) t ∈ B 2 ( M ) satisfies
∣ μ j + t j n ∣ ≥ ∣ μ j ∣ − M n ≥ δ 2 , j ∉ S 0 . \left|\mu_j+\frac{t_j}{\sqrt{n}}\right|\ge|\mu_j|-\frac{M}{\sqrt{n}}\ge\frac{\delta}{2},\qquad j\notin S_0. μ j + n t j ≥ ∣ μ j ∣ − n M ≥ 2 δ , j ∈ / S 0 .
Using x Φ ( − x ) ≤ ϕ ( x ) x\Phi(-x)\le\phi(x) x Φ ( − x ) ≤ ϕ ( x ) again bounds each term in the second sum by
∣ μ j + t j n ∣ Φ ( − n Σ j j ∣ μ j + t j n ∣ ) ≤ λ 2 π n e − n δ 2 / ( 8 λ ) . \left|\mu_j+\frac{t_j}{\sqrt{n}}\right|\Phi\left(-\frac{\sqrt{n}}{\sqrt{\Sigma_{jj}}}\left|\mu_j+\frac{t_j}{\sqrt{n}}\right|\right)\le\frac{\sqrt{\lambda}}{\sqrt{2\pi n}}e^{-n\delta^2/(8\lambda)}. μ j + n t j Φ ( − Σ j j n μ j + n t j ) ≤ 2 π n λ e − n δ 2 / ( 8 λ ) .
Consequently,
R n , M Σ ( μ ) ≤ sup t ∈ B 2 ( M ) E μ n , t , Σ R ( w ∗ ( x ˉ ) ) ≤ min { M d − s , λ ( d − s ) } 2 n + s λ 2 π n e − n δ 2 / ( 8 λ ) . R^\Sigma_{n,M}(\mu)\le\sup_{t\in B_2(M)}\mathbb{E}_{\mu_{n,t},\Sigma}R(w_*(\bar{x}))\le\frac{\min\{M\sqrt{d-s},\sqrt{\lambda}(d-s)\}}{2\sqrt{n}}+\frac{s\sqrt{\lambda}}{\sqrt{2\pi n}}e^{-n\delta^2/(8\lambda)}. R n , M Σ ( μ ) ≤ t ∈ B 2 ( M ) sup E μ n , t , Σ R ( w ∗ ( x ˉ )) ≤ 2 n min { M d − s , λ ( d − s )} + 2 π n s λ e − n δ 2 / ( 8 λ ) .
For 0 < s < d 0<s<d 0 < s < d and fixed d , μ , M d,\mu,M d , μ , M , the exponential term is eventually at most min { M d − s , λ ( d − s ) } / ( 2 n ) \min\{M\sqrt{d-s},\sqrt{\lambda}(d-s)\}/(2\sqrt{n}) min { M d − s , λ ( d − s )} / ( 2 n ) . If s = 0 s=0 s = 0 , that term is absent. The upper bound is uniform over Σ ∈ C \Sigma\in\mathcal{C} Σ ∈ C and uses the same plug-in rule. Combining it with the preceding lower bound gives, for s < d s<d s < d and all sufficiently large n n n ,
Φ ( − 1 ) min { M d − s , λ ( d − s ) } n ≤ R n , M Σ ( μ ) ≤ min { M d − s , λ ( d − s ) } n . (115) \frac{\Phi(-1)\min\{M\sqrt{d-s},\sqrt{\lambda}(d-s)\}}{\sqrt{n}}\le R^\Sigma_{n,M}(\mu)\le\frac{\min\{M\sqrt{d-s},\sqrt{\lambda}(d-s)\}}{\sqrt{n}}. \tag*{(115)} n Φ ( − 1 ) min { M d − s , λ ( d − s )} ≤ R n , M Σ ( μ ) ≤ n min { M d − s , λ ( d − s )} . (115)
Thus (E.3) holds with the stated constants, and the same plug-in policy attains the local minimax rate.
If s < d s<d s < d , then for any j ∈ S 0 j\in S_0 j ∈ S 0 and M > 0 M>0 M > 0 , ψ ( M e j ) + ψ ( − M e j ) = M > 2 ψ ( 0 ) = 0 \psi(Me_j)+\psi(-Me_j)=M>2\psi(0)=0 ψ ( M e j ) + ψ ( − M e j ) = M > 2 ψ ( 0 ) = 0 . Thus ψ \psi ψ is nonaffine on B 2 ( M ) B_2(M) B 2 ( M ) , and V M , Σ ( ψ ) > 0 V_{M,\Sigma}(\psi)>0 V M , Σ ( ψ ) > 0 by Theorem 3.3. The exact identity (E.2) in particular gives
lim n → ∞ n R n , M Σ ( μ ) = V M , Σ ( ψ ) . \lim_{n\to\infty}\sqrt{n}R^\Sigma_{n,M}(\mu)=V_{M,\Sigma}(\psi). n → ∞ lim n R n , M Σ ( μ ) = V M , Σ ( ψ ) .
Now we consider the case s = d s=d s = d . In this case S 0 = ∅ S_0=\varnothing S 0 = ∅ and δ = min 1 ≤ j ≤ d ∣ μ j ∣ > 0 \delta=\min_{1\le j\le d}|\mu_j|>0 δ = min 1 ≤ j ≤ d ∣ μ j ∣ > 0 . If M / n < δ M/\sqrt{n}<\delta M / n < δ , then μ j + t j / n \mu_j+t_j/\sqrt{n} μ j + t j / n shares the same sign as μ j \mu_j μ j for every j j j and every t ∈ B 2 ( M ) t\in B_2(M) t ∈ B 2 ( M ) . Consequently,
w ∗ , j ( μ n , t ) = w ∗ , j ( μ ) = I { μ j > 0 } , j = 1 , … , d , t ∈ B 2 ( M ) . w_{*,j}(\mu_{n,t})=w_{*,j}(\mu)=\mathbb{I}\{\mu_j>0\},\qquad j=1,\ldots,d,\quad t\in B_2(M). w ∗ , j ( μ n , t ) = w ∗ , j ( μ ) = I { μ j > 0 } , j = 1 , … , d , t ∈ B 2 ( M ) .
The constant policy w ∗ ( μ ) w_*(\mu) w ∗ ( μ ) is admissible in the local benchmark and is optimal throughout the local ball. Therefore,
0 ≤ R n , M Σ ( μ ) ≤ sup t ∈ B 2 ( M ) E μ n , t , Σ R ( w ∗ ( μ ) ) = 0. 0\le R^\Sigma_{n,M}(\mu)\le\sup_{t\in B_2(M)}\mathbb{E}_{\mu_{n,t},\Sigma}R(w_*(\mu))=0. 0 ≤ R n , M Σ ( μ ) ≤ t ∈ B 2 ( M ) sup E μ n , t , Σ R ( w ∗ ( μ )) = 0.
Equivalently, ψ ≡ 0 \psi\equiv0 ψ ≡ 0 and (E.2) gives the same conclusion.
For SAA, coordinatewise optimality gives w ∗ , j ( x ˉ ) = I { x ˉ j > 0 } w_{*,j}(\bar{x})=\mathbb{I}\{\bar{x}_j>0\} w ∗ , j ( x ˉ ) = I { x ˉ j > 0 } . If μ n , t , j > 0 \mu_{n,t,j}>0 μ n , t , j > 0 , choosing 0 instead of 1 loses μ n , t , j \mu_{n,t,j} μ n , t , j ; if μ n , t , j < 0 \mu_{n,t,j}<0 μ n , t , j < 0 , choosing 1 instead of 0 loses − μ n , t , j -\mu_{n,t,j} − μ n , t , j . Thus
R ( w ∗ ( x ˉ ) ) = ∑ j = 1 d ∣ μ n , t , j ∣ I { x ˉ j μ n , t , j < 0 } . R(w_*(\bar{x}))=\sum_{j=1}^{d}|\mu_{n,t,j}|\mathbb{I}\{\bar{x}_j\mu_{n,t,j}<0\}. R ( w ∗ ( x ˉ )) = j = 1 ∑ d ∣ μ n , t , j ∣ I { x ˉ j μ n , t , j < 0 } .
Under the true mean μ n , t \mu_{n,t} μ n , t , x ˉ j ∼ N ( μ n , t , j , Σ j j / n ) \bar{x}_j\sim\mathcal{N}(\mu_{n,t,j},\Sigma_{jj}/n) x ˉ j ∼ N ( μ n , t , j , Σ j j / n ) . Hence
P μ n , t , Σ { x ˉ j μ n , t , j < 0 } = { Φ ( − n μ n , t , j / Σ j j ) , μ n , t , j > 0 , 1 − Φ ( − n μ n , t , j / Σ j j ) , μ n , t , j < 0 = Φ ( − n ∣ μ n , t , j ∣ Σ j j ) . \mathbb{P}_{\mu_{n,t},\Sigma}\{\bar{x}_j\mu_{n,t,j}<0\}=
\begin{cases}
\Phi(-\sqrt{n}\mu_{n,t,j}/\sqrt{\Sigma_{jj}}), & \mu_{n,t,j}>0,\\
1-\Phi(-\sqrt{n}\mu_{n,t,j}/\sqrt{\Sigma_{jj}}), & \mu_{n,t,j}<0
\end{cases}
=\Phi\left(-\frac{\sqrt{n}|\mu_{n,t,j}|}{\sqrt{\Sigma_{jj}}}\right). P μ n , t , Σ { x ˉ j μ n , t , j < 0 } = { Φ ( − n μ n , t , j / Σ j j ) , 1 − Φ ( − n μ n , t , j / Σ j j ) , μ n , t , j > 0 , μ n , t , j < 0 = Φ ( − Σ j j n ∣ μ n , t , j ∣ ) .
Moreover, if M / n ≤ δ / 2 M/\sqrt{n}\le\delta/2 M / n ≤ δ /2 , then for every t ∈ B 2 ( M ) t\in B_2(M) t ∈ B 2 ( M ) and every j j j ,
∣ μ n , t , j ∣ = ∣ μ j + t j n ∣ ≥ ∣ μ j ∣ − ∣ t j ∣ n ≥ δ − M n ≥ δ 2 . |\mu_{n,t,j}|=\left|\mu_j+\frac{t_j}{\sqrt{n}}\right|\ge|\mu_j|-\frac{|t_j|}{\sqrt{n}}\ge\delta-\frac{M}{\sqrt{n}}\ge\frac{\delta}{2}. ∣ μ n , t , j ∣ = μ j + n t j ≥ ∣ μ j ∣ − n ∣ t j ∣ ≥ δ − n M ≥ 2 δ .
For x ≥ 0 x\ge0 x ≥ 0 , the Mills’ ratio gives x Φ ( − x ) ≤ ϕ ( x ) x\Phi(-x)\le\phi(x) x Φ ( − x ) ≤ ϕ ( x ) . Applying this inequality with x = n ∣ μ n , t , j ∣ / Σ j j x=\sqrt{n}|\mu_{n,t,j}|/\sqrt{\Sigma_{jj}} x = n ∣ μ n , t , j ∣/ Σ j j gives
∣ μ n , t , j ∣ Φ ( − n ∣ μ n , t , j ∣ Σ j j ) ≤ Σ j j 2 π n exp ( − n ∣ μ n , t , j ∣ 2 2 Σ j j ) ≤ λ ˉ 2 π n exp ( − n δ 2 8 λ ˉ ) . |\mu_{n,t,j}|\Phi\left(-\frac{\sqrt{n}|\mu_{n,t,j}|}{\sqrt{\Sigma_{jj}}}\right)
\le\frac{\sqrt{\Sigma_{jj}}}{\sqrt{2\pi n}}\exp\left(-\frac{n|\mu_{n,t,j}|^2}{2\Sigma_{jj}}\right)
\le\frac{\sqrt{\bar{\lambda}}}{\sqrt{2\pi n}}\exp\left(-\frac{n\delta^2}{8\bar{\lambda}}\right). ∣ μ n , t , j ∣Φ ( − Σ j j n ∣ μ n , t , j ∣ ) ≤ 2 π n Σ j j exp ( − 2 Σ j j n ∣ μ n , t , j ∣ 2 ) ≤ 2 π n λ ˉ exp ( − 8 λ ˉ n δ 2 ) .
Here the second inequality uses Σ j j ≤ λ ˉ \Sigma_{jj}\le\bar{\lambda} Σ j j ≤ λ ˉ and
n ∣ μ n , t , j ∣ 2 2 Σ j j ≥ n ( δ / 2 ) 2 2 λ ˉ = n δ 2 8 λ ˉ . \frac{n|\mu_{n,t,j}|^2}{2\Sigma_{jj}}\ge\frac{n(\delta/2)^2}{2\bar{\lambda}}=\frac{n\delta^2}{8\bar{\lambda}}. 2 Σ j j n ∣ μ n , t , j ∣ 2 ≥ 2 λ ˉ n ( δ /2 ) 2 = 8 λ ˉ n δ 2 .
By linearity of expectation, summing the coordinate losses yields the following bound, regardless of dependence between coordinates:
sup t ∈ B 2 ( M ) E μ , n , t , Σ R ( w ∗ ( x ˉ ) ) ≤ d λ 2 π n exp ( − n δ 2 8 λ ) , Σ ∈ C . \sup_{t\in B_2(M)} \mathbb{E}_{\mu,n,t,\Sigma} R(w_\ast(\bar{x})) \le\frac{d\sqrt{\lambda}}{\sqrt{2\pi n}}\exp\left(-\frac{n\delta^2}{8\lambda}\right),\qquad\Sigma\in\mathcal{C}. t ∈ B 2 ( M ) sup E μ , n , t , Σ R ( w ∗ ( x ˉ )) ≤ 2 π n d λ exp ( − 8 λ n δ 2 ) , Σ ∈ C .
For fixed d d d , μ \mu μ , and M M M , this bound holds for every n ≥ 4 M 2 / δ 2 n\ge4M^2/\delta^2 n ≥ 4 M 2 / δ 2 , uniformly over Σ ∈ C \Sigma\in\mathcal{C} Σ ∈ C . Since δ 2 / ( 8 λ ) > 0 \delta^2/(8\lambda)>0 δ 2 / ( 8 λ ) > 0 , SAA has exponentially decaying maximal expected regret on each fixed local ball.
A strongly convex set with center-dependent rates: details for Example 4.9
Consider the strongly convex set
Ω = B 4 / 3 2 = { w : ∣ w 1 ∣ 4 / 3 + ∣ w 2 ∣ 4 / 3 ≤ 1 } , \Omega=B_{4/3}^2=\{w:|w_1|^{4/3}+|w_2|^{4/3}\le1\}, Ω = B 4/3 2 = { w : ∣ w 1 ∣ 4/3 + ∣ w 2 ∣ 4/3 ≤ 1 } ,
which gives f ∗ ( μ ) = ( μ 1 4 + μ 2 4 ) 1 / 4 f_\ast(\mu)=(\mu_1^4+\mu_2^4)^{1/4} f ∗ ( μ ) = ( μ 1 4 + μ 2 4 ) 1/4 . The local minimax rate depends on the reference center μ \mu μ even though the feasible set remains fixed. We first take μ = e 1 \mu=e_1 μ = e 1 , where w ∗ ( e 1 ) = e 1 w_\ast(e_1)=e_1 w ∗ ( e 1 ) = e 1 and H e 1 = ∇ 2 f ∗ ( e 1 ) = 0 H_{e_1}=\nabla^2f_\ast(e_1)=0 H e 1 = ∇ 2 f ∗ ( e 1 ) = 0 . Uniformly for bounded t t t ,
f ∗ ( e 1 + ε t ) = 1 + ε t 1 + ε 4 t 2 4 4 + o ( ε 4 ) . f_\ast(e_1+\varepsilon t)=1+\varepsilon t_1+\frac{\varepsilon^4t_2^4}{4}+o(\varepsilon^4). f ∗ ( e 1 + εt ) = 1 + ε t 1 + 4 ε 4 t 2 4 + o ( ε 4 ) .
Theorem Theorem 3.3 therefore applies with a ( ε ) = ε 4 a(\varepsilon)=\varepsilon^4 a ( ε ) = ε 4 and ψ ( t ) = t 2 4 / 4 \psi(t)=t_2^4/4 ψ ( t ) = t 2 4 /4 , yielding
R n , M Σ ( e 1 ) ≍ M n − 2 R^\Sigma_{n,M}(e_1)\asymp_M n^{-2} R n , M Σ ( e 1 ) ≍ M n − 2
for every fixed M > 0 M>0 M > 0 .
More generally, at any fixed nonzero point on either coordinate axis, the first nonzero local remainder is quartic in the transverse coordinate, giving the same n − 2 n^{-2} n − 2 rate.
Next take μ = 2 − 1 / 4 ( 1 , 1 ) T \mu=2^{-1/4}(1,1)^\mathsf{T} μ = 2 − 1/4 ( 1 , 1 ) T , so that f ∗ ( μ ) = 1 f_\ast(\mu)=1 f ∗ ( μ ) = 1 as before. Direct differentiation gives
H μ = 3 2 2 ( 1 − 1 − 1 1 ) , r μ ( ε t ) = ε 2 3 ( t 1 − t 2 ) 2 4 2 + o ( ε 2 ) , H_\mu=\frac{3}{2\sqrt{2}}\begin{pmatrix}1&-1\\-1&1\end{pmatrix},\qquad r_\mu(\varepsilon t)=\varepsilon^2\frac{3(t_1-t_2)^2}{4\sqrt{2}}+o(\varepsilon^2), H μ = 2 2 3 ( 1 − 1 − 1 1 ) , r μ ( εt ) = ε 2 4 2 3 ( t 1 − t 2 ) 2 + o ( ε 2 ) ,
uniformly for bounded t t t . The limit ψ ( t ) = 3 ( t 1 − t 2 ) 2 / ( 4 2 ) \psi(t)=3(t_1-t_2)^2/(4\sqrt{2}) ψ ( t ) = 3 ( t 1 − t 2 ) 2 / ( 4 2 ) is nonaffine on every B 2 ( M ) B_2(M) B 2 ( M ) with M > 0 M>0 M > 0 . Therefore,
R n , M Σ ( 2 − 1 / 4 ( 1 , 1 ) T ) ≍ M n − 1 . R^\Sigma_{n,M}\left(2^{-1/4}(1,1)^\mathsf{T}\right)\asymp_M n^{-1}. R n , M Σ ( 2 − 1/4 ( 1 , 1 ) T ) ≍ M n − 1 .
Theorem Theorem 3.2 also gives the sharp constant when M → ∞ M\to\infty M → ∞ :
lim M → ∞ lim n → ∞ n R n , M Σ ( 2 − 1 / 4 ( 1 , 1 ) T ) = 1 2 tr ( Σ H μ ) = 3 4 2 ( Σ 11 + Σ 22 − 2 Σ 12 ) . \lim_{M\to\infty}\lim_{n\to\infty}nR^\Sigma_{n,M}\left(2^{-1/4}(1,1)^\mathsf{T}\right)=\frac{1}{2}\operatorname{tr}(\Sigma H_\mu)=\frac{3}{4\sqrt{2}}(\Sigma_{11}+\Sigma_{22}-2\Sigma_{12}). M → ∞ lim n → ∞ lim n R n , M Σ ( 2 − 1/4 ( 1 , 1 ) T ) = 2 1 tr ( Σ H μ ) = 4 2 3 ( Σ 11 + Σ 22 − 2 Σ 12 ) .
At any fixed center with μ 1 μ 2 ≠ 0 \mu_1\mu_2\ne0 μ 1 μ 2 = 0 , the Hessian is nonzero and the same n − 1 n^{-1} n − 1 rate holds. Here the ambiguity set is A = { 0 } A=\{0\} A = { 0 } and, under the polar-norm convention, dist ( μ , A ) = f ∗ ( μ ) \operatorname{dist}(\mu,A)=f_\ast(\mu) dist ( μ , A ) = f ∗ ( μ ) .
In particular, the two displayed nonzero centers have the same distance dist ( μ , A ) = 1 \operatorname{dist}(\mu,A)=1 dist ( μ , A ) = 1 , yet their local minimax rates differ.
Finally, take μ = 0 \mu=0 μ = 0 . With the convention w ∗ ( 0 ) = 0 \mathbf{w}_*(0)=0 w ∗ ( 0 ) = 0 , positive homogeneity gives the exact identity
r 0 ( ε t ) = ε ∥ t ∥ 4 , R n , M Σ ( 0 ) = 1 n V M , Σ ( t ↦ ∥ t ∥ 4 ) ≍ M n − 1 / 2 . r_0(\varepsilon t)=\varepsilon\|t\|_4,\qquad R^\Sigma_{n,M}(0)=\frac{1}{\sqrt{n}}V_{M,\Sigma}(t\mapsto\|t\|_4)\asymp_M n^{-1/2}. r 0 ( εt ) = ε ∥ t ∥ 4 , R n , M Σ ( 0 ) = n 1 V M , Σ ( t ↦ ∥ t ∥ 4 ) ≍ M n − 1/2 .
The positivity follows because ∥ t ∥ 4 \|t\|_4 ∥ t ∥ 4 is nonaffine on every B 2 ( M ) B_2(M) B 2 ( M ) with M > 0 M>0 M > 0 . Thus this single strongly convex set exhibits n − 2 n^{-2} n − 2 , n − 1 n^{-1} n − 1 , and n − 1 / 2 n^{-1/2} n − 1/2 local rates at different centers.
All of these rates concern neighborhoods of radius M / n M/\sqrt{n} M / n with M M M fixed. In contrast, every fixed-radius neighborhood of e 1 e_1 e 1 contains interior points with nonzero Hessian and consequently has an n − 1 n^{-1} n − 1 minimax lower bound. If its closure excludes the origin, boundedness of the Hessian on a larger neighborhood and Gaussian tail bounds also give an O ( n − 1 ) O(n^{-1}) O ( n − 1 ) upper bound for the plug-in policy.
More generally, for d ≥ 2 d\ge2 d ≥ 2 , consider
Ω = B 4 / 3 d = { w : ∑ j = 1 d ∣ w j ∣ 4 / 3 ≤ 1 } . \Omega=B^d_{4/3}=\left\{\mathbf{w}:\sum_{j=1}^{d}|w_j|^{4/3}\le1\right\}. Ω = B 4/3 d = { w : j = 1 ∑ d ∣ w j ∣ 4/3 ≤ 1 } .
which gives f ∗ ( μ ) = ∥ μ ∥ 4 f_*(\mu)=\|\mu\|_4 f ∗ ( μ ) = ∥ μ ∥ 4 . For μ ≠ 0 \mu\ne0 μ = 0 , denote
S 0 = { j ∈ [ d ] : μ j = 0 } , s = ∥ μ ∥ 0 = d − ∣ S 0 ∣ . S_0=\{j\in[d]:\mu_j=0\},\qquad s=\|\mu\|_0=d-|S_0|. S 0 = { j ∈ [ d ] : μ j = 0 } , s = ∥ μ ∥ 0 = d − ∣ S 0 ∣.
The maximizer is unique, with w ∗ , j ( μ ) = μ j 3 / ∥ μ ∥ 4 3 w_{*,j}(\mu)=\mu_j^3/\|\mu\|_4^3 w ∗ , j ( μ ) = μ j 3 /∥ μ ∥ 4 3 . Thus A = { 0 } A=\{0\} A = { 0 } and the polar-norm convention gives dist ( μ , A ) = ∥ μ ∥ 4 \operatorname{dist}(\mu,A)=\|\mu\|_4 dist ( μ , A ) = ∥ μ ∥ 4 .
The strong-convexity constant can be chosen independently of d d d . To verify this, set F ( w ) = ∥ w ∥ 4 / 3 2 / 2 F(\mathbf{w})=\|\mathbf{w}\|_{4/3}^2/2 F ( w ) = ∥ w ∥ 4/3 2 /2 . At points with nonzero coordinates, differentiation and Hölder’s inequality give
z T ∇ 2 F ( w ) z ≥ 1 3 ∥ w ∥ 4 / 3 2 / 3 ∑ j = 1 d ∣ w j ∣ − 2 / 3 z j 2 ≥ 1 3 ∥ z ∥ 4 / 3 2 . \mathbf{z}^{\mathsf{T}}\nabla^2F(\mathbf{w})\mathbf{z}\ge\frac{1}{3}\|\mathbf{w}\|_{4/3}^{2/3}\sum_{j=1}^{d}|w_j|^{-2/3}z_j^2\ge\frac{1}{3}\|\mathbf{z}\|_{4/3}^2. z T ∇ 2 F ( w ) z ≥ 3 1 ∥ w ∥ 4/3 2/3 j = 1 ∑ d ∣ w j ∣ − 2/3 z j 2 ≥ 3 1 ∥ z ∥ 4/3 2 .
Integrating along line segments and extending to zero coordinates by continuity proves that F F F is 1 / 3 1/3 1/3 -strongly convex in the ℓ 4 / 3 \ell_{4/3} ℓ 4/3 norm. Since F ( w ∗ ( μ ) ) = 1 / 2 F(\mathbf{w}_*(\mu))=1/2 F ( w ∗ ( μ )) = 1/2 and ∇ F ( w ∗ ( μ ) ) = μ / ∥ μ ∥ 4 \nabla F(\mathbf{w}_*(\mu))=\mu/\|\mu\|_4 ∇ F ( w ∗ ( μ )) = μ /∥ μ ∥ 4 , it follows that
F ( w ) ≥ 1 2 + μ T ( w − w ∗ ( μ ) ) ∥ μ ∥ 4 + 1 6 ∥ w − w ∗ ( μ ) ∥ 4 / 3 2 . F(\mathbf{w})\ge\frac{1}{2}+\frac{\mu^{\mathsf{T}}(\mathbf{w}-\mathbf{w}_*(\mu))}{\|\mu\|_4}+\frac{1}{6}\|\mathbf{w}-\mathbf{w}_*(\mu)\|_{4/3}^2. F ( w ) ≥ 2 1 + ∥ μ ∥ 4 μ T ( w − w ∗ ( μ )) + 6 1 ∥ w − w ∗ ( μ ) ∥ 4/3 2 .
For w ∈ Ω \mathbf{w}\in\Omega w ∈ Ω , we have F ( w ) ≤ 1 / 2 F(\mathbf{w})\le1/2 F ( w ) ≤ 1/2 , hence
μ T ( w ∗ ( μ ) − w ) ≥ ∥ μ ∥ 4 6 ∥ w − w ∗ ( μ ) ∥ 4 / 3 2 . \mu^{\mathsf{T}}(\mathbf{w}_*(\mu)-\mathbf{w})\ge\frac{\|\mu\|_4}{6}\|\mathbf{w}-\mathbf{w}_*(\mu)\|_{4/3}^2. μ T ( w ∗ ( μ ) − w ) ≥ 6 ∥ μ ∥ 4 ∥ w − w ∗ ( μ ) ∥ 4/3 2 .
Thus (11) holds with c Ω = 1 / 6 c_\Omega=1/6 c Ω = 1/6 . Also, for G ∼ N ( 0 , I d ) \mathbf{G}\sim\mathcal{N}(0,I_d) G ∼ N ( 0 , I d ) , norm comparison and Jensen’s inequality yield
2 π d 1 / 4 = d − 3 / 4 E ∥ G ∥ 1 ≤ G ( Ω ) = E ∥ G ∥ 4 ≤ ( E ∑ j = 1 d G j 4 ) 1 / 4 = ( 3 d ) 1 / 4 . \sqrt{\frac{2}{\pi}}d^{1/4}=d^{-3/4}\mathbb{E}\|\mathbf{G}\|_1\le\mathcal{G}(\Omega)=\mathbb{E}\|\mathbf{G}\|_4\le\left(\mathbb{E}\sum_{j=1}^{d}G_j^4\right)^{1/4}=(3d)^{1/4}. π 2 d 1/4 = d − 3/4 E ∥ G ∥ 1 ≤ G ( Ω ) = E ∥ G ∥ 4 ≤ ( E j = 1 ∑ d G j 4 ) 1/4 = ( 3 d ) 1/4 .
Conditional Jensen’s inequality, applied after adding independent Gaussian noise, also gives
λ ‾ G ( Ω ) ≤ G ( Σ 1 / 2 Ω ) ≤ λ ‾ G ( Ω ) . \sqrt{\underline{\lambda}}\,\mathcal{G}(\Omega) \le\mathcal{G}(\Sigma^{1/2}\Omega) \le\sqrt{\overline{\lambda}}\,\mathcal{G}(\Omega). λ G ( Ω ) ≤ G ( Σ 1/2 Ω ) ≤ λ G ( Ω ) .
Consequently, the right-hand side of (14) has order d / ( n ∥ μ ∥ 4 ) \sqrt{d}/(n\lVert\mu\rVert_4) d / ( n ∥ μ ∥ 4 ) , with constants depending only on the fixed spectral bounds.
For every nonzero μ \boldsymbol{\mu} μ , the support function is smooth, and direct differentiation gives
tr ( H μ ) = 3 ∥ μ ∥ 4 3 ∑ j = 1 d μ j 2 − 3 ∥ μ ∥ 4 7 ∑ j = 1 d μ j 6 . \operatorname{tr}(H_{\mu}) = \frac{3}{\lVert\boldsymbol{\mu}\rVert_4^3}\sum_{j=1}^{d}\mu_j^2-\frac{3}{\lVert\boldsymbol{\mu}\rVert_4^7}\sum_{j=1}^{d}\mu_j^6. tr ( H μ ) = ∥ μ ∥ 4 3 3 j = 1 ∑ d μ j 2 − ∥ μ ∥ 4 7 3 j = 1 ∑ d μ j 6 .
For general Σ \Sigma Σ , the corresponding weighted trace is
tr ( Σ H μ ) = 3 ∥ μ ∥ 4 3 ∑ j Σ j j μ j 2 − 3 ∥ μ ∥ 4 7 ∑ i , j Σ i j μ i 3 μ j 3 , \operatorname{tr}(\Sigma H_{\mu}) = \frac{3}{\lVert\boldsymbol{\mu}\rVert_4^3}\sum_j\Sigma_{jj}\mu_j^2-\frac{3}{\lVert\boldsymbol{\mu}\rVert_4^7}\sum_{i,j}\Sigma_{ij}\mu_i^3\mu_j^3, tr ( Σ H μ ) = ∥ μ ∥ 4 3 3 j ∑ Σ j j μ j 2 − ∥ μ ∥ 4 7 3 i , j ∑ Σ ij μ i 3 μ j 3 ,
because ( H μ ) i j = 3 μ i 2 I { i = j } / ∥ μ ∥ 4 3 − 3 μ i 3 μ j 3 / ∥ μ ∥ 4 7 (H_{\mu})_{ij}=3\mu_i^2\mathbb{I}\{i=j\}/\lVert\boldsymbol{\mu}\rVert_4^3-3\mu_i^3\mu_j^3/\lVert\boldsymbol{\mu}\rVert_4^7 ( H μ ) ij = 3 μ i 2 I { i = j } / ∥ μ ∥ 4 3 − 3 μ i 3 μ j 3 / ∥ μ ∥ 4 7 . Moreover,
λ ‾ tr ( H μ ) ≤ tr ( Σ H μ ) ≤ λ ‾ tr ( H μ ) . \underline{\lambda}\operatorname{tr}(H_{\mu})\leq\operatorname{tr}(\Sigma H_{\mu})\leq\overline{\lambda}\operatorname{tr}(H_{\mu}). λ tr ( H μ ) ≤ tr ( Σ H μ ) ≤ λ tr ( H μ ) .
Suppose now that the s s s nonzero coordinates have equal absolute value, so ∣ μ j ∣ = ∥ μ ∥ 4 s − 1 / 4 |\mu_j|=\lVert\boldsymbol{\mu}\rVert_4s^{-1/4} ∣ μ j ∣ = ∥ μ ∥ 4 s − 1/4 for j ∉ S 0 j\notin S_0 j ∈ / S 0 . Then
∑ j = 1 d μ j 2 = ∥ μ ∥ 4 2 s , ∑ j = 1 d μ j 6 = ∥ μ ∥ 4 6 s , \sum_{j=1}^{d}\mu_j^2=\lVert\boldsymbol{\mu}\rVert_4^2\sqrt{s},\qquad\sum_{j=1}^{d}\mu_j^6=\frac{\lVert\boldsymbol{\mu}\rVert_4^6}{\sqrt{s}}, j = 1 ∑ d μ j 2 = ∥ μ ∥ 4 2 s , j = 1 ∑ d μ j 6 = s ∥ μ ∥ 4 6 ,
tr ( H μ ) = 3 ∥ μ ∥ 4 ( s − 1 s ) . \operatorname{tr}(H_{\mu})=\frac{3}{\lVert\boldsymbol{\mu}\rVert_4}\left(\sqrt{s}-\frac{1}{\sqrt{s}}\right). tr ( H μ ) = ∥ μ ∥ 4 3 ( s − s 1 ) .
For s ≥ 2 s\geq2 s ≥ 2 , Theorem 3.2 therefore gives
lim M → ∞ lim n → ∞ n R n , M Σ ( μ ) = 1 2 tr ( Σ H μ ) ≍ s ∥ μ ∥ 4 . \lim_{M\to\infty}\lim_{n\to\infty}nR_{n,M}^{\Sigma}(\mu)=\frac{1}{2}\operatorname{tr}(\Sigma H_{\mu})\asymp\frac{\sqrt{s}}{\lVert\boldsymbol{\mu}\rVert_4}. M → ∞ lim n → ∞ lim n R n , M Σ ( μ ) = 2 1 tr ( Σ H μ ) ≍ ∥ μ ∥ 4 s .
The comparison constants in the last relation depend only on the fixed spectral bounds and are independent of d d d and s ≥ 2 s\geq2 s ≥ 2 . The plug-in policy has the same asymptotic risk constant uniformly on every fixed local ball by (24) . Thus the fast rate matches the sharp local asymptotic scale when s ≍ d s\asymp d s ≍ d . For 2 ≤ s ≪ d 2\leq s\ll d 2 ≤ s ≪ d , its n − 1 n^{-1} n − 1 power is optimal, but its geometric coefficient exceeds the sharp one by a factor of order d / s \sqrt{d/s} d / s .
When s = 1 s=1 s = 1 , the Hessian vanishes. Coordinate permutations and sign changes preserve Ω \Omega Ω and B 2 ( M ) B_2(M) B 2 ( M ) ; the same transformation must be applied to the covariance matrix, which remains in C \mathcal{C} C . Since the following calculation holds for every Σ ∈ C \Sigma\in\mathcal{C} Σ ∈ C , it suffices to take μ = μ 1 e 1 \boldsymbol{\mu}=\mu_1e_1 μ = μ 1 e 1 with μ 1 > 0 \mu_1>0 μ 1 > 0 , where w ∗ ( μ ) = e 1 w_*(\boldsymbol{\mu})=e_1 w ∗ ( μ ) = e 1 . Uniformly for bounded t t t ,
f ∗ ( μ 1 e 1 + ε t ) = ( ( μ 1 + ε t 1 ) 4 + ε 4 ∑ j = 2 d t j 4 ) 1 / 4 = μ 1 + ε t 1 + ε 4 4 μ 1 3 ∑ j = 2 d t j 4 + o ( ε 4 ) . \begin{aligned}
f_*(\mu_1e_1+\varepsilon t)=\left((\mu_1+\varepsilon t_1)^4+\varepsilon^4\sum_{j=2}^{d}t_j^4\right)^{1/4} \\
&=\mu_1+\varepsilon t_1+\frac{\varepsilon^4}{4\mu_1^3}\sum_{j=2}^{d}t_j^4+o(\varepsilon^4).
\end{aligned} f ∗ ( μ 1 e 1 + εt ) = ( ( μ 1 + ε t 1 ) 4 + ε 4 j = 2 ∑ d t j 4 ) 1/4 = μ 1 + ε t 1 + 4 μ 1 3 ε 4 j = 2 ∑ d t j 4 + o ( ε 4 ) .
Hence Theorem 3.3 applies with a ( ε ) = ε 4 a(\varepsilon)=\varepsilon^4 a ( ε ) = ε 4 and ψ ( t ) = ∑ j = 2 d t j 4 / ( 4 μ 1 3 ) \psi(t)=\sum_{j=2}^{d}t_j^4/(4\mu_1^3) ψ ( t ) = ∑ j = 2 d t j 4 / ( 4 μ 1 3 ) , which is nonaffine on every B 2 ( M ) B_2(M) B 2 ( M ) with M > 0 M>0 M > 0 . For fixed d , μ 1 , M d,\mu_1,M d , μ 1 , M ,
n 2 R n , M Σ ( μ 1 e 1 ) → V M , Σ ( t ↦ 1 4 μ 1 3 ∑ j = 2 d t j 4 ) . n^2R^\Sigma_{n,M}(\mu_1e_1)\to V_{M,\Sigma}\left(t\mapsto\frac{1}{4\mu_1^3}\sum_{j=2}^{d}t_j^4\right). n 2 R n , M Σ ( μ 1 e 1 ) → V M , Σ ( t ↦ 4 μ 1 3 1 j = 2 ∑ d t j 4 ) .
The limit is positive, and the plug-in conclusion of the same theorem gives
R n , M Σ ( μ 1 e 1 ) ≍ sup t ∈ B 2 ( M ) E μ n , t , Σ R ( w ∗ ( x ˉ ) ) ≍ n − 2 . R^\Sigma_{n,M}(\mu_1e_1)\asymp\sup_{t\in B_2(M)}\mathbb{E}_{\mu_{n,t},\Sigma}R(w_*(\bar{x}))\asymp n^{-2}. R n , M Σ ( μ 1 e 1 ) ≍ t ∈ B 2 ( M ) sup E μ n , t , Σ R ( w ∗ ( x ˉ )) ≍ n − 2 .
At zero, Corollary 4.1 and Theorem 3.3 give the n − 1 / 2 n^{-1/2} n − 1/2 rate. The plug-in rule attains all three rates, uniformly over Σ ∈ C \Sigma\in\mathcal{C} Σ ∈ C , without using Σ \Sigma Σ .
Optimal value versus optimal decision on ℓ p ⋆ \ell_{p^\star} ℓ p ⋆ balls
Let d ≥ 2 d\ge2 d ≥ 2 , 1 < p < ∞ 1<p<\infty 1 < p < ∞ , p ⋆ = p / ( p − 1 ) p^\star=p/(p-1) p ⋆ = p / ( p − 1 ) , and Ω = B p ⋆ d \Omega=B_{p^\star}^d Ω = B p ⋆ d . By Hölder duality,
f ∗ ( z ) = ∥ z ∥ p . f_*(z)=\lVert z\rVert_p. f ∗ ( z ) = ∥ z ∥ p .
and, for z ≠ 0 z\ne0 z = 0 , the unique optimizer has coordinates
w ∗ , j ( z ) = sgn ( z j ) ∣ z j ∣ p − 1 ∥ z ∥ p p − 1 . w_{*,j}(z)=\frac{\operatorname{sgn}(z_j)|z_j|^{p-1}}{\lVert z\rVert_p^{p-1}}. w ∗ , j ( z ) = ∥ z ∥ p p − 1 sgn ( z j ) ∣ z j ∣ p − 1 .
Fix μ = μ 1 e 1 \mu=\mu_1e_1 μ = μ 1 e 1 , μ 1 > 0 \mu_1>0 μ 1 > 0 . For t t t in a fixed bounded set and sufficiently small ε \varepsilon ε , μ 1 + ε t 1 > 0 \mu_1+\varepsilon t_1>0 μ 1 + ε t 1 > 0 , and
f ∗ ( μ + ε t ) = ( μ 1 + ε t 1 ) { 1 + ε p ∑ j = 2 d ∣ t j ∣ p ( μ 1 + ε t 1 ) p } 1 / p = μ 1 + ε t 1 + ε p p ( μ 1 + ε t 1 ) p − 1 ∑ j = 2 d ∣ t j ∣ p + o ( ε p ) = μ 1 + ε t 1 + ε p p μ 1 p − 1 ∑ j = 2 d ∣ t j ∣ p + o ( ε p ) , \begin{aligned}
f_*(\mu+\varepsilon t)
&=(\mu_1+\varepsilon t_1)\left\{1+\frac{\varepsilon^p\sum_{j=2}^{d}|t_j|^p}{(\mu_1+\varepsilon t_1)^p}\right\}^{1/p}\\
&=\mu_1+\varepsilon t_1+\frac{\varepsilon^p}{p(\mu_1+\varepsilon t_1)^{p-1}}\sum_{j=2}^{d}|t_j|^p+o(\varepsilon^p)\\
&=\mu_1+\varepsilon t_1+\frac{\varepsilon^p}{p\mu_1^{p-1}}\sum_{j=2}^{d}|t_j|^p+o(\varepsilon^p),
\end{aligned} f ∗ ( μ + εt ) = ( μ 1 + ε t 1 ) { 1 + ( μ 1 + ε t 1 ) p ε p ∑ j = 2 d ∣ t j ∣ p } 1/ p = μ 1 + ε t 1 + p ( μ 1 + ε t 1 ) p − 1 ε p j = 2 ∑ d ∣ t j ∣ p + o ( ε p ) = μ 1 + ε t 1 + p μ 1 p − 1 ε p j = 2 ∑ d ∣ t j ∣ p + o ( ε p ) ,
uniformly on that bounded set. Since w ∗ ( μ ) = e 1 w_*(\mu)=e_1 w ∗ ( μ ) = e 1 ,
r μ ( ε t ) = ε p p μ 1 p − 1 ∑ j = 2 d ∣ t j ∣ p + o ( ε p ) . r_\mu(\varepsilon t)=\frac{\varepsilon^p}{p\mu_1^{p-1}}\sum_{j=2}^{d}|t_j|^p+o(\varepsilon^p). r μ ( εt ) = p μ 1 p − 1 ε p j = 2 ∑ d ∣ t j ∣ p + o ( ε p ) .
Thus a ( ε ) = ε p a(\varepsilon)=\varepsilon^p a ( ε ) = ε p and ψ ( t ) = ( p μ 1 p − 1 ) − 1 ∑ j = 2 d ∣ t j ∣ p \psi(t)=(p\mu_1^{p-1})^{-1}\sum_{j=2}^{d}|t_j|^p ψ ( t ) = ( p μ 1 p − 1 ) − 1 ∑ j = 2 d ∣ t j ∣ p . Since d ≥ 2 d\ge2 d ≥ 2 , the profile is nonaffine on every B 2 ( M ) B_2(M) B 2 ( M ) with M > 0 M>0 M > 0 . Theorem 3.3 therefore yields
R n , M Σ ( μ ) ≍ sup t ∈ B 2 ( M ) E μ n , t , Σ R ( w ∗ ( x ˉ ) ) ≍ n − p / 2 . R^\Sigma_{n,M}(\mu)\asymp\sup_{t\in B_2(M)}\mathbb{E}_{\mu_{n,t},\Sigma}R(w_*(\bar{x}))\asymp n^{-p/2}. R n , M Σ ( μ ) ≍ t ∈ B 2 ( M ) sup E μ n , t , Σ R ( w ∗ ( x ˉ )) ≍ n − p /2 .
Under the reference mean μ \boldsymbol{\mu} μ , write x ˉ = μ + Z / n \bar{x}=\boldsymbol{\mu}+\boldsymbol{Z}/\sqrt{n} x ˉ = μ + Z / n , where Z ∼ N ( 0 , Σ ) \boldsymbol{Z}\sim\mathcal{N}(0,\boldsymbol{\Sigma}) Z ∼ N ( 0 , Σ ) . Since ∇ f ∗ ( μ ) = e 1 \nabla f_*(\boldsymbol{\mu})=\boldsymbol{e}_1 ∇ f ∗ ( μ ) = e 1 , by the delta method,
n ( f ∗ ( μ + Z / n ) − f ∗ ( μ ) ) ⟹ Z 1 , \sqrt{n}\left(f_*(\boldsymbol{\mu}+\boldsymbol{Z}/\sqrt{n})-f_*(\boldsymbol{\mu})\right)\Longrightarrow Z_1, n ( f ∗ ( μ + Z / n ) − f ∗ ( μ ) ) ⟹ Z 1 ,
where Z 1 ∼ N ( 0 , Σ 11 ) Z_1\sim\mathcal{N}(0,\Sigma_{11}) Z 1 ∼ N ( 0 , Σ 11 ) . Since f ∗ ( ⋅ ) = ∥ ⋅ ∥ p f_*(\cdot)=\lVert\cdot\rVert_p f ∗ ( ⋅ ) = ∥ ⋅ ∥ p , the reverse triangle inequality also gives
n ∣ f ∗ ( μ + Z / n ) − f ∗ ( μ ) ∣ ≤ ∥ Z ∥ p . \sqrt{n}\left|f_*(\boldsymbol{\mu}+\boldsymbol{Z}/\sqrt{n})-f_*(\boldsymbol{\mu})\right|\leq\lVert\boldsymbol{Z}\rVert_p. n f ∗ ( μ + Z / n ) − f ∗ ( μ ) ≤ ∥ Z ∥ p .
Both E ∥ Z ∥ p \mathbb{E}\lVert\boldsymbol{Z}\rVert_p E ∥ Z ∥ p and E ∥ Z ∥ p 2 \mathbb{E}\lVert\boldsymbol{Z}\rVert_p^2 E ∥ Z ∥ p 2 are finite. DCT therefore yields
n E μ , Σ ∣ f ∗ ( x ˉ ) − f ∗ ( μ ) ∣ ⟶ E ∣ Z 1 ∣ = 2 Σ 11 / π , \sqrt{n}\mathbb{E}_{\boldsymbol{\mu},\boldsymbol{\Sigma}}\left|f_*(\bar{x})-f_*(\boldsymbol{\mu})\right|\longrightarrow\mathbb{E}|Z_1|=\sqrt{2\Sigma_{11}/\pi}, n E μ , Σ ∣ f ∗ ( x ˉ ) − f ∗ ( μ ) ∣ ⟶ E ∣ Z 1 ∣ = 2 Σ 11 / π ,
n E μ , Σ ( f ∗ ( x ˉ ) − f ∗ ( μ ) ) 2 ⟶ E Z 1 2 = Σ 11 . n\mathbb{E}_{\boldsymbol{\mu},\boldsymbol{\Sigma}}\left(f_*(\bar{x})-f_*(\boldsymbol{\mu})\right)^2\longrightarrow\mathbb{E}Z_1^2=\Sigma_{11}. n E μ , Σ ( f ∗ ( x ˉ ) − f ∗ ( μ ) ) 2 ⟶ E Z 1 2 = Σ 11 .
Now we discuss the local minimax value-estimation rates. Fix M > 0 M>0 M > 0 . Following the local minimax framework of van der Vaart [36 ] , let
f ^ n = f ^ n ( x 1 , … , x n ) , f ^ n : ( R d ) n → R , \widehat{f}_n=\widehat{f}_n(x_1,\ldots,x_n),\qquad\widehat{f}_n:(\mathbb{R}^d)^n\to\mathbb{R}, f n = f n ( x 1 , … , x n ) , f n : ( R d ) n → R ,
range over all Borel measurable estimators of the scalar value f ∗ ( μ n , t ) f_*(\boldsymbol{\mu}_{n,t}) f ∗ ( μ n , t ) . An estimator may depend on the fixed benchmark center μ \boldsymbol{\mu} μ , covariance Σ \boldsymbol{\Sigma} Σ , and radius M M M , but must be the same rule for every unknown t ∈ B 2 ( M ) t\in B_2(M) t ∈ B 2 ( M ) . The minimax risk takes the supremum over t t t for each such rule and then the infimum over the rules. The plug-in estimator is f ∗ ( x ˉ ) f_*(\bar{x}) f ∗ ( x ˉ ) .
For the upper bound, the reverse triangle inequality gives
sup t ∈ B 2 ( M ) E μ n , t , Σ ∣ f ∗ ( x ˉ ) − f ∗ ( μ n , t ) ∣ ≤ E ∥ Z ∥ p n , \sup_{t\in B_2(M)}\mathbb{E}_{\boldsymbol{\mu}_{n,t},\boldsymbol{\Sigma}}\left|f_*(\bar{x})-f_*(\boldsymbol{\mu}_{n,t})\right|\leq\frac{\mathbb{E}\lVert\boldsymbol{Z}\rVert_p}{\sqrt{n}}, t ∈ B 2 ( M ) sup E μ n , t , Σ f ∗ ( x ˉ ) − f ∗ ( μ n , t ) ≤ n E ∥ Z ∥ p ,
sup t ∈ B 2 ( M ) E μ n , t , Σ ( f ∗ ( x ˉ ) − f ∗ ( μ n , t ) ) 2 ≤ E ∥ Z ∥ p 2 n . \sup_{t\in B_2(M)}\mathbb{E}_{\boldsymbol{\mu}_{n,t},\boldsymbol{\Sigma}}\left(f_*(\bar{x})-f_*(\boldsymbol{\mu}_{n,t})\right)^2\leq\frac{\mathbb{E}\lVert\boldsymbol{Z}\rVert_p^2}{n}. t ∈ B 2 ( M ) sup E μ n , t , Σ ( f ∗ ( x ˉ ) − f ∗ ( μ n , t ) ) 2 ≤ n E ∥ Z ∥ p 2 .
Both right-hand sides are finite. Moreover, ∥ Z ∥ p ≤ d max { 1 / p − 1 / 2 , 0 } ∥ Z ∥ 2 \lVert\boldsymbol{Z}\rVert_p\leq d^{\max\{1/p-1/2,0\}}\lVert\boldsymbol{Z}\rVert_2 ∥ Z ∥ p ≤ d m a x { 1/ p − 1/2 , 0 } ∥ Z ∥ 2 and E ∥ Z ∥ 2 2 = tr ( Σ ) ≤ d λ \mathbb{E}\lVert\boldsymbol{Z}\rVert_2^2=\operatorname{tr}(\boldsymbol{\Sigma})\leq d\lambda E ∥ Z ∥ 2 2 = tr ( Σ ) ≤ d λ , so the upper bounds are uniform over Σ ∈ C \boldsymbol{\Sigma}\in\mathcal{C} Σ ∈ C .
For the lower bound, fix h = min { M , 1 } > 0 h=\min\{M,1\}>0 h = min { M , 1 } > 0 and use the prior π \pi π assigning probability 1 / 2 1/2 1/2 to each of T = ± h e 1 ∈ B 2 ( M ) T=\pm h\boldsymbol{e}_1\in B_2(M) T = ± h e 1 ∈ B 2 ( M ) . Since μ = μ 1 e 1 \boldsymbol{\mu}=\mu_1\boldsymbol{e}_1 μ = μ 1 e 1 and μ 1 > 0 \mu_1>0 μ 1 > 0 , for h / n < μ 1 h/\sqrt{n}<\mu_1 h / n < μ 1 ,
f ∗ ( μ + h e 1 / n ) − f ∗ ( μ − h e 1 / n ) = 2 h n . f_*(\boldsymbol{\mu}+h\boldsymbol{e}_1/\sqrt{n})-f_*(\boldsymbol{\mu}-h\boldsymbol{e}_1/\sqrt{n})=\frac{2h}{\sqrt{n}}. f ∗ ( μ + h e 1 / n ) − f ∗ ( μ − h e 1 / n ) = n 2 h .
For every estimator f ^ n \widehat{f}_n f n , its maximal risk dominates its average risk at these two alternatives:
sup t ∈ B 2 ( M ) E μ n , t , Σ ∣ f ^ n − f ∗ ( μ n , t ) ∣ ≥ 1 2 E μ + h e 1 / n , Σ ∣ f ^ n − μ 1 − h / n ∣ + 1 2 E μ − h e 1 / n , Σ ∣ f ^ n − μ 1 + h / n ∣ = E π , Σ ∣ f ^ n − f ∗ ( μ n , T ) ∣ . (116) \sup_{t\in B_2(M)}\mathbb{E}_{\boldsymbol{\mu}_{n,t},\boldsymbol{\Sigma}}\left|\widehat{f}_n-f_*(\boldsymbol{\mu}_{n,t})\right|
\geq\frac{1}{2}\mathbb{E}_{\boldsymbol{\mu}+h\boldsymbol{e}_1/\sqrt{n},\boldsymbol{\Sigma}}\left|\widehat{f}_n-\mu_1-h/\sqrt{n}\right|
+\frac{1}{2}\mathbb{E}_{\boldsymbol{\mu}-h\boldsymbol{e}_1/\sqrt{n},\boldsymbol{\Sigma}}\left|\widehat{f}_n-\mu_1+h/\sqrt{n}\right|
=\mathbb{E}_{\pi,\boldsymbol{\Sigma}}\left|\widehat{f}_n-f_*(\boldsymbol{\mu}_{n,T})\right|. \tag*{(116)} t ∈ B 2 ( M ) sup E μ n , t , Σ f n − f ∗ ( μ n , t ) ≥ 2 1 E μ + h e 1 / n , Σ f n − μ 1 − h / n + 2 1 E μ − h e 1 / n , Σ f n − μ 1 + h / n = E π , Σ f n − f ∗ ( μ n , T ) . (116)
We compute the minimum of this Bayes risk. The sample mean is sufficient, so with Y = n ( x ˉ − μ ) = T + Z Y=\sqrt{n}(\bar{x}-\mu)=T+Z Y = n ( x ˉ − μ ) = T + Z , q ( Y ) : = P π , Σ { T = h e 1 ∣ Y } = P π , Σ { T = h e 1 ∣ x 1 , … , x n } q(Y):=\mathbb{P}_{\pi,\Sigma}\{T=he_1\mid Y\}=\mathbb{P}_{\pi,\Sigma}\{T=he_1\mid x_1,\ldots,x_n\} q ( Y ) := P π , Σ { T = h e 1 ∣ Y } = P π , Σ { T = h e 1 ∣ x 1 , … , x n } . Conditional on the full sample, the expected absolute loss of a scalar output a ∈ R a\in\mathbb{R} a ∈ R is
q ( Y ) ∣ a − μ 1 − h / n ∣ + ( 1 − q ( Y ) ) ∣ a − μ 1 + h / n ∣ ≥ min { q ( Y ) , 1 − q ( Y ) } ( ∣ a − μ 1 − h / n ∣ + ∣ a − μ 1 + h / n ∣ ) ≥ 2 h n min { q ( Y ) , 1 − q ( Y ) } . \begin{aligned}
q(Y)|a-\mu_1-h/\sqrt{n}|+(1-q(Y))|a-\mu_1+h/\sqrt{n}|&\\
\geq\min\{q(Y),1-q(Y)\}\bigl(|a-\mu_1-h/\sqrt{n}|+|a-\mu_1+h/\sqrt{n}|\bigr)&\\
\geq\frac{2h}{\sqrt{n}}\min\{q(Y),1-q(Y)\}.&
\end{aligned} q ( Y ) ∣ a − μ 1 − h / n ∣ + ( 1 − q ( Y )) ∣ a − μ 1 + h / n ∣ ≥ min { q ( Y ) , 1 − q ( Y )} ( ∣ a − μ 1 − h / n ∣ + ∣ a − μ 1 + h / n ∣ ) ≥ n 2 h min { q ( Y ) , 1 − q ( Y )} .
Equality is attained by a = μ 1 + h / n a=\mu_1+h/\sqrt{n} a = μ 1 + h / n when q ( Y ) ≥ 1 / 2 q(Y)\geq1/2 q ( Y ) ≥ 1/2 and by a = μ 1 − h / n a=\mu_1-h/\sqrt{n} a = μ 1 − h / n when q ( Y ) < 1 / 2 q(Y)<1/2 q ( Y ) < 1/2 . These choices are measurable functions of Y Y Y . Thus minimizing the conditional loss and taking expectations gives
inf f ^ n E π , Σ ∣ f ^ n − f ∗ ( μ n , T ) ∣ = 2 h n E π , Σ min { q ( Y ) , 1 − q ( Y ) } . (117) \inf_{\widehat{f}_n}\mathbb{E}_{\pi,\Sigma}|\widehat{f}_n-f_*(\mu_{n,T})|=\frac{2h}{\sqrt{n}}\mathbb{E}_{\pi,\Sigma}\min\{q(Y),1-q(Y)\}. \tag*{(117)} f n inf E π , Σ ∣ f n − f ∗ ( μ n , T ) ∣ = n 2 h E π , Σ min { q ( Y ) , 1 − q ( Y )} . (117)
To evaluate the last expectation, the Gaussian densities give
log q ( Y ) 1 − q ( Y ) = − 1 2 ( Y − h e 1 ) ⊤ Σ − 1 ( Y − h e 1 ) + 1 2 ( Y + h e 1 ) ⊤ Σ − 1 ( Y + h e 1 ) = 2 h e 1 ⊤ Σ − 1 Y . \begin{aligned}
\log\frac{q(Y)}{1-q(Y)}=-\frac{1}{2}(Y-he_1)^\top\Sigma^{-1}(Y-he_1)+\frac{1}{2}(Y+he_1)^\top\Sigma^{-1}(Y+he_1) \\
&=2he_1^\top\Sigma^{-1}Y.
\end{aligned} log 1 − q ( Y ) q ( Y ) = − 2 1 ( Y − h e 1 ) ⊤ Σ − 1 ( Y − h e 1 ) + 2 1 ( Y + h e 1 ) ⊤ Σ − 1 ( Y + h e 1 ) = 2 h e 1 ⊤ Σ − 1 Y .
Note that e 1 ⊤ Σ − 1 Y ∣ T = ± h e 1 ∼ N ( ± h e 1 ⊤ Σ − 1 e 1 , e 1 ⊤ Σ − 1 e 1 ) e_1^\top\Sigma^{-1}Y\mid T=\pm he_1\sim\mathcal{N}(\pm he_1^\top\Sigma^{-1}e_1,e_1^\top\Sigma^{-1}e_1) e 1 ⊤ Σ − 1 Y ∣ T = ± h e 1 ∼ N ( ± h e 1 ⊤ Σ − 1 e 1 , e 1 ⊤ Σ − 1 e 1 ) . Hence
E π , Σ min { q ( Y ) , 1 − q ( Y ) } = 1 2 P π , Σ { e 1 ⊤ Σ − 1 Y < 0 ∣ T = h e 1 } + 1 2 P π , Σ { e 1 ⊤ Σ − 1 Y ≥ 0 ∣ T = − h e 1 } = Φ ( − h e 1 ⊤ Σ − 1 e 1 ) ≥ Φ ( − h / λ ) , (118) \begin{aligned}
\mathbb{E}_{\pi,\Sigma}\min\{q(Y),1-q(Y)\}
&=\frac{1}{2}\mathbb{P}_{\pi,\Sigma}\{e_1^\top\Sigma^{-1}Y<0\mid T=he_1\}\\
&\quad+\frac{1}{2}\mathbb{P}_{\pi,\Sigma}\{e_1^\top\Sigma^{-1}Y\geq0\mid T=-he_1\}\\
&=\Phi\left(-h\sqrt{e_1^\top\Sigma^{-1}e_1}\right)\geq\Phi(-h/\sqrt{\lambda}), \tag*{(118)}
\end{aligned} E π , Σ min { q ( Y ) , 1 − q ( Y )} = 2 1 P π , Σ { e 1 ⊤ Σ − 1 Y < 0 ∣ T = h e 1 } + 2 1 P π , Σ { e 1 ⊤ Σ − 1 Y ≥ 0 ∣ T = − h e 1 } = Φ ( − h e 1 ⊤ Σ − 1 e 1 ) ≥ Φ ( − h / λ ) , (118)
where we use e 1 ⊤ Σ − 1 e 1 ≤ 1 / λ e_1^\top\Sigma^{-1}e_1\leq1/\lambda e 1 ⊤ Σ − 1 e 1 ≤ 1/ λ . Combining Equations (E.5), (E.6) and (E.7) gives
inf f ^ n sup t ∈ B 2 ( M ) E μ n , t , Σ ∣ f ^ n − f ∗ ( μ n , t ) ∣ ≥ inf f ^ n E π , Σ ∣ f ^ n − f ∗ ( μ n , T ) ∣ ≥ 2 h n Φ ( − h / λ ) . \inf_{\widehat{f}_n}\sup_{t\in B_2(M)}\mathbb{E}_{\mu_{n,t},\Sigma}|\widehat{f}_n-f_*(\mu_{n,t})|\geq\inf_{\widehat{f}_n}\mathbb{E}_{\pi,\Sigma}|\widehat{f}_n-f_*(\mu_{n,T})|\geq\frac{2h}{\sqrt{n}}\Phi(-h/\sqrt{\lambda}). f n inf t ∈ B 2 ( M ) sup E μ n , t , Σ ∣ f n − f ∗ ( μ n , t ) ∣ ≥ f n inf E π , Σ ∣ f n − f ∗ ( μ n , T ) ∣ ≥ n 2 h Φ ( − h / λ ) .
The minimax risk is at most the maximal risk of f ∗ ( x ˉ ) f_*(\bar{x}) f ∗ ( x ˉ ) , so the upper and lower bounds prove
inf f ^ n sup t ∈ B 2 ( M ) E μ n , t , Σ ∣ f ^ n − f ∗ ( μ n , t ) ∣ ≍ n − 1 / 2 . \inf_{\widehat{f}_n}\sup_{t\in B_2(M)}\mathbb{E}_{\mu_{n,t},\Sigma}|\widehat{f}_n-f_*(\mu_{n,t})|\asymp n^{-1/2}. f n inf t ∈ B 2 ( M ) sup E μ n , t , Σ ∣ f n − f ∗ ( μ n , t ) ∣ ≍ n − 1/2 .
For squared loss, by Jensen’s inequality, for every f ^ n \widehat{f}_n f n ,
sup t ∈ B 2 ( M ) E μ n , t , Σ ( f ^ n − f ∗ ( μ n , t ) ) 2 ≥ sup t ∈ B 2 ( M ) ( E μ n , t , Σ ∣ f ^ n − f ∗ ( μ n , t ) ∣ ) 2 = ( sup t ∈ B 2 ( M ) E μ n , t , Σ ∣ f ^ n − f ∗ ( μ n , t ) ∣ ) 2 . \begin{aligned}
\sup_{t\in B_2(M)}\mathbb{E}_{\mu_{n,t},\Sigma}(\widehat{f}_n-f_*(\mu_{n,t}))^2\geq\sup_{t\in B_2(M)}\left(\mathbb{E}_{\mu_{n,t},\Sigma}|\widehat{f}_n-f_*(\mu_{n,t})|\right)^2 \\
&=\left(\sup_{t\in B_2(M)}\mathbb{E}_{\mu_{n,t},\Sigma}|\widehat{f}_n-f_*(\mu_{n,t})|\right)^2.
\end{aligned} t ∈ B 2 ( M ) sup E μ n , t , Σ ( f n − f ∗ ( μ n , t ) ) 2 ≥ t ∈ B 2 ( M ) sup ( E μ n , t , Σ ∣ f n − f ∗ ( μ n , t ) ∣ ) 2 = ( t ∈ B 2 ( M ) sup E μ n , t , Σ ∣ f n − f ∗ ( μ n , t ) ∣ ) 2 .
Taking the infimum over f ^ n \widehat{f}_n f n and using monotonicity of squaring on [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) gives
inf f ^ n sup t ∈ B 2 ( M ) E μ n , t , Σ ( f ^ n − f ∗ ( μ n , t ) ) 2 ≥ ( inf f ^ n sup t ∈ B 2 ( M ) E μ n , t , Σ ∣ f ^ n − f ∗ ( μ n , t ) ∣ ) 2 ≥ 4 h 2 n Φ ( − h / λ ) 2 . \begin{aligned}
\inf_{\widehat{f}_n}\sup_{t\in B_2(M)}\mathbb{E}_{\mu_{n,t},\Sigma}\left(\widehat{f}_n-f_*(\mu_{n,t})\right)^2 \ge\left(\inf_{\widehat{f}_n}\sup_{t\in B_2(M)}\mathbb{E}_{\mu_{n,t},\Sigma}\left|\widehat{f}_n-f_*(\mu_{n,t})\right|\right)^2 \\
&\ge\frac{4h^2}{n}\Phi\left(-h/\sqrt{\lambda}\right)^2.
\end{aligned} f n inf t ∈ B 2 ( M ) sup E μ n , t , Σ ( f n − f ∗ ( μ n , t ) ) 2 ≥ ( f n inf t ∈ B 2 ( M ) sup E μ n , t , Σ f n − f ∗ ( μ n , t ) ) 2 ≥ n 4 h 2 Φ ( − h / λ ) 2 .
Together with the squared-loss upper bound for f ∗ ( x ˉ ) f_*(\bar{x}) f ∗ ( x ˉ ) , this proves
inf f ^ n sup t ∈ B 2 ( M ) E μ n , t , Σ ( f ^ n − f ∗ ( μ n , t ) ) 2 ≍ n − 1 . \inf_{\widehat{f}_n}\sup_{t\in B_2(M)}\mathbb{E}_{\mu_{n,t},\Sigma}\left(\widehat{f}_n-f_*(\mu_{n,t})\right)^2 \asymp n^{-1}. f n inf t ∈ B 2 ( M ) sup E μ n , t , Σ ( f n − f ∗ ( μ n , t ) ) 2 ≍ n − 1 .
Both comparisons hold uniformly over Σ ∈ C \Sigma\in\mathcal{C} Σ ∈ C for fixed M > 0 M>0 M > 0 .
A non-power local rate: details for Example 4.11
A non-power scale can be necessary even when the optimal decision is unique. Choose 0 < δ < e − 2 0<\delta<e^{-2} 0 < δ < e − 2 and define, for ∣ s ∣ ≤ δ |s|\le\delta ∣ s ∣ ≤ δ ,
g ( 0 ) = 0 , g ( s ) = ∣ s ∣ log ( 1 / ∣ s ∣ ) ( s ≠ 0 ) . g(0)=0,\qquad g(s)=\frac{|s|}{\log(1/|s|)}\quad(s\ne0). g ( 0 ) = 0 , g ( s ) = log ( 1/∣ s ∣ ) ∣ s ∣ ( s = 0 ) .
The choice of g g g separates differentiability from higher-order power growth:
g ( s ) ∣ s ∣ = 1 log ( 1 / ∣ s ∣ ) → 0 , \frac{g(s)}{|s|}=\frac{1}{\log(1/|s|)}\to0, ∣ s ∣ g ( s ) = log ( 1/∣ s ∣ ) 1 → 0 ,
g ( s ) ∣ s ∣ α = ∣ s ∣ 1 − α log ( 1 / ∣ s ∣ ) → + ∞ ( α > 1 ) , s → 0. \frac{g(s)}{|s|^\alpha}=\frac{|s|^{1-\alpha}}{\log(1/|s|)}\to+\infty\quad(\alpha>1),\qquad s\to0. ∣ s ∣ α g ( s ) = log ( 1/∣ s ∣ ) ∣ s ∣ 1 − α → + ∞ ( α > 1 ) , s → 0.
The first limit gives g ′ ( 0 ) = 0 g'(0)=0 g ′ ( 0 ) = 0 , while the second rules out an O ( ∣ s ∣ α ) O(|s|^\alpha) O ( ∣ s ∣ α ) remainder for any α > 1 \alpha>1 α > 1 . The function g g g is even and continuously differentiable, with
g ′ ( s ) = sgn ( s ) { 1 log ( 1 / ∣ s ∣ ) + 1 [ log ( 1 / ∣ s ∣ ) ] 2 } → 0 ( s → 0 , s ≠ 0 ) . g'(s)=\operatorname{sgn}(s)\left\{\frac{1}{\log(1/|s|)}+\frac{1}{[\log(1/|s|)]^2}\right\}\to0\quad(s\to0,\ s\ne0). g ′ ( s ) = sgn ( s ) { log ( 1/∣ s ∣ ) 1 + [ log ( 1/∣ s ∣ ) ] 2 1 } → 0 ( s → 0 , s = 0 ) .
g g g is convex because its derivative is continuous at zero and
g ′ ′ ( s ) = log ( 1 / s ) + 2 s [ log ( 1 / s ) ] 3 > 0 , 0 < s < δ . g''(s)=\frac{\log(1/s)+2}{s[\log(1/s)]^3}>0,\qquad0<s<\delta. g ′′ ( s ) = s [ log ( 1/ s ) ] 3 log ( 1/ s ) + 2 > 0 , 0 < s < δ .
We realize 1 + g 1+g 1 + g as a section of a support function by using the intercept and slope of each tangent line as the coordinates of a feasible point:
Ω = conv ( { 0 } ∪ { ( 1 + g ( s ) − s g ′ ( s ) , g ′ ( s ) ) : ∣ s ∣ ≤ δ } ) , μ = e 1 . \Omega=\operatorname{conv}\left(\{0\}\cup\{(1+g(s)-sg'(s),g'(s)):|s|\le\delta\}\right),\qquad\mu=e_1. Ω = conv ( { 0 } ∪ {( 1 + g ( s ) − s g ′ ( s ) , g ′ ( s )) : ∣ s ∣ ≤ δ } ) , μ = e 1 .
The generating curve { ( 1 + g ( s ) − s g ′ ( s ) , g ′ ( s ) ) : ∣ s ∣ ≤ δ } \{(1+g(s)-sg'(s),g'(s)):|s|\le\delta\} {( 1 + g ( s ) − s g ′ ( s ) , g ′ ( s )) : ∣ s ∣ ≤ δ } is continuous on [ − δ , δ ] [-\delta,\delta] [ − δ , δ ] , so its union with { 0 } \{0\} { 0 } is compact. Its convex hull is compact in R 2 \mathbb{R}^2 R 2 , and Ω \Omega Ω is therefore compact, convex, and contains 0 0 0 . For s ≠ 0 s\ne0 s = 0 ,
1 + g ( s ) − s g ′ ( s ) = 1 − ∣ s ∣ [ log ( 1 / ∣ s ∣ ) ] 2 < 1 , 1+g(s)-sg'(s)=1-\frac{|s|}{[\log(1/|s|)]^2}<1, 1 + g ( s ) − s g ′ ( s ) = 1 − [ log ( 1/∣ s ∣ ) ] 2 ∣ s ∣ < 1 ,
whereas s = 0 s=0 s = 0 gives the point ( 1 , 0 ) T = e 1 (1,0)^{\mathsf T}=\mathbf e_1 ( 1 , 0 ) T = e 1 . Every point of Ω \Omega Ω is a finite convex combination of generating points, i.e., the origin or points of the form ( 1 + g ( s ) − s g ′ ( s ) , g ′ ( s ) ) (1+g(s)-sg'(s),g'(s)) ( 1 + g ( s ) − s g ′ ( s ) , g ′ ( s )) for some ∣ s ∣ ≤ δ |s|\le\delta ∣ s ∣ ≤ δ ; its first coordinate can equal 1 1 1 only if all points receiving positive weight are e 1 \mathbf e_1 e 1 . Hence
f ∗ ( e 1 ) = 1 , W ∗ ( e 1 ) = { e 1 } , ∇ f ∗ ( e 1 ) = e 1 . f_*(\mathbf e_1)=1,\qquad\mathcal W_*(\mathbf e_1)=\{\mathbf e_1\},\qquad\nabla f_*(\mathbf e_1)=\mathbf e_1. f ∗ ( e 1 ) = 1 , W ∗ ( e 1 ) = { e 1 } , ∇ f ∗ ( e 1 ) = e 1 .
For ∣ u ∣ ≤ δ |u|\le\delta ∣ u ∣ ≤ δ , convexity gives g ( s ) + g ′ ( s ) ( u − s ) ≤ g ( u ) g(s)+g'(s)(u-s)\le g(u) g ( s ) + g ′ ( s ) ( u − s ) ≤ g ( u ) , with equality at s = u s=u s = u . Taking the supremum of the corresponding linear functionals therefore yields
f ∗ ( ( 1 , u ) T ) = max { 0 , sup ∣ s ∣ ≤ δ [ 1 + g ( s ) − s g ′ ( s ) + u g ′ ( s ) ] } = 1 + g ( u ) . f_*((1,u)^{\mathsf T})=\max\left\{0,\sup_{|s|\le\delta}\left[1+g(s)-sg'(s)+ug'(s)\right]\right\}
=1+g(u). f ∗ (( 1 , u ) T ) = max { 0 , ∣ s ∣ ≤ δ sup [ 1 + g ( s ) − s g ′ ( s ) + u g ′ ( s ) ] } = 1 + g ( u ) .
Here 1 + g ( u ) ≥ 1 1+g(u)\ge1 1 + g ( u ) ≥ 1 , so the feasible point 0 \mathbf0 0 does not affect the maximum. Fix M > 0 M>0 M > 0 and take ε M ≤ min { 1 / 2 , δ / 2 } \varepsilon M\le\min\{1/2,\delta/2\} εM ≤ min { 1/2 , δ /2 } . For every t ∈ B 2 ( M ) \mathbf t\in B_2(M) t ∈ B 2 ( M ) ,
1 + ε t 1 ≥ 1 2 , ∣ ε t 2 1 + ε t 1 ∣ ≤ 2 ε M ≤ δ . 1+\varepsilon t_1\ge\frac{1}{2},\qquad\left|\frac{\varepsilon t_2}{1+\varepsilon t_1}\right|\le2\varepsilon M\le\delta. 1 + ε t 1 ≥ 2 1 , 1 + ε t 1 ε t 2 ≤ 2 εM ≤ δ .
Positive homogeneity of f ∗ f_* f ∗ and the section identity give
f ∗ ( e 1 + ε t ) = ( 1 + ε t 1 ) f ∗ ( ( 1 , ε t 2 1 + ε t 1 ) T ) = ( 1 + ε t 1 ) [ 1 + g ( ε t 2 1 + ε t 1 ) ] . f_*(\mathbf e_1+\varepsilon\mathbf t)=(1+\varepsilon t_1)f_*\left(\left(1,\frac{\varepsilon t_2}{1+\varepsilon t_1}\right)^{\mathsf T}\right)
=(1+\varepsilon t_1)\left[1+g\left(\frac{\varepsilon t_2}{1+\varepsilon t_1}\right)\right]. f ∗ ( e 1 + ε t ) = ( 1 + ε t 1 ) f ∗ ( ( 1 , 1 + ε t 1 ε t 2 ) T ) = ( 1 + ε t 1 ) [ 1 + g ( 1 + ε t 1 ε t 2 ) ] .
Hence
r e 1 ( ε t ) = f ∗ ( e 1 + ε t ) − 1 − ε t 1 = ( 1 + ε t 1 ) g ( ε t 2 1 + ε t 1 ) . r_{\mathbf e_1}(\varepsilon\mathbf t)=f_*(\mathbf e_1+\varepsilon\mathbf t)-1-\varepsilon t_1
=(1+\varepsilon t_1)g\left(\frac{\varepsilon t_2}{1+\varepsilon t_1}\right). r e 1 ( ε t ) = f ∗ ( e 1 + ε t ) − 1 − ε t 1 = ( 1 + ε t 1 ) g ( 1 + ε t 1 ε t 2 ) .
To identify the local scale, set a ( ε ) = ε / log ( 1 / ε ) a(\varepsilon)=\varepsilon/\log(1/\varepsilon) a ( ε ) = ε / log ( 1/ ε ) for 0 < ε < 1 0<\varepsilon<1 0 < ε < 1 . If t 2 = 0 t_2=0 t 2 = 0 , the preceding identity and g ( 0 ) = 0 g(0)=0 g ( 0 ) = 0 give r e 1 ( ε t ) = 0 r_{\mathbf e_1}(\varepsilon\mathbf t)=0 r e 1 ( ε t ) = 0 for all sufficiently small ε \varepsilon ε . If t 2 ≠ 0 t_2\ne0 t 2 = 0 , substituting the definition of g g g gives
r e 1 ( ε t ) = ( 1 + ε t 1 ) ε ∣ t 2 ∣ / ( 1 + ε t 1 ) log ( ( 1 + ε t 1 ) / ( ε ∣ t 2 ∣ ) ) = ε ∣ t 2 ∣ log ( 1 / ε ) + log ( 1 + ε t 1 ) − log ∣ t 2 ∣ . r_{\mathbf e_1}(\varepsilon\mathbf t)
=(1+\varepsilon t_1)\frac{\varepsilon|t_2|/(1+\varepsilon t_1)}{\log((1+\varepsilon t_1)/(\varepsilon|t_2|))}
=\frac{\varepsilon|t_2|}{\log(1/\varepsilon)+\log(1+\varepsilon t_1)-\log|t_2|}. r e 1 ( ε t ) = ( 1 + ε t 1 ) log (( 1 + ε t 1 ) / ( ε ∣ t 2 ∣ )) ε ∣ t 2 ∣/ ( 1 + ε t 1 ) = log ( 1/ ε ) + log ( 1 + ε t 1 ) − log ∣ t 2 ∣ ε ∣ t 2 ∣ .
Consequently,
r e 1 ( ε t ) a ( ε ) = ∣ t 2 ∣ 1 + log ( 1 + ε t 1 ) − log ∣ t 2 ∣ log ( 1 / ε ) → ∣ t 2 ∣ . \frac{r_{\mathbf e_1}(\varepsilon\mathbf t)}{a(\varepsilon)}
=\frac{|t_2|}{1+\dfrac{\log(1+\varepsilon t_1)-\log|t_2|}{\log(1/\varepsilon)}}\to|t_2|. a ( ε ) r e 1 ( ε t ) = 1 + log ( 1/ ε ) log ( 1 + ε t 1 ) − log ∣ t 2 ∣ ∣ t 2 ∣ → ∣ t 2 ∣.
Indeed, for fixed t \mathbf t t with t 2 ≠ 0 t_2\ne0 t 2 = 0 , log ( 1 + ε t 1 ) → 0 \log(1+\varepsilon t_1)\to0 log ( 1 + ε t 1 ) → 0 , log ∣ t 2 ∣ \log|t_2| log ∣ t 2 ∣ is finite, and log ( 1 / ε ) → + ∞ \log(1/\varepsilon)\to+\infty log ( 1/ ε ) → + ∞ . Combining the two cases proves ψ ( t ) = ∣ t 2 ∣ \psi(\mathbf t)=|t_2| ψ ( t ) = ∣ t 2 ∣ . Proposition C.1 upgrades this pointwise limit to
lim ε ↓ 0 sup t ∈ B 2 ( M ) ∣ r e 1 ( ε t ) a ( ε ) − ∣ t 2 ∣ ∣ = 0 , M > 0. \lim_{\varepsilon\downarrow0}\sup_{\mathbf t\in B_2(M)}
\left|\frac{r_{\mathbf e_1}(\varepsilon\mathbf t)}{a(\varepsilon)}-|t_2|\right|=0,\qquad M>0. ε ↓ 0 lim t ∈ B 2 ( M ) sup a ( ε ) r e 1 ( ε t ) − ∣ t 2 ∣ = 0 , M > 0.
Since ψ ( t ) = ∣ t 2 ∣ \psi(t)=|t_2| ψ ( t ) = ∣ t 2 ∣ is not affine on any B 2 ( M ) B_2(M) B 2 ( M ) with M > 0 M>0 M > 0 and a ( n − 1 / 2 ) = 2 / ( n log n ) a(n^{-1/2})=2/(\sqrt{n}\log n) a ( n − 1/2 ) = 2/ ( n log n ) for n > 1 n>1 n > 1 , Theorem 3.3 gives
n log n 2 R n , M Σ ( e 1 ) → V M , Σ ( ψ ) > 0. \frac{\sqrt{n}\log n}{2}R^\Sigma_{n,M}(e_1)\to V_{M,\Sigma}(\psi)>0. 2 n log n R n , M Σ ( e 1 ) → V M , Σ ( ψ ) > 0.
The same theorem gives the matching SAA rate:
R n , M Σ ( e 1 ) ≍ sup t ∈ B 2 ( M ) E μ n , t , Σ R ( w ∗ ( x ˉ ) ) ≍ 1 n log n , M > 0. R^\Sigma_{n,M}(e_1)\asymp\sup_{t\in B_2(M)}\mathbb{E}_{\mu_{n,t,\Sigma}}R(w_*(\bar{x}))\asymp\frac{1}{\sqrt{n}\log n},\qquad M>0. R n , M Σ ( e 1 ) ≍ t ∈ B 2 ( M ) sup E μ n , t , Σ R ( w ∗ ( x ˉ )) ≍ n log n 1 , M > 0.
The comparison constants can be chosen uniformly over Σ ∈ C \Sigma\in\mathcal{C} Σ ∈ C , and SAA uses neither the center nor Σ \Sigma Σ . Along the transverse direction, the remainder satisfies, for every α > 1 \alpha>1 α > 1 ,
ε − 1 r e 1 ( ε e 2 ) = 1 log ( 1 / ε ) → 0 , \varepsilon^{-1}r_{e_1}(\varepsilon e_2)=\frac{1}{\log(1/\varepsilon)}\to0, ε − 1 r e 1 ( ε e 2 ) = log ( 1/ ε ) 1 → 0 ,
ε − α r e 1 ( ε e 2 ) = ε 1 − α log ( 1 / ε ) → + ∞ . \varepsilon^{-\alpha}r_{e_1}(\varepsilon e_2)=\frac{\varepsilon^{1-\alpha}}{\log(1/\varepsilon)}\to+\infty. ε − α r e 1 ( ε e 2 ) = log ( 1/ ε ) ε 1 − α → + ∞.
Thus the first-order remainder vanishes, but no pure-power scale ε α \varepsilon^\alpha ε α with α > 1 \alpha>1 α > 1 yields a finite local profile. The logarithmic factor captures the intermediate scale and produces the corresponding non-power regret rate.
Technical Results for KL Neighborhoods
Proof of Corollary 5.1
Proof. Write the local alternative as μ + Σ 1 / 2 u / n \mu+\Sigma^{1/2}u/\sqrt{n} μ + Σ 1/2 u / n with u ∈ B 2 ( M ) u\in B_2(M) u ∈ B 2 ( M ) and set
Y = Σ − 1 / 2 n ( x ˉ − μ ) = u + G , G ∼ N ( 0 , I d ) . Y=\Sigma^{-1/2}\sqrt{n}(\bar{x}-\mu)=u+G,\qquad G\sim\mathcal{N}(0,I_d). Y = Σ − 1/2 n ( x ˉ − μ ) = u + G , G ∼ N ( 0 , I d ) .
The transformed experiment is therefore the standard Gaussian shift experiment on B 2 ( M ) B_2(M) B 2 ( M ) . Under (C.1), its normalized support-function remainder is
r μ ( Σ 1 / 2 u / n ) a ( n − 1 / 2 ) ⟶ ( ψ ∘ Σ 1 / 2 ) ( u ) \frac{r_\mu(\Sigma^{1/2}u/\sqrt{n})}{a(n^{-1/2})}\longrightarrow(\psi\circ\Sigma^{1/2})(u) a ( n − 1/2 ) r μ ( Σ 1/2 u / n ) ⟶ ( ψ ∘ Σ 1/2 ) ( u )
uniformly on bounded sets. Applying Theorem 3.3 to this transformed experiment proves (45) and the rate-optimality assertion for SAA. In the smooth case, ψ ( t ) = t ⊤ H μ t / 2 \psi(t)=t^\top H_\mu t/2 ψ ( t ) = t ⊤ H μ t /2 , so ψ ( Σ 1 / 2 u ) = u ⊤ Σ 1 / 2 H μ Σ 1 / 2 u / 2 \psi(\Sigma^{1/2}u)=u^\top\Sigma^{1/2}H_\mu\Sigma^{1/2}u/2 ψ ( Σ 1/2 u ) = u ⊤ Σ 1/2 H μ Σ 1/2 u /2 . The smooth formulas follow from Theorem 3.2 .
Uniform properties of the truncated mean
Lemma F.1 (Means and truncated observations in a KL neighborhood). Assume (42) . Write x i = ( x i , 1 , … , x i , d ) ⊤ x_i=(x_{i,1},\ldots,x_{i,d})^\top x i = ( x i , 1 , … , x i , d ) ⊤ , i = 1 , … , n i=1,\ldots,n i = 1 , … , n , for the i.i.d. observations with law Q Q Q , and set
τ n : = log 2 ( n + 1 ) , x i , j ( n ) : = sgn ( x i , j ) min { ∣ x i , j ∣ , τ n } , i = 1 , … , n , j = 1 , … , d , \tau_n:=\log^2(n+1),\qquad x_{i,j}^{(n)}:=\operatorname{sgn}(x_{i,j})\min\{|x_{i,j}|,\tau_n\},\qquad i=1,\ldots,n,\ j=1,\ldots,d, τ n := log 2 ( n + 1 ) , x i , j ( n ) := sgn ( x i , j ) min { ∣ x i , j ∣ , τ n } , i = 1 , … , n , j = 1 , … , d , μ ~ n : = 1 n ∑ i = 1 n x i ( n ) , Z n , Q : = n ( μ ~ n − μ Q ) . (119) \widetilde{\mu}_n:=\frac{1}{n}\sum_{i=1}^{n}x_i^{(n)},\qquad Z_{n,Q}:=\sqrt{n}(\widetilde{\mu}_n-\mu_Q). \tag*{(119)} μ n := n 1 i = 1 ∑ n x i ( n ) , Z n , Q := n ( μ n − μ Q ) . (119)
Here x i ( n ) = ( x i , 1 ( n ) , … , x i , d ( n ) ) ⊤ x_i^{(n)}=(x_{i,1}^{(n)},\ldots,x_{i,d}^{(n)})^\top x i ( n ) = ( x i , 1 ( n ) , … , x i , d ( n ) ) ⊤ . Expectations E Q \mathbb{E}_{Q} E Q involving x 1 x_1 x 1 are with respect to its one-observation law Q Q Q ; E P \mathbb{E}_{P} E P denotes integration of the same one-observation function under the reference law P P P . Expectations of sample statistics are denoted by E Q ⊗ n \mathbb{E}_{Q^{\otimes n}} E Q ⊗ n . For each fixed M > 0 M>0 M > 0 , all Q ∈ Q n , M K L ( P ) Q\in\mathcal{Q}_{n,M}^{\mathrm{KL}}(P) Q ∈ Q n , M KL ( P ) have finite first moments, and
sup Q ∈ Q n , M K L ( P ) n ∥ μ Q − μ ∥ 2 = O ( 1 ) . (120) \sup_{Q\in\mathcal{Q}_{n,M}^{\mathrm{KL}}(P)}\sqrt{n}\lVert\mu_Q-\mu\rVert_2=O(1). \tag*{(120)} Q ∈ Q n , M KL ( P ) sup n ∥ μ Q − μ ∥ 2 = O ( 1 ) . (120)
sup Q ∈ Q n , M K L ( P ) n ( μ Q − μ ) ⊤ Σ − 1 ( μ Q − μ ) ≤ M 2 + o ( 1 ) . (121) \sup_{Q\in\mathcal{Q}_{n,M}^{\mathrm{KL}}(P)}n(\mu_Q-\mu)^\top\Sigma^{-1}(\mu_Q-\mu)\le M^2+o(1). \tag*{(121)} Q ∈ Q n , M KL ( P ) sup n ( μ Q − μ ) ⊤ Σ − 1 ( μ Q − μ ) ≤ M 2 + o ( 1 ) . (121)
sup Q ∈ Q n , M K L ( P ) ∥ E Q x 1 ( n ) − μ Q ∥ 2 = O ( n − 1 ) . (122) \sup_{Q\in\mathcal{Q}_{n,M}^{\mathrm{KL}}(P)}\left\lVert\mathbb{E}_{Q}x_1^{(n)}-\mu_Q\right\rVert_2=O(n^{-1}). \tag*{(122)} Q ∈ Q n , M KL ( P ) sup E Q x 1 ( n ) − μ Q 2 = O ( n − 1 ) . (122)
sup Q ∈ Q n , M K L ( P ) ∥ Cov Q ( x 1 ( n ) ) − Σ ∥ o p → 0. (123) \sup_{Q\in\mathcal{Q}_{n,M}^{\mathrm{KL}}(P)}\left\lVert\operatorname{Cov}_{Q}(x_1^{(n)})-\Sigma\right\rVert_{\mathrm{op}}\to0. \tag*{(123)} Q ∈ Q n , M KL ( P ) sup Cov Q ( x 1 ( n ) ) − Σ op → 0. (123)
sup n l a r g e sup Q ∈ Q n , M K L ( P ) E Q ⊗ n ∥ Z n , Q ∥ 2 q < ∞ , q > 0. (124) \sup_{n\ \mathrm{large}}\ \sup_{Q\in\mathcal{Q}_{n,M}^{\mathrm{KL}}(P)}\mathbb{E}_{Q^{\otimes n}}\lVert Z_{n,Q}\rVert_2^q<\infty,\qquad q>0. \tag*{(124)} n large sup Q ∈ Q n , M KL ( P ) sup E Q ⊗ n ∥ Z n , Q ∥ 2 q < ∞ , q > 0. (124)
For every sequence Q n ∈ Q n , M K L ( P ) Q_n\in\mathcal{Q}_{n,M}^{\mathrm{KL}}(P) Q n ∈ Q n , M KL ( P ) , Z n , Q n → d N ( 0 , Σ ) Z_{n,Q_n}\xrightarrow{d}\mathcal{N}(0,\Sigma) Z n , Q n d N ( 0 , Σ ) . Moreover, for each fixed ϵ > 0 \epsilon>0 ϵ > 0 and an independent G ∼ N ( 0 , I d ) \mathbf{G}\sim\mathcal{N}(0,I_d) G ∼ N ( 0 , I d ) ,
sup Q ∈ Q n , M K L ( P ) ∥ L Q ⊗ n ( Z n , Q + ϵ G ) − N ( 0 , Σ + ϵ 2 I d ) ∥ T V → 0. (125) \sup_{Q\in\mathcal{Q}_{n,M}^{\mathrm{KL}}(P)}\left\lVert\mathcal{L}_{Q^{\otimes n}}(Z_{n,Q}+\epsilon\mathbf{G})-\mathcal{N}(0,\Sigma+\epsilon^2I_d)\right\rVert_{\mathrm{TV}}\to0. \tag*{(125)} Q ∈ Q n , M KL ( P ) sup L Q ⊗ n ( Z n , Q + ϵ G ) − N ( 0 , Σ + ϵ 2 I d ) TV → 0. (125)
The law in (F.7) includes the independent G \mathbf{G} G .
Proof. (42) implies, for some γ > 0 \gamma>0 γ > 0 ,
E P e γ ∥ x 1 ∥ 2 < ∞ . \mathbb{E}_{P}e^{\gamma\lVert x_1\rVert_2}<\infty. E P e γ ∥ x 1 ∥ 2 < ∞.
Indeed, for sufficiently small γ > 0 \gamma>0 γ > 0 , e γ ∥ x 1 ∥ 2 ≤ e γ ∑ j ∣ x 1 , j ∣ ≤ ∑ v ∈ { − 1 , 1 } d e γ v ⊤ x 1 e^{\gamma\lVert x_1\rVert_2}\le e^{\gamma\sum_j\lvert x_{1,j}\rvert}\le\sum_{v\in\{-1,1\}^d}e^{\gamma v^\top x_1} e γ ∥ x 1 ∥ 2 ≤ e γ ∑ j ∣ x 1 , j ∣ ≤ ∑ v ∈ { − 1 , 1 } d e γ v ⊤ x 1 . For bounded measurable g g g , nonnegativity of D K L ( Q ∥ e g P / E P e g ) D_{\mathrm{KL}}(Q\Vert e^gP/\mathbb{E}_{P}e^g) D KL ( Q ∥ e g P / E P e g ) gives the entropy inequality
E Q g ≤ D K L ( Q ∥ P ) + log E P e g . (126) \mathbb{E}_{Q}g\le D_{\mathrm{KL}}(Q\Vert P)+\log\mathbb{E}_{P}e^g. \tag*{(126)} E Q g ≤ D KL ( Q ∥ P ) + log E P e g . (126)
For nonnegative g g g with E P e g < ∞ \mathbb{E}_{P}e^g<\infty E P e g < ∞ , apply this inequality to g ∧ K g\wedge K g ∧ K and let K → ∞ K\to\infty K → ∞ by monotone convergence. Taking g ( x 1 ) = γ ∥ x 1 ∥ 2 g(x_1)=\gamma\lVert x_1\rVert_2 g ( x 1 ) = γ ∥ x 1 ∥ 2 proves
E Q ∥ x 1 ∥ 2 ≤ D K L ( Q ∥ P ) + log E P e γ ∥ x 1 ∥ 2 γ < ∞ . \mathbb{E}_{Q}\lVert x_1\rVert_2\le\frac{D_{\mathrm{KL}}(Q\Vert P)+\log\mathbb{E}_{P}e^{\gamma\lVert x_1\rVert_2}}{\gamma}<\infty. E Q ∥ x 1 ∥ 2 ≤ γ D KL ( Q ∥ P ) + log E P e γ ∥ x 1 ∥ 2 < ∞.
For g ( x 1 ) = v ⊤ ( x 1 − μ ) g(x_1)=v^\top(x_1-\mu) g ( x 1 ) = v ⊤ ( x 1 − μ ) with ∥ v ∥ 2 < η \lVert v\rVert_2<\eta ∥ v ∥ 2 < η , we have
E Q ∣ g ∣ ≤ ∥ v ∥ 2 ( E Q ∥ x 1 ∥ 2 + ∥ μ ∥ 2 ) < ∞ , log E P e g = Λ P ( v ) < ∞ . \mathbb{E}_{Q}|g|\le\lVert v\rVert_2(\mathbb{E}_{Q}\lVert x_1\rVert_2+\lVert\mu\rVert_2)<\infty,\qquad\log\mathbb{E}_{P}e^g=\Lambda_P(v)<\infty. E Q ∣ g ∣ ≤ ∥ v ∥ 2 ( E Q ∥ x 1 ∥ 2 + ∥ μ ∥ 2 ) < ∞ , log E P e g = Λ P ( v ) < ∞.
Thus the same log-density identity is integrable under Q Q Q and gives
v ⊤ ( μ Q − μ ) ≤ D K L ( Q ∥ P ) + Λ P ( v ) . v^\top(\mu_Q-\mu)\le D_{\mathrm{KL}}(Q\Vert P)+\Lambda_P(v). v ⊤ ( μ Q − μ ) ≤ D KL ( Q ∥ P ) + Λ P ( v ) .
There exist C P , ρ P > 0 C_P,\rho_P>0 C P , ρ P > 0 such that Λ P ( v ) ≤ C P ∥ v ∥ 2 2 / 2 \Lambda_P(v)\le C_P\|v\|_2^2/2 Λ P ( v ) ≤ C P ∥ v ∥ 2 2 /2 for ∥ v ∥ 2 ≤ ρ P \|v\|_2\le\rho_P ∥ v ∥ 2 ≤ ρ P . For sufficiently large n n n , when μ Q ≠ μ \mu_Q\ne\mu μ Q = μ , v = n − 1 / 2 ( μ Q − μ ) / ∥ μ Q − μ ∥ 2 v=n^{-1/2}(\mu_Q-\mu)/\|\mu_Q-\mu\|_2 v = n − 1/2 ( μ Q − μ ) /∥ μ Q − μ ∥ 2 satisfies ∥ v ∥ 2 ≤ ρ P \|v\|_2\le\rho_P ∥ v ∥ 2 ≤ ρ P , and therefore
n ∥ μ Q − μ ∥ 2 ≤ M 2 + C P 2 , \sqrt{n}\|\mu_Q-\mu\|_2\le\frac{M^2+C_P}{2}, n ∥ μ Q − μ ∥ 2 ≤ 2 M 2 + C P ,
which proves (F.2). Thus, for sufficiently large n n n , h = μ Q − μ h=\mu_Q-\mu h = μ Q − μ belongs to the domain of θ ( h ) \theta(h) θ ( h ) in Lemma H.2, uniformly for large n n n . Since E Q ( x 1 − μ ) = h \mathbb{E}_Q(x_1-\mu)=h E Q ( x 1 − μ ) = h , and log d P h d P \log\frac{\mathrm{d}P_h}{\mathrm{d}P} log d P d P h is Q Q Q -integrable,
D K L ( Q ∥ P ) = D K L ( Q ∥ P h ) + θ ( h ) T h − Λ P ( θ ( h ) ) , (127) D_{\mathrm{KL}}(Q\Vert P)=D_{\mathrm{KL}}(Q\Vert P_h)+\theta(h)^\mathsf{T}h-\Lambda_P(\theta(h)), \tag*{(127)} D KL ( Q ∥ P ) = D KL ( Q ∥ P h ) + θ ( h ) T h − Λ P ( θ ( h )) , (127)
and (H.12) gives
θ ( h ) T h − Λ P ( θ ( h ) ) = 1 2 h T Σ − 1 h + O ( ∥ h ∥ 2 3 ) . \theta(h)^\mathsf{T}h-\Lambda_P(\theta(h))=\frac{1}{2}h^\mathsf{T}\Sigma^{-1}h+O(\|h\|_2^3). θ ( h ) T h − Λ P ( θ ( h )) = 2 1 h T Σ − 1 h + O ( ∥ h ∥ 2 3 ) .
Thus,
1 2 h T Σ − 1 h ≤ D K L ( Q ∥ P ) − D K L ( Q ∥ P h ) + C ∥ h ∥ 2 3 ≤ M 2 2 n + C ∥ h ∥ 2 3 \frac{1}{2}h^\mathsf{T}\Sigma^{-1}h\le D_{\mathrm{KL}}(Q\Vert P)-D_{\mathrm{KL}}(Q\Vert P_h)+C\|h\|_2^3\le\frac{M^2}{2n}+C\|h\|_2^3 2 1 h T Σ − 1 h ≤ D KL ( Q ∥ P ) − D KL ( Q ∥ P h ) + C ∥ h ∥ 2 3 ≤ 2 n M 2 + C ∥ h ∥ 2 3
for some constant C > 0 C>0 C > 0 and small h h h . By (F.2), n ∥ h ∥ 2 3 = o ( 1 ) n\|h\|_2^3=o(1) n ∥ h ∥ 2 3 = o ( 1 ) , which proves (F.3).
For (F.4), choose 0 < γ 1 < γ 0<\gamma_1<\gamma 0 < γ 1 < γ and let g ( x 1 ) = γ 1 ∥ x 1 ∥ 2 2 I { ∥ x 1 ∥ 2 > τ } g(x_1)=\gamma_1\|x_1\|_2^2\mathbb{I}\{\|x_1\|_2>\tau\} g ( x 1 ) = γ 1 ∥ x 1 ∥ 2 2 I { ∥ x 1 ∥ 2 > τ } for some constant τ > 0 \tau>0 τ > 0 . Then
E P e g ≤ 1 + E P [ e γ 1 ∥ x 1 ∥ 2 2 I { ∥ x 1 ∥ 2 > τ } ] ≤ 1 + e − ( γ − γ 1 ) τ E P e γ ∥ x 1 ∥ 2 2 , \mathbb{E}_P e^g\le1+\mathbb{E}_P[e^{\gamma_1\|x_1\|_2^2}\mathbb{I}\{\|x_1\|_2>\tau\}]\le1+e^{-(\gamma-\gamma_1)\tau}\mathbb{E}_P e^{\gamma\|x_1\|_2^2}, E P e g ≤ 1 + E P [ e γ 1 ∥ x 1 ∥ 2 2 I { ∥ x 1 ∥ 2 > τ }] ≤ 1 + e − ( γ − γ 1 ) τ E P e γ ∥ x 1 ∥ 2 2 ,
and therefore
E Q g = γ 1 E Q [ ∥ x 1 ∥ 2 2 I { ∥ x 1 ∥ 2 > τ } ] ≤ D K L ( Q ∥ P ) + log ( 1 + e − ( γ − γ 1 ) τ E P e γ ∥ x 1 ∥ 2 2 ) ≤ M 2 2 n + e − ( γ − γ 1 ) τ E P e γ ∥ x 1 ∥ 2 2 . \begin{aligned}
\mathbb{E}_Q g=\gamma_1\mathbb{E}_Q[\|x_1\|_2^2\mathbb{I}\{\|x_1\|_2>\tau\}]\le D_{\mathrm{KL}}(Q\Vert P)+\log(1+e^{-(\gamma-\gamma_1)\tau}\mathbb{E}_P e^{\gamma\|x_1\|_2^2}) \\
&\le\frac{M^2}{2n}+e^{-(\gamma-\gamma_1)\tau}\mathbb{E}_P e^{\gamma\|x_1\|_2^2}.
\end{aligned} E Q g = γ 1 E Q [ ∥ x 1 ∥ 2 2 I { ∥ x 1 ∥ 2 > τ }] ≤ D KL ( Q ∥ P ) + log ( 1 + e − ( γ − γ 1 ) τ E P e γ ∥ x 1 ∥ 2 2 ) ≤ 2 n M 2 + e − ( γ − γ 1 ) τ E P e γ ∥ x 1 ∥ 2 2 .
Since the observations are identically distributed,
E Q ⊗ n μ ~ n = 1 n ∑ i = 1 n E Q x i ( n ) = E Q x 1 ( n ) , μ Q = E Q x 1 . \mathbb{E}_{Q^{\otimes n}}\widetilde{\mu}_n=\frac{1}{n}\sum_{i=1}^n\mathbb{E}_Q x_i^{(n)}=\mathbb{E}_Q x_1^{(n)},\qquad\mu_Q=\mathbb{E}_Qx_1. E Q ⊗ n μ n = n 1 i = 1 ∑ n E Q x i ( n ) = E Q x 1 ( n ) , μ Q = E Q x 1 .
Moreover, ∥ x 1 − x 1 ( n ) ∥ 2 ≤ ∥ x 1 ∥ 2 I { ∥ x 1 ∥ 2 > τ n } \|x_1-x_1^{(n)}\|_2\le\|x_1\|_2\mathbb{I}\{\|x_1\|_2>\tau_n\} ∥ x 1 − x 1 ( n ) ∥ 2 ≤ ∥ x 1 ∥ 2 I { ∥ x 1 ∥ 2 > τ n } . Taking τ = τ n \tau=\tau_n τ = τ n gives
sup Q ∈ Q n , M K L ( P ) E Q ∥ x 1 ( n ) − x 1 ∥ 2 ≤ sup Q ∈ Q n , M K L ( P ) E Q [ ∥ x 1 ∥ 2 I { ∥ x 1 ∥ 2 > τ n } ] ≤ γ 1 − 1 [ M 2 2 n + e − ( γ − γ 1 ) τ n E P e γ ∥ x 1 ∥ 2 2 ] = O ( n − 1 ) . (128) \begin{aligned}
\sup_{Q\in\mathcal{Q}_{n,M}^{\mathrm{KL}}(P)}\mathbb{E}_Q\|x_1^{(n)}-x_1\|_2\le\sup_{Q\in\mathcal{Q}_{n,M}^{\mathrm{KL}}(P)}\mathbb{E}_Q[\|x_1\|_2\mathbb{I}\{\|x_1\|_2>\tau_n\}] \\
&\le\gamma_1^{-1}\left[\frac{M^2}{2n}+e^{-(\gamma-\gamma_1)\tau_n}\mathbb{E}_P e^{\gamma\|x_1\|_2^2}\right]=O(n^{-1}). \tag*{(128)}
\end{aligned} Q ∈ Q n , M KL ( P ) sup E Q ∥ x 1 ( n ) − x 1 ∥ 2 ≤ Q ∈ Q n , M KL ( P ) sup E Q [ ∥ x 1 ∥ 2 I { ∥ x 1 ∥ 2 > τ n }] ≤ γ 1 − 1 [ 2 n M 2 + e − ( γ − γ 1 ) τ n E P e γ ∥ x 1 ∥ 2 2 ] = O ( n − 1 ) . (128)
Here τ n = log 2 ( n + 1 ) \tau_n=\log^2(n+1) τ n = log 2 ( n + 1 ) . Since
∥ E Q ⊗ n μ ~ n − μ Q ∥ 2 = ∥ E Q ( x 1 ( n ) − x 1 ) ∥ 2 ≤ E Q ∥ x 1 ( n ) − x 1 ∥ 2 , \|\mathbb{E}_{Q^{\otimes n}}\widetilde{\mu}_n-\mu_Q\|_2=\|\mathbb{E}_Q(x_1^{(n)}-x_1)\|_2\le\mathbb{E}_Q\|x_1^{(n)}-x_1\|_2, ∥ E Q ⊗ n μ n − μ Q ∥ 2 = ∥ E Q ( x 1 ( n ) − x 1 ) ∥ 2 ≤ E Q ∥ x 1 ( n ) − x 1 ∥ 2 ,
this also proves (F.4).
Pinsker’s inequality gives
sup Q ∈ Q n , M K L ( P ) ∥ Q − P ∥ T V ≤ sup Q ∈ Q n , M K L ( P ) 1 2 D K L ( Q ∥ P ) ≤ M 2 n . \sup_{Q\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)}\lVert Q-P\rVert_{\mathrm{TV}}
\leq
\sup_{Q\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)}
\sqrt{\frac{1}{2}D_{\mathrm{KL}}(Q\|P)}
\leq\frac{M}{2\sqrt{n}}. Q ∈ Q n , M KL ( P ) sup ∥ Q − P ∥ TV ≤ Q ∈ Q n , M KL ( P ) sup 2 1 D KL ( Q ∥ P ) ≤ 2 n M .
For every fixed k > 0 k>0 k > 0 , ∥ x 1 ( n ) ∥ 2 ≤ d τ n \lVert x_1^{(n)}\rVert_2\leq\sqrt{d}\tau_n ∥ x 1 ( n ) ∥ 2 ≤ d τ n , and hence
sup Q ∈ Q n , M K L ( P ) ∣ E Q ∥ x 1 ( n ) ∥ 2 k − E P ∥ x 1 ( n ) ∥ 2 k ∣ ≤ 2 ( d τ n ) k sup Q ∈ Q n , M K L ( P ) ∥ Q − P ∥ T V ≤ M ( d τ n ) k n = o ( 1 ) . (129) \sup_{Q\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)}
\left|\mathbb{E}_Q\lVert x_1^{(n)}\rVert_2^k-\mathbb{E}_P\lVert x_1^{(n)}\rVert_2^k\right|
\leq2(\sqrt{d}\tau_n)^k
\sup_{Q\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)}
\lVert Q-P\rVert_{\mathrm{TV}}
\leq\frac{M(\sqrt{d}\tau_n)^k}{\sqrt{n}}=o(1). \tag*{(129)} Q ∈ Q n , M KL ( P ) sup E Q ∥ x 1 ( n ) ∥ 2 k − E P ∥ x 1 ( n ) ∥ 2 k ≤ 2 ( d τ n ) k Q ∈ Q n , M KL ( P ) sup ∥ Q − P ∥ TV ≤ n M ( d τ n ) k = o ( 1 ) . (129)
Since ∥ x 1 ( n ) ∥ 2 k ↑ ∥ x 1 ∥ 2 k \lVert x_1^{(n)}\rVert_2^k\uparrow\lVert x_1\rVert_2^k ∥ x 1 ( n ) ∥ 2 k ↑ ∥ x 1 ∥ 2 k pointwise, monotone convergence and the exponential-moment bound under P P P give
E P ∥ x 1 ( n ) ∥ 2 k → E P ∥ x 1 ∥ 2 k < ∞ . \mathbb{E}_P\lVert x_1^{(n)}\rVert_2^k\to\mathbb{E}_P\lVert x_1\rVert_2^k<\infty. E P ∥ x 1 ( n ) ∥ 2 k → E P ∥ x 1 ∥ 2 k < ∞.
Combining this limit with (F.11) gives, for every fixed k > 0 k>0 k > 0 ,
sup n sup Q ∈ Q n , M K L ( P ) E Q ∥ x 1 ( n ) ∥ 2 k < ∞ . (130) \sup_n\sup_{Q\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)}
\mathbb{E}_Q\lVert x_1^{(n)}\rVert_2^k<\infty. \tag*{(130)} n sup Q ∈ Q n , M KL ( P ) sup E Q ∥ x 1 ( n ) ∥ 2 k < ∞. (130)
For the first and second clipped coordinate moments, Pinsker’s inequality and the bounds ∣ x 1 , j ( n ) ∣ ≤ τ n \lvert x_{1,j}^{(n)}\rvert\leq\tau_n ∣ x 1 , j ( n ) ∣ ≤ τ n and ∣ x 1 , j ( n ) x 1 , k ( n ) ∣ ≤ τ n 2 \lvert x_{1,j}^{(n)}x_{1,k}^{(n)}\rvert\leq\tau_n^2 ∣ x 1 , j ( n ) x 1 , k ( n ) ∣ ≤ τ n 2 give
sup Q ∣ E Q x 1 , j ( n ) − E P x 1 , j ( n ) ∣ ≤ M τ n n = o ( 1 ) , \sup_Q\left|\mathbb{E}_Qx_{1,j}^{(n)}-\mathbb{E}_Px_{1,j}^{(n)}\right|
\leq\frac{M\tau_n}{\sqrt{n}}=o(1), Q sup E Q x 1 , j ( n ) − E P x 1 , j ( n ) ≤ n M τ n = o ( 1 ) ,
sup Q ∣ E Q [ x 1 , j ( n ) x 1 , k ( n ) ] − E P [ x 1 , j ( n ) x 1 , k ( n ) ] ∣ ≤ M τ n 2 n = o ( 1 ) . \sup_Q\left|\mathbb{E}_Q[x_{1,j}^{(n)}x_{1,k}^{(n)}]-\mathbb{E}_P[x_{1,j}^{(n)}x_{1,k}^{(n)}]\right|
\leq\frac{M\tau_n^2}{\sqrt{n}}=o(1). Q sup E Q [ x 1 , j ( n ) x 1 , k ( n ) ] − E P [ x 1 , j ( n ) x 1 , k ( n ) ] ≤ n M τ n 2 = o ( 1 ) .
Dominated convergence under P P P , using ∣ x 1 , j ( n ) ∣ ≤ ∣ x 1 , j ∣ \lvert x_{1,j}^{(n)}\rvert\leq\lvert x_{1,j}\rvert ∣ x 1 , j ( n ) ∣ ≤ ∣ x 1 , j ∣ and ∣ x 1 , j ( n ) x 1 , k ( n ) ∣ ≤ ( x 1 , j 2 + x 1 , k 2 ) / 2 \lvert x_{1,j}^{(n)}x_{1,k}^{(n)}\rvert\leq(x_{1,j}^2+x_{1,k}^2)/2 ∣ x 1 , j ( n ) x 1 , k ( n ) ∣ ≤ ( x 1 , j 2 + x 1 , k 2 ) /2 , identifies the limits as μ j \mu_j μ j and E P x 1 , j x 1 , k \mathbb{E}_P x_{1,j}x_{1,k} E P x 1 , j x 1 , k . Expanding each covariance entry and using fixed d d d proves (F.5) uniformly over Q Q Q .
For an integer k ≥ 1 k\geq1 k ≥ 1 , expand the 2 k 2k 2 k th moment of n − 1 / 2 ∑ i = 1 n ( x i , j ( n ) − E Q x 1 , j ( n ) ) n^{-1/2}\sum_{i=1}^n(x_{i,j}^{(n)}-\mathbb{E}_Qx_{1,j}^{(n)}) n − 1/2 ∑ i = 1 n ( x i , j ( n ) − E Q x 1 , j ( n ) ) . By independence and centering, any term with an index occurring only once is zero. The remaining terms contain at most k k k distinct indices, and their number is at most C k n k C_kn^k C k n k . Hölder’s inequality and (F.12) with exponent 2 k 2k 2 k therefore give
E Q ⊗ n ∣ 1 n ∑ i = 1 n ( x i , j ( n ) − E Q x 1 , j ( n ) ) ∣ 2 k ≤ C k E Q ∣ x 1 , j ( n ) − E Q x 1 , j ( n ) ∣ 2 k ≤ C k , P , M . \mathbb{E}_{Q^{\otimes n}}\left\lvert\frac{1}{\sqrt{n}}\sum_{i=1}^n(x_{i,j}^{(n)}-\mathbb{E}_Qx_{1,j}^{(n)})\right\rvert^{2k}
\leq C_k\mathbb{E}_Q\lvert x_{1,j}^{(n)}-\mathbb{E}_Qx_{1,j}^{(n)}\rvert^{2k}
\leq C_{k,P,M}. E Q ⊗ n n 1 i = 1 ∑ n ( x i , j ( n ) − E Q x 1 , j ( n ) ) 2 k ≤ C k E Q ∣ x 1 , j ( n ) − E Q x 1 , j ( n ) ∣ 2 k ≤ C k , P , M .
Combined with (F.4), summing over coordinates and using ∥ z ∥ 2 2 k ≤ d k − 1 ∑ j ∣ z j ∣ 2 k \lVert z\rVert_2^{2k}\leq d^{k-1}\sum_j\lvert z_j\rvert^{2k} ∥ z ∥ 2 2 k ≤ d k − 1 ∑ j ∣ z j ∣ 2 k proves (F.6); every fixed q > 0 q>0 q > 0 is bounded by a sufficiently large even moment.
Fix v ∈ R d v\in\mathbb{R}^d v ∈ R d and put U n , i = x i ( n ) − E Q x 1 ( n ) U_{n,i}=x_i^{(n)}-\mathbb{E}_Qx_1^{(n)} U n , i = x i ( n ) − E Q x 1 ( n ) . The third moments of U n , 1 U_{n,1} U n , 1 are uniformly bounded by (F.12). Taylor’s inequality ∣ e i u − 1 − i u + u 2 / 2 ∣ ≤ ∣ u ∣ 3 / 6 \lvert e^{iu}-1-iu+u^2/2\rvert\leq\lvert u\rvert^3/6 ∣ e i u − 1 − i u + u 2 /2 ∣ ≤ ∣ u ∣ 3 /6 and independence give
E Q ⊗ n e i v ⊤ n − 1 / 2 ∑ i U n , i = [ 1 − v ⊤ Cov Q ( x 1 ( n ) ) v 2 n + O v , P , M ( n − 3 / 2 ) ] n ⟶ e − v ⊤ Σ v / 2 . (131) \mathbb{E}_{Q^{\otimes n}}e^{iv^\top n^{-1/2}\sum_iU_{n,i}}
=
\left[1-\frac{v^\top\operatorname{Cov}_Q(x_1^{(n)})v}{2n}+O_{v,P,M}(n^{-3/2})\right]^n
\longrightarrow e^{-v^\top\Sigma v/2}. \tag*{(131)} E Q ⊗ n e i v ⊤ n − 1/2 ∑ i U n , i = [ 1 − 2 n v ⊤ Cov Q ( x 1 ( n ) ) v + O v , P , M ( n − 3/2 ) ] n ⟶ e − v ⊤ Σ v /2 . (131)
uniformly over Q Q Q , by (F.5) and the expansion log ( 1 + z ) = z + O ( ∣ z ∣ 2 ) \log(1+z)=z+O(|z|^2) log ( 1 + z ) = z + O ( ∣ z ∣ 2 ) near 1. The centering correction n ( E Q x 1 ( n ) − μ Q ) \sqrt{n}(\mathbb{E}_Q x_1^{(n)}-\mu_Q) n ( E Q x 1 ( n ) − μ Q ) is uniformly O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) by (F.4); hence
sup Q ∣ E Q ⊗ n e i v ⊤ Z n , Q − e − v ⊤ Σ v / 2 ∣ → 0. \sup_Q\left|\mathbb{E}_{Q^{\otimes n}}e^{iv^\top Z_{n,Q}}-e^{-v^\top\Sigma v/2}\right|\to0. Q sup E Q ⊗ n e i v ⊤ Z n , Q − e − v ⊤ Σ v /2 → 0.
Lévy’s continuity theorem proves the stated weak limit along every sequence Q n Q_n Q n .
For completeness, fix ϵ > 0 \epsilon>0 ϵ > 0 and let G ∼ N ( 0 , I d ) G\sim\mathcal{N}(0,I_d) G ∼ N ( 0 , I d ) be independent of the sample. All suprema over Q Q Q in this proof are over Q n , M K L ( P ) Q^{\mathrm{KL}}_{n,M}(P) Q n , M KL ( P ) . Independence gives
E e i v ⊤ ( Z n , Q + ϵ G ) = e − ϵ 2 ∥ v ∥ 2 2 / 2 E Q ⊗ n e i v ⊤ Z n , Q . \mathbb{E}e^{iv^\top(Z_{n,Q}+\epsilon G)}=e^{-\epsilon^2\|v\|_2^2/2}\mathbb{E}_{Q^{\otimes n}}e^{iv^\top Z_{n,Q}}. E e i v ⊤ ( Z n , Q + ϵ G ) = e − ϵ 2 ∥ v ∥ 2 2 /2 E Q ⊗ n e i v ⊤ Z n , Q .
The Gaussian factor is integrable. Fourier inversion gives continuous densities p n , Q , ϵ p_{n,Q,\epsilon} p n , Q , ϵ and p ϵ p_\epsilon p ϵ for the smoothed statistic and N ( 0 , Σ + ϵ 2 I d ) \mathcal{N}(0,\Sigma+\epsilon^2I_d) N ( 0 , Σ + ϵ 2 I d ) , respectively, with
sup Q ∥ p n , Q , ϵ − p ϵ ∥ ∞ ≤ ( 2 π ) − d ∫ e − ϵ 2 ∥ v ∥ 2 2 / 2 sup Q ∣ E e i v ⊤ Z n , Q − e − v ⊤ Σ v / 2 ∣ d v → 0. (132) \sup_Q\|p_{n,Q,\epsilon}-p_\epsilon\|_\infty\le(2\pi)^{-d}\int e^{-\epsilon^2\|v\|_2^2/2}\sup_Q\left|\mathbb{E}e^{iv^\top Z_{n,Q}}-e^{-v^\top\Sigma v/2}\right|\,dv\to0. \tag*{(132)} Q sup ∥ p n , Q , ϵ − p ϵ ∥ ∞ ≤ ( 2 π ) − d ∫ e − ϵ 2 ∥ v ∥ 2 2 /2 Q sup E e i v ⊤ Z n , Q − e − v ⊤ Σ v /2 d v → 0. (132)
For every fixed v v v , the supremum inside the integral tends to zero and is at most 2. DCT with the integrable bound 2 e − ϵ 2 ∥ v ∥ 2 2 / 2 2e^{-\epsilon^2\|v\|_2^2/2} 2 e − ϵ 2 ∥ v ∥ 2 2 /2 proves the displayed limit. The supremum is measurable: the uniform first-moment bound from (F.6) makes these characteristic functions uniformly Lipschitz in v v v , and hence their supremum of absolute differences is continuous for sufficiently large n n n .
For L > 0 L>0 L > 0 , split the density integral over B 2 ( L ) B_2(L) B 2 ( L ) and its complement. The uniform second-moment bound in (F.6) and Markov’s inequality give
sup Q ∫ ∣ p n , Q , ϵ − p ϵ ∣ ≤ vol ( B 2 ( L ) ) sup Q ∥ p n , Q , ϵ − p ϵ ∥ ∞ + C P , M , ϵ L − 2 . \sup_Q\int|p_{n,Q,\epsilon}-p_\epsilon|\le\operatorname{vol}(B_2(L))\sup_Q\|p_{n,Q,\epsilon}-p_\epsilon\|_\infty+C_{P,M,\epsilon}L^{-2}. Q sup ∫ ∣ p n , Q , ϵ − p ϵ ∣ ≤ vol ( B 2 ( L )) Q sup ∥ p n , Q , ϵ − p ϵ ∥ ∞ + C P , M , ϵ L − 2 .
First let n → ∞ n\to\infty n → ∞ , then L → ∞ L\to\infty L → ∞ , and use the density formula for total variation. This proves (F.7) for every fixed $\epsilon>0.
Proof of Theorem 5.2
Proof. We first prove (48) . Fix 0 < δ < M 0<\delta<M 0 < δ < M and use g L g_L g L , Γ L \Gamma_L Γ L , J L J_L J L and Q n , u ( L ) Q_{n,u}^{(L)} Q n , u ( L ) from Lemma H.1. Since J L → Σ 1 / 2 J_L\to\Sigma^{1/2} J L → Σ 1/2 as L → ∞ L\to\infty L → ∞ , fix L L L sufficiently large that Γ L ≻ 0 \Gamma_L\succ0 Γ L ≻ 0 and J L J_L J L is invertible. Throughout the sampling limit below, L L L and δ \delta δ are fixed. Equation (H.3), with ϕ ( s ) = s log s − s + 1 \phi(s)=s\log s-s+1 ϕ ( s ) = s log s − s + 1 , gives
sup u ∈ B 2 ( M − δ ) n D K L ( Q n , u ( L ) ∥ P ) ≤ ( M − δ ) 2 2 + o ( 1 ) < M 2 2 . \sup_{u\in B_2(M-\delta)}nD_{\mathrm{KL}}(Q_{n,u}^{(L)}\mathbin{\|}P)\le\frac{(M-\delta)^2}{2}+o(1)<\frac{M^2}{2}. u ∈ B 2 ( M − δ ) sup n D KL ( Q n , u ( L ) ∥ P ) ≤ 2 ( M − δ ) 2 + o ( 1 ) < 2 M 2 .
for all sufficiently large n n n . The density ratios d Q n , u ( L ) / d P dQ_{n,u}^{(L)}/dP d Q n , u ( L ) / d P in (H.2) belong to [ 1 / 2 , 3 / 2 ] [1/2,3/2] [ 1/2 , 3/2 ] for large n n n , so
sup u ∈ B 2 ( M − δ ) E Q n , u ( L ) ∥ x ∥ 2 ≤ 3 2 E P ∥ x ∥ 2 < ∞ . \sup_{u\in B_2(M-\delta)}\mathbb{E}_{Q_{n,u}^{(L)}}\|x\|^2\le\frac{3}{2}\mathbb{E}_P\|x\|^2<\infty. u ∈ B 2 ( M − δ ) sup E Q n , u ( L ) ∥ x ∥ 2 ≤ 2 3 E P ∥ x ∥ 2 < ∞.
Consequently,
{ Q n , u ( L ) : u ∈ B 2 ( M − δ ) } ⊆ Q n , M K L ( P ) , E Q n , u ( L ) X = μ + J L u n . \left\{Q_{n,u}^{(L)}:u\in B_2(M-\delta)\right\}\subseteq Q_{n,M}^{\mathrm{KL}}(P),\qquad\mathbb{E}_{Q_{n,u}^{(L)}}X=\mu+\frac{J_Lu}{\sqrt{n}}. { Q n , u ( L ) : u ∈ B 2 ( M − δ ) } ⊆ Q n , M KL ( P ) , E Q n , u ( L ) X = μ + n J L u .
The second identity is from (H.2). Under P ⊗ n P^{\otimes n} P ⊗ n , the sample likelihoods L n ( L ) L_n^{(L)} L n ( L ) have strictly positive continuous paths, and Lemma H.1 gives
L n ( L ) → d [ u ↦ e u ⊤ Y − ∥ u ∥ 2 2 / 2 ] in C ( B 2 ( M − δ ) ) , Y ∼ N ( 0 , I d ) , L_n^{(L)}\xrightarrow{d}\left[u\mapsto e^{u^\top Y-\lVert u\rVert_2^2/2}\right]\quad\text{in }C(B_2(M-\delta)),\qquad Y\sim\mathcal{N}(0,I_d), L n ( L ) d [ u ↦ e u ⊤ Y − ∥ u ∥ 2 2 /2 ] in C ( B 2 ( M − δ )) , Y ∼ N ( 0 , I d ) ,
sup n l a r g e E P ⊗ n sup u ∈ B 2 ( M − δ ) ∣ L n ( L ) ( u ) ∣ 2 < ∞ . \sup_{n\ \mathrm{large}}\mathbb{E}_{P^{\otimes n}}\sup_{u\in B_2(M-\delta)}\left|L_n^{(L)}(u)\right|^2<\infty. n large sup E P ⊗ n u ∈ B 2 ( M − δ ) sup L n ( L ) ( u ) 2 < ∞.
Thus (H.13) and its required moment bound hold on B 2 ( M − δ ) B_2(M-\delta) B 2 ( M − δ ) .
For every fixed policy w ^ n \widehat{w}_n w n , the submodel inclusion gives
sup Q ∈ Q n , M K L ( P ) E Q ⊗ n R Q ( w ^ n ) ≥ sup u ∈ B 2 ( M − δ ) E ( Q n , u ( L ) ) ⊗ n ℓ n ( J L u , w ^ n ) . \sup_{Q\in Q_{n,M}^{\mathrm{KL}}(P)}\mathbb{E}_{Q^{\otimes n}}R_Q(\widehat{w}_n)\geq\sup_{u\in B_2(M-\delta)}\mathbb{E}_{(Q_{n,u}^{(L)})^{\otimes n}}\ell_n(J_Lu,\widehat{w}_n). Q ∈ Q n , M KL ( P ) sup E Q ⊗ n R Q ( w n ) ≥ u ∈ B 2 ( M − δ ) sup E ( Q n , u ( L ) ) ⊗ n ℓ n ( J L u , w n ) .
Taking the infimum over the same class of sample-based policies on both sides gives
R n , M K L ( P ) ≥ inf w ^ n sup u ∈ B 2 ( M − δ ) E ( Q n , u ( L ) ) ⊗ n ℓ n ( J L u , w ^ n ) . \mathcal{R}_{n,M}^{\mathrm{KL}}(P)\geq\inf_{\widehat{w}_n}\sup_{u\in B_2(M-\delta)}\mathbb{E}_{(Q_{n,u}^{(L)})^{\otimes n}}\ell_n(J_Lu,\widehat{w}_n). R n , M KL ( P ) ≥ w n inf u ∈ B 2 ( M − δ ) sup E ( Q n , u ( L ) ) ⊗ n ℓ n ( J L u , w n ) .
All assumptions of Lemma H.3 have now been verified with radius M − δ M-\delta M − δ and Γ = J L \Gamma=J_L Γ = J L , so
lim inf n → ∞ R n , M K L ( P ) a ( n − 1 / 2 ) ≥ V M − δ , I d ( ψ ∘ J L ) . \liminf_{n\to\infty}\frac{\mathcal{R}_{n,M}^{\mathrm{KL}}(P)}{a(n^{-1/2})}\geq V_{M-\delta,I_d}(\psi\circ J_L). n → ∞ lim inf a ( n − 1/2 ) R n , M KL ( P ) ≥ V M − δ , I d ( ψ ∘ J L ) .
This inequality holds for every sufficiently large fixed L L L . By J L → Σ 1 / 2 J_L\to\Sigma^{1/2} J L → Σ 1/2 and continuity in Lemma H.4, letting first L → ∞ L\to\infty L → ∞ and then δ ↓ 0 \delta\downarrow0 δ ↓ 0 gives
lim inf n → ∞ R n , M K L ( P ) a ( n − 1 / 2 ) ≥ V M − δ , I d ( ψ ∘ Σ 1 / 2 ) → δ ↓ 0 V M , I d ( ψ ∘ Σ 1 / 2 ) . \liminf_{n\to\infty}\frac{\mathcal{R}_{n,M}^{\mathrm{KL}}(P)}{a(n^{-1/2})}\geq V_{M-\delta,I_d}(\psi\circ\Sigma^{1/2})\xrightarrow{\delta\downarrow0}V_{M,I_d}(\psi\circ\Sigma^{1/2}). n → ∞ lim inf a ( n − 1/2 ) R n , M KL ( P ) ≥ V M − δ , I d ( ψ ∘ Σ 1/2 ) δ ↓ 0 V M , I d ( ψ ∘ Σ 1/2 ) .
This proves (48) under the finite-second-moment conditions.
For the upper bound, now additionally assume (42) , and fix δ , ϵ , υ > 0 \delta,\epsilon,\upsilon>0 δ , ϵ , υ > 0 , with δ < 1 \delta<1 δ < 1 . Lemma F.1 implies
t Q : = n ( μ Q − μ ) ∈ Σ 1 / 2 B 2 ( M + δ ) ⊂ B 2 ( L ) , L : = ∥ Σ 1 / 2 ∥ o p ( M + 1 ) , t_Q:=\sqrt{n}(\mu_Q-\mu)\in\Sigma^{1/2}B_2(M+\delta)\subset B_2(L),\qquad L:=\lVert\Sigma^{1/2}\rVert_{\mathrm{op}}(M+1), t Q := n ( μ Q − μ ) ∈ Σ 1/2 B 2 ( M + δ ) ⊂ B 2 ( L ) , L := ∥ Σ 1/2 ∥ op ( M + 1 ) ,
uniformly over Q ∈ Q n , M K L ( P ) Q\in Q_{n,M}^{\mathrm{KL}}(P) Q ∈ Q n , M KL ( P ) , for all sufficiently large n n n . All suprema over Q Q Q below are over Q n , M K L ( P ) Q_{n,M}^{\mathrm{KL}}(P) Q n , M KL ( P ) . The invertible transformation y ↦ Σ − 1 / 2 y y\mapsto\Sigma^{-1/2}y y ↦ Σ − 1/2 y identifies the experiment N ( t , Σ ) \mathcal{N}(t,\Sigma) N ( t , Σ ) , t = Σ 1 / 2 u t=\Sigma^{1/2}u t = Σ 1/2 u , with N ( u , I d ) \mathcal{N}(u,I_d) N ( u , I d ) , and preserves the loss ℓ n ( Σ 1 / 2 u , w ) \ell_n(\Sigma^{1/2}u,w) ℓ n ( Σ 1/2 u , w ) . Thus Lemma H.4 applies.
In the experiment Y ∼ N ( t , Σ ) Y\sim\mathcal{N}(t,\Sigma) Y ∼ N ( t , Σ ) , t ∈ Σ 1 / 2 B 2 ( M + δ ) t\in\Sigma^{1/2}B_2(M+\delta) t ∈ Σ 1/2 B 2 ( M + δ ) , there exists a kernel δ n υ ( d w ∣ y ) \delta_n^\upsilon(dw\mid y) δ n υ ( d w ∣ y ) with δ n υ ( Ω n , L ∣ y ) = 1 \delta_n^\upsilon(\Omega_{n,L}\mid y)=1 δ n υ ( Ω n , L ∣ y ) = 1 for every y ∈ R d y\in\mathbb{R}^d y ∈ R d , satisfying
sup t ∈ Σ 1 / 2 B 2 ( M + δ ) E N ( t , Σ ) ∫ ℓ n ( t , w ) δ n υ ( d w ∣ Y ) ≤ inf δ ′ sup u ∈ B 2 ( M + δ ) E N ( u , I d ) ∫ Ω ℓ n ( Σ 1 / 2 u , w ) δ ′ ( d w ∣ Y ) + υ a ( n − 1 / 2 ) . \sup_{t\in\Sigma^{1/2}B_2(M+\delta)}\mathbb{E}_{\mathcal{N}(t,\Sigma)}\int\ell_n(t,w)\,\delta_n^\upsilon(dw\mid Y)
\leq\inf_{\delta'}\sup_{u\in B_2(M+\delta)}\mathbb{E}_{\mathcal{N}(u,I_d)}\int_{\Omega}\ell_n(\Sigma^{1/2}u,w)\,\delta'(dw\mid Y)+\upsilon a(n^{-1/2}). t ∈ Σ 1/2 B 2 ( M + δ ) sup E N ( t , Σ ) ∫ ℓ n ( t , w ) δ n υ ( d w ∣ Y ) ≤ δ ′ inf u ∈ B 2 ( M + δ ) sup E N ( u , I d ) ∫ Ω ℓ n ( Σ 1/2 u , w ) δ ′ ( d w ∣ Y ) + υ a ( n − 1/2 ) .
Let G ∼ N ( 0 , I d ) G \sim\mathcal{N}(0,I_d) G ∼ N ( 0 , I d ) be independent of the sample and apply δ n ν \delta_n^\nu δ n ν to
Y n ′ = n ( μ ~ n − μ ) + ϵ G = t Q + Z n , Q + ϵ G . Y_n'=\sqrt{n}(\widetilde{\mu}_n-\mu)+\epsilon G=t_Q+Z_{n,Q}+\epsilon G. Y n ′ = n ( μ n − μ ) + ϵ G = t Q + Z n , Q + ϵ G .
For fixed 0 < ϵ ≤ 1 0<\epsilon\leq1 0 < ϵ ≤ 1 , the Gaussian KL formula and Pinsker’s inequality yield
∥ N ( 0 , Σ + ϵ 2 I d ) − N ( 0 , Σ ) ∥ T V ≤ C Σ ϵ 2 , C Σ = tr ( Σ − 2 ) / 8 . \left\|\mathcal{N}(0,\Sigma+\epsilon^2 I_d)-\mathcal{N}(0,\Sigma)\right\|_{\mathrm{TV}}\leq C_\Sigma\epsilon^2,\qquad C_\Sigma=\sqrt{\operatorname{tr}(\Sigma^{-2})/8}. N ( 0 , Σ + ϵ 2 I d ) − N ( 0 , Σ ) TV ≤ C Σ ϵ 2 , C Σ = tr ( Σ − 2 ) /8 .
Thus (F.7) and translation invariance imply
sup Q ∥ L ( Y n ′ ) − N ( t Q , Σ ) ∥ T V ≤ o n ( 1 ) + C Σ ϵ 2 . (133) \sup_Q\left\|\mathcal{L}(Y_n')-\mathcal{N}(t_Q,\Sigma)\right\|_{\mathrm{TV}}\leq o_n(1)+C_\Sigma\epsilon^2. \tag*{(133)} Q sup ∥ L ( Y n ′ ) − N ( t Q , Σ ) ∥ TV ≤ o n ( 1 ) + C Σ ϵ 2 . (133)
The laws include the independent G G G . Here o n ( 1 ) o_n(1) o n ( 1 ) is uniform in Q Q Q for each fixed ϵ \epsilon ϵ ; the limit n → ∞ n\to\infty n → ∞ precedes ϵ ↓ 0 \epsilon\downarrow0 ϵ ↓ 0 . By (B.9),
0 ≤ ∫ ℓ n ( t Q , w ) δ n ν ( d w ∣ y ) ≤ 2 b 3 L ( n − 1 / 2 ) , y ∈ R d . 0\leq\int\ell_n(t_Q,w)\delta_n^\nu(\mathrm{d}w\mid y)\leq2b_{3L}(n^{-1/2}),\qquad y\in\mathbb{R}^d. 0 ≤ ∫ ℓ n ( t Q , w ) δ n ν ( d w ∣ y ) ≤ 2 b 3 L ( n − 1/2 ) , y ∈ R d .
For 0 ≤ g ≤ C 0\leq g\leq C 0 ≤ g ≤ C ,
∣ E P 1 g − E P 2 g ∣ = ∣ ∫ 0 C ( P 1 { g > s } − P 2 { g > s } ) d s ∣ ≤ C ∥ P 1 − P 2 ∥ T V . (134) \left|\mathbb{E}_{P_1}g-\mathbb{E}_{P_2}g\right|=\left|\int_0^C\left(P_1\{g>s\}-P_2\{g>s\}\right)\,\mathrm{d}s\right|\leq C\|P_1-P_2\|_{\mathrm{TV}}. \tag*{(134)} ∣ E P 1 g − E P 2 g ∣ = ∫ 0 C ( P 1 { g > s } − P 2 { g > s } ) d s ≤ C ∥ P 1 − P 2 ∥ TV . (134)
Apply (F.16) with g Q ( y ) = ∫ ℓ n ( t Q , w ) δ n ν ( d w ∣ y ) g_Q(y)=\int\ell_n(t_Q,w)\delta_n^\nu(\mathrm{d}w\mid y) g Q ( y ) = ∫ ℓ n ( t Q , w ) δ n ν ( d w ∣ y ) . The loss bound is uniform in Q Q Q , y y y , and n n n after normalization:
sup Q E g Q ( Y n ′ ) ≤ sup Q E N ( t Q , Σ ) g Q ( Y ) (135) \sup_Q\mathbb{E}g_Q(Y_n')\leq\sup_Q\mathbb{E}_{\mathcal{N}(t_Q,\Sigma)}g_Q(Y) \tag*{(135)} Q sup E g Q ( Y n ′ ) ≤ Q sup E N ( t Q , Σ ) g Q ( Y ) (135)
+ 2 b 3 L ( n − 1 / 2 ) { o n ( 1 ) + C Σ ϵ 2 } . \qquad+2b_{3L}(n^{-1/2})\{o_n(1)+C_\Sigma\epsilon^2\}. + 2 b 3 L ( n − 1/2 ) { o n ( 1 ) + C Σ ϵ 2 } .
In particular, b 3 L ( n − 1 / 2 ) / a ( n − 1 / 2 ) = O ( 1 ) b_{3L}(n^{-1/2})/a(n^{-1/2})=O(1) b 3 L ( n − 1/2 ) / a ( n − 1/2 ) = O ( 1 ) by the local expansion. This restriction of the action space is essential: the global loss bound on Ω \Omega Ω alone need not be of order a ( n − 1 / 2 ) a(n^{-1/2}) a ( n − 1/2 ) . For all sufficiently large n n n , the mean bound above gives
{ t Q : Q ∈ Q n , M K L ( P ) } ⊆ Σ 1 / 2 B 2 ( M + δ ) . \{t_Q:Q\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)\}\subseteq\Sigma^{1/2}B_2(M+\delta). { t Q : Q ∈ Q n , M KL ( P )} ⊆ Σ 1/2 B 2 ( M + δ ) .
Consequently,
sup Q E N ( t Q , Σ ) ∫ ℓ n ( t Q , w ) δ n ν ( d w ∣ Y ) ≤ sup t ∈ Σ 1 / 2 B 2 ( M + δ ) E N ( t , Σ ) ∫ ℓ n ( t , w ) δ n ν ( d w ∣ Y ) . (136) \sup_Q\mathbb{E}_{\mathcal{N}(t_Q,\Sigma)}\int\ell_n(t_Q,w)\delta_n^\nu(\mathrm{d}w\mid Y)
\leq\sup_{t\in\Sigma^{1/2}B_2(M+\delta)}\mathbb{E}_{\mathcal{N}(t,\Sigma)}\int\ell_n(t,w)\delta_n^\nu(\mathrm{d}w\mid Y). \tag*{(136)} Q sup E N ( t Q , Σ ) ∫ ℓ n ( t Q , w ) δ n ν ( d w ∣ Y ) ≤ t ∈ Σ 1/2 B 2 ( M + δ ) sup E N ( t , Σ ) ∫ ℓ n ( t , w ) δ n ν ( d w ∣ Y ) . (136)
Combining (F.17)–(F.18) bounds the randomized rule by the nearly optimal Gaussian risk plus the displayed transfer error. The expectation on the left includes the independent G G G . For each fixed sample, average the selected action over both G G G and the kernel δ n ν \delta_n^\nu δ n ν . This defines the deterministic policy
w ^ n ( x 1 , … , x n ) : = ∫ R d ∫ Ω n , L w δ n ν ( d w ∣ n ( μ ~ n − μ ) + ϵ g ) N ( 0 , I d ) ( d g ) . \widehat{w}_n(x_1,\ldots,x_n):=\int_{\mathbb{R}^d}\int_{\Omega_{n,L}}w\,\delta_n^\nu(\mathrm{d}w\mid\sqrt{n}(\widetilde{\mu}_n-\mu)+\epsilon g)\,\mathcal{N}(0,I_d)(\mathrm{d}g). w n ( x 1 , … , x n ) := ∫ R d ∫ Ω n , L w δ n ν ( d w ∣ n ( μ n − μ ) + ϵ g ) N ( 0 , I d ) ( d g ) .
The bounded Markov-kernel integral is measurable, and compactness and convexity give w ^ n ∈ Ω n , L \hat{w}_n \in\Omega_{n,L} w ^ n ∈ Ω n , L for every sample. Since regret is affine in the action, averaging preserves risk exactly:
E Q ⊗ n R Q ( w ^ n ) = E ∫ ℓ n ( t Q , w ) δ n v ( d w ∣ Y n ′ ) . \mathbb{E}_{Q^{\otimes n}} R_Q(\hat{w}_n) = \mathbb{E}\int\ell_n(t_Q,w)\delta_n^v(\mathrm{d}w \mid Y_n'). E Q ⊗ n R Q ( w ^ n ) = E ∫ ℓ n ( t Q , w ) δ n v ( d w ∣ Y n ′ ) .
Consequently, the maximal risk of this deterministic policy bounds R n , M K L ( P ) \mathcal{R}^{\mathrm{KL}}_{n,M}(P) R n , M KL ( P ) from above.
Also, the uniform local expansion implies
∣ b 3 L ( n − 1 / 2 ) a ( n − 1 / 2 ) − sup t ∈ B 2 ( 3 L ) ψ ( t ) ∣ ≤ sup t ∈ B 2 ( 3 L ) ∣ r μ ( t / n ) a ( n − 1 / 2 ) − ψ ( t ) ∣ → 0. \left|\frac{b_{3L}(n^{-1/2})}{a(n^{-1/2})}-\sup_{t\in B_2(3L)}\psi(t)\right|
\leq\sup_{t\in B_2(3L)}\left|\frac{r_\mu(t/\sqrt{n})}{a(n^{-1/2})}-\psi(t)\right|\to0. a ( n − 1/2 ) b 3 L ( n − 1/2 ) − t ∈ B 2 ( 3 L ) sup ψ ( t ) ≤ t ∈ B 2 ( 3 L ) sup a ( n − 1/2 ) r μ ( t / n ) − ψ ( t ) → 0.
Dividing the risk bound by a ( n − 1 / 2 ) a(n^{-1/2}) a ( n − 1/2 ) , taking n → ∞ n\to\infty n → ∞ with δ , ϵ , υ \delta,\epsilon,\upsilon δ , ϵ , υ fixed, and applying (H.18) gives
lim sup n → ∞ R n , M K L ( P ) a ( n − 1 / 2 ) ≤ V M + δ , I d ( ψ ∘ Σ 1 / 2 ) + υ + 2 C Σ ϵ 2 sup t ∈ B 2 ( 3 L ) ψ ( t ) . \limsup_{n\to\infty}\frac{\mathcal{R}^{\mathrm{KL}}_{n,M}(P)}{a(n^{-1/2})}
\leq V_{M+\delta,I_d}(\psi\circ\Sigma^{1/2})+\upsilon+2C_{\Sigma}\epsilon^2\sup_{t\in B_2(3L)}\psi(t). n → ∞ lim sup a ( n − 1/2 ) R n , M KL ( P ) ≤ V M + δ , I d ( ψ ∘ Σ 1/2 ) + υ + 2 C Σ ϵ 2 t ∈ B 2 ( 3 L ) sup ψ ( t ) .
Letting υ ↓ 0 \upsilon\downarrow0 υ ↓ 0 , ϵ ↓ 0 \epsilon\downarrow0 ϵ ↓ 0 , and δ ↓ 0 \delta\downarrow0 δ ↓ 0 and using continuity in Lemma H.4 proves the upper bound and hence (49) .
If ψ \psi ψ is affine on Σ 1 / 2 B 2 ( M ) \Sigma^{1/2}B_2(M) Σ 1/2 B 2 ( M ) , then ψ ∘ Σ 1 / 2 \psi\circ\Sigma^{1/2} ψ ∘ Σ 1/2 is affine on B 2 ( M ) B_2(M) B 2 ( M ) , and the conditional-mean identity makes every Bayes Jensen gap zero. Thus V M , I d ( ψ ∘ Σ 1 / 2 ) = 0 V_{M,I_d}(\psi\circ\Sigma^{1/2})=0 V M , I d ( ψ ∘ Σ 1/2 ) = 0 . If ψ \psi ψ is nonaffine on Σ 1 / 2 B 2 ( M ) \Sigma^{1/2}B_2(M) Σ 1/2 B 2 ( M ) , Theorem 3.3 gives V M , I d ( ψ ∘ Σ 1 / 2 ) > 0 V_{M,I_d}(\psi\circ\Sigma^{1/2})>0 V M , I d ( ψ ∘ Σ 1/2 ) > 0 . Choose L L L large enough that t Q ∈ B 2 ( L ) t_Q\in B_2(L) t Q ∈ B 2 ( L ) for large n n n and sup B 2 ( L ) ψ > 0 \sup_{B_2(L)}\psi>0 sup B 2 ( L ) ψ > 0 . Note that the deterministic argument leading to (C.6) applies with x ˉ \bar{x} x ˉ replaced by μ ~ n \tilde{\mu}_n μ ~ n . With a similar bound, (C.5) gives
R Q ( w ∗ ( μ ~ n ) ) ≤ r μ ( t Q / n ) + r μ ( ( t Q + 2 Z n , Q ) / n ) ≤ b L ( n − 1 / 2 ) [ 1 + C b ( 1 + 2 ∥ Z n , Q ∥ 2 L ) c b ] . \begin{aligned}
R_Q(w_*(\tilde{\mu}_n))\leq r_\mu(t_Q/\sqrt{n})+r_\mu((t_Q+2Z_{n,Q})/\sqrt{n}) \\
&\leq b_L(n^{-1/2})\left[1+C_b\left(1+\frac{2\|Z_{n,Q}\|_2}{L}\right)^{c_b}\right].
\end{aligned} R Q ( w ∗ ( μ ~ n )) ≤ r μ ( t Q / n ) + r μ (( t Q + 2 Z n , Q ) / n ) ≤ b L ( n − 1/2 ) [ 1 + C b ( 1 + L 2∥ Z n , Q ∥ 2 ) c b ] .
Choose an integer q ≥ max { 1 , c b } q\geq\max\{1,c_b\} q ≥ max { 1 , c b } . By (F.6),
sup Q E Q ⊗ n ( 1 + 2 ∥ Z n , Q ∥ 2 L ) c b ≤ C L ( 1 + sup Q E Q ⊗ n ∥ Z n , Q ∥ 2 q ) ≤ C P , M , L . \sup_Q\mathbb{E}_{Q^{\otimes n}}\left(1+\frac{2\|Z_{n,Q}\|_2}{L}\right)^{c_b}
\leq C_L\left(1+\sup_Q\mathbb{E}_{Q^{\otimes n}}\|Z_{n,Q}\|_2^q\right)\leq C_{P,M,L}. Q sup E Q ⊗ n ( 1 + L 2∥ Z n , Q ∥ 2 ) c b ≤ C L ( 1 + Q sup E Q ⊗ n ∥ Z n , Q ∥ 2 q ) ≤ C P , M , L .
Moreover, (C.3) gives
b L ( n − 1 / 2 ) a ( n − 1 / 2 ) → sup t ∈ B 2 ( L ) ψ ( t ) ∈ ( 0 , ∞ ) . \frac{b_L(n^{-1/2})}{a(n^{-1/2})}\to\sup_{t\in B_2(L)}\psi(t)\in(0,\infty). a ( n − 1/2 ) b L ( n − 1/2 ) → t ∈ B 2 ( L ) sup ψ ( t ) ∈ ( 0 , ∞ ) .
Taking expectations in the deterministic bound therefore yields
sup Q E Q ⊗ n R Q ( w ∗ ( μ ~ n ) ) ≤ C P , M b L ( n − 1 / 2 ) ≤ C P , M ′ a ( n − 1 / 2 ) . (137) \sup_Q\mathbb{E}_{Q^{\otimes n}}R_Q(w_*(\tilde{\mu}_n))\leq C_{P,M}b_L(n^{-1/2})\leq C'_{P,M}a(n^{-1/2}). \tag*{(137)} Q sup E Q ⊗ n R Q ( w ∗ ( μ ~ n )) ≤ C P , M b L ( n − 1/2 ) ≤ C P , M ′ a ( n − 1/2 ) . (137)
Since the limiting constant is positive, (49) gives, for all sufficiently large n n n ,
1 2 V M , I d ( ψ ∘ Σ 1 / 2 ) a ( n − 1 / 2 ) ≤ R n , M K L ( P ) ≤ sup Q E Q ⊗ n R Q ( w ∗ ( μ ~ n ) ) . \frac{1}{2}V_{M,I_d}(\psi\circ\Sigma^{1/2})a(n^{-1/2})\leq\mathcal{R}^{\mathrm{KL}}_{n,M}(P)\leq\sup_Q\mathbb{E}_{Q^{\otimes n}}R_Q(w_*(\tilde{\mu}_n)). 2 1 V M , I d ( ψ ∘ Σ 1/2 ) a ( n − 1/2 ) ≤ R n , M KL ( P ) ≤ Q sup E Q ⊗ n R Q ( w ∗ ( μ ~ n )) .
The second inequality follows from the definition of the minimax risk and feasibility of the truncated plug-in policy. Together with the upper bound just proved, this gives (50) .
If the limiting constant is zero, (49) instead gives R n , M K L ( P ) = o ( a ( n − 1 / 2 ) ) \mathcal{R}^{\mathrm{KL}}_{n,M}(P)=o(a(n^{-1/2})) R n , M KL ( P ) = o ( a ( n − 1/2 )) .
Proof of Corollary 5.3
Proof. The expansion has a ( ε ) = ε 2 a(\varepsilon)=\varepsilon^2 a ( ε ) = ε 2 and ψ ( t ) = t ⊤ H μ t / 2 \psi(t)=t^\top H_\mu t/2 ψ ( t ) = t ⊤ H μ t /2 . Theorem 5.2 and the quadratic Gaussian limit give (51) –(52) .
For (53) , fix M M M and take all suprema over Q n , M K L ( P ) Q^{\mathrm{KL}}_{n,M}(P) Q n , M KL ( P ) . Write t Q = n ( μ Q − μ ) t_Q=\sqrt{n}(\mu_Q-\mu) t Q = n ( μ Q − μ ) and Z = Z n , Q Z=Z_{n,Q} Z = Z n , Q . By (F.2), sup Q ∥ t Q ∥ 2 ≤ L 0 \sup_Q\|t_Q\|_2\leq L_0 sup Q ∥ t Q ∥ 2 ≤ L 0 for large n n n . Choose r > 0 r>0 r > 0 so that f ∗ f_* f ∗ is C 2 C^2 C 2 near μ + B 2 ( r ) \mu+B_2(r) μ + B 2 ( r ) , and let C r C_r C r bound its Hessian there. For large n n n , μ Q ∈ μ + B 2 ( r / 2 ) \mu_Q\in\mu+B_2(r/2) μ Q ∈ μ + B 2 ( r /2 ) . On A n , Q = { ∥ μ ~ n − μ ∥ 2 ≤ r } A_{n,Q}=\{\|\widetilde{\mu}_n-\mu\|_2\leq r\} A n , Q = { ∥ μ n − μ ∥ 2 ≤ r } , the support identity and Taylor’s integral formula yield
n R Q ( w ∗ ( μ ~ n ) ) = ∫ 0 1 ( 1 − s ) Z ⊤ ∇ 2 f ∗ ( μ + t Q + ( 1 − s ) Z n ) Z d s . nR_Q(w_*(\widetilde{\mu}_n))=\int_0^1(1-s)Z^\top\nabla^2f_*\left(\mu+\frac{t_Q+(1-s)Z}{\sqrt{n}}\right)Z\,\mathrm{d}s. n R Q ( w ∗ ( μ n )) = ∫ 0 1 ( 1 − s ) Z ⊤ ∇ 2 f ∗ ( μ + n t Q + ( 1 − s ) Z ) Z d s .
Put D n , Q = n R Q ( w ∗ ( μ ~ n ) ) − Z ⊤ H μ Z / 2 D_{n,Q}=nR_Q(w_*(\widetilde{\mu}_n))-Z^\top H_\mu Z/2 D n , Q = n R Q ( w ∗ ( μ n )) − Z ⊤ H μ Z /2 . For each fixed L L L and sufficiently large n n n , { ∥ Z ∥ 2 ≤ L } ⊆ A n , Q \{\|Z\|_2\leq L\}\subseteq A_{n,Q} { ∥ Z ∥ 2 ≤ L } ⊆ A n , Q uniformly over Q Q Q . Continuity of the Hessian gives
sup Q E [ ∣ D n , Q ∣ I { ∥ Z ∥ 2 ≤ L } ] ≤ L 2 2 sup ∥ h ∥ 2 ≤ ( L 0 + L ) / n ∥ ∇ 2 f ∗ ( μ + h ) − H μ ∥ o p → 0. \sup_Q\mathbb{E}[|D_{n,Q}|\mathbb{I}\{\|Z\|_2\leq L\}]
\leq\frac{L^2}{2}\sup_{\|h\|_2\leq(L_0+L)/\sqrt{n}}\|\nabla^2f_*(\mu+h)-H_\mu\|_{\mathrm{op}}\to0. Q sup E [ ∣ D n , Q ∣ I { ∥ Z ∥ 2 ≤ L }] ≤ 2 L 2 ∥ h ∥ 2 ≤ ( L 0 + L ) / n sup ∥ ∇ 2 f ∗ ( μ + h ) − H μ ∥ op → 0.
On A n , Q A_{n,Q} A n , Q , ∣ D n , Q ∣ ≤ ( C r + ∥ H μ ∥ o p ) ∥ Z ∥ 2 2 / 2 |D_{n,Q}|\leq(C_r+\|H_\mu\|_{\mathrm{op}})\|Z\|_2^2/2 ∣ D n , Q ∣ ≤ ( C r + ∥ H μ ∥ op ) ∥ Z ∥ 2 2 /2 . The uniform fourth-moment bound in (F.6) therefore implies
sup Q E [ ∣ D n , Q ∣ I A n , Q I { ∥ Z ∥ 2 > L } ] ≤ C L − 2 . \sup_Q\mathbb{E}[|D_{n,Q}|\mathbb{I}_{A_{n,Q}}\mathbb{I}\{\|Z\|_2>L\}]\leq CL^{-2}. Q sup E [ ∣ D n , Q ∣ I A n , Q I { ∥ Z ∥ 2 > L }] ≤ C L − 2 .
On A n , Q c A_{n,Q}^{c} A n , Q c , ∥ Z ∥ 2 > r n − L 0 ≥ r n / 2 \|Z\|_2>r\sqrt{n}-L_0\geq r\sqrt{n}/2 ∥ Z ∥ 2 > r n − L 0 ≥ r n /2 . Since every feasible action has regret at most 2 R Ω ∥ μ Q ∥ 2 ≤ C 2R_\Omega\|\mu_Q\|_2\leq C 2 R Ω ∥ μ Q ∥ 2 ≤ C , the same fourth-moment bound gives
sup Q E [ ∣ D n , Q ∣ I A n , Q c ] ≤ C n sup Q P { ∥ Z ∥ 2 > r n / 2 } + C sup Q E [ ∥ Z ∥ 2 2 I { ∥ Z ∥ 2 > r n / 2 } ] = O ( n − 1 ) . \begin{aligned}
\sup_Q\mathbb{E}[|D_{n,Q}|\mathbb{I}_{A_{n,Q}^{c}}]
&\leq Cn\sup_Q\mathbb{P}\{\|Z\|_2>r\sqrt{n}/2\}\\
&\quad+C\sup_Q\mathbb{E}[\|Z\|_2^2\mathbb{I}\{\|Z\|_2>r\sqrt{n}/2\}]\\
&=O(n^{-1}).
\end{aligned} Q sup E [ ∣ D n , Q ∣ I A n , Q c ] ≤ C n Q sup P { ∥ Z ∥ 2 > r n /2 } + C Q sup E [ ∥ Z ∥ 2 2 I { ∥ Z ∥ 2 > r n /2 }] = O ( n − 1 ) .
First taking n → ∞ n\to\infty n → ∞ and then L → ∞ L\to\infty L → ∞ proves sup Q E ∣ D n , Q ∣ → 0 \sup_Q\mathbb{E}|D_{n,Q}|\to0 sup Q E ∣ D n , Q ∣ → 0 .
Finally, independence gives
E Z Z ⊤ = Cov Q ( x 1 ( n ) ) + n b n , Q b n , Q ⊤ , b n , Q = E Q x 1 ( n ) − μ Q . \mathbb{E}ZZ^\top=\operatorname{Cov}_Q(x_1^{(n)})+nb_{n,Q}b_{n,Q}^\top,\qquad b_{n,Q}=\mathbb{E}_Qx_1^{(n)}-\mu_Q. E Z Z ⊤ = Cov Q ( x 1 ( n ) ) + n b n , Q b n , Q ⊤ , b n , Q = E Q x 1 ( n ) − μ Q .
Equations (F.5) and (F.4) imply uniform convergence of this matrix to Σ \Sigma Σ . Taking traces against H μ H_\mu H μ and combining with the L 1 L^1 L 1 approximation proves (53) .
Boundary growth on KL neighborhoods
Theorem F.2 (Boundary growth and matching rates on a KL neighborhood). Let d ≥ 2 d\geq2 d ≥ 2 , let Ω \Omega Ω be compact and convex with nonempty interior, and suppose μ ≠ 0 \mu\ne0 μ = 0 has a unique maximizer w ∗ ( μ ) w_*(\mu) w ∗ ( μ ) . Retain n μ = μ / ∥ μ ∥ 2 n_\mu=\mu/\|\mu\|_2 n μ = μ /∥ μ ∥ 2 and the orthonormal basis matrix U μ U_\mu U μ of n μ ⊥ n_\mu^\perp n μ ⊥ . For
the lower bound, assume only E P ∥ x ∥ 2 2 < ∞ \mathbb{E}_{P}\lVert x\rVert_{2}^{2}<\infty E P ∥ x ∥ 2 2 < ∞ and Σ = Cov P ( x ) ≻ 0 \Sigma=\operatorname{Cov}_{P}(x)\succ0 Σ = Cov P ( x ) ≻ 0 . Suppose that, for some ρ , κ ˉ , β > 0 \rho,\bar{\kappa},\beta>0 ρ , κ ˉ , β > 0 , a boundary profile ϱ μ \varrho_{\mu} ϱ μ satisfies
w ∗ ( μ ) + U μ z − n μ ϱ μ ( z ) ∈ Ω , ϱ μ ( 0 ) = 0 , w_{*}(\mu)+U_{\mu}z-n_{\mu}\varrho_{\mu}(z)\in\Omega,\qquad\varrho_{\mu}(0)=0, w ∗ ( μ ) + U μ z − n μ ϱ μ ( z ) ∈ Ω , ϱ μ ( 0 ) = 0 ,
0 ≤ ϱ μ ( z ) ≤ κ ˉ 1 + β ∥ z ∥ 2 1 + β , ∥ z ∥ 2 ≤ ρ . (138) 0\leq\varrho_{\mu}(z)\leq\frac{\bar{\kappa}}{1+\beta}\lVert z\rVert_{2}^{1+\beta},\qquad\lVert z\rVert_{2}\leq\rho. \tag*{(138)} 0 ≤ ϱ μ ( z ) ≤ 1 + β κ ˉ ∥ z ∥ 2 1 + β , ∥ z ∥ 2 ≤ ρ . (138)
For every M > 0 M>0 M > 0 , every unit v ∈ R d − 1 v\in\mathbb{R}^{d-1} v ∈ R d − 1 , and every h > 0 h>0 h > 0 with h ∥ Σ − 1 / 2 U μ v ∥ 2 < M h\lVert\Sigma^{-1/2}U_{\mu}v\rVert_{2}<M h ∥ Σ − 1/2 U μ v ∥ 2 < M ,
lim inf n → ∞ n ( 1 + β ) / ( 2 β ) R n , M K L ( P ) ≥ 2 Φ ( − h ∥ Σ − 1 / 2 U μ v ∥ 2 ) β 1 + β ( κ ˉ ∥ μ ∥ 2 ) − 1 / β h ( 1 + β ) / β > 0. (139) \liminf_{n\to\infty}n^{(1+\beta)/(2\beta)}\mathcal{R}^{\mathrm{KL}}_{n,M}(P)\geq2\Phi\left(-h\lVert\Sigma^{-1/2}U_{\mu}v\rVert_{2}\right)\frac{\beta}{1+\beta}(\bar{\kappa}\lVert\mu\rVert_{2})^{-1/\beta}h^{(1+\beta)/\beta}>0. \tag*{(139)} n → ∞ lim inf n ( 1 + β ) / ( 2 β ) R n , M KL ( P ) ≥ 2Φ ( − h ∥ Σ − 1/2 U μ v ∥ 2 ) 1 + β β ( κ ˉ ∥ μ ∥ 2 ) − 1/ β h ( 1 + β ) / β > 0. (139)
For the matching upper bound, assume additionally (42) . Suppose that ϱ μ \varrho_{\mu} ϱ μ describes the entire local feasible set and has a corresponding lower growth bound: for some 0 < κ ≤ κ ˉ 0<\kappa\leq\bar{\kappa} 0 < κ ≤ κ ˉ , after decreasing ρ \rho ρ if necessary,
w ∗ ( μ ) + U μ z − n μ b ∈ Ω ⟺ ϱ μ ( z ) ≤ b , ∥ z ∥ 2 ≤ ρ , ∣ b ∣ ≤ ρ , w_{*}(\mu)+U_{\mu}z-n_{\mu}b\in\Omega\quad\Longleftrightarrow\quad\varrho_{\mu}(z)\leq b,\qquad\lVert z\rVert_{2}\leq\rho,\quad|b|\leq\rho, w ∗ ( μ ) + U μ z − n μ b ∈ Ω ⟺ ϱ μ ( z ) ≤ b , ∥ z ∥ 2 ≤ ρ , ∣ b ∣ ≤ ρ ,
κ 1 + β ∥ z ∥ 2 1 + β ≤ ϱ μ ( z ) ≤ κ ˉ 1 + β ∥ z ∥ 2 1 + β , ∥ z ∥ 2 ≤ ρ . (140) \frac{\kappa}{1+\beta}\lVert z\rVert_{2}^{1+\beta}\leq\varrho_{\mu}(z)\leq\frac{\bar{\kappa}}{1+\beta}\lVert z\rVert_{2}^{1+\beta},\qquad\lVert z\rVert_{2}\leq\rho. \tag*{(140)} 1 + β κ ∥ z ∥ 2 1 + β ≤ ϱ μ ( z ) ≤ 1 + β κ ˉ ∥ z ∥ 2 1 + β , ∥ z ∥ 2 ≤ ρ . (140)
Then, for every fixed M > 0 M>0 M > 0 and all sufficiently large n n n ,
c P , M n − ( 1 + β ) / ( 2 β ) ≤ R n , M K L ( P ) ≤ sup Q ∈ Q n , M K L ( P ) E Q ⊗ n R Q ( w ∗ ( μ ~ n ) ) ≤ C P , M n − ( 1 + β ) / ( 2 β ) , (141) c_{P,M}n^{-(1+\beta)/(2\beta)}\leq\mathcal{R}^{\mathrm{KL}}_{n,M}(P)\leq\sup_{Q\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)}\mathbb{E}_{Q^{\otimes n}}R_{Q}(w_{*}(\widetilde{\mu}_{n}))\leq C_{P,M}n^{-(1+\beta)/(2\beta)}, \tag*{(141)} c P , M n − ( 1 + β ) / ( 2 β ) ≤ R n , M KL ( P ) ≤ Q ∈ Q n , M KL ( P ) sup E Q ⊗ n R Q ( w ∗ ( μ n )) ≤ C P , M n − ( 1 + β ) / ( 2 β ) , (141)
where 0 < c P , M ≤ C P , M < ∞ 0<c_{P,M}\leq C_{P,M}<\infty 0 < c P , M ≤ C P , M < ∞ may also depend on the fixed geometry.
Proof. We first prove the lower bound. Fix any v ∈ R d − 1 v\in\mathbb{R}^{d-1} v ∈ R d − 1 with ∥ v ∥ 2 = 1 \lVert v\rVert_{2}=1 ∥ v ∥ 2 = 1 , fix h > 0 h>0 h > 0 with h ∥ Σ − 1 / 2 U μ v ∥ 2 < M h\lVert\Sigma^{-1/2}U_{\mu}v\rVert_{2}<M h ∥ Σ − 1/2 U μ v ∥ 2 < M , and take t = h U μ v t=hU_{\mu}v t = h U μ v . For 0 ≤ b ≤ ρ 0\leq b\leq\rho 0 ≤ b ≤ ρ , (F.20) gives w ∗ ( μ ) + b U μ v − n μ ϱ μ ( b v ) ∈ Ω w_{*}(\mu)+bU_{\mu}v-n_{\mu}\varrho_{\mu}(bv)\in\Omega w ∗ ( μ ) + b U μ v − n μ ϱ μ ( b v ) ∈ Ω . Since μ T U μ v = n μ T U μ v = 0 \mu^{\mathsf{T}}U_{\mu}v=n_{\mu}^{\mathsf{T}}U_{\mu}v=0 μ T U μ v = n μ T U μ v = 0 , and U μ T U μ = I d − 1 U_{\mu}^{\mathsf{T}}U_{\mu}=I_{d-1} U μ T U μ = I d − 1 , the support-function definition gives
r μ ( t / n ) = sup w ∈ Ω ( μ + t / n ) T ( w − w ∗ ( μ ) ) ≥ ( μ + h U μ v / n ) T ( b U μ v − n μ ϱ μ ( b v ) ) = h b n − ∥ μ ∥ 2 ϱ μ ( b v ) ≥ h b n − κ ˉ ∥ μ ∥ 2 1 + β b 1 + β . \begin{aligned}
r_{\mu}(t/\sqrt{n})=\sup_{w\in\Omega}(\mu+t/\sqrt{n})^{\mathsf{T}}(w-w_{*}(\mu)) \\
&\geq(\mu+hU_{\mu}v/\sqrt{n})^{\mathsf{T}}(bU_{\mu}v-n_{\mu}\varrho_{\mu}(bv)) \\
&=\frac{hb}{\sqrt{n}}-\lVert\mu\rVert_{2}\varrho_{\mu}(bv) \\
&\geq\frac{hb}{\sqrt{n}}-\frac{\bar{\kappa}\lVert\mu\rVert_{2}}{1+\beta}b^{1+\beta}.
\end{aligned} r μ ( t / n ) = w ∈ Ω sup ( μ + t / n ) T ( w − w ∗ ( μ )) ≥ ( μ + h U μ v / n ) T ( b U μ v − n μ ϱ μ ( b v )) = n hb − ∥ μ ∥ 2 ϱ μ ( b v ) ≥ n hb − 1 + β κ ˉ ∥ μ ∥ 2 b 1 + β .
The same calculation with − b v -bv − b v gives the identical bound for r μ ( − t / n ) r_{\mu}(-t/\sqrt{n}) r μ ( − t / n ) . The concave function b ↦ h b n − κ ˉ ∥ μ ∥ 2 1 + β b 1 + β b\mapsto\frac{hb}{\sqrt{n}}-\frac{\bar{\kappa}\lVert\mu\rVert_{2}}{1+\beta}b^{1+\beta} b ↦ n hb − 1 + β κ ˉ ∥ μ ∥ 2 b 1 + β on the last line is maximized over b ≥ 0 b\geq0 b ≥ 0 at
b = ( h κ ˉ ∥ μ ∥ 2 n ) 1 / β . b=\left(\frac{h}{\bar{\kappa}\lVert\mu\rVert_{2}\sqrt{n}}\right)^{1/\beta}. b = ( κ ˉ ∥ μ ∥ 2 n h ) 1/ β .
For sufficiently large n n n , h / n ≤ κ ˉ ∥ μ ∥ 2 ρ β h/\sqrt{n}\leq\bar{\kappa}\lVert\mu\rVert_{2}\rho^{\beta} h / n ≤ κ ˉ ∥ μ ∥ 2 ρ β , so this value belongs to [ 0 , ρ ] [0,\rho] [ 0 , ρ ] . Substitution yields
min { r μ ( t / n ) , r μ ( − t / n ) } ≥ β 1 + β ( h n ) ( 1 + β ) / β ( κ ˉ ∥ μ ∥ 2 ) − 1 / β . (142) \min\{r_{\mu}(t/\sqrt{n}),r_{\mu}(-t/\sqrt{n})\}\geq\frac{\beta}{1+\beta}\left(\frac{h}{\sqrt{n}}\right)^{(1+\beta)/\beta}(\bar{\kappa}\lVert\mu\rVert_{2})^{-1/\beta}. \tag*{(142)} min { r μ ( t / n ) , r μ ( − t / n )} ≥ 1 + β β ( n h ) ( 1 + β ) / β ( κ ˉ ∥ μ ∥ 2 ) − 1/ β . (142)
Use the tilted construction of distributions in Lemma H.1 . Since J L → Σ 1 / 2 J_L \to\Sigma^{1/2} J L → Σ 1/2 , for every sufficiently large fixed L L L the vector u = J L − 1 t u = J_L^{-1}t u = J L − 1 t satisfies ∥ u ∥ 2 < M \lVert u\rVert_2 < M ∥ u ∥ 2 < M . Equations (H.2) and (H.3), with ϕ ( s ) = s log s − s + 1 \phi(s) = s\log s - s + 1 ϕ ( s ) = s log s − s + 1 , give
μ Q n , ± u ( L ) = μ ± t n , n D K L ( Q n , ± u ( L ) ∥ P ) → ∥ u ∥ 2 2 2 < M 2 2 . \mu_{Q_{n,\pm u}^{(L)}} = \mu\pm\frac{t}{\sqrt{n}}, \qquad nD_{\mathrm{KL}}\left(Q_{n,\pm u}^{(L)}\mathbin\Vert P\right) \to\frac{\lVert u\rVert_2^2}{2} < \frac{M^2}{2}. μ Q n , ± u ( L ) = μ ± n t , n D KL ( Q n , ± u ( L ) ∥ P ) → 2 ∥ u ∥ 2 2 < 2 M 2 .
Both distributions belong to Q n , M K L ( P ) Q_{n,M}^{\mathrm{KL}}(P) Q n , M KL ( P ) for large n n n .
Use the equal-probability prior on Q n , u ( L ) Q_{n,u}^{(L)} Q n , u ( L ) and Q n , − u ( L ) Q_{n,-u}^{(L)} Q n , − u ( L ) , whose mean perturbations are t t t and − t -t − t . The same binary-prior argument in the proof of Proposition 3.4 applies. Since both distributions belong to Q n , M K L ( P ) Q_{n,M}^{\mathrm{KL}}(P) Q n , M KL ( P ) , every measurable sample-based policy w ^ n \widehat{w}_n w n satisfies
sup Q ∈ Q n , M K L ( P ) E Q ⊗ n R Q ( w ^ n ) ≥ 1 2 E ( Q n , u ( L ) ) ⊗ n ℓ n ( t , w ^ n ) + 1 2 E ( Q n , − u ( L ) ) ⊗ n ℓ n ( − t , w ^ n ) . \sup_{Q\in Q_{n,M}^{\mathrm{KL}}(P)}\mathbb{E}_{Q^{\otimes n}}R_Q(\widehat{w}_n) \geq\frac{1}{2}\mathbb{E}_{(Q_{n,u}^{(L)})^{\otimes n}}\ell_n(t,\widehat{w}_n) + \frac{1}{2}\mathbb{E}_{(Q_{n,-u}^{(L)})^{\otimes n}}\ell_n(-t,\widehat{w}_n). Q ∈ Q n , M KL ( P ) sup E Q ⊗ n R Q ( w n ) ≥ 2 1 E ( Q n , u ( L ) ) ⊗ n ℓ n ( t , w n ) + 2 1 E ( Q n , − u ( L ) ) ⊗ n ℓ n ( − t , w n ) .
Taking the infimum over the same class of policies gives
R n , M K L ( P ) ≥ inf w ^ n { 1 2 E ( Q n , u ( L ) ) ⊗ n ℓ n ( t , w ^ n ) + 1 2 E ( Q n , − u ( L ) ) ⊗ n ℓ n ( − t , w ^ n ) } . \mathcal{R}_{n,M}^{\mathrm{KL}}(P) \geq\inf_{\widehat{w}_n}\left\{\frac{1}{2}\mathbb{E}_{(Q_{n,u}^{(L)})^{\otimes n}}\ell_n(t,\widehat{w}_n) + \frac{1}{2}\mathbb{E}_{(Q_{n,-u}^{(L)})^{\otimes n}}\ell_n(-t,\widehat{w}_n)\right\}. R n , M KL ( P ) ≥ w n inf { 2 1 E ( Q n , u ( L ) ) ⊗ n ℓ n ( t , w n ) + 2 1 E ( Q n , − u ( L ) ) ⊗ n ℓ n ( − t , w n ) } .
The right side is the minimum Bayes risk for this two-point prior.
For 0 ≤ λ ≤ 1 0 \leq\lambda\leq1 0 ≤ λ ≤ 1 , by convexity of r μ r_\mu r μ , note that
λ r μ ( t / n ) + ( 1 − λ ) r μ ( − t / n ) − r μ ( ( 2 λ − 1 ) t / n ) ≥ min { λ , 1 − λ } { r μ ( t / n ) + r μ ( − t / n ) } . (143) \lambda r_\mu(t/\sqrt{n}) + (1-\lambda)r_\mu(-t/\sqrt{n}) - r_\mu((2\lambda-1)t/\sqrt{n})
\geq\min\{\lambda,1-\lambda\}\{r_\mu(t/\sqrt{n}) + r_\mu(-t/\sqrt{n})\}. \tag*{(143)} λ r μ ( t / n ) + ( 1 − λ ) r μ ( − t / n ) − r μ (( 2 λ − 1 ) t / n ) ≥ min { λ , 1 − λ } { r μ ( t / n ) + r μ ( − t / n )} . (143)
Now let λ \lambda λ be the posterior probability of Q n , u ( L ) Q_{n,u}^{(L)} Q n , u ( L ) given the sample. Bayes’ formula gives
λ = 1 2 L n ( L ) ( u ) 1 2 L n ( L ) ( u ) + 1 2 L n ( L ) ( − u ) = L n ( L ) ( u ) L n ( L ) ( u ) + L n ( L ) ( − u ) . \lambda= \frac{\frac{1}{2}L_n^{(L)}(u)}{\frac{1}{2}L_n^{(L)}(u)+\frac{1}{2}L_n^{(L)}(-u)}
= \frac{L_n^{(L)}(u)}{L_n^{(L)}(u)+L_n^{(L)}(-u)}. λ = 2 1 L n ( L ) ( u ) + 2 1 L n ( L ) ( − u ) 2 1 L n ( L ) ( u ) = L n ( L ) ( u ) + L n ( L ) ( − u ) L n ( L ) ( u ) .
The denominator is positive. Conditional on the sample, the expected regret of an action w ∈ Ω w \in\Omega w ∈ Ω is
λ ℓ n ( t , w ) + ( 1 − λ ) ℓ n ( − t , w ) = λ f ∗ ( μ + t / n ) + ( 1 − λ ) f ∗ ( μ − t / n ) − w ⊤ ( μ + ( 2 λ − 1 ) t n ) ≥ λ f ∗ ( μ + t / n ) + ( 1 − λ ) f ∗ ( μ − t / n ) − f ∗ ( μ + ( 2 λ − 1 ) t / n ) = λ r μ ( t / n ) + ( 1 − λ ) r μ ( − t / n ) − r μ ( ( 2 λ − 1 ) t / n ) . \begin{aligned}
\lambda\ell_n(t,w) + (1-\lambda)\ell_n(-t,w)
&= \lambda f_*(\mu+t/\sqrt{n}) + (1-\lambda)f_*(\mu-t/\sqrt{n}) - w^\top\left(\mu+\frac{(2\lambda-1)t}{\sqrt{n}}\right) \\
&\geq\lambda f_*(\mu+t/\sqrt{n}) + (1-\lambda)f_*(\mu-t/\sqrt{n}) - f_*(\mu+(2\lambda-1)t/\sqrt{n}) \\
&= \lambda r_\mu(t/\sqrt{n}) + (1-\lambda)r_\mu(-t/\sqrt{n}) - r_\mu((2\lambda-1)t/\sqrt{n}).
\end{aligned} λ ℓ n ( t , w ) + ( 1 − λ ) ℓ n ( − t , w ) = λ f ∗ ( μ + t / n ) + ( 1 − λ ) f ∗ ( μ − t / n ) − w ⊤ ( μ + n ( 2 λ − 1 ) t ) ≥ λ f ∗ ( μ + t / n ) + ( 1 − λ ) f ∗ ( μ − t / n ) − f ∗ ( μ + ( 2 λ − 1 ) t / n ) = λ r μ ( t / n ) + ( 1 − λ ) r μ ( − t / n ) − r μ (( 2 λ − 1 ) t / n ) .
where the equality is attained by the Bayes policy w ∗ ( μ + ( 2 λ − 1 ) t / n ) w_*(\mu+(2\lambda-1)t/\sqrt{n}) w ∗ ( μ + ( 2 λ − 1 ) t / n ) . The marginal sample law under the prior is 1 2 ( Q n , u ( L ) ) ⊗ n + 1 2 ( Q n , − u ( L ) ) ⊗ n \frac{1}{2}(Q_{n,u}^{(L)})^{\otimes n}+\frac{1}{2}(Q_{n,-u}^{(L)})^{\otimes n} 2 1 ( Q n , u ( L ) ) ⊗ n + 2 1 ( Q n , − u ( L ) ) ⊗ n , whose density relative to P ⊗ n P^{\otimes n} P ⊗ n is ( L n ( L ) ( u ) + L n ( L ) ( − u ) ) / 2 (L_n^{(L)}(u)+L_n^{(L)}(-u))/2 ( L n ( L ) ( u ) + L n ( L ) ( − u )) /2 . Since the Bayes policy w ∗ ( μ + ( 2 λ − 1 ) t / n ) w_*(\mu+(2\lambda-1)t/\sqrt{n}) w ∗ ( μ + ( 2 λ − 1 ) t / n ) attains the conditional minimum for each sample, its integrated risk equals
inf w ^ n { 1 2 E ( Q n , u ( L ) ) ⊗ n ℓ n ( t , w ^ n ) + 1 2 E ( Q n , − u ( L ) ) ⊗ n ℓ n ( − t , w ^ n ) } = E P ⊗ n [ L n ( L ) ( u ) + L n ( L ) ( − u ) 2 { λ r μ ( t / n ) + ( 1 − λ ) r μ ( − t / n ) − r μ ( ( 2 λ − 1 ) t / n ) } ] . (144) \begin{aligned}
\inf_{\widehat{w}_n}\left\{\frac{1}{2}\mathbb{E}_{(Q_{n,u}^{(L)})^{\otimes n}}\ell_n(t,\widehat{w}_n) + \frac{1}{2}\mathbb{E}_{(Q_{n,-u}^{(L)})^{\otimes n}}\ell_n(-t,\widehat{w}_n)\right\}
= \mathbb{E}_{P^{\otimes n}}\left[\frac{L_n^{(L)}(u)+L_n^{(L)}(-u)}{2}\left\{\lambda r_\mu(t/\sqrt{n}) + (1-\lambda)r_\mu(-t/\sqrt{n}) - r_\mu((2\lambda-1)t/\sqrt{n})\right\}\right]. \tag*{(144)}
\end{aligned} w n inf { 2 1 E ( Q n , u ( L ) ) ⊗ n ℓ n ( t , w n ) + 2 1 E ( Q n , − u ( L ) ) ⊗ n ℓ n ( − t , w n ) } = E P ⊗ n [ 2 L n ( L ) ( u ) + L n ( L ) ( − u ) { λ r μ ( t / n ) + ( 1 − λ ) r μ ( − t / n ) − r μ (( 2 λ − 1 ) t / n ) } ] . (144)
where
L n ( L ) ( u ) + L n ( L ) ( − u ) 2 min { λ , 1 − λ } = L n ( L ) ( u ) + L n ( L ) ( − u ) 2 min { L n ( L ) ( u ) , L n ( L ) ( − u ) } L n ( L ) ( u ) + L n ( L ) ( − u ) = 1 2 min { L n ( L ) ( u ) , L n ( L ) ( − u ) } . \frac{L_n^{(L)}(u)+L_n^{(L)}(-u)}{2}\min\{\lambda,1-\lambda\}
=
\frac{L_n^{(L)}(u)+L_n^{(L)}(-u)}{2}
\frac{\min\{L_n^{(L)}(u),L_n^{(L)}(-u)\}}{L_n^{(L)}(u)+L_n^{(L)}(-u)}
=
\frac{1}{2}\min\{L_n^{(L)}(u),L_n^{(L)}(-u)\}. 2 L n ( L ) ( u ) + L n ( L ) ( − u ) min { λ , 1 − λ } = 2 L n ( L ) ( u ) + L n ( L ) ( − u ) L n ( L ) ( u ) + L n ( L ) ( − u ) min { L n ( L ) ( u ) , L n ( L ) ( − u )} = 2 1 min { L n ( L ) ( u ) , L n ( L ) ( − u )} .
Applying (F.25) inside (F.26) gives
R n , M K L ( P ) ≥ 1 2 ( r μ ( t / n ) + r μ ( − t / n ) ) E P ⊗ n min { L n ( L ) ( u ) , L n ( L ) ( − u ) } . (145) \mathcal{R}_{n,M}^{\mathrm{KL}}(P)\geq\frac{1}{2}\left(r_\mu(t/\sqrt{n})+r_\mu(-t/\sqrt{n})\right)\mathbb{E}_{P^{\otimes n}}\min\{L_n^{(L)}(u),L_n^{(L)}(-u)\}. \tag*{(145)} R n , M KL ( P ) ≥ 2 1 ( r μ ( t / n ) + r μ ( − t / n ) ) E P ⊗ n min { L n ( L ) ( u ) , L n ( L ) ( − u )} . (145)
By (H.4) and the continuous mapping theorem, for Y ∼ N ( 0 , I d ) Y\sim\mathcal{N}(0,I_d) Y ∼ N ( 0 , I d ) ,
( L n ( L ) ( u ) , L n ( L ) ( − u ) ) → d ( e u ⊤ Y − ∥ u ∥ 2 2 / 2 , e − u ⊤ Y − ∥ u ∥ 2 2 / 2 ) , \left(L_n^{(L)}(u),L_n^{(L)}(-u)\right)\xrightarrow{d}\left(e^{u^\top Y-\|u\|_2^2/2},e^{-u^\top Y-\|u\|_2^2/2}\right), ( L n ( L ) ( u ) , L n ( L ) ( − u ) ) d ( e u ⊤ Y − ∥ u ∥ 2 2 /2 , e − u ⊤ Y − ∥ u ∥ 2 2 /2 ) ,
which, again by the continuous mapping theorem, yields
min { L n ( L ) ( u ) , L n ( L ) ( − u ) } → d min { e u ⊤ Y − ∥ u ∥ 2 2 / 2 , e − u ⊤ Y − ∥ u ∥ 2 2 / 2 } = e − ∣ u ⊤ Y ∣ − ∥ u ∥ 2 2 / 2 . \min\left\{L_n^{(L)}(u),L_n^{(L)}(-u)\right\}\xrightarrow{d}\min\left\{e^{u^\top Y-\|u\|_2^2/2},e^{-u^\top Y-\|u\|_2^2/2}\right\}=e^{-|u^\top Y|-\|u\|_2^2/2}. min { L n ( L ) ( u ) , L n ( L ) ( − u ) } d min { e u ⊤ Y − ∥ u ∥ 2 2 /2 , e − u ⊤ Y − ∥ u ∥ 2 2 /2 } = e − ∣ u ⊤ Y ∣ − ∥ u ∥ 2 2 /2 .
Note that min { L n ( L ) ( u ) , L n ( L ) ( − u ) } \min\{L_n^{(L)}(u),L_n^{(L)}(-u)\} min { L n ( L ) ( u ) , L n ( L ) ( − u )} is uniformly integrable, since
sup n l a r g e E P ⊗ n min { L n ( L ) ( u ) , L n ( L ) ( − u ) } 2 ≤ sup n l a r g e E P ⊗ n ∥ L n ( L ) ∥ ∞ 2 < ∞ . \sup_{n\ \mathrm{large}}\mathbb{E}_{P^{\otimes n}}\min\{L_n^{(L)}(u),L_n^{(L)}(-u)\}^2
\leq
\sup_{n\ \mathrm{large}}\mathbb{E}_{P^{\otimes n}}\|L_n^{(L)}\|_\infty^2<\infty. n large sup E P ⊗ n min { L n ( L ) ( u ) , L n ( L ) ( − u ) } 2 ≤ n large sup E P ⊗ n ∥ L n ( L ) ∥ ∞ 2 < ∞.
Hence
1 2 E P ⊗ n min { L n ( L ) ( u ) , L n ( L ) ( − u ) } → 1 2 E Y ∼ N ( 0 , I d ) e − ∣ u ⊤ Y ∣ − ∥ u ∥ 2 2 / 2 = 1 2 π ∫ 0 ∞ e − ( s + ∥ u ∥ 2 ) 2 / 2 d s = Φ ( − ∥ u ∥ 2 ) . (146) \begin{aligned}
\frac{1}{2}\mathbb{E}_{P^{\otimes n}}\min\{L_n^{(L)}(u),L_n^{(L)}(-u)\}
\to
\frac{1}{2}\mathbb{E}_{Y\sim\mathcal{N}(0,I_d)}e^{-|u^\top Y|-\|u\|_2^2/2} \\
&=
\frac{1}{\sqrt{2\pi}}\int_0^\infty e^{-(s+\|u\|_2)^2/2}\,ds
=
\Phi(-\|u\|_2). \tag*{(146)}
\end{aligned} 2 1 E P ⊗ n min { L n ( L ) ( u ) , L n ( L ) ( − u )} → 2 1 E Y ∼ N ( 0 , I d ) e − ∣ u ⊤ Y ∣ − ∥ u ∥ 2 2 /2 = 2 π 1 ∫ 0 ∞ e − ( s + ∥ u ∥ 2 ) 2 /2 d s = Φ ( − ∥ u ∥ 2 ) . (146)
By (F.24),
r μ ( t / n ) + r μ ( − t / n ) ≥ 2 β 1 + β ( κ ˉ ∥ μ ∥ 2 ) − 1 / β h ( 1 + β ) / β n − ( 1 + β ) / ( 2 β ) . r_\mu(t/\sqrt{n})+r_\mu(-t/\sqrt{n})
\geq
2\frac{\beta}{1+\beta}(\bar{\kappa}\|\mu\|_2)^{-1/\beta}h^{(1+\beta)/\beta}n^{-(1+\beta)/(2\beta)}. r μ ( t / n ) + r μ ( − t / n ) ≥ 2 1 + β β ( κ ˉ ∥ μ ∥ 2 ) − 1/ β h ( 1 + β ) / β n − ( 1 + β ) / ( 2 β ) .
Substituting this into (F.27) and multiplying by n ( 1 + β ) / ( 2 β ) n^{(1+\beta)/(2\beta)} n ( 1 + β ) / ( 2 β ) gives, for each sufficiently large fixed L L L and all sufficiently large n n n ,
n ( 1 + β ) / ( 2 β ) R n , M K L ( P ) ≥ 2 β 1 + β ( κ ˉ ∥ μ ∥ 2 ) − 1 / β h ( 1 + β ) / β [ 1 2 E P ⊗ n min { L n ( L ) ( u ) , L n ( L ) ( − u ) } ] , n^{(1+\beta)/(2\beta)}\mathcal{R}_{n,M}^{\mathrm{KL}}(P)
\geq
2\frac{\beta}{1+\beta}(\bar{\kappa}\|\mu\|_2)^{-1/\beta}h^{(1+\beta)/\beta}
\left[\frac{1}{2}\mathbb{E}_{P^{\otimes n}}\min\{L_n^{(L)}(u),L_n^{(L)}(-u)\}\right], n ( 1 + β ) / ( 2 β ) R n , M KL ( P ) ≥ 2 1 + β β ( κ ˉ ∥ μ ∥ 2 ) − 1/ β h ( 1 + β ) / β [ 2 1 E P ⊗ n min { L n ( L ) ( u ) , L n ( L ) ( − u )} ] ,
where u = J L − 1 t u=J_L^{-1}t u = J L − 1 t . For this fixed L L L , (F.28) therefore yields
lim inf n → ∞ n ( 1 + β ) / ( 2 β ) R n , M K L ( P ) ≥ 2 Φ ( − ∥ J L − 1 t ∥ 2 ) β 1 + β ( κ ˉ ∥ μ ∥ 2 ) − 1 / β h ( 1 + β ) / β . (147) \liminf_{n\to\infty}n^{(1+\beta)/(2\beta)}\mathcal{R}_{n,M}^{\mathrm{KL}}(P)
\geq
2\Phi(-\|J_L^{-1}t\|_2)\frac{\beta}{1+\beta}(\bar{\kappa}\|\mu\|_2)^{-1/\beta}h^{(1+\beta)/\beta}. \tag*{(147)} n → ∞ lim inf n ( 1 + β ) / ( 2 β ) R n , M KL ( P ) ≥ 2Φ ( − ∥ J L − 1 t ∥ 2 ) 1 + β β ( κ ˉ ∥ μ ∥ 2 ) − 1/ β h ( 1 + β ) / β . (147)
This inequality holds for every sufficiently large fixed L L L , while its left side is independent of L L L . Since J L → Σ 1 / 2 J_L\to\Sigma^{1/2} J L → Σ 1/2 and Σ ≻ 0 \Sigma\succ0 Σ ≻ 0 , continuity of matrix inversion gives J L − 1 → Σ − 1 / 2 J_L^{-1}\to\Sigma^{-1/2} J L − 1 → Σ − 1/2 in operator norm as L → ∞ L\to\infty L → ∞ . Consequently,
∣ ∥ J L − 1 t ∥ 2 − ∥ Σ − 1 / 2 t ∥ 2 ∣ ≤ ∥ ( J L − 1 − Σ − 1 / 2 ) t ∥ 2 ≤ ∥ J L − 1 − Σ − 1 / 2 ∥ o p ∥ t ∥ 2 → L → ∞ 0 , \left|\|J_L^{-1}t\|_2-\|\Sigma^{-1/2}t\|_2\right|
\leq
\|(J_L^{-1}-\Sigma^{-1/2})t\|_2
\leq
\|J_L^{-1}-\Sigma^{-1/2}\|_{\mathrm{op}}\|t\|_2
\xrightarrow[L\to\infty]{}0, ∥ J L − 1 t ∥ 2 − ∥ Σ − 1/2 t ∥ 2 ≤ ∥ ( J L − 1 − Σ − 1/2 ) t ∥ 2 ≤ ∥ J L − 1 − Σ − 1/2 ∥ op ∥ t ∥ 2 L → ∞ 0 ,
where ∥ Σ − 1 / 2 t ∥ 2 = h ∥ Σ − 1 / 2 U μ v ∥ 2 \lVert\Sigma^{-1/2}t\rVert_2 = h\lVert\Sigma^{-1/2}U_\mu v\rVert_2 ∥ Σ − 1/2 t ∥ 2 = h ∥ Σ − 1/2 U μ v ∥ 2 . Letting L → ∞ L \to\infty L → ∞ on the right side of (F.29), and using continuity of Φ \Phi Φ gives
lim inf n → ∞ n ( 1 + β ) / ( 2 β ) R n , M K L ( P ) ≥ 2 Φ ( − h ∥ Σ − 1 / 2 U μ v ∥ 2 ) β 1 + β ( κ ˉ ∥ μ ∥ 2 ) − 1 / β h ( 1 + β ) / β \liminf_{n\to\infty} n^{(1+\beta)/(2\beta)}R_{n,M}^{\mathrm{KL}}(P) \ge2\Phi\left(-h\lVert\Sigma^{-1/2}U_\mu v\rVert_2\right)\frac{\beta}{1+\beta}(\bar{\kappa}\lVert\mu\rVert_2)^{-1/\beta}h^{(1+\beta)/\beta} n → ∞ lim inf n ( 1 + β ) / ( 2 β ) R n , M KL ( P ) ≥ 2Φ ( − h ∥ Σ − 1/2 U μ v ∥ 2 ) 1 + β β ( κ ˉ ∥ μ ∥ 2 ) − 1/ β h ( 1 + β ) / β
This proves (F.21), with n → ∞ n \to\infty n → ∞ taken first for each fixed L L L , followed by L → ∞ L \to\infty L → ∞ .
For the upper bound, uniqueness and compactness imply that every maximizer at μ + h \mu+h μ + h converges to w ∗ ( μ ) w_*(\mu) w ∗ ( μ ) as h → 0 h \to0 h → 0 . Otherwise, a sequence of maximizers outside a fixed neighborhood would have a cluster point w ≠ w ∗ ( μ ) w \ne w_*(\mu) w = w ∗ ( μ ) satisfying μ ⊤ w = f ∗ ( μ ) \mu^\top w=f_*(\mu) μ ⊤ w = f ∗ ( μ ) . Decrease ρ \rho ρ so that κ ˉ ρ β / ( 1 + β ) ≤ 1 \bar{\kappa}\rho^\beta/(1+\beta)\le1 κ ˉ ρ β / ( 1 + β ) ≤ 1 . The upper growth bound in (F.22) then gives
0 ≤ ϱ μ ( z ) ≤ κ ˉ 1 + β ∥ z ∥ 2 1 + β ≤ κ ˉ 1 + β ρ 1 + β ≤ ρ , ∥ z ∥ 2 ≤ ρ . 0\le\varrho_\mu(z)\le\frac{\bar{\kappa}}{1+\beta}\lVert z\rVert_2^{1+\beta}\le\frac{\bar{\kappa}}{1+\beta}\rho^{1+\beta}\le\rho,\qquad\lVert z\rVert_2\le\rho. 0 ≤ ϱ μ ( z ) ≤ 1 + β κ ˉ ∥ z ∥ 2 1 + β ≤ 1 + β κ ˉ ρ 1 + β ≤ ρ , ∥ z ∥ 2 ≤ ρ .
For this smaller fixed ρ \rho ρ , all maximizers still belong to the corresponding coordinate neighborhood for sufficiently small h h h . The local feasible points have the form
w = w ∗ ( μ ) + U μ z − n μ b , ∥ z ∥ 2 ≤ ρ , ϱ μ ( z ) ≤ b ≤ ρ . w=w_*(\mu)+U_\mu z-n_\mu b,\qquad\lVert z\rVert_2\le\rho,\qquad\varrho_\mu(z)\le b\le\rho. w = w ∗ ( μ ) + U μ z − n μ b , ∥ z ∥ 2 ≤ ρ , ϱ μ ( z ) ≤ b ≤ ρ .
Here b b b is the inward normal displacement. Since μ ⊤ U μ = 0 ⊤ \mu^\top U_\mu=0^\top μ ⊤ U μ = 0 ⊤ and μ ⊤ n μ = ∥ μ ∥ 2 \mu^\top n_\mu=\lVert\mu\rVert_2 μ ⊤ n μ = ∥ μ ∥ 2 ,
( μ + h ) ⊤ ( w − w ∗ ( μ ) ) = ( μ + h ) ⊤ ( U μ z − n μ b ) = ( U μ ⊤ h ) ⊤ z − ( ∥ μ ∥ 2 + n μ ⊤ h ) b . \begin{aligned}
(\mu+h)^\top(w-w_*(\mu))&=(\mu+h)^\top(U_\mu z-n_\mu b)\\
&=(U_\mu^\top h)^\top z-(\lVert\mu\rVert_2+n_\mu^\top h)b.
\end{aligned} ( μ + h ) ⊤ ( w − w ∗ ( μ )) = ( μ + h ) ⊤ ( U μ z − n μ b ) = ( U μ ⊤ h ) ⊤ z − (∥ μ ∥ 2 + n μ ⊤ h ) b .
For ∥ h ∥ 2 < ∥ μ ∥ 2 / 2 \lVert h\rVert_2<\lVert\mu\rVert_2/2 ∥ h ∥ 2 < ∥ μ ∥ 2 /2 ,
∥ μ ∥ 2 + n μ ⊤ h ≥ ∥ μ ∥ 2 − ∥ h ∥ 2 > 1 2 ∥ μ ∥ 2 > 0. \lVert\mu\rVert_2+n_\mu^\top h\ge\lVert\mu\rVert_2-\lVert h\rVert_2>\frac{1}{2}\lVert\mu\rVert_2>0. ∥ μ ∥ 2 + n μ ⊤ h ≥ ∥ μ ∥ 2 − ∥ h ∥ 2 > 2 1 ∥ μ ∥ 2 > 0.
For each fixed z z z , the objective ( μ + h ) ⊤ ( w − w ∗ ( μ ) ) (\mu+h)^\top(w-w_*(\mu)) ( μ + h ) ⊤ ( w − w ∗ ( μ )) is therefore decreasing in b b b and is maximized at b = ϱ μ ( z ) b=\varrho_\mu(z) b = ϱ μ ( z ) . By definition of r μ r_\mu r μ , for sufficiently small h h h ,
r μ ( h ) = sup w ∈ Ω ( μ + h ) ⊤ ( w − w ∗ ( μ ) ) = sup ∥ z ∥ 2 ≤ ρ ϱ μ ( z ) ≤ b ≤ ρ { ( U μ ⊤ h ) ⊤ z − ( ∥ μ ∥ 2 + n μ ⊤ h ) b } = sup ∥ z ∥ 2 ≤ ρ { ( U μ ⊤ h ) ⊤ z − ( ∥ μ ∥ 2 + n μ ⊤ h ) ϱ μ ( z ) } ≤ sup ∥ z ∥ 2 ≤ ρ { ( U μ ⊤ h ) ⊤ z − κ ( ∥ μ ∥ 2 + n μ ⊤ h ) 1 + β ∥ z ∥ 2 1 + β } ≤ sup z ∈ R d − 1 { ( U μ ⊤ h ) ⊤ z − κ ( ∥ μ ∥ 2 + n μ ⊤ h ) 1 + β ∥ z ∥ 2 1 + β } = sup s ≥ 0 { ∥ U μ ⊤ h ∥ 2 s − κ ( ∥ μ ∥ 2 + n μ ⊤ h ) 1 + β s 1 + β } . \begin{aligned}
r_\mu(h)&=\sup_{w\in\Omega}(\mu+h)^\top(w-w_*(\mu))\\
&=\sup_{\substack{\lVert z\rVert_2\le\rho\\ \varrho_\mu(z)\le b\le\rho}}\left\{(U_\mu^\top h)^\top z-(\lVert\mu\rVert_2+n_\mu^\top h)b\right\}\\
&=\sup_{\lVert z\rVert_2\le\rho}\left\{(U_\mu^\top h)^\top z-(\lVert\mu\rVert_2+n_\mu^\top h)\varrho_\mu(z)\right\}\\
&\le\sup_{\lVert z\rVert_2\le\rho}\left\{(U_\mu^\top h)^\top z-\frac{\kappa(\lVert\mu\rVert_2+n_\mu^\top h)}{1+\beta}\lVert z\rVert_2^{1+\beta}\right\}\\
&\le\sup_{z\in\mathbb{R}^{d-1}}\left\{(U_\mu^\top h)^\top z-\frac{\kappa(\lVert\mu\rVert_2+n_\mu^\top h)}{1+\beta}\lVert z\rVert_2^{1+\beta}\right\}\\
&=\sup_{s\ge0}\left\{\lVert U_\mu^\top h\rVert_2s-\frac{\kappa(\lVert\mu\rVert_2+n_\mu^\top h)}{1+\beta}s^{1+\beta}\right\}.
\end{aligned} r μ ( h ) = w ∈ Ω sup ( μ + h ) ⊤ ( w − w ∗ ( μ )) = ∥ z ∥ 2 ≤ ρ ϱ μ ( z ) ≤ b ≤ ρ sup { ( U μ ⊤ h ) ⊤ z − (∥ μ ∥ 2 + n μ ⊤ h ) b } = ∥ z ∥ 2 ≤ ρ sup { ( U μ ⊤ h ) ⊤ z − (∥ μ ∥ 2 + n μ ⊤ h ) ϱ μ ( z ) } ≤ ∥ z ∥ 2 ≤ ρ sup { ( U μ ⊤ h ) ⊤ z − 1 + β κ (∥ μ ∥ 2 + n μ ⊤ h ) ∥ z ∥ 2 1 + β } ≤ z ∈ R d − 1 sup { ( U μ ⊤ h ) ⊤ z − 1 + β κ (∥ μ ∥ 2 + n μ ⊤ h ) ∥ z ∥ 2 1 + β } = s ≥ 0 sup { ∥ U μ ⊤ h ∥ 2 s − 1 + β κ (∥ μ ∥ 2 + n μ ⊤ h ) s 1 + β } .
where for nonzero U μ ⊤ h U_\mu^\top h U μ ⊤ h , the last term is maximized at
s = ( ∥ U μ ⊤ h ∥ 2 κ ( ∥ μ ∥ 2 + n μ ⊤ h ) ) 1 / β . s=\left(\frac{\lVert U_\mu^\top h\rVert_2}{\kappa(\lVert\mu\rVert_2+n_\mu^\top h)}\right)^{1/\beta}. s = ( κ (∥ μ ∥ 2 + n μ ⊤ h ) ∥ U μ ⊤ h ∥ 2 ) 1/ β .
At this value,
r μ ( h ) ≤ β 1 + β ∥ U μ ⊤ h ∥ 2 ( ∥ U μ ⊤ h ∥ 2 κ ( ∥ μ ∥ 2 + n μ ⊤ h ) ) 1 / β = β 1 + β [ κ ( ∥ μ ∥ 2 + n μ ⊤ h ) ] − 1 / β ∥ U μ ⊤ h ∥ 2 ( 1 + β ) / β . \begin{aligned}
r_\mu(h)\le\frac{\beta}{1+\beta}\lVert U_\mu^\top h\rVert_2\left(\frac{\lVert U_\mu^\top h\rVert_2}{\kappa(\lVert\mu\rVert_2+n_\mu^\top h)}\right)^{1/\beta} \\
&=\frac{\beta}{1+\beta}\left[\kappa(\lVert\mu\rVert_2+n_\mu^\top h)\right]^{-1/\beta}\lVert U_\mu^\top h\rVert_2^{(1+\beta)/\beta}.
\end{aligned} r μ ( h ) ≤ 1 + β β ∥ U μ ⊤ h ∥ 2 ( κ (∥ μ ∥ 2 + n μ ⊤ h ) ∥ U μ ⊤ h ∥ 2 ) 1/ β = 1 + β β [ κ (∥ μ ∥ 2 + n μ ⊤ h ) ] − 1/ β ∥ U μ ⊤ h ∥ 2 ( 1 + β ) / β .
If U μ ⊤ h = 0 U_\mu^\top h=0 U μ ⊤ h = 0 , the supremum is attained at s = 0 s=0 s = 0 with value zero, and the same bound holds. Consequently, for some r , C > 0 r,C>0 r , C > 0 , r μ ( h ) ≤ C ∥ h ∥ 2 ( 1 + β ) / β r_\mu(h)\le C\lVert h\rVert_2^{(1+\beta)/\beta} r μ ( h ) ≤ C ∥ h ∥ 2 ( 1 + β ) / β on B 2 ( r ) B_2(r) B 2 ( r ) . Outside this ball, the global Lipschitz bound gives
r μ ( h ) ≤ 2 R Ω ∥ h ∥ 2 ≤ 2 R Ω r − 1 / β ∥ h ∥ 2 ( 1 + β ) / β . r_\mu(h)\le2R_\Omega\lVert h\rVert_2\le2R_\Omega r^{-1/\beta}\lVert h\rVert_2^{(1+\beta)/\beta}. r μ ( h ) ≤ 2 R Ω ∥ h ∥ 2 ≤ 2 R Ω r − 1/ β ∥ h ∥ 2 ( 1 + β ) / β .
After enlarging C C C we therefore have
0 ≤ r μ ( h ) ≤ C ∥ h ∥ 2 ( 1 + β ) / β , h ∈ R d . (148) 0\le r_\mu(h)\le C\lVert h\rVert_2^{(1+\beta)/\beta},\qquad h\in\mathbb{R}^d. \tag*{(148)} 0 ≤ r μ ( h ) ≤ C ∥ h ∥ 2 ( 1 + β ) / β , h ∈ R d . (148)
Write t Q = n ( μ Q − μ ) t_Q=\sqrt{n}(\mu_Q-\mu) t Q = n ( μ Q − μ ) and retain Z n , Q Z_{n,Q} Z n , Q from (47) . The deterministic subgradient argument in (C.6) applies with the selected maximizer at μ ~ n = μ + ( t Q + Z n , Q ) / n \widetilde{\mu}_n=\mu+(t_Q+Z_{n,Q})/\sqrt{n} μ n = μ + ( t Q + Z n , Q ) / n . Together with (F.30), it gives
R Q ( w ∗ ( μ ~ n ) ) ≤ r μ ( t Q / n ) + r μ ( ( t Q + 2 Z n , Q ) / n ) ≤ C n − ( 1 + β ) / ( 2 β ) { ∥ t Q ∥ 2 ( 1 + β ) / β + ∥ t Q + 2 Z n , Q ∥ 2 ( 1 + β ) / β } . \begin{aligned}
R_Q(w_*(\widetilde{\mu}_n))\le r_\mu(t_Q/\sqrt{n})+r_\mu((t_Q+2Z_{n,Q})/\sqrt{n}) \\
&\le Cn^{-(1+\beta)/(2\beta)}\left\{\lVert t_Q\rVert_2^{(1+\beta)/\beta}+\lVert t_Q+2Z_{n,Q}\rVert_2^{(1+\beta)/\beta}\right\}.
\end{aligned} R Q ( w ∗ ( μ n )) ≤ r μ ( t Q / n ) + r μ (( t Q + 2 Z n , Q ) / n ) ≤ C n − ( 1 + β ) / ( 2 β ) { ∥ t Q ∥ 2 ( 1 + β ) / β + ∥ t Q + 2 Z n , Q ∥ 2 ( 1 + β ) / β } .
The uniform mean bound (F.2) and moment bound (F.6) imply
sup Q ∈ Q n , M K L ( P ) E Q ⊗ n R Q ( w ∗ ( μ ~ n ) ) ≤ C P , M n − ( 1 + β ) / ( 2 β ) . \sup_{Q\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)}\mathbb{E}_{Q^{\otimes n}}R_Q(w_*(\widetilde{\mu}_n))\le C_{P,M}n^{-(1+\beta)/(2\beta)}. Q ∈ Q n , M KL ( P ) sup E Q ⊗ n R Q ( w ∗ ( μ n )) ≤ C P , M n − ( 1 + β ) / ( 2 β ) .
Together with (F.21) and the definition of R n , M K L ( P ) \mathcal{R}^{\mathrm{KL}}_{n,M}(P) R n , M KL ( P ) , this proves (F.23). □
Remark F.3 (Boundary regularity and distance). The local C 1 , β C^{1,\beta} C 1 , β argument in the proof of Proposition 4.5, for 0 < β ≤ 1 0<\beta\le1 0 < β ≤ 1 , verifies (F.20) and hence the KL lower bound. The matching upper bound additionally uses the lower growth and local feasible-set description in (F.22). For the boundary graph in (F.22), the identities
n μ ⊤ ( w ∗ ( μ ) − w ) = ϱ μ ( z ) , U μ ⊤ ( w − w ∗ ( μ ) ) = z , w = w ∗ ( μ ) + U μ z − n μ ϱ μ ( z ) , n_\mu^\top(w_*(\mu)-w)=\varrho_\mu(z),\qquad U_\mu^\top(w-w_*(\mu))=z,\qquad w=w_*(\mu)+U_\mu z-n_\mu\varrho_\mu(z), n μ ⊤ ( w ∗ ( μ ) − w ) = ϱ μ ( z ) , U μ ⊤ ( w − w ∗ ( μ )) = z , w = w ∗ ( μ ) + U μ z − n μ ϱ μ ( z ) ,
show that the local support-distance condition (D.17) is equivalent to
ϱ μ ( z ) ≥ κ 1 + β ∥ z ∥ 2 1 + β . \varrho_\mu(z)\ge\frac{\kappa}{1+\beta}\lVert z\rVert_2^{1+\beta}. ϱ μ ( z ) ≥ 1 + β κ ∥ z ∥ 2 1 + β .
on a sufficiently small coordinate ball. Thus, for 0 < β ≤ 1 0 < \beta\le1 0 < β ≤ 1 , a local C 1 , β C^{1,\beta} C 1 , β boundary together with (D.17) implies (F.22) and the matching KL rates in (F.23), under the moment assumptions of Theorem F.2 . In particular, the strong-convexity and positive-principal-curvature conditions verified in Lemma D.3 give the corresponding quadratic growth conclusions.
If dist ( μ , A ) > 0 \operatorname{dist}(\mu,A)>0 dist ( μ , A ) > 0 , homogeneity (D.1) rewrites the geometric factor in (F.21) as
( κ ˉ ∥ μ ∥ 2 ) − 1 / β = ( dist ( n μ , A ) κ ˉ dist ( μ , A ) ) 1 / β . (\bar{\kappa}\lVert\mu\rVert_2)^{-1/\beta}=\left(\frac{\operatorname{dist}(n_\mu,A)}{\bar{\kappa}\operatorname{dist}(\mu,A)}\right)^{1/\beta}. ( κ ˉ ∥ μ ∥ 2 ) − 1/ β = ( κ ˉ dist ( μ , A ) dist ( n μ , A ) ) 1/ β .
Remark F.4 (First-order rates and the role of exponential perturbations). For every compact convex Ω \Omega Ω , Corollary 4.1 and Theorem 5.2 give
lim n → ∞ n R n , M K L ( P ) = V M , I d ( ψ 1 ∘ Σ 1 / 2 ) . \lim_{n\to\infty}\sqrt{n}\,R_{n,M}^{\mathrm{KL}}(P)=V_{M,\mathrm{Id}}\left(\psi_1\circ\Sigma^{1/2}\right). n → ∞ lim n R n , M KL ( P ) = V M , Id ( ψ 1 ∘ Σ 1/2 ) .
Because Σ ≻ 0 \Sigma\succ0 Σ ≻ 0 , this constant is positive for every M > 0 M>0 M > 0 if W ∗ ( μ ) \mathcal{W}_*(\mu) W ∗ ( μ ) contains more than one point; it is zero if W ∗ ( μ ) \mathcal{W}_*(\mu) W ∗ ( μ ) is a singleton. The geometric expansions in the preceding examples therefore give their respective nonzero rates on KL neighborhoods as well, with Gaussian constant V M , I d ( ψ ∘ Σ 1 / 2 ) V_{M,\mathrm{Id}}(\psi\circ\Sigma^{1/2}) V M , Id ( ψ ∘ Σ 1/2 ) and the truncated plug-in policy attaining the rate.
The neighborhood (46) allows all dominated Q Q Q satisfying its entropy bound, rather than only the exponential family (H.10). Its mean perturbations satisfy
n ( μ Q − μ ) ∈ Σ 1 / 2 B 2 ( M + o ( 1 ) ) , \sqrt{n}(\mu_Q-\mu)\in\Sigma^{1/2}B_2(M+o(1)), n ( μ Q − μ ) ∈ Σ 1/2 B 2 ( M + o ( 1 )) ,
so its fixed-M M M constant is generally different from V M , Σ ( ψ ) V_{M,\Sigma}(\psi) V M , Σ ( ψ ) for the Gaussian experiment on a Euclidean mean ball in § 3 . The bounded-score tilts in Lemma H.1 supply the lower-bound submodel in the proof of (49) ; the upper bound additionally controls distributions outside that submodel.
Proofs for Ordinary and Truncated SAA
Proof of Theorem 6.1
Proof. For this proof, write
ℓ ψ ( t , z ) : = ψ ( t ) − ψ ( t + z ) + ∇ ψ ( t + z ) T z \ell_\psi(t,z):=\psi(t)-\psi(t+z)+\nabla\psi(t+z)^\mathsf{T}z ℓ ψ ( t , z ) := ψ ( t ) − ψ ( t + z ) + ∇ ψ ( t + z ) T z
where ψ \psi ψ is differentiable at t + z t+z t + z , and assign value zero on the remaining set. By Rademacher’s theorem, the finite convex function ψ \psi ψ on R d \mathbb{R}^d R d is locally Lipschitz and differentiable Lebesgue almost everywhere. Since Σ ≻ 0 \Sigma\succ0 Σ ≻ 0 , the assigned values do not affect E ℓ ψ ( t , Z ) \mathbb{E}\ell_\psi(t,Z) E ℓ ψ ( t , Z ) for any fixed t t t .
For u ∈ R d u\in\mathbb{R}^d u ∈ R d , the selected maximizer gives the subgradient
w ∗ ( μ + u / n ) − w ∗ ( μ ) n a ( n − 1 / 2 ) ∈ ∂ ( r μ ( ⋅ / n ) a ( n − 1 / 2 ) ) ( u ) . \frac{w_*(\mu+u/\sqrt{n})-w_*(\mu)}{\sqrt{n}a(n^{-1/2})}\in\partial\left(\frac{r_\mu(\cdot/\sqrt{n})}{a(n^{-1/2})}\right)(u). n a ( n − 1/2 ) w ∗ ( μ + u / n ) − w ∗ ( μ ) ∈ ∂ ( a ( n − 1/2 ) r μ ( ⋅ / n ) ) ( u ) .
Suppose u n → u u_n \to u u n → u and ψ \psi ψ is differentiable at u u u . For every unit vector v v v and every h > 0 h>0 h > 0 , the subgradient inequalities give
r μ ( u n / n ) − r μ ( ( u n − h v ) / n ) h a ( n − 1 / 2 ) ≤ v ⊤ w ∗ ( μ + u n / n ) − w ∗ ( μ ) n a ( n − 1 / 2 ) ≤ r μ ( ( u n + h v ) / n ) − r μ ( u n / n ) h a ( n − 1 / 2 ) . \frac{r_\mu(u_n/\sqrt{n})-r_\mu((u_n-hv)/\sqrt{n})}{h\,a(n^{-1/2})}
\leq v^\top\frac{w_*(\mu+u_n/\sqrt{n})-w_*(\mu)}{\sqrt{n}\,a(n^{-1/2})}
\leq\frac{r_\mu((u_n+hv)/\sqrt{n})-r_\mu(u_n/\sqrt{n})}{h\,a(n^{-1/2})}. h a ( n − 1/2 ) r μ ( u n / n ) − r μ (( u n − h v ) / n ) ≤ v ⊤ n a ( n − 1/2 ) w ∗ ( μ + u n / n ) − w ∗ ( μ ) ≤ h a ( n − 1/2 ) r μ (( u n + h v ) / n ) − r μ ( u n / n ) .
For fixed h > 0 h>0 h > 0 , u n u_n u n , u n − h v u_n-hv u n − h v and u n + h v u_n+hv u n + h v belong to the compact ball B 2 ( ∥ u ∥ 2 + h + 1 ) B_2(\lVert u\rVert_2+h+1) B 2 (∥ u ∥ 2 + h + 1 ) for sufficiently large n n n . (C.1) and continuity of ψ \psi ψ therefore imply
ψ ( u ) − ψ ( u − h v ) h ≤ lim inf n → ∞ v ⊤ w ∗ ( μ + u n / n ) − w ∗ ( μ ) n a ( n − 1 / 2 ) ≤ lim sup n → ∞ v ⊤ w ∗ ( μ + u n / n ) − w ∗ ( μ ) n a ( n − 1 / 2 ) ≤ ψ ( u + h v ) − ψ ( u ) h . \frac{\psi(u)-\psi(u-hv)}{h}
\leq\liminf_{n\to\infty}v^\top\frac{w_*(\mu+u_n/\sqrt{n})-w_*(\mu)}{\sqrt{n}\,a(n^{-1/2})}
\leq\limsup_{n\to\infty}v^\top\frac{w_*(\mu+u_n/\sqrt{n})-w_*(\mu)}{\sqrt{n}\,a(n^{-1/2})}
\leq\frac{\psi(u+hv)-\psi(u)}{h}. h ψ ( u ) − ψ ( u − h v ) ≤ n → ∞ lim inf v ⊤ n a ( n − 1/2 ) w ∗ ( μ + u n / n ) − w ∗ ( μ ) ≤ n → ∞ lim sup v ⊤ n a ( n − 1/2 ) w ∗ ( μ + u n / n ) − w ∗ ( μ ) ≤ h ψ ( u + h v ) − ψ ( u ) .
Letting h ↓ 0 h\downarrow0 h ↓ 0 , the two endpoints converge to v ⊤ ∇ ψ ( u ) v^\top\nabla\psi(u) v ⊤ ∇ ψ ( u ) . Taking v v v to be each coordinate vector proves
w ∗ ( μ + u n / n ) − w ∗ ( μ ) n a ( n − 1 / 2 ) → ∇ ψ ( u ) . (149) \frac{w_*(\mu+u_n/\sqrt{n})-w_*(\mu)}{\sqrt{n}\,a(n^{-1/2})}\to\nabla\psi(u). \tag*{(149)} n a ( n − 1/2 ) w ∗ ( μ + u n / n ) − w ∗ ( μ ) → ∇ ψ ( u ) . (149)
Now let Q n ∈ Q n , M K L ( P ) Q_n\in Q^{\mathrm{KL}}_{n,M}(P) Q n ∈ Q n , M KL ( P ) , suppose t Q n → t t_{Q_n}\to t t Q n → t , and let μ ^ n \hat{\mu}_n μ ^ n be any sample-based statistic such that
n ( μ ^ n − μ Q n ) → d Z ∼ N ( 0 , Σ ) . \sqrt{n}(\hat{\mu}_n-\mu_{Q_n})\xrightarrow{d} Z\sim\mathcal{N}(0,\Sigma). n ( μ ^ n − μ Q n ) d Z ∼ N ( 0 , Σ ) .
With z = n ( μ ^ n − μ Q n ) z=\sqrt{n}(\hat{\mu}_n-\mu_{Q_n}) z = n ( μ ^ n − μ Q n ) , the exact support-function identity is
R Q n ( w ∗ ( μ ^ n ) ) a ( n − 1 / 2 ) = r μ ( t Q n / n ) − r μ ( ( t Q n + z ) / n ) a ( n − 1 / 2 ) + ( w ∗ ( μ + ( t Q n + z ) / n ) − w ∗ ( μ ) ) ⊤ z n a ( n − 1 / 2 ) . \frac{R_{Q_n}(w_*(\hat{\mu}_n))}{a(n^{-1/2})}
=
\frac{r_\mu(t_{Q_n}/\sqrt{n})-r_\mu((t_{Q_n}+z)/\sqrt{n})}{a(n^{-1/2})}
+
\frac{(w_*(\mu+(t_{Q_n}+z)/\sqrt{n})-w_*(\mu))^\top z}{\sqrt{n}\,a(n^{-1/2})}. a ( n − 1/2 ) R Q n ( w ∗ ( μ ^ n )) = a ( n − 1/2 ) r μ ( t Q n / n ) − r μ (( t Q n + z ) / n ) + n a ( n − 1/2 ) ( w ∗ ( μ + ( t Q n + z ) / n ) − w ∗ ( μ ) ) ⊤ z .
On a Skorokhod representation, the centered statistic z = n ( μ ^ n − μ Q n ) z=\sqrt{n}(\hat{\mu}_n-\mu_{Q_n}) z = n ( μ ^ n − μ Q n ) converges almost surely to Z Z Z . (C.1) and (G.1) apply almost surely, because ψ \psi ψ is differentiable at t + Z t+Z t + Z almost surely. Thus
R Q n ( w ∗ ( μ ^ n ) ) a ( n − 1 / 2 ) → d ℓ ψ ( t , Z ) . (150) \frac{R_{Q_n}(w_*(\hat{\mu}_n))}{a(n^{-1/2})}\xrightarrow{d}\ell_\psi(t,Z). \tag*{(150)} a ( n − 1/2 ) R Q n ( w ∗ ( μ ^ n )) d ℓ ψ ( t , Z ) . (150)
The same argument applies to any subsequence along which t Q n t_{Q_n} t Q n converges. Choose L L L with t Q ∈ B 2 ( L ) t_Q\in B_2(L) t Q ∈ B 2 ( L ) for all sufficiently large n n n and sup B 2 ( L ) ψ > 0 \sup_{B_2(L)}\psi>0 sup B 2 ( L ) ψ > 0 . The deterministic subgradient argument leading to (C.6) applies to any vectors t , z t,z t , z : expand the regret using the selected maximizer at μ + ( t + z ) / n \mu+(t+z)/\sqrt{n} μ + ( t + z ) / n , and compare its support value at μ + ( t + 2 z ) / n \mu+(t+2z)/\sqrt{n} μ + ( t + 2 z ) / n . It gives
f ∗ ( μ + t / n ) − w ∗ ( μ + ( t + z ) / n ) ⊤ ( μ + t / n ) ≤ r μ ( t / n ) + r μ ( ( t + 2 z ) / n ) − 2 r μ ( ( t + z ) / n ) ≤ b L ( n − 1 / 2 ) + b L ( n − 1 / 2 ( 1 + 2 ∥ z ∥ 2 L ) ) , t ∈ B 2 ( L ) . f_*(\mu+t/\sqrt{n})-w_*(\mu+(t+z)/\sqrt{n})^\top(\mu+t/\sqrt{n})
\leq r_\mu(t/\sqrt{n})+r_\mu((t+2z)/\sqrt{n})-2r_\mu((t+z)/\sqrt{n})
\leq b_L(n^{-1/2})+b_L\left(n^{-1/2}\left(1+\frac{2\lVert z\rVert_2}{L}\right)\right),\qquad t\in B_2(L). f ∗ ( μ + t / n ) − w ∗ ( μ + ( t + z ) / n ) ⊤ ( μ + t / n ) ≤ r μ ( t / n ) + r μ (( t + 2 z ) / n ) − 2 r μ (( t + z ) / n ) ≤ b L ( n − 1/2 ) + b L ( n − 1/2 ( 1 + L 2 ∥ z ∥ 2 ) ) , t ∈ B 2 ( L ) .
Since b L ( n − 1 / 2 ) / a ( n − 1 / 2 ) b_L(n^{-1/2})/a(n^{-1/2}) b L ( n − 1/2 ) / a ( n − 1/2 ) converges to sup B 2 ( L ) ψ ∈ ( 0 , ∞ ) \sup_{B_2(L)}\psi\in(0,\infty) sup B 2 ( L ) ψ ∈ ( 0 , ∞ ) , (C.5) gives a constant C C C such that, for t ∈ B 2 ( L ) t \in B_2(L) t ∈ B 2 ( L ) and z ∈ R d z \in\mathbb{R}^d z ∈ R d , for sufficiently large n n n ,
0 ≤ f ∗ ( μ + t / n ) − w ∗ ( μ + ( t + z ) / n ) T ( μ + t / n ) a ( n − 1 / 2 ) ≤ b L ( n − 1 / 2 ) + b L ( n − 1 / 2 ( 1 + 2 ∥ z ∥ 2 / L ) ) c b a ( n − 1 / 2 ) ≤ b L ( n − 1 / 2 ) a ( n − 1 / 2 ) [ 1 + C b ( 1 + 2 ∥ z ∥ 2 L ) c b ] ≤ C ( 1 + ∥ z ∥ 2 ) c b . (151) \begin{aligned}
0 &\le\frac{f_*(\mu+t/\sqrt{n})-w_*(\mu+(t+z)/\sqrt{n})^\mathsf{T}(\mu+t/\sqrt{n})}{a(n^{-1/2})}\\
&\le\frac{b_L(n^{-1/2})+b_L\left(n^{-1/2}(1+2\lVert z\rVert_2/L)\right)^{c_b}}{a(n^{-1/2})}\\
&\le\frac{b_L(n^{-1/2})}{a(n^{-1/2})}\left[1+C_b\left(1+\frac{2\lVert z\rVert_2}{L}\right)^{c_b}\right]\\
&\le C(1+\lVert z\rVert_2)^{c_b}. \tag*{(151)}
\end{aligned} 0 ≤ a ( n − 1/2 ) f ∗ ( μ + t / n ) − w ∗ ( μ + ( t + z ) / n ) T ( μ + t / n ) ≤ a ( n − 1/2 ) b L ( n − 1/2 ) + b L ( n − 1/2 ( 1 + 2 ∥ z ∥ 2 / L ) ) c b ≤ a ( n − 1/2 ) b L ( n − 1/2 ) [ 1 + C b ( 1 + L 2 ∥ z ∥ 2 ) c b ] ≤ C ( 1 + ∥ z ∥ 2 ) c b . (151)
Here C b , c b < ∞ C_b,c_b<\infty C b , c b < ∞ are defined in (C.5) for L L L . Note that
f ∗ ( μ + t / n ) − w ∗ ( μ + ( t + z ) / n ) T ( μ + t / n ) a ( n − 1 / 2 ) = r μ ( t / n ) − r μ ( ( t + z ) / n ) a ( n − 1 / 2 ) + [ w ∗ ( μ + ( t + z ) / n ) − w ∗ ( μ ) n a ( n − 1 / 2 ) ] T z → ℓ ψ ( t , z ) \begin{aligned}
\frac{f_*(\mu+t/\sqrt{n})-w_*(\mu+(t+z)/\sqrt{n})^\mathsf{T}(\mu+t/\sqrt{n})}{a(n^{-1/2})} \\
&= \frac{r_\mu(t/\sqrt{n})-r_\mu((t+z)/\sqrt{n})}{a(n^{-1/2})}+\left[\frac{w_*(\mu+(t+z)/\sqrt{n})-w_*(\mu)}{\sqrt{n}a(n^{-1/2})}\right]^\mathsf{T}z \\
&\to\ell_\psi(t,z)
\end{aligned} a ( n − 1/2 ) f ∗ ( μ + t / n ) − w ∗ ( μ + ( t + z ) / n ) T ( μ + t / n ) = a ( n − 1/2 ) r μ ( t / n ) − r μ (( t + z ) / n ) + [ n a ( n − 1/2 ) w ∗ ( μ + ( t + z ) / n ) − w ∗ ( μ ) ] T z → ℓ ψ ( t , z )
as n → ∞ n\to\infty n → ∞ . Passing to the limit in (G.3) gives ℓ ψ ( t , z ) ≤ C ( 1 + ∥ z ∥ 2 ) c b < ∞ \ell_\psi(t,z)\le C(1+\lVert z\rVert_2)^{c_b}<\infty ℓ ψ ( t , z ) ≤ C ( 1 + ∥ z ∥ 2 ) c b < ∞ wherever ψ \psi ψ is differentiable at t + z t+z t + z . Convexity of ψ \psi ψ also gives ℓ ψ ≥ 0 \ell_\psi\ge0 ℓ ψ ≥ 0 . Hence sup t ∈ B 2 ( L ) E ℓ ψ ( t , Z ) \sup_{t\in B_2(L)}\mathbb{E}\ell_\psi(t,Z) sup t ∈ B 2 ( L ) E ℓ ψ ( t , Z ) is finite.
Let t j → t t_j\to t t j → t and enlarge the fixed radius L L L to contain this sequence and its limit. For almost every z z z , ψ \psi ψ is differentiable at t + z t+z t + z and at every t j + z t_j+z t j + z . The same subgradient inequalities used in (G.1), now applied to the fixed function ψ \psi ψ , give ∇ ψ ( t j + z ) → ∇ ψ ( t + z ) \nabla\psi(t_j+z)\to\nabla\psi(t+z) ∇ ψ ( t j + z ) → ∇ ψ ( t + z ) . Hence ℓ ψ ( t j , Z ) → ℓ ψ ( t , Z ) \ell_\psi(t_j,Z)\to\ell_\psi(t,Z) ℓ ψ ( t j , Z ) → ℓ ψ ( t , Z ) almost surely. The envelope (G.3) on the larger ball is Gaussian-integrable, so dominated convergence yields
E ℓ ψ ( t j , Z ) → E ℓ ψ ( t , Z ) . (152) \mathbb{E}\ell_\psi(t_j,Z)\to\mathbb{E}\ell_\psi(t,Z). \tag*{(152)} E ℓ ψ ( t j , Z ) → E ℓ ψ ( t , Z ) . (152)
Thus the function t ↦ E ℓ ψ ( t , Z ) t\mapsto\mathbb{E}\ell_\psi(t,Z) t ↦ E ℓ ψ ( t , Z ) is continuous, and the supremum in (54) is finite, attained, and equal to the supremum over ∥ Σ − 1 / 2 t ∥ 2 < M \lVert\Sigma^{-1/2}t\rVert_2<M ∥ Σ − 1/2 t ∥ 2 < M .
For μ ^ n = μ ~ n \widehat{\mu}_n=\widetilde{\mu}_n μ n = μ n , Lemma F.1 gives n ( μ ~ n − μ Q n ) = Z n , Q n → d Z \sqrt{n}(\widetilde{\mu}_n-\mu_{Q_n})=Z_{n,Q_n}\xrightarrow{d}Z n ( μ n − μ Q n ) = Z n , Q n d Z , i.e., the required convergence in distribution. Its moment bounds and (G.3) give
sup n l a r g e sup Q E Q ⊗ n [ ( R Q ( w ∗ ( μ ~ n ) ) a ( n − 1 / 2 ) ) 2 ] ≤ C sup n l a r g e sup Q E Q ⊗ n ( 1 + ∥ Z n , Q ∥ 2 ) 2 c b < ∞ . \sup_{n\ \mathrm{large}}\sup_{Q}\mathbb{E}_{Q^{\otimes n}}\left[\left(\frac{R_Q(w_*(\widetilde{\mu}_n))}{a(n^{-1/2})}\right)^2\right]\le C\sup_{n\ \mathrm{large}}\sup_Q\mathbb{E}_{Q^{\otimes n}}(1+\lVert Z_{n,Q}\rVert_2)^{2c_b}<\infty. n large sup Q sup E Q ⊗ n [ ( a ( n − 1/2 ) R Q ( w ∗ ( μ n )) ) 2 ] ≤ C n large sup Q sup E Q ⊗ n ( 1 + ∥ Z n , Q ∥ 2 ) 2 c b < ∞.
Thus the normalized losses are uniformly integrable. More generally, the polynomial envelope requires only a uniformly bounded moment of order q > c b q>c_b q > c b to obtain a bounded ( 1 + ρ ) (1+\rho) ( 1 + ρ ) th loss moment for some ρ > 0 \rho>0 ρ > 0 . Lemma F.1 supplies every fixed q q q , so this step remains valid even when a ( n − 1 / 2 ) = o ( n − 1 ) a(n^{-1/2})=o(n^{-1}) a ( n − 1/2 ) = o ( n − 1 ) . Applying (G.2) along any sequence Q n ∈ Q n , M K L ( P ) Q_n\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P) Q n ∈ Q n , M KL ( P ) with t Q n → t t_{Q_n}\to t t Q n → t gives
E Q n ⊗ n R Q n ( w ∗ ( μ ~ n ) ) a ( n − 1 / 2 ) → E ℓ ψ ( t , Z ) . (153) \frac{\mathbb{E}_{Q_n^{\otimes n}}R_{Q_n}(w_*(\widetilde{\mu}_n))}{a(n^{-1/2})}\to\mathbb{E}\ell_\psi(t,Z). \tag*{(153)} a ( n − 1/2 ) E Q n ⊗ n R Q n ( w ∗ ( μ n )) → E ℓ ψ ( t , Z ) . (153)
Since all possible t Q t_Q t Q , for all sufficiently large n n n and all Q ∈ Q n , M K L ( P ) Q \in\mathcal{Q}_{n,M}^{\mathrm{KL}}(P) Q ∈ Q n , M KL ( P ) , lie in the same fixed compact ball B 2 ( L ) B_2(L) B 2 ( L ) , the subsequence argument below gives
sup Q ∣ E Q ⊗ n R Q ( w ∗ ( μ ~ n ) ) a ( n − 1 / 2 ) − E ℓ ψ ( t Q , Z ) ∣ → 0. (154) \sup_Q \left|\frac{\mathbb{E}_{Q^{\otimes n}}R_Q(w_*(\widetilde{\mu}_n))}{a(n^{-1/2})}-\mathbb{E}\ell_\psi(t_Q,Z)\right| \to0. \tag*{(154)} Q sup a ( n − 1/2 ) E Q ⊗ n R Q ( w ∗ ( μ n )) − E ℓ ψ ( t Q , Z ) → 0. (154)
Otherwise, if the supremum did not converge to zero, there would be a subsequence and distributions Q n Q_n Q n along which the difference ∣ E Q n ⊗ n R Q n ( w ∗ ( μ ~ n ) ) / a ( n − 1 / 2 ) − E ℓ ψ ( t Q n , Z ) ∣ \left|\mathbb{E}_{Q_n^{\otimes n}}R_{Q_n}(w_*(\widetilde{\mu}_n))/a(n^{-1/2})-\mathbb{E}\ell_\psi(t_{Q_n},Z)\right| E Q n ⊗ n R Q n ( w ∗ ( μ n )) / a ( n − 1/2 ) − E ℓ ψ ( t Q n , Z ) stays bounded away from zero. The bound on t Q n t_{Q_n} t Q n gives a further subsequence with t Q n → t t_{Q_n}\to t t Q n → t . The distributional limit and uniform integrability yield
E Q n ⊗ n R Q n ( w ∗ ( μ ~ n ) ) a ( n − 1 / 2 ) → E ℓ ψ ( t , Z ) , E ℓ ψ ( t Q n , Z ) → E ℓ ψ ( t , Z ) , \frac{\mathbb{E}_{Q_n^{\otimes n}}R_{Q_n}(w_*(\widetilde{\mu}_n))}{a(n^{-1/2})}\to\mathbb{E}\ell_\psi(t,Z), \qquad\mathbb{E}\ell_\psi(t_{Q_n},Z)\to\mathbb{E}\ell_\psi(t,Z), a ( n − 1/2 ) E Q n ⊗ n R Q n ( w ∗ ( μ n )) → E ℓ ψ ( t , Z ) , E ℓ ψ ( t Q n , Z ) → E ℓ ψ ( t , Z ) ,
a contradiction.
For the ordinary mean x ˉ \bar{x} x ˉ , (F.10) gives
sup Q E Q ⊗ n ∥ Δ n ∥ 2 ≤ sup Q E Q ∥ x 1 − x 1 ( n ) ∥ 2 = O ( n − 1 ) . (155) \sup_Q\mathbb{E}_{Q^{\otimes n}}\|\Delta_n\|_2\leq\sup_Q\mathbb{E}_Q\|x_1-x_1^{(n)}\|_2=O(n^{-1}). \tag*{(155)} Q sup E Q ⊗ n ∥ Δ n ∥ 2 ≤ Q sup E Q ∥ x 1 − x 1 ( n ) ∥ 2 = O ( n − 1 ) . (155)
For every h > 0 h>0 h > 0 , Markov’s inequality gives
sup Q Q ⊗ n { n ∥ Δ n ∥ 2 > h } ≤ n h sup Q E ∥ Δ n ∥ 2 = O ( n − 1 / 2 ) → 0. \sup_Q Q^{\otimes n}\{\sqrt{n}\|\Delta_n\|_2>h\}\leq\frac{\sqrt{n}}{h}\sup_Q\mathbb{E}\|\Delta_n\|_2=O(n^{-1/2})\to0. Q sup Q ⊗ n { n ∥ Δ n ∥ 2 > h } ≤ h n Q sup E ∥ Δ n ∥ 2 = O ( n − 1/2 ) → 0.
Since
n ( x ˉ − μ Q ) = Z n , Q + n Δ n , \sqrt{n}(\bar{x}-\mu_Q)=Z_{n,Q}+\sqrt{n}\Delta_n, n ( x ˉ − μ Q ) = Z n , Q + n Δ n ,
Slutsky’s theorem gives n ( x ˉ − μ Q n ) → d Z ∼ N ( 0 , Σ ) \sqrt{n}(\bar{x}-\mu_{Q_n})\xrightarrow{d}Z\sim N(0,\Sigma) n ( x ˉ − μ Q n ) d Z ∼ N ( 0 , Σ ) . Equation (G.2) therefore applies to μ ^ n = x ˉ \widehat{\mu}_n=\bar{x} μ n = x ˉ , i.e.,
R Q n ( w ∗ ( x ˉ ) ) a ( n − 1 / 2 ) → d ℓ ψ ( t , Z ) . \frac{R_{Q_n}(w_*(\bar{x}))}{a(n^{-1/2})}\xrightarrow{d}\ell_\psi(t,Z). a ( n − 1/2 ) R Q n ( w ∗ ( x ˉ )) d ℓ ψ ( t , Z ) .
For each fixed K > 0 K>0 K > 0 , convergence in distribution implies convergence of the expectations after applying the bounded continuous function x ↦ min { max { x , 0 } , K } x\mapsto\min\{\max\{x,0\},K\} x ↦ min { max { x , 0 } , K } , i.e.,
lim n → ∞ E Q n ⊗ n min { R Q n ( w ∗ ( x ˉ ) ) a ( n − 1 / 2 ) , K } = E min { ℓ ψ ( t , Z ) , K } . \lim_{n\to\infty}\mathbb{E}_{Q_n^{\otimes n}}\min\left\{\frac{R_{Q_n}(w_*(\bar{x}))}{a(n^{-1/2})},K\right\}=\mathbb{E}\min\{\ell_\psi(t,Z),K\}. n → ∞ lim E Q n ⊗ n min { a ( n − 1/2 ) R Q n ( w ∗ ( x ˉ )) , K } = E min { ℓ ψ ( t , Z ) , K } .
R Q n ( w ∗ ( x ˉ ) ) R_{Q_n}(w_*(\bar{x})) R Q n ( w ∗ ( x ˉ )) and ℓ ψ \ell_\psi ℓ ψ are nonnegative. Letting K → ∞ K\to\infty K → ∞ and using MCT therefore gives
lim inf n → ∞ E Q n ⊗ n R Q n ( w ∗ ( x ˉ ) ) a ( n − 1 / 2 ) ≥ E ℓ ψ ( t , Z ) if t Q n → t . (156) \liminf_{n\to\infty}\frac{\mathbb{E}_{Q_n^{\otimes n}}R_{Q_n}(w_*(\bar{x}))}{a(n^{-1/2})}\geq\mathbb{E}\ell_\psi(t,Z)\quad\text{if }t_{Q_n}\to t. \tag*{(156)} n → ∞ lim inf a ( n − 1/2 ) E Q n ⊗ n R Q n ( w ∗ ( x ˉ )) ≥ E ℓ ψ ( t , Z ) if t Q n → t . (156)
By (F.3), every cluster point of t Q t_Q t Q lies in Σ 1 / 2 B 2 ( M ) \Sigma^{1/2}B_2(M) Σ 1/2 B 2 ( M ) . Conversely, for every fixed t t t with ∥ Σ − 1 / 2 t ∥ 2 < M \|\Sigma^{-1/2}t\|_2<M ∥ Σ − 1/2 t ∥ 2 < M , the exponential perturbation P t / n P_{t/\sqrt{n}} P t / n has mean μ + t / n \mu+t/\sqrt{n} μ + t / n and, by (H.12), satisfies
D K L ( P t / n ∥ P ) = t ⊤ Σ − 1 t 2 n + O ( n − 3 / 2 ) < M 2 2 n D_{\mathrm{KL}}(P_{t/\sqrt{n}}\Vert P)=\frac{t^\top\Sigma^{-1}t}{2n}+O(n^{-3/2})<\frac{M^2}{2n} D KL ( P t / n ∥ P ) = 2 n t ⊤ Σ − 1 t + O ( n − 3/2 ) < 2 n M 2
for large n n n . Fix any t t t with ∥ Σ − 1 / 2 t ∥ 2 < M \lVert\Sigma^{-1/2}t\rVert_2 < M ∥ Σ − 1/2 t ∥ 2 < M and take Q n = P t / n Q_n = P_t/\sqrt{n} Q n = P t / n . The mean parametrization gives μ Q n = μ + t / n \mu_{Q_n} = \mu+ t/\sqrt{n} μ Q n = μ + t / n , and t Q n = n ( μ Q n − μ ) = t t_{Q_n} = \sqrt{n}(\mu_{Q_n} - \mu) = t t Q n = n ( μ Q n − μ ) = t . Since Q n ∈ Q n , M K L ( P ) Q_n \in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P) Q n ∈ Q n , M KL ( P ) for all sufficiently large n n n , the definitions of the maximal regrets imply
W n , M t r ( P ) ≥ E Q n ⊗ n R Q n ( w ∗ ( μ ~ n ) ) , W n , M m e a n ( P ) ≥ E Q n ⊗ n R Q n ( w ∗ ( x ˉ ) ) . W^{\mathrm{tr}}_{n,M}(P) \geq\mathbb{E}_{Q_n^{\otimes n}} R_{Q_n}(w_*(\widetilde{\mu}_n)), \qquad W^{\mathrm{mean}}_{n,M}(P) \geq\mathbb{E}_{Q_n^{\otimes n}} R_{Q_n}(w_*(\bar{x})). W n , M tr ( P ) ≥ E Q n ⊗ n R Q n ( w ∗ ( μ n )) , W n , M mean ( P ) ≥ E Q n ⊗ n R Q n ( w ∗ ( x ˉ )) .
Dividing by a ( n − 1 / 2 ) a(n^{-1/2}) a ( n − 1/2 ) and applying (G.5) and (G.8) yields
lim inf n → ∞ W n , M t r ( P ) a ( n − 1 / 2 ) ≥ lim n → ∞ E Q n ⊗ n R Q n ( w ∗ ( μ ~ n ) ) a ( n − 1 / 2 ) = E ℓ ψ ( t , Z ) , \liminf_{n\to\infty} \frac{W^{\mathrm{tr}}_{n,M}(P)}{a(n^{-1/2})} \geq\lim_{n\to\infty} \frac{\mathbb{E}_{Q_n^{\otimes n}}R_{Q_n}(w_*(\widetilde{\mu}_n))}{a(n^{-1/2})} = \mathbb{E}\ell_\psi(t,Z), n → ∞ lim inf a ( n − 1/2 ) W n , M tr ( P ) ≥ n → ∞ lim a ( n − 1/2 ) E Q n ⊗ n R Q n ( w ∗ ( μ n )) = E ℓ ψ ( t , Z ) ,
lim inf n → ∞ W n , M m e a n ( P ) a ( n − 1 / 2 ) ≥ lim inf n → ∞ E Q n ⊗ n R Q n ( w ∗ ( x ˉ ) ) a ( n − 1 / 2 ) ≥ E ℓ ψ ( t , Z ) . \liminf_{n\to\infty} \frac{W^{\mathrm{mean}}_{n,M}(P)}{a(n^{-1/2})} \geq\liminf_{n\to\infty} \frac{\mathbb{E}_{Q_n^{\otimes n}}R_{Q_n}(w_*(\bar{x}))}{a(n^{-1/2})} \geq\mathbb{E}\ell_\psi(t,Z). n → ∞ lim inf a ( n − 1/2 ) W n , M mean ( P ) ≥ n → ∞ lim inf a ( n − 1/2 ) E Q n ⊗ n R Q n ( w ∗ ( x ˉ )) ≥ E ℓ ψ ( t , Z ) .
Both left sides are independent of the chosen t t t , so each is at least sup ∥ Σ − 1 / 2 t ∥ 2 < M E ℓ ψ ( t , Z ) \sup_{\lVert\Sigma^{-1/2}t\rVert_2<M}\mathbb{E}\ell_\psi(t,Z) sup ∥ Σ − 1/2 t ∥ 2 < M E ℓ ψ ( t , Z ) . By (G.4),
sup ∥ Σ − 1 / 2 t ∥ 2 < M E ℓ ψ ( t , Z ) = sup ∥ Σ − 1 / 2 t ∥ 2 ≤ M E ℓ ψ ( t , Z ) = C M p l u g ( ψ , Σ ) , \sup_{\lVert\Sigma^{-1/2}t\rVert_2<M}\mathbb{E}\ell_\psi(t,Z) = \sup_{\lVert\Sigma^{-1/2}t\rVert_2\leq M}\mathbb{E}\ell_\psi(t,Z) = C_M^{\mathrm{plug}}(\psi,\Sigma), ∥ Σ − 1/2 t ∥ 2 < M sup E ℓ ψ ( t , Z ) = ∥ Σ − 1/2 t ∥ 2 ≤ M sup E ℓ ψ ( t , Z ) = C M plug ( ψ , Σ ) ,
and therefore
lim inf n → ∞ W n , M t r ( P ) a ( n − 1 / 2 ) ≥ C M p l u g ( ψ , Σ ) , \liminf_{n\to\infty} \frac{W^{\mathrm{tr}}_{n,M}(P)}{a(n^{-1/2})} \geq C_M^{\mathrm{plug}}(\psi,\Sigma), n → ∞ lim inf a ( n − 1/2 ) W n , M tr ( P ) ≥ C M plug ( ψ , Σ ) ,
lim inf n → ∞ W n , M m e a n ( P ) a ( n − 1 / 2 ) ≥ C M p l u g ( ψ , Σ ) . \liminf_{n\to\infty} \frac{W^{\mathrm{mean}}_{n,M}(P)}{a(n^{-1/2})} \geq C_M^{\mathrm{plug}}(\psi,\Sigma). n → ∞ lim inf a ( n − 1/2 ) W n , M mean ( P ) ≥ C M plug ( ψ , Σ ) .
For the reverse bound, choose Q n Q_n Q n within a ( n − 1 / 2 ) / n a(n^{-1/2})/n a ( n − 1/2 ) / n of the supremum defining W n , M t r W^{\mathrm{tr}}_{n,M} W n , M tr . The normalized risks are bounded by (F.19). Along a subsequence attaining their limsup, compactness gives a further subsequence with t Q n → t t_{Q_n}\to t t Q n → t ; (F.3) puts t ∈ Σ 1 / 2 B 2 ( M ) t\in\Sigma^{1/2}B_2(M) t ∈ Σ 1/2 B 2 ( M ) . Equations (G.6) and (G.4) then give
lim sup n → ∞ W n , M t r ( P ) a ( n − 1 / 2 ) = E ℓ ψ ( t , Z ) ≤ C M p l u g ( ψ , Σ ) . \limsup_{n\to\infty}\frac{W^{\mathrm{tr}}_{n,M}(P)}{a(n^{-1/2})} = \mathbb{E}\ell_\psi(t,Z) \leq C_M^{\mathrm{plug}}(\psi,\Sigma). n → ∞ lim sup a ( n − 1/2 ) W n , M tr ( P ) = E ℓ ψ ( t , Z ) ≤ C M plug ( ψ , Σ ) .
This proves (55) . Finally, Theorem 5.2 and R n , M K L ( P ) ≤ W n , M t r ( P ) R^{\mathrm{KL}}_{n,M}(P)\leq W^{\mathrm{tr}}_{n,M}(P) R n , M KL ( P ) ≤ W n , M tr ( P ) imply
C M p l u g ( ψ , Σ ) ≥ V M , I d ( ψ ∘ Σ 1 / 2 ) > 0. C_M^{\mathrm{plug}}(\psi,\Sigma) \geq V_{M,\mathrm{Id}}(\psi\circ\Sigma^{1/2}) > 0. C M plug ( ψ , Σ ) ≥ V M , Id ( ψ ∘ Σ 1/2 ) > 0.
The two remaining comparisons follow because W n , M t r ( P ) / a ( n − 1 / 2 ) W^{\mathrm{tr}}_{n,M}(P)/a(n^{-1/2}) W n , M tr ( P ) / a ( n − 1/2 ) converges to this positive constant and W n , M m e a n ( P ) / a ( n − 1 / 2 ) W^{\mathrm{mean}}_{n,M}(P)/a(n^{-1/2}) W n , M mean ( P ) / a ( n − 1/2 ) has at least the same lower limit.
Proof of Proposition 6.2
Proof. By definition,
R Q ( w ∗ ( x ˉ ) ) = r μ ( μ Q − μ ) − r μ ( x ˉ − μ ) + ( w ∗ ( x ˉ ) − w ∗ ( μ ) ) ⊤ ( x ˉ − μ Q ) . R_Q(w_*(\bar{x})) = r_\mu(\mu_Q-\mu)-r_\mu(\bar{x}-\mu)+(w_*(\bar{x})-w_*(\mu))^\top(\bar{x}-\mu_Q). R Q ( w ∗ ( x ˉ )) = r μ ( μ Q − μ ) − r μ ( x ˉ − μ ) + ( w ∗ ( x ˉ ) − w ∗ ( μ ) ) ⊤ ( x ˉ − μ Q ) .
Combined with the subgradient inequality
r μ ( 2 x ˉ − μ Q − μ ) ≥ r μ ( x ˉ − μ ) + ( w ∗ ( x ˉ ) − w ∗ ( μ ) ) ⊤ ( x ˉ − μ Q ) , r_\mu(2\bar{x}-\mu_Q-\mu) \geq r_\mu(\bar{x}-\mu)+(w_*(\bar{x})-w_*(\mu))^\top(\bar{x}-\mu_Q), r μ ( 2 x ˉ − μ Q − μ ) ≥ r μ ( x ˉ − μ ) + ( w ∗ ( x ˉ ) − w ∗ ( μ ) ) ⊤ ( x ˉ − μ Q ) ,
we have
R Q ( w ∗ ( x ˉ ) ) ≤ r μ ( μ Q − μ ) + r μ ( 2 x ˉ − μ Q − μ ) . R_Q(\mathbf{w}_*(\bar{x})) \le r_\mu(\mu_Q-\mu)+r_\mu(2\bar{x}-\mu_Q-\mu). R Q ( w ∗ ( x ˉ )) ≤ r μ ( μ Q − μ ) + r μ ( 2 x ˉ − μ Q − μ ) .
Since 2 x ˉ − μ Q − μ = ( 4 μ ~ n − 2 μ Q − 2 μ ) / 2 + ( 4 Δ n ) / 2 2\bar{x}-\mu_Q-\mu=(4\widetilde{\mu}_n-2\mu_Q-2\mu)/2+(4\Delta_n)/2 2 x ˉ − μ Q − μ = ( 4 μ n − 2 μ Q − 2 μ ) /2 + ( 4 Δ n ) /2 , convexity of r μ r_\mu r μ yields
0 ≤ R Q ( w ∗ ( x ˉ ) ) ≤ r μ ( μ Q − μ ) + 1 2 r μ ( 4 μ ~ n − 2 μ Q − 2 μ ) + 1 2 r μ ( 4 Δ n ) . (157) 0\le R_Q(\mathbf{w}_*(\bar{x}))\le r_\mu(\mu_Q-\mu)+\frac{1}{2}r_\mu(4\widetilde{\mu}_n-2\mu_Q-2\mu)+\frac{1}{2}r_\mu(4\Delta_n). \tag*{(157)} 0 ≤ R Q ( w ∗ ( x ˉ )) ≤ r μ ( μ Q − μ ) + 2 1 r μ ( 4 μ n − 2 μ Q − 2 μ ) + 2 1 r μ ( 4 Δ n ) . (157)
Write Z n , Q = n ( μ ~ n − μ Q ) Z_{n,Q}=\sqrt{n}(\widetilde{\mu}_n-\mu_Q) Z n , Q = n ( μ n − μ Q ) as in (47) . Then
μ Q − μ = t Q n , 4 μ ~ n − 2 μ Q − 2 μ = 2 t Q + 4 Z n , Q n . \mu_Q-\mu=\frac{t_Q}{\sqrt{n}},\qquad4\widetilde{\mu}_n-2\mu_Q-2\mu=\frac{2t_Q+4Z_{n,Q}}{\sqrt{n}}. μ Q − μ = n t Q , 4 μ n − 2 μ Q − 2 μ = n 2 t Q + 4 Z n , Q .
Choose L L L with t Q ∈ B 2 ( L ) t_Q\in B_2(L) t Q ∈ B 2 ( L ) and sup B 2 ( L ) ψ > 0 \sup_{B_2(L)}\psi>0 sup B 2 ( L ) ψ > 0 . By (C.3) and (C.5), the normalized sum
A n , Q : = r μ ( μ Q − μ ) + r μ ( 4 μ ~ n − 2 μ Q − 2 μ ) / 2 a ( n − 1 / 2 ) A_{n,Q}:=\frac{r_\mu(\mu_Q-\mu)+r_\mu(4\widetilde{\mu}_n-2\mu_Q-2\mu)/2}{a(n^{-1/2})} A n , Q := a ( n − 1/2 ) r μ ( μ Q − μ ) + r μ ( 4 μ n − 2 μ Q − 2 μ ) /2
satisfies, by ∥ 2 t Q + 4 Z n , Q ∥ 2 ≤ L ( 2 + 4 ∥ Z n , Q ∥ 2 / L ) \lVert2t_Q+4Z_{n,Q}\rVert_2\le L(2+4\lVert Z_{n,Q}\rVert_2/L) ∥ 2 t Q + 4 Z n , Q ∥ 2 ≤ L ( 2 + 4 ∥ Z n , Q ∥ 2 / L ) ,
0 ≤ A n , Q ≤ b L ( n − 1 / 2 ) a ( n − 1 / 2 ) [ 1 + C b 2 ( 2 + 4 ∥ Z n , Q ∥ 2 / L ) c b ] ≤ C ( 1 + ∥ Z n , Q ∥ 2 ) c b . 0\le A_{n,Q}\le\frac{b_L(n^{-1/2})}{a(n^{-1/2})}\left[1+\frac{C_b}{2}(2+4\lVert Z_{n,Q}\rVert_2/L)^{c_b}\right]\le C(1+\lVert Z_{n,Q}\rVert_2)^{c_b}. 0 ≤ A n , Q ≤ a ( n − 1/2 ) b L ( n − 1/2 ) [ 1 + 2 C b ( 2 + 4 ∥ Z n , Q ∥ 2 / L ) c b ] ≤ C ( 1 + ∥ Z n , Q ∥ 2 ) c b .
for some C , c b C,c_b C , c b which may depend on L L L . Lemma F.1 therefore gives
sup n l a r g e sup Q E A n , Q 2 < ∞ . \sup_{n\ \mathrm{large}}\sup_Q\mathbb{E}A_{n,Q}^2<\infty. n large sup Q sup E A n , Q 2 < ∞.
Taking expectations in (G.9) proves the first inequality in (56) . Moreover,
0 ≤ r μ ( 4 Δ n ) ≤ 8 R Ω ∥ Δ n ∥ 2 . 0\le r_\mu(4\Delta_n)\le8R_\Omega\lVert\Delta_n\rVert_2. 0 ≤ r μ ( 4 Δ n ) ≤ 8 R Ω ∥ Δ n ∥ 2 .
Equation (G.7) proves the second inequality. Under (57) , the positive minimax limit in Theorem 5.2 supplies the matching lower bound.
Suppose now that (58) holds. For nonnegative x , A , B x,A,B x , A , B with x ≤ A + B x\le A+B x ≤ A + B , splitting according to A > K / 2 A>K/2 A > K /2 gives
x I { x > K } ≤ 2 A I { A > K / 2 } + 2 B . x\mathbb{I}\{x>K\}\le2A\mathbb{I}\{A>K/2\}+2B. x I { x > K } ≤ 2 A I { A > K /2 } + 2 B .
Indeed, on { A ≤ K / 2 , x > K } \{A\le K/2,x>K\} { A ≤ K /2 , x > K } we have B > K / 2 ≥ A B>K/2\ge A B > K /2 ≥ A , hence x ≤ 2 B x\le2B x ≤ 2 B ; on { A > K / 2 } \{A>K/2\} { A > K /2 } use x ≤ A + B x\le A+B x ≤ A + B . Set
L n , Q : = R Q ( w ∗ ( x ˉ ) ) a ( n − 1 / 2 ) . L_{n,Q}:=\frac{R_Q(\mathbf{w}_*(\bar{x}))}{a(n^{-1/2})}. L n , Q := a ( n − 1/2 ) R Q ( w ∗ ( x ˉ )) .
Apply the inequality with x = L n , Q x=L_{n,Q} x = L n , Q , A = A n , Q A=A_{n,Q} A = A n , Q , and B = r μ ( 4 Δ n ) / ( 2 a ( n − 1 / 2 ) ) B=r_\mu(4\Delta_n)/(2a(n^{-1/2})) B = r μ ( 4 Δ n ) / ( 2 a ( n − 1/2 )) . Equation (G.9) then implies
sup Q E [ L n , Q I { L n , Q > K } ] ≤ 2 sup Q E [ A n , Q I { A n , Q > K / 2 } ] + sup Q E r μ ( 4 Δ n ) a ( n − 1 / 2 ) ≤ 4 K sup Q E A n , Q 2 + sup Q E r μ ( 4 Δ n ) a ( n − 1 / 2 ) . \begin{aligned}
\sup_Q\mathbb{E}[L_{n,Q}\mathbb{I}\{L_{n,Q}>K\}]
\le2\sup_Q\mathbb{E}[A_{n,Q}\mathbb{I}\{A_{n,Q}>K/2\}]
+\frac{\sup_Q\mathbb{E}r_\mu(4\Delta_n)}{a(n^{-1/2})} \\
&\le\frac{4}{K}\sup_Q\mathbb{E}A_{n,Q}^2+\frac{\sup_Q\mathbb{E}r_\mu(4\Delta_n)}{a(n^{-1/2})}.
\end{aligned} Q sup E [ L n , Q I { L n , Q > K }] ≤ 2 Q sup E [ A n , Q I { A n , Q > K /2 }] + a ( n − 1/2 ) sup Q E r μ ( 4 Δ n ) ≤ K 4 Q sup E A n , Q 2 + a ( n − 1/2 ) sup Q E r μ ( 4 Δ n ) .
By (58) and the uniform second-moment bound for A n , Q A_{n,Q} A n , Q , for every K > 0 K>0 K > 0 ,
lim sup n → ∞ sup Q E Q ⊗ n [ R Q ( w ∗ ( x ˉ ) ) a ( n − 1 / 2 ) I { R Q ( w ∗ ( x ˉ ) ) a ( n − 1 / 2 ) > K } ] ≤ C K . \limsup_{n\to\infty}\sup_{Q}\mathbb{E}_{Q}^{\otimes n}\left[\frac{R_Q(\mathbf{w}_*(\bar{x}))}{a(n^{-1/2})}\mathbb{I}\left\{\frac{R_Q(\mathbf{w}_*(\bar{x}))}{a(n^{-1/2})}>K\right\}\right]\leq\frac{C}{K}. n → ∞ lim sup Q sup E Q ⊗ n [ a ( n − 1/2 ) R Q ( w ∗ ( x ˉ )) I { a ( n − 1/2 ) R Q ( w ∗ ( x ˉ )) > K } ] ≤ K C .
Consequently,
lim K → ∞ lim sup n → ∞ sup Q E Q ⊗ n [ R Q ( w ∗ ( x ˉ ) ) a ( n − 1 / 2 ) I { R Q ( w ∗ ( x ˉ ) ) a ( n − 1 / 2 ) > K } ] = 0. (158) \lim_{K\to\infty}\limsup_{n\to\infty}\sup_{Q}\mathbb{E}_{Q}^{\otimes n}\left[\frac{R_Q(\mathbf{w}_*(\bar{x}))}{a(n^{-1/2})}\mathbb{I}\left\{\frac{R_Q(\mathbf{w}_*(\bar{x}))}{a(n^{-1/2})}>K\right\}\right]=0. \tag*{(158)} K → ∞ lim n → ∞ lim sup Q sup E Q ⊗ n [ a ( n − 1/2 ) R Q ( w ∗ ( x ˉ )) I { a ( n − 1/2 ) R Q ( w ∗ ( x ˉ )) > K } ] = 0. (158)
Fix any sequence Q n ∈ Q n , M K L ( P ) Q_n\in\mathcal{Q}_{n,M}^{\mathrm{KL}}(P) Q n ∈ Q n , M KL ( P ) such that t Q n → t t_{Q_n}\to t t Q n → t . Lemma F.1 gives
n ( μ ~ n − μ Q n ) → d Z , Z ∼ N ( 0 , Σ ) . \sqrt{n}(\widetilde{\mu}_n-\mu_{Q_n})\xrightarrow{d}Z,\qquad Z\sim N(0,\Sigma). n ( μ n − μ Q n ) d Z , Z ∼ N ( 0 , Σ ) .
Since Δ n = x ˉ − μ ~ n \Delta_n=\bar{x}-\widetilde{\mu}_n Δ n = x ˉ − μ n , Markov’s inequality and (G.7) imply, for every ε > 0 \varepsilon>0 ε > 0 ,
P Q n ⊗ n { n ∥ Δ n ∥ 2 > ε } ≤ n ε E Q n ⊗ n ∥ Δ n ∥ 2 = O ( n − 1 / 2 ) → 0. \mathbb{P}_{Q_n}^{\otimes n}\{\sqrt{n}\|\Delta_n\|_2>\varepsilon\}\leq\frac{\sqrt{n}}{\varepsilon}\mathbb{E}_{Q_n}^{\otimes n}\|\Delta_n\|_2=O(n^{-1/2})\to0. P Q n ⊗ n { n ∥ Δ n ∥ 2 > ε } ≤ ε n E Q n ⊗ n ∥ Δ n ∥ 2 = O ( n − 1/2 ) → 0.
Slutsky’s theorem therefore gives
n ( x ˉ − μ Q n ) = n ( μ ~ n − μ Q n ) + n Δ n → d Z . \sqrt{n}(\bar{x}-\mu_{Q_n})=\sqrt{n}(\widetilde{\mu}_n-\mu_{Q_n})+\sqrt{n}\Delta_n\xrightarrow{d}Z. n ( x ˉ − μ Q n ) = n ( μ n − μ Q n ) + n Δ n d Z .
Thus by (G.2), taking μ ^ n = x ˉ \widehat{\mu}_n=\bar{x} μ n = x ˉ yields
R Q n ( w ∗ ( x ˉ ) ) a ( n − 1 / 2 ) → d ℓ ψ ( t , Z ) . \frac{R_{Q_n}(\mathbf{w}_*(\bar{x}))}{a(n^{-1/2})}\xrightarrow{d}\ell_\psi(t,Z). a ( n − 1/2 ) R Q n ( w ∗ ( x ˉ )) d ℓ ψ ( t , Z ) .
For every fixed K > 0 K>0 K > 0 , applying the bounded continuous function x ↦ min { max { x , 0 } , K } x\mapsto\min\{\max\{x,0\},K\} x ↦ min { max { x , 0 } , K } gives
E Q n ⊗ n min { R Q n ( w ∗ ( x ˉ ) ) a ( n − 1 / 2 ) , K } → E min { ℓ ψ ( t , Z ) , K } . \mathbb{E}_{Q_n}^{\otimes n}\min\left\{\frac{R_{Q_n}(\mathbf{w}_*(\bar{x}))}{a(n^{-1/2})},K\right\}\to\mathbb{E}\min\{\ell_\psi(t,Z),K\}. E Q n ⊗ n min { a ( n − 1/2 ) R Q n ( w ∗ ( x ˉ )) , K } → E min { ℓ ψ ( t , Z ) , K } .
Since both losses are nonnegative, by (G.10),
0 ≤ E Q n ⊗ n R Q n ( w ∗ ( x ˉ ) ) a ( n − 1 / 2 ) − E Q n ⊗ n min { R Q n ( w ∗ ( x ˉ ) ) a ( n − 1 / 2 ) , K } ≤ E Q n ⊗ n [ R Q n ( w ∗ ( x ˉ ) ) a ( n − 1 / 2 ) I { R Q n ( w ∗ ( x ˉ ) ) a ( n − 1 / 2 ) > K } ] , (159) \begin{aligned}
0\leq\frac{\mathbb{E}_{Q_n}^{\otimes n}R_{Q_n}(\mathbf{w}_*(\bar{x}))}{a(n^{-1/2})}-\mathbb{E}_{Q_n}^{\otimes n}\min\left\{\frac{R_{Q_n}(\mathbf{w}_*(\bar{x}))}{a(n^{-1/2})},K\right\} \\
&\leq\mathbb{E}_{Q_n}^{\otimes n}\left[\frac{R_{Q_n}(\mathbf{w}_*(\bar{x}))}{a(n^{-1/2})}\mathbb{I}\left\{\frac{R_{Q_n}(\mathbf{w}_*(\bar{x}))}{a(n^{-1/2})}>K\right\}\right], \tag*{(159)}
\end{aligned} 0 ≤ a ( n − 1/2 ) E Q n ⊗ n R Q n ( w ∗ ( x ˉ )) − E Q n ⊗ n min { a ( n − 1/2 ) R Q n ( w ∗ ( x ˉ )) , K } ≤ E Q n ⊗ n [ a ( n − 1/2 ) R Q n ( w ∗ ( x ˉ )) I { a ( n − 1/2 ) R Q n ( w ∗ ( x ˉ )) > K } ] , (159)
0 ≤ E ℓ ψ ( t , Z ) − E min { ℓ ψ ( t , Z ) , K } ≤ E [ ℓ ψ ( t , Z ) I { ℓ ψ ( t , Z ) > K } ] . 0\leq\mathbb{E}\ell_\psi(t,Z)-\mathbb{E}\min\{\ell_\psi(t,Z),K\}\leq\mathbb{E}[\ell_\psi(t,Z)\mathbb{I}\{\ell_\psi(t,Z)>K\}]. 0 ≤ E ℓ ψ ( t , Z ) − E min { ℓ ψ ( t , Z ) , K } ≤ E [ ℓ ψ ( t , Z ) I { ℓ ψ ( t , Z ) > K }] .
Asymptotic uniform integrability makes
lim sup n → ∞ E Q n ⊗ n [ R Q n ( w ∗ ( x ˉ ) ) a ( n − 1 / 2 ) I { R Q n ( w ∗ ( x ˉ ) ) a ( n − 1 / 2 ) > K } ] → 0 , \limsup_{n\to\infty}\mathbb{E}_{Q_n}^{\otimes n}\left[\frac{R_{Q_n}(\mathbf{w}_*(\bar{x}))}{a(n^{-1/2})}\mathbb{I}\left\{\frac{R_{Q_n}(\mathbf{w}_*(\bar{x}))}{a(n^{-1/2})}>K\right\}\right]\to0, n → ∞ lim sup E Q n ⊗ n [ a ( n − 1/2 ) R Q n ( w ∗ ( x ˉ )) I { a ( n − 1/2 ) R Q n ( w ∗ ( x ˉ )) > K } ] → 0 ,
as K → ∞ K\to\infty K → ∞ . Passing to the limit in (G.3) and using finiteness of every fixed Gaussian moment gives
0 ≤ E ℓ ψ ( t , Z ) ≤ C E ( 1 + ∥ Z ∥ 2 ) c b < ∞ . 0\leq\mathbb{E}\ell_\psi(t,Z)\leq C\mathbb{E}(1+\|Z\|_2)^{c_b}<\infty. 0 ≤ E ℓ ψ ( t , Z ) ≤ C E ( 1 + ∥ Z ∥ 2 ) c b < ∞.
Hence E [ ℓ ψ ( t , Z ) 1 { ℓ ψ ( t , Z ) > K } ] → 0 \mathbb{E}[\ell_{\psi}(t,\boldsymbol{Z})\mathbb{1}\{\ell_{\psi}(t,\boldsymbol{Z}) > K\}] \to0 E [ ℓ ψ ( t , Z ) 1 { ℓ ψ ( t , Z ) > K }] → 0 as K → ∞ K \to\infty K → ∞ . Letting first n → ∞ n \to\infty n → ∞ and then K → ∞ K \to\infty K → ∞ in (G.11) therefore gives
E Q n ⊗ n R Q n ( w ∗ ( x ˉ ) ) a ( n − 1 / 2 ) → E ℓ ψ ( t , Z ) . (160) \frac{\mathbb{E}_{Q_n \otimes n}R_{Q_n}(\boldsymbol{w}_*(\bar{\boldsymbol{x}}))}{a(n^{-1/2})} \to\mathbb{E}\ell_{\psi}(t,\boldsymbol{Z}). \tag*{(160)} a ( n − 1/2 ) E Q n ⊗ n R Q n ( w ∗ ( x ˉ )) → E ℓ ψ ( t , Z ) . (160)
All possible t Q t_{Q} t Q belong to the same fixed compact ball B 2 ( L ) B_2(L) B 2 ( L ) for sufficiently large n n n . Suppose there exists some ε > 0 \varepsilon> 0 ε > 0 , a subsequence n j n_j n j , and Q n j ∈ Q n j , M K L ( P ) Q_{n_j} \in\mathcal{Q}^{\mathrm{KL}}_{n_j,M}(P) Q n j ∈ Q n j , M KL ( P ) such that
∣ E Q n j ⊗ n j R Q n j ( w ∗ ( x ˉ ) ) a ( n j − 1 / 2 ) − E ℓ ψ ( t Q n j , Z ) ∣ ≥ ε . \left|\frac{\mathbb{E}_{Q_{n_j}^{\otimes n_j}}R_{Q_{n_j}}(\boldsymbol{w}_*(\bar{\boldsymbol{x}}))}{a(n_j^{-1/2})}-\mathbb{E}\ell_{\psi}(t_{Q_{n_j}},\boldsymbol{Z})\right| \ge\varepsilon. a ( n j − 1/2 ) E Q n j ⊗ n j R Q n j ( w ∗ ( x ˉ )) − E ℓ ψ ( t Q n j , Z ) ≥ ε .
Passing to a further subsequence implies that there exists some t ∈ B 2 ( L ) t \in B_2(L) t ∈ B 2 ( L ) , such that t Q n j → t ∈ B 2 ( L ) t_{Q_{n_j}} \to t \in B_2(L) t Q n j → t ∈ B 2 ( L ) . (G.12) and (G.4) imply
E Q n j ⊗ n j R Q n j ( w ∗ ( x ˉ ) ) a ( n j − 1 / 2 ) → E ℓ ψ ( t , Z ) , E ℓ ψ ( t Q n j , Z ) → E ℓ ψ ( t , Z ) , \frac{\mathbb{E}_{Q_{n_j}^{\otimes n_j}}R_{Q_{n_j}}(\boldsymbol{w}_*(\bar{\boldsymbol{x}}))}{a(n_j^{-1/2})} \to\mathbb{E}\ell_{\psi}(t,\boldsymbol{Z}), \qquad\mathbb{E}\ell_{\psi}(t_{Q_{n_j}},\boldsymbol{Z}) \to\mathbb{E}\ell_{\psi}(t,\boldsymbol{Z}), a ( n j − 1/2 ) E Q n j ⊗ n j R Q n j ( w ∗ ( x ˉ )) → E ℓ ψ ( t , Z ) , E ℓ ψ ( t Q n j , Z ) → E ℓ ψ ( t , Z ) ,
a contradiction. Thus
sup Q ∣ E Q ⊗ n R Q ( w ∗ ( x ˉ ) ) a ( n − 1 / 2 ) − E ℓ ψ ( t Q , Z ) ∣ → 0. \sup_Q\left|\frac{\mathbb{E}_{Q\otimes n}R_Q(\boldsymbol{w}_*(\bar{\boldsymbol{x}}))}{a(n^{-1/2})}-\mathbb{E}\ell_{\psi}(t_Q,\boldsymbol{Z})\right| \to0. Q sup a ( n − 1/2 ) E Q ⊗ n R Q ( w ∗ ( x ˉ )) − E ℓ ψ ( t Q , Z ) → 0.
In particular,
∣ W n , M m e a n ( P ) a ( n − 1 / 2 ) − sup Q E ℓ ψ ( t Q , Z ) ∣ ≤ sup Q ∣ E Q ⊗ n R Q ( w ∗ ( x ˉ ) ) a ( n − 1 / 2 ) − E ℓ ψ ( t Q , Z ) ∣ → 0. \left|\frac{W^{\mathrm{mean}}_{n,M}(P)}{a(n^{-1/2})}-\sup_Q\mathbb{E}\ell_{\psi}(t_Q,\boldsymbol{Z})\right|
\leq\sup_Q\left|\frac{\mathbb{E}_{Q\otimes n}R_Q(\boldsymbol{w}_*(\bar{\boldsymbol{x}}))}{a(n^{-1/2})}-\mathbb{E}\ell_{\psi}(t_Q,\boldsymbol{Z})\right| \to0. a ( n − 1/2 ) W n , M mean ( P ) − Q sup E ℓ ψ ( t Q , Z ) ≤ Q sup a ( n − 1/2 ) E Q ⊗ n R Q ( w ∗ ( x ˉ )) − E ℓ ψ ( t Q , Z ) → 0.
By (F.3),
sup Q ∥ Σ − 1 / 2 t Q ∥ 2 2 ≤ M 2 + o ( 1 ) . \sup_Q \left\|\Sigma^{-1/2}t_Q\right\|_2^2 \leq M^2+o(1). Q sup Σ − 1/2 t Q 2 2 ≤ M 2 + o ( 1 ) .
For each n n n , choose Q n ∈ Q n , M K L ( P ) Q_n \in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P) Q n ∈ Q n , M KL ( P ) such that
0 ≤ sup Q ∈ Q n , M K L ( P ) E ℓ ψ ( t Q , Z ) − E ℓ ψ ( t Q n , Z ) ≤ 1 n . 0 \leq\sup_{Q\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)} \mathbb{E}\ell_{\psi}(t_Q,\boldsymbol{Z})-\mathbb{E}\ell_{\psi}(t_{Q_n},\boldsymbol{Z}) \leq\frac{1}{n}. 0 ≤ Q ∈ Q n , M KL ( P ) sup E ℓ ψ ( t Q , Z ) − E ℓ ψ ( t Q n , Z ) ≤ n 1 .
All possible t Q t_Q t Q belong to the same fixed compact ball B 2 ( L ) B_2(L) B 2 ( L ) for sufficiently large n n n . Continuity in (G.4) therefore implies
0 ≤ sup Q ∈ Q n , M K L ( P ) E ℓ ψ ( t Q , Z ) ≤ sup t ∈ B 2 ( L ) E ℓ ψ ( t , Z ) < ∞ . 0 \leq\sup_{Q\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)}\mathbb{E}\ell_{\psi}(t_Q,\boldsymbol{Z}) \leq\sup_{t\in B_2(L)}\mathbb{E}\ell_{\psi}(t,\boldsymbol{Z}) < \infty. 0 ≤ Q ∈ Q n , M KL ( P ) sup E ℓ ψ ( t Q , Z ) ≤ t ∈ B 2 ( L ) sup E ℓ ψ ( t , Z ) < ∞.
Choose a subsequence n j n_j n j along which
sup Q ∈ Q n j , M K L ( P ) E ℓ ψ ( t Q , Z ) → lim sup n → ∞ sup Q ∈ Q n , M K L ( P ) E ℓ ψ ( t Q , Z ) . \sup_{Q\in\mathcal{Q}^{\mathrm{KL}}_{n_j,M}(P)}\mathbb{E}\ell_{\psi}(t_Q,\boldsymbol{Z}) \to\limsup_{n\to\infty}\sup_{Q\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)}\mathbb{E}\ell_{\psi}(t_Q,\boldsymbol{Z}). Q ∈ Q n j , M KL ( P ) sup E ℓ ψ ( t Q , Z ) → n → ∞ lim sup Q ∈ Q n , M KL ( P ) sup E ℓ ψ ( t Q , Z ) .
Since t Q n j ∈ B 2 ( L ) t_{Q_{n_j}} \in B_2(L) t Q n j ∈ B 2 ( L ) , compactness gives a further subsequence, still denoted by n j n_j n j , and t ∈ B 2 ( L ) t \in B_2(L) t ∈ B 2 ( L ) such that t Q n j → t t_{Q_{n_j}} \to t t Q n j → t . By (F.3),
∥ Σ − 1 / 2 t ∥ 2 2 = lim j → ∞ ∥ Σ − 1 / 2 t Q n j ∥ 2 2 ≤ M 2 , t ∈ Σ 1 / 2 B 2 ( M ) . \|\Sigma^{-1/2}t\|_2^2=\lim_{j\to\infty}\|\Sigma^{-1/2}t_{Q_{n_j}}\|_2^2\le M^2,\qquad t\in\Sigma^{1/2}B_2(M). ∥ Σ − 1/2 t ∥ 2 2 = j → ∞ lim ∥ Σ − 1/2 t Q n j ∥ 2 2 ≤ M 2 , t ∈ Σ 1/2 B 2 ( M ) .
Equation (G.4) gives
E ℓ ψ ( t Q n j , Z ) → E ℓ ψ ( t , Z ) . \mathbb{E}\ell_\psi(t_{Q_{n_j}},Z)\to\mathbb{E}\ell_\psi(t,Z). E ℓ ψ ( t Q n j , Z ) → E ℓ ψ ( t , Z ) .
Combining these limits with the approximation error 1 / n j → 0 1/n_j\to0 1/ n j → 0 yields
lim sup n → ∞ sup Q ∈ Q n , M K L ( P ) E ℓ ψ ( t Q , Z ) = lim j → ∞ E ℓ ψ ( t Q n j , Z ) = E ℓ ψ ( t , Z ) ≤ sup t ∈ Σ 1 / 2 B 2 ( M ) E ℓ ψ ( t , Z ) = C M p l u g ( ψ , Σ ) . \begin{aligned}
\limsup_{n\to\infty}\sup_{Q\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)}\mathbb{E}\ell_\psi(t_Q,Z)=\lim_{j\to\infty}\mathbb{E}\ell_\psi(t_{Q_{n_j}},Z)=\mathbb{E}\ell_\psi(t,Z) \\
&\le\sup_{t\in\Sigma^{1/2}B_2(M)}\mathbb{E}\ell_\psi(t,Z)=C_M^{\mathrm{plug}}(\psi,\Sigma).
\end{aligned} n → ∞ lim sup Q ∈ Q n , M KL ( P ) sup E ℓ ψ ( t Q , Z ) = j → ∞ lim E ℓ ψ ( t Q n j , Z ) = E ℓ ψ ( t , Z ) ≤ t ∈ Σ 1/2 B 2 ( M ) sup E ℓ ψ ( t , Z ) = C M plug ( ψ , Σ ) .
Conversely, fix t t t with ∥ Σ − 1 / 2 t ∥ 2 < M \|\Sigma^{-1/2}t\|_2<M ∥ Σ − 1/2 t ∥ 2 < M . Equation (H.12) gives
n D K L ( P t / n ∥ P ) = 1 2 t ⊤ Σ − 1 t + O ( n − 1 / 2 ) < M 2 2 nD_{\mathrm{KL}}(P_{t/\sqrt{n}}\|P)=\frac{1}{2}t^\top\Sigma^{-1}t+O(n^{-1/2})<\frac{M^2}{2} n D KL ( P t / n ∥ P ) = 2 1 t ⊤ Σ − 1 t + O ( n − 1/2 ) < 2 M 2
for sufficiently large n n n . Thus Q n = P t / n ∈ Q n , M K L ( P ) Q_n=P_{t/\sqrt{n}}\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P) Q n = P t / n ∈ Q n , M KL ( P ) and t Q n = t t_{Q_n}=t t Q n = t , so
lim inf n → ∞ sup Q E ℓ ψ ( t Q , Z ) ≥ sup ∥ Σ − 1 / 2 t ∥ 2 < M E ℓ ψ ( t , Z ) = C M p l u g ( ψ , Σ ) . \liminf_{n\to\infty}\sup_Q\mathbb{E}\ell_\psi(t_Q,Z)\ge\sup_{\|\Sigma^{-1/2}t\|_2<M}\mathbb{E}\ell_\psi(t,Z)=C_M^{\mathrm{plug}}(\psi,\Sigma). n → ∞ lim inf Q sup E ℓ ψ ( t Q , Z ) ≥ ∥ Σ − 1/2 t ∥ 2 < M sup E ℓ ψ ( t , Z ) = C M plug ( ψ , Σ ) .
The last equality follows from (G.4), by approximating each boundary point t t t with ( 1 − j − 1 ) t (1-j^{-1})t ( 1 − j − 1 ) t , j ≥ 2 j\ge2 j ≥ 2 . Consequently,
W n , M m e a n ( P ) a ( n − 1 / 2 ) → C M p l u g ( ψ , Σ ) . \frac{W_{n,M}^{\mathrm{mean}}(P)}{a(n^{-1/2})}\to C_M^{\mathrm{plug}}(\psi,\Sigma). a ( n − 1/2 ) W n , M mean ( P ) → C M plug ( ψ , Σ ) .
Together with (55) , this proves (59) .
Finally, (G.7) and r μ ( 4 Δ n ) ≤ 8 R Ω ∥ Δ n ∥ 2 r_\mu(4\Delta_n)\le8R_\Omega\|\Delta_n\|_2 r μ ( 4 Δ n ) ≤ 8 R Ω ∥ Δ n ∥ 2 give
0 ≤ sup Q E Q ⊗ n r μ ( 4 Δ n ) a ( n − 1 / 2 ) ≤ C n a ( n − 1 / 2 ) . 0\le\frac{\sup_Q\mathbb{E}_{Q^{\otimes n}}r_\mu(4\Delta_n)}{a(n^{-1/2})}\le\frac{C}{na(n^{-1/2})}. 0 ≤ a ( n − 1/2 ) sup Q E Q ⊗ n r μ ( 4 Δ n ) ≤ na ( n − 1/2 ) C .
If lim inf n → ∞ n a ( n − 1 / 2 ) > 0 \liminf_{n\to\infty}na(n^{-1/2})>0 lim inf n → ∞ na ( n − 1/2 ) > 0 , the right side is bounded, proving (57) and hence rate-optimality. If n a ( n − 1 / 2 ) → ∞ na(n^{-1/2})\to\infty na ( n − 1/2 ) → ∞ , the right side converges to zero, proving (58) and hence (59) .
Proof of Corollary 6.3
Proof. Choose 0 < c 1 < c 0<c_1<c 0 < c 1 < c . Applying (F.8) to g ( x ) = c 1 ∥ x ∥ 2 I { ∥ x ∥ 2 > τ n } g(x)=c_1\|x\|_2\mathbb{I}\{\|x\|_2>\tau_n\} g ( x ) = c 1 ∥ x ∥ 2 I { ∥ x ∥ 2 > τ n } gives
sup Q E Q [ ∥ x ∥ 2 I { ∥ x ∥ 2 > τ n } ] ≤ 1 c 1 { M 2 2 n + e − ( c − c 1 ) τ n 2 E P e c ∥ x ∥ 2 2 } = O ( n − 1 ) . \sup_Q\mathbb{E}_Q[\|x\|_2\mathbb{I}\{\|x\|_2>\tau_n\}]\le\frac{1}{c_1}\left\{\frac{M^2}{2n}+e^{-(c-c_1)\tau_n^2}\mathbb{E}_P e^{c\|x\|_2^2}\right\}=O(n^{-1}). Q sup E Q [ ∥ x ∥ 2 I { ∥ x ∥ 2 > τ n }] ≤ c 1 1 { 2 n M 2 + e − ( c − c 1 ) τ n 2 E P e c ∥ x ∥ 2 2 } = O ( n − 1 ) .
Here e − ( c − c 1 ) log 2 ζ ( n + 1 ) = o ( n − k ) e^{-(c-c_1)\log^2\zeta(n+1)}=o(n^{-k}) e − ( c − c 1 ) l o g 2 ζ ( n + 1 ) = o ( n − k ) for every fixed k > 0 k>0 k > 0 . Let D n , i : = x i − x i ( n ) D_{n,i}:=x_i-x_i^{(n)} D n , i := x i − x i ( n ) . Since
∥ D n , i ∥ 2 ≤ ∥ x i ∥ 2 I { ∥ x i ∥ 2 > τ n } , \|D_{n,i}\|_2\le\|x_i\|_2\mathbb{I}\{\|x_i\|_2>\tau_n\}, ∥ D n , i ∥ 2 ≤ ∥ x i ∥ 2 I { ∥ x i ∥ 2 > τ n } ,
we have, uniformly in Q Q Q ,
E Q ∥ D n , 1 ∥ 2 ζ ≤ C n , ∥ E Q D n , 1 ∥ 2 ≤ C n τ n ζ − 1 . \mathbb{E}_{Q}\lVert D_{n,1}\rVert_{2}^{\zeta} \le\frac{C}{n}, \qquad\lVert\mathbb{E}_{Q}D_{n,1}\rVert_{2} \le\frac{C}{n\tau_n^{\zeta-1}}. E Q ∥ D n , 1 ∥ 2 ζ ≤ n C , ∥ E Q D n , 1 ∥ 2 ≤ n τ n ζ − 1 C .
For ζ ≥ 2 \zeta\ge2 ζ ≥ 2 we also have
E Q ∥ D n , 1 ∥ 2 2 ≤ E Q [ ∥ x ∥ 2 2 I { ∥ x ∥ 2 > τ n } ] ≤ C n τ n ζ − 2 . \mathbb{E}_{Q}\lVert D_{n,1}\rVert_{2}^{2} \le\mathbb{E}_{Q}\bigl[\lVert x\rVert_{2}^{2}\mathbb{I}\{\lVert x\rVert_{2}>\tau_n\}\bigr] \le\frac{C}{n\tau_n^{\zeta-2}}. E Q ∥ D n , 1 ∥ 2 2 ≤ E Q [ ∥ x ∥ 2 2 I {∥ x ∥ 2 > τ n } ] ≤ n τ n ζ − 2 C .
For independent centered real variables U 1 , … , U n U_1,\ldots,U_n U 1 , … , U n with finite ζ \zeta ζ th absolute moments, the von Bahr–Esseen inequality (von Bahr and Esseen, [38 ] ) gives, for 1 ≤ ζ ≤ 2 1\le\zeta\le2 1 ≤ ζ ≤ 2 ,
E ∣ ∑ i U i ∣ ζ ≤ 2 ∑ i E ∣ U i ∣ ζ . (161) \mathbb{E}\left|\sum_i U_i\right|^{\zeta}\le2\sum_i\mathbb{E}|U_i|^{\zeta}. \tag*{(161)} E i ∑ U i ζ ≤ 2 i ∑ E ∣ U i ∣ ζ . (161)
For ζ > 2 \zeta>2 ζ > 2 , Rosenthal’s inequality (Rosenthal, [30 ] ) gives
E ∣ ∑ i U i ∣ ζ ≤ C ζ { ( ∑ i E U i 2 ) ζ / 2 + ∑ i E ∣ U i ∣ ζ } . (162) \mathbb{E}\left|\sum_i U_i\right|^{\zeta}\le C_{\zeta}\left\{\left(\sum_i\mathbb{E}U_i^2\right)^{\zeta/2}+\sum_i\mathbb{E}|U_i|^{\zeta}\right\}. \tag*{(162)} E i ∑ U i ζ ≤ C ζ ⎩ ⎨ ⎧ ( i ∑ E U i 2 ) ζ /2 + i ∑ E ∣ U i ∣ ζ ⎭ ⎬ ⎫ . (162)
Applying these scalar inequalities coordinatewise gives the corresponding vector inequalities, with constants depending also on the fixed d d d . Note that D n , 1 , … , D n , n D_{n,1},\ldots,D_{n,n} D n , 1 , … , D n , n are i.i.d. under Q ⊗ n Q^{\otimes n} Q ⊗ n . Thus
Δ n = E Q D n , 1 + 1 n ∑ i = 1 n ( D n , i − E Q D n , 1 ) , \Delta_n=\mathbb{E}_{Q}D_{n,1}+\frac{1}{n}\sum_{i=1}^{n}(D_{n,i}-\mathbb{E}_{Q}D_{n,1}), Δ n = E Q D n , 1 + n 1 i = 1 ∑ n ( D n , i − E Q D n , 1 ) ,
where the items in the second term are independent and centered. For ζ ≥ 1 \zeta\ge1 ζ ≥ 1 , the inequality ∥ a + b ∥ 2 ζ ≤ 2 ζ − 1 ( ∥ a ∥ 2 ζ + ∥ b ∥ 2 ζ ) \lVert a+b\rVert_{2}^{\zeta}\le2^{\zeta-1}(\lVert a\rVert_{2}^{\zeta}+\lVert b\rVert_{2}^{\zeta}) ∥ a + b ∥ 2 ζ ≤ 2 ζ − 1 (∥ a ∥ 2 ζ + ∥ b ∥ 2 ζ ) gives
E Q ⊗ n ∥ Δ n ∥ 2 ζ ≤ 2 ζ − 1 ∥ E Q D n , 1 ∥ 2 ζ + 2 ζ − 1 n ζ E Q ⊗ n ∥ ∑ i = 1 n ( D n , i − E Q D n , 1 ) ∥ 2 ζ . \mathbb{E}_{Q^{\otimes n}}\lVert\Delta_n\rVert_{2}^{\zeta}\le2^{\zeta-1}\lVert\mathbb{E}_{Q}D_{n,1}\rVert_{2}^{\zeta}+\frac{2^{\zeta-1}}{n^{\zeta}}\mathbb{E}_{Q^{\otimes n}}\left\lVert\sum_{i=1}^{n}(D_{n,i}-\mathbb{E}_{Q}D_{n,1})\right\rVert_{2}^{\zeta}. E Q ⊗ n ∥ Δ n ∥ 2 ζ ≤ 2 ζ − 1 ∥ E Q D n , 1 ∥ 2 ζ + n ζ 2 ζ − 1 E Q ⊗ n i = 1 ∑ n ( D n , i − E Q D n , 1 ) 2 ζ .
Jensen’s inequality also gives
E Q ∥ D n , 1 − E Q D n , 1 ∥ 2 ζ ≤ 2 ζ − 1 ( E Q ∥ D n , 1 ∥ 2 ζ + ∥ E Q D n , 1 ∥ 2 ζ ) ≤ 2 ζ E Q ∥ D n , 1 ∥ 2 ζ . \mathbb{E}_{Q}\lVert D_{n,1}-\mathbb{E}_{Q}D_{n,1}\rVert_{2}^{\zeta}\le2^{\zeta-1}\left(\mathbb{E}_{Q}\lVert D_{n,1}\rVert_{2}^{\zeta}+\lVert\mathbb{E}_{Q}D_{n,1}\rVert_{2}^{\zeta}\right)\le2^{\zeta}\mathbb{E}_{Q}\lVert D_{n,1}\rVert_{2}^{\zeta}. E Q ∥ D n , 1 − E Q D n , 1 ∥ 2 ζ ≤ 2 ζ − 1 ( E Q ∥ D n , 1 ∥ 2 ζ + ∥ E Q D n , 1 ∥ 2 ζ ) ≤ 2 ζ E Q ∥ D n , 1 ∥ 2 ζ .
For ζ > 2 \zeta>2 ζ > 2 , the second moment in Rosenthal’s inequality satisfies
E Q ∥ D n , 1 − E Q D n , 1 ∥ 2 2 = E Q ∥ D n , 1 ∥ 2 2 − ∥ E Q D n , 1 ∥ 2 2 ≤ E Q ∥ D n , 1 ∥ 2 2 . \mathbb{E}_{Q}\lVert D_{n,1}-\mathbb{E}_{Q}D_{n,1}\rVert_{2}^{2}=\mathbb{E}_{Q}\lVert D_{n,1}\rVert_{2}^{2}-\lVert\mathbb{E}_{Q}D_{n,1}\rVert_{2}^{2}\le\mathbb{E}_{Q}\lVert D_{n,1}\rVert_{2}^{2}. E Q ∥ D n , 1 − E Q D n , 1 ∥ 2 2 = E Q ∥ D n , 1 ∥ 2 2 − ∥ E Q D n , 1 ∥ 2 2 ≤ E Q ∥ D n , 1 ∥ 2 2 .
Applying (G.13) and absorbing these factors into C ζ C_{\zeta} C ζ yields, for 1 ≤ ζ ≤ 2 1\le\zeta\le2 1 ≤ ζ ≤ 2 ,
E ∥ Δ n ∥ 2 ζ ≤ C ζ { ∥ E Q D n , 1 ∥ 2 ζ + n 1 − ζ E Q ∥ D n , 1 ∥ 2 ζ } = O ( n − ζ ) . \mathbb{E}\lVert\Delta_n\rVert_{2}^{\zeta}\le C_{\zeta}\left\{\lVert\mathbb{E}_{Q}D_{n,1}\rVert_{2}^{\zeta}+n^{1-\zeta}\mathbb{E}_{Q}\lVert D_{n,1}\rVert_{2}^{\zeta}\right\}=O(n^{-\zeta}). E ∥ Δ n ∥ 2 ζ ≤ C ζ { ∥ E Q D n , 1 ∥ 2 ζ + n 1 − ζ E Q ∥ D n , 1 ∥ 2 ζ } = O ( n − ζ ) .
For ζ > 2 \zeta>2 ζ > 2 ,
E ∥ Δ n ∥ 2 ζ ≤ C ζ ∥ E Q D n , 1 ∥ 2 ζ + C ζ n 1 − ζ E Q ∥ D n , 1 ∥ 2 ζ + C ζ n − ζ / 2 ( E Q ∥ D n , 1 ∥ 2 2 ) ζ / 2 = O ( n − ζ ) . \mathbb{E}\lVert\Delta_n\rVert_{2}^{\zeta}\le C_{\zeta}\lVert\mathbb{E}_{Q}D_{n,1}\rVert_{2}^{\zeta}+C_{\zeta}n^{1-\zeta}\mathbb{E}_{Q}\lVert D_{n,1}\rVert_{2}^{\zeta}+C_{\zeta}n^{-\zeta/2}\left(\mathbb{E}_{Q}\lVert D_{n,1}\rVert_{2}^{2}\right)^{\zeta/2}=O(n^{-\zeta}). E ∥ Δ n ∥ 2 ζ ≤ C ζ ∥ E Q D n , 1 ∥ 2 ζ + C ζ n 1 − ζ E Q ∥ D n , 1 ∥ 2 ζ + C ζ n − ζ /2 ( E Q ∥ D n , 1 ∥ 2 2 ) ζ /2 = O ( n − ζ ) .
All these estimates are uniform in Q Q Q , proving sup Q E Q ⊗ n ∥ Δ n ∥ 2 ζ = O ( n − ζ ) \sup_Q \mathbb{E}_{Q^{\otimes n}}\|\Delta_n\|_2^\zeta=O(n^{-\zeta}) sup Q E Q ⊗ n ∥ Δ n ∥ 2 ζ = O ( n − ζ ) .
The uniform local expansion on B 2 ( 1 ) B_2(1) B 2 ( 1 ) gives, for some r > 0 r>0 r > 0 ,
sup ∥ v ∥ 2 ≤ 1 r μ ( ε v ) ε α ≤ 1 + sup B 2 ( 1 ) ψ < ∞ , 0 < ε ≤ r . \sup_{\|v\|_2\leq1}\frac{r_\mu(\varepsilon v)}{\varepsilon^\alpha}\leq1+\sup_{B_2(1)}\psi<\infty,\qquad0<\varepsilon\leq r. ∥ v ∥ 2 ≤ 1 sup ε α r μ ( ε v ) ≤ 1 + B 2 ( 1 ) sup ψ < ∞ , 0 < ε ≤ r .
For 0 < ∥ h ∥ 2 ≤ r 0<\|h\|_2\leq r 0 < ∥ h ∥ 2 ≤ r , take ε = ∥ h ∥ 2 \varepsilon=\|h\|_2 ε = ∥ h ∥ 2 and v = h / ∥ h ∥ 2 v=h/\|h\|_2 v = h /∥ h ∥ 2 to obtain r μ ( h ) ≤ C ∥ h ∥ 2 α r_\mu(h)\leq C\|h\|_2^\alpha r μ ( h ) ≤ C ∥ h ∥ 2 α . For s ∈ [ 1 , α ] s\in[1,\alpha] s ∈ [ 1 , α ] , use ∥ h ∥ 2 α ≤ r α − s ∥ h ∥ 2 s \|h\|_2^\alpha\leq r^{\alpha-s}\|h\|_2^s ∥ h ∥ 2 α ≤ r α − s ∥ h ∥ 2 s on ∥ h ∥ 2 ≤ r \|h\|_2\leq r ∥ h ∥ 2 ≤ r , and ∥ h ∥ 2 ≤ r 1 − s ∥ h ∥ 2 s \|h\|_2\leq r^{1-s}\|h\|_2^s ∥ h ∥ 2 ≤ r 1 − s ∥ h ∥ 2 s on ∥ h ∥ 2 > r \|h\|_2>r ∥ h ∥ 2 > r . Together with the global bound r μ ( h ) ≤ 2 R Ω ∥ h ∥ 2 r_\mu(h)\leq2R_\Omega\|h\|_2 r μ ( h ) ≤ 2 R Ω ∥ h ∥ 2 , this yields
0 ≤ r μ ( h ) ≤ C s ∥ h ∥ 2 s , h ∈ R d . 0\leq r_\mu(h)\leq C_s\|h\|_2^s,\qquad h\in\mathbb{R}^d. 0 ≤ r μ ( h ) ≤ C s ∥ h ∥ 2 s , h ∈ R d .
Taking s = min { α , ζ } s=\min\{\alpha,\zeta\} s = min { α , ζ } and applying sup Q E Q ⊗ n ∥ Δ n ∥ 2 ζ = O ( n − ζ ) \sup_Q\mathbb{E}_{Q^{\otimes n}}\|\Delta_n\|_2^\zeta=O(n^{-\zeta}) sup Q E Q ⊗ n ∥ Δ n ∥ 2 ζ = O ( n − ζ ) gives
sup Q E r μ ( 4 Δ n ) ≤ C ( sup Q E ∥ Δ n ∥ 2 ζ ) min { α , ζ } / ζ = O ( n − min { α , ζ } ) . (163) \sup_Q\mathbb{E}r_\mu(4\Delta_n)\leq C\left(\sup_Q\mathbb{E}\|\Delta_n\|_2^\zeta\right)^{\min\{\alpha,\zeta\}/\zeta}=O(n^{-\min\{\alpha,\zeta\}}). \tag*{(163)} Q sup E r μ ( 4 Δ n ) ≤ C ( Q sup E ∥ Δ n ∥ 2 ζ ) m i n { α , ζ } / ζ = O ( n − m i n { α , ζ } ) . (163)
By Proposition 6.2,
W n , M m e a n ( P ) ≤ C { n − α / 2 + n − min { α , ζ } } ≤ C ′ { n − α / 2 + n − ζ } . W^{\mathrm{mean}}_{n,M}(P)\leq C\{n^{-\alpha/2}+n^{-\min\{\alpha,\zeta\}}\}\leq C'\{n^{-\alpha/2}+n^{-\zeta}\}. W n , M mean ( P ) ≤ C { n − α /2 + n − m i n { α , ζ } } ≤ C ′ { n − α /2 + n − ζ } .
If α ≤ 2 ζ \alpha\leq2\zeta α ≤ 2 ζ , then min { α , ζ } ≥ α / 2 \min\{\alpha,\zeta\}\geq\alpha/2 min { α , ζ } ≥ α /2 , and (57) holds. If α < 2 ζ \alpha<2\zeta α < 2 ζ , then min { α , ζ } > α / 2 \min\{\alpha,\zeta\}>\alpha/2 min { α , ζ } > α /2 , and (58) holds. The strict rate separation for α > 2 ζ \alpha>2\zeta α > 2 ζ is verified in Example 6.5.
Proof of Corollary 6.4
Proof. Choose r > 0 r>0 r > 0 so that f ∗ f_* f ∗ is C 2 C^2 C 2 near μ + B 2 ( r ) \mu+B_2(r) μ + B 2 ( r ) , and let C r C_r C r bound the operator norm of its Hessian there. All suprema over Q Q Q in this proof are over Q n , M K L ( P ) Q^{\mathrm{KL}}_{n,M}(P) Q n , M KL ( P ) . (G.7) gives
sup Q E Q ⊗ n ∥ x ~ − μ ~ n ∥ 2 = O ( n − 1 ) . \sup_Q\mathbb{E}_{Q^{\otimes n}}\|\widetilde{x}-\widetilde{\mu}_n\|_2=O(n^{-1}). Q sup E Q ⊗ n ∥ x − μ n ∥ 2 = O ( n − 1 ) .
Recall Z n , Q = n ( μ ~ n − μ Q ) Z_{n,Q}=\sqrt{n}(\widetilde{\mu}_n-\mu_Q) Z n , Q = n ( μ n − μ Q ) from (47) . Taking moment order q = 2 q=2 q = 2 in (F.6) gives
sup Q E Q ⊗ n ∥ μ ~ n − μ Q ∥ 2 2 = 1 n sup Q E Q ⊗ n ∥ Z n , Q ∥ 2 2 = O ( n − 1 ) . \sup_Q\mathbb{E}_{Q^{\otimes n}}\|\widetilde{\mu}_n-\mu_Q\|_2^2=\frac{1}{n}\sup_Q\mathbb{E}_{Q^{\otimes n}}\|Z_{n,Q}\|_2^2=O(n^{-1}). Q sup E Q ⊗ n ∥ μ n − μ Q ∥ 2 2 = n 1 Q sup E Q ⊗ n ∥ Z n , Q ∥ 2 2 = O ( n − 1 ) .
By (F.2), sup Q ∥ μ Q − μ ∥ 2 = O ( n − 1 / 2 ) \sup_Q\|\mu_Q-\mu\|_2=O(n^{-1/2}) sup Q ∥ μ Q − μ ∥ 2 = O ( n − 1/2 ) , so for sufficiently large n n n , sup Q ∥ μ Q − μ ∥ 2 ≤ r / 8 \sup_Q\|\mu_Q-\mu\|_2\leq r/8 sup Q ∥ μ Q − μ ∥ 2 ≤ r /8 . Consequently, for every such n n n and every Q Q Q in the neighborhood,
{ ∥ μ ~ n − μ ∥ 2 > r / 4 } ⊆ { ∥ μ ~ n − μ Q ∥ 2 > r / 8 } = { ∥ Z n , Q ∥ 2 > r n / 8 } . \{\|\widetilde{\mu}_n-\mu\|_2>r/4\}\subseteq\{\|\widetilde{\mu}_n-\mu_Q\|_2>r/8\}=\{\|Z_{n,Q}\|_2>r\sqrt{n}/8\}. { ∥ μ n − μ ∥ 2 > r /4 } ⊆ { ∥ μ n − μ Q ∥ 2 > r /8 } = { ∥ Z n , Q ∥ 2 > r n /8 } .
Markov’s inequality and (F.6) with moment order q = 4 q=4 q = 4 therefore imply
sup Q Q ⊗ n { ∥ μ ~ n − μ ∥ 2 > r / 4 } ≤ sup Q Q ⊗ n { ∥ Z n , Q ∥ 2 > r n / 8 } ≤ 8 4 r 4 n 2 sup Q E Q ⊗ n ∥ Z n , Q ∥ 2 4 = O ( n − 2 ) , \begin{aligned}
\sup_Q Q^{\otimes n}\{\|\widetilde{\mu}_n-\mu\|_2>r/4\}\leq\sup_Q Q^{\otimes n}\{\|Z_{n,Q}\|_2>r\sqrt{n}/8\} \\
&\leq\frac{8^4}{r^4n^2}\sup_Q\mathbb{E}_{Q^{\otimes n}}\|Z_{n,Q}\|_2^4=O(n^{-2}),
\end{aligned} Q sup Q ⊗ n { ∥ μ n − μ ∥ 2 > r /4 } ≤ Q sup Q ⊗ n { ∥ Z n , Q ∥ 2 > r n /8 } ≤ r 4 n 2 8 4 Q sup E Q ⊗ n ∥ Z n , Q ∥ 2 4 = O ( n − 2 ) ,
where the implied constant may depend on the fixed r r r .
On the event E n : = { ∥ μ ~ n − μ ∥ 2 ≤ r / 4 , ∥ x ˉ − μ ~ n ∥ 2 ≤ r / 4 } E_n := \{\|\tilde{\mu}_n-\mu\|_2 \le r/4,\ \|\bar{x}-\tilde{\mu}_n\|_2 \le r/4\} E n := { ∥ μ ~ n − μ ∥ 2 ≤ r /4 , ∥ x ˉ − μ ~ n ∥ 2 ≤ r /4 } , we have
∥ x ˉ − μ ∥ 2 ≤ ∥ x ˉ − μ ~ n ∥ 2 + ∥ μ ~ n − μ ∥ 2 ≤ r / 2 < r , ∥ μ Q − μ ∥ 2 ≤ r / 8 < r . \|\bar{x}-\mu\|_2 \le\|\bar{x}-\tilde{\mu}_n\|_2 + \|\tilde{\mu}_n-\mu\|_2 \le r/2 < r,\qquad\|\mu_Q-\mu\|_2 \le r/8 < r. ∥ x ˉ − μ ∥ 2 ≤ ∥ x ˉ − μ ~ n ∥ 2 + ∥ μ ~ n − μ ∥ 2 ≤ r /2 < r , ∥ μ Q − μ ∥ 2 ≤ r /8 < r .
Thus x ˉ , μ Q ∈ μ + B 2 ( r ) \bar{x},\mu_Q \in\mu+B_2(r) x ˉ , μ Q ∈ μ + B 2 ( r ) , and convexity of the ball gives
( 1 − s ) x ˉ + s μ Q ∈ μ + B 2 ( r ) , 0 ≤ s ≤ 1. (1-s)\bar{x}+s\mu_Q \in\mu+B_2(r),\qquad0\le s\le1. ( 1 − s ) x ˉ + s μ Q ∈ μ + B 2 ( r ) , 0 ≤ s ≤ 1.
Taylor’s formula implies
R Q ( w ∗ ( x ˉ ) ) = f ∗ ( μ Q ) − f ∗ ( x ˉ ) − ∇ f ∗ ( x ˉ ) T ( μ Q − x ˉ ) = ∫ 0 1 ( 1 − s ) ( μ Q − x ˉ ) T ∇ 2 f ∗ ( x ˉ + s ( μ Q − x ˉ ) ) ( μ Q − x ˉ ) d s ≤ C r 2 ∥ x ˉ − μ Q ∥ 2 2 ≤ C r ∥ μ ~ n − μ Q ∥ 2 2 + C r ∥ x ˉ − μ ~ n ∥ 2 2 ≤ C r ∥ μ ~ n − μ Q ∥ 2 2 + C r r 4 ∥ x ˉ − μ ~ n ∥ 2 . \begin{aligned}
R_Q(w_*(\bar{x})) &= f_*(\mu_Q)-f_*(\bar{x})-\nabla f_*(\bar{x})^\mathsf{T}(\mu_Q-\bar{x}) \\
&= \int_0^1 (1-s)(\mu_Q-\bar{x})^\mathsf{T}\nabla^2f_*(\bar{x}+s(\mu_Q-\bar{x}))(\mu_Q-\bar{x})\,\mathrm{d}s \\
&\le\frac{C_r}{2}\|\bar{x}-\mu_Q\|_2^2 \\
&\le C_r\|\tilde{\mu}_n-\mu_Q\|_2^2+C_r\|\bar{x}-\tilde{\mu}_n\|_2^2 \\
&\le C_r\|\tilde{\mu}_n-\mu_Q\|_2^2+\frac{C_r r}{4}\|\bar{x}-\tilde{\mu}_n\|_2.
\end{aligned} R Q ( w ∗ ( x ˉ )) = f ∗ ( μ Q ) − f ∗ ( x ˉ ) − ∇ f ∗ ( x ˉ ) T ( μ Q − x ˉ ) = ∫ 0 1 ( 1 − s ) ( μ Q − x ˉ ) T ∇ 2 f ∗ ( x ˉ + s ( μ Q − x ˉ )) ( μ Q − x ˉ ) d s ≤ 2 C r ∥ x ˉ − μ Q ∥ 2 2 ≤ C r ∥ μ ~ n − μ Q ∥ 2 2 + C r ∥ x ˉ − μ ~ n ∥ 2 2 ≤ C r ∥ μ ~ n − μ Q ∥ 2 2 + 4 C r r ∥ x ˉ − μ ~ n ∥ 2 .
On the complement, compactness gives R Q ( w ∗ ( x ˉ ) ) ≤ 2 R Ω ∥ μ Q ∥ 2 R_Q(w_*(\bar{x}))\le2R_\Omega\|\mu_Q\|_2 R Q ( w ∗ ( x ˉ )) ≤ 2 R Ω ∥ μ Q ∥ 2 , and Markov’s inequality gives
sup Q Q ⊗ n ( E n c ) ≤ O ( n − 2 ) + 4 r sup Q E Q ⊗ n ∥ x ˉ − μ ~ n ∥ 2 = O ( n − 1 ) . \sup_Q Q^{\otimes n}(E_n^c)\le O(n^{-2})+\frac{4}{r}\sup_Q\mathbb{E}_{Q^{\otimes n}}\|\bar{x}-\tilde{\mu}_n\|_2=O(n^{-1}). Q sup Q ⊗ n ( E n c ) ≤ O ( n − 2 ) + r 4 Q sup E Q ⊗ n ∥ x ˉ − μ ~ n ∥ 2 = O ( n − 1 ) .
Taking expectations proves (62) . If H μ ≠ 0 H_\mu\ne0 H μ = 0 , then Σ ≻ 0 \Sigma\succ0 Σ ≻ 0 and H μ ⪰ 0 H_\mu\succeq0 H μ ⪰ 0 imply t ↦ t T Σ 1 / 2 H μ Σ 1 / 2 t / 2 t\mapsto t^\mathsf{T}\Sigma^{1/2}H_\mu\Sigma^{1/2}t/2 t ↦ t T Σ 1/2 H μ Σ 1/2 t /2 is nonaffine on every B 2 ( M ) B_2(M) B 2 ( M ) , M > 0 M>0 M > 0 . Hence (51) has a positive finite fixed-M M M constant, and
R n , M K L ( P ) ≍ n − 1 ≍ sup Q ∈ Q n , M K L ( P ) E Q ⊗ n R Q ( w ∗ ( x ˉ ) ) . \mathcal{R}^{\mathrm{KL}}_{n,M}(P)\asymp n^{-1}\asymp\sup_{Q\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P)}\mathbb{E}_{Q^{\otimes n}}R_Q(w_*(\bar{x})). R n , M KL ( P ) ≍ n − 1 ≍ Q ∈ Q n , M KL ( P ) sup E Q ⊗ n R Q ( w ∗ ( x ˉ )) .
Detailed construction for Example 6.5
Fix ζ ≥ 1 \zeta\ge1 ζ ≥ 1 and α > 1 \alpha>1 α > 1 . Let x 1 x_1 x 1 and x 2 x_2 x 2 be independent under P P P , with
P { x 1 = 1 / 2 } = P { x 1 = 3 / 2 } = 1 2 , p x 2 ( z ) = e − ∣ z ∣ ζ ∫ R e − ∣ u ∣ ζ d u . (164) \mathbb{P}\{x_1=1/2\}=\mathbb{P}\{x_1=3/2\}=\frac{1}{2},\qquad p_{x_2}(z)=\frac{e^{-|z|^\zeta}}{\int_{\mathbb{R}}e^{-|u|^\zeta}\,\mathrm{d}u}. \tag*{(164)} P { x 1 = 1/2 } = P { x 1 = 3/2 } = 2 1 , p x 2 ( z ) = ∫ R e − ∣ u ∣ ζ d u e − ∣ z ∣ ζ . (164)
Then μ = e 1 \mu=e_1 μ = e 1 , Σ ≻ 0 \Sigma\succ0 Σ ≻ 0 , and (60) holds for a sufficiently small c > 0 c>0 c > 0 . Take Ω = B α / ( α − 1 ) 2 \Omega=B_{\alpha/(\alpha-1)}^2 Ω = B α / ( α − 1 ) 2 , so f ∗ ( z ) = ∥ z ∥ α f_*(z)=\|z\|_\alpha f ∗ ( z ) = ∥ z ∥ α . For bounded t t t and sufficiently small ε > 0 \varepsilon>0 ε > 0 ,
r e 1 ( ε t ) = ( 1 + ε t 1 ) { ( 1 + ∣ ε t 2 1 + ε t 1 ∣ α ) 1 / α − 1 } = ε α ∣ t 2 ∣ α α + o ( ε α ) . \begin{aligned}
r_{e_1}(\varepsilon t)
&=(1+\varepsilon t_1)\left\{\left(1+\left|\frac{\varepsilon t_2}{1+\varepsilon t_1}\right|^\alpha\right)^{1/\alpha}-1\right\}\\
&=\frac{\varepsilon^\alpha|t_2|^\alpha}{\alpha}+o(\varepsilon^\alpha).
\end{aligned} r e 1 ( εt ) = ( 1 + ε t 1 ) { ( 1 + 1 + ε t 1 ε t 2 α ) 1/ α − 1 } = α ε α ∣ t 2 ∣ α + o ( ε α ) .
uniformly on every fixed ball. Thus ψ ( t ) = ∣ t 2 ∣ α / α \psi(t)=|t_2|^\alpha/\alpha ψ ( t ) = ∣ t 2 ∣ α / α , a ( ε ) = ε α a(\varepsilon)=\varepsilon^\alpha a ( ε ) = ε α , and Theorem 5.2 gives, for every M > 0 M>0 M > 0 ,
R n , M K L ( P ) ≲ W n , M t r ( P ) ≲ n − α / 2 . \mathcal{R}^{\mathrm{KL}}_{n,M}(P)\lesssim W^{\mathrm{tr}}_{n,M}(P)\lesssim n^{-\alpha/2}. R n , M KL ( P ) ≲ W n , M tr ( P ) ≲ n − α /2 .
For this family, we claim that the bound in Corollary 6.3 is sharp in rate:
W n , M m e a n ( P ) ≲ n − α / 2 + n − ζ . (165) W^{\mathrm{mean}}_{n,M}(P)\lesssim n^{-\alpha/2}+n^{-\zeta}. \tag*{(165)} W n , M mean ( P ) ≲ n − α /2 + n − ζ . (165)
To prove the additional n − ζ n^{-\zeta} n − ζ lower bound, set
B n = { n ≤ ∣ x 2 ∣ ≤ n + 1 } , ε n = M 2 4 n ζ + 1 , Q n = ( 1 − ε n ) P + ε n P ( ⋅ ∣ B n ) . (166) B_n=\{n\le|x_2|\le n+1\},\qquad\varepsilon_n=\frac{M^2}{4n^{\zeta+1}},\qquad Q_n=(1-\varepsilon_n)P+\varepsilon_nP(\,\cdot\mid B_n). \tag*{(166)} B n = { n ≤ ∣ x 2 ∣ ≤ n + 1 } , ε n = 4 n ζ + 1 M 2 , Q n = ( 1 − ε n ) P + ε n P ( ⋅ ∣ B n ) . (166)
For sufficiently large n n n , 0 < ε n < 1 0<\varepsilon_n<1 0 < ε n < 1 and 0 < P ( B n ) < 1 0<P(B_n)<1 0 < P ( B n ) < 1 . The conditional law satisfies
d P ( ⋅ ∣ B n ) d P = I B n P ( B n ) , d Q n d P = 1 − ε n + ε n P ( B n ) I B n . \frac{\mathrm{d}P(\,\cdot\mid B_n)}{\mathrm{d}P}=\frac{\mathbb{I}_{B_n}}{P(B_n)},\qquad\frac{\mathrm{d}Q_n}{\mathrm{d}P}=1-\varepsilon_n+\frac{\varepsilon_n}{P(B_n)}\mathbb{I}_{B_n}. d P d P ( ⋅ ∣ B n ) = P ( B n ) I B n , d P d Q n = 1 − ε n + P ( B n ) ε n I B n .
Thus Q n ≪ P Q_n\ll P Q n ≪ P , and by convexity of u ↦ u log u u\mapsto u\log u u ↦ u log u on [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) , with 0 log 0 = 0 0\log0=0 0 log 0 = 0 ,
( 1 − ε n + ε n I B n P ( B n ) ) log ( 1 − ε n + ε n I B n P ( B n ) ) ≤ 1 log 1 + ε n I B n P ( B n ) log ( I B n P ( B n ) ) . \left(1-\varepsilon_n+\frac{\varepsilon_n\mathbb{I}_{B_n}}{P(B_n)}\right)\log\left(1-\varepsilon_n+\frac{\varepsilon_n\mathbb{I}_{B_n}}{P(B_n)}\right)\le1\log1+\varepsilon_n\frac{\mathbb{I}_{B_n}}{P(B_n)}\log\left(\frac{\mathbb{I}_{B_n}}{P(B_n)}\right). ( 1 − ε n + P ( B n ) ε n I B n ) log ( 1 − ε n + P ( B n ) ε n I B n ) ≤ 1 log 1 + ε n P ( B n ) I B n log ( P ( B n ) I B n ) .
Integrating with respect to P P P yields
D K L ( Q n ∥ P ) ≤ ε n ∫ B n 1 P ( B n ) log 1 P ( B n ) d P = ε n log 1 P ( B n ) . D_{\mathrm{KL}}(Q_n\mathbin\Vert P)\le\varepsilon_n\int_{B_n}\frac{1}{P(B_n)}\log\frac{1}{P(B_n)}\,\mathrm{d}P=\varepsilon_n\log\frac{1}{P(B_n)}. D KL ( Q n ∥ P ) ≤ ε n ∫ B n P ( B n ) 1 log P ( B n ) 1 d P = ε n log P ( B n ) 1 .
Let
c ζ : = log ( 1 2 ∫ R e − ∣ u ∣ ζ d u ) . c_\zeta:=\log\left(\frac{1}{2}\int_{\mathbb{R}}e^{-|u|^\zeta}\,\mathrm{d}u\right). c ζ := log ( 2 1 ∫ R e − ∣ u ∣ ζ d u ) .
Then
n ζ + c ζ ≤ log 1 P ( B n ) ≤ ( n + 1 ) ζ + c ζ , n^\zeta+c_\zeta\le\log\frac{1}{P(B_n)}\le(n+1)^\zeta+c_\zeta, n ζ + c ζ ≤ log P ( B n ) 1 ≤ ( n + 1 ) ζ + c ζ ,
i.e.,
log 1 P ( B n ) ∼ n ζ . \log\frac{1}{P(B_n)}\sim n^\zeta. log P ( B n ) 1 ∼ n ζ .
Consequently,
D K L ( Q n ∥ P ) ≤ ε n log 1 P ( B n ) ∼ M 2 4 n < M 2 2 n . (167) D_{\mathrm{KL}}(Q_n\mathbin\Vert P)\le\varepsilon_n\log\frac{1}{P(B_n)}\sim\frac{M^2}{4n}<\frac{M^2}{2n}. \tag*{(167)} D KL ( Q n ∥ P ) ≤ ε n log P ( B n ) 1 ∼ 4 n M 2 < 2 n M 2 . (167)
By symmetry in x 2 x_2 x 2 and independence of x 1 x_1 x 1 , note that μ Q n = e 1 \mu_{Q_n}=e_1 μ Q n = e 1 . Since E P ∥ x ∥ 2 < ∞ \mathbb{E}_P\lVert x\rVert_2<\infty E P ∥ x ∥ 2 < ∞ and ∥ x ∥ 2 ≤ ( 3 / 2 ) 2 + ( n + 1 ) 2 \lVert x\rVert_2\le\sqrt{(3/2)^2+(n+1)^2} ∥ x ∥ 2 ≤ ( 3/2 ) 2 + ( n + 1 ) 2 on B n B_n B n , we also have E Q n ∥ x ∥ 2 < ∞ \mathbb{E}_{Q_n}\lVert x\rVert_2<\infty E Q n ∥ x ∥ 2 < ∞ . Hence Q n ∈ Q n , M K L ( P ) Q_n\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P) Q n ∈ Q n , M KL ( P ) for all sufficiently large n n n .
In the mixture representation, each observation independently selects the component P P P with probability 1 − ε n 1-\varepsilon_n 1 − ε n or P ( ⋅ ∣ B n ) P(\,\cdot\mid B_n) P ( ⋅ ∣ B n ) with probability ε n \varepsilon_n ε n , and is then drawn from the selected component. Let N n N_n N n denote the number selecting P ( ⋅ ∣ B n ) P(\,\cdot\mid B_n) P ( ⋅ ∣ B n ) . Then
N n ∼ Bin ( n , ε n ) , P { N n = 1 } = n ε n ( 1 − ε n ) n − 1 ∼ M 2 4 n ζ . (168) N_n\sim\operatorname{Bin}(n,\varepsilon_n),\qquad\mathbb{P}\{N_n=1\}=n\varepsilon_n(1-\varepsilon_n)^{n-1}\sim\frac{M^2}{4n^\zeta}. \tag*{(168)} N n ∼ Bin ( n , ε n ) , P { N n = 1 } = n ε n ( 1 − ε n ) n − 1 ∼ 4 n ζ M 2 . (168)
since n ε n → 0 n\varepsilon_n \to0 n ε n → 0 . Conditional on N n = 1 N_n = 1 N n = 1 , the exceptional index is the unique sample index whose selected component is P ( ⋅ ∣ B n ) P(\cdot\mid B_n) P ( ⋅ ∣ B n ) . It is uniformly distributed over { 1 , … , n } \{1,\ldots,n\} { 1 , … , n } . Since the sample mean is invariant under permutations, its conditional law is the same for every exceptional index. It therefore suffices to calculate with exceptional index 1. By independence, centering, and Chebyshev’s inequality, for every h > 0 h > 0 h > 0 ,
E [ 1 n ∑ i = 2 n x i , 2 ∣ N n = 1 , the exceptional index is 1 ] = 0 , \mathbb{E}\left[\left.\frac{1}{n}\sum_{i=2}^{n}x_{i,2}\,\right|\,N_n=1,\text{ the exceptional index is }1\right]=0, E [ n 1 i = 2 ∑ n x i , 2 N n = 1 , the exceptional index is 1 ] = 0 ,
Var ( 1 n ∑ i = 2 n x i , 2 ∣ N n = 1 , the exceptional index is 1 ) = n − 1 n 2 Var P ( x 2 ) , \operatorname{Var}\left(\left.\frac{1}{n}\sum_{i=2}^{n}x_{i,2}\,\right|\,N_n=1,\text{ the exceptional index is }1\right)=\frac{n-1}{n^2}\operatorname{Var}_{P}(x_2), Var ( n 1 i = 2 ∑ n x i , 2 N n = 1 , the exceptional index is 1 ) = n 2 n − 1 Var P ( x 2 ) ,
P { ∣ 1 n ∑ i = 2 n x i , 2 ∣ > h ∣ N n = 1 , the exceptional index is 1 } ≤ ( n − 1 ) Var P ( x 2 ) n 2 h 2 → 0. \mathbb{P}\left\{\left.\left|\frac{1}{n}\sum_{i=2}^{n}x_{i,2}\right|>h\,\right|\,N_n=1,\text{ the exceptional index is }1\right\}\leq\frac{(n-1)\operatorname{Var}_{P}(x_2)}{n^2h^2}\to0. P { n 1 i = 2 ∑ n x i , 2 > h N n = 1 , the exceptional index is 1 } ≤ n 2 h 2 ( n − 1 ) Var P ( x 2 ) → 0.
On this conditional event, ∣ x 1 , 2 ∣ / n ∈ [ 1 , 1 + 1 / n ] |x_{1,2}|/n\in[1,1+1/n] ∣ x 1 , 2 ∣/ n ∈ [ 1 , 1 + 1/ n ] , and the triangle inequality gives
∣ ∣ x ˉ 2 ∣ − 1 ∣ ≤ ∣ ∣ x ˉ 2 ∣ − ∣ x 1 , 2 ∣ n ∣ + ∣ ∣ x 1 , 2 ∣ n − 1 ∣ ≤ ∣ 1 n ∑ i = 2 n x i , 2 ∣ + 1 n . \left|\left|\bar{x}_2\right|-1\right|\leq\left|\left|\bar{x}_2\right|-\frac{|x_{1,2}|}{n}\right|+\left|\frac{|x_{1,2}|}{n}-1\right|\leq\left|\frac{1}{n}\sum_{i=2}^{n}x_{i,2}\right|+\frac{1}{n}. ∣ ∣ x ˉ 2 ∣ − 1 ∣ ≤ ∣ x ˉ 2 ∣ − n ∣ x 1 , 2 ∣ + n ∣ x 1 , 2 ∣ − 1 ≤ n 1 i = 2 ∑ n x i , 2 + n 1 .
Thus ∣ x ˉ 2 ∣ → 1 |\bar{x}_2|\to1 ∣ x ˉ 2 ∣ → 1 in conditional probability, also when conditioning only on N n = 1 N_n=1 N n = 1 by the permutation invariance above. Independence of the two coordinates under P P P and the definition of B n B_n B n give P { x 1 = 1 / 2 ∣ B n } = P { x 1 = 3 / 2 ∣ B n } = 1 2 \mathbb{P}\{x_1=1/2\mid B_n\}=\mathbb{P}\{x_1=3/2\mid B_n\}=\frac{1}{2} P { x 1 = 1/2 ∣ B n } = P { x 1 = 3/2 ∣ B n } = 2 1 . Consequently, for every selection of mixture components, the first coordinates are independent with the same mean 1 and variance 1 / 4 1/4 1/4 . Hence
E [ x ˉ 1 ∣ N n = 1 ] = 1 , Var ( x ˉ 1 ∣ N n = 1 ) = 1 4 n , P { ∣ x ˉ 1 − 1 ∣ > h ∣ N n = 1 } ≤ 1 4 n h 2 → 0. \mathbb{E}[\bar{x}_1\mid N_n=1]=1,\qquad\operatorname{Var}(\bar{x}_1\mid N_n=1)=\frac{1}{4n},\qquad\mathbb{P}\{|\bar{x}_1-1|>h\mid N_n=1\}\leq\frac{1}{4nh^2}\to0. E [ x ˉ 1 ∣ N n = 1 ] = 1 , Var ( x ˉ 1 ∣ N n = 1 ) = 4 n 1 , P { ∣ x ˉ 1 − 1∣ > h ∣ N n = 1 } ≤ 4 n h 2 1 → 0.
We have proved
x ˉ 1 → 1 , ∣ x ˉ 2 ∣ → 1 in probability conditional on N n = 1. (169) \bar{x}_1\to1,\qquad|\bar{x}_2|\to1\quad\text{in probability conditional on }N_n=1. \tag*{(169)} x ˉ 1 → 1 , ∣ x ˉ 2 ∣ → 1 in probability conditional on N n = 1. (169)
For z ≠ 0 z\ne0 z = 0 , the maximizer over Ω = B α / ( α − 1 ) 2 \Omega=B_{\alpha/(\alpha-1)}^2 Ω = B α / ( α − 1 ) 2 is
w ⋆ , j ( z ) = sgn ( z j ) ∣ z j ∣ α − 1 ∥ z ∥ α − 1 α − 1 , j = 1 , 2. w_{\star,j}(z)=\frac{\operatorname{sgn}(z_j)|z_j|^{\alpha-1}}{\lVert z\rVert_{\alpha-1}^{\alpha-1}},\qquad j=1,2. w ⋆ , j ( z ) = ∥ z ∥ α − 1 α − 1 sgn ( z j ) ∣ z j ∣ α − 1 , j = 1 , 2.
Since μ Q n = e 1 \mu Q_n=e_1 μ Q n = e 1 and f ⋆ ( e 1 ) = 1 f_{\star}(e_1)=1 f ⋆ ( e 1 ) = 1 , for z 1 > 0 z_1>0 z 1 > 0 ,
R Q n ( w ⋆ ( z ) ) = f ⋆ ( e 1 ) − w ⋆ ( z ) T e 1 = 1 − z 1 α − 1 ( z 1 α + ∣ z 2 ∣ α ) ( α − 1 ) / α = 1 − ( 1 + ∣ z 2 / z 1 ∣ α ) − ( α − 1 ) / α . (170) \begin{aligned}
R_{Q_n}(w_{\star}(z))&=f_{\star}(e_1)-w_{\star}(z)^\mathsf{T}e_1\\
&=1-\frac{z_1^{\alpha-1}}{(z_1^\alpha+|z_2|^\alpha)^{(\alpha-1)/\alpha}}
=1-\left(1+|z_2/z_1|^\alpha\right)^{-(\alpha-1)/\alpha}. \tag*{(170)}
\end{aligned} R Q n ( w ⋆ ( z )) = f ⋆ ( e 1 ) − w ⋆ ( z ) T e 1 = 1 − ( z 1 α + ∣ z 2 ∣ α ) ( α − 1 ) / α z 1 α − 1 = 1 − ( 1 + ∣ z 2 / z 1 ∣ α ) − ( α − 1 ) / α . (170)
Since x ˉ 1 ≥ 1 / 2 \bar{x}_1\geq1/2 x ˉ 1 ≥ 1/2 , this regret belongs to [ 0 , 1 ] [0,1] [ 0 , 1 ] . Continuity and the conditional convergence ∣ x ˉ 2 ∣ / x ˉ 1 → 1 |\bar{x}_2|/\bar{x}_1\to1 ∣ x ˉ 2 ∣/ x ˉ 1 → 1 imply convergence in conditional probability to 1 − 2 − ( α − 1 ) / α 1-2^{-(\alpha-1)/\alpha} 1 − 2 − ( α − 1 ) / α . Boundedness then gives
E [ R Q n ( w ⋆ ( x ˉ ) ) ∣ N n = 1 ] → 1 − 2 − ( α − 1 ) / α > 0. (171) \mathbb{E}[R_{Q_n}(w_{\star}(\bar{x}))\mid N_n=1]\to1-2^{-(\alpha-1)/\alpha}>0. \tag*{(171)} E [ R Q n ( w ⋆ ( x ˉ )) ∣ N n = 1 ] → 1 − 2 − ( α − 1 ) / α > 0. (171)
By Q n ∈ Q n , M K L ( P ) Q_n \in Q_{n,M}^{\mathrm{KL}}(P) Q n ∈ Q n , M KL ( P ) and nonnegativity of regret,
W n , M m e a n ( P ) ≥ E Q n ⊗ n R Q n ( w ∗ ( x ˉ ) ) ≥ P { N n = 1 } E [ R Q n ( w ∗ ( x ˉ ) ) ∣ N n = 1 ] = M 2 4 n ζ { 1 − 2 − ( α − 1 ) / α + o ( 1 ) } . \begin{aligned}
W_{n,M}^{\mathrm{mean}}(P) \ge\mathbb{E}_{Q_n^{\otimes n}}R_{Q_n}(w_*(\bar{x})) \\
&\ge\mathbb{P}\{N_n=1\}\mathbb{E}[R_{Q_n}(w_*(\bar{x})) \mid N_n=1] \\
&= \frac{M^2}{4n\zeta}\{1-2^{-(\alpha-1)/\alpha}+o(1)\}.
\end{aligned} W n , M mean ( P ) ≥ E Q n ⊗ n R Q n ( w ∗ ( x ˉ )) ≥ P { N n = 1 } E [ R Q n ( w ∗ ( x ˉ )) ∣ N n = 1 ] = 4 n ζ M 2 { 1 − 2 − ( α − 1 ) / α + o ( 1 )} .
The minimax lower bound supplies W n , M m e a n ( P ) ≳ n − α / 2 W_{n,M}^{\mathrm{mean}}(P) \gtrsim n^{-\alpha/2} W n , M mean ( P ) ≳ n − α /2 . Combining the two lower bounds with (61) proves (G.17). In particular, if α > 2 ζ \alpha> 2\zeta α > 2 ζ ,
W n , M m e a n ( P ) W n , M t r ( P ) ≳ n α / 2 − ζ → ∞ . \frac{W_{n,M}^{\mathrm{mean}}(P)}{W_{n,M}^{\mathrm{tr}}(P)} \gtrsim n^{\alpha/2-\zeta} \to\infty. W n , M tr ( P ) W n , M mean ( P ) ≳ n α /2 − ζ → ∞.
Hence the truncated policy has strictly smaller maximal regret for every sufficiently large n n n in this regime. In particular, for ( ζ , α ) = ( 1 , 4 ) (\zeta,\alpha)=(1,4) ( ζ , α ) = ( 1 , 4 ) , x 2 x_2 x 2 is standard Laplace, Ω = B 4 / 3 2 \Omega=B_{4/3}^2 Ω = B 4/3 2 , and
W n , M m e a n ( P ) ≲ n − 1 , W n , M t r ( P ) ≳ R n , M K L ( P ) ≳ n − 2 . W_{n,M}^{\mathrm{mean}}(P)\lesssim n^{-1}, \qquad W_{n,M}^{\mathrm{tr}}(P)\gtrsim\mathcal{R}_{n,M}^{\mathrm{KL}}(P)\gtrsim n^{-2}. W n , M mean ( P ) ≲ n − 1 , W n , M tr ( P ) ≳ R n , M KL ( P ) ≳ n − 2 .
For the boundary ( ζ , α ) = ( 1 , 2 ) (\zeta,\alpha)=(1,2) ( ζ , α ) = ( 1 , 2 ) , Ω = B 2 2 \Omega=B_2^2 Ω = B 2 2 , and μ = e 1 \mu=e_1 μ = e 1 . We show that the same construction gives a strict constant gap between ordinary SAA and truncated SAA for every fixed M > 0 M>0 M > 0 . In (G.16), x 2 x_2 x 2 now has density e − ∣ z ∣ / 2 e^{-|z|/2} e − ∣ z ∣/2 , so Σ = diag ( 1 / 4 , 2 ) \Sigma=\operatorname{diag}(1/4,2) Σ = diag ( 1/4 , 2 ) , ψ ( t ) = t 2 2 / 2 \psi(t)=t_2^2/2 ψ ( t ) = t 2 2 /2 , and H e 1 = diag ( 0 , 1 ) H_{e_1}=\operatorname{diag}(0,1) H e 1 = diag ( 0 , 1 ) . Consequently, Corollary 5.3 gives the truncated-SAA constant
lim n → ∞ n W n , M t r ( P ) = 1 2 tr ( Σ H e 1 ) = 1. \lim_{n\to\infty} nW_{n,M}^{\mathrm{tr}}(P)=\frac{1}{2}\operatorname{tr}(\Sigma H_{e_1})=1. n → ∞ lim n W n , M tr ( P ) = 2 1 tr ( Σ H e 1 ) = 1.
We first calculate the contribution from samples drawn entirely from P P P . Since μ P = μ Q n = e 1 \mu_P=\mu_{Q_n}=e_1 μ P = μ Q n = e 1 , (G.22) with α = 2 \alpha=2 α = 2 gives
R P ( w ∗ ( x ˉ ) ) = 1 − x ˉ 1 x ˉ 1 2 + x ˉ 2 2 = 1 − ( 1 + x ˉ 2 2 x ˉ 1 2 ) − 1 / 2 . R_P(w_*(\bar{x}))=1-\frac{\bar{x}_1}{\sqrt{\bar{x}_1^2+\bar{x}_2^2}}=1-\left(1+\frac{\bar{x}_2^2}{\bar{x}_1^2}\right)^{-1/2}. R P ( w ∗ ( x ˉ )) = 1 − x ˉ 1 2 + x ˉ 2 2 x ˉ 1 = 1 − ( 1 + x ˉ 1 2 x ˉ 2 2 ) − 1/2 .
Under P ⊗ n P^{\otimes n} P ⊗ n , x ˉ 1 \bar{x}_1 x ˉ 1 and x ˉ 2 \bar{x}_2 x ˉ 2 are independent. By (G.16),
1 2 ≤ x ˉ 1 ≤ 3 2 , E P ⊗ n x ˉ 1 = 1 , Var P ⊗ n ( x ˉ 1 ) = 1 4 n , \frac{1}{2}\le\bar{x}_1\le\frac{3}{2}, \qquad\mathbb{E}_{P^{\otimes n}}\bar{x}_1=1, \qquad\operatorname{Var}_{P^{\otimes n}}(\bar{x}_1)=\frac{1}{4n}, 2 1 ≤ x ˉ 1 ≤ 2 3 , E P ⊗ n x ˉ 1 = 1 , Var P ⊗ n ( x ˉ 1 ) = 4 n 1 ,
so x ˉ 1 → 1 \bar{x}_1\to1 x ˉ 1 → 1 in probability. The Laplace moments E P x 2 2 = 2 \mathbb{E}_P x_2^2=2 E P x 2 2 = 2 and E P x 2 4 = 24 \mathbb{E}_P x_2^4=24 E P x 2 4 = 24 give
E P ⊗ n x ˉ 2 2 = 2 n , E P ⊗ n x ˉ 2 4 = n E P x 2 4 + 3 n ( n − 1 ) ( E P x 2 2 ) 2 n 4 = 12 n 2 + 12 n 3 . \mathbb{E}_{P^{\otimes n}}\bar{x}_2^2=\frac{2}{n}, \qquad\mathbb{E}_{P^{\otimes n}}\bar{x}_2^4=\frac{n\mathbb{E}_P x_2^4+3n(n-1)(\mathbb{E}_P x_2^2)^2}{n^4}=\frac{12}{n^2}+\frac{12}{n^3}. E P ⊗ n x ˉ 2 2 = n 2 , E P ⊗ n x ˉ 2 4 = n 4 n E P x 2 4 + 3 n ( n − 1 ) ( E P x 2 2 ) 2 = n 2 12 + n 3 12 .
Taylor’s formula yields the uniform remainder bound
∣ 1 − ( 1 + u ) − 1 / 2 − u 2 ∣ ≤ 3 8 u 2 , u ≥ 0. \left|1-(1+u)^{-1/2}-\frac{u}{2}\right|\le\frac{3}{8}u^2,\qquad u\ge0. 1 − ( 1 + u ) − 1/2 − 2 u ≤ 8 3 u 2 , u ≥ 0.
Substituting u = x ˉ 2 2 / x ˉ 1 2 u=\bar{x}_2^2/\bar{x}_1^2 u = x ˉ 2 2 / x ˉ 1 2 and using x ˉ 1 − 4 ≤ 16 \bar{x}_1^{-4}\le16 x ˉ 1 − 4 ≤ 16 therefore gives
∣ n E P ⊗ n R P ( w ∗ ( x ˉ ) ) − n 2 E P ⊗ n x ˉ 2 2 x ˉ 1 2 ∣ ≤ 6 n E P ⊗ n x ˉ 2 4 = O ( n − 1 ) . \left|n\mathbb{E}_{P^{\otimes n}}R_P(w_*(\bar{x}))-\frac{n}{2}\mathbb{E}_{P^{\otimes n}}\frac{\bar{x}_2^2}{\bar{x}_1^2}\right|\le6n\mathbb{E}_{P^{\otimes n}}\bar{x}_2^4=O(n^{-1}). n E P ⊗ n R P ( w ∗ ( x ˉ )) − 2 n E P ⊗ n x ˉ 1 2 x ˉ 2 2 ≤ 6 n E P ⊗ n x ˉ 2 4 = O ( n − 1 ) .
Independence of x ˉ 1 \bar{x}_1 x ˉ 1 and x ˉ 2 \bar{x}_2 x ˉ 2 , and E P ⊗ n x ˉ 2 2 = 2 / n \mathbb{E}_{P^{\otimes n}}\bar{x}_2^2=2/n E P ⊗ n x ˉ 2 2 = 2/ n imply
n E P ⊗ n R P ( w ∗ ( x ˉ ) ) = n 2 E P ⊗ n [ x ˉ 1 − 2 ] E P ⊗ n x ˉ 2 2 + O ( n − 1 ) = E P ⊗ n [ x ˉ 1 − 2 ] + O ( n − 1 ) → 1. (172) n\mathbb{E}_{P^{\otimes n}}R_P(w_*(\bar{x}))=\frac{n}{2}\mathbb{E}_{P^{\otimes n}}[\bar{x}_1^{-2}]\mathbb{E}_{P^{\otimes n}}\bar{x}_2^2+O(n^{-1})=\mathbb{E}_{P^{\otimes n}}[\bar{x}_1^{-2}]+O(n^{-1})\to1. \tag*{(172)} n E P ⊗ n R P ( w ∗ ( x ˉ )) = 2 n E P ⊗ n [ x ˉ 1 − 2 ] E P ⊗ n x ˉ 2 2 + O ( n − 1 ) = E P ⊗ n [ x ˉ 1 − 2 ] + O ( n − 1 ) → 1. (172)
The last limit follows from x ˉ 1 − 2 → 1 \bar{x}_1^{-2}\to1 x ˉ 1 − 2 → 1 in probability and 0 ≤ x ˉ 1 − 2 ≤ 4 0\leq\bar{x}_1^{-2}\leq4 0 ≤ x ˉ 1 − 2 ≤ 4 .
Next, (G.18) gives ε n = M 2 / ( 4 n 2 ) \varepsilon_n=M^2/(4n^2) ε n = M 2 / ( 4 n 2 ) , and (G.20) implies
P { N n = 0 } = ( 1 − ε n ) n → 1 , n P { N n = 1 } → M 2 4 . \mathbb{P}\{N_n=0\}=(1-\varepsilon_n)^n\to1,\qquad n\mathbb{P}\{N_n=1\}\to\frac{M^2}{4}. P { N n = 0 } = ( 1 − ε n ) n → 1 , n P { N n = 1 } → 4 M 2 .
Conditional on N n = 0 N_n=0 N n = 0 , the sample law is P ⊗ n P^{\otimes n} P ⊗ n , so (G.24) supplies the contribution 1 1 1 after multiplying the risk by n n n . Conditional on N n = 1 N_n=1 N n = 1 , (G.21) gives ∣ x ˉ 2 ∣ / x ˉ 1 → 1 |\bar{x}_2|/\bar{x}_1\to1 ∣ x ˉ 2 ∣/ x ˉ 1 → 1 in probability. The regret formula (G.22) and its expectation limit (G.23) then yield
n P { N n = 1 } E [ R Q n ( w ∗ ( x ˉ ) ) ∣ N n = 1 ] → M 2 4 ( 1 − 2 − 1 / 2 ) . n\mathbb{P}\{N_n=1\}\mathbb{E}[R_{Q_n}(w_*(\bar{x}))\mid N_n=1]\to\frac{M^2}{4}(1-2^{-1/2}). n P { N n = 1 } E [ R Q n ( w ∗ ( x ˉ )) ∣ N n = 1 ] → 4 M 2 ( 1 − 2 − 1/2 ) .
The KL bound (G.19) ensures that Q n ∈ Q n , M K L ( P ) Q_n\in\mathcal{Q}^{\mathrm{KL}}_{n,M}(P) Q n ∈ Q n , M KL ( P ) for all sufficiently large n n n . Since μ Q n = μ P \mu_{Q_n}=\mu_P μ Q n = μ P , retaining the nonnegative contributions from N n = 0 N_n=0 N n = 0 and N n = 1 N_n=1 N n = 1 gives
W n , M m e a n ( P ) ≥ E Q n ⊗ n R Q n ( w ∗ ( x ˉ ) ) ≥ ( 1 − ε n ) n E P ⊗ n R P ( w ∗ ( x ˉ ) ) + P { N n = 1 } E [ R Q n ( w ∗ ( x ˉ ) ) ∣ N n = 1 ] . \begin{aligned}
W^{\mathrm{mean}}_{n,M}(P)\geq\mathbb{E}_{Q_n^{\otimes n}}R_{Q_n}(w_*(\bar{x})) \\
&\geq(1-\varepsilon_n)^n\mathbb{E}_{P^{\otimes n}}R_P(w_*(\bar{x}))+\mathbb{P}\{N_n=1\}\mathbb{E}[R_{Q_n}(w_*(\bar{x}))\mid N_n=1].
\end{aligned} W n , M mean ( P ) ≥ E Q n ⊗ n R Q n ( w ∗ ( x ˉ )) ≥ ( 1 − ε n ) n E P ⊗ n R P ( w ∗ ( x ˉ )) + P { N n = 1 } E [ R Q n ( w ∗ ( x ˉ )) ∣ N n = 1 ] .
Multiplying by n n n and using the preceding limits, we obtain
lim inf n → ∞ n W n , M m e a n ( P ) ≥ 1 + M 2 4 ( 1 − 2 − 1 / 2 ) > 1 = lim n → ∞ n W n , M t r ( P ) . \liminf_{n\to\infty}nW^{\mathrm{mean}}_{n,M}(P)\geq1+\frac{M^2}{4}(1-2^{-1/2})>1=\lim_{n\to\infty}nW^{\mathrm{tr}}_{n,M}(P). n → ∞ lim inf n W n , M mean ( P ) ≥ 1 + 4 M 2 ( 1 − 2 − 1/2 ) > 1 = n → ∞ lim n W n , M tr ( P ) .
Thus the N n = 0 N_n=0 N n = 0 contribution matches the truncated-SAA constant, while the N n = 1 N_n=1 N n = 1 contribution makes the ordinary-SAA lower bound strictly larger for every M > 0 M>0 M > 0 .
Remark G.1 (Scope of the optimality criteria). The criteria (57) and (58) retain the direction of the truncation error through r μ ( 4 Δ n ) r_\mu(4\Delta_n) r μ ( 4 Δ n ) . Corollary 6.3 gives sufficient conditions depending only on α \alpha α and ζ \zeta ζ ; Example 6.5 proves their sharpness in rate over the stated class of reference distributions. For a particular ( P , Ω ) (P,\Omega) ( P , Ω ) , the directional criterion may apply even when these scalar conditions do not.
If P P P has bounded support, then x j ( n ) = x j x_j^{(n)}=x_j x j ( n ) = x j P P P -almost surely for all sufficiently large n n n , and hence Q Q Q -almost surely for every Q ≪ P Q\ll P Q ≪ P . Thus W n , M m e a n ( P ) = W n , M t r ( P ) W^{\mathrm{mean}}_{n,M}(P)=W^{\mathrm{tr}}_{n,M}(P) W n , M mean ( P ) = W n , M tr ( P ) eventually. For unbounded support, (G.7) still gives the same sequential Gaussian fluctuation limit, while (58) ensures convergence of the expected normalized regrets.
Auxiliary Local-Asymptotic Lemmas
Lemma H.1 (Duchi–Ruan tilts with bounded scores). Let P P P be a Borel probability measure on R d \mathbb{R}^d R d with
E P ∥ x ∥ 2 2 < ∞ , μ = E P x , Σ = Cov P ( x ) ≻ 0. \mathbb{E}_P\lVert x\rVert_2^2<\infty,\qquad\mu=\mathbb{E}_P x,\qquad\Sigma=\operatorname{Cov}_P(x)\succ0. E P ∥ x ∥ 2 2 < ∞ , μ = E P x , Σ = Cov P ( x ) ≻ 0.
Following Duchi and Ruan (2021, Section 3.1), let h D R : R → [ − 1 , 1 ] h_{\mathrm{DR}}:\mathbb{R}\to[-1,1] h DR : R → [ − 1 , 1 ] be C 3 C^3 C 3 , with h D R ( s ) = s h_{\mathrm{DR}}(s)=s h DR ( s ) = s for ∣ s ∣ ≤ 1 / 2 |s|\le1/2 ∣ s ∣ ≤ 1/2 , h D R ′ ≥ 0 h'_{\mathrm{DR}}\ge0 h DR ′ ≥ 0 , and bounded first three derivatives. For L > 0 L>0 L > 0 , define
g L ( x ) : = ( x − μ ) I { ∥ x − μ ∥ 2 ≤ L } − E P [ ( x − μ ) I { ∥ x − μ ∥ 2 ≤ L } ] , g_L(x):=(x-\mu)\mathbb{I}\{\lVert x-\mu\rVert_2\le L\}-\mathbb{E}_P[(x-\mu)\mathbb{I}\{\lVert x-\mu\rVert_2\le L\}], g L ( x ) := ( x − μ ) I {∥ x − μ ∥ 2 ≤ L } − E P [( x − μ ) I {∥ x − μ ∥ 2 ≤ L }] ,
Γ L : = E P [ g L g L ⊤ ] . \Gamma_L:=\mathbb{E}_P[g_Lg_L^\top]. Γ L := E P [ g L g L ⊤ ] .
Then Γ L → L → ∞ Σ \Gamma_L \xrightarrow{L\to\infty}\Sigma Γ L L → ∞ Σ in operator norm, so Γ L ≻ 0 \Gamma_L\succ0 Γ L ≻ 0 for sufficiently large L L L . For such L L L , set
J L : = E P [ ( x − μ ) g L ⊤ ] Γ L − 1 / 2 . J_L:=\mathbb{E}_P[(x-\mu)g_L^\top]\Gamma_L^{-1/2}. J L := E P [( x − μ ) g L ⊤ ] Γ L − 1/2 .
We have J L → L → ∞ Σ 1 / 2 J_L\xrightarrow{L\to\infty}\Sigma^{1/2} J L L → ∞ Σ 1/2 in operator norm, and J L J_L J L is invertible for sufficiently large L L L . Fix such an L L L and M > 0 M>0 M > 0 . In the following assertions, n → ∞ n\to\infty n → ∞ with L , M L,M L , M fixed, and the thresholds for sufficiently large n n n may depend on P , L , M P,L,M P , L , M . The suprema over u ∈ B 2 ( M ) u\in B_2(M) u ∈ B 2 ( M ) express uniformity in u u u for these fixed L , M L,M L , M . For all sufficiently large n n n and u ∈ B 2 ( M ) u\in B_2(M) u ∈ B 2 ( M ) , define
d Q n , u ( L ) d P ( x ) : = 1 + h D R ( n − 1 / 2 u ⊤ Γ L − 1 / 2 g L ( x ) ) E P [ 1 + h D R ( n − 1 / 2 u ⊤ Γ L − 1 / 2 g L ( x ) ) ] . (173) \frac{\mathrm{d}Q_{n,u}^{(L)}}{\mathrm{d}P}(x):=\frac{1+h_{\mathrm{DR}}(n^{-1/2}u^\top\Gamma_L^{-1/2}g_L(x))}{\mathbb{E}_P[1+h_{\mathrm{DR}}(n^{-1/2}u^\top\Gamma_L^{-1/2}g_L(x))]}. \tag*{(173)} d P d Q n , u ( L ) ( x ) := E P [ 1 + h DR ( n − 1/2 u ⊤ Γ L − 1/2 g L ( x ))] 1 + h DR ( n − 1/2 u ⊤ Γ L − 1/2 g L ( x )) . (173)
Uniformly over this ball, these measures are equivalent to P P P , and
d Q n , u ( L ) d P ( x ) = 1 + u ⊤ Γ L − 1 / 2 g L ( x ) n , E Q n , u ( L ) x = μ + J L u n . (174) \frac{\mathrm{d}Q_{n,u}^{(L)}}{\mathrm{d}P}(x)=1+\frac{u^\top\Gamma_L^{-1/2}g_L(x)}{\sqrt{n}},\qquad\mathbb{E}_{Q_{n,u}^{(L)}}x=\mu+\frac{J_Lu}{\sqrt{n}}. \tag*{(174)} d P d Q n , u ( L ) ( x ) = 1 + n u ⊤ Γ L − 1/2 g L ( x ) , E Q n , u ( L ) x = μ + n J L u . (174)
For every proper lower semicontinuous convex function ϕ : [ 0 , ∞ ) → [ 0 , ∞ ] \phi:[0,\infty)\to[0,\infty] ϕ : [ 0 , ∞ ) → [ 0 , ∞ ] with ϕ ( 1 ) = ϕ ′ ( 1 ) = 0 \phi(1)=\phi'(1)=0 ϕ ( 1 ) = ϕ ′ ( 1 ) = 0 , ϕ ∈ C 2 \phi\in C^2 ϕ ∈ C 2 near 1 1 1 , and 0 < ϕ ′ ′ ( 1 ) < ∞ 0<\phi''(1)<\infty 0 < ϕ ′′ ( 1 ) < ∞ ,
sup u ∈ B 2 ( M ) ∣ n E P ϕ ( d Q n , u ( L ) d P ) − ϕ ′ ′ ( 1 ) 2 ∥ u ∥ 2 2 ∣ → n → ∞ 0. (175) \sup_{u\in B_2(M)}\left|n\mathbb{E}_P\phi\left(\frac{\mathrm{d}Q_{n,u}^{(L)}}{\mathrm{d}P}\right)-\frac{\phi''(1)}{2}\lVert u\rVert_2^2\right|\xrightarrow{n\to\infty}0. \tag*{(175)} u ∈ B 2 ( M ) sup n E P ϕ ( d P d Q n , u ( L ) ) − 2 ϕ ′′ ( 1 ) ∥ u ∥ 2 2 n → ∞ 0. (175)
In particular, for ϕ ( s ) = s log s − s + 1 \phi(s)=s\log s-s+1 ϕ ( s ) = s log s − s + 1 , the expectation is D K L ( Q n , u ( L ) ∥ P ) D_{\mathrm{KL}}(Q_{n,u}^{(L)}\mathbin{\|}P) D KL ( Q n , u ( L ) ∥ P ) and ϕ ′ ′ ( 1 ) = 1 \phi''(1)=1 ϕ ′′ ( 1 ) = 1 . Under P ⊗ n P^{\otimes n} P ⊗ n , write
L n ( L ) ( u ) : = d ( Q n , u ( L ) ) ⊗ n d P ⊗ n ( x 1 , … , x n ) . L_n^{(L)}(u):=\frac{\mathrm{d}(Q_{n,u}^{(L)})^{\otimes n}}{\mathrm{d}P^{\otimes n}}(x_1,\ldots,x_n). L n ( L ) ( u ) := d P ⊗ n d ( Q n , u ( L ) ) ⊗ n ( x 1 , … , x n ) .
These likelihoods have strictly positive continuous sample paths on B 2 ( M ) B_2(M) B 2 ( M ) , with
sup n l a r g e E P ⊗ n ∥ L n ( L ) ∥ ∞ 2 < ∞ , \sup_{n\ \mathrm{large}}\mathbb{E}_{P^{\otimes n}}\lVert L_n^{(L)}\rVert_\infty^2<\infty, n large sup E P ⊗ n ∥ L n ( L ) ∥ ∞ 2 < ∞ ,
and
L n ( L ) → d n → ∞ [ u ↦ e u ⊤ Y − ∥ u ∥ 2 2 / 2 ] in C ( B 2 ( M ) ) , Y ∼ N ( 0 , I d ) . (176) L_n^{(L)}\xrightarrow{d}_{n\to\infty}\left[u\mapsto e^{u^\top Y-\lVert u\rVert_2^2/2}\right]\quad\text{in }C(B_2(M)),\qquad Y\sim\mathcal{N}(0,I_d). \tag*{(176)} L n ( L ) d n → ∞ [ u ↦ e u ⊤ Y − ∥ u ∥ 2 2 /2 ] in C ( B 2 ( M )) , Y ∼ N ( 0 , I d ) . (176)
The L → ∞ L\to\infty L → ∞ limits concern the approximation of x − μ x-\mu x − μ by the bounded scores. When combined with the sampling limits, one first lets n → ∞ n\to\infty n → ∞ for each fixed L L L , and then lets L → ∞ L\to\infty L → ∞ .
Proof. Centering and truncation give E P g L = 0 \mathbb{E}_{P}g_L = 0 E P g L = 0 , ∥ g L ∥ 2 ≤ 2 L \|g_L\|_2 \le2L ∥ g L ∥ 2 ≤ 2 L , and E P ∥ g L ∥ 2 2 ≤ tr ( Σ ) \mathbb{E}_{P}\|g_L\|_2^2 \le\operatorname{tr}(\Sigma) E P ∥ g L ∥ 2 2 ≤ tr ( Σ ) .
Note that
g L − ( x − μ ) = − ( x − μ ) I { ∥ x − μ ∥ 2 > L } + E P [ ( x − μ ) I { ∥ x − μ ∥ 2 > L } ] , g_L-(x-\mu)=-(x-\mu)\mathbb{I}\{\|x-\mu\|_2>L\}+\mathbb{E}_{P}[(x-\mu)\mathbb{I}\{\|x-\mu\|_2>L\}], g L − ( x − μ ) = − ( x − μ ) I { ∥ x − μ ∥ 2 > L } + E P [( x − μ ) I { ∥ x − μ ∥ 2 > L }] ,
which, by DCT, yields
E P ∥ g L − ( x − μ ) ∥ 2 2 ≤ E P [ ∥ x − μ ∥ 2 2 I { ∥ x − μ ∥ 2 > L } ] → L → ∞ 0. \mathbb{E}_{P}\|g_L-(x-\mu)\|_2^2\le\mathbb{E}_{P}\left[\|x-\mu\|_2^2\mathbb{I}\{\|x-\mu\|_2>L\}\right]\xrightarrow[L\to\infty]{}0. E P ∥ g L − ( x − μ ) ∥ 2 2 ≤ E P [ ∥ x − μ ∥ 2 2 I { ∥ x − μ ∥ 2 > L } ] L → ∞ 0.
Cauchy-Schwarz inequality therefore gives
∥ E P [ ( x − μ ) g L T ] − Σ ∥ op = ∥ E P [ ( x − μ ) ( g L − ( x − μ ) ) T ] ∥ op ≤ E P [ ∥ x − μ ∥ 2 ∥ g L − ( x − μ ) ∥ 2 ] ≤ tr ( Σ ) ( E P ∥ g L − ( x − μ ) ∥ 2 2 ) 1 / 2 → L → ∞ 0 , \begin{aligned}
\left\|\mathbb{E}_{P}[(x-\mu)g_L^{\mathsf T}]-\Sigma\right\|_{\operatorname{op}}
&=\left\|\mathbb{E}_{P}[(x-\mu)(g_L-(x-\mu))^{\mathsf T}]\right\|_{\operatorname{op}}\\
&\le\mathbb{E}_{P}\left[\|x-\mu\|_2\|g_L-(x-\mu)\|_2\right]\\
&\le\sqrt{\operatorname{tr}(\Sigma)}\left(\mathbb{E}_{P}\|g_L-(x-\mu)\|_2^2\right)^{1/2}\xrightarrow[L\to\infty]{}0,
\end{aligned} E P [( x − μ ) g L T ] − Σ op = E P [( x − μ ) ( g L − ( x − μ ) ) T ] op ≤ E P [ ∥ x − μ ∥ 2 ∥ g L − ( x − μ ) ∥ 2 ] ≤ tr ( Σ ) ( E P ∥ g L − ( x − μ ) ∥ 2 2 ) 1/2 L → ∞ 0 ,
i.e., E P [ ( x − μ ) g L T ] → L → ∞ Σ \mathbb{E}_{P}[(x-\mu)g_L^{\mathsf T}]\xrightarrow[L\to\infty]{}\Sigma E P [( x − μ ) g L T ] L → ∞ Σ . Write
g L g L T − ( x − μ ) ( x − μ ) T = ( g L − ( x − μ ) ) g L T + ( x − μ ) ( g L − ( x − μ ) ) T , g_Lg_L^{\mathsf T}-(x-\mu)(x-\mu)^{\mathsf T}=(g_L-(x-\mu))g_L^{\mathsf T}+(x-\mu)(g_L-(x-\mu))^{\mathsf T}, g L g L T − ( x − μ ) ( x − μ ) T = ( g L − ( x − μ )) g L T + ( x − μ ) ( g L − ( x − μ ) ) T ,
which further gives
∥ Γ L − Σ ∥ op ≤ ( E P ∥ g L − ( x − μ ) ∥ 2 2 ) 1 / 2 [ ( E P ∥ g L ∥ 2 2 ) 1 / 2 + ( E P ∥ x − μ ∥ 2 2 ) 1 / 2 ] ≤ 2 tr ( Σ ) ( E P ∥ g L − ( x − μ ) ∥ 2 2 ) 1 / 2 → L → ∞ 0. \begin{aligned}
\|\Gamma_L-\Sigma\|_{\operatorname{op}}
&\le\left(\mathbb{E}_{P}\|g_L-(x-\mu)\|_2^2\right)^{1/2}\left[\left(\mathbb{E}_{P}\|g_L\|_2^2\right)^{1/2}+\left(\mathbb{E}_{P}\|x-\mu\|_2^2\right)^{1/2}\right]\\
&\le2\sqrt{\operatorname{tr}(\Sigma)}\left(\mathbb{E}_{P}\|g_L-(x-\mu)\|_2^2\right)^{1/2}\xrightarrow[L\to\infty]{}0.
\end{aligned} ∥ Γ L − Σ ∥ op ≤ ( E P ∥ g L − ( x − μ ) ∥ 2 2 ) 1/2 [ ( E P ∥ g L ∥ 2 2 ) 1/2 + ( E P ∥ x − μ ∥ 2 2 ) 1/2 ] ≤ 2 tr ( Σ ) ( E P ∥ g L − ( x − μ ) ∥ 2 2 ) 1/2 L → ∞ 0.
Since Σ ≻ 0 \Sigma\succ0 Σ ≻ 0 ,
J L → L → ∞ Σ Σ − 1 / 2 = Σ 1 / 2 , λ min ( Γ L ) → L → ∞ λ min ( Σ ) > 0 , det J L → L → ∞ det ( Σ 1 / 2 ) > 0. J_L\xrightarrow[L\to\infty]{}\Sigma\Sigma^{-1/2}=\Sigma^{1/2},\qquad\lambda_{\min}(\Gamma_L)\xrightarrow[L\to\infty]{}\lambda_{\min}(\Sigma)>0,\qquad\det J_L\xrightarrow[L\to\infty]{}\det(\Sigma^{1/2})>0. J L L → ∞ Σ Σ − 1/2 = Σ 1/2 , λ m i n ( Γ L ) L → ∞ λ m i n ( Σ ) > 0 , det J L L → ∞ det ( Σ 1/2 ) > 0.
Fix L L L with Γ L ≻ 0 \Gamma_L\succ0 Γ L ≻ 0 and J L J_L J L invertible, and fix M > 0 M>0 M > 0 . Throughout the remainder of the proof, all limits are as n → ∞ n\to\infty n → ∞ with these L , M L,M L , M fixed. For sufficiently large n n n ,
sup u ∈ B 2 ( M ) sup x ∈ R d ∣ u T Γ L − 1 / 2 g L ( x ) n ∣ ≤ 2 M L ∥ Γ L − 1 / 2 ∥ op n ≤ 1 2 . \sup_{u\in B_2(M)}\sup_{x\in\mathbb{R}^d}\left|\frac{u^{\mathsf T}\Gamma_L^{-1/2}g_L(x)}{\sqrt{n}}\right|\le\frac{2ML\|\Gamma_L^{-1/2}\|_{\operatorname{op}}}{\sqrt{n}}\le\frac{1}{2}. u ∈ B 2 ( M ) sup x ∈ R d sup n u T Γ L − 1/2 g L ( x ) ≤ n 2 M L ∥ Γ L − 1/2 ∥ op ≤ 2 1 .
Thus h D R h_{\mathrm{DR}} h DR equals the identity at every argument in (H.1), and its denominator equals
1 + u T Γ L − 1 / 2 E P g L n = 1. 1+\frac{u^{\mathsf T}\Gamma_L^{-1/2}\mathbb{E}_{P}g_L}{\sqrt{n}}=1. 1 + n u T Γ L − 1/2 E P g L = 1.
Consequently,
1 2 ≤ d Q n , u ( L ) d P ≤ 3 2 , E Q n , u ( L ) ∥ x ∥ 2 2 ≤ 3 2 E P ∥ x ∥ 2 2 < ∞ , \frac{1}{2}\le\frac{\mathrm{d}Q_{n,u}^{(L)}}{\mathrm{d}P}\le\frac{3}{2},\qquad\mathbb{E}_{Q_{n,u}^{(L)}}\|x\|_2^2\le\frac{3}{2}\mathbb{E}_{P}\|x\|_2^2<\infty, 2 1 ≤ d P d Q n , u ( L ) ≤ 2 3 , E Q n , u ( L ) ∥ x ∥ 2 2 ≤ 2 3 E P ∥ x ∥ 2 2 < ∞ ,
and
E Q n , u ( L ) x − μ = E P [ ( x − μ ) g L T ] Γ L − 1 / 2 u n = J L u n . \mathbb{E}_{Q_{n,u}^{(L)}}x-\mu=\frac{\mathbb{E}_{P}[(x-\mu)g_L^{\mathsf T}]\Gamma_L^{-1/2}u}{\sqrt{n}}=\frac{J_Lu}{\sqrt{n}}. E Q n , u ( L ) x − μ = n E P [( x − μ ) g L T ] Γ L − 1/2 u = n J L u .
This proves (H.2).
Taylor’s formula at 1 gives, for sufficiently small ∣ z ∣ |z| ∣ z ∣ ,
∣ ϕ ( 1 + z ) − ϕ ′ ′ ( 1 ) 2 z 2 ∣ ≤ z 2 2 sup ∣ v ∣ ≤ ∣ z ∣ ∣ ϕ ′ ′ ( 1 + v ) − ϕ ′ ′ ( 1 ) ∣ . \left|\phi(1+z)-\frac{\phi^{\prime\prime}(1)}{2}z^2\right|\leq\frac{z^2}{2}\sup_{|v|\leq|z|}\left|\phi^{\prime\prime}(1+v)-\phi^{\prime\prime}(1)\right|. ϕ ( 1 + z ) − 2 ϕ ′′ ( 1 ) z 2 ≤ 2 z 2 ∣ v ∣ ≤ ∣ z ∣ sup ∣ ϕ ′′ ( 1 + v ) − ϕ ′′ ( 1 ) ∣ .
Since E P ( u T Γ L − 1 / 2 g L ) 2 = ∥ u ∥ 2 2 \mathbb{E}_{P}(u^{\mathsf{T}}\Gamma_{L}^{-1/2}g_{L})^2=\|u\|_2^2 E P ( u T Γ L − 1/2 g L ) 2 = ∥ u ∥ 2 2 , when n → ∞ n\to\infty n → ∞ ,
sup u ∈ B 2 ( M ) ∣ n E P ϕ ( d Q n , u ( L ) d P ) − ϕ ′ ′ ( 1 ) 2 ∥ u ∥ 2 2 ∣ ≤ M 2 2 sup ∣ v ∣ ≤ 2 M ∥ Γ L − 1 / 2 ∥ o p / n ∣ ϕ ′ ′ ( 1 + v ) − ϕ ′ ′ ( 1 ) ∣ → 0. \sup_{u\in B_2(M)}\left|n\mathbb{E}_{P}\phi\left(\frac{dQ_{n,u}^{(L)}}{dP}\right)-\frac{\phi^{\prime\prime}(1)}{2}\|u\|_2^2\right|\leq\frac{M^2}{2}\sup_{|v|\leq2M\|\Gamma_L^{-1/2}\|_{\mathrm{op}}/\sqrt{n}}\left|\phi^{\prime\prime}(1+v)-\phi^{\prime\prime}(1)\right|\to0. u ∈ B 2 ( M ) sup n E P ϕ ( d P d Q n , u ( L ) ) − 2 ϕ ′′ ( 1 ) ∥ u ∥ 2 2 ≤ 2 M 2 ∣ v ∣ ≤ 2 M ∥ Γ L − 1/2 ∥ op / n sup ∣ ϕ ′′ ( 1 + v ) − ϕ ′′ ( 1 ) ∣ → 0.
This proves (H.3).
For ∣ z ∣ ≤ 1 / 2 |z|\leq1/2 ∣ z ∣ ≤ 1/2 , ∣ log ( 1 + z ) − z + z 2 / 2 ∣ ≤ C ∣ z ∣ 3 |\log(1+z)-z+z^2/2|\leq C|z|^3 ∣ log ( 1 + z ) − z + z 2 /2∣ ≤ C ∣ z ∣ 3 . Applying this to each likelihood factor gives, uniformly over u ∈ B 2 ( M ) u\in B_2(M) u ∈ B 2 ( M ) ,
log L n ( L ) ( u ) = u T 1 n ∑ i = 1 n Γ L − 1 / 2 g L ( x i ) − 1 2 u T ( 1 n ∑ i = 1 n Γ L − 1 / 2 g L ( x i ) g L ( x i ) T Γ L − 1 / 2 ) u + O L , M ( n − 1 / 2 ) . (177) \log L_n^{(L)}(u)=u^{\mathsf{T}}\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\Gamma_L^{-1/2}g_L(x_i)-\frac{1}{2}u^{\mathsf{T}}\left(\frac{1}{n}\sum_{i=1}^{n}\Gamma_L^{-1/2}g_L(x_i)g_L(x_i)^{\mathsf{T}}\Gamma_L^{-1/2}\right)u+O_{L,M}(n^{-1/2}). \tag*{(177)} log L n ( L ) ( u ) = u T n 1 i = 1 ∑ n Γ L − 1/2 g L ( x i ) − 2 1 u T ( n 1 i = 1 ∑ n Γ L − 1/2 g L ( x i ) g L ( x i ) T Γ L − 1/2 ) u + O L , M ( n − 1/2 ) . (177)
Note that Γ L − 1 / 2 g L ( x ) \Gamma_L^{-1/2}g_L(x) Γ L − 1/2 g L ( x ) has mean zero and covariance I d I_d I d . CLT and LLN yield
1 n ∑ i = 1 n Γ L − 1 / 2 g L ( x i ) → n → ∞ d N ( 0 , I d ) , \frac{1}{\sqrt{n}}\sum_{i=1}^{n}\Gamma_L^{-1/2}g_L(x_i)\xrightarrow[n\to\infty]{d}\mathcal{N}(0,I_d), n 1 i = 1 ∑ n Γ L − 1/2 g L ( x i ) d n → ∞ N ( 0 , I d ) ,
1 n ∑ i = 1 n Γ L − 1 / 2 g L ( x i ) g L ( x i ) T Γ L − 1 / 2 → n → ∞ P I d . \frac{1}{n}\sum_{i=1}^{n}\Gamma_L^{-1/2}g_L(x_i)g_L(x_i)^{\mathsf{T}}\Gamma_L^{-1/2}\xrightarrow[n\to\infty]{\mathbb{P}}I_d. n 1 i = 1 ∑ n Γ L − 1/2 g L ( x i ) g L ( x i ) T Γ L − 1/2 P n → ∞ I d .
The latter convergence is in operator norm, since d d d is fixed. Equip C ( B 2 ( M ) ) C(B_2(M)) C ( B 2 ( M )) with the supremum norm. Since ∣ u T A u ∣ ≤ M 2 ∥ A ∥ o p |u^{\mathsf{T}}Au|\leq M^2\|A\|_{\mathrm{op}} ∣ u T A u ∣ ≤ M 2 ∥ A ∥ op for u ∈ B 2 ( M ) u\in B_2(M) u ∈ B 2 ( M ) , (H.5) and LLN give
sup u ∈ B 2 ( M ) ∣ log L n ( L ) ( u ) − { u T 1 n ∑ i = 1 n Γ L − 1 / 2 g L ( x i ) − 1 2 ∥ u ∥ 2 2 } ∣ ≤ M 2 2 ∥ 1 n ∑ i = 1 n Γ L − 1 / 2 g L ( x i ) g L ( x i ) T Γ L − 1 / 2 − I d ∥ o p + C L , M n − 1 / 2 → n → ∞ P 0. (178) \sup_{u\in B_2(M)}\left|\log L_n^{(L)}(u)-\left\{u^{\mathsf{T}}\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\Gamma_L^{-1/2}g_L(x_i)-\frac{1}{2}\|u\|_2^2\right\}\right|
\leq\frac{M^2}{2}\left\|\frac{1}{n}\sum_{i=1}^{n}\Gamma_L^{-1/2}g_L(x_i)g_L(x_i)^{\mathsf{T}}\Gamma_L^{-1/2}-I_d\right\|_{\mathrm{op}}+C_{L,M}n^{-1/2}\xrightarrow[n\to\infty]{\mathbb{P}}0. \tag*{(178)} u ∈ B 2 ( M ) sup log L n ( L ) ( u ) − { u T n 1 i = 1 ∑ n Γ L − 1/2 g L ( x i ) − 2 1 ∥ u ∥ 2 2 } ≤ 2 M 2 n 1 i = 1 ∑ n Γ L − 1/2 g L ( x i ) g L ( x i ) T Γ L − 1/2 − I d op + C L , M n − 1/2 P n → ∞ 0. (178)
The map
y ⟼ [ u ⟼ u T y − 1 2 ∥ u ∥ 2 2 ] , R d ⟶ C ( B 2 ( M ) ) , y\longmapsto\left[u\longmapsto u^{\mathsf{T}}y-\frac{1}{2}\|u\|_2^2\right],\qquad\mathbb{R}^d\longrightarrow C(B_2(M)), y ⟼ [ u ⟼ u T y − 2 1 ∥ u ∥ 2 2 ] , R d ⟶ C ( B 2 ( M )) ,
is continuous, because
sup u ∈ B 2 ( M ) ∣ u T ( y − y ′ ) ∣ ≤ M ∥ y − y ′ ∥ 2 . \sup_{u\in B_2(M)}|u^{\mathsf{T}}(y-y^{\prime})|\leq M\|y-y^{\prime}\|_2. u ∈ B 2 ( M ) sup ∣ u T ( y − y ′ ) ∣ ≤ M ∥ y − y ′ ∥ 2 .
Thus CLT and the continuous mapping theorem imply
[ u ⟼ u T 1 n ∑ i = 1 n Γ L − 1 / 2 g L ( x i ) − 1 2 ∥ u ∥ 2 2 ] → n → ∞ d [ u ⟼ u T Y − 1 2 ∥ u ∥ 2 2 ] in C ( B 2 ( M ) ) , \left[u\longmapsto u^{\mathsf{T}}\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\Gamma_L^{-1/2}g_L(x_i)-\frac{1}{2}\|u\|_2^2\right]\xrightarrow[n\to\infty]{d}\left[u\longmapsto u^{\mathsf{T}}Y-\frac{1}{2}\|u\|_2^2\right]\quad\text{in }C(B_2(M)), [ u ⟼ u T n 1 i = 1 ∑ n Γ L − 1/2 g L ( x i ) − 2 1 ∥ u ∥ 2 2 ] d n → ∞ [ u ⟼ u T Y − 2 1 ∥ u ∥ 2 2 ] in C ( B 2 ( M )) ,
where Y ∼ N ( 0 , I d ) \mathbf{Y} \sim\mathcal{N}(\mathbf{0}, I_d) Y ∼ N ( 0 , I d ) . Together with (H.6), Slutsky’s theorem in C ( B 2 ( M ) ) C(B_2(M)) C ( B 2 ( M )) yields
log L n ( L ) → d n → ∞ [ u ↦ u ⊤ Y − 1 2 ∥ u ∥ 2 2 ] in C ( B 2 ( M ) ) . \log L_n^{(L)} \xrightarrow{d}_{n\to\infty} \left[u \mapsto u^{\top}\mathbf{Y} - \frac{1}{2}\lVert u\rVert_2^2\right] \text{ in } C(B_2(M)). log L n ( L ) d n → ∞ [ u ↦ u ⊤ Y − 2 1 ∥ u ∥ 2 2 ] in C ( B 2 ( M )) .
Finally, exponentiation is continuous on this space. Applying the continuous mapping theorem once more proves
L n ( L ) → d n → ∞ [ u ↦ e u ⊤ Y − ∥ u ∥ 2 2 / 2 ] in C ( B 2 ( M ) ) , L_n^{(L)} \xrightarrow{d}_{n\to\infty} \left[u \mapsto e^{u^{\top}\mathbf{Y}-\lVert u\rVert_2^2/2}\right] \text{ in } C(B_2(M)), L n ( L ) d n → ∞ [ u ↦ e u ⊤ Y − ∥ u ∥ 2 2 /2 ] in C ( B 2 ( M )) ,
which is (H.4). All these limits hold for fixed L L L , M M M under P ⊗ n P^{\otimes n} P ⊗ n .
To prove its moment bound, use log ( 1 + z ) ≤ z \log(1+z) \le z log ( 1 + z ) ≤ z to obtain
∥ L n ( L ) ∥ ∞ 2 ≤ exp ( 2 M ∥ 1 n ∑ i = 1 n Γ L − 1 / 2 g L ( x i ) ∥ 2 ) . \lVert L_n^{(L)}\rVert_\infty^2 \le\exp\left(2M\left\lVert\frac{1}{\sqrt{n}}\sum_{i=1}^n \Gamma_L^{-1/2}g_L(x_i)\right\rVert_2\right). ∥ L n ( L ) ∥ ∞ 2 ≤ exp ( 2 M n 1 i = 1 ∑ n Γ L − 1/2 g L ( x i ) 2 ) .
For every fixed v ∈ R d \mathbf{v}\in\mathbb{R}^d v ∈ R d , e z ≤ 1 + z + z 2 e ∣ z ∣ / 2 e^z\le1+z+z^2e^{\lvert z\rvert}/2 e z ≤ 1 + z + z 2 e ∣ z ∣ /2 and centering imply
E P e v ⊤ Γ L − 1 / 2 g L / n ≤ 1 + ∥ v ∥ 2 2 2 n e 2 L ∥ Γ L − 1 / 2 v ∥ 2 / n . \mathbb{E}_P e^{\mathbf{v}^{\top}\Gamma_L^{-1/2}g_L/\sqrt{n}} \le1+\frac{\lVert\mathbf{v}\rVert_2^2}{2n}e^{2L\lVert\Gamma_L^{-1/2}\mathbf{v}\rVert_2/\sqrt{n}}. E P e v ⊤ Γ L − 1/2 g L / n ≤ 1 + 2 n ∥ v ∥ 2 2 e 2 L ∥ Γ L − 1/2 v ∥ 2 / n .
For sufficiently large n n n , independence therefore gives
E P ⊗ n exp ( v ⊤ 1 n ∑ i = 1 n Γ L − 1 / 2 g L ( x i ) ) ≤ e ∥ v ∥ 2 2 . \mathbb{E}_{P^{\otimes n}}\exp\left(\mathbf{v}^{\top}\frac{1}{\sqrt{n}}\sum_{i=1}^n\Gamma_L^{-1/2}g_L(x_i)\right)\le e^{\lVert\mathbf{v}\rVert_2^2}. E P ⊗ n exp ( v ⊤ n 1 i = 1 ∑ n Γ L − 1/2 g L ( x i ) ) ≤ e ∥ v ∥ 2 2 .
Finally, e c ∥ z ∥ 2 ≤ ∑ v ∈ { − 1 , 1 } d e c v ⊤ z e^{c\lVert z\rVert_2}\le\sum_{\mathbf{v}\in\{-1,1\}^d}e^{c\mathbf{v}^{\top}z} e c ∥ z ∥ 2 ≤ ∑ v ∈ { − 1 , 1 } d e c v ⊤ z yields, for all sufficiently large n n n ,
E P ⊗ n ∥ L n ( L ) ∥ ∞ 2 ≤ ∑ v ∈ { − 1 , 1 } d E P ⊗ n exp ( 2 M v ⊤ 1 n ∑ i = 1 n Γ L − 1 / 2 g L ( x i ) ) ≤ 2 d e 4 M 2 d < ∞ . \mathbb{E}_{P^{\otimes n}}\lVert L_n^{(L)}\rVert_\infty^2
\le\sum_{\mathbf{v}\in\{-1,1\}^d}\mathbb{E}_{P^{\otimes n}}\exp\left(2M\mathbf{v}^{\top}\frac{1}{\sqrt{n}}\sum_{i=1}^n\Gamma_L^{-1/2}g_L(x_i)\right)
\le2^d e^{4M^2d}<\infty. E P ⊗ n ∥ L n ( L ) ∥ ∞ 2 ≤ v ∈ { − 1 , 1 } d ∑ E P ⊗ n exp ( 2 M v ⊤ n 1 i = 1 ∑ n Γ L − 1/2 g L ( x i ) ) ≤ 2 d e 4 M 2 d < ∞.
Lemma H.2 (Mean parametrization and the local KL expansion). Under (42) , there exist ρ P , ε P , C P > 0 \rho_P,\varepsilon_P,C_P>0 ρ P , ε P , C P > 0 and a C ∞ C^\infty C ∞ map θ ( h ) \boldsymbol{\theta}(\mathbf{h}) θ ( h ) on a neighborhood of B 2 ( ε P ) B_2(\varepsilon_P) B 2 ( ε P ) such that θ ( 0 ) = 0 \boldsymbol{\theta}(\mathbf{0})=\mathbf{0} θ ( 0 ) = 0 , ∇ Λ P ( θ ( h ) ) = h \nabla\Lambda_P(\boldsymbol{\theta}(\mathbf{h}))=\mathbf{h} ∇ Λ P ( θ ( h )) = h , and
∥ θ ( h ) ∥ 2 ≤ min { ρ P , C P ∥ h ∥ 2 } , ∥ h ∥ 2 ≤ ε P . (179) \lVert\boldsymbol{\theta}(\mathbf{h})\rVert_2\le\min\{\rho_P,C_P\lVert\mathbf{h}\rVert_2\},\qquad\lVert\mathbf{h}\rVert_2\le\varepsilon_P. \tag*{(179)} ∥ θ ( h ) ∥ 2 ≤ min { ρ P , C P ∥ h ∥ 2 } , ∥ h ∥ 2 ≤ ε P . (179) Moreover,
∥ θ ( h ) − Σ − 1 h ∥ 2 ≤ C P ∥ h ∥ 2 2 , ∥ h ∥ 2 ≤ ε P . (180) \lVert\boldsymbol{\theta}(\mathbf{h})-\Sigma^{-1}\mathbf{h}\rVert_2\le C_P\lVert\mathbf{h}\rVert_2^2,\qquad\lVert\mathbf{h}\rVert_2\le\varepsilon_P. \tag*{(180)} ∥ θ ( h ) − Σ − 1 h ∥ 2 ≤ C P ∥ h ∥ 2 2 , ∥ h ∥ 2 ≤ ε P . (180) and, for ∥ θ ∥ 2 ≤ ρ P \lVert\boldsymbol{\theta}\rVert_2\le\rho_P ∥ θ ∥ 2 ≤ ρ P ,
0 ≤ Λ P ( θ ) ≤ C P 2 ∥ θ ∥ 2 2 , ∣ Λ P ( θ ) − 1 2 θ ⊤ Σ θ ∣ ≤ C P 6 ∥ θ ∥ 2 3 . (181) 0\le\Lambda_P(\boldsymbol{\theta})\le\frac{C_P}{2}\lVert\boldsymbol{\theta}\rVert_2^2,\qquad\left\lvert\Lambda_P(\boldsymbol{\theta})-\frac{1}{2}\boldsymbol{\theta}^{\top}\Sigma\boldsymbol{\theta}\right\rvert\le\frac{C_P}{6}\lVert\boldsymbol{\theta}\rVert_2^3. \tag*{(181)} 0 ≤ Λ P ( θ ) ≤ 2 C P ∥ θ ∥ 2 2 , Λ P ( θ ) − 2 1 θ ⊤ Σ θ ≤ 6 C P ∥ θ ∥ 2 3 . (181) For ∥ h ∥ 2 ≤ ε P \lVert\mathbf{h}\rVert_2\le\varepsilon_P ∥ h ∥ 2 ≤ ε P , define
d P h d P ( x ) : = exp { θ ( h ) ⊤ ( x − μ ) − Λ P ( θ ( h ) ) } . (182) \frac{\mathrm{d}P_{\mathbf{h}}}{\mathrm{d}P}(x):=\exp\{\boldsymbol{\theta}(\mathbf{h})^{\top}(x-\boldsymbol{\mu})-\Lambda_P(\boldsymbol{\theta}(\mathbf{h}))\}. \tag*{(182)} d P d P h ( x ) := exp { θ ( h ) ⊤ ( x − μ ) − Λ P ( θ ( h ))} . (182)
Then P h P_h P h is a probability measure equivalent to P P P , with P 0 = P P_0 = P P 0 = P and E P h x = μ + h \mathbb{E}_{P_h}x = \mu+ h E P h x = μ + h . Uniformly as h → 0 h \to0 h → 0 ,
Λ P ( θ ( h ) ) = 1 2 h ⊤ Σ − 1 h + O ( ∥ h ∥ 2 3 ) , (183) \Lambda_P(\theta(h)) = \frac{1}{2}h^{\top}\Sigma^{-1}h + O(\lVert h\rVert_2^3), \tag*{(183)} Λ P ( θ ( h )) = 2 1 h ⊤ Σ − 1 h + O (∥ h ∥ 2 3 ) , (183)
and
D K L ( P h ∥ P ) = θ ( h ) ⊤ h − Λ P ( θ ( h ) ) = 1 2 h ⊤ Σ − 1 h + O ( ∥ h ∥ 2 3 ) . (184) D_{\mathrm{KL}}(P_h\Vert P) = \theta(h)^{\top}h - \Lambda_P(\theta(h)) = \frac{1}{2}h^{\top}\Sigma^{-1}h + O(\lVert h\rVert_2^3). \tag*{(184)} D KL ( P h ∥ P ) = θ ( h ) ⊤ h − Λ P ( θ ( h )) = 2 1 h ⊤ Σ − 1 h + O (∥ h ∥ 2 3 ) . (184)
Proof. Fix 0 < ρ < η / ( 4 d ) 0 < \rho< \eta/(4\sqrt{d}) 0 < ρ < η / ( 4 d ) . Then
E P e 4 ρ ∥ x − μ ∥ 2 ≤ ∑ v ∈ { − 1 , 1 } d E P e 4 ρ v ⊤ ( x − μ ) = ∑ v ∈ { − 1 , 1 } d e Λ P ( 4 ρ v ) < ∞ . \mathbb{E}_P e^{4\rho\lVert x-\mu\rVert_2} \le\sum_{v\in\{-1,1\}^d}\mathbb{E}_P e^{4\rho v^{\top}(x-\mu)} = \sum_{v\in\{-1,1\}^d}e^{\Lambda_P(4\rho v)} < \infty. E P e 4 ρ ∥ x − μ ∥ 2 ≤ v ∈ { − 1 , 1 } d ∑ E P e 4 ρ v ⊤ ( x − μ ) = v ∈ { − 1 , 1 } d ∑ e Λ P ( 4 ρ v ) < ∞.
For each integer k ≥ 0 k \ge0 k ≥ 0 ,
sup ∥ θ ∥ 2 ≤ 7 ρ / 2 ∥ x − μ ∥ 2 k e θ ⊤ ( x − μ ) ≤ k ! ( 2 / ρ ) k e 4 ρ ∥ x − μ ∥ 2 . \sup_{\lVert\theta\rVert_2\le7\rho/2}\lVert x-\mu\rVert_2^k e^{\theta^{\top}(x-\mu)} \le k!(2/\rho)^k e^{4\rho\lVert x-\mu\rVert_2}. ∥ θ ∥ 2 ≤ 7 ρ /2 sup ∥ x − μ ∥ 2 k e θ ⊤ ( x − μ ) ≤ k ! ( 2/ ρ ) k e 4 ρ ∥ x − μ ∥ 2 .
Differentiation under the integral in (43) gives Λ P ∈ C ∞ \Lambda_P \in C^\infty Λ P ∈ C ∞ on a neighborhood of B 2 ( 3 ρ ) B_2(3\rho) B 2 ( 3 ρ ) , with
∇ Λ P ( θ ) = E P [ ( x − μ ) e θ ⊤ ( x − μ ) ] E P e θ ⊤ ( x − μ ) \nabla\Lambda_P(\theta) = \frac{\mathbb{E}_P[(x-\mu)e^{\theta^{\top}(x-\mu)}]}{\mathbb{E}_P e^{\theta^{\top}(x-\mu)}} ∇ Λ P ( θ ) = E P e θ ⊤ ( x − μ ) E P [( x − μ ) e θ ⊤ ( x − μ ) ]
and
∇ 2 Λ P ( θ ) = E P [ ( x − μ ) ( x − μ ) ⊤ e θ ⊤ ( x − μ ) ] E P e θ ⊤ ( x − μ ) − ∇ Λ P ( θ ) ∇ Λ P ( θ ) ⊤ . \nabla^2\Lambda_P(\theta) = \frac{\mathbb{E}_P[(x-\mu)(x-\mu)^{\top}e^{\theta^{\top}(x-\mu)}]}{\mathbb{E}_P e^{\theta^{\top}(x-\mu)}} - \nabla\Lambda_P(\theta)\nabla\Lambda_P(\theta)^{\top}. ∇ 2 Λ P ( θ ) = E P e θ ⊤ ( x − μ ) E P [( x − μ ) ( x − μ ) ⊤ e θ ⊤ ( x − μ ) ] − ∇ Λ P ( θ ) ∇ Λ P ( θ ) ⊤ .
In particular, Λ P ( 0 ) = 0 \Lambda_P(0) = 0 Λ P ( 0 ) = 0 , ∇ Λ P ( 0 ) = 0 \nabla\Lambda_P(0) = 0 ∇ Λ P ( 0 ) = 0 , and ∇ 2 Λ P ( 0 ) = Σ ≻ 0 \nabla^2\Lambda_P(0) = \Sigma\succ0 ∇ 2 Λ P ( 0 ) = Σ ≻ 0 . Choose 0 < ρ P ≤ ρ 0 < \rho_P \le\rho 0 < ρ P ≤ ρ with
inf ∥ θ ∥ 2 ≤ 3 ρ P λ min ( ∇ 2 Λ P ( θ ) ) ≥ λ / 2 > 0. \inf_{\lVert\theta\rVert_2\le3\rho_P}\lambda_{\min}(\nabla^2\Lambda_P(\theta)) \ge\lambda/2 > 0. ∥ θ ∥ 2 ≤ 3 ρ P inf λ m i n ( ∇ 2 Λ P ( θ )) ≥ λ /2 > 0.
The inverse function theorem applied to ∇ Λ P \nabla\Lambda_P ∇ Λ P at 0 0 0 gives a C ∞ C^\infty C ∞ inverse θ ( h ) \theta(h) θ ( h ) on a neighborhood of 0 0 0 . Choose ε P > 0 \varepsilon_P > 0 ε P > 0 so that this inverse is defined on a neighborhood of B 2 ( ε P ) B_2(\varepsilon_P) B 2 ( ε P ) and θ ( B 2 ( ε P ) ) ⊆ B 2 ( ρ P ) \theta(B_2(\varepsilon_P)) \subseteq B_2(\rho_P) θ ( B 2 ( ε P )) ⊆ B 2 ( ρ P ) . The chain rule gives
∇ 2 Λ P ( θ ( h ) ) D θ ( h ) = I d , \nabla^2\Lambda_P(\theta(h))D\theta(h) = I_d, ∇ 2 Λ P ( θ ( h )) D θ ( h ) = I d ,
so D θ ( 0 ) = Σ − 1 D\theta(0) = \Sigma^{-1} D θ ( 0 ) = Σ − 1 . Taylor’s formula yields
θ ( h ) = ∫ 0 1 D θ ( s h ) [ h ] d s \theta(h) = \int_0^1 D\theta(sh)[h]\,\mathrm{d}s θ ( h ) = ∫ 0 1 D θ ( s h ) [ h ] d s
and
θ ( h ) − Σ − 1 h = ∫ 0 1 ( 1 − s ) D 2 θ ( s h ) [ h , h ] d s . \theta(h) - \Sigma^{-1}h = \int_0^1(1-s)D^2\theta(sh)[h,h]\,\mathrm{d}s. θ ( h ) − Σ − 1 h = ∫ 0 1 ( 1 − s ) D 2 θ ( s h ) [ h , h ] d s .
Take C P ≥ 1 C_P \ge1 C P ≥ 1 bounding ∥ D θ ∥ o p \lVert D\theta\rVert_{\mathrm{op}} ∥ D θ ∥ op , ∥ D 2 θ ∥ o p \lVert D^2\theta\rVert_{\mathrm{op}} ∥ D 2 θ ∥ op , ∥ ∇ 2 Λ P ∥ o p \lVert\nabla^2\Lambda_P\rVert_{\mathrm{op}} ∥ ∇ 2 Λ P ∥ op , and ∥ D 3 Λ P ∥ o p \lVert D^3\Lambda_P\rVert_{\mathrm{op}} ∥ D 3 Λ P ∥ op on the respective compact balls. Then (H.7) and (H.8) follow.
By the definition of P h P_h P h ,
∫ d P h = e − Λ P ( θ ( h ) ) E P e θ ( h ) ⊤ ( x − μ ) = 1. \int\mathrm{d}P_h = e^{-\Lambda_P(\theta(h))}\mathbb{E}_P e^{\theta(h)^\top(x-\mu)} = 1. ∫ d P h = e − Λ P ( θ ( h )) E P e θ ( h ) ⊤ ( x − μ ) = 1.
The derivative formulas for Λ P \Lambda_P Λ P give
E P h ( x − μ ) = ∇ Λ P ( θ ( h ) ) = h \mathbb{E}_{P_h}(x-\mu) = \nabla\Lambda_P(\theta(h)) = h E P h ( x − μ ) = ∇ Λ P ( θ ( h )) = h
and P 0 = P P_0 = P P 0 = P since θ ( 0 ) = 0 \theta(0) = 0 θ ( 0 ) = 0 and Λ P ( 0 ) = 0 \Lambda_P(0) = 0 Λ P ( 0 ) = 0 . Also, 0 < d P h / d P < ∞ 0 < \mathrm{d}P_h/\mathrm{d}P < \infty 0 < d P h / d P < ∞ P P P -almost surely, proving mutual absolute continuity.
Finally, Λ P ( θ ) ≥ E P [ θ ⊤ ( x − μ ) ] = 0 \Lambda_P(\theta) \geq\mathbb{E}_P[\theta^\top(x-\mu)] = 0 Λ P ( θ ) ≥ E P [ θ ⊤ ( x − μ )] = 0 , and Taylor’s formula gives
Λ P ( θ ) = ∫ 0 1 ( 1 − s ) θ ⊤ ∇ 2 Λ P ( s θ ) θ d s \Lambda_P(\theta) = \int_0^1 (1-s)\theta^\top\nabla^2\Lambda_P(s\theta)\theta\,\mathrm{d}s Λ P ( θ ) = ∫ 0 1 ( 1 − s ) θ ⊤ ∇ 2 Λ P ( s θ ) θ d s
and
Λ P ( θ ) − 1 2 θ ⊤ Σ θ = 1 2 ∫ 0 1 ( 1 − s ) 2 D 3 Λ P ( s θ ) [ θ , θ , θ ] d s . \Lambda_P(\theta) - \frac{1}{2}\theta^\top\Sigma\theta= \frac{1}{2}\int_0^1 (1-s)^2D^3\Lambda_P(s\theta)[\theta,\theta,\theta] \,\mathrm{d}s. Λ P ( θ ) − 2 1 θ ⊤ Σ θ = 2 1 ∫ 0 1 ( 1 − s ) 2 D 3 Λ P ( s θ ) [ θ , θ , θ ] d s .
For ∥ θ ∥ 2 ≤ 3 ρ P \|\theta\|_2 \leq3\rho_P ∥ θ ∥ 2 ≤ 3 ρ P , the bound on D 3 Λ P D^3\Lambda_P D 3 Λ P gives
∣ Λ P ( θ ) − 1 2 θ ⊤ Σ θ ∣ ≤ 1 2 ∫ 0 1 ( 1 − s ) 2 C P ∥ θ ∥ 2 3 d s = C P 6 ∥ θ ∥ 2 3 . \left|\Lambda_P(\theta) - \frac{1}{2}\theta^\top\Sigma\theta\right| \leq\frac{1}{2}\int_0^1(1-s)^2C_P\|\theta\|_2^3 \,\mathrm{d}s = \frac{C_P}{6}\|\theta\|_2^3. Λ P ( θ ) − 2 1 θ ⊤ Σ θ ≤ 2 1 ∫ 0 1 ( 1 − s ) 2 C P ∥ θ ∥ 2 3 d s = 6 C P ∥ θ ∥ 2 3 .
These identities and the bound on ∇ 2 Λ P \nabla^2\Lambda_P ∇ 2 Λ P prove (H.9).
Since C P ≥ 1 C_P \geq1 C P ≥ 1 , ∥ Σ ∥ o p ≤ C P \|\Sigma\|_{\mathrm{op}} \leq C_P ∥Σ ∥ op ≤ C P , and ∥ Σ − 1 ∥ o p ≤ C P \|\Sigma^{-1}\|_{\mathrm{op}} \leq C_P ∥ Σ − 1 ∥ op ≤ C P , they also give, for ∥ h ∥ 2 ≤ ε P \|h\|_2 \leq\varepsilon_P ∥ h ∥ 2 ≤ ε P ,
∣ Λ P ( θ ( h ) ) − 1 2 h ⊤ Σ − 1 h ∣ ≤ ∣ Λ P ( θ ( h ) ) − 1 2 θ ( h ) ⊤ Σ θ ( h ) ∣ + 1 2 ∣ h ⊤ Σ − 1 h − θ ( h ) ⊤ Σ θ ( h ) ∣ ≤ C P 6 ∥ θ ( h ) ∥ 2 3 + 1 2 ∣ ( θ ( h ) − Σ − 1 h ) ⊤ Σ ( θ ( h ) + Σ − 1 h ) ∣ ≤ C P 6 ∥ θ ( h ) ∥ 2 3 + ∥ Σ ∥ o p 2 ∥ θ ( h ) − Σ − 1 h ∥ 2 ( ∥ θ ( h ) ∥ 2 + ∥ Σ − 1 h ∥ 2 ) . \begin{aligned}
\left|\Lambda_P(\theta(h))-\frac{1}{2}h^\top\Sigma^{-1}h\right|
&\leq\left|\Lambda_P(\theta(h))-\frac{1}{2}\theta(h)^\top\Sigma\theta(h)\right|+\frac{1}{2}\left|h^\top\Sigma^{-1}h-\theta(h)^\top\Sigma\theta(h)\right|\\
&\leq\frac{C_P}{6}\|\theta(h)\|_2^3+\frac{1}{2}\left|(\theta(h)-\Sigma^{-1}h)^\top\Sigma(\theta(h)+\Sigma^{-1}h)\right|\\
&\leq\frac{C_P}{6}\|\theta(h)\|_2^3+\frac{\|\Sigma\|_{\mathrm{op}}}{2}\|\theta(h)-\Sigma^{-1}h\|_2(\|\theta(h)\|_2+\|\Sigma^{-1}h\|_2).
\end{aligned} Λ P ( θ ( h )) − 2 1 h ⊤ Σ − 1 h ≤ Λ P ( θ ( h )) − 2 1 θ ( h ) ⊤ Σ θ ( h ) + 2 1 h ⊤ Σ − 1 h − θ ( h ) ⊤ Σ θ ( h ) ≤ 6 C P ∥ θ ( h ) ∥ 2 3 + 2 1 ( θ ( h ) − Σ − 1 h ) ⊤ Σ ( θ ( h ) + Σ − 1 h ) ≤ 6 C P ∥ θ ( h ) ∥ 2 3 + 2 ∥Σ ∥ op ∥ θ ( h ) − Σ − 1 h ∥ 2 ( ∥ θ ( h ) ∥ 2 + ∥ Σ − 1 h ∥ 2 ) .
Consequently, (H.11) follows from
∣ Λ P ( θ ( h ) ) − 1 2 h ⊤ Σ − 1 h ∣ ≤ ( C P 4 / 6 + C P 3 ) ∥ h ∥ 2 3 ≤ 2 C P 4 ∥ h ∥ 2 3 . \left|\Lambda_P(\theta(h))-\frac{1}{2}h^\top\Sigma^{-1}h\right| \leq(C_P^4/6+C_P^3)\|h\|_2^3 \leq2C_P^4\|h\|_2^3. Λ P ( θ ( h )) − 2 1 h ⊤ Σ − 1 h ≤ ( C P 4 /6 + C P 3 ) ∥ h ∥ 2 3 ≤ 2 C P 4 ∥ h ∥ 2 3 .
Finally, taking the expectation of the log density under P h P_h P h gives
D K L ( P h ∥ P ) = E P h [ θ ( h ) ⊤ ( x − μ ) − Λ P ( θ ( h ) ) ] = θ ( h ) ⊤ h − Λ P ( θ ( h ) ) . \begin{aligned}
D_{\mathrm{KL}}(P_h\|P) &= \mathbb{E}_{P_h}\left[\theta(h)^\top(x-\mu)-\Lambda_P(\theta(h))\right]\\
&=\theta(h)^\top h-\Lambda_P(\theta(h)).
\end{aligned} D KL ( P h ∥ P ) = E P h [ θ ( h ) ⊤ ( x − μ ) − Λ P ( θ ( h )) ] = θ ( h ) ⊤ h − Λ P ( θ ( h )) .
By (H.8),
∣ θ ( h ) ⊤ h − h ⊤ Σ − 1 h ∣ ≤ ∥ θ ( h ) − Σ − 1 h ∥ 2 ∥ h ∥ 2 ≤ C P ∥ h ∥ 2 3 . \left|\theta(h)^\top h-h^\top\Sigma^{-1}h\right| \leq\|\theta(h)-\Sigma^{-1}h\|_2\|h\|_2 \leq C_P\|h\|_2^3. θ ( h ) ⊤ h − h ⊤ Σ − 1 h ≤ ∥ θ ( h ) − Σ − 1 h ∥ 2 ∥ h ∥ 2 ≤ C P ∥ h ∥ 2 3 .
Together with (H.11), this yields, uniformly as h → 0 h\to0 h → 0 ,
D K L ( P h ∥ P ) = 1 2 h ⊤ Σ − 1 h + O ( ∥ h ∥ 2 3 ) . D_{\mathrm{KL}}(P_h\|P)=\frac{1}{2}h^\top\Sigma^{-1}h+O(\|h\|_2^3). D KL ( P h ∥ P ) = 2 1 h ⊤ Σ − 1 h + O ( ∥ h ∥ 2 3 ) .
Lemma H.3 (A Bayes lower bound from a local likelihood limit). Let P P P be a reference law with E P ∥ x ∥ 2 < ∞ \mathbb{E}_P\lVert x\rVert_2 < \infty E P ∥ x ∥ 2 < ∞ and E P x = μ \mathbb{E}_P x = \mu E P x = μ . Suppose (C.1) holds, and fix M > 0 M > 0 M > 0 and an invertible matrix Γ ∈ R d × d \Gamma\in\mathbb{R}^{d \times d} Γ ∈ R d × d . Consider a submodel Q n , u ≪ P Q_{n,u} \ll P Q n , u ≪ P , u ∈ B 2 ( M ) u \in B_2(M) u ∈ B 2 ( M ) , with
E Q n , u x = μ + Γ u n , L n ( u ) : = d Q n , u ⊗ n d P ⊗ n ( x 1 , … , x n ) . \mathbb{E}_{Q_{n,u}} x = \mu+ \frac{\Gamma u}{\sqrt{n}}, \qquad L_n(u) := \frac{\mathrm{d}Q_{n,u}^{\otimes n}}{\mathrm{d}P^{\otimes n}}(x_1,\ldots,x_n). E Q n , u x = μ + n Γ u , L n ( u ) := d P ⊗ n d Q n , u ⊗ n ( x 1 , … , x n ) . Assume L n L_n L n has strictly positive continuous sample paths on B 2 ( M ) B_2(M) B 2 ( M ) for all sufficiently large n n n , and, under P ⊗ n P^{\otimes n} P ⊗ n ,
L n ⟶ d L in C ( B 2 ( M ) ) , L ( u ) = e u T Y − ∥ u ∥ 2 2 / 2 , Y ∼ N ( 0 , I d ) , (185) L_n \overset{d}{\longrightarrow} L \quad\text{in } C(B_2(M)), \qquad L(u) = e^{u^{\mathsf{T}}Y-\lVert u\rVert_2^2/2}, \qquad Y \sim\mathcal{N}(0,I_d), \tag*{(185)} L n ⟶ d L in C ( B 2 ( M )) , L ( u ) = e u T Y − ∥ u ∥ 2 2 /2 , Y ∼ N ( 0 , I d ) , (185) with sup n l a r g e E P ⊗ n ∥ L n ∥ ∞ 2 < ∞ \sup_{n\ \mathrm{large}}\mathbb{E}_{P^{\otimes n}}\lVert L_n\rVert_\infty^2 < \infty sup n large E P ⊗ n ∥ L n ∥ ∞ 2 < ∞ . Then
lim inf n → ∞ inf w ^ n sup u ∈ B 2 ( M ) E Q n , u ⊗ n ℓ n ( Γ u , w ^ n ) a ( n − 1 / 2 ) ≥ V M , I d ( ψ ∘ Γ ) . (186) \liminf_{n\to\infty}\inf_{\widehat{w}_n}\sup_{u\in B_2(M)}\frac{\mathbb{E}_{Q_{n,u}^{\otimes n}}\ell_n(\Gamma u,\widehat{w}_n)}{a(n^{-1/2})} \ge V_{M,I_d}(\psi\circ\Gamma). \tag*{(186)} n → ∞ lim inf w n inf u ∈ B 2 ( M ) sup a ( n − 1/2 ) E Q n , u ⊗ n ℓ n ( Γ u , w n ) ≥ V M , I d ( ψ ∘ Γ ) . (186) The same lower bound applies to any distribution neighborhood containing this submodel for all sufficiently large n n n . The infimum is over measurable sample-based policies into Ω \Omega Ω .
Proof. Since C ( B 2 ( M ) ) C(B_2(M)) C ( B 2 ( M )) is complete and separable, the Skorokhod representation theorem supplies copies of L n L_n L n , L L L for which ∥ L n − L ∥ ∞ → 0 \lVert L_n-L\rVert_\infty\to0 ∥ L n − L ∥ ∞ → 0 almost surely. Their laws, including the stated moment bounds, are preserved. Also,
E ∥ L ∥ ∞ 2 ≤ E e 2 M ∥ Y ∥ 2 < ∞ . \mathbb{E}\lVert L\rVert_\infty^2 \le\mathbb{E}e^{2M\lVert Y\rVert_2} < \infty. E ∥ L ∥ ∞ 2 ≤ E e 2 M ∥ Y ∥ 2 < ∞.
For K > 0 K > 0 K > 0 ,
E ∥ L n − L ∥ ∞ ≤ E min { ∥ L n − L ∥ ∞ , K } + K − 1 E ∥ L n − L ∥ ∞ 2 . \mathbb{E}\lVert L_n-L\rVert_\infty\le\mathbb{E}\min\{\lVert L_n-L\rVert_\infty,K\}+K^{-1}\mathbb{E}\lVert L_n-L\rVert_\infty^2. E ∥ L n − L ∥ ∞ ≤ E min {∥ L n − L ∥ ∞ , K } + K − 1 E ∥ L n − L ∥ ∞ 2 .
Letting first n → ∞ n \to\infty n → ∞ and then K → ∞ K \to\infty K → ∞ gives, on this coupling,
E ∥ L n − L ∥ ∞ → 0. (187) \mathbb{E}\lVert L_n-L\rVert_\infty\to0. \tag*{(187)} E ∥ L n − L ∥ ∞ → 0. (187)
For π ∈ P ( B 2 ( M ) ) \pi\in\mathcal{P}(B_2(M)) π ∈ P ( B 2 ( M )) and positive continuous l l l , define
J π ( l ) : = ∫ ψ ( Γ u ) l ( u ) π ( d u ) − ( ∫ l d π ) ψ ( Γ ∫ u l ( u ) π ( d u ) ∫ l d π ) . J_\pi(l) := \int\psi(\Gamma u)l(u)\,\pi(\mathrm{d}u)-\left(\int l\,\mathrm{d}\pi\right)\psi\left(\Gamma\frac{\int ul(u)\,\pi(\mathrm{d}u)}{\int l\,\mathrm{d}\pi}\right). J π ( l ) := ∫ ψ ( Γ u ) l ( u ) π ( d u ) − ( ∫ l d π ) ψ ( Γ ∫ l d π ∫ u l ( u ) π ( d u ) ) .
All integrals are over B 2 ( M ) B_2(M) B 2 ( M ) . Since ∫ l d π > 0 \int l\,\mathrm{d}\pi> 0 ∫ l d π > 0 , the measure l d π / ∫ l d π l\,\mathrm{d}\pi/\int l\,\mathrm{d}\pi l d π / ∫ l d π is a probability measure on B 2 ( M ) B_2(M) B 2 ( M ) , and
J π ( l ) = ( ∫ l d π ) { ∫ ψ ( Γ u ) l ( u ) π ( d u ) ∫ l d π − ψ ( Γ ∫ u l d π ∫ l d π ) } ≥ 0. J_\pi(l)=\left(\int l\,\mathrm{d}\pi\right)\left\{\frac{\int\psi(\Gamma u)l(u)\,\pi(\mathrm{d}u)}{\int l\,\mathrm{d}\pi}-\psi\left(\Gamma\frac{\int ul\,\mathrm{d}\pi}{\int l\,\mathrm{d}\pi}\right)\right\}\ge0. J π ( l ) = ( ∫ l d π ) { ∫ l d π ∫ ψ ( Γ u ) l ( u ) π ( d u ) − ψ ( Γ ∫ l d π ∫ u l d π ) } ≥ 0.
For l = L n l=L_n l = L n , Bayes’ formula identifies L n d π / ∫ L n d π L_n\,\mathrm{d}\pi/\int L_n\,\mathrm{d}\pi L n d π / ∫ L n d π as the posterior distribution, while ∫ L n d π \int L_n\,\mathrm{d}\pi ∫ L n d π is the density of the prior mixture of the sample laws relative to P ⊗ n P^{\otimes n} P ⊗ n . Thus J π ( L n ) J_\pi(L_n) J π ( L n ) is the posterior Jensen gap multiplied by this mixture density. The finite convex function ψ ∘ Γ \psi\circ\Gamma ψ ∘ Γ is bounded and Lipschitz on B 2 ( M ) B_2(M) B 2 ( M ) . Choose C C C bounding its absolute value and a Lipschitz constant there. For positive continuous l , k l,k l , k , their weighted means belong to B 2 ( M ) B_2(M) B 2 ( M ) , since
∥ ∫ u l d π ∫ l d π ∥ 2 ≤ ∫ ∥ u ∥ 2 l d π ∫ l d π ≤ M , ∥ ∫ u k d π ∫ k d π ∥ 2 ≤ M . \left\|\frac{\int ul\,\mathrm{d}\pi}{\int l\,\mathrm{d}\pi}\right\|_2 \le\frac{\int\|u\|_2l\,\mathrm{d}\pi}{\int l\,\mathrm{d}\pi} \le M,\qquad\left\|\frac{\int uk\,\mathrm{d}\pi}{\int k\,\mathrm{d}\pi}\right\|_2 \le M. ∫ l d π ∫ u l d π 2 ≤ ∫ l d π ∫ ∥ u ∥ 2 l d π ≤ M , ∫ k d π ∫ u k d π 2 ≤ M .
The identity
( ∫ l d π ) ( ∫ u l d π ∫ l d π − ∫ u k d π ∫ k d π ) = ∫ u ( l − k ) d π − ( ∫ ( l − k ) d π ) ∫ u k d π ∫ k d π \left(\int l\,\mathrm{d}\pi\right)\left(\frac{\int ul\,\mathrm{d}\pi}{\int l\,\mathrm{d}\pi}-\frac{\int uk\,\mathrm{d}\pi}{\int k\,\mathrm{d}\pi}\right)=\int u(l-k)\,\mathrm{d}\pi-\left(\int(l-k)\,\mathrm{d}\pi\right)\frac{\int uk\,\mathrm{d}\pi}{\int k\,\mathrm{d}\pi} ( ∫ l d π ) ( ∫ l d π ∫ u l d π − ∫ k d π ∫ u k d π ) = ∫ u ( l − k ) d π − ( ∫ ( l − k ) d π ) ∫ k d π ∫ u k d π
therefore gives
( ∫ l d π ) ∥ ∫ u l d π ∫ l d π − ∫ u k d π ∫ k d π ∥ 2 ≤ ∥ ∫ u ( l − k ) d π ∥ 2 + M ∣ ∫ ( l − k ) d π ∣ ≤ 2 M ∥ l − k ∥ ∞ . \left(\int l\,\mathrm{d}\pi\right)\left\|\frac{\int ul\,\mathrm{d}\pi}{\int l\,\mathrm{d}\pi}-\frac{\int uk\,\mathrm{d}\pi}{\int k\,\mathrm{d}\pi}\right\|_2
\le\left\|\int u(l-k)\,\mathrm{d}\pi\right\|_2+M\left|\int(l-k)\,\mathrm{d}\pi\right|
\le2M\|l-k\|_\infty. ( ∫ l d π ) ∫ l d π ∫ u l d π − ∫ k d π ∫ u k d π 2 ≤ ∫ u ( l − k ) d π 2 + M ∫ ( l − k ) d π ≤ 2 M ∥ l − k ∥ ∞ .
The first integral in J π ( l ) − J π ( k ) J_\pi(l)-J_\pi(k) J π ( l ) − J π ( k ) is bounded by C ∥ l − k ∥ ∞ C\|l-k\|_\infty C ∥ l − k ∥ ∞ . For the posterior term, add and subtract the expression with mass ∫ l d π \int l\,\mathrm{d}\pi ∫ l d π and the k k k -weighted mean. The preceding mean bound and the common Lipschitz constant give ( 2 C M + C ) ∥ l − k ∥ ∞ (2CM+C)\|l-k\|_\infty ( 2 C M + C ) ∥ l − k ∥ ∞ . Combining the two bounds gives
∣ J π ( l ) − J π ( k ) ∣ ≤ 2 C ( 1 + M ) ∥ l − k ∥ ∞ , π ∈ P ( B 2 ( M ) ) . (188) |J_\pi(l)-J_\pi(k)|\le2C(1+M)\|l-k\|_\infty,\qquad\pi\in\mathcal{P}(B_2(M)). \tag*{(188)} ∣ J π ( l ) − J π ( k ) ∣ ≤ 2 C ( 1 + M ) ∥ l − k ∥ ∞ , π ∈ P ( B 2 ( M )) . (188)
The constant is independent of π \pi π , and no uniform positive lower bound on ∫ l d π \int l\,\mathrm{d}\pi ∫ l d π or ∫ k d π \int k\,\mathrm{d}\pi ∫ k d π is needed. On the coupling used in (H.15), preservation of the marginal laws gives, for every π \pi π ,
∣ E P ⊗ n J π ( L n ) − E N ( 0 , I d ) J π ( L ) ∣ = ∣ E [ J π ( L n ) − J π ( L ) ] ∣ ≤ E ∣ J π ( L n ) − J π ( L ) ∣ ≤ 2 C ( 1 + M ) E ∥ L n − L ∥ ∞ . \left|\mathbb{E}_{\mathbb{P}^{\otimes n}}J_\pi(L_n)-\mathbb{E}_{\mathcal{N}(0,I_d)}J_\pi(L)\right|
=\left|\mathbb{E}[J_\pi(L_n)-J_\pi(L)]\right|
\le\mathbb{E}|J_\pi(L_n)-J_\pi(L)|
\le2C(1+M)\mathbb{E}\|L_n-L\|_\infty. E P ⊗ n J π ( L n ) − E N ( 0 , I d ) J π ( L ) = ∣ E [ J π ( L n ) − J π ( L )] ∣ ≤ E ∣ J π ( L n ) − J π ( L ) ∣ ≤ 2 C ( 1 + M ) E ∥ L n − L ∥ ∞ .
The expectations on the right are taken on the common coupling space. Taking the supremum over π \pi π and using (H.15) yields
sup π ∈ P ( B 2 ( M ) ) ∣ E P ⊗ n J π ( L n ) − E N ( 0 , I d ) J π ( L ) ∣ ≤ 2 C ( 1 + M ) E ∥ L n − L ∥ ∞ ⟶ 0. (189) \sup_{\pi\in\mathcal{P}(B_2(M))}
\left|\mathbb{E}_{\mathbb{P}^{\otimes n}}J_\pi(L_n)-\mathbb{E}_{\mathcal{N}(0,I_d)}J_\pi(L)\right|
\le2C(1+M)\mathbb{E}\|L_n-L\|_\infty\longrightarrow0. \tag*{(189)} π ∈ P ( B 2 ( M )) sup E P ⊗ n J π ( L n ) − E N ( 0 , I d ) J π ( L ) ≤ 2 C ( 1 + M ) E ∥ L n − L ∥ ∞ ⟶ 0. (189)
Under the prior T ∼ π T\sim\pi T ∼ π and conditional sample law Q n , T ⊗ n Q^{\otimes n}_{n,T} Q n , T ⊗ n , write E π \mathbb{E}_\pi E π for the joint prior-sampling expectation. Since ∥ T ∥ 2 ≤ M \|T\|_2\le M ∥ T ∥ 2 ≤ M almost surely, conditional Jensen’s inequality gives ∥ E π [ T ∣ x 1 , … , x n ] ∥ 2 ≤ E π [ ∥ T ∥ 2 ∣ x 1 , … , x n ] ≤ M \|\mathbb{E}_\pi[T\mid x_1,\ldots,x_n]\|_2\le\mathbb{E}_\pi[\|T\|_2\mid x_1,\ldots,x_n]\le M ∥ E π [ T ∣ x 1 , … , x n ] ∥ 2 ≤ E π [ ∥ T ∥ 2 ∣ x 1 , … , x n ] ≤ M . For a fixed action w ∈ Ω w\in\Omega w ∈ Ω , the posterior expected loss is
E π [ ℓ n ( Γ T , w ) ∣ x 1 , … , x n ] = E π [ f ∗ ( μ + T ⊤ n ) | x 1 , … , x n ] − w ⊤ ( μ + E π [ T ∣ x 1 , … , x n ] n ) . \mathbb{E}_\pi[\ell_n(\Gamma T,w)\mid x_1,\ldots,x_n]
=\mathbb{E}_\pi\left[f_*\left(\mu+\frac{T^\top}{\sqrt{n}}\right)\middle|x_1,\ldots,x_n\right]
-w^\top\left(\mu+\frac{\mathbb{E}_\pi[T\mid x_1,\ldots,x_n]}{\sqrt{n}}\right). E π [ ℓ n ( Γ T , w ) ∣ x 1 , … , x n ] = E π [ f ∗ ( μ + n T ⊤ ) x 1 , … , x n ] − w ⊤ ( μ + n E π [ T ∣ x 1 , … , x n ] ) .
The first term is independent of w \boldsymbol{w} w . Maximizing the linear term over Ω \Omega Ω therefore gives the Bayes rule
w ∗ ( μ + E π [ T ∣ x 1 , … , x n ] n ) , \boldsymbol{w}_* \left(\mu+ \frac{\mathbb{E}_{\pi}[T \mid x_1,\ldots,x_n]}{\sqrt{n}}\right), w ∗ ( μ + n E π [ T ∣ x 1 , … , x n ] ) ,
and its integrated regret satisfies
inf w ^ n E π ℓ ( Γ T , w ^ n ) = E π [ f ∗ ( μ + Γ T n ) − f ∗ ( μ + Γ E π [ T ∣ x 1 , … , x n ] n ) ] = E π [ r μ ( Γ T n ) − r μ ( Γ E π [ T ∣ x 1 , … , x n ] n ) ] . \inf_{\widehat{\boldsymbol{w}}_n}\mathbb{E}_{\pi}\ell(\Gamma T,\widehat{\boldsymbol{w}}_n)=\mathbb{E}_{\pi}\left[f_*\left(\mu+\frac{\Gamma T}{\sqrt{n}}\right)-f_*\left(\mu+\frac{\Gamma\mathbb{E}_{\pi}[T\mid x_1,\ldots,x_n]}{\sqrt{n}}\right)\right]
=\mathbb{E}_{\pi}\left[r_{\mu}\left(\frac{\Gamma T}{\sqrt{n}}\right)-r_{\mu}\left(\frac{\Gamma\mathbb{E}_{\pi}[T\mid x_1,\ldots,x_n]}{\sqrt{n}}\right)\right]. w n inf E π ℓ ( Γ T , w n ) = E π [ f ∗ ( μ + n Γ T ) − f ∗ ( μ + n Γ E π [ T ∣ x 1 , … , x n ] ) ] = E π [ r μ ( n Γ T ) − r μ ( n Γ E π [ T ∣ x 1 , … , x n ] ) ] .
Bayes’ formula gives
E π [ T ∣ x 1 , … , x n ] = ∫ u L n ( u ) π ( d u ) ∫ L n d π . \mathbb{E}_{\pi}[T\mid x_1,\ldots,x_n]=\frac{\int uL_n(u)\,\pi(du)}{\int L_n\,d\pi}. E π [ T ∣ x 1 , … , x n ] = ∫ L n d π ∫ u L n ( u ) π ( d u ) .
For the first term of J π ( L n ) J_{\pi}(L_n) J π ( L n ) , using E P ⊗ n L n ( u ) = 1 \mathbb{E}_{P\otimes n}L_n(u)=1 E P ⊗ n L n ( u ) = 1 yields
E P ⊗ n ∫ ψ ( Γ u ) L n ( u ) π ( d u ) = ∫ ψ ( Γ u ) π ( d u ) = E π ψ ( Γ T ) . \mathbb{E}_{P\otimes n}\int\psi(\Gamma u)L_n(u)\,\pi(du)=\int\psi(\Gamma u)\,\pi(du)=\mathbb{E}_{\pi}\psi(\Gamma T). E P ⊗ n ∫ ψ ( Γ u ) L n ( u ) π ( d u ) = ∫ ψ ( Γ u ) π ( d u ) = E π ψ ( Γ T ) .
For the second term, the density of the prior mixture of the sample laws relative to P ⊗ n P^{\otimes n} P ⊗ n is ∫ L n d π \int L_n\,d\pi ∫ L n d π , so
E P ⊗ n [ ( ∫ L n d π ) ψ ( Γ ∫ u L n ( u ) π ( d u ) ∫ L n d π ) ] = E π ψ ( Γ E π [ T ∣ x 1 , … , x n ] ) . \mathbb{E}_{P\otimes n}\left[\left(\int L_n\,d\pi\right)\psi\left(\Gamma\frac{\int uL_n(u)\,\pi(du)}{\int L_n\,d\pi}\right)\right]=\mathbb{E}_{\pi}\psi\left(\Gamma\mathbb{E}_{\pi}[T\mid x_1,\ldots,x_n]\right). E P ⊗ n [ ( ∫ L n d π ) ψ ( Γ ∫ L n d π ∫ u L n ( u ) π ( d u ) ) ] = E π ψ ( Γ E π [ T ∣ x 1 , … , x n ] ) .
Consequently,
E P ⊗ n J π ( L n ) = E π [ ψ ( Γ T ) − ψ ( Γ E π [ T ∣ x 1 , … , x n ] ) ] . \mathbb{E}_{P\otimes n}J_{\pi}(L_n)=\mathbb{E}_{\pi}\left[\psi(\Gamma T)-\psi\left(\Gamma\mathbb{E}_{\pi}[T\mid x_1,\ldots,x_n]\right)\right]. E P ⊗ n J π ( L n ) = E π [ ψ ( Γ T ) − ψ ( Γ E π [ T ∣ x 1 , … , x n ] ) ] .
Both arguments belong to the same fixed ball:
∥ Γ T ∥ 2 ≤ ∥ T ∥ o p M , ∥ Γ E π [ T ∣ x 1 , … , x n ] ∥ 2 ≤ ∥ T ∥ o p M . \|\Gamma T\|_2\leq\|T\|_{\mathrm{op}}M,\qquad\|\Gamma\mathbb{E}_{\pi}[T\mid x_1,\ldots,x_n]\|_2\leq\|T\|_{\mathrm{op}}M. ∥Γ T ∥ 2 ≤ ∥ T ∥ op M , ∥Γ E π [ T ∣ x 1 , … , x n ] ∥ 2 ≤ ∥ T ∥ op M .
Write
m n : = E π [ T ∣ x 1 , … , x n ] . m_n:=\mathbb{E}_{\pi}[T\mid x_1,\ldots,x_n]. m n := E π [ T ∣ x 1 , … , x n ] .
The triangle inequality and the uniform local expansion therefore give
∣ inf w ^ n E π ℓ ( Γ T , w ^ n ) a ( n − 1 / 2 ) − E P ⊗ n J π ( L n ) ∣ ≤ E π ∣ r μ ( Γ T / n ) a ( n − 1 / 2 ) − ψ ( Γ T ) ∣ + E π ∣ r μ ( Γ m n / n ) a ( n − 1 / 2 ) − ψ ( Γ m n ) ∣ ≤ 2 sup t ∈ B 2 ( ∥ T ∥ o p M ) ∣ r μ ( t / n ) a ( n − 1 / 2 ) − ψ ( t ) ∣ → 0 , \left|\frac{\inf_{\widehat{\boldsymbol{w}}_n}\mathbb{E}_{\pi}\ell(\Gamma T,\widehat{\boldsymbol{w}}_n)}{a(n^{-1/2})}-\mathbb{E}_{P\otimes n}J_{\pi}(L_n)\right|
\leq\mathbb{E}_{\pi}\left|\frac{r_{\mu}(\Gamma T/\sqrt{n})}{a(n^{-1/2})}-\psi(\Gamma T)\right|+\mathbb{E}_{\pi}\left|\frac{r_{\mu}(\Gamma m_n/\sqrt{n})}{a(n^{-1/2})}-\psi(\Gamma m_n)\right|
\leq2\sup_{t\in B_2(\|T\|_{\mathrm{op}}M)}\left|\frac{r_{\mu}(t/\sqrt{n})}{a(n^{-1/2})}-\psi(t)\right|\to0, a ( n − 1/2 ) inf w n E π ℓ ( Γ T , w n ) − E P ⊗ n J π ( L n ) ≤ E π a ( n − 1/2 ) r μ ( Γ T / n ) − ψ ( Γ T ) + E π a ( n − 1/2 ) r μ ( Γ m n / n ) − ψ ( Γ m n ) ≤ 2 t ∈ B 2 ( ∥ T ∥ op M ) sup a ( n − 1/2 ) r μ ( t / n ) − ψ ( t ) → 0 ,
uniformly in π \pi π . In the Gaussian experiment, the same mixture calculation gives
E N ( 0 , I d ) J π ( L ) = E π , I d [ ψ ( Γ T ) − ψ ( Γ m π , I d ( Y ) ) ] . \mathbb{E}_{N(0,I_d)}J_{\pi}(L)=\mathbb{E}_{\pi,I_d}\left[\psi(\Gamma T)-\psi(\Gamma m_{\pi,I_d}(Y))\right]. E N ( 0 , I d ) J π ( L ) = E π , I d [ ψ ( Γ T ) − ψ ( Γ m π , I d ( Y )) ] .
For every sample-based policy w ^ n \widehat{w}_n w n and every π ∈ P ( B 2 ( M ) ) \pi\in\mathcal{P}(B_2(M)) π ∈ P ( B 2 ( M )) ,
sup u ∈ B 2 ( M ) E Q n , u n ℓ n ( Γ u , w ^ n ) ≥ ∫ B 2 ( M ) E Q n , u n ℓ n ( Γ u , w ^ n ) π ( d u ) = E π ℓ n ( Γ T , w ^ n ) . \sup_{u\in B_2(M)} \mathbb{E}_{Q_{n,u}^{n}}\ell_n(\Gamma u,\widehat{w}_n) \ge\int_{B_2(M)} \mathbb{E}_{Q_{n,u}^{n}}\ell_n(\Gamma u,\widehat{w}_n)\,\pi(\mathrm{d}u)=\mathbb{E}_{\pi}\ell_n(\Gamma T,\widehat{w}_n). u ∈ B 2 ( M ) sup E Q n , u n ℓ n ( Γ u , w n ) ≥ ∫ B 2 ( M ) E Q n , u n ℓ n ( Γ u , w n ) π ( d u ) = E π ℓ n ( Γ T , w n ) .
Taking the infimum over the same policy class on both sides and then the supremum over π \pi π gives
inf w ^ n sup u ∈ B 2 ( M ) E Q n , u n ℓ n ( Γ u , w ^ n ) ≥ sup π ∈ P ( B 2 ( M ) ) inf w ^ n E π ℓ n ( Γ T , w ^ n ) . \inf_{\widehat{w}_n}\sup_{u\in B_2(M)}\mathbb{E}_{Q_{n,u}^{n}}\ell_n(\Gamma u,\widehat{w}_n)\ge\sup_{\pi\in\mathcal{P}(B_2(M))}\inf_{\widehat{w}_n}\mathbb{E}_{\pi}\ell_n(\Gamma T,\widehat{w}_n). w n inf u ∈ B 2 ( M ) sup E Q n , u n ℓ n ( Γ u , w n ) ≥ π ∈ P ( B 2 ( M )) sup w n inf E π ℓ n ( Γ T , w n ) .
The preceding uniform approximation and (H.17) imply
sup π ∈ P ( B 2 ( M ) ) ∣ inf w ^ n E π ℓ n ( Γ T , w ^ n ) a ( n − 1 / 2 ) − E N ( 0 , I d ) J π ( L ) ∣ → 0. \sup_{\pi\in\mathcal{P}(B_2(M))}\left|\frac{\inf_{\widehat{w}_n}\mathbb{E}_{\pi}\ell_n(\Gamma T,\widehat{w}_n)}{a(n^{-1/2})}-\mathbb{E}_{\mathcal{N}(0,I_d)}J_{\pi}(L)\right|\to0. π ∈ P ( B 2 ( M )) sup a ( n − 1/2 ) inf w n E π ℓ n ( Γ T , w n ) − E N ( 0 , I d ) J π ( L ) → 0.
Using ∣ sup u − sup v ∣ ≤ sup ∣ u − v ∣ |\sup u-\sup v|\leq\sup|u-v| ∣ sup u − sup v ∣ ≤ sup ∣ u − v ∣ on P ( B 2 ( M ) ) \mathcal{P}(B_2(M)) P ( B 2 ( M )) , we therefore obtain
sup π ∈ P ( B 2 ( M ) ) inf w ^ n E π ℓ n ( Γ T , w ^ n ) a ( n − 1 / 2 ) → sup π ∈ P ( B 2 ( M ) ) E N ( 0 , I d ) J π ( L ) . \sup_{\pi\in\mathcal{P}(B_2(M))}\frac{\inf_{\widehat{w}_n}\mathbb{E}_{\pi}\ell_n(\Gamma T,\widehat{w}_n)}{a(n^{-1/2})}\to\sup_{\pi\in\mathcal{P}(B_2(M))}\mathbb{E}_{\mathcal{N}(0,I_d)}J_{\pi}(L). π ∈ P ( B 2 ( M )) sup a ( n − 1/2 ) inf w n E π ℓ n ( Γ T , w n ) → π ∈ P ( B 2 ( M )) sup E N ( 0 , I d ) J π ( L ) .
By the Gaussian identity above and the definition of V M , I d V_{M,I_d} V M , I d ,
sup π ∈ P ( B 2 ( M ) ) E N ( 0 , I d ) J π ( L ) = sup π ∈ P ( B 2 ( M ) ) E π , I d [ ( ψ ∘ Γ ) ( T ) − ( ψ ∘ Γ ) ( m π , I d ( Y ) ) ] = V M , I d ( ψ ∘ Γ ) . \sup_{\pi\in\mathcal{P}(B_2(M))}\mathbb{E}_{\mathcal{N}(0,I_d)}J_{\pi}(L)
=\sup_{\pi\in\mathcal{P}(B_2(M))}\mathbb{E}_{\pi,I_d}\left[(\psi\circ\Gamma)(T)-(\psi\circ\Gamma)(m_{\pi,I_d}(Y))\right]
=V_{M,I_d}(\psi\circ\Gamma). π ∈ P ( B 2 ( M )) sup E N ( 0 , I d ) J π ( L ) = π ∈ P ( B 2 ( M )) sup E π , I d [ ( ψ ∘ Γ ) ( T ) − ( ψ ∘ Γ ) ( m π , I d ( Y )) ] = V M , I d ( ψ ∘ Γ ) .
Consequently,
lim inf n → ∞ inf w ^ n sup u ∈ B 2 ( M ) E Q n , u n ℓ n ( Γ u , w ^ n ) a ( n − 1 / 2 ) ≥ lim n → ∞ sup π ∈ P ( B 2 ( M ) ) inf w ^ n E π ℓ n ( Γ T , w ^ n ) a ( n − 1 / 2 ) = V M , I d ( ψ ∘ Γ ) . \liminf_{n\to\infty}\inf_{\widehat{w}_n}\frac{\sup_{u\in B_2(M)}\mathbb{E}_{Q_{n,u}^{n}}\ell_n(\Gamma u,\widehat{w}_n)}{a(n^{-1/2})}
\geq\lim_{n\to\infty}\sup_{\pi\in\mathcal{P}(B_2(M))}\frac{\inf_{\widehat{w}_n}\mathbb{E}_{\pi}\ell_n(\Gamma T,\widehat{w}_n)}{a(n^{-1/2})}
=V_{M,I_d}(\psi\circ\Gamma). n → ∞ lim inf w n inf a ( n − 1/2 ) sup u ∈ B 2 ( M ) E Q n , u n ℓ n ( Γ u , w n ) ≥ n → ∞ lim π ∈ P ( B 2 ( M )) sup a ( n − 1/2 ) inf w n E π ℓ n ( Γ T , w n ) = V M , I d ( ψ ∘ Γ ) .
This proves (H.14) using the minimax–Bayes inequality.
Lemma H.4 (Gaussian comparison on an ellipsoid). Suppose (C.1) holds. For an invertible matrix Γ ∈ R d × d \Gamma\in\mathbb{R}^{d\times d} Γ ∈ R d × d , consider the observation Y ∼ N ( u , I d ) \mathbf{Y}\sim\mathcal{N}(\mathbf{u},I_d) Y ∼ N ( u , I d ) , u ∈ B 2 ( M ) \mathbf{u}\in B_2(M) u ∈ B 2 ( M ) , and loss ℓ n ( Γ u , w ) \ell_n(\Gamma\mathbf{u},\mathbf{w}) ℓ n ( Γ u , w ) . Then
lim n → ∞ inf δ sup u ∈ B 2 ( M ) E N ( u , I d ) ∫ Ω ℓ n ( Γ u , w ) δ ( d w ∣ Y ) a ( n − 1 / 2 ) = V M , I d ( ψ ∘ Γ ) . (190) \lim_{n\to\infty}\frac{\inf_{\delta}\sup_{\mathbf{u}\in B_2(M)}\mathbb{E}_{\mathcal{N}(\mathbf{u},I_d)}\int_{\Omega}\ell_n(\Gamma\mathbf{u},\mathbf{w})\,\delta(\mathrm{d}\mathbf{w}\mid\mathbf{Y})}{a(n^{-1/2})}=V_{M,I_d}(\psi\circ\Gamma). \tag*{(190)} n → ∞ lim a ( n − 1/2 ) inf δ sup u ∈ B 2 ( M ) E N ( u , I d ) ∫ Ω ℓ n ( Γ u , w ) δ ( d w ∣ Y ) = V M , I d ( ψ ∘ Γ ) . (190) Here δ \delta δ ranges over Markov kernels from R d \mathbb{R}^{d} R d to Ω \Omega Ω , with δ ( d w ∣ y ) \delta(\mathrm{d}\mathbf{w}\mid\mathbf{y}) δ ( d w ∣ y ) specifying the conditional distribution of w \mathbf{w} w given Y = y \mathbf{Y}=\mathbf{y} Y = y . The infimum is unchanged by restricting the kernels δ \delta δ to Ω n , L \Omega_{n,L} Ω n , L for any L ≥ ∥ Γ ∥ o p M L\geq\|\Gamma\|_{\mathrm{op}}M L ≥ ∥Γ ∥ op M . This restriction means δ ( Ω n , L ∣ y ) = 1 \delta(\Omega_{n,L}\mid\mathbf{y})=1 δ ( Ω n , L ∣ y ) = 1 for every y ∈ R d \mathbf{y}\in\mathbb{R}^{d} y ∈ R d . The map ( M , Γ ) ↦ V M , I d ( ψ ∘ Γ ) (M,\Gamma)\mapsto V_{M,I_d}(\psi\circ\Gamma) ( M , Γ ) ↦ V M , I d ( ψ ∘ Γ ) is continuous for M > 0 M>0 M > 0 and invertible Γ \Gamma Γ .
Proof. Fix n n n and L ≥ ∥ Γ ∥ o p M L \ge\lVert\Gamma\rVert_{\mathrm{op}}M L ≥ ∥ Γ ∥ op M . The parameter space B 2 ( M ) B_2(M) B 2 ( M ) is compact, and the experiment Y ∼ N ( u , I d ) Y \sim\mathcal{N}(u,I_d) Y ∼ N ( u , I d ) has a strictly positive Lebesgue density. For u , v ∈ B 2 ( M ) u,v \in B_2(M) u , v ∈ B 2 ( M ) , the Gaussian KL formula and Pinsker’s inequality give
D K L ( N ( u , I d ) ∥ N ( v , I d ) ) = 1 2 ∥ u − v ∥ 2 2 , ∥ N ( u , I d ) − N ( v , I d ) ∥ T V ≤ 1 2 ∥ u − v ∥ 2 . D_{\mathrm{KL}}\bigl(\mathcal{N}(u,I_d)\Vert\mathcal{N}(v,I_d)\bigr)=\frac{1}{2}\lVert u-v\rVert_2^2,\qquad\lVert\mathcal{N}(u,I_d)-\mathcal{N}(v,I_d)\rVert_{\mathrm{TV}}\le\frac{1}{2}\lVert u-v\rVert_2. D KL ( N ( u , I d ) ∥ N ( v , I d ) ) = 2 1 ∥ u − v ∥ 2 2 , ∥ N ( u , I d ) − N ( v , I d ) ∥ TV ≤ 2 1 ∥ u − v ∥ 2 .
The loss ( u , w ) ↦ ℓ n ( Γ u , w ) (u,w)\mapsto\ell_n(\Gamma u,w) ( u , w ) ↦ ℓ n ( Γ u , w ) is jointly continuous and affine in w w w , and
0 ≤ ℓ n ( Γ u , w ) ≤ 2 R Ω ( ∥ u ∥ 2 + L / n ) , ( u , w ) ∈ B 2 ( M ) × Ω . 0\le\ell_n(\Gamma u,w)\le2R_{\Omega}(\lVert u\rVert_2+L/\sqrt{n}),\qquad(u,w)\in B_2(M)\times\Omega. 0 ≤ ℓ n ( Γ u , w ) ≤ 2 R Ω (∥ u ∥ 2 + L / n ) , ( u , w ) ∈ B 2 ( M ) × Ω.
For either decision space Ω \Omega Ω or Ω n , L \Omega_{n,L} Ω n , L , compactness and convexity hold; for Ω n , L \Omega_{n,L} Ω n , L these were proved in Lemma B.2. The class of all Markov kernels from R d \mathbb{R}^d R d into the chosen decision space is nonempty, convex, and weakly closed in the sense used in Lemma B.1. The same statistical minimax argument therefore applies to this experiment and loss.
For π ∈ P ( B 2 ( M ) ) \pi\in\mathcal{P}(B_2(M)) π ∈ P ( B 2 ( M )) , let T ∼ π T\sim\pi T ∼ π and Y = T + G Y=T+G Y = T + G , where G ∼ N ( 0 , I d ) G\sim\mathcal{N}(0,I_d) G ∼ N ( 0 , I d ) is independent of T T T . Then m π , I d ( Y ) = E π [ T ∣ Y ] m_{\pi,I_d}(Y)=\mathbb{E}_{\pi}[T\mid Y] m π , I d ( Y ) = E π [ T ∣ Y ] , and ∥ m π , I d ( Y ) ∥ 2 ≤ E π [ ∥ T ∥ 2 ∣ Y ] ≤ M \lVert m_{\pi,I_d}(Y)\rVert_2\le\mathbb{E}_{\pi}[\lVert T\rVert_2\mid Y]\le M ∥ m π , I d ( Y ) ∥ 2 ≤ E π [∥ T ∥ 2 ∣ Y ] ≤ M . For w ∈ Ω w\in\Omega w ∈ Ω , the posterior expected loss is
E π [ ℓ n ( Γ T , w ) ∣ Y ] = E π [ f ∗ ( μ + Γ T n ) | Y ] − w ⊤ ( μ + Γ m π , I d ( Y ) n ) . \mathbb{E}_{\pi}[\ell_n(\Gamma T,w)\mid Y]=\mathbb{E}_{\pi}\left[f_*\left(\mu+\frac{\Gamma T}{\sqrt{n}}\right)\middle|Y\right]-w^\top\left(\mu+\frac{\Gamma m_{\pi,I_d}(Y)}{\sqrt{n}}\right). E π [ ℓ n ( Γ T , w ) ∣ Y ] = E π [ f ∗ ( μ + n Γ T ) Y ] − w ⊤ ( μ + n Γ m π , I d ( Y ) ) .
The first term does not depend on w w w . Hence a Bayes rule is
w ∗ ( μ + Γ m π , I d ( Y ) n ) ∈ arg max w ∈ Ω ( μ + Γ m π , I d ( Y ) n ) ⊤ w ⊆ Ω n , L , w_*\left(\mu+\frac{\Gamma m_{\pi,I_d}(Y)}{\sqrt{n}}\right)\in\underset{w\in\Omega}{\operatorname{arg\,max}}\left(\mu+\frac{\Gamma m_{\pi,I_d}(Y)}{\sqrt{n}}\right)^\top w\subseteq\Omega_{n,L}, w ∗ ( μ + n Γ m π , I d ( Y ) ) ∈ w ∈ Ω arg max ( μ + n Γ m π , I d ( Y ) ) ⊤ w ⊆ Ω n , L ,
since ∥ Γ m π , I d ( Y ) ∥ 2 ≤ ∥ Γ ∥ o p M ≤ L \lVert\Gamma m_{\pi,I_d}(Y)\rVert_2\le\lVert\Gamma\rVert_{\mathrm{op}}M\le L ∥ Γ m π , I d ( Y ) ∥ 2 ≤ ∥ Γ ∥ op M ≤ L . Thus every prior admits a Bayes rule in Ω n , L \Omega_{n,L} Ω n , L , and its Bayes risk is the same for the restricted (i.e., Ω n , L \Omega_{n,L} Ω n , L ) and unrestricted (i.e., Ω \Omega Ω ) decision spaces. Applying the minimax identity of the form of (B.2) to both spaces proves the assertion about restricting the kernels.
The unrestricted Bayes risk equals
inf δ E π ∫ Ω ℓ n ( Γ T , w ) δ ( d w ∣ Y ) = E π [ f ∗ ( μ + Γ T n ) − f ∗ ( μ + Γ m π , I d ( Y ) n ) ] = E π [ r μ ( Γ T n ) − r μ ( Γ m π , I d ( Y ) n ) ] . \begin{aligned}
\inf_{\delta}\mathbb{E}_{\pi}\int_{\Omega}\ell_n(\Gamma T,w)\,\delta(\mathrm{d}w\mid Y)
=\mathbb{E}_{\pi}\left[f_*\left(\mu+\frac{\Gamma T}{\sqrt{n}}\right)-f_*\left(\mu+\frac{\Gamma m_{\pi,I_d}(Y)}{\sqrt{n}}\right)\right] \\
&=\mathbb{E}_{\pi}\left[r_{\mu}\left(\frac{\Gamma T}{\sqrt{n}}\right)-r_{\mu}\left(\frac{\Gamma m_{\pi,I_d}(Y)}{\sqrt{n}}\right)\right].
\end{aligned} δ inf E π ∫ Ω ℓ n ( Γ T , w ) δ ( d w ∣ Y ) = E π [ f ∗ ( μ + n Γ T ) − f ∗ ( μ + n Γ m π , I d ( Y ) ) ] = E π [ r μ ( n Γ T ) − r μ ( n Γ m π , I d ( Y ) ) ] .
Consequently, the minimax identity of the form of (B.2) gives the exact equality
inf δ sup u ∈ B 2 ( M ) E N ( u , I d ) ∫ Ω ℓ n ( Γ u , w ) δ ( d w ∣ Y ) = sup π ∈ P ( B 2 ( M ) ) E π [ r μ ( Γ T n ) − r μ ( Γ m π , I d ( Y ) n ) ] . \begin{aligned}
\inf_{\delta}\sup_{u\in B_2(M)}\mathbb{E}_{\mathcal{N}(u,I_d)}\int_{\Omega}\ell_n(\Gamma u,w)\,\delta(\mathrm{d}w\mid Y) \\
&=\sup_{\pi\in\mathcal{P}(B_2(M))}\mathbb{E}_{\pi}\left[r_{\mu}\left(\frac{\Gamma T}{\sqrt{n}}\right)-r_{\mu}\left(\frac{\Gamma m_{\pi,I_d}(Y)}{\sqrt{n}}\right)\right].
\end{aligned} δ inf u ∈ B 2 ( M ) sup E N ( u , I d ) ∫ Ω ℓ n ( Γ u , w ) δ ( d w ∣ Y ) = π ∈ P ( B 2 ( M )) sup E π [ r μ ( n Γ T ) − r μ ( n Γ m π , I d ( Y ) ) ] .
Both Γ T \Gamma T Γ T and Γ m π , I d ( Y ) \Gamma m_{\pi,I_d}(Y) Γ m π , I d ( Y ) belong to B 2 ( L ) B_2(L) B 2 ( L ) . The uniform expansion (C.1) therefore implies
sup π ∈ P ( B 2 ( M ) ) ∣ E π [ r μ ( Γ T / n ) − r μ ( Γ m π , I d ( Y ) / n ) ] a ( n − 1 / 2 ) − E π [ ψ ( Γ T ) − ψ ( Γ m π , I d ( Y ) ) ] ∣ \sup_{\pi\in\mathcal{P}(B_2(M))}\left|\frac{\mathbb{E}_{\pi}\left[r_{\mu}(\Gamma T/\sqrt{n})-r_{\mu}(\Gamma m_{\pi,I_d}(Y)/\sqrt{n})\right]}{a(n^{-1/2})}-\mathbb{E}_{\pi}\left[\psi(\Gamma T)-\psi(\Gamma m_{\pi,I_d}(Y))\right]\right| π ∈ P ( B 2 ( M )) sup a ( n − 1/2 ) E π [ r μ ( Γ T / n ) − r μ ( Γ m π , I d ( Y ) / n ) ] − E π [ ψ ( Γ T ) − ψ ( Γ m π , I d ( Y )) ]
≤ 2 sup t ∈ B 2 ( L ) ∣ r μ ( t / n ) a ( n − 1 / 2 ) − ψ ( t ) ∣ → 0. \le2\sup_{t\in B_2(L)}\left|\frac{r_{\mu}(t/\sqrt{n})}{a(n^{-1/2})}-\psi(t)\right|\to0. ≤ 2 t ∈ B 2 ( L ) sup a ( n − 1/2 ) r μ ( t / n ) − ψ ( t ) → 0.
Since
sup π ∈ P ( B 2 ( M ) ) E π [ ψ ( Γ T ) − ψ ( Γ m π , I d ( Y ) ) ] = V M , I d ( ψ ∘ Γ ) , \sup_{\pi\in\mathcal{P}(B_2(M))}\mathbb{E}_{\pi}\left[\psi(\Gamma T)-\psi(\Gamma m_{\pi,I_d}(Y))\right]=V_{M,I_d}(\psi\circ\Gamma), π ∈ P ( B 2 ( M )) sup E π [ ψ ( Γ T ) − ψ ( Γ m π , I d ( Y )) ] = V M , I d ( ψ ∘ Γ ) ,
taking suprema over the same priors proves (H.18).
To prove continuity, let ( M k , Γ k ) → ( M , Γ ) (M_k,\Gamma_k)\to(M,\Gamma) ( M k , Γ k ) → ( M , Γ ) with M k , M > 0 M_k,M>0 M k , M > 0 and Γ k , Γ \Gamma_k,\Gamma Γ k , Γ invertible. Parametrize the prior by T = M U T=MU T = M U , where U ∼ π ∈ P ( B 2 ( 1 ) ) U\sim\pi\in\mathcal{P}(B_2(1)) U ∼ π ∈ P ( B 2 ( 1 )) . The correspondence between priors on B 2 ( M ) B_2(M) B 2 ( M ) and B 2 ( 1 ) B_2(1) B 2 ( 1 ) is bijective. Given U = u U=u U = u , Y ∼ N ( M u , I d ) Y\sim\mathcal{N}(Mu,I_d) Y ∼ N ( M u , I d ) , and
l M ( u , y ) = e M u T y − M 2 ∥ u ∥ 2 2 / 2 , E π [ U ∣ Y = y ] = ∫ u l M ( u , y ) π ( d u ) ∫ l M ( u , y ) π ( d u ) . l_M(u,y)=e^{M u^\mathsf{T}y-M^2\lVert u\rVert_2^2/2},\qquad\mathbb{E}_{\pi}[U\mid Y=y]=\frac{\int u l_M(u,y)\,\pi(du)}{\int l_M(u,y)\,\pi(du)}. l M ( u , y ) = e M u T y − M 2 ∥ u ∥ 2 2 /2 , E π [ U ∣ Y = y ] = ∫ l M ( u , y ) π ( d u ) ∫ u l M ( u , y ) π ( d u ) .
All integrals below are over B 2 ( 1 ) B_2(1) B 2 ( 1 ) . For l = l M k l=l_{M_k} l = l M k or l = l M l=l_M l = l M , define
J π ( Γ , M ; l ) : = E N ( 0 , I d ) [ ∫ ψ ( Γ M u ) l ( u , Y ) π ( d u ) − ( ∫ l ( u , Y ) π ( d u ) ) ψ ( Γ M ∫ u l ( u , Y ) π ( d u ) ∫ l ( u , Y ) π ( d u ) ) ] . \begin{aligned}
J_\pi(\Gamma,M;l)&:=\mathbb{E}_{\mathcal{N}(0,I_d)}\left[\int\psi(\Gamma Mu)l(u,Y)\,\pi(du)\right.\\
&\qquad\left.-\left(\int l(u,Y)\,\pi(du)\right)\psi\left(\Gamma M\frac{\int u l(u,Y)\,\pi(du)}{\int l(u,Y)\,\pi(du)}\right)\right].
\end{aligned} J π ( Γ , M ; l ) := E N ( 0 , I d ) [ ∫ ψ ( Γ M u ) l ( u , Y ) π ( d u ) − ( ∫ l ( u , Y ) π ( d u ) ) ψ ( Γ M ∫ l ( u , Y ) π ( d u ) ∫ u l ( u , Y ) π ( d u ) ) ] .
The arguments ( Γ , M ) (\Gamma,M) ( Γ , M ) determine the function u ↦ ψ ( Γ M u ) u\mapsto\psi(\Gamma Mu) u ↦ ψ ( Γ M u ) , whereas l l l determines the mixture density and posterior mean. The mixture-density calculation used for J π J_\pi J π gives
V M , I d ( ψ ∘ Γ ) = sup π ∈ P ( B 2 ( 1 ) ) J π ( Γ , M ; l M ) , V M k , I d ( ψ ∘ Γ k ) = sup π ∈ P ( B 2 ( 1 ) ) J π ( Γ k , M k ; l M k ) . V_{M,I_d}(\psi\circ\Gamma)=\sup_{\pi\in\mathcal{P}(B_2(1))}J_\pi(\Gamma,M;l_M),\qquad V_{M_k,I_d}(\psi\circ\Gamma_k)=\sup_{\pi\in\mathcal{P}(B_2(1))}J_\pi(\Gamma_k,M_k;l_{M_k}). V M , I d ( ψ ∘ Γ ) = π ∈ P ( B 2 ( 1 )) sup J π ( Γ , M ; l M ) , V M k , I d ( ψ ∘ Γ k ) = π ∈ P ( B 2 ( 1 )) sup J π ( Γ k , M k ; l M k ) .
Choose a fixed L L L with M k ∥ Γ k ∥ o p ≤ L M_k\lVert\Gamma_k\rVert_{\mathrm{op}}\le L M k ∥ Γ k ∥ op ≤ L and M ∥ Γ ∥ o p ≤ L M\lVert\Gamma\rVert_{\mathrm{op}}\le L M ∥ Γ ∥ op ≤ L for all sufficiently large k k k . The finite convex function ψ \psi ψ is bounded and Lipschitz on B 2 ( L ) B_2(L) B 2 ( L ) . Since M k Γ k → M Γ M_k\Gamma_k\to M\Gamma M k Γ k → M Γ ,
sup u ∈ B 2 ( 1 ) ∣ ψ ( Γ k M k u ) − ψ ( Γ M u ) ∣ ≤ C ∥ M k Γ k − M Γ ∥ o p → 0. \sup_{u\in B_2(1)}\left|\psi(\Gamma_kM_ku)-\psi(\Gamma Mu)\right|\le C\lVert M_k\Gamma_k-M\Gamma\rVert_{\mathrm{op}}\to0. u ∈ B 2 ( 1 ) sup ∣ ψ ( Γ k M k u ) − ψ ( Γ M u ) ∣ ≤ C ∥ M k Γ k − M Γ ∥ op → 0.
Also, u ↦ ψ ( Γ k M k u ) u\mapsto\psi(\Gamma_kM_ku) u ↦ ψ ( Γ k M k u ) and u ↦ ψ ( Γ M u ) u\mapsto\psi(\Gamma Mu) u ↦ ψ ( Γ M u ) have a common bound and Lipschitz constant on B 2 ( 1 ) B_2(1) B 2 ( 1 ) , after increasing C C C if necessary. For each fixed y y y ,
sup u ∈ B 2 ( 1 ) ∣ log l M k ( u , y ) − log l M ( u , y ) ∣ ≤ ∣ M k − M ∣ ∥ y ∥ 2 + 1 2 ∣ M k 2 − M 2 ∣ → 0. \sup_{u\in B_2(1)}\left|\log l_{M_k}(u,y)-\log l_M(u,y)\right|\le|M_k-M|\lVert y\rVert_2+\frac{1}{2}|M_k^2-M^2|\to0. u ∈ B 2 ( 1 ) sup ∣ log l M k ( u , y ) − log l M ( u , y ) ∣ ≤ ∣ M k − M ∣ ∥ y ∥ 2 + 2 1 ∣ M k 2 − M 2 ∣ → 0.
Thus sup u ∈ B 2 ( 1 ) ∣ l M k ( u , y ) − l M ( u , y ) ∣ → 0 \sup_{u\in B_2(1)}|l_{M_k}(u,y)-l_M(u,y)|\to0 sup u ∈ B 2 ( 1 ) ∣ l M k ( u , y ) − l M ( u , y ) ∣ → 0 . Since M k M_k M k is bounded, for a constant C C C and all sufficiently large k k k ,
sup u ∈ B 2 ( 1 ) ∣ l M k ( u , y ) − l M ( u , y ) ∣ ≤ 2 e C ∥ y ∥ 2 , E N ( 0 , I d ) e C ∥ Y ∥ 2 < ∞ . \sup_{u\in B_2(1)}|l_{M_k}(u,y)-l_M(u,y)|\le2e^{C\lVert y\rVert_2},\qquad\mathbb{E}_{\mathcal{N}(0,I_d)}e^{C\lVert Y\rVert_2}<\infty. u ∈ B 2 ( 1 ) sup ∣ l M k ( u , y ) − l M ( u , y ) ∣ ≤ 2 e C ∥ y ∥ 2 , E N ( 0 , I d ) e C ∥ Y ∥ 2 < ∞.
DCT gives
E N ( 0 , I d ) sup u ∈ B 2 ( 1 ) ∣ l M k ( u , Y ) − l M ( u , Y ) ∣ → 0. \mathbb{E}_{\mathcal{N}(0,I_d)}\sup_{u\in B_2(1)}|l_{M_k}(u,Y)-l_M(u,Y)|\to0. E N ( 0 , I d ) u ∈ B 2 ( 1 ) sup ∣ l M k ( u , Y ) − l M ( u , Y ) ∣ → 0.
Fix π ∈ P ( B 2 ( 1 ) ) \pi\in\mathcal{P}(B_2(1)) π ∈ P ( B 2 ( 1 )) . Adding and subtracting J π ( Γ , M ; l M k ) J_\pi(\Gamma,M;l_{M_k}) J π ( Γ , M ; l M k ) gives
J π ( Γ k , M k ; l M k ) − J π ( Γ , M ; l M ) = [ J π ( Γ k , M k ; l M k ) − J π ( Γ , M ; l M k ) ] + [ J π ( Γ , M ; l M k ) − J π ( Γ , M ; l M ) ] . \begin{aligned}
J_\pi(\Gamma_k,M_k;l_{M_k})-J_\pi(\Gamma,M;l_M)
={}&[J_\pi(\Gamma_k,M_k;l_{M_k})-J_\pi(\Gamma,M;l_{M_k})]\\
&+[J_\pi(\Gamma,M;l_{M_k})-J_\pi(\Gamma,M;l_M)].
\end{aligned} J π ( Γ k , M k ; l M k ) − J π ( Γ , M ; l M ) = [ J π ( Γ k , M k ; l M k ) − J π ( Γ , M ; l M k )] + [ J π ( Γ , M ; l M k ) − J π ( Γ , M ; l M )] .
For the first difference, only Γ k M k \Gamma_kM_k Γ k M k is replaced by Γ M \Gamma M Γ M . The posterior mean is unchanged, and
∥ ∫ u l M k ( u , Y ) π ( d u ) ∫ l M k ( u , Y ) π ( d u ) ∥ 2 ≤ ∫ ∥ u ∥ 2 l M k ( u , Y ) π ( d u ) ∫ l M k ( u , Y ) π ( d u ) ≤ 1. \left\lVert\frac{\int u l_{M_k}(u,Y)\,\pi(\mathrm{d}u)}{\int l_{M_k}(u,Y)\,\pi(\mathrm{d}u)} \right\rVert_2 \leq\frac{\int\lVert u\rVert_2 l_{M_k}(u,Y)\,\pi(\mathrm{d}u)}{\int l_{M_k}(u,Y)\,\pi(\mathrm{d}u)} \leq1. ∫ l M k ( u , Y ) π ( d u ) ∫ u l M k ( u , Y ) π ( d u ) 2 ≤ ∫ l M k ( u , Y ) π ( d u ) ∫ ∥ u ∥ 2 l M k ( u , Y ) π ( d u ) ≤ 1.
Thus both u u u and this posterior mean belong to B 2 ( 1 ) B_2(1) B 2 ( 1 ) . Each of the two terms in the functional changes in absolute value by at most
sup u ∈ B 2 ( 1 ) ∣ ψ ( Γ k M k u ) − ψ ( Γ M u ) ∣ ∫ l M k ( u , Y ) π ( d u ) . \sup_{u\in B_2(1)} \left\lvert\psi(\Gamma_kM_ku)-\psi(\Gamma Mu)\right\rvert\int l_{M_k}(u,Y)\,\pi(\mathrm{d}u). u ∈ B 2 ( 1 ) sup ∣ ψ ( Γ k M k u ) − ψ ( Γ M u ) ∣ ∫ l M k ( u , Y ) π ( d u ) .
Moreover, Tonelli’s theorem and the likelihood-ratio identity give
E N ( 0 , I d ) ∫ l M k ( u , Y ) π ( d u ) = ∫ E N ( 0 , I d ) l M k ( u , Y ) π ( d u ) = ∫ 1 π ( d u ) = 1. \mathbb{E}_{\mathcal{N}(0,I_d)}\int l_{M_k}(u,Y)\,\pi(\mathrm{d}u)=\int\mathbb{E}_{\mathcal{N}(0,I_d)}l_{M_k}(u,Y)\,\pi(\mathrm{d}u)=\int1\,\pi(\mathrm{d}u)=1. E N ( 0 , I d ) ∫ l M k ( u , Y ) π ( d u ) = ∫ E N ( 0 , I d ) l M k ( u , Y ) π ( d u ) = ∫ 1 π ( d u ) = 1.
Consequently,
∣ J π ( Γ k , M k ; l M k ) − J π ( Γ , M ; l M k ) ∣ ≤ 2 sup u ∈ B 2 ( 1 ) ∣ ψ ( Γ k M k u ) − ψ ( Γ M u ) ∣ . \left\lvert J_\pi(\Gamma_k,M_k;l_{M_k})-J_\pi(\Gamma,M;l_{M_k})\right\rvert\leq2\sup_{u\in B_2(1)}\left\lvert\psi(\Gamma_kM_ku)-\psi(\Gamma Mu)\right\rvert. ∣ J π ( Γ k , M k ; l M k ) − J π ( Γ , M ; l M k ) ∣ ≤ 2 u ∈ B 2 ( 1 ) sup ∣ ψ ( Γ k M k u ) − ψ ( Γ M u ) ∣ .
For the second difference, ( Γ , M ) (\Gamma,M) ( Γ , M ) is fixed and only l M k l_{M_k} l M k is replaced by l M l_M l M . For each fixed y y y , the functions l M k ( ⋅ , y ) l_{M_k}(\cdot,y) l M k ( ⋅ , y ) and l M ( ⋅ , y ) l_M(\cdot,y) l M ( ⋅ , y ) are positive and continuous on B 2 ( 1 ) B_2(1) B 2 ( 1 ) . The function u ↦ ψ ( Γ M u ) u\mapsto\psi(\Gamma Mu) u ↦ ψ ( Γ M u ) is bounded and Lipschitz there, with both bounds at most C C C . Thus (H.16) applies with the radius in that estimate equal to 1, its matrix equal to Γ M \Gamma M Γ M , and its two weight functions equal to l M k ( ⋅ , y ) l_{M_k}(\cdot,y) l M k ( ⋅ , y ) and l M ( ⋅ , y ) l_M(\cdot,y) l M ( ⋅ , y ) . The two integrands defining J π ( Γ , M ; l M k ) J_\pi(\Gamma,M;l_{M_k}) J π ( Γ , M ; l M k ) and J π ( Γ , M ; l M ) J_\pi(\Gamma,M;l_M) J π ( Γ , M ; l M ) are precisely the corresponding J π J_\pi J π functionals. Taking expectation of the pointwise bound and using ∣ E A − E B ∣ ≤ E ∣ A − B ∣ \lvert\mathbb{E}A-\mathbb{E}B\rvert\leq\mathbb{E}\lvert A-B\rvert ∣ E A − E B ∣ ≤ E ∣ A − B ∣ gives
∣ J π ( Γ , M ; l M k ) − J π ( Γ , M ; l M ) ∣ ≤ 4 C E N ( 0 , I d ) sup u ∈ B 2 ( 1 ) ∣ l M k ( u , Y ) − l M ( u , Y ) ∣ . \left\lvert J_\pi(\Gamma,M;l_{M_k})-J_\pi(\Gamma,M;l_M)\right\rvert\leq4C\mathbb{E}_{\mathcal{N}(0,I_d)}\sup_{u\in B_2(1)}\left\lvert l_{M_k}(u,Y)-l_M(u,Y)\right\rvert. ∣ J π ( Γ , M ; l M k ) − J π ( Γ , M ; l M ) ∣ ≤ 4 C E N ( 0 , I d ) u ∈ B 2 ( 1 ) sup ∣ l M k ( u , Y ) − l M ( u , Y ) ∣ .
Both bounds are independent of π \pi π . Taking suprema over the same priors gives
∣ V M k , I d ( ψ ∘ Γ k ) − V M , I d ( ψ ∘ Γ ) ∣ ≤ sup π ∈ P ( B 2 ( 1 ) ) ∣ J π ( Γ k , M k ; l M k ) − J π ( Γ , M ; l M ) ∣ ≤ 2 sup u ∈ B 2 ( 1 ) ∣ ψ ( Γ k M k u ) − ψ ( Γ M u ) ∣ + 4 C E N ( 0 , I d ) sup u ∈ B 2 ( 1 ) ∣ l M k ( u , Y ) − l M ( u , Y ) ∣ → 0. \begin{aligned}
\left\lvert V_{M_k,I_d}(\psi\circ\Gamma_k)-V_{M,I_d}(\psi\circ\Gamma)\right\rvert
&\leq\sup_{\pi\in\mathcal{P}(B_2(1))}\left\lvert J_\pi(\Gamma_k,M_k;l_{M_k})-J_\pi(\Gamma,M;l_M)\right\rvert\\
&\leq2\sup_{u\in B_2(1)}\left\lvert\psi(\Gamma_kM_ku)-\psi(\Gamma Mu)\right\rvert\\
&\quad+4C\mathbb{E}_{\mathcal{N}(0,I_d)}\sup_{u\in B_2(1)}\left\lvert l_{M_k}(u,Y)-l_M(u,Y)\right\rvert\to0.
\end{aligned} ∣ V M k , I d ( ψ ∘ Γ k ) − V M , I d ( ψ ∘ Γ ) ∣ ≤ π ∈ P ( B 2 ( 1 )) sup ∣ J π ( Γ k , M k ; l M k ) − J π ( Γ , M ; l M ) ∣ ≤ 2 u ∈ B 2 ( 1 ) sup ∣ ψ ( Γ k M k u ) − ψ ( Γ M u ) ∣ + 4 C E N ( 0 , I d ) u ∈ B 2 ( 1 ) sup ∣ l M k ( u , Y ) − l M ( u , Y ) ∣ → 0.
This proves the asserted continuity.