Introduction
The extraction of information from complex, multi-modal distributions that are specified only implicitly through evaluations of their densities is a long-standing problem in applied sciences. A promising approach is approximate sampling, that is, generating realizations from a probability measure that is ‘sufficiently close’ to the target distribution. This methodology is frequently used in computational statistics, machine learning, and statistical physics, where the desired distribution often depends on an inverse-temperature parameter. As the inverse temperature increases, intuitively, the distribution becomes increasingly concentrated. While this concentration can make it possible to identify a particularly high probability area, it also poses significant challenges for sampling and can substantially reduce the efficiency of the sampling scheme. We propose a multi-modal Laplace mixture approximation that can serve as a proxy for the target distribution and under suitable assumptions remains capable to capture the asymptotic behavior when the ‘concentration’ increases.
For formalizing the setting let ( π n ) n ∈ N (\pi_n)_{n \in\mathbb{N}} ( π n ) n ∈ N R d \mathbb{R}^d R d σ \sigma σ
π n ( d x ) : = 1 Z n e − n Φ n ( x ) π 0 ( d x ) , with Z n : = ∫ R d e − n Φ n ( x ) π 0 ( d x ) , (1) \pi_n(\mathrm{d}x) := \frac{1}{Z_n}e^{-n\Phi_n(x)}\pi_0(\mathrm{d}x), \quad\text{with} \quad Z_n := \int_{\mathbb{R}^d}e^{-n\Phi_n(x)}\pi_0(\mathrm{d}x), \tag*{(1)} π n ( d x ) := Z n 1 e − n Φ n ( x ) π 0 ( d x ) , with Z n := ∫ R d e − n Φ n ( x ) π 0 ( d x ) , (1) where Φ n : R d → [ 0 , ∞ ) \Phi_n:\mathbb{R}^d \to[0,\infty) Φ n : R d → [ 0 , ∞ ) n ∈ N n \in\mathbb{N} n ∈ N π 0 \pi_0 π 0 R d \mathbb{R}^d R d y y y
Y = G n ( X ) + η n with X ∼ π 0 . Y = G_n(X) + \eta_n \quad\text{with} \quad X \sim\pi_0. Y = G n ( X ) + η n with X ∼ π 0 . ‘Concentration behavior’ for increasing n n n
a) Assume that the observation operator G n = G G_n = G G n = G o ∈ N o \in\mathbb{N} o ∈ N y ∈ R o y \in\mathbb{R}^o y ∈ R o G : R d → R o G:\mathbb{R}^d \to\mathbb{R}^o G : R d → R o n n n η n ∼ N ( 0 , n − 1 Γ ) \eta_n \sim\mathcal{N}(0,n^{-1}\Gamma) η n ∼ N ( 0 , n − 1 Γ ) Γ ∈ R o × o \Gamma\in\mathbb{R}^{o \times o} Γ ∈ R o × o
Φ n ( x ) : = 1 2 ∥ y − G ( x ) ∥ Γ − 1 2 , with ∥ z ∥ Γ − 1 : = ∥ Γ − 1 / 2 z ∥ \Phi_n(x) := \frac{1}{2}\lVert y - G(x)\rVert_{\Gamma^{-1}}^2, \quad\text{with} \quad\lVert z\rVert_{\Gamma^{-1}} := \lVert\Gamma^{-1/2}z\rVert Φ n ( x ) := 2 1 ∥ y − G ( x ) ∥ Γ − 1 2 , with ∥ z ∥ Γ − 1 := ∥ Γ − 1/2 z ∥ and Euclidean norm ∥ z ∥ \lVert z\rVert ∥ z ∥ z ∈ R o z \in\mathbb{R}^o z ∈ R o
b) For G G G y = ( y 1 , … , y n ) ∈ R n ⋅ o y = (y_1,\ldots,y_n) \in\mathbb{R}^{n \cdot o} y = ( y 1 , … , y n ) ∈ R n ⋅ o y 1 , … , y n ∈ R o y_1,\ldots,y_n \in\mathbb{R}^o y 1 , … , y n ∈ R o η n ∼ N ( 0 , σ 2 Id n o ) \eta_n \sim\mathcal{N}(0,\sigma^2\operatorname{Id}_{no}) η n ∼ N ( 0 , σ 2 Id n o ) σ > 0 \sigma> 0 σ > 0 Id n o ∈ R n o × n o \operatorname{Id}_{no} \in\mathbb{R}^{no \times no} Id n o ∈ R n o × n o
G n = ( G , … , G ) : R d → R n ⋅ o , G_n = (G,\ldots,G):\mathbb{R}^d \to\mathbb{R}^{n \cdot o}, G n = ( G , … , G ) : R d → R n ⋅ o , such that (for i = 1 , … , n i = 1,\ldots,n i = 1 , … , n y i y_i y i G ( X ) + ξ i G(X) + \xi_i G ( X ) + ξ i η n = ( ξ 1 , … , ξ n ) \eta_n = (\xi_1,\ldots,\xi_n) η n = ( ξ 1 , … , ξ n )
Φ n ( x ) : = 1 2 n σ 2 ∑ k = 1 n ∥ y k − G ( x ) ∥ 2 . \Phi_n(x) := \frac{1}{2n\sigma^2}\sum_{k=1}^{n}\lVert y_k - G(x)\rVert^2. Φ n ( x ) := 2 n σ 2 1 k = 1 ∑ n ∥ y k − G ( x ) ∥ 2 . Scenario a) is referred to as small noise limit case and scenario b) as large data limit case. Let us mention here that we will mostly focus on the small noise limit framework in the theoretical analysis but also consider the large data limit in our numerical experiments.
For large n ∈ N n \in\mathbb{N} n ∈ N [8 ] , and Bayesian machine learning, cf. [4 ] , for dealing with a concentrated posterior measure π n \pi_n π n Laplace approximation (L A LA L A L π n \mathcal{L}_{\pi_n} L π n π n \pi_n π n n → ∞ n \to\infty n → ∞ [10 , 11 , 14 , 15 , 25 , 24 , 26 ] .
A particularly remarkable recent result obtained by Katsoulakis and Sverkoiny in [15 ] provides an explicit upper bound of the total variation distance between π n \pi_n π n L π n \mathcal{L}_{\pi_n} L π n d n − 1 / 2 d\,n^{-1/2} d n − 1/2 π n \pi_n π n L π n K \mathcal{L}_{\pi_n}^{K} L π n K K K K Φ n \Phi_n Φ n multi-modal case for Φ n = Φ \Phi_n = \Phi Φ n = Φ n n n [13 ] . There, the author shows that when argmin x ∈ R d Φ ( x ) = { x 1 , … , x K } \operatorname*{argmin}_{x \in\mathbb{R}^d}\Phi(x) = \{x_1,\ldots,x_K\} argmin x ∈ R d Φ ( x ) = { x 1 , … , x K } K ∈ N K \in\mathbb{N} K ∈ N π n \pi_n π n δ x 1 , … , δ x K \delta_{x_1},\ldots,\delta_{x_K} δ x 1 , … , δ x K [10 ] , where the p p p p ∈ [ 1 , ∞ ) p \in[1,\infty) p ∈ [ 1 , ∞ ) π n \pi_n π n n − 1 / 2 p n^{-1/2p} n − 1/2 p
Our contribution. We study the behavior and benefits of the LA for the computation of expectations w.r.t. π n \pi_n π n π n \pi_n π n [8 ] . In the first part of the paper, we generalize the convergence result of [24 Theorem 2] to the case of multimodal LA and multimodal π n \pi_n π n [11 ] we also derive a non-asymptotic bound in Hellinger distance. A particular attention is given to explicitly stating all the constants involved in this bound, especially their dependence on the dimension d d d π n \pi_n π n n → ∞ n \to\infty n → ∞ n − 1 + δ n^{-1+\delta} n − 1 + δ δ > 0 \delta> 0 δ > 0 n → ∞ n \to\infty n → ∞ n − 1 / 2 n^{-1/2} n − 1/2
Outline. In § 2 we introduce the general setting, assumptions and the multimodal LA. § 3 establishes the convergence of LA to π n \pi_n π n § 4.2 we study the independent Metropolis–Hastings algorithm using the LA as proposal. Afterwards, both methods and statements are considered in several numerical experiments in Section 5. Finally, § 6 concludes.
Preliminaries and Laplace mixture approximation
Notation and Preliminaries
We denote the usual inner product on R d \mathbb{R}^d R d ⟨ ⋅ , ⋅ ⟩ \langle\cdot,\cdot\rangle ⟨ ⋅ , ⋅ ⟩ ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ ∥ ∥ ∥ ⋅ ∥ ∥ ∥ \|\!\|\!\|\cdot\|\!\|\!\| ∥ ∥ ∥ ⋅ ∥ ∥ ∥ A ∈ R d × d A \in\mathbb{R}^{d \times d} A ∈ R d × d ∥ ⋅ ∥ A = ∥ A 1 / 2 ⋅ ∥ \| \cdot\|_A = \|A^{1/2}\cdot\| ∥ ⋅ ∥ A = ∥ A 1/2 ⋅ ∥
For a given probability measure π \pi π R d \mathbb{R}^d R d L p ( π ) L^p(\pi) L p ( π ) p p p f : R d → R f:\mathbb{R}^d \to\mathbb{R} f : R d → R L p ( π ) L^p(\pi) L p ( π ) ∥ ⋅ ∥ L p ( π ) \| \cdot\|_{L^p(\pi)} ∥ ⋅ ∥ L p ( π ) f ∈ L 1 ( π ) f \in L^1(\pi) f ∈ L 1 ( π ) E π [ f ] : = ∫ R d f d π \mathbb{E}_{\pi}[f] := \int_{\mathbb{R}^d} f\,\mathrm{d}\pi E π [ f ] := ∫ R d f d π f ∈ L 2 ( π ) f \in L^2(\pi) f ∈ L 2 ( π ) Var π ( f ) : = E π [ ( f − E π [ f ] ) 2 ] \operatorname{Var}_{\pi}(f) := \mathbb{E}_{\pi}[(f-\mathbb{E}_{\pi}[f])^2] Var π ( f ) := E π [( f − E π [ f ] ) 2 ]
In the following we work with three classical distances and divergences of probability measures π \pi π π ~ \widetilde{\pi} π R d \mathbb{R}^d R d d T V d_{\mathrm{TV}} d TV d H d_{\mathrm{H}} d H
d T V ( π , π ~ ) : = 1 2 ∫ R d ∣ d π d μ − d π ~ d μ ∣ d μ , d H ( π , π ~ ) : = ( ∫ R d ∣ d π d μ − d π ~ d μ ∣ 2 d μ ) 1 / 2 , d_{\mathrm{TV}}(\pi,\widetilde{\pi}) := \frac{1}{2}\int_{\mathbb{R}^d}\left|\frac{\mathrm{d}\pi}{\mathrm{d}\mu}-\frac{\mathrm{d}\widetilde{\pi}}{\mathrm{d}\mu}\right|\,\mathrm{d}\mu,\qquad d_{\mathrm{H}}(\pi,\widetilde{\pi}) := \left(\int_{\mathbb{R}^d}\left|\sqrt{\frac{\mathrm{d}\pi}{\mathrm{d}\mu}}-\sqrt{\frac{\mathrm{d}\widetilde{\pi}}{\mathrm{d}\mu}}\right|^2\,\mathrm{d}\mu\right)^{1/2}, d TV ( π , π ) := 2 1 ∫ R d d μ d π − d μ d π d μ , d H ( π , π ) := ∫ R d d μ d π − d μ d π 2 d μ 1/2 , where μ \mu μ π \pi π π ~ \widetilde{\pi} π R d \mathbb{R}^d R d d π d μ \frac{\mathrm{d}\pi}{\mathrm{d}\mu} d μ d π d π ~ d μ \frac{\mathrm{d}\widetilde{\pi}}{\mathrm{d}\mu} d μ d π χ 2 \chi^2 χ 2 π ~ \tilde{\pi} π ~ π \pi π
χ 2 ( π ~ ∥ π ) : = ∫ R d ( d π ~ d π − 1 ) 2 d π ∈ [ 0 , ∞ ] \chi^2(\tilde{\pi}\mathbin\Vert\pi) := \int_{\mathbb{R}^d} \left(\frac{\mathrm{d}\tilde{\pi}}{\mathrm{d}\pi}-1\right)^2 \mathrm{d}\pi\in[0,\infty] χ 2 ( π ~ ∥ π ) := ∫ R d ( d π d π ~ − 1 ) 2 d π ∈ [ 0 , ∞ ] for the case that π ~ \tilde{\pi} π ~ π \pi π χ 2 ( π ~ ∥ π ) : = ∞ \chi^2(\tilde{\pi}\mathbin\Vert\pi) := \infty χ 2 ( π ~ ∥ π ) := ∞ χ 2 \chi^2 χ 2 d π ~ d π \frac{\mathrm{d}\tilde{\pi}}{\mathrm{d}\pi} d π d π ~ d T V d_{\mathrm{TV}} d TV d H d_{\mathrm{H}} d H [9 Equation (8)] , that
d H 2 ( π , π ~ ) 2 ≤ d T V ( π , π ~ ) ≤ d H ( π , π ~ ) (2) \frac{d_{\mathrm{H}}^2(\pi,\tilde{\pi})}{2} \le d_{\mathrm{TV}}(\pi,\tilde{\pi}) \le d_{\mathrm{H}}(\pi,\tilde{\pi}) \tag*{(2)} 2 d H 2 ( π , π ~ ) ≤ d TV ( π , π ~ ) ≤ d H ( π , π ~ ) (2) and that the Hellinger distance allows to control differences of expectations of square integrable functions
∣ E π [ f ] − E π ~ [ f ] ∣ ≤ ( 2 ∥ f ∥ L 2 ( π ) 2 + 2 ∥ f ∥ L 2 ( π ~ ) 2 ) 1 / 2 d H ( π , π ~ ) , ∀ f ∈ L 2 ( π ) ∩ L 2 ( π ~ ) , (3) \left|\mathbb{E}_{\pi}[f]-\mathbb{E}_{\tilde{\pi}}[f]\right| \le\left(2\lVert f\rVert_{L^2(\pi)}^2+2\lVert f\rVert_{L^2(\tilde{\pi})}^2\right)^{1/2}d_{\mathrm{H}}(\pi,\tilde{\pi}),\qquad\forall f\in L^2(\pi)\cap L^2(\tilde{\pi}), \tag*{(3)} ∣ E π [ f ] − E π ~ [ f ] ∣ ≤ ( 2 ∥ f ∥ L 2 ( π ) 2 + 2 ∥ f ∥ L 2 ( π ~ ) 2 ) 1/2 d H ( π , π ~ ) , ∀ f ∈ L 2 ( π ) ∩ L 2 ( π ~ ) , (3) see [3 Lemma 21] .
Remark 2.1. Note, that the proof of the bound (3) extends to general positive and finite measures π , π ~ \pi,\tilde{\pi} π , π ~ R d \mathbb{R}^d R d
Moreover, the Hellinger distance can be bounded by the χ 2 \chi^2 χ 2
d H ( π , π ~ ) ≤ χ 2 ( π ~ ∥ π ) (4) d_{\mathrm{H}}(\pi,\tilde{\pi})\le\chi^2(\tilde{\pi}\mathbin\Vert\pi) \tag*{(4)} d H ( π , π ~ ) ≤ χ 2 ( π ~ ∥ π ) (4) see [28 Section 2.4] . In general the converse inequality does not hold, but in case of ∥ d π ~ d π ∥ ∞ : = sup x ∈ R d ∣ d π ~ d π ( x ) ∣ < ∞ \left\lVert\frac{\mathrm{d}\tilde{\pi}}{\mathrm{d}\pi}\right\rVert_\infty:= \sup_{x\in\mathbb{R}^d}\left|\frac{\mathrm{d}\tilde{\pi}}{\mathrm{d}\pi}(x)\right|<\infty d π d π ~ ∞ := sup x ∈ R d d π d π ~ ( x ) < ∞
χ 2 ( π ~ ∥ π ) ≤ ( ∥ d π ~ d π ∥ ∞ + 1 ) 2 d H ( π , π ~ ) 2 , (5) \chi^2(\tilde{\pi}\mathbin\Vert\pi)\le\left(\sqrt{\left\lVert\frac{\mathrm{d}\tilde{\pi}}{\mathrm{d}\pi}\right\rVert_\infty}+1\right)^2d_{\mathrm{H}}(\pi,\tilde{\pi})^2, \tag*{(5)} χ 2 ( π ~ ∥ π ) ≤ ( d π d π ~ ∞ + 1 ) 2 d H ( π , π ~ ) 2 , (5) which follows by using ( t − 1 ) 2 = ( t − 1 ) 2 ( t + 1 ) 2 (t-1)^2=(\sqrt{t}-1)^2(\sqrt{t}+1)^2 ( t − 1 ) 2 = ( t − 1 ) 2 ( t + 1 ) 2 t ≥ 0 t\ge0 t ≥ 0 t = d π ~ d π ( x ) t=\frac{\mathrm{d}\tilde{\pi}}{\mathrm{d}\pi}(x) t = d π d π ~ ( x )
Laplace mixture approximation
We assume that the prior distribution π 0 \pi_0 π 0 (1) is absolutely continuous with respect to the Lebesgue measure, with density π 0 : R d → ( 0 , ∞ ) \pi_0:\mathbb{R}^d\to(0,\infty) π 0 : R d → ( 0 , ∞ ) π n \pi_n π n
π n ( d x ) : = 1 Z n e − n I n ( x ) d x , with I n : = Φ n − 1 n log ( π 0 ) . \pi_n(\mathrm{d}x):=\frac{1}{Z_n}\mathrm{e}^{-nI_n(x)}\mathrm{d}x,\qquad\text{with}\qquad I_n:=\Phi_n-\frac{1}{n}\log(\pi_0). π n ( d x ) := Z n 1 e − n I n ( x ) d x , with I n := Φ n − n 1 log ( π 0 ) . As a standard assumption for the well-definedness of the Laplace approximation we impose the following differentiability requirements. For this let us denote with C r ( R d , [ 0 , ∞ ) ) C^r(\mathbb{R}^d,[0,\infty)) C r ( R d , [ 0 , ∞ )) r r r R d \mathbb{R}^d R d [ 0 , ∞ ) [0,\infty) [ 0 , ∞ )
Standing assumption. For each n ∈ N n \in\mathbb{N} n ∈ N I n ∈ C 2 ( R d , [ 0 , ∞ ) ) I_n \in C^2(\mathbb{R}^d,[0,\infty)) I n ∈ C 2 ( R d , [ 0 , ∞ )) K ∈ N K \in\mathbb{N} K ∈ N x n ( 1 ) , … , x n ( K ) ∈ R d x_n^{(1)},\ldots,x_n^{(K)} \in\mathbb{R}^d x n ( 1 ) , … , x n ( K ) ∈ R d n ∈ N n \in\mathbb{N} n ∈ N k = 1 , … , K k=1,\ldots,K k = 1 , … , K
π n ( k ) : = e − n I n ( x n ( k ) ) > 0 , H n ( k ) : = ∇ 2 I n ( x n ( k ) ) \pi_n^{(k)} := \mathrm{e}^{-nI_n(x_n^{(k)})} > 0, \qquad H_n^{(k)} := \nabla^2 I_n(x_n^{(k)}) π n ( k ) := e − n I n ( x n ( k ) ) > 0 , H n ( k ) := ∇ 2 I n ( x n ( k ) ) and assume that the matrices H n ( k ) H_n^{(k)} H n ( k )
Under the standing assumption , we define the multi-modal version of the LA of π n \pi_n π n n ∈ N n \in\mathbb{N} n ∈ N [8 ] ), as
L π n K : = ∑ k = 1 K c n , k N ( x n ( k ) , ( n H n ( k ) ) − 1 ) (6) \mathcal{L}_{\pi_n}^{K} := \sum_{k=1}^{K} c_{n,k}\mathcal{N}\left(x_n^{(k)},(nH_n^{(k)})^{-1}\right) \tag*{(6)} L π n K := k = 1 ∑ K c n , k N ( x n ( k ) , ( n H n ( k ) ) − 1 ) (6) with coefficients c n , k ∈ [ 0 , 1 ] c_{n,k} \in[0,1] c n , k ∈ [ 0 , 1 ] ∑ k = 1 K c n , k = 1 \sum_{k=1}^{K}c_{n,k}=1 ∑ k = 1 K c n , k = 1
c n , k : = det ( ( n H n ( k ) ) − 1 ) 1 2 π n ( k ) ∑ j = 1 K det ( ( n H n ( j ) ) − 1 ) 1 2 π n ( j ) . c_{n,k} := \frac{\det\left((nH_n^{(k)})^{-1}\right)^{\frac{1}{2}}\pi_n^{(k)}}{\sum_{j=1}^{K}\det\left((nH_n^{(j)})^{-1}\right)^{\frac{1}{2}}\pi_n^{(j)}}. c n , k := ∑ j = 1 K det ( ( n H n ( j ) ) − 1 ) 2 1 π n ( j ) det ( ( n H n ( k ) ) − 1 ) 2 1 π n ( k ) . This LA probability measure L π n K \mathcal{L}_{\pi_n}^{K} L π n K L π n K \mathcal{L}_{\pi_n}^{K} L π n K x ∈ R d x \in\mathbb{R}^d x ∈ R d
L π n K ( x ) = 1 Z ~ n ∑ k = 1 K π n ( k ) exp ( − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 ) , with Z ~ n : = ( 2 π n ) d / 2 ∑ j = 1 K det ( ( H n ( j ) ) − 1 / 2 ) π n ( j ) . (7) \mathcal{L}_{\pi_n}^{K}(x)=\frac{1}{\widetilde{Z}_n}\sum_{k=1}^{K}\pi_n^{(k)}\exp\left(-\frac{n}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2\right), \quad\text{with } \widetilde{Z}_n:=\left(\frac{2\pi}{n}\right)^{d/2}\sum_{j=1}^{K}\det\left((H_n^{(j)})^{-1/2}\right)\pi_n^{(j)}. \tag*{(7)} L π n K ( x ) = Z n 1 k = 1 ∑ K π n ( k ) exp ( − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 ) , with Z n := ( n 2 π ) d /2 j = 1 ∑ K det ( ( H n ( j ) ) − 1/2 ) π n ( j ) . (7) Remark 2.2. We comment on the definition of the multi-modal LA. In principle, there are at least two ways to build it based on a Gaussian mixture since there are two types of local minima: those that vanish as n n n I n : R → [ 0 , ∞ ) I_n:\mathbb{R}\to[0,\infty) I n : R → [ 0 , ∞ )
I n ( x ) = ( x 2 − 1 ) 2 + 1 n ( x + 1 ) 2 , I_n(x)=(x^2-1)^2+\frac{1}{n}(x+1)^2, I n ( x ) = ( x 2 − 1 ) 2 + n 1 ( x + 1 ) 2 , we have a r g m i n x ∈ R I n ( x ) = { − 1 , 1 2 ( 1 + 1 − 2 n ) } \mathop{\rm arg\,min}_{x\in\mathbb{R}} I_n(x)=\left\{-1,\frac{1}{2}\left(1+\sqrt{1-\frac{2}{n}}\right)\right\} arg min x ∈ R I n ( x ) = { − 1 , 2 1 ( 1 + 1 − n 2 ) } π n \pi_n π n δ − 1 + e − 4 δ 1 1 + e − 4 \frac{\delta_{-1}+\mathrm{e}^{-4}\delta_1}{1+\mathrm{e}^{-4}} 1 + e − 4 δ − 1 + e − 4 δ 1
I n ( x ) = ( x 2 − 1 ) 2 + 1 n ( x + 1 ) 2 , I_n(x)=(x^2-1)^2+\frac{1}{\sqrt{n}}(x+1)^2, I n ( x ) = ( x 2 − 1 ) 2 + n 1 ( x + 1 ) 2 , for which a r g m i n x ∈ R I n ( x ) = { − 1 , 1 2 ( 1 + 1 − 2 n ) } \mathop{\rm arg\,min}_{x\in\mathbb{R}} I_n(x)=\left\{-1,\frac{1}{2}\left(1+\sqrt{1-\frac{2}{\sqrt{n}}}\right)\right\} arg min x ∈ R I n ( x ) = { − 1 , 2 1 ( 1 + 1 − n 2 ) } π n \pi_n π n δ − 1 \delta_{-1} δ − 1 0 0 0 n n n
Convergence in Hellinger distance
In this section, we investigate the convergence of the Hellinger distance between π n \pi_n π n L π n K \mathcal{L}_{\pi_n}^{K} L π n K x n ( 1 ) , … , x n ( K ) x_n^{(1)},\ldots,x_n^{(K)} x n ( 1 ) , … , x n ( K ) V n k , R , q V_n^{k,R,q} V n k , R , q R , q ≥ 0 R,q \ge0 R , q ≥ 0
V n k , R , q : = { x ∈ R d : ∥ x − x n ( k ) ∥ H n ( k ) 2 ≤ R n − 2 q } , n ∈ N , k = 1 , … , K . V_n^{k,R,q} := \left\{x \in\mathbb{R}^d : \|x-x_n^{(k)}\|_{H_n^{(k)}}^2 \le Rn^{-2q}\right\}, \qquad n \in\mathbb{N},\ k=1,\ldots,K. V n k , R , q := { x ∈ R d : ∥ x − x n ( k ) ∥ H n ( k ) 2 ≤ R n − 2 q } , n ∈ N , k = 1 , … , K . Moreover, we introduce the ‘tail area’ as V ‾ n R , q : = R d ∖ ( ⋃ k = 1 K V n k , R , q ) \overline{V}_n^{R,q} := \mathbb{R}^d \setminus\left(\bigcup_{k=1}^K V_n^{k,R,q}\right) V n R , q := R d ∖ ( ⋃ k = 1 K V n k , R , q )
Remark 3.1. From the definition it is visible that for any k = 1 , … , K k=1,\ldots,K k = 1 , … , K V n k , R , q V_n^{k,R,q} V n k , R , q n − q n^{-q} n − q n n n π n \pi_n π n n n n
We are now in a position to impose differentiability conditions on I n I_n I n
Assumption 1. For any n ∈ N n \in\mathbb{N} n ∈ N I n ∈ C 3 ( R d , [ 0 , ∞ ) ) I_n \in C^3(\mathbb{R}^d,[0,\infty)) I n ∈ C 3 ( R d , [ 0 , ∞ )) q ≥ 0 q \ge0 q ≥ 0 R , ρ , M ∈ ( 0 , ∞ ) R,\rho,M \in(0,\infty) R , ρ , M ∈ ( 0 , ∞ )
(i) For any n ∈ N n \in\mathbb{N} n ∈ N V n 1 , R , q , … , V n K , R , q V_n^{1,R,q},\ldots,V_n^{K,R,q} V n 1 , R , q , … , V n K , R , q
(ii) The matrices ( H n ( k ) ) k ∈ { 1 , … , K } \left(H_n^{(k)}\right)_{k\in\{1,\ldots,K\}} ( H n ( k ) ) k ∈ { 1 , … , K } ρ \rho ρ k ∈ { 1 , … , K } k\in\{1,\ldots,K\} k ∈ { 1 , … , K } n ∈ N n\in\mathbb{N} n ∈ N
H n ( k ) ≥ ρ 2 I d , H_n^{(k)} \ge\rho^2\mathbb{I}_d, H n ( k ) ≥ ρ 2 I d , where the inequality is taken in the sense of positive definite matrices.
(iii) The third-order derivative tensors D 3 I n ( x ) D^3I_n(x) D 3 I n ( x ) M M M V n k , R , q V_n^{k,R,q} V n k , R , q k ∈ { 1 , … , K } k\in\{1,\ldots,K\} k ∈ { 1 , … , K } n ∈ N n\in\mathbb{N} n ∈ N x ∈ V n k , R , q x\in V_n^{k,R,q} x ∈ V n k , R , q h ∈ R d h\in\mathbb{R}^d h ∈ R d
∣ D 3 I n ( x ) ( h , h , h ) ∣ ≤ 6 M ∥ h ∥ 3 . \left|D^3I_n(x)(h,h,h)\right| \le6M\|h\|^3. D 3 I n ( x ) ( h , h , h ) ≤ 6 M ∥ h ∥ 3 . Remark 3.2. Let us already here note that (ii) of Assumption 1 implies that for each n ∈ N n\in\mathbb{N} n ∈ N k ∈ { 1 , … , K } k\in\{1,\ldots,K\} k ∈ { 1 , … , K } x ∈ R d x\in\mathbb{R}^d x ∈ R d
∥ x − x n ( k ) ∥ ≤ ∥ ( H n ( k ) ) − 1 / 2 ∥ ∥ x − x n ( k ) ∥ H n ( k ) ≤ 1 ρ ∥ x − x n ( k ) ∥ H n ( k ) . \|x-x_n^{(k)}\| \le\left\|\left(H_n^{(k)}\right)^{-1/2}\right\|\|x-x_n^{(k)}\|_{H_n^{(k)}} \le\frac{1}{\rho}\|x-x_n^{(k)}\|_{H_n^{(k)}}. ∥ x − x n ( k ) ∥ ≤ ( H n ( k ) ) − 1/2 ∥ x − x n ( k ) ∥ H n ( k ) ≤ ρ 1 ∥ x − x n ( k ) ∥ H n ( k ) . Asymptotic result for q = 0 q=0 q = 0
Now, we establish a generalization of [24 ] , Theorem 2 to the multi-modal setting. We consider the case q = 0 q=0 q = 0 V n k , R , q = V n k , R , 0 V_n^{k,R,q}=V_n^{k,R,0} V n k , R , q = V n k , R , 0 n n n
Assumption 2. For R > 0 R > 0 R > 0 0 < ε < 1 0 < \varepsilon< 1 0 < ε < 1 n ∗ ∈ N n^* \in\mathbb{N} n ∗ ∈ N n ≥ n ∗ n \ge n^* n ≥ n ∗
inf x ∈ V ‾ n R , 0 I n ( x ) > ε , ∑ j = 1 K det ( ( H n ( j ) ) − 1 ) π n ( j ) > ε , (8) \inf_{x \in\overline{V}_{n}^{R,0}} I_n(x) > \varepsilon,\qquad\sum_{j=1}^{K} \det\left((H_n^{(j)})^{-1}\right)\pi_n^{(j)} > \varepsilon, \tag*{(8)} x ∈ V n R , 0 inf I n ( x ) > ε , j = 1 ∑ K det ( ( H n ( j ) ) − 1 ) π n ( j ) > ε , (8) and there is a function g : R d → [ 0 , ∞ ) g : \mathbb{R}^d \to[0,\infty) g : R d → [ 0 , ∞ ) g 1 − ε g^{1-\varepsilon} g 1 − ε x ∈ R d x \in\mathbb{R}^d x ∈ R d n ≥ n ∗ n \ge n^* n ≥ n ∗ exp ( − n I n ( x ) ) ≤ g ( x ) \exp(-nI_n(x)) \le g(x) exp ( − n I n ( x )) ≤ g ( x )
Remark 3.3. If for a given R > 0 R > 0 R > 0
lim n → ∞ inf x ∈ V ‾ n R , 0 I n ( x ) > 0 and lim n → ∞ ∑ j = 1 K det ( ( H n ( j ) ) − 1 ) π n ( j ) > 0 , \lim_{n \to\infty}\inf_{x \in\overline{V}_{n}^{R,0}} I_n(x) > 0 \quad\text{and} \quad\lim_{n \to\infty}\sum_{j=1}^{K}\det\left((H_n^{(j)})^{-1}\right)\pi_n^{(j)} > 0, n → ∞ lim x ∈ V n R , 0 inf I n ( x ) > 0 and n → ∞ lim j = 1 ∑ K det ( ( H n ( j ) ) − 1 ) π n ( j ) > 0 , then there is an ε > 0 \varepsilon> 0 ε > 0 n ∗ ∈ N n^* \in\mathbb{N} n ∗ ∈ N (8) holds for any n ≥ n ∗ n \ge n^* n ≥ n ∗
Note that the combination of Assumption 1 and Assumption 2 is adapted from the requirements formulated in [24 ] to cover the multi-modal setting. Now we state one of our main theorems that is proven in Appendix § 7 .§ 3 .
Theorem 3.1. For q = 0 q = 0 q = 0 ρ , M ∈ ( 0 , ∞ ) \rho, M \in(0,\infty) ρ , M ∈ ( 0 , ∞ ) R ≤ ρ 6 16 M 2 R \le\frac{\rho^6}{16M^2} R ≤ 16 M 2 ρ 6 ε ∈ ( 0 , 1 ) \varepsilon\in(0,1) ε ∈ ( 0 , 1 )
d H ( L π n K , π n ) = ( ∫ R d [ π n ( x ) − L π n K ( x ) ] 2 d x ) 1 / 2 ∈ O ( n − 1 / 2 ) . d_H(\mathcal{L}_{\pi_n}^{K},\pi_n)=\left(\int_{\mathbb{R}^d}\left[\sqrt{\pi_n(x)}-\sqrt{\mathcal{L}_{\pi_n}^{K}(x)}\right]^2\mathrm{d}x\right)^{1/2}\in O\left(n^{-1/2}\right). d H ( L π n K , π n ) = ( ∫ R d [ π n ( x ) − L π n K ( x ) ] 2 d x ) 1/2 ∈ O ( n − 1/2 ) . The idea of the proof of this theorem follows the classical approach of ‘splitting the integral’ appearing in the definition of the Hellinger distance into several parts and bound those suitably. The corresponding parts are determined by decomposing R d \mathbb{R}^d R d I n I_n I n [24 Theorem 2] in a multi-modal setting. It is asymptotic in the sense that it relies on Lebesgue’s dominated convergence theorem. Furthermore, we see in the proof that the dependence on d d d
Non-asymptotic result for q ∈ [ 1 3 , 1 2 ) q \in[\frac{1}{3},\frac{1}{2}) q ∈ [ 3 1 , 2 1 )
From a theoretical and applied perspective, for example in the Bayesian inverse problem framework, it is particularly interesting, on the one hand, to have a fully explicit non-asymptotic estimate, and on the other hand, to achieve a ‘nice’ dependence on d d d V n K , R , q V_n^{K,R,q} V n K , R , q n n n q ∈ [ 1 3 , 1 2 ) q \in[\frac{1}{3},\frac{1}{2}) q ∈ [ 3 1 , 2 1 )
Assumption 3. For R , q , δ > 0 R,q,\delta> 0 R , q , δ > 0 n ∈ N n \in\mathbb{N} n ∈ N x ∈ V ‾ n R , q x \in\overline{V}_n^{R,q} x ∈ V n R , q
e − n I n ( x ) ≤ ∑ k = 1 K π n ( k ) exp ( − n δ ∥ x − x n ( k ) ∥ H n ( k ) 2 ) . \mathrm{e}^{-nI_n(x)} \le\sum_{k=1}^{K}\pi_n^{(k)}\exp\left(-n\delta\left\|x-x_n^{(k)}\right\|_{H_n^{(k)}}^2\right). e − n I n ( x ) ≤ k = 1 ∑ K π n ( k ) exp ( − n δ x − x n ( k ) H n ( k ) 2 ) . Remark 3.4. Intuitively, Assumption 3 3 3 is more restrictive than Assumption 2 2 2 since it essentially imposes at least ‘Gaussian decay’ at infinity, whereas Assumption 2 2 2 also allows for heavier ‘tail behaviors’. For instance, for the I n : R → [ 0 , ∞ ) I_n\colon\mathbb{R}\to[0,\infty) I n : R → [ 0 , ∞ ) with I n ( x ) = x 2 + 1 − 1 I_n(x)=\sqrt{x^2+1}-1 I n ( x ) = x 2 + 1 − 1 there is an α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) such that Assumption 2 2 2 is satisfied, but there are no numbers R , q , δ ∈ ( 0 , ∞ ) R,q,\delta\in(0,\infty) R , q , δ ∈ ( 0 , ∞ ) such that Assumption 3 3 3 is true.
Note that the combination of Assumption 1 1 1 3 3 3 [11 Assumption 3.3] and extended to the multi-modal posterior distribution framework. It particularly allows us to obtain an explicit non-asymptotic result, in contrast to the asymptotic statement of Theorem 3.1. We provide this result now and present its proof in Appendix § 7 .
Theorem 3.2. For q ∈ [ 1 3 , 1 2 ) q\in\left[\frac{1}{3},\frac{1}{2}\right) q ∈ [ 3 1 , 2 1 ) ρ , M , δ ∈ ( 0 , ∞ ) \rho,M,\delta\in(0,\infty) ρ , M , δ ∈ ( 0 , ∞ ) R = ρ 2 / M 2 3 R=\rho^2/M^{\frac{2}{3}} R = ρ 2 / M 3 2 1 1 1 3 3 3 n ∈ N n\in\mathbb{N} n ∈ N
d H ( L π n K , π n ) ≤ 13 d 3 2 M n 1 2 ρ 3 + 2 d 2 + 2 exp ( − ρ 2 min { 1 , 2 δ } 8 M 2 3 n ( 1 − 2 q ) ) [ ( K + 1 ) + max { 1 , ( 2 δ ) − d 2 } ] . d_{\mathrm{H}}\left(\mathcal{L}_{\pi_n}^{K},\pi_n\right)\leq\frac{13d^{\frac{3}{2}}M}{n^{\frac{1}{2}}\rho^3}+2^{\frac{d}{2}+2}\exp\left(-\frac{\rho^2\min\{1,2\delta\}}{8M^{\frac{2}{3}}}n^{(1-2q)}\right)\left[(K+1)+\max\{1,(2\delta)^{-\frac{d}{2}}\}\right]. d H ( L π n K , π n ) ≤ n 2 1 ρ 3 13 d 2 3 M + 2 2 d + 2 exp ( − 8 M 3 2 ρ 2 min { 1 , 2 δ } n ( 1 − 2 q ) ) [ ( K + 1 ) + max { 1 , ( 2 δ ) − 2 d } ] . A consequence of the previous theorem is the following illustrative result, where we provide the (technical) proof in Appendix § 7 .4.
Corollary 3.1. Under the assumptions of Theorem 3.2 for δ ≥ 1 / 2 \delta\geq1/2 δ ≥ 1/2 the bound simplifies to
d H ( L π n K , π n ) ≤ 13 d 3 2 M n 1 2 ρ 3 + 2 d 2 + 2 ( K + 2 ) exp ( − ρ 2 n ( 1 − 2 q ) 8 M 2 3 ) , (9) d_{\mathrm{H}}\left(\mathcal{L}_{\pi_n}^{K},\pi_n\right)\leq\frac{13d^{\frac{3}{2}}M}{n^{\frac{1}{2}}\rho^3}+2^{\frac{d}{2}+2}(K+2)\exp\left(-\frac{\rho^2n^{(1-2q)}}{8M^{\frac{2}{3}}}\right), \tag*{(9)} d H ( L π n K , π n ) ≤ n 2 1 ρ 3 13 d 2 3 M + 2 2 d + 2 ( K + 2 ) exp ( − 8 M 3 2 ρ 2 n ( 1 − 2 q ) ) , (9) such that d H ( L π n K , π n ) ≤ 14 d 3 2 M n 1 2 ρ 3 d_{\mathrm{H}}\left(\mathcal{L}_{\pi_n}^{K},\pi_n\right)\leq\frac{14d^{\frac{3}{2}}M}{n^{\frac{1}{2}}\rho^3} d H ( L π n K , π n ) ≤ n 2 1 ρ 3 14 d 2 3 M
n ≥ max { [ 16 M 2 3 ρ 2 ( 1 − 2 q ) log ( 16 M 2 3 ρ 2 ( 1 − 2 q ) ) ] 1 − 2 q , [ 16 M 2 3 ρ 2 log ( 2 d 2 + 2 ( K + 2 ) ρ 3 M ) ] 1 1 − 2 q } . (10) n\geq\max\left\{\left[\frac{16M^{\frac{2}{3}}}{\rho^2(1-2q)}\log\left(\frac{16M^{\frac{2}{3}}}{\rho^2(1-2q)}\right)\right]^{1-2q},\left[\frac{16M^{\frac{2}{3}}}{\rho^2}\log\left(\frac{2^{\frac{d}{2}+2}(K+2)\rho^3}{M}\right)\right]^{\frac{1}{1-2q}}\right\}. \tag*{(10)} n ≥ max ⎩ ⎨ ⎧ [ ρ 2 ( 1 − 2 q ) 16 M 3 2 log ( ρ 2 ( 1 − 2 q ) 16 M 3 2 ) ] 1 − 2 q , [ ρ 2 16 M 3 2 log ( M 2 2 d + 2 ( K + 2 ) ρ 3 ) ] 1 − 2 q 1 ⎭ ⎬ ⎫ . (10) Let us interpret the latter bound regarding the dependence on the dimension d d d n n n q ∈ [ 1 3 , 1 2 ) q\in\left[\frac{1}{3},\frac{1}{2}\right) q ∈ [ 3 1 , 2 1 ) ρ , M , δ ∈ [ 1 2 , ∞ ) \rho,M,\delta\in\left[\frac{1}{2},\infty\right) ρ , M , δ ∈ [ 2 1 , ∞ ) (10) we see that there is a constant c > 0 c>0 c > 0 q , ρ , M , δ q,\rho,M,\delta q , ρ , M , δ ε ∈ ( 0 , 1 ) \varepsilon\in(0,1) ε ∈ ( 0 , 1 ) n ≥ c ( d 3 2 ε − 2 + d 1 1 − 2 q ) n\geq c(d^{\frac{3}{2}}\varepsilon^{-2}+d^{\frac{1}{1-2q}}) n ≥ c ( d 2 3 ε − 2 + d 1 − 2 q 1 ) d H ( L π n K , π n ) ≤ ε d_{\mathrm{H}}\left(\mathcal{L}_{\pi_n}^{K},\pi_n\right)\leq\varepsilon d H ( L π n K , π n ) ≤ ε
Another consequence of the previous theorem is an explicit bound on χ 2 ( π n ∥ L π n K ) \chi^2(\pi_n\mathbin\Vert\mathcal{L}_{\pi_n}^{K}) χ 2 ( π n ∥ L π n K ) (5) and an auxiliary estimate (also used in the proof of Theorem 3.1, see Lemma A.3) to control the appearing ratio of normalizing constants. It is proven in Section A.5.
Corollary 3.2. Under the assumptions of Theorem 3.2 for δ ≥ 1 / 2 \delta\geq1/2 δ ≥ 1/2 and
n ≥ max { δ 2 e 2 d 3 M 2 ρ 6 , [ 4 M 2 3 ρ 2 log ( 2 2 + d 2 ( K + 2 ) ) ] 1 1 − 2 q } (11) n\geq\max\left\{\frac{\delta^2e^2d^3M^2}{\rho^6},\left[\frac{4M^{\frac{2}{3}}}{\rho^2}\log\left(2^{2+\frac{d}{2}}(K+2)\right)\right]^{\frac{1}{1-2q}}\right\} \tag*{(11)} n ≥ max ⎩ ⎨ ⎧ ρ 6 δ 2 e 2 d 3 M 2 , [ ρ 2 4 M 3 2 log ( 2 2 + 2 d ( K + 2 ) ) ] 1 − 2 q 1 ⎭ ⎬ ⎫ (11) we have π n ( x ) L π n K ( x ) ≤ 2 e \frac{\pi_n(x)}{\mathcal{L}_{\pi_n}^{K}(x)}\leq2e L π n K ( x ) π n ( x ) ≤ 2 e as well as
χ 2 ( π n ∥ L π n K ) ≤ 12 d H ( L π n K , π n ) 2 ≤ 4056 d 3 M 2 n ρ 6 + 2 d + 9 ( K + 2 ) 2 exp ( − ρ 2 n ( 1 − 2 q ) 4 M 2 3 ) . \chi^2\left(\pi_n\mathbin\Vert\mathcal{L}_{\pi_n}^{K}\right)\leq12d_{\mathrm{H}}\left(\mathcal{L}_{\pi_n}^{K},\pi_n\right)^2\leq4056\frac{d^3M^2}{n\rho^6}+2^{d+9}(K+2)^2\exp\left(-\frac{\rho^2n^{(1-2q)}}{4M^{\frac{2}{3}}}\right). χ 2 ( π n ∥ L π n K ) ≤ 12 d H ( L π n K , π n ) 2 ≤ 4056 n ρ 6 d 3 M 2 + 2 d + 9 ( K + 2 ) 2 exp ( − 4 M 3 2 ρ 2 n ( 1 − 2 q ) ) . We finish this section with a remark on the idea of proof of Theorem 3.1.
Remark 3.5. In contrast to Theorem 3.1 in Theorem 3.2 we use that the ‘tail’ of the distribution is comparable to that of a normal distribution, cf. Assumption 3. With that and requiring for increasing n n n V n k , R , q V_n^{k,R,q} V n k , R , q d d d n n n n − q n^{-q} n − q π n \pi_n π n V n k , R , q V_n^{k,R,q} V n k , R , q
Laplace-based Monte Carlo for expectations
Now we are interested in the computation of the expectation of a quantity of interest f : R d → R f:\mathbb{R}^d \to\mathbb{R} f : R d → R π n \pi_n π n f f f π n \pi_n π n ( Ω , A , P ) (\Omega,\mathcal{A},\mathbb{P}) ( Ω , A , P ) ν \nu ν R d \mathbb{R}^d R d π n \pi_n π n w n : = d π n d ν w_n := \frac{\mathrm{d}\pi_n}{\mathrm{d}\nu} w n := d ν d π n
E π n [ f ] : = ∫ R d f d π n = ∫ R d f ( x ) w n ( x ) ν ( d x ) . \mathbb{E}_{\pi_n}[f] := \int_{\mathbb{R}^d} f\,\mathrm{d}\pi_n = \int_{\mathbb{R}^d} f(x)\,w_n(x)\,\nu(\mathrm{d}x). E π n [ f ] := ∫ R d f d π n = ∫ R d f ( x ) w n ( x ) ν ( d x ) . Unfortunately, it is quite common that w n w_n w n w n ∗ : R d → [ 0 , ∞ ) w_n^*:\mathbb{R}^d \to[0,\infty) w n ∗ : R d → [ 0 , ∞ ) w n w_n w n
E π n [ f ] = ∫ R d f ( x ) w n ∗ ( x ) ν ( d x ) ∫ R d w n ∗ ( x ) ν ( d x ) . (12) \mathbb{E}_{\pi_n}[f] = \frac{\int_{\mathbb{R}^d} f(x)w_n^*(x)\nu(\mathrm{d}x)}{\int_{\mathbb{R}^d} w_n^*(x)\nu(\mathrm{d}x)}. \tag*{(12)} E π n [ f ] = ∫ R d w n ∗ ( x ) ν ( d x ) ∫ R d f ( x ) w n ∗ ( x ) ν ( d x ) . (12) Laplace-based importance sampling
For n ∈ N n \in\mathbb{N} n ∈ N ( X i , n ) i ∈ N (X_{i,n})_{i\in\mathbb{N}} ( X i , n ) i ∈ N ν \nu ν self-normalized importance sampling (SNIS) w.r.t. ν \nu ν
IS π n , ν N ( f ) : = ∑ i = 1 N w n ∗ ( X i , n ) f ( X i , n ) ∑ j = 1 N w n ∗ ( X j , n ) . \operatorname{IS}^{N}_{\pi_n,\nu}(f) := \frac{\sum_{i=1}^{N} w_n^*(X_{i,n})f(X_{i,n})}{\sum_{j=1}^{N} w_n^*(X_{j,n})}. IS π n , ν N ( f ) := ∑ j = 1 N w n ∗ ( X j , n ) ∑ i = 1 N w n ∗ ( X i , n ) f ( X i , n ) . It is well-known that the strong law of large numbers yields consistency of SNIS, see [19 Theorem 9.2] , i.e., we have for any n ∈ N n \in\mathbb{N} n ∈ N
IS π n , ν N ( f ) → N → ∞ a . s . E π n [ f ] . \operatorname{IS}^{N}_{\pi_n,\nu}(f) \xrightarrow[N\to\infty]{\mathrm{a.s.}} \mathbb{E}_{\pi_n}[f]. IS π n , ν N ( f ) a.s. N → ∞ E π n [ f ] . Moreover, by the central limit theorem and Slutzky’s lemma one can prove the asymptotic normality of the SNIS estimator, i.e., we have
N ( IS π n , ν N ( f ) − E π n [ f ] ) → N → ∞ d N ( 0 , σ n 2 ( f ) ) \sqrt{N}\left(\operatorname{IS}^{N}_{\pi_n,\nu}(f)-\mathbb{E}_{\pi_n}[f]\right) \xrightarrow[N\to\infty]{\mathrm{d}} \mathcal{N}\left(0,\sigma_n^2(f)\right) N ( IS π n , ν N ( f ) − E π n [ f ] ) d N → ∞ N ( 0 , σ n 2 ( f ) ) where → d \xrightarrow{\mathrm{d}} d asymptotic variance of SNIS is finite, that is,
σ n 2 ( f ) : = E ν [ w n 2 ( f − E π n [ f ] ) 2 ] < ∞ . (13) \sigma_n^2(f) := \mathbb{E}_{\nu}\left[w_n^2\left(f-\mathbb{E}_{\pi_n}[f]\right)^2\right] < \infty. \tag*{(13)} σ n 2 ( f ) := E ν [ w n 2 ( f − E π n [ f ] ) 2 ] < ∞. (13) All of these statements also apply if ν \nu ν n n n ν n \nu_n ν n π n \pi_n π n
The asymptotic variance serves as a measure of the efficiency of SNIS. Although it depends on f f f f f f ∥ f ∥ ∞ < ∞ \|f\|_\infty< \infty ∥ f ∥ ∞ < ∞ σ n 2 ( f ) ≤ 4 ∥ f ∥ ∞ 2 E ν [ w n 2 ] \sigma_n^2(f) \leq4\|f\|_\infty^2 \mathbb{E}_\nu[w_n^2] σ n 2 ( f ) ≤ 4∥ f ∥ ∞ 2 E ν [ w n 2 ] E ν [ w n 2 ] \mathbb{E}_\nu[w_n^2] E ν [ w n 2 ] χ 2 ( π n ∥ ν ) + 1 \chi^2(\pi_n \mathbin\Vert\nu) + 1 χ 2 ( π n ∥ ν ) + 1 [1 Section 2.3] .
This motivates another common measure of efficiency for SNIS: the effective sample size (ESS) . It is defined as
ESS n ( N ) : = ( ∑ j = 1 N w n ∗ ( X j , n ) ) 2 ∑ i = 1 N ( w n ∗ ( X i , n ) ) 2 . (14) \operatorname{ESS}_n(N) := \frac{\left(\sum_{j=1}^{N} w_n^*(X_{j,n})\right)^2}{\sum_{i=1}^{N} (w_n^*(X_{i,n}))^2}. \tag*{(14)} ESS n ( N ) := ∑ i = 1 N ( w n ∗ ( X i , n ) ) 2 ( ∑ j = 1 N w n ∗ ( X j , n ) ) 2 . (14) Thus, ESS n ( N ) \operatorname{ESS}_n(N) ESS n ( N ) ESS n ( N ) ≤ N \operatorname{ESS}_n(N) \leq N ESS n ( N ) ≤ N [1 ] , the expression N ESS n ( N ) − 1 \frac{N}{\operatorname{ESS}_n(N)} - 1 ESS n ( N ) N − 1 χ 2 ( π n ∥ ν ) \chi^2(\pi_n \mathbin\Vert\nu) χ 2 ( π n ∥ ν ) χ 2 ( π n ∥ ν ) < ∞ \chi^2(\pi_n \mathbin\Vert\nu) < \infty χ 2 ( π n ∥ ν ) < ∞
ESS n ( N ) N → N → ∞ a . s . 1 χ 2 ( π n ∥ ν ) + 1 . \frac{\operatorname{ESS}_n(N)}{N} \xrightarrow[N\to\infty]{\mathrm{a.s.}} \frac{1}{\chi^2(\pi_n \mathbin\Vert\nu)+1}. N ESS n ( N ) a.s. N → ∞ χ 2 ( π n ∥ ν ) + 1 1 . and therefore for bounded f f f χ 2 ( π n ∥ ν ) < ∞ \chi^2(\pi_n \mathbin\Vert\nu) < \infty χ 2 ( π n ∥ ν ) < ∞
σ n 2 ( f ) ≤ 4 ∥ f ∥ ∞ 2 lim N → ∞ ( ESS n ( N ) N ) − 1 . \sigma_n^2(f) \leq4\|f\|_\infty^2 \lim_{N\to\infty}\left(\frac{\operatorname{ESS}_n(N)}{N}\right)^{-1}. σ n 2 ( f ) ≤ 4∥ f ∥ ∞ 2 N → ∞ lim ( N ESS n ( N ) ) − 1 . Hence, 1 χ 2 ( π n ∥ ν ) + 1 \frac{1}{\chi^2(\pi_n \mathbin\Vert\nu)+1} χ 2 ( π n ∥ ν ) + 1 1 asymptotic efficiency of SNIS as the limit of the effective sample size and we set
eff π n I S ( ν ) : = 1 χ 2 ( π n ∥ ν ) + 1 ∈ [ 0 , 1 ] . \operatorname{eff}_{\pi_n}^{\mathrm{IS}}(\nu) := \frac{1}{\chi^2(\pi_n \mathbin\Vert\nu)+1} \in[0,1]. eff π n IS ( ν ) := χ 2 ( π n ∥ ν ) + 1 1 ∈ [ 0 , 1 ] . Prior-based importance sampling. We first consider ν = π 0 \nu= \pi_0 ν = π 0 n → ∞ n \to\infty n → ∞ π n \pi_n π n
Theorem 4.1. Let Φ : R d → [ 0 , ∞ ) \Phi: \mathbb{R}^d \to[0,\infty) Φ : R d → [ 0 , ∞ ) n n n I n ( x ) : = Φ ( x ) − 1 n log ( π 0 ( x ) ) I_n(x) := \Phi(x) - \frac{1}{n}\log(\pi_0(x)) I n ( x ) := Φ ( x ) − n 1 log ( π 0 ( x )) n ∈ N n \in\mathbb{N} n ∈ N x ∈ R d x \in\mathbb{R}^d x ∈ R d
there is a unique global minimizer x ∗ ∈ R d x_* \in\mathbb{R}^d x ∗ ∈ R d Φ \Phi Φ H ∗ : = ∇ 2 Φ ( x ∗ ) H_* := \nabla^2\Phi(x_*) H ∗ := ∇ 2 Φ ( x ∗ )
for every r > 0 r > 0 r > 0 inf x ∈ B r c ( x ∗ ) Φ ( x ) > Φ ( x ∗ ) \inf_{x\in B_r^c(x_*)}\Phi(x) > \Phi(x_*) inf x ∈ B r c ( x ∗ ) Φ ( x ) > Φ ( x ∗ ) B r ( x ∗ ) : = { x ∈ R d : ∥ x − x ∗ ∥ ≤ r } B_r(x_*) := \{x \in\mathbb{R}^d : \|x-x_*\| \leq r\} B r ( x ∗ ) := { x ∈ R d : ∥ x − x ∗ ∥ ≤ r } B r c ( x ∗ ) : = R d \ B r ( x ∗ ) B_r^c(x_*) := \mathbb{R}^d \backslash B_r(x_*) B r c ( x ∗ ) := R d \ B r ( x ∗ )
Then, we have eff π n I S ( π 0 ) ∈ O ( n − d / 2 ) \operatorname{eff}_{\pi_n}^{\mathrm{IS}}(\pi_0) \in O\left(n^{-d/2}\right) eff π n IS ( π 0 ) ∈ O ( n − d /2 )
Proof. We apply the classical Laplace method for the asymptotics of integrals [30 Section IX.5] twice and obtain as n → ∞ n \to\infty n → ∞
∫ R d exp ( − n Φ ( x ) ) π 0 ( x ) d x = exp ( − n Φ ( x ∗ ) ) n − d / 2 ( c 0 ( π 0 ) + O ( n − 1 ) ) , and \int_{\mathbb{R}^d} \exp\left(-n\Phi(x)\right)\pi_0(x)\,\mathrm{d}x = \exp\left(-n\Phi(x_*)\right)n^{-d/2}\left(c_0(\pi_0) + O\left(n^{-1}\right)\right), \qquad\text{and} ∫ R d exp ( − n Φ ( x ) ) π 0 ( x ) d x = exp ( − n Φ ( x ∗ ) ) n − d /2 ( c 0 ( π 0 ) + O ( n − 1 ) ) , and ∫ R d exp ( − 2 n Φ ( x ) ) π 0 ( x ) d x = exp ( − 2 n Φ ( x ∗ ) ) ( 2 n ) − d / 2 ( c 0 ( π 0 ) + O ( n − 1 ) ) , \int_{\mathbb{R}^d} \exp\left(-2n\Phi(x)\right)\pi_0(x)\,\mathrm{d}x = \exp\left(-2n\Phi(x_*)\right)(2n)^{-d/2}\left(c_0(\pi_0) + O\left(n^{-1}\right)\right), ∫ R d exp ( − 2 n Φ ( x ) ) π 0 ( x ) d x = exp ( − 2 n Φ ( x ∗ ) ) ( 2 n ) − d /2 ( c 0 ( π 0 ) + O ( n − 1 ) ) , where c 0 ( π 0 ) : = det ( 2 π H ∗ − 1 ) π 0 ( x ∗ ) > 0 c_0(\pi_0) := \sqrt{\det(2\pi H_*^{-1})}\pi_0(x_*) > 0 c 0 ( π 0 ) := det ( 2 π H ∗ − 1 ) π 0 ( x ∗ ) > 0 Z n = ∫ R d exp ( − n Φ ( x ) ) π 0 ( x ) d x Z_n = \int_{\mathbb{R}^d} \exp(-n\Phi(x))\pi_0(x)\,dx Z n = ∫ R d exp ( − n Φ ( x )) π 0 ( x ) d x
eff π n I S ( π 0 ) = 1 E π 0 ( w n 2 ) = ( ∫ R d exp ( − n Φ ( x ) ) π 0 ( x ) d x ) 2 ∫ R d exp ( − 2 n Φ ( x ) ) π 0 ( x ) d x = n − d ( c 0 ( π 0 ) + O ( n − 1 ) ) 2 ( 2 n ) − d / 2 ( c 0 ( π 0 ) + O ( n − 1 ) ) , \operatorname{eff}^{\mathrm{IS}}_{\pi_n}(\pi_0) = \frac{1}{\mathbb{E}_{\pi_0}(w_n^2)} = \frac{\left(\int_{\mathbb{R}^d}\exp(-n\Phi(x))\pi_0(x)\,dx\right)^2}{\int_{\mathbb{R}^d}\exp(-2n\Phi(x))\pi_0(x)\,dx} = \frac{n^{-d}\left(c_0(\pi_0) + O\left(n^{-1}\right)\right)^2}{(2n)^{-d/2}\left(c_0(\pi_0) + O\left(n^{-1}\right)\right)}, eff π n IS ( π 0 ) = E π 0 ( w n 2 ) 1 = ∫ R d exp ( − 2 n Φ ( x )) π 0 ( x ) d x ( ∫ R d exp ( − n Φ ( x )) π 0 ( x ) d x ) 2 = ( 2 n ) − d /2 ( c 0 ( π 0 ) + O ( n − 1 ) ) n − d ( c 0 ( π 0 ) + O ( n − 1 ) ) 2 , which implies the claim.
This theorem shows that the prior-based importance sampling suffers from a curse of dimensionality and also becomes less efficient as the target distribution π n \pi_n π n
Laplace-based importance sampling. Next we consider SNIS based on the multi-modal version of the LA, i.e., ν = L π n K \nu= \mathcal{L}_{\pi_n}^K ν = L π n K
w n ∗ ( x ) : = exp ( − n I n ( x ) ) ∑ k = 1 K π n ( k ) exp ( − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 ) , x ∈ R d . (15) w_n^*(x) := \frac{\exp(-nI_n(x))}{\sum_{k=1}^K \pi_n^{(k)} \exp\left(-\frac{n}{2}\left\|x-x_n^{(k)}\right\|_{H_n^{(k)}}^2\right)}, \qquad x \in\mathbb{R}^d. \tag*{(15)} w n ∗ ( x ) := ∑ k = 1 K π n ( k ) exp ( − 2 n x − x n ( k ) H n ( k ) 2 ) exp ( − n I n ( x )) , x ∈ R d . (15) We start with statements about the asymptotic efficiency eff π n I S ( L π n K ) \operatorname{eff}_{\pi_n}^{\mathrm{IS}}(\mathcal{L}_{\pi_n}^K) eff π n IS ( L π n K )
Corollary 4.1. For q ∈ [ 1 3 , 1 2 ) q \in\left[\frac{1}{3},\frac{1}{2}\right) q ∈ [ 3 1 , 2 1 ) δ ≥ 1 / 2 \delta\ge1/2 δ ≥ 1/2 ρ , M ∈ ( 0 , ∞ ) \rho,M \in(0,\infty) ρ , M ∈ ( 0 , ∞ ) R = ρ 2 / M 2 3 R = \rho^2/M^{\frac{2}{3}} R = ρ 2 / M 3 2 Assumption 1 and Assumption 3 be satisfied. Then, for Laplace-based importance sampling we have
1 − eff π n I S ( L π n K ) ∈ O ( n − 1 ) . 1-\operatorname{eff}_{\pi_n}^{\mathrm{IS}}(\mathcal{L}_{\pi_n}^K) \in O\left(n^{-1}\right). 1 − eff π n IS ( L π n K ) ∈ O ( n − 1 ) . However, particularly, Assumption 3 is not necessarily satisfied in Bayesian inference with δ ≥ 1 2 \delta\ge\frac{1}{2} δ ≥ 2 1 ESS n ( N ) \operatorname{ESS}_n(N) ESS n ( N ) Assumption 1 . For this, we formulate the following additional requirement.
Assumption 4. Let Φ ∈ C 2 ( R d , [ 0 , ∞ ) ) \Phi\in C^2(\mathbb{R}^d,[0,\infty)) Φ ∈ C 2 ( R d , [ 0 , ∞ )) n n n I n ( x ) : = Φ ( x ) − 1 n log ( π 0 ( x ) ) I_n(x) := \Phi(x) - \frac{1}{n}\log(\pi_0(x)) I n ( x ) := Φ ( x ) − n 1 log ( π 0 ( x )) n ∈ N n \in\mathbb{N} n ∈ N x ∈ R d x \in\mathbb{R}^d x ∈ R d Φ \Phi Φ K ∈ N K \in\mathbb{N} K ∈ N x ∗ ( 1 ) , … , x ∗ ( K ) ∈ R d x_*^{(1)},\ldots,x_*^{(K)} \in\mathbb{R}^d x ∗ ( 1 ) , … , x ∗ ( K ) ∈ R d Φ ( x ∗ ( k ) ) = 0 \Phi(x_*^{(k)}) = 0 Φ ( x ∗ ( k ) ) = 0 ∇ 2 Φ ( x ∗ ( k ) ) \nabla^2\Phi(x_*^{(k)}) ∇ 2 Φ ( x ∗ ( k ) )
lim n → ∞ x n ( k ) = x ∗ ( k ) , k = 1 , … , K . \lim_{n\to\infty}x_n^{(k)}=x_*^{(k)}, \qquad k=1,\ldots,K. n → ∞ lim x n ( k ) = x ∗ ( k ) , k = 1 , … , K . Remark 4.2. It is worth to emphasize here that the conditions of Assumption 4 4 4 4.1 4.1 4.1 § 2 . These just provide a refinement of requirements that can be further exploited and allow new conclusions.
In our (asymptotic) analysis of the effective sample size of SNIS we exploit the fact that for increasing ‘concentration’ (i.e. increasing n n n L π n K \mathcal{L}_{\pi_n}^{K} L π n K
Proposition 4.1. Let R > 0 R>0 R > 0 N ∈ N N\in\mathbb{N} N ∈ N q ∈ [ 0 , 1 / 2 ) q\in[0,1/2) q ∈ [ 0 , 1/2 ) X 1 , n , … , X N , n X_{1,n},\ldots,X_{N,n} X 1 , n , … , X N , n L π n K \mathcal{L}_{\pi_n}^{K} L π n K
G n , N : = ⋂ i = 1 N { min k = 1 , … , K ∥ X i , n − x n ( k ) ∥ H n ( k ) 2 ≤ R n − 2 q } . (16) G_{n,N}:=\bigcap_{i=1}^{N}\left\{\min_{k=1,\ldots,K}\left\|X_{i,n}-x_{n}^{(k)}\right\|_{H_{n}^{(k)}}^{2}\leq Rn^{-2q}\right\}. \tag*{(16)} G n , N := i = 1 ⋂ N { k = 1 , … , K min X i , n − x n ( k ) H n ( k ) 2 ≤ R n − 2 q } . (16) Then, lim n → ∞ P ( G n , N ) = 1 \lim_{n\to\infty}\mathbb{P}(G_{n,N})=1 lim n → ∞ P ( G n , N ) = 1
The proof can be found in Section B.1. As a consequence of these former result we are able to add the following tool which is proven in Section B.2.
Lemma 4.1. For q ∈ ( 1 3 , 1 2 ) q\in\left(\frac{1}{3},\frac{1}{2}\right) q ∈ ( 3 1 , 2 1 ) R , ρ , M ∈ ( 0 , ∞ ) R,\rho,M\in(0,\infty) R , ρ , M ∈ ( 0 , ∞ ) 1 1 1 4 4 4 N ∈ N N\in\mathbb{N} N ∈ N X 1 , n , … , X N , n X_{1,n},\ldots,X_{N,n} X 1 , n , … , X N , n L π n K \mathcal{L}_{\pi_n}^{K} L π n K G n , N G_{n,N} G n , N (16) . Then, there exists a constant c > 0 c>0 c > 0 n 0 ∈ N n_0\in\mathbb{N} n 0 ∈ N n ∈ N n\in\mathbb{N} n ∈ N n ≥ n 0 n\geq n_0 n ≥ n 0 ESS n ( N ) \operatorname{ESS}_{n}(N) ESS n ( N ) X 1 , n , … , X N , n X_{1,n},\ldots,X_{N,n} X 1 , n , … , X N , n 5 G n , N G_{n,N} G n , N
∣ ESS n ( N ) − N ∣ ≤ c N n 2 − 6 q . (17) \left|\operatorname{ESS}_{n}(N)-N\right|\leq cNn^{2-6q}. \tag*{(17)} ∣ ESS n ( N ) − N ∣ ≤ c N n 2 − 6 q . (17) Combining the former two results leads to the following asymptotic statement regarding the effective sample size.
Theorem 4.2. For q ∈ ( 1 3 , 1 2 ) q\in\left(\frac{1}{3},\frac{1}{2}\right) q ∈ ( 3 1 , 2 1 ) R , ρ , M ∈ ( 0 , ∞ ) R,\rho,M\in(0,\infty) R , ρ , M ∈ ( 0 , ∞ ) 1 1 1 4 4 4 N ∈ N N\in\mathbb{N} N ∈ N X 1 , n , … , X N , n X_{1,n},\ldots,X_{N,n} X 1 , n , … , X N , n L π n K \mathcal{L}_{\pi_n}^{K} L π n K ESS n ( N ) \operatorname{ESS}_{n}(N) ESS n ( N ) X 1 , n , … , X N , n X_{1,n},\ldots,X_{N,n} X 1 , n , … , X N , n γ ∈ ( 0 , 6 q − 2 ) \gamma\in(0,6q-2) γ ∈ ( 0 , 6 q − 2 )
n γ ∣ ESS n ( N ) − N ∣ → n → ∞ P 0. n^{\gamma}\left|\operatorname{ESS}_{n}(N)-N\right|\xrightarrow[n\to\infty]{\mathbb{P}}0. n γ ∣ ESS n ( N ) − N ∣ P n → ∞ 0. where → n → ∞ P \xrightarrow[n\to\infty]{\mathbb{P}} P n → ∞
Proof. Let ε > 0 \varepsilon>0 ε > 0 c > 0 c>0 c > 0 n 0 n_0 n 0 4.1 4.1 4.1 n ∈ N n\in\mathbb{N} n ∈ N n ≥ n 0 n\geq n_0 n ≥ n 0 n ≥ ( c N / ε ) 1 / ( 6 q − 2 − γ ) n\geq(cN/\varepsilon)^{1/(6q-2-\gamma)} n ≥ ( c N / ε ) 1/ ( 6 q − 2 − γ ) G n , N G_{n,N} G n , N (16) ), we have
n γ ∣ ESS n ( N ) − N ∣ ≤ c N n γ + 2 − 6 q ≤ ε . n^{\gamma}\left|\operatorname{ESS}_{n}(N)-N\right|\leq cNn^{\gamma+2-6q}\leq\varepsilon. n γ ∣ ESS n ( N ) − N ∣ ≤ c N n γ + 2 − 6 q ≤ ε . Therefore
lim n → ∞ P ( n γ ∣ ESS n ( N ) − N ∣ ≤ ε ) ≥ lim n → ∞ P ( n γ ∣ ESS n ( N ) − N ∣ ≤ ε and G n , N ) = lim n → ∞ P ( G n , N ) . \lim_{n\to\infty}\mathbb{P}\left(n^{\gamma}\left|\operatorname{ESS}_{n}(N)-N\right|\leq\varepsilon\right)\geq\lim_{n\to\infty}\mathbb{P}\left(n^{\gamma}\left|\operatorname{ESS}_{n}(N)-N\right|\leq\varepsilon\ \text{and}\ G_{n,N}\right)=\lim_{n\to\infty}\mathbb{P}(G_{n,N}). n → ∞ lim P ( n γ ∣ ESS n ( N ) − N ∣ ≤ ε ) ≥ n → ∞ lim P ( n γ ∣ ESS n ( N ) − N ∣ ≤ ε and G n , N ) = n → ∞ lim P ( G n , N ) . By Proposition 4.1 4.1 4.1 lim n → ∞ P ( G n , N ) = 1 \lim_{n\to\infty}\mathbb{P}(G_{n,N})=1 lim n → ∞ P ( G n , N ) = 1 □ \square □
Laplace-based independent Metropolis-Hastings algorithm
We consider an alternative to SNIS for the approximation of E π n [ f ] \mathbb{E}_{\pi_n}[f] E π n [ f ] Independent Metropolis-Hastings (IMH) algorithm. The idea is to construct a Markov chain that can be used for approximate sampling of π n \pi_n π n ν \nu ν R d \mathbb{R}^d R d w n : = d π n d ν w_n := \frac{\mathrm{d}\pi_n}{\mathrm{d}\nu} w n := d ν d π n w n ∗ w_n^* w n ∗ ν \nu ν π n \pi_n π n ( X i , n ) i ∈ N (X_{i,n})_{i\in\mathbb{N}} ( X i , n ) i ∈ N i ∈ N i \in\mathbb{N} i ∈ N X i , n = x ∈ R d X_{i,n}=x\in\mathbb{R}^d X i , n = x ∈ R d X i + 1 , n X_{i+1,n} X i + 1 , n Y ∼ ν Y\sim\nu Y ∼ ν y y y acceptance probability
α n ( x , y ) = 1 ∧ w n ∗ ( y ) w n ∗ ( x ) , \alpha_n(x,y)=1\wedge\frac{w_n^*(y)}{w_n^*(x)}, α n ( x , y ) = 1 ∧ w n ∗ ( x ) w n ∗ ( y ) , accept the proposal y y y X i + 1 , n = y X_{i+1,n}=y X i + 1 , n = y X i + 1 , n = x X_{i+1,n}=x X i + 1 , n = x X i + 1 , n X_{i+1,n} X i + 1 , n X i , n = x X_{i,n}=x X i , n = x
I M n ( x , d y ) : = α n ( x , y ) ν ( d y ) + r n ( x ) δ x ( d y ) , \mathrm{IM}_n(x,\mathrm{d}y):=\alpha_n(x,y)\nu(\mathrm{d}y)+r_n(x)\delta_x(\mathrm{d}y), IM n ( x , d y ) := α n ( x , y ) ν ( d y ) + r n ( x ) δ x ( d y ) , where r n ( x ) : = ∫ R d ( 1 − α n ( x , y ) ) ν ( d y ) r_n(x):=\int_{\mathbb{R}^d}(1-\alpha_n(x,y))\,\nu(\mathrm{d}y) r n ( x ) := ∫ R d ( 1 − α n ( x , y )) ν ( d y ) δ x \delta_x δ x x x x R d \mathbb{R}^d R d ( X i , n ) i ∈ N (X_{i,n})_{i\in\mathbb{N}} ( X i , n ) i ∈ N I M n \mathrm{IM}_n IM n 6
lim t → ∞ d T V ( I M n t ( x , ⋅ ) , π n ) = 0 ∀ x ∈ R d . \lim_{t\to\infty}d_{\mathrm{TV}}\left(\mathrm{IM}_n^t(x,\cdot),\pi_n\right)=0 \qquad\forall x\in\mathbb{R}^d. t → ∞ lim d TV ( IM n t ( x , ⋅ ) , π n ) = 0 ∀ x ∈ R d . Moreover, for any π n \pi_n π n f : R d → R f:\mathbb{R}^d\to\mathbb{R} f : R d → R [27 Theorem 3] , such that
I M H π n , ν N ( f ) → N → ∞ a . s . E π n [ f ] , I M H π n , ν N ( f ) : = 1 N ∑ i = 1 N f ( X i , n ) . \mathrm{IMH}_{\pi_n,\nu}^N(f)\xrightarrow[N\to\infty]{\mathrm{a.s.}}\mathbb{E}_{\pi_n}[f],\qquad\mathrm{IMH}_{\pi_n,\nu}^N(f):=\frac{1}{N}\sum_{i=1}^{N}f(X_{i,n}). IMH π n , ν N ( f ) a.s. N → ∞ E π n [ f ] , IMH π n , ν N ( f ) := N 1 i = 1 ∑ N f ( X i , n ) . If the function w n w_n w n
∥ w n ∥ ∞ : = sup x ∈ R d w n ( x ) < ∞ , (18) \lVert w_n\rVert_\infty:=\sup_{x\in\mathbb{R}^d}w_n(x)<\infty, \tag*{(18)} ∥ w n ∥ ∞ := x ∈ R d sup w n ( x ) < ∞ , (18) then the IMH Markov chain is uniformly ergodic (cf. [17 ] ) and, by [29 Theorem 2] , we have
sup x ∈ R d d T V ( I M n t ( x , ⋅ ) , π n ) = ( 1 − 1 ∥ w n ∥ ∞ ) t , t ∈ N . (19) \sup_{x\in\mathbb{R}^d}d_{\mathrm{TV}}\left(\mathrm{IM}_n^t(x,\cdot),\pi_n\right)=\left(1-\frac{1}{\lVert w_n\rVert_\infty}\right)^t,\qquad t\in\mathbb{N}. \tag*{(19)} x ∈ R d sup d TV ( IM n t ( x , ⋅ ) , π n ) = ( 1 − ∥ w n ∥ ∞ 1 ) t , t ∈ N . (19) In this case, there also holds a central limit theorem (cf. [20 Corollary 2.1] ) for I M H π n , ν N ( f ) \mathrm{IMH}_{\pi_n,\nu}^N(f) IMH π n , ν N ( f )
N ( I M H π n , ν N ( f ) − E π n [ f ] ) → N → ∞ d N ( 0 , σ ~ n 2 ( f ) ) , f ∈ L 2 ( π n ) , \sqrt{N}\left(\mathrm{IMH}_{\pi_n,\nu}^N(f)-\mathbb{E}_{\pi_n}[f]\right)\xrightarrow[N\to\infty]{\mathrm{d}}\mathcal{N}\left(0,\widetilde{\sigma}_n^2(f)\right),\qquad f\in L^2(\pi_n), N ( IMH π n , ν N ( f ) − E π n [ f ] ) d N → ∞ N ( 0 , σ n 2 ( f ) ) , f ∈ L 2 ( π n ) , where the asymptotic variance is given by
σ ~ n 2 ( f ) : = Var π n ( f ) ( 1 + 2 ∑ i = 1 ∞ Corr ( f ( X ~ 1 , n ) , f ( X ~ i , n ) ) ) , (20) \widetilde{\sigma}_n^2(f) := \operatorname{Var}_{\pi_n}(f)\left(1 + 2\sum_{i=1}^{\infty} \operatorname{Corr}(f(\widetilde{X}_{1,n}), f(\widetilde{X}_{i,n}))\right), \tag*{(20)} σ n 2 ( f ) := Var π n ( f ) ( 1 + 2 i = 1 ∑ ∞ Corr ( f ( X 1 , n ) , f ( X i , n )) ) , (20) for ( X ~ i , n ) i ∈ N (\widetilde{X}_{i,n})_{i\in\mathbb{N}} ( X i , n ) i ∈ N π n \pi_n π n Corr ( ⋅ , ⋅ ) \operatorname{Corr}(\cdot,\cdot) Corr ( ⋅ , ⋅ ) [21 Theorem 3.34] , we have
lim N → ∞ N ⋅ E ∣ IMH π n , ν N ( f ) − E π n [ f ] ∣ 2 = σ ~ n 2 ( f ) . \lim_{N\to\infty} N\cdot\mathbb{E}\left|\operatorname{IMH}_{\pi_n,\nu}^{N}(f)-\mathbb{E}_{\pi_n}[f]\right|^2=\widetilde{\sigma}_n^2(f). N → ∞ lim N ⋅ E IMH π n , ν N ( f ) − E π n [ f ] 2 = σ n 2 ( f ) . Therefore, we are interested in the n n n M n : L 2 ( π n ) → L 2 ( π n ) M_n:L^2(\pi_n)\to L^2(\pi_n) M n : L 2 ( π n ) → L 2 ( π n ) IM n \operatorname{IM}_n IM n
( M n f ) ( x ) : = ∫ R d f ( y ) IM n ( x , d y ) , f ∈ L 2 ( π n ) . (M_nf)(x):=\int_{\mathbb{R}^d}f(y)\operatorname{IM}_n(x,\mathrm{d}y),\qquad f\in L^2(\pi_n). ( M n f ) ( x ) := ∫ R d f ( y ) IM n ( x , d y ) , f ∈ L 2 ( π n ) . Define L 0 2 ( π n ) ⊆ L 2 ( π n ) L_0^2(\pi_n)\subseteq L^2(\pi_n) L 0 2 ( π n ) ⊆ L 2 ( π n ) f : R d → R f:\mathbb{R}^d\to\mathbb{R} f : R d → R E π n [ f ] = 0 \mathbb{E}_{\pi_n}[f]=0 E π n [ f ] = 0 L 2 ( π n ) L^2(\pi_n) L 2 ( π n ) M n M_n M n
gap π n ( M n ) : = 1 − ∥ M n ∥ π n , ∥ M n ∥ π n : = sup f ∈ L 0 2 ( π n ) , f ≠ 0 ∥ M n f ∥ L 2 ( π n ) ∥ f ∥ L 2 ( π n ) . \operatorname{gap}_{\pi_n}(M_n):=1-\lVert M_n\rVert_{\pi_n},\qquad\lVert M_n\rVert_{\pi_n}:=\sup_{f\in L_0^2(\pi_n),f\ne0}\frac{\lVert M_nf\rVert_{L^2(\pi_n)}}{\lVert f\rVert_{L^2(\pi_n)}}. gap π n ( M n ) := 1 − ∥ M n ∥ π n , ∥ M n ∥ π n := f ∈ L 0 2 ( π n ) , f = 0 sup ∥ f ∥ L 2 ( π n ) ∥ M n f ∥ L 2 ( π n ) . Clearly, gap π n ( M n ) ∈ [ 0 , 1 ] \operatorname{gap}_{\pi_n}(M_n)\in[0,1] gap π n ( M n ) ∈ [ 0 , 1 ] [22 p. 6] , that
σ ~ n 2 ( f ) ≤ 2 ∥ f ∥ 2 , π n gap π n ( M n ) , ∀ f ∈ L 2 ( π n ) . \widetilde{\sigma}_n^2(f)\leq\frac{2\lVert f\rVert_{2,\pi_n}}{\operatorname{gap}_{\pi_n}(M_n)},\qquad\forall f\in L^2(\pi_n). σ n 2 ( f ) ≤ gap π n ( M n ) 2 ∥ f ∥ 2 , π n , ∀ f ∈ L 2 ( π n ) . Therefore, non-deteriorate n n n gap π n ( M n ) \operatorname{gap}_{\pi_n}(M_n) gap π n ( M n ) n n n [21 ] or the recent discussions in [12 ] . For the spectral gap of the IMH Markov operator we obtain the following lower bound, which allows a similar conclusion as (19) , cf. [29 ] .
Proposition 4.2. If ∥ w n ∥ ∞ < ∞ \lVert w_n\rVert_\infty<\infty ∥ w n ∥ ∞ < ∞
gap π n ( M n ) ≥ 1 ∥ w n ∥ ∞ . (21) \operatorname{gap}_{\pi_n}(M_n)\geq\frac{1}{\lVert w_n\rVert_\infty}. \tag*{(21)} gap π n ( M n ) ≥ ∥ w n ∥ ∞ 1 . (21) In particular, by [17 Theorem 2.1] , if ∥ w n ∥ ∞ = ∞ \lVert w_n\rVert_\infty=\infty ∥ w n ∥ ∞ = ∞ [6 ] , in that case gap π n ( M n ) = 0 \operatorname{gap}_{\pi_n}(M_n)=0 gap π n ( M n ) = 0 (19) for the total variation convergence) a dichotomy of its existence, determined by whether ∥ w n ∥ ∞ \lVert w_n\rVert_\infty ∥ w n ∥ ∞ [2 , 7 , 17 , 29 ] , but in our setting not really explicitly stated, such that we provide a proof in Section C.1.
Besides the asymptotic variance and/or spectral gap we provide seemingly less prominent, easier to handle quantities, that, however, have less direct consequences for stabilization. One is the averaged acceptance probability of IMH w.r.t. proposal distribution ν \nu ν
E π n ⊗ ν [ α n ] : = ∫ R d ∫ R d α n ( x , y ) ν ( d y ) π n ( d x ) ∈ [ 0 , 1 ] , (22) \mathbb{E}_{\pi_n\otimes\nu}[\alpha_n]:=\int_{\mathbb{R}^d}\int_{\mathbb{R}^d}\alpha_n(x,y)\,\nu(\mathrm{d}y)\pi_n(\mathrm{d}x)\in[0,1], \tag*{(22)} E π n ⊗ ν [ α n ] := ∫ R d ∫ R d α n ( x , y ) ν ( d y ) π n ( d x ) ∈ [ 0 , 1 ] , (22) and the other one is the expected squared jump distance in a direction v ∈ R d v \in\mathbb{R}^{d} v ∈ R d
ρ n v : = ∫ R d ∫ R d ∣ v T x − v T y ∣ 2 α n ( x , y ) ν ( d y ) π n ( d x ) . (23) \rho_{n}^{v} := \int_{\mathbb{R}^{d}} \int_{\mathbb{R}^{d}} \left|v^{T}x-v^{T}y\right|^{2}\alpha_{n}(x,y)\nu(\mathrm{d}y)\pi_{n}(\mathrm{d}x). \tag*{(23)} ρ n v := ∫ R d ∫ R d v T x − v T y 2 α n ( x , y ) ν ( d y ) π n ( d x ) . (23) Note, that ρ n v \rho_{n}^{v} ρ n v π n \pi_{n} π n v v v ( X ~ i , n ) i ∈ N (\widetilde{X}_{i,n})_{i\in\mathbb{N}} ( X i , n ) i ∈ N ν \nu ν π n \pi_{n} π n
Corr ( v T X ~ i , n , v T X ~ i + 1 , n ) = 1 − ρ n v 2 Var X ∼ π n ( v T X ) . (24) \operatorname{Corr}(v^{T}\widetilde{X}_{i,n},v^{T}\widetilde{X}_{i+1,n})=1-\frac{\rho_{n}^{v}}{2\operatorname{Var}_{X\sim\pi_{n}}(v^{T}X)}. \tag*{(24)} Corr ( v T X i , n , v T X i + 1 , n ) = 1 − 2 Var X ∼ π n ( v T X ) ρ n v . (24) In the case of an averaged acceptance probability and/or an expected squared jump distance that is zero the convergence behavior of the IMH Markov chain is ‘bad’. Namely, on average no proposed state is accepted and/or one stays always at the current state and got stuck. Therefore, asymptotic in n n n § 2 .
Lemma 4.2. For approximate sampling of π n \pi_{n} π n ν \nu ν
1 − 2 d T V ( π n , ν ) ≤ E π n ⊗ ν [ α n ] ≤ 1 − d T V ( π n , ν ) . 1-2d_{\mathrm{TV}}(\pi_{n},\nu)\leq\mathbb{E}_{\pi_{n}\otimes\nu}[\alpha_{n}]\leq1-d_{\mathrm{TV}}(\pi_{n},\nu). 1 − 2 d TV ( π n , ν ) ≤ E π n ⊗ ν [ α n ] ≤ 1 − d TV ( π n , ν ) . Moreover, if ν \nu ν π n \pi_{n} π n v ∈ R d v\in\mathbb{R}^{d} v ∈ R d ( X ~ i , n ) i ∈ N (\widetilde{X}_{i,n})_{i\in\mathbb{N}} ( X i , n ) i ∈ N ν \nu ν π n \pi_{n} π n
∣ Corr ( v T X ~ i , n , v T X ~ i + 1 , n ) ∣ ≤ 2 ∫ R d ∫ R d ∣ v T x − v T y ∣ 4 [ ν ( d y ) + π n ( d y ) ] π n ( d x ) Var X ∼ π n ( v T X ) 2 d H ( π n , ν ) . \left|\operatorname{Corr}(v^{T}\widetilde{X}_{i,n},v^{T}\widetilde{X}_{i+1,n})\right|\leq\sqrt{\frac{2\int_{\mathbb{R}^{d}}\int_{\mathbb{R}^{d}}\left|v^{T}x-v^{T}y\right|^{4}[\nu(\mathrm{d}y)+\pi_{n}(\mathrm{d}y)]\pi_{n}(\mathrm{d}x)}{\operatorname{Var}_{X\sim\pi_{n}}(v^{T}X)^{2}}}\,d_{\mathrm{H}}(\pi_{n},\nu). Corr ( v T X i , n , v T X i + 1 , n ) ≤ Var X ∼ π n ( v T X ) 2 2 ∫ R d ∫ R d ∣ v T x − v T y ∣ 4 [ ν ( d y ) + π n ( d y )] π n ( d x ) d H ( π n , ν ) . Laplace-based IMH. We now consider choosing the (multimodal) Laplace approximation as proposal distribution ν = L π n K \nu=\mathcal{L}_{\pi_{n}}^{K} ν = L π n K Theorem 3.2 , together with Lemma 4.2, we obtain the following result.
Theorem 4.3. Let the assumptions of Theorem 3.1 or Theorem 3.2 be satisfied. Then the average acceptance probability of Laplace-based IMH with proposal distribution L π n K \mathcal{L}_{\pi_{n}}^{K} L π n K
1 − E π n ⊗ L π n K [ α n ] ∈ O ( n − 1 / 2 ) . 1-\mathbb{E}_{\pi_{n}\otimes\mathcal{L}_{\pi_{n}}^{K}}[\alpha_{n}]\in\mathcal{O}(n^{-1/2}). 1 − E π n ⊗ L π n K [ α n ] ∈ O ( n − 1/2 ) . Moreover, if in addition to possesses a fourth moment and π n \pi_{n} π n x ∗ ( 1 ) , … , x ∗ ( K ) x_{*}^{(1)},\ldots,x_{*}^{(K)} x ∗ ( 1 ) , … , x ∗ ( K ) K ≥ 2 K\geq2 K ≥ 2 v ∈ R d v\in\mathbb{R}^{d} v ∈ R d lim n → ∞ Var X ∼ π n ( v T X ) > 0 \lim_{n\to\infty}\operatorname{Var}_{X\sim\pi_{n}}(v^{T}X)>0 lim n → ∞ Var X ∼ π n ( v T X ) > 0 ( X ~ i , n ) i ∈ N (\widetilde{X}_{i,n})_{i\in\mathbb{N}} ( X i , n ) i ∈ N L π n K \mathcal{L}_{\pi_{n}}^{K} L π n K π n \pi_{n} π n
∣ Corr ( v T X ~ i , n , v T X ~ i + 1 , n ) ∣ ∈ O ( n − 1 / 2 ) \left|\operatorname{Corr}(v^{T}\widetilde{X}_{i,n},v^{T}\widetilde{X}_{i+1,n})\right|\in\mathcal{O}(n^{-1/2}) Corr ( v T X i , n , v T X i + 1 , n ) ∈ O ( n − 1/2 ) Proof. The first statement is an immediate consequence of Lemma 4.2 and equation (2) , such that
1 − E π n ⊗ L π n K [ α n ] ≤ 2 d T V ( π n , L π n K ) ≤ 2 d H ( π n , L π n K ) , 1-\mathbb{E}_{\pi_{n}\otimes\mathcal{L}_{\pi_{n}}^{K}}[\alpha_{n}]\leq2d_{\mathrm{TV}}(\pi_{n},\mathcal{L}_{\pi_{n}}^{K})\leq2d_{\mathrm{H}}(\pi_{n},\mathcal{L}_{\pi_{n}}^{K}), 1 − E π n ⊗ L π n K [ α n ] ≤ 2 d TV ( π n , L π n K ) ≤ 2 d H ( π n , L π n K ) , which, combined with the statements of Theorem 3.1 or Theorem 3.2 , respectively, verify the claim. The second statement follows analogously by Theorem 3.1 or Theorem 3.2 , respectively, and noting that the finite fourth moment of π 0 \pi_0 π 0
∫ R d ∫ R d ∣ v T x − v T y ∣ 4 [ K π n K ( d y ) + π n ( d y ) ] π n ( d x ) < ∞ . \int_{\mathbb{R}^d}\int_{\mathbb{R}^d} |v^T x-v^T y|^4 \left[\mathcal{K}_{\pi_n}^K(\mathrm{d}y)+\pi_n(\mathrm{d}y)\right]\pi_n(\mathrm{d}x)<\infty. ∫ R d ∫ R d ∣ v T x − v T y ∣ 4 [ K π n K ( d y ) + π n ( d y ) ] π n ( d x ) < ∞. Together with the posterior concentration assumption and either Assumption 2 or Assumption 3, this further implies that
∫ R d ∫ R d ∣ v T x − v T y ∣ 4 [ K π n K ( d y ) + π n ( d y ) ] π n ( d x ) ∈ O ( 1 ) . \int_{\mathbb{R}^d}\int_{\mathbb{R}^d} |v^T x-v^T y|^4 \left[\mathcal{K}_{\pi_n}^K(\mathrm{d}y)+\pi_n(\mathrm{d}y)\right]\pi_n(\mathrm{d}x)\in\mathcal{O}(1). ∫ R d ∫ R d ∣ v T x − v T y ∣ 4 [ K π n K ( d y ) + π n ( d y ) ] π n ( d x ) ∈ O ( 1 ) . Since lim n → ∞ Var X ∼ π n ( v T X ) > 0 \lim_{n\to\infty}\operatorname{Var}_{X\sim\pi_n}(v^T X)>0 lim n → ∞ Var X ∼ π n ( v T X ) > 0 n → ∞ n\to\infty n → ∞ O ( n − 1 / 2 ) \mathcal{O}(n^{-1/2}) O ( n − 1/2 )
Remark 4.3. The assumption in Theorem 4.3 about the fourth moment of π 0 \pi_0 π 0 ∫ R d ∫ R d ∣ v T x − v T y ∣ 4 [ K π n K ( d y ) + π n ( d y ) ] π n ( d x ) \int_{\mathbb{R}^d}\int_{\mathbb{R}^d} |v^T x-v^T y|^4 [\mathcal{K}_{\pi_n}^K(\mathrm{d}y)+\pi_n(\mathrm{d}y)]\pi_n(\mathrm{d}x) ∫ R d ∫ R d ∣ v T x − v T y ∣ 4 [ K π n K ( d y ) + π n ( d y )] π n ( d x ) π n \pi_n π n lim n → ∞ Var X ∼ π n ( v T X ) = 0 \lim_{n\to\infty}\operatorname{Var}_{X\sim\pi_n}(v^T X)=0 lim n → ∞ Var X ∼ π n ( v T X ) = 0 Var X ∼ π n ( v T X ) \operatorname{Var}_{X\sim\pi_n}(v^T X) Var X ∼ π n ( v T X ) O ( n − 1 / 2 ) \mathcal{O}(n^{-1/2}) O ( n − 1/2 ) ∫ R d ∫ R d ∣ v T x − v T y ∣ 4 [ K π n K ( d y ) + π n ( d y ) ] π n ( d x ) \int_{\mathbb{R}^d}\int_{\mathbb{R}^d} |v^T x-v^T y|^4 [\mathcal{K}_{\pi_n}^K(\mathrm{d}y)+\pi_n(\mathrm{d}y)]\pi_n(\mathrm{d}x) ∫ R d ∫ R d ∣ v T x − v T y ∣ 4 [ K π n K ( d y ) + π n ( d y )] π n ( d x ) O ( n − 1 / 2 ) \mathcal{O}(n^{-1/2}) O ( n − 1/2 ) [23 ] , § 4.2 for more details in the unimodal case.
Next, we establish an even stronger statement about the spectral gap of the Laplace-based IMH given the setting of Corollary 3.2. Its proof relies also to some extent of the arguments delivered in Section A.5.
Theorem 4.4. Let the assumptions of Corollary 3.2 be satisfied with δ > 1 / 2 \delta>1/2 δ > 1/2 ν n = K π n K \nu_n=\mathcal{K}_{\pi_n}^K ν n = K π n K M n M_n M n
1 − gap π n ( M n ) ∈ O ( n 1 − 3 q ) . 1-\operatorname{gap}_{\pi_n}(M_n)\in\mathcal{O}(n^{1-3q}). 1 − gap π n ( M n ) ∈ O ( n 1 − 3 q ) . Proof. We provide an upper bound of ∥ w n ∥ ∞ \|w_n\|_\infty ∥ w n ∥ ∞ w n ( x ) = Z n K Z n w n ∗ ( x ) w_n(x)=\frac{Z_n^K}{Z_n}\,w_n^*(x) w n ( x ) = Z n Z n K w n ∗ ( x ) (43) holds Z n K Z n ≤ ( 1 − κ n , 1 2 ) − 1 \frac{Z_n^K}{Z_n}\le\left(1-\frac{\kappa_{n,1}}{2}\right)^{-1} Z n Z n K ≤ ( 1 − 2 κ n , 1 ) − 1 κ n , 1 \kappa_{n,1} κ n , 1 w n ∗ w_n^* w n ∗ x ∈ ⋃ k = 1 K V n k , R , q x\in\bigcup_{k=1}^K V_n^{k,R,q} x ∈ ⋃ k = 1 K V n k , R , q
w n ∗ ( x ) ≤ e n 1 − 3 q . w_n^*(x)\le e^{n^{1-3q}}. w n ∗ ( x ) ≤ e n 1 − 3 q . In contrast to what we relied on in the proof of Corollary 3.2, for x ∈ V ‾ n R , q x\in\overline{V}_n^{R,q} x ∈ V n R , q
w n ∗ ( x ) ≤ ∑ k = 1 K π n ( k ) exp ( − n δ ∥ x − x n ( k ) ∥ H n ( k ) 2 ) ∑ j = 1 K π n ( j ) exp ( − n 2 ∥ x − x n ( j ) ∥ H n ( j ) 2 ) ≤ ∑ k = 1 K e − n ( δ − 1 2 ) ∥ x − x n ( k ) ∥ H n ( k ) 2 ≤ K e − ( δ − 1 2 ) R n 1 − 2 q , w_n^*(x)\le\sum_{k=1}^K \pi_n^{(k)}\frac{\exp\left(-n\delta\|x-x_n^{(k)}\|_{H_n^{(k)}}^2\right)}{\sum_{j=1}^K\pi_n^{(j)}\exp\left(-\frac{n}{2}\|x-x_n^{(j)}\|_{H_n^{(j)}}^2\right)}
\le\sum_{k=1}^K e^{-n(\delta-\frac{1}{2})\|x-x_n^{(k)}\|_{H_n^{(k)}}^2}\le Ke^{-(\delta-\frac{1}{2})Rn^{1-2q},} w n ∗ ( x ) ≤ k = 1 ∑ K π n ( k ) ∑ j = 1 K π n ( j ) exp ( − 2 n ∥ x − x n ( j ) ∥ H n ( j ) 2 ) exp ( − n δ ∥ x − x n ( k ) ∥ H n ( k ) 2 ) ≤ k = 1 ∑ K e − n ( δ − 2 1 ) ∥ x − x n ( k ) ∥ H n ( k ) 2 ≤ K e − ( δ − 2 1 ) R n 1 − 2 q , such that
sup x ∈ R d w n ∗ ( x ) ≤ max { e n 1 − 3 q , K e − ( δ − 1 2 ) R n 1 − 2 q } . \sup_{x\in\mathbb{R}^d}w_n^*(x)\le\max\left\{e^{n^{1-3q}},Ke^{-(\delta-\frac{1}{2})Rn^{1-2q}}\right\}. x ∈ R d sup w n ∗ ( x ) ≤ max { e n 1 − 3 q , K e − ( δ − 2 1 ) R n 1 − 2 q } . By q ∈ [ 1 / 3 , 1 / 2 ) q \in[1/3,1/2) q ∈ [ 1/3 , 1/2 ) c > 0 c > 0 c > 0 n ∈ N n \in\mathbb{N} n ∈ N ∥ w n ∗ ∥ ∞ ≤ e c n 1 − 3 q \lVert w_n^* \rVert_\infty\le e^{cn^{1-3q}} ∥ w n ∗ ∥ ∞ ≤ e c n 1 − 3 q ∥ w n ∥ ∞ ≤ ( 1 − K n , 1 2 ) − 1 e c n 1 − 3 q \lVert w_n \rVert_\infty\le\left(1 - \frac{\mathcal{K}_{n,1}}{2}\right)^{-1} e^{cn^{1-3q}} ∥ w n ∥ ∞ ≤ ( 1 − 2 K n , 1 ) − 1 e c n 1 − 3 q (21) , also for sufficiently large n ∈ N n \in\mathbb{N} n ∈ N
1 − gap π n ( M n ) ≤ 1 − 1 ∥ w n ∥ ∞ ≤ 1 − ( 1 − K n , 1 2 ) e − c n 1 − 3 q ≤ 1 − e − c n 1 − 3 q + K n , 1 2 e − c n 1 − 3 q 1 - \operatorname{gap}_{\pi_n}(M_n) \le1 - \frac{1}{\lVert w_n \rVert_\infty} \le1 - \left(1 - \frac{\mathcal{K}_{n,1}}{2}\right)e^{-cn^{1-3q}} \le1 - e^{-cn^{1-3q}} + \frac{\mathcal{K}_{n,1}}{2}e^{-cn^{1-3q}} 1 − gap π n ( M n ) ≤ 1 − ∥ w n ∥ ∞ 1 ≤ 1 − ( 1 − 2 K n , 1 ) e − c n 1 − 3 q ≤ 1 − e − c n 1 − 3 q + 2 K n , 1 e − c n 1 − 3 q ≤ 1 − e − c n 1 − 3 q + K n , 1 ≤ c n 1 − 3 q + K n , 1 , \le1 - e^{-cn^{1-3q}} + \mathcal{K}_{n,1} \le cn^{1-3q} + \mathcal{K}_{n,1}, ≤ 1 − e − c n 1 − 3 q + K n , 1 ≤ c n 1 − 3 q + K n , 1 , where in the last equality we applied 1 − e − x ≤ x 1 - e^{-x} \le x 1 − e − x ≤ x x ∈ R x \in\mathbb{R} x ∈ R K n , 1 ∈ O ( n − 1 / 2 ) \mathcal{K}_{n,1} \in O(n^{-1/2}) K n , 1 ∈ O ( n − 1/2 ) q ∈ [ 1 / 3 , 1 / 2 ) q \in[1/3,1/2) q ∈ [ 1/3 , 1/2 ) c n 1 − 3 q + K n , 1 ∈ O ( n 1 − 3 q ) cn^{1-3q} + \mathcal{K}_{n,1} \in O(n^{1-3q}) c n 1 − 3 q + K n , 1 ∈ O ( n 1 − 3 q )
Theorem 4.4 shows that the spectral gap of the Markov operator M n M_n M n n → ∞ n \to\infty n → ∞ 1 − 3 q < 0 1 - 3q < 0 1 − 3 q < 0 q ∈ ( 1 / 3 , 1 / 2 ) q \in(1/3,1/2) q ∈ ( 1/3 , 1/2 ) K π n K \mathcal{K}_{\pi_n}^K K π n K π n \pi_n π n δ > 1 / 2 \delta> 1/2 δ > 1/2 [5 ] establishes polynomial convergence rates in total variation under suitable moment assumptions. Such behavior, however, is not analyzed in the present work.
Numerical Illustrations
We now illustrate our theoretical findings on the Laplace-based importance sampling and Laplace-based independent Metropolis–Hastings algorithm at two examples. The first one is a rather academic test example from [24 ] . The second example is concerned with sampling the posterior in Bayesian neural networks where multimodal posteriors occur naturally by the symmetric architecture of the network.
Simple Example
We consider a Gaussian prior π 0 = N ( 0 , I d ) \pi_0 = \mathcal{N}(0,I_d) π 0 = N ( 0 , I d ) R d \mathbb{R}^d R d d ∈ { 1 , 2 , … , 7 } d \in\{1,2,\ldots,7\} d ∈ { 1 , 2 , … , 7 }
Y = G ( X ) + η n Y = \mathcal{G}(X) + \eta_n Y = G ( X ) + η n where X ∼ π 0 X \sim\pi_0 X ∼ π 0 η n \eta_n η n G ( x ) = ( G 1 ( x ) , … , G d ( x ) ) T \mathcal{G}(x) = (\mathcal{G}_1(x),\ldots,\mathcal{G}_d(x))^\mathsf{T} G ( x ) = ( G 1 ( x ) , … , G d ( x ) ) T
G 1 ( x ) = x 1 2 + x 1 , G 2 ( x ) = 2 x 2 + x 1 2 , G 3 ( x ) = exp ( x 3 ) + x 2 , G 4 ( x ) = x 4 + 2 x 3 , \mathcal{G}_1(x) = x_1^2 + x_1,\quad\mathcal{G}_2(x) = 2x_2 + x_1^2,\quad\mathcal{G}_3(x) = \exp(x_3) + x_2,\quad\mathcal{G}_4(x) = x_4 + 2x_3, G 1 ( x ) = x 1 2 + x 1 , G 2 ( x ) = 2 x 2 + x 1 2 , G 3 ( x ) = exp ( x 3 ) + x 2 , G 4 ( x ) = x 4 + 2 x 3 , G 5 ( x ) = x 5 2 + x 4 , G 6 ( x ) = x 6 + x 5 , G 7 ( x ) = x 7 + 2 x 6 \mathcal{G}_5(x) = x_5^2 + x_4,\quad\mathcal{G}_6(x) = x_6 + x_5,\quad\mathcal{G}_7(x) = x_7 + 2x_6 G 5 ( x ) = x 5 2 + x 4 , G 6 ( x ) = x 6 + x 5 , G 7 ( x ) = x 7 + 2 x 6 for x = ( x 1 , … , x d ) T ∈ R d x = (x_1,\ldots,x_d)^\mathsf{T} \in\mathbb{R}^d x = ( x 1 , … , x d ) T ∈ R d π n \pi_n π n (1)
Small noise limit: a single observation on increasing precision is given y n = 2 d + η n y_n = 2_d + \eta_n y n = 2 d + η n 2 d : = ( 2 , … , 2 ) ∈ R d 2_d := (2,\ldots,2) \in\mathbb{R}^d 2 d := ( 2 , … , 2 ) ∈ R d η n ∼ N ( 0 , 0.1 n − 1 I d ) \eta_n \sim\mathcal{N}(0,0.1\,n^{-1}I_d) η n ∼ N ( 0 , 0.1 n − 1 I d ) π n \pi_n π n (1) with
Φ n ( x ) : = 10 2 ∥ y n − G ( x ) ∥ 2 \Phi_n(x) := \frac{10}{2}\lVert y_n - \mathcal{G}(x)\rVert^2 Φ n ( x ) := 2 10 ∥ y n − G ( x ) ∥ 2 Large data limit: an increasing number of observations of same precision are given, i.e., y i = 2 d + η i y_i = \mathbf{2}_d + \eta_i y i = 2 d + η i i = 1 , … , n i = 1,\ldots,n i = 1 , … , n η i ∼ N ( 0 , 0.1 I d ) \eta_i \sim N(0,0.1I_d) η i ∼ N ( 0 , 0.1 I d ) π n \pi_n π n (1) with
Φ n ( x ) : = 10 2 n ∑ i = 1 n ∥ y i − G ( x ) ∥ 2 . \Phi_n(x) := \frac{10}{2n}\sum_{i=1}^{n}\lVert y_i-\mathcal{G}(x)\rVert^2. Φ n ( x ) := 2 n 10 i = 1 ∑ n ∥ y i − G ( x ) ∥ 2 . In each case the resulting posteriors are multimodal: for d ∈ { 1 , … , 4 } d \in\{1,\ldots,4\} d ∈ { 1 , … , 4 } K = 2 K=2 K = 2 d ∈ { 5 , 6 , 7 } d \in\{5,6,7\} d ∈ { 5 , 6 , 7 } K = 4 K=4 K = 4 π n \pi_n π n
Laplace-based importance sampling: We use N = 10 3 N=10^3 N = 1 0 3 ν n = L π n K \nu_n=L_{\pi_n}^{K} ν n = L π n K Figure 1 illustrate the asymptotic behavior of the effective samples size for increasing n n n d d d 1 − E S S N 1-\frac{\mathrm{ESS}}{N} 1 − N ESS n − 1 n^{-1} n − 1 1 − E S S N 1-\frac{\mathrm{ESS}}{N} 1 − N ESS n − 1 n^{-1} n − 1 Figure 1 a dependence of the multiplicative constant on the dimension, i.e., the effective sample size is smaller in the case of d ∈ { 3 , 4 } d \in\{3,4\} d ∈ { 3 , 4 } d ∈ { 1 , 2 } d \in\{1,2\} d ∈ { 1 , 2 } G \mathcal{G} G
Figure 1. Performance of Laplace-based importance sampling for the simple example considered in Section 5.1.
For comparison, we also plot ( E S S − 1 ) / N (\mathrm{ESS}-1)/N ( ESS − 1 ) / N N = 10 5 N=10^5 N = 1 0 5 Figure 3 . The results show that ( E S S − 1 ) / N (\mathrm{ESS}-1)/N ( ESS − 1 ) / N n n n n → ∞ n \to\infty n → ∞ n → ∞ n \to\infty n → ∞
Figure 3. Convergence of ( E S S − 1 ) / N (\mathrm{ESS}-1)/N ( ESS − 1 ) / N 2 D 2D 2 D
Laplace-based IMH: We only consider the small noise limit here and examine how the average acceptance probability and the lag-one autocorrelation behave as n n n 10 6 10^6 1 0 6 Figure 2 , the average acceptance probability converges to one with a rate of n − 1 / 2 n^{-1/2} n − 1/2 10 6 10^6 1 0 6 Figure 2 and observe that it decays to zero at an approximate rate of n − 1 / 2 n^{-1/2} n − 1/2 d d d G \mathcal{G} G N N N σ ~ n 2 ( f ) \widetilde{\sigma}_n^2(f) σ n 2 ( f ) Figure 4 we observe that 1 − E S S / N 1-\mathrm{ESS}/N 1 − ESS / N n − 1 / 2 n^{-1/2} n − 1/2 1 − E S S / N ≥ 1 − gap π n ( M n ) 2 ∥ f ∥ 2 , π n 1-\mathrm{ESS}/N \ge1-\frac{\operatorname{gap}_{\pi_n}(M_n)}{2\|f\|_{2,\pi_n}} 1 − ESS / N ≥ 1 − 2∥ f ∥ 2 , π n gap π n ( M n ) Theorem 4.4 although its assumptions are not satisfied here.
Figure 2. (a): Convergence of the deviation of the average acceptance probability from 1; (b): Convergence of the absolute value of lag-one autocorrelation
Figure 4. Convergence of 1 − E S S / N 1-\mathrm{ESS}/N 1 − ESS / N
Bayesian neural network
Neural networks are concatenations of simple functions consisting of affine layers and non-linear activation functions:
F ( ξ ) = F L ∘ ⋯ ∘ F 1 ( ξ ) , ξ ∈ R N F(\xi)=F_L\circ\cdots\circ F_1(\xi),\qquad\xi\in\mathbb{R}^{N} F ( ξ ) = F L ∘ ⋯ ∘ F 1 ( ξ ) , ξ ∈ R N where each F k : R N k − 1 → R N k F_k:\mathbb{R}^{N_{k-1}}\to\mathbb{R}^{N_k} F k : R N k − 1 → R N k
F ( ξ ) = ϕ ( W k + b k ) , W k ∈ R N k × N k − 1 , b k ∈ R N k , F(\xi)=\phi(W_k+b_k),\qquad W_k\in\mathbb{R}^{N_k\times N_{k-1}},\qquad b_k\in\mathbb{R}^{N_k}, F ( ξ ) = ϕ ( W k + b k ) , W k ∈ R N k × N k − 1 , b k ∈ R N k , with chosen activation function ϕ : R → R \phi:\mathbb{R}\to\mathbb{R} ϕ : R → R N k N_k N k k k k b k b_k b k biases and the entries in the matrices W k W_k W k weights . In machine learning these biases and weights are the parameters to be trained given observational data of the unknown function to be approximated by the neural network.
In order to capture uncertainty in the model, i.e., uncertainty about the learned biases and weights, a classical approach is to consider Bayesian neural networks where prior distributions for weights and biases are conditioned on observational data and the resulting posterior quantifies the limited knowledge about the neural network.
For simplicity, we consider a Bayesian neural network F : R → R F:\mathbb{R}\to\mathbb{R} F : R → R L = 2 L=2 L = 2 N 0 = 1 N_0=1 N 0 = 1 N 1 = 2 N_1=2 N 1 = 2 N 2 = 1 N_2=1 N 2 = 1 Figure 5 a, of the form
Figure 5. (a): Bayesian Neural Network with W 1 = [ w 1 ( 1 ) w 1 ( 2 ) ] W_1 = \left[w_1^{(1)}\quad w_1^{(2)}\right] W 1 = [ w 1 ( 1 ) w 1 ( 2 ) ] b 1 = ( b 1 ( 1 ) , b 1 ( 2 ) ) b_1 = \left(b_1^{(1)}, b_1^{(2)}\right) b 1 = ( b 1 ( 1 ) , b 1 ( 2 ) ) W 2 = [ w 2 ( 1 ) w 2 ( 2 ) ] W_2 = \begin{bmatrix} w_2^{(1)} \\ w_2^{(2)} \end{bmatrix} W 2 = [ w 2 ( 1 ) w 2 ( 2 ) ] b 1 = b 2 ( 1 ) b_1 = b_2^{(1)} b 1 = b 2 ( 1 )
F ( ξ ) = F 2 ( ϕ ( F 1 ( ξ ) ) ) , F(\xi)=F_2\left(\phi\left(F_1(\xi)\right)\right), F ( ξ ) = F 2 ( ϕ ( F 1 ( ξ ) ) ) , where we choose as activation function the tanh function ϕ ( t ) = e t − e − t e t + e − t \phi(t)=\frac{e^t-e^{-t}}{e^t+e^{-t}} ϕ ( t ) = e t + e − t e t − e − t
x = ( w 1 ( 1 ) , w 1 ( 2 ) , b 1 ( 1 ) , b 1 ( 2 ) , w 2 ( 1 ) , w 2 ( 2 ) , b 2 ( 1 ) ) T ∈ R 7 . x=\left(w_1^{(1)},w_1^{(2)},b_1^{(1)},b_1^{(2)},w_2^{(1)},w_2^{(2)},b_2^{(1)}\right)^{\mathsf{T}}\in\mathbb{R}^{7}. x = ( w 1 ( 1 ) , w 1 ( 2 ) , b 1 ( 1 ) , b 1 ( 2 ) , w 2 ( 1 ) , w 2 ( 2 ) , b 2 ( 1 ) ) T ∈ R 7 . The neural network, thus, becomes a parametrized mapping F ( ⋅ ; x ) : R → R F(\cdot;x):\mathbb{R}\to\mathbb{R} F ( ⋅ ; x ) : R → R π 0 = N ( 0 , 20 ⋅ I d ) \pi_0=\mathcal{N}(0,20\cdot I_d) π 0 = N ( 0 , 20 ⋅ I d ) x ∈ R d x\in\mathbb{R}^d x ∈ R d
y n = G ( X ) + η n , G ( x ) = ( F ( ξ 1 ; x ) ⋯ F ( ξ m ; x ) ) T , η n ∼ N ( 0 , n − 1 σ 2 I m ) y_n=\mathcal{G}(X)+\eta_n,\qquad\mathcal{G}(x)=\left(F(\xi_1;x)\ \cdots\ F(\xi_m;x)\right)^{\mathsf{T}},\qquad\eta_n\sim\mathcal{N}\left(0,n^{-1}\sigma^2 I_m\right) y n = G ( X ) + η n , G ( x ) = ( F ( ξ 1 ; x ) ⋯ F ( ξ m ; x ) ) T , η n ∼ N ( 0 , n − 1 σ 2 I m ) at m = 100 m=100 m = 100 ξ 1 , … , ξ m ∈ [ − 5 , 5 ] \xi_1,\ldots,\xi_m\in[-5,5] ξ 1 , … , ξ m ∈ [ − 5 , 5 ] σ 2 = 10 \sigma^2=10 σ 2 = 10 X ∼ π 0 X\sim\pi_0 X ∼ π 0 η n \eta_n η n π n \pi_n π n x x x (1) with
Φ n : = 1 20 ∥ y n − G ( x ) ∥ 2 . \Phi_n:=\frac{1}{20}\lVert y_n-\mathcal{G}(x)\rVert^2. Φ n := 20 1 ∥ y n − G ( x ) ∥ 2 . In the experiment we generated the observed data by y n = G ( x t r u e ) + η n y_n=\mathcal{G}(x_{\mathrm{true}})+\eta_n y n = G ( x true ) + η n
x t r u e = ( 1 , − 2 , 3 , 3 , 2 , 2 , 1 ) T . x_{\mathrm{true}}=(1,-2,3,3,2,2,1)^{\mathsf{T}}. x true = ( 1 , − 2 , 3 , 3 , 2 , 2 , 1 ) T . We then consider three cases derived from fixing some of the parameters of the network to the true values and inferring only the d ∈ { 2 , 4 , 7 } d\in\{2,4,7\} d ∈ { 2 , 4 , 7 }
4-dimensional case: we infer only all the weights, i.e., w 1 ( 1 ) w_1^{(1)} w 1 ( 1 ) w 1 ( 2 ) w_1^{(2)} w 1 ( 2 ) w 2 ( 1 ) w_2^{(1)} w 2 ( 1 ) w 2 ( 2 ) w_2^{(2)} w 2 ( 2 )
Again, we apply Laplace-based importance sampling and Laplace-based IMH to sample from the multimodal posterior π n \pi_n π n n n n d ∈ { 2 , 4 } d \in\{2,4\} d ∈ { 2 , 4 } K = 2 K = 2 K = 2 d = 7 d = 7 d = 7 K = 8 K = 8 K = 8
Results for Laplace-based importance sampling: We use N = 10 3 N = 10^3 N = 1 0 3 n n n Figure 5 b the effective sample size converges to the total sample size N N N n − 1 n^{-1} n − 1 d d d G \mathcal{G} G d = 2 d = 2 d = 2 d = 4 d = 4 d = 4 G \mathcal{G} G d d d d d d ( E S S − 1 ) / N (\mathrm{ESS}-1)/N ( ESS − 1 ) / N N = 10 5 N = 10^5 N = 1 0 5 n n n ( E S S − 1 ) / N (\mathrm{ESS}-1)/N ( ESS − 1 ) / N n → ∞ n \to\infty n → ∞ Figure 7 .
Figure 7. Convergence of ( 1 − E S S ) / N (1-\mathrm{ESS})/N ( 1 − ESS ) / N
Results for Laplace-based-IMH: As before we run a single Markov chain with 10 6 10^6 1 0 6 Figure 6 , the average acceptance probability converges again to one and the lag-one autocorrelation again to zero each with rate n − 1 / 2 n^{-1/2} n − 1/2 Figure 8 shows that the effective sample size converges to the total number of iterations at rate n − 1 / 2 n^{-1/2} n − 1/2
Figure 6. Laplace-based independent Metropolis–Hastings algorithm: (a): Convergence of the deviation of the average acceptance probability from 1. (b): Convergence of the absolute value of lag-one autocorrelation.
Figure 8. Convergence of 1 − E S S / N 1-\mathrm{ESS}/N 1 − ESS / N
Conclusion
In this work we analysed a multi-modal Laplace approximation L π n K \mathcal{L}_{\pi_n}^{K} L π n K I n I_n I n π n \pi_n π n π n \pi_n π n L π n K \mathcal{L}_{\pi_n}^{K} L π n K n − 1 / 2 n^{-1/2} n − 1/2 d d d χ 2 \chi^2 χ 2 n − 1 n^{-1} n − 1 n − d / 2 n^{-d/2} n − d /2 n − 1 / 2 n^{-1/2} n − 1/2
Several questions remain open. For instance, our results on the effective sample size require all local minima of I n I_n I n Φ \Phi Φ δ > 1 / 2 \delta> 1/2 δ > 1/2
Proof of Hellinger convergence
Before proceeding to the proof of the main results, we state useful lemmas that allow us to keep as good as possible track on the dimensional dependence of our estimates
Preliminary Lemmas
This first lemma gives an explicit control on appearing moments of the standard multivariate normal distribution in R d \mathbb{R}^d R d
Lemma A.1. With Γ \Gamma Γ n ∈ N n \in\mathbb{N} n ∈ N p ∈ { 1 , 2 } p \in\{1,2\} p ∈ { 1 , 2 }
∫ R d ∥ x ∥ 3 p e − n 2 ∥ x ∥ 2 d x = 2 3 p 2 ( 2 π ) d 2 Γ ( 3 p + d 2 ) n 3 p + d 2 Γ ( d 2 ) ≤ 2 3 p − 2 ( 2 π n ) d 2 ( d n ) 3 p 2 . \int_{\mathbb{R}^d} \lVert x\rVert^{3p} e^{-\frac{n}{2}\lVert x\rVert^2}\,\mathrm{d}x = \frac{2^{\frac{3p}{2}}(2\pi)^{\frac{d}{2}}\Gamma\left(\frac{3p+d}{2}\right)}{n^{\frac{3p+d}{2}}\Gamma\left(\frac{d}{2}\right)} \leq2^{3p-2}\left(\frac{2\pi}{n}\right)^{\frac{d}{2}}\left(\frac{d}{n}\right)^{\frac{3p}{2}}. ∫ R d ∥ x ∥ 3 p e − 2 n ∥ x ∥ 2 d x = n 2 3 p + d Γ ( 2 d ) 2 2 3 p ( 2 π ) 2 d Γ ( 2 3 p + d ) ≤ 2 3 p − 2 ( n 2 π ) 2 d ( n d ) 2 3 p . Proof. The first equality is the ( 3 p ) (3p) ( 3 p ) R d \mathbb{R}^d R d p = 2 p=2 p = 2 Γ ( z + 1 ) = z ⋅ Γ ( z ) \Gamma(z+1)=z\cdot\Gamma(z) Γ ( z + 1 ) = z ⋅ Γ ( z ) z > 0 z>0 z > 0
Γ ( d 2 + 3 ) Γ ( d 2 ) = d ( d + 2 ) ( d + 4 ) 8 ≤ d 3 8 ( 1 + 14 d ) ≤ 2 d 3 , \frac{\Gamma\left(\frac{d}{2}+3\right)}{\Gamma\left(\frac{d}{2}\right)}=\frac{d(d+2)(d+4)}{8}\leq\frac{d^3}{8}\left(1+\frac{14}{d}\right)\leq2d^3, Γ ( 2 d ) Γ ( 2 d + 3 ) = 8 d ( d + 2 ) ( d + 4 ) ≤ 8 d 3 ( 1 + d 14 ) ≤ 2 d 3 , such that the inequality holds in that case. Secondly, for p = 1 p=1 p = 1 Γ ( z + 1 / 2 ) ≤ z ⋅ Γ ( z ) \Gamma(z+1/2)\leq\sqrt{z}\cdot\Gamma(z) Γ ( z + 1/2 ) ≤ z ⋅ Γ ( z ) z > 0 z>0 z > 0 [16 Lemma 6] ), we have
Γ ( d 2 + 3 2 ) Γ ( d 2 ) = ( d 2 + 1 2 ) Γ ( d 2 + 1 2 ) Γ ( d 2 ) ≤ ( d + 1 ) d 2 2 ≤ d 3 / 2 2 , \frac{\Gamma\left(\frac{d}{2}+\frac{3}{2}\right)}{\Gamma\left(\frac{d}{2}\right)}=\frac{\left(\frac{d}{2}+\frac{1}{2}\right)\Gamma\left(\frac{d}{2}+\frac{1}{2}\right)}{\Gamma\left(\frac{d}{2}\right)}\leq\frac{(d+1)\sqrt{d}}{2\sqrt{2}}\leq\frac{d^{3/2}}{\sqrt{2}}, Γ ( 2 d ) Γ ( 2 d + 2 3 ) = Γ ( 2 d ) ( 2 d + 2 1 ) Γ ( 2 d + 2 1 ) ≤ 2 2 ( d + 1 ) d ≤ 2 d 3/2 , such that the desired bound follows.
The next lemma gives an upper bound on Gaussian tails in arbitrary dimension.
Lemma A.2. Let Σ ∈ R d × d \Sigma\in\mathbb{R}^{d \times d} Σ ∈ R d × d z ∈ R d z \in\mathbb{R}^{d} z ∈ R d r , c > 0 r,c>0 r , c > 0
∫ ∥ x − z ∥ Σ 2 ≥ r e − n c ∥ x − z ∥ Σ 2 d x ≤ e − n c r 2 det ( Σ − 1 / 2 ) ( 2 π n c ) d 2 . \int_{\lVert x-z\rVert_{\Sigma}^{2} \ge r} e^{-nc\lVert x-z\rVert_{\Sigma}^{2}}\,\mathrm{d}x \le e^{-\frac{ncr}{2}} \det\left(\Sigma^{-1/2}\right) \left(\frac{2\pi}{nc}\right)^{\frac{d}{2}}. ∫ ∥ x − z ∥ Σ 2 ≥ r e − n c ∥ x − z ∥ Σ 2 d x ≤ e − 2 n cr det ( Σ − 1/2 ) ( n c 2 π ) 2 d . Proof. A change of variable x ↦ Σ 1 / 2 ( x − z ) x \mapsto\Sigma^{1/2}(x-z) x ↦ Σ 1/2 ( x − z ) Σ − 1 / 2 \Sigma^{-1/2} Σ − 1/2
∫ ∥ x − z ∥ Σ 2 ≥ r e − n c ∥ x − z ∥ Σ 2 d x = ∫ ∥ y ∥ 2 ≥ r e − n c ∥ y ∥ 2 det ( Σ − 1 / 2 ) d z = det ( Σ − 1 / 2 ) ( π n c ) d 2 P ( ∥ Z ∥ 2 ≥ r ) , \int_{\lVert x-z\rVert_{\Sigma}^{2} \ge r} e^{-nc\lVert x-z\rVert_{\Sigma}^{2}}\,\mathrm{d}x = \int_{\lVert y\rVert^{2} \ge r} e^{-nc\lVert y\rVert^{2}} \det\left(\Sigma^{-1/2}\right)\,\mathrm{d}z = \det\left(\Sigma^{-1/2}\right)\left(\frac{\pi}{nc}\right)^{\frac{d}{2}}\mathbb{P}\left(\lVert Z\rVert^{2} \ge r\right), ∫ ∥ x − z ∥ Σ 2 ≥ r e − n c ∥ x − z ∥ Σ 2 d x = ∫ ∥ y ∥ 2 ≥ r e − n c ∥ y ∥ 2 det ( Σ − 1/2 ) d z = det ( Σ − 1/2 ) ( n c π ) 2 d P ( ∥ Z ∥ 2 ≥ r ) , where Z ∼ N ( 0 , ( 2 n c ) − 1 I d ) Z \sim\mathcal{N}(0,(2nc)^{-1}I_d) Z ∼ N ( 0 , ( 2 n c ) − 1 I d ) P ( ∥ Z ∥ 2 ≥ r ) ≤ E [ h ( ∥ Z ∥ 2 ) ] h ( r ) \mathbb{P}(\lVert Z\rVert^{2} \ge r) \le\frac{\mathbb{E}[h(\lVert Z\rVert^{2})]}{h(r)} P (∥ Z ∥ 2 ≥ r ) ≤ h ( r ) E [ h (∥ Z ∥ 2 )] h ( s ) = e n c s 2 h(s)=e^{\frac{ncs}{2}} h ( s ) = e 2 n cs E [ e n c 2 ∥ Z ∥ 2 ] = 2 d 2 \mathbb{E}\left[e^{\frac{nc}{2}\lVert Z\rVert^{2}}\right]=2^{\frac{d}{2}} E [ e 2 n c ∥ Z ∥ 2 ] = 2 2 d
Finally for the convenience of the reader we add a standard inequality without proof, which is a few times used in the following in different settings. We have
∣ ∑ j ∈ J a j ∣ p ≤ ∣ J ∣ p − 1 ∑ j ∈ J ∣ a j ∣ p (25) \left|\sum_{j\in J} a_j\right|^p \le|J|^{p-1}\sum_{j\in J}|a_j|^p \tag*{(25)} j ∈ J ∑ a j p ≤ ∣ J ∣ p − 1 j ∈ J ∑ ∣ a j ∣ p (25) for any p ≥ 1 p \ge1 p ≥ 1 J J J ∣ J ∣ |J| ∣ J ∣ ( a j ) j ∈ J ⊂ R (a_j)_{j\in J} \subset\mathbb{R} ( a j ) j ∈ J ⊂ R
Main Lemma for proving theo:MainpropHelling2 and theo:MainpropHelling
Lemma A.3. With the objects of the standing assumption, for any n ∈ N n \in\mathbb{N} n ∈ N p ∈ { 1 , 2 } p \in\{1,2\} p ∈ { 1 , 2 }
K n , p : = 2 Z ~ n K ∫ R d ∣ e − n p I n ( x ) − ( ∑ j = 1 K π ~ n ( j ) exp ( − n 2 ∥ x − x n ( j ) ∥ H ~ n ( j ) 2 ) ) 1 p ∣ p d x , \mathcal{K}_{n,p}:=\frac{2}{\widetilde{Z}_n^K}\int_{\mathbb{R}^{d}}\left|e^{-\frac{n}{p}I_n(x)}-\left(\sum_{j=1}^{K}\widetilde{\pi}_n^{(j)}\exp\left(-\frac{n}{2}\lVert x-x_n^{(j)}\rVert_{\widetilde{H}_n^{(j)}}^{2}\right)\right)^{\frac{1}{p}}\right|^p\mathrm{d}x, K n , p := Z n K 2 ∫ R d e − p n I n ( x ) − ( j = 1 ∑ K π n ( j ) exp ( − 2 n ∥ x − x n ( j ) ∥ H n ( j ) 2 ) ) p 1 p d x , where Z ~ n K \widetilde{Z}_n^K Z n K
(a) Let Assumption 1 and Assumption 2 with q = 0 q=0 q = 0 ρ , M ∈ ( 0 , ∞ ) \rho,M\in(0,\infty) ρ , M ∈ ( 0 , ∞ ) R ≤ ρ 6 16 M 2 R\le\frac{\rho^6}{16M^2} R ≤ 16 M 2 ρ 6 ε ∈ ( 0 , 1 ) \varepsilon\in(0,1) ε ∈ ( 0 , 1 ) p ∈ { 1 , 2 } p\in\{1,2\} p ∈ { 1 , 2 } n ∈ N n\in\mathbb{N} n ∈ N
K n , p ≤ 2 d 2 − 2 ( 2 11 2 d 3 2 M ρ 3 p ) p n − p 2 + O ( e − n 4 min { R , ε 2 } ) . (26) \mathcal{K}_{n,p}\le2^{\frac{d}{2}-2}\left(\frac{2^{\frac{11}{2}}d^{\frac{3}{2}}M}{\rho^3p}\right)^p n^{-\frac{p}{2}}+O\left(e^{-\frac{n}{4}\min\{R,\varepsilon^2\}}\right). \tag*{(26)} K n , p ≤ 2 2 d − 2 ( ρ 3 p 2 2 11 d 2 3 M ) p n − 2 p + O ( e − 4 n m i n { R , ε 2 } ) . (26) (b) Let Assumption 1 and Assumption 3 with q ∈ [ 1 3 , 1 2 ) q\in\left[\frac{1}{3},\frac{1}{2}\right) q ∈ [ 3 1 , 2 1 ) ρ , M , δ ∈ ( 0 , ∞ ) \rho,M,\delta\in(0,\infty) ρ , M , δ ∈ ( 0 , ∞ ) R = ρ 2 M 2 3 R=\frac{\rho^2}{M^{\frac{2}{3}}} R = M 3 2 ρ 2 p ∈ { 1 , 2 } p\in\{1,2\} p ∈ { 1 , 2 } n ∈ N n\in\mathbb{N} n ∈ N
K n , p ≤ e 4 ( 2 4 d 3 2 M ρ 3 p ) p n − p 2 + 2 p + d 2 ( K ( K − 1 ) p − 1 + 1 + ( 2 δ ) − d 2 ) exp ( − ρ 2 min { 1 , 2 δ } 4 M 2 3 n 1 − 2 q ) . (27) \mathcal{K}_{n,p}\le\frac{e}{4}\left(\frac{2^{4}d^{\frac{3}{2}}M}{\rho^3p}\right)^p n^{-\frac{p}{2}}+2^{p+\frac{d}{2}}\left(K(K-1)^{p-1}+1+(2\delta)^{-\frac{d}{2}}\right)\exp\left(-\frac{\rho^2\min\{1,2\delta\}}{4M^{\frac{2}{3}}}n^{1-2q}\right). \tag*{(27)} K n , p ≤ 4 e ( ρ 3 p 2 4 d 2 3 M ) p n − 2 p + 2 p + 2 d ( K ( K − 1 ) p − 1 + 1 + ( 2 δ ) − 2 d ) exp ( − 4 M 3 2 ρ 2 min { 1 , 2 δ } n 1 − 2 q ) . (27) Proof. Define
I n ( 0 ) : = 2 p − 1 ∫ V n R , q exp ( − n I n ( x ) ) d x + 2 p − 1 ∑ k = 1 K π n ( k ) ∫ V n R , q exp ( − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x (28) \mathcal{I}_{n}^{(0)} := 2^{p-1}\int_{V_{n}^{R,q}} \exp(-nI_n(x))\,\mathrm{d}x + 2^{p-1}\sum_{k=1}^{K}\pi_n^{(k)}\int_{V_{n}^{R,q}} \exp\left(-\frac{n}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2\right)\,\mathrm{d}x \tag*{(28)} I n ( 0 ) := 2 p − 1 ∫ V n R , q exp ( − n I n ( x )) d x + 2 p − 1 k = 1 ∑ K π n ( k ) ∫ V n R , q exp ( − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x (28) and note that by virtue of (25) we obtain
∫ V n R , q ∣ ( exp ( − n I n ( x ) ) ) 1 / p − ( ∑ j = 1 K π n ( j ) exp ( − n 2 ∥ x − x n ( j ) ∥ H n ( j ) 2 ) ) 1 / p ∣ p d x ≤ I n ( 0 ) . (29) \int_{V_n^{R,q}}\left|\left(\exp(-nI_n(x))\right)^{1/p}-\left(\sum_{j=1}^{K}\pi_n^{(j)}\exp\left(-\frac{n}{2}\lVert x-x_n^{(j)}\rVert_{H_n^{(j)}}^2\right)\right)^{1/p}\right|^p\mathrm{d}x\leq\mathcal{I}_n^{(0)}. \tag*{(29)} ∫ V n R , q ( exp ( − n I n ( x )) ) 1/ p − ( j = 1 ∑ K π n ( j ) exp ( − 2 n ∥ x − x n ( j ) ∥ H n ( j ) 2 ) ) 1/ p p d x ≤ I n ( 0 ) . (29) For k = 1 , … , K k=1,\ldots,K k = 1 , … , K
I n ( k ) : = ∫ V n k , R , q ∣ ( exp ( − n I n ( x ) ) ) 1 / p − ( ∑ j = 1 K π n ( j ) exp ( − n 2 ∥ x − x n ( j ) ∥ H n ( j ) 2 ) ) 1 / p ∣ p d x \mathcal{I}_n^{(k)}:=\int_{V_n^{k,R,q}}\left|\left(\exp(-nI_n(x))\right)^{1/p}-\left(\sum_{j=1}^{K}\pi_n^{(j)}\exp\left(-\frac{n}{2}\lVert x-x_n^{(j)}\rVert_{H_n^{(j)}}^2\right)\right)^{1/p}\right|^p\mathrm{d}x I n ( k ) := ∫ V n k , R , q ( exp ( − n I n ( x )) ) 1/ p − ( j = 1 ∑ K π n ( j ) exp ( − 2 n ∥ x − x n ( j ) ∥ H n ( j ) 2 ) ) 1/ p p d x such that, by decomposing R d \mathbb{R}^d R d (29) , we obtain K n , p ≤ 2 Z ~ n K ∑ k = 1 K I n ( k ) + 2 Z ~ n K I n ( 0 ) \mathcal{K}_{n,p}\leq\frac{2}{\widetilde{Z}_n^K}\sum_{k=1}^{K}\mathcal{I}_n^{(k)}+\frac{2}{\widetilde{Z}_n^K}\mathcal{I}_n^{(0)} K n , p ≤ Z n K 2 ∑ k = 1 K I n ( k ) + Z n K 2 I n ( 0 ) k = 1 , … , K k=1,\ldots,K k = 1 , … , K
I n ( k ) , 1 : = ∫ V n k , R , q ∣ ( exp ( − n I n ( x ) ) ) 1 / p − ( π n ( k ) exp ( − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 ) ) 1 / p ∣ p d x \mathcal{I}_n^{(k),1}:=\int_{V_n^{k,R,q}}\left|\left(\exp(-nI_n(x))\right)^{1/p}-\left(\pi_n^{(k)}\exp\left(-\frac{n}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2\right)\right)^{1/p}\right|^p\mathrm{d}x I n ( k ) , 1 := ∫ V n k , R , q ( exp ( − n I n ( x )) ) 1/ p − ( π n ( k ) exp ( − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 ) ) 1/ p p d x I n ( k ) , 2 : = ∫ V n k , R , q ∣ ( π n ( k ) exp ( − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 ) ) 1 / p − ( ∑ j = 1 K π n ( j ) exp ( − n 2 ∥ x − x n ( j ) ∥ H n ( j ) 2 ) ) 1 / p ∣ p d x \mathcal{I}_n^{(k),2}:=\int_{V_n^{k,R,q}}\left|\left(\pi_n^{(k)}\exp\left(-\frac{n}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2\right)\right)^{1/p}-\left(\sum_{j=1}^{K}\pi_n^{(j)}\exp\left(-\frac{n}{2}\lVert x-x_n^{(j)}\rVert_{H_n^{(j)}}^2\right)\right)^{1/p}\right|^p\mathrm{d}x I n ( k ) , 2 := ∫ V n k , R , q ( π n ( k ) exp ( − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 ) ) 1/ p − ( j = 1 ∑ K π n ( j ) exp ( − 2 n ∥ x − x n ( j ) ∥ H n ( j ) 2 ) ) 1/ p p d x applying (25) suitably yields
I n ( k ) ≤ 2 p − 1 ( I n ( k ) , 1 + I n ( k ) , 2 ) . (30) \mathcal{I}_n^{(k)}\leq2^{p-1}\left(\mathcal{I}_n^{(k),1}+\mathcal{I}_n^{(k),2}\right). \tag*{(30)} I n ( k ) ≤ 2 p − 1 ( I n ( k ) , 1 + I n ( k ) , 2 ) . (30) Furthermore
I n ( k ) , 1 = ∫ V n k , R , q ∣ ( exp ( − n I n ( x ) ) π n ( k ) exp ( − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 ) ) 1 / p − 1 ∣ p π n ( k ) exp ( − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x . (31) \mathcal{I}_n^{(k),1}=\int_{V_n^{k,R,q}}\left|\left(\frac{\exp(-nI_n(x))}{\pi_n^{(k)}\exp\left(-\frac{n}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2\right)}\right)^{1/p}-1\right|^p\pi_n^{(k)}\exp\left(-\frac{n}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2\right)\,\mathrm{d}x. \tag*{(31)} I n ( k ) , 1 = ∫ V n k , R , q π n ( k ) exp ( − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 ) exp ( − n I n ( x )) 1/ p − 1 p π n ( k ) exp ( − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x . (31) The second order Taylor expansion of I n I_n I n x ∈ V n k , R , q x\in V_n^{k,R,q} x ∈ V n k , R , q
I n ( x ) = I n ( x n ( k ) ) + ⟨ ∇ I n ( x n ( k ) ) , x − x n ( k ) ⟩ + 1 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 + R n ( k ) ( x ) , I_n(x)=I_n(x_n^{(k)})+\left\langle\nabla I_n(x_n^{(k)}),x-x_n^{(k)}\right\rangle+\frac{1}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2+R_n^{(k)}(x), I n ( x ) = I n ( x n ( k ) ) + ⟨ ∇ I n ( x n ( k ) ) , x − x n ( k ) ⟩ + 2 1 ∥ x − x n ( k ) ∥ H n ( k ) 2 + R n ( k ) ( x ) , with R n ( k ) ( x ) : = 1 6 D 3 I n ( ξ x ) ( x − x n ( k ) , x − x n ( k ) , x − x n ( k ) ) R_n^{(k)}(x):=\frac{1}{6}D^3I_n(\xi_x)(x-x_n^{(k)},x-x_n^{(k)},x-x_n^{(k)}) R n ( k ) ( x ) := 6 1 D 3 I n ( ξ x ) ( x − x n ( k ) , x − x n ( k ) , x − x n ( k ) ) ξ x ∈ V n k , R , q \xi_x\in V_n^{k,R,q} ξ x ∈ V n k , R , q x n ( k ) x_n^{(k)} x n ( k ) I n I_n I n ∇ I n ( x n ( k ) ) = 0 \nabla I_n(x_n^{(k)})=0 ∇ I n ( x n ( k ) ) = 0 (31) yields
I n k , ( 1 ) = π n ( k ) ∫ V n k , R , q ∣ exp ( − n p R n ( k ) ( x ) ) − 1 ∣ p exp ( − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x , \mathcal{I}_n^{k,(1)}=\pi_n^{(k)}\int_{V_n^{k,R,q}}\left|\exp\left(-\frac{n}{p}R_n^{(k)}(x)\right)-1\right|^p\exp\left(-\frac{n}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2\right)\,\mathrm{d}x, I n k , ( 1 ) = π n ( k ) ∫ V n k , R , q exp ( − p n R n ( k ) ( x ) ) − 1 p exp ( − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x , which, by using the inequality ∣ exp ( − t ) − 1 ∣ ≤ exp ( ∣ t ∣ ) ∣ t ∣ |\exp(-t)-1|\leq\exp(|t|)|t| ∣ exp ( − t ) − 1∣ ≤ exp ( ∣ t ∣ ) ∣ t ∣ t ∈ R t\in\mathbb{R} t ∈ R
I n k , ( 1 ) ≤ π n ( k ) ∫ V n k , R , q exp ( n ∣ R n ( k ) ( x ) ∣ ) ∣ n p R n ( k ) ( x ) ∣ p exp ( − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x . \mathcal{I}_n^{k,(1)}\leq\pi_n^{(k)}\int_{V_n^{k,R,q}}\exp\left(n|R_n^{(k)}(x)|\right)\left|\frac{n}{p}R_n^{(k)}(x)\right|^p\exp\left(-\frac{n}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2\right)\,\mathrm{d}x. I n k , ( 1 ) ≤ π n ( k ) ∫ V n k , R , q exp ( n ∣ R n ( k ) ( x ) ∣ ) p n R n ( k ) ( x ) p exp ( − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x . Exploiting Assumption 1, in particular (iii) and (ii) (via Remark 3.2), we obtain
I n k , ( 1 ) ≤ π n ( k ) M p ( n p ) ∫ V n k , R , q ∥ x − x n ( k ) ∥ 3 p exp ( − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 + n M ∥ x − x n ( k ) ∥ H n ( k ) 3 ) d x ≤ π n ( k ) M p ( n p ) ∫ V n k , R , q ∥ x − x n ( k ) ∥ 3 p exp ( − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 + M n ρ 3 ∥ x − x n ( k ) ∥ H n ( k ) 3 ) d x . (32) \begin{aligned}
\mathcal{I}_{n}^{k,(1)} \leq\pi_n^{(k)} M^p \binom{n}{p} \int_{V_n^{k,R,q}} \lVert x-x_n^{(k)}\rVert^{3p}\exp\left(-\frac{n}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2+nM\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^3\right)\,dx \\
&\leq\pi_n^{(k)} M^p \binom{n}{p} \int_{V_n^{k,R,q}} \lVert x-x_n^{(k)}\rVert^{3p}\exp\left(-\frac{n}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2+\frac{Mn}{\rho^3}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^3\right)\,dx. \tag*{(32)}
\end{aligned} I n k , ( 1 ) ≤ π n ( k ) M p ( p n ) ∫ V n k , R , q ∥ x − x n ( k ) ∥ 3 p exp ( − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 + n M ∥ x − x n ( k ) ∥ H n ( k ) 3 ) d x ≤ π n ( k ) M p ( p n ) ∫ V n k , R , q ∥ x − x n ( k ) ∥ 3 p exp ( − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 + ρ 3 M n ∥ x − x n ( k ) ∥ H n ( k ) 3 ) d x . (32) Now we estimate I n ( k ) , 2 \mathcal{I}_n^{(k),2} I n ( k ) , 2 V n k , R , q ⊆ R d ∖ V n j , R , q V_n^{k,R,q}\subseteq\mathbb{R}^d\setminus V_n^{j,R,q} V n k , R , q ⊆ R d ∖ V n j , R , q j ∈ { 1 , … , K } j\in\{1,\ldots,K\} j ∈ { 1 , … , K } j ≠ k j\neq k j = k
I n ( k ) , 2 = ∫ V n k , R , q ∣ ∑ j = 1 j ≠ k K ( π n ( j ) ) 1 / p exp ( − n 2 p ∥ x − x n ( j ) ∥ H n ( j ) 2 ) ∣ p d x ≤ ( K − 1 ) p − 1 ∑ j = 1 j ≠ k K π n ( j ) ∫ V n k , R , q exp ( − n 2 ∥ x − x n ( j ) ∥ H n ( j ) 2 ) d x ≤ ( K − 1 ) p − 1 ∑ j = 1 j ≠ k K π n ( j ) ∫ R d ∖ V n j , R , q exp ( − n 2 ∥ x − x n ( j ) ∥ H n ( j ) 2 ) d x ≤ ( K − 1 ) p − 1 ( 4 π n ) d / 2 e − n 1 − 2 q R 4 ∑ j = 1 j ≠ k K π n ( j ) det ( ( H n ( j ) ) − 1 / 2 ) . (33) \begin{aligned}
\mathcal{I}_n^{(k),2}=\int_{V_n^{k,R,q}}\left|\sum_{\substack{j=1\\j\neq k}}^K\left(\pi_n^{(j)}\right)^{1/p}\exp\left(-\frac{n}{2p}\lVert x-x_n^{(j)}\rVert_{H_n^{(j)}}^2\right)\right|^p\,dx \\
&\leq(K-1)^{p-1}\sum_{\substack{j=1\\j\neq k}}^K\pi_n^{(j)}\int_{V_n^{k,R,q}}\exp\left(-\frac{n}{2}\lVert x-x_n^{(j)}\rVert_{H_n^{(j)}}^2\right)\,dx \\
&\leq(K-1)^{p-1}\sum_{\substack{j=1\\j\neq k}}^K\pi_n^{(j)}\int_{\mathbb{R}^d\setminus V_n^{j,R,q}}\exp\left(-\frac{n}{2}\lVert x-x_n^{(j)}\rVert_{H_n^{(j)}}^2\right)\,dx \\
&\leq(K-1)^{p-1}\left(\frac{4\pi}{n}\right)^{d/2}e^{-\frac{n^{1-2q}R}{4}}\sum_{\substack{j=1\\j\neq k}}^K\pi_n^{(j)}\det\left((H_n^{(j)})^{-1/2}\right). \tag*{(33)}
\end{aligned} I n ( k ) , 2 = ∫ V n k , R , q j = 1 j = k ∑ K ( π n ( j ) ) 1/ p exp ( − 2 p n ∥ x − x n ( j ) ∥ H n ( j ) 2 ) p d x ≤ ( K − 1 ) p − 1 j = 1 j = k ∑ K π n ( j ) ∫ V n k , R , q exp ( − 2 n ∥ x − x n ( j ) ∥ H n ( j ) 2 ) d x ≤ ( K − 1 ) p − 1 j = 1 j = k ∑ K π n ( j ) ∫ R d ∖ V n j , R , q exp ( − 2 n ∥ x − x n ( j ) ∥ H n ( j ) 2 ) d x ≤ ( K − 1 ) p − 1 ( n 4 π ) d /2 e − 4 n 1 − 2 q R j = 1 j = k ∑ K π n ( j ) det ( ( H n ( j ) ) − 1/2 ) . (33) In the following we further consider I n ( k ) , 1 \mathcal{I}_n^{(k),1} I n ( k ) , 1 I n ( k ) , 2 \mathcal{I}_n^{(k),2} I n ( k ) , 2 I n ( 0 ) \mathcal{I}_n^{(0)} I n ( 0 )
Addressing statement (a): We further bound I n ( k ) , 1 \mathcal{I}_n^{(k),1} I n ( k ) , 1 q = 0 q=0 q = 0 R ≤ ρ 6 16 M 2 R\leq\frac{\rho^6}{16M^2} R ≤ 16 M 2 ρ 6
I n ( k ) , 1 ≤ π n ( k ) M p ( n p ) ∫ V n k , R , 0 ∥ x − x n ( k ) ∥ 3 p exp ( − ( n 2 − n R M ρ 3 ) ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x ≤ π n ( k ) M p ( n p ) ∫ V n k , R , 0 ∥ x − x n ( k ) ∥ 3 p exp ( − n 4 ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x ≤ π n ( k ) M p ( n p ) ∫ R d ∥ 2 ( H n ( k ) ) − 1 / 2 φ ( x ) ∥ 3 p exp ( − n 2 ∥ φ ( x ) ∥ 2 ) d x , \begin{aligned}
\mathcal{I}_n^{(k),1}\leq\pi_n^{(k)}M^p\binom{n}{p}\int_{V_n^{k,R,0}}\lVert x-x_n^{(k)}\rVert^{3p}\exp\left(-\left(\frac{n}{2}-\frac{n\sqrt{R}M}{\rho^3}\right)\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2\right)\,dx \\
&\leq\pi_n^{(k)}M^p\binom{n}{p}\int_{V_n^{k,R,0}}\lVert x-x_n^{(k)}\rVert^{3p}\exp\left(-\frac{n}{4}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2\right)\,dx \\
&\leq\pi_n^{(k)}M^p\binom{n}{p}\int_{\mathbb{R}^d}\lVert\sqrt{2}(H_n^{(k)})^{-1/2}\varphi(x)\rVert^{3p}\exp\left(-\frac{n}{2}\lVert\varphi(x)\rVert^2\right)\,dx,
\end{aligned} I n ( k ) , 1 ≤ π n ( k ) M p ( p n ) ∫ V n k , R , 0 ∥ x − x n ( k ) ∥ 3 p exp ( − ( 2 n − ρ 3 n R M ) ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x ≤ π n ( k ) M p ( p n ) ∫ V n k , R , 0 ∥ x − x n ( k ) ∥ 3 p exp ( − 4 n ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x ≤ π n ( k ) M p ( p n ) ∫ R d ∥ 2 ( H n ( k ) ) − 1/2 φ ( x ) ∥ 3 p exp ( − 2 n ∥ φ ( x ) ∥ 2 ) d x , where φ ( x ) : = 1 2 ( H n ( k ) ) 1 / 2 ( x − x n ( k ) ) . \varphi(x):=\frac{1}{\sqrt{2}}(H_n^{(k)})^{1/2}(x-x_n^{(k)}). φ ( x ) := 2 1 ( H n ( k ) ) 1/2 ( x − x n ( k ) ) .
I n ( k ) , 1 ≤ π n ( k ) M p ( n p ) 2 3 p 2 + d 2 ∥ ( H n ( k ) ) − 1 / 2 ∥ 3 p det ( ( H n ( k ) ) − 1 / 2 ) ∫ R d ∥ y ∥ 3 p e − n 2 ∥ y ∥ 2 d y . \mathcal{I}_n^{(k),1}\leq\pi_n^{(k)}M^p\binom{n}{p}2^{\frac{3p}{2}+\frac{d}{2}}\left\lVert(H_n^{(k)})^{-1/2}\right\rVert^{3p}\det\left((H_n^{(k)})^{-1/2}\right)\int_{\mathbb{R}^d}\lVert y\rVert^{3p}e^{-\frac{n}{2}\lVert y\rVert^2}\,dy. I n ( k ) , 1 ≤ π n ( k ) M p ( p n ) 2 2 3 p + 2 d ( H n ( k ) ) − 1/2 3 p det ( ( H n ( k ) ) − 1/2 ) ∫ R d ∥ y ∥ 3 p e − 2 n ∥ y ∥ 2 d y . Furthermore, Lemma A.1 and (ii) of Assumption 1 imply
I n ( k ) , 1 ≤ π n ( k ) M p ( n p ) 2 3 p 2 + d 2 ∥ ( H n ( k ) ) − 1 / 2 ∥ 3 p det ( ( H n ( k ) ) − 1 / 2 ) 2 3 p − 2 ( 2 π n ) d 2 ( d n ) 3 p 2 ≤ π n ( k ) ( 2 9 2 d 3 2 M ρ 3 p ) p ( 2 π n ) d 2 2 d 2 − 2 det ( ( H n ( k ) ) − 1 / 2 ) n − p 2 . (34) \begin{aligned}
\mathcal{I}_n^{(k),1}\leq\pi_n^{(k)}M^p\binom{n}{p}2^{\frac{3p}{2}+\frac{d}{2}}\left\lVert(H_n^{(k)})^{-1/2}\right\rVert^{3p}\det\left((H_n^{(k)})^{-1/2}\right)2^{3p-2}\left(\frac{2\pi}{n}\right)^{\frac{d}{2}}\left(\frac{d}{n}\right)^{\frac{3p}{2}} \\
&\leq\pi_n^{(k)}\left(\frac{2^{\frac{9}{2}}d^{\frac{3}{2}}M}{\rho^3p}\right)^p\left(\frac{2\pi}{n}\right)^{\frac{d}{2}}2^{\frac{d}{2}-2}\det\left((H_n^{(k)})^{-1/2}\right)n^{-\frac{p}{2}}. \tag*{(34)}
\end{aligned} I n ( k ) , 1 ≤ π n ( k ) M p ( p n ) 2 2 3 p + 2 d ( H n ( k ) ) − 1/2 3 p det ( ( H n ( k ) ) − 1/2 ) 2 3 p − 2 ( n 2 π ) 2 d ( n d ) 2 3 p ≤ π n ( k ) ( ρ 3 p 2 2 9 d 2 3 M ) p ( n 2 π ) 2 d 2 2 d − 2 det ( ( H n ( k ) ) − 1/2 ) n − 2 p . (34) Taking the definition of Z ~ n K \widetilde{Z}_n^K Z n K (7) into account and using the bounds of (30) , (33) (with q = 0 q=0 q = 0 (34) yields
2 Z ~ n K ∑ k = 1 K I n ( k ) ≤ 2 p Z ~ n K ∑ k = 1 K ( I n ( k ) , 1 + I n ( k ) , 2 ) \frac{2}{\widetilde{Z}_n^K}\sum_{k=1}^{K}\mathcal{I}_n^{(k)}\leq\frac{2^p}{\widetilde{Z}_n^K}\sum_{k=1}^{K}\left(\mathcal{I}_n^{(k),1}+\mathcal{I}_n^{(k),2}\right) Z n K 2 k = 1 ∑ K I n ( k ) ≤ Z n K 2 p k = 1 ∑ K ( I n ( k ) , 1 + I n ( k ) , 2 ) \leq2^{\frac{d}{2}-2}\left(\frac{2^{\frac{11}{2}d^{\frac{3}{2}}M}{\rho^3p}}\right)^p n^{-\frac{p}{2}}+2^{p+\frac{d}{2}}(K-1)^{p-1}e^{-\frac{nR}{4}}\sum_{k=1}^{K}\frac{\sum_{j=1,j\ne k}^{K}\pi_n^{(j)}\det\left((H_n^{(j)})^{-1/2}\right)}{\sum_{i=1}^{K}\pi_n^{(i)}\det\left((H_n^{(i)})^{-1/2}\right)}
\leq2^{\frac{d}{2}-2}\left(\frac{2^{\frac{11}{2}d^{\frac{3}{2}}M}{\rho^3p}}\right)^p n^{-\frac{p}{2}}+2^{p+\frac{d}{2}}K(K-1)^{p-1}e^{-\frac{nR}{4}}. \tag*{(35)}
For deriving the desired bound of K n , p \mathcal{K}_{n,p} K n , p (35) we further estimate the ratio 2 I n ( 0 ) / Z ~ n K 2\mathcal{I}_n^{(0)}/\widetilde{Z}_n^K 2 I n ( 0 ) / Z n K I n ( 0 ) \mathcal{I}_n^{(0)} I n ( 0 ) V n R , 0 ⊆ R d ∖ V n k , R , 0 V_n^{R,0}\subseteq\mathbb{R}^d\setminus V_n^{k,R,0} V n R , 0 ⊆ R d ∖ V n k , R , 0 k ∈ { 1 , … , K } k\in\{1,\ldots,K\} k ∈ { 1 , … , K } (7) we obtain
∑ k = 1 K π n ( k ) ∫ V n R , 0 exp ( − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x ≤ ∑ k = 1 K π n ( k ) ∫ R d ∖ V n k , R , 0 exp ( − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x \sum_{k=1}^{K}\pi_n^{(k)}\int_{V_n^{R,0}}\exp\left(-\frac{n}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2\right)\mathrm{d}x\leq\sum_{k=1}^{K}\pi_n^{(k)}\int_{\mathbb{R}^d\setminus V_n^{k,R,0}}\exp\left(-\frac{n}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2\right)\mathrm{d}x k = 1 ∑ K π n ( k ) ∫ V n R , 0 exp ( − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x ≤ k = 1 ∑ K π n ( k ) ∫ R d ∖ V n k , R , 0 exp ( − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x ≤ 2 d 2 e − n R 4 ( 2 π n ) d 2 ∑ k = 1 K π n ( k ) det ( ( H n ( k ) ) − 1 / 2 ) = 2 d 2 e − n R 4 Z ~ n K , \leq2^{\frac{d}{2}}e^{-\frac{nR}{4}}\left(\frac{2\pi}{n}\right)^{\frac{d}{2}}\sum_{k=1}^{K}\pi_n^{(k)}\det\left((H_n^{(k)})^{-1/2}\right)=2^{\frac{d}{2}}e^{-\frac{nR}{4}}\widetilde{Z}_n^K, ≤ 2 2 d e − 4 n R ( n 2 π ) 2 d k = 1 ∑ K π n ( k ) det ( ( H n ( k ) ) − 1/2 ) = 2 2 d e − 4 n R Z n K , such that 2 p Z ~ n K ∑ k = 1 K π n ( k ) ∫ V n R , q exp ( − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x ∈ O ( e − n R 4 ) \frac{2^p}{\widetilde{Z}_n^K}\sum_{k=1}^{K}\pi_n^{(k)}\int_{V_n^{R,q}}\exp\left(-\frac{n}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2\right)\mathrm{d}x\in\mathcal{O}\left(e^{-\frac{nR}{4}}\right) Z n K 2 p ∑ k = 1 K π n ( k ) ∫ V n R , q exp ( − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x ∈ O ( e − 4 n R )
By Assumption [2 ] for any n ≥ n ∗ n\geq n^* n ≥ n ∗ x ∈ V ‾ n R , 0 x\in\overline{V}_n^{R,0} x ∈ V n R , 0 I n ( x ) ≥ ε I_n(x)\geq\varepsilon I n ( x ) ≥ ε g n ( x ) : = 1 V ‾ n R , 0 ( x ) exp ( − n I n ( x ) + n ε 2 ) g_n(x):=\mathbf{1}_{\overline{V}_n^{R,0}}(x)\exp(-nI_n(x)+n\varepsilon^2) g n ( x ) := 1 V n R , 0 ( x ) exp ( − n I n ( x ) + n ε 2 ) g n ( x ) ≤ exp ( − n ( 1 − ε ) ε ) g_n(x)\leq\exp(-n(1-\varepsilon)\varepsilon) g n ( x ) ≤ exp ( − n ( 1 − ε ) ε ) lim n → ∞ g n ( x ) = 0 \lim_{n\to\infty}g_n(x)=0 lim n → ∞ g n ( x ) = 0 [2 ] we obtain
g n ( x ) ≤ exp ( − n ( 1 − ε ) I n ( x ) ) ≤ g ( x ) 1 − ε , g_n(x)\leq\exp(-n(1-\varepsilon)I_n(x))\leq g(x)^{1-\varepsilon}, g n ( x ) ≤ exp ( − n ( 1 − ε ) I n ( x )) ≤ g ( x ) 1 − ε , with g 1 − ε g^{1-\varepsilon} g 1 − ε
lim n → ∞ e n ε 2 ∫ V n R , 0 exp ( − n I n ( x ) ) d x ≤ lim n → ∞ ∫ R d g n ( x ) d x = ∫ R d lim n → ∞ g n ( x ) d x = 0 , \lim_{n\to\infty}e^{n\varepsilon^2}\int_{V_n^{R,0}}\exp(-nI_n(x))\mathrm{d}x\leq\lim_{n\to\infty}\int_{\mathbb{R}^d}g_n(x)\mathrm{d}x=\int_{\mathbb{R}^d}\lim_{n\to\infty}g_n(x)\mathrm{d}x=0, n → ∞ lim e n ε 2 ∫ V n R , 0 exp ( − n I n ( x )) d x ≤ n → ∞ lim ∫ R d g n ( x ) d x = ∫ R d n → ∞ lim g n ( x ) d x = 0 , i.e., there is an n ~ ∈ N \widetilde{n}\in\mathbb{N} n ∈ N c R , ε > 0 c_{R,\varepsilon}>0 c R , ε > 0 n ≥ n ~ n\geq\widetilde{n} n ≥ n ∫ V n R , 0 exp ( − n I n ( x ) ) d x ≤ c R , ε e − n ε 2 \int_{V_n^{R,0}}\exp(-nI_n(x))\mathrm{d}x\leq c_{R,\varepsilon}e^{-n\varepsilon^2} ∫ V n R , 0 exp ( − n I n ( x )) d x ≤ c R , ε e − n ε 2
By further exploiting Assumption [2 ] we have Z ~ n K ≥ ( 2 π n ) d / 2 ε \widetilde{Z}_n^K\geq\left(\frac{2\pi}{n}\right)^{d/2}\varepsilon Z n K ≥ ( n 2 π ) d /2 ε n ≥ n ~ n\geq\widetilde{n} n ≥ n
2 p Z ~ n K ∫ V n R , 0 exp ( − n I n ( x ) ) d x ≤ 2 p ε ( n 2 π ) d / 2 c R , ε e − n ε 2 ∈ O ( e − n ε 2 4 ) , \frac{2^p}{\widetilde{Z}_n^K}\int_{V_n^{R,0}}\exp(-nI_n(x))\mathrm{d}x\leq\frac{2^p}{\varepsilon}\left(\frac{n}{2\pi}\right)^{d/2}c_{R,\varepsilon}e^{-n\varepsilon^2}\in\mathcal{O}\left(e^{-\frac{n\varepsilon^2}{4}}\right), Z n K 2 p ∫ V n R , 0 exp ( − n I n ( x )) d x ≤ ε 2 p ( 2 π n ) d /2 c R , ε e − n ε 2 ∈ O ( e − 4 n ε 2 ) , since sup n ∈ N n d / 2 e 3 n ε 2 / 4 < ∞ \sup_{n\in\mathbb{N}}\frac{n^{d/2}}{e^{3n\varepsilon^2/4}}<\infty sup n ∈ N e 3 n ε 2 /4 n d /2 < ∞ 2 Z ~ n K I n ( 0 ) ∈ O ( e − n ε 2 / 2 + e − n R 4 ) ∈ O ( e − n 4 min { R , ε 2 } ) \frac{2}{\widetilde{Z}_n^K}\mathcal{I}_n^{(0)}\in\mathcal{O}\left(e^{-n\varepsilon^2/2}+e^{-\frac{nR}{4}}\right)\in\mathcal{O}\left(e^{-\frac{n}{4}\min\{R,\varepsilon^2\}}\right) Z n K 2 I n ( 0 ) ∈ O ( e − n ε 2 /2 + e − 4 n R ) ∈ O ( e − 4 n m i n { R , ε 2 } )
Now we conclude with the final estimate for statement (a). Combining (35) with 2 Z ~ n K I n ( 0 ) ∈ O ( e − n 4 min { R , ε 2 } ) \frac{2}{\widetilde{Z}_n^K}\mathcal{I}_n^{(0)}\in\mathcal{O}\left(e^{-\frac{n}{4}\min\{R,\varepsilon^2\}}\right) Z n K 2 I n ( 0 ) ∈ O ( e − 4 n m i n { R , ε 2 } )
K n , p ≤ 2 d 2 − 2 ( 2 11 2 d 3 2 M ρ 3 p ) p n − p 2 + O ( e − n 4 min { R , ε 2 } ) . (36) \mathcal{K}_{n,p}\leq2^{\frac{d}{2}-2}\left(2^{\frac{11}{2}d^{\frac{3}{2}}M}{\rho^3p}\right)^p n^{-\frac{p}{2}}+\mathcal{O}\left(e^{-\frac{n}{4}\min\{R,\varepsilon^2\}}\right). \tag*{(36)} K n , p ≤ 2 2 d − 2 ( 2 2 11 d 2 3 M ρ 3 p ) p n − 2 p + O ( e − 4 n m i n { R , ε 2 } ) . (36) Addressing statement (b): We start again with bounding I n ( k ) , 1 \mathcal{I}_{n}^{(k),1} I n ( k ) , 1 (32) . For q > 1 / 3 q>1/3 q > 1/3 R = ρ 2 / M 2 3 R=\rho^2/M^{\frac{2}{3}} R = ρ 2 / M 3 2
I n ( k ) , 1 ≤ π n ( k ) M p ( n p ) p exp ( M R 3 2 n 1 − 3 q ρ 3 ) ∫ V n k , R , q ∥ x − x n ( k ) ∥ 3 p exp ( − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x ≤ e π n ( k ) M p ( n p ) p ∫ V n k , R , q ∥ x − x n ( k ) ∥ 3 p exp ( − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x ≤ e π n ( k ) M p ( n p ) p ∫ R d ∥ ( H n ( k ) ) − 1 / 2 φ ( x ) ∥ 3 p exp ( − n 2 ∥ φ ( x ) ∥ 2 ) d x , \begin{aligned}
\mathcal{I}_{n}^{(k),1}\leq\pi_{n}^{(k)}M^p\left(\frac{n}{p}\right)^p\exp\left(\frac{M R^{\frac{3}{2}}n^{1-3q}}{\rho^3}\right)\int_{\mathcal{V}_{n}^{k,R,q}}\|x-x_n^{(k)}\|^{3p}\exp\left(-\frac{n}{2}\|x-x_n^{(k)}\|_{H_n^{(k)}}^2\right)\,dx \\
&\leq e\pi_{n}^{(k)}M^p\left(\frac{n}{p}\right)^p\int_{\mathcal{V}_{n}^{k,R,q}}\|x-x_n^{(k)}\|^{3p}\exp\left(-\frac{n}{2}\|x-x_n^{(k)}\|_{H_n^{(k)}}^2\right)\,dx \\
&\leq e\pi_{n}^{(k)}M^p\left(\frac{n}{p}\right)^p\int_{\mathbb{R}^d}\left\|(H_n^{(k)})^{-1/2}\varphi(x)\right\|^{3p}\exp\left(-\frac{n}{2}\|\varphi(x)\|^2\right)\,dx,
\end{aligned} I n ( k ) , 1 ≤ π n ( k ) M p ( p n ) p exp ( ρ 3 M R 2 3 n 1 − 3 q ) ∫ V n k , R , q ∥ x − x n ( k ) ∥ 3 p exp ( − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x ≤ e π n ( k ) M p ( p n ) p ∫ V n k , R , q ∥ x − x n ( k ) ∥ 3 p exp ( − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x ≤ e π n ( k ) M p ( p n ) p ∫ R d ( H n ( k ) ) − 1/2 φ ( x ) 3 p exp ( − 2 n ∥ φ ( x ) ∥ 2 ) d x , where φ ( x ) : = ( H n ( k ) ) 1 / 2 ( x − x n ( k ) ) . \varphi(x):=(H_n^{(k)})^{1/2}(x-x_n^{(k)}). φ ( x ) := ( H n ( k ) ) 1/2 ( x − x n ( k ) ) .
I n ( k ) , 1 ≤ e π n ( k ) M p ( n p ) p ∥ ( H n ( k ) ) − 1 / 2 ∥ 3 p det ( ( H n ( k ) ) − 1 / 2 ) ∫ R d ∥ y ∥ 3 p e − n 2 ∥ y ∥ 2 d y . \mathcal{I}_{n}^{(k),1}\leq e\pi_{n}^{(k)}M^p\left(\frac{n}{p}\right)^p\left\|(H_n^{(k)})^{-1/2}\right\|^{3p}\det\left((H_n^{(k)})^{-1/2}\right)\int_{\mathbb{R}^d}\|y\|^{3p}e^{-\frac{n}{2}\|y\|^2}\,dy. I n ( k ) , 1 ≤ e π n ( k ) M p ( p n ) p ( H n ( k ) ) − 1/2 3 p det ( ( H n ( k ) ) − 1/2 ) ∫ R d ∥ y ∥ 3 p e − 2 n ∥ y ∥ 2 d y . Furthermore, Lemma A.1 and (ii) of Assumption 1 imply
I n ( k ) , 1 ≤ e π n ( k ) M p ( n p ) p ∥ ( H n ( k ) ) − 1 / 2 ∥ 3 p det ( ( H n ( k ) ) − 1 / 2 ) 2 3 p − 2 ( 2 π n ) d 2 ( d n ) 3 p 2 ≤ e 4 π n ( k ) ( 2 3 d 3 2 M ρ 3 p ) p ( 2 π n ) d 2 det ( ( H n ( k ) ) − 1 / 2 ) n − p 2 . (37) \begin{aligned}
\mathcal{I}_{n}^{(k),1}\leq e\pi_{n}^{(k)}M^p\left(\frac{n}{p}\right)^p\left\|(H_n^{(k)})^{-1/2}\right\|^{3p}\det\left((H_n^{(k)})^{-1/2}\right)2^{3p-2}\left(\frac{2\pi}{n}\right)^{\frac{d}{2}}\left(\frac{d}{n}\right)^{\frac{3p}{2}} \\
&\leq\frac{e}{4}\pi_{n}^{(k)}\left(\frac{2^3d^{\frac{3}{2}}M}{\rho^3p}\right)^p\left(\frac{2\pi}{n}\right)^{\frac{d}{2}}\det\left((H_n^{(k)})^{-1/2}\right)n^{-\frac{p}{2}}. \tag*{(37)}
\end{aligned} I n ( k ) , 1 ≤ e π n ( k ) M p ( p n ) p ( H n ( k ) ) − 1/2 3 p det ( ( H n ( k ) ) − 1/2 ) 2 3 p − 2 ( n 2 π ) 2 d ( n d ) 2 3 p ≤ 4 e π n ( k ) ( ρ 3 p 2 3 d 2 3 M ) p ( n 2 π ) 2 d det ( ( H n ( k ) ) − 1/2 ) n − 2 p . (37) Taking the definition of Z ~ n K \widetilde{Z}_n^K Z n K (7) into account and using the bounds of (30) , (33) (with R = ρ 2 / M 2 3 R=\rho^2/M^{\frac{2}{3}} R = ρ 2 / M 3 2 (37) yields
2 Z ~ n K ∑ k = 1 K I n ( k ) ≤ 2 p Z ~ n K ∑ k = 1 K ( I n ( k ) , 1 + I n ( k ) , 2 ) ≤ 2 p Z ~ n K ∑ k = 1 K [ e 4 π n ( k ) ( 2 3 d 3 2 M ρ 3 p ) p ( 2 π n ) d 2 det ( ( H n ( k ) ) − 1 / 2 ) n − p 2 \begin{aligned}
\frac{2}{\widetilde{Z}_n^K}\sum_{k=1}^K\mathcal{I}_n^{(k)}
\leq\frac{2^p}{\widetilde{Z}_n^K}\sum_{k=1}^K\left(\mathcal{I}_n^{(k),1}+\mathcal{I}_n^{(k),2}\right) \\
&\leq\frac{2^p}{\widetilde{Z}_n^K}\sum_{k=1}^K\left[\frac{e}{4}\pi_n^{(k)}\left(\frac{2^3d^{\frac{3}{2}}M}{\rho^3p}\right)^p\left(\frac{2\pi}{n}\right)^{\frac{d}{2}}\det\left((H_n^{(k)})^{-1/2}\right)n^{-\frac{p}{2}}
\right.
\end{aligned} Z n K 2 k = 1 ∑ K I n ( k ) ≤ Z n K 2 p k = 1 ∑ K ( I n ( k ) , 1 + I n ( k ) , 2 ) ≤ Z n K 2 p k = 1 ∑ K [ 4 e π n ( k ) ( ρ 3 p 2 3 d 2 3 M ) p ( n 2 π ) 2 d det ( ( H n ( k ) ) − 1/2 ) n − 2 p + ( K − 1 ) p − 1 ( 4 π n ) d 2 exp ( − ρ 2 n ( 1 − 2 q ) 4 M 2 3 ) ∑ j = 1 , j ≠ k K π n ( j ) det ( ( H n ( j ) ) − 1 / 2 ) ] = e 4 ( 2 4 d 3 2 M ρ 3 p ) p n − p 2 + 2 p + d 2 ( K − 1 ) p − 1 exp ( − ρ 2 n ( 1 − 2 q ) 4 M 2 3 ) ∑ k = 1 K ∑ j = 1 , j ≠ k K π n ( j ) det ( ( H n ( j ) ) − 1 / 2 ) ∑ i = 1 K π n ( i ) det ( ( H n ( i ) ) − 1 / 2 ) ≤ e 4 ( 2 4 d 3 2 M ρ 3 p ) p n − p 2 + 2 p + d 2 K ( K − 1 ) p − 1 exp ( − ρ 2 n ( 1 − 2 q ) 4 M 2 3 ) . (38) \begin{aligned}
\left.
\qquad+(K-1)^{p-1}\left(\frac{4\pi}{n}\right)^{\frac{d}{2}}\exp\left(-\frac{\rho^2n^{(1-2q)}}{4M^{\frac{2}{3}}}\right)\sum_{j=1,j\ne k}^{K}\pi_n^{(j)}\det\left((H_n^{(j)})^{-1/2}\right)\right] \\
&= \frac{e}{4}\left(\frac{2^4d^{\frac{3}{2}}M}{\rho^3p}\right)^p n^{-\frac{p}{2}}+2^{p+\frac{d}{2}}(K-1)^{p-1}\exp\left(-\frac{\rho^2n^{(1-2q)}}{4M^{\frac{2}{3}}}\right)
\frac{\sum_{k=1}^K\sum_{j=1,j\ne k}^{K}\pi_n^{(j)}\det\left((H_n^{(j)})^{-1/2}\right)}
{\sum_{i=1}^{K}\pi_n^{(i)}\det\left((H_n^{(i)})^{-1/2}\right)} \\
&\leq\frac{e}{4}\left(\frac{2^4d^{\frac{3}{2}}M}{\rho^3p}\right)^p n^{-\frac{p}{2}}+2^{p+\frac{d}{2}}K(K-1)^{p-1}\exp\left(-\frac{\rho^2n^{(1-2q)}}{4M^{\frac{2}{3}}}\right). \tag*{(38)}
\end{aligned} + ( K − 1 ) p − 1 ( n 4 π ) 2 d exp ( − 4 M 3 2 ρ 2 n ( 1 − 2 q ) ) j = 1 , j = k ∑ K π n ( j ) det ( ( H n ( j ) ) − 1/2 ) = 4 e ( ρ 3 p 2 4 d 2 3 M ) p n − 2 p + 2 p + 2 d ( K − 1 ) p − 1 exp ( − 4 M 3 2 ρ 2 n ( 1 − 2 q ) ) ∑ i = 1 K π n ( i ) det ( ( H n ( i ) ) − 1/2 ) ∑ k = 1 K ∑ j = 1 , j = k K π n ( j ) det ( ( H n ( j ) ) − 1/2 ) ≤ 4 e ( ρ 3 p 2 4 d 2 3 M ) p n − 2 p + 2 p + 2 d K ( K − 1 ) p − 1 exp ( − 4 M 3 2 ρ 2 n ( 1 − 2 q ) ) . (38) Now we further estimate I n ( 0 ) \mathcal{I}_n^{(0)} I n ( 0 ) I n ( 0 ) / Z ~ n K \mathcal{I}_n^{(0)}/\widetilde{Z}_n^K I n ( 0 ) / Z n K I n ( 0 ) \mathcal{I}_n^{(0)} I n ( 0 ) (28) ) satisfies
2 p − 1 ∫ V n R , q exp ( − n L n ( x ) ) d x ≤ 2 p − 1 ∫ V n R , q ∑ k = 1 K π n ( k ) exp ( − n δ ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x 2^{p-1}\int_{\mathcal{V}_n^{R,q}}\exp(-nL_n(x))\,dx\leq2^{p-1}\int_{\mathcal{V}_n^{R,q}}\sum_{k=1}^{K}\pi_n^{(k)}\exp\left(-n\delta\|x-x_n^{(k)}\|_{H_n^{(k)}}^2\right)\,dx 2 p − 1 ∫ V n R , q exp ( − n L n ( x )) d x ≤ 2 p − 1 ∫ V n R , q k = 1 ∑ K π n ( k ) exp ( − n δ ∥ x − x n ( k ) ∥ H n ( k ) 2 ) d x such that
I n ( 0 ) ≤ 2 p − 1 ∫ V n R , q ∑ k = 1 K [ π n ( k ) exp ( − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 ) + π n ( k ) exp ( − n δ ∥ x − x n ( k ) ∥ H n ( k ) 2 ) ] d x \mathcal{I}^{(0)}_n \leq2^{p-1}\int_{\mathcal{V}^{R,q}_n}\sum_{k=1}^{K}\left[\pi_n^{(k)}\exp\left(-\frac{n}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2\right)+\pi_n^{(k)}\exp\left(-n\delta\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2\right)\right]\mathrm{d}x I n ( 0 ) ≤ 2 p − 1 ∫ V n R , q k = 1 ∑ K [ π n ( k ) exp ( − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 ) + π n ( k ) exp ( − n δ ∥ x − x n ( k ) ∥ H n ( k ) 2 ) ] d x ≤ 2 p − 1 ∑ k = 1 K π n ( k ) ∫ R d ∖ V n R , q [ exp ( − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 ) + exp ( − n δ ∥ x − x n ( k ) ∥ H n ( k ) 2 ) ] d x , \leq2^{p-1}\sum_{k=1}^{K}\pi_n^{(k)}\int_{\mathbb{R}^d\setminus\mathcal{V}^{R,q}_n}\left[\exp\left(-\frac{n}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2\right)+\exp\left(-n\delta\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2\right)\right]\mathrm{d}x, ≤ 2 p − 1 k = 1 ∑ K π n ( k ) ∫ R d ∖ V n R , q [ exp ( − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 ) + exp ( − n δ ∥ x − x n ( k ) ∥ H n ( k ) 2 ) ] d x , where the last inequality follows by the fact that V ‾ n R , q ⊆ V n k , R , q \overline{\mathcal{V}}^{R,q}_n\subseteq\mathcal{V}^{k,R,q}_n V n R , q ⊆ V n k , R , q k ∈ { 1 , … , K } k\in\{1,\ldots,K\} k ∈ { 1 , … , K }
I n ( 0 ) ≤ 2 p − 1 ∑ k = 1 K π n ( k ) det ( ( H n ( k ) ) − 1 / 2 ) ( ( 4 π n ) d 2 e − R 4 n 1 − 2 q + ( 2 π n δ ) d 2 e − δ R 2 n 1 − 2 q ) \mathcal{I}^{(0)}_n\leq2^{p-1}\sum_{k=1}^{K}\pi_n^{(k)}\det\left((H_n^{(k)})^{-1/2}\right)\left(\left(\frac{4\pi}{n}\right)^{\frac{d}{2}}e^{-\frac{R}{4}n^{1-2q}}+\left(\frac{2\pi}{n\delta}\right)^{\frac{d}{2}}e^{-\frac{\delta R}{2}n^{1-2q}}\right) I n ( 0 ) ≤ 2 p − 1 k = 1 ∑ K π n ( k ) det ( ( H n ( k ) ) − 1/2 ) ( ( n 4 π ) 2 d e − 4 R n 1 − 2 q + ( n δ 2 π ) 2 d e − 2 δ R n 1 − 2 q ) ≤ 2 p − 1 ( 4 d 2 e − R 4 n 1 − 2 q + 2 d 2 δ − d 2 e − δ R 2 n 1 − 2 q ) ( π n ) d 2 ∑ k = 1 K π n ( k ) det ( ( H n ( k ) ) − 1 / 2 ) \leq2^{p-1}\left(4^{\frac{d}{2}}e^{-\frac{R}{4}n^{1-2q}}+2^{\frac{d}{2}}\delta^{-\frac{d}{2}}e^{-\frac{\delta R}{2}n^{1-2q}}\right)\left(\frac{\pi}{n}\right)^{\frac{d}{2}}\sum_{k=1}^{K}\pi_n^{(k)}\det\left((H_n^{(k)})^{-1/2}\right) ≤ 2 p − 1 ( 4 2 d e − 4 R n 1 − 2 q + 2 2 d δ − 2 d e − 2 δ R n 1 − 2 q ) ( n π ) 2 d k = 1 ∑ K π n ( k ) det ( ( H n ( k ) ) − 1/2 ) ≤ 2 p − 1 + d 2 e − R 4 min { 1 , 2 δ } n 1 − 2 q ( 2 d 2 + δ − d 2 ) ( π n ) d 2 ∑ k = 1 K π n ( k ) det ( ( H n ( k ) ) − 1 / 2 ) . \leq2^{p-1+\frac{d}{2}}e^{-\frac{R}{4}\min\{1,2\delta\}n^{1-2q}}\left(2^{\frac{d}{2}}+\delta^{-\frac{d}{2}}\right)\left(\frac{\pi}{n}\right)^{\frac{d}{2}}\sum_{k=1}^{K}\pi_n^{(k)}\det\left((H_n^{(k)})^{-1/2}\right). ≤ 2 p − 1 + 2 d e − 4 R m i n { 1 , 2 δ } n 1 − 2 q ( 2 2 d + δ − 2 d ) ( n π ) 2 d k = 1 ∑ K π n ( k ) det ( ( H n ( k ) ) − 1/2 ) . Plugging (7) and R = ρ 2 / M 2 / 3 R=\rho^2/M^{2/3} R = ρ 2 / M 2/3
2 Z ~ n K I n ( 0 ) ≤ 2 p ( 2 d 2 + δ − d 2 ) exp ( − ρ 2 min { 1 , 2 δ } 4 M 2 / 3 n 1 − 2 q ) , \frac{2}{\widetilde{Z}_n^K}\mathcal{I}^{(0)}_n\leq2^p\left(2^{\frac{d}{2}}+\delta^{-\frac{d}{2}}\right)\exp\left(-\frac{\rho^2\min\{1,2\delta\}}{4M^{2/3}}n^{1-2q}\right), Z n K 2 I n ( 0 ) ≤ 2 p ( 2 2 d + δ − 2 d ) exp ( − 4 M 2/3 ρ 2 min { 1 , 2 δ } n 1 − 2 q ) , such that we finally obtain with (38) that
K n , p ≤ e 4 ( 2 4 d d 2 M ρ 3 p ) p n − p 2 + 2 p + d 2 ( K ( K − 1 ) p − 1 + 1 + ( 2 δ ) − d 2 ) exp ( − ρ 2 min { 1 , 2 δ } 4 M 2 / 3 n 1 − 2 q ) , \mathcal{K}_{n,p}\leq\frac{\mathrm{e}}{4}\left(\frac{2^4d^{\frac{d}{2}}M}{\rho^3p}\right)^p n^{-\frac{p}{2}}+2^{p+\frac{d}{2}}\left(K(K-1)^{p-1}+1+(2\delta)^{-\frac{d}{2}}\right)\exp\left(-\frac{\rho^2\min\{1,2\delta\}}{4M^{2/3}}n^{1-2q}\right), K n , p ≤ 4 e ( ρ 3 p 2 4 d 2 d M ) p n − 2 p + 2 p + 2 d ( K ( K − 1 ) p − 1 + 1 + ( 2 δ ) − 2 d ) exp ( − 4 M 2/3 ρ 2 min { 1 , 2 δ } n 1 − 2 q ) , which finishes the proof. □
Proof of theo:MainpropHelling2 and theo:MainpropHelling
Proof of Theorem 3.1 and Theorem 3.2
Define
K ~ n , 1 : = 2 Z n ( ( Z ~ n K ) − 1 / 2 − Z n − 1 / 2 ) 2 , \widetilde{\mathcal{K}}_{n,1}:=2Z_n\left((\widetilde{Z}_n^K)^{-1/2}-Z_n^{-1/2}\right)^2, K n , 1 := 2 Z n ( ( Z n K ) − 1/2 − Z n − 1/2 ) 2 , K ~ n , 2 : = 2 Z ~ n K ∫ R d ∣ e − n 2 I n ( x ) − ( ∑ j = 1 K π n ( j ) exp ( − n 2 ∥ x − x n ( j ) ∥ H n ( j ) 2 ) ) 1 2 ∣ 2 d x \widetilde{\mathcal{K}}_{n,2}:=\frac{2}{\widetilde{Z}_n^K}\int_{\mathbb{R}^d}\left|\mathrm{e}^{-\frac{n}{2}I_n(x)}-\left(\sum_{j=1}^{K}\pi_n^{(j)}\exp\left(-\frac{n}{2}\lVert x-x_n^{(j)}\rVert_{H_n^{(j)}}^2\right)\right)^{\frac{1}{2}}\right|^2\mathrm{d}x K n , 2 := Z n K 2 ∫ R d e − 2 n I n ( x ) − ( j = 1 ∑ K π n ( j ) exp ( − 2 n ∥ x − x n ( j ) ∥ H n ( j ) 2 ) ) 2 1 2 d x and note that by suitable application of (25) we have
d H 2 ( P π n K , π n ) = ∫ R d [ π n ( x ) − P π n K ( x ) ] 2 d x ≤ K ~ n , 1 + K ~ n , 2 . d_H^2(\mathcal{P}_{\pi_n}^K,\pi_n)=\int_{\mathbb{R}^d}\left[\sqrt{\pi_n(x)}-\sqrt{\mathcal{P}_{\pi_n}^K(x)}\right]^2\mathrm{d}x\leq\widetilde{\mathcal{K}}_{n,1}+\widetilde{\mathcal{K}}_{n,2}. d H 2 ( P π n K , π n ) = ∫ R d [ π n ( x ) − P π n K ( x ) ] 2 d x ≤ K n , 1 + K n , 2 . Moreover, by the definition of K n , 1 , K n , 2 \mathcal{K}_{n,1},\mathcal{K}_{n,2} K n , 1 , K n , 2 K ~ n , 2 = K n , 2 \widetilde{\mathcal{K}}_{n,2}=\mathcal{K}_{n,2} K n , 2 = K n , 2
K ~ n , 1 = 2 Z n ( ( Z ~ n K ) − 1 − Z n − 1 ) 2 ( ( Z ~ n K ) − 1 / 2 + Z n − 1 / 2 ) 2 ≤ 2 Z n ( ( Z ~ n K ) − 1 − Z n − 1 ) 2 ( Z n − 1 / 2 ) 2 = 2 ( Z n − Z ~ n K ) 2 ( Z ~ n K ) 2 ≤ K n , 1 2 2 . \widetilde{\mathcal{K}}_{n,1}=\frac{2Z_n\left((\widetilde{Z}_n^K)^{-1}-Z_n^{-1}\right)^2}{\left((\widetilde{Z}_n^K)^{-1/2}+Z_n^{-1/2}\right)^2}\leq\frac{2Z_n\left((\widetilde{Z}_n^K)^{-1}-Z_n^{-1}\right)^2}{(Z_n^{-1/2})^2}=\frac{2(Z_n-\widetilde{Z}_n^K)^2}{(\widetilde{Z}_n^K)^2}\leq\frac{\mathcal{K}_{n,1}^2}{2}. K n , 1 = ( ( Z n K ) − 1/2 + Z n − 1/2 ) 2 2 Z n ( ( Z n K ) − 1 − Z n − 1 ) 2 ≤ ( Z n − 1/2 ) 2 2 Z n ( ( Z n K ) − 1 − Z n − 1 ) 2 = ( Z n K ) 2 2 ( Z n − Z n K ) 2 ≤ 2 K n , 1 2 . Proof of Theorem 3.1. Applying statement (a) of Lemma A.3 with p = 1 p=1 p = 1 p = 2 p=2 p = 2 (25) , leads to
d H 2 ( L π n K , π n ) ≤ 2 11 d 3 M 2 ρ 6 ( 2 d − 4 + 2 d 2 − 4 ) n − 1 + O ( e − n 4 min { R , ε 2 } ) , d_{\mathrm{H}}^{2}(\mathcal{L}_{\pi_n}^{K},\pi_n) \leq\frac{2^{11}d^{3}M^{2}}{\rho^{6}}\left(2^{d-4}+2^{\frac{d}{2}-4}\right)n^{-1}+\mathcal{O}\left(e^{-\frac{n}{4}\min\{R,\varepsilon^{2}\}}\right), d H 2 ( L π n K , π n ) ≤ ρ 6 2 11 d 3 M 2 ( 2 d − 4 + 2 2 d − 4 ) n − 1 + O ( e − 4 n m i n { R , ε 2 } ) , such that d H ( L π n K , π n ) ∈ O ( n − 1 2 ) d_{\mathrm{H}}(\mathcal{L}_{\pi_n}^{K},\pi_n)\in\mathcal{O}(n^{-\frac{1}{2}}) d H ( L π n K , π n ) ∈ O ( n − 2 1 )
Proof of Theorem 3.2. Applying statement (b) of Lemma A.3 with p = 1 p=1 p = 1 p = 2 p=2 p = 2
d H 2 ( L π n K , π n ) ≤ e 2 4 d 3 M 2 n ρ 6 ( e + 1 ) + 4 exp ( − ρ 2 min { 1 , 2 δ } 2 M 2 3 n ( 1 − 2 q ) ) [ 2 d ( K + 1 + ( 2 δ ) − d 2 ) 2 ] + 4 exp ( − ρ 2 min { 1 , 2 δ } 4 M 2 3 n ( 1 − 2 q ) ) [ 2 d 2 ( K 2 − K + 1 + ( 2 δ ) − d 2 ) ] ≤ e 2 4 d 3 M 2 n ρ 6 ( e + 1 ) + 2 d + 2 exp ( − ρ 2 min { 1 , 2 δ } 4 M 2 3 n ( 1 − 2 q ) ) [ 2 ( K + 1 ) 2 + 2 ( 2 δ ) − d + K 2 + ( 2 δ ) − d 2 ] ≤ e 2 4 d 3 M 2 n ρ 6 ( e + 1 ) + 12 ⋅ 2 d exp ( − ρ 2 min { 1 , 2 δ } 4 M 2 3 n ( 1 − 2 q ) ) [ ( K + 1 ) 2 + max { 1 , ( 2 δ ) − d } ] , \begin{aligned}
d_{\mathrm{H}}^{2}(\mathcal{L}_{\pi_n}^{K},\pi_n) \leq\frac{e2^{4}d^{3}M^{2}}{n\rho^{6}}(e+1)+4\exp\left(-\frac{\rho^{2}\min\{1,2\delta\}}{2M^{\frac{2}{3}}}n^{(1-2q)}\right)\left[2^{d}\left(K+1+(2\delta)^{-\frac{d}{2}}\right)^{2}\right] \\
&+4\exp\left(-\frac{\rho^{2}\min\{1,2\delta\}}{4M^{\frac{2}{3}}}n^{(1-2q)}\right)\left[2^{\frac{d}{2}}\left(K^{2}-K+1+(2\delta)^{-\frac{d}{2}}\right)\right] \\
&\leq\frac{e2^{4}d^{3}M^{2}}{n\rho^{6}}(e+1)+2^{d+2}\exp\left(-\frac{\rho^{2}\min\{1,2\delta\}}{4M^{\frac{2}{3}}}n^{(1-2q)}\right)\left[2(K+1)^{2}+2(2\delta)^{-d}+K^{2}+(2\delta)^{-\frac{d}{2}}\right] \\
&\leq\frac{e2^{4}d^{3}M^{2}}{n\rho^{6}}(e+1)+12\cdot2^{d}\exp\left(-\frac{\rho^{2}\min\{1,2\delta\}}{4M^{\frac{2}{3}}}n^{(1-2q)}\right)\left[(K+1)^{2}+\max\{1,(2\delta)^{-d}\}\right],
\end{aligned} d H 2 ( L π n K , π n ) ≤ n ρ 6 e 2 4 d 3 M 2 ( e + 1 ) + 4 exp ( − 2 M 3 2 ρ 2 min { 1 , 2 δ } n ( 1 − 2 q ) ) [ 2 d ( K + 1 + ( 2 δ ) − 2 d ) 2 ] + 4 exp ( − 4 M 3 2 ρ 2 min { 1 , 2 δ } n ( 1 − 2 q ) ) [ 2 2 d ( K 2 − K + 1 + ( 2 δ ) − 2 d ) ] ≤ n ρ 6 e 2 4 d 3 M 2 ( e + 1 ) + 2 d + 2 exp ( − 4 M 3 2 ρ 2 min { 1 , 2 δ } n ( 1 − 2 q ) ) [ 2 ( K + 1 ) 2 + 2 ( 2 δ ) − d + K 2 + ( 2 δ ) − 2 d ] ≤ n ρ 6 e 2 4 d 3 M 2 ( e + 1 ) + 12 ⋅ 2 d exp ( − 4 M 3 2 ρ 2 min { 1 , 2 δ } n ( 1 − 2 q ) ) [ ( K + 1 ) 2 + max { 1 , ( 2 δ ) − d } ] , where also (25) and 2 ( 2 δ ) − d + ( 2 δ ) − d 2 ≤ 3 max { 1 , ( 2 δ ) − d } 2(2\delta)^{-d}+(2\delta)^{-\frac{d}{2}}\leq3\max\{1,(2\delta)^{-d}\} 2 ( 2 δ ) − d + ( 2 δ ) − 2 d ≤ 3 max { 1 , ( 2 δ ) − d }
d H ( L π n K , π n ) ≤ 13 d 3 2 M n 1 2 ρ 3 + 2 d 2 + 2 exp ( − ρ 2 min { 1 , 2 δ } 8 M 2 3 n ( 1 − 2 q ) ) [ ( K + 1 ) + max { 1 , ( 2 δ ) − d 2 } ] . d_{\mathrm{H}}(\mathcal{L}_{\pi_n}^{K},\pi_n)\leq\frac{13d^{\frac{3}{2}}M}{n^{\frac{1}{2}}\rho^{3}}+2^{\frac{d}{2}+2}\exp\left(-\frac{\rho^{2}\min\{1,2\delta\}}{8M^{\frac{2}{3}}}n^{(1-2q)}\right)\left[(K+1)+\max\{1,(2\delta)^{-\frac{d}{2}}\}\right]. d H ( L π n K , π n ) ≤ n 2 1 ρ 3 13 d 2 3 M + 2 2 d + 2 exp ( − 8 M 3 2 ρ 2 min { 1 , 2 δ } n ( 1 − 2 q ) ) [ ( K + 1 ) + max { 1 , ( 2 δ ) − 2 d } ] . Proof of cor: accesablestatement
We start with a purely analytical auxiliary estimate.
Lemma A.4. For C > 0 C>0 C > 0 q ∈ [ 0 , 1 / 2 ) q\in[0,1/2) q ∈ [ 0 , 1/2 ) D ≥ 1 + e − 1 1 − 2 q D\geq\frac{1+e^{-1}}{1-2q} D ≥ 1 − 2 q 1 + e − 1 n ∈ N n\in\mathbb{N} n ∈ N n ≥ ( D C log ( D C ) ) 1 − 2 q n\geq\left(\frac{D}{C}\log\left(\frac{D}{C}\right)\right)^{1-2q} n ≥ ( C D log ( C D ) ) 1 − 2 q exp ( − C n 1 − 2 q ) ≤ n − 1 \exp(-Cn^{1-2q})\leq n^{-1} exp ( − C n 1 − 2 q ) ≤ n − 1
Proof. Define f : ( 0 , ∞ ) → ( 0 , ∞ ) f:(0,\infty)\to(0,\infty) f : ( 0 , ∞ ) → ( 0 , ∞ ) f ( x ) : = log x x 1 − 2 q f(x):=\frac{\log x}{x^{1-2q}} f ( x ) := x 1 − 2 q l o g x exp ( − C n 1 − 2 q ) ≤ n − 1 \exp(-Cn^{1-2q})\leq n^{-1} exp ( − C n 1 − 2 q ) ≤ n − 1 f ( n ) ≤ C f(n)\leq C f ( n ) ≤ C f f f n ∗ : = e 1 1 − 2 q n^{*}:=e^{\frac{1}{1-2q}} n ∗ := e 1 − 2 q 1 f ( n ∗ ) = ( ( 1 − 2 q ) e ) − 1 f(n^{*})=((1-2q)e)^{-1} f ( n ∗ ) = (( 1 − 2 q ) e ) − 1 C > f ( n ∗ ) C>f(n^{*}) C > f ( n ∗ ) f ( n ) ≤ f ( n ∗ ) ≤ C f(n)\leq f(n^{*})\leq C f ( n ) ≤ f ( n ∗ ) ≤ C n ∈ N n\in\mathbb{N} n ∈ N 0 < C ≤ f ( n ∗ ) 0<C\leq f(n^{*}) 0 < C ≤ f ( n ∗ ) f f f x > n ∗ x>n^{*} x > n ∗
n ≥ ( D C log ( D C ) ) 1 − 2 q ≥ ( ( e + 1 ) log ( e + 1 ) ) 1 − 2 q > n ∗ , n\geq\left(\frac{D}{C}\log\left(\frac{D}{C}\right)\right)^{1-2q}\geq\left((e+1)\log(e+1)\right)^{1-2q}>n^{*}, n ≥ ( C D log ( C D ) ) 1 − 2 q ≥ ( ( e + 1 ) log ( e + 1 ) ) 1 − 2 q > n ∗ , where we used that D C ≥ e + 1 \frac{D}{C}\geq e+1 C D ≥ e + 1 x ↦ x log x x\mapsto x\log x x ↦ x log x x > e − 1 x>e^{-1} x > e − 1 n > n ∗ n>n^{*} n > n ∗
f ( n ) ≤ log [ ( D C ) 1 − 2 q [ log ( D C ) ] 1 − 2 q ] D C log ( D C ) = C D ( 1 1 − 2 q + f ( [ log ( D C ) ] 1 − 2 q ) ) ≤ C D ( 1 1 − 2 q + f ( n ∗ ) ) = C ( 1 + e − 1 ) D ( 1 − 2 q ) ≤ C , \begin{aligned}
f(n)\leq\frac{\log\left[\left(\frac{D}{C}\right)^{1-2q}\left[\log\left(\frac{D}{C}\right)\right]^{1-2q}\right]}{\frac{D}{C}\log\left(\frac{D}{C}\right)}=\frac{C}{D}\left(\frac{1}{1-2q}+f\left(\left[\log\left(\frac{D}{C}\right)\right]^{1-2q}\right)\right) \\
&\leq\frac{C}{D}\left(\frac{1}{1-2q}+f(n^{*})\right)=\frac{C(1+e^{-1})}{D(1-2q)}\leq C,
\end{aligned} f ( n ) ≤ C D log ( C D ) log [ ( C D ) 1 − 2 q [ log ( C D ) ] 1 − 2 q ] = D C ( 1 − 2 q 1 + f ( [ log ( C D ) ] 1 − 2 q ) ) ≤ D C ( 1 − 2 q 1 + f ( n ∗ ) ) = D ( 1 − 2 q ) C ( 1 + e − 1 ) ≤ C , where the last inequality follows by D ≥ 1 + e − 1 1 − 2 q D\geq\frac{1+e^{-1}}{1-2q} D ≥ 1 − 2 q 1 + e − 1
Proof of Corollary 3.1. Note that (9) follows immediately from Theorem 3.1 and the fact that δ ≥ 1 / 2 \delta\ge1/2 δ ≥ 1/2 n n n C = ρ 2 8 M 2 / 3 C = \frac{\rho^2}{8M^{2/3}} C = 8 M 2/3 ρ 2 D = 2 1 − 2 q D = \frac{2}{1-2q} D = 1 − 2 q 2 n 1 : = [ 16 M 2 / 3 ρ 2 ( 1 − 2 q ) log ( 16 M 2 / 3 ρ 2 ( 1 − 2 q ) ) ] 1 1 − 2 q n_1 := \left[\frac{16M^{2/3}}{\rho^2(1-2q)}\log\left(\frac{16M^{2/3}}{\rho^2(1-2q)}\right)\right]^{\frac{1}{1-2q}} n 1 := [ ρ 2 ( 1 − 2 q ) 16 M 2/3 log ( ρ 2 ( 1 − 2 q ) 16 M 2/3 ) ] 1 − 2 q 1
exp ( − ρ 2 n 1 − 2 q 16 M 2 / 3 ) ≤ n − 1 2 for any n ≥ n 1 . (39) \exp\left(-\frac{\rho^2 n^{1-2q}}{16M^{2/3}}\right) \le n^{-\frac{1}{2}}\qquad\text{for any }n\ge n_1. \tag*{(39)} exp ( − 16 M 2/3 ρ 2 n 1 − 2 q ) ≤ n − 2 1 for any n ≥ n 1 . (39) Furthermore, for n 2 : = [ 16 M 2 / 3 ρ 2 log ( 2 d 2 + 2 ( K + 2 ) ρ 3 M ) ] 1 1 − 2 q ≥ [ 16 M 2 / 3 ρ 2 log ( 2 d 2 + 2 ( K + 2 ) ρ 3 d 3 / 2 M ) ] 1 1 − 2 q n_2 := \left[\frac{16M^{2/3}}{\rho^2}\log\left(\frac{2^{\frac{d}{2}+2}(K+2)\rho^3}{M}\right)\right]^{\frac{1}{1-2q}} \ge\left[\frac{16M^{2/3}}{\rho^2}\log\left(\frac{2^{\frac{d}{2}+2}(K+2)\rho^3}{d^{3/2}M}\right)\right]^{\frac{1}{1-2q}} n 2 := [ ρ 2 16 M 2/3 log ( M 2 2 d + 2 ( K + 2 ) ρ 3 ) ] 1 − 2 q 1 ≥ [ ρ 2 16 M 2/3 log ( d 3/2 M 2 2 d + 2 ( K + 2 ) ρ 3 ) ] 1 − 2 q 1
exp ( − ρ 2 n 1 − 2 q 16 M 2 / 3 ) ≤ d 3 2 M 2 d 2 + 2 ( K + 2 ) ρ 3 for any n ≥ n 2 . (40) \exp\left(-\frac{\rho^2 n^{1-2q}}{16M^{2/3}}\right) \le\frac{d^{\frac{3}{2}}M}{2^{\frac{d}{2}+2}(K+2)\rho^3}\qquad\text{for any }n\ge n_2. \tag*{(40)} exp ( − 16 M 2/3 ρ 2 n 1 − 2 q ) ≤ 2 2 d + 2 ( K + 2 ) ρ 3 d 2 3 M for any n ≥ n 2 . (40) Consequently, for any n ∈ N n\in\mathbb{N} n ∈ N n ≥ max { n 1 , n 2 } n\ge\max\{n_1,n_2\} n ≥ max { n 1 , n 2 }
2 d 2 + 2 ( K + 2 ) exp ( − ρ 2 n 1 − 2 q 8 M 2 3 ) ≤ 2 d 2 + 2 ( K + 2 ) n 1 2 exp ( − ρ 2 n 1 − 2 q 16 M 2 3 ) ≤ M d 3 2 n 1 2 ρ 3 , 2^{\frac{d}{2}+2}(K+2)\exp\left(-\frac{\rho^2n^{1-2q}}{8M^{\frac{2}{3}}}\right)
\le\frac{2^{\frac{d}{2}+2}(K+2)}{n^{\frac{1}{2}}}\exp\left(-\frac{\rho^2n^{1-2q}}{16M^{\frac{2}{3}}}\right)
\le\frac{Md^{\frac{3}{2}}}{n^{\frac{1}{2}\rho^3}}, 2 2 d + 2 ( K + 2 ) exp ( − 8 M 3 2 ρ 2 n 1 − 2 q ) ≤ n 2 1 2 2 d + 2 ( K + 2 ) exp ( − 16 M 3 2 ρ 2 n 1 − 2 q ) ≤ n 2 1 ρ 3 M d 2 3 , which in combination with (9) proves the desired statement. □ \square □
Proof of theo:PropChiSq
It is sufficient to verify that π n ( x ) K π n ( x ) ≤ 2 e \frac{\pi_n(x)}{\mathcal{K}_{\pi_n}(x)}\le2e K π n ( x ) π n ( x ) ≤ 2 e x ∈ R d x\in\mathbb{R}^d x ∈ R d n ∈ N n\in\mathbb{N} n ∈ N
Firstly, we prove that π n ( x ) K π n ( x ) ≤ e Z ~ n k Z n \frac{\pi_n(x)}{\mathcal{K}_{\pi_n}(x)}\le\frac{e^{\widetilde{Z}_n^k}}{Z_n} K π n ( x ) π n ( x ) ≤ Z n e Z n k x ∈ R d x\in\mathbb{R}^d x ∈ R d n ∈ N n\in\mathbb{N} n ∈ N ω n ∗ \omega_n^* ω n ∗ π n ( x ) K π n ( x ) = Z ~ n k Z n ω n ∗ ( x ) \frac{\pi_n(x)}{\mathcal{K}_{\pi_n}(x)}=\frac{\widetilde{Z}_n^k}{Z_n}\omega_n^*(x) K π n ( x ) π n ( x ) = Z n Z n k ω n ∗ ( x ) x ∈ ⋃ j = 1 K V n j , R , q x\in\bigcup_{j=1}^K V_n^{j,R,q} x ∈ ⋃ j = 1 K V n j , R , q k ∈ { 1 , … , K } k\in\{1,\ldots,K\} k ∈ { 1 , … , K } x ∈ V n k , R , q x\in V_n^{k,R,q} x ∈ V n k , R , q
ω n ∗ ( x ) = e − n I n ( x ) π n ( k ) exp ( − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 ) ( 1 + ∑ j = 1 , j ≠ k K π n ( j ) π n ( k ) exp ( − n 2 ∥ x − x n ( j ) ∥ H n ( j ) 2 + n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 ) ) , \omega_n^*(x)=\frac{e^{-nI_n(x)}}{\pi_n^{(k)}\exp\left(-\frac{n}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2\right)\left(1+\sum_{j=1,j\ne k}^K\frac{\pi_n^{(j)}}{\pi_n^{(k)}}\exp\left(-\frac{n}{2}\lVert x-x_n^{(j)}\rVert_{H_n^{(j)}}^2+\frac{n}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2\right)\right)}, ω n ∗ ( x ) = π n ( k ) exp ( − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 ) ( 1 + ∑ j = 1 , j = k K π n ( k ) π n ( j ) exp ( − 2 n ∥ x − x n ( j ) ∥ H n ( j ) 2 + 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 ) ) e − n I n ( x ) , which implies
ω n ∗ ( x ) ≤ exp ( − n ( I n ( x ) − I n ( x n ( k ) ) − 1 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 ) ) , \omega_n^*(x)\le\exp\left(-n\left(I_n(x)-I_n(x_n^{(k)})-\frac{1}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2\right)\right), ω n ∗ ( x ) ≤ exp ( − n ( I n ( x ) − I n ( x n ( k ) ) − 2 1 ∥ x − x n ( k ) ∥ H n ( k ) 2 ) ) , where we also used the definition of π n ( k ) \pi_n^{(k)} π n ( k ) x ∈ V n k , R , q x\in V_n^{k,R,q} x ∈ V n k , R , q I n I_n I n
I n ( x ) = I n ( x n ( k ) ) + ⟨ ∇ I n ( x n ( k ) ) , x − x n ( k ) ⟩ + 1 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 + R n ( k ) ( x ) , I_n(x)=I_n(x_n^{(k)})+\langle\nabla I_n(x_n^{(k)}),x-x_n^{(k)}\rangle+\frac{1}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2+R_n^{(k)}(x), I n ( x ) = I n ( x n ( k ) ) + ⟨ ∇ I n ( x n ( k ) ) , x − x n ( k ) ⟩ + 2 1 ∥ x − x n ( k ) ∥ H n ( k ) 2 + R n ( k ) ( x ) , with R n ( k ) ( x ) = 1 6 D 3 I n ( ξ x ) ( x − x n ( k ) , x − x n ( k ) , x − x n ( k ) ) R_n^{(k)}(x)=\frac{1}{6}D^3I_n(\xi_x)(x-x_n^{(k)},x-x_n^{(k)},x-x_n^{(k)}) R n ( k ) ( x ) = 6 1 D 3 I n ( ξ x ) ( x − x n ( k ) , x − x n ( k ) , x − x n ( k ) ) ξ x ∈ V n k , R , q \xi_x\in V_n^{k,R,q} ξ x ∈ V n k , R , q ∇ I n ( x n ( k ) ) = 0 \nabla I_n(x_n^{(k)})=0 ∇ I n ( x n ( k ) ) = 0 x n ( k ) x_n^{(k)} x n ( k ) Assumption 1 (iii) and Remark 3.2, we obtain
ω n ∗ ( x ) ≤ e n ∣ R n ( k ) ( x ) ∣ ≤ exp ( M n ρ 3 ∥ x − x n ( k ) ∥ H n ( k ) 3 ) . (41) \omega_n^*(x)\le e^{n|R_n^{(k)}(x)|}\le\exp\left(\frac{Mn}{\rho^3}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^3\right). \tag*{(41)} ω n ∗ ( x ) ≤ e n ∣ R n ( k ) ( x ) ∣ ≤ exp ( ρ 3 M n ∥ x − x n ( k ) ∥ H n ( k ) 3 ) . (41) By the definition of V n k , R , q V_n^{k,R,q} V n k , R , q R = ρ 2 / M 2 3 R=\rho^2/M^{\frac{2}{3}} R = ρ 2 / M 3 2 x ∈ V n k , R , q x\in V_n^{k,R,q} x ∈ V n k , R , q
w n ∗ ( x ) ≤ e n 1 − 3 q ≤ e , (42) w_n^*(x)\le e^{n^{1-3q}}\le e, \tag*{(42)} w n ∗ ( x ) ≤ e n 1 − 3 q ≤ e , (42) where the last inequality follows by q ∈ [ 1 / 3 , 1 / 2 ) q\in[1/3,1/2) q ∈ [ 1/3 , 1/2 ) x ∈ V ‾ n R , q x\in\overline{V}_n^{R,q} x ∈ V n R , q
w n ∗ ( x ) ≤ ∑ k = 1 K π n ( k ) exp ( − n δ ∥ x − x n ( k ) ∥ H n ( k ) 2 ) ∑ j = 1 K π n ( j ) exp ( − n 2 ∥ x − x n ( j ) ∥ H n ( j ) 2 ) ≤ 1 , w_n^*(x)\le\frac{\sum_{k=1}^{K}\pi_n^{(k)}\exp\left(-n\delta\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2\right)}{\sum_{j=1}^{K}\pi_n^{(j)}\exp\left(-\frac{n}{2}\lVert x-x_n^{(j)}\rVert_{H_n^{(j)}}^2\right)}\le1, w n ∗ ( x ) ≤ ∑ j = 1 K π n ( j ) exp ( − 2 n ∥ x − x n ( j ) ∥ H n ( j ) 2 ) ∑ k = 1 K π n ( k ) exp ( − n δ ∥ x − x n ( k ) ∥ H n ( k ) 2 ) ≤ 1 , where the last inequality follows by δ ≥ 1 / 2 \delta\ge1/2 δ ≥ 1/2 exp ( − n δ ∥ x − x n ( j ) ∥ H n ( j ) 2 ) ≤ exp ( − n 2 ∥ x − x n ( j ) ∥ H n ( j ) 2 ) \exp\left(-n\delta\lVert x-x_n^{(j)}\rVert_{H_n^{(j)}}^2\right)\le\exp\left(-\frac{n}{2}\lVert x-x_n^{(j)}\rVert_{H_n^{(j)}}^2\right) exp ( − n δ ∥ x − x n ( j ) ∥ H n ( j ) 2 ) ≤ exp ( − 2 n ∥ x − x n ( j ) ∥ H n ( j ) 2 ) j ∈ { 1 , … , K } j\in\{1,\ldots,K\} j ∈ { 1 , … , K } x ∈ R d x\in\mathbb{R}^d x ∈ R d π n ( x ) K π n K ( x ) ≤ e Z ~ n K Z n \frac{\pi_n(x)}{\mathcal{K}_{\pi_n}^{K}(x)}\le\frac{e\widetilde{Z}_n^K}{Z_n} K π n K ( x ) π n ( x ) ≤ Z n e Z n K K n , 1 \mathcal{K}_{n,1} K n , 1
∣ Z n Z ~ n K − 1 ∣ = 1 Z ~ n K ∣ Z ~ n K − Z n ∣ ≤ 1 Z ~ n K ∫ R d ∣ e − n I n ( x ) − ∑ j = 1 K π n ( j ) exp ( − n 2 ∥ x − x n ( j ) ∥ H n ( j ) 2 ) ∣ d x = K n , 1 2 , \left|\frac{Z_n}{\widetilde{Z}_n^K}-1\right|=\frac{1}{\widetilde{Z}_n^K}\left|\widetilde{Z}_n^K-Z_n\right|\le\frac{1}{\widetilde{Z}_n^K}\int_{\mathbb{R}^d}\left|e^{-nI_n(x)}-\sum_{j=1}^{K}\pi_n^{(j)}\exp\left(-\frac{n}{2}\lVert x-x_n^{(j)}\rVert_{H_n^{(j)}}^2\right)\right|\mathrm{d}x=\frac{\mathcal{K}_{n,1}}{2}, Z n K Z n − 1 = Z n K 1 Z n K − Z n ≤ Z n K 1 ∫ R d e − n I n ( x ) − j = 1 ∑ K π n ( j ) exp ( − 2 n ∥ x − x n ( j ) ∥ H n ( j ) 2 ) d x = 2 K n , 1 , such that
1 − K n , 1 2 ≤ Z n Z ~ n K implies Z ~ n K Z n ≤ ( 1 − K n , 1 2 ) − 1 = 2 2 − K n , 1 . (43) 1-\frac{\mathcal{K}_{n,1}}{2}\le\frac{Z_n}{\widetilde{Z}_n^K}\quad\text{implies}\quad\frac{\widetilde{Z}_n^K}{Z_n}\le\left(1-\frac{\mathcal{K}_{n,1}}{2}\right)^{-1}=\frac{2}{2-\mathcal{K}_{n,1}}. \tag*{(43)} 1 − 2 K n , 1 ≤ Z n K Z n implies Z n Z n K ≤ ( 1 − 2 K n , 1 ) − 1 = 2 − K n , 1 2 . (43) For n n n (11) it follows
4 e d / 3 M ρ 3 n 1 / 2 ≤ 1 2 and 2 1 + d 2 ( K + 2 ) exp ( − ρ 2 4 M 2 / 3 n 1 − 2 q ) ≤ 1 2 \frac{4e^{d/3}M}{\rho^3n^{1/2}}\le\frac{1}{2}\quad\text{and}\quad2^{1+\frac{d}{2}}(K+2)\exp\left(-\frac{\rho^2}{4M^{2/3}}n^{1-2q}\right)\le\frac{1}{2} ρ 3 n 1/2 4 e d /3 M ≤ 2 1 and 2 1 + 2 d ( K + 2 ) exp ( − 4 M 2/3 ρ 2 n 1 − 2 q ) ≤ 2 1 such that by (b) of Lemma A.3 (with δ ≥ 1 / 2 \delta\ge1/2 δ ≥ 1/2 K n , 1 ≤ 1 \mathcal{K}_{n,1}\le1 K n , 1 ≤ 1 n n n Z ~ n K Z n ≤ 2 \frac{\widetilde{Z}_n^K}{Z_n}\le2 Z n Z n K ≤ 2 π n ( x ) K π n K ( x ) ≤ 2 e \frac{\pi_n(x)}{\mathcal{K}_{\pi_n}^K(x)}\le2e K π n K ( x ) π n ( x ) ≤ 2 e
Proofs of Laplace-based importance sampling
We provide proofs that have been postponed in Section 4.1.
Proof of Proposition 4.1.
Proof. For any k ∈ { 1 , … , K } k\in\{1,\ldots,K\} k ∈ { 1 , … , K } i ∈ { 1 , … , N } i\in\{1,\ldots,N\} i ∈ { 1 , … , N }
A i , n : = { min j = 1 , … , K ∥ X i , n − x n ( k ) ∥ H n ( k ) 2 > R n − 2 q } = { X i , n ∈ ⋂ j = 1 K ( V n j , R , q ) c } ⊆ { X i , n ∈ ( V n k , R , q ) c } . A_{i,n}:=\left\{\min_{j=1,\ldots,K}\lVert X_{i,n}-x_n^{(k)}\rVert_{H_n^{(k)}}^2>Rn^{-2q}\right\}=\left\{X_{i,n}\in\bigcap_{j=1}^{K}(V_n^{j,R,q})^c\right\}\subseteq\left\{X_{i,n}\in(V_n^{k,R,q})^c\right\}. A i , n := { j = 1 , … , K min ∥ X i , n − x n ( k ) ∥ H n ( k ) 2 > R n − 2 q } = { X i , n ∈ j = 1 ⋂ K ( V n j , R , q ) c } ⊆ { X i , n ∈ ( V n k , R , q ) c } . Consequently, we obtain
P ( A i , n ) = ∫ ⋂ k = 1 K ( V n k , R , q ) c 1 Z ~ n K ∑ k = 1 K π n ( k ) e − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 d x = ∑ k = 1 K π n ( k ) Z ~ n K ∫ ⋂ j = 1 K ( V n j , R , q ) c e − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 d x \mathbb{P}(A_{i,n})=\int_{\bigcap_{k=1}^{K}(V_n^{k,R,q})^c}\frac{1}{\widetilde{Z}_n^K}\sum_{k=1}^{K}\pi_n^{(k)}e^{-\frac{n}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2}\,\mathrm{d}x=\sum_{k=1}^{K}\frac{\pi_n^{(k)}}{\widetilde{Z}_n^K}\int_{\bigcap_{j=1}^{K}(V_n^{j,R,q})^c}e^{-\frac{n}{2}\lVert x-x_n^{(k)}\rVert_{H_n^{(k)}}^2}\,\mathrm{d}x P ( A i , n ) = ∫ ⋂ k = 1 K ( V n k , R , q ) c Z n K 1 k = 1 ∑ K π n ( k ) e − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 d x = k = 1 ∑ K Z n K π n ( k ) ∫ ⋂ j = 1 K ( V n j , R , q ) c e − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 d x and, by Lemma A.2 we have
∫ ⋂ j = 1 K ( V n j , R , q ) c e − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 d x ≤ ∫ ( V n k , R , q ) c e − n 2 ∥ x − x n ( k ) ∥ H n ( k ) 2 d x ≤ e − R n 1 − 2 q 4 ( 4 π n ) d 2 det ( ( H n ( k ) ) − 1 ) 1 / 2 . \int_{\bigcap_{j=1}^{K}(V_{n}^{j,R,q})^{c}} \mathrm{e}^{-\frac{n}{2}\lVert x-x_{n}^{(k)}\rVert_{H_{n}^{(k)}}^{2}} \, \mathrm{d}x \leq\int_{(V_{n}^{k,R,q})^{c}} \mathrm{e}^{-\frac{n}{2}\lVert x-x_{n}^{(k)}\rVert_{H_{n}^{(k)}}^{2}} \, \mathrm{d}x \leq\mathrm{e}^{-\frac{R n^{1-2q}}{4}}\left(\frac{4\pi}{n}\right)^{\frac{d}{2}}\det\left((H_{n}^{(k)})^{-1}\right)^{1/2}. ∫ ⋂ j = 1 K ( V n j , R , q ) c e − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 d x ≤ ∫ ( V n k , R , q ) c e − 2 n ∥ x − x n ( k ) ∥ H n ( k ) 2 d x ≤ e − 4 R n 1 − 2 q ( n 4 π ) 2 d det ( ( H n ( k ) ) − 1 ) 1/2 . By Z ~ n K = ( 2 π n ) d / 2 ∑ j = 1 K det ( ( H n ( j ) ) − 1 ) 1 / 2 π n ( j ) \widetilde{Z}_{n}^{K}=\left(\frac{2\pi}{n}\right)^{d/2}\sum_{j=1}^{K}\det\left((H_{n}^{(j)})^{-1}\right)^{1/2}\pi_{n}^{(j)} Z n K = ( n 2 π ) d /2 ∑ j = 1 K det ( ( H n ( j ) ) − 1 ) 1/2 π n ( j )
P ( A i , n ) ≤ ∑ k = 1 K π n ( k ) Z ~ n K e − R n 1 − 2 q 4 ( 4 π n ) d 2 det ( ( H n ( k ) ) − 1 ) 1 / 2 = 2 d 2 − R n 1 − 2 q 4 , \mathbb{P}(A_{i,n})\leq\sum_{k=1}^{K}\frac{\pi_{n}^{(k)}}{\widetilde{Z}_{n}^{K}}\mathrm{e}^{-\frac{R n^{1-2q}}{4}}\left(\frac{4\pi}{n}\right)^{\frac{d}{2}}\det\left((H_{n}^{(k)})^{-1}\right)^{1/2}=2^{d}2^{-\frac{R n^{1-2q}}{4}}, P ( A i , n ) ≤ k = 1 ∑ K Z n K π n ( k ) e − 4 R n 1 − 2 q ( n 4 π ) 2 d det ( ( H n ( k ) ) − 1 ) 1/2 = 2 d 2 − 4 R n 1 − 2 q , such that lim n → ∞ P ( A i , n ) = 0 \lim_{n\to\infty}\mathbb{P}(A_{i,n})=0 lim n → ∞ P ( A i , n ) = 0 G n , N = ⋂ i = 1 N A i , n c G_{n,N}=\bigcap_{i=1}^{N}A_{i,n}^{c} G n , N = ⋂ i = 1 N A i , n c A 1 , n , … , A N , n A_{1,n},\ldots,A_{N,n} A 1 , n , … , A N , n
lim n → ∞ P ( G n , N ) = lim n → ∞ ∏ i = 1 N P ( A i , n c ) = ∏ i = 1 N lim n → ∞ P ( A i , n c ) = 1 , \lim_{n\to\infty}\mathbb{P}(G_{n,N})=\lim_{n\to\infty}\prod_{i=1}^{N}\mathbb{P}(A_{i,n}^{c})=\prod_{i=1}^{N}\lim_{n\to\infty}\mathbb{P}(A_{i,n}^{c})=1, n → ∞ lim P ( G n , N ) = n → ∞ lim i = 1 ∏ N P ( A i , n c ) = i = 1 ∏ N n → ∞ lim P ( A i , n c ) = 1 , which finishes the proof.
Proof of Lemma 4.1.
Before we prove Lemma 4.1 we need some auxiliary results.
Lemma B.1. Under Assumption 1 and Assumption 4 we have lim n → ∞ π n ( k ) = π 0 ( x ∗ ( k ) ) \lim_{n\to\infty}\pi_{n}^{(k)}=\pi_{0}(x_{*}^{(k)}) lim n → ∞ π n ( k ) = π 0 ( x ∗ ( k ) )
Proof. Observe that
π n ( k ) = exp ( − n Φ ( x n ( k ) ) ) π 0 ( x n ( k ) ) = exp ( − n ( Φ ( x n ( k ) ) − Φ ( x ∗ ( k ) ) + Φ ( x ∗ ( k ) ) ) ) π 0 ( x n ( k ) ) . \pi_{n}^{(k)}=\exp\left(-n\Phi(x_{n}^{(k)})\right)\pi_{0}(x_{n}^{(k)})=\exp\left(-n\left(\Phi(x_{n}^{(k)})-\Phi(x_{*}^{(k)})+\Phi(x_{*}^{(k)})\right)\right)\pi_{0}(x_{n}^{(k)}). π n ( k ) = exp ( − n Φ ( x n ( k ) ) ) π 0 ( x n ( k ) ) = exp ( − n ( Φ ( x n ( k ) ) − Φ ( x ∗ ( k ) ) + Φ ( x ∗ ( k ) ) ) ) π 0 ( x n ( k ) ) . Due to ∇ 2 Φ ( x ∗ ( k ) ) > 0 \nabla^{2}\Phi(x_{*}^{(k)})>0 ∇ 2 Φ ( x ∗ ( k ) ) > 0 Φ ∈ C 3 ( R d , [ 0 , ∞ ) ) \Phi\in C^{3}(\mathbb{R}^{d},[0,\infty)) Φ ∈ C 3 ( R d , [ 0 , ∞ )) Φ \Phi Φ x ∗ ( k ) x_{*}^{(k)} x ∗ ( k ) c ^ > 0 \widehat{c}>0 c > 0
Φ ( x n ( k ) ) − Φ ( x ∗ ( k ) ) ≤ 1 2 c ^ ∥ ∇ Φ ( x n ( k ) ) ∥ 2 . \Phi(x_{n}^{(k)})-\Phi(x_{*}^{(k)})\leq\frac{1}{2\widehat{c}}\lVert\nabla\Phi(x_{n}^{(k)})\rVert^{2}. Φ ( x n ( k ) ) − Φ ( x ∗ ( k ) ) ≤ 2 c 1 ∥ ∇Φ ( x n ( k ) ) ∥ 2 . Moreover, since ∇ Φ ( x n ( k ) ) = 1 n ∇ log π 0 ( x n ( k ) ) \nabla\Phi(x_{n}^{(k)})=\frac{1}{n}\nabla\log\pi_{0}(x_{n}^{(k)}) ∇Φ ( x n ( k ) ) = n 1 ∇ log π 0 ( x n ( k ) )
Φ ( x n ( k ) ) − Φ ( x ∗ ( k ) ) ≤ 1 2 c ^ n 2 ∥ ∇ log π 0 ( x n ( k ) ) ∥ 2 . \Phi(x_{n}^{(k)})-\Phi(x_{*}^{(k)})\leq\frac{1}{2\widehat{c}n^{2}}\lVert\nabla\log\pi_{0}(x_{n}^{(k)})\rVert^{2}. Φ ( x n ( k ) ) − Φ ( x ∗ ( k ) ) ≤ 2 c n 2 1 ∥ ∇ log π 0 ( x n ( k ) ) ∥ 2 . Since lim n → ∞ x n ( k ) = x ∗ ( k ) \lim_{n\to\infty}x_{n}^{(k)}=x_{*}^{(k)} lim n → ∞ x n ( k ) = x ∗ ( k ) π 0 ∈ C 2 ( R d , ( 0 , ∞ ) ) \pi_{0}\in C^{2}(\mathbb{R}^{d},(0,\infty)) π 0 ∈ C 2 ( R d , ( 0 , ∞ )) lim n → ∞ ∥ ∇ log π 0 ( x n ( k ) ) ∥ = ∥ ∇ log π 0 ( x ∗ ( k ) ) ∥ \lim_{n\to\infty}\lVert\nabla\log\pi_{0}(x_{n}^{(k)})\rVert=\lVert\nabla\log\pi_{0}(x_{*}^{(k)})\rVert lim n → ∞ ∥ ∇ log π 0 ( x n ( k ) )∥ = ∥ ∇ log π 0 ( x ∗ ( k ) )∥ C > 0 C>0 C > 0 n n n n 0 ∈ N n_{0}\in\mathbb{N} n 0 ∈ N n ≥ n 0 n\geq n_{0} n ≥ n 0
Φ ( x n ( k ) ) − Φ ( x ∗ ( k ) ) ≤ C n − 2 . \Phi(x_{n}^{(k)})-\Phi(x_{*}^{(k)})\leq Cn^{-2}. Φ ( x n ( k ) ) − Φ ( x ∗ ( k ) ) ≤ C n − 2 . Hence, since Φ ( x ∗ ( k ) ) = 0 \Phi(x_{*}^{(k)})=0 Φ ( x ∗ ( k ) ) = 0 π 0 ∈ C 2 ( R d , ( 0 , ∞ ) ) \pi_{0}\in C^{2}(\mathbb{R}^{d},(0,\infty)) π 0 ∈ C 2 ( R d , ( 0 , ∞ )) lim n → ∞ π n ( k ) = π 0 ( x ∗ ( k ) ) . \lim_{n\to\infty}\pi_{n}^{(k)}=\pi_{0}(x_{*}^{(k)}). lim n → ∞ π n ( k ) = π 0 ( x ∗ ( k ) ) .
We provide two inequalities w.r.t. random variables that hold pointwise restricted to a suitable measurable subset of the probability space.
Proposition B.1. For R > 0 R > 0 R > 0 q ∈ [ 0 , 1 / 2 ) q \in[0,1/2) q ∈ [ 0 , 1/2 ) ρ , M ∈ ( 0 , ∞ ) \rho,M \in(0,\infty) ρ , M ∈ ( 0 , ∞ ) Assumption 1 and Assumption 4 be satisfied. For N ∈ N N \in\mathbb{N} N ∈ N X 1 , n , … , X N , n X_{1,n},\ldots,X_{N,n} X 1 , n , … , X N , n i d L π n K K \mathrm{id}\ L^K_{\pi_n^K} id L π n K K G n , N G_{n,N} G n , N (16) . Furthermore, for any i ∈ { 1 , … , N } i \in\{1,\ldots,N\} i ∈ { 1 , … , N }
κ i , n : = argmin ℓ ∈ { 1 , … , K } ∥ X i , n − x n ( ℓ ) ∥ H n ( ℓ ) 2 . (44) \kappa_{i,n} := \operatorname*{argmin}_{\ell\in\{1,\ldots,K\}} \left\|X_{i,n}-x_n^{(\ell)}\right\|_{H_n^{(\ell)}}^2. \tag*{(44)} κ i , n := ℓ ∈ { 1 , … , K } argmin X i , n − x n ( ℓ ) H n ( ℓ ) 2 . (44) Then, there are numbers c 1 , c 2 > 0 c_1,c_2 > 0 c 1 , c 2 > 0 n 1 ∈ N n_1 \in\mathbb{N} n 1 ∈ N i ∈ { 1 , … , N } i \in\{1,\ldots,N\} i ∈ { 1 , … , N } n ∈ N n \in\mathbb{N} n ∈ N n ≥ n 1 n \ge n_1 n ≥ n 1 G n , N G_{n,N} G n , N
∑ ℓ = 1 , ℓ ≠ κ i , n K π n ( ℓ ) exp ( − n 2 ∥ X i , n − x n ( ℓ ) ∥ H n ( ℓ ) 2 ) π n ( κ i , n ) exp ( − n 2 ∥ X i , n − x n ( κ i , n ) ∥ H n ( κ i , n ) 2 ) ≤ c 1 e − c 2 n . (45) \frac{\sum_{\ell=1,\ell\ne\kappa_{i,n}}^K \pi_n^{(\ell)}\exp\left(-\frac{n}{2}\left\|X_{i,n}-x_n^{(\ell)}\right\|_{H_n^{(\ell)}}^2\right)}
{\pi_n^{(\kappa_{i,n})}\exp\left(-\frac{n}{2}\left\|X_{i,n}-x_n^{(\kappa_{i,n})}\right\|_{H_n^{(\kappa_{i,n})}}^2\right)}
\le c_1e^{-c_2n}. \tag*{(45)} π n ( κ i , n ) exp ( − 2 n X i , n − x n ( κ i , n ) H n ( κ i , n ) 2 ) ∑ ℓ = 1 , ℓ = κ i , n K π n ( ℓ ) exp ( − 2 n X i , n − x n ( ℓ ) H n ( ℓ ) 2 ) ≤ c 1 e − c 2 n . (45) Moreover, for w n ∗ w_n^* w n ∗ (15) ) there are numbers c 3 > 0 c_3 > 0 c 3 > 0 n 2 ∈ N n_2 \in\mathbb{N} n 2 ∈ N i ∈ { 1 , … , N } i \in\{1,\ldots,N\} i ∈ { 1 , … , N } n ∈ N n \in\mathbb{N} n ∈ N n ≥ n 2 n \ge n_2 n ≥ n 2 G n , N G_{n,N} G n , N
1 − c 3 n 1 − 3 q ≤ w n ∗ ( X i , n ) ≤ 1 + c 3 n 1 − 3 q . (46) 1-c_3n^{1-3q}\le w_n^*(X_{i,n})\le1+c_3n^{1-3q}. \tag*{(46)} 1 − c 3 n 1 − 3 q ≤ w n ∗ ( X i , n ) ≤ 1 + c 3 n 1 − 3 q . (46) Proof. We start with proving (45) . For this define
γ : = max j , ℓ ∈ { 1 , … , K } j ≠ ℓ π 0 ( x ∗ ( ℓ ) ) π 0 ( x ∗ ( j ) ) and γ n : = max j , ℓ ∈ { 1 , … , K } j ≠ ℓ π n ( ℓ ) π n ( j ) . \gamma:= \max_{\substack{j,\ell\in\{1,\ldots,K\}\\j\ne\ell}}\frac{\pi_0(x_*^{(\ell)})}{\pi_0(x_*^{(j)})}
\qquad\text{and}\qquad
\gamma_n := \max_{\substack{j,\ell\in\{1,\ldots,K\}\\j\ne\ell}}\frac{\pi_n^{(\ell)}}{\pi_n^{(j)}}. γ := j , ℓ ∈ { 1 , … , K } j = ℓ max π 0 ( x ∗ ( j ) ) π 0 ( x ∗ ( ℓ ) ) and γ n := j , ℓ ∈ { 1 , … , K } j = ℓ max π n ( j ) π n ( ℓ ) . Note that, by Lemma B.1 we have lim n → ∞ γ n = γ \lim_{n\to\infty}\gamma_n=\gamma lim n → ∞ γ n = γ Assumption 1 (part (i)) there is a constant α > 0 \alpha>0 α > 0
min j , ℓ ∈ { 1 , … , K } j ≠ ℓ ∥ x n ( j ) − x n ( ℓ ) ∥ H n ( ℓ ) ≥ α . \min_{\substack{j,\ell\in\{1,\ldots,K\}\\j\ne\ell}}\left\|x_n^{(j)}-x_n^{(\ell)}\right\|_{H_n^{(\ell)}}\ge\alpha. j , ℓ ∈ { 1 , … , K } j = ℓ min x n ( j ) − x n ( ℓ ) H n ( ℓ ) ≥ α . Now, choose n 1 ∈ N n_1\in\mathbb{N} n 1 ∈ N n ∈ N n\in\mathbb{N} n ∈ N n ≥ n 1 n\ge n_1 n ≥ n 1 γ n ≤ 2 γ \gamma_n\le2\gamma γ n ≤ 2 γ α > 4 R n − q \alpha>4\sqrt{Rn^{-q}} α > 4 R n − q i ∈ { 1 , … , N } i\in\{1,\ldots,N\} i ∈ { 1 , … , N } ( X i , n (X_{i,n} ( X i , n κ i , n ) \kappa_{i,n}) κ i , n ) G n , N G_{n,N} G n , N
min ℓ ∈ { 1 , … , K } ∥ X i , n − x n ( ℓ ) ∥ H n ( ℓ ) 2 = ∥ X i , n − x n ( κ i , n ) ∥ H n ( κ i , n ) 2 ≤ R n − 2 q \min_{\ell\in\{1,\ldots,K\}}\left\|X_{i,n}-x_n^{(\ell)}\right\|_{H_n^{(\ell)}}^2
=\left\|X_{i,n}-x_n^{(\kappa_{i,n})}\right\|_{H_n^{(\kappa_{i,n})}}^2\le Rn^{-2q} ℓ ∈ { 1 , … , K } min X i , n − x n ( ℓ ) H n ( ℓ ) 2 = X i , n − x n ( κ i , n ) H n ( κ i , n ) 2 ≤ R n − 2 q and, consequently, for any ℓ ∈ { 1 , … , K } \ell\in\{1,\ldots,K\} ℓ ∈ { 1 , … , K } ℓ ≠ κ i , n \ell\ne\kappa_{i,n} ℓ = κ i , n
∥ X i , n − x n ( ℓ ) ∥ H n ( ℓ ) 2 ≥ ( ∥ X i , n − x n ( κ i , n ) ∥ H n ( ℓ ) − ∥ x n ( κ i , n ) − x n ( ℓ ) ∥ H n ( ℓ ) ) 2 ≥ ( α − R n − q ) 2 . \left\|X_{i,n}-x_n^{(\ell)}\right\|_{H_n^{(\ell)}}^2\ge\left(\left\|X_{i,n}-x_n^{(\kappa_{i,n})}\right\|_{H_n^{(\ell)}}-\left\|x_n^{(\kappa_{i,n})}-x_n^{(\ell)}\right\|_{H_n^{(\ell)}}\right)^2\ge\left(\alpha-\sqrt{Rn^{-q}}\right)^2. X i , n − x n ( ℓ ) H n ( ℓ ) 2 ≥ ( X i , n − x n ( κ i , n ) H n ( ℓ ) − x n ( κ i , n ) − x n ( ℓ ) H n ( ℓ ) ) 2 ≥ ( α − R n − q ) 2 . Hence, for n ≥ n 1 n\ge n_1 n ≥ n 1
∑ ℓ = 1 , ℓ ≠ κ i , n K π n ( ℓ ) exp ( − n 2 ∥ X i , n − x n ( ℓ ) ∥ H n ( ℓ ) 2 ) π n ( κ i , n ) exp ( − n 2 ∥ X i , n − x n ( κ i , n ) ∥ H n ( κ i , n ) 2 ) = ∑ ℓ = 1 , ℓ ≠ κ i , n K π n ( ℓ ) π n ( κ i , n ) exp ( − n 2 ( ∥ X i , n − x n ( ℓ ) ∥ H n ( ℓ ) 2 − ∥ X i , n − x n ( κ i , n ) ∥ H n ( κ i , n ) 2 ) ) ≤ ∑ ℓ = 1 , ℓ ≠ κ i , n K γ n exp ( − n 2 ( ( α − R n − q ) 2 − R n − 2 q ) ) ≤ ∑ ℓ = 1 , ℓ ≠ κ i , n K 2 γ exp ( − α n ( α 2 − R n − q ) ) ≤ 2 ( K − 1 ) γ exp ( − α 2 4 n ) , \begin{aligned}
\frac{\sum_{\ell=1,\ell\ne\kappa_{i,n}}^K \pi_n^{(\ell)}\exp\left(-\frac{n}{2}\left\|X_{i,n}-x_n^{(\ell)}\right\|_{H_n^{(\ell)}}^2\right)}
{\pi_n^{(\kappa_{i,n})}\exp\left(-\frac{n}{2}\left\|X_{i,n}-x_n^{(\kappa_{i,n})}\right\|_{H_n^{(\kappa_{i,n})}}^2\right)}
&=\sum_{\ell=1,\ell\ne\kappa_{i,n}}^K\frac{\pi_n^{(\ell)}}{\pi_n^{(\kappa_{i,n})}}\exp\left(-\frac{n}{2}\left(\left\|X_{i,n}-x_n^{(\ell)}\right\|_{H_n^{(\ell)}}^2-\left\|X_{i,n}-x_n^{(\kappa_{i,n})}\right\|_{H_n^{(\kappa_{i,n})}}^2\right)\right)\\
&\le\sum_{\ell=1,\ell\ne\kappa_{i,n}}^K\gamma_n\exp\left(-\frac{n}{2}\left((\alpha-\sqrt{Rn^{-q}})^2-Rn^{-2q}\right)\right)\\
&\le\sum_{\ell=1,\ell\ne\kappa_{i,n}}^K2\gamma\exp\left(-\alpha n\left(\frac{\alpha}{2}-\sqrt{Rn^{-q}}\right)\right)\le2(K-1)\gamma\exp\left(-\frac{\alpha^2}{4}n\right),
\end{aligned} π n ( κ i , n ) exp ( − 2 n X i , n − x n ( κ i , n ) H n ( κ i , n ) 2 ) ∑ ℓ = 1 , ℓ = κ i , n K π n ( ℓ ) exp ( − 2 n X i , n − x n ( ℓ ) H n ( ℓ ) 2 ) = ℓ = 1 , ℓ = κ i , n ∑ K π n ( κ i , n ) π n ( ℓ ) exp ( − 2 n ( X i , n − x n ( ℓ ) H n ( ℓ ) 2 − X i , n − x n ( κ i , n ) H n ( κ i , n ) 2 ) ) ≤ ℓ = 1 , ℓ = κ i , n ∑ K γ n exp ( − 2 n ( ( α − R n − q ) 2 − R n − 2 q ) ) ≤ ℓ = 1 , ℓ = κ i , n ∑ K 2 γ exp ( − α n ( 2 α − R n − q ) ) ≤ 2 ( K − 1 ) γ exp ( − 4 α 2 n ) , such that (45) holds with c 1 = 2 ( K − 1 ) γ c_1 = 2(K-1)\gamma c 1 = 2 ( K − 1 ) γ c 2 = α 2 / 4 c_2 = \alpha^2/4 c 2 = α 2 /4
Now we turn to the proof of (46) . Let i ∈ { 1 , … , N } i \in\{1,\ldots,N\} i ∈ { 1 , … , N } ∇ I n ( x n ( κ i , n ) ) = 0 \nabla I_n(x_n^{(\kappa_{i,n})})=0 ∇ I n ( x n ( κ i , n ) ) = 0 I n I_n I n x n ( κ i , n ) x_n^{(\kappa_{i,n})} x n ( κ i , n )
I n ( X i , n ) = I n ( x n ( κ i , n ) ) + 1 2 ∥ X i , n − x n ( κ i , n ) ∥ H n ( κ i , n ) 2 + R n ( κ i , n ) ( X i , n ) , I_n(X_{i,n}) = I_n\left(x_n^{(\kappa_{i,n})}\right) + \frac{1}{2}\left\|X_{i,n}-x_n^{(\kappa_{i,n})}\right\|_{H_n^{(\kappa_{i,n})}}^2 + R_n^{(\kappa_{i,n})}(X_{i,n}), I n ( X i , n ) = I n ( x n ( κ i , n ) ) + 2 1 X i , n − x n ( κ i , n ) H n ( κ i , n ) 2 + R n ( κ i , n ) ( X i , n ) , where R n ( κ i , n ) ( X i , n ) R_n^{(\kappa_{i,n})}(X_{i,n}) R n ( κ i , n ) ( X i , n )
exp ( − n I n ( X i , n ) ) = exp ( − n ( I n ( x n ( κ i , n ) ) + 1 2 ∥ X i , n − x n ( κ i , n ) ∥ H n ( κ i , n ) 2 + R n ( κ i , n ) ( X i , n ) ) ) . \exp\left(-nI_n(X_{i,n})\right)=\exp\left(-n\left(I_n\left(x_n^{(\kappa_{i,n})}\right)+\frac{1}{2}\left\|X_{i,n}-x_n^{(\kappa_{i,n})}\right\|_{H_n^{(\kappa_{i,n})}}^2+R_n^{(\kappa_{i,n})}(X_{i,n})\right)\right). exp ( − n I n ( X i , n ) ) = exp ( − n ( I n ( x n ( κ i , n ) ) + 2 1 X i , n − x n ( κ i , n ) H n ( κ i , n ) 2 + R n ( κ i , n ) ( X i , n ) ) ) . As before, all random variables ( X i , n (X_{i,n} ( X i , n κ i , n ) \kappa_{i,n}) κ i , n ) G n , N G_{n,N} G n , N ∥ X i , n − x n ( κ i , n ) ∥ H n ( κ i , n ) 2 ≤ R n − 2 q \left\|X_{i,n}-x_n^{(\kappa_{i,n})}\right\|_{H_n^{(\kappa_{i,n})}}^2 \le Rn^{-2q} X i , n − x n ( κ i , n ) H n ( κ i , n ) 2 ≤ R n − 2 q H n ( k ) H_n^{(k)} H n ( k ) ρ \rho ρ I n I_n I n M M M V n k , R , q V_n^{k,R,q} V n k , R , q k ∈ { 1 , … , K } k \in\{1,\ldots,K\} k ∈ { 1 , … , K }
∣ n R n ( κ i , n ) ( X i , n ) ∣ ≤ 6 M 3 ! ρ 3 ∥ X i , n − x n ( κ i , n ) ∥ 3 ≤ M ρ − 3 R 3 / 2 n 1 − 3 q . (47) \left|nR_n^{(\kappa_{i,n})}(X_{i,n})\right|\le\frac{6M}{3!\rho^3}\left\|X_{i,n}-x_n^{(\kappa_{i,n})}\right\|^3 \le M\rho^{-3}R^{3/2}n^{1-3q}. \tag*{(47)} n R n ( κ i , n ) ( X i , n ) ≤ 3 ! ρ 3 6 M X i , n − x n ( κ i , n ) 3 ≤ M ρ − 3 R 3/2 n 1 − 3 q . (47) Then
w n ∗ ( X i , n ) = exp ( − n I n ( X i , n ) ) ∑ ℓ = 1 K π n ( ℓ ) exp ( − n 2 ∥ X i , n − x n ( ℓ ) ∥ H n ( ℓ ) 2 ) = exp ( − n R n ( κ i , n ) ( X i , n ) ) 1 + ∑ ℓ = 1 , ℓ ≠ κ i , n K π n ( ℓ ) exp ( − n 2 ∥ X i , n − x n ( ℓ ) ∥ H n ( ℓ ) 2 ) π n ( κ i , n ) exp ( − n 2 ∥ X i , n − x n ( κ i , n ) ∥ H n ( κ i , n ) 2 ) . ≤ exp ( − n R n ( κ i , n ) ( X i , n ) ) ≤ exp ( M ρ − 3 R 3 / 2 n 1 − 3 q ) ≤ 1 + exp ( M ρ − 3 R 3 / 2 ) n 1 − 3 q , \begin{aligned}
w_n^*(X_{i,n})=\frac{\exp\left(-nI_n(X_{i,n})\right)}{\sum_{\ell=1}^{K}\pi_n^{(\ell)}\exp\left(-\frac{n}{2}\left\|X_{i,n}-x_n^{(\ell)}\right\|_{H_n^{(\ell)}}^2\right)}
=\frac{\exp\left(-nR_n^{(\kappa_{i,n})}(X_{i,n})\right)}{1+\frac{\sum_{\ell=1,\ell\ne\kappa_{i,n}}^{K}\pi_n^{(\ell)}\exp\left(-\frac{n}{2}\left\|X_{i,n}-x_n^{(\ell)}\right\|_{H_n^{(\ell)}}^2\right)}{\pi_n^{(\kappa_{i,n})}\exp\left(-\frac{n}{2}\left\|X_{i,n}-x_n^{(\kappa_{i,n})}\right\|_{H_n^{(\kappa_{i,n})}}^2\right)}}. \\
&\le\exp\left(-nR_n^{(\kappa_{i,n})}(X_{i,n})\right)\le\exp\left(M\rho^{-3}R^{3/2}n^{1-3q}\right)\le1+\exp\left(M\rho^{-3}R^{3/2}\right)n^{1-3q},
\end{aligned} w n ∗ ( X i , n ) = ∑ ℓ = 1 K π n ( ℓ ) exp ( − 2 n X i , n − x n ( ℓ ) H n ( ℓ ) 2 ) exp ( − n I n ( X i , n ) ) = 1 + π n ( κ i , n ) e x p ( − 2 n X i , n − x n ( κ i , n ) H n ( κ i , n ) 2 ) ∑ ℓ = 1 , ℓ = κ i , n K π n ( ℓ ) e x p ( − 2 n X i , n − x n ( ℓ ) H n ( ℓ ) 2 ) exp ( − n R n ( κ i , n ) ( X i , n ) ) . ≤ exp ( − n R n ( κ i , n ) ( X i , n ) ) ≤ exp ( M ρ − 3 R 3/2 n 1 − 3 q ) ≤ 1 + exp ( M ρ − 3 R 3/2 ) n 1 − 3 q , where in the last inequality we exploited n 1 − 3 q ≤ 1 n^{1-3q}\le1 n 1 − 3 q ≤ 1 q > 1 / 3 q>1/3 q > 1/3 z ∈ [ 0 , 1 ] z\in[0,1] z ∈ [ 0 , 1 ] a ≥ 0 a\ge0 a ≥ 0 e a z ≤ 1 + e a z e^{az}\le1+e^az e a z ≤ 1 + e a z
w n ∗ ( X i , n ) − 1 ≤ exp ( M ρ − 3 R 3 / 2 ) n 1 − 3 q . (48) w_n^*(X_{i,n})-1\le\exp\left(M\rho^{-3}R^{3/2}\right)n^{1-3q}. \tag*{(48)} w n ∗ ( X i , n ) − 1 ≤ exp ( M ρ − 3 R 3/2 ) n 1 − 3 q . (48) We now verify the corresponding lower bound. By (45) for n ≥ n 1 n\ge n_1 n ≥ n 1
w n ∗ ( X i , n ) = exp ( − n R n ( κ i , n ) ( X i , n ) ) 1 + ∑ ℓ = 1 , ℓ ≠ κ i , n K π n ( ℓ ) exp ( − n 2 ∥ X i , n − x n ( ℓ ) ∥ H n ( ℓ ) 2 ) π n ( κ i , n ) exp ( − n 2 ∥ X i , n − x n ( κ i , n ) ∥ H n ( κ i , n ) 2 ) ≥ exp ( − n R n ( κ i , n ) ( X i , n ) ) 1 + c 1 e − c 2 n . w_n^*(X_{i,n})=\frac{\exp\left(-nR_n^{(\kappa_{i,n})}(X_{i,n})\right)}{1+\frac{\sum_{\ell=1,\ell\ne\kappa_{i,n}}^{K}\pi_n^{(\ell)}\exp\left(-\frac{n}{2}\left\|X_{i,n}-x_n^{(\ell)}\right\|_{H_n^{(\ell)}}^2\right)}{\pi_n^{(\kappa_{i,n})}\exp\left(-\frac{n}{2}\left\|X_{i,n}-x_n^{(\kappa_{i,n})}\right\|_{H_n^{(\kappa_{i,n})}}^2\right)}}\ge\frac{\exp\left(-nR_n^{(\kappa_{i,n})}(X_{i,n})\right)}{1+c_1e^{-c_2n}}. w n ∗ ( X i , n ) = 1 + π n ( κ i , n ) e x p ( − 2 n X i , n − x n ( κ i , n ) H n ( κ i , n ) 2 ) ∑ ℓ = 1 , ℓ = κ i , n K π n ( ℓ ) e x p ( − 2 n X i , n − x n ( ℓ ) H n ( ℓ ) 2 ) exp ( − n R n ( κ i , n ) ( X i , n ) ) ≥ 1 + c 1 e − c 2 n exp ( − n R n ( κ i , n ) ( X i , n ) ) . Exploiting (47) and that for any z ≥ 0 z\ge0 z ≥ 0 e − z ≥ 1 − z e^{-z}\ge1-z e − z ≥ 1 − z
exp ( − n R n κ i , n ( X i , n ) ) ≥ exp ( − M ρ − 3 R 3 / 2 n 1 − 3 q ) ≥ 1 − M ρ − 3 R 3 / 2 n 1 − 3 q . \exp\left(-nR_n^{\kappa_{i,n}}(X_{i,n})\right)\ge\exp\left(-M\rho^{-3}R^{3/2}n^{1-3q}\right)\ge1-M\rho^{-3}R^{3/2}n^{1-3q}. exp ( − n R n κ i , n ( X i , n ) ) ≥ exp ( − M ρ − 3 R 3/2 n 1 − 3 q ) ≥ 1 − M ρ − 3 R 3/2 n 1 − 3 q . We choose n 2 ∈ N n_2\in\mathbb{N} n 2 ∈ N n 2 ≥ n 1 n_2\ge n_1 n 2 ≥ n 1 n ∈ N n\in\mathbb{N} n ∈ N n ≥ n 2 n\ge n_2 n ≥ n 2 c 1 e − c 2 n ≤ M ρ − 3 R 3 / 2 n 1 − 3 q c_1e^{-c_2n}\le M\rho^{-3}R^{3/2}n^{1-3q} c 1 e − c 2 n ≤ M ρ − 3 R 3/2 n 1 − 3 q
w n ∗ ( X i , n ) ≥ 1 − M ρ − 3 R 3 / 2 n 1 − 3 q 1 + c 1 e − c 2 n ≥ 1 − M ρ − 3 R 3 / 2 n 1 − 3 q − c 1 e − c 2 n ≥ 1 − 2 M ρ − 3 R 3 / 2 n 1 − 3 q . w_n^*(X_{i,n})\ge\frac{1-M\rho^{-3}R^{3/2}n^{1-3q}}{1+c_1e^{-c_2n}}\ge1-M\rho^{-3}R^{3/2}n^{1-3q}-c_1e^{-c_2n}\ge1-2M\rho^{-3}R^{3/2}n^{1-3q}. w n ∗ ( X i , n ) ≥ 1 + c 1 e − c 2 n 1 − M ρ − 3 R 3/2 n 1 − 3 q ≥ 1 − M ρ − 3 R 3/2 n 1 − 3 q − c 1 e − c 2 n ≥ 1 − 2 M ρ − 3 R 3/2 n 1 − 3 q . In combination with (48) , for c 3 : = max { 2 M ρ − 3 R 3 / 2 , exp ( M ρ − 3 R 3 / 2 ) } c_3:=\max\{2M\rho^{-3}R^{3/2},\exp(M\rho^{-3}R^{3/2})\} c 3 := max { 2 M ρ − 3 R 3/2 , exp ( M ρ − 3 R 3/2 )} n ≥ n 2 n\ge n_2 n ≥ n 2
1 − c 3 n 1 − 3 q ≤ w n ∗ ( X i , n ) ≤ 1 + c 3 n 1 − 3 q . 1-c_3n^{1-3q}\le w_n^*(X_{i,n})\le1+c_3n^{1-3q}. 1 − c 3 n 1 − 3 q ≤ w n ∗ ( X i , n ) ≤ 1 + c 3 n 1 − 3 q . We collected all auxiliary tools together and are now able to prove Lemma 4.1.
Proof of Lemma 4.1. Define w ‾ n ∗ ( X i , n ) : = N − 1 ∑ i = 1 N w n ∗ ( X i , n ) \overline{w}_n^*(X_{i,n}) := N^{-1}\sum_{i=1}^{N} w_n^*(X_{i,n}) w n ∗ ( X i , n ) := N − 1 ∑ i = 1 N w n ∗ ( X i , n )
∣ ESS n ( N ) − N ∣ = ∣ ( ∑ i = 1 N w n ∗ ( X i , n ) ) 2 ∑ i = 1 N w n ∗ ( X i , n ) 2 − N ∣ = ∣ ( ∑ i = 1 N w n ∗ ( X i , n ) ) 2 − N ∑ i = 1 N w n ∗ ( X i , n ) 2 ∑ i = 1 N w n ∗ ( X i , n ) 2 ∣ = N ∣ ( N − 1 ∑ i = 1 N w n ∗ ( X i , n ) ) 2 − N − 1 ∑ i = 1 N w n ∗ ( X i , n ) 2 N − 1 ∑ i = 1 N w n ∗ ( X i , n ) 2 ∣ = ∑ i = 1 N ( w n ∗ ( X i , n ) − w ‾ n ∗ ( X i , n ) ) 2 N − 1 ∑ i = 1 N w n ∗ ( X i , n ) 2 . \begin{aligned}
\left|\operatorname{ESS}_n(N)-N\right|=\left|\frac{\left(\sum_{i=1}^{N}w_n^*(X_{i,n})\right)^2}{\sum_{i=1}^{N}w_n^*(X_{i,n})^2}-N\right|=\left|\frac{\left(\sum_{i=1}^{N}w_n^*(X_{i,n})\right)^2-N\sum_{i=1}^{N}w_n^*(X_{i,n})^2}{\sum_{i=1}^{N}w_n^*(X_{i,n})^2}\right| \\
&=N\left|\frac{\left(N^{-1}\sum_{i=1}^{N}w_n^*(X_{i,n})\right)^2-N^{-1}\sum_{i=1}^{N}w_n^*(X_{i,n})^2}{N^{-1}\sum_{i=1}^{N}w_n^*(X_{i,n})^2}\right|=\frac{\sum_{i=1}^{N}\left(w_n^*(X_{i,n})-\overline{w}_n^*(X_{i,n})\right)^2}{N^{-1}\sum_{i=1}^{N}w_n^*(X_{i,n})^2}.
\end{aligned} ∣ ESS n ( N ) − N ∣ = ∑ i = 1 N w n ∗ ( X i , n ) 2 ( ∑ i = 1 N w n ∗ ( X i , n ) ) 2 − N = ∑ i = 1 N w n ∗ ( X i , n ) 2 ( ∑ i = 1 N w n ∗ ( X i , n ) ) 2 − N ∑ i = 1 N w n ∗ ( X i , n ) 2 = N N − 1 ∑ i = 1 N w n ∗ ( X i , n ) 2 ( N − 1 ∑ i = 1 N w n ∗ ( X i , n ) ) 2 − N − 1 ∑ i = 1 N w n ∗ ( X i , n ) 2 = N − 1 ∑ i = 1 N w n ∗ ( X i , n ) 2 ∑ i = 1 N ( w n ∗ ( X i , n ) − w n ∗ ( X i , n ) ) 2 . As in the proof before all random variables are considered restricted to G n , N G_{n,N} G n , N n 2 ∈ N n_2\in\mathbb{N} n 2 ∈ N c 3 > 0 c_3>0 c 3 > 0 Proposition B.1 we have that (46) holds true. Consequently, we have for n ≥ n 2 n\geq n_2 n ≥ n 2
1 − c 3 n 1 − 3 q ≤ w n ∗ ( X i , n ) ≤ 1 + c 3 n 1 − 3 q , (49) 1-c_3n^{1-3q}\leq w_n^*(X_{i,n})\leq1+c_3n^{1-3q}, \tag*{(49)} 1 − c 3 n 1 − 3 q ≤ w n ∗ ( X i , n ) ≤ 1 + c 3 n 1 − 3 q , (49) which readily implies
− 2 c 3 n 1 − 3 q ≤ w n ∗ ( X i , n ) − w ‾ n ∗ ( X i , n ) ≤ 2 c 3 n 1 − 3 q . -2c_3n^{1-3q}\leq w_n^*(X_{i,n})-\overline{w}_n^*(X_{i,n})\leq2c_3n^{1-3q}. − 2 c 3 n 1 − 3 q ≤ w n ∗ ( X i , n ) − w n ∗ ( X i , n ) ≤ 2 c 3 n 1 − 3 q . The latter inequality can be rewritten as
∣ w n ∗ ( X i , n ) − w ‾ n ∗ ( X i , n ) ∣ ≤ 2 c 3 n 1 − 3 q . \left|w_n^*(X_{i,n})-\overline{w}_n^*(X_{i,n})\right|\leq2c_3n^{1-3q}. ∣ w n ∗ ( X i , n ) − w n ∗ ( X i , n ) ∣ ≤ 2 c 3 n 1 − 3 q . Now, we choose n 0 ∈ N n_0\in\mathbb{N} n 0 ∈ N n 0 ≥ n 2 n_0\geq n_2 n 0 ≥ n 2 n ∈ N n\in\mathbb{N} n ∈ N n ≥ n 0 n\geq n_0 n ≥ n 0 1 − c 3 n 1 − 3 q ≥ 1 / 2 1-c_3n^{1-3q}\geq1/2 1 − c 3 n 1 − 3 q ≥ 1/2 (49) is 1 / 2 1/2 1/2 n ∈ N n\in\mathbb{N} n ∈ N n ≥ n 0 n\geq n_0 n ≥ n 0
∑ i = 1 N ( w n ∗ ( X i , n ) − w ‾ n ∗ ( X i , n ) ) 2 ≤ 4 N c 3 2 n 2 − 6 q and 1 4 ≤ N − 1 ∑ i = 1 N w n ∗ ( X i , n ) 2 . \sum_{i=1}^{N}\left(w_n^*(X_{i,n})-\overline{w}_n^*(X_{i,n})\right)^2\leq4Nc_3^2n^{2-6q}\qquad\text{and}\qquad\frac{1}{4}\leq N^{-1}\sum_{i=1}^{N}w_n^*(X_{i,n})^2. i = 1 ∑ N ( w n ∗ ( X i , n ) − w n ∗ ( X i , n ) ) 2 ≤ 4 N c 3 2 n 2 − 6 q and 4 1 ≤ N − 1 i = 1 ∑ N w n ∗ ( X i , n ) 2 . Consequently, for all n ∈ N n\in\mathbb{N} n ∈ N n ≥ n 0 n\geq n_0 n ≥ n 0 c : = 8 c 3 2 c:=8c_3^2 c := 8 c 3 2 ∣ ESS n ( N ) − N ∣ ≤ c N n 2 − 6 q \left|\operatorname{ESS}_n(N)-N\right|\leq cNn^{2-6q} ∣ ESS n ( N ) − N ∣ ≤ c N n 2 − 6 q □ \square □
Proofs regarding IMH algorithm
Proof of Proposition 4.2
For any x ∈ R d x\in\mathbb{R}^d x ∈ R d A ⊆ R d A\subseteq\mathbb{R}^d A ⊆ R d
IM n ( x , A ) ≥ ∫ A α n ( x , y ) ν ( d y ) = ∫ A ( 1 w n ∗ ( y ) ∧ 1 w n ∗ ( x ) ) w n ∗ ( y ) ν ( d y ) ≥ ∫ R d w n ∗ ( z ) ν ( d z ) ∥ w n ∗ ∥ ∞ π n ( A ) = π n ( A ) ∥ w n ∥ ∞ , \operatorname{IM}_n(x,A)\geq\int_A\alpha_n(x,y)\nu(\mathrm{d}y)=\int_A\left(\frac{1}{w_n^*(y)}\wedge\frac{1}{w_n^*(x)}\right)w_n^*(y)\nu(\mathrm{d}y)\geq\frac{\int_{\mathbb{R}^d}w_n^*(z)\nu(\mathrm{d}z)}{\lVert w_n^*\rVert_\infty}\pi_n(A)=\frac{\pi_n(A)}{\lVert w_n\rVert_\infty}, IM n ( x , A ) ≥ ∫ A α n ( x , y ) ν ( d y ) = ∫ A ( w n ∗ ( y ) 1 ∧ w n ∗ ( x ) 1 ) w n ∗ ( y ) ν ( d y ) ≥ ∥ w n ∗ ∥ ∞ ∫ R d w n ∗ ( z ) ν ( d z ) π n ( A ) = ∥ w n ∥ ∞ π n ( A ) , where the last equality follows by w n ( x ) = w n ∗ ( x ) ∫ R d w n ∗ ( z ) ν ( d z ) w_n(x)=\frac{w_n^*(x)}{\int_{\mathbb{R}^d}w_n^*(z)\nu(\mathrm{d}z)} w n ( x ) = ∫ R d w n ∗ ( z ) ν ( d z ) w n ∗ ( x ) γ : = ∥ w n ∥ ∞ − 1 \gamma:=\lVert w_n\rVert_\infty^{-1} γ := ∥ w n ∥ ∞ − 1
R n ( x , A ) : = IM n ( x , A ) − γ π n ( A ) 1 − γ R_n(x,A):=\frac{\operatorname{IM}_n(x,A)-\gamma\pi_n(A)}{1-\gamma} R n ( x , A ) := 1 − γ IM n ( x , A ) − γ π n ( A ) is a transition kernel with stationary distribution π n \pi_n π n
IM n ( x , A ) = γ π n ( A ) + ( 1 − γ ) R n ( x , A ) . \operatorname{IM}_n(x,A)=\gamma\pi_n(A)+(1-\gamma)R_n(x,A). IM n ( x , A ) = γ π n ( A ) + ( 1 − γ ) R n ( x , A ) . Using that, for f ∈ L 0 2 ( π n ) f \in L^2_0(\pi_n) f ∈ L 0 2 ( π n ) f ≠ 0 f \ne0 f = 0
∥ M n f ∥ L 2 ( π n ) 2 = ∫ R d ∣ ∫ R d f ( y ) I M n ( x , d y ) ∣ 2 π n ( d x ) = ( 1 − γ ) 2 ∫ R d ∣ ∫ R d f ( y ) R n ( x , d y ) ∣ 2 π n ( d x ) ≤ ( 1 − γ ) 2 ∫ R d ∫ R d ∣ f ( y ) ∣ 2 R n ( x , d y ) π n ( d x ) = ( 1 − γ ) 2 ∫ R d ∣ f ( y ) ∣ 2 π n ( d y ) = ( 1 − γ ) 2 ∥ f ∥ L 2 ( π n ) 2 , \begin{aligned}
\|M_n f\|_{L^2(\pi_n)}^2 = \int_{\mathbb{R}^d} \left|\int_{\mathbb{R}^d} f(y) I M_n(x,dy)\right|^2 \pi_n(dx) = (1-\gamma)^2 \int_{\mathbb{R}^d} \left|\int_{\mathbb{R}^d} f(y) R_n(x,dy)\right|^2 \pi_n(dx) \\
&\le(1-\gamma)^2 \int_{\mathbb{R}^d}\int_{\mathbb{R}^d} |f(y)|^2 R_n(x,dy)\,\pi_n(dx) = (1-\gamma)^2 \int_{\mathbb{R}^d} |f(y)|^2 \pi_n(dy) = (1-\gamma)^2\|f\|_{L^2(\pi_n)}^2,
\end{aligned} ∥ M n f ∥ L 2 ( π n ) 2 = ∫ R d ∫ R d f ( y ) I M n ( x , d y ) 2 π n ( d x ) = ( 1 − γ ) 2 ∫ R d ∫ R d f ( y ) R n ( x , d y ) 2 π n ( d x ) ≤ ( 1 − γ ) 2 ∫ R d ∫ R d ∣ f ( y ) ∣ 2 R n ( x , d y ) π n ( d x ) = ( 1 − γ ) 2 ∫ R d ∣ f ( y ) ∣ 2 π n ( d y ) = ( 1 − γ ) 2 ∥ f ∥ L 2 ( π n ) 2 , where we used Jensen’s inequality and stationarity of R n R_n R n π n \pi_n π n ∥ M n ∥ π n ≤ 1 − γ \|M_n\|_{\pi_n} \le1-\gamma ∥ M n ∥ π n ≤ 1 − γ
Proof of Lemma 4.2
By using a ∧ b = 1 2 ( a + b − ∣ b − a ∣ ) a \wedge b = \frac{1}{2}(a+b-|b-a|) a ∧ b = 2 1 ( a + b − ∣ b − a ∣ ) a , b > 0 a,b>0 a , b > 0
E π n ⊗ ν [ α n ] = ∫ R d ∫ R d ( 1 ∧ w n ( y ) w n ( x ) ) ν ( d y ) π n ( d x ) = 1 2 + 1 2 ∫ R d ∫ R d w n ( y ) w n ( x ) ν ( d y ) w n ( x ) ν ( d x ) \begin{aligned}
\mathbb{E}_{\pi_n\otimes\nu}[\alpha_n] = \int_{\mathbb{R}^d}\int_{\mathbb{R}^d}\left(1\wedge\frac{w_n(y)}{w_n(x)}\right)\nu(dy)\pi_n(dx) \\
&= \frac{1}{2}+\frac{1}{2}\int_{\mathbb{R}^d}\int_{\mathbb{R}^d}\frac{w_n(y)}{w_n(x)}\nu(dy)w_n(x)\nu(dx)
\end{aligned} E π n ⊗ ν [ α n ] = ∫ R d ∫ R d ( 1 ∧ w n ( x ) w n ( y ) ) ν ( d y ) π n ( d x ) = 2 1 + 2 1 ∫ R d ∫ R d w n ( x ) w n ( y ) ν ( d y ) w n ( x ) ν ( d x ) − 1 2 ∫ R d ∫ R d ∣ 1 − w n ( y ) w n ( x ) ∣ ν ( d y ) w n ( x ) ν ( d x ) = 1 − 1 2 ∫ R d ∫ R d ∣ w n ( x ) − w n ( y ) ∣ ν ( d y ) ν ( d x ) = 1 − d T V ( π n ⊗ ν , ν ⊗ π n ) \begin{aligned}
\quad-\frac{1}{2}\int_{\mathbb{R}^d}\int_{\mathbb{R}^d}\left|1-\frac{w_n(y)}{w_n(x)}\right|\nu(dy)w_n(x)\nu(dx) \\
&=1-\frac{1}{2}\int_{\mathbb{R}^d}\int_{\mathbb{R}^d}|w_n(x)-w_n(y)|\nu(dy)\nu(dx)=1-d_{\mathrm{TV}}(\pi_n\otimes\nu,\nu\otimes\pi_n)
\end{aligned} − 2 1 ∫ R d ∫ R d 1 − w n ( x ) w n ( y ) ν ( d y ) w n ( x ) ν ( d x ) = 1 − 2 1 ∫ R d ∫ R d ∣ w n ( x ) − w n ( y ) ∣ ν ( d y ) ν ( d x ) = 1 − d TV ( π n ⊗ ν , ν ⊗ π n ) cf. also [18 Theorem 1] for general MH algorithms. The first statement on the average acceptance rate then follows by
d T V ( π n , ν ) ≤ d T V ( π n ⊗ ν , ν ⊗ π n ) ≤ 2 d T V ( π n , ν ) d_{\mathrm{TV}}(\pi_n,\nu)\le d_{\mathrm{TV}}(\pi_n\otimes\nu,\nu\otimes\pi_n)\le2d_{\mathrm{TV}}(\pi_n,\nu) d TV ( π n , ν ) ≤ d TV ( π n ⊗ ν , ν ⊗ π n ) ≤ 2 d TV ( π n , ν ) where the first inequality is due to the data processing inequality for general f f f
d T V ( π n ⊗ ν , ν ⊗ π n ) ≤ 1 2 ∫ R d ∫ R d ∣ w n ( x ) − 1 ∣ + ∣ 1 − w n ( y ) ∣ ν ( d y ) ν ( d x ) = 2 d T V ( π n , ν ) . d_{\mathrm{TV}}(\pi_n\otimes\nu,\nu\otimes\pi_n)\le\frac{1}{2}\int_{\mathbb{R}^d}\int_{\mathbb{R}^d}|w_n(x)-1|+|1-w_n(y)|\,\nu(dy)\nu(dx)=2d_{\mathrm{TV}}(\pi_n,\nu). d TV ( π n ⊗ ν , ν ⊗ π n ) ≤ 2 1 ∫ R d ∫ R d ∣ w n ( x ) − 1∣ + ∣1 − w n ( y ) ∣ ν ( d y ) ν ( d x ) = 2 d TV ( π n , ν ) . For the second statement about the lag-one autocorrelation let us first define for probability measures μ , μ ~ \mu,\widetilde{\mu} μ , μ R 2 d \mathbb{R}^{2d} R 2 d ν 2 : = ν ⊗ ν \nu_2:=\nu\otimes\nu ν 2 := ν ⊗ ν d μ d ν 2 , d μ ~ d ν 2 \frac{d\mu}{d\nu_2},\frac{d\widetilde{\mu}}{d\nu_2} d ν 2 d μ , d ν 2 d μ μ ∧ μ ~ \mu\wedge\widetilde{\mu} μ ∧ μ R 2 d \mathbb{R}^{2d} R 2 d
[ μ ∧ μ ~ ] ( A ) : = ∫ A ( d μ d ν 2 ( z ) ∧ d μ ~ d ν 2 ( z ) ) ν 2 ( d z ) , [\mu\wedge\widetilde{\mu}](A):=\int_A\left(\frac{d\mu}{d\nu_2}(z)\wedge\frac{d\widetilde{\mu}}{d\nu_2}(z)\right)\nu_2(dz), [ μ ∧ μ ] ( A ) := ∫ A ( d ν 2 d μ ( z ) ∧ d ν 2 d μ ( z ) ) ν 2 ( d z ) , for arbitrary Borel measurable A ⊆ R 2 d A\subseteq\mathbb{R}^{2d} A ⊆ R 2 d
ρ n ν = ∫ R d ∫ R d ∣ v T x − v T y ∣ 2 ( 1 ∧ w n ( y ) w n ( y ) ) ν ( d y ) π n ( d x ) = ∫ R d ∫ R d ∣ v T x − v T y ∣ 2 ( w n ( x ) ∧ w n ( y ) ) ν ( d y ) ν ( d x ) = [ π n ⊗ ν ∧ ν ⊗ π n ] ( ∣ Δ f v ∣ 2 ) \begin{aligned}
\rho_n^\nu=\int_{\mathbb{R}^d}\int_{\mathbb{R}^d}|v^T x-v^T y|^2\left(1\wedge\frac{w_n(y)}{w_n(y)}\right)\nu(dy)\pi_n(dx) \\
&=\int_{\mathbb{R}^d}\int_{\mathbb{R}^d}|v^T x-v^T y|^2(w_n(x)\wedge w_n(y))\,\nu(dy)\nu(dx)=[\pi_n\otimes\nu\wedge\nu\otimes\pi_n](|\Delta f_v|^2)
\end{aligned} ρ n ν = ∫ R d ∫ R d ∣ v T x − v T y ∣ 2 ( 1 ∧ w n ( y ) w n ( y ) ) ν ( d y ) π n ( d x ) = ∫ R d ∫ R d ∣ v T x − v T y ∣ 2 ( w n ( x ) ∧ w n ( y )) ν ( d y ) ν ( d x ) = [ π n ⊗ ν ∧ ν ⊗ π n ] ( ∣Δ f v ∣ 2 ) where f v ( x ) : = v T x f_v(x):=v^T x f v ( x ) := v T x f : R d → R f:\mathbb{R}^d\to\mathbb{R} f : R d → R
[ π n ⊗ ν ∧ ν ⊗ π n ] ( ∣ Δ f ∣ 2 ) : = ∫ R d ∫ R d ∣ f ( x ) − f ( y ) ∣ 2 [ π n ⊗ ν ∧ ν ⊗ π n ] ( d x d y ) . [\pi_n\otimes\nu\wedge\nu\otimes\pi_n](|\Delta f|^2):=\int_{\mathbb{R}^d}\int_{\mathbb{R}^d}|f(x)-f(y)|^2[\pi_n\otimes\nu\wedge\nu\otimes\pi_n](dxdy). [ π n ⊗ ν ∧ ν ⊗ π n ] ( ∣Δ f ∣ 2 ) := ∫ R d ∫ R d ∣ f ( x ) − f ( y ) ∣ 2 [ π n ⊗ ν ∧ ν ⊗ π n ] ( d x d y ) . The latter notation can be readily adapted to other product measures, such that
2 Var π n ( f ν ) = [ π n ⊗ π n ∧ π n ⊗ π n ] ( ∣ Δ f ν ∣ 2 ) 2\operatorname{Var}_{\pi_n}(f_\nu)=[\pi_n\otimes\pi_n\wedge\pi_n\otimes\pi_n](|\Delta f_\nu|^2) 2 Var π n ( f ν ) = [ π n ⊗ π n ∧ π n ⊗ π n ] ( ∣Δ f ν ∣ 2 ) and by (3) and Remark 2.1 we have
∣ ρ n ν − 2 Var π n ( f ν ) ∣ = ∣ [ π n ⊗ ν ∧ ν ⊗ π n ] ( ∣ Δ f ν ∣ 2 ) − [ π n ⊗ π n ∧ π n ⊗ π n ] ( ∣ Δ f ν ∣ 2 ) ∣ ≤ 2 [ π n ⊗ ν ∧ ν ⊗ π n ] ( ∣ Δ f ν ∣ 4 ) + 2 [ π n ⊗ π n ∧ π n ⊗ π n ] ( ∣ Δ f ν ∣ 4 ) ⋅ d H ( π n ⊗ ν ∧ ν ⊗ π n , π n ⊗ π n ∧ π n ⊗ π n ) . (50) \begin{aligned}
|\rho_n^\nu-2\operatorname{Var}_{\pi_n}(f_\nu)|
&=\left|[\pi_n\otimes\nu\wedge\nu\otimes\pi_n](|\Delta f_\nu|^2)-[\pi_n\otimes\pi_n\wedge\pi_n\otimes\pi_n](|\Delta f_\nu|^2)\right|\\
&\le\sqrt{2}[\pi_n\otimes\nu\wedge\nu\otimes\pi_n](|\Delta f_\nu|^4)+2[\pi_n\otimes\pi_n\wedge\pi_n\otimes\pi_n](|\Delta f_\nu|^4)\\
&\qquad\cdot d_{\mathrm H}(\pi_n\otimes\nu\wedge\nu\otimes\pi_n,\pi_n\otimes\pi_n\wedge\pi_n\otimes\pi_n). \tag*{(50)}
\end{aligned} ∣ ρ n ν − 2 Var π n ( f ν ) ∣ = [ π n ⊗ ν ∧ ν ⊗ π n ] ( ∣Δ f ν ∣ 2 ) − [ π n ⊗ π n ∧ π n ⊗ π n ] ( ∣Δ f ν ∣ 2 ) ≤ 2 [ π n ⊗ ν ∧ ν ⊗ π n ] ( ∣Δ f ν ∣ 4 ) + 2 [ π n ⊗ π n ∧ π n ⊗ π n ] ( ∣Δ f ν ∣ 4 ) ⋅ d H ( π n ⊗ ν ∧ ν ⊗ π n , π n ⊗ π n ∧ π n ⊗ π n ) . (50) Furthermore
d H 2 ( π n ⊗ ν ∧ ν ⊗ π n , π n ⊗ π n ∧ π n ⊗ π n ) = ∫ R d ∫ R d ( w n ( x ) ∧ w n ( y ) − w n ( x ) w n ( y ) ) 2 ν ( d x ) ν ( d y ) ≤ ∫ R d ∫ R d ( w n ( x ) − w n ( x ) w n ( y ) ) 2 ν ( d x ) ν ( d y ) + ∫ R d ∫ R d ( w n ( y ) − w n ( x ) w n ( y ) ) 2 ν ( d x ) ν ( d y ) = 2 d H 2 ( π n , ν ) . \begin{aligned}
d_{\mathrm H}^2(\pi_n\otimes\nu\wedge\nu\otimes\pi_n,\pi_n\otimes\pi_n\wedge\pi_n\otimes\pi_n)
&=\int_{\mathbb{R}^d}\int_{\mathbb{R}^d}\left(\sqrt{w_n(x)\wedge w_n(y)}-\sqrt{w_n(x)w_n(y)}\right)^2\nu(dx)\nu(dy)\\
&\le\int_{\mathbb{R}^d}\int_{\mathbb{R}^d}\left(\sqrt{w_n(x)}-\sqrt{w_n(x)w_n(y)}\right)^2\nu(dx)\nu(dy)\\
&\qquad+\int_{\mathbb{R}^d}\int_{\mathbb{R}^d}\left(\sqrt{w_n(y)}-\sqrt{w_n(x)w_n(y)}\right)^2\nu(dx)\nu(dy)=2d_{\mathrm H}^2(\pi_n,\nu).
\end{aligned} d H 2 ( π n ⊗ ν ∧ ν ⊗ π n , π n ⊗ π n ∧ π n ⊗ π n ) = ∫ R d ∫ R d ( w n ( x ) ∧ w n ( y ) − w n ( x ) w n ( y ) ) 2 ν ( d x ) ν ( d y ) ≤ ∫ R d ∫ R d ( w n ( x ) − w n ( x ) w n ( y ) ) 2 ν ( d x ) ν ( d y ) + ∫ R d ∫ R d ( w n ( y ) − w n ( x ) w n ( y ) ) 2 ν ( d x ) ν ( d y ) = 2 d H 2 ( π n , ν ) . Finally, by a ∧ b ≤ a + b a\wedge b\le a+b a ∧ b ≤ a + b a , b > 0 a,b>0 a , b > 0
[ π n ⊗ ν ∧ ν ⊗ π n ] ( ∣ Δ f ν ∣ 4 ) = ∫ R d ∫ R d ∣ v T x − v T y ∣ 4 ( w n ( x ) ∧ w n ( y ) ) ν ( d y ) ν ( d x ) ≤ 2 ∫ R d ∫ R d ∣ v T x − v T y ∣ 4 π n ( d x ) ν ( d y ) . \begin{aligned}
[\pi_n\otimes\nu\wedge\nu\otimes\pi_n](|\Delta f_\nu|^4)
&=\int_{\mathbb{R}^d}\int_{\mathbb{R}^d}|v^Tx-v^Ty|^4(w_n(x)\wedge w_n(y))\nu(dy)\nu(dx)\\
&\le2\int_{\mathbb{R}^d}\int_{\mathbb{R}^d}|v^Tx-v^Ty|^4\pi_n(dx)\nu(dy).
\end{aligned} [ π n ⊗ ν ∧ ν ⊗ π n ] ( ∣Δ f ν ∣ 4 ) = ∫ R d ∫ R d ∣ v T x − v T y ∣ 4 ( w n ( x ) ∧ w n ( y )) ν ( d y ) ν ( d x ) ≤ 2 ∫ R d ∫ R d ∣ v T x − v T y ∣ 4 π n ( d x ) ν ( d y ) . which by exploiting (24) implies the statement.
