Introduction
We consider an inverse problem
y = A ( x ) + δ ϵ , (1) y = A(x) + \delta\epsilon, \tag*{(1)} y = A ( x ) + δ ϵ , (1) where y ∈ R m y \in\mathbb{R}^m y ∈ R m x ∈ X x \in X x ∈ X X X X A : X → R m A : X \to\mathbb{R}^m A : X → R m ϵ ∈ R m ∼ N ( 0 , I m ) \epsilon\in\mathbb{R}^m \sim\mathcal{N}(0,I_m) ϵ ∈ R m ∼ N ( 0 , I m ) δ > 0 \delta> 0 δ > 0
We adopt the Bayesian approach [18 , 45 ] and assume to have some prior knowledge of the distribution of the unknown available, modelled by the probability measure μ 0 \mu_0 μ 0 μ ( ⋅ ∣ y ) \mu(\cdot\mid y) μ ( ⋅ ∣ y ) x x x y y y (1) via Bayes’ formula, expressed through the Radon-Nikodym derivative
d μ d μ 0 ( x ∣ y ) = 1 Z ( y ) exp ( − 1 2 δ 2 ∥ A ( x ) − y ∥ 2 2 ) in L 1 ( μ 0 ) \frac{\mathrm{d}\mu}{\mathrm{d}\mu_0}(x\mid y) = \frac{1}{Z(y)} \exp\left(-\frac{1}{2\delta^2}\left\|A(x)-y\right\|_2^2\right) \quad\text{in } L^1(\mu_0) d μ 0 d μ ( x ∣ y ) = Z ( y ) 1 exp ( − 2 δ 2 1 ∥ A ( x ) − y ∥ 2 2 ) in L 1 ( μ 0 ) with normalising constant
Z ( y ) = ∫ X exp ( − 1 2 δ 2 ∥ A ( x ) − y ∥ 2 2 ) μ 0 ( d x ) . Z(y) = \int_X \exp\left(-\frac{1}{2\delta^2}\left\|A(x)-y\right\|_2^2\right)\,\mu_0(\mathrm{d}x). Z ( y ) = ∫ X exp ( − 2 δ 2 1 ∥ A ( x ) − y ∥ 2 2 ) μ 0 ( d x ) . For more details, see e.g. [45 ] . In what follows, we write p ( y ∣ x ) = ( 2 π δ 2 ) − m / 2 exp ( − 1 2 δ 2 ∥ A ( x ) − y ∥ 2 ) p(y|x) = (2\pi\delta^2)^{-m/2}\exp\left(-\frac{1}{2\delta^2}\lVert A(x)-y\rVert^2\right) p ( y ∣ x ) = ( 2 π δ 2 ) − m /2 exp ( − 2 δ 2 1 ∥ A ( x ) − y ∥ 2 ) p ( y ) = ( 2 π δ 2 ) − m / 2 Z ( y ) p(y) = (2\pi\delta^2)^{-m/2}Z(y) p ( y ) = ( 2 π δ 2 ) − m /2 Z ( y ) y y y
In many practical settings, the forward operator is either not precisely known or computationally expensive to evaluate. Therefore, it is often convenient to study the problem using a less computationally intensive approximate model, say, by replacing the partial differential equation (PDE) solver associated with A A A [20 , 33 , 6 , 7 , 1 ] . In the inverse problems literature, such questions are also tackled in the context of stability analysis related to the Bayesian posterior [46 ] .
In this paper, we focus on an early likelihood-based surrogate model for inverse problems, introduced in [1 ] and known as the approximation error model (AEM). The central idea is that, given a computationally efficient forward approximation A ^ \widehat{A} A A ^ \widehat{A} A
p ^ ( y ∣ x ; ξ , Γ ) = N ( A ^ ( x ) + ξ , Γ ) , (2) \widehat{p}(y|x;\xi,\Gamma)=\mathcal{N}(\widehat{A}(x)+\xi,\Gamma), \tag*{(2)} p ( y ∣ x ; ξ , Γ ) = N ( A ( x ) + ξ , Γ ) , (2) where ξ ∈ R m \xi\in\mathbb{R}^m ξ ∈ R m Γ ∈ S ( m ) ⊂ R m × m \Gamma\in\mathcal{S}(m)\subset\mathbb{R}^{m\times m} Γ ∈ S ( m ) ⊂ R m × m S ( m ) \mathcal{S}(m) S ( m ) p ^ ( y ; ξ , Γ ) \widehat{p}(y;\xi,\Gamma) p ( y ; ξ , Γ ) y y y (2) . The approximate likelihood and the prior give rise to an approximative posterior
d μ ^ d μ 0 ( x ∣ y ; ξ , Γ ) = p ^ ( y ∣ x ; ξ , Γ ) p ^ ( y ; ξ , Γ ) in L 1 ( μ 0 ) . (3) \frac{\mathrm{d}\widehat{\mu}}{\mathrm{d}\mu_0}(x|y;\xi,\Gamma)=\frac{\widehat{p}(y|x;\xi,\Gamma)}{\widehat{p}(y;\xi,\Gamma)}\quad\text{in }L^1(\mu_0). \tag*{(3)} d μ 0 d μ ( x ∣ y ; ξ , Γ ) = p ( y ; ξ , Γ ) p ( y ∣ x ; ξ , Γ ) in L 1 ( μ 0 ) . (3) Further, parameters ( ξ , Γ ) (\xi,\Gamma) ( ξ , Γ ) μ ^ ( ⋅ ∣ y ) \widehat{\mu}(\cdot|y) μ ( ⋅ ∣ y ) μ ( ⋅ ∣ y ) \mu(\cdot|y) μ ( ⋅ ∣ y )
In our view, the AEM approach has sparked significant attention in the application domain (see e.g. [1 , 21 , 34 , 35 , 32 , 39 , 15 , 50 ] ) due to three primary reasons: first, the easy-to-use formulation combining a Gaussian likelihood with the given arbitrary approximative scheme for the forward mapping. Second, intuitive interpretability of parameters ξ \xi ξ Γ \Gamma Γ [18 ] ). Third, the empirical evidence of improved posterior modeling reported in many works [1 , 18 , 23 , 38 , 3 ] .
The immediate fundamental question in AEM is how to effectively choose the parameters ξ \xi ξ Γ \Gamma Γ enhanced error approach (EEA) [18 ] . In EEA, the parameters ( ξ , Γ ) (\xi,\Gamma) ( ξ , Γ )
ξ ∗ = E λ ( y − A ^ ( x ) ) , Γ ∗ = E λ [ ( y − A ^ ( x ) − ξ ∗ ) ⊗ ( y − A ^ ( x ) − ξ ∗ ) ] , (4) \xi_*=\mathbb{E}^{\lambda}\left(y-\widehat{A}(x)\right),\qquad
\Gamma_*=\mathbb{E}^{\lambda}\left[\left(y-\widehat{A}(x)-\xi_*\right)\otimes\left(y-\widehat{A}(x)-\xi_*\right)\right], \tag*{(4)} ξ ∗ = E λ ( y − A ( x ) ) , Γ ∗ = E λ [ ( y − A ( x ) − ξ ∗ ) ⊗ ( y − A ( x ) − ξ ∗ ) ] , (4) where λ \lambda λ ( x , y ) (x,y) ( x , y ) X × R m X\times\mathbb{R}^m X × R m (1) and the prior μ 0 \mu_0 μ 0 ξ ∗ \xi_* ξ ∗ Γ ∗ \Gamma_* Γ ∗ (1) .
In this paper, we adopt a novel perspective on AEM by exploring the optimal choice of parametrisation ( ξ , Γ ) (\xi,\Gamma) ( ξ , Γ ) (3) . We show that, as expected, the EEA can be interpreted as fitting the best possible likelihood approximation when averaged over the prior. This leaves open the key question of our paper: how to directly optimise the posterior error given prior information, whether in the form of a closed-form distribution or empirical data?
Our contribution
The main contribution of this paper is to propose and analyse an optimality condition for the surrogate posterior by minimising the expected information loss , i.e., we solve
( ξ ^ , Γ ^ ) = argmin ξ , Γ E P KL ( μ ( ⋅ ∣ y ) ∥ μ ^ ( ⋅ ∣ y ; ξ , Γ ) ) , (5) (\widehat{\xi},\widehat{\Gamma})=\underset{\xi,\Gamma}{\operatorname{argmin}}\ \mathbb{E}^{\mathbb{P}}\operatorname{KL}\left(\mu(\cdot|y)\,\|\,\widehat{\mu}(\cdot|y;\xi,\Gamma)\right), \tag*{(5)} ( ξ , Γ ) = ξ , Γ argmin E P KL ( μ ( ⋅ ∣ y ) ∥ μ ( ⋅ ∣ y ; ξ , Γ ) ) , (5) where the Kullback–Leibler (KL) divergence is defined in (6) ahead. We refer to the resulting criterion as the optimal approximation error (OAE) approach, and to any corresponding minimiser ( ξ ^ , Γ ^ ) (\widehat{\xi},\widehat{\Gamma}) ( ξ , Γ ) [17 ] . In what follows, we write G : = Γ − 1 G := \Gamma^{-1} G := Γ − 1 (5) reduces to
( ξ ^ , G ^ ) = argmin ξ , G J ( ξ , G ) , where J ( ξ , G ) = − E λ log ( d μ ^ ( ⋅ ∣ y ; ξ , G ) d μ 0 ( x ) ) , (\hat{\xi},\hat{G}) = \underset{\xi,G}{\operatorname{argmin}}\ \mathcal{J}(\xi,G), \qquad\text{where}\quad\mathcal{J}(\xi,G) = -\mathbb{E}^{\lambda}\log\left(\frac{\mathrm{d}\hat{\mu}(\cdot\mid y;\xi,G)}{\mathrm{d}\mu_0}(x)\right), ( ξ ^ , G ^ ) = ξ , G argmin J ( ξ , G ) , where J ( ξ , G ) = − E λ log ( d μ 0 d μ ^ ( ⋅ ∣ y ; ξ , G ) ( x ) ) , see Lemma 2.3.
Our key findings are as follows:
( ξ ∗ , G ∗ ) = argmin ξ , G E μ 0 KL ( p ( ⋅ ∣ x ) ∥ p ^ ( ⋅ ∣ x ; ξ , G ) ) . (\xi_*,G_*) = \underset{\xi,G}{\operatorname{argmin}}\ \mathbb{E}^{\mu_0}\operatorname{KL}\left(p(\cdot\mid x)\mathbin{\|}\hat{p}(\cdot\mid x;\xi,G)\right). ( ξ ∗ , G ∗ ) = ξ , G argmin E μ 0 KL ( p ( ⋅ ∣ x ) ∥ p ^ ( ⋅ ∣ x ; ξ , G ) ) . This confirms that EEA is optimal in a likelihood-averaged sense, but leaves open whether directly optimising the posterior error in (5) yields further benefits.
In Section 2 we establish the fundamental properties of the functional J \mathcal{J} J J \mathcal{J} J ( ξ , G ) (\xi,G) ( ξ , G )
In the special case of a linear surrogate A ^ \hat{A} A ^ μ 0 \mu_0 μ 0 § 3 ), we derive J \mathcal{J} J ξ \xi ξ ξ ^ = E λ ( y − A ^ ( x ) ) \hat{\xi} = \mathbb{E}^{\lambda}(y-\hat{A}(x)) ξ ^ = E λ ( y − A ^ ( x )) (4) .
For practical optimisation, in § 4 we first exhibit that the infimum is in general not attained and further the level sets of J \mathcal{J} J J α : = J + α R \mathcal{J}_{\alpha}:=\mathcal{J}+\alpha R J α := J + α R α > 0 \alpha>0 α > 0 R ( G ) = δ 2 tr ( G ) − log det G R(G)=\delta^2\operatorname{tr}(G)-\log\det G R ( G ) = δ 2 tr ( G ) − log det G Γ = G − 1 \Gamma=G^{-1} Γ = G − 1 J α \mathcal{J}_{\alpha} J α J α \mathcal{J}_{\alpha} J α [2 ] .
In § 5 we devise two plug-in estimators from empirical data, a single-loop estimator for the Gaussian linear case and a double-loop estimator for the general case, and prove in Theorems 5.4 and 5.9 that they concentrate on the set of minimisers of the regularised objective J α \mathcal{J}_{\alpha} J α M → ∞ M\to\infty M → ∞
Finally, we validate the proposed approach on a linear CT-based and a nonlinear PDE-based (Darcy flow) inverse problem in § 6 , where OAE is shown to reduce the expected posterior KL divergence and improve mean-squared error relative to both the unmitigated and EEA-corrected likelihoods.
Bayesian inverse problems often seek to infer quantities such as coefficients of PDEs or integral equations that are inherently infinite dimensional. While any numerical solution requires a discretisation of the problem, there has been a significant effort to develop methods that are independent of the choice of discretisation [25 , 26 , 27 ] . In line with [45 ] , we seek discretisation-invariant methods and avoid discretisation until the last possible moment. The core results of § 2 hold for unknowns taking values in a separable, potentially infinite-dimensional, Banach space X X X y ∈ R m y\in\mathbb{R}^{m} y ∈ R m ( ξ , Γ ) (\xi,\Gamma) ( ξ , Γ ) § 3 specialises to Hilbert spaces and § 5 to X = R n X=\mathbb{R}^{n} X = R n
Literature overview
Variational inference is a classical technique that approximates a posterior by a member of a tractable family of distributions. It rests on writing the log-evidence as the sum of the so-called evidence lower bound and the KL divergence from the approximating distribution to the posterior. Since the log-evidence does not depend on the approximating distribution, maximising the former is equivalent to minimising the latter. See e.g. [4 Section 9.4] . Our objective (5) has the same structure (Lemma 2.3), but with two differences: the approximating family is the set of AEM posteriors (3) , parametrised by ( ξ , Γ ) (\xi,\Gamma) ( ξ , Γ ) KL ( μ ∥ μ ^ ) \operatorname{KL}(\mu\mathbin\Vert\hat{\mu}) KL ( μ ∥ μ ^ ) [37 , 31 ] where mode-seeking approximation of general measures in a class of Gaussian or Gaussian mixtures are explored.
The literature on surrogate modelling in inverse problems is very vast. The importance of accounting for a misspecified forward model is well-studied, e.g. in the works of [6 , 20 , 18 , 44 ] . Classical strategies include the complete error model [19 ] , which infers the unknown and the model error jointly, at the cost of a significantly higher-dimensional inference problem. Another idea presented in the literature is based on a two-stage sampling algorithm. Therein samples generated utilising the surrogate are used to speed up an MCMC algorithm employing the exact forward map [51 , 24 ] .
We mention a few of the many further strategies discussed in the literature. An adaptive scheme that constructs local approximations of the posterior distribution depending on measurement-data is proposed in [28 ] . The authors of [8 ] propose an iterative scheme updating the posterior densities through the model error. A framework that enables embedded internal model adjustment is proposed in [40 , 41 ] . Therein an additive modification of model parameters using a Polynomial Chaos representation is utilised. In [32 ] , a learned model correction utilising measurement data is discussed based on a variational approach. Perhaps the closest contribution to ours is given in the recent work of [11 ] , where the introduced distance of an approximate posterior to the posterior distribution in a KL divergence based metric is studied. The authors analyse multiple posterior approximations including EEA and the complete error model. Further the choice of the observation operator is analysed from the viewpoint of mitigating model error.
Convex and non-convex minimisation problems based on the Kurdyka-Lojasiewicz property have been studied in [2 , 49 ] , including convergence guarantees for numerical algorithms. These results are based on the Kurdyka-Lojasiewicz inequalities [22 , 30 ] and have since been applied in numerous works (see e.g. [10 , 47 , 9 , 29 ] ).
Structure of the paper. § 2 introduces the OAE functional formally, relates it to the EEA, and establishes smoothness (Theorem 2.5 ) and bi-convexity (Theorem 2.7 ). § 3 specialises to linear surrogates A ^ \hat{A} A ^ J \mathcal{J} J ξ \xi ξ Theorem 3.1 ). § 4 addresses the practical optimisation of J \mathcal{J} J J α \mathcal{J}_{\alpha} J α § 5 studies stability of the method: we establish consistency of an empirical estimator as the number of training samples grows (Theorems 5.4 and 5.9 ). § 6 validates the approach numerically on a linear CT-based inverse problem and a nonlinear PDE-based inverse problem governed by Darcy’s law. § 7 draws concluding remarks and gives an outlook on future work.
Optimal approximation error
For two probability measures μ \mu μ ν \nu ν X X X μ ≪ ν \mu\ll\nu μ ≪ ν
KL ( μ ∥ ν ) = E μ log ( d μ d ν ( x ) ) . (6) \operatorname{KL}(\mu\mathbin\Vert\nu) = \mathbb{E}^{\mu} \log\left(\frac{\mathrm{d}\mu}{\mathrm{d}\nu}(x)\right). \tag*{(6)} KL ( μ ∥ ν ) = E μ log ( d ν d μ ( x ) ) . (6) Notice that KL divergence is not symmetric; in the following we use the arrangement above with the aim to optimise ν \nu ν μ \mu μ mean-seeking . Depending on the arrangement, different behaviour of the approximation is enforced. If μ \mu μ ν \nu ν ν \nu ν μ \mu μ μ \mu μ [4 , 31 ] ).
For convenience, in what follows, we reparametrise the likelihood approximation in (2) with respect to the mean-shift ξ \xi ξ G : = Γ − 1 G := \Gamma^{-1} G := Γ − 1 ∥ y ∥ G 2 = ⟨ y , G y ⟩ \lVert y\rVert_{G}^{2} = \langle y, Gy\rangle ∥ y ∥ G 2 = ⟨ y , G y ⟩
Let us next study the EEA method from the perspective of a minimisation problem. Let us define a functional K : R m × S ( m ) → R \mathcal{K} : \mathbb{R}^{m} \times\mathcal{S}(m) \to\mathbb{R} K : R m × S ( m ) → R
K ( ξ , G ) = E μ 0 KL ( p ( ⋅ ∣ x ) ∥ p ^ ( ⋅ ∣ x ; ξ , G ) ) (7) \mathcal{K}(\xi,G) = \mathbb{E}^{\mu_{0}}\operatorname{KL}\bigl(p(\cdot\mid x)\mathbin\Vert\hat{p}(\cdot\mid x;\xi,G)\bigr) \tag*{(7)} K ( ξ , G ) = E μ 0 KL ( p ( ⋅ ∣ x ) ∥ p ^ ( ⋅ ∣ x ; ξ , G ) ) (7) In order to prove that the objective functional K \mathcal{K} K
Assumption 1. The operators A ∈ { A , A ^ } \mathcal{A} \in\{\mathcal{A},\widehat{\mathcal{A}}\} A ∈ { A , A } μ 0 \mu_0 μ 0
(A1) (Lipschitz) There exists L 1 > 0 L_1 > 0 L 1 > 0
∥ A ( x ) − A ( x ′ ) ∥ ≤ L 1 ∥ x − x ′ ∥ \|\mathcal{A}(x)-\mathcal{A}(x')\| \le L_1 \|x-x'\| ∥ A ( x ) − A ( x ′ ) ∥ ≤ L 1 ∥ x − x ′ ∥ (A2) (sub-Gaussian prior) There exists L 2 > 0 L_2 > 0 L 2 > 0
E μ 0 exp ( L 2 ∥ x ∥ 2 ) < ∞ \mathbb{E}^{\mu_0}\exp\left(L_2\|x\|^2\right)<\infty E μ 0 exp ( L 2 ∥ x ∥ 2 ) < ∞ The assumption ensures the boundedness of several key expectations and moment terms, see Theorem 2.2 and Lemma 2.4 ahead. We point out now, that Assumptions 1 are inherited from [16 ] . Indeed, condition (iii) of [16 Assumption 3.1] follows from (A1). Hence the auxiliary estimates of [16 Section 3.1] , which we use throughout, apply. We emphasise that these estimates, unlike the stability theorems of that work, do not require any coupling between L 1 L_1 L 1 L 2 L_2 L 2
Remark 2.1. We point out that {(1.4)} implies Γ ∗ ⪰ δ 2 I \Gamma_* \succeq\delta^2 I Γ ∗ ⪰ δ 2 I e = y − A ^ ( x ) − ξ ∗ e=y-\widehat{\mathcal{A}}(x)-\xi_* e = y − A ( x ) − ξ ∗
Γ ∗ = Cov ( e ) = Cov ( E ( e ∣ x ) ) + E ( Cov ( e ∣ x ) ) = Cov ( E ( e ∣ x ) ) + δ 2 I m ⪰ δ 2 I . \begin{aligned}
\Gamma_* = \operatorname{Cov}(e)=\operatorname{Cov}\left(\mathbb{E}(e\mid x)\right)+\mathbb{E}\left(\operatorname{Cov}(e\mid x)\right) \\
&= \operatorname{Cov}\left(\mathbb{E}(e\mid x)\right)+\delta^2 I_m \succeq\delta^2 I.
\end{aligned} Γ ∗ = Cov ( e ) = Cov ( E ( e ∣ x ) ) + E ( Cov ( e ∣ x ) ) = Cov ( E ( e ∣ x ) ) + δ 2 I m ⪰ δ 2 I . This is a natural condition for Γ ∗ \Gamma_* Γ ∗
Theorem 2.2. Let ξ ∗ ∈ R m \xi_* \in\mathbb{R}^m ξ ∗ ∈ R m Γ ∗ ∈ S ( m ) \Gamma_* \in S(m) Γ ∗ ∈ S ( m ) G ∗ = Γ ∗ − 1 G_*=\Gamma_*^{-1} G ∗ = Γ ∗ − 1 ( ξ ∗ , G ∗ ) (\xi_*,G_*) ( ξ ∗ , G ∗ ) K \mathcal{K} K
Proof of Theorem 2.2.
K ( ξ , G ) = ∫ X ∫ R m log p ( y ∣ x ) p ^ ( y ∣ x ; ξ , G ) p ( y ∣ x ) d y μ 0 ( d x ) \mathcal{K}(\xi,G)=\int_X\int_{\mathbb{R}^m}\log\frac{p(y\mid x)}{\widehat{p}(y\mid x;\xi,G)}p(y\mid x)\,dy\,\mu_0(dx) K ( ξ , G ) = ∫ X ∫ R m log p ( y ∣ x ; ξ , G ) p ( y ∣ x ) p ( y ∣ x ) d y μ 0 ( d x ) = const − E λ log p ^ ( y ∣ x ; ξ , G ) . =\operatorname{const}-\mathbb{E}^{\lambda}\log\widehat{p}(y\mid x;\xi,G). = const − E λ log p ( y ∣ x ; ξ , G ) . where const : = ∫ X ∫ R m log p ( y ∣ x ) p ( y ∣ x ) d y μ 0 ( d x ) \operatorname{const}:=\int_X\int_{\mathbb{R}^m}\log p(y\mid x)p(y\mid x)\,dy\,\mu_0(dx) const := ∫ X ∫ R m log p ( y ∣ x ) p ( y ∣ x ) d y μ 0 ( d x ) ( ξ , G ) (\xi,G) ( ξ , G ) K \mathcal{K} K
K ( ξ , G ) = const + m 2 log 2 π − 1 2 log det G + 1 2 E λ ∥ A ^ ( x ) − y + ξ ∥ G 2 . \mathcal{K}(\xi,G)=\operatorname{const}+\frac{m}{2}\log2\pi-\frac{1}{2}\log\det G+\frac{1}{2}\mathbb{E}^{\lambda}\left\|\widehat{\mathcal{A}}(x)-y+\xi\right\|_G^2. K ( ξ , G ) = const + 2 m log 2 π − 2 1 log det G + 2 1 E λ A ( x ) − y + ξ G 2 . We optimise over ξ \xi ξ G G G G G G
1 2 E λ ∥ A ^ ( x ) − y + ξ ∥ G 2 \frac{1}{2}\mathbb{E}^{\lambda}\left\|\widehat{\mathcal{A}}(x)-y+\xi\right\|_G^2 2 1 E λ A ( x ) − y + ξ G 2 is minimised in ξ \xi ξ E λ [ y − A ^ ( x ) − ξ ] = 0 \mathbb{E}^{\lambda}[y-\widehat{\mathcal{A}}(x)-\xi]=0 E λ [ y − A ( x ) − ξ ] = 0 ξ ∗ \xi_* ξ ∗ G G G e = y − A ^ ( x ) − ξ ∗ e=y-\widehat{\mathcal{A}}(x)-\xi_* e = y − A ( x ) − ξ ∗
E λ ∥ e ∥ G 2 = E λ tr ( G e e T ) = tr ( G Cov λ ( e ) ) . \mathbb{E}^{\lambda}\|e\|_G^2=\mathbb{E}^{\lambda}\operatorname{tr}(Gee^{\mathsf{T}})=\operatorname{tr}\left(G\operatorname{Cov}^{\lambda}(e)\right). E λ ∥ e ∥ G 2 = E λ tr ( G e e T ) = tr ( G Cov λ ( e ) ) . To differentiate K \mathcal{K} K G ↦ − log det G G\mapsto-\log\det G G ↦ − log det G S ( m ) S(m) S ( m ) D G [ Δ G ] D_G[\Delta_G] D G [ Δ G ] Δ G ∈ Sym ( m , R ) \Delta_G\in\operatorname{Sym}(m,\mathbb{R}) Δ G ∈ Sym ( m , R ) [36 ] that
D G [ Δ G ] log det G = tr ( G − 1 Δ G ) . D_G[\Delta_G]\log\det G=\operatorname{tr}(G^{-1}\Delta_G). D G [ Δ G ] log det G = tr ( G − 1 Δ G ) . Plugging these observations into the definition of K \mathcal{K} K
D G [ Δ G ] K ( ξ ∗ , G ) = 1 2 tr [ ( Cov λ ( e ) − G − 1 ) Δ G ] . D_G[\Delta_G]\mathcal{K}(\xi_*,G)=\frac{1}{2}\operatorname{tr}\left[\left(\operatorname{Cov}^{\lambda}(e)-G^{-1}\right)\Delta_G\right]. D G [ Δ G ] K ( ξ ∗ , G ) = 2 1 tr [ ( Cov λ ( e ) − G − 1 ) Δ G ] . This derivative vanishes for G ∗ G_* G ∗ D G 2 [ Δ G , Δ G ] D_G^2[\Delta_G,\Delta_G] D G 2 [ Δ G , Δ G ] Δ G \Delta_G Δ G [36 Section 2.2] we get for Δ G ≠ 0 \Delta_G\ne0 Δ G = 0
D G 2 [ Δ G , Δ G ] K ( ξ ∗ , G ) = 1 2 tr ( G − 1 Δ G G − 1 Δ G ) > 0 , D_G^2[\Delta_G,\Delta_G]\mathcal{K}(\xi_*,G)=\frac{1}{2}\operatorname{tr}\left(G^{-1}\Delta_GG^{-1}\Delta_G\right)>0, D G 2 [ Δ G , Δ G ] K ( ξ ∗ , G ) = 2 1 tr ( G − 1 Δ G G − 1 Δ G ) > 0 , for every G ∈ S ( m ) G\in S(m) G ∈ S ( m ) K \mathcal{K} K G G G ξ \xi ξ ξ ∗ \xi_* ξ ∗ G G G ( ξ ∗ , G ∗ ) (\xi_*,G_*) ( ξ ∗ , G ∗ )
What Theorem 2.2 indicates is that the EEA approach indeed provides best possible likelihood approximation when averaged over the prior. However, as we will later see, this can differ from the best possible approximation in terms of posterior distributions. We now define the objective functional J \mathcal{J} J R m × S ( m ) \mathbb{R}^{m} \times\mathcal{S}(m) R m × S ( m )
J : R m × S ( m ) → R , J ( ξ , G ) = − E λ log ( d μ ^ ( ⋅ ∣ y ; ξ , G ) d μ 0 ( x ) ) . (8) \mathcal{J}:\mathbb{R}^{m} \times\mathcal{S}(m) \to\mathbb{R},\quad\mathcal{J}(\xi,G)=-\mathbb{E}^{\lambda}\log\left(\frac{\mathrm{d}\hat{\mu}(\cdot\mid y;\xi,G)}{\mathrm{d}\mu_{0}}(x)\right). \tag*{(8)} J : R m × S ( m ) → R , J ( ξ , G ) = − E λ log ( d μ 0 d μ ^ ( ⋅ ∣ y ; ξ , G ) ( x ) ) . (8) The functional J \mathcal{J} J
Lemma 2.3. Let Assumption 1 hold. It holds for any ( ξ , G ) ∈ R m × S ( m ) (\xi,G) \in\mathbb{R}^{m} \times\mathcal{S}(m) ( ξ , G ) ∈ R m × S ( m )
E p KL ( μ ( ⋅ ∣ y ) ∥ μ ^ ( ⋅ ∣ y ; ξ , G ) ) = E p KL ( μ ( ⋅ ∣ y ) ∥ μ 0 ) + J ( ξ , G ) . (9) \mathbb{E}^{p}\operatorname{KL}\left(\mu(\cdot\mid y)\mathrel{\|}\hat{\mu}(\cdot\mid y;\xi,G)\right)=\mathbb{E}^{p}\operatorname{KL}\left(\mu(\cdot\mid y)\mathrel{\|}\mu_{0}\right)+\mathcal{J}(\xi,G). \tag*{(9)} E p KL ( μ ( ⋅ ∣ y ) ∥ μ ^ ( ⋅ ∣ y ; ξ , G ) ) = E p KL ( μ ( ⋅ ∣ y ) ∥ μ 0 ) + J ( ξ , G ) . (9) Proof of Lemma 2.3 . By definition of KL divergence:
E p KL ( μ ( ⋅ ∣ y ) ∥ μ ^ ( ⋅ ∣ y ; ξ , G ) ) = E λ log ( d μ ( ⋅ ∣ y ) d μ ^ ( ⋅ ∣ y ; ξ , G ) ( x ) ) . (10) \mathbb{E}^{p}\operatorname{KL}\left(\mu(\cdot\mid y)\mathrel{\|}\hat{\mu}(\cdot\mid y;\xi,G)\right)=\mathbb{E}^{\lambda}\log\left(\frac{\mathrm{d}\mu(\cdot\mid y)}{\mathrm{d}\hat{\mu}(\cdot\mid y;\xi,G)}(x)\right). \tag*{(10)} E p KL ( μ ( ⋅ ∣ y ) ∥ μ ^ ( ⋅ ∣ y ; ξ , G ) ) = E λ log ( d μ ^ ( ⋅ ∣ y ; ξ , G ) d μ ( ⋅ ∣ y ) ( x ) ) . (10) Note that all measures μ 0 , μ ( ⋅ ∣ y ) , μ ^ ( ⋅ ∣ y ; ξ , G ) \mu_{0},\mu(\cdot\mid y),\hat{\mu}(\cdot\mid y;\xi,G) μ 0 , μ ( ⋅ ∣ y ) , μ ^ ( ⋅ ∣ y ; ξ , G ) X X X y ∈ R m y \in\mathbb{R}^{m} y ∈ R m ξ ∈ R m \xi\in\mathbb{R}^{m} ξ ∈ R m G ∈ S ( m ) G \in\mathcal{S}(m) G ∈ S ( m )
log ( d μ ( ⋅ ∣ y ) d μ ^ ( ⋅ ∣ y ; ξ , G ) ( x ) ) = log ( d μ ( ⋅ ∣ y ) d μ 0 ( x ) ) − log ( d μ ^ ( ⋅ ∣ y ; ξ , G ) d μ 0 ( x ) ) . (11) \log\left(\frac{\mathrm{d}\mu(\cdot\mid y)}{\mathrm{d}\hat{\mu}(\cdot\mid y;\xi,G)}(x)\right)=\log\left(\frac{\mathrm{d}\mu(\cdot\mid y)}{\mathrm{d}\mu_{0}}(x)\right)-\log\left(\frac{\mathrm{d}\hat{\mu}(\cdot\mid y;\xi,G)}{\mathrm{d}\mu_{0}}(x)\right). \tag*{(11)} log ( d μ ^ ( ⋅ ∣ y ; ξ , G ) d μ ( ⋅ ∣ y ) ( x ) ) = log ( d μ 0 d μ ( ⋅ ∣ y ) ( x ) ) − log ( d μ 0 d μ ^ ( ⋅ ∣ y ; ξ , G ) ( x ) ) . (11) Plugging into (10) yields the claim. By Lemma 2.4 ahead, both quantities on the right-hand side are finite under Assumption 1. □
Since the term E p KL ( μ ( ⋅ ∣ y ) ∥ μ 0 ) \mathbb{E}^{p}\operatorname{KL}(\mu(\cdot\mid y)\mathrel{\|}\mu_{0}) E p KL ( μ ( ⋅ ∣ y ) ∥ μ 0 ) ξ \xi ξ G G G (9) also minimise J \mathcal{J} J J \mathcal{J} J J \mathcal{J} J
Lemma 2.4 (J \mathcal{J} J Suppose Assumption 1 holds. Then it holds for every ( ξ , G ) ∈ R m × S ( m ) (\xi,G) \in\mathbb{R}^{m} \times\mathcal{S}(m) ( ξ , G ) ∈ R m × S ( m )
− ∞ < − E p KL ( μ ( ⋅ ∣ y ) ∥ μ 0 ) ≤ J ( ξ , G ) < ∞ . -\infty<-\mathbb{E}^{p}\operatorname{KL}\left(\mu(\cdot\mid y)\mathrel{\|}\mu_{0}\right)\leq\mathcal{J}(\xi,G)<\infty. − ∞ < − E p KL ( μ ( ⋅ ∣ y ) ∥ μ 0 ) ≤ J ( ξ , G ) < ∞. Proof. We begin with the lower bound. By (11) we have
J ( ξ , G ) ≥ − E p KL ( μ ( ⋅ ∣ y ) ∥ μ 0 ) . \mathcal{J}(\xi,G)\geq-\mathbb{E}^{p}\operatorname{KL}\left(\mu(\cdot\mid y)\mathrel{\|}\mu_{0}\right). J ( ξ , G ) ≥ − E p KL ( μ ( ⋅ ∣ y ) ∥ μ 0 ) . By straightforward calculations
E p KL ( μ ( ⋅ ∣ y ) ∥ μ 0 ) = ∬ X × R m log p ( y ∣ x ) p ( y ) λ ( d x , d y ) = − m 2 log ( 2 π δ 2 ) − 1 2 δ 2 E λ ∥ A ( x ) − y ∥ 2 − ∫ R m p ( y ) log p ( y ) d y . \begin{aligned}
\mathbb{E}^{p}\operatorname{KL}\left(\mu(\cdot\mid y)\mathrel{\|}\mu_{0}\right)
&=\iint_{X\times\mathbb{R}^{m}}\log\frac{p(y\mid x)}{p(y)}\,\lambda(\mathrm{d}x,\mathrm{d}y)\\
&=-\frac{m}{2}\log(2\pi\delta^{2})-\frac{1}{2\delta^{2}}\mathbb{E}^{\lambda}\left\|A(x)-y\right\|^{2}-\int_{\mathbb{R}^{m}}p(y)\log p(y)\,\mathrm{d}y.
\end{aligned} E p KL ( μ ( ⋅ ∣ y ) ∥ μ 0 ) = ∬ X × R m log p ( y ) p ( y ∣ x ) λ ( d x , d y ) = − 2 m log ( 2 π δ 2 ) − 2 δ 2 1 E λ ∥ A ( x ) − y ∥ 2 − ∫ R m p ( y ) log p ( y ) d y . The second term equals − m 2 -\frac{m}{2} − 2 m (1) and the final term of right-hand side is finite by [16 Lemma 3.11] . It is used that p ( y ) = c m Z ^ ( y ) p(y)=c_{m}\hat{Z}(y) p ( y ) = c m Z ^ ( y )
J ( ξ , G ) = 1 2 E λ ∥ A ^ ( x ) − y + ξ ∥ G 2 + E p log Z ^ ( y ; ξ , G ) . (12) \mathcal{J}(\xi,G)=\frac{1}{2}\mathbb{E}^{\lambda}\left\|\hat{A}(x)-y+\xi\right\|_{G}^{2}+\mathbb{E}^{p}\log\hat{Z}(y;\xi,G). \tag*{(12)} J ( ξ , G ) = 2 1 E λ A ^ ( x ) − y + ξ G 2 + E p log Z ^ ( y ; ξ , G ) . (12) where
Z ^ ( y ; ξ , G ) = E μ 0 exp ( − 1 2 ∥ A ^ ( x ′ ) − y + ξ ∥ G 2 ) . \hat{Z}(y;\xi,G)=\mathbb{E}^{\mu_{0}}\exp\left(-\frac{1}{2}\left\|\hat{A}(x')-y+\xi\right\|_{G}^{2}\right). Z ^ ( y ; ξ , G ) = E μ 0 exp ( − 2 1 A ^ ( x ′ ) − y + ξ G 2 ) . The first term in (12) is bounded by an application of triangle inequality in conjunction with [16 Corollary 3.8] , making the observation that λ \lambda λ ( x , y ) (x,y) ( x , y ) [16 Lemma 3.11] . □
The following theorem establishes that the objective function is smooth.
Theorem 2.5 ( J \mathcal{J} J Suppose Assumption 1 holds. The objective functional J \mathcal{J} J
( D η γ J ) ( ξ , G ) = − E λ [ D η γ log ( d μ ^ ( ⋅ ∣ y ; ξ , G ) d μ 0 ( x ) ) ] , (D_{\eta}^{\gamma}\mathcal{J})(\xi,G)=-\mathbb{E}^{\lambda}\left[D_{\eta}^{\gamma}\log\left(\frac{d\widehat{\mu}(\cdot\mid y;\xi,G)}{d\mu_{0}}(x)\right)\right], ( D η γ J ) ( ξ , G ) = − E λ [ D η γ log ( d μ 0 d μ ( ⋅ ∣ y ; ξ , G ) ( x ) ) ] , where γ = 1 , 2 \gamma=1,2 γ = 1 , 2 η ∈ { ξ , G } γ \eta\in\{\xi,G\}^{\gamma} η ∈ { ξ , G } γ
The following lemma will be helpful to establish dominating functions to justify differentiation under the integral (see [14 Theorem 2.27] ) in our setting.
Lemma 2.6. Let Assumption 1 hold. Then it holds that for any compact U ⊂ R m × S ( m ) U\subset\mathbb{R}^{m}\times\mathcal{S}(m) U ⊂ R m × S ( m ) k ∈ N k\in\mathbb{N} k ∈ N C ( U , k ) < ∞ C(U,k)<\infty C ( U , k ) < ∞
sup ( ξ , G ) ∈ U E P E μ ^ ( ⋅ ∣ y ; ξ , G ) ( ∥ x ∥ k + ∥ y ∥ k ) ≤ C ( U , k ) . \sup_{(\xi,G)\in U}\mathbb{E}^{\mathbb{P}}\mathbb{E}^{\widehat{\mu}(\cdot\mid y;\xi,G)}\left(\lVert x\rVert^{k}+\lVert y\rVert^{k}\right)\le C(U,k). ( ξ , G ) ∈ U sup E P E μ ( ⋅ ∣ y ; ξ , G ) ( ∥ x ∥ k + ∥ y ∥ k ) ≤ C ( U , k ) . Proof. We show that this follows from the arguments of [43 ] . Notice first by [16 Lemma 3.10] applied to ( y − ξ ) (y-\xi) ( y − ξ ) ( ξ , G ) ∈ U (\xi,G)\in U ( ξ , G ) ∈ U ε > 0 \varepsilon>0 ε > 0 C ( U ) C(U) C ( U ) κ ( ε , U ) > 0 \kappa(\varepsilon,U)>0 κ ( ε , U ) > 0
∥ A ^ ( x ) − y + ξ ∥ G 2 ≤ C ( U ) ( 1 + ∥ x ∥ 2 2 + ∥ y − ξ ∥ 2 2 ) ≤ 2 C ( U ) ( 1 + ∥ ξ ∥ 2 2 + ∥ x ∥ 2 2 + ∥ y ∥ 2 2 ) . \left\lVert\widehat{A}(x)-y+\xi\right\rVert_{G}^{2}\le C(U)\left(1+\lVert x\rVert_{2}^{2}+\lVert y-\xi\rVert_{2}^{2}\right)\le2C(U)\left(1+\lVert\xi\rVert_{2}^{2}+\lVert x\rVert_{2}^{2}+\lVert y\rVert_{2}^{2}\right). A ( x ) − y + ξ G 2 ≤ C ( U ) ( 1 + ∥ x ∥ 2 2 + ∥ y − ξ ∥ 2 2 ) ≤ 2 C ( U ) ( 1 + ∥ ξ ∥ 2 2 + ∥ x ∥ 2 2 + ∥ y ∥ 2 2 ) . By [16 Lemma 3.10] and Young’s inequality
− C ( U ) − κ ( ε , U ) ∥ ξ ∥ 2 − ε ∥ x ∥ 2 2 + κ ( ε , U ) 2 ∥ y ∥ 2 ≤ − C ( U ) − ε ∥ x ∥ 2 2 + κ ( ε , U ) ∥ y − ξ ∥ 2 ≤ ∥ A ^ ( x ) − y + ξ ∥ G 2 . -C(U)-\kappa(\varepsilon,U)\lVert\xi\rVert^{2}-\varepsilon\lVert x\rVert_{2}^{2}+\frac{\kappa(\varepsilon,U)}{2}\lVert y\rVert^{2}\le-C(U)-\varepsilon\lVert x\rVert_{2}^{2}+\kappa(\varepsilon,U)\lVert y-\xi\rVert^{2}\le\left\lVert\widehat{A}(x)-y+\xi\right\rVert_{G}^{2}. − C ( U ) − κ ( ε , U ) ∥ ξ ∥ 2 − ε ∥ x ∥ 2 2 + 2 κ ( ε , U ) ∥ y ∥ 2 ≤ − C ( U ) − ε ∥ x ∥ 2 2 + κ ( ε , U ) ∥ y − ξ ∥ 2 ≤ A ( x ) − y + ξ G 2 . Hence the Assumptions of [43 Lemma A.3] are satisfied and we may use that there exists q > 1 q>1 q > 1
sup ( ξ , G ) ∈ U E λ ( d μ ^ ( ⋅ ∣ y ; ξ , G ) d μ ( ⋅ ∣ y ) ) q < ∞ . \sup_{(\xi,G)\in U}\mathbb{E}^{\lambda}\left(\frac{d\widehat{\mu}(\cdot\mid y;\xi,G)}{d\mu(\cdot\mid y)}\right)^{q}<\infty. ( ξ , G ) ∈ U sup E λ ( d μ ( ⋅ ∣ y ) d μ ( ⋅ ∣ y ; ξ , G ) ) q < ∞. We can now prove the claim by using Hölder’s inequality for q q q r = q q − 1 r=\frac{q}{q-1} r = q − 1 q
sup ( ξ , G ) ∈ U E P E μ ^ ( ⋅ ∣ y ; ξ , G ) ( ∥ x ∥ k + ∥ y ∥ k ) = sup ( ξ , G ) ∈ U E λ ( ( ∥ x ∥ k + ∥ y ∥ k ) d μ ^ ( ⋅ ∣ y ; ξ , G ) d μ ( ⋅ ∣ y ) ) ≤ ( E λ ( ∥ x ∥ k + ∥ y ∥ k ) r ) 1 / r sup ( ξ , G ) ∈ U ( E λ ( d μ ^ ( ⋅ ∣ y ; ξ , G ) d μ ( ⋅ ∣ y ) ) q ) 1 / q . \begin{aligned}
\sup_{(\xi,G)\in U}\mathbb{E}^{\mathbb{P}}\mathbb{E}^{\widehat{\mu}(\cdot\mid y;\xi,G)}\left(\lVert x\rVert^{k}+\lVert y\rVert^{k}\right)
&=\sup_{(\xi,G)\in U}\mathbb{E}^{\lambda}\left(\left(\lVert x\rVert^{k}+\lVert y\rVert^{k}\right)\frac{d\widehat{\mu}(\cdot\mid y;\xi,G)}{d\mu(\cdot\mid y)}\right)\\
&\le\left(\mathbb{E}^{\lambda}\left(\lVert x\rVert^{k}+\lVert y\rVert^{k}\right)^{r}\right)^{1/r}
\sup_{(\xi,G)\in U}\left(\mathbb{E}^{\lambda}\left(\frac{d\widehat{\mu}(\cdot\mid y;\xi,G)}{d\mu(\cdot\mid y)}\right)^{q}\right)^{1/q}.
\end{aligned} ( ξ , G ) ∈ U sup E P E μ ( ⋅ ∣ y ; ξ , G ) ( ∥ x ∥ k + ∥ y ∥ k ) = ( ξ , G ) ∈ U sup E λ ( ( ∥ x ∥ k + ∥ y ∥ k ) d μ ( ⋅ ∣ y ) d μ ( ⋅ ∣ y ; ξ , G ) ) ≤ ( E λ ( ∥ x ∥ k + ∥ y ∥ k ) r ) 1/ r ( ξ , G ) ∈ U sup ( E λ ( d μ ( ⋅ ∣ y ) d μ ( ⋅ ∣ y ; ξ , G ) ) q ) 1/ q . Proof of Theorem 2.5. We compute the Fréchet derivatives D η γ [ Δ η ] D_{\eta}^{\gamma}[\Delta_{\eta}] D η γ [ Δ η ] Δ η \Delta_{\eta} Δ η
f x , y ( ξ , G ) : = − log ( d μ ^ ( ⋅ ∣ y ; ξ , G ) d μ 0 ( x ) ) = 1 2 ∥ A ^ ( x ) − y + ξ ∥ G 2 + log E μ 0 exp ( − 1 2 ∥ A ^ ( x ′ ) − y + ξ ∥ G 2 ) . f_{x,y}(\xi,G):=-\log\left(\frac{d\widehat{\mu}(\cdot\mid y;\xi,G)}{d\mu_{0}}(x)\right)=\frac{1}{2}\left\lVert\widehat{A}(x)-y+\xi\right\rVert_{G}^{2}+\log\mathbb{E}^{\mu_{0}}\exp\left(-\frac{1}{2}\left\lVert\widehat{A}(x')-y+\xi\right\rVert_{G}^{2}\right). f x , y ( ξ , G ) := − log ( d μ 0 d μ ( ⋅ ∣ y ; ξ , G ) ( x ) ) = 2 1 A ( x ) − y + ξ G 2 + log E μ 0 exp ( − 2 1 A ( x ′ ) − y + ξ G 2 ) . Introduce the notation
g x , y ( ξ , G ; Δ ξ ) = ( A ^ ( x ) − y + ξ ) ⊤ G Δ ξ h x , y ( ξ , G ; Δ G ) = 1 2 ∥ A ^ ( x ) − y + ξ ∥ Δ G 2 k x , y ( ξ ; Δ ξ , Δ G ) = ( A ^ ( x ) − y + ξ ) ⊤ Δ G Δ ξ \begin{aligned}
g_{x,y}(\xi,G;\Delta_{\xi})&=\left(\widehat{A}(x)-y+\xi\right)^{\top}G\Delta_{\xi}\\
h_{x,y}(\xi,G;\Delta_{G})&=\frac{1}{2}\left\lVert\widehat{A}(x)-y+\xi\right\rVert_{\Delta_{G}}^{2}\\
k_{x,y}(\xi;\Delta_{\xi},\Delta_{G})&=\left(\widehat{A}(x)-y+\xi\right)^{\top}\Delta_{G}\Delta_{\xi}
\end{aligned} g x , y ( ξ , G ; Δ ξ ) h x , y ( ξ , G ; Δ G ) k x , y ( ξ ; Δ ξ , Δ G ) = ( A ( x ) − y + ξ ) ⊤ G Δ ξ = 2 1 A ( x ) − y + ξ Δ G 2 = ( A ( x ) − y + ξ ) ⊤ Δ G Δ ξ and notice that for C C C
∣ g x , y ( ξ , G ; Δ ξ ) ∣ + ∣ h x , y ( ξ , G ; Δ G ) ∣ + ∣ k x , y ( ξ ; Δ ξ , Δ G ) ∣ + ∥ A ^ ( x ) − y + ξ ∥ G 2 ≤ C ( 1 + ∥ Δ G ∥ + ∥ Δ ξ ∥ + ∥ Δ G ∥ ∥ Δ ξ ∥ ) ( ( ∥ x ∥ + 1 ) 2 + ( ∥ y ∥ + 1 ) 2 + ( ∥ ξ ∥ + 1 ) 2 ) . (13) \begin{aligned}
&|g_{x,y}(\xi,G;\Delta_{\xi})|+|h_{x,y}(\xi,G;\Delta_{G})|+|k_{x,y}(\xi;\Delta_{\xi},\Delta_{G})|+\left\lVert\widehat{A}(x)-y+\xi\right\rVert_{G}^{2}\\
&\le C(1+\lVert\Delta_{G}\rVert+\lVert\Delta_{\xi}\rVert+\lVert\Delta_{G}\rVert\lVert\Delta_{\xi}\rVert)\left((\lVert x\rVert+1)^{2}+(\lVert y\rVert+1)^{2}+(\lVert\xi\rVert+1)^{2}\right). \tag*{(13)}
\end{aligned} ∣ g x , y ( ξ , G ; Δ ξ ) ∣ + ∣ h x , y ( ξ , G ; Δ G ) ∣ + ∣ k x , y ( ξ ; Δ ξ , Δ G ) ∣ + A ( x ) − y + ξ G 2 ≤ C ( 1 + ∥ Δ G ∥ + ∥ Δ ξ ∥ + ∥ Δ G ∥ ∥ Δ ξ ∥) ( (∥ x ∥ + 1 ) 2 + (∥ y ∥ + 1 ) 2 + (∥ ξ ∥ + 1 ) 2 ) . (13) where [16 Corollary 3.8] was used to bound ∣ A ^ ( x ) ∣ \lvert\widehat{A}(x)\rvert ∣ A ( x )∣ ∥ x ∥ + 1 \lVert x\rVert+ 1 ∥ x ∥ + 1
D ξ [ Δ ξ ] Z ^ ( y ; ξ , G ) = − E μ 0 [ g x ′ , y ( ξ , G ; Δ ξ ) exp ( − 1 2 ∥ A ^ ( x ′ ) − y + ξ ∥ G 2 ) ] D_{\xi}[\Delta_{\xi}]\widehat{Z}(y;\xi,G)=-\mathbb{E}^{\mu_{0}}\left[g_{x',y}(\xi,G;\Delta_{\xi})\exp\left(-\frac{1}{2}\left\lVert\widehat{A}(x')-y+\xi\right\rVert_{G}^{2}\right)\right] D ξ [ Δ ξ ] Z ( y ; ξ , G ) = − E μ 0 [ g x ′ , y ( ξ , G ; Δ ξ ) exp ( − 2 1 A ( x ′ ) − y + ξ G 2 ) ] D G [ Δ G ] Z ^ ( y ; ξ , G ) = − E μ 0 [ h x ′ , y ( ξ , G ; Δ G ) exp ( − 1 2 ∥ A ^ ( x ′ ) − y + ξ ∥ G 2 ) ] . D_{G}[\Delta_{G}]\widehat{Z}(y;\xi,G)=-\mathbb{E}^{\mu_{0}}\left[h_{x',y}(\xi,G;\Delta_{G})\exp\left(-\frac{1}{2}\left\lVert\widehat{A}(x')-y+\xi\right\rVert_{G}^{2}\right)\right]. D G [ Δ G ] Z ( y ; ξ , G ) = − E μ 0 [ h x ′ , y ( ξ , G ; Δ G ) exp ( − 2 1 A ( x ′ ) − y + ξ G 2 ) ] . Notice that the exponential term is bounded by one and g , h g,h g , h (13) . We can find a dominating function (locally in ( ξ , G ) (\xi,G) ( ξ , G ) μ 0 \mu_{0} μ 0
The following derivatives may be taken under the integral with respect to λ \lambda λ (13) together with Lemma 2.6 and since λ \lambda λ
D ξ [ Δ ξ ] f x , y ( ξ , G ) = g x , y ( ξ , G ; Δ ξ ) − E μ ^ ( ⋅ ∣ y ; ξ , G ) g x ′ , y ( ξ , G ; Δ ξ ) D G [ Δ G ] f x , y ( ξ , G ) = h x , y ( ξ , G ; Δ G ) − E μ ^ ( ⋅ ∣ y ; ξ , G ) h x ′ , y ( ξ , G ; Δ G ) . (14) \begin{aligned}
D_{\xi}[\Delta_{\xi}]f_{x,y}(\xi,G)&=g_{x,y}(\xi,G;\Delta_{\xi})-\mathbb{E}^{\widehat{\mu}(\cdot\mid y;\xi,G)}g_{x',y}(\xi,G;\Delta_{\xi})\\
D_{G}[\Delta_{G}]f_{x,y}(\xi,G)&=h_{x,y}(\xi,G;\Delta_{G})-\mathbb{E}^{\widehat{\mu}(\cdot\mid y;\xi,G)}h_{x',y}(\xi,G;\Delta_{G}). \tag*{(14)}
\end{aligned} D ξ [ Δ ξ ] f x , y ( ξ , G ) D G [ Δ G ] f x , y ( ξ , G ) = g x , y ( ξ , G ; Δ ξ ) − E μ ( ⋅ ∣ y ; ξ , G ) g x ′ , y ( ξ , G ; Δ ξ ) = h x , y ( ξ , G ; Δ G ) − E μ ( ⋅ ∣ y ; ξ , G ) h x ′ , y ( ξ , G ; Δ G ) . (14) We move on to the second order derivatives and justify differentiation under the integral by using the estimate (13) giving a polynomial bound for the integrands together with Lemma 2.6.
D ξ , ξ 2 [ Δ ξ , Δ ξ ] f x , y ( ξ , G ) = Var μ ^ ( ⋅ ∣ y ; ξ , G ) g x ′ , y ( ξ , G ; Δ ξ ) D G , G 2 [ Δ G , Δ G ] f x , y ( ξ , G ) = Var μ ^ ( ⋅ ∣ y ; ξ , G ) h x ′ , y ( ξ , G ; Δ G ) D ξ , G 2 [ Δ ξ , Δ G ] f x , y ( ξ , G ) = k x , y ( ξ ; Δ ξ , Δ G ) − E μ ^ ( ⋅ ∣ y ; ξ , G ) k x ′ , y ( ξ ; Δ ξ , Δ G ) + Cov μ ^ ( ⋅ ∣ y ; ξ , G ) ( g x ′ , y ( ξ , G ; Δ ξ ) , h x ′ , y ( ξ , G ; Δ G ) ) . (15) \begin{aligned}
D_{\xi,\xi}^{2}[\Delta_{\xi},\Delta_{\xi}]f_{x,y}(\xi,G)&=\operatorname{Var}^{\widehat{\mu}(\cdot\mid y;\xi,G)}g_{x',y}(\xi,G;\Delta_{\xi})\\
D_{G,G}^{2}[\Delta_{G},\Delta_{G}]f_{x,y}(\xi,G)&=\operatorname{Var}^{\widehat{\mu}(\cdot\mid y;\xi,G)}h_{x',y}(\xi,G;\Delta_{G})\\
D_{\xi,G}^{2}[\Delta_{\xi},\Delta_{G}]f_{x,y}(\xi,G)&=k_{x,y}(\xi;\Delta_{\xi},\Delta_{G})-\mathbb{E}^{\widehat{\mu}(\cdot\mid y;\xi,G)}k_{x',y}(\xi;\Delta_{\xi},\Delta_{G})\\
&\quad+\operatorname{Cov}^{\widehat{\mu}(\cdot\mid y;\xi,G)}\left(g_{x',y}(\xi,G;\Delta_{\xi}),h_{x',y}(\xi,G;\Delta_{G})\right). \tag*{(15)}
\end{aligned} D ξ , ξ 2 [ Δ ξ , Δ ξ ] f x , y ( ξ , G ) D G , G 2 [ Δ G , Δ G ] f x , y ( ξ , G ) D ξ , G 2 [ Δ ξ , Δ G ] f x , y ( ξ , G ) = Var μ ( ⋅ ∣ y ; ξ , G ) g x ′ , y ( ξ , G ; Δ ξ ) = Var μ ( ⋅ ∣ y ; ξ , G ) h x ′ , y ( ξ , G ; Δ G ) = k x , y ( ξ ; Δ ξ , Δ G ) − E μ ( ⋅ ∣ y ; ξ , G ) k x ′ , y ( ξ ; Δ ξ , Δ G ) + Cov μ ( ⋅ ∣ y ; ξ , G ) ( g x ′ , y ( ξ , G ; Δ ξ ) , h x ′ , y ( ξ , G ; Δ G ) ) . (15) Differentiation under the integral follows from the arguments above.
Theorem 2.5 gives a framework to express and analyse both the first- and second-order optimality conditions through the study of stationary points. In addition to local optima, we next establish a convexity property of J \mathcal{J} J
Theorem 2.7. Suppose Assumption 1 holds. The functional J \mathcal{J} J ( ξ , G ) (\xi,G) ( ξ , G ) ( ξ ^ , G ^ ) ∈ R m × S ( m ) (\widehat{\xi},\widehat{G})\in\mathbb{R}^{m}\times\mathcal{S}(m) ( ξ , G ) ∈ R m × S ( m ) ( ξ ^ , G ^ ) (\widehat{\xi},\widehat{G}) ( ξ , G )
E μ 0 [ A ^ ( x ) ] = E P μ ^ ( ⋅ ∣ y ; ξ ^ , G ^ ) [ A ^ ( x ′ ) ] \mathbb{E}^{\mu_{0}}\left[\widehat{A}(x)\right]=\mathbb{E}^{\mathbb{P}\widehat{\mu}(\cdot\mid y;\widehat{\xi},\widehat{G})}\left[\widehat{A}(x')\right] E μ 0 [ A ( x ) ] = E P μ ( ⋅ ∣ y ; ξ , G ) [ A ( x ′ ) ] E λ [ A ^ ( x ) − y + ξ ^ ] ⊗ [ A ^ ( x ) − y + ξ ^ ] = E P μ ^ ( ⋅ ∣ y ; ξ ^ , G ^ ) [ A ^ ( x ′ ) − y + ξ ^ ] ⊗ [ A ^ ( x ′ ) − y + ξ ^ ] . \mathbb{E}^{\lambda}\left[\widehat{A}(x)-y+\widehat{\xi}\right]\otimes\left[\widehat{A}(x)-y+\widehat{\xi}\right]=\mathbb{E}^{\mathbb{P}\widehat{\mu}(\cdot\mid y;\widehat{\xi},\widehat{G})}\left[\widehat{A}(x')-y+\widehat{\xi}\right]\otimes\left[\widehat{A}(x')-y+\widehat{\xi}\right]. E λ [ A ( x ) − y + ξ ] ⊗ [ A ( x ) − y + ξ ] = E P μ ( ⋅ ∣ y ; ξ , G ) [ A ( x ′ ) − y + ξ ] ⊗ [ A ( x ′ ) − y + ξ ] . Proof. Bi-convexity of J \mathcal{J} J (15) : the second derivatives D ξ , ξ 2 D_{\xi,\xi}^{2} D ξ , ξ 2 D G , G 2 D_{G,G}^{2} D G , G 2 J \mathcal{J} J D ξ D_{\xi} D ξ D G D_{G} D G J \mathcal{J} J (14) :
D ξ [ Δ ξ ] J ( ξ , G ) = E λ [ ( A ^ ( x ) − y + ξ ) T G Δ ξ ] − E P μ ^ ( ⋅ ∣ y ; ξ , G ) [ ( A ^ ( x ′ ) − y + ξ ) T G Δ ξ ] = ( E μ 0 [ A ^ ( x ) ] − E P [ y ] + ξ − E P μ ^ ( ⋅ ∣ y ; ξ , G ) [ A ^ ( x ′ ) ] + E P [ y ] − ξ ) T G Δ ξ = ( E μ 0 [ A ^ ( x ) ] − E P μ ^ ( ⋅ ∣ y ; ξ , G ) [ A ^ ( x ′ ) ] ) T G Δ ξ = ! 0. (16) \begin{aligned}
D_{\xi}[\Delta_{\xi}]\mathcal{J}(\xi,G)&=\mathbb{E}^{\lambda}\left[\left(\widehat{A}(x)-y+\xi\right)^{\mathsf{T}}G\Delta_{\xi}\right]-\mathbb{E}^{\mathbb{P}\widehat{\mu}(\cdot\mid y;\xi,G)}\left[\left(\widehat{A}(x')-y+\xi\right)^{\mathsf{T}}G\Delta_{\xi}\right]\\
&=\left(\mathbb{E}^{\mu_{0}}\left[\widehat{A}(x)\right]-\mathbb{E}^{\mathbb{P}}[y]+\xi-\mathbb{E}^{\mathbb{P}\widehat{\mu}(\cdot\mid y;\xi,G)}\left[\widehat{A}(x')\right]+\mathbb{E}^{\mathbb{P}}[y]-\xi\right)^{\mathsf{T}}G\Delta_{\xi}\\
&=\left(\mathbb{E}^{\mu_{0}}\left[\widehat{A}(x)\right]-\mathbb{E}^{\mathbb{P}\widehat{\mu}(\cdot\mid y;\xi,G)}\left[\widehat{A}(x')\right]\right)^{\mathsf{T}}G\Delta_{\xi}\overset{!}{=}0. \tag*{(16)}
\end{aligned} D ξ [ Δ ξ ] J ( ξ , G ) = E λ [ ( A ( x ) − y + ξ ) T G Δ ξ ] − E P μ ( ⋅ ∣ y ; ξ , G ) [ ( A ( x ′ ) − y + ξ ) T G Δ ξ ] = ( E μ 0 [ A ( x ) ] − E P [ y ] + ξ − E P μ ( ⋅ ∣ y ; ξ , G ) [ A ( x ′ ) ] + E P [ y ] − ξ ) T G Δ ξ = ( E μ 0 [ A ( x ) ] − E P μ ( ⋅ ∣ y ; ξ , G ) [ A ( x ′ ) ] ) T G Δ ξ = ! 0. (16) For the other derivative:
D G [ Δ G ] J ( ξ , G ) = 1 2 ( E λ [ ∥ A ^ ( x ) − y + ξ ∥ Δ G 2 ] − E P μ ^ ( ⋅ ∣ y ; ξ , G ) [ ∥ A ^ ( x ′ ) − y + ξ ∥ Δ G 2 ] ) = ! 0. (17) D_{G}[\Delta_{G}]\mathcal{J}(\xi,G)=\frac{1}{2}\left(\mathbb{E}^{\lambda}\left[\left\lVert\widehat{A}(x)-y+\xi\right\rVert_{\Delta_{G}}^{2}\right]-\mathbb{E}^{\mathbb{P}\widehat{\mu}(\cdot\mid y;\xi,G)}\left[\left\lVert\widehat{A}(x')-y+\xi\right\rVert_{\Delta_{G}}^{2}\right]\right)\overset{!}{=}0. \tag*{(17)} D G [ Δ G ] J ( ξ , G ) = 2 1 ( E λ [ A ( x ) − y + ξ Δ G 2 ] − E P μ ( ⋅ ∣ y ; ξ , G ) [ A ( x ′ ) − y + ξ Δ G 2 ] ) = ! 0. (17) We show now that for fixed G ∈ S ( m ) G\in\mathcal{S}(m) G ∈ S ( m ) ξ ^ \widehat{\xi} ξ J ( ⋅ , G ) \mathcal{J}(\cdot,G) J ( ⋅ , G ) J \mathcal{J} J ( Δ ξ , Δ G ) ∈ R m × Sym ( m , R ) (\Delta_{\xi},\Delta_{G})\in\mathbb{R}^{m}\times\operatorname{Sym}(m,\mathbb{R}) ( Δ ξ , Δ G ) ∈ R m × Sym ( m , R ) D [ Δ ξ , Δ G ] 2 D^2_{[\Delta_\xi,\Delta_G]} D [ Δ ξ , Δ G ] 2 J \mathcal{J} J ( ξ ^ , G ) (\hat{\xi},G) ( ξ ^ , G )
D [ Δ ξ , Δ G ] 2 J ( ξ ^ , G ) = D ξ , ξ 2 [ Δ ξ , Δ ξ ] J ( ξ ^ , G ) + D G , G 2 [ Δ G , Δ G ] J ( ξ ^ , G ) + 2 D ξ , G 2 [ Δ ξ , Δ G ] J ( ξ ^ , G ) = E p Var μ ^ ( ⋅ ∣ y ; ξ ^ , G ) ( g x ′ , y ( ξ ^ , G ; Δ ξ ) ) + E p Var μ ^ ( ⋅ ∣ y ; ξ ^ , G ) ( h x ′ , y ( ξ ^ , G ; Δ G ) ) + 2 E λ k x , y ( ξ ^ ; Δ ξ , Δ G ) − 2 E p E μ ^ ( ⋅ ∣ y ; ξ ^ , G ) k x ′ , y ( ξ ^ ; Δ ξ , Δ G ) + 2 E p Cov μ ^ ( ⋅ ∣ y ; ξ ^ , G ) ( g x ′ , y ( ξ ^ , G ; Δ ξ ) , h x ′ , y ( ξ ^ , G ; Δ G ) ) . \begin{aligned}
D^2_{[\Delta_\xi,\Delta_G]}\mathcal{J}(\hat{\xi},G)
&=D^2_{\xi,\xi}[\Delta_\xi,\Delta_\xi]\mathcal{J}(\hat{\xi},G)+D^2_{G,G}[\Delta_G,\Delta_G]\mathcal{J}(\hat{\xi},G)+2D^2_{\xi,G}[\Delta_\xi,\Delta_G]\mathcal{J}(\hat{\xi},G)\\
&=\mathbb{E}^{p}\operatorname{Var}^{\hat{\mu}(\cdot\mid y;\hat{\xi},G)}(g_{x',y}(\hat{\xi},G;\Delta_\xi))+\mathbb{E}^{p}\operatorname{Var}^{\hat{\mu}(\cdot\mid y;\hat{\xi},G)}(h_{x',y}(\hat{\xi},G;\Delta_G))\\
&\quad+2\mathbb{E}^{\lambda}k_{x,y}(\hat{\xi};\Delta_\xi,\Delta_G)-2\mathbb{E}^{p}\mathbb{E}^{\hat{\mu}(\cdot\mid y;\hat{\xi},G)}k_{x',y}(\hat{\xi};\Delta_\xi,\Delta_G)\\
&\quad+2\mathbb{E}^{p}\operatorname{Cov}^{\hat{\mu}(\cdot\mid y;\hat{\xi},G)}\left(g_{x',y}(\hat{\xi},G;\Delta_\xi),h_{x',y}(\hat{\xi},G;\Delta_G)\right).
\end{aligned} D [ Δ ξ , Δ G ] 2 J ( ξ ^ , G ) = D ξ , ξ 2 [ Δ ξ , Δ ξ ] J ( ξ ^ , G ) + D G , G 2 [ Δ G , Δ G ] J ( ξ ^ , G ) + 2 D ξ , G 2 [ Δ ξ , Δ G ] J ( ξ ^ , G ) = E p Var μ ^ ( ⋅ ∣ y ; ξ ^ , G ) ( g x ′ , y ( ξ ^ , G ; Δ ξ )) + E p Var μ ^ ( ⋅ ∣ y ; ξ ^ , G ) ( h x ′ , y ( ξ ^ , G ; Δ G )) + 2 E λ k x , y ( ξ ^ ; Δ ξ , Δ G ) − 2 E p E μ ^ ( ⋅ ∣ y ; ξ ^ , G ) k x ′ , y ( ξ ^ ; Δ ξ , Δ G ) + 2 E p Cov μ ^ ( ⋅ ∣ y ; ξ ^ , G ) ( g x ′ , y ( ξ ^ , G ; Δ ξ ) , h x ′ , y ( ξ ^ , G ; Δ G ) ) . From the condition (16) and the definition of k k k ξ ^ \hat{\xi} ξ ^
E λ k x , y ( ξ ^ ; Δ ξ , Δ G ) = E p E μ ^ ( ⋅ ∣ y ; ξ ^ , G ) k x ′ , y ( ξ ^ ; Δ ξ , Δ G ) . \mathbb{E}^{\lambda}k_{x,y}(\hat{\xi};\Delta_\xi,\Delta_G)=\mathbb{E}^{p}\mathbb{E}^{\hat{\mu}(\cdot\mid y;\hat{\xi},G)}k_{x',y}(\hat{\xi};\Delta_\xi,\Delta_G). E λ k x , y ( ξ ^ ; Δ ξ , Δ G ) = E p E μ ^ ( ⋅ ∣ y ; ξ ^ , G ) k x ′ , y ( ξ ^ ; Δ ξ , Δ G ) . For this reason, the second derivative (evaluated at ( ξ ^ , G ) (\hat{\xi},G) ( ξ ^ , G )
D [ Δ ξ , Δ G ] 2 J ( ξ ^ , G ) = E p Var μ ^ ( ⋅ ∣ y ; ξ ^ , G ) g x ′ , y ( ξ ^ , G ; Δ ξ ) + E p Var μ ^ ( ⋅ ∣ y ; ξ ^ , G ) h x ′ , y ( ξ ^ , G ; Δ G ) + 2 E p Cov μ ^ ( ⋅ ∣ y ; ξ ^ , G ) ( g x ′ , y ( ξ ^ , G ; Δ ξ ) , h x ′ , y ( ξ ^ , G ; Δ G ) ) = E p Var μ ^ ( ⋅ ∣ y ; ξ ^ , G ) [ g x ′ , y ( ξ ^ , G ; Δ ξ ) + h x ′ , y ( ξ ^ , G ; Δ G ) ] ≥ 0. (18) \begin{aligned}
D^2_{[\Delta_\xi,\Delta_G]}\mathcal{J}(\hat{\xi},G)
&=\mathbb{E}^{p}\operatorname{Var}^{\hat{\mu}(\cdot\mid y;\hat{\xi},G)}g_{x',y}(\hat{\xi},G;\Delta_\xi)+\mathbb{E}^{p}\operatorname{Var}^{\hat{\mu}(\cdot\mid y;\hat{\xi},G)}h_{x',y}(\hat{\xi},G;\Delta_G)\\
&\quad+2\mathbb{E}^{p}\operatorname{Cov}^{\hat{\mu}(\cdot\mid y;\hat{\xi},G)}\left(g_{x',y}(\hat{\xi},G;\Delta_\xi),h_{x',y}(\hat{\xi},G;\Delta_G)\right)\\
&=\mathbb{E}^{p}\operatorname{Var}^{\hat{\mu}(\cdot\mid y;\hat{\xi},G)}\left[g_{x',y}(\hat{\xi},G;\Delta_\xi)+h_{x',y}(\hat{\xi},G;\Delta_G)\right]\geq0. \tag*{(18)}
\end{aligned} D [ Δ ξ , Δ G ] 2 J ( ξ ^ , G ) = E p Var μ ^ ( ⋅ ∣ y ; ξ ^ , G ) g x ′ , y ( ξ ^ , G ; Δ ξ ) + E p Var μ ^ ( ⋅ ∣ y ; ξ ^ , G ) h x ′ , y ( ξ ^ , G ; Δ G ) + 2 E p Cov μ ^ ( ⋅ ∣ y ; ξ ^ , G ) ( g x ′ , y ( ξ ^ , G ; Δ ξ ) , h x ′ , y ( ξ ^ , G ; Δ G ) ) = E p Var μ ^ ( ⋅ ∣ y ; ξ ^ , G ) [ g x ′ , y ( ξ ^ , G ; Δ ξ ) + h x ′ , y ( ξ ^ , G ; Δ G ) ] ≥ 0. (18) Theorem 2.7 provides an interesting characterisation of the critical points.
In the special case of linear models A ^ \hat{A} A ^ μ 0 \mu_0 μ 0 ξ ^ \hat{\xi} ξ ^ G ^ \hat{G} G ^ Theorem 3.1 it is shown that
ξ ^ = E λ ( y − A ^ ( x ) ) , \hat{\xi}=\mathbb{E}^{\lambda}\left(y-\hat{A}(x)\right), ξ ^ = E λ ( y − A ^ ( x ) ) , and hence ξ ^ \hat{\xi} ξ ^ J \mathcal{J} J K \mathcal{K} K
Linear approximations with Gaussian prior
In this section we specialise to the case of a Gaussian prior together with a linear surrogate model A ^ \hat{A} A ^ Z ^ \hat{Z} Z ^ § 5 ahead for details. Finally, the linear-Gaussian structure allows us to say more: in Theorem 3.1 below we derive a closed-form expression for J \mathcal{J} J ξ ^ \hat{\xi} ξ ^
Assume now that A ^ \hat{A} A ^ X = H X=H X = H μ 0 = N ( x ˉ , C 0 ) \mu_0=\mathcal{N}(\bar{x},C_0) μ 0 = N ( x ˉ , C 0 ) x ˉ ∈ H \bar{x}\in H x ˉ ∈ H C 0 ∈ L + ( H ) C_0\in L_+(H) C 0 ∈ L + ( H ) L + ( H ) L_+(H) L + ( H ) μ ^ ( ⋅ ∣ y ; ξ , G ) = N ( m ^ ( y ; ξ , G ) , C ^ ( G ) ) \hat{\mu}(\cdot\mid y;\xi,G)=\mathcal{N}(\hat{m}(y;\xi,G),\hat{C}(G)) μ ^ ( ⋅ ∣ y ; ξ , G ) = N ( m ^ ( y ; ξ , G ) , C ^ ( G ))
m ^ ( y ; ξ , G ) = x ˉ + C 0 A ^ ∗ ( G − 1 + A ^ C 0 A ^ ∗ ) − 1 ( y − A ^ x ˉ − ξ ) \hat{m}(y;\xi,G)=\bar{x}+C_0\hat{A}^*\left(G^{-1}+\hat{A}C_0\hat{A}^*\right)^{-1}(y-\hat{A}\bar{x}-\xi) m ^ ( y ; ξ , G ) = x ˉ + C 0 A ^ ∗ ( G − 1 + A ^ C 0 A ^ ∗ ) − 1 ( y − A ^ x ˉ − ξ ) C ^ ( G ) = C 0 − C 0 A ^ ∗ ( G − 1 + A ^ C 0 A ^ ∗ ) − 1 A ^ C 0 . (19) \hat{C}(G)=C_0-C_0\hat{A}^*\left(G^{-1}+\hat{A}C_0\hat{A}^*\right)^{-1}\hat{A}C_0. \tag*{(19)} C ^ ( G ) = C 0 − C 0 A ^ ∗ ( G − 1 + A ^ C 0 A ^ ∗ ) − 1 A ^ C 0 . (19) In the following, we derive J \mathcal{J} J X = H X=H X = H ξ ^ \hat{\xi} ξ ^
Theorem 3.1. Let the Assumption 1 hold and the assumptions of this section and abbreviate M ( G ) : = G − 1 + A ^ C 0 A ^ ∗ M(G):=G^{-1}+\hat{A}C_0\hat{A}^* M ( G ) := G − 1 + A ^ C 0 A ^ ∗ B ( G ) : = C 0 1 / 2 A ^ ∗ M ( G ) − 1 A ^ C 0 1 / 2 B(G):=C_0^{1/2}\hat{A}^*M(G)^{-1}\hat{A}C_0^{1/2} B ( G ) := C 0 1/2 A ^ ∗ M ( G ) − 1 A ^ C 0 1/2
J ( ξ , G ) = 1 2 log det ( I − B ( G ) ) + 1 2 E λ ∥ A ^ ( x − m ^ ( y ; ξ , G ) ) ∥ G 2 \mathcal{J}(\xi,G)=\frac{1}{2}\log\det\left(I-B(G)\right)+\frac{1}{2}\mathbb{E}^{\lambda}\left\|\hat{A}\left(x-\hat{m}(y;\xi,G)\right)\right\|_{G}^{2} J ( ξ , G ) = 2 1 log det ( I − B ( G ) ) + 2 1 E λ A ^ ( x − m ^ ( y ; ξ , G ) ) G 2 + 1 2 E p ∥ C 0 1 / 2 A ^ ∗ M ( G ) − 1 ( y − A ^ x ˉ − ξ ) ∥ H 2 − E λ ⟨ A ^ ∗ M ( G ) − 1 y , x − x ˉ ⟩ H . +\frac{1}{2}\mathbb{E}^{p}\left\|C_0^{1/2}\hat{A}^*M(G)^{-1}\left(y-\hat{A}\bar{x}-\xi\right)\right\|_{H}^{2}-\mathbb{E}^{\lambda}\left\langle\hat{A}^*M(G)^{-1}y,x-\bar{x}\right\rangle_{H}. + 2 1 E p C 0 1/2 A ^ ∗ M ( G ) − 1 ( y − A ^ x ˉ − ξ ) H 2 − E λ ⟨ A ^ ∗ M ( G ) − 1 y , x − x ˉ ⟩ H . For every fixed G ∈ S ( m ) G \in\mathcal{S}(m) G ∈ S ( m ) ξ ∈ R m \xi\in\mathbb{R}^{m} ξ ∈ R m
ξ ^ = E λ ( y − A ^ x ) , (20) \widehat{\xi} = \mathbb{E}^{\lambda}(y-\widehat{A}x), \tag*{(20)} ξ = E λ ( y − A x ) , (20) independently of G G G ξ ^ + M ( G ) ker A ^ ∗ \widehat{\xi}+M(G)\ker\widehat{A}^{*} ξ + M ( G ) ker A ∗ ξ ^ \widehat{\xi} ξ Range ( A ^ ) = R m \operatorname{Range}(\widehat{A})=\mathbb{R}^{m} Range ( A ) = R m
Since the optimiser for ξ \xi ξ J \mathcal{J} J G G G
Proof of Theorem 3.1 . By the Feldman-Hájek theorem, μ ^ ( ⋅ ∣ y ; ξ , G ) \widehat{\mu}(\cdot|y;\xi,G) μ ( ⋅ ∣ y ; ξ , G ) μ 0 \mu_{0} μ 0
d μ ^ ( ⋅ ∣ y ; ξ , G ) d μ 0 ( x ) = d N ( m ^ , C 0 ) d N ( x ‾ , C 0 ) ( x ) d N ( m ^ , C ^ ) d N ( m ^ , C 0 ) ( x ) , \frac{\mathrm{d}\widehat{\mu}(\cdot|y;\xi,G)}{\mathrm{d}\mu_{0}}(x)=\frac{\mathrm{d}\mathcal{N}(\widehat{m},C_{0})}{\mathrm{d}\mathcal{N}(\overline{x},C_{0})}(x)\frac{\mathrm{d}\mathcal{N}(\widehat{m},\widehat{C})}{\mathrm{d}\mathcal{N}(\widehat{m},C_{0})}(x), d μ 0 d μ ( ⋅ ∣ y ; ξ , G ) ( x ) = d N ( x , C 0 ) d N ( m , C 0 ) ( x ) d N ( m , C 0 ) d N ( m , C ) ( x ) , μ 0 \mu_{0} μ 0 μ 0 \mu_{0} μ 0
d N ( m ^ , C 0 ) d N ( x ‾ , C 0 ) ( x ) = exp ( − 1 2 ∥ m ^ − x ‾ ∥ C 0 2 + ⟨ m ^ − x ‾ , x − x ‾ ⟩ C 0 ) . \frac{\mathrm{d}\mathcal{N}(\widehat{m},C_{0})}{\mathrm{d}\mathcal{N}(\overline{x},C_{0})}(x)
=
\exp\left(-\frac{1}{2}\lVert\widehat{m}-\overline{x}\rVert_{C_{0}}^{2}+\langle\widehat{m}-\overline{x},x-\overline{x}\rangle_{C_{0}}\right). d N ( x , C 0 ) d N ( m , C 0 ) ( x ) = exp ( − 2 1 ∥ m − x ∥ C 0 2 + ⟨ m − x , x − x ⟩ C 0 ) . where ⟨ ⋅ , ⋅ ⟩ C 0 \langle\cdot,\cdot\rangle_{C_{0}} ⟨ ⋅ , ⋅ ⟩ C 0 [13 ] for details. By using the property ⟨ C 0 z , ⋅ ⟩ C 0 = ⟨ z , ⋅ ⟩ H \langle C_{0}z,\cdot\rangle_{C_{0}}=\langle z,\cdot\rangle_{H} ⟨ C 0 z , ⋅ ⟩ C 0 = ⟨ z , ⋅ ⟩ H [42 Section 3.2] ), μ 0 \mu_{0} μ 0 z z z
d N ( m ^ , C 0 ) d N ( x ‾ , C 0 ) ( x ) = exp ( − 1 2 ∥ C 0 1 / 2 A ^ ∗ M − 1 ( y − A ^ x ‾ − ξ ) ∥ H 2 + ⟨ A ^ ∗ M − 1 ( y − A ^ x ‾ − ξ ) , x − x ‾ ⟩ H ) . \frac{\mathrm{d}\mathcal{N}(\widehat{m},C_{0})}{\mathrm{d}\mathcal{N}(\overline{x},C_{0})}(x)
=
\exp\left(-\frac{1}{2}\left\lVert C_{0}^{1/2}\widehat{A}^{*}M^{-1}(y-\widehat{A}\overline{x}-\xi)\right\rVert_{H}^{2}
+\left\langle\widehat{A}^{*}M^{-1}(y-\widehat{A}\overline{x}-\xi),x-\overline{x}\right\rangle_{H}\right). d N ( x , C 0 ) d N ( m , C 0 ) ( x ) = exp ( − 2 1 C 0 1/2 A ∗ M − 1 ( y − A x − ξ ) H 2 + ⟨ A ∗ M − 1 ( y − A x − ξ ) , x − x ⟩ H ) . Notice that B ( G ) B(G) B ( G ) B ( G ) B(G) B ( G )
( I − B ( G ) ) − 1 = C 0 1 / 2 A ^ ∗ G A ^ C 0 1 / 2 + I , (21) (I-B(G))^{-1}=C_{0}^{1/2}\widehat{A}^{*}G\widehat{A}C_{0}^{1/2}+I, \tag*{(21)} ( I − B ( G ) ) − 1 = C 0 1/2 A ∗ G A C 0 1/2 + I , (21) is strictly positive definite. By [12 Proposition 1.3.11] , we can write for D ( G ) : = B ( G ) ( I − B ( G ) ) − 1 = ( I − B ( G ) ) − 1 − I D(G):=B(G)(I-B(G))^{-1}=(I-B(G))^{-1}-I D ( G ) := B ( G ) ( I − B ( G ) ) − 1 = ( I − B ( G ) ) − 1 − I
d N ( m ^ , C ^ ) d N ( m ^ , C 0 ) ( x ) = det ( I − B ( G ) ) − 1 / 2 exp ( − 1 2 ⟨ D ( G ) C 0 − 1 / 2 ( x − m ^ ) , C 0 − 1 / 2 ( x − m ^ ) ⟩ H ) , (22) \frac{\mathrm{d}\mathcal{N}(\widehat{m},\widehat{C})}{\mathrm{d}\mathcal{N}(\widehat{m},C_{0})}(x)
=
\det(I-B(G))^{-1/2}
\exp\left(-\frac{1}{2}\left\langle D(G)C_{0}^{-1/2}(x-\widehat{m}),C_{0}^{-1/2}(x-\widehat{m})\right\rangle_{H}\right), \tag*{(22)} d N ( m , C 0 ) d N ( m , C ) ( x ) = det ( I − B ( G ) ) − 1/2 exp ( − 2 1 ⟨ D ( G ) C 0 − 1/2 ( x − m ) , C 0 − 1/2 ( x − m ) ⟩ H ) , (22) where det \det det (21) into (22) we find
− E λ log ( d N ( m ^ , C ^ ) d N ( m ^ , C 0 ) ( x ) ) = 1 2 log det ( I − B ( G ) ) + 1 2 E λ ∥ A ^ ( x − m ^ ) ∥ G 2 . -\mathbb{E}^{\lambda}\log\left(\frac{\mathrm{d}\mathcal{N}(\widehat{m},\widehat{C})}{\mathrm{d}\mathcal{N}(\widehat{m},C_{0})}(x)\right)
=
\frac{1}{2}\log\det(I-B(G))
+\frac{1}{2}\mathbb{E}^{\lambda}\left\lVert\widehat{A}(x-\widehat{m})\right\rVert_{G}^{2}. − E λ log ( d N ( m , C 0 ) d N ( m , C ) ( x ) ) = 2 1 log det ( I − B ( G )) + 2 1 E λ A ( x − m ) G 2 . Introducing the notations r = y − A ^ x ‾ − ξ r=y-\widehat{A}\overline{x}-\xi r = y − A x − ξ r ‾ = E p y − A ^ x ‾ − ξ \overline{r}=\mathbb{E}^{p}y-\widehat{A}\overline{x}-\xi r = E p y − A x − ξ
J ( ξ , G ) = 1 2 log det ( I − B ( G ) ) + 1 2 E λ ∥ A ^ ( x − m ^ ) ∥ G 2 + 1 2 E p ∥ C 0 1 / 2 A ^ ∗ M − 1 r ∥ H 2 − E λ ⟨ A ^ ∗ M − 1 y , x − x ‾ ⟩ H . \mathcal{J}(\xi,G)
=
\frac{1}{2}\log\det(I-B(G))
+\frac{1}{2}\mathbb{E}^{\lambda}\left\lVert\widehat{A}(x-\widehat{m})\right\rVert_{G}^{2}
+\frac{1}{2}\mathbb{E}^{p}\left\lVert C_{0}^{1/2}\widehat{A}^{*}M^{-1}r\right\rVert_{H}^{2}
-\mathbb{E}^{\lambda}\langle\widehat{A}^{*}M^{-1}y,x-\overline{x}\rangle_{H}. J ( ξ , G ) = 2 1 log det ( I − B ( G )) + 2 1 E λ A ( x − m ) G 2 + 2 1 E p C 0 1/2 A ∗ M − 1 r H 2 − E λ ⟨ A ∗ M − 1 y , x − x ⟩ H . We first note that
E λ A ^ ( x − m ^ ) = E λ A ^ ( x − x ‾ − C 0 A ^ ∗ M − 1 r ) = − A ^ C 0 A ^ ∗ M − 1 r ‾ . \mathbb{E}^{\lambda}\widehat{A}(x-\widehat{m})
=
\mathbb{E}^{\lambda}\widehat{A}\left(x-\overline{x}-C_{0}\widehat{A}^{*}M^{-1}r\right)
=
-\widehat{A}C_{0}\widehat{A}^{*}M^{-1}\overline{r}. E λ A ( x − m ) = E λ A ( x − x − C 0 A ∗ M − 1 r ) = − A C 0 A ∗ M − 1 r . By isolating the ξ \xi ξ
E p ∥ C 0 1 / 2 A ^ ∗ M ( G ) − 1 r ∥ H 2 + E λ ∥ A ^ ( x − m ^ ) ∥ G 2 = ∥ C 0 1 / 2 A ^ ∗ M − 1 r ‾ ∥ H 2 + ∥ A ^ C 0 A ^ ∗ M − 1 r ‾ ∥ G 2 + tr ( Cov λ ( C 0 1 / 2 A ^ ∗ M ( G ) − 1 y ) ) + tr ( G Cov λ A ^ ( x − m ^ ) ) . \mathbb{E}^{p}\left\lVert C_{0}^{1/2}\widehat{A}^{*}M(G)^{-1}r\right\rVert_{H}^{2}
+\mathbb{E}^{\lambda}\left\lVert\widehat{A}(x-\widehat{m})\right\rVert_{G}^{2}
=
\left\lVert C_{0}^{1/2}\widehat{A}^{*}M^{-1}\overline{r}\right\rVert_{H}^{2}
+\left\lVert\widehat{A}C_{0}\widehat{A}^{*}M^{-1}\overline{r}\right\rVert_{G}^{2}
+\operatorname{tr}\left(\operatorname{Cov}^{\lambda}\left(C_{0}^{1/2}\widehat{A}^{*}M(G)^{-1}y\right)\right)
+\operatorname{tr}\left(G\operatorname{Cov}^{\lambda}\widehat{A}(x-\widehat{m})\right). E p C 0 1/2 A ∗ M ( G ) − 1 r H 2 + E λ A ( x − m ) G 2 = C 0 1/2 A ∗ M − 1 r H 2 + A C 0 A ∗ M − 1 r G 2 + tr ( Cov λ ( C 0 1/2 A ∗ M ( G ) − 1 y ) ) + tr ( G Cov λ A ( x − m ) ) . The minimisers are given by r ‾ = E λ ( y − A ^ ( x ) ) − ξ ∈ ker ( A ^ ∗ M − 1 ) \overline{r}=\mathbb{E}^{\lambda}(y-\widehat{A}(x))-\xi\in\ker(\widehat{A}^{*}M^{-1}) r = E λ ( y − A ( x )) − ξ ∈ ker ( A ∗ M − 1 ) ξ ^ = E λ ( y − A ^ ( x ) ) \widehat{\xi}=\mathbb{E}^{\lambda}(y-\widehat{A}(x)) ξ = E λ ( y − A ( x )) G ∈ S ( m ) G\in\mathcal{S}(m) G ∈ S ( m )
argmin ξ ∈ R m J ( ξ , G ) = ξ ^ + M ker ( A ^ ∗ ) . \underset{\xi\in\mathbb{R}^{m}}{\operatorname{argmin}}\ \mathcal{J}(\xi,G)=\widehat{\xi}+M\ker(\widehat{A}^{*}). ξ ∈ R m argmin J ( ξ , G ) = ξ + M ker ( A ∗ ) . The following remark can be helpful in evaluating the objective function in the case where prior and exact and approximate posterior distribution admit a known density even in the non-Gaussian-linear setting.
In particular, in the Gaussian-linear setting and when X = R n X=\mathbb{R}^{n} X = R n
J ( ξ , G ) = 1 2 E μ ^ ∥ x − m ^ ( y ; ξ , G ) ∥ C ^ ( G ) − 1 2 + 1 2 log det C ^ ( G ) + E μ 0 log p 0 ( x ) + n 2 log ( 2 π ) . (23) \mathcal{J}(\xi,G)=\frac{1}{2}\mathbb{E}^{\hat{\mu}}\left\|x-\hat{m}(y;\xi,G)\right\|_{\hat{C}(G)^{-1}}^{2}+\frac{1}{2}\log\det\hat{C}(G)+\mathbb{E}^{\mu_{0}}\log p_{0}(x)+\frac{n}{2}\log(2\pi). \tag*{(23)} J ( ξ , G ) = 2 1 E μ ^ ∥ x − m ^ ( y ; ξ , G ) ∥ C ^ ( G ) − 1 2 + 2 1 log det C ^ ( G ) + E μ 0 log p 0 ( x ) + 2 n log ( 2 π ) . (23) In finite dimensions the posterior mean and covariance satisfy
m ^ ( y ; ξ , G ) = C ^ ( G ) ( C 0 − 1 x ˉ + A ^ T G ( y − ξ ) ) , C ^ ( G ) = ( C 0 − 1 + A ^ T G A ^ ) − 1 . (24) \hat{m}(y;\xi,G)=\hat{C}(G)\left(C_{0}^{-1}\bar{x}+\hat{A}^{\mathsf{T}}G(y-\xi)\right),\qquad\hat{C}(G)=\left(C_{0}^{-1}+\hat{A}^{\mathsf{T}}G\hat{A}\right)^{-1}. \tag*{(24)} m ^ ( y ; ξ , G ) = C ^ ( G ) ( C 0 − 1 x ˉ + A ^ T G ( y − ξ ) ) , C ^ ( G ) = ( C 0 − 1 + A ^ T G A ^ ) − 1 . (24) Optimisation of the regularised objective functional
This section considers practical optimisation of the functional J \mathcal{J} J
Regularised objective functional
We immediately observe three potential limitations in the problem formulation:
(i) Non-attainment of the infimum. Let X = R X=\mathbb{R} X = R A = I 1 = − A ^ A=I_{1}=-\hat{A} A = I 1 = − A ^ μ 0 = N ( 0 , 1 2 ) \mu_{0}=\mathcal{N}(0,\frac{1}{2}) μ 0 = N ( 0 , 2 1 ) δ = 1 \delta=1 δ = 1
J ( ξ , G ) = 3 G ( G + 1 ) + ξ 2 G 2 2 ( 2 + G ) + log 2 2 − log ( 2 + G ) 2 , \mathcal{J}(\xi,G)=\frac{3G(G+1)+\xi^{2}G^{2}}{2(2+G)}+\frac{\log2}{2}-\frac{\log(2+G)}{2}, J ( ξ , G ) = 2 ( 2 + G ) 3 G ( G + 1 ) + ξ 2 G 2 + 2 log 2 − 2 log ( 2 + G ) , from where it follows that a r g m i n G ∈ cl ( S ( 1 ) ) J ( 0 , G ) = 0 \mathop{\rm argmin}_{G\in\operatorname{cl}(S(1))}\mathcal{J}(0,G)=0 argmin G ∈ cl ( S ( 1 )) J ( 0 , G ) = 0 0 ∉ S ( 1 ) 0\notin S(1) 0 ∈ / S ( 1 )
(ii) Unbounded level sets for G G G Suppose X = R k X=\mathbb{R}^{k} X = R k k < m k<m k < m μ 0 \mu_{0} μ 0 A ^ ∈ R m × k \hat{A}\in\mathbb{R}^{m\times k} A ^ ∈ R m × k A ^ \hat{A} A ^ A ^ T \hat{A}^{\mathsf{T}} A ^ T (24) , that for some z ≠ 0 z\ne0 z = 0 A ^ T z = 0 \hat{A}^{\mathsf{T}}z=0 A ^ T z = 0 t ≥ 0 t\ge0 t ≥ 0 ξ ∈ R m \xi\in\mathbb{R}^{m} ξ ∈ R m
J ( ξ , G + t z ⊗ z ) = J ( ξ , G ) . (25) \mathcal{J}(\xi,G+tz\otimes z)=\mathcal{J}(\xi,G). \tag*{(25)} J ( ξ , G + t z ⊗ z ) = J ( ξ , G ) . (25) Thus J \mathcal{J} J { G + t z ⊗ z : t ≥ 0 } \{G+tz\otimes z:t\ge0\} { G + t z ⊗ z : t ≥ 0 }
(iii) Unbounded level sets for ξ \xi ξ Suppose again X = R k X=\mathbb{R}^{k} X = R k k < m k<m k < m μ 0 \mu_{0} μ 0 A ^ ∈ R m × k \hat{A}\in\mathbb{R}^{m\times k} A ^ ∈ R m × k ξ \xi ξ G G G t ∈ R t\in\mathbb{R} t ∈ R
J ( ξ + t G − 1 z , G ) = J ( ξ , G ) . \mathcal{J}(\xi+tG^{-1}z,G)=\mathcal{J}(\xi,G). J ( ξ + t G − 1 z , G ) = J ( ξ , G ) . Thus J ( ⋅ , G ) \mathcal{J}(\cdot,G) J ( ⋅ , G ) { ξ + t G − 1 z : t ∈ R } \{\xi+tG^{-1}z:t\in\mathbb{R}\} { ξ + t G − 1 z : t ∈ R }
The difficulties in (i), (ii) are resolved by adding an appropriate regularisation term for G G G A ^ \hat{A} A ^ μ 0 \mu_{0} μ 0 R m \mathbb{R}^{m} R m ξ \xi ξ S ( m ) S(m) S ( m )
Lemma 4.1. The function
R : S ( m ) → R , G ↦ δ 2 tr ( G ) − log det G , \mathcal{R} : \mathcal{S}(m) \to\mathbb{R}, \qquad G \mapsto\delta^{2}\operatorname{tr}(G) - \log\det G, R : S ( m ) → R , G ↦ δ 2 tr ( G ) − log det G , has the following properties:
(i) R \mathcal{R} R S ( m ) \mathcal{S}(m) S ( m )
(ii) Let { G n } n = 1 ∞ ⊂ S ( m ) \{G_n\}_{n=1}^{\infty} \subset\mathcal{S}(m) { G n } n = 1 ∞ ⊂ S ( m ) γ max ( G n ) → ∞ \gamma_{\max}(G_n) \to\infty γ m a x ( G n ) → ∞ γ min ( G n ) → 0 \gamma_{\min}(G_n) \to0 γ m i n ( G n ) → 0 γ max \gamma_{\max} γ m a x γ min \gamma_{\min} γ m i n R ( G n ) → ∞ \mathcal{R}(G_n) \to\infty R ( G n ) → ∞
(iii) R \mathcal{R} R δ − 2 I m ∈ S ( m ) \delta^{-2}I_m \in\mathcal{S}(m) δ − 2 I m ∈ S ( m )
Proof. (i) Convexity. The map G ↦ tr G G \mapsto\operatorname{tr}G G ↦ tr G G ↦ − log det G G \mapsto-\log\det G G ↦ − log det G S ( m ) \mathcal{S}(m) S ( m ) − log det -\log\det − log det [5 ] ). As a sum of convex functions, R \mathcal{R} R S ( m ) \mathcal{S}(m) S ( m )
(ii) Coercivity. Write R ( G ) = ∑ i = 1 m s ( γ i ( G ) ) \mathcal{R}(G) = \sum_{i=1}^{m} s(\gamma_i(G)) R ( G ) = ∑ i = 1 m s ( γ i ( G )) γ i ( G ) \gamma_i(G) γ i ( G ) G G G s ( γ ) : = δ 2 γ − log γ s(\gamma) := \delta^{2}\gamma- \log\gamma s ( γ ) := δ 2 γ − log γ s s s s ( δ − 2 ) s(\delta^{-2}) s ( δ − 2 ) s ( γ ) → ∞ s(\gamma) \to\infty s ( γ ) → ∞ γ → 0 + \gamma\to0^{+} γ → 0 + γ → ∞ \gamma\to\infty γ → ∞ γ max ( G n ) → ∞ \gamma_{\max}(G_n) \to\infty γ m a x ( G n ) → ∞ R ( G n ) ≥ ( m − 1 ) s ( δ − 2 ) + s ( γ max ( G n ) ) → ∞ \mathcal{R}(G_n) \ge(m-1)s(\delta^{-2}) + s(\gamma_{\max}(G_n)) \to\infty R ( G n ) ≥ ( m − 1 ) s ( δ − 2 ) + s ( γ m a x ( G n )) → ∞ γ min ( G n ) → 0 \gamma_{\min}(G_n) \to0 γ m i n ( G n ) → 0
(iii) Unique minimiser. Setting the derivative to zero, the first-order optimality condition for G ↦ δ 2 tr G − log det G G \mapsto\delta^{2}\operatorname{tr}G - \log\det G G ↦ δ 2 tr G − log det G δ 2 I − G − 1 = 0 \delta^{2}I - G^{-1} = 0 δ 2 I − G − 1 = 0 G = δ − 2 I G = \delta^{-2}I G = δ − 2 I − log det -\log\det − log det S ( m ) \mathcal{S}(m) S ( m )
We emphasise that R ( G ) = 2 KL ( N ( 0 , δ 2 I ) ∥ N ( 0 , G − 1 ) ) + m ( 1 + log δ 2 ) \mathcal{R}(G) = 2\operatorname{KL}\left(\mathcal{N}(0,\delta^{2}I)\,\middle\|\,\mathcal{N}(0,G^{-1})\right) + m(1+\log\delta^{2}) R ( G ) = 2 KL ( N ( 0 , δ 2 I ) N ( 0 , G − 1 ) ) + m ( 1 + log δ 2 )
For α > 0 \alpha> 0 α > 0
J α : R m × S ( m ) → R , J α ( ξ , G ) = J ( ξ , G ) + α R ( G ) . \mathcal{J}_{\alpha} : \mathbb{R}^{m} \times\mathcal{S}(m) \to\mathbb{R}, \qquad\mathcal{J}_{\alpha}(\xi,G) = \mathcal{J}(\xi,G) + \alpha\mathcal{R}(G). J α : R m × S ( m ) → R , J α ( ξ , G ) = J ( ξ , G ) + α R ( G ) . The regularisation parameter α > 0 \alpha> 0 α > 0 Lemma 4.1 (ii), the sublevel sets of J α \mathcal{J}_{\alpha} J α G G G S ( m ) \mathcal{S}(m) S ( m ) ξ \xi ξ
Proximal Alternating Minimisation
We adopt a proximal alternating minimisation strategy, which interleaves exact minimisation over each block variable with a quadratic proximal penalty that stabilises the iterates. Convergence of such algorithms under the Kurdyka–Łojasiewicz (KL) property has been established in [2 , 49 ] ; we recall the relevant definitions before stating our results.
Denote by ∂ f \partial f ∂ f f f f dom ( ∂ f ) \operatorname{dom}(\partial f) dom ( ∂ f ) ∞ \infty ∞ dom ( ∂ f ) \operatorname{dom}(\partial f) dom ( ∂ f )
Definition 4.2 (Kurdyka–Łojasiewicz property [2 ] ). A lower semi-continuous function f : R ℓ → R ∪ { + ∞ } f : \mathbb{R}^{\ell} \to\mathbb{R} \cup\{+\infty\} f : R ℓ → R ∪ { + ∞ } z ∗ ∈ dom ( ∂ f ) z^{*} \in\operatorname{dom}(\partial f) z ∗ ∈ dom ( ∂ f ) η ∈ ( 0 , + ∞ ) \eta\in(0,+\infty) η ∈ ( 0 , + ∞ ) U U U z ∗ z^{*} z ∗ φ : [ 0 , η ) → R ≥ 0 \varphi: [0,\eta) \to\mathbb{R}_{\ge0} φ : [ 0 , η ) → R ≥ 0
(i) φ ( 0 ) = 0 \varphi(0) = 0 φ ( 0 ) = 0
(ii) φ ∈ C 1 ( 0 , η ) \varphi\in C^{1}(0,\eta) φ ∈ C 1 ( 0 , η ) φ ′ > 0 \varphi' > 0 φ ′ > 0 ( 0 , η ) (0,\eta) ( 0 , η )
(iii) for all z ∈ U z \in U z ∈ U f ( z ∗ ) < f ( z ) < f ( z ∗ ) + η f(z^{*}) < f(z) < f(z^{*})+\eta f ( z ∗ ) < f ( z ) < f ( z ∗ ) + η
φ ′ ( f ( z ) − f ( z ∗ ) ) ⋅ dist ( 0 , ∂ f ( z ) ) ≥ 1. \varphi'(f(z)-f(z^{*})) \cdot\operatorname{dist}(0,\partial f(z)) \ge1. φ ′ ( f ( z ) − f ( z ∗ )) ⋅ dist ( 0 , ∂ f ( z )) ≥ 1. The KL inequality quantifies how fast the function values separate from the critical value, and is the key analytic tool in the convergence proofs of [2 ] .
Remark 4.3. The following classes of functions satisfy the KL property and will be relevant below.
(a) Morse functions. A C 2 C^2 C 2 f f f Morse function if the Hessian D 2 f ( x ˉ ) D^2f(\bar{x}) D 2 f ( x ˉ ) x ˉ \bar{x} x ˉ D 2 f ( x ˉ ) D^2f(\bar{x}) D 2 f ( x ˉ ) [2 ] .
(b) Non-critical points. Any proper lower semi-continuous function satisfies the KL property at every non-critical point [2 ] .
We now define the proximal alternating minimisation sequence.
Definition 4.4 (Proximal alternating minimisation). Let f : R ℓ 1 × R ℓ 2 → R ∪ { + ∞ } f : \mathbb{R}^{\ell_1} \times\mathbb{R}^{\ell_2} \to\mathbb{R} \cup\{+\infty\} f : R ℓ 1 × R ℓ 2 → R ∪ { + ∞ } ( z 0 , w 0 ) ∈ R ℓ 1 × R ℓ 2 (z_0,w_0) \in\mathbb{R}^{\ell_1} \times\mathbb{R}^{\ell_2} ( z 0 , w 0 ) ∈ R ℓ 1 × R ℓ 2 ( z k , w k ) k ≥ 1 (z_k,w_k)_{k \ge1} ( z k , w k ) k ≥ 1
z k ∈ argmin z ∈ R ℓ 1 [ f ( z , w k − 1 ) + 1 ζ k 1 ∥ z − z k − 1 ∥ 2 ] , z_k \in\underset{z\in\mathbb{R}^{\ell_1}}{\operatorname{argmin}}\left[f(z,w_{k-1})+\frac{1}{\zeta_k^1}\lVert z-z_{k-1}\rVert^2\right], z k ∈ z ∈ R ℓ 1 argmin [ f ( z , w k − 1 ) + ζ k 1 1 ∥ z − z k − 1 ∥ 2 ] , w k ∈ argmin w ∈ R ℓ 2 [ f ( z k , w ) + 1 ζ k 2 ∥ w − w k − 1 ∥ 2 ] . w_k \in\underset{w\in\mathbb{R}^{\ell_2}}{\operatorname{argmin}}\left[f(z_k,w)+\frac{1}{\zeta_k^2}\lVert w-w_{k-1}\rVert^2\right]. w k ∈ w ∈ R ℓ 2 argmin [ f ( z k , w ) + ζ k 2 1 ∥ w − w k − 1 ∥ 2 ] . where the proximal parameters satisfy ζ k 1 , ζ k 2 ∈ ( r − , r + ) \zeta_k^1,\zeta_k^2 \in(r_-,r_+) ζ k 1 , ζ k 2 ∈ ( r − , r + ) 0 < r − < r + 0<r_-<r_+ 0 < r − < r +
Remark 4.5. The proximal terms 1 ζ k 1 ∥ z − z k − 1 ∥ 2 \frac{1}{\zeta_k^1}\lVert z-z_{k-1}\rVert^2 ζ k 1 1 ∥ z − z k − 1 ∥ 2 1 ζ k 2 ∥ w − w k − 1 ∥ 2 \frac{1}{\zeta_k^2}\lVert w-w_{k-1}\rVert^2 ζ k 2 1 ∥ w − w k − 1 ∥ 2 ζ k 1 , ζ k 2 \zeta_k^1,\zeta_k^2 ζ k 1 , ζ k 2 [2 ] .
The central result of [2 ] states that, if the objective functional satisfies the KL property at every point of its domain, then (under mild assumptions) any bounded sequence generated by the proximal alternating minimisation algorithm converges to a critical point of f f f
Theorem 4.6 ([2 Theorems 8–10] ). Let f : R ℓ 1 × R ℓ 2 → R ∪ { + ∞ } f : \mathbb{R}^{\ell_1} \times\mathbb{R}^{\ell_2} \to\mathbb{R} \cup\{+\infty\} f : R ℓ 1 × R ℓ 2 → R ∪ { + ∞ } f ( z , w ) = f 1 ( z ) + Q ( z , w ) + f 2 ( w ) f(z,w)=f_1(z)+Q(z,w)+f_2(w) f ( z , w ) = f 1 ( z ) + Q ( z , w ) + f 2 ( w ) f 1 , f 2 f_1,f_2 f 1 , f 2 Q Q Q f f f f ( ⋅ , w 0 ) f(\cdot,w_0) f ( ⋅ , w 0 ) ( z k , w k ) (z_k,w_k) ( z k , w k ) Definition 4.4 with ζ k 1 , ζ k 2 ∈ ( r − , r + ) \zeta_k^1,\zeta_k^2\in(r_-,r_+) ζ k 1 , ζ k 2 ∈ ( r − , r + )
If ( z k , w k ) (z_k,w_k) ( z k , w k ) f f f ( z k , w k ) (z_k,w_k) ( z k , w k ) ∑ k = 1 ∞ ∥ ( z k , w k ) − ( z k − 1 , w k − 1 ) ∥ < ∞ \sum_{k=1}^{\infty}\lVert(z_k,w_k)-(z_{k-1},w_{k-1})\rVert<\infty ∑ k = 1 ∞ ∥( z k , w k ) − ( z k − 1 , w k − 1 )∥ < ∞ ( z k , w k ) (z_k,w_k) ( z k , w k ) f f f
Furthermore, if ( z ^ , w ^ ) (\hat{z},\hat{w}) ( z ^ , w ^ ) f f f f f f ( z ^ , w ^ ) (\hat{z},\hat{w}) ( z ^ , w ^ ) f f f
The first statement is the content of the proof of [2 Theorem 9] , which applies [2 Theorem 8] at a cluster point of the iterates; the second is [2 Theorem 10] .
We note that convergence to the global optimiser can only be guaranteed if the iterates are initialised close to the global optimiser. Since the objective function in the following may have multiple local optima, we will not be able to ensure convergence to global optimisers.
As we see in the following lemma, when f f f [2 Remark 5] ).
Lemma 4.7. Let the assumptions of Theorem 4.6 hold and assume that f f f ( z k , w k ) (z_k,w_k) ( z k , w k ) Definition 4.4 and with ζ k 1 , ζ k 2 ∈ ( r − , r + ) \zeta_k^1,\zeta_k^2\in(r_-,r_+) ζ k 1 , ζ k 2 ∈ ( r − , r + )
Proof. By [2 Lemma 5] , the sequence f ( z k , w k ) f(z_k,w_k) f ( z k , w k ) ( z k , w k ) (z_k,w_k) ( z k , w k ) ∥ ( z n k , w n k ) ∥ → ∞ \lVert(z_{n_k},w_{n_k})\rVert\to\infty ∥( z n k , w n k )∥ → ∞ n k n_k n k f f f f ( z n k , w n k ) → ∞ f(z_{n_k},w_{n_k})\to\infty f ( z n k , w n k ) → ∞
Remark 4.8. Theorem 4.6 is stated for functions on R ℓ 1 × R ℓ 2 \mathbb{R}^{\ell_1}\times\mathbb{R}^{\ell_2} R ℓ 1 × R ℓ 2 Q ∈ C 1 Q\in C^1 Q ∈ C 1 I α \mathcal{I}_{\alpha} I α R m × S ( m ) \mathbb{R}^m\times\mathcal{S}(m) R m × S ( m ) S ( m ) \mathcal{S}(m) S ( m ) I \mathcal{I} I
We reduce to that setting as follows. Throughout, we identify Sym ( m , R ) \operatorname{Sym}(m,\mathbb{R}) Sym ( m , R ) R m ′ \mathbb{R}^{m'} R m ′ m ′ : = m ( m + 1 ) / 2 m' := m(m+1)/2 m ′ := m ( m + 1 ) /2 S ( m ) \mathcal{S}(m) S ( m ) R m ′ \mathbb{R}^{m'} R m ′ L 0 : = J α ( ξ 0 , G 0 ) L_0 := \mathcal{J}_{\alpha}(\xi_0,G_0) L 0 := J α ( ξ 0 , G 0 ) κ : = E p KL ( μ ( ⋅ ∣ y ) ∥ μ 0 ) \kappa:= \mathbb{E}^{p}\operatorname{KL}(\mu(\cdot\mid y)\mathbin\Vert\mu_0) κ := E p KL ( μ ( ⋅ ∣ y ) ∥ μ 0 ) B : = { G ∈ S ( m ) : α R ( G ) ≤ L 0 + κ } \mathcal{B} := \{G \in\mathcal{S}(m) : \alpha\mathcal{R}(G) \le L_0+\kappa\} B := { G ∈ S ( m ) : α R ( G ) ≤ L 0 + κ } S ( m ) \mathcal{S}(m) S ( m ) [2 ] and J ≥ − κ \mathcal{J} \ge-\kappa J ≥ − κ G k G_k G k B \mathcal{B} B ∂ S ( m ) \partial\mathcal{S}(m) ∂ S ( m ) χ ∈ C ∞ ( R m ′ ) \chi\in C^{\infty}(\mathbb{R}^{m'}) χ ∈ C ∞ ( R m ′ ) S ( m ) \mathcal{S}(m) S ( m ) χ ≡ 1 \chi\equiv1 χ ≡ 1 N \mathcal{N} N B \mathcal{B} B f 1 ≡ 0 f_1 \equiv0 f 1 ≡ 0 Q : = χ J − ( 1 − χ ) κ ∈ C 2 ( R m × R m ′ ) Q := \chi\mathcal{J}-(1-\chi)\kappa\in C^2(\mathbb{R}^{m}\times\mathbb{R}^{m'}) Q := χ J − ( 1 − χ ) κ ∈ C 2 ( R m × R m ′ ) f 2 : = α R f_2 := \alpha\mathcal{R} f 2 := α R + ∞ +\infty + ∞ S ( m ) \mathcal{S}(m) S ( m ) Q ≥ − κ Q \ge-\kappa Q ≥ − κ Q = J Q=\mathcal{J} Q = J R m × N \mathbb{R}^{m}\times\mathcal{N} R m × N Q + f 2 > L 0 Q+f_2>L_0 Q + f 2 > L 0 R m × ( S ( m ) ∖ B ) \mathbb{R}^{m}\times(\mathcal{S}(m)\setminus\mathcal{B}) R m × ( S ( m ) ∖ B ) f 1 + Q + f 2 f_1+Q+f_2 f 1 + Q + f 2 J α \mathcal{J}_{\alpha} J α R m × B \mathbb{R}^{m}\times\mathcal{B} R m × B
To establish the Morse property for J α \mathcal{J}_{\alpha} J α (18) do not vanish.
Assumption 2. No non-zero vector v ∈ R m v \in\mathbb{R}^{m} v ∈ R m v ⋅ A ^ v\cdot\widehat{A} v ⋅ A μ 0 \mu_0 μ 0
Lemma 4.12. The covariance matrix
C = E μ [ ( A ^ ( x ) − E μ A ^ ( x ) ) ⊗ ( A ^ ( x ) − E μ A ^ ( x ) ) ] , C=\mathbb{E}^{\mu}\left[(\widehat{A}(x)-\mathbb{E}^{\mu}\widehat{A}(x))\otimes(\widehat{A}(x)-\mathbb{E}^{\mu}\widehat{A}(x))\right], C = E μ [ ( A ( x ) − E μ A ( x )) ⊗ ( A ( x ) − E μ A ( x )) ] , where μ = μ ^ ( ⋅ ∣ y ; ξ , G ) \mu=\widehat{\mu}(\cdot\mid y;\xi,G) μ = μ ( ⋅ ∣ y ; ξ , G ) v ∈ R m v\in\mathbb{R}^{m} v ∈ R m v ⋅ A ^ v\cdot\widehat{A} v ⋅ A μ 0 \mu_0 μ 0
Proof. First observe that the Radon–Nikodym derivative
d μ d μ 0 ( x ∣ y ; ξ , G ) = 1 Z ^ ( y ; ξ , G ) exp ( − 1 2 ∥ A ^ ( x ) − y + ξ ∥ G 2 ) > 0 (26) \frac{\mathrm{d}\mu}{\mathrm{d}\mu_0}(x\mid y;\xi,G)=\frac{1}{\widehat{Z}(y;\xi,G)}\exp\left(-\frac{1}{2}\|\widehat{A}(x)-y+\xi\|_{G}^{2}\right)>0 \tag*{(26)} d μ 0 d μ ( x ∣ y ; ξ , G ) = Z ( y ; ξ , G ) 1 exp ( − 2 1 ∥ A ( x ) − y + ξ ∥ G 2 ) > 0 (26) for any x ∈ X x\in X x ∈ X y , ξ ∈ R m y,\xi\in\mathbb{R}^{m} y , ξ ∈ R m G ∈ S ( m ) G\in\mathcal{S}(m) G ∈ S ( m ) B ⊂ X \mathcal{B}\subset X B ⊂ X μ ( B ∣ y ; ξ , G ) = 0 \mu(\mathcal{B}\mid y;\xi,G)=0 μ ( B ∣ y ; ξ , G ) = 0 μ 0 ( B ) = 0 \mu_0(\mathcal{B})=0 μ 0 ( B ) = 0 μ 0 \mu_0 μ 0 μ ( ⋅ ∣ y ; ξ , G ) \mu(\cdot\mid y;\xi,G) μ ( ⋅ ∣ y ; ξ , G ) μ ( ⋅ ∣ y ; ξ , G ) \mu(\cdot\mid y;\xi,G) μ ( ⋅ ∣ y ; ξ , G ) μ 0 \mu_0 μ 0
For any v ∈ R m v\in\mathbb{R}^{m} v ∈ R m
v T C v = Var μ ( v ⋅ A ^ ( x ) ) = ∥ v ⋅ A ^ ( x ) − E μ [ v ⋅ A ^ ( x ) ] ∥ L 2 ( μ ) 2 . v^{\mathsf{T}}Cv=\operatorname{Var}^{\mu}(v\cdot\widehat{A}(x))=\|v\cdot\widehat{A}(x)-\mathbb{E}^{\mu}[v\cdot\widehat{A}(x)]\|_{L^2(\mu)}^2. v T C v = Var μ ( v ⋅ A ( x )) = ∥ v ⋅ A ( x ) − E μ [ v ⋅ A ( x )] ∥ L 2 ( μ ) 2 . This vanishes if and only if v ⋅ A ^ v\cdot\widehat{A} v ⋅ A μ \mu μ v T C v > 0 v^{\mathsf{T}}Cv>0 v T C v > 0 v ≠ 0 v\ne0 v = 0 C ≻ 0 C\succ0 C ≻ 0 v ⋅ A ^ v\cdot\widehat{A} v ⋅ A μ \mu μ □ \square □
Lemma 4.13. Let Assumptions 1 and 2 be satisfied. Then the functional J α \mathcal{J}_{\alpha} J α J α \mathcal{J}_{\alpha} J α
Proof.
J α \mathcal{J}_{\alpha} J α ( ξ ^ , G ^ ) ∈ R m × R m ′ (\hat{\xi},\hat{G})\in\mathbb{R}^{m}\times\mathbb{R}^{m'} ( ξ ^ , G ^ ) ∈ R m × R m ′ J α \mathcal{J}_{\alpha} J α ( Δ ξ , Δ G ) (\Delta_{\xi},\Delta_{G}) ( Δ ξ , Δ G ) Δ G = 0 \Delta_{G}=0 Δ G = 0 J ( ⋅ , G ) − J α ( ⋅ , G ) \mathcal{J}(\cdot,G)-\mathcal{J}_{\alpha}(\cdot,G) J ( ⋅ , G ) − J α ( ⋅ , G ) G ∈ S ( m ) G\in\mathcal{S}(m) G ∈ S ( m ) (18) we find
D [ Δ ξ , 0 ] 2 J α ( ξ ^ , G ^ ) = D [ Δ ξ , 0 ] 2 J ( ξ ^ , G ^ ) = E p Var μ ^ ( ⋅ ∣ y ; ξ ^ , G ^ ) [ A ^ ( x ) T G ^ Δ ξ ] . \begin{aligned}
D^{2}_{[\Delta_{\xi},0]}\mathcal{J}_{\alpha}(\hat{\xi},\hat{G})=D^{2}_{[\Delta_{\xi},0]}\mathcal{J}(\hat{\xi},\hat{G}) \\
&=\mathbb{E}^{p}\operatorname{Var}^{\hat{\mu}(\cdot\mid y;\hat{\xi},\hat{G})}\left[\hat{A}(x)^{\mathsf{T}}\hat{G}\Delta_{\xi}\right].
\end{aligned} D [ Δ ξ , 0 ] 2 J α ( ξ ^ , G ^ ) = D [ Δ ξ , 0 ] 2 J ( ξ ^ , G ^ ) = E p Var μ ^ ( ⋅ ∣ y ; ξ ^ , G ^ ) [ A ^ ( x ) T G ^ Δ ξ ] . By Lemma 4.12 applied with v = G ^ Δ ξ ≠ 0 v=\hat{G}\Delta_{\xi}\ne0 v = G ^ Δ ξ = 0 D [ Δ ξ , 0 ] 2 J α ( ξ ^ , G ^ ) > 0 D^{2}_{[\Delta_{\xi},0]}\mathcal{J}_{\alpha}(\hat{\xi},\hat{G})>0 D [ Δ ξ , 0 ] 2 J α ( ξ ^ , G ^ ) > 0 Δ ξ ∈ R m ∖ { 0 } \Delta_{\xi}\in\mathbb{R}^{m}\setminus\{0\} Δ ξ ∈ R m ∖ { 0 } Δ G ≠ 0 \Delta_{G}\ne0 Δ G = 0
D [ Δ ξ , Δ G ] 2 J α ( ξ ^ , G ^ ) = D [ Δ ξ , Δ G ] 2 J ( ξ ^ , G ^ ) + α D [ Δ G , Δ G ] 2 R ( G ^ ) = E p Var μ ^ ( ⋅ ∣ y ; ξ ^ , G ^ ) [ A ^ ( x ) T G ^ Δ ξ + 1 2 ∥ A ^ ( x ) − y + ξ ^ ∥ Δ G 2 ] + α D [ Δ G , Δ G ] 2 R ( G ^ ) ≥ α D [ Δ G , Δ G ] 2 R ( G ^ ) > 0 , \begin{aligned}
D^{2}_{[\Delta_{\xi},\Delta_{G}]}\mathcal{J}_{\alpha}(\hat{\xi},\hat{G})=D^{2}_{[\Delta_{\xi},\Delta_{G}]}\mathcal{J}(\hat{\xi},\hat{G})+\alpha D^{2}_{[\Delta_{G},\Delta_{G}]}\mathcal{R}(\hat{G}) \\
&=\mathbb{E}^{p}\operatorname{Var}^{\hat{\mu}(\cdot\mid y;\hat{\xi},\hat{G})}\left[\hat{A}(x)^{\mathsf{T}}\hat{G}\Delta_{\xi}+\frac{1}{2}\left\|\hat{A}(x)-y+\hat{\xi}\right\|_{\Delta_{G}}^{2}\right]+\alpha D^{2}_{[\Delta_{G},\Delta_{G}]}\mathcal{R}(\hat{G}) \\
&\ge\alpha D^{2}_{[\Delta_{G},\Delta_{G}]}\mathcal{R}(\hat{G})>0,
\end{aligned} D [ Δ ξ , Δ G ] 2 J α ( ξ ^ , G ^ ) = D [ Δ ξ , Δ G ] 2 J ( ξ ^ , G ^ ) + α D [ Δ G , Δ G ] 2 R ( G ^ ) = E p Var μ ^ ( ⋅ ∣ y ; ξ ^ , G ^ ) [ A ^ ( x ) T G ^ Δ ξ + 2 1 A ^ ( x ) − y + ξ ^ Δ G 2 ] + α D [ Δ G , Δ G ] 2 R ( G ^ ) ≥ α D [ Δ G , Δ G ] 2 R ( G ^ ) > 0 , where it was used that α D [ Δ G , Δ G ] 2 R ( G ^ ) > 0 \alpha D^{2}_{[\Delta_{G},\Delta_{G}]}\mathcal{R}(\hat{G})>0 α D [ Δ G , Δ G ] 2 R ( G ^ ) > 0 Δ G ≠ 0 \Delta_{G}\ne0 Δ G = 0 R \mathcal{R} R J α \mathcal{J}_{\alpha} J α
Remark 4.14. The proof of Lemma 4.13 crucially relies on the fact that the cross term involving k k k J \mathcal{J} J ξ \xi ξ (18) . Regularising ξ \xi ξ (18) no longer holds. Hence the cross term may contribute negatively to the Hessian, and positive definiteness, and hence the Morse property, can fail.
We can now define a proximal alternating minimisation sequence corresponding to the minimisation of J α \mathcal{J}_{\alpha} J α ξ 0 , G 0 ∈ R m × S ( m ) \xi_{0},G_{0}\in\mathbb{R}^{m}\times\mathcal{S}(m) ξ 0 , G 0 ∈ R m × S ( m ) 0 < r − < r + 0<r_{-}<r_{+} 0 < r − < r + ζ k 1 , ζ k 2 ∈ ( r − , r + ) \zeta_{k}^{1},\zeta_{k}^{2}\in(r_{-},r_{+}) ζ k 1 , ζ k 2 ∈ ( r − , r + ) k ≥ 1 k\ge1 k ≥ 1
ξ k ∈ arg min ξ ∈ R m [ J α ( ξ , G k − 1 ) + 1 ζ k 1 ∥ ξ − ξ k − 1 ∥ 2 ] , \xi_{k}\in\underset{\xi\in\mathbb{R}^{m}}{\arg\min}\left[\mathcal{J}_{\alpha}(\xi,G_{k-1})+\frac{1}{\zeta_{k}^{1}}\|\xi-\xi_{k-1}\|^{2}\right], ξ k ∈ ξ ∈ R m arg min [ J α ( ξ , G k − 1 ) + ζ k 1 1 ∥ ξ − ξ k − 1 ∥ 2 ] , G k ∈ arg min G ∈ S ( m ) [ J α ( ξ k , G ) + 1 ζ k 2 ∥ G − G k − 1 ∥ 2 ] . G_{k}\in\underset{G\in\mathcal{S}(m)}{\arg\min}\left[\mathcal{J}_{\alpha}(\xi_{k},G)+\frac{1}{\zeta_{k}^{2}}\|G-G_{k-1}\|^{2}\right]. G k ∈ G ∈ S ( m ) arg min [ J α ( ξ k , G ) + ζ k 2 1 ∥ G − G k − 1 ∥ 2 ] . Since we may use Theorem 4.6 , the sequence ( ξ k , G k ) (\xi_{k},G_{k}) ( ξ k , G k ) Lemma 4.7 , we can show that the objective function is coercive to ensure boundedness of the iterates and hence convergence to a critical point.. For G G G ξ \xi ξ ( p , λ ) (p,\lambda) ( p , λ ) ( 1 M ∑ i = 1 M δ y i , 1 M ∑ i = 1 M δ x i , y i ) \left(\frac{1}{M}\sum_{i=1}^{M}\delta_{y_i},\frac{1}{M}\sum_{i=1}^{M}\delta_{x_i,y_i}\right) ( M 1 ∑ i = 1 M δ y i , M 1 ∑ i = 1 M δ x i , y i ) § 5 .
Lemma 4.15 (Coercive decomposition). Let λ ~ M \widetilde{\lambda}_{M} λ M X × R m X\times\mathbb{R}^{m} X × R m p ~ M \widetilde{p}_{M} p M R m \mathbb{R}^{m} R m
A ^ λ ~ M : = E λ ~ M A ^ ( x ) , \widehat{A}^{\widetilde{\lambda}_{M}}:=\mathbb{E}^{\widetilde{\lambda}_{M}}\widehat{A}(x), A λ M := E λ M A ( x ) , denote by ⟨ ⋅ , ⋅ ⟩ + \langle\cdot,\cdot\rangle_{+} ⟨ ⋅ , ⋅ ⟩ + ⟨ ⋅ , ⋅ ⟩ \langle\cdot,\cdot\rangle ⟨ ⋅ , ⋅ ⟩
C ∗ ( λ ~ M ) : = inf ∥ v ∥ G 2 = 1 , γ ( G ) ∈ [ γ min , γ max ] E μ 0 [ exp ( − 1 2 ∥ A ^ ( x ) − A ^ λ ~ M ∥ G 2 ) ⟨ A ^ λ ~ M − A ^ ( x ) , v ⟩ G + ] . C_{*}(\widetilde{\lambda}_{M}):=\underset{\|v\|_{G}^{2}=1,\ \gamma(G)\in[\gamma_{\min},\gamma_{\max}]}{\inf}\mathbb{E}^{\mu^{0}}\left[\exp\left(-\frac{1}{2}\left\|\widehat{A}(x)-\widehat{A}^{\widetilde{\lambda}_{M}}\right\|_{G}^{2}\right)\left\langle\widehat{A}^{\widetilde{\lambda}_{M}}-\widehat{A}(x),v\right\rangle_{G}^{+}\right]. C ∗ ( λ M ) := ∥ v ∥ G 2 = 1 , γ ( G ) ∈ [ γ m i n , γ m a x ] inf E μ 0 [ exp ( − 2 1 A ( x ) − A λ M G 2 ) ⟨ A λ M − A ( x ) , v ⟩ G + ] . For G ∈ S ( m ) G\in\mathcal{S}(m) G ∈ S ( m ) [ γ min , γ max ] [\gamma_{\min},\gamma_{\max}] [ γ m i n , γ m a x ]
J ~ M ( ξ , G ) : = 1 2 E λ ~ M ∥ A ^ ( x ) − y + ξ ∥ G 2 + E p ~ M log E μ 0 exp ( − 1 2 ∥ A ^ ( x ) − y + ξ ∥ G 2 ) (27) \widetilde{\mathcal{J}}^{M}(\xi,G):=\frac{1}{2}\mathbb{E}^{\widetilde{\lambda}_{M}}\left\|\widehat{A}(x)-y+\xi\right\|_{G}^{2}+\mathbb{E}^{\widetilde{p}_{M}}\log\mathbb{E}^{\mu^{0}}\exp\left(-\frac{1}{2}\left\|\widehat{A}(x)-y+\xi\right\|_{G}^{2}\right) \tag*{(27)} J M ( ξ , G ) := 2 1 E λ M A ( x ) − y + ξ G 2 + E p M log E μ 0 exp ( − 2 1 A ( x ) − y + ξ G 2 ) (27) ≥ − E λ ~ M ⟨ A ^ ( x ) − A ^ λ ~ M , y ⟩ G + E p ~ M log ( ∥ A ^ λ ~ M − y + ξ ∥ G C ∗ ( λ ~ M ) ) . \ge-\mathbb{E}^{\widetilde{\lambda}_{M}}\left\langle\widehat{A}(x)-\widehat{A}^{\widetilde{\lambda}_{M}},y\right\rangle_{G}+\mathbb{E}^{\widetilde{p}_{M}}\log\left(\left\|\widehat{A}^{\widetilde{\lambda}_{M}}-y+\xi\right\|_{G}C_{*}(\widetilde{\lambda}_{M})\right). ≥ − E λ M ⟨ A ( x ) − A λ M , y ⟩ G + E p M log ( A λ M − y + ξ G C ∗ ( λ M ) ) . Proof. We have
∥ A ^ ( x ) − y + ξ ∥ G 2 = ∥ A ^ ( x ) − A ^ λ ~ M ∥ G 2 + ∥ A ^ λ ~ M − y + ξ ∥ G 2 + 2 ⟨ A ^ ( x ) − A ^ λ ~ M , A ^ λ ~ M − y + ξ ⟩ G . \left\|\widehat{A}(x)-y+\xi\right\|_{G}^{2}=\left\|\widehat{A}(x)-\widehat{A}^{\widetilde{\lambda}^{M}}\right\|_{G}^{2}+\left\|\widehat{A}^{\widetilde{\lambda}^{M}}-y+\xi\right\|_{G}^{2}+2\left\langle\widehat{A}(x)-\widehat{A}^{\widetilde{\lambda}^{M}},\widehat{A}^{\widetilde{\lambda}^{M}}-y+\xi\right\rangle_{G}. A ( x ) − y + ξ G 2 = A ( x ) − A λ M G 2 + A λ M − y + ξ G 2 + 2 ⟨ A ( x ) − A λ M , A λ M − y + ξ ⟩ G . Taking expectations and using that ∥ A ^ λ ~ M − y + ξ ∥ G 2 \left\|\widehat{A}^{\widetilde{\lambda}^{M}}-y+\xi\right\|_{G}^{2} A λ M − y + ξ G 2 x x x μ 0 \mu_{0} μ 0
J ~ M ( ξ , G ) = 1 2 E λ ~ M ∥ A ^ ( x ) − A ^ λ ~ M ∥ G 2 + E λ ~ M ⟨ A ^ ( x ) − A ^ λ ~ M , A ^ λ ~ M + ξ ⟩ G − E λ ~ M ⟨ A ^ ( x ) − A ^ λ ~ M , y ⟩ G + 1 2 E λ ~ M ∥ A ^ λ ~ M − y + ξ ∥ G 2 − 1 2 E p ~ M ∥ A ^ λ ~ M − y + ξ ∥ G 2 + E p ~ M log E μ 0 exp ( − 1 2 ∥ A ^ ( x ) − A ^ λ ~ M ∥ G 2 − ⟨ A ^ ( x ) − A ^ λ ~ M , A ^ λ ~ M − y + ξ ⟩ G ) . \begin{aligned}
\widetilde{\mathcal{J}}^{M}(\xi,G)
={}&\frac{1}{2}\mathbb{E}^{\widetilde{\lambda}^{M}}\left\|\widehat{A}(x)-\widehat{A}^{\widetilde{\lambda}^{M}}\right\|_{G}^{2}
+\mathbb{E}^{\widetilde{\lambda}^{M}}\left\langle\widehat{A}(x)-\widehat{A}^{\widetilde{\lambda}^{M}},\widehat{A}^{\widetilde{\lambda}^{M}}+\xi\right\rangle_{G}\\
&-\mathbb{E}^{\widetilde{\lambda}^{M}}\left\langle\widehat{A}(x)-\widehat{A}^{\widetilde{\lambda}^{M}},y\right\rangle_{G}
+\frac{1}{2}\mathbb{E}^{\widetilde{\lambda}^{M}}\left\|\widehat{A}^{\widetilde{\lambda}^{M}}-y+\xi\right\|_{G}^{2}
-\frac{1}{2}\mathbb{E}^{\widetilde{p}^{M}}\left\|\widehat{A}^{\widetilde{\lambda}^{M}}-y+\xi\right\|_{G}^{2}\\
&+\mathbb{E}^{\widetilde{p}^{M}}\log\mathbb{E}^{\mu_{0}}\exp\left(-\frac{1}{2}\left\|\widehat{A}(x)-\widehat{A}^{\widetilde{\lambda}^{M}}\right\|_{G}^{2}
-\left\langle\widehat{A}(x)-\widehat{A}^{\widetilde{\lambda}^{M}},\widehat{A}^{\widetilde{\lambda}^{M}}-y+\xi\right\rangle_{G}\right).
\end{aligned} J M ( ξ , G ) = 2 1 E λ M A ( x ) − A λ M G 2 + E λ M ⟨ A ( x ) − A λ M , A λ M + ξ ⟩ G − E λ M ⟨ A ( x ) − A λ M , y ⟩ G + 2 1 E λ M A λ M − y + ξ G 2 − 2 1 E p M A λ M − y + ξ G 2 + E p M log E μ 0 exp ( − 2 1 A ( x ) − A λ M G 2 − ⟨ A ( x ) − A λ M , A λ M − y + ξ ⟩ G ) . Since p ~ M \widetilde{p}_{M} p M λ ~ M \widetilde{\lambda}_{M} λ M
J ~ M ( ξ , G ) ≥ − E λ ~ M ⟨ A ^ ( x ) − A ^ λ ~ M , y ⟩ G + E p ~ M log E μ 0 exp ( − 1 2 ∥ A ^ ( x ) − A ^ λ ~ M ∥ G 2 − ⟨ A ^ ( x ) − A ^ λ ~ M , A ^ λ ~ M − y + ξ ⟩ G ) . \widetilde{\mathcal{J}}^{M}(\xi,G)\geq-\mathbb{E}^{\widetilde{\lambda}^{M}}\left\langle\widehat{A}(x)-\widehat{A}^{\widetilde{\lambda}^{M}},y\right\rangle_{G}
+\mathbb{E}^{\widetilde{p}^{M}}\log\mathbb{E}^{\mu_{0}}\exp\left(-\frac{1}{2}\left\|\widehat{A}(x)-\widehat{A}^{\widetilde{\lambda}^{M}}\right\|_{G}^{2}
-\left\langle\widehat{A}(x)-\widehat{A}^{\widetilde{\lambda}^{M}},\widehat{A}^{\widetilde{\lambda}^{M}}-y+\xi\right\rangle_{G}\right). J M ( ξ , G ) ≥ − E λ M ⟨ A ( x ) − A λ M , y ⟩ G + E p M log E μ 0 exp ( − 2 1 A ( x ) − A λ M G 2 − ⟨ A ( x ) − A λ M , A λ M − y + ξ ⟩ G ) . We introduce
( A ^ λ ~ M − y + ξ ) = r v ^ , r : = ∥ A ^ λ ~ M − y + ξ ∥ G , v ^ : = ( A ^ λ ~ M − y + ξ ) / ∥ A ^ λ ~ M − y + ξ ∥ G . \left(\widehat{A}^{\widetilde{\lambda}^{M}}-y+\xi\right)=r\widehat{v},\qquad r:=\left\|\widehat{A}^{\widetilde{\lambda}^{M}}-y+\xi\right\|_{G},\qquad\widehat{v}:=\left(\widehat{A}^{\widetilde{\lambda}^{M}}-y+\xi\right)\big/\left\|\widehat{A}^{\widetilde{\lambda}^{M}}-y+\xi\right\|_{G}. ( A λ M − y + ξ ) = r v , r := A λ M − y + ξ G , v := ( A λ M − y + ξ ) / A λ M − y + ξ G . By using e t ≥ t + e^{t}\geq t^{+} e t ≥ t + t ∈ R t\in\mathbb{R} t ∈ R
log E μ 0 exp ( − 1 2 ∥ A ^ ( x ) − A ^ λ ~ M ∥ G 2 − ⟨ A ^ ( x ) − A ^ λ ~ M , A ^ λ ~ M − y + ξ ⟩ G ) ≥ log E μ 0 exp ( − 1 2 ∥ A ^ ( x ) − A ^ λ ~ M ∥ G 2 ) ⟨ A ^ λ ~ M − A ^ ( x ) , A ^ λ ~ M − y + ξ ⟩ G + = log [ r E μ 0 exp ( − 1 2 ∥ A ^ ( x ) − A ^ λ ~ M ∥ G 2 ) ⟨ A ^ λ ~ M − A ^ ( x ) , v ^ ⟩ G + ] ≥ log r + log C ∗ ( λ ~ M ) . (28) \begin{aligned}
\log\mathbb{E}^{\mu_{0}}\exp\left(-\frac{1}{2}\left\|\widehat{A}(x)-\widehat{A}^{\widetilde{\lambda}^{M}}\right\|_{G}^{2}
-\left\langle\widehat{A}(x)-\widehat{A}^{\widetilde{\lambda}^{M}},\widehat{A}^{\widetilde{\lambda}^{M}}-y+\xi\right\rangle_{G}\right)
&\geq\log\mathbb{E}^{\mu_{0}}\exp\left(-\frac{1}{2}\left\|\widehat{A}(x)-\widehat{A}^{\widetilde{\lambda}^{M}}\right\|_{G}^{2}\right)
\left\langle\widehat{A}^{\widetilde{\lambda}^{M}}-\widehat{A}(x),\widehat{A}^{\widetilde{\lambda}^{M}}-y+\xi\right\rangle_{G}^{+}\\
&=\log\left[r\mathbb{E}^{\mu_{0}}\exp\left(-\frac{1}{2}\left\|\widehat{A}(x)-\widehat{A}^{\widetilde{\lambda}^{M}}\right\|_{G}^{2}\right)
\left\langle\widehat{A}^{\widetilde{\lambda}^{M}}-\widehat{A}(x),\widehat{v}\right\rangle_{G}^{+}\right]\\
&\geq\log r+\log C_{*}\left(\widetilde{\lambda}_{M}\right). \tag*{(28)}
\end{aligned} log E μ 0 exp ( − 2 1 A ( x ) − A λ M G 2 − ⟨ A ( x ) − A λ M , A λ M − y + ξ ⟩ G ) ≥ log E μ 0 exp ( − 2 1 A ( x ) − A λ M G 2 ) ⟨ A λ M − A ( x ) , A λ M − y + ξ ⟩ G + = log [ r E μ 0 exp ( − 2 1 A ( x ) − A λ M G 2 ) ⟨ A λ M − A ( x ) , v ⟩ G + ] ≥ log r + log C ∗ ( λ M ) . (28) Proposition 4.16. Suppose Assumption 1 is satisfied. If Assumption 2 is additionally satisfied, then the iterates ( ξ k , G k ) (\xi_{k},G_{k}) ( ξ k , G k ) J α \mathcal{J}_{\alpha} J α ξ 0 ∈ R m \xi_{0}\in\mathbb{R}^{m} ξ 0 ∈ R m ( ξ n ) n ∈ N (\xi_{n})_{n\in\mathbb{N}} ( ξ n ) n ∈ N J ( ξ n , G ) = J ( ξ 0 , G ) \mathcal{J}(\xi_{n},G)=\mathcal{J}(\xi_{0},G) J ( ξ n , G ) = J ( ξ 0 , G )
Proof. Let first Assumption 2 hold. By Lemma 4.7 , we need to show that J α \mathcal{J}_{\alpha} J α J α ( ξ k , G k ) → ∞ \mathcal{J}_{\alpha}(\xi_{k},G_{k})\to\infty J α ( ξ k , G k ) → ∞ ( ξ k , G k ) ∈ R m × S ( m ) (\xi_{k},G_{k})\in\mathbb{R}^{m}\times\mathcal{S}(m) ( ξ k , G k ) ∈ R m × S ( m ) ( ξ k , G k ) → ∞ (\xi_{k},G_{k})\to\infty ( ξ k , G k ) → ∞
Assume now ξ k → ∞ \xi_{k}\to\infty ξ k → ∞ G k G_{k} G k G k G_{k} G k k ∈ N k\in\mathbb{N} k ∈ N [ γ min , γ max ] [\gamma_{\min},\gamma_{\max}] [ γ m i n , γ m a x ] G k G_{k} G k
C ∗ ( λ ) = inf ∥ v ∥ G 2 = 1 , γ ( G ) ∈ [ γ min , γ max ] E μ 0 [ exp ( − 1 2 ∥ A ^ ( x ) − A ^ μ 0 ∥ G 2 ) ⟨ A ^ μ 0 − A ^ ( x ) , v ⟩ G + ] . C_{*}(\lambda)=\inf_{\|v\|_{G}^{2}=1,\ \gamma(G)\in[\gamma_{\min},\gamma_{\max}]}\mathbb{E}^{\mu_{0}}\left[\exp\left(-\frac{1}{2}\left\|\widehat{A}(x)-\widehat{A}^{\mu_{0}}\right\|_{G}^{2}\right)\left\langle\widehat{A}^{\mu_{0}}-\widehat{A}(x),v\right\rangle_{G}^{+}\right]. C ∗ ( λ ) = ∥ v ∥ G 2 = 1 , γ ( G ) ∈ [ γ m i n , γ m a x ] inf E μ 0 [ exp ( − 2 1 A ( x ) − A μ 0 G 2 ) ⟨ A μ 0 − A ( x ) , v ⟩ G + ] . We show now that C ∗ > 0 C_{*}>0 C ∗ > 0 C ∗ = 0 C_{*}=0 C ∗ = 0 v ∈ R m v\in\mathbb{R}^{m} v ∈ R m ⟨ A ^ μ 0 − A ^ ( x ) , v ⟩ G ≤ 0 \left\langle\widehat{A}^{\mu_{0}}-\widehat{A}(x),v\right\rangle_{G}\leq0 ⟨ A μ 0 − A ( x ) , v ⟩ G ≤ 0 E μ 0 ⟨ A ^ μ 0 − A ^ ( x ) , v ⟩ G = 0 \mathbb{E}^{\mu_{0}}\left\langle\widehat{A}^{\mu_{0}}-\widehat{A}(x),v\right\rangle_{G}=0 E μ 0 ⟨ A μ 0 − A ( x ) , v ⟩ G = 0 ⟨ A ^ μ 0 − A ^ ( x ) , v ⟩ G = 0 \left\langle\widehat{A}^{\mu_{0}}-\widehat{A}(x),v\right\rangle_{G}=0 ⟨ A μ 0 − A ( x ) , v ⟩ G = 0 ( p , λ ) (p,\lambda) ( p , λ ) ξ k → ∞ \xi_k \to\infty ξ k → ∞
J α ( ξ k , G k ) ≥ α R ( δ − 2 I ) + E p log ( C ∗ ( λ ) γ min ∥ A ^ μ 0 − y + ξ k ∥ 2 ) − E λ ⟨ A ^ ( x ) − A ^ μ 0 , y ⟩ G k . \mathcal{J}_{\alpha}(\xi_k,G_k) \ge\alpha R(\delta^{-2}I) + \mathbb{E}^{p}\log\left(C_*(\lambda)\sqrt{\gamma_{\min}}\left\|\widehat{A}^{\mu_0}-y+\xi_k\right\|_2\right)-\mathbb{E}^{\lambda}\langle\widehat{A}(x)-\widehat{A}^{\mu_0},y\rangle_{G_k}. J α ( ξ k , G k ) ≥ α R ( δ − 2 I ) + E p log ( C ∗ ( λ ) γ m i n A μ 0 − y + ξ k 2 ) − E λ ⟨ A ( x ) − A μ 0 , y ⟩ G k . The last term is bounded uniformly in k k k G k G_k G k [ γ min , γ max ] [\gamma_{\min},\gamma_{\max}] [ γ m i n , γ m a x ] λ \lambda λ
E k : = { ∥ y − A ^ μ 0 ∥ 2 ≤ ∥ ξ k ∥ 2 / 2 } E_k:=\left\{\left\|y-\widehat{A}^{\mu_0}\right\|_2\leq\|\xi_k\|_2/2\right\} E k := { y − A μ 0 2 ≤ ∥ ξ k ∥ 2 /2 } we have ∥ A ^ μ 0 − y + ξ k ∥ 2 ≥ ∥ ξ k ∥ 2 / 2 \left\|\widehat{A}^{\mu_0}-y+\xi_k\right\|_2\geq\|\xi_k\|_2/2 A μ 0 − y + ξ k 2 ≥ ∥ ξ k ∥ 2 /2 sup c ∈ R m E p log − ∥ y − c ∥ 2 < ∞ \sup_{c\in\mathbb{R}^m}\mathbb{E}^{p}\log^{-}\|y-c\|_2<\infty sup c ∈ R m E p log − ∥ y − c ∥ 2 < ∞ p p p p ( E k ) → 1 p(E_k)\to1 p ( E k ) → 1 E p log ∥ A ^ μ 0 − y + ξ k ∥ 2 → ∞ \mathbb{E}^{p}\log\left\|\widehat{A}^{\mu_0}-y+\xi_k\right\|_2\to\infty E p log A μ 0 − y + ξ k 2 → ∞ J α ( ξ k , G k ) → ∞ \mathcal{J}_{\alpha}(\xi_k,G_k)\to\infty J α ( ξ k , G k ) → ∞
If Assumption [2 ] is violated, then there exists v ≠ 0 v\ne0 v = 0 μ 0 \mu_0 μ 0 ⟨ A ^ ( x ) − A ^ μ 0 , v ⟩ G = 0 \langle\widehat{A}(x)-\widehat{A}^{\mu_0},v\rangle_G=0 ⟨ A ( x ) − A μ 0 , v ⟩ G = 0 ξ r : = r v + ξ 0 \xi_r:=rv+\xi_0 ξ r := r v + ξ 0 r > 0 r>0 r > 0 ( p , λ ) (p,\lambda) ( p , λ )
J α ( ξ r , G ) = α R ( G ) + 1 2 E λ ∥ A ^ ( x ) − A ^ μ 0 ∥ G 2 − E λ ⟨ A ^ ( x ) − A ^ μ 0 , y ⟩ G + E p log E μ 0 exp ( − 1 2 ∥ A ^ ( x ) − A ^ μ 0 ∥ G 2 − ⟨ A ^ ( x ) − A ^ μ 0 , A ^ μ 0 − y + ξ 0 ⟩ G − r ⟨ A ^ ( x ) − A ^ μ 0 , v ⟩ G ) = J α ( ξ 0 , G ) . \begin{aligned}
\mathcal{J}_{\alpha}(\xi_r,G)
&=\alpha R(G)+\tfrac12\mathbb{E}^{\lambda}\left\|\widehat{A}(x)-\widehat{A}^{\mu_0}\right\|_G^2-\mathbb{E}^{\lambda}\langle\widehat{A}(x)-\widehat{A}^{\mu_0},y\rangle_G\\
&\quad+\mathbb{E}^{p}\log\mathbb{E}^{\mu_0}\exp\left(-\tfrac12\left\|\widehat{A}(x)-\widehat{A}^{\mu_0}\right\|_G^2-\langle\widehat{A}(x)-\widehat{A}^{\mu_0},\widehat{A}^{\mu_0}-y+\xi_0\rangle_G-r\langle\widehat{A}(x)-\widehat{A}^{\mu_0},v\rangle_G\right)\\
&=\mathcal{J}_{\alpha}(\xi_0,G).
\end{aligned} J α ( ξ r , G ) = α R ( G ) + 2 1 E λ A ( x ) − A μ 0 G 2 − E λ ⟨ A ( x ) − A μ 0 , y ⟩ G + E p log E μ 0 exp ( − 2 1 A ( x ) − A μ 0 G 2 − ⟨ A ( x ) − A μ 0 , A μ 0 − y + ξ 0 ⟩ G − r ⟨ A ( x ) − A μ 0 , v ⟩ G ) = J α ( ξ 0 , G ) . Proposition 4.17 (Empirical objectives). Let λ ~ \widetilde{\lambda} λ R n × R m \mathbb{R}^n\times\mathbb{R}^m R n × R m p ~ \widetilde{p} p R m \mathbb{R}^m R m y y y p ~ \widetilde{p} p μ ~ 0 ( y ) \widetilde{\mu}_0^{(y)} μ 0 ( y ) R n \mathbb{R}^n R n
J ~ α ( ξ , G ) : = α R ( G ) + 1 2 E λ ~ ∥ A ^ ( x ) − y + ξ ∥ G 2 + E p ~ log E μ ~ 0 ( y ) exp ( − 1 2 ∥ A ^ ( x ′ ) − y + ξ ∥ G 2 ) . \widetilde{\mathcal{J}}_{\alpha}(\xi,G):=\alpha R(G)+\tfrac12\mathbb{E}^{\widetilde{\lambda}}\left\|\widehat{A}(x)-y+\xi\right\|_G^2+\mathbb{E}^{\widetilde{p}}\log\mathbb{E}^{\widetilde{\mu}_0^{(y)}}\exp\left(-\tfrac12\left\|\widehat{A}(x')-y+\xi\right\|_G^2\right). J α ( ξ , G ) := α R ( G ) + 2 1 E λ A ( x ) − y + ξ G 2 + E p log E μ 0 ( y ) exp ( − 2 1 A ( x ′ ) − y + ξ G 2 ) . Assume
( E 1 ) (E1) ( E 1 ) y ∈ supp p ~ y\in\operatorname{supp}\widetilde{p} y ∈ supp p v ≠ 0 v\ne0 v = 0 v ⋅ A ^ v\cdot\widehat{A} v ⋅ A supp μ ~ 0 ( y ) \operatorname{supp}\widetilde{\mu}_0^{(y)} supp μ 0 ( y )
( E 2 ) (E2) ( E 2 ) A ^ λ ~ : = E λ ~ A ^ ( x ) \widehat{A}^{\widetilde{\lambda}}:=\mathbb{E}^{\widetilde{\lambda}}\widehat{A}(x) A λ := E λ A ( x ) conv A ^ ( supp μ ~ 0 ( y ) ) \operatorname{conv}\widehat{A}(\operatorname{supp}\widetilde{\mu}_0^{(y)}) conv A ( supp μ 0 ( y ) ) y ∈ supp p ~ y\in\operatorname{supp}\widetilde{p} y ∈ supp p
Then J ~ α \widetilde{\mathcal{J}}_{\alpha} J α J ~ α \widetilde{\mathcal{J}}_{\alpha} J α
Proof. The proofs of Theorem 2.7 and Lemma 4.13 use only that p ~ \widetilde{p} p y y y λ ~ \widetilde{\lambda} λ y y y (16) and of the cross term in (18) ) and Lemma 4.12 for the inner measure, which is (E1). Lemma 4.15 holds with μ 0 \mu_0 μ 0 μ ~ 0 ( y ) \widetilde{\mu}_0^{(y)} μ 0 ( y ) E p ~ \mathbb{E}^{\widetilde{p}} E p C ∗ ( λ ~ ) > 0 C_*(\widetilde{\lambda})>0 C ∗ ( λ ) > 0 C ∗ = 0 C_*=0 C ∗ = 0 A ^ λ ~ \widehat{A}^{\widetilde{\lambda}} A λ A ^ ( supp μ ~ 0 ( y ) ) \widehat{A}(\operatorname{supp}\widetilde{\mu}_0^{(y)}) A ( supp μ 0 ( y ) )
Remark 4.18. For the single-loop estimator of Section 5, where μ ~ 0 ( y ) = μ 0 \widetilde{\mu}_0^{(y)}=\mu_0 μ 0 ( y ) = μ 0 A ^ \widehat{A} A [2 ] and (E2) holds automatically, since A ^ # μ 0 \widehat{A}_{\#}\mu_0 A # μ 0
For the double-loop estimator, where μ ~ 0 ( y j ) \widetilde{\mu}_0^{(y_j)} μ 0 ( y j ) ( x i , j ) i = 1 L (x_{i,j})_{i=1}^{L} ( x i , j ) i = 1 L A ^ ( supp μ 0 ) \widehat{A}(\operatorname{supp}\mu_0) A ( supp μ 0 ) R m \mathbb{R}^m R m A ^ μ 0 \widehat{A}^{\mu_0} A μ 0 conv supp ( A ^ # μ 0 ) \operatorname{conv}\operatorname{supp}(\widehat{A}_{\#}\mu_0) conv supp ( A # μ 0 ) L , M → ∞ L,M\to\infty L , M → ∞ L L L J ~ α \widetilde{\mathcal{J}}_{\alpha} J α ξ \xi ξ
Concentration under empirical approximations
In many applications, the full Bayesian joint distribution λ \lambda λ
Recall from (12) that
J ( ξ , G ) = 1 2 E λ ∥ A ^ ( x ) − y + ξ ∥ G 2 + E P log Z ^ ( y ; ξ , G ) , Z ^ ( y ; ξ , G ) = E μ 0 exp ( − 1 2 ∥ A ^ ( x ) − y + ξ ∥ G 2 ) . \mathcal{J}(\xi,G)=\frac{1}{2}\mathbb{E}^{\lambda}\left\|\widehat{A}(x)-y+\xi\right\|_{G}^{2}+\mathbb{E}^{\mathbb{P}}\log\widehat{Z}(y;\xi,G),\qquad\widehat{Z}(y;\xi,G)=\mathbb{E}^{\mu_{0}}\exp\left(-\frac{1}{2}\left\|\widehat{A}(x)-y+\xi\right\|_{G}^{2}\right). J ( ξ , G ) = 2 1 E λ A ( x ) − y + ξ G 2 + E P log Z ( y ; ξ , G ) , Z ( y ; ξ , G ) = E μ 0 exp ( − 2 1 A ( x ) − y + ξ G 2 ) . An obstruction to a direct sample-average approximation of J \mathcal{J} J E P log Z ^ ( y ; ξ , G ) \mathbb{E}^{\mathbb{P}}\log\widehat{Z}(y;\xi,G) E P log Z ( y ; ξ , G ) λ \lambda λ ( x j , y j ) (x_j,y_j) ( x j , y j ) Z ^ ( y ; ξ , G ) \widehat{Z}(y;\xi,G) Z ( y ; ξ , G ) μ 0 \mu_0 μ 0 Z ^ ( y ; ξ , G ) \widehat{Z}(y;\xi,G) Z ( y ; ξ , G ) X = R n X=\mathbb{R}^n X = R n
Definition 5.1 (Single-loop estimators). Given i.i.d. samples ( x j , y j ) j = 1 M ∼ λ (x_j,y_j)_{j=1}^{M}\sim\lambda ( x j , y j ) j = 1 M ∼ λ μ 0 ∼ N ( x ˉ , C 0 ) \mu_0\sim\mathcal{N}(\bar{x},C_0) μ 0 ∼ N ( x ˉ , C 0 ) R n \mathbb{R}^n R n p 0 p_0 p 0 A ^ \widehat{A} A
J ^ α M , ∞ ( ξ , G ) : = α R ( G ) + 1 2 log det C ^ ( G ) + 1 2 M ∑ j = 1 M ∥ x j − m ^ ( y j ; ξ , G ) ∥ C ^ ( G ) − 1 2 + 1 M ∑ j = 1 M log p 0 ( x j ) + n 2 log ( 2 π ) , \widehat{\mathcal{J}}_{\alpha}^{M,\infty}(\xi,G):=\alpha R(G)+\frac{1}{2}\log\det\widehat{C}(G)+\frac{1}{2M}\sum_{j=1}^{M}\left\|x_j-\widehat{m}(y_j;\xi,G)\right\|_{\widehat{C}(G)^{-1}}^{2}+\frac{1}{M}\sum_{j=1}^{M}\log p_0(x_j)+\frac{n}{2}\log(2\pi), J α M , ∞ ( ξ , G ) := α R ( G ) + 2 1 log det C ( G ) + 2 M 1 j = 1 ∑ M ∥ x j − m ( y j ; ξ , G ) ∥ C ( G ) − 1 2 + M 1 j = 1 ∑ M log p 0 ( x j ) + 2 n log ( 2 π ) , which serves as an approximation of (23) . Indeed, by the definition of q ^ \widehat{q} q ( x , y ) ∈ R n × R m (x,y)\in\mathbb{R}^n\times\mathbb{R}^m ( x , y ) ∈ R n × R m
1 2 ∥ A ^ ( x ) − y + ξ ∥ G 2 + log Z ^ ( y ; ξ , G ) = log p 0 ( x ) − log q ^ ( x ∣ y ; ξ , G ) , \frac{1}{2}\left\|\widehat{A}(x)-y+\xi\right\|_{G}^{2}+\log\widehat{Z}(y;\xi,G)=\log p_0(x)-\log\widehat{q}(x|y;\xi,G), 2 1 A ( x ) − y + ξ G 2 + log Z ( y ; ξ , G ) = log p 0 ( x ) − log q ( x ∣ y ; ξ , G ) , so that J ^ α M , ∞ \widehat{\mathcal{J}}_{\alpha}^{M,\infty} J α M , ∞ (27) for ( p ~ M , λ ~ M ) = ( 1 M ∑ j = 1 M δ y j , 1 M ∑ j = 1 M δ x j , y j ) (\widetilde{p}_M,\widetilde{\lambda}_M)=\left(\frac{1}{M}\sum_{j=1}^{M}\delta_{y_j},\frac{1}{M}\sum_{j=1}^{M}\delta_{x_j,y_j}\right) ( p M , λ M ) = ( M 1 ∑ j = 1 M δ y j , M 1 ∑ j = 1 M δ x j , y j ) 1 M ∑ j = 1 M log p 0 ( x j ) \frac{1}{M}\sum_{j=1}^{M}\log p_0(x_j) M 1 ∑ j = 1 M log p 0 ( x j ) ξ , G \xi,G ξ , G
We recall that posterior mean and covariance satisfy
m ^ ( y ; ξ , G ) = C ^ ( G ) ( C 0 − 1 x ˉ + A ^ T G ( y − ξ ) ) , C ^ ( G ) = ( C 0 − 1 + A ^ T G A ^ ) − 1 . \widehat{m}(y;\xi,G)=\widehat{C}(G)\left(C_0^{-1}\bar{x}+\widehat{A}^{\mathsf T}G(y-\xi)\right),\qquad\widehat{C}(G)=\left(C_0^{-1}+\widehat{A}^{\mathsf T}G\widehat{A}\right)^{-1}. m ( y ; ξ , G ) = C ( G ) ( C 0 − 1 x ˉ + A T G ( y − ξ ) ) , C ( G ) = ( C 0 − 1 + A T G A ) − 1 . We recall that convergence of the alternating optimisation algorithm is guaranteed for empirical estimators as pointed out in Proposition 4.17 and Remark 4.18. Our objective is to show that a sequence of optimisers of the empirical single-loop problems J ^ α M , ∞ \widehat{\mathcal{J}}_{\alpha}^{M,\infty} J α M , ∞
Theorem 5.2 ([48 Theorem 5.14] ). Denote by Θ \Theta Θ d d d l : X × R m × Θ → R l:X\times\mathbb{R}^{m}\times\Theta\to\mathbb{R} l : X × R m × Θ → R θ ∈ Θ \theta\in\Theta θ ∈ Θ ( x , y ) ∈ X × R m (x,y)\in X\times\mathbb{R}^{m} ( x , y ) ∈ X × R m U ⊂ Θ U\subset\Theta U ⊂ Θ
E λ [ inf θ ∈ U l θ ( x , y ) ] > − ∞ . (29) \mathbb{E}^{\lambda}\left[\inf_{\theta\in U}l_{\theta}(x,y)\right]>-\infty. \tag*{(29)} E λ [ θ ∈ U inf l θ ( x , y ) ] > − ∞. (29) Assume that the set of minimisers Θ 0 \Theta_0 Θ 0 θ ↦ E λ l θ ( x , y ) \theta\mapsto\mathbb{E}^{\lambda}l_{\theta}(x,y) θ ↦ E λ l θ ( x , y ) ( θ ^ M ) M ∈ N (\widehat{\theta}_M)_{M\in\mathbb{N}} ( θ M ) M ∈ N
1 M ∑ j = 1 M l θ ^ M ( x j , y j ) ≤ 1 M ∑ j = 1 M l θ 0 ( x j , y j ) + o P ( 1 ) . (30) \frac{1}{M}\sum_{j=1}^{M}l_{\widehat{\theta}_M}(x_j,y_j)\leq\frac{1}{M}\sum_{j=1}^{M}l_{\theta_0}(x_j,y_j)+o_P(1). \tag*{(30)} M 1 j = 1 ∑ M l θ M ( x j , y j ) ≤ M 1 j = 1 ∑ M l θ 0 ( x j , y j ) + o P ( 1 ) . (30) Then for any ε > 0 \varepsilon>0 ε > 0 K ⊂ Θ K\subset\Theta K ⊂ Θ
P ( d ( θ ^ M , Θ 0 ) > ε ∧ θ ^ M ∈ K ) → 0. \mathbb{P}\left(d(\widehat{\theta}_M,\Theta_0)>\varepsilon\land\widehat{\theta}_M\in K\right)\to0. P ( d ( θ M , Θ 0 ) > ε ∧ θ M ∈ K ) → 0. Exact minimisers of θ ↦ 1 M ∑ j = 1 M ℓ θ ( x j , y j ) \theta\mapsto\frac{1}{M}\sum_{j=1}^{M}\ell_\theta(x_j,y_j) θ ↦ M 1 ∑ j = 1 M ℓ θ ( x j , y j ) (30) trivially; the weaker form is needed for the plug-in estimator of Corollary 5.5 below. To ensure the required containment of the empirical estimators in a compact set with high probability, we will use the following lemma.
Lemma 5.3. Let Assumptions 1 and 2 hold. Assume that μ 0 = N ( x ˉ , C 0 ) \mu_0=\mathcal{N}(\bar{x},C_0) μ 0 = N ( x ˉ , C 0 ) p 0 p_0 p 0 A ^ : R n → R m \widehat{A}:\mathbb{R}^n\to\mathbb{R}^m A : R n → R m
( ξ ^ M , G ^ M ) ∈ argmin ( ξ , G ) ∈ R m × S ( m ) J ^ α M , ∞ ( ξ , G ) . (31) (\widehat{\xi}_M,\widehat{G}_M)\in\underset{(\xi,G)\in\mathbb{R}^m\times\mathcal{S}(m)}{\operatorname{argmin}}\ \widehat{\mathcal{J}}_{\alpha}^{M,\infty}(\xi,G). \tag*{(31)} ( ξ M , G M ) ∈ ( ξ , G ) ∈ R m × S ( m ) argmin J α M , ∞ ( ξ , G ) . (31) Then for every κ > 0 \kappa>0 κ > 0 K κ ⊂ R m × S ( m ) K_\kappa\subset\mathbb{R}^m\times\mathcal{S}(m) K κ ⊂ R m × S ( m )
P ( ( ξ ^ M , G ^ M ) ∈ K κ for all M ∈ N ) ≥ 1 − κ . \mathbb{P}\left((\widehat{\xi}_M,\widehat{G}_M)\in K_\kappa\text{ for all }M\in\mathbb{N}\right)\ge1-\kappa. P ( ( ξ M , G M ) ∈ K κ for all M ∈ N ) ≥ 1 − κ . Proof. Write ℓ M : = 1 M ∑ j = 1 M log p 0 ( x j ) + n 2 log ( 2 π ) \ell_M:=\frac{1}{M}\sum_{j=1}^{M}\log p_0(x_j)+\frac{n}{2}\log(2\pi) ℓ M := M 1 ∑ j = 1 M log p 0 ( x j ) + 2 n log ( 2 π ) E μ 0 log p 0 ( x ) > − ∞ \mathbb{E}^{\mu_0}\log p_0(x)>-\infty E μ 0 log p 0 ( x ) > − ∞ M M M
J ^ α M , ∞ ( 0 , I m ) − ℓ M → α R ( I m ) + 1 2 log det C ^ ( I m ) + 1 2 E μ 0 ∥ x − m ^ ( y ; 0 , I m ) ∥ C ^ ( I m ) − 1 2 < ∞ \widehat{\mathcal{J}}_{\alpha}^{M,\infty}(0,I_m)-\ell_M\to\alpha\mathcal{R}(I_m)+\frac{1}{2}\log\det\widehat{C}(I_m)+\frac{1}{2}\mathbb{E}^{\mu_0}\left\|x-\widehat{m}(y;0,I_m)\right\|_{\widehat{C}(I_m)^{-1}}^2<\infty J α M , ∞ ( 0 , I m ) − ℓ M → α R ( I m ) + 2 1 log det C ( I m ) + 2 1 E μ 0 ∥ x − m ( y ; 0 , I m ) ∥ C ( I m ) − 1 2 < ∞ almost surely, so that by optimality
B : = sup M ∈ N ( J ^ α M , ∞ ( ξ ^ M , G ^ M ) − ℓ M ) ≤ sup M ∈ N ( J ^ α M , ∞ ( 0 , I m ) − ℓ M ) < ∞ B:=\sup_{M\in\mathbb{N}}\left(\widehat{\mathcal{J}}_{\alpha}^{M,\infty}(\widehat{\xi}_M,\widehat{G}_M)-\ell_M\right)\le\sup_{M\in\mathbb{N}}\left(\widehat{\mathcal{J}}_{\alpha}^{M,\infty}(0,I_m)-\ell_M\right)<\infty B := M ∈ N sup ( J α M , ∞ ( ξ M , G M ) − ℓ M ) ≤ M ∈ N sup ( J α M , ∞ ( 0 , I m ) − ℓ M ) < ∞ almost surely.
Confinement of G G G Since the quadratic term is non-negative and log det C ^ ( G ) = − log det ( C 0 − 1 + A ^ T G A ^ ) \log\det\widehat{C}(G)=-\log\det(C_0^{-1}+\widehat{A}^{\mathsf{T}}G\widehat{A}) log det C ( G ) = − log det ( C 0 − 1 + A T G A ) ( ξ , G ) ∈ R m × S ( m ) (\xi,G)\in\mathbb{R}^m\times\mathcal{S}(m) ( ξ , G ) ∈ R m × S ( m )
J ^ α M , ∞ ( ξ , G ) − ℓ M ≥ α R ( G ) − 1 2 log det ( C 0 − 1 + A ^ T G A ^ ) . \widehat{\mathcal{J}}_{\alpha}^{M,\infty}(\xi,G)-\ell_M\ge\alpha\mathcal{R}(G)-\frac{1}{2}\log\det\left(C_0^{-1}+\widehat{A}^{\mathsf{T}}G\widehat{A}\right). J α M , ∞ ( ξ , G ) − ℓ M ≥ α R ( G ) − 2 1 log det ( C 0 − 1 + A T G A ) . Since A ^ T G A ^ ⪯ ∥ A ^ ∥ F 2 tr ( G ) I n \widehat{A}^{\mathsf{T}}G\widehat{A}\preceq\lVert\widehat{A}\rVert_F^2\operatorname{tr}(G)I_n A T G A ⪯ ∥ A ∥ F 2 tr ( G ) I n
det ( C 0 − 1 ( I + C 0 1 / 2 A ^ T G A ^ C 0 1 / 2 ) ) ≤ det C 0 − 1 ( 1 + ∥ C 0 1 / 2 ∥ F 2 ∥ A ^ ∥ F 2 tr ( G ) ) n \det\left(C_0^{-1}\left(I+C_0^{1/2}\widehat{A}^{\mathsf{T}}G\widehat{A}C_0^{1/2}\right)\right)\le\det C_0^{-1}\left(1+\left\|C_0^{1/2}\right\|_F^2\left\|\widehat{A}\right\|_F^2\operatorname{tr}(G)\right)^n det ( C 0 − 1 ( I + C 0 1/2 A T G A C 0 1/2 ) ) ≤ det C 0 − 1 ( 1 + C 0 1/2 F 2 A F 2 tr ( G ) ) n and hence, for all M ∈ N M\in\mathbb{N} M ∈ N
B ≥ α R ( G ^ M ) + log det ( C 0 1 / 2 ) − n 2 log ( 1 + ∥ C 0 1 / 2 ∥ F 2 ∥ A ^ ∥ F 2 tr ( G ^ M ) ) . B\ge\alpha\mathcal{R}(\widehat{G}_M)+\log\det\left(C_0^{1/2}\right)-\frac{n}{2}\log\left(1+\left\|C_0^{1/2}\right\|_F^2\left\|\widehat{A}\right\|_F^2\operatorname{tr}(\widehat{G}_M)\right). B ≥ α R ( G M ) + log det ( C 0 1/2 ) − 2 n log ( 1 + C 0 1/2 F 2 A F 2 tr ( G M ) ) . By Lemma 4.1 (ii), R ( G ^ M ) \mathcal{R}(\widehat{G}_M) R ( G M ) γ max ( G ^ M ) \gamma_{\max}(\widehat{G}_M) γ m a x ( G M ) γ min ( G ^ M ) → 0 \gamma_{\min}(\widehat{G}_M)\to0 γ m i n ( G M ) → 0 γ max ( G ^ M ) \gamma_{\max}(\widehat{G}_M) γ m a x ( G M ) 0 < γ min ≤ γ max < ∞ 0<\gamma_{\min}\le\gamma_{\max}<\infty 0 < γ m i n ≤ γ m a x < ∞ G ^ M \widehat{G}_M G M [ γ min , γ max ] [\gamma_{\min},\gamma_{\max}] [ γ m i n , γ m a x ] M ∈ N M\in\mathbb{N} M ∈ N
Confinement of ξ \xi ξ For fixed G ∈ S ( m ) G\in\mathcal{S}(m) G ∈ S ( m ) ξ ↦ J ^ α M , ∞ ( ξ , G ) \xi\mapsto\widehat{\mathcal{J}}_{\alpha}^{M,\infty}(\xi,G) ξ ↦ J α M , ∞ ( ξ , G ) ∂ ξ m ^ ( y ; ξ , G ) = − C ^ ( G ) A ^ T G \partial_\xi\widehat{m}(y;\xi,G)=-\widehat{C}(G)\widehat{A}^{\mathsf{T}}G ∂ ξ m ( y ; ξ , G ) = − C ( G ) A T G
∇ ξ J ^ α M , ∞ ( ξ , G ) = G A ^ 1 M ∑ j = 1 M ( x j − m ^ ( y j ; ξ , G ) ) = G A ^ ( x ˉ M − C ^ ( G ) ( C 0 − 1 x ˉ + A ^ T G ( y ˉ M − ξ ) ) ) . \nabla_\xi\widehat{\mathcal{J}}_{\alpha}^{M,\infty}(\xi,G)=G\widehat{A}\frac{1}{M}\sum_{j=1}^{M}\left(x_j-\widehat{m}(y_j;\xi,G)\right)=G\widehat{A}\left(\bar{x}_M-\widehat{C}(G)\left(C_0^{-1}\bar{x}+\widehat{A}^{\mathsf{T}}G(\bar{y}_M-\xi)\right)\right). ∇ ξ J α M , ∞ ( ξ , G ) = G A M 1 j = 1 ∑ M ( x j − m ( y j ; ξ , G ) ) = G A ( x ˉ M − C ( G ) ( C 0 − 1 x ˉ + A T G ( y ˉ M − ξ ) ) ) . where x ˉ M : = 1 M ∑ j = 1 M x j \bar{x}_M:=\frac{1}{M}\sum_{j=1}^{M}x_j x ˉ M := M 1 ∑ j = 1 M x j y ˉ M : = 1 M ∑ j = 1 M y j \bar{y}_M:=\frac{1}{M}\sum_{j=1}^{M}y_j y ˉ M := M 1 ∑ j = 1 M y j A ^ \widehat{A} A A ^ C ^ ( G ) A ^ T \widehat{A}\widehat{C}(G)\widehat{A}^{\mathsf{T}} A C ( G ) A T ξ \xi ξ
ξ ^ M = y ˉ M − ( A ^ C ^ ( G ^ M ) A ^ T G ^ M ) − 1 A ^ ( x ˉ M − C ^ ( G ^ M ) C 0 − 1 x ˉ ) . (32) \widehat{\xi}_M=\bar{y}_M-\left(\widehat{A}\widehat{C}(\widehat{G}_M)\widehat{A}^{\mathsf{T}}\widehat{G}_M\right)^{-1}\widehat{A}\left(\bar{x}_M-\widehat{C}(\widehat{G}_M)C_0^{-1}\bar{x}\right). \tag*{(32)} ξ M = y ˉ M − ( A C ( G M ) A T G M ) − 1 A ( x ˉ M − C ( G M ) C 0 − 1 x ˉ ) . (32) The sample means converge almost surely by the law of large numbers, and G ↦ ( A ^ C ^ ( G ) A ^ T G ) − 1 G\mapsto\left(\widehat{A}\widehat{C}(G)\widehat{A}^{\mathsf{T}}G\right)^{-1} G ↦ ( A C ( G ) A T G ) − 1 { G ∈ S ( m ) : γ ( G ) ∈ [ γ min , γ max ] } \{G\in\mathcal{S}(m):\gamma(G)\in[\gamma_{\min},\gamma_{\max}]\} { G ∈ S ( m ) : γ ( G ) ∈ [ γ m i n , γ m a x ]} sup M ∈ N ∥ ξ ^ M ∥ 2 < ∞ \sup_{M\in\mathbb{N}}\lVert\widehat{\xi}_M\rVert_2<\infty sup M ∈ N ∥ ξ M ∥ 2 < ∞
From almost-sure to uniform confinement. Set
ρ : = max { sup M ∈ N γ max ( G ^ M ) , sup M ∈ N γ min ( G ^ M ) − 1 , sup M ∈ N ∥ ξ ^ M ∥ 2 } . \rho:= \max\left\{\sup_{M\in\mathbb{N}}\gamma_{\max}(\widehat{G}_M),\ \sup_{M\in\mathbb{N}}\gamma_{\min}(\widehat{G}_M)^{-1},\ \sup_{M\in\mathbb{N}}\|\widehat{\xi}_M\|_2\right\}. ρ := max { M ∈ N sup γ m a x ( G M ) , M ∈ N sup γ m i n ( G M ) − 1 , M ∈ N sup ∥ ξ M ∥ 2 } . We have shown that ρ < ∞ \rho< \infty ρ < ∞ { ρ ≤ n } \{\rho\le n\} { ρ ≤ n } { ρ < ∞ } \{\rho< \infty\} { ρ < ∞ } P \mathbb{P} P P ( ρ ≤ n ) → 1 \mathbb{P}(\rho\le n) \to1 P ( ρ ≤ n ) → 1 n → ∞ n \to\infty n → ∞ [14 Theorem 1.8] ). Hence for every κ > 0 \kappa> 0 κ > 0 ρ κ < ∞ \rho_\kappa< \infty ρ κ < ∞ P ( ρ > ρ κ ) < κ \mathbb{P}(\rho> \rho_\kappa) < \kappa P ( ρ > ρ κ ) < κ
K κ : = { ( ξ , G ) ∈ R m × S ( m ) : ∥ ξ ∥ 2 ≤ ρ κ , ρ κ − 1 I m ⪯ G ⪯ ρ κ I m } K_\kappa:= \left\{(\xi,G)\in\mathbb{R}^m\times\mathcal{S}(m):\|\xi\|_2\le\rho_\kappa,\ \rho_\kappa^{-1}I_m\preceq G\preceq\rho_\kappa I_m\right\} K κ := { ( ξ , G ) ∈ R m × S ( m ) : ∥ ξ ∥ 2 ≤ ρ κ , ρ κ − 1 I m ⪯ G ⪯ ρ κ I m } is a deterministic compact subset of R m × S ( m ) \mathbb{R}^m\times\mathcal{S}(m) R m × S ( m ) { ρ ≤ ρ κ } ⊂ { ( ξ ^ M , G ^ M ) ∈ K κ ∀ M } \{\rho\le\rho_\kappa\}\subset\{(\widehat{\xi}_M,\widehat{G}_M)\in K_\kappa\ \forall M\} { ρ ≤ ρ κ } ⊂ {( ξ M , G M ) ∈ K κ ∀ M }
P ( ( ξ ^ M , G ^ M ) ∈ K κ ∀ M ) ≥ P ( ρ ≤ ρ κ ) > 1 − κ . \mathbb{P}\left((\widehat{\xi}_M,\widehat{G}_M)\in K_\kappa\ \forall M\right)\ge\mathbb{P}(\rho\le\rho_\kappa)>1-\kappa. P ( ( ξ M , G M ) ∈ K κ ∀ M ) ≥ P ( ρ ≤ ρ κ ) > 1 − κ . Theorem 5.4
Theorem 5.4. Let d d d R m × S ( m ) \mathbb{R}^m\times\mathcal{S}(m) R m × S ( m ) μ 0 = N ( x ‾ , C 0 ) \mu_0=\mathcal{N}(\overline{x},C_0) μ 0 = N ( x , C 0 ) p 0 p_0 p 0 A ^ : R n → R m \widehat{A}:\mathbb{R}^n\to\mathbb{R}^m A : R n → R m
Θ 0 : = argmin ( ξ , G ) ∈ R m × S ( m ) J α ( ξ , G ) \Theta_0:=\underset{(\xi,G)\in\mathbb{R}^m\times\mathcal{S}(m)}{\operatorname{argmin}}\ \mathcal{J}_\alpha(\xi,G) Θ 0 := ( ξ , G ) ∈ R m × S ( m ) argmin J α ( ξ , G ) is non-empty (Remark 4.10), and the estimator ( ξ ^ M , G ^ M ) (\widehat{\xi}_M,\widehat{G}_M) ( ξ M , G M ) (31) for the objective J ^ α M , ∞ \widehat{\mathcal{J}}_\alpha^{M,\infty} J α M , ∞ ε > 0 \varepsilon>0 ε > 0
P [ d ( ( ξ ^ M , G ^ M ) , Θ 0 ) > ε ] ⟶ 0 , M → ∞ . \mathbb{P}\left[d\left((\widehat{\xi}_M,\widehat{G}_M),\Theta_0\right)>\varepsilon\right]\longrightarrow0,\qquad M\to\infty. P [ d ( ( ξ M , G M ) , Θ 0 ) > ε ] ⟶ 0 , M → ∞. Proof. We apply Theorem 5.2 with θ = ( ξ , G ) \theta=(\xi,G) θ = ( ξ , G )
ℓ θ ( x , y ) : = α R ( G ) + 1 2 log det C ^ ( G ) + 1 2 ∥ x − m ^ ( y ; ξ , G ) ∥ C ^ ( G ) − 1 2 + log p 0 ( x ) , \ell_\theta(x,y):=\alpha R(G)+\frac{1}{2}\log\det\widehat{C}(G)+\frac{1}{2}\left\|x-\widehat{m}(y;\xi,G)\right\|_{\widehat{C}(G)^{-1}}^2+\log p_0(x), ℓ θ ( x , y ) := α R ( G ) + 2 1 log det C ( G ) + 2 1 ∥ x − m ( y ; ξ , G ) ∥ C ( G ) − 1 2 + log p 0 ( x ) , so that J ^ α M , ∞ ( θ ) = 1 M ∑ j = 1 M ℓ θ ( x j , y j ) \widehat{\mathcal{J}}_\alpha^{M,\infty}(\theta)=\frac{1}{M}\sum_{j=1}^{M}\ell_\theta(x_j,y_j) J α M , ∞ ( θ ) = M 1 ∑ j = 1 M ℓ θ ( x j , y j ) E ℓ θ ( x , y ) = J α ( θ ) \mathbb{E}\ell_\theta(x,y)=\mathcal{J}_\alpha(\theta) E ℓ θ ( x , y ) = J α ( θ ) (23) ; hence Θ 0 \Theta_0 Θ 0 E ℓ θ \mathbb{E}\ell_\theta E ℓ θ θ ↦ ℓ θ ( x , y ) \theta\mapsto\ell_\theta(x,y) θ ↦ ℓ θ ( x , y ) ℓ θ ( x , y ) ≥ α R ( G ) + 1 2 log det C ^ ( G ) + log p 0 ( x ) \ell_\theta(x,y)\ge\alpha R(G)+\frac{1}{2}\log\det\widehat{C}(G)+\log p_0(x) ℓ θ ( x , y ) ≥ α R ( G ) + 2 1 log det C ( G ) + log p 0 ( x ) θ \theta θ E μ 0 log p 0 ( x ) > − ∞ \mathbb{E}^{\mu_0}\log p_0(x)>-\infty E μ 0 log p 0 ( x ) > − ∞ ( ξ ^ M , G ^ M ) (\widehat{\xi}_M,\widehat{G}_M) ( ξ M , G M ) κ > 0 \kappa>0 κ > 0
lim M → ∞ P [ d ( ( ξ ^ M , G ^ M ) , Θ 0 ) > ε ] ≤ lim M → ∞ P [ d ( ( ξ ^ M , G ^ M ) , Θ 0 ) > ε , ( ξ ^ M , G ^ M ) ∈ K κ ] + lim M → ∞ P [ ( ξ ^ M , G ^ M ) ∉ K κ ] . \lim_{M\to\infty}\mathbb{P}\left[d\left((\widehat{\xi}_M,\widehat{G}_M),\Theta_0\right)>\varepsilon\right]
\le
\lim_{M\to\infty}\mathbb{P}\left[d\left((\widehat{\xi}_M,\widehat{G}_M),\Theta_0\right)>\varepsilon,\ (\widehat{\xi}_M,\widehat{G}_M)\in K_\kappa\right]
+\lim_{M\to\infty}\mathbb{P}\left[(\widehat{\xi}_M,\widehat{G}_M)\notin K_\kappa\right]. M → ∞ lim P [ d ( ( ξ M , G M ) , Θ 0 ) > ε ] ≤ M → ∞ lim P [ d ( ( ξ M , G M ) , Θ 0 ) > ε , ( ξ M , G M ) ∈ K κ ] + M → ∞ lim P [ ( ξ M , G M ) ∈ / K κ ] . By applying Theorem 5.2 , the first term equals zero, whereas by Lemma 5.3 , the second term is bounded by κ > 0 \kappa>0 κ > 0
The following corollary highlights a particular way to drop Assumption 2 by cleverly defining estimators of ξ \xi ξ
Corollary 5.5. Let Assumption 1 hold. Assume that μ 0 = N ( x ‾ , C 0 ) \mu_0=\mathcal{N}(\overline{x},C_0) μ 0 = N ( x , C 0 ) p 0 p_0 p 0 A ^ : R n → R m \widehat{A}:\mathbb{R}^n\to\mathbb{R}^m A : R n → R m
ξ ^ M = 1 M ∑ j = 1 M ( y j − A ^ ( x j ) ) , G ^ M = argmin G ∈ S ( m ) J ^ α M , ∞ ( ξ ^ M , G ) . \widehat{\xi}_M=\frac{1}{M}\sum_{j=1}^{M}\left(y_j-\widehat{A}(x_j)\right),\qquad
\widehat{G}_M=\underset{G\in\mathcal{S}(m)}{\operatorname{argmin}}\ \widehat{\mathcal{J}}_\alpha^{M,\infty}(\widehat{\xi}_M,G). ξ M = M 1 j = 1 ∑ M ( y j − A ( x j ) ) , G M = G ∈ S ( m ) argmin J α M , ∞ ( ξ M , G ) . Then
P [ d ( ( ξ ^ M , G ^ M ) , Θ 0 ) > ε ] ⟶ 0 , M → ∞ . \mathbb{P}\left[d\left((\widehat{\xi}_M,\widehat{G}_M),\Theta_0\right)>\varepsilon\right]\longrightarrow0,\qquad M\to\infty. P [ d ( ( ξ M , G M ) , Θ 0 ) > ε ] ⟶ 0 , M → ∞. Proof. By the law of large numbers, ξ ^ M → ξ ^ : = E λ ( y − A ^ ( x ) ) \widehat{\xi}_M\to\widehat{\xi}:=\mathbb{E}^{\lambda}(y-\widehat{A}(x)) ξ M → ξ := E λ ( y − A ( x )) ( ξ ^ M ) M ∈ N (\widehat{\xi}_M)_{M\in\mathbb{N}} ( ξ M ) M ∈ N Theorem 3.1 , ξ ^ \widehat{\xi} ξ J ( ⋅ , G ) \mathcal{J}(\cdot,G) J ( ⋅ , G ) G ∈ S ( m ) G\in\mathcal{S}(m) G ∈ S ( m ) min ( ξ , G ) J α = min G J α ( ξ ^ , G ) \min_{(\xi,G)}\mathcal{J}_\alpha=\min_G\mathcal{J}_\alpha(\widehat{\xi},G) min ( ξ , G ) J α = min G J α ( ξ , G ) G ^ \widehat{G} G J α ( ξ ^ , ⋅ ) \mathcal{J}_\alpha(\widehat{\xi},\cdot) J α ( ξ , ⋅ ) ( ξ ^ , G ^ ) ∈ Θ 0 (\widehat{\xi},\widehat{G}) \in\Theta_0 ( ξ , G ) ∈ Θ 0 (30) with θ 0 = ( ξ ^ , G ^ ) \theta_0=(\widehat{\xi},\widehat{G}) θ 0 = ( ξ , G ) l θ l_{\theta} l θ G ^ M \widehat{G}_M G M
J ^ α M , ∞ ( ξ ^ M , G ^ M ) ≤ J ^ α M , ∞ ( ξ ^ M , G ^ ) , \widehat{\mathcal{J}}_{\alpha}^{M,\infty}(\widehat{\xi}_M,\widehat{G}_M) \leq\widehat{\mathcal{J}}_{\alpha}^{M,\infty}(\widehat{\xi}_M,\widehat{G}), J α M , ∞ ( ξ M , G M ) ≤ J α M , ∞ ( ξ M , G ) , and ξ ↦ J ^ α M , ∞ ( ξ , G ^ ) \xi\mapsto\widehat{\mathcal{J}}_{\alpha}^{M,\infty}(\xi,\widehat{G}) ξ ↦ J α M , ∞ ( ξ , G ) x j x_j x j y j y_j y j ξ ^ M → ξ ^ \widehat{\xi}_M \to\widehat{\xi} ξ M → ξ J ^ α M , ∞ ( ξ ^ M , G ^ ) − J ^ α M , ∞ ( ξ ^ , G ^ ) → 0 \widehat{\mathcal{J}}_{\alpha}^{M,\infty}(\widehat{\xi}_M,\widehat{G})-\widehat{\mathcal{J}}_{\alpha}^{M,\infty}(\widehat{\xi},\widehat{G}) \to0 J α M , ∞ ( ξ M , G ) − J α M , ∞ ( ξ , G ) → 0 G ^ M \widehat{G}_M G M Lemma 5.3 , using ( ξ ^ M , I m ) (\widehat{\xi}_M,I_m) ( ξ M , I m ) J ^ α M , ∞ ( ξ ^ M , I m ) − ℓ M \widehat{\mathcal{J}}_{\alpha}^{M,\infty}(\widehat{\xi}_M,I_m)-\ell_M J α M , ∞ ( ξ M , I m ) − ℓ M ξ ^ M \widehat{\xi}_M ξ M M M M G G G
More generally, if μ 0 \mu_0 μ 0 A ^ \widehat{A} A Z ^ \widehat{Z} Z J \mathcal{J} J Z ^ ( y ; ξ , G ) \widehat{Z}(y;\xi,G) Z ( y ; ξ , G ) μ 0 \mu_0 μ 0 A ^ \widehat{A} A Z ^ \widehat{Z} Z μ 0 \mu_0 μ 0 A ^ \widehat{A} A Z ^ ( y ; ξ , G ) \widehat{Z}(y;\xi,G) Z ( y ; ξ , G ) μ 0 \mu_0 μ 0 ( x j , y j ) (x_j,y_j) ( x j , y j ) y j y_j y j
Definition 5.6 (Double-loop estimator). Given a coercive function L : N → N L:\mathbb{N}\to\mathbb{N} L : N → N ( x j , y j ) j = 1 M ∼ λ (x_j,y_j)_{j=1}^{M}\sim\lambda ( x j , y j ) j = 1 M ∼ λ ( ( x i , j ) i = 1 L ( M ) ) j = 1 M ∼ μ 0 ((x_{i,j})_{i=1}^{L(M)})_{j=1}^{M}\sim\mu_0 (( x i , j ) i = 1 L ( M ) ) j = 1 M ∼ μ 0
J ^ α M , L ( ξ , G ) = α R ( G ) + 1 2 M ∑ j = 1 M ∥ A ^ ( x j ) − y j + ξ ∥ G 2 + 1 M ∑ j = 1 M log 1 L ( M ) ∑ i = 1 L ( M ) exp ( − 1 2 ∥ A ^ ( x i , j ) − y j + ξ ∥ G 2 ) \widehat{\mathcal{J}}_{\alpha}^{M,L}(\xi,G)=\alpha R(G)+\frac{1}{2M}\sum_{j=1}^{M}\left\|\widehat{A}(x_j)-y_j+\xi\right\|_{G}^{2}+\frac{1}{M}\sum_{j=1}^{M}\log\frac{1}{L(M)}\sum_{i=1}^{L(M)}\exp\left(-\frac{1}{2}\left\|\widehat{A}(x_{i,j})-y_j+\xi\right\|_{G}^{2}\right) J α M , L ( ξ , G ) = α R ( G ) + 2 M 1 j = 1 ∑ M A ( x j ) − y j + ξ G 2 + M 1 j = 1 ∑ M log L ( M ) 1 i = 1 ∑ L ( M ) exp ( − 2 1 A ( x i , j ) − y j + ξ G 2 ) We recall the discussion in Proposition 4.17 and Remark 4.18, by which the previously developed theory is applicable under the data-dependent conditions.
For double-loop estimators we will make the following assumption of differentiability of A A A A ^ \widehat{A} A
Assumption 3. Assume that X = R n X=\mathbb{R}^{n} X = R n L 3 > 0 L_3>0 L 3 > 0 A , A ^ ∈ C 1 ( R n ) A,\widehat{A}\in C^{1}(\mathbb{R}^{n}) A , A ∈ C 1 ( R n )
∥ ∇ A ^ ( x ) ∥ + ∥ ∇ A ( x ) ∥ ≤ L 3 ( 1 + ∥ x ∥ ) . \left\|\nabla\widehat{A}(x)\right\|+\left\|\nabla A(x)\right\|\leq L_3(1+\|x\|). ∇ A ( x ) + ∥ ∇ A ( x ) ∥ ≤ L 3 ( 1 + ∥ x ∥ ) . Consistency of the double loop estimator is established in [43 ] for very general function-valued likelihood approximations for functions f ∈ F f\in\mathcal{F} f ∈ F F \mathcal{F} F C 1 C^{1} C 1 R n × R m \mathbb{R}^{n}\times\mathbb{R}^{m} R n × R m
E λ f ( x , y ) + E P λ log E μ 0 exp [ − f ( x , y ) ] , \mathbb{E}^{\lambda}f(x,y)+\mathbb{E}^{\mathbb{P}\vphantom{\lambda}}\log\mathbb{E}^{\mu_0}\exp[-f(x,y)], E λ f ( x , y ) + E P λ log E μ 0 exp [ − f ( x , y )] , which coincides with (12) for f = f ξ , G f=f_{\xi,G} f = f ξ , G
f ξ , G ( x , y ) = 1 2 ∥ A ^ ( x ) − y + ξ ∥ G 2 , F = { f ξ , G , ( ξ , G ) ∈ K } ⊂ C 2 ( R n × R m ) , f_{\xi,G}(x,y)=\frac{1}{2}\left\|\widehat{A}(x)-y+\xi\right\|_{G}^{2},\qquad\mathcal{F}=\{f_{\xi,G},(\xi,G)\in K\}\subset C^{2}(\mathbb{R}^{n}\times\mathbb{R}^{m}), f ξ , G ( x , y ) = 2 1 A ( x ) − y + ξ G 2 , F = { f ξ , G , ( ξ , G ) ∈ K } ⊂ C 2 ( R n × R m ) , whenever ξ \xi ξ G G G K K K R m × S ( m ) \mathbb{R}^{m}\times\mathcal{S}(m) R m × S ( m ) A A A C 1 C^{1} C 1
A metric measuring the discrepancy between potentials after accounting for their y y y
d Φ ( ( ξ 1 , G 1 ) , ( ξ 2 , G 2 ) ) = ∥ f ξ 1 , G 1 − f ξ 2 , G 2 + log E μ 0 exp ( − f ξ 1 , G 1 ) − log E μ 0 exp ( − f ξ 2 , G 2 ) ∥ L 1 ( H × R m , λ ) . d_{\Phi}\left((\xi_1,G_1),(\xi_2,G_2)\right)=\left\|f_{\xi_1,G_1}-f_{\xi_2,G_2}+\log\mathbb{E}^{\mu_0}\exp(-f_{\xi_1,G_1})-\log\mathbb{E}^{\mu_0}\exp(-f_{\xi_2,G_2})\right\|_{L^{1}(H\times\mathbb{R}^{m},\lambda)}. d Φ ( ( ξ 1 , G 1 ) , ( ξ 2 , G 2 ) ) = ∥ f ξ 1 , G 1 − f ξ 2 , G 2 + log E μ 0 exp ( − f ξ 1 , G 1 ) − log E μ 0 exp ( − f ξ 2 , G 2 ) ∥ L 1 ( H × R m , λ ) . We reiterate that d Φ d_{\Phi} d Φ R m × S ( m ) \mathbb{R}^{m}\times\mathcal{S}(m) R m × S ( m ) ξ M \xi_M ξ M d Φ d_{\Phi} d Φ R m × S ( m ) \mathbb{R}^{m}\times\mathcal{S}(m) R m × S ( m ) v T A ^ v^{\mathsf{T}}\widehat{A} v T A μ 0 \mu_0 μ 0 v ∈ R m v\in\mathbb{R}^{m} v ∈ R m G G G (25) .
Lemma 5.7. Let Assumptions [1 ] hold and let K K K R m × S ( m ) \mathbb{R}^{m} \times\mathcal{S}(m) R m × S ( m ) ( ξ k , G k ) ⊂ K (\xi_k,G_k) \subset K ( ξ k , G k ) ⊂ K
( ξ k , G k ) → ( ξ , G ) ∈ K (\xi_k,G_k) \to(\xi,G) \in K ( ξ k , G k ) → ( ξ , G ) ∈ K with respect to the Euclidean norm on R m × S ( m ) \mathbb{R}^{m} \times\mathcal{S}(m) R m × S ( m ) d Φ d_{\Phi} d Φ [2 ] holds, then convergence with respect to d Φ d_{\Phi} d Φ
Proof. Assume ( ξ k , G k ) → ( ξ , G ) ∈ K (\xi_k,G_k) \to(\xi,G) \in K ( ξ k , G k ) → ( ξ , G ) ∈ K [43 Lemma 2.6] it holds for some p ≥ p 0 > 1 p \ge p_0 > 1 p ≥ p 0 > 1 C C C
d Φ ( ( ξ n , G n ) , ( ξ , G ) ) ≤ C ∥ ∥ A ^ ( x ) − y + ξ n ∥ G n 2 − ∥ A ^ ( x ) − y + ξ ∥ G 2 ∥ L p ( λ ) . d_{\Phi}\bigl((\xi_n,G_n),(\xi,G)\bigr) \le C \left\lVert\left\lVert\widehat{A}(x)-y+\xi_n\right\rVert_{G_n}^{2}-\left\lVert\widehat{A}(x)-y+\xi\right\rVert_{G}^{2}\right\rVert_{L_p(\lambda)}. d Φ ( ( ξ n , G n ) , ( ξ , G ) ) ≤ C A ( x ) − y + ξ n G n 2 − A ( x ) − y + ξ G 2 L p ( λ ) . The dominated convergence theorem applies by the arguments used in Theorem 2.5 and hence convergence with respect to d Φ d_{\Phi} d Φ
Assume now that ( ξ k , G k ) → ( ξ , G ) ∈ K (\xi_k,G_k) \to(\xi,G) \in K ( ξ k , G k ) → ( ξ , G ) ∈ K d Φ d_{\Phi} d Φ K K K d Φ d_{\Phi} d Φ ( ξ n k , G n k ) → ( ξ † , G † ) ∈ K (\xi_{n_k},G_{n_k}) \to(\xi^\dagger,G^\dagger) \in K ( ξ n k , G n k ) → ( ξ † , G † ) ∈ K
d Φ ( ( ξ † , G † ) , ( ξ , G ) ) ≤ d Φ ( ( ξ † , G † ) , ( ξ n k , G n k ) ) + d Φ ( ( ξ n k , G n k ) , ( ξ , G ) ) → 0 , d_{\Phi}\bigl((\xi^\dagger,G^\dagger),(\xi,G)\bigr) \le d_{\Phi}\bigl((\xi^\dagger,G^\dagger),(\xi_{n_k},G_{n_k})\bigr)+d_{\Phi}\bigl((\xi_{n_k},G_{n_k}),(\xi,G)\bigr) \to0, d Φ ( ( ξ † , G † ) , ( ξ , G ) ) ≤ d Φ ( ( ξ † , G † ) , ( ξ n k , G n k ) ) + d Φ ( ( ξ n k , G n k ) , ( ξ , G ) ) → 0 , Hence d Φ ( ( ξ † , G † ) , ( ξ , G ) ) = 0 d_{\Phi}\bigl((\xi^\dagger,G^\dagger),(\xi,G)\bigr)=0 d Φ ( ( ξ † , G † ) , ( ξ , G ) ) = 0 [43 Lemma 2.4] this implies that the term
∥ A ^ ( x ) − y + ξ ∥ G 2 − ∥ A ^ ( x ) − y + ξ † ∥ G † 2 \left\lVert\widehat{A}(x)-y+\xi\right\rVert_{G}^{2}-\left\lVert\widehat{A}(x)-y+\xi^\dagger\right\rVert_{G^\dagger}^{2} A ( x ) − y + ξ G 2 − A ( x ) − y + ξ † G † 2 depends only on y y y λ \lambda λ
A ^ ( x ) T ( G − G † ) A ^ ( x ) + 2 A ^ ( x ) T ( G ξ − G † ξ † + ( G † − G ) y ) + y T ( G − G † ) y (33) \widehat{A}(x)^{\mathsf{T}}(G-G^\dagger)\widehat{A}(x)+2\widehat{A}(x)^{\mathsf{T}}(G\xi-G^\dagger\xi^\dagger+(G^\dagger-G)y)+y^{\mathsf{T}}(G-G^\dagger)y \tag*{(33)} A ( x ) T ( G − G † ) A ( x ) + 2 A ( x ) T ( G ξ − G † ξ † + ( G † − G ) y ) + y T ( G − G † ) y (33) depends only on y y y λ \lambda λ p p p R m \mathbb{R}^{m} R m (33) at y y y y + z y+z y + z z ∈ R m z \in\mathbb{R}^{m} z ∈ R m A ^ ( x ) T ( G † − G ) z \widehat{A}(x)^{\mathsf{T}}(G^\dagger-G)z A ( x ) T ( G † − G ) z μ 0 \mu_0 μ 0 x x x z z z [2 ] implies G = G † G=G^\dagger G = G † (33) gives that A ^ ( x ) T G ( ξ − ξ † ) \widehat{A}(x)^{\mathsf{T}}G(\xi-\xi^\dagger) A ( x ) T G ( ξ − ξ † ) μ 0 \mu_0 μ 0 x x x [2 ] together with invertibility of G G G ξ = ξ † \xi=\xi^\dagger ξ = ξ † ( ξ n , G n ) ⊂ K (\xi_n,G_n) \subset K ( ξ n , G n ) ⊂ K ( ξ , G ) (\xi,G) ( ξ , G ) K K K ( ξ † , G † ) = ( ξ , G ) (\xi^\dagger,G^\dagger)=(\xi,G) ( ξ † , G † ) = ( ξ , G ) ( ξ n , G n ) → ( ξ , G ) (\xi_n,G_n) \to(\xi,G) ( ξ n , G n ) → ( ξ , G )
Following [43 ] , we will use the following theorem to show consistency of the double-loop estimators.
Theorem 5.8 (Consistency, [48 Theorem 5.7] ). Let ( D , d ) (D,d) ( D , d ) ϕ : D → R \phi:D \to\mathbb{R} ϕ : D → R D 0 : = argmin f ∈ D ϕ ( f ) D_0 := \operatorname*{argmin}_{f\in D}\phi(f) D 0 := argmin f ∈ D ϕ ( f ) ε > 0 \varepsilon> 0 ε > 0
inf f ∈ D : d ( f , D 0 ) > ε ϕ ( f ) > inf f ∈ D ϕ ( f ) (34) \inf_{f\in D:\,d(f,D_0)>\varepsilon}\phi(f)>\inf_{f\in D}\phi(f) \tag*{(34)} f ∈ D : d ( f , D 0 ) > ε inf ϕ ( f ) > f ∈ D inf ϕ ( f ) (34) and
sup f ∈ D ∣ ϕ N ( f ) − ϕ ( f ) ∣ → 0 (35) \sup_{f\in D}\left|\phi_N(f)-\phi(f)\right| \to0 \tag*{(35)} f ∈ D sup ∣ ϕ N ( f ) − ϕ ( f ) ∣ → 0 (35) in probability. Then any f ^ N \widehat{f}_N f N ϕ N ( f ^ N ) ≤ inf f ∈ D ϕ N ( f ) + o P ( 1 ) \phi_N(\widehat{f}_N) \le\inf_{f\in D}\phi_N(f)+o_P(1) ϕ N ( f N ) ≤ inf f ∈ D ϕ N ( f ) + o P ( 1 ) d ( f ^ N , D 0 ) → 0 d(\widehat{f}_N,D_0) \to0 d ( f N , D 0 ) → 0
Proof. For f 0 ∈ D 0 f_0 \in D_0 f 0 ∈ D 0 (35) gives ϕ ( f ^ N ) ≤ ϕ N ( f ^ N ) + o P ( 1 ) ≤ ϕ N ( f 0 ) + o P ( 1 ) ≤ ϕ ( f 0 ) + o P ( 1 ) \phi(\widehat{f}_N) \le\phi_N(\widehat{f}_N)+o_P(1) \le\phi_N(f_0)+o_P(1) \le\phi(f_0)+o_P(1) ϕ ( f N ) ≤ ϕ N ( f N ) + o P ( 1 ) ≤ ϕ N ( f 0 ) + o P ( 1 ) ≤ ϕ ( f 0 ) + o P ( 1 ) ϕ ( f ^ N ) − inf D ϕ = o P ( 1 ) \phi(\widehat{f}_N)-\inf_D\phi=o_P(1) ϕ ( f N ) − inf D ϕ = o P ( 1 ) (34) yields the claim.
Theorem 5.9. Let Assumptions [1 ] and [3 ] hold, let K K K R m × S ( m ) \mathbb{R}^{m} \times\mathcal{S}(m) R m × S ( m ) Θ 0 : = argmin ( ξ , G ) ∈ K J α ( ξ , G ) \Theta_0 := \operatorname*{argmin}_{(\xi,G)\in K}\mathcal{J}_{\alpha}(\xi,G) Θ 0 := argmin ( ξ , G ) ∈ K J α ( ξ , G ) J α \mathcal{J}_{\alpha} J α
( ξ ^ M , G ^ M ) ∈ argmin ( ξ , G ) ∈ K J ^ α M , L ( ξ , G ) , (\widehat{\xi}_M,\widehat{G}_M) \in\operatorname*{argmin}_{(\xi,G)\in K}\widehat{\mathcal{J}}_{\alpha}^{M,L}(\xi,G), ( ξ M , G M ) ∈ ( ξ , G ) ∈ K argmin J α M , L ( ξ , G ) , it holds for every ε > 0 \varepsilon> 0 ε > 0
P ( d Φ ( ( ξ ^ M , G ^ M ) , Θ 0 ) > ε ) → 0 as M → ∞ . \mathbb{P}\left(d_{\Phi}\bigl((\widehat{\xi}_M,\widehat{G}_M),\Theta_0\bigr)>\varepsilon\right)\to0 \quad\text{as } M\to\infty. P ( d Φ ( ( ξ M , G M ) , Θ 0 ) > ε ) → 0 as M → ∞. If Assumption [2 ] holds, the same holds with d Φ d_{\Phi} d Φ
Proof of Theorem 5.9. By [43 Theorem 3.6] applied with
F 1 : = conv ( { 1 2 ∥ A ^ ( x ) − y + ξ ∥ G 2 : ( ξ , G ) ∈ K } ∪ { 1 2 δ 2 ∥ A ( x ) − y ∥ 2 } ) , \mathcal{F}_1 := \operatorname{conv}\left(\left\{\frac{1}{2}\left\|\widehat{A}(x)-y+\xi\right\|_{G}^{2}:(\xi,G)\in K\right\}\cup\left\{\frac{1}{2\delta^{2}}\|A(x)-y\|^{2}\right\}\right), F 1 := conv ( { 2 1 A ( x ) − y + ξ G 2 : ( ξ , G ) ∈ K } ∪ { 2 δ 2 1 ∥ A ( x ) − y ∥ 2 } ) , and since the regularisation term cancels in the difference, we get
sup ( ξ , G ) ∈ K ∣ J ^ α M , L ( ξ , G ) − J α ( ξ , G ) ∣ ≤ sup f ∈ F 1 ∣ ϕ N ( f ) − ϕ ( f ) ∣ → 0 \sup_{(\xi,G)\in K}\left|\widehat{J}_{\alpha}^{M,L}(\xi,G)-J_{\alpha}(\xi,G)\right|\leq\sup_{f\in\mathcal{F}_1}\left|\phi_N(f)-\phi(f)\right|\to0 ( ξ , G ) ∈ K sup J α M , L ( ξ , G ) − J α ( ξ , G ) ≤ f ∈ F 1 sup ∣ ϕ N ( f ) − ϕ ( f ) ∣ → 0 in probability, which is (35) for D = K D=K D = K ϕ = J α \phi=J_{\alpha} ϕ = J α ϕ N = J ^ α M , L \phi_N=\widehat{J}_{\alpha}^{M,L} ϕ N = J α M , L (34) with d = d Φ d=d_{\Phi} d = d Φ D 0 = Θ 0 D_0=\Theta_0 D 0 = Θ 0 ε > 0 \varepsilon>0 ε > 0
inf ( ξ , G ) ∈ K , d Φ ( ( ξ , G ) , Θ 0 ) > ε J α ( ξ , G ) > c ∗ : = min ( ξ , G ) ∈ K J α ( ξ , G ) . \inf_{\substack{(\xi,G)\in K,\,d_{\Phi}((\xi,G),\Theta_0)>\varepsilon}}J_{\alpha}(\xi,G)>c^*:=\min_{(\xi,G)\in K}J_{\alpha}(\xi,G). ( ξ , G ) ∈ K , d Φ (( ξ , G ) , Θ 0 ) > ε inf J α ( ξ , G ) > c ∗ := ( ξ , G ) ∈ K min J α ( ξ , G ) . Suppose, for contradiction, that for some ε > 0 \varepsilon>0 ε > 0 ( ξ n , G n ) ⊂ K (\xi_n,G_n)\subset K ( ξ n , G n ) ⊂ K
d Φ ( ( ξ n , G n ) , Θ 0 ) ≥ ε for all n ∈ N , (36) d_{\Phi}((\xi_n,G_n),\Theta_0)\geq\varepsilon\qquad\text{for all }n\in\mathbb{N}, \tag*{(36)} d Φ (( ξ n , G n ) , Θ 0 ) ≥ ε for all n ∈ N , (36) and J α ( ξ n , G n ) → c ∗ J_{\alpha}(\xi_n,G_n)\to c^* J α ( ξ n , G n ) → c ∗ K K K ( ξ n , G n ) → ( ξ ∗ , G ∗ ) ∈ K (\xi_n,G_n)\to(\xi^*,G^*)\in K ( ξ n , G n ) → ( ξ ∗ , G ∗ ) ∈ K J α J_{\alpha} J α J α ( ξ ∗ , G ∗ ) = c ∗ J_{\alpha}(\xi^*,G^*)=c^* J α ( ξ ∗ , G ∗ ) = c ∗ ( ξ ∗ , G ∗ ) ∈ Θ 0 (\xi^*,G^*)\in\Theta_0 ( ξ ∗ , G ∗ ) ∈ Θ 0 d Φ ( ( ξ n , G n ) , ( ξ ∗ , G ∗ ) ) → 0 d_{\Phi}((\xi_n,G_n),(\xi^*,G^*))\to0 d Φ (( ξ n , G n ) , ( ξ ∗ , G ∗ )) → 0 d Φ ( ( ξ n , G n ) , Θ 0 ) → 0 d_{\Phi}((\xi_n,G_n),\Theta_0)\to0 d Φ (( ξ n , G n ) , Θ 0 ) → 0 (36) . Theorem 5.8 now yields the first claim.
Under Assumption 2, for every ε > 0 \varepsilon>0 ε > 0 η > 0 \eta>0 η > 0 d Φ ( θ , Θ 0 ) ≤ η d_{\Phi}(\theta,\Theta_0)\leq\eta d Φ ( θ , Θ 0 ) ≤ η dist ( θ , Θ 0 ) ≤ ε \operatorname{dist}(\theta,\Theta_0)\leq\varepsilon dist ( θ , Θ 0 ) ≤ ε θ ∈ K \theta\in K θ ∈ K θ n ∈ K \theta_n\in K θ n ∈ K ϑ n ∈ Θ 0 \vartheta_n\in\Theta_0 ϑ n ∈ Θ 0 d Φ ( θ n , ϑ n ) → 0 d_{\Phi}(\theta_n,\vartheta_n)\to0 d Φ ( θ n , ϑ n ) → 0 dist ( θ n , Θ 0 ) > ε \operatorname{dist}(\theta_n,\Theta_0)>\varepsilon dist ( θ n , Θ 0 ) > ε θ n → θ ∗ \theta_n\to\theta^* θ n → θ ∗ ϑ n → ϑ ∗ ∈ Θ 0 \vartheta_n\to\vartheta^*\in\Theta_0 ϑ n → ϑ ∗ ∈ Θ 0 d Φ ( θ ∗ , ϑ ∗ ) = 0 d_{\Phi}(\theta^*,\vartheta^*)=0 d Φ ( θ ∗ , ϑ ∗ ) = 0 θ ∗ = ϑ ∗ \theta^*=\vartheta^* θ ∗ = ϑ ∗ dist ( θ ∗ , Θ 0 ) ≥ ε \operatorname{dist}(\theta^*,\Theta_0)\geq\varepsilon dist ( θ ∗ , Θ 0 ) ≥ ε
Remark 5.10. The compactness of K K K Z ^ \widehat{Z} Z λ ~ M \widetilde{\lambda}_M λ M G G G ξ \xi ξ (9) . However, the inner samples ( x i , j ) (x_{i,j}) ( x i , j ) Z ^ ( y j ; ξ , G ) \widehat{Z}(y_j;\xi,G) Z ( y j ; ξ , G ) μ 0 \mu_0 μ 0 ( x j , y j ) (x_j,y_j) ( x j , y j ) λ \lambda λ (9) does not hold for the double-loop objective, and the confinement argument for G G G ξ \xi ξ
Numerical experiments
Computed tomography
We model the forward map A A A 50 × 50 50\times50 50 × 50 n = 2500 n=2500 n = 2500 0 0 0 170 170 170 m = 47 ⋅ 50 = 2350 m=47\cdot50=2350 m = 47 ⋅ 50 = 2350 δ = 0.005 \delta=0.005 δ = 0.005 μ 0 = N ( 0 , C 0 ) \mu_0=\mathcal{N}(0,C_0) μ 0 = N ( 0 , C 0 ) C 0 C_0 C 0
k ( x , y ) = 1 σ 2 exp ( − ∥ x − y ∥ 2 2 2 l 2 ) , k(x,y)=\frac{1}{\sigma^2}\exp\left(-\frac{\|x-y\|_2^2}{2l^2}\right), k ( x , y ) = σ 2 1 exp ( − 2 l 2 ∥ x − y ∥ 2 2 ) , with length-scale l = 0.1 ( 50 ) − 1 / 2 l=0.1(50)^{-1/2} l = 0.1 ( 50 ) − 1/2 σ 2 = 50 \sigma^2=50 σ 2 = 50 Figure 1 .
Figure 1. Ground truth x 0 x_0 x 0 μ 0 \mu_0 μ 0
The surrogate A ^ \widehat{A} A 5 5 5 177 177 177 0 0 0 170 170 170 A ^ \widehat{A} A A A A A ^ \widehat{A} A
We compare three choices of the Gaussian likelihood: the surrogate likelihood without any correction, i.e. ξ = 0 \xi=0 ξ = 0 G = δ − 2 I m G=\delta^{-2}I_m G = δ − 2 I m (4) (EEA); and the optimal approximation error approach (5) (OAE). Since the setting is Gaussian-linear, OAE uses closed-form objective function given in § 3 and α = 0 \alpha=0 α = 0 G G G G ∗ G_\ast G ∗ (24) . The expected KL divergence is evaluated in closed form..
Figure 2 shows posterior samples for the three methods and for the true posterior, Figure 3 shows the posterior mean, its error with respect to the ground truth of Figure 1 , the pointwise posterior standard deviation and the pixels covered by a 2 σ 2\sigma 2 σ Table 1 collects summary metrics.
Figure 2. Posterior samples for the CT problem. Rows correspond, from top to bottom, to None, EEA, OAE and the true posterior.
Table 1. Error metrics for the CT-imaging problem. All norms are averaged over the image pixels. In the quantities in the two last rows, x 0 x_0 x 0 Figure 1 and y 0 y_0 y 0
Figure 3. For all methods, we report the posterior mean, difference posterior mean to the ground truth, diagonal of the posterior covariance, and the pixels contained in a confidence interval based on the posterior mean and variance. Top row corresponds to None, second row to EEA, third row to OAE and final row to the true posterior
Table 1 shows that OAE reduces the expected posterior KL divergence from 342 (EEA) to 239, and the error of the posterior mean by a factor of about 2.5, both in expectation and for the single draw x 0 x_0 x 0 ∥ C − C ^ ∥ F \lVert C-\widehat{C}\rVert_F ∥ C − C ∥ F δ − 2 I m \delta^{-2}I_m δ − 2 I m
Darcy Flow
We seek to recover the permeability σ ∈ C 1 ( Ω ) \sigma\in C^{1}(\Omega) σ ∈ C 1 ( Ω ) Ω = [ 0 , 1 ] 2 \Omega= [0,1]^{2} Ω = [ 0 , 1 ] 2 u ∈ C 2 ( Ω ) u \in C^{2}(\Omega) u ∈ C 2 ( Ω ) g ∈ C ( Ω ) g \in C(\Omega) g ∈ C ( Ω ) F : σ ↦ u F:\sigma\mapsto u F : σ ↦ u
− ∇ ⋅ ( σ ∇ u ) = g z ∈ Ω u = 0 z ∈ ∂ Ω . \begin{aligned}
-\nabla\cdot(\sigma\nabla u) &= g && z \in\Omega\\
u &= 0 && z \in\partial\Omega.
\end{aligned} − ∇ ⋅ ( σ ∇ u ) u = g = 0 z ∈ Ω z ∈ ∂ Ω. The surrogate A ^ \widehat{A} A F F F M = 5000 M = 5000 M = 5000 ( x j , y j ) ∼ λ (x_j,y_j) \sim\lambda ( x j , y j ) ∼ λ J ^ α M , L \widehat{J}_{\alpha}^{M,L} J α M , L α = 0.001 \alpha= 0.001 α = 0.001 μ 0 \mu_0 μ 0 Z ^ ( y j ; ξ , G ) \widehat{Z}(y_j;\xi,G) Z ( y j ; ξ , G ) L = 100 L = 100 L = 100
The surrogate and true posterior are sampled by the same random-walk Metropolis–Hastings algorithm with 6000 samples after a burn-in of 20,000 and a thinning with the factor t t t m ( y 0 ) m(y_0) m ( y 0 ) C ( y 0 ) C(y_0) C ( y 0 ) m ^ ( y 0 ) \widehat{m}(y_0) m ( y 0 ) C ^ ( y 0 ) \widehat{C}(y_0) C ( y 0 ) Figure 5 are computed from these chains.
Figure 5. Posterior samples for all methods. Top row corresponds to None, second row to EEA third row to OAE, and final row to the true posterior
Table 2 collects the error metrics, Figure 5 shows posterior samples and Figure 6 compares the posterior means, their errors, the pointwise posterior standard deviation and the credible-band coverage with the ground truth.
Table 2: Error metrics for the 2D Darcy-flow problem. All norms are averaged over the image pixels. In the quantities in the two last rows, x 0 x_0 x 0 Figure 4 and y 0 y_0 y 0
Table 2.
Figure 4. Ground truth sample from the prior
Figure 6. For all methods, we report the posterior mean, difference posterior mean to the ground truth, diagonal of the posterior covariance, and the pixels contained in a confidence interval based on the posterior mean and variance. Rows correspond to (top to bottom) None, EEA, OAE, true posterior
The picture is consistent with the CT experiment. OAE halves the error of the posterior mean relative to EEA (0.0015 versus 0.0027) and reduces the expected KL divergence, though here the gain over EEA is modest (454 versus 475), while both improve by an order of magnitude on the uncorrected likelihood.
Coverage of the 2 σ 2\sigma 2 σ
Conclusions
We introduced the optimal approximation error (OAE) framework for calibrating the noise parameters of the approximation error model by directly minimising the expected KL divergence between the true and surrogate posteriors. We showed that OAE subsumes the classical enhanced error approach as the variant that instead targets the expected divergence between likelihoods. We established smoothness, bi-convexity, and, after regularisation, coercivity and the Kurdyka–Łojasiewicz property, yielding a proximal alternating minimisation scheme with guaranteed convergence to a critical point. In the Gaussian linear case we obtained the objective function and the optimal mean shift in closed form. More generally, we introduced an empirical estimator and proved its consistency as the training sample size grows. The framework applies in a discretisation-invariant manner to unknowns in infinite-dimensional Banach spaces, and our experiments on a linear CT problem and a nonlinear Darcy-flow problem show that OAE improves the posterior mean and the expected KL divergence relative to the uncorrected and EEA-corrected likelihoods, with credible-band coverage comparable to that of EEA and of the true posterior.
Several directions remain open. The present framework parametrises the noise covariance by a full precision matrix G ∈ S ( m ) G \in\mathcal{S}(m) G ∈ S ( m ) m m m ( ξ , G ) (\xi,G) ( ξ , G )
Acknowledgments
Part of this research was carried out while F.S. was at TU Wien, Austria. Funding was received from the European Research Council (ERC) under the Horizon 2020 research and innovation program of the European Union (Grant agreement No. 101125225). This work was supported by the Research Council of Finland (353094, 348504 and 359183). Support by the Austrian Science Fund (FWF) grant I6667-N is acknowledged
AI disclosure
During the preparation of this manuscript, the authors used Claude Fable 5.1 to support proofreading and editing of the text and to improve its clarity. The authors reviewed, validated and critically assessed all AI-assisted outputs and take full responsibility for the content of the manuscript.
