Introduction
Diffusion models ([50 , 54 , 55 , 51 , 21 ] (Ho et al. 2020)), as the backbone of modern image and video generation engines ([44 ] ), are one of the driving forces behind the recent remarkable advances of generative artificial intelligence. Since their debut, diffusion models have been adopted across many scientific and engineering domains, such as biological structure prediction ([2 , 56 ] ), robotics ([19 , 33 ] ), materials ([23 ] ), and many others.
A central idea behind score-based diffusion models is to perturb the data with progressively stronger Gaussian noise and learn the associated score functions, namely the gradients of the log densities at different noise levels, for instance, via score matching ([30 ] ). These scores then guide an iterative denoising process that transforms noise into samples resembling those from the data distribution ([54 , 55 ] ). Complementing this empirical success, a flurry of recent theory has established polynomial bounds on the number of sampling steps required when the score functions are accurate, without imposing any smoothness or log-concavity assumptions on the data distribution (see, e.g., [17 , 5 , 39 ] (Li et al. 2024)).
Diffusion models as priors
The expressive power of diffusion models makes them attractive priors for ill-posed inverse problems, where the goal is to recover high-dimensional signals from a small number of noisy measurements. Of particular interest is the plug-and-play framework ([10 ] ), in which the same pretrained diffusion model is reused to solve different instances of inverse problems. This flexibility is especially valuable in domains where the same class of signals is observed through different sensing modalities and under varying noise conditions, allowing the substantial cost of training a diffusion model to be amortized across reconstruction tasks.
Posterior sampling provides a natural approach to use a diffusion prior for inverse problems, where one can generate multiple samples from the resulting posterior as candidate reconstructions. However, turning an unconditional diffusion model into a posterior sampler introduces additional difficulties. Running the reverse diffusion directly for posterior sampling requires the scores of the measurement-conditioned distributions at intermediate noise levels, which are not supplied by the pretrained diffusion model. Many empirical methods are thus developed to circumvent this limitation, and have achieved qualitatively impressive performance (see, e.g., [32 , 1 , 52 , 53 ] ).
Despite these empirical advances, the theoretical understanding of diffusion-based inverse solvers remains severely limited. Existing algorithms often come only with asymptotic consistency for sampling from the true posterior [26 , 15 , 35 ] (Wu et al. 2023; Xu and Chi 2024). When it comes to non-asymptotic guarantees, they can require exponentially many sampling steps in the worst case (Xu and Chi, 2024), rely on strong assumptions on the algorithm’s initialization [41 , 62 ] , or constrain the data distribution to strict log-concavity [62 ] or small susceptibility [9 ] to allow more efficient sampling guarantees. Indeed, under standard computational hardness assumptions, Gupta et al. [28 ] ; ); Bruna and Han [9 ] showed that posterior sampling can be intractable even for linear inverse problems whose unconditional diffusion models admit efficient sampling. These computational barriers rule out a general polynomial-time guarantee for diffusion-based inverse solvers, and motivate the search for structured inverse problems that admit efficient inference and are more representative of where such solvers succeed in practice.
Our contribution
To understand when diffusion priors admit efficient inference, we adopt sparse recovery from noisy linear measurements, a classical problem in compressed sensing [24 , 14 , 12 ] , as our testbed. We assume access to a pretrained diffusion model that captures a sparsity-promoting Bernoulli–Gaussian prior, under which each coordinate is either zero or drawn from a Gaussian distribution. This setting retains the key challenge associated with a non-log-concave data prior, while remains tractable enough to yield practically meaningful insight into the statistical and computational efficiency of diffusion-based inverse solvers beyond the worst case.
Concretely, we analyze a simple likelihood-guided annealed Langevin algorithm [32 ] that combines the scores of the noise-perturbed prior with progressively stronger likelihood guidance as the diffusion noise level decreases. We show the algorithm succeeds within a polynomial number of iterations at near-optimal sample complexities for a broad family of sensing matrix designs, including sub-Gaussian random matrices, randomly subsampled orthonormal bases, and deterministic incoherent tight frames.
For fixed sparse signals, we establish that under suitable noise and signal strength conditions, with high probability, the reconstructed signal enjoys near-optimal ℓ 2 \ell_2 ℓ 2 recovery guarantees on the order of
up to logarithmic factors, where k k k is the sparsity level, and σ \sigma σ is the noise level.
α + E s c \alpha+ \sqrt{\mathcal{E}_{\mathrm{sc}}} α + E sc
after a final denoising step, where α > 0 \alpha> 0 α > 0 is the target accuracy, and E s c \mathcal{E}_{\mathrm{sc}} E sc is the score estimation error.
Both guarantees explicitly account for the score estimation error, where they degenerate gracefully as it grows. To the best of our knowledge, our result provides the first polynomial-time posterior sampling algorithm at the near-optimal sample complexity linear in the expected sparsity level using random sensing ensembles, answering an open problem raised in Kumar et al. [38 ] . Together, these results identify a concrete and benign setting in which a simple diffusion-based solver provably achieves accurate recovery and posterior sampling under a non-log-concave prior, with guarantees that are efficient both statistically and computationally.
Related works
Several recent works have sought non-asymptotic convergence guarantees for diffusion-based posterior sampling in general inverse problems. The convergence rate in Xu and Chi (2024) depends critically on the spectral gap of a Markov chain tied to the dynamics of their underlying algorithm, which can become exponentially small in the problem dimension in the worst case. Leong and Traonmilin [41 ] developed a deterministic recovery theory leveraging the connection between diffusion and denoising operators Kadkhodaie and Simoncelli [36 ] . However, their theory does not provide a convergence guarantee with polynomial complexity, since it hinges on an unspecified initial burn-in phase. A notable recent development is Xun et al. [62 ] , which developed a provably efficient posterior sampling algorithm based on annealed Langevin dynamics with polynomial iteration complexities, assuming a log-concave prior. They subsequently went beyond log-concavity by assuming access to direct measurements of the signal of interest at sufficiently low noise levels, which are generally unavailable in practice. Our work removes these restrictions in the setting of sparse recovery with few measurements. Last but not least, Xu [60 ] showed that diffusion-based posterior sampling is efficient for sparse signal denoising, corresponding to the special case of compressed sensing with an identity sensing matrix. The present paper builds on Xu [60 ] , but significantly goes beyond it by handling general sensing matrices.
We also note that Bayesian approaches have long been applied to high-dimensional sparse linear regression, e.g., George and McCulloch [27 ] ; Ishwaran and Rao [31 ] ; Yang et al. [63 ] . Due to the challenge of sampling from multi-modal distributions, recent efforts have focused on developing efficient sampling algorithms tailored to sparsity-inducing distributions, such as the spike-and-slab prior [43 , 37 , 34 , 38 ] . Although the goal of our work is not to compete with existing sparse recovery algorithms, it offers a complementary–and competitive–algorithmic perspective with efficient sampling guarantees, and highlights the benefit of annealing provided naturally by diffusion priors.
Notation. Before proceeding, we introduce several useful notation used throughout the paper. Define the sub-Gaussian norm of a real random variable R R R as
∥ R ∥ ψ 2 : = inf { v > 0 : E e R 2 / v 2 ≤ 2 } . \lVert R\rVert_{\psi_2} := \inf\{v > 0 : \mathbb{E}e^{R^2/v^2} \le2\}. ∥ R ∥ ψ 2 := inf { v > 0 : E e R 2 / v 2 ≤ 2 } .
Throughout the paper, we use c c c and C C C to denote positive universal constants whose values may change between occurrences, where c c c is typically taken to be small enough and C C C to be large enough. For a matrix B B B , we let B i j B_{ij} B ij be its ( i , j ) (i,j) ( i , j ) -th entry, B j B_j B j be its i i i -th column, and for index sets J J J and K K K , let B J K B_{JK} B J K be the submatrix with rows in J J J and columns in K K K , and B J B_J B J be the submatrix with columns in J J J . The diagonal part of a matrix B B B is denoted by diag B \operatorname{diag} B diag B , while diag ( c i ) \operatorname{diag}(c_i) diag ( c i ) is the diagonal matrix with entries c i c_i c i . We write [ n ] : = { 1 , … , n } [n] := \{1,\ldots,n\} [ n ] := { 1 , … , n } and supp x : = { i : x i ≠ 0 } \operatorname{supp} x := \{i : x_i \ne0\} supp x := { i : x i = 0 } . Let ∥ x ∥ \lVert x\rVert ∥ x ∥ be the ℓ 2 \ell_2 ℓ 2 norm of a vector x x x , and ∥ A ∥ \lVert A\rVert ∥ A ∥ be the spectral norm of a matrix A A A . The law of a random variable X X X is denoted by L ( X ) \mathcal{L}(X) L ( X ) . The product μ ⊗ ν \mu\otimes\nu μ ⊗ ν is the joint law of independent draws with laws μ , ν \mu,\nu μ , ν ; the law of their sum has convolution law μ ∗ ν \mu* \nu μ ∗ ν .
Problem formulation
We aim at solving an underdetermined linear inverse problem of the form
y = A x ⋆ + σ ζ , ζ ∼ N ( 0 , I m ) , (1) y = Ax_\star+ \sigma\zeta,\qquad\zeta\sim\mathcal{N}(0,I_m), \tag*{(1)} y = A x ⋆ + σ ζ , ζ ∼ N ( 0 , I m ) , (1)
where A ∈ R m × n A \in\mathbb{R}^{m\times n} A ∈ R m × n is the sensing or measurement matrix, x ⋆ x_\star x ⋆ is the signal of interest, and σ ζ ∈ R m \sigma\zeta\in\mathbb{R}^m σ ζ ∈ R m is the additive Gaussian noise independent of the signal, with a known noise level σ > 0 \sigma> 0 σ > 0 .
Sparse signal prior. Our canonical model of the signal prior is the Bernoulli–Gaussian model , which provides a simple and analytically convenient description of sparsity.
Definition 1 (Bernoulli–Gaussian model). Let s , n ∈ N s,n \in\mathbb{N} s , n ∈ N , with s < n s < n s < n . A random vector
X = ( X 1 , … , X n ) ∈ R n X = (X_1,\ldots,X_n) \in\mathbb{R}^n X = ( X 1 , … , X n ) ∈ R n is said to follow the Bernoulli–Gaussian model with sparsity level s s s and dimension n n n , denoted as B G ( s , n ) \mathrm{BG}(s,n) BG ( s , n ) , if its coordinates are independent and identically distributed according to
X i = B i Z i ∼ ( 1 − s n ) δ 0 + s n N ( 0 , 1 ) , i = 1 , … , n , X_i = B_i Z_i \sim\left(1-\frac{s}{n}\right)\delta_0 + \frac{s}{n}\mathcal{N}(0,1),\qquad i=1,\ldots,n, X i = B i Z i ∼ ( 1 − n s ) δ 0 + n s N ( 0 , 1 ) , i = 1 , … , n , where δ 0 \delta_0 δ 0 is a point mass at zero, B i ∼ Bernoulli ( s / n ) B_i \sim\operatorname{Bernoulli}(s/n) B i ∼ Bernoulli ( s / n ) and Z i ∼ N ( 0 , 1 ) Z_i \sim\mathcal{N}(0,1) Z i ∼ N ( 0 , 1 ) are all mutually independent.
Each coordinate of X X X is thus zero with probability 1 − s / n 1-s/n 1 − s / n and standard Gaussian otherwise. The expected support size of X X X is E [ ∣ supp ( X ) ∣ ] = s \mathbb{E}[\lvert\operatorname{supp}(X)\rvert] = s E [∣ supp ( X )∣] = s , and therefore, we refer to s s s as the sparsity level.
Posterior sampling with diffusion models We consider a Bayesian approach to signal recovery, even when the underlying ground truth signal x ⋆ x_\star x ⋆ may not be generated from a probabilistic model. We postulate the ground truth signal x ⋆ x_\star x ⋆ follows a prior distribution p ⋆ p_\star p ⋆ , for example, BG ( s , n ) \operatorname{BG}(s,n) BG ( s , n ) for some postulated sparsity level s s s . We attempt to solve the inverse problem in (1) by sampling from the posterior distribution of x ⋆ x_\star x ⋆ given y y y , using Bayes’ formula:
p ⋆ ( x ∣ y ) ∝ p ⋆ ( x ) p ( y ∣ x ⋆ = x ) = p ⋆ ( x ) exp ( L ( x ; y ) ) , (2) p_\star(x\mid y) \propto p_\star(x)\,p(y\mid x_\star=x)=p_\star(x)\exp\bigl(\mathcal{L}(x;y)\bigr), \tag*{(2)} p ⋆ ( x ∣ y ) ∝ p ⋆ ( x ) p ( y ∣ x ⋆ = x ) = p ⋆ ( x ) exp ( L ( x ; y ) ) , (2)
where the log-likelihood function of the measurement model (1) is given by
L ( x ; y ) : = − 1 2 σ 2 ∥ y − A x ∥ 2 + C σ . (3) \mathcal{L}(x;y):=-\frac{1}{2\sigma^2}\lVert y-Ax\rVert^2+C_\sigma. \tag*{(3)} L ( x ; y ) := − 2 σ 2 1 ∥ y − A x ∥ 2 + C σ . (3)
Here, C σ C_\sigma C σ is a constant depending only on σ \sigma σ .
Rather than developing a posterior sampler tailored to the sparse prior, we instead use sparse recovery as a testbed for a broader question: can a diffusion model (Song et al. [55 ] ), which describes p ⋆ p_\star p ⋆ only implicitly, enable provably efficient posterior sampling? Concretely, consider the smoothed prior
p t : = p ⋆ ∗ N ( 0 , t I n ) , t > 0 , (4) p_t:=p_\star* \mathcal{N}(0,tI_n),\qquad t>0, \tag*{(4)} p t := p ⋆ ∗ N ( 0 , t I n ) , t > 0 , (4)
which is the law of X + t ξ X+\sqrt{t}\xi X + t ξ , where X ∼ p ⋆ X\sim p_\star X ∼ p ⋆ and ξ ∼ N ( 0 , I n ) \xi\sim\mathcal{N}(0,I_n) ξ ∼ N ( 0 , I n ) are independent. We assume access to an estimate s ^ t ( x ) \widehat{s}_t(x) s t ( x ) of the score function
s t ( x ) : = ∇ log p t ( x ) s_t(x):=\nabla\log p_t(x) s t ( x ) := ∇ log p t ( x )
of the smoothed prior p t p_t p t , which is well-defined since p t p_t p t is smooth and positive. In practice, the score functions can be learned, for instance, via score matching (Hyvärinen [30 ] ). Given accurate scores, many diffusion samplers provably generate samples whose law is close to p ⋆ p_\star p ⋆ (see, e.g., Benton et al. [5 ] ; Chen et al. [17 ] ; Lee et al. [39 ] ; Li et al. (2024)), and hence, the diffusion model (as specified by s ^ t ( x ) \widehat{s}_t(x) s t ( x ) ) encodes the prior information. Nonetheless, we cannot directly access an analytical form of p ⋆ p_\star p ⋆ .
Goal. Our goal is to recover the signal of interest x ⋆ x_\star x ⋆ from the measurement y y y , assuming access to a diffusion model (as specified by s ^ t ( x ) \widehat{s}_t(x) s t ( x ) ), in a provably efficient manner, both statistically and computationally.
Diffusion-based inverse solver
Many diffusion-based posterior samplers add a guidance term to the reverse-time dynamics of the diffusion model, which steers the iterates to be consistent with the measurements (Chung et al. [20 ] ). For tractability, we analyze one of the simplest schemes of this type, annealed Langevin dynamics with a likelihood guidance term, proposed in Jalal et al. [32 ] . While this may not be the fastest horse in the race, its simplicity makes a precise analysis possible for deeper mathematical insights.
Tilted targets. To begin with, consider a family of tilted targets. For each level t > 0 t>0 t > 0 , let
Π t ( x ) ∝ p t ( x ) exp ( − ∥ A x − y ∥ 2 2 a t ) , a t : = σ 2 + 2 t , (5) \Pi_t(x)\propto p_t(x)\exp\left(-\frac{\lVert Ax-y\rVert^2}{2a_t}\right),\qquad a_t:=\sigma^2+2t, \tag*{(5)} Π t ( x ) ∝ p t ( x ) exp ( − 2 a t ∥ A x − y ∥ 2 ) , a t := σ 2 + 2 t , (5)
where p t p_t p t is the smoothed prior in (4) . It is easy to check that Π t \Pi_t Π t is the conditional law of X X X given the prior X ∼ p t X\sim p_t X ∼ p t and y = A X + w y=AX+w y = A X + w , where w ∼ N ( 0 , a t I m ) w\sim\mathcal{N}(0,a_tI_m) w ∼ N ( 0 , a t I m ) is independent of X X X , with the variance a t a_t a t adding a smoothing term 2 t 2t 2 t to the noise variance σ 2 \sigma^2 σ 2 . Decreasing the level t t t moves the tilted target Π t ( x ) \Pi_t(x) Π t ( x ) from a nearly Gaussian law (since p t p_t p t is close to N ( 0 , t I n ) \mathcal{N}(0,tI_n) N ( 0 , t I n ) when t t t is large), toward the posterior of the signal given y y y (since p t → p ⋆ p_t \to p_\star p t → p ⋆ and a t → σ 2 a_t \to\sigma^2 a t → σ 2 as t → 0 t \to0 t → 0 ).
At a fixed level t t t , the target Π t \Pi_t Π t is the stationary law of the Langevin diffusion
d X u = ( ∇ log p t ( X u ) + A T ( y − A X u ) a t ) d u + 2 d B u , u ≥ 0 , (6) \mathrm{d}X_u = \left(\nabla\log p_t(X_u) + \frac{A^\mathsf{T}(y-AX_u)}{a_t}\right)\mathrm{d}u + \sqrt{2}\,\mathrm{d}B_u,\qquad u \ge0, \tag*{(6)} d X u = ( ∇ log p t ( X u ) + a t A T ( y − A X u ) ) d u + 2 d B u , u ≥ 0 , (6)
where ( B u ) u ≥ 0 (B_u)_{u\ge0} ( B u ) u ≥ 0 is a standard Brownian motion in R n \mathbb{R}^n R n , and the drift is the score of the tilted target
∇ log Π t ( x ) = ∇ log p t ( x ) + A T ( y − A x ) a t . (7) \nabla\log\Pi_t(x) = \nabla\log p_t(x) + \frac{A^\mathsf{T}(y-Ax)}{a_t}. \tag*{(7)} ∇ log Π t ( x ) = ∇ log p t ( x ) + a t A T ( y − A x ) . (7)
Score-based annealed Langevin sampler. Originally proposed in Jalal et al. [32 ] , our sampler discretizes (6) while decreasing the level along a schedule T = t 0 > t 1 > ⋯ > t N = τ T=t_0>t_1>\cdots>t_N=\tau T = t 0 > t 1 > ⋯ > t N = τ specified below for a total of N N N steps, with the prior score ∇ log p t ( ⋅ ) \nabla\log p_t(\cdot) ∇ log p t ( ⋅ ) replaced by the estimate s ^ t ( ⋅ ) \widehat{s}_t(\cdot) s t ( ⋅ ) . Starting from an initial iterate X ^ 0 \widehat{X}_0 X 0 , it takes one Euler–Maruyama step of size h h h at each level:
X ^ j + 1 = X ^ j + h [ s ^ t j ( X ^ j ) + A T ( y − A X ^ j ) a t j ] + 2 h ξ j , 0 ≤ j < N , (8) \widehat{X}_{j+1}=\widehat{X}_j+h\left[\widehat{s}_{t_j}(\widehat{X}_j)+\frac{A^\mathsf{T}(y-A\widehat{X}_j)}{a_{t_j}}\right]+\sqrt{2h}\,\xi_j,\qquad0\le j<N, \tag*{(8)} X j + 1 = X j + h [ s t j ( X j ) + a t j A T ( y − A X j ) ] + 2 h ξ j , 0 ≤ j < N , (8)
where ξ 0 , … , ξ N − 1 \xi_0,\ldots,\xi_{N-1} ξ 0 , … , ξ N − 1 are i.i.d. N ( 0 , I n ) \mathcal{N}(0,I_n) N ( 0 , I n ) vectors. At high levels, both the smoothed prior and the tilted target are close to broad Gaussian laws, so the updates are essentially Langevin steps toward a Gaussian target and the iterates move freely. As the level decreases, the two terms in the drift acquire distinct roles. The prior score pulls the iterates toward sparse signals, while the guidance pulls them toward signals consistent with the observation. Together, these two forces localize the final iterate near the true signal.
We now specify the choice of algorithmic parameters that will be analyzed in the sequel. Fix an iteration budget N ∈ N N\in\mathbb{N} N ∈ N and a terminal level 0 < τ ≤ 1 0<\tau\le1 0 < τ ≤ 1 at which the annealing stops. We set
T : = n τ T:=\frac{n}{\tau} T := τ n
and the schedule decreases from t 0 = T t_0=T t 0 = T to t N = τ t_N=\tau t N = τ . The remaining parameters are chosen as follows.
t j = T ( τ T ) j / N , j = 0 , 1 , … , N . (9) t_j=T\left(\frac{\tau}{T}\right)^{j/N},\qquad j=0,1,\ldots,N. \tag*{(9)} t j = T ( T τ ) j / N , j = 0 , 1 , … , N . (9)
The exponential schedule (9) is standard in implementations related to the variance-exploding diffusion models (Song and Ermon [54 ] ; Song et al. [55 ] ).
h = N − 2 / 3 . (10) h=N^{-2/3}. \tag*{(10)} h = N − 2/3 . (10)
The step size (10) balances the discretization error of the Euler–Maruyama steps against the movement of the targets Π t j \Pi_{t_j} Π t j between consecutive levels. As N N N grows, each step becomes finer while the total Langevin time N h = N 1 / 3 Nh=N^{1/3} N h = N 1/3 increases.
exp ( − ∥ x ∥ 2 2 T − ∥ A x − y ∥ 2 2 a T ) , (11) \exp\left(-\frac{\|x\|^2}{2T}-\frac{\|Ax-y\|^2}{2a_T}\right), \tag*{(11)} exp ( − 2 T ∥ x ∥ 2 − 2 a T ∥ A x − y ∥ 2 ) , (11)
that is, from N ( μ 0 , Σ 0 ) \mathcal{N}(\mu_0,\Sigma_0) N ( μ 0 , Σ 0 ) with Σ 0 : = ( I n / T + A T A / a T ) − 1 \Sigma_0:=\left(I_n/T+A^\mathsf{T}A/a_T\right)^{-1} Σ 0 := ( I n / T + A T A / a T ) − 1 and μ 0 : = Σ 0 A T y / a T \mu_0:=\Sigma_0A^\mathsf{T}y/a_T μ 0 := Σ 0 A T y / a T , which can be sampled exactly. This law is the first target Π T \Pi_T Π T with p T p_T p T replaced by N ( 0 , T I n ) \mathcal{N}(0,TI_n) N ( 0 , T I n ) , and a large initial level T = n / τ T=n/\tau T = n / τ makes the two close.
Main theorems
We now state the main theorems, which provide high-probability recovery guarantee for fixed sparse signals (Theorem 1), and posterior sampling guarantees (Theorem 2). Define the following quantity
Q : = T + n s = n τ + n s , (12) Q := T + \frac{n}{s} = \frac{n}{\tau} + \frac{n}{s}, \tag*{(12)} Q := T + s n = τ n + s n , (12)
which will be used to describe the required number of iterations.
Sensing matrix ensemble. The following definition collects the sensing matrix ensembles covered by our theory, including both random and deterministic sensing matrix designs.
Definition 2 (Matrix families). Let A = [ A i j ] ∈ R m × n A = [A_{ij}] \in\mathbb{R}^{m \times n} A = [ A ij ] ∈ R m × n with m ≤ n m \le n m ≤ n , which is independent of the signal and the noise.
Independent sub-Gaussian entries. The variables m A i j \sqrt{m}A_{ij} m A ij are independent and identically distributed, centered, unit-variance, and of ψ 2 \psi_2 ψ 2 norm at most κ ≥ 1 \kappa\ge1 κ ≥ 1 .
Randomly subsampled bases. Let U ∈ R n × n U \in\mathbb{R}^{n \times n} U ∈ R n × n be a fixed orthonormal matrix with ∣ U i j ∣ ≤ κ b / n |U_{ij}| \le\kappa_b/\sqrt{n} ∣ U ij ∣ ≤ κ b / n , where κ b ≥ 1 \kappa_b \ge1 κ b ≥ 1 . Let A = n / m R I U A = \sqrt{n/m}R_{\mathcal{I}}U A = n / m R I U , where I \mathcal{I} I is a uniformly random subset of [ n ] [n] [ n ] of size m m m and R I R_{\mathcal{I}} R I extracts the rows of U U U indexed by I \mathcal{I} I .
Deterministic frames. Let A A A be a fixed, and incoherent tight frame, where there exists a fixed κ f ≥ 1 \kappa_f \ge1 κ f ≥ 1 ,
∥ A i ∥ = 1 , A A T = ( n / m ) I m , μ ( A ) : = max i ≠ j ∣ ⟨ a i , a j ⟩ ∣ ≤ κ f m . \lVert A_i\rVert= 1,\qquad AA^\mathsf{T} = (n/m)I_m,\qquad\mu(A) := \max_{i \ne j}|\langle a_i,a_j\rangle| \le\frac{\kappa_f}{\sqrt{m}}. ∥ A i ∥ = 1 , A A T = ( n / m ) I m , μ ( A ) := i = j max ∣ ⟨ a i , a j ⟩ ∣ ≤ m κ f . Here, A i A_i A i is the i i i -th column of A A A , and μ ( A ) \mu(A) μ ( A ) is called the coherence of $A.
Definition 2 encompasses several popular models considered in the literature ([24 , 14 , 25 , 3 ] ), for example, Gaussian and Rademacher matrices, subsampled Hadamard and Fourier matrices, and incoherent frames such as Gabor frames.
Score estimation error. We measure the accuracy of the score estimate by its L 2 L^2 L 2 error along the tilted targets (5) , summed over the levels used by the sampler. For a matrix A A A and a signal x ⋆ x^\star x ⋆ , the score error at A A A and x ⋆ x^\star x ⋆ is
E s c ( A , x ⋆ ) : = E ζ ∑ j < N ∫ ∥ s ^ t j ( x ) − ∇ log p t j ( x ) ∥ 2 Π t j ( d x ) , y = A x ⋆ + σ ζ . (13) \mathcal{E}_{\mathrm{sc}}(A,x^\star) := \mathbb{E}_{\zeta}\sum_{j<N}\int\left\lVert\widehat{s}_{t_j}(x)-\nabla\log p_{t_j}(x)\right\rVert^2\Pi_{t_j}(dx),\qquad y=Ax^\star+\sigma\zeta. \tag*{(13)} E sc ( A , x ⋆ ) := E ζ j < N ∑ ∫ s t j ( x ) − ∇ log p t j ( x ) 2 Π t j ( d x ) , y = A x ⋆ + σ ζ . (13)
where Π t j \Pi_{t_j} Π t j is the tilted target at the data ( A , y ) (A,y) ( A , y ) and the expectation is over the noise ζ \zeta ζ . Averaging over the measurement matrix gives the score error E s c ( x ⋆ ) : = E A E s c ( A , x ⋆ ) \mathcal{E}_{\mathrm{sc}}(x^\star):=\mathbb{E}_A\mathcal{E}_{\mathrm{sc}}(A,x^\star) E sc ( x ⋆ ) := E A E sc ( A , x ⋆ ) at a fixed signal x ⋆ x^\star x ⋆ , and averaging further over a random signal X ⋆ ∼ p ⋆ X^\star\sim p_\star X ⋆ ∼ p ⋆ gives the score error E s c : = E X ⋆ E s c ( X ⋆ ) \mathcal{E}_{\mathrm{sc}}:=\mathbb{E}_{X^\star}\mathcal{E}_{\mathrm{sc}}(X^\star) E sc := E X ⋆ E sc ( X ⋆ ) .
Assumptions of this type are standard in the analysis of diffusion samplers, where the score error is usually measured under the marginals p t p_t p t (see, e.g., [5 ] . In an inverse problem, the tilted distribution Π t \Pi_t Π t instead reflects the information carried by the observation y y y , and is used in collecting the score estimation error ([9 ] ). Due to annealing, the assumption captures the intuition that, as the inverse problem solver approaches the true signal, score accuracy in a neighborhood of x ⋆ x_\star x ⋆ becomes increasingly important.
Recovery guarantees for fixed signals
We first consider recovery of a fixed sparse signal, without assuming that it follows the prior.
Theorem 1 (Recovery of a fixed signal). Suppose x ⋆ ∈ R n x^\star\in\mathbb{R}^n x ⋆ ∈ R n is a k k k -sparse signal with support S S S , observed through (1) with a measurement matrix from one of the families in Definition 2. The sampler uses the prior
model p ⋆ = BG ( s , n ) p_\star= \operatorname{BG}(s,n) p ⋆ = BG ( s , n ) , with prior sparsity parameter 1 ≤ s ≤ n / 2 1 \le s \le n/2 1 ≤ s ≤ n /2 . There are universal constants c , C > 0 c,C > 0 c , C > 0 such that the following holds. Fix 0 < α < 1 0 < \alpha< 1 0 < α < 1 . Suppose that
τ ≤ c min { σ 2 n , α } , N ≥ C Q 18 , s ≤ c σ log ( e n / α ) . (14) \tau\le c\min\left\{\frac{\sigma^2}{n},\alpha\right\}, \qquad N \ge CQ^{18}, \qquad s \le\frac{c}{\sigma\sqrt{\log(en/\alpha)}}. \tag*{(14)} τ ≤ c min { n σ 2 , α } , N ≥ C Q 18 , s ≤ σ log ( e n / α ) c . (14)
If the signal x ⋆ x^\star x ⋆ and the noise σ \sigma σ satisfy
min i ∈ S ∣ x i ⋆ ∣ ≥ C σ log ( e n / ( α τ ) ) , σ ∥ x ⋆ ∥ ≤ c log ( e n / α ) , (15) \min_{i\in S}|x_i^\star| \ge C\sigma\sqrt{\log(en/(\alpha\tau))}, \qquad\sigma\lVert x^\star\rVert\le c\sqrt{\log(en/\alpha)}, \tag*{(15)} i ∈ S min ∣ x i ⋆ ∣ ≥ C σ log ( e n / ( α τ )) , σ ∥ x ⋆ ∥ ≤ c log ( e n / α ) , (15)
and the sample size satisfies
m ≥ C ⋅ { κ 8 ( 1 + s + k + ∥ x ⋆ ∥ 2 ) log 3 ( e n / τ ) log ( e n ) , for i.i.d. sub-Gaussian matrices , κ b 2 ( 1 + s + k + ∥ x ⋆ ∥ 2 ) log 4 ( e n / τ ) log 3 ( e n ) , for randomly subsampled bases , κ f 2 ( 1 + s + k + ∥ x ⋆ ∥ 2 ) 2 log 4 ( e n / τ ) , for deterministic frames , (16) m \ge C\cdot
\begin{cases}
\kappa^8(1+s+k+\lVert x^\star\rVert^2)\log^3(en/\tau)\log(en), & \text{for i.i.d. sub-Gaussian matrices},\\
\kappa_b^2(1+s+k+\lVert x^\star\rVert^2)\log^4(en/\tau)\log^3(en), & \text{for randomly subsampled bases},\\
\kappa_f^2(1+s+k+\lVert x^\star\rVert^2)^2\log^4(en/\tau), & \text{for deterministic frames},
\end{cases} \tag*{(16)} m ≥ C ⋅ ⎩ ⎨ ⎧ κ 8 ( 1 + s + k + ∥ x ⋆ ∥ 2 ) log 3 ( e n / τ ) log ( e n ) , κ b 2 ( 1 + s + k + ∥ x ⋆ ∥ 2 ) log 4 ( e n / τ ) log 3 ( e n ) , κ f 2 ( 1 + s + k + ∥ x ⋆ ∥ 2 ) 2 log 4 ( e n / τ ) , for i.i.d. sub-Gaussian matrices , for randomly subsampled bases , for deterministic frames , (16)
with κ \kappa κ , κ b \kappa_b κ b and κ f \kappa_f κ f as in Definition 2. Then, with probability at least 1 − α − C E s c ( x ⋆ ) 1-\alpha-C\mathcal{E}_{\mathrm{sc}}(x^\star) 1 − α − C E sc ( x ⋆ ) over the measurement matrix, the noise, and the algorithmic randomness, we have
∥ X ^ N − x ⋆ ∥ ≤ C σ ( k + log e n α ) . (17) \lVert\widehat{X}_N-x^\star\rVert\le C\sigma\left(\sqrt{k}+\log\frac{en}{\alpha}\right). \tag*{(17)} ∥ X N − x ⋆ ∥ ≤ C σ ( k + log α e n ) . (17)
We first comment on the parameter choices in (14) . The terminal level τ \tau τ must be small relative to the noise level, so that the smoothing at the final level adds noise comparable to the measurement noise. On the other hand, the prior sparsity level s s s needs to scale inverse proportionally to the noise level, in order to prevent spurious contributions from the noise into the sparse support. Turning to the condition (15) , it prescribes two competing forces on the ground truth signal. The first condition says that the minimum signal strength needs to be not too small with respect to the noise level, which allows reliable detection of its support from noise. The second condition controls the bias of the signal from the Gaussian prior for nonzero entries.
Under these suitable signal and noise conditions, Theorem 1 establishes that, a simple likelihood-guided annealed Langevin sampler can recover fixed sparse signals at the near minimax-optimal rate (ignoring logarithmic factors) [64 , 11 ]
∥ X ^ N − x ⋆ ∥ ≲ σ k \lVert\widehat{X}_N-x^\star\rVert\lesssim\sigma\sqrt{k} ∥ X N − x ⋆ ∥ ≲ σ k
from a small number of measurements using polynomially many score evaluations, despite the lack of log-concavity of its prior. In particular, the sparse signals do not necessarily need to agree with typical signals drawn from the prior, e.g., by allowing the sparsity level k k k of the true signal to be different from the sparsity level s s s postulated by the prior. A few discussions are in order.
Sample efficiency. As long as the sparsity level of the Bernoulli-Gaussian prior satisfies s = O ( k ) s=O(k) s = O ( k ) and the signal strength satisfies ∥ x ⋆ ∥ 2 = O ( k ) \lVert x^\star\rVert^2=O(k) ∥ x ⋆ ∥ 2 = O ( k ) , then the proposed algorithm succeeds when the sample complexity m m m is on the order of O ( k ) O(k) O ( k ) for random matrix ensembles (including sub-Gaussian sensing matrices or randomly subsampled bases), and on the order of O ( k 2 ) O(k^2) O ( k 2 ) for deterministic frames, up to logarithmic factors. For the random ensembles, this sample complexity is near optimal with respect to the sparsity level [24 , 14 ] , and for deterministic frames it matches the recovery guarantees based on the worst-case coherence [25 ] .
Computational efficiency. The iteration complexity N N N is polynomial in Q = n / τ + n / s Q=n/\tau+n/s Q = n / τ + n / s ,1 which is in turn polynomial in n n n , 1 / σ 1/\sigma 1/ σ and 1 / α 1/\alpha 1/ α when τ \tau τ is taken at the upper end of (14) . This certifies diffusion models as computationally efficient prior models in sparse recovery for the first time , without requiring a warm start or unknown burn-in periods [62 , 41 ] .
Score estimation error. With exact scores, the success probability in Theorem 1 is at least 1 − α 1-\alpha 1 − α . For approximate scores, the additional failure probability is controlled by ϵ s c ( x ⋆ ) \epsilon_{\mathrm{sc}}(x^\star) ϵ sc ( x ⋆ ) , which degrades gracefully with the increase of the score estimation error.
Remark 2 (Unknown sparsifying basis). Theorem 1 extends to signals that are sparse in an unknown orthonormal basis, where x ⋆ = Ψ θ ⋆ x^\star=\Psi\theta^\star x ⋆ = Ψ θ ⋆ is sparse in some orthonormal basis Ψ \Psi Ψ . Since N ( 0 , t I n ) \mathcal{N}(0,tI_n) N ( 0 , t I n ) is invariant under orthonormal transformations, the smoothed prior of the diffusion model is x ↦ p t ( θ ) x\mapsto p_t(\theta) x ↦ p t ( θ ) , where θ = Ψ ⊤ x \theta=\Psi^\top x θ = Ψ ⊤ x , and its score is Ψ ( ∇ log p t ) ( Ψ ⊤ x ) \Psi(\nabla\log p_t)(\Psi^\top x) Ψ ( ∇ log p t ) ( Ψ ⊤ x ) . Multiplying (8) by Ψ ⊤ \Psi^\top Ψ ⊤ shows that the iterates θ ^ j : = Ψ ⊤ X ^ j \hat{\theta}_j:=\Psi^\top\hat{X}_j θ ^ j := Ψ ⊤ X ^ j satisfy
θ ^ j + 1 = θ ^ j + h [ s ^ θ ( t j , θ ^ j ) + ( A Ψ ) ⊤ ( y − A Ψ θ ^ j ) a t j ] + 2 h Ψ ⊤ ξ j , \hat{\theta}_{j+1}=\hat{\theta}_j+h\left[\hat{s}_{\theta}(t_j,\hat{\theta}_j)+\frac{(A\Psi)^\top(y-A\Psi\hat{\theta}_j)}{a_{t_j}}\right]+\sqrt{2h}\,\Psi^\top\xi_j, θ ^ j + 1 = θ ^ j + h [ s ^ θ ( t j , θ ^ j ) + a t j ( A Ψ ) ⊤ ( y − A Ψ θ ^ j ) ] + 2 h Ψ ⊤ ξ j ,
where s ^ θ ( t , θ ) \hat{s}_{\theta}(t,\theta) s ^ θ ( t , θ ) estimates ∇ log p t \nabla\log p_t ∇ log p t , and the vectors Ψ ⊤ ξ j \Psi^\top\xi_j Ψ ⊤ ξ j are again i.i.d. N ( 0 , I n ) \mathcal{N}(0,I_n) N ( 0 , I n ) . This is the sampler for the prior B G ( s , n ) \mathrm{BG}(s,n) BG ( s , n ) with the measurement matrix A Ψ A\Psi A Ψ , and the initialization (11) and the score error (13) transform in the same way. When A A A has i.i.d. Gaussian entries, A Ψ A\Psi A Ψ has the same law as A A A , and Theorem 1 applied to θ ⋆ \theta^\star θ ⋆ yields the bound (17) for ∥ X ^ N − x ⋆ ∥ = ∥ θ ^ N − θ ⋆ ∥ \|\hat{X}_N-x^\star\|=\|\hat{\theta}_N-\theta^\star\| ∥ X ^ N − x ⋆ ∥ = ∥ θ ^ N − θ ⋆ ∥ , with the signal conditions (15) imposed on θ ⋆ \theta^\star θ ⋆ . More generally, the conclusion holds whenever A Ψ A\Psi A Ψ belongs to one of the families in Definition 2, which for randomly subsampled bases requires U Ψ U\Psi U Ψ to satisfy the incoherence condition.
Posterior sampling guarantees
We now turn to the case that the ground truth signal indeed follows the prior, X ⋆ ∼ p ⋆ = B G ( s , n ) X^\star\sim p_\star=\mathrm{BG}(s,n) X ⋆ ∼ p ⋆ = BG ( s , n ) , independently of the measurement matrix. In this case, we are interested in obtaining posterior sampling guarantees, where the recovery follows a distribution close to the posterior distribution p ⋆ ( ⋅ ∣ y ) p_\star(\cdot\mid y) p ⋆ ( ⋅ ∣ y ) defined in (2) in total variation.
For the sparse recovery task with Bernoulli-Gaussian priors, the vanilla annealed Langevin sampler does not readily yield a posterior sampling guarantee. This is because the Gaussian perturbation in the final update (8) makes X ^ N \hat{X}_N X ^ N nonzero with probability one for all coordinates, whereas the posterior in (2) assigns positive mass to vectors with zero coordinates. To obtain a total-variation guarantee, we follow the Langevin updates with an endpoint draw that is defined as follows.
Definition 3 (Endpoint kernel and endpoint draw). For t > 0 t>0 t > 0 and x ∈ R n x\in\mathbb{R}^n x ∈ R n , the endpoint kernel D t ( x , ⋅ ) D_t(x,\cdot) D t ( x , ⋅ ) is the conditional law of Z Z Z given Z + t ξ = x Z+\sqrt{t}\,\xi=x Z + t ξ = x , where Z ∼ p ⋆ Z\sim p_\star Z ∼ p ⋆ and ξ ∼ N ( 0 , I n ) \xi\sim\mathcal{N}(0,I_n) ξ ∼ N ( 0 , I n ) are independent. The endpoint draw is Z ^ ∼ D τ ( X ^ N , ⋅ ) \hat{Z}\sim D_\tau(\hat{X}_N,\cdot) Z ^ ∼ D τ ( X ^ N , ⋅ ) , generated using fresh randomness independent of the data and the Langevin chain.
In other words, the endpoint draw applies an additional posterior denoising step on X ^ N \hat{X}_N X ^ N , assuming additive Gaussian noise at the noise level τ \tau τ . In general, the endpoint draw can be implemented efficiently using the same diffusion model for unconditional generation, as discussed in Xu and Chi (2024). For the Bernoulli–Gaussian prior B G ( s , n ) \mathrm{BG}(s,n) BG ( s , n ) , the endpoint draw admits a simple closed-form expression. Each coordinate of z ∼ D t ( x , ⋅ ) z\sim D_t(x,\cdot) z ∼ D t ( x , ⋅ ) is independently generated by
z i = { 0 with probability 1 − ζ t ( x i ) , N ( x i 1 + t , t 1 + t ) otherwise , (18) z_i=
\begin{cases}
0 & \text{with probability }1-\zeta_t(x_i),\\
\mathcal{N}\left(\dfrac{x_i}{1+t},\dfrac{t}{1+t}\right) & \text{otherwise},
\end{cases} \tag*{(18)} z i = ⎩ ⎨ ⎧ 0 N ( 1 + t x i , 1 + t t ) with probability 1 − ζ t ( x i ) , otherwise , (18)
where
ζ t ( v ) : = [ 1 + n − s s 1 + 1 t e − v 2 / [ 2 t ( 1 + t ) ] ] − 1 \zeta_t(v):=\left[1+\frac{n-s}{s}\sqrt{1+\frac{1}{t}}e^{-v^2/[2t(1+t)]}\right]^{-1} ζ t ( v ) := [ 1 + s n − s 1 + t 1 e − v 2 / [ 2 t ( 1 + t )] ] − 1
is the conditional probability that a coordinate is nonzero given its noisy value v v v . Armed with this endpoint draw, we are ready to present the posterior sampling guarantee in total variation.
Theorem 2 (Posterior sampling). Suppose X ⋆ ∼ p ⋆ = B G ( s , n ) X_\star\sim p_\star=\mathrm{BG}(s,n) X ⋆ ∼ p ⋆ = BG ( s , n ) , with 1 ≤ s ≤ n / 2 1\le s\le n/2 1 ≤ s ≤ n /2 , is observed through (1) with a measurement matrix from one of the families in Definition 2 independent of X ⋆ X^\star X ⋆ . There are universal constants c , C > 0 c,C>0 c , C > 0 such that the following holds. Fix 0 < α < 1 0<\alpha<1 0 < α < 1 . Suppose that
τ ≤ c min { α 2 , α σ 2 n 2 ( 1 + s / σ 2 ) } , N ≥ C Q 18 , (19) \tau\le c\min\left\{\alpha^2,\frac{\alpha\sigma^2}{n^2(1+s/\sigma^2)}\right\},\qquad N\ge CQ^{18}, \tag*{(19)} τ ≤ c min { α 2 , n 2 ( 1 + s / σ 2 ) α σ 2 } , N ≥ C Q 18 , (19)
and the sample size satisfies
m ≥ C ⋅ { κ 8 ( s + log ( 1 / α ) ) log 3 ( e n / τ ) log ( e n ) , for i.i.d. sub-Gaussian matrices , κ b 2 ( s + log ( 1 / α ) ) log 4 ( e n / τ ) log 3 ( e n ) , for randomly subsampled bases , κ f 2 ( s + log ( 1 / α ) ) 2 log 4 ( e n / τ ) , for deterministic frames , (20) m \ge C\cdot
\begin{cases}
\kappa^{8}\left(s+\log(1/\alpha)\right)\log^{3}(en/\tau)\log(en), & \text{for i.i.d. sub-Gaussian matrices},\\
\kappa_b^{2}\left(s+\log(1/\alpha)\right)\log^{4}(en/\tau)\log^{3}(en), & \text{for randomly subsampled bases},\\
\kappa_f^{2}\left(s+\log(1/\alpha)\right)^{2}\log^{4}(en/\tau), & \text{for deterministic frames},
\end{cases} \tag*{(20)} m ≥ C ⋅ ⎩ ⎨ ⎧ κ 8 ( s + log ( 1/ α ) ) log 3 ( e n / τ ) log ( e n ) , κ b 2 ( s + log ( 1/ α ) ) log 4 ( e n / τ ) log 3 ( e n ) , κ f 2 ( s + log ( 1/ α ) ) 2 log 4 ( e n / τ ) , for i.i.d. sub-Gaussian matrices , for randomly subsampled bases , for deterministic frames , (20)
with κ \kappa κ , κ b \kappa_b κ b and κ f \kappa_f κ f as in Definition 2. Then the law of the endpoint draw Z ^ ∼ D τ ( X ^ N , ⋅ ) \widehat{Z}\sim D_\tau(\widehat{X}_N,\cdot) Z ∼ D τ ( X N , ⋅ ) satisfies
E TV ( L ( Z ^ ∣ A , y ) , p ⋆ ( ⋅ ∣ y ) ) ≲ α + E s c , \mathbb{E}\operatorname{TV}\left(\mathcal{L}(\widehat{Z}\mid A,y),p_\star(\cdot\mid y)\right)\lesssim\alpha+\sqrt{\mathcal{E}_{\mathrm{sc}}}, E TV ( L ( Z ∣ A , y ) , p ⋆ ( ⋅ ∣ y ) ) ≲ α + E sc ,
where p ⋆ ( ⋅ ∣ y ) p_\star(\cdot\mid y) p ⋆ ( ⋅ ∣ y ) is the true posterior in (2) . Here, E s c \mathcal{E}_{\mathrm{sc}} E sc is the averaged score error defined in (13) .
Theorem 2 establishes that, when the signal follows the Bernoulli–Gaussian prior, the likelihood-guided annealed Langevin sampler, followed by an endpoint draw, approximates the posterior in total variation up to an order of
α + E s c \alpha+\sqrt{\mathcal{E}_{\mathrm{sc}}} α + E sc
using polynomially many score evaluations, as soon as the sample complexity m m m is on the order of s + log ( 1 / α ) s+\log(1/\alpha) s + log ( 1/ α ) for random matrix ensembles (including sub-Gaussian sensing matrices or randomly subsampled bases), and on the order of ( s + log ( 1 / α ) ) 2 \left(s+\log(1/\alpha)\right)^2 ( s + log ( 1/ α ) ) 2 for deterministic frames, up to logarithmic factors. Furthermore, the performance guarantee degenerates gracefully as the score estimation error increases, analogous to that in unconditional diffusion guarantees [17 Theorem 2] .
To the best of our knowledge, Theorem 2 provides the first polynomial-time posterior sampling algorithm at the near-optimal sample complexity O ~ ( s ) \widetilde{O}(s) O ( s ) using random sensing ensembles, improving upon the sample complexities of several existing posterior sampling algorithms [37 , 38 ] that are not based on diffusion models.
Remark 3. Compared to Theorem 1, Theorem 2 holds for every σ > 0 \sigma>0 σ > 0 , without the prior sparsity restriction (14) or the signal conditions (15) . The early stopping level in (19) must be small enough to control both the sampling error relative to Π τ \Pi_\tau Π τ and the difference between the endpoint distribution and the desired posterior.
Proof outline
We provide an outline of our analysis. The central machinery is an annealed transfer inequality , detailed in § 5.1 , which bounds the distance between the laws of the output of the sampler and the terminal target Π τ \Pi_\tau Π τ in (5) under certain weak variance inequality that can be certified for the sensing ensembles of interest. We then move on to establishing Theorem 1 in § 5.2 and Theorem 2 in § 5.3 , by studying the relations of the terminal target Π τ \Pi_\tau Π τ with signal reconstruction and posterior sampling.
We also introduce some notation used throughout the proof. For a probability measure μ \mu μ , write E μ f = μ f : = ∫ f d μ \mathbb{E}_{\mu}f=\mu f:=\int f\,\mathrm{d}\mu E μ f = μ f := ∫ f d μ . For a real square-integrable function f f f and a bounded function g g g , respectively, define
Var μ f : = μ ( f − μ f ) 2 = μ f 2 − ( μ f ) 2 , osc ( g ) : = sup g − inf g . \operatorname{Var}_{\mu}f:=\mu(f-\mu f)^2=\mu f^2-(\mu f)^2,\qquad\operatorname{osc}(g):=\sup g-\inf g. Var μ f := μ ( f − μ f ) 2 = μ f 2 − ( μ f ) 2 , osc ( g ) := sup g − inf g .
A Markov kernel K ( x , ⋅ ) K(x,\cdot) K ( x , ⋅ ) is a probability measure for each x x x , measurable in x x x on every measurable set. We let
K f ( x ) : = ∫ f ( z ) K ( x , d z ) , ( μ K ) ( ⋅ ) : = ∫ K ( x , ⋅ ) μ ( d x ) . Kf(x):=\int f(z)K(x,\mathrm{d}z),\qquad(\mu K)(\cdot):=\int K(x,\cdot)\mu(\mathrm{d}x). K f ( x ) := ∫ f ( z ) K ( x , d z ) , ( μ K ) ( ⋅ ) := ∫ K ( x , ⋅ ) μ ( d x ) .
For probability measures μ 1 , μ 2 \mu_1,\mu_2 μ 1 , μ 2 with densities p , q p,q p , q with respect to a common dominating measure λ \lambda λ , the squared Hellinger distance is
H 2 ( μ 1 , μ 2 ) : = ∫ ( p − q ) 2 d λ = 2 − 2 ∫ p q d λ . (21) H^2(\mu_1,\mu_2):=\int\left(\sqrt{p}-\sqrt{q}\right)^2\mathrm{d}\lambda=2-2\int\sqrt{pq}\,\mathrm{d}\lambda. \tag*{(21)} H 2 ( μ 1 , μ 2 ) := ∫ ( p − q ) 2 d λ = 2 − 2 ∫ pq d λ . (21)
Under the parameter choices of the main theorems, T = n / τ T=n/\tau T = n / τ , τ ≤ 1 ≤ n \tau\le1\le n τ ≤ 1 ≤ n and 1 ≤ s ≤ n / 2 1\le s\le n/2 1 ≤ s ≤ n /2 . Hence the quantity Q Q Q of (12) satisfies
Q ≥ T ≥ max { n , 1 / τ } , Q ≥ n s ≥ 2. (22) Q\ge T\ge\max\{n,1/\tau\},\qquad Q\ge\frac{n}{s}\ge2. \tag*{(22)} Q ≥ T ≥ max { n , 1/ τ } , Q ≥ s n ≥ 2. (22)
The annealed transfer inequality
Given a fixed instance ( A , y ) (A,y) ( A , y ) following (1) , we define the conditional law of the output of the sampler (8) as
ρ N : = L ( X ^ N ∣ A , y ) , \rho_N := \mathcal{L}(\widehat{X}_N \mid A,y), ρ N := L ( X N ∣ A , y ) ,
and let μ i n i t \mu_{\mathrm{init}} μ init be the law of X ^ 0 \widehat{X}_0 X 0 in (11) given the data pair ( A , y ) (A,y) ( A , y ) . By (7) , the drift of the update (8) at x x x is
∇ log Π t j ( x ) + Δ j ( x ) , where Δ j ( x ) : = s ^ t j ( x ) − ∇ log p t j ( x ) \nabla\log\Pi_{t_j}(x) + \Delta_j(x), \qquad\text{where } \Delta_j(x) := \widehat{s}_{t_j}(x) - \nabla\log p_{t_j}(x) ∇ log Π t j ( x ) + Δ j ( x ) , where Δ j ( x ) := s t j ( x ) − ∇ log p t j ( x )
is the error of the score estimate. Since the vectors ξ j \xi_j ξ j are independent of ( A , y ) (A,y) ( A , y ) and of X ^ 0 \widehat{X}_0 X 0 , each update, conditioned on ( A , y ) (A,y) ( A , y ) , is a draw from
K ^ j ( x , ⋅ ) : = N ( x + h [ ∇ log Π t j ( x ) + Δ j ( x ) ] , 2 h I n ) . (23) \widehat{K}_j(x,\cdot) := \mathcal{N}\left(x+h\left[\nabla\log\Pi_{t_j}(x)+\Delta_j(x)\right],2hI_n\right). \tag*{(23)} K j ( x , ⋅ ) := N ( x + h [ ∇ log Π t j ( x ) + Δ j ( x ) ] , 2 h I n ) . (23)
a Langevin step toward Π t j \Pi_{t_j} Π t j with the additional drift Δ j \Delta_j Δ j . We can then write the conditional law of the sampler output as
ρ N = μ i n i t K ^ 0 ⋯ K ^ N − 1 . (24) \rho_N = \mu_{\mathrm{init}}\widehat{K}_0\cdots\widehat{K}_{N-1}. \tag*{(24)} ρ N = μ init K 0 ⋯ K N − 1 . (24)
The annealed transfer inequality, which bounds the difference between ρ N \rho_N ρ N and Π τ \Pi_\tau Π τ at fixed instances by a sum of error terms, requires certain weak variance inequality for the tilted targets, stated as follows.
Definition 4 (Weak variance inequality). A probability measure γ \gamma γ on R n \mathbb{R}^n R n satisfies a weak variance inequality with constant V ≥ 0 V \ge0 V ≥ 0 and defect δ ≥ 0 \delta\ge0 δ ≥ 0 if every bounded smooth g g g with bounded gradient satisfies
Var γ g ≤ V γ ∥ ∇ g ∥ 2 + δ osc ( g ) 2 . (25) \operatorname{Var}_{\gamma} g \le V\gamma\lVert\nabla g\rVert^2+\delta\operatorname{osc}(g)^2. \tag*{(25)} Var γ g ≤ V γ ∥ ∇ g ∥ 2 + δ osc ( g ) 2 . (25)
The weak variance inequality (25) has the form of a weak Poincaré inequality (Röckner and Wang, [47 ] ) at a fixed defect level. We now state the annealed transfer inequality for a fixed instance, assuming the tilted targets have smooth scores and satisfy the weak variance inequality.
Proposition 1 (Annealed transfer). Suppose that there exists some L , V ≥ 0 L,V \ge0 L , V ≥ 0 such that for every j < N j<N j < N , the score ∇ log Π t j \nabla\log\Pi_{t_j} ∇ log Π t j is L L L -Lipschitz and Π t j \Pi_{t_j} Π t j satisfies the weak variance inequality of Definition 4 with constant V V V and defect δ j ≥ 0 \delta_j \ge0 δ j ≥ 0 . Then for every g : R n → [ 0 , 1 ] g:\mathbb{R}^n\to[0,1] g : R n → [ 0 , 1 ] and every 0 < λ ≤ 1 0<\lambda\le1 0 < λ ≤ 1 , the following holds
ρ N g ≤ ( 1 + λ ) Π τ g + C E λ + C E d e f , (26) \rho_N g \le(1+\lambda)\Pi_\tau g+\frac{C\mathcal{E}}{\lambda}+C\mathcal{E}_{\mathrm{def}}, \tag*{(26)} ρ N g ≤ ( 1 + λ ) Π τ g + λ C E + C E def , (26) where
E : = H 2 ( Π T , μ i n i t ) + ∑ j < N h Π t j ∥ Δ j ∥ 2 + n L 2 N h 2 + ( 1 + V h ) ∑ j < N H 2 ( Π t j , Π t j + 1 ) , (27) \mathcal{E}:=H^2(\Pi_T,\mu_{\mathrm{init}})+\sum_{j<N}h\Pi_{t_j}\lVert\Delta_j\rVert^2+nL^2Nh^2+\left(1+\frac{V}{h}\right)\sum_{j<N}H^2(\Pi_{t_j},\Pi_{t_{j+1}}), \tag*{(27)} E := H 2 ( Π T , μ init ) + j < N ∑ h Π t j ∥ Δ j ∥ 2 + n L 2 N h 2 + ( 1 + h V ) j < N ∑ H 2 ( Π t j , Π t j + 1 ) , (27) E d e f : = ∑ j < N δ j H 2 ( Π t j , Π t j + 1 ) . (28) \mathcal{E}_{\mathrm{def}}:=\sum_{j<N}\sqrt{\delta_j}\,H^2(\Pi_{t_j},\Pi_{t_{j+1}}). \tag*{(28)} E def := j < N ∑ δ j H 2 ( Π t j , Π t j + 1 ) . (28) Here, H 2 H^2 H 2 denotes the squared Hellinger distance defined in (21) .
The proof of this proposition can be found in Appendix § 8 .§ 1 . Proposition 1 establishes the proximity between ρ N \rho_N ρ N and Π τ \Pi_\tau Π τ , by showing that the expectation ρ N g \rho_N g ρ N g under the sampler output is close to Π τ g \Pi_\tau g Π τ g under the terminal target for every g g g up to a few error terms. The parameter λ \lambda λ balances the contribution of Π τ g \Pi_\tau g Π τ g and E \mathcal{E} E , where the latter contains the following terms:
Initialization error. The squared Hellinger distance H 2 ( Π T , μ i n i t ) H^2(\Pi_T,\mu_{\mathrm{init}}) H 2 ( Π T , μ init ) measures the mismatch between the first target Π T \Pi_T Π T and the initial law μ i n i t \mu_{\mathrm{init}} μ init , which in turn measures the mismatch between p T p_T p T and N ( 0 , T I n ) \mathcal{N}(0,TI_n) N ( 0 , T I n ) , a term that diminishes quickly as T T T becomes large.
Score estimation error. The term ∑ j < N h Π t j ∥ Δ j ∥ 2 \sum_{j<N}h\Pi_{t_j}\lVert\Delta_j\rVert^2 ∑ j < N h Π t j ∥ Δ j ∥ 2 , which is linear in the step size h h h , accumulates the mean squared score error under the tilted targets.
Discretization error. The term n L 2 N h 2 nL^{2}Nh^{2} n L 2 N h 2 , which is quadratic in the step size h h h , bounds the error from the total N N N Euler–Maruyama steps. Since h = N − 2 / 3 h=N^{-2/3} h = N − 2/3 in (10) , this term decreases at a rate of N − 1 / 3 N^{-1/3} N − 1/3 as the number of steps N N N increases.
Target movement error. The term ( 1 + V / h ) ∑ j < N H 2 ( Π t j , Π t j + 1 ) (1+V/h)\sum_{j<N}H^{2}(\Pi_{t_j},\Pi_{t_{j+1}}) ( 1 + V / h ) ∑ j < N H 2 ( Π t j , Π t j + 1 ) accounts for the change between consecutive targets, measured by their squared Hellinger distances. In particular, the factor 1 + V / h 1+V/h 1 + V / h scales inverse proportionally with the step size h h h .
Last but not least, the additional term E d e f \mathcal{E}_{\mathrm{def}} E def also accounts for the change between consecutive targets as a weighted sum of their Hellinger distances. It then boils down to 1) establishing the desired properties of the tilted targets Π t j \Pi_{t_j} Π t j to apply Proposition 1, and 2) controlling the error terms in (27) when averaged over the measurement noise.
The matrix condition. In order to make use of Proposition 1, we need to supply the desired smoothness and weak variance inequality for the tilted targets { Π t j } j < N \{\Pi_{t_j}\}_{j<N} { Π t j } j < N under appropriate conditions of the sensing matrix. To begin, for a fixed signal x ⋆ x^\star x ⋆ with support S S S , define its cap as
q : = ⌈ 14 ( 1 + s + ∣ S ∣ + ∥ x ⋆ ∥ 2 ) log ( 2 Q ) ⌉ . (29) q := \left\lceil14(1+s+|S|+\|x^\star\|^{2})\log(2Q)\right\rceil. \tag*{(29)} q := ⌈ 14 ( 1 + s + ∣ S ∣ + ∥ x ⋆ ∥ 2 ) log ( 2 Q ) ⌉ . (29)
and set ε : = c ε / log ( 2 Q ) \varepsilon:=c_{\varepsilon}/\log(2Q) ε := c ε / log ( 2 Q ) , where c ε ∈ ( 0 , 1 / 16 ] c_{\varepsilon}\in(0,1/16] c ε ∈ ( 0 , 1/16 ] is a sufficiently small universal constant. Since log ( 2 Q ) ≥ 1 \log(2Q)\ge1 log ( 2 Q ) ≥ 1 by (22) , these choices give
ε ≤ c ε ≤ 1 16 , q ≥ ∣ S ∣ + ∥ x ⋆ ∥ 2 ≥ ∣ S ∣ . (30) \varepsilon\le c_{\varepsilon}\le\frac{1}{16},\qquad q\ge|S|+\|x^\star\|^{2}\ge|S|. \tag*{(30)} ε ≤ c ε ≤ 16 1 , q ≥ ∣ S ∣ + ∥ x ⋆ ∥ 2 ≥ ∣ S ∣. (30)
Let
∀ t > 0 : Σ t : = a t I m + t A A T , B t : = I n + A T Σ t − 1 A , (31) \forall t>0:\qquad\Sigma_t:=a_t I_m+tAA^{\mathsf{T}},\qquad B_t:=I_n+A^{\mathsf{T}}\Sigma_t^{-1}A, \tag*{(31)} ∀ t > 0 : Σ t := a t I m + t A A T , B t := I n + A T Σ t − 1 A , (31)
where a t a_t a t is defined in (5). We define the following matrix condition.
Definition 5 (Matrix condition). The matrix event A ( q ) \mathcal{A}(q) A ( q ) of x ⋆ x^\star x ⋆ is the set of matrices A A A that satisfy
min i ∥ A i ∥ 2 ≥ 1 2 , (32) \min_i \|A_i\|^2\ge\frac{1}{2}, \tag*{(32)} i min ∥ A i ∥ 2 ≥ 2 1 , (32) ∥ A ∥ 2 ≤ 2 n , (33) \|A\|^2\le2n, \tag*{(33)} ∥ A ∥ 2 ≤ 2 n , (33) ∥ ( B t − diag B t ) J J ∥ ≤ ε min i ∈ J B t , i i for every τ ≤ t ≤ T and J ⊆ [ n ] with 1 ≤ ∣ J ∣ ≤ 2 q . (34) \|(B_t-\operatorname{diag} B_t)_{JJ}\|\le\varepsilon\min_{i\in J}B_{t,ii}\qquad\text{for every }\tau\le t\le T\text{ and }J\subseteq[n]\text{ with }1\le|J|\le2q. \tag*{(34)} ∥ ( B t − diag B t ) J J ∥ ≤ ε i ∈ J min B t , ii for every τ ≤ t ≤ T and J ⊆ [ n ] with 1 ≤ ∣ J ∣ ≤ 2 q . (34)
Here, A i A_i A i denotes the i i i -th column of A A A .
The last condition (34) , which we refer to as the block condition, requires B t B_t B t to be close to its diagonal approximation on every set of at most 2 q 2q 2 q coordinates. This is reminiscent of the celebrated restricted isometry property [13 , 57 ] , but requires the matrix B t B_t B t to behave like a well-conditioned diagonal matrix along the diffusion path t ∈ [ τ , T ] t\in[\tau,T] t ∈ [ τ , T ] . For fixed x ⋆ x^\star x ⋆ , whether A A A satisfies the matrix condition depends on the sensing matrix ensemble, independently of the observation noise. Luckily, this event holds with high probability under the sample size of the main theorems for the sensing ensembles of interest, and we defer the verification to Appendix § 10 .
The averaged annealed transfer inequality. The matrix condition allows us to certify the desired properties of the tilted targets { Π t j } j < N \{\Pi_{t_j}\}_{j<N} { Π t j } j < N to apply Proposition 1. Averaging over the observation noise ζ \zeta ζ , we can bound the error terms and obtain the next lemma summarizing the resulting error in terms of the terminal level τ \tau τ and the score error E s c ( A , x ⋆ ) \mathcal{E}_{\mathrm{sc}}(A,x^\star) E sc ( A , x ⋆ ) .
Lemma 1 (Averaged annealed transfer inequality). There are universal constants C 0 , C C_0,C C 0 , C such that the following holds. Let x ⋆ x^\star x ⋆ be a signal with ∥ x ⋆ ∥ 2 ≤ n \|x^\star\|^2\le n ∥ x ⋆ ∥ 2 ≤ n , and A ∈ A ( q ) A\in\mathcal{A}(q) A ∈ A ( q ) satisfy the matrix condition of Definition 5 for q q q defined in (29) . Suppose that N ≥ C 0 Q 18 N\ge C_0Q^{18} N ≥ C 0 Q 18 . Then, with y = A x ⋆ + σ ζ y=Ax^\star+\sigma\zeta y = A x ⋆ + σ ζ , every g : R n → [ 0 , 1 ] g:\mathbb{R}^n\to[0,1] g : R n → [ 0 , 1 ] and every 0 < λ ≤ 1 0<\lambda\le1 0 < λ ≤ 1 ,
E ζ ρ N g ≤ ( 1 + λ ) E ζ Π τ g + C λ ( τ + E s c ( A , x ⋆ ) ) , (35) \mathbb{E}_{\zeta}\rho_N g\le(1+\lambda)\mathbb{E}_{\zeta}\Pi_\tau g+\frac{C}{\lambda}\left(\tau+\mathcal{E}_{\mathrm{sc}}(A,x^\star)\right), \tag*{(35)} E ζ ρ N g ≤ ( 1 + λ ) E ζ Π τ g + λ C ( τ + E sc ( A , x ⋆ ) ) , (35) E ζ TV ( ρ N , Π τ ) ≤ C ( τ + E s c ( A , x ⋆ ) ) 1 / 2 . (36) \mathbb{E}_{\zeta}\operatorname{TV}(\rho_N,\Pi_\tau)\le C\left(\tau+\mathcal{E}_{\mathrm{sc}}(A,x^\star)\right)^{1/2}. \tag*{(36)} E ζ TV ( ρ N , Π τ ) ≤ C ( τ + E sc ( A , x ⋆ ) ) 1/2 . (36)
The proof of this lemma can be found in Appendix § 8 .§ 3 . Our main theorems will be established by making use of (35) and (36) , respectively.
Proof of Theorem 1
For the fixed signal x ⋆ x^\star x ⋆ with support S S S , define the failure set
B ( x ⋆ ) : = { x : ∥ x − x ⋆ ∥ > C r a d σ ( ∣ S ∣ + log ( e n / α ) ) } , (37) \mathcal{B}(x^\star) := \left\{x : \left\|x-x^\star\right\| > C_{\mathrm{rad}} \sigma\left(\sqrt{|S|}+\log(en/\alpha)\right)\right\}, \tag*{(37)} B ( x ⋆ ) := { x : ∥ x − x ⋆ ∥ > C rad σ ( ∣ S ∣ + log ( e n / α ) ) } , (37)
where C r a d C_{\mathrm{rad}} C rad is the universal constant of Lemma 16 . Whenever it is clear from the context, we use B : = B ( x ⋆ ) \mathcal{B}:=\mathcal{B}(x^\star) B := B ( x ⋆ ) when the signal x ⋆ x^\star x ⋆ is fixed, and use A \mathcal{A} A for the event that A A A satisfies the matrix condition of Definition 5.
Since the sample-size condition (16) implies ∥ x ⋆ ∥ 2 ≤ m ≤ n \left\|x^\star\right\|^2\leq m\leq n ∥ x ⋆ ∥ 2 ≤ m ≤ n , and the iteration bound in (14) gives N ≥ C 0 Q 18 N\geq C_0Q^{18} N ≥ C 0 Q 18 when its constant C C C is at least the constant C 0 C_0 C 0 of Lemma 1, applying Lemma 1 for every A ∈ A A\in\mathcal{A} A ∈ A with g = 1 B g=\mathbf{1}_{\mathcal{B}} g = 1 B and λ = 1 \lambda=1 λ = 1 gives
E ζ ρ N ( B ) ≤ 2 E ζ Π τ ( B ) + C ( τ + E s c ( A , x ⋆ ) ) . (38) \mathbb{E}_{\zeta}\rho_N(\mathcal{B})\leq2\mathbb{E}_{\zeta}\Pi_\tau(\mathcal{B})+C(\tau+\mathcal{E}_{\mathrm{sc}}(A,x^\star)). \tag*{(38)} E ζ ρ N ( B ) ≤ 2 E ζ Π τ ( B ) + C ( τ + E sc ( A , x ⋆ )) . (38)
Since A \mathcal{A} A depends only on A A A , we can decompose the error and average (38) over the sensing matrix as
P ( X ^ N ∈ B ) ≤ P ( A c ) + E A [ 1 A E ζ ρ N ( B ) ] ≤ P ( A c ) + 2 E A [ 1 A E ζ Π τ ( B ) ] + C ( τ + E s c ( x ⋆ ) ) , (39) \begin{aligned}
\mathbb{P}(\widehat{X}_N\in\mathcal{B})\leq\mathbb{P}(\mathcal{A}^{c})+\mathbb{E}_{A}\left[\mathbf{1}_{\mathcal{A}}\mathbb{E}_{\zeta}\rho_N(\mathcal{B})\right] \\
&\leq\mathbb{P}(\mathcal{A}^{c})+2\mathbb{E}_{A}\left[\mathbf{1}_{\mathcal{A}}\mathbb{E}_{\zeta}\Pi_\tau(\mathcal{B})\right]+C(\tau+\mathcal{E}_{\mathrm{sc}}(x^\star)), \tag*{(39)}
\end{aligned} P ( X N ∈ B ) ≤ P ( A c ) + E A [ 1 A E ζ ρ N ( B ) ] ≤ P ( A c ) + 2 E A [ 1 A E ζ Π τ ( B ) ] + C ( τ + E sc ( x ⋆ )) , (39)
where E A E s c ( A , x ⋆ ) = E s c ( x ⋆ ) \mathbb{E}_{A}\mathcal{E}_{\mathrm{sc}}(A,x^\star)=\mathcal{E}_{\mathrm{sc}}(x^\star) E A E sc ( A , x ⋆ ) = E sc ( x ⋆ ) . Under the assumptions (14) and (16) , together with ∣ S ∣ ≤ k |S|\leq k ∣ S ∣ ≤ k , it gives (cf. Proposition 3)
P ( A c ) ≤ α . \mathbb{P}(\mathcal{A}^{c})\leq\alpha. P ( A c ) ≤ α .
We next turn to the second term in (39) , which measures the mass of the terminal target Π τ \Pi_\tau Π τ outside the recovery ball B c \mathcal{B}^{c} B c . This is supplied in the following lemma, whose proof is deferred to Appendix § 9 .§ 1 .
Lemma 2 (Terminal accuracy). Under the conditions of Theorem 1, for every A ∈ A A\in\mathcal{A} A ∈ A , it holds
E ζ Π τ ( B ) ≤ C α . \mathbb{E}_{\zeta}\Pi_\tau(\mathcal{B})\leq C\alpha. E ζ Π τ ( B ) ≤ C α .
Finally, τ ≤ α \tau\leq\alpha τ ≤ α by (14) , so substituting these bounds into (39) gives, for some universal constant C 1 ≥ 1 C_1\geq1 C 1 ≥ 1 ,
P ( X ^ N ∈ B ) ≤ C 1 ( α + E s c ( x ⋆ ) ) . (40) \mathbb{P}(\widehat{X}_N\in\mathcal{B})\leq C_1(\alpha+\mathcal{E}_{\mathrm{sc}}(x^\star)). \tag*{(40)} P ( X N ∈ B ) ≤ C 1 ( α + E sc ( x ⋆ )) . (40)
Since ∣ S ∣ ≤ k |S|\leq k ∣ S ∣ ≤ k , applying this bound with α / C 1 \alpha/C_1 α / C 1 in place of α \alpha α proves Theorem 1 after adjusting the universal constants.
Proof of Theorem 2
We begin by noting that the endpoint draw has conditional law
L ( Z ^ ∣ A , y ) = ρ N D τ , where ρ N = L ( X ^ N ∣ A , y ) . \mathcal{L}(\widehat{Z}\mid A,y)=\rho_ND_\tau,\qquad\text{where }\rho_N=\mathcal{L}(\widehat{X}_N\mid A,y). L ( Z ∣ A , y ) = ρ N D τ , where ρ N = L ( X N ∣ A , y ) .
Denoting the true posterior p ⋆ : = p ⋆ ( ⋅ ∣ y ) p^\star:=p^\star(\cdot\mid y) p ⋆ := p ⋆ ( ⋅ ∣ y ) , we can bound that, for every A A A and y y y ,
TV ( ρ N D τ , p ⋆ A , y ) ≤ TV ( ρ N D τ , Π τ D τ ) + TV ( Π τ D τ , p ⋆ A , y ) ≤ TV ( ρ N , Π τ ) + TV ( Π τ D τ , p ⋆ A , y ) . (41) \begin{aligned}
\operatorname{TV}(\rho_ND_\tau,p_\star^{A,y})\leq\operatorname{TV}(\rho_ND_\tau,\Pi_\tau D_\tau)+\operatorname{TV}(\Pi_\tau D_\tau,p_\star^{A,y}) \\
&\leq\operatorname{TV}(\rho_N,\Pi_\tau)+\operatorname{TV}(\Pi_\tau D_\tau,p_\star^{A,y}). \tag*{(41)}
\end{aligned} TV ( ρ N D τ , p ⋆ A , y ) ≤ TV ( ρ N D τ , Π τ D τ ) + TV ( Π τ D τ , p ⋆ A , y ) ≤ TV ( ρ N , Π τ ) + TV ( Π τ D τ , p ⋆ A , y ) . (41)
Here, the first line follows the triangle inequality, and the second line follows the contraction of total variation under D τ D_\tau D τ . The two terms in the resulting bound (41) measure the sampler error relative to Π τ \Pi_\tau Π τ , and the discrepancy between the denoised terminal target Π τ D τ \Pi_\tau D_\tau Π τ D τ and the true posterior p ⋆ A , y p_\star^{A,y} p ⋆ A , y , respectively.
Define the event
G : = { ( A , x ⋆ ) : A ∈ A , ∥ x ⋆ ∥ 2 ≤ n } . (42) \mathcal{G}:=\left\{(A,x^\star):A\in\mathcal{A},\ \left\|x^\star\right\|^2\leq n\right\}. \tag*{(42)} G := { ( A , x ⋆ ) : A ∈ A , ∥ x ⋆ ∥ 2 ≤ n } . (42)
where A \mathcal{A} A is the matrix event of x ⋆ x^\star x ⋆ . We also use G \mathcal{G} G for the event ( A , X ⋆ ) ∈ G (A,X^\star)\in\mathcal{G} ( A , X ⋆ ) ∈ G , which depends on the sensing matrix and the signal but not on the observation noise. Since total variation is at most one, we can split (41) according to G \mathcal{G} G and bound
E TV ( ρ N D τ , p ⋆ A , y ) ≤ P ( G c ) + E [ 1 G TV ( ρ N , Π τ ) ] + E [ 1 G TV ( Π τ D τ , p ⋆ A , y ) ] . (43) \mathbb{E}\operatorname{TV}(\rho_ND_\tau,p_\star^{A,y})\leq\mathbb{P}(\mathcal{G}^{c})+\mathbb{E}\left[\mathbf{1}_{\mathcal{G}}\operatorname{TV}(\rho_N,\Pi_\tau)\right]+\mathbb{E}\left[\mathbf{1}_{\mathcal{G}}\operatorname{TV}(\Pi_\tau D_\tau,p_\star^{A,y})\right]. \tag*{(43)} E TV ( ρ N D τ , p ⋆ A , y ) ≤ P ( G c ) + E [ 1 G TV ( ρ N , Π τ ) ] + E [ 1 G TV ( Π τ D τ , p ⋆ A , y ) ] . (43)
By the sample size condition and (19) , it follows (cf. Proposition 3)
P ( G c ) ≤ 2 α . \mathbb{P}(\mathcal{G}^{c}) \le2\alpha. P ( G c ) ≤ 2 α .
The iteration bound of Theorem 2 gives N ≥ C 0 Q 18 N \ge C_0Q^{18} N ≥ C 0 Q 18 when its constant C C C is at least the constant C 0 C_0 C 0 of Lemma 1. For every fixed pair ( A , x ⋆ ) ∈ G (A,x^\star) \in\mathcal{G} ( A , x ⋆ ) ∈ G , Lemma 1 then gives, with y = A x ⋆ + σ ζ y=Ax^\star+\sigma\zeta y = A x ⋆ + σ ζ ,
E ζ TV ( ρ N , Π τ ) ≤ C τ + E s c ( A , x ⋆ ) . \mathbb{E}_{\zeta}\operatorname{TV}(\rho_N,\Pi_\tau) \le C\sqrt{\tau}+\mathcal{E}_{\mathrm{sc}}(A,x^\star). E ζ TV ( ρ N , Π τ ) ≤ C τ + E sc ( A , x ⋆ ) .
Averaging over ( A , X ⋆ ) (A,X^\star) ( A , X ⋆ ) , using Jensen’s inequality and E [ 1 G E s c ( A , X ⋆ ) ] ≤ E s c \mathbb{E}[\mathbf{1}_{\mathcal{G}}\mathcal{E}_{\mathrm{sc}}(A,X^\star)]\le\mathcal{E}_{\mathrm{sc}} E [ 1 G E sc ( A , X ⋆ )] ≤ E sc , yields
E [ 1 G TV ( ρ N , Π τ ) ] = E A , X ⋆ [ 1 G E ζ TV ( ρ N , Π τ ) ] ≤ C τ + C E s c ≤ C α + C E s c , \mathbb{E}[\mathbf{1}_{\mathcal{G}}\operatorname{TV}(\rho_N,\Pi_\tau)] = \mathbb{E}_{A,X^\star}\left[\mathbf{1}_{\mathcal{G}}\mathbb{E}_{\zeta}\operatorname{TV}(\rho_N,\Pi_\tau)\right] \le C\sqrt{\tau}+C\sqrt{\mathcal{E}_{\mathrm{sc}}}\le C\alpha+C\sqrt{\mathcal{E}_{\mathrm{sc}}}, E [ 1 G TV ( ρ N , Π τ )] = E A , X ⋆ [ 1 G E ζ TV ( ρ N , Π τ ) ] ≤ C τ + C E sc ≤ C α + C E sc ,
where the last inequality uses τ ≤ α 2 \tau\le\alpha^2 τ ≤ α 2 .
Finally, we bound the last term in (43) , which is controlled by the lemma below.
Lemma 3 (Comparison of terminal target). Let X ⋆ ∼ BG ( s , n ) X^\star\sim\operatorname{BG}(s,n) X ⋆ ∼ BG ( s , n ) , let A ∈ R m × n A\in\mathbb{R}^{m\times n} A ∈ R m × n , with m ≤ n m\le n m ≤ n , be independent of X ⋆ X^\star X ⋆ , and let y = A X ⋆ + σ ζ y=AX^\star+\sigma\zeta y = A X ⋆ + σ ζ . Then, for every τ > 0 \tau>0 τ > 0 ,
E [ 1 { ∥ A ∥ 2 ≤ 2 n } TV ( Π τ D τ , p ⋆ A , y ) ] ≤ 4 n 2 τ σ 2 ( 1 + s σ 2 ) . \mathbb{E}\left[\mathbf{1}_{\{\lVert A\rVert^2\le2n\}}\operatorname{TV}(\Pi_\tau\mathcal{D}_\tau,p_{\star}^{A,y})\right]\le\frac{4n^2\tau}{\sigma^2}\left(1+\frac{s}{\sigma^2}\right). E [ 1 {∥ A ∥ 2 ≤ 2 n } TV ( Π τ D τ , p ⋆ A , y ) ] ≤ σ 2 4 n 2 τ ( 1 + σ 2 s ) .
The proof of this lemma can be found in Appendix § 9 .§ 2 . Since the event G \mathcal{G} G implies ∥ A ∥ 2 ≤ 2 n \lVert A\rVert^2\le2n ∥ A ∥ 2 ≤ 2 n ,
E [ 1 G TV ( Π τ D τ , p ⋆ A , y ) ] ≤ E [ 1 { ∥ A ∥ 2 ≤ 2 n } TV ( Π τ D τ , p ⋆ A , y ) ] ≤ 4 n 2 τ σ 2 ( 1 + s σ 2 ) ≤ α , \mathbb{E}\left[\mathbf{1}_{\mathcal{G}}\operatorname{TV}(\Pi_\tau\mathcal{D}_\tau,p_{\star}^{A,y})\right]\le\mathbb{E}\left[\mathbf{1}_{\{\lVert A\rVert^2\le2n\}}\operatorname{TV}(\Pi_\tau\mathcal{D}_\tau,p_{\star}^{A,y})\right]\le\frac{4n^2\tau}{\sigma^2}\left(1+\frac{s}{\sigma^2}\right)\le\alpha, E [ 1 G TV ( Π τ D τ , p ⋆ A , y ) ] ≤ E [ 1 {∥ A ∥ 2 ≤ 2 n } TV ( Π τ D τ , p ⋆ A , y ) ] ≤ σ 2 4 n 2 τ ( 1 + σ 2 s ) ≤ α ,
where the last inequality follows from (19) for sufficiently small c ≤ 1 / 4 c\le1/4 c ≤ 1/4 . Substituting the three bounds into (43) gives
E TV ( L ( Z ^ ∣ A , y ) , p ⋆ A , y ) ≲ α + E s c . (44) \mathbb{E}\operatorname{TV}(\mathcal{L}(\widehat{Z}\mid A,y),p_{\star}^{A,y})\lesssim\alpha+\sqrt{\mathcal{E}_{\mathrm{sc}}}. \tag*{(44)} E TV ( L ( Z ∣ A , y ) , p ⋆ A , y ) ≲ α + E sc . (44)
which proves Theorem 2.
Discussion
We establish that the likelihood-guided annealed Langevin sampler (8) provides statistically and computationally efficient sparse recovery guarantees under a Bernoulli-Gaussian prior, when the prior information is provided implicitly via a diffusion model. This provides a first step toward a benign mathematical characterization of diffusion-based inverse solvers, when the prior model exhibits non-log-concavity. The analysis highlights several directions for future work.
Beyond sparse priors. The Bernoulli-Gaussian prior provides a fruitful test ground for establishing the finite-time performance of diffusion-based inverse solvers. However, many applications involve richer priors involving low-dimensional structures. Extending the result to broader classes of low-dimensional priors would help clarify the extent to which diffusion-based solvers adapt to intrinsic low-dimensional structures inherent in real-world data.
Other types of solvers. In practice, diffusion-based inverse problems are often solved using a wide range of tools, including predictor-corrector methods, proximal or projection-based updates, and plug-and-play samplers. Developing a unified theory that covers such algorithms would be extremely valuable.
Preliminaries
This appendix collects the general tools and the latent representation of the tilted targets used in the proofs.
Technical background
Probability distances. For the squared Hellinger distance defined in (21) , the next lemma collects some standard comparisons, whose proofs can be found in Polyanskiy and Wu [45 ] .
Lemma 4. (Polyanskiy and Wu [45 ] ) Let ν \nu ν be a σ \sigma σ -finite measure, and let p ( x , z ) , q ( x , z ) ≥ 0 p(x,z),q(x,z)\ge0 p ( x , z ) , q ( x , z ) ≥ 0 be functions such that
p ( x ) : = ∫ p ( x , z ) ν ( d z ) , q ( x ) : = ∫ q ( x , z ) ν ( d z ) p(x) := \int p(x,z)\,\nu(\mathrm{d}z), \qquad q(x) := \int q(x,z)\,\nu(\mathrm{d}z) p ( x ) := ∫ p ( x , z ) ν ( d z ) , q ( x ) := ∫ q ( x , z ) ν ( d z ) are probability densities.
The mixtures satisfy
H 2 ( p , q ) ≤ ∬ ( p ( x , z ) − q ( x , z ) ) 2 ν ( d z ) d x . (45) \mathrm{H}^{2}(p,q) \le\iint\left(\sqrt{p(x,z)}-\sqrt{q(x,z)}\right)^{2}\nu(\mathrm{d}z)\,\mathrm{d}x. \tag*{(45)} H 2 ( p , q ) ≤ ∬ ( p ( x , z ) − q ( x , z ) ) 2 ν ( d z ) d x . (45) If p ( x , z ) = w ( z ) p z ( x ) p(x,z)=w(z)p_z(x) p ( x , z ) = w ( z ) p z ( x ) and q ( x , z ) = w ( z ) q z ( x ) q(x,z)=w(z)q_z(x) q ( x , z ) = w ( z ) q z ( x ) for probability densities p z , q z p_z,q_z p z , q z and a common mixing density w w w , then the right side of (45) equals ∫ H 2 ( p z , q z ) w ( z ) ν ( d z ) \int\mathrm{H}^{2}(p_z,q_z)w(z)\nu(\mathrm{d}z) ∫ H 2 ( p z , q z ) w ( z ) ν ( d z ) .
For Σ ≻ 0 \Sigma\succ0 Σ ≻ 0 and vectors m 1 , m 2 m_1,m_2 m 1 , m 2 , with d Σ : = ( m 1 − m 2 ) ⊤ Σ − 1 ( m 1 − m 2 ) d_\Sigma:= (m_1-m_2)^\top\Sigma^{-1}(m_1-m_2) d Σ := ( m 1 − m 2 ) ⊤ Σ − 1 ( m 1 − m 2 ) ,
H 2 ( N ( m 1 , Σ ) , N ( m 2 , Σ ) ) = 2 − 2 e − d Σ / 8 ≤ d Σ / 4. (46) \mathrm{H}^{2}\left(\mathcal{N}(m_1,\Sigma),\mathcal{N}(m_2,\Sigma)\right)=2-2e^{-d_\Sigma/8}\le d_\Sigma/4. \tag*{(46)} H 2 ( N ( m 1 , Σ ) , N ( m 2 , Σ ) ) = 2 − 2 e − d Σ /8 ≤ d Σ /4. (46)
For μ ≪ ν \mu\ll\nu μ ≪ ν , the relative entropy and chi-square divergence are
KL ( μ ∥ ν ) : = ∫ log d μ d ν d μ , χ 2 ( μ ∥ ν ) : = ∫ ( d μ d ν − 1 ) 2 d ν . \operatorname{KL}(\mu\Vert\nu) := \int\log\frac{\mathrm{d}\mu}{\mathrm{d}\nu}\,\mathrm{d}\mu,\qquad\chi^2(\mu\Vert\nu) := \int\left(\frac{\mathrm{d}\mu}{\mathrm{d}\nu}-1\right)^2\,\mathrm{d}\nu. KL ( μ ∥ ν ) := ∫ log d ν d μ d μ , χ 2 ( μ ∥ ν ) := ∫ ( d ν d μ − 1 ) 2 d ν .
Both are defined to be + ∞ +\infty + ∞ when μ \mu μ is not absolutely continuous with respect to ν \nu ν . For probability measures μ 1 , μ 2 \mu_1,\mu_2 μ 1 , μ 2 with densities p , q p,q p , q with respect to a common dominating measure λ \lambda λ , the total-variation distance is defined as
TV ( μ 1 , μ 2 ) : = sup E ∣ μ 1 ( E ) − μ 2 ( E ) ∣ = 1 2 ∫ ∣ p − q ∣ d λ . \operatorname{TV}(\mu_1,\mu_2) := \sup_E |\mu_1(E)-\mu_2(E)| = \frac{1}{2}\int|p-q|\,\mathrm{d}\lambda. TV ( μ 1 , μ 2 ) := E sup ∣ μ 1 ( E ) − μ 2 ( E ) ∣ = 2 1 ∫ ∣ p − q ∣ d λ .
A Markov kernel does not increase total variation: TV ( μ K , ν K ) ≤ TV ( μ , ν ) \operatorname{TV}(\mu K,\nu K)\leq\operatorname{TV}(\mu,\nu) TV ( μ K , ν K ) ≤ TV ( μ , ν ) .
Poincaré constants. A probability μ \mu μ on R d \mathbb{R}^d R d has Poincaré constant at most K K K if
Var μ f ≤ K μ ∥ ∇ f ∥ 2 \operatorname{Var}_{\mu} f \leq K\mu\|\nabla f\|^2 Var μ f ≤ K μ ∥∇ f ∥ 2
for every smooth f f f with compact support. By standard approximation using cutoff and mollification, the above inequality also holds for every C 1 C^1 C 1 function f f f with μ f 2 < ∞ \mu f^2<\infty μ f 2 < ∞ and μ ∥ ∇ f ∥ 2 < ∞ \mu\|\nabla f\|^2<\infty μ ∥∇ f ∥ 2 < ∞ , in particular for bounded C 1 C^1 C 1 functions with bounded gradient.
The following lemma collects some useful standard properties of Poincaré inequalities; see Chafaï [16 ] for details. In particular, Item 5 is the strongly log-concave case of the Brascamp–Lieb inequality [8 ] , in the form of Bobkov and Ledoux [6 ] .
Lemma 5 (Properties for Poincaré constants). The following statements are true.
A product measure has Poincaré constant at most the maximum of those of its factors.
The image of a law under an affine map x ↦ T x + x 0 x\mapsto Tx+x_0 x ↦ T x + x 0 has Poincaré constant at most ∥ T ∥ 2 \|T\|^2 ∥ T ∥ 2 times that of the law.
The Gaussian N ( m , Σ ) \mathcal{N}(m,\Sigma) N ( m , Σ ) has Poincaré constant at most ∥ Σ ∥ \|\Sigma\| ∥Σ∥ , including for singular Σ \Sigma Σ .
The law of a sum of independent random vectors has Poincaré constant at most the sum of theirs.
If μ ( d x ) ∝ e − Ψ ( x ) d x \mu(\mathrm{d}x)\propto e^{-\Psi(x)}\mathrm{d}x μ ( d x ) ∝ e − Ψ ( x ) d x for a smooth Ψ \Psi Ψ with ∇ 2 Ψ ⪰ a I \nabla^2\Psi\succeq aI ∇ 2 Ψ ⪰ a I for some a > 0 a>0 a > 0 , then μ \mu μ has Poincaré constant at most 1 / a 1/a 1/ a .
Gaussian mixture representation of the tilted targets
To analyze the tilted targets Π t \Pi_t Π t , we express them as mixtures of Gaussian distributions indexed by the supports of an underlying Bernoulli–Gaussian vector. This representation separates the conditional Gaussian structure from the distribution over supports. It provides the common formulas used to establish the weak variance inequality in Appendix § 8 .§ 2 and the terminal guarantees in Appendix § 9 .
The auxiliary model. Introduce the auxiliary model
Z ∼ p ⋆ , X t = Z + t ξ , Y ~ t = A X t + a t ξ ′ , Z\sim p_\star,\qquad X_t=Z+\sqrt{t}\,\xi,\qquad\widetilde{Y}_t=AX_t+\sqrt{a_t}\,\xi', Z ∼ p ⋆ , X t = Z + t ξ , Y t = A X t + a t ξ ′ ,
where the latent vector Z Z Z , ξ ∼ N ( 0 , I n ) \xi\sim\mathcal{N}(0,I_n) ξ ∼ N ( 0 , I n ) and ξ ′ ∼ N ( 0 , I m ) \xi'\sim\mathcal{N}(0,I_m) ξ ′ ∼ N ( 0 , I m ) are independent. Since X t X_t X t has density p t p_t p t , Bayes’ formula gives
L ( X t ∣ Y ~ t = y ) ∝ p t ( x ) exp ( − ∥ A x − y ∥ 2 2 a t ) , \mathcal{L}(X_t\mid\widetilde{Y}_t=y)\propto p_t(x)\exp\left(-\frac{\|Ax-y\|^2}{2a_t}\right), L ( X t ∣ Y t = y ) ∝ p t ( x ) exp ( − 2 a t ∥ A x − y ∥ 2 ) ,
which is exactly Π t \Pi_t Π t in (5) . Writing the auxiliary observation Y ~ t \widetilde{Y}_t Y t in terms of Z Z Z gives
Y ~ t = A Z + w t , where w t ∼ N ( 0 , Σ t ) \widetilde{Y}_t=AZ+w_t,\qquad\text{where } w_t\sim\mathcal{N}(0,\Sigma_t) Y t = A Z + w t , where w t ∼ N ( 0 , Σ t )
independent of Z Z Z , with Σ t = a t I m + t A A T \Sigma_t=a_tI_m+tAA^{\mathsf T} Σ t = a t I m + t A A T in (31) . Define the latent law
P t A , y : = L ( Z ∣ Y ~ t = y ) , (47) \mathcal{P}_t^{A,y}:=\mathcal{L}(Z\mid\widetilde{Y}_t=y), \tag*{(47)} P t A , y := L ( Z ∣ Y t = y ) , (47)
and write P t \mathcal{P}_t P t when A A A and y y y are fixed.
The forward kernel. The conditional law that maps the latent vector Z Z Z to the target X t X_t X t is the forward kernel
F t A , y ( z , ⋅ ) : = L ( X t ∣ Z = z , Y ~ t = y ) , \mathcal{F}_t^{A,y}(z,\cdot):=\mathcal{L}(X_t\mid Z=z,\widetilde{Y}_t=y), F t A , y ( z , ⋅ ) := L ( X t ∣ Z = z , Y t = y ) ,
which we abbreviate as F t \mathcal{F}_t F t . Conditioning first on Z Z Z in the joint law of ( Z , X t ) (Z,X_t) ( Z , X t ) given Y ~ t = y \widetilde{Y}_t=y Y t = y gives
P t F t = Π t , (48) \mathcal{P}_t\mathcal{F}_t=\Pi_t, \tag*{(48)} P t F t = Π t , (48)
which expresses the target as the latent law P t \mathcal{P}_t P t passed through the forward kernel F t \mathcal{F}_t F t . By straightforward calculation, we have
F t A , y ( z , ⋅ ) = N ( M t z + t b t , t M t ) , (49) \mathcal{F}_t^{A,y}(z,\cdot)=\mathcal{N}(M_tz+tb_t,tM_t), \tag*{(49)} F t A , y ( z , ⋅ ) = N ( M t z + t b t , t M t ) , (49)
where
M t : = I n − t G t , b t : = A T Σ t − 1 y , with G t : = A T Σ t − 1 A . M_t:=I_n-tG_t,\qquad b_t:=A^\mathsf{T}\Sigma_t^{-1}y,\qquad\text{with}\qquad G_t:=A^\mathsf{T}\Sigma_t^{-1}A. M t := I n − t G t , b t := A T Σ t − 1 y , with G t := A T Σ t − 1 A .
Indeed, Σ t = a t I m + t A A T \Sigma_t=a_tI_m+tAA^\mathsf{T} Σ t = a t I m + t A A T gives M t = ( I n + ( t / a t ) A T A ) − 1 M_t=(I_n+(t/a_t)A^\mathsf{T}A)^{-1} M t = ( I n + ( t / a t ) A T A ) − 1 and M t A T = a t A T Σ t − 1 M_tA^\mathsf{T}=a_tA^\mathsf{T}\Sigma_t^{-1} M t A T = a t A T Σ t − 1 . Completing the square in x x x therefore gives covariance t M t tM_t t M t and mean M t z + t b t M_tz+tb_t M t z + t b t .
Latent supports and Gaussian components. For a fixed level t t t , it follows from Bayes’ formula that
P t ( d z ) ∝ e − ( y − A z ) T Σ t − 1 ( y − A z ) / 2 p ⋆ ( d z ) ∝ e ( b t ) T z − z T G t z / 2 p ⋆ ( d z ) . (50) \mathcal{P}_t(dz)\propto e^{-(y-Az)^\mathsf{T}\Sigma_t^{-1}(y-Az)/2}p_\star(dz)\propto e^{(b_t)^\mathsf{T}z-z^\mathsf{T}G_tz/2}p_\star(dz). \tag*{(50)} P t ( d z ) ∝ e − ( y − A z ) T Σ t − 1 ( y − A z ) /2 p ⋆ ( d z ) ∝ e ( b t ) T z − z T G t z /2 p ⋆ ( d z ) . (50)
Define the latent support S t : = supp z S_t:=\operatorname{supp}z S t := supp z under P t \mathcal{P}_t P t . For W ⊆ [ n ] W\subseteq[n] W ⊆ [ n ] , write P t W : = P t ( ⋅ ∣ S t = W ) \mathcal{P}_t^W:=\mathcal{P}_t(\cdot\mid S_t=W) P t W := P t ( ⋅ ∣ S t = W ) . Combining (48) with the support decomposition gives
Π t = ∑ W ⊆ [ n ] P t { S t = W } P t W F t . (51) \Pi_t=\sum_{W\subseteq[n]}\mathcal{P}_t\{S_t=W\}\mathcal{P}_t^W\mathcal{F}_t. \tag*{(51)} Π t = W ⊆ [ n ] ∑ P t { S t = W } P t W F t . (51)
Let B W W : = ( B t ) W W B_{WW}:=(B_t)_{WW} B W W := ( B t ) W W , G W W : = ( G t ) W W G_{WW}:=(G_t)_{WW} G W W := ( G t ) W W and b W : = ( b t ) W b_W:=(b_t)_W b W := ( b t ) W , and ω : = s / ( n − s ) \omega:=s/(n-s) ω := s / ( n − s ) be the prior odds of a coordinate, then by some calculation, we have P t { S t = W } = w W / ∑ V ⊆ [ n ] w V > 0 \mathcal{P}_t\{S_t=W\}=w_W/\sum_{V\subseteq[n]}w_V>0 P t { S t = W } = w W / ∑ V ⊆ [ n ] w V > 0 with
w W : = ω ∣ W ∣ det ( B W W ) − 1 / 2 e ( b W ) T B W W − 1 b W / 2 , w ∅ = 1 , (52) w_W:=\omega^{|W|}\det(B_{WW})^{-1/2}e^{(b_W)^\mathsf{T}B_{WW}^{-1}b_W/2},\qquad w_\varnothing=1, \tag*{(52)} w W := ω ∣ W ∣ det ( B W W ) − 1/2 e ( b W ) T B W W − 1 b W /2 , w ∅ = 1 , (52)
and, under P t W \mathcal{P}_t^W P t W ,
z W ∼ N ( B W W − 1 b W , B W W − 1 ) , z W c = 0. (53) z_W\sim\mathcal{N}(B_{WW}^{-1}b_W,B_{WW}^{-1}),\qquad z_{W^c}=0. \tag*{(53)} z W ∼ N ( B W W − 1 b W , B W W − 1 ) , z W c = 0. (53)
Therefore, each component P t W F t \mathcal{P}_t^W\mathcal{F}_t P t W F t in (51) is Gaussian by (53) and the affine Gaussian formula (49) . This representation permits separate control of the support probabilities and of the distributions within the components.
Annealed transfer inequalities
This appendix first proves Proposition 1 in Section B.1, then derives the weak variance inequality from the matrix condition in Section B.2, and finally prove Lemma 1 in Section B.3.
Proof of Proposition 1
We now begin to prove Proposition 1. We will first compare each update with an invariant kernel that preserves the current target. We then construct the invariant kernels and the variance they remove, and finally bound their discrepancy from the sampler updates.
Step 1: general transfer inequality. For a Markov kernel K K K , define
Var K f ( x ) : = K ( f 2 ) ( x ) − ( K f ( x ) ) 2 , \operatorname{Var}^{K} f(x) := K(f^{2})(x) - (Kf(x))^{2}, Var K f ( x ) := K ( f 2 ) ( x ) − ( K f ( x ) ) 2 ,
which is the variance of f f f after one transition from x x x . The following lemma compares the sampler update with comparison kernels that are invariant to the current target at each step.
Lemma 6 (Kernels with invariant measures). Let Π t 0 , … , Π t N \Pi_{t_0},\ldots,\Pi_{t_N} Π t 0 , … , Π t N and μ \mu μ be probability measures on R n \mathbb{R}^{n} R n , and for j < N j<N j < N let P j P_j P j be Markov kernels on R n \mathbb{R}^{n} R n with Π t j P j = Π t j \Pi_{t_j}P_j=\Pi_{t_j} Π t j P j = Π t j . Suppose that for some β ≥ 1 \beta\ge1 β ≥ 1 and δ j ≥ 0 \delta_j\ge0 δ j ≥ 0 , every j < N j<N j < N and every g : R n → [ 0 , 1 ] g:\mathbb{R}^{n}\to[0,1] g : R n → [ 0 , 1 ] satisfy
Var Π t j g ≤ β Π t j Var P j g + δ j . (54) \operatorname{Var}_{\Pi_{t_j}} g \le\beta\Pi_{t_j}\operatorname{Var}^{P_j}g+\delta_j. \tag*{(54)} Var Π t j g ≤ β Π t j Var P j g + δ j . (54) Set
e j : = ∫ H 2 ( P j ( x , ⋅ ) , K ^ j ( x , ⋅ ) ) Π t j ( d x ) , ρ N : = μ K ^ 0 ⋯ K ^ N − 1 . e_j:=\int H^{2}\left(P_j(x,\cdot),\widehat{K}_j(x,\cdot)\right)\Pi_{t_j}(dx),\qquad\rho_N:=\mu\widehat{K}_0\cdots\widehat{K}_{N-1}. e j := ∫ H 2 ( P j ( x , ⋅ ) , K j ( x , ⋅ ) ) Π t j ( d x ) , ρ N := μ K 0 ⋯ K N − 1 . Then every g : R n → [ 0 , 1 ] g:\mathbb{R}^{n}\to[0,1] g : R n → [ 0 , 1 ] and every 0 < λ ≤ 1 0<\lambda\le1 0 < λ ≤ 1 satisfy
ρ N g ≤ ( 1 + λ ) Π t N g + C λ ( H 2 ( Π t 0 , μ ) + ∑ j < N [ e j + β H 2 ( Π t j , Π t j + 1 ) ] ) + C ∑ j < N δ j H 2 ( Π t j , Π t j + 1 ) . (55) \rho_Ng\le(1+\lambda)\Pi_{t_N}g+\frac{C}{\lambda}\left(H^{2}(\Pi_{t_0},\mu)+\sum_{j<N}\left[e_j+\beta H^{2}(\Pi_{t_j},\Pi_{t_{j+1}})\right]\right)+C\sum_{j<N}\sqrt{\delta_jH^{2}(\Pi_{t_j},\Pi_{t_{j+1}})}. \tag*{(55)} ρ N g ≤ ( 1 + λ ) Π t N g + λ C H 2 ( Π t 0 , μ ) + j < N ∑ [ e j + β H 2 ( Π t j , Π t j + 1 ) ] + C j < N ∑ δ j H 2 ( Π t j , Π t j + 1 ) . (55)
The proof is given in Appendix § 8 .§ 1.1 . Lemma 6 reduces Proposition 1 to constructing invariant kernels satisfying (50) with β = 1 + V / h \beta=1+V/h β = 1 + V / h and establishing e j ≤ h Π t j ∥ Δ j ∥ 2 + 2 n L 2 h 2 e_j\le h\Pi_{t_j}\|\Delta_j\|^{2}+2nL^{2}h^{2} e j ≤ h Π t j ∥ Δ j ∥ 2 + 2 n L 2 h 2 . These two estimates are supplied in the next two steps.
Step 2: comparison kernel and variance removal. We construct the comparison kernels for Lemma 6, first establish their invariance, and then identify the variance they remove. Fix j < N j<N j < N , write φ h \varphi_h φ h for the density of N ( 0 , h I n ) \mathcal{N}(0,hI_n) N ( 0 , h I n ) , and define
r u ( d x ′ ) : = φ h ( u − x ′ ) Π t j ( d x ′ ) ∫ φ h ( u − x ′ ′ ) Π t j ( d x ′ ′ ) . r_u(dx'):=\frac{\varphi_h(u-x')\Pi_{t_j}(dx')}{\int\varphi_h(u-x'')\Pi_{t_j}(dx'')}. r u ( d x ′ ) := ∫ φ h ( u − x ′′ ) Π t j ( d x ′′ ) φ h ( u − x ′ ) Π t j ( d x ′ ) .
The comparison kernel P j ( x , ⋅ ) P_j(x,\cdot) P j ( x , ⋅ ) is defined as the law of X ′ X' X ′ obtained by drawing X ~ ∼ N ( x , h I n ) \widetilde{X}\sim\mathcal{N}(x,hI_n) X ∼ N ( x , h I n ) and then X ′ ∼ r X ~ X'\sim r_{\widetilde{X}} X ′ ∼ r X . This is one step of the proximal sampler [40 , 18 ] . To establish invariance, let X ∼ Π t j X\sim\Pi_{t_j} X ∼ Π t j , and X ~ : = X + h ξ \widetilde{X}:=X+\sqrt{h}\,\xi X := X + h ξ with ξ \xi ξ standard Gaussian and independent of X X X , and draw X ′ ∼ r X ~ X'\sim r_{\widetilde{X}} X ′ ∼ r X given ( X , X ~ ) (X,\widetilde{X}) ( X , X ) . By Bayes’ formula, r X ~ r_{\widetilde{X}} r X is the conditional law of X X X given X ~ \widetilde{X} X . Thus X X X and X ′ X' X ′ are conditionally independent and identically distributed given X ~ \widetilde{X} X , and in particular have the same marginal law Π t j \Pi_{t_j} Π t j . Since X ′ X' X ′ given X X X has law P j ( X , ⋅ ) P_j(X,\cdot) P j ( X , ⋅ ) , we obtain
Π t j P j = L ( X ′ ) = Π t j . \Pi_{t_j}P_j=\mathcal{L}(X')=\Pi_{t_j}. Π t j P j = L ( X ′ ) = Π t j .
Next, fix g : R n → [ 0 , 1 ] g:\mathbb{R}^{n}\to[0,1] g : R n → [ 0 , 1 ] . Since E [ g ( X ′ ) ∣ X ] = P j g ( X ) \mathbb{E}[g(X')\mid X]=P_jg(X) E [ g ( X ′ ) ∣ X ] = P j g ( X ) and X , X ′ ∼ Π t j X,X'\sim\Pi_{t_j} X , X ′ ∼ Π t j , the law of total variance gives
Π t j Var P j g = E Var ( g ( X ′ ) ∣ X ) = Var Π t j g − Var Π t j ( P j g ) . (56) \Pi_{t_j}\operatorname{Var}^{P_j}g=\mathbb{E}\operatorname{Var}(g(X')\mid X)=\operatorname{Var}_{\Pi_{t_j}}g-\operatorname{Var}_{\Pi_{t_j}}(P_jg). \tag*{(56)} Π t j Var P j g = E Var ( g ( X ′ ) ∣ X ) = Var Π t j g − Var Π t j ( P j g ) . (56)
The next lemma verifies the hypothesis (50) of Lemma 6 for the comparison kernels, with β = 1 + V / h \beta=1+V/h β = 1 + V / h . The proof of this lemma can be found in Appendix § 8 .§ 1.2 .
Lemma 7 (Variance removed by the comparison kernel). Under the hypotheses of Proposition 1, for j < N j<N j < N , with P j P_j P j the comparison kernel of Π t j \Pi_{t_j} Π t j , every g : R n → [ 0 , 1 ] g:\mathbb{R}^{n}\to[0,1] g : R n → [ 0 , 1 ] satisfies
Var Π t j g ≤ ( 1 + V h ) Π t j Var P j g + δ j . \operatorname{Var}_{\Pi_{t_j}}g\le\left(1+\frac{V}{h}\right)\Pi_{t_j}\operatorname{Var}^{P_j}g+\delta_j. Var Π t j g ≤ ( 1 + h V ) Π t j Var P j g + δ j .
Lemma 7 corresponds to the Poincaré-based analysis in [18 Theorem 4] , with the weak variance inequality (25) in place of a Poincaré inequality.
Step 3: bound the discrepancy error and conclusion. It remains to control the discrepancy between the comparison kernel and the sampler update. The next lemma bounds the term e j e_j e j of Lemma 6 by the score and discretization terms, whose proof can be found in Appendix § 8 .§ 1.3 .
Lemma 8 (Comparison with the update). Under the hypotheses of Proposition 1, for j < N j<N j < N , the comparison kernel P j P_j P j of Π t j \Pi_{t_j} Π t j and the update K ^ j \widehat{K}_j K j satisfy
∫ H 2 ( P j ( x , ⋅ ) , K ^ j ( x , ⋅ ) ) Π t j ( d x ) ≤ h Π t j ∥ Δ j ∥ 2 + 2 n L 2 h 2 . (57) \int H^2(P_j(x,\cdot),\widehat{K}_j(x,\cdot))\,\Pi_{t_j}(\mathrm{d}x)\le h\Pi_{t_j}\lVert\Delta_j\rVert^2+2nL^2h^2. \tag*{(57)} ∫ H 2 ( P j ( x , ⋅ ) , K j ( x , ⋅ )) Π t j ( d x ) ≤ h Π t j ∥ Δ j ∥ 2 + 2 n L 2 h 2 . (57)
We are ready to apply Lemma 6 with μ = μ i n i t \mu=\mu_{\mathrm{init}} μ = μ init and β = 1 + V / h ≥ 1 \beta=1+V/h\ge1 β = 1 + V / h ≥ 1 , by Lemma 7. Summing the bounds of Lemma 8 gives
∑ j < N e j ≤ ∑ j < N h Π t j ∥ Δ j ∥ 2 + 2 n L 2 N h 2 . \sum_{j<N}e_j\le\sum_{j<N}h\Pi_{t_j}\lVert\Delta_j\rVert^2+2nL^2Nh^2. j < N ∑ e j ≤ j < N ∑ h Π t j ∥ Δ j ∥ 2 + 2 n L 2 N h 2 .
Substituting this bound and β = 1 + V / h \beta=1+V/h β = 1 + V / h in (55) gives (26) with the errors defined in (27). This proves Proposition 1.
Proof of Lemma 6
For 0 < λ ≤ 1 0<\lambda\le1 0 < λ ≤ 1 , let Φ λ ( u ) : = u + λ u 2 \Phi_\lambda(u):=u+\lambda u^2 Φ λ ( u ) := u + λ u 2 for u ∈ R u\in\mathbb{R} u ∈ R . We use the following lemma, whose proof is given below.
Lemma 9 (Square-root inequalities). Let μ 1 , μ 2 \mu_1,\mu_2 μ 1 , μ 2 be probability measures on R n \mathbb{R}^n R n and g g g a bounded function. Then
∣ μ 2 g − μ 1 g ∣ ≤ 2 Var μ 1 g ⋅ H 2 ( μ 1 , μ 2 ) + osc ( g ) H 2 ( μ 1 , μ 2 ) . (58) \lvert\mu_2g-\mu_1g\rvert\le2\sqrt{\operatorname{Var}_{\mu_1}g\cdot H^2(\mu_1,\mu_2)}+\operatorname{osc}(g)H^2(\mu_1,\mu_2). \tag*{(58)} ∣ μ 2 g − μ 1 g ∣ ≤ 2 Var μ 1 g ⋅ H 2 ( μ 1 , μ 2 ) + osc ( g ) H 2 ( μ 1 , μ 2 ) . (58)
If moreover 0 ≤ g ≤ 1 0\le g\le1 0 ≤ g ≤ 1 and 0 < λ ≤ 1 0<\lambda\le1 0 < λ ≤ 1 , then
Φ λ ( μ 2 g ) ≤ μ 1 Φ λ ( g ) − λ 2 Var μ 1 g + 21 λ H 2 ( μ 1 , μ 2 ) , (59) \Phi_\lambda(\mu_2g)\le\mu_1\Phi_\lambda(g)-\frac{\lambda}{2}\operatorname{Var}_{\mu_1}g+\frac{21}{\lambda}H^2(\mu_1,\mu_2), \tag*{(59)} Φ λ ( μ 2 g ) ≤ μ 1 Φ λ ( g ) − 2 λ Var μ 1 g + λ 21 H 2 ( μ 1 , μ 2 ) , (59)
μ 2 g ≤ μ 1 Φ λ ( g ) + 2 λ H 2 ( μ 1 , μ 2 ) . (60) \mu_2g\le\mu_1\Phi_\lambda(g)+\frac{2}{\lambda}H^2(\mu_1,\mu_2). \tag*{(60)} μ 2 g ≤ μ 1 Φ λ ( g ) + λ 2 H 2 ( μ 1 , μ 2 ) . (60)
Fix g : R n → [ 0 , 1 ] g:\mathbb{R}^n\to[0,1] g : R n → [ 0 , 1 ] and 0 < λ < 1 0<\lambda<1 0 < λ < 1 , and define g N : = g g_N:=g g N := g and g j : = K ^ j g j + 1 g_j:=\widehat{K}_jg_{j+1} g j := K j g j + 1 . Then 0 ≤ g j ≤ 1 0\le g_j\le1 0 ≤ g j ≤ 1 and ρ N g = μ g 0 \rho_Ng=\mu g_0 ρ N g = μ g 0 . For each j < N j<N j < N , apply (59) to g j + 1 g_{j+1} g j + 1 with μ 1 = P j ( x , ⋅ ) \mu_1=P_j(x,\cdot) μ 1 = P j ( x , ⋅ ) and μ 2 = K ^ j ( x , ⋅ ) \mu_2=\widehat{K}_j(x,\cdot) μ 2 = K j ( x , ⋅ ) . Integrating against Π t j \Pi_{t_j} Π t j and using Π t j P j = Π t j \Pi_{t_j}P_j=\Pi_{t_j} Π t j P j = Π t j gives
Π t j Φ λ ( g j ) ≤ Π t j Φ λ ( g j + 1 ) − λ 2 Π t j Var P j g j + 1 + 21 λ e j . \Pi_{t_j}\Phi_\lambda(g_j)\le\Pi_{t_j}\Phi_\lambda(g_{j+1})-\frac{\lambda}{2}\Pi_{t_j}\operatorname{Var}^{P_j}g_{j+1}+\frac{21}{\lambda}e_j. Π t j Φ λ ( g j ) ≤ Π t j Φ λ ( g j + 1 ) − 2 λ Π t j Var P j g j + 1 + λ 21 e j .
Since Φ λ \Phi_\lambda Φ λ maps [ 0 , 1 ] [0,1] [ 0 , 1 ] into [ 0 , 1 + λ ] [0,1+\lambda] [ 0 , 1 + λ ] and is ( 1 + 2 λ ) (1+2\lambda) ( 1 + 2 λ ) -Lipschitz, we have osc Φ λ ( g j + 1 ) ≤ 1 + λ \operatorname{osc}\Phi_\lambda(g_{j+1})\le1+\lambda osc Φ λ ( g j + 1 ) ≤ 1 + λ and Var Π t j Φ λ ( g j + 1 ) ≤ ( 1 + 2 λ ) 2 Var Π t j g j + 1 \operatorname{Var}_{\Pi_{t_j}}\Phi_\lambda(g_{j+1})\le(1+2\lambda)^2\operatorname{Var}_{\Pi_{t_j}}g_{j+1} Var Π t j Φ λ ( g j + 1 ) ≤ ( 1 + 2 λ ) 2 Var Π t j g j + 1 . Applying (58) to Φ λ ( g j + 1 ) \Phi_\lambda(g_{j+1}) Φ λ ( g j + 1 ) yields
Π t j Φ λ ( g j + 1 ) − Π t j + 1 Φ λ ( g j + 1 ) ≤ ( 2 + 4 λ ) Var Π t j g j + 1 ⋅ H 2 ( Π t j , Π t j + 1 ) + ( 1 + λ ) H 2 ( Π t j , Π t j + 1 ) ≤ λ 2 Π t j Var P j g j + 1 + 20 β λ H 2 ( Π t j , Π t j + 1 ) + 6 δ j H 2 ( Π t j , Π t j + 1 ) . \begin{aligned}
\Pi_{t_j}\Phi_\lambda(g_{j+1})-\Pi_{t_{j+1}}\Phi_\lambda(g_{j+1})
&\le(2+4\lambda)\sqrt{\operatorname{Var}_{\Pi_{t_j}}g_{j+1}\cdot H^2(\Pi_{t_j},\Pi_{t_{j+1}})}\\
&\quad+(1+\lambda)H^2(\Pi_{t_j},\Pi_{t_{j+1}})\\
&\le\frac{\lambda}{2}\Pi_{t_j}\operatorname{Var}^{P_j}g_{j+1}
+\frac{20\beta}{\lambda}H^2(\Pi_{t_j},\Pi_{t_{j+1}})
+6\sqrt{\delta_j}H^2(\Pi_{t_j},\Pi_{t_{j+1}}).
\end{aligned} Π t j Φ λ ( g j + 1 ) − Π t j + 1 Φ λ ( g j + 1 ) ≤ ( 2 + 4 λ ) Var Π t j g j + 1 ⋅ H 2 ( Π t j , Π t j + 1 ) + ( 1 + λ ) H 2 ( Π t j , Π t j + 1 ) ≤ 2 λ Π t j Var P j g j + 1 + λ 20 β H 2 ( Π t j , Π t j + 1 ) + 6 δ j H 2 ( Π t j , Π t j + 1 ) .
where the last inequality uses (54) , a + b ≤ a + b \sqrt{a+b}\le\sqrt{a}+\sqrt{b} a + b ≤ a + b , and Young’s inequality, with 2 ( 1 + 2 λ ) 2 + λ ( 1 + λ ) / β ≤ 20 2(1+2\lambda)^2+\lambda(1+\lambda)/\beta\le20 2 ( 1 + 2 λ ) 2 + λ ( 1 + λ ) / β ≤ 20 and 2 + 4 λ ≤ 6 2+4\lambda\le6 2 + 4 λ ≤ 6 for 0 < λ ≤ 1 ≤ β 0<\lambda\le1\le\beta 0 < λ ≤ 1 ≤ β . Combining the two displays cancels the variance terms and gives
Π t j Φ λ ( g j ) ≤ Π t j + 1 Φ λ ( g j + 1 ) + 21 λ ( e j + β H 2 ( Π t j , Π t j + 1 ) ) + 6 δ j H 2 ( Π t j , Π t j + 1 ) . (61) \Pi_{t_j}\Phi_\lambda(g_j)\le\Pi_{t_{j+1}}\Phi_\lambda(g_{j+1})+\frac{21}{\lambda}\left(e_j+\beta H^2(\Pi_{t_j},\Pi_{t_{j+1}})\right)+6\sqrt{\delta_j}H^2(\Pi_{t_j},\Pi_{t_{j+1}}). \tag*{(61)} Π t j Φ λ ( g j ) ≤ Π t j + 1 Φ λ ( g j + 1 ) + λ 21 ( e j + β H 2 ( Π t j , Π t j + 1 ) ) + 6 δ j H 2 ( Π t j , Π t j + 1 ) . (61)
Finally, applying (60) to g 0 g_0 g 0 , with μ 1 = Π t 0 \mu_1=\Pi_{t_0} μ 1 = Π t 0 and μ 2 = μ \mu_2=\mu μ 2 = μ , and summing (61) over j < N j<N j < N , we obtain
ρ N g = μ g 0 ≤ Π t 0 Φ λ ( g 0 ) + 2 λ H 2 ( Π t 0 , μ ) ≤ Π t N Φ λ ( g ) + 2 λ H 2 ( Π t 0 , μ ) + 21 λ ∑ j < N ( e j + β H 2 ( Π t j , Π t j + 1 ) ) + 6 ∑ j < N δ j H 2 ( Π t j , Π t j + 1 ) . \begin{aligned}
\rho_Ng=\mu g_0
&\le\Pi_{t_0}\Phi_\lambda(g_0)+\frac{2}{\lambda}H^2(\Pi_{t_0},\mu)\\
&\le\Pi_{t_N}\Phi_\lambda(g)+\frac{2}{\lambda}H^2(\Pi_{t_0},\mu)
+\frac{21}{\lambda}\sum_{j<N}\left(e_j+\beta H^2(\Pi_{t_j},\Pi_{t_{j+1}})\right)\\
&\quad+6\sum_{j<N}\sqrt{\delta_j}H^2(\Pi_{t_j},\Pi_{t_{j+1}}).
\end{aligned} ρ N g = μ g 0 ≤ Π t 0 Φ λ ( g 0 ) + λ 2 H 2 ( Π t 0 , μ ) ≤ Π t N Φ λ ( g ) + λ 2 H 2 ( Π t 0 , μ ) + λ 21 j < N ∑ ( e j + β H 2 ( Π t j , Π t j + 1 ) ) + 6 j < N ∑ δ j H 2 ( Π t j , Π t j + 1 ) .
Using Φ λ ( g ) ≤ ( 1 + λ ) g \Phi_\lambda(g)\le(1+\lambda)g Φ λ ( g ) ≤ ( 1 + λ ) g proves (55) with $C=21.
Proof of Lemma 9. Since both sides of (58) are unchanged when a constant is added to g g g , we assume μ 1 g = 0 \mu_1 g=0 μ 1 g = 0 . It then follows that ∣ g ∣ ≤ osc ( g ) |g|\leq\operatorname{osc}(g) ∣ g ∣ ≤ osc ( g ) and μ 1 g 2 = Var μ 1 g \mu_1g^2=\operatorname{Var}_{\mu_1}g μ 1 g 2 = Var μ 1 g . With p , q p,q p , q the densities of μ 1 , μ 2 \mu_1,\mu_2 μ 1 , μ 2 with respect to a common dominating measure,
μ 2 g − μ 1 g = ∫ g ⋅ 2 p ( q − p ) + ∫ g ( q − p ) 2 \mu_2g-\mu_1g=\int g\cdot2\sqrt{p}(\sqrt{q}-\sqrt{p})+\int g(\sqrt{q}-\sqrt{p})^2 μ 2 g − μ 1 g = ∫ g ⋅ 2 p ( q − p ) + ∫ g ( q − p ) 2
≤ 2 Var μ 1 g ⋅ H 2 ( μ 1 , μ 2 ) + osc ( g ) H 2 ( μ 1 , μ 2 ) , \leq2\sqrt{\operatorname{Var}_{\mu_1}g\cdot\mathrm{H}^2(\mu_1,\mu_2)}+\operatorname{osc}(g)\mathrm{H}^2(\mu_1,\mu_2), ≤ 2 Var μ 1 g ⋅ H 2 ( μ 1 , μ 2 ) + osc ( g ) H 2 ( μ 1 , μ 2 ) ,
where the second line follows from Cauchy–Schwarz and the definitions of squared Hellinger distance in (21) . Repeating the same bound for − g -g − g gives (58) .
Since the function Φ λ \Phi_\lambda Φ λ is increasing and ( 1 + 2 λ ) (1+2\lambda) ( 1 + 2 λ ) -Lipschitz on [ 0 , 1 ] [0,1] [ 0 , 1 ] , and μ 1 Φ λ ( g ) = Φ λ ( μ 1 g ) + λ Var μ 1 g \mu_1\Phi_\lambda(g)=\Phi_\lambda(\mu_1g)+\lambda\operatorname{Var}_{\mu_1}g μ 1 Φ λ ( g ) = Φ λ ( μ 1 g ) + λ Var μ 1 g , with (58) and osc ( g ) ≤ 1 \operatorname{osc}(g)\leq1 osc ( g ) ≤ 1 ,
Φ λ ( μ 2 g ) ≤ Φ λ ( μ 1 g ) + ( 1 + 2 λ ) ∣ μ 2 g − μ 1 g ∣ ≤ μ 1 Φ λ ( g ) − λ Var μ 1 g + ( 2 + 4 λ ) Var μ 1 g ⋅ H 2 ( μ 1 , μ 2 ) + ( 1 + 2 λ ) H 2 ( μ 1 , μ 2 ) ≤ μ 1 Φ λ ( g ) − λ 2 Var μ 1 g + 21 λ H 2 ( μ 1 , μ 2 ) , \begin{aligned}
\Phi_\lambda(\mu_2g)\leq\Phi_\lambda(\mu_1g)+(1+2\lambda)|\mu_2g-\mu_1g| \\
&\leq\mu_1\Phi_\lambda(g)-\lambda\operatorname{Var}_{\mu_1}g+(2+4\lambda)\sqrt{\operatorname{Var}_{\mu_1}g\cdot\mathrm{H}^2(\mu_1,\mu_2)}+(1+2\lambda)\mathrm{H}^2(\mu_1,\mu_2) \\
&\leq\mu_1\Phi_\lambda(g)-\frac{\lambda}{2}\operatorname{Var}_{\mu_1}g+\frac{21}{\lambda}\mathrm{H}^2(\mu_1,\mu_2),
\end{aligned} Φ λ ( μ 2 g ) ≤ Φ λ ( μ 1 g ) + ( 1 + 2 λ ) ∣ μ 2 g − μ 1 g ∣ ≤ μ 1 Φ λ ( g ) − λ Var μ 1 g + ( 2 + 4 λ ) Var μ 1 g ⋅ H 2 ( μ 1 , μ 2 ) + ( 1 + 2 λ ) H 2 ( μ 1 , μ 2 ) ≤ μ 1 Φ λ ( g ) − 2 λ Var μ 1 g + λ 21 H 2 ( μ 1 , μ 2 ) ,
where the last inequality uses 2 a b ≤ a 2 + b 2 2ab\leq a^2+b^2 2 ab ≤ a 2 + b 2 and 2 ( 1 + 2 λ ) 2 + λ ( 1 + 2 λ ) ≤ 21 2(1+2\lambda)^2+\lambda(1+2\lambda)\leq21 2 ( 1 + 2 λ ) 2 + λ ( 1 + 2 λ ) ≤ 21 for 0 < λ ≤ 1 0<\lambda\leq1 0 < λ ≤ 1 . This proves (59) .
Turning to (60) , with (58) and 2 Var μ 1 g ⋅ H 2 ( μ 1 , μ 2 ) ≤ λ Var μ 1 g + λ − 1 H 2 ( μ 1 , μ 2 ) 2\sqrt{\operatorname{Var}_{\mu_1}g\cdot\mathrm{H}^2(\mu_1,\mu_2)}\leq\lambda\operatorname{Var}_{\mu_1}g+\lambda^{-1}\mathrm{H}^2(\mu_1,\mu_2) 2 Var μ 1 g ⋅ H 2 ( μ 1 , μ 2 ) ≤ λ Var μ 1 g + λ − 1 H 2 ( μ 1 , μ 2 ) ,
μ 2 g ≤ μ 1 g + λ Var μ 1 g + ( 1 + λ − 1 ) H 2 ( μ 1 , μ 2 ) ≤ μ 1 Φ λ ( g ) + 2 λ H 2 ( μ 1 , μ 2 ) , \mu_2g\leq\mu_1g+\lambda\operatorname{Var}_{\mu_1}g+(1+\lambda^{-1})\mathrm{H}^2(\mu_1,\mu_2)\leq\mu_1\Phi_\lambda(g)+\frac{2}{\lambda}\mathrm{H}^2(\mu_1,\mu_2), μ 2 g ≤ μ 1 g + λ Var μ 1 g + ( 1 + λ − 1 ) H 2 ( μ 1 , μ 2 ) ≤ μ 1 Φ λ ( g ) + λ 2 H 2 ( μ 1 , μ 2 ) ,
since μ 1 Φ λ ( g ) = μ 1 g + λ μ 1 g 2 ≥ μ 1 g + λ Var μ 1 g \mu_1\Phi_\lambda(g)=\mu_1g+\lambda\mu_1g^2\geq\mu_1g+\lambda\operatorname{Var}_{\mu_1}g μ 1 Φ λ ( g ) = μ 1 g + λ μ 1 g 2 ≥ μ 1 g + λ Var μ 1 g and 1 ≤ 1 / λ 1\leq1/\lambda 1 ≤ 1/ λ . □
Proof of Lemma 7
Since P j g P_jg P j g is a Gaussian convolution of a [ 0 , 1 ] [0,1] [ 0 , 1 ] -valued function, it is smooth with bounded gradient and values in [ 0 , 1 ] [0,1] [ 0 , 1 ] . Applying the weak variance inequality (25) to P j g P_jg P j g , with osc ( P j g ) ≤ 1 \operatorname{osc}(P_jg)\leq1 osc ( P j g ) ≤ 1 , gives
Var Π t j ( P j g ) ≤ V Π t j ∥ ∇ P j g ∥ 2 + δ j . \operatorname{Var}_{\Pi_{t_j}}(P_jg)\leq V\Pi_{t_j}\|\nabla P_jg\|^2+\delta_j. Var Π t j ( P j g ) ≤ V Π t j ∥∇ P j g ∥ 2 + δ j .
It remains to bound the gradient term by the conditional variance Var P j g \operatorname{Var}^{P_j}g Var P j g . Let X , X ~ , X ′ X,\widetilde{X},X' X , X , X ′ be constructed as in Step 2. For any unit vector e e e , differentiating the Gaussian density gives
e ⊤ ∇ P j g ( x ) = h − 1 ∫ e ⊤ ( u − x ) ϕ h ( u − x ) r u g d u = h − 1 E [ e ⊤ ( X ~ − x ) { g ( X ′ ) − P j g ( x ) } ∣ X = x ] . \begin{aligned}
e^\top\nabla P_jg(x)=h^{-1}\int e^\top(u-x)\phi_h(u-x)r_ug\,du \\
&=h^{-1}\mathbb{E}\left[e^\top(\widetilde{X}-x)\{g(X')-P_jg(x)\}\mid X=x\right].
\end{aligned} e ⊤ ∇ P j g ( x ) = h − 1 ∫ e ⊤ ( u − x ) ϕ h ( u − x ) r u g d u = h − 1 E [ e ⊤ ( X − x ) { g ( X ′ ) − P j g ( x )} ∣ X = x ] .
Here differentiation is justified by 0 ≤ r u g ≤ 1 0\leq r_ug\leq1 0 ≤ r u g ≤ 1 and integrability of the Gaussian density’s derivatives. The second equality uses E [ g ( X ′ ) ∣ X , X ~ ] = r X ~ g \mathbb{E}[g(X')\mid X,\widetilde{X}]=r_{\widetilde{X}}g E [ g ( X ′ ) ∣ X , X ] = r X g and E [ X ~ − x ∣ X = x ] = 0 \mathbb{E}[\widetilde{X}-x\mid X=x]=0 E [ X − x ∣ X = x ] = 0 . Since E [ ( e ⊤ ( X ~ − x ) ) 2 ∣ X = x ] = h \mathbb{E}[(e^\top(\widetilde{X}-x))^2\mid X=x]=h E [( e ⊤ ( X − x ) ) 2 ∣ X = x ] = h , Cauchy–Schwarz yields
∣ e ⊤ ∇ P j g ( x ) ∣ 2 ≤ h − 1 Var ( g ( X ′ ) ∣ X = x ) = h − 1 Var P j g ( x ) . |e^\top\nabla P_jg(x)|^2\leq h^{-1}\operatorname{Var}(g(X')\mid X=x)=h^{-1}\operatorname{Var}^{P_j}g(x). ∣ e ⊤ ∇ P j g ( x ) ∣ 2 ≤ h − 1 Var ( g ( X ′ ) ∣ X = x ) = h − 1 Var P j g ( x ) .
Taking the supremum over unit vectors e e e and averaging under Π t j \Pi_{t_j} Π t j gives Π t j ∥ ∇ P j g ∥ 2 ≤ h − 1 Π t j Var P j g \Pi_{t_j}\|\nabla P_jg\|^2\leq h^{-1}\Pi_{t_j}\operatorname{Var}^{P_j}g Π t j ∥∇ P j g ∥ 2 ≤ h − 1 Π t j Var P j g . Combining this bound with the opening display and the variance decomposition (56) , we conclude that
Var Π t j g = Π t j Var P j g + Var Π t j ( P j g ) ≤ ( 1 + V h ) Π t j Var P j g + δ j . \operatorname{Var}_{\Pi_{t_j}}g=\Pi_{t_j}\operatorname{Var}^{P_j}g+\operatorname{Var}_{\Pi_{t_j}}(P_jg)\leq\left(1+\frac{V}{h}\right)\Pi_{t_j}\operatorname{Var}^{P_j}g+\delta_j. Var Π t j g = Π t j Var P j g + Var Π t j ( P j g ) ≤ ( 1 + h V ) Π t j Var P j g + δ j .
Proof of Lemma 8
Assume Π t j ∥ Δ j ∥ 2 < ∞ \Pi_{t_j}\lVert\Delta_j\rVert^2 < \infty Π t j ∥ Δ j ∥ 2 < ∞ , since otherwise the bound is immediate. Fix x x x . The comparison kernel P j ( x , ⋅ ) P_j(x,\cdot) P j ( x , ⋅ ) and the sampler update K ^ j ( x , ⋅ ) \widehat K_j(x,\cdot) K j ( x , ⋅ ) are mixtures over u ∼ N ( x , h I n ) u\sim\mathcal{N}(x,hI_n) u ∼ N ( x , h I n ) , with respective conditional laws r u r_u r u and
γ u : = N ( u + h [ ∇ log Π t j ( x ) + Δ j ( x ) ] , h I n ) . \gamma_u := \mathcal{N}\left(u+h[\nabla\log\Pi_{t_j}(x)+\Delta_j(x)],hI_n\right). γ u := N ( u + h [ ∇ log Π t j ( x ) + Δ j ( x )] , h I n ) .
Thus items 1 and 2 of Lemma 4 give
H 2 ( P j ( x , ⋅ ) , K ^ j ( x , ⋅ ) ) ≤ ∫ ϕ h ( u − x ) H 2 ( r u , γ u ) d u . \mathrm{H}^2(P_j(x,\cdot),\widehat K_j(x,\cdot))\leq\int\phi_h(u-x)\mathrm{H}^2(r_u,\gamma_u)\,\mathrm{d}u. H 2 ( P j ( x , ⋅ ) , K j ( x , ⋅ )) ≤ ∫ ϕ h ( u − x ) H 2 ( r u , γ u ) d u .
To bound the distance between these conditional laws, fix u u u and let ψ : = d r u / d γ u \psi:=\sqrt{\mathrm{d}r_u/\mathrm{d}\gamma_u} ψ := d r u / d γ u . Expanding the Gaussian exponents shows that ψ ( x ′ ) 2 \psi(x')^2 ψ ( x ′ ) 2 is proportional to Π t j ( x ′ ) e − [ ∇ log Π t j ( x ) + Δ j ( x ) ] ⊤ x ′ \Pi_{t_j}(x')e^{-[\nabla\log\Pi_{t_j}(x)+\Delta_j(x)]^\top x'} Π t j ( x ′ ) e − [ ∇ l o g Π t j ( x ) + Δ j ( x ) ] ⊤ x ′ , and hence
∇ ψ ( x ′ ) = 1 2 ψ ( x ′ ) [ ∇ log Π t j ( x ′ ) − ∇ log Π t j ( x ) − Δ j ( x ) ] . \nabla\psi(x')=\frac{1}{2}\psi(x')\left[\nabla\log\Pi_{t_j}(x')-\nabla\log\Pi_{t_j}(x)-\Delta_j(x)\right]. ∇ ψ ( x ′ ) = 2 1 ψ ( x ′ ) [ ∇ log Π t j ( x ′ ) − ∇ log Π t j ( x ) − Δ j ( x ) ] .
The Gaussian factor in r u r_u r u gives r u ∥ x ′ ∥ 2 < ∞ r_u\lVert x'\rVert^2<\infty r u ∥ x ′ ∥ 2 < ∞ , so Lipschitz continuity of ∇ log Π t j \nabla\log\Pi_{t_j} ∇ log Π t j implies γ u ∥ ∇ ψ ∥ 2 < ∞ \gamma_u\lVert\nabla\psi\rVert^2<\infty γ u ∥ ∇ ψ ∥ 2 < ∞ . Together with γ u ψ 2 = 1 \gamma_u\psi^2=1 γ u ψ 2 = 1 , this verifies the hypotheses of the extended Poincaré inequality in Appendix § 7 .§ 1 . By Cauchy–Schwarz, 0 ≤ γ u ψ ≤ 1 0\leq\gamma_u\psi\leq1 0 ≤ γ u ψ ≤ 1 ; applying the Gaussian Poincaré inequality with constant h h h , item 3 of Lemma 5, therefore gives
H 2 ( r u , γ u ) = 2 ( 1 − γ u ψ ) ≤ 2 ( 1 − ( γ u ψ ) 2 ) = 2 Var γ u ψ \mathrm{H}^2(r_u,\gamma_u)=2(1-\gamma_u\psi)\leq2(1-(\gamma_u\psi)^2)=2\operatorname{Var}_{\gamma_u}\psi H 2 ( r u , γ u ) = 2 ( 1 − γ u ψ ) ≤ 2 ( 1 − ( γ u ψ ) 2 ) = 2 Var γ u ψ
≤ 2 h γ u ∥ ∇ ψ ∥ 2 = h 2 r u ∥ ∇ log Π t j − ∇ log Π t j ( x ) − Δ j ( x ) ∥ 2 . \leq2h\gamma_u\lVert\nabla\psi\rVert^2=\frac{h}{2}r_u\left\lVert\nabla\log\Pi_{t_j}-\nabla\log\Pi_{t_j}(x)-\Delta_j(x)\right\rVert^2. ≤ 2 h γ u ∥ ∇ ψ ∥ 2 = 2 h r u ∇ log Π t j − ∇ log Π t j ( x ) − Δ j ( x ) 2 .
Let X , X ~ , X ′ X,\widetilde X,X' X , X , X ′ be constructed as in Step 2. Combining the two Hellinger bounds and averaging over X ∼ Π t j X\sim\Pi_{t_j} X ∼ Π t j yields
∫ H 2 ( P j ( x , ⋅ ) , K ^ j ( x , ⋅ ) ) Π t j ( d x ) ≤ h 2 E ∥ ∇ log Π t j ( X ′ ) − ∇ log Π t j ( X ) − Δ j ( X ) ∥ 2 ≤ h Π t j ∥ Δ j ∥ 2 + h E ∥ ∇ log Π t j ( X ′ ) − ∇ log Π t j ( X ) ∥ 2 , \begin{aligned}
\int\mathrm{H}^2(P_j(x,\cdot),\widehat K_j(x,\cdot))\Pi_{t_j}(\mathrm{d}x)
\leq\frac{h}{2}\mathbb{E}\left\lVert\nabla\log\Pi_{t_j}(X')-\nabla\log\Pi_{t_j}(X)-\Delta_j(X)\right\rVert^2 \\
&\leq h\Pi_{t_j}\lVert\Delta_j\rVert^2+h\mathbb{E}\left\lVert\nabla\log\Pi_{t_j}(X')-\nabla\log\Pi_{t_j}(X)\right\rVert^2,
\end{aligned} ∫ H 2 ( P j ( x , ⋅ ) , K j ( x , ⋅ )) Π t j ( d x ) ≤ 2 h E ∇ log Π t j ( X ′ ) − ∇ log Π t j ( X ) − Δ j ( X ) 2 ≤ h Π t j ∥ Δ j ∥ 2 + h E ∇ log Π t j ( X ′ ) − ∇ log Π t j ( X ) 2 ,
where the last step uses ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 + 2 ∥ b ∥ 2 \lVert a+b\rVert^2\leq2\lVert a\rVert^2+2\lVert b\rVert^2 ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 + 2 ∥ b ∥ 2 . It remains to bound the drift increment in the second term. The centered difference ∇ log Π t j ( X ) − ∇ log Π t j ( X ~ ) \nabla\log\Pi_{t_j}(X)-\nabla\log\Pi_{t_j}(\widetilde X) ∇ log Π t j ( X ) − ∇ log Π t j ( X ) is square-integrable because ∇ log Π t j \nabla\log\Pi_{t_j} ∇ log Π t j is L L L -Lipschitz and E ∥ X − X ~ ∥ 2 = n h \mathbb{E}\lVert X-\widetilde X\rVert^2=nh E ∥ X − X ∥ 2 = nh . Since X X X and X ′ X' X ′ are conditionally i.i.d. given X ~ \widetilde X X , subtracting ∇ log Π t j ( X ~ ) \nabla\log\Pi_{t_j}(\widetilde X) ∇ log Π t j ( X ) from both drift values and expanding the square gives
E ∥ ∇ log Π t j ( X ′ ) − ∇ log Π t j ( X ) ∥ 2 = 2 E ∥ ∇ log Π t j ( X ) − ∇ log Π t j ( X ~ ) ∥ 2 \mathbb{E}\left\lVert\nabla\log\Pi_{t_j}(X')-\nabla\log\Pi_{t_j}(X)\right\rVert^2
=2\mathbb{E}\left\lVert\nabla\log\Pi_{t_j}(X)-\nabla\log\Pi_{t_j}(\widetilde X)\right\rVert^2 E ∇ log Π t j ( X ′ ) − ∇ log Π t j ( X ) 2 = 2 E ∇ log Π t j ( X ) − ∇ log Π t j ( X ) 2
− 2 E ∥ E [ ∇ log Π t j ( X ) − ∇ log Π t j ( X ~ ) ∣ X ~ ] ∥ 2 -2\mathbb{E}\left\lVert\mathbb{E}\left[\nabla\log\Pi_{t_j}(X)-\nabla\log\Pi_{t_j}(\widetilde X)\mid\widetilde X\right]\right\rVert^2 − 2 E E [ ∇ log Π t j ( X ) − ∇ log Π t j ( X ) ∣ X ] 2
≤ 2 L 2 E ∥ X − X ~ ∥ 2 = 2 n h L 2 . \leq2L^2\mathbb{E}\lVert X-\widetilde X\rVert^2=2nhL^2. ≤ 2 L 2 E ∥ X − X ∥ 2 = 2 nh L 2 .
Substituting this estimate proves (57) .
Weak variance inequality and containment
Recall the latent representation and the latent support S t : = supp z S_t:=\operatorname{supp}z S t := supp z under P t \mathcal{P}_t P t defined in Appendix § 7 .§ 2 . We define the following large-support mass as follows.
Definition 6 (Large-support mass). For a signal x ⋆ x^\star x ⋆ with cap q q q from (28), its large-support mass at a matrix A A A , an observation y ∈ R m y\in\mathbb{R}^m y ∈ R m and a level t t t is
δ t : = P t A , y { ∣ S t ∣ > q } . \delta_t:=\mathcal{P}_t^{A,y}\{|S_t|>q\}. δ t := P t A , y { ∣ S t ∣ > q } . This definition measures how the latent variable is concentrated around sparse signals.
Proposition 2 (Weak variance inequality and containment). Let x ⋆ x^\star x ⋆ be a signal with cap q q q and matrix event A = A ( q ) \mathcal{A}=\mathcal{A}(q) A = A ( q ) , and let δ t \delta_t δ t be its large-support mass. The following statements hold.
Weak variance inequality. For every A ∈ A A\in\mathcal{A} A ∈ A , y ∈ R m y\in\mathbb{R}^m y ∈ R m and τ ≤ t ≤ T \tau\leq t\leq T τ ≤ t ≤ T , every bounded smooth g g g with bounded gradient satisfies
Var Π t g ≤ V t Π t ∥ ∇ g ∥ 2 + δ t osc ( g ) 2 , V t : = t + C n s ( 1 + t − 1 / 2 ) , (62) \operatorname{Var}_{\Pi_t}g\leq V_t\Pi_t\lVert\nabla g\rVert^2+\delta_t\operatorname{osc}(g)^2,\qquad V_t:=t+C\frac{n}{s}(1+t^{-1/2}), \tag*{(62)} Var Π t g ≤ V t Π t ∥ ∇ g ∥ 2 + δ t osc ( g ) 2 , V t := t + C s n ( 1 + t − 1/2 ) , (62) where C C C is a universal constant.
Containment. For every matrix A A A , with y = A x ⋆ + σ ζ y=Ax^\star+\sigma\zeta y = A x ⋆ + σ ζ , the large-support mass satisfies
E ζ δ t ≤ ( 2 Q ) − 14 , t ≥ τ . (63) \mathbb{E}_{\zeta}\delta_t\leq(2Q)^{-14},\qquad t\geq\tau. \tag*{(63)} E ζ δ t ≤ ( 2 Q ) − 14 , t ≥ τ . (63)
The rest of this subsection is devoted to proving Proposition 2. Steps 1–4 establish the weak variance inequality, and Step 5 bounds the defect on average over the observation noise. Let g g g be bounded and smooth with bounded gradient, and let b = b t b=b_t b = b t , G = G t G=G_t G = G t and B = B t B=B_t B = B t for a fixed t t t , where we drop the subscript t t t for notation simplicity.
Step 1: symmetrization. Let Z Z Z and Z ′ Z' Z ′ be independent with law P t \mathsf{P}_t P t , and draw X X X and X ′ X' X ′ independently from F t ( Z , ⋅ ) \mathcal{F}_t(Z,\cdot) F t ( Z , ⋅ ) and F t ( Z ′ , ⋅ ) \mathcal{F}_t(Z',\cdot) F t ( Z ′ , ⋅ ) . By (48) , X X X and X ′ X' X ′ are independent with law Π t \Pi_t Π t . Set
V : = supp Z ∪ supp Z ′ , F : = g ( X ) − g ( X ′ ) . \mathcal{V}:=\operatorname{supp}Z\cup\operatorname{supp}Z',\qquad F:=g(X)-g(X'). V := supp Z ∪ supp Z ′ , F := g ( X ) − g ( X ′ ) .
Conditioning on the union V \mathcal{V} V preserves exchangeability of the two copies. Therefore
E [ F ∣ V ] = 0 , Var Π t g = 1 2 E F 2 = 1 2 E Var ( F ∣ V ) . \mathbb{E}[F\mid\mathcal{V}]=0,\qquad\operatorname{Var}_{\Pi_t}g=\frac{1}{2}\mathbb{E}F^2=\frac{1}{2}\mathbb{E}\operatorname{Var}(F\mid\mathcal{V}). E [ F ∣ V ] = 0 , Var Π t g = 2 1 E F 2 = 2 1 E Var ( F ∣ V ) .
We call a union V \mathcal{V} V covered if ∣ V ∣ ≤ 2 q |\mathcal{V}|\leq2q ∣ V ∣ ≤ 2 q . Otherwise, if ∣ V ∣ > 2 q |\mathcal{V}|>2q ∣ V ∣ > 2 q , it requires at least one copy to have support larger than q q q ,
P ( ∣ V ∣ > 2 q ) ≤ 2 δ t . (64) \mathbb{P}(|\mathcal{V}|>2q)\leq2\delta_t. \tag*{(64)} P ( ∣ V ∣ > 2 q ) ≤ 2 δ t . (64)
It thus suffices to show that, for every covered union J J J , the conditional law of ( X , X ′ ) (X,X') ( X , X ′ ) given V = J \mathcal{V}=J V = J has Poincaré constant at most V t V_t V t . Indeed, applying this bound to g ( X ) − g ( X ′ ) g(X)-g(X') g ( X ) − g ( X ′ ) on covered unions and using F 2 ≤ osc ( g ) 2 F^2\leq\operatorname{osc}(g)^2 F 2 ≤ osc ( g ) 2 on the complement would give
Var Π t g ≤ 1 2 V t E [ ∥ ∇ g ( X ) ∥ 2 + ∥ ∇ g ( X ′ ) ∥ 2 ] + 1 2 P ( ∣ V ∣ > 2 q ) osc ( g ) 2 ≤ V t Π t ∥ ∇ g ∥ 2 + δ t osc ( g ) 2 . \begin{aligned}
\operatorname{Var}_{\Pi_t}g\leq\frac{1}{2}V_t\mathbb{E}\left[\lVert\nabla g(X)\rVert^2+\lVert\nabla g(X')\rVert^2\right]+\frac{1}{2}\mathbb{P}(|\mathcal{V}|>2q)\operatorname{osc}(g)^2 \\
&\leq V_t\Pi_t\lVert\nabla g\rVert^2+\delta_t\operatorname{osc}(g)^2.
\end{aligned} Var Π t g ≤ 2 1 V t E [ ∥ ∇ g ( X ) ∥ 2 + ∥ ∇ g ( X ′ ) ∥ 2 ] + 2 1 P ( ∣ V ∣ > 2 q ) osc ( g ) 2 ≤ V t Π t ∥ ∇ g ∥ 2 + δ t osc ( g ) 2 .
We establish the conditional Poincaré bound by identifying the pair law, controlling its interaction, and passing it through the forward kernel.
Step 2: identify the conditional pair law. Fix a nonempty covered union J J J with P ( V = J ) > 0 \mathbb{P}(\mathcal{V}=J)>0 P ( V = J ) > 0 . We express the conditional law of the coordinate pairs ( Z i , Z i ′ ) i ∈ J (Z_i,Z'_i)_{i\in J} ( Z i , Z i ′ ) i ∈ J as a quadratic tilt of a product of pair laws. On V = J \mathcal{V}=J V = J , both copies vanish off J J J , and at least one coordinate in each pair ( Z i , Z i ′ ) (Z_i,Z'_i) ( Z i , Z i ′ ) , i ∈ J i\in J i ∈ J , is nonzero. Thus conditioning on the union imposes a separate restriction ( Z i , Z i ′ ) ≠ ( 0 , 0 ) (Z_i,Z'_i)\neq(0,0) ( Z i , Z i ′ ) = ( 0 , 0 ) at each coordinate.
Writing ω = s / ( n − s ) \omega=s/(n-s) ω = s / ( n − s ) for the prior odds, (50) and the product form of the prior give
P t ( d z ) ∝ e b T z − z T B z / 2 ν ( d z ) , ν ( d z ) : = ⨂ i = 1 n [ δ 0 ( d z i ) + ω ( 2 π ) − 1 / 2 d z i ] . \mathsf{P}_t(dz)\propto e^{b^\mathsf{T}z-z^\mathsf{T}Bz/2}\nu(dz),\qquad\nu(dz):=\bigotimes_{i=1}^{n}\left[\delta_0(dz_i)+\omega(2\pi)^{-1/2}\,dz_i\right]. P t ( d z ) ∝ e b T z − z T B z /2 ν ( d z ) , ν ( d z ) := i = 1 ⨂ n [ δ 0 ( d z i ) + ω ( 2 π ) − 1/2 d z i ] .
To obtain the positive semidefinite interaction required by Lemma 11 below, we slightly increase the diagonal reference precisions by setting
B ∗ , J : = min j ∈ J B j j , b i r e f : = B i i + ϵ B ∗ , J ( i ∈ J ) . B_{*,J}:=\min_{j\in J}B_{jj},\qquad b_i^{\mathrm{ref}}:=B_{ii}+\epsilon B_{*,J}\quad(i\in J). B ∗ , J := j ∈ J min B j j , b i ref := B ii + ϵ B ∗ , J ( i ∈ J ) .
where ε \varepsilon ε is defined in § 5.1 . The block condition (34) gives
Ξ : = diag ( b i r e f ) − B J , J = ε B ∗ , J I − ( B − diag B ) J , J , 0 ⪯ Ξ ⪯ 2 ε B ∗ , J I . \Xi:= \operatorname{diag}(b_i^{\mathrm{ref}}) - B_{J,J} = \varepsilon B_{*,J}I - (B - \operatorname{diag} B)_{J,J}, \qquad0 \preceq\Xi\preceq2\varepsilon B_{*,J}I. Ξ := diag ( b i ref ) − B J , J = ε B ∗ , J I − ( B − diag B ) J , J , 0 ⪯ Ξ ⪯ 2 ε B ∗ , J I .
The quadratic tilt exactly compensates for the increased reference precisions: for z ∈ R J z \in\mathbb{R}^{J} z ∈ R J ,
e b J ⊤ z − z ⊤ B J , J z / 2 = e z ⊤ Ξ z / 2 ∏ i ∈ J e b i z i − b i r e f z i 2 / 2 . e^{b_J^\top z-z^\top B_{J,J}z/2}=e^{z^\top\Xi z/2}\prod_{i\in J}e^{b_i z_i-b_i^{\mathrm{ref}}z_i^2/2}. e b J ⊤ z − z ⊤ B J , J z /2 = e z ⊤ Ξ z /2 i ∈ J ∏ e b i z i − b i ref z i 2 /2 .
To combine these diagonal factors with the coordinatewise restrictions, fix B 0 > 0 B_0 > 0 B 0 > 0 and define
ρ u ( d z ) : = δ 0 ( d z ) + ω ( 2 π ) − 1 / 2 e u z − B 0 z 2 / 2 d z , u ∈ R . \rho_u(dz):=\delta_0(dz)+\omega(2\pi)^{-1/2}e^{uz-B_0z^2/2}\,dz,\qquad u\in\mathbb{R}. ρ u ( d z ) := δ 0 ( d z ) + ω ( 2 π ) − 1/2 e u z − B 0 z 2 /2 d z , u ∈ R .
For θ ∈ R 2 \theta\in\mathbb{R}^2 θ ∈ R 2 , let μ θ \mu_\theta μ θ be the probability obtained by normalizing the restriction of ρ θ 1 ⊗ ρ θ 2 \rho_{\theta_1}\otimes\rho_{\theta_2} ρ θ 1 ⊗ ρ θ 2 to R 2 ∖ { ( 0 , 0 ) } \mathbb{R}^2\setminus\{(0,0)\} R 2 ∖ {( 0 , 0 )} . This removes the joint atom at the origin while retaining the two axis components. For each i ∈ J i\in J i ∈ J , take μ ( i ) = μ ( b i , b i ) \mu^{(i)}=\mu_{(b_i,b_i)} μ ( i ) = μ ( b i , b i ) with B 0 = b i r e f B_0=b_i^{\mathrm{ref}} B 0 = b i ref . Multiplying the two copies of the preceding factorization and imposing the pairwise restrictions shows that w : = ( ( Z i , Z i ′ ) ) i ∈ J w:=((Z_i,Z_i') )_{i\in J} w := (( Z i , Z i ′ ) ) i ∈ J has conditional law
μ H ( d w ) ∝ e w ⊤ H w / 2 μ ( d w ) , μ : = ⨂ i ∈ J μ ( i ) , 0 ⪯ H ⪯ 2 ε B ∗ , J I , (65) \mu_H(dw)\propto e^{w^\top Hw/2}\mu(dw),\qquad\mu:=\bigotimes_{i\in J}\mu^{(i)},\qquad0\preceq H\preceq2\varepsilon B_{*,J}I, \tag*{(65)} μ H ( d w ) ∝ e w ⊤ H w /2 μ ( d w ) , μ := i ∈ J ⨂ μ ( i ) , 0 ⪯ H ⪯ 2 ε B ∗ , J I , (65)
where H H H acts as Ξ \Xi Ξ on each copy: w ⊤ H w = Z J ⊤ Ξ Z J + ( Z J ′ ) ⊤ Ξ Z J ′ w^\top Hw=Z_J^\top\Xi Z_J+(Z'_J)^\top\Xi Z'_J w ⊤ H w = Z J ⊤ Ξ Z J + ( Z J ′ ) ⊤ Ξ Z J ′ . The spectral bounds on H H H follow from those on Ξ \Xi Ξ , and the normalizing constants of the individual pair laws are absorbed into the overall normalizer.
Step 3: bound the pair laws and their interaction. We first record covariance and smoothed Poincaré bounds for the pair laws, uniformly over all linear tilts under Gaussian smoothing. This uniformity will make the resulting conditional Poincaré bound independent of the observation y y y . The proof of this lemma can be found in Appendix § 8 .2.1.
For a probability μ \mu μ on R d \mathbb{R}^{d} R d whose Laplace transform M ( η ) : = ∫ e η ⊤ w μ ( d w ) \mathcal{M}(\eta):=\int e^{\eta^\top w}\,\mu(dw) M ( η ) := ∫ e η ⊤ w μ ( d w ) is finite for every η ∈ R d \eta\in\mathbb{R}^{d} η ∈ R d , write μ η ( d w ) : = M ( η ) − 1 e η ⊤ w μ ( d w ) \mu^\eta(dw):=\mathcal{M}(\eta)^{-1}e^{\eta^\top w}\mu(dw) μ η ( d w ) := M ( η ) − 1 e η ⊤ w μ ( d w ) for its normalized linear tilts.
Lemma 10 (A Gaussian plane and its axes). Let B 0 > 0 B_0>0 B 0 > 0 and r > 0 r>0 r > 0 , and set
v : = 1 / B 0 , β : = B 0 / ω , log + x : = max ( 0 , log x ) . v:=1/B_0,\qquad\beta:=\sqrt{B_0}/\omega,\qquad\log_+x:=\max(0,\log x). v := 1/ B 0 , β := B 0 / ω , log + x := max ( 0 , log x ) . For every θ ∈ R 2 \theta\in\mathbb{R}^2 θ ∈ R 2 , the pair law μ θ \mu_\theta μ θ defined in Step 2 has the following properties:
μ θ η = μ θ + η \mu_\theta^\eta=\mu_{\theta+\eta} μ θ η = μ θ + η for every η ∈ R 2 \eta\in\mathbb{R}^2 η ∈ R 2 ,
the covariance of μ θ \mu_\theta μ θ is at most C v ( 1 + log + β ) I Cv(1+\log_+\beta)I C v ( 1 + log + β ) I ,
the law μ θ ∗ N ( 0 , r I 2 ) \mu_\theta*\mathcal{N}(0,rI_2) μ θ ∗ N ( 0 , r I 2 ) has Poincaré constant at most ( v + r ) + 2 β ( v + r ) 3 / 2 r − 1 / 2 (v+r)+2\beta(v+r)^{3/2}r^{-1/2} ( v + r ) + 2 β ( v + r ) 3/2 r − 1/2 .
In particular the bounds of items 2 and 3 hold for every linear tilt of μ θ \mu_\theta μ θ .
The next lemma combines a covariance bound that controls the quadratic interaction, and a smoothed Poincaré bound to control the resulting law. Its proof adapts the strategy of Bauerschmidt and Bodineau [4 Section 1] to a variance inequality, using a representation as a mixture of linear tilts over a strongly log-concave field. The proof is given in Appendix § 8 .2.2.
Lemma 11 (Weak interaction). Let μ \mu μ be a probability on R d ˉ \mathbb{R}^{\bar d} R d ˉ with Laplace transform finite everywhere, and r ≥ 0 r\geq0 r ≥ 0 . Suppose that for every η ∈ R d ˉ \eta\in\mathbb{R}^{\bar d} η ∈ R d ˉ the tilt μ η \mu^\eta μ η has covariance at most v 0 I v_0I v 0 I and μ η ∗ N ( 0 , r I ) \mu^\eta*\mathcal{N}(0,rI) μ η ∗ N ( 0 , r I ) has Poincaré constant at most K 0 K_0 K 0 . Let H H H be symmetric with H ⪰ 0 H\succeq0 H ⪰ 0 and v ˉ : = v 0 ∥ H ∥ < 1 \bar v:=v_0\lVert H\rVert<1 v ˉ := v 0 ∥ H ∥ < 1 . Then
∫ e w ⊤ H w / 2 μ ( d w ) < ∞ \int e^{w^\top Hw/2}\mu(dw)<\infty ∫ e w ⊤ H w /2 μ ( d w ) < ∞ , so that μ H ( d w ) ∝ e w ⊤ H w / 2 μ ( d w ) \mu_H(dw)\propto e^{w^\top Hw/2}\mu(dw) μ H ( d w ) ∝ e w ⊤ H w /2 μ ( d w ) is a probability,
the law μ H ∗ N ( 0 , r I ) \mu_H*\mathcal{N}(0,rI) μ H ∗ N ( 0 , r I ) has Poincaré constant at most K 0 / ( 1 − v ˉ ) K_0/(1-\bar v) K 0 / ( 1 − v ˉ ) .
Step 4: establish the conditional Poincaré inequality. Fix J ≠ ∅ J \ne\varnothing J = ∅ with ∣ J ∣ ≤ 2 q |J| \le2q ∣ J ∣ ≤ 2 q and P ( V = J ) > 0 \mathbb{P}(V=J)>0 P ( V = J ) > 0 , and work conditionally on V = J V=J V = J . Take μ \mu μ , H H H and b i r e f b_i^{\mathrm{ref}} b i ref from the representation (65) in Step 2. Set r : = min { t , 1 } r:=\min\{t,1\} r := min { t , 1 } . We first bound the Poincaré constant of μ H ∗ N ( 0 , r I ) \mu_H * \mathcal{N}(0,rI) μ H ∗ N ( 0 , r I ) , then realize this smoothing using part of the noise in the forward kernel.
We begin by verifying the uniform bounds on linear tilts required by Lemma 11 . Each pair law is a finite mixture of Gaussians, so μ \mu μ has Laplace transform finite everywhere, and a tilt of μ \mu μ is the product of tilts of the pair laws. For each pair i i i , Lemma 10 applies with B 0 = b i r e f B_0=b_i^{\mathrm{ref}} B 0 = b i ref , so v = ( b i r e f ) − 1 v=(b_i^{\mathrm{ref}})^{-1} v = ( b i ref ) − 1 and β = b i r e f / ω \beta=\sqrt{b_i^{\mathrm{ref}}}/\omega β = b i ref / ω . Since 0 ≺ M t ⪯ I n 0 \prec M_t \preceq I_n 0 ≺ M t ⪯ I n , we have 0 ⪯ G = t − 1 ( I n − M t ) ⪯ t − 1 I n 0 \preceq G=t^{-1}(I_n-M_t)\preceq t^{-1}I_n 0 ⪯ G = t − 1 ( I n − M t ) ⪯ t − 1 I n . Thus B = I + G B=I+G B = I + G , ε ≤ 1 / 2 \varepsilon\le1/2 ε ≤ 1/2 and (22) give
1 ≤ B ∗ , J ≤ b i r e f ≤ ( 1 + ε ) ( 1 + t − 1 ) ≤ 4 Q , ω − 1 ≤ n / s ≤ Q . 1 \le B_{*,J} \le b_i^{\mathrm{ref}} \le(1+\varepsilon)(1+t^{-1}) \le4Q,\qquad\omega^{-1}\le n/s\le Q. 1 ≤ B ∗ , J ≤ b i ref ≤ ( 1 + ε ) ( 1 + t − 1 ) ≤ 4 Q , ω − 1 ≤ n / s ≤ Q .
Consequently v ≤ B ∗ , J − 1 v\le B_{*,J}^{-1} v ≤ B ∗ , J − 1 and β ≤ 2 Q 3 / 2 \beta\le2Q^{3/2} β ≤ 2 Q 3/2 , so 1 + log + β ≤ C log ( 2 Q ) 1+\log^+\beta\le C\log(2Q) 1 + log + β ≤ C log ( 2 Q ) . By Lemma 10 (1)–(2) and the block diagonal covariance of a product, every tilt μ η \mu^\eta μ η satisfies
Cov ( μ η ) ⪯ v 0 I , v 0 : = C c o v log ( 2 Q ) B ∗ , J , \operatorname{Cov}(\mu^\eta)\preceq v_0 I,\qquad v_0:=\frac{C_{\mathrm{cov}}\log(2Q)}{B_{*,J}}, Cov ( μ η ) ⪯ v 0 I , v 0 := B ∗ , J C cov log ( 2 Q ) ,
for a universal constant C c o v C_{\mathrm{cov}} C cov .
To bound the smoothed tilts, combine the preceding bounds on b i r e f b_i^{\mathrm{ref}} b i ref with r ( 1 + t − 1 ) ≤ 2 r(1+t^{-1})\le2 r ( 1 + t − 1 ) ≤ 2 and r ≤ 1 r\le1 r ≤ 1 to obtain
r ≤ 3 v , v ≤ 1 , v + r ≤ min { 2 , 4 v } , β v = ω − 1 ≤ n s . r\le3v,\qquad v\le1,\qquad v+r\le\min\{2,4v\},\qquad\beta\sqrt{v}=\omega^{-1}\le\frac{n}{s}. r ≤ 3 v , v ≤ 1 , v + r ≤ min { 2 , 4 v } , β v = ω − 1 ≤ s n .
Lemma 10 (3) therefore bounds the Poincaré constant of each smoothed pair law by
( v + r ) + 2 β ( v + r ) 3 / 2 r − 1 / 2 ≤ 2 + 16 ω − 1 v r − 1 / 2 ≤ K 0 : = C n s ( 1 + t − 1 / 2 ) , (v+r)+2\beta(v+r)^{3/2}r^{-1/2}\le2+16\omega^{-1}vr^{-1/2}\le K_0:=C\frac{n}{s}(1+t^{-1/2}), ( v + r ) + 2 β ( v + r ) 3/2 r − 1/2 ≤ 2 + 16 ω − 1 v r − 1/2 ≤ K 0 := C s n ( 1 + t − 1/2 ) ,
where we also used r − 1 / 2 ≤ 1 + t − 1 / 2 r^{-1/2}\le1+t^{-1/2} r − 1/2 ≤ 1 + t − 1/2 and n / s ≥ 1 n/s\ge1 n / s ≥ 1 . By Lemma 10 (1), this bound holds uniformly over all linear tilts of each pair law. Since μ η ∗ N ( 0 , r I ) \mu^\eta*\mathcal{N}(0,rI) μ η ∗ N ( 0 , r I ) is the product of these smoothed tilts, tensorization in Lemma 5(1) gives it Poincaré constant at most K 0 K_0 K 0 .
These estimates allow us to control the quadratic interaction. Indeed, (65) and ε = c ε / log ( 2 Q ) \varepsilon=c_\varepsilon/\log(2Q) ε = c ε / log ( 2 Q ) give
v 0 ∥ H ∥ ≤ C c o v log ( 2 Q ) B ∗ , J ⋅ 2 ε B ∗ , J = 2 C c o v c ε ≤ 1 2 , v_0\|H\|\le\frac{C_{\mathrm{cov}}\log(2Q)}{B_{*,J}}\cdot2\varepsilon B_{*,J}=2C_{\mathrm{cov}}c_\varepsilon\le\frac{1}{2}, v 0 ∥ H ∥ ≤ B ∗ , J C cov log ( 2 Q ) ⋅ 2 ε B ∗ , J = 2 C cov c ε ≤ 2 1 ,
provided c ε ≤ 1 / ( 4 C c o v ) c_\varepsilon\le1/(4C_{\mathrm{cov}}) c ε ≤ 1/ ( 4 C cov ) . Lemma 11 then shows that w ~ : = w + v ξ ~ \widetilde{w}:=w+\sqrt{v}\,\widetilde{\xi} w := w + v ξ , where ξ ~ \widetilde{\xi} ξ is an independent standard Gaussian, has Poincaré constant at most 2 K 0 2K_0 2 K 0 .
It remains to pass this bound through the forward kernel. Since 0 ≺ M t ⪯ I 0\prec M_t\preceq I 0 ≺ M t ⪯ I and r ≤ t r\le t r ≤ t , the matrix R : = t M t − r ( M t ) [ n ] J ( M t ) J [ n ] R:=tM_t-r(M_t)_{[n]J}(M_t)_{J[n]} R := t M t − r ( M t ) [ n ] J ( M t ) J [ n ] satisfies
0 ⪯ t M t − r M t 2 ⪯ R ⪯ t I . 0\preceq tM_t-rM_t^2\preceq R\preceq tI. 0 ⪯ t M t − r M t 2 ⪯ R ⪯ t I .
Write ξ ~ = ( ( ξ ~ i , ξ ~ i ′ ) ) i ∈ J \widetilde{\xi}=((\widetilde{\xi}_i,\widetilde{\xi}'_i))_{i\in J} ξ = (( ξ i , ξ i ′ ) ) i ∈ J , and let ξ ‾ , ξ ‾ ′ \overline{\xi},\overline{\xi}' ξ , ξ ′ be independent standard Gaussians, independent of ( w , ξ ~ ) (w,\widetilde{\xi}) ( w , ξ ) . By the definition of R R R , the noise in each copy can be split into the smoothing term and a residual Gaussian with covariance R R R . Since Z Z Z and Z ′ Z' Z ′ vanish off J J J , (49) gives the joint representation
X = d ( M t ) [ n ] J ( Z J + r ( ξ ~ i ) i ∈ J ) + t b + R 1 / 2 ξ ‾ , X \mathrel{\mathop{=}\limits^{d}}(M_t)_{[n]J}\left(Z_J+\sqrt{r}\,(\widetilde{\xi}_i)_{i\in J}\right)+tb+R^{1/2}\overline{\xi}, X = d ( M t ) [ n ] J ( Z J + r ( ξ i ) i ∈ J ) + t b + R 1/2 ξ ,
X ′ = d ( M t ) [ n ] J ( Z J ′ + r ( ξ ~ i ′ ) i ∈ J ) + t b + R 1 / 2 ξ ‾ ′ . X' \mathrel{\mathop{=}\limits^{d}}(M_t)_{[n]J}\left(Z'_J+\sqrt{r}\,(\widetilde{\xi}'_i)_{i\in J}\right)+tb+R^{1/2}\overline{\xi}'. X ′ = d ( M t ) [ n ] J ( Z J ′ + r ( ξ i ′ ) i ∈ J ) + t b + R 1/2 ξ ′ .
Thus ( X , X ′ ) (X,X') ( X , X ′ ) is an affine image of w ~ \widetilde{w} w with linear part of norm at most ∥ M t ∥ ≤ 1 \|M_t\|\le1 ∥ M t ∥ ≤ 1 , plus an independent Gaussian with covariance diag ( R , R ) ⪯ t I \operatorname{diag}(R,R)\preceq tI diag ( R , R ) ⪯ t I . Lemma 5(2)–(4) then bounds the Poincaré constant of the conditional law of ( X , X ′ ) (X,X') ( X , X ′ ) given V = J V=J V = J by
2 K 0 + t = V t , 2K_0+t=V_t, 2 K 0 + t = V t ,
which defines the constant in V t V_t V t . For V = ∅ V=\varnothing V = ∅ only the Gaussian remains, and the constant is at most t ≤ V t t\le V_t t ≤ V t . The reduction in Step 1 now gives the variance inequality (62) .
Step 5: bound the large-support mass. To prove (63) , fix an arbitrary matrix A A A , a signal x ⋆ x^\star x ⋆ with support S S S , the observation y = A x ⋆ + σ ζ y=Ax^\star+\sigma\zeta y = A x ⋆ + σ ζ and a level t ≥ τ t\ge\tau t ≥ τ . Write G = G t G=G_t G = G t . We bound the normalizing constant of P t P_t P t from below uniformly in the observation noise, then bound the expected unnormalized mass of large supports by their prior probability.
Expanding the residual y − A z = σ ζ − A ( z − x ⋆ ) y-Az=\sigma\zeta-A(z-x^\star) y − A z = σ ζ − A ( z − x ⋆ ) in (50) and discarding the term independent of z z z , we obtain, with η : = σ A ⊤ Σ t − 1 ζ \eta:=\sigma A^\top\Sigma_t^{-1}\zeta η := σ A ⊤ Σ t − 1 ζ and T : = { z : ∣ supp z ∣ > q } T:=\{z:|\operatorname{supp}z|>q\} T := { z : ∣ supp z ∣ > q } ,
P t ( d z ) ∝ e − U ( z ) p ⋆ ( d z ) , U ( z ) : = 1 2 ( z − x ⋆ ) ⊤ G ( z − x ⋆ ) − η ⊤ ( z − x ⋆ ) , δ t = ∫ T e − U d p ⋆ ∫ e − U d p ⋆ . P_t(dz)\propto e^{-U(z)}p_\star(dz),\qquad U(z):=\frac{1}{2}(z-x^\star)^\top G(z-x^\star)-\eta^\top(z-x^\star),\qquad\delta_t=\frac{\int_T e^{-U}\,dp_\star}{\int e^{-U}\,dp_\star}. P t ( d z ) ∝ e − U ( z ) p ⋆ ( d z ) , U ( z ) := 2 1 ( z − x ⋆ ) ⊤ G ( z − x ⋆ ) − η ⊤ ( z − x ⋆ ) , δ t = ∫ e − U d p ⋆ ∫ T e − U d p ⋆ .
To bound the denominator, let ρ \rho ρ be the law of a vector that vanishes off S S S and is N ( x ⋆ , r I S ) \mathcal{N}(x^\star,rI_S) N ( x ⋆ , r I S ) on S S S , where r : = min { t , 1 } r:=\min\{t,1\} r := min { t , 1 } ; for S = ∅ S=\varnothing S = ∅ , take ρ = δ 0 \rho=\delta_0 ρ = δ 0 . Since ρ \rho ρ has mean x ⋆ x^\star x ⋆ , the linear noise term in U U U has zero expectation under ρ \rho ρ . Hence ρ U = 1 2 r tr G S S ≤ ∣ S ∣ / 2 \rho U=\frac{1}{2}r\operatorname{tr}G_{SS}\le|S|/2 ρ U = 2 1 r tr G S S ≤ ∣ S ∣/2 , using G = t − 1 ( I n − M t ) ⪯ t − 1 I n G=t^{-1}(I_n-M_t)\preceq t^{-1}I_n G = t − 1 ( I n − M t ) ⪯ t − 1 I n and r ≤ t r\le t r ≤ t . On vectors with support S S S , p ⋆ p_\star p ⋆ has density ( s / n ) ∣ S ∣ ( 1 − s / n ) n − ∣ S ∣ (s/n)^{|S|}(1-s/n)^{n-|S|} ( s / n ) ∣ S ∣ ( 1 − s / n ) n − ∣ S ∣ times the standard Gaussian density on R S \mathbb{R}^S R S . In particular, ρ ≪ p ⋆ \rho\ll p_\star ρ ≪ p ⋆ , and
KL ( ρ ∥ p ⋆ ) = ∣ S ∣ log n s + ( n − ∣ S ∣ ) log ( 1 + ω ) + 1 2 ( ∣ S ∣ r + ∥ x ⋆ ∥ 2 − ∣ S ∣ − ∣ S ∣ log r ) ≤ 3 2 ∣ S ∣ log ( 2 Q ) + 2 s + 1 2 ∥ x ⋆ ∥ 2 . \begin{aligned}
\operatorname{KL}(\rho\|p_\star)=|S|\log\frac{n}{s}+(n-|S|)\log(1+\omega)+\frac{1}{2}\left(|S|r+\|x^\star\|^2-|S|-|S|\log r\right) \\
&\le\frac{3}{2}|S|\log(2Q)+2s+\frac{1}{2}\|x^\star\|^2.
\end{aligned} KL ( ρ ∥ p ⋆ ) = ∣ S ∣ log s n + ( n − ∣ S ∣ ) log ( 1 + ω ) + 2 1 ( ∣ S ∣ r + ∥ x ⋆ ∥ 2 − ∣ S ∣ − ∣ S ∣ log r ) ≤ 2 3 ∣ S ∣ log ( 2 Q ) + 2 s + 2 1 ∥ x ⋆ ∥ 2 .
Here we used n / s ≤ 2 Q n/s\le2Q n / s ≤ 2 Q , r ≤ 1 r\le1 r ≤ 1 and 1 / r ≤ max { 1 , 1 / τ } ≤ Q 1/r\le\max\{1,1/\tau\}\le Q 1/ r ≤ max { 1 , 1/ τ } ≤ Q , by (22) , together with ( n − ∣ S ∣ ) log ( 1 + ω ) ≤ n ω ≤ 2 s (n-|S|)\log(1+\omega)\le n\omega\le2s ( n − ∣ S ∣ ) log ( 1 + ω ) ≤ nω ≤ 2 s . Jensen’s inequality now yields
∫ e − U d p ⋆ ≥ ∫ e − U − log ( d ρ / d p ⋆ ) d ρ ≥ e − ρ U − KL ( ρ ∥ p ⋆ ) ≥ ( 2 Q ) − 3 ∣ S ∣ / 2 e − ∣ S ∣ / 2 − 2 s − 1 2 ∥ x ⋆ ∥ 2 . \int e^{-U}\,dp_\star\ge\int e^{-U-\log(d\rho/dp_\star)}\,d\rho\ge e^{-\rho U-\operatorname{KL}(\rho\|p_\star)}\ge(2Q)^{-3|S|/2}e^{-|S|/2-2s-\frac{1}{2}\|x^\star\|^2}. ∫ e − U d p ⋆ ≥ ∫ e − U − l o g ( d ρ / d p ⋆ ) d ρ ≥ e − ρ U − KL ( ρ ∥ p ⋆ ) ≥ ( 2 Q ) − 3∣ S ∣/2 e − ∣ S ∣/2 − 2 s − 2 1 ∥ x ⋆ ∥ 2 .
This lower bound holds for every realization of ζ \zeta ζ .
We next bound the numerator after averaging over the noise. For each z z z , set v : = z − x ⋆ v:=z-x^\star v := z − x ⋆ . The noise enters U U U only through the centered Gaussian vector η \eta η , whose covariance satisfies Cov η = σ 2 A ⊤ Σ t − 2 A ⪯ A ⊤ Σ t − 1 A = G \operatorname{Cov}\eta=\sigma^2A^\top\Sigma_t^{-2}A\preceq A^\top\Sigma_t^{-1}A=G Cov η = σ 2 A ⊤ Σ t − 2 A ⪯ A ⊤ Σ t − 1 A = G , since Σ t ⪰ σ 2 I m \Sigma_t\succeq\sigma^2I_m Σ t ⪰ σ 2 I m . Thus E ζ e η ⊤ v = e v ⊤ ( Cov η ) v / 2 ≤ e v ⊤ G v / 2 \mathbb{E}_\zeta e^{\eta^\top v}=e^{v^\top(\operatorname{Cov}\eta)v/2}\le e^{v^\top Gv/2} E ζ e η ⊤ v = e v ⊤ ( Cov η ) v /2 ≤ e v ⊤ G v /2 , so the weight e − U ( z ) e^{-U(z)} e − U ( z ) has expectation at most one for every z z z . Under p ⋆ p_\star p ⋆ , the support size is binomial with parameters n n n and s / n s/n s / n . Tonelli’s theorem therefore gives
E ζ ∫ T e − U d p ⋆ = ∫ T e − v ⊤ G v / 2 E ζ e η ⊤ v p ⋆ ( d z ) ≤ p ⋆ ( T ) ≤ e − q ( 1 + ( e − 1 ) s n ) n ≤ e ( e − 1 ) s − q . \mathbb{E}_\zeta\int_T e^{-U}\,dp_\star=\int_T e^{-v^\top Gv/2}\mathbb{E}_\zeta e^{\eta^\top v}\,p_\star(dz)\le p_\star(T)\le e^{-q}\left(1+(e-1)\frac{s}{n}\right)^n\le e^{(e-1)s-q}. E ζ ∫ T e − U d p ⋆ = ∫ T e − v ⊤ G v /2 E ζ e η ⊤ v p ⋆ ( d z ) ≤ p ⋆ ( T ) ≤ e − q ( 1 + ( e − 1 ) n s ) n ≤ e ( e − 1 ) s − q .
where the second bound is Markov’s inequality for e ∣ supp z ∣ e^{|\operatorname{supp}z|} e ∣ supp z ∣ and the third is 1 + u ≤ e u 1+u\le e^u 1 + u ≤ e u .
Since the denominator bound is independent of ζ \zeta ζ , dividing the expected numerator by this bound controls E ζ δ t \mathbb{E}_\zeta\delta_t E ζ δ t . Using log ( 2 Q ) ≥ 1 \log(2Q)\ge1 log ( 2 Q ) ≥ 1 , e + 1 ≤ 4 e+1\le4 e + 1 ≤ 4 and q ≥ 14 ( 1 + s + ∣ S ∣ + ∥ x ⋆ ∥ 2 ) log ( 2 Q ) q\ge14(1+s+|S|+\|x^\star\|^2)\log(2Q) q ≥ 14 ( 1 + s + ∣ S ∣ + ∥ x ⋆ ∥ 2 ) log ( 2 Q ) , we obtain
E ζ δ t ≤ exp { 3 2 ∣ S ∣ log ( 2 Q ) + ∣ S ∣ 2 + ( e + 1 ) s + 1 2 ∥ x ⋆ ∥ 2 − q } ≤ exp { ( 2 ∣ S ∣ + 2 ∥ x ⋆ ∥ 2 + 4 s ) log ( 2 Q ) − q } ≤ ( 2 Q ) − 14 . \begin{aligned}
\mathbb{E}_\zeta\delta_t\le\exp\left\{\frac{3}{2}|S|\log(2Q)+\frac{|S|}{2}+(e+1)s+\frac{1}{2}\|x^\star\|^2-q\right\} \\
&\le\exp\left\{(2|S|+2\|x^\star\|^2+4s)\log(2Q)-q\right\}\le(2Q)^{-14}.
\end{aligned} E ζ δ t ≤ exp { 2 3 ∣ S ∣ log ( 2 Q ) + 2 ∣ S ∣ + ( e + 1 ) s + 2 1 ∥ x ⋆ ∥ 2 − q } ≤ exp { ( 2∣ S ∣ + 2∥ x ⋆ ∥ 2 + 4 s ) log ( 2 Q ) − q } ≤ ( 2 Q ) − 14 .
This proves (63) and completes the proof of Proposition 2.
Proof of Lemma 10
Mixture representation and linear tilts. Completing the square gives, for u ∈ R u\in\mathbb{R} u ∈ R ,
ω ( 2 π ) − 1 / 2 e u z − B 0 z 2 / 2 d z = k u N ( u v , v ) ( d z ) , k u : = ω v e u 2 v / 2 . \omega(2\pi)^{-1/2}e^{uz-B_0z^2/2}\,dz=k_u\mathcal{N}(uv,v)(dz),\qquad k_u:=\omega\sqrt{v}e^{u^2v/2}. ω ( 2 π ) − 1/2 e u z − B 0 z 2 /2 d z = k u N ( uv , v ) ( d z ) , k u := ω v e u 2 v /2 .
Set m : = θ v m:=\theta v m := θ v and γ : = N ( m , v I 2 ) \gamma:=\mathcal{N}(m,vI_2) γ := N ( m , v I 2 ) , and let γ i \gamma_i γ i be the law of X ∼ γ X\sim\gamma X ∼ γ with coordinate i i i replaced by zero. Expanding ρ θ 1 ⊗ ρ θ 2 \rho_{\theta_1}\otimes\rho_{\theta_2} ρ θ 1 ⊗ ρ θ 2 , removing the atom at the origin, and dividing by k θ 1 k θ 2 k_{\theta_1}k_{\theta_2} k θ 1 k θ 2 yields
μ θ = Z − 1 ( γ + λ 1 γ 1 + λ 2 γ 2 ) , λ i : = k θ i − 1 = β e − m i 2 / ( 2 v ) , Z : = 1 + λ 1 + λ 2 . \mu_\theta=\mathcal{Z}^{-1}(\gamma+\lambda_1\gamma_1+\lambda_2\gamma_2),\qquad\lambda_i:=k_{\theta_i}^{-1}=\beta e^{-m_i^2/(2v)},\qquad\mathcal{Z}:=1+\lambda_1+\lambda_2. μ θ = Z − 1 ( γ + λ 1 γ 1 + λ 2 γ 2 ) , λ i := k θ i − 1 = β e − m i 2 / ( 2 v ) , Z := 1 + λ 1 + λ 2 .
Multiplication by e η ⊤ z e^{\eta^\top z} e η ⊤ z changes θ \theta θ to θ + η \theta+\eta θ + η and preserves the restriction to R 2 ∖ { 0 } \mathbb{R}^2\setminus\{0\} R 2 ∖ { 0 } , proving item 1. It remains to bound the covariance and the smoothed Poincaré constant uniformly in m m m .
Covariance. The covariance is bounded by the second moment about m m m . Under γ \gamma γ this moment is v I 2 vI_2 v I 2 , while under γ i \gamma_i γ i it is v ( I 2 − e i e i T ) + m i 2 e i e i T v(I_2-e_i e_i^{\mathsf T})+m_i^2e_i e_i^{\mathsf T} v ( I 2 − e i e i T ) + m i 2 e i e i T , where e i e_i e i is the i i i th coordinate vector. Thus
Cov ( μ θ ) ⪯ v I 2 + ∑ i = 1 2 λ i Z m i 2 e i e i T . \operatorname{Cov}(\mu_\theta)\preceq vI_2+\sum_{i=1}^{2}\frac{\lambda_i}{Z}m_i^2e_i e_i^{\mathsf T}. Cov ( μ θ ) ⪯ v I 2 + i = 1 ∑ 2 Z λ i m i 2 e i e i T .
Writing ϱ i : = m i 2 / ( 2 v ) \varrho_i:=m_i^2/(2v) ϱ i := m i 2 / ( 2 v ) and using λ i / Z ≤ min { 1 , λ i } \lambda_i/Z\le\min\{1,\lambda_i\} λ i / Z ≤ min { 1 , λ i } , we obtain
λ i Z m i 2 ≤ 2 v ϱ i min { 1 , e log + β − ϱ i } ≤ 2 v ( 1 + log + β ) . \frac{\lambda_i}{Z}m_i^2\le2v\varrho_i\min\{1,e^{\log^+\beta-\varrho_i}\}\le2v(1+\log^+\beta). Z λ i m i 2 ≤ 2 v ϱ i min { 1 , e l o g + β − ϱ i } ≤ 2 v ( 1 + log + β ) .
For the last inequality, use ϱ i ≤ log + β \varrho_i\le\log^+\beta ϱ i ≤ log + β when this holds; otherwise write ϱ i = log + β + y \varrho_i=\log^+\beta+y ϱ i = log + β + y and use y e − y ≤ 1 ye^{-y}\le1 y e − y ≤ 1 for y ≥ 0 y\ge0 y ≥ 0 . Since the last matrix sum is diagonal, this proves Cov ( μ θ ) ⪯ v ( 3 + 2 log + β ) I 2 ⪯ 3 v ( 1 + log + β ) I 2 \operatorname{Cov}(\mu_\theta)\preceq v(3+2\log^+\beta)I_2\preceq3v(1+\log^+\beta)I_2 Cov ( μ θ ) ⪯ v ( 3 + 2 log + β ) I 2 ⪯ 3 v ( 1 + log + β ) I 2 , and hence item 2.
Smoothed Poincaré inequality. Let γ ~ : = γ ∗ N ( 0 , r I 2 ) = N ( m , ( v + r ) I 2 ) \widetilde{\gamma}:=\gamma*\mathcal{N}(0,rI_2)=\mathcal{N}(m,(v+r)I_2) γ := γ ∗ N ( 0 , r I 2 ) = N ( m , ( v + r ) I 2 ) and γ ~ i : = γ i ∗ N ( 0 , r I 2 ) \widetilde{\gamma}_i:=\gamma_i*\mathcal{N}(0,rI_2) γ i := γ i ∗ N ( 0 , r I 2 ) . The latter differs from γ ~ \widetilde{\gamma} γ only in coordinate i i i , where its law is N ( 0 , r ) \mathcal{N}(0,r) N ( 0 , r ) instead of N ( m i , v + r ) \mathcal{N}(m_i,v+r) N ( m i , v + r ) . The smoothed mixture is
μ ~ : = μ θ ∗ N ( 0 , r I 2 ) = Z − 1 ( γ ~ + λ 1 γ ~ 1 + λ 2 γ ~ 2 ) . \widetilde{\mu}:=\mu_\theta*\mathcal{N}(0,rI_2)=Z^{-1}(\widetilde{\gamma}+\lambda_1\widetilde{\gamma}_1+\lambda_2\widetilde{\gamma}_2). μ := μ θ ∗ N ( 0 , r I 2 ) = Z − 1 ( γ + λ 1 γ 1 + λ 2 γ 2 ) .
For a smooth compactly supported f f f , centering at γ ~ f \widetilde{\gamma}f γ f gives
Var μ ~ f ≤ μ ~ ( f − γ ~ f ) 2 = 1 Z { Var γ ~ f + ∑ i = 1 2 λ i [ Var γ ~ i f + ∣ γ ~ i f − γ ~ f ∣ 2 ] } . \operatorname{Var}_{\widetilde{\mu}}f\le\widetilde{\mu}(f-\widetilde{\gamma}f)^2=\frac{1}{Z}\left\{\operatorname{Var}_{\widetilde{\gamma}}f+\sum_{i=1}^{2}\lambda_i\left[\operatorname{Var}_{\widetilde{\gamma}_i}f+\lvert\widetilde{\gamma}_if-\widetilde{\gamma}f\rvert^2\right]\right\}. Var μ f ≤ μ ( f − γ f ) 2 = Z 1 { Var γ f + i = 1 ∑ 2 λ i [ Var γ i f + ∣ γ i f − γ f ∣ 2 ] } .
Each component has Poincaré constant at most v + r v+r v + r by 5(3). To control the differences of component means, Cauchy–Schwarz gives
∣ γ ~ i f − γ ~ f ∣ 2 = ∣ γ ~ [ ( d γ ~ i d γ ~ − 1 ) ( f − γ ~ f ) ] ∣ 2 ≤ χ 2 ( γ ~ i ∥ γ ~ ) Var γ ~ f . \lvert\widetilde{\gamma}_if-\widetilde{\gamma}f\rvert^2=\left|\widetilde{\gamma}\left[\left(\frac{d\widetilde{\gamma}_i}{d\widetilde{\gamma}}-1\right)(f-\widetilde{\gamma}f)\right]\right|^2\le\chi^2(\widetilde{\gamma}_i\mathbin{\|}\widetilde{\gamma})\operatorname{Var}_{\widetilde{\gamma}}f. ∣ γ i f − γ f ∣ 2 = γ [ ( d γ d γ i − 1 ) ( f − γ f ) ] 2 ≤ χ 2 ( γ i ∥ γ ) Var γ f .
Since Z − 1 γ ~ ≤ μ ~ Z^{-1}\widetilde{\gamma}\le\widetilde{\mu} Z − 1 γ ≤ μ , substitution into the variance bound yields
Var μ ~ f ≤ ( v + r ) ( 1 + ∑ i = 1 2 λ i χ 2 ( γ ~ i ∥ γ ~ ) ) μ ~ ∥ ∇ f ∥ 2 . \operatorname{Var}_{\widetilde{\mu}}f\le(v+r)\left(1+\sum_{i=1}^{2}\lambda_i\chi^2(\widetilde{\gamma}_i\mathbin{\|}\widetilde{\gamma})\right)\widetilde{\mu}\lVert\nabla f\rVert^2. Var μ f ≤ ( v + r ) ( 1 + i = 1 ∑ 2 λ i χ 2 ( γ i ∥ γ ) ) μ ∥ ∇ f ∥ 2 .
It remains to bound the weighted chi-square divergences uniformly in m m m . For 0 < a < 2 b 0<a<2b 0 < a < 2 b , completing the square in the one-dimensional Gaussian integral gives
∫ R φ a ( x ) 2 φ b ( x − m ‾ ) d x = b a ( 2 b − a ) exp ( m ‾ 2 2 b − a ) , \int_{\mathbb{R}}\frac{\varphi_a(x)^2}{\varphi_b(x-\overline{m})}\,dx=\frac{b}{\sqrt{a(2b-a)}}\exp\left(\frac{\overline{m}^2}{2b-a}\right), ∫ R φ b ( x − m ) φ a ( x ) 2 d x = a ( 2 b − a ) b exp ( 2 b − a m 2 ) ,
where φ a \varphi_a φ a denotes the N ( 0 , a ) \mathcal{N}(0,a) N ( 0 , a ) density. Apply this identity in coordinate i i i with a = r a=r a = r , b = v + r b=v+r b = v + r , and m ‾ = m i \overline{m}=m_i m = m i . Multiplying by λ i = β e − m i 2 / ( 2 v ) \lambda_i=\beta e^{-m_i^2/(2v)} λ i = β e − m i 2 / ( 2 v ) gives the cancellation
λ i [ 1 + χ 2 ( γ ~ i ∥ γ ~ ) ] = β ( v + r ) r ( 2 v + r ) exp ( − m i 2 r 2 v ( 2 v + r ) ) ≤ β v + r r . \lambda_i\left[1+\chi^2(\widetilde{\gamma}_i\mathbin{\|}\widetilde{\gamma})\right]=\frac{\beta(v+r)}{\sqrt{r(2v+r)}}\exp\left(-\frac{m_i^2r}{2v(2v+r)}\right)\le\beta\sqrt{\frac{v+r}{r}}. λ i [ 1 + χ 2 ( γ i ∥ γ ) ] = r ( 2 v + r ) β ( v + r ) exp ( − 2 v ( 2 v + r ) m i 2 r ) ≤ β r v + r .
The same bound holds for λ i χ 2 ( γ ~ i ∥ γ ~ ) \lambda_i\chi^2(\widetilde{\gamma}_i\mathbin{\|}\widetilde{\gamma}) λ i χ 2 ( γ i ∥ γ ) . Summing over the two axes therefore proves
Var μ ~ f ≤ [ ( v + r ) + 2 β ( v + r ) 3 / 2 r − 1 / 2 ] μ ~ ∥ ∇ f ∥ 2 , \operatorname{Var}_{\widetilde{\mu}}f\le\left[(v+r)+2\beta(v+r)^{3/2}r^{-1/2}\right]\widetilde{\mu}\lVert\nabla f\rVert^2, Var μ f ≤ [ ( v + r ) + 2 β ( v + r ) 3/2 r − 1/2 ] μ ∥ ∇ f ∥ 2 ,
which is item 3.
Proof of Lemma 11
We represent μ H \mu_H μ H as a mixture of linear tilts of μ \mu μ . The covariance hypothesis controls the mixing field, while the Poincaré hypothesis controls the smoothed conditional laws.
The auxiliary field and its normalizer. Finiteness of M \mathcal{M} M everywhere permits differentiation under the integral and gives ∇ 2 log M ( η ) = Cov ( μ η ) \nabla^2 \log\mathcal{M}(\eta)=\operatorname{Cov}(\mu^\eta) ∇ 2 log M ( η ) = Cov ( μ η ) . Define
Ψ ( g ) : = ∥ g ∥ 2 2 − log M ( H 1 / 2 g ) , ∇ 2 Ψ ( g ) = I − H 1 / 2 Cov ( μ H 1 / 2 g ) H 1 / 2 ⪰ ( 1 − v ˉ ) I . \Psi(g):=\frac{\lVert g\rVert^2}{2}-\log\mathcal{M}(H^{1/2}g),\qquad\nabla^2\Psi(g)=I-H^{1/2}\operatorname{Cov}(\mu^{H^{1/2}g})H^{1/2}\succeq(1-\bar{v})I. Ψ ( g ) := 2 ∥ g ∥ 2 − log M ( H 1/2 g ) , ∇ 2 Ψ ( g ) = I − H 1/2 Cov ( μ H 1/2 g ) H 1/2 ⪰ ( 1 − v ˉ ) I .
Since v ˉ < 1 \bar{v}<1 v ˉ < 1 ,
Ψ ( g ) ≥ Ψ ( 0 ) + ∇ Ψ ( 0 ) ⊤ g + 1 − v ˉ 2 ∥ g ∥ 2 , \Psi(g)\geq\Psi(0)+\nabla\Psi(0)^\top g+\frac{1-\bar{v}}{2}\lVert g\rVert^2, Ψ ( g ) ≥ Ψ ( 0 ) + ∇Ψ ( 0 ) ⊤ g + 2 1 − v ˉ ∥ g ∥ 2 ,
so e − Ψ e^{-\Psi} e − Ψ is integrable. Tonelli’s theorem and the Gaussian integral in g g g now give
∫ e − Ψ ( g ) d g = ∬ e − ∥ g ∥ 2 / 2 + g ⊤ H 1 / 2 w d g μ ( d w ) = ( 2 π ) d ˉ / 2 ∫ e w ⊤ H w / 2 μ ( d w ) < ∞ , \int e^{-\Psi(g)}\,dg=\iint e^{-\lVert g\rVert^2/2+g^\top H^{1/2}w}\,dg\,\mu(dw)=(2\pi)^{\bar{d}/2}\int e^{w^\top Hw/2}\,\mu(dw)<\infty, ∫ e − Ψ ( g ) d g = ∬ e − ∥ g ∥ 2 /2 + g ⊤ H 1/2 w d g μ ( d w ) = ( 2 π ) d ˉ /2 ∫ e w ⊤ H w /2 μ ( d w ) < ∞ ,
proving item 1. Normalize the joint measure e − ∥ g ∥ 2 / 2 + g ⊤ H 1 / 2 w d g μ ( d w ) e^{-\lVert g\rVert^2/2+g^\top H^{1/2}w}\,dg\,\mu(dw) e − ∥ g ∥ 2 /2 + g ⊤ H 1/2 w d g μ ( d w ) , and let ( g , w ) (g,w) ( g , w ) have this law. Then
w ∼ μ H , g ∼ e − Ψ ( g ) d g ∫ e − Ψ , w ∣ g ∼ μ η , η = H 1 / 2 g . w\sim\mu_H,\qquad g\sim\frac{e^{-\Psi(g)}\,dg}{\int e^{-\Psi}},\qquad w\mid g\sim\mu^\eta,\qquad\eta=H^{1/2}g. w ∼ μ H , g ∼ ∫ e − Ψ e − Ψ ( g ) d g , w ∣ g ∼ μ η , η = H 1/2 g .
By Lemma 5(5), the law of g g g has Poincaré constant at most ( 1 − v ˉ ) − 1 (1-\bar{v})^{-1} ( 1 − v ˉ ) − 1 .
Variance decomposition. Let f f f be smooth with compact support, let ξ ˇ \check{\xi} ξ ˇ be standard Gaussian and independent of ( g , w ) (g,w) ( g , w ) , and set w ~ : = w + r ξ ˇ \widetilde{w}:=w+\sqrt{r}\check{\xi} w := w + r ξ ˇ . Given g g g , the law of w ~ \widetilde{w} w is μ η ∗ N ( 0 , r I ) \mu^\eta* N(0,rI) μ η ∗ N ( 0 , r I ) , so
Var ( f ( w ~ ) ∣ g ) ≤ K 0 E [ ∥ ∇ f ( w ~ ) ∥ 2 ∣ g ] . \operatorname{Var}(f(\widetilde{w})\mid g)\leq K_0\mathbb{E}[\lVert\nabla f(\widetilde{w})\rVert^2\mid g]. Var ( f ( w ) ∣ g ) ≤ K 0 E [∥ ∇ f ( w ) ∥ 2 ∣ g ] .
To control the variance of the conditional mean f ˉ ( g ) : = E [ f ( w ~ ) ∣ g ] \bar{f}(g):=\mathbb{E}[f(\widetilde{w})\mid g] f ˉ ( g ) := E [ f ( w ) ∣ g ] , differentiate the tilt:
∇ f ˉ ( g ) = H 1 / 2 Cov ( w , f ( w ~ ) ∣ g ) . \nabla\bar{f}(g)=H^{1/2}\operatorname{Cov}(w,f(\widetilde{w})\mid g). ∇ f ˉ ( g ) = H 1/2 Cov ( w , f ( w ) ∣ g ) .
For every unit vector e e e , Cauchy–Schwarz and the covariance hypothesis imply
∣ e ⊤ ∇ f ˉ ( g ) ∣ 2 ≤ Var ( f ( w ~ ) ∣ g ) e ⊤ H 1 / 2 Cov ( w ∣ g ) H 1 / 2 e ≤ v ˉ Var ( f ( w ~ ) ∣ g ) . \lvert e^\top\nabla\bar{f}(g)\rvert^2\leq\operatorname{Var}(f(\widetilde{w})\mid g)e^\top H^{1/2}\operatorname{Cov}(w\mid g)H^{1/2}e\leq\bar{v}\operatorname{Var}(f(\widetilde{w})\mid g). ∣ e ⊤ ∇ f ˉ ( g ) ∣ 2 ≤ Var ( f ( w ) ∣ g ) e ⊤ H 1/2 Cov ( w ∣ g ) H 1/2 e ≤ v ˉ Var ( f ( w ) ∣ g ) .
Taking the supremum over unit vectors gives
∥ ∇ f ˉ ( g ) ∥ 2 ≤ v ˉ Var ( f ( w ~ ) ∣ g ) . \lVert\nabla\bar{f}(g)\rVert^2\leq\bar{v}\operatorname{Var}(f(\widetilde{w})\mid g). ∥ ∇ f ˉ ( g ) ∥ 2 ≤ v ˉ Var ( f ( w ) ∣ g ) .
In particular, f ˉ \bar{f} f ˉ is bounded and smooth with bounded gradient, so the Poincaré inequality for g g g applies. The conditional variance decomposition therefore yields
Var f ( w ~ ) = E Var ( f ( w ~ ) ∣ g ) + Var f ˉ ( g ) ≤ E Var ( f ( w ~ ) ∣ g ) + E ∥ ∇ f ˉ ( g ) ∥ 2 1 − v ˉ ≤ E Var ( f ( w ~ ) ∣ g ) 1 − v ˉ ≤ K 0 1 − v ˉ E ∥ ∇ f ( w ~ ) ∥ 2 . \begin{aligned}
\operatorname{Var}f(\widetilde{w})&=\mathbb{E}\operatorname{Var}(f(\widetilde{w})\mid g)+\operatorname{Var}\bar{f}(g)\\
&\leq\mathbb{E}\operatorname{Var}(f(\widetilde{w})\mid g)+\frac{\mathbb{E}\lVert\nabla\bar{f}(g)\rVert^2}{1-\bar{v}}\\
&\leq\frac{\mathbb{E}\operatorname{Var}(f(\widetilde{w})\mid g)}{1-\bar{v}}\leq\frac{K_0}{1-\bar{v}}\mathbb{E}\lVert\nabla f(\widetilde{w})\rVert^2.
\end{aligned} Var f ( w ) = E Var ( f ( w ) ∣ g ) + Var f ˉ ( g ) ≤ E Var ( f ( w ) ∣ g ) + 1 − v ˉ E ∥ ∇ f ˉ ( g ) ∥ 2 ≤ 1 − v ˉ E Var ( f ( w ) ∣ g ) ≤ 1 − v ˉ K 0 E ∥ ∇ f ( w ) ∥ 2 .
Since w ~ ∼ μ H ∗ N ( 0 , r I ) \widetilde{w}\sim\mu_H*N(0,rI) w ∼ μ H ∗ N ( 0 , r I ) , this proves item 2. The argument also covers singular H H H and r = 0 r=0 r = 0 , since it uses only H 1 / 2 H^{1/2} H 1/2 and independent Gaussian smoothing, which vanishes when r = 0 r=0 r = 0 .
Proof of Lemma 1
We prove Lemma 1 by verifying the hypotheses of Proposition 1 and establishing the bounds
E ζ E ≤ C ( τ + ε s c ( A , x ⋆ ) ) , E ζ E d e f ≤ C τ \mathbb{E}_{\zeta}\mathcal{E}\leq C\bigl(\tau+\varepsilon_{\mathrm{sc}}(A,x^\star)\bigr),\qquad\mathbb{E}_{\zeta}\mathcal{E}_{\mathrm{def}}\leq C\tau E ζ E ≤ C ( τ + ε sc ( A , x ⋆ ) ) , E ζ E def ≤ C τ
for its error terms in (27).
Step 1: verify the transfer hypotheses. The following lemma gives the required drift regularity. The proof of this lemma can be found in Appendix § 8 .3.1.
Lemma 12 (Drift). *For every matrix A A A with ∥ A ∥ 2 ≤ 2 n \lVert A\rVert^2 \le2n ∥ A ∥ 2 ≤ 2 n and τ ≤ t ≤ T \tau\le t \le T τ ≤ t ≤ T , the target drift ∇ log Π t \nabla\log\Pi_t ∇ log Π t in (7) is globally L ∗ L_\ast L ∗ -Lipschitz, where
L ∗ : = 2 τ + 2 ( n − s ) s τ 3 / 2 + n τ satisfies n L ∗ 2 ≤ C Q 5 . (66) L_\ast:= \frac{2}{\tau}+\frac{\sqrt{2}(n-s)}{s\,\tau^{3/2}}+\frac{n}{\tau}\qquad\text{satisfies}\qquad nL_\ast^2 \le CQ^5. \tag*{(66)} L ∗ := τ 2 + s τ 3/2 2 ( n − s ) + τ n satisfies n L ∗ 2 ≤ C Q 5 . (66)
The norm condition of Definition 5 gives ∥ A ∥ 2 ≤ 2 n \lVert A\rVert^2 \le2n ∥ A ∥ 2 ≤ 2 n , so Lemma 12 makes the drift of each Π t j \Pi_{t_j} Π t j L ∗ L_\ast L ∗ -Lipschitz. Meanwhile, Proposition 2 gives the weak variance inequality (62) for every fixed y y y . Since τ ≤ t ≤ T \tau\le t \le T τ ≤ t ≤ T and τ ≤ 1 ≤ s \tau\le1 \le s τ ≤ 1 ≤ s , we have 1 + t − 1 / 2 ≤ 2 τ − 1 / 2 1+t^{-1/2}\le2\tau^{-1/2} 1 + t − 1/2 ≤ 2 τ − 1/2 and n s τ − 1 / 2 ≤ n τ \frac{n}{s}\tau^{-1/2}\le\frac{n}{\tau} s n τ − 1/2 ≤ τ n , so (22) gives
V t ≤ T + 2 C n s τ − 1 / 2 ≤ ( 1 + 2 C ) Q ( τ ≤ t ≤ T ) V_t \le T+2C\frac{n}{s}\tau^{-1/2}\le(1+2C)Q\qquad(\tau\le t\le T) V t ≤ T + 2 C s n τ − 1/2 ≤ ( 1 + 2 C ) Q ( τ ≤ t ≤ T )
for some constant C C C . Set C V : = 1 + 2 C C_V:=1+2C C V := 1 + 2 C and V : = C V Q V:=C_VQ V := C V Q . Since (25) persists when its variance constant increases, each Π t j \Pi_{t_j} Π t j satisfies it with the common constant V V V and defect δ j ( y ) : = δ t j ( y ) \delta_j(y):=\delta_{t_j}(y) δ j ( y ) := δ t j ( y ) . The containment bound (63) gives
E ζ δ j ( A x ⋆ + σ ζ ) ≤ ( 2 Q ) − 14 j < N . \mathbb{E}_{\zeta}\delta_j(Ax^\star+\sigma\zeta)\le(2Q)^{-14}\qquad j<N. E ζ δ j ( A x ⋆ + σ ζ ) ≤ ( 2 Q ) − 14 j < N .
Together with the drift bound, this verifies the hypotheses of Proposition 1 for every fixed instance ( A , y ) (A,y) ( A , y ) , with L = L ∗ L=L_\ast L = L ∗ and V = C V Q V=C_VQ V = C V Q .
Step 2: bound the averaged errors. The following two lemmas control the initialization error and target movement along the annealing schedule.
Lemma 13 (Initialization). *For a signal x ⋆ x^\star x ⋆ with ∥ x ⋆ ∥ 2 ≤ n \lVert x^\star\rVert^2\le n ∥ x ⋆ ∥ 2 ≤ n and y = A x ⋆ + σ ζ y=Ax^\star+\sigma\zeta y = A x ⋆ + σ ζ ,
E ζ H 2 ( Π T , μ i n i t ) ≤ n T = τ . (67) \mathbb{E}_{\zeta}H^2(\Pi_T,\mu_{\mathrm{init}})\le\frac{n}{T}=\tau. \tag*{(67)} E ζ H 2 ( Π T , μ init ) ≤ T n = τ . (67)
Lemma 14 (Target movement). *For two levels τ ≤ t ′ ≤ t ≤ T \tau\le t'\le t\le T τ ≤ t ′ ≤ t ≤ T and every matrix A A A and every y ∈ R m y\in\mathbb{R}^m y ∈ R m ,
H 2 ( Π t , Π t ′ ) ≤ 1 2 log 2 ( t / t ′ ) ( n + R ( y ) ) 2 , R ( y ) : = y T ( a τ I m + τ A A T ) − 1 y . (68) H^2(\Pi_t,\Pi_{t'})\le\frac{1}{2}\log^2(t/t')(n+R(y))^2,\qquad R(y):=y^\mathsf{T}(a_\tau I_m+\tau AA^\mathsf{T})^{-1}y. \tag*{(68)} H 2 ( Π t , Π t ′ ) ≤ 2 1 log 2 ( t / t ′ ) ( n + R ( y ) ) 2 , R ( y ) := y T ( a τ I m + τ A A T ) − 1 y . (68)
*If x ⋆ x^\star x ⋆ is a signal with ∥ x ⋆ ∥ 2 ≤ n \lVert x^\star\rVert^2\le n ∥ x ⋆ ∥ 2 ≤ n and y = A x ⋆ + σ ζ y=Ax^\star+\sigma\zeta y = A x ⋆ + σ ζ , then E ζ ( n + R ( y ) ) 2 ≤ C Q 2 \mathbb{E}_{\zeta}(n+R(y))^2\le CQ^2 E ζ ( n + R ( y ) ) 2 ≤ C Q 2 . Consequently the movements between consecutive targets of the sampler satisfy the movement bound
E ζ H 2 ( Π t j , Π t j + 1 ) ≤ C Q 3 N 2 , j < N . (69) \mathbb{E}_{\zeta}H^2(\Pi_{t_j},\Pi_{t_{j+1}})\le\frac{CQ^3}{N^2},\qquad j<N. \tag*{(69)} E ζ H 2 ( Π t j , Π t j + 1 ) ≤ N 2 C Q 3 , j < N . (69)
The proof of the above two lemmas can be found in Appendix § 8 .3.2 and Appendix § 8 .3.3.
By averaging the error decomposition in Proposition 1 over the measurement noise, and combining the above lemmas, we obtain, using h = N − 2 / 3 h=N^{-2/3} h = N − 2/3 ,
E ζ E ≤ τ + E s c ( A , x ⋆ ) + C Q 5 N h 2 + C Q 4 N h = τ + E s c ( A , x ⋆ ) + C Q 5 N − 1 / 3 + C Q 4 N − 1 / 3 . \begin{aligned}
\mathbb{E}_{\zeta}\mathcal{E}\le\tau+\mathcal{E}_{\mathrm{sc}}(A,x^\star)+CQ^5Nh^2+\frac{CQ^4}{Nh} \\
&=\tau+\mathcal{E}_{\mathrm{sc}}(A,x^\star)+CQ^5N^{-1/3}+CQ^4N^{-1/3}.
\end{aligned} E ζ E ≤ τ + E sc ( A , x ⋆ ) + C Q 5 N h 2 + N h C Q 4 = τ + E sc ( A , x ⋆ ) + C Q 5 N − 1/3 + C Q 4 N − 1/3 .
Since N ≥ C 0 Q 18 N\ge C_0Q^{18} N ≥ C 0 Q 18 with C 0 C_0 C 0 sufficiently large, the last two terms sum to at most Q − 1 ≤ τ Q^{-1}\le\tau Q − 1 ≤ τ , by (22) . Consequently,
E ζ E ≤ 2 τ + E s c ( A , x ⋆ ) . \mathbb{E}_{\zeta}\mathcal{E}\le2\tau+\mathcal{E}_{\mathrm{sc}}(A,x^\star). E ζ E ≤ 2 τ + E sc ( A , x ⋆ ) .
For the defect term, the Cauchy–Schwarz inequality gives
E ζ E d e f = ∑ j < N E ζ δ j H 2 ( Π t j , Π t j + 1 ) ≤ ∑ j < N ( E ζ δ j ) 1 / 2 ( E ζ H 2 ( Π t j , Π t j + 1 ) ) 1 / 2 ≤ N ( 2 Q ) − 7 C Q 3 / 2 N ≤ C Q ≤ C τ . \begin{aligned}
\mathbb{E}_{\zeta}\mathcal{E}_{\mathrm{def}}
=\sum_{j<N}\mathbb{E}_{\zeta}\sqrt{\delta_jH^2(\Pi_{t_j},\Pi_{t_{j+1}})} \\
&\le\sum_{j<N}(\mathbb{E}_{\zeta}\delta_j)^{1/2}(\mathbb{E}_{\zeta}H^2(\Pi_{t_j},\Pi_{t_{j+1}}))^{1/2}
\le N(2Q)^{-7}\frac{CQ^{3/2}}{N}\le\frac{C}{Q}\le C\tau.
\end{aligned} E ζ E def = j < N ∑ E ζ δ j H 2 ( Π t j , Π t j + 1 ) ≤ j < N ∑ ( E ζ δ j ) 1/2 ( E ζ H 2 ( Π t j , Π t j + 1 ) ) 1/2 ≤ N ( 2 Q ) − 7 N C Q 3/2 ≤ Q C ≤ C τ .
Step 3: deduce the two conclusions. Averaging (26) over the noise and inserting these two bounds gives (35) , since λ ≤ 1 \lambda\le1 λ ≤ 1 .
To prove (36) , we take the supremum over events at fixed ( A , y ) (A,y) ( A , y ) before averaging. Applying (26) to an indicator g g g and its complement, and using 0 ≤ Π τ g ≤ 1 0 \le\Pi_{\tau}g \le1 0 ≤ Π τ g ≤ 1 , gives
∣ ρ N g − Π τ g ∣ ≤ λ + C E λ + C E d e f . \lvert\rho_N g-\Pi_{\tau}g\rvert\le\lambda+\frac{C\mathcal{E}}{\lambda}+C\mathcal{E}_{\mathrm{def}}. ∣ ρ N g − Π τ g ∣ ≤ λ + λ C E + C E def .
For 0 < E ≤ 1 0<\mathcal{E}\le1 0 < E ≤ 1 , choose λ = E \lambda=\sqrt{\mathcal{E}} λ = E ; when E = 0 \mathcal{E}=0 E = 0 , let λ ↓ 0 \lambda\downarrow0 λ ↓ 0 . Taking the supremum over indicators gives
TV ( ρ N , Π τ ) ≤ C E + C E d e f . \operatorname{TV}(\rho_N,\Pi_{\tau})\le C\sqrt{\mathcal{E}}+C\mathcal{E}_{\mathrm{def}}. TV ( ρ N , Π τ ) ≤ C E + C E def .
This bound also holds when E > 1 \mathcal{E}>1 E > 1 , since total variation is at most one. Averaging over the noise and applying Jensen’s inequality now give
E ζ TV ( ρ N , Π τ ) ≤ C ( E ζ E ) 1 / 2 + C E ζ E d e f ≤ C ( 2 τ + ε s c ( A , x ⋆ ) ) 1 / 2 + C τ , \mathbb{E}_{\zeta}\operatorname{TV}(\rho_N,\Pi_{\tau})\le C\left(\mathbb{E}_{\zeta}\mathcal{E}\right)^{1/2}+C\mathbb{E}_{\zeta}\mathcal{E}_{\mathrm{def}}\le C\left(2\tau+\varepsilon_{\mathrm{sc}}(A,x^{\star})\right)^{1/2}+C\tau, E ζ TV ( ρ N , Π τ ) ≤ C ( E ζ E ) 1/2 + C E ζ E def ≤ C ( 2 τ + ε sc ( A , x ⋆ ) ) 1/2 + C τ ,
and τ ≤ τ 1 / 2 \tau\le\tau^{1/2} τ ≤ τ 1/2 , as τ ≤ 1 \tau\le1 τ ≤ 1 , gives (36) .
Proof of Lemma 12
By (7) , it suffices to bound the Hessian of the log prior and the Jacobian of the residual term. Since p t p_t p t is a product measure, ∇ 2 log p t \nabla^2\log p_t ∇ 2 log p t is diagonal, with entries given by the derivative of the scalar mixture score.
Write s t s_t s t for the logarithmic derivative of the density of ( 1 − s / n ) N ( 0 , t ) + ( s / n ) N ( 0 , 1 + t ) (1-s/n)\mathcal{N}(0,t)+(s/n)\mathcal{N}(0,1+t) ( 1 − s / n ) N ( 0 , t ) + ( s / n ) N ( 0 , 1 + t ) , and set
v : = x 2 2 t ( 1 + t ) , ϱ t : = ω t 1 + t , ς : = ς t ( x ) = ϱ t e v 1 + ϱ t e v , v:=\frac{x^2}{2t(1+t)},\qquad\varrho_t:=\omega\sqrt{\frac{t}{1+t}},\qquad\varsigma:=\varsigma_t(x)=\frac{\varrho_t e^v}{1+\varrho_t e^v}, v := 2 t ( 1 + t ) x 2 , ϱ t := ω 1 + t t , ς := ς t ( x ) = 1 + ϱ t e v ϱ t e v ,
where ω = s / ( n − s ) \omega=s/(n-s) ω = s / ( n − s ) and ς t \varsigma_t ς t is the weight of the second component at x x x , as in (18) . Direct differentiation gives
s t ( x ) = − x t + x ς t ( 1 + t ) , s t ′ ( x ) = − 1 t + ς + 2 v ς ( 1 − ς ) t ( 1 + t ) . s_t(x)=-\frac{x}{t}+\frac{x\varsigma}{t(1+t)},\qquad s_t'(x)=-\frac{1}{t}+\frac{\varsigma+2v\varsigma(1-\varsigma)}{t(1+t)}. s t ( x ) = − t x + t ( 1 + t ) x ς , s t ′ ( x ) = − t 1 + t ( 1 + t ) ς + 2 v ς ( 1 − ς ) .
Since 0 ≤ ς ≤ 1 0\le\varsigma\le1 0 ≤ ς ≤ 1 , ς ( 1 − ς ) ≤ ϱ t − 1 e − v \varsigma(1-\varsigma)\le\varrho_t^{-1}e^{-v} ς ( 1 − ς ) ≤ ϱ t − 1 e − v and v e − v ≤ 1 / e ve^{-v}\le1/e v e − v ≤ 1/ e ,
∣ s t ′ ( x ) ∣ ≤ 1 t + 1 + 2 / ( e ϱ t ) t ( 1 + t ) ≤ 2 + ϱ t − 1 t ≤ 2 τ + 2 ( n − s ) s τ 3 / 2 . \lvert s_t'(x)\rvert\le\frac{1}{t}+\frac{1+2/(e\varrho_t)}{t(1+t)}\le\frac{2+\varrho_t^{-1}}{t}\le\frac{2}{\tau}+\frac{\sqrt{2}(n-s)}{s\tau^{3/2}}. ∣ s t ′ ( x )∣ ≤ t 1 + t ( 1 + t ) 1 + 2/ ( e ϱ t ) ≤ t 2 + ϱ t − 1 ≤ τ 2 + s τ 3/2 2 ( n − s ) .
The last inequality uses t ≥ τ t\ge\tau t ≥ τ , τ ≤ 1 \tau\le1 τ ≤ 1 and ϱ t − 1 = ω − 1 1 + 1 / t ≤ 2 / τ ω − 1 \varrho_t^{-1}=\omega^{-1}\sqrt{1+1/t}\le\sqrt{2/\tau}\,\omega^{-1} ϱ t − 1 = ω − 1 1 + 1/ t ≤ 2/ τ ω − 1 .
The residual term has Jacobian − A T A / ( 2 t ) -A^{\mathsf T}A/(2t) − A T A / ( 2 t ) , whose norm is at most ∥ A ∥ 2 / ( 2 t ) ≤ n / τ \lVert A\rVert^2/(2t)\le n/\tau ∥ A ∥ 2 / ( 2 t ) ≤ n / τ . Adding this to the preceding bound shows that ∥ ∇ 2 log Π t ( x ) ∥ ≤ L ⋆ \lVert\nabla^2\log\Pi_t(x)\rVert\le L_{\star} ∥ ∇ 2 log Π t ( x )∥ ≤ L ⋆ for every x x x , and hence that the drift is globally L ⋆ L_{\star} L ⋆ -Lipschitz.
Finally, (22) gives n / τ ≤ Q n/\tau\le Q n / τ ≤ Q , 1 / τ ≤ Q 1/\tau\le Q 1/ τ ≤ Q and ω − 1 ≤ n / s ≤ Q \omega^{-1}\le n/s\le Q ω − 1 ≤ n / s ≤ Q . Thus
n L ⋆ = 2 n τ + 2 n τ ω − 1 τ + n τ n ≤ 2 Q + 2 Q 5 / 2 + Q 3 / 2 ≤ C Q 5 / 2 , \sqrt{n}L_{\star}=\frac{2\sqrt{n}}{\tau}+\sqrt{\frac{2n}{\tau}}\frac{\omega^{-1}}{\tau}+\frac{n}{\tau}\sqrt{n}\le2Q+\sqrt{2}Q^{5/2}+Q^{3/2}\le CQ^{5/2}, n L ⋆ = τ 2 n + τ 2 n τ ω − 1 + τ n n ≤ 2 Q + 2 Q 5/2 + Q 3/2 ≤ C Q 5/2 ,
and squaring gives (66) .
Proof of Lemma 13
The forward-kernel representation reduces the initialization error to a second moment of the latent law. Indeed, by (49) ,
γ z : = F T ( z , ⋅ ) = N ( M T z + T b T , T M T ) . \gamma_z:=\mathcal{F}_T(z,\cdot)=\mathcal{N}(M_Tz+Tb_T,TM_T). γ z := F T ( z , ⋅ ) = N ( M T z + T b T , T M T ) .
The target Π T \Pi_T Π T is the mixture of γ z \gamma_z γ z over z ∼ P T z\sim P_T z ∼ P T by (48) , whereas μ i n i t = γ 0 \mu_{\mathrm{init}}=\gamma_0 μ init = γ 0 by (11) . These Gaussians have common covariance T M T TM_T T M T and mean difference M T z M_Tz M T z . Since 0 ≺ M T ⪯ I n 0\prec M_T\preceq I_n 0 ≺ M T ⪯ I n , the mixture inequality and Gaussian formula in Lemma 4 give
H 2 ( Π T , μ i n i t ) ≤ ∫ H 2 ( γ z , γ 0 ) P T ( d z ) ≤ 1 4 ∫ ( M T z ) T ( T M T ) − 1 ( M T z ) P T ( d z ) = E P T [ Z T M T Z ] 4 T ≤ E P T ∥ Z ∥ 2 4 T . \begin{aligned}
\mathrm{H}^2(\Pi_T,\mu_{\mathrm{init}})\le\int\mathrm{H}^2(\gamma_z,\gamma_0)\,P_T(dz) \\
&\le\frac{1}{4}\int(M_Tz)^{\mathsf T}(TM_T)^{-1}(M_Tz)\,P_T(dz)=\frac{\mathbb{E}_{P_T}[Z^{\mathsf T}M_TZ]}{4T}\le\frac{\mathbb{E}_{P_T}\lVert Z\rVert^2}{4T}.
\end{aligned} H 2 ( Π T , μ init ) ≤ ∫ H 2 ( γ z , γ 0 ) P T ( d z ) ≤ 4 1 ∫ ( M T z ) T ( T M T ) − 1 ( M T z ) P T ( d z ) = 4 T E P T [ Z T M T Z ] ≤ 4 T E P T ∥ Z ∥ 2 .
To bound this second moment, condition on the latent support S T = W S_T = W S T = W . By (53) and B W W = ( B T ) W W ⪰ I W B_{WW} = (B_T)_{WW} \succeq I_W B W W = ( B T ) W W ⪰ I W ,
E Π T [ ∥ Z ∥ 2 ∣ S T = W ] = tr B W W − 1 + ∥ B W W − 1 ( b T ) W ∥ 2 ≤ ∣ W ∣ + ∥ b T ∥ 2 ≤ n + ∥ b T ∥ 2 . \mathbb{E}_{\Pi_T}\left[\lVert Z\rVert^2 \mid S_T = W\right] = \operatorname{tr} B_{WW}^{-1} + \left\lVert B_{WW}^{-1}(b_T)_W\right\rVert^2 \leq|W| + \lVert b_T\rVert^2 \leq n + \lVert b_T\rVert^2. E Π T [ ∥ Z ∥ 2 ∣ S T = W ] = tr B W W − 1 + B W W − 1 ( b T ) W 2 ≤ ∣ W ∣ + ∥ b T ∥ 2 ≤ n + ∥ b T ∥ 2 .
For W = ∅ W = \varnothing W = ∅ , the conditional second moment is zero, so the same bound holds. Averaging over W W W yields
H 2 ( Π T , μ i n i t ) ≤ n + ∥ b T ∥ 2 4 T . H^2(\Pi_T,\mu_{\mathrm{init}}) \leq\frac{n+\lVert b_T\rVert^2}{4T}. H 2 ( Π T , μ init ) ≤ 4 T n + ∥ b T ∥ 2 .
It remains to average ∥ b T ∥ 2 \lVert b_T\rVert^2 ∥ b T ∥ 2 over the observation noise. Substituting y = A x ⋆ + σ ζ y = Ax^\star+ \sigma\zeta y = A x ⋆ + σ ζ gives b T = G T x ⋆ + σ A ⊤ Σ T − 1 ζ b_T = G_Tx^\star+ \sigma A^\top\Sigma_T^{-1}\zeta b T = G T x ⋆ + σ A ⊤ Σ T − 1 ζ . The identities for M T M_T M T imply 0 ⪯ G T = T − 1 ( I n − M T ) ⪯ T − 1 I n 0 \preceq G_T = T^{-1}(I_n-M_T) \preceq T^{-1}I_n 0 ⪯ G T = T − 1 ( I n − M T ) ⪯ T − 1 I n , while Σ T ⪰ σ 2 I m \Sigma_T \succeq\sigma^2 I_m Σ T ⪰ σ 2 I m gives σ 2 A ⊤ Σ T − 2 A ⪯ G T \sigma^2 A^\top\Sigma_T^{-2}A \preceq G_T σ 2 A ⊤ Σ T − 2 A ⪯ G T . Consequently, using ∥ x ⋆ ∥ 2 ≤ n \lVert x^\star\rVert^2 \leq n ∥ x ⋆ ∥ 2 ≤ n and T = n / τ ≥ 1 T=n/\tau\geq1 T = n / τ ≥ 1 ,
E ζ ∥ b T ∥ 2 = ∥ G T x ⋆ ∥ 2 + σ 2 tr ( A ⊤ Σ T − 2 A ) ≤ ∥ x ⋆ ∥ 2 T 2 + tr G T ≤ n T 2 + n T ≤ 2 n , \mathbb{E}_{\zeta}\lVert b_T\rVert^2 = \lVert G_Tx^\star\rVert^2 + \sigma^2\operatorname{tr}(A^\top\Sigma_T^{-2}A) \leq\frac{\lVert x^\star\rVert^2}{T^2}+\operatorname{tr}G_T \leq\frac{n}{T^2}+\frac{n}{T}\leq2n, E ζ ∥ b T ∥ 2 = ∥ G T x ⋆ ∥ 2 + σ 2 tr ( A ⊤ Σ T − 2 A ) ≤ T 2 ∥ x ⋆ ∥ 2 + tr G T ≤ T 2 n + T n ≤ 2 n ,
and hence E ζ H 2 ( Π T , μ i n i t ) ≤ 3 n / ( 4 T ) ≤ n / T \mathbb{E}_{\zeta}H^2(\Pi_T,\mu_{\mathrm{init}})\leq3n/(4T)\leq n/T E ζ H 2 ( Π T , μ init ) ≤ 3 n / ( 4 T ) ≤ n / T , as claimed.
Proof of Lemma 14
We control the target movement through the joint conditional law of ( Z , X r ) (Z,X_r) ( Z , X r ) given Y ~ r = y \widetilde{Y}_r=y Y r = y in the auxiliary model of Appendix § 7 .§ 2 .
Step 1: reduce the movement to a residual moment. For r ∈ [ t ′ , t ] r\in[t',t] r ∈ [ t ′ , t ] , define the joint density with respect to ν 0 ( d z , d x ) : = p ⋆ ( d z ) d x \nu_0(dz,dx):=p_\star(dz)\,dx ν 0 ( d z , d x ) := p ⋆ ( d z ) d x by
J r ( z , x ) : = exp { − ∥ x − z ∥ 2 2 r − ∥ A x − y ∥ 2 2 a r } , Z r : = ∫ J r d ν 0 , ϖ r : = J r / Z r . J_r(z,x):=\exp\left\{-\frac{\lVert x-z\rVert^2}{2r}-\frac{\lVert Ax-y\rVert^2}{2a_r}\right\},\qquad Z_r:=\int J_r\,d\nu_0,\qquad\varpi_r:=J_r/Z_r. J r ( z , x ) := exp { − 2 r ∥ x − z ∥ 2 − 2 a r ∥ A x − y ∥ 2 } , Z r := ∫ J r d ν 0 , ϖ r := J r / Z r .
The Gaussian prefactors depend only on r r r and cancel upon normalization, so ϖ r \varpi_r ϖ r is the conditional density of ( Z , X r ) (Z,X_r) ( Z , X r ) given Y ~ r = y \widetilde{Y}_r=y Y r = y . Its x x x -marginal is Π r \Pi_r Π r ; hence (45) gives
H 2 ( Π t , Π t ′ ) ≤ ∥ ϖ t − ϖ t ′ ∥ L 2 ( ν 0 ) 2 . H^2(\Pi_t,\Pi_{t'})\leq\left\lVert\sqrt{\varpi_t}-\sqrt{\varpi_{t'}}\right\rVert_{L^2(\nu_0)}^2. H 2 ( Π t , Π t ′ ) ≤ ∥ ϖ t − ϖ t ′ ∥ L 2 ( ν 0 ) 2 .
The derivative of log J r \log J_r log J r is controlled by the normalized residual energy
R r : = ∥ x − z ∥ 2 r + ∥ A x − y ∥ 2 a r . R_r:=\frac{\lVert x-z\rVert^2}{r}+\frac{\lVert Ax-y\rVert^2}{a_r}. R r := r ∥ x − z ∥ 2 + a r ∥ A x − y ∥ 2 .
Indeed, d a r / d r = 2 da_r/dr=2 d a r / d r = 2 and r / a r ≤ 1 / 2 r/a_r\leq1/2 r / a r ≤ 1/2 give
0 ≤ χ r : = ∂ log r log J r = ∥ x − z ∥ 2 2 r + r a r ∥ A x − y ∥ 2 a r ≤ 1 2 R r . 0\leq\chi_r:=\partial_{\log r}\log J_r=\frac{\lVert x-z\rVert^2}{2r}+\frac{r}{a_r}\frac{\lVert Ax-y\rVert^2}{a_r}\leq\frac{1}{2}R_r. 0 ≤ χ r := ∂ l o g r log J r = 2 r ∥ x − z ∥ 2 + a r r a r ∥ A x − y ∥ 2 ≤ 2 1 R r .
The Gaussian tails on each support, uniformly over r ∈ [ t ′ , t ] ⊂ ( 0 , ∞ ) r\in[t',t]\subset(0,\infty) r ∈ [ t ′ , t ] ⊂ ( 0 , ∞ ) , justify differentiation under the integral and in L 2 ( ν 0 ) L^2(\nu_0) L 2 ( ν 0 ) . Thus ∂ log r log Z r = E ϖ r χ r \partial_{\log r}\log Z_r=\mathbb{E}_{\varpi_r}\chi_r ∂ l o g r log Z r = E ϖ r χ r , where E ϖ r \mathbb{E}_{\varpi_r} E ϖ r denotes expectation under the joint density, and
∂ log r ϖ r = 1 2 ϖ r ( χ r − E ϖ r χ r ) , ∥ ∂ log r ϖ r ∥ L 2 ( ν 0 ) 2 = 1 4 Var ϖ r ( χ r ) ≤ 1 16 E ϖ r R r 2 . \partial_{\log r}\sqrt{\varpi_r}=\frac{1}{2}\sqrt{\varpi_r}\left(\chi_r-\mathbb{E}_{\varpi_r}\chi_r\right),\qquad\left\lVert\partial_{\log r}\sqrt{\varpi_r}\right\rVert_{L^2(\nu_0)}^2=\frac{1}{4}\operatorname{Var}_{\varpi_r}(\chi_r)\leq\frac{1}{16}\mathbb{E}_{\varpi_r}R_r^2. ∂ l o g r ϖ r = 2 1 ϖ r ( χ r − E ϖ r χ r ) , ∥ ∂ l o g r ϖ r ∥ L 2 ( ν 0 ) 2 = 4 1 Var ϖ r ( χ r ) ≤ 16 1 E ϖ r R r 2 .
The fundamental theorem of calculus in log r \log r log r and Cauchy–Schwarz therefore yield
H 2 ( Π t , Π t ′ ) ≤ log t t ′ ∫ t ′ t ∥ ∂ log r ϖ r ∥ L 2 ( ν 0 ) 2 d r r ≤ log ( t / t ′ ) 16 ∫ t ′ t E ϖ r R r 2 d r r . \begin{aligned}
H^2(\Pi_t,\Pi_{t'})\leq\log\frac{t}{t'}\int_{t'}^t\left\lVert\partial_{\log r}\sqrt{\varpi_r}\right\rVert_{L^2(\nu_0)}^2\frac{dr}{r} \\
&\leq\frac{\log(t/t')}{16}\int_{t'}^t\mathbb{E}_{\varpi_r}R_r^2\frac{dr}{r}.
\end{aligned} H 2 ( Π t , Π t ′ ) ≤ log t ′ t ∫ t ′ t ∥ ∂ l o g r ϖ r ∥ L 2 ( ν 0 ) 2 r d r ≤ 16 log ( t / t ′ ) ∫ t ′ t E ϖ r R r 2 r d r .
It remains to prove E ϖ r R r 2 ≤ 8 ( n + R ( y ) ) 2 \mathbb{E}_{\varpi_r}R_r^2\leq8(n+\mathcal{R}(y))^2 E ϖ r R r 2 ≤ 8 ( n + R ( y ) ) 2 uniformly in r ∈ [ t ′ , t ] r\in[t',t] r ∈ [ t ′ , t ] . Step 2: bound the residual fourth moment. Under the conditional law defining ϖ r \varpi_r ϖ r , the auxiliary model gives X r − Z = r ξ X_r-Z=\sqrt{r}\,\xi X r − Z = r ξ and y − A X r = a r ξ ′ y-AX_r=\sqrt{a_r}\,\xi' y − A X r = a r ξ ′ , so R r = ∥ ξ ∥ 2 + ∥ ξ ′ ∥ 2 R_r=\|\xi\|^2+\|\xi'\|^2 R r = ∥ ξ ∥ 2 + ∥ ξ ′ ∥ 2 . Before conditioning on the observation, given supp Z = W \operatorname{supp}Z=W supp Z = W , the vector z ~ : = ( Z W , ξ , ξ ′ ) \widetilde z:=(Z_W,\xi,\xi') z := ( Z W , ξ , ξ ′ ) is standard Gaussian and
Y ~ r = F z ~ , F : = [ A W , r A , a r I m ] . \widetilde Y_r=F\widetilde z,\qquad F:=[A_W,\sqrt{r}A,\sqrt{a_r}I_m]. Y r = F z , F := [ A W , r A , a r I m ] .
Since F F F has rank m m m , Gaussian conditioning on Y ~ r = y \widetilde Y_r=y Y r = y gives
z ~ = z ~ 0 + g , z ~ 0 : = F T ( F F T ) − 1 y , g ∼ N ( 0 , P ker F ) , \widetilde z=\widetilde z_0+g,\qquad\widetilde z_0:=F^\mathsf{T}(FF^\mathsf{T})^{-1}y,\qquad g\sim\mathcal{N}(0,P_{\ker F}), z = z 0 + g , z 0 := F T ( F F T ) − 1 y , g ∼ N ( 0 , P k e r F ) ,
where P ker F P_{\ker F} P k e r F is the orthogonal projection onto ker F \ker F ker F . The mean z ~ 0 \widetilde z_0 z 0 is orthogonal to ker F \ker F ker F , and F F T = A W A W T + Σ r ⪰ Σ τ FF^\mathsf{T}=A_WA_W^\mathsf{T}+\Sigma_r\succeq\Sigma_\tau F F T = A W A W T + Σ r ⪰ Σ τ . Consequently,
R r ≤ ∥ z ~ ∥ 2 = ∥ z ~ 0 ∥ 2 + ∥ g ∥ 2 , ∥ z ~ 0 ∥ 2 = y T ( F F T ) − 1 y ≤ R ( y ) . R_r\leq\|\widetilde z\|^2=\|\widetilde z_0\|^2+\|g\|^2,\qquad\|\widetilde z_0\|^2=y^\mathsf{T}(FF^\mathsf{T})^{-1}y\leq R(y). R r ≤ ∥ z ∥ 2 = ∥ z 0 ∥ 2 + ∥ g ∥ 2 , ∥ z 0 ∥ 2 = y T ( F F T ) − 1 y ≤ R ( y ) .
Moreover, ∥ g ∥ 2 \|g\|^2 ∥ g ∥ 2 is chi-square with dim ker F = ∣ W ∣ + n ≤ 2 n \dim\ker F=|W|+n\leq2n dim ker F = ∣ W ∣ + n ≤ 2 n degrees of freedom. Its first two moments therefore give
E [ R r 2 ∣ Y ~ r = y , supp Z = W ] ≤ E ( ∥ z ~ 0 ∥ 2 + ∥ g ∥ 2 ) 2 ≤ ( R ( y ) + ∣ W ∣ + n ) 2 + 2 ( ∣ W ∣ + n ) \begin{aligned}
\mathbb{E}[R_r^2\mid\widetilde Y_r=y,\operatorname{supp}Z=W]\leq\mathbb{E}\bigl(\|\widetilde z_0\|^2+\|g\|^2\bigr)^2 \\
&\leq\bigl(R(y)+|W|+n\bigr)^2+2(|W|+n)
\end{aligned} E [ R r 2 ∣ Y r = y , supp Z = W ] ≤ E ( ∥ z 0 ∥ 2 + ∥ g ∥ 2 ) 2 ≤ ( R ( y ) + ∣ W ∣ + n ) 2 + 2 ( ∣ W ∣ + n )
≤ 8 ( n + R ( y ) ) 2 . \leq8(n+R(y))^2. ≤ 8 ( n + R ( y ) ) 2 .
This also covers W = ∅ W=\varnothing W = ∅ . Averaging over the conditional support law establishes the required moment bound. Substitution into Step 1 gives
H 2 ( Π t , Π t ′ ) ≤ 1 2 log 2 ( t / t ′ ) ( n + R ( y ) ) 2 , H^2(\Pi_t,\Pi_{t'})\leq\frac{1}{2}\log^2(t/t')(n+R(y))^2, H 2 ( Π t , Π t ′ ) ≤ 2 1 log 2 ( t / t ′ ) ( n + R ( y ) ) 2 ,
which proves (68) .
Step 3: average over the noise and apply the schedule. Let y = A x ⋆ + σ ζ y=Ax^\star+\sigma\zeta y = A x ⋆ + σ ζ , with ∥ x ⋆ ∥ 2 ≤ n \|x^\star\|^2\leq n ∥ x ⋆ ∥ 2 ≤ n . Since
Σ τ = [ τ A , a τ I m ] [ τ A , a τ I m ] T , \Sigma_\tau=[\sqrt{\tau}A,\sqrt{a_\tau}I_m][\sqrt{\tau}A,\sqrt{a_\tau}I_m]^\mathsf{T}, Σ τ = [ τ A , a τ I m ] [ τ A , a τ I m ] T ,
the minimum-norm characterization of R ( y ) R(y) R ( y ) gives
R ( y ) = min τ A v + a τ w = y ( ∥ v ∥ 2 + ∥ w ∥ 2 ) ≤ ∥ x ⋆ ∥ 2 τ + σ 2 ∥ ζ ∥ 2 a τ , R(y)=\min_{\sqrt{\tau}Av+\sqrt{a_\tau}w=y}\bigl(\|v\|^2+\|w\|^2\bigr)\leq\frac{\|x^\star\|^2}{\tau}+\frac{\sigma^2\|\zeta\|^2}{a_\tau}, R ( y ) = τ A v + a τ w = y min ( ∥ v ∥ 2 + ∥ w ∥ 2 ) ≤ τ ∥ x ⋆ ∥ 2 + a τ σ 2 ∥ ζ ∥ 2 ,
where the inequality uses the feasible pair ( v , w ) = ( x ⋆ / τ , σ ζ / a τ ) (v,w)=(x^\star/\sqrt{\tau},\sigma\zeta/\sqrt{a_\tau}) ( v , w ) = ( x ⋆ / τ , σ ζ / a τ ) . By (22) and σ 2 ≤ a τ \sigma^2\leq a_\tau σ 2 ≤ a τ , it follows that n + R ( y ) ≤ 2 Q + ∥ ζ ∥ 2 n+R(y)\leq2Q+\|\zeta\|^2 n + R ( y ) ≤ 2 Q + ∥ ζ ∥ 2 . Since ζ ∼ N ( 0 , I m ) \zeta\sim\mathcal{N}(0,I_m) ζ ∼ N ( 0 , I m ) and m ≤ n ≤ Q m\leq n\leq Q m ≤ n ≤ Q ,
E ζ ( n + R ( y ) ) 2 ≤ 4 Q 2 + 4 Q m + m ( m + 2 ) ≤ C Q 2 . \mathbb{E}_\zeta(n+R(y))^2\leq4Q^2+4Qm+m(m+2)\leq CQ^2. E ζ ( n + R ( y ) ) 2 ≤ 4 Q 2 + 4 Q m + m ( m + 2 ) ≤ C Q 2 .
Finally, the geometric schedule (9) and T / τ = n / τ 2 ≤ Q 2 T/\tau=n/\tau^2\leq Q^2 T / τ = n / τ 2 ≤ Q 2 imply
log t j t j + 1 = log ( T / τ ) N ≤ 2 log Q N . \log\frac{t_j}{t_{j+1}}=\frac{\log(T/\tau)}{N}\leq\frac{2\log Q}{N}. log t j + 1 t j = N log ( T / τ ) ≤ N 2 log Q .
Applying (68) , averaging over ζ \zeta ζ , and using log 2 Q ≤ Q \log^2Q\leq Q log 2 Q ≤ Q , we obtain
E ζ H 2 ( Π t j , Π t j + 1 ) ≤ 2 log 2 Q N 2 E ζ ( n + R ( y ) ) 2 ≤ C Q 3 N 2 , \mathbb{E}_\zeta H^2(\Pi_{t_j},\Pi_{t_{j+1}})\leq\frac{2\log^2Q}{N^2}\mathbb{E}_\zeta(n+R(y))^2\leq\frac{CQ^3}{N^2}, E ζ H 2 ( Π t j , Π t j + 1 ) ≤ N 2 2 log 2 Q E ζ ( n + R ( y ) ) 2 ≤ N 2 C Q 3 ,
as required.
Terminal guarantees
This appendix proves the accuracy bound of Lemma 2 and the posterior-comparison bound of Lemma 3 for the terminal target.
Proof of Lemma 2
Throughout this subsection, we write ℓ : = log ( e n / α ) \ell:= \log(en/\alpha) ℓ := log ( e n / α ) , which satisfies ℓ ≥ 1 \ell\ge1 ℓ ≥ 1 for n ≥ 1 n \ge1 n ≥ 1 and 0 < α < 1 0 < \alpha< 1 0 < α < 1 .
Bounds at the terminal level. We record the scales used in the supporting arguments. Taking c ≤ 1 / 2 c \le1/2 c ≤ 1/2 in the hypothesis on τ \tau τ , we have 2 Q ≥ 2 n / τ ≥ 2 n / ( c α ) ≥ e n / α 2Q \ge2n/\tau\ge2n/(c\alpha) \ge en/\alpha 2 Q ≥ 2 n / τ ≥ 2 n / ( c α ) ≥ e n / α , and hence
log ( 2 Q ) ≥ ℓ , ε = c ε log ( 2 Q ) ≤ c ε ℓ . (70) \log(2Q) \ge\ell,\qquad\varepsilon= \frac{c_\varepsilon}{\log(2Q)} \le\frac{c_\varepsilon}{\ell}. \tag*{(70)} log ( 2 Q ) ≥ ℓ , ε = log ( 2 Q ) c ε ≤ ℓ c ε . (70)
Also τ ≤ σ 2 / ( 2 n ) ≤ σ 2 \tau\le\sigma^2/(2n) \le\sigma^2 τ ≤ σ 2 / ( 2 n ) ≤ σ 2 . With ∥ A ∥ 2 ≤ 2 n \lVert A\rVert^2 \le2n ∥ A ∥ 2 ≤ 2 n from A \mathcal{A} A ,
Σ τ = a τ I m + τ A A ⊤ ⪯ ( σ 2 + 2 τ + 2 n τ ) I m ⪯ 4 σ 2 I m . \Sigma_\tau= a_\tau I_m + \tau AA^\top\preceq(\sigma^2 + 2\tau+ 2n\tau)I_m \preceq4\sigma^2 I_m. Σ τ = a τ I m + τ A A ⊤ ⪯ ( σ 2 + 2 τ + 2 n τ ) I m ⪯ 4 σ 2 I m .
Consequently, G τ ⪰ A ⊤ A / ( 4 σ 2 ) G_\tau\succeq A^\top A/(4\sigma^2) G τ ⪰ A ⊤ A / ( 4 σ 2 ) , and ∥ A i ∥ 2 ≥ 1 / 2 \lVert A_i\rVert^2 \ge1/2 ∥ A i ∥ 2 ≥ 1/2 gives
τ ≤ σ 2 , B τ , i i ≥ G τ , i i ≥ ∥ A i ∥ 2 4 σ 2 ≥ 1 8 σ 2 ( i ∈ [ n ] ) . (71) \tau\le\sigma^2,\qquad B_{\tau,ii} \ge G_{\tau,ii} \ge\frac{\lVert A_i\rVert^2}{4\sigma^2} \ge\frac{1}{8\sigma^2}\qquad(i \in[n]). \tag*{(71)} τ ≤ σ 2 , B τ , ii ≥ G τ , ii ≥ 4 σ 2 ∥ A i ∥ 2 ≥ 8 σ 2 1 ( i ∈ [ n ]) . (71)
We show that the terminal target assigns small mass to the failure set B \mathcal{B} B of (37) . The argument splits the latent supports into three classes.
Definition 7 (Classes of supports). Let x ⋆ x^\star x ⋆ be a signal with support S S S and cap q q q , and set k e x : = ⌈ log ( 1 / α ) ⌉ k_{\mathrm{ex}} := \lceil\log(1/\alpha)\rceil k ex := ⌈ log ( 1/ α )⌉ . A support W ⊆ [ n ] W \subseteq[n] W ⊆ [ n ] is large if ∣ W ∣ > q |W| > q ∣ W ∣ > q , good if S ⊆ W S \subseteq W S ⊆ W and ∣ W ∖ S ∣ < k e x |W \setminus S| < k_{\mathrm{ex}} ∣ W ∖ S ∣ < k ex , and bad if it is neither large nor good.
A good support contains the support of the signal and few false coordinates. Moreover, k e x − 1 ≤ log ( 1 / α ) ≤ log ( 2 Q ) k_{\mathrm{ex}} - 1 \le\log(1/\alpha) \le\log(2Q) k ex − 1 ≤ log ( 1/ α ) ≤ log ( 2 Q ) , since Q ≥ n / τ Q \ge n/\tau Q ≥ n / τ and τ ≤ α \tau\le\alpha τ ≤ α . Thus (29) gives ∣ W ∣ ≤ ∣ S ∣ + k e x − 1 ≤ q |W| \le|S| + k_{\mathrm{ex}} - 1 \le q ∣ W ∣ ≤ ∣ S ∣ + k ex − 1 ≤ q for every good support, so the three classes are disjoint. A bad support either omits a coordinate of the signal or has many false coordinates. For each support W W W , write Γ W : = P τ W F τ \Gamma_W := P_\tau^W F_\tau Γ W := P τ W F τ for its Gaussian component in (51) . Since every support is large, bad or good and Γ W ( B ) ≤ 1 \Gamma_W(\mathcal{B}) \le1 Γ W ( B ) ≤ 1 ,
Π τ ( B ) = ∑ W P τ { S τ = W } Γ W ( B ) ≤ δ τ + P τ { S τ is bad } + ∑ W g o o d P τ { S τ = W } Γ W ( B ) . (72) \Pi_\tau(\mathcal{B}) = \sum_W \mathcal{P}_\tau\{S_\tau= W\}\Gamma_W(\mathcal{B}) \le\delta_\tau+ \mathcal{P}_\tau\{S_\tau\text{ is bad}\} + \sum_{W\ \mathrm{good}} \mathcal{P}_\tau\{S_\tau= W\}\Gamma_W(\mathcal{B}). \tag*{(72)} Π τ ( B ) = W ∑ P τ { S τ = W } Γ W ( B ) ≤ δ τ + P τ { S τ is bad } + W good ∑ P τ { S τ = W } Γ W ( B ) . (72)
The first term is controlled by Proposition 2. The next two terms are handled by the next two lemmas. Lemma 15 bounds the mass of the bad supports, and Lemma 16 shows that the components of the good supports are concentrated near the signal.
Lemma 15 (Bad supports). Under the conditions of Theorem 1 , for every A ∈ A A \in\mathcal{A} A ∈ A , it holds
E ζ P τ { S τ is bad } ≤ C α . \mathbb{E}_\zeta\mathcal{P}_\tau\{S_\tau\text{ is bad}\} \le C\alpha. E ζ P τ { S τ is bad } ≤ C α . The proof of this lemma can be found in Appendix § 9 .§ 1.1 .
Lemma 16 (Good supports). Suppose the conditions of Theorem 1 and A ∈ A A \in\mathcal{A} A ∈ A hold. For a support W W W , let Γ W : = P τ W F τ \Gamma_W := P_\tau^W F_\tau Γ W := P τ W F τ be the component of Π τ \Pi_\tau Π τ on W W W in (51) . There is a universal constant C r a d C_{\mathrm{rad}} C rad such that, with B ( x ⋆ ) \mathcal{B}(x^\star) B ( x ⋆ ) defined by (37) with this constant, there is an event of the noise of probability at least 1 − α 1-\alpha 1 − α on which every good support W W W satisfies Γ W ( B ( x ⋆ ) ) ≤ α \Gamma_W(\mathcal{B}(x^\star)) \le\alpha Γ W ( B ( x ⋆ )) ≤ α .
The proof of this lemma can be found in Appendix § 9 .§ 1.2 .
Let E \mathcal{E} E be the noise event of Lemma 16 , with P ζ ( E c ) ≤ α \mathbb{P}_\zeta(\mathcal{E}^{c}) \le\alpha P ζ ( E c ) ≤ α . Averaging (72) over the noise gives
E ζ Π τ ( B ) ≤ E ζ δ τ + E ζ P τ { S τ is bad } + α P ζ ( E ) + P ζ ( E c ) ≤ E ζ δ τ + E ζ P τ { S τ is bad } + 2 α ≤ C α . (73) \begin{aligned}
\mathbb{E}_\zeta\Pi_\tau(\mathcal{B}) &\le\mathbb{E}_\zeta\delta_\tau+ \mathbb{E}_\zeta\mathcal{P}_\tau\{S_\tau\text{ is bad}\} + \alpha\mathbb{P}_\zeta(\mathcal{E}) + \mathbb{P}_\zeta(\mathcal{E}^{c}) \\
&\le\mathbb{E}_\zeta\delta_\tau+ \mathbb{E}_\zeta\mathcal{P}_\tau\{S_\tau\text{ is bad}\} + 2\alpha\le C\alpha. \tag*{(73)}
\end{aligned} E ζ Π τ ( B ) ≤ E ζ δ τ + E ζ P τ { S τ is bad } + α P ζ ( E ) + P ζ ( E c ) ≤ E ζ δ τ + E ζ P τ { S τ is bad } + 2 α ≤ C α . (73)
Here, the last bound uses Proposition 2 with ( 2 Q ) − 14 ≤ 1 / Q ≤ τ ≤ α (2Q)^{-14} \le1/Q \le\tau\le\alpha ( 2 Q ) − 14 ≤ 1/ Q ≤ τ ≤ α , by (22) , and Lemma 15 .
Proof of Lemma 15
We bound the bad-support mass by fractional moments of the support-weight ratios. A product estimate for these moments reduces the sum over supports to two scalar bounds: one for false additions and one for omitted true coordinates.
Step 1: reduction to fractional moments. For any 0 < ι ≤ 1 0 < \iota\le1 0 < ι ≤ 1 , the latent mass of W W W is at most min { 1 , w W / w S } ≤ ( w W / w S ) ι \min\{1,w_W/w_S\} \le(w_W/w_S)^\iota min { 1 , w W / w S } ≤ ( w W / w S ) ι . Hence
P τ { S τ is bad } ≤ ∑ W b a d min { 1 , w W w S } ≤ ∑ W b a d ( w W w S ) ι . \mathbb{P}_{\tau}\{S_{\tau}\text{ is bad}\} \le\sum_{W\ \mathrm{bad}}\min\left\{1,\frac{w_W}{w_S}\right\} \le\sum_{W\ \mathrm{bad}}\left(\frac{w_W}{w_S}\right)^\iota. P τ { S τ is bad } ≤ W bad ∑ min { 1 , w S w W } ≤ W bad ∑ ( w S w W ) ι .
The support S S S is good, so every bad support differs from S S S and has size at most q q q . The first moments are too large for this purpose: for a support with a single false coordinate, the first moment can be of the order of the prior odds ω \omega ω , and summing over the coordinates gives order s s s . The next lemma uses ι \iota ι close to one to keep n 1 − ι n^{1-\iota} n 1 − ι bounded, while retaining a positive margin for Gaussian integration.
Lemma 17 (A fractional moment of one weight ratio). There are universal constants c ∈ ( 0 , 1 / 2 ] c \in(0,1/2] c ∈ ( 0 , 1/2 ] and C , C a > 0 C,C_a > 0 C , C a > 0 such that the following holds. Suppose that τ ≤ c min { σ 2 / n , α } \tau\le c\min\{\sigma^2/n,\alpha\} τ ≤ c min { σ 2 / n , α } , and let x ⋆ x^\star x ⋆ have support S S S , cap q q q , matrix event A A A , and σ ∥ x ⋆ ∥ ≤ c ℓ \sigma\lVert x^\star\rVert\le c\sqrt{\ell} σ ∥ x ⋆ ∥ ≤ c ℓ . Let A ∈ A A \in\mathcal{A} A ∈ A , y = A x ⋆ + σ ζ y=Ax^\star+\sigma\zeta y = A x ⋆ + σ ζ , and let w W w_W w W be the weights (52) at level τ \tau τ . Write B = B τ B=B_\tau B = B τ and set
ι : = 1 − 1 2 ℓ , ι 1 : = ι ( 1 + ε ) 1 − ε , \iota:= 1-\frac{1}{2\ell}, \qquad\iota_1 := \frac{\iota(1+\varepsilon)}{1-\varepsilon}, ι := 1 − 2 ℓ 1 , ι 1 := 1 − ε ι ( 1 + ε ) ,
so that 1 / 2 ≤ ι < 1 1/2 \le\iota< 1 1/2 ≤ ι < 1 and 1 − ι 1 ≥ 1 / ( 4 ℓ ) 1-\iota_1 \ge1/(4\ell) 1 − ι 1 ≥ 1/ ( 4 ℓ ) . Define
χ i : = ω ι ( ( 1 − ε ) B i i ) − ι / 2 1 − ι 1 ( i ∉ S ) , \chi_i := \frac{\omega^\iota\bigl((1-\varepsilon)B_{ii}\bigr)^{-\iota/2}}{\sqrt{1-\iota_1}}
\qquad(i\notin S), χ i := 1 − ι 1 ω ι ( ( 1 − ε ) B ii ) − ι /2 ( i ∈ / S ) ,
ψ i ′ : = ( e n α ) C a ω − ι B i i / 2 1 − ι 1 e − ι ( 1 − ε ) B i i ( x i ⋆ ) 2 / 8 ( i ∈ S ) . \psi_i' := \left(\frac{e n}{\alpha}\right)^{C_a}\frac{\omega^{-\iota B_{ii}/2}}{\sqrt{1-\iota_1}}e^{-\iota(1-\varepsilon)B_{ii}(x_i^\star)^2/8}
\qquad(i\in S). ψ i ′ := ( α e n ) C a 1 − ι 1 ω − ι B ii /2 e − ι ( 1 − ε ) B ii ( x i ⋆ ) 2 /8 ( i ∈ S ) .
Then, for every support W ≠ S W\ne S W = S with ∣ W ∣ ≤ q |W|\le q ∣ W ∣ ≤ q ,
E ζ ( w W w S ) ι ≤ C ∏ i ∈ W ∖ S χ i ∏ i ∈ S ∖ W ψ i ′ . (74) \mathbb{E}_{\zeta}\left(\frac{w_W}{w_S}\right)^\iota
\le C\prod_{i\in W\setminus S}\chi_i\prod_{i\in S\setminus W}\psi_i'. \tag*{(74)} E ζ ( w S w W ) ι ≤ C i ∈ W ∖ S ∏ χ i i ∈ S ∖ W ∏ ψ i ′ . (74)
Here χ i \chi_i χ i is the factor associated with adding a false coordinate, while ψ i ′ \psi_i' ψ i ′ is the factor associated with omitting a true coordinate. The proof is given in Appendix § 9 .§ 1.3 .
Step 2: sum over the bad supports. For each support W W W , write F = W ∖ S F=W\setminus S F = W ∖ S and S 0 = S ∖ W S_0=S\setminus W S 0 = S ∖ W . Applying Lemma 17 to the preceding reduction gives
E ζ P τ { S τ is bad } ≤ C ∑ W b a d ∏ i ∈ F χ i ∏ i ∈ S 0 ψ i ′ . \mathbb{E}_{\zeta}\mathbb{P}_{\tau}\{S_{\tau}\text{ is bad}\}
\le C\sum_{W\ \mathrm{bad}}\prod_{i\in F}\chi_i\prod_{i\in S_0}\psi_i'. E ζ P τ { S τ is bad } ≤ C W bad ∑ i ∈ F ∏ χ i i ∈ S 0 ∏ ψ i ′ .
The map W ↦ ( F , S 0 ) W\mapsto(F,S_0) W ↦ ( F , S 0 ) is a bijection onto the pairs of a subset of [ n ] ∖ S [n]\setminus S [ n ] ∖ S and a subset of S S S , and a bad support has S 0 ≠ ∅ S_0\ne\varnothing S 0 = ∅ or ∣ F ∣ ≥ k e x |F|\ge k_{\mathrm{ex}} ∣ F ∣ ≥ k ex . Enlarging the sum to all such pairs, with χ t o t : = ∑ i ∉ S χ i \chi_{\mathrm{tot}}:=\sum_{i\notin S}\chi_i χ tot := ∑ i ∈ / S χ i and ψ t o t : = ∑ i ∈ S ψ i ′ \psi_{\mathrm{tot}}:=\sum_{i\in S}\psi_i' ψ tot := ∑ i ∈ S ψ i ′ ,
E ζ P τ { S τ is bad } ≤ C [ ∏ i ∉ S ( 1 + χ i ) ( ∏ i ∈ S ( 1 + ψ i ′ ) − 1 ) + ∑ ∣ F ∣ ≥ k e x ∏ i ∈ F χ i ] ≤ C e χ t o t ( e ψ t o t − 1 + χ t o t k e x ) . (75) \begin{aligned}
\mathbb{E}_{\zeta}\mathbb{P}_{\tau}\{S_{\tau}\text{ is bad}\}
\le C\left[\prod_{i\notin S}(1+\chi_i)\left(\prod_{i\in S}(1+\psi_i')-1\right)+\sum_{|F|\ge k_{\mathrm{ex}}}\prod_{i\in F}\chi_i\right] \\
&\le C e^{\chi_{\mathrm{tot}}}\left(e^{\psi_{\mathrm{tot}}}-1+\chi_{\mathrm{tot}}^{k_{\mathrm{ex}}}\right). \tag*{(75)}
\end{aligned} E ζ P τ { S τ is bad } ≤ C i ∈ / S ∏ ( 1 + χ i ) ( i ∈ S ∏ ( 1 + ψ i ′ ) − 1 ) + ∣ F ∣ ≥ k ex ∑ i ∈ F ∏ χ i ≤ C e χ tot ( e ψ tot − 1 + χ tot k ex ) . (75)
where the last sum is over the subsets F F F of [ n ] ∖ S [n]\setminus S [ n ] ∖ S . The second bound uses 1 + u ≤ e u 1+u\le e^u 1 + u ≤ e u , and that the products over the subsets of size j j j add up to at most χ t o t j / j ! \chi_{\mathrm{tot}}^j/j! χ tot j / j ! , with ∑ j ≥ k e x χ t o t j / j ! ≤ e χ t o t χ t o t k e x \sum_{j\ge k_{\mathrm{ex}}}\chi_{\mathrm{tot}}^j/j!\le e^{\chi_{\mathrm{tot}}}\chi_{\mathrm{tot}}^{k_{\mathrm{ex}}} ∑ j ≥ k ex χ tot j / j ! ≤ e χ tot χ tot k ex .
Step 3: bound false additions and omissions. It remains to establish the two scalar bounds
χ t o t ≤ e − 1 , ψ t o t ≤ 2 α . \chi_{\mathrm{tot}} \le e^{-1}, \qquad\psi_{\mathrm{tot}} \le2\alpha. χ tot ≤ e − 1 , ψ tot ≤ 2 α .
The prior-sparsity condition will give the first bound, which controls excess false coordinates. The minimum-signal condition will give the second, which controls the omission of any true coordinate. The choice k e x = ⌈ log ( 1 / α ) ⌉ k_{\mathrm{ex}} = \lceil\log(1/\alpha)\rceil k ex = ⌈ log ( 1/ α )⌉ then ensures χ t o t ≤ α \chi_{\mathrm{tot}} \le\alpha χ tot ≤ α .
Using B i i ≥ c σ − 2 B_{ii} \ge c\sigma^{-2} B ii ≥ c σ − 2 , ε ≤ 1 / 2 \varepsilon\le1/2 ε ≤ 1/2 and n ω ≤ 2 s n\omega\le2s nω ≤ 2 s , we obtain
χ t o t ≤ n ω ι ( C σ ) ℓ ≤ 2 ℓ n 1 − ι ( C s σ ) ℓ . \chi_{\mathrm{tot}} \le n\omega^{\iota}(C\sigma)^{\ell} \le2\sqrt{\ell}\,n^{1-\iota}(Cs\sigma)^{\ell}. χ tot ≤ n ω ι ( C σ ) ℓ ≤ 2 ℓ n 1 − ι ( C s σ ) ℓ .
Here n 1 − ι = n 1 / ( 2 ℓ ) ≤ e 1 / 2 n^{1-\iota} = n^{1/(2\ell)} \le e^{1/2} n 1 − ι = n 1/ ( 2 ℓ ) ≤ e 1/2 , as log n ≤ ℓ \log n \le\ell log n ≤ ℓ . The hypothesis s ≤ c / ( σ ℓ ) s \le c/(\sigma\sqrt{\ell}) s ≤ c / ( σ ℓ ) gives C s σ ≤ C c / ℓ Cs\sigma\le Cc/\sqrt{\ell} C s σ ≤ C c / ℓ , which is at most 1 / ℓ 1/\sqrt{\ell} 1/ ℓ for c c c small enough, and then ( C s σ ) ℓ ℓ ≤ ( C c ) 1 / 2 ℓ ( 1 − ι ) / 2 ≤ 2 ( C c ) 1 / 2 (Cs\sigma)^{\ell}\sqrt{\ell} \le(Cc)^{1/2}\ell^{(1-\iota)/2} \le2(Cc)^{1/2} ( C s σ ) ℓ ℓ ≤ ( C c ) 1/2 ℓ ( 1 − ι ) /2 ≤ 2 ( C c ) 1/2 , since ι ≥ 1 / 2 \iota\ge1/2 ι ≥ 1/2 and ℓ 1 / ( 4 ℓ ) ≤ 2 \ell^{1/(4\ell)} \le2 ℓ 1/ ( 4 ℓ ) ≤ 2 . Hence χ t o t ≤ C c 1 / 2 ≤ 1 / e \chi_{\mathrm{tot}} \le Cc^{1/2} \le1/e χ tot ≤ C c 1/2 ≤ 1/ e for c c c small enough.
To bound ψ t o t \psi_{\mathrm{tot}} ψ tot , fix i ∈ S i \in S i ∈ S . Since ω ≤ 1 ≤ B i i \omega\le1 \le B_{ii} ω ≤ 1 ≤ B ii , and α < 1 ≤ s \alpha< 1 \le s α < 1 ≤ s ,
ω − ι ≤ ω − 1 ≤ n s ≤ e n α , B i i ℓ / 2 ≤ B i i 1 / 2 . \omega^{-\iota} \le\omega^{-1} \le\frac{n}{s} \le\frac{en}{\alpha}, \qquad B_{ii}^{\ell/2} \le B_{ii}^{1/2}. ω − ι ≤ ω − 1 ≤ s n ≤ α e n , B ii ℓ /2 ≤ B ii 1/2 .
Set u : = α τ / ( e n ) u := \alpha\tau/(en) u := α τ / ( e n ) , so that u ≤ 1 / e u \le1/e u ≤ 1/ e . The bound min i ∈ S ∣ x i ⋆ ∣ ≥ C d e t σ log ( e n / ( α τ ) ) \min_{i\in S}|x_i^\star| \ge C_{\mathrm{det}}\sigma\sqrt{\log(en/(\alpha\tau))} min i ∈ S ∣ x i ⋆ ∣ ≥ C det σ log ( e n / ( α τ )) in (15) , where C d e t C_{\mathrm{det}} C det is the constant C C C there, gives ( x i ⋆ ) 2 ≥ C d e t 2 σ 2 log ( 1 / u ) (x_i^\star)^2 \ge C_{\mathrm{det}}^2\sigma^2\log(1/u) ( x i ⋆ ) 2 ≥ C det 2 σ 2 log ( 1/ u ) for i ∈ S i \in S i ∈ S , and ι ( 1 − ε ) / 8 ≥ 1 / 32 \iota(1-\varepsilon)/8 \ge1/32 ι ( 1 − ε ) /8 ≥ 1/32 , so
ψ i ′ ≤ ( e n α ) C a + 1 2 ℓ f ( B i i ) , f ( θ ) : = θ e − θ C d e t 2 σ 2 log ( 1 / u ) / 32 . \psi_i' \le\left(\frac{en}{\alpha}\right)^{C_a+1}2\sqrt{\ell}\,f(B_{ii}), \qquad f(\theta) := \sqrt{\theta}\,e^{-\theta C_{\mathrm{det}}^2\sigma^2\log(1/u)/32}. ψ i ′ ≤ ( α e n ) C a + 1 2 ℓ f ( B ii ) , f ( θ ) := θ e − θ C det 2 σ 2 l o g ( 1/ u ) /32 .
Let c B ≤ 1 c_B \le1 c B ≤ 1 be the constant in B i i ≥ c B σ − 2 B_{ii} \ge c_B\sigma^{-2} B ii ≥ c B σ − 2 . Since log ( 1 / u ) ≥ 1 \log(1/u) \ge1 log ( 1/ u ) ≥ 1 , the function f f f is decreasing on [ c B σ − 2 , ∞ ) [c_B\sigma^{-2},\infty) [ c B σ − 2 , ∞ ) once C d e t C_{\mathrm{det}} C det is large enough in terms of c B c_B c B , and
f ( B i i ) ≤ f ( c B σ 2 ) ≤ u c B C d e t 2 / 32 σ . f(B_{ii}) \le f\left(\frac{c_B}{\sigma^2}\right) \le\frac{u^{c_BC_{\mathrm{det}}^2/32}}{\sigma}. f ( B ii ) ≤ f ( σ 2 c B ) ≤ σ u c B C det 2 /32 .
By the hypothesis on τ \tau τ , σ 2 ≥ τ \sigma^2 \ge\tau σ 2 ≥ τ , and τ = u e n / α ≥ e u \tau= uen/\alpha\ge eu τ = u e n / α ≥ e u , so σ − 1 ≤ u − 1 / 2 \sigma^{-1} \le u^{-1/2} σ − 1 ≤ u − 1/2 . Also e n / α = τ / u ≤ 1 / u en/\alpha= \tau/u \le1/u e n / α = τ / u ≤ 1/ u and ℓ ≤ log ( 1 / u ) ≤ u − 1 / 2 \sqrt{\ell} \le\sqrt{\log(1/u)} \le u^{-1/2} ℓ ≤ log ( 1/ u ) ≤ u − 1/2 . These bounds and ∣ S ∣ ≤ n ≤ α / ( e u ) |S| \le n \le\alpha/(eu) ∣ S ∣ ≤ n ≤ α / ( e u ) give
ψ t o t = ∑ i ∈ S ψ i ′ ≤ 2 α u c B C d e t 2 / 32 − C a − 3 ≤ 2 α \psi_{\mathrm{tot}} = \sum_{i\in S}\psi_i' \le2\alpha\,u^{c_BC_{\mathrm{det}}^2/32-C_a-3} \le2\alpha ψ tot = i ∈ S ∑ ψ i ′ ≤ 2 α u c B C det 2 /32 − C a − 3 ≤ 2 α
once C d e t C_{\mathrm{det}} C det is large enough in terms of c B c_B c B and C a C_a C a , since u ≤ 1 u \le1 u ≤ 1 .
Finally, since χ t o t ≤ 1 / e \chi_{\mathrm{tot}} \le1/e χ tot ≤ 1/ e and k e x ≥ log ( 1 / α ) k_{\mathrm{ex}} \ge\log(1/\alpha) k ex ≥ log ( 1/ α ) , we have χ t o t k e x ≤ e − k e x ≤ α \chi_{\mathrm{tot}}^{k_{\mathrm{ex}}} \le e^{-k_{\mathrm{ex}}} \le\alpha χ tot k ex ≤ e − k ex ≤ α . Combining this with (75) , e u − 1 ≤ u e u e^u-1 \le ue^u e u − 1 ≤ u e u and ψ t o t ≤ 2 α ≤ 2 \psi_{\mathrm{tot}} \le2\alpha\le2 ψ tot ≤ 2 α ≤ 2 gives
E ζ P τ { S τ is bad } ≤ C e 1 / e ( 2 α e 2 + α ) ≤ C α . \mathbb{E}_{\zeta}\mathbb{P}_{\tau}\{S_{\tau}\text{ is bad}\} \le Ce^{1/e}(2\alpha e^2+\alpha) \le C\alpha. E ζ P τ { S τ is bad } ≤ C e 1/ e ( 2 α e 2 + α ) ≤ C α .
Proof of Lemma 16
The observation noise enters Γ W \Gamma_W Γ W only through its mean. Because the support weights depend on the same noise, we first control the means on one noise event that holds for all good supports. For each noise realization in this event, we bound the failure mass within each Gaussian component.
Step 1: identify the component law and covariance. Write G = G τ G=G_{\tau} G = G τ , M = M τ M=M_{\tau} M = M τ , B = B τ B=B_{\tau} B = B τ and b = b τ b=b_{\tau} b = b τ . For any support W W W , (53) gives z W ∼ N ( B W W − 1 b W , B W W − 1 ) z_W \sim\mathcal{N}(B_{WW}^{-1}b_W,B_{WW}^{-1}) z W ∼ N ( B W W − 1 b W , B W W − 1 ) under P W \mathcal{P}^W P W , with z W c = 0 z_{W^c}=0 z W c = 0 . By (49) , a draw from Γ W \Gamma_W Γ W is M [ n ] W z W + τ b M_{[n]W}z_W+\tau b M [ n ] W z W + τ b plus an independent N ( 0 , τ M ) \mathcal{N}(0,\tau M) N ( 0 , τ M ) vector. Thus Γ W = N ( L W y , C W ) \Gamma_W=\mathcal{N}(L_Wy,C_W) Γ W = N ( L W y , C W ) , where
L W : = M [ n ] W B W W − 1 A W T Σ τ − 1 + τ A T Σ τ − 1 , C W : = M [ n ] W B W W − 1 M W [ n ] + τ M . (76) L_W := M_{[n]W}B_{WW}^{-1}A_W^{\mathsf T}\Sigma_{\tau}^{-1}+\tau A^{\mathsf T}\Sigma_{\tau}^{-1}, \qquad C_W := M_{[n]W}B_{WW}^{-1}M_{W[n]}+\tau M. \tag*{(76)} L W := M [ n ] W B W W − 1 A W T Σ τ − 1 + τ A T Σ τ − 1 , C W := M [ n ] W B W W − 1 M W [ n ] + τ M . (76)
For W = ∅ W=\varnothing W = ∅ , this gives L W y = τ b L_Wy=\tau b L W y = τ b and C W = τ M C_W=\tau M C W = τ M .
Let W W W be good, so ∣ W ∣ ≤ q |W| \le q ∣ W ∣ ≤ q as noted after Definition 7. For W ≠ ∅ W \ne\varnothing W = ∅ , the block condition (34) applies since ∣ W ∣ ≤ q |W| \le q ∣ W ∣ ≤ q , and gives
λ min ( B W W ) ≥ min i ∈ W B i i − ∥ B W W − ( diag B ) W W ∥ ≥ ( 1 − ε ) min i ∈ W B i i ≥ ( 1 − ε ) min i B i i . \lambda_{\min}(B_{WW}) \ge\min_{i \in W} B_{ii} - \lVert B_{WW} - (\operatorname{diag} B)_{WW} \rVert\ge(1-\varepsilon)\min_{i \in W} B_{ii} \ge(1-\varepsilon)\min_i B_{ii}. λ m i n ( B W W ) ≥ i ∈ W min B ii − ∥ B W W − ( diag B ) W W ∥ ≥ ( 1 − ε ) i ∈ W min B ii ≥ ( 1 − ε ) i min B ii .
Thus ∥ B W W − 1 ∥ ≤ ( ( 1 − ε ) min i B i i ) − 1 \lVert B_{WW}^{-1} \rVert\le((1-\varepsilon)\min_i B_{ii})^{-1} ∥ B W W − 1 ∥ ≤ (( 1 − ε ) min i B ii ) − 1 . Also M = ( I n + ( τ / a τ ) A ⊤ A ) − 1 M=(I_n+(\tau/a_\tau)A^\top A)^{-1} M = ( I n + ( τ / a τ ) A ⊤ A ) − 1 satisfies 0 ≺ M ⪯ I n 0 \prec M \preceq I_n 0 ≺ M ⪯ I n , hence ∥ M ∥ ≤ 1 \lVert M\rVert\le1 ∥ M ∥ ≤ 1 . With ε ≤ 1 / 2 \varepsilon\le1/2 ε ≤ 1/2 , (71) and τ ≤ σ 2 / n \tau\le\sigma^2/n τ ≤ σ 2 / n , we obtain for this matrix and the covariance (76)
∥ B W W − 1 ∥ ≤ C σ 2 , tr C W ≤ tr B W W − 1 + τ n ≤ C σ 2 ( ∣ W ∣ + 1 ) , ∥ C W ∥ ≤ C σ 2 . (77) \lVert B_{WW}^{-1}\rVert\le C\sigma^2,\qquad\operatorname{tr} C_W \le\operatorname{tr} B_{WW}^{-1}+\tau n \le C\sigma^2(|W|+1),\qquad\lVert C_W\rVert\le C\sigma^2. \tag*{(77)} ∥ B W W − 1 ∥ ≤ C σ 2 , tr C W ≤ tr B W W − 1 + τ n ≤ C σ 2 ( ∣ W ∣ + 1 ) , ∥ C W ∥ ≤ C σ 2 . (77)
These bounds also hold for W = ∅ W=\varnothing W = ∅ , when C W = τ M C_W=\tau M C W = τ M .
Step 2: control all component means. Substituting y = A x ⋆ + σ ζ y=Ax^\star+\sigma\zeta y = A x ⋆ + σ ζ into b = A ⊤ Σ τ − 1 y b=A^\top\Sigma_\tau^{-1}y b = A ⊤ Σ τ − 1 y gives b = G x ⋆ + η b=Gx^\star+\eta b = G x ⋆ + η , where η : = σ A ⊤ Σ τ − 1 ζ \eta:=\sigma A^\top\Sigma_\tau^{-1}\zeta η := σ A ⊤ Σ τ − 1 ζ is centered Gaussian. Since S ⊆ W S\subseteq W S ⊆ W , ( G x ⋆ ) W = G W W x W ⋆ = B W W x W ⋆ − x W ⋆ (Gx^\star)_W=G_{WW}x_W^\star=B_{WW}x_W^\star-x_W^\star ( G x ⋆ ) W = G W W x W ⋆ = B W W x W ⋆ − x W ⋆ , so B W W − 1 ( G x ⋆ ) W = x W ⋆ − B W W − 1 x W ⋆ B_{WW}^{-1}(Gx^\star)_W=x_W^\star-B_{WW}^{-1}x_W^\star B W W − 1 ( G x ⋆ ) W = x W ⋆ − B W W − 1 x W ⋆ . With M [ n ] W x W ⋆ = M x ⋆ M_{[n]W}x_W^\star=Mx^\star M [ n ] W x W ⋆ = M x ⋆ , the identity M + τ G = I M+\tau G=I M + τ G = I gives
L W y − x ⋆ = − M [ n ] W B W W − 1 x W ⋆ + V W + τ η , V W : = M [ n ] W B W W − 1 η W . L_Wy-x^\star=-M_{[n]W}B_{WW}^{-1}x_W^\star+V_W+\tau\eta,\qquad V_W:=M_{[n]W}B_{WW}^{-1}\eta_W. L W y − x ⋆ = − M [ n ] W B W W − 1 x W ⋆ + V W + τ η , V W := M [ n ] W B W W − 1 η W .
The three terms are the shrinkage bias, the observation noise fitted on W W W , and the common observation-noise contribution in the forward-kernel mean, respectively. By ∥ M ∥ ≤ 1 \lVert M\rVert\le1 ∥ M ∥ ≤ 1 , the bound on ∥ B W W − 1 ∥ \lVert B_{WW}^{-1}\rVert ∥ B W W − 1 ∥ in (77) and the bound σ 2 ∥ x ⋆ ∥ 2 ≤ ℓ \sigma^2\lVert x^\star\rVert^2\le\ell σ 2 ∥ x ⋆ ∥ 2 ≤ ℓ from (15) , the first term has norm at most C σ 2 ∥ x ⋆ ∥ ≤ C σ ℓ C\sigma^2\lVert x^\star\rVert\le C\sigma\sqrt{\ell} C σ 2 ∥ x ⋆ ∥ ≤ C σ ℓ . Since Σ τ ⪰ a τ I m ⪰ σ I m \Sigma_\tau\succeq a_\tau I_m\succeq\sigma I_m Σ τ ⪰ a τ I m ⪰ σ I m , we have Cov η = σ 2 A ⊤ Σ τ − 2 A ⪯ G \operatorname{Cov}\eta=\sigma^2A^\top\Sigma_\tau^{-2}A\preceq G Cov η = σ 2 A ⊤ Σ τ − 2 A ⪯ G . Together with G W W ⪯ B W W G_{WW}\preceq B_{WW} G W W ⪯ B W W , this gives Cov V W ⪯ M [ n ] W B W W − 1 M W [ n ] \operatorname{Cov}V_W\preceq M_{[n]W}B_{WW}^{-1}M_{W[n]} Cov V W ⪯ M [ n ] W B W W − 1 M W [ n ] , whose trace and norm are at most C σ 2 ∣ W ∣ C\sigma^2|W| C σ 2 ∣ W ∣ and C σ 2 C\sigma^2 C σ 2 . Also G = τ − 1 ( I n − M ) ⪯ τ − 1 I n G=\tau^{-1}(I_n-M)\preceq\tau^{-1}I_n G = τ − 1 ( I n − M ) ⪯ τ − 1 I n , so Cov ( τ η ) ⪯ τ 2 G ⪯ τ I n \operatorname{Cov}(\tau\eta)\preceq\tau^2G\preceq\tau I_n Cov ( τ η ) ⪯ τ 2 G ⪯ τ I n , with trace at most τ n ≤ σ 2 \tau n\le\sigma^2 τ n ≤ σ 2 and norm at most τ ≤ σ 2 \tau\le\sigma^2 τ ≤ σ 2 .
To control the observation noise uniformly over good supports, we use the Gaussian concentration inequality (Boucheron et al., [7 ] , Theorem 5.6) for the ∥ Σ 1 / 2 ∥ \lVert\Sigma^{1/2}\rVert ∥ Σ 1/2 ∥ -Lipschitz function g ↦ ∥ Σ 1 / 2 g ∥ g\mapsto\lVert\Sigma^{1/2}g\rVert g ↦ ∥ Σ 1/2 g ∥ of a standard Gaussian vector g g g . With E ∥ Σ 1 / 2 g ∥ ≤ ( tr Σ ) 1 / 2 \mathbb{E}\lVert\Sigma^{1/2}g\rVert\le(\operatorname{tr}\Sigma)^{1/2} E ∥ Σ 1/2 g ∥ ≤ ( tr Σ ) 1/2 it gives, for a centered Gaussian vector V V V with covariance Σ \Sigma Σ and 0 < δ < 1 0<\delta<1 0 < δ < 1 ,
P ( ∥ V ∥ > tr Σ + 2 ∥ Σ ∥ log ( 1 / δ ) ) ≤ δ . (78) \mathbb{P}\left(\lVert V\rVert>\sqrt{\operatorname{tr}\Sigma}+\sqrt{2\lVert\Sigma\rVert\log(1/\delta)}\right)\le\delta. \tag*{(78)} P ( ∥ V ∥ > tr Σ + 2 ∥ Σ ∥ log ( 1/ δ ) ) ≤ δ . (78)
The good supports number ∑ j < k e x ( n − ∣ S ∣ j ) ≤ 2 n k e x − 1 \sum_{j<k_{\mathrm{ex}}}\binom{n-|S|}{j}\le2n^{k_{\mathrm{ex}}-1} ∑ j < k ex ( j n − ∣ S ∣ ) ≤ 2 n k ex − 1 , as n ≥ 2 n\ge2 n ≥ 2 . Apply (78) with failure probability α / ( 4 n k e x − 1 ) \alpha/(4n^{k_{\mathrm{ex}}-1}) α / ( 4 n k ex − 1 ) to each V W V_W V W , and with failure probability α / 2 \alpha/2 α /2 to τ η \tau\eta τ η . A union bound gives an event E 2 \mathcal{E}_2 E 2 of the noise with P ( E 2 c ) ≤ α \mathbb{P}(\mathcal{E}_2^c)\le\alpha P ( E 2 c ) ≤ α on which every good W W W satisfies
∥ V W ∥ ≤ C σ ( ∣ W ∣ + log ( 4 n k e x − 1 / α ) ) , ∥ τ η ∥ ≤ C σ ( 1 + log ( 2 / α ) ) . \lVert V_W\rVert\le C\sigma\left(\sqrt{|W|}+\sqrt{\log(4n^{k_{\mathrm{ex}}-1}/\alpha)}\right),\qquad\lVert\tau\eta\rVert\le C\sigma\left(1+\sqrt{\log(2/\alpha)}\right). ∥ V W ∥ ≤ C σ ( ∣ W ∣ + log ( 4 n k ex − 1 / α ) ) , ∥ τ η ∥ ≤ C σ ( 1 + log ( 2/ α ) ) .
The cost of uniformity over good supports is
log 4 n k e x − 1 α = log 4 + ( k e x − 1 ) log n + log ( 1 / α ) ≤ C ℓ 2 , \log\frac{4n^{k_{\mathrm{ex}}-1}}{\alpha}=\log4+(k_{\mathrm{ex}}-1)\log n+\log(1/\alpha)\le C\ell^2, log α 4 n k ex − 1 = log 4 + ( k ex − 1 ) log n + log ( 1/ α ) ≤ C ℓ 2 ,
since k e x − 1 ≤ ℓ k_{\mathrm{ex}}-1\le\ell k ex − 1 ≤ ℓ , log n ≤ ℓ \log n\le\ell log n ≤ ℓ and ℓ ≥ 1 \ell\ge1 ℓ ≥ 1 . Together with ∣ W ∣ ≤ ∣ S ∣ + ℓ \sqrt{|W|}\le\sqrt{|S|}+\sqrt{\ell} ∣ W ∣ ≤ ∣ S ∣ + ℓ , this bounds all the component means within C σ ( ∣ S ∣ + ℓ ) C\sigma(\sqrt{|S|}+\ell) C σ ( ∣ S ∣ + ℓ ) of x ⋆ x^\star x ⋆ on E 2 \mathcal{E}_2 E 2 .
Step 3: bound fluctuations within each component. On E 2 \mathcal{E}_2 E 2 , a draw x ∼ Γ W x\sim\Gamma_W x ∼ Γ W satisfies ∥ x − x ⋆ ∥ ≤ ∥ L W y − x ⋆ ∥ + ∥ x − L W y ∥ \lVert x-x^\star\rVert\le\lVert L_Wy-x^\star\rVert+\lVert x-L_Wy\rVert ∥ x − x ⋆ ∥ ≤ ∥ L W y − x ⋆ ∥ + ∥ x − L W y ∥ . By (78) and the bounds on tr C W \operatorname{tr}C_W tr C W and ∥ C W ∥ \lVert C_W\rVert ∥ C W ∥ in (77) , the second term is at most C σ ( ∣ W ∣ + 1 + log ( 1 / α ) ) C\sigma(\sqrt{|W|+1}+\sqrt{\log(1/\alpha)}) C σ ( ∣ W ∣ + 1 + log ( 1/ α ) ) except on a set of Γ W \Gamma_W Γ W -mass at most α \alpha α . This fluctuation has scale at most C σ ( ∣ S ∣ + ℓ ) C\sigma(\sqrt{|S|}+\sqrt{\ell}) C σ ( ∣ S ∣ + ℓ ) . Combining it with the uniform mean bound proves Γ W ( B ) ≤ α \Gamma_W(B)\le\alpha Γ W ( B ) ≤ α on E 2 \mathcal{E}_2 E 2 for a universal constant C r a d C_{\mathrm{rad}} C rad . Thus the σ ℓ \sigma\ell σ ℓ term in the recovery radius comes from uniform control of the means across good supports. Since P ( E 2 ) ≤ α \mathbb{P}(\mathcal{E}_2)\le\alpha P ( E 2 ) ≤ α , the good components contribute at most 2 α 2\alpha 2 α after averaging over the observation noise, as used in (73) .
Proof of Lemma 17
We derive the product estimate (74) by writing the exact weight ratio through Schur complements, integrating the Gaussian noise, and bounding the deterministic shifts. First, the terminal bounds (70) give ε ≤ c ε / ℓ \varepsilon\le c_{\varepsilon}/\ell ε ≤ c ε / ℓ , and c ε ≤ 1 / 16 c_{\varepsilon} \le1/16 c ε ≤ 1/16 . Since ε ≤ 1 / 2 \varepsilon\le1/2 ε ≤ 1/2 ,
ι 1 = ι ( 1 + 2 ε 1 − ε ) ≤ ( 1 − 1 2 ℓ ) ( 1 + 1 4 ℓ ) ≤ 1 − 1 4 ℓ . \iota_1=\iota\left(1+\frac{2\varepsilon}{1-\varepsilon}\right)\leq\left(1-\frac{1}{2\ell}\right)\left(1+\frac{1}{4\ell}\right)\leq1-\frac{1}{4\ell}. ι 1 = ι ( 1 + 1 − ε 2 ε ) ≤ ( 1 − 2 ℓ 1 ) ( 1 + 4 ℓ 1 ) ≤ 1 − 4 ℓ 1 .
Thus 1 − ι 1 ≥ 1 / ( 4 ℓ ) 1-\iota_1\geq1/(4\ell) 1 − ι 1 ≥ 1/ ( 4 ℓ ) , as claimed. Fix W ≠ S W\neq S W = S with ∣ W ∣ ≤ q |W|\leq q ∣ W ∣ ≤ q , and write
W ‾ : = S ∪ W , F : = W ∖ S , S 0 : = S ∖ W , B ∗ : = min i B i i . \overline{W}:=S\cup W,\qquad F:=W\setminus S,\qquad S_0:=S\setminus W,\qquad B_*:=\min_i B_{ii}. W := S ∪ W , F := W ∖ S , S 0 := S ∖ W , B ∗ := i min B ii .
Empty determinants and products are understood to equal one, and quadratic forms on empty coordinate sets equal zero.
Step 1: the exact weight ratio. Set G = G τ G=G_\tau G = G τ , b = b τ b=b_\tau b = b τ , D : = ( diag B ) W ‾ W ‾ D:=(\operatorname{diag}B)_{\overline{W}\overline{W}} D := ( diag B ) W W and β ~ : = B W ‾ W ‾ − 1 b W ‾ \widetilde{\beta}:=B_{\overline{W}\overline{W}}^{-1}b_{\overline{W}} β := B W W − 1 b W . Since W ≠ S W\neq S W = S and ∣ W ‾ ∣ ≤ q + ∣ S ∣ ≤ 2 q |\overline{W}|\leq q+|S|\leq2q ∣ W ∣ ≤ q + ∣ S ∣ ≤ 2 q by (30) , the block condition (34) gives ∥ B W ‾ W ‾ − D ∥ ≤ ε min i ∈ W ‾ B i i \lVert B_{\overline{W}\overline{W}}-D\rVert\leq\varepsilon\min_{i\in\overline{W}}B_{ii} ∥ B W W − D ∥ ≤ ε min i ∈ W B ii . As D ⪰ ( min i ∈ W ‾ B i i ) I D\succeq(\min_{i\in\overline{W}}B_{ii})I D ⪰ ( min i ∈ W B ii ) I , this implies − ε D ⪯ B W ‾ W ‾ − D ⪯ ε D -\varepsilon D\preceq B_{\overline{W}\overline{W}}-D\preceq\varepsilon D − ε D ⪯ B W W − D ⪯ ε D , hence ( 1 − ε ) D ⪯ B W ‾ W ‾ ⪯ ( 1 + ε ) D (1-\varepsilon)D\preceq B_{\overline{W}\overline{W}}\preceq(1+\varepsilon)D ( 1 − ε ) D ⪯ B W W ⪯ ( 1 + ε ) D .
For ( J , K ) = ( S , F ) (J,K)=(S,F) ( J , K ) = ( S , F ) or ( W , S 0 ) (W,S_0) ( W , S 0 ) , the Schur complement is Υ K = B K K − B K J B J J − 1 B J K \Upsilon_K=B_{KK}-B_{KJ}B_{JJ}^{-1}B_{JK} Υ K = B K K − B K J B J J − 1 B J K . Completing the square in v J v_J v J gives
v K ⊤ Υ K v K = min v J ∈ R J v ⊤ B W ‾ W ‾ v v_K^\top\Upsilon_Kv_K=\min_{v_J\in\mathbb{R}^J}v^\top B_{\overline{W}\overline{W}}v v K ⊤ Υ K v K = v J ∈ R J min v ⊤ B W W v
for each fixed v K ∈ R K v_K\in\mathbb{R}^K v K ∈ R K , with the minimum attained at v J = − B J J − 1 B J K v K v_J=-B_{JJ}^{-1}B_{JK}v_K v J = − B J J − 1 B J K v K . The lower comparison with D D D yields Υ K ⪰ ( 1 − ε ) D K K \Upsilon_K\succeq(1-\varepsilon)D_{KK} Υ K ⪰ ( 1 − ε ) D K K , while taking v J = 0 v_J=0 v J = 0 gives Υ K ⪯ B K K ⪯ ( 1 + ε ) D K K \Upsilon_K\preceq B_{KK}\preceq(1+\varepsilon)D_{KK} Υ K ⪯ B K K ⪯ ( 1 + ε ) D K K . Hadamard’s inequality gives det Υ K ≤ ∏ i ∈ K ( Υ K ) i i ≤ ∏ i ∈ K B i i \det\Upsilon_K\leq\prod_{i\in K}(\Upsilon_K)_{ii}\leq\prod_{i\in K}B_{ii} det Υ K ≤ ∏ i ∈ K ( Υ K ) ii ≤ ∏ i ∈ K B ii , and the determinant is monotone on positive definite matrices. Thus, for K = F , S 0 K=F,S_0 K = F , S 0 ,
( 1 − ε ) D K K ⪯ Υ K ⪯ ( 1 + ε ) D K K , det Υ S 0 ≤ ∏ i ∈ S 0 B i i , det Υ F ≥ ∏ i ∈ F ( 1 − ε ) B i i . (79) (1-\varepsilon)D_{KK}\preceq\Upsilon_K\preceq(1+\varepsilon)D_{KK},\qquad\det\Upsilon_{S_0}\leq\prod_{i\in S_0}B_{ii},\qquad\det\Upsilon_F\geq\prod_{i\in F}(1-\varepsilon)B_{ii}. \tag*{(79)} ( 1 − ε ) D K K ⪯ Υ K ⪯ ( 1 + ε ) D K K , det Υ S 0 ≤ i ∈ S 0 ∏ B ii , det Υ F ≥ i ∈ F ∏ ( 1 − ε ) B ii . (79)
To compare the two weights, identify vectors supported on J ⊆ W ‾ J\subseteq\overline{W} J ⊆ W with their coordinate vectors in R J ⊆ R W ‾ \mathbb{R}^J\subseteq\mathbb{R}^{\overline{W}} R J ⊆ R W , and let ( J , K ) (J,K) ( J , K ) be ( S , F ) (S,F) ( S , F ) or ( W , S 0 ) (W,S_0) ( W , S 0 ) . Every v ∈ R J v\in\mathbb{R}^J v ∈ R J satisfies
2 b J ⊤ v − v ⊤ B J J v = 2 b W ‾ ⊤ v − v ⊤ B W ‾ W ‾ v = β ~ ⊤ B W ‾ W ‾ β ~ − ( β ~ − v ) ⊤ B W ‾ W ‾ ( β ~ − v ) , 2b_J^\top v-v^\top B_{JJ}v=2b_{\overline{W}}^\top v-v^\top B_{\overline{W}\overline{W}}v=\widetilde{\beta}^\top B_{\overline{W}\overline{W}}\widetilde{\beta}-(\widetilde{\beta}-v)^\top B_{\overline{W}\overline{W}}(\widetilde{\beta}-v), 2 b J ⊤ v − v ⊤ B J J v = 2 b W ⊤ v − v ⊤ B W W v = β ⊤ B W W β − ( β − v ) ⊤ B W W ( β − v ) ,
and β ~ − v \widetilde{\beta}-v β − v ranges over the extensions of β ~ K \widetilde{\beta}_K β K . Maximizing over v v v and using the variational identity above give b J ⊤ B J J − 1 b J = β ~ ⊤ B W ‾ W ‾ β ~ − β ~ K ⊤ Υ K β ~ K b_J^\top B_{JJ}^{-1}b_J=\widetilde{\beta}^\top B_{\overline{W}\overline{W}}\widetilde{\beta}-\widetilde{\beta}_K^\top\Upsilon_K\widetilde{\beta}_K b J ⊤ B J J − 1 b J = β ⊤ B W W β − β K ⊤ Υ K β K . With (52) and the block determinant identity det B W ‾ W ‾ = det B J J det Υ K \det B_{\overline{W}\overline{W}}=\det B_{JJ}\det\Upsilon_K det B W W = det B J J det Υ K , this gives
w W w S = w ∣ F ∣ − ∣ S 0 ∣ det Υ S 0 det Υ F exp { 1 2 β ~ F ⊤ Υ F β ~ F − 1 2 β ~ S 0 ⊤ Υ S 0 β ~ S 0 } . (80) \frac{w_W}{w_S}=w^{|F|-|S_0|}\sqrt{\frac{\det\Upsilon_{S_0}}{\det\Upsilon_F}}\exp\left\{\frac{1}{2}\widetilde{\beta}_F^\top\Upsilon_F\widetilde{\beta}_F-\frac{1}{2}\widetilde{\beta}_{S_0}^\top\Upsilon_{S_0}\widetilde{\beta}_{S_0}\right\}. \tag*{(80)} w S w W = w ∣ F ∣ − ∣ S 0 ∣ det Υ F det Υ S 0 exp { 2 1 β F ⊤ Υ F β F − 2 1 β S 0 ⊤ Υ S 0 β S 0 } . (80)
Step 2: average the Gaussian noise. Define the deterministic shifts
a : = B W ‾ W ‾ − 1 x W ‾ ⋆ , m F : = Υ F 1 / 2 ( x ⋆ − a ) F . a:=B_{\overline{W}\overline{W}}^{-1}x^\star_{\overline{W}},\qquad m_F:=\Upsilon_F^{1/2}(x^\star-a)_F. a := B W W − 1 x W ⋆ , m F := Υ F 1/2 ( x ⋆ − a ) F .
Substituting y = A x ⋆ + σ ζ y=Ax^\star+\sigma\zeta y = A x ⋆ + σ ζ into b = A ⊤ Σ τ − 1 y b=A^\top\Sigma_\tau^{-1}y b = A ⊤ Σ τ − 1 y gives b = G x ⋆ + η b=Gx^\star+\eta b = G x ⋆ + η , where η : = σ A ⊤ Σ τ − 1 ζ \eta:=\sigma A^\top\Sigma_\tau^{-1}\zeta η := σ A ⊤ Σ τ − 1 ζ is centered Gaussian. Since Σ τ ⪰ a τ I m ⪰ σ 2 I m \Sigma_\tau\succeq a_\tau I_m\succeq\sigma^2 I_m Σ τ ⪰ a τ I m ⪰ σ 2 I m , its covariance satisfies Cov η = σ 2 A ⊤ Σ τ − 2 A ⪯ G \operatorname{Cov}\eta=\sigma^2A^\top\Sigma_\tau^{-2}A\preceq G Cov η = σ 2 A ⊤ Σ τ − 2 A ⪯ G . As supp x ⋆ ⊆ W ‾ \operatorname{supp}x^\star\subseteq\overline{W} supp x ⋆ ⊆ W and G W ‾ W ‾ = B W ‾ W ‾ − I ⪯ B W ‾ W ‾ G_{\overline{W}\overline{W}}=B_{\overline{W}\overline{W}}-I\preceq B_{\overline{W}\overline{W}} G W W = B W W − I ⪯ B W W ,
β ~ = x W ‾ ⋆ − a + B W ‾ W ‾ − 1 η W ‾ , Cov β ~ = B W ‾ W ‾ − 1 Cov ( η W ‾ ) B W ‾ W ‾ − 1 ⪯ B W ‾ W ‾ − 1 ⪯ ( 1 − ε ) − 1 D − 1 . \widetilde{\beta}=x^\star_{\overline{W}}-a+B_{\overline{W}\overline{W}}^{-1}\eta_{\overline{W}},\qquad\operatorname{Cov}\widetilde{\beta}=B_{\overline{W}\overline{W}}^{-1}\operatorname{Cov}(\eta_{\overline{W}})B_{\overline{W}\overline{W}}^{-1}\preceq B_{\overline{W}\overline{W}}^{-1}\preceq(1-\varepsilon)^{-1}D^{-1}. β = x W ⋆ − a + B W W − 1 η W , Cov β = B W W − 1 Cov ( η W ) B W W − 1 ⪯ B W W − 1 ⪯ ( 1 − ε ) − 1 D − 1 .
To bound the exponent, set r : = D 1 / 2 β ~ r:=D^{1/2}\widetilde{\beta} r := D 1/2 β , r ‾ : = E ζ r = D 1 / 2 ( x ⋆ − a ) W ‾ \overline{r}:=\mathbb{E}_\zeta r=D^{1/2}(x^\star-a)_{\overline{W}} r := E ζ r = D 1/2 ( x ⋆ − a ) W and z : = r − r ‾ z:=r-\overline{r} z := r − r , so that z z z is centered Gaussian with Cov z ⪯ ( 1 − ε ) − 1 I \operatorname{Cov}z\preceq(1-\varepsilon)^{-1}I Cov z ⪯ ( 1 − ε ) − 1 I by the last step. By (79) , the exponent in (80) is at most 1 2 ( 1 + ε ) ∥ r F ∥ 2 − 1 2 ( 1 − ε ) ∥ r S 0 ∥ 2 \frac{1}{2}(1+\varepsilon)\lVert r_F\rVert^2-\frac{1}{2}(1-\varepsilon)\lVert r_{S_0}\rVert^2 2 1 ( 1 + ε ) ∥ r F ∥ 2 − 2 1 ( 1 − ε ) ∥ r S 0 ∥ 2 , and ∥ r S 0 ∥ 2 ≥ 1 2 ∥ r ‾ S 0 ∥ 2 − ∥ z S 0 ∥ 2 \lVert r_{S_0}\rVert^2\geq\frac{1}{2}\lVert\overline{r}_{S_0}\rVert^2-\lVert z_{S_0}\rVert^2 ∥ r S 0 ∥ 2 ≥ 2 1 ∥ r S 0 ∥ 2 − ∥ z S 0 ∥ 2 . Raising (80) to the power ℓ \ell ℓ therefore gives
( w W w S ) ℓ ≤ w ℓ ( ∣ F ∣ − ∣ S 0 ∣ ) ( det Υ S 0 det Υ F ) ℓ / 2 e f ( z ) e − ℓ ( 1 − ε ) ∥ r ‾ S 0 ∥ 2 / 4 . (81) \left(\frac{w_W}{w_S}\right)^\ell\leq w^{\ell(|F|-|S_0|)}\left(\frac{\det\Upsilon_{S_0}}{\det\Upsilon_F}\right)^{\ell/2}e^{f(z)}e^{-\ell(1-\varepsilon)\lVert\overline{r}_{S_0}\rVert^2/4}. \tag*{(81)} ( w S w W ) ℓ ≤ w ℓ ( ∣ F ∣ − ∣ S 0 ∣ ) ( det Υ F det Υ S 0 ) ℓ /2 e f ( z ) e − ℓ ( 1 − ε ) ∥ r S 0 ∥ 2 /4 . (81)
with the convex function
f ( z ) : = ι 2 ( ( 1 + ε ) ∥ r ˉ F + z F ∥ 2 + ( 1 − ε ) ∥ z S 0 ∥ 2 ) . f(z) := \frac{\iota}{2}\left((1+\varepsilon)\lVert\bar r_F+z_F\rVert^2+(1-\varepsilon)\lVert z_{S_0}\rVert^2\right). f ( z ) := 2 ι ( ( 1 + ε ) ∥ r ˉ F + z F ∥ 2 + ( 1 − ε ) ∥ z S 0 ∥ 2 ) .
We now enlarge the covariance in this convex upper bound. Let w ∼ N ( 0 , ( 1 − ε ) − 1 I − Cov z ) w \sim\mathcal{N}(0,(1-\varepsilon)^{-1}I-\operatorname{Cov}z) w ∼ N ( 0 , ( 1 − ε ) − 1 I − Cov z ) be independent of z z z . Jensen’s inequality for e f ( z ) e^{f(z)} e f ( z ) , conditionally on z z z , gives E e f ( z ) ≤ E e f ( z + w ) \mathbb{E}e^{f(z)} \leq\mathbb{E}e^{f(z+w)} E e f ( z ) ≤ E e f ( z + w ) , and the coordinates of z + w z+w z + w are independent with variance v : = ( 1 − ε ) − 1 v:=(1-\varepsilon)^{-1} v := ( 1 − ε ) − 1 . For real μ \mu μ and γ \gamma γ , a scalar g 0 ∼ N ( 0 , v ) g_0\sim\mathcal{N}(0,v) g 0 ∼ N ( 0 , v ) with γ v < 1 \gamma v<1 γ v < 1 satisfies
E e γ ( μ + g 0 ) 2 / 2 = ( 1 − γ v ) − 1 / 2 e γ μ 2 / ( 2 ( 1 − γ v ) ) . \mathbb{E}e^{\gamma(\mu+g_0)^2/2}=(1-\gamma v)^{-1/2}e^{\gamma\mu^2/(2(1-\gamma v))}. E e γ ( μ + g 0 ) 2 /2 = ( 1 − γ v ) − 1/2 e γ μ 2 / ( 2 ( 1 − γ v )) .
On F F F we take γ = ι ( 1 + ε ) \gamma=\iota(1+\varepsilon) γ = ι ( 1 + ε ) and μ = r ˉ i \mu=\bar r_i μ = r ˉ i , so that γ v = ι 1 \gamma v=\iota_1 γ v = ι 1 . On S 0 S_0 S 0 we take γ = ι ( 1 − ε ) \gamma=\iota(1-\varepsilon) γ = ι ( 1 − ε ) and μ = 0 \mu=0 μ = 0 , so that γ v = ι ≤ ι 1 \gamma v=\iota\leq\iota_1 γ v = ι ≤ ι 1 . As r ˉ F = − D F F 1 / 2 a F \bar r_F=-D_{FF}^{1/2}a_F r ˉ F = − D F F 1/2 a F and m F = − Υ F 1 / 2 a F m_F=-\Upsilon_F^{1/2}a_F m F = − Υ F 1/2 a F , (79) gives ∥ r ˉ F ∥ 2 ≤ ∥ m F ∥ 2 / ( 1 − ε ) \lVert\bar r_F\rVert^2\leq\lVert m_F\rVert^2/(1-\varepsilon) ∥ r ˉ F ∥ 2 ≤ ∥ m F ∥ 2 / ( 1 − ε ) . Hence
E ζ e f ( z ) ≤ ( 1 − ι 1 ) − ( ∣ F ∣ + ∣ S 0 ∣ ) / 2 exp { ι ( 1 + ε ) ∥ r ˉ F ∥ 2 2 ( 1 − ι 1 ) } ≤ ( 1 − ι 1 ) − ( ∣ F ∣ + ∣ S 0 ∣ ) / 2 exp { ι ( 1 + ε ) ∥ m F ∥ 2 2 ( 1 − ε ) ( 1 − ι 1 ) } . \mathbb{E}_{\zeta}e^{f(z)}\leq(1-\iota_1)^{-(|F|+|S_0|)/2}\exp\left\{\frac{\iota(1+\varepsilon)\lVert\bar r_F\rVert^2}{2(1-\iota_1)}\right\}\leq(1-\iota_1)^{-(|F|+|S_0|)/2}\exp\left\{\frac{\iota(1+\varepsilon)\lVert m_F\rVert^2}{2(1-\varepsilon)(1-\iota_1)}\right\}. E ζ e f ( z ) ≤ ( 1 − ι 1 ) − ( ∣ F ∣ + ∣ S 0 ∣ ) /2 exp { 2 ( 1 − ι 1 ) ι ( 1 + ε ) ∥ r ˉ F ∥ 2 } ≤ ( 1 − ι 1 ) − ( ∣ F ∣ + ∣ S 0 ∣ ) /2 exp { 2 ( 1 − ε ) ( 1 − ι 1 ) ι ( 1 + ε ) ∥ m F ∥ 2 } .
For the omitted coordinates, let S 0 ≠ ∅ S_0\neq\varnothing S 0 = ∅ . As r ˉ i 2 = B i i ( x i ⋆ − a i ) 2 ≥ 1 2 B i i ( x i ⋆ ) 2 − B i i a i 2 \bar r_i^2=B_{ii}(x_i^\star-a_i)^2\geq\frac12B_{ii}(x_i^\star)^2-B_{ii}a_i^2 r ˉ i 2 = B ii ( x i ⋆ − a i ) 2 ≥ 2 1 B ii ( x i ⋆ ) 2 − B ii a i 2 and ( 1 − ε ) a ⊤ D a ≤ a ⊤ B W ˉ W ˉ a (1-\varepsilon)a^\top Da\leq a^\top B_{\bar W\bar W}a ( 1 − ε ) a ⊤ D a ≤ a ⊤ B W ˉ W ˉ a ,
e − ι ( 1 − ε ) ∥ r ˉ S 0 ∥ 2 / 4 ≤ e ι ( 1 − ε ) a ⊤ D a / 4 ∏ i ∈ S 0 e − ι ( 1 − ε ) B i i ( x i ⋆ ) 2 / 8 ≤ e ι a ⊤ B W ˉ W ˉ a / 4 ∏ i ∈ S 0 e − ι ( 1 − ε ) B i i ( x i ⋆ ) 2 / 8 . e^{-\iota(1-\varepsilon)\lVert\bar r_{S_0}\rVert^2/4}\leq e^{\iota(1-\varepsilon)a^\top Da/4}\prod_{i\in S_0}e^{-\iota(1-\varepsilon)B_{ii}(x_i^\star)^2/8}\leq e^{\iota a^\top B_{\bar W\bar W}a/4}\prod_{i\in S_0}e^{-\iota(1-\varepsilon)B_{ii}(x_i^\star)^2/8}. e − ι ( 1 − ε ) ∥ r ˉ S 0 ∥ 2 /4 ≤ e ι ( 1 − ε ) a ⊤ D a /4 i ∈ S 0 ∏ e − ι ( 1 − ε ) B ii ( x i ⋆ ) 2 /8 ≤ e ι a ⊤ B W ˉ W ˉ a /4 i ∈ S 0 ∏ e − ι ( 1 − ε ) B ii ( x i ⋆ ) 2 /8 .
If S 0 = ∅ S_0=\varnothing S 0 = ∅ , the left side is one.
Finally, by (79) ,
ω ι ( ∣ F ∣ − ∣ S 0 ∣ ) ( det Υ S 0 det Υ F ) ι / 2 ( 1 − ι 1 ) − ( ∣ F ∣ + ∣ S 0 ∣ ) / 2 ≤ ∏ i ∈ F ω ι ( ( 1 − ε ) B i i ) − ι / 2 1 − ι 1 ∏ i ∈ S 0 ω − ι B i i ι / 2 1 − ι 1 . \omega^{\iota(|F|-|S_0|)}\left(\frac{\det\Upsilon_{S_0}}{\det\Upsilon_F}\right)^{\iota/2}(1-\iota_1)^{-(|F|+|S_0|)/2}\leq\prod_{i\in F}\frac{\omega^{\iota}((1-\varepsilon)B_{ii})^{-\iota/2}}{\sqrt{1-\iota_1}}\prod_{i\in S_0}\frac{\omega^{-\iota}B_{ii}^{\iota/2}}{\sqrt{1-\iota_1}}. ω ι ( ∣ F ∣ − ∣ S 0 ∣ ) ( det Υ F det Υ S 0 ) ι /2 ( 1 − ι 1 ) − ( ∣ F ∣ + ∣ S 0 ∣ ) /2 ≤ i ∈ F ∏ 1 − ι 1 ω ι (( 1 − ε ) B ii ) − ι /2 i ∈ S 0 ∏ 1 − ι 1 ω − ι B ii ι /2 .
For i ∈ S i\in S i ∈ S , write ψ i : = ( e n / α ) − C a ψ i ′ \psi_i:=(en/\alpha)^{-C_a}\psi_i' ψ i := ( e n / α ) − C a ψ i ′ . Taking E ζ \mathbb{E}_{\zeta} E ζ in (81) and inserting the last three displays gives
E ζ ( ω W ω S ) ι ≤ exp { ι ( 1 + ε ) ∥ m F ∥ 2 2 ( 1 − ε ) ( 1 − ι 1 ) } e ι a ⊤ B W ˉ W ˉ a / 4 ∏ i ∈ F χ i ∏ i ∈ S 0 ψ i , (82) \mathbb{E}_{\zeta}\left(\frac{\omega_W}{\omega_S}\right)^\iota\leq\exp\left\{\frac{\iota(1+\varepsilon)\lVert m_F\rVert^2}{2(1-\varepsilon)(1-\iota_1)}\right\}e^{\iota a^\top B_{\bar W\bar W}a/4}\prod_{i\in F}\chi_i\prod_{i\in S_0}\psi_i, \tag*{(82)} E ζ ( ω S ω W ) ι ≤ exp { 2 ( 1 − ε ) ( 1 − ι 1 ) ι ( 1 + ε ) ∥ m F ∥ 2 } e ι a ⊤ B W ˉ W ˉ a /4 i ∈ F ∏ χ i i ∈ S 0 ∏ ψ i , (82)
where the factor e ι a ⊤ B W ˉ W ˉ a / 4 e^{\iota a^\top B_{\bar W\bar W}a/4} e ι a ⊤ B W ˉ W ˉ a /4 is omitted when S 0 = ∅ S_0=\varnothing S 0 = ∅ .
Step 3: absorb the deterministic shifts. As x F ⋆ = 0 x_F^\star=0 x F ⋆ = 0 , eliminating a S a_S a S from B W ˉ W ˉ a = x W ˉ ⋆ B_{\bar W\bar W}a=x_{\bar W}^\star B W ˉ W ˉ a = x W ˉ ⋆ gives Υ F a F = − B F S B S S − 1 x S ⋆ \Upsilon_Fa_F=-B_{FS}B_{SS}^{-1}x_S^\star Υ F a F = − B F S B S S − 1 x S ⋆ . Hence
m F = − Υ F 1 / 2 a F = Υ F − 1 / 2 B F S B S S − 1 x S ⋆ . m_F=-\Upsilon_F^{1/2}a_F=\Upsilon_F^{-1/2}B_{FS}B_{SS}^{-1}x_S^\star. m F = − Υ F 1/2 a F = Υ F − 1/2 B F S B S S − 1 x S ⋆ .
Set B ⋆ , W ˉ : = min i ∈ W ˉ B i i ≥ B ⋆ B_{\star,\bar W}:=\min_{i\in\bar W}B_{ii}\geq B_\star B ⋆ , W ˉ := min i ∈ W ˉ B ii ≥ B ⋆ . The block B F S B_{FS} B F S of B W ˉ W ˉ − D B_{\bar W\bar W}-D B W ˉ W ˉ − D has norm at most ε B ⋆ , W ˉ \varepsilon B_{\star,\bar W} ε B ⋆ , W ˉ , and Υ F \Upsilon_F Υ F , B S S B_{SS} B S S and B W ˉ W ˉ B_{\bar W\bar W} B W ˉ W ˉ are at least ( 1 − ε ) B ⋆ , W ˉ I (1-\varepsilon)B_{\star,\bar W}I ( 1 − ε ) B ⋆ , W ˉ I by (79) and the preceding comparison with D D D . Hence
∥ m F ∥ 2 ≤ ε 2 B ⋆ , W ˉ 2 ∥ x ⋆ ∥ 2 ( 1 − ε ) 3 B ⋆ , W ˉ 3 ≤ ε 2 ∥ x ⋆ ∥ 2 ( 1 − ε ) 3 B ⋆ , a ⊤ B W ˉ W ˉ a = ( x W ˉ ⋆ ) ⊤ B W ˉ W ˉ − 1 x W ˉ ⋆ ≤ ∥ x ⋆ ∥ 2 ( 1 − ε ) B ⋆ . \lVert m_F\rVert^2\leq\frac{\varepsilon^2B_{\star,\bar W}^2\lVert x^\star\rVert^2}{(1-\varepsilon)^3B_{\star,\bar W}^3}\leq\frac{\varepsilon^2\lVert x^\star\rVert^2}{(1-\varepsilon)^3B_\star},\qquad a^\top B_{\bar W\bar W}a=(x_{\bar W}^\star)^\top B_{\bar W\bar W}^{-1}x_{\bar W}^\star\leq\frac{\lVert x^\star\rVert^2}{(1-\varepsilon)B_\star}. ∥ m F ∥ 2 ≤ ( 1 − ε ) 3 B ⋆ , W ˉ 3 ε 2 B ⋆ , W ˉ 2 ∥ x ⋆ ∥ 2 ≤ ( 1 − ε ) 3 B ⋆ ε 2 ∥ x ⋆ ∥ 2 , a ⊤ B W ˉ W ˉ a = ( x W ˉ ⋆ ) ⊤ B W ˉ W ˉ − 1 x W ˉ ⋆ ≤ ( 1 − ε ) B ⋆ ∥ x ⋆ ∥ 2 .
By (71) , B ⋆ ≥ 1 / ( 8 σ 2 ) B_\star\geq1/(8\sigma^2) B ⋆ ≥ 1/ ( 8 σ 2 ) , and the signal-norm hypothesis gives σ 2 ∥ x ⋆ ∥ 2 ≤ ℓ \sigma^2\lVert x^\star\rVert^2\leq\ell σ 2 ∥ x ⋆ ∥ 2 ≤ ℓ . Together with (70) , the preceding bounds imply
∥ m F ∥ 2 ≤ C ε 2 σ 2 ∥ x ⋆ ∥ 2 ≤ C ε 2 ℓ , a ⊤ B W ˉ W ˉ a ≤ C σ 2 ∥ x ⋆ ∥ 2 ≤ C ℓ . \lVert m_F\rVert^2\leq C\varepsilon^2\sigma^2\lVert x^\star\rVert^2\leq\frac{C\varepsilon^2}{\ell},\qquad a^\top B_{\bar W\bar W}a\leq C\sigma^2\lVert x^\star\rVert^2\leq C\ell. ∥ m F ∥ 2 ≤ C ε 2 σ 2 ∥ x ⋆ ∥ 2 ≤ ℓ C ε 2 , a ⊤ B W ˉ W ˉ a ≤ C σ 2 ∥ x ⋆ ∥ 2 ≤ C ℓ .
Since 1 − ι 1 ≥ 1 / ( 4 ℓ ) 1-\iota_1\geq1/(4\ell) 1 − ι 1 ≥ 1/ ( 4 ℓ ) , the first exponential in (82) is at most a universal constant. Choose a universal C a C_a C a large enough that the second exponential is at most e C a ℓ = ( e n / α ) C a e^{C_a\ell}=(en/\alpha)^{C_a} e C a ℓ = ( e n / α ) C a , uniformly over W W W . When S 0 ≠ ∅ S_0\neq\varnothing S 0 = ∅ , this factor is at most ∏ i ∈ S 0 ( e n / α ) C a \prod_{i\in S_0}(en/\alpha)^{C_a} ∏ i ∈ S 0 ( e n / α ) C a , so it is absorbed into ∏ i ∈ S 0 ψ i \prod_{i\in S_0}\psi_i ∏ i ∈ S 0 ψ i . When S 0 = ∅ S_0=\varnothing S 0 = ∅ , the second exponential is absent and the omitted-coordinate product equals one. Substitution into (82) proves (74) .
Proof of Lemma 3
We first identify the denoised terminal target as a posterior with a modified observation covariance, and then compare it with the true posterior.
The modified observation covariance. For a matrix A A A , an observation y ∈ R m y \in\mathbb{R}^{m} y ∈ R m and any positive definite Σ ∈ R m × m \Sigma\in\mathbb{R}^{m \times m} Σ ∈ R m × m , define
ν Σ A , y ( d z ) ∝ e − ( y − A z ) T Σ − 1 ( y − A z ) / 2 p ⋆ ( d z ) . (83) \nu_{\Sigma}^{A,y}(dz) \propto e^{-(y-Az)^{\mathsf{T}}\Sigma^{-1}(y-Az)/2}p_{\star}(dz). \tag*{(83)} ν Σ A , y ( d z ) ∝ e − ( y − A z ) T Σ − 1 ( y − A z ) /2 p ⋆ ( d z ) . (83)
This is the conditional law of Z ∼ p ⋆ Z \sim p_{\star} Z ∼ p ⋆ given A Z + w = y AZ+w=y A Z + w = y , where w ∼ N ( 0 , Σ ) w \sim\mathcal{N}(0,\Sigma) w ∼ N ( 0 , Σ ) is independent of Z Z Z .
Fix A A A , y y y and t > 0 t>0 t > 0 , and recall the auxiliary model of Appendix § 7 .§ 2 . The endpoint kernel of Definition 3 is D t ( x , ⋅ ) = L ( Z ∣ X t = x ) D_t(x,\cdot)=\mathcal{L}(Z\mid X_t=x) D t ( x , ⋅ ) = L ( Z ∣ X t = x ) . Since Y ~ t \widetilde{Y}_t Y t depends on Z Z Z only through X t X_t X t , this is also the conditional law of Z Z Z given X t = x X_t=x X t = x and Y ~ t = y \widetilde{Y}_t=y Y t = y . Integrating against Π t = L ( X t ∣ Y ~ t = y ) \Pi_t=\mathcal{L}(X_t\mid\widetilde{Y}_t=y) Π t = L ( X t ∣ Y t = y ) therefore gives Π t D t = P t \Pi_tD_t=\mathcal{P}_t Π t D t = P t . Combining this identity with (50) and the true observation model yields
Π t D t = P t = ν Σ t A , y , p ⋆ A , y = ν σ 2 I m A , y . (84) \Pi_tD_t=\mathcal{P}_t=\nu_{\Sigma_t}^{A,y},\qquad p_{\star}^{A,y}=\nu_{\sigma^2 I_m}^{A,y}. \tag*{(84)} Π t D t = P t = ν Σ t A , y , p ⋆ A , y = ν σ 2 I m A , y . (84)
Thus the endpoint draw applied to the target yields a posterior whose observation covariance is Σ t = σ 2 I m + t ( 2 I m + A A T ) \Sigma_t=\sigma^2I_m+t(2I_m+AA^{\mathsf{T}}) Σ t = σ 2 I m + t ( 2 I m + A A T ) . At the terminal level,
Π τ D τ = ν σ 2 I m + Ξ A , y , Ξ : = τ ( 2 I m + A A T ) ⪰ 0. (85) \Pi_{\tau}D_{\tau}=\nu_{\sigma^2I_m+\Xi}^{A,y},\qquad\Xi:=\tau(2I_m+AA^{\mathsf{T}})\succeq0. \tag*{(85)} Π τ D τ = ν σ 2 I m + Ξ A , y , Ξ := τ ( 2 I m + A A T ) ⪰ 0. (85)
Comparison and averaging. The following bound at fixed data controls the total variation between this law and the true posterior by the size of Ξ \Xi Ξ .
Lemma 18 (Comparison at fixed data). Let A A A be a matrix, y ∈ R m y\in\mathbb{R}^{m} y ∈ R m and Ξ ∈ R m × m \Xi\in\mathbb{R}^{m\times m} Ξ ∈ R m × m symmetric with $\Xi\succeq 0. Then
TV ( ν σ 2 I m + Ξ A , y , p ⋆ A , y ) ≤ ∥ Ξ ∥ 2 σ 2 ( m + ∥ y ∥ 2 σ 2 ) . (86) \operatorname{TV}\left(\nu_{\sigma^2I_m+\Xi}^{A,y},p_{\star}^{A,y}\right)\leq\frac{\lVert\Xi\rVert}{2\sigma^2}\left(m+\frac{\lVert y\rVert^2}{\sigma^2}\right). \tag*{(86)} TV ( ν σ 2 I m + Ξ A , y , p ⋆ A , y ) ≤ 2 σ 2 ∥ Ξ ∥ ( m + σ 2 ∥ y ∥ 2 ) . (86)
The proof of this lemma can be found in Appendix § 9 .2.1.
We now finish the proof of Lemma 3. Fix A A A satisfying ∥ A ∥ 2 ≤ 2 n \lVert A\rVert^2\leq2n ∥ A ∥ 2 ≤ 2 n . The covariance perturbation in (85) satisfies ∥ Ξ ∥ ≤ τ ( 2 + 2 n ) ≤ 4 n τ \lVert\Xi\rVert\leq\tau(2+2n)\leq4n\tau ∥ Ξ ∥ ≤ τ ( 2 + 2 n ) ≤ 4 n τ . Since X ⋆ X^\star X ⋆ is independent of A A A and has independent, centered coordinates with second moment s / n s/n s / n ,
E [ ∥ y ∥ 2 ∣ A ] = s n ∥ A ∥ F 2 + σ 2 m ≤ 2 s m + σ 2 m , \mathbb{E}[\lVert y\rVert^2\mid A]=\frac{s}{n}\lVert A\rVert_F^2+\sigma^2m\leq2sm+\sigma^2m, E [∥ y ∥ 2 ∣ A ] = n s ∥ A ∥ F 2 + σ 2 m ≤ 2 s m + σ 2 m ,
where we used ∥ A ∥ F 2 ≤ m ∥ A ∥ 2 ≤ 2 n m \lVert A\rVert_F^2\leq m\lVert A\rVert^2\leq2nm ∥ A ∥ F 2 ≤ m ∥ A ∥ 2 ≤ 2 nm . Applying (86) and averaging over the observation at this fixed A A A gives
E [ TV ( Π τ D τ , p ⋆ A , y ) ∣ A ] ≤ 2 n τ σ 2 ( m + E [ ∥ y ∥ 2 ∣ A ] σ 2 ) ≤ 4 n m τ σ 2 ( 1 + s σ 2 ) . \begin{aligned}
\mathbb{E}\left[\operatorname{TV}\left(\Pi_{\tau}D_{\tau},p_{\star}^{A,y}\right)\mid A\right]\leq\frac{2n\tau}{\sigma^2}\left(m+\frac{\mathbb{E}[\lVert y\rVert^2\mid A]}{\sigma^2}\right) \\
&\leq\frac{4nm\tau}{\sigma^2}\left(1+\frac{s}{\sigma^2}\right).
\end{aligned} E [ TV ( Π τ D τ , p ⋆ A , y ) ∣ A ] ≤ σ 2 2 n τ ( m + σ 2 E [∥ y ∥ 2 ∣ A ] ) ≤ σ 2 4 nm τ ( 1 + σ 2 s ) .
Averaging this bound over A A A on the event { ∥ A ∥ 2 ≤ 2 n } \{\lVert A\rVert^2\leq2n\} {∥ A ∥ 2 ≤ 2 n } and using m ≤ n m\leq n m ≤ n proves the claim.
Proof of Lemma 18
We interpolate between the two noise covariances to reduce the comparison to a residual bound, then establish that bound uniformly over the latent supports.
Step 1: reduction to a residual bound. For 0 ≤ v ≤ 1 0\leq v\leq1 0 ≤ v ≤ 1 set
Σ ( v ) : = σ 2 I m + v Ξ , ν ( v ) : = ν Σ ( v ) A , y . \Sigma^{(v)}:=\sigma^2I_m+v\Xi,\qquad\nu^{(v)}:=\nu_{\Sigma^{(v)}}^{A,y}. Σ ( v ) := σ 2 I m + v Ξ , ν ( v ) := ν Σ ( v ) A , y .
so that ν ( 0 ) = p ⋆ A , y \nu^{(0)}=p_{\star}^{A,y} ν ( 0 ) = p ⋆ A , y and ν ( 1 ) = ν σ 2 I m + Ξ A , y \nu^{(1)}=\nu_{\sigma^2I_m+\Xi}^{A,y} ν ( 1 ) = ν σ 2 I m + Ξ A , y . The unnormalized density of ν ( v ) \nu^{(v)} ν ( v ) with respect to p ⋆ p_{\star} p ⋆ is e − ϕ v e^{-\phi_v} e − ϕ v with ϕ v ( z ) : = ( y − A z ) T ( Σ ( v ) ) − 1 ( y − A z ) / 2 \phi_v(z):=(y-Az)^{\mathsf{T}}(\Sigma^{(v)})^{-1}(y-Az)/2 ϕ v ( z ) := ( y − A z ) T ( Σ ( v ) ) − 1 ( y − A z ) /2 , and − ϕ v -\phi_v − ϕ v has v v v -derivative
u v ( z ) : = 1 2 ( y − A z ) T ( Σ ( v ) ) − 1 Ξ ( Σ ( v ) ) − 1 ( y − A z ) ≥ 0. u_v(z):=\frac{1}{2}(y-Az)^{\mathsf{T}}(\Sigma^{(v)})^{-1}\Xi(\Sigma^{(v)})^{-1}(y-Az)\geq0. u v ( z ) := 2 1 ( y − A z ) T ( Σ ( v ) ) − 1 Ξ ( Σ ( v ) ) − 1 ( y − A z ) ≥ 0.
Differentiation under the integral is justified by the finite second moment of p ⋆ p_\star p ⋆ and the bound Σ ( v ) ⪰ σ 2 I m \Sigma^{(v)} \succeq\sigma^2 I_m Σ ( v ) ⪰ σ 2 I m . Differentiating the normalized density gives ∂ v ν ( v ) = ( u v − ν ( v ) u v ) ν ( v ) \partial_v \nu^{(v)}=(u_v-\nu^{(v)}u_v)\nu^{(v)} ∂ v ν ( v ) = ( u v − ν ( v ) u v ) ν ( v ) , where the centering term comes from differentiating the normalizing constant. Thus
TV ( ν ( 1 ) , ν ( 0 ) ) ≤ 1 2 ∫ 0 1 ∣ ν ( v ) u v − ν ( v ) u v ∣ d v ≤ ∫ 0 1 ν ( v ) u v d v , \operatorname{TV}(\nu^{(1)},\nu^{(0)})\leq\frac{1}{2}\int_0^1\left|\nu^{(v)}u_v-\nu^{(v)}u_v\right|\,dv\leq\int_0^1\nu^{(v)}u_v\,dv, TV ( ν ( 1 ) , ν ( 0 ) ) ≤ 2 1 ∫ 0 1 ν ( v ) u v − ν ( v ) u v d v ≤ ∫ 0 1 ν ( v ) u v d v ,
where the second bound is ∣ a − b ∣ ≤ a + b |a-b|\leq a+b ∣ a − b ∣ ≤ a + b for a , b ≥ 0 a,b\geq0 a , b ≥ 0 .
Since Σ ( v ) ⪰ σ 2 I m \Sigma^{(v)}\succeq\sigma^2 I_m Σ ( v ) ⪰ σ 2 I m , the matrix ( Σ ( v ) ) − 1 / 2 Ξ ( Σ ( v ) ) − 1 / 2 (\Sigma^{(v)})^{-1/2}\Xi(\Sigma^{(v)})^{-1/2} ( Σ ( v ) ) − 1/2 Ξ ( Σ ( v ) ) − 1/2 is at most ( ∥ Ξ ∥ / σ 2 ) I m (\|\Xi\|/\sigma^2)I_m ( ∥Ξ∥/ σ 2 ) I m , and
u v ≤ ∥ Ξ ∥ 2 σ 2 ∥ ( Σ ( v ) ) − 1 / 2 ( y − A z ) ∥ 2 . u_v\leq\frac{\|\Xi\|}{2\sigma^2}\left\|(\Sigma^{(v)})^{-1/2}(y-Az)\right\|^2. u v ≤ 2 σ 2 ∥Ξ∥ ( Σ ( v ) ) − 1/2 ( y − A z ) 2 .
Combining the last two displays yields
TV ( ν ( 1 ) , ν ( 0 ) ) ≤ ∥ Ξ ∥ 2 σ 2 ∫ 0 1 E ν ( v ) [ ∥ ( Σ ( v ) ) − 1 / 2 ( y − A z ) ∥ 2 ] d v . \operatorname{TV}(\nu^{(1)},\nu^{(0)})\leq\frac{\|\Xi\|}{2\sigma^2}\int_0^1\mathbb{E}_{\nu^{(v)}}\left[\left\|(\Sigma^{(v)})^{-1/2}(y-Az)\right\|^2\right]\,dv. TV ( ν ( 1 ) , ν ( 0 ) ) ≤ 2 σ 2 ∥Ξ∥ ∫ 0 1 E ν ( v ) [ ( Σ ( v ) ) − 1/2 ( y − A z ) 2 ] d v .
It therefore suffices to bound the integrand by m + ∥ y ∥ 2 / σ 2 m+\|y\|^2/\sigma^2 m + ∥ y ∥ 2 / σ 2 , uniformly in v ∈ [ 0 , 1 ] v\in[0,1] v ∈ [ 0 , 1 ] .
Step 2: bound the residual uniformly over supports. We prove the same bound conditional on every support, so no estimate of the posterior support probabilities is needed. Fix v ∈ [ 0 , 1 ] v\in[0,1] v ∈ [ 0 , 1 ] , condition under ν ( v ) \nu^{(v)} ν ( v ) on the support W W W of z z z , including W = ∅ W=\varnothing W = ∅ , and set F : = ( Σ ( v ) ) − 1 / 2 A W F:=(\Sigma^{(v)})^{-1/2}A_W F := ( Σ ( v ) ) − 1/2 A W and y ~ : = ( Σ ( v ) ) − 1 / 2 y \widetilde{y}:=(\Sigma^{(v)})^{-1/2}y y := ( Σ ( v ) ) − 1/2 y . Under this conditional law, z W z_W z W is Gaussian with precision I + F ⊤ F I+F^\top F I + F ⊤ F and mean ( I + F ⊤ F ) − 1 F ⊤ y ~ (I+F^\top F)^{-1}F^\top\widetilde{y} ( I + F ⊤ F ) − 1 F ⊤ y . The identity
F ( I + F ⊤ F ) − 1 F ⊤ = I m − ( I m + F F ⊤ ) − 1 F(I+F^\top F)^{-1}F^\top=I_m-(I_m+FF^\top)^{-1} F ( I + F ⊤ F ) − 1 F ⊤ = I m − ( I m + F F ⊤ ) − 1
shows that the whitened residual y ~ − F z W \widetilde{y}-Fz_W y − F z W has mean ( I m + F F ⊤ ) − 1 y ~ (I_m+FF^\top)^{-1}\widetilde{y} ( I m + F F ⊤ ) − 1 y and covariance bounded by I m I_m I m . Consequently,
E ν ( v ) [ ∥ y ~ − F z W ∥ 2 ∣ W ] = ∥ ( I m + F F ⊤ ) − 1 y ~ ∥ 2 + tr ( F ( I + F ⊤ F ) − 1 F ⊤ ) ≤ ∥ y ∥ 2 σ 2 + m , \mathbb{E}_{\nu^{(v)}}\left[\left\|\widetilde{y}-Fz_W\right\|^2\mid W\right]=\left\|(I_m+FF^\top)^{-1}\widetilde{y}\right\|^2+\operatorname{tr}\left(F(I+F^\top F)^{-1}F^\top\right)\leq\frac{\|y\|^2}{\sigma^2}+m, E ν ( v ) [ ∥ y − F z W ∥ 2 ∣ W ] = ( I m + F F ⊤ ) − 1 y 2 + tr ( F ( I + F ⊤ F ) − 1 F ⊤ ) ≤ σ 2 ∥ y ∥ 2 + m ,
since ∥ ( I m + F F ⊤ ) − 1 ∥ ≤ 1 \|(I_m+FF^\top)^{-1}\|\leq1 ∥ ( I m + F F ⊤ ) − 1 ∥ ≤ 1 and ∥ y ~ ∥ ≤ ∥ y ∥ / σ \|\widetilde{y}\|\leq\|y\|/\sigma ∥ y ∥ ≤ ∥ y ∥/ σ . Averaging over the posterior support law gives the required uniform residual bound and proves (86) .
Verification of the matrix condition
This appendix verifies the matrix condition of Definition Definition 5 for the three sensing families. The following proposition supplies the probability bounds used in § 5.2 and § 5.3 .
Proposition 3 (The matrix event). There are universal constants c , C > 0 c,C>0 c , C > 0 such that the following holds. Fix 0 < α < 1 0<\alpha<1 0 < α < 1 and 1 ≤ s ≤ n / 2 1\leq s\leq n/2 1 ≤ s ≤ n /2 , and suppose that the measurement matrix comes from one of the families in Definition 2.
Let x ⋆ x^\star x ⋆ be a fixed signal with support S S S , and let A \mathcal{A} A be its matrix event. If τ ≤ c α \tau\leq c\alpha τ ≤ c α and m m m satisfies (16) with ∣ S ∣ |S| ∣ S ∣ in place of k k k and with this constant C C C , then P ( A ∉ A ) ≤ α \mathbb{P}(\mathcal{A}\notin\mathcal{A})\leq\alpha P ( A ∈ / A ) ≤ α .
If X ⋆ ∼ BG ( s , n ) X^\star\sim\operatorname{BG}(s,n) X ⋆ ∼ BG ( s , n ) is independent of the measurement matrix, τ ≤ c α \tau\leq c\alpha τ ≤ c α , and m m m satisfies (20) with this constant C C C , then, with A \mathcal{A} A the matrix event of X ⋆ X^\star X ⋆ , P ( A ∉ A or ∥ X ⋆ ∥ 2 > n ) ≤ 2 α \mathbb{P}(\mathcal{A}\notin\mathcal{A}\text{ or }\|X^\star\|^2>n)\leq2\alpha P ( A ∈ / A or ∥ X ⋆ ∥ 2 > n ) ≤ 2 α .
In Theorems 1 and 2, the constants c c c are taken small enough and the constants C C C large enough for this proposition to apply. We first verify the matrix event under the following sufficient measurement conditions, expressed in terms of the cap q q q , the tolerance ε \varepsilon ε , and a failure probability 0 < α 0 < 1 0<\alpha_0<1 0 < α 0 < 1 :
m ≥ C ⋆ κ ⋆ 8 ε − 2 { 2 q log ( e n ) + log ( e n / α 0 ) } (independent sub-Gaussian entries), (87) m\geq C_\star\kappa_\star^8\varepsilon^{-2}\{2q\log(en)+\log(en/\alpha_0)\}\qquad\text{(independent sub-Gaussian entries),} \tag*{(87)} m ≥ C ⋆ κ ⋆ 8 ε − 2 { 2 q log ( e n ) + log ( e n / α 0 )} (independent sub-Gaussian entries), (87)
m ≥ 2 C ⋆ κ ⋆ 2 ε − 2 q { log 4 ( e n ) + log ( e / α 0 ) } (sampled bases), (88) m\geq2C_\star\kappa_\star^2\varepsilon^{-2}q\{\log^4(en)+\log(e/\alpha_0)\}\qquad\text{(sampled bases),} \tag*{(88)} m ≥ 2 C ⋆ κ ⋆ 2 ε − 2 q { log 4 ( e n ) + log ( e / α 0 )} (sampled bases), (88)
m ≥ 4 C ⋆ κ ⋆ 2 ε − 2 q 2 (deterministic frames). (89) m\geq4C_\star\kappa_\star^2\varepsilon^{-2}q^2\qquad\text{(deterministic frames).} \tag*{(89)} m ≥ 4 C ⋆ κ ⋆ 2 ε − 2 q 2 (deterministic frames). (89)
Here C ∗ ≥ 1 C_* \ge1 C ∗ ≥ 1 is a universal constant chosen large enough for all three families. We begin with sampled bases and frames, for which tightness gives B t = I + β t A ⊤ A B_t = I + \beta_t A^\top A B t = I + β t A ⊤ A , so restricted isometry and coherence bounds suffice. For independent entries, we prove one uniform resolvent estimate and use a push-through identity to obtain the block condition. Finally, we substitute the prescribed q q q and ε \varepsilon ε and pass from fixed to random signals to prove the two parts of Proposition 3.
The matrix quantities B t B_t B t , q q q and ε \varepsilon ε are defined in § 5.1 , and ε ≤ 1 / 2 \varepsilon\le1/2 ε ≤ 1/2 by (30) . For a fixed signal, the event A \mathcal{A} A depends only on the measurement matrix. Since ∥ diag R ∥ ≤ ∥ R ∥ \lVert\operatorname{diag} R\rVert\le\lVert R\rVert ∥ diag R ∥ ≤ ∥ R ∥ , every square matrix R R R satisfies
∥ R − diag R ∥ ≤ 2 ∥ R ∥ . (90) \lVert R - \operatorname{diag} R\rVert\le2\lVert R\rVert. \tag*{(90)} ∥ R − diag R ∥ ≤ 2 ∥ R ∥ . (90)
Sampled orthonormal bases and frames
Reduction to restricted isometry. Both families satisfy A A ⊤ = ( n / m ) I m AA^\top= (n/m)I_m A A ⊤ = ( n / m ) I m . We show that the matrix event holds whenever
∥ ( A ⊤ A − I ) J J ∥ ≤ ε / 3 for every J ⊆ [ n ] with 1 ≤ ∣ J ∣ ≤ 2 q . (91) \lVert(A^\top A - I)_{J J}\rVert\le\varepsilon/3 \qquad\text{for every } J \subseteq[n] \text{ with } 1 \le|J| \le2q. \tag*{(91)} ∥( A ⊤ A − I ) J J ∥ ≤ ε /3 for every J ⊆ [ n ] with 1 ≤ ∣ J ∣ ≤ 2 q . (91)
Indeed, ∥ A ∥ 2 = n / m ≤ 2 n \lVert A\rVert^2 = n/m \le2n ∥ A ∥ 2 = n / m ≤ 2 n , and the singletons give ∥ A i ∥ 2 ≥ 1 − ε / 3 ≥ 2 / 3 \lVert A_i\rVert^2 \ge1 - \varepsilon/3 \ge2/3 ∥ A i ∥ 2 ≥ 1 − ε /3 ≥ 2/3 . Since Σ t = ( a t + t n / m ) I m \Sigma_t = (a_t + tn/m)I_m Σ t = ( a t + t n / m ) I m ,
B t = I + β t A ⊤ A , β t : = { σ 2 + t ( 2 + n / m ) } − 1 . B_t = I + \beta_t A^\top A, \qquad\beta_t := \{\sigma^2 + t(2+n/m)\}^{-1}. B t = I + β t A ⊤ A , β t := { σ 2 + t ( 2 + n / m ) } − 1 .
The off-diagonal part of ( B t ) J J (B_t)_{J J} ( B t ) J J is β t \beta_t β t times that of ( A ⊤ A − I ) J J (A^\top A-I)_{J J} ( A ⊤ A − I ) J J , so by (90)
∥ ( B t − diag B t ) J J ∥ ≤ 2 ε β t 3 ≤ ε B t , i i ( i ∈ J ) , \lVert(B_t-\operatorname{diag} B_t)_{J J}\rVert\le\frac{2\varepsilon\beta_t}{3} \le\varepsilon B_{t,ii} \qquad(i\in J), ∥( B t − diag B t ) J J ∥ ≤ 3 2 ε β t ≤ ε B t , ii ( i ∈ J ) ,
since B t , i i = 1 + β t ∥ A i ∥ 2 ≥ 2 β t / 3 B_{t,ii}=1+\beta_t\lVert A_i\rVert^2\ge2\beta_t/3 B t , ii = 1 + β t ∥ A i ∥ 2 ≥ 2 β t /3 .
Sampled orthonormal bases. Let F F F be the matrix of the rows of U U U retained independently with probability m / n m/n m / n , scaled by n / m \sqrt{n/m} n / m . Given that the number of retained rows is m m m , F F F has the law of A A A . The restricted isometry theorem for subsampled bounded orthonormal systems, due to Candès and Tao [14 ] , Rudelson and Vershynin [48 ] , and Rauhut [46 ] , in the form (Dirksen [22 ] , Theorem 4.1, Eq. (18)), states that
( 1 − δ ) ∥ x ∥ 2 ≤ ∥ F x ∥ 2 ≤ ( 1 + δ ) ∥ x ∥ 2 (1-\delta)\lVert x\rVert^2 \le\lVert F x\rVert^2 \le(1+\delta)\lVert x\rVert^2 ( 1 − δ ) ∥ x ∥ 2 ≤ ∥ F x ∥ 2 ≤ ( 1 + δ ) ∥ x ∥ 2
for every 2 q 2q 2 q -sparse vector x x x fails with probability at most α 1 \alpha_1 α 1 if
m ≥ 2 C D i r κ b 2 δ − 2 q { log 2 ( 2 e q ) log ( e m ) log ( e n ) + log ( 1 / α 1 ) } , (92) m \ge2C_{\mathrm{Dir}}\kappa_b^2\delta^{-2}q\{\log^2(2eq)\log(em)\log(en)+\log(1/\alpha_1)\}, \tag*{(92)} m ≥ 2 C Dir κ b 2 δ − 2 q { log 2 ( 2 e q ) log ( e m ) log ( e n ) + log ( 1/ α 1 )} , (92)
where C D i r C_{\mathrm{Dir}} C Dir is the universal constant of Dirksen [22 ] , Theorem 4.1. The normalization and coordinate bound of that theorem match Definition 2. Replacing the maximum of its two terms by their sum and enlarging the logarithms only increases the sufficient bound; the factor log ( e m ) \log(em) log ( e m ) is removed below using m ≤ n m\le n m ≤ n .
To obtain the bound for a uniform subset, note that the retained cardinality is binomial with parameters ( n , m / n ) (n,m/n) ( n , m / n ) . Its mean m m m is an integer and hence a mode, of probability at least 1 / ( n + 1 ) 1/(n+1) 1/ ( n + 1 ) . Conditional on this cardinality, the retained set is a uniform m m m -subset with the same scaling. So with α 1 : = α 0 / ( n + 1 ) \alpha_1:=\alpha_0/(n+1) α 1 := α 0 / ( n + 1 ) the restricted isometry fails for the uniform subset with probability at most α 0 \alpha_0 α 0 .
To verify (92) , recall that κ b ≥ 1 \kappa_b\ge1 κ b ≥ 1 because the rows of U U U have unit norm. Together with C ∗ ≥ 1 C_*\ge1 C ∗ ≥ 1 and ε ≤ 1 \varepsilon\le1 ε ≤ 1 , condition (88) gives m ≥ 2 q m\ge2q m ≥ 2 q , so 2 q ≤ m ≤ n 2q\le m\le n 2 q ≤ m ≤ n . With also n + 1 ≤ 2 n n+1\le2n n + 1 ≤ 2 n , the bracket in (92) at α 1 = α 0 / ( n + 1 ) \alpha_1=\alpha_0/(n+1) α 1 = α 0 / ( n + 1 ) satisfies
log 2 ( 2 e q ) log ( e m ) log ( e n ) + log n + 1 α 0 ≤ log 4 ( e n ) + log ( 2 n ) + log 1 α 0 ≤ 3 { log 4 ( e n ) + log 1 α 0 } , \log^2(2eq)\log(em)\log(en)+\log\frac{n+1}{\alpha_0}
\le\log^4(en)+\log(2n)+\log\frac{1}{\alpha_0}
\le3\left\{\log^4(en)+\log\frac{1}{\alpha_0}\right\}, log 2 ( 2 e q ) log ( e m ) log ( e n ) + log α 0 n + 1 ≤ log 4 ( e n ) + log ( 2 n ) + log α 0 1 ≤ 3 { log 4 ( e n ) + log α 0 1 } ,
the second bound because log ( 2 n ) ≤ log ( e n ) ≤ log 4 ( e n ) \log(2n)\le\log(en)\le\log^4(en) log ( 2 n ) ≤ log ( e n ) ≤ log 4 ( e n ) . With δ = ε / 3 \delta=\varepsilon/3 δ = ε /3 and log ( 1 / α 0 ) ≤ log ( e / α 0 ) \log(1/\alpha_0)\le\log(e/\alpha_0) log ( 1/ α 0 ) ≤ log ( e / α 0 ) , condition (88) therefore implies (92) when C ∗ ≥ C b : = max { 1 , 27 C D i r } C_*\ge C_b:=\max\{1,27C_{\mathrm{Dir}}\} C ∗ ≥ C b := max { 1 , 27 C Dir } .
Applying the restricted isometry with δ = ε / 3 \delta=\varepsilon/3 δ = ε /3 to vectors supported on J J J gives (91) . Distinct sampled rows of U U U are orthonormal, so A A ⊤ = ( n / m ) I m AA^\top=(n/m)I_m A A ⊤ = ( n / m ) I m , and the preceding reduction gives A \mathcal{A} A with probability at least 1 − α 0 1-\alpha_0 1 − α 0 .
Deterministic frames. The symmetric matrix ( A ⊤ A − I ) J J (A^\top A-I)_{JJ} ( A ⊤ A − I ) J J has zero diagonal and off-diagonal entries of magnitude at most κ f / m \kappa_f/\sqrt{m} κ f / m . By Gershgorin’s bound and (89) with C ∗ ≥ 9 C_* \ge9 C ∗ ≥ 9 ,
∥ ( A ⊤ A − I ) J , J ∥ ≤ 2 q κ f m ≤ ε 3 ( ∣ J ∣ ≤ 2 q ) . \left\|(A^\top A-I)_{J,J}\right\| \le\frac{2q\kappa_f}{\sqrt{m}} \le\frac{\varepsilon}{3}\qquad(|J|\le2q). ( A ⊤ A − I ) J , J ≤ m 2 q κ f ≤ 3 ε ( ∣ J ∣ ≤ 2 q ) .
As A A ⊤ = ( n / m ) I m AA^\top=(n/m)I_m A A ⊤ = ( n / m ) I m , the reduction above gives A \mathcal{A} A .
Independent sub-Gaussian entries
Here A A ⊤ AA^\top A A ⊤ need not be a multiple of the identity. We instead condition on the columns outside a support J J J and prove that the inside columns remain nearly isotropic after applying the outside resolvent.
For J ⊆ [ n ] J\subseteq[n] J ⊆ [ n ] with 1 ≤ ∣ J ∣ ≤ 2 q 1\le|J|\le2q 1 ≤ ∣ J ∣ ≤ 2 q and 0 ≤ v ≤ 1 / 2 0\le v\le1/2 0 ≤ v ≤ 1/2 , set
Θ J : = A J c A J c ⊤ = ∑ i ∉ J A i A i ⊤ , γ J : = n − ∣ J ∣ m , \Theta_J:=A_{J^c}A_{J^c}^\top=\sum_{i\notin J}A_iA_i^\top,\qquad\gamma_J:=\frac{n-|J|}{m}, Θ J := A J c A J c ⊤ = i ∈ / J ∑ A i A i ⊤ , γ J := m n − ∣ J ∣ ,
M J ( v ) : = ( I m + v Θ J ) − 1 , ι J ( v ) : = m − 1 tr M J ( v ) , W J ( v ) : = A J ⊤ M J ( v ) A J . M_J(v):=(I_m+v\Theta_J)^{-1},\qquad\iota_J(v):=m^{-1}\operatorname{tr}M_J(v),\qquad W_J(v):=A_J^\top M_J(v)A_J. M J ( v ) := ( I m + v Θ J ) − 1 , ι J ( v ) := m − 1 tr M J ( v ) , W J ( v ) := A J ⊤ M J ( v ) A J .
Lemma 19 (Uniform resolvent concentration). *There is a universal constant C e ≥ 1 C_e\ge1 C e ≥ 1 such that, for independent sub-Gaussian entries, the measurement condition (87) with C ∗ ≥ C e C_*\ge C_e C ∗ ≥ C e implies that, with probability at least 1 − α 0 1-\alpha_0 1 − α 0 ,
∥ W J ( v ) − ι J ( v ) I J ∥ ≤ ε 4 ι J ( v ) for every J ⊆ [ n ] , 1 ≤ ∣ J ∣ ≤ 2 q , and 0 ≤ v ≤ 1 / 2. (93) \left\|W_J(v)-\iota_J(v)I_J\right\|\le\frac{\varepsilon}{4}\iota_J(v)\qquad\text{for every }J\subseteq[n],\ 1\le|J|\le2q,\ \text{and }0\le v\le1/2. \tag*{(93)} ∥ W J ( v ) − ι J ( v ) I J ∥ ≤ 4 ε ι J ( v ) for every J ⊆ [ n ] , 1 ≤ ∣ J ∣ ≤ 2 q , and 0 ≤ v ≤ 1/2. (93) The proof is deferred to Section D.2.1.
From the resolvent estimate to the matrix event. On the event of Lemma 19, the column and norm conditions of A A A come from (93) at v = 0 v=0 v = 0 , and the block condition from (93) at v = t / a t v=t/a_t v = t / a t .
At v = 0 v=0 v = 0 , W J ( 0 ) = A J ⊤ A J W_J(0)=A_J^\top A_J W J ( 0 ) = A J ⊤ A J and ι J ( 0 ) = 1 \iota_J(0)=1 ι J ( 0 ) = 1 , so the singletons give ∣ ∥ A i ∥ 2 − 1 ∣ ≤ ε / 4 \left|\|A_i\|^2-1\right|\le\varepsilon/4 ∥ A i ∥ 2 − 1 ≤ ε /4 . Hence
min i ∥ A i ∥ 2 ≥ 1 − ε / 4 ≥ 1 2 , ∥ A ∥ 2 ≤ ∥ A ∥ F 2 ≤ ( 1 + ε / 4 ) n ≤ 2 n . \min_i\|A_i\|^2\ge1-\varepsilon/4\ge\frac12,\qquad\|A\|^2\le\|A\|_F^2\le(1+\varepsilon/4)n\le2n. i min ∥ A i ∥ 2 ≥ 1 − ε /4 ≥ 2 1 , ∥ A ∥ 2 ≤ ∥ A ∥ F 2 ≤ ( 1 + ε /4 ) n ≤ 2 n .
Fix τ ≤ t ≤ T \tau\le t\le T τ ≤ t ≤ T and take v : = t / a t v:=t/a_t v := t / a t , so that 0 < v ≤ 1 / 2 0<v\le1/2 0 < v ≤ 1/2 and Σ t = a t ( I m + v A A ⊤ ) \Sigma_t=a_t(I_m+vAA^\top) Σ t = a t ( I m + v A A ⊤ ) . Fix J J J with 1 ≤ ∣ J ∣ ≤ 2 q 1\le|J|\le2q 1 ≤ ∣ J ∣ ≤ 2 q and write M M M , W W W and ι \iota ι for M J ( v ) M_J(v) M J ( v ) , W J ( v ) W_J(v) W J ( v ) and ι J ( v ) \iota_J(v) ι J ( v ) . Since I m + v A A ⊤ = M − 1 + v A J A J ⊤ I_m+vAA^\top=M^{-1}+vA_JA_J^\top I m + v A A ⊤ = M − 1 + v A J A J ⊤ ,
( I m + v A A ⊤ ) M A J = A J ( I J + v W ) , (I_m+vAA^\top)MA_J=A_J(I_J+vW), ( I m + v A A ⊤ ) M A J = A J ( I J + v W ) ,
so ( I m + v A A ⊤ ) − 1 A J = M A J ( I J + v W ) − 1 (I_m+vAA^\top)^{-1}A_J=MA_J(I_J+vW)^{-1} ( I m + v A A ⊤ ) − 1 A J = M A J ( I J + v W ) − 1 and
( B t ) J J = I J + a t − 1 A J ⊤ ( I m + v A A ⊤ ) − 1 A J = I J + a t − 1 f ( W ) , f ( w ) : = w 1 + v w . (B_t)_{JJ}=I_J+a_t^{-1}A_J^\top(I_m+vAA^\top)^{-1}A_J=I_J+a_t^{-1}f(W),\qquad f(w):=\frac{w}{1+vw}. ( B t ) J J = I J + a t − 1 A J ⊤ ( I m + v A A ⊤ ) − 1 A J = I J + a t − 1 f ( W ) , f ( w ) := 1 + v w w .
On [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) the function f f f is increasing with f ′ ≤ 1 f'\le1 f ′ ≤ 1 , and W ⪰ 0 W\succeq0 W ⪰ 0 , so by (93)
∥ f ( W ) − f ( ι ) I J ∥ ≤ ∥ W − ι I J ∥ ≤ ε ι 4 . \|f(W)-f(\iota)I_J\|\le\|W-\iota I_J\|\le\frac{\varepsilon\iota}{4}. ∥ f ( W ) − f ( ι ) I J ∥ ≤ ∥ W − ι I J ∥ ≤ 4 ε ι .
Since M ⪯ I m M\preceq I_m M ⪯ I m , we have ι ≤ 1 \iota\le1 ι ≤ 1 , and hence f ( ι ) ≥ 2 ι / 3 f(\iota)\ge2\iota/3 f ( ι ) ≥ 2 ι /3 . The off-diagonal part of ( B t ) J J (B_t)_{JJ} ( B t ) J J is a t − 1 a_t^{-1} a t − 1 times that of f ( W ) − f ( ι ) I J f(W)-f(\iota)I_J f ( W ) − f ( ι ) I J , so
∥ ( B t − diag B t ) J J ∥ ≤ ε ι 2 a t , ( B t ) i i ≥ f ( ι ) − ε ι / 4 a t ≥ ι 2 a t ( i ∈ J ) , \|(B_t-\operatorname{diag}B_t)_{JJ}\|\le\frac{\varepsilon\iota}{2a_t},\qquad(B_t)_{ii}\ge\frac{f(\iota)-\varepsilon\iota/4}{a_t}\ge\frac{\iota}{2a_t}\qquad(i\in J), ∥ ( B t − diag B t ) J J ∥ ≤ 2 a t ε ι , ( B t ) ii ≥ a t f ( ι ) − ε ι /4 ≥ 2 a t ι ( i ∈ J ) ,
where the first bound is (90) , and the second drops the 1 1 1 and uses ε ≤ 1 / 2 \varepsilon\le1/2 ε ≤ 1/2 . Hence
∥ ( B t − diag B t ) J J ∥ ≤ ε min i ∈ J B t , i i ( 1 ≤ ∣ J ∣ ≤ 2 q ) . \|(B_t-\operatorname{diag}B_t)_{JJ}\|\le\varepsilon\min_{i\in J}B_{t,ii}\qquad(1\le|J|\le2q). ∥ ( B t − diag B t ) J J ∥ ≤ ε i ∈ J min B t , ii ( 1 ≤ ∣ J ∣ ≤ 2 q ) .
Thus (87) with C ∗ ≥ C e C_*\ge C_e C ∗ ≥ C e gives P ( A ∉ A ) ≤ α 0 \mathbb{P}(A\notin\mathcal{A})\le\alpha_0 P ( A ∈ / A ) ≤ α 0 .
Proof of Lemma 19
Proof. We control the outside spectrum and the inside columns on its partial spectral projections. A nonnegative expansion in those projections then gives uniformity in v v v without discretizing the interval.
Let Y Y Y be a random vector with independent centered coordinates of ψ 2 \psi_2 ψ 2 norm at most κ \kappa κ , and let H H H be a deterministic real symmetric matrix with squared Frobenius norm ∥ H ∥ F 2 : = tr ( H ⊤ H ) \lVert H\rVert_F^2 := \operatorname{tr}(H^\top H) ∥ H ∥ F 2 := tr ( H ⊤ H ) . By the Hanson–Wright inequality [49 ] , in the form [58 Theorem 6.2.1] for the ψ 2 \psi_2 ψ 2 norm of Definition 2, there is a universal c H W > 0 c_{\mathrm{HW}}>0 c HW > 0 such that, for every z ≥ 0 z \ge0 z ≥ 0 ,
P { ∣ Y ⊤ H Y − E Y ⊤ H Y ∣ > z } ≤ 2 exp [ − c H W min { z 2 κ 4 ∥ H ∥ F 2 , z κ 2 ∥ H ∥ } ] . (94) \mathbb{P}\left\{\left|Y^\top H Y-\mathbb{E}Y^\top H Y\right|>z\right\}\le2\exp\left[-c_{\mathrm{HW}}\min\left\{\frac{z^2}{\kappa^4\lVert H\rVert_F^2},\frac{z}{\kappa^2\lVert H\rVert}\right\}\right]. \tag*{(94)} P { Y ⊤ H Y − E Y ⊤ H Y > z } ≤ 2 exp [ − c HW min { κ 4 ∥ H ∥ F 2 z 2 , κ 2 ∥ H ∥ z } ] . (94)
The unit sphere of R d \mathbb{R}^d R d has a 1 / 4 1/4 1/4 -net X \mathcal{X} X with ∣ X ∣ ≤ 9 d \lvert\mathcal{X}\rvert\le9^d ∣ X ∣ ≤ 9 d [58 Corollary 4.2.13] , and every symmetric H H H on R d \mathbb{R}^d R d satisfies [58 Exercise 4.4.3(b]
∥ H ∥ ≤ 2 max x ∈ X ∣ x ⊤ H x ∣ . (95) \lVert H\rVert\le2\max_{x\in\mathcal{X}}\lvert x^\top Hx\rvert. \tag*{(95)} ∥ H ∥ ≤ 2 x ∈ X max ∣ x ⊤ H x ∣ . (95)
The pairs of a set J ⊆ [ n ] J\subseteq[n] J ⊆ [ n ] with ∣ J ∣ ≤ 2 q \lvert J\rvert\le2q ∣ J ∣ ≤ 2 q and a point of a 1 / 4 1/4 1/4 -net of the sphere of R J \mathbb{R}^J R J number at most
∑ j ≤ 2 q ( n j ) 9 j ≤ ∑ j ≤ 2 q ( 9 n ) j ≤ ( 9 e n ) 2 q . (96) \sum_{j\le2q}\binom{n}{j}9^j\le\sum_{j\le2q}(9n)^j\le(9en)^{2q}. \tag*{(96)} j ≤ 2 q ∑ ( j n ) 9 j ≤ j ≤ 2 q ∑ ( 9 n ) j ≤ ( 9 e n ) 2 q . (96)
where the first bound is ( n j ) ≤ n j \binom{n}{j}\le n^j ( j n ) ≤ n j and the second bounds the geometric sum by 9 8 ( 9 n ) 2 q \frac{9}{8}(9n)^{2q} 8 9 ( 9 n ) 2 q .
Set
K 1 : = max { 1 , ( 3 + log 9 ) / c H W } , θ : = 3 c H W { 2 q log ( 9 e n ) + log ( 4 n / α 0 ) } . K_1:=\max\{1,(3+\log9)/c_{\mathrm{HW}}\},\qquad\theta:=\frac{3}{c_{\mathrm{HW}}}\{2q\log(9en)+\log(4n/\alpha_0)\}. K 1 := max { 1 , ( 3 + log 9 ) / c HW } , θ := c HW 3 { 2 q log ( 9 e n ) + log ( 4 n / α 0 )} .
Step 1: control of the outside spectrum. Fix J J J with 1 ≤ ∣ J ∣ ≤ 2 q 1\le\lvert J\rvert\le2q 1 ≤ ∣ J ∣ ≤ 2 q . We show that, except on an event of probability at most 2 e − 3 m 2e^{-3m} 2 e − 3 m ,
∥ Θ J − γ J I m ∥ ≤ γ J / 2 + 2 K 1 κ 4 , (97) \lVert\Theta_J-\gamma_J I_m\rVert\le\gamma_J/2+2K_1\kappa^4, \tag*{(97)} ∥ Θ J − γ J I m ∥ ≤ γ J /2 + 2 K 1 κ 4 , (97)
and consequently
∥ M J ( v ) ∥ ≤ 7 K 1 κ 4 ι J ( v ) ( 0 ≤ v ≤ 1 / 2 ) . (98) \lVert M_J(v)\rVert\le7K_1\kappa^4\iota_J(v)\qquad(0\le v\le1/2). \tag*{(98)} ∥ M J ( v )∥ ≤ 7 K 1 κ 4 ι J ( v ) ( 0 ≤ v ≤ 1/2 ) . (98)
If ∣ J ∣ = n \lvert J\rvert=n ∣ J ∣ = n , then Θ J = 0 \Theta_J=0 Θ J = 0 , γ J = 0 \gamma_J=0 γ J = 0 and M J ( v ) = I m M_J(v)=I_m M J ( v ) = I m , so both bounds hold on the whole space. For the remainder of this step, assume ∣ J ∣ < n \lvert J\rvert<n ∣ J ∣ < n .
Fix a unit w ∈ R m w\in\mathbb{R}^m w ∈ R m . Then w ⊤ Θ J w = m − 1 ∑ i ∉ J ( w ⊤ m A i ) 2 w^\top\Theta_Jw=m^{-1}\sum_{i\notin J}(w^\top\sqrt{m}A_i)^2 w ⊤ Θ J w = m − 1 ∑ i ∈ / J ( w ⊤ m A i ) 2 is the quadratic form Y ⊤ H Y Y^\top HY Y ⊤ H Y of the concatenated entries Y Y Y of m A J c \sqrt{m}A_{J^c} m A J c , with H H H the block diagonal matrix of n − ∣ J ∣ n-\lvert J\rvert n − ∣ J ∣ copies of w w ⊤ / m ww^\top/m w w ⊤ / m . Thus
E Y ⊤ H Y = γ J , ∥ H ∥ = 1 m , ∥ H ∥ F 2 = γ J m . \mathbb{E}Y^\top HY=\gamma_J,\qquad\lVert H\rVert=\frac{1}{m},\qquad\lVert H\rVert_F^2=\frac{\gamma_J}{m}. E Y ⊤ H Y = γ J , ∥ H ∥ = m 1 , ∥ H ∥ F 2 = m γ J .
Set z : = γ J / 4 + K 1 κ 4 z:=\gamma_J/4+K_1\kappa^4 z := γ J /4 + K 1 κ 4 . Then
z 2 κ 4 ∥ H ∥ F 2 ≥ K 1 κ 4 γ J m κ 4 γ J = K 1 m , z κ 2 ∥ H ∥ ≥ K 1 κ 2 m ≥ K 1 m , \frac{z^2}{\kappa^4\lVert H\rVert_F^2}\ge\frac{K_1\kappa^4\gamma_J m}{\kappa^4\gamma_J}=K_1m,\qquad\frac{z}{\kappa^2\lVert H\rVert}\ge K_1\kappa^2m\ge K_1m, κ 4 ∥ H ∥ F 2 z 2 ≥ κ 4 γ J K 1 κ 4 γ J m = K 1 m , κ 2 ∥ H ∥ z ≥ K 1 κ 2 m ≥ K 1 m ,
where the first bound is ( a + b ) 2 ≥ 4 a b (a+b)^2\ge4ab ( a + b ) 2 ≥ 4 ab and the last uses κ ≥ 1 \kappa\ge1 κ ≥ 1 , which holds by Definition 2. By (94) , ∣ w ⊤ Θ J w − γ J ∣ ≤ z \lvert w^\top\Theta_Jw-\gamma_J\rvert\le z ∣ w ⊤ Θ J w − γ J ∣ ≤ z fails with probability at most 2 e − c H W K 1 m 2e^{-c_{\mathrm{HW}}K_1m} 2 e − c HW K 1 m .
Apply (95) to Θ J − γ J I m \Theta_J-\gamma_JI_m Θ J − γ J I m over a 1 / 4 1/4 1/4 -net of the sphere of R m \mathbb{R}^m R m . Where the preceding bound holds at every point of the net, ∥ Θ J − γ J I m ∥ ≤ 2 z \lVert\Theta_J-\gamma_JI_m\rVert\le2z ∥ Θ J − γ J I m ∥ ≤ 2 z , which is (97) . Since c H W K 1 ≥ 3 + log 9 c_{\mathrm{HW}}K_1\ge3+\log9 c HW K 1 ≥ 3 + log 9 , this event fails with probability at most
2 ⋅ 9 m e − c H W K 1 m ≤ 2 e − 3 m . 2\cdot9^m e^{-c_{\mathrm{HW}}K_1m}\le2e^{-3m}. 2 ⋅ 9 m e − c HW K 1 m ≤ 2 e − 3 m .
The eigenvalues of M J M_J M J are the numbers ( 1 + v λ ) − 1 (1+v\lambda)^{-1} ( 1 + v λ ) − 1 over the eigenvalues λ \lambda λ of Θ J \Theta_J Θ J . By (97) , γ J / 2 ≤ λ min + 2 K 1 κ 4 \gamma_J/2\le\lambda_{\min}+2K_1\kappa^4 γ J /2 ≤ λ m i n + 2 K 1 κ 4 and λ max ≤ 3 γ J / 2 + 2 K 1 κ 4 \lambda_{\max}\le3\gamma_J/2+2K_1\kappa^4 λ m a x ≤ 3 γ J /2 + 2 K 1 κ 4 , so λ max ≤ 3 λ min + 8 K 1 κ 4 \lambda_{\max}\le3\lambda_{\min}+8K_1\kappa^4 λ m a x ≤ 3 λ m i n + 8 K 1 κ 4 . Hence
∥ M J ∥ ι J ≤ 1 + v λ max 1 + v λ min ≤ 1 + 3 v λ min + 8 v K 1 κ 4 1 + v λ min ≤ 3 + 8 v K 1 κ 4 ≤ 7 K 1 κ 4 , \frac{\lVert M_J\rVert}{\iota_J}\le\frac{1+v\lambda_{\max}}{1+v\lambda_{\min}}\le\frac{1+3v\lambda_{\min}+8vK_1\kappa^4}{1+v\lambda_{\min}}\le3+8vK_1\kappa^4\le7K_1\kappa^4, ι J ∥ M J ∥ ≤ 1 + v λ m i n 1 + v λ m a x ≤ 1 + v λ m i n 1 + 3 v λ m i n + 8 v K 1 κ 4 ≤ 3 + 8 v K 1 κ 4 ≤ 7 K 1 κ 4 ,
where the first bound uses ∥ M J ∥ = ( 1 + v λ min ) − 1 \lVert M_J\rVert=(1+v\lambda_{\min})^{-1} ∥ M J ∥ = ( 1 + v λ m i n ) − 1 and ι J ≥ ( 1 + v λ max ) − 1 \iota_J\ge(1+v\lambda_{\max})^{-1} ι J ≥ ( 1 + v λ m a x ) − 1 , and the last uses v ≤ 1 / 2 v\le1/2 v ≤ 1/2 and K 1 κ 4 ≥ 1 K_1\kappa^4\ge1 K 1 κ 4 ≥ 1 . This is (98) .
Step 2: concentration on the partial spectral projections. Let 0 ≤ λ 1 < ⋯ < λ l 0 \le\lambda_1 < \cdots< \lambda_l 0 ≤ λ 1 < ⋯ < λ l be the distinct eigenvalues of Θ J \Theta_J Θ J and E 1 , … , E l E_1,\ldots,E_l E 1 , … , E l their eigenprojections, and set
E ‾ j : = { E 1 + ⋯ + E j , 1 ≤ j ≤ l , I m , l < j ≤ m , r j : = tr E ‾ j , \overline{E}_j :=
\begin{cases}
E_1+\cdots+E_j, & 1 \le j \le l,\\
I_m, & l<j\le m,
\end{cases}
\qquad r_j:=\operatorname{tr}\overline{E}_j, E j := { E 1 + ⋯ + E j , I m , 1 ≤ j ≤ l , l < j ≤ m , r j := tr E j ,
so that r j ≥ 1 r_j \ge1 r j ≥ 1 is the rank of E ‾ j \overline{E}_j E j . Since l ≤ m l \le m l ≤ m , we have E ‾ l = I m \overline{E}_l=I_m E l = I m , and the index set { 1 , … , m } \{1,\ldots,m\} { 1 , … , m } of the partial sums does not depend on the draw.
Given the columns outside J J J , these projections are fixed and independent of A J A_J A J . We show that
∥ A J T E ‾ j A J − r j m I J ∥ ≤ 2 κ 2 r j θ + θ m ( 1 ≤ j ≤ m ) (99) \left\|A_J^{\mathsf T}\overline{E}_jA_J-\frac{r_j}{m}I_J\right\|\le2\kappa^2\frac{\sqrt{r_j\theta}+\theta}{m}\qquad(1\le j\le m) \tag*{(99)} A J T E j A J − m r j I J ≤ 2 κ 2 m r j θ + θ ( 1 ≤ j ≤ m ) (99)
fails with conditional probability at most 2 m 9 ∣ J ∣ e − c H W θ 2m\,9^{|J|}e^{-c_{\mathrm{HW}}\theta} 2 m 9 ∣ J ∣ e − c HW θ .
Condition on the columns outside J J J . For a unit x ∈ R J x\in\mathbb{R}^J x ∈ R J , x T A J T E ‾ j A J x x^{\mathsf T}A_J^{\mathsf T}\overline{E}_jA_Jx x T A J T E j A J x is the quadratic form Y T H Y Y^{\mathsf T}HY Y T H Y of the concatenated entries Y Y Y of m A J \sqrt{m}A_J m A J , which are independent of the outside columns, with H : = m − 1 x x T ⊗ E ‾ j H:=m^{-1}xx^{\mathsf T}\otimes\overline{E}_j H := m − 1 x x T ⊗ E j . Thus
E Y T H Y = r j m , ∥ H ∥ = 1 m , ∥ H ∥ F 2 = r j m 2 . \mathbb{E}Y^{\mathsf T}HY=\frac{r_j}{m},\qquad\|H\|=\frac{1}{m},\qquad\|H\|_F^2=\frac{r_j}{m^2}. E Y T H Y = m r j , ∥ H ∥ = m 1 , ∥ H ∥ F 2 = m 2 r j .
Set z j : = κ 2 ( r j θ + θ ) / m z_j:=\kappa^2(\sqrt{r_j\theta}+\theta)/m z j := κ 2 ( r j θ + θ ) / m . Then
z j 2 κ 4 ∥ H ∥ F 2 = ( r j θ + θ ) 2 r j ≥ θ , z j κ 2 ∥ H ∥ = r j θ + θ ≥ θ , \frac{z_j^2}{\kappa^4\|H\|_F^2}=\frac{(\sqrt{r_j\theta}+\theta)^2}{r_j}\ge\theta,\qquad\frac{z_j}{\kappa^2\|H\|}=\sqrt{r_j\theta}+\theta\ge\theta, κ 4 ∥ H ∥ F 2 z j 2 = r j ( r j θ + θ ) 2 ≥ θ , κ 2 ∥ H ∥ z j = r j θ + θ ≥ θ ,
so by (94) the bound ∣ x T A J T E ‾ j A J x − r j / m ∣ ≤ z j \lvert x^{\mathsf T}A_J^{\mathsf T}\overline{E}_jA_Jx-r_j/m\rvert\le z_j ∣ x T A J T E j A J x − r j / m ∣ ≤ z j fails with conditional probability at most 2 e − c H W θ 2e^{-c_{\mathrm{HW}}\theta} 2 e − c HW θ .
By (95) over a 1 / 4 1/4 1/4 -net of the sphere of R J \mathbb{R}^J R J , which has at most 9 ∣ J ∣ 9^{|J|} 9 ∣ J ∣ points, and a union over 1 ≤ j ≤ m 1\le j\le m 1 ≤ j ≤ m , the bound (99) , whose right side is 2 z j 2z_j 2 z j , fails with conditional probability at most 2 m 9 ∣ J ∣ e − c H W θ 2m\,9^{|J|}e^{-c_{\mathrm{HW}}\theta} 2 m 9 ∣ J ∣ e − c HW θ . This bound does not depend on the outside columns, so by Fubini’s theorem it holds unconditionally.
Step 3: uniformity along the resolvent path. Suppose that (98) and (99) hold for this J J J , and that m ≥ K 2 κ 8 ε − 2 θ m\ge K_2\kappa^8\varepsilon^{-2}\theta m ≥ K 2 κ 8 ε − 2 θ , where K 2 ≥ 1 K_2\ge1 K 2 ≥ 1 is a universal constant to be chosen below. The same projection event controls every v v v , because M J ( v ) M_J(v) M J ( v ) is a nonnegative combination of the partial spectral projections.
Fix v v v and write M J = ∑ j ≤ l f j E j M_J=\sum_{j\le l}f_jE_j M J = ∑ j ≤ l f j E j with f j : = ( 1 + v λ j ) − 1 f_j:=(1+v\lambda_j)^{-1} f j := ( 1 + v λ j ) − 1 , which is nonincreasing in j j j . Summation by parts gives
M J = ∑ j ≤ l ϖ j E ‾ j , ϖ l : = f l , ϖ j : = f j − f j + 1 ( j < l ) , M_J=\sum_{j\le l}\varpi_j\overline{E}_j,\qquad\varpi_l:=f_l,\qquad\varpi_j:=f_j-f_{j+1}\ (j<l), M J = j ≤ l ∑ ϖ j E j , ϖ l := f l , ϖ j := f j − f j + 1 ( j < l ) ,
and the weights satisfy
ϖ j ≥ 0 , ∑ j ϖ j = f 1 = ∥ M J ∥ , ∑ j ϖ j r j = tr M J = m ι J , ∑ j ϖ j r j ≤ ∥ M J ∥ m ι J , \varpi_j\ge0,\qquad\sum_j\varpi_j=f_1=\|M_J\|,\qquad\sum_j\varpi_jr_j=\operatorname{tr}M_J=m\iota_J,\qquad\sum_j\varpi_j\sqrt{r_j}\le\sqrt{\|M_J\|m\iota_J}, ϖ j ≥ 0 , j ∑ ϖ j = f 1 = ∥ M J ∥ , j ∑ ϖ j r j = tr M J = m ι J , j ∑ ϖ j r j ≤ ∥ M J ∥ m ι J ,
the last by Cauchy–Schwarz.
Since W J = ∑ j ϖ j A J T E ‾ j A J W_J=\sum_j\varpi_jA_J^{\mathsf T}\overline{E}_jA_J W J = ∑ j ϖ j A J T E j A J and ι J = ∑ j ϖ j r j / m \iota_J=\sum_j\varpi_jr_j/m ι J = ∑ j ϖ j r j / m ,
∥ W J − ι J I J ∥ ≤ ∑ j ϖ j ∥ A J T E ‾ j A J − r j m I J ∥ ≤ 2 κ 2 ∑ j ϖ j r j θ + θ m ≤ 2 κ 2 { θ ∥ M J ∥ ι J m + θ m ∥ M J ∥ } , \|W_J-\iota_JI_J\|\le\sum_j\varpi_j\left\|A_J^{\mathsf T}\overline{E}_jA_J-\frac{r_j}{m}I_J\right\|\le2\kappa^2\sum_j\varpi_j\frac{\sqrt{r_j\theta}+\theta}{m}\le2\kappa^2\left\{\sqrt{\frac{\theta\|M_J\|\iota_J}{m}}+\frac{\theta}{m}\|M_J\|\right\}, ∥ W J − ι J I J ∥ ≤ j ∑ ϖ j A J T E j A J − m r j I J ≤ 2 κ 2 j ∑ ϖ j m r j θ + θ ≤ 2 κ 2 { m θ ∥ M J ∥ ι J + m θ ∥ M J ∥ } ,
by the triangle inequality, (99) and the properties of the weights.
Dividing by ι J \iota_J ι J and applying (98) ,
∥ W J − ι J I J ∥ ι J ≤ 2 κ 2 { 7 K 1 κ 4 θ m + 7 K 1 κ 4 θ m } ≤ 2 ε { 7 K 1 K 2 + 7 K 1 K 2 } ≤ ε 4 , \frac{\|W_J-\iota_JI_J\|}{\iota_J}\le2\kappa^2\left\{\sqrt{\frac{7K_1\kappa^4\theta}{m}}+\frac{7K_1\kappa^4\theta}{m}\right\}\le2\varepsilon\left\{\sqrt{\frac{7K_1}{K_2}}+\frac{7K_1}{K_2}\right\}\le\frac{\varepsilon}{4}, ι J ∥ W J − ι J I J ∥ ≤ 2 κ 2 { m 7 K 1 κ 4 θ + m 7 K 1 κ 4 θ } ≤ 2 ε { K 2 7 K 1 + K 2 7 K 1 } ≤ 4 ε ,
where the second bound uses κ 4 θ / m ≤ ε / K 2 \kappa^4\sqrt{\theta/m}\le\varepsilon/\sqrt{K_2} κ 4 θ / m ≤ ε / K 2 and κ 6 θ / m ≤ ε / K 2 \kappa^6\theta/m\le\varepsilon/K_2 κ 6 θ / m ≤ ε / K 2 , which follow from the assumed lower bound on m m m since κ ≥ 1 \kappa\ge1 κ ≥ 1 and ε ≤ 1 \varepsilon\le1 ε ≤ 1 , and the last bound holds for K 2 K_2 K 2 large enough in terms of K 1 K_1 K 1 .
Step 4: simultaneous control over all supports. Set C e : = max { 4 , 12 K 2 / c H W } C_{\mathrm{e}} := \max\{4,12K_{2}/c_{\mathrm{HW}}\} C e := max { 4 , 12 K 2 / c HW } , and let C ∗ ≥ C e C_{\ast} \ge C_{\mathrm{e}} C ∗ ≥ C e . The inequalities log ( 9 e n ) ≤ 4 log ( e n ) \log(9en) \le4\log(en) log ( 9 e n ) ≤ 4 log ( e n ) and log ( 4 / α 0 ) ≤ log ( 4 n / α 0 ) ≤ 2 log ( e n / α 0 ) \log(4/\alpha_{0}) \le\log(4n/\alpha_{0}) \le2\log(en/\alpha_{0}) log ( 4/ α 0 ) ≤ log ( 4 n / α 0 ) ≤ 2 log ( e n / α 0 ) give
θ ≤ 12 c H W { 2 q log ( e n ) + log ( e n / α 0 ) } . \theta\le\frac{12}{c_{\mathrm{HW}}}\{2q\log(en)+\log(en/\alpha_{0})\}. θ ≤ c HW 12 { 2 q log ( e n ) + log ( e n / α 0 )} .
and with these inequalities, κ ≥ 1 \kappa\ge1 κ ≥ 1 , ε ≤ 1 \varepsilon\le1 ε ≤ 1 and C ∗ ≥ C e C_{\ast} \ge C_{\mathrm{e}} C ∗ ≥ C e , the condition (87) gives
m ≥ K 2 κ 8 ε − 2 θ , m ≥ 2 q log ( 9 e n ) + log ( 4 / α 0 ) . m \ge K_{2}\kappa^{8}\varepsilon^{-2}\theta,\qquad m \ge2q\log(9en)+\log(4/\alpha_{0}). m ≥ K 2 κ 8 ε − 2 θ , m ≥ 2 q log ( 9 e n ) + log ( 4/ α 0 ) .
By (96) and the definition of θ \theta θ , the failures of (99) over all J J J total at most
2 m exp { 2 q log ( 9 e n ) − c H W θ } = 2 m exp { − 4 q log ( 9 e n ) − 3 log ( 4 n / α 0 ) } ≤ 2 m ( α 0 / 4 n ) 3 ≤ α 0 / 2 , 2m\exp\{2q\log(9en)-c_{\mathrm{HW}}\theta\}=2m\exp\{-4q\log(9en)-3\log(4n/\alpha_{0})\}\le2m(\alpha_{0}/4n)^{3}\le\alpha_{0}/2, 2 m exp { 2 q log ( 9 e n ) − c HW θ } = 2 m exp { − 4 q log ( 9 e n ) − 3 log ( 4 n / α 0 )} ≤ 2 m ( α 0 /4 n ) 3 ≤ α 0 /2 ,
using m ≤ n m\le n m ≤ n and α 0 ≤ 1 \alpha_{0}\le1 α 0 ≤ 1 in the last step. The number of sets J J J is also at most (96) , so by the second bound on m m m the failures of (97) total at most
2 exp { 2 q log ( 9 e n ) − 3 m } ≤ 2 exp { − 4 q log ( 9 e n ) − 3 log ( 4 / α 0 ) } ≤ 2 ( α 0 / 4 ) 3 ≤ α 0 / 2. 2\exp\{2q\log(9en)-3m\}\le2\exp\{-4q\log(9en)-3\log(4/\alpha_{0})\}\le2(\alpha_{0}/4)^{3}\le\alpha_{0}/2. 2 exp { 2 q log ( 9 e n ) − 3 m } ≤ 2 exp { − 4 q log ( 9 e n ) − 3 log ( 4/ α 0 )} ≤ 2 ( α 0 /4 ) 3 ≤ α 0 /2.
Outside these failures, which total at most α 0 \alpha_{0} α 0 , Step 3 gives (93) for every J J J and every 0 ≤ v ≤ 1 / 2 0\le v\le1/2 0 ≤ v ≤ 1/2 . □
Proof of Proposition 3
Choose C ∗ : = max { 9 , C e , C b } C_{\ast}:=\max\{9,C_{\mathrm{e}},C_{\mathrm{b}}\} C ∗ := max { 9 , C e , C b } . The preceding arguments show that each measurement condition implies P ( A ∉ A ) ≤ α 0 \mathbb{P}(A\notin\mathcal{A})\le\alpha_{0} P ( A ∈ / A ) ≤ α 0 for its family. It remains to derive those conditions from the sample bounds in Proposition 3.
Step 1: fixed signals. Take c ≤ 1 c\le1 c ≤ 1 , so that τ ≤ c α \tau\le c\alpha τ ≤ c α implies τ ≤ α < 1 \tau\le\alpha<1 τ ≤ α < 1 . We apply the measurement conditions with α 0 = α \alpha_{0}=\alpha α 0 = α .
Since s ≥ 1 ≥ τ s\ge1\ge\tau s ≥ 1 ≥ τ , Q = n / τ + n / s ≤ 2 n / τ Q=n/\tau+n/s\le2n/\tau Q = n / τ + n / s ≤ 2 n / τ , and as 4 n / τ ≤ ( e n / τ ) 2 4n/\tau\le(en/\tau)^{2} 4 n / τ ≤ ( e n / τ ) 2 ,
log ( 2 Q ) ≤ log ( 4 n / τ ) ≤ 2 log ( e n / τ ) . \log(2Q)\le\log(4n/\tau)\le2\log(en/\tau). log ( 2 Q ) ≤ log ( 4 n / τ ) ≤ 2 log ( e n / τ ) .
Set k 0 : = 1 + s + ∣ S ∣ + ∥ x ⋆ ∥ 2 ≥ 1 k_{0}:=1+s+|S|+\lVert x^{\star}\rVert^{2}\ge1 k 0 := 1 + s + ∣ S ∣ + ∥ x ⋆ ∥ 2 ≥ 1 . Since q ≤ 14 k 0 log ( 2 Q ) + 1 q\le14k_{0}\log(2Q)+1 q ≤ 14 k 0 log ( 2 Q ) + 1 by the definition of the cap, log ( e n / τ ) ≥ 1 \log(en/\tau)\ge1 log ( e n / τ ) ≥ 1 and α ≥ τ \alpha\ge\tau α ≥ τ ,
q ≤ C k 0 log ( e n / τ ) , ε − 2 = log 2 ( 2 Q ) c ε 2 ≤ 4 log 2 ( e n / τ ) c ε 2 , log ( e n / α ) , log ( e / α ) ≤ log ( e n / τ ) . q\le Ck_{0}\log(en/\tau),\qquad\varepsilon^{-2}=\frac{\log^{2}(2Q)}{c_{\varepsilon}^{2}}\le\frac{4\log^{2}(en/\tau)}{c_{\varepsilon}^{2}},\qquad\log(en/\alpha),\ \log(e/\alpha)\le\log(en/\tau). q ≤ C k 0 log ( e n / τ ) , ε − 2 = c ε 2 log 2 ( 2 Q ) ≤ c ε 2 4 log 2 ( e n / τ ) , log ( e n / α ) , log ( e / α ) ≤ log ( e n / τ ) .
Substituting these bounds and using log ( e n ) ≥ 1 \log(en)\ge1 log ( e n ) ≥ 1 , k 0 ≥ 1 k_{0}\ge1 k 0 ≥ 1 and log 4 ( e n ) + log ( e n / τ ) ≤ 2 log ( e n / τ ) log 3 ( e n ) \log^{4}(en)+\log(en/\tau)\le2\log(en/\tau)\log^{3}(en) log 4 ( e n ) + log ( e n / τ ) ≤ 2 log ( e n / τ ) log 3 ( e n ) , which holds as τ ≤ 1 \tau\le1 τ ≤ 1 , the right sides of (87) , (88) and (89) are at most
C c ε − 2 C ∗ κ 8 k 0 log 3 ( e n / τ ) log ( e n ) , C c ε − 2 C ∗ κ b 2 k 0 log 4 ( e n / τ ) log 3 ( e n ) , C c ε − 2 C ∗ κ b 2 k 0 2 log 4 ( e n / τ ) Cc_{\varepsilon}^{-2}C_{\ast}\kappa^{8}k_{0}\log^{3}(en/\tau)\log(en),\qquad Cc_{\varepsilon}^{-2}C_{\ast}\kappa_{\mathrm{b}}^{2}k_{0}\log^{4}(en/\tau)\log^{3}(en),\qquad Cc_{\varepsilon}^{-2}C_{\ast}\kappa_{\mathrm{b}}^{2}k_{0}^{2}\log^{4}(en/\tau) C c ε − 2 C ∗ κ 8 k 0 log 3 ( e n / τ ) log ( e n ) , C c ε − 2 C ∗ κ b 2 k 0 log 4 ( e n / τ ) log 3 ( e n ) , C c ε − 2 C ∗ κ b 2 k 0 2 log 4 ( e n / τ )
respectively. Therefore, taking the universal constant C C C in (16) large enough in terms of C ∗ C_{\ast} C ∗ and c ε c_{\varepsilon} c ε ensures the measurement condition of the relevant family and hence P ( A ∉ A ) ≤ α \mathbb{P}(A\notin\mathcal{A})\le\alpha P ( A ∈ / A ) ≤ α .
Step 2: random signals. Item 2 gives P ( G c ) ≤ 2 α \mathbb{P}(\mathcal{G}^{c})\le2\alpha P ( G c ) ≤ 2 α for the event G \mathcal{G} G in (42) . It follows from item 1 by bounding the support size and the energy of the random signal.
Let C 2 C_{2} C 2 be a constant for which item 1 holds. Item 1 then holds with every larger constant, since a larger constant in (16) strengthens its hypothesis, and we take C ≥ max { 36 C 2 , 4 } C\ge\max\{36C_{2},4\} C ≥ max { 36 C 2 , 4 } in the proposition. Set k α : = 4 ( s + log ( 1 / α ) ) k_{\alpha}:=4(s+\log(1/\alpha)) k α := 4 ( s + log ( 1/ α )) . The proof applies item 1 to every signal with ∣ S ∣ + ∥ x ⋆ ∥ 2 ≤ k α |S|+\lVert x^{\star}\rVert^{2}\le k_{\alpha} ∣ S ∣ + ∥ x ⋆ ∥ 2 ≤ k α , and bounds the probability that the random signal is larger.
Let x ⋆ x^{\star} x ⋆ be a signal with support S S S , matrix event A A A and ∣ S ∣ + ∥ x ⋆ ∥ 2 ≤ k α |S|+\lVert x^{\star}\rVert^{2}\le k_{\alpha} ∣ S ∣ + ∥ x ⋆ ∥ 2 ≤ k α . Since s ≥ 1 s\ge1 s ≥ 1 , we have 1 + s + ∣ S ∣ + ∥ x ⋆ ∥ 2 ≤ 6 ( s + log ( 1 / α ) ) 1+s+|S|+\lVert x^{\star}\rVert^{2}\le6(s+\log(1/\alpha)) 1 + s + ∣ S ∣ + ∥ x ⋆ ∥ 2 ≤ 6 ( s + log ( 1/ α )) . The right-hand side of (16) is at most quadratic in 1 + s + k + ∥ x ⋆ ∥ 2 1+s+k+\lVert x^{\star}\rVert^{2} 1 + s + k + ∥ x ⋆ ∥ 2 , so (20) with the constant C C C implies (16) with ∣ S ∣ |S| ∣ S ∣ in place of k k k and with the constant C / 36 ≥ C 2 C/36\ge C_{2} C /36 ≥ C 2 . Item 1 then gives P ( A ∉ A ) ≤ α \mathbb{P}(A\notin\mathcal{A})\le\alpha P ( A ∈ / A ) ≤ α . Moreover ∥ x ⋆ ∥ 2 ≤ k α ≤ m ≤ n \lVert x^{\star}\rVert^{2}\le k_{\alpha}\le m\le n ∥ x ⋆ ∥ 2 ≤ k α ≤ m ≤ n , where k α ≤ m k_{\alpha}\le m k α ≤ m holds by (20) with C ≥ 4 C\ge4 C ≥ 4 , as the family parameters and the logarithms there are at least one. The coordinates of X ⋆ X^\star X ⋆ are independent, and ∣ supp X ⋆ ∣ + ∥ X ⋆ ∥ 2 = Y : = ∑ i 1 { X i ⋆ ≠ 0 } ( 1 + ( X i ⋆ ) 2 ) |\operatorname{supp} X^\star|+\lVert X^\star\rVert^2=Y:=\sum_i 1\{X_i^\star\ne0\}(1+(X_i^\star)^2) ∣ supp X ⋆ ∣ + ∥ X ⋆ ∥ 2 = Y := ∑ i 1 { X i ⋆ = 0 } ( 1 + ( X i ⋆ ) 2 ) . For g ∼ N ( 0 , 1 ) g\sim\mathcal{N}(0,1) g ∼ N ( 0 , 1 ) , e 1 / 4 E e g 2 / 4 = e 1 / 4 2 ≤ 2 e^{1/4}\mathbb{E}e^{g^2/4}=e^{1/4}\sqrt{2}\le2 e 1/4 E e g 2 /4 = e 1/4 2 ≤ 2 , so
E e Y / 4 = ( 1 − s n + s n e 1 / 4 E e g 2 / 4 ) n ≤ ( 1 + s n ) n ≤ e s . \mathbb{E}e^{Y/4}=\left(1-\frac{s}{n}+\frac{s}{n}e^{1/4}\mathbb{E}e^{g^2/4}\right)^n\le\left(1+\frac{s}{n}\right)^n\le e^s. E e Y /4 = ( 1 − n s + n s e 1/4 E e g 2 /4 ) n ≤ ( 1 + n s ) n ≤ e s .
Markov’s inequality applied to e Y / 4 e^{Y/4} e Y /4 gives P ( Y > k α ) ≤ e s − k α / 4 = α \mathbb{P}(Y>k_\alpha)\le e^{s-k_\alpha/4}=\alpha P ( Y > k α ) ≤ e s − k α /4 = α .
Let A \mathcal{A} A be the matrix event of X ⋆ X^\star X ⋆ . On the event Y ≤ k α Y\le k_\alpha Y ≤ k α we have ∥ X ⋆ ∥ 2 ≤ n \lVert X^\star\rVert^2\le n ∥ X ⋆ ∥ 2 ≤ n by the fixed-signal argument above, so there the event { A ∉ A or ∥ X ⋆ ∥ 2 > n } \{\mathcal{A}\notin\mathcal{A}\text{ or }\lVert X^\star\rVert^2>n\} { A ∈ / A or ∥ X ⋆ ∥ 2 > n } requires A ∉ A \mathcal{A}\notin\mathcal{A} A ∈ / A . The measurement matrix is independent of X ⋆ X^\star X ⋆ , so the preceding bounds give
P ( A ∉ A or ∥ X ⋆ ∥ 2 > n ) ≤ P ( Y > k α ) + E 1 { Y ≤ k α } P ( A ∉ A ∣ X ⋆ ) ≤ 2 α . \mathbb{P}\left(\mathcal{A}\notin\mathcal{A}\text{ or }\lVert X^\star\rVert^2>n\right)\le\mathbb{P}(Y>k_\alpha)+\mathbb{E}\,1\{Y\le k_\alpha\}\mathbb{P}(\mathcal{A}\notin\mathcal{A}\mid X^\star)\le2\alpha. P ( A ∈ / A or ∥ X ⋆ ∥ 2 > n ) ≤ P ( Y > k α ) + E 1 { Y ≤ k α } P ( A ∈ / A ∣ X ⋆ ) ≤ 2 α .