Introduction
Flow matching ([34 , 1 , 35 ] ) has established itself as a powerful and versatile framework for generative modeling, achieving compelling performance in image generation ([17 ] ), speech synthesis ([26 ] ), and even molecule generation ([54 ] ) and protein structure generation ([8 ] ). In a nutshell, flow matching seeks to transport a source distribution toward a target data distribution by means of an ordinary differential equation (ODE):
d d t X t = v t ( X t ) , t ∈ [ 0 , 1 ) , X 0 ∼ P b a s e , (1) \frac{\mathrm{d}}{\mathrm{d}t}X_t=v_t(X_t),\quad t\in[0,1),\quad X_0\sim P_{\mathrm{base}}, \tag*{(1)} d t d X t = v t ( X t ) , t ∈ [ 0 , 1 ) , X 0 ∼ P base , (1)
where P b a s e P_{\mathrm{base}} P base represents a source distribution on R d \mathbb{R}^{d} R d that is easy to sample from, and v t : R d → R d v_t:\mathbb{R}^{d}\to\mathbb{R}^{d} v t : R d → R d stands for a time-dependent vector field (often referred to as the velocity field). Ideally, the distribution of X t X_t X t converges to the target data distribution — denoted by P d a t a P_{\mathrm{data}} P data — as t t t approaches 1. Flow matching is typically composed of two stages: during training, the population vector field v t ( ⋅ ) v_t(\cdot) v t ( ⋅ ) is learned by regression using samples drawn from the source and target distributions; during sampling, one draws X 0 ∼ P b a s e X_0\sim P_{\mathrm{base}} X 0 ∼ P base and numerically solves the ODE ((1) ) from this initialization X 0 X_0 X 0 , using a standard approach such as Euler’s method, to obtain an approximate sample from P d a t a P_{\mathrm{data}} P data .
The popularity of flow matching stems in part from its simulation-free training, empirical effectiveness, conceptual simplicity, and flexibility in choosing the source distribution and, in some formulations, the source-target coupling [35 , 57 , 45 ] . Given the widespread adoption of flow matching in generative modeling, an important practical consideration is its sampling efficiency. As with ODE-based samplers for diffusion models, generating a sample with flow matching requires numerically solving an ODE driven by a learned velocity field, typically by discretizing time into a grid and evaluating the field at each step. A finer time grid can reduce discretization error and improve sampling accuracy, but it also requires more evaluations of the velocity field, each typically involving a forward pass through a large neural network. To understand this tradeoff and guide the choice of time grid in practice, the present paper seeks to characterize the iteration complexity of the flow matching sampler, namely, the number of steps needed to achieve a prescribed sampling accuracy.
Notably, while substantial work has been devoted to establishing convergence theory for diffusion models — whose probability flow ODEs can be formulated as flow matching with Gaussian sources [2 ] — the analyses and insights therein do not directly carry over to flow matching with non-Gaussian sources. This theoretical gap matters in practice because non-Gaussian sources can be highly versatile: they support sampling from heavy-tailed or multimodal distributions, data-to-data transport, and the generation of non-Euclidean data. This motivates a closer examination of the role of non-Gaussian sources and the theoretical support for them.
Non-Gaussian sources in flow matching
Conventionally, Gaussian sources are the standard choice in diffusion models: the forward process of a diffusion model gradually adds Gaussian noise until the data distribution approaches a Gaussian source, while preserving tractable forward transition distributions [22 , 52 , 53 ] . However, Gaussian sources may be less well suited to target distributions with heavy tails or to data with discrete or geometric structure. Alternative source distributions have therefore been explored, with tractability as a key consideration; examples include Gamma and α \alpha α -stable noise with suitable closure properties under addition [40 , 62 , 46 , 50 ] , and discrete corruption schemes that allow multi-step transition probabilities to be computed tractably [4 ] .
By contrast, flow matching offers greater flexibility in the choice of source distribution. Given samples from any source and target distributions, the velocity field can be learned through regression. For heavy-tailed targets, prior work explored multivariate t t t distributions as sources, reporting improved performance over Gaussian-source baselines, while other work proposed to learn quantile-based source distributions that adapt to heavy-tailed data [43 , 19 , 10 ] . Gaussian mixture sources have also been used effectively to generate multimodal targets [38 , 58 ] . Other work employed non-Euclidean source distributions to generate discrete data or data lying on a manifold, such as DNA sequences, proteins, crystals, and geographic locations [56 , 12 , 39 , 8 , 55 ] . In addition, flow matching has been used for data-to-data transport, such as translating images between domains, where the source is itself a non-Gaussian data distribution [35 , 57 ] . Despite these promising applications of non-Gaussian sources, the effects of source choice on training and sampling remain under debate [15 , 27 ] . It is largely unclear to what extent existing convergence results for standard Gaussian sources carry over to more general sources.
A few recent studies began to analyze the convergence behavior of flow matching and its variants with non-Gaussian sources. Some assumed the learned velocity field is bounded [20 ] , an assumption that may be restrictive for source distributions with unbounded support. Others imposed tail conditions on the target data distribution [19 ] or introduced Gaussian smoothing along the flow trajectory through stochastic interpolants [36 , 37 , 51 ] . This motivates the following question:
For commonly used non-Gaussian sources, can we establish nonasymptotic convergence guarantees without introducing Gaussian smoothing or imposing stronger assumptions on the target?
This paper
Main contributions. In this paper, we address the above question by analyzing the Euler sampler for flow matching when the true velocity field is available. Our results hold under the following conditions on the source and target distributions, which are satisfied by a broad class of commonly used distributions:
The source distribution P b a s e P_{\mathrm{base}} P base has an everywhere positive and continuously differentiable density on R d \mathbb{R}^{d} R d , finite total variance, and an L L L -Lipschitz score function.
The target distribution P d a t a P_{\mathrm{data}} P data has a finite second moment that is not overly large relative to the source’s second moment.
Under these conditions, we prove that the Euler sampler for flow matching achieves a total-variation-based error of at most ε \varepsilon ε using at most
O ~ ( L d TotVar b a s e ε ) (2) \widetilde{O}\left(\frac{Ld\,\operatorname{TotVar}_{\mathrm{base}}}{\varepsilon}\right) \tag*{(2)} O ( ε L d TotVar base ) (2)
evaluations of the velocity field, where TotVar b a s e : = E Z ∼ P b a s e [ ∥ Z − E [ Z ] ∥ 2 ] \operatorname{TotVar}_{\mathrm{base}} := \mathbb{E}_{Z \sim P_{\mathrm{base}}}\left[\left\|Z-\mathbb{E}[Z]\right\|^{2}\right] TotVar base := E Z ∼ P base [ ∥ Z − E [ Z ] ∥ 2 ] represents the total variance of the source distribution.
Our convergence theory is established under substantially milder assumptions on the source and target distributions than those imposed in prior work. For the source distribution, the assumptions of a Lipschitz score and a finite second moment are satisfied, for example, by multivariate Gaussian distributions with positive definite covariance matrices, multivariate Student’s t t t distributions with more than two degrees of freedom, and finite Gaussian mixtures whose components share a positive definite covariance matrix; note that both Student’s t t t distributions and Gaussian mixtures have been used as non-Gaussian sources in flow matching applications [43 , 38 ] . For the target distribution, we require only a finite second moment that is not too large relative to that of the source, with no additional tail assumptions. These results demonstrate that flow matching samplers can efficiently transport a broad class of non-Gaussian source distributions, including heavy-tailed distributions, to fairly general target distributions.
From a technical standpoint, our analysis rests on carefully characterizing the nonlinear score of the source distribution and establishing a restricted injectivity property for the Euler map. Both of these technical steps are more involved than in the Gaussian case, the latter of which features an affine source score and a globally injective Euler map.
Organization. The remainder of this paper is organized as follows. § 2 formally introduces the flow matching sampler. § 3 presents our main convergence theory and its consequences, followed by additional related work in § 4 . § 5 outlines the proof strategy and highlights the key steps for handling non-Gaussian source distributions. § 6 proves the main theorem, with proofs of auxiliary results deferred to the appendix. § 7 concludes the paper by discussing directions for future work.
Notation. We use the following notational conventions throughout. Capital letters X ⋆ , Z , X t , Y n X_{\star}, Z, X_{t}, Y_{n} X ⋆ , Z , X t , Y n denote random vectors, while lowercase letters x ⋆ , z , x , y n x_{\star}, z, x, y_{n} x ⋆ , z , x , y n denote fixed values or integration variables. For any random vector W W W , let P W P_{W} P W denote its law and p W p_{W} p W its Lebesgue density when it exists. We write p X ⋆ ∣ X t ( ⋅ ∣ x ) p_{X_{\star} \mid X_{t}}(\cdot\mid x) p X ⋆ ∣ X t ( ⋅ ∣ x ) for the density of the conditional law of X ⋆ X_{\star} X ⋆ given X t = x X_{t}=x X t = x , with respect to P d a t a = P X ⋆ P_{\mathrm{data}}=P_{X_{\star}} P data = P X ⋆ . For probability laws P P P and Q Q Q , define their total-variation (TV) distance as
TV ( P , Q ) = 1 2 ∫ ∣ d P − d Q ∣ . \operatorname{TV}(P,Q)=\frac{1}{2}\int\lvert\mathrm{d}P-\mathrm{d}Q\rvert. TV ( P , Q ) = 2 1 ∫ ∣ d P − d Q ∣ .
We also adopt the Jacobian convention [ ∇ x u ] i j = ∂ u i / ∂ x j [\nabla x u]_{ij}=\partial u_{i}/\partial x_{j} [ ∇ xu ] ij = ∂ u i / ∂ x j . For a symmetric matrix Σ \Sigma Σ , λ min ( Σ ) \lambda_{\min}(\Sigma) λ m i n ( Σ ) denotes the smallest eigenvalue of Σ \Sigma Σ .
We use f ( ⋅ ) = O ( g ( ⋅ ) ) f(\cdot)=O(g(\cdot)) f ( ⋅ ) = O ( g ( ⋅ )) to mean that f ≤ C 0 g f\le C_{0}g f ≤ C 0 g for some universal constant C 0 > 0 C_{0}>0 C 0 > 0 . The notation O ~ ( ⋅ ) \widetilde{O}(\cdot) O ( ⋅ ) has the same meaning as O ( ⋅ ) O(\cdot) O ( ⋅ ) , except that it hides the logarithmic factors. Unless stated otherwise, C , c > 0 C,c>0 C , c > 0 denote numerical constants whose values may change from line to line.
Preliminaries
In this section, we formalize the flow-matching setup and the Euler sampler studied in this paper.
Flow matching setup. Recall that P b a s e P_{\mathrm{base}} P base and P d a t a P_{\mathrm{data}} P data denote the source and target distributions on R d \mathbb{R}^{d} R d , respectively. Flow matching models the transport from a source distribution P b a s e P_{\mathrm{base}} P base to P d a t a P_{\mathrm{data}} P data through an ODE driven by a time-dependent velocity field:
d X t d t = v t ( X t ) , X 0 ∼ P b a s e , (3) \frac{\mathrm{d}X_t}{\mathrm{d}t}=v_t(X_t), \qquad X_0\sim P_{\mathrm{base}}, \tag*{(3)} d t d X t = v t ( X t ) , X 0 ∼ P base , (3)
where the velocity field { v t ( ⋅ ) } t ∈ [ 0 , 1 ) \{v_t(\cdot)\}_{t\in[0,1)} { v t ( ⋅ ) } t ∈ [ 0 , 1 ) is assumed to be sufficiently regular to ensure existence and uniqueness of solutions to this ODE. Let ϕ t : R d → R d \phi_t:\mathbb{R}^{d}\to\mathbb{R}^{d} ϕ t : R d → R d denote the corresponding flow map, which maps each initial condition at time 0 0 0 to the solution at time t t t , i.e.,
X t = ϕ t ( X 0 ) . (4) X_t=\phi_t(X_0). \tag*{(4)} X t = ϕ t ( X 0 ) . (4)
Throughout this paper, we let P t = P X t P_t=P_{X_t} P t = P X t denote the law of X t X_t X t for 0 ≤ t < 1 0\le t<1 0 ≤ t < 1 . We assume that P b a s e P_{\mathrm{base}} P base and each P t P_t P t (0 ≤ t < 1 0\le t<1 0 ≤ t < 1 ) admit densities p b a s e p_{\mathrm{base}} p base and p t p_t p t , respectively; in particular, p 0 = p b a s e p_0=p_{\mathrm{base}} p 0 = p base . For an ideal velocity field, P t P_t P t converges weakly to P d a t a P_{\mathrm{data}} P data as t ↑ 1 t\uparrow1 t ↑ 1 .
Flow matching trains a continuous normalizing flow by regressing a parameterized velocity field onto a target field that generates a probability path { P t } t ∈ [ 0 , 1 ] \{P_t\}_{t\in[0,1]} { P t } t ∈ [ 0 , 1 ] connecting the source and target distributions. A core idea behind flow matching is to construct this path using conditional flows, which we introduce next.
Conditional flows. A useful way to construct a probability path from P b a s e P_{\mathrm{base}} P base to P d a t a P_{\mathrm{data}} P data is to first specify how a source sample moves toward a given target sample, then average over target samples. More precisely, let X ⋆ ∼ P d a t a X_{\star}\sim P_{\mathrm{data}} X ⋆ ∼ P data and Z ∼ P b a s e Z\sim P_{\mathrm{base}} Z ∼ P base be independent samples from the target and source distributions. Conditional on X ⋆ = x ⋆ X_{\star}=x_{\star} X ⋆ = x ⋆ , consider the flow defined by
d X t d t = v t ( X t ∣ x ⋆ ) , X 0 = Z , v t ( x ∣ x ⋆ ) = x ⋆ − x 1 − t , t ∈ [ 0 , 1 ) . (5) \frac{\mathrm{d}X_t}{\mathrm{d}t}=v_t(X_t\mid x_{\star}), \qquad X_0=Z, \qquad v_t(x\mid x_{\star})=\frac{x_{\star}-x}{1-t}, \qquad t\in[0,1). \tag*{(5)} d t d X t = v t ( X t ∣ x ⋆ ) , X 0 = Z , v t ( x ∣ x ⋆ ) = 1 − t x ⋆ − x , t ∈ [ 0 , 1 ) . (5)
In words, v t ( ⋅ ∣ x ⋆ ) v_t(\cdot\mid x_{\star}) v t ( ⋅ ∣ x ⋆ ) is the conditional velocity field that points from the current state toward the fixed target x ⋆ x_{\star} x ⋆ . Solving this ODE (5) for each target value yields the linear interpolant
X t = ( 1 − t ) Z + t x ⋆ , t ∈ [ 0 , 1 ) , (6) X_t=(1-t)Z+tx_{\star}, \qquad t\in[0,1), \tag*{(6)} X t = ( 1 − t ) Z + t x ⋆ , t ∈ [ 0 , 1 ) , (6)
where X 1 = x ⋆ X_1=x_{\star} X 1 = x ⋆ by continuous extension. For t < 1 t<1 t < 1 , the resulting conditional density obeys
p X t ∣ X ⋆ ( x ∣ x ⋆ ) = 1 ( 1 − t ) d p b a s e ( x − t x ⋆ 1 − t ) . (7) p_{X_t\mid X_{\star}}(x\mid x_{\star})=\frac{1}{(1-t)^d}p_{\mathrm{base}}\left(\frac{x-tx_{\star}}{1-t}\right). \tag*{(7)} p X t ∣ X ⋆ ( x ∣ x ⋆ ) = ( 1 − t ) d 1 p base ( 1 − t x − t x ⋆ ) . (7)
This straight-line interpolation corresponds to the conditional optimal transport path in Lipman et al. [34 ] ; it was also used by Liu et al. [35 ] to construct 1-rectified flow.
Marginal flow. Marginalizing the conditional densities over X ⋆ ∼ P d a t a X_{\star}\sim P_{\mathrm{data}} X ⋆ ∼ P data yields the marginal probability path
p t ( x ) : = p X t ( x ) = ∫ R d p X t ∣ X ⋆ ( x ∣ x ⋆ ) P d a t a ( d x ⋆ ) , t ∈ [ 0 , 1 ) , (8) p_t(x):=p_{X_t}(x)=\int_{\mathbb{R}^{d}}p_{X_t\mid X_{\star}}(x\mid x_{\star})P_{\mathrm{data}}(\mathrm{d}x_{\star}), \qquad t\in[0,1), \tag*{(8)} p t ( x ) := p X t ( x ) = ∫ R d p X t ∣ X ⋆ ( x ∣ x ⋆ ) P data ( d x ⋆ ) , t ∈ [ 0 , 1 ) , (8)
with p 0 = p b a s e p_0=p_{\mathrm{base}} p 0 = p base , and P t P_t P t converges weakly to P d a t a P_{\mathrm{data}} P data as t ↑ 1 t\uparrow1 t ↑ 1 . The corresponding marginal velocity field satisfies
v t ( x ) = E [ v t ( x ∣ X ⋆ ) ∣ X t = x ] = E [ X ⋆ − Z ∣ X t = x ] = E [ X ⋆ ∣ X t = x ] − x 1 − t . (9) v_t(x)=\mathbb{E}[v_t(x\mid X_{\star})\mid X_t=x]=\mathbb{E}[X_{\star}-Z\mid X_t=x]=\frac{\mathbb{E}[X_{\star}\mid X_t=x]-x}{1-t}. \tag*{(9)} v t ( x ) = E [ v t ( x ∣ X ⋆ ) ∣ X t = x ] = E [ X ⋆ − Z ∣ X t = x ] = 1 − t E [ X ⋆ ∣ X t = x ] − x . (9)
Here, the conditional expectation is taken under the joint law of ( X t , X ⋆ ) (X_t,X_{\star}) ( X t , X ⋆ ) corresponding to
X t = ( 1 − t ) Z + t X ⋆ , Z ∼ P b a s e , X ⋆ ∼ P d a t a , (10) X_t=(1-t)Z+tX_{\star}, \qquad Z\sim P_{\mathrm{base}}, \qquad X_{\star}\sim P_{\mathrm{data}}, \tag*{(10)} X t = ( 1 − t ) Z + t X ⋆ , Z ∼ P base , X ⋆ ∼ P data , (10)
with Z Z Z and X ⋆ X_\star X ⋆ independent. Under suitable integrability and regularity assumptions, the marginal density and velocity field satisfy the continuity equation or transport equation Lipman et al., [34 ] :
∂ t p t + ∇ x ⋅ ( p t v t ) = 0. \partial_t p_t + \nabla_x \cdot(p_t v_t) = 0. ∂ t p t + ∇ x ⋅ ( p t v t ) = 0.
Consequently, under these regularity conditions, the solution of ODE (3) with v t ( ⋅ ) v_t(\cdot) v t ( ⋅ ) defined in (9) satisfies X t ∼ P t X_t \sim P_t X t ∼ P t at each time t t t .
Euler discretization for sampling. To generate samples in practice, the continuous-time ODE must be approximated numerically. We focus on the Euler sampler — a standard, first-order discretization of the continuous-time ODE — with a rescaled initialization and early stopping. Given a time grid 0 < t 0 < t 1 < ⋯ < t N < 1 0 < t_0 < t_1 < \cdots< t_N < 1 0 < t 0 < t 1 < ⋯ < t N < 1 of N + 1 N + 1 N + 1 time points, the Euler method proceeds as
Y 0 = d ( 1 − t 0 ) Z , Y n = Y n − 1 + v t n − 1 ( Y n − 1 ) ( t n − t n − 1 ) , n = 1 , … , N . (11) Y_0 \mathrel{\mathop{=}^{\mathrm{d}}} (1-t_0)Z,\qquad Y_n = Y_{n-1} + v_{t_{n-1}}(Y_{n-1})(t_n-t_{n-1}),\qquad n=1,\ldots,N. \tag*{(11)} Y 0 = d ( 1 − t 0 ) Z , Y n = Y n − 1 + v t n − 1 ( Y n − 1 ) ( t n − t n − 1 ) , n = 1 , … , N . (11)
At each step, the velocity field is evaluated at the left endpoint t n − 1 t_{n-1} t n − 1 , and the sampler outputs Y N Y_N Y N . As will be specified in § 3.1 , both t 0 t_0 t 0 and 1 − t N 1-t_N 1 − t N are chosen to be sufficiently small. The positive initial time t 0 t_0 t 0 is a technical modification that requires only rescaling the initial noise. Early stopping at t N < 1 t_N < 1 t N < 1 is common in the literature [11 , 16 ] . As we will shortly see, this allows us to establish total-variation-based guarantees relative to the intermediate law P t N P_{t_N} P t N (which has a continuously differentiable density p t N p_{t_N} p t N ), since the data distribution P d a t a P_{\mathrm{data}} P data may be singular.
Remark 1. In this paper, we focus on the sampling error and therefore consider the Euler sampler with the population velocity field v t v_t v t , as defined in (11) . In practice, one often uses a learned velocity field v ^ t \widehat{v}_t v t , trained by regressing X ⋆ − Z X_\star-Z X ⋆ − Z upon X t = ( 1 − t ) Z + t X ⋆ X_t=(1-t)Z+tX_\star X t = ( 1 − t ) Z + t X ⋆ .
Main results
In this section, we establish convergence guarantees for the Euler method for flow matching, by bounding the total-variation distance between the sampler’s output distribution P Y N P_{Y_N} P Y N and the law P t N = P X t N P_{t_N}=P_{X_{t_N}} P t N = P X t N of the interpolant at time t N t_N t N . Note that the law P t N P_{t_N} P t N has a continuously differentiable density and converges weakly to the data distribution P d a t a P_{\mathrm{data}} P data as t N ↑ 1 t_N \uparrow1 t N ↑ 1 .
Assumptions
We begin by stating the assumptions needed for our theoretical analysis, where the first assumption is concerned with the source distribution.
Assumption 1. The source distribution P b a s e P_{\mathrm{base}} P base has a strictly positive, continuously differentiable (C 1 C^1 C 1 ) density p b a s e p_{\mathrm{base}} p base on R d \mathbb{R}^d R d , and obeys E Z ∼ P b a s e [ ∥ Z − E [ Z ] ∥ 2 2 ] < ∞ \mathbb{E}_{Z\sim P_{\mathrm{base}}}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right] < \infty E Z ∼ P base [ ∥ Z − E [ Z ] ∥ 2 2 ] < ∞ . Additionally, its score function s ( ⋅ ) = ∇ log p b a s e ( ⋅ ) s(\cdot)=\nabla\log p_{\mathrm{base}}(\cdot) s ( ⋅ ) = ∇ log p base ( ⋅ ) is L L L -Lipschitz, namely,
∥ s ( z ) − s ( z ′ ) ∥ 2 ≤ L ∥ z − z ′ ∥ 2 for all z , z ′ ∈ R d . (12) \lVert s(z)-s(z')\rVert_2 \le L\lVert z-z'\rVert_2\qquad\text{for all }z,z'\in\mathbb{R}^d. \tag*{(12)} ∥ s ( z ) − s ( z ′ ) ∥ 2 ≤ L ∥ z − z ′ ∥ 2 for all z , z ′ ∈ R d . (12)
This assumption concerns only the source distribution, which can be chosen by the practitioner. It is satisfied, for example, by multivariate Gaussian distributions with positive-definite covariance matrices, multivariate Student’s t t t distributions with more than two degrees of freedom, and finite Gaussian mixtures whose components share a positive-definite covariance matrix. Student’s t t t distributions and Gaussian mixtures have both been used as non-Gaussian sources in flow-matching applications [43 , 38 ] . For a new source distribution, the assumption can be checked by verifying the regularity of its density and score.
The next assumption relates the target’s second moment to the source’s centered second moment.
Assumption 2. Let X ⋆ ∼ P d a t a X_\star\sim P_{\mathrm{data}} X ⋆ ∼ P data and Z ∼ P b a s e Z \sim P_{\mathrm{base}} Z ∼ P base . Assume that there exists a fixed numerical constant c R > 0 c_R > 0 c R > 0 such that
E [ ∥ X ⋆ ∥ 2 2 ] ≤ ( N + 1 ) c R E [ ∥ Z − E [ Z ] ∥ 2 2 ] . (13) \mathbb{E}\left[\lVert X_\star\rVert_2^2\right] \le(N+1)^{c_R}\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]. \tag*{(13)} E [ ∥ X ⋆ ∥ 2 2 ] ≤ ( N + 1 ) c R E [ ∥ Z − E [ Z ] ∥ 2 2 ] . (13)
Assumption 2 allows the target’s second moment to grow polynomially with the number N N N of sampling steps, while controlling it relative to the source’s centered second moment. In particular, it favors choosing a source that is not too concentrated relative to the target.
Last but not least, we impose a condition on the discretization time grid. It requires the initial and stopping times to lie near the endpoints and controls the step sizes relative to their distance from those endpoints.
Assumption 3. Fix constants c 0 ≥ c R + 4 c_0 \ge c_R + 4 c 0 ≥ c R + 4 and c s > 0 c_s > 0 c s > 0 , and suppose c s log ( N + 1 ) / ( N + 1 ) ≤ 1 / 2 c_s \log(N + 1)/(N + 1) \le1/2 c s log ( N + 1 ) / ( N + 1 ) ≤ 1/2 . The sampling times 0 < t 0 < t 1 < ⋯ < t N < 1 0 < t_0 < t_1 < \cdots< t_N < 1 0 < t 0 < t 1 < ⋯ < t N < 1 satisfy
t 0 ≤ ( N + 1 ) − c 0 / 2 , 1 − t N ≤ ( N + 1 ) − c 0 / 2 , t_0 \le(N+1)^{-c_0/2}, \qquad1-t_N \le(N+1)^{-c_0/2}, t 0 ≤ ( N + 1 ) − c 0 /2 , 1 − t N ≤ ( N + 1 ) − c 0 /2 , t n − t n − 1 t n − 1 ( 1 − t n ) ≤ c s log ( N + 1 ) N + 1 , 1 ≤ n ≤ N . (14) \frac{t_n-t_{n-1}}{t_{n-1}(1-t_n)} \le\frac{c_s\log(N+1)}{N+1}, \qquad1 \le n \le N. \tag*{(14)} t n − 1 ( 1 − t n ) t n − t n − 1 ≤ N + 1 c s log ( N + 1 ) , 1 ≤ n ≤ N . (14)
For concreteness, we note that Assumption 3 can be met by, for example, either of the following schedules:
(i) t n = ( 1 + ( N + 1 ) c 0 ( 1 / 2 − n / N ) ) − 1 t_n = \left(1 + (N+1)^{c_0(1/2-n/N)}\right)^{-1} t n = ( 1 + ( N + 1 ) c 0 ( 1/2 − n / N ) ) − 1 .
(ii) The times increase geometrically near zero, and approach one geometrically near the end:
t n = { ε q n , 0 ≤ n ≤ ⌊ N / 2 ⌋ , 1 − ε q N − n , ⌊ N / 2 ⌋ < n ≤ N , t_n =
\begin{cases}
\varepsilon q^n, & 0 \le n \le\lfloor N/2\rfloor,\\
1-\varepsilon q^{N-n}, & \lfloor N/2\rfloor< n \le N,
\end{cases} t n = { ε q n , 1 − ε q N − n , 0 ≤ n ≤ ⌊ N /2 ⌋ , ⌊ N /2 ⌋ < n ≤ N ,
where ε = ( N + 1 ) − c 0 / 2 \varepsilon= (N+1)^{-c_0/2} ε = ( N + 1 ) − c 0 /2 and q = ( 2 ε ) − 2 / N q = (2\varepsilon)^{-2/N} q = ( 2 ε ) − 2/ N .
Both schedules use smaller steps near the endpoints. Related grids also appear in convergence analyses of diffusion models ([25 ] ).
Convergence theory and implications
We are now positioned to state the main convergence guarantee. For notational convenience, we use
T o t V a r b a s e : = E Z ∼ P b a s e [ ∥ Z − E [ Z ] ∥ 2 2 ] (15) \mathrm{TotVar}_{\mathrm{base}} := \mathbb{E}_{Z\sim P_{\mathrm{base}}}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right] \tag*{(15)} TotVar base := E Z ∼ P base [ ∥ Z − E [ Z ] ∥ 2 2 ] (15)
to denote the total variance of the source distribution P b a s e P_{\mathrm{base}} P base throughout the rest of the paper.
Theorem 1 (Sources with a finite second moment). Suppose N ≥ 2 N \ge2 N ≥ 2 , and Assumptions 1, 2 and 3 hold. Then the Euler sampler for flow matching (cf. (11) ) satisfies
TV ( P Y N , P t N ) ≤ C L ( d + log N ) T o t V a r b a s e log 2 N N (16) \operatorname{TV}(P_{Y_N},P_{t_N}) \le\frac{CL(d+\log N)\mathrm{TotVar}_{\mathrm{base}}\log^2 N}{N} \tag*{(16)} TV ( P Y N , P t N ) ≤ N C L ( d + log N ) TotVar base log 2 N (16) for some universal constant C > 0 C > 0 C > 0 .
Theorem 1 develops an upper bound on the TV distance between the Euler sampler’s output distribution P Y N P_{Y_N} P Y N and the true marginal law P t N = P X t N P_{t_N}=P_{X_{t_N}} P t N = P X t N . By Assumption 3, the sampler is terminated at a time t N t_N t N satisfying 1 − ( N + 1 ) − c 0 / 2 ≤ t N < 1 1-(N+1)^{-c_0/2} \le t_N < 1 1 − ( N + 1 ) − c 0 /2 ≤ t N < 1 . Note that P t N P_{t_N} P t N is a slightly perturbed version of the target distribution, representing the law of
t N X ⋆ + ( 1 − t N ) Z ∼ P t N t_N X_\star+ (1-t_N)Z \sim P_{t_N} t N X ⋆ + ( 1 − t N ) Z ∼ P t N
with ( X ⋆ , Z ) ∼ P d a t a × P b a s e (X_\star,Z) \sim P_{\mathrm{data}} \times P_{\mathrm{base}} ( X ⋆ , Z ) ∼ P data × P base . As alluded to previously, we compare the sampler with this intermediate law because the target distribution P d a t a P_{\mathrm{data}} P data may be singular and need not have a density, whereas P t N P_{t_N} P t N has a continuously differentiable density. This way of comparison is also common in convergence analyses of diffusion models ([11 , 16 ] ). Furthermore, it is noteworthy that Theorem 1 is established under minimal assumptions on the source and target distributions: a Lipschitz score condition on the source, a mild second-moment bound on the target (relative to the source), and a standard stepsize condition. This confirms the flow matching sampler’s capability in efficiently transporting fairly general source distributions to target distributions.
A few additional remarks are as follows.
Comparison with prior convergence theory for non-Gaussian sources. Perhaps the most closely related results to ours are those of Guan et al. [19 ] . Guan et al. [19 ] established Wasserstein bounds for flow matching in the dual space with a Student-t ν t_{\nu} t ν source, encompassing Euclidean flow matching as a special case. However, its guarantee requires a bounded target density in dual coordinates satisfying p 1 ( x ) ≲ ∥ x ∥ − 2 d − ν − 2 p_1(x) \lesssim\lVert x\rVert^{-2d-\nu-2} p 1 ( x ) ≲ ∥ x ∥ − 2 d − ν − 2 for ∥ x ∥ ≥ 1 \lVert x\rVert\ge1 ∥ x ∥ ≥ 1 . This condition excludes discrete targets and target densities with heavier tails. Guan et al. [20 ] established TV bounds for Riemannian flow matching, with Euclidean flow matching covered by its result for Hadamard manifolds. They imposed additional regularity assumptions on the velocity fields, including boundedness assumptions. Notably, the boundedness condition may fail to hold even for a standard Gaussian source.
Consequences of Theorem 1 for commonly used source distributions. The source distribution mainly affects the TV-based guarantee through the factor L T o t V a r b a s e L_{\mathrm{TotVar}_{\mathrm{base}}} L TotVar base . Here, we evaluate this quantity analytically for a few examples; the calculations are postponed to Appendix § 9 .§ 1 .
Gaussian source. Let Z ∼ N ( μ , Σ ) Z \sim\mathcal{N}(\mu,\Sigma) Z ∼ N ( μ , Σ ) with mean μ ∈ R d \mu\in\mathbb{R}^{d} μ ∈ R d and covariance matrix Σ ≻ 0 \Sigma\succ0 Σ ≻ 0 . Then
L T o t V a r b a s e = tr ( Σ ) λ min ( Σ ) . (17) L_{\mathrm{TotVar}_{\mathrm{base}}} = \frac{\operatorname{tr}(\Sigma)}{\lambda_{\min}(\Sigma)}. \tag*{(17)} L TotVar base = λ m i n ( Σ ) tr ( Σ ) . (17)
Multivariate Student’s t t t source. Let Z ∼ t ν ( μ , Σ ) Z \sim t_{\nu}(\mu,\Sigma) Z ∼ t ν ( μ , Σ ) have ν > 2 \nu> 2 ν > 2 degrees of freedom, location μ ∈ R d \mu\in\mathbb{R}^{d} μ ∈ R d , and scale matrix Σ ≻ 0 \Sigma\succ0 Σ ≻ 0 . In this case, one has
L T o t V a r b a s e = ν + d ν − 2 tr ( Σ ) λ min ( Σ ) . (18) L_{\mathrm{TotVar}_{\mathrm{base}}} = \frac{\nu+d}{\nu-2}\frac{\operatorname{tr}(\Sigma)}{\lambda_{\min}(\Sigma)}. \tag*{(18)} L TotVar base = ν − 2 ν + d λ m i n ( Σ ) tr ( Σ ) . (18)
Z ∼ ∑ k = 1 m π k N ( μ k , Σ ) , Z \sim\sum_{k=1}^{m} \pi_k \mathcal{N}(\mu_k,\Sigma), Z ∼ k = 1 ∑ m π k N ( μ k , Σ ) ,
where π k > 0 \pi_k > 0 π k > 0 , ∑ k = 1 m π k = 1 \sum_{k=1}^{m}\pi_k = 1 ∑ k = 1 m π k = 1 , μ k ∈ R d \mu_k \in\mathbb{R}^{d} μ k ∈ R d , and Σ ≻ 0 \Sigma\succ0 Σ ≻ 0 . Then one has
L T o t V a r b a s e = tr ( Σ ) + B λ min ( Σ ) max { 1 , Δ 2 4 − 1 } , (19) L_{\mathrm{TotVar}_{\mathrm{base}}} = \frac{\operatorname{tr}(\Sigma)+B}{\lambda_{\min}(\Sigma)}\max\left\{1,\frac{\Delta^2}{4}-1\right\}, \tag*{(19)} L TotVar base = λ m i n ( Σ ) tr ( Σ ) + B max { 1 , 4 Δ 2 − 1 } , (19)
where we take μ ˉ = ∑ k π k μ k \bar{\mu} = \sum_k \pi_k\mu_k μ ˉ = ∑ k π k μ k and
B = ∑ k = 1 m π k ∥ μ k − μ ˉ ∥ 2 2 , Δ 2 = max 1 ≤ i , j ≤ m ( μ i − μ j ) T Σ − 1 ( μ i − μ j ) . B = \sum_{k=1}^{m}\pi_k\lVert\mu_k-\bar{\mu}\rVert_2^2,\qquad\Delta^2 = \max_{1\le i,j\le m}(\mu_i-\mu_j)^{\mathsf{T}}\Sigma^{-1}(\mu_i-\mu_j). B = k = 1 ∑ m π k ∥ μ k − μ ˉ ∥ 2 2 , Δ 2 = 1 ≤ i , j ≤ m max ( μ i − μ j ) T Σ − 1 ( μ i − μ j ) .
In each of the above examples, L T o t V a r b a s e L_{\mathrm{TotVar}_{\mathrm{base}}} L TotVar base depends on λ min ( Σ ) − 1 \lambda_{\min}(\Sigma)^{-1} λ m i n ( Σ ) − 1 together with a measure of total spread, e.g., tr ( Σ ) \operatorname{tr}(\Sigma) tr ( Σ ) . This suggests the potential benefit of a well-conditioned source: a source with better-conditioned covariance may facilitate faster sampling for a broad class of target distributions.
Related work
The flow matching framework was first introduced by Lipman et al. [34 ] as a simulation-free approach to learning velocity fields for a range of probability paths, including diffusion paths and conditional optimal transport (OT) paths. In concurrent work, Liu et al. [35 ] introduced rectified flow using straight-line interpolation between source and target samples, which coincides with the interpolation rule that Lipman et al. [34 ] used in their conditional OT path when the terminal noise is zero. Liu et al. [35 ] also proposed reflow, using endpoint pairs from a previously learned flow to train a new flow with straighter trajectories. Pooladian et al. [45 ] and Tong et al. [57 ] pursued a similar goal by coupling source and target samples within each minibatch. In complementary work, Albergo and Vanden-Eijnden [1 ] introduced a stochastic interpolant framework that supports both linear and trigonometric interpolation; Albergo et al. [2 ] later extended it to allow auxiliary noise in the interpolant. In the current paper, we focus on the flow induced by straight-line interpolation between independent source and target samples, which is also the 1-rectified flow of Liu et al. [35 ] .
Our work is intimately related to prior convergence theory developed for both flow matching and diffusion models, which often considers two types of error. The first is the statistical error in estimating the velocity field or score function from a finite training sample ([42 , 18 , 16 ] ). The second is the sampling error, which concerns how discretization affects the accuracy of the sampler. We focus on the sampling error in this work and assume that the true velocity field is available. For diffusion models, there is a rich literature on sampling convergence under Gaussian corruption, including results for the deterministic DDIM sampler ([52 , 13 , 31 , 33 ] ), the stochastic DDPM sampler ([22 , 11 , 5 , 47 , 23 , 25 ] (Li and Yan 2024; Li and Yan 2025)), and their higher-order variants ([30 , 60 , 32 , 24 ] ). The connection between rectified flow and DDPM was leveraged to study low-dimensional adaptation of rectified flow in Roy et al. [49 ] under Gaussian sources. A few studies also develop convergence theory for specific non-Gaussian corruptions, such as α \alpha α -stable noise ([21 ] ).
A strand of recent work has studied convergence properties for flow matching and its variants (e.g., (Liu et al. (2025, 2026); Silveri et al. (2026); Benton et al. (2024); Cheng et al. (2024); Bansal et al. (2025); Roy et al. (2026); Guan et al. (2026a,b); Wang et al. (2026b); Chambolle and Hertrich (2026); Xie and Cheng (2026); Pham et al. (2026))). For instance, Liu et al. [36 , 37 ] ; Silveri et al. [51 ] studied ODE or SDE samplers for stochastic interpolants that introduce Gaussian noise along the flow trajectory. Guan et al. [19 ] established Wasserstein bounds for mirror flow matching with a Student’s t t t source under a polynomial tail condition on the target density. Guan et al. [20 ] established total variation bounds for Riemannian flow matching under regularity and boundedness assumptions on the population and learned velocity fields. In comparison, the present paper establishes a total variation bound for the Euler sampler with linear interpolation and no auxiliary Gaussian noise. Our result applies to a broad class of non-Gaussian sources and requires neither a bounded velocity field nor a polynomial tail bound on the target density.
Technical novelty
Our theory accommodates general non-Gaussian sources, including heavy-tailed sources such as the multivariate Student’s t t t distributions. Accordingly, we do not assume exponential tails or higher moments beyond the second. This creates two major obstacles in our analysis: the one-step Euler map does not inherit the injectivity of Gaussian source, and the local density-comparison error analysis cannot assume arbitrarily high-order moments to be finite; we also need to deal with multiple remainder terms due to non-Gaussian source throughout the proof. In this section, we briefly describe our main proof framework and highlight our strategies for tackling non-Gaussian source-related challenges.
Decomposition of the final TV error. First of all, different from prior analysis for DDIM convergence ([31 ] ) where density ratio error at each step is accumulated across the whole sampling trajectory, we make the following observation:
TV ( P Y N , P t N ) ≤ TV ( P Y 0 , P t 0 ) + ∑ n = 1 N TV ( P Y ^ n , P t n ) \operatorname{TV}(P_{Y_N}, P_{t_N}) \le\operatorname{TV}(P_{Y_0}, P_{t_0}) + \sum_{n=1}^{N} \operatorname{TV}(P_{\widehat{Y}_n}, P_{t_n}) TV ( P Y N , P t N ) ≤ TV ( P Y 0 , P t 0 ) + n = 1 ∑ N TV ( P Y n , P t n )
where Y ^ n = T n ( X t n − 1 ) \widehat{Y}_n = T_n(X_{t_{n-1}}) Y n = T n ( X t n − 1 ) and T n T_n T n is the one-step Euler update function:
T n ( X t n − 1 ) = X t n − 1 + ( t n − t n − 1 ) v t n − 1 ( X t n − 1 ) . T_n(X_{t_{n-1}}) = X_{t_{n-1}} + (t_n - t_{n-1})v_{t_{n-1}}(X_{t_{n-1}}). T n ( X t n − 1 ) = X t n − 1 + ( t n − t n − 1 ) v t n − 1 ( X t n − 1 ) .
P t n P_{t_n} P t n is the distribution of X t n = X t n − 1 + ( t n − t n − 1 ) ( X ⋆ − Z ) X_{t_n} = X_{t_{n-1}} + (t_n - t_{n-1})(X_{\star} - Z) X t n = X t n − 1 + ( t n − t n − 1 ) ( X ⋆ − Z ) . Thus, each summand compares one Euler update from the exact marginal P t n − 1 P_{t_{n-1}} P t n − 1 with the next marginal.
When the map T n T_n T n is C 1 C^1 C 1 and globally injective with image space R d \mathbb{R}^d R d , then for every Borel set A A A ,
P ( Y ^ n ∈ A ) = ∫ T n − 1 ( A ) p t n − 1 ( y ) d y \mathbb{P}(\widehat{Y}_n \in A) = \int_{T_n^{-1}(A)} p_{t_{n-1}}(y)\,\mathrm{d}y P ( Y n ∈ A ) = ∫ T n − 1 ( A ) p t n − 1 ( y ) d y
P ( X t n ∈ A ) = ∫ T n − 1 ( A ) p t n ( T n ( y ) ) ∣ det ∇ T n ( y ) ∣ d y . \mathbb{P}(X_{t_n} \in A)=\int_{T_n^{-1}(A)}p_{t_n}(T_n(y))\left|\det\nabla T_n(y)\right|\,\mathrm{d}y. P ( X t n ∈ A ) = ∫ T n − 1 ( A ) p t n ( T n ( y )) ∣ det ∇ T n ( y ) ∣ d y .
Hence,
TV ( P ^ Y n , P t n ) = ∫ p t n − 1 ( x ) [ 1 − p t n ( T n ( x ) ) ∣ det ∇ T n ( x ) ∣ p t n − 1 ( x ) ] + d x . \operatorname{TV}(\widehat{P}_{Y_n},P_{t_n})=\int p_{t_{n-1}}(x)\left[1-\frac{p_{t_n}(T_n(x))\left|\det\nabla T_n(x)\right|}{p_{t_{n-1}}(x)}\right]_{+}\,\mathrm{d}x. TV ( P Y n , P t n ) = ∫ p t n − 1 ( x ) [ 1 − p t n − 1 ( x ) p t n ( T n ( x )) ∣ det ∇ T n ( x ) ∣ ] + d x .
Consequently, a lower bound on the combined log-density and log-determinant ratio suffices. However, the global injectivity of T n T_n T n follows for a standard Gaussian source but may not hold for more general sources. In fact,
∇ T n ( x ) = 1 − t n 1 − t n − 1 I d − t n − t n − 1 t n − 1 ( 1 − t n − 1 ) Cov ( Z , s ( Z ) ∣ X t n − 1 = x ) = Z ∼ N ( 0 , I d ) 1 − t n 1 − t n − 1 I d + t n − t n − 1 t n − 1 ( 1 − t n − 1 ) Cov ( Z ∣ X t n − 1 = x ) ⪰ 1 − t n 1 − t n − 1 I d , (20) \begin{aligned}
\nabla T_n(x)&=\frac{1-t_n}{1-t_{n-1}}I_d-\frac{t_n-t_{n-1}}{t_{n-1}(1-t_{n-1})}\operatorname{Cov}(Z,s(Z)\mid X_{t_{n-1}}=x)\\
&\underset{Z\sim\mathcal{N}(0,I_d)}{=}\frac{1-t_n}{1-t_{n-1}}I_d+\frac{t_n-t_{n-1}}{t_{n-1}(1-t_{n-1})}\operatorname{Cov}(Z\mid X_{t_{n-1}}=x)\succeq\frac{1-t_n}{1-t_{n-1}}I_d, \tag*{(20)}
\end{aligned} ∇ T n ( x ) = 1 − t n − 1 1 − t n I d − t n − 1 ( 1 − t n − 1 ) t n − t n − 1 Cov ( Z , s ( Z ) ∣ X t n − 1 = x ) Z ∼ N ( 0 , I d ) = 1 − t n − 1 1 − t n I d + t n − 1 ( 1 − t n − 1 ) t n − t n − 1 Cov ( Z ∣ X t n − 1 = x ) ⪰ 1 − t n − 1 1 − t n I d , (20)
The second equality uses the standard Gaussian score s ( z ) = − z s(z)=-z s ( z ) = − z . The uniform positive lower bound on ∇ T n \nabla T_n ∇ T n gives strong monotonicity and hence global injectivity. For a general source, − Cov ( Z , s ( Z ) ∣ X t n − 1 = x ) -\operatorname{Cov}(Z,s(Z)\mid X_{t_{n-1}}=x) − Cov ( Z , s ( Z ) ∣ X t n − 1 = x ) need not be symmetric positive semidefinite, so the same argument does not apply. We therefore construct a Borel set G n ⊆ R d G_n\subseteq\mathbb{R}^d G n ⊆ R d on which T n T_n T n is injective and has positive determinant. The localized comparison at each step t n t_n t n then takes the following form:
TV ( P ^ Y n , P t n ) ≤ P ( X t n − 1 ∉ G n ) + ∫ G n p t n − 1 ( x ) [ 1 − p t n ( T n ( x ) ) det ∇ T n ( x ) p t n − 1 ( x ) ] + d x . (21) \operatorname{TV}(\widehat{P}_{Y_n},P_{t_n})\leq\mathbb{P}(X_{t_{n-1}}\notin G_n)+\int_{G_n}p_{t_{n-1}}(x)\left[1-\frac{p_{t_n}(T_n(x))\det\nabla T_n(x)}{p_{t_{n-1}}(x)}\right]_{+}\,\mathrm{d}x. \tag*{(21)} TV ( P Y n , P t n ) ≤ P ( X t n − 1 ∈ / G n ) + ∫ G n p t n − 1 ( x ) [ 1 − p t n − 1 ( x ) p t n ( T n ( x )) det ∇ T n ( x ) ] + d x . (21)
Lower bounds for the density ratio and Jacobian determinant. For x ∈ G n x\in G_n x ∈ G n , we need a lower bound close to zero for log [ p t n ( T n ( x ) ) / p t n − 1 ( x ) ] + log det ∇ T n ( x ) \log[p_{t_n}(T_n(x))/p_{t_{n-1}}(x)]+\log\det\nabla T_n(x) log [ p t n ( T n ( x )) / p t n − 1 ( x )] + log det ∇ T n ( x ) . One main obstacle is that the score function of the source distribution s ( Z ) s(Z) s ( Z ) , a key quantity in both the density ratio term and the Jacobian determinant term, no longer takes the simple form s ( Z ) = − Z s(Z)=-Z s ( Z ) = − Z as in the Gaussian case. Luckily, with the aid of the Lipschitz property of the score function, we obtain the following two lower bounds:
log p t n ( T n ( x ) ) p t n − 1 ( x ) ≥ − ( t n − t n − 1 ) tr ∇ v t n − 1 ( x ) − 3 L 2 ( t n − t n − 1 t n − 1 ( 1 − t n ) ) 2 E [ ∥ Z − E [ Z ∣ X t n − 1 = x ] ∥ 2 2 ∣ X t n − 1 = x ] , log det ∇ T n ( x ) ≥ + ( t n − t n − 1 ) tr ∇ v t n − 1 ( x ) − 4 3 ( t n − t n − 1 ) 2 ∥ ∇ v t n − 1 ( x ) ∥ F 2 . (22) \begin{aligned}
\log\frac{p_{t_n}(T_n(x))}{p_{t_{n-1}}(x)}
&\geq-(t_n-t_{n-1})\operatorname{tr}\nabla v_{t_{n-1}}(x)\\
&\quad-\frac{3L}{2}\left(\frac{t_n-t_{n-1}}{t_{n-1}(1-t_n)}\right)^2\mathbb{E}\left[\left\|Z-\mathbb{E}[Z\mid X_{t_{n-1}}=x]\right\|_2^2\mid X_{t_{n-1}}=x\right],\\
\log\det\nabla T_n(x)
&\geq+(t_n-t_{n-1})\operatorname{tr}\nabla v_{t_{n-1}}(x)-\frac{4}{3}(t_n-t_{n-1})^2\left\|\nabla v_{t_{n-1}}(x)\right\|_{\mathrm{F}}^2. \tag*{(22)}
\end{aligned} log p t n − 1 ( x ) p t n ( T n ( x )) log det ∇ T n ( x ) ≥ − ( t n − t n − 1 ) tr ∇ v t n − 1 ( x ) − 2 3 L ( t n − 1 ( 1 − t n ) t n − t n − 1 ) 2 E [ ∥ Z − E [ Z ∣ X t n − 1 = x ] ∥ 2 2 ∣ X t n − 1 = x ] , ≥ + ( t n − t n − 1 ) tr ∇ v t n − 1 ( x ) − 3 4 ( t n − t n − 1 ) 2 ∥ ∇ v t n − 1 ( x ) ∥ F 2 . (22)
The first-order contributions cancel, while the remaining cross-covariance coefficient is quadratic in step size t n − t n − 1 t_n-t_{n-1} t n − t n − 1 . To achieve the above, only the second moment of Z Z Z and the Lipschitz property of the source score function are needed here. Summing the second-order terms in (22) across 1 ≤ n ≤ N 1\leq n\leq N 1 ≤ n ≤ N , we have a good control for ∑ n = 1 N TV ( P ^ Y n , P t n ) \sum_{n=1}^{N}\operatorname{TV}(\widehat{P}_{Y_n},P_{t_n}) ∑ n = 1 N TV ( P Y n , P t n ) up to the sum of the discarded probabilities P ( X t n − 1 ∉ G n ) \mathbb{P}(X_{t_{n-1}}\notin G_n) P ( X t n − 1 ∈ / G n ) .
Proof of restricted injectivity. Finally, we give a brief description over the construction of G n G_n G n and the restricted injectivity proof, the key technical novelty of our proof. We also note that this may be of independent interest for future works dealing with non-Gaussian sources, where T n T_n T n is likely not injective over the whole R d \mathbb{R}^d R d . In particular, motivated by (20) , we construct G n G_n G n such that the second term in (20) is well-controlled to not ruin the positive definiteness of ∇ T n \nabla T_n ∇ T n :
G n = { x : E [ ∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 = x ] ≤ A 1 , E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 = x ] ≤ A 2 } . G_n=\left\{x:\mathbb{E}[\|s(Z)\|_2^2\mid X_{t_{n-1}}=x]\leq A_1,\ \mathbb{E}[\|Z-\mathbb{E}[Z]\|_2^2\mid X_{t_{n-1}}=x]\leq A_2\right\}. G n = { x : E [ ∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 = x ] ≤ A 1 , E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 = x ] ≤ A 2 } .
where A 1 ≲ d A_1\lesssim d A 1 ≲ d and A 2 ≳ N 2 / d A_2\gtrsim N^2/d A 2 ≳ N 2 / d up to logarithmic factors. Here, although Z Z Z may be heavy-tailed, its score is sub-Gaussian, implied by the mild Lipschitz property of the score (see Negrea (2022); Altschuler and Chewi [3 ] ; also Lemma 2). Combining sub-Gaussian tail control for s ( Z ) s(Z) s ( Z ) and a Markovian tail control for Z Z Z gives a good control for P ( X t n − 1 ∉ G n ) \mathbb{P}(X_{t_{n-1}}\notin G_n) P ( X t n − 1 ∈ / G n ) .
Although our constructed good set G n G_n G n ensures ∇ T n \nabla T_n ∇ T n (symmetrized) to have lower bounded minimum eigenvalues, T n T_n T n ’s injectivity on G n G_n G n does not follow directly, as G n G_n G n may not be a convex set. Instead, we design a new proof argument by showing that T n ( x ) = T n ( y ) T_n(x)=T_n(y) T n ( x ) = T n ( y ) implies that the posterior mean of the source Z Z Z given X t n − 1 = x X_{t_{n-1}}=x X t n − 1 = x and X t n − 1 = y X_{t_{n-1}}=y X t n − 1 = y must be sufficiently close, making it impossible for x ≠ y x\ne y x = y to happen. Appendix § 9 .§ 6 gives the argument for Lemma 5, the restricted injectivity statement.
Finally, putting all pieces together and summing the TV errors over the grid 1 ≤ n ≤ N 1\le n\le N 1 ≤ n ≤ N , we arrive at our final result:
TV ( P Y N , P t N ) ≲ L TotVar b a s e ( d + log N ) log 2 N N . \operatorname{TV}(P_{Y_N},P_{t_N})\lesssim L\operatorname{TotVar}_{\mathrm{base}}(d+\log N)\frac{\log^2 N}{N}. TV ( P Y N , P t N ) ≲ L TotVar base ( d + log N ) N log 2 N .
Analysis
In this section, we prove Theorem 1 by decomposing the total error into initialization and one-step discretization errors.
Preparation. The following lemma introduces two important properties for the sampling times and its proof is deferred to Section A.2.
Lemma 1. Under Assumption 3, we have
t n − t n − 1 ≤ c s log ( N + 1 ) N + 1 t n − 1 ( 1 − t n − 1 ) , 1 ≤ n ≤ N , ∑ n = 1 N ( t n − t n − 1 t n − 1 ( 1 − t n ) ) r ≤ c s r log r ( N + 1 ) ( N + 1 ) r − 1 , r ≥ 1. (23) \begin{aligned}
t_n-t_{n-1} &\le\frac{c_s\log(N+1)}{N+1}t_{n-1}(1-t_{n-1}), \qquad1\le n\le N,\\
\sum_{n=1}^{N}\left(\frac{t_n-t_{n-1}}{t_{n-1}(1-t_n)}\right)^r &\le c_s^r\frac{\log^r(N+1)}{(N+1)^{r-1}}, \qquad r\ge1. \tag*{(23)}
\end{aligned} t n − t n − 1 n = 1 ∑ N ( t n − 1 ( 1 − t n ) t n − t n − 1 ) r ≤ N + 1 c s log ( N + 1 ) t n − 1 ( 1 − t n − 1 ) , 1 ≤ n ≤ N , ≤ c s r ( N + 1 ) r − 1 log r ( N + 1 ) , r ≥ 1. (23)
The next lemma provides some key properties of the score function, with its proof postponed to Section A.7. The sub-Gaussian property of Lipschitz score functions has also been introduced in Negrea (2022); Altschuler and Chewi [3 ] .
Lemma 2 (Score and second-moment estimates). Under Assumption 1, for every v ∈ R d v\in\mathbb{R}^d v ∈ R d ,
E [ e v ⊤ s ( Z ) ] ≤ e L ∥ v ∥ 2 / 2 , E [ s ( Z ) ] = 0 , E [ s ( Z ) s ( Z ) ⊤ ] ⪯ L I d . (24) \mathbb{E}\left[e^{v^\top s(Z)}\right]\le e^{L\lVert v\rVert^2/2},\qquad\mathbb{E}[s(Z)]=0,\qquad\mathbb{E}[s(Z)s(Z)^\top]\preceq L I_d. \tag*{(24)} E [ e v ⊤ s ( Z ) ] ≤ e L ∥ v ∥ 2 /2 , E [ s ( Z )] = 0 , E [ s ( Z ) s ( Z ) ⊤ ] ⪯ L I d . (24) Moreover, we have
E [ e ∥ s ( Z ) ∥ 2 / ( 4 L ) ] ≤ 2 d / 2 , (25) \mathbb{E}\left[e^{\lVert s(Z)\rVert^2/(4L)}\right]\le2^{d/2}, \tag*{(25)} E [ e ∥ s ( Z ) ∥ 2 / ( 4 L ) ] ≤ 2 d /2 , (25) E [ ( Z − E [ Z ] ) s ( Z ) ⊤ ] = − I d , (26) \mathbb{E}[(Z-\mathbb{E}[Z])s(Z)^\top]=-I_d, \tag*{(26)} E [( Z − E [ Z ]) s ( Z ) ⊤ ] = − I d , (26) L E [ ∥ Z − E [ Z ] ∥ 2 ] ≥ d . (27) L\mathbb{E}[\lVert Z-\mathbb{E}[Z]\rVert^2]\ge d. \tag*{(27)} L E [∥ Z − E [ Z ] ∥ 2 ] ≥ d . (27)
In addition, the lemma below shows that the initialization error can be well-controlled.
Lemma 3 (Initialization error). Suppose that Assumptions 1, 2, and 3 hold. Then
KL ( P t 0 ∥ P Y 0 ) ≤ L t 0 2 2 ( 1 − t 0 ) 2 E [ ∥ X ⋆ ∥ 2 ] , (28) \operatorname{KL}(P_{t_0}\mathbin\Vert P_{Y_0})\le\frac{L t_0^2}{2(1-t_0)^2}\mathbb{E}\left[\lVert X_\star\rVert^2\right], \tag*{(28)} KL ( P t 0 ∥ P Y 0 ) ≤ 2 ( 1 − t 0 ) 2 L t 0 2 E [ ∥ X ⋆ ∥ 2 ] , (28) TV ( P Y 0 , P t 0 ) ≤ t 0 2 ( 1 − t 0 ) L E [ ∥ X ⋆ ∥ 2 ] ≤ L E [ ∥ Z − E [ Z ] ∥ 2 ] ( N + 1 ) 2 . (29) \operatorname{TV}(P_{Y_0},P_{t_0})\le\frac{t_0}{2(1-t_0)}\sqrt{L\mathbb{E}\left[\lVert X_\star\rVert^2\right]}\le\frac{\sqrt{L\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert^2\right]}}{(N+1)^2}. \tag*{(29)} TV ( P Y 0 , P t 0 ) ≤ 2 ( 1 − t 0 ) t 0 L E [ ∥ X ⋆ ∥ 2 ] ≤ ( N + 1 ) 2 L E [ ∥ Z − E [ Z ] ∥ 2 ] . (29)
The proof of Lemma 3 is given in Appendix § 9 .§ 3 .
Step 1: Decomposing the total variation error. We let
Y ^ n = X t n − 1 + ( t n − t n − 1 ) v t n − 1 ( X t n − 1 ) = T n ( X t n − 1 ) , P ^ n = P Y ^ n . (30) \widehat{Y}_n=X_{t_{n-1}}+(t_n-t_{n-1})v_{t_{n-1}}(X_{t_{n-1}})=T_n(X_{t_{n-1}}),\qquad\widehat{P}_n=P_{\widehat{Y}_n}. \tag*{(30)} Y n = X t n − 1 + ( t n − t n − 1 ) v t n − 1 ( X t n − 1 ) = T n ( X t n − 1 ) , P n = P Y n . (30)
Since total variation contracts under a measurable map, one has
TV ( P Y n , P t n ) ≤ TV ( P Y n , P ^ n ) + TV ( P ^ n , P t n ) ≤ TV ( P Y n − 1 , P t n − 1 ) + TV ( P ^ n , P t n ) . \operatorname{TV}(P_{Y_n},P_{t_n}) \le\operatorname{TV}(P_{Y_n},\widehat{P}_n) + \operatorname{TV}(\widehat{P}_n,P_{t_n})
\le\operatorname{TV}(P_{Y_{n-1}},P_{t_{n-1}}) + \operatorname{TV}(\widehat{P}_n,P_{t_n}). TV ( P Y n , P t n ) ≤ TV ( P Y n , P n ) + TV ( P n , P t n ) ≤ TV ( P Y n − 1 , P t n − 1 ) + TV ( P n , P t n ) .
Repeatedly applying the previous inequality leads to
TV ( P Y N , P t N ) ≤ TV ( P Y 0 , P t 0 ) + ∑ n = 1 N TV ( P ^ n , P t n ) . (31) \operatorname{TV}(P_{Y_N},P_{t_N}) \le\operatorname{TV}(P_{Y_0},P_{t_0}) + \sum_{n=1}^{N}\operatorname{TV}(\widehat{P}_n,P_{t_n}). \tag*{(31)} TV ( P Y N , P t N ) ≤ TV ( P Y 0 , P t 0 ) + n = 1 ∑ N TV ( P n , P t n ) . (31)
To bound TV ( P Y N , P t N ) \operatorname{TV}(P_{Y_N},P_{t_N}) TV ( P Y N , P t N ) , it suffices to control TV ( P ^ n , P t n ) \operatorname{TV}(\widehat{P}_n,P_{t_n}) TV ( P n , P t n ) for each n n n . We also define the following “good” event:
G n = { x : E [ ∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 = x ] ≤ 4 L { d + 12 log ( N + 1 ) } , G_n=\left\{x:\mathbb{E}[\lVert s(Z)\rVert_2^2\mid X_{t_{n-1}}=x]\le4L\{d+12\log(N+1)\},
\right. G n = { x : E [∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 = x ] ≤ 4 L { d + 12 log ( N + 1 )} ,
E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 = x ] ≤ t n − 1 2 ( 1 − t n ) 2 256 L ( t n − t n − 1 ) 2 { d + 12 log ( N + 1 ) } } . (32) \left.
\mathbb{E}[\lVert Z-\mathbb{E}[Z]\rVert_2^2\mid X_{t_{n-1}}=x]\le\frac{t_{n-1}^2(1-t_n)^2}{256L(t_n-t_{n-1})^2\{d+12\log(N+1)\}}
\right\}. \tag*{(32)} E [∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 = x ] ≤ 256 L ( t n − t n − 1 ) 2 { d + 12 log ( N + 1 )} t n − 1 2 ( 1 − t n ) 2 } . (32)
The following two lemmas give the regularity and injectivity needed for the one-step comparison.
Lemma 4 (Regularity needed for the one-step comparison). Under Assumption 1, for every fixed t ∈ ( 0 , 1 ) t\in(0,1) t ∈ ( 0 , 1 ) , p t p_t p t is strictly positive and C 1 C^1 C 1 . The posterior mean x ↦ E [ X ⋆ ∣ X t = x ] x\mapsto\mathbb{E}[X_\star\mid X_t=x] x ↦ E [ X ⋆ ∣ X t = x ] is finite and continuously differentiable on R d \mathbb{R}^d R d . Moreover, for every x ∈ R d x\in\mathbb{R}^d x ∈ R d ,
∫ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) P d a t a ( d x ⋆ ) = ∇ x ∫ p X ⋆ ∣ X t ( x ⋆ ∣ x ) P d a t a ( d x ⋆ ) = ∇ x 1 = 0. (33) \int\nabla_xp_{X_\star\mid X_t}(x_\star\mid x)P_{\mathrm{data}}(\mathrm{d}x_\star)
=\nabla_x\int p_{X_\star\mid X_t}(x_\star\mid x)P_{\mathrm{data}}(\mathrm{d}x_\star)
=\nabla_x1=0. \tag*{(33)} ∫ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) P data ( d x ⋆ ) = ∇ x ∫ p X ⋆ ∣ X t ( x ⋆ ∣ x ) P data ( d x ⋆ ) = ∇ x 1 = 0. (33) and
∇ x E [ X ⋆ ∣ X t = x ] = ∫ x ⋆ [ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) ] T P d a t a ( d x ⋆ ) = ∫ ( x ⋆ − x ) [ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) ] T P d a t a ( d x ⋆ ) . (34) \nabla_x\mathbb{E}[X_\star\mid X_t=x]
=\int x_\star[\nabla_xp_{X_\star\mid X_t}(x_\star\mid x)]^{\mathsf{T}}P_{\mathrm{data}}(\mathrm{d}x_\star)
=\int(x_\star-x)[\nabla_xp_{X_\star\mid X_t}(x_\star\mid x)]^{\mathsf{T}}P_{\mathrm{data}}(\mathrm{d}x_\star). \tag*{(34)} ∇ x E [ X ⋆ ∣ X t = x ] = ∫ x ⋆ [ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) ] T P data ( d x ⋆ ) = ∫ ( x ⋆ − x ) [ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) ] T P data ( d x ⋆ ) . (34) In particular,
v t ( x ) = E [ X ⋆ ∣ X t = x ] − x 1 − t (35) v_t(x)=\frac{\mathbb{E}[X_\star\mid X_t=x]-x}{1-t} \tag*{(35)} v t ( x ) = 1 − t E [ X ⋆ ∣ X t = x ] − x (35) is C 1 C^1 C 1 . The two functions
x ⟼ E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t = x ] , x ⟼ E [ ∥ s ( Z ) ∥ 2 2 ∣ X t = x ] x\longmapsto\mathbb{E}[\lVert Z-\mathbb{E}[Z]\rVert_2^2\mid X_t=x],
\qquad
x\longmapsto\mathbb{E}[\lVert s(Z)\rVert_2^2\mid X_t=x] x ⟼ E [∥ Z − E [ Z ] ∥ 2 2 ∣ X t = x ] , x ⟼ E [∥ s ( Z ) ∥ 2 2 ∣ X t = x ] are finite and continuous. Consequently, T n T_n T n is C 1 C^1 C 1 and G n G_n G n is closed.
Lemma 5 (Injectivity on the restricted set). Under Assumptions 1 and 3, for every 1 ≤ n ≤ N 1\le n\le N 1 ≤ n ≤ N , the map T n T_n T n is injective on G n G_n G n . Moreover, for every x ∈ G n x\in G_n x ∈ G n ,
( t n − t n − 1 ) ∥ ∇ x v t n − 1 ( x ) ∥ o p ≤ 5 8 , det ∇ x T n ( x ) > 0. (36) (t_n-t_{n-1})\lVert\nabla_xv_{t_{n-1}}(x)\rVert_{\mathrm{op}}\le\frac{5}{8},
\qquad
\det\nabla_xT_n(x)>0. \tag*{(36)} ( t n − t n − 1 ) ∥ ∇ x v t n − 1 ( x ) ∥ op ≤ 8 5 , det ∇ x T n ( x ) > 0. (36)
The proofs of Lemmas 4 and 5 are postponed to Sections A.4 and A.6, respectively.
For any event A ⊆ R d A\subseteq\mathbb{R}^d A ⊆ R d , Lemmas 4 and 5 allow us to change variables on G n ∩ T n − 1 ( A ) G_n\cap T_n^{-1}(A) G n ∩ T n − 1 ( A ) and obtain
∫ A p t n ( y ) d y ≥ ∫ A ∩ T n ( G n ) p t n ( y ) d y = ∫ G n ∩ T n − 1 ( A ) p t n ( T n ( x ) ) det ∇ T n ( x ) d x . (37) \int_A p_{t_n}(y)\,\mathrm{d}y
\ge\int_{A\cap T_n(G_n)}p_{t_n}(y)\,\mathrm{d}y
=\int_{G_n\cap T_n^{-1}(A)}p_{t_n}(T_n(x))\det\nabla T_n(x)\,\mathrm{d}x. \tag*{(37)} ∫ A p t n ( y ) d y ≥ ∫ A ∩ T n ( G n ) p t n ( y ) d y = ∫ G n ∩ T n − 1 ( A ) p t n ( T n ( x )) det ∇ T n ( x ) d x . (37)
Since Y ^ n = T n ( X t n − 1 ) \widehat{Y}_n=T_n(X_{t_{n-1}}) Y n = T n ( X t n − 1 ) , we know that
P ( Y ^ n ∈ A ) − ∫ A p t n ( y ) d y ≤ P ( X t n − 1 ∉ G n ) + P ( Y ^ n ∈ A , X t n − 1 ∈ G n ) − ∫ A p t n ( y ) d y ≤ P ( X t n − 1 ∉ G n ) + ∫ G n ∩ T n − 1 ( A ) [ p t n − 1 ( x ) − p t n ( T n ( x ) ) det ∇ T n ( x ) ] d x ≤ P ( X t n − 1 ∉ G n ) + ∫ G n [ p t n − 1 ( x ) − p t n ( T n ( x ) ) det ∇ T n ( x ) ] + d x . \begin{aligned}
\mathbb{P}(\widehat{Y}_n\in A)-\int_Ap_{t_n}(y)\,\mathrm{d}y
\le\mathbb{P}(X_{t_{n-1}}\notin G_n)+\mathbb{P}(\widehat{Y}_n\in A,X_{t_{n-1}}\in G_n)-\int_Ap_{t_n}(y)\,\mathrm{d}y \\
&\le\mathbb{P}(X_{t_{n-1}}\notin G_n)+\int_{G_n\cap T_n^{-1}(A)}\left[p_{t_{n-1}}(x)-p_{t_n}(T_n(x))\det\nabla T_n(x)\right]\,dx \\
&\le\mathbb{P}(X_{t_{n-1}}\notin G_n)+\int_{G_n}\left[p_{t_{n-1}}(x)-p_{t_n}(T_n(x))\det\nabla T_n(x)\right]_+\,dx.
\end{aligned} P ( Y n ∈ A ) − ∫ A p t n ( y ) d y ≤ P ( X t n − 1 ∈ / G n ) + P ( Y n ∈ A , X t n − 1 ∈ G n ) − ∫ A p t n ( y ) d y ≤ P ( X t n − 1 ∈ / G n ) + ∫ G n ∩ T n − 1 ( A ) [ p t n − 1 ( x ) − p t n ( T n ( x )) det ∇ T n ( x ) ] d x ≤ P ( X t n − 1 ∈ / G n ) + ∫ G n [ p t n − 1 ( x ) − p t n ( T n ( x )) det ∇ T n ( x ) ] + d x .
Taking the supremum over A A A yields
TV ( P ^ n , P t n ) ≤ P ( X t n − 1 ∉ G n ) + ∫ G n p t n − 1 ( x ) [ 1 − p t n ( T n ( x ) ) det ∇ T n ( x ) p t n − 1 ( x ) ] + d x . (38) \operatorname{TV}(\widehat{P}_n,P_{t_n})\le\mathbb{P}(X_{t_{n-1}}\notin G_n)+\int_{G_n}p_{t_{n-1}}(x)\left[1-\frac{p_{t_n}(T_n(x))\det\nabla T_n(x)}{p_{t_{n-1}}(x)}\right]_+\,dx. \tag*{(38)} TV ( P n , P t n ) ≤ P ( X t n − 1 ∈ / G n ) + ∫ G n p t n − 1 ( x ) [ 1 − p t n − 1 ( x ) p t n ( T n ( x )) det ∇ T n ( x ) ] + d x . (38)
In the following, we will provide a lower bound for p t n ( T n ( x ) ) det ∇ T n ( x ) p t n − 1 ( x ) \frac{p_{t_n}(T_n(x))\det\nabla T_n(x)}{p_{t_{n-1}}(x)} p t n − 1 ( x ) p t n ( T n ( x )) d e t ∇ T n ( x ) .
Expanding the Jacobian determinant
Step 2: Expanding the Jacobian determinant. First, we deal with det ∇ x T n ( x ) \det\nabla_xT_n(x) det ∇ x T n ( x ) . Recall that
∇ x T n ( x ) = I d + ( t n − t n − 1 ) ∇ x v t n − 1 ( x ) . (39) \nabla_xT_n(x)=I_d+(t_n-t_{n-1})\nabla_xv_{t_{n-1}}(x). \tag*{(39)} ∇ x T n ( x ) = I d + ( t n − t n − 1 ) ∇ x v t n − 1 ( x ) . (39)
In view of (36) in Lemma 5 , we have ∥ ∇ T n ( x ) − I d ∥ o p ≤ 5 / 8 \|\nabla T_n(x)-I_d\|_{\mathrm{op}}\le5/8 ∥∇ T n ( x ) − I d ∥ op ≤ 5/8 and det ∇ T n ( x ) > 0 \det\nabla T_n(x)>0 det ∇ T n ( x ) > 0 for every x ∈ G n x\in G_n x ∈ G n . For any real matrix E E E with ∥ E ∥ o p ≤ 5 / 8 \|E\|_{\mathrm{op}}\le5/8 ∥ E ∥ op ≤ 5/8 , the convergent logarithmic series tells us that
∣ log det ( I d + E ) − Tr E ∣ ≤ ∑ ℓ = 2 ∞ ∣ Tr ( E ℓ ) ∣ ℓ ≤ ∑ ℓ = 2 ∞ ∣ ⟨ E ⊤ , E ℓ − 1 ⟩ ∣ ℓ ≤ ∑ ℓ = 2 ∞ ∥ E ∥ F ∥ E ℓ − 1 ∥ F ℓ ≤ ∥ E ∥ F 2 ∑ ℓ = 2 ∞ ∥ E ∥ o p ℓ − 2 ℓ ≤ ∥ E ∥ F 2 2 ( 1 − ∥ E ∥ o p ) ≤ 4 3 ∥ E ∥ F 2 . (40) \left|\log\det(I_d+E)-\operatorname{Tr}E\right|
\le\sum_{\ell=2}^{\infty}\frac{|\operatorname{Tr}(E^\ell)|}{\ell}
\le\sum_{\ell=2}^{\infty}\frac{|\langle E^\top,E^{\ell-1}\rangle|}{\ell}
\le\sum_{\ell=2}^{\infty}\frac{\|E\|_{\mathrm{F}}\|E^{\ell-1}\|_{\mathrm{F}}}{\ell}
\le\|E\|_{\mathrm{F}}^2\sum_{\ell=2}^{\infty}\frac{\|E\|_{\mathrm{op}}^{\ell-2}}{\ell}
\le\frac{\|E\|_{\mathrm{F}}^2}{2(1-\|E\|_{\mathrm{op}})}
\le\frac{4}{3}\|E\|_{\mathrm{F}}^2. \tag*{(40)} ∣ log det ( I d + E ) − Tr E ∣ ≤ ℓ = 2 ∑ ∞ ℓ ∣ Tr ( E ℓ ) ∣ ≤ ℓ = 2 ∑ ∞ ℓ ∣ ⟨ E ⊤ , E ℓ − 1 ⟩ ∣ ≤ ℓ = 2 ∑ ∞ ℓ ∥ E ∥ F ∥ E ℓ − 1 ∥ F ≤ ∥ E ∥ F 2 ℓ = 2 ∑ ∞ ℓ ∥ E ∥ op ℓ − 2 ≤ 2 ( 1 − ∥ E ∥ op ) ∥ E ∥ F 2 ≤ 3 4 ∥ E ∥ F 2 . (40)
For any x ∈ G n x\in G_n x ∈ G n , setting E = ∇ T n ( x ) − I d E=\nabla T_n(x)-I_d E = ∇ T n ( x ) − I d in (40) and using (39) yield
∣ log ( det ∇ x T n ( x ) ) − ( t n − t n − 1 ) Tr ( ∇ x v t n − 1 ( x ) ) ∣ ≤ 4 3 ( t n − t n − 1 ) 2 ∥ ∇ x v t n − 1 ( x ) ∥ F 2 . (41) \left|\log\left(\det\nabla_xT_n(x)\right)-(t_n-t_{n-1})\operatorname{Tr}\left(\nabla_xv_{t_{n-1}}(x)\right)\right|
\le\frac{4}{3}(t_n-t_{n-1})^2\left\|\nabla_xv_{t_{n-1}}(x)\right\|_{\mathrm{F}}^2. \tag*{(41)} ∣ log ( det ∇ x T n ( x ) ) − ( t n − t n − 1 ) Tr ( ∇ x v t n − 1 ( x ) ) ∣ ≤ 3 4 ( t n − t n − 1 ) 2 ∥ ∇ x v t n − 1 ( x ) ∥ F 2 . (41)
Step 3: Expanding the density ratio. For any fixed n n n , let y n − 1 y_{n-1} y n − 1 denote a fixed point and y n = T n ( y n − 1 ) y_n=T_n(y_{n-1}) y n = T n ( y n − 1 ) , where
T n ( x ) = x + ( t n − t n − 1 ) v t n − 1 ( x ) . T_n(x)=x+(t_n-t_{n-1})v_{t_{n-1}}(x). T n ( x ) = x + ( t n − t n − 1 ) v t n − 1 ( x ) .
Then we know the density p t n ( y ) p_{t_n}(y) p t n ( y ) at y = y n y=y_n y = y n is
p t n ( y n ) = 1 ( 1 − t n ) d ∫ p b a s e ( y n − t n x ⋆ 1 − t n ) P d a t a ( d x ⋆ ) = ( 1 − t n − 1 1 − t n ) d p t n − 1 ( y n − 1 ) ∫ p X ⋆ ∣ X t n − 1 ( x ⋆ ∣ y n − 1 ) p b a s e ( ( y n − t n x ⋆ ) / ( 1 − t n ) ) p b a s e ( ( y n − 1 − t n − 1 x ⋆ ) / ( 1 − t n − 1 ) ) P d a t a ( d x ⋆ ) , (42) \begin{aligned}
p_{t_n}(y_n)=\frac{1}{(1-t_n)^d}\int p_{\mathrm{base}}\left(\frac{y_n-t_nx_\star}{1-t_n}\right)P_{\mathrm{data}}(dx_\star) \\
&=\left(\frac{1-t_{n-1}}{1-t_n}\right)^d p_{t_{n-1}}(y_{n-1})\int p_{X_\star\mid X_{t_{n-1}}}(x_\star\mid y_{n-1})\frac{p_{\mathrm{base}}\left((y_n-t_nx_\star)/(1-t_n)\right)}{p_{\mathrm{base}}\left((y_{n-1}-t_{n-1}x_\star)/(1-t_{n-1})\right)}P_{\mathrm{data}}(dx_\star), \tag*{(42)}
\end{aligned} p t n ( y n ) = ( 1 − t n ) d 1 ∫ p base ( 1 − t n y n − t n x ⋆ ) P data ( d x ⋆ ) = ( 1 − t n 1 − t n − 1 ) d p t n − 1 ( y n − 1 ) ∫ p X ⋆ ∣ X t n − 1 ( x ⋆ ∣ y n − 1 ) p base ( ( y n − 1 − t n − 1 x ⋆ ) / ( 1 − t n − 1 ) ) p base ( ( y n − t n x ⋆ ) / ( 1 − t n ) ) P data ( d x ⋆ ) , (42)
where the last equation holds due to Bayes’ formula
p X ⋆ ∣ X t n − 1 ( x ⋆ ∣ y n − 1 ) = p b a s e ( y n − 1 − t n − 1 x ⋆ 1 − t n − 1 ) ( 1 − t n − 1 ) d p t n − 1 ( y n − 1 ) . p_{X_\star\mid X_{t_{n-1}}}(x_\star\mid y_{n-1})=
\frac{p_{\mathrm{base}}\left(\frac{y_{n-1}-t_{n-1}x_\star}{1-t_{n-1}}\right)}
{(1-t_{n-1})^d p_{t_{n-1}}(y_{n-1})}. p X ⋆ ∣ X t n − 1 ( x ⋆ ∣ y n − 1 ) = ( 1 − t n − 1 ) d p t n − 1 ( y n − 1 ) p base ( 1 − t n − 1 y n − 1 − t n − 1 x ⋆ ) .
We collect the identities and bounds for the velocity Jacobian used below.
Lemma 6 (Velocity Jacobian). Under Assumption 1, for every t ∈ ( 0 , 1 ) t\in(0,1) t ∈ ( 0 , 1 ) and x ∈ R d x\in\mathbb{R}^d x ∈ R d , conditional on X t = x X_t=x X t = x , one has
t ( X ⋆ − E [ X ⋆ ∣ X t = x ] ) = − ( 1 − t ) ( Z − E [ Z ∣ X t = x ] ) . (43) t\left(X_\star-\mathbb{E}[X_\star\mid X_t=x]\right)=-(1-t)\left(Z-\mathbb{E}[Z\mid X_t=x]\right). \tag*{(43)} t ( X ⋆ − E [ X ⋆ ∣ X t = x ] ) = − ( 1 − t ) ( Z − E [ Z ∣ X t = x ] ) . (43)
Moreover,
∇ x v t ( x ) = − 1 1 − t I − 1 t ( 1 − t ) E [ ( Z − E [ Z ∣ X t = x ] ) s ( Z ) T ∣ X t = x ] . (44) \nabla_x v_t(x)=-\frac{1}{1-t}I-\frac{1}{t(1-t)}\mathbb{E}\left[(Z-\mathbb{E}[Z\mid X_t=x])s(Z)^{\mathsf{T}}\mid X_t=x\right]. \tag*{(44)} ∇ x v t ( x ) = − 1 − t 1 I − t ( 1 − t ) 1 E [ ( Z − E [ Z ∣ X t = x ]) s ( Z ) T ∣ X t = x ] . (44)
In addition,
∥ ∇ v t ( x ) + I d 1 − t ∥ F 2 ≤ E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t = x ] E [ ∥ s ( Z ) ∥ 2 2 ∣ X t = x ] t 2 ( 1 − t ) 2 , (45) \left\|\nabla v_t(x)+\frac{I_d}{1-t}\right\|_{\mathrm{F}}^2\leq\frac{\mathbb{E}\left[\|Z-\mathbb{E}[Z]\|_2^2\mid X_t=x\right]\mathbb{E}\left[\|s(Z)\|_2^2\mid X_t=x\right]}{t^2(1-t)^2}, \tag*{(45)} ∇ v t ( x ) + 1 − t I d F 2 ≤ t 2 ( 1 − t ) 2 E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t = x ] E [ ∥ s ( Z ) ∥ 2 2 ∣ X t = x ] , (45)
∥ ∇ v t ( x ) ∥ F 2 ≤ 2 d ( 1 − t ) 2 + 2 E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t = x ] E [ ∥ s ( Z ) ∥ 2 2 ∣ X t = x ] t 2 ( 1 − t ) 2 . (46) \|\nabla v_t(x)\|_{\mathrm{F}}^2\leq\frac{2d}{(1-t)^2}+\frac{2\mathbb{E}\left[\|Z-\mathbb{E}[Z]\|_2^2\mid X_t=x\right]\mathbb{E}\left[\|s(Z)\|_2^2\mid X_t=x\right]}{t^2(1-t)^2}. \tag*{(46)} ∥∇ v t ( x ) ∥ F 2 ≤ ( 1 − t ) 2 2 d + t 2 ( 1 − t ) 2 2 E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t = x ] E [ ∥ s ( Z ) ∥ 2 2 ∣ X t = x ] . (46)
The proof is given in Appendix A.5.
For the integration variable x ⋆ x_\star x ⋆ , we let z = ( y n − 1 − t n − 1 x ⋆ ) / ( 1 − t n − 1 ) z=(y_{n-1}-t_{n-1}x_\star)/(1-t_{n-1}) z = ( y n − 1 − t n − 1 x ⋆ ) / ( 1 − t n − 1 ) . Applying (43) at t = t n − 1 t=t_{n-1} t = t n − 1 and x = y n − 1 x=y_{n-1} x = y n − 1 gives
y n − t n x ⋆ 1 − t n − y n − 1 − t n − 1 x ⋆ 1 − t n − 1 = − t n − t n − 1 ( 1 − t n ) ( 1 − t n − 1 ) x ⋆ + y n − 1 1 − t n + t n − t n − 1 1 − t n E [ X ⋆ − y n − 1 1 − t n − 1 ∣ X t n − 1 = y n − 1 ] \begin{aligned}
\frac{y_n-t_nx_\star}{1-t_n}-\frac{y_{n-1}-t_{n-1}x_\star}{1-t_{n-1}} \\
&=-\frac{t_n-t_{n-1}}{(1-t_n)(1-t_{n-1})}x_\star+\frac{y_{n-1}}{1-t_n}+\frac{t_n-t_{n-1}}{1-t_n}\mathbb{E}\left[\left.\frac{X_\star-y_{n-1}}{1-t_{n-1}}\right|X_{t_{n-1}}=y_{n-1}\right]
\end{aligned} 1 − t n y n − t n x ⋆ − 1 − t n − 1 y n − 1 − t n − 1 x ⋆ = − ( 1 − t n ) ( 1 − t n − 1 ) t n − t n − 1 x ⋆ + 1 − t n y n − 1 + 1 − t n t n − t n − 1 E [ 1 − t n − 1 X ⋆ − y n − 1 X t n − 1 = y n − 1 ]
− y n − 1 1 − t n − 1 = − t n − t n − 1 ( 1 − t n ) ( 1 − t n − 1 ) ( x ⋆ − E [ X ⋆ ∣ X t n − 1 = y n − 1 ] ) = t n − t n − 1 t n − 1 ( 1 − t n ) ( z − E [ Z ∣ X t n − 1 = y n − 1 ] ) . (47) \begin{aligned}
\quad-\frac{y_{n-1}}{1-t_{n-1}} \\
&=-\frac{t_n-t_{n-1}}{(1-t_n)(1-t_{n-1})}\left(x_\star-\mathbb{E}[X_\star\mid X_{t_{n-1}}=y_{n-1}]\right) \\
&=\frac{t_n-t_{n-1}}{t_{n-1}(1-t_n)}\left(z-\mathbb{E}[Z\mid X_{t_{n-1}}=y_{n-1}]\right). \tag*{(47)}
\end{aligned} − 1 − t n − 1 y n − 1 = − ( 1 − t n ) ( 1 − t n − 1 ) t n − t n − 1 ( x ⋆ − E [ X ⋆ ∣ X t n − 1 = y n − 1 ] ) = t n − 1 ( 1 − t n ) t n − t n − 1 ( z − E [ Z ∣ X t n − 1 = y n − 1 ] ) . (47)
Since s ( ⋅ ) s(\cdot) s ( ⋅ ) is L L L -Lipschitz, by Taylor’s expansion, we have
log p b a s e ( y n − t n x ⋆ 1 − t n ) − log p b a s e ( y n − 1 − t n − 1 x ⋆ 1 − t n − 1 ) = − t n − t n − 1 ( 1 − t n ) ( 1 − t n − 1 ) s ( z ) T ( x ⋆ − E [ X ⋆ ∣ X t n − 1 = y n − 1 ] ) + r n ( 2 ) ( x ⋆ ) , (48) \begin{aligned}
\log p_{\mathrm{base}}\left(\frac{y_n-t_nx_\star}{1-t_n}\right)-\log p_{\mathrm{base}}\left(\frac{y_{n-1}-t_{n-1}x_\star}{1-t_{n-1}}\right) \\
&=-\frac{t_n-t_{n-1}}{(1-t_n)(1-t_{n-1})}s(z)^{\mathsf{T}}\left(x_\star-\mathbb{E}[X_\star\mid X_{t_{n-1}}=y_{n-1}]\right)+r_n^{(2)}(x_\star), \tag*{(48)}
\end{aligned} log p base ( 1 − t n y n − t n x ⋆ ) − log p base ( 1 − t n − 1 y n − 1 − t n − 1 x ⋆ ) = − ( 1 − t n ) ( 1 − t n − 1 ) t n − t n − 1 s ( z ) T ( x ⋆ − E [ X ⋆ ∣ X t n − 1 = y n − 1 ] ) + r n ( 2 ) ( x ⋆ ) , (48)
where the remainder term satisfies
∣ r n ( 2 ) ( x ⋆ ) ∣ ≤ L ( t n − t n − 1 ) 2 2 t n − 1 2 ( 1 − t n ) 2 ∥ z − E [ Z ∣ X t n − 1 = y n − 1 ] ∥ 2 2 . (49) \left|r_n^{(2)}(x_\star)\right|\leq\frac{L(t_n-t_{n-1})^2}{2t_{n-1}^2(1-t_n)^2}\left\|z-\mathbb{E}[Z\mid X_{t_{n-1}}=y_{n-1}]\right\|_2^2. \tag*{(49)} r n ( 2 ) ( x ⋆ ) ≤ 2 t n − 1 2 ( 1 − t n ) 2 L ( t n − t n − 1 ) 2 ∥ z − E [ Z ∣ X t n − 1 = y n − 1 ] ∥ 2 2 . (49)
We define
ξ n : = s ( Z ) T ( X ⋆ − E [ X ⋆ ∣ X t n − 1 = y n − 1 ] ) − E [ s ( Z ) T ( X ⋆ − E [ X ⋆ ∣ X t n − 1 = y n − 1 ] ) ∣ X t n − 1 = y n − 1 ] . (50) \xi_n:=s(Z)^{\mathsf{T}}\left(X_\star-\mathbb{E}[X_\star\mid X_{t_{n-1}}=y_{n-1}]\right)-\mathbb{E}\left[\left.s(Z)^{\mathsf{T}}\left(X_\star-\mathbb{E}[X_\star\mid X_{t_{n-1}}=y_{n-1}]\right)\right|X_{t_{n-1}}=y_{n-1}\right]. \tag*{(50)} ξ n := s ( Z ) T ( X ⋆ − E [ X ⋆ ∣ X t n − 1 = y n − 1 ] ) − E [ s ( Z ) T ( X ⋆ − E [ X ⋆ ∣ X t n − 1 = y n − 1 ] ) X t n − 1 = y n − 1 ] . (50)
Then
E [ ξ n ∣ X t n − 1 = y n − 1 ] = 0. (51) \mathbb{E}[\xi_n\mid X_{t_{n-1}}=y_{n-1}]=0. \tag*{(51)} E [ ξ n ∣ X t n − 1 = y n − 1 ] = 0. (51)
Combining (42) , (43) , (48) and (50) , one arrives at
log p t n ( y n ) p t n − 1 ( y n − 1 ) = d log ( 1 − t n − 1 1 − t n ) − t n − t n − 1 ( 1 − t n ) ( 1 − t n − 1 ) E [ s ( Z ) T ( X ⋆ − E [ X ⋆ ∣ X t n − 1 = y n − 1 ] ) ∣ X t n − 1 = y n − 1 ] \begin{aligned}
\log\frac{p_{t_n}(y_n)}{p_{t_{n-1}}(y_{n-1})} \\
&=d\log\left(\frac{1-t_{n-1}}{1-t_n}\right)-\frac{t_n-t_{n-1}}{(1-t_n)(1-t_{n-1})}\mathbb{E}\left[\left.s(Z)^{\mathsf{T}}\left(X_\star-\mathbb{E}[X_\star\mid X_{t_{n-1}}=y_{n-1}]\right)\right|X_{t_{n-1}}=y_{n-1}\right]
\end{aligned} log p t n − 1 ( y n − 1 ) p t n ( y n ) = d log ( 1 − t n 1 − t n − 1 ) − ( 1 − t n ) ( 1 − t n − 1 ) t n − t n − 1 E [ s ( Z ) T ( X ⋆ − E [ X ⋆ ∣ X t n − 1 = y n − 1 ] ) X t n − 1 = y n − 1 ]
+ log E [ exp ( − t n − t n − 1 ( 1 − t n ) ( 1 − t n − 1 ) ξ n + r n ( 2 ) ( X ⋆ ) ) ∣ X t n − 1 = y n − 1 ] = d log ( 1 − t n − 1 1 − t n ) + t n − t n − 1 ( 1 − t n ) t n − 1 E [ s ( Z ) T ( Z − E [ Z ∣ X t n − 1 = y n − 1 ] ) ∣ X t n − 1 = y n − 1 ] \begin{aligned}
\quad+\log\mathbb{E}\left[\left.\exp\left(-\frac{t_n-t_{n-1}}{(1-t_n)(1-t_{n-1})}\xi_n+r_n^{(2)}(X_\star)\right)\right|X_{t_{n-1}}=y_{n-1}\right] \\
&=d\log\left(\frac{1-t_{n-1}}{1-t_n}\right)+\frac{t_n-t_{n-1}}{(1-t_n)t_{n-1}}\mathbb{E}\left[\left.s(Z)^{\mathsf{T}}\left(Z-\mathbb{E}[Z\mid X_{t_{n-1}}=y_{n-1}]\right)\right|X_{t_{n-1}}=y_{n-1}\right]
\end{aligned} + log E [ exp ( − ( 1 − t n ) ( 1 − t n − 1 ) t n − t n − 1 ξ n + r n ( 2 ) ( X ⋆ ) ) X t n − 1 = y n − 1 ] = d log ( 1 − t n 1 − t n − 1 ) + ( 1 − t n ) t n − 1 t n − t n − 1 E [ s ( Z ) T ( Z − E [ Z ∣ X t n − 1 = y n − 1 ] ) X t n − 1 = y n − 1 ]
+ log E [ exp ( − t n − t n − 1 ( 1 − t n ) ( 1 − t n − 1 ) ξ n + r n ( 2 ) ( X ⋆ ) ) ∣ X t n − 1 = y n − 1 ] . (52) \quad+\log\mathbb{E}\left[\left.\exp\left(-\frac{t_n-t_{n-1}}{(1-t_n)(1-t_{n-1})}\xi_n+r_n^{(2)}(X_\star)\right)\right|X_{t_{n-1}}=y_{n-1}\right]. \tag*{(52)} + log E [ exp ( − ( 1 − t n ) ( 1 − t n − 1 ) t n − t n − 1 ξ n + r n ( 2 ) ( X ⋆ ) ) X t n − 1 = y n − 1 ] . (52)
Step 4: Controlling log p t n ( T n ( x ) ) det ∇ T n ( x ) p t n − 1 ( x ) \log\frac{p_{t_n}(T_n(x))\det\nabla T_n(x)}{p_{t_{n-1}}(x)} log p t n − 1 ( x ) p t n ( T n ( x )) d e t ∇ T n ( x ) . Taking the trace in (44) of Lemma 6 gives
Tr ( ∇ v t n − 1 ( y n − 1 ) ) = − d 1 − t n − 1 − 1 t n − 1 ( 1 − t n − 1 ) E [ s ( Z ) T ( Z − E [ Z ∣ X t n − 1 = y n − 1 ] ∣ X t n − 1 = y n − 1 ) ] . \operatorname{Tr}(\nabla v_{t_{n-1}}(y_{n-1}))=-\frac{d}{1-t_{n-1}}-\frac{1}{t_{n-1}(1-t_{n-1})}\mathbb{E}\left[s(Z)^\mathsf{T}\left(Z-\mathbb{E}[Z\mid X_{t_{n-1}}=y_{n-1}]\mid X_{t_{n-1}}=y_{n-1}\right)\right]. Tr ( ∇ v t n − 1 ( y n − 1 )) = − 1 − t n − 1 d − t n − 1 ( 1 − t n − 1 ) 1 E [ s ( Z ) T ( Z − E [ Z ∣ X t n − 1 = y n − 1 ] ∣ X t n − 1 = y n − 1 ) ] .
Combining (39), (50), the previous equation and the fact t n − t n − 1 t n − 1 ( 1 1 − t n − 1 1 − t n − 1 ) = ( t n − t n − 1 ) 2 t n − 1 ( 1 − t n − 1 ) ( 1 − t n ) \frac{t_n-t_{n-1}}{t_{n-1}}\left(\frac{1}{1-t_n}-\frac{1}{1-t_{n-1}}\right)=\frac{(t_n-t_{n-1})^2}{t_{n-1}(1-t_{n-1})(1-t_n)} t n − 1 t n − t n − 1 ( 1 − t n 1 − 1 − t n − 1 1 ) = t n − 1 ( 1 − t n − 1 ) ( 1 − t n ) ( t n − t n − 1 ) 2 , we have, for any y n − 1 ∈ G n y_{n-1}\in G_n y n − 1 ∈ G n ,
log ( p t n ( y n ) det ∇ T n ( y n − 1 ) p t n − 1 ( y n − 1 ) ) ≥ − d ( t n − t n − 1 ) 1 − t n − 1 + d log ( 1 − t n − 1 1 − t n ) + ( t n − t n − 1 ) 2 t n − 1 ( 1 − t n − 1 ) ( 1 − t n ) E [ s ( Z ) T ( Z − E [ Z ∣ X t n − 1 = y n − 1 ] ∣ X t n − 1 = y n − 1 ) ] − 4 3 ( t n − t n − 1 ) 2 ∥ ∇ v t n − 1 ( y n − 1 ) ∥ F 2 + log E [ exp ( − t n − t n − 1 ( 1 − t n ) ( 1 − t n − 1 ) ξ n + r n ( 2 ) ( X ∗ ) ) ∣ X t n − 1 = y n − 1 ] . (53) \begin{aligned}
\log\left(\frac{p_{t_n}(y_n)\det\nabla T_n(y_{n-1})}{p_{t_{n-1}}(y_{n-1})}\right)
&\geq-\frac{d(t_n-t_{n-1})}{1-t_{n-1}}+d\log\left(\frac{1-t_{n-1}}{1-t_n}\right)\\
&\quad+\frac{(t_n-t_{n-1})^2}{t_{n-1}(1-t_{n-1})(1-t_n)}\mathbb{E}\left[s(Z)^\mathsf{T}\left(Z-\mathbb{E}[Z\mid X_{t_{n-1}}=y_{n-1}]\mid X_{t_{n-1}}=y_{n-1}\right)\right]\\
&\quad-\frac{4}{3}(t_n-t_{n-1})^2\left\|\nabla v_{t_{n-1}}(y_{n-1})\right\|_{\mathrm{F}}^2\\
&\quad+\log\mathbb{E}\left[\exp\left(-\frac{t_n-t_{n-1}}{(1-t_n)(1-t_{n-1})}\xi_n+r_n^{(2)}(X_*)\right)\mid X_{t_{n-1}}=y_{n-1}\right]. \tag*{(53)}
\end{aligned} log ( p t n − 1 ( y n − 1 ) p t n ( y n ) det ∇ T n ( y n − 1 ) ) ≥ − 1 − t n − 1 d ( t n − t n − 1 ) + d log ( 1 − t n 1 − t n − 1 ) + t n − 1 ( 1 − t n − 1 ) ( 1 − t n ) ( t n − t n − 1 ) 2 E [ s ( Z ) T ( Z − E [ Z ∣ X t n − 1 = y n − 1 ] ∣ X t n − 1 = y n − 1 ) ] − 3 4 ( t n − t n − 1 ) 2 ∥ ∇ v t n − 1 ( y n − 1 ) ∥ F 2 + log E [ exp ( − ( 1 − t n ) ( 1 − t n − 1 ) t n − t n − 1 ξ n + r n ( 2 ) ( X ∗ ) ) ∣ X t n − 1 = y n − 1 ] . (53)
Since s ( ⋅ ) s(\cdot) s ( ⋅ ) is L L L -Lipschitz, the following lower bound holds:
E [ s ( Z ) T ( Z − E [ Z ∣ X t n − 1 = y n − 1 ] ∣ X t n − 1 = y n − 1 ) ] = E [ ( s ( Z ) − s ( E [ Z ∣ X t n − 1 = y n − 1 ] ) ) T ( Z − E [ Z ∣ X t n − 1 = y n − 1 ] ∣ X t n − 1 = y n − 1 ) ] ≥ − L E [ ∥ Z − E [ Z ∣ X t n − 1 = y n − 1 ] ∥ 2 2 ∣ X t n − 1 = y n − 1 ] . (54) \begin{aligned}
\mathbb{E}\left[s(Z)^\mathsf{T}\left(Z-\mathbb{E}[Z\mid X_{t_{n-1}}=y_{n-1}]\mid X_{t_{n-1}}=y_{n-1}\right)\right]
&=\mathbb{E}\left[(s(Z)-s(\mathbb{E}[Z\mid X_{t_{n-1}}=y_{n-1}]))^\mathsf{T}\left(Z-\mathbb{E}[Z\mid X_{t_{n-1}}=y_{n-1}]\mid X_{t_{n-1}}=y_{n-1}\right)\right]\\
&\geq-L\mathbb{E}\left[\left\|Z-\mathbb{E}[Z\mid X_{t_{n-1}}=y_{n-1}]\right\|_2^2\mid X_{t_{n-1}}=y_{n-1}\right]. \tag*{(54)}
\end{aligned} E [ s ( Z ) T ( Z − E [ Z ∣ X t n − 1 = y n − 1 ] ∣ X t n − 1 = y n − 1 ) ] = E [ ( s ( Z ) − s ( E [ Z ∣ X t n − 1 = y n − 1 ]) ) T ( Z − E [ Z ∣ X t n − 1 = y n − 1 ] ∣ X t n − 1 = y n − 1 ) ] ≥ − L E [ ∥ Z − E [ Z ∣ X t n − 1 = y n − 1 ] ∥ 2 2 ∣ X t n − 1 = y n − 1 ] . (54)
Jensen’s inequality also tells us
log E [ exp ( − t n − t n − 1 ( 1 − t n ) ( 1 − t n − 1 ) ξ n + r n ( 2 ) ( X ∗ ) ) ∣ X t n − 1 = y n − 1 ] ≥ E [ − t n − t n − 1 ( 1 − t n ) ( 1 − t n − 1 ) ξ n + r n ( 2 ) ( X ∗ ) ∣ X t n − 1 = y n − 1 ] = citep‘(49)‘ E [ r n ( 2 ) ( X ∗ ) ∣ X t n − 1 = y n − 1 ] ≥ citep‘(47)‘ − L ( t n − t n − 1 ) 2 2 t n − 1 2 ( 1 − t n ) 2 E [ ∥ Z − E [ Z ∣ X t n − 1 = y n − 1 ] ∥ 2 2 ∣ X t n − 1 = y n − 1 ] . (55) \begin{aligned}
\log\mathbb{E}\left[\exp\left(-\frac{t_n-t_{n-1}}{(1-t_n)(1-t_{n-1})}\xi_n+r_n^{(2)}(X_*)\right)\mid X_{t_{n-1}}=y_{n-1}\right]
&\geq\mathbb{E}\left[-\frac{t_n-t_{n-1}}{(1-t_n)(1-t_{n-1})}\xi_n+r_n^{(2)}(X_*)\mid X_{t_{n-1}}=y_{n-1}\right]\\
&\overset{\text{\scriptsize{citep}`(49)`}}{=}\mathbb{E}\left[r_n^{(2)}(X_*)\mid X_{t_{n-1}}=y_{n-1}\right]\\
&\overset{\text{\scriptsize{citep}`(47)`}}{\geq}-\frac{L(t_n-t_{n-1})^2}{2t_{n-1}^2(1-t_n)^2}\mathbb{E}\left[\left\|Z-\mathbb{E}[Z\mid X_{t_{n-1}}=y_{n-1}]\right\|_2^2\mid X_{t_{n-1}}=y_{n-1}\right]. \tag*{(55)}
\end{aligned} log E [ exp ( − ( 1 − t n ) ( 1 − t n − 1 ) t n − t n − 1 ξ n + r n ( 2 ) ( X ∗ ) ) ∣ X t n − 1 = y n − 1 ] ≥ E [ − ( 1 − t n ) ( 1 − t n − 1 ) t n − t n − 1 ξ n + r n ( 2 ) ( X ∗ ) ∣ X t n − 1 = y n − 1 ] = citep ‘(49)‘ E [ r n ( 2 ) ( X ∗ ) ∣ X t n − 1 = y n − 1 ] ≥ citep ‘(47)‘ − 2 t n − 1 2 ( 1 − t n ) 2 L ( t n − t n − 1 ) 2 E [ ∥ Z − E [ Z ∣ X t n − 1 = y n − 1 ] ∥ 2 2 ∣ X t n − 1 = y n − 1 ] . (55)
Putting (51), (52) and (53) together, we obtain that for any x ∈ G n x\in G_n x ∈ G n ,
log ( p t n ( T n ( x ) ) det ∇ T n ( x ) p t n − 1 ( x ) ) ≥ − 3 L ( t n − t n − 1 ) 2 2 t n − 1 2 ( 1 − t n ) 2 E [ ∥ Z − E [ Z ∣ X t n − 1 = x ] ∥ 2 2 ∣ X t n − 1 = x ] − 4 3 ( t n − t n − 1 ) 2 ∥ ∇ v t n − 1 ( x ) ∥ F 2 ≥ − 3 ( t n − t n − 1 ) 2 t n − 1 2 ( 1 − t n ) 2 ( d + L E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 = x ] + E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 = x ] E [ ∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 = x ] ) . (56) \begin{aligned}
\log\left(\frac{p_{t_n}(T_n(x))\det\nabla T_n(x)}{p_{t_{n-1}}(x)}\right)
&\geq-\frac{3L(t_n-t_{n-1})^2}{2t_{n-1}^2(1-t_n)^2}\mathbb{E}\left[\left\|Z-\mathbb{E}[Z\mid X_{t_{n-1}}=x]\right\|_2^2\mid X_{t_{n-1}}=x\right]\\
&\quad-\frac{4}{3}(t_n-t_{n-1})^2\left\|\nabla v_{t_{n-1}}(x)\right\|_{\mathrm{F}}^2\\
&\geq-3\frac{(t_n-t_{n-1})^2}{t_{n-1}^2(1-t_n)^2}\left(d+L\mathbb{E}\left[\left\|Z-\mathbb{E}[Z]\right\|_2^2\mid X_{t_{n-1}}=x\right]\right.\\
&\qquad\left.+\mathbb{E}\left[\left\|Z-\mathbb{E}[Z]\right\|_2^2\mid X_{t_{n-1}}=x\right]\mathbb{E}\left[\left\|s(Z)\right\|_2^2\mid X_{t_{n-1}}=x\right]\right). \tag*{(56)}
\end{aligned} log ( p t n − 1 ( x ) p t n ( T n ( x )) det ∇ T n ( x ) ) ≥ − 2 t n − 1 2 ( 1 − t n ) 2 3 L ( t n − t n − 1 ) 2 E [ ∥ Z − E [ Z ∣ X t n − 1 = x ] ∥ 2 2 ∣ X t n − 1 = x ] − 3 4 ( t n − t n − 1 ) 2 ∥ ∇ v t n − 1 ( x ) ∥ F 2 ≥ − 3 t n − 1 2 ( 1 − t n ) 2 ( t n − t n − 1 ) 2 ( d + L E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 = x ] + E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 = x ] E [ ∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 = x ] ) . (56)
Here, the first inequality comes from − a − log ( 1 − a ) ≥ 0 -a-\log(1-a)\geq0 − a − log ( 1 − a ) ≥ 0 with a = ( t n − t n − 1 ) / ( 1 − t n − 1 ) a=(t_n-t_{n-1})/(1-t_{n-1}) a = ( t n − t n − 1 ) / ( 1 − t n − 1 ) and t n − 1 ( 1 − t n ) / ( 1 − t n − 1 ) ≤ 1 t_{n-1}(1-t_n)/(1-t_{n-1})\leq1 t n − 1 ( 1 − t n ) / ( 1 − t n − 1 ) ≤ 1 by Assumption 3. The second inequality uses (44) and the fact that E [ Z ∣ X t n − 1 = x ] \mathbb{E}[Z\mid X_{t_{n-1}}=x] E [ Z ∣ X t n − 1 = x ] minimizes the conditional mean squared error.
Finally, inserting (56) into (38) and using [ 1 − e − v ] + ≤ v [1-e^{-v}]_{+}\le v [ 1 − e − v ] + ≤ v for v ≥ 0 v\ge0 v ≥ 0 yields
TV ( P ^ Y n , P t n ) ≤ P ( X t n − 1 ∉ G n ) + 3 ( t n − t n − 1 ) 2 t n − 1 2 ( 1 − t n ) 2 { d + L E [ ∥ Z − E [ Z ] ∥ 2 2 ] + E [ 1 { X t n − 1 ∈ G n } E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 ] E [ ∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 ] ] } . (57) \operatorname{TV}(\widehat{P}_{Y_n},P_{t_n})\le\mathbb{P}(X_{t_{n-1}}\notin G_n)+3\frac{(t_n-t_{n-1})^2}{t_{n-1}^2(1-t_n)^2}\left\{d+L\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]+\mathbb{E}\left[1_{\{X_{t_{n-1}}\in G_n\}}\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\mid X_{t_{n-1}}\right]\mathbb{E}\left[\lVert s(Z)\rVert_2^2\mid X_{t_{n-1}}\right]\right]\right\}. \tag*{(57)} TV ( P Y n , P t n ) ≤ P ( X t n − 1 ∈ / G n ) + 3 t n − 1 2 ( 1 − t n ) 2 ( t n − t n − 1 ) 2 { d + L E [ ∥ Z − E [ Z ] ∥ 2 2 ] + E [ 1 { X t n − 1 ∈ G n } E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 ] E [ ∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 ] ] } . (57)
Step 5: Putting it all together. By virtue of (25) in Lemma 2, conditional Jensen’s inequality and the tower property, one has
E [ exp { E [ ∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 ] 4 L } ] ≤ E [ E [ exp { ∥ s ( Z ) ∥ 2 2 4 L } ∣ X t n − 1 ] ] = E [ exp { ∥ s ( Z ) ∥ 2 2 4 L } ] ≤ 2 d / 2 . \mathbb{E}\left[\exp\left\{\frac{\mathbb{E}[\lVert s(Z)\rVert_2^2\mid X_{t_{n-1}}]}{4L}\right\}\right]\le\mathbb{E}\left[\mathbb{E}\left[\left.\exp\left\{\frac{\lVert s(Z)\rVert_2^2}{4L}\right\}\right|X_{t_{n-1}}\right]\right]=\mathbb{E}\left[\exp\left\{\frac{\lVert s(Z)\rVert_2^2}{4L}\right\}\right]\le2^{d/2}. E [ exp { 4 L E [∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 ] } ] ≤ E [ E [ exp { 4 L ∥ s ( Z ) ∥ 2 2 } X t n − 1 ] ] = E [ exp { 4 L ∥ s ( Z ) ∥ 2 2 } ] ≤ 2 d /2 .
Applying Markov’s inequality to this exponential yields
P ( E [ ∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 ] > 4 L { d + 12 log ( N + 1 ) } ) = P ( exp { E [ ∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 ] 4 L } > e d + 12 log ( N + 1 ) ) ≤ e − d − 12 log ( N + 1 ) E [ exp { E [ ∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 ] 4 L } ] ≤ exp ( d 2 log 2 − d − 12 log ( N + 1 ) ) ≤ ( N + 1 ) − 12 . (58) \begin{aligned}
\mathbb{P}\left(\mathbb{E}[\lVert s(Z)\rVert_2^2\mid X_{t_{n-1}}]>4L\{d+12\log(N+1)\}\right)
=\mathbb{P}\left(\exp\left\{\frac{\mathbb{E}[\lVert s(Z)\rVert_2^2\mid X_{t_{n-1}}]}{4L}\right\}>e^{d+12\log(N+1)}\right) \\
&\le e^{-d-12\log(N+1)}\mathbb{E}\left[\exp\left\{\frac{\mathbb{E}[\lVert s(Z)\rVert_2^2\mid X_{t_{n-1}}]}{4L}\right\}\right]
\le\exp\left(\frac{d}{2}\log2-d-12\log(N+1)\right)
\le(N+1)^{-12}. \tag*{(58)}
\end{aligned} P ( E [∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 ] > 4 L { d + 12 log ( N + 1 )} ) = P ( exp { 4 L E [∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 ] } > e d + 12 l o g ( N + 1 ) ) ≤ e − d − 12 l o g ( N + 1 ) E [ exp { 4 L E [∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 ] } ] ≤ exp ( 2 d log 2 − d − 12 log ( N + 1 ) ) ≤ ( N + 1 ) − 12 . (58)
In addition, Markov’s inequality implies that
P ( E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 ] > t n − 1 2 ( 1 − t n ) 2 256 L ( t n − t n − 1 ) 2 { d + 12 log ( N + 1 ) } ) ≤ 256 L ( t n − t n − 1 ) 2 { d + 12 log ( N + 1 ) } t n − 1 2 ( 1 − t n ) 2 E [ E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 ] ] = 256 L ( t n − t n − 1 ) 2 { d + 12 log ( N + 1 ) } t n − 1 2 ( 1 − t n ) 2 E [ ∥ Z − E [ Z ] ∥ 2 2 ] . (59) \begin{aligned}
\mathbb{P}\left(\mathbb{E}[\lVert Z-\mathbb{E}[Z]\rVert_2^2\mid X_{t_{n-1}}]>\frac{t_{n-1}^2(1-t_n)^2}{256L(t_n-t_{n-1})^2\{d+12\log(N+1)\}}\right) \\
&\le\frac{256L(t_n-t_{n-1})^2\{d+12\log(N+1)\}}{t_{n-1}^2(1-t_n)^2}\mathbb{E}\left[\mathbb{E}[\lVert Z-\mathbb{E}[Z]\rVert_2^2\mid X_{t_{n-1}}]\right]
=\frac{256L(t_n-t_{n-1})^2\{d+12\log(N+1)\}}{t_{n-1}^2(1-t_n)^2}\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]. \tag*{(59)}
\end{aligned} P ( E [∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 ] > 256 L ( t n − t n − 1 ) 2 { d + 12 log ( N + 1 )} t n − 1 2 ( 1 − t n ) 2 ) ≤ t n − 1 2 ( 1 − t n ) 2 256 L ( t n − t n − 1 ) 2 { d + 12 log ( N + 1 )} E [ E [∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 ] ] = t n − 1 2 ( 1 − t n ) 2 256 L ( t n − t n − 1 ) 2 { d + 12 log ( N + 1 )} E [ ∥ Z − E [ Z ] ∥ 2 2 ] . (59)
In view of (32) , (58) , (59) and the union bound, we have
P ( X t n − 1 ∉ G n ) ≤ ( N + 1 ) − 12 + 256 L ( t n − t n − 1 ) 2 { d + 12 log ( N + 1 ) } t n − 1 2 ( 1 − t n ) 2 E [ ∥ Z − E [ Z ] ∥ 2 2 ] . (60) \mathbb{P}(X_{t_{n-1}}\notin G_n)\le(N+1)^{-12}+\frac{256L(t_n-t_{n-1})^2\{d+12\log(N+1)\}}{t_{n-1}^2(1-t_n)^2}\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]. \tag*{(60)} P ( X t n − 1 ∈ / G n ) ≤ ( N + 1 ) − 12 + t n − 1 2 ( 1 − t n ) 2 256 L ( t n − t n − 1 ) 2 { d + 12 log ( N + 1 )} E [ ∥ Z − E [ Z ] ∥ 2 2 ] . (60)
On the event { X t n − 1 ∈ G n } \{X_{t_{n-1}}\in G_n\} { X t n − 1 ∈ G n } , its conditional score factor E [ ∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 = x ] \mathbb{E}[\lVert s(Z)\rVert_2^2\mid X_{t_{n-1}}=x] E [∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 = x ] is at most 4 L { d + 12 log ( N + 1 ) } 4L\{d+12\log(N+1)\} 4 L { d + 12 log ( N + 1 )} , and consequently
E [ 1 G n ( X t n − 1 ) E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 ] E [ ∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 ] ] ≤ 4 L { d + 12 log ( N + 1 ) } E [ 1 G n ( X t n − 1 ) E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 ] ] \mathbb{E}\left[1_{G_n}(X_{t_{n-1}})\mathbb{E}[\lVert Z-\mathbb{E}[Z]\rVert_2^2\mid X_{t_{n-1}}]\mathbb{E}[\lVert s(Z)\rVert_2^2\mid X_{t_{n-1}}]\right]
\le4L\{d+12\log(N+1)\}\mathbb{E}\left[1_{G_n}(X_{t_{n-1}})\mathbb{E}[\lVert Z-\mathbb{E}[Z]\rVert_2^2\mid X_{t_{n-1}}]\right] E [ 1 G n ( X t n − 1 ) E [∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 ] E [∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 ] ] ≤ 4 L { d + 12 log ( N + 1 )} E [ 1 G n ( X t n − 1 ) E [∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 ] ]
≤ 4 L { d + 12 log ( N + 1 ) } E [ E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 ] ] = 4 L { d + 12 log ( N + 1 ) } E [ ∥ Z − E [ Z ] ∥ 2 2 ] . (61) \le4L\{d+12\log(N+1)\}\mathbb{E}\left[\mathbb{E}[\lVert Z-\mathbb{E}[Z]\rVert_2^2\mid X_{t_{n-1}}]\right]
=4L\{d+12\log(N+1)\}\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]. \tag*{(61)} ≤ 4 L { d + 12 log ( N + 1 )} E [ E [∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 ] ] = 4 L { d + 12 log ( N + 1 )} E [ ∥ Z − E [ Z ] ∥ 2 2 ] . (61)
Substituting (60) and (61) into (57) gives
TV ( P ^ Y n , P t n ) ≤ ( N + 1 ) − 12 + 256 L ( t n − t n − 1 ) 2 { d + 12 log ( N + 1 ) } t n − 1 2 ( 1 − t n ) 2 E [ ∥ Z − E [ Z ] ∥ 2 2 ] + 3 ( t n − t n − 1 ) 2 t n − 1 2 ( 1 − t n ) 2 { d + L E [ ∥ Z − E [ Z ] ∥ 2 2 ] + 4 L { d + 12 log ( N + 1 ) } E [ ∥ Z − E [ Z ] ∥ 2 2 ] } . ≤ ( N + 1 ) − 12 + ( t n − t n − 1 ) 2 t n − 1 2 ( 1 − t n ) 2 { 4 d + 4 L E [ ∥ Z − E [ Z ] ∥ 2 2 ] + 300 L { d + 12 log ( N + 1 ) } E [ ∥ Z − E [ Z ] ∥ 2 2 ] } . \begin{aligned}
\operatorname{TV}(\widehat{P}_{Y_n},P_{t_n})\le(N+1)^{-12}+\frac{256L(t_n-t_{n-1})^2\{d+12\log(N+1)\}}{t_{n-1}^2(1-t_n)^2}\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]
+\frac{3(t_n-t_{n-1})^2}{t_{n-1}^2(1-t_n)^2}\left\{d+L\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]+4L\{d+12\log(N+1)\}\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]\right\}. \\
&\le(N+1)^{-12}+\frac{(t_n-t_{n-1})^2}{t_{n-1}^2(1-t_n)^2}\left\{4d+4L\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]+300L\{d+12\log(N+1)\}\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]\right\}.
\end{aligned} TV ( P Y n , P t n ) ≤ ( N + 1 ) − 12 + t n − 1 2 ( 1 − t n ) 2 256 L ( t n − t n − 1 ) 2 { d + 12 log ( N + 1 )} E [ ∥ Z − E [ Z ] ∥ 2 2 ] + t n − 1 2 ( 1 − t n ) 2 3 ( t n − t n − 1 ) 2 { d + L E [ ∥ Z − E [ Z ] ∥ 2 2 ] + 4 L { d + 12 log ( N + 1 )} E [ ∥ Z − E [ Z ] ∥ 2 2 ] } . ≤ ( N + 1 ) − 12 + t n − 1 2 ( 1 − t n ) 2 ( t n − t n − 1 ) 2 { 4 d + 4 L E [ ∥ Z − E [ Z ] ∥ 2 2 ] + 300 L { d + 12 log ( N + 1 )} E [ ∥ Z − E [ Z ] ∥ 2 2 ] } .
We know from (27) in Lemma 2 that d ≤ L E [ ∥ Z − E [ Z ] ∥ 2 2 ] d \le L\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right] d ≤ L E [ ∥ Z − E [ Z ] ∥ 2 2 ] . Therefore, we have
4 d + 4 L E [ ∥ Z − E [ Z ] ∥ 2 2 ] ≤ 8 L E [ ∥ Z − E [ Z ] ∥ 2 2 ] ≤ 8 L { d + 12 log ( N + 1 ) } E [ ∥ Z − E [ Z ] ∥ 2 2 ] . 4d+4L\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]\le8L\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]\le8L\{d+12\log(N+1)\}\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]. 4 d + 4 L E [ ∥ Z − E [ Z ] ∥ 2 2 ] ≤ 8 L E [ ∥ Z − E [ Z ] ∥ 2 2 ] ≤ 8 L { d + 12 log ( N + 1 )} E [ ∥ Z − E [ Z ] ∥ 2 2 ] .
Combining these inequalities gives
TV ( P Y n , P t n ) ≤ ( N + 1 ) − 12 + C L ( t n − t n − 1 ) 2 { d + log ( N + 1 ) } t n − 1 2 ( 1 − t n ) 2 E [ ∥ Z − E [ Z ] ∥ 2 2 ] , (62) \operatorname{TV}(P_{Y_n},P_{t_n})\le(N+1)^{-12}+\frac{CL(t_n-t_{n-1})^2\{d+\log(N+1)\}}{t_{n-1}^2(1-t_n)^2}\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right], \tag*{(62)} TV ( P Y n , P t n ) ≤ ( N + 1 ) − 12 + t n − 1 2 ( 1 − t n ) 2 C L ( t n − t n − 1 ) 2 { d + log ( N + 1 )} E [ ∥ Z − E [ Z ] ∥ 2 2 ] , (62)
The grid estimate (23) with r = 2 r=2 r = 2 gives
∑ n = 1 N ( t n − t n − 1 ) 2 t n − 1 2 ( 1 − t n ) 2 ≤ c s 2 log 2 ( N + 1 ) N + 1 . \sum_{n=1}^{N}\frac{(t_n-t_{n-1})^2}{t_{n-1}^2(1-t_n)^2}\le\frac{c_s^2\log^2(N+1)}{N+1}. n = 1 ∑ N t n − 1 2 ( 1 − t n ) 2 ( t n − t n − 1 ) 2 ≤ N + 1 c s 2 log 2 ( N + 1 ) .
Summing the one-step bound (62) , we obtain
∑ n = 1 N TV ( P Y n , P t n ) ≤ 1 ( N + 1 ) 11 + C L { d + log ( N + 1 ) } log 2 ( N + 1 ) N + 1 E [ ∥ Z − E [ Z ] ∥ 2 2 ] . \sum_{n=1}^{N}\operatorname{TV}(P_{Y_n},P_{t_n})\le\frac{1}{(N+1)^{11}}+CL\{d+\log(N+1)\}\frac{\log^2(N+1)}{N+1}\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]. n = 1 ∑ N TV ( P Y n , P t n ) ≤ ( N + 1 ) 11 1 + C L { d + log ( N + 1 )} N + 1 log 2 ( N + 1 ) E [ ∥ Z − E [ Z ] ∥ 2 2 ] .
Putting (31) , (29) and the previous inequality together, we arrive at
TV ( P Y N , P t N ) ≤ L E [ ∥ Z − E [ Z ] ∥ 2 2 ] ( N + 1 ) 2 + ( N + 1 ) − 11 + C L { d + log ( N + 1 ) } log 2 ( N + 1 ) N + 1 E [ ∥ Z − E [ Z ] ∥ 2 2 ] . \operatorname{TV}(P_{Y_N},P_{t_N})\le\frac{\sqrt{L\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]}}{(N+1)^2}+(N+1)^{-11}+CL\{d+\log(N+1)\}\frac{\log^2(N+1)}{N+1}\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]. TV ( P Y N , P t N ) ≤ ( N + 1 ) 2 L E [ ∥ Z − E [ Z ] ∥ 2 2 ] + ( N + 1 ) − 11 + C L { d + log ( N + 1 )} N + 1 log 2 ( N + 1 ) E [ ∥ Z − E [ Z ] ∥ 2 2 ] .
In addition, (27) tells us
1 ≤ L E [ ∥ Z − E [ Z ] ∥ 2 2 ] ≤ L E [ ∥ Z − E [ Z ] ∥ 2 2 ] . 1\le\sqrt{L\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]}\le L\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]. 1 ≤ L E [ ∥ Z − E [ Z ] ∥ 2 2 ] ≤ L E [ ∥ Z − E [ Z ] ∥ 2 2 ] .
As a consequence, one has
L E [ ∥ Z − E [ Z ] ∥ 2 2 ] ( N + 1 ) 2 + ( N + 1 ) − 11 ≤ 2 L E [ ∥ Z − E [ Z ] ∥ 2 2 ] ( N + 1 ) 2 . \frac{\sqrt{L\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]}}{(N+1)^2}+(N+1)^{-11}\le\frac{2L\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]}{(N+1)^2}. ( N + 1 ) 2 L E [ ∥ Z − E [ Z ] ∥ 2 2 ] + ( N + 1 ) − 11 ≤ ( N + 1 ) 2 2 L E [ ∥ Z − E [ Z ] ∥ 2 2 ] .
Therefore, we have
TV ( P Y N , P t N ) ≲ L ( d + log N ) E [ ∥ Z − E [ Z ] ∥ 2 2 ] log 2 N N , \operatorname{TV}(P_{Y_N},P_{t_N})\lesssim L(d+\log N)\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]\frac{\log^2 N}{N}, TV ( P Y N , P t N ) ≲ L ( d + log N ) E [ ∥ Z − E [ Z ] ∥ 2 2 ] N log 2 N ,
which finishes the proof of Theorem 1.
Discussion
In this work, we have established the first non-asymptotic convergence guarantee for the Euler sampler in flow matching with a general non-Gaussian source. Compared with prior work, our analysis does not require boundedness assumptions on the velocity field or additional tail conditions on the target distribution. We assume only that the source distribution has a continuously differentiable density that is positive everywhere on R d \mathbb{R}^d R d , a finite second moment, and a Lipschitz score function. These mild assumptions accommodate, among many other examples, correlated Gaussian distributions, multivariate Student’s t t t distributions, and Gaussian mixtures with a shared covariance. To establish our theory, we have developed new proof techniques for handling nonlinear source score functions and proving restricted injectivity of the Euler map, which extend beyond the global injectivity available for Gaussian sources. These techniques may be of independent interest.
The theory of flow matching with general source distributions remains in its early stages, leaving a number of important questions open. For instance, our current analysis assumes access to the exact population velocity field; extending it to account for velocity estimation error and quantifying its effect on sampling accuracy are natural next steps. It is also unclear whether particular source choices, such as multivariate Student’s t t t distributions, can accelerate convergence for heavy-tailed targets. Another related question concerns the choice of source-target coupling. In image-to-image and text-to-image generation, many approaches train on paired source and target samples to preserve the desired correspondence during sampling. Developing convergence guarantees for dependent source-target couplings could clarify which couplings lead to faster sampling. Another direction is to extend our theory from continuous distributions on Euclidean space to discrete and Riemannian flow matching, where generating structured data is a central task.
Acknowledgments
Y. Chen is supported in part by the Alfred P. Sloan Research Fellowship, the ONR grant N00014-25-1-2344, the AFOSR grant FA9550261B178, and the Wharton AI & Analytics Initiative’s AI Research Fund. G. Li is supported in part by the Chinese University of Hong Kong Direct Grant for Research and the Hong Kong Research Grants Council ECS 2191363. This work is also supported in part by the NSF under Cooperative Agreement No. 2433450.
Proofs of auxiliary lemmas
Proof of examples in Section 3.2
Note that all three examples of source distributions have strictly positive C 1 C^1 C 1 densities on R d \mathbb{R}^d R d . We develop upper bounds for L TotVar b a s e L\operatorname{TotVar}_{\mathrm{base}} L TotVar base below. For each source, we shall upper bound the Lipschitz constant L L L by sup x ∥ ∇ s ( x ) ∥ o p \sup_x \lVert\nabla s(x)\rVert_{\mathrm{op}} sup x ∥ ∇ s ( x ) ∥ op . To see this, we observe that
∥ s ( z ) − s ( z ′ ) ∥ 2 = ∥ ∫ 0 1 ∇ s ( z ′ + θ ( z − z ′ ) ) ( z − z ′ ) d θ ∥ 2 ≤ ∫ 0 1 ∥ ∇ s ( z ′ + θ ( z − z ′ ) ) ∥ o p ∥ z − z ′ ∥ 2 d θ ≤ ( sup x ∥ ∇ s ( x ) ∥ o p ) ∥ z − z ′ ∥ 2 . \begin{aligned}
\lVert s(z)-s(z')\rVert_2 = \left\lVert\int_0^1 \nabla s(z' + \theta(z-z'))(z-z')\,\mathrm{d}\theta\right\rVert_2 \\
&\le\int_0^1 \lVert\nabla s(z' + \theta(z-z'))\rVert_{\mathrm{op}}\lVert z-z'\rVert_2\,\mathrm{d}\theta\le\left(\sup_x \lVert\nabla s(x)\rVert_{\mathrm{op}}\right)\lVert z-z'\rVert_2.
\end{aligned} ∥ s ( z ) − s ( z ′ ) ∥ 2 = ∫ 0 1 ∇ s ( z ′ + θ ( z − z ′ )) ( z − z ′ ) d θ 2 ≤ ∫ 0 1 ∥ ∇ s ( z ′ + θ ( z − z ′ )) ∥ op ∥ z − z ′ ∥ 2 d θ ≤ ( x sup ∥ ∇ s ( x ) ∥ op ) ∥ z − z ′ ∥ 2 .
Gaussian source. For Z ∼ N ( μ , Σ ) Z\sim\mathcal{N}(\mu,\Sigma) Z ∼ N ( μ , Σ ) with Σ ≻ 0 \Sigma\succ0 Σ ≻ 0 , one can show that
s ( z ) = ∇ z [ − 1 2 ( z − μ ) ⊤ Σ − 1 ( z − μ ) ] = − Σ − 1 ( z − μ ) , s(z)=\nabla_z\left[-\frac{1}{2}(z-\mu)^\top\Sigma^{-1}(z-\mu)\right]=-\Sigma^{-1}(z-\mu), s ( z ) = ∇ z [ − 2 1 ( z − μ ) ⊤ Σ − 1 ( z − μ ) ] = − Σ − 1 ( z − μ ) ,
∇ s ( z ) = − Σ − 1 , L = ∥ Σ − 1 ∥ o p = 1 λ min ( Σ ) , \nabla s(z)=-\Sigma^{-1},\qquad L=\lVert\Sigma^{-1}\rVert_{\mathrm{op}}=\frac{1}{\lambda_{\min}(\Sigma)}, ∇ s ( z ) = − Σ − 1 , L = ∥ Σ − 1 ∥ op = λ m i n ( Σ ) 1 ,
TotVar b a s e = tr E [ ( Z − μ ) ( Z − μ ) ⊤ ] = tr ( Σ ) . \operatorname{TotVar}_{\mathrm{base}}=\operatorname{tr}\mathbb{E}[(Z-\mu)(Z-\mu)^\top]=\operatorname{tr}(\Sigma). TotVar base = tr E [( Z − μ ) ( Z − μ ) ⊤ ] = tr ( Σ ) .
Therefore, we have L TotVar b a s e = tr ( Σ ) λ min ( Σ ) L\operatorname{TotVar}_{\mathrm{base}}=\frac{\operatorname{tr}(\Sigma)}{\lambda_{\min}(\Sigma)} L TotVar base = λ m i n ( Σ ) tr ( Σ ) .
Multivariate Student-t t t source. Let ν > 2 \nu>2 ν > 2 and let Σ ≻ 0 \Sigma\succ0 Σ ≻ 0 be the scale matrix. Set y = Σ − 1 / 2 ( z − μ ) y=\Sigma^{-1/2}(z-\mu) y = Σ − 1/2 ( z − μ ) and q = ∥ y ∥ 2 2 q=\lVert y\rVert_2^2 q = ∥ y ∥ 2 2 . It is straightforward to show the following:
p b a s e ( z ) ∝ ∣ Σ ∣ − 1 / 2 ( 1 + q / ν ) − ( ν + d ) / 2 , ∇ z q = 2 Σ − 1 ( z − μ ) , p_{\mathrm{base}}(z)\propto|\Sigma|^{-1/2}(1+q/\nu)^{-(\nu+d)/2},\qquad\nabla_z q=2\Sigma^{-1}(z-\mu), p base ( z ) ∝ ∣Σ ∣ − 1/2 ( 1 + q / ν ) − ( ν + d ) /2 , ∇ z q = 2 Σ − 1 ( z − μ ) ,
s ( z ) = − ν + d 2 ∇ z q ν + q = − ν + d ν + q Σ − 1 ( z − μ ) , s(z)=-\frac{\nu+d}{2}\frac{\nabla_z q}{\nu+q}=-\frac{\nu+d}{\nu+q}\Sigma^{-1}(z-\mu), s ( z ) = − 2 ν + d ν + q ∇ z q = − ν + q ν + d Σ − 1 ( z − μ ) ,
∇ s ( z ) = − ν + d ν + q Σ − 1 + 2 ( ν + d ) ( ν + q ) 2 Σ − 1 ( z − μ ) ( z − μ ) ⊤ Σ − 1 = − ν + d ν + q Σ − 1 / 2 ( I d − 2 y y ⊤ ν + q ) Σ − 1 / 2 . \nabla s(z)=-\frac{\nu+d}{\nu+q}\Sigma^{-1}+\frac{2(\nu+d)}{(\nu+q)^2}\Sigma^{-1}(z-\mu)(z-\mu)^\top\Sigma^{-1}
=-\frac{\nu+d}{\nu+q}\Sigma^{-1/2}\left(I_d-\frac{2yy^\top}{\nu+q}\right)\Sigma^{-1/2}. ∇ s ( z ) = − ν + q ν + d Σ − 1 + ( ν + q ) 2 2 ( ν + d ) Σ − 1 ( z − μ ) ( z − μ ) ⊤ Σ − 1 = − ν + q ν + d Σ − 1/2 ( I d − ν + q 2 y y ⊤ ) Σ − 1/2 .
Since
0 ⪯ y y ⊤ ⪯ q I d , − I d ⪯ ν − q ν + q I d ⪯ I d − 2 y y ⊤ ν + q ⪯ I d , 0\preceq yy^\top\preceq qI_d,\qquad-I_d\preceq\frac{\nu-q}{\nu+q}I_d\preceq I_d-\frac{2yy^\top}{\nu+q}\preceq I_d, 0 ⪯ y y ⊤ ⪯ q I d , − I d ⪯ ν + q ν − q I d ⪯ I d − ν + q 2 y y ⊤ ⪯ I d ,
we obtain the Lipschitz constant as follows
∥ ∇ s ( z ) ∥ o p ≤ ν + d ν + q ∥ Σ − 1 / 2 ∥ o p 2 ∥ I d − 2 y y T ν + q ∥ o p ≤ ν + d ( ν + q ) λ min ( Σ ) ≤ ν + d ν λ min ( Σ ) = : L . \lVert\nabla s(z)\rVert_{\mathrm{op}} \le\frac{\nu+d}{\nu+q}\lVert\Sigma^{-1/2}\rVert_{\mathrm{op}}^{2}\left\lVert I_d-\frac{2yy^{\mathsf{T}}}{\nu+q}\right\rVert_{\mathrm{op}} \le\frac{\nu+d}{(\nu+q)\lambda_{\min}(\Sigma)} \le\frac{\nu+d}{\nu\lambda_{\min}(\Sigma)}=:L. ∥ ∇ s ( z ) ∥ op ≤ ν + q ν + d ∥ Σ − 1/2 ∥ op 2 I d − ν + q 2 y y T op ≤ ( ν + q ) λ m i n ( Σ ) ν + d ≤ ν λ m i n ( Σ ) ν + d =: L .
Using independent G ∼ N ( 0 , I d ) G\sim\mathcal{N}(0,I_d) G ∼ N ( 0 , I d ) and S ∼ χ ν 2 S\sim\chi_{\nu}^{2} S ∼ χ ν 2 , one can write
Z = d μ + ν / S Σ 1 / 2 G , E [ Z ] = E [ μ + ν / S Σ 1 / 2 E [ G ∣ S ] ] = μ . Z\mathrel{\stackrel{\mathrm{d}}{=}}\mu+\sqrt{\nu/S}\,\Sigma^{1/2}G,\qquad\mathbb{E}[Z]=\mathbb{E}\left[\mu+\sqrt{\nu/S}\,\Sigma^{1/2}\mathbb{E}[G\mid S]\right]=\mu. Z = d μ + ν / S Σ 1/2 G , E [ Z ] = E [ μ + ν / S Σ 1/2 E [ G ∣ S ] ] = μ .
Hence, we can bound the total variance of Z Z Z as follows
TotVar b a s e = ν E [ S − 1 G T Σ G ] = ν E [ S − 1 ] tr ( Σ E [ G G T ] ) = ν ν − 2 tr ( Σ ) . \operatorname{TotVar}_{\mathrm{base}}=\nu\mathbb{E}[S^{-1}G^{\mathsf{T}}\Sigma G]=\nu\mathbb{E}[S^{-1}]\operatorname{tr}\left(\Sigma\mathbb{E}[GG^{\mathsf{T}}]\right)=\frac{\nu}{\nu-2}\operatorname{tr}(\Sigma). TotVar base = ν E [ S − 1 G T Σ G ] = ν E [ S − 1 ] tr ( Σ E [ G G T ] ) = ν − 2 ν tr ( Σ ) .
Therefore, we have
L TotVar b a s e = ν + d ν λ min ( Σ ) ν tr ( Σ ) ν − 2 = ν + d ν − 2 tr ( Σ ) λ min ( Σ ) . L\operatorname{TotVar}_{\mathrm{base}}=\frac{\nu+d}{\nu\lambda_{\min}(\Sigma)}\frac{\nu\operatorname{tr}(\Sigma)}{\nu-2}=\frac{\nu+d}{\nu-2}\frac{\operatorname{tr}(\Sigma)}{\lambda_{\min}(\Sigma)}. L TotVar base = ν λ m i n ( Σ ) ν + d ν − 2 ν tr ( Σ ) = ν − 2 ν + d λ m i n ( Σ ) tr ( Σ ) .
Gaussian mixture with a common covariance. Let ϕ Σ \phi_{\Sigma} ϕ Σ be the N ( 0 , Σ ) \mathcal{N}(0,\Sigma) N ( 0 , Σ ) density and define
w k ( z ) = π k ϕ Σ ( z − μ k ) p b a s e ( z ) = π k ϕ Σ ( z − μ k ) ∑ l = 1 m π l ϕ Σ ( z − μ l ) , m z = ∑ k w k ( z ) μ k . w_k(z)=\frac{\pi_k\phi_{\Sigma}(z-\mu_k)}{p_{\mathrm{base}}(z)}=\frac{\pi_k\phi_{\Sigma}(z-\mu_k)}{\sum_{l=1}^{m}\pi_l\phi_{\Sigma}(z-\mu_l)},\qquad m_z=\sum_k w_k(z)\mu_k. w k ( z ) = p base ( z ) π k ϕ Σ ( z − μ k ) = ∑ l = 1 m π l ϕ Σ ( z − μ l ) π k ϕ Σ ( z − μ k ) , m z = k ∑ w k ( z ) μ k .
We can then calculate the score function as follows:
s ( z ) = ∑ k π k ∇ z ϕ Σ ( z − μ k ) p b a s e ( z ) = − ∑ k w k Σ − 1 ( z − μ k ) = Σ − 1 ( m z − z ) . s(z)=\frac{\sum_k\pi_k\nabla_z\phi_{\Sigma}(z-\mu_k)}{p_{\mathrm{base}}(z)}=-\sum_k w_k\Sigma^{-1}(z-\mu_k)=\Sigma^{-1}(m_z-z). s ( z ) = p base ( z ) ∑ k π k ∇ z ϕ Σ ( z − μ k ) = − k ∑ w k Σ − 1 ( z − μ k ) = Σ − 1 ( m z − z ) .
To bound its gradient, we note that
∇ w k ( z ) = w k [ − Σ − 1 ( z − μ k ) − s ( z ) ] = w k Σ − 1 ( μ k − m z ) , \nabla w_k(z)=w_k\left[-\Sigma^{-1}(z-\mu_k)-s(z)\right]=w_k\Sigma^{-1}(\mu_k-m_z), ∇ w k ( z ) = w k [ − Σ − 1 ( z − μ k ) − s ( z ) ] = w k Σ − 1 ( μ k − m z ) ,
∇ m z = ∑ k μ k ( ∇ w k ) T = ∑ k w k μ k ( μ k − m z ) T Σ − 1 = ∑ k w k ( μ k − m z ) ( μ k − m z ) T Σ − 1 , \nabla m_z=\sum_k\mu_k(\nabla w_k)^{\mathsf{T}}=\sum_k w_k\mu_k(\mu_k-m_z)^{\mathsf{T}}\Sigma^{-1}=\sum_k w_k(\mu_k-m_z)(\mu_k-m_z)^{\mathsf{T}}\Sigma^{-1}, ∇ m z = k ∑ μ k ( ∇ w k ) T = k ∑ w k μ k ( μ k − m z ) T Σ − 1 = k ∑ w k ( μ k − m z ) ( μ k − m z ) T Σ − 1 ,
which implies that
∇ s ( z ) = − Σ − 1 + Σ − 1 [ ∑ k w k ( μ k − m z ) ( μ k − m z ) T ] Σ − 1 = Σ − 1 / 2 ( C z − I d ) Σ − 1 / 2 , \nabla s(z)=-\Sigma^{-1}+\Sigma^{-1}\left[\sum_k w_k(\mu_k-m_z)(\mu_k-m_z)^{\mathsf{T}}\right]\Sigma^{-1}=\Sigma^{-1/2}(C_z-I_d)\Sigma^{-1/2}, ∇ s ( z ) = − Σ − 1 + Σ − 1 [ k ∑ w k ( μ k − m z ) ( μ k − m z ) T ] Σ − 1 = Σ − 1/2 ( C z − I d ) Σ − 1/2 ,
where
C z = Σ − 1 / 2 [ ∑ k w k ( μ k − ∑ j w j μ j ) ( μ k − ∑ j w j μ j ) T ] Σ − 1 / 2 . C_z=\Sigma^{-1/2}\left[\sum_k w_k\left(\mu_k-\sum_jw_j\mu_j\right)\left(\mu_k-\sum_jw_j\mu_j\right)^{\mathsf{T}}\right]\Sigma^{-1/2}. C z = Σ − 1/2 k ∑ w k ( μ k − j ∑ w j μ j ) ( μ k − j ∑ w j μ j ) T Σ − 1/2 .
For v ∈ R d v\in\mathbb{R}^{d} v ∈ R d , set r k = v T Σ − 1 / 2 μ k r_k=v^{\mathsf{T}}\Sigma^{-1/2}\mu_k r k = v T Σ − 1/2 μ k , r ˉ = ∑ k w k r k \bar r=\sum_k w_kr_k r ˉ = ∑ k w k r k , r − = min k r k r_-=\min_k r_k r − = min k r k , r + = max k r k r_+=\max_k r_k r + = max k r k , and c = ( r − + r + ) / 2 c=(r_-+r_+)/2 c = ( r − + r + ) /2 . Then
0 ≤ v T C z v = ∑ k w k ( r k − r ˉ ) 2 = ∑ k w k ( r k − c ) 2 − ( r ˉ − c ) 2 ≤ ∑ k w k ( r + − r − ) 2 4 = 1 4 max i , j ∣ v T Σ − 1 / 2 ( μ i − μ j ) ∣ 2 . \begin{aligned}
0\le v^{\mathsf{T}}C_zv=\sum_k w_k(r_k-\bar r)^2=\sum_k w_k(r_k-c)^2-(\bar r-c)^2 \\
&\le\sum_k w_k\frac{(r_+-r_-)^2}{4}=\frac{1}{4}\max_{i,j}\left|v^{\mathsf{T}}\Sigma^{-1/2}(\mu_i-\mu_j)\right|^2.
\end{aligned} 0 ≤ v T C z v = k ∑ w k ( r k − r ˉ ) 2 = k ∑ w k ( r k − c ) 2 − ( r ˉ − c ) 2 ≤ k ∑ w k 4 ( r + − r − ) 2 = 4 1 i , j max v T Σ − 1/2 ( μ i − μ j ) 2 .
By the definition of maximum component separation Δ \Delta Δ , we further have
v T C z v ≤ ∥ v ∥ 2 2 4 max i , j ∥ Σ − 1 / 2 ( μ i − μ j ) ∥ 2 2 = Δ 2 4 ∥ v ∥ 2 2 . v^{\mathsf{T}}C_zv\le\frac{\lVert v\rVert_2^2}{4}\max_{i,j}\lVert\Sigma^{-1/2}(\mu_i-\mu_j)\rVert_2^2=\frac{\Delta^2}{4}\lVert v\rVert_2^2. v T C z v ≤ 4 ∥ v ∥ 2 2 i , j max ∥ Σ − 1/2 ( μ i − μ j ) ∥ 2 2 = 4 Δ 2 ∥ v ∥ 2 2 .
Consequently,
0 ⪯ C z ⪯ Δ 2 4 I d ⟹ − I d ⪯ C z − I d ⪯ ( Δ 2 4 − 1 ) I d , 0 \preceq C_z \preceq\frac{\Delta^2}{4}I_d \quad\Longrightarrow\quad-I_d \preceq C_z-I_d \preceq\left(\frac{\Delta^2}{4}-1\right)I_d, 0 ⪯ C z ⪯ 4 Δ 2 I d ⟹ − I d ⪯ C z − I d ⪯ ( 4 Δ 2 − 1 ) I d ,
∥ ∇ s ( z ) ∥ o p ≤ ∥ Σ − 1 / 2 ∥ o p 2 ∥ C z − I d ∥ o p ≤ max { 1 , Δ 2 / 4 − 1 } λ min ( Σ ) = : L . \lVert\nabla s(z)\rVert_{\mathrm{op}} \leq\lVert\Sigma^{-1/2}\rVert_{\mathrm{op}}^2 \lVert C_z-I_d\rVert_{\mathrm{op}} \leq\frac{\max\{1,\Delta^2/4-1\}}{\lambda_{\min}(\Sigma)}=:L. ∥ ∇ s ( z ) ∥ op ≤ ∥ Σ − 1/2 ∥ op 2 ∥ C z − I d ∥ op ≤ λ m i n ( Σ ) max { 1 , Δ 2 /4 − 1 } =: L .
Finally, to compute the total variance T o t V a r b a s e \mathrm{TotVar}_{\mathrm{base}} TotVar base , let P ( K = k ) = π k \mathbb{P}(K=k)=\pi_k P ( K = k ) = π k and G ∼ N ( 0 , I d ) G\sim\mathcal{N}(0,I_d) G ∼ N ( 0 , I d ) be independent. Then we have
Z = d μ K + Σ 1 / 2 G , E [ Z ] = ∑ k π k μ k + Σ 1 / 2 E [ G ] = μ ˉ . Z \overset{\mathrm{d}}{=} \mu_K+\Sigma^{1/2}G,\qquad\mathbb{E}[Z]=\sum_k\pi_k\mu_k+\Sigma^{1/2}\mathbb{E}[G]=\bar{\mu}. Z = d μ K + Σ 1/2 G , E [ Z ] = k ∑ π k μ k + Σ 1/2 E [ G ] = μ ˉ .
Therefore, we have
T o t V a r b a s e = ∑ k π k E [ ∥ μ k − μ ˉ + Σ 1 / 2 G ∥ 2 2 ] = ∑ k π k [ ∥ μ k − μ ˉ ∥ 2 2 + 2 ( μ k − μ ˉ ) ⊤ Σ 1 / 2 E [ G ] + tr ( Σ E [ G G ⊤ ] ) ] = B + tr ( Σ ) , \begin{aligned}
\mathrm{TotVar}_{\mathrm{base}}=\sum_k\pi_k\mathbb{E}\left[\lVert\mu_k-\bar{\mu}+\Sigma^{1/2}G\rVert_2^2\right] \\
&=\sum_k\pi_k\left[\lVert\mu_k-\bar{\mu}\rVert_2^2+2(\mu_k-\bar{\mu})^\top\Sigma^{1/2}\mathbb{E}[G]+\operatorname{tr}\left(\Sigma\mathbb{E}[GG^\top]\right)\right] \\
&=B+\operatorname{tr}(\Sigma),
\end{aligned} TotVar base = k ∑ π k E [ ∥ μ k − μ ˉ + Σ 1/2 G ∥ 2 2 ] = k ∑ π k [ ∥ μ k − μ ˉ ∥ 2 2 + 2 ( μ k − μ ˉ ) ⊤ Σ 1/2 E [ G ] + tr ( Σ E [ G G ⊤ ] ) ] = B + tr ( Σ ) ,
L T o t V a r b a s e = tr ( Σ ) + B λ min ( Σ ) max { 1 , Δ 2 4 − 1 } . L\mathrm{TotVar}_{\mathrm{base}}=\frac{\operatorname{tr}(\Sigma)+B}{\lambda_{\min}(\Sigma)}\max\left\{1,\frac{\Delta^2}{4}-1\right\}. L TotVar base = λ m i n ( Σ ) tr ( Σ ) + B max { 1 , 4 Δ 2 − 1 } .
Proof of Lemma 1
Since t n > t n − 1 t_n>t_{n-1} t n > t n − 1 , Assumption 3 yields
t n − t n − 1 ≤ c s log ( N + 1 ) N + 1 t n − 1 ( 1 − t n ) ≤ c s log ( N + 1 ) N + 1 t n − 1 ( 1 − t n − 1 ) . t_n-t_{n-1}\leq\frac{c_s\log(N+1)}{N+1}t_{n-1}(1-t_n)\leq\frac{c_s\log(N+1)}{N+1}t_{n-1}(1-t_{n-1}). t n − t n − 1 ≤ N + 1 c s log ( N + 1 ) t n − 1 ( 1 − t n ) ≤ N + 1 c s log ( N + 1 ) t n − 1 ( 1 − t n − 1 ) .
Consequently, for every r ≥ 1 r\geq1 r ≥ 1 ,
∑ n = 1 N { t n − t n − 1 t n − 1 ( 1 − t n ) } r ≤ N { c s log ( N + 1 ) N + 1 } r ≤ c s r log r ( N + 1 ) ( N + 1 ) r − 1 . \sum_{n=1}^{N}\left\{\frac{t_n-t_{n-1}}{t_{n-1}(1-t_n)}\right\}^r\leq N\left\{\frac{c_s\log(N+1)}{N+1}\right\}^r\leq c_s^r\frac{\log^r(N+1)}{(N+1)^{r-1}}. n = 1 ∑ N { t n − 1 ( 1 − t n ) t n − t n − 1 } r ≤ N { N + 1 c s log ( N + 1 ) } r ≤ c s r ( N + 1 ) r − 1 log r ( N + 1 ) .
This proves (23) .
Proof of Lemma 3
Since
p Y 0 ( y ) = 1 ( 1 − t 0 ) d p b a s e ( y 1 − t 0 ) , p X t 0 ∣ X ∗ ( y ∣ x ∗ ) = 1 ( 1 − t 0 ) d p b a s e ( y − t 0 x ∗ 1 − t 0 ) , p_{Y_0}(y)=\frac{1}{(1-t_0)^d}p_{\mathrm{base}}\left(\frac{y}{1-t_0}\right),\qquad p_{X_{t_0}\mid X_*}(y\mid x_*)=\frac{1}{(1-t_0)^d}p_{\mathrm{base}}\left(\frac{y-t_0x_*}{1-t_0}\right), p Y 0 ( y ) = ( 1 − t 0 ) d 1 p base ( 1 − t 0 y ) , p X t 0 ∣ X ∗ ( y ∣ x ∗ ) = ( 1 − t 0 ) d 1 p base ( 1 − t 0 y − t 0 x ∗ ) ,
we have
p X t 0 ∣ X ∗ ( y ∣ x ∗ ) p Y 0 ( y ) = 1 ( 1 − t 0 ) d p b a s e ( y − t 0 x ∗ 1 − t 0 ) 1 ( 1 − t 0 ) d p b a s e ( y 1 − t 0 ) = p b a s e ( y − t 0 x ∗ 1 − t 0 ) p b a s e ( y 1 − t 0 ) . \frac{p_{X_{t_0}\mid X_*}(y\mid x_*)}{p_{Y_0}(y)}=\frac{\frac{1}{(1-t_0)^d}p_{\mathrm{base}}\left(\frac{y-t_0x_*}{1-t_0}\right)}{\frac{1}{(1-t_0)^d}p_{\mathrm{base}}\left(\frac{y}{1-t_0}\right)}=\frac{p_{\mathrm{base}}\left(\frac{y-t_0x_*}{1-t_0}\right)}{p_{\mathrm{base}}\left(\frac{y}{1-t_0}\right)}. p Y 0 ( y ) p X t 0 ∣ X ∗ ( y ∣ x ∗ ) = ( 1 − t 0 ) d 1 p base ( 1 − t 0 y ) ( 1 − t 0 ) d 1 p base ( 1 − t 0 y − t 0 x ∗ ) = p base ( 1 − t 0 y ) p base ( 1 − t 0 y − t 0 x ∗ ) .
For fixed z , x ∗ ∈ R d z,x_*\in\mathbb{R}^d z , x ∗ ∈ R d , (93) implies that
log p b a s e ( z ) p b a s e ( z + t 0 x ∗ / ( 1 − t 0 ) ) ≤ − t 0 1 − t 0 x ∗ ⊤ s ( z ) + L t 0 2 2 ( 1 − t 0 ) 2 ∥ x ∗ ∥ 2 2 . (63) \log\frac{p_{\mathrm{base}}(z)}{p_{\mathrm{base}}(z+t_0x_*/(1-t_0))}\leq-\frac{t_0}{1-t_0}x_*^\top s(z)+\frac{Lt_0^2}{2(1-t_0)^2}\lVert x_*\rVert_2^2. \tag*{(63)} log p base ( z + t 0 x ∗ / ( 1 − t 0 )) p base ( z ) ≤ − 1 − t 0 t 0 x ∗ ⊤ s ( z ) + 2 ( 1 − t 0 ) 2 L t 0 2 ∥ x ∗ ∥ 2 2 . (63)
∣ log p b a s e ( z ) p b a s e ( z + t 0 x ∗ / ( 1 − t 0 ) ) ∣ ≤ t 0 1 − t 0 ∥ x ∗ ∥ 2 ∥ s ( z ) ∥ 2 + L t 0 2 2 ( 1 − t 0 ) 2 ∥ x ∗ ∥ 2 2 . (64) \left|\log\frac{p_{\mathrm{base}}(z)}{p_{\mathrm{base}}(z+t_0x_*/(1-t_0))}\right|\leq\frac{t_0}{1-t_0}\lVert x_*\rVert_2\lVert s(z)\rVert_2+\frac{Lt_0^2}{2(1-t_0)^2}\lVert x_*\rVert_2^2. \tag*{(64)} log p base ( z + t 0 x ∗ / ( 1 − t 0 )) p base ( z ) ≤ 1 − t 0 t 0 ∥ x ∗ ∥ 2 ∥ s ( z ) ∥ 2 + 2 ( 1 − t 0 ) 2 L t 0 2 ∥ x ∗ ∥ 2 2 . (64)
Therefore, we can bound the KL divergence as follows:
KL ( P X t 0 ∣ X ⋆ ( ⋅ ∣ x ⋆ ) ∥ P Y 0 ) = ∫ 1 ( 1 − t 0 ) d p b a s e ( y − t 0 x ⋆ 1 − t 0 ) log p b a s e ( ( y − t 0 x ⋆ ) / ( 1 − t 0 ) ) p b a s e ( y / ( 1 − t 0 ) ) d y = ∫ p b a s e ( z ) log p b a s e ( z ) p b a s e ( z + t 0 x ⋆ / ( 1 − t 0 ) ) d z ≤ − t 0 1 − t 0 x ⋆ T E [ s ( Z ) ] + L t 0 2 2 ( 1 − t 0 ) 2 ∥ x ⋆ ∥ 2 2 = xref‘(24)‘ L t 0 2 2 ( 1 − t 0 ) 2 ∥ x ⋆ ∥ 2 2 . (65) \begin{aligned}
\operatorname{KL}(P_{X_{t_0}\mid X_\star}(\cdot\mid x_\star)\|P_{Y_0})
&= \int\frac{1}{(1-t_0)^d}p_{\mathrm{base}}\left(\frac{y-t_0x_\star}{1-t_0}\right)\log\frac{p_{\mathrm{base}}\left((y-t_0x_\star)/(1-t_0)\right)}{p_{\mathrm{base}}(y/(1-t_0))}\,\mathrm{d}y\\
&= \int p_{\mathrm{base}}(z)\log\frac{p_{\mathrm{base}}(z)}{p_{\mathrm{base}}(z+t_0x_\star/(1-t_0))}\,\mathrm{d}z\\
&\le-\frac{t_0}{1-t_0}x_\star^{\mathsf{T}}\mathbb{E}[s(Z)]+\frac{Lt_0^2}{2(1-t_0)^2}\lVert x_\star\rVert_2^2\\
&\overset{\text{\scriptsize{xref}`(24)`}}{=} \frac{Lt_0^2}{2(1-t_0)^2}\lVert x_\star\rVert_2^2. \tag*{(65)}
\end{aligned} KL ( P X t 0 ∣ X ⋆ ( ⋅ ∣ x ⋆ ) ∥ P Y 0 ) = ∫ ( 1 − t 0 ) d 1 p base ( 1 − t 0 y − t 0 x ⋆ ) log p base ( y / ( 1 − t 0 )) p base ( ( y − t 0 x ⋆ ) / ( 1 − t 0 ) ) d y = ∫ p base ( z ) log p base ( z + t 0 x ⋆ / ( 1 − t 0 )) p base ( z ) d z ≤ − 1 − t 0 t 0 x ⋆ T E [ s ( Z )] + 2 ( 1 − t 0 ) 2 L t 0 2 ∥ x ⋆ ∥ 2 2 = xref ‘(24)‘ 2 ( 1 − t 0 ) 2 L t 0 2 ∥ x ⋆ ∥ 2 2 . (65)
Since X ⋆ X_\star X ⋆ and Z Z Z are independent, we have
p t 0 ( y ) = ∫ p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) P d a t a ( d x ⋆ ) . p_{t_0}(y)=\int p_{X_{t_0}\mid X_\star}(y\mid x_\star)\,P_{\mathrm{data}}(\mathrm{d}x_\star). p t 0 ( y ) = ∫ p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) P data ( d x ⋆ ) .
Furthermore, recognizing that the function r ↦ r log r r\mapsto r\log r r ↦ r log r is convex, Jensen’s inequality yields
p t 0 ( y ) p Y 0 ( y ) log p t 0 ( y ) p Y 0 ( y ) = [ ∫ p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) p Y 0 ( y ) P d a t a ( d x ⋆ ) ] log [ ∫ p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) p Y 0 ( y ) P d a t a ( d x ⋆ ) ] ≤ ∫ p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) p Y 0 ( y ) log p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) p Y 0 ( y ) P d a t a ( d x ⋆ ) , \begin{aligned}
\frac{p_{t_0}(y)}{p_{Y_0}(y)}\log\frac{p_{t_0}(y)}{p_{Y_0}(y)}
=
\left[\int\frac{p_{X_{t_0}\mid X_\star}(y\mid x_\star)}{p_{Y_0}(y)}\,P_{\mathrm{data}}(\mathrm{d}x_\star)\right]
\log\left[\int\frac{p_{X_{t_0}\mid X_\star}(y\mid x_\star)}{p_{Y_0}(y)}\,P_{\mathrm{data}}(\mathrm{d}x_\star)\right] \\
&\le\int\frac{p_{X_{t_0}\mid X_\star}(y\mid x_\star)}{p_{Y_0}(y)}
\log\frac{p_{X_{t_0}\mid X_\star}(y\mid x_\star)}{p_{Y_0}(y)}\,P_{\mathrm{data}}(\mathrm{d}x_\star),
\end{aligned} p Y 0 ( y ) p t 0 ( y ) log p Y 0 ( y ) p t 0 ( y ) = [ ∫ p Y 0 ( y ) p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) P data ( d x ⋆ ) ] log [ ∫ p Y 0 ( y ) p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) P data ( d x ⋆ ) ] ≤ ∫ p Y 0 ( y ) p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) log p Y 0 ( y ) p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) P data ( d x ⋆ ) ,
or equivalently,
p t 0 ( y ) log p t 0 ( y ) p Y 0 ( y ) ≤ ∫ p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) log p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) p Y 0 ( y ) P d a t a ( d x ⋆ ) . p_{t_0}(y)\log\frac{p_{t_0}(y)}{p_{Y_0}(y)}
\le\int p_{X_{t_0}\mid X_\star}(y\mid x_\star)\log\frac{p_{X_{t_0}\mid X_\star}(y\mid x_\star)}{p_{Y_0}(y)}\,P_{\mathrm{data}}(\mathrm{d}x_\star). p t 0 ( y ) log p Y 0 ( y ) p t 0 ( y ) ≤ ∫ p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) log p Y 0 ( y ) p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) P data ( d x ⋆ ) .
In view of (64) and Assumption 2, we know that
∫ ∫ p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) ∣ log p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) p Y 0 ( y ) ∣ d y P d a t a ( d x ⋆ ) ≤ t 0 1 − t 0 L d E [ ∥ X ⋆ ∥ 2 ] + L t 0 2 2 ( 1 − t 0 ) 2 E [ ∥ X ⋆ ∥ 2 2 ] < ∞ . \int\int p_{X_{t_0}\mid X_\star}(y\mid x_\star)\left|\log\frac{p_{X_{t_0}\mid X_\star}(y\mid x_\star)}{p_{Y_0}(y)}\right|\,\mathrm{d}y\,P_{\mathrm{data}}(\mathrm{d}x_\star)
\le\frac{t_0}{1-t_0}\sqrt{L}d\,\mathbb{E}\left[\lVert X_\star\rVert_2\right]+\frac{Lt_0^2}{2(1-t_0)^2}\mathbb{E}\left[\lVert X_\star\rVert_2^2\right]<\infty. ∫∫ p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) log p Y 0 ( y ) p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) d y P data ( d x ⋆ ) ≤ 1 − t 0 t 0 L d E [ ∥ X ⋆ ∥ 2 ] + 2 ( 1 − t 0 ) 2 L t 0 2 E [ ∥ X ⋆ ∥ 2 2 ] < ∞.
Here E [ ∥ X ⋆ ∥ 2 ] ≤ ( E [ ∥ X ⋆ ∥ 2 2 ] ) 1 / 2 < ∞ \mathbb{E}\left[\lVert X_\star\rVert_2\right]\le\left(\mathbb{E}\left[\lVert X_\star\rVert_2^2\right]\right)^{1/2}<\infty E [ ∥ X ⋆ ∥ 2 ] ≤ ( E [ ∥ X ⋆ ∥ 2 2 ] ) 1/2 < ∞ by Assumption 2. In addition, we know that
p t 0 ( y ) log p t 0 ( y ) p Y 0 ( y ) ≥ − p Y 0 ( y ) e p_{t_0}(y)\log\frac{p_{t_0}(y)}{p_{Y_0}(y)}\ge-\frac{p_{Y_0}(y)}{e} p t 0 ( y ) log p Y 0 ( y ) p t 0 ( y ) ≥ − e p Y 0 ( y )
since r log r ≥ − 1 / e r\log r\ge-1/e r log r ≥ − 1/ e . Applying Fubini’s theorem yields
KL ( P t 0 ∥ P Y 0 ) = ∫ p t 0 ( y ) log p t 0 ( y ) p Y 0 ( y ) d y ≤ ∫ ( ∫ p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) log p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) p Y 0 ( y ) P d a t a ( d x ⋆ ) ) d y = ∫ ( ∫ p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) log p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) p Y 0 ( y ) d y ) P d a t a ( d x ⋆ ) = ∫ KL ( P X t 0 ∣ X ⋆ ( ⋅ ∣ x ⋆ ) ∥ P Y 0 ) P d a t a ( d x ⋆ ) ≤ xref‘(62)‘ ∫ L t 0 2 2 ( 1 − t 0 ) 2 ∥ x ⋆ ∥ 2 2 P d a t a ( d x ⋆ ) = L t 0 2 2 ( 1 − t 0 ) 2 E [ ∥ X ⋆ ∥ 2 2 ] . \begin{aligned}
\operatorname{KL}(P_{t_0}\|P_{Y_0})
&=\int p_{t_0}(y)\log\frac{p_{t_0}(y)}{p_{Y_0}(y)}\,\mathrm{d}y\\
&\le\int\left(\int p_{X_{t_0}\mid X_\star}(y\mid x_\star)\log\frac{p_{X_{t_0}\mid X_\star}(y\mid x_\star)}{p_{Y_0}(y)}\,P_{\mathrm{data}}(\mathrm{d}x_\star)\right)\mathrm{d}y\\
&=\int\left(\int p_{X_{t_0}\mid X_\star}(y\mid x_\star)\log\frac{p_{X_{t_0}\mid X_\star}(y\mid x_\star)}{p_{Y_0}(y)}\,\mathrm{d}y\right)P_{\mathrm{data}}(\mathrm{d}x_\star)\\
&=\int\operatorname{KL}(P_{X_{t_0}\mid X_\star}(\cdot\mid x_\star)\|P_{Y_0})\,P_{\mathrm{data}}(\mathrm{d}x_\star)\\
&\overset{\text{\scriptsize{xref}`(62)`}}{\le}\int\frac{Lt_0^2}{2(1-t_0)^2}\lVert x_\star\rVert_2^2\,P_{\mathrm{data}}(\mathrm{d}x_\star)\\
&=\frac{Lt_0^2}{2(1-t_0)^2}\mathbb{E}\left[\lVert X_\star\rVert_2^2\right].
\end{aligned} KL ( P t 0 ∥ P Y 0 ) = ∫ p t 0 ( y ) log p Y 0 ( y ) p t 0 ( y ) d y ≤ ∫ ( ∫ p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) log p Y 0 ( y ) p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) P data ( d x ⋆ ) ) d y = ∫ ( ∫ p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) log p Y 0 ( y ) p X t 0 ∣ X ⋆ ( y ∣ x ⋆ ) d y ) P data ( d x ⋆ ) = ∫ KL ( P X t 0 ∣ X ⋆ ( ⋅ ∣ x ⋆ ) ∥ P Y 0 ) P data ( d x ⋆ ) ≤ xref ‘(62)‘ ∫ 2 ( 1 − t 0 ) 2 L t 0 2 ∥ x ⋆ ∥ 2 2 P data ( d x ⋆ ) = 2 ( 1 − t 0 ) 2 L t 0 2 E [ ∥ X ⋆ ∥ 2 2 ] .
This proves (28) .
Pinsker’s inequality further shows that
TV ( P Y 0 , P t 0 ) = TV ( P t 0 , P Y 0 ) ≤ 1 2 KL ( P t 0 ∥ P Y 0 ) ≤ L t 0 2 4 ( 1 − t 0 ) 2 E [ ∥ X ⋆ ∥ 2 2 ] = t 0 2 ( 1 − t 0 ) L E [ ∥ X ⋆ ∥ 2 2 ] , \operatorname{TV}(P_{Y_0},P_{t_0})=\operatorname{TV}(P_{t_0},P_{Y_0})\le\sqrt{\frac{1}{2}\operatorname{KL}(P_{t_0}\|P_{Y_0})}\le\sqrt{\frac{Lt_0^2}{4(1-t_0)^2}\mathbb{E}\left[\lVert X^\star\rVert_2^2\right]}=\frac{t_0}{2(1-t_0)}\sqrt{L\mathbb{E}\left[\lVert X^\star\rVert_2^2\right]}, TV ( P Y 0 , P t 0 ) = TV ( P t 0 , P Y 0 ) ≤ 2 1 KL ( P t 0 ∥ P Y 0 ) ≤ 4 ( 1 − t 0 ) 2 L t 0 2 E [ ∥ X ⋆ ∥ 2 2 ] = 2 ( 1 − t 0 ) t 0 L E [ ∥ X ⋆ ∥ 2 2 ] ,
Since t 0 ≤ ( N + 1 ) − c 0 / 2 < 1 / 2 t_0\le(N+1)^{-c_0/2}<1/2 t 0 ≤ ( N + 1 ) − c 0 /2 < 1/2 , 1 / ( 2 ( 1 − t 0 ) ) ≤ 1 1/(2(1-t_0))\le1 1/ ( 2 ( 1 − t 0 )) ≤ 1 . Consequently, by Assumptions 2 and 3, we have
TV ( P Y 0 , P t 0 ) ≤ t 0 L E [ ∥ X ⋆ ∥ 2 2 ] ≤ t 0 ( N + 1 ) c R / 2 L E [ ∥ Z − E [ Z ] ∥ 2 2 ] ≤ ( N + 1 ) − ( c 0 − c R ) / 2 L E [ ∥ Z − E [ Z ] ∥ 2 2 ] ≤ L E [ ∥ Z − E [ Z ] ∥ 2 2 ] ( N + 1 ) 2 . \begin{aligned}
\operatorname{TV}(P_{Y_0},P_{t_0})\le t_0\sqrt{L\mathbb{E}\left[\lVert X^\star\rVert_2^2\right]} \\
&\le t_0(N+1)^{c_R/2}\sqrt{L\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]} \\
&\le(N+1)^{-(c_0-c_R)/2}\sqrt{L\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]} \\
&\le\frac{\sqrt{L\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]}}{(N+1)^2}.
\end{aligned} TV ( P Y 0 , P t 0 ) ≤ t 0 L E [ ∥ X ⋆ ∥ 2 2 ] ≤ t 0 ( N + 1 ) c R /2 L E [ ∥ Z − E [ Z ] ∥ 2 2 ] ≤ ( N + 1 ) − ( c 0 − c R ) /2 L E [ ∥ Z − E [ Z ] ∥ 2 2 ] ≤ ( N + 1 ) 2 L E [ ∥ Z − E [ Z ] ∥ 2 2 ] .
Therefore, we have finished the proof of (29) .
Proof of Lemma 4
For every y , z ∈ R d y,z\in\mathbb{R}^d y , z ∈ R d , by virtue of (93) , one has
p b a s e ( y ) ≥ p b a s e ( z ) exp ( ∥ s ( z ) ∥ 2 2 2 L ) exp ( − L 2 ∥ y − z − s ( z ) L ∥ 2 2 ) . p_{\mathrm{base}}(y)\ge p_{\mathrm{base}}(z)\exp\left(\frac{\lVert s(z)\rVert_2^2}{2L}\right)\exp\left(-\frac{L}{2}\left\lVert y-z-\frac{s(z)}{L}\right\rVert_2^2\right). p base ( y ) ≥ p base ( z ) exp ( 2 L ∥ s ( z ) ∥ 2 2 ) exp ( − 2 L y − z − L s ( z ) 2 2 ) .
Multiplying by 1 + ∥ y − E [ Z ] ∥ 2 2 1+\lVert y-\mathbb{E}[Z]\rVert_2^2 1 + ∥ y − E [ Z ] ∥ 2 2 and integrating in y y y on both sides, we obtain
1 + E [ ∥ Z − E [ Z ] ∥ 2 2 ] ≥ p b a s e ( z ) exp ( ∥ s ( z ) ∥ 2 2 2 L ) ∫ R d ( 1 + ∥ y − E [ Z ] ∥ 2 2 ) exp ( − L 2 ∥ y − z − s ( z ) L ∥ 2 2 ) d y . 1+\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]\ge p_{\mathrm{base}}(z)\exp\left(\frac{\lVert s(z)\rVert_2^2}{2L}\right)\int_{\mathbb{R}^d}\left(1+\lVert y-\mathbb{E}[Z]\rVert_2^2\right)\exp\left(-\frac{L}{2}\left\lVert y-z-\frac{s(z)}{L}\right\rVert_2^2\right)\,dy. 1 + E [ ∥ Z − E [ Z ] ∥ 2 2 ] ≥ p base ( z ) exp ( 2 L ∥ s ( z ) ∥ 2 2 ) ∫ R d ( 1 + ∥ y − E [ Z ] ∥ 2 2 ) exp ( − 2 L y − z − L s ( z ) 2 2 ) d y .
We let w = y − z − s ( z ) / L w=y-z-s(z)/L w = y − z − s ( z ) / L . Then we have
∫ R d ( 1 + ∥ y − E [ Z ] ∥ 2 2 ) exp ( − L 2 ∥ y − z − s ( z ) L ∥ 2 2 ) d y = ∫ R d ( 1 + ∥ w + z + s ( z ) L − E [ Z ] ∥ 2 2 ) exp ( − L 2 ∥ w ∥ 2 2 ) d w = ( 1 + ∥ z + s ( z ) L − E [ Z ] ∥ 2 2 ) ∫ R d exp ( − L 2 ∥ w ∥ 2 2 ) d w + 2 ( z + s ( z ) L − E [ Z ] ) T ∫ R d w exp ( − L 2 ∥ w ∥ 2 2 ) d w + ∫ R d ∥ w ∥ 2 2 exp ( − L 2 ∥ w ∥ 2 2 ) d w = ( 2 π L ) d / 2 ( 1 + ∥ z + s ( z ) L − E [ Z ] ∥ 2 2 + d L ) , \begin{aligned}
\int_{\mathbb{R}^d}\left(1+\lVert y-\mathbb{E}[Z]\rVert_2^2\right)\exp\left(-\frac{L}{2}\left\lVert y-z-\frac{s(z)}{L}\right\rVert_2^2\right)\,dy
&=\int_{\mathbb{R}^d}\left(1+\left\lVert w+z+\frac{s(z)}{L}-\mathbb{E}[Z]\right\rVert_2^2\right)\exp\left(-\frac{L}{2}\lVert w\rVert_2^2\right)\,dw\\
&=\left(1+\left\lVert z+\frac{s(z)}{L}-\mathbb{E}[Z]\right\rVert_2^2\right)\int_{\mathbb{R}^d}\exp\left(-\frac{L}{2}\lVert w\rVert_2^2\right)\,dw\\
&\quad+2\left(z+\frac{s(z)}{L}-\mathbb{E}[Z]\right)^{\mathsf{T}}\int_{\mathbb{R}^d}w\exp\left(-\frac{L}{2}\lVert w\rVert_2^2\right)\,dw\\
&\quad+\int_{\mathbb{R}^d}\lVert w\rVert_2^2\exp\left(-\frac{L}{2}\lVert w\rVert_2^2\right)\,dw\\
&=\left(\frac{2\pi}{L}\right)^{d/2}\left(1+\left\lVert z+\frac{s(z)}{L}-\mathbb{E}[Z]\right\rVert_2^2+\frac{d}{L}\right),
\end{aligned} ∫ R d ( 1 + ∥ y − E [ Z ] ∥ 2 2 ) exp ( − 2 L y − z − L s ( z ) 2 2 ) d y = ∫ R d ( 1 + w + z + L s ( z ) − E [ Z ] 2 2 ) exp ( − 2 L ∥ w ∥ 2 2 ) d w = ( 1 + z + L s ( z ) − E [ Z ] 2 2 ) ∫ R d exp ( − 2 L ∥ w ∥ 2 2 ) d w + 2 ( z + L s ( z ) − E [ Z ] ) T ∫ R d w exp ( − 2 L ∥ w ∥ 2 2 ) d w + ∫ R d ∥ w ∥ 2 2 exp ( − 2 L ∥ w ∥ 2 2 ) d w = ( L 2 π ) d /2 ( 1 + z + L s ( z ) − E [ Z ] 2 2 + L d ) ,
where the last equation makes use of
∫ R d e − L ∥ w ∥ 2 2 / 2 d w = ( 2 π L ) d / 2 , \int_{\mathbb{R}^d}e^{-L\lVert w\rVert_2^2/2}\,dw=\left(\frac{2\pi}{L}\right)^{d/2}, ∫ R d e − L ∥ w ∥ 2 2 /2 d w = ( L 2 π ) d /2 ,
∫ R d w exp ( − L 2 ∥ w ∥ 2 2 ) d w = 0 , \int_{\mathbb{R}^d} w\exp\left(-\frac{L}{2}\lVert w\rVert_2^2\right)\,dw=0, ∫ R d w exp ( − 2 L ∥ w ∥ 2 2 ) d w = 0 ,
∫ R d ∥ w ∥ 2 2 e − L ∥ w ∥ 2 2 / 2 d w = d L ( 2 π L ) d / 2 . \int_{\mathbb{R}^d}\lVert w\rVert_2^2e^{-L\lVert w\rVert_2^2/2}\,dw=\frac{d}{L}\left(\frac{2\pi}{L}\right)^{d/2}. ∫ R d ∥ w ∥ 2 2 e − L ∥ w ∥ 2 2 /2 d w = L d ( L 2 π ) d /2 .
Therefore, we have
p b a s e ( z ) exp ( ∥ s ( z ) ∥ 2 2 2 L ) ( 1 + ∥ z + s ( z ) L − E [ Z ] ∥ 2 2 + d L ) ≤ ( L 2 π ) d / 2 ( 1 + E [ ∥ Z − E [ Z ] ∥ 2 2 ] ) . p_{\mathrm{base}}(z)\exp\left(\frac{\lVert s(z)\rVert_2^2}{2L}\right)\left(1+\left\lVert z+\frac{s(z)}{L}-\mathbb{E}[Z]\right\rVert_2^2+\frac{d}{L}\right)\leq\left(\frac{L}{2\pi}\right)^{d/2}\left(1+\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]\right). p base ( z ) exp ( 2 L ∥ s ( z ) ∥ 2 2 ) ( 1 + z + L s ( z ) − E [ Z ] 2 2 + L d ) ≤ ( 2 π L ) d /2 ( 1 + E [ ∥ Z − E [ Z ] ∥ 2 2 ] ) .
Moreover, the inequality ∥ v + w ∥ 2 2 ≤ 2 ∥ v ∥ 2 2 + 2 ∥ w ∥ 2 2 \lVert v+w\rVert_2^2\leq2\lVert v\rVert_2^2+2\lVert w\rVert_2^2 ∥ v + w ∥ 2 2 ≤ 2 ∥ v ∥ 2 2 + 2 ∥ w ∥ 2 2 implies
1 + ∥ z − E [ Z ] ∥ 2 2 ≤ 1 + 2 ∥ z + s ( z ) L − E [ Z ] ∥ 2 2 + 2 ∥ s ( z ) ∥ 2 2 L 2 1+\lVert z-\mathbb{E}[Z]\rVert_2^2\leq1+2\left\lVert z+\frac{s(z)}{L}-\mathbb{E}[Z]\right\rVert_2^2+\frac{2\lVert s(z)\rVert_2^2}{L^2} 1 + ∥ z − E [ Z ] ∥ 2 2 ≤ 1 + 2 z + L s ( z ) − E [ Z ] 2 2 + L 2 2 ∥ s ( z ) ∥ 2 2
≤ 2 ( 1 + ∥ s ( z ) ∥ 2 2 L 2 ) ( 1 + ∥ z + s ( z ) L − E [ Z ] ∥ 2 2 + d L ) . \leq2\left(1+\frac{\lVert s(z)\rVert_2^2}{L^2}\right)\left(1+\left\lVert z+\frac{s(z)}{L}-\mathbb{E}[Z]\right\rVert_2^2+\frac{d}{L}\right). ≤ 2 ( 1 + L 2 ∥ s ( z ) ∥ 2 2 ) ( 1 + z + L s ( z ) − E [ Z ] 2 2 + L d ) .
Combining the last two inequalities leads to
p b a s e ( z ) ( 1 + ∥ z − E [ Z ] ∥ 2 2 ) ( 1 + ∥ s ( z ) ∥ 2 2 ) p_{\mathrm{base}}(z)(1+\lVert z-\mathbb{E}[Z]\rVert_2^2)(1+\lVert s(z)\rVert_2^2) p base ( z ) ( 1 + ∥ z − E [ Z ] ∥ 2 2 ) ( 1 + ∥ s ( z ) ∥ 2 2 )
≤ 2 ( L 2 π ) d / 2 ( 1 + E [ ∥ Z − E [ Z ] ∥ 2 2 ] ) ( 1 + ∥ s ( z ) ∥ 2 2 L 2 ) ( 1 + ∥ s ( z ) ∥ 2 2 ) exp ( − ∥ s ( z ) ∥ 2 2 2 L ) ≤ C d , L ( 1 + ∥ s ( z ) ∥ 2 2 ) exp ( − ∥ s ( z ) ∥ 2 2 2 L ) ≤ C d , L ( 1 + 16 L 2 e 2 ) , \begin{aligned}
\leq2\left(\frac{L}{2\pi}\right)^{d/2}\left(1+\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]\right)\left(1+\frac{\lVert s(z)\rVert_2^2}{L^2}\right)(1+\lVert s(z)\rVert_2^2)\exp\left(-\frac{\lVert s(z)\rVert_2^2}{2L}\right) \\
&\leq C_{d,L}(1+\lVert s(z)\rVert_2^2)\exp\left(-\frac{\lVert s(z)\rVert_2^2}{2L}\right)\leq C_{d,L}\left(1+\frac{16L^2}{e^2}\right),
\end{aligned} ≤ 2 ( 2 π L ) d /2 ( 1 + E [ ∥ Z − E [ Z ] ∥ 2 2 ] ) ( 1 + L 2 ∥ s ( z ) ∥ 2 2 ) ( 1 + ∥ s ( z ) ∥ 2 2 ) exp ( − 2 L ∥ s ( z ) ∥ 2 2 ) ≤ C d , L ( 1 + ∥ s ( z ) ∥ 2 2 ) exp ( − 2 L ∥ s ( z ) ∥ 2 2 ) ≤ C d , L ( 1 + e 2 16 L 2 ) ,
where C d , L C_{d,L} C d , L is a constant depending on L L L , d d d and the source moments. Here the penultimate inequality makes use of the fact
( 1 + x L 2 ) ( 1 + x ) = 1 + ( 1 + 1 L 2 ) x + x 2 L 2 ≤ 2 ( 1 + 1 L 2 ) ( 1 + x 2 ) \left(1+\frac{x}{L^2}\right)(1+x)=1+\left(1+\frac{1}{L^2}\right)x+\frac{x^2}{L^2}\leq2\left(1+\frac{1}{L^2}\right)(1+x^2) ( 1 + L 2 x ) ( 1 + x ) = 1 + ( 1 + L 2 1 ) x + L 2 x 2 ≤ 2 ( 1 + L 2 1 ) ( 1 + x 2 )
for x ≥ 0 x\geq0 x ≥ 0 ; the last inequality comes from sup u ≥ 0 u 2 e − u / ( 2 L ) = 16 L 2 / e 2 \sup_{u\geq0}u^2e^{-u/(2L)}=16L^2/e^2 sup u ≥ 0 u 2 e − u / ( 2 L ) = 16 L 2 / e 2 . As a result, we know that
sup z ∈ R d p b a s e ( z ) ( 1 + ∥ z − E [ Z ] ∥ 2 2 ) ( 1 + ∥ s ( z ) ∥ 2 2 ) < ∞ . (66) \sup_{z\in\mathbb{R}^d}p_{\mathrm{base}}(z)(1+\lVert z-\mathbb{E}[Z]\rVert_2^2)(1+\lVert s(z)\rVert_2^2)<\infty. \tag*{(66)} z ∈ R d sup p base ( z ) ( 1 + ∥ z − E [ Z ] ∥ 2 2 ) ( 1 + ∥ s ( z ) ∥ 2 2 ) < ∞. (66)
Proving p t p_t p t and E [ X ⋆ ∣ X t = x ] \mathbb{E}[X_\star\mid X_t=x] E [ X ⋆ ∣ X t = x ] are C 1 C^1 C 1 . For any fixed t ∈ ( 0 , 1 ) t\in(0,1) t ∈ ( 0 , 1 ) , the mixture density and the posterior mean are
p t ( x ) = 1 ( 1 − t ) d ∫ p b a s e ( x − t x ⋆ 1 − t ) P d a t a ( d x ⋆ ) , E [ X ⋆ ∣ X t = x ] = ∫ x ⋆ p b a s e ( x − t x ⋆ 1 − t ) P d a t a ( d x ⋆ ) ∫ p b a s e ( x − t x ⋆ 1 − t ) P d a t a ( d x ⋆ ) . p_t(x)=\frac{1}{(1-t)^d}\int p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)P_{\mathrm{data}}(dx_\star),\qquad\mathbb{E}[X_\star\mid X_t=x]=\frac{\int x_\star p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)P_{\mathrm{data}}(dx_\star)}{\int p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)P_{\mathrm{data}}(dx_\star)}. p t ( x ) = ( 1 − t ) d 1 ∫ p base ( 1 − t x − t x ⋆ ) P data ( d x ⋆ ) , E [ X ⋆ ∣ X t = x ] = ∫ p base ( 1 − t x − t x ⋆ ) P data ( d x ⋆ ) ∫ x ⋆ p base ( 1 − t x − t x ⋆ ) P data ( d x ⋆ ) .
For ∥ x ∥ 2 ≤ R + 1 \lVert x\rVert_2\leq R+1 ∥ x ∥ 2 ≤ R + 1 , the identity
x ⋆ = x − ( 1 − t ) E [ Z ] t − 1 − t t ( x − t x ⋆ 1 − t − E [ Z ] ) x_\star=\frac{x-(1-t)\mathbb{E}[Z]}{t}-\frac{1-t}{t}\left(\frac{x-tx_\star}{1-t}-\mathbb{E}[Z]\right) x ⋆ = t x − ( 1 − t ) E [ Z ] − t 1 − t ( 1 − t x − t x ⋆ − E [ Z ] )
implies
1 + ∥ x ⋆ ∥ 2 ≤ 1 + R + 1 + ( 1 − t ) ∥ E [ Z ] ∥ 2 t + 1 − t t ∥ x − t x ⋆ 1 − t − E [ Z ] ∥ 2 ≤ ( 1 + R + 1 + ( 1 − t ) ∥ E [ Z ] ∥ 2 t + 1 − t t ) ( 1 + ∥ x − t x ⋆ 1 − t − E [ Z ] ∥ 2 2 ) = R + 2 + ( 1 − t ) ∥ E [ Z ] ∥ 2 t ( 1 + ∥ x − t x ⋆ 1 − t − E [ Z ] ∥ 2 2 ) . (67) \begin{aligned}
1+\lVert x_\star\rVert_2\leq1+\frac{R+1+(1-t)\lVert\mathbb{E}[Z]\rVert_2}{t}+\frac{1-t}{t}\left\lVert\frac{x-tx_\star}{1-t}-\mathbb{E}[Z]\right\rVert_2 \\
&\leq\left(1+\frac{R+1+(1-t)\lVert\mathbb{E}[Z]\rVert_2}{t}+\frac{1-t}{t}\right)\left(1+\left\lVert\frac{x-tx_\star}{1-t}-\mathbb{E}[Z]\right\rVert_2^2\right) \\
&= \frac{R+2+(1-t)\lVert\mathbb{E}[Z]\rVert_2}{t}\left(1+\left\lVert\frac{x-tx_\star}{1-t}-\mathbb{E}[Z]\right\rVert_2^2\right). \tag*{(67)}
\end{aligned} 1 + ∥ x ⋆ ∥ 2 ≤ 1 + t R + 1 + ( 1 − t ) ∥ E [ Z ] ∥ 2 + t 1 − t 1 − t x − t x ⋆ − E [ Z ] 2 ≤ ( 1 + t R + 1 + ( 1 − t ) ∥ E [ Z ] ∥ 2 + t 1 − t ) ( 1 + 1 − t x − t x ⋆ − E [ Z ] 2 2 ) = t R + 2 + ( 1 − t ) ∥ E [ Z ] ∥ 2 ( 1 + 1 − t x − t x ⋆ − E [ Z ] 2 2 ) . (67)
It follows from (66) that, uniformly over ∥ x ∥ 2 ≤ R + 1 \lVert x\rVert_2\le R+1 ∥ x ∥ 2 ≤ R + 1 and x ⋆ ∈ R d x_\star\in\mathbb{R}^d x ⋆ ∈ R d ,
( 1 + ∥ x ⋆ ∥ 2 ) p b a s e ( x − t x ⋆ 1 − t ) ( 1 + ∥ s ( x − t x ⋆ 1 − t ) ∥ 2 ) ≤ C R , t p b a s e ( x − t x ⋆ 1 − t ) ( 1 + ∥ x − t x ⋆ 1 − t − E [ Z ] ∥ 2 2 ) ( 1 + ∥ s ( x − t x ⋆ 1 − t ) ∥ 2 2 ) ≤ C R , t , d , L . \begin{aligned}
\left(1+\lVert x_\star\rVert_2\right)p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)\left(1+\left\lVert s\left(\frac{x-tx_\star}{1-t}\right)\right\rVert_2\right) \\
&\le C_{R,t}p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)\left(1+\left\lVert\frac{x-tx_\star}{1-t}-\mathbb{E}[Z]\right\rVert_2^2\right)\left(1+\left\lVert s\left(\frac{x-tx_\star}{1-t}\right)\right\rVert_2^2\right)\le C_{R,t,d,L}.
\end{aligned} ( 1 + ∥ x ⋆ ∥ 2 ) p base ( 1 − t x − t x ⋆ ) ( 1 + s ( 1 − t x − t x ⋆ ) 2 ) ≤ C R , t p base ( 1 − t x − t x ⋆ ) ( 1 + 1 − t x − t x ⋆ − E [ Z ] 2 2 ) ( 1 + s ( 1 − t x − t x ⋆ ) 2 2 ) ≤ C R , t , d , L .
Here, C R , t C_{R,t} C R , t (resp. C R , t , d , L C_{R,t,d,L} C R , t , d , L ) is a constant depending on R , t R,t R , t (resp. R , t , d , L R,t,d,L R , t , d , L ) and the source moments. Therefore, the numerator of the posterior mean is absolutely integrable, and its denominator is finite and strictly positive.
For each coordinate vector e j e_j e j , every x x x with ∥ x ∥ 2 ≤ R \lVert x\rVert_2\le R ∥ x ∥ 2 ≤ R , and 0 < ∣ ε ∣ < 1 0<|\varepsilon|<1 0 < ∣ ε ∣ < 1 , the fundamental theorem of calculus reveals that
p b a s e ( x + ε e j − t x ⋆ 1 − t ) − p b a s e ( x − t x ⋆ 1 − t ) ε = 1 1 − t ∫ 0 1 p b a s e ( x + r ε e j − t x ⋆ 1 − t ) s j ( x + r ε e j − t x ⋆ 1 − t ) d r . \frac{p_{\mathrm{base}}\left(\frac{x+\varepsilon e_j-tx_\star}{1-t}\right)-p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)}{\varepsilon}
=\frac{1}{1-t}\int_0^1p_{\mathrm{base}}\left(\frac{x+r\varepsilon e_j-tx_\star}{1-t}\right)s_j\left(\frac{x+r\varepsilon e_j-tx_\star}{1-t}\right)\,\mathrm{d}r. ε p base ( 1 − t x + ε e j − t x ⋆ ) − p base ( 1 − t x − t x ⋆ ) = 1 − t 1 ∫ 0 1 p base ( 1 − t x + r ε e j − t x ⋆ ) s j ( 1 − t x + r ε e j − t x ⋆ ) d r .
Applying the Dominated Convergence Theorem therefore gives
∂ j ∫ p b a s e ( x − t x ⋆ 1 − t ) P d a t a ( d x ⋆ ) = 1 1 − t ∫ p b a s e ( x − t x ⋆ 1 − t ) s j ( x − t x ⋆ 1 − t ) P d a t a ( d x ⋆ ) , \partial_j\int p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)P_{\mathrm{data}}(\mathrm{d}x_\star)=\frac{1}{1-t}\int p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)s_j\left(\frac{x-tx_\star}{1-t}\right)P_{\mathrm{data}}(\mathrm{d}x_\star), ∂ j ∫ p base ( 1 − t x − t x ⋆ ) P data ( d x ⋆ ) = 1 − t 1 ∫ p base ( 1 − t x − t x ⋆ ) s j ( 1 − t x − t x ⋆ ) P data ( d x ⋆ ) ,
∂ j ∫ ( x ⋆ ) i p b a s e ( x − t x ⋆ 1 − t ) P d a t a ( d x ⋆ ) = 1 1 − t ∫ ( x ⋆ ) i p b a s e ( x − t x ⋆ 1 − t ) s j ( x − t x ⋆ 1 − t ) P d a t a ( d x ⋆ ) . \partial_j\int(x_\star)_i p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)P_{\mathrm{data}}(\mathrm{d}x_\star)=\frac{1}{1-t}\int(x_\star)_i p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)s_j\left(\frac{x-tx_\star}{1-t}\right)P_{\mathrm{data}}(\mathrm{d}x_\star). ∂ j ∫ ( x ⋆ ) i p base ( 1 − t x − t x ⋆ ) P data ( d x ⋆ ) = 1 − t 1 ∫ ( x ⋆ ) i p base ( 1 − t x − t x ⋆ ) s j ( 1 − t x − t x ⋆ ) P data ( d x ⋆ ) .
Since both right-hand sides are continuous, we know that the unnormalized integrals ∫ p b a s e ( x − t x ⋆ 1 − t ) P d a t a ( d x ⋆ ) \int p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)P_{\mathrm{data}}(\mathrm{d}x_\star) ∫ p base ( 1 − t x − t x ⋆ ) P data ( d x ⋆ ) and ∫ x ⋆ p b a s e ( x − t x ⋆ 1 − t ) P d a t a ( d x ⋆ ) \int x_\star p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)P_{\mathrm{data}}(\mathrm{d}x_\star) ∫ x ⋆ p base ( 1 − t x − t x ⋆ ) P data ( d x ⋆ ) are C 1 C^1 C 1 , and so is p t p_t p t . Since the denominator of the posterior mean is positive, the quotient rule shows that the posterior mean is C 1 C^1 C 1 as well.
Proving (33) and (32). By Bayes’ formula and the quotient rule, we have
p X ⋆ ∣ X t ( x ⋆ ∣ x ) = p b a s e ( x − t x ⋆ 1 − t ) ( 1 − t ) d p t ( x ) , p_{X_\star\mid X_t}(x_\star\mid x)=\frac{p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)}{(1-t)^d p_t(x)}, p X ⋆ ∣ X t ( x ⋆ ∣ x ) = ( 1 − t ) d p t ( x ) p base ( 1 − t x − t x ⋆ ) ,
∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) = p b a s e ( x − t x ⋆ 1 − t ) ( 1 − t ) d + 1 p t ( x ) s ( x − t x ⋆ 1 − t ) − p b a s e ( x − t x ⋆ 1 − t ) ( 1 − t ) d p t ( x ) 2 ∇ p t ( x ) . \nabla_xp_{X_\star\mid X_t}(x_\star\mid x)=\frac{p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)}{(1-t)^{d+1}p_t(x)}s\left(\frac{x-tx_\star}{1-t}\right)-\frac{p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)}{(1-t)^d p_t(x)^2}\nabla p_t(x). ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) = ( 1 − t ) d + 1 p t ( x ) p base ( 1 − t x − t x ⋆ ) s ( 1 − t x − t x ⋆ ) − ( 1 − t ) d p t ( x ) 2 p base ( 1 − t x − t x ⋆ ) ∇ p t ( x ) .
For every fixed x x x , the domination established above shows that both terms in the second equation are absolutely integrable with respect to P d a t a P_{\mathrm{data}} P data , even after multiplication by 1 + ∥ x ⋆ ∥ 2 1+\lVert x_\star\rVert_2 1 + ∥ x ⋆ ∥ 2 . Consequently, the derivative formula for the unnormalized density implies
∫ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) P d a t a ( d x ⋆ ) = ∇ p t ( x ) p t ( x ) − ∇ p t ( x ) p t ( x ) = 0 = ∇ x 1 = ∇ x ∫ p X ⋆ ∣ X t ( x ⋆ ∣ x ) P d a t a ( d x ⋆ ) , \begin{aligned}
\int\nabla_xp_{X_\star\mid X_t}(x_\star\mid x)P_{\mathrm{data}}(\mathrm{d}x_\star)=\frac{\nabla p_t(x)}{p_t(x)}-\frac{\nabla p_t(x)}{p_t(x)}=0 \\
&=\nabla_x1=\nabla_x\int p_{X_\star\mid X_t}(x_\star\mid x)P_{\mathrm{data}}(\mathrm{d}x_\star),
\end{aligned} ∫ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) P data ( d x ⋆ ) = p t ( x ) ∇ p t ( x ) − p t ( x ) ∇ p t ( x ) = 0 = ∇ x 1 = ∇ x ∫ p X ⋆ ∣ X t ( x ⋆ ∣ x ) P data ( d x ⋆ ) ,
which proves (33) . Moreover, applying the quotient rule to the posterior mean and using the derivative formulas for the unnormalized integrals, we obtain
∇ x E [ X ⋆ ∣ X t = x ] = ∇ x [ ∫ x ⋆ p b a s e ( x − t x ⋆ 1 − t ) P d a t a ( d x ⋆ ) ( 1 − t ) d p t ( x ) ] (68) \nabla_x\mathbb{E}[X_\star\mid X_t=x]=\nabla_x\left[\frac{\int x_\star p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)P_{\mathrm{data}}(\mathrm{d}x_\star)}{(1-t)^d p_t(x)}\right] \tag*{(68)} ∇ x E [ X ⋆ ∣ X t = x ] = ∇ x [ ( 1 − t ) d p t ( x ) ∫ x ⋆ p base ( 1 − t x − t x ⋆ ) P data ( d x ⋆ ) ] (68)
= 1 ( 1 − t ) d + 1 p t ( x ) ∫ x ⋆ p b a s e ( x − t x ⋆ 1 − t ) s ( x − t x ⋆ 1 − t ) T P d a t a ( d x ⋆ ) − 1 ( 1 − t ) d p t ( x ) 2 [ ∫ x ⋆ p b a s e ( x − t x ⋆ 1 − t ) P d a t a ( d x ⋆ ) ] [ ∇ p t ( x ) ] T = ∫ x ⋆ [ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) ] T P d a t a ( d x ⋆ ) = ∫ ( x ⋆ − x ) [ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) ] T P d a t a ( d x ⋆ ) + x [ ∫ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) P d a t a ( d x ⋆ ) ] T = ∫ ( x ⋆ − x ) [ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) ] T P d a t a ( d x ⋆ ) . \begin{aligned}
&= \frac{1}{(1-t)^{d+1}p_t(x)}\int x_\star p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)s\left(\frac{x-tx_\star}{1-t}\right)^{\mathsf{T}}P_{\mathrm{data}}(\mathrm{d}x_\star)\\
&\quad-\frac{1}{(1-t)^d p_t(x)^2}\left[\int x_\star p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)P_{\mathrm{data}}(\mathrm{d}x_\star)\right]\left[\nabla p_t(x)\right]^{\mathsf{T}}\\
&=\int x_\star\left[\nabla_xp_{X_\star\mid X_t}(x_\star\mid x)\right]^{\mathsf{T}}P_{\mathrm{data}}(\mathrm{d}x_\star)\\
&=\int(x_\star-x)\left[\nabla_xp_{X_\star\mid X_t}(x_\star\mid x)\right]^{\mathsf{T}}P_{\mathrm{data}}(\mathrm{d}x_\star)+x\left[\int\nabla_xp_{X_\star\mid X_t}(x_\star\mid x)P_{\mathrm{data}}(\mathrm{d}x_\star)\right]^{\mathsf{T}}\\
&=\int(x_\star-x)\left[\nabla_xp_{X_\star\mid X_t}(x_\star\mid x)\right]^{\mathsf{T}}P_{\mathrm{data}}(\mathrm{d}x_\star).
\end{aligned} = ( 1 − t ) d + 1 p t ( x ) 1 ∫ x ⋆ p base ( 1 − t x − t x ⋆ ) s ( 1 − t x − t x ⋆ ) T P data ( d x ⋆ ) − ( 1 − t ) d p t ( x ) 2 1 [ ∫ x ⋆ p base ( 1 − t x − t x ⋆ ) P data ( d x ⋆ ) ] [ ∇ p t ( x ) ] T = ∫ x ⋆ [ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) ] T P data ( d x ⋆ ) = ∫ ( x ⋆ − x ) [ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) ] T P data ( d x ⋆ ) + x [ ∫ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) P data ( d x ⋆ ) ] T = ∫ ( x ⋆ − x ) [ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) ] T P data ( d x ⋆ ) .
Here the last equation comes from (33) . This proves (32), and hence v t ( x ) = ( E [ X ⋆ ∣ X t = x ] − x ) / ( 1 − t ) v_t(x)=(\mathbb{E}[X_\star\mid X_t=x]-x)/(1-t) v t ( x ) = ( E [ X ⋆ ∣ X t = x ] − x ) / ( 1 − t ) is C 1 C^1 C 1 .
Proving continuity of the conditional second moments. Since Z = ( x − t X ⋆ ) / ( 1 − t ) Z=(x-tX_\star)/(1-t) Z = ( x − t X ⋆ ) / ( 1 − t ) when X t = x X_t=x X t = x , Bayes’ formula gives
E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t = x ] = ∫ ∥ x − t x ⋆ 1 − t − E [ Z ] ∥ 2 2 p b a s e ( x − t x ⋆ 1 − t ) P d a t a ( d x ⋆ ) ∫ p b a s e ( x − t x ⋆ 1 − t ) P d a t a ( d x ⋆ ) , \mathbb{E}[\|Z-\mathbb{E}[Z]\|_2^2\mid X_t=x]=\frac{\int\left\|\frac{x-tx_\star}{1-t}-\mathbb{E}[Z]\right\|_2^2p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)P_{\mathrm{data}}(\mathrm{d}x_\star)}{\int p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)P_{\mathrm{data}}(\mathrm{d}x_\star)}, E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t = x ] = ∫ p base ( 1 − t x − t x ⋆ ) P data ( d x ⋆ ) ∫ 1 − t x − t x ⋆ − E [ Z ] 2 2 p base ( 1 − t x − t x ⋆ ) P data ( d x ⋆ ) ,
E [ ∥ s ( Z ) ∥ 2 2 ∣ X t = x ] = ∫ ∥ s ( x − t x ⋆ 1 − t ) ∥ 2 2 p b a s e ( x − t x ⋆ 1 − t ) P d a t a ( d x ⋆ ) ∫ p b a s e ( x − t x ⋆ 1 − t ) P d a t a ( d x ⋆ ) . \mathbb{E}[\|s(Z)\|_2^2\mid X_t=x]=\frac{\int\left\|s\left(\frac{x-tx_\star}{1-t}\right)\right\|_2^2p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)P_{\mathrm{data}}(\mathrm{d}x_\star)}{\int p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)P_{\mathrm{data}}(\mathrm{d}x_\star)}. E [ ∥ s ( Z ) ∥ 2 2 ∣ X t = x ] = ∫ p base ( 1 − t x − t x ⋆ ) P data ( d x ⋆ ) ∫ s ( 1 − t x − t x ⋆ ) 2 2 p base ( 1 − t x − t x ⋆ ) P data ( d x ⋆ ) .
By virtue of (66) , both numerator integrands are bounded uniformly over x , x ⋆ ∈ R d x,x_\star\in\mathbb{R}^d x , x ⋆ ∈ R d . Since p b a s e p_{\mathrm{base}} p base and s s s are continuous, the Dominated Convergence Theorem shows that both numerators are finite and continuous. Their common denominator is positive and continuous, so both conditional second moments are finite and continuous as well. Therefore, T n ( x ) = x + ( t n − t n − 1 ) v t n − 1 ( x ) T_n(x)=x+(t_n-t_{n-1})v_{t_{n-1}}(x) T n ( x ) = x + ( t n − t n − 1 ) v t n − 1 ( x ) is C 1 C^1 C 1 , and G n G_n G n is closed. In particular, G n G_n G n is Borel.
Proof of Lemma 6
Proof. Recall from (5) and (9) that
v t ( x ) = E [ v t ( x ∣ X ⋆ ) ∣ X t = x ] = E [ X ⋆ − x 1 − t ∣ X t = x ] = ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) x ⋆ − x 1 − t P d a t a ( d x ⋆ ) . v_t(x)=\mathbb{E}[v_t(x\mid X_\star)\mid X_t=x]=\mathbb{E}\left[\left.\frac{X_\star-x}{1-t}\right|X_t=x\right]
=\int_{x_\star}p_{X_\star\mid X_t}(x_\star\mid x)\frac{x_\star-x}{1-t}P_{\mathrm{data}}(\mathrm{d}x_\star). v t ( x ) = E [ v t ( x ∣ X ⋆ ) ∣ X t = x ] = E [ 1 − t X ⋆ − x X t = x ] = ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) 1 − t x ⋆ − x P data ( d x ⋆ ) .
Consequently, by (32) in Lemma 4, one has
∇ x v t ( x ) = 1 1 − t ( ∇ x E [ X ⋆ ∣ X t = x ] − I ) = 1 1 − t I + 1 1 − t ∫ x ⋆ x ⋆ [ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) ] T P d a t a ( d x ⋆ ) = − 1 1 − t I + 1 1 − t ∫ x ⋆ ( x ⋆ − x ) [ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) ] T P d a t a ( d x ⋆ ) . \begin{aligned}
\nabla_xv_t(x)=\frac{1}{1-t}\left(\nabla_x\mathbb{E}[X_\star\mid X_t=x]-I\right) \\
&=\frac{1}{1-t}I+\frac{1}{1-t}\int_{x_\star}x_\star\left[\nabla_xp_{X_\star\mid X_t}(x_\star\mid x)\right]^{\mathsf{T}}P_{\mathrm{data}}(\mathrm{d}x_\star) \\
&=-\frac{1}{1-t}I+\frac{1}{1-t}\int_{x_\star}(x_\star-x)\left[\nabla_xp_{X_\star\mid X_t}(x_\star\mid x)\right]^{\mathsf{T}}P_{\mathrm{data}}(\mathrm{d}x_\star).
\end{aligned} ∇ x v t ( x ) = 1 − t 1 ( ∇ x E [ X ⋆ ∣ X t = x ] − I ) = 1 − t 1 I + 1 − t 1 ∫ x ⋆ x ⋆ [ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) ] T P data ( d x ⋆ ) = − 1 − t 1 I + 1 − t 1 ∫ x ⋆ ( x ⋆ − x ) [ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) ] T P data ( d x ⋆ ) .
For any fixed x ⋆ x_\star x ⋆ , Bayes’ rule and the fact that ∇ x p ( x ) = p ( x ) ∇ x log p ( x ) \nabla_xp(x)=p(x)\nabla_x\log p(x) ∇ x p ( x ) = p ( x ) ∇ x log p ( x ) imply that
∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) = p X ⋆ ∣ X t ( x ⋆ ∣ x ) ∇ x log p X ⋆ ∣ X t ( x ⋆ ∣ x ) = p X ⋆ ∣ X t ( x ⋆ ∣ x ) ∇ x log ( p X t ∣ X ⋆ ( x ∣ x ⋆ ) p t ( x ) ) = p X ⋆ ∣ X t ( x ⋆ ∣ x ) [ ∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) − ∇ x log p t ( x ) ] . (69) \begin{aligned}
\nabla_xp_{X_\star\mid X_t}(x_\star\mid x)=p_{X_\star\mid X_t}(x_\star\mid x)\nabla_x\log p_{X_\star\mid X_t}(x_\star\mid x) \\
&= p_{X_\star\mid X_t}(x_\star\mid x)\nabla_x \log\left(\frac{p_{X_t \mid X_\star}(x \mid x_\star)}{p_t(x)}\right) \\
&= p_{X_\star\mid X_t}(x_\star\mid x)\left[\nabla_x \log p_{X_t \mid X_\star}(x \mid x_\star)-\nabla_x \log p_t(x)\right]. \tag*{(69)}
\end{aligned} ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) = p X ⋆ ∣ X t ( x ⋆ ∣ x ) ∇ x log p X ⋆ ∣ X t ( x ⋆ ∣ x ) = p X ⋆ ∣ X t ( x ⋆ ∣ x ) ∇ x log ( p t ( x ) p X t ∣ X ⋆ ( x ∣ x ⋆ ) ) = p X ⋆ ∣ X t ( x ⋆ ∣ x ) [ ∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) − ∇ x log p t ( x ) ] . (69)
Putting the previous two identities together, we obtain
∇ x v t ( x ) = − 1 1 − t I + 1 1 − t ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) ( x ⋆ − x ) [ ∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) − ∇ x log p t ( x ) ] ⊤ P d a t a ( d x ⋆ ) . (70) \nabla_x v_t(x)=-\frac{1}{1-t}I+\frac{1}{1-t}\int_{x_\star}p_{X_\star\mid X_t}(x_\star\mid x)(x_\star-x)\left[\nabla_x\log p_{X_t\mid X_\star}(x\mid x_\star)-\nabla_x\log p_t(x)\right]^\top P_{\mathrm{data}}(\mathrm{d}x_\star). \tag*{(70)} ∇ x v t ( x ) = − 1 − t 1 I + 1 − t 1 ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) ( x ⋆ − x ) [ ∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) − ∇ x log p t ( x ) ] ⊤ P data ( d x ⋆ ) . (70)
In addition, (64) and (33) in Lemma 4 together show that for any fixed y ∈ R d y\in\mathbb{R}^d y ∈ R d ,
∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) y [ ∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) − ∇ x log p t ( x ) ] ⊤ P d a t a ( d x ⋆ ) = ∫ x ⋆ y [ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) ] ⊤ P d a t a ( d x ⋆ ) = y [ ∇ x ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) P d a t a ( d x ⋆ ) ] ⊤ = y ( ∇ x 1 ) ⊤ = 0. (71) \begin{aligned}
\int_{x_\star}p_{X_\star\mid X_t}(x_\star\mid x)y\left[\nabla_x\log p_{X_t\mid X_\star}(x\mid x_\star)-\nabla_x\log p_t(x)\right]^\top P_{\mathrm{data}}(\mathrm{d}x_\star) \\
&=\int_{x_\star}y\left[\nabla_xp_{X_\star\mid X_t}(x_\star\mid x)\right]^\top P_{\mathrm{data}}(\mathrm{d}x_\star) \\
&=y\left[\nabla_x\int_{x_\star}p_{X_\star\mid X_t}(x_\star\mid x)P_{\mathrm{data}}(\mathrm{d}x_\star)\right]^\top\\
&=y(\nabla_x1)^\top=0. \tag*{(71)}
\end{aligned} ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) y [ ∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) − ∇ x log p t ( x ) ] ⊤ P data ( d x ⋆ ) = ∫ x ⋆ y [ ∇ x p X ⋆ ∣ X t ( x ⋆ ∣ x ) ] ⊤ P data ( d x ⋆ ) = y [ ∇ x ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) P data ( d x ⋆ ) ] ⊤ = y ( ∇ x 1 ) ⊤ = 0. (71)
Setting y = E [ X ⋆ ∣ X t = x ] − x y=\mathbb{E}[X_\star\mid X_t=x]-x y = E [ X ⋆ ∣ X t = x ] − x in (71) yields
∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) ( x ⋆ − x ) [ ∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) − ∇ x log p t ( x ) ] ⊤ P d a t a ( d x ⋆ ) = ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) ( x ⋆ − E [ X ⋆ ∣ X t = x ] ) [ ∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) − ∇ x log p t ( x ) ] ⊤ P d a t a ( d x ⋆ ) + ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) ( E [ X ⋆ ∣ X t = x ] − x ) [ ∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) − ∇ x log p t ( x ) ] ⊤ P d a t a ( d x ⋆ ) = ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) ( x ⋆ − E [ X ⋆ ∣ X t = x ] ) [ ∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) − ∇ x log p t ( x ) ] ⊤ P d a t a ( d x ⋆ ) = ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) ( x ⋆ − E [ X ⋆ ∣ X t = x ] ) [ ∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) ] ⊤ P d a t a ( d x ⋆ ) \begin{aligned}
\int_{x_\star}p_{X_\star\mid X_t}(x_\star\mid x)(x_\star-x)\left[\nabla_x\log p_{X_t\mid X_\star}(x\mid x_\star)-\nabla_x\log p_t(x)\right]^\top P_{\mathrm{data}}(\mathrm{d}x_\star) \\
&=\int_{x_\star}p_{X_\star\mid X_t}(x_\star\mid x)(x_\star-\mathbb{E}[X_\star\mid X_t=x])\left[\nabla_x\log p_{X_t\mid X_\star}(x\mid x_\star)-\nabla_x\log p_t(x)\right]^\top P_{\mathrm{data}}(\mathrm{d}x_\star) \\
&+\int_{x_\star}p_{X_\star\mid X_t}(x_\star\mid x)(\mathbb{E}[X_\star\mid X_t=x]-x)\left[\nabla_x\log p_{X_t\mid X_\star}(x\mid x_\star)-\nabla_x\log p_t(x)\right]^\top P_{\mathrm{data}}(\mathrm{d}x_\star) \\
&=\int_{x_\star}p_{X_\star\mid X_t}(x_\star\mid x)(x_\star-\mathbb{E}[X_\star\mid X_t=x])\left[\nabla_x\log p_{X_t\mid X_\star}(x\mid x_\star)-\nabla_x\log p_t(x)\right]^\top P_{\mathrm{data}}(\mathrm{d}x_\star) \\
&=\int_{x_\star}p_{X_\star\mid X_t}(x_\star\mid x)(x_\star-\mathbb{E}[X_\star\mid X_t=x])\left[\nabla_x\log p_{X_t\mid X_\star}(x\mid x_\star)\right]^\top P_{\mathrm{data}}(\mathrm{d}x_\star)
\end{aligned} ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) ( x ⋆ − x ) [ ∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) − ∇ x log p t ( x ) ] ⊤ P data ( d x ⋆ ) = ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) ( x ⋆ − E [ X ⋆ ∣ X t = x ]) [ ∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) − ∇ x log p t ( x ) ] ⊤ P data ( d x ⋆ ) + ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) ( E [ X ⋆ ∣ X t = x ] − x ) [ ∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) − ∇ x log p t ( x ) ] ⊤ P data ( d x ⋆ ) = ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) ( x ⋆ − E [ X ⋆ ∣ X t = x ]) [ ∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) − ∇ x log p t ( x ) ] ⊤ P data ( d x ⋆ ) = ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) ( x ⋆ − E [ X ⋆ ∣ X t = x ]) [ ∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) ] ⊤ P data ( d x ⋆ )
− [ ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) ( x ⋆ − E [ X ⋆ ∣ X t = x ] ) P d a t a ( d x ⋆ ) ] [ ∇ x log p t ( x ) ] ⊤ = ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) ( x ⋆ − E [ X ⋆ ∣ X t = x ] ) [ ∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) ] ⊤ P d a t a ( d x ⋆ ) \begin{aligned}
-\left[\int_{x_\star}p_{X_\star\mid X_t}(x_\star\mid x)(x_\star-\mathbb{E}[X_\star\mid X_t=x])P_{\mathrm{data}}(\mathrm{d}x_\star)\right]\left[\nabla_x\log p_t(x)\right]^\top\\
&=\int_{x_\star}p_{X_\star\mid X_t}(x_\star\mid x)(x_\star-\mathbb{E}[X_\star\mid X_t=x])\left[\nabla_x\log p_{X_t\mid X_\star}(x\mid x_\star)\right]^\top P_{\mathrm{data}}(\mathrm{d}x_\star)
\end{aligned} − [ ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) ( x ⋆ − E [ X ⋆ ∣ X t = x ]) P data ( d x ⋆ ) ] [ ∇ x log p t ( x ) ] ⊤ = ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) ( x ⋆ − E [ X ⋆ ∣ X t = x ]) [ ∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) ] ⊤ P data ( d x ⋆ )
− E [ X ⋆ − E [ X ⋆ ∣ X t = x ] ∣ X t = x ] [ ∇ x log p t ( x ) ] ⊤ = ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) ( x ⋆ − E [ X ⋆ ∣ X t = x ] ) [ ∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) ] ⊤ P d a t a ( d x ⋆ ) . \begin{aligned}
-\mathbb{E}\left[X_\star-\mathbb{E}[X_\star\mid X_t=x]\mid X_t=x\right]\left[\nabla_x\log p_t(x)\right]^\top\\
&=\int_{x_\star}p_{X_\star\mid X_t}(x_\star\mid x)(x_\star-\mathbb{E}[X_\star\mid X_t=x])\left[\nabla_x\log p_{X_t\mid X_\star}(x\mid x_\star)\right]^\top P_{\mathrm{data}}(\mathrm{d}x_\star).
\end{aligned} − E [ X ⋆ − E [ X ⋆ ∣ X t = x ] ∣ X t = x ] [ ∇ x log p t ( x ) ] ⊤ = ∫ x ⋆ p X ⋆ ∣ X t ( x ⋆ ∣ x ) ( x ⋆ − E [ X ⋆ ∣ X t = x ]) [ ∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) ] ⊤ P data ( d x ⋆ ) .
Equation (7) together with the chain rule implies
∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) = ∇ x log ( ( 1 − t ) − d p b a s e ( x − t x ⋆ 1 − t ) ) = ∇ x log p b a s e ( x − t x ⋆ 1 − t ) = ( ∇ x x − t x ⋆ 1 − t ) ⊤ ∇ z log p b a s e ( x − t x ⋆ 1 − t ) = 1 1 − t ∇ z log p b a s e ( x − t x ⋆ 1 − t ) . (72) \begin{aligned}
\nabla_x\log p_{X_t\mid X_\star}(x\mid x_\star)=\nabla_x\log\left((1-t)^{-d}p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right)\right) \\
&=\nabla_x\log p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right) \\
&=\left(\nabla_x\frac{x-tx_\star}{1-t}\right)^\top\nabla_z\log p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right) \\
&=\frac{1}{1-t}\nabla_z\log p_{\mathrm{base}}\left(\frac{x-tx_\star}{1-t}\right). \tag*{(72)}
\end{aligned} ∇ x log p X t ∣ X ⋆ ( x ∣ x ⋆ ) = ∇ x log ( ( 1 − t ) − d p base ( 1 − t x − t x ⋆ ) ) = ∇ x log p base ( 1 − t x − t x ⋆ ) = ( ∇ x 1 − t x − t x ⋆ ) ⊤ ∇ z log p base ( 1 − t x − t x ⋆ ) = 1 − t 1 ∇ z log p base ( 1 − t x − t x ⋆ ) . (72)
Combining the previous two equations together, we have
∫ x ∗ p X ∗ ∣ X t ( x ∗ ∣ x ) ( x ∗ − x ) [ ∇ x log p X t ∣ X ∗ ( x ∣ x ∗ ) − ∇ x log p t ( x ) ] T P d a t a ( d x ∗ ) = 1 1 − t ∫ x ∗ p X ∗ ∣ X t ( x ∗ ∣ x ) ( x ∗ − E [ X ∗ ∣ X t = x ] ) [ ∇ z log p b a s e ( x − t x ∗ 1 − t ) ] T P d a t a ( d x ∗ ) = 1 1 − t ∫ x ∗ p X ∗ ∣ X t ( x ∗ ∣ x ) ( x ∗ − E [ X ∗ ∣ X t = x ] ) × [ ∇ z log p b a s e ( x − t x ∗ 1 − t ) − ∇ z log p b a s e ( x − t E [ X ∗ ∣ X t = x ] 1 − t ) ] T P d a t a ( d x ∗ ) + 1 1 − t ∫ x ∗ p X ∗ ∣ X t ( x ∗ ∣ x ) ( x ∗ − E [ X ∗ ∣ X t = x ] ) [ ∇ z log p b a s e ( x − t E [ X ∗ ∣ X t = x ] 1 − t ) ] T P d a t a ( d x ∗ ) = 1 1 − t ∫ x ∗ p X ∗ ∣ X t ( x ∗ ∣ x ) ( x ∗ − E [ X ∗ ∣ X t = x ] ) × [ ∇ z log p b a s e ( x − t x ∗ 1 − t ) − ∇ z log p b a s e ( x − t E [ X ∗ ∣ X t = x ] 1 − t ) ] T P d a t a ( d x ∗ ) + 1 1 − t E [ ( X ∗ − E [ X ∗ ∣ X t = x ] ) ∣ X t = x ] [ ∇ z log p b a s e ( x − t E [ X ∗ ∣ X t = x ] 1 − t ) ] T = 1 1 − t ∫ x ∗ p X ∗ ∣ X t ( x ∗ ∣ x ) ( x ∗ − E [ X ∗ ∣ X t = x ] ) × [ s ( x − t x ∗ 1 − t ) − s ( x − t E [ X ∗ ∣ X t = x ] 1 − t ) ] P d a t a ( d x ∗ ) = 1 1 − t E [ ( X ∗ − E [ X ∗ ∣ X t = x ] ) [ s ( x − t X ∗ 1 − t ) − s ( x − t E [ X ∗ ∣ X t = x ] 1 − t ) ] T ∣ X t = x ] . (73) \begin{aligned}
&\int_{x_*}p_{X_*|X_t}(x_*|x)(x_*-x)\left[\nabla_x\log p_{X_t|X_*}(x|x_*)-\nabla_x\log p_t(x)\right]^{\mathsf T}P_{\mathrm{data}}(\mathrm{d}x_*)\\
&=\frac{1}{1-t}\int_{x_*}p_{X_*|X_t}(x_*|x)(x_*-\mathbb{E}[X_*|X_t=x])\left[\nabla_z\log p_{\mathrm{base}}\left(\frac{x-tx_*}{1-t}\right)\right]^{\mathsf T}P_{\mathrm{data}}(\mathrm{d}x_*)\\
&=\frac{1}{1-t}\int_{x_*}p_{X_*|X_t}(x_*|x)(x_*-\mathbb{E}[X_*|X_t=x])\\
&\qquad\times\left[\nabla_z\log p_{\mathrm{base}}\left(\frac{x-tx_*}{1-t}\right)-\nabla_z\log p_{\mathrm{base}}\left(\frac{x-t\mathbb{E}[X_*|X_t=x]}{1-t}\right)\right]^{\mathsf T}P_{\mathrm{data}}(\mathrm{d}x_*)\\
&\quad+\frac{1}{1-t}\int_{x_*}p_{X_*|X_t}(x_*|x)(x_*-\mathbb{E}[X_*|X_t=x])\left[\nabla_z\log p_{\mathrm{base}}\left(\frac{x-t\mathbb{E}[X_*|X_t=x]}{1-t}\right)\right]^{\mathsf T}P_{\mathrm{data}}(\mathrm{d}x_*)\\
&=\frac{1}{1-t}\int_{x_*}p_{X_*|X_t}(x_*|x)(x_*-\mathbb{E}[X_*|X_t=x])\\
&\qquad\times\left[\nabla_z\log p_{\mathrm{base}}\left(\frac{x-tx_*}{1-t}\right)-\nabla_z\log p_{\mathrm{base}}\left(\frac{x-t\mathbb{E}[X_*|X_t=x]}{1-t}\right)\right]^{\mathsf T}P_{\mathrm{data}}(\mathrm{d}x_*)\\
&\quad+\frac{1}{1-t}\mathbb{E}\left[(X_*-\mathbb{E}[X_*|X_t=x])\mid X_t=x\right]\left[\nabla_z\log p_{\mathrm{base}}\left(\frac{x-t\mathbb{E}[X_*|X_t=x]}{1-t}\right)\right]^{\mathsf T}\\
&=\frac{1}{1-t}\int_{x_*}p_{X_*|X_t}(x_*|x)(x_*-\mathbb{E}[X_*|X_t=x])\\
&\qquad\times\left[s\left(\frac{x-tx_*}{1-t}\right)-s\left(\frac{x-t\mathbb{E}[X_*|X_t=x]}{1-t}\right)\right]P_{\mathrm{data}}(\mathrm{d}x_*)\\
&=\frac{1}{1-t}\mathbb{E}\left[\left(X_*-\mathbb{E}[X_*|X_t=x]\right)\left[s\left(\frac{x-tX_*}{1-t}\right)-s\left(\frac{x-t\mathbb{E}[X_*|X_t=x]}{1-t}\right)\right]^{\mathsf T}\mid X_t=x\right]. \tag*{(73)}
\end{aligned} ∫ x ∗ p X ∗ ∣ X t ( x ∗ ∣ x ) ( x ∗ − x ) [ ∇ x log p X t ∣ X ∗ ( x ∣ x ∗ ) − ∇ x log p t ( x ) ] T P data ( d x ∗ ) = 1 − t 1 ∫ x ∗ p X ∗ ∣ X t ( x ∗ ∣ x ) ( x ∗ − E [ X ∗ ∣ X t = x ]) [ ∇ z log p base ( 1 − t x − t x ∗ ) ] T P data ( d x ∗ ) = 1 − t 1 ∫ x ∗ p X ∗ ∣ X t ( x ∗ ∣ x ) ( x ∗ − E [ X ∗ ∣ X t = x ]) × [ ∇ z log p base ( 1 − t x − t x ∗ ) − ∇ z log p base ( 1 − t x − t E [ X ∗ ∣ X t = x ] ) ] T P data ( d x ∗ ) + 1 − t 1 ∫ x ∗ p X ∗ ∣ X t ( x ∗ ∣ x ) ( x ∗ − E [ X ∗ ∣ X t = x ]) [ ∇ z log p base ( 1 − t x − t E [ X ∗ ∣ X t = x ] ) ] T P data ( d x ∗ ) = 1 − t 1 ∫ x ∗ p X ∗ ∣ X t ( x ∗ ∣ x ) ( x ∗ − E [ X ∗ ∣ X t = x ]) × [ ∇ z log p base ( 1 − t x − t x ∗ ) − ∇ z log p base ( 1 − t x − t E [ X ∗ ∣ X t = x ] ) ] T P data ( d x ∗ ) + 1 − t 1 E [ ( X ∗ − E [ X ∗ ∣ X t = x ]) ∣ X t = x ] [ ∇ z log p base ( 1 − t x − t E [ X ∗ ∣ X t = x ] ) ] T = 1 − t 1 ∫ x ∗ p X ∗ ∣ X t ( x ∗ ∣ x ) ( x ∗ − E [ X ∗ ∣ X t = x ]) × [ s ( 1 − t x − t x ∗ ) − s ( 1 − t x − t E [ X ∗ ∣ X t = x ] ) ] P data ( d x ∗ ) = 1 − t 1 E [ ( X ∗ − E [ X ∗ ∣ X t = x ] ) [ s ( 1 − t x − t X ∗ ) − s ( 1 − t x − t E [ X ∗ ∣ X t = x ] ) ] T ∣ X t = x ] . (73)
Conditional on X t = x X_t=x X t = x , one has
t ( X ∗ − E [ X ∗ ∣ X t = x ] ) = − ( 1 − t ) ( Z − E [ Z ∣ X t = x ] ) . t(X_*-\mathbb{E}[X_*|X_t=x])=-(1-t)(Z-\mathbb{E}[Z|X_t=x]). t ( X ∗ − E [ X ∗ ∣ X t = x ]) = − ( 1 − t ) ( Z − E [ Z ∣ X t = x ]) .
Combining (65), (68) and the previous equation, one has
∇ x v t ( x ) = − 1 1 − t I − 1 t ( 1 − t ) E [ ( Z − E [ Z ∣ X t = x ] ) [ s ( Z ) − s ( E [ Z ∣ X t = x ] ) ] T ∣ X t = x ] . (74) \nabla_xv_t(x)=-\frac{1}{1-t}I-\frac{1}{t(1-t)}\mathbb{E}\left[(Z-\mathbb{E}[Z|X_t=x])[s(Z)-s(\mathbb{E}[Z|X_t=x])]^{\mathsf T}\mid X_t=x\right]. \tag*{(74)} ∇ x v t ( x ) = − 1 − t 1 I − t ( 1 − t ) 1 E [ ( Z − E [ Z ∣ X t = x ]) [ s ( Z ) − s ( E [ Z ∣ X t = x ]) ] T ∣ X t = x ] . (74)
Since s ( ⋅ ) s(\cdot) s ( ⋅ ) is L L L -Lipschitz, one has
∥ ∇ v t ( x ) + I d 1 − t ∥ o p ≤ ∥ ∇ v t ( x ) + I d 1 − t ∥ F ≤ 1 t ( 1 − t ) E [ ∥ Z − E [ Z ∣ X t = x ] ∥ 2 ∥ s ( Z ) − s ( E [ Z ∣ X t = x ] ) ∥ 2 ∣ X t = x ] ≤ L t ( 1 − t ) E [ ∥ Z − E [ Z ∣ X t = x ] ∥ 2 2 ∣ X t = x ] . (75) \begin{aligned}
\left\|\nabla v_t(x)+\frac{I_d}{1-t}\right\|_{\mathrm{op}}\leq\left\|\nabla v_t(x)+\frac{I_d}{1-t}\right\|_{\mathrm{F}} \\
&\leq\frac{1}{t(1-t)}\mathbb{E}\left[\|Z-\mathbb{E}[Z|X_t=x]\|_2\|s(Z)-s(\mathbb{E}[Z|X_t=x])\|_2\mid X_t=x\right] \\
&\leq\frac{L}{t(1-t)}\mathbb{E}\left[\|Z-\mathbb{E}[Z|X_t=x]\|_2^2\mid X_t=x\right]. \tag*{(75)}
\end{aligned} ∇ v t ( x ) + 1 − t I d op ≤ ∇ v t ( x ) + 1 − t I d F ≤ t ( 1 − t ) 1 E [ ∥ Z − E [ Z ∣ X t = x ] ∥ 2 ∥ s ( Z ) − s ( E [ Z ∣ X t = x ]) ∥ 2 ∣ X t = x ] ≤ t ( 1 − t ) L E [ ∥ Z − E [ Z ∣ X t = x ] ∥ 2 2 ∣ X t = x ] . (75)
Moreover, the tower law and (69) also imply that
∇ x v t ( x ) = − 1 1 − t I − 1 t ( 1 − t ) E [ ( Z − E [ Z ∣ X t = x ] ) s ( Z ) T ∣ X t = x ] . \nabla_xv_t(x)=-\frac{1}{1-t}I-\frac{1}{t(1-t)}\mathbb{E}\left[(Z-\mathbb{E}[Z|X_t=x])s(Z)^{\mathsf T}\mid X_t=x\right]. ∇ x v t ( x ) = − 1 − t 1 I − t ( 1 − t ) 1 E [ ( Z − E [ Z ∣ X t = x ]) s ( Z ) T ∣ X t = x ] .
The Cauchy–Schwarz inequality tells us that
∥ ∇ v t ( x ) + I d 1 − t ∥ F 2 ≤ E [ ∥ Z − E [ Z ∣ X t = x ] ∥ 2 2 ∣ X t = x ] E [ ∥ s ( Z ) ∥ 2 2 ∣ X t = x ] t 2 ( 1 − t ) 2 ≤ E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t = x ] E [ ∥ s ( Z ) ∥ 2 2 ∣ X t = x ] t 2 ( 1 − t ) 2 . \begin{aligned}
\left\|\nabla v_t(x)+\frac{I_d}{1-t}\right\|_{\mathrm{F}}^2\leq\frac{\mathbb{E}[\|Z-\mathbb{E}[Z|X_t=x]\|_2^2\mid X_t=x]\mathbb{E}[\|s(Z)\|_2^2\mid X_t=x]}{t^2(1-t)^2} \\
&\leq\frac{\mathbb{E}[\|Z-\mathbb{E}[Z]\|_2^2\mid X_t=x]\mathbb{E}[\|s(Z)\|_2^2\mid X_t=x]}{t^2(1-t)^2}.
\end{aligned} ∇ v t ( x ) + 1 − t I d F 2 ≤ t 2 ( 1 − t ) 2 E [ ∥ Z − E [ Z ∣ X t = x ] ∥ 2 2 ∣ X t = x ] E [ ∥ s ( Z ) ∥ 2 2 ∣ X t = x ] ≤ t 2 ( 1 − t ) 2 E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t = x ] E [ ∥ s ( Z ) ∥ 2 2 ∣ X t = x ] .
Then the basic inequality ∥ A + B ∥ F 2 ≤ 2 ( ∥ A ∥ F 2 + ∥ B ∥ F 2 ) \|A+B\|_{\mathrm F}^2\leq2(\|A\|_{\mathrm F}^2+\|B\|_{\mathrm F}^2) ∥ A + B ∥ F 2 ≤ 2 ( ∥ A ∥ F 2 + ∥ B ∥ F 2 ) implies (44).
Proof of Lemma 5
Proof. We fix 1 ≤ n ≤ N 1 \le n \le N 1 ≤ n ≤ N . In view of Lemma 4 , we know that T n T_n T n is C 1 C^1 C 1 and G n G_n G n is Borel.
Proving injectivity on G n G_n G n . We prove injectivity by contradiction. Suppose that x , y ∈ G n x,y \in G_n x , y ∈ G n satisfy T n ( x ) = T n ( y ) T_n(x)=T_n(y) T n ( x ) = T n ( y ) , and let α = ( y − x ) / ( 1 − t n − 1 ) \alpha=(y-x)/(1-t_{n-1}) α = ( y − x ) / ( 1 − t n − 1 ) . Since
T n ( x ) = x + t n − t n − 1 t n − 1 ( x − E [ Z ∣ X t n − 1 = x ] ) , T_n(x)=x+\frac{t_n-t_{n-1}}{t_{n-1}}\left(x-\mathbb{E}[Z\mid X_{t_{n-1}}=x]\right), T n ( x ) = x + t n − 1 t n − t n − 1 ( x − E [ Z ∣ X t n − 1 = x ] ) ,
we know that
E [ Z ∣ X t n − 1 = y ] − E [ Z ∣ X t n − 1 = x ] = t n t n − t n − 1 ( y − x ) = t n ( 1 − t n − 1 ) t n − t n − 1 α . (76) \mathbb{E}[Z\mid X_{t_{n-1}}=y]-\mathbb{E}[Z\mid X_{t_{n-1}}=x]=\frac{t_n}{t_n-t_{n-1}}(y-x)=\frac{t_n(1-t_{n-1})}{t_n-t_{n-1}}\alpha. \tag*{(76)} E [ Z ∣ X t n − 1 = y ] − E [ Z ∣ X t n − 1 = x ] = t n − t n − 1 t n ( y − x ) = t n − t n − 1 t n ( 1 − t n − 1 ) α . (76)
By Jensen’s inequality and the definition of G n G_n G n in (32) , we have
∥ α ∥ 2 ≤ t n − t n − 1 ( 1 − t n − 1 ) t n ( ∥ E [ Z ∣ X t n − 1 = y ] − E [ Z ] ∥ 2 + ∥ E [ Z ∣ X t n − 1 = x ] − E [ Z ] ∥ 2 ) ≤ t n − t n − 1 ( 1 − t n − 1 ) t n { E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 = y ] + E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 = x ] } ≤ t n − 1 ( 1 − t n ) 8 ( 1 − t n − 1 ) t n L { d + 12 log ( N + 1 ) } ≤ 1 8 L { d + 12 log ( N + 1 ) } . (77) \begin{aligned}
\lVert\alpha\rVert_2\leq\frac{t_n-t_{n-1}}{(1-t_{n-1})t_n}\left(\left\lVert\mathbb{E}[Z\mid X_{t_{n-1}}=y]-\mathbb{E}[Z]\right\rVert_2+\left\lVert\mathbb{E}[Z\mid X_{t_{n-1}}=x]-\mathbb{E}[Z]\right\rVert_2\right) \\
&\leq\frac{t_n-t_{n-1}}{(1-t_{n-1})t_n}\left\{\sqrt{\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\mid X_{t_{n-1}}=y\right]}+\sqrt{\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\mid X_{t_{n-1}}=x\right]}\right\} \\
&\leq\frac{t_{n-1}(1-t_n)}{8(1-t_{n-1})t_n\sqrt{L\{d+12\log(N+1)\}}}\leq\frac{1}{8\sqrt{L\{d+12\log(N+1)\}}}. \tag*{(77)}
\end{aligned} ∥ α ∥ 2 ≤ ( 1 − t n − 1 ) t n t n − t n − 1 ( ∥ E [ Z ∣ X t n − 1 = y ] − E [ Z ] ∥ 2 + ∥ E [ Z ∣ X t n − 1 = x ] − E [ Z ] ∥ 2 ) ≤ ( 1 − t n − 1 ) t n t n − t n − 1 { E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 = y ] + E [ ∥ Z − E [ Z ] ∥ 2 2 ∣ X t n − 1 = x ] } ≤ 8 ( 1 − t n − 1 ) t n L { d + 12 log ( N + 1 )} t n − 1 ( 1 − t n ) ≤ 8 L { d + 12 log ( N + 1 )} 1 . (77)
Here the last inequality arises from ( 1 − t n − 1 ) t n = t n − 1 ( 1 − t n ) + ( t n − t n − 1 ) ≥ t n − 1 ( 1 − t n ) (1-t_{n-1})t_n=t_{n-1}(1-t_n)+(t_n-t_{n-1})\geq t_{n-1}(1-t_n) ( 1 − t n − 1 ) t n = t n − 1 ( 1 − t n ) + ( t n − t n − 1 ) ≥ t n − 1 ( 1 − t n ) .
Let P P P and Q Q Q be the posterior laws of X ⋆ X_\star X ⋆ given X t n − 1 = x X_{t_{n-1}}=x X t n − 1 = x and X t n − 1 = y X_{t_{n-1}}=y X t n − 1 = y , respectively. We let
ℓ ( x ⋆ ) : = log d Q d P ( x ⋆ ) = log p b a s e ( y − t n − 1 x ⋆ 1 − t n − 1 ) − log p b a s e ( x − t n − 1 x ⋆ 1 − t n − 1 ) + log p t n − 1 ( x ) p t n − 1 ( y ) . (78) \ell(x_\star):=\log\frac{\mathrm{d}Q}{\mathrm{d}P}(x_\star)=\log p_{\mathrm{base}}\left(\frac{y-t_{n-1}x_\star}{1-t_{n-1}}\right)-\log p_{\mathrm{base}}\left(\frac{x-t_{n-1}x_\star}{1-t_{n-1}}\right)+\log\frac{p_{t_{n-1}}(x)}{p_{t_{n-1}}(y)}. \tag*{(78)} ℓ ( x ⋆ ) := log d P d Q ( x ⋆ ) = log p base ( 1 − t n − 1 y − t n − 1 x ⋆ ) − log p base ( 1 − t n − 1 x − t n − 1 x ⋆ ) + log p t n − 1 ( y ) p t n − 1 ( x ) . (78)
Applying (93) at the two endpoints reveals that
∣ ℓ ( x ⋆ ) − log p t n − 1 ( x ) p t n − 1 ( y ) − α T s ( x − t n − 1 x ⋆ 1 − t n − 1 ) ∣ ≤ L 2 ∥ y − x 1 − t n − 1 ∥ 2 2 , (79) \left|\ell(x_\star)-\log\frac{p_{t_{n-1}}(x)}{p_{t_{n-1}}(y)}-\alpha^{\mathsf{T}}s\left(\frac{x-t_{n-1}x_\star}{1-t_{n-1}}\right)\right|\leq\frac{L}{2}\left\|\frac{y-x}{1-t_{n-1}}\right\|_2^2, \tag*{(79)} ℓ ( x ⋆ ) − log p t n − 1 ( y ) p t n − 1 ( x ) − α T s ( 1 − t n − 1 x − t n − 1 x ⋆ ) ≤ 2 L 1 − t n − 1 y − x 2 2 , (79)
∣ ℓ ( x ⋆ ) − log p t n − 1 ( x ) p t n − 1 ( y ) − α T s ( y − t n − 1 x ⋆ 1 − t n − 1 ) ∣ ≤ L 2 ∥ y − x 1 − t n − 1 ∥ 2 2 . (80) \left|\ell(x_\star)-\log\frac{p_{t_{n-1}}(x)}{p_{t_{n-1}}(y)}-\alpha^{\mathsf{T}}s\left(\frac{y-t_{n-1}x_\star}{1-t_{n-1}}\right)\right|\leq\frac{L}{2}\left\|\frac{y-x}{1-t_{n-1}}\right\|_2^2. \tag*{(80)} ℓ ( x ⋆ ) − log p t n − 1 ( y ) p t n − 1 ( x ) − α T s ( 1 − t n − 1 y − t n − 1 x ⋆ ) ≤ 2 L 1 − t n − 1 y − x 2 2 . (80)
Moreover, by the definition of G n G_n G n , we have
E P [ ∥ s ( x − t n − 1 X ⋆ 1 − t n − 1 ) ∥ 2 2 ] = E [ ∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 = x ] ≤ 4 L { d + 12 log ( N + 1 ) } , (81) \mathbb{E}_P\left[\left\|s\left(\frac{x-t_{n-1}X_\star}{1-t_{n-1}}\right)\right\|_2^2\right]=\mathbb{E}\left[\lVert s(Z)\rVert_2^2\mid X_{t_{n-1}}=x\right]\leq4L\{d+12\log(N+1)\}, \tag*{(81)} E P [ s ( 1 − t n − 1 x − t n − 1 X ⋆ ) 2 2 ] = E [ ∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 = x ] ≤ 4 L { d + 12 log ( N + 1 )} , (81)
E Q [ ∥ s ( y − t n − 1 X ⋆ 1 − t n − 1 ) ∥ 2 2 ] = E [ ∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 = y ] ≤ 4 L { d + 12 log ( N + 1 ) } . (82) \mathbb{E}_Q\left[\left\|s\left(\frac{y-t_{n-1}X_\star}{1-t_{n-1}}\right)\right\|_2^2\right]=\mathbb{E}\left[\lVert s(Z)\rVert_2^2\mid X_{t_{n-1}}=y\right]\leq4L\{d+12\log(N+1)\}. \tag*{(82)} E Q [ s ( 1 − t n − 1 y − t n − 1 X ⋆ ) 2 2 ] = E [ ∥ s ( Z ) ∥ 2 2 ∣ X t n − 1 = y ] ≤ 4 L { d + 12 log ( N + 1 )} . (82)
Thus ℓ ( X ⋆ ) \ell(X_\star) ℓ ( X ⋆ ) has finite second moments under both laws. In particular, both relative entropies are finite, and
0 ≤ KL ( P ∥ Q ) + KL ( Q ∥ P ) = E Q [ ℓ ( X ⋆ ) ] − E P [ ℓ ( X ⋆ ) ] = E Q [ ℓ ( X ⋆ ) − log p t n − 1 ( x ) p t n − 1 ( y ) ] − E P [ ℓ ( X ⋆ ) − log p t n − 1 ( x ) p t n − 1 ( y ) ] \begin{aligned}
0\leq\operatorname{KL}(P\mathbin{\|}Q)+\operatorname{KL}(Q\mathbin{\|}P)=\mathbb{E}_Q[\ell(X_\star)]-\mathbb{E}_P[\ell(X_\star)] \\
&=\mathbb{E}_Q\left[\ell(X_\star)-\log\frac{p_{t_{n-1}}(x)}{p_{t_{n-1}}(y)}\right]-\mathbb{E}_P\left[\ell(X_\star)-\log\frac{p_{t_{n-1}}(x)}{p_{t_{n-1}}(y)}\right]
\end{aligned} 0 ≤ KL ( P ∥ Q ) + KL ( Q ∥ P ) = E Q [ ℓ ( X ⋆ )] − E P [ ℓ ( X ⋆ )] = E Q [ ℓ ( X ⋆ ) − log p t n − 1 ( y ) p t n − 1 ( x ) ] − E P [ ℓ ( X ⋆ ) − log p t n − 1 ( y ) p t n − 1 ( x ) ]
≤ (74a)–(75b) 4 L { d + 12 log ( N + 1 ) } ∥ α ∥ 2 + L ∥ α ∥ 2 2 \mathrel{\overset{\text{\scriptsize(74a)--(75b)}}{\leq}}4\sqrt{L\{d+12\log(N+1)\}}\lVert\alpha\rVert_2+L\lVert\alpha\rVert_2^2 ≤ (74a)–(75b) 4 L { d + 12 log ( N + 1 )} ∥ α ∥ 2 + L ∥ α ∥ 2 2
≤ (72) 1 2 + 1 64 { d + 12 log ( N + 1 ) } ≤ 9 16 . (83) \mathrel{\overset{\text{\scriptsize(72)}}{\leq}}\frac{1}{2}+\frac{1}{64\{d+12\log(N+1)\}}\leq\frac{9}{16}. \tag*{(83)} ≤ (72) 2 1 + 64 { d + 12 log ( N + 1 )} 1 ≤ 16 9 . (83)
The same two Taylor inequalities also allow us to control the variances. More specifically, using ( u + v ) 2 ≤ 2 u 2 + 2 v 2 (u+v)^2 \le2u^2+2v^2 ( u + v ) 2 ≤ 2 u 2 + 2 v 2 , we obtain
Var P ( ℓ ( X ⋆ ) ) + Var Q ( ℓ ( X ⋆ ) ) ≤ E P [ ( ℓ ( X ⋆ ) − log p t n − 1 ( x ) p t n − 1 ( y ) ) 2 ] + E Q [ ( ℓ ( X ⋆ ) − log p t n − 1 ( x ) p t n − 1 ( y ) ) 2 ] ≤ 2 ∥ α ∥ 2 2 { E P [ ∥ s ( x − t n − 1 X ⋆ 1 − t n − 1 ) ∥ 2 2 ] + E Q [ ∥ s ( y − t n − 1 X ⋆ 1 − t n − 1 ) ∥ 2 2 ] } + L 2 ∥ α ∥ 2 4 ≤ 16 L { d + 12 log ( N + 1 ) } ∥ α ∥ 2 2 + L 2 ∥ α ∥ 2 4 . (84) \begin{aligned}
\operatorname{Var}_{P}(\ell(X_\star))+\operatorname{Var}_{Q}(\ell(X_\star))
&\le\mathbb{E}_{P}\left[\left(\ell(X_\star)-\log\frac{p_{t_{n-1}}(x)}{p_{t_{n-1}}(y)}\right)^2\right]
+\mathbb{E}_{Q}\left[\left(\ell(X_\star)-\log\frac{p_{t_{n-1}}(x)}{p_{t_{n-1}}(y)}\right)^2\right]\\
&\le2\|\alpha\|_2^2\left\{\mathbb{E}_{P}\left[\left\|s\left(\frac{x-t_{n-1}X_\star}{1-t_{n-1}}\right)\right\|_2^2\right]
+\mathbb{E}_{Q}\left[\left\|s\left(\frac{y-t_{n-1}X_\star}{1-t_{n-1}}\right)\right\|_2^2\right]\right\}
+L^2\|\alpha\|_2^4\\
&\le16L\{d+12\log(N+1)\}\|\alpha\|_2^2+L^2\|\alpha\|_2^4. \tag*{(84)}
\end{aligned} Var P ( ℓ ( X ⋆ )) + Var Q ( ℓ ( X ⋆ )) ≤ E P [ ( ℓ ( X ⋆ ) − log p t n − 1 ( y ) p t n − 1 ( x ) ) 2 ] + E Q [ ( ℓ ( X ⋆ ) − log p t n − 1 ( y ) p t n − 1 ( x ) ) 2 ] ≤ 2∥ α ∥ 2 2 { E P [ s ( 1 − t n − 1 x − t n − 1 X ⋆ ) 2 2 ] + E Q [ s ( 1 − t n − 1 y − t n − 1 X ⋆ ) 2 2 ] } + L 2 ∥ α ∥ 2 4 ≤ 16 L { d + 12 log ( N + 1 )} ∥ α ∥ 2 2 + L 2 ∥ α ∥ 2 4 . (84)
We next compare expectations under P P P and Q Q Q . Since ( exp ( u ) − 1 ) u ≥ 2 ( exp ( u ) − 1 ) 2 / ( exp ( u ) + 1 ) (\exp(u)-1)u\ge2(\exp(u)-1)^2/(\exp(u)+1) ( exp ( u ) − 1 ) u ≥ 2 ( exp ( u ) − 1 ) 2 / ( exp ( u ) + 1 ) for every u ∈ R u\in\mathbb{R} u ∈ R and
exp ( ℓ ) d P = d Q , (85) \exp(\ell)\,dP=dQ, \tag*{(85)} exp ( ℓ ) d P = d Q , (85)
we have
∫ ( exp ( ℓ ) − 1 ) 2 exp ( ℓ ) + 1 d P ≤ 1 2 ∫ ( exp ( ℓ ) − 1 ) ℓ d P = 1 2 ∫ ( d Q d P − 1 ) log d Q d P d P = KL ( P ∥ Q ) + KL ( Q ∥ P ) 2 . (86) \int\frac{(\exp(\ell)-1)^2}{\exp(\ell)+1}\,dP
\le\frac{1}{2}\int(\exp(\ell)-1)\ell\,dP
=\frac{1}{2}\int\left(\frac{dQ}{dP}-1\right)\log\frac{dQ}{dP}\,dP
=\frac{\operatorname{KL}(P\mathbin{\|}Q)+\operatorname{KL}(Q\mathbin{\|}P)}{2}. \tag*{(86)} ∫ exp ( ℓ ) + 1 ( exp ( ℓ ) − 1 ) 2 d P ≤ 2 1 ∫ ( exp ( ℓ ) − 1 ) ℓ d P = 2 1 ∫ ( d P d Q − 1 ) log d P d Q d P = 2 KL ( P ∥ Q ) + KL ( Q ∥ P ) . (86)
For any real-valued f f f with finite second moments under both laws, we have
E Q [ f ( X ⋆ ) ] − E P [ f ( X ⋆ ) ] = E Q [ f ( X ⋆ ) − E P [ f ( X ⋆ ) ] + E Q [ f ( X ⋆ ) ] 2 ] − E P [ f ( X ⋆ ) − E P [ f ( X ⋆ ) ] + E Q [ f ( X ⋆ ) ] 2 ] = ∫ ( f − E P [ f ( X ⋆ ) ] + E Q [ f ( X ⋆ ) ] 2 ) ( exp ( ℓ ) − 1 ) d P . (87) \begin{aligned}
\mathbb{E}_{Q}[f(X_\star)]-\mathbb{E}_{P}[f(X_\star)]
&=\mathbb{E}_{Q}\left[f(X_\star)-\frac{\mathbb{E}_{P}[f(X_\star)]+\mathbb{E}_{Q}[f(X_\star)]}{2}\right]\\
&\quad-\mathbb{E}_{P}\left[f(X_\star)-\frac{\mathbb{E}_{P}[f(X_\star)]+\mathbb{E}_{Q}[f(X_\star)]}{2}\right]\\
&=\int\left(f-\frac{\mathbb{E}_{P}[f(X_\star)]+\mathbb{E}_{Q}[f(X_\star)]}{2}\right)(\exp(\ell)-1)\,dP. \tag*{(87)}
\end{aligned} E Q [ f ( X ⋆ )] − E P [ f ( X ⋆ )] = E Q [ f ( X ⋆ ) − 2 E P [ f ( X ⋆ )] + E Q [ f ( X ⋆ )] ] − E P [ f ( X ⋆ ) − 2 E P [ f ( X ⋆ )] + E Q [ f ( X ⋆ )] ] = ∫ ( f − 2 E P [ f ( X ⋆ )] + E Q [ f ( X ⋆ )] ) ( exp ( ℓ ) − 1 ) d P . (87)
By virtue of the Cauchy–Schwarz inequality, (86) and (87) , it follows that
∣ E Q [ f ( X ⋆ ) ] − E P [ f ( X ⋆ ) ] ∣ 2 ≤ [ ∫ ( f − E P [ f ( X ⋆ ) ] + E Q [ f ( X ⋆ ) ] 2 ) 2 ( exp ( ℓ ) + 1 ) d P ] [ ∫ ( exp ( ℓ ) − 1 ) 2 exp ( ℓ ) + 1 d P ] ≤ KL ( P ∥ Q ) + KL ( Q ∥ P ) 2 { E P [ ( f ( X ⋆ ) − E P [ f ( X ⋆ ) ] + E Q [ f ( X ⋆ ) ] 2 ) 2 ] + E Q [ ( f ( X ⋆ ) − E P [ f ( X ⋆ ) ] + E Q [ f ( X ⋆ ) ] 2 ) 2 ] } = KL ( P ∥ Q ) + KL ( Q ∥ P ) 2 { Var P ( f ( X ⋆ ) ) + Var Q ( f ( X ⋆ ) ) + 1 2 ∣ E Q [ f ( X ⋆ ) ] − E P [ f ( X ⋆ ) ] ∣ 2 } . \begin{aligned}
\left|\mathbb{E}_{Q}[f(X_\star)]-\mathbb{E}_{P}[f(X_\star)]\right|^2
&\le\left[\int\left(f-\frac{\mathbb{E}_{P}[f(X_\star)]+\mathbb{E}_{Q}[f(X_\star)]}{2}\right)^2(\exp(\ell)+1)\,dP\right]
\left[\int\frac{(\exp(\ell)-1)^2}{\exp(\ell)+1}\,dP\right]\\
&\le\frac{\operatorname{KL}(P\mathbin{\|}Q)+\operatorname{KL}(Q\mathbin{\|}P)}{2}
\left\{\mathbb{E}_{P}\left[\left(f(X_\star)-\frac{\mathbb{E}_{P}[f(X_\star)]+\mathbb{E}_{Q}[f(X_\star)]}{2}\right)^2\right]\right.\\
&\qquad\left.+\mathbb{E}_{Q}\left[\left(f(X_\star)-\frac{\mathbb{E}_{P}[f(X_\star)]+\mathbb{E}_{Q}[f(X_\star)]}{2}\right)^2\right]\right\}\\
&=\frac{\operatorname{KL}(P\mathbin{\|}Q)+\operatorname{KL}(Q\mathbin{\|}P)}{2}
\left\{\operatorname{Var}_{P}(f(X_\star))+\operatorname{Var}_{Q}(f(X_\star))
+\frac{1}{2}\left|\mathbb{E}_{Q}[f(X_\star)]-\mathbb{E}_{P}[f(X_\star)]\right|^2\right\}.
\end{aligned} ∣ E Q [ f ( X ⋆ )] − E P [ f ( X ⋆ )] ∣ 2 ≤ [ ∫ ( f − 2 E P [ f ( X ⋆ )] + E Q [ f ( X ⋆ )] ) 2 ( exp ( ℓ ) + 1 ) d P ] [ ∫ exp ( ℓ ) + 1 ( exp ( ℓ ) − 1 ) 2 d P ] ≤ 2 KL ( P ∥ Q ) + KL ( Q ∥ P ) { E P [ ( f ( X ⋆ ) − 2 E P [ f ( X ⋆ )] + E Q [ f ( X ⋆ )] ) 2 ] + E Q [ ( f ( X ⋆ ) − 2 E P [ f ( X ⋆ )] + E Q [ f ( X ⋆ )] ) 2 ] } = 2 KL ( P ∥ Q ) + KL ( Q ∥ P ) { Var P ( f ( X ⋆ )) + Var Q ( f ( X ⋆ )) + 2 1 ∣ E Q [ f ( X ⋆ )] − E P [ f ( X ⋆ )] ∣ 2 } .
Rearranging the terms and making use of (83) yields
∣ E Q [ f ( X ⋆ ) ] − E P [ f ( X ⋆ ) ] ∣ 2 ≤ { KL ( P ∥ Q ) + KL ( Q ∥ P ) } { Var P ( f ( X ⋆ ) ) + Var Q ( f ( X ⋆ ) ) } . (88) \left|\mathbb{E}_{Q}[f(X_\star)]-\mathbb{E}_{P}[f(X_\star)]\right|^2
\le\{\operatorname{KL}(P\mathbin{\|}Q)+\operatorname{KL}(Q\mathbin{\|}P)\}
\{\operatorname{Var}_{P}(f(X_\star))+\operatorname{Var}_{Q}(f(X_\star))\}. \tag*{(88)} ∣ E Q [ f ( X ⋆ )] − E P [ f ( X ⋆ )] ∣ 2 ≤ { KL ( P ∥ Q ) + KL ( Q ∥ P )} { Var P ( f ( X ⋆ )) + Var Q ( f ( X ⋆ ))} . (88)
By letting f = ℓ f=\ell f = ℓ and using E Q [ ℓ ( X ⋆ ) ] − E P [ ℓ ( X ⋆ ) ] = KL ( P ∥ Q ) + KL ( Q ∥ P ) \mathbb{E}_{Q}[\ell(X_\star)]-\mathbb{E}_{P}[\ell(X_\star)]=\operatorname{KL}(P\mathbin{\|}Q)+\operatorname{KL}(Q\mathbin{\|}P) E Q [ ℓ ( X ⋆ )] − E P [ ℓ ( X ⋆ )] = KL ( P ∥ Q ) + KL ( Q ∥ P ) , we obtain
KL ( P ∥ Q ) + KL ( Q ∥ P ) ≤ Var P ( ℓ ( X ⋆ ) ) + Var Q ( ℓ ( X ⋆ ) ) . (89) \operatorname{KL}(P\mathbin{\|}Q)+\operatorname{KL}(Q\mathbin{\|}P)
\le\operatorname{Var}_{P}(\ell(X_\star))+\operatorname{Var}_{Q}(\ell(X_\star)). \tag*{(89)} KL ( P ∥ Q ) + KL ( Q ∥ P ) ≤ Var P ( ℓ ( X ⋆ )) + Var Q ( ℓ ( X ⋆ )) . (89)
Putting (84) and (89) together, one has
∣ E Q [ f ( X ⋆ ) ] − E P [ f ( X ⋆ ) ] ∣ 2 ≤ ( 16 L { d + 12 log ( N + 1 ) } ∥ α ∥ 2 2 + L 2 ∥ α ∥ 2 4 ) { Var P ( f ( X ⋆ ) ) + Var Q ( f ( X ⋆ ) ) } . (90) \begin{aligned}
\left|\mathbb{E}_{Q}[f(X_\star)]-\mathbb{E}_{P}[f(X_\star)]\right|^2 \\
&\le\left(16L\{d+12\log(N+1)\}\|\alpha\|_2^2+L^2\|\alpha\|_2^4\right)\{\operatorname{Var}_P(f(X_\star))+\operatorname{Var}_Q(f(X_\star))\}. \tag*{(90)}
\end{aligned} ∣ E Q [ f ( X ⋆ )] − E P [ f ( X ⋆ )] ∣ 2 ≤ ( 16 L { d + 12 log ( N + 1 )} ∥ α ∥ 2 2 + L 2 ∥ α ∥ 2 4 ) { Var P ( f ( X ⋆ )) + Var Q ( f ( X ⋆ ))} . (90)
For every unit vector v ∈ R d v\in\mathbb{R}^d v ∈ R d , we let
f ( x ⋆ ) = v ⊤ x − t n − 1 x ⋆ 1 − t n − 1 . f(x_\star)=v^\top\frac{x-t_{n-1}x_\star}{1-t_{n-1}}. f ( x ⋆ ) = v ⊤ 1 − t n − 1 x − t n − 1 x ⋆ .
Under P P P , the vector ( x − t n − 1 X ⋆ ) / ( 1 − t n − 1 ) (x-t_{n-1}X_\star)/(1-t_{n-1}) ( x − t n − 1 X ⋆ ) / ( 1 − t n − 1 ) has the conditional law of Z Z Z given X t n − 1 = x X_{t_{n-1}}=x X t n − 1 = x . Under Q Q Q ,
y − t n − 1 X ⋆ 1 − t n − 1 = x − t n − 1 X ⋆ 1 − t n − 1 + y − x 1 − t n − 1 \frac{y-t_{n-1}X_\star}{1-t_{n-1}}=\frac{x-t_{n-1}X_\star}{1-t_{n-1}}+\frac{y-x}{1-t_{n-1}} 1 − t n − 1 y − t n − 1 X ⋆ = 1 − t n − 1 x − t n − 1 X ⋆ + 1 − t n − 1 y − x
has the conditional law of Z Z Z given X t n − 1 = y X_{t_{n-1}}=y X t n − 1 = y . By the definition of G n G_n G n , we have
Var P ( f ( X ⋆ ) ) + Var Q ( f ( X ⋆ ) ) = Var [ v ⊤ Z ∣ X t n − 1 = x ] + Var [ v ⊤ ( Z − y − x 1 − t n − 1 ) ∣ X t n − 1 = y ] = E [ ( v ⊤ ( Z − E [ Z ∣ X t n − 1 = x ] ) ) 2 ∣ X t n − 1 = x ] + E [ ( v ⊤ ( Z − E [ Z ∣ X t n − 1 = y ] ) ) 2 ∣ X t n − 1 = y ] ≤ E [ ∥ Z − E [ Z ∣ X t n − 1 = x ] ∥ 2 2 ∣ X t n − 1 = x ] + E [ ∥ Z − E [ Z ∣ X t n − 1 = y ] ∥ 2 2 ∣ X t n − 1 = y ] ≤ t n − 1 2 ( 1 − t n ) 2 128 L ( t n − t n − 1 ) 2 { d + 12 log ( N + 1 ) } . (91) \begin{aligned}
\operatorname{Var}_P(f(X_\star))+\operatorname{Var}_Q(f(X_\star))
&=\operatorname{Var}\left[v^\top Z\mid X_{t_{n-1}}=x\right]\\
&\quad+\operatorname{Var}\left[v^\top\left(Z-\frac{y-x}{1-t_{n-1}}\right)\mid X_{t_{n-1}}=y\right]\\
&=\mathbb{E}\left[(v^\top(Z-\mathbb{E}[Z\mid X_{t_{n-1}}=x]))^2\mid X_{t_{n-1}}=x\right]\\
&\quad+\mathbb{E}\left[(v^\top(Z-\mathbb{E}[Z\mid X_{t_{n-1}}=y]))^2\mid X_{t_{n-1}}=y\right]\\
&\le\mathbb{E}\left[\|Z-\mathbb{E}[Z\mid X_{t_{n-1}}=x]\|_2^2\mid X_{t_{n-1}}=x\right]\\
&\quad+\mathbb{E}\left[\|Z-\mathbb{E}[Z\mid X_{t_{n-1}}=y]\|_2^2\mid X_{t_{n-1}}=y\right]\\
&\le\frac{t_{n-1}^2(1-t_n)^2}{128L(t_n-t_{n-1})^2\{d+12\log(N+1)\}}. \tag*{(91)}
\end{aligned} Var P ( f ( X ⋆ )) + Var Q ( f ( X ⋆ )) = Var [ v ⊤ Z ∣ X t n − 1 = x ] + Var [ v ⊤ ( Z − 1 − t n − 1 y − x ) ∣ X t n − 1 = y ] = E [ ( v ⊤ ( Z − E [ Z ∣ X t n − 1 = x ]) ) 2 ∣ X t n − 1 = x ] + E [ ( v ⊤ ( Z − E [ Z ∣ X t n − 1 = y ]) ) 2 ∣ X t n − 1 = y ] ≤ E [ ∥ Z − E [ Z ∣ X t n − 1 = x ] ∥ 2 2 ∣ X t n − 1 = x ] + E [ ∥ Z − E [ Z ∣ X t n − 1 = y ] ∥ 2 2 ∣ X t n − 1 = y ] ≤ 128 L ( t n − t n − 1 ) 2 { d + 12 log ( N + 1 )} t n − 1 2 ( 1 − t n ) 2 . (91)
Taking the supremum over unit vectors v v v yields
∥ E Q [ x − t n − 1 X ⋆ 1 − t n − 1 ] − E P [ x − t n − 1 X ⋆ 1 − t n − 1 ] ∥ 2 2 = sup v : ∥ v ∥ 2 = 1 [ v ⊤ ( E Q [ x − t n − 1 X ⋆ 1 − t n − 1 ] − E P [ x − t n − 1 X ⋆ 1 − t n − 1 ] ) ] 2 = sup v : ∥ v ∥ 2 = 1 [ E Q [ v ⊤ x − t n − 1 X ⋆ 1 − t n − 1 ] − E P [ v ⊤ x − t n − 1 X ⋆ 1 − t n − 1 ] ] 2 \begin{aligned}
\left\|\mathbb{E}_Q\left[\frac{x-t_{n-1}X_\star}{1-t_{n-1}}\right]-\mathbb{E}_P\left[\frac{x-t_{n-1}X_\star}{1-t_{n-1}}\right]\right\|_2^2 \\
&=\sup_{v:\|v\|_2=1}\left[v^\top\left(\mathbb{E}_Q\left[\frac{x-t_{n-1}X_\star}{1-t_{n-1}}\right]-\mathbb{E}_P\left[\frac{x-t_{n-1}X_\star}{1-t_{n-1}}\right]\right)\right]^2 \\
&=\sup_{v:\|v\|_2=1}\left[\mathbb{E}_Q\left[v^\top\frac{x-t_{n-1}X_\star}{1-t_{n-1}}\right]-\mathbb{E}_P\left[v^\top\frac{x-t_{n-1}X_\star}{1-t_{n-1}}\right]\right]^2
\end{aligned} E Q [ 1 − t n − 1 x − t n − 1 X ⋆ ] − E P [ 1 − t n − 1 x − t n − 1 X ⋆ ] 2 2 = v : ∥ v ∥ 2 = 1 sup [ v ⊤ ( E Q [ 1 − t n − 1 x − t n − 1 X ⋆ ] − E P [ 1 − t n − 1 x − t n − 1 X ⋆ ] ) ] 2 = v : ∥ v ∥ 2 = 1 sup [ E Q [ v ⊤ 1 − t n − 1 x − t n − 1 X ⋆ ] − E P [ v ⊤ 1 − t n − 1 x − t n − 1 X ⋆ ] ] 2
By (90) and (91) ,
≤ t n − 1 2 ( 1 − t n ) 2 128 L ( t n − t n − 1 ) 2 { d + 12 log ( N + 1 ) } ( 16 L { d + 12 log ( N + 1 ) } ∥ α ∥ 2 2 + L 2 ∥ α ∥ 2 4 ) . (92) \le\frac{t_{n-1}^2(1-t_n)^2}{128L(t_n-t_{n-1})^2\{d+12\log(N+1)\}}\left(16L\{d+12\log(N+1)\}\|\alpha\|_2^2+L^2\|\alpha\|_2^4\right). \tag*{(92)} ≤ 128 L ( t n − t n − 1 ) 2 { d + 12 log ( N + 1 )} t n − 1 2 ( 1 − t n ) 2 ( 16 L { d + 12 log ( N + 1 )} ∥ α ∥ 2 2 + L 2 ∥ α ∥ 2 4 ) . (92)
On the other hand, (76) gives
E Q [ x − t n − 1 X ⋆ 1 − t n − 1 ] − E P [ x − t n − 1 X ⋆ 1 − t n − 1 ] = E [ Z ∣ X t n − 1 = y ] − E [ Z ∣ X t n − 1 = x ] − α = ( ( 1 − t n − 1 ) t n t n − t n − 1 − 1 ) α = t n − 1 ( 1 − t n ) t n − t n − 1 α . \begin{aligned}
\mathbb{E}_Q\left[\frac{x-t_{n-1}X_\star}{1-t_{n-1}}\right]-\mathbb{E}_P\left[\frac{x-t_{n-1}X_\star}{1-t_{n-1}}\right]=\mathbb{E}[Z\mid X_{t_{n-1}}=y]-\mathbb{E}[Z\mid X_{t_{n-1}}=x]-\alpha\\
&=\left(\frac{(1-t_{n-1})t_n}{t_n-t_{n-1}}-1\right)\alpha=\frac{t_{n-1}(1-t_n)}{t_n-t_{n-1}}\alpha.
\end{aligned} E Q [ 1 − t n − 1 x − t n − 1 X ⋆ ] − E P [ 1 − t n − 1 x − t n − 1 X ⋆ ] = E [ Z ∣ X t n − 1 = y ] − E [ Z ∣ X t n − 1 = x ] − α = ( t n − t n − 1 ( 1 − t n − 1 ) t n − 1 ) α = t n − t n − 1 t n − 1 ( 1 − t n ) α .
If α ≠ 0 \alpha\ne0 α = 0 , substituting this identity into (92) and cancelling the positive factor t n − 1 2 ( 1 − t n ) 2 ∥ α ∥ 2 2 / ( t n − t n − 1 ) 2 t_{n-1}^2(1-t_n)^2\|\alpha\|_2^2/(t_n-t_{n-1})^2 t n − 1 2 ( 1 − t n ) 2 ∥ α ∥ 2 2 / ( t n − t n − 1 ) 2 gives
1 ≤ 1 8 + L ∥ α ∥ 2 2 128 { d + 12 log ( N + 1 ) } ≤ (72) 1 8 + 1 8192 { d + 12 log ( N + 1 ) } 2 < 1 , 1\le\frac{1}{8}+\frac{L\|\alpha\|_2^2}{128\{d+12\log(N+1)\}}\mathrel{\mathop{\le}^{\text{\scriptsize(72)}}}\frac{1}{8}+\frac{1}{8192\{d+12\log(N+1)\}^2}<1, 1 ≤ 8 1 + 128 { d + 12 log ( N + 1 )} L ∥ α ∥ 2 2 ≤ (72) 8 1 + 8192 { d + 12 log ( N + 1 ) } 2 1 < 1 ,
a contradiction! Therefore, α = 0 \alpha=0 α = 0 and x = y x=y x = y , which proves injectivity on G n G_n G n .
Controlling the Jacobian on G n G_n G n . For every x ∈ G n x\in G_n x ∈ G n , (45) and (32) imply that
∥ ∇ x v t n − 1 ( x ) + I d 1 − t n − 1 ∥ F ≤ 1 t n − 1 ( 1 − t n − 1 ) { t n − 1 2 ( 1 − t n ) 2 256 L ( t n − t n − 1 ) 2 { d + 12 log ( N + 1 ) } ⋅ 4 L { d + 12 log ( N + 1 ) } } 1 / 2 = 1 − t n 8 ( 1 − t n − 1 ) ( t n − t n − 1 ) . \begin{aligned}
\left\|\nabla_xv_{t_{n-1}}(x)+\frac{I_d}{1-t_{n-1}}\right\|_F \\
&\le\frac{1}{t_{n-1}(1-t_{n-1})}\left\{\frac{t_{n-1}^{2}(1-t_n)^2}{256L(t_n-t_{n-1})^2\{d+12\log(N+1)\}}\cdot4L\{d+12\log(N+1)\}\right\}^{1/2} \\
&= \frac{1-t_n}{8(1-t_{n-1})(t_n-t_{n-1})}.
\end{aligned} ∇ x v t n − 1 ( x ) + 1 − t n − 1 I d F ≤ t n − 1 ( 1 − t n − 1 ) 1 { 256 L ( t n − t n − 1 ) 2 { d + 12 log ( N + 1 )} t n − 1 2 ( 1 − t n ) 2 ⋅ 4 L { d + 12 log ( N + 1 )} } 1/2 = 8 ( 1 − t n − 1 ) ( t n − t n − 1 ) 1 − t n .
Consequently, we know from Assumption 3 that
( t n − t n − 1 ) ∥ ∇ x v t n − 1 ( x ) ∥ o p ≤ t n − t n − 1 1 − t n − 1 + 1 − t n 8 ( 1 − t n − 1 ) ≤ t n − t n − 1 t n − 1 ( 1 − t n ) + 1 8 ≤ 5 8 . (t_n-t_{n-1})\lVert\nabla_xv_{t_{n-1}}(x)\rVert_{\mathrm{op}}\le\frac{t_n-t_{n-1}}{1-t_{n-1}}+\frac{1-t_n}{8(1-t_{n-1})}\le\frac{t_n-t_{n-1}}{t_{n-1}(1-t_n)}+\frac{1}{8}\le\frac{5}{8}. ( t n − t n − 1 ) ∥ ∇ x v t n − 1 ( x ) ∥ op ≤ 1 − t n − 1 t n − t n − 1 + 8 ( 1 − t n − 1 ) 1 − t n ≤ t n − 1 ( 1 − t n ) t n − t n − 1 + 8 1 ≤ 8 5 .
In particular, I d + θ ( t n − t n − 1 ) ∇ x v t n − 1 ( x ) I_d+\theta(t_n-t_{n-1})\nabla_xv_{t_{n-1}}(x) I d + θ ( t n − t n − 1 ) ∇ x v t n − 1 ( x ) is nonsingular for every 0 ≤ θ ≤ 1 0\le\theta\le1 0 ≤ θ ≤ 1 . Its determinant is continuous in θ \theta θ and equals one at θ = 0 \theta=0 θ = 0 , so det ∇ x T n ( x ) > 0 \det\nabla_xT_n(x)>0 det ∇ x T n ( x ) > 0 . This proves (36) .
Proof of Lemma 2
Proof of (24) . For every z , v ∈ R d z,v\in\mathbb{R}^d z , v ∈ R d , the fundamental theorem of calculus shows that
log p b a s e ( z + v ) − log p b a s e ( z ) − v T s ( z ) = ∫ 0 1 v T [ s ( z + θ v ) − s ( z ) ] d θ . \log p_{\mathrm{base}}(z+v)-\log p_{\mathrm{base}}(z)-v^{\mathsf{T}}s(z)=\int_0^1v^{\mathsf{T}}[s(z+\theta v)-s(z)]\,\mathrm{d}\theta. log p base ( z + v ) − log p base ( z ) − v T s ( z ) = ∫ 0 1 v T [ s ( z + θ v ) − s ( z )] d θ .
Using the Lipschitz property of s s s , we obtain
∣ log p b a s e ( z + v ) − log p b a s e ( z ) − v T s ( z ) ∣ ≤ ∫ 0 1 ∥ v ∥ 2 L θ ∥ v ∥ 2 d θ = L 2 ∥ v ∥ 2 2 , (93) \left|\log p_{\mathrm{base}}(z+v)-\log p_{\mathrm{base}}(z)-v^{\mathsf{T}}s(z)\right|\le\int_0^1\lVert v\rVert_2L\theta\lVert v\rVert_2\,\mathrm{d}\theta=\frac{L}{2}\lVert v\rVert_2^2, \tag*{(93)} log p base ( z + v ) − log p base ( z ) − v T s ( z ) ≤ ∫ 0 1 ∥ v ∥ 2 L θ ∥ v ∥ 2 d θ = 2 L ∥ v ∥ 2 2 , (93)
which further implies that
p b a s e ( z ) e v T s ( z ) ≤ e L ∥ v ∥ 2 2 / 2 p b a s e ( z + v ) . p_{\mathrm{base}}(z)e^{v^{\mathsf{T}}s(z)}\le e^{L\lVert v\rVert_2^2/2}p_{\mathrm{base}}(z+v). p base ( z ) e v T s ( z ) ≤ e L ∥ v ∥ 2 2 /2 p base ( z + v ) .
Integrating and then changing variables by z ′ = z + v z'=z+v z ′ = z + v yields
E [ e v T s ( Z ) ] ≤ e L ∥ v ∥ 2 2 / 2 ∫ p b a s e ( z + v ) d z = e L ∥ v ∥ 2 2 / 2 ∫ p b a s e ( z ′ ) d z ′ = e L ∥ v ∥ 2 2 / 2 . (94) \mathbb{E}[e^{v^{\mathsf{T}}s(Z)}]\le e^{L\lVert v\rVert_2^2/2}\int p_{\mathrm{base}}(z+v)\,\mathrm{d}z=e^{L\lVert v\rVert_2^2/2}\int p_{\mathrm{base}}(z')\,\mathrm{d}z'=e^{L\lVert v\rVert_2^2/2}. \tag*{(94)} E [ e v T s ( Z ) ] ≤ e L ∥ v ∥ 2 2 /2 ∫ p base ( z + v ) d z = e L ∥ v ∥ 2 2 /2 ∫ p base ( z ′ ) d z ′ = e L ∥ v ∥ 2 2 /2 . (94)
Next, we derive the mean and covariance statements. Fix v ∈ R d v\in\mathbb{R}^d v ∈ R d . Since
∣ v T s ( Z ) ∣ ≤ e v T s ( Z ) + e − v T s ( Z ) , \lvert v^{\mathsf{T}}s(Z)\rvert\le e^{v^{\mathsf{T}}s(Z)}+e^{-v^{\mathsf{T}}s(Z)}, ∣ v T s ( Z )∣ ≤ e v T s ( Z ) + e − v T s ( Z ) ,
we know from (94) that E [ ∣ v T s ( Z ) ∣ ] < ∞ \mathbb{E}[\lvert v^{\mathsf{T}}s(Z)\rvert]<\infty E [∣ v T s ( Z )∣] < ∞ . For every λ > 0 \lambda>0 λ > 0 , Jensen’s inequality and (94) together reveal that
λ E [ v T s ( Z ) ] ≤ log E [ e λ v T s ( Z ) ] ≤ L λ 2 ∥ v ∥ 2 2 2 , \lambda\mathbb{E}[v^{\mathsf{T}}s(Z)]\le\log\mathbb{E}[e^{\lambda v^{\mathsf{T}}s(Z)}]\le\frac{L\lambda^2\lVert v\rVert_2^2}{2}, λ E [ v T s ( Z )] ≤ log E [ e λ v T s ( Z ) ] ≤ 2 L λ 2 ∥ v ∥ 2 2 ,
− λ E [ v T s ( Z ) ] ≤ log E [ e − λ v T s ( Z ) ] ≤ L λ 2 ∥ v ∥ 2 2 2 . -\lambda\mathbb{E}[v^{\mathsf{T}}s(Z)]\le\log\mathbb{E}[e^{-\lambda v^{\mathsf{T}}s(Z)}]\le\frac{L\lambda^2\lVert v\rVert_2^2}{2}. − λ E [ v T s ( Z )] ≤ log E [ e − λ v T s ( Z ) ] ≤ 2 L λ 2 ∥ v ∥ 2 2 .
Dividing by λ \lambda λ and letting λ ↓ 0 \lambda\downarrow0 λ ↓ 0 shows that E [ v T s ( Z ) ] = 0 \mathbb{E}[v^{\mathsf{T}}s(Z)]=0 E [ v T s ( Z )] = 0 . Taking v v v to be each coordinate vector proves
E [ s ( Z ) ] = 0. \mathbb{E}[s(Z)]=0. E [ s ( Z )] = 0.
For every λ > 0 \lambda>0 λ > 0 , adding (94) for λ v \lambda v λ v and − λ v -\lambda v − λ v leads to
E [ 2 ( cosh ( λ v T s ( Z ) ) − 1 ) λ 2 ] = 1 λ 2 ( E [ e λ v T s ( Z ) ] + E [ e − λ v T s ( Z ) ] − 2 ) ≤ 2 ( e L λ 2 ∥ v ∥ 2 2 / 2 − 1 ) λ 2 . \mathbb{E}\left[\frac{2(\cosh(\lambda v^{\mathsf{T}}s(Z))-1)}{\lambda^2}\right]=\frac{1}{\lambda^2}\left(\mathbb{E}[e^{\lambda v^{\mathsf{T}}s(Z)}]+\mathbb{E}[e^{-\lambda v^{\mathsf{T}}s(Z)}]-2\right)\le\frac{2(e^{L\lambda^2\lVert v\rVert_2^2/2}-1)}{\lambda^2}. E [ λ 2 2 ( cosh ( λ v T s ( Z )) − 1 ) ] = λ 2 1 ( E [ e λ v T s ( Z ) ] + E [ e − λ v T s ( Z ) ] − 2 ) ≤ λ 2 2 ( e L λ 2 ∥ v ∥ 2 2 /2 − 1 ) .
Note that 2 ( cosh ( λ v T s ( Z ) ) − 1 ) λ 2 \frac{2(\cosh(\lambda v^{\mathsf{T}}s(Z))-1)}{\lambda^2} λ 2 2 ( c o s h ( λ v T s ( Z )) − 1 ) is nonnegative and converges pointwise to ( v T s ( Z ) ) 2 (v^{\mathsf{T}}s(Z))^2 ( v T s ( Z ) ) 2 as λ ↓ 0 \lambda\downarrow0 λ ↓ 0 . Therefore, Fatou’s lemma implies
E [ ( v T s ( Z ) ) 2 ] ≤ lim inf λ ↓ 0 E [ 2 ( cosh ( λ v T s ( Z ) ) − 1 ) λ 2 ] ≤ lim λ ↓ 0 2 ( e L λ 2 ∥ v ∥ 2 2 / 2 − 1 ) λ 2 = L ∥ v ∥ 2 2 . \mathbb{E}[(v^{\mathsf{T}}s(Z))^2]\le\liminf_{\lambda\downarrow0}\mathbb{E}\left[\frac{2(\cosh(\lambda v^{\mathsf{T}}s(Z))-1)}{\lambda^2}\right]\le\lim_{\lambda\downarrow0}\frac{2(e^{L\lambda^2\lVert v\rVert_2^2/2}-1)}{\lambda^2}=L\lVert v\rVert_2^2. E [( v T s ( Z ) ) 2 ] ≤ λ ↓ 0 lim inf E [ λ 2 2 ( cosh ( λ v T s ( Z )) − 1 ) ] ≤ λ ↓ 0 lim λ 2 2 ( e L λ 2 ∥ v ∥ 2 2 /2 − 1 ) = L ∥ v ∥ 2 2 .
Thus every coordinate of s ( Z ) s(Z) s ( Z ) has a finite second moment, and v T E [ s ( Z ) s ( Z ) T ] v ≤ L ∥ v ∥ 2 2 v^{\mathsf{T}}\mathbb{E}[s(Z)s(Z)^{\mathsf{T}}]v\le L\lVert v\rVert_2^2 v T E [ s ( Z ) s ( Z ) T ] v ≤ L ∥ v ∥ 2 2 for every v v v . This proves all three statements in (24) .
Proof of (25) . For every fixed z z z , recognizing that
− ∥ g ∥ 2 2 2 + g T s ( z ) 2 L = − 1 2 ∥ g − s ( z ) 2 L ∥ 2 2 + ∥ s ( z ) ∥ 2 2 4 L , -\frac{\lVert g\rVert_2^2}{2}+\frac{g^{\mathsf{T}}s(z)}{\sqrt{2L}}=-\frac{1}{2}\left\lVert g-\frac{s(z)}{\sqrt{2L}}\right\rVert_2^2+\frac{\lVert s(z)\rVert_2^2}{4L}, − 2 ∥ g ∥ 2 2 + 2 L g T s ( z ) = − 2 1 g − 2 L s ( z ) 2 2 + 4 L ∥ s ( z ) ∥ 2 2 ,
we obtain
1 ( 2 π ) d / 2 ∫ R d e − ∥ g ∥ 2 2 / 2 + g T s ( z ) / 2 L d g = e ∥ s ( z ) ∥ 2 2 / ( 4 L ) . \frac{1}{(2\pi)^{d/2}}\int_{\mathbb{R}^d}e^{-\lVert g\rVert_2^2/2+g^{\mathsf{T}}s(z)/\sqrt{2L}}\,\mathrm{d}g=e^{\lVert s(z)\rVert_2^2/(4L)}. ( 2 π ) d /2 1 ∫ R d e − ∥ g ∥ 2 2 /2 + g T s ( z ) / 2 L d g = e ∥ s ( z ) ∥ 2 2 / ( 4 L ) .
Then Tonelli’s theorem and (24) together show that
E [ e ∥ s ( Z ) ∥ 2 2 / ( 4 L ) ] = 1 ( 2 π ) d / 2 ∫ R d e − ∥ g ∥ 2 2 / 2 E [ e g T s ( Z ) / 2 L ] d g ≤ 1 ( 2 π ) d / 2 ∫ R d e − ∥ g ∥ 2 2 / 2 e ∥ g ∥ 2 2 / 4 d g = 1 ( 2 π ) d / 2 ∏ j = 1 d ∫ R e − g j 2 / 4 d g j = ( 4 π ) d / 2 ( 2 π ) d / 2 = 2 d / 2 . \begin{aligned}
\mathbb{E}\left[e^{\lVert s(Z)\rVert_2^2/(4L)}\right]=\frac{1}{(2\pi)^{d/2}}\int_{\mathbb{R}^d}e^{-\lVert g\rVert_2^2/2}\mathbb{E}\left[e^{g^{\mathsf{T}}s(Z)/\sqrt{2L}}\right]\,\mathrm{d}g \\
&\leq\frac{1}{(2\pi)^{d/2}}\int_{\mathbb{R}^d}e^{-\lVert g\rVert_2^2/2}e^{\lVert g\rVert_2^2/4}\,\mathrm{d}g \\
&=\frac{1}{(2\pi)^{d/2}}\prod_{j=1}^{d}\int_{\mathbb{R}}e^{-g_j^2/4}\,\mathrm{d}g_j=\frac{(4\pi)^{d/2}}{(2\pi)^{d/2}}=2^{d/2}.
\end{aligned} E [ e ∥ s ( Z ) ∥ 2 2 / ( 4 L ) ] = ( 2 π ) d /2 1 ∫ R d e − ∥ g ∥ 2 2 /2 E [ e g T s ( Z ) / 2 L ] d g ≤ ( 2 π ) d /2 1 ∫ R d e − ∥ g ∥ 2 2 /2 e ∥ g ∥ 2 2 /4 d g = ( 2 π ) d /2 1 j = 1 ∏ d ∫ R e − g j 2 /4 d g j = ( 2 π ) d /2 ( 4 π ) d /2 = 2 d /2 .
Therefore, (25) holds.
Proofs of (26) and (27) . Eqn. (26) is a direct consequence of Rao [48 ] and we prove it for completeness. More specifically, we let
ϕ ( z ; θ ) = p b a s e ( z − θ ) , T ( z ) = z − E [ Z ] . \phi(z;\theta)=p_{\mathrm{base}}(z-\theta),\qquad T(z)=z-\mathbb{E}[Z]. ϕ ( z ; θ ) = p base ( z − θ ) , T ( z ) = z − E [ Z ] .
Then we have
∫ R d T ( z ) ϕ ( z ; θ ) d z = θ (95) \int_{\mathbb{R}^d}T(z)\phi(z;\theta)\,\mathrm{d}z=\theta \tag*{(95)} ∫ R d T ( z ) ϕ ( z ; θ ) d z = θ (95)
and
∇ θ log ϕ ( z ; θ ) = − s ( z − θ ) . \nabla_\theta\log\phi(z;\theta)=-s(z-\theta). ∇ θ log ϕ ( z ; θ ) = − s ( z − θ ) .
Moreover, since E [ ∥ Z ∥ 2 2 ] < ∞ \mathbb{E}[\lVert Z\rVert_2^2]<\infty E [∥ Z ∥ 2 2 ] < ∞ and
E [ ∥ s ( Z ) ∥ 2 2 ] ≤ 2 ( ∥ s ( E [ Z ] ) ∥ 2 2 + E [ ∥ s ( Z ) − s ( E [ Z ] ) ∥ 2 2 ] ) \mathbb{E}[\lVert s(Z)\rVert_2^2]\leq2\left(\lVert s(\mathbb{E}[Z])\rVert_2^2+\mathbb{E}[\lVert s(Z)-s(\mathbb{E}[Z])\rVert_2^2]\right) E [∥ s ( Z ) ∥ 2 2 ] ≤ 2 ( ∥ s ( E [ Z ]) ∥ 2 2 + E [∥ s ( Z ) − s ( E [ Z ]) ∥ 2 2 ] )
≤ 2 ( ∥ s ( E [ Z ] ) ∥ 2 2 + L 2 E [ ∥ Z − E [ Z ] ∥ 2 2 ] ) < ∞ , \leq2\left(\lVert s(\mathbb{E}[Z])\rVert_2^2+L^2\mathbb{E}[\lVert Z-\mathbb{E}[Z]\rVert_2^2]\right)<\infty, ≤ 2 ( ∥ s ( E [ Z ]) ∥ 2 2 + L 2 E [∥ Z − E [ Z ] ∥ 2 2 ] ) < ∞ ,
we know that for every θ ∈ R d \theta\in\mathbb{R}^d θ ∈ R d and i , j ∈ { 1 , … , d } i,j\in\{1,\ldots,d\} i , j ∈ { 1 , … , d } , the integral on the right below is absolutely convergent. For the location family and affine T T T , the fundamental theorem of calculus and Fubini’s theorem justify differentiating the expectation, and
∂ ∂ θ j ∫ R d T i ( z ) ϕ ( z ; θ ) d z = ∫ R d T i ( z ) ∂ ϕ ( z ; θ ) ∂ θ j d z = δ i j . \frac{\partial}{\partial\theta_j}\int_{\mathbb{R}^d}T_i(z)\phi(z;\theta)\,\mathrm{d}z=\int_{\mathbb{R}^d}T_i(z)\frac{\partial\phi(z;\theta)}{\partial\theta_j}\,\mathrm{d}z=\delta_{ij}. ∂ θ j ∂ ∫ R d T i ( z ) ϕ ( z ; θ ) d z = ∫ R d T i ( z ) ∂ θ j ∂ ϕ ( z ; θ ) d z = δ ij .
Equivalently, we have
∫ R d T ( z ) ( ∇ θ log ϕ ( z ; θ ) ) T ϕ ( z ; θ ) d z = I d , \int_{\mathbb{R}^d}T(z)\left(\nabla_\theta\log\phi(z;\theta)\right)^{\mathsf{T}}\phi(z;\theta)\,\mathrm{d}z=I_d, ∫ R d T ( z ) ( ∇ θ log ϕ ( z ; θ ) ) T ϕ ( z ; θ ) d z = I d ,
Evaluating the identity at θ = 0 \theta=0 θ = 0 therefore gives
E [ ( Z − E [ Z ] ) s ( Z ) T ] = − I d . \mathbb{E}\left[(Z-\mathbb{E}[Z])s(Z)^{\mathsf{T}}\right]=-I_d. E [ ( Z − E [ Z ]) s ( Z ) T ] = − I d .
By virtue of the Cauchy–Schwarz inequality, we have
d = ∣ E [ ( Z − E [ Z ] ) T s ( Z ) ] ∣ ≤ E [ ∥ Z − E [ Z ] ∥ 2 ∥ s ( Z ) ∥ 2 ] ≤ ( E [ ∥ Z − E [ Z ] ∥ 2 2 ] ) 1 / 2 ( E [ ∥ s ( Z ) ∥ 2 2 ] ) 1 / 2 . d=\left|\mathbb{E}\left[(Z-\mathbb{E}[Z])^{\mathsf{T}}s(Z)\right]\right|\leq\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2\lVert s(Z)\rVert_2\right]\leq\left(\mathbb{E}\left[\lVert Z-\mathbb{E}[Z]\rVert_2^2\right]\right)^{1/2}\left(\mathbb{E}\left[\lVert s(Z)\rVert_2^2\right]\right)^{1/2}. d = E [ ( Z − E [ Z ] ) T s ( Z ) ] ≤ E [ ∥ Z − E [ Z ] ∥ 2 ∥ s ( Z ) ∥ 2 ] ≤ ( E [ ∥ Z − E [ Z ] ∥ 2 2 ] ) 1/2 ( E [ ∥ s ( Z ) ∥ 2 2 ] ) 1/2 .
In addition, (24) shows that
E [ ∥ s ( Z ) ∥ 2 2 ] = Tr ( E [ s ( Z ) s ( Z ) T ] ) ≤ L d . \mathbb{E}\left[\lVert s(Z)\rVert_2^2\right]=\operatorname{Tr}\left(\mathbb{E}[s(Z)s(Z)^{\mathsf{T}}]\right)\leq Ld. E [ ∥ s ( Z ) ∥ 2 2 ] = Tr ( E [ s ( Z ) s ( Z ) T ] ) ≤ L d .
Putting the previous two inequalities together, we arrive at (27) .