Partial preliminary results of this work appeared in International Conference on Machine Learning 2026 [26 ] .
Introduction
Diffusion probabilistic models (DPMs) have rapidly emerged as a leading paradigm in modern generative modeling, achieving state-of-the-art performance across a wide range applications in generative AI [62 , 61 , 59 , 15 ] (Ho et al. 2020). Building on ideas from non-equilibrium thermodynamics [58 ] , a DPM first defines a forward process that gradually injects noise into the clean data drawn from the true data distribution, and then generates new samples by reversing this process, turning pure noise into high-fidelity outputs. The reversal is driven by a pre-trained neural network that approximate the score functions of the noisy data distributions. DPMs now have shown remarkable success across various domains, including image synthesis [53 , 52 , 54 ] , audio generation [32 ] , language models [55 ] (Li et al. 2022; Li et al. 2026), and molecular design [22 ] , underscoring their versatility and impact. See Yang et al. [65 ] ; Croitoru et al. [13 ] ; Lai et al. [34 ] for overviews of recent development.
Once a DPM is trained, sampling amounts to discretizing a differential equation that describes the reverse process: either the diffusion SDE [51 , 2 ] (Ho et al. 2020) or the diffusion ODE [59 ] (Lu et al. 2022a). In practice, a DPM is typically trained as a noise prediction model : given a noisy sample and a time step, it predicts the noise added to the original clean data at that time in the forward process. Since subtracting this noise prediction from the noisy sample yields an estimate of the clean data, the same network equivalently acts as a clean-data predictor. As a result, sample quality relies on two factors: how the diffusion SDE/ODE is discretized, and how the pre-trained DPM is queried along the way.
Diffusion samplers can be broadly classified into two categories, depending on whether noise is injected beyond the initial step. Stochastic samplers (e.g., DDPM (Ho et al., 2020)) add Gaussian noise with during sampling iterations, while deterministic samplers (e.g., DDIM [59 ] ) add none beyond initialization. Stochastic samplers tend to produce more diverse samples and give a more natural framework for classifier guidance, which often improves the quality of conditional samples. For example, Figure 1 compares Stable Diffusion samples generated with the prompts (a) “A hand pets a cat in a suitcase” and (b) “A rodeo area with men in cowboy hats, rodeo clowns and an audience in the stands”. For the first prompt, the stochastic samplers SDE-DPMSolver++1 and SDE-DPMSolver++2 (see (18) and (19) for their update rules) produce clearly recognizable cats, whereas the deterministic samplers DDIM and DPMSolver-3 (see (32) for the latter’s update rule) primarily depict the suitcase. For the second prompt, the stochastic samplers produce cowboy hats and audience in stands. The corresponding details are less clearly resolved in the deterministic samples. Nevertheless, these benefits come at a price. Empirically, stochastic samplers converge far more slowly than their deterministic counterparts. For instance, On CIFAR-10 with the EDM model, DDPM needs roughly 40 iterations to achieve lower than 3 in Fréchet inception distance (FID) [20 ] , a metric used to assess the quality of images, whereas DDIM achieves comparable quality in fewer than 15 iterations. Since each iteration costs one neural function evaluation of the pre-trained DPM and sampling speed is often the primary bottleneck in deployment, this gap has driven growing interest in deterministic samplers.
Figure 1. Stable Diffusion samples at different NFEs for the cat and rodeo prompts described in the text. In these examples, SDE-DPMSolver++1 and SDE-DPMSolver++2 depict the cat and human figures more clearly than DDIM and DPMSolver-3.
Higher-order discretization is a widely used training-free approach to accelerating deterministic diffusion sampling (Lu et al., 2022a,b; Zhao et al., 2023). For stochastic samplers, however, increasing the order of a deterministic approximation does not automatically improve the strong convergence order to the underlying diffusion SDE. Strong convergence measures the discrepancy between the numerical and continuous trajectories driven by the same Brownian motion, whereas distributional convergence measures the discrepancy between their laws or between the generated and target distributions. In particular, we show that SDE-DPM-Solver++2 remains first-order convergent despite its higher-order approximation.
This gap motivates the central question of this work:
Can we design a fast stochastic sampler that narrows the performance gap with deterministic samplers?
Our contributions
We begin by analyzing the convergence rates of existing deterministic and stochastic samplers, including DDIM, DPMSolver(Lu et al., 2022a), and SDE-DPMSolver++(Lu et al., 2022a). For deterministic samplers, prior works have established that a higher-order approximation achieves a corresponding higher-order convergence rate. For instance, DPMSolver-p p p converges at a rate of O ~ ( M − p ) \widetilde{O}(M^{-p}) O ( M − p ) , where M M M denotes the number of iterations. This work reveals that this correspondence between approximation order and convergence order does not hold for the stochastic SDE-DPM-Solver++ schemes. In particular, SDE-DPM-Solver++2 remains first-order convergent in the strong sense, as established by matching upper and worst-case lower bounds. The reason is that the injected noise makes the generated samples vary too much at each step, for a higher-order expansion to help. This offers one explanation for the practical inefficiency of stochastic samplers.
Next, we explore a simple but under-explored idea: a forward-value discretization of the diffusion ODE. Given the current iterate and time, the idealized forward-value update would evaluate the DPM at the quantities corresponding the next time rather than the current one. We observe that such an update scheme can perform effectively even in the extremely low-NFE regime, which motivates a practical sampler that approximates it. The resulting sampler, named Forward DPMSolver (F-DPMSolver ), first produces a rough estimate of the next iterate using a vanilla update (e.g., DDIM) with cached model predictions, and then combines the current iterate with a DPM evaluation at this estimate. Critically, it maintains one model evaluation per iteration by reusing the previous evaluation, so its number of model evaluations (NFE) is the same as that of the vanilla sampler. Empirical results show that F-DPMSolver improves FID substantially over higher-order samplers including DPMSolver-2, DPMSolver-3 (Lu et al., 2022b), and UniPC-3 (Zhao et al., [71 ] ), even though its convergence rate still scales as Θ ~ ( 1 / M ) \widetilde{\Theta}(1/M) Θ ( 1/ M ) . These results convey a surprising insight: a first-order sampler can also achieve high practical sampling efficiency.
Inspired by this insight, we extend the forward-value idea to stochastic samplers and propose a fast first-order stochastic sampler, termed Forward DDPM (F-DDPM ). Whereas the deterministic sampler evaluates the DPM at a lookahead estimate for the next time step, the stochastic sampler uses an estimate at a carefully chosen cleaner intermediate time to account for subsequent noise injection. To construct this estimate, we form a tailored noise estimate that combines the available noise prediction with the injected noise from the previous step. We also establish a first-order strong error upper bound for F-DDPM. Empirically, F-DDPM consistently improves FID over the evaluated stochastic baselines at the same NFEs and substantially narrows the performance gap with deterministic samplers. With 1.5 times as many NFEs, it achieves sample quality comparable to or better than the evaluated deterministic baselines in many tested settings.
Other related work
Training-free/based acceleration. Existing acceleration strategies for diffusion sampling can be broadly divided into training-free and training-based approaches. Training-free methods reuse a fixed pre-trained DPM and speed up sampling solely by modifying the sampling update rule, making them broadly applicable to off-the-shelf DPMs. The methods discussed in the introduction belong to this category. In contrast, training-based strategies introduce an additional stage of training (e.g., distillation (Luhman and Luhman, [49 ] ; Salmans and Ho, [56 ] ; Meng et al., [50 ] ; Zhang et al., [69 ] ) and consistency models (Li et al., 2024b; Song et al., [60 ] )). They aim to shorten sampling trajectories by adapting pre-trained models into related architectures, at the cost of additional training computation. Since our focus is training-free acceleration, we next summarize related work from both practical and theoretical viewpoints.
Training-free acceleration: practice. Most practical training-free accelerations are driven by higher-order numerical discretizations of the reverse-time SDE/ODE. For ODE-based samplers, a prominent line of work exploits the structure of the diffusion ODE to design higher-order solvers, including DPM-Solver++ (Lu et al., 2022a,b), exponential-integrator-based methods (Zhang and Chen, [70 ] ), and predictor-corrector-based schemes such as UniPC (Zhao et al., [71 ] ). For SDE-based samplers, acceleration is comparatively less explored due to the intrinsic difficulty of SDE discretization, but notable progress includes stochastic Improved Euler’s methods (Jolicoeur-Martineau et al., [28 ] ), stochastic Adams methods (Xue et al., [64 ] ), and stochastic Runge–Kutta methods (Wu et al., 2024).
Training-free acceleration: theory. Convergence theory has been established for a wide range of diffusion samplers [10 , 35 , 7 , 9 , 3 , 27 , 68 ] (Li et al. 2023; Huang et al. 2024a; Li and Yan 2024; Li et al. 2025a; Cai et al. 2026), mostly focusing on standard DDPM and DDIM. For provable acceleration, most existing results likewise concentrate on higher-order discretization and on controlling the resulting discretization error for the reverse-time dynamics (Li and Cai, 2024; Li and Jiao, 2024; Jiao and Li, 2024; Huang et al., 2024a,b; Li et al., 2024a; Yu and Yu, 2025; Li et al., 2025b; Gupta et al., 2024; Wu et al., 2024). Beyond designing improved solvers, another line of work accelerates generation via parallel sampling that implements multiple denoising steps in parallel [57 , 8 , 19 ] . In addition to continuous diffusion models, sampling acceleration has also been studied for discrete diffusion models [11 , 16 , 72 , 17 , 12 ] (Li and Cai 2025; Cai and Li 2026).
Notation. For integer n ≥ 1 n \ge1 n ≥ 1 , we denote [ n ] : = { 1 , 2 , … , n } [n] := \{1, 2, \ldots, n\} [ n ] := { 1 , 2 , … , n } . For vector x \boldsymbol{x} x , we denote by ∥ x ∥ 2 \|\boldsymbol{x}\|_2 ∥ x ∥ 2 or ∥ x ∥ \|\boldsymbol{x}\| ∥ x ∥ its ℓ 2 \ell_2 ℓ 2 norm. For matrix A \boldsymbol{A} A , we denote by ∥ A ∥ \|\boldsymbol{A}\| ∥ A ∥ or ∥ A ∥ o p \|\boldsymbol{A}\|_{\mathrm{op}} ∥ A ∥ op its operator norm. In the rest of the paper, we use λ ( t ) \lambda(t) λ ( t ) or λ t \lambda_t λ t interchangeably to denote the mapping from the parameter t t t to λ \lambda λ . For random vector x t \boldsymbol{x}_t x t defined in (1) , we let q t q_t q t denote its probability density function. For function f ( M ) f(M) f ( M ) , we write f ( M ) = O ( M − k ) f(M) = O(M^{-k}) f ( M ) = O ( M − k ) if there exists some constant C > 0 C > 0 C > 0 such that ∣ f ( M ) ∣ ≤ C M − k |f(M)| \le CM^{-k} ∣ f ( M ) ∣ ≤ C M − k for all M ≥ 1 M \ge1 M ≥ 1 . Similarly, we denote f ( M ) = Ω ( M − k ) f(M) = \Omega(M^{-k}) f ( M ) = Ω ( M − k ) if ∣ f ( M ) ∣ ≥ c M − k |f(M)| \ge cM^{-k} ∣ f ( M ) ∣ ≥ c M − k holds for some constant c > 0 c > 0 c > 0 , and f ( M ) = Θ ( M − k ) f(M) = \Theta(M^{-k}) f ( M ) = Θ ( M − k ) if both f ( M ) = O ( M − k ) f(M) = O(M^{-k}) f ( M ) = O ( M − k ) and f ( M ) = Ω ( M − k ) f(M) = \Omega(M^{-k}) f ( M ) = Ω ( M − k ) hold. Finally, we say f ( M ) = o ( M − k ) f(M) = o(M^{-k}) f ( M ) = o ( M − k ) if lim M → ∞ M k f ( M ) = 0 \lim_{M \to\infty} M^k f(M) = 0 lim M → ∞ M k f ( M ) = 0 .
Preliminaries
In this section, we review the basics of DPM sampling.
Diffusion processes and parameterization
Forward and reverse processes. We consider a forward process ( x t ) 0 ≤ t ≤ T (\boldsymbol{x}_t)_{0 \le t \le T} ( x t ) 0 ≤ t ≤ T in R d \mathbb{R}^d R d whose marginal distribution at time t ∈ [ 0 , T ] t \in[0,T] t ∈ [ 0 , T ] satisfies
x t = d α t x 0 + σ t z , (1) \boldsymbol{x}_t \mathrel{\mathop{=}^{\mathrm{d}}} \alpha_t \boldsymbol{x}_0 + \sigma_t \boldsymbol{z}, \tag*{(1)} x t = d α t x 0 + σ t z , (1)
where x 0 ∼ q 0 \boldsymbol{x}_0 \sim q_0 x 0 ∼ q 0 is independent of the Gaussian noise z ∼ N ( 0 , I d ) \boldsymbol{z} \sim\mathcal{N}(\boldsymbol{0}, \boldsymbol{I}_d) z ∼ N ( 0 , I d ) . We denote the distribution of x t \boldsymbol{x}_t x t , or its density, by q t q_t q t . The noise schedules α t \alpha_t α t and σ t \sigma_t σ t are positive for t ∈ ( 0 , T ) t \in(0,T) t ∈ ( 0 , T ) and are chosen so that the signal-to-noise (SNR) ratio α t / σ t \alpha_t/\sigma_t α t / σ t decreases strictly with t t t and the terminal distribution is approximately Gaussian: q T ≈ N ( 0 , σ ~ 2 I d ) q_T \approx\mathcal{N}(0,\widetilde{\sigma}^2\boldsymbol{I}_d) q T ≈ N ( 0 , σ 2 I d ) .
These forward marginals can be obtained from the SDE
d x t = f ( t ) x t d t + g ( t ) d w t , t ∈ [ 0 , T ] ; x 0 ∼ q 0 , (2) \mathrm{d}\boldsymbol{x}_t = f(t)\boldsymbol{x}_t\,\mathrm{d}t + g(t)\,\mathrm{d}\boldsymbol{w}_t,\qquad t \in[0,T];\qquad\boldsymbol{x}_0 \sim q_0, \tag*{(2)} d x t = f ( t ) x t d t + g ( t ) d w t , t ∈ [ 0 , T ] ; x 0 ∼ q 0 , (2)
where ( w t ) t ≥ 0 (\boldsymbol{w}_t)_{t \ge0} ( w t ) t ≥ 0 denotes a standard d d d -dimensional Brownian motion. The coefficients are determined by the noise schedules as follows:
f ( t ) = d log α t d t and g 2 ( t ) = d σ t 2 d t − 2 σ t 2 d log α t d t . f(t) = \frac{\mathrm{d}\log\alpha_t}{\mathrm{d}t}\qquad\text{and}\qquad g^2(t) = \frac{\mathrm{d}\sigma_t^2}{\mathrm{d}t} - 2\sigma_t^2\frac{\mathrm{d}\log\alpha_t}{\mathrm{d}t}. f ( t ) = d t d log α t and g 2 ( t ) = d t d σ t 2 − 2 σ t 2 d t d log α t .
For further details on this formulation, we refer to Kingma et al. [31 ] .
The time-reversal result of Anderson [1 ] gives the reverse SDE associated with (2) , which is solved from T T T to 0 0 0 :
d x t = [ f ( t ) x t − g 2 ( t ) s t ∗ ( x t ) ] d t + g ( t ) d w ~ t , t ∈ [ 0 , T ] ; x T ∼ q T . (3) \mathrm{d}\boldsymbol{x}_t = \left[f(t)\boldsymbol{x}_t - g^2(t)s_t^*(\boldsymbol{x}_t)\right]\mathrm{d}t + g(t)\,\mathrm{d}\widetilde{\boldsymbol{w}}_t,\qquad t \in[0,T];\qquad\boldsymbol{x}_T \sim q_T. \tag*{(3)} d x t = [ f ( t ) x t − g 2 ( t ) s t ∗ ( x t ) ] d t + g ( t ) d w t , t ∈ [ 0 , T ] ; x T ∼ q T . (3)
where ( w ~ t ) t ∈ [ 0 , T ] (\widetilde{\boldsymbol{w}}_t)_{t \in[0,T]} ( w t ) t ∈ [ 0 , T ] denotes standard Brownian motion running backward from T T T to 0 0 0 , and s t ∗ ( ⋅ ) : = ∇ log q t ( ⋅ ) s_t^*(\cdot) := \nabla\log q_t(\cdot) s t ∗ ( ⋅ ) := ∇ log q t ( ⋅ ) is the score of the time-t t t marginal. Sampling can alternatively be based on a deterministic process governed by the probability flow ODE :
x ˙ t = f ( t ) x t − 1 2 g 2 ( t ) s t ∗ ( x t ) , t ∈ [ 0 , T ] ; x T ∼ q T . (4) \dot{\boldsymbol{x}}_t = f(t)\boldsymbol{x}_t - \frac{1}{2}g^2(t)s_t^*(\boldsymbol{x}_t),\qquad t \in[0,T];\qquad\boldsymbol{x}_T \sim q_T. \tag*{(4)} x ˙ t = f ( t ) x t − 2 1 g 2 ( t ) s t ∗ ( x t ) , t ∈ [ 0 , T ] ; x T ∼ q T . (4)
At every time t t t , both reverse processes have marginal distribution q t q_t q t , matching that of the forward process; see Song et al. [62 ] . With the exact score s t ⋆ ( ⋅ ) s_t^\star(\cdot) s t ⋆ ( ⋅ ) and initialization from q T q_T q T , both reverse processes recover q 0 q_0 q 0 . In practice, sampling is approximate because the score is estimated and q T q_T q T is replaced by a Gaussian approximation.
Noise and data prediction. A noise prediction model ϵ θ ( x , t ) \epsilon_\theta(x,t) ϵ θ ( x , t ) , parameterized by θ \theta θ , approximates the scaled score function − σ t s t ⋆ ( x ) -\sigma_t s_t^\star(x) − σ t s t ⋆ ( x ) . The ideal noise predictor satisfies
ϵ θ ⋆ ( x , t ) = − σ t s t ⋆ ( x ) = E [ z ∣ α t x 0 + σ t z = x ] . (5) \epsilon_{\theta^\star}(x,t)=-\sigma_t s_t^\star(x)=\mathbb{E}\left[z\mid\alpha_t x_0+\sigma_t z=x\right]. \tag*{(5)} ϵ θ ⋆ ( x , t ) = − σ t s t ⋆ ( x ) = E [ z ∣ α t x 0 + σ t z = x ] . (5)
For any noise predictor, the corresponding data predictor is defined by
μ θ ( x , t ) : = x − σ t ϵ θ ( x , t ) α t . (6) \mu_\theta(x,t):=\frac{x-\sigma_t\epsilon_\theta(x,t)}{\alpha_t}. \tag*{(6)} μ θ ( x , t ) := α t x − σ t ϵ θ ( x , t ) . (6)
In particular, the ideal data predictor satisfies
μ θ ⋆ ( x , t ) = E [ x 0 ∣ α t x 0 + σ t z = x ] . (7) \mu_{\theta^\star}(x,t)=\mathbb{E}\left[x_0\mid\alpha_t x_0+\sigma_t z=x\right]. \tag*{(7)} μ θ ⋆ ( x , t ) = E [ x 0 ∣ α t x 0 + σ t z = x ] . (7)
These conditional expectations motivate the names: ϵ θ \epsilon_\theta ϵ θ and μ θ \mu_\theta μ θ predict the noise z z z and clean data x 0 x_0 x 0 , respectively, from the noisy input.
Time reparameterization and sampling grid. We use the time parameterization
λ ( t ) = λ t : = log ( α t / σ t ) , (8) \lambda(t)=\lambda_t:=\log(\alpha_t/\sigma_t), \tag*{(8)} λ ( t ) = λ t := log ( α t / σ t ) , (8)
which is strictly decreasing on ( 0 , T ) (0,T) ( 0 , T ) . Its inverse is denoted by t ( λ ) t(\lambda) t ( λ ) . For notational convenience, we use the abbreviations
ϵ t : = ϵ θ ( x t , t ) and μ t : = μ θ ( x t , t ) \epsilon_t:=\epsilon_\theta(x_t,t) \qquad\text{and}\qquad\mu_t:=\mu_\theta(x_t,t) ϵ t := ϵ θ ( x t , t ) and μ t := μ θ ( x t , t )
whenever it is clear from context.
For numerical sampling, we discretize a truncated interval [ t M , t 0 ] ⊂ ( 0 , T ) [t_M,t_0]\subset(0,T) [ t M , t 0 ] ⊂ ( 0 , T ) using M + 1 M+1 M + 1 time points:
T > t 0 > t 1 > ⋯ > t M − 1 > t M > 0. (9) T>t_0>t_1>\cdots>t_{M-1}>t_M>0. \tag*{(9)} T > t 0 > t 1 > ⋯ > t M − 1 > t M > 0. (9)
The corresponding step sizes in the log-SNR variable are
δ i : = λ t i − λ t i − 1 > 0 , i = 1 , … , M . (10) \delta_i:=\lambda_{t_i}-\lambda_{t_{i-1}}>0,\qquad i=1,\ldots,M. \tag*{(10)} δ i := λ t i − λ t i − 1 > 0 , i = 1 , … , M . (10)
We initialize x t 0 ∼ N ( 0 , σ t 0 2 I d ) x_{t_0}\sim\mathcal{N}(0,\sigma_{t_0}^2I_d) x t 0 ∼ N ( 0 , σ t 0 2 I d ) , which is a Gaussian approximation to q t 0 q_{t_0} q t 0 . Both deterministic and stochastic samplers then generate x t 1 , … , x t M x_{t_1},\ldots,x_{t_M} x t 1 , … , x t M sequentially to approximate the corresponding diffusion ODE or SDE introduced below.
Diffusion ODE and deterministic samplers
Diffusion ODE. Replacing the score function s t ⋆ ( x ) s_t^\star(x) s t ⋆ ( x ) with its approximation − ϵ θ ( x , t ) / σ t -\epsilon_\theta(x,t)/\sigma_t − ϵ θ ( x , t ) / σ t in the probability flow ODE (4) yields the diffusion ODE :
x ˙ t = f ( t ) x t + g 2 ( t ) 2 σ t ϵ θ ( x t , t ) , t ∈ [ t M , t 0 ] . (11) \dot{x}_t=f(t)x_t+\frac{g^2(t)}{2\sigma_t}\epsilon_\theta(x_t,t),\qquad t\in[t_M,t_0]. \tag*{(11)} x ˙ t = f ( t ) x t + 2 σ t g 2 ( t ) ϵ θ ( x t , t ) , t ∈ [ t M , t 0 ] . (11)
Under time reparameterization (8) , its solution x t x_t x t at any time t < s t<s t < s can be expressed as
x t = α t α s x s − α t ∫ λ ( s ) λ ( t ) e − λ ϵ θ ( x t ( λ ) , t ( λ ) ) d λ (12) x_t=\frac{\alpha_t}{\alpha_s}x_s-\alpha_t\int_{\lambda(s)}^{\lambda(t)}e^{-\lambda}\epsilon_\theta(x_{t(\lambda)},t(\lambda))\,\mathrm{d}\lambda \tag*{(12)} x t = α s α t x s − α t ∫ λ ( s ) λ ( t ) e − λ ϵ θ ( x t ( λ ) , t ( λ )) d λ (12)
= σ t σ s x s + σ t ∫ λ ( s ) λ ( t ) e λ μ θ ( x t ( λ ) , t ( λ ) ) d λ . (13) =\frac{\sigma_t}{\sigma_s}x_s+\sigma_t\int_{\lambda(s)}^{\lambda(t)}e^\lambda\mu_\theta(x_{t(\lambda)},t(\lambda))\,\mathrm{d}\lambda. \tag*{(13)} = σ s σ t x s + σ t ∫ λ ( s ) λ ( t ) e λ μ θ ( x t ( λ ) , t ( λ )) d λ . (13)
Diffusion ODE solvers. Starting from x t 0 x_{t_0} x t 0 , an ODE solver produces the states x t 1 , … , x t M x_{t_1}, \ldots, x_{t_M} x t 1 , … , x t M in sequence, each approximating the continuous solution at the corresponding grid point. The deterministic DDIM sampler Song et al., [59 ] admits the following first-order update:
x t i = α t i α t i − 1 x t i − 1 − α t i ( σ t i − 1 α t i − 1 − σ t i α t i ) ε t i − 1 = α t i α t i − 1 x t i − 1 − σ t i ( e δ i − 1 ) ε t i − 1 . (14) x_{t_i}=\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}x_{t_{i-1}}-\alpha_{t_i}\left(\frac{\sigma_{t_{i-1}}}{\alpha_{t_{i-1}}}-\frac{\sigma_{t_i}}{\alpha_{t_i}}\right)\varepsilon_{t_{i-1}}=\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}x_{t_{i-1}}-\sigma_{t_i}\left(e^{\delta_i}-1\right)\varepsilon_{t_{i-1}}. \tag*{(14)} x t i = α t i − 1 α t i x t i − 1 − α t i ( α t i − 1 σ t i − 1 − α t i σ t i ) ε t i − 1 = α t i − 1 α t i x t i − 1 − σ t i ( e δ i − 1 ) ε t i − 1 . (14)
Equivalently, using the data prediction parameterization (6) , the DDIM update becomes
x t i = σ t i σ t i − 1 x t i − 1 + σ t i ( α t i σ t i − α t i − 1 σ t i − 1 ) μ t i − 1 . (15) x_{t_i}=\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}x_{t_{i-1}}+\sigma_{t_i}\left(\frac{\alpha_{t_i}}{\sigma_{t_i}}-\frac{\alpha_{t_{i-1}}}{\sigma_{t_{i-1}}}\right)\mu_{t_{i-1}}. \tag*{(15)} x t i = σ t i − 1 σ t i x t i − 1 + σ t i ( σ t i α t i − σ t i − 1 α t i − 1 ) μ t i − 1 . (15)
Some popular higher-order multistep solvers approximate the noise predictor along the trajectory by polynomial interpolation of previous model evaluations. For example, DPMSolver-p p p uses ε t i − 1 , … , ε t i − p \varepsilon_{t_{i-1}},\ldots,\varepsilon_{t_{i-p}} ε t i − 1 , … , ε t i − p to approximate the integral in (12) (Lu et al., 2022a). We provide the general update and the commonly used cases p = 2 , 3 p=2,3 p = 2 , 3 in Appendix A.1.
Diffusion SDE and stochastic samplers
Diffusion SDE. Given a noise predictor ε θ ( x , t ) \varepsilon_\theta(x,t) ε θ ( x , t ) , we can instantiate the SDE (3) by substituting the unknown score function s t ∗ ( x ) s_t^*(x) s t ∗ ( x ) with − ε θ ( x , t ) / σ t -\varepsilon_\theta(x,t)/\sigma_t − ε θ ( x , t ) / σ t , which yields the diffusion SDE :
d x t = [ f ( t ) x t + g 2 ( t ) σ t ε θ ( x t , t ) ] d t + g ( t ) d w ~ t , t ∈ [ t M , t 0 ] . (16) \mathrm{d}x_t=\left[f(t)x_t+\frac{g^2(t)}{\sigma_t}\varepsilon_\theta(x_t,t)\right]\mathrm{d}t+g(t)\mathrm{d}\widetilde{w}_t,\qquad t\in[t_M,t_0]. \tag*{(16)} d x t = [ f ( t ) x t + σ t g 2 ( t ) ε θ ( x t , t ) ] d t + g ( t ) d w t , t ∈ [ t M , t 0 ] . (16)
Under the time reparameterization (8) , the solution x t x_t x t to the diffusion SDE (16) at any time t < s t<s t < s admits the data prediction representation
x t = σ t σ s e − ( λ t − λ s ) x s + 2 α t ∫ λ s λ t e − 2 ( λ t − λ ) μ θ ( x t ( λ ) , t ( λ ) ) d λ + 2 σ t ∫ λ s λ t e − ( λ t − λ ) d w ‾ λ , (17) x_t=\frac{\sigma_t}{\sigma_s}e^{-(\lambda_t-\lambda_s)}x_s+2\alpha_t\int_{\lambda_s}^{\lambda_t}e^{-2(\lambda_t-\lambda)}\mu_\theta(x_{t(\lambda)},t(\lambda))\mathrm{d}\lambda+\sqrt{2\sigma_t}\int_{\lambda_s}^{\lambda_t}e^{-(\lambda_t-\lambda)}\mathrm{d}\overline{w}_\lambda, \tag*{(17)} x t = σ s σ t e − ( λ t − λ s ) x s + 2 α t ∫ λ s λ t e − 2 ( λ t − λ ) μ θ ( x t ( λ ) , t ( λ )) d λ + 2 σ t ∫ λ s λ t e − ( λ t − λ ) d w λ , (17)
Here, w ‾ λ \overline{w}_\lambda w λ is a standard d d d -dimensional Brownian motion in the increasing variable λ \lambda λ , obtained by time-changing the reverse-time noise. For a detailed derivation, see Lu et al. (2022b).
Diffusion SDE solvers. We consider two stochastic samplers based on (17) (Lu et al., 2022b). In both updates, z 1 , … , z M z_1,\ldots,z_M z 1 , … , z M are independent standard Gaussian vectors, independent of the initial state.
x t i = e − δ i σ t i σ t i − 1 x t i − 1 + α t i ( 1 − e − 2 δ i ) μ θ ( x t i − 1 , t i − 1 ) + σ t i 1 − e − 2 δ i z i . (18) x_{t_i}=e^{-\delta_i}\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}x_{t_{i-1}}+\alpha_{t_i}\left(1-e^{-2\delta_i}\right)\mu_\theta(x_{t_{i-1}},t_{i-1})+\sigma_{t_i}\sqrt{1-e^{-2\delta_i}}z_i. \tag*{(18)} x t i = e − δ i σ t i − 1 σ t i x t i − 1 + α t i ( 1 − e − 2 δ i ) μ θ ( x t i − 1 , t i − 1 ) + σ t i 1 − e − 2 δ i z i . (18)
x t i = e − δ i σ t i σ t i − 1 x t i − 1 + α t i ( 1 − e − 2 δ i ) μ θ ( x t i − 1 , t i − 1 ) + 1 2 δ i α t i ( 1 − e − 2 δ i ) D t i + σ t i 1 − e − 2 δ i z i , (19) x_{t_i}=e^{-\delta_i}\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}x_{t_{i-1}}+\alpha_{t_i}\left(1-e^{-2\delta_i}\right)\mu_\theta(x_{t_{i-1}},t_{i-1})+\frac{1}{2}\delta_i\alpha_{t_i}\left(1-e^{-2\delta_i}\right)D_{t_i}+\sigma_{t_i}\sqrt{1-e^{-2\delta_i}}z_i, \tag*{(19)} x t i = e − δ i σ t i − 1 σ t i x t i − 1 + α t i ( 1 − e − 2 δ i ) μ θ ( x t i − 1 , t i − 1 ) + 2 1 δ i α t i ( 1 − e − 2 δ i ) D t i + σ t i 1 − e − 2 δ i z i , (19)
where
D t 1 = 0 , D t i : = μ θ ( x t i − 1 , t i − 1 ) − μ θ ( x t i − 2 , t i − 2 ) λ t i − 1 − λ t i − 2 , i = 2 , … , M . D_{t_1}=0,\qquad D_{t_i}:=\frac{\mu_\theta(x_{t_{i-1}},t_{i-1})-\mu_\theta(x_{t_{i-2}},t_{i-2})}{\lambda_{t_{i-1}}-\lambda_{t_{i-2}}},\qquad i=2,\ldots,M. D t 1 = 0 , D t i := λ t i − 1 − λ t i − 2 μ θ ( x t i − 1 , t i − 1 ) − μ θ ( x t i − 2 , t i − 2 ) , i = 2 , … , M .
Main results
In this section, we introduce a forward-value stochastic sampler and present the convergence guarantees of both the existing and our proposed stochastic samplers.
Forward-value discretization
This subsection reviews the forward-value discretization framework and its deterministic implementations from our preliminary conference version (Jiao et al., 2026). This framework provides the basis for the stochastic extension in § 3.2 .
Ideal forward-value discretization. The DDIM update (15) approximates the data predictor along each integration interval by its value at the current iterate. With few sampling steps, the model evaluation placement can substantially affect sample quality. We therefore consider evaluating the data predictor at the next state-time pair, leading to the ideal forward-value discretization:
x t i f o r = σ t i σ t i − 1 x t i − 1 f o r + ( α t i − σ t i α t i − 1 σ t i − 1 ) μ θ ( x t i f o r , t i ) (20) x_{t_i}^{\mathrm{for}} = \frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}x_{t_{i-1}}^{\mathrm{for}} + \left(\alpha_{t_i} - \frac{\sigma_{t_i}\alpha_{t_{i-1}}}{\sigma_{t_{i-1}}}\right)\mu_\theta\left(x_{t_i}^{\mathrm{for}}, t_i\right) \tag*{(20)} x t i for = σ t i − 1 σ t i x t i − 1 for + ( α t i − σ t i − 1 σ t i α t i − 1 ) μ θ ( x t i for , t i ) (20)
Here, “forward-value” refers to evaluation at the next endpoint along the sampling direction, even though it is a discretization of the reverse process. Unlike DDIM, this update is implicit because the unknown next state x t i f o r x_{t_i}^{\mathrm{for}} x t i for also appears in the model input. To obtain an explicit sampler, we replace this input with an estimate of the next state. Under suitable regularity and stability conditions, both DDIM and (20) converge to the same diffusion ODE as the maximum step size vanishes. Our focus is their behavior with few sampling steps.
Figure 2 explores forward-value discretization with few sampling steps. For this experiment, we use five DDIM substeps within each interval ( t i − 1 , t i ) (t_{i-1}, t_i) ( t i − 1 , t i ) to obtain an endpoint estimate at t i t_i t i , and then use the data prediction at that estimate in the forward-value update. This reference implementation approximates endpoint evaluation without solving the implicit equation (20) . It achieves lower FID than DDIM and DPMSolver-2 in the tested settings on CIFAR-10 and ImageNet-64. Since the endpoint estimates require additional model evaluations, these results motivate the exploration of forward-value evaluation but do not establish an efficiency gain at matched NFE budgets. This raises the practical question: can we approximate forward-value evaluation while retaining one model evaluation per iteration?
Figure 2. FID comparison of DDIM, DPMSolver-2, and a forward-value discretization on CIFAR-10 (EDM-uncond) and ImageNet-64 (EDM2-S), versus the number of outer iterations M M M . The forward-value discretization uses additional model evaluations to estimate endpoints. FID is computed using 50K generated samples.
F-DPMSolver: first-order forward-value deterministic sampler. We next describe a practical F-DPMSolver, which replaces the implicit endpoint in (20) with a lookahead estimate. At iteration i = 1 , … , M i = 1,\ldots,M i = 1 , … , M , we first construct a lookahead estimate x ^ t i \hat{x}_{t_i} x ^ t i of the next state using only information available from Algorithm 1 F-DDIM
Input: Data prediction model μ θ ( x , t ) \mu_\theta(x,t) μ θ ( x , t ) , time grid t 0 > ⋯ > t M t_0 > \cdots> t_M t 0 > ⋯ > t M , noise schedule ( α t , σ t ) (\alpha_t,\sigma_t) ( α t , σ t ) , and initialization x t 0 x_{t_0} x t 0 .
Initialize ε ~ t 0 ← x t 0 \tilde{\varepsilon}_{t_0} \leftarrow x_{t_0} ε ~ t 0 ← x t 0 .
for i = 1 , … , M i = 1,\ldots,M i = 1 , … , M do
Compute a lookahead estimate
x ^ t i ← α t i α t i − 1 x t i − 1 − ( α t i σ t i − 1 α t i − 1 − σ t i ) ε ~ t i − 1 . \widehat{x}_{t_i} \leftarrow\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}x_{t_{i-1}}-\left(\frac{\alpha_{t_i}\sigma_{t_{i-1}}}{\alpha_{t_{i-1}}}-\sigma_{t_i}\right)\tilde{\varepsilon}_{t_{i-1}}. x t i ← α t i − 1 α t i x t i − 1 − ( α t i − 1 α t i σ t i − 1 − σ t i ) ε ~ t i − 1 .
Update
x t i ← σ t i σ t i − 1 x t i − 1 + ( α t i − σ t i α t i − 1 σ t i − 1 ) μ θ ( x ^ t i , t i ) . x_{t_i} \leftarrow\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}x_{t_{i-1}}+\left(\alpha_{t_i}-\frac{\sigma_{t_i}\alpha_{t_{i-1}}}{\sigma_{t_{i-1}}}\right)\mu_\theta(\widehat{x}_{t_i},t_i). x t i ← σ t i − 1 σ t i x t i − 1 + ( α t i − σ t i − 1 σ t i α t i − 1 ) μ θ ( x t i , t i ) .
Set and cache ε ~ t i ← ( x ^ t i − α t i μ θ ( x ^ t i , t i ) ) / σ t i \tilde{\varepsilon}_{t_i} \leftarrow(\widehat{x}_{t_i}-\alpha_{t_i}\mu_\theta(\widehat{x}_{t_i},t_i))/\sigma_{t_i} ε ~ t i ← ( x t i − α t i μ θ ( x t i , t i )) / σ t i .
end for
Output: Generated sample x t M x_{t_M} x t M .
earlier iterations, without a new model evaluation. We then evaluate the data prediction model at ( x ^ t i , t i ) (\widehat{x}_{t_i},t_i) ( x t i , t i ) and update the current iterate via
x t i = σ t i σ t i − 1 x t i − 1 + ( α t i − σ t i α t i − 1 σ t i − 1 ) μ θ ( x ^ t i , t i ) . (21) x_{t_i}=\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}x_{t_{i-1}}+\left(\alpha_{t_i}-\frac{\sigma_{t_i}\alpha_{t_{i-1}}}{\sigma_{t_{i-1}}}\right)\mu_\theta(\widehat{x}_{t_i},t_i). \tag*{(21)} x t i = σ t i − 1 σ t i x t i − 1 + ( α t i − σ t i − 1 σ t i α t i − 1 ) μ θ ( x t i , t i ) . (21)
This update has the same form as DDIM (15) , except that the current-state evaluation μ θ ( x t i − 1 , t i − 1 ) \mu_\theta(x_{t_{i-1}},t_{i-1}) μ θ ( x t i − 1 , t i − 1 ) is replaced with the lookahead evaluation μ θ ( x ^ t i , t i ) \mu_\theta(\widehat{x}_{t_i},t_i) μ θ ( x t i , t i ) .
The data predictor output gives the noise prediction
ε ~ t i : = x ^ t i − α t i μ θ ( x ^ t i , t i ) σ t i = ε θ ( x ^ t i , t i ) . (22) \tilde{\varepsilon}_{t_i}:=\frac{\widehat{x}_{t_i}-\alpha_{t_i}\mu_\theta(\widehat{x}_{t_i},t_i)}{\sigma_{t_i}}=\varepsilon_\theta(\widehat{x}_{t_i},t_i). \tag*{(22)} ε ~ t i := σ t i x t i − α t i μ θ ( x t i , t i ) = ε θ ( x t i , t i ) . (22)
The noise prediction ε ~ t i \tilde{\varepsilon}_{t_i} ε ~ t i is cached for subsequent lookahead steps. We initialize the cache by setting ε ~ t 0 = x t 0 \tilde{\varepsilon}_{t_0}=x_{t_0} ε ~ t 0 = x t 0 without querying the model. Thus, each iteration requires exactly one new model evaluation, giving a total NFE of M M M . Algorithm 3 in Appendix § 7 .§ 3 summarizes the framework.
F-DDIM. A simple choice of lookahead predictor is the DDIM update formula (14) with the cached noise estimate in place of a fresh model evaluation:
x ^ t i = α t i α t i − 1 x t i − 1 − ( α t i σ t i − 1 α t i − 1 − σ t i ) ε ~ t i − 1 . (23) \widehat{x}_{t_i}=\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}x_{t_{i-1}}-\left(\frac{\alpha_{t_i}\sigma_{t_{i-1}}}{\alpha_{t_{i-1}}}-\sigma_{t_i}\right)\tilde{\varepsilon}_{t_{i-1}}. \tag*{(23)} x t i = α t i − 1 α t i x t i − 1 − ( α t i − 1 α t i σ t i − 1 − σ t i ) ε ~ t i − 1 . (23)
For i ≥ 2 i \ge2 i ≥ 2 , the cached estimate is ε ~ t i − 1 = ε θ ( x ^ t i − 1 , t i − 1 ) \tilde{\varepsilon}_{t_{i-1}}=\varepsilon_\theta(\widehat{x}_{t_{i-1}},t_{i-1}) ε ~ t i − 1 = ε θ ( x t i − 1 , t i − 1 ) . Thus, the predictor starts from the updated state x t i − 1 x_{t_{i-1}} x t i − 1 but reuses the model output computed at the previous lookahead point x ^ t i − 1 \widehat{x}_{t_{i-1}} x t i − 1 . We call this instantiation F-DDIM and give its complete procedure in Algorithm 1.
Higher-order lookahead predictors. The framework also accommodates predictors that use multiple cached noise estimates to construct the lookahead prediction. We denote the variant based on a p p p -th order DPMSolver predictor by F-DPMSolver-p . At iteration i i i , we use a predictor of order k i = min { p , i } k_i=\min\{p,i\} k i = min { p , i } with the cached estimates ε ~ t i − 1 , … , ε ~ t i − k i \tilde{\varepsilon}_{t_{i-1}},\ldots,\tilde{\varepsilon}_{t_{i-k_i}} ε ~ t i − 1 , … , ε ~ t i − k i . The procedure starts with the F-DDIM predictor and increases the predictor order as more cached outputs become available, without additional model evaluations. For example, the F-DPMSolver-2 predictor uses cached estimates ε ~ t i − 1 \tilde{\varepsilon}_{t_{i-1}} ε ~ t i − 1 and ε ~ t i − 2 \tilde{\varepsilon}_{t_{i-2}} ε ~ t i − 2 to construct x ^ t i \widehat{x}_{t_i} x t i for i ≥ 2 i \ge2 i ≥ 2 . Here, p p p specifies the lookahead predictor; it does not imply p p p -th order convergence of the resulting sampler.
The deterministic framework shifts model evaluation to a lookahead point while retaining one model evaluation per iteration. Its stochastic extension must additionally account for the noise injected during each update, which affects both the appropriate evaluation time and the construction of the cached noise estimate. We address these two issues in § 3.2 .
Algorithm 2 F-DDPM
Input: Data prediction model μ θ ( x , t ) \mu_\theta(x,t) μ θ ( x , t ) , time grid t 0 > ⋯ > t M t_0 > \cdots> t_M t 0 > ⋯ > t M , noise schedule ( α t , σ t ) (\alpha_t,\sigma_t) ( α t , σ t ) , initialization x t 0 x_{t_0} x t 0 .
Compute λ t i ← log ( α t i / σ t i ) \lambda_{t_i} \leftarrow\log(\alpha_{t_i}/\sigma_{t_i}) λ t i ← log ( α t i / σ t i ) for i = 0 , … , M i=0,\ldots,M i = 0 , … , M
Generate Gaussian noise z 1 , … , z M ∼ i . i . d . N ( 0 , I d ) z_1,\ldots,z_M \mathrel{\overset{\mathrm{i.i.d.}}{\sim}} \mathcal{N}(0,I_d) z 1 , … , z M ∼ i.i.d. N ( 0 , I d )
Initialize ϵ ~ t 0 ← x t 0 \widetilde{\epsilon}_{t_0} \leftarrow x_{t_0} ϵ t 0 ← x t 0
for i = 1 , … , M i=1,\ldots,M i = 1 , … , M do
Set δ i ← λ t i − λ t i − 1 \delta_i \leftarrow\lambda_{t_i}-\lambda_{t_{i-1}} δ i ← λ t i − λ t i − 1 and compute t i ′ t_i' t i ′ such that λ t i ′ = λ t i + δ i \lambda_{t_i'}=\lambda_{t_i}+\delta_i λ t i ′ = λ t i + δ i
Compute a lookahead estimate x ^ t i ′ ← α t i ′ α t i − 1 x t i − 1 − σ t i ′ ( e 2 δ i − 1 ) ϵ ~ t i − 1 \widehat{x}_{t_i'} \leftarrow\frac{\alpha_{t_i'}}{\alpha_{t_{i-1}}}x_{t_{i-1}}-\sigma_{t_i'}(e^{2\delta_i}-1)\widetilde{\epsilon}_{t_{i-1}} x t i ′ ← α t i − 1 α t i ′ x t i − 1 − σ t i ′ ( e 2 δ i − 1 ) ϵ t i − 1
Update x t i ← e − δ i σ t i σ t i − 1 x t i − 1 + α t i ( 1 − e − 2 δ i ) μ θ ( x ^ t i ′ , t i ′ ) + σ t i 1 − e − 2 δ i z i x_{t_i} \leftarrow e^{-\delta_i}\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}x_{t_{i-1}}+\alpha_{t_i}(1-e^{-2\delta_i})\mu_\theta(\widehat{x}_{t_i'},t_i')+\sigma_{t_i}\sqrt{1-e^{-2\delta_i}}z_i x t i ← e − δ i σ t i − 1 σ t i x t i − 1 + α t i ( 1 − e − 2 δ i ) μ θ ( x t i ′ , t i ′ ) + σ t i 1 − e − 2 δ i z i .
Set and cache ϵ ~ t i ← e − δ i ϵ θ ( x ^ t i ′ , t i ′ ) + 1 − e − 2 δ i z i \widetilde{\epsilon}_{t_i}\leftarrow e^{-\delta_i}\epsilon_\theta(\widehat{x}_{t_i'},t_i')+\sqrt{1-e^{-2\delta_i}}z_i ϵ t i ← e − δ i ϵ θ ( x t i ′ , t i ′ ) + 1 − e − 2 δ i z i , with ϵ θ ( x ^ t i ′ , t i ′ ) = ( x ^ t i ′ − α t i ′ μ θ ( x ^ t i ′ , t i ′ ) ) / σ t i ′ \epsilon_\theta(\widehat{x}_{t_i'},t_i')=(\widehat{x}_{t_i'}-\alpha_{t_i'}\mu_\theta(\widehat{x}_{t_i'},t_i'))/\sigma_{t_i'} ϵ θ ( x t i ′ , t i ′ ) = ( x t i ′ − α t i ′ μ θ ( x t i ′ , t i ′ )) / σ t i ′
end for
Output: Generated sample x t M x_{t_M} x t M .
F-DDPM: first-order forward-value stochastic sampler
Building on the forward-value discretization framework, we introduce F-DDPM , a fast stochastic sampler that serves as the forward-value counterpart of DDPM under the data-prediction parametrization in (18) . The complete procedure is summarized in Algorithm 2.
At a high level, F-DDPM has the same form as DDPM, and the only difference is where the data prediction model μ θ \mu_\theta μ θ is evaluated. DDPM evaluates it at the current state-time pair ( x t i − 1 , t i − 1 ) (x_{t_{i-1}},t_{i-1}) ( x t i − 1 , t i − 1 ) , whereas F-DDPM evaluates it at a lookahead state-time pair ( x ^ t i ′ , t i ′ ) (\widehat{x}_{t_i'},t_i') ( x t i ′ , t i ′ ) built from information available through step i − 1 i-1 i − 1 only. Concretely, for iteration i = 1 , … , M i=1,\ldots,M i = 1 , … , M , we update the iterate x t i − 1 x_{t_{i-1}} x t i − 1 via
x t i = e − δ i σ t i σ t i − 1 x t i − 1 + α t i ( 1 − e − 2 δ i ) μ θ ( x ^ t i ′ , t i ′ ) + σ t i 1 − e − 2 δ i z i , (24) x_{t_i}=e^{-\delta_i}\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}x_{t_{i-1}}+\alpha_{t_i}(1-e^{-2\delta_i})\mu_\theta(\widehat{x}_{t_i'},t_i')+\sigma_{t_i}\sqrt{1-e^{-2\delta_i}}z_i, \tag*{(24)} x t i = e − δ i σ t i − 1 σ t i x t i − 1 + α t i ( 1 − e − 2 δ i ) μ θ ( x t i ′ , t i ′ ) + σ t i 1 − e − 2 δ i z i , (24)
where z i ∼ i . i . d . N ( 0 , I d ) z_i\mathrel{\overset{\mathrm{i.i.d.}}{\sim}}\mathcal{N}(0,I_d) z i ∼ i.i.d. N ( 0 , I d ) for i = 1 , … , M i=1,\ldots,M i = 1 , … , M are independent Gaussian noise and δ i : = λ t i − λ t i − 1 \delta_i:=\lambda_{t_i}-\lambda_{t_{i-1}} δ i := λ t i − λ t i − 1 is defined in (10) .
Below, let us specify the evaluation time t i ′ t_i' t i ′ and the lookahead estimate x ^ t i ′ \widehat{x}_{t_i'} x t i ′ .
Evaluation time t i ′ t_i' t i ′ . Recall that for the deterministic sampler F-DPMSolver in (21) , the forward-value evaluation is taken at the end time t i t_i t i of the current step. In contrast, F-DDPM evaluates the data predictor at a cleaner time t i ′ < t i t_i'<t_i t i ′ < t i , chosen to account for the noise injected during the stochastic update.
For each i ≥ 1 i\ge1 i ≥ 1 , let t i ′ t_i' t i ′ be the time whose log-SNR λ t i ′ \lambda_{t_i'} λ t i ′ exceeds that of the end time t i t_i t i by δ i > 0 \delta_i>0 δ i > 0 :
λ t i ′ = λ t i + δ i = λ t i − 1 + 2 δ i . (25) \lambda_{t_i'}=\lambda_{t_i}+\delta_i=\lambda_{t_{i-1}}+2\delta_i. \tag*{(25)} λ t i ′ = λ t i + δ i = λ t i − 1 + 2 δ i . (25)
Since λ t \lambda_t λ t is decreasing in t t t , the evaluation time t i ′ < t i t_i'<t_i t i ′ < t i lies past the current end time t i t_i t i along the sampling direction. When the time grid is uniform in log-SNR, we have t i ′ = t i + 1 t_i'=t_{i+1} t i ′ = t i + 1 for i = 1 , … , M − 1 i=1,\ldots,M-1 i = 1 , … , M − 1 .
To motivate this choice, denote by x t ⋆ x_t^\star x t ⋆ the solution to the SDE (2) for the forward process. Conditioned on the state x t i − 1 ⋆ x_{t_{i-1}}^\star x t i − 1 ⋆ at time t i − 1 t_{i-1} t i − 1 and the clean data x 0 ⋆ x_0^\star x 0 ⋆ , the posterior distribution of the state x t i ⋆ x_{t_i}^\star x t i ⋆ at time t i t_i t i is Gaussian with covariance σ t i 2 ( 1 − e − 2 δ i ) I d \sigma_{t_i}^2(1-e^{-2\delta_i})I_d σ t i 2 ( 1 − e − 2 δ i ) I d and mean
e − δ i σ t i σ t i − 1 x t i − 1 ⋆ + α t i ( 1 − e − 2 δ i ) x 0 ⋆ = α t i x 0 ⋆ + σ t i e − δ i ϵ (26) e^{-\delta_i}\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}x_{t_{i-1}}^\star+\alpha_{t_i}(1-e^{-2\delta_i})x_0^\star=\alpha_{t_i}x_0^\star+\sigma_{t_i}e^{-\delta_i}\epsilon \tag*{(26)} e − δ i σ t i − 1 σ t i x t i − 1 ⋆ + α t i ( 1 − e − 2 δ i ) x 0 ⋆ = α t i x 0 ⋆ + σ t i e − δ i ϵ (26)
where ϵ = ( x t i − 1 ⋆ − α t i − 1 x 0 ⋆ ) / σ t i − 1 ∼ N ( 0 , I d ) \epsilon=(x_{t_{i-1}}^\star-\alpha_{t_{i-1}}x_0^\star)/\sigma_{t_{i-1}}\sim\mathcal{N}(0,I_d) ϵ = ( x t i − 1 ⋆ − α t i − 1 x 0 ⋆ ) / σ t i − 1 ∼ N ( 0 , I d ) is independent of x 0 ⋆ x_0^\star x 0 ⋆ . The i i i -th update of DDPM is thus approximate sampling from this posterior distribution with x 0 ⋆ x_0^\star x 0 ⋆ replaced by μ θ ( x t i − 1 , t i − 1 ) \mu_\theta(x_{t_{i-1}},t_{i-1}) μ θ ( x t i − 1 , t i − 1 ) . Now, notice that the right-hand side of (26) can be interpreted as noisy data with log-SNR
log ( α t i σ t i e − δ i ) = λ t i + δ i , \log\left(\frac{\alpha_{t_i}}{\sigma_{t_i}e^{-\delta_i}}\right)=\lambda_{t_i}+\delta_i, log ( σ t i e − δ i α t i ) = λ t i + δ i ,
which is higher than λ t i \lambda_{t_i} λ t i at time t i t_i t i . Matching this log-SNR with λ t i ′ = log ( α t i ′ / σ t i ′ ) \lambda_{t_i'} = \log(\alpha_{t_i'}/\sigma_{t_i'}) λ t i ′ = log ( α t i ′ / σ t i ′ ) yields the choice of t i ′ t_i' t i ′ in (25) . The injected Gaussian noise with variance σ t i 2 ( 1 − e − 2 δ i ) \sigma_{t_i}^{2}(1-e^{-2\delta_i}) σ t i 2 ( 1 − e − 2 δ i ) then restores the total noise variance σ t i 2 \sigma_{t_i}^{2} σ t i 2 at time t i t_i t i .
This choice also yields a useful algebraic interpretation. By the definition of t i ′ t_i' t i ′ , one can rewrite the update rule of DDPM in (18) as
x t i = α t i α t i ′ [ σ t i ′ σ t i − 1 x t i − 1 + σ t i ′ ( α t i ′ σ t i ′ − α t i − 1 σ t i − 1 ) μ θ ( x t i − 1 , t i − 1 ) ] + σ t i 1 − e − 2 δ i z i (27) \mathbf{x}_{t_i}=\frac{\alpha_{t_i}}{\alpha_{t_i'}}\left[\frac{\sigma_{t_i'}}{\sigma_{t_{i-1}}}\mathbf{x}_{t_{i-1}}+\sigma_{t_i'}\left(\frac{\alpha_{t_i'}}{\sigma_{t_i'}}-\frac{\alpha_{t_{i-1}}}{\sigma_{t_{i-1}}}\right)\mu_\theta(\mathbf{x}_{t_{i-1}},t_{i-1})\right]+\sigma_{t_i}\sqrt{1-e^{-2\delta_i}}z_i \tag*{(27)} x t i = α t i ′ α t i [ σ t i − 1 σ t i ′ x t i − 1 + σ t i ′ ( σ t i ′ α t i ′ − σ t i − 1 α t i − 1 ) μ θ ( x t i − 1 , t i − 1 ) ] + σ t i 1 − e − 2 δ i z i (27)
The quantity in the square bracket corresponds to a one-step DDIM update from t i − 1 t_{i-1} t i − 1 to t i ′ t_i' t i ′ . Therefore, the stochastic update of DDPM can be interpreted as (1) taking a DDIM step to a cleaner t i ′ t_i' t i ′ , (2) rescaling the result to the signal level α t i \alpha_{t_i} α t i , and (3) injecting Gaussian noise to obtain the total noise level σ t i \sigma_{t_i} σ t i at time t i t_i t i .
The ideal forward-value rule applied to this DDIM step from t i − 1 t_{i-1} t i − 1 to t i ′ t_i' t i ′ would replace its start-time evaluation μ θ ( x t i − 1 , t i − 1 ) \mu_\theta(\mathbf{x}_{t_{i-1}},t_{i-1}) μ θ ( x t i − 1 , t i − 1 ) with the end-time evaluation μ θ ( x t i ′ , t i ′ ) \mu_\theta(\mathbf{x}_{t_i'},t_i') μ θ ( x t i ′ , t i ′ ) . Here, x t i ′ \mathbf{x}_{t_i'} x t i ′ denotes the idealized endpoint of this deterministic substep, rather than an already generated stochastic iterate. As x t i ′ \mathbf{x}_{t_i'} x t i ′ is not available at step i − 1 i-1 i − 1 , we approximate it with a lookahead estimate x ^ t i ′ \widehat{\mathbf{x}}_{t_i'} x t i ′ constructed using only information available up to step i − 1 i-1 i − 1 , leading to the modified term in the square bracket:
σ t i ′ σ t i − 1 x t i − 1 + σ t i ′ ( α t i ′ σ t i ′ − α t i − 1 σ t i − 1 ) μ θ ( x ^ t i ′ , t i ′ ) . \frac{\sigma_{t_i'}}{\sigma_{t_{i-1}}}\mathbf{x}_{t_{i-1}}+\sigma_{t_i'}\left(\frac{\alpha_{t_i'}}{\sigma_{t_i'}}-\frac{\alpha_{t_{i-1}}}{\sigma_{t_{i-1}}}\right)\mu_\theta(\widehat{\mathbf{x}}_{t_i'},t_i'). σ t i − 1 σ t i ′ x t i − 1 + σ t i ′ ( σ t i ′ α t i ′ − σ t i − 1 α t i − 1 ) μ θ ( x t i ′ , t i ′ ) .
Plugging this into (27) yields the update rule of F-DDPM in (24) . It remains to specify the lookahead estimate x ^ t i ′ \widehat{\mathbf{x}}_{t_i'} x t i ′ .
Lookahead estimate x ^ t i ′ \widehat{\mathbf{x}}_{t_i'} x t i ′ . Given the evaluation time t i ′ t_i' t i ′ , we construct the lookahead estimate x ^ t i ′ \widehat{\mathbf{x}}_{t_i'} x t i ′ by taking a single DDIM step (14) from t i − 1 t_{i-1} t i − 1 to t i ′ t_i' t i ′ , written in the noise-prediction form:
x ^ t i ′ = α t i ′ α t i − 1 x t i − 1 − α t i ′ ( σ t i − 1 α t i − 1 − σ t i ′ α t i ′ ) ϵ ~ t i − 1 = α t i ′ α t i − 1 x t i − 1 − σ t i ′ ( e 2 δ i − 1 ) ϵ ~ t i − 1 . (28) \widehat{\mathbf{x}}_{t_i'}=\frac{\alpha_{t_i'}}{\alpha_{t_{i-1}}}\mathbf{x}_{t_{i-1}}-\alpha_{t_i'}\left(\frac{\sigma_{t_{i-1}}}{\alpha_{t_{i-1}}}-\frac{\sigma_{t_i'}}{\alpha_{t_i'}}\right)\widetilde{\boldsymbol{\epsilon}}_{t_{i-1}}=\frac{\alpha_{t_i'}}{\alpha_{t_{i-1}}}\mathbf{x}_{t_{i-1}}-\sigma_{t_i'}\left(e^{2\delta_i}-1\right)\widetilde{\boldsymbol{\epsilon}}_{t_{i-1}}. \tag*{(28)} x t i ′ = α t i − 1 α t i ′ x t i − 1 − α t i ′ ( α t i − 1 σ t i − 1 − α t i ′ σ t i ′ ) ϵ t i − 1 = α t i − 1 α t i ′ x t i − 1 − σ t i ′ ( e 2 δ i − 1 ) ϵ t i − 1 . (28)
Here, ϵ ~ t i − 1 \widetilde{\boldsymbol{\epsilon}}_{t_{i-1}} ϵ t i − 1 is an estimate of the noise prediction ϵ θ ( x t i − 1 , t i − 1 ) \boldsymbol{\epsilon}_\theta(\mathbf{x}_{t_{i-1}},t_{i-1}) ϵ θ ( x t i − 1 , t i − 1 ) using model evaluations from the previous step i − 1 i-1 i − 1 for 1 ≥ 2 1\geq2 1 ≥ 2 . This avoids an additional model evaluation for ϵ θ ( x t i − 1 , t i − 1 ) \boldsymbol{\epsilon}_\theta(\mathbf{x}_{t_{i-1}},t_{i-1}) ϵ θ ( x t i − 1 , t i − 1 ) at the current step.
For i > 1 i>1 i > 1 , writing (27) at step i − 1 i-1 i − 1 and replacing the square bracket by the quantity it approximates, namely x t i − 1 ′ \mathbf{x}_{t_{i-1}'} x t i − 1 ′ , we obtain
x t i − 1 ≈ α t i − 1 α t i − 1 ′ x t i − 1 ′ + σ t i − 1 1 − e − 2 δ i − 1 z i − 1 = α t i − 1 α t i − 1 ′ ( α t i − 1 ′ μ θ ( x t i − 1 ′ , t i − 1 ′ ) + σ t i − 1 ′ ϵ θ ( x t i − 1 ′ , t i − 1 ′ ) ) + σ t i − 1 1 − e − 2 δ i − 1 z i − 1 = α t i − 1 μ θ ( x t i − 1 ′ , t i − 1 ′ ) + σ t i − 1 ′ α t i − 1 ′ α t i − 1 ϵ θ ( x t i − 1 ′ , t i − 1 ′ ) + σ t i − 1 1 − e − 2 δ i − 1 z i − 1 . \begin{aligned}
\mathbf{x}_{t_{i-1}}\approx\frac{\alpha_{t_{i-1}}}{\alpha_{t_{i-1}'}}\mathbf{x}_{t_{i-1}'}+\sigma_{t_{i-1}}\sqrt{1-e^{-2\delta_{i-1}}}z_{i-1} \\
&=\frac{\alpha_{t_{i-1}}}{\alpha_{t_{i-1}'}}\left(\alpha_{t_{i-1}'}\mu_\theta(\mathbf{x}_{t_{i-1}'},t_{i-1}')+\sigma_{t_{i-1}'}\boldsymbol{\epsilon}_\theta(\mathbf{x}_{t_{i-1}'},t_{i-1}')\right)+\sigma_{t_{i-1}}\sqrt{1-e^{-2\delta_{i-1}}}z_{i-1} \\
&=\alpha_{t_{i-1}}\mu_\theta(\mathbf{x}_{t_{i-1}'},t_{i-1}')+\frac{\sigma_{t_{i-1}'}}{\alpha_{t_{i-1}'}}\alpha_{t_{i-1}}\boldsymbol{\epsilon}_\theta(\mathbf{x}_{t_{i-1}'},t_{i-1}')+\sigma_{t_{i-1}}\sqrt{1-e^{-2\delta_{i-1}}}z_{i-1}.
\end{aligned} x t i − 1 ≈ α t i − 1 ′ α t i − 1 x t i − 1 ′ + σ t i − 1 1 − e − 2 δ i − 1 z i − 1 = α t i − 1 ′ α t i − 1 ( α t i − 1 ′ μ θ ( x t i − 1 ′ , t i − 1 ′ ) + σ t i − 1 ′ ϵ θ ( x t i − 1 ′ , t i − 1 ′ ) ) + σ t i − 1 1 − e − 2 δ i − 1 z i − 1 = α t i − 1 μ θ ( x t i − 1 ′ , t i − 1 ′ ) + α t i − 1 ′ σ t i − 1 ′ α t i − 1 ϵ θ ( x t i − 1 ′ , t i − 1 ′ ) + σ t i − 1 1 − e − 2 δ i − 1 z i − 1 .
where the second line applies the data/noise prediction identity (6) at t i − 1 ′ t_{i-1}' t i − 1 ′ . Meanwhile, by the same identity at t i − 1 t_{i-1} t i − 1 , we have
x t i − 1 = α t i − 1 μ θ ( x t i − 1 , t i − 1 ) + σ t i − 1 ϵ θ ( x t i − 1 , t i − 1 ) . \mathbf{x}_{t_{i-1}}=\alpha_{t_{i-1}}\mu_\theta(\mathbf{x}_{t_{i-1}},t_{i-1})+\sigma_{t_{i-1}}\boldsymbol{\epsilon}_\theta(\mathbf{x}_{t_{i-1}},t_{i-1}). x t i − 1 = α t i − 1 μ θ ( x t i − 1 , t i − 1 ) + σ t i − 1 ϵ θ ( x t i − 1 , t i − 1 ) .
If the data predictions change little between the endpoint and intermediate time, i.e., μ θ ( x t i − 1 ′ , t i − 1 ′ ) ≈ μ θ ( x t i − 1 , t i − 1 ) \mu_\theta(\mathbf{x}_{t_{i-1}'},t_{i-1}')\approx\mu_\theta(\mathbf{x}_{t_{i-1}},t_{i-1}) μ θ ( x t i − 1 ′ , t i − 1 ′ ) ≈ μ θ ( x t i − 1 , t i − 1 ) , comparing the two expressions above suggests
ϵ θ ( x t i − 1 , t i − 1 ) ≈ e − δ i − 1 ϵ θ ( x t i − 1 ′ , t i − 1 ′ ) + 1 − e − 2 δ i − 1 z i − 1 . \boldsymbol{\epsilon}_\theta(\mathbf{x}_{t_{i-1}},t_{i-1})\approx e^{-\delta_{i-1}}\boldsymbol{\epsilon}_\theta(\mathbf{x}_{t_{i-1}'},t_{i-1}')+\sqrt{1-e^{-2\delta_{i-1}}}z_{i-1}. ϵ θ ( x t i − 1 , t i − 1 ) ≈ e − δ i − 1 ϵ θ ( x t i − 1 ′ , t i − 1 ′ ) + 1 − e − 2 δ i − 1 z i − 1 .
Replacing the idealized endpoint x t i − 1 ′ \mathbf{x}_{t_{i-1}'} x t i − 1 ′ with the available lookahead estimate x ^ t i − 1 ′ \widehat{\mathbf{x}}_{t_{i-1}'} x t i − 1 ′ therefore motivates the practical estimator
ϵ ~ t i − 1 : = e − δ i − 1 ϵ θ ( x ^ t i − 1 ′ , t i − 1 ′ ) + 1 − e − 2 δ i − 1 z i − 1 . (29) \widetilde{\boldsymbol{\epsilon}}_{t_{i-1}}:=e^{-\delta_{i-1}}\boldsymbol{\epsilon}_\theta(\widehat{\mathbf{x}}_{t_{i-1}'},t_{i-1}')+\sqrt{1-e^{-2\delta_{i-1}}}z_{i-1}. \tag*{(29)} ϵ t i − 1 := e − δ i − 1 ϵ θ ( x t i − 1 ′ , t i − 1 ′ ) + 1 − e − 2 δ i − 1 z i − 1 . (29)
For i = 1 i=1 i = 1 , we simply set ϵ ~ t 0 = x t 0 \widetilde{\epsilon}_{t_0}=x_{t_0} ϵ t 0 = x t 0 as in F-DPMSolver.
To conclude, each iteration of F-DDPM requires only one new model evaluation, at ( x ^ t i , t i ′ ) (\widehat{x}_{t_i},t_i') ( x t i , t i ′ ) —the resulting noise prediction is used both to compute the data prediction in (24) and to form the noise estimate for the next iteration. Thus, F-DDPM uses M M M model evaluations in total.
Convergence analysis
We analyze the trajectory discretization errors of the forward-value samplers and the baseline diffusion ODE and SDE solvers. The numerical sampler and its continuous reference process use the same initial state. In the stochastic setting, they are additionally coupled through the same Brownian motion. These convergence guarantees characterize approximation to the continuous dynamics and complement the few-step sample-quality comparisons in § 4 .
Convergence criteria. We consider time grids { t i } i = 0 M \{t_i\}_{i=0}^{M} { t i } i = 0 M with fixed endpoints t 0 t_0 t 0 and t M t_M t M , corresponding to a fixed finite log-SNR interval. Convergence rates are measured as M → ∞ M \to\infty M → ∞ . For deterministic samplers, we fix the initial state x t 0 x_{t_0} x t 0 and measure the Euclidean error at the final time.
Definition 1. A deterministic diffusion sampler has convergence order at least k k k if
∥ x t M − x t M ⋆ ∥ 2 = O ( 1 / M k ) , \lVert x_{t_M}-x_{t_M}^{\star}\rVert_2=O(1/M^k), ∥ x t M − x t M ⋆ ∥ 2 = O ( 1/ M k ) ,
where x t ⋆ x_t^{\star} x t ⋆ is the exact solution to the diffusion ODE (12) when initialized at x t 0 ⋆ = x t 0 x_{t_0}^{\star}=x_{t_0} x t 0 ⋆ = x t 0 . In particular, for 0 ≤ t ≤ t 0 0 \le t \le t_0 0 ≤ t ≤ t 0 , it satisfies
x t ⋆ = α t α t 0 x t 0 ⋆ − α t ∫ λ t 0 λ t e − λ ϵ θ ( x t ⋆ ( λ ) , t ( λ ) ) d λ , 0 ≤ t ≤ t 0 . (30) x_t^{\star}=\frac{\alpha_t}{\alpha_{t_0}}x_{t_0}^{\star}-\alpha_t\int_{\lambda_{t_0}}^{\lambda_t}e^{-\lambda}\epsilon_{\theta}(x_{t}^{\star}(\lambda),t(\lambda))\,\mathrm{d}\lambda,\qquad0\le t\le t_0. \tag*{(30)} x t ⋆ = α t 0 α t x t 0 ⋆ − α t ∫ λ t 0 λ t e − λ ϵ θ ( x t ⋆ ( λ ) , t ( λ )) d λ , 0 ≤ t ≤ t 0 . (30)
For stochastic samplers, we measure the expected Euclidean error under a common initialization and Brownian motion.
Definition 2. A stochastic diffusion sampler has convergence order at least k k k if
E [ ∥ x t M − x t M ⋆ ∥ 2 ] = O ( 1 / M k ) , \mathbb{E}\left[\lVert x_{t_M}-x_{t_M}^{\star}\rVert_2\right]=O(1/M^k), E [ ∥ x t M − x t M ⋆ ∥ 2 ] = O ( 1/ M k ) ,
where x t ⋆ x_t^{\star} x t ⋆ is the exact solution to the diffusion SDE (17) , initialized at x t 0 ⋆ = x t 0 x_{t_0}^{\star}=x_{t_0} x t 0 ⋆ = x t 0 and driven by the same Brownian ω ‾ λ \overline{\omega}_{\lambda} ω λ as the numerical sampler. In particular, for 0 ≤ t < s ≤ t 0 0\le t<s\le t_0 0 ≤ t < s ≤ t 0 , it satisfies
x t ⋆ = σ t σ s e − ( λ t − λ s ) x s ⋆ + 2 α t ∫ λ s λ t e − 2 ( λ t − λ ) μ θ ( x t ⋆ ( λ ) , λ ) d λ + 2 σ t ∫ λ s λ t e − ( λ t − λ ) d ω ‾ λ . (31) x_t^{\star}=\frac{\sigma_t}{\sigma_s}e^{-(\lambda_t-\lambda_s)}x_s^{\star}+2\alpha_t\int_{\lambda_s}^{\lambda_t}e^{-2(\lambda_t-\lambda)}\mu_{\theta}(x_t^{\star}(\lambda),\lambda)\,\mathrm{d}\lambda+\sqrt{2}\sigma_t\int_{\lambda_s}^{\lambda_t}e^{-(\lambda_t-\lambda)}\mathrm{d}\overline{\omega}_{\lambda}. \tag*{(31)} x t ⋆ = σ s σ t e − ( λ t − λ s ) x s ⋆ + 2 α t ∫ λ s λ t e − 2 ( λ t − λ ) μ θ ( x t ⋆ ( λ ) , λ ) d λ + 2 σ t ∫ λ s λ t e − ( λ t − λ ) d ω λ . (31)
The Gaussian vectors z i z_i z i in the stochastic updates are coupled to this Brownian motion through
σ t i 1 − e − 2 δ i z i = 2 σ t i ∫ λ t i − 1 λ t i e − ( λ t i − λ ) d ω ‾ λ . \sigma_{t_i}\sqrt{1-e^{-2\delta_i}}z_i=\sqrt{2}\sigma_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-(\lambda_{t_i}-\lambda)}\mathrm{d}\overline{\omega}_{\lambda}. σ t i 1 − e − 2 δ i z i = 2 σ t i ∫ λ t i − 1 λ t i e − ( λ t i − λ ) d ω λ .
Regularity assumptions. We impose conditions on the sampling grids, noise schedules, and prediction model to control the discretization errors. The model assumptions are stated in terms of the data predictor μ θ \mu_{\theta} μ θ in (6) , which has one-to-one correspondence with the noise predictor ϵ θ \epsilon_{\theta} ϵ θ .
Assumption 1. We assume that the time grid satisfies max i ∈ [ M ] δ i = O ( 1 / M ) \max_{i\in[M]}\delta_i=O(1/M) max i ∈ [ M ] δ i = O ( 1/ M ) and max i ∈ [ M ] δ i / min i ∈ [ M ] δ i = O ( 1 ) \max_{i\in[M]}\delta_i/\min_{i\in[M]}\delta_i=O(1) max i ∈ [ M ] δ i / min i ∈ [ M ] δ i = O ( 1 ) . The noise schedules α t ( λ ) \alpha_t(\lambda) α t ( λ ) and σ t ( λ ) \sigma_t(\lambda) σ t ( λ ) are positive and twice continuously differentiable on [ λ t 0 , max 1 ≤ m ≤ M λ t m ′ ] [\lambda_{t_0},\max_{1\le m\le M}\lambda_{t_m'}] [ λ t 0 , max 1 ≤ m ≤ M λ t m ′ ] , with α t ( λ ) \alpha_t(\lambda) α t ( λ ) nondecreasing and σ t ( λ ) \sigma_t(\lambda) σ t ( λ ) nonincreasing as functions of λ \lambda λ . In addition, for every x ∈ R d x\in\mathbb{R}^d x ∈ R d and λ ∈ [ λ t 0 , max 1 ≤ m ≤ M λ t m ′ ] \lambda\in[\lambda_{t_0},\max_{1\le m\le M}\lambda_{t_m'}] λ ∈ [ λ t 0 , max 1 ≤ m ≤ M λ t m ′ ] , we assume that the data predictor μ θ ( x , t ) \mu_{\theta}(x,t) μ θ ( x , t ) satisfies
(A1) ∥ μ θ ( x , t ) ∥ ≤ L μ \lVert\mu_{\theta}(x,t)\rVert\le L_{\mu} ∥ μ θ ( x , t )∥ ≤ L μ ;
(A2) ∥ ∇ x μ θ ( x , t ) ∥ ≤ L x / α t \lVert\nabla_x\mu_{\theta}(x,t)\rVert\le L_x/\alpha_t ∥ ∇ x μ θ ( x , t )∥ ≤ L x / α t ;
∥ ∇ x 2 μ θ ( x , t ) ∥ ≤ H x ; (32) \lVert\nabla_x^2 \mu_\theta(x,t) \rVert\le H_x; \tag*{(32)} ∥ ∇ x 2 μ θ ( x , t )∥ ≤ H x ; (32)
∥ ∂ ∂ λ μ θ ( x , t ( λ ) ) ∥ ≤ L t ; (33) \left\lVert\frac{\partial}{\partial\lambda}\mu_\theta(x,t(\lambda)) \right\rVert\le L_t; \tag*{(33)} ∂ λ ∂ μ θ ( x , t ( λ )) ≤ L t ; (33)
∥ ∂ 2 ∂ λ 2 μ θ ( x , t ( λ ) ) ∥ ≤ H t ; (34) \left\lVert\frac{\partial^2}{\partial\lambda^2}\mu_\theta(x,t(\lambda)) \right\rVert\le H_t; \tag*{(34)} ∂ λ 2 ∂ 2 μ θ ( x , t ( λ )) ≤ H t ; (34)
∥ ∂ ∂ λ ∇ x μ θ ( x , t ( λ ) ) ∥ ≤ H . (35) \left\lVert\frac{\partial}{\partial\lambda}\nabla_x\mu_\theta(x,t(\lambda)) \right\rVert\le H. \tag*{(35)} ∂ λ ∂ ∇ x μ θ ( x , t ( λ )) ≤ H . (35)
Assumption 1 imposes a maximum log-SNR step size of O ( M − 1 ) O(M^{-1}) O ( M − 1 ) , a bounded ratio between the largest and smallest steps, monotonicity and smoothness of the noise schedules, and boundedness and regularity of the data predictor. The maximum-step condition and spatial Lipschitz regularity have been assumed in diffusion solver analyses ([71 ] (Lu et al. 2022a)). The bounded step ratio simplifies the treatment of nonuniform grids, and is mainly used for the analysis of SDE-DPMSolver++2. The derivative conditions support the Taylor expansions and remainder estimates used in our proofs. Related total-derivative assumptions along ODE trajectories appear in DPM-Solver and UniPC ([71 ] (Lu et al. 2022a)), while stochastic solver analyses also impose spatial and temporal coefficient regularity ([18 ] ).
Convergence characterization of deterministic samplers
Previous analyses establish error upper bounds for deterministic diffusion solvers; see, for example, ((Lu et al., 2022a, Theorem 3.2)) and ([73 Theorem 3.3] ). We complement these guarantees with matching worst-case lower bounds for DDIM and DPMSolver-2. Notably, these lower bounds cannot be avoided even in the most ideal setting when the true score functions are accessible (i.e., there is no score matching error).
Theorem 1. Under Assumption 1, the following convergence guarantees hold:
In addition, for any time grid { t i } i = 0 M \{t_i\}_{i=0}^{M} { t i } i = 0 M satisfying δ i = λ t i − λ t i − 1 = O ( M − 1 ) \delta_i=\lambda_{t_i}-\lambda_{t_{i-1}}=O(M^{-1}) δ i = λ t i − λ t i − 1 = O ( M − 1 ) , the following bounds hold, each for some target distribution:
Theorem 1 shows that the error upper bounds for DDIM and DPMSolver-2 are sharp. For these deterministic methods, the higher-order approximation improves the convergence order from 1 to 2. The proof is deferred to Appendix § 10 .
Convergence characterization of stochastic samplers
We next analyze SDE-DPMSolver++1 and SDE-DPMSolver++2. Theorem 2 gives first-order error upper bounds for both methods under the expected Euclidean error in Definition 2.
Theorem 2. Under Assumption 1, both SDE-DPMSolver++1 and SDE-DPMSolver++2 satisfy the first-order error bound in Definition 2.
The proof is deferred to Appendix § 11 . We establish matching worst-case lower bounds next. These bounds hold even with exact score functions, showing that the first-order rates cannot generally be improved for these methods.
Theorem 3. Consider any time grid { t i } i = 0 M \{t_i\}_{i=0}^{M} { t i } i = 0 M satisfying
δ i = λ t i − λ t i − 1 = O ( M − 1 ) , and max i δ i min i δ i = O ( 1 ) . \delta_i=\lambda_{t_i}-\lambda_{t_{i-1}}=O(M^{-1}), \qquad\text{and} \qquad\frac{\max_i \delta_i}{\min_i \delta_i}=O(1). δ i = λ t i − λ t i − 1 = O ( M − 1 ) , and min i δ i max i δ i = O ( 1 ) . For each of SDE-DPMSolver++1 and SDE-DPMSolver++2, there exists a target distribution for which its convergence order is at most 1.
The proof of this result is postponed to Appendix § 12 . Together, Theorems 2 and 3 establish sharp first-order convergence rates for both methods. In particular, the higher-order approximation used by SDE-DPMSolver++2 does not improve its strong convergence order over SDE-DPMSolver++1. Thus, increasing the order of a deterministic approximation does not automatically improve the strong convergence order of the resulting stochastic sampler.
Convergence of forward-value samplers
We now analyze the forward-value samplers introduced in § 3.1 and § 3.2 . We first establish a first-order error upper bound for F-DDIM and show that this rate is sharp in the worst case.
Theorem 4. Under Assumption 1, F-DDIM satisfies the first-order error bound in Definition 1. Moreover, for any time grids { λ t i } i = 0 M \{\lambda_{t_i}\}_{i=0}^{M} { λ t i } i = 0 M satisfying δ i = O ( M − 1 ) \delta_i = O(M^{-1}) δ i = O ( M − 1 ) , there exists a target distribution for which the convergence order of F-DDIM is at most 1.
The proof is postponed to Appendix § 13 . Higher-order discretization is a common approach to improving diffusion sampling with few steps. Theorem 4 , together with the experiments in § 4.1 , shows that good few-step sample quality can also be achieved by a first-order sampler: F-DDIM remains first-order convergent while attaining competitive or better FID than higher-order baselines at small NFE budgets. This highlights model evaluation placement as a design principle that complements improvements in convergence order. The distinction is particularly relevant to stochastic sampling, where the higher-order approximation in SDE-DPMSolver++2 does not improve its first-order strong convergence rate. These observations explain the forward-value design of F-DDPM in § 3.2 , which applies this principle while accounting for the injected noise.
For completeness, we also establish a first-order error upper bound for F-DDPM.
Theorem 5. Under Assumption 1, F-DDPM satisfies the first-order error bound in Definition 2.
The detailed proof of this result is postponed to Appendix § 14 . This guarantee complements the empirical results in § 4 tells that: F-DDPM improves sample quality at small NFE budgets while retaining a first-order strong convergence guarantee.
Experiments
In this section, we evaluate the forward-value samplers from two perspectives. First, we examine whether the first-order deterministic sampler F-DDIM achieves competitive sample quality at small NFEs. Second, we compare F-DDPM with stochastic baselines at matched NFEs and with deterministic samplers using 1.5 times their NFEs.
Models and datasets. We evaluate the samplers on CIFAR-10, ImageNet, LSUN, and FFHQ datasets. For CIFAR-10 ([33 ] ), we use unconditional and class-conditional EDM models ([29 ] ). For ImageNet ([14 ] ), we use EDM2-S and EDM2-L at resolution 64 × 64 64 \times64 64 × 64 , and EDM2-XS and EDM2-XXL at resolution 512 × 512 512 \times512 512 × 512 . For LSUN bedrooms ((Yu et al., 2015)) and FFHQ ([30 ] ), we use pre-trained latent diffusion models ([53 , 4 ] ) with noise predictors.
Evaluation settings. We measure sample quality using FID computed from 50K generated samples, with lower scores indicating better quality. In the FID tables, bold indicates the lowest FID in each row. Computational budgets are reported as the number of neural function evaluations (NFE). For LSUN and FFHQ, we compute the reference statistics ourselves; the resulting FID scores may therefore differ from those reported in other studies. The reference statistics and the code used to generate them are available at https://github.com/Na-Li66/F-DPMSolver .
For EDM and EDM2, we directly construct the sampling noise levels using their native ρ \rho ρ -parameterized schedule. For the latent diffusion models considered here, the noise predictor is trained at discrete noise levels specified by a reference schedule { ( α j r e f , σ j r e f ) } j = 0 M r e f \{(\alpha^{\mathrm{ref}}_j,\sigma^{\mathrm{ref}}_j)\}_{j=0}^{M^{\mathrm{ref}}} {( α j ref , σ j ref ) } j = 0 M ref . To perform sampling with M M M iterations, we select M + 1 M+1 M + 1 levels from this schedule according to
r i = round ( i M r e f M ) , r_i = \operatorname{round}\left(\frac{iM^{\mathrm{ref}}}{M}\right), r i = round ( M i M ref ) ,
α t i = α r i r e f , σ t i = σ r i r e f , i = 0 , … , M . \alpha_{t_i}=\alpha^{\mathrm{ref}}_{r_i}, \qquad\sigma_{t_i}=\sigma^{\mathrm{ref}}_{r_i}, \qquad i=0,\ldots,M. α t i = α r i ref , σ t i = σ r i ref , i = 0 , … , M .
Evaluation of F-DDIM
We first evaluate whether F-DDIM, as a first-order sampler, can achieve good few-step sample quality.
Experimental setup. We compare F-DDIM with deterministic DDIM, a second-order DPM Solver (DPMSolver-2, see (31)), a third-order DPM Solver (DPMSolver-3, see (32)), and a third-order predictor-corrector-based sampler (UniPC-3, see (42) and (41) ). All methods are evaluated at NFE budgets in { 4 , 5 , 6 , 8 , 10 } \{4,5,6,8,10\} { 4 , 5 , 6 , 8 , 10 } . The baseline update formulas are given in Appendix § 7 .§ 1 .
Quantitative results (FID). Table 1 and Table 2 , together with Table 7 and Table 8 in Appendix § 7 , report the FID results. F-DDIM substantially improves upon DDIM across the tested settings. On ImageNet-64 and ImageNet-512, it achieves the lowest FID among the compared methods at all tested NFEs and model sizes reported in Table 2 and Table 7 . On unconditional CIFAR-10, F-DDIM achieves the lowest FID at NFE budgets 4, 5, 6, and 8. On conditional CIFAR-10, it achieves the lowest FID at budgets 4, 5, and 6, while UniPC-3 performs better at budgets 8 and 10. These results demonstrate that F-DDIM, as a first-order sampler, can achieve competitive few-step sample quality relative to higher-order deterministic baselines.
Table 1. FID on unconditional and conditional CIFAR-10 with EDM.
Table 2. FID on ImageNet-512 with EDM2-XS/XXL.
Table 7. FID on ImageNet-64 with EDM2-S/L.
Table 8. FID on FFHQ and LSUN bedrooms with latent diffusion models.
Additional results, including generated samples, runtime comparisons, larger NFE budgets, and higher-order lookahead predictors, are provided in Appendix § 7 .
Evaluation of F-DDPM
Experimental setup. We compare F-DDPM with SDE-DPMSolver++1 (17a) and SDE-DPMSolver++2 (17b) at NFEs in { 6 , 9 , 12 , 15 } \{6, 9, 12, 15\} { 6 , 9 , 12 , 15 } . We also compare these stochastic samplers with DDIM, DPMSolver-3, and F-DDIM at NFEs in { 4 , 6 , 8 , 10 } \{4, 6, 8, 10\} { 4 , 6 , 8 , 10 } , respectively. Thus, each stochastic sampler uses 1.5 times the NFE of the deterministic methods in the corresponding table row. The noise schedules follow the subsampling procedure described above.
For the latent diffusion models considered here, model evaluations are restricted to supported reference noise levels. At iteration i i i , we select the noise level whose signal-to-noise ratio is closest to that of the lookahead time t i ′ t_i' t i ′ :
j i ∗ ∈ arg min j ∈ J i r e f ∣ α t i ′ σ t i ′ − α j r e f σ j r e f ∣ j_i^* \in\operatorname*{arg\,min}_{j\in\mathcal{J}_i^{\mathrm{ref}}} \left|\frac{\alpha_{t_i'}}{\sigma_{t_i'}}-\frac{\alpha_j^{\mathrm{ref}}}{\sigma_j^{\mathrm{ref}}}\right| j i ∗ ∈ j ∈ J i ref arg min σ t i ′ α t i ′ − σ j ref α j ref
where
J i r e f = { 0 ≤ j ≤ M r e f : α j r e f σ j r e f ≥ α t i ′ σ t i ′ } . \mathcal{J}_i^{\mathrm{ref}}=\left\{0\leq j\leq M^{\mathrm{ref}}:\frac{\alpha_j^{\mathrm{ref}}}{\sigma_j^{\mathrm{ref}}}\geq\frac{\alpha_{t_i'}}{\sigma_{t_i'}}\right\}. J i ref = { 0 ≤ j ≤ M ref : σ j ref α j ref ≥ σ t i ′ α t i ′ } .
The selected reference noise level is used for both the model evaluation and the corresponding F-DDPM update coefficients.
Comparison with stochastic baselines. Table 3 –Table 6 compare the stochastic samplers across various NFEs. F-DDPM achieves the lowest FID among the three stochastic methods in every tested setting. For example, on unconditional CIFAR-10 with 9 model evaluations, F-DDPM achieves an FID of 8.63, compared with 55.52 for SDE-DPMSolver++1 and 17.89 for SDE-DPMSolver++2. These improvements are consistent across datasets, model sizes, conditional and unconditional settings, and pixel-space and latent-space generation.
Table 3. FID on unconditional and conditional CIFAR-10.
Table 6. FID on LSUN bedrooms and FFHQ with latent diffusion models.
Comparison with deterministic baselines. The same tables compare the stochastic samplers using 1.5 B 1.5B 1.5 B model evaluations with deterministic samplers using B B B evaluations, where B ∈ { 4 , 6 , 8 , 10 } B\in\{4,6,8,10\} B ∈ { 4 , 6 , 8 , 10 } . Under this budget allocation, F-DDPM achieves lower FID than all three deterministic baselines on unconditional CIFAR-10 and ImageNet-512. It also outperforms all three on conditional CIFAR-10 at B = 4 , 6 , 8 B=4,6,8 B = 4 , 6 , 8 and on LSUN at B = 6 , 8 , 10 B=6,8,10 B = 6 , 8 , 10 . Deterministic samplers retain an advantage on ImageNet-64 and in several FFHQ settings. Overall, these comparisons show that F-DDPM narrows the empirical sampling-efficiency gap with deterministic methods, while still using a larger NFE budget.
Table 4. FID on ImageNet-64 with EDM2-S/L.
Table 5. FID on ImageNet-512 with EDM2-XS/XXL.
Qualitative comparisons. Figure 3 and Figure 4 show representative samples on ImageNet-64 and FFHQ. Additional examples on CIFAR-10, ImageNet-512, and LSUN are provided in Figure 13 , Figure 14 , Figure 15 , and Figure 16 in the appendix. Within each comparison, the stochastic samplers use the same initial noise and Brownian motion. At small NFEs, F-DDPM often produces clearer structures and finer visual details than the stochastic baselines, consistent with the FID comparisons. For example, the ImageNet-64 samples show more distinct object boundaries and background details.
Figure 3. Qualitative comparison of F-DDPM, SDE-DPMSolver++1, and SDE-DPMSolver++2 on ImageNet-64 using EDM2-S and EDM2-L. All methods use the NFE indicated above each column.
Figure 4. Qualitative comparison of F-DDPM, SDE-DPMSolver++1, and SDE-DPMSolver++2 on FFHQ using a pre-trained latent diffusion model. All methods use the NFE indicated above each column.
Figure 13. Qualitative comparisons between F-DDPM (ours) and SDE-DPMSolver++1, and SDE-DPMSolver++2. Images are sampled from pre-trained unconditional EDM on CIFAR10 dataset.
Figure 14. Qualitative comparisons between F-DDPM (ours) and SDE-DPMSolver++1, and SDE-DPMSolver++2. Images are sampled from pre-trained conditional EDM on CIFAR10 dataset.
Figure 15. Qualitative comparisons between F-DDPM (ours) and SDE-DPMSolver++1, and SDE-DPMSolver++2. Images are sampled from pre-trained EDM2 with XS and XXL size on ImageNet512 dataset.
Figure 16. Qualitative comparisons between F-DDPM (ours) and SDE-DPMSolver++1, and SDE-DPMSolver++2. Images are sampled from pre-trained latent diffusion model on LSUN bedroom dataset.
Discussion
We studied forward-value discretization as a training-free approach for accelerating diffusion sampling. Our analysis establishes first-order strong error bounds and matching worst-case lower bounds for SDE-DPMSolver++1 and SDE-DPMSolver++2. Thus, the higher-order approximation in SDE-DPMSolver++2 does not improve its strong convergence order. Together with the first-order characterization of F-DDIM and its competitive few-step sample quality, these results highlight model evaluation placement as a design principle that complements increasing the convergence order.
Building on the deterministic forward-value framework, we developed F-DDPM by accounting for noise injection in the choice of evaluation time and the construction of the lookahead estimate. The method requires one model evaluation per iteration and admits a first-order strong convergence guarantee. Across the tested settings, F-DDPM achieves lower FID than the stochastic baselines at the same NFE budgets. With 1.5 times the NFE of the deterministic baselines, it also achieves competitive sample quality in many settings, although deterministic methods retain an advantage in others. These results demonstrate the practical potential of forward-value evaluation for reducing the efficiency gap between stochastic and deterministic sampling.
Several questions remain open. First, the convergence guarantees do not explain why changing evaluation placement improves few-step sample quality. Second, the behavior of the method under classifier guidance and classifier-free guidance deserves further study, where guidance strength can amplify errors and alter the effective dynamics. Finally, applications to inverse problems, such as deblurring and super-resolution, require further investigation of how forward-value updates interact with measurement consistency constraints.
Acknowledgements
G. Li is supported in part by the Chinese University of Hong Kong Direct Grant for Research and the Hong Kong Research Grants Council ECS 24305724 and GRF 14307525. C. Cai is supported by the NSF CAREER Award CCF-2541600 and the NSF grant DMS-2515333. Y. Wei is supported in part by the NSF grants CCF-2106778, CCF-2418156 and CAREER award DMS-2143215, also by the NSF under Cooperative Agreement No. 2433450.
Additional sampler details and experimental results
Baseline deterministic samplers
We describe the deterministic baseline samplers used in § 4.1 .
Multistep DPMSolver updates. For the noise-prediction-based multistep solvers, we write ε t j = ε θ ( x t j , t j ) \varepsilon_{t_j} = \varepsilon_\theta(x_{t_j}, t_j) ε t j = ε θ ( x t j , t j ) . To construct a p p p -th order update, we approximate the noise predictor along the trajectory by a polynomial of degree at most p − 1 p-1 p − 1 , interpolating the previous p p p model evaluations. Integrating this polynomial in (12) gives (Lu et al., 2022a)
x t i = ODESolver p ( x t i − 1 , ε t i − 1 , ⋯ , ε t i − p ) : = α t i α t i − 1 x t i − 1 − α t i e − λ t i − 1 ∑ k = 0 p − 1 φ k ( δ i ) k ! y k + 1 , i ≥ p , (36) \begin{aligned}
x_{t_i} &= \operatorname{ODESolver}_p(x_{t_{i-1}}, \varepsilon_{t_{i-1}}, \cdots, \varepsilon_{t_{i-p}}) \\
&:= \frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}x_{t_{i-1}}-\alpha_{t_i}e^{-\lambda_{t_{i-1}}}\sum_{k=0}^{p-1}\frac{\varphi_k(\delta_i)}{k!}y_{k+1}, \qquad i \ge p, \tag*{(36)}
\end{aligned} x t i = ODESolver p ( x t i − 1 , ε t i − 1 , ⋯ , ε t i − p ) := α t i − 1 α t i x t i − 1 − α t i e − λ t i − 1 k = 0 ∑ p − 1 k ! φ k ( δ i ) y k + 1 , i ≥ p , (36)
where scalar functions φ k \varphi_k φ k are defined by
φ 0 ( x ) = 1 − e − x , \varphi_0(x)=1-e^{-x}, φ 0 ( x ) = 1 − e − x ,
φ k ( x ) = ∫ 0 x e − λ λ k d λ = k φ k − 1 ( x ) − x k e − x , k = 1 , ⋯ , p − 1. \varphi_k(x)=\int_0^x e^{-\lambda}\lambda^k\,\mathrm{d}\lambda=k\varphi_{k-1}(x)-x^ke^{-x}, \qquad k=1,\cdots,p-1. φ k ( x ) = ∫ 0 x e − λ λ k d λ = k φ k − 1 ( x ) − x k e − x , k = 1 , ⋯ , p − 1.
The vectors y k + 1 y_{k+1} y k + 1 define the interpolating polynomial
P i ( λ ) = ∑ k = 0 p − 1 ( λ − λ t i − 1 ) k k ! y k + 1 , P_i(\lambda)=\sum_{k=0}^{p-1}\frac{(\lambda-\lambda_{t_{i-1}})^k}{k!}y_{k+1}, P i ( λ ) = k = 0 ∑ p − 1 k ! ( λ − λ t i − 1 ) k y k + 1 ,
which satisfies P i ( λ t i − j ) = ε t i − j P_i(\lambda_{t_{i-j}})=\varepsilon_{t_{i-j}} P i ( λ t i − j ) = ε t i − j for j = 1 , … , p j=1,\ldots,p j = 1 , … , p . Equivalently, the matrix Y = [ y 1 , … , y p ] ⊤ ∈ R p × d Y=[y_1,\ldots,y_p]^\top\in\mathbb{R}^{p\times d} Y = [ y 1 , … , y p ] ⊤ ∈ R p × d solves
A Y = B , A = [ ( λ t i − j − λ t i − 1 ) k − 1 ( k − 1 ) ! ] 1 ≤ j , k ≤ p ∈ R p × p , B = [ ε t i − j ] 1 ≤ j ≤ p ∈ R p × d . AY=B,\qquad A=\left[\frac{(\lambda_{t_{i-j}}-\lambda_{t_{i-1}})^{k-1}}{(k-1)!}\right]_{1\le j,k\le p}\in\mathbb{R}^{p\times p},\qquad B=[\varepsilon_{t_{i-j}}]_{1\le j\le p}\in\mathbb{R}^{p\times d}. A Y = B , A = [ ( k − 1 )! ( λ t i − j − λ t i − 1 ) k − 1 ] 1 ≤ j , k ≤ p ∈ R p × p , B = [ ε t i − j ] 1 ≤ j ≤ p ∈ R p × d .
The cases p = 2 p=2 p = 2 and p = 3 p=3 p = 3 give the following DPMSolver-2 and DPMSolver-3 updates.
DPMSolver-2. The first state x t 1 x_{t_1} x t 1 is initialized using one DDIM step (14) . For i = 2 , … , M i=2,\ldots,M i = 2 , … , M , the update is
x t i = ODESolver 2 ( x t i − 1 , ε t i − 1 , ε t i − 2 ) : = α t i α t i − 1 x t i − 1 + α t i [ ( e − λ t i − e − λ t i − 1 ) ε t i − 1 + ϕ 1 , i D t i ] , (37) \begin{aligned}
x_{t_i}&=\operatorname{ODESolver}_2(x_{t_{i-1}},\varepsilon_{t_{i-1}},\varepsilon_{t_{i-2}})\\
&:=\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}x_{t_{i-1}}+\alpha_{t_i}\left[(e^{-\lambda_{t_i}}-e^{-\lambda_{t_{i-1}}})\varepsilon_{t_{i-1}}+\phi_{1,i}D_{t_i}\right], \tag*{(37)}
\end{aligned} x t i = ODESolver 2 ( x t i − 1 , ε t i − 1 , ε t i − 2 ) := α t i − 1 α t i x t i − 1 + α t i [ ( e − λ t i − e − λ t i − 1 ) ε t i − 1 + ϕ 1 , i D t i ] , (37)
where
D t i : = ε t i − 1 − ε t i − 2 δ i − 1 , ϕ 1 , i : = ( δ i + 1 ) e − λ t i − e − λ t i − 1 , i ≥ 2. (38) D_{t_i}:=\frac{\varepsilon_{t_{i-1}}-\varepsilon_{t_{i-2}}}{\delta_{i-1}},\qquad\phi_{1,i}:=(\delta_i+1)e^{-\lambda_{t_i}}-e^{-\lambda_{t_{i-1}}},\qquad i\ge2. \tag*{(38)} D t i := δ i − 1 ε t i − 1 − ε t i − 2 , ϕ 1 , i := ( δ i + 1 ) e − λ t i − e − λ t i − 1 , i ≥ 2. (38)
DPMSolver-3. The first state x t 1 x_{t_1} x t 1 is initialized using one DDIM step (14) , and the second state x t 2 x_{t_2} x t 2 is computed using DPMSolver-2 (31) with ϵ t 0 \epsilon_{t_0} ϵ t 0 and ϵ t 1 \epsilon_{t_1} ϵ t 1 . Once three previous model evaluations are available, the update for i = 3 , … , M i=3,\ldots,M i = 3 , … , M is
x t i = ODESolver3 ( x t i − 1 , ϵ t i − 1 , ϵ t i − 2 , ϵ t i − 3 ) : = α t i α t i − 1 x t i − 1 + α t i [ ( e − λ t i − e − λ t i − 1 ) ϵ t i − 1 + ϕ 1 , i ( λ t i − 1 − λ t i − 3 ) D 1 , t i − ( λ t i − 1 − λ t i − 2 ) D 2 , t i λ t i − 2 − λ t i − 3 + ϕ 2 , i D 1 , t i − D 2 , t i λ t i − 2 − λ t i − 3 ] , (39) \begin{aligned}
x_{t_i} &= \operatorname{ODESolver3}(x_{t_{i-1}},\epsilon_{t_{i-1}},\epsilon_{t_{i-2}},\epsilon_{t_{i-3}}) \\
&:= \frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}x_{t_{i-1}}+\alpha_{t_i}\left[\left(e^{-\lambda_{t_i}}-e^{-\lambda_{t_{i-1}}}\right)\epsilon_{t_{i-1}}\right.\\
&\qquad\left.+\phi_{1,i}\frac{(\lambda_{t_{i-1}}-\lambda_{t_{i-3}})D_{1,t_i}-(\lambda_{t_{i-1}}-\lambda_{t_{i-2}})D_{2,t_i}}{\lambda_{t_{i-2}}-\lambda_{t_{i-3}}}+\phi_{2,i}\frac{D_{1,t_i}-D_{2,t_i}}{\lambda_{t_{i-2}}-\lambda_{t_{i-3}}}\right], \tag*{(39)}
\end{aligned} x t i = ODESolver3 ( x t i − 1 , ϵ t i − 1 , ϵ t i − 2 , ϵ t i − 3 ) := α t i − 1 α t i x t i − 1 + α t i [ ( e − λ t i − e − λ t i − 1 ) ϵ t i − 1 + ϕ 1 , i λ t i − 2 − λ t i − 3 ( λ t i − 1 − λ t i − 3 ) D 1 , t i − ( λ t i − 1 − λ t i − 2 ) D 2 , t i + ϕ 2 , i λ t i − 2 − λ t i − 3 D 1 , t i − D 2 , t i ] , (39)
where ϕ 1 , i \phi_{1,i} ϕ 1 , i is defined in (38) , and
D 1 , t i : = ϵ t i − 1 − ϵ t i − 2 λ t i − 1 − λ t i − 2 , D 2 , t i : = ϵ t i − 1 − ϵ t i − 3 λ t i − 1 − λ t i − 3 , ϕ 2 , i : = δ i 2 e − λ t i + 2 ϕ 1 , i , i ≥ 3. (40) D_{1,t_i}:=\frac{\epsilon_{t_{i-1}}-\epsilon_{t_{i-2}}}{\lambda_{t_{i-1}}-\lambda_{t_{i-2}}},\qquad
D_{2,t_i}:=\frac{\epsilon_{t_{i-1}}-\epsilon_{t_{i-3}}}{\lambda_{t_{i-1}}-\lambda_{t_{i-3}}},\qquad
\phi_{2,i}:=\delta_i^2e^{-\lambda_{t_i}}+2\phi_{1,i},\qquad i\ge3. \tag*{(40)} D 1 , t i := λ t i − 1 − λ t i − 2 ϵ t i − 1 − ϵ t i − 2 , D 2 , t i := λ t i − 1 − λ t i − 3 ϵ t i − 1 − ϵ t i − 3 , ϕ 2 , i := δ i 2 e − λ t i + 2 ϕ 1 , i , i ≥ 3. (40)
UniPC-3. UniPC-3 combines a predictor with a corrector (Zhao et al., [71 ] ). Let x t i x_{t_i} x t i denote the predicted state and x t i c o r x_{t_i}^{\mathrm{cor}} x t i cor the corrected state. The model is evaluated at the predicted state, ϵ t i = ϵ θ ( x t i , t i ) \epsilon_{t_i}=\epsilon_\theta(x_{t_i},t_i) ϵ t i = ϵ θ ( x t i , t i ) . The corrected initialization is x t 0 c o r = x t 0 x_{t_0}^{\mathrm{cor}}=x_{t_0} x t 0 cor = x t 0 . In the first step, x t 1 x_{t_1} x t 1 is computed using one DDIM step (14) from x t 0 x_{t_0} x t 0 , and x t 1 c o r x_{t_1}^{\mathrm{cor}} x t 1 cor is computed using DPMSolver-2 (31) with ϵ t 0 \epsilon_{t_0} ϵ t 0 and ϵ t 1 \epsilon_{t_1} ϵ t 1 . For i ≥ 2 i\ge2 i ≥ 2 , the corrected state x t i − 1 c o r x_{t_{i-1}}^{\mathrm{cor}} x t i − 1 cor is used to compute the next predicted state x t i x_{t_i} x t i using DPMSolver-2 (31) with ϵ t i − 1 \epsilon_{t_{i-1}} ϵ t i − 1 and ϵ t i − 2 \epsilon_{t_{i-2}} ϵ t i − 2 .
x t i = ODESolver2 ( x t i − 1 c o r , ϵ t i − 1 , ϵ t i − 2 ) , (41) x_{t_i}=\operatorname{ODESolver2}(x_{t_{i-1}}^{\mathrm{cor}},\epsilon_{t_{i-1}},\epsilon_{t_{i-2}}), \tag*{(41)} x t i = ODESolver2 ( x t i − 1 cor , ϵ t i − 1 , ϵ t i − 2 ) , (41)
where ODESolver2 is given in (31). We then evaluate ϵ t i = ϵ θ ( x t i , t i ) \epsilon_{t_i}=\epsilon_\theta(x_{t_i},t_i) ϵ t i = ϵ θ ( x t i , t i ) and use this output to correct the current interval:
x t i c o r = ODESolver3 ( x t i − 1 c o r , ϵ t i , ϵ t i − 1 , ϵ t i − 2 ) , (42) x_{t_i}^{\mathrm{cor}}=\operatorname{ODESolver3}(x_{t_{i-1}}^{\mathrm{cor}},\epsilon_{t_i},\epsilon_{t_{i-1}},\epsilon_{t_{i-2}}), \tag*{(42)} x t i cor = ODESolver3 ( x t i − 1 cor , ϵ t i , ϵ t i − 1 , ϵ t i − 2 ) , (42)
where ODESolver3 is defined in (32). At the final step, we return the predicted state x t M x_{t_M} x t M without evaluating the model at t M t_M t M or applying the corrector. Including the initial model evaluation at t 0 t_0 t 0 , the total NFE is M M M .
Additional results for F-DDIM
Quantitative results (FID). Table 7 and Table 8 supplement the results in § 4.1 . On ImageNet-64, F-DDIM achieves the lowest FID among the compared methods across all tested NFEs and model sizes. On FFHQ, it achieves the lowest FID at all tested NFEs except NFE = 4, where DPMSolver-3 performs slightly better. On LSUN bedrooms, F-DDIM performs best at NFE = 5, 6, while DPMSolver-3 performs best at NFE = 4 and DPMSolver-2 at NFE = 8, 10. Across these settings, F-DDIM consistently improves upon DDIM and remains competitive with higher-order methods.
Qualitative comparisons. Figure 5 –Figure 10 show additional samples on ImageNet-64, ImageNet-512, FFHQ, LSUN bedrooms, and CIFAR-10. The comparisons cover conditional and unconditional generation using both pixel-space and latent-space models. At the same NFE, F-DDIM often produces clearer structures and finer details than DDIM. In the examples shown, the higher-order solvers (DPMSolver-2/3 and UniPC-3) more often exhibit localized artifacts, particularly with few sampling steps.
Figure 5. Qualitative comparisons between F-DDIM (ours) and DDIM, DPMSolver-2, DPMSolver-3, and UniPC-3. Images are sampled from pre-trained EDM2 with S and L size on ImageNet64 dataset.
Figure 10. Qualitative comparisons between F-DDIM (ours) and DDIM, DPMSolver-2, DPMSolver-3, and UniPC-3. Images are sampled from pre-trained latent diffusion model on LSUN bedroom dataset.
Figure 6. Qualitative comparisons between F-DDIM (ours) and DDIM, DPMSolver-2, DPMSolver-3, and UniPC-3. Images are sampled from pre-trained EDM2 with XS and XXL size on ImageNet512 dataset.
Figure 7. Qualitative comparisons between F-DDIM (ours) and DDIM, DPMSolver-2, DPMSolver-3, and UniPC-3. Images are sampled from pre-trained unconditional EDM on CIFAR10 dataset.
Figure 8. Qualitative comparisons between F-DDIM (ours) and DDIM, DPMSolver-2, DPMSolver-3, and UniPC-3. Images are sampled from pre-trained conditional EDM on CIFAR10 dataset.
Figure 9. Qualitative comparisons between F-DDIM (ours) and DDIM, DPMSolver-2, DPMSolver-3, and UniPC-3. Images are sampled from pre-trained latent diffusion model on FFHQ dataset.
Higher-order lookahead predictors
Algorithm 3 summarizes the general F-DPMSolver framework using a data prediction model. We evaluate F-DPMSolver-2, which constructs the lookahead state using the DPMSolver-2 update with cached noise predictions:
x ^ t i = ODESolver2 ( x t i − 1 , ε ~ t i − 1 , ε ~ t i − 2 ) , i ≥ 2. \widehat{x}_{t_i} = \operatorname{ODESolver2}\left(x_{t_{i-1}}, \widetilde{\varepsilon}_{t_{i-1}}, \widetilde{\varepsilon}_{t_{i-2}}\right), \qquad i \ge2. x t i = ODESolver2 ( x t i − 1 , ε t i − 1 , ε t i − 2 ) , i ≥ 2.
The first lookahead state is constructed using (23) , with ε ~ t 0 = x t 0 \widetilde{\varepsilon}_{t_0}=x_{t_0} ε t 0 = x t 0 . The higher-order predictor changes only the lookahead construction; each iteration still requires one new model evaluation.
Algorithm 3 F-DPMSolver
Input: Data prediction model μ θ ( x , t ) \mu_\theta(x,t) μ θ ( x , t ) , time grid t 0 > ⋯ > t M t_0>\cdots>t_M t 0 > ⋯ > t M , noise schedule ( α t , σ t ) (\alpha_t,\sigma_t) ( α t , σ t ) , initialization x t 0 x_{t_0} x t 0 , and a lookahead predictor using cached noise estimates.
Initialize ε ~ t 0 ← x t 0 \widetilde{\varepsilon}_{t_0}\leftarrow x_{t_0} ε t 0 ← x t 0 .
for i = 1 , … , M i=1,\ldots,M i = 1 , … , M do
Construct x ^ t i \widehat{x}_{t_i} x t i from x t i − 1 x_{t_{i-1}} x t i − 1 and the cached estimates { ε ~ t j } j = 0 i − 1 \{\widetilde{\varepsilon}_{t_j}\}_{j=0}^{i-1} { ε t j } j = 0 i − 1 , without a new model evaluation.
Update
x t i ← σ t i − 1 σ t i x t i − 1 + ( α t i − σ t i α t i − 1 σ t i − 1 ) μ θ ( x ^ t i , t i ) . x_{t_i}\leftarrow\frac{\sigma_{t_{i-1}}}{\sigma_{t_i}}x_{t_{i-1}}+\left(\alpha_{t_i}-\frac{\sigma_{t_i}\alpha_{t_{i-1}}}{\sigma_{t_{i-1}}}\right)\mu_\theta(\widehat{x}_{t_i},t_i). x t i ← σ t i σ t i − 1 x t i − 1 + ( α t i − σ t i − 1 σ t i α t i − 1 ) μ θ ( x t i , t i ) . Set and cache ε ~ t i ← ( x ^ t i − α t i μ θ ( x ^ t i , t i ) ) / σ t i \widetilde{\varepsilon}_{t_i}\leftarrow(\widehat{x}_{t_i}-\alpha_{t_i}\mu_\theta(\widehat{x}_{t_i},t_i))/\sigma_{t_i} ε t i ← ( x t i − α t i μ θ ( x t i , t i )) / σ t i .
end for
Output: Generated sample x t M x_{t_M} x t M .
Quantitative results (FID). Tables 9 and 10 compare F-DDIM and F-DPMSolver-2 at the same NFE. F-DPMSolver-2 achieves lower FID in all reported CIFAR-10 settings and most ImageNet-512 settings. The exception is EDM2-XS at NFE = 10, where F-DDIM achieves an FID of 4.57, compared with 4.90 for F-DPMSolver-2. These results show that a higher-order lookahead predictor can further improve sample quality in most cases. The predictor order does not determine the convergence order of the forward-value sampler. Under the lookahead accuracy condition (181) , Theorem 7 in Appendix § 13 guarantees that the sampler F-DPMSolver-2 has the first-order convergence rate of the ideal forward-value sampler.
Table 9. FID comparison of F-DDIM and F-DPMSolver-2 on CIFAR-10.
Table 10. FID comparison of F-DDIM and F-DPMSolver-2 on ImageNet-512.
Qualitative comparisons. Figure 11 and Figure 12 compare the samples generated by F-DDIM and F-DPMSolver-2. The higher-order lookahead predictor can improve visual details at small NFEs, consistent with the FID comparisons.
Figure 11. Qualitative comparison of F-DDIM and F-DPMSolver-2 on CIFAR-10 using unconditional and conditional EDM models.
Figure 12. Qualitative comparison of F-DDIM and F-DPMSolver-2 on ImageNet-512 using EDM2-XS and EDM2-XXL.
Larger NFE budgets and runtime comparisons
We examine F-DDIM at larger NFE budgets on ImageNet-512 using EDM2-XXL and compare runtime on ImageNet-64 using EDM2-S.
Larger NFE results. Table 11 reports FID at NFEs 10, 12, 15, and 20. F-DDIM achieves the lowest reported FID at NFE=10, 12, and 15, and matches UniPC-3 at NFE=20. These results show that F-DDIM remains comparable to higher-order baselines over the tested range of larger NFEs.
Table 11. FID at larger NFE budgets on ImageNet-512 with EDM2-XXL.
Runtime comparison. Table 12 compares runtime on ImageNet-64 using EDM2-S. Let T A ( M ) T_A(M) T A ( M ) denote the runtime of sampler A A A with M M M model evaluations. We report the normalized runtime
Table 12. Normalized runtime on ImageNet-64 with EDM2-S. DDIM with NFE = 10 is assigned a value of 10.
R A ( M ) = 10 T A ( M ) T D D I M ( 10 ) . R_A(M) = 10 \frac{T_A(M)}{T_{\mathrm{DDIM}}(10)}. R A ( M ) = 10 T DDIM ( 10 ) T A ( M ) .
A table entry of the form M × c M \times c M × c denotes R A ( M ) = M c R_A(M) = Mc R A ( M ) = M c . Thus, c c c measures the average runtime per model evaluation relative to that of DDIM with NFE = 10. F-DDIM and F-DPMSolver-2 have runtimes similar to DDIM and DPMSolver-2 at the same NFE. This supports using NFE as a practical proxy for runtime when comparing these samplers under the tested setting.
Additional samples from F-DDPM
This subsection supplements the qualitative comparisons in § 4.2 . Figure 13 and Figure 14 show unconditional and class-conditional CIFAR-10 samples, while Figure 15 and Figure 16 show results on ImageNet-512 and LSUN bedrooms, respectively. Within each comparison, F-DDPM and the stochastic baselines use the same NFE, initial noise, and Brownian motion. The examples illustrate that F-DDPM often produces clearer structures and finer details at small NFEs, complementing the quantitative results in Table 3 –Table 6 .
Proof of deterministic samplers convergence lower bound (Theorem 1)
Let us consider the case where the target distribution is an isotropic Gaussian distribution q 0 = N ( 0 , γ 2 I d ) q_0 = \mathcal{N}(0, \gamma^2 I_d) q 0 = N ( 0 , γ 2 I d ) for some constant γ ≥ 0 \gamma\ge0 γ ≥ 0 .
Preliminaries. Before proceeding to the main proof, we first introduce some preliminary results that will be frequently used in the following analysis.
By the choice of the forward process (1) , the distribution q t q_t q t of x t x_t x t at any time t ≥ 0 t \ge0 t ≥ 0 satisfies
x t ∼ q t = N ( 0 , ( α t 2 γ 2 + σ t 2 ) I d ) , x_t \sim q_t = \mathcal{N}\left(0,(\alpha_t^2\gamma^2+\sigma_t^2)I_d\right), x t ∼ q t = N ( 0 , ( α t 2 γ 2 + σ t 2 ) I d ) ,
and its score function s t ⋆ ( ⋅ ) s_t^\star(\cdot) s t ⋆ ( ⋅ ) can be computed in closed form as
s t ⋆ ( x ) = ∇ x log q t ( x ) = ∇ x ( − ∥ x ∥ 2 2 ( α t 2 γ 2 + σ t 2 ) ) = − x α t 2 γ 2 + σ t 2 . s_t^\star(x)=\nabla_x\log q_t(x)=\nabla_x\left(-\frac{\lVert x\rVert^2}{2(\alpha_t^2\gamma^2+\sigma_t^2)}\right)=-\frac{x}{\alpha_t^2\gamma^2+\sigma_t^2}. s t ⋆ ( x ) = ∇ x log q t ( x ) = ∇ x ( − 2 ( α t 2 γ 2 + σ t 2 ) ∥ x ∥ 2 ) = − α t 2 γ 2 + σ t 2 x .
Combining this with the relationship between the noise predictor and score function ϵ θ ( x t , t ) = − σ t s t ⋆ ( x t ) \epsilon_\theta(x_t,t)=-\sigma_t s_t^\star(x_t) ϵ θ ( x t , t ) = − σ t s t ⋆ ( x t ) , we can derive the noise predictor at noise level t ( λ ) t(\lambda) t ( λ ) for any λ ∈ R \lambda\in\mathbb{R} λ ∈ R as
ϵ θ ( x t ( λ ) , t ( λ ) ) = − σ t ( λ ) s t ( λ ) ⋆ ( x t ( λ ) ) = σ t ( λ ) x t ( λ ) α t ( λ ) 2 γ 2 + σ t ( λ ) 2 = e − λ x t ( λ ) α t ( λ ) ( γ 2 + e − 2 λ ) . (43) \epsilon_\theta(x_{t(\lambda)},t(\lambda))=-\sigma_{t(\lambda)}s_{t(\lambda)}^\star(x_{t(\lambda)})=\frac{\sigma_{t(\lambda)}x_{t(\lambda)}}{\alpha_{t(\lambda)}^2\gamma^2+\sigma_{t(\lambda)}^2}=\frac{e^{-\lambda}x_{t(\lambda)}}{\alpha_{t(\lambda)}(\gamma^2+e^{-2\lambda})}. \tag*{(43)} ϵ θ ( x t ( λ ) , t ( λ )) = − σ t ( λ ) s t ( λ ) ⋆ ( x t ( λ ) ) = α t ( λ ) 2 γ 2 + σ t ( λ ) 2 σ t ( λ ) x t ( λ ) = α t ( λ ) ( γ 2 + e − 2 λ ) e − λ x t ( λ ) . (43)
where the last line holds due to the fact that σ t ( λ ) = α t ( λ ) e − λ \sigma_{t(\lambda)}=\alpha_{t(\lambda)}e^{-\lambda} σ t ( λ ) = α t ( λ ) e − λ .
We claim that in the isotropic Gaussian case, the exact solution x t ⋆ x_t^\star x t ⋆ of the diffusion ODE (12) remains colinear with the initial point x t 0 ⋆ x_{t_0}^\star x t 0 ⋆ . Specifically, for any λ ∈ [ λ t 0 , λ t M ] \lambda\in[\lambda_{t_0},\lambda_{t_M}] λ ∈ [ λ t 0 , λ t M ] , we have
x t ( λ ) ⋆ = κ t ( λ ) ⋆ x t 0 ⋆ with κ t ( λ ) ⋆ = α t ( λ ) α t 0 γ 2 + e − 2 λ γ 2 + e − 2 λ ( t 0 ) . (44) x_{t(\lambda)}^\star=\kappa_{t(\lambda)}^\star x_{t_0}^\star\quad\text{with}\quad\kappa_{t(\lambda)}^\star=\frac{\alpha_{t(\lambda)}}{\alpha_{t_0}}\sqrt{\frac{\gamma^2+e^{-2\lambda}}{\gamma^2+e^{-2\lambda(t_0)}}}. \tag*{(44)} x t ( λ ) ⋆ = κ t ( λ ) ⋆ x t 0 ⋆ with κ t ( λ ) ⋆ = α t 0 α t ( λ ) γ 2 + e − 2 λ ( t 0 ) γ 2 + e − 2 λ . (44)
To verify this, let us substitute the expression (44) into the right-hand-side of the diffusion ODE (12) . For arbitrary t , s ≥ 0 t,s \ge0 t , s ≥ 0 , straightforward calculation yields
α t α s x s ⋆ − α t ∫ λ ( s ) λ ( t ) e − λ ϵ θ ( x t ( λ ) ⋆ , t ( λ ) ) d λ = ( a ) α t α s x s ⋆ − ∫ λ ( s ) λ ( t ) α t α t ( λ ) e − 2 λ γ 2 + e − 2 λ x t ( λ ) ⋆ d λ = ( b ) x s ⋆ ( α t α s − ∫ λ ( s ) λ ( t ) α t α t ( λ ) e − 2 λ κ t ( λ ) ⋆ γ 2 + e − 2 λ κ s ⋆ d λ ) = ( c ) x s ⋆ ( α t α s − α t α s γ 2 + e − 2 λ ( s ) ∫ λ ( s ) λ ( t ) e − 2 λ γ 2 + e − 2 λ d λ ) = α t x s ⋆ α s ( 1 − 1 γ 2 + e − 2 λ ( s ) ( γ 2 + e − 2 λ ( t ) − γ 2 + e − 2 λ ( s ) ) ) = α t x s ⋆ α s γ 2 + e − 2 λ ( t ) γ 2 + e − 2 λ ( s ) = κ t ⋆ κ s ⋆ x s ⋆ = x t ⋆ , (45) \begin{aligned}
\frac{\alpha_t}{\alpha_s}x_s^\star-\alpha_t\int_{\lambda(s)}^{\lambda(t)}e^{-\lambda}\epsilon_\theta(x_{t(\lambda)}^\star,t(\lambda))\,\mathrm{d}\lambda
&\overset{(a)}{=}\frac{\alpha_t}{\alpha_s}x_s^\star-\int_{\lambda(s)}^{\lambda(t)}\frac{\alpha_t}{\alpha_{t(\lambda)}}\frac{e^{-2\lambda}}{\gamma^2+e^{-2\lambda}}x_{t(\lambda)}^\star\,\mathrm{d}\lambda\\
&\overset{(b)}{=}x_s^\star\left(\frac{\alpha_t}{\alpha_s}-\int_{\lambda(s)}^{\lambda(t)}\frac{\alpha_t}{\alpha_{t(\lambda)}}\frac{e^{-2\lambda}\kappa_{t(\lambda)}^\star}{\gamma^2+e^{-2\lambda}\kappa_s^\star}\,\mathrm{d}\lambda\right)\\
&\overset{(c)}{=}x_s^\star\left(\frac{\alpha_t}{\alpha_s}-\frac{\alpha_t}{\alpha_s\sqrt{\gamma^2+e^{-2\lambda(s)}}}\int_{\lambda(s)}^{\lambda(t)}\frac{e^{-2\lambda}}{\sqrt{\gamma^2+e^{-2\lambda}}}\,\mathrm{d}\lambda\right)\\
&=\frac{\alpha_t x_s^\star}{\alpha_s}\left(1-\frac{1}{\sqrt{\gamma^2+e^{-2\lambda(s)}}}\left(\sqrt{\gamma^2+e^{-2\lambda(t)}}-\sqrt{\gamma^2+e^{-2\lambda(s)}}\right)\right)\\
&=\frac{\alpha_t x_s^\star}{\alpha_s}\sqrt{\frac{\gamma^2+e^{-2\lambda(t)}}{\gamma^2+e^{-2\lambda(s)}}}
=\frac{\kappa_t^\star}{\kappa_s^\star}x_s^\star=x_t^\star, \tag*{(45)}
\end{aligned} α s α t x s ⋆ − α t ∫ λ ( s ) λ ( t ) e − λ ϵ θ ( x t ( λ ) ⋆ , t ( λ )) d λ = ( a ) α s α t x s ⋆ − ∫ λ ( s ) λ ( t ) α t ( λ ) α t γ 2 + e − 2 λ e − 2 λ x t ( λ ) ⋆ d λ = ( b ) x s ⋆ ( α s α t − ∫ λ ( s ) λ ( t ) α t ( λ ) α t γ 2 + e − 2 λ κ s ⋆ e − 2 λ κ t ( λ ) ⋆ d λ ) = ( c ) x s ⋆ ( α s α t − α s γ 2 + e − 2 λ ( s ) α t ∫ λ ( s ) λ ( t ) γ 2 + e − 2 λ e − 2 λ d λ ) = α s α t x s ⋆ ( 1 − γ 2 + e − 2 λ ( s ) 1 ( γ 2 + e − 2 λ ( t ) − γ 2 + e − 2 λ ( s ) ) ) = α s α t x s ⋆ γ 2 + e − 2 λ ( s ) γ 2 + e − 2 λ ( t ) = κ s ⋆ κ t ⋆ x s ⋆ = x t ⋆ , (45)
where (a) applies (43) ; (b) and (c) uses the expression of x t ( λ ) ⋆ x_{t(\lambda)}^\star x t ( λ ) ⋆ from (44) ; and the penultimate line holds because d d t γ 2 + e − 2 λ = − e − 2 λ γ 2 + e − 2 λ \frac{\mathrm{d}}{\mathrm{d}t}\sqrt{\gamma^2+e^{-2\lambda}}=-\frac{e^{-2\lambda}}{\sqrt{\gamma^2+e^{-2\lambda}}} d t d γ 2 + e − 2 λ = − γ 2 + e − 2 λ e − 2 λ . Setting s = t 0 s=t_0 s = t 0 establishes the claim.
Moreover, the calculation in (45) implies the following identity regarding κ t i ⋆ \kappa_{t_i}^\star κ t i ⋆ for any i ∈ [ M ] i \in[M] i ∈ [ M ] :
κ t i ⋆ = α t i α t i − 1 ( 1 − 1 γ 2 + e − 2 λ t i − 1 ∫ λ t i − 1 λ t i e − 2 λ γ 2 + e − 2 λ d λ ) κ t i − 1 ⋆ , (46) \kappa_{t_i}^\star=\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}\left(1-\frac{1}{\sqrt{\gamma^2+e^{-2\lambda_{t_{i-1}}}}}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\frac{e^{-2\lambda}}{\sqrt{\gamma^2+e^{-2\lambda}}}\,\mathrm{d}\lambda\right)\kappa_{t_{i-1}}^\star, \tag*{(46)} κ t i ⋆ = α t i − 1 α t i ( 1 − γ 2 + e − 2 λ t i − 1 1 ∫ λ t i − 1 λ t i γ 2 + e − 2 λ e − 2 λ d λ ) κ t i − 1 ⋆ , (46)
which will be frequently used in the following proof.
With the above preparation in hand, we shall establish the lower bound for deterministic DDIM (14) and second-order ODE solver (31) separately.
Lower bound for deterministic DDIM. In this paragraph, let x t i x_{t_i} x t i denote the iterates produced by the deterministic DDIM sampler.
As shown in (43) , the noise predictor ϵ θ ( x , t ) \epsilon_\theta(x,t) ϵ θ ( x , t ) is colinear with x x x . Together with the DDIM update rule (14) , this implies that the i i i -th iterate x t i x_{t_i} x t i is aligned with the initial point x t 0 x_{t_0} x t 0 for all i ∈ [ M ] i \in[M] i ∈ [ M ] . Therefore, we can write
x t i = κ t i x t 0 x_{t_i} = \kappa_{t_i}x_{t_0} x t i = κ t i x t 0
where κ t i ∈ R \kappa_{t_i} \in\mathbb{R} κ t i ∈ R denotes the scalar coefficient.
Substituting x t i = κ t i x t 0 x_{t_i} = \kappa_{t_i}x_{t_0} x t i = κ t i x t 0 and the expression (43) for ϵ θ \epsilon_\theta ϵ θ into the DDIM update (14) yields the following recurrence for κ t i \kappa_{t_i} κ t i :
κ t 0 = 1 , and κ t i = α t i α t i − 1 ( 1 − e − λ t i − 1 γ 2 + e − 2 λ t i − 1 ∫ λ t i − 1 λ t i e − λ d λ ) κ t i − 1 , i ≥ 1. (47) \kappa_{t_0} = 1,\quad\text{and}\quad\kappa_{t_i} = \frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}\left(1-\frac{e^{-\lambda_{t_{i-1}}}}{\gamma^2+e^{-2\lambda_{t_{i-1}}}}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-\lambda}\,\mathrm{d}\lambda\right)\kappa_{t_{i-1}},\quad i\geq1. \tag*{(47)} κ t 0 = 1 , and κ t i = α t i − 1 α t i ( 1 − γ 2 + e − 2 λ t i − 1 e − λ t i − 1 ∫ λ t i − 1 λ t i e − λ d λ ) κ t i − 1 , i ≥ 1. (47)
Comparing (47) with the corresponding identity for κ t i ⋆ \kappa_{t_i}^{\star} κ t i ⋆ in (46) , and using κ t 0 ⋆ = κ t 0 = 1 \kappa_{t_0}^{\star}=\kappa_{t_0}=1 κ t 0 ⋆ = κ t 0 = 1 , we obtain that for any 1 ≤ i ≤ M 1\leq i\leq M 1 ≤ i ≤ M ,
κ t i ⋆ − κ t i = a i ( κ t i − 1 ⋆ − κ t i − 1 ) + κ t i − 1 ⋆ α t i α t i − 1 γ 2 + e − 2 λ t i − 1 ∫ λ t i − 1 λ t i e − λ ( e − λ t i − 1 γ 2 + e − 2 λ t i − 1 − e − λ γ 2 + e − 2 λ ) d λ = a i ( κ t i − 1 ⋆ − κ t i − 1 ) + α t i α t 0 γ 2 + e − 2 λ t 0 ∫ λ t i − 1 λ t i e − λ ( e − λ t i − 1 γ 2 + e − 2 λ t i − 1 − e − λ γ 2 + e − 2 λ ) d λ , (48) \begin{aligned}
\kappa_{t_i}^{\star}-\kappa_{t_i}=a_i\left(\kappa_{t_{i-1}}^{\star}-\kappa_{t_{i-1}}\right)+\kappa_{t_{i-1}}^{\star}\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}\sqrt{\gamma^2+e^{-2\lambda_{t_{i-1}}}}}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-\lambda}\left(\frac{e^{-\lambda_{t_{i-1}}}}{\sqrt{\gamma^2+e^{-2\lambda_{t_{i-1}}}}}-\frac{e^{-\lambda}}{\sqrt{\gamma^2+e^{-2\lambda}}}\right)\,\mathrm{d}\lambda\\
&=a_i\left(\kappa_{t_{i-1}}^{\star}-\kappa_{t_{i-1}}\right)+\frac{\alpha_{t_i}}{\alpha_{t_0}\sqrt{\gamma^2+e^{-2\lambda_{t_0}}}}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-\lambda}\left(\frac{e^{-\lambda_{t_{i-1}}}}{\sqrt{\gamma^2+e^{-2\lambda_{t_{i-1}}}}}-\frac{e^{-\lambda}}{\sqrt{\gamma^2+e^{-2\lambda}}}\right)\,\mathrm{d}\lambda, \tag*{(48)}
\end{aligned} κ t i ⋆ − κ t i = a i ( κ t i − 1 ⋆ − κ t i − 1 ) + κ t i − 1 ⋆ α t i − 1 γ 2 + e − 2 λ t i − 1 α t i ∫ λ t i − 1 λ t i e − λ ( γ 2 + e − 2 λ t i − 1 e − λ t i − 1 − γ 2 + e − 2 λ e − λ ) d λ = a i ( κ t i − 1 ⋆ − κ t i − 1 ) + α t 0 γ 2 + e − 2 λ t 0 α t i ∫ λ t i − 1 λ t i e − λ ( γ 2 + e − 2 λ t i − 1 e − λ t i − 1 − γ 2 + e − 2 λ e − λ ) d λ , (48)
where the coefficient a i a_i a i is defined as
a i ≔ α t i α t i − 1 ( 1 − e − λ t i − 1 γ 2 + e − 2 λ t i − 1 ∫ λ t i − 1 λ t i e − λ d λ ) , 1 ≤ i ≤ M , (49) a_i\coloneqq\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}\left(1-\frac{e^{-\lambda_{t_{i-1}}}}{\gamma^2+e^{-2\lambda_{t_{i-1}}}}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-\lambda}\,\mathrm{d}\lambda\right),\quad1\leq i\leq M, \tag*{(49)} a i : = α t i − 1 α t i ( 1 − γ 2 + e − 2 λ t i − 1 e − λ t i − 1 ∫ λ t i − 1 λ t i e − λ d λ ) , 1 ≤ i ≤ M , (49)
and the second line follows from the expression of κ t i − 1 ⋆ \kappa_{t_{i-1}}^{\star} κ t i − 1 ⋆ from (44) .
Notice that for any γ > 0 \gamma>0 γ > 0 ,
e − λ t i − 1 γ 2 + e − 2 λ t i − 1 − e − λ γ 2 + e − 2 λ ≥ 0 , ∀ λ ∈ [ λ t i − 1 , λ t i ] . \frac{e^{-\lambda_{t_{i-1}}}}{\sqrt{\gamma^2+e^{-2\lambda_{t_{i-1}}}}}-\frac{e^{-\lambda}}{\sqrt{\gamma^2+e^{-2\lambda}}}\geq0,\quad\forall\lambda\in[\lambda_{t_{i-1}},\lambda_{t_i}]. γ 2 + e − 2 λ t i − 1 e − λ t i − 1 − γ 2 + e − 2 λ e − λ ≥ 0 , ∀ λ ∈ [ λ t i − 1 , λ t i ] .
Plugging this into (48) leads to κ t i ⋆ ≥ κ t i \kappa_{t_i}^{\star}\geq\kappa_{t_i} κ t i ⋆ ≥ κ t i for all 0 ≤ i ≤ M 0\leq i\leq M 0 ≤ i ≤ M . Thus, the difference between the diffusion ODE and the deterministic DDIM trajectories at step M M M satisfies
∥ x t M − x t M ⋆ ∥ = ( κ t M ⋆ − κ t M ) ∥ x t 0 ∥ . \left\|x_{t_M}-x_{t_M}^{\star}\right\|=\left(\kappa_{t_M}^{\star}-\kappa_{t_M}\right)\left\|x_{t_0}\right\|. x t M − x t M ⋆ = ( κ t M ⋆ − κ t M ) ∥ x t 0 ∥ .
Therefore, to establish that the deterministic DDIM has convergence order at most one, it is sufficient to show that κ t M ⋆ − κ t M = Ω ( 1 / M ) \kappa_{t_M}^{\star}-\kappa_{t_M}=\Omega(1/M) κ t M ⋆ − κ t M = Ω ( 1/ M ) .
To this end, note that the integral on the right-hand-side of (48) can be lower bounded by
∫ λ t i − 1 λ t i e − λ ( e − λ t i − 1 γ 2 + e − 2 λ t i − 1 − e − λ γ 2 + e − 2 λ ) d λ = ∫ λ t i − 1 λ t i e − λ ∫ λ t i − 1 λ ′ − d d λ e − λ γ 2 + e − 2 λ d λ d λ ′ = ∫ λ t i − 1 λ t i ( e − λ − e − λ t i ) γ 2 e 2 λ ( γ 2 e 2 λ + 1 ) 3 2 d λ \begin{aligned}
\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-\lambda}\left(\frac{e^{-\lambda_{t_{i-1}}}}{\sqrt{\gamma^2+e^{-2\lambda_{t_{i-1}}}}}-\frac{e^{-\lambda}}{\sqrt{\gamma^2+e^{-2\lambda}}}\right)\,\mathrm{d}\lambda\\
&=\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-\lambda}\int_{\lambda_{t_{i-1}}}^{\lambda'}-\frac{\mathrm{d}}{\mathrm{d}\lambda}\frac{e^{-\lambda}}{\sqrt{\gamma^2+e^{-2\lambda}}}\,\mathrm{d}\lambda\,\mathrm{d}\lambda' \\
&= \int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}} \left(e^{-\lambda}-e^{-\lambda_{t_i}}\right)\frac{\gamma^2 e^{2\lambda}}{\left(\gamma^2 e^{2\lambda}+1\right)^{\frac{3}{2}}}\,\mathrm{d}\lambda
\end{aligned} ∫ λ t i − 1 λ t i e − λ ( γ 2 + e − 2 λ t i − 1 e − λ t i − 1 − γ 2 + e − 2 λ e − λ ) d λ = ∫ λ t i − 1 λ t i e − λ ∫ λ t i − 1 λ ′ − d λ d γ 2 + e − 2 λ e − λ d λ d λ ′ = ∫ λ t i − 1 λ t i ( e − λ − e − λ t i ) ( γ 2 e 2 λ + 1 ) 2 3 γ 2 e 2 λ d λ
≥ ( a ) γ 2 δ i 2 e − λ t i e 2 λ t i − 1 2 ( γ 2 e 2 λ t i + 1 ) 3 2 ≥ ( b ) γ 2 δ i 2 e λ t i 4 ( γ 2 e 2 λ t i + 1 ) 3 2 (50) \mathrel{\overset{\mathrm{(a)}}{\ge}} \frac{\gamma^2\delta_i^2 e^{-\lambda_{t_i}}e^{2\lambda_{t_{i-1}}}}{2\left(\gamma^2e^{2\lambda_{t_i}}+1\right)^{\frac{3}{2}}}
\mathrel{\overset{\mathrm{(b)}}{\ge}} \frac{\gamma^2\delta_i^2e^{\lambda_{t_i}}}{4\left(\gamma^2e^{2\lambda_{t_i}}+1\right)^{\frac{3}{2}}} \tag*{(50)} ≥ ( a ) 2 ( γ 2 e 2 λ t i + 1 ) 2 3 γ 2 δ i 2 e − λ t i e 2 λ t i − 1 ≥ ( b ) 4 ( γ 2 e 2 λ t i + 1 ) 2 3 γ 2 δ i 2 e λ t i (50)
where (a) holds because λ t i \lambda_{t_i} λ t i is increasing in i i i and thus
∫ λ t i − 1 λ t i ( e − λ − e − λ t i ) γ 2 e 2 λ ( γ 2 e 2 λ + 1 ) 3 2 d λ ≥ γ 2 e 2 λ t i − 1 ( γ 2 e 2 λ t i + 1 ) 3 2 ∫ λ t i − 1 λ t i ( e − λ − e − λ t i ) d λ = γ 2 e 2 λ t i − 1 e − λ t i ( γ 2 e 2 λ t i + 1 ) 3 2 ( e δ i − 1 − δ i ) ≥ γ 2 e 2 λ t i − 1 e − λ t i δ i 2 2 ( γ 2 e 2 λ t i + 1 ) 3 2 . \begin{aligned}
\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\left(e^{-\lambda}-e^{-\lambda_{t_i}}\right)\frac{\gamma^2e^{2\lambda}}{\left(\gamma^2e^{2\lambda}+1\right)^{\frac{3}{2}}}\,\mathrm{d}\lambda
\ge\frac{\gamma^2e^{2\lambda_{t_{i-1}}}}{\left(\gamma^2e^{2\lambda_{t_i}}+1\right)^{\frac{3}{2}}}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\left(e^{-\lambda}-e^{-\lambda_{t_i}}\right)\,\mathrm{d}\lambda\\
&= \frac{\gamma^2e^{2\lambda_{t_{i-1}}}e^{-\lambda_{t_i}}}{\left(\gamma^2e^{2\lambda_{t_i}}+1\right)^{\frac{3}{2}}}\left(e^{\delta_i}-1-\delta_i\right)
\ge\frac{\gamma^2e^{2\lambda_{t_{i-1}}}e^{-\lambda_{t_i}}\delta_i^2}{2\left(\gamma^2e^{2\lambda_{t_i}}+1\right)^{\frac{3}{2}}}.
\end{aligned} ∫ λ t i − 1 λ t i ( e − λ − e − λ t i ) ( γ 2 e 2 λ + 1 ) 2 3 γ 2 e 2 λ d λ ≥ ( γ 2 e 2 λ t i + 1 ) 2 3 γ 2 e 2 λ t i − 1 ∫ λ t i − 1 λ t i ( e − λ − e − λ t i ) d λ = ( γ 2 e 2 λ t i + 1 ) 2 3 γ 2 e 2 λ t i − 1 e − λ t i ( e δ i − 1 − δ i ) ≥ 2 ( γ 2 e 2 λ t i + 1 ) 2 3 γ 2 e 2 λ t i − 1 e − λ t i δ i 2 .
and (b) is true because e 2 λ t i − 1 − λ t i = e λ t i e − 2 δ i ≥ e λ t i / 2 e^{2\lambda_{t_{i-1}}-\lambda_{t_i}}=e^{\lambda_{t_i}}e^{-2\delta_i}\ge e^{\lambda_{t_i}}/2 e 2 λ t i − 1 − λ t i = e λ t i e − 2 δ i ≥ e λ t i /2 provided that δ i = O ( 1 / M ) ≤ log ( 2 ) / 2 \delta_i=O(1/M)\le\log(2)/2 δ i = O ( 1/ M ) ≤ log ( 2 ) /2 . Substituting (50) into (48) , we obtain
κ t i ⋆ − κ t i ≥ a i ( κ t i − 1 ⋆ − κ t i − 1 ) + 1 α t 0 γ 2 + e − 2 λ t 0 γ 2 δ i 2 α t i e λ t i 4 ( γ 2 e 2 λ t i + 1 ) 3 2 , ∀ 1 ≤ i ≤ M . (51) \kappa_{t_i}^{\star}-\kappa_{t_i}\ge a_i\left(\kappa_{t_{i-1}}^{\star}-\kappa_{t_{i-1}}\right)+\frac{1}{\alpha_{t_0}\sqrt{\gamma^2+e^{-2\lambda_{t_0}}}}\frac{\gamma^2\delta_i^2\alpha_{t_i}e^{\lambda_{t_i}}}{4\left(\gamma^2e^{2\lambda_{t_i}}+1\right)^{\frac{3}{2}}},\quad\forall\,1\le i\le M. \tag*{(51)} κ t i ⋆ − κ t i ≥ a i ( κ t i − 1 ⋆ − κ t i − 1 ) + α t 0 γ 2 + e − 2 λ t 0 1 4 ( γ 2 e 2 λ t i + 1 ) 2 3 γ 2 δ i 2 α t i e λ t i , ∀ 1 ≤ i ≤ M . (51)
Applying (51) recursively allows us to bound the final error as
κ t M ⋆ − κ t M ≥ 1 α t 0 γ 2 + e − 2 λ t 0 ∑ i = 1 M ∏ j = i + 1 M a j γ 2 δ i 2 α t i e λ t i 4 ( γ 2 e 2 λ t i + 1 ) 3 2 . (52) \kappa_{t_M}^{\star}-\kappa_{t_M}\ge\frac{1}{\alpha_{t_0}\sqrt{\gamma^2+e^{-2\lambda_{t_0}}}}\sum_{i=1}^{M}\prod_{j=i+1}^{M}a_j\frac{\gamma^2\delta_i^2\alpha_{t_i}e^{\lambda_{t_i}}}{4\left(\gamma^2e^{2\lambda_{t_i}}+1\right)^{\frac{3}{2}}}. \tag*{(52)} κ t M ⋆ − κ t M ≥ α t 0 γ 2 + e − 2 λ t 0 1 i = 1 ∑ M j = i + 1 ∏ M a j 4 ( γ 2 e 2 λ t i + 1 ) 2 3 γ 2 δ i 2 α t i e λ t i . (52)
To control the right-hand-side of (52) , let us introduce a subset of steps:
S : = { 1 ≤ i ≤ M : c 1 ≤ e λ t i − 1 ≤ e λ t i ≤ c 2 } , (53) \mathcal{S}:=\left\{1\le i\le M:c_1\le e^{\lambda_{t_{i-1}}}\le e^{\lambda_{t_i}}\le c_2\right\}, \tag*{(53)} S := { 1 ≤ i ≤ M : c 1 ≤ e λ t i − 1 ≤ e λ t i ≤ c 2 } , (53)
where c 1 c_1 c 1 , c 2 c_2 c 2 are two positive constants satisfying e λ t 0 < c 1 < c 2 < e λ t M e^{\lambda_{t_0}}<c_1<c_2<e^{\lambda_{t_M}} e λ t 0 < c 1 < c 2 < e λ t M . For sufficiently large M M M ,
∑ i ∈ S δ i ≥ log c 2 − log c 1 − 2 max i δ i ≥ 1 2 ( log c 2 − log c 1 ) . \sum_{i\in\mathcal{S}}\delta_i\ge\log c_2-\log c_1-2\max_i\delta_i\ge\frac{1}{2}(\log c_2-\log c_1). i ∈ S ∑ δ i ≥ log c 2 − log c 1 − 2 i max δ i ≥ 2 1 ( log c 2 − log c 1 ) .
Note that e − λ t i ≤ e − λ e^{-\lambda_{t_i}}\le e^{-\lambda} e − λ t i ≤ e − λ and γ 2 + e − 2 λ t i ≥ ( γ 2 + e − 2 λ ) e − 2 δ i \gamma^2+e^{-2\lambda_{t_i}}\ge(\gamma^2+e^{-2\lambda})e^{-2\delta_i} γ 2 + e − 2 λ t i ≥ ( γ 2 + e − 2 λ ) e − 2 δ i for all λ ∈ [ λ t i − 1 , λ t i ] \lambda\in[\lambda_{t_{i-1}},\lambda_{t_i}] λ ∈ [ λ t i − 1 , λ t i ] . This together with the definition of a i a_i a i in (49) imply the following lower bound
a i ≥ α t i α t i − 1 ( 1 − ∫ λ t i − 1 λ t i e − λ t i γ 2 + e − 2 λ t i e − λ d λ ) ≥ α t i α t i − 1 ( 1 − e 2 δ i ∫ λ t i − 1 λ t i e − 2 λ γ 2 + e − 2 λ d λ ) ≥ α t i α t i − 1 ( 1 − 5 4 ∫ λ t i − 1 λ t i e − 2 λ γ 2 + e − 2 λ d λ ) \begin{aligned}
a_i\ge\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}\left(1-\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\frac{e^{-\lambda_{t_i}}}{\gamma^2+e^{-2\lambda_{t_i}}}e^{-\lambda}\,\mathrm{d}\lambda\right) \\
&\ge\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}\left(1-e^{2\delta_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\frac{e^{-2\lambda}}{\gamma^2+e^{-2\lambda}}\,\mathrm{d}\lambda\right)
\ge\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}\left(1-\frac{5}{4}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\frac{e^{-2\lambda}}{\gamma^2+e^{-2\lambda}}\,\mathrm{d}\lambda\right)
\end{aligned} a i ≥ α t i − 1 α t i ( 1 − ∫ λ t i − 1 λ t i γ 2 + e − 2 λ t i e − λ t i e − λ d λ ) ≥ α t i − 1 α t i ( 1 − e 2 δ i ∫ λ t i − 1 λ t i γ 2 + e − 2 λ e − 2 λ d λ ) ≥ α t i − 1 α t i ( 1 − 4 5 ∫ λ t i − 1 λ t i γ 2 + e − 2 λ e − 2 λ d λ )
provided that 2 δ i = O ( 1 / M ) ≤ log ( 5 / 4 ) 2\delta_i=O(1/M)\le\log(5/4) 2 δ i = O ( 1/ M ) ≤ log ( 5/4 ) . This further allows us to lower bound the cumulative product ∏ j = i + 1 M a j \prod_{j=i+1}^{M}a_j ∏ j = i + 1 M a j for any i ∈ S i\in\mathcal{S} i ∈ S :
∏ j = i + 1 M a j ≥ α t M α t i ∏ j = i + 1 M ( 1 − 5 4 ∫ λ t j − 1 λ t j e − 2 λ γ 2 + e − 2 λ d λ ) ≥ α t M α t i ( 1 − 5 4 ∑ j = i + 1 M ∫ λ t j − 1 λ t j e − 2 λ γ 2 + e − 2 λ d λ ) ≥ α t M α t i ( 1 − 5 4 ∑ e λ t j − 1 ≥ c 1 ∫ λ t j − 1 λ t j e − 2 λ γ 2 + e − 2 λ d λ ) ≥ ( a ) α t M α t i ( 1 − 5 8 log ( 1 + γ − 2 c 1 − 2 ) ) ≥ α t M 8 α t i , (54) \begin{aligned}
\prod_{j=i+1}^{M}a_j\ge\frac{\alpha_{t_M}}{\alpha_{t_i}}\prod_{j=i+1}^{M}\left(1-\frac{5}{4}\int_{\lambda_{t_{j-1}}}^{\lambda_{t_j}}\frac{e^{-2\lambda}}{\gamma^2+e^{-2\lambda}}\,\mathrm{d}\lambda\right) \\
&\ge\frac{\alpha_{t_M}}{\alpha_{t_i}}\left(1-\frac{5}{4}\sum_{j=i+1}^{M}\int_{\lambda_{t_{j-1}}}^{\lambda_{t_j}}\frac{e^{-2\lambda}}{\gamma^2+e^{-2\lambda}}\,\mathrm{d}\lambda\right) \\
&\ge\frac{\alpha_{t_M}}{\alpha_{t_i}}\left(1-\frac{5}{4}\sum_{e^{\lambda_{t_{j-1}}}\ge c_1}\int_{\lambda_{t_{j-1}}}^{\lambda_{t_j}}\frac{e^{-2\lambda}}{\gamma^2+e^{-2\lambda}}\,\mathrm{d}\lambda\right)\mathrel{\overset{(a)}{\ge}}\frac{\alpha_{t_M}}{\alpha_{t_i}}\left(1-\frac{5}{8}\log(1+\gamma^{-2}c_1^{-2})\right)\ge\frac{\alpha_{t_M}}{8\alpha_{t_i}}, \tag*{(54)}
\end{aligned} j = i + 1 ∏ M a j ≥ α t i α t M j = i + 1 ∏ M ( 1 − 4 5 ∫ λ t j − 1 λ t j γ 2 + e − 2 λ e − 2 λ d λ ) ≥ α t i α t M ( 1 − 4 5 j = i + 1 ∑ M ∫ λ t j − 1 λ t j γ 2 + e − 2 λ e − 2 λ d λ ) ≥ α t i α t M 1 − 4 5 e λ t j − 1 ≥ c 1 ∑ ∫ λ t j − 1 λ t j γ 2 + e − 2 λ e − 2 λ d λ ≥ ( a ) α t i α t M ( 1 − 8 5 log ( 1 + γ − 2 c 1 − 2 ) ) ≥ 8 α t i α t M , (54)
provided that log ( 1 + γ − 2 c 1 − 2 ) ≤ 7 / 5 \log(1+\gamma^{-2}c_1^{-2})\le7/5 log ( 1 + γ − 2 c 1 − 2 ) ≤ 7/5 , where (a) holds because
∑ e λ t j − 1 ≥ c 1 ∫ λ t j − 1 λ t j e − 2 λ γ 2 + e − 2 λ d λ ≤ ∫ log c 1 ∞ e − 2 λ γ 2 + e − 2 λ d λ = 1 2 log ( γ 2 + e − 2 λ ) ∣ ∞ log c 1 = 1 2 log ( 1 + γ − 2 c 1 − 2 ) . \sum_{e^{\lambda_{t_{j-1}}}\ge c_1}\int_{\lambda_{t_{j-1}}}^{\lambda_{t_j}}\frac{e^{-2\lambda}}{\gamma^2+e^{-2\lambda}}\,\mathrm{d}\lambda\le\int_{\log c_1}^{\infty}\frac{e^{-2\lambda}}{\gamma^2+e^{-2\lambda}}\,\mathrm{d}\lambda=\left.\frac{1}{2}\log(\gamma^2+e^{-2\lambda})\right|_{\infty}^{\log c_1}=\frac{1}{2}\log(1+\gamma^{-2}c_1^{-2}). e λ t j − 1 ≥ c 1 ∑ ∫ λ t j − 1 λ t j γ 2 + e − 2 λ e − 2 λ d λ ≤ ∫ l o g c 1 ∞ γ 2 + e − 2 λ e − 2 λ d λ = 2 1 log ( γ 2 + e − 2 λ ) ∞ l o g c 1 = 2 1 log ( 1 + γ − 2 c 1 − 2 ) .
To finish up, substituting (54) into (52) yields
κ t M ∗ − κ t M ≥ 1 α t 0 γ 2 + e − 2 λ t 0 ∑ i ∈ S ∏ j = i + 1 M a j γ 2 δ i 2 α t i e λ t i 4 ( γ 2 e 2 λ t i + 1 ) 3 / 2 \kappa^*_{t_M}-\kappa_{t_M}\ge\frac{1}{\alpha_{t_0}\sqrt{\gamma^2+e^{-2\lambda_{t_0}}}}\sum_{i\in S}\prod_{j=i+1}^{M}a_j\frac{\gamma^2\delta_i^2\alpha_{t_i}e^{\lambda_{t_i}}}{4(\gamma^2e^{2\lambda_{t_i}}+1)^{3/2}} κ t M ∗ − κ t M ≥ α t 0 γ 2 + e − 2 λ t 0 1 i ∈ S ∑ j = i + 1 ∏ M a j 4 ( γ 2 e 2 λ t i + 1 ) 3/2 γ 2 δ i 2 α t i e λ t i
≥ ( a ) α t M α t 0 γ 2 + e − 2 λ t 0 γ 2 c 1 32 ( γ 2 c 2 2 + 1 ) 3 / 2 ∑ i ∈ S δ i 2 \mathrel{\overset{(a)}{\ge}}\frac{\alpha_{t_M}}{\alpha_{t_0}\sqrt{\gamma^2+e^{-2\lambda_{t_0}}}}\frac{\gamma^2c_1}{32(\gamma^2c_2^2+1)^{3/2}}\sum_{i\in S}\delta_i^2 ≥ ( a ) α t 0 γ 2 + e − 2 λ t 0 α t M 32 ( γ 2 c 2 2 + 1 ) 3/2 γ 2 c 1 i ∈ S ∑ δ i 2
≥ ( b ) α t M α t 0 γ 2 + e − 2 λ t 0 γ 2 c 1 32 ( γ 2 c 2 2 + 1 ) 3 / 2 ( log c 2 − log c 1 ) 2 4 M = Ω ( M − 1 ) , (55) \mathrel{\overset{(b)}{\ge}}\frac{\alpha_{t_M}}{\alpha_{t_0}\sqrt{\gamma^2+e^{-2\lambda_{t_0}}}}\frac{\gamma^2c_1}{32(\gamma^2c_2^2+1)^{3/2}}\frac{(\log c_2-\log c_1)^2}{4M}=\Omega(M^{-1}), \tag*{(55)} ≥ ( b ) α t 0 γ 2 + e − 2 λ t 0 α t M 32 ( γ 2 c 2 2 + 1 ) 3/2 γ 2 c 1 4 M ( log c 2 − log c 1 ) 2 = Ω ( M − 1 ) , (55)
where (a) holds because e λ t i / ( γ 2 e 2 λ t i + 1 ) 3 / 2 ≥ c 1 / ( γ 2 c 2 2 + 1 ) 3 / 2 e^{\lambda_{t_i}}/(\gamma^2e^{2\lambda_{t_i}}+1)^{3/2}\ge c_1/(\gamma^2c_2^2+1)^{3/2} e λ t i / ( γ 2 e 2 λ t i + 1 ) 3/2 ≥ c 1 / ( γ 2 c 2 2 + 1 ) 3/2 , and (b) is true due to the Cauchy-Schwarz inequality ( ∑ i ∈ S δ i ) 2 ≤ ∣ S ∣ ∑ i ∈ S δ i 2 (\sum_{i\in S}\delta_i)^2\le|S|\sum_{i\in S}\delta_i^2 ( ∑ i ∈ S δ i ) 2 ≤ ∣ S ∣ ∑ i ∈ S δ i 2 and the fact that ∑ i ∈ S δ i ≥ 1 2 ( log c 2 − log c 1 ) \sum_{i\in S}\delta_i\ge\frac{1}{2}(\log c_2-\log c_1) ∑ i ∈ S δ i ≥ 2 1 ( log c 2 − log c 1 ) . This completes the proof for the lower bound of deterministic DDIM.
Lower bound for second-order ODE solver. In this paragraph, we use x t i x_{t_i} x t i to represent the iterates generated by the second-order ODE solver (31). Similar to the analysis for the deterministic DDIM, we know that ϵ θ ( x , t ) \epsilon_\theta(x,t) ϵ θ ( x , t ) is colinear with x x x . In particular, we can express the i i i -th iterate as
x t i = κ t i x t 0 , x_{t_i}=\kappa_{t_i}x_{t_0}, x t i = κ t i x t 0 ,
We initialize the method with one DDIM step, so κ t 0 = 1 \kappa_{t_0}=1 κ t 0 = 1 and κ t 1 \kappa_{t_1} κ t 1 is given by (47) with i = 1 i=1 i = 1 . In particular, (48) and smoothness give 0 ≤ κ t 1 ∗ − κ t 1 = O ( δ 1 2 ) 0\le\kappa^*_{t_1}-\kappa_{t_1}=O(\delta_1^2) 0 ≤ κ t 1 ∗ − κ t 1 = O ( δ 1 2 ) . For subsequent steps, the coefficient κ t i \kappa_{t_i} κ t i satisfies that for any i ≥ 2 i\ge2 i ≥ 2 ,
κ t i = α t i α t i − 1 κ t i − 1 − α t i ∫ λ t i − 1 λ t i e − λ d λ − e − λ t i − 1 κ t i − 1 α t i − 1 ( γ 2 + e − 2 λ t i − 1 ) − α t i ∫ λ t i − 1 λ t i ( λ − λ t i − 1 ) e − λ d λ e − λ t i − 1 κ t i − 1 α t i − 1 ( γ 2 + e − 2 λ t i − 1 ) − e − λ t i − 2 κ t i − 2 α t i − 2 ( γ 2 + e − 2 λ t i − 2 ) λ t i − 1 − λ t i − 2 . (56) \kappa_{t_i}=\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}\kappa_{t_{i-1}}-\alpha_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-\lambda}\,\mathrm{d}\lambda-\frac{e^{-\lambda_{t_{i-1}}}\kappa_{t_{i-1}}}{\alpha_{t_{i-1}}(\gamma^2+e^{-2\lambda_{t_{i-1}}})}
-\alpha_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}(\lambda-\lambda_{t_{i-1}})e^{-\lambda}\,\mathrm{d}\lambda\frac{\frac{e^{-\lambda_{t_{i-1}}}\kappa_{t_{i-1}}}{\alpha_{t_{i-1}}(\gamma^2+e^{-2\lambda_{t_{i-1}}})}-\frac{e^{-\lambda_{t_{i-2}}}\kappa_{t_{i-2}}}{\alpha_{t_{i-2}}(\gamma^2+e^{-2\lambda_{t_{i-2}}})}}{\lambda_{t_{i-1}}-\lambda_{t_{i-2}}}. \tag*{(56)} κ t i = α t i − 1 α t i κ t i − 1 − α t i ∫ λ t i − 1 λ t i e − λ d λ − α t i − 1 ( γ 2 + e − 2 λ t i − 1 ) e − λ t i − 1 κ t i − 1 − α t i ∫ λ t i − 1 λ t i ( λ − λ t i − 1 ) e − λ d λ λ t i − 1 − λ t i − 2 α t i − 1 ( γ 2 + e − 2 λ t i − 1 ) e − λ t i − 1 κ t i − 1 − α t i − 2 ( γ 2 + e − 2 λ t i − 2 ) e − λ t i − 2 κ t i − 2 . (56)
Combining this with the recurrence for κ ∗ \kappa^* κ ∗ given in (46) , we can decompose the difference κ t i ∗ − κ t i \kappa^*_{t_i}-\kappa_{t_i} κ t i ∗ − κ t i as follows for i ≥ 2 i\ge2 i ≥ 2 :
κ t i ∗ − κ t i = a i ( κ t i − 1 ∗ − κ t i − 1 ) + b i + c i , (57) \kappa^*_{t_i}-\kappa_{t_i}=a_i(\kappa^*_{t_{i-1}}-\kappa_{t_{i-1}})+b_i+c_i, \tag*{(57)} κ t i ∗ − κ t i = a i ( κ t i − 1 ∗ − κ t i − 1 ) + b i + c i , (57)
where a i a_i a i is define in (49) , and b i b_i b i and c i c_i c i are given by
b i : = − α t i ∫ λ t i − 1 λ t i ( λ − λ t i − 1 ) e − λ d λ ⋅ 1 λ t i − 1 − λ t i − 2 ( e − λ t i − 1 ( κ t i − 1 ∗ − κ t i − 1 ) α t i − 1 ( γ 2 + e − 2 λ t i − 1 ) − e − λ t i − 2 ( κ t i − 2 ∗ − κ t i − 2 ) α t i − 2 ( γ 2 + e − 2 λ t i − 2 ) ) , (58) b_i:=-\alpha_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}(\lambda-\lambda_{t_{i-1}})e^{-\lambda}\,\mathrm{d}\lambda\cdot\frac{1}{\lambda_{t_{i-1}}-\lambda_{t_{i-2}}}\left(\frac{e^{-\lambda_{t_{i-1}}}(\kappa^*_{t_{i-1}}-\kappa_{t_{i-1}})}{\alpha_{t_{i-1}}(\gamma^2+e^{-2\lambda_{t_{i-1}}})}-\frac{e^{-\lambda_{t_{i-2}}}(\kappa^*_{t_{i-2}}-\kappa_{t_{i-2}})}{\alpha_{t_{i-2}}(\gamma^2+e^{-2\lambda_{t_{i-2}}})}\right), \tag*{(58)} b i := − α t i ∫ λ t i − 1 λ t i ( λ − λ t i − 1 ) e − λ d λ ⋅ λ t i − 1 − λ t i − 2 1 ( α t i − 1 ( γ 2 + e − 2 λ t i − 1 ) e − λ t i − 1 ( κ t i − 1 ∗ − κ t i − 1 ) − α t i − 2 ( γ 2 + e − 2 λ t i − 2 ) e − λ t i − 2 ( κ t i − 2 ∗ − κ t i − 2 ) ) , (58)
c i : = − α t i ∫ λ t i − 1 λ t i ( e − 2 λ γ 2 + e − 2 λ κ t i − 1 ∗ α t i − 1 γ 2 + e − 2 λ t i − 1 − e − λ t i − 1 − λ κ t i − 1 ∗ α t i − 1 ( γ 2 + e − 2 λ t i − 1 ) − ( λ − λ t i − 1 ) e − λ 1 λ t i − 1 − λ t i − 2 ( e − λ t i − 1 − λ t i − 1 κ t i − 1 ∗ α t i − 1 ( γ 2 + e − 2 λ t i − 1 ) − e − λ t i − 2 κ t i − 2 ∗ α t i − 2 ( γ 2 + e − 2 λ t i − 2 ) ) ) d λ . (59) c_i:=-\alpha_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\left(\frac{e^{-2\lambda}}{\gamma^2+e^{-2\lambda}}\frac{\kappa^*_{t_{i-1}}}{\alpha_{t_{i-1}}\sqrt{\gamma^2+e^{-2\lambda_{t_{i-1}}}}}-\frac{e^{-\lambda_{t_{i-1}}-\lambda}\kappa^*_{t_{i-1}}}{\alpha_{t_{i-1}}(\gamma^2+e^{-2\lambda_{t_{i-1}}})}-(\lambda-\lambda_{t_{i-1}})e^{-\lambda}\frac{1}{\lambda_{t_{i-1}}-\lambda_{t_{i-2}}}\left(\frac{e^{-\lambda_{t_{i-1}}-\lambda_{t_{i-1}}}\kappa^*_{t_{i-1}}}{\alpha_{t_{i-1}}(\gamma^2+e^{-2\lambda_{t_{i-1}}})}-\frac{e^{-\lambda_{t_{i-2}}}\kappa^*_{t_{i-2}}}{\alpha_{t_{i-2}}(\gamma^2+e^{-2\lambda_{t_{i-2}}})}\right)\right)\,\mathrm{d}\lambda. \tag*{(59)} c i := − α t i ∫ λ t i − 1 λ t i ( γ 2 + e − 2 λ e − 2 λ α t i − 1 γ 2 + e − 2 λ t i − 1 κ t i − 1 ∗ − α t i − 1 ( γ 2 + e − 2 λ t i − 1 ) e − λ t i − 1 − λ κ t i − 1 ∗ − ( λ − λ t i − 1 ) e − λ λ t i − 1 − λ t i − 2 1 ( α t i − 1 ( γ 2 + e − 2 λ t i − 1 ) e − λ t i − 1 − λ t i − 1 κ t i − 1 ∗ − α t i − 2 ( γ 2 + e − 2 λ t i − 2 ) e − λ t i − 2 κ t i − 2 ∗ ) ) d λ . (59)
Applying recursion to (57) yields
κ t M ∗ − κ t M = ∏ j = 2 M a j ( κ t 1 ∗ − κ t 1 ) + ∑ i = 2 M ∏ j = i + 1 M a j ( b i + c i ) . (60) \begin{aligned}
\kappa^*_{t_M}-\kappa_{t_M}&=\prod_{j=2}^{M}a_j(\kappa^*_{t_1}-\kappa_{t_1})\\
&\quad+\sum_{i=2}^{M}\prod_{j=i+1}^{M}a_j(b_i+c_i). \tag*{(60)}
\end{aligned} κ t M ∗ − κ t M = j = 2 ∏ M a j ( κ t 1 ∗ − κ t 1 ) + i = 2 ∑ M j = i + 1 ∏ M a j ( b i + c i ) . (60)
In what follows, we shall analyze c i c_i c i , b i b_i b i , and a i a_i a i separately.
f ( λ ) : = e − λ γ 2 + e − 2 λ . f(\lambda):=\frac{\mathrm{e}^{-\lambda}}{\sqrt{\gamma^2+\mathrm{e}^{-2\lambda}}}. f ( λ ) := γ 2 + e − 2 λ e − λ .
Using the expression of κ t j ∗ \kappa^*_{t_j} κ t j ∗ in (44) , the following factors appearing in the definition of c i c_i c i can be rewritten in terms of f ( ⋅ ) f(\cdot) f ( ⋅ ) for any i ≥ 2 i\ge2 i ≥ 2 :
e − λ t j κ t j ∗ α t j ( γ 2 + e − 2 λ t j ) = 1 α t 0 γ 2 + e − 2 λ t 0 e − λ t j γ 2 + e − 2 λ t j = f ( λ t j ) α t 0 γ 2 + e − 2 λ t 0 , j = i , i − 1 , i − 2 , \frac{\mathrm{e}^{-\lambda_{t_j}}\kappa^*_{t_j}}{\alpha_{t_j}(\gamma^2+\mathrm{e}^{-2\lambda_{t_j}})}
=\frac{1}{\alpha_{t_0}\sqrt{\gamma^2+\mathrm{e}^{-2\lambda_{t_0}}}}\frac{\mathrm{e}^{-\lambda_{t_j}}}{\sqrt{\gamma^2+\mathrm{e}^{-2\lambda_{t_j}}}}
=\frac{f(\lambda_{t_j})}{\alpha_{t_0}\sqrt{\gamma^2+\mathrm{e}^{-2\lambda_{t_0}}}},
\qquad j=i,i-1,i-2, α t j ( γ 2 + e − 2 λ t j ) e − λ t j κ t j ∗ = α t 0 γ 2 + e − 2 λ t 0 1 γ 2 + e − 2 λ t j e − λ t j = α t 0 γ 2 + e − 2 λ t 0 f ( λ t j ) , j = i , i − 1 , i − 2 ,
e − 2 λ γ 2 + e − 2 λ κ t i − 1 ∗ α t i − 1 γ 2 + e − 2 λ t i − 1 = e − λ f ( λ ) α t 0 γ 2 + e − 2 λ t 0 . (61) \frac{\mathrm{e}^{-2\lambda}}{\sqrt{\gamma^2+\mathrm{e}^{-2\lambda}}}\frac{\kappa^*_{t_{i-1}}}{\alpha_{t_{i-1}}\sqrt{\gamma^2+\mathrm{e}^{-2\lambda_{t_{i-1}}}}}
=\frac{\mathrm{e}^{-\lambda}f(\lambda)}{\alpha_{t_0}\sqrt{\gamma^2+\mathrm{e}^{-2\lambda_{t_0}}}}. \tag*{(61)} γ 2 + e − 2 λ e − 2 λ α t i − 1 γ 2 + e − 2 λ t i − 1 κ t i − 1 ∗ = α t 0 γ 2 + e − 2 λ t 0 e − λ f ( λ ) . (61)
Consequently, the term c i c_i c i can be expressed as:
c i = − α t i α t 0 γ 2 + e − 2 λ t 0 ∫ λ t i − 1 λ t i e − λ ( f ( λ ) − f ( λ t i − 1 ) − ( λ − λ t i − 1 ) f ( λ t i − 1 ) − f ( λ t i − 2 ) λ t i − 1 − λ t i − 2 ) d λ . (62) c_i=-\frac{\alpha_{t_i}}{\alpha_{t_0}\sqrt{\gamma^2+\mathrm{e}^{-2\lambda_{t_0}}}}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\mathrm{e}^{-\lambda}\left(f(\lambda)-f(\lambda_{t_{i-1}})-(\lambda-\lambda_{t_{i-1}})\frac{f(\lambda_{t_{i-1}})-f(\lambda_{t_{i-2}})}{\lambda_{t_{i-1}}-\lambda_{t_{i-2}}}\right)\,\mathrm{d}\lambda. \tag*{(62)} c i = − α t 0 γ 2 + e − 2 λ t 0 α t i ∫ λ t i − 1 λ t i e − λ ( f ( λ ) − f ( λ t i − 1 ) − ( λ − λ t i − 1 ) λ t i − 1 − λ t i − 2 f ( λ t i − 1 ) − f ( λ t i − 2 ) ) d λ . (62)
The derivative of f ( λ ) f(\lambda) f ( λ ) can be computed as
f ′ ( λ ) = − γ 2 e 2 λ ( γ 2 e 2 λ + 1 ) 3 2 , f ′ ′ ( λ ) = − γ 2 e 2 λ ( 2 − γ 2 e 2 λ ) ( γ 2 e 2 λ + 1 ) 5 2 . (63) f'(\lambda)=-\frac{\gamma^2\mathrm{e}^{2\lambda}}{(\gamma^2\mathrm{e}^{2\lambda}+1)^{\frac{3}{2}}},
\qquad
f''(\lambda)=-\frac{\gamma^2\mathrm{e}^{2\lambda}(2-\gamma^2\mathrm{e}^{2\lambda})}{(\gamma^2\mathrm{e}^{2\lambda}+1)^{\frac{5}{2}}}. \tag*{(63)} f ′ ( λ ) = − ( γ 2 e 2 λ + 1 ) 2 3 γ 2 e 2 λ , f ′′ ( λ ) = − ( γ 2 e 2 λ + 1 ) 2 5 γ 2 e 2 λ ( 2 − γ 2 e 2 λ ) . (63)
By a second-order Taylor’s expansion and the fact that δ i − 1 = O ( M − 1 ) \delta_{i-1}=O(M^{-1}) δ i − 1 = O ( M − 1 ) and δ i = O ( M − 1 ) \delta_i=O(M^{-1}) δ i = O ( M − 1 ) , we obtain
f ( λ ) − f ( λ t i − 1 ) − ( λ − λ t i − 1 ) f ( λ t i − 1 ) − f ( λ t i − 2 ) λ t i − 1 − λ t i − 2 = (a) ( λ − λ t i − 1 ) ( f ′ ( λ t i − 1 ) − f ( λ t i − 1 ) − f ( λ t i − 2 ) λ t i − 1 − λ t i − 2 ) + 1 2 f ′ ′ ( λ t i − 1 ) ( λ − λ t i − 1 ) 2 + O ( M − 3 ) = (b) 1 2 f ′ ′ ( λ t i − 1 ) ( λ t i − 1 − λ t i − 2 ) ( λ − λ t i − 1 ) + 1 2 f ′ ′ ( λ t i − 1 ) ( λ − λ t i − 1 ) 2 + O ( M − 3 ) = 1 2 f ′ ′ ( λ t i − 1 ) ( λ − λ t i − 2 ) ( λ − λ t i − 1 ) + O ( M − 3 ) . \begin{aligned}
f(\lambda)-f(\lambda_{t_{i-1}})-(\lambda-\lambda_{t_{i-1}})\frac{f(\lambda_{t_{i-1}})-f(\lambda_{t_{i-2}})}{\lambda_{t_{i-1}}-\lambda_{t_{i-2}}}
&\overset{\text{(a)}}{=}(\lambda-\lambda_{t_{i-1}})\left(f'(\lambda_{t_{i-1}})-\frac{f(\lambda_{t_{i-1}})-f(\lambda_{t_{i-2}})}{\lambda_{t_{i-1}}-\lambda_{t_{i-2}}}\right)\\
&\quad+\frac{1}{2}f''(\lambda_{t_{i-1}})(\lambda-\lambda_{t_{i-1}})^2+O(M^{-3})\\
&\overset{\text{(b)}}{=}\frac{1}{2}f''(\lambda_{t_{i-1}})(\lambda_{t_{i-1}}-\lambda_{t_{i-2}})(\lambda-\lambda_{t_{i-1}})\\
&\quad+\frac{1}{2}f''(\lambda_{t_{i-1}})(\lambda-\lambda_{t_{i-1}})^2+O(M^{-3})\\
&=\frac{1}{2}f''(\lambda_{t_{i-1}})(\lambda-\lambda_{t_{i-2}})(\lambda-\lambda_{t_{i-1}})+O(M^{-3}).
\end{aligned} f ( λ ) − f ( λ t i − 1 ) − ( λ − λ t i − 1 ) λ t i − 1 − λ t i − 2 f ( λ t i − 1 ) − f ( λ t i − 2 ) = (a) ( λ − λ t i − 1 ) ( f ′ ( λ t i − 1 ) − λ t i − 1 − λ t i − 2 f ( λ t i − 1 ) − f ( λ t i − 2 ) ) + 2 1 f ′′ ( λ t i − 1 ) ( λ − λ t i − 1 ) 2 + O ( M − 3 ) = (b) 2 1 f ′′ ( λ t i − 1 ) ( λ t i − 1 − λ t i − 2 ) ( λ − λ t i − 1 ) + 2 1 f ′′ ( λ t i − 1 ) ( λ − λ t i − 1 ) 2 + O ( M − 3 ) = 2 1 f ′′ ( λ t i − 1 ) ( λ − λ t i − 2 ) ( λ − λ t i − 1 ) + O ( M − 3 ) .
where (a) holds because f ( λ ) − f ( λ t i − 1 ) = f ′ ( λ t i − 1 ) ( λ − λ t i − 1 ) + 1 2 f ′ ′ ( λ t i − 1 ) ( λ − λ t i − 1 ) 2 + O ( M − 3 ) f(\lambda)-f(\lambda_{t_{i-1}})=f'(\lambda_{t_{i-1}})(\lambda-\lambda_{t_{i-1}})+\frac{1}{2}f''(\lambda_{t_{i-1}})(\lambda-\lambda_{t_{i-1}})^2+O(M^{-3}) f ( λ ) − f ( λ t i − 1 ) = f ′ ( λ t i − 1 ) ( λ − λ t i − 1 ) + 2 1 f ′′ ( λ t i − 1 ) ( λ − λ t i − 1 ) 2 + O ( M − 3 ) , and (b) is true because f ( λ t i − 1 ) − f ( λ t i − 2 ) = f ′ ( λ t i − 1 ) ( λ t i − 1 − λ t i − 2 ) − 1 2 f ′ ′ ( λ t i − 1 ) ( λ t i − 1 − λ t i − 2 ) 2 + O ( M − 3 ) f(\lambda_{t_{i-1}})-f(\lambda_{t_{i-2}})=f'(\lambda_{t_{i-1}})(\lambda_{t_{i-1}}-\lambda_{t_{i-2}})-\frac{1}{2}f''(\lambda_{t_{i-1}})(\lambda_{t_{i-1}}-\lambda_{t_{i-2}})^2+O(M^{-3}) f ( λ t i − 1 ) − f ( λ t i − 2 ) = f ′ ( λ t i − 1 ) ( λ t i − 1 − λ t i − 2 ) − 2 1 f ′′ ( λ t i − 1 ) ( λ t i − 1 − λ t i − 2 ) 2 + O ( M − 3 ) . Substituting this into (62) results in the following bound for c i c_i c i :
c i ≡ (a) − α t i f ′ ′ ( λ t i − 1 ) 2 α t 0 γ 2 + e − 2 λ t 0 ∫ λ t i − 1 λ t i e − λ ( λ − λ t i − 2 ) ( λ − λ t i − 1 ) d λ + O ( M − 4 ) c_i\overset{\text{(a)}}{\equiv}-\frac{\alpha_{t_i}f''(\lambda_{t_{i-1}})}{2\alpha_{t_0}\sqrt{\gamma^2+\mathrm{e}^{-2\lambda_{t_0}}}}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\mathrm{e}^{-\lambda}(\lambda-\lambda_{t_{i-2}})(\lambda-\lambda_{t_{i-1}})\,\mathrm{d}\lambda+O(M^{-4}) c i ≡ (a) − 2 α t 0 γ 2 + e − 2 λ t 0 α t i f ′′ ( λ t i − 1 ) ∫ λ t i − 1 λ t i e − λ ( λ − λ t i − 2 ) ( λ − λ t i − 1 ) d λ + O ( M − 4 )
≡ (b) − α t i e − λ t i − 1 f ′ ′ ( λ t i − 1 ) 2 α t 0 γ 2 + e − 2 λ t 0 ∫ λ t i − 1 λ t i ( λ − λ t i − 2 ) ( λ − λ t i − 1 ) d λ + O ( M − 4 ) = − α t i e − λ t i − 1 f ′ ′ ( λ t i − 1 ) 2 α t 0 γ 2 + e − 2 λ t 0 1 6 δ i 2 ( 2 δ i + 3 δ i − 1 ) + O ( M − 4 ) = − α t i e − λ t i − 1 f ′ ′ ( λ t i − 1 ) 12 α t 0 γ 2 + e − 2 λ t 0 δ i 2 ( 2 δ i + 3 δ i − 1 ) + O ( M − 4 ) , (64) \begin{aligned}
\overset{\text{(b)}}{\equiv}-\frac{\alpha_{t_i}\mathrm{e}^{-\lambda_{t_{i-1}}}f''(\lambda_{t_{i-1}})}{2\alpha_{t_0}\sqrt{\gamma^2+\mathrm{e}^{-2\lambda_{t_0}}}}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}(\lambda-\lambda_{t_{i-2}})(\lambda-\lambda_{t_{i-1}})\,\mathrm{d}\lambda+O(M^{-4}) \\
&= -\frac{\alpha_{t_i}e^{-\lambda_{t_{i-1}}}f^{\prime\prime}(\lambda_{t_{i-1}})}{2\alpha_{t_0}\sqrt{\gamma^2+e^{-2\lambda_{t_0}}}}\frac{1}{6}\delta_i^2\left(2\delta_i+3\delta_{i-1}\right)+O(M^{-4})\\
= -\frac{\alpha_{t_i}e^{-\lambda_{t_{i-1}}}f^{\prime\prime}(\lambda_{t_{i-1}})}{12\alpha_{t_0}\sqrt{\gamma^2+e^{-2\lambda_{t_0}}}}\delta_i^2\left(2\delta_i+3\delta_{i-1}\right)+O(M^{-4}), \tag*{(64)}
\end{aligned} ≡ (b) − 2 α t 0 γ 2 + e − 2 λ t 0 α t i e − λ t i − 1 f ′′ ( λ t i − 1 ) ∫ λ t i − 1 λ t i ( λ − λ t i − 2 ) ( λ − λ t i − 1 ) d λ + O ( M − 4 ) = − 12 α t 0 γ 2 + e − 2 λ t 0 α t i e − λ t i − 1 f ′′ ( λ t i − 1 ) δ i 2 ( 2 δ i + 3 δ i − 1 ) + O ( M − 4 ) , = − 2 α t 0 γ 2 + e − 2 λ t 0 α t i e − λ t i − 1 f ′′ ( λ t i − 1 ) 6 1 δ i 2 ( 2 δ i + 3 δ i − 1 ) + O ( M − 4 ) (64)
where (a) holds because
∣ α t i α t 0 γ 2 + e − 2 λ t 0 ∫ λ t i − 1 λ t i e − λ O ( M − 3 ) d λ ∣ ≤ ∣ α t i e − λ t i − 1 α t 0 γ 2 + e − 2 λ t 0 ∣ δ i ⋅ O ( M − 3 ) = O ( M − 4 ) , \left|\frac{\alpha_{t_i}}{\alpha_{t_0}\sqrt{\gamma^2+e^{-2\lambda_{t_0}}}}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-\lambda}O(M^{-3})\,\mathrm{d}\lambda\right|
\leq\left|\frac{\alpha_{t_i}e^{-\lambda_{t_{i-1}}}}{\alpha_{t_0}\sqrt{\gamma^2+e^{-2\lambda_{t_0}}}}\right|\delta_i\cdot O(M^{-3})=O(M^{-4}), α t 0 γ 2 + e − 2 λ t 0 α t i ∫ λ t i − 1 λ t i e − λ O ( M − 3 ) d λ ≤ α t 0 γ 2 + e − 2 λ t 0 α t i e − λ t i − 1 δ i ⋅ O ( M − 3 ) = O ( M − 4 ) ,
and (b) is true because
∣ ∫ λ t i − 1 λ t i ( e − λ − e − λ t i − 1 ) ( λ − λ t i − 2 ) ( λ − λ t i − 1 ) d λ ∣ ≤ ( e − λ t i − 1 − e − λ t i ) ∫ λ t i − 1 λ t i ( λ − λ t i − 2 ) ( λ − λ t i − 1 ) d λ ≤ e − λ t i − 1 δ i 1 6 δ i 2 ( 2 δ i + 3 δ i − 1 ) = O ( M − 4 ) . \begin{aligned}
\left|\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}(e^{-\lambda}-e^{-\lambda_{t_{i-1}}})(\lambda-\lambda_{t_{i-2}})(\lambda-\lambda_{t_{i-1}})\,\mathrm{d}\lambda\right|
\leq(e^{-\lambda_{t_{i-1}}}-e^{-\lambda_{t_i}})\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}(\lambda-\lambda_{t_{i-2}})(\lambda-\lambda_{t_{i-1}})\,\mathrm{d}\lambda\\
&\leq e^{-\lambda_{t_{i-1}}}\delta_i\frac{1}{6}\delta_i^2\left(2\delta_i+3\delta_{i-1}\right)=O(M^{-4}).
\end{aligned} ∫ λ t i − 1 λ t i ( e − λ − e − λ t i − 1 ) ( λ − λ t i − 2 ) ( λ − λ t i − 1 ) d λ ≤ ( e − λ t i − 1 − e − λ t i ) ∫ λ t i − 1 λ t i ( λ − λ t i − 2 ) ( λ − λ t i − 1 ) d λ ≤ e − λ t i − 1 δ i 6 1 δ i 2 ( 2 δ i + 3 δ i − 1 ) = O ( M − 4 ) .
g t i : = e − λ t i γ 2 + e − 2 λ t i κ t i ⋆ − κ t i α t i . g_{t_i}:=\frac{e^{-\lambda_{t_i}}}{\gamma^2+e^{-2\lambda_{t_i}}}\frac{\kappa_{t_i}^{\star}-\kappa_{t_i}}{\alpha_{t_i}}. g t i := γ 2 + e − 2 λ t i e − λ t i α t i κ t i ⋆ − κ t i .
By definition, b i b_i b i can be expressed in terms of g t i − 1 g_{t_{i-1}} g t i − 1 and g t i − 2 g_{t_{i-2}} g t i − 2 as:
b i = − α t i ∫ λ t i − 1 λ t i ( λ − λ t i − 1 ) e − λ d λ g t i − 1 − g t i − 2 λ t i − 1 − λ t i − 2 . (65) b_i=-\alpha_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}(\lambda-\lambda_{t_{i-1}})e^{-\lambda}\,\mathrm{d}\lambda\frac{g_{t_{i-1}}-g_{t_{i-2}}}{\lambda_{t_{i-1}}-\lambda_{t_{i-2}}}. \tag*{(65)} b i = − α t i ∫ λ t i − 1 λ t i ( λ − λ t i − 1 ) e − λ d λ λ t i − 1 − λ t i − 2 g t i − 1 − g t i − 2 . (65)
Therefore it suffices to analyze the term g t i − 1 − g t i − 2 λ t i − 1 − λ t i − 2 \frac{g_{t_{i-1}}-g_{t_{i-2}}}{\lambda_{t_{i-1}}-\lambda_{t_{i-2}}} λ t i − 1 − λ t i − 2 g t i − 1 − g t i − 2 . Leveraging that κ t i ⋆ − κ t i = α t i γ 2 + e − 2 λ t i e − λ t i g t i \kappa_{t_i}^{\star}-\kappa_{t_i}=\alpha_{t_i}\frac{\gamma^2+e^{-2\lambda_{t_i}}}{e^{-\lambda_{t_i}}}g_{t_i} κ t i ⋆ − κ t i = α t i e − λ t i γ 2 + e − 2 λ t i g t i , recurrence for κ t i ⋆ − κ t i \kappa_{t_i}^{\star}-\kappa_{t_i} κ t i ⋆ − κ t i in (57) also establishes a recurrence for g t i g_{t_i} g t i as:
g t i = α t i − 1 e − λ t i ( γ 2 + e − 2 λ t i − 1 ) α t i ( γ 2 + e − 2 λ t i ) e − λ t i − 1 a i g t i − 1 − e − λ t i γ 2 + e − 2 λ t i ∫ λ t i − 1 λ t i ( λ − λ t i − 1 ) e − λ d λ g t i − 1 − g t i − 2 λ t i − 1 − λ t i − 2 + e − λ t i α t i ( γ 2 + e − 2 λ t i ) c i . (66) \begin{aligned}
g_{t_i}={}&\frac{\alpha_{t_{i-1}}e^{-\lambda_{t_i}}(\gamma^2+e^{-2\lambda_{t_{i-1}}})}{\alpha_{t_i}(\gamma^2+e^{-2\lambda_{t_i}})e^{-\lambda_{t_{i-1}}}}a_i g_{t_{i-1}}\\
&-\frac{e^{-\lambda_{t_i}}}{\gamma^2+e^{-2\lambda_{t_i}}}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}(\lambda-\lambda_{t_{i-1}})e^{-\lambda}\,\mathrm{d}\lambda\frac{g_{t_{i-1}}-g_{t_{i-2}}}{\lambda_{t_{i-1}}-\lambda_{t_{i-2}}}\\
&+\frac{e^{-\lambda_{t_i}}}{\alpha_{t_i}(\gamma^2+e^{-2\lambda_{t_i}})}c_i. \tag*{(66)}
\end{aligned} g t i = α t i ( γ 2 + e − 2 λ t i ) e − λ t i − 1 α t i − 1 e − λ t i ( γ 2 + e − 2 λ t i − 1 ) a i g t i − 1 − γ 2 + e − 2 λ t i e − λ t i ∫ λ t i − 1 λ t i ( λ − λ t i − 1 ) e − λ d λ λ t i − 1 − λ t i − 2 g t i − 1 − g t i − 2 + α t i ( γ 2 + e − 2 λ t i ) e − λ t i c i . (66)
We further have
g t i − g t i − 1 λ t i − λ t i − 1 = ( α t i − 1 e − λ t i ( γ 2 + e − 2 λ t i − 1 ) α t i ( γ 2 + e − 2 λ t i ) e − λ t i − 1 a i − 1 ) g t i − 1 δ i − e − λ t i δ i ( γ 2 + e − 2 λ t i ) ∫ λ t i − 1 λ t i ( λ − λ t i − 1 ) e − λ d λ g t i − 1 − g t i − 2 λ t i − 1 − λ t i − 2 + e − λ t i α t i ( γ 2 + e − 2 λ t i ) c i δ i . (67) \begin{aligned}
\frac{g_{t_i}-g_{t_{i-1}}}{\lambda_{t_i}-\lambda_{t_{i-1}}}={}&\left(\frac{\alpha_{t_{i-1}}e^{-\lambda_{t_i}}(\gamma^2+e^{-2\lambda_{t_{i-1}}})}{\alpha_{t_i}(\gamma^2+e^{-2\lambda_{t_i}})e^{-\lambda_{t_{i-1}}}}a_i-1\right)\frac{g_{t_{i-1}}}{\delta_i}\\
&-\frac{e^{-\lambda_{t_i}}}{\delta_i(\gamma^2+e^{-2\lambda_{t_i}})}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}(\lambda-\lambda_{t_{i-1}})e^{-\lambda}\,\mathrm{d}\lambda\frac{g_{t_{i-1}}-g_{t_{i-2}}}{\lambda_{t_{i-1}}-\lambda_{t_{i-2}}}\\
&+\frac{e^{-\lambda_{t_i}}}{\alpha_{t_i}(\gamma^2+e^{-2\lambda_{t_i}})}\frac{c_i}{\delta_i}. \tag*{(67)}
\end{aligned} λ t i − λ t i − 1 g t i − g t i − 1 = ( α t i ( γ 2 + e − 2 λ t i ) e − λ t i − 1 α t i − 1 e − λ t i ( γ 2 + e − 2 λ t i − 1 ) a i − 1 ) δ i g t i − 1 − δ i ( γ 2 + e − 2 λ t i ) e − λ t i ∫ λ t i − 1 λ t i ( λ − λ t i − 1 ) e − λ d λ λ t i − 1 − λ t i − 2 g t i − 1 − g t i − 2 + α t i ( γ 2 + e − 2 λ t i ) e − λ t i δ i c i . (67)
Let us control three quantities on the right-hand-side of (67) separately. Regarding the third term, identity (64) gives
e − λ t i α t i ( γ 2 + e − 2 λ t i ) ∣ c i ∣ δ i ≤ e − λ t i e − λ t i − 1 ∣ f ′ ′ ( λ t i − 1 ) ∣ 12 α t 0 γ 2 + e − 2 λ t 0 δ i ( 2 δ i + 3 δ i − 1 ) + O ( M − 3 ) = O ( M − 2 ) . (68) \frac{e^{-\lambda_{t_i}}}{\alpha_{t_i}(\gamma^2+e^{-2\lambda_{t_i}})}\frac{|c_i|}{\delta_i}
\leq\frac{e^{-\lambda_{t_i}}e^{-\lambda_{t_{i-1}}}|f^{\prime\prime}(\lambda_{t_{i-1}})|}{12\alpha_{t_0}\sqrt{\gamma^2+e^{-2\lambda_{t_0}}}}\delta_i\left(2\delta_i+3\delta_{i-1}\right)+O(M^{-3})=O(M^{-2}). \tag*{(68)} α t i ( γ 2 + e − 2 λ t i ) e − λ t i δ i ∣ c i ∣ ≤ 12 α t 0 γ 2 + e − 2 λ t 0 e − λ t i e − λ t i − 1 ∣ f ′′ ( λ t i − 1 ) ∣ δ i ( 2 δ i + 3 δ i − 1 ) + O ( M − 3 ) = O ( M − 2 ) . (68)
Regarding the second term, provided that δ i = O ( M − 1 ) ≤ log 2 \delta_i = O(M^{-1}) \le\log2 δ i = O ( M − 1 ) ≤ log 2 , the coefficient before g t i − 1 − g t i − 2 λ t i − 1 − λ t i − 2 \frac{g_{t_i-1}-g_{t_i-2}}{\lambda_{t_i-1}-\lambda_{t_i-2}} λ t i − 1 − λ t i − 2 g t i − 1 − g t i − 2 is bounded by:
e − λ t i δ i ( γ 2 + e − 2 λ t i ) ∫ λ t i − 1 λ t i ( λ − λ t i − 1 ) e − λ d λ ≤ e − λ t i − λ t i − 1 2 ( γ 2 + e − 2 λ t i ) δ i ≤ 1 2 e δ i δ i ≤ δ i . (69) \frac{e^{-\lambda_{t_i}}}{\delta_i(\gamma^2+e^{-2\lambda_{t_i}})}\int_{\lambda_{t_i-1}}^{\lambda_{t_i}}(\lambda-\lambda_{t_i-1})e^{-\lambda}\,\mathrm{d}\lambda\le\frac{e^{-\lambda_{t_i}-\lambda_{t_i-1}}}{2(\gamma^2+e^{-2\lambda_{t_i}})}\delta_i \le\frac{1}{2}e^{\delta_i}\delta_i \le\delta_i. \tag*{(69)} δ i ( γ 2 + e − 2 λ t i ) e − λ t i ∫ λ t i − 1 λ t i ( λ − λ t i − 1 ) e − λ d λ ≤ 2 ( γ 2 + e − 2 λ t i ) e − λ t i − λ t i − 1 δ i ≤ 2 1 e δ i δ i ≤ δ i . (69)
Moreover, by the definition of a i a_i a i in (49) , the first term is simplified as
1 δ i ( α t i − 1 e − λ t i ( γ 2 + e − 2 λ t i − 1 ) α t i ( γ 2 + e − 2 λ t i ) e − λ t i − 1 a i − 1 ) = 1 δ i ( e − λ t i ( γ 2 + e − 2 λ t i − 1 ) ( γ 2 + e − 2 λ t i ) e − λ t i − 1 − e − λ t i γ 2 + e − 2 λ t i ∫ λ t i − 1 λ t i e − λ d λ − 1 ) = 1 δ i ( e − λ t i − 1 ( 1 − e − δ i ) ( e − λ t i − 1 − λ t i − γ 2 ) ( γ 2 + e − 2 λ t i ) e − λ t i − 1 − e − λ t i γ 2 + e − 2 λ t i ∫ λ t i − 1 λ t i e − λ d λ ) = − 1 δ i γ 2 ( 1 − e − δ i ) γ 2 + e − 2 λ t i . (70) \begin{aligned}
\frac{1}{\delta_i}\left(\frac{\alpha_{t_i-1}e^{-\lambda_{t_i}}(\gamma^2+e^{-2\lambda_{t_i-1}})}{\alpha_{t_i}(\gamma^2+e^{-2\lambda_{t_i}})e^{-\lambda_{t_i-1}}}a_i-1\right) \\
&=\frac{1}{\delta_i}\left(\frac{e^{-\lambda_{t_i}}(\gamma^2+e^{-2\lambda_{t_i-1}})}{(\gamma^2+e^{-2\lambda_{t_i}})e^{-\lambda_{t_i-1}}}-\frac{e^{-\lambda_{t_i}}}{\gamma^2+e^{-2\lambda_{t_i}}}\int_{\lambda_{t_i-1}}^{\lambda_{t_i}}e^{-\lambda}\,\mathrm{d}\lambda-1\right) \\
&=\frac{1}{\delta_i}\left(\frac{e^{-\lambda_{t_i-1}}(1-e^{-\delta_i})(e^{-\lambda_{t_i-1}-\lambda_{t_i}}-\gamma^2)}{(\gamma^2+e^{-2\lambda_{t_i}})e^{-\lambda_{t_i-1}}}-\frac{e^{-\lambda_{t_i}}}{\gamma^2+e^{-2\lambda_{t_i}}}\int_{\lambda_{t_i-1}}^{\lambda_{t_i}}e^{-\lambda}\,\mathrm{d}\lambda\right) \\
&=-\frac{1}{\delta_i}\frac{\gamma^2(1-e^{-\delta_i})}{\gamma^2+e^{-2\lambda_{t_i}}}. \tag*{(70)}
\end{aligned} δ i 1 ( α t i ( γ 2 + e − 2 λ t i ) e − λ t i − 1 α t i − 1 e − λ t i ( γ 2 + e − 2 λ t i − 1 ) a i − 1 ) = δ i 1 ( ( γ 2 + e − 2 λ t i ) e − λ t i − 1 e − λ t i ( γ 2 + e − 2 λ t i − 1 ) − γ 2 + e − 2 λ t i e − λ t i ∫ λ t i − 1 λ t i e − λ d λ − 1 ) = δ i 1 ( ( γ 2 + e − 2 λ t i ) e − λ t i − 1 e − λ t i − 1 ( 1 − e − δ i ) ( e − λ t i − 1 − λ t i − γ 2 ) − γ 2 + e − 2 λ t i e − λ t i ∫ λ t i − 1 λ t i e − λ d λ ) = − δ i 1 γ 2 + e − 2 λ t i γ 2 ( 1 − e − δ i ) . (70)
where the second line holds because
e − λ t i ( γ 2 + e − 2 λ t i − 1 ) ( γ 2 + e − 2 λ t i ) e − λ t i − 1 − 1 = e − λ t i ( γ 2 + e − 2 λ t i − 1 ) − ( γ 2 + e − 2 λ t i ) e − λ t i − 1 ( γ 2 + e − 2 λ t i ) e − λ t i − 1 = γ 2 ( e − λ t i − e − λ t i − 1 ) + e − λ t i e − λ t i − 1 ( e − λ t i − 1 − e − λ t i ) ( γ 2 + e − 2 λ t i ) e − λ t i − 1 = ( e − λ t i − 1 − e − λ t i ) ( e − λ t i e − λ t i − 1 − γ 2 ) ( γ 2 + e − 2 λ t i ) e − λ t i − 1 = e − λ t i − 1 ( 1 − e − δ i ) ( e − λ t i e − λ t i − 1 − γ 2 ) ( γ 2 + e − 2 λ t i ) e − λ t i − 1 , (71) \begin{aligned}
\frac{e^{-\lambda_{t_i}}(\gamma^2+e^{-2\lambda_{t_i-1}})}{(\gamma^2+e^{-2\lambda_{t_i}})e^{-\lambda_{t_i-1}}}-1
=\frac{e^{-\lambda_{t_i}}(\gamma^2+e^{-2\lambda_{t_i-1}})-(\gamma^2+e^{-2\lambda_{t_i}})e^{-\lambda_{t_i-1}}}{(\gamma^2+e^{-2\lambda_{t_i}})e^{-\lambda_{t_i-1}}} \\
&=\frac{\gamma^2(e^{-\lambda_{t_i}}-e^{-\lambda_{t_i-1}})+e^{-\lambda_{t_i}}e^{-\lambda_{t_i-1}}(e^{-\lambda_{t_i-1}}-e^{-\lambda_{t_i}})}{(\gamma^2+e^{-2\lambda_{t_i}})e^{-\lambda_{t_i-1}}} \\
&=\frac{(e^{-\lambda_{t_i-1}}-e^{-\lambda_{t_i}})(e^{-\lambda_{t_i}}e^{-\lambda_{t_i-1}}-\gamma^2)}{(\gamma^2+e^{-2\lambda_{t_i}})e^{-\lambda_{t_i-1}}} \\
&=\frac{e^{-\lambda_{t_i-1}}(1-e^{-\delta_i})(e^{-\lambda_{t_i}}e^{-\lambda_{t_i-1}}-\gamma^2)}{(\gamma^2+e^{-2\lambda_{t_i}})e^{-\lambda_{t_i-1}}}, \tag*{(71)}
\end{aligned} ( γ 2 + e − 2 λ t i ) e − λ t i − 1 e − λ t i ( γ 2 + e − 2 λ t i − 1 ) − 1 = ( γ 2 + e − 2 λ t i ) e − λ t i − 1 e − λ t i ( γ 2 + e − 2 λ t i − 1 ) − ( γ 2 + e − 2 λ t i ) e − λ t i − 1 = ( γ 2 + e − 2 λ t i ) e − λ t i − 1 γ 2 ( e − λ t i − e − λ t i − 1 ) + e − λ t i e − λ t i − 1 ( e − λ t i − 1 − e − λ t i ) = ( γ 2 + e − 2 λ t i ) e − λ t i − 1 ( e − λ t i − 1 − e − λ t i ) ( e − λ t i e − λ t i − 1 − γ 2 ) = ( γ 2 + e − 2 λ t i ) e − λ t i − 1 e − λ t i − 1 ( 1 − e − δ i ) ( e − λ t i e − λ t i − 1 − γ 2 ) , (71)
and the last identity holds since
e − λ t i − 1 ( 1 − e − δ i ) ( e − λ t i − 1 − λ t i − γ 2 ) ( γ 2 + e − 2 λ t i ) e − λ t i − 1 − e − λ t i γ 2 + e − 2 λ t i ∫ λ t i − 1 λ t i e − λ d λ = ( 1 − e − δ i ) ( e − λ t i − 1 − λ t i − γ 2 ) γ 2 + e − 2 λ t i − e − λ t i γ 2 + e − 2 λ t i ( e − λ t i − 1 − e − λ t i ) = − γ 2 ( 1 − e − δ i ) γ 2 + e − 2 λ t i + e − λ t i − 1 − λ t i − e − 2 λ t i − e − λ t i ( e − λ t i − 1 − e − λ t i ) γ 2 + e − 2 λ t i = − γ 2 ( 1 − e − δ i ) γ 2 + e − 2 λ t i . \begin{aligned}
\frac{e^{-\lambda_{t_i-1}}(1-e^{-\delta_i})(e^{-\lambda_{t_i-1}-\lambda_{t_i}}-\gamma^2)}{(\gamma^2+e^{-2\lambda_{t_i}})e^{-\lambda_{t_i-1}}}-\frac{e^{-\lambda_{t_i}}}{\gamma^2+e^{-2\lambda_{t_i}}}\int_{\lambda_{t_i-1}}^{\lambda_{t_i}}e^{-\lambda}\,\mathrm{d}\lambda\\
&=\frac{(1-e^{-\delta_i})(e^{-\lambda_{t_i-1}-\lambda_{t_i}}-\gamma^2)}{\gamma^2+e^{-2\lambda_{t_i}}}-\frac{e^{-\lambda_{t_i}}}{\gamma^2+e^{-2\lambda_{t_i}}}(e^{-\lambda_{t_i-1}}-e^{-\lambda_{t_i}}) \\
&=-\frac{\gamma^2(1-e^{-\delta_i})}{\gamma^2+e^{-2\lambda_{t_i}}}+\frac{e^{-\lambda_{t_i-1}-\lambda_{t_i}}-e^{-2\lambda_{t_i}}-e^{-\lambda_{t_i}}(e^{-\lambda_{t_i-1}}-e^{-\lambda_{t_i}})}{\gamma^2+e^{-2\lambda_{t_i}}} \\
&=-\frac{\gamma^2(1-e^{-\delta_i})}{\gamma^2+e^{-2\lambda_{t_i}}}.
\end{aligned} ( γ 2 + e − 2 λ t i ) e − λ t i − 1 e − λ t i − 1 ( 1 − e − δ i ) ( e − λ t i − 1 − λ t i − γ 2 ) − γ 2 + e − 2 λ t i e − λ t i ∫ λ t i − 1 λ t i e − λ d λ = γ 2 + e − 2 λ t i ( 1 − e − δ i ) ( e − λ t i − 1 − λ t i − γ 2 ) − γ 2 + e − 2 λ t i e − λ t i ( e − λ t i − 1 − e − λ t i ) = − γ 2 + e − 2 λ t i γ 2 ( 1 − e − δ i ) + γ 2 + e − 2 λ t i e − λ t i − 1 − λ t i − e − 2 λ t i − e − λ t i ( e − λ t i − 1 − e − λ t i ) = − γ 2 + e − 2 λ t i γ 2 ( 1 − e − δ i ) .
Equation (70) can be further simplified by leveraging the fact that 1 − e − δ i ≤ δ i 1-e^{-\delta_i}\le\delta_i 1 − e − δ i ≤ δ i :
∣ 1 δ i ( α t i − 1 e − λ t i ( γ 2 + e − 2 λ t i − 1 ) α t i ( γ 2 + e − 2 λ t i ) e − λ t i − 1 a i − 1 ) ∣ ≤ γ 2 γ 2 + e − 2 λ t i ≤ 1. (72) \left|\frac{1}{\delta_i}\left(\frac{\alpha_{t_i-1}e^{-\lambda_{t_i}}(\gamma^2+e^{-2\lambda_{t_i-1}})}{\alpha_{t_i}(\gamma^2+e^{-2\lambda_{t_i}})e^{-\lambda_{t_i-1}}}a_i-1\right)\right|\le\frac{\gamma^2}{\gamma^2+e^{-2\lambda_{t_i}}}\le1. \tag*{(72)} δ i 1 ( α t i ( γ 2 + e − 2 λ t i ) e − λ t i − 1 α t i − 1 e − λ t i ( γ 2 + e − 2 λ t i − 1 ) a i − 1 ) ≤ γ 2 + e − 2 λ t i γ 2 ≤ 1. (72)
Finally, plugging the above bounds (69) , (68) , and (72) into (67) , we find that the absolute value of g t i − g t i − 1 λ t i − λ t i − 1 \frac{g_{t_i}-g_{t_i-1}}{\lambda_{t_i}-\lambda_{t_i-1}} λ t i − λ t i − 1 g t i − g t i − 1 is bounded by
∣ g t i − g t i − 1 λ t i − λ t i − 1 ∣ ≤ δ i ∣ g t i − 1 − g t i − 2 λ t i − 1 − λ t i − 2 ∣ + ∣ g t i − 1 ∣ + O ( M − 2 ) . \left|\frac{g_{t_i}-g_{t_i-1}}{\lambda_{t_i}-\lambda_{t_i-1}}\right|\le\delta_i\left|\frac{g_{t_i-1}-g_{t_i-2}}{\lambda_{t_i-1}-\lambda_{t_i-2}}\right|+|g_{t_i-1}|+O(M^{-2}). λ t i − λ t i − 1 g t i − g t i − 1 ≤ δ i λ t i − 1 − λ t i − 2 g t i − 1 − g t i − 2 + ∣ g t i − 1 ∣ + O ( M − 2 ) .
According to the fact that the convergence order of DPMSolver-2 is 2 (Lu et al., 2022a, Theorem 3.2, [73 Theorem 3.3] ), we have g t i − 1 = O ( M − 2 ) g_{t_{i-1}}=O(M^{-2}) g t i − 1 = O ( M − 2 ) . The DDIM initialization gives g t 0 = 0 g_{t_0}=0 g t 0 = 0 and g t 1 = O ( δ 1 2 ) g_{t_1}=O(\delta_1^2) g t 1 = O ( δ 1 2 ) , so ( g t 1 − g t 0 ) / δ 1 = O ( δ 1 ) (g_{t_1}-g_{t_0})/\delta_1=O(\delta_1) ( g t 1 − g t 0 ) / δ 1 = O ( δ 1 ) . Applying the recursion gives:
∣ g t i − g t i − 1 λ t i − λ t i − 1 ∣ = O ( M − 2 ) , 2 ≤ i ≤ M , and g t 1 − g t 0 δ 1 = O ( M − 1 ) . (73) \left|\frac{g_{t_i}-g_{t_{i-1}}}{\lambda_{t_i}-\lambda_{t_{i-1}}}\right|=O(M^{-2}),\quad2\leq i\leq M,\qquad\text{and}\qquad\frac{g_{t_1}-g_{t_0}}{\delta_1}=O(M^{-1}). \tag*{(73)} λ t i − λ t i − 1 g t i − g t i − 1 = O ( M − 2 ) , 2 ≤ i ≤ M , and δ 1 g t 1 − g t 0 = O ( M − 1 ) . (73)
Substituting (73) into (65) , we bound the term ∣ b i ∣ |b_i| ∣ b i ∣ for i ≥ 3 i\geq3 i ≥ 3 by
∣ b i ∣ ≤ α t i ∫ λ t i − 1 λ t i ( λ − λ t i − 1 ) e − λ d λ ∣ g t i − 1 − g t i − 2 λ t i − 1 − λ t i − 2 ∣ ≤ 1 2 α t i e − λ t i − 1 δ i 2 ∣ g t i − 1 − g t i − 2 λ t i − 1 − λ t i − 2 ∣ = O ( M − 4 ) , i ≥ 3. (74) \begin{aligned}
|b_i|\leq\alpha_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}(\lambda-\lambda_{t_{i-1}})e^{-\lambda}\,\mathrm{d}\lambda\left|\frac{g_{t_{i-1}}-g_{t_{i-2}}}{\lambda_{t_{i-1}}-\lambda_{t_{i-2}}}\right| \\
&\leq\frac{1}{2}\alpha_{t_i}e^{-\lambda_{t_{i-1}}}\delta_i^2\left|\frac{g_{t_{i-1}}-g_{t_{i-2}}}{\lambda_{t_{i-1}}-\lambda_{t_{i-2}}}\right|=O\left(M^{-4}\right),\qquad i\geq3. \tag*{(74)}
\end{aligned} ∣ b i ∣ ≤ α t i ∫ λ t i − 1 λ t i ( λ − λ t i − 1 ) e − λ d λ λ t i − 1 − λ t i − 2 g t i − 1 − g t i − 2 ≤ 2 1 α t i e − λ t i − 1 δ i 2 λ t i − 1 − λ t i − 2 g t i − 1 − g t i − 2 = O ( M − 4 ) , i ≥ 3. (74)
For i = 2 i=2 i = 2 , the initial bound (73) gives
∣ b 2 ∣ = O ( δ 2 2 δ 1 ) = O ( M − 3 ) . |b_2|=O(\delta_2^2\delta_1)=O(M^{-3}). ∣ b 2 ∣ = O ( δ 2 2 δ 1 ) = O ( M − 3 ) .
The total contribution of the b i b_i b i terms satisfies
∑ i = 2 M ∣ ( ∏ j = i + 1 M a j ) b i ∣ ≤ C ( O ( M − 3 ) + M ⋅ O ( M − 4 ) ) = O ( M − 3 ) . \sum_{i=2}^{M}\left|\left(\prod_{j=i+1}^{M}a_j\right)b_i\right|\leq C\left(O(M^{-3})+M\cdot O(M^{-4})\right)=O(M^{-3}). i = 2 ∑ M ( j = i + 1 ∏ M a j ) b i ≤ C ( O ( M − 3 ) + M ⋅ O ( M − 4 ) ) = O ( M − 3 ) .
δ ~ i : = e − λ t i − 1 γ 2 + e − 2 λ t i − 1 ∫ λ t i − 1 λ t i e − λ d λ . \widetilde{\delta}_i:=\frac{e^{-\lambda_{t_{i-1}}}}{\gamma^2+e^{-2\lambda_{t_{i-1}}}}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-\lambda}\,\mathrm{d}\lambda. δ i := γ 2 + e − 2 λ t i − 1 e − λ t i − 1 ∫ λ t i − 1 λ t i e − λ d λ .
By the definition of a i a_i a i in (49) , we have
∏ j = i + 1 M a j = α t M α t i ∏ j = i + 1 M ( 1 − δ ~ j ) = ( a ) α t M α t i exp ( − ∑ j = i + 1 M δ ~ j + O ( ∑ j = i + 1 M δ ~ j 2 ) ) . (75) \prod_{j=i+1}^{M}a_j=\frac{\alpha_{t_M}}{\alpha_{t_i}}\prod_{j=i+1}^{M}(1-\widetilde{\delta}_j)\mathrel{\overset{(a)}{=}}\frac{\alpha_{t_M}}{\alpha_{t_i}}\exp\left(-\sum_{j=i+1}^{M}\widetilde{\delta}_j+O\left(\sum_{j=i+1}^{M}\widetilde{\delta}_j^2\right)\right). \tag*{(75)} j = i + 1 ∏ M a j = α t i α t M j = i + 1 ∏ M ( 1 − δ j ) = ( a ) α t i α t M exp ( − j = i + 1 ∑ M δ j + O ( j = i + 1 ∑ M δ j 2 ) ) . (75)
where (a) holds because
log ( 1 − δ ~ j ) = − δ ~ j + O ( δ ~ j 2 ) . (76) \log(1-\widetilde{\delta}_j)=-\widetilde{\delta}_j+O(\widetilde{\delta}_j^2). \tag*{(76)} log ( 1 − δ j ) = − δ j + O ( δ j 2 ) . (76)
We can compute the sum of δ ~ j \widetilde{\delta}_j δ j and δ ~ j 2 \widetilde{\delta}_j^2 δ j 2 as
∑ j = i + 1 M δ ~ j = ∑ j = i + 1 M e − λ t j − 1 γ 2 + e − 2 λ t j − 1 ∫ λ t j − 1 λ t j e − λ d λ = ( a ) ∑ j = i + 1 M ∫ λ t j − 1 λ t j e − 2 λ γ 2 + e − 2 λ d λ + O ( M − 1 ) = ∫ λ t i λ t M e − 2 λ γ 2 + e − 2 λ d λ + O ( M − 1 ) \begin{aligned}
\sum_{j=i+1}^{M}\widetilde{\delta}_j=\sum_{j=i+1}^{M}\frac{e^{-\lambda_{t_{j-1}}}}{\gamma^2+e^{-2\lambda_{t_{j-1}}}}\int_{\lambda_{t_{j-1}}}^{\lambda_{t_j}}e^{-\lambda}\,\mathrm{d}\lambda\mathrel{\overset{(a)}{=}}\sum_{j=i+1}^{M}\int_{\lambda_{t_{j-1}}}^{\lambda_{t_j}}\frac{e^{-2\lambda}}{\gamma^2+e^{-2\lambda}}\,\mathrm{d}\lambda+O(M^{-1}) \\
&=\int_{\lambda_{t_i}}^{\lambda_{t_M}}\frac{e^{-2\lambda}}{\gamma^2+e^{-2\lambda}}\,\mathrm{d}\lambda+O(M^{-1})
\end{aligned} j = i + 1 ∑ M δ j = j = i + 1 ∑ M γ 2 + e − 2 λ t j − 1 e − λ t j − 1 ∫ λ t j − 1 λ t j e − λ d λ = ( a ) j = i + 1 ∑ M ∫ λ t j − 1 λ t j γ 2 + e − 2 λ e − 2 λ d λ + O ( M − 1 ) = ∫ λ t i λ t M γ 2 + e − 2 λ e − 2 λ d λ + O ( M − 1 )
= ( b ) 1 2 log ( γ 2 + e − 2 λ t i γ 2 + e − 2 λ t M ) + O ( M − 1 ) . (77) \mathrel{\overset{(b)}{=}}\frac{1}{2}\log\left(\frac{\gamma^2+e^{-2\lambda_{t_i}}}{\gamma^2+e^{-2\lambda_{t_M}}}\right)+O(M^{-1}). \tag*{(77)} = ( b ) 2 1 log ( γ 2 + e − 2 λ t M γ 2 + e − 2 λ t i ) + O ( M − 1 ) . (77)
where (a) holds because
∫ λ t j − 1 λ t j ∣ e − λ t j − 1 γ 2 + e − 2 λ t j − 1 − e − λ γ 2 + e − 2 λ ∣ e − λ d λ ≤ e − λ t j − 1 max λ ∈ [ λ t j − 1 , λ t j ] ∣ d d λ e − λ γ 2 + e − 2 λ ∣ ∫ λ t j − 1 λ t j ( λ − λ t j − 1 ) d λ = δ j 2 2 e − λ t j − 1 max λ ∈ [ λ t j − 1 , λ t j ] e − λ ∣ γ 2 − e − 2 λ ∣ ( γ 2 + e − 2 λ ) 2 ≤ e δ j 2 δ j 2 = O ( M − 2 ) . \begin{aligned}
\int_{\lambda_{t_{j-1}}}^{\lambda_{t_j}}\left|\frac{e^{-\lambda_{t_{j-1}}}}{\gamma^2+e^{-2\lambda_{t_{j-1}}}}-\frac{e^{-\lambda}}{\gamma^2+e^{-2\lambda}}\right|e^{-\lambda}\,\mathrm{d}\lambda
\leq e^{-\lambda_{t_{j-1}}}\max_{\lambda\in[\lambda_{t_{j-1}},\lambda_{t_j}]}\left|\frac{\mathrm{d}}{\mathrm{d}\lambda}\frac{e^{-\lambda}}{\gamma^2+e^{-2\lambda}}\right|\int_{\lambda_{t_{j-1}}}^{\lambda_{t_j}}(\lambda-\lambda_{t_{j-1}})\,\mathrm{d}\lambda\\
&=\frac{\delta_j^2}{2}e^{-\lambda_{t_{j-1}}}\max_{\lambda\in[\lambda_{t_{j-1}},\lambda_{t_j}]}\frac{e^{-\lambda}|\gamma^2-e^{-2\lambda}|}{(\gamma^2+e^{-2\lambda})^2}\leq\frac{e^{\delta_j}}{2}\delta_j^2=O(M^{-2}).
\end{aligned} ∫ λ t j − 1 λ t j γ 2 + e − 2 λ t j − 1 e − λ t j − 1 − γ 2 + e − 2 λ e − λ e − λ d λ ≤ e − λ t j − 1 λ ∈ [ λ t j − 1 , λ t j ] max d λ d γ 2 + e − 2 λ e − λ ∫ λ t j − 1 λ t j ( λ − λ t j − 1 ) d λ = 2 δ j 2 e − λ t j − 1 λ ∈ [ λ t j − 1 , λ t j ] max ( γ 2 + e − 2 λ ) 2 e − λ ∣ γ 2 − e − 2 λ ∣ ≤ 2 e δ j δ j 2 = O ( M − 2 ) .
and (b) arises from
∫ λ t i λ t M e − 2 λ γ 2 + e − 2 λ d λ = 1 2 log ( γ 2 + e − 2 λ ) ∣ λ t M λ t i = 1 2 log ( γ 2 + e − 2 λ t i γ 2 + e − 2 λ t M ) . \int_{\lambda_{t_i}}^{\lambda_{t_M}} \frac{e^{-2\lambda}}{\gamma^2+e^{-2\lambda}}\,\mathrm{d}\lambda= \left.\frac{1}{2}\log(\gamma^2+e^{-2\lambda})\right|_{\lambda_{t_M}}^{\lambda_{t_i}} = \frac{1}{2}\log\left(\frac{\gamma^2+e^{-2\lambda_{t_i}}}{\gamma^2+e^{-2\lambda_{t_M}}}\right). ∫ λ t i λ t M γ 2 + e − 2 λ e − 2 λ d λ = 2 1 log ( γ 2 + e − 2 λ ) λ t M λ t i = 2 1 log ( γ 2 + e − 2 λ t M γ 2 + e − 2 λ t i ) .
In addition, the sum of δ ~ j 2 \tilde{\delta}_j^2 δ ~ j 2 is computed as
∑ j = i + 1 M δ ~ j 2 ≤ ∑ i = 1 M e − 2 λ t i − 1 ( γ 2 + e − 2 λ t i − 1 ) 2 ( ∫ λ t i − 1 λ t i e − λ d λ ) 2 ≤ ∑ i = 1 M δ i 2 e − 4 λ t i − 1 ( γ 2 + e − 2 λ t i − 1 ) 2 ≤ ∑ i = 1 M δ i 2 = O ( M − 1 ) . (78) \sum_{j=i+1}^{M}\tilde{\delta}_j^2 \leq\sum_{i=1}^{M}\frac{e^{-2\lambda_{t_{i-1}}}}{(\gamma^2+e^{-2\lambda_{t_{i-1}}})^2}\left(\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-\lambda}\,\mathrm{d}\lambda\right)^2 \leq\sum_{i=1}^{M}\frac{\delta_i^2e^{-4\lambda_{t_{i-1}}}}{(\gamma^2+e^{-2\lambda_{t_{i-1}}})^2}\leq\sum_{i=1}^{M}\delta_i^2=O(M^{-1}). \tag*{(78)} j = i + 1 ∑ M δ ~ j 2 ≤ i = 1 ∑ M ( γ 2 + e − 2 λ t i − 1 ) 2 e − 2 λ t i − 1 ( ∫ λ t i − 1 λ t i e − λ d λ ) 2 ≤ i = 1 ∑ M ( γ 2 + e − 2 λ t i − 1 ) 2 δ i 2 e − 4 λ t i − 1 ≤ i = 1 ∑ M δ i 2 = O ( M − 1 ) . (78)
Substituting (77) and (78) into (75) , we have
∏ j = i + 1 M a j = α t M α t i γ 2 + e − 2 λ t M γ 2 + e − 2 λ t i ( 1 + O ( M − 1 ) ) . (79) \prod_{j=i+1}^{M}a_j=\frac{\alpha_{t_M}}{\alpha_{t_i}}\sqrt{\frac{\gamma^2+e^{-2\lambda_{t_M}}}{\gamma^2+e^{-2\lambda_{t_i}}}}\left(1+O(M^{-1})\right). \tag*{(79)} j = i + 1 ∏ M a j = α t i α t M γ 2 + e − 2 λ t i γ 2 + e − 2 λ t M ( 1 + O ( M − 1 ) ) . (79)
Since a j > 0 a_j>0 a j > 0 and κ t 1 ∗ − κ t 1 ≥ 0 \kappa_{t_1}^*-\kappa_{t_1}\geq0 κ t 1 ∗ − κ t 1 ≥ 0 , the initialization term in (60) is nonnegative and can be omitted when deriving a lower bound. Finally, plugging the above bounds (79) , (74) , and (64) into (60) , the discretization error at the last step is bounded below by
κ t M ∗ − κ t M ≥ α t M γ 2 + e − 2 λ t M 12 α t 0 γ 2 + e − 2 λ t 0 ∑ i = 2 M e − λ t i − 1 f ′ ′ ( λ t i − 1 ) γ 2 + e − 2 λ t i δ i 2 ( 2 δ i + 3 δ i − 1 ) + O ( M − 3 ) = α t M γ 2 + e − 2 λ t M 12 α t 0 γ 2 + e − 2 λ t 0 ∑ i = 2 M e − λ t i − 1 γ 2 e 2 λ t i − 1 ( 2 − γ 2 e 2 λ t i − 1 ) ( γ 2 e 2 λ t i − 1 + 1 ) 5 / 2 δ i 2 ( 2 δ i + 3 δ i − 1 ) + O ( M − 3 ) = α t M γ 2 + e − 2 λ t M 12 α t 0 γ 2 + e − 2 λ t 0 ∑ i = 2 M γ 2 e 2 λ t i − 1 ( 2 − γ 2 e 2 λ t i − 1 ) ( γ 2 e 2 λ t i − 1 + 1 ) 3 δ i 2 ( 2 δ i + 3 δ i − 1 ) + O ( M − 3 ) , \begin{aligned}
\kappa_{t_M}^*-\kappa_{t_M}\geq\frac{\alpha_{t_M}\sqrt{\gamma^2+e^{-2\lambda_{t_M}}}}{12\alpha_{t_0}\sqrt{\gamma^2+e^{-2\lambda_{t_0}}}}\sum_{i=2}^{M}\frac{e^{-\lambda_{t_{i-1}}}f''(\lambda_{t_{i-1}})}{\sqrt{\gamma^2+e^{-2\lambda_{t_i}}}}\delta_i^2(2\delta_i+3\delta_{i-1})+O(M^{-3}) \\
&=\frac{\alpha_{t_M}\sqrt{\gamma^2+e^{-2\lambda_{t_M}}}}{12\alpha_{t_0}\sqrt{\gamma^2+e^{-2\lambda_{t_0}}}}\sum_{i=2}^{M}\frac{e^{-\lambda_{t_{i-1}}}\gamma^2e^{2\lambda_{t_{i-1}}}(2-\gamma^2e^{2\lambda_{t_{i-1}}})}{(\gamma^2e^{2\lambda_{t_{i-1}}}+1)^{5/2}}\delta_i^2(2\delta_i+3\delta_{i-1})+O(M^{-3}) \\
&=\frac{\alpha_{t_M}\sqrt{\gamma^2+e^{-2\lambda_{t_M}}}}{12\alpha_{t_0}\sqrt{\gamma^2+e^{-2\lambda_{t_0}}}}\sum_{i=2}^{M}\frac{\gamma^2e^{2\lambda_{t_{i-1}}}(2-\gamma^2e^{2\lambda_{t_{i-1}}})}{(\gamma^2e^{2\lambda_{t_{i-1}}}+1)^3}\delta_i^2(2\delta_i+3\delta_{i-1})+O(M^{-3}),
\end{aligned} κ t M ∗ − κ t M ≥ 12 α t 0 γ 2 + e − 2 λ t 0 α t M γ 2 + e − 2 λ t M i = 2 ∑ M γ 2 + e − 2 λ t i e − λ t i − 1 f ′′ ( λ t i − 1 ) δ i 2 ( 2 δ i + 3 δ i − 1 ) + O ( M − 3 ) = 12 α t 0 γ 2 + e − 2 λ t 0 α t M γ 2 + e − 2 λ t M i = 2 ∑ M ( γ 2 e 2 λ t i − 1 + 1 ) 5/2 e − λ t i − 1 γ 2 e 2 λ t i − 1 ( 2 − γ 2 e 2 λ t i − 1 ) δ i 2 ( 2 δ i + 3 δ i − 1 ) + O ( M − 3 ) = 12 α t 0 γ 2 + e − 2 λ t 0 α t M γ 2 + e − 2 λ t M i = 2 ∑ M ( γ 2 e 2 λ t i − 1 + 1 ) 3 γ 2 e 2 λ t i − 1 ( 2 − γ 2 e 2 λ t i − 1 ) δ i 2 ( 2 δ i + 3 δ i − 1 ) + O ( M − 3 ) ,
where the second line inserts the second-order derivative of f f f from (63) . Taking γ = e − λ t M \gamma=e^{-\lambda_{t_M}} γ = e − λ t M , we have 2 − γ 2 e 2 λ t i − 1 ≥ 0 2-\gamma^2e^{2\lambda_{t_{i-1}}}\geq0 2 − γ 2 e 2 λ t i − 1 ≥ 0 for all 0 ≤ i − 1 ≤ M 0\leq i-1\leq M 0 ≤ i − 1 ≤ M . Therefore, we arrive at the desired result:
κ t M ∗ − κ t M ≥ α t M γ 2 + e − 2 λ t M 12 α t 0 γ 2 + e − 2 λ t 0 ∑ i ∈ S γ 2 e 2 λ t i − 1 ( 2 − γ 2 e 2 λ t i − 1 ) ( γ 2 e 2 λ t i − 1 + 1 ) 3 δ i 2 ( 2 δ i + 3 δ i − 1 ) + O ( M − 3 ) ≥ 2 α t M γ 2 + e − 2 λ t M γ 2 c 1 2 ( 2 − γ 2 c 2 2 ) 12 α t 0 γ 2 + e − 2 λ t 0 ( γ 2 c 2 2 + 1 ) 3 ∑ i ∈ S δ i 3 + O ( M − 3 ) ≥ 2 α t M γ 2 + e − 2 λ t M γ 2 c 1 2 ( 2 − γ 2 c 2 2 ) 12 α t 0 γ 2 + e − 2 λ t 0 ( γ 2 c 2 2 + 1 ) 3 ( log c 2 − log c 1 ) 3 8 M 2 + O ( M − 3 ) = Ω ( M − 2 ) , \begin{aligned}
\kappa_{t_M}^*-\kappa_{t_M}\geq\frac{\alpha_{t_M}\sqrt{\gamma^2+e^{-2\lambda_{t_M}}}}{12\alpha_{t_0}\sqrt{\gamma^2+e^{-2\lambda_{t_0}}}}\sum_{i\in S}\frac{\gamma^2e^{2\lambda_{t_{i-1}}}(2-\gamma^2e^{2\lambda_{t_{i-1}}})}{(\gamma^2e^{2\lambda_{t_{i-1}}}+1)^3}\delta_i^2(2\delta_i+3\delta_{i-1})+O(M^{-3}) \\
&\geq\frac{2\alpha_{t_M}\sqrt{\gamma^2+e^{-2\lambda_{t_M}}}\gamma^2c_1^2(2-\gamma^2c_2^2)}{12\alpha_{t_0}\sqrt{\gamma^2+e^{-2\lambda_{t_0}}}(\gamma^2c_2^2+1)^3}\sum_{i\in S}\delta_i^3+O(M^{-3}) \\
&\geq\frac{2\alpha_{t_M}\sqrt{\gamma^2+e^{-2\lambda_{t_M}}}\gamma^2c_1^2(2-\gamma^2c_2^2)}{12\alpha_{t_0}\sqrt{\gamma^2+e^{-2\lambda_{t_0}}}(\gamma^2c_2^2+1)^3}\frac{(\log c_2-\log c_1)^3}{8M^2}+O(M^{-3})=\Omega(M^{-2}),
\end{aligned} κ t M ∗ − κ t M ≥ 12 α t 0 γ 2 + e − 2 λ t 0 α t M γ 2 + e − 2 λ t M i ∈ S ∑ ( γ 2 e 2 λ t i − 1 + 1 ) 3 γ 2 e 2 λ t i − 1 ( 2 − γ 2 e 2 λ t i − 1 ) δ i 2 ( 2 δ i + 3 δ i − 1 ) + O ( M − 3 ) ≥ 12 α t 0 γ 2 + e − 2 λ t 0 ( γ 2 c 2 2 + 1 ) 3 2 α t M γ 2 + e − 2 λ t M γ 2 c 1 2 ( 2 − γ 2 c 2 2 ) i ∈ S ∑ δ i 3 + O ( M − 3 ) ≥ 12 α t 0 γ 2 + e − 2 λ t 0 ( γ 2 c 2 2 + 1 ) 3 2 α t M γ 2 + e − 2 λ t M γ 2 c 1 2 ( 2 − γ 2 c 2 2 ) 8 M 2 ( log c 2 − log c 1 ) 3 + O ( M − 3 ) = Ω ( M − 2 ) ,
where S S S is defined in (53) , and the last inequality uses Holder inequality that
1 2 ( log c 2 − log c 1 ) ≤ ∑ i ∈ S δ i ≤ ( ∑ i ∈ S δ i 3 ) 1 3 ( ∑ i ∈ S 1 ) 2 3 ≤ ( ∑ i ∈ S δ i 3 ) 1 3 M 2 3 . \frac{1}{2}(\log c_2-\log c_1)\leq\sum_{i\in S}\delta_i\leq\left(\sum_{i\in S}\delta_i^3\right)^{\frac{1}{3}}\left(\sum_{i\in S}1\right)^{\frac{2}{3}}\leq\left(\sum_{i\in S}\delta_i^3\right)^{\frac{1}{3}}M^{\frac{2}{3}}. 2 1 ( log c 2 − log c 1 ) ≤ i ∈ S ∑ δ i ≤ ( i ∈ S ∑ δ i 3 ) 3 1 ( i ∈ S ∑ 1 ) 3 2 ≤ ( i ∈ S ∑ δ i 3 ) 3 1 M 3 2 .
This finishes the proof.
Proof of stochastic samplers convergence upper bounds (Theorem 2)
We begin by analyzing the discretization scheme in (18) .
Convergence analysis of SDE-DPMSolver++
We first state a preliminary moment bound, whose proof is deferred to the end of this section:
E [ ∥ x t i ⋆ ∥ 2 2 ] = O ( 1 ) , 0 ≤ i ≤ M . (80) \mathbb{E}[\lVert x_{t_i}^{\star}\rVert_2^2] = O(1), \qquad0 \le i \le M. \tag*{(80)} E [∥ x t i ⋆ ∥ 2 2 ] = O ( 1 ) , 0 ≤ i ≤ M . (80)
Equipped with this estimate, we proceed to establish (18) in four steps.
Step 1: decompose the error x t i − x t i ⋆ x_{t_i}-x_{t_i}^{\star} x t i − x t i ⋆ . Subtracting (31) from (18) gives
x t i − x t i ⋆ = σ t i σ t i − 1 e − δ i ( x t i − 1 − x t i − 1 ⋆ ) + 2 α t i ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) ( μ ( x t i − 1 , t i − 1 ) − μ ( x t ( λ ) ⋆ , t ( λ ) ) ) d λ = σ t i σ t i − 1 e − δ i ( x t i − 1 − x t i − 1 ⋆ ) + α t i ( 1 − e − 2 δ i ) ( μ ( x t i − 1 , t i − 1 ) − μ ( x t i − 1 ⋆ , t i − 1 ) ) + 2 α t i ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) ( μ ( x t i − 1 ⋆ , t i − 1 ) − μ ( x t ( λ ) ⋆ , t ( λ ) ) ) d λ . (81) \begin{aligned}
x_{t_i}-x_{t_i}^{\star}
&= \frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}e^{-\delta_i}(x_{t_{i-1}}-x_{t_{i-1}}^{\star}) + 2\alpha_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-2(\lambda_{t_i}-\lambda)}\bigl(\mu(x_{t_{i-1}},t_{i-1})-\mu(x_{t(\lambda)}^{\star},t(\lambda))\bigr)\,d\lambda\\
&= \frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}e^{-\delta_i}(x_{t_{i-1}}-x_{t_{i-1}}^{\star})+\alpha_{t_i}(1-e^{-2\delta_i})\bigl(\mu(x_{t_{i-1}},t_{i-1})-\mu(x_{t_{i-1}}^{\star},t_{i-1})\bigr) \\
&\quad+2\alpha_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-2(\lambda_{t_i}-\lambda)}\bigl(\mu(x_{t_{i-1}}^{\star},t_{i-1})-\mu(x_{t(\lambda)}^{\star},t(\lambda))\bigr)\,d\lambda. \tag*{(81)}
\end{aligned} x t i − x t i ⋆ = σ t i − 1 σ t i e − δ i ( x t i − 1 − x t i − 1 ⋆ ) + 2 α t i ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) ( μ ( x t i − 1 , t i − 1 ) − μ ( x t ( λ ) ⋆ , t ( λ )) ) d λ = σ t i − 1 σ t i e − δ i ( x t i − 1 − x t i − 1 ⋆ ) + α t i ( 1 − e − 2 δ i ) ( μ ( x t i − 1 , t i − 1 ) − μ ( x t i − 1 ⋆ , t i − 1 ) ) + 2 α t i ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) ( μ ( x t i − 1 ⋆ , t i − 1 ) − μ ( x t ( λ ) ⋆ , t ( λ )) ) d λ . (81)
To treat the last integral, we expand the integrand μ ( x t i − 1 ⋆ , t i − 1 ) − μ ( x t ( λ ) ⋆ , t ( λ ) ) \mu(x_{t_{i-1}}^{\star},t_{i-1})-\mu(x_{t(\lambda)}^{\star},t(\lambda)) μ ( x t i − 1 ⋆ , t i − 1 ) − μ ( x t ( λ ) ⋆ , t ( λ )) via Taylor’s theorem:
μ ( x t i − 1 ⋆ , t i − 1 ) − μ ( x t ( λ ) ⋆ , t ( λ ) ) = ∂ ∂ λ μ ( x t i − 1 ⋆ , t i − 1 ) ( λ t i − 1 − λ ) + ∇ x μ ( x t i − 1 ⋆ , λ t i − 1 ) T ( x t i − 1 ⋆ − x t ( λ ) ⋆ ) + O ( δ i 2 ) + O ( ∥ x t i − 1 ⋆ − x t ( λ ) ⋆ ∥ 2 2 ) . (82) \begin{aligned}
\mu(x_{t_{i-1}}^{\star},t_{i-1})-\mu(x_{t(\lambda)}^{\star},t(\lambda))
&= \frac{\partial}{\partial\lambda}\mu(x_{t_{i-1}}^{\star},t_{i-1})(\lambda_{t_{i-1}}-\lambda) \\
&\quad+ \nabla_x\mu(x_{t_{i-1}}^{\star},\lambda_{t_{i-1}})^{\mathsf{T}}(x_{t_{i-1}}^{\star}-x_{t(\lambda)}^{\star}) + O(\delta_i^2)+O(\lVert x_{t_{i-1}}^{\star}-x_{t(\lambda)}^{\star}\rVert_2^2). \tag*{(82)}
\end{aligned} μ ( x t i − 1 ⋆ , t i − 1 ) − μ ( x t ( λ ) ⋆ , t ( λ )) = ∂ λ ∂ μ ( x t i − 1 ⋆ , t i − 1 ) ( λ t i − 1 − λ ) + ∇ x μ ( x t i − 1 ⋆ , λ t i − 1 ) T ( x t i − 1 ⋆ − x t ( λ ) ⋆ ) + O ( δ i 2 ) + O (∥ x t i − 1 ⋆ − x t ( λ ) ⋆ ∥ 2 2 ) . (82)
From the SDE dynamics (31) , we have
x t ( λ ) ⋆ − x t i − 1 ⋆ = ( σ t ( λ ) σ t i − 1 e − ( λ − λ t i − 1 ) − 1 ) x t i − 1 ⋆ + 2 α t ( λ ) ∫ λ t i − 1 λ e − 2 ( λ − λ ′ ) μ ( x t ( λ ′ ) ⋆ , t ( λ ′ ) ) d λ ′ + 2 σ t ( λ ) ∫ λ t i − 1 λ e − ( λ − λ ′ ) d w ‾ t ( λ ′ ) . x_{t(\lambda)}^{\star}-x_{t_{i-1}}^{\star}=\left(\frac{\sigma_{t(\lambda)}}{\sigma_{t_{i-1}}}e^{-(\lambda-\lambda_{t_{i-1}})}-1\right)x_{t_{i-1}}^{\star}+2\alpha_{t(\lambda)}\int_{\lambda_{t_{i-1}}}^{\lambda}e^{-2(\lambda-\lambda')}\mu(x_{t(\lambda')}^{\star},t(\lambda'))\,d\lambda'+\sqrt{2}\sigma_{t(\lambda)}\int_{\lambda_{t_{i-1}}}^{\lambda}e^{-(\lambda-\lambda')}\,d\overline{w}_{t(\lambda')}. x t ( λ ) ⋆ − x t i − 1 ⋆ = ( σ t i − 1 σ t ( λ ) e − ( λ − λ t i − 1 ) − 1 ) x t i − 1 ⋆ + 2 α t ( λ ) ∫ λ t i − 1 λ e − 2 ( λ − λ ′ ) μ ( x t ( λ ′ ) ⋆ , t ( λ ′ )) d λ ′ + 2 σ t ( λ ) ∫ λ t i − 1 λ e − ( λ − λ ′ ) d w t ( λ ′ ) .
Substituting this into (82) yields
μ ( x t i − 1 ⋆ , t i − 1 ) − μ ( x t ( λ ) ⋆ , t ( λ ) ) = ∂ ∂ λ μ ( x t i − 1 ⋆ , t i − 1 ) ( λ t i − 1 − λ ) − ( σ t ( λ ) σ t i − 1 e − ( λ − λ t i − 1 ) − 1 ) ∇ x μ ( x t i − 1 ⋆ , λ t i − 1 ) T x t i − 1 ⋆ − 2 α t ( λ ) ∫ λ t i − 1 λ e − 2 ( λ − λ ′ ) ∇ μ ( x t i − 1 ⋆ , λ t i − 1 ) T μ ( x t ( λ ′ ) ⋆ , λ ′ ) d λ ′ − 2 σ t ( λ ) ∇ μ ( x t i − 1 ⋆ , λ t i − 1 ) T ∫ λ t i − 1 λ e − ( λ − λ ′ ) d w ‾ t ( λ ′ ) + O ( δ i 2 ) + O ( ∥ x t i − 1 ⋆ − x t ( λ ) ⋆ ∥ 2 2 ) . (83) \begin{aligned}
\mu(x_{t_{i-1}}^{\star},t_{i-1})-\mu(x_{t(\lambda)}^{\star},t(\lambda))
&= \frac{\partial}{\partial\lambda}\mu(x_{t_{i-1}}^{\star},t_{i-1})(\lambda_{t_{i-1}}-\lambda) \\
&\quad-\left(\frac{\sigma_{t(\lambda)}}{\sigma_{t_{i-1}}}e^{-(\lambda-\lambda_{t_{i-1}})}-1\right)\nabla_x\mu(x_{t_{i-1}}^{\star},\lambda_{t_{i-1}})^{\mathsf{T}}x_{t_{i-1}}^{\star} \\
&\quad-2\alpha_{t(\lambda)}\int_{\lambda_{t_{i-1}}}^{\lambda}e^{-2(\lambda-\lambda')}\nabla\mu(x_{t_{i-1}}^{\star},\lambda_{t_{i-1}})^{\mathsf{T}}\mu(x_{t(\lambda')}^{\star},\lambda')\,d\lambda' \\
&\quad-\sqrt{2}\sigma_{t(\lambda)}\nabla\mu(x_{t_{i-1}}^{\star},\lambda_{t_{i-1}})^{\mathsf{T}}\int_{\lambda_{t_{i-1}}}^{\lambda}e^{-(\lambda-\lambda')}\,d\overline{w}_{t(\lambda')} \\
&\quad+O(\delta_i^2)+O(\lVert x_{t_{i-1}}^{\star}-x_{t(\lambda)}^{\star}\rVert_2^2). \tag*{(83)}
\end{aligned} μ ( x t i − 1 ⋆ , t i − 1 ) − μ ( x t ( λ ) ⋆ , t ( λ )) = ∂ λ ∂ μ ( x t i − 1 ⋆ , t i − 1 ) ( λ t i − 1 − λ ) − ( σ t i − 1 σ t ( λ ) e − ( λ − λ t i − 1 ) − 1 ) ∇ x μ ( x t i − 1 ⋆ , λ t i − 1 ) T x t i − 1 ⋆ − 2 α t ( λ ) ∫ λ t i − 1 λ e − 2 ( λ − λ ′ ) ∇ μ ( x t i − 1 ⋆ , λ t i − 1 ) T μ ( x t ( λ ′ ) ⋆ , λ ′ ) d λ ′ − 2 σ t ( λ ) ∇ μ ( x t i − 1 ⋆ , λ t i − 1 ) T ∫ λ t i − 1 λ e − ( λ − λ ′ ) d w t ( λ ′ ) + O ( δ i 2 ) + O (∥ x t i − 1 ⋆ − x t ( λ ) ⋆ ∥ 2 2 ) . (83)
We now estimate the integral appearing in the last term of (81) . The contributions from the terms in (83) are bounded as follows. First,
∥ ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) ∂ ∂ λ μ ( x t i − 1 ⋆ , t i − 1 ) ( λ t i − 1 − λ ) d λ ∥ 2 ≤ L t ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) ( λ − λ t i − 1 ) d λ = L t ( δ i 2 − 1 4 + 1 4 e − 2 δ i ) ≤ ( a ) L t δ i 2 , \left\lVert\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-2(\lambda_{t_i}-\lambda)}\frac{\partial}{\partial\lambda}\mu(x_{t_{i-1}}^{\star},t_{i-1})(\lambda_{t_{i-1}}-\lambda)\,d\lambda\right\rVert_2
\le L_t\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-2(\lambda_{t_i}-\lambda)}(\lambda-\lambda_{t_{i-1}})\,d\lambda
= L_t\left(\frac{\delta_i}{2}-\frac{1}{4}+\frac{1}{4}e^{-2\delta_i}\right)\overset{(a)}{\le}L_t\delta_i^2, ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) ∂ λ ∂ μ ( x t i − 1 ⋆ , t i − 1 ) ( λ t i − 1 − λ ) d λ 2 ≤ L t ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) ( λ − λ t i − 1 ) d λ = L t ( 2 δ i − 4 1 + 4 1 e − 2 δ i ) ≤ ( a ) L t δ i 2 ,
where (a) holds for sufficiently small δ i ≤ δ \delta_i\le\delta δ i ≤ δ . Second,
∣ ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) ( σ t ( λ ) σ t i − 1 e − ( λ − λ t i − 1 ) − 1 ) d λ ∣ = ( b ) ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) ( 1 − σ t ( λ ) σ t i − 1 e − ( λ − λ t i − 1 ) ) d λ ≤ ( c ) 2 δ i ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) d λ = δ i ( 1 − e − 2 δ i ) ≤ 2 δ i 2 , \begin{aligned}
\left\lvert\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-2(\lambda_{t_i}-\lambda)}\left(\frac{\sigma_{t(\lambda)}}{\sigma_{t_{i-1}}}e^{-(\lambda-\lambda_{t_{i-1}})}-1\right)\,d\lambda\right\rvert
&\overset{(b)}{=}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-2(\lambda_{t_i}-\lambda)}\left(1-\frac{\sigma_{t(\lambda)}}{\sigma_{t_{i-1}}}e^{-(\lambda-\lambda_{t_{i-1}})}\right)\,d\lambda\\
&\overset{(c)}{\le}2\delta_i\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-2(\lambda_{t_i}-\lambda)}\,d\lambda\\
&=\delta_i(1-e^{-2\delta_i})\le2\delta_i^2,
\end{aligned} ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) ( σ t i − 1 σ t ( λ ) e − ( λ − λ t i − 1 ) − 1 ) d λ = ( b ) ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) ( 1 − σ t i − 1 σ t ( λ ) e − ( λ − λ t i − 1 ) ) d λ ≤ ( c ) 2 δ i ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) d λ = δ i ( 1 − e − 2 δ i ) ≤ 2 δ i 2 ,
where the equality (b) uses λ ≤ λ t i \lambda\le\lambda_{t_i} λ ≤ λ t i and σ t ( λ ) / σ t i − 1 ≤ 1 \sigma_t(\lambda)/\sigma_{t_{i-1}} \le1 σ t ( λ ) / σ t i − 1 ≤ 1 , and the inequality (c) follows from
1 − σ t ( λ ) σ t i − 1 e − ( λ − λ t i − 1 ) ≤ 1 − α t i − 1 σ t ( λ ) σ t i − 1 α t ( λ ) e − ( λ − λ t i − 1 ) = 1 − e λ t i − 1 − λ − ( λ − λ t i − 1 ) = 1 − e 2 ( λ t i − 1 − λ ) ≤ 2 ( λ − λ t i − 1 ) ≤ 2 δ i . (84) 1-\frac{\sigma_t(\lambda)}{\sigma_{t_{i-1}}}e^{-(\lambda-\lambda_{t_{i-1}})}
\le1-\frac{\alpha_{t_{i-1}}\sigma_t(\lambda)}{\sigma_{t_{i-1}}\alpha_t(\lambda)}e^{-(\lambda-\lambda_{t_{i-1}})}
=1-e^{\lambda_{t_{i-1}}-\lambda-(\lambda-\lambda_{t_{i-1}})}
=1-e^{2(\lambda_{t_{i-1}}-\lambda)}
\le2(\lambda-\lambda_{t_{i-1}})\le2\delta_i. \tag*{(84)} 1 − σ t i − 1 σ t ( λ ) e − ( λ − λ t i − 1 ) ≤ 1 − σ t i − 1 α t ( λ ) α t i − 1 σ t ( λ ) e − ( λ − λ t i − 1 ) = 1 − e λ t i − 1 − λ − ( λ − λ t i − 1 ) = 1 − e 2 ( λ t i − 1 − λ ) ≤ 2 ( λ − λ t i − 1 ) ≤ 2 δ i . (84)
Third, the integral involving the drift term is bounded by
2 ∥ ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) α t ( λ ) ∫ λ t i − 1 λ e − 2 ( λ − λ ′ ) ∇ μ ( x t i − 1 ⋆ , λ t i − 1 ) ⊤ μ ( x t ( λ ′ ) ⋆ , λ ′ ) d λ ′ d λ ∥ 2 2\left\|\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-2(\lambda_{t_i}-\lambda)}\alpha_t(\lambda)\int_{\lambda_{t_{i-1}}}^{\lambda}e^{-2(\lambda-\lambda')}\nabla\mu(x^\star_{t_{i-1}},\lambda_{t_{i-1}})^\top\mu(x^\star_{t(\lambda')},\lambda')d\lambda'd\lambda\right\|_2 2 ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) α t ( λ ) ∫ λ t i − 1 λ e − 2 ( λ − λ ′ ) ∇ μ ( x t i − 1 ⋆ , λ t i − 1 ) ⊤ μ ( x t ( λ ′ ) ⋆ , λ ′ ) d λ ′ d λ 2
≤ 2 α t i L x α t i − 1 L μ ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) ∫ λ t i − 1 λ e − 2 ( λ − λ ′ ) d λ ′ d λ = α t i L x α t i − 1 L μ ( 1 2 − 1 2 e − 2 δ i − δ i e − 2 δ i ) ≤ 2 α t i L x α t i − 1 L μ δ i 2 . \begin{aligned}
\le2\alpha_{t_i}\frac{L_x}{\alpha_{t_{i-1}}}L_\mu\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-2(\lambda_{t_i}-\lambda)}\int_{\lambda_{t_{i-1}}}^{\lambda}e^{-2(\lambda-\lambda')}d\lambda'd\lambda\\
&=\alpha_{t_i}\frac{L_x}{\alpha_{t_{i-1}}}L_\mu\left(\frac{1}{2}-\frac{1}{2}e^{-2\delta_i}-\delta_i e^{-2\delta_i}\right)
\le2\alpha_{t_i}\frac{L_x}{\alpha_{t_{i-1}}}L_\mu\delta_i^2.
\end{aligned} ≤ 2 α t i α t i − 1 L x L μ ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) ∫ λ t i − 1 λ e − 2 ( λ − λ ′ ) d λ ′ d λ = α t i α t i − 1 L x L μ ( 2 1 − 2 1 e − 2 δ i − δ i e − 2 δ i ) ≤ 2 α t i α t i − 1 L x L μ δ i 2 .
Finally, the quadratic remainder term satisfies
δ i 2 ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) d λ = δ i 2 2 ( 1 − e − 2 δ i ) ≤ δ i 3 . \delta_i^2\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-2(\lambda_{t_i}-\lambda)}d\lambda
=\frac{\delta_i^2}{2}\left(1-e^{-2\delta_i}\right)\le\delta_i^3. δ i 2 ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) d λ = 2 δ i 2 ( 1 − e − 2 δ i ) ≤ δ i 3 .
Collecting these estimates, the integral in (81) reduces to
∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) ( μ ( x t i − 1 ⋆ , λ t i − 1 ) − μ ( x t ( λ ) ⋆ , λ ) ) d λ = O ( δ i 2 ) + ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) O ( ∥ x t i − 1 ⋆ − x t ( λ ) ⋆ ∥ 2 2 ) d λ − ∇ x μ ( x t i − 1 ⋆ , λ t i − 1 ) ⊤ x t i − 1 ⋆ O ( δ i 2 ) \begin{aligned}
\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-2(\lambda_{t_i}-\lambda)}
\left(\mu(x^\star_{t_{i-1}},\lambda_{t_{i-1}})-\mu(x^\star_{t(\lambda)},\lambda)\right)d\lambda\\
&=O(\delta_i^2)+\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-2(\lambda_{t_i}-\lambda)}O(\|x^\star_{t_{i-1}}-x^\star_{t(\lambda)}\|_2^2)d\lambda
-\nabla_x\mu(x^\star_{t_{i-1}},\lambda_{t_{i-1}})^\top x^\star_{t_{i-1}}O(\delta_i^2)
\end{aligned} ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) ( μ ( x t i − 1 ⋆ , λ t i − 1 ) − μ ( x t ( λ ) ⋆ , λ ) ) d λ = O ( δ i 2 ) + ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) O ( ∥ x t i − 1 ⋆ − x t ( λ ) ⋆ ∥ 2 2 ) d λ − ∇ x μ ( x t i − 1 ⋆ , λ t i − 1 ) ⊤ x t i − 1 ⋆ O ( δ i 2 )
− 2 ∇ μ ( x t i − 1 ⋆ , λ t i − 1 ) ⊤ ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) σ t ( λ ) ∫ λ t i − 1 λ e − ( λ − λ ′ ) d w ‾ t ( λ ′ ) d λ ′ . (85) -\sqrt{2}\nabla\mu(x^\star_{t_{i-1}},\lambda_{t_{i-1}})^\top
\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-2(\lambda_{t_i}-\lambda)}\sigma_t(\lambda)
\int_{\lambda_{t_{i-1}}}^{\lambda}e^{-(\lambda-\lambda')}\,d\overline{w}_t(\lambda')d\lambda'. \tag*{(85)} − 2 ∇ μ ( x t i − 1 ⋆ , λ t i − 1 ) ⊤ ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) σ t ( λ ) ∫ λ t i − 1 λ e − ( λ − λ ′ ) d w t ( λ ′ ) d λ ′ . (85)
Here, the notation O ( δ i 2 ) O(\delta_i^2) O ( δ i 2 ) denotes a constant or a vector whose ℓ 2 \ell_2 ℓ 2 -norm is bounded by c δ i 2 c\delta_i^2 c δ i 2 , where c c c is a constant independent of the number of iterations M M M . Then we have simplified the last term in the right-hand-side of (81) .
For the second term on the right-hand-side of (81) , a direct Taylor expansion yields
α t i ( 1 − e − 2 δ i ) ( μ ( x t i − 1 , λ t i − 1 ) − μ ( x t i − 1 ⋆ , λ t i − 1 ) ) = α t i ( 1 − e − 2 δ i ) ∇ x μ ( x t i − 1 ⋆ , λ t i − 1 ) ⊤ ( x t i − 1 − x t i − 1 ⋆ ) + O ( δ i ∥ x t i − 1 − x t i − 1 ⋆ ∥ 2 2 ) , (86) \begin{aligned}
\alpha_{t_i}(1-e^{-2\delta_i})\left(\mu(x_{t_{i-1}},\lambda_{t_{i-1}})-\mu(x^\star_{t_{i-1}},\lambda_{t_{i-1}})\right) \\
&=\alpha_{t_i}(1-e^{-2\delta_i})\nabla_x\mu(x^\star_{t_{i-1}},\lambda_{t_{i-1}})^\top(x_{t_{i-1}}-x^\star_{t_{i-1}})
+O(\delta_i\|x_{t_{i-1}}-x^\star_{t_{i-1}}\|_2^2), \tag*{(86)}
\end{aligned} α t i ( 1 − e − 2 δ i ) ( μ ( x t i − 1 , λ t i − 1 ) − μ ( x t i − 1 ⋆ , λ t i − 1 ) ) = α t i ( 1 − e − 2 δ i ) ∇ x μ ( x t i − 1 ⋆ , λ t i − 1 ) ⊤ ( x t i − 1 − x t i − 1 ⋆ ) + O ( δ i ∥ x t i − 1 − x t i − 1 ⋆ ∥ 2 2 ) , (86)
where the remainder estimate follows from the bound
α t i ( 1 − e − 2 δ i ) ≤ 2 α t i δ i = O ( δ i ) . \alpha_{t_i}(1-e^{-2\delta_i})\le2\alpha_{t_i}\delta_i=O(\delta_i). α t i ( 1 − e − 2 δ i ) ≤ 2 α t i δ i = O ( δ i ) .
Substituting (85) and (86) into (81) yields the final error decomposition:
x t i − x t i ⋆ = ( σ t i σ t i − 1 e − δ i I + α t i ( 1 − e − 2 δ i ) ∇ x μ ( x t i − 1 ⋆ , λ t i − 1 ) ⊤ ) ( x t i − 1 − x t i − 1 ⋆ ) + E t i , 1 + E t i , 2 , (87) x_{t_i}-x^\star_{t_i}
=\left(\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}e^{-\delta_i}I+\alpha_{t_i}(1-e^{-2\delta_i})\nabla_x\mu(x^\star_{t_{i-1}},\lambda_{t_{i-1}})^\top\right)(x_{t_{i-1}}-x^\star_{t_{i-1}})+\mathcal{E}_{t_i,1}+\mathcal{E}_{t_i,2}, \tag*{(87)} x t i − x t i ⋆ = ( σ t i − 1 σ t i e − δ i I + α t i ( 1 − e − 2 δ i ) ∇ x μ ( x t i − 1 ⋆ , λ t i − 1 ) ⊤ ) ( x t i − 1 − x t i − 1 ⋆ ) + E t i , 1 + E t i , 2 , (87)
where the residual terms E t i , 1 \mathcal{E}_{t_i,1} E t i , 1 and E t i , 2 \mathcal{E}_{t_i,2} E t i , 2 are defined as
E t i , 1 : = O ( δ i 2 ) + ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) O ( ∥ x t i − 1 ⋆ − x t ( λ ) ⋆ ∥ 2 2 ) d λ − ∇ x μ ( x t i − 1 ⋆ , λ t i − 1 ) ⊤ x t i − 1 ⋆ O ( δ i 2 ) + O ( δ i ∥ x t i − 1 − x t i − 1 ⋆ ∥ 2 2 ) , (88) \mathcal{E}_{t_i,1}:=O(\delta_i^2)+\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-2(\lambda_{t_i}-\lambda)}O(\|x^\star_{t_{i-1}}-x^\star_{t(\lambda)}\|_2^2)d\lambda-\nabla_x\mu(x^\star_{t_{i-1}},\lambda_{t_{i-1}})^\top x^\star_{t_{i-1}}O(\delta_i^2)+O(\delta_i\|x_{t_{i-1}}-x^\star_{t_{i-1}}\|_2^2), \tag*{(88)} E t i , 1 := O ( δ i 2 ) + ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) O ( ∥ x t i − 1 ⋆ − x t ( λ ) ⋆ ∥ 2 2 ) d λ − ∇ x μ ( x t i − 1 ⋆ , λ t i − 1 ) ⊤ x t i − 1 ⋆ O ( δ i 2 ) + O ( δ i ∥ x t i − 1 − x t i − 1 ⋆ ∥ 2 2 ) , (88)
E t i , 2 : = − 2 2 α t i ∇ μ ( x t i − 1 ⋆ , λ t i − 1 ) ⊤ ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) σ t ( λ ) ∫ λ t i − 1 λ e − ( λ − λ ′ ) d w ‾ t ( λ ′ ) d λ ′ . (89) \mathcal{E}_{t_i,2}:=-2\sqrt{2}\alpha_{t_i}\nabla\mu(x^\star_{t_{i-1}},\lambda_{t_{i-1}})^\top
\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-2(\lambda_{t_i}-\lambda)}\sigma_t(\lambda)
\int_{\lambda_{t_{i-1}}}^{\lambda}e^{-(\lambda-\lambda')}\,d\overline{w}_t(\lambda')d\lambda'. \tag*{(89)} E t i , 2 := − 2 2 α t i ∇ μ ( x t i − 1 ⋆ , λ t i − 1 ) ⊤ ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) σ t ( λ ) ∫ λ t i − 1 λ e − ( λ − λ ′ ) d w t ( λ ′ ) d λ ′ . (89)
Here E t i , 1 \mathcal{E}_{t_i,1} E t i , 1 collects the approximation errors, while E t i , 2 \mathcal{E}_{t_i,2} E t i , 2 captures the stochastic integral contribution.
Step 2: recursion for the expectation E [ ∥ x t i − x t i ⋆ ∥ ] \mathbb{E}[\lVert x_{t_i}-x^\star_{t_i}\rVert] E [∥ x t i − x t i ⋆ ∥] . For notational convenience, we introduce the matrices A t i A_{t_i} A t i , which depend on x t i − 1 ⋆ x^\star_{t_{i-1}} x t i − 1 ⋆ , as
A t i : = σ t i σ t i − 1 e − δ i I d + α t i ( 1 − e − 2 δ i ) ∇ x μ ( x t i − 1 ⋆ , λ t i − 1 ) ⊤ (90) A_{t_i}:=\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}e^{-\delta_i}I_d+\alpha_{t_i}(1-e^{-2\delta_i})\nabla_x\mu(x^\star_{t_{i-1}},\lambda_{t_{i-1}})^\top \tag*{(90)} A t i := σ t i − 1 σ t i e − δ i I d + α t i ( 1 − e − 2 δ i ) ∇ x μ ( x t i − 1 ⋆ , λ t i − 1 ) ⊤ (90)
This definition allows us to rewrite (87) compactly. By recursion, we have
x t M − x t M ⋆ = A t M ( x t M − 1 ⋆ ) ( x t M − 1 − x t M − 1 ⋆ ) + E t M , 1 + E t M , 2 = ∑ m = 1 M ∏ j = m + 1 M A t j ( E t m , 1 + E t m , 2 ) , (91) x_{t_M}-x^\star_{t_M}=A_{t_M}(x^\star_{t_{M-1}})(x_{t_{M-1}}-x^\star_{t_{M-1}})+\mathcal{E}_{t_M,1}+\mathcal{E}_{t_M,2}=\sum_{m=1}^{M}\prod_{j=m+1}^{M}A_{t_j}\left(\mathcal{E}_{t_m,1}+\mathcal{E}_{t_m,2}\right), \tag*{(91)} x t M − x t M ⋆ = A t M ( x t M − 1 ⋆ ) ( x t M − 1 − x t M − 1 ⋆ ) + E t M , 1 + E t M , 2 = m = 1 ∑ M j = m + 1 ∏ M A t j ( E t m , 1 + E t m , 2 ) , (91)
where we adopt the convention that the empty product equals the identity matrix. Taking expectations and applying the triangle inequality gives the recursive bound
E [ ∥ x t M − x t M ⋆ ∥ 2 ] ≤ ∑ m = 1 M E [ ∏ j = m + 1 M ∥ A t j ∥ o p ∥ E t m , 1 ∥ 2 ] + E [ ∥ ∑ m = 1 M ∏ j = m + 1 M A t j E t m , 2 ∥ 2 ] . (92) \mathbb{E}[\lVert x_{t_M}-x^\star_{t_M}\rVert_2]\leq\sum_{m=1}^{M}\mathbb{E}\left[\prod_{j=m+1}^{M}\lVert A_{t_j}\rVert_{\mathrm{op}}\lVert\mathcal{E}_{t_m,1}\rVert_2\right]+\mathbb{E}\left[\left\lVert\sum_{m=1}^{M}\prod_{j=m+1}^{M}A_{t_j}\mathcal{E}_{t_m,2}\right\rVert_2\right]. \tag*{(92)} E [∥ x t M − x t M ⋆ ∥ 2 ] ≤ m = 1 ∑ M E [ j = m + 1 ∏ M ∥ A t j ∥ op ∥ E t m , 1 ∥ 2 ] + E m = 1 ∑ M j = m + 1 ∏ M A t j E t m , 2 2 . (92)
To control the product of operator norms, we first estimate a single factor. For each i i i , we have
∥ A t i ( x t i − 1 ⋆ ) ∥ o p ≤ σ t i σ t i − 1 e − δ i + α t i ( 1 − e − 2 δ i ) L x α t i − 1 ≤ ( a ) 1 + L x σ t i σ t i − 1 e − δ i ( e 2 δ i − 1 ) ≤ ( b ) 1 + 4 L x δ i , (93) \lVert A_{t_i}(x^\star_{t_{i-1}})\rVert_{\mathrm{op}}\leq\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}e^{-\delta_i}+\alpha_{t_i}(1-e^{-2\delta_i})\frac{L_x}{\alpha_{t_{i-1}}}\overset{(a)}{\leq}1+L_x\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}e^{-\delta_i}(e^{2\delta_i}-1)\overset{(b)}{\leq}1+4L_x\delta_i, \tag*{(93)} ∥ A t i ( x t i − 1 ⋆ ) ∥ op ≤ σ t i − 1 σ t i e − δ i + α t i ( 1 − e − 2 δ i ) α t i − 1 L x ≤ ( a ) 1 + L x σ t i − 1 σ t i e − δ i ( e 2 δ i − 1 ) ≤ ( b ) 1 + 4 L x δ i , (93)
where the second inequality (a) follows from σ t i ≤ σ t i − 1 \sigma_{t_i}\leq\sigma_{t_{i-1}} σ t i ≤ σ t i − 1 and the last inequality (b) uses e − δ i ( e 2 δ i − 1 ) ≤ 4 δ i e^{-\delta_i}(e^{2\delta_i}-1)\leq4\delta_i e − δ i ( e 2 δ i − 1 ) ≤ 4 δ i for δ i ≤ 1 / 2 \delta_i\leq1/2 δ i ≤ 1/2 . Consequently, the cumulative product satisfies
∥ ∏ j = m + 1 M A t j ∥ ≤ ∏ j = m + 1 M ∥ A t j ∥ o p ≤ ∏ j = m + 1 M ( 1 + 4 L x δ j ) ≤ exp ( 4 L x ∑ j = m + 1 M δ j ) = O ( 1 ) , (94) \left\lVert\prod_{j=m+1}^{M}A_{t_j}\right\rVert\leq\prod_{j=m+1}^{M}\lVert A_{t_j}\rVert_{\mathrm{op}}\leq\prod_{j=m+1}^{M}(1+4L_x\delta_j)
\leq\exp\left(4L_x\sum_{j=m+1}^{M}\delta_j\right)=O(1), \tag*{(94)} j = m + 1 ∏ M A t j ≤ j = m + 1 ∏ M ∥ A t j ∥ op ≤ j = m + 1 ∏ M ( 1 + 4 L x δ j ) ≤ exp ( 4 L x j = m + 1 ∑ M δ j ) = O ( 1 ) , (94)
where the last equality uses the fact that ∑ j = m + 1 M δ j ≤ ∑ j = 1 M δ j = O ( 1 ) \sum_{j=m+1}^{M}\delta_j\leq\sum_{j=1}^{M}\delta_j=O(1) ∑ j = m + 1 M δ j ≤ ∑ j = 1 M δ j = O ( 1 ) . Substituting this bound into (92) yields
E [ ∥ x t M − x t M ⋆ ∥ 2 ] ≲ ∑ m = 1 M E [ ∥ E t m , 1 ∥ 2 ] + E [ ∥ ∑ m = 1 M ∏ j = m + 1 M A t j E t m , 2 ∥ 2 ] . (95) \mathbb{E}[\lVert x_{t_M}-x^\star_{t_M}\rVert_2]\lesssim\sum_{m=1}^{M}\mathbb{E}[\lVert\mathcal{E}_{t_m,1}\rVert_2]+\mathbb{E}\left[\left\lVert\sum_{m=1}^{M}\prod_{j=m+1}^{M}A_{t_j}\mathcal{E}_{t_m,2}\right\rVert_2\right]. \tag*{(95)} E [∥ x t M − x t M ⋆ ∥ 2 ] ≲ m = 1 ∑ M E [∥ E t m , 1 ∥ 2 ] + E m = 1 ∑ M j = m + 1 ∏ M A t j E t m , 2 2 . (95)
In the following two steps, we will bound these two contributions separately.
Step 3: Analysis of E t m , 2 \mathcal{E}_{t_m,2} E t m , 2 . From its definition in (89) , the stochastic term E t m , 2 \mathcal{E}_{t_m,2} E t m , 2 can be rewritten as
E t m , 2 = − 2 2 α t m e − 2 λ t m ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ ∫ λ t m − 1 λ t m ∫ λ ′ λ t m e λ σ t ( λ ) e λ ′ d λ d ω ˉ t ( λ ′ ) = − 2 2 α t m e − 2 λ t m ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ ∫ λ t m − 1 λ t m e λ ′ [ ∫ λ ′ λ t m α t ( λ ) d λ ] d ω ˉ t ( λ ′ ) = : ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ z ˉ t m , (96) \begin{aligned}
\mathcal{E}_{t_m,2}=-2\sqrt{2\alpha_{t_m}}e^{-2\lambda_{t_m}}\nabla\mu(x^\star_{t_{m-1}},\lambda_{t_{m-1}})^\top\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}\int_{\lambda'}^{\lambda_{t_m}}e^{\lambda}\sigma_t(\lambda)e^{\lambda'}\,\mathrm{d}\lambda\,\mathrm{d}\bar{\omega}_t(\lambda') \\
&=-2\sqrt{2\alpha_{t_m}}e^{-2\lambda_{t_m}}\nabla\mu(x^\star_{t_{m-1}},\lambda_{t_{m-1}})^\top\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}e^{\lambda'}\left[\int_{\lambda'}^{\lambda_{t_m}}\alpha_t(\lambda)\,\mathrm{d}\lambda\right]\mathrm{d}\bar{\omega}_t(\lambda') \\
&=:\nabla\mu(x^\star_{t_{m-1}},\lambda_{t_{m-1}})^\top\bar{z}_{t_m}, \tag*{(96)}
\end{aligned} E t m , 2 = − 2 2 α t m e − 2 λ t m ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ ∫ λ t m − 1 λ t m ∫ λ ′ λ t m e λ σ t ( λ ) e λ ′ d λ d ω ˉ t ( λ ′ ) = − 2 2 α t m e − 2 λ t m ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ ∫ λ t m − 1 λ t m e λ ′ [ ∫ λ ′ λ t m α t ( λ ) d λ ] d ω ˉ t ( λ ′ ) =: ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ z ˉ t m , (96)
where the noise vector z ˉ t m \bar{z}_{t_m} z ˉ t m depends on the Brownian motion increment over the interval [ λ t m − 1 , λ t m ] [\lambda_{t_{m-1}},\lambda_{t_m}] [ λ t m − 1 , λ t m ] , and is defined as
z ˉ t m : = − 2 2 α t m e − 2 λ t m ∫ λ t m − 1 λ t m e λ ′ [ ∫ λ ′ λ t m α t ( λ ) d λ ] d ω ˉ t ( λ ′ ) . (97) \bar{z}_{t_m}:=-2\sqrt{2\alpha_{t_m}}e^{-2\lambda_{t_m}}\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}e^{\lambda'}\left[\int_{\lambda'}^{\lambda_{t_m}}\alpha_t(\lambda)\,\mathrm{d}\lambda\right]\mathrm{d}\bar{\omega}_t(\lambda'). \tag*{(97)} z ˉ t m := − 2 2 α t m e − 2 λ t m ∫ λ t m − 1 λ t m e λ ′ [ ∫ λ ′ λ t m α t ( λ ) d λ ] d ω ˉ t ( λ ′ ) . (97)
Define the cumulative propagation matrices by
P 0 = I d , P i = A t i P i − 1 = ∏ j = 1 i A t j , i ≥ 1. (98) P_0 = I_d,\qquad P_i = A_{t_i}P_{i-1} = \prod_{j=1}^{i} A_{t_j},\quad i \ge1. \tag*{(98)} P 0 = I d , P i = A t i P i − 1 = j = 1 ∏ i A t j , i ≥ 1. (98)
Note that
e − 2 δ i ≤ σ t i σ t i − 1 e − δ i ≤ 1 , ∥ A t i − I d ∥ o p ≤ ( 2 + 4 L x ) δ i . e^{-2\delta_i} \le\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}e^{-\delta_i} \le1,\qquad\lVert A_{t_i}-I_d\rVert_{\mathrm{op}} \le(2+4L_x)\delta_i. e − 2 δ i ≤ σ t i − 1 σ t i e − δ i ≤ 1 , ∥ A t i − I d ∥ op ≤ ( 2 + 4 L x ) δ i .
Here the first inequality uses α t i ≥ α t i − 1 \alpha_{t_i} \ge\alpha_{t_{i-1}} α t i ≥ α t i − 1 , and the second follows from the same Jacobian bound as in (93) . Writing C A = 2 + 4 L x C_A=2+4L_x C A = 2 + 4 L x , for sufficiently small δ \delta δ we have
∥ A t i − 1 ∥ o p ≤ ( 1 − C A δ i ) − 1 ≤ exp ( 2 C A δ i ) . \lVert A_{t_i}^{-1}\rVert_{\mathrm{op}} \le(1-C_A\delta_i)^{-1} \le\exp(2C_A\delta_i). ∥ A t i − 1 ∥ op ≤ ( 1 − C A δ i ) − 1 ≤ exp ( 2 C A δ i ) .
Together with (94) and ∑ i δ i = O ( 1 ) \sum_i\delta_i=O(1) ∑ i δ i = O ( 1 ) , this gives a deterministic constant C C C , independent of M M M , such that
max 0 ≤ i ≤ M ( ∥ P i ∥ o p + ∥ P i − 1 ∥ o p ) ≤ C . \max_{0\le i\le M}\left(\lVert P_i\rVert_{\mathrm{op}}+\lVert P_i^{-1}\rVert_{\mathrm{op}}\right)\le C. 0 ≤ i ≤ M max ( ∥ P i ∥ op + ∥ P i − 1 ∥ op ) ≤ C .
Then the factor in the right-hand-side of (95) becomes
∑ m = 1 M ∏ j = m + 1 M A t j E t m , 2 = ∑ m = 1 M A t M ⋯ A t m + 1 E t m , 2 = P M ∑ m = 1 M P m − 1 E t m , 2 . (99) \sum_{m=1}^{M}\prod_{j=m+1}^{M}A_{t_j}\mathcal{E}_{t_m,2}
=\sum_{m=1}^{M}A_{t_M}\cdots A_{t_{m+1}}\mathcal{E}_{t_m,2}
=P_M\sum_{m=1}^{M}P_m^{-1}\mathcal{E}_{t_m,2}. \tag*{(99)} m = 1 ∑ M j = m + 1 ∏ M A t j E t m , 2 = m = 1 ∑ M A t M ⋯ A t m + 1 E t m , 2 = P M m = 1 ∑ M P m − 1 E t m , 2 . (99)
Substituting (96) into (99) yields
∑ m = 1 M ∏ j = m + 1 M A t j E t m , 2 = P M ∑ m = 1 M P m − 1 ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ z ‾ t m . (100) \sum_{m=1}^{M}\prod_{j=m+1}^{M}A_{t_j}\mathcal{E}_{t_m,2}
=P_M\sum_{m=1}^{M}P_m^{-1}\nabla\mu(x^{\star}_{t_{m-1}},\lambda_{t_{m-1}})^{\top}\overline{z}_{t_m}. \tag*{(100)} m = 1 ∑ M j = m + 1 ∏ M A t j E t m , 2 = P M m = 1 ∑ M P m − 1 ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ z t m . (100)
To analyze this sum, define the filtration generated by the initial state and the Brownian increments up to the current log-SNR:
F t i = σ ( x t 0 ⋆ , w ‾ t ( λ ) − w ‾ t 0 : λ t 0 ≤ λ ≤ λ t i ) . (101) \mathcal{F}_{t_i}=\sigma\left(x^{\star}_{t_0},\overline{w}_{t}(\lambda)-\overline{w}_{t_0}:\lambda_{t_0}\le\lambda\le\lambda_{t_i}\right). \tag*{(101)} F t i = σ ( x t 0 ⋆ , w t ( λ ) − w t 0 : λ t 0 ≤ λ ≤ λ t i ) . (101)
The vector z ‾ t m \overline{z}_{t_m} z t m is independent of F t m − 1 \mathcal{F}_{t_{m-1}} F t m − 1 and has zero mean. Its conditional second moment satisfies
E [ ∥ z ‾ t m ∥ 2 2 ∣ F t m − 1 ] ≤ 8 α t m d e − 4 λ t m ∫ λ t m − 1 λ t m e 2 λ ′ [ ∫ λ ′ λ t m α t ( λ ) d λ ] 2 d λ ′ \mathbb{E}[\lVert\overline{z}_{t_m}\rVert_2^2\mid\mathcal{F}_{t_{m-1}}]
\le8\alpha_{t_m}d e^{-4\lambda_{t_m}}
\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}e^{2\lambda'}\left[\int_{\lambda'}^{\lambda_{t_m}}\alpha_t(\lambda)\,\mathrm{d}\lambda\right]^2\mathrm{d}\lambda' E [∥ z t m ∥ 2 2 ∣ F t m − 1 ] ≤ 8 α t m d e − 4 λ t m ∫ λ t m − 1 λ t m e 2 λ ′ [ ∫ λ ′ λ t m α t ( λ ) d λ ] 2 d λ ′
≤ 8 α t m d e − 4 λ t m ∫ λ t m − 1 λ t m e 2 λ ′ ( λ t m − λ ′ ) ∫ λ ′ λ t m α t 2 ( λ ) d λ d λ ′ \le8\alpha_{t_m}d e^{-4\lambda_{t_m}}
\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}e^{2\lambda'}(\lambda_{t_m}-\lambda')\int_{\lambda'}^{\lambda_{t_m}}\alpha_t^2(\lambda)\,\mathrm{d}\lambda\,\mathrm{d}\lambda' ≤ 8 α t m d e − 4 λ t m ∫ λ t m − 1 λ t m e 2 λ ′ ( λ t m − λ ′ ) ∫ λ ′ λ t m α t 2 ( λ ) d λ d λ ′
≤ 8 α t m d e − 4 λ t m ∫ λ t m − 1 λ t m e 2 λ ′ δ m 2 α t m 2 d λ ′ \le8\alpha_{t_m}d e^{-4\lambda_{t_m}}
\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}e^{2\lambda'}\delta_m^2\alpha_{t_m}^2\,\mathrm{d}\lambda' ≤ 8 α t m d e − 4 λ t m ∫ λ t m − 1 λ t m e 2 λ ′ δ m 2 α t m 2 d λ ′
≤ 8 d δ m 2 α t m 3 e − 4 λ t m ( e 2 λ t m − e 2 λ t m − 1 ) ≤ 16 d δ m 3 α t m 3 e − 2 λ t m = O ( δ m 3 ) , (102) \le8d\delta_m^2\alpha_{t_m}^3e^{-4\lambda_{t_m}}\left(e^{2\lambda_{t_m}}-e^{2\lambda_{t_{m-1}}}\right)
\le16d\delta_m^3\alpha_{t_m}^3e^{-2\lambda_{t_m}}
=O(\delta_m^3), \tag*{(102)} ≤ 8 d δ m 2 α t m 3 e − 4 λ t m ( e 2 λ t m − e 2 λ t m − 1 ) ≤ 16 d δ m 3 α t m 3 e − 2 λ t m = O ( δ m 3 ) , (102)
where the second inequality follows from Jensen’s inequality, and the third uses the monotonicity α t ( λ ) ≤ α t m \alpha_t(\lambda)\le\alpha_{t_m} α t ( λ ) ≤ α t m .
Since P m − 1 P_m^{-1} P m − 1 and ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) \nabla\mu(x^{\star}_{t_{m-1}},\lambda_{t_{m-1}}) ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) are F t m − 1 \mathcal{F}_{t_{m-1}} F t m − 1 -measurable, and (102) , the expected norm squares of the vector ∑ m = 1 M P m − 1 ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ z ‾ t m \sum_{m=1}^{M}P_m^{-1}\nabla\mu(x^{\star}_{t_{m-1}},\lambda_{t_{m-1}})^{\top}\overline{z}_{t_m} ∑ m = 1 M P m − 1 ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ z t m can be computed by:
E [ ∥ ∑ m = 1 M P m − 1 ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ z ‾ t m ∥ 2 2 ] = ∑ m = 1 M E [ ∥ P m − 1 ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ z ‾ t m ∥ 2 2 ] \begin{aligned}
\mathbb{E}\left[\left\lVert\sum_{m=1}^{M}P_m^{-1}\nabla\mu(x^{\star}_{t_{m-1}},\lambda_{t_{m-1}})^{\top}\overline{z}_{t_m}\right\rVert_2^2\right] \\
&=\sum_{m=1}^{M}\mathbb{E}\left[\left\lVert P_m^{-1}\nabla\mu(x^{\star}_{t_{m-1}},\lambda_{t_{m-1}})^{\top}\overline{z}_{t_m}\right\rVert_2^2\right]
\end{aligned} E m = 1 ∑ M P m − 1 ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ z t m 2 2 = m = 1 ∑ M E [ P m − 1 ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ z t m 2 2 ]
+ 2 ∑ n > m E [ ⟨ P n − 1 ∇ μ ( x t n − 1 ⋆ , λ t n − 1 ) ⊤ z ˉ t n , P m − 1 ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ z ˉ t m ⟩ ] ≤ C ∑ m = 1 M δ m 3 = O ( δ 2 ) , (103) +2\sum_{n>m}\mathbb{E}\left[\left\langle P_n^{-1}\nabla\mu(x_{t_{n-1}}^\star,\lambda_{t_{n-1}})^\top\bar{z}_{t_n},P_m^{-1}\nabla\mu(x_{t_{m-1}}^\star,\lambda_{t_{m-1}})^\top\bar{z}_{t_m}\right\rangle\right]
\le C\sum_{m=1}^{M}\delta_m^3=O(\delta^2), \tag*{(103)} + 2 n > m ∑ E [ ⟨ P n − 1 ∇ μ ( x t n − 1 ⋆ , λ t n − 1 ) ⊤ z ˉ t n , P m − 1 ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ z ˉ t m ⟩ ] ≤ C m = 1 ∑ M δ m 3 = O ( δ 2 ) , (103)
where the last inequality uses (102) and
E [ ∥ P m − 1 ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ z ˉ t m ∥ 2 2 ] = E [ E [ ∥ P m − 1 ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ z ˉ t m ∥ 2 2 | F t m − 1 ] ] ≤ C E [ E [ ∥ z ˉ t m ∥ 2 2 | F t m − 1 ] ] = O ( δ m 3 ) , \mathbb{E}\left[\left\|P_m^{-1}\nabla\mu(x_{t_{m-1}}^\star,\lambda_{t_{m-1}})^\top\bar{z}_{t_m}\right\|_2^2\right]
=\mathbb{E}\left[\mathbb{E}\left[\left\|P_m^{-1}\nabla\mu(x_{t_{m-1}}^\star,\lambda_{t_{m-1}})^\top\bar{z}_{t_m}\right\|_2^2\middle|\mathcal{F}_{t_{m-1}}\right]\right]
\le C\mathbb{E}\left[\mathbb{E}\left[\|\bar{z}_{t_m}\|_2^2\middle|\mathcal{F}_{t_{m-1}}\right]\right]=O(\delta_m^3), E [ P m − 1 ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ z ˉ t m 2 2 ] = E [ E [ P m − 1 ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ z ˉ t m 2 2 F t m − 1 ] ] ≤ C E [ E [ ∥ z ˉ t m ∥ 2 2 F t m − 1 ] ] = O ( δ m 3 ) ,
and the fact that the cross terms vanish due to the independence of z ˉ t n \bar{z}_{t_n} z ˉ t n and F t n − 1 \mathcal{F}_{t_{n-1}} F t n − 1 . Finally, combining (99) and (100) with Jensen’s inequality yields the required bound for the second term in (95) :
E ∥ ∑ m = 1 M ∏ j = m + 1 M A t j E t m , 2 ∥ 2 ≲ ( E ∥ ∑ m = 1 M P m − 1 ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ z ˉ t m ∥ 2 2 ) 1 / 2 = O ( δ ) . (104) \mathbb{E}\left\|\sum_{m=1}^{M}\prod_{j=m+1}^{M}A_{t_j}\mathcal{E}_{t_m,2}\right\|_2
\lesssim\left(\mathbb{E}\left\|\sum_{m=1}^{M}P_m^{-1}\nabla\mu(x_{t_{m-1}}^\star,\lambda_{t_{m-1}})^\top\bar{z}_{t_m}\right\|_2^2\right)^{1/2}=O(\delta). \tag*{(104)} E m = 1 ∑ M j = m + 1 ∏ M A t j E t m , 2 2 ≲ E m = 1 ∑ M P m − 1 ∇ μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ z ˉ t m 2 2 1/2 = O ( δ ) . (104)
For later use, we also estimate E [ ∥ x t m − 1 ⋆ − x t ( λ ) ⋆ ∥ 2 2 ] \mathbb{E}[\|x_{t_{m-1}}^\star-x_{t(\lambda)}^\star\|_2^2] E [ ∥ x t m − 1 ⋆ − x t ( λ ) ⋆ ∥ 2 2 ] . From (31) , we have
x t ( λ ) ⋆ − x t m − 1 ⋆ = ( σ t ( λ ) σ t m − 1 e − ( λ − λ t m − 1 ) − 1 ) x t m − 1 ⋆ + 2 α t ( λ ) ∫ λ t m − 1 λ e − 2 ( λ − λ ′ ) μ ( x t ( λ ′ ) ⋆ , λ ′ ) d λ ′ + 2 σ t ( λ ) ∫ λ t m − 1 λ e − ( λ − λ ′ ) d w ˉ t ( λ ′ ) . (105) x_{t(\lambda)}^\star-x_{t_{m-1}}^\star
=\left(\frac{\sigma_{t(\lambda)}}{\sigma_{t_{m-1}}}e^{-(\lambda-\lambda_{t_{m-1}})}-1\right)x_{t_{m-1}}^\star
+2\alpha_{t(\lambda)}\int_{\lambda_{t_{m-1}}}^{\lambda}e^{-2(\lambda-\lambda')}\mu(x_{t(\lambda')}^\star,\lambda')\mathrm{d}\lambda'
+\sqrt{2}\sigma_{t(\lambda)}\int_{\lambda_{t_{m-1}}}^{\lambda}e^{-(\lambda-\lambda')}\mathrm{d}\bar{w}_{t(\lambda')}. \tag*{(105)} x t ( λ ) ⋆ − x t m − 1 ⋆ = ( σ t m − 1 σ t ( λ ) e − ( λ − λ t m − 1 ) − 1 ) x t m − 1 ⋆ + 2 α t ( λ ) ∫ λ t m − 1 λ e − 2 ( λ − λ ′ ) μ ( x t ( λ ′ ) ⋆ , λ ′ ) d λ ′ + 2 σ t ( λ ) ∫ λ t m − 1 λ e − ( λ − λ ′ ) d w ˉ t ( λ ′ ) . (105)
Using the coefficient bound in (84) and Assumption A1, we obtain
E [ ∥ x t ( λ ) ⋆ − x t m − 1 ⋆ ∥ 2 2 ] ≲ δ m 2 E [ ∥ x t m − 1 ⋆ ∥ 2 2 ] + α t ( λ ) 2 L μ 2 ( ∫ λ t m − 1 λ e − 2 ( λ − λ ′ ) d λ ′ ) 2 + E [ ∥ 2 σ t ( λ ) ∫ λ t m − 1 λ e − ( λ − λ ′ ) d w ˉ t ( λ ′ ) ∥ 2 2 ] . (106) \mathbb{E}\left[\|x_{t(\lambda)}^\star-x_{t_{m-1}}^\star\|_2^2\right]
\lesssim\delta_m^2\mathbb{E}\left[\|x_{t_{m-1}}^\star\|_2^2\right]
+\alpha_{t(\lambda)}^2L_\mu^2\left(\int_{\lambda_{t_{m-1}}}^{\lambda}e^{-2(\lambda-\lambda')}\mathrm{d}\lambda'\right)^2
+\mathbb{E}\left[\left\|\sqrt{2}\sigma_{t(\lambda)}\int_{\lambda_{t_{m-1}}}^{\lambda}e^{-(\lambda-\lambda')}\mathrm{d}\bar{w}_{t(\lambda')}\right\|_2^2\right]. \tag*{(106)} E [ ∥ x t ( λ ) ⋆ − x t m − 1 ⋆ ∥ 2 2 ] ≲ δ m 2 E [ ∥ x t m − 1 ⋆ ∥ 2 2 ] + α t ( λ ) 2 L μ 2 ( ∫ λ t m − 1 λ e − 2 ( λ − λ ′ ) d λ ′ ) 2 + E 2 σ t ( λ ) ∫ λ t m − 1 λ e − ( λ − λ ′ ) d w ˉ t ( λ ′ ) 2 2 . (106)
The three terms are bounded as follows:
∫ λ t m − 1 λ e − 2 ( λ − λ ′ ) d λ ′ = 1 2 ( 1 − e − 2 ( λ − λ t m − 1 ) ) ≤ λ − λ t m − 1 ≤ δ m , \int_{\lambda_{t_{m-1}}}^{\lambda}e^{-2(\lambda-\lambda')}\mathrm{d}\lambda'
=\frac{1}{2}\left(1-e^{-2(\lambda-\lambda_{t_{m-1}})}\right)
\le\lambda-\lambda_{t_{m-1}}\le\delta_m, ∫ λ t m − 1 λ e − 2 ( λ − λ ′ ) d λ ′ = 2 1 ( 1 − e − 2 ( λ − λ t m − 1 ) ) ≤ λ − λ t m − 1 ≤ δ m ,
E [ ∥ 2 σ t ( λ ) ∫ λ t m − 1 λ e − ( λ − λ ′ ) d w ˉ t ( λ ′ ) ∥ 2 2 ] = 2 d σ t ( λ ) 2 ∫ λ t m − 1 λ e − 2 ( λ − λ ′ ) d λ ′ ≤ 2 d σ t ( λ ) 2 δ m . \mathbb{E}\left[\left\|\sqrt{2}\sigma_{t(\lambda)}\int_{\lambda_{t_{m-1}}}^{\lambda}e^{-(\lambda-\lambda')}\mathrm{d}\bar{w}_{t(\lambda')}\right\|_2^2\right]
=2d\sigma_{t(\lambda)}^2\int_{\lambda_{t_{m-1}}}^{\lambda}e^{-2(\lambda-\lambda')}\mathrm{d}\lambda'
\le2d\sigma_{t(\lambda)}^2\delta_m. E 2 σ t ( λ ) ∫ λ t m − 1 λ e − ( λ − λ ′ ) d w ˉ t ( λ ′ ) 2 2 = 2 d σ t ( λ ) 2 ∫ λ t m − 1 λ e − 2 ( λ − λ ′ ) d λ ′ ≤ 2 d σ t ( λ ) 2 δ m .
Together with the moment bound E [ ∥ x t m − 1 ⋆ ∥ 2 2 ] = O ( 1 ) \mathbb{E}[\|x_{t_{m-1}}^\star\|_2^2]=O(1) E [ ∥ x t m − 1 ⋆ ∥ 2 2 ] = O ( 1 ) from (80) , we conclude
E [ ∥ x t ( λ ) ⋆ − x t m − 1 ⋆ ∥ 2 2 ] ≲ δ m 2 + α t ( λ ) 2 L μ 2 δ m + 2 d σ t ( λ ) 2 δ m = O ( δ m ) . (107) \mathbb{E}\left[\|x_{t(\lambda)}^\star-x_{t_{m-1}}^\star\|_2^2\right]
\lesssim\delta_m^2+\alpha_{t(\lambda)}^2L_\mu^2\delta_m+2d\sigma_{t(\lambda)}^2\delta_m=O(\delta_m). \tag*{(107)} E [ ∥ x t ( λ ) ⋆ − x t m − 1 ⋆ ∥ 2 2 ] ≲ δ m 2 + α t ( λ ) 2 L μ 2 δ m + 2 d σ t ( λ ) 2 δ m = O ( δ m ) . (107)
Step 4: Analysis of E t m , 1 \mathcal{E}_{t_m,1} E t m , 1 . We now estimate the first term on the right-hand-side of (95) . From the definition of ∥ E t m , 1 ∥ 2 \|\mathcal{E}_{t_m,1}\|_2 ∥ E t m , 1 ∥ 2 in (88) , we have
E [ ∥ E t m , 1 ∥ 2 ] ≲ δ m 2 + ∫ λ t m − 1 λ t m e − 2 ( λ t m − λ ) E [ ∥ x t m − 1 ⋆ − x t ( λ ) ⋆ ∥ 2 2 ] d λ + δ m 2 E [ ∥ ∇ x μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ x t m − 1 ⋆ ∥ 2 ] + δ m E [ ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] . (108) \mathbb{E}[\|\mathcal{E}_{t_m,1}\|_2]\lesssim\delta_m^2+\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}e^{-2(\lambda_{t_m}-\lambda)}\mathbb{E}[\|x_{t_{m-1}}^\star-x_{t(\lambda)}^\star\|_2^2]\mathrm{d}\lambda
+\delta_m^2\mathbb{E}[\|\nabla_x\mu(x_{t_{m-1}}^\star,\lambda_{t_{m-1}})^\top x_{t_{m-1}}^\star\|_2]
+\delta_m\mathbb{E}[\|x_{t_{m-1}}-x_{t_{m-1}}^\star\|_2^2]. \tag*{(108)} E [ ∥ E t m , 1 ∥ 2 ] ≲ δ m 2 + ∫ λ t m − 1 λ t m e − 2 ( λ t m − λ ) E [ ∥ x t m − 1 ⋆ − x t ( λ ) ⋆ ∥ 2 2 ] d λ + δ m 2 E [ ∥ ∇ x μ ( x t m − 1 ⋆ , λ t m − 1 ) ⊤ x t m − 1 ⋆ ∥ 2 ] + δ m E [ ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] . (108)
We bound each term on the right-hand side of (108) as follows. First, from (107) , we have E [ ∥ x t m − 1 ⋆ − x t ( λ ) ⋆ ∥ 2 2 ] = O ( δ m ) \mathbb{E}[\|x_{t_{m-1}}^\star-x_{t(\lambda)}^\star\|_2^2]=O(\delta_m) E [ ∥ x t m − 1 ⋆ − x t ( λ ) ⋆ ∥ 2 2 ] = O ( δ m ) . Consequently,
∫ λ t m − 1 λ t m e − 2 ( λ t m − λ ) E [ ∥ x t m − 1 ⋆ − x t ( λ ) ⋆ ∥ 2 2 ] d λ = O ( δ m ) 1 2 ( 1 − e − 2 δ m ) = O ( δ m 2 ) . (109) \int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}e^{-2(\lambda_{t_m}-\lambda)}\mathbb{E}[\|x_{t_{m-1}}^\star-x_{t(\lambda)}^\star\|_2^2]\mathrm{d}\lambda
=O(\delta_m)\frac{1}{2}(1-e^{-2\delta_m})=O(\delta_m^2). \tag*{(109)} ∫ λ t m − 1 λ t m e − 2 ( λ t m − λ ) E [ ∥ x t m − 1 ⋆ − x t ( λ ) ⋆ ∥ 2 2 ] d λ = O ( δ m ) 2 1 ( 1 − e − 2 δ m ) = O ( δ m 2 ) . (109)
Second, by Assumption A2 and the moment bound (80) ,
δ m 2 E [ ∥ ∇ x μ ( x t m − 1 ⋆ , λ t m − 1 ) T x t m − 1 ⋆ ∥ 2 2 ] ≤ L x δ m 2 α t m − 1 E [ ∥ x t m − 1 ⋆ ∥ 2 2 ] = O ( δ m 2 ) . (110) \delta_m^2 \mathbb{E}\left[\left\|\nabla_x\mu\left(x^\star_{t_{m-1}},\lambda_{t_{m-1}}\right)^\mathsf{T}x^\star_{t_{m-1}}\right\|_2^2\right] \le\frac{L_x\delta_m^2}{\alpha_{t_{m-1}}}\sqrt{\mathbb{E}\left[\left\|x^\star_{t_{m-1}}\right\|_2^2\right]}=O(\delta_m^2). \tag*{(110)} δ m 2 E [ ∇ x μ ( x t m − 1 ⋆ , λ t m − 1 ) T x t m − 1 ⋆ 2 2 ] ≤ α t m − 1 L x δ m 2 E [ x t m − 1 ⋆ 2 2 ] = O ( δ m 2 ) . (110)
Third, the remaining term E [ ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] \mathbb{E}\left[\left\|x_{t_{m-1}}-x^\star_{t_{m-1}}\right\|_2^2\right] E [ x t m − 1 − x t m − 1 ⋆ 2 2 ] is precisely the quantity we aim to control. To proceed, we first derive a crude bound for it. From (81) and the Lipschitz property of data predictor with respect to x x x , we have
∥ x t m − x t m ⋆ ∥ 2 ≤ σ t m σ t m − 1 e − δ m ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 + α t m α t m − 1 L x ( 1 − e − 2 δ m ) ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 + 2 α t m ∥ ∫ λ t m − 1 λ t m e − 2 ( λ t m − λ ) ( μ ( x t m − 1 ⋆ , λ t m − 1 ) − μ ( x t ( λ ) ⋆ , λ ) ) d λ ∥ 2 . (111) \left\|x_{t_m}-x^\star_{t_m}\right\|_2 \le\frac{\sigma_{t_m}}{\sigma_{t_{m-1}}}e^{-\delta_m}\left\|x_{t_{m-1}}-x^\star_{t_{m-1}}\right\|_2+\frac{\alpha_{t_m}}{\alpha_{t_{m-1}}}L_x\left(1-e^{-2\delta_m}\right)\left\|x_{t_{m-1}}-x^\star_{t_{m-1}}\right\|_2+2\alpha_{t_m}\left\|\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}e^{-2(\lambda_{t_m}-\lambda)}\left(\mu\left(x^\star_{t_{m-1}},\lambda_{t_{m-1}}\right)-\mu\left(x^\star_{t(\lambda)},\lambda\right)\right)\mathrm{d}\lambda\right\|_2. \tag*{(111)} x t m − x t m ⋆ 2 ≤ σ t m − 1 σ t m e − δ m x t m − 1 − x t m − 1 ⋆ 2 + α t m − 1 α t m L x ( 1 − e − 2 δ m ) x t m − 1 − x t m − 1 ⋆ 2 + 2 α t m ∫ λ t m − 1 λ t m e − 2 ( λ t m − λ ) ( μ ( x t m − 1 ⋆ , λ t m − 1 ) − μ ( x t ( λ ) ⋆ , λ ) ) d λ 2 . (111)
For notational convenience, denote the integral term by
B t m : = α t m ∥ ∫ λ t m − 1 λ t m e − 2 ( λ t m − λ ) ( μ ( x t m − 1 ⋆ , λ t m − 1 ) − μ ( x t ( λ ) ⋆ , λ ) ) d λ ∥ 2 . (112) B_{t_m}:=\alpha_{t_m}\left\|\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}e^{-2(\lambda_{t_m}-\lambda)}\left(\mu\left(x^\star_{t_{m-1}},\lambda_{t_{m-1}}\right)-\mu\left(x^\star_{t(\lambda)},\lambda\right)\right)\mathrm{d}\lambda\right\|_2. \tag*{(112)} B t m := α t m ∫ λ t m − 1 λ t m e − 2 ( λ t m − λ ) ( μ ( x t m − 1 ⋆ , λ t m − 1 ) − μ ( x t ( λ ) ⋆ , λ ) ) d λ 2 . (112)
Since σ t m ≤ σ t m − 1 \sigma_{t_m}\le\sigma_{t_{m-1}} σ t m ≤ σ t m − 1 , we have
σ t m σ t m − 1 e − δ m ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 + α t m α t m − 1 L x ( 1 − e − 2 δ m ) ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 ≤ ( σ t m σ t m − 1 e − δ m + 2 α t m L x δ m α t m − 1 ) ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 ≤ ( 1 + 2 α t m L x δ m α t m − 1 ) ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 . (113) \begin{aligned}
\frac{\sigma_{t_m}}{\sigma_{t_{m-1}}}e^{-\delta_m}\left\|x_{t_{m-1}}-x^\star_{t_{m-1}}\right\|_2+\frac{\alpha_{t_m}}{\alpha_{t_{m-1}}}L_x\left(1-e^{-2\delta_m}\right)\left\|x_{t_{m-1}}-x^\star_{t_{m-1}}\right\|_2
&\le\left(\frac{\sigma_{t_m}}{\sigma_{t_{m-1}}}e^{-\delta_m}+\frac{2\alpha_{t_m}L_x\delta_m}{\alpha_{t_{m-1}}}\right)\left\|x_{t_{m-1}}-x^\star_{t_{m-1}}\right\|_2\\
&\le\left(1+\frac{2\alpha_{t_m}L_x\delta_m}{\alpha_{t_{m-1}}}\right)\left\|x_{t_{m-1}}-x^\star_{t_{m-1}}\right\|_2. \tag*{(113)}
\end{aligned} σ t m − 1 σ t m e − δ m x t m − 1 − x t m − 1 ⋆ 2 + α t m − 1 α t m L x ( 1 − e − 2 δ m ) x t m − 1 − x t m − 1 ⋆ 2 ≤ ( σ t m − 1 σ t m e − δ m + α t m − 1 2 α t m L x δ m ) x t m − 1 − x t m − 1 ⋆ 2 ≤ ( 1 + α t m − 1 2 α t m L x δ m ) x t m − 1 − x t m − 1 ⋆ 2 . (113)
Next, we estimate the second moment of B t m B_{t_m} B t m . Splitting via the triangle inequality gives
E [ B t m 2 ] ≤ 2 α t m 2 E [ ∥ ∫ λ t m − 1 λ t m e − 2 ( λ t m − λ ) ( μ ( x t m − 1 ⋆ , λ t m − 1 ) − μ ( x t ( λ ) ⋆ , λ t m − 1 ) ) d λ ∥ 2 2 ] + 2 α t m 2 E [ ∥ ∫ λ t m − 1 λ t m e − 2 ( λ t m − λ ) ( μ ( x t ( λ ) ⋆ , λ t m − 1 ) − μ ( x t ( λ ) ⋆ , λ ) ) d λ ∥ 2 2 ] ≤ ( a ) 2 δ m α t m 2 E [ ∫ λ t m − 1 λ t m e − 4 ( λ t m − λ ) ∥ μ ( x t m − 1 ⋆ , λ t m − 1 ) − μ ( x t ( λ ) ⋆ , λ t m − 1 ) ∥ 2 2 d λ ] + 2 δ m α t m 2 E [ ∫ λ t m − 1 λ t m e − 4 ( λ t m − λ ) ∥ μ ( x t ( λ ) ⋆ , λ t m − 1 ) − μ ( x t ( λ ) ⋆ , λ ) ∥ 2 2 d λ ] ≤ ( b ) 2 δ m α t m 2 L x 2 α t m − 1 2 ∫ λ t m − 1 λ t m e − 4 ( λ t m − λ ) E [ ∥ x t m − 1 ⋆ − x t ( λ ) ⋆ ∥ 2 2 ] d λ + 2 δ m α t m 2 L t 2 ∫ λ t m − 1 λ t m e − 4 ( λ t m − λ ) ( λ t m − 1 − λ ) 2 d λ ≤ ( c ) δ m 2 ∫ λ t m − 1 λ t m e − 4 ( λ t m − λ ) d λ + δ m 3 ∫ λ t m − 1 λ t m e − 4 ( λ t m − λ ) d λ = O ( δ m 3 ) . (114) \begin{aligned}
\mathbb{E}[B_{t_m}^2]
&\le2\alpha_{t_m}^2\mathbb{E}\left[\left\|\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}e^{-2(\lambda_{t_m}-\lambda)}\left(\mu\left(x^\star_{t_{m-1}},\lambda_{t_{m-1}}\right)-\mu\left(x^\star_{t(\lambda)},\lambda_{t_{m-1}}\right)\right)\mathrm{d}\lambda\right\|_2^2\right]\\
&\quad+2\alpha_{t_m}^2\mathbb{E}\left[\left\|\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}e^{-2(\lambda_{t_m}-\lambda)}\left(\mu\left(x^\star_{t(\lambda)},\lambda_{t_{m-1}}\right)-\mu\left(x^\star_{t(\lambda)},\lambda\right)\right)\mathrm{d}\lambda\right\|_2^2\right]\\
&\overset{(a)}{\le}2\delta_m\alpha_{t_m}^2\mathbb{E}\left[\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}e^{-4(\lambda_{t_m}-\lambda)}\left\|\mu\left(x^\star_{t_{m-1}},\lambda_{t_{m-1}}\right)-\mu\left(x^\star_{t(\lambda)},\lambda_{t_{m-1}}\right)\right\|_2^2\mathrm{d}\lambda\right]\\
&\quad+2\delta_m\alpha_{t_m}^2\mathbb{E}\left[\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}e^{-4(\lambda_{t_m}-\lambda)}\left\|\mu\left(x^\star_{t(\lambda)},\lambda_{t_{m-1}}\right)-\mu\left(x^\star_{t(\lambda)},\lambda\right)\right\|_2^2\mathrm{d}\lambda\right]\\
&\overset{(b)}{\le}\frac{2\delta_m\alpha_{t_m}^2L_x^2}{\alpha_{t_{m-1}}^2}\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}e^{-4(\lambda_{t_m}-\lambda)}\mathbb{E}\left[\left\|x^\star_{t_{m-1}}-x^\star_{t(\lambda)}\right\|_2^2\right]\mathrm{d}\lambda\\
&\quad+2\delta_m\alpha_{t_m}^2L_t^2\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}e^{-4(\lambda_{t_m}-\lambda)}(\lambda_{t_{m-1}}-\lambda)^2\mathrm{d}\lambda\\
&\overset{(c)}{\le}\delta_m^2\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}e^{-4(\lambda_{t_m}-\lambda)}\mathrm{d}\lambda+\delta_m^3\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}e^{-4(\lambda_{t_m}-\lambda)}\mathrm{d}\lambda=O(\delta_m^3). \tag*{(114)}
\end{aligned} E [ B t m 2 ] ≤ 2 α t m 2 E ∫ λ t m − 1 λ t m e − 2 ( λ t m − λ ) ( μ ( x t m − 1 ⋆ , λ t m − 1 ) − μ ( x t ( λ ) ⋆ , λ t m − 1 ) ) d λ 2 2 + 2 α t m 2 E ∫ λ t m − 1 λ t m e − 2 ( λ t m − λ ) ( μ ( x t ( λ ) ⋆ , λ t m − 1 ) − μ ( x t ( λ ) ⋆ , λ ) ) d λ 2 2 ≤ ( a ) 2 δ m α t m 2 E [ ∫ λ t m − 1 λ t m e − 4 ( λ t m − λ ) μ ( x t m − 1 ⋆ , λ t m − 1 ) − μ ( x t ( λ ) ⋆ , λ t m − 1 ) 2 2 d λ ] + 2 δ m α t m 2 E [ ∫ λ t m − 1 λ t m e − 4 ( λ t m − λ ) μ ( x t ( λ ) ⋆ , λ t m − 1 ) − μ ( x t ( λ ) ⋆ , λ ) 2 2 d λ ] ≤ ( b ) α t m − 1 2 2 δ m α t m 2 L x 2 ∫ λ t m − 1 λ t m e − 4 ( λ t m − λ ) E [ x t m − 1 ⋆ − x t ( λ ) ⋆ 2 2 ] d λ + 2 δ m α t m 2 L t 2 ∫ λ t m − 1 λ t m e − 4 ( λ t m − λ ) ( λ t m − 1 − λ ) 2 d λ ≤ ( c ) δ m 2 ∫ λ t m − 1 λ t m e − 4 ( λ t m − λ ) d λ + δ m 3 ∫ λ t m − 1 λ t m e − 4 ( λ t m − λ ) d λ = O ( δ m 3 ) . (114)
where (a) uses Jensen’s inequality (with the interval length δ m \delta_m δ m ), (b) follows from Assumptions A2 and A4, and (c) uses (107) . Combining (113) and (114) with (111) , we obtain
E [ ∥ x t m − x t m ⋆ ∥ 2 2 ] ≤ ( 1 + 2 α t m L x δ m α t m − 1 ) 2 E [ ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] + 4 ( 1 + 2 α t m L x δ m α t m − 1 ) E [ B t m ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 ] + 4 E [ B t m 2 ] ≤ ( a ) ( 1 + 2 α t m L x δ m α t m − 1 ) 2 E [ ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] + 2 ( 1 + 2 α t m L x δ m α t m − 1 ) O ( δ m 1.5 ) E [ ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] + O ( δ m 3 ) . (115) \begin{aligned}
\mathbb{E}\left[\left\|x_{t_m}-x^\star_{t_m}\right\|_2^2\right]
&\le\left(1+\frac{2\alpha_{t_m}L_x\delta_m}{\alpha_{t_{m-1}}}\right)^2\mathbb{E}\left[\left\|x_{t_{m-1}}-x^\star_{t_{m-1}}\right\|_2^2\right]+4\left(1+\frac{2\alpha_{t_m}L_x\delta_m}{\alpha_{t_{m-1}}}\right)\mathbb{E}\left[B_{t_m}\left\|x_{t_{m-1}}-x^\star_{t_{m-1}}\right\|_2\right]+4\mathbb{E}[B_{t_m}^2]\\
&\overset{(a)}{\le}\left(1+\frac{2\alpha_{t_m}L_x\delta_m}{\alpha_{t_{m-1}}}\right)^2\mathbb{E}\left[\left\|x_{t_{m-1}}-x^\star_{t_{m-1}}\right\|_2^2\right]\\
&\quad+2\left(1+\frac{2\alpha_{t_m}L_x\delta_m}{\alpha_{t_{m-1}}}\right)O(\delta_m^{1.5})\sqrt{\mathbb{E}\left[\left\|x_{t_{m-1}}-x^\star_{t_{m-1}}\right\|_2^2\right]}+O(\delta_m^3). \tag*{(115)}
\end{aligned} E [ x t m − x t m ⋆ 2 2 ] ≤ ( 1 + α t m − 1 2 α t m L x δ m ) 2 E [ x t m − 1 − x t m − 1 ⋆ 2 2 ] + 4 ( 1 + α t m − 1 2 α t m L x δ m ) E [ B t m x t m − 1 − x t m − 1 ⋆ 2 ] + 4 E [ B t m 2 ] ≤ ( a ) ( 1 + α t m − 1 2 α t m L x δ m ) 2 E [ x t m − 1 − x t m − 1 ⋆ 2 2 ] + 2 ( 1 + α t m − 1 2 α t m L x δ m ) O ( δ m 1.5 ) E [ x t m − 1 − x t m − 1 ⋆ 2 2 ] + O ( δ m 3 ) . (115)
where (a) uses (114) and Cauchy-Schwarz
E [ B t m ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 ] ≤ E [ B t m 2 ] E [ ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] = O ( δ m 1.5 ) E [ ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] . \mathbb{E}\left[B_{t_m}\left\|x_{t_{m-1}}-x^\star_{t_{m-1}}\right\|_2\right]\leq\sqrt{\mathbb{E}\left[B_{t_m}^2\right]\mathbb{E}\left[\left\|x_{t_{m-1}}-x^\star_{t_{m-1}}\right\|_2^2\right]}=O\left(\delta_m^{1.5}\right)\sqrt{\mathbb{E}\left[\left\|x_{t_{m-1}}-x^\star_{t_{m-1}}\right\|_2^2\right]}. E [ B t m x t m − 1 − x t m − 1 ⋆ 2 ] ≤ E [ B t m 2 ] E [ x t m − 1 − x t m − 1 ⋆ 2 2 ] = O ( δ m 1.5 ) E [ x t m − 1 − x t m − 1 ⋆ 2 2 ] .
For sufficiently small δ m \delta_m δ m , we have 2 α t m L x δ m α t m − 1 ≤ 1 \frac{2\alpha_{t_m}L_x\delta_m}{\alpha_{t_{m-1}}}\leq1 α t m − 1 2 α t m L x δ m ≤ 1 and
δ m 1.5 E [ ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] ≤ 1 2 δ m 2 + 1 2 δ m E [ ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] . \delta_m^{1.5}\sqrt{\mathbb{E}\left[\left\|x_{t_{m-1}}-x^\star_{t_{m-1}}\right\|_2^2\right]}\leq\frac{1}{2}\delta_m^2+\frac{1}{2}\delta_m\mathbb{E}\left[\left\|x_{t_{m-1}}-x^\star_{t_{m-1}}\right\|_2^2\right]. δ m 1.5 E [ x t m − 1 − x t m − 1 ⋆ 2 2 ] ≤ 2 1 δ m 2 + 2 1 δ m E [ x t m − 1 − x t m − 1 ⋆ 2 2 ] .
Therefore we have
E [ ∥ x t m − x t m ⋆ ∥ 2 2 ] ≤ ( 1 + 5 α t m L x δ m α t m − 1 + O ( δ m ) ) E [ ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] + O ( δ m 2 ) \mathbb{E}\left[\left\|x_{t_m}-x^\star_{t_m}\right\|_2^2\right]\leq\left(1+\frac{5\alpha_{t_m}L_x\delta_m}{\alpha_{t_{m-1}}}+O(\delta_m)\right)\mathbb{E}\left[\left\|x_{t_{m-1}}-x^\star_{t_{m-1}}\right\|_2^2\right]+O(\delta_m^2) E [ x t m − x t m ⋆ 2 2 ] ≤ ( 1 + α t m − 1 5 α t m L x δ m + O ( δ m ) ) E [ x t m − 1 − x t m − 1 ⋆ 2 2 ] + O ( δ m 2 )
≲ ∑ i = 1 m ∏ j = i + 1 m ( 1 + 5 α t j L x δ j α t j − 1 + O ( δ j ) ) δ i 2 \lesssim\sum_{i=1}^{m}\prod_{j=i+1}^{m}\left(1+\frac{5\alpha_{t_j}L_x\delta_j}{\alpha_{t_{j-1}}}+O(\delta_j)\right)\delta_i^2 ≲ i = 1 ∑ m j = i + 1 ∏ m ( 1 + α t j − 1 5 α t j L x δ j + O ( δ j ) ) δ i 2
≲ ∑ i = 1 m exp ( ∑ j = i + 1 m ( 5 α t j L x δ j α t j − 1 + O ( δ j ) ) ) δ i 2 \lesssim\sum_{i=1}^{m}\exp\left(\sum_{j=i+1}^{m}\left(\frac{5\alpha_{t_j}L_x\delta_j}{\alpha_{t_{j-1}}}+O(\delta_j)\right)\right)\delta_i^2 ≲ i = 1 ∑ m exp ( j = i + 1 ∑ m ( α t j − 1 5 α t j L x δ j + O ( δ j ) ) ) δ i 2
≲ ( a ) ∑ i = 1 m δ i 2 = O ( δ ) . (116) \underset{(a)}{\lesssim}\sum_{i=1}^{m}\delta_i^2=O(\delta). \tag*{(116)} ( a ) ≲ i = 1 ∑ m δ i 2 = O ( δ ) . (116)
where (a) uses ∑ j = i + 1 m δ j ≤ ∑ j = 1 M δ j = O ( 1 ) \sum_{j=i+1}^{m}\delta_j\leq\sum_{j=1}^{M}\delta_j=O(1) ∑ j = i + 1 m δ j ≤ ∑ j = 1 M δ j = O ( 1 ) . Substituting (109) , (110) , and (116) into (108) yields the desired bound:
E [ ∥ E t m , 1 ∥ 2 2 ] = O ( δ 2 ) . (117) \mathbb{E}\left[\left\|\mathcal{E}_{t_m,1}\right\|_2^2\right]=O(\delta^2). \tag*{(117)} E [ ∥ E t m , 1 ∥ 2 2 ] = O ( δ 2 ) . (117)
Hence the first term on the right-hand-side of (95) satisfies
∑ m = 1 M E [ ∥ E t m , 1 ∥ 2 2 ] ≲ M δ 2 = O ( δ ) . (118) \sum_{m=1}^{M}\mathbb{E}\left[\left\|\mathcal{E}_{t_m,1}\right\|_2^2\right]\lesssim M\delta^2=O(\delta). \tag*{(118)} m = 1 ∑ M E [ ∥ E t m , 1 ∥ 2 2 ] ≲ M δ 2 = O ( δ ) . (118)
Finally, combining (104) and (118) into (95) yields
E [ ∥ x t M − x t M ⋆ ∥ 2 2 ] = O ( δ ) . (119) \mathbb{E}\left[\left\|x_{t_M}-x^\star_{t_M}\right\|_2^2\right]=O(\delta). \tag*{(119)} E [ x t M − x t M ⋆ 2 2 ] = O ( δ ) . (119)
This completes the proof.
Proof of (80) . From the explicit solution of the SDE in (17) , we have
x t i ⋆ = σ t i σ t 0 e − ( λ t i − λ t 0 ) x t 0 + 2 α t i ∫ λ t 0 λ t i e − 2 ( λ t i − λ ) μ ( x t ( λ ) ⋆ , λ ) d λ + 2 σ t i ∫ λ t 0 λ t i e − ( λ t i − λ ) d w ‾ t ( λ ) . (120) x^\star_{t_i}=\frac{\sigma_{t_i}}{\sigma_{t_0}}e^{-(\lambda_{t_i}-\lambda_{t_0})}x_{t_0}+2\alpha_{t_i}\int_{\lambda_{t_0}}^{\lambda_{t_i}}e^{-2(\lambda_{t_i}-\lambda)}\mu(x^\star_{t(\lambda)},\lambda)\,d\lambda+\sqrt{2}\sigma_{t_i}\int_{\lambda_{t_0}}^{\lambda_{t_i}}e^{-(\lambda_{t_i}-\lambda)}d\overline{w}_t(\lambda). \tag*{(120)} x t i ⋆ = σ t 0 σ t i e − ( λ t i − λ t 0 ) x t 0 + 2 α t i ∫ λ t 0 λ t i e − 2 ( λ t i − λ ) μ ( x t ( λ ) ⋆ , λ ) d λ + 2 σ t i ∫ λ t 0 λ t i e − ( λ t i − λ ) d w t ( λ ) . (120)
Under Assumption A1, the data predictor term is uniformly bounded by L μ L_\mu L μ . Applying the triangle inequality yields
∥ x t i ⋆ ∥ 2 ≤ σ t i σ t 0 e − ( λ t i − λ t 0 ) ∥ x t 0 ∥ 2 + 2 α t i L μ ∫ λ t 0 λ t i e − 2 ( λ t i − λ ) d λ + 2 σ t i ∥ ∫ λ t 0 λ t i e − ( λ t i − λ ) d w ‾ t ( λ ) ∥ 2 . (121) \left\|x^\star_{t_i}\right\|_2\leq\frac{\sigma_{t_i}}{\sigma_{t_0}}e^{-(\lambda_{t_i}-\lambda_{t_0})}\left\|x_{t_0}\right\|_2+2\alpha_{t_i}L_\mu\int_{\lambda_{t_0}}^{\lambda_{t_i}}e^{-2(\lambda_{t_i}-\lambda)}d\lambda+\sqrt{2}\sigma_{t_i}\left\|\int_{\lambda_{t_0}}^{\lambda_{t_i}}e^{-(\lambda_{t_i}-\lambda)}d\overline{w}_t(\lambda)\right\|_2. \tag*{(121)} x t i ⋆ 2 ≤ σ t 0 σ t i e − ( λ t i − λ t 0 ) ∥ x t 0 ∥ 2 + 2 α t i L μ ∫ λ t 0 λ t i e − 2 ( λ t i − λ ) d λ + 2 σ t i ∫ λ t 0 λ t i e − ( λ t i − λ ) d w t ( λ ) 2 . (121)
Taking expectations and using the elementary inequality ( a + b + c ) 2 ≤ 3 a 2 + 3 b 2 + 3 c 2 (a+b+c)^2\leq3a^2+3b^2+3c^2 ( a + b + c ) 2 ≤ 3 a 2 + 3 b 2 + 3 c 2 , we obtain
E [ ∥ x t i ⋆ ∥ 2 2 ] ≤ σ t i 2 σ t 0 2 e − 2 ( λ t i − λ t 0 ) E [ ∥ x t 0 ∥ 2 2 ] + 12 α t i 2 L μ 2 + 6 σ t i 2 E [ ∥ ∫ λ t 0 λ t i e − ( λ t i − λ ) d w ‾ t ( λ ) ∥ 2 2 ] = O ( 1 ) , (122) \mathbb{E}\left[\left\|x^\star_{t_i}\right\|_2^2\right]\leq\frac{\sigma_{t_i}^2}{\sigma_{t_0}^2}e^{-2(\lambda_{t_i}-\lambda_{t_0})}\mathbb{E}\left[\left\|x_{t_0}\right\|_2^2\right]+12\alpha_{t_i}^2L_\mu^2+6\sigma_{t_i}^2\mathbb{E}\left[\left\|\int_{\lambda_{t_0}}^{\lambda_{t_i}}e^{-(\lambda_{t_i}-\lambda)}d\overline{w}_t(\lambda)\right\|_2^2\right]=O(1), \tag*{(122)} E [ x t i ⋆ 2 2 ] ≤ σ t 0 2 σ t i 2 e − 2 ( λ t i − λ t 0 ) E [ ∥ x t 0 ∥ 2 2 ] + 12 α t i 2 L μ 2 + 6 σ t i 2 E ∫ λ t 0 λ t i e − ( λ t i − λ ) d w t ( λ ) 2 2 = O ( 1 ) , (122)
where the last equation uses E [ ∥ x t 0 ∥ 2 2 ] = d σ t 0 2 = O ( 1 ) \mathbb{E}\left[\left\|x_{t_0}\right\|_2^2\right]=d\sigma_{t_0}^2=O(1) E [ ∥ x t 0 ∥ 2 2 ] = d σ t 0 2 = O ( 1 ) , and
E [ ∥ ∫ λ t 0 λ t i e − ( λ t i − λ ) d w ‾ t ( λ ) ∥ 2 2 ] = d ∫ λ t 0 λ t i e − 2 ( λ t i − λ ) d λ ≤ d 2 . (123) \mathbb{E}\left[\left\|\int_{\lambda_{t_0}}^{\lambda_{t_i}}e^{-(\lambda_{t_i}-\lambda)}d\overline{w}_t(\lambda)\right\|_2^2\right]=d\int_{\lambda_{t_0}}^{\lambda_{t_i}}e^{-2(\lambda_{t_i}-\lambda)}d\lambda\leq\frac{d}{2}. \tag*{(123)} E ∫ λ t 0 λ t i e − ( λ t i − λ ) d w t ( λ ) 2 2 = d ∫ λ t 0 λ t i e − 2 ( λ t i − λ ) d λ ≤ 2 d . (123)
This completes the proof of the preliminary bound.
Convergence analysis of SDE-DPMSolver++2
For notational convenience, we denote by D t i \mathbf{D}_{t_i} D t i the difference term D 1 \mathbf{D}_1 D 1 that appears when updating x t i − 1 \mathbf{x}_{t_{i-1}} x t i − 1 to x t i \mathbf{x}_{t_i} x t i . Similar to the analysis in Section C.1, we first establish the following preliminary bound, whose proof is deferred to the end of this section:
E [ ∥ x t i ∥ 2 2 ] = O ( 1 ) , 0 ≤ i ≤ M . (124) \mathbb{E}[\|\mathbf{x}_{t_i}\|_2^2] = O(1), \qquad0 \le i \le M. \tag*{(124)} E [ ∥ x t i ∥ 2 2 ] = O ( 1 ) , 0 ≤ i ≤ M . (124)
Step 1: some crude bounds. Before proceeding with the main proof, we derive some crude estimates for D t i \mathbf{D}_{t_i} D t i and x t i − 1 − x t i − 2 \mathbf{x}_{t_{i-1}}-\mathbf{x}_{t_{i-2}} x t i − 1 − x t i − 2 . We begin with D t i \mathbf{D}_{t_i} D t i . By its definition in (19) ,
∥ D t i ∥ 2 = 1 δ i − 1 ∥ μ θ ( x t i − 1 , t i − 1 ) − μ θ ( x t i − 2 , t i − 2 ) ∥ 2 ≤ 1 δ i − 1 ∥ μ θ ( x t i − 1 , t i − 1 ) − μ θ ( x t i − 2 , t i − 1 ) ∥ 2 + 1 δ i − 1 ∥ μ θ ( x t i − 2 , t i − 1 ) − μ θ ( x t i − 2 , t i − 2 ) ∥ 2 ≤ L x δ i − 1 α t i − 1 ∥ x t i − 1 − x t i − 2 ∥ 2 + L t , (125) \begin{aligned}
\|\mathbf{D}_{t_i}\|_2 = \frac{1}{\delta_{i-1}}\|\boldsymbol{\mu}_{\theta}(\mathbf{x}_{t_{i-1}},t_{i-1})-\boldsymbol{\mu}_{\theta}(\mathbf{x}_{t_{i-2}},t_{i-2})\|_2 \\
&\le\frac{1}{\delta_{i-1}}\|\boldsymbol{\mu}_{\theta}(\mathbf{x}_{t_{i-1}},t_{i-1})-\boldsymbol{\mu}_{\theta}(\mathbf{x}_{t_{i-2}},t_{i-1})\|_2+\frac{1}{\delta_{i-1}}\|\boldsymbol{\mu}_{\theta}(\mathbf{x}_{t_{i-2}},t_{i-1})-\boldsymbol{\mu}_{\theta}(\mathbf{x}_{t_{i-2}},t_{i-2})\|_2 \\
&\le\frac{L_x}{\delta_{i-1}\alpha_{t_{i-1}}}\|\mathbf{x}_{t_{i-1}}-\mathbf{x}_{t_{i-2}}\|_2+L_t, \tag*{(125)}
\end{aligned} ∥ D t i ∥ 2 = δ i − 1 1 ∥ μ θ ( x t i − 1 , t i − 1 ) − μ θ ( x t i − 2 , t i − 2 ) ∥ 2 ≤ δ i − 1 1 ∥ μ θ ( x t i − 1 , t i − 1 ) − μ θ ( x t i − 2 , t i − 1 ) ∥ 2 + δ i − 1 1 ∥ μ θ ( x t i − 2 , t i − 1 ) − μ θ ( x t i − 2 , t i − 2 ) ∥ 2 ≤ δ i − 1 α t i − 1 L x ∥ x t i − 1 − x t i − 2 ∥ 2 + L t , (125)
where the last inequality follows from Assumptions 1 (A2) and (A4). Next, from the update rule (19) , we have
x t i − 1 − x t i − 2 = ( e − δ i − 1 σ t i − 1 σ t i − 2 − 1 ) x t i − 2 + α t i − 1 ( 1 − e − 2 δ i − 1 ) μ θ ( x t i − 2 , t i − 2 ) \mathbf{x}_{t_{i-1}}-\mathbf{x}_{t_{i-2}}=\left(e^{-\delta_{i-1}}\frac{\sigma_{t_{i-1}}}{\sigma_{t_{i-2}}}-1\right)\mathbf{x}_{t_{i-2}}+\alpha_{t_{i-1}}(1-e^{-2\delta_{i-1}})\boldsymbol{\mu}_{\theta}(\mathbf{x}_{t_{i-2}},t_{i-2}) x t i − 1 − x t i − 2 = ( e − δ i − 1 σ t i − 2 σ t i − 1 − 1 ) x t i − 2 + α t i − 1 ( 1 − e − 2 δ i − 1 ) μ θ ( x t i − 2 , t i − 2 )
+ α t i − 1 2 ( 1 − e − 2 δ i − 1 ) δ i − 1 D t i − 1 + σ t i − 1 1 − e − 2 δ i − 1 z i − 1 , (126) \qquad+\frac{\alpha_{t_{i-1}}}{2}(1-e^{-2\delta_{i-1}})\delta_{i-1}\mathbf{D}_{t_{i-1}}+\sigma_{t_{i-1}}\sqrt{1-e^{-2\delta_{i-1}}}\mathbf{z}_{i-1}, \tag*{(126)} + 2 α t i − 1 ( 1 − e − 2 δ i − 1 ) δ i − 1 D t i − 1 + σ t i − 1 1 − e − 2 δ i − 1 z i − 1 , (126)
Using bound in (84) , we obtain
∥ x t i − 1 − x t i − 2 ∥ 2 2 ≤ 2 δ i − 1 ∥ x t i − 2 ∥ 2 2 + 2 α t i − 1 δ i − 1 ∥ μ θ ( x t i − 2 , t i − 2 ) ∥ 2 2 + α t i − 1 δ i − 1 2 ∥ D t i − 1 ∥ 2 2 + σ t i − 1 2 δ i − 1 ∥ z i − 1 ∥ 2 \|\mathbf{x}_{t_{i-1}}-\mathbf{x}_{t_{i-2}}\|_2^2\le2\delta_{i-1}\|\mathbf{x}_{t_{i-2}}\|_2^2+2\alpha_{t_{i-1}}\delta_{i-1}\|\boldsymbol{\mu}_{\theta}(\mathbf{x}_{t_{i-2}},t_{i-2})\|_2^2+\alpha_{t_{i-1}}\delta_{i-1}^2\|\mathbf{D}_{t_{i-1}}\|_2^2+\sigma_{t_{i-1}}\sqrt{2\delta_{i-1}}\|\mathbf{z}_{i-1}\|_2 ∥ x t i − 1 − x t i − 2 ∥ 2 2 ≤ 2 δ i − 1 ∥ x t i − 2 ∥ 2 2 + 2 α t i − 1 δ i − 1 ∥ μ θ ( x t i − 2 , t i − 2 ) ∥ 2 2 + α t i − 1 δ i − 1 2 ∥ D t i − 1 ∥ 2 2 + σ t i − 1 2 δ i − 1 ∥ z i − 1 ∥ 2
≤ 2 δ i − 1 ∥ x t i − 2 ∥ 2 2 + 2 α t i − 1 δ i − 1 L μ + α t i − 1 δ i − 1 2 ∥ D t i − 1 ∥ 2 2 + σ t i − 1 2 δ i − 1 ∥ z i − 1 ∥ 2 , (127) \le2\delta_{i-1}\|\mathbf{x}_{t_{i-2}}\|_2^2+2\alpha_{t_{i-1}}\delta_{i-1}L_\mu+\alpha_{t_{i-1}}\delta_{i-1}^2\|\mathbf{D}_{t_{i-1}}\|_2^2+\sigma_{t_{i-1}}\sqrt{2\delta_{i-1}}\|\mathbf{z}_{i-1}\|_2, \tag*{(127)} ≤ 2 δ i − 1 ∥ x t i − 2 ∥ 2 2 + 2 α t i − 1 δ i − 1 L μ + α t i − 1 δ i − 1 2 ∥ D t i − 1 ∥ 2 2 + σ t i − 1 2 δ i − 1 ∥ z i − 1 ∥ 2 , (127)
where the last inequality uses Assumption 1 (A1). Substituting (127) into (125) yields the following recursion for ∥ D t i ∥ 2 2 \|\mathbf{D}_{t_i}\|_2^2 ∥ D t i ∥ 2 2 :
∥ D t i ∥ 2 2 ≤ 2 L x α t i − 1 ∥ x t i − 2 ∥ 2 2 + 2 L x L μ + L x δ i − 1 ∥ D t i − 1 ∥ 2 2 + 2 L x σ t i − 1 δ i − 1 α t i − 1 ∥ z i − 1 ∥ 2 + L t ≤ ∑ m = 1 i − 1 ( L x δ ) i − m − 1 ( 2 L x α t m ∥ x t m − 1 ∥ 2 2 + 2 L x L μ + L t + 2 L x σ t m δ m α t m ∥ z m ∥ 2 ) . (128) \begin{aligned}
\|\mathbf{D}_{t_i}\|_2^2\le\frac{2L_x}{\alpha_{t_{i-1}}}\|\mathbf{x}_{t_{i-2}}\|_2^2+2L_xL_\mu+L_x\delta_{i-1}\|\mathbf{D}_{t_{i-1}}\|_2^2+\frac{\sqrt{2}L_x\sigma_{t_{i-1}}}{\sqrt{\delta_{i-1}}\alpha_{t_{i-1}}}\|\mathbf{z}_{i-1}\|_2+L_t \\
&\le\sum_{m=1}^{i-1}(L_x\delta)^{i-m-1}\left(\frac{2L_x}{\alpha_{t_m}}\|\mathbf{x}_{t_{m-1}}\|_2^2+2L_xL_\mu+L_t+\frac{\sqrt{2}L_x\sigma_{t_m}}{\sqrt{\delta_m}\alpha_{t_m}}\|\mathbf{z}_m\|_2\right). \tag*{(128)}
\end{aligned} ∥ D t i ∥ 2 2 ≤ α t i − 1 2 L x ∥ x t i − 2 ∥ 2 2 + 2 L x L μ + L x δ i − 1 ∥ D t i − 1 ∥ 2 2 + δ i − 1 α t i − 1 2 L x σ t i − 1 ∥ z i − 1 ∥ 2 + L t ≤ m = 1 ∑ i − 1 ( L x δ ) i − m − 1 ( α t m 2 L x ∥ x t m − 1 ∥ 2 2 + 2 L x L μ + L t + δ m α t m 2 L x σ t m ∥ z m ∥ 2 ) . (128)
By taking expectation, we get
E [ ∥ D t i ∥ 2 2 ] ≤ 3 E [ ( ∑ m = 1 i − 1 ( L x δ ) i − m − 1 ∥ x t m − 1 ∥ 2 ) 2 ] + 3 ( ∑ m = 1 i − 1 ( L x δ ) i − m − 1 ( 2 L x L μ + L t ) ) 2 \mathbb{E}[\|\mathbf{D}_{t_i}\|_2^2]\le3\mathbb{E}\left[\left(\sum_{m=1}^{i-1}(L_x\delta)^{i-m-1}\|\mathbf{x}_{t_{m-1}}\|_2\right)^2\right]+3\left(\sum_{m=1}^{i-1}(L_x\delta)^{i-m-1}(2L_xL_\mu+L_t)\right)^2 E [ ∥ D t i ∥ 2 2 ] ≤ 3 E ( m = 1 ∑ i − 1 ( L x δ ) i − m − 1 ∥ x t m − 1 ∥ 2 ) 2 + 3 ( m = 1 ∑ i − 1 ( L x δ ) i − m − 1 ( 2 L x L μ + L t ) ) 2
+ 3 E [ ( ∑ m = 1 i − 1 ( L x δ ) i − m − 1 2 L x σ t m δ m α t m ∥ z m ∥ 2 ) 2 ] . (129) \qquad+3\mathbb{E}\left[\left(\sum_{m=1}^{i-1}(L_x\delta)^{i-m-1}\frac{\sqrt{2}L_x\sigma_{t_m}}{\sqrt{\delta_m}\alpha_{t_m}}\|\mathbf{z}_m\|_2\right)^2\right]. \tag*{(129)} + 3 E ( m = 1 ∑ i − 1 ( L x δ ) i − m − 1 δ m α t m 2 L x σ t m ∥ z m ∥ 2 ) 2 . (129)
We now estimate each of the three terms on the right-hand side of (129) . For the first term, by Cauchy-Schwarz and the moment bound E [ ∥ x t m − 1 ∥ 2 2 ] = O ( 1 ) \mathbb{E}[\|\mathbf{x}_{t_{m-1}}\|_2^2]=O(1) E [ ∥ x t m − 1 ∥ 2 2 ] = O ( 1 ) , we have
E [ ( ∑ m = 1 i − 1 ( L x δ ) i − m − 1 ∥ x t m − 1 ∥ 2 ) 2 ] ≤ E [ ∑ m = 1 i − 1 ( L x δ ) i − m − 1 ∑ m = 1 i − 1 ( L x δ ) i − m − 1 ∥ x t m − 1 ∥ 2 2 ] ≤ 1 1 − L x δ ∑ m = 1 i − 1 ( L x δ ) i − m − 1 E [ ∥ x t m − 1 ∥ 2 2 ] \begin{aligned}
\mathbb{E}\left[\left(\sum_{m=1}^{i-1}(L_x\delta)^{i-m-1}\|\mathbf{x}_{t_{m-1}}\|_2\right)^2\right]\le\mathbb{E}\left[\sum_{m=1}^{i-1}(L_x\delta)^{i-m-1}\sum_{m=1}^{i-1}(L_x\delta)^{i-m-1}\|\mathbf{x}_{t_{m-1}}\|_2^2\right] \\
&\le\frac{1}{1-L_x\delta}\sum_{m=1}^{i-1}(L_x\delta)^{i-m-1}\mathbb{E}[\|\mathbf{x}_{t_{m-1}}\|_2^2]
\end{aligned} E ( m = 1 ∑ i − 1 ( L x δ ) i − m − 1 ∥ x t m − 1 ∥ 2 ) 2 ≤ E [ m = 1 ∑ i − 1 ( L x δ ) i − m − 1 m = 1 ∑ i − 1 ( L x δ ) i − m − 1 ∥ x t m − 1 ∥ 2 2 ] ≤ 1 − L x δ 1 m = 1 ∑ i − 1 ( L x δ ) i − m − 1 E [ ∥ x t m − 1 ∥ 2 2 ]
e n d s m i d = 1 ≤ ( 1 1 − L x δ ) 2 = O ( 1 ) , \begin{aligned}
{ends_mid=1} \\
&\le\left(\frac{1}{1-L_x\delta}\right)^2=O(1),
\end{aligned} e n d s m i d = 1 ≤ ( 1 − L x δ 1 ) 2 = O ( 1 ) ,
for sufficiently small δ \delta δ such that L x δ ≤ 1 / 2 L_x\delta\le1/2 L x δ ≤ 1/2 . For the second term, we have
( ∑ m = 1 i − 1 ( L x δ ) i − m − 1 ( 2 L x L μ + L t ) ) 2 ≤ ( 2 L x L μ + L t ) 2 ( 1 − L x δ ) 2 = O ( 1 ) . \left(\sum_{m=1}^{i-1}(L_x\delta)^{i-m-1}(2L_xL_\mu+L_t)\right)^2\le\frac{(2L_xL_\mu+L_t)^2}{(1-L_x\delta)^2}=O(1). ( m = 1 ∑ i − 1 ( L x δ ) i − m − 1 ( 2 L x L μ + L t ) ) 2 ≤ ( 1 − L x δ ) 2 ( 2 L x L μ + L t ) 2 = O ( 1 ) .
For the third term, since z m ∼ N ( 0 , I d ) z_m\sim\mathcal{N}(0,I_d) z m ∼ N ( 0 , I d ) are independent standard Gaussian vectors, we have E [ ∥ z m ∥ 2 2 ] = d \mathbb{E}[\lVert z_m\rVert_2^2]=d E [∥ z m ∥ 2 2 ] = d . Applying Cauchy-Schwarz yields
E [ ( ∑ m = 1 i − 1 ( L x δ ) i − m − 1 2 L x σ t m δ m α t m ∥ z m ∥ 2 ) 2 ] ≤ E [ ∑ m = 1 i − 1 ( L x δ ) i − m − 1 ( ∑ m = 1 i − 1 ( L x δ ) i − m − 1 2 L x 2 σ t m 2 δ m α t m 2 ∥ z m ∥ 2 2 ) ] ≤ 1 1 − L x δ ∑ m = 1 i − 1 ( L x δ ) i − m − 1 2 d L x 2 σ t m 2 δ m α t m 2 = O ( 1 δ ) , \begin{aligned}
\mathbb{E}\left[\left(\sum_{m=1}^{i-1}(L_x\delta)^{i-m-1}\frac{\sqrt{2}L_x\sigma_{t_m}}{\sqrt{\delta_m}\alpha_{t_m}}\lVert z_m\rVert_2\right)^2\right]
\le\mathbb{E}\left[\sum_{m=1}^{i-1}(L_x\delta)^{i-m-1}\left(\sum_{m=1}^{i-1}(L_x\delta)^{i-m-1}\frac{2L_x^2\sigma_{t_m}^2}{\delta_m\alpha_{t_m}^2}\lVert z_m\rVert_2^2\right)\right] \\
&\le\frac{1}{1-L_x\delta}\sum_{m=1}^{i-1}(L_x\delta)^{i-m-1}\frac{2dL_x^2\sigma_{t_m}^2}{\delta_m\alpha_{t_m}^2}=O\left(\frac{1}{\delta}\right),
\end{aligned} E ( m = 1 ∑ i − 1 ( L x δ ) i − m − 1 δ m α t m 2 L x σ t m ∥ z m ∥ 2 ) 2 ≤ E [ m = 1 ∑ i − 1 ( L x δ ) i − m − 1 ( m = 1 ∑ i − 1 ( L x δ ) i − m − 1 δ m α t m 2 2 L x 2 σ t m 2 ∥ z m ∥ 2 2 ) ] ≤ 1 − L x δ 1 m = 1 ∑ i − 1 ( L x δ ) i − m − 1 δ m α t m 2 2 d L x 2 σ t m 2 = O ( δ 1 ) ,
where we have used the assumption that max δ i / min δ i = O ( 1 ) \max\delta_i/\min\delta_i=O(1) max δ i / min δ i = O ( 1 ) . Combining the above three estimates into (129) yields
E [ ∥ D t i ∥ 2 2 ] = O ( 1 δ ) . (130) \mathbb{E}[\lVert D_{t_i}\rVert_2^2]=O\left(\frac{1}{\delta}\right). \tag*{(130)} E [∥ D t i ∥ 2 2 ] = O ( δ 1 ) . (130)
Finally, substituting (130) into (127) , we obtain
E [ ∥ x t i − 1 − x t i − 2 ∥ 2 2 ] ≤ 16 δ i − 1 2 E [ ∥ x t i − 2 ∥ 2 2 ] + 16 α t i − 1 2 δ i − 1 2 L μ 2 + 4 α t i − 1 2 δ i − 1 4 E [ ∥ D t i − 1 ∥ 2 2 ] + 8 σ t i − 1 2 δ i − 1 E [ ∥ z i − 1 ∥ 2 2 ] \mathbb{E}[\lVert x_{t_{i-1}}-x_{t_{i-2}}\rVert_2^2]\le16\delta_{i-1}^2\mathbb{E}[\lVert x_{t_{i-2}}\rVert_2^2]+16\alpha_{t_{i-1}}^2\delta_{i-1}^2L_\mu^2+4\alpha_{t_{i-1}}^2\delta_{i-1}^4\mathbb{E}[\lVert D_{t_{i-1}}\rVert_2^2]+8\sigma_{t_{i-1}}^2\delta_{i-1}\mathbb{E}[\lVert z_{i-1}\rVert_2^2] E [∥ x t i − 1 − x t i − 2 ∥ 2 2 ] ≤ 16 δ i − 1 2 E [∥ x t i − 2 ∥ 2 2 ] + 16 α t i − 1 2 δ i − 1 2 L μ 2 + 4 α t i − 1 2 δ i − 1 4 E [∥ D t i − 1 ∥ 2 2 ] + 8 σ t i − 1 2 δ i − 1 E [∥ z i − 1 ∥ 2 2 ]
≲ δ i − 1 2 + δ i − 1 3 + δ i − 1 = O ( δ i − 1 ) . (131) \lesssim\delta_{i-1}^2+\delta_{i-1}^3+\delta_{i-1}=O(\delta_{i-1}). \tag*{(131)} ≲ δ i − 1 2 + δ i − 1 3 + δ i − 1 = O ( δ i − 1 ) . (131)
Step 2: decomposition of errors. We are now ready to analyze the convergence rate of (19) . Repeating the arguments in (81) -(87) , we obtain
x t i − x t i ∗ = A t i ( x t i − 1 − x t i − 1 ∗ ) + α t i 2 ( 1 − e − 2 δ i ) δ i D t i + E t i , 1 + E t i , 2 , (132) x_{t_i}-x_{t_i}^*=A_{t_i}(x_{t_{i-1}}-x_{t_{i-1}}^*)+\frac{\alpha_{t_i}}{2}(1-e^{-2\delta_i})\delta_iD_{t_i}+\mathcal{E}_{t_i,1}+\mathcal{E}_{t_i,2}, \tag*{(132)} x t i − x t i ∗ = A t i ( x t i − 1 − x t i − 1 ∗ ) + 2 α t i ( 1 − e − 2 δ i ) δ i D t i + E t i , 1 + E t i , 2 , (132)
where the matrix A t i A_{t_i} A t i is defined in (90) .
For i ≥ 2 i\ge2 i ≥ 2 , we apply the Taylor expansion at ( x t i − 2 , t i − 1 ) (x_{t_{i-2}},t_{i-1}) ( x t i − 2 , t i − 1 ) . From the definition of D t i D_{t_i} D t i ,
D t i = 1 δ i − 1 ( μ θ ( x t i − 1 , t i − 1 ) − μ θ ( x t i − 2 , t i − 2 ) ) = 1 δ i − 1 ( μ θ ( x t i − 1 , t i − 1 ) − μ θ ( x t i − 2 , t i − 1 ) + μ θ ( x t i − 2 , t i − 1 ) − μ θ ( x t i − 2 , t i − 2 ) ) = 1 δ i − 1 ∇ μ θ ( x t i − 2 , t i − 1 ) ⊤ ( x t i − 1 − x t i − 2 ) + 1 δ i − 1 O ( ∥ x t i − 1 − x t i − 2 ∥ 2 2 ) + ∂ ∂ λ μ θ ( x t i − 2 , t i − 2 ) + O ( δ i − 1 ) . \begin{aligned}
D_{t_i}=\frac{1}{\delta_{i-1}}\left(\mu_\theta(x_{t_{i-1}},t_{i-1})-\mu_\theta(x_{t_{i-2}},t_{i-2})\right) \\
&=\frac{1}{\delta_{i-1}}\left(\mu_\theta(x_{t_{i-1}},t_{i-1})-\mu_\theta(x_{t_{i-2}},t_{i-1})+\mu_\theta(x_{t_{i-2}},t_{i-1})-\mu_\theta(x_{t_{i-2}},t_{i-2})\right) \\
&=\frac{1}{\delta_{i-1}}\nabla\mu_\theta(x_{t_{i-2}},t_{i-1})^\top(x_{t_{i-1}}-x_{t_{i-2}})+\frac{1}{\delta_{i-1}}O(\lVert x_{t_{i-1}}-x_{t_{i-2}}\rVert_2^2)+\frac{\partial}{\partial\lambda}\mu_\theta(x_{t_{i-2}},t_{i-2})+O(\delta_{i-1}).
\end{aligned} D t i = δ i − 1 1 ( μ θ ( x t i − 1 , t i − 1 ) − μ θ ( x t i − 2 , t i − 2 ) ) = δ i − 1 1 ( μ θ ( x t i − 1 , t i − 1 ) − μ θ ( x t i − 2 , t i − 1 ) + μ θ ( x t i − 2 , t i − 1 ) − μ θ ( x t i − 2 , t i − 2 ) ) = δ i − 1 1 ∇ μ θ ( x t i − 2 , t i − 1 ) ⊤ ( x t i − 1 − x t i − 2 ) + δ i − 1 1 O (∥ x t i − 1 − x t i − 2 ∥ 2 2 ) + ∂ λ ∂ μ θ ( x t i − 2 , t i − 2 ) + O ( δ i − 1 ) .
Substituting the expression for x t i − 1 − x t i − 2 x_{t_{i-1}}-x_{t_{i-2}} x t i − 1 − x t i − 2 from (126) yields for i ≥ 2 i\ge2 i ≥ 2 ,
D t i = 1 δ i − 1 ( e − δ i − 1 σ t i − 1 σ t i − 2 − 1 ) ∇ μ θ ( x t i − 2 , t i − 1 ) ⊤ x t i − 2 + α t i − 1 δ i − 1 ( 1 − e − 2 δ i − 1 ) ∇ μ θ ( x t i − 2 , t i − 1 ) ⊤ μ θ ( x t i − 2 , t i − 2 ) + α t i − 1 2 ( 1 − e − 2 δ i − 1 ) ∇ μ θ ( x t i − 2 , t i − 1 ) ⊤ D t i − 1 + 1 δ i − 1 O ( ∥ x t i − 1 − x t i − 2 ∥ 2 2 ) + ∂ ∂ λ μ θ ( x t i − 2 , t i − 2 ) + O ( δ i − 1 ) + σ t i − 1 δ i − 1 1 − e − 2 δ i − 1 ∇ μ θ ( x t i − 2 , t i − 1 ) ⊤ z i − 1 = : E t i , 3 + E t i , 4 , (133) \begin{aligned}
D_{t_i}=\frac{1}{\delta_{i-1}}\left(e^{-\delta_{i-1}}\frac{\sigma_{t_{i-1}}}{\sigma_{t_{i-2}}}-1\right)\nabla\mu_\theta(x_{t_{i-2}},t_{i-1})^\top x_{t_{i-2}}+\frac{\alpha_{t_{i-1}}}{\delta_{i-1}}(1-e^{-2\delta_{i-1}})\nabla\mu_\theta(x_{t_{i-2}},t_{i-1})^\top\mu_\theta(x_{t_{i-2}},t_{i-2}) \\
&+\frac{\alpha_{t_{i-1}}}{2}(1-e^{-2\delta_{i-1}})\nabla\mu_\theta(x_{t_{i-2}},t_{i-1})^\top D_{t_{i-1}}+\frac{1}{\delta_{i-1}}O(\lVert x_{t_{i-1}}-x_{t_{i-2}}\rVert_2^2)+\frac{\partial}{\partial\lambda}\mu_\theta(x_{t_{i-2}},t_{i-2}) \\
&+O(\delta_{i-1})+\frac{\sigma_{t_{i-1}}}{\delta_{i-1}}\sqrt{1-e^{-2\delta_{i-1}}}\nabla\mu_\theta(x_{t_{i-2}},t_{i-1})^\top z_{i-1} \\
&=:\mathcal{E}_{t_i,3}+\mathcal{E}_{t_i,4}, \tag*{(133)}
\end{aligned} D t i = δ i − 1 1 ( e − δ i − 1 σ t i − 2 σ t i − 1 − 1 ) ∇ μ θ ( x t i − 2 , t i − 1 ) ⊤ x t i − 2 + δ i − 1 α t i − 1 ( 1 − e − 2 δ i − 1 ) ∇ μ θ ( x t i − 2 , t i − 1 ) ⊤ μ θ ( x t i − 2 , t i − 2 ) + 2 α t i − 1 ( 1 − e − 2 δ i − 1 ) ∇ μ θ ( x t i − 2 , t i − 1 ) ⊤ D t i − 1 + δ i − 1 1 O (∥ x t i − 1 − x t i − 2 ∥ 2 2 ) + ∂ λ ∂ μ θ ( x t i − 2 , t i − 2 ) + O ( δ i − 1 ) + δ i − 1 σ t i − 1 1 − e − 2 δ i − 1 ∇ μ θ ( x t i − 2 , t i − 1 ) ⊤ z i − 1 =: E t i , 3 + E t i , 4 , (133)
where the residual terms E t i , 3 \mathcal{E}_{t_i,3} E t i , 3 and E t i , 4 \mathcal{E}_{t_i,4} E t i , 4 are defined as
E t i , 3 : = 1 δ i − 1 ( e − δ i − 1 σ t i − 1 σ t i − 2 − 1 ) ∇ μ θ ( x t i − 2 , t i − 1 ) ⊤ x t i − 2 + α t i − 1 δ i − 1 ( 1 − e − 2 δ i − 1 ) ∇ μ θ ( x t i − 2 , t i − 1 ) ⊤ μ θ ( x t i − 2 , t i − 2 ) \mathcal{E}_{t_i,3}:=\frac{1}{\delta_{i-1}}\left(e^{-\delta_{i-1}}\frac{\sigma_{t_{i-1}}}{\sigma_{t_{i-2}}}-1\right)\nabla\mu_\theta(x_{t_{i-2}},t_{i-1})^\top x_{t_{i-2}}+\frac{\alpha_{t_{i-1}}}{\delta_{i-1}}(1-e^{-2\delta_{i-1}})\nabla\mu_\theta(x_{t_{i-2}},t_{i-1})^\top\mu_\theta(x_{t_{i-2}},t_{i-2}) E t i , 3 := δ i − 1 1 ( e − δ i − 1 σ t i − 2 σ t i − 1 − 1 ) ∇ μ θ ( x t i − 2 , t i − 1 ) ⊤ x t i − 2 + δ i − 1 α t i − 1 ( 1 − e − 2 δ i − 1 ) ∇ μ θ ( x t i − 2 , t i − 1 ) ⊤ μ θ ( x t i − 2 , t i − 2 )
α t i − 1 2 ( 1 − e − 2 δ i − 1 ) ∇ μ θ ( x t i − 2 , t i − 1 ) ⊤ D t i − 1 + 1 δ i − 1 O ( ∥ x t i − 1 − x t i − 2 ∥ 2 2 ) + ∂ ∂ λ μ θ ( x t i − 2 , t i − 2 ) + O ( δ i − 1 ) (134) \frac{\alpha_{t_{i-1}}}{2}(1-e^{-2\delta_{i-1}})\nabla\mu_\theta(x_{t_{i-2}},t_{i-1})^\top D_{t_{i-1}}+\frac{1}{\delta_{i-1}}O(\lVert x_{t_{i-1}}-x_{t_{i-2}}\rVert_2^2)+\frac{\partial}{\partial\lambda}\mu_\theta(x_{t_{i-2}},t_{i-2})+O(\delta_{i-1}) \tag*{(134)} 2 α t i − 1 ( 1 − e − 2 δ i − 1 ) ∇ μ θ ( x t i − 2 , t i − 1 ) ⊤ D t i − 1 + δ i − 1 1 O (∥ x t i − 1 − x t i − 2 ∥ 2 2 ) + ∂ λ ∂ μ θ ( x t i − 2 , t i − 2 ) + O ( δ i − 1 ) (134)
E t i , 4 : = σ t i − 1 δ i − 1 1 − e − 2 δ i − 1 ∇ μ θ ( x t i − 2 , t i − 1 ) ⊤ z i − 1 , (135) \mathcal{E}_{t_i,4}:=\frac{\sigma_{t_{i-1}}}{\delta_{i-1}}\sqrt{1-e^{-2\delta_{i-1}}}\nabla\mu_\theta(x_{t_{i-2}},t_{i-1})^\top z_{i-1}, \tag*{(135)} E t i , 4 := δ i − 1 σ t i − 1 1 − e − 2 δ i − 1 ∇ μ θ ( x t i − 2 , t i − 1 ) ⊤ z i − 1 , (135)
and E t 1 , 3 = E t 1 , 4 = 0 \mathcal{E}_{t_1,3}=\mathcal{E}_{t_1,4}=0 E t 1 , 3 = E t 1 , 4 = 0 . Denote c 1 = 0 c_1=0 c 1 = 0 and c i = α t i 2 ( 1 − e − 2 δ i ) δ i c_i=\frac{\alpha_{t_i}}{2}(1-e^{-2\delta_i})\delta_i c i = 2 α t i ( 1 − e − 2 δ i ) δ i for i ≥ 2 i\ge2 i ≥ 2 . Substituting (133) into (132) gives
x t i − x t i ⋆ = A t i ( x t i − 1 − x t i − 1 ⋆ ) + E t i , 1 + E t i , 2 + c i E t i , 3 + c i E t i , 4 x_{t_i}-x^\star_{t_i}=A_{t_i}(x_{t_{i-1}}-x^\star_{t_{i-1}})+\mathcal{E}_{t_i,1}+\mathcal{E}_{t_i,2}+c_i\mathcal{E}_{t_i,3}+c_i\mathcal{E}_{t_i,4} x t i − x t i ⋆ = A t i ( x t i − 1 − x t i − 1 ⋆ ) + E t i , 1 + E t i , 2 + c i E t i , 3 + c i E t i , 4
where we have used the bound
α t i 2 ( 1 − e − 2 δ i ) δ i ≤ α t i δ i 2 = O ( δ i 2 ) . \frac{\alpha_{t_i}}{2}(1-e^{-2\delta_i})\delta_i\leq\alpha_{t_i}\delta_i^2=O(\delta_i^2). 2 α t i ( 1 − e − 2 δ i ) δ i ≤ α t i δ i 2 = O ( δ i 2 ) .
By recursion, we get the final decomposition
x t M − x t M ⋆ = ∑ m = 1 M ∏ j = m + 1 M A t j ( E t m , 1 + E t m , 2 + c m E t m , 3 + c m E t m , 4 ) . (136) x_{t_M}-x^\star_{t_M}=\sum_{m=1}^{M}\prod_{j=m+1}^{M}A_{t_j}\left(\mathcal{E}_{t_m,1}+\mathcal{E}_{t_m,2}+c_m\mathcal{E}_{t_m,3}+c_m\mathcal{E}_{t_m,4}\right). \tag*{(136)} x t M − x t M ⋆ = m = 1 ∑ M j = m + 1 ∏ M A t j ( E t m , 1 + E t m , 2 + c m E t m , 3 + c m E t m , 4 ) . (136)
Taking expectations and applying the uniform product bound (94) , we obtain
E [ ∥ x t M − x t M ⋆ ∥ 2 ] ≲ ∑ m = 1 M E [ ∥ E t m , 1 ∥ 2 ] + E [ ∥ ∑ m = 1 M ∏ j = m + 1 M A t j E t m , 2 ∥ 2 ] + ∑ m = 1 M δ m 2 E [ ∥ E t m , 3 ∥ 2 ] + E [ ∥ ∑ m = 2 M c m ∏ j = m + 1 M A t j E t m , 4 ∥ 2 ] . (137) \mathbb{E}[\lVert x_{t_M}-x^\star_{t_M}\rVert_2]\lesssim\sum_{m=1}^{M}\mathbb{E}[\lVert\mathcal{E}_{t_m,1}\rVert_2]+\mathbb{E}\left[\left\lVert\sum_{m=1}^{M}\prod_{j=m+1}^{M}A_{t_j}\mathcal{E}_{t_m,2}\right\rVert_2\right]
+\sum_{m=1}^{M}\delta_m^2\mathbb{E}[\lVert\mathcal{E}_{t_m,3}\rVert_2]+\mathbb{E}\left[\left\lVert\sum_{m=2}^{M}c_m\prod_{j=m+1}^{M}A_{t_j}\mathcal{E}_{t_m,4}\right\rVert_2\right]. \tag*{(137)} E [∥ x t M − x t M ⋆ ∥ 2 ] ≲ m = 1 ∑ M E [∥ E t m , 1 ∥ 2 ] + E m = 1 ∑ M j = m + 1 ∏ M A t j E t m , 2 2 + m = 1 ∑ M δ m 2 E [∥ E t m , 3 ∥ 2 ] + E m = 2 ∑ M c m j = m + 1 ∏ M A t j E t m , 4 2 . (137)
Step 3: control each term separately. We now bound the four terms on the right-hand side of (137) separately.
∑ m = 1 M E [ ∥ E t m , 1 ∥ 2 ] = O ( δ ) , (138) \sum_{m=1}^{M}\mathbb{E}[\lVert\mathcal{E}_{t_m,1}\rVert_2]=O(\delta), \tag*{(138)} m = 1 ∑ M E [∥ E t m , 1 ∥ 2 ] = O ( δ ) , (138)
Recalling the analysis in Step 4 of Section C.1, it suffices to prove the second-moment bound
E [ ∥ x t m − x t m ⋆ ∥ 2 2 ] = O ( δ ) . \mathbb{E}[\lVert x_{t_m}-x^\star_{t_m}\rVert_2^2]=O(\delta). E [∥ x t m − x t m ⋆ ∥ 2 2 ] = O ( δ ) .
To this end, we repeat the arguments leading to (111) , which now include the additional drift-difference term:
∥ x t m − x t m ⋆ ∥ 2 2 ≤ σ t m − 1 σ t m − 1 e − δ m ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 + α t m α t m − 1 L x ( 1 − e − 2 δ m ) ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 + 2 B t m + 1 2 δ m α t m ( 1 − e − 2 δ m ) ∥ D t m ∥ 2 , (139) \lVert x_{t_m}-x^\star_{t_m}\rVert_2^2\leq\frac{\sigma_{t_m-1}}{\sigma_{t_{m-1}}}e^{-\delta_m}\lVert x_{t_{m-1}}-x^\star_{t_{m-1}}\rVert_2^2+\frac{\alpha_{t_m}}{\alpha_{t_{m-1}}}L_x(1-e^{-2\delta_m})\lVert x_{t_{m-1}}-x^\star_{t_{m-1}}\rVert_2^2+2B_{t_m}+\frac{1}{2}\delta_m\alpha_{t_m}(1-e^{-2\delta_m})\lVert D_{t_m}\rVert_2, \tag*{(139)} ∥ x t m − x t m ⋆ ∥ 2 2 ≤ σ t m − 1 σ t m − 1 e − δ m ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 + α t m − 1 α t m L x ( 1 − e − 2 δ m ) ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 + 2 B t m + 2 1 δ m α t m ( 1 − e − 2 δ m ) ∥ D t m ∥ 2 , (139)
where B t m B_{t_m} B t m is defined in (112) . Repeating the derivation of (115) , we obtain
E [ ∥ x t m − x t m ⋆ ∥ 2 2 ] ≤ ( 1 + 2 α t m L x δ m α t m − 1 ) 2 E [ ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] + 2 ( 1 + 2 α t m L x δ m α t m − 1 ) E [ ( 2 B t m + O ( δ m 2 ) ∥ D t m ∥ 2 ) ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 ] + E [ ( 2 B t m + O ( δ m 2 ) ∥ D t m ∥ 2 ) 2 ] ≤ ( 1 + 2 α t m L x δ m α t m − 1 ) 2 E [ ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] + 2 ( 1 + 2 α t m L x δ m α t m − 1 ) O ( δ m 1.5 ) E [ ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] + O ( δ m 3 ) ≤ ( 1 + 5 α t m L x δ m α t m − 1 + O ( δ m ) ) E [ ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] + O ( δ m 2 ) . (140) \begin{aligned}
\mathbb{E}[\lVert x_{t_m}-x^\star_{t_m}\rVert_2^2]
&\leq\left(1+\frac{2\alpha_{t_m}L_x\delta_m}{\alpha_{t_{m-1}}}\right)^2\mathbb{E}[\lVert x_{t_{m-1}}-x^\star_{t_{m-1}}\rVert_2^2]\\
&\quad+2\left(1+\frac{2\alpha_{t_m}L_x\delta_m}{\alpha_{t_{m-1}}}\right)\mathbb{E}\left[(2B_{t_m}+O(\delta_m^2)\lVert D_{t_m}\rVert_2)\lVert x_{t_{m-1}}-x^\star_{t_{m-1}}\rVert_2\right]\\
&\quad+\mathbb{E}[(2B_{t_m}+O(\delta_m^2)\lVert D_{t_m}\rVert_2)^2]\\
&\leq\left(1+\frac{2\alpha_{t_m}L_x\delta_m}{\alpha_{t_{m-1}}}\right)^2\mathbb{E}[\lVert x_{t_{m-1}}-x^\star_{t_{m-1}}\rVert_2^2] \\
&+2\left(1+\frac{2\alpha_{t_m}L_x\delta_m}{\alpha_{t_{m-1}}}\right)O(\delta_m^{1.5})\sqrt{\mathbb{E}\left[\left\|x_{t_{m-1}}-x^\star_{t_{m-1}}\right\|_2^2\right]}+O(\delta_m^3) \\
&\leq\left(1+\frac{5\alpha_{t_m}L_x\delta_m}{\alpha_{t_{m-1}}}+O(\delta_m)\right)\mathbb{E}\left[\left\|x_{t_{m-1}}-x^\star_{t_{m-1}}\right\|_2^2\right]+O(\delta_m^2). \tag*{(140)}
\end{aligned} E [∥ x t m − x t m ⋆ ∥ 2 2 ] ≤ ( 1 + α t m − 1 2 α t m L x δ m ) 2 E [∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] + 2 ( 1 + α t m − 1 2 α t m L x δ m ) E [ ( 2 B t m + O ( δ m 2 ) ∥ D t m ∥ 2 ) ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 ] + E [( 2 B t m + O ( δ m 2 ) ∥ D t m ∥ 2 ) 2 ] ≤ ( 1 + α t m − 1 2 α t m L x δ m ) 2 E [∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] + 2 ( 1 + α t m − 1 2 α t m L x δ m ) O ( δ m 1.5 ) E [ x t m − 1 − x t m − 1 ⋆ 2 2 ] + O ( δ m 3 ) ≤ ( 1 + α t m − 1 5 α t m L x δ m + O ( δ m ) ) E [ x t m − 1 − x t m − 1 ⋆ 2 2 ] + O ( δ m 2 ) . (140)
where (a) holds since
E [ ( 2 B t m + O ( δ m 2 ) ∥ D t m ∥ 2 ) 2 ] ≲ E [ B t m 2 ] + δ m 4 E [ ∥ D t m ∥ 2 2 ] = O ( δ m 3 ) . \mathbb{E}\left[(2B_{t_m}+O(\delta_m^2)\|D_{t_m}\|_2)^2\right]\lesssim\mathbb{E}[B_{t_m}^2]+\delta_m^4\mathbb{E}[\|D_{t_m}\|_2^2]=O(\delta_m^3). E [ ( 2 B t m + O ( δ m 2 ) ∥ D t m ∥ 2 ) 2 ] ≲ E [ B t m 2 ] + δ m 4 E [ ∥ D t m ∥ 2 2 ] = O ( δ m 3 ) .
By recursion, we have
E [ ∥ x t m − x t m ⋆ ∥ 2 2 ] ≲ ∑ i = 1 m exp ( ∑ j = i + 1 m ( 5 α t j L x δ j α t j − 1 + O ( δ j ) ) ) δ i 2 = O ( δ ) . (141) \mathbb{E}\left[\|x_{t_m}-x^\star_{t_m}\|_2^2\right]\lesssim\sum_{i=1}^{m}\exp\left(\sum_{j=i+1}^{m}\left(\frac{5\alpha_{t_j}L_x\delta_j}{\alpha_{t_{j-1}}}+O(\delta_j)\right)\right)\delta_i^2=O(\delta). \tag*{(141)} E [ ∥ x t m − x t m ⋆ ∥ 2 2 ] ≲ i = 1 ∑ m exp ( j = i + 1 ∑ m ( α t j − 1 5 α t j L x δ j + O ( δ j ) ) ) δ i 2 = O ( δ ) . (141)
This establishes (138) .
Bounding the second term. Recall the definition of E t m , 2 E_{t_m,2} E t m , 2 (cf. (89) ), which is independent of the update rule for x t i x_{t_i} x t i . Therefore, the analysis in Appendix § 11 .§ 1 remains valid, and we have
E [ ∥ ∑ m = 1 M ∏ i = m + 1 M A t i E t m , 2 ∥ 2 ] = O ( δ ) . (142) \mathbb{E}\left[\left\|\sum_{m=1}^{M}\prod_{i=m+1}^{M}A_{t_i}E_{t_m,2}\right\|_2\right]=O(\delta). \tag*{(142)} E [ m = 1 ∑ M i = m + 1 ∏ M A t i E t m , 2 2 ] = O ( δ ) . (142)
Bounding the third term. For the definition of E t i , 3 E_{t_i,3} E t i , 3 (cf. (134) ), we have
E [ ∥ E t m , 3 ∥ 2 ] ≤ 1 δ m − 1 ∣ e − δ m − 1 σ t m − 1 σ t m − 2 − 1 ∣ L x α t m − 1 E [ ∥ x t m − 2 ∥ 2 ] + α t m − 1 δ m − 1 ( 1 − e − 2 δ m − 1 ) L x L μ α t m − 1 + α t m − 1 2 ( 1 − e − 2 δ m − 1 ) L x α t m − 1 E [ ∥ D t m − 1 ∥ 2 ] + 1 δ m − 1 O ( E [ ∥ x t m − 1 − x t m − 2 ∥ 2 2 ] ) + L t + O ( δ m − 1 ) \begin{aligned}
\mathbb{E}[\|E_{t_m,3}\|_2]\leq\frac{1}{\delta_{m-1}}\left|e^{-\delta_{m-1}}\frac{\sigma_{t_{m-1}}}{\sigma_{t_{m-2}}}-1\right|\frac{L_x}{\alpha_{t_{m-1}}}\mathbb{E}[\|x_{t_{m-2}}\|_2]+\frac{\alpha_{t_{m-1}}}{\delta_{m-1}}(1-e^{-2\delta_{m-1}})\frac{L_xL_\mu}{\alpha_{t_{m-1}}} \\
&+\frac{\alpha_{t_{m-1}}}{2}(1-e^{-2\delta_{m-1}})\frac{L_x}{\alpha_{t_{m-1}}}\mathbb{E}[\|D_{t_{m-1}}\|_2]+\frac{1}{\delta_{m-1}}O(\mathbb{E}[\|x_{t_{m-1}}-x_{t_{m-2}}\|_2^2])+L_t+O(\delta_{m-1})
\end{aligned} E [ ∥ E t m , 3 ∥ 2 ] ≤ δ m − 1 1 e − δ m − 1 σ t m − 2 σ t m − 1 − 1 α t m − 1 L x E [ ∥ x t m − 2 ∥ 2 ] + δ m − 1 α t m − 1 ( 1 − e − 2 δ m − 1 ) α t m − 1 L x L μ + 2 α t m − 1 ( 1 − e − 2 δ m − 1 ) α t m − 1 L x E [ ∥ D t m − 1 ∥ 2 ] + δ m − 1 1 O ( E [ ∥ x t m − 1 − x t m − 2 ∥ 2 2 ]) + L t + O ( δ m − 1 )
≲ (a) E [ ∥ x t m − 2 ∥ 2 ] + O ( 1 ) + O ( δ m − 1 ) E [ ∥ D t m − 1 ∥ 2 ] + 1 δ m − 1 E [ ∥ x t m − 1 − x t m − 2 ∥ 2 2 ] ≅ (b) O ( 1 ) , \underset{\text{(a)}}{\lesssim}\mathbb{E}[\|x_{t_{m-2}}\|_2]+O(1)+O(\delta_{m-1})\mathbb{E}[\|D_{t_{m-1}}\|_2]+\frac{1}{\delta_{m-1}}\mathbb{E}[\|x_{t_{m-1}}-x_{t_{m-2}}\|_2^2]\underset{\text{(b)}}{\cong}O(1), (a) ≲ E [ ∥ x t m − 2 ∥ 2 ] + O ( 1 ) + O ( δ m − 1 ) E [ ∥ D t m − 1 ∥ 2 ] + δ m − 1 1 E [ ∥ x t m − 1 − x t m − 2 ∥ 2 2 ] (b) ≅ O ( 1 ) ,
where (a) uses the elementary bound (84) , and (b) applies the crude estimates (124) , (130) , and (131) from Step 1. Consequently, the third term in (137) is bounded by
∑ m = 1 M δ m 2 E [ ∥ E t m , 3 ∥ 2 ] ≲ ∑ m = 1 M δ m 2 = O ( δ ) . (143) \sum_{m=1}^{M}\delta_m^2\mathbb{E}[\|E_{t_m,3}\|_2]\lesssim\sum_{m=1}^{M}\delta_m^2=O(\delta). \tag*{(143)} m = 1 ∑ M δ m 2 E [ ∥ E t m , 3 ∥ 2 ] ≲ m = 1 ∑ M δ m 2 = O ( δ ) . (143)
Bounding the fourth term. We use the cumulative matrices P i P_i P i and the filtration introduced in Section C.1. Since the Jacobian in (135) is evaluated at x t m − 2 x_{t_{m-2}} x t m − 2 , it is F t m − 2 \mathcal{F}_{t_{m-2}} F t m − 2 -measurable. Independence of z m − 1 z_{m-1} z m − 1 and F t m − 2 \mathcal{F}_{t_{m-2}} F t m − 2 gives
E [ E t m , 4 ∣ F t m − 2 ] = 0 , E [ ∥ E t m , 4 ∥ 2 2 ∣ F t m − 2 ] ≤ C δ m − 1 , m ≥ 2. (144) \mathbb{E}[E_{t_m,4}\mid\mathcal{F}_{t_{m-2}}]=0,\qquad\mathbb{E}[\|E_{t_m,4}\|_2^2\mid\mathcal{F}_{t_{m-2}}]\leq\frac{C}{\delta_{m-1}},\qquad m\geq2. \tag*{(144)} E [ E t m , 4 ∣ F t m − 2 ] = 0 , E [ ∥ E t m , 4 ∥ 2 2 ∣ F t m − 2 ] ≤ δ m − 1 C , m ≥ 2. (144)
The matrix P m − 1 P_m^{-1} P m − 1 still depends on the preceding noise increment. We therefore write
P m − 1 c m E t m , 4 = P m − 1 − 1 c m E t m , 4 + R m , P_m^{-1}c_mE_{t_m,4}=P_{m-1}^{-1}c_mE_{t_m,4}+R_m, P m − 1 c m E t m , 4 = P m − 1 − 1 c m E t m , 4 + R m ,
R m = c m ( P m − 1 − P m − 1 − 1 ) E t m , 4 . R_m=c_m(P_m^{-1}-P_{m-1}^{-1})E_{t_m,4}. R m = c m ( P m − 1 − P m − 1 − 1 ) E t m , 4 .
Using the identity ∏ i = m + 1 M A t i = P M P m − 1 \prod_{i=m+1}^{M}A_{t_i}=P_MP_m^{-1} ∏ i = m + 1 M A t i = P M P m − 1 and the bound for P M P_M P M yields
E [ ∥ ∑ m = 2 M c m ∏ i = m + 1 M A t i E t m , 4 ∥ 2 ] = E [ ∥ P M ∑ m = 2 M P m − 1 c m E t m , 4 ∥ 2 ] ≤ C E ∥ ∑ m = 2 M P m − 1 − 1 c m E t m , 4 ∥ 2 + C E [ ∑ m = 2 M ∥ R m ∥ 2 ] . (145) \begin{aligned}
\mathbb{E}\left[\left\|\sum_{m=2}^{M}c_m\prod_{i=m+1}^{M}A_{t_i}E_{t_m,4}\right\|_2\right]=\mathbb{E}\left[\left\|P_M\sum_{m=2}^{M}P_m^{-1}c_mE_{t_m,4}\right\|_2\right] \\
&\leq CE\left\Vert\sum_{m=2}^{M}P_{m-1}^{-1}c_m\mathcal{E}_{t_m,4}\right\Vert_2+CE\left[\sum_{m=2}^{M}\lVert R_m\rVert_2\right]. \tag*{(145)}
\end{aligned} E [ m = 2 ∑ M c m i = m + 1 ∏ M A t i E t m , 4 2 ] = E [ P M m = 2 ∑ M P m − 1 c m E t m , 4 2 ] ≤ C E m = 2 ∑ M P m − 1 − 1 c m E t m , 4 2 + C E [ m = 2 ∑ M ∥ R m ∥ 2 ] . (145)
By the bounds for P i − 1 P_i^{-1} P i − 1 and A t i − 1 A_{t_i}^{-1} A t i − 1 established previously, we have
∥ P m − 1 − P m − 1 − 1 ∥ o p = ∥ P m − 1 − 1 ( A t m − 1 − I d ) ∥ o p ≤ C δ m . \lVert P_{m}^{-1}-P_{m-1}^{-1}\rVert_{\mathrm{op}}=\lVert P_{m-1}^{-1}(A_{t_m}^{-1}-I_d)\rVert_{\mathrm{op}}\leq C\delta_m. ∥ P m − 1 − P m − 1 − 1 ∥ op = ∥ P m − 1 − 1 ( A t m − 1 − I d ) ∥ op ≤ C δ m .
Since c m = O ( δ m 2 ) c_m=O(\delta_m^2) c m = O ( δ m 2 ) and the statistical property in (144) , the remainder satisfies
E [ ∑ m = 2 M ∥ R m ∥ 2 ] ≤ ∑ m = 2 M E ∥ R m ∥ 2 ≤ C ∑ m = 2 M δ m 3 δ m − 1 ≤ C δ 3 / 2 ∑ m = 2 M δ m = O ( δ 3 / 2 ) . (146) \mathbb{E}\left[\sum_{m=2}^{M}\lVert R_m\rVert_2\right]\leq\sum_{m=2}^{M}\mathbb{E}\lVert R_m\rVert_2\leq C\sum_{m=2}^{M}\frac{\delta_m^3}{\sqrt{\delta_{m-1}}}\leq C\delta^{3/2}\sum_{m=2}^{M}\delta_m=O(\delta^{3/2}). \tag*{(146)} E [ m = 2 ∑ M ∥ R m ∥ 2 ] ≤ m = 2 ∑ M E ∥ R m ∥ 2 ≤ C m = 2 ∑ M δ m − 1 δ m 3 ≤ C δ 3/2 m = 2 ∑ M δ m = O ( δ 3/2 ) . (146)
Moreover, P m − 1 − 1 P_{m-1}^{-1} P m − 1 − 1 is F t m − 2 \mathcal{F}_{t_{m-2}} F t m − 2 -measurable. Hence the first term in the right-hand-side of (145) satisfies
E ∥ ∑ m = 2 M P m − 1 − 1 c m E t m , 4 ∥ 2 2 = ∑ m = 2 M E ∥ P m − 1 − 1 c m E t m , 4 ∥ 2 2 (147) \mathbb{E}\left\Vert\sum_{m=2}^{M}P_{m-1}^{-1}c_m\mathcal{E}_{t_m,4}\right\Vert_2^2=\sum_{m=2}^{M}\mathbb{E}\left\Vert P_{m-1}^{-1}c_m\mathcal{E}_{t_m,4}\right\Vert_2^2 \tag*{(147)} E m = 2 ∑ M P m − 1 − 1 c m E t m , 4 2 2 = m = 2 ∑ M E P m − 1 − 1 c m E t m , 4 2 2 (147)
≤ C ∑ m = 2 M δ m 4 δ m − 1 ≤ C ∑ m = 2 M δ m 3 = O ( δ 2 ) , (148) \leq C\sum_{m=2}^{M}\frac{\delta_m^4}{\delta_{m-1}}\leq C\sum_{m=2}^{M}\delta_m^3=O(\delta^2), \tag*{(148)} ≤ C m = 2 ∑ M δ m − 1 δ m 4 ≤ C m = 2 ∑ M δ m 3 = O ( δ 2 ) , (148)
where the first equality follows from E [ E t m , 4 ∣ F t m − 2 ] = 0 \mathbb{E}[\mathcal{E}_{t_m,4}\mid\mathcal{F}_{t_{m-2}}]=0 E [ E t m , 4 ∣ F t m − 2 ] = 0 and the F t m − 2 \mathcal{F}_{t_{m-2}} F t m − 2 -measurability of P m − 1 − 1 P_{m-1}^{-1} P m − 1 − 1 . Collecting (145) , (146) , and (147) yields
E ∥ ∑ m = 2 M c m ∏ i = m + 1 M A t i E t m , 4 ∥ 2 ≤ C ( E ∥ ∑ m = 2 M P m − 1 − 1 c m E t m , 4 ∥ 2 2 ) 1 / 2 + C ∑ m = 2 M E ∥ R m ∥ 2 = O ( δ ) + O ( δ 3 / 2 ) = O ( δ ) . (149) \begin{aligned}
\mathbb{E}\left\Vert\sum_{m=2}^{M}c_m\prod_{i=m+1}^{M}A_{t_i}\mathcal{E}_{t_m,4}\right\Vert_2
\leq C\left(\mathbb{E}\left\Vert\sum_{m=2}^{M}P_{m-1}^{-1}c_m\mathcal{E}_{t_m,4}\right\Vert_2^2\right)^{1/2}
+C\sum_{m=2}^{M}\mathbb{E}\lVert R_m\rVert_2 \\
&=O(\delta)+O(\delta^{3/2})=O(\delta). \tag*{(149)}
\end{aligned} E m = 2 ∑ M c m i = m + 1 ∏ M A t i E t m , 4 2 ≤ C E m = 2 ∑ M P m − 1 − 1 c m E t m , 4 2 2 1/2 + C m = 2 ∑ M E ∥ R m ∥ 2 = O ( δ ) + O ( δ 3/2 ) = O ( δ ) . (149)
Finally, collecting (138) , (142) , (143) , and (149) into (137) yields the desired result:
E [ ∥ x t M − x t M ⋆ ∥ 2 2 ] = O ( δ ) . \mathbb{E}[\lVert x_{t_M}-x_{t_M}^{\star}\rVert_2^2]=O(\delta). E [∥ x t M − x t M ⋆ ∥ 2 2 ] = O ( δ ) .
This completes the proof.
Proof of (124) . Recall that we have already established E [ ∥ x t m ⋆ ∥ 2 2 ] = O ( 1 ) \mathbb{E}[\lVert x_{t_m}^{\star}\rVert_2^2]=O(1) E [∥ x t m ⋆ ∥ 2 2 ] = O ( 1 ) in (80) . Therefore, it suffices to prove
E [ ∥ x t m − x t m ⋆ ∥ 2 2 ] = O ( 1 ) . \mathbb{E}[\lVert x_{t_m}-x_{t_m}^{\star}\rVert_2^2]=O(1). E [∥ x t m − x t m ⋆ ∥ 2 2 ] = O ( 1 ) .
From the decomposition in (140) , we have
E [ ∥ x t m − x t m ⋆ ∥ 2 2 ] ≤ ( 1 + 2 α t m L x δ m α t m − 1 ) E [ ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] + 2 ( 1 + 2 α t m L x δ m α t m − 1 ) E [ ( 2 B t m + O ( δ m 2 ) ∥ D t m ∥ 2 ) ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] + E [ ( 2 B t m + O ( δ m 2 ) ∥ D t m ∥ 2 ) 2 ] . (150) \begin{aligned}
\mathbb{E}[\lVert x_{t_m}-x_{t_m}^{\star}\rVert_2^2]
&\leq\left(1+\frac{2\alpha_{t_m}L_x\delta_m}{\alpha_{t_{m-1}}}\right)\mathbb{E}[\lVert x_{t_{m-1}}-x_{t_{m-1}}^{\star}\rVert_2^2]\\
&\quad+2\left(1+\frac{2\alpha_{t_m}L_x\delta_m}{\alpha_{t_{m-1}}}\right)\mathbb{E}\left[(2B_{t_m}+O(\delta_m^2)\lVert D_{t_m}\rVert_2)\lVert x_{t_{m-1}}-x_{t_{m-1}}^{\star}\rVert_2^2\right]\\
&\quad+\mathbb{E}[(2B_{t_m}+O(\delta_m^2)\lVert D_{t_m}\rVert_2)^2]. \tag*{(150)}
\end{aligned} E [∥ x t m − x t m ⋆ ∥ 2 2 ] ≤ ( 1 + α t m − 1 2 α t m L x δ m ) E [∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] + 2 ( 1 + α t m − 1 2 α t m L x δ m ) E [ ( 2 B t m + O ( δ m 2 ) ∥ D t m ∥ 2 ) ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] + E [( 2 B t m + O ( δ m 2 ) ∥ D t m ∥ 2 ) 2 ] . (150)
By Assumption A1, the drift term is uniformly bounded by L μ L_\mu L μ , i.e., ∥ μ θ ( x t m − 1 , t m − 1 ) ∥ 2 ≤ L μ \lVert\mu_\theta(x_{t_{m-1}},t_{m-1})\rVert_2\leq L_\mu ∥ μ θ ( x t m − 1 , t m − 1 ) ∥ 2 ≤ L μ . Consequently,
∥ D t m ∥ 2 ≤ 1 δ m − 1 ( ∥ μ θ ( x t m − 1 , t m − 1 ) ∥ 2 + ∥ μ θ ( x t m − 2 , t m − 2 ) ∥ 2 ) ≤ 2 L μ δ m − 1 . \lVert D_{t_m}\rVert_2\leq\frac{1}{\delta_{m-1}}\left(\lVert\mu_\theta(x_{t_{m-1}},t_{m-1})\rVert_2+\lVert\mu_\theta(x_{t_{m-2}},t_{m-2})\rVert_2\right)\leq\frac{2L_\mu}{\delta_{m-1}}. ∥ D t m ∥ 2 ≤ δ m − 1 1 ( ∥ μ θ ( x t m − 1 , t m − 1 ) ∥ 2 + ∥ μ θ ( x t m − 2 , t m − 2 ) ∥ 2 ) ≤ δ m − 1 2 L μ .
Using the assumption δ m / δ m − 1 = O ( 1 ) \delta_m/\delta_{m-1}=O(1) δ m / δ m − 1 = O ( 1 ) and the bound E [ B t m 2 ] = O ( δ m 3 ) \mathbb{E}[B_{t_m}^2]=O(\delta_m^3) E [ B t m 2 ] = O ( δ m 3 ) in (114) , (which is independent of the update rule for x t m x_{t_m} x t m ), we obtain
E [ ( 2 B t m + O ( δ m 2 ) ∥ D t m ∥ 2 ) 2 ] ≲ E [ B t m 2 ] + δ m 4 E [ ∥ D t m ∥ 2 2 ] ≲ E [ B t m 2 ] + δ m 4 δ m − 1 2 = O ( δ m 2 ) . \mathbb{E}\left[(2B_{t_m}+O(\delta_m^2)\lVert D_{t_m}\rVert_2)^2\right]\lesssim\mathbb{E}[B_{t_m}^2]+\delta_m^4\mathbb{E}[\lVert D_{t_m}\rVert_2^2]\lesssim\mathbb{E}[B_{t_m}^2]+\frac{\delta_m^4}{\delta_{m-1}^2}=O(\delta_m^2). E [ ( 2 B t m + O ( δ m 2 ) ∥ D t m ∥ 2 ) 2 ] ≲ E [ B t m 2 ] + δ m 4 E [∥ D t m ∥ 2 2 ] ≲ E [ B t m 2 ] + δ m − 1 2 δ m 4 = O ( δ m 2 ) .
Substituting this estimate into (150) yields
E [ ∥ x t m − x t m ⋆ ∥ 2 2 ] ≤ ( 1 + 5 α t m L x δ m α t m − 1 + O ( δ m ) ) 2 E [ ∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] + O ( δ m 3 ) E [ ∥ D t m ∥ 2 2 ] + O ( 1 δ m ) E [ B t m 2 ] ≲ ∑ i = 1 m exp ( ∑ j = i + 1 m ( 5 α t j L x δ j α t j − 1 + O ( δ j ) ) ) δ i = O ( 1 ) . (151) \begin{aligned}
\mathbb{E}[\lVert x_{t_m}-x_{t_m}^\star\rVert_2^2]
&\leq\left(1+\frac{5\alpha_{t_m}L_x\delta_m}{\alpha_{t_m-1}}+O(\delta_m)\right)^2\mathbb{E}[\lVert x_{t_{m-1}}-x_{t_{m-1}}^\star\rVert_2^2]\\
&\quad+O(\delta_m^3)\mathbb{E}[\lVert D_{t_m}\rVert_2^2]+O\left(\frac{1}{\delta_m}\right)\mathbb{E}[B_{t_m}^2]\\
&\lesssim\sum_{i=1}^{m}\exp\left(\sum_{j=i+1}^{m}\left(\frac{5\alpha_{t_j}L_x\delta_j}{\alpha_{t_{j-1}}}+O(\delta_j)\right)\right)\delta_i=O(1). \tag*{(151)}
\end{aligned} E [∥ x t m − x t m ⋆ ∥ 2 2 ] ≤ ( 1 + α t m − 1 5 α t m L x δ m + O ( δ m ) ) 2 E [∥ x t m − 1 − x t m − 1 ⋆ ∥ 2 2 ] + O ( δ m 3 ) E [∥ D t m ∥ 2 2 ] + O ( δ m 1 ) E [ B t m 2 ] ≲ i = 1 ∑ m exp ( j = i + 1 ∑ m ( α t j − 1 5 α t j L x δ j + O ( δ j ) ) ) δ i = O ( 1 ) . (151)
This completes the proof of (124) .
Proof of stochastic samplers convergence lower bounds (Theorem 3)
Assume that x 0 ∼ N ( 0 , γ 2 I d ) x_0\sim\mathcal{N}(0,\gamma^2 I_d) x 0 ∼ N ( 0 , γ 2 I d ) follows the Gaussian distribution with isotropic covariance matrix, where γ > 0 \gamma>0 γ > 0 is fixed. In this case, the score function reduces to
s t ⋆ ( x ) = ∇ log q t ( x ) = − x α t 2 γ 2 + σ t 2 = − 1 α t 2 γ 2 + e − 2 λ t x . s_t^\star(x)=\nabla\log q_t(x)=-\frac{x}{\alpha_t^2\gamma^2+\sigma_t^2}=-\frac{1}{\alpha_t^2\gamma^2+e^{-2\lambda_t}}x. s t ⋆ ( x ) = ∇ log q t ( x ) = − α t 2 γ 2 + σ t 2 x = − α t 2 γ 2 + e − 2 λ t 1 x .
Then the data predictor is given by
μ θ ( x , t ) = 1 α t ( x − σ t ϵ θ ( x , t ) ) = 1 α t ( x + σ t 2 s t ⋆ ( x ) ) = 1 α t ( 1 − e − 2 λ t γ 2 + e − 2 λ t ) x = γ 2 γ 2 + e − 2 λ t x α t . (152) \begin{aligned}
\mu_\theta(x,t)&=\frac{1}{\alpha_t}\left(x-\sigma_t\epsilon_\theta(x,t)\right)=\frac{1}{\alpha_t}\left(x+\sigma_t^2s_t^\star(x)\right)\\
&=\frac{1}{\alpha_t}\left(1-\frac{e^{-2\lambda_t}}{\gamma^2+e^{-2\lambda_t}}\right)x=\frac{\gamma^2}{\gamma^2+e^{-2\lambda_t}}\frac{x}{\alpha_t}. \tag*{(152)}
\end{aligned} μ θ ( x , t ) = α t 1 ( x − σ t ϵ θ ( x , t ) ) = α t 1 ( x + σ t 2 s t ⋆ ( x ) ) = α t 1 ( 1 − γ 2 + e − 2 λ t e − 2 λ t ) x = γ 2 + e − 2 λ t γ 2 α t x . (152)
Substituting into the SDE trajectory (31) yields
x t ⋆ = σ t σ t 0 e − ( λ t − λ t 0 ) x t 0 ⋆ + 2 α t ∫ λ t 0 λ t γ 2 e − 2 ( λ t − λ ) x t ( λ ) ⋆ γ 2 + e − 2 λ 1 α t ( λ ) d λ + 2 σ t ∫ λ t 0 λ t e − ( λ t − λ ) d ω ‾ t ( λ ) . (153) x_t^\star=\frac{\sigma_t}{\sigma_{t_0}}e^{-(\lambda_t-\lambda_{t_0})}x_{t_0}^\star+2\alpha_t\int_{\lambda_{t_0}}^{\lambda_t}\frac{\gamma^2e^{-2(\lambda_t-\lambda)}x_{t(\lambda)}^\star}{\gamma^2+e^{-2\lambda}}\frac{1}{\alpha_{t(\lambda)}}\,\mathrm{d}\lambda+\sqrt{2}\sigma_t\int_{\lambda_{t_0}}^{\lambda_t}e^{-(\lambda_t-\lambda)}\,\mathrm{d}\overline{\omega}_t(\lambda). \tag*{(153)} x t ⋆ = σ t 0 σ t e − ( λ t − λ t 0 ) x t 0 ⋆ + 2 α t ∫ λ t 0 λ t γ 2 + e − 2 λ γ 2 e − 2 ( λ t − λ ) x t ( λ ) ⋆ α t ( λ ) 1 d λ + 2 σ t ∫ λ t 0 λ t e − ( λ t − λ ) d ω t ( λ ) . (153)
Define
y t ⋆ : = e λ t σ t x t ⋆ . (154) y_t^\star:=\frac{e^{\lambda_t}}{\sigma_t}x_t^\star. \tag*{(154)} y t ⋆ := σ t e λ t x t ⋆ . (154)
which therefore conforms to
y t ⋆ = e λ t σ t 0 e − ( λ t − λ t 0 ) x t 0 ⋆ + 2 e 2 λ t ∫ λ t 0 λ t γ 2 e − 2 ( λ t − λ ) x t ( λ ) ⋆ γ 2 + e − 2 λ 1 α t ( λ ) d λ + 2 e λ t ∫ λ t 0 λ t e − ( λ t − λ ) d ω ‾ t ( λ ) = y t 0 ⋆ + 2 e 2 λ t ∫ λ t 0 λ t γ 2 e − 2 λ γ 2 + e − 2 λ y t ( λ ) ⋆ d λ + 2 e λ t ∫ λ t 0 λ t e − ( λ t − λ ) d ω ‾ t ( λ ) . (155) \begin{aligned}
y_t^\star&=\frac{e^{\lambda_t}}{\sigma_{t_0}}e^{-(\lambda_t-\lambda_{t_0})}x_{t_0}^\star+2e^{2\lambda_t}\int_{\lambda_{t_0}}^{\lambda_t}\frac{\gamma^2e^{-2(\lambda_t-\lambda)}x_{t(\lambda)}^\star}{\gamma^2+e^{-2\lambda}}\frac{1}{\alpha_{t(\lambda)}}\,\mathrm{d}\lambda+\sqrt{2}e^{\lambda_t}\int_{\lambda_{t_0}}^{\lambda_t}e^{-(\lambda_t-\lambda)}\,\mathrm{d}\overline{\omega}_t(\lambda)\\
&=y_{t_0}^\star+2e^{2\lambda_t}\int_{\lambda_{t_0}}^{\lambda_t}\frac{\gamma^2e^{-2\lambda}}{\gamma^2+e^{-2\lambda}}y_{t(\lambda)}^\star\,\mathrm{d}\lambda+\sqrt{2}e^{\lambda_t}\int_{\lambda_{t_0}}^{\lambda_t}e^{-(\lambda_t-\lambda)}\,\mathrm{d}\overline{\omega}_t(\lambda). \tag*{(155)}
\end{aligned} y t ⋆ = σ t 0 e λ t e − ( λ t − λ t 0 ) x t 0 ⋆ + 2 e 2 λ t ∫ λ t 0 λ t γ 2 + e − 2 λ γ 2 e − 2 ( λ t − λ ) x t ( λ ) ⋆ α t ( λ ) 1 d λ + 2 e λ t ∫ λ t 0 λ t e − ( λ t − λ ) d ω t ( λ ) = y t 0 ⋆ + 2 e 2 λ t ∫ λ t 0 λ t γ 2 + e − 2 λ γ 2 e − 2 λ y t ( λ ) ⋆ d λ + 2 e λ t ∫ λ t 0 λ t e − ( λ t − λ ) d ω t ( λ ) . (155)
We conclude that the solution to (155) is given by
y t ⋆ = 1 + γ 2 e 2 λ t 1 + γ 2 e 2 λ t 0 y t 0 ⋆ + 2 ( 1 + γ 2 e 2 λ t ) ∫ λ t 0 λ t e λ 1 + γ 2 e 2 λ d ω ‾ t ( λ ) . (156) y_t^\star=\frac{1+\gamma^2e^{2\lambda_t}}{1+\gamma^2e^{2\lambda_{t_0}}}y_{t_0}^\star+\sqrt{2}(1+\gamma^2e^{2\lambda_t})\int_{\lambda_{t_0}}^{\lambda_t}\frac{e^\lambda}{1+\gamma^2e^{2\lambda}}\,\mathrm{d}\overline{\omega}_t(\lambda). \tag*{(156)} y t ⋆ = 1 + γ 2 e 2 λ t 0 1 + γ 2 e 2 λ t y t 0 ⋆ + 2 ( 1 + γ 2 e 2 λ t ) ∫ λ t 0 λ t 1 + γ 2 e 2 λ e λ d ω t ( λ ) . (156)
To check this, we insert it into the right-hand-side of (155) and obtain
y t 0 ∗ + 2 e 2 λ t ∫ λ t 0 λ t γ 2 e − 2 λ t γ 2 + e − 2 λ y t ∗ ( λ ) d λ + 2 e λ t ∫ λ t 0 λ t e − ( λ t − λ ) d ω ˉ t ( λ ) = y t 0 ∗ + 2 e 2 λ t ∫ λ t 0 λ t γ 2 e − 2 λ t γ 2 + e − 2 λ 1 + γ 2 e 2 λ 1 + γ 2 e 2 λ t 0 d λ y t 0 ∗ + 2 2 e 2 λ t ∫ λ t 0 λ t γ 2 e − 2 λ t γ 2 + e − 2 λ ( 1 + γ 2 e 2 λ ) ∫ λ t 0 λ e λ ′ 1 + γ 2 e 2 λ ′ d ω ˉ t ( λ ′ ) d λ + 2 e λ t ∫ λ t 0 λ t e − ( λ t − λ ) d ω ˉ t ( λ ) \begin{aligned}
y_{t_0}^*+2e^{2\lambda t}\int_{\lambda_{t_0}}^{\lambda_t}\frac{\gamma^2e^{-2\lambda_t}}{\gamma^2+e^{-2\lambda}}y_t^*(\lambda)\,\mathrm{d}\lambda+\sqrt{2}e^{\lambda t}\int_{\lambda_{t_0}}^{\lambda_t}e^{-(\lambda_t-\lambda)}\,\mathrm{d}\bar{\omega}_t(\lambda)\\
={}&y_{t_0}^*+2e^{2\lambda t}\int_{\lambda_{t_0}}^{\lambda_t}\frac{\gamma^2e^{-2\lambda_t}}{\gamma^2+e^{-2\lambda}}\frac{1+\gamma^2e^{2\lambda}}{1+\gamma^2e^{2\lambda_{t_0}}}\,\mathrm{d}\lambda\,y_{t_0}^*+2\sqrt{2}e^{2\lambda t}\int_{\lambda_{t_0}}^{\lambda_t}\frac{\gamma^2e^{-2\lambda_t}}{\gamma^2+e^{-2\lambda}}(1+\gamma^2e^{2\lambda})\int_{\lambda_{t_0}}^\lambda\frac{e^{\lambda'}}{1+\gamma^2e^{2\lambda'}}\,\mathrm{d}\bar{\omega}_t(\lambda')\,\mathrm{d}\lambda\\
&+\sqrt{2}e^{\lambda t}\int_{\lambda_{t_0}}^{\lambda_t}e^{-(\lambda_t-\lambda)}\,\mathrm{d}\bar{\omega}_t(\lambda)
\end{aligned} y t 0 ∗ + 2 e 2 λ t ∫ λ t 0 λ t γ 2 + e − 2 λ γ 2 e − 2 λ t y t ∗ ( λ ) d λ + 2 e λ t ∫ λ t 0 λ t e − ( λ t − λ ) d ω ˉ t ( λ ) = y t 0 ∗ + 2 e 2 λ t ∫ λ t 0 λ t γ 2 + e − 2 λ γ 2 e − 2 λ t 1 + γ 2 e 2 λ t 0 1 + γ 2 e 2 λ d λ y t 0 ∗ + 2 2 e 2 λ t ∫ λ t 0 λ t γ 2 + e − 2 λ γ 2 e − 2 λ t ( 1 + γ 2 e 2 λ ) ∫ λ t 0 λ 1 + γ 2 e 2 λ ′ e λ ′ d ω ˉ t ( λ ′ ) d λ + 2 e λ t ∫ λ t 0 λ t e − ( λ t − λ ) d ω ˉ t ( λ )
(a) 1 + γ 2 e 2 λ t 1 + γ 2 e 2 λ t 0 y t 0 ∗ + 2 2 e 2 λ t ∫ λ t 0 λ t γ 2 e − 2 λ t γ 2 + e − 2 λ ( 1 + γ 2 e 2 λ ) ∫ λ t 0 λ e λ ′ 1 + γ 2 e 2 λ ′ d ω ˉ t ( λ ′ ) d λ + 2 e λ t ∫ λ t 0 λ t e − ( λ t − λ ) d ω ˉ t ( λ ) , (157) \text{(a)}\quad\frac{1+\gamma^2e^{2\lambda_t}}{1+\gamma^2e^{2\lambda_{t_0}}}y_{t_0}^*+2\sqrt{2}e^{2\lambda t}\int_{\lambda_{t_0}}^{\lambda_t}\frac{\gamma^2e^{-2\lambda_t}}{\gamma^2+e^{-2\lambda}}(1+\gamma^2e^{2\lambda})\int_{\lambda_{t_0}}^\lambda\frac{e^{\lambda'}}{1+\gamma^2e^{2\lambda'}}\,\mathrm{d}\bar{\omega}_t(\lambda')\,\mathrm{d}\lambda+\sqrt{2}e^{\lambda t}\int_{\lambda_{t_0}}^{\lambda_t}e^{-(\lambda_t-\lambda)}\,\mathrm{d}\bar{\omega}_t(\lambda), \tag*{(157)} (a) 1 + γ 2 e 2 λ t 0 1 + γ 2 e 2 λ t y t 0 ∗ + 2 2 e 2 λ t ∫ λ t 0 λ t γ 2 + e − 2 λ γ 2 e − 2 λ t ( 1 + γ 2 e 2 λ ) ∫ λ t 0 λ 1 + γ 2 e 2 λ ′ e λ ′ d ω ˉ t ( λ ′ ) d λ + 2 e λ t ∫ λ t 0 λ t e − ( λ t − λ ) d ω ˉ t ( λ ) , (157)
where (a) holds since
1 + 2 e 2 λ t ∫ λ t 0 λ t γ 2 e − 2 λ t γ 2 + e − 2 λ 1 + γ 2 e 2 λ 1 + γ 2 e 2 λ t 0 d λ = 1 + 2 e 2 λ t γ 2 e − 2 λ t 1 + γ 2 e 2 λ t 0 ∫ λ t 0 λ t e 2 λ d λ = 1 + γ 2 1 + γ 2 e 2 λ t 0 ( e 2 λ t − e 2 λ t 0 ) = 1 + γ 2 e 2 λ t 1 + γ 2 e 2 λ t 0 . \begin{aligned}
1+2e^{2\lambda t}\int_{\lambda_{t_0}}^{\lambda_t}\frac{\gamma^2e^{-2\lambda_t}}{\gamma^2+e^{-2\lambda}}\frac{1+\gamma^2e^{2\lambda}}{1+\gamma^2e^{2\lambda_{t_0}}}\,\mathrm{d}\lambda
&=1+2e^{2\lambda t}\frac{\gamma^2e^{-2\lambda_t}}{1+\gamma^2e^{2\lambda_{t_0}}}\int_{\lambda_{t_0}}^{\lambda_t}e^{2\lambda}\,\mathrm{d}\lambda\\
&=1+\frac{\gamma^2}{1+\gamma^2e^{2\lambda_{t_0}}}\left(e^{2\lambda_t}-e^{2\lambda_{t_0}}\right)\\
&=\frac{1+\gamma^2e^{2\lambda_t}}{1+\gamma^2e^{2\lambda_{t_0}}}.
\end{aligned} 1 + 2 e 2 λ t ∫ λ t 0 λ t γ 2 + e − 2 λ γ 2 e − 2 λ t 1 + γ 2 e 2 λ t 0 1 + γ 2 e 2 λ d λ = 1 + 2 e 2 λ t 1 + γ 2 e 2 λ t 0 γ 2 e − 2 λ t ∫ λ t 0 λ t e 2 λ d λ = 1 + 1 + γ 2 e 2 λ t 0 γ 2 ( e 2 λ t − e 2 λ t 0 ) = 1 + γ 2 e 2 λ t 0 1 + γ 2 e 2 λ t .
For the noise term, we have
2 e 2 λ t ∫ λ t 0 λ t γ 2 e − 2 λ t γ 2 + e − 2 λ ( 1 + γ 2 e 2 λ ) ∫ λ t 0 λ e λ ′ 1 + γ 2 e 2 λ ′ d ω ˉ t ( λ ′ ) d λ = 2 ∫ λ t 0 λ t γ 2 e 2 λ ∫ λ t 0 λ e λ ′ 1 + γ 2 e 2 λ ′ d ω ˉ t ( λ ′ ) d λ = 2 ∫ λ t 0 λ t γ 2 e λ ′ 1 + γ 2 e 2 λ ′ ∫ λ ′ λ t e 2 λ d λ d ω ˉ t ( λ ′ ) = ∫ λ t 0 λ t γ 2 e λ ′ ( e 2 λ t − e 2 λ ′ ) 1 + γ 2 e 2 λ ′ d ω ˉ t ( λ ′ ) . \begin{aligned}
2e^{2\lambda t}\int_{\lambda_{t_0}}^{\lambda_t}\frac{\gamma^2e^{-2\lambda_t}}{\gamma^2+e^{-2\lambda}}(1+\gamma^2e^{2\lambda})\int_{\lambda_{t_0}}^\lambda\frac{e^{\lambda'}}{1+\gamma^2e^{2\lambda'}}\,\mathrm{d}\bar{\omega}_t(\lambda')\,\mathrm{d}\lambda\\
&=2\int_{\lambda_{t_0}}^{\lambda_t}\gamma^2e^{2\lambda}\int_{\lambda_{t_0}}^\lambda\frac{e^{\lambda'}}{1+\gamma^2e^{2\lambda'}}\,\mathrm{d}\bar{\omega}_t(\lambda')\,\mathrm{d}\lambda\\
&=2\int_{\lambda_{t_0}}^{\lambda_t}\frac{\gamma^2e^{\lambda'}}{1+\gamma^2e^{2\lambda'}}\int_{\lambda'}^{\lambda_t}e^{2\lambda}\,\mathrm{d}\lambda\,\mathrm{d}\bar{\omega}_t(\lambda')\\
&=\int_{\lambda_{t_0}}^{\lambda_t}\frac{\gamma^2e^{\lambda'}(e^{2\lambda_t}-e^{2\lambda'})}{1+\gamma^2e^{2\lambda'}}\,\mathrm{d}\bar{\omega}_t(\lambda').
\end{aligned} 2 e 2 λ t ∫ λ t 0 λ t γ 2 + e − 2 λ γ 2 e − 2 λ t ( 1 + γ 2 e 2 λ ) ∫ λ t 0 λ 1 + γ 2 e 2 λ ′ e λ ′ d ω ˉ t ( λ ′ ) d λ = 2 ∫ λ t 0 λ t γ 2 e 2 λ ∫ λ t 0 λ 1 + γ 2 e 2 λ ′ e λ ′ d ω ˉ t ( λ ′ ) d λ = 2 ∫ λ t 0 λ t 1 + γ 2 e 2 λ ′ γ 2 e λ ′ ∫ λ ′ λ t e 2 λ d λ d ω ˉ t ( λ ′ ) = ∫ λ t 0 λ t 1 + γ 2 e 2 λ ′ γ 2 e λ ′ ( e 2 λ t − e 2 λ ′ ) d ω ˉ t ( λ ′ ) .
and therefore
2 2 e 2 λ t ∫ λ t 0 λ t γ 2 e − 2 λ t γ 2 + e − 2 λ ( 1 + γ 2 e 2 λ ) ∫ λ t 0 λ e λ ′ 1 + γ 2 e 2 λ ′ d ω ˉ t ( λ ′ ) d λ + 2 e λ t ∫ λ t 0 λ t e − ( λ t − λ ) d ω ˉ t ( λ ) = 2 ∫ λ t 0 λ t ( γ 2 e λ ( e 2 λ t − e 2 λ ) 1 + γ 2 e 2 λ + e λ ) d ω ˉ t ( λ ) = 2 ∫ λ t 0 λ t ( e λ γ 2 e 2 λ t + 1 1 + γ 2 e 2 λ ) d ω ˉ t ( λ ) = 2 ( 1 + γ 2 e 2 λ t ) ∫ λ t 0 λ t e λ 1 + γ 2 e 2 λ d ω ˉ t ( λ ) . \begin{aligned}
&2\sqrt{2}e^{2\lambda t}\int_{\lambda_{t_0}}^{\lambda_t}\frac{\gamma^2e^{-2\lambda_t}}{\gamma^2+e^{-2\lambda}}(1+\gamma^2e^{2\lambda})\int_{\lambda_{t_0}}^\lambda\frac{e^{\lambda'}}{1+\gamma^2e^{2\lambda'}}\,\mathrm{d}\bar{\omega}_t(\lambda')\,\mathrm{d}\lambda+\sqrt{2}e^{\lambda t}\int_{\lambda_{t_0}}^{\lambda_t}e^{-(\lambda_t-\lambda)}\,\mathrm{d}\bar{\omega}_t(\lambda)\\
&=\sqrt{2}\int_{\lambda_{t_0}}^{\lambda_t}\left(\frac{\gamma^2e^\lambda(e^{2\lambda_t}-e^{2\lambda})}{1+\gamma^2e^{2\lambda}}+e^\lambda\right)\,\mathrm{d}\bar{\omega}_t(\lambda)
=\sqrt{2}\int_{\lambda_{t_0}}^{\lambda_t}\left(\frac{e^\lambda\gamma^2e^{2\lambda_t}+1}{1+\gamma^2e^{2\lambda}}\right)\,\mathrm{d}\bar{\omega}_t(\lambda)\\
&=\sqrt{2}(1+\gamma^2e^{2\lambda_t})\int_{\lambda_{t_0}}^{\lambda_t}\frac{e^\lambda}{1+\gamma^2e^{2\lambda}}\,\mathrm{d}\bar{\omega}_t(\lambda).
\end{aligned} 2 2 e 2 λ t ∫ λ t 0 λ t γ 2 + e − 2 λ γ 2 e − 2 λ t ( 1 + γ 2 e 2 λ ) ∫ λ t 0 λ 1 + γ 2 e 2 λ ′ e λ ′ d ω ˉ t ( λ ′ ) d λ + 2 e λ t ∫ λ t 0 λ t e − ( λ t − λ ) d ω ˉ t ( λ ) = 2 ∫ λ t 0 λ t ( 1 + γ 2 e 2 λ γ 2 e λ ( e 2 λ t − e 2 λ ) + e λ ) d ω ˉ t ( λ ) = 2 ∫ λ t 0 λ t ( 1 + γ 2 e 2 λ e λ γ 2 e 2 λ t + 1 ) d ω ˉ t ( λ ) = 2 ( 1 + γ 2 e 2 λ t ) ∫ λ t 0 λ t 1 + γ 2 e 2 λ e λ d ω ˉ t ( λ ) .
Inserting into (157) yields
y t 0 ∗ + 2 e 2 λ t ∫ λ t 0 λ t γ 2 e − 2 λ t γ 2 + e − 2 λ y t ∗ ( λ ) d λ + 2 e λ t ∫ λ t 0 λ t e − ( λ t − λ ) d ω ˉ t ( λ ) = 1 + γ 2 e 2 λ t 1 + γ 2 e 2 λ t 0 y t 0 ∗ + 2 ( 1 + γ 2 e 2 λ t ) ∫ λ t 0 λ t e λ 1 + γ 2 e 2 λ d ω ˉ t ( λ ) = y t ∗ . \begin{aligned}
y_{t_0}^*+2e^{2\lambda t}\int_{\lambda_{t_0}}^{\lambda_t}\frac{\gamma^2e^{-2\lambda_t}}{\gamma^2+e^{-2\lambda}}y_t^*(\lambda)\,\mathrm{d}\lambda+\sqrt{2}e^{\lambda t}\int_{\lambda_{t_0}}^{\lambda_t}e^{-(\lambda_t-\lambda)}\,\mathrm{d}\bar{\omega}_t(\lambda)\\
=\frac{1+\gamma^2e^{2\lambda_t}}{1+\gamma^2e^{2\lambda_{t_0}}}y_{t_0}^*+\sqrt{2}(1+\gamma^2e^{2\lambda_t})\int_{\lambda_{t_0}}^{\lambda_t}\frac{e^\lambda}{1+\gamma^2e^{2\lambda}}\,\mathrm{d}\bar{\omega}_t(\lambda)=y_t^*.
\end{aligned} y t 0 ∗ + 2 e 2 λ t ∫ λ t 0 λ t γ 2 + e − 2 λ γ 2 e − 2 λ t y t ∗ ( λ ) d λ + 2 e λ t ∫ λ t 0 λ t e − ( λ t − λ ) d ω ˉ t ( λ ) = 1 + γ 2 e 2 λ t 0 1 + γ 2 e 2 λ t y t 0 ∗ + 2 ( 1 + γ 2 e 2 λ t ) ∫ λ t 0 λ t 1 + γ 2 e 2 λ e λ d ω ˉ t ( λ ) = y t ∗ .
Hence we have verified that (156) is the solution to (155) . Recalling the definition of y t ⋆ y_t^\star y t ⋆ in (154) , the solution to (153) is given by
x t ⋆ = σ t e − λ t y t ⋆ = σ t e λ t − 2 λ t 0 γ 2 + e − 2 λ t γ 2 + e − 2 λ t 0 x t 0 ⋆ + 2 σ t e λ t ( γ 2 + e − 2 λ t ) ∫ λ t 0 λ t e λ 1 + γ 2 e 2 λ d ω ‾ λ = σ t σ t 0 e λ t − λ t 0 γ 2 + e − 2 λ t γ 2 + e − 2 λ t 0 x t 0 ⋆ + 2 α t ( γ 2 + e − 2 λ t ) ∫ λ t 0 λ t e λ 1 + γ 2 e 2 λ d ω ‾ λ = α t α t 0 γ 2 + e − 2 λ t γ 2 + e − 2 λ t 0 x t 0 ⋆ + 2 α t ( γ 2 + e − 2 λ t ) ∫ λ t 0 λ t e λ 1 + γ 2 e 2 λ d ω ‾ λ . (158) x_t^\star= \sigma_t e^{-\lambda_t}y_t^\star= \sigma_t e^{\lambda_t-2\lambda_{t_0}}\frac{\gamma^2+e^{-2\lambda_t}}{\gamma^2+e^{-2\lambda_{t_0}}}x_{t_0}^\star+\sqrt{2}\sigma_t e^{\lambda_t}(\gamma^2+e^{-2\lambda_t})\int_{\lambda_{t_0}}^{\lambda_t}\frac{e^\lambda}{1+\gamma^2e^{2\lambda}}\mathrm{d}\overline{\omega}_\lambda
\\
=\frac{\sigma_t}{\sigma_{t_0}}e^{\lambda_t-\lambda_{t_0}}\frac{\gamma^2+e^{-2\lambda_t}}{\gamma^2+e^{-2\lambda_{t_0}}}x_{t_0}^\star+\sqrt{2}\alpha_t(\gamma^2+e^{-2\lambda_t})\int_{\lambda_{t_0}}^{\lambda_t}\frac{e^\lambda}{1+\gamma^2e^{2\lambda}}\mathrm{d}\overline{\omega}_\lambda
\\
=\frac{\alpha_t}{\alpha_{t_0}}\frac{\gamma^2+e^{-2\lambda_t}}{\gamma^2+e^{-2\lambda_{t_0}}}x_{t_0}^\star+\sqrt{2}\alpha_t(\gamma^2+e^{-2\lambda_t})\int_{\lambda_{t_0}}^{\lambda_t}\frac{e^\lambda}{1+\gamma^2e^{2\lambda}}\mathrm{d}\overline{\omega}_\lambda. \tag*{(158)} x t ⋆ = σ t e − λ t y t ⋆ = σ t e λ t − 2 λ t 0 γ 2 + e − 2 λ t 0 γ 2 + e − 2 λ t x t 0 ⋆ + 2 σ t e λ t ( γ 2 + e − 2 λ t ) ∫ λ t 0 λ t 1 + γ 2 e 2 λ e λ d ω λ = σ t 0 σ t e λ t − λ t 0 γ 2 + e − 2 λ t 0 γ 2 + e − 2 λ t x t 0 ⋆ + 2 α t ( γ 2 + e − 2 λ t ) ∫ λ t 0 λ t 1 + γ 2 e 2 λ e λ d ω λ = α t 0 α t γ 2 + e − 2 λ t 0 γ 2 + e − 2 λ t x t 0 ⋆ + 2 α t ( γ 2 + e − 2 λ t ) ∫ λ t 0 λ t 1 + γ 2 e 2 λ e λ d ω λ . (158)
It also holds that for any λ t i \lambda_{t_i} λ t i ,
x t i ⋆ = α t i α t i − 1 γ 2 + e − 2 λ t i γ 2 + e − 2 λ t i − 1 x t i − 1 ⋆ + 2 α t i ( γ 2 + e − 2 λ t i ) ∫ λ t i − 1 λ t i e λ 1 + γ 2 e 2 λ d ω ‾ t ( λ ) . (159) x_{t_i}^\star=\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}\frac{\gamma^2+e^{-2\lambda_{t_i}}}{\gamma^2+e^{-2\lambda_{t_{i-1}}}}x_{t_{i-1}}^\star+\sqrt{2}\alpha_{t_i}(\gamma^2+e^{-2\lambda_{t_i}})\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\frac{e^\lambda}{1+\gamma^2e^{2\lambda}}\mathrm{d}\overline{\omega}_t(\lambda). \tag*{(159)} x t i ⋆ = α t i − 1 α t i γ 2 + e − 2 λ t i − 1 γ 2 + e − 2 λ t i x t i − 1 ⋆ + 2 α t i ( γ 2 + e − 2 λ t i ) ∫ λ t i − 1 λ t i 1 + γ 2 e 2 λ e λ d ω t ( λ ) . (159)
Below we shall analyze the discretization error of (18) and (19) separately.
Convergence analysis of SDE-DPMSolver++1
Substituting the expression of data predictor in (152) into the update rule (18) yields
x t i = e − δ i σ t i − 1 σ t i − 1 x t i − 1 + α t i α t i − 1 γ 2 ( 1 − e − 2 δ i ) γ 2 + e − 2 λ t i − 1 x t i − 1 + σ t i 1 − e − 2 δ i z i = e − 2 δ i α t i α t i − 1 x t i − 1 + α t i α t i − 1 γ 2 ( 1 − e − 2 δ i ) γ 2 + e − 2 λ t i − 1 x t i − 1 + σ t i 1 − e − 2 δ i z i = α t i α t i − 1 γ 2 + e − 2 λ t i γ 2 + e − 2 λ t i − 1 x t i − 1 + 2 σ t i ∫ λ t i − 1 λ t i e − ( λ t i − λ ) d ω ‾ t ( λ ) . (160) x_{t_i}=e^{-\delta_i}\frac{\sigma_{t_{i-1}}}{\sigma_{t_i-1}}x_{t_{i-1}}+\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}\frac{\gamma^2(1-e^{-2\delta_i})}{\gamma^2+e^{-2\lambda_{t_{i-1}}}}x_{t_{i-1}}+\sigma_{t_i}\sqrt{1-e^{-2\delta_i}}z_i
\\
=e^{-2\delta_i}\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}x_{t_{i-1}}+\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}\frac{\gamma^2(1-e^{-2\delta_i})}{\gamma^2+e^{-2\lambda_{t_{i-1}}}}x_{t_{i-1}}+\sigma_{t_i}\sqrt{1-e^{-2\delta_i}}z_i
\\
=\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}\frac{\gamma^2+e^{-2\lambda_{t_i}}}{\gamma^2+e^{-2\lambda_{t_{i-1}}}}x_{t_{i-1}}+\sqrt{2}\sigma_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-(\lambda_{t_i}-\lambda)}\mathrm{d}\overline{\omega}_t(\lambda). \tag*{(160)} x t i = e − δ i σ t i − 1 σ t i − 1 x t i − 1 + α t i − 1 α t i γ 2 + e − 2 λ t i − 1 γ 2 ( 1 − e − 2 δ i ) x t i − 1 + σ t i 1 − e − 2 δ i z i = e − 2 δ i α t i − 1 α t i x t i − 1 + α t i − 1 α t i γ 2 + e − 2 λ t i − 1 γ 2 ( 1 − e − 2 δ i ) x t i − 1 + σ t i 1 − e − 2 δ i z i = α t i − 1 α t i γ 2 + e − 2 λ t i − 1 γ 2 + e − 2 λ t i x t i − 1 + 2 σ t i ∫ λ t i − 1 λ t i e − ( λ t i − λ ) d ω t ( λ ) . (160)
Subtracting the x t i ⋆ x_{t_i}^\star x t i ⋆ in (159) from (160) yields
x t i − x t i ⋆ = α t i α t i − 1 γ 2 + e − 2 λ t i γ 2 + e − 2 λ t i − 1 ( x t i − 1 − x t i − 1 ⋆ ) + 2 σ t i ∫ λ t i − 1 λ t i e − ( λ t i − λ ) d ω ‾ t ( λ ) − 2 α t i ( γ 2 + e − 2 λ t i ) ∫ λ t i − 1 λ t i e λ 1 + γ 2 e 2 λ d ω ‾ t ( λ ) = α t i α t i − 1 γ 2 + e − 2 λ t i γ 2 + e − 2 λ t i − 1 ( x t i − 1 − x t i − 1 ⋆ ) + 2 α t i ∫ λ t i − 1 λ t i e λ ( e − 2 λ t i − γ 2 + e − 2 λ t i 1 + γ 2 e 2 λ ) d ω ‾ t ( λ ) = α t i α t i − 1 γ 2 + e − 2 λ t i γ 2 + e − 2 λ t i − 1 ( x t i − 1 − x t i − 1 ⋆ ) − 2 α t i γ 2 ∫ λ t i − 1 λ t i e λ 1 − e − 2 ( λ t i − λ ) 1 + γ 2 e 2 λ d ω ‾ t ( λ ) . (161) x_{t_i}-x_{t_i}^\star=\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}\frac{\gamma^2+e^{-2\lambda_{t_i}}}{\gamma^2+e^{-2\lambda_{t_{i-1}}}}(x_{t_{i-1}}-x_{t_{i-1}}^\star)+\sqrt{2}\sigma_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-(\lambda_{t_i}-\lambda)}\mathrm{d}\overline{\omega}_t(\lambda)-\sqrt{2}\alpha_{t_i}(\gamma^2+e^{-2\lambda_{t_i}})\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\frac{e^\lambda}{1+\gamma^2e^{2\lambda}}\mathrm{d}\overline{\omega}_t(\lambda)
\\
=\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}\frac{\gamma^2+e^{-2\lambda_{t_i}}}{\gamma^2+e^{-2\lambda_{t_{i-1}}}}(x_{t_{i-1}}-x_{t_{i-1}}^\star)+\sqrt{2}\alpha_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^\lambda\left(e^{-2\lambda_{t_i}}-\frac{\gamma^2+e^{-2\lambda_{t_i}}}{1+\gamma^2e^{2\lambda}}\right)\mathrm{d}\overline{\omega}_t(\lambda)
\\
=\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}\frac{\gamma^2+e^{-2\lambda_{t_i}}}{\gamma^2+e^{-2\lambda_{t_{i-1}}}}(x_{t_{i-1}}-x_{t_{i-1}}^\star)-\sqrt{2}\alpha_{t_i}\gamma^2\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^\lambda\frac{1-e^{-2(\lambda_{t_i}-\lambda)}}{1+\gamma^2e^{2\lambda}}\mathrm{d}\overline{\omega}_t(\lambda). \tag*{(161)} x t i − x t i ⋆ = α t i − 1 α t i γ 2 + e − 2 λ t i − 1 γ 2 + e − 2 λ t i ( x t i − 1 − x t i − 1 ⋆ ) + 2 σ t i ∫ λ t i − 1 λ t i e − ( λ t i − λ ) d ω t ( λ ) − 2 α t i ( γ 2 + e − 2 λ t i ) ∫ λ t i − 1 λ t i 1 + γ 2 e 2 λ e λ d ω t ( λ ) = α t i − 1 α t i γ 2 + e − 2 λ t i − 1 γ 2 + e − 2 λ t i ( x t i − 1 − x t i − 1 ⋆ ) + 2 α t i ∫ λ t i − 1 λ t i e λ ( e − 2 λ t i − 1 + γ 2 e 2 λ γ 2 + e − 2 λ t i ) d ω t ( λ ) = α t i − 1 α t i γ 2 + e − 2 λ t i − 1 γ 2 + e − 2 λ t i ( x t i − 1 − x t i − 1 ⋆ ) − 2 α t i γ 2 ∫ λ t i − 1 λ t i e λ 1 + γ 2 e 2 λ 1 − e − 2 ( λ t i − λ ) d ω t ( λ ) . (161)
By recursion, we obtain
x t M − x t M ⋆ = − 2 γ 2 ∑ m = 1 M ∏ i = m + 1 M α t i α t i − 1 γ 2 + e − 2 λ t i γ 2 + e − 2 λ t i − 1 α t m ∫ λ t m − 1 λ t m e λ 1 − e − 2 ( λ t m − λ ) 1 + γ 2 e 2 λ d ω ‾ t ( λ ) = − 2 γ 2 α t M ( γ 2 + e − 2 λ t M ) ∑ m = 1 M 1 γ 2 + e − 2 λ t m ∫ λ t m − 1 λ t m e λ 1 − e − 2 ( λ t m − λ ) 1 + γ 2 e 2 λ d ω ‾ t ( λ ) . x_{t_M}-x_{t_M}^\star=-\sqrt{2}\gamma^2\sum_{m=1}^{M}\prod_{i=m+1}^{M}\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}\frac{\gamma^2+e^{-2\lambda_{t_i}}}{\gamma^2+e^{-2\lambda_{t_{i-1}}}}\alpha_{t_m}\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}e^\lambda\frac{1-e^{-2(\lambda_{t_m}-\lambda)}}{1+\gamma^2e^{2\lambda}}\mathrm{d}\overline{\omega}_t(\lambda)
\\
=-\sqrt{2}\gamma^2\alpha_{t_M}(\gamma^2+e^{-2\lambda_{t_M}})\sum_{m=1}^{M}\frac{1}{\gamma^2+e^{-2\lambda_{t_m}}}\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}e^\lambda\frac{1-e^{-2(\lambda_{t_m}-\lambda)}}{1+\gamma^2e^{2\lambda}}\mathrm{d}\overline{\omega}_t(\lambda). x t M − x t M ⋆ = − 2 γ 2 m = 1 ∑ M i = m + 1 ∏ M α t i − 1 α t i γ 2 + e − 2 λ t i − 1 γ 2 + e − 2 λ t i α t m ∫ λ t m − 1 λ t m e λ 1 + γ 2 e 2 λ 1 − e − 2 ( λ t m − λ ) d ω t ( λ ) = − 2 γ 2 α t M ( γ 2 + e − 2 λ t M ) m = 1 ∑ M γ 2 + e − 2 λ t m 1 ∫ λ t m − 1 λ t m e λ 1 + γ 2 e 2 λ 1 − e − 2 ( λ t m − λ ) d ω t ( λ ) .
Note that ω ‾ t ( λ ) \overline{\omega}_t(\lambda) ω t ( λ ) is Brownian motion, x t M − x t M ⋆ x_{t_M}-x_{t_M}^\star x t M − x t M ⋆ is a centered Gaussian random vector with second moment
E [ ∥ x t M − x t M ⋆ ∥ 2 2 ] = 2 d γ 4 α t M 2 ( γ 2 + e − 2 λ t M ) 2 ∑ m = 1 M 1 ( γ 2 + e − 2 λ t m ) 2 ∫ λ t m − 1 λ t m e 2 λ ( 1 − e − 2 ( λ t m − λ ) ) 2 ( 1 + γ 2 e 2 λ ) 2 d λ . \mathbb{E}[\lVert x_{t_M}-x_{t_M}^\star\rVert_2^2]=2d\gamma^4\alpha_{t_M}^2(\gamma^2+e^{-2\lambda_{t_M}})^2\sum_{m=1}^{M}\frac{1}{(\gamma^2+e^{-2\lambda_{t_m}})^2}\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}e^{2\lambda}\frac{(1-e^{-2(\lambda_{t_m}-\lambda)})^2}{(1+\gamma^2e^{2\lambda})^2}\mathrm{d}\lambda. E [∥ x t M − x t M ⋆ ∥ 2 2 ] = 2 d γ 4 α t M 2 ( γ 2 + e − 2 λ t M ) 2 m = 1 ∑ M ( γ 2 + e − 2 λ t m ) 2 1 ∫ λ t m − 1 λ t m e 2 λ ( 1 + γ 2 e 2 λ ) 2 ( 1 − e − 2 ( λ t m − λ ) ) 2 d λ .
Note that ( γ 2 + e − 2 λ t M ) / ( γ 2 + e − 2 λ t m ) ≥ ( γ 2 + e − 2 λ t M ) / ( γ 2 + e − 2 λ t 0 ) = Ω ( 1 ) (\gamma^2+e^{-2\lambda t_M})/(\gamma^2+e^{-2\lambda t_m}) \ge(\gamma^2+e^{-2\lambda t_M})/(\gamma^2+e^{-2\lambda t_0}) = \Omega(1) ( γ 2 + e − 2 λ t M ) / ( γ 2 + e − 2 λ t m ) ≥ ( γ 2 + e − 2 λ t M ) / ( γ 2 + e − 2 λ t 0 ) = Ω ( 1 ) and
1 − e − 2 ( λ t m − λ ) ≥ λ t m − λ , for δ m ≤ 1 / 2. 1-e^{-2(\lambda t_m-\lambda)} \ge\lambda t_m-\lambda,\qquad\text{for }\delta_m\le1/2. 1 − e − 2 ( λ t m − λ ) ≥ λ t m − λ , for δ m ≤ 1/2.
We introduce λ t m ′ = ( λ t m − 1 + λ t m ) / 2 \lambda'_{t_m}=(\lambda_{t_{m-1}}+\lambda_{t_m})/2 λ t m ′ = ( λ t m − 1 + λ t m ) /2 and obtain
E [ ∥ x t M − x t M ∗ ∥ 2 ] ≳ ∑ m = 1 M ∫ λ t m − 1 λ t m ′ γ 2 e 2 λ ( λ t m − λ ) 2 ( 1 + γ 2 e 2 λ ) 2 d λ ≥ ∑ m = 1 M ∫ λ t m − 1 λ t m ′ γ 2 e 2 λ ( λ t m − λ ) 2 ( 1 + γ 2 e 2 λ ) 2 d λ ≥ 1 4 ∑ m = 1 M δ m 2 ∫ λ t m − 1 λ t m ′ γ 2 e 2 λ ( 1 + γ 2 e 2 λ ) 2 d λ = 1 8 ∑ m = 1 M δ m 2 ( 1 1 + γ 2 e 2 λ t m − 1 − 1 1 + γ 2 e 2 λ t m ′ ) ≥ 1 8 ∑ m = 1 M δ m 2 γ 2 ( e 2 λ t m ′ − e 2 λ t m − 1 ) ( 1 + γ 2 e 2 λ t m ′ ) 2 ≥ 1 8 ∑ m = 1 M δ m 3 γ 2 e 2 λ t m − 1 ( 1 + γ 2 e 2 λ t m ′ ) 2 ≳ ∑ m = 1 M δ m 3 ≥ ( ∑ m = 1 M δ m M 2 / 3 ) 3 = Ω ( M − 2 ) . \begin{aligned}
\mathbb{E}[\lVert x_{t_M}-x^*_{t_M}\rVert^2]
&\gtrsim\sum_{m=1}^{M}\int_{\lambda_{t_{m-1}}}^{\lambda'_{t_m}}
\frac{\gamma^2e^{2\lambda}(\lambda_{t_m}-\lambda)^2}{(1+\gamma^2e^{2\lambda})^2}\,d\lambda\\
&\ge\sum_{m=1}^{M}\int_{\lambda_{t_{m-1}}}^{\lambda'_{t_m}}
\frac{\gamma^2e^{2\lambda}(\lambda_{t_m}-\lambda)^2}{(1+\gamma^2e^{2\lambda})^2}\,d\lambda\\
&\ge\frac{1}{4}\sum_{m=1}^{M}\delta_m^2\int_{\lambda_{t_{m-1}}}^{\lambda'_{t_m}}
\frac{\gamma^2e^{2\lambda}}{(1+\gamma^2e^{2\lambda})^2}\,d\lambda\\
&= \frac{1}{8}\sum_{m=1}^{M}\delta_m^2\left(\frac{1}{1+\gamma^2e^{2\lambda_{t_{m-1}}}}-\frac{1}{1+\gamma^2e^{2\lambda'_{t_m}}}\right)\\
&\ge\frac{1}{8}\sum_{m=1}^{M}\frac{\delta_m^2\gamma^2(e^{2\lambda'_{t_m}}-e^{2\lambda_{t_{m-1}}})}{(1+\gamma^2e^{2\lambda'_{t_m}})^2}
\ge\frac{1}{8}\sum_{m=1}^{M}\frac{\delta_m^3\gamma^2e^{2\lambda_{t_{m-1}}}}{(1+\gamma^2e^{2\lambda'_{t_m}})^2}\\
&\gtrsim\sum_{m=1}^{M}\delta_m^3
\ge\left(\frac{\sum_{m=1}^{M}\delta_m}{M^{2/3}}\right)^3
=\Omega(M^{-2}).
\end{aligned} E [∥ x t M − x t M ∗ ∥ 2 ] ≳ m = 1 ∑ M ∫ λ t m − 1 λ t m ′ ( 1 + γ 2 e 2 λ ) 2 γ 2 e 2 λ ( λ t m − λ ) 2 d λ ≥ m = 1 ∑ M ∫ λ t m − 1 λ t m ′ ( 1 + γ 2 e 2 λ ) 2 γ 2 e 2 λ ( λ t m − λ ) 2 d λ ≥ 4 1 m = 1 ∑ M δ m 2 ∫ λ t m − 1 λ t m ′ ( 1 + γ 2 e 2 λ ) 2 γ 2 e 2 λ d λ = 8 1 m = 1 ∑ M δ m 2 ( 1 + γ 2 e 2 λ t m − 1 1 − 1 + γ 2 e 2 λ t m ′ 1 ) ≥ 8 1 m = 1 ∑ M ( 1 + γ 2 e 2 λ t m ′ ) 2 δ m 2 γ 2 ( e 2 λ t m ′ − e 2 λ t m − 1 ) ≥ 8 1 m = 1 ∑ M ( 1 + γ 2 e 2 λ t m ′ ) 2 δ m 3 γ 2 e 2 λ t m − 1 ≳ m = 1 ∑ M δ m 3 ≥ ( M 2/3 ∑ m = 1 M δ m ) 3 = Ω ( M − 2 ) .
Consequently, x t M − x t M ∗ x_{t_M}-x^*_{t_M} x t M − x t M ∗ is a centered Gaussian random vector with second moment Ω ( M − 2 ) \Omega(M^{-2}) Ω ( M − 2 ) . Then we have E [ ∥ x t M − x t M ∗ ∥ 2 ] = Ω ( M − 1 ) \mathbb{E}[\lVert x_{t_M}-x^*_{t_M}\rVert^2]=\Omega(M^{-1}) E [∥ x t M − x t M ∗ ∥ 2 ] = Ω ( M − 1 ) and complete the proof.
Convergence analysis of SDE-DPMSolver++2
Similar to the analysis in (161) , we have
x t i − x t i ∗ = α t i α t i − 1 γ 2 + e − 2 λ t i γ 2 + e − 2 λ t i − 1 ( x t i − 1 − x t i − 1 ∗ ) + 1 2 δ i α t i ( 1 − e − 2 δ i ) D t i − 2 α t i γ 2 ∫ λ t i − 1 λ t i e λ 1 − e − 2 ( λ t i − λ ) 1 + γ 2 e 2 λ d ω ˉ t ( λ ) . (162) x_{t_i}-x^*_{t_i}
=\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}\frac{\gamma^2+e^{-2\lambda_{t_i}}}{\gamma^2+e^{-2\lambda_{t_{i-1}}}}(x_{t_{i-1}}-x^*_{t_{i-1}})
+\frac{1}{2}\delta_i\alpha_{t_i}(1-e^{-2\delta_i})D_{t_i}
-\sqrt{2}\alpha_{t_i}\gamma^2\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^\lambda\frac{1-e^{-2(\lambda_{t_i}-\lambda)}}{1+\gamma^2e^{2\lambda}}\,d\bar{\omega}_t(\lambda). \tag*{(162)} x t i − x t i ∗ = α t i − 1 α t i γ 2 + e − 2 λ t i − 1 γ 2 + e − 2 λ t i ( x t i − 1 − x t i − 1 ∗ ) + 2 1 δ i α t i ( 1 − e − 2 δ i ) D t i − 2 α t i γ 2 ∫ λ t i − 1 λ t i e λ 1 + γ 2 e 2 λ 1 − e − 2 ( λ t i − λ ) d ω ˉ t ( λ ) . (162)
We define D t 1 ∗ = 0 D^*_{t_1}=0 D t 1 ∗ = 0 and for i ≥ 2 i\ge2 i ≥ 2 ,
D t i ∗ = 1 δ i − 1 ( μ θ ( x t i − 1 ∗ , t i − 1 ) − μ θ ( x t i − 2 ∗ , t i − 2 ) ) = 1 δ i − 1 ( γ 2 γ 2 + e − 2 λ t i − 1 x t i − 1 ∗ α t i − 1 − γ 2 γ 2 + e − 2 λ t i − 2 x t i − 2 ∗ α t i − 2 ) (163) \begin{aligned}
D^*_{t_i}
&=\frac{1}{\delta_{i-1}}\left(\mu_\theta(x^*_{t_{i-1}},t_{i-1})-\mu_\theta(x^*_{t_{i-2}},t_{i-2})\right)\\
&=\frac{1}{\delta_{i-1}}\left(\frac{\gamma^2}{\gamma^2+e^{-2\lambda_{t_{i-1}}}}\frac{x^*_{t_{i-1}}}{\alpha_{t_{i-1}}}
-\frac{\gamma^2}{\gamma^2+e^{-2\lambda_{t_{i-2}}}}\frac{x^*_{t_{i-2}}}{\alpha_{t_{i-2}}}\right) \tag*{(163)}
\end{aligned} D t i ∗ = δ i − 1 1 ( μ θ ( x t i − 1 ∗ , t i − 1 ) − μ θ ( x t i − 2 ∗ , t i − 2 ) ) = δ i − 1 1 ( γ 2 + e − 2 λ t i − 1 γ 2 α t i − 1 x t i − 1 ∗ − γ 2 + e − 2 λ t i − 2 γ 2 α t i − 2 x t i − 2 ∗ ) (163)
≡ 1 δ i − 1 [ γ 2 γ 2 + e − 2 λ t i − 1 ( γ 2 + e − 2 λ t i − 1 γ 2 + e − 2 λ t i − 2 x t i − 2 ∗ α t i − 2 + 2 ( γ 2 + e − 2 λ t i − 1 ) ∫ λ t i − 2 λ t i − 1 e λ 1 + γ 2 e 2 λ d ω ˉ t ( λ ) ) − γ 2 γ 2 + e − 2 λ t i − 2 x t i − 2 ∗ α t i − 2 ] = 2 γ 2 δ i − 1 ∫ λ t i − 2 λ t i − 1 e λ 1 + γ 2 e 2 λ d ω ˉ t ( λ ) , (164) \begin{aligned}
&\equiv\frac{1}{\delta_{i-1}}\left[
\frac{\gamma^2}{\gamma^2+e^{-2\lambda_{t_{i-1}}}}
\left(\frac{\gamma^2+e^{-2\lambda_{t_{i-1}}}}{\gamma^2+e^{-2\lambda_{t_{i-2}}}}\frac{x^*_{t_{i-2}}}{\alpha_{t_{i-2}}}
+\sqrt{2}(\gamma^2+e^{-2\lambda_{t_{i-1}}})\int_{\lambda_{t_{i-2}}}^{\lambda_{t_{i-1}}}\frac{e^\lambda}{1+\gamma^2e^{2\lambda}}\,d\bar{\omega}_t(\lambda)\right)
-\frac{\gamma^2}{\gamma^2+e^{-2\lambda_{t_{i-2}}}}\frac{x^*_{t_{i-2}}}{\alpha_{t_{i-2}}}
\right]\\
&=\frac{\sqrt{2}\gamma^2}{\delta_{i-1}}\int_{\lambda_{t_{i-2}}}^{\lambda_{t_{i-1}}}\frac{e^\lambda}{1+\gamma^2e^{2\lambda}}\,d\bar{\omega}_t(\lambda), \tag*{(164)}
\end{aligned} ≡ δ i − 1 1 [ γ 2 + e − 2 λ t i − 1 γ 2 ( γ 2 + e − 2 λ t i − 2 γ 2 + e − 2 λ t i − 1 α t i − 2 x t i − 2 ∗ + 2 ( γ 2 + e − 2 λ t i − 1 ) ∫ λ t i − 2 λ t i − 1 1 + γ 2 e 2 λ e λ d ω ˉ t ( λ ) ) − γ 2 + e − 2 λ t i − 2 γ 2 α t i − 2 x t i − 2 ∗ ] = δ i − 1 2 γ 2 ∫ λ t i − 2 λ t i − 1 1 + γ 2 e 2 λ e λ d ω ˉ t ( λ ) , (164)
where (a) applies (159) . Substituting into (162) further yields
x t i − x t i ∗ = α t i α t i − 1 γ 2 + e − 2 λ t i γ 2 + e − 2 λ t i − 1 ( x t i − 1 − x t i − 1 ∗ ) + 1 2 δ i α t i ( 1 − e − 2 δ i ) D t i ∗ − 2 α t i γ 2 ∫ λ t i − 1 λ t i e λ 1 − e − 2 ( λ t i − λ ) 1 + γ 2 e 2 λ d ω ˉ t ( λ ) + 1 2 δ i α t i ( 1 − e − 2 δ i ) ( D t i − D t i ∗ ) = α t i α t i − 1 γ 2 + e − 2 λ t i γ 2 + e − 2 λ t i − 1 ( x t i − 1 − x t i − 1 ⋆ ) + 2 2 δ i α t i ( 1 − e − 2 δ i ) γ 2 δ i − 1 ∫ λ t i − 2 λ t i − 1 e λ 1 + γ 2 e 2 λ d w ˉ t ( λ ) \begin{aligned}
x_{t_i}-x^*_{t_i}
&=\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}\frac{\gamma^2+e^{-2\lambda_{t_i}}}{\gamma^2+e^{-2\lambda_{t_{i-1}}}}(x_{t_{i-1}}-x^*_{t_{i-1}})
+\frac{1}{2}\delta_i\alpha_{t_i}(1-e^{-2\delta_i})D^*_{t_i}\\
&\quad-\sqrt{2}\alpha_{t_i}\gamma^2\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^\lambda\frac{1-e^{-2(\lambda_{t_i}-\lambda)}}{1+\gamma^2e^{2\lambda}}\,d\bar{\omega}_t(\lambda)
+\frac{1}{2}\delta_i\alpha_{t_i}(1-e^{-2\delta_i})(D_{t_i}-D^*_{t_i}) \\
&= \frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}\frac{\gamma^2+e^{-2\lambda t_i}}{\gamma^2+e^{-2\lambda t_{i-1}}}(x_{t_{i-1}}-x^\star_{t_{i-1}})+\frac{\sqrt{2}}{2}\delta_i\alpha_{t_i}(1-e^{-2\delta_i})\frac{\gamma^2}{\delta_{i-1}}\int_{\lambda_{t_{i-2}}}^{\lambda_{t_{i-1}}}\frac{e^\lambda}{1+\gamma^2e^{2\lambda}}\,\mathrm{d}\bar{w}_t(\lambda)
\end{aligned} x t i − x t i ∗ = α t i − 1 α t i γ 2 + e − 2 λ t i − 1 γ 2 + e − 2 λ t i ( x t i − 1 − x t i − 1 ∗ ) + 2 1 δ i α t i ( 1 − e − 2 δ i ) D t i ∗ − 2 α t i γ 2 ∫ λ t i − 1 λ t i e λ 1 + γ 2 e 2 λ 1 − e − 2 ( λ t i − λ ) d ω ˉ t ( λ ) + 2 1 δ i α t i ( 1 − e − 2 δ i ) ( D t i − D t i ∗ ) = α t i − 1 α t i γ 2 + e − 2 λ t i − 1 γ 2 + e − 2 λ t i ( x t i − 1 − x t i − 1 ⋆ ) + 2 2 δ i α t i ( 1 − e − 2 δ i ) δ i − 1 γ 2 ∫ λ t i − 2 λ t i − 1 1 + γ 2 e 2 λ e λ d w ˉ t ( λ )
− 2 α t i γ 2 ∫ λ t i − 1 λ t i e λ 1 − e − 2 ( λ t i − λ ) 1 + γ 2 e 2 λ d w ˉ t ( λ ) + 1 2 δ i α t i ( 1 − e − 2 δ i ) ( D t i − D t i ⋆ ) . -\sqrt{2}\alpha_{t_i}\gamma^2\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^\lambda\frac{1-e^{-2(\lambda_{t_i}-\lambda)}}{1+\gamma^2e^{2\lambda}}\,\mathrm{d}\bar{w}_t(\lambda)+\frac{1}{2}\delta_i\alpha_{t_i}(1-e^{-2\delta_i})(D_{t_i}-D^\star_{t_i}). − 2 α t i γ 2 ∫ λ t i − 1 λ t i e λ 1 + γ 2 e 2 λ 1 − e − 2 ( λ t i − λ ) d w ˉ t ( λ ) + 2 1 δ i α t i ( 1 − e − 2 δ i ) ( D t i − D t i ⋆ ) .
By recursion, we obtain
x t M − x t M ⋆ = ∑ m = 2 M α t M α t m γ 2 + e − 2 λ t M γ 2 + e − 2 λ t m 2 2 δ m α t m ( 1 − e − 2 δ m ) γ 2 δ m − 1 ∫ λ t m − 2 λ t m − 1 e λ 1 + γ 2 e 2 λ d w ˉ t ( λ ) x_{t_M}-x^\star_{t_M}=\sum_{m=2}^{M}\frac{\alpha_{t_M}}{\alpha_{t_m}}\frac{\gamma^2+e^{-2\lambda t_M}}{\gamma^2+e^{-2\lambda t_m}}\frac{\sqrt{2}}{2}\delta_m\alpha_{t_m}(1-e^{-2\delta_m})\frac{\gamma^2}{\delta_{m-1}}\int_{\lambda_{t_{m-2}}}^{\lambda_{t_{m-1}}}\frac{e^\lambda}{1+\gamma^2e^{2\lambda}}\,\mathrm{d}\bar{w}_t(\lambda) x t M − x t M ⋆ = m = 2 ∑ M α t m α t M γ 2 + e − 2 λ t m γ 2 + e − 2 λ t M 2 2 δ m α t m ( 1 − e − 2 δ m ) δ m − 1 γ 2 ∫ λ t m − 2 λ t m − 1 1 + γ 2 e 2 λ e λ d w ˉ t ( λ )
− ∑ m = 1 M α t M α t m γ 2 + e − 2 λ t M γ 2 + e − 2 λ t m 2 α t m γ 2 ∫ λ t m − 1 λ t m e λ 1 − e − 2 ( λ t m − λ ) 1 + γ 2 e 2 λ d w ˉ t ( λ ) + ∑ m = 1 M α t M α t m γ 2 + e − 2 λ t M γ 2 + e − 2 λ t m 1 2 δ m α t m ( 1 − e − 2 δ m ) ( D t m − D t m ⋆ ) = 2 α t M γ 2 ∑ m = 1 M − 1 γ 2 + e − 2 λ t M γ 2 + e − 2 λ t m ∫ λ t m − 1 λ t m e λ 1 + γ 2 e 2 λ ( γ 2 + e − 2 λ t m γ 2 + e − 2 λ t m + 1 δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m − 1 + e − 2 ( λ t m − λ ) ) d w ˉ t ( λ ) \begin{aligned}
-\sum_{m=1}^{M}\frac{\alpha_{t_M}}{\alpha_{t_m}}\frac{\gamma^2+e^{-2\lambda t_M}}{\gamma^2+e^{-2\lambda t_m}}\sqrt{2}\alpha_{t_m}\gamma^2\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}e^\lambda\frac{1-e^{-2(\lambda_{t_m}-\lambda)}}{1+\gamma^2e^{2\lambda}}\,\mathrm{d}\bar{w}_t(\lambda) \\
&+\sum_{m=1}^{M}\frac{\alpha_{t_M}}{\alpha_{t_m}}\frac{\gamma^2+e^{-2\lambda t_M}}{\gamma^2+e^{-2\lambda t_m}}\frac{1}{2}\delta_m\alpha_{t_m}(1-e^{-2\delta_m})(D_{t_m}-D^\star_{t_m}) \\
&=\sqrt{2}\alpha_{t_M}\gamma^2\sum_{m=1}^{M-1}\frac{\gamma^2+e^{-2\lambda t_M}}{\gamma^2+e^{-2\lambda t_m}}\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}\frac{e^\lambda}{1+\gamma^2e^{2\lambda}}\left(\frac{\gamma^2+e^{-2\lambda t_m}}{\gamma^2+e^{-2\lambda t_{m+1}}}\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m}-1+e^{-2(\lambda_{t_m}-\lambda)}\right)\mathrm{d}\bar{w}_t(\lambda)
\end{aligned} − m = 1 ∑ M α t m α t M γ 2 + e − 2 λ t m γ 2 + e − 2 λ t M 2 α t m γ 2 ∫ λ t m − 1 λ t m e λ 1 + γ 2 e 2 λ 1 − e − 2 ( λ t m − λ ) d w ˉ t ( λ ) + m = 1 ∑ M α t m α t M γ 2 + e − 2 λ t m γ 2 + e − 2 λ t M 2 1 δ m α t m ( 1 − e − 2 δ m ) ( D t m − D t m ⋆ ) = 2 α t M γ 2 m = 1 ∑ M − 1 γ 2 + e − 2 λ t m γ 2 + e − 2 λ t M ∫ λ t m − 1 λ t m 1 + γ 2 e 2 λ e λ ( γ 2 + e − 2 λ t m + 1 γ 2 + e − 2 λ t m 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) − 1 + e − 2 ( λ t m − λ ) ) d w ˉ t ( λ )
− ∫ λ t M − 1 λ t M e λ 1 − e − 2 ( λ t M − λ ) 1 + γ 2 e 2 λ d w ˉ t ( λ ) + ∑ m = 1 M α t M α t m γ 2 + e − 2 λ t M γ 2 + e − 2 λ t m 1 2 δ m α t m ( 1 − e − 2 δ m ) ( D t m − D t m ⋆ ) . -\int_{\lambda_{t_{M-1}}}^{\lambda_{t_M}}e^\lambda\frac{1-e^{-2(\lambda_{t_M}-\lambda)}}{1+\gamma^2e^{2\lambda}}\,\mathrm{d}\bar{w}_t(\lambda)+\sum_{m=1}^{M}\frac{\alpha_{t_M}}{\alpha_{t_m}}\frac{\gamma^2+e^{-2\lambda t_M}}{\gamma^2+e^{-2\lambda t_m}}\frac{1}{2}\delta_m\alpha_{t_m}(1-e^{-2\delta_m})(D_{t_m}-D^\star_{t_m}). − ∫ λ t M − 1 λ t M e λ 1 + γ 2 e 2 λ 1 − e − 2 ( λ t M − λ ) d w ˉ t ( λ ) + m = 1 ∑ M α t m α t M γ 2 + e − 2 λ t m γ 2 + e − 2 λ t M 2 1 δ m α t m ( 1 − e − 2 δ m ) ( D t m − D t m ⋆ ) .
Note that the sum of the first two terms is a centered Gaussian random vector. Denote its covariance matrix by V 2 I d V^2I_d V 2 I d . Then we have
E [ ∥ x t M − x t M ⋆ ∥ 2 ] ≥ V d 2 − ∑ m = 1 M α t M α t m γ 2 + e − 2 λ t M γ 2 + e − 2 λ t m 1 2 δ m α t m ( 1 − e − 2 δ m ) E [ ∥ D t m − D t m ⋆ ∥ 2 ] (165) \mathbb{E}[\lVert x_{t_M}-x^\star_{t_M}\rVert^2]\geq\frac{V\sqrt{d}}{\sqrt{2}}-\sum_{m=1}^{M}\frac{\alpha_{t_M}}{\alpha_{t_m}}\frac{\gamma^2+e^{-2\lambda t_M}}{\gamma^2+e^{-2\lambda t_m}}\frac{1}{2}\delta_m\alpha_{t_m}(1-e^{-2\delta_m})\mathbb{E}[\lVert D_{t_m}-D^\star_{t_m}\rVert^2] \tag*{(165)} E [∥ x t M − x t M ⋆ ∥ 2 ] ≥ 2 V d − m = 1 ∑ M α t m α t M γ 2 + e − 2 λ t m γ 2 + e − 2 λ t M 2 1 δ m α t m ( 1 − e − 2 δ m ) E [∥ D t m − D t m ⋆ ∥ 2 ] (165)
The remaining proof is divided into two steps. We first establish that V 2 = Ω ( M − 2 ) V^2=\Omega(M^{-2}) V 2 = Ω ( M − 2 ) , namely
V 2 = 2 α t M 2 γ 4 ∑ m = 1 M − 1 ( γ 2 + e − 2 λ t M γ 2 + e − 2 λ t m ) 2 ∫ λ t m − 1 λ t m e 2 λ ( 1 + γ 2 e 2 λ ) 2 ( γ 2 + e − 2 λ t m γ 2 + e − 2 λ t m + 1 δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m − 1 + e − 2 ( λ t m − λ ) ) 2 d λ + ∫ λ t M − 1 λ t M e 2 λ ( 1 − e − 2 ( λ t M − λ ) 1 + γ 2 e 2 λ ) 2 d λ = Ω ( δ 2 ) (166) \begin{aligned}
V^2=2\alpha_{t_M}^2\gamma^4\sum_{m=1}^{M-1}\left(\frac{\gamma^2+e^{-2\lambda t_M}}{\gamma^2+e^{-2\lambda t_m}}\right)^2\int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}\frac{e^{2\lambda}}{(1+\gamma^2e^{2\lambda})^2}\left(\frac{\gamma^2+e^{-2\lambda t_m}}{\gamma^2+e^{-2\lambda t_{m+1}}}\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m}-1+e^{-2(\lambda_{t_m}-\lambda)}\right)^2\mathrm{d}\lambda\\
&+\int_{\lambda_{t_{M-1}}}^{\lambda_{t_M}}e^{2\lambda}\left(\frac{1-e^{-2(\lambda_{t_M}-\lambda)}}{1+\gamma^2e^{2\lambda}}\right)^2\mathrm{d}\lambda=\Omega(\delta^2) \tag*{(166)}
\end{aligned} V 2 = 2 α t M 2 γ 4 m = 1 ∑ M − 1 ( γ 2 + e − 2 λ t m γ 2 + e − 2 λ t M ) 2 ∫ λ t m − 1 λ t m ( 1 + γ 2 e 2 λ ) 2 e 2 λ ( γ 2 + e − 2 λ t m + 1 γ 2 + e − 2 λ t m 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) − 1 + e − 2 ( λ t m − λ ) ) 2 d λ + ∫ λ t M − 1 λ t M e 2 λ ( 1 + γ 2 e 2 λ 1 − e − 2 ( λ t M − λ ) ) 2 d λ = Ω ( δ 2 ) (166)
and then show that
∑ m = 1 M α t M α t m γ 2 + e − 2 λ t M γ 2 + e − 2 λ t m 1 2 δ m α t m ( 1 − e − 2 δ m ) E [ ∥ D t m − D t m ⋆ ∥ 2 ] = O ( δ 1.5 ) . (167) \sum_{m=1}^{M}\frac{\alpha_{t_M}}{\alpha_{t_m}}\frac{\gamma^2+e^{-2\lambda t_M}}{\gamma^2+e^{-2\lambda t_m}}\frac{1}{2}\delta_m\alpha_{t_m}(1-e^{-2\delta_m})\mathbb{E}[\lVert D_{t_m}-D^\star_{t_m}\rVert^2]=O(\delta^{1.5}). \tag*{(167)} m = 1 ∑ M α t m α t M γ 2 + e − 2 λ t m γ 2 + e − 2 λ t M 2 1 δ m α t m ( 1 − e − 2 δ m ) E [∥ D t m − D t m ⋆ ∥ 2 ] = O ( δ 1.5 ) . (167)
Collecting them into (165) directly leads to the desired result.
Proof of (166) . We begin with the following lower bound:
( γ 2 + e − 2 λ t m γ 2 + e − 2 λ t m + 1 δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m − 1 + e − 2 ( λ t m − λ ) ) 2 ≥ ( δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m − 1 + e − 2 ( λ t m − λ ) ) 2 + 2 ( γ 2 + e − 2 λ t m γ 2 + e − 2 λ t m + 1 − 1 ) δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m ( δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m − 1 + e − 2 ( λ t m − λ ) ) (168) \begin{aligned}
\left(\frac{\gamma^2+e^{-2\lambda t_m}}{\gamma^2+e^{-2\lambda t_{m+1}}}\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m}-1+e^{-2(\lambda_{t_m}-\lambda)}\right)^2
\geq\left(\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m}-1+e^{-2(\lambda_{t_m}-\lambda)}\right)^2 \\
&+2\left(\frac{\gamma^2+e^{-2\lambda t_m}}{\gamma^2+e^{-2\lambda t_{m+1}}}-1\right)\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m}\left(\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m}-1+e^{-2(\lambda_{t_m}-\lambda)}\right) \tag*{(168)}
\end{aligned} ( γ 2 + e − 2 λ t m + 1 γ 2 + e − 2 λ t m 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) − 1 + e − 2 ( λ t m − λ ) ) 2 ≥ ( 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) − 1 + e − 2 ( λ t m − λ ) ) 2 + 2 ( γ 2 + e − 2 λ t m + 1 γ 2 + e − 2 λ t m − 1 ) 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) ( 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) − 1 + e − 2 ( λ t m − λ ) ) (168)
For the second term in the right-hand-side of (168) , we have
( γ 2 + e − 2 λ t m γ 2 + e − 2 λ t m + 1 − 1 ) δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m ≤ e − 2 λ t m + 1 ( e 2 δ m + 1 − 1 ) γ 2 + e − 2 λ t m + 1 δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m ≤ 2 e − 2 λ t m + 1 γ 2 + e − 2 λ t m + 1 δ m + 1 3 δ m + O ( δ 3 ) , \begin{aligned}
\left(\frac{\gamma^2+e^{-2\lambda t_m}}{\gamma^2+e^{-2\lambda t_{m+1}}}-1\right)\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m}
\le\frac{e^{-2\lambda t_{m+1}}(e^{2\delta_{m+1}}-1)}{\gamma^2+e^{-2\lambda t_{m+1}}}\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m} \\
&\le\frac{2e^{-2\lambda t_{m+1}}}{\gamma^2+e^{-2\lambda t_{m+1}}}\frac{\delta_{m+1}^3}{\delta_m}+O(\delta^3),
\end{aligned} ( γ 2 + e − 2 λ t m + 1 γ 2 + e − 2 λ t m − 1 ) 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) ≤ γ 2 + e − 2 λ t m + 1 e − 2 λ t m + 1 ( e 2 δ m + 1 − 1 ) 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) ≤ γ 2 + e − 2 λ t m + 1 2 e − 2 λ t m + 1 δ m δ m + 1 3 + O ( δ 3 ) ,
δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m − 1 + e − 2 ( λ t m − λ ) = δ m + 1 2 δ m − 2 ( λ t m − λ ) + O ( δ 2 ) ≥ − 2 ( λ t m − λ ) + O ( δ 2 ) . \begin{aligned}
\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m}-1+e^{-2(\lambda t_m-\lambda)}
=\frac{\delta_{m+1}^2}{\delta_m}-2(\lambda t_m-\lambda)+O(\delta^2) \\
&\ge-2(\lambda t_m-\lambda)+O(\delta^2).
\end{aligned} 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) − 1 + e − 2 ( λ t m − λ ) = δ m δ m + 1 2 − 2 ( λ t m − λ ) + O ( δ 2 ) ≥ − 2 ( λ t m − λ ) + O ( δ 2 ) .
which leads to the following expression for the second term:
2 ( γ 2 + e − 2 λ t m γ 2 + e − 2 λ t m + 1 − 1 ) δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m ( δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m − 1 + e − 2 ( λ t m − λ ) ) ≥ − 4 e − 2 λ t m + 1 γ 2 + e − 2 λ t m + 1 δ m + 1 3 δ m ( 2 ( λ t m − λ ) ) + O ( δ 4 ) \begin{aligned}
2\left(\frac{\gamma^2+e^{-2\lambda t_m}}{\gamma^2+e^{-2\lambda t_{m+1}}}-1\right)\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m}\left(\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m}-1+e^{-2(\lambda t_m-\lambda)}\right) \\
&\ge-\frac{4e^{-2\lambda t_{m+1}}}{\gamma^2+e^{-2\lambda t_{m+1}}}\frac{\delta_{m+1}^3}{\delta_m}\left(2(\lambda t_m-\lambda)\right)+O(\delta^4)
\end{aligned} 2 ( γ 2 + e − 2 λ t m + 1 γ 2 + e − 2 λ t m − 1 ) 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) ( 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) − 1 + e − 2 ( λ t m − λ ) ) ≥ − γ 2 + e − 2 λ t m + 1 4 e − 2 λ t m + 1 δ m δ m + 1 3 ( 2 ( λ t m − λ ) ) + O ( δ 4 )
Moreover, substituting it into the integral in (166) yields
∫ λ t m − 1 λ t m 2 e 2 λ ( 1 + γ 2 e 2 λ ) 2 ( γ 2 + e − 2 λ t m γ 2 + e − 2 λ t m + 1 − 1 ) δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m ( δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m − 1 + e − 2 ( λ t m − λ ) ) d λ ≥ − ∫ λ t m − 1 λ t m 8 e 2 λ ( 1 + γ 2 e 2 λ ) 2 e − 2 λ t m + 1 γ 2 + e − 2 λ t m + 1 δ m + 1 3 δ m ( λ t m − λ ) d λ + O ( δ 5 ) ≥ − 4 e 2 λ t m ( 1 + γ 2 e 2 λ t m − 1 ) 2 e − 2 λ t m + 1 γ 2 + e − 2 λ t m + 1 δ m + 1 3 δ m + O ( δ 5 ) = O ( δ 4 ) . (169) \begin{aligned}
\int_{\lambda t_{m-1}}^{\lambda t_m}\frac{2e^{2\lambda}}{(1+\gamma^2e^{2\lambda})^2}\left(\frac{\gamma^2+e^{-2\lambda t_m}}{\gamma^2+e^{-2\lambda t_{m+1}}}-1\right)\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m}\left(\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m}-1+e^{-2(\lambda t_m-\lambda)}\right)\,d\lambda\\
&\ge-\int_{\lambda t_{m-1}}^{\lambda t_m}\frac{8e^{2\lambda}}{(1+\gamma^2e^{2\lambda})^2}\frac{e^{-2\lambda t_{m+1}}}{\gamma^2+e^{-2\lambda t_{m+1}}}\frac{\delta_{m+1}^3}{\delta_m}(\lambda t_m-\lambda)\,d\lambda+O(\delta^5) \\
&\ge-\frac{4e^{2\lambda t_m}}{(1+\gamma^2e^{2\lambda t_{m-1}})^2}\frac{e^{-2\lambda t_{m+1}}}{\gamma^2+e^{-2\lambda t_{m+1}}}\delta_{m+1}^3\delta_m+O(\delta^5)=O(\delta^4). \tag*{(169)}
\end{aligned} ∫ λ t m − 1 λ t m ( 1 + γ 2 e 2 λ ) 2 2 e 2 λ ( γ 2 + e − 2 λ t m + 1 γ 2 + e − 2 λ t m − 1 ) 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) ( 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) − 1 + e − 2 ( λ t m − λ ) ) d λ ≥ − ∫ λ t m − 1 λ t m ( 1 + γ 2 e 2 λ ) 2 8 e 2 λ γ 2 + e − 2 λ t m + 1 e − 2 λ t m + 1 δ m δ m + 1 3 ( λ t m − λ ) d λ + O ( δ 5 ) ≥ − ( 1 + γ 2 e 2 λ t m − 1 ) 2 4 e 2 λ t m γ 2 + e − 2 λ t m + 1 e − 2 λ t m + 1 δ m + 1 3 δ m + O ( δ 5 ) = O ( δ 4 ) . (169)
For the first term in the right-hand-side of (168) , we have
∫ λ t m − 1 λ t m e 2 λ ( 1 + γ 2 e 2 λ ) 2 ( δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m − 1 + e − 2 ( λ t m − λ ) ) 2 d λ ≥ ∫ λ t m − 1 λ t m e 2 λ t m − 1 ( 1 + γ 2 e 2 λ t m ) 2 ( δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m − 1 + e − 2 ( λ t m − λ ) ) 2 d λ . (170) \begin{aligned}
\int_{\lambda t_{m-1}}^{\lambda t_m}\frac{e^{2\lambda}}{(1+\gamma^2e^{2\lambda})^2}\left(\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m}-1+e^{-2(\lambda t_m-\lambda)}\right)^2\,d\lambda\\
&\ge\int_{\lambda t_{m-1}}^{\lambda t_m}\frac{e^{2\lambda t_{m-1}}}{(1+\gamma^2e^{2\lambda t_m})^2}\left(\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m}-1+e^{-2(\lambda t_m-\lambda)}\right)^2\,d\lambda. \tag*{(170)}
\end{aligned} ∫ λ t m − 1 λ t m ( 1 + γ 2 e 2 λ ) 2 e 2 λ ( 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) − 1 + e − 2 ( λ t m − λ ) ) 2 d λ ≥ ∫ λ t m − 1 λ t m ( 1 + γ 2 e 2 λ t m ) 2 e 2 λ t m − 1 ( 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) − 1 + e − 2 ( λ t m − λ ) ) 2 d λ . (170)
By direct calculus, we obtain
∫ λ t m − 1 λ t m ( δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m − 1 + e − 2 ( λ t m − λ ) ) 2 d λ = ∫ λ t m − 1 λ t m e − 4 ( λ t m − λ ) d λ + 2 ( δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m − 1 ) ∫ λ t m − 1 λ t m e − 2 ( λ t m − λ ) d λ + ( δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m − 1 ) 2 δ m = 1 4 ( 1 − e − 4 δ m ) + ( δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m − 1 ) ( 1 − e − 2 δ m ) + ( δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m − 1 ) 2 δ m . (171) \begin{aligned}
\int_{\lambda t_{m-1}}^{\lambda t_m}\left(\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m}-1+e^{-2(\lambda t_m-\lambda)}\right)^2\,d\lambda
&=\int_{\lambda t_{m-1}}^{\lambda t_m}e^{-4(\lambda t_m-\lambda)}\,d\lambda\\
&\quad+2\left(\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m}-1\right)\int_{\lambda t_{m-1}}^{\lambda t_m}e^{-2(\lambda t_m-\lambda)}\,d\lambda\\
&\quad+\left(\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m}-1\right)^2\delta_m \\
&=\frac{1}{4}(1-e^{-4\delta_m})+\left(\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m}-1\right)(1-e^{-2\delta_m})+\left(\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m}-1\right)^2\delta_m. \tag*{(171)}
\end{aligned} ∫ λ t m − 1 λ t m ( 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) − 1 + e − 2 ( λ t m − λ ) ) 2 d λ = ∫ λ t m − 1 λ t m e − 4 ( λ t m − λ ) d λ + 2 ( 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) − 1 ) ∫ λ t m − 1 λ t m e − 2 ( λ t m − λ ) d λ + ( 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) − 1 ) 2 δ m = 4 1 ( 1 − e − 4 δ m ) + ( 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) − 1 ) ( 1 − e − 2 δ m ) + ( 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) − 1 ) 2 δ m . (171)
Taylor expansion gives that
1 − e − 4 δ m = 4 δ m − 8 δ m 2 + 32 3 δ m 3 + O ( δ m 4 ) , 1-e^{-4\delta_m}=4\delta_m-8\delta_m^2+\frac{32}{3}\delta_m^3+O(\delta_m^4), 1 − e − 4 δ m = 4 δ m − 8 δ m 2 + 3 32 δ m 3 + O ( δ m 4 ) ,
1 − e − 2 δ m = 2 δ m − 2 δ m 2 + 4 3 δ m 3 + O ( δ m 4 ) . 1-e^{-2\delta_m}=2\delta_m-2\delta_m^2+\frac{4}{3}\delta_m^3+O(\delta_m^4). 1 − e − 2 δ m = 2 δ m − 2 δ m 2 + 3 4 δ m 3 + O ( δ m 4 ) .
Substituting into the right-hand-side of (171) yields
∫ λ t m − 1 λ t m ( δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m − 1 + e − 2 ( λ t m − λ ) ) 2 d λ = δ m − 2 δ m 2 + 8 3 δ m 3 + ( δ m + 1 δ m ( δ m + 1 − δ m + 1 2 + 2 3 δ m + 1 3 + O ( δ m + 1 4 ) ) − 1 ) ( 2 δ m − 2 δ m 2 + 4 3 δ m 3 + O ( δ m 4 ) ) + ( δ m + 1 δ m ( δ m + 1 − δ m + 1 2 + 2 3 δ m + 1 3 + O ( δ m + 1 4 ) ) − 1 ) 2 δ m + O ( δ 4 ) = 4 3 δ m 3 − 2 δ m δ m + 1 2 + δ m + 1 4 δ m + O ( δ 4 ) = 1 3 δ m 3 + ( δ m + 1 2 − δ m 2 ) 2 δ m + O ( δ 4 ) ≥ 1 3 δ m 3 + O ( δ 4 ) . \begin{aligned}
\int_{\lambda t_{m-1}}^{\lambda t_m}\left(\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m}-1+e^{-2(\lambda t_m-\lambda)}\right)^2\,d\lambda\\
&= \delta_m-2\delta_m^2+\frac{8}{3}\delta_m^3+\left(\frac{\delta_{m+1}}{\delta_m}\left(\delta_{m+1}-\delta_{m+1}^2+\frac{2}{3}\delta_{m+1}^3+O(\delta_{m+1}^4)\right)-1\right)\left(2\delta_m-2\delta_m^2+\frac{4}{3}\delta_m^3+O(\delta_m^4)\right)
+\left(\frac{\delta_{m+1}}{\delta_m}\left(\delta_{m+1}-\delta_{m+1}^2+\frac{2}{3}\delta_{m+1}^3+O(\delta_{m+1}^4)\right)-1\right)^2\delta_m+O(\delta^4) \\
&=\frac{4}{3}\delta_m^3-2\delta_m\delta_{m+1}^2+\frac{\delta_{m+1}^4}{\delta_m}+O(\delta^4)
=\frac{1}{3}\delta_m^3+\frac{(\delta_{m+1}^2-\delta_m^2)^2}{\delta_m}+O(\delta^4)
\geq\frac{1}{3}\delta_m^3+O(\delta^4).
\end{aligned} ∫ λ t m − 1 λ t m ( 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) − 1 + e − 2 ( λ t m − λ ) ) 2 d λ = δ m − 2 δ m 2 + 3 8 δ m 3 + ( δ m δ m + 1 ( δ m + 1 − δ m + 1 2 + 3 2 δ m + 1 3 + O ( δ m + 1 4 ) ) − 1 ) ( 2 δ m − 2 δ m 2 + 3 4 δ m 3 + O ( δ m 4 ) ) + ( δ m δ m + 1 ( δ m + 1 − δ m + 1 2 + 3 2 δ m + 1 3 + O ( δ m + 1 4 ) ) − 1 ) 2 δ m + O ( δ 4 ) = 3 4 δ m 3 − 2 δ m δ m + 1 2 + δ m δ m + 1 4 + O ( δ 4 ) = 3 1 δ m 3 + δ m ( δ m + 1 2 − δ m 2 ) 2 + O ( δ 4 ) ≥ 3 1 δ m 3 + O ( δ 4 ) .
Combining with (170) yields
∫ λ t m − 1 λ t m e 2 λ ( 1 + γ 2 e 2 λ ) 2 ( δ m + 1 ( 1 − e − 2 δ m + 1 ) 2 δ m − 1 + e − 2 ( λ t m − λ ) ) 2 d λ ≥ e 2 λ t m − 1 ( 1 + γ 2 e 2 λ t m ) 2 ( 1 3 δ m 3 + O ( δ 4 ) ) . (172) \int_{\lambda_{t_{m-1}}}^{\lambda_{t_m}}\frac{e^{2\lambda}}{(1+\gamma^2e^{2\lambda})^2}\left(\frac{\delta_{m+1}(1-e^{-2\delta_{m+1}})}{2\delta_m}-1+e^{-2(\lambda_{t_m}-\lambda)}\right)^2\mathrm{d}\lambda
\geq\frac{e^{2\lambda_{t_{m-1}}}}{(1+\gamma^2e^{2\lambda_{t_m}})^2}\left(\frac{1}{3}\delta_m^3+O(\delta^4)\right). \tag*{(172)} ∫ λ t m − 1 λ t m ( 1 + γ 2 e 2 λ ) 2 e 2 λ ( 2 δ m δ m + 1 ( 1 − e − 2 δ m + 1 ) − 1 + e − 2 ( λ t m − λ ) ) 2 d λ ≥ ( 1 + γ 2 e 2 λ t m ) 2 e 2 λ t m − 1 ( 3 1 δ m 3 + O ( δ 4 ) ) . (172)
Substituting into (166) yields
V 2 ≥ 2 α t M 2 γ 4 ∑ m = 1 M − 1 ( γ 2 + e − 2 λ t M γ 2 + e − 2 λ t m ) 2 e 2 λ t m − 1 ( 1 + γ 2 e 2 λ t m ) 2 ( 1 3 δ m 3 + O ( δ 4 ) ) V^2\geq2\alpha_{t_M}^2\gamma^4\sum_{m=1}^{M-1}\left(\frac{\gamma^2+e^{-2\lambda_{t_M}}}{\gamma^2+e^{-2\lambda_{t_m}}}\right)^2\frac{e^{2\lambda_{t_m-1}}}{(1+\gamma^2e^{2\lambda_{t_m}})^2}\left(\frac{1}{3}\delta_m^3+O(\delta^4)\right) V 2 ≥ 2 α t M 2 γ 4 m = 1 ∑ M − 1 ( γ 2 + e − 2 λ t m γ 2 + e − 2 λ t M ) 2 ( 1 + γ 2 e 2 λ t m ) 2 e 2 λ t m − 1 ( 3 1 δ m 3 + O ( δ 4 ) )
≥ 2 α t M 2 γ 4 ( γ 2 + e − 2 λ t M γ 2 + e − 2 λ t 1 ) 2 e − 2 λ t M − 2 δ M ( γ 2 + e − 2 λ t M ) 2 ∑ m = 1 M − 1 ( 1 3 δ m 3 + O ( δ 4 ) ) ≳ δ 2 . \geq2\alpha_{t_M}^2\gamma^4\left(\frac{\gamma^2+e^{-2\lambda_{t_M}}}{\gamma^2+e^{-2\lambda_{t_1}}}\right)^2\frac{e^{-2\lambda_{t_M}-2\delta_M}}{(\gamma^2+e^{-2\lambda_{t_M}})^2}\sum_{m=1}^{M-1}\left(\frac{1}{3}\delta_m^3+O(\delta^4)\right)\gtrsim\delta^2. ≥ 2 α t M 2 γ 4 ( γ 2 + e − 2 λ t 1 γ 2 + e − 2 λ t M ) 2 ( γ 2 + e − 2 λ t M ) 2 e − 2 λ t M − 2 δ M m = 1 ∑ M − 1 ( 3 1 δ m 3 + O ( δ 4 ) ) ≳ δ 2 .
where the last inequality holds for sufficiently large M M M , since
∑ m = 1 M − 1 δ m 3 ≥ ( λ t M − λ t 0 − δ M ) 3 ( M − 1 ) 2 = Ω ( M − 2 ) = Ω ( δ 2 ) . \sum_{m=1}^{M-1}\delta_m^3\geq\frac{(\lambda_{t_M}-\lambda_{t_0}-\delta_M)^3}{(M-1)^2}=\Omega(M^{-2})=\Omega(\delta^2). m = 1 ∑ M − 1 δ m 3 ≥ ( M − 1 ) 2 ( λ t M − λ t 0 − δ M ) 3 = Ω ( M − 2 ) = Ω ( δ 2 ) .
This completes the proof of (166) .
Proof of (167) . By the expression for data predictor in (152) and the definition of D t i ⋆ D_{t_i}^\star D t i ⋆ in (163) , we have
D t i − D t i ⋆ = 1 δ i − 1 ( γ 2 γ 2 + e − 2 λ t i − 1 x t i − 1 − x t i − 1 ⋆ α t i − 1 − γ 2 γ 2 + e − 2 λ t i − 2 x t i − 2 − x t i − 2 ⋆ α t i − 2 ) . (173) D_{t_i}-D_{t_i}^\star=\frac{1}{\delta_{i-1}}\left(\frac{\gamma^2}{\gamma^2+e^{-2\lambda_{t_{i-1}}}}\frac{x_{t_{i-1}}-x_{t_{i-1}}^\star}{\alpha_{t_{i-1}}}-\frac{\gamma^2}{\gamma^2+e^{-2\lambda_{t_{i-2}}}}\frac{x_{t_{i-2}}-x_{t_{i-2}}^\star}{\alpha_{t_{i-2}}}\right). \tag*{(173)} D t i − D t i ⋆ = δ i − 1 1 ( γ 2 + e − 2 λ t i − 1 γ 2 α t i − 1 x t i − 1 − x t i − 1 ⋆ − γ 2 + e − 2 λ t i − 2 γ 2 α t i − 2 x t i − 2 − x t i − 2 ⋆ ) . (173)
While by (162) , the above difference is calculated as
γ 2 γ 2 + e − 2 λ t i − 1 x t i − 1 − x t i − 1 ⋆ α t i − 1 − γ 2 γ 2 + e − 2 λ t i − 2 x t i − 2 − x t i − 2 ⋆ α t i − 2 = γ 2 δ i − 1 ( 1 − e − 2 δ i − 1 ) 2 ( γ 2 + e − 2 λ t i − 1 ) D t i − 1 − 2 γ 4 γ 2 + e − 2 λ t i − 1 ∫ λ t i − 2 λ t i − 1 e λ 1 − e − 2 ( λ t i − 1 − λ ) 1 + γ 2 e 2 λ d w ‾ t ( λ ) . \frac{\gamma^2}{\gamma^2+e^{-2\lambda_{t_{i-1}}}}\frac{x_{t_{i-1}}-x_{t_{i-1}}^\star}{\alpha_{t_{i-1}}}-\frac{\gamma^2}{\gamma^2+e^{-2\lambda_{t_{i-2}}}}\frac{x_{t_{i-2}}-x_{t_{i-2}}^\star}{\alpha_{t_{i-2}}}
=\frac{\gamma^2\delta_{i-1}(1-e^{-2\delta_{i-1}})}{2(\gamma^2+e^{-2\lambda_{t_{i-1}}})}D_{t_{i-1}}-\frac{\sqrt{2}\gamma^4}{\gamma^2+e^{-2\lambda_{t_{i-1}}}}\int_{\lambda_{t_{i-2}}}^{\lambda_{t_{i-1}}}e^\lambda\frac{1-e^{-2(\lambda_{t_{i-1}}-\lambda)}}{1+\gamma^2e^{2\lambda}}\mathrm{d}\overline{w}_t(\lambda). γ 2 + e − 2 λ t i − 1 γ 2 α t i − 1 x t i − 1 − x t i − 1 ⋆ − γ 2 + e − 2 λ t i − 2 γ 2 α t i − 2 x t i − 2 − x t i − 2 ⋆ = 2 ( γ 2 + e − 2 λ t i − 1 ) γ 2 δ i − 1 ( 1 − e − 2 δ i − 1 ) D t i − 1 − γ 2 + e − 2 λ t i − 1 2 γ 4 ∫ λ t i − 2 λ t i − 1 e λ 1 + γ 2 e 2 λ 1 − e − 2 ( λ t i − 1 − λ ) d w t ( λ ) .
Substituting it into (173) further yields
D t i − D t i ⋆ = γ 2 ( 1 − e − 2 δ i − 1 ) 2 ( γ 2 + e − 2 λ t i − 1 ) D t i − 1 − 2 γ 4 δ i − 1 ( γ 2 + e − 2 λ t i − 1 ) ∫ λ t i − 2 λ t i − 1 e λ 1 − e − 2 ( λ t i − 1 − λ ) 1 + γ 2 e 2 λ d w ‾ t ( λ ) = γ 2 ( 1 − e − 2 δ i − 1 ) 2 ( γ 2 + e − 2 λ t i − 1 ) ( D t i − 1 ⋆ + D t i − 1 − D t i − 1 ⋆ ) − 2 γ 4 δ i − 1 ( γ 2 + e − 2 λ t i − 1 ) ∫ λ t i − 2 λ t i − 1 e λ 1 − e − 2 ( λ t i − 1 − λ ) 1 + γ 2 e 2 λ d w ‾ t ( λ ) . (174) \begin{aligned}
D_{t_i}-D_{t_i}^\star=\frac{\gamma^2(1-e^{-2\delta_{i-1}})}{2(\gamma^2+e^{-2\lambda_{t_{i-1}}})}D_{t_{i-1}}-\frac{\sqrt{2}\gamma^4}{\delta_{i-1}(\gamma^2+e^{-2\lambda_{t_{i-1}}})}\int_{\lambda_{t_{i-2}}}^{\lambda_{t_{i-1}}}e^\lambda\frac{1-e^{-2(\lambda_{t_{i-1}}-\lambda)}}{1+\gamma^2e^{2\lambda}}\mathrm{d}\overline{w}_t(\lambda) \\
&=\frac{\gamma^2(1-e^{-2\delta_{i-1}})}{2(\gamma^2+e^{-2\lambda_{t_{i-1}}})}\left(D_{t_{i-1}}^\star+D_{t_{i-1}}-D_{t_{i-1}}^\star\right)-\frac{\sqrt{2}\gamma^4}{\delta_{i-1}(\gamma^2+e^{-2\lambda_{t_{i-1}}})}\int_{\lambda_{t_{i-2}}}^{\lambda_{t_{i-1}}}e^\lambda\frac{1-e^{-2(\lambda_{t_{i-1}}-\lambda)}}{1+\gamma^2e^{2\lambda}}\mathrm{d}\overline{w}_t(\lambda). \tag*{(174)}
\end{aligned} D t i − D t i ⋆ = 2 ( γ 2 + e − 2 λ t i − 1 ) γ 2 ( 1 − e − 2 δ i − 1 ) D t i − 1 − δ i − 1 ( γ 2 + e − 2 λ t i − 1 ) 2 γ 4 ∫ λ t i − 2 λ t i − 1 e λ 1 + γ 2 e 2 λ 1 − e − 2 ( λ t i − 1 − λ ) d w t ( λ ) = 2 ( γ 2 + e − 2 λ t i − 1 ) γ 2 ( 1 − e − 2 δ i − 1 ) ( D t i − 1 ⋆ + D t i − 1 − D t i − 1 ⋆ ) − δ i − 1 ( γ 2 + e − 2 λ t i − 1 ) 2 γ 4 ∫ λ t i − 2 λ t i − 1 e λ 1 + γ 2 e 2 λ 1 − e − 2 ( λ t i − 1 − λ ) d w t ( λ ) . (174)
For the last term, we have
E [ ∥ ∫ λ t i − 2 λ t i − 1 e λ 1 − e − 2 ( λ t i − 1 − λ ) 1 + γ 2 e 2 λ d w ‾ t ( λ ) ∥ 2 2 ] = ∫ λ t i − 2 λ t i − 1 e 2 λ ( 1 − e − 2 ( λ t i − 1 − λ ) ) 2 ( 1 + γ 2 e 2 λ ) 2 d λ ≤ ∫ λ t i − 2 λ t i − 1 e 2 λ t i − 1 4 ( λ t i − 1 − λ ) 2 ( 1 + γ 2 e 2 λ t i − 2 ) 2 d λ = 4 δ i − 1 3 e 2 λ t i − 1 3 ( 1 + γ 2 e 2 λ t i − 2 ) 2 = 4 δ i − 1 3 e 2 λ t i − 1 − 4 λ t i − 2 3 ( γ 2 + e − 2 λ t i − 2 ) 2 \begin{aligned}
\mathbb{E}\left[\left\|\int_{\lambda_{t_{i-2}}}^{\lambda_{t_{i-1}}}e^\lambda\frac{1-e^{-2(\lambda_{t_{i-1}}-\lambda)}}{1+\gamma^2e^{2\lambda}}\mathrm{d}\overline{w}_t(\lambda)\right\|_2^2\right]
=\int_{\lambda_{t_{i-2}}}^{\lambda_{t_{i-1}}}e^{2\lambda}\frac{(1-e^{-2(\lambda_{t_{i-1}}-\lambda)})^2}{(1+\gamma^2e^{2\lambda})^2}\mathrm{d}\lambda\\
&\leq\int_{\lambda_{t_{i-2}}}^{\lambda_{t_{i-1}}}e^{2\lambda_{t_{i-1}}}\frac{4(\lambda_{t_{i-1}}-\lambda)^2}{(1+\gamma^2e^{2\lambda_{t_{i-2}}})^2}\mathrm{d}\lambda\\
&= \frac{4\delta_{i-1}^{3}e^{2\lambda t_{i-1}}}{3(1+\gamma^{2}e^{2\lambda t_{i-2}})^{2}} = \frac{4\delta_{i-1}^{3}e^{2\lambda t_{i-1}-4\lambda t_{i-2}}}{3(\gamma^{2}+e^{-2\lambda t_{i-2}})^{2}}
\end{aligned} E ∫ λ t i − 2 λ t i − 1 e λ 1 + γ 2 e 2 λ 1 − e − 2 ( λ t i − 1 − λ ) d w t ( λ ) 2 2 = ∫ λ t i − 2 λ t i − 1 e 2 λ ( 1 + γ 2 e 2 λ ) 2 ( 1 − e − 2 ( λ t i − 1 − λ ) ) 2 d λ ≤ ∫ λ t i − 2 λ t i − 1 e 2 λ t i − 1 ( 1 + γ 2 e 2 λ t i − 2 ) 2 4 ( λ t i − 1 − λ ) 2 d λ = 3 ( 1 + γ 2 e 2 λ t i − 2 ) 2 4 δ i − 1 3 e 2 λ t i − 1 = 3 ( γ 2 + e − 2 λ t i − 2 ) 2 4 δ i − 1 3 e 2 λ t i − 1 − 4 λ t i − 2
≲ δ i − 1 3 e − 2 λ t i − 2 ( γ 2 + e − 2 λ t i − 2 ) 2 . \lesssim\frac{\delta_{i-1}^{3}e^{-2\lambda t_{i-2}}}{(\gamma^{2}+e^{-2\lambda t_{i-2}})^{2}}. ≲ ( γ 2 + e − 2 λ t i − 2 ) 2 δ i − 1 3 e − 2 λ t i − 2 .
Note that the factor before D t i − 1 − D t i − 1 ⋆ D_{t_{i-1}}-D^{\star}_{t_{i-1}} D t i − 1 − D t i − 1 ⋆ in (174) satisfies
γ 2 ( 1 − e − 2 δ i − 1 ) 2 ( γ 2 + e − 2 λ t i − 1 ) ≤ 1 − e − 2 δ i − 1 2 ≤ δ i − 1 . \frac{\gamma^{2}(1-e^{-2\delta_{i-1}})}{2(\gamma^{2}+e^{-2\lambda t_{i-1}})}
\le\frac{1-e^{-2\delta_{i-1}}}{2}\le\delta_{i-1}. 2 ( γ 2 + e − 2 λ t i − 1 ) γ 2 ( 1 − e − 2 δ i − 1 ) ≤ 2 1 − e − 2 δ i − 1 ≤ δ i − 1 .
Therefore we have
E [ ∥ D t i − D t i ⋆ ∥ 2 ] ≤ δ i − 1 E [ ∥ D t i − 1 − D t i − 1 ⋆ ∥ 2 ] + δ i − 1 E [ ∥ D t i − 1 ⋆ ∥ ] + O ( γ 4 δ i − 1 0.5 e − λ t i − 2 ( γ 2 + e − 2 λ t i − 2 ) 2 ) (175) \mathbb{E}[\lVert D_{t_i}-D^{\star}_{t_i}\rVert^{2}]
\le\delta_{i-1}\mathbb{E}[\lVert D_{t_{i-1}}-D^{\star}_{t_{i-1}}\rVert^{2}]
+\delta_{i-1}\mathbb{E}[\lVert D^{\star}_{t_{i-1}}\rVert]
+O\left(\frac{\gamma^{4}\delta_{i-1}^{0.5}e^{-\lambda t_{i-2}}}{(\gamma^{2}+e^{-2\lambda t_{i-2}})^{2}}\right) \tag*{(175)} E [∥ D t i − D t i ⋆ ∥ 2 ] ≤ δ i − 1 E [∥ D t i − 1 − D t i − 1 ⋆ ∥ 2 ] + δ i − 1 E [∥ D t i − 1 ⋆ ∥] + O ( ( γ 2 + e − 2 λ t i − 2 ) 2 γ 4 δ i − 1 0.5 e − λ t i − 2 ) (175)
By (164) , we have
E [ ∥ D t i − 1 ⋆ ∥ ] 2 ≤ E [ ∥ D t i − 1 ⋆ ∥ 2 ] = 2 γ 4 δ i − 2 2 ∫ λ t i − 3 λ t i − 2 e 2 λ ( 1 + γ 2 e 2 λ ) 2 d λ ≤ 2 γ 4 δ i − 2 e − 2 λ t i − 3 δ i − 2 2 ( γ 2 + e − 2 λ t i − 2 ) 2 = 2 γ 4 e − 2 λ t i − 3 δ i − 2 ( γ 2 + e − 2 λ t i − 2 ) 2 . (176) \begin{aligned}
\mathbb{E}[\lVert D^{\star}_{t_{i-1}}\rVert]^{2}
\le\mathbb{E}[\lVert D^{\star}_{t_{i-1}}\rVert^{2}]
= \frac{2\gamma^{4}}{\delta_{i-2}^{2}}\int_{\lambda t_{i-3}}^{\lambda t_{i-2}}\frac{e^{2\lambda}}{(1+\gamma^{2}e^{2\lambda})^{2}}\,\mathrm{d}\lambda\\
&\le\frac{2\gamma^{4}\delta_{i-2}e^{-2\lambda t_{i-3}}}{\delta_{i-2}^{2}(\gamma^{2}+e^{-2\lambda t_{i-2}})^{2}}
= \frac{2\gamma^{4}e^{-2\lambda t_{i-3}}}{\delta_{i-2}(\gamma^{2}+e^{-2\lambda t_{i-2}})^{2}}. \tag*{(176)}
\end{aligned} E [∥ D t i − 1 ⋆ ∥ ] 2 ≤ E [∥ D t i − 1 ⋆ ∥ 2 ] = δ i − 2 2 2 γ 4 ∫ λ t i − 3 λ t i − 2 ( 1 + γ 2 e 2 λ ) 2 e 2 λ d λ ≤ δ i − 2 2 ( γ 2 + e − 2 λ t i − 2 ) 2 2 γ 4 δ i − 2 e − 2 λ t i − 3 = δ i − 2 ( γ 2 + e − 2 λ t i − 2 ) 2 2 γ 4 e − 2 λ t i − 3 . (176)
Substituting into (175) yields
E [ ∥ D t i − D t i ⋆ ∥ 2 ] ≲ O ( δ ) . \mathbb{E}[\lVert D_{t_i}-D^{\star}_{t_i}\rVert^{2}]\lesssim O(\sqrt{\delta}). E [∥ D t i − D t i ⋆ ∥ 2 ] ≲ O ( δ ) .
Substituting into the left-hand-side of (167) yields
∑ m = 1 M α t M α t m γ 2 + e − 2 λ t M γ 2 + e − 2 λ t m 1 2 δ m α t m ( 1 − e − 2 δ m ) E [ ∥ D t m − D t m ⋆ ∥ 2 ] ≤ ∑ m = 1 M α t M α t m γ 2 + e − 2 λ t M γ 2 + e − 2 λ t m 1 2 δ m α t m ( 1 − e − 2 δ m ) O ( δ ) ≤ O ( δ ) ∑ m = 1 M α t M ( γ 2 + e − 2 λ t M ) γ 2 + e − 2 λ t m δ m 2 = O ( δ 1.5 ) . (177) \begin{aligned}
\sum_{m=1}^{M}\frac{\alpha_{t_M}}{\alpha_{t_m}}\frac{\gamma^{2}+e^{-2\lambda t_M}}{\gamma^{2}+e^{-2\lambda t_m}}\frac{1}{2}\delta_m\alpha_{t_m}(1-e^{-2\delta_m})\mathbb{E}[\lVert D_{t_m}-D^{\star}_{t_m}\rVert^{2}] \\
&\le\sum_{m=1}^{M}\frac{\alpha_{t_M}}{\alpha_{t_m}}\frac{\gamma^{2}+e^{-2\lambda t_M}}{\gamma^{2}+e^{-2\lambda t_m}}\frac{1}{2}\delta_m\alpha_{t_m}(1-e^{-2\delta_m})O(\sqrt{\delta}) \\
&\le O(\sqrt{\delta})\sum_{m=1}^{M}\frac{\alpha_{t_M}(\gamma^{2}+e^{-2\lambda t_M})}{\gamma^{2}+e^{-2\lambda t_m}}\delta_m^{2}
=O(\delta^{1.5}). \tag*{(177)}
\end{aligned} m = 1 ∑ M α t m α t M γ 2 + e − 2 λ t m γ 2 + e − 2 λ t M 2 1 δ m α t m ( 1 − e − 2 δ m ) E [∥ D t m − D t m ⋆ ∥ 2 ] ≤ m = 1 ∑ M α t m α t M γ 2 + e − 2 λ t m γ 2 + e − 2 λ t M 2 1 δ m α t m ( 1 − e − 2 δ m ) O ( δ ) ≤ O ( δ ) m = 1 ∑ M γ 2 + e − 2 λ t m α t M ( γ 2 + e − 2 λ t M ) δ m 2 = O ( δ 1.5 ) . (177)
This completes the proof.
Proof of convergence order upper bound for F-DDIM (Theorem 4)
We write δ = max i ∈ [ M ] δ i = O ( M − 1 ) \delta=\max_{i\in[M]}\delta_i=O(M^{-1}) δ = max i ∈ [ M ] δ i = O ( M − 1 ) . To establish Theorem 4 , we first consider the convergence order of the following (idealized) first-order forward-value discretization of the diffusion ODE (12) . For each iteration i = 1 , … , M i=1,\ldots,M i = 1 , … , M , given the current iterate x t i − 1 f o r x^{\mathrm{for}}_{t_{i-1}} x t i − 1 for at time t i − 1 t_{i-1} t i − 1 , we compute the next iterate x t i f o r x^{\mathrm{for}}_{t_i} x t i for by
x t i f o r = σ t i σ t i − 1 x t i − 1 f o r + ( α t i − σ t i α t i − 1 σ t i − 1 ) μ θ ( x t i − 1 f o r , t i ) . (178) x^{\mathrm{for}}_{t_i}
=\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}x^{\mathrm{for}}_{t_{i-1}}
+\left(\alpha_{t_i}-\frac{\sigma_{t_i}\alpha_{t_{i-1}}}{\sigma_{t_{i-1}}}\right)\mu_{\theta}(x^{\mathrm{for}}_{t_{i-1}},t_i). \tag*{(178)} x t i for = σ t i − 1 σ t i x t i − 1 for + ( α t i − σ t i − 1 σ t i α t i − 1 ) μ θ ( x t i − 1 for , t i ) . (178)
For comparison, the deterministic DDIM update (14) can be written in the data-prediction form as
x t i b c k = σ t i σ t i − 1 x t i − 1 b c k + ( α t i − σ t i α t i − 1 σ t i − 1 ) μ θ ( x t i − 1 b c k , t i − 1 ) . (179) x^{\mathrm{bck}}_{t_i}
=\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}x^{\mathrm{bck}}_{t_{i-1}}
+\left(\alpha_{t_i}-\frac{\sigma_{t_i}\alpha_{t_{i-1}}}{\sigma_{t_{i-1}}}\right)\mu_{\theta}(x^{\mathrm{bck}}_{t_{i-1}},t_{i-1}). \tag*{(179)} x t i bck = σ t i − 1 σ t i x t i − 1 bck + ( α t i − σ t i − 1 σ t i α t i − 1 ) μ θ ( x t i − 1 bck , t i − 1 ) . (179)
The only difference between (178) and (179) is whether the data prediction model μ θ ( x , t ) \mu_\theta(x,t) μ θ ( x , t ) is evaluated at the next state/time ( x t i b c k , t i ) (x^{\mathrm{bck}}_{t_i},t_i) ( x t i bck , t i ) , or the current state/time ( x t i − 1 b c k , t i − 1 ) (x^{\mathrm{bck}}_{t_{i-1}},t_{i-1}) ( x t i − 1 bck , t i − 1 ) . As a note, while the update rule of the naive forward-value discretization (178) is not directly implementable, it serves as a clean lens for understanding how evaluation placement affects discretization error.
Interestingly, it turns out that the first-order discretization errors introduced by these two strategies have exactly opposite signs, as formalized in the following theorem. The proof is postponed to Section E.1.
Theorem 6. Under Assumption 1, the idealized forward-value discretization scheme (178) has convergence order 1. Moreover, let x t M f o r x^{\mathrm{for}}_{t_M} x t M for and x t M b c k x^{\mathrm{bck}}_{t_M} x t M bck denote the outputs of the forward-value discretization (178) and DDIM (179) , respectively, and let x t M ⋆ x^\star_{t_M} x t M ⋆ be the exact ODE solution defined in (30) , with the common initialization x t 0 b c k = x t 0 f o r = x t 0 ⋆ x^{\mathrm{bck}}_{t_0}=x^{\mathrm{for}}_{t_0}=x^\star_{t_0} x t 0 bck = x t 0 for = x t 0 ⋆ . Then we have
∥ x t M b c k − x t M ⋆ + x t M f o r − x t M ⋆ ∥ 2 = O ( 1 / M 2 ) . (180) \left\|x^{\mathrm{bck}}_{t_M}-x^\star_{t_M}+x^{\mathrm{for}}_{t_M}-x^\star_{t_M}\right\|_2=O(1/M^2). \tag*{(180)} x t M bck − x t M ⋆ + x t M for − x t M ⋆ 2 = O ( 1/ M 2 ) . (180)
Theorem 6 implies that the leading O ( 1 / M ) O(1/M) O ( 1/ M ) errors of the forward-value and backward-value discretizations cancel to second order:
x t M b c k − x t M ⋆ ≈ − ( x t M f o r − x t M ⋆ ) . x^{\mathrm{bck}}_{t_M}-x^\star_{t_M}\approx-\left(x^{\mathrm{for}}_{t_M}-x^\star_{t_M}\right). x t M bck − x t M ⋆ ≈ − ( x t M for − x t M ⋆ ) .
Conceptually, this suggests that if one can (even approximately) access forward-value information, then combining it with the standard backward-value update can greatly reduce the most harmful part of the discretization error.
Next, we provide the convergence guarantee of the proposed sampler in Theorem 7 below; with proof postponed to Section E.2.
Theorem 7. Suppose that Assumption 1 holds. In addition, assume that the one-step lookahead estimate x ^ t i \hat{x}_{t_i} x ^ t i satisfies
∥ x ^ t i − x t ⋆ ( x t i − 1 , t i − 1 ) ∥ = o ( 1 / M ) , i ≥ 2 , and ∥ x ^ t 1 − x t ⋆ ( x t 0 , t 0 ) ∥ = O ( 1 / M ) , (181) \left\|\hat{x}_{t_i}-x^\star_t(x_{t_{i-1}},t_{i-1})\right\|=o(1/M),\quad i\geq2,\qquad\text{and}\qquad\left\|\hat{x}_{t_1}-x^\star_t(x_{t_0},t_0)\right\|=O(1/M), \tag*{(181)} ∥ x ^ t i − x t ⋆ ( x t i − 1 , t i − 1 ) ∥ = o ( 1/ M ) , i ≥ 2 , and ∥ x ^ t 1 − x t ⋆ ( x t 0 , t 0 ) ∥ = O ( 1/ M ) , (181) where x t ⋆ ( x t i − 1 , t i − 1 ) x^\star_t(x_{t_{i-1}},t_{i-1}) x t ⋆ ( x t i − 1 , t i − 1 ) denotes the solution to the diffusion ODE at time t ≤ t i − 1 t\leq t_{i-1} t ≤ t i − 1 when initialized at time t i − 1 t_{i-1} t i − 1 with state x t i − 1 x_{t_{i-1}} x t i − 1 , i.e.,
x t ⋆ ( x t i − 1 , t i − 1 ) = α t α t i − 1 x t i − 1 − α t ∫ λ t i − 1 λ t e − λ ϵ θ ( x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) , t ( λ ) ) d λ , ∀ t ≤ t i − 1 . (182) x^\star_t(x_{t_{i-1}},t_{i-1})=\frac{\alpha_t}{\alpha_{t_{i-1}}}x_{t_{i-1}}-\alpha_t\int_{\lambda_{t_{i-1}}}^{\lambda_t}e^{-\lambda}\epsilon_\theta(x^\star_{t(\lambda)}(x_{t_{i-1}},t_{i-1}),t(\lambda))\,\mathrm{d}\lambda,\quad\forall\ t\leq t_{i-1}. \tag*{(182)} x t ⋆ ( x t i − 1 , t i − 1 ) = α t i − 1 α t x t i − 1 − α t ∫ λ t i − 1 λ t e − λ ϵ θ ( x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) , t ( λ )) d λ , ∀ t ≤ t i − 1 . (182) Then the proposed sampler (21) satisfies
∥ x t M − x t M f o r ∥ = o ( 1 / M ) , \left\|x_{t_M}-x^{\mathrm{for}}_{t_M}\right\|=o(1/M), x t M − x t M for = o ( 1/ M ) , where x t f o r x^{\mathrm{for}}_t x t for represents the idealized forward-value iterate defined in (178) with the same initialization as x t x_t x t . In particular, the convergence order of the sampler (21) is 1. For F-DDIM in Algorithm 1, (181) holds uniformly for i ≥ 2 i\geq2 i ≥ 2 ; the initialization ϵ ~ t 0 = x t 0 \tilde{\epsilon}_{t_0}=x_{t_0} ϵ ~ t 0 = x t 0 gives only an O ( 1 / M ) O(1/M) O ( 1/ M ) lookahead error at i = 1 i=1 i = 1 . Nevertheless,
max i ∈ [ M ] ∥ x t i − x t i f o r ∥ = O ( 1 / M 2 ) , \max_{i\in[M]}\left\|x_{t_i}-x^{\mathrm{for}}_{t_i}\right\|=O(1/M^2), i ∈ [ M ] max x t i − x t i for = O ( 1/ M 2 ) , so F-DDIM also has convergence order 1.
This theorem guarantees that lookahead errors satisfying (181) are sufficiently small for the implemented method to inherit the first-order rate of the idealized forward-value scheme. Theorem 6 together with Theorem 7 directly yields Theorem 4. For the remainder of this section, we focus on the proofs of Theorems 6 and 7.
For ease of presentation, we use ∇ k μ ( x , t ) \nabla^k\mu(x,t) ∇ k μ ( x , t ) to denote the k k k -th order gradient of μ ( x , t ) \mu(x,t) μ ( x , t ) with respect to x x x . Before diving into the proof details, we note that the diffusion ODE in (12) can be equivalently written in terms of the data prediction model μ θ \mu_\theta μ θ . Indeed, (12) comes from the ODE (4) . Using the approximation s t ⋆ ( x , t ) ≈ ( α t μ θ ( x , t ) − x ) / σ t 2 s^\star_t(x,t)\approx(\alpha_t\mu_\theta(x,t)-x)/\sigma_t^2 s t ⋆ ( x , t ) ≈ ( α t μ θ ( x , t ) − x ) / σ t 2 and substituting it into (4) yields the following diffusion ODE associated with a data prediction model:
x ˙ t = ( f ( t ) + g 2 ( t ) 2 σ t 2 ) x t − α t g 2 ( t ) 2 σ t 2 μ θ ( x t , t ) , x T ∼ N ( 0 , σ ~ 2 I d ) . (183) \dot{x}_t=\left(f(t)+\frac{g^2(t)}{2\sigma_t^2}\right)x_t-\frac{\alpha_tg^2(t)}{2\sigma_t^2}\mu_\theta(x_t,t),\qquad x_T\sim\mathcal{N}(0,\widetilde{\sigma}^2I_d). \tag*{(183)} x ˙ t = ( f ( t ) + 2 σ t 2 g 2 ( t ) ) x t − 2 σ t 2 α t g 2 ( t ) μ θ ( x t , t ) , x T ∼ N ( 0 , σ 2 I d ) . (183)
Solving this ODE from T T T to 0 0 0 again produces a sample x 0 \mathbf{x}_0 x 0 . Under the same change of variables λ ( t ) \lambda(t) λ ( t ) , the solution admits the integral form: for t < s t<s t < s ,
x t = σ t σ s x s + σ t ∫ λ ( s ) λ ( t ) e λ μ θ ( x t ( λ ) , t ( λ ) ) d λ . (184) \mathbf{x}_t=\frac{\sigma_t}{\sigma_s}\mathbf{x}_s+\sigma_t\int_{\lambda(s)}^{\lambda(t)}e^\lambda\boldsymbol{\mu}_\theta(\mathbf{x}_{t(\lambda)},t(\lambda))\,\mathrm{d}\lambda. \tag*{(184)} x t = σ s σ t x s + σ t ∫ λ ( s ) λ ( t ) e λ μ θ ( x t ( λ ) , t ( λ )) d λ . (184)
Finally, we introduce the total derivative of a function μ θ ( x t ( λ ) , t ( λ ) ) \boldsymbol{\mu}_\theta(\mathbf{x}_{t(\lambda)},t(\lambda)) μ θ ( x t ( λ ) , t ( λ )) with respect to log-SNR λ \lambda λ along a trajectory x t ( λ ) \mathbf{x}_{t(\lambda)} x t ( λ ) as
d d λ μ θ ( x t ( λ ) , t ( λ ) ) = ∇ x μ θ ( x t ( λ ) , t ( λ ) ) d x t ( λ ) d λ + ∂ μ θ ∂ t ( x t ( λ ) , t ( λ ) ) d t ( λ ) d λ . (185) \frac{\mathrm{d}}{\mathrm{d}\lambda}\boldsymbol{\mu}_\theta(\mathbf{x}_{t(\lambda)},t(\lambda))=\nabla_{\mathbf{x}}\boldsymbol{\mu}_\theta(\mathbf{x}_{t(\lambda)},t(\lambda))\frac{\mathrm{d}\mathbf{x}_{t(\lambda)}}{\mathrm{d}\lambda}+\frac{\partial\boldsymbol{\mu}_\theta}{\partial t}(\mathbf{x}_{t(\lambda)},t(\lambda))\frac{\mathrm{d}t(\lambda)}{\mathrm{d}\lambda}. \tag*{(185)} d λ d μ θ ( x t ( λ ) , t ( λ )) = ∇ x μ θ ( x t ( λ ) , t ( λ )) d λ d x t ( λ ) + ∂ t ∂ μ θ ( x t ( λ ) , t ( λ )) d λ d t ( λ ) . (185)
On a fixed finite log-SNR interval I = [ λ t 0 , λ t M ] I=[\lambda_{t_0},\lambda_{t_M}] I = [ λ t 0 , λ t M ] , by the integral form (184) and the Assumption 1 (A1), there exists R < ∞ R<\infty R < ∞ such that
max λ ∈ I ∥ x t ( λ ) ∥ ≤ R . \max_{\lambda\in I}\lVert\mathbf{x}_{t(\lambda)}\rVert\le R. λ ∈ I max ∥ x t ( λ ) ∥ ≤ R .
Let
α min = min λ ∈ I α λ , and α max = max λ ∈ I α λ \alpha_{\min}=\min_{\lambda\in I}\alpha_\lambda,\qquad\text{and}\qquad\alpha_{\max}=\max_{\lambda\in I}\alpha_\lambda α m i n = λ ∈ I min α λ , and α m a x = λ ∈ I max α λ
The derivative of μ θ ( x t ( λ ) , t ( λ ) ) \boldsymbol{\mu}_\theta(\mathbf{x}_{t(\lambda)},t(\lambda)) μ θ ( x t ( λ ) , t ( λ )) with respect to λ \lambda λ is bounded by
∥ d d λ μ θ ( x t ( λ ) , t ( λ ) ) ∥ ≤ ∥ ∇ x μ θ ( x t ( λ ) , t ( λ ) ) ∥ ∥ d x t ( λ ) d λ ∥ + ∥ ∂ μ θ ∂ t ( x t ( λ ) , t ( λ ) ) d t ( λ ) d λ ∥ ≤ L x α t ( λ ) ∥ d log σ λ d λ x t ( λ ) + α λ μ θ ( x t ( λ ) , t ( λ ) ) ∥ + L t ≤ L x α min ( R max λ ∈ I ∣ d log σ λ d λ ∣ + α max L μ ) + L t = : L ~ t , (186) \begin{aligned}
\left\lVert\frac{\mathrm{d}}{\mathrm{d}\lambda}\boldsymbol{\mu}_\theta(\mathbf{x}_{t(\lambda)},t(\lambda))\right\rVert
&\le\left\lVert\nabla_{\mathbf{x}}\boldsymbol{\mu}_\theta(\mathbf{x}_{t(\lambda)},t(\lambda))\right\rVert\left\lVert\frac{\mathrm{d}\mathbf{x}_{t(\lambda)}}{\mathrm{d}\lambda}\right\rVert+\left\lVert\frac{\partial\boldsymbol{\mu}_\theta}{\partial t}(\mathbf{x}_{t(\lambda)},t(\lambda))\frac{\mathrm{d}t(\lambda)}{\mathrm{d}\lambda}\right\rVert\\
&\le\frac{L_x}{\alpha_{t(\lambda)}}\left\lVert\frac{\mathrm{d}\log\sigma_\lambda}{\mathrm{d}\lambda}\mathbf{x}_{t(\lambda)}+\alpha_\lambda\boldsymbol{\mu}_\theta(\mathbf{x}_{t(\lambda)},t(\lambda))\right\rVert+L_t\\
&\le\frac{L_x}{\alpha_{\min}}\left(R\max_{\lambda\in I}\left|\frac{\mathrm{d}\log\sigma_\lambda}{\mathrm{d}\lambda}\right|+\alpha_{\max}L_\mu\right)+L_t=:\widetilde{L}_t, \tag*{(186)}
\end{aligned} d λ d μ θ ( x t ( λ ) , t ( λ )) ≤ ∇ x μ θ ( x t ( λ ) , t ( λ )) d λ d x t ( λ ) + ∂ t ∂ μ θ ( x t ( λ ) , t ( λ )) d λ d t ( λ ) ≤ α t ( λ ) L x d λ d log σ λ x t ( λ ) + α λ μ θ ( x t ( λ ) , t ( λ )) + L t ≤ α m i n L x ( R λ ∈ I max d λ d log σ λ + α m a x L μ ) + L t =: L t , (186)
where boundedness follows from , Assumption 1 (A1), and the fixed finite interval. We have also used
d x t ( λ ) d λ = σ λ d d λ x t ( λ ) σ λ + x t ( λ ) σ λ d σ λ d λ = α λ μ θ ( x t ( λ ) , t ( λ ) ) + d log σ λ d λ x t ( λ ) . (187) \frac{\mathrm{d}\mathbf{x}_{t(\lambda)}}{\mathrm{d}\lambda}=\sigma_\lambda\frac{\mathrm{d}}{\mathrm{d}\lambda}\frac{\mathbf{x}_{t(\lambda)}}{\sigma_\lambda}+\frac{\mathbf{x}_{t(\lambda)}}{\sigma_\lambda}\frac{\mathrm{d}\sigma_\lambda}{\mathrm{d}\lambda}
=\alpha_\lambda\boldsymbol{\mu}_\theta(\mathbf{x}_{t(\lambda)},t(\lambda))+\frac{\mathrm{d}\log\sigma_\lambda}{\mathrm{d}\lambda}\mathbf{x}_{t(\lambda)}. \tag*{(187)} d λ d x t ( λ ) = σ λ d λ d σ λ x t ( λ ) + σ λ x t ( λ ) d λ d σ λ = α λ μ θ ( x t ( λ ) , t ( λ )) + d λ d log σ λ x t ( λ ) . (187)
Similarly, the second derivative is bounded by
∥ d 2 d λ 2 μ θ ( x t ( λ ) , t ( λ ) ) ∥ ≤ ∥ ∇ x 2 μ θ ( x t ( λ ) , t ( λ ) ) ∥ ∥ d x t ( λ ) d λ ∥ 2 + 2 ∥ ∂ ∂ λ ∇ x μ θ ( x , t ( λ ) ) ∣ x = x t ( λ ) ∥ ∥ d x t ( λ ) d λ ∥ + ∥ ∇ x μ θ ( x t ( λ ) , t ( λ ) ) ∥ ∥ d 2 x t ( λ ) d λ 2 ∥ + ∥ ∂ 2 μ θ ∂ λ 2 ( x , t ( λ ) ) ∣ x = x t ( λ ) ∥ ≤ H x ∥ d x t ( λ ) d λ ∥ 2 + 2 H ∥ d x t ( λ ) d λ ∥ + L x α t ( λ ) ∥ d 2 x t ( λ ) d λ 2 ∥ + H t ≤ H x V 2 + 2 H V + L x α min ( R max λ ∈ I ∣ d 2 log σ λ d λ 2 ∣ + V max λ ∈ I ∣ d log σ λ d λ ∣ ) + L μ max λ ∈ I ∣ d α λ d λ ∣ + α max L ~ t + H t = : H ~ t , \begin{aligned}
\left\lVert\frac{\mathrm{d}^2}{\mathrm{d}\lambda^2}\boldsymbol{\mu}_\theta(\mathbf{x}_{t(\lambda)},t(\lambda))\right\rVert
&\le\left\lVert\nabla_{\mathbf{x}}^2\boldsymbol{\mu}_\theta(\mathbf{x}_{t(\lambda)},t(\lambda))\right\rVert\left\lVert\frac{\mathrm{d}\mathbf{x}_{t(\lambda)}}{\mathrm{d}\lambda}\right\rVert^2+2\left\lVert\left.\frac{\partial}{\partial\lambda}\nabla_{\mathbf{x}}\boldsymbol{\mu}_\theta(\mathbf{x},t(\lambda))\right|_{\mathbf{x}=\mathbf{x}_{t(\lambda)}}\right\rVert\left\lVert\frac{\mathrm{d}\mathbf{x}_{t(\lambda)}}{\mathrm{d}\lambda}\right\rVert\\
&\quad+\left\lVert\nabla_{\mathbf{x}}\boldsymbol{\mu}_\theta(\mathbf{x}_{t(\lambda)},t(\lambda))\right\rVert\left\lVert\frac{\mathrm{d}^2\mathbf{x}_{t(\lambda)}}{\mathrm{d}\lambda^2}\right\rVert+\left\lVert\left.\frac{\partial^2\boldsymbol{\mu}_\theta}{\partial\lambda^2}(\mathbf{x},t(\lambda))\right|_{\mathbf{x}=\mathbf{x}_{t(\lambda)}}\right\rVert\\
&\le H_x\left\lVert\frac{\mathrm{d}\mathbf{x}_{t(\lambda)}}{\mathrm{d}\lambda}\right\rVert^2+2H\left\lVert\frac{\mathrm{d}\mathbf{x}_{t(\lambda)}}{\mathrm{d}\lambda}\right\rVert+\frac{L_x}{\alpha_{t(\lambda)}}\left\lVert\frac{\mathrm{d}^2\mathbf{x}_{t(\lambda)}}{\mathrm{d}\lambda^2}\right\rVert+H_t\\
&\le H_xV^2+2HV\\
&\quad+\frac{L_x}{\alpha_{\min}}\left(R\max_{\lambda\in I}\left|\frac{\mathrm{d}^2\log\sigma_\lambda}{\mathrm{d}\lambda^2}\right|+V\max_{\lambda\in I}\left|\frac{\mathrm{d}\log\sigma_\lambda}{\mathrm{d}\lambda}\right|\right)\\
&\quad+L_\mu\max_{\lambda\in I}\left|\frac{\mathrm{d}\alpha_\lambda}{\mathrm{d}\lambda}\right|+\alpha_{\max}\widetilde{L}_t+H_t=:\widetilde{H}_t,
\end{aligned} d λ 2 d 2 μ θ ( x t ( λ ) , t ( λ )) ≤ ∇ x 2 μ θ ( x t ( λ ) , t ( λ )) d λ d x t ( λ ) 2 + 2 ∂ λ ∂ ∇ x μ θ ( x , t ( λ )) x = x t ( λ ) d λ d x t ( λ ) + ∇ x μ θ ( x t ( λ ) , t ( λ )) d λ 2 d 2 x t ( λ ) + ∂ λ 2 ∂ 2 μ θ ( x , t ( λ )) x = x t ( λ ) ≤ H x d λ d x t ( λ ) 2 + 2 H d λ d x t ( λ ) + α t ( λ ) L x d λ 2 d 2 x t ( λ ) + H t ≤ H x V 2 + 2 H V + α m i n L x ( R λ ∈ I max d λ 2 d 2 log σ λ + V λ ∈ I max d λ d log σ λ ) + L μ λ ∈ I max d λ d α λ + α m a x L t + H t =: H t ,
where we have used (187) and the fact that
d 2 x t ( λ ) d λ 2 = d d λ ( d log σ λ d λ x t ( λ ) + α λ μ θ ( x t ( λ ) , t ( λ ) ) ) = d 2 log σ λ d λ 2 x t ( λ ) + d log σ λ d λ d x t ( λ ) d λ + d α λ d λ μ θ ( x t ( λ ) , t ( λ ) ) + α λ d d λ μ θ ( x t ( λ ) , t ( λ ) ) . \begin{aligned}
\frac{\mathrm{d}^2\mathbf{x}_{t(\lambda)}}{\mathrm{d}\lambda^2}
&=\frac{\mathrm{d}}{\mathrm{d}\lambda}\left(\frac{\mathrm{d}\log\sigma_\lambda}{\mathrm{d}\lambda}\mathbf{x}_{t(\lambda)}+\alpha_\lambda\boldsymbol{\mu}_\theta(\mathbf{x}_{t(\lambda)},t(\lambda))\right)\\
&=\frac{\mathrm{d}^2\log\sigma_\lambda}{\mathrm{d}\lambda^2}\mathbf{x}_{t(\lambda)}+\frac{\mathrm{d}\log\sigma_\lambda}{\mathrm{d}\lambda}\frac{\mathrm{d}\mathbf{x}_{t(\lambda)}}{\mathrm{d}\lambda}+\frac{\mathrm{d}\alpha_\lambda}{\mathrm{d}\lambda}\boldsymbol{\mu}_\theta(\mathbf{x}_{t(\lambda)},t(\lambda))+\alpha_\lambda\frac{\mathrm{d}}{\mathrm{d}\lambda}\boldsymbol{\mu}_\theta(\mathbf{x}_{t(\lambda)},t(\lambda)).
\end{aligned} d λ 2 d 2 x t ( λ ) = d λ d ( d λ d log σ λ x t ( λ ) + α λ μ θ ( x t ( λ ) , t ( λ )) ) = d λ 2 d 2 log σ λ x t ( λ ) + d λ d log σ λ d λ d x t ( λ ) + d λ d α λ μ θ ( x t ( λ ) , t ( λ )) + α λ d λ d μ θ ( x t ( λ ) , t ( λ )) .
Proof of Theorem 6
Proof for the first-order forward-value discretization scheme. By the diffusion ODE with the data prediction model (see (184) ), for each i ∈ [ M ] i \in[M] i ∈ [ M ] , we can express x t i ⋆ x_{t_i}^{\star} x t i ⋆ in terms of x t i − 1 ⋆ x_{t_{i-1}}^{\star} x t i − 1 ⋆ as
x t i ⋆ = σ t i σ t i − 1 x t i − 1 ⋆ + σ t i ∫ λ t i − 1 λ t i e λ μ θ ( x t ( λ ) ⋆ , t ( λ ) ) d λ . (188) x_{t_i}^{\star} = \frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}x_{t_{i-1}}^{\star}+\sigma_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{\lambda}\mu_{\theta}(x_{t(\lambda)}^{\star},t(\lambda))\,\mathrm{d}\lambda. \tag*{(188)} x t i ⋆ = σ t i − 1 σ t i x t i − 1 ⋆ + σ t i ∫ λ t i − 1 λ t i e λ μ θ ( x t ( λ ) ⋆ , t ( λ )) d λ . (188)
Combining this identity with the forward-value discretization (178) , we can decompose the distance between the iterates x t i f o r x_{t_i}^{\mathrm{for}} x t i for and x t i ⋆ x_{t_i}^{\star} x t i ⋆ as
x t i f o r − x t i ⋆ = σ t i σ t i − 1 ( x t i − 1 f o r − x t i − 1 ⋆ ) + σ t i ∫ λ t i − 1 λ t i e λ ( μ θ ( x t i f o r , t i ) − μ θ ( x t ( λ ) ⋆ , t ( λ ) ) ) d λ = σ t i σ t i − 1 ( x t i − 1 f o r − x t i − 1 ⋆ ) + σ t i ∫ λ t i − 1 λ t i e λ d λ ⋅ ( μ θ ( x t i f o r , t i ) − μ θ ( x t i ⋆ , t i ) ) + σ t i ∫ λ t i − 1 λ t i e λ ( μ θ ( x t i ⋆ , t i ) − μ θ ( x t ( λ ) ⋆ , t ( λ ) ) ) d λ . (189) \begin{aligned}
x_{t_i}^{\mathrm{for}}-x_{t_i}^{\star}
&=\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}\left(x_{t_{i-1}}^{\mathrm{for}}-x_{t_{i-1}}^{\star}\right)+\sigma_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{\lambda}\left(\mu_{\theta}(x_{t_i}^{\mathrm{for}},t_i)-\mu_{\theta}(x_{t(\lambda)}^{\star},t(\lambda))\right)\,\mathrm{d}\lambda\\
&=\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}\left(x_{t_{i-1}}^{\mathrm{for}}-x_{t_{i-1}}^{\star}\right)+\sigma_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{\lambda}\,\mathrm{d}\lambda\cdot\left(\mu_{\theta}(x_{t_i}^{\mathrm{for}},t_i)-\mu_{\theta}(x_{t_i}^{\star},t_i)\right)\\
&\quad+\sigma_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{\lambda}\left(\mu_{\theta}(x_{t_i}^{\star},t_i)-\mu_{\theta}(x_{t(\lambda)}^{\star},t(\lambda))\right)\,\mathrm{d}\lambda. \tag*{(189)}
\end{aligned} x t i for − x t i ⋆ = σ t i − 1 σ t i ( x t i − 1 for − x t i − 1 ⋆ ) + σ t i ∫ λ t i − 1 λ t i e λ ( μ θ ( x t i for , t i ) − μ θ ( x t ( λ ) ⋆ , t ( λ )) ) d λ = σ t i − 1 σ t i ( x t i − 1 for − x t i − 1 ⋆ ) + σ t i ∫ λ t i − 1 λ t i e λ d λ ⋅ ( μ θ ( x t i for , t i ) − μ θ ( x t i ⋆ , t i ) ) + σ t i ∫ λ t i − 1 λ t i e λ ( μ θ ( x t i ⋆ , t i ) − μ θ ( x t ( λ ) ⋆ , t ( λ )) ) d λ . (189)
Let us control the last two terms on the right-hand-side of (189) separately. For the second term, we can leverage the Lipschitz property of μ θ ( x , t ) \mu_{\theta}(x,t) μ θ ( x , t ) with respect to x x x (see Assumption 1 (A2)) to derive
∥ σ t i ∫ λ t i − 1 λ t i e λ d λ ⋅ ( μ θ ( x t i f o r , t i ) − μ θ ( x t i ⋆ , t i ) ) ∥ ≤ σ t i e λ t i δ i L x α t i ∥ x t i f o r − x t i ⋆ ∥ = δ i L x ∥ x t i f o r − x t i ⋆ ∥ , (190) \left\|\sigma_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{\lambda}\,\mathrm{d}\lambda\cdot\left(\mu_{\theta}(x_{t_i}^{\mathrm{for}},t_i)-\mu_{\theta}(x_{t_i}^{\star},t_i)\right)\right\|\leq\sigma_{t_i}e^{\lambda_{t_i}}\delta_i\frac{L_x}{\alpha_{t_i}}\left\|x_{t_i}^{\mathrm{for}}-x_{t_i}^{\star}\right\|
=\delta_iL_x\left\|x_{t_i}^{\mathrm{for}}-x_{t_i}^{\star}\right\|, \tag*{(190)} σ t i ∫ λ t i − 1 λ t i e λ d λ ⋅ ( μ θ ( x t i for , t i ) − μ θ ( x t i ⋆ , t i ) ) ≤ σ t i e λ t i δ i α t i L x x t i for − x t i ⋆ = δ i L x x t i for − x t i ⋆ , (190)
where we used λ t i − 1 ≤ λ t i \lambda_{t_{i-1}}\leq\lambda_{t_i} λ t i − 1 ≤ λ t i and λ t = log ( α t / σ t ) \lambda_t=\log(\alpha_t/\sigma_t) λ t = log ( α t / σ t ) . Similarly, we can control the third term by the total-derivative bound L ~ t \widetilde{L}_t L t along the exact ODE trajectory:
∥ σ t i ∫ λ t i − 1 λ t i e λ ( μ θ ( x t i ⋆ , t i ) − μ θ ( x t ( λ ) ⋆ , t ( λ ) ) ) d λ ∥ ≤ σ t i e λ t i ∫ λ t i − 1 λ t i L ~ t ( λ t i − λ ) d λ = 1 2 σ t i e λ t i L ~ t δ i 2 = 1 2 α t i L ~ t δ i 2 = O ( M − 2 ) , (191) \begin{aligned}
\left\|\sigma_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{\lambda}\left(\mu_{\theta}(x_{t_i}^{\star},t_i)-\mu_{\theta}(x_{t(\lambda)}^{\star},t(\lambda))\right)\,\mathrm{d}\lambda\right\|
&\leq\sigma_{t_i}e^{\lambda_{t_i}}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\widetilde{L}_t(\lambda_{t_i}-\lambda)\,\mathrm{d}\lambda=\frac{1}{2}\sigma_{t_i}e^{\lambda_{t_i}}\widetilde{L}_t\delta_i^2\\
&=\frac{1}{2}\alpha_{t_i}\widetilde{L}_t\delta_i^2=O(M^{-2}), \tag*{(191)}
\end{aligned} σ t i ∫ λ t i − 1 λ t i e λ ( μ θ ( x t i ⋆ , t i ) − μ θ ( x t ( λ ) ⋆ , t ( λ )) ) d λ ≤ σ t i e λ t i ∫ λ t i − 1 λ t i L t ( λ t i − λ ) d λ = 2 1 σ t i e λ t i L t δ i 2 = 2 1 α t i L t δ i 2 = O ( M − 2 ) , (191)
where the last step arises from the condition that δ i = O ( M − 1 ) \delta_i=O(M^{-1}) δ i = O ( M − 1 ) . Substituting (190) and (191) into (189) , we find that for each i ∈ [ M ] i \in[M] i ∈ [ M ] ,
∥ x t i f o r − x t i ⋆ ∥ ≤ σ t i σ t i − 1 ∥ x t i − 1 f o r − x t i − 1 ⋆ ∥ + L x δ i ∥ x t i f o r − x t i ⋆ ∥ + O ( M − 2 ) , (192) \left\|x_{t_i}^{\mathrm{for}}-x_{t_i}^{\star}\right\|\leq\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}\left\|x_{t_{i-1}}^{\mathrm{for}}-x_{t_{i-1}}^{\star}\right\|+L_x\delta_i\left\|x_{t_i}^{\mathrm{for}}-x_{t_i}^{\star}\right\|+O(M^{-2}), \tag*{(192)} x t i for − x t i ⋆ ≤ σ t i − 1 σ t i x t i − 1 for − x t i − 1 ⋆ + L x δ i x t i for − x t i ⋆ + O ( M − 2 ) , (192)
or equivalently,
∥ x t i f o r − x t i ⋆ ∥ ≤ σ t i σ t i − 1 ( 1 − L x δ i ) ∥ x t i − 1 f o r − x t i − 1 ⋆ ∥ + O ( M − 2 ) , (193) \left\|x_{t_i}^{\mathrm{for}}-x_{t_i}^{\star}\right\|\leq\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}(1-L_x\delta_i)}\left\|x_{t_{i-1}}^{\mathrm{for}}-x_{t_{i-1}}^{\star}\right\|+O(M^{-2}), \tag*{(193)} x t i for − x t i ⋆ ≤ σ t i − 1 ( 1 − L x δ i ) σ t i x t i − 1 for − x t i − 1 ⋆ + O ( M − 2 ) , (193)
where the last step arises from the fact that 1 − L x δ i = 1 − O ( 1 / M ) ≍ 1 1-L_x\delta_i=1-O(1/M)\asymp1 1 − L x δ i = 1 − O ( 1/ M ) ≍ 1 .
Applying (193) recursively allows us to bound the final error as
∥ x t M f o r − x t M ⋆ ∥ ≤ O ( M − 2 ) ∑ i = 1 M ∏ j = i + 1 M σ t j σ t j − 1 ( 1 − L x δ j ) = O ( M − 1 ) . (194) \left\|x_{t_M}^{\mathrm{for}}-x_{t_M}^{\star}\right\|\leq O(M^{-2})\sum_{i=1}^{M}\prod_{j=i+1}^{M}\frac{\sigma_{t_j}}{\sigma_{t_{j-1}}(1-L_x\delta_j)}=O(M^{-1}). \tag*{(194)} x t M for − x t M ⋆ ≤ O ( M − 2 ) i = 1 ∑ M j = i + 1 ∏ M σ t j − 1 ( 1 − L x δ j ) σ t j = O ( M − 1 ) . (194)
Here, the first step holds as x t 0 f o r = x t 0 ⋆ x_{t_0}^{\mathrm{for}}=x_{t_0}^{\star} x t 0 for = x t 0 ⋆ and the last step follows from
∑ i = 1 M ∏ j = i + 1 M σ t j σ t j − 1 ( 1 − L x δ j ) ≤ ∑ i = 1 M σ t M σ t i exp ( 2 L x ∑ j = i + 1 M δ j ) ≤ M exp ( 2 L x ( λ t M − λ t 0 ) ) = O ( M ) . \sum_{i=1}^{M}\prod_{j=i+1}^{M}\frac{\sigma_{t_j}}{\sigma_{t_{j-1}}(1-L_x\delta_j)}
\leq\sum_{i=1}^{M}\frac{\sigma_{t_M}}{\sigma_{t_i}}\exp\left(2L_x\sum_{j=i+1}^{M}\delta_j\right)
\leq M\exp\left(2L_x(\lambda_{t_M}-\lambda_{t_0})\right)=O(M). i = 1 ∑ M j = i + 1 ∏ M σ t j − 1 ( 1 − L x δ j ) σ t j ≤ i = 1 ∑ M σ t i σ t M exp ( 2 L x j = i + 1 ∑ M δ j ) ≤ M exp ( 2 L x ( λ t M − λ t 0 ) ) = O ( M ) .
We used 1 / ( 1 − x ) ≤ exp ( 2 x ) 1/(1-x)\leq\exp(2x) 1/ ( 1 − x ) ≤ exp ( 2 x ) for 0 ≤ x ≤ 1 / 2 0\leq x\leq1/2 0 ≤ x ≤ 1/2 , L x δ ≤ 1 / 2 L_x\delta\leq1/2 L x δ ≤ 1/2 for sufficiently large M M M , and σ t M ≤ σ t i \sigma_{t_M}\leq\sigma_{t_i} σ t M ≤ σ t i . The same argument gives max i ∈ [ M ] ∥ x t i f o r − x t i ⋆ ∥ = O ( M − 1 ) \max_{i\in[M]}\left\|x_{t_i}^{\mathrm{for}}-x_{t_i}^{\star}\right\|=O(M^{-1}) max i ∈ [ M ] x t i for − x t i ⋆ = O ( M − 1 ) . This justifies that the convergence order of the first-order, forward-value discretization scheme is equal to 1.
Proof of Claim (169). Let us fix an arbitrary i ∈ [ M ] i \in[M] i ∈ [ M ] . Combining the expression of x t i ⋆ x_{t_i}^{\star} x t i ⋆ from (188) with the update rules for the deterministic DDIM x t i b c k x_{t_i}^{\mathrm{bck}} x t i bck (see (14) ) and the first-order forward-value discretization x t i f o r x_{t_i}^{\mathrm{for}} x t i for (see (178) ), we can express the target difference as
x t i b c k − x t i ⋆ + x t i f o r − x t i ⋆ = σ t i σ t i − 1 ( x t i − 1 b c k + x t i − 1 f o r − 2 x t i − 1 ⋆ ) + σ t i ( ∫ λ t i − 1 λ t i e λ ( μ θ ( x t i − 1 b c k , t i − 1 ) + μ θ ( x t i f o r , t i ) − 2 μ θ ( x t ⋆ ( λ ) , t ( λ ) ) ) d λ ) = σ t i σ t i − 1 ( x t i − 1 ⋆ + x t i − 1 f o r − 2 x t i − 1 ⋆ ) + σ t i ( ∫ λ t i − 1 λ t i e λ ( μ θ ( x t i − 1 ⋆ , t i − 1 ) + μ θ ( x t i ⋆ , t i ) − 2 μ θ ( x t ⋆ ( λ ) , t ( λ ) ) ) d λ ) + σ t i ∫ λ t i − 1 λ t i e λ ( μ θ ( x t i − 1 b c k , t i − 1 ) + μ θ ( x t i − 1 f o r , t i − 1 ) − 2 μ θ ( x t i − 1 ⋆ , t i − 1 ) ) d λ + σ t i ∫ λ t i − 1 λ t i e λ ( μ θ ( x t i f o r , t i ) − μ θ ( x t i ⋆ , t i ) − μ θ ( x t i − 1 f o r , t i − 1 ) + μ θ ( x t i − 1 ⋆ , t i − 1 ) ) d λ . (195) \begin{aligned}
x_{t_i}^{\mathrm{bck}}-x_{t_i}^{\star}+x_{t_i}^{\mathrm{for}}-x_{t_i}^{\star}
={}&\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}\left(x_{t_{i-1}}^{\mathrm{bck}}+x_{t_{i-1}}^{\mathrm{for}}-2x_{t_{i-1}}^{\star}\right)\\
&+\sigma_{t_i}\left(\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{\lambda}\left(\mu_{\theta}\left(x_{t_{i-1}}^{\mathrm{bck}},t_{i-1}\right)+\mu_{\theta}\left(x_{t_i}^{\mathrm{for}},t_i\right)-2\mu_{\theta}\left(x_t^{\star}(\lambda),t(\lambda)\right)\right)\,d\lambda\right)\\
={}&\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}\left(x_{t_{i-1}}^{\star}+x_{t_{i-1}}^{\mathrm{for}}-2x_{t_{i-1}}^{\star}\right)\\
&+\sigma_{t_i}\left(\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{\lambda}\left(\mu_{\theta}\left(x_{t_{i-1}}^{\star},t_{i-1}\right)+\mu_{\theta}\left(x_{t_i}^{\star},t_i\right)-2\mu_{\theta}\left(x_t^{\star}(\lambda),t(\lambda)\right)\right)\,d\lambda\right)\\
&+\sigma_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{\lambda}\left(\mu_{\theta}\left(x_{t_{i-1}}^{\mathrm{bck}},t_{i-1}\right)+\mu_{\theta}\left(x_{t_{i-1}}^{\mathrm{for}},t_{i-1}\right)-2\mu_{\theta}\left(x_{t_{i-1}}^{\star},t_{i-1}\right)\right)\,d\lambda\\
&+\sigma_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{\lambda}\left(\mu_{\theta}\left(x_{t_i}^{\mathrm{for}},t_i\right)-\mu_{\theta}\left(x_{t_i}^{\star},t_i\right)-\mu_{\theta}\left(x_{t_{i-1}}^{\mathrm{for}},t_{i-1}\right)+\mu_{\theta}\left(x_{t_{i-1}}^{\star},t_{i-1}\right)\right)\,d\lambda. \tag*{(195)}
\end{aligned} x t i bck − x t i ⋆ + x t i for − x t i ⋆ = = σ t i − 1 σ t i ( x t i − 1 bck + x t i − 1 for − 2 x t i − 1 ⋆ ) + σ t i ( ∫ λ t i − 1 λ t i e λ ( μ θ ( x t i − 1 bck , t i − 1 ) + μ θ ( x t i for , t i ) − 2 μ θ ( x t ⋆ ( λ ) , t ( λ ) ) ) d λ ) σ t i − 1 σ t i ( x t i − 1 ⋆ + x t i − 1 for − 2 x t i − 1 ⋆ ) + σ t i ( ∫ λ t i − 1 λ t i e λ ( μ θ ( x t i − 1 ⋆ , t i − 1 ) + μ θ ( x t i ⋆ , t i ) − 2 μ θ ( x t ⋆ ( λ ) , t ( λ ) ) ) d λ ) + σ t i ∫ λ t i − 1 λ t i e λ ( μ θ ( x t i − 1 bck , t i − 1 ) + μ θ ( x t i − 1 for , t i − 1 ) − 2 μ θ ( x t i − 1 ⋆ , t i − 1 ) ) d λ + σ t i ∫ λ t i − 1 λ t i e λ ( μ θ ( x t i for , t i ) − μ θ ( x t i ⋆ , t i ) − μ θ ( x t i − 1 for , t i − 1 ) + μ θ ( x t i − 1 ⋆ , t i − 1 ) ) d λ . (195)
In what follows, we shall analyze χ 1 \chi_1 χ 1 , χ 2 \chi_2 χ 2 , and χ 3 \chi_3 χ 3 separately.
Controlling χ 1 \chi_1 χ 1 . For simplicity of notation, let us denote μ ( λ ) : = μ θ ( x t ⋆ ( λ ) , t ( λ ) ) \mu(\lambda):=\mu_{\theta}(x_t^{\star}(\lambda),t(\lambda)) μ ( λ ) := μ θ ( x t ⋆ ( λ ) , t ( λ )) . We begin with decomposing χ 1 \chi_1 χ 1 as
χ 1 = e λ t i ∫ λ t i − 1 λ t i ( μ θ ( x t i − 1 ⋆ , t i − 1 ) + μ θ ( x t i ⋆ , t i ) − 2 μ θ ( x t ⋆ ( λ ) , t ( λ ) ) ) d λ + ∫ λ t i − 1 λ t i ( e λ − e λ t i ) ( μ θ ( x t i − 1 ⋆ , t i − 1 ) + μ θ ( x t i ⋆ , t i ) − 2 μ θ ( x t ⋆ ( λ ) , t ( λ ) ) ) d λ . (196) \begin{aligned}
\chi_1={}&e^{\lambda_{t_i}}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\left(\mu_{\theta}\left(x_{t_{i-1}}^{\star},t_{i-1}\right)+\mu_{\theta}\left(x_{t_i}^{\star},t_i\right)-2\mu_{\theta}\left(x_t^{\star}(\lambda),t(\lambda)\right)\right)\,d\lambda\\
&+\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\left(e^{\lambda}-e^{\lambda_{t_i}}\right)\left(\mu_{\theta}\left(x_{t_{i-1}}^{\star},t_{i-1}\right)+\mu_{\theta}\left(x_{t_i}^{\star},t_i\right)-2\mu_{\theta}\left(x_t^{\star}(\lambda),t(\lambda)\right)\right)\,d\lambda. \tag*{(196)}
\end{aligned} χ 1 = e λ t i ∫ λ t i − 1 λ t i ( μ θ ( x t i − 1 ⋆ , t i − 1 ) + μ θ ( x t i ⋆ , t i ) − 2 μ θ ( x t ⋆ ( λ ) , t ( λ ) ) ) d λ + ∫ λ t i − 1 λ t i ( e λ − e λ t i ) ( μ θ ( x t i − 1 ⋆ , t i − 1 ) + μ θ ( x t i ⋆ , t i ) − 2 μ θ ( x t ⋆ ( λ ) , t ( λ ) ) ) d λ . (196)
and control these two terms individually.
∫ λ t i − 1 λ t i ( μ θ ( x t i − 1 ⋆ , t i − 1 ) + μ θ ( x t i ⋆ , t i ) − 2 μ θ ( x t ⋆ ( λ ) , t ( λ ) ) ) d λ = ∫ λ t i − 1 λ t i ( μ ( λ t i − 1 ) + μ ( λ t i ) − 2 μ ( λ ) ) d λ = 2 ∫ λ t i − 1 λ t i ( μ ( λ t i ) − μ ( λ t i − 1 ) λ t i − λ t i − 1 ( λ − λ t i − 1 ) + μ ( λ t i − 1 ) − μ ( λ ) ) d λ + ∫ λ t i − 1 λ t i ( μ ( λ t i ) − μ ( λ t i − 1 ) − 2 ( μ ( λ t i ) − μ ( λ t i − 1 ) ) λ t i − λ t i − 1 ( λ − λ t i − 1 ) ) d λ = 2 ∫ λ t i − 1 λ t i ( μ ( λ t i ) − μ ( λ t i − 1 ) λ t i − λ t i − 1 ( λ − λ t i − 1 ) + μ ( λ t i − 1 ) − μ ( λ ) ) d λ . (197) \begin{aligned}
&\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\left(\mu_{\theta}\left(x_{t_{i-1}}^{\star},t_{i-1}\right)+\mu_{\theta}\left(x_{t_i}^{\star},t_i\right)-2\mu_{\theta}\left(x_t^{\star}(\lambda),t(\lambda)\right)\right)\,d\lambda\\
={}&\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\left(\mu(\lambda_{t_{i-1}})+\mu(\lambda_{t_i})-2\mu(\lambda)\right)\,d\lambda\\
={}&2\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\left(\frac{\mu(\lambda_{t_i})-\mu(\lambda_{t_{i-1}})}{\lambda_{t_i}-\lambda_{t_{i-1}}}(\lambda-\lambda_{t_{i-1}})+\mu(\lambda_{t_{i-1}})-\mu(\lambda)\right)\,d\lambda\\
&+\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\left(\mu(\lambda_{t_i})-\mu(\lambda_{t_{i-1}})-\frac{2\left(\mu(\lambda_{t_i})-\mu(\lambda_{t_{i-1}})\right)}{\lambda_{t_i}-\lambda_{t_{i-1}}}(\lambda-\lambda_{t_{i-1}})\right)\,d\lambda\\
={}&2\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\left(\frac{\mu(\lambda_{t_i})-\mu(\lambda_{t_{i-1}})}{\lambda_{t_i}-\lambda_{t_{i-1}}}(\lambda-\lambda_{t_{i-1}})+\mu(\lambda_{t_{i-1}})-\mu(\lambda)\right)\,d\lambda. \tag*{(197)}
\end{aligned} = = = ∫ λ t i − 1 λ t i ( μ θ ( x t i − 1 ⋆ , t i − 1 ) + μ θ ( x t i ⋆ , t i ) − 2 μ θ ( x t ⋆ ( λ ) , t ( λ ) ) ) d λ ∫ λ t i − 1 λ t i ( μ ( λ t i − 1 ) + μ ( λ t i ) − 2 μ ( λ ) ) d λ 2 ∫ λ t i − 1 λ t i ( λ t i − λ t i − 1 μ ( λ t i ) − μ ( λ t i − 1 ) ( λ − λ t i − 1 ) + μ ( λ t i − 1 ) − μ ( λ ) ) d λ + ∫ λ t i − 1 λ t i ( μ ( λ t i ) − μ ( λ t i − 1 ) − λ t i − λ t i − 1 2 ( μ ( λ t i ) − μ ( λ t i − 1 ) ) ( λ − λ t i − 1 ) ) d λ 2 ∫ λ t i − 1 λ t i ( λ t i − λ t i − 1 μ ( λ t i ) − μ ( λ t i − 1 ) ( λ − λ t i − 1 ) + μ ( λ t i − 1 ) − μ ( λ ) ) d λ . (197)
where the last identity holds because
∫ λ t i − 1 λ t i ( μ ( λ t i ) − μ ( λ t i − 1 ) − 2 ( μ ( λ t i ) − μ ( λ t i − 1 ) ) λ t i − λ t i − 1 ( λ − λ t i − 1 ) ) d λ = ( μ ( λ t i ) − μ ( λ t i − 1 ) ) δ i − 2 ( μ ( λ t i ) − μ ( λ t i − 1 ) ) λ t i − λ t i − 1 ∫ λ t i − 1 λ t i ( λ − λ t i − 1 ) d λ = ( μ ( λ t i ) − μ ( λ t i − 1 ) ) δ i − 2 ( μ ( λ t i ) − μ ( λ t i − 1 ) ) δ i ⋅ 1 2 δ i 2 = 0. \begin{aligned}
&\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\left(\mu(\lambda_{t_i})-\mu(\lambda_{t_{i-1}})-\frac{2\left(\mu(\lambda_{t_i})-\mu(\lambda_{t_{i-1}})\right)}{\lambda_{t_i}-\lambda_{t_{i-1}}}(\lambda-\lambda_{t_{i-1}})\right)\,d\lambda\\
={}&\left(\mu(\lambda_{t_i})-\mu(\lambda_{t_{i-1}})\right)\delta_i-\frac{2\left(\mu(\lambda_{t_i})-\mu(\lambda_{t_{i-1}})\right)}{\lambda_{t_i}-\lambda_{t_{i-1}}}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}(\lambda-\lambda_{t_{i-1}})\,d\lambda\\
&= \left(\mu(\lambda_{t_i})-\mu(\lambda_{t_{i-1}})\right)\delta_i-\frac{2\left(\mu(\lambda_{t_i})-\mu(\lambda_{t_{i-1}})\right)}{\delta_i}\cdot\frac{1}{2}\delta_i^2=0.
\end{aligned} = ∫ λ t i − 1 λ t i ( μ ( λ t i ) − μ ( λ t i − 1 ) − λ t i − λ t i − 1 2 ( μ ( λ t i ) − μ ( λ t i − 1 ) ) ( λ − λ t i − 1 ) ) d λ ( μ ( λ t i ) − μ ( λ t i − 1 ) ) δ i − λ t i − λ t i − 1 2 ( μ ( λ t i ) − μ ( λ t i − 1 ) ) ∫ λ t i − 1 λ t i ( λ − λ t i − 1 ) d λ = ( μ ( λ t i ) − μ ( λ t i − 1 ) ) δ i − δ i 2 ( μ ( λ t i ) − μ ( λ t i − 1 ) ) ⋅ 2 1 δ i 2 = 0.
Given the expression in (197) , combining Taylor’s theorem with the total second-derivative bound H ~ t \widetilde{H}_t H t , we can derive
∥ ∫ λ t i − 1 λ t i ( μ ( λ t i ) − μ ( λ t i − 1 ) λ t i − λ t i − 1 ( λ − λ t i − 1 ) + μ ( λ t i − 1 ) − μ ( λ ) ) d λ ∥ ≤ ∥ ∫ λ t i − 1 λ t i ( μ ( λ t i ) − μ ( λ t i − 1 ) λ t i − λ t i − 1 ( λ − λ t i − 1 ) − μ ′ ( λ t i − 1 ) ( λ − λ t i − 1 ) ) d λ ∥ + H ~ t ∫ λ t i − 1 λ t i ( λ − λ t i − 1 ) 2 d λ ≤ ∥ μ ( λ t i ) − μ ( λ t i − 1 ) λ t i − λ t i − 1 − μ ′ ( λ t i − 1 ) ∥ ∫ λ t i − 1 λ t i ( λ − λ t i − 1 ) d λ + H ~ t 3 δ i 3 . (198) \begin{aligned}
\left\|\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\left(\frac{\mu(\lambda_{t_i})-\mu(\lambda_{t_{i-1}})}{\lambda_{t_i}-\lambda_{t_{i-1}}}(\lambda-\lambda_{t_{i-1}})+\mu(\lambda_{t_{i-1}})-\mu(\lambda)\right)\mathrm{d}\lambda\right\| \\
&\leq\left\|\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\left(\frac{\mu(\lambda_{t_i})-\mu(\lambda_{t_{i-1}})}{\lambda_{t_i}-\lambda_{t_{i-1}}}(\lambda-\lambda_{t_{i-1}})-\mu'(\lambda_{t_{i-1}})(\lambda-\lambda_{t_{i-1}})\right)\mathrm{d}\lambda\right\|
+\widetilde{H}_t\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}(\lambda-\lambda_{t_{i-1}})^2\mathrm{d}\lambda\\
&\leq\left\|\frac{\mu(\lambda_{t_i})-\mu(\lambda_{t_{i-1}})}{\lambda_{t_i}-\lambda_{t_{i-1}}}-\mu'(\lambda_{t_{i-1}})\right\|\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}(\lambda-\lambda_{t_{i-1}})\mathrm{d}\lambda+\frac{\widetilde{H}_t}{3}\delta_i^3. \tag*{(198)}
\end{aligned} ∫ λ t i − 1 λ t i ( λ t i − λ t i − 1 μ ( λ t i ) − μ ( λ t i − 1 ) ( λ − λ t i − 1 ) + μ ( λ t i − 1 ) − μ ( λ ) ) d λ ≤ ∫ λ t i − 1 λ t i ( λ t i − λ t i − 1 μ ( λ t i ) − μ ( λ t i − 1 ) ( λ − λ t i − 1 ) − μ ′ ( λ t i − 1 ) ( λ − λ t i − 1 ) ) d λ + H t ∫ λ t i − 1 λ t i ( λ − λ t i − 1 ) 2 d λ ≤ λ t i − λ t i − 1 μ ( λ t i ) − μ ( λ t i − 1 ) − μ ′ ( λ t i − 1 ) ∫ λ t i − 1 λ t i ( λ − λ t i − 1 ) d λ + 3 H t δ i 3 . (198)
Applying the same total second-derivative bound again, we can further bound the first term on the right-hand side of (198) as
∥ μ ( λ t i ) − μ ( λ t i − 1 ) λ t i − λ t i − 1 − μ ′ ( λ t i − 1 ) ∥ ≤ ∥ μ ′ ( λ t i − 1 ) λ t i − λ t i − 1 λ t i − λ t i − 1 − μ ′ ( λ t i − 1 ) ∥ + H ~ t δ i = H ~ t δ i . \left\|\frac{\mu(\lambda_{t_i})-\mu(\lambda_{t_{i-1}})}{\lambda_{t_i}-\lambda_{t_{i-1}}}-\mu'(\lambda_{t_{i-1}})\right\|
\leq\left\|\mu'(\lambda_{t_{i-1}})\frac{\lambda_{t_i}-\lambda_{t_{i-1}}}{\lambda_{t_i}-\lambda_{t_{i-1}}}-\mu'(\lambda_{t_{i-1}})\right\|+\widetilde{H}_t\delta_i=\widetilde{H}_t\delta_i. λ t i − λ t i − 1 μ ( λ t i ) − μ ( λ t i − 1 ) − μ ′ ( λ t i − 1 ) ≤ μ ′ ( λ t i − 1 ) λ t i − λ t i − 1 λ t i − λ t i − 1 − μ ′ ( λ t i − 1 ) + H t δ i = H t δ i .
Substituted into (198) , this yields
∥ ∫ λ t i − 1 λ t i ( μ ( λ t i ) − μ ( λ t i − 1 ) λ t i − λ t i − 1 ( λ − λ t i − 1 ) + μ ( λ t i − 1 ) − μ ( λ ) ) d λ ∥ ≤ 1 2 H ~ t δ i 3 + 1 3 H ~ t δ i 3 = O ( M − 3 ) . (199) \left\|\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\left(\frac{\mu(\lambda_{t_i})-\mu(\lambda_{t_{i-1}})}{\lambda_{t_i}-\lambda_{t_{i-1}}}(\lambda-\lambda_{t_{i-1}})+\mu(\lambda_{t_{i-1}})-\mu(\lambda)\right)\mathrm{d}\lambda\right\|
\leq\frac{1}{2}\widetilde{H}_t\delta_i^3+\frac{1}{3}\widetilde{H}_t\delta_i^3=O(M^{-3}). \tag*{(199)} ∫ λ t i − 1 λ t i ( λ t i − λ t i − 1 μ ( λ t i ) − μ ( λ t i − 1 ) ( λ − λ t i − 1 ) + μ ( λ t i − 1 ) − μ ( λ ) ) d λ ≤ 2 1 H t δ i 3 + 3 1 H t δ i 3 = O ( M − 3 ) . (199)
Finally, plugging (199) into (197) , the first term on the right-hand side of (196) can be bounded as
∫ λ t i − 1 λ t i ( μ θ ( x t i − 1 ⋆ , t i − 1 ) + μ θ ( x t i ⋆ , t i ) − 2 μ θ ( x t ( λ ) ⋆ , t ( λ ) ) ) d λ = O ( M − 3 ) . (200) \int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\left(\mu_\theta(x^\star_{t_{i-1}},t_{i-1})+\mu_\theta(x^\star_{t_i},t_i)-2\mu_\theta(x^\star_{t(\lambda)},t(\lambda))\right)\mathrm{d}\lambda=O(M^{-3}). \tag*{(200)} ∫ λ t i − 1 λ t i ( μ θ ( x t i − 1 ⋆ , t i − 1 ) + μ θ ( x t i ⋆ , t i ) − 2 μ θ ( x t ( λ ) ⋆ , t ( λ )) ) d λ = O ( M − 3 ) . (200)
– Turning to the second term, one can derive
∥ ∫ λ t i − 1 λ t i ( e λ − e λ t i ) ( μ θ ( x t i − 1 ⋆ , t i − 1 ) + μ θ ( x t i ⋆ , t i ) − 2 μ θ ( x t ( λ ) ⋆ , t ( λ ) ) ) d λ ∥ ≤ ∣ e λ t i − 1 − e λ t i ∣ ∫ λ t i − 1 λ t i ∥ μ θ ( x t i − 1 ⋆ , t i − 1 ) + μ θ ( x t i ⋆ , t i ) − 2 μ θ ( x t ( λ ) ⋆ , t ( λ ) ) ∥ d λ ≤ e λ t i δ i ∫ λ t i − 1 λ t i L ~ t ( λ − λ t i − 1 + λ t i − λ ) d λ = e λ t i L ~ t δ i 3 = O ( M − 3 ) . (201) \begin{aligned}
\left\|\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\left(\mathrm{e}^{\lambda}-\mathrm{e}^{\lambda_{t_i}}\right)\left(\mu_\theta(x^\star_{t_{i-1}},t_{i-1})+\mu_\theta(x^\star_{t_i},t_i)-2\mu_\theta(x^\star_{t(\lambda)},t(\lambda))\right)\mathrm{d}\lambda\right\| \\
&\leq\left|\mathrm{e}^{\lambda_{t_{i-1}}}-\mathrm{e}^{\lambda_{t_i}}\right|\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\left\|\mu_\theta(x^\star_{t_{i-1}},t_{i-1})+\mu_\theta(x^\star_{t_i},t_i)-2\mu_\theta(x^\star_{t(\lambda)},t(\lambda))\right\|\mathrm{d}\lambda\\
&\leq\mathrm{e}^{\lambda_{t_i}}\delta_i\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\widetilde{L}_t(\lambda-\lambda_{t_{i-1}}+\lambda_{t_i}-\lambda)\mathrm{d}\lambda\\
&=\mathrm{e}^{\lambda_{t_i}}\widetilde{L}_t\delta_i^3=O(M^{-3}). \tag*{(201)}
\end{aligned} ∫ λ t i − 1 λ t i ( e λ − e λ t i ) ( μ θ ( x t i − 1 ⋆ , t i − 1 ) + μ θ ( x t i ⋆ , t i ) − 2 μ θ ( x t ( λ ) ⋆ , t ( λ )) ) d λ ≤ e λ t i − 1 − e λ t i ∫ λ t i − 1 λ t i μ θ ( x t i − 1 ⋆ , t i − 1 ) + μ θ ( x t i ⋆ , t i ) − 2 μ θ ( x t ( λ ) ⋆ , t ( λ )) d λ ≤ e λ t i δ i ∫ λ t i − 1 λ t i L t ( λ − λ t i − 1 + λ t i − λ ) d λ = e λ t i L t δ i 3 = O ( M − 3 ) . (201)
where the second inequality uses the total-derivative bound L ~ t \widetilde{L}_t L t along the exact trajectory.
– Putting (200) and (201) together, we conclude that
∥ χ 1 ∥ = O ( M − 3 ) . (202) \|\chi_1\|=O(M^{-3}). \tag*{(202)} ∥ χ 1 ∥ = O ( M − 3 ) . (202)
• Controlling χ 2 \chi_2 χ 2 . From Taylor’s theorem and Assumption 1 (A3), we can bound
∥ μ θ ( x t i − 1 b c k , t i − 1 ) + μ θ ( x t i − 1 f o r , t i − 1 ) − 2 μ θ ( x t i − 1 ⋆ , t i − 1 ) − ∇ μ θ ( x t i − 1 ⋆ , t i − 1 ) ( x t i − 1 b c k + x t i − 1 f o r − 2 x t i − 1 ⋆ ) ∥ ≤ H x ( ∥ x t i − 1 b c k − x t i − 1 ⋆ ∥ 2 + ∥ x t i − 1 f o r − x t i − 1 ⋆ ∥ 2 ) = O ( M − 2 ) , \begin{aligned}
\left\|\mu_\theta(x^{\mathrm{bck}}_{t_{i-1}},t_{i-1})+\mu_\theta(x^{\mathrm{for}}_{t_{i-1}},t_{i-1})-2\mu_\theta(x^\star_{t_{i-1}},t_{i-1})-\nabla\mu_\theta(x^\star_{t_{i-1}},t_{i-1})\left(x^{\mathrm{bck}}_{t_{i-1}}+x^{\mathrm{for}}_{t_{i-1}}-2x^\star_{t_{i-1}}\right)\right\| \\
&\le H_x\left(\left\|x_{t_{i-1}}^{\mathrm{bck}}-x_{t_{i-1}}^\star\right\|^2+\left\|x_{t_{i-1}}^{\mathrm{for}}-x_{t_{i-1}}^\star\right\|^2\right)=O(M^{-2}),
\end{aligned} μ θ ( x t i − 1 bck , t i − 1 ) + μ θ ( x t i − 1 for , t i − 1 ) − 2 μ θ ( x t i − 1 ⋆ , t i − 1 ) − ∇ μ θ ( x t i − 1 ⋆ , t i − 1 ) ( x t i − 1 bck + x t i − 1 for − 2 x t i − 1 ⋆ ) ≤ H x ( x t i − 1 bck − x t i − 1 ⋆ 2 + x t i − 1 for − x t i − 1 ⋆ 2 ) = O ( M − 2 ) ,
where the last step follows from the first-order bounds for DDIM and the forward-value iterates. This allows us to bound χ 2 \chi_2 χ 2 as
∥ χ 2 ∥ ≤ ∫ λ t i − 1 λ t i e λ ( ∥ ∇ μ θ ( x t i − 1 ⋆ , t i − 1 ) ( x t i − 1 b c k + x t i − 1 f o r − 2 x t i − 1 ⋆ ) ∥ + O ( M − 2 ) ) d λ ≤ ( e λ t i − e λ t i − 1 ) ( L x α t i − 1 ∥ x t i − 1 b c k + x t i − 1 f o r − 2 x t i − 1 ⋆ ∥ + O ( M − 2 ) ) ≤ e λ t i δ i L x α t i − 1 ∥ x t i − 1 b c k + x t i − 1 f o r − 2 x t i − 1 ⋆ ∥ + e λ t i δ i O ( M − 2 ) = L x e δ i δ i σ t i − 1 ∥ x t i − 1 b c k + x t i − 1 f o r − 2 x t i − 1 ⋆ ∥ + O ( M − 3 ) , (203) \begin{aligned}
\left\|\chi_2\right\|\le\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^\lambda\left(\left\|\nabla\mu_\theta\left(x_{t_{i-1}}^\star,t_{i-1}\right)\left(x_{t_{i-1}}^{\mathrm{bck}}+x_{t_{i-1}}^{\mathrm{for}}-2x_{t_{i-1}}^\star\right)\right\|+O(M^{-2})\right)\mathrm{d}\lambda\\
&\le\left(e^{\lambda_{t_i}}-e^{\lambda_{t_{i-1}}}\right)\left(\frac{L_x}{\alpha_{t_{i-1}}}\left\|x_{t_{i-1}}^{\mathrm{bck}}+x_{t_{i-1}}^{\mathrm{for}}-2x_{t_{i-1}}^\star\right\|+O(M^{-2})\right) \\
&\le e^{\lambda_{t_i}}\delta_i\frac{L_x}{\alpha_{t_{i-1}}}\left\|x_{t_{i-1}}^{\mathrm{bck}}+x_{t_{i-1}}^{\mathrm{for}}-2x_{t_{i-1}}^\star\right\|+e^{\lambda_{t_i}}\delta_iO(M^{-2}) \\
&=\frac{L_xe^{\delta_i}\delta_i}{\sigma_{t_{i-1}}}\left\|x_{t_{i-1}}^{\mathrm{bck}}+x_{t_{i-1}}^{\mathrm{for}}-2x_{t_{i-1}}^\star\right\|+O(M^{-3}), \tag*{(203)}
\end{aligned} ∥ χ 2 ∥ ≤ ∫ λ t i − 1 λ t i e λ ( ∇ μ θ ( x t i − 1 ⋆ , t i − 1 ) ( x t i − 1 bck + x t i − 1 for − 2 x t i − 1 ⋆ ) + O ( M − 2 ) ) d λ ≤ ( e λ t i − e λ t i − 1 ) ( α t i − 1 L x x t i − 1 bck + x t i − 1 for − 2 x t i − 1 ⋆ + O ( M − 2 ) ) ≤ e λ t i δ i α t i − 1 L x x t i − 1 bck + x t i − 1 for − 2 x t i − 1 ⋆ + e λ t i δ i O ( M − 2 ) = σ t i − 1 L x e δ i δ i x t i − 1 bck + x t i − 1 for − 2 x t i − 1 ⋆ + O ( M − 3 ) , (203)
where the second line applies Assumption 1 (A2) and the last step holds as λ t = log ( α t / σ t ) \lambda_t=\log(\alpha_t/\sigma_t) λ t = log ( α t / σ t ) and δ i = O ( M − 1 ) \delta_i=O(M^{-1}) δ i = O ( M − 1 ) .
∥ μ θ ( x t i f o r , t i ) − μ θ ( x t i ⋆ , t i ) − μ θ ( x t i − 1 f o r , t i − 1 ) + μ θ ( x t i − 1 ⋆ , t i − 1 ) ∥ ≤ ∥ ∇ μ θ ( x t i ⋆ , t i ) ( x t i f o r − x t i ⋆ ) − ∇ μ θ ( x t i − 1 ⋆ , t i − 1 ) ( x t i f o r − x t i ⋆ ) ∥ \begin{aligned}
\left\|\mu_\theta\left(x_{t_i}^{\mathrm{for}},t_i\right)-\mu_\theta\left(x_{t_i}^\star,t_i\right)-\mu_\theta\left(x_{t_{i-1}}^{\mathrm{for}},t_{i-1}\right)+\mu_\theta\left(x_{t_{i-1}}^\star,t_{i-1}\right)\right\| \\
&\le\left\|\nabla\mu_\theta\left(x_{t_i}^\star,t_i\right)\left(x_{t_i}^{\mathrm{for}}-x_{t_i}^\star\right)-\nabla\mu_\theta\left(x_{t_{i-1}}^\star,t_{i-1}\right)\left(x_{t_i}^{\mathrm{for}}-x_{t_i}^\star\right)\right\|
\end{aligned} μ θ ( x t i for , t i ) − μ θ ( x t i ⋆ , t i ) − μ θ ( x t i − 1 for , t i − 1 ) + μ θ ( x t i − 1 ⋆ , t i − 1 ) ≤ ∇ μ θ ( x t i ⋆ , t i ) ( x t i for − x t i ⋆ ) − ∇ μ θ ( x t i − 1 ⋆ , t i − 1 ) ( x t i for − x t i ⋆ )
+ H x ∥ x t i f o r − x t i ⋆ ∥ 2 + H x ∥ x t i − 1 f o r − x t i − 1 ⋆ ∥ 2 ≤ ∥ ( ∇ μ θ ( x t i ⋆ , t i ) − ∇ μ θ ( x t i − 1 ⋆ , t i − 1 ) ) ( x t i f o r − x t i ⋆ ) ∥ \begin{aligned}
\quad+H_x\left\|x_{t_i}^{\mathrm{for}}-x_{t_i}^\star\right\|^2+H_x\left\|x_{t_{i-1}}^{\mathrm{for}}-x_{t_{i-1}}^\star\right\|^2 \\
&\le\left\|\left(\nabla\mu_\theta\left(x_{t_i}^\star,t_i\right)-\nabla\mu_\theta\left(x_{t_{i-1}}^\star,t_{i-1}\right)\right)\left(x_{t_i}^{\mathrm{for}}-x_{t_i}^\star\right)\right\|
\end{aligned} + H x x t i for − x t i ⋆ 2 + H x x t i − 1 for − x t i − 1 ⋆ 2 ≤ ( ∇ μ θ ( x t i ⋆ , t i ) − ∇ μ θ ( x t i − 1 ⋆ , t i − 1 ) ) ( x t i for − x t i ⋆ )
+ ∥ ∇ μ θ ( x t i − 1 ⋆ , t i − 1 ) ∥ ∥ x t i f o r − x t i ⋆ − x t i − 1 f o r + x t i − 1 ⋆ ∥ + O ( M − 2 ) , (204) \quad+\left\|\nabla\mu_\theta\left(x_{t_{i-1}}^\star,t_{i-1}\right)\right\|\left\|x_{t_i}^{\mathrm{for}}-x_{t_i}^\star-x_{t_{i-1}}^{\mathrm{for}}+x_{t_{i-1}}^\star\right\|+O(M^{-2}), \tag*{(204)} + ∇ μ θ ( x t i − 1 ⋆ , t i − 1 ) x t i for − x t i ⋆ − x t i − 1 for + x t i − 1 ⋆ + O ( M − 2 ) , (204)
where the last step applies (194) .
To control the first term on the right-hand-side of (204) , Assumption 1 (A3) and (A6), together with the bound V V V in (186) , give
∥ ∇ μ θ ( x t i ⋆ , t i ) − ∇ μ θ ( x t i − 1 ⋆ , t i − 1 ) ∥ ≤ H x ∥ x t i ⋆ − x t i − 1 ⋆ ∥ + H δ i ≤ ( H x V + H ) δ i = O ( M − 1 ) . (205) \left\|\nabla\mu_\theta\left(x_{t_i}^\star,t_i\right)-\nabla\mu_\theta\left(x_{t_{i-1}}^\star,t_{i-1}\right)\right\|\le H_x\left\|x_{t_i}^\star-x_{t_{i-1}}^\star\right\|+H\delta_i\le(H_xV+H)\delta_i=O(M^{-1}). \tag*{(205)} ∇ μ θ ( x t i ⋆ , t i ) − ∇ μ θ ( x t i − 1 ⋆ , t i − 1 ) ≤ H x x t i ⋆ − x t i − 1 ⋆ + H δ i ≤ ( H x V + H ) δ i = O ( M − 1 ) . (205)
Therefore the first term is bounded by
∥ ( ∇ μ θ ( x t i ⋆ , t i ) − ∇ μ θ ( x t i − 1 ⋆ , t i − 1 ) ) ( x t i f o r − x t i ⋆ ) ∥ ≤ ∥ ∇ μ θ ( x t i ⋆ , t i ) − ∇ μ θ ( x t i − 1 ⋆ , t i − 1 ) ∥ ∥ x t i f o r − x t i ⋆ ∥ = O ( M − 2 ) , (206) \begin{aligned}
\left\|\left(\nabla\mu_\theta\left(x_{t_i}^\star,t_i\right)-\nabla\mu_\theta\left(x_{t_{i-1}}^\star,t_{i-1}\right)\right)\left(x_{t_i}^{\mathrm{for}}-x_{t_i}^\star\right)\right\| \\
&\le\left\|\nabla\mu_\theta\left(x_{t_i}^\star,t_i\right)-\nabla\mu_\theta\left(x_{t_{i-1}}^\star,t_{i-1}\right)\right\|\left\|x_{t_i}^{\mathrm{for}}-x_{t_i}^\star\right\|=O(M^{-2}), \tag*{(206)}
\end{aligned} ( ∇ μ θ ( x t i ⋆ , t i ) − ∇ μ θ ( x t i − 1 ⋆ , t i − 1 ) ) ( x t i for − x t i ⋆ ) ≤ ∇ μ θ ( x t i ⋆ , t i ) − ∇ μ θ ( x t i − 1 ⋆ , t i − 1 ) x t i for − x t i ⋆ = O ( M − 2 ) , (206)
where the last equation applies (194) again.
In addition, putting collectively what we have shown in (189) , (190) , and (191) , one knows that
∥ x t i f o r − x t i ⋆ − x t i − 1 f o r + x t i − 1 ⋆ ∥ ≤ ∣ σ t i σ t i − 1 − 1 ∣ ∥ x t i − 1 f o r − x t i − 1 ⋆ ∥ + L x δ i ∥ x t i f o r − x t i ⋆ ∥ + O ( M − 2 ) . \left\|x_{t_i}^{\mathrm{for}}-x_{t_i}^\star-x_{t_{i-1}}^{\mathrm{for}}+x_{t_{i-1}}^\star\right\|\le\left|\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}-1\right|\left\|x_{t_{i-1}}^{\mathrm{for}}-x_{t_{i-1}}^\star\right\|+L_x\delta_i\left\|x_{t_i}^{\mathrm{for}}-x_{t_i}^\star\right\|+O(M^{-2}). x t i for − x t i ⋆ − x t i − 1 for + x t i − 1 ⋆ ≤ σ t i − 1 σ t i − 1 x t i − 1 for − x t i − 1 ⋆ + L x δ i x t i for − x t i ⋆ + O ( M − 2 ) .
Since σ t i ≤ σ t i − 1 \sigma_{t_i}\le\sigma_{t_{i-1}} σ t i ≤ σ t i − 1 , we have
∣ σ t i σ t i − 1 − 1 ∣ = 1 − σ t i σ t i − 1 = 1 − α t i α t i − 1 e − ( λ t i − λ t i − 1 ) ≤ 1 − e − δ i ≤ δ i , \left|\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}-1\right|=1-\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}=1-\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}e^{-(\lambda_{t_i}-\lambda_{t_{i-1}})}\le1-e^{-\delta_i}\le\delta_i, σ t i − 1 σ t i − 1 = 1 − σ t i − 1 σ t i = 1 − α t i − 1 α t i e − ( λ t i − λ t i − 1 ) ≤ 1 − e − δ i ≤ δ i ,
which together with (193) leads to
∥ x t i f o r − x t i ⋆ − x t i − 1 f o r + x t i − 1 ⋆ ∥ ≤ δ i ∥ x t i − 1 f o r − x t i ⋆ ∥ + L x δ i ∥ x t i f o r − x t i ⋆ ∥ + O ( M − 2 ) = O ( M − 2 ) . (207) \left\|x_{t_i}^{\mathrm{for}}-x_{t_i}^\star-x_{t_{i-1}}^{\mathrm{for}}+x_{t_{i-1}}^\star\right\|\le\delta_i\left\|x_{t_{i-1}}^{\mathrm{for}}-x_{t_i}^\star\right\|+L_x\delta_i\left\|x_{t_i}^{\mathrm{for}}-x_{t_i}^\star\right\|+O(M^{-2})=O(M^{-2}). \tag*{(207)} x t i for − x t i ⋆ − x t i − 1 for + x t i − 1 ⋆ ≤ δ i x t i − 1 for − x t i ⋆ + L x δ i x t i for − x t i ⋆ + O ( M − 2 ) = O ( M − 2 ) . (207)
Therefore the second term in the right-hand-side of (204) is bounded by
∥ ∇ μ θ ( x t i − 1 ⋆ , t i − 1 ) ∥ ∥ x t i f o r − x t i ⋆ − x t i − 1 f o r + x t i − 1 ⋆ ∥ ≤ L x α t i − 1 O ( M − 2 ) ≤ L x α min O ( M − 2 ) = O ( M − 2 ) . (208) \left\|\nabla\mu_\theta\left(x_{t_{i-1}}^\star,t_{i-1}\right)\right\|\left\|x_{t_i}^{\mathrm{for}}-x_{t_i}^\star-x_{t_{i-1}}^{\mathrm{for}}+x_{t_{i-1}}^\star\right\|\le\frac{L_x}{\alpha_{t_{i-1}}}O(M^{-2})\le\frac{L_x}{\alpha_{\min}}O(M^{-2})=O(M^{-2}). \tag*{(208)} ∇ μ θ ( x t i − 1 ⋆ , t i − 1 ) x t i for − x t i ⋆ − x t i − 1 for + x t i − 1 ⋆ ≤ α t i − 1 L x O ( M − 2 ) ≤ α m i n L x O ( M − 2 ) = O ( M − 2 ) . (208)
where we used α t i − 1 ≥ α min > 0 \alpha_{t_{i-1}} \ge\alpha_{\min} > 0 α t i − 1 ≥ α m i n > 0 .
Finally, plugging the above bound into the definition of χ 3 \chi_3 χ 3 , we conclude that
∥ χ 3 ∥ ≤ ∫ λ t i − 1 λ t i e λ d λ ⋅ ∥ μ θ ( x t i f o r , t i ) − μ θ ( x t i ⋆ , t i ) − μ θ ( x t i − 1 f o r , t i − 1 ) + μ θ ( x t i − 1 ⋆ , t i − 1 ) ∥ ≤ e λ t i δ i ∥ μ θ ( x t i f o r , t i ) − μ θ ( x t i ⋆ , t i ) − μ θ ( x t i − 1 f o r , t i − 1 ) + μ θ ( x t i − 1 ⋆ , t i − 1 ) ∥ = O ( M − 3 ) . (209) \begin{aligned}
\lVert\chi_3\rVert&\le\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}} e^\lambda\,\mathrm{d}\lambda\cdot\lVert\mu_\theta(x^{\mathrm{for}}_{t_i},t_i)-\mu_\theta(x^\star_{t_i},t_i)-\mu_\theta(x^{\mathrm{for}}_{t_{i-1}},t_{i-1})+\mu_\theta(x^\star_{t_{i-1}},t_{i-1})\rVert\\
&\le e^{\lambda_{t_i}}\delta_i\lVert\mu_\theta(x^{\mathrm{for}}_{t_i},t_i)-\mu_\theta(x^\star_{t_i},t_i)-\mu_\theta(x^{\mathrm{for}}_{t_{i-1}},t_{i-1})+\mu_\theta(x^\star_{t_{i-1}},t_{i-1})\rVert=O(M^{-3}). \tag*{(209)}
\end{aligned} ∥ χ 3 ∥ ≤ ∫ λ t i − 1 λ t i e λ d λ ⋅ ∥ μ θ ( x t i for , t i ) − μ θ ( x t i ⋆ , t i ) − μ θ ( x t i − 1 for , t i − 1 ) + μ θ ( x t i − 1 ⋆ , t i − 1 )∥ ≤ e λ t i δ i ∥ μ θ ( x t i for , t i ) − μ θ ( x t i ⋆ , t i ) − μ θ ( x t i − 1 for , t i − 1 ) + μ θ ( x t i − 1 ⋆ , t i − 1 )∥ = O ( M − 3 ) . (209)
Putting bounds for χ 1 \chi_1 χ 1 , χ 2 \chi_2 χ 2 , and χ 3 \chi_3 χ 3 together. Finally, combining the bounds in (202) , (203) , and (209) , and then plugging it into (195) , we find that for any i ≥ 1 i \ge1 i ≥ 1 ,
∥ x t i b c k − x t i ⋆ + x t i f o r − x t i ⋆ ∥ ≤ σ t i σ t i − 1 ( 1 + L x e δ i δ i ) ∥ x t i − 1 b c k + x t i − 1 f o r − 2 x t i − 1 ⋆ ∥ + σ t i O ( M − 3 ) . \left\lVert x^{\mathrm{bck}}_{t_i}-x^\star_{t_i}+x^{\mathrm{for}}_{t_i}-x^\star_{t_i}\right\rVert
\le\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}\left(1+L_xe^{\delta_i}\delta_i\right)\left\lVert x^{\mathrm{bck}}_{t_{i-1}}+x^{\mathrm{for}}_{t_{i-1}}-2x^\star_{t_{i-1}}\right\rVert+\sigma_{t_i}O(M^{-3}). x t i bck − x t i ⋆ + x t i for − x t i ⋆ ≤ σ t i − 1 σ t i ( 1 + L x e δ i δ i ) x t i − 1 bck + x t i − 1 for − 2 x t i − 1 ⋆ + σ t i O ( M − 3 ) .
Applying this inequality recursively from i = 1 i=1 i = 1 to i = M i=M i = M , and noting that x t 0 b c k + x t 0 f o r − 2 x t 0 ⋆ = 0 x^{\mathrm{bck}}_{t_0}+x^{\mathrm{for}}_{t_0}-2x^\star_{t_0}=0 x t 0 bck + x t 0 for − 2 x t 0 ⋆ = 0 , we obtain
∥ x t M b c k − x t M ⋆ + x t M f o r − x t M ⋆ ∥ ≤ ∑ i = 1 M ∏ j = i + 1 M σ t j σ t j − 1 ( 1 + L x e δ j δ j ) ⋅ σ t i O ( M − 3 ) ≤ ∑ i = 1 M ∏ j = i + 1 M σ t j σ t j − 1 exp ( L x e δ j δ j ) ⋅ σ t i O ( M − 3 ) = ∑ i = 1 M σ t M exp ( L x e δ ( λ t M − λ t i ) ) ⋅ O ( M − 3 ) ≤ M σ t M exp ( L x e δ ( λ t M − λ t 0 ) ) ⋅ O ( M − 3 ) = O ( M − 2 ) , \begin{aligned}
\left\lVert x^{\mathrm{bck}}_{t_M}-x^\star_{t_M}+x^{\mathrm{for}}_{t_M}-x^\star_{t_M}\right\rVert
&\le\sum_{i=1}^{M}\prod_{j=i+1}^{M}\frac{\sigma_{t_j}}{\sigma_{t_{j-1}}}\left(1+L_xe^{\delta_j}\delta_j\right)\cdot\sigma_{t_i}O(M^{-3})\\
&\le\sum_{i=1}^{M}\prod_{j=i+1}^{M}\frac{\sigma_{t_j}}{\sigma_{t_{j-1}}}\exp(L_xe^{\delta_j}\delta_j)\cdot\sigma_{t_i}O(M^{-3})\\
&=\sum_{i=1}^{M}\sigma_{t_M}\exp\left(L_xe^\delta(\lambda_{t_M}-\lambda_{t_i})\right)\cdot O(M^{-3})\\
&\le M\sigma_{t_M}\exp\left(L_xe^\delta(\lambda_{t_M}-\lambda_{t_0})\right)\cdot O(M^{-3})=O(M^{-2}),
\end{aligned} x t M bck − x t M ⋆ + x t M for − x t M ⋆ ≤ i = 1 ∑ M j = i + 1 ∏ M σ t j − 1 σ t j ( 1 + L x e δ j δ j ) ⋅ σ t i O ( M − 3 ) ≤ i = 1 ∑ M j = i + 1 ∏ M σ t j − 1 σ t j exp ( L x e δ j δ j ) ⋅ σ t i O ( M − 3 ) = i = 1 ∑ M σ t M exp ( L x e δ ( λ t M − λ t i ) ) ⋅ O ( M − 3 ) ≤ M σ t M exp ( L x e δ ( λ t M − λ t 0 ) ) ⋅ O ( M − 3 ) = O ( M − 2 ) ,
where the second step is true because 1 + x ≤ e x 1+x\le e^x 1 + x ≤ e x for any x ∈ R x\in\mathbb{R} x ∈ R , and the last line holds as λ t M − λ t i ≤ λ t M − λ t 0 \lambda_{t_M}-\lambda_{t_i}\le\lambda_{t_M}-\lambda_{t_0} λ t M − λ t i ≤ λ t M − λ t 0 for all 0 ≤ i ≤ M 0\le i\le M 0 ≤ i ≤ M .
Proof of Theorem 7
Comparing the update rules of our sampler (21) and that of the idealized forward-value discretization (178) , one can express the difference between their iterates as
x t i − x t i f o r = σ t i σ t i − 1 ( x t i − 1 − x t i − 1 f o r ) + σ t i ∫ λ t i − 1 λ t i e λ d λ ⋅ ( μ θ ( x ^ t i , t i ) − μ θ ( x t i f o r , t i ) ) , x_{t_i}-x^{\mathrm{for}}_{t_i}
=\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}\left(x_{t_{i-1}}-x^{\mathrm{for}}_{t_{i-1}}\right)+\sigma_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^\lambda\,\mathrm{d}\lambda\cdot\left(\mu_\theta(\widehat{x}_{t_i},t_i)-\mu_\theta(x^{\mathrm{for}}_{t_i},t_i)\right), x t i − x t i for = σ t i − 1 σ t i ( x t i − 1 − x t i − 1 for ) + σ t i ∫ λ t i − 1 λ t i e λ d λ ⋅ ( μ θ ( x t i , t i ) − μ θ ( x t i for , t i ) ) ,
for any i ≥ 1 i\ge1 i ≥ 1 . Leveraging the Lipschitz property of μ θ ( x t i , t i ) \mu_\theta(x_{t_i},t_i) μ θ ( x t i , t i ) , we can bound
∥ x t i − x t i f o r ∥ ≤ σ t i σ t i − 1 ∥ x t i − 1 − x t i − 1 f o r ∥ + σ t i ∫ λ t i − 1 λ t i e λ d λ ⋅ ∥ μ θ ( x t i f o r , t i ) − μ θ ( x ^ t i , t i ) ∥ ≤ σ t i σ t i − 1 ∥ x t i − 1 − x t i − 1 f o r ∥ + σ t i e λ t i δ i L x α t i ∥ x t i f o r − x ^ t i ∥ ≤ σ t i σ t i − 1 ∥ x t i − 1 − x t i − 1 f o r ∥ + δ i L x ∥ x t i f o r − x t i ⋆ ( x t i − 1 , t i − 1 ) ∥ + δ i L x ∥ x t i ⋆ ( x t i − 1 , t i − 1 ) − x ^ t i ∥ , (210) \begin{aligned}
\left\lVert x_{t_i}-x^{\mathrm{for}}_{t_i}\right\rVert
&\le\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}\left\lVert x_{t_{i-1}}-x^{\mathrm{for}}_{t_{i-1}}\right\rVert+\sigma_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^\lambda\,\mathrm{d}\lambda\cdot\left\lVert\mu_\theta(x^{\mathrm{for}}_{t_i},t_i)-\mu_\theta(\widehat{x}_{t_i},t_i)\right\rVert\\
&\le\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}\left\lVert x_{t_{i-1}}-x^{\mathrm{for}}_{t_{i-1}}\right\rVert+\sigma_{t_i}e^{\lambda_{t_i}}\delta_i\frac{L_x}{\alpha_{t_i}}\left\lVert x^{\mathrm{for}}_{t_i}-\widehat{x}_{t_i}\right\rVert\\
&\le\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}\left\lVert x_{t_{i-1}}-x^{\mathrm{for}}_{t_{i-1}}\right\rVert+\delta_iL_x\left\lVert x^{\mathrm{for}}_{t_i}-x^\star_{t_i}(x_{t_{i-1}},t_{i-1})\right\rVert\\
&\qquad+\delta_iL_x\left\lVert x^\star_{t_i}(x_{t_{i-1}},t_{i-1})-\widehat{x}_{t_i}\right\rVert, \tag*{(210)}
\end{aligned} x t i − x t i for ≤ σ t i − 1 σ t i x t i − 1 − x t i − 1 for + σ t i ∫ λ t i − 1 λ t i e λ d λ ⋅ μ θ ( x t i for , t i ) − μ θ ( x t i , t i ) ≤ σ t i − 1 σ t i x t i − 1 − x t i − 1 for + σ t i e λ t i δ i α t i L x x t i for − x t i ≤ σ t i − 1 σ t i x t i − 1 − x t i − 1 for + δ i L x x t i for − x t i ⋆ ( x t i − 1 , t i − 1 ) + δ i L x x t i ⋆ ( x t i − 1 , t i − 1 ) − x t i , (210)
where the last equation uses σ t i e λ t i / α t i = e λ t i e − λ t i = 1 \sigma_{t_i}e^{\lambda_{t_i}}/\alpha_{t_i}=e^{\lambda_{t_i}}e^{-\lambda_{t_i}}=1 σ t i e λ t i / α t i = e λ t i e − λ t i = 1 .
We claim that the second term on the right-hand-side of (210) satisfies the following bound:
∥ x t i f o r − x t i ⋆ ( x t i − 1 , t i − 1 ) ∥ ≤ σ t i σ t i − 1 ( 1 − δ i L x ) ∥ x t i − 1 f o r − x t i − 1 ∥ + σ t i e λ t i L ~ x δ i 2 2 ( 1 − δ i L x ) . (211) \left\lVert x^{\mathrm{for}}_{t_i}-x^\star_{t_i}(x_{t_{i-1}},t_{i-1})\right\rVert
\le\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}(1-\delta_iL_x)}\left\lVert x^{\mathrm{for}}_{t_{i-1}}-x_{t_{i-1}}\right\rVert+\frac{\sigma_{t_i}e^{\lambda_{t_i}}\widetilde{L}_x\delta_i^2}{2(1-\delta_iL_x)}. \tag*{(211)} x t i for − x t i ⋆ ( x t i − 1 , t i − 1 ) ≤ σ t i − 1 ( 1 − δ i L x ) σ t i x t i − 1 for − x t i − 1 + 2 ( 1 − δ i L x ) σ t i e λ t i L x δ i 2 . (211)
The proof is deferred to the end of this section.
Armed with (211) , for i ≥ 2 i \ge2 i ≥ 2 , we can plug it into (210) to obtain the following key relationship regarding the difference between our sampler x t i x_{t_i} x t i and the idealized forward-value discretization x t i f o r x_{t_i}^{\mathrm{for}} x t i for :
∥ x t i − x t i f o r ∥ ≤ σ t i σ t i − 1 ∥ x t i − 1 − x t i − 1 f o r ∥ + δ i L x ( σ t i σ t i − 1 ( 1 − δ i L x ) ∥ x t i − 1 f o r − x t i − 1 ∥ + σ t i e λ t i L x L ~ t δ i 2 2 ( 1 − δ i L x ) ) + δ i L x ∥ x t i ⋆ ( x t i − 1 , t i − 1 ) − x ^ t i ∥ = σ t i ∥ x t i − 1 f o r − x t i − 1 ∥ σ t i − 1 ( 1 − δ i L x ) + σ t i e λ t i L x L ~ t δ i 3 2 ( 1 − δ i L x ) + δ i L x ∥ x t i ⋆ ( x t i − 1 , t i − 1 ) − x ^ t i ∥ = σ t i ∥ x t i − 1 f o r − x t i − 1 ∥ σ t i − 1 ( 1 − δ i L x ) + o ( M − 2 ) , (212) \begin{aligned}
\left\|x_{t_i}-x_{t_i}^{\mathrm{for}}\right\|
&\le\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}\left\|x_{t_{i-1}}-x_{t_{i-1}}^{\mathrm{for}}\right\|+\delta_iL_x\left(\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}(1-\delta_iL_x)}\left\|x_{t_{i-1}}^{\mathrm{for}}-x_{t_{i-1}}\right\|+\frac{\sigma_{t_i}e^{\lambda_{t_i}L_x}\tilde{L}_t\delta_i^2}{2(1-\delta_iL_x)}\right)\\
&\quad+\delta_iL_x\left\|x_{t_i}^{\star}(x_{t_{i-1}},t_{i-1})-\hat{x}_{t_i}\right\|\\
&=\frac{\sigma_{t_i}\left\|x_{t_{i-1}}^{\mathrm{for}}-x_{t_{i-1}}\right\|}{\sigma_{t_{i-1}}(1-\delta_iL_x)}+\frac{\sigma_{t_i}e^{\lambda_{t_i}L_x}\tilde{L}_t\delta_i^3}{2(1-\delta_iL_x)}+\delta_iL_x\left\|x_{t_i}^{\star}(x_{t_{i-1}},t_{i-1})-\hat{x}_{t_i}\right\|\\
&=\frac{\sigma_{t_i}\left\|x_{t_{i-1}}^{\mathrm{for}}-x_{t_{i-1}}\right\|}{\sigma_{t_{i-1}}(1-\delta_iL_x)}+o(M^{-2}), \tag*{(212)}
\end{aligned} x t i − x t i for ≤ σ t i − 1 σ t i x t i − 1 − x t i − 1 for + δ i L x ( σ t i − 1 ( 1 − δ i L x ) σ t i x t i − 1 for − x t i − 1 + 2 ( 1 − δ i L x ) σ t i e λ t i L x L ~ t δ i 2 ) + δ i L x x t i ⋆ ( x t i − 1 , t i − 1 ) − x ^ t i = σ t i − 1 ( 1 − δ i L x ) σ t i x t i − 1 for − x t i − 1 + 2 ( 1 − δ i L x ) σ t i e λ t i L x L ~ t δ i 3 + δ i L x x t i ⋆ ( x t i − 1 , t i − 1 ) − x ^ t i = σ t i − 1 ( 1 − δ i L x ) σ t i x t i − 1 for − x t i − 1 + o ( M − 2 ) , (212)
where the last equation uses (181) , namely ∥ x t i ⋆ ( x t i − 1 , t i − 1 ) − x ^ t i ∥ = o ( M − 1 ) \left\|x_{t_i}^{\star}(x_{t_{i-1}},t_{i-1})-\hat{x}_{t_i}\right\|=o(M^{-1}) x t i ⋆ ( x t i − 1 , t i − 1 ) − x ^ t i = o ( M − 1 ) and δ i = O ( M − 1 ) \delta_i=O(M^{-1}) δ i = O ( M − 1 ) . For the first step, the common initialization x t 0 = x t 0 f o r x_{t_0}=x_{t_0}^{\mathrm{for}} x t 0 = x t 0 for and (210) –(211) give
∥ x t 1 − x t 1 f o r ∥ ≤ σ t 1 e λ t 1 L x L ~ t δ 1 3 2 ( 1 − δ 1 L x ) + δ 1 L x ∥ x t 1 ⋆ ( x t 0 , t 0 ) − x ^ t 1 ∥ = O ( M − 3 ) + O ( M − 2 ) = O ( M − 2 ) , \begin{aligned}
\left\|x_{t_1}-x_{t_1}^{\mathrm{for}}\right\|
&\le\frac{\sigma_{t_1}e^{\lambda_{t_1}L_x}\tilde{L}_t\delta_1^3}{2(1-\delta_1L_x)}+\delta_1L_x\left\|x_{t_1}^{\star}(x_{t_0},t_0)-\hat{x}_{t_1}\right\|\\
&=O(M^{-3})+O(M^{-2})=O(M^{-2}),
\end{aligned} x t 1 − x t 1 for ≤ 2 ( 1 − δ 1 L x ) σ t 1 e λ t 1 L x L ~ t δ 1 3 + δ 1 L x x t 1 ⋆ ( x t 0 , t 0 ) − x ^ t 1 = O ( M − 3 ) + O ( M − 2 ) = O ( M − 2 ) ,
where we used δ 1 = O ( M − 1 ) \delta_1=O(M^{-1}) δ 1 = O ( M − 1 ) and the first-step bound in (181) . Applying relationship (212) recursively from i = 2 i=2 i = 2 , and using the first-step bound ∥ x t 1 − x t 1 f o r ∥ = O ( M − 2 ) \left\|x_{t_1}-x_{t_1}^{\mathrm{for}}\right\|=O(M^{-2}) x t 1 − x t 1 for = O ( M − 2 ) established above, we can bound the final error at t M t_M t M as
∥ x t M − x t M f o r ∥ ≤ ∏ j = 2 M σ t j σ t j − 1 ( 1 − δ j L x ) ∥ x t 1 − x t 1 f o r ∥ + ∑ i = 2 M ∏ j = i + 1 M σ t j σ t j − 1 ( 1 − δ j L x ) ⋅ o ( M − 2 ) . (213) \left\|x_{t_M}-x_{t_M}^{\mathrm{for}}\right\|\le\prod_{j=2}^{M}\frac{\sigma_{t_j}}{\sigma_{t_{j-1}}(1-\delta_jL_x)}\left\|x_{t_1}-x_{t_1}^{\mathrm{for}}\right\|+\sum_{i=2}^{M}\prod_{j=i+1}^{M}\frac{\sigma_{t_j}}{\sigma_{t_{j-1}}(1-\delta_jL_x)}\cdot o(M^{-2}). \tag*{(213)} x t M − x t M for ≤ j = 2 ∏ M σ t j − 1 ( 1 − δ j L x ) σ t j x t 1 − x t 1 for + i = 2 ∑ M j = i + 1 ∏ M σ t j − 1 ( 1 − δ j L x ) σ t j ⋅ o ( M − 2 ) . (213)
To finish up, note that
∏ j = i + 1 M σ t j σ t j − 1 ( 1 − δ j L x ) ≤ ( a ) ∏ j = i + 1 M σ t j σ t j − 1 exp ( 2 δ j L x ) = σ t M σ t i exp ( 2 L x ∑ j = i + 1 M δ j ) = σ t M σ t i exp ( 2 L x ( λ t M − λ t i ) ) ≤ ( b ) exp ( 2 L x ( λ t M − λ t 0 ) ) , \begin{aligned}
\prod_{j=i+1}^{M}\frac{\sigma_{t_j}}{\sigma_{t_{j-1}}(1-\delta_jL_x)}
&\overset{(a)}{\le}\prod_{j=i+1}^{M}\frac{\sigma_{t_j}}{\sigma_{t_{j-1}}}\exp(2\delta_jL_x)
=\frac{\sigma_{t_M}}{\sigma_{t_i}}\exp\left(2L_x\sum_{j=i+1}^{M}\delta_j\right)\\
&=\frac{\sigma_{t_M}}{\sigma_{t_i}}\exp\left(2L_x(\lambda_{t_M}-\lambda_{t_i})\right)\\
&\overset{(b)}{\le}\exp\left(2L_x(\lambda_{t_M}-\lambda_{t_0})\right),
\end{aligned} j = i + 1 ∏ M σ t j − 1 ( 1 − δ j L x ) σ t j ≤ ( a ) j = i + 1 ∏ M σ t j − 1 σ t j exp ( 2 δ j L x ) = σ t i σ t M exp ( 2 L x j = i + 1 ∑ M δ j ) = σ t i σ t M exp ( 2 L x ( λ t M − λ t i ) ) ≤ ( b ) exp ( 2 L x ( λ t M − λ t 0 ) ) ,
where (a) holds because 1 / ( 1 − x ) ≤ exp ( 2 x ) 1/(1-x)\le\exp(2x) 1/ ( 1 − x ) ≤ exp ( 2 x ) for 0 ≤ x ≤ 1 / 2 0\le x\le1/2 0 ≤ x ≤ 1/2 and δ j L x = O ( M − 1 ) ≤ 1 / 2 \delta_jL_x=O(M^{-1})\le1/2 δ j L x = O ( M − 1 ) ≤ 1/2 ; (b) is true because of the assumption σ t i ≥ σ t M \sigma_{t_i}\ge\sigma_{t_M} σ t i ≥ σ t M and λ t M − λ t i ≤ λ t M − λ t 0 \lambda_{t_M}-\lambda_{t_i}\le\lambda_{t_M}-\lambda_{t_0} λ t M − λ t i ≤ λ t M − λ t 0 for all 0 ≤ i ≤ M 0\le i\le M 0 ≤ i ≤ M . Using this bound reaches the advertised result:
∥ x t M − x t M f o r ∥ ≤ exp ( 2 L x ( λ t M − λ t 0 ) ) ( O ( M − 2 ) + ( M − 1 ) o ( M − 2 ) ) = o ( M − 1 ) . (214) \begin{aligned}
\left\|x_{t_M}-x_{t_M}^{\mathrm{for}}\right\|
&\le\exp\left(2L_x(\lambda_{t_M}-\lambda_{t_0})\right)\left(O(M^{-2})+(M-1)o(M^{-2})\right)\\
&=o(M^{-1}). \tag*{(214)}
\end{aligned} x t M − x t M for ≤ exp ( 2 L x ( λ t M − λ t 0 ) ) ( O ( M − 2 ) + ( M − 1 ) o ( M − 2 ) ) = o ( M − 1 ) . (214)
It remains to prove the claim (211) . Towards this, note that by the diffusion ODE (184) , we can rewrite x t i ⋆ ( x t i − 1 , t i − 1 ) x_{t_i}^{\star}(x_{t_{i-1}},t_{i-1}) x t i ⋆ ( x t i − 1 , t i − 1 ) defined in (182) in terms of the data predictor μ θ \mu_\theta μ θ as
x t i ⋆ ( x t i − 1 , t i − 1 ) = σ t i σ t i − 1 x t i − 1 + σ t i ∫ λ t i − 1 λ t i e λ μ θ ( x t i ⋆ ( x t i − 1 , t i − 1 ) , t ( λ ) ) d λ . x_{t_i}^{\star}(x_{t_{i-1}},t_{i-1})=\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}x_{t_{i-1}}+\sigma_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^\lambda\mu_\theta\left(x_{t_i}^{\star}(x_{t_{i-1}},t_{i-1}),t(\lambda)\right)\,\mathrm{d}\lambda. x t i ⋆ ( x t i − 1 , t i − 1 ) = σ t i − 1 σ t i x t i − 1 + σ t i ∫ λ t i − 1 λ t i e λ μ θ ( x t i ⋆ ( x t i − 1 , t i − 1 ) , t ( λ ) ) d λ .
Combined with the definition of x t i f o r x_{t_i}^{\mathrm{for}} x t i for (cf. (178) ), the above representation allows us to decompose their difference as
x t i f o r − x t i ⋆ ( x t i − 1 , t i − 1 ) = σ t i σ t i − 1 ( x t i − 1 f o r − x t i − 1 ) + σ t i ∫ λ t i − 1 λ t i e λ ( μ θ ( x t i f o r , t i ) − μ θ ( x t i ⋆ ( x t i − 1 , t i − 1 ) , t ( λ ) ) ) d λ x_{t_i}^{\mathrm{for}}-x_{t_i}^{\star}(x_{t_{i-1}},t_{i-1})=\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}\left(x_{t_{i-1}}^{\mathrm{for}}-x_{t_{i-1}}\right)+\sigma_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^\lambda\left(\mu_\theta(x_{t_i}^{\mathrm{for}},t_i)-\mu_\theta\left(x_{t_i}^{\star}(x_{t_{i-1}},t_{i-1}),t(\lambda)\right)\right)\,\mathrm{d}\lambda x t i for − x t i ⋆ ( x t i − 1 , t i − 1 ) = σ t i − 1 σ t i ( x t i − 1 for − x t i − 1 ) + σ t i ∫ λ t i − 1 λ t i e λ ( μ θ ( x t i for , t i ) − μ θ ( x t i ⋆ ( x t i − 1 , t i − 1 ) , t ( λ ) ) ) d λ
x ^ t i − x t i ⋆ ( x t i − 1 , t i − 1 ) = σ t i σ t i − 1 ( x t i − 1 f o r − x t i − 1 ) + σ t i ∫ λ t i − 1 λ t i e λ d λ ⋅ ( μ θ ( x t i f o r , t i ) − μ θ ( x t i ⋆ ( x t i − 1 , t i − 1 ) , t i ) ) + σ t i ∫ λ t i − 1 λ t i e λ ( μ θ ( x t i ⋆ ( x t i − 1 , t i − 1 ) , t i ) − μ θ ( x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) , t ( λ ) ) ) d λ . (215) \begin{aligned}
\widehat{x}_{t_i}-x_{t_i}^{\star}(x_{t_{i-1}},t_{i-1})
={}&\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}\left(x_{t_{i-1}}^{\mathrm{for}}-x_{t_{i-1}}\right)
+\sigma_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^\lambda\,\mathrm{d}\lambda\cdot\left(\mu_\theta(x_{t_i}^{\mathrm{for}},t_i)-\mu_\theta(x_{t_i}^{\star}(x_{t_{i-1}},t_{i-1}),t_i)\right)\\
&+\sigma_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^\lambda\left(\mu_\theta(x_{t_i}^{\star}(x_{t_{i-1}},t_{i-1}),t_i)-\mu_\theta(x_{t(\lambda)}^{\star}(x_{t_{i-1}},t_{i-1}),t(\lambda))\right)\,\mathrm{d}\lambda. \tag*{(215)}
\end{aligned} x t i − x t i ⋆ ( x t i − 1 , t i − 1 ) = σ t i − 1 σ t i ( x t i − 1 for − x t i − 1 ) + σ t i ∫ λ t i − 1 λ t i e λ d λ ⋅ ( μ θ ( x t i for , t i ) − μ θ ( x t i ⋆ ( x t i − 1 , t i − 1 ) , t i ) ) + σ t i ∫ λ t i − 1 λ t i e λ ( μ θ ( x t i ⋆ ( x t i − 1 , t i − 1 ) , t i ) − μ θ ( x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) , t ( λ )) ) d λ . (215)
Let us control the second and third quantities on the right-hand-side of (215) separately. Regarding the second term, we can invoke a similar argument as in (190) to bound it as
∥ σ t i ∫ λ t i − 1 λ t i e λ d λ ⋅ ( μ θ ( x t i f o r , t i ) − μ θ ( x t i ⋆ ( x t i − 1 , t i − 1 ) , t i ) ) ∥ ≤ σ t i e λ t i L x α t i ∥ x t i f o r − x t i ⋆ ( x t i − 1 , t i − 1 ) ∥ = δ i L x ∥ x t i f o r − x t i ⋆ ( x t i − 1 , t i − 1 ) ∥ . (216) \begin{aligned}
\left\|\sigma_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^\lambda\,\mathrm{d}\lambda\cdot\left(\mu_\theta(x_{t_i}^{\mathrm{for}},t_i)-\mu_\theta(x_{t_i}^{\star}(x_{t_{i-1}},t_{i-1}),t_i)\right)\right\|
\le\sigma_{t_i}e^{\lambda_{t_i}}\frac{L_x}{\alpha_{t_i}}\left\|x_{t_i}^{\mathrm{for}}-x_{t_i}^{\star}(x_{t_{i-1}},t_{i-1})\right\| \\
&=\delta_iL_x\left\|x_{t_i}^{\mathrm{for}}-x_{t_i}^{\star}(x_{t_{i-1}},t_{i-1})\right\|. \tag*{(216)}
\end{aligned} σ t i ∫ λ t i − 1 λ t i e λ d λ ⋅ ( μ θ ( x t i for , t i ) − μ θ ( x t i ⋆ ( x t i − 1 , t i − 1 ) , t i ) ) ≤ σ t i e λ t i α t i L x x t i for − x t i ⋆ ( x t i − 1 , t i − 1 ) = δ i L x x t i for − x t i ⋆ ( x t i − 1 , t i − 1 ) . (216)
where the first inequality uses λ t i − 1 ≤ λ t i \lambda_{t_{i-1}}\le\lambda_{t_i} λ t i − 1 ≤ λ t i and the Lipschitz condition of μ θ \mu_\theta μ θ from Assumption 1 (A2), and the last equation uses λ t i = log ( α t i / σ t i ) \lambda_{t_i}=\log(\alpha_{t_i}/\sigma_{t_i}) λ t i = log ( α t i / σ t i ) . As for the third quantity, repeating a similar argument for (191) , one can bound it as
σ t i ∥ ∫ λ t i − 1 λ t i e λ ( μ θ ( x t i ⋆ ( x t i − 1 , t i − 1 ) , t i ) − μ θ ( x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) , t ( λ ) ) ) d λ ∥ ≤ 1 2 σ t i e λ t i L ~ t δ i 2 . \sigma_{t_i}\left\|\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^\lambda\left(\mu_\theta(x_{t_i}^{\star}(x_{t_{i-1}},t_{i-1}),t_i)-\mu_\theta(x_{t(\lambda)}^{\star}(x_{t_{i-1}},t_{i-1}),t(\lambda))\right)\,\mathrm{d}\lambda\right\|
\le\frac{1}{2}\sigma_{t_i}e^{\lambda_{t_i}}\widetilde{L}_t\delta_i^2. σ t i ∫ λ t i − 1 λ t i e λ ( μ θ ( x t i ⋆ ( x t i − 1 , t i − 1 ) , t i ) − μ θ ( x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) , t ( λ )) ) d λ ≤ 2 1 σ t i e λ t i L t δ i 2 .
Plugging these two bounds into (215) and rearranging the inequality, we finish the proof of the claim (211) .
Proof of F-DDIM. Finally, we verify the lookahead accuracy of F-DDIM satisfies (181) . Let
E i = ∥ x ^ t i − x t i ⋆ ( x t i − 1 , t i − 1 ) ∥ . E_i=\left\|\widehat{x}_{t_i}-x_{t_i}^{\star}(x_{t_{i-1}},t_{i-1})\right\|. E i = x t i − x t i ⋆ ( x t i − 1 , t i − 1 ) .
We first record uniform boundedness of the iterates. Since ∥ μ θ ∥ ≤ L μ \|\mu_\theta\|\le L_\mu ∥ μ θ ∥ ≤ L μ and σ t i / σ t i − 1 ≤ 1 \sigma_{t_i}/\sigma_{t_{i-1}}\le1 σ t i / σ t i − 1 ≤ 1 , the corrected update gives
∥ x t i ∥ ≤ ∥ x t 0 ∥ + α max L μ ∑ j = 1 i δ j = O ( 1 ) . \|x_{t_i}\|\le\|x_{t_0}\|+\alpha_{\max}L_\mu\sum_{j=1}^{i}\delta_j=O(1). ∥ x t i ∥ ≤ ∥ x t 0 ∥ + α m a x L μ j = 1 ∑ i δ j = O ( 1 ) .
For i ≥ 2 i\ge2 i ≥ 2 , the update rule of x ^ t i \widehat{x}_{t_i} x t i gives
∥ x ^ t i ∥ ≤ C ∥ x t i − 1 ∥ + C δ ∥ x ^ t i − 1 ∥ = O ( 1 ) . \|\widehat{x}_{t_i}\|\le C\|x_{t_{i-1}}\|+C\delta\|\widehat{x}_{t_{i-1}}\|=O(1). ∥ x t i ∥ ≤ C ∥ x t i − 1 ∥ + C δ ∥ x t i − 1 ∥ = O ( 1 ) .
The local exact trajectories are also bounded by the integral form (184) . Thus there exists a ball ∥ x ∥ ≤ R \|x\|\le R ∥ x ∥ ≤ R that contains all these states and trajectories.
We next establish the derivative bounds of the noise predictor ϵ θ \epsilon_\theta ϵ θ . From ϵ θ = x / σ t − e λ t μ θ \epsilon_\theta=x/\sigma_t-e^{\lambda_t}\mu_\theta ϵ θ = x / σ t − e λ t μ θ , Assumption 1 (A1), (A2), and (A4) give the following inequalities for all x x x in a fixed ball ∥ x ∥ ≤ R \|x\|\le R ∥ x ∥ ≤ R and λ ∈ [ λ t i − 1 , λ t i ] \lambda\in[\lambda_{t_{i-1}},\lambda_{t_i}] λ ∈ [ λ t i − 1 , λ t i ] :
∇ x ϵ θ = I − α t ∇ x μ θ σ t , ∥ ∇ x ϵ θ ∥ ≤ 1 + L x σ t ≤ 1 + L x σ t i = : K x , i , \nabla_x\epsilon_\theta=\frac{I-\alpha_t\nabla_x\mu_\theta}{\sigma_t},\qquad
\|\nabla_x\epsilon_\theta\|\le\frac{1+L_x}{\sigma_t}\le\frac{1+L_x}{\sigma_{t_i}}=:K_{x,i}, ∇ x ϵ θ = σ t I − α t ∇ x μ θ , ∥ ∇ x ϵ θ ∥ ≤ σ t 1 + L x ≤ σ t i 1 + L x =: K x , i ,
∂ ∂ λ ϵ θ = − 1 σ t 2 ∂ ∂ t ( λ ) σ t ( λ ) x − e λ ( μ θ + ∂ ∂ t ( λ ) μ θ ) , ∥ ∂ ∂ λ ϵ θ ∥ ≤ R max λ ∣ ∂ ∂ λ log σ t ( λ ) ∣ σ t i + e λ t i ( L μ + L t ) = : K t , i . \frac{\partial}{\partial\lambda}\epsilon_\theta=-\frac{1}{\sigma_t^2}\frac{\partial}{\partial t(\lambda)}\sigma_t(\lambda)x-e^\lambda\left(\mu_\theta+\frac{\partial}{\partial t(\lambda)}\mu_\theta\right),\qquad
\left\|\frac{\partial}{\partial\lambda}\epsilon_\theta\right\|\le\frac{R\max_\lambda\left|\frac{\partial}{\partial\lambda}\log\sigma_t(\lambda)\right|}{\sigma_{t_i}}+e^{\lambda_{t_i}}(L_\mu+L_t)=:K_{t,i}. ∂ λ ∂ ϵ θ = − σ t 2 1 ∂ t ( λ ) ∂ σ t ( λ ) x − e λ ( μ θ + ∂ t ( λ ) ∂ μ θ ) , ∂ λ ∂ ϵ θ ≤ σ t i R max λ ∂ λ ∂ log σ t ( λ ) + e λ t i ( L μ + L t ) =: K t , i .
Consider F-DDIM with its cached noise predictions. We have
x ^ t i − x t i ⋆ ( x t i − 1 , t i − 1 ) = α t i ∫ λ t i − 1 λ t i e − λ ϵ θ ( x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) , t ( λ ) ) d λ − ( α t i σ t i − 1 α t i − 1 − σ t i ) ϵ ~ t i − 1 = α t i ∫ λ t i − 1 λ t i e − λ ( ϵ θ ( x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) , t ( λ ) ) − ϵ ~ t i − 1 ) d λ . \begin{aligned}
\widehat{x}_{t_i}-x_{t_i}^{\star}(x_{t_{i-1}},t_{i-1})
=\alpha_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-\lambda}\epsilon_\theta(x_{t(\lambda)}^{\star}(x_{t_{i-1}},t_{i-1}),t(\lambda))\,\mathrm{d}\lambda-\left(\frac{\alpha_{t_i}\sigma_{t_{i-1}}}{\alpha_{t_{i-1}}}-\sigma_{t_i}\right)\widetilde{\epsilon}_{t_{i-1}} \\
&=\alpha_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-\lambda}\left(\epsilon_\theta(x_{t(\lambda)}^{\star}(x_{t_{i-1}},t_{i-1}),t(\lambda))-\widetilde{\epsilon}_{t_{i-1}}\right)\,\mathrm{d}\lambda.
\end{aligned} x t i − x t i ⋆ ( x t i − 1 , t i − 1 ) = α t i ∫ λ t i − 1 λ t i e − λ ϵ θ ( x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) , t ( λ )) d λ − ( α t i − 1 α t i σ t i − 1 − σ t i ) ϵ t i − 1 = α t i ∫ λ t i − 1 λ t i e − λ ( ϵ θ ( x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) , t ( λ )) − ϵ t i − 1 ) d λ .
For i ≥ 2 i\ge2 i ≥ 2 , taking norms and using the definition ϵ ~ t i − 1 = ϵ θ ( x ^ t i − 1 , t i − 1 ) \widetilde{\epsilon}_{t_{i-1}}=\epsilon_\theta(\widehat{x}_{t_{i-1}},t_{i-1}) ϵ t i − 1 = ϵ θ ( x t i − 1 , t i − 1 ) gives
E i ≤ α t i ∫ λ t i − 1 λ t i e − λ ∥ ϵ θ ( x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) , t ( λ ) ) − ϵ ~ t i − 1 ∥ d λ ≤ α t i ∫ λ t i − 1 λ t i e − λ ( K x , i ∥ x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) − x ^ t i − 1 ∥ + K t , i ( λ − λ t i − 1 ) ) d λ ≤ σ t i e δ i K x , i ∫ λ t i − 1 λ t i ∥ x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) − x ^ t i − 1 ∥ d λ + K t , i σ t i e δ i δ i 2 , i ≥ 2. (217) \begin{aligned}
E_i\le\alpha_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-\lambda}\left\|\epsilon_\theta(x_{t(\lambda)}^{\star}(x_{t_{i-1}},t_{i-1}),t(\lambda))-\widetilde{\epsilon}_{t_{i-1}}\right\|\,\mathrm{d}\lambda\\
&\le\alpha_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}} e^{-\lambda}\left(K_{x,i}\left\|\boldsymbol{x}^{\star}_{t(\lambda)}(x_{t_{i-1}},t_{i-1})-\widehat{x}_{t_{i-1}}\right\|+K_{t,i}(\lambda-\lambda_{t_{i-1}})\right)\,\mathrm{d}\lambda\\
&\le\sigma_{t_i}e^{\delta_i}K_{x,i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}\left\|\boldsymbol{x}^{\star}_{t(\lambda)}(x_{t_{i-1}},t_{i-1})-\widehat{x}_{t_{i-1}}\right\|\,\mathrm{d}\lambda+K_{t,i}\sigma_{t_i}e^{\delta_i}\delta_i^2,\quad i\ge2. \tag*{(217)}
\end{aligned} E i ≤ α t i ∫ λ t i − 1 λ t i e − λ ϵ θ ( x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) , t ( λ )) − ϵ t i − 1 d λ ≤ α t i ∫ λ t i − 1 λ t i e − λ ( K x , i x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) − x t i − 1 + K t , i ( λ − λ t i − 1 ) ) d λ ≤ σ t i e δ i K x , i ∫ λ t i − 1 λ t i x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) − x t i − 1 d λ + K t , i σ t i e δ i δ i 2 , i ≥ 2. (217)
It remains to prove that
∥ x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) − x ^ t i − 1 ∥ = O ( δ i ) , λ t i − 1 ≤ λ ≤ λ t i , i ≥ 2. (218) \left\|\boldsymbol{x}^{\star}_{t(\lambda)}(x_{t_{i-1}},t_{i-1})-\widehat{x}_{t_{i-1}}\right\|=O(\delta_i),\quad\lambda_{t_{i-1}}\le\lambda\le\lambda_{t_i},\quad i\ge2. \tag*{(218)} x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) − x t i − 1 = O ( δ i ) , λ t i − 1 ≤ λ ≤ λ t i , i ≥ 2. (218)
Towards this, we first note that
∥ x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) − x ^ t i − 1 ∥ ≤ ∥ x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) − x t i − 1 ∥ + ∥ x t i − 1 − x ^ t i − 1 ∥ . (219) \left\|\boldsymbol{x}^{\star}_{t(\lambda)}(x_{t_{i-1}},t_{i-1})-\widehat{x}_{t_{i-1}}\right\|\le\left\|\boldsymbol{x}^{\star}_{t(\lambda)}(x_{t_{i-1}},t_{i-1})-x_{t_{i-1}}\right\|+\left\|x_{t_{i-1}}-\widehat{x}_{t_{i-1}}\right\|. \tag*{(219)} x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) − x t i − 1 ≤ x t ( λ ) ⋆ ( x t i − 1 , t i − 1 ) − x t i − 1 + ∥ x t i − 1 − x t i − 1 ∥ . (219)
The ODE (184) and Assumption 1 (A1) bound the first term by C δ i C\delta_i C δ i , uniformly on the fixed finite λ \lambda λ interval. For the second term, inserting the update rules of x t i x_{t_i} x t i and x ^ t i \widehat{x}_{t_i} x t i for i ≥ 2 i\ge2 i ≥ 2 yields
x t i − x ^ t i = ( σ t i α t i − 1 σ t i − 1 − α t i ) ( μ θ ( x ^ t i − 1 , t i − 1 ) − μ θ ( x ^ t i , t i ) ) + ( σ t i σ t i − 1 − α t i α t i − 1 ) ( x t i − 1 − x ^ t i − 1 ) = α t i ( e − δ i − 1 ) ( μ θ ( x ^ t i − 1 , t i − 1 ) − μ θ ( x t i − 1 , t i − 1 ) + μ θ ( x t i − 1 , t i − 1 ) − μ θ ( x ^ t i , t i ) ) + α t i α t i − 1 ( e − δ i − 1 ) ( x t i − 1 − x ^ t i − 1 ) . (220) \begin{aligned}
x_{t_i}-\widehat{x}_{t_i}=\left(\frac{\sigma_{t_i}\alpha_{t_{i-1}}}{\sigma_{t_{i-1}}}-\alpha_{t_i}\right)\left(\mu_\theta(\widehat{x}_{t_{i-1}},t_{i-1})-\mu_\theta(\widehat{x}_{t_i},t_i)\right)+\left(\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}-\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}\right)(x_{t_{i-1}}-\widehat{x}_{t_{i-1}}) \\
&=\alpha_{t_i}\left(e^{-\delta_i}-1\right)\left(\mu_\theta(\widehat{x}_{t_{i-1}},t_{i-1})-\mu_\theta(x_{t_{i-1}},t_{i-1})+\mu_\theta(x_{t_{i-1}},t_{i-1})-\mu_\theta(\widehat{x}_{t_i},t_i)\right)
+\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}\left(e^{-\delta_i}-1\right)(x_{t_{i-1}}-\widehat{x}_{t_{i-1}}). \tag*{(220)}
\end{aligned} x t i − x t i = ( σ t i − 1 σ t i α t i − 1 − α t i ) ( μ θ ( x t i − 1 , t i − 1 ) − μ θ ( x t i , t i ) ) + ( σ t i − 1 σ t i − α t i − 1 α t i ) ( x t i − 1 − x t i − 1 ) = α t i ( e − δ i − 1 ) ( μ θ ( x t i − 1 , t i − 1 ) − μ θ ( x t i − 1 , t i − 1 ) + μ θ ( x t i − 1 , t i − 1 ) − μ θ ( x t i , t i ) ) + α t i − 1 α t i ( e − δ i − 1 ) ( x t i − 1 − x t i − 1 ) . (220)
Using Assumption 1 (A2) and (A4) yields
∥ x t i − x ^ t i ∥ ≤ α t i α t i − 1 L x δ i ( ∥ x t i − 1 − x ^ t i − 1 ∥ + ∥ x t i − 1 − x ^ t i ∥ ) + α t i L t δ i 2 + α t i α t i − 1 δ i ∥ x t i − 1 − x ^ t i − 1 ∥ ≤ C δ i ∥ x t i − 1 − x ^ t i − 1 ∥ + C δ i 2 , i ≥ 2. (221) \begin{aligned}
\left\|x_{t_i}-\widehat{x}_{t_i}\right\|\le\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}L_x\delta_i\left(\left\|x_{t_{i-1}}-\widehat{x}_{t_{i-1}}\right\|+\left\|x_{t_{i-1}}-\widehat{x}_{t_i}\right\|\right)
+\alpha_{t_i}L_t\delta_i^2+\frac{\alpha_{t_i}}{\alpha_{t_{i-1}}}\delta_i\left\|x_{t_{i-1}}-\widehat{x}_{t_{i-1}}\right\| \\
&\le C\delta_i\left\|x_{t_{i-1}}-\widehat{x}_{t_{i-1}}\right\|+C\delta_i^2,\quad i\ge2. \tag*{(221)}
\end{aligned} ∥ x t i − x t i ∥ ≤ α t i − 1 α t i L x δ i ( ∥ x t i − 1 − x t i − 1 ∥ + ∥ x t i − 1 − x t i ∥ ) + α t i L t δ i 2 + α t i − 1 α t i δ i ∥ x t i − 1 − x t i − 1 ∥ ≤ C δ i ∥ x t i − 1 − x t i − 1 ∥ + C δ i 2 , i ≥ 2. (221)
Here the update rule of x ^ t i \widehat{x}_{t_i} x t i gives ∥ x t i − 1 − x ^ t i ∥ = O ( δ i ) \left\|x_{t_{i-1}}-\widehat{x}_{t_i}\right\|=O(\delta_i) ∥ x t i − 1 − x t i ∥ = O ( δ i ) . The initialization gives
∥ x t 1 − x ^ t 1 ∥ ≤ α t 1 α t 0 ∣ σ t 0 − 1 ∣ ( 1 − e − δ 1 ) ∥ x t 0 ∥ + α t 1 ( 1 − e − δ 1 ) ∥ μ θ ( x ^ t 1 , t 1 ) ∥ = O ( δ 1 ) . \left\|x_{t_1}-\widehat{x}_{t_1}\right\|\le\frac{\alpha_{t_1}}{\alpha_{t_0}}\left|\sigma_{t_0}-1\right|(1-e^{-\delta_1})\left\|x_{t_0}\right\|+\alpha_{t_1}(1-e^{-\delta_1})\left\|\mu_\theta(\widehat{x}_{t_1},t_1)\right\|=O(\delta_1). ∥ x t 1 − x t 1 ∥ ≤ α t 0 α t 1 ∣ σ t 0 − 1 ∣ ( 1 − e − δ 1 ) ∥ x t 0 ∥ + α t 1 ( 1 − e − δ 1 ) ∥ μ θ ( x t 1 , t 1 ) ∥ = O ( δ 1 ) .
Induction for sufficiently small δ \delta δ therefore gives
max 2 ≤ i ≤ M ∥ x t i − x ^ t i ∥ = O ( δ 2 ) , \max_{2\le i\le M}\left\|x_{t_i}-\widehat{x}_{t_i}\right\|=O(\delta^2), 2 ≤ i ≤ M max ∥ x t i − x t i ∥ = O ( δ 2 ) ,
which completes the proof.
Proof of convergence order upper bound for F-DDPM (Theorem 5)
We begin with deriving the following preliminary bounds for x t i x_{t_i} x t i and x ^ t i ′ \widehat{x}_{t_i}^{\prime} x t i ′ :
E [ ∥ x t i ∥ 2 2 ] = O ( 1 ) , E [ ∥ x ^ t i ′ ∥ 2 2 ] = O ( 1 ) . (222) \mathbb{E}\left[\left\|x_{t_i}\right\|_2^2\right]=O(1),\qquad\mathbb{E}\left[\left\|\widehat{x}_{t_i}^{\prime}\right\|_2^2\right]=O(1). \tag*{(222)} E [ ∥ x t i ∥ 2 2 ] = O ( 1 ) , E [ x t i ′ 2 2 ] = O ( 1 ) . (222)
The proof is postponed to the end of this section.
Note that the difference between the update rules in (18) and F-DDPM sampler (24) is that the data predictor μ ( x t i − 1 , t i − 1 ) \mu(x_{t_{i-1}},t_{i-1}) μ ( x t i − 1 , t i − 1 ) is replaced by μ ( x ^ t i ′ , t i ′ ) \mu(\widehat{x}_{t_i}^{\prime},t_i^{\prime}) μ ( x t i ′ , t i ′ ) . Similar to the analysis in (81) , we obtain
x t i − x t i ⋆ = σ t i σ t i − 1 e − δ i ( x t i − 1 − x t i − 1 ⋆ ) + α t i ( 1 − e − 2 δ i ) ( μ ( x ^ t i ′ , t i ′ ) − μ ( x t i − 1 ⋆ , t i − 1 ) ) + 2 α t i ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) ( μ ( x t i − 1 ⋆ , t i − 1 ) − μ ( x t ( λ ) ⋆ , t ( λ ) ) ) d λ = σ t i σ t i − 1 e − δ i ( x t i − 1 − x t i − 1 ⋆ ) + α t i ( 1 − e − 2 δ i ) ( μ ( x t i − 1 , t i − 1 ) − μ ( x t i − 1 ⋆ , t i − 1 ) ) + α t i ( 1 − e − 2 δ i ) ( μ ( x ^ t i , t i ′ ) − μ ( x t i − 1 , t i − 1 ) ) + 2 α t i ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) ( μ ( x t i − 1 ⋆ , t i − 1 ) − μ ( x t ( λ ) ⋆ , t ( λ ) ) ) d λ (223) \begin{aligned}
x_{t_i}-x_{t_i}^{\star}=\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}e^{-\delta_i}(x_{t_{i-1}}-x_{t_{i-1}}^{\star})+\alpha_{t_i}(1-e^{-2\delta_i})\left(\mu(\widehat{x}_{t_i}^{\prime},t_i^{\prime})-\mu(x_{t_{i-1}}^{\star},t_{i-1})\right) \\
&+2\alpha_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-2(\lambda_{t_i}-\lambda)}\left(\mu(x^{\star}_{t_{i-1}},t_{i-1})-\mu(x^{\star}_{t(\lambda)},t(\lambda))\right)\,\mathrm{d}\lambda\\
&=\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}e^{-\delta_i}\left(x_{t_{i-1}}-x^{\star}_{t_{i-1}}\right)+\alpha_{t_i}\left(1-e^{-2\delta_i}\right)\left(\mu(x_{t_{i-1}},t_{i-1})-\mu(x^{\star}_{t_{i-1}},t_{i-1})\right) \\
&+\alpha_{t_i}\left(1-e^{-2\delta_i}\right)\left(\mu(\widehat{x}_{t_i},t_i')-\mu(x_{t_{i-1}},t_{i-1})\right) \\
&+2\alpha_{t_i}\int_{\lambda_{t_{i-1}}}^{\lambda_{t_i}}e^{-2(\lambda_{t_i}-\lambda)}\left(\mu(x^{\star}_{t_{i-1}},t_{i-1})-\mu(x^{\star}_{t(\lambda)},t(\lambda))\right)\,\mathrm{d}\lambda \tag*{(223)}
\end{aligned} x t i − x t i ⋆ = σ t i − 1 σ t i e − δ i ( x t i − 1 − x t i − 1 ⋆ ) + α t i ( 1 − e − 2 δ i ) ( μ ( x t i ′ , t i ′ ) − μ ( x t i − 1 ⋆ , t i − 1 ) ) + 2 α t i ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) ( μ ( x t i − 1 ⋆ , t i − 1 ) − μ ( x t ( λ ) ⋆ , t ( λ )) ) d λ = σ t i − 1 σ t i e − δ i ( x t i − 1 − x t i − 1 ⋆ ) + α t i ( 1 − e − 2 δ i ) ( μ ( x t i − 1 , t i − 1 ) − μ ( x t i − 1 ⋆ , t i − 1 ) ) + α t i ( 1 − e − 2 δ i ) ( μ ( x t i , t i ′ ) − μ ( x t i − 1 , t i − 1 ) ) + 2 α t i ∫ λ t i − 1 λ t i e − 2 ( λ t i − λ ) ( μ ( x t i − 1 ⋆ , t i − 1 ) − μ ( x t ( λ ) ⋆ , t ( λ )) ) d λ (223)
Repeating arguments in Step 1 in Appendix § 11 .§ 1 yields
x t i − x t i ⋆ = A t i ( x t i − 1 − x t i − 1 ⋆ ) + E t i , 1 + E t i , 2 + α t i ( 1 − e − 2 δ i ) ( μ ( x ^ t i , t i ′ ) − μ ( x t i − 1 , t i − 1 ) ) , (224) x_{t_i}-x^{\star}_{t_i}=A_{t_i}\left(x_{t_{i-1}}-x^{\star}_{t_{i-1}}\right)+\mathcal{E}_{t_i,1}+\mathcal{E}_{t_i,2}+\alpha_{t_i}\left(1-e^{-2\delta_i}\right)\left(\mu(\widehat{x}_{t_i},t_i')-\mu(x_{t_{i-1}},t_{i-1})\right), \tag*{(224)} x t i − x t i ⋆ = A t i ( x t i − 1 − x t i − 1 ⋆ ) + E t i , 1 + E t i , 2 + α t i ( 1 − e − 2 δ i ) ( μ ( x t i , t i ′ ) − μ ( x t i − 1 , t i − 1 ) ) , (224)
where the residual terms E t i , 1 \mathcal{E}_{t_i,1} E t i , 1 and E t i , 2 \mathcal{E}_{t_i,2} E t i , 2 are defined in (79). By recursion, we obtain
E [ ∥ x t M − x t M ⋆ ∥ 2 ] ≤ ∑ m = 1 M E [ ∥ ∏ j = m + 1 M ∥ A t j ∥ o p ∥ E t m , 1 ∥ 2 ] + E [ ∥ ∑ m = 1 M ∏ j = m + 1 M A t j E t m , 2 ∥ 2 ] + E [ ∥ ∑ m = 1 M ∏ j = m + 1 M A t j α t m ( 1 − e − 2 δ m ) ( μ ( x ^ t m , t m ′ ) − μ ( x t m − 1 , t m − 1 ) ) ∥ 2 ] (225) \begin{aligned}
\mathbb{E}[\|x_{t_M}-x^{\star}_{t_M}\|_2]\leq\sum_{m=1}^{M}\mathbb{E}[\|\prod_{j=m+1}^{M}\|A_{t_j}\|_{\mathrm{op}}\|\mathcal{E}_{t_m,1}\|_2]+\mathbb{E}[\|\sum_{m=1}^{M}\prod_{j=m+1}^{M}A_{t_j}\mathcal{E}_{t_m,2}\|_2] \\
&+\mathbb{E}\left[\left\|\sum_{m=1}^{M}\prod_{j=m+1}^{M}A_{t_j}\alpha_{t_m}\left(1-e^{-2\delta_m}\right)\left(\mu(\widehat{x}_{t_m},t_m')-\mu(x_{t_{m-1}},t_{m-1})\right)\right\|_2\right] \tag*{(225)}
\end{aligned} E [ ∥ x t M − x t M ⋆ ∥ 2 ] ≤ m = 1 ∑ M E [ ∥ j = m + 1 ∏ M ∥ A t j ∥ op ∥ E t m , 1 ∥ 2 ] + E [ ∥ m = 1 ∑ M j = m + 1 ∏ M A t j E t m , 2 ∥ 2 ] + E m = 1 ∑ M j = m + 1 ∏ M A t j α t m ( 1 − e − 2 δ m ) ( μ ( x t m , t m ′ ) − μ ( x t m − 1 , t m − 1 ) ) 2 (225)
Below we shall bound these three terms separately.
Control the second term. Note that the definition of E t m , 2 \mathcal{E}_{t_m,2} E t m , 2 is independent of the update rule, and consequently the bound derived in Step 3 in Appendix § 11 .§ 1 still holds, i.e.,
E [ ∥ ∑ m = 1 M ∏ j = m + 1 M A t j E t m , 2 ∥ 2 ] = O ( δ ) . (226) \mathbb{E}\left[\left\|\sum_{m=1}^{M}\prod_{j=m+1}^{M}A_{t_j}\mathcal{E}_{t_m,2}\right\|_2\right]=O(\delta). \tag*{(226)} E m = 1 ∑ M j = m + 1 ∏ M A t j E t m , 2 2 = O ( δ ) . (226)
E [ ∥ ∑ m = 1 M ∏ j = m + 1 M A t j α t m ( 1 − e − 2 δ m ) ( μ ( x ^ t m , t m ′ ) − μ ( x t m − 1 , t m − 1 ) ) ∥ 2 ] ≲ ∑ m = 1 M δ m E [ ∥ x ^ t m ′ − x t m − 1 ∥ 2 ] + ∑ m = 1 M δ m 2 , (227) \mathbb{E}\left[\left\|\sum_{m=1}^{M}\prod_{j=m+1}^{M}A_{t_j}\alpha_{t_m}\left(1-e^{-2\delta_m}\right)\left(\mu(\widehat{x}_{t_m},t_m')-\mu(x_{t_{m-1}},t_{m-1})\right)\right\|_2\right]
\lesssim\sum_{m=1}^{M}\delta_m\mathbb{E}\left[\left\|\widehat{x}_{t_m}'-x_{t_{m-1}}\right\|_2\right]+\sum_{m=1}^{M}\delta_m^2, \tag*{(227)} E m = 1 ∑ M j = m + 1 ∏ M A t j α t m ( 1 − e − 2 δ m ) ( μ ( x t m , t m ′ ) − μ ( x t m − 1 , t m − 1 ) ) 2 ≲ m = 1 ∑ M δ m E [ x t m ′ − x t m − 1 2 ] + m = 1 ∑ M δ m 2 , (227)
where we have used
∥ μ ( x ^ t m ′ , t m ′ ) − μ ( x t m − 1 , t m − 1 ) ∥ 2 ≤ ∥ μ ( x ^ t m ′ , t m − 1 ) − μ ( x t m − 1 , t m − 1 ) ∥ 2 + ∥ μ ( x ^ t m ′ , t m ′ ) − μ ( x ^ t m ′ , t m − 1 ) ∥ 2 ≤ L x α t m − 1 ∥ x ^ t m ′ − x t m − 1 ∥ 2 + L t ( λ t m + δ m − λ t m − 1 ) ≤ L x α t m − 1 ∥ x ^ t m ′ − x t m − 1 ∥ 2 + 2 L t δ m . (228) \begin{aligned}
\left\|\mu(\widehat{x}_{t_m}',t_m')-\mu(x_{t_{m-1}},t_{m-1})\right\|_2 \\
&\leq\left\|\mu(\widehat{x}_{t_m}',t_{m-1})-\mu(x_{t_{m-1}},t_{m-1})\right\|_2+\left\|\mu(\widehat{x}_{t_m}',t_m')-\mu(\widehat{x}_{t_m}',t_{m-1})\right\|_2 \\
&\leq\frac{L_x}{\alpha_{t_{m-1}}}\left\|\widehat{x}_{t_m}'-x_{t_{m-1}}\right\|_2+L_t(\lambda_{t_m}+\delta_m-\lambda_{t_{m-1}}) \\
&\leq\frac{L_x}{\alpha_{t_{m-1}}}\left\|\widehat{x}_{t_m}'-x_{t_{m-1}}\right\|_2+2L_t\delta_m. \tag*{(228)}
\end{aligned} μ ( x t m ′ , t m ′ ) − μ ( x t m − 1 , t m − 1 ) 2 ≤ μ ( x t m ′ , t m − 1 ) − μ ( x t m − 1 , t m − 1 ) 2 + μ ( x t m ′ , t m ′ ) − μ ( x t m ′ , t m − 1 ) 2 ≤ α t m − 1 L x x t m ′ − x t m − 1 2 + L t ( λ t m + δ m − λ t m − 1 ) ≤ α t m − 1 L x x t m ′ − x t m − 1 2 + 2 L t δ m . (228)
According to the (28) and (29) , we have
x ^ t m ′ = α t m ′ α t m − 1 x t m − 1 − σ t m ′ ( e 2 δ m − 1 ) ϵ ~ t m − 1 = α t m ′ α t m − 1 x t m − 1 − σ t m ′ ( e 2 δ m − 1 ) ( e − δ m − 1 ϵ θ ( x ^ t m − 1 ′ , t m − 1 ′ ) + 1 − e − 2 δ m − 1 z m − 1 ) = α t m α t m − 1 x t m − 1 − σ t m σ t m − 1 e − δ m ( e 2 δ m − 1 ) ( x ^ t m − 1 ′ − α t m − 1 ′ μ ( x ^ t m − 1 ′ , t m − 1 ′ ) ) − σ t m ′ ( e 2 δ m − 1 ) 1 − e − 2 δ m − 1 z m − 1 . (229) \begin{aligned}
\widehat{x}_{t_m}'=\frac{\alpha_{t_m}'}{\alpha_{t_{m-1}}}x_{t_{m-1}}-\sigma_{t_m}'\left(e^{2\delta_m}-1\right)\widetilde{\epsilon}_{t_{m-1}} \\
&=\frac{\alpha_{t_m}'}{\alpha_{t_{m-1}}}x_{t_{m-1}}-\sigma_{t_m}'\left(e^{2\delta_m}-1\right)\left(e^{-\delta_{m-1}}\epsilon_{\theta}(\widehat{x}_{t_{m-1}}',t_{m-1}')+\sqrt{1-e^{-2\delta_{m-1}}}z_{m-1}\right) \\
&= \frac{\alpha_{t_m}}{\alpha_{t_{m-1}}}x_{t_{m-1}}-\frac{\sigma_{t_m}}{\sigma_{t_{m-1}}}e^{-\delta_m}(e^{2\delta_m}-1)\left(\widehat{x}_{t'_{m-1}}-\alpha_{t'_{m-1}}\mu(\widehat{x}_{t'_{m-1}},t'_{m-1})\right)
-\sigma_{t_m}'(e^{2\delta_m}-1)\sqrt{1-e^{-2\delta_{m-1}}}z_{m-1}. \tag*{(229)}
\end{aligned} x t m ′ = α t m − 1 α t m ′ x t m − 1 − σ t m ′ ( e 2 δ m − 1 ) ϵ t m − 1 = α t m − 1 α t m ′ x t m − 1 − σ t m ′ ( e 2 δ m − 1 ) ( e − δ m − 1 ϵ θ ( x t m − 1 ′ , t m − 1 ′ ) + 1 − e − 2 δ m − 1 z m − 1 ) = α t m − 1 α t m x t m − 1 − σ t m − 1 σ t m e − δ m ( e 2 δ m − 1 ) ( x t m − 1 ′ − α t m − 1 ′ μ ( x t m − 1 ′ , t m − 1 ′ ) ) − σ t m ′ ( e 2 δ m − 1 ) 1 − e − 2 δ m − 1 z m − 1 . (229)
Consequently, for sufficiently small δ \delta δ such that e 2 δ ≤ 1.5 e^{2\delta}\le1.5 e 2 δ ≤ 1.5 , we have
∥ x ^ t m ′ − x t m − 1 ∥ 2 ≤ ( α t m ′ α t m − 1 − 1 ) ∥ x t m − 1 ∥ 2 + 3 δ m ∥ x ^ t m ′ ∥ 2 + 3 α t m ′ δ m ∥ μ ( x ^ t m − 1 ′ , t m − 1 ′ ) ∥ 2 + 3 σ t m ′ ( e 2 δ m − 1 ) 1 − e − 2 δ m − 1 ∥ z m − 1 ∥ 2 \left\|\widehat{x}_{t'_m}-x_{t_{m-1}}\right\|_2
\le\left(\frac{\alpha_{t'_m}}{\alpha_{t_{m-1}}}-1\right)\left\|x_{t_{m-1}}\right\|_2+3\delta_m\left\|\widehat{x}_{t'_m}\right\|_2+3\alpha_{t'_m}\delta_m\left\|\mu(\widehat{x}_{t'_{m-1}},t'_{m-1})\right\|_2
+3\sigma_{t'_m}(e^{2\delta_m}-1)\sqrt{1-e^{-2\delta_{m-1}}}\left\|z_{m-1}\right\|_2 x t m ′ − x t m − 1 2 ≤ ( α t m − 1 α t m ′ − 1 ) ∥ x t m − 1 ∥ 2 + 3 δ m x t m ′ 2 + 3 α t m ′ δ m μ ( x t m − 1 ′ , t m − 1 ′ ) 2 + 3 σ t m ′ ( e 2 δ m − 1 ) 1 − e − 2 δ m − 1 ∥ z m − 1 ∥ 2
≤ ( a ) 3 δ m ∥ x t m − 1 ∥ 2 + 3 δ m ∥ x ^ t m ′ ∥ 2 + 3 α t m ′ δ m L μ + 3 σ t m ′ δ m 3 δ m − 1 ∥ z m − 1 ∥ 2 . \underset{(a)}{\le}3\delta_m\left\|x_{t_{m-1}}\right\|_2+3\delta_m\left\|\widehat{x}_{t'_m}\right\|_2+3\alpha_{t'_m}\delta_mL_\mu+3\sigma_{t'_m}\delta_m\sqrt{3\delta_{m-1}}\left\|z_{m-1}\right\|_2. ( a ) ≤ 3 δ m ∥ x t m − 1 ∥ 2 + 3 δ m x t m ′ 2 + 3 α t m ′ δ m L μ + 3 σ t m ′ δ m 3 δ m − 1 ∥ z m − 1 ∥ 2 .
where (a) holds since Assumption A1 and the fact that
α t m ′ α t m − 1 − 1 ≤ α t m ′ σ t m − 1 α t m − 1 σ t m ′ − 1 = e 2 δ m − 1 ≤ 3 δ m . \frac{\alpha_{t'_m}}{\alpha_{t_{m-1}}}-1\le\frac{\alpha_{t'_m}\sigma_{t_{m-1}}}{\alpha_{t_{m-1}}\sigma_{t'_m}}-1=e^{2\delta_m}-1\le3\delta_m. α t m − 1 α t m ′ − 1 ≤ α t m − 1 σ t m ′ α t m ′ σ t m − 1 − 1 = e 2 δ m − 1 ≤ 3 δ m .
Taking square and expectation further gives
E [ ∥ x ^ t m ′ − x t m − 1 ∥ 2 2 ] ≤ 36 δ m 2 E [ ∥ x t m − 1 ∥ 2 2 ] + 36 δ m 2 E [ ∥ x ^ t m ′ ∥ 2 2 ] + 36 α t m ′ 2 δ m 2 L μ 2 + 108 σ t m ′ 2 δ m 2 δ m − 1 d \mathbb{E}\left[\left\|\widehat{x}_{t'_m}-x_{t_{m-1}}\right\|_2^2\right]\le36\delta_m^2\mathbb{E}\left[\left\|x_{t_{m-1}}\right\|_2^2\right]+36\delta_m^2\mathbb{E}\left[\left\|\widehat{x}_{t'_m}\right\|_2^2\right]+36\alpha_{t'_m}^2\delta_m^2L_\mu^2+108\sigma_{t'_m}^2\delta_m^2\delta_{m-1}d E [ x t m ′ − x t m − 1 2 2 ] ≤ 36 δ m 2 E [ ∥ x t m − 1 ∥ 2 2 ] + 36 δ m 2 E [ x t m ′ 2 2 ] + 36 α t m ′ 2 δ m 2 L μ 2 + 108 σ t m ′ 2 δ m 2 δ m − 1 d
= ( a ) O ( δ m 2 ) , (230) \underset{(a)}{=}O(\delta_m^2), \tag*{(230)} ( a ) = O ( δ m 2 ) , (230)
where (a) arises from (222) . While for m = 1 m=1 m = 1 , we have
x ^ t 1 ′ − x t 0 = α t 1 ′ α t 0 x t 0 − σ t 1 ′ ( e 2 δ 1 − 1 ) ϵ ~ t 0 − x t 0 = ( α t 1 ′ α t 0 − σ t 1 ′ ( e 2 δ 1 − 1 ) − 1 ) x t 0 . (231) \widehat{x}_{t'_1}-x_{t_0}=\frac{\alpha_{t'_1}}{\alpha_{t_0}}x_{t_0}-\sigma_{t'_1}(e^{2\delta_1}-1)\widetilde{\epsilon}_{t_0}-x_{t_0}
=\left(\frac{\alpha_{t'_1}}{\alpha_{t_0}}-\sigma_{t'_1}(e^{2\delta_1}-1)-1\right)x_{t_0}. \tag*{(231)} x t 1 ′ − x t 0 = α t 0 α t 1 ′ x t 0 − σ t 1 ′ ( e 2 δ 1 − 1 ) ϵ t 0 − x t 0 = ( α t 0 α t 1 ′ − σ t 1 ′ ( e 2 δ 1 − 1 ) − 1 ) x t 0 . (231)
Therefore we have E [ ∥ x ^ t 1 ′ − x t 0 ∥ ] = O ( 1 ) \mathbb{E}[\|\widehat{x}_{t'_1}-x_{t_0}\|]=O(1) E [ ∥ x t 1 ′ − x t 0 ∥ ] = O ( 1 ) . Substituting into (227) yields
E [ ∥ ∑ m = 1 M ∏ j = m + 1 M A t j α t m ( 1 − e − 2 δ m ) ( μ ( x ^ t m ′ , t m ′ ) − μ ( x t m − 1 , t m − 1 ) ) ∥ 2 ] ≲ ∑ m = 1 M δ m E [ ∥ x ^ t m ′ − x t m − 1 ∥ 2 2 ] + ∑ m = 1 M δ m 2 \mathbb{E}\left[\left\|\sum_{m=1}^{M}\prod_{j=m+1}^{M}A_{t_j}\alpha_{t_m}(1-e^{-2\delta_m})\left(\mu(\widehat{x}_{t'_m},t'_m)-\mu(x_{t_{m-1}},t_{m-1})\right)\right\|_2\right]
\lesssim\sum_{m=1}^{M}\delta_m\sqrt{\mathbb{E}\left[\left\|\widehat{x}_{t'_m}-x_{t_{m-1}}\right\|_2^2\right]}+\sum_{m=1}^{M}\delta_m^2 E m = 1 ∑ M j = m + 1 ∏ M A t j α t m ( 1 − e − 2 δ m ) ( μ ( x t m ′ , t m ′ ) − μ ( x t m − 1 , t m − 1 ) ) 2 ≲ m = 1 ∑ M δ m E [ x t m ′ − x t m − 1 2 2 ] + m = 1 ∑ M δ m 2
≲ ∑ m = 2 M δ m 2 + δ 1 = O ( δ ) . (232) \lesssim\sum_{m=2}^{M}\delta_m^2+\delta_1=O(\delta). \tag*{(232)} ≲ m = 2 ∑ M δ m 2 + δ 1 = O ( δ ) . (232)
Control the first term. The analysis on E t m , 1 E_{t_m,1} E t m , 1 established in Step 4 in Appendix C.1 relies on the conclusion
E [ ∥ x t m − x t m ∗ ∥ 2 2 ] = O ( δ ) . (233) \mathbb{E}\left[\left\|x_{t_m}-x^*_{t_m}\right\|_2^2\right]=O(\delta). \tag*{(233)} E [ x t m − x t m ∗ 2 2 ] = O ( δ ) . (233)
Armed with (233) , the previous analysis implies that
∑ m = 1 M E [ ∥ E t m , 1 ∥ 2 ] = O ( δ ) . (234) \sum_{m=1}^{M}\mathbb{E}\left[\left\|E_{t_m,1}\right\|_2\right]=O(\delta). \tag*{(234)} m = 1 ∑ M E [ ∥ E t m , 1 ∥ 2 ] = O ( δ ) . (234)
The remaining proof focuses on establishing (233) for the update rule in (24) . By analysis in (111) , we have
∥ x t m − x t m ∗ ∥ 2 ≤ σ t m σ t m − 1 e − δ m ∥ x t m − 1 − x t m − 1 ∗ ∥ 2 + α t m ( 1 − e − 2 δ m ) ∥ μ ( x ^ t m ′ , t m ′ ) − μ ( x t m − 1 ∗ , t m − 1 ) ∥ 2 + 2 B t m \left\|x_{t_m}-x^*_{t_m}\right\|_2\le\frac{\sigma_{t_m}}{\sigma_{t_{m-1}}}e^{-\delta_m}\left\|x_{t_{m-1}}-x^*_{t_{m-1}}\right\|_2+\alpha_{t_m}(1-e^{-2\delta_m})\left\|\mu(\widehat{x}_{t'_m},t'_m)-\mu(x^*_{t_{m-1}},t_{m-1})\right\|_2+2B_{t_m} x t m − x t m ∗ 2 ≤ σ t m − 1 σ t m e − δ m x t m − 1 − x t m − 1 ∗ 2 + α t m ( 1 − e − 2 δ m ) μ ( x t m ′ , t m ′ ) − μ ( x t m − 1 ∗ , t m − 1 ) 2 + 2 B t m
≤ ( a ) σ t m σ t m − 1 e − δ m ∥ x t m − 1 − x t m − 1 ∗ ∥ 2 + α t m α t m − 1 L x ( 1 − e − 2 δ m ) ∥ x ^ t m ′ − x t m − 1 ∗ ∥ 2 + 2 α t m L t ( 1 − e − 2 δ m ) δ m + 2 B t m ≤ σ t m σ t m − 1 e − δ m ∥ x t m − 1 − x t m − 1 ∗ ∥ 2 + α t m α t m − 1 L x ( 1 − e − 2 δ m ) ∥ x t m − 1 − x t m − 1 ∗ ∥ 2 + α t m α t m − 1 L x ( 1 − e − 2 δ m ) ∥ x ^ t m ′ − x t m − 1 ∥ 2 + 2 α t m L t ( 1 − e − 2 δ m ) δ m + 2 B t m , (235) \begin{aligned}
\underset{(a)}{\le}\frac{\sigma_{t_m}}{\sigma_{t_{m-1}}}e^{-\delta_m}\left\|x_{t_{m-1}}-x^*_{t_{m-1}}\right\|_2+\frac{\alpha_{t_m}}{\alpha_{t_{m-1}}}L_x(1-e^{-2\delta_m})\left\|\widehat{x}_{t'_m}-x^*_{t_{m-1}}\right\|_2 \\
&+2\alpha_{t_m}L_t(1-e^{-2\delta_m})\delta_m+2B_{t_m} \\
&\le\frac{\sigma_{t_m}}{\sigma_{t_{m-1}}}e^{-\delta_m}\lVert x_{t_{m-1}}-x^*_{t_{m-1}}\rVert_2+\frac{\alpha_{t_m}}{\alpha_{t_{m-1}}}L_x(1-e^{-2\delta_m})\lVert x_{t_{m-1}}-x^*_{t_{m-1}}\rVert_2
+\frac{\alpha_{t_m}}{\alpha_{t_{m-1}}}L_x(1-e^{-2\delta_m})\lVert\widehat{x}_{t'_m}-x_{t_{m-1}}\rVert_2+2\alpha_{t_m}L_t(1-e^{-2\delta_m})\delta_m+2B_{t_m}, \tag*{(235)}
\end{aligned} ( a ) ≤ σ t m − 1 σ t m e − δ m x t m − 1 − x t m − 1 ∗ 2 + α t m − 1 α t m L x ( 1 − e − 2 δ m ) x t m ′ − x t m − 1 ∗ 2 + 2 α t m L t ( 1 − e − 2 δ m ) δ m + 2 B t m ≤ σ t m − 1 σ t m e − δ m ∥ x t m − 1 − x t m − 1 ∗ ∥ 2 + α t m − 1 α t m L x ( 1 − e − 2 δ m ) ∥ x t m − 1 − x t m − 1 ∗ ∥ 2 + α t m − 1 α t m L x ( 1 − e − 2 δ m ) ∥ x t m ′ − x t m − 1 ∥ 2 + 2 α t m L t ( 1 − e − 2 δ m ) δ m + 2 B t m , (235)
where B t m B_{t_m} B t m is defined in (112) and is independent of the update rule, and (a) uses (228) . Note that by (230) ,
E [ ( α t m α t m − 1 L x ( 1 − e − 2 δ m ) ∥ x ^ t m ′ − x t m − 1 ∥ 2 + 2 α t m L t ( 1 − e − 2 δ m ) δ m + 2 B t m ) 2 ] ≲ δ m 2 E [ ∥ x ^ t m ′ − x t m − 1 ∥ 2 2 ] + δ m 4 + E [ B t m 2 ] = ( a ) O ( δ m 3 ) , \mathbb{E}\left[\left(\frac{\alpha_{t_m}}{\alpha_{t_{m-1}}}L_x(1-e^{-2\delta_m})\lVert\widehat{x}_{t'_m}-x_{t_{m-1}}\rVert_2+2\alpha_{t_m}L_t(1-e^{-2\delta_m})\delta_m+2B_{t_m}\right)^2\right]
\lesssim\delta_m^2\mathbb{E}\left[\lVert\widehat{x}_{t'_m}-x_{t_{m-1}}\rVert_2^2\right]+\delta_m^4+\mathbb{E}[B_{t_m}^2]\overset{(a)}{=}O(\delta_m^3), E [ ( α t m − 1 α t m L x ( 1 − e − 2 δ m ) ∥ x t m ′ − x t m − 1 ∥ 2 + 2 α t m L t ( 1 − e − 2 δ m ) δ m + 2 B t m ) 2 ] ≲ δ m 2 E [ ∥ x t m ′ − x t m − 1 ∥ 2 2 ] + δ m 4 + E [ B t m 2 ] = ( a ) O ( δ m 3 ) ,
where (a) comes from (230) and the bound for the second moment of B t m B_{t_m} B t m in (114) . Repeating the analysis in (115) - (116) yields (233) .
Collecting (226) , (232) , and (234) into (225) finally yields
E [ ∥ x t M − x t M ∗ ∥ 2 ] ≤ O ( δ ) . (236) \mathbb{E}[\lVert x_{t_M}-x^*_{t_M}\rVert_2]\le O(\delta). \tag*{(236)} E [∥ x t M − x t M ∗ ∥ 2 ] ≤ O ( δ ) . (236)
This completes the proof.
Proof of (222) . According to the update rule (24) , we have
x t i = e − δ i σ t i σ t i − 1 x t i − 1 + α t i ( 1 − e − 2 δ i ) μ θ ( x ^ t i ′ , t i ′ ) + σ t i 1 − e − 2 δ i z i = e − ( λ t i − λ t 0 ) σ t i σ t 0 x t 0 + ∑ m = 1 i ∏ j = m + 1 i ( e − δ j σ t j σ t j − 1 ) ( α t m ( 1 − e − 2 δ m ) μ θ ( x ^ t m ′ , t m ′ ) + σ t m 1 − e − 2 δ m z m ) = e − ( λ t i − λ t 0 ) σ t i σ t 0 x t 0 + σ t i ∑ m = 1 i e − ( λ t i − λ t m ) α t m σ t m ( 1 − e − 2 δ m ) μ θ ( x ^ t m ′ , t m ′ ) + σ t i ∑ m = 1 i e − ( λ t i − λ t m ) 1 − e − 2 δ m z m (237) \begin{aligned}
x_{t_i}=e^{-\delta_i}\frac{\sigma_{t_i}}{\sigma_{t_{i-1}}}x_{t_{i-1}}+\alpha_{t_i}(1-e^{-2\delta_i})\mu_\theta(\widehat{x}_{t'_i},t'_i)+\sigma_{t_i}\sqrt{1-e^{-2\delta_i}}z_i \\
&=e^{-(\lambda_{t_i}-\lambda_{t_0})}\frac{\sigma_{t_i}}{\sigma_{t_0}}x_{t_0}+\sum_{m=1}^{i}\prod_{j=m+1}^{i}\left(e^{-\delta_j}\frac{\sigma_{t_j}}{\sigma_{t_{j-1}}}\right)\left(\alpha_{t_m}(1-e^{-2\delta_m})\mu_\theta(\widehat{x}_{t'_m},t'_m)+\sigma_{t_m}\sqrt{1-e^{-2\delta_m}}z_m\right) \\
&=e^{-(\lambda_{t_i}-\lambda_{t_0})}\frac{\sigma_{t_i}}{\sigma_{t_0}}x_{t_0}+\sigma_{t_i}\sum_{m=1}^{i}e^{-(\lambda_{t_i}-\lambda_{t_m})}\frac{\alpha_{t_m}}{\sigma_{t_m}}(1-e^{-2\delta_m})\mu_\theta(\widehat{x}_{t'_m},t'_m)+\sigma_{t_i}\sum_{m=1}^{i}e^{-(\lambda_{t_i}-\lambda_{t_m})}\sqrt{1-e^{-2\delta_m}}z_m \tag*{(237)}
\end{aligned} x t i = e − δ i σ t i − 1 σ t i x t i − 1 + α t i ( 1 − e − 2 δ i ) μ θ ( x t i ′ , t i ′ ) + σ t i 1 − e − 2 δ i z i = e − ( λ t i − λ t 0 ) σ t 0 σ t i x t 0 + m = 1 ∑ i j = m + 1 ∏ i ( e − δ j σ t j − 1 σ t j ) ( α t m ( 1 − e − 2 δ m ) μ θ ( x t m ′ , t m ′ ) + σ t m 1 − e − 2 δ m z m ) = e − ( λ t i − λ t 0 ) σ t 0 σ t i x t 0 + σ t i m = 1 ∑ i e − ( λ t i − λ t m ) σ t m α t m ( 1 − e − 2 δ m ) μ θ ( x t m ′ , t m ′ ) + σ t i m = 1 ∑ i e − ( λ t i − λ t m ) 1 − e − 2 δ m z m (237)
Note that z m z_m z m are independent. Taking square and expectation yields
E [ ∥ x t i ∥ 2 2 ] ≤ 3 e − 2 ( λ t i − λ t 0 ) σ t i 2 σ t 0 2 E [ ∥ x t 0 ∥ 2 2 ] + 3 σ t i 2 L μ 2 ( ∑ m = 1 i e − ( λ t i − λ t m ) α t m σ t m ( 1 − e − 2 δ m ) ) 2 + 3 σ t i 2 ∑ m = 1 i e − 2 ( λ t i − λ t m ) ( 1 − e − 2 δ m ) E [ ∥ z m ∥ 2 2 ] ≲ 1 + ( ∑ m = 1 i δ m ) 2 + ∑ m = 1 i δ m = O ( 1 ) . (238) \mathbb{E}[\lVert x_{t_i}\rVert_2^2]\le3e^{-2(\lambda_{t_i}-\lambda_{t_0})}\frac{\sigma_{t_i}^2}{\sigma_{t_0}^2}\mathbb{E}[\lVert x_{t_0}\rVert_2^2]+3\sigma_{t_i}^2L_\mu^2\left(\sum_{m=1}^{i}e^{-(\lambda_{t_i}-\lambda_{t_m})}\frac{\alpha_{t_m}}{\sigma_{t_m}}(1-e^{-2\delta_m})\right)^2
+3\sigma_{t_i}^2\sum_{m=1}^{i}e^{-2(\lambda_{t_i}-\lambda_{t_m})}(1-e^{-2\delta_m})\mathbb{E}[\lVert z_m\rVert_2^2]
\lesssim1+\left(\sum_{m=1}^{i}\delta_m\right)^2+\sum_{m=1}^{i}\delta_m=O(1). \tag*{(238)} E [∥ x t i ∥ 2 2 ] ≤ 3 e − 2 ( λ t i − λ t 0 ) σ t 0 2 σ t i 2 E [∥ x t 0 ∥ 2 2 ] + 3 σ t i 2 L μ 2 ( m = 1 ∑ i e − ( λ t i − λ t m ) σ t m α t m ( 1 − e − 2 δ m ) ) 2 + 3 σ t i 2 m = 1 ∑ i e − 2 ( λ t i − λ t m ) ( 1 − e − 2 δ m ) E [∥ z m ∥ 2 2 ] ≲ 1 + ( m = 1 ∑ i δ m ) 2 + m = 1 ∑ i δ m = O ( 1 ) . (238)
As for the expectation of ∥ x ^ t m ′ ∥ 2 2 \lVert\widehat{x}_{t'_m}\rVert_2^2 ∥ x t m ′ ∥ 2 2 , (229) implies that
∥ x ^ t m ′ ∥ 2 ≤ α t m ′ α t m − 1 ∥ x t m − 1 ∥ 2 + σ t m ′ σ t m − 1 ′ e − δ m − 1 ( e 2 δ m − 1 ) ∥ x ^ t m − 1 ′ ∥ 2 + e − 2 δ m + δ m − 1 − δ m − 1 ( e 2 δ m − 1 ) α t m ′ ∥ μ ( x ^ t m − 1 ′ , t m − 1 ′ ) ∥ 2 + σ t m ′ ( e 2 δ m − 1 ) 1 − e − 2 δ m − 1 ∥ z m − 1 ∥ 2 ≤ α t m ′ α t m − 1 ∥ x t m − 1 ∥ 2 + 4 δ m ∥ x ^ t m − 1 ′ ∥ 2 + 2 α t m ′ L μ δ m + 2 2 σ t m ′ e 2 δ m δ m δ m − 1 ∥ z m − 1 ∥ 2 . (239) \begin{aligned}
\lVert\widehat{x}_{t'_m}\rVert_2\le\frac{\alpha_{t'_m}}{\alpha_{t_{m-1}}}\lVert x_{t_{m-1}}\rVert_2+\frac{\sigma'_{t_m}}{\sigma_{t_{m-1}}'}e^{-\delta_{m-1}}(e^{2\delta_m}-1)\lVert\widehat{x}_{t'_{m-1}}\rVert_2+e^{-2\delta_m+\delta_{m-1}-\delta_{m-1}}(e^{2\delta_m}-1)\alpha_{t'_m}\lVert\mu(\widehat{x}_{t'_{m-1}},t'_{m-1})\rVert_2
+\sigma_{t'_m}(e^{2\delta_m}-1)\sqrt{1-e^{-2\delta_{m-1}}}\lVert z_{m-1}\rVert_2 \\
&\le\frac{\alpha_{t'_m}}{\alpha_{t_{m-1}}}\lVert x_{t_{m-1}}\rVert_2+4\delta_m\lVert\widehat{x}_{t'_{m-1}}\rVert_2+2\alpha_{t'_m}L_\mu\delta_m+2\sqrt{2}\sigma_{t'_m}e^{2\delta_m}\delta_m\sqrt{\delta_{m-1}}\lVert z_{m-1}\rVert_2. \tag*{(239)}
\end{aligned} ∥ x t m ′ ∥ 2 ≤ α t m − 1 α t m ′ ∥ x t m − 1 ∥ 2 + σ t m − 1 ′ σ t m ′ e − δ m − 1 ( e 2 δ m − 1 ) ∥ x t m − 1 ′ ∥ 2 + e − 2 δ m + δ m − 1 − δ m − 1 ( e 2 δ m − 1 ) α t m ′ ∥ μ ( x t m − 1 ′ , t m − 1 ′ ) ∥ 2 + σ t m ′ ( e 2 δ m − 1 ) 1 − e − 2 δ m − 1 ∥ z m − 1 ∥ 2 ≤ α t m − 1 α t m ′ ∥ x t m − 1 ∥ 2 + 4 δ m ∥ x t m − 1 ′ ∥ 2 + 2 α t m ′ L μ δ m + 2 2 σ t m ′ e 2 δ m δ m δ m − 1 ∥ z m − 1 ∥ 2 . (239)
By taking square and expectation, we have
E [ ∥ x ^ t m ′ ∥ 2 2 ] ≤ 4 α t m ′ 2 α t m − 1 2 E [ ∥ x t m − 1 ∥ 2 2 ] + 64 δ m 2 E [ ∥ x ^ t m − 1 ′ ∥ 2 2 ] + 16 α t m ′ 2 L μ 2 δ m 2 + 32 σ t m ′ 2 e 4 δ m δ m 2 δ m − 1 E [ ∥ z m − 1 ∥ 2 2 ] . \mathbb{E}[\lVert\widehat{x}_{t'_m}\rVert_2^2]\le\frac{4\alpha_{t'_m}^2}{\alpha_{t_{m-1}}^2}\mathbb{E}[\lVert x_{t_{m-1}}\rVert_2^2]+64\delta_m^2\mathbb{E}[\lVert\widehat{x}_{t'_{m-1}}\rVert_2^2]+16\alpha_{t'_m}^2L_\mu^2\delta_m^2+32\sigma_{t'_m}^2e^{4\delta_m}\delta_m^2\delta_{m-1}\mathbb{E}[\lVert z_{m-1}\rVert_2^2]. E [∥ x t m ′ ∥ 2 2 ] ≤ α t m − 1 2 4 α t m ′ 2 E [∥ x t m − 1 ∥ 2 2 ] + 64 δ m 2 E [∥ x t m − 1 ′ ∥ 2 2 ] + 16 α t m ′ 2 L μ 2 δ m 2 + 32 σ t m ′ 2 e 4 δ m δ m 2 δ m − 1 E [∥ z m − 1 ∥ 2 2 ] .
For 64 δ m 2 ≤ 1 / 2 64\delta_m^2 \le1/2 64 δ m 2 ≤ 1/2 , note that E [ ∥ z m − 1 ∥ 2 2 ] = d \mathbb{E}[\lVert z_{m-1}\rVert_2^2] = d E [∥ z m − 1 ∥ 2 2 ] = d and E [ ∥ x t m − 1 ∥ 2 2 ] = O ( 1 ) \mathbb{E}[\lVert x_{t_{m-1}}\rVert_2^2] = O(1) E [∥ x t m − 1 ∥ 2 2 ] = O ( 1 ) , we have
E [ ∥ x ~ t m ′ ∥ 2 2 ] ≲ max i ≤ m − 1 E [ ∥ x t i ∥ 2 2 ] + O ( δ 2 ) = O ( 1 ) . \mathbb{E}[\lVert\tilde{x}_{t'_m}\rVert_2^2] \lesssim\max_{i\le m-1} \mathbb{E}[\lVert x_{t_i}\rVert_2^2] + O(\delta^2) = O(1). E [∥ x ~ t m ′ ∥ 2 2 ] ≲ i ≤ m − 1 max E [∥ x t i ∥ 2 2 ] + O ( δ 2 ) = O ( 1 ) .
where the last inequality uses (238) .