PhD thesis 2026
Acknowledgements
I would like to thank my supervisors, François-Xavier Briol and Arthur Gretton, for their guidance on how to do sound research, how to become an independent researcher, and how to develop good taste in scientific problems. Arthur told me at the start of my PhD that the best way to learn a field is to write a paper on it; this has motivated me to explore many different areas. FX taught me to write with the audience always in mind; this has also become a habit of mine, although I am still working hard to do it as well as he does. Beyond research, I have learned a great deal from both of you about academic life more broadly. You have been the best possible combination of supervisors I could have hoped for. I am also grateful to Zhu Michael Li and Pierre Glaser, who acted as unofficial supervisors at the beginning of my PhD and taught me how to read and write proper proofs, almost hand by hand. Without their tutorials, my understanding of kernel methods are largely limited to linear algebra. I am especially grateful to Zhu Michael Li for his introducing me to many different pubs and restaurants around the Gatsby building. I would also like to thank Anna Korba, Lester Mackey, Chris Oates, Bharath K. Sriperumbudur, and Taiji Suzuki for their advice and support during our collaborations. Although they were not my official supervisors, they have had a substantial influence on my research taste and outlook. My three-month internship at RIKEN in Tokyo was one of the most unforgettable experiences of my PhD. Tokyo is an amazing city, and I hope to visit it again.
I would like to thank my other collaborators, Xiaoyuan Cheng, Ruocheng Guo, Heishiro Kanagawa, Sophia Kang, Toni Karvonen, Yang Liu, Dimitri Meunier, Aratrika Mustafi, Masha Naslidnyk, Atsushi Nitanda, Louis Sharrock, Zikai Steve Shen, Ingo Steinwart, Chenyang Tian, and Jean-François Ton. This thesis would not have been possible without your assistance and contributions. I am grateful to Kevin Han Huang for introducing me to friends from Warwick and for being willing to talk with me during some of my most difficult moments, and to Zhuo Sun, Jiaye Teng, and Zhiheng Zhang for introducing me to many academic friends in Shanghai. I am also grateful to Ben Dai, Fanghui Liu, Lei Shi, Weijie Su, and Lei Wu for giving me valuable advice on academic careers.
I would like to thank William Laplante for many wonderful evenings spent over hotpot and Biangbiang noodles. It has been amazing to meet someone from a completely different cultural background with whom I share so many personal traits. I am also grateful to the many people at the Gatsby Unit, including Hugh Dance, Peter Orbanz, Yedi Zhang, and others whom I did not always have the courage to reach out and talk to. It has been a great pleasure to spend time at the Gatsby Unit. It is a genuinely wonderful place for research, and my time there will remain among my best memories. From the Foundational AI Centre, I would like to thank William Bankes, David Chanin, Boyu Chen, Tiantian He, Shuofei Qiao, Yuxuan Shu, Seongho Son, Jiayi Wang, Lorenz Wolf, Bin Wu, Tianqi Wu, Moucheng Xu, Ziming Xu, Zhuangjian Yang, Ning Zhang, Mingtian Zhang, and Caroline Zhu for the wonderful events, retreats, and time spent together. In particular, I would like to thank Bin Wu, Tianqi Wu, Ning Zhang, Zhuangjian Yang, and Ziming Xu for helping me recover my stolen iPhone. I would like to thank my friends from the Fundamentals of Statistical Machine Learning group (FSML) in the Department of Statistical Science at UCL. The regular Monday group meetings and lunches helped me keep track of my PhD progress and encouraged me to socialize with others. Such an atmosphere is very rare. I would also like to thank my office mates, Wei Bu, Simon Damm, Nicola Gnecco, Zheng He, Christopher Marunoi, and Matthew Tilley, for the inspiring conversations in the visitor office. I would like to thank my friends met in Tokyo, Wei Huang, Zhanpeng Zhou, Tom Wakayama, Sho Sonada, Dake Bu, Bingrui Li, and Charles, for all the interesting conversations during lunch and dinners. I would also like to thank Tim G. J. Ruder and Weizhong Zhang who have taken me all the way to this path of machine learning back in my undergraduate. I would like to thank my roommates, Hengyi Wang, Yuxiang Wu, and Zhengyao Jiang, for their support and for the time we spent together playing VR, Junqi, and watching NBA games. I hope our friendship will last forever.
Finally, I would like to thank my family. I am deeply grateful to my parents, BinWei Chen and Jun Wu, for their support throughout my childhood. During high school, my obsession with the Mathematical Olympiad placed a considerable financial burden on my family. Although this path was uncertain and, in many ways, unrealistic, my parents continued to support my dream. Without their trust and continuing support, I would not have had the confidence and courage to pursue a PhD. I am also deeply grateful to my grandparents, Guomei Dong and Qifa Wu, for taking such good care of me when I was young. I wish them good health, and I will always remember their love and care.
I would also like to thank my girlfriend Churong Peng, and soon-to-be wife at the time of writing this thesis, who may already be my wife by the time it is submitted. Her support has meant a great deal to me throughout my PhD. Our one-hour WeChat video calls were the happiest part of each day. I believe a happy future is ahead of us.
Chapter 1
Introduction
Comparing two probability distributions P \mathbb{P} P Q \mathbb{Q} Q
Among the many possible choices of divergence or discrepancy D D D flexibility : different choices of kernel induce different topologies on probability measures (Sriperumbudur et al., 2010; Barp et al., 2024); 2) computability : MMD admits explicit plug-in estimators from finite samples from both distributions (Gretton et al., 2012); 3) sample complexity : for some commonly-used kernels, its sample complexity does not suffer from the curse of dimensionality ((Gretton et al., 2012; Tolstikhin et al., 2016)). Together, these favourable statistical and computational properties make MMD useful for a wide class of algorithms that approximate or model a target distribution by minimizing a computable discrepancy.
This thesis studies this optimization problem for the squared MMD discrepancy, MMD 2 ( P , Q ) \operatorname{MMD}^{2}(\mathbb{P}, \mathbb{Q}) MMD 2 ( P , Q ) P \mathbb{P} P Q \mathbb{Q} Q
Numerical integration: suppose we wish to approximate an intractable integral ∫ f ( x ) Q ( d x ) \int f(x)\mathbb{Q}(\mathrm{d}x) ∫ f ( x ) Q ( d x ) Q \mathbb{Q} Q { x i } i = 1 N \{x_i\}_{i=1}^{N} { x i } i = 1 N { w i } i = 1 N \{w_i\}_{i=1}^{N} { w i } i = 1 N ∑ n = 1 N w n f ( x n ) \sum_{n=1}^{N} w_n f(x_n) ∑ n = 1 N w n f ( x n ) f f f ∑ i = 1 N w i δ x i \sum_{i=1}^{N} w_i\delta_{x_i} ∑ i = 1 N w i δ x i Q \mathbb{Q} Q w 1 = ⋯ = w N = 1 / N w_1 = \cdots= w_N = 1/N w 1 = ⋯ = w N = 1/ N
Numerical Integration
min w 1 , … , w N ∈ R N MMD 2 ( ∑ i = 1 N w i δ x i , Q ) \min_{w_1,\ldots,w_N\in\mathbb{R}^{N}} \operatorname{MMD}^{2}\left(\sum_{i=1}^{N} w_i\delta_{x_i},\mathbb{Q}\right) w 1 , … , w N ∈ R N min MMD 2 ( i = 1 ∑ N w i δ x i , Q ) Sampling, generative modelling, and coreset selection: the goal is to construct samples { x i } i = 1 N \{x_i\}_{i=1}^{N} { x i } i = 1 N 1 N ∑ i = 1 N δ x i \frac{1}{N}\sum_{i=1}^{N}\delta_{x_i} N 1 ∑ i = 1 N δ x i Q \mathbb{Q} Q
perspective, this amounts to choosing the empirical measure 1 N ∑ i = 1 N δ x i \frac{1}{N}\sum_{i=1}^{N}\delta_{x_i} N 1 ∑ i = 1 N δ x i Q \mathbb{Q} Q Q \mathbb{Q} Q Q \mathbb{Q} Q
Sampling / Generative Modelling
min x 1 , … , x N MMD 2 ( 1 N ∑ i = 1 N δ x i , Q ) \min_{x_1,\ldots,x_N}\operatorname{MMD}^{2}\left(\frac{1}{N}\sum_{i=1}^{N}\delta_{x_i},\mathbb{Q}\right) x 1 , … , x N min MMD 2 ( N 1 i = 1 ∑ N δ x i , Q ) Statistical inference: given a model family { P θ : θ ∈ Θ } \{\mathbb{P}_{\theta}:\theta\in\Theta\} { P θ : θ ∈ Θ } θ \theta θ P θ \mathbb{P}_{\theta} P θ Q \mathbb{Q} Q P θ \mathbb{P}_{\theta} P θ Q \mathbb{Q} Q
Statistical Inference
min θ ∈ Θ MMD 2 ( P θ , Q ) \min_{\theta\in\Theta}\operatorname{MMD}^{2}(\mathbb{P}_{\theta},\mathbb{Q}) θ ∈ Θ min MMD 2 ( P θ , Q ) My contributions
This thesis develops new methods and theory for MMD minimization in the three settings described above.
For numerical integration, arg min w 1 , … , w N MMD 2 ( ∑ i = 1 N w i δ x i , Q ) \operatorname*{arg\,min}_{w_1,\ldots,w_N} \operatorname{MMD}^{2}\left(\sum_{i=1}^{N} w_i\delta_{x_i},\mathbb{Q}\right) arg min w 1 , … , w N MMD 2 ( ∑ i = 1 N w i δ x i , Q )
For sampling and generative modelling, the finite-dimensional problem arg min x 1 , … , x N MMD 2 ( 1 N ∑ i = 1 N δ x i , Q ) \operatorname*{arg\,min}_{x_1,\ldots,x_N} \operatorname{MMD}^{2}\left(\frac{1}{N}\sum_{i=1}^{N}\delta_{x_i},\mathbb{Q}\right) arg min x 1 , … , x N MMD 2 ( N 1 ∑ i = 1 N δ x i , Q ) P \mathcal{P} P arg min P ∈ P MMD 2 ( P , Q ) \operatorname*{arg\,min}_{\mathbb{P}\in\mathcal{P}} \operatorname{MMD}^{2}\left(\mathbb{P},\mathbb{Q}\right) arg min P ∈ P MMD 2 ( P , Q ) P ↦ MMD 2 ( P , Q ) \mathbb{P}\mapsto\operatorname{MMD}^{2}\left(\mathbb{P},\mathbb{Q}\right) P ↦ MMD 2 ( P , Q ) Q \mathbb{Q} Q
For statistical inference, despite the growing popularity of minimum MMD estimators, comparatively little work has addressed the optimization problem, arg min θ ∈ Θ MMD 2 ( P θ , Q ) \operatorname*{arg\,min}_{\theta\in\Theta} \operatorname{MMD}^{2}\left(\mathbb{P}_{\theta},\mathbb{Q}\right) arg min θ ∈ Θ MMD 2 ( P θ , Q ) θ \theta θ
gradient-based methods can at best be expected to converge to stationary points. In Chapter 5, I propose a new descent scheme for minimizing the parametric MMD objective, and prove that it can converge to a (not necessarily unique) global minimizer under explicit assumptions. This chapter is based on a recently arXived manuscript (Kang et al., 2026).
Beyond the chapters above, I have also contributed as a primary author to the following publications or archival papers, during the course of the PhD:
Z. Chen, T. Karvonen, H. Kanagawa, F.-X. Briol, and C. Oates. Stationary MMD Points for Cubature. Proceedings of the 43rd International Conference on Machine Learning, 2026 (Chen et al., 2025b);
Z. Chen, H. Kanagawa, F.-X. Briol, C. J. Oates, and L. Mackey. Thinned Mean Field Langevin Dynamics. Proceedings of the 43rd International Conference on Machine Learning, 2026 (Chen et al., 2026);
C. Tian, B. K. Sriperumbudur, A. Gretton, and Z. Chen. Sobolev Regularized MMD Gradient Flow. arXiv, 2026 (Tian et al., 2026);
Z. Chen, R. Guo, J.-F. Ton, and Y. Liu. Conformal Counterfactual Inference under Hidden Confounding. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024 (Chen et al., 2024a);
Z. Chen, A. Nitanda, A. Gretton, and T. Suzuki. Towards a Unified Analysis of Neural Networks in Nonparametric Instrumental Variable Regression: Optimization and Generalization. arXiv, 2025 (Chen et al., 2025e);
Z. Shen, Z. Chen, D. Meunier, I. Steinwart, A. Gretton, and Z. Li. Nonparametric Instrumental Variable Regression with Observed Covariates. arXiv, 2025 (Shen et al., 2025b);
S. S. Kang, F.-X. Briol, T. Karvonen, and Z. Chen. BayesSum: Bayesian Quadrature in Discrete Spaces. Proceedings of the Second International Conference on Probabilistic Numerics, 2026 (Kang et al., 2025).
Table 1.1. Table 1.1: Frequently used function spaces and probability-measure notation.
Background
Reproducing Kernel Hilbert Spaces
Let X ⊆ R d \mathcal{X} \subseteq\mathbb{R}^{d} X ⊆ R d k : X × X → R k : \mathcal{X} \times\mathcal{X} \to\mathbb{R} k : X × X → R N ∈ N N \in\mathbb{N} N ∈ N x 1 : N ∈ X N x_{1:N} \in\mathcal{X}^{N} x 1 : N ∈ X N
k ( x 1 : N , x 1 : N ) : = [ k ( x 1 , x 1 ) ⋯ k ( x 1 , x N ) ⋮ ⋱ ⋮ k ( x N , x 1 ) ⋯ k ( x N , x N ) ] k(x_{1:N}, x_{1:N}) := \begin{bmatrix}
k(x_{1}, x_{1}) & \cdots& k(x_{1}, x_{N}) \\
\vdots& \ddots& \vdots\\
k(x_{N}, x_{1}) & \cdots& k(x_{N}, x_{N})
\end{bmatrix} k ( x 1 : N , x 1 : N ) := k ( x 1 , x 1 ) ⋮ k ( x N , x 1 ) ⋯ ⋱ ⋯ k ( x 1 , x N ) ⋮ k ( x N , x N ) is positive semi-definite. In other words, α T k ( x 1 : N , x 1 : N ) α ≥ 0 \alpha^{\mathsf{T}} k(x_{1:N}, x_{1:N}) \alpha\ge0 α T k ( x 1 : N , x 1 : N ) α ≥ 0 α ∈ R N \alpha\in\mathbb{R}^{N} α ∈ R N
The associated RKHS H k \mathcal{H}_{k} H k X \mathcal{X} X ⟨ ⋅ , ⋅ ⟩ H k \langle\cdot, \cdot\rangle_{\mathcal{H}_{k}} ⟨ ⋅ , ⋅ ⟩ H k ∥ ⋅ ∥ H k \lVert\cdot\rVert_{\mathcal{H}_{k}} ∥ ⋅ ∥ H k
For all x ∈ X x \in\mathcal{X} x ∈ X k ( x , ⋅ ) ∈ H k k(x, \cdot) \in\mathcal{H}_{k} k ( x , ⋅ ) ∈ H k
The reproducing property holds: for all h ∈ H k h \in\mathcal{H}_{k} h ∈ H k x ∈ X x \in\mathcal{X} x ∈ X h ( x ) = ⟨ h , k ( x , ⋅ ) ⟩ H k h(x) = \langle h, k(x, \cdot) \rangle_{\mathcal{H}_{k}} h ( x ) = ⟨ h , k ( x , ⋅ ) ⟩ H k
Alternatively speaking, H k \mathcal{H}_{k} H k { k ( x , ⋅ ) : x ∈ X } \{k(x, \cdot) : x \in\mathcal{X}\} { k ( x , ⋅ ) : x ∈ X } [2 Chapter 7] :
The linear kernel k ( x , y ) = x T y k(x,y)=x^{\mathsf{T}}y k ( x , y ) = x T y R d \mathbb{R}^{d} R d f ( x ) = θ T x f(x)=\theta^{\mathsf{T}}x f ( x ) = θ T x θ ∈ R d \theta\in\mathbb{R}^{d} θ ∈ R d θ \theta θ
The polynomial kernel k ( x , y ) = ( x T y + c ) p k(x,y)=(x^{\mathsf{T}}y+c)^{p} k ( x , y ) = ( x T y + c ) p p p p c > 0 c>0 c > 0 p p p c = 0 c=0 c = 0
The Gaussian kernel k ( x , y ) = exp ( − ∥ x − y ∥ 2 2 ℓ 2 ) k(x,y)=\exp\left(-\frac{\lVert x-y\rVert^{2}}{2\ell^{2}}\right) k ( x , y ) = exp ( − 2 ℓ 2 ∥ x − y ∥ 2 ) ℓ \ell ℓ
Matérn kernels of order ν \nu ν ν + d / 2 \nu+d/2 ν + d /2 ν ∈ { 1 / 2 , 3 / 2 , … } \nu\in\{1/2,3/2,\ldots\} ν ∈ { 1/2 , 3/2 , … } k k k ⌊ ν ⌋ \lfloor\nu\rfloor ⌊ ν ⌋ ρ = ∥ x − x ′ ∥ \rho=\lVert x-x'\rVert ρ = ∥ x − x ′ ∥
ν = 1 / 2 : k ν ( x , x ′ ) = τ 2 exp ( − ρ ℓ ) , ν = 3 / 2 : k ν ( x , x ′ ) = τ 2 ( 1 + 3 ρ ℓ ) exp ( − 3 ρ ℓ ) , ν = 5 / 2 : k ν ( x , x ′ ) = τ 2 ( 1 + 5 ρ ℓ + 5 ρ 2 3 ℓ 2 ) exp ( − 5 ρ ℓ ) . \begin{aligned}
\nu=1/2:\quad&k_{\nu}(x,x')=\tau^{2}\exp\left(-\frac{\rho}{\ell}\right),\\
\nu=3/2:\quad&k_{\nu}(x,x')=\tau^{2}\left(1+\frac{\sqrt{3}\rho}{\ell}\right)\exp\left(-\frac{\sqrt{3}\rho}{\ell}\right),\\
\nu=5/2:\quad&k_{\nu}(x,x')=\tau^{2}\left(1+\frac{\sqrt{5}\rho}{\ell}+\frac{5\rho^{2}}{3\ell^{2}}\right)\exp\left(-\frac{\sqrt{5}\rho}{\ell}\right).
\end{aligned} ν = 1/2 : ν = 3/2 : ν = 5/2 : k ν ( x , x ′ ) = τ 2 exp ( − ℓ ρ ) , k ν ( x , x ′ ) = τ 2 ( 1 + ℓ 3 ρ ) exp ( − ℓ 3 ρ ) , k ν ( x , x ′ ) = τ 2 ( 1 + ℓ 5 ρ + 3 ℓ 2 5 ρ 2 ) exp ( − ℓ 5 ρ ) . For an open set X ⊆ R d \mathcal{X}\subseteq\mathbb{R}^{d} X ⊆ R d α ∈ N 0 d \alpha\in\mathbb{N}_{0}^{d} α ∈ N 0 d v v v weak derivative ∂ α f \partial^{\alpha}f ∂ α f f f f
∫ X f ( x ) ∂ α φ ( x ) d x = ( − 1 ) ∣ α ∣ ∫ X v ( x ) φ ( x ) d x \int_{\mathcal{X}}f(x)\partial^{\alpha}\varphi(x)\,\mathrm{d}x=(-1)^{|\alpha|}\int_{\mathcal{X}}v(x)\varphi(x)\,\mathrm{d}x ∫ X f ( x ) ∂ α φ ( x ) d x = ( − 1 ) ∣ α ∣ ∫ X v ( x ) φ ( x ) d x for every smooth, compactly supported test function φ ∈ C c ∞ ( X ) \varphi\in C^\infty_c(\mathcal{X}) φ ∈ C c ∞ ( X )
Definition 2.1 (Sobolev space). For an integer s ≥ 0 s \ge0 s ≥ 0 X ⊆ R d \mathcal{X} \subseteq\mathbb{R}^d X ⊆ R d W 2 s ( X ) W_2^s(\mathcal{X}) W 2 s ( X ) s s s
W 2 s ( X ) : = { f ∈ L 2 ( X ) : ∂ α f ∈ L 2 ( X ) for every multi-index α with ∣ α ∣ ≤ s } . (1) W_2^s(\mathcal{X}) := \{f \in L_2(\mathcal{X}) : \partial^\alpha f \in L_2(\mathcal{X}) \text{ for every multi-index } \alpha\text{ with } |\alpha| \le s\}. \tag*{(1)} W 2 s ( X ) := { f ∈ L 2 ( X ) : ∂ α f ∈ L 2 ( X ) for every multi-index α with ∣ α ∣ ≤ s } . (1) Its norm is
∥ f ∥ W 2 s ( X ) : = ( ∑ ∣ α ∣ ≤ s ∥ ∂ α f ∥ L 2 ( X ) 2 ) 1 / 2 . \lVert f \rVert_{W_2^s(\mathcal{X})} := \left(\sum_{|\alpha|\le s} \lVert\partial^\alpha f \rVert_{L_2(\mathcal{X})}^2\right)^{1/2}. ∥ f ∥ W 2 s ( X ) := ∣ α ∣ ≤ s ∑ ∥ ∂ α f ∥ L 2 ( X ) 2 1/2 . We next review two technical constructions built from kernels and RKHSs: kernel integral operators and the maximum mean discrepancy . The kernel integral operator and its RKHS counterpart will be used to bound the regression errors underlying nested kernel quadrature in Chapter 3, and to construct the regularized discrepancy and analyse its Wasserstein gradient flow in Chapter 4. The maximum mean discrepancy will be used extensively in all the following chapters of this thesis.
Kernel integral operator: When the kernel k k k ∫ k ( x , x ) d P ( x ) < ∞ \int\sqrt{k(x,x)}\,\mathrm{d}\mathbb{P}(x) < \infty ∫ k ( x , x ) d P ( x ) < ∞ P \mathbb{P} P H \mathcal{H} H L 2 ( P ) L_2(\mathbb{P}) L 2 ( P ) ι P : H → L 2 ( P ) \iota_{\mathbb{P}} : \mathcal{H} \to L_2(\mathbb{P}) ι P : H → L 2 ( P ) f ↦ f f \mapsto f f ↦ f ι P ∗ : L 2 ( P ) → H \iota_{\mathbb{P}}^* : L_2(\mathbb{P}) \to\mathcal{H} ι P ∗ : L 2 ( P ) → H ι P ∗ f ( ⋅ ) = ∫ k ( x , ⋅ ) f ( x ) d P ( x ) \iota_{\mathbb{P}}^*f(\cdot) = \int k(x,\cdot)f(x)\mathrm{d}\mathbb{P}(x) ι P ∗ f ( ⋅ ) = ∫ k ( x , ⋅ ) f ( x ) d P ( x ) ι P \iota_{\mathbb{P}} ι P L 2 ( P ) L_2(\mathbb{P}) L 2 ( P ) T P : = ι P ι P ∗ T_{\mathbb{P}} := \iota_{\mathbb{P}}\iota_{\mathbb{P}}^* T P := ι P ι P ∗ integral operator , and a H \mathcal{H} H Σ P : = ι P ∗ ι P = ∫ k ( ⋅ , x ) ⊗ k ( ⋅ , x ) d P ( x ) \Sigma_{\mathbb{P}} := \iota_{\mathbb{P}}^*\iota_{\mathbb{P}} = \int k(\cdot,x) \otimes k(\cdot,x)\mathrm{d}\mathbb{P}(x) Σ P := ι P ∗ ι P = ∫ k ( ⋅ , x ) ⊗ k ( ⋅ , x ) d P ( x ) ( a ⊗ b ) c : = ⟨ b , c ⟩ H a (a \otimes b)c := \langle b,c\rangle_{\mathcal{H}}a ( a ⊗ b ) c := ⟨ b , c ⟩ H a a , b , c ∈ H a,b,c \in\mathcal{H} a , b , c ∈ H covariance operator . T P T_{\mathbb{P}} T P { e i } i ≥ 1 \{e_i\}_{i\ge1} { e i } i ≥ 1 L 2 ( P ) L_2(\mathbb{P}) L 2 ( P ) ϱ 1 ≥ ⋯ ϱ i ≥ ⋯ ≥ 0 \varrho_1 \ge\cdots\varrho_i \ge\cdots\ge0 ϱ 1 ≥ ⋯ ϱ i ≥ ⋯ ≥ 0
Maximum mean discrepancy: When the kernel k k k ∫ k ( x , x ) d P ( x ) < ∞ \int\sqrt{k(x,x)}\,\mathrm{d}\mathbb{P}(x) < \infty ∫ k ( x , x ) d P ( x ) < ∞ P \mathbb{P} P P \mathbb{P} P X \mathcal{X} X H k \mathcal{H}_k H k m P : = ∫ X k ( x , ⋅ ) d P ( x ) m_{\mathbb{P}} := \int_{\mathcal{X}} k(x,\cdot)\,\mathrm{d}\mathbb{P}(x) m P := ∫ X k ( x , ⋅ ) d P ( x ) E X ∼ P [ h ( X ) ] = ⟨ h , m P ⟩ H k \mathbb{E}_{X\sim\mathbb{P}}[h(X)] = \langle h,m_{\mathbb{P}}\rangle_{\mathcal{H}_k} E X ∼ P [ h ( X )] = ⟨ h , m P ⟩ H k h ∈ H k h \in\mathcal{H}_k h ∈ H k P \mathbb{P} P Q \mathbb{Q} Q
MMD ( P , Q ) : = ∥ m P − m Q ∥ H k = sup ∥ h ∥ H k ≤ 1 ∣ E P [ h ( X ) ] − E Q [ h ( Y ) ] ∣ . \operatorname{MMD}(\mathbb{P},\mathbb{Q}) := \lVert m_{\mathbb{P}} - m_{\mathbb{Q}}\rVert_{\mathcal{H}_k} = \sup_{\lVert h\rVert_{\mathcal{H}_k}\leq1} \left|\mathbb{E}_{\mathbb{P}}[h(X)]-\mathbb{E}_{\mathbb{Q}}[h(Y)]\right|. MMD ( P , Q ) := ∥ m P − m Q ∥ H k = ∥ h ∥ H k ≤ 1 sup ∣ E P [ h ( X )] − E Q [ h ( Y )] ∣ . Although the MMD is defined through a supremum, the reproducing property and the Cauchy–Schwarz inequality imply that it admits a closed-form expression in terms of kernel expectations. In particular, for independent X , X ′ ∼ P X,X' \sim\mathbb{P} X , X ′ ∼ P Y , Y ′ ∼ Q Y,Y' \sim\mathbb{Q} Y , Y ′ ∼ Q
MMD k 2 ( P , Q ) = E [ k ( X , X ′ ) ] − 2 E [ k ( X , Y ) ] + E [ k ( Y , Y ′ ) ] . \operatorname{MMD}_{k}^{2}(\mathbb{P},\mathbb{Q}) = \mathbb{E}[k(X,X')] - 2\mathbb{E}[k(X,Y)] + \mathbb{E}[k(Y,Y')]. MMD k 2 ( P , Q ) = E [ k ( X , X ′ )] − 2 E [ k ( X , Y )] + E [ k ( Y , Y ′ )] . A kernel k k k characteristic if its mean embedding is injective on the class of probability measures for which the embedding exists: m P = m Q m_{\mathbb{P}} = m_{\mathbb{Q}} m P = m Q P = Q \mathbb{P} = \mathbb{Q} P = Q MMD k ( P , Q ) = 0 \operatorname{MMD}_k(\mathbb{P},\mathbb{Q}) = 0 MMD k ( P , Q ) = 0 P = Q \mathbb{P} = \mathbb{Q} P = Q
MiniMMD for Integration
In this section, we provide background of kernel quadrature (Sommariva and Vianello, 2006; Rasmussen and Ghahramani, 2002; Briol et al., 2019b), a kernel-based Monte Carlo method derived by minimizing the MMD between the weighted empirical distribution and the integration measure.
Consider an intractable expectation I = E X ∼ Q [ h ( X ) ] = ∫ h ( x ) d Q ( x ) I = \mathbb{E}_{X\sim\mathbb{Q}}[h(X)] = \int h(x)\,\mathrm{d}\mathbb{Q}(x) I = E X ∼ Q [ h ( X )] = ∫ h ( x ) d Q ( x ) x 1 : N x_{1:N} x 1 : N N N N Q \mathbb{Q} Q I I I
I ≈ 1 N ∑ n = 1 N h ( x n ) I \approx\frac{1}{N}\sum_{n=1}^{N} h(x_n) I ≈ N 1 ∑ n = 1 N h ( x n ) h h h H k \mathcal{H}_k H k k k k
Assuming h ∈ H k h \in\mathcal{H}_k h ∈ H k E X ∼ Q [ k ( X , X ) ] < ∞ \mathbb{E}_{X \sim\mathbb{Q}}[\sqrt{k(X,X)}] < \infty E X ∼ Q [ k ( X , X ) ] < ∞ I ^ K Q = ∑ n = 1 N w n h ( x n ) \hat{I}_{\mathrm{KQ}} = \sum_{n=1}^{N} w_n h(x_n) I ^ KQ = ∑ n = 1 N w n h ( x n )
∣ I − I ^ K Q ∣ = ∣ E X ∼ Q [ h ( X ) ] − ∑ n = 1 N w n h ( x n ) ∣ ≤ ∥ h ∥ H k ∥ m Q ( ⋅ ) − ∑ n = 1 N w n k ( x n , ⋅ ) ∥ H k = ∥ h ∥ H k MMD ( ∑ i = 1 N w i δ x i , Q ) , \begin{aligned}
\left|I-\hat{I}_{\mathrm{KQ}}\right|
&=\left|\mathbb{E}_{X\sim\mathbb{Q}}[h(X)]-\sum_{n=1}^{N}w_nh(x_n)\right|\\
&\leq\|h\|_{\mathcal{H}_k}\left\|m_{\mathbb{Q}}(\cdot)-\sum_{n=1}^{N}w_nk(x_n,\cdot)\right\|_{\mathcal{H}_k}\\
&=\|h\|_{\mathcal{H}_k}\operatorname{MMD}\left(\sum_{i=1}^{N}w_i\delta_{x_i},\mathbb{Q}\right),
\end{aligned} I − I ^ KQ = E X ∼ Q [ h ( X )] − n = 1 ∑ N w n h ( x n ) ≤ ∥ h ∥ H k m Q ( ⋅ ) − n = 1 ∑ N w n k ( x n , ⋅ ) H k = ∥ h ∥ H k MMD ( i = 1 ∑ N w i δ x i , Q ) , where m Q ( ⋅ ) = E X ∼ Q [ k ( X , ⋅ ) ] m_{\mathbb{Q}}(\cdot)=\mathbb{E}_{X\sim\mathbb{Q}}[k(X,\cdot)] m Q ( ⋅ ) = E X ∼ Q [ k ( X , ⋅ )] kernel mean embedding (KME) of Q \mathbb{Q} Q H k \mathcal{H}_k H k arg min w 1 : N MMD 2 ( ∑ i = 1 N w i δ x i , Q ) \mathop{\arg\min}_{w_{1:N}}\operatorname{MMD}^{2}(\sum_{i=1}^{N}w_i\delta_{x_i},\mathbb{Q}) arg min w 1 : N MMD 2 ( ∑ i = 1 N w i δ x i , Q )
I ^ K Q : = m Q ( x 1 : N ) K − 1 h ( x 1 : N ) , (2) \hat{I}_{\mathrm{KQ}} := m_{\mathbb{Q}}(x_{1:N})K^{-1}h(x_{1:N}), \tag*{(2)} I ^ KQ := m Q ( x 1 : N ) K − 1 h ( x 1 : N ) , (2) where K = k ( x 1 : N , x 1 : N ) ∈ R N × N K=k(x_{1:N},x_{1:N})\in\mathbb{R}^{N\times N} K = k ( x 1 : N , x 1 : N ) ∈ R N × N m Q ( x 1 : N ) = ( m Q ( x 1 ) , … , m Q ( x N ) ) m_{\mathbb{Q}}(x_{1:N})=(m_{\mathbb{Q}}(x_1),\ldots,m_{\mathbb{Q}}(x_N)) m Q ( x 1 : N ) = ( m Q ( x 1 ) , … , m Q ( x N )) h ( x 1 : N ) = ( h ( x 1 ) , … , h ( x N ) ) T h(x_{1:N})=(h(x_1),\ldots,h(x_N))^{\mathsf{T}} h ( x 1 : N ) = ( h ( x 1 ) , … , h ( x N ) ) T w 1 : N = m Q ( x 1 : N ) K − 1 w_{1:N}=m_{\mathbb{Q}}(x_{1:N})K^{-1} w 1 : N = m Q ( x 1 : N ) K − 1 λ I d \lambda I_d λ I d K K K
KQ uses the structural information h ∈ H k h\in\mathcal{H}_k h ∈ H k ∣ I − I ^ K Q ∣ \left|I-\hat{I}_{\mathrm{KQ}}\right| I − I ^ KQ N → ∞ N\to\infty N → ∞ H k \mathcal{H}_k H k W 2 s ( X ) W_2^s(\mathcal{X}) W 2 s ( X ) s > d / 2 s>d/2 s > d /2 O ( N − s / d ) O(N^{-s/d}) O ( N − s / d ) O ( N − 1 / 2 ) O(N^{-1/2}) O ( N − 1/2 ) λ = 0 \lambda= 0 λ = 0 (2) and require the Gram matrix K \mathbf{K} K λ ≲ N − 2 s d ( log N ) 2 s + 2 d \lambda\lesssim N^{-\frac{2s}{d}}(\log N)^{\frac{2s+2}{d}} λ ≲ N − d 2 s ( log N ) d 2 s + 2 K \mathbf{K} K
Despite the optimality of the KQ convergence rate, the rate constant can be reduced by selecting points x 1 : N x_{1:N} x 1 : N
Compared with MC, KQ has two main drawbacks: the worst-case computational cost is O ( N 3 ) O(N^3) O ( N 3 ) m Q m_{\mathsf{Q}} m Q tbl:1 in Briol et al. (2019b) or the catalogue in Wenger et al. (2021), or replace the kernel with a Stein reproducing kernel (Oates et al., 2017, 2019; Si et al., 2021; Sun et al., 2023).
In Chapter 3, we tackle both drawbacks through a change-of-variable trick. Suppose we can find a continuous transformation Φ \Phi Φ x 1 : N = Φ ( u 1 : N ) x_{1:N} = \Phi(u_{1:N}) x 1 : N = Φ ( u 1 : N ) u 1 : N u_{1:N} u 1 : N U \mathcal{U} U ∫ U h ( Φ ( u ) ) d U ( u ) \int_{\mathcal{U}} h(\Phi(u))\mathrm{d}\mathcal{U}(u) ∫ U h ( Φ ( u )) d U ( u ) h : X → R h:\mathcal{X}\to\mathbb{R} h : X → R h ∘ Φ : U → R h\circ\Phi:\mathcal{U}\to\mathbb{R} h ∘ Φ : U → R I ^ K Q = μ U ( u 1 : N ) K U − 1 ( h ∘ Φ ) ( u 1 : N ) \widehat{I}_{\mathrm{KQ}}=\mu_{\mathcal{U}}(u_{1:N})K_{\mathcal{U}}^{-1}(h\circ\Phi)(u_{1:N}) I KQ = μ U ( u 1 : N ) K U − 1 ( h ∘ Φ ) ( u 1 : N ) K U = k U ( u 1 : N , u 1 : N ) K_{\mathcal{U}}=k_{\mathcal{U}}(u_{1:N},u_{1:N}) K U = k U ( u 1 : N , u 1 : N ) U \mathcal{U} U μ U ( u 1 : N ) K U − 1 \mu_{\mathcal{U}}(u_{1:N})K_{\mathcal{U}}^{-1} μ U ( u 1 : N ) K U − 1 O ( N ) O(N) O ( N ) Φ \Phi Φ
Before concluding, we note that the KQ estimator is often called Bayesian quadrature (B Q BQ B Q h ( x 1 : N ) h(x_{1:N}) h ( x 1 : N )
MiniMMD for Sampling, Generative Modelling and Coreset Selection
In this section, we provide background on MMD gradient flow, which is the Wasserstein gradient flow of the squared MMD discrepancy, and show how it can be used for sampling, generative modelling and coreset selection.
Wasserstein gradient flow Let F : P 2 ( R d ) → R \mathcal{F}:\mathcal{P}_{2}(\mathbb{R}^{d})\to\mathbb{R} F : P 2 ( R d ) → R π \pi π F \mathcal{F} F F ( ⋅ ) = D ( ⋅ ∥ π ) \mathcal{F}(\cdot)=\mathcal{D}(\cdot\Vert\pi) F ( ⋅ ) = D ( ⋅ ∥ π ) D \mathcal{D} D F \mathcal{F} F F : R d → R \mathcal{F}:\mathbb{R}^{d}\to\mathbb{R} F : R d → R x 0 x_{0} x 0 ( x t ) t ≥ 0 (x_{t})_{t\geq0} ( x t ) t ≥ 0 ∂ t x t = − v ( x t ) \partial_{t}x_{t}=-v(x_{t}) ∂ t x t = − v ( x t ) v v v F \mathcal{F} F
Gradient flows extend from Euclidean spaces to more general metric spaces (Ambrosio et al., 2005). When the metric space is P 2 ( R d ) \mathcal{P}_2(\mathbb{R}^d) P 2 ( R d ) μ 0 ∈ P 2 ( R d ) \mu_0 \in\mathcal{P}_2(\mathbb{R}^d) μ 0 ∈ P 2 ( R d ) ( μ t ) t ≥ 0 (\mu_t)_{t \ge0} ( μ t ) t ≥ 0 F \mathcal{F} F ( P 2 ( R d ) , W 2 ) (\mathcal{P}_2(\mathbb{R}^d), W_2) ( P 2 ( R d ) , W 2 )
∂ t μ t + ∇ ⋅ ( μ t v t ) = 0 , (3) \partial_t \mu_t + \nabla\cdot(\mu_t v_t) = 0, \tag*{(3)} ∂ t μ t + ∇ ⋅ ( μ t v t ) = 0 , (3) where v t v_t v t F \mathcal{F} F μ t \mu_t μ t μ t \mu_t μ t v t v_t v t F \mathcal{F} F t t t v t v_t v t ( μ t ) t ≥ 0 (\mu_t)_{t \ge0} ( μ t ) t ≥ 0 P 2 ( R d ) \mathcal{P}_2(\mathbb{R}^d) P 2 ( R d )
d X t = − v t ( X t ) d t , Law ( X t ) = μ t . (4) \mathrm{d}X_t = -v_t(X_t)\mathrm{d}t,\qquad\operatorname{Law}(X_t) = \mu_t. \tag*{(4)} d X t = − v t ( X t ) d t , Law ( X t ) = μ t . (4) Equations (3) and (4) give complementary viewpoints: the former is Eulerian and describes density evolution, whereas the latter is Lagrangian and follows individual particle trajectories.
In practice, WGF implementations are usually based on a finite-particle discretization of the Lagrangian formulation. Given a fixed time horizon S ∈ N S \in\mathbb{N} S ∈ N { x 0 ( i ) } i = 1 N ∼ μ 0 \{x_0^{(i)}\}_{i=1}^N \sim\mu_0 { x 0 ( i ) } i = 1 N ∼ μ 0
x s + 1 ( i ) = x s ( i ) − γ v [ μ ^ s ] ( x s ( i ) ) , μ ^ s = 1 N ∑ j = 1 N δ x s ( j ) , (5) x_{s+1}^{(i)} = x_s^{(i)} - \gamma v[\hat{\mu}_s](x_s^{(i)}),\qquad\hat{\mu}_s = \frac{1}{N}\sum_{j=1}^{N}\delta_{x_s^{(j)}}, \tag*{(5)} x s + 1 ( i ) = x s ( i ) − γ v [ μ ^ s ] ( x s ( i ) ) , μ ^ s = N 1 j = 1 ∑ N δ x s ( j ) , (5) for s ∈ [ S ] s \in[S] s ∈ [ S ] γ > 0 \gamma> 0 γ > 0 μ s + 1 = ( Id − γ v s ) # μ s \mu_{s+1} = (\operatorname{Id} - \gamma v_s)_{\#}\mu_s μ s + 1 = ( Id − γ v s ) # μ s
A constant-speed Wasserstein geodesic ( ρ t ) t ∈ [ 0 , 1 ] (\rho_t)_{t\in[0,1]} ( ρ t ) t ∈ [ 0 , 1 ] W 2 ( ρ s , ρ t ) = ∣ s − t ∣ W 2 ( ρ 0 , ρ 1 ) W_2(\rho_s,\rho_t) = |s-t|W_2(\rho_0,\rho_1) W 2 ( ρ s , ρ t ) = ∣ s − t ∣ W 2 ( ρ 0 , ρ 1 ) s , t ∈ [ 0 , 1 ] s,t \in[0,1] s , t ∈ [ 0 , 1 ] F \mathcal{F} F geodesically convex if it is convex along every such geodesic, that is, F ( ρ t ) ≤ ( 1 − t ) F ( ρ 0 ) + t F ( ρ 1 ) \mathcal{F}(\rho_t) \le(1-t)\mathcal{F}(\rho_0) + t\mathcal{F}(\rho_1) F ( ρ t ) ≤ ( 1 − t ) F ( ρ 0 ) + t F ( ρ 1 ) geodesically Λ \Lambda Λ convex if
F ( ρ t ) ≤ ( 1 − t ) F ( ρ 0 ) + t F ( ρ 1 ) − Λ 2 t ( 1 − t ) W 2 2 ( ρ 0 , ρ 1 ) . \mathcal{F}(\rho_t) \le(1-t)\mathcal{F}(\rho_0) + t\mathcal{F}(\rho_1) - \frac{\Lambda}{2}t(1-t)W_2^2(\rho_0,\rho_1). F ( ρ t ) ≤ ( 1 − t ) F ( ρ 0 ) + t F ( ρ 1 ) − 2 Λ t ( 1 − t ) W 2 2 ( ρ 0 , ρ 1 ) . This generalizes Euclidean convexity by replacing straight lines with geodesics ((Ambrosio et al., 2005; Villani et al., 2009)).
As with gradient descent in Euclidean spaces, a central tool for characterizing the convergence of Wasserstein gradient descent is the geodesic convexity and smoothness of the functional F \mathcal{F} F F \mathcal{F} F ϕ ∈ C c ∞ ( R d ) \phi\in C^\infty_c(\mathbb{R}^d) ϕ ∈ C c ∞ ( R d ) 1 μ \mu μ ρ t = ( Id + t ∇ ϕ ) # μ \rho_t = (\operatorname{Id} + t\nabla\phi)_{\#}\mu ρ t = ( Id + t ∇ ϕ ) # μ 0 ≤ t ≤ 1 0 \le t \le1 0 ≤ t ≤ 1 F : P 2 ( R d ) → R \mathcal{F} : \mathcal{P}_2(\mathbb{R}^d) \to\mathbb{R} F : P 2 ( R d ) → R μ \mu μ Hess F ∣ μ \operatorname{Hess}\mathcal{F}|_\mu Hess F ∣ μ L 2 ( μ ) L^2(\mu) L 2 ( μ ) L 2 ( μ ) L^2(\mu) L 2 ( μ ) 2
⟨ Hess F ∣ μ ∇ ϕ , ∇ ϕ ⟩ L 2 ( μ ) = d 2 d t 2 ∣ t = 0 F ( ρ t ) . \left\langle\operatorname{Hess}\mathcal{F}|_\mu\nabla\phi,\nabla\phi\right\rangle_{L^2(\mu)} = \left.\frac{\mathrm{d}^2}{\mathrm{d}t^2}\right|_{t=0}\mathcal{F}(\rho_t). ⟨ Hess F ∣ μ ∇ ϕ , ∇ ϕ ⟩ L 2 ( μ ) = d t 2 d 2 t = 0 F ( ρ t ) . A functional F \mathcal{F} F geodesically M M M smooth at μ \mu μ
⟨ Hess F ∣ μ ∇ ϕ , ∇ ϕ ⟩ L 2 ( μ ) ≤ M ∥ ∇ ϕ ∥ L 2 ( μ ) 2 , \left\langle\operatorname{Hess}\mathcal{F}|_\mu\nabla\phi,\nabla\phi\right\rangle_{L^2(\mu)} \le M\|\nabla\phi\|_{L^2(\mu)}^2, ⟨ Hess F ∣ μ ∇ ϕ , ∇ ϕ ⟩ L 2 ( μ ) ≤ M ∥∇ ϕ ∥ L 2 ( μ ) 2 , and geodesically Λ \Lambda Λ μ \mu μ
⟨ Hess F ∣ μ ∇ φ , ∇ φ ⟩ L 2 ( μ ) ≥ Λ ∥ ∇ φ ∥ L 2 ( μ ) 2 . \left\langle\operatorname{Hess}\mathcal{F}|_{\mu}\nabla\varphi,\nabla\varphi\right\rangle_{L^{2}(\mu)} \geq\Lambda\lVert\nabla\varphi\rVert_{L^{2}(\mu)}^{2}. ⟨ Hess F ∣ μ ∇ φ , ∇ φ ⟩ L 2 ( μ ) ≥ Λ ∥ ∇ φ ∥ L 2 ( μ ) 2 . Additionally, F \mathcal{F} F geodesically semiconvex if − ∞ < Λ < 0 -\infty< \Lambda< 0 − ∞ < Λ < 0 geodesically strongly convex if Λ > 0 \Lambda> 0 Λ > 0
MMD and KSD gradient flows Given a target probability measure π ∈ P 2 ( R d ) \pi\in\mathcal{P}_{2}(\mathbb{R}^{d}) π ∈ P 2 ( R d ) F M M D ( ⋅ ) = 1 2 MMD 2 ( ⋅ ∥ π ) \mathcal{F}_{\mathrm{MMD}}(\cdot) = \frac{1}{2}\operatorname{MMD}^{2}(\cdot\lVert\pi) F MMD ( ⋅ ) = 2 1 MMD 2 ( ⋅ ∥ π )
arg min P ∈ P 2 ( R d ) MMD 2 ( P , Q ) . \underset{\mathbb{P}\in\mathcal{P}_{2}(\mathbb{R}^{d})}{\operatorname{arg\,min}}\ \operatorname{MMD}^{2}(\mathbb{P},\mathbb{Q}). P ∈ P 2 ( R d ) arg min MMD 2 ( P , Q ) . The MMD (Gretton et al., 2012) between two probability measures is the RKHS norm of the difference between their kernel mean embeddings.
For F M M D ( μ ) = 1 2 MMD 2 ( μ , π ) \mathcal{F}_{\mathrm{MMD}}(\mu) = \frac{1}{2}\operatorname{MMD}^{2}(\mu,\pi) F MMD ( μ ) = 2 1 MMD 2 ( μ , π )
v M M D [ μ ] ( ⋅ ) = ∫ ∇ 2 k ( x , ⋅ ) d ( μ − π ) ( x ) . v_{\mathrm{MMD}}[\mu](\cdot) = \int\nabla_{2}k(x,\cdot)\,\mathrm{d}(\mu-\pi)(x). v MMD [ μ ] ( ⋅ ) = ∫ ∇ 2 k ( x , ⋅ ) d ( μ − π ) ( x ) . This makes the MMD flow well suited to generative modeling because v M M D [ μ ] v_{\mathrm{MMD}}[\mu] v MMD [ μ ] μ \mu μ π \pi π M M M { y ( i ) } i = 1 M \{y^{(i)}\}_{i=1}^{M} { y ( i ) } i = 1 M π \pi π N N N { x s ( j ) } j = 1 N \{x_{s}^{(j)}\}_{j=1}^{N} { x s ( j ) } j = 1 N s s s
v M M D [ μ ^ s ] ( ⋅ ) ≈ 1 N ∑ j = 1 N ∇ 2 k ( x s ( j ) , ⋅ ) − 1 M ∑ j = 1 M ∇ 2 k ( y ( j ) , ⋅ ) . v_{\mathrm{MMD}}[\hat{\mu}_{s}](\cdot) \approx\frac{1}{N}\sum_{j=1}^{N}\nabla_{2}k(x_{s}^{(j)},\cdot)-\frac{1}{M}\sum_{j=1}^{M}\nabla_{2}k(y^{(j)},\cdot). v MMD [ μ ^ s ] ( ⋅ ) ≈ N 1 j = 1 ∑ N ∇ 2 k ( x s ( j ) , ⋅ ) − M 1 j = 1 ∑ M ∇ 2 k ( y ( j ) , ⋅ ) . The MMD flow and closely related MMD-based objectives have therefore been used to construct generative modeling algorithms, including MMD-GAN variants (Li et al., 2017; Bi’nkowski et al., 2018; Arbel et al., 2018), sample-based MMD flows (Arbel et al., 2019), Riesz-kernel and sliced MMD flows (Hertrich et al., 2023, 2024b,a), and adaptive-kernel flows based on learned features (Galashov et al., 2025a). However, the Wasserstein Hessian of F M M D \mathcal{F}_{\mathrm{MMD}} F MMD
The kernel Stein discrepancy (KSD) is a special instance of MMD under a target-dependent kernel k π k_{\pi} k π ∫ k π ( x , y ) d π ( x ) = 0 \int k_{\pi}(x,y)\mathrm{d}\pi(x)=0 ∫ k π ( x , y ) d π ( x ) = 0 y ∈ R d y\in\mathbb{R}^{d} y ∈ R d s ( x ) = ∇ log π ( x ) \boldsymbol{s}(x)=\nabla\log\pi(x) s ( x ) = ∇ log π ( x ) k : R d × R d → R k:\mathbb{R}^{d}\times\mathbb{R}^{d}\to\mathbb{R} k : R d × R d → R
k π ( x , y ) = s ( x ) T s ( y ) k ( x , y ) + s ( x ) T ∇ 2 k ( x , y ) + ∇ 1 k ( x , y ) T s ( y ) + Tr ( ∇ 1 ∇ 2 k ( x , y ) ) . (6) k_{\pi}(x,y)=\boldsymbol{s}(x)^{\mathsf{T}}\boldsymbol{s}(y)k(x,y)+\boldsymbol{s}(x)^{\mathsf{T}}\nabla_{2}k(x,y)
+\nabla_{1}k(x,y)^{\mathsf{T}}\boldsymbol{s}(y)+\operatorname{Tr}(\nabla_{1}\nabla_{2}k(x,y)). \tag*{(6)} k π ( x , y ) = s ( x ) T s ( y ) k ( x , y ) + s ( x ) T ∇ 2 k ( x , y ) + ∇ 1 k ( x , y ) T s ( y ) + Tr ( ∇ 1 ∇ 2 k ( x , y )) . (6) Under suitable tail conditions on the base kernel, the Stein identity makes the KSD flow particularly suitable for sampling from a target π \pi π s s s
v K S D [ μ ^ s ] ( ⋅ ) = 1 N ∑ j = 1 N ∇ 2 k π ( x s ( j ) , ⋅ ) . v_{\mathrm{KSD}}[\hat{\mu}_{s}](\cdot)=\frac{1}{N}\sum_{j=1}^{N}\nabla_{2}k_{\pi}(x_{s}^{(j)},\cdot). v KSD [ μ ^ s ] ( ⋅ ) = N 1 j = 1 ∑ N ∇ 2 k π ( x s ( j ) , ⋅ ) . MiniMMD for Statistical Inference
In this section, we provide background on minimum MMD inference, which is a novel statistical inference method using squared MMD as the discrepancy measure (Briol et al., 2019a; Alquier and Gerber, 2025; Chérief-Abdellatif and (Alquier, 2020; Chérief-Abdellatif and Näf, 2025).
Parameter estimation seeks to recover the unknown parameters governing a data-generating mechanism from observed data. Given a parametric family of statistical models { P θ : θ ∈ Θ } \{P_\theta: \theta\in\Theta\} { P θ : θ ∈ Θ } Θ ⊆ R p \Theta\subseteq\mathbb{R}^p Θ ⊆ R p Q Q Q
Recently, a growing body of work has considered using the maximum mean discrepancy (MMD), a metric based on embeddings of probability measures into a reproducing kernel Hilbert space (Gretton et al., 2012; Muandet et al., 2017), for minimum-distance estimation (Briol et al., 2019a). Minimum MMD estimators have been studied in simulator-based statistical inference (Briol et al., 2019a; Niu et al., 2023b; Bharti et al., 2023a), robust and generalized Bayesian inference (Chérief-Abdellatif and Alquier, 2020; Dellaporta et al., 2022), parametric regression and covariate-shift settings (Chérief-Abdellatif and Alquier, 2020), and goodness-of-fit or two-sample testing through MMD-based discrepancy statistics (Gretton et al., 2012; Muandet et al., 2017). The appeal of the method comes from the particular structure of the MMD objective, which we recall next.
Given observations Y 1 , … , Y m ∼ Q Y_1,\ldots,Y_m \sim Q Y 1 , … , Y m ∼ Q Q ^ m = m − 1 ∑ i = 1 m δ Y i \widehat{Q}_m = m^{-1}\sum_{i=1}^{m}\delta_{Y_i} Q m = m − 1 ∑ i = 1 m δ Y i { P θ : θ ∈ Θ } \{P_\theta: \theta\in\Theta\} { P θ : θ ∈ Θ }
θ ^ m ∈ arg min θ ∈ Θ MMD 2 ( P θ , Q ^ m ) . \widehat{\theta}_m \in\mathop{\arg\min}_{\theta\in\Theta}\operatorname{MMD}^2(P_\theta,\widehat{Q}_m). θ m ∈ arg min θ ∈ Θ MMD 2 ( P θ , Q m ) . This parallels maximum likelihood in that both procedures choose a model distribution closest to the data, but the discrepancy is now the RKHS distance between kernel mean embeddings rather than the Kullback–Leibler divergence.
The minimum MMD estimator enjoys the following two desirable properties:
Likelihood-free computation. For simulator-based models of the form P θ = ( G θ ) # ρ P_{\theta} = (G_{\theta})_{\#}\rho P θ = ( G θ ) # ρ X i = G θ ( U i ) X_i = G_{\theta}(U_i) X i = G θ ( U i ) U i ∼ ρ U_i \sim\rho U i ∼ ρ k ( X , X ′ ) k(X, X') k ( X , X ′ ) k ( X , Y ) k(X, Y) k ( X , Y ) k ( Y , Y ′ ) k(Y, Y') k ( Y , Y ′ ) p θ ( y ) p_{\theta}(y) p θ ( y ) ∇ y log p θ ( y ) \nabla_y \log p_{\theta}(y) ∇ y log p θ ( y )
Robustness. If k k k Q ^ m \widehat{Q}_m Q m k ( x , y ) = exp ( − ∥ x − y ∥ 2 / ( 2 σ 2 ) ) k(x,y) = \exp(-\lVert x-y\rVert^2/(2\sigma^2)) k ( x , y ) = exp ( − ∥ x − y ∥ 2 / ( 2 σ 2 )) k ( x , y ) = exp ( − ∥ x − y ∥ / σ ) k(x,y) = \exp(-\lVert x-y\rVert/\sigma) k ( x , y ) = exp ( − ∥ x − y ∥ / σ ) sup x , y ∣ k ( x , y ) ∣ < ∞ \sup_{x,y} \lvert k(x,y)\rvert< \infty sup x , y ∣ k ( x , y )∣ < ∞
Chapter 3
Minimum MMD for Integration in Nested Expectations
The results in this chapter were published in the following paper:
Zonghao Chen, Masha Naslidnyk, and François-Xavier Briol. Nested Expectations with Kernel Quadrature. International Conference of Machine Learning, 2025.
Introduction
Many decisions under uncertainty require a nonlinear assessment of an expected outcome, averaged over uncertain scenarios. For example, Bayesian experimental design averages the information gained from possible observations, while lookahead Bayesian optimisation evaluates the benefit of a future experiment after averaging over its possible outcomes. These calculations involve nested expectations: an inner average describes the outcome within a scenario, and an outer average accounts for uncertainty across scenarios. Evaluating them efficiently is important because they are often repeated while comparing candidate decisions, and each inner evaluation may require an expensive simulation or model fit.
We consider the computational task of estimating such a nested expectation, which is the expectation of a function that itself depends on another unknown conditional expectation. More precisely, let Q \mathbb{Q} Q q q q Θ \Theta Θ P θ \mathbb{P}_{\theta} P θ p θ p_{\theta} p θ X ⊆ R d X \mathcal{X} \subseteq\mathbb{R}^{d_X} X ⊆ R d X θ ∈ Θ ⊆ R d Θ \theta\in\Theta\subseteq\mathbb{R}^{d_{\Theta}} θ ∈ Θ ⊆ R d Θ f : R → R f : \mathbb{R} \to\mathbb{R} f : R → R g : X × Θ → R g : \mathcal{X} \times\Theta\to\mathbb{R} g : X × Θ → R
I : = E θ ∼ Q [ f ( E X ∼ P θ [ g ( X , θ ) ] ) ] = ∫ Θ f ( ∫ X g ( x , θ ) p θ ( x ) d x ⏟ inner conditional expectation ) q ( θ ) d θ . (7) \begin{aligned}
I &:= \mathbb{E}_{\theta\sim\mathbb{Q}}\left[f\left(\mathbb{E}_{X \sim\mathbb{P}_{\theta}}[g(X,\theta)]\right)\right] \\
&= \int_{\Theta} f\left(\underbrace{\int_{\mathcal{X}} g(x,\theta)p_{\theta}(x)\,dx}_{\text{inner conditional expectation}}\right)q(\theta)\,d\theta. \tag*{(7)}
\end{aligned} I := E θ ∼ Q [ f ( E X ∼ P θ [ g ( X , θ )] ) ] = ∫ Θ f inner conditional expectation ∫ X g ( x , θ ) p θ ( x ) d x q ( θ ) d θ . (7) Nested expectations arise within a wide range of tasks, such as the computation of objectives in Bayesian experimental design Beck et al. (2020); Goda et al. (2020); Rainforth et al. (2024), of acquisition functions in active learning and Bayesian optimisation Ginsbourger and Le Riche (2010); Yang et al. (2024), of objectives in distributionally-robust optimisation Shapiro et al. (2023); Bariletto and Ho (2024); Dellaporta et al. (2024), and of statistical divergences (Song et al., 2020; Kanagawa et al., 2023). Computing nested expectations is also a key task beyond machine learning, including in fields ranging from simulation-based uncertainty quantification Gordy and Juneja (2010); Giles and Haji-Ali (2019) to manufacturing Andradóttir and Glynn (2016) and geology Goda et al. (2018).
The estimation of nested expectations is particularly challenging since there are two levels of intractability: the inner conditional expectation, and the outer expectation, both of which must be approximated accurately in order to approximate the nested expectation I I I nested Monte Carlo (NMC) Lee and Glynn (2003); Hong and Juneja (2009); Rainforth et al. (2018). It approximates the inner and outer expectations using Monte Carlo estimators with N N N T T T O ( Δ − 3 ) \mathcal{O}(\Delta^{-3}) O ( Δ − 3 ) O ( Δ − 4 ) \mathcal{O}(\Delta^{-4}) O ( Δ − 4 ) g g g Δ \Delta Δ Δ = 0.01 \Delta= 0.01 Δ = 0.01 g g g
This issue has led to the development of a number of methods aiming to reduce the cost. Bartuska et al. (2023) proposed replacing the Monte Carlo estimators with quasi-Monte Carlo (QMC) Dick et al. (2013). This algorithm, called nested QMC (NQMC) , requires only O ( Δ − 2.5 ) \mathcal{O}(\Delta^{-2.5}) O ( Δ − 2.5 ) Δ \Delta Δ Δ = 0.01 \Delta= 0.01 Δ = 0.01 f f f multi-level Monte Carlo (MLMC) and showed that this can further reduce the number of function evaluations to O ( Δ − 2 ) \mathcal{O}(\Delta^{-2}) O ( Δ − 2 ) Δ = 0.01 \Delta= 0.01 Δ = 0.01 f f f g g g f f f g g g
To fill this gap in the literature, we propose a novel algorithm called nested kernel quadrature (NKQ) , which is presented in § 5.2 . NKQ replaces the inner and outer MC estimators of NMC with kernel quadrature (KQ) estimators (Sommariva and Vianello, 2006). We show in § 5.3 that NKQ requires only O ~ ( Δ − d X s X − d Θ s Θ ) \widetilde{\mathcal{O}}(\Delta^{-\frac{d_{\mathcal{X}}}{s_{\mathcal{X}}}-\frac{d_{\Theta}}{s_{\Theta}}}) O ( Δ − s X d X − s Θ d Θ ) Δ \Delta Δ O ~ \widetilde{\mathcal{O}} O O \mathcal{O} O s X s_{\mathcal{X}} s X s Θ s_{\Theta} s Θ f f f g g g X \mathcal{X} X Θ \Theta Θ s X > d X / 2 s_{\mathcal{X}} > d_{\mathcal{X}}/2 s X > d X /2 s Θ > d Θ / 2 s_{\Theta} > d_{\Theta}/2 s Θ > d Θ /2 O ( Δ − 4 ) \mathcal{O}(\Delta^{-4}) O ( Δ − 4 ) O ( Δ − 2 ) O(\Delta^{-2}) O ( Δ − 2 ) Δ = 0.01 \Delta= 0.01 Δ = 0.01 § 5.4 , where we show that NKQ can provide significant accuracy gains in synthetic benchmarks and Bayesian optimisation. Moreover, we show that NKQ can be combined with QMC and MLMC, providing an avenue to further accelerate convergence.
Nested Kernel Quadrature
We can now present our novel algorithm: nested kernel quadrature (NKQ). To simplify the formulas, we write
J ( θ ) : = E X ∼ P θ [ g ( X , θ ) ] , F ( θ ) : = f ( J ( θ ) ) , (8) J(\theta) := \mathbb{E}_{X \sim P_\theta}[g(X,\theta)], \qquad F(\theta) := f(J(\theta)), \tag*{(8)} J ( θ ) := E X ∼ P θ [ g ( X , θ )] , F ( θ ) := f ( J ( θ )) , (8) so that the nested expectation in (7) can be written as I = E θ ∼ Q [ F ( θ ) ] I = \mathbb{E}_{\theta\sim Q}[F(\theta)] I = E θ ∼ Q [ F ( θ )]
θ 1 : T : = [ θ 1 , … , θ T ] T ∈ Θ T , x 1 : N ( t ) : = [ x 1 ( t ) , … , x N ( t ) ] T ∈ X N , g ( x 1 : N ( t ) , θ t ) : = [ g ( x 1 ( t ) , θ t ) , … , g ( x N ( t ) , θ t ) ] ∈ R N , \begin{aligned}
\theta_{1:T} &:= [\theta_1,\ldots,\theta_T]^\mathsf{T} \in\Theta^T,\\
x_{1:N}^{(t)} &:= [x_1^{(t)},\ldots,x_N^{(t)}]^\mathsf{T} \in\mathcal{X}^N,\\
g(x_{1:N}^{(t)},\theta_t) &:= [g(x_1^{(t)},\theta_t),\ldots,g(x_N^{(t)},\theta_t)] \in\mathbb{R}^N,
\end{aligned} θ 1 : T x 1 : N ( t ) g ( x 1 : N ( t ) , θ t ) := [ θ 1 , … , θ T ] T ∈ Θ T , := [ x 1 ( t ) , … , x N ( t ) ] T ∈ X N , := [ g ( x 1 ( t ) , θ t ) , … , g ( x N ( t ) , θ t )] ∈ R N , for all t ∈ { 1 , … , T } t \in\{1,\ldots,T\} t ∈ { 1 , … , T } f f f § 5.3 . Using the same number of function evaluations N N N θ t \theta_t θ t
Stage I For each t ∈ { 1 , … , T } t \in\{1,\ldots,T\} t ∈ { 1 , … , T } J J J θ t \theta_t θ t N N N x 1 : N ( t ) x_{1:N}^{(t)} x 1 : N ( t ) g ( x 1 : N ( t ) , θ t ) g(x_{1:N}^{(t)},\theta_t) g ( x 1 : N ( t ) , θ t )
J ^ K Q ( θ t ) : = μ P θ t ( x 1 : N ( t ) ) ( K X ( t ) + N λ X I N ) − 1 g ( x 1 : N ( t ) , θ t ) . (9) \hat{J}_{\mathrm{KQ}}(\theta_t) := \mu_{P_{\theta_t}}\left(x_{1:N}^{(t)}\right)\left(\mathbf{K}_{\mathcal{X}}^{(t)} + N\lambda_{\mathcal{X}}\mathbf{I}_N\right)^{-1}g\left(x_{1:N}^{(t)},\theta_t\right). \tag*{(9)} J ^ KQ ( θ t ) := μ P θ t ( x 1 : N ( t ) ) ( K X ( t ) + N λ X I N ) − 1 g ( x 1 : N ( t ) , θ t ) . (9) Here k X k_{\mathcal{X}} k X X \mathcal{X} X μ P θ t ( ⋅ ) = E X ∼ P θ t [ k X ( X , ⋅ ) ] \mu_{P_{\theta_t}}(\cdot) = \mathbb{E}_{X\sim P_{\theta_t}}[k_{\mathcal{X}}(X,\cdot)] μ P θ t ( ⋅ ) = E X ∼ P θ t [ k X ( X , ⋅ )] P θ t P_{\theta_t} P θ t K X ( t ) = k X ( x 1 : N ( t ) , x 1 : N ( t ) ) \mathbf{K}_{\mathcal{X}}^{(t)} = k_{\mathcal{X}}(x_{1:N}^{(t)},x_{1:N}^{(t)}) K X ( t ) = k X ( x 1 : N ( t ) , x 1 : N ( t ) ) N × N N \times N N × N k X k_{\mathcal{X}} k X t ∈ { 1 , … , T } t \in\{1,\ldots,T\} t ∈ { 1 , … , T } f f f F ^ K Q ( θ t ) = f ( J ^ K Q ( θ t ) ) \hat{F}_{\mathrm{KQ}}(\theta_t) = f(\hat{J}_{\mathrm{KQ}}(\theta_t)) F ^ KQ ( θ t ) = f ( J ^ KQ ( θ t ))
Stage II We use a KQ estimator to approximate the outer expectation using the output of Stage I:
I ^ N K Q : = μ Q ( θ 1 : T ) ( K Θ + T λ Θ I T ) − 1 F ^ K Q ( θ 1 : T ) . (10) \hat{I}_{\mathrm{NKQ}} := \mu_Q(\theta_{1:T})\left(\mathbf{K}_{\Theta} + T\lambda_{\Theta}\mathbf{I}_T\right)^{-1}\hat{F}_{\mathrm{KQ}}(\theta_{1:T}). \tag*{(10)} I ^ NKQ := μ Q ( θ 1 : T ) ( K Θ + T λ Θ I T ) − 1 F ^ KQ ( θ 1 : T ) . (10) Here k Θ k_{\Theta} k Θ Θ \Theta Θ μ Q = E θ ∼ Q [ k Θ ( θ , ⋅ ) ] \mu_Q = \mathbb{E}_{\theta\sim Q}[k_{\Theta}(\theta,\cdot)] μ Q = E θ ∼ Q [ k Θ ( θ , ⋅ )] Q Q Q K Θ = k Θ ( θ 1 : T , θ 1 : T ) \mathbf{K}_{\Theta} = k_{\Theta}(\theta_{1:T},\theta_{1:T}) K Θ = k Θ ( θ 1 : T , θ 1 : T ) T × T T \times T T × T
Combining stage I and II, NKQ can be expressed in a single equation as a nesting of two quadrature rules:
I ^ N K Q = ∑ t = 1 T w t Θ f ( ∑ n = 1 N w n , t X g ( x n ( t ) , θ t ) ) , (11) \hat{I}_{\mathrm{NKQ}} = \sum_{t=1}^{T}w_t^{\Theta}f\left(\sum_{n=1}^{N}w_{n,t}^{\mathcal{X}}g(x_n^{(t)},\theta_t)\right), \tag*{(11)} I ^ NKQ = t = 1 ∑ T w t Θ f ( n = 1 ∑ N w n , t X g ( x n ( t ) , θ t ) ) , (11) where w 1 , t X , … , w N , t X w_{1,t}^{\mathcal{X}},\ldots,w_{N,t}^{\mathcal{X}} w 1 , t X , … , w N , t X J ^ K Q ( θ t ) \hat{J}_{\mathrm{KQ}}(\theta_t) J ^ KQ ( θ t ) w 1 Θ , … , w T Θ w_1^{\Theta},\ldots,w_T^{\Theta} w 1 Θ , … , w T Θ λ X = λ Θ = 0 \lambda_{\mathcal{X}} = \lambda_{\Theta} = 0 λ X = λ Θ = 0 f f f
Note that NMC can be recovered by taking all stage I weights to be 1 / N 1/N 1/ N 1 / T 1/T 1/ T § 5.3 ) and empirical performances (§ 5.4 ).
NKQ inherits the two main drawbacks of KQ. Firstly, solving the linear systems to obtain the stage I and II weights has a worst-case computational complexity of O ( T N 3 + T 3 ) O(TN^3+T^3) O ( T N 3 + T 3 ) μ P θ t \mu_{P_{\theta_t}} μ P θ t t ∈ { 1 , … , T } t \in\{1,\ldots,T\} t ∈ { 1 , … , T } μ Q \mu_Q μ Q O ( T N + T ) O(TN+T) O ( T N + T )
NKQ requires the selection of hyperparameters, including for the kernels in both stage I and II. We typically take k X k_X k X k Θ k_\Theta k Θ f f f g g g Theorem 3.1 ; see § 5.3 for details). This leaves us with a choice of kernel hyperparameters which include lengthscales γ X , γ Θ > 0 \gamma_X,\gamma_\Theta> 0 γ X , γ Θ > 0 A X , A Θ > 0 A_X,A_\Theta> 0 A X , A Θ > 0 λ X = λ 0 , X N − 2 s X d X ( log N ) 2 s X + 2 d X \lambda_X = \lambda_{0,X}N^{-\frac{2s_X}{d_X}}(\log N)^{\frac{2s_X+2}{d_X}} λ X = λ 0 , X N − d X 2 s X ( log N ) d X 2 s X + 2 λ Θ = λ 0 , Θ T − 2 s Θ d Θ ( log T ) 2 s Θ + 2 d Θ \lambda_\Theta= \lambda_{0,\Theta}T^{-\frac{2s_\Theta}{d_\Theta}}(\log T)^{\frac{2s_\Theta+2}{d_\Theta}} λ Θ = λ 0 , Θ T − d Θ 2 s Θ ( log T ) d Θ 2 s Θ + 2 Theorem 3.1 , where λ 0 , X , λ 0 , Θ \lambda_{0,X},\lambda_{0,\Theta} λ 0 , X , λ 0 , Θ { 0.01 , 0.1 , 1.0 } \{0.01, 0.1, 1.0\} { 0.01 , 0.1 , 1.0 } A X = A Θ = 1 A_X = A_\Theta= 1 A X = A Θ = 1
Before presenting our theoretical results, we briefly comment on the connection with existing KQ methods. If we could evaluate the exact expression for the inner conditional expectation J ( θ ) J(\theta) J ( θ ) I I I I ˉ K Q = μ Q ( θ 1 : T ) ( K Θ + T λ Θ I T ) − 1 F ( θ 1 : T ) \bar{I}_{\mathrm{KQ}} = \mu_Q(\theta_{1:T})(K_\Theta+ T\lambda_\Theta I_T)^{-1}F(\theta_{1:T}) I ˉ KQ = μ Q ( θ 1 : T ) ( K Θ + T λ Θ I T ) − 1 F ( θ 1 : T ) (10) , NKQ can thus be seen as KQ with noisy function values F ^ K Q ( θ 1 : T ) \widehat{F}_{\mathrm{KQ}}(\theta_{1:T}) F KQ ( θ 1 : T ) F ( θ 1 : T ) F(\theta_{1:T}) F ( θ 1 : T ) (10) ). Although it is proved in Cai et al. (2023) that noisy observations make KQ converge at a slower rate, we prove that the stage II observation noise is of the same order as the stage I error, and consequently, we can still treat stage II KQ as noiseless kernel ridge regression and the additional error caused by the stage II observation noise would be subsumed by the stage I error (See Theorem 3.3). NKQ is also closely related to a family of regression-based methods for estimating conditional expectations Longstaff and Schwartz (2001); Chen et al. (2024c). Indeed, with a slight modification of Stage II in (10) , we can obtain an estimator of J ( θ ) J(\theta) J ( θ ) conditional kernel quadrature (CKQ)
J ^ C K Q ( θ ) : = k Θ ( θ , θ 1 : T ) ( K Θ + T λ Θ I T ) − 1 J ^ K Q ( θ 1 : T ) . (12) \widehat{J}_{\mathrm{CKQ}}(\theta) := k_{\Theta}(\theta,\theta_{1:T})(K_{\Theta}+T\lambda_{\Theta}I_{T})^{-1}\widehat{J}_{\mathrm{KQ}}(\theta_{1:T}). \tag*{(12)} J CKQ ( θ ) := k Θ ( θ , θ 1 : T ) ( K Θ + T λ Θ I T ) − 1 J KQ ( θ 1 : T ) . (12) CKQ highly resembles conditional BQ (CBQ) (Chen et al., 2024c); the difference is in stage II, where CBQ uses heteroskedastic Gaussian process regression whilst CKQ uses kernel ridge regression. Interestingly, the proof in this paper leads to a much better rate for CKQ than the best known rate for CBQ (see Theorem 3.4).
Change of Variable Trick for Kernel Quadrature
As discussed above, the two major bottlenecks of KQ/NKQ are:
The need for a closed-form KME E X ∼ P [ k ( X , x ) ] \mathbb{E}_{X\sim\mathbb{P}}[k(X,x)] E X ∼ P [ k ( X , x )]
The O ( N 3 ) \mathcal{O}(N^{3}) O ( N 3 ) k ( x 1 : N , x 1 : N ) k(x_{1:N},x_{1:N}) k ( x 1 : N , x 1 : N )
The “change of variable” trick can address both challenges. Here, we only present the idea for KQ but the same holds for NKQ in both stages.
The integral of interest is I = ∫ X h ( x ) P ( d x ) I=\int_{\mathcal{X}}h(x)\mathbb{P}(dx) I = ∫ X h ( x ) P ( d x ) Φ \Phi Φ X = Φ ( U ) ∼ P X=\Phi(U)\sim\mathbb{P} X = Φ ( U ) ∼ P U ∼ U U\sim\mathbb{U} U ∼ U U = [ 0 , 1 ] d \mathcal{U}=[0,1]^{d} U = [ 0 , 1 ] d Φ \Phi Φ P \mathbb{P} P I I I I = ∫ U h ( Φ ( u ) ) d U ( u ) I=\int_{\mathcal{U}}h(\Phi(u))\mathrm{d}\mathbb{U}(u) I = ∫ U h ( Φ ( u )) d U ( u ) h : X → R h:\mathcal{X}\to\mathbb{R} h : X → R h ∘ Φ : U → R h\circ\Phi:\mathcal{U}\to\mathbb{R} h ∘ Φ : U → R
I ^ K Q = E U ∼ U [ k U ( U , u 1 : N ) ] ( k U ( u 1 : N , u 1 : N ) + N λ I N ) − 1 ( h ∘ Φ ) ( u 1 : N ) . \widehat{I}_{\mathrm{KQ}}=\mathbb{E}_{U\sim\mathbb{U}}[k_{\mathcal{U}}(U,u_{1:N})](k_{\mathcal{U}}(u_{1:N},u_{1:N})+N\lambda I_{N})^{-1}(h\circ\Phi)(u_{1:N}). I KQ = E U ∼ U [ k U ( U , u 1 : N )] ( k U ( u 1 : N , u 1 : N ) + N λ I N ) − 1 ( h ∘ Φ ) ( u 1 : N ) . Here u 1 : N \boldsymbol{u}_{1:N} u 1 : N U \mathcal{U} U k U k_{\mathcal{U}} k U U \mathcal{U} U U \mathcal{U} U k U ( u 1 : N , u 1 : N ) k_{\mathcal{U}}(u_{1:N},u_{1:N}) k U ( u 1 : N , u 1 : N ) E U ∼ U [ k U ( U , u 1 : N ) ] \mathbb{E}_{U\sim\mathcal{U}}[k_{\mathcal{U}}(U,u_{1:N})] E U ∼ U [ k U ( U , u 1 : N )] h h h Φ \Phi Φ w 1 : N K Q = E U ∼ U [ k U ( U , u 1 : N ) ] ( k U ( u 1 : N , u 1 : N ) + N λ I d N ) − 1 w_{1:N}^{\mathrm{KQ}}=\mathbb{E}_{U\sim\mathcal{U}}[k_{\mathcal{U}}(U,u_{1:N})]\left(k_{\mathcal{U}}(u_{1:N},u_{1:N})+N\lambda I_{d_N}\right)^{-1} w 1 : N KQ = E U ∼ U [ k U ( U , u 1 : N )] ( k U ( u 1 : N , u 1 : N ) + N λ I d N ) − 1 ∑ i = 1 N w i K Q h ( x i ) \sum_{i=1}^{N}w_i^{\mathrm{KQ}}h(x_i) ∑ i = 1 N w i KQ h ( x i ) x i = Φ ( u i ) x_i=\Phi(u_i) x i = Φ ( u i ) O ( N 3 ) \mathcal{O}(N^3) O ( N 3 ) O ( N ) \mathcal{O}(N) O ( N )
For NKQ, we apply this construction at both stages: write θ = Ψ ( V ) ∼ Q \theta=\Psi(V)\sim Q θ = Ψ ( V ) ∼ Q X = Φ θ ( U ) ∼ P θ X=\Phi_\theta(U)\sim P_\theta X = Φ θ ( U ) ∼ P θ V V V U U U g ( Φ Ψ ( v ) ( u ) , Ψ ( v ) ) g(\Phi_{\Psi(v)}(u),\Psi(v)) g ( Φ Ψ ( v ) ( u ) , Ψ ( v )) θ \theta θ O ( N 3 + T 3 ) \mathcal{O}(N^3+T^3) O ( N 3 + T 3 ) O ( N T + T ) \mathcal{O}(NT+T) O ( N T + T )
Theoretical Results
In this section, we derive a convergence rate for the absolute error ∣ I ^ N K Q − I ∣ |\widehat{I}_{\mathrm{NKQ}}-I| ∣ I NKQ − I ∣ N , T → ∞ N,T\to\infty N , T → ∞ k k k R d \mathbb{R}^{d} R d k ( x , x ′ ) = Ψ ( x − x ′ ) k(x,x')=\Psi(x-x') k ( x , x ′ ) = Ψ ( x − x ′ ) Ψ \Psi Ψ Ψ ^ ( ω ) \widehat{\Psi}(\omega) Ψ ( ω ) R d \mathbb{R}^{d} R d X \mathcal{X} X k k k Ψ ^ ( ω ) \widehat{\Psi}(\omega) Ψ ( ω ) O ( 1 + ∥ ω ∥ 2 ) − s \mathcal{O}\left(1+\lVert\omega\rVert^{2}\right)^{-s} O ( 1 + ∥ ω ∥ 2 ) − s ω → ∞ \omega\to\infty ω → ∞ s > d / 2 s>d/2 s > d /2 H k \mathcal{H}_k H k W 2 s ( X ) W_{2}^{s}(\mathcal{X}) W 2 s ( X ) c 1 , c 2 > 0 c_1,c_2>0 c 1 , c 2 > 0 c 1 ∥ h ∥ H k ≤ ∥ h ∥ W 2 s ( X ) ≤ c 2 ∥ h ∥ H k c_1\lVert h\rVert_{\mathcal{H}_k} \le\lVert h\rVert_{W_2^s(\mathcal{X})} \le c_2\lVert h\rVert_{\mathcal{H}_k} c 1 ∥ h ∥ H k ≤ ∥ h ∥ W 2 s ( X ) ≤ c 2 ∥ h ∥ H k h ∈ H k h \in\mathcal{H}_k h ∈ H k Sobolev reproducing kernel of smoothness s s s . An important example of Sobolev kernel is the Matérn kernel—the RKHS of a Matérn-ν \nu ν W 2 s ( X ) W_2^s(\mathcal{X}) W 2 s ( X ) s = ν + d / 2 s = \nu+ d/2 s = ν + d /2 sup x ∈ X k ( x , x ) ≤ κ \sup_{x \in\mathcal{X}} k(x,x) \le\kappa sup x ∈ X k ( x , x ) ≤ κ κ \kappa κ ∥ f ∥ s , 2 : = ∥ f ∥ W 2 s ( X ) \lVert f\rVert_{s,2} := \lVert f\rVert_{W_2^s(\mathcal{X})} ∥ f ∥ s , 2 := ∥ f ∥ W 2 s ( X )
Theorem 3.1. Let X = [ 0 , 1 ] d X \mathcal{X} = [0,1]^{d_{\mathcal{X}}} X = [ 0 , 1 ] d X Θ = [ 0 , 1 ] d Θ \Theta= [0,1]^{d_{\Theta}} Θ = [ 0 , 1 ] d Θ q q q p θ p_\theta p θ Q \mathbb{Q} Q P θ \mathbb{P}_\theta P θ Θ \Theta Θ X \mathcal{X} X θ 1 : T \theta_{1:T} θ 1 : T Q \mathbb{Q} Q x 1 : N ( t ) x_{1:N}^{(t)} x 1 : N ( t ) P θ t \mathbb{P}_{\theta_t} P θ t t ∈ { 1 , ⋯ , T } t \in\{1,\cdots,T\} t ∈ { 1 , ⋯ , T } k X k_{\mathcal{X}} k X k Θ k_{\Theta} k Θ s X > d X / 2 s_{\mathcal{X}} > d_{\mathcal{X}}/2 s X > d X /2 s Θ > d Θ / 2 s_{\Theta} > d_{\Theta}/2 s Θ > d Θ /2
(1) There exist G 0 , Θ , G 1 , Θ , G 0 , X , G 1 , X > 0 G_{0,\Theta}, G_{1,\Theta}, G_{0,\mathcal{X}}, G_{1,\mathcal{X}} > 0 G 0 , Θ , G 1 , Θ , G 0 , X , G 1 , X > 0 G 0 , Θ ≤ q ( θ ) ≤ G 1 , Θ G_{0,\Theta} \le q(\theta) \le G_{1,\Theta} G 0 , Θ ≤ q ( θ ) ≤ G 1 , Θ G 0 , X ≤ p θ ( x ) ≤ G 1 , X G_{0,\mathcal{X}} \le p_\theta(x) \le G_{1,\mathcal{X}} G 0 , X ≤ p θ ( x ) ≤ G 1 , X θ ∈ Θ \theta\in\Theta θ ∈ Θ x ∈ X x \in\mathcal{X} x ∈ X
(2) There exists S 1 > 0 S_1 > 0 S 1 > 0 θ ∈ Θ \theta\in\Theta θ ∈ Θ β ∈ N d Θ \beta\in\mathbb{N}^{d_{\Theta}} β ∈ N d Θ ∣ β ∣ ≤ s Θ |\beta| \le s_{\Theta} ∣ β ∣ ≤ s Θ ∥ D θ β g ( ⋅ , θ ) ∥ s X , 2 ≤ S 1 \lVert D_\theta^\beta g(\cdot,\theta)\rVert_{s_{\mathcal{X}},2} \le S_1 ∥ D θ β g ( ⋅ , θ ) ∥ s X , 2 ≤ S 1
(3) There exist S 2 , S 3 > 0 S_2,S_3 > 0 S 2 , S 3 > 0 x ∈ X x \in\mathcal{X} x ∈ X ∥ g ( x , ⋅ ) ∥ s Θ , 2 ≤ S 2 \lVert g(x,\cdot)\rVert_{s_{\Theta},2} \le S_2 ∥ g ( x , ⋅ ) ∥ s Θ , 2 ≤ S 2 ∥ θ ↦ p θ ( x ) ∥ s Θ , 2 ≤ S 3 ≤ 1 \lVert\theta\mapsto p_\theta(x)\rVert_{s_{\Theta},2} \le S_3 \le1 ∥ θ ↦ p θ ( x ) ∥ s Θ , 2 ≤ S 3 ≤ 1
(4) There exists S 4 > 0 S_4 > 0 S 4 > 0 f f f s Θ + 1 s_{\Theta}+1 s Θ + 1 S 4 S_4 S 4
Then, there exists N 0 , T 0 ∈ N + N_0,T_0 \in\mathbb{N}^{+} N 0 , T 0 ∈ N + N > N 0 N > N_0 N > N 0 T > T 0 T > T_0 T > T 0 λ X ≍ N − 2 s X d X ( log N ) 2 s X + 2 d X \lambda_{\mathcal{X}} \asymp N^{-\frac{2s_{\mathcal{X}}}{d_{\mathcal{X}}}}(\log N)^{\frac{2s_{\mathcal{X}}+2}{d_{\mathcal{X}}}} λ X ≍ N − d X 2 s X ( log N ) d X 2 s X + 2 λ Θ ≍ T − 2 s Θ d Θ ( log T ) 2 s Θ + 2 d Θ \lambda_{\Theta} \asymp T^{-\frac{2s_{\Theta}}{d_{\Theta}}}(\log T)^{\frac{2s_{\Theta}+2}{d_{\Theta}}} λ Θ ≍ T − d Θ 2 s Θ ( log T ) d Θ 2 s Θ + 2
∣ I − I ^ N K Q ∣ ≤ τ ( C 1 N − s X d X ( log N ) s X + 1 d X + C 2 T − s Θ d Θ ( log T ) s Θ + 1 d Θ ) , \left|I-\widehat{I}_{NKQ}\right| \le\tau\left(C_1N^{-\frac{s_{\mathcal{X}}}{d_{\mathcal{X}}}}(\log N)^{\frac{s_{\mathcal{X}}+1}{d_{\mathcal{X}}}}+C_2T^{-\frac{s_{\Theta}}{d_{\Theta}}}(\log T)^{\frac{s_{\Theta}+1}{d_{\Theta}}}\right), I − I N K Q ≤ τ ( C 1 N − d X s X ( log N ) d X s X + 1 + C 2 T − d Θ s Θ ( log T ) d Θ s Θ + 1 ) , which holds with probability at least 1 − 8 e − τ 1-8e^{-\tau} 1 − 8 e − τ C 1 , C 2 C_1,C_2 C 1 , C 2 N , T , τ N,T,\tau N , T , τ
Corollary 3.2. Suppose all assumptions in Theorem 3.1 hold. If we set N = O ~ ( Δ − d X s X ) N=\widetilde{O}(\Delta^{-\frac{d_{\mathcal{X}}}{s_{\mathcal{X}}}}) N = O ( Δ − s X d X ) T = O ~ ( Δ − d Θ s Θ ) T=\widetilde{O}(\Delta^{-\frac{d_{\Theta}}{s_{\Theta}}}) T = O ( Δ − s Θ d Θ ) N × T = O ~ ( Δ − d X s X − d Θ s Θ ) N \times T=\widetilde{O}(\Delta^{-\frac{d_{\mathcal{X}}}{s_{\mathcal{X}}}-\frac{d_{\Theta}}{s_{\Theta}}}) N × T = O ( Δ − s X d X − s Θ d Θ )
are sufficient to guarantee that ∣ I − I ^ N K Q ∣ ≤ Δ |I-\widehat{I}_{\mathrm{NKQ}}| \le\Delta ∣ I − I NKQ ∣ ≤ Δ
To prove these results, we can decompose ∣ I − I ^ N K Q ∣ |I-\widehat{I}_{\mathrm{NKQ}}| ∣ I − I NKQ ∣ N − s X / d X ( log N ) s X / d X + 1 N^{-s_X/d_X}(\log N)^{s_X/d_X+1} N − s X / d X ( log N ) s X / d X + 1 T − s Θ / d Θ ( log T ) s Θ / d Θ + 1 T^{-s_\Theta/d_\Theta}(\log T)^{s_\Theta/d_\Theta+1} T − s Θ / d Θ ( log T ) s Θ / d Θ + 1 F ^ K Q ( θ 1 : T ) \widehat{F}_{\mathrm{KQ}}(\theta_{1:T}) F KQ ( θ 1 : T ) ∣ I − I ^ N K Q ∣ |I-\widehat{I}_{\mathrm{NKQ}}| ∣ I − I NKQ ∣
We now briefly discuss our assumptions. Assumption (1) is mild and allows L 2 ( P θ ) L_2(P_\theta) L 2 ( P θ ) L 2 ( Q ) L_2(Q) L 2 ( Q ) L 2 ( X ) L_2(\mathcal{X}) L 2 ( X ) L 2 ( Θ ) L_2(\Theta) L 2 ( Θ ) f , g f,g f , g p p p Theorem 3.1 shows that for NKQ to have a fast convergence rate, one ought to use Sobolev kernels which are as smooth as possible in both stages. Furthermore, when s X = s Θ = ∞ s_X=s_\Theta=\infty s X = s Θ = ∞
observations { θ t , F ( θ t ) } t = 1 T \{\theta_t,F(\theta_t)\}_{t=1}^{T} { θ t , F ( θ t ) } t = 1 T O ~ ( N − s X d X ) \widetilde{\mathcal{O}}(N^{-\frac{s_X}{d_X}}) O ( N − d X s X ) { F ^ K Q ( θ t ) } t = 1 T \{\hat{F}_{\mathrm{KQ}}(\theta_t)\}_{t=1}^{T} { F ^ KQ ( θ t ) } t = 1 T O ~ ( N − s X d X ) \widetilde{\mathcal{O}}(N^{-\frac{s_X}{d_X}}) O ( N − d X s X )
Remark 3.4 (Convergence rate for CKQ). For the CKQ estimator J ^ C K Q \hat{J}_{\mathrm{CKQ}} J ^ CKQ (12) ) that approximates the parametric / conditional expectation J ( θ ) J(\theta) J ( θ ) θ ∈ Θ \theta\in\Theta θ ∈ Θ stage I error and can be shown to be O ~ ( N − s X d X ) \widetilde{\mathcal{O}}(N^{-\frac{s_X}{d_X}}) O ( N − d X s X ) stage II error and can be shown to be O ~ ( T − s Θ d Θ ) \widetilde{\mathcal{O}}(T^{-\frac{s_\Theta}{d_\Theta}}) O ( T − d Θ s Θ ) ∥ J − J ^ C K Q ∥ L 2 ( Q ) = O ~ ( N − s X d X + T − s Θ d Θ ) \lVert J-\hat{J}_{\mathrm{CKQ}}\rVert_{L_2(Q)}=\widetilde{\mathcal{O}}(N^{-\frac{s_X}{d_X}}+T^{-\frac{s_\Theta}{d_\Theta}}) ∥ J − J ^ CKQ ∥ L 2 ( Q ) = O ( N − d X s X + T − d Θ s Θ ) 1 − 8 e − τ 1-8e^{-\tau} 1 − 8 e − τ O ( N − s X d X + T − 1 4 ) \mathcal{O}(N^{-\frac{s_X}{d_X}}+T^{-\frac{1}{4}}) O ( N − d X s X + T − 4 1 ) s Θ d Θ > 1 2 > 1 4 \frac{s_\Theta}{d_\Theta}>\frac{1}{2}>\frac{1}{4} d Θ s Θ > 2 1 > 4 1 λ X , λ Θ \lambda_X,\lambda_\Theta λ X , λ Θ λ Θ \lambda_\Theta λ Θ
In Table 3.1 , we compare the cost of all methods evaluated by the number of evaluations required to ensure ∣ I ^ − I ∣ ≤ Δ |\hat{I}-I|\leq\Delta ∣ I ^ − I ∣ ≤ Δ g , p , f g,p,f g , p , f d X s X + d Θ s Θ < 2 \frac{d_X}{s_X}+\frac{d_\Theta}{s_\Theta}<2 s X d X + s Θ d Θ < 2
Table 3.1. Cost of methods for nested expectations, measured through the number of function evaluations required to ensure ∣ I − I ^ ∣ ≤ Δ |I-\hat{I}| \le\Delta ∣ I − I ^ ∣ ≤ Δ r r r Δ − r \Delta^{-r} Δ − r
Experiments
We now illustrate NKQ over a range of applications, including some where the theory does not hold but where we still observe significant gains in accuracy. The code to reproduce all experiments is available at https://github.com/hudsonchen/nest_kq.
Synthetic Experiment We start by verifying the bound in Theorem 3.1 using the following synthetic example: Q = U [ 0 , 1 ] \mathbb{Q}=\mathcal{U}[0,1] Q = U [ 0 , 1 ] P θ = U [ 0 , 1 ] P_\theta=\mathcal{U}[0,1] P θ = U [ 0 , 1 ] g ( x , θ ) = x 5 2 + θ 5 2 g(x,\theta)=x^{\frac{5}{2}}+\theta^{\frac{5}{2}} g ( x , θ ) = x 2 5 + θ 2 5 f ( z ) = z 2 f(z)=z^2 f ( z ) = z 2 I = 0.4115 I=0.4115 I = 0.4115 I I I θ 1 : T ∼ U [ 0 , 1 ] \theta_{1:T}\sim\mathcal{U}[0,1] θ 1 : T ∼ U [ 0 , 1 ] x 1 : N ( t ) ∼ U [ 0 , 1 ] x_{1:N}^{(t)}\sim\mathcal{U}[0,1] x 1 : N ( t ) ∼ U [ 0 , 1 ] t ∈ { 1 , … , T } t\in\{1,\ldots,T\} t ∈ { 1 , … , T } Theorem 3.1 are satisfied with s X = s Θ = 2 s_X=s_\Theta=2 s X = s Θ = 2 d X = d Θ = 1 d_X=d_\Theta=1 d X = d Θ = 1 Δ \Delta Δ N = T = Δ − 0.5 N=T=\Delta^{-0.5} N = T = Δ − 0.5 Theorem 3.2 . On the other hand, based on Theorem 3 of Rainforth et al. (2018), the optimal way of assigning samples for NMC is to choose N = T = Δ − 1 N=\sqrt{T}=\Delta^{-1} N = T = Δ − 1
In Figure 3.1 Top, we see that the optimal choice of N N N T T T = Δ − 1 =\Delta^{-1} = Δ − 1 = Δ − 3 =\Delta^{-3} = Δ − 3 Theorem 3.1 and Rainforth et al. (2018, Theorem 3) are indeed realized. We also adapt the synthetic problem to higher dimensions (d X = d Θ = d d_X=d_\Theta=d d X = d Θ = d Figure 3.1 Bottom that the performance gap between NKQ and NMC closes down as dimension grows. Such behaviour is expected because the cost of NKQ is O ~ ( Δ − d X s X − d Θ s Θ ) \widetilde{O}\left(\Delta^{-\frac{d_X}{s_X}-\frac{d_\Theta}{s_\Theta}}\right) O ( Δ − s X d X − s Θ d Θ ) d X d_X d X d Θ d_\Theta d Θ
Figure 3.1. Synthetic experiment. Top: Verification of theoretical results. The thin grey lines are theoretical rates of Δ = Cost − 1 \Delta= \operatorname{Cost}^{-1} Δ = Cost − 1 Δ = Cost − 1 / 3 \Delta= \operatorname{Cost}^{-1/3} Δ = Cost − 1/3 Bottom: Comparison of NKQ and NMC as dimension increases. Results are averaged over 1000 independent runs, while shaded regions give the 25%-75% quantiles.
Bayesian Optimization We conclude with an application in Bayesian optimization. Typical acquisition functions are greedy approaches that maximize the immediate reward, while look-ahead acquisition functions optimize accumulated reward over a planning horizon, which results in reduced number of required function evaluations (Ginsbourger and Le Riche, 2010; González et al., 2016; Wu and Frazier, 2019; Yang et al., 2024). The utility of a two-step look ahead acquisition functions can be written as the following nested expectation.
α ( z ; D ) : = E f ∣ D [ g ( f ∣ D , z ) + max z ′ E f ∣ D ′ [ g ( f ∣ D ′ , z ′ ) ] ] \alpha(z; \mathcal{D}) := \mathbb{E}_{f_{\mid\mathcal{D}}}\left[g(f_{\mid\mathcal{D}}, z) + \max_{z'} \mathbb{E}_{f_{\mid\mathcal{D}'}}\left[g(f_{\mid\mathcal{D}'}, z')\right]\right] α ( z ; D ) := E f ∣ D [ g ( f ∣ D , z ) + z ′ max E f ∣ D ′ [ g ( f ∣ D ′ , z ′ ) ] ] where f ∣ D f_{\mid\mathcal{D}} f ∣ D f ∣ D ′ f_{\mid\mathcal{D}'} f ∣ D ′ D \mathcal{D} D D ′ : = D ∪ ( z , f ∣ D ( z ) ) \mathcal{D}' := \mathcal{D} \cup(z, f_{\mid\mathcal{D}}(z)) D ′ := D ∪ ( z , f ∣ D ( z )) f ∣ D f_{\mid D} f ∣ D D 0 \mathcal{D}_0 D 0 g g g q = 2 q = 2 q = 2 z = ( z 1 , z 2 ) z = (z_1, z_2) z = ( z 1 , z 2 ) g ( f ∣ D , z ) = max j = 1 , 2 ( f ∣ D ( z j ) − r max ) , 0 ) g(f_{\mid D}, z) = \max_{j=1,2}(f_{\mid D}(z_j) - r_{\max}), 0) g ( f ∣ D , z ) = max j = 1 , 2 ( f ∣ D ( z j ) − r m a x ) , 0 ) r max r_{\max} r m a x f ∣ D f_{\mid D} f ∣ D f ∣ D 0 f_{\mid D^0} f ∣ D 0 z z z z ∗ z^* z ∗ f ∣ D 0 ( z 1 , z 2 ) f_{\mid D^0}(z_1,z_2) f ∣ D 0 ( z 1 , z 2 ) x x x f ∣ D ( z 1 , z 2 ) f_{\mid D}(z_1,z_2) f ∣ D ( z 1 , z 2 ) θ \theta θ (7) (i.e. d X = d Θ = 2 d_{\mathsf{X}} = d_{\mathsf{\Theta}} = 2 d X = d Θ = 2 f ∣ D 0 , f ∣ D f_{\mid D^0}, f_{\mid D} f ∣ D 0 , f ∣ D max \max max s X = 1 s_{\mathsf{X}} = 1 s X = 1 Θ \Theta Θ
We benchmark NKQ, NMC and MLMC on three synthetic tasks from BoTorch (Balandat et al., 2020). For NKQ, both k X k_{\mathsf{X}} k X k Θ k_{\mathsf{\Theta}} k Θ 1 2 \frac{1}{2} 2 1 Q \mathbb{Q} Q P θ \mathbb{P}_{\theta} P θ [ 0 , 1 ] d [0,1]^d [ 0 , 1 ] d O ( T × N ) \mathcal{O}(T \times N) O ( T × N ) Δ = 0.01 \Delta= 0.01 Δ = 0.01 Table 3.1 , we choose N = T = Δ − 2 N = T = \Delta^{-2} N = T = Δ − 2 N = T = Δ − 1 N = T = \Delta^{-1} N = T = Δ − 1 ∥ max z ∈ D S f B B ( z ) − f B B ( z ∗ ) ∥ 2 ∥ max z ∈ D 0 f B B ( z ) − f B B ( z ∗ ) ∥ 2 \frac{\left\|\max_{z\in\mathcal{D}_S} f_{\mathrm{BB}}(z)-f_{\mathrm{BB}}(z^*)\right\|^2}{\left\|\max_{z\in\mathcal{D}_0} f_{\mathrm{BB}}(z)-f_{\mathrm{BB}}(z^*)\right\|^2} ∥ m a x z ∈ D 0 f BB ( z ) − f BB ( z ∗ ) ∥ 2 ∥ m a x z ∈ D S f BB ( z ) − f BB ( z ∗ ) ∥ 2 D 0 \mathcal{D}_0 D 0 D S \mathcal{D}_S D S S S S f B B f_{\mathrm{BB}} f BB f B B ( z ∗ ) f_{\mathrm{BB}}(z^*) f BB ( z ∗ )
In Figure 3.2 , we compare the efficiency of each method by plotting their NMSE against cumulative computational time in wall clock. We can see that NKQ achieves the lowest NMSE among all methods under a fixed amount of computational time in all three datasets, even though the assumptions of Theorem 3.1 are not all satisfied. Since the Dropwave, Ackley, and Cosine8 functions are synthetic and computationally cheap, we expect the advantages of NKQ to be more pronounced for Bayesian optimization on real-world expensive problems. Furthermore, many other utility functions in Bayesian optimization—such as predictive entropy search—involve nested expectations (Balandat et al., 2020). We leave the empirical evaluation of NKQ on these utility functions to future work.
Figure 3.2. Bayesian optimization with look ahead acquisition function. The plots are NMSE against accumulative wall clock time. Results are averaged over 100 independent runs, while shaded regions give the 25%-75% quantiles.
Conclusion
This paper introduces a novel estimator for nested expectations based on kernel quadrature. We prove in Theorem 3.1 that our method has a faster rate of convergence than existing methods provided that the problem has sufficient smoothness. This theoretical result is consistent with the empirical evidence in several numerical experiments. Additionally, even when the problem is not as smooth as the theory requires, NKQ can still outperform baseline methods potentially due to the use of non-equal weights.
Following our work, there remain a number of interesting future problems and we now highlight two main ones. Firstly, we propose a combination of KQ and MLMC that we call MLKQ. However, we believe our current theoretical rate for MLKQ is sub-optimal due to the sub-optimal allocation of samples at each level. Further work will therefore be needed to determine whether this is a viable approach in some cases. Secondly, for applications where function evaluations are extremely expensive, NKQ could be extended to its Bayesian counterpart. This would allow us to use the finite sample uncertainty quantification for adaptive selection of samples, which could further improve performance. Finally, establishing minimax lower bounds for nested expectation remains an open and compelling problem. The main difficulty lies in its two-stage structure. To the best of our knowledge, existing minimax lower bounds for two-stage problem typically reduce the problem to a one-stage problem before deriving the bound; see, for example, Chen and Reiss (2011, Chapter 3), Meunier et al. (2024, Appendix F.1), and Zhang et al. (2025). However, it remains unclear how to directly establish meaningful minimax lower bounds of genuinely two-stage problems, such as our nested expectation.
Minimum MMD for Generative Modelling
The results in this chapter were published in the following paper:
Zonghao Chen, Aratrika Mustafi, Pierre Glaser, Anna Korba, Arthur Gretton, and Bharath K. Sriperumbudur.
(De)-regularized Maximum Mean Discrepancy Gradient Flow.
Journal of Machine Learning Research , 2025.
Introduction
Many applications in computational statistics and machine learning involve approximating a probability distribution π \pi π R d \mathbb{R}^{d} R d π \pi π π \pi π π \pi π π \pi π π \pi π R d \mathbb{R}^{d} R d manifold hypothesis : high-dimensional observations often lie on or near a manifold of much lower dimension (Fefferman et al., 2016).
Recently, a popular framework to perform this task involves solving a minimization problem in P ( R d ) \mathcal{P}(\mathbb{R}^{d}) P ( R d ) R d \mathbb{R}^{d} R d dissimilarity function D ( ⋅ ∥ π ) \mathcal{D}(\cdot\|\pi) D ( ⋅ ∥ π ) D ( ν ∥ π ) = 0 \mathcal{D}(\nu\|\pi)=0 D ( ν ∥ π ) = 0 ν = π \nu=\pi ν = π π \pi π π \pi π P 2 ( R d ) \mathcal{P}_{2}(\mathbb{R}^{d}) P 2 ( R d ) F = D ( ⋅ ∥ π ) \mathcal{F}=\mathcal{D}(\cdot\|\pi) F = D ( ⋅ ∥ π ) F \mathcal{F} F
In generative modeling, the choice of D ( ⋅ ∥ π ) \mathcal{D}(\cdot\|\pi) D ( ⋅ ∥ π ) π \pi π π \pi π global minimizer. Combined, these two properties ensure that, in the large sample limit, this algorithm generates new samples from the target. Unfortunately, verifying these two properties simultaneously has proved to be a surprisingly challenging task. For instance, recent approaches based on Maximum Mean Discrepancy (MMD) (Arbel et al., 2019; Hertrich et al., 2024b), the sliced-Wasserstein distance (Liutkus et al., 2019) and the Sinkhorn divergence (Genevay et al., 2018) typically admit consistent finite-sample implementations, but their global convergence guarantees—when they exist—do not apply to most practical targets π \pi π D \mathcal{D} D f f f χ 2 \chi^{2} χ 2 π ∝ e − V \pi\propto e^{-V} π ∝ e − V V V V χ 2 \chi^{2} χ 2 f f f
In the face of the trade-offs present in the current approaches, a natural question arises: Does there exist a divergence functional D ( ⋅ ∥ π ) \mathcal{D}(\cdot\|\pi) D ( ⋅ ∥ π ) In this work, we take a step towards a positive answer by constructing a “de-regularized” variant of the Maximum Mean Discrepancy (DrMMD) and its associated Wasserstein gradient flow. We prove that the DrMMD gradient flow converges exponentially to the global minimum up to a controllable barrier term for targets π \pi π χ 2 \chi^{2} χ 2 f f f π \pi π
In this paper, we make the following assumption on our kernel.
Assumption 4.1. k : R d × R d → R k : \mathbb{R}^{d} \times\mathbb{R}^{d} \to\mathbb{R} k : R d × R d → R c 0 c_{0} c 0
and there exists K > 0 K > 0 K > 0 sup x k ( x , x ) ≤ K \sup_x k(x,x) \le K sup x k ( x , x ) ≤ K
Assumption 4.2. k : R d × R d → R k : \mathbb{R}^{d} \times\mathbb{R}^{d} \to\mathbb{R} k : R d × R d → R i , j ∈ { 1 , ⋯ , d } i,j \in\{1,\cdots,d\} i , j ∈ { 1 , ⋯ , d } ∂ i ∂ i + d k \partial_i\partial_{i+d}k ∂ i ∂ i + d k ∂ i ∂ j ∂ i + d ∂ j + d k \partial_i\partial_j\partial_{i+d}\partial_{j+d}k ∂ i ∂ j ∂ i + d ∂ j + d k K 1 d , K 2 d > 0 K_{1d}, K_{2d} > 0 K 1 d , K 2 d > 0 ∥ ∇ 1 k ( x , ⋅ ) ∥ H d : = ∑ i = 1 d ∥ ∂ i k ( x , ⋅ ) ∥ H ≤ K 1 d \|\nabla_1 k(x,\cdot)\|_{\mathcal{H}^d} := \sum_{i=1}^{d} \|\partial_i k(x,\cdot)\|_{\mathcal{H}} \le\sqrt{K_{1d}} ∥ ∇ 1 k ( x , ⋅ ) ∥ H d := ∑ i = 1 d ∥ ∂ i k ( x , ⋅ ) ∥ H ≤ K 1 d ∥ H 1 k ( x , ⋅ ) ∥ H d × d : = ∑ i , j = 1 d ∥ ∂ i ∂ j k ( x , ⋅ ) ∥ H ≤ K 2 d \|\mathbf{H}_1 k(x,\cdot)\|_{\mathcal{H}^{d\times d}} := \sum_{i,j=1}^{d} \|\partial_i\partial_j k(x,\cdot)\|_{\mathcal{H}} \le\sqrt{K_{2d}} ∥ H 1 k ( x , ⋅ ) ∥ H d × d := ∑ i , j = 1 d ∥ ∂ i ∂ j k ( x , ⋅ ) ∥ H ≤ K 2 d x ∈ R d x \in\mathbb{R}^{d} x ∈ R d
Many kernels satisfy both Assumptions 4.1 and 4.2, including the class of bounded, continuous, and translation invariant kernels on R d \mathbb{R}^{d} R d ∇ 1 k ( ⋅ , x ) \nabla_1 k(\cdot,x) ∇ 1 k ( ⋅ , x ) H 1 k ( ⋅ , x ) \mathbf{H}_1 k(\cdot,x) H 1 k ( ⋅ , x ) x x x ν \nu ν ν + d 2 ≥ 2 \nu+ \frac{d}{2} \ge2 ν + 2 d ≥ 2
(De)-regularized Maximum Mean Discrepancy (DrMMD)
In this section, we introduce a (de)-regularized version of maximum mean discrepancy, or DrMMD in short. The DrMMD is rooted in a unified representation of the MMD and the χ 2 \chi^2 χ 2
Proposition 4.3 (MMD and χ 2 \chi^2 χ 2 d μ d π − 1 ∈ L 2 ( π ) \frac{d\mu}{d\pi} - 1 \in L^2(\pi) d π d μ − 1 ∈ L 2 ( π ) μ , π ∈ P 2 ( R d ) \mu,\pi\in\mathcal{P}_2(\mathbb{R}^{d}) μ , π ∈ P 2 ( R d )
MMD 2 ( μ ∥ π ) = ∥ T π 1 / 2 ( d μ d π − 1 ) ∥ L 2 ( π ) 2 and χ 2 ( μ ∥ π ) = ∥ Id ( d μ d π − 1 ) ∥ L 2 ( π ) 2 \operatorname{MMD}^{2}(\mu\|\pi) = \left\|\mathcal{T}_{\pi}^{1/2}\left(\frac{d\mu}{d\pi} - 1\right)\right\|_{L^2(\pi)}^2 \quad\text{and} \quad\chi^2(\mu\|\pi) = \left\|\operatorname{Id}\left(\frac{d\mu}{d\pi} - 1\right)\right\|_{L^2(\pi)}^2 MMD 2 ( μ ∥ π ) = T π 1/2 ( d π d μ − 1 ) L 2 ( π ) 2 and χ 2 ( μ ∥ π ) = Id ( d π d μ − 1 ) L 2 ( π ) 2 Remark 4.4. The χ 2 \chi^2 χ 2 χ 2 \chi^2 χ 2 d μ d π − 1 \frac{d\mu}{d\pi}-1 d π d μ − 1 χ 2 \chi^2 χ 2 L 2 ( π ) L^2(\pi) L 2 ( π ) M M D 2 \mathrm{MMD}^2 MMD 2 T π 1 / 2 \mathcal{T}_{\pi}^{1/2} T π 1/2 L 2 ( π ) L^2(\pi) L 2 ( π ) T π 1 / 2 \mathcal{T}_{\pi}^{1/2} T π 1/2 T π \mathcal{T}_{\pi} T π k k k F M M D ( ⋅ ) = M M D 2 ( ⋅ ∥ π ) \mathcal{F}_{\mathrm{MMD}}(\cdot)=\mathrm{MMD}^2(\cdot\mathbin{\|}\pi) F MMD ( ⋅ ) = MMD 2 ( ⋅ ∥ π ) F χ 2 ( ⋅ ) = χ 2 ( ⋅ ∥ π ) \mathcal{F}_{\chi^2}(\cdot)=\chi^2(\cdot\mathbin{\|}\pi) F χ 2 ( ⋅ ) = χ 2 ( ⋅ ∥ π ) π \pi π
With these facts in mind, we introduce a divergence whose purpose is to combine the beneficial properties of both the χ 2 \chi^2 χ 2 L 2 ( π ) L^2(\pi) L 2 ( π ) d μ d π − 1 \frac{d\mu}{d\pi}-1 d π d μ − 1 I d \mathrm{Id} Id T π 1 / 2 \mathcal{T}_{\pi}^{1/2} T π 1/2 ( ( T π + λ I d ) − 1 T π ) 1 / 2 \left((\mathcal{T}_{\pi}+\lambda\mathrm{Id})^{-1}\mathcal{T}_{\pi}\right)^{1/2} ( ( T π + λ Id ) − 1 T π ) 1/2 λ > 0 \lambda>0 λ > 0 ( ( T π + λ I d ) − 1 T π ) 1 / 2 \left((\mathcal{T}_{\pi}+\lambda\mathrm{Id})^{-1}\mathcal{T}_{\pi}\right)^{1/2} ( ( T π + λ Id ) − 1 T π ) 1/2 T π 1 / 2 \mathcal{T}_{\pi}^{1/2} T π 1/2
Definition 4.5 (DrMMD). Suppose d μ d π − 1 ∈ L 2 ( π ) \frac{d\mu}{d\pi}-1\in L^2(\pi) d π d μ − 1 ∈ L 2 ( π ) μ , π ∈ P 2 ( R d ) \mu,\pi\in\mathcal{P}_2(\mathbb{R}^d) μ , π ∈ P 2 ( R d ) μ , π ∈ P 2 ( R d ) \mu,\pi\in\mathcal{P}_2(\mathbb{R}^d) μ , π ∈ P 2 ( R d )
DrMMD ( μ ∥ π ) = ( 1 + λ ) ∥ ( ( T π + λ I d ) − 1 T π ) 1 / 2 ( d μ d π − 1 ) ∥ L 2 ( π ) 2 , (13) \operatorname{DrMMD}(\mu\mathbin{\|}\pi)=(1+\lambda)\left\|\left((\mathcal{T}_{\pi}+\lambda\mathrm{Id})^{-1}\mathcal{T}_{\pi}\right)^{1/2}\left(\frac{d\mu}{d\pi}-1\right)\right\|_{L^2(\pi)}^2, \tag*{(13)} DrMMD ( μ ∥ π ) = ( 1 + λ ) ( ( T π + λ Id ) − 1 T π ) 1/2 ( d π d μ − 1 ) L 2 ( π ) 2 , (13) where λ > 0 \lambda>0 λ > 0
While all three operators, Id \operatorname{Id} Id T π \mathcal{T}_{\pi} T π ( T π + λ Id ) − 1 T π (\mathcal{T}_{\pi}+\lambda\operatorname{Id})^{-1}\mathcal{T}_{\pi} ( T π + λ Id ) − 1 T π L 2 ( π ) L^{2}(\pi) L 2 ( π ) Id \operatorname{Id} Id 1 1 1 T π \mathcal{T}_{\pi} T π ( q i ) i ≥ 1 (q_i)_{i\geq1} ( q i ) i ≥ 1 i → ∞ i\to\infty i → ∞ ( T π + λ Id ) − 1 T π (\mathcal{T}_{\pi}+\lambda\operatorname{Id})^{-1}\mathcal{T}_{\pi} ( T π + λ Id ) − 1 T π ( q i / ( q i + λ ) ) i ≥ 1 (q_i/(q_i+\lambda))_{i\geq1} ( q i / ( q i + λ ) ) i ≥ 1 1 1 1 λ \lambda λ i → ∞ i\to\infty i → ∞ ( T π + λ Id ) − 1 T π (\mathcal{T}_{\pi}+\lambda\operatorname{Id})^{-1}\mathcal{T}_{\pi} ( T π + λ Id ) − 1 T π
One of the stated purposes of the DrMMD is to retain the computational benefits of the MMD, which are crucial for its use in particle algorithms for generative modeling. To this end, we provide an alternative representation of DrMMD which does not involve the density ratio d μ d π \frac{d\mu}{d\pi} d π d μ
Proposition 4.6 (Density ratio–free and variational formulations). DrMMD can be alternately represented as
DrMMD ( μ ∥ π ) = ( 1 + λ ) ∥ ( Σ π + λ Id ) − 1 2 ( m μ − m π ) ∥ H 2 (14) \operatorname{DrMMD}(\mu\|\pi)=(1+\lambda)\left\|(\Sigma_{\pi}+\lambda\operatorname{Id})^{-\frac{1}{2}}(m_{\mu}-m_{\pi})\right\|_{\mathcal{H}}^{2} \tag*{(14)} DrMMD ( μ ∥ π ) = ( 1 + λ ) ( Σ π + λ Id ) − 2 1 ( m μ − m π ) H 2 (14) = ( 1 + λ ) sup h ∈ H { ∫ h d μ − ∫ ( h 2 4 + h ) d π − λ 4 ∥ h ∥ H 2 } . (15) =(1+\lambda)\sup_{h\in\mathcal{H}}\left\{\int h\,d\mu-\int\left(\frac{h^{2}}{4}+h\right)d\pi-\frac{\lambda}{4}\|h\|_{\mathcal{H}}^{2}\right\}. \tag*{(15)} = ( 1 + λ ) h ∈ H sup { ∫ h d μ − ∫ ( 4 h 2 + h ) d π − 4 λ ∥ h ∥ H 2 } . (15) with h μ , π ∗ = 2 ( Σ π + λ Id ) − 1 ( m μ − m π ) h_{\mu,\pi}^{*}=2(\Sigma_{\pi}+\lambda\operatorname{Id})^{-1}(m_{\mu}-m_{\pi}) h μ , π ∗ = 2 ( Σ π + λ Id ) − 1 ( m μ − m π )
The proof is in Section B.1.2. The density ratio-free representation (4.2) contains three expectations under μ \mu μ π \pi π m μ m_{\mu} m μ m π m_{\pi} m π Σ π \Sigma_{\pi} Σ π { x i } i = 1 M ∼ μ \{x_i\}_{i=1}^{M}\sim\mu { x i } i = 1 M ∼ μ { y i } i = 1 N ∼ π \{y_i\}_{i=1}^{N}\sim\pi { y i } i = 1 N ∼ π μ \mu μ π \pi π μ ^ = 1 M ∑ i = 1 M δ x i \hat{\mu}=\frac{1}{M}\sum_{i=1}^{M}\delta_{x_i} μ ^ = M 1 ∑ i = 1 M δ x i π ^ = 1 N ∑ i = 1 N δ y i \hat{\pi}=\frac{1}{N}\sum_{i=1}^{N}\delta_{y_i} π ^ = N 1 ∑ i = 1 N δ y i
The density ratio-free representation (4.2) frames DrMMD as acting on the difference of m μ m_{\mu} m μ m π m_{\pi} m π ( 1 + λ ) (1+\lambda) ( 1 + λ ) k ~ \tilde{k} k ~
k ~ ( x , x ′ ) = ⟨ ( Σ π + λ Id ) − 1 2 k ( x , ⋅ ) , ( Σ π + λ Id ) − 1 2 k ( x ′ , ⋅ ) ⟩ H . (16) \tilde{k}(x,x')=\left\langle(\Sigma_{\pi}+\lambda\operatorname{Id})^{-\frac{1}{2}}k(x,\cdot),(\Sigma_{\pi}+\lambda\operatorname{Id})^{-\frac{1}{2}}k(x',\cdot)\right\rangle_{\mathcal{H}}. \tag*{(16)} k ~ ( x , x ′ ) = ⟨ ( Σ π + λ Id ) − 2 1 k ( x , ⋅ ) , ( Σ π + λ Id ) − 2 1 k ( x ′ , ⋅ ) ⟩ H . (16) The derivations are provided in Section B.1.3. The kernel k ~ \tilde{k} k ~ H ~ \tilde{\mathcal{H}} H ~
Properties of DrMMD
In this section, we establish various properties of DrMMD. As discussed earlier, DrMMD is constructed to interpolate between χ 2 \chi^2 χ 2
Proposition 4.7 (Interpolation property). Let μ , π ∈ P 2 ( R d ) \mu,\pi\in\mathcal{P}_2(\mathbb{R}^d) μ , π ∈ P 2 ( R d ) d μ d π − 1 ∈ L 2 ( π ) \frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\in L^2(\pi) d π d μ − 1 ∈ L 2 ( π )
lim λ → 0 DrMMD ( μ ∥ π ) = χ 2 ( μ ∥ π ) , and lim λ → ∞ DrMMD ( μ ∥ π ) = MMD 2 ( μ ∥ π ) . \lim_{\lambda\to0}\operatorname{DrMMD}(\mu\Vert\pi)=\chi^2(\mu\Vert\pi), \qquad\text{and} \qquad\lim_{\lambda\to\infty}\operatorname{DrMMD}(\mu\Vert\pi)=\operatorname{MMD}^2(\mu\Vert\pi). λ → 0 lim DrMMD ( μ ∥ π ) = χ 2 ( μ ∥ π ) , and λ → ∞ lim DrMMD ( μ ∥ π ) = MMD 2 ( μ ∥ π ) . Theorem 4.7, whose proof can be found in Section B.1.4, shows that DrMMD asymptotically becomes a probability divergence in the small and large λ \lambda λ any fixed value of λ \lambda λ
Proposition 4.8 (DrMMD is a probability divergence). Under Assumption 4.1, for any λ ∈ ( 0 , ∞ ) \lambda\in(0,\infty) λ ∈ ( 0 , ∞ )
DrMMD ( μ ∥ π ) ≥ 0 \operatorname{DrMMD}(\mu\Vert\pi) \ge0 DrMMD ( μ ∥ π ) ≥ 0 μ = π \mu=\pi μ = π DrMMD \operatorname{DrMMD} DrMMD DrMMD ( μ n ∥ π ) → 0 \operatorname{DrMMD}(\mu_n\Vert\pi)\to0 DrMMD ( μ n ∥ π ) → 0 μ n \mu_n μ n π ∈ P 2 ( R d ) \pi\in\mathcal{P}_2(\mathbb{R}^d) π ∈ P 2 ( R d ) n → ∞ n\to\infty n → ∞
As MMD with c 0 c_0 c 0 λ > 0 \lambda>0 λ > 0 DrMMD \operatorname{DrMMD} DrMMD χ 2 \chi^2 χ 2
Wasserstein Gradient Flow of DrMMD
Having introduced the DrMMD in the previous section, we now construct and analyze its Wasserstein Gradient Flow (WGF). As discussed in § 4.3 , WGFs define dynamics ( μ t ) t ≥ 0 (\mu_t)_{t\ge0} ( μ t ) t ≥ 0 F \mathcal{F} F μ t \mu_t μ t Fréchet subdifferential of F \mathcal{F} F μ t \mu_t μ t π \pi π F DrMMD ( ⋅ ) = DrMMD ( ⋅ ∥ π ) \mathcal{F}_{\operatorname{DrMMD}}(\cdot)=\operatorname{DrMMD}(\cdot\Vert\pi) F DrMMD ( ⋅ ) = DrMMD ( ⋅ ∥ π ) μ t \mu_t μ t π \pi π
To fulfill this potential, we formally establish that μ t \mu_t μ t π \pi π F \mathcal{F} F χ 2 \chi^2 χ 2
DrMMD flow: Definition, existence, and uniqueness
To prove that the DrMMD flow is well-defined and admits solutions, the key is to show that the F D r M M D \mathcal{F}_{\mathrm{DrMMD}} F DrMMD
Proposition 4.10 (DrMMD gradient flow). Let λ > 0 \lambda> 0 λ > 0 and μ 0 , π ∈ P 2 ( R d ) \mu_0,\pi\in\mathcal{P}_2(\mathbb{R}^d) μ 0 , π ∈ P 2 ( R d ) Under Assumptions 4.1 and 4.2, the functional F D r M M D \mathcal{F}_{\mathrm{DrMMD}} F DrMMD admits Fréchet subdifferential of the form ( 1 + λ ) ∇ h μ t , π ∗ (1+\lambda)\nabla h^*_{\mu_t,\pi} ( 1 + λ ) ∇ h μ t , π ∗ where h μ t , π ∗ h^*_{\mu_t,\pi} h μ t , π ∗ is the witness function defined in Theorem 4.6. Consequently, the DrMMD flow is well-defined and is the solution to the following equation
∂ t μ t − ∇ ⋅ ( μ t ( 1 + λ ) ∇ h μ t , π ∗ ) = 0. (17) \partial_t\mu_t-\nabla\cdot\left(\mu_t(1+\lambda)\nabla h^*_{\mu_t,\pi}\right)=0. \tag*{(17)} ∂ t μ t − ∇ ⋅ ( μ t ( 1 + λ ) ∇ h μ t , π ∗ ) = 0. (17) In addition, the DrMMD flow starting at μ 0 \mu_0 μ 0 is unique because F D r M M D \mathcal{F}_{\mathrm{DrMMD}} F DrMMD is semiconvex, i.e., for any ϕ ∈ C c ∞ ( R d ) \phi\in C^\infty_c(\mathbb{R}^d) ϕ ∈ C c ∞ ( R d ) and μ ∈ P 2 ( R d ) \mu\in\mathcal{P}_2(\mathbb{R}^d) μ ∈ P 2 ( R d )
∣ ⟨ Hess F D r M M D ∣ μ ∇ ϕ , ∇ ϕ ⟩ L 2 ( μ ) ∣ ≤ 2 ( 1 + λ ) 2 K K 2 d + K 1 d λ ∥ ∇ ϕ ∥ L 2 ( μ ) 2 . (18) \left|\left\langle\operatorname{Hess}\mathcal{F}_{\mathrm{DrMMD}}|_\mu\nabla\phi,\nabla\phi\right\rangle_{L^2(\mu)}\right|\leq2(1+\lambda)\frac{2\sqrt{K K_{2d}+K_{1d}}}{\lambda}\lVert\nabla\phi\rVert_{L^2(\mu)}^2. \tag*{(18)} ⟨ Hess F DrMMD ∣ μ ∇ ϕ , ∇ ϕ ⟩ L 2 ( μ ) ≤ 2 ( 1 + λ ) λ 2 K K 2 d + K 1 d ∥ ∇ ϕ ∥ L 2 ( μ ) 2 . (18) :::
The proof is in Section B.1.6. By recalling the discussion of Wasserstein Hessian in § 4.3 , (4.6) indicates that F D r M M D \mathcal{F}_{\mathrm{DrMMD}} F DrMMD geodesically smooth and geodesically semiconvex , which is expected because DrMMD is equivalent to MMD with a regularized kernel k ~ \tilde{k} k ~ (16) and F M M D \mathcal{F}_{\mathrm{MMD}} F MMD geodesically smooth and geodesically semiconvex (Arbel et al., 2019, Proposition 5).
Theorem 4.10 is proved concurrently in Corollaries 14 and 20 of Stein et al. (2025) relying on formulation via Moreau envelopes, while our proof uses the closed-form expression for DrMMD.
Near-global convergence of DrMMD flow
Having defined the DrMMD flow, we are now concerned with its convergence to the target π \pi π χ 2 \chi^2 χ 2 χ 2 \chi^2 χ 2 asymptotically positive as λ → 0 \lambda\to0 λ → 0 π \pi π non-asymptotic convergence rate, we take another route and show that the DrMMD flow converges to π \pi π λ \lambda λ π \pi π
Near-Geodesic Convexity of F D r M M D \mathcal{F}_{\mathrm{DrMMD}} F DrMMD
One popular approach to proving that the DrMMD flow ( μ t ) t ≥ 0 (\mu_t)_{t\geq0} ( μ t ) t ≥ 0 π \pi π F D r M M D \mathcal{F}_{\mathrm{DrMMD}} F DrMMD geodesically convex (Ambrosio et al., 2005, Theorem 4.0.4), or, equivalently in our definition in § 4.3 , its Wasserstein Hessian is positive definite. In the next proposition, we show that the Wasserstein Hessian of F D r M M D \mathcal{F}_{\mathrm{DrMMD}} F DrMMD λ \lambda λ λ → 0 \lambda\to0 λ → 0
Proposition 4.11 (Near-geodesic convexity of F D r M M D \mathcal{F}_{\mathrm{DrMMD}} F DrMMD Let μ , π ∈ P 2 ( R d ) \mu,\pi\in\mathcal{P}_2(\mathbb{R}^d) μ , π ∈ P 2 ( R d ) μ , π ≪ L d \mu,\pi\ll\mathcal{L}^d μ , π ≪ L d ϕ ∈ C c ∞ ( R d ) \phi\in C^\infty_c(\mathbb{R}^d) ϕ ∈ C c ∞ ( R d ) Under Assumptions 4.1 and 4.2, let π \pi π be α \alpha α strongly log-concave, i.e., π ∝ exp ( − V ) \pi\propto\exp(-V) π ∝ exp ( − V ) H V ⪰ α I d \mathbf{H}V \succeq\alpha\mathrm{Id} H V ⪰ α Id and assume additionally that x ↦ H V ( x ) x \mapsto\mathbf{H}V(x) x ↦ H V ( x ) is continuous. Then for all μ \mu μ such that x ↦ ∇ log μ ( x ) x \mapsto\nabla\log\mu(x) x ↦ ∇ log μ ( x ) is
continuous and d μ d π − 1 ∈ H \frac{d\mu}{d\pi}-1 \in\mathcal{H} d π d μ − 1 ∈ H
⟨ Hess F D r M M D | μ ∇ ϕ , ∇ ϕ ⟩ L 2 ( μ ) ≥ α ( 1 + λ ) ∫ d μ d π ( x ) ∥ ∇ ϕ ( x ) ∥ 2 d μ ( x ) − R ( λ , μ , ∇ ϕ ) , (19) \left\langle\operatorname{Hess}\mathcal{F}_{\mathrm{DrMMD}}\middle|_{\mu}\nabla\phi,\nabla\phi\right\rangle_{L^{2}(\mu)} \ge\alpha(1+\lambda)\int\frac{d\mu}{d\pi}(x)\lVert\nabla\phi(x)\rVert^{2}d\mu(x)-R(\lambda,\mu,\nabla\phi), \tag*{(19)} ⟨ Hess F DrMMD μ ∇ ϕ , ∇ ϕ ⟩ L 2 ( μ ) ≥ α ( 1 + λ ) ∫ d π d μ ( x ) ∥ ∇ ϕ ( x ) ∥ 2 d μ ( x ) − R ( λ , μ , ∇ ϕ ) , (19) where lim λ → 0 R ( λ , μ , ∇ ϕ ) = 0 \lim_{\lambda\to0}R(\lambda,\mu,\nabla\phi)=0 lim λ → 0 R ( λ , μ , ∇ ϕ ) = 0
The proof can be found in Section B.1.7. To obtain this result, we relate the Wasserstein Hessian of F D r M M D \mathcal{F}_{\mathrm{DrMMD}} F DrMMD F χ 2 \mathcal{F}_{\chi^{2}} F χ 2 d μ d π − 1 ∈ H \frac{d\mu}{d\pi}-1 \in\mathcal{H} d π d μ − 1 ∈ H λ → 0 \lambda\to0 λ → 0 χ 2 \chi^{2} χ 2 F χ 2 \mathcal{F}_{\chi^{2}} F χ 2 α \alpha α π \pi π (19) . It is noteworthy that although DrMMD can be viewed as squared MMD with a regularized kernel k ~ \tilde{k} k ~ k k k χ 2 \chi^{2} χ 2
Remark 4.12 (Geodesic convexity/smoothness trade-off in F D r M M D \mathcal{F}_{\mathrm{DrMMD}} F DrMMD F D r M M D \mathcal{F}_{\mathrm{DrMMD}} F DrMMD (18) and (19) respectively via upper and lower bounds on the Wasserstein Hessian of F D r M M D \mathcal{F}_{\mathrm{DrMMD}} F DrMMD (18) and (19) impose contradictory conditions on the (de)-regularization parameter λ \lambda λ (18) indicates that DrMMD is smoother if λ \lambda λ (19) indicates that DrMMD is more convex if λ \lambda λ F D r M M D \mathcal{F}_{\mathrm{DrMMD}} F DrMMD
Remark 4.13. Theorem 4.11 shows that for fixed μ \mu μ ∇ ϕ \nabla\phi ∇ ϕ λ \lambda λ ⟨ Hess F D r M M D | μ ∇ ϕ , ∇ ϕ ⟩ L 2 ( μ ) > 0 \left\langle\operatorname{Hess}\mathcal{F}_{\mathrm{DrMMD}}\middle|_{\mu}\nabla\phi,\nabla\phi\right\rangle_{L^{2}(\mu)} > 0 ⟨ Hess F DrMMD μ ∇ ϕ , ∇ ϕ ⟩ L 2 ( μ ) > 0 μ \mu μ R ( λ , μ , ∇ ϕ ) R(\lambda,\mu,\nabla\phi) R ( λ , μ , ∇ ϕ ) λ → 0 \lambda\to0 λ → 0 π \pi π non-asymptotic near-global convergence.
Near-Global Convergence of DrMMD flow via Poincaré inequality
Even when a functional F \mathcal{F} F ( μ t ) t ≥ 0 (\mu_t)_{t \ge0} ( μ t ) t ≥ 0 π \pi π χ 2 \chi^2 χ 2 ( ν t ) t ≥ 0 (\nu_t)_{t \ge0} ( ν t ) t ≥ 0 π \pi π ( ν t ) t ≥ 0 (\nu_t)_{t \ge0} ( ν t ) t ≥ 0 π \pi π
KL ( ν T ∥ π ) ≤ exp ( − 2 T C P ) KL ( ν 0 ∥ π ) . (20) \operatorname{KL}(\nu_T \mathbin{\|} \pi) \le\exp\left(-\frac{2T}{C_P}\right)\operatorname{KL}(\nu_0 \mathbin{\|} \pi). \tag*{(20)} KL ( ν T ∥ π ) ≤ exp ( − C P 2 T ) KL ( ν 0 ∥ π ) . (20) Recall that π \pi π f : R d → R f : \mathbb{R}^d \to\mathbb{R} f : R d → R f , ∇ f ∈ L 2 ( π ) f, \nabla f \in L^2(\pi) f , ∇ f ∈ L 2 ( π ) C P > 0 C_P > 0 C P > 0
∫ f ( x ) 2 d π ( x ) − ( ∫ f ( x ) d π ( x ) ) 2 ≤ C P ∥ ∇ f ∥ L 2 ( π ) 2 . (21) \int f(x)^2 \, \mathrm{d}\pi(x) - \left(\int f(x) \, \mathrm{d}\pi(x)\right)^2 \le C_P \lVert\nabla f \rVert_{L^2(\pi)}^2. \tag*{(21)} ∫ f ( x ) 2 d π ( x ) − ( ∫ f ( x ) d π ( x ) ) 2 ≤ C P ∥ ∇ f ∥ L 2 ( π ) 2 . (21) The smallest constant C P C_P C P (21) holds is called the Poincaré constant. The Poincaré inequality is widely used for studying the convergence of Langevin diffusions ((Chewi et al., 2024)) and χ 2 \chi^2 χ 2 π \pi π
Given the interpolation property of DrMMD to χ 2 \chi^2 χ 2 (20) suggests investigating the convergence of the DrMMD flow ( μ t ) t ≥ 0 (\mu_t)_{t \ge0} ( μ t ) t ≥ 0 KL ( μ t ∥ π ) \operatorname{KL}(\mu_t\Vert\pi) KL ( μ t ∥ π )
Theorem 4.14 (KL control of the DrMMD flow). Suppose k k k π \pi π ( μ t ) t ≥ 0 (\mu_t)_{t\geq0} ( μ t ) t ≥ 0
π \pi π C P C_{\mathrm{P}} C P
μ t , π ≪ L d \mu_t,\pi\ll\mathcal{L}^d μ t , π ≪ L d
d μ t d π − 1 ∈ Ran ( T π r ) \frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}-1 \in\operatorname{Ran}(T_\pi^r) d π d μ t − 1 ∈ Ran ( T π r ) r > 0 r>0 r > 0 i.e., there exists q t ∈ L 2 ( π ) q_t\in L^2(\pi) q t ∈ L 2 ( π ) d μ t d π − 1 = T π r q t \frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}-1=T_\pi^r q_t d π d μ t − 1 = T π r q t
∥ ∇ ( log π ) ⊤ ∇ ( d μ t d π ) ∥ L 2 ( π ) ≤ J t \left\|\nabla(\log\pi)^\top\nabla\left(\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}\right)\right\|_{L^2(\pi)}\leq\mathcal{J}_t ∇ ( log π ) ⊤ ∇ ( d π d μ t ) L 2 ( π ) ≤ J t ∥ Δ ( d μ t d π ) ∥ L 2 ( π ) ≤ I t \left\|\Delta\left(\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}\right)\right\|_{L^2(\pi)}\leq\mathcal{I}_t Δ ( d π d μ t ) L 2 ( π ) ≤ I t
For all i = 1 , … , d i=1,\ldots,d i = 1 , … , d lim x → ∞ ( h μ t , π ∗ ( x ) − 2 d μ t d π ( x ) ) ( ∂ i d μ t d π ( x ) ) π ( x ) → 0 \lim_{x\to\infty}\left(h^*_{\mu_t,\pi}(x)-2\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}(x)\right)\left(\partial_i\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}(x)\right)\pi(x)\to0 lim x → ∞ ( h μ t , π ∗ ( x ) − 2 d π d μ t ( x ) ) ( ∂ i d π d μ t ( x ) ) π ( x ) → 0
Then, for any T ≥ 0 T\geq0 T ≥ 0
KL ( μ T ∥ π ) ≤ exp ( − 2 ( 1 + λ ) C P T ) KL ( μ 0 ∥ π ) + 4 ( 1 + λ ) λ r ∫ 0 T exp ( − 2 ( 1 + λ ) C P ( T − t ) ) ∥ q t ∥ L 2 ( π ) ( J t + I t ) d t . (22) \operatorname{KL}(\mu_T\Vert\pi)\leq\exp\left(-\frac{2(1+\lambda)}{C_{\mathrm{P}}}T\right)\operatorname{KL}(\mu_0\Vert\pi)
+4(1+\lambda)\lambda^r\int_0^T\exp\left(-\frac{2(1+\lambda)}{C_{\mathrm{P}}}(T-t)\right)\lVert q_t\rVert_{L^2(\pi)}(\mathcal{J}_t+\mathcal{I}_t)\,\mathrm{d}t. \tag*{(22)} KL ( μ T ∥ π ) ≤ exp ( − C P 2 ( 1 + λ ) T ) KL ( μ 0 ∥ π ) + 4 ( 1 + λ ) λ r ∫ 0 T exp ( − C P 2 ( 1 + λ ) ( T − t ) ) ∥ q t ∥ L 2 ( π ) ( J t + I t ) d t . (22) The proof, which can be found in Section B.1.8, leverages the fact that the DrMMD can approximate not only the χ 2 \chi^2 χ 2 Wasserstein gradient . The DrMMD’s approximation properties can be combined with functional inequalities to obtain an upper-bound for the continuous-time dissipation of KL divergence along the flow, given by:
d d t KL ( μ t ∥ π ) ≤ − 2 ( 1 + λ ) C P KL ( μ t ∥ π ) + 4 ( 1 + λ ) λ r ∥ q t ∥ L 2 ( π ) ( J t + I t ) ⏟ Approximation error , (23) \frac{\mathrm{d}}{\mathrm{d}t}\operatorname{KL}(\mu_t\Vert\pi)\leq-\frac{2(1+\lambda)}{C_{\mathrm{P}}}\operatorname{KL}(\mu_t\Vert\pi)+\underbrace{4(1+\lambda)\lambda^r\lVert q_t\rVert_{L^2(\pi)}(\mathcal{J}_t+\mathcal{I}_t)}_{\text{Approximation error}}, \tag*{(23)} d t d KL ( μ t ∥ π ) ≤ − C P 2 ( 1 + λ ) KL ( μ t ∥ π ) + Approximation error 4 ( 1 + λ ) λ r ∥ q t ∥ L 2 ( π ) ( J t + I t ) , (23) from which Theorem 4.14 follows upon applying the Gronwall’s lemma (Gronwall, 1919). The first term is strictly negative, while the second term is an approximation error term arising from DrMMD not perfectly matching the χ 2 \chi^2 χ 2 λ > 0 \lambda> 0 λ > 0 λ = 0 \lambda= 0 λ = 0 Theorem 4.14 recovers the exponential decay of KL divergence along χ 2 \chi^2 χ 2 (20) .
Corollary 4.16 (Near global convergence of the DrMMD flow). In addition to the assumptions of Theorem 4.14 , if ∥ q t ∥ L 2 ( π ) ≤ Q \lVert q_t\rVert_{L^2(\pi)} \le Q ∥ q t ∥ L 2 ( π ) ≤ Q J t ≤ J , I t ≤ I \mathcal{J}_t \le\mathcal{J}, \mathcal{I}_t \le\mathcal{I} J t ≤ J , I t ≤ I for all 0 ≤ t ≤ T 0 \le t \le T 0 ≤ t ≤ T where Q Q Q J \mathcal{J} J and I \mathcal{I} I are universal constants independent of λ \lambda λ then for any T ≥ 0 T \ge0 T ≥ 0
KL ( μ T ∥ π ) ≤ exp ( − 2 ( 1 + λ ) C P T ) KL ( μ 0 ∥ π ) + 2 λ r C P Q ( J + I ) . \operatorname{KL}(\mu_T \mathbin\Vert\pi) \le\exp\left(-\frac{2(1+\lambda)}{C_P}T\right)\operatorname{KL}(\mu_0 \mathbin\Vert\pi) + 2\lambda^{r}C_PQ(\mathcal{J}+\mathcal{I}). KL ( μ T ∥ π ) ≤ exp ( − C P 2 ( 1 + λ ) T ) KL ( μ 0 ∥ π ) + 2 λ r C P Q ( J + I ) . The proof of Theorem 4.16 follows directly from upper-bounding the second term of (22) with universal constants Q Q Q J \mathcal{J} J I \mathcal{I} I λ → 0 \lambda\to0 λ → 0 ∥ q t ∥ L 2 ( π ) ≤ Q \lVert q_t\rVert_{L^2(\pi)} \le Q ∥ q t ∥ L 2 ( π ) ≤ Q J t ≤ J \mathcal{J}_t \le\mathcal{J} J t ≤ J I t ≤ I \mathcal{I}_t \le\mathcal{I} I t ≤ I 0 ≤ t ≤ T 0 \le t \le T 0 ≤ t ≤ T O ( λ r ) O(\lambda^r) O ( λ r )
Therefore, in the continuous time regime, to have a smaller approximation error, it is beneficial to use a small (de)-regularization parameter λ \lambda λ χ 2 \chi^2 χ 2 λ \lambda λ C P C_P C P
Previously, Arbel et al. (2019) established (sublinear) global convergence of the MMD flow in terms of MMD distance by assuming that a Łojasiewicz inequality (or a variant of it if additionally performing noise injection, see Arbel et al., 2018, Proposition 8) holds along the flow. Our result thus complements that of Arbel et al. (2019) by showing that MMD-type functionals can achieve near-global convergence for targets satisfying a Poincaré inequality regardless of whether such inequalities hold, by studying their behavior in the f f f
may wonder why we focus on the gradient flow of DrMMD(⋅ ∥ π \cdot\Vert\pi ⋅ ∥ π π ∥ ⋅ \pi\Vert\cdot π ∥ ⋅ Theorem 4.14 shows that DrMMD(⋅ ∥ π \cdot\Vert\pi ⋅ ∥ π π \pi π π ∥ ⋅ \pi\Vert\cdot π ∥ ⋅ π \pi π (13) , rather than with respect to μ t \mu_t μ t
Computing the DrMMD flow exactly is intractable since the dynamics are in continuous time, and in practice, we often only have samples from π \pi π
Time-discretized DrMMD flow
In this section, we first construct and analyze the forward Euler scheme of (17) , a simple time-discretization of the DrMMD flow which we call DrMMD Descent. Compared to the DrMMD flow, the convergence of DrMMD descent is affected by an additional smoothness-related time-discretization error that blows up as λ \lambda λ § 6.4.2 an alternative discrete-time scheme that adapts the value of the regularization coefficient λ \lambda λ Adaptive DrMMD Descent . We show that this scheme converges in the KL divergence up to a barrier term that vanishes as the discretization step size goes to zero.
DrMMD Descent
The forward Euler discretization of the DrMMD flow (or DrMMD Descent in short) with step size γ > 0 \gamma> 0 γ > 0 ( μ n ) n ∈ N (\mu_n)_{n\in\mathbb{N}} ( μ n ) n ∈ N
μ n + 1 = ( Id − γ ( 1 + λ ) ∇ h μ n , π ∗ ) # μ n , μ n = 0 = μ 0 , (25) \mu_{n+1}=\left(\operatorname{Id}-\gamma(1+\lambda)\nabla h^{*}_{\mu_n,\pi}\right)_{\#}\mu_n,\qquad\mu_{n=0}=\mu_0, \tag*{(25)} μ n + 1 = ( Id − γ ( 1 + λ ) ∇ h μ n , π ∗ ) # μ n , μ n = 0 = μ 0 , (25) where h μ n , π ∗ = 2 ( Σ π + λ Id ) − 1 ( m μ n − m π ) h^{*}_{\mu_n,\pi}=2\left(\Sigma_\pi+\lambda\operatorname{Id}\right)^{-1}\left(m_{\mu_n}-m_\pi\right) h μ n , π ∗ = 2 ( Σ π + λ Id ) − 1 ( m μ n − m π ) γ \gamma γ 2
Next, we turn to study the convergence of DrMMD descent. One way is to treat DrMMD as MMD with a regularized kernel k ~ \tilde{k} k ~ λ \lambda λ F D r M M D \mathcal{F}_{\mathrm{DrMMD}} F DrMMD § 6.3 that exploits the interpolation property of DrMMD towards χ 2 \chi^2 χ 2 π \pi π λ \lambda λ
Proposition 4.18 (Descent lemma in KL). Suppose k k k π \pi π ( μ n ) n ∈ N (\mu_n)_{n\in\mathbb{N}} ( μ n ) n ∈ N
π \pi π C P C_P C P β \beta β π ∝ exp ( − V ) \pi\propto\exp(-V) π ∝ exp ( − V ) H V ⪯ β Id \operatorname{H}V \preceq\beta\operatorname{Id} H V ⪯ β Id
μ n , π ≪ L d \mu_n,\pi\ll\mathcal{L}^d μ n , π ≪ L d
d μ n d π − 1 ∈ Ran ( T π r ) \frac{\mathrm{d}\mu_n}{\mathrm{d}\pi}-1\in\operatorname{Ran}(T_\pi^r) d π d μ n − 1 ∈ Ran ( T π r ) r > 0 r>0 r > 0 q n ∈ L 2 ( π ) q_n\in L^2(\pi) q n ∈ L 2 ( π )
d μ n d π − 1 = T π r q n . \frac{\mathrm{d}\mu_n}{\mathrm{d}\pi}-1=T_\pi^r q_n. d π d μ n − 1 = T π r q n . ∥ q n ∥ L 2 ( π ) ≤ Q \lVert q_n\rVert_{L^2(\pi)} \leq Q ∥ q n ∥ L 2 ( π ) ≤ Q ∥ ∇ V ∇ ( d μ n d π ) ∥ L 2 ( π ) ≤ J \left\lVert\nabla V \nabla\left(\frac{d\mu_n}{d\pi}\right)\right\rVert_{L^2(\pi)} \leq\mathcal{J} ∇ V ∇ ( d π d μ n ) L 2 ( π ) ≤ J ∥ Δ ( d μ n d π ) ∥ L 2 ( π ) ≤ I \left\lVert\Delta\left(\frac{d\mu_n}{d\pi}\right)\right\rVert_{L^2(\pi)} \leq\mathcal{I} Δ ( d π d μ n ) L 2 ( π ) ≤ I n = 1 , … , n max n = 1,\ldots,n_{\max} n = 1 , … , n m a x
For all i = 1 , … , d i = 1,\ldots,d i = 1 , … , d lim x → ∞ ( h μ n , π ∗ ( x ) − 2 d μ n d π ( x ) ) ( ∂ i d μ n d π ( x ) ) π ( x ) → 0 \lim_{x\to\infty}\left(h_{\mu_n,\pi}^{*}(x) - 2\frac{d\mu_n}{d\pi}(x)\right)\left(\partial_i\frac{d\mu_n}{d\pi}(x)\right)\pi(x) \to0 lim x → ∞ ( h μ n , π ∗ ( x ) − 2 d π d μ n ( x ) ) ( ∂ i d π d μ n ( x ) ) π ( x ) → 0
There exists a constant 1 < ζ < 2 1 < \zeta< 2 1 < ζ < 2 n = 1 , … , n max n = 1,\ldots,n_{\max} n = 1 , … , n m a x γ \gamma γ
γ ≤ ζ − 1 2 ζ ( 1 + λ ) χ 2 ( μ n ∥ π ) K 2 d λ (26) \gamma\leq\frac{\zeta- 1}{2\zeta(1+\lambda)\sqrt{\chi^2(\mu_n\mathbin\Vert\pi)\frac{K_{2d}}{\lambda}}} \tag*{(26)} γ ≤ 2 ζ ( 1 + λ ) χ 2 ( μ n ∥ π ) λ K 2 d ζ − 1 (26) Then for all 0 ≤ n ≤ n max 0 \leq n \leq n_{\max} 0 ≤ n ≤ n m a x 0 < λ ≤ 1 0 < \lambda\leq1 0 < λ ≤ 1
KL ( μ n + 1 ∥ π ) − KL ( μ n ∥ π ) ≤ − 2 C P 2 χ 2 ( μ n ∥ π ) γ + 4 γ λ Q ( J + I ) ⏟ Approximation error + 8 γ 2 ( β + ζ 2 ) χ 2 ( μ n ∥ π ) K 1 d + K 2 d λ . (27) \begin{aligned}
\operatorname{KL}(\mu_{n+1}\mathbin\Vert\pi) - \operatorname{KL}(\mu_n\mathbin\Vert\pi)
&\leq-\frac{2}{C_P^2}\chi^2(\mu_n\mathbin\Vert\pi)\gamma\\
&\quad+ 4\gamma^\lambda Q(\mathcal{J}+\mathcal{I}) \\
&\qquad\qquad\underbrace{\hspace{3.2cm}}_{\text{Approximation error}} \\
&\quad+ 8\gamma^2(\beta+\zeta^2)\chi^2(\mu_n\mathbin\Vert\pi)\frac{K_{1d}+K_{2d}}{\lambda} . \tag*{(27)}
\end{aligned} KL ( μ n + 1 ∥ π ) − KL ( μ n ∥ π ) ≤ − C P 2 2 χ 2 ( μ n ∥ π ) γ + 4 γ λ Q ( J + I ) Approximation error + 8 γ 2 ( β + ζ 2 ) χ 2 ( μ n ∥ π ) λ K 1 d + K 2 d . (27) The proof is provided in Section B.1.9. Conditions 1-5 of Theorem 4.18 are similar to conditions 1-5 of Theorem 4.14, which assumes a functional inequality on the target π \pi π d μ n d π − 1 \frac{d\mu_n}{d\pi}-1 d π d μ n − 1 d μ n d π − 1 \frac{d\mu_n}{d\pi}-1 d π d μ n − 1 H V ⪯ β Id HV \preceq\beta\operatorname{Id} H V ⪯ β Id ∇ V \nabla V ∇ V s ∈ [ 0 , 1 ] s \in[0,1] s ∈ [ 0 , 1 ] γ \gamma γ γ \gamma γ Theorem 4.21 and Theorem 4.22 are presented.
Adaptive DrMMD Descent
The DrMMD flow and descent dynamics are defined for a value of λ \lambda λ λ \lambda λ λ \lambda λ (27) . This sequence, which we term Adaptive DrMMD descent , is given by
μ n + 1 = ( Id − γ ( 1 + λ ) ∇ h μ n , π ∗ ) # μ n , λ n = ( 2 γ χ 2 ( μ n ∥ π ) ( β + ζ 2 ) ( K 1 d + K 2 d ) Q ( J + I ) ) 1 r + 1 . (28) \begin{aligned}
\mu_{n+1} &= \left(\operatorname{Id} - \gamma(1+\lambda)\nabla h^{*}_{\mu_n,\pi}\right)_{\#}\mu_n,\\
\lambda_n &= \left(2\gamma\chi^2(\mu_n\mathbin{\|}\pi)\frac{(\beta+\zeta^2)(K_{1d}+K_{2d})}{Q(\mathcal{J}+\mathcal{I})}\right)^{\frac{1}{r+1}}. \tag*{(28)}
\end{aligned} μ n + 1 λ n = ( Id − γ ( 1 + λ ) ∇ h μ n , π ∗ ) # μ n , = ( 2 γ χ 2 ( μ n ∥ π ) Q ( J + I ) ( β + ζ 2 ) ( K 1 d + K 2 d ) ) r + 1 1 . (28) The particular choice of λ n \lambda_n λ n (27) . This optimal choice indicates that λ n \lambda_n λ n χ 2 ( μ n ∥ π ) \chi^2(\mu_n\Vert\pi) χ 2 ( μ n ∥ π ) λ n \lambda_n λ n μ n \mu_n μ n π \pi π λ n \lambda_n λ n χ 2 \chi^2 χ 2 μ n \mu_n μ n π \pi π V V V β \beta β λ n \lambda_n λ n V V V § 6.4.1 , since the Wasserstein gradient updating μ n \mu_n μ n (28) constitutes a significant departure from related works in Wasserstein gradient descent ((Glaser et al., 2021; Arbel et al., 2019; Korba et al., 2021; Hertrich et al., 2024b, 2023; Chewi et al., 2020)).
Remark 4.20 (Adaptive kernel). Recent applications of MMD-based generative modeling algorithms with adaptive kernels (in particular, time-dependent kernel hyperparameters) demonstrate improved empirical performance over fixed kernels in both Wasserstein gradient flow on MMD ((Galashov et al., 2025a)) and generative adversarial networks with an MMD critic ((Li et al., 2017; Arbel et al., 2018)): the latter can be related to gradient flow on the critic where μ n \mu_n μ n k ~ \widetilde{k} k λ n \lambda_n λ n π \pi π μ t \mu_t μ t
By leveraging the quasi-descent lemma in KL divergence in Theorem 4.18, we are able to establish the following theorem, which provides a near-global convergence result of the Adaptive DrMMD gradient descent iterates in KL divergence.
Theorem 4.21 (Near-global convergence of adaptive DrMMD gradient descent). Suppose k k k Assumptions 4.1 and 4.2 and K ≤ 1 K \le1 K ≤ 1 suppose the target π \pi π ( μ n ) n ∈ N (\mu_n)_{n\in\mathbb{N}} ( μ n ) n ∈ N
π \pi π C P C_P C P β \beta β π ∝ exp ( − V ) \pi\propto\exp(-V) π ∝ exp ( − V ) H V ≤ β I d \mathbf{H}V \le\beta\mathbf{I}d H V ≤ β I d
μ n , π ≪ L d \mu_n,\pi\ll\mathcal{L}^d μ n , π ≪ L d
d μ n d π − 1 ∈ Ran ( T π r ) \frac{\mathrm{d}\mu_n}{\mathrm{d}\pi}-1 \in\operatorname{Ran}(T_\pi^r) d π d μ n − 1 ∈ Ran ( T π r ) r > 0 r>0 r > 0 q n ∈ L 2 ( π ) q_n \in L^2(\pi) q n ∈ L 2 ( π ) d μ n d π − 1 = T π r q n \frac{\mathrm{d}\mu_n}{\mathrm{d}\pi}-1=T_\pi^r q_n d π d μ n − 1 = T π r q n
∥ q n ∥ L 2 ( π ) ≤ Q \lVert q_n\rVert_{L^2(\pi)}\le Q ∥ q n ∥ L 2 ( π ) ≤ Q ∥ ∇ V ∇ ( d μ n d π ) ∥ L 2 ( π ) ≤ J \left\lVert\nabla V\nabla\left(\frac{\mathrm{d}\mu_n}{\mathrm{d}\pi}\right)\right\rVert_{L^2(\pi)}\le\mathcal{J} ∇ V ∇ ( d π d μ n ) L 2 ( π ) ≤ J ∥ Δ ( d μ n d π ) ∥ L 2 ( π ) ≤ I \left\lVert\Delta\left(\frac{\mathrm{d}\mu_n}{\mathrm{d}\pi}\right)\right\rVert_{L^2(\pi)}\le\mathcal{I} Δ ( d π d μ n ) L 2 ( π ) ≤ I n = 1 , … , n max n=1,\ldots,n_{\max} n = 1 , … , n m a x
For all i = 1 , … , d i=1,\ldots,d i = 1 , … , d lim x → ∞ ( h μ n , π ∗ ( x ) − 2 d μ n d π ( x ) ) ( ∂ i d μ n d π ( x ) ) π ( x ) → 0 \lim_{x\to\infty}\left(h_{\mu_n,\pi}^{*}(x)-2\frac{\mathrm{d}\mu_n}{\mathrm{d}\pi}(x)\right)\left(\partial_i\frac{\mathrm{d}\mu_n}{\mathrm{d}\pi}(x)\right)\pi(x)\to0 lim x → ∞ ( h μ n , π ∗ ( x ) − 2 d π d μ n ( x ) ) ( ∂ i d π d μ n ( x ) ) π ( x ) → 0
There exists a constant 1 < ζ < 2 1<\zeta<2 1 < ζ < 2 γ \gamma γ
γ ≤ 1 8 ( ζ − 1 ζ ) 2 r + 2 2 r + 1 ( 1 Q ) ( 1 J + I ) 1 2 r + 1 ( 1 K 2 d β + ζ 2 ) r 2 r + 1 ∧ 1 4 ζ − 1 ζ 1 Q 2 K 2 d ∧ C P 2 ∧ 1. (29) \begin{aligned}
\gamma\le{}& \frac{1}{8}\left(\frac{\zeta-1}{\zeta}\right)^{\frac{2r+2}{2r+1}}\left(\frac{1}{Q}\right)\left(\frac{1}{\mathcal{J}+\mathcal{I}}\right)^{\frac{1}{2r+1}}\left(\frac{1}{K_{2d}\beta+\zeta^2}\right)^{\frac{r}{2r+1}} \\
&{}\wedge\frac{1}{4}\frac{\zeta-1}{\zeta}\frac{1}{Q^2K_{2d}}\wedge\frac{C_P}{2}\wedge1. \tag*{(29)}
\end{aligned} γ ≤ 8 1 ( ζ ζ − 1 ) 2 r + 1 2 r + 2 ( Q 1 ) ( J + I 1 ) 2 r + 1 1 ( K 2 d β + ζ 2 1 ) 2 r + 1 r ∧ 4 1 ζ ζ − 1 Q 2 K 2 d 1 ∧ 2 C P ∧ 1. (29) Then by taking parameter λ n = ( 2 γ χ 2 ( μ n ∥ π ) ( β + ζ 2 ) ( K 1 d + K 2 d ) Q ( J + I ) ) 1 r + 1 ∧ 1 \lambda_n=\left(2\gamma\chi^2(\mu_n\Vert\pi)^{\frac{(\beta+\zeta^2)(K_{1d}+K_{2d})}{Q(\mathcal{J}+\mathcal{I})}}\right)^{\frac{1}{r+1}}\wedge1 λ n = ( 2 γ χ 2 ( μ n ∥ π ) Q ( J + I ) ( β + ζ 2 ) ( K 1 d + K 2 d ) ) r + 1 1 ∧ 1
KL ( μ n max ∥ π ) ≤ exp ( − 2 n max γ C P ) KL ( μ 0 ∥ π ) + 4 γ r r + 1 C P Q 2 r + 1 r + 1 ( ( K 1 d + K 2 d ) ( β + ζ 2 ) ) r r + 1 ( J + I ) 1 r + 1 . (30) \begin{aligned}
\operatorname{KL}(\mu_{n_{\max}}\Vert\pi) \le{}& \exp\left(-\frac{2n_{\max}\gamma}{C_P}\right)\operatorname{KL}(\mu_0\Vert\pi) \\
&{}+4\gamma^{\frac{r}{r+1}}C_PQ^{\frac{2r+1}{r+1}}\left((K_{1d}+K_{2d})(\beta+\zeta^2)\right)^{\frac{r}{r+1}}(\mathcal{J}+\mathcal{I})^{\frac{1}{r+1}}. \tag*{(30)}
\end{aligned} KL ( μ n m a x ∥ π ) ≤ exp ( − C P 2 n m a x γ ) KL ( μ 0 ∥ π ) + 4 γ r + 1 r C P Q r + 1 2 r + 1 ( ( K 1 d + K 2 d ) ( β + ζ 2 ) ) r + 1 r ( J + I ) r + 1 1 . (30) The proof can be found in Section B.1.10. The conditions 1-5 of Theorem 4.21 are the same as conditions 1-5 of Theorem 4.18. Since the RHS of (29) are all constants, condition 6 is satisfied when the step size γ \gamma γ
The implication of Theorem 4.21 is that DrMMD gradient descent exhibits an exponential rate of convergence (linear convergence) in terms of KL divergence up to an extra barrier of order O ( γ r r + 1 ) O(\gamma^{\frac{r}{r+1}}) O ( γ r + 1 r ) λ n \lambda_n λ n Theorem 4.21 is reminiscent of the convergence result of the Langevin Monte Carlo sampling algorithm, whose KL divergence also decreases exponentially up to an extra barrier, but of order O ( γ ) O(\gamma) O ( γ ) Theorem 4.14 , in which the barrier can be made arbitrarily small by taking small enough regularization, taking the step size γ \gamma γ Theorem 4.21 to be arbitrarily small will significantly impact the rate of convergence, even though it is exponential in terms of n max n_{\max} n m a x
Theorem 4.22 (Global convergence of DrMMD gradient descent). Suppose that k k k ( μ n ) n ∈ N (\mu_n)_{n\in\mathbb{N}} ( μ n ) n ∈ N π \pi π λ n \lambda_n λ n γ n \gamma_n γ n γ n \gamma_n γ n
γ n ≤ 1 ( K 1 d + K 2 d ) ( β + ζ 2 ) ( 1 Q ( J + I ) ) 1 r ( 1 8 C P ) r + 1 r χ 2 ( μ n ∥ π ) 1 r , (31) \gamma_n \le\frac{1}{(K_{1d}+K_{2d})(\beta+\zeta^2)}\left(\frac{1}{Q(\mathcal{J}+\mathcal{I})}\right)^{\frac{1}{r}}\left(\frac{1}{8C_P}\right)^{\frac{r+1}{r}}\chi^2(\mu_n\Vert\pi)^{\frac{1}{r}}, \tag*{(31)} γ n ≤ ( K 1 d + K 2 d ) ( β + ζ 2 ) 1 ( Q ( J + I ) 1 ) r 1 ( 8 C P 1 ) r r + 1 χ 2 ( μ n ∥ π ) r 1 , (31) for all n = 1 , ⋯ , n max n=1,\cdots,n_{\max} n = 1 , ⋯ , n m a x
KL ( μ n max ∥ π ) ≤ ∏ n = 1 n max ( 1 − 1 C P γ n ) KL ( μ 0 ∥ π ) . (32) \operatorname{KL}(\mu_{n_{\max}}\Vert\pi) \le\prod_{n=1}^{n_{\max}}\left(1-\frac{1}{C_P}\gamma_n\right)\operatorname{KL}(\mu_0\Vert\pi). \tag*{(32)} KL ( μ n m a x ∥ π ) ≤ n = 1 ∏ n m a x ( 1 − C P 1 γ n ) KL ( μ 0 ∥ π ) . (32) The proof can be found in Section B.1.11. Compared with (30) , (32) provides a cleaner upper bound without the barrier term and leads to global convergence.
complexity in this case because the growth rate of { n s } s ≥ 1 \{n_s\}_{s\ge1} { n s } s ≥ 1
Remark 4.24 (Step size γ \gamma γ γ \gamma γ (29) and Theorem 4.22 imposes an additional condition in (31) . These conditions subsume the condition (26) on step size in Theorem 4.18. (See derivations in Section B.1.10.) The conditions (29) and (31) become more stringent as the potential V V V β \beta β r r r Q Q Q J \mathcal{J} J I \mathcal{I} I
Practical adaptive DrMMD particle descent
In practice, we are given fixed target samples { x ( i ) } i = 1 M \{x^{(i)}\}_{i=1}^{M} { x ( i ) } i = 1 M { y 0 ( i ) } i = 1 N \{y_0^{(i)}\}_{i=1}^{N} { y 0 ( i ) } i = 1 N π ^ = M − 1 ∑ i = 1 M δ x ( i ) \hat{\pi}=M^{-1}\sum_{i=1}^{M}\delta_{x^{(i)}} π ^ = M − 1 ∑ i = 1 M δ x ( i ) μ ^ n = N − 1 ∑ i = 1 N δ y n ( i ) \hat{\mu}_n=N^{-1}\sum_{i=1}^{N}\delta_{y_n^{(i)}} μ ^ n = N − 1 ∑ i = 1 N δ y n ( i ) h μ ^ n , π ^ ∗ h^{*}_{\hat{\mu}_n,\hat{\pi}} h μ ^ n , π ^ ∗
As indicated in (28) , the regularization parameter λ n \lambda_n λ n χ 2 ( μ n ∥ π ) 1 r + 1 \chi^2(\mu_n\mathbin{\|}\pi)^{\frac{1}{r+1}} χ 2 ( μ n ∥ π ) r + 1 1 χ 2 ( μ n ∥ π ) \chi^2(\mu_n\mathbin{\|}\pi) χ 2 ( μ n ∥ π ) r r r χ 2 ( μ n ∥ π ) \chi^2(\mu_n\mathbin{\|}\pi) χ 2 ( μ n ∥ π ) DrMMD ( μ ^ n ∥ π ^ ) \operatorname{DrMMD}(\hat{\mu}_n\mathbin{\|}\hat{\pi}) DrMMD ( μ ^ n ∥ π ^ ) χ 2 ( μ n ∥ π ) \chi^2(\mu_n\mathbin{\|}\pi) χ 2 ( μ n ∥ π ) r r r { 0.1 , 0.5 , 1.0 } \{0.1, 0.5, 1.0\} { 0.1 , 0.5 , 1.0 } γ \gamma γ (29) , which contains several constants that cannot generally be computed. In practice, our approach has been to select γ \gamma γ
This proxy avoids density estimation and permits adaptive particle updates even when the empirical measures have different supports. Regularization also Algorithm 4.1 DrMMD particle descent (Chen et al. (2025c))
Input: Target samples { x ( i ) } i = 1 M ∼ π \{x^{(i)}\}_{i=1}^{M} \sim\pi { x ( i ) } i = 1 M ∼ π { y 0 ( i ) } i = 1 N ∼ μ 0 \{y^{(i)}_0\}_{i=1}^{N} \sim\mu_0 { y 0 ( i ) } i = 1 N ∼ μ 0 γ \gamma γ λ 0 \lambda_0 λ 0 n max n_{\max} n m a x r r r
for n = 0 n = 0 n = 0 n max − 1 n_{\max}-1 n m a x − 1 do
Compute witness function h μ ^ n , π ^ ∗ h^{*}_{\hat{\mu}_n,\hat{\pi}} h μ ^ n , π ^ ∗
Compute DrMMD( μ ^ n , π ^ ) (\hat{\mu}_n,\hat{\pi}) ( μ ^ n , π ^ ) h μ ^ n , π ^ ∗ h^{*}_{\hat{\mu}_n,\hat{\pi}} h μ ^ n , π ^ ∗
Rescale regularization coefficient λ n ∝ DrMMD ( μ ^ n , π ^ ) 1 / ( r + 1 ) \lambda_n \propto\operatorname{DrMMD}(\hat{\mu}_n,\hat{\pi})^{1/(r+1)} λ n ∝ DrMMD ( μ ^ n , π ^ ) 1/ ( r + 1 )
Update particles, for i = 1 , … , N i = 1,\ldots,N i = 1 , … , N
y n + 1 ( i ) = y n ( i ) − γ ( 1 + λ n ) ∇ h μ ^ n , π ^ ∗ ( y n ( i ) ) . y^{(i)}_{n+1}=y^{(i)}_n-\gamma(1+\lambda_n)\nabla h^{*}_{\hat{\mu}_n,\hat{\pi}}(y^{(i)}_n). y n + 1 ( i ) = y n ( i ) − γ ( 1 + λ n ) ∇ h μ ^ n , π ^ ∗ ( y n ( i ) ) . end for
Output: { y n max ( i ) } i = 1 N \{y^{(i)}_{n_{\max}}\}_{i=1}^{N} { y n m a x ( i ) } i = 1 N
stabilizes the kernel linear system; in the experiments, we enforce the positive lower bound λ ~ = 10 − 3 \tilde{\lambda}=10^{-3} λ ~ = 1 0 − 3
The substitution is nevertheless a practical approximation to the theoretical schedule. At fixed positive λ \lambda λ χ 2 \chi^2 χ 2 χ 2 \chi^2 χ 2 λ n \lambda_n λ n λ n → 0 \lambda_n \to0 λ n → 0 Theorems 4.21 and 4.22 , which assume the population schedule and additional regularity, do not automatically apply to this empirical adaptive rule. Its effectiveness in the experiments is empirical evidence rather than a guarantee for arbitrary singular targets.
Experiments
In this section, we demonstrate the superior empirical performance of the proposed DrMMD descent in various experimental settings.
Three ring experiment
We follow the experimental set-up in Glaser et al. (2021) in which the target distribution π ( ∙ ) \pi(\bullet) π ( ∙ ) R 2 \mathbb{R}^{2} R 2 μ 0 ( ∙ ) \mu_{0}(\bullet) μ 0 ( ∙ ) f f f π \pi π μ 0 \mu_{0} μ 0 χ 2 \chi^{2} χ 2
We sample N = M = 300 N = M = 300 N = M = 300 λ \lambda λ n max = 100 , 000 n_{\max} = 100,000 n m a x = 100 , 000 k ( x , x ′ ) = exp ( − 0.5 ∥ x − x ′ ∥ 2 / l 2 ) k(x,x') = \exp(-0.5\lVert x - x'\rVert^{2}/l^{2}) k ( x , x ′ ) = exp ( − 0.5 ∥ x − x ′ ∥ 2 / l 2 ) l = 0.3 l = 0.3 l = 0.3 γ = 10 − 2 \gamma= 10^{-2} γ = 1 0 − 2 γ = 10 − 3 \gamma= 10^{-3} γ = 1 0 − 3 λ ~ = 10 − 3 \tilde{\lambda} = 10^{-3} λ ~ = 1 0 − 3 r r r { 0.1 , 0.5 , 1.0 } \{0.1, 0.5, 1.0\} { 0.1 , 0.5 , 1.0 }
From Figure 4.2 Left and Middle, we can see that DrMMD descent outperforms MMD, KALE, and χ 2 \chi^{2} χ 2 π \pi π Figure 4.1 is an animation plot visualizing the evolution of particles under these descent schemes, which demonstrates that both KALE and DrMMD descent are sensitive to the mismatch of support and stay concentrated in the support of the target π \pi π π \pi π χ 2 \chi^{2} χ 2 χ 2 \chi^{2} χ 2 f f f χ 2 \chi^{2} χ 2
Figure 4.1. Animation of MMD, KALE, χ 2 \chi^{2} χ 2
Figure 4.2. Left & Middle: Comparison of DrMMD descent with adaptive λ \lambda λ π \pi π Right: Comparison of MMD descent with and without noise injection, DrMMD descent with and without noise injection on training student/teacher networks in terms of validation MMD2 ^2 2
Gradient flow for training student/teacher networks
Next, we consider a large-scale setting following Arbel et al. (2019), where a student network is trained to imitate the outputs of a teacher network. We consider a two-layer neural network of the form
ψ ( z , x ) = G ( b 1 + W 1 σ ( W 0 z + b 0 ) ) , \psi(z,x) = G\left(b^{1} + W^{1}\sigma\left(W^{0}z + b^{0}\right)\right), ψ ( z , x ) = G ( b 1 + W 1 σ ( W 0 z + b 0 ) ) , where σ \sigma σ x x x ( b 1 , W 1 , b 0 , W 0 ) ∈ R d (b^{1}, W^{1}, b^{0}, W^{0}) \in\mathbb{R}^{d} ( b 1 , W 1 , b 0 , W 0 ) ∈ R d G G G G : R → R , x ↦ exp ( − 1 4 x 2 ) G : \mathbb{R} \to\mathbb{R}, x \mapsto\exp\left(-\frac{1}{4}x^{2}\right) G : R → R , x ↦ exp ( − 4 1 x 2 ) Ψ T ( z , π ) = ∫ ψ ( z , x ) d π ( x ) \Psi^{T}(z,\pi)=\int\psi(z,x)d\pi(x) Ψ T ( z , π ) = ∫ ψ ( z , x ) d π ( x ) π \pi π Ψ S ( z , μ ) = ∫ ψ ( z , x ) d μ ( x ) \Psi^{S}(z,\mu)=\int\psi(z,x)d\mu(x) Ψ S ( z , μ ) = ∫ ψ ( z , x ) d μ ( x ) μ \mu μ μ \mu μ π \pi π 3
min μ ∈ P 2 ( R d ) E z ∼ P d a t a ( Ψ T ( z , π ) − Ψ S ( z , μ ) ) 2 , (33) \min_{\mu\in\mathcal{P}_{2}(\mathbb{R}^{d})}\mathbb{E}_{z\sim\mathbb{P}_{\mathrm{data}}}\left(\Psi^{T}(z,\pi)-\Psi^{S}(z,\mu)\right)^{2}, \tag*{(33)} μ ∈ P 2 ( R d ) min E z ∼ P data ( Ψ T ( z , π ) − Ψ S ( z , μ ) ) 2 , (33) where P d a t a \mathbb{P}_{\mathrm{data}} P data
k ( x , x ′ ) = E z ∼ P d a t a [ ψ ( z , x ) T ψ ( z , x ′ ) ] , k(x,x')=\mathbb{E}_{z\sim\mathbb{P}_{\mathrm{data}}}\left[\psi(z,x)^{\mathsf{T}}\psi(z,x')\right], k ( x , x ′ ) = E z ∼ P data [ ψ ( z , x ) T ψ ( z , x ′ ) ] , then the objective of (33) can be equivalently expressed as
min μ ∈ P 2 ( R d ) ∬ X k ( x , x ′ ) d ( π − μ ) ( x ) d ( π − μ ) ( x ′ ) , \min_{\mu\in\mathcal{P}_{2}(\mathbb{R}^{d})}\iint_{\mathcal{X}} k(x,x')d(\pi-\mu)(x)d(\pi-\mu)(x'), μ ∈ P 2 ( R d ) min ∬ X k ( x , x ′ ) d ( π − μ ) ( x ) d ( π − μ ) ( x ′ ) , which is precisely the MMD ( μ ∥ π ) 2 \operatorname{MMD}(\mu\|\pi)^{2} MMD ( μ ∥ π ) 2 k k k G ( x ) = exp ( − 1 4 x 2 ) G(x)=\exp\left(-\frac{1}{4}x^{2}\right) G ( x ) = exp ( − 4 1 x 2 ) k k k
Therefore, the training of the student network with objective (4.21) can be treated as an optimization problem of MMD 2 \operatorname{MMD}^{2} MMD 2 P 2 ( R d ) \mathcal{P}_{2}(\mathbb{R}^{d}) P 2 ( R d ) geodesically convex; therefore, noise injection has been proposed to escape these local optima.
With support from Theorems 4.14 and Theorem 4.21 on the convergence of DrMMD flow and its descent scheme, we propose to minimize DrMMD ( μ ∥ π ) \operatorname{DrMMD}(\mu\|\pi) DrMMD ( μ ∥ π ) MMD ( μ ∥ π ) 2 \operatorname{MMD}(\mu\|\pi)^2 MMD ( μ ∥ π ) 2 (33) , the favorable convergence performance of DrMMD descent should result in a smaller MMD ( μ ∥ π ) 2 \operatorname{MMD}(\mu\|\pi)^2 MMD ( μ ∥ π ) 2
In our experimental setting, we are given M = 10 M=10 M = 10 x ( 1 ) , … , x ( M ) x^{(1)},\ldots,x^{(M)} x ( 1 ) , … , x ( M ) π = N ( 0 , I d ) \pi=\mathcal{N}(0,\mathrm{Id}) π = N ( 0 , Id ) N = 1000 N=1000 N = 1000 y 0 ( 1 ) , … , y 0 ( N ) y_0^{(1)},\ldots,y_0^{(N)} y 0 ( 1 ) , … , y 0 ( N ) μ 0 = N ( 0 , 10 − 3 I d ) \mu_0=\mathcal{N}(0,10^{-3}\mathrm{Id}) μ 0 = N ( 0 , 1 0 − 3 Id ) λ 0 = 0.1 \lambda_0=0.1 λ 0 = 0.1 γ = 0.1 \gamma=0.1 γ = 0.1 λ ~ = 10 − 3 \widetilde{\lambda}=10^{-3} λ = 1 0 − 3 r r r { 0.1 , 0.5 , 1.0 } \{0.1,0.5,1.0\} { 0.1 , 0.5 , 1.0 } P d a t a \mathbb{P}_{\mathrm{data}} P data R p \mathbb{R}^p R p p = 50 p=50 p = 50 P d a t a \mathbb{P}_{\mathrm{data}} P data k ( x , x ′ ) = E z ∼ P d a t a ⟨ ψ ( z , x ) , ψ ( z , x ′ ) ⟩ k(x,x')=\mathbb{E}_{z\sim\mathbb{P}_{\mathrm{data}}}\langle\psi(z,x),\psi(z,x')\rangle k ( x , x ′ ) = E z ∼ P data ⟨ ψ ( z , x ) , ψ ( z , x ′ )⟩ n max = 15 , 000 n_{\max}=15,000 n m a x = 15 , 000 MMD 2 ( μ n max ∥ π ) \operatorname{MMD}^2(\mu_{n_{\max}}\|\pi) MMD 2 ( μ n m a x ∥ π ) k k k
In Figure 4.2 Right, we report the performance of MMD descent (with and without noise injection) along with the DrMMD descent (with and without noise injection) in terms of MMD distance on the validation dataset. We can see that the DrMMD descent does not get stuck in a local optimum, and leads to much lower validation MMD ( μ n max ∥ π ) 2 \operatorname{MMD}(\mu_{n_{\max}}\|\pi)^2 MMD ( μ n m a x ∥ π ) 2 Theorem 4.21 , the kernel k k k π \pi π
The code to reproduce all the experiments can be found in the following GitHub repository. https://github.com/hudsonchen/DrMMD.
Discussion
In this paper, we introduced (de)-regularization of the MMD (called DrMMD) and its associated Wasserstein gradient flow. As an interpolation between the MMD and χ 2 \chi^2 χ 2 π \pi π f f f χ 2 \chi^2 χ 2 π \pi π χ 2 \chi^2 χ 2 π \pi π
Minimum MMD for Statistical Inference
The results in this chapter were published in the following paper:
Sophia Seulkee Kang, Louis Sharrock, Xiaoyuan Cheng, François-Xavier Briol, and Zonghao Chen. A Gradient Flow Perspective on Minimum MMD Estimation. arXiv:2607.03871.
All theoretical results in this chapter are due to the author. Experiments are carried out by Sophia Seulkee Kang and are included to support the theory. As the last author, the author also played a supervisory role in guiding the overall research direction and presentation of the work.
Introduction
Parameter estimation seeks to recover the unknown parameters governing a data-generating mechanism from observed data. Given a parametric family of statistical models { P θ : θ ∈ Θ } \{P_\theta: \theta\in\Theta\} { P θ : θ ∈ Θ } Θ ⊆ R p \Theta\subseteq\mathbb{R}^p Θ ⊆ R p Q \mathbb{Q} Q
Recently, a growing body of work has considered using the maximum mean discrepancy (MMD), a metric based on embeddings of probability measures into a reproducing kernel Hilbert space (Gretton et al., 2012; Muandet et al., 2017), leading to the so-called minimum MMD estimator (Briol et al., 2019a). This choice is attractive for both statistical and computational reasons. Statistically, minimum MMD estimators are robust to model misspecification, such as outlier contamination and covariate shift (Briol et al., 2019a; Chérief-Abdellatif and Alquier, 2020; Dellaporta and Damoulas, 2026). Computationally, they remain well defined even when the likelihood is unavailable or intractable; such settings are commonly referred to as likelihood-free inference (Niu et al., 2023a; Dellaporta et al., 2022; Bharti et al., 2023b). These favorable properties have also led to numerous applications of minimum MMD estimators, including for hypothesis testing (Key et al., 2025; Schrab et al., 2023; Brück et al., 2025; Fazeli-Asl et al., 2024), parametric regression (Alquier and Gerber, 2024), time-series modeling (Alquier et al., 2026), distributionally robust optimization (Dellaporta et al., 2026), generalizations of Bayesian inference (Dellaporta et al., 2022; Dellaporta and Damoulas, 2026; Chen et al., 2025a), calibration of stochastic simulation models in operations research (Su and Klabjan, 2025), copula modeling (Alquier et al., 2023), inference under missing data (Chérief-Abdellatif and Näf, 2025), and learning of summary statistics (Khoo et al., 2026).
Despite the growing popularity of these estimators, very little work has focused on solving the associated optimization problem
arg min θ ∈ Θ MMD 2 ( P θ , Q ) , \underset{\theta\in\Theta}{\arg\min}\operatorname{MMD}^{2}(\mathbb{P}_{\theta}, \mathbb{Q}), θ ∈ Θ arg min MMD 2 ( P θ , Q ) , compared to its accompanying statistical theory. Existing methods typically rely on (stochastic) gradient descent (Briol et al., 2019a; Chérief-Abdellatif and Alquier, 2020), including the recent R-package implementation of Alquier and Gerber (2025), and this can at best be expected to converge to stationary points due to the non-convexity of the objective. To the best of our knowledge, the only existing optimization guarantee is Proposition 5.2 of Chérief-Abdellatif and Alquier (2022), which assumes convexity of the objective, a condition that rarely holds in practice; see Figure 5.1 for an illustration on a simple two-mode Gaussian mixture model with unknown variances. Therefore, an open problem remains:
Figure 5.1. Illustrative example. Both the target distribution Q \mathbb{Q} Q P θ \mathbb{P}_{\theta} P θ θ = ( σ 1 , σ 2 ) \theta= (\sigma_1, \sigma_2) θ = ( σ 1 , σ 2 )
“Can we design a gradient-based optimization scheme for minimum MMD estimation with convergence guarantees beyond convex objectives?”
In this paper, we make progress on this problem by proposing a new descent scheme for minimizing the MMD objective, and prove that it can converge to a (not necessarily unique) global minimizer under explicit assumptions. Our approach builds on MMD gradient flow, a descent scheme for minimizing the squared MMD directly over the space of probability measures, without imposing a parametric model representation (Arbel et al., 2019). Existing MMD gradient flows have been studied primarily in the contexts of generative modelling (Galashov et al., 2025a; Hertrich et al., 2024b; Altekrüger et al., 2023; Hertrich et al., 2024a; Chen et al., 2025c; Tian et al., 2026) and sampling (Korba et al., 2021; Chen et al., 2025b; Tian et al., 2026), and are typically run with a fixed kernel throughout the dynamics. We depart from this fixed-kernel formulation by introducing an adaptive nonparametric MMD descent scheme ( P t ) t ∈ N (\mathbb{P}_t)_{t\in\mathbb{N}} ( P t ) t ∈ N ( ℓ t ) t ∈ N (\ell_t)_{t\in\mathbb{N}} ( ℓ t ) t ∈ N ℓ > 0 \ell> 0 ℓ > 0 lim t → ∞ MMD ( P t , Q ) = 0 \lim_{t\to\infty}\operatorname{MMD}(\mathbb{P}_t,\mathbb{Q}) = 0 lim t → ∞ MMD ( P t , Q ) = 0
Motivated by this global convergence result, we then design a descent scheme such that the parametric distributions ( P θ t ) t ∈ N (\mathbb{P}_{\theta_t})_{t\in\mathbb{N}} ( P θ t ) t ∈ N ( P t ) t ∈ N (\mathbb{P}_t)_{t\in\mathbb{N}} ( P t ) t ∈ N t t t θ \theta θ preconditioned gradient descent (PGD). We next prove in Theorem 5.10 that, under gradient dominance and projection residual conditions, the parametric models optimized under PGD, denoted as ( P θ t P G D ) t ∈ N (\mathbb{P}^{\mathrm{PGD}}_{\theta_t})_{t\in\mathbb{N}} ( P θ t PGD ) t ∈ N lim t → ∞ MMD ( P θ t P G D , Q ) = min θ ∈ Θ MMD ( P θ P G D , Q ) \lim_{t\to\infty}\operatorname{MMD}(\mathbb{P}^{\mathrm{PGD}}_{\theta_t},\mathbb{Q}) = \min_{\theta\in\Theta}\operatorname{MMD}(\mathbb{P}^{\mathrm{PGD}}_{\theta},\mathbb{Q}) lim t → ∞ MMD ( P θ t PGD , Q ) = min θ ∈ Θ MMD ( P θ PGD , Q )
Nonparametric MMD Descent with Adaptive Lengthscale
In this section, we first establish a convergence guarantee for nonparametric MMD descent with adaptive kernel lengthscales. We propose to initiate the MMD gradient flow algorithm with a large lengthscale, then to monotonically reduce it towards some target value as the number of iterations increases. This is known empirically to improve the convergence behavior of MMD flows (Chen et al., 2021; Galashov et al., 2025a; Chen et al., 2025c), and to even be useful for other statistical applications of the MMD (Schrab et al., 2023; Biggs et al., 2023). The intuition for adaptive lengthscale is analogous to that of annealing methods: adapting the objective function throughout the optimisation routine can significantly improve exploration. We also note that adaptive lengthscale will be straightforward to use directly in the context of parametric updates. Hence, the convergence guarantee proved here provides one foundation for the optimization analysis of parametric MMD descent in § 7.3 .
Let f ℓ , P , Q f_{\ell,\mathbb{P},\mathbb{Q}} f ℓ , P , Q P \mathbb{P} P Q \mathbb{Q} Q k ℓ k_\ell k ℓ MMD ℓ ( P , Q ) \operatorname{MMD}_{\ell}(\mathbb{P},\mathbb{Q}) MMD ℓ ( P , Q ) ℓ \ell ℓ ( ℓ t ) t ∈ N (\ell_t)_{t\in\mathbb{N}} ( ℓ t ) t ∈ N
P t + 1 = ( Id − γ t ∇ f ℓ t , P t , Q ) # P t . (34) \mathbb{P}_{t+1} = \left(\operatorname{Id} - \gamma_t \nabla f_{\ell_t,\mathbb{P}_t,\mathbb{Q}}\right)_{\#}\mathbb{P}_t. \tag*{(34)} P t + 1 = ( Id − γ t ∇ f ℓ t , P t , Q ) # P t . (34) In the remainder of this paper, we focus explicitly on the Gaussian and Matérn kernels, due to their widespread use for minimum MMD estimation (Briol et al., 2019a; Chérief-Abdellatif and Alquier, 2020); see Theorem C.15 for an extension to other kernels.
Assumption 5.1 (Kernel). Let k ℓ k_\ell k ℓ Gaussian kernel
k ℓ ( x , y ) = exp ( − ∥ x − y ∥ 2 2 ℓ 2 ) , k_\ell(x,y) = \exp\left(-\frac{\lVert x-y\rVert^2}{2\ell^2}\right), k ℓ ( x , y ) = exp ( − 2 ℓ 2 ∥ x − y ∥ 2 ) , or the Matérn kernel with smoothness parameter s > 1 s > 1 s > 1
k ℓ ( x , y ) = 2 1 − s Γ ( s ) ( 2 s ∥ x − y ∥ ℓ ) s K s ( 2 s ∥ x − y ∥ ℓ ) , k_{\ell}(x,y)=\frac{2^{1-s}}{\Gamma(s)}\left(\frac{\sqrt{2s}\lVert x-y\rVert}{\ell}\right)^{s}K_s\left(\frac{\sqrt{2s}\lVert x-y\rVert}{\ell}\right), k ℓ ( x , y ) = Γ ( s ) 2 1 − s ( ℓ 2 s ∥ x − y ∥ ) s K s ( ℓ 2 s ∥ x − y ∥ ) , Here, ℓ > 0 \ell> 0 ℓ > 0 Γ \Gamma Γ K s K_s K s
Theorem 5.2 (Global convergence of nonparametric MMD descent). Let kernel k k k Q ∈ P 2 ( R d ) \mathbb{Q} \in\mathcal{P}_2(\mathbb{R}^d) Q ∈ P 2 ( R d ) P 0 ∈ P 2 ( R d ) \mathbb{P}_0 \in\mathcal{P}_2(\mathbb{R}^d) P 0 ∈ P 2 ( R d ) ( P t ) t ∈ N (\mathbb{P}_t)_{t\in\mathbb{N}} ( P t ) t ∈ N (34) with lengthscale ℓ t \ell_t ℓ t C > 0 C > 0 C > 0 t t t ( γ t ) t ∈ N (\gamma_t)_{t\in\mathbb{N}} ( γ t ) t ∈ N γ t ∈ ( 0 , 1 2 C ) \gamma_t \in(0,\frac{1}{2C}) γ t ∈ ( 0 , 2 C 1 ) lim t → ∞ γ t = 0 \lim_{t\to\infty}\gamma_t = 0 lim t → ∞ γ t = 0 ∑ t = 0 ∞ γ t = ∞ \sum_{t=0}^{\infty}\gamma_t = \infty ∑ t = 0 ∞ γ t = ∞ ( ℓ t ) t ∈ N (\ell_t)_{t\in\mathbb{N}} ( ℓ t ) t ∈ N lim t → ∞ ℓ t = ℓ > 0 \lim_{t\to\infty}\ell_t = \ell> 0 lim t → ∞ ℓ t = ℓ > 0 ( ℓ t ℓ − 1 ) d ∥ ∇ f ℓ t , P t , Q ∥ L 2 d ( P t ) 2 ≥ C MMD ℓ 2 ( P t , Q ) (\ell_t\ell^{-1})^d\lVert\nabla f_{\ell_t,\mathbb{P}_t,\mathbb{Q}}\rVert_{L_2^d(\mathbb{P}_t)}^2 \ge C\operatorname{MMD}_{\ell}^2(\mathbb{P}_t,\mathbb{Q}) ( ℓ t ℓ − 1 ) d ∥ ∇ f ℓ t , P t , Q ∥ L 2 d ( P t ) 2 ≥ C MMD ℓ 2 ( P t , Q ) t ∈ N t \in\mathbb{N} t ∈ N lim t → ∞ MMD ℓ 2 ( P t , Q ) = 0 \lim_{t\to\infty}\operatorname{MMD}_{\ell}^2(\mathbb{P}_t,\mathbb{Q}) = 0 lim t → ∞ MMD ℓ 2 ( P t , Q ) = 0
The proof is in Section C.1. Theorem 5.2 shows that, under its step-size and trajectory-wise gradient-dominance assumptions, an appropriate kernel lengthscale sequence ensures that P t \mathbb{P}_t P t Q \mathbb{Q} Q MMD ℓ 2 ( P t , Q ) \operatorname{MMD}_{\ell}^2(\mathbb{P}_t,\mathbb{Q}) MMD ℓ 2 ( P t , Q ) ( ℓ t ) t ∈ N (\ell_t)_{t\in\mathbb{N}} ( ℓ t ) t ∈ N
The condition of a diminishing yet cumulatively unbounded step size is standard in convex optimization (e.g., (Moulines and Bach, 2011)). The lengthscale condition can be interpreted as a trajectory-wise gradient dominance condition, and can be verified a posteriori ; see § 7.4 . Since ℓ t \ell_t ℓ t ℓ \ell ℓ P t \mathbb{P}_t P t Q \mathbb{Q} Q ℓ t / ℓ > 1 \ell_t/\ell> 1 ℓ t / ℓ > 1 d d d C C C realized trajectory.
Parametric MMD Descent with Adaptive Lengthscale
In this section, we propose a new approach for training parametric MMD models. The guiding principle is to design a parametric update that remains as faithful as possible to nonparametric MMD descent, and leverage the convergence result in Theorem 5.2 .
Given an unknown data generating distribution Q \mathbb{Q} Q { P θ : θ ∈ Θ } \{\mathbb{P}_{\theta} : \theta\in\Theta\} { P θ : θ ∈ Θ } Θ ⊆ R p \Theta\subseteq\mathbb{R}^{p} Θ ⊆ R p arg min θ ∈ Θ MMD 2 ( P θ , Q ) \operatorname*{arg\,min}_{\theta\in\Theta} \operatorname{MMD}^{2}(\mathbb{P}_{\theta}, \mathbb{Q}) arg min θ ∈ Θ MMD 2 ( P θ , Q ) ℓ > 0 \ell> 0 ℓ > 0 P θ = ( G θ ) # ρ \mathbb{P}_{\theta} = (G_{\theta})_{\#}\rho P θ = ( G θ ) # ρ ρ ∈ P 2 ( Z ) \rho\in\mathcal{P}_{2}(\mathcal{Z}) ρ ∈ P 2 ( Z ) Z ⊆ R q \mathcal{Z} \subseteq\mathbb{R}^{q} Z ⊆ R q G θ : Z → R d G_{\theta} : \mathcal{Z} \to\mathbb{R}^{d} G θ : Z → R d θ ∈ Θ \theta\in\Theta θ ∈ Θ Θ \Theta Θ q q q d d d P θ \mathbb{P}_{\theta} P θ θ \theta θ θ t + 1 = θ t − γ t Δ θ t ∈ Θ \theta_{t+1} = \theta_t - \gamma_t \Delta\theta_t \in\Theta θ t + 1 = θ t − γ t Δ θ t ∈ Θ t ∈ N t \in\mathbb{N} t ∈ N Θ \Theta Θ Θ \Theta Θ
P θ t + 1 = ( G θ t + 1 ) # ρ = ( G θ t − γ t Δ θ t ) # ρ , ∀ t ∈ N . (35) \mathbb{P}_{\theta_{t+1}} = (G_{\theta_{t+1}})_{\#}\rho= (G_{\theta_t-\gamma_t\Delta\theta_t})_{\#}\rho,\quad\forall t \in\mathbb{N}. \tag*{(35)} P θ t + 1 = ( G θ t + 1 ) # ρ = ( G θ t − γ t Δ θ t ) # ρ , ∀ t ∈ N . (35) Here, γ t > 0 \gamma_t > 0 γ t > 0 Δ θ t ∈ R p \Delta\theta_t \in\mathbb{R}^p Δ θ t ∈ R p Δ θ t \Delta\theta_t Δ θ t 1 2 MMD ℓ 2 ( P θ , Q ) \frac{1}{2}\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta}, \mathbb{Q}) 2 1 MMD ℓ 2 ( P θ , Q ) θ \theta θ
Δ θ t G D = ∇ θ [ 1 2 MMD ℓ 2 ( P θ , Q ) ] = E z ∼ ρ [ J θ G θ t ( z ) T ∇ f ℓ , P θ t , Q ( G θ t ( z ) ) ] , (36) \Delta\theta_t^{\mathrm{GD}} = \nabla_{\theta}\left[\frac{1}{2}\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta}, \mathbb{Q})\right] = \mathbb{E}_{z\sim\rho}\left[\mathbf{J}_{\theta}G_{\theta_t}(z)^{\mathsf{T}}\nabla f_{\ell,\mathbb{P}_{\theta_t},\mathbb{Q}}(G_{\theta_t}(z))\right], \tag*{(36)} Δ θ t GD = ∇ θ [ 2 1 MMD ℓ 2 ( P θ , Q ) ] = E z ∼ ρ [ J θ G θ t ( z ) T ∇ f ℓ , P θ t , Q ( G θ t ( z )) ] , (36) where J θ G θ t ( z ) ∈ R d × p \mathbf{J}_{\theta}G_{\theta_t}(z) \in\mathbb{R}^{d\times p} J θ G θ t ( z ) ∈ R d × p θ ↦ G θ ( z ) \theta\mapsto G_{\theta}(z) θ ↦ G θ ( z ) θ t \theta_t θ t ( P θ t G D ) t ∈ N (\mathbb{P}^{\mathrm{GD}}_{\theta_t})_{t\in\mathbb{N}} ( P θ t GD ) t ∈ N parametric MMD descent . Although this straightforward update scheme has been used in the majority of the literature, it remains unclear whether it can guarantee lim t → ∞ MMD ℓ ( P θ t G D , Q ) = min θ ∈ Θ MMD ℓ ( P θ , Q ) \lim_{t\to\infty}\operatorname{MMD}_{\ell}(\mathbb{P}^{\mathrm{GD}}_{\theta_t},\mathbb{Q}) = \min_{\theta\in\Theta}\operatorname{MMD}_{\ell}(\mathbb{P}_{\theta},\mathbb{Q}) lim t → ∞ MMD ℓ ( P θ t GD , Q ) = min θ ∈ Θ MMD ℓ ( P θ , Q )
Motivated by the global convergence of nonparametric MMD descent established in Theorem 5.2 , we introduce an alternative parameter update Δ θ \Delta\theta Δ θ MMD ℓ 2 ( P t + 1 , Q ) \operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{t+1},\mathbb{Q}) MMD ℓ 2 ( P t + 1 , Q ) MMD ℓ 2 ( P θ t + 1 , Q ) \operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta_{t+1}},\mathbb{Q}) MMD ℓ 2 ( P θ t + 1 , Q ) γ t \gamma_t γ t P t ≈ P θ t = ( G θ t ) # ρ \mathbb{P}_t \approx\mathbb{P}_{\theta_t} = (G_{\theta_t})_{\#}\rho P t ≈ P θ t = ( G θ t ) # ρ t t t t + 1 t+1 t + 1 γ t \gamma_t γ t
MMD ℓ 2 ( P t + 1 , Q ) = MMD ℓ 2 ( P t , Q ) − 2 γ t E z ∼ ρ [ ( ∇ f ℓ , P t , Q ( G θ t ( z ) ) T ∇ f ℓ , P t , Q ( G θ t ( z ) ) ) ] + O ( γ t 2 ) \operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{t+1},\mathbb{Q}) = \operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{t},\mathbb{Q}) - 2\gamma_t\mathbb{E}_{z\sim\rho}\left[\begin{pmatrix}\nabla f_{\ell,\mathbb{P}_t,\mathbb{Q}}(G_{\theta_t}(z))^{\mathsf{T}} \\ \nabla f_{\ell,\mathbb{P}_t,\mathbb{Q}}(G_{\theta_t}(z))\end{pmatrix}\right] + O(\gamma_t^2) MMD ℓ 2 ( P t + 1 , Q ) = MMD ℓ 2 ( P t , Q ) − 2 γ t E z ∼ ρ [ ( ∇ f ℓ , P t , Q ( G θ t ( z ) ) T ∇ f ℓ , P t , Q ( G θ t ( z )) ) ] + O ( γ t 2 ) MMD ℓ 2 ( P θ t + 1 , Q ) = MMD ℓ 2 ( P θ t , Q ) − 2 γ t E z ∼ ρ [ ( ∇ f ℓ , P θ t , Q ( G θ t ( z ) ) T J θ G θ t ( z ) Δ θ ) ] + O ( γ t 2 ) . \operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta_{t+1}},\mathbb{Q}) = \operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta_t},\mathbb{Q}) - 2\gamma_t\mathbb{E}_{z\sim\rho}\left[\begin{pmatrix}\nabla f_{\ell,\mathbb{P}_{\theta_t},\mathbb{Q}}(G_{\theta_t}(z))^{\mathsf{T}} \\ \mathbf{J}_{\theta}G_{\theta_t}(z)\Delta\theta\end{pmatrix}\right] + O(\gamma_t^2). MMD ℓ 2 ( P θ t + 1 , Q ) = MMD ℓ 2 ( P θ t , Q ) − 2 γ t E z ∼ ρ [ ( ∇ f ℓ , P θ t , Q ( G θ t ( z ) ) T J θ G θ t ( z ) Δ θ ) ] + O ( γ t 2 ) . Comparing the first-order term in γ t \gamma_t γ t Δ θ \Delta\theta Δ θ J θ G θ t ( ⋅ ) Δ θ ≈ ∇ f ℓ t , P t , Q ( G θ t ( ⋅ ) ) J_{\theta}G_{\theta_t}(\cdot)\Delta\theta\approx\nabla f_{\ell_t,\mathsf{P}_t,\mathsf{Q}}(G_{\theta_t}(\cdot)) J θ G θ t ( ⋅ ) Δ θ ≈ ∇ f ℓ t , P t , Q ( G θ t ( ⋅ )) projection of the vector field ∇ f ℓ t , P t , Q ( G θ t ( ⋅ ) ) \nabla f_{\ell_t,\mathsf{P}_t,\mathsf{Q}}(G_{\theta_t}(\cdot)) ∇ f ℓ t , P t , Q ( G θ t ( ⋅ )) span { J θ G θ t ( ⋅ ) u : u ∈ R p } ⊂ L 2 d ( ρ ) \operatorname{span}\{J_{\theta}G_{\theta_t}(\cdot)u:u\in\mathbb{R}^{p}\}\subset L_2^d(\rho) span { J θ G θ t ( ⋅ ) u : u ∈ R p } ⊂ L 2 d ( ρ ) Figure 5.2 ). This subspace consists of the directions that can be realized by infinitesimal updates of the parametric model at θ t \theta_t θ t
Figure 5.2. A geometric interpretation of the projection.
J t ( Δ θ t ) : = E z ∼ ρ [ ∥ J θ G θ t ( z ) Δ θ t − ∇ f ℓ t , P θ t , Q ( G θ t ( z ) ) ∥ 2 ] . (37) \mathcal{J}_t(\Delta\theta_t) := \mathbb{E}_{z\sim\rho}\left[\left\|J_{\theta}G_{\theta_t}(z)\Delta\theta_t-\nabla f_{\ell_t,\mathsf{P}_{\theta_t},\mathsf{Q}}(G_{\theta_t}(z))\right\|^2\right]. \tag*{(37)} J t ( Δ θ t ) := E z ∼ ρ [ J θ G θ t ( z ) Δ θ t − ∇ f ℓ t , P θ t , Q ( G θ t ( z )) 2 ] . (37) Since J t ( Δ θ t ) \mathcal{J}_t(\Delta\theta_t) J t ( Δ θ t ) Δ θ t ∈ R p \Delta\theta_t\in\mathbb{R}^{p} Δ θ t ∈ R p
Δ θ t P G D = { E z ∼ ρ [ J θ G θ t ( z ) ⊤ J θ G θ t ( z ) ] + λ t I d } − 1 × E z ∼ ρ [ J θ G θ t ( z ) ⊤ ∇ f ℓ t , P θ t , Q ( G θ t ( z ) ) ] . (38) \Delta\theta_t^{\mathrm{PGD}}=\left\{\mathbb{E}_{z\sim\rho}\left[J_{\theta}G_{\theta_t}(z)^\top J_{\theta}G_{\theta_t}(z)\right]+\lambda_t I_d\right\}^{-1}
\times\mathbb{E}_{z\sim\rho}\left[J_{\theta}G_{\theta_t}(z)^\top\nabla f_{\ell_t,\mathsf{P}_{\theta_t},\mathsf{Q}}(G_{\theta_t}(z))\right]. \tag*{(38)} Δ θ t PGD = { E z ∼ ρ [ J θ G θ t ( z ) ⊤ J θ G θ t ( z ) ] + λ t I d } − 1 × E z ∼ ρ [ J θ G θ t ( z ) ⊤ ∇ f ℓ t , P θ t , Q ( G θ t ( z )) ] . (38) where we include a ridge regularization λ t ≥ 0 \lambda_t\ge0 λ t ≥ 0 λ t = 0 \lambda_t=0 λ t = 0 ∇ f ℓ t , P θ t , Q ∘ G θ t \nabla f_{\ell_t,\mathsf{P}_{\theta_t},\mathsf{Q}}\circ G_{\theta_t} ∇ f ℓ t , P θ t , Q ∘ G θ t L 2 ( ρ ) L_2(\rho) L 2 ( ρ ) span { J θ G θ ( ⋅ ) u : u ∈ R p } \operatorname{span}\{J_{\theta}G_{\theta}(\cdot)u:u\in\mathbb{R}^{p}\} span { J θ G θ ( ⋅ ) u : u ∈ R p } E z ∼ ρ [ J θ G θ t ( z ) ⊤ J θ G θ t ( z ) ] \mathbb{E}_{z\sim\rho}\left[J_{\theta}G_{\theta_t}(z)^\top J_{\theta}G_{\theta_t}(z)\right] E z ∼ ρ [ J θ G θ t ( z ) ⊤ J θ G θ t ( z ) ] ρ \rho ρ λ t I d \lambda_t I_d λ t I d
At each iterate t t t λ t \lambda_t λ t approximation error of Δ θ t P G D \Delta\theta_t^{\mathrm{PGD}} Δ θ t PGD approximation error from using a second-order Taylor expansion. See Theorem 5.8 for details. Compared with the standard update Δ θ t G D \Delta\theta_t^{\mathrm{GD}} Δ θ t GD (36) , our update Δ θ t P G D \Delta\theta_t^{\mathrm{PGD}} Δ θ t PGD preconditioning matrix. For this reason, we refer to the resulting sequence ( P θ t P G D ) t ∈ N (\mathsf{P}_{\theta_t}^{\mathrm{PGD}})_{t\in\mathbb{N}} ( P θ t PGD ) t ∈ N preconditioned parametric MMD descent.
Before we proceed to present an upper bound that controls the gap between the nonparametric and preconditioned parametric updates after one step, we first introduce the following assumptions.
Assumption 5.3 (Regularity of G θ G_\theta G θ Suppose that θ ↦ G θ ( z ) \theta\mapsto G_\theta(z) θ ↦ G θ ( z ) is twice continuously differentiable. For all θ ∈ Θ \theta\in\Theta θ ∈ Θ and for all i , j ∈ { 1 , … , p } i,j \in\{1,\ldots,p\} i , j ∈ { 1 , … , p } suppose that
E z ∼ ρ [ ∥ G θ ( z ) ∥ 2 ] ≤ T 1 , E z ∼ ρ [ ∥ ∂ θ i G θ ( z ) ∥ 2 ] ≤ T 2 and E z ∼ ρ [ ∥ ∂ θ j ∂ θ i G θ ( z ) ∥ 2 ] ≤ T 3 . \mathbb{E}_{z\sim\rho}[\lVert G_\theta(z)\rVert^2] \leq\mathcal{T}_1,\quad\mathbb{E}_{z\sim\rho}[\lVert\partial_{\theta_i}G_\theta(z)\rVert^2] \leq\mathcal{T}_2\quad\text{and}\quad\mathbb{E}_{z\sim\rho}[\lVert\partial_{\theta_j}\partial_{\theta_i}G_\theta(z)\rVert^2] \leq\mathcal{T}_3. E z ∼ ρ [∥ G θ ( z ) ∥ 2 ] ≤ T 1 , E z ∼ ρ [∥ ∂ θ i G θ ( z ) ∥ 2 ] ≤ T 2 and E z ∼ ρ [∥ ∂ θ j ∂ θ i G θ ( z ) ∥ 2 ] ≤ T 3 . For notational simplicity, we set T 1 = T 2 = T 3 = 1 \mathcal{T}_1=\mathcal{T}_2=\mathcal{T}_3=1 T 1 = T 2 = T 3 = 1
Assumption 5.4 (Projection residual). For each t ∈ N t\in\mathbb{N} t ∈ N suppose that there exists a minimizer u t ∗ ∈ arg min u ∈ R p J t ( u ) u_t^*\in\arg\min_{u\in\mathbb{R}^p}\mathcal{J}_t(u) u t ∗ ∈ arg min u ∈ R p J t ( u ) such that, for some constants 0 ≤ R < 1 0\leq\mathcal{R}<1 0 ≤ R < 1 and 0 < U < ∞ 0<\mathcal{U}<\infty 0 < U < ∞ independent of t t t
J t ( u t ∗ ) ≤ R 2 ∥ ∇ f ℓ t , P θ t , Q ∥ L 2 ( P θ t ) 2 , ∥ u t ∗ ∥ ≤ U . \mathcal{J}_t(u_t^*)\leq\mathcal{R}^2\lVert\nabla f_{\ell_t,\mathbb{P}_{\theta_t},\mathbb{Q}}\rVert^2_{L^2(\mathbb{P}_{\theta_t})},\qquad\lVert u_t^*\rVert\leq\mathcal{U}. J t ( u t ∗ ) ≤ R 2 ∥ ∇ f ℓ t , P θ t , Q ∥ L 2 ( P θ t ) 2 , ∥ u t ∗ ∥ ≤ U . Assumption 5.3 serves two purposes. First, the condition E z ∼ ρ [ ∥ G θ ( z ) ∥ 2 ] ≤ 1 \mathbb{E}_{z\sim\rho}[\lVert G_\theta(z)\rVert^2]\leq1 E z ∼ ρ [∥ G θ ( z ) ∥ 2 ] ≤ 1 { P θ : θ ∈ Θ } ⊆ P 2 ( R d ) \{\mathbb{P}_\theta:\theta\in\Theta\}\subseteq\mathcal{P}_2(\mathbb{R}^d) { P θ : θ ∈ Θ } ⊆ P 2 ( R d ) E z ∼ ρ [ ∥ ∂ θ i G θ ( z ) ∥ 2 ] ≤ 1 \mathbb{E}_{z\sim\rho}[\lVert\partial_{\theta_i}G_\theta(z)\rVert^2]\leq1 E z ∼ ρ [∥ ∂ θ i G θ ( z ) ∥ 2 ] ≤ 1 E z ∼ ρ [ ∥ ∂ θ j ∂ θ i G θ ( z ) ∥ 2 ] ≤ 1 p r o v i d e t h e r e g u l a r i t y n e e d e d t o c o n t r o l f i r s t − a n d s e c o n d − o r d e r v a r i a t i o n s o f t h e m o d e l \mathbb{E}_{z\sim\rho}[\lVert\partial_{\theta_j}\partial_{\theta_i}G_\theta(z)\rVert^2]\leq1 provide the regularity needed to control first- and second-order variations of the model E z ∼ ρ [∥ ∂ θ j ∂ θ i G θ ( z ) ∥ 2 ] ≤ 1 p r o v i d e t h er e g u l a r i t y n ee d e d t oco n t r o l f i r s t − an d seco n d − or d er v a r ia t i o n so f t h e m o d e l θ \theta θ (36) .
Assumption 5.4 admits a natural geometric interpretation. For the first inequality of Assumption 5.4, R \mathcal{R} R t t t ∣ sin ( α ) ∣ |\sin(\alpha)| ∣ sin ( α ) ∣ α \alpha α Figure 5.2 . Hence the condition R < 1 \mathcal{R}<1 R < 1 ∇ f ℓ t , P θ t , Q ∘ G θ t ( ⋅ ) \nabla f_{\ell_t,\mathbb{P}_{\theta_t},\mathbb{Q}}\circ G_{\theta_t}(\cdot) ∇ f ℓ t , P θ t , Q ∘ G θ t ( ⋅ ) tangent subspace span { J θ G θ t ( ⋅ ) u : u ∈ R p } \operatorname{span}\{J_\theta G_{\theta_t}(\cdot)u:u\in\mathbb{R}^p\} span { J θ G θ t ( ⋅ ) u : u ∈ R p }
preconditioned descent scheme is not expected to make progress in reducing the MMD. The second inequality of Assumption 5.4 requires the exact-projection coefficient, displayed by the black vector in Figure 5.2 , to have a uniform-in-t t t
Example 5.5 (Sufficient conditions and examples for Assumption 5.4). A sufficient condition for Assumption 5.4 to hold, is that ∇ f ℓ , P θ , Q ( G θ ( ⋅ ) ) ∈ span { J θ G θ ( ⋅ ) u , u ∈ R } \nabla f_{\ell,\mathrm{P}_\theta,\mathrm{Q}}(G_\theta(\cdot)) \in\operatorname{span}\{J_\theta G_\theta(\cdot)u, u \in\mathbb{R}\} ∇ f ℓ , P θ , Q ( G θ ( ⋅ )) ∈ span { J θ G θ ( ⋅ ) u , u ∈ R } t t t R = 0 \mathcal{R}=0 R = 0 Q \mathbb{Q} Q R d \mathbb{R}^d R d t t t P θ = ( G θ ) # Unif ( S d − 1 ) \mathbb{P}_\theta=(G_\theta)_\#\operatorname{Unif}(\mathbb{S}^{d-1}) P θ = ( G θ ) # Unif ( S d − 1 ) G θ : z ↦ θ z G_\theta:z\mapsto\theta z G θ : z ↦ θ z θ ∈ R \theta\in\mathbb{R} θ ∈ R k k k
Proposition 5.7 (Closeness of P θ 1 P G D \mathbb{P}_{\theta_1}^{\mathrm{PGD}} P θ 1 PGD P t \mathbb{P}_t P t Let kernel k k k Q ∈ P 2 ( R d ) \mathbb{Q} \in\mathcal{P}_2(\mathbb{R}^d) Q ∈ P 2 ( R d ) ρ ∈ P 2 ( Z ) \rho\in\mathcal{P}_2(\mathbb{Z}) ρ ∈ P 2 ( Z ) P 0 = P θ 0 \mathbb{P}_0 = \mathbb{P}_{\theta_0} P 0 = P θ 0 γ 0 > 0 \gamma_0 > 0 γ 0 > 0 P 1 \mathbb{P}_1 P 1 P θ 1 P G D \mathbb{P}_{\theta_1}^{\mathrm{PGD}} P θ 1 PGD λ 0 ∝ ( γ 0 ρ 2 ( 1 + ℓ − 1 ) ℓ 0 − 2 ) 2 5 \lambda_0 \propto(\gamma_0\rho^2(1+\ell^{-1})\ell_0^{-2})^{\frac{2}{5}} λ 0 ∝ ( γ 0 ρ 2 ( 1 + ℓ − 1 ) ℓ 0 − 2 ) 5 2
MMD ℓ 2 ( P θ 1 P G D , Q ) − MMD ℓ 2 ( P 1 , Q ) ≤ 2 γ 0 R ∥ ∇ f ℓ , P 0 , Q ∥ L 2 d ( P 0 ) 2 + C k γ 0 R ℓ 0 − 4 ( ℓ 0 2 − ℓ 2 ) + o ( γ 0 ) . \operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta_1}^{\mathrm{PGD}},\mathbb{Q})-\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_1,\mathbb{Q}) \leq2\gamma_0\mathfrak{R}\lVert\nabla f_{\ell,\mathbb{P}_0,\mathbb{Q}}\rVert_{L_2^d(\mathbb{P}_0)}^2+C_k\gamma_0\mathfrak{R}\ell_0^{-4}(\ell_0^2-\ell^2)+o(\gamma_0). MMD ℓ 2 ( P θ 1 PGD , Q ) − MMD ℓ 2 ( P 1 , Q ) ≤ 2 γ 0 R ∥ ∇ f ℓ , P 0 , Q ∥ L 2 d ( P 0 ) 2 + C k γ 0 R ℓ 0 − 4 ( ℓ 0 2 − ℓ 2 ) + o ( γ 0 ) . Here, C k C_k C k
The proof is given in Section C.2.1. Theorem 5.7 gives a one-step comparison between the PGD update and nonparametric MMD descent, measured by their squared MMD distance to Q \mathbb{Q} Q cannot be represented, to first order, by any parametric update on the parameter θ \theta θ ℓ \ell ℓ ℓ 0 \ell_0 ℓ 0 o ( γ 0 ) o(\gamma_0) o ( γ 0 )
Remark 5.8 (Tradeoff with λ 0 \lambda_0 λ 0 λ 0 \lambda_0 λ 0 γ 0 λ 0 d ℓ − 1 \gamma_0\sqrt{\lambda_0}\sqrt{d}\ell^{-1} γ 0 λ 0 d ℓ − 1 λ 0 \lambda_0 λ 0 λ 0 − 2 γ 0 2 d ℓ − 2 ℓ 0 − 2 \lambda_0^{-2}\gamma_0^2\sqrt{d}\ell^{-2}\ell_0^{-2} λ 0 − 2 γ 0 2 d ℓ − 2 ℓ 0 − 2 λ 0 \lambda_0 λ 0
Remark 5.9 (Connection to natural gradient). Natural gradient descent is another preconditioned gradient method requiring fewer iterations than standard gradient descent to reach convergence (Amari, 1998; Hoffman et al., 2013; Pascanu and Bengio, 2014; Zhang et al., 2019; Martens, 2020). It can be interpreted as the steepest descent direction when the parameter space is endowed with the local geometry induced by the KL divergence (Amari and Nagaoka, 2000). For the MMD objective, a natural analogue is to endow the parameter space with the local geometry induced by the squared MMD distance (Briol et al., 2019a, Section 2.3). This leads to the preconditioning matrix (cf. Section C.3)
{ E z , z ′ ∼ ρ [ J θ G θ ( z ) ⊤ ∇ 1 , 2 k ℓ ( G θ ( z ) , G θ ( z ′ ) ) J θ G θ ( z ′ ) ] } − 1 . (39) \left\{\mathbb{E}_{z,z'\sim\rho}\left[J_\theta G_\theta(z)^\top\nabla_{1,2}k_\ell\left(G_\theta(z),G_\theta(z')\right)J_\theta G_\theta(z')\right]\right\}^{-1}. \tag*{(39)} { E z , z ′ ∼ ρ [ J θ G θ ( z ) ⊤ ∇ 1 , 2 k ℓ ( G θ ( z ) , G θ ( z ′ ) ) J θ G θ ( z ′ ) ] } − 1 . (39) This matrix is different from our preconditioning matrix
{ E z ∼ ρ [ J θ G θ t ( z ) ⊤ J θ G θ t ( z ) ] } − 1 \left\{\mathbb{E}_{z\sim\rho}\left[J_\theta G_{\theta_t}(z)^\top J_\theta G_{\theta_t}(z)\right]\right\}^{-1} { E z ∼ ρ [ J θ G θ t ( z ) ⊤ J θ G θ t ( z ) ] } − 1 in Eq. (38) , which arises instead from minimizing, after one step, the discrepancy between parametric and nonparametric descent.
In practice, the target distribution Q \mathbb{Q} Q m m m Q ^ m \widehat{\mathbb{Q}}_m Q m Q \mathbb{Q} Q Q ^ m \widehat{\mathbb{Q}}_m Q m ( P θ t , m P G D ) t ∈ N \left(\mathbb{P}_{\theta_{t,m}}^{\mathrm{PGD}}\right)_{t\in\mathbb{N}} ( P θ t , m PGD ) t ∈ N θ t + 1 = θ t − γ t Δ t , m P G D \theta_{t+1}=\theta_t-\gamma_t\Delta_{t,m}^{\mathrm{PGD}} θ t + 1 = θ t − γ t Δ t , m PGD
Δ t , m P G D = { E z ∼ ρ [ J θ G θ t ( z ) ⊤ J θ G θ t ( z ) ] + λ t I d } − 1 × E z ∼ ρ [ J θ G θ t ( z ) ⊤ ∇ f ℓ t , P θ t , Q ^ m ( G θ t ( z ) ) ] . (40) \Delta_{t,m}^{\mathrm{PGD}}=\left\{\mathbb{E}_{z\sim\rho}\left[J_\theta G_{\theta_t}(z)^\top J_\theta G_{\theta_t}(z)\right]+\lambda_t\mathrm{Id}\right\}^{-1}
\times\mathbb{E}_{z\sim\rho}\left[J_\theta G_{\theta_t}(z)^\top\nabla f_{\ell_t,\mathbb{P}_{\theta_t},\widehat{\mathbb{Q}}_m}\left(G_{\theta_t}(z)\right)\right]. \tag*{(40)} Δ t , m PGD = { E z ∼ ρ [ J θ G θ t ( z ) ⊤ J θ G θ t ( z ) ] + λ t Id } − 1 × E z ∼ ρ [ J θ G θ t ( z ) ⊤ ∇ f ℓ t , P θ t , Q m ( G θ t ( z ) ) ] . (40) Theorem 5.10 (Global convergence of PGD scheme ). Let kernel k k k Q ∈ P 2 ( R d ) \mathbb{Q}\in\mathcal{P}_2(\mathbb{R}^d) Q ∈ P 2 ( R d ) C > 0 C>0 C > 0 t t t ( γ t ) t ∈ N (\gamma_t)_{t\in\mathbb{N}} ( γ t ) t ∈ N γ t ∈ ( 0 , 1 2 C ( 1 − R ) ) \gamma_t\in\left(0,\frac{1}{2C(1-\mathcal{R})}\right) γ t ∈ ( 0 , 2 C ( 1 − R ) 1 ) lim t → ∞ γ t = 0 \lim_{t\to\infty}\gamma_t=0 lim t → ∞ γ t = 0 ∑ t = 0 ∞ γ t = ∞ \sum_{t=0}^{\infty}\gamma_t=\infty ∑ t = 0 ∞ γ t = ∞ ( ℓ t ) t ∈ N (\ell_t)_{t\in\mathbb{N}} ( ℓ t ) t ∈ N
satisfying lim t → ∞ ℓ t = ℓ > 0 \lim_{t \to\infty} \ell_t = \ell> 0 lim t → ∞ ℓ t = ℓ > 0 λ t ∝ ( γ t p 2 ( 1 + ℓ − 1 ) ℓ t − 2 ) 2 5 \lambda_t \propto\left(\gamma_t p^2(1+\ell^{-1})\ell_t^{-2}\right)^{\frac{2}{5}} λ t ∝ ( γ t p 2 ( 1 + ℓ − 1 ) ℓ t − 2 ) 5 2 T ∈ N T \in\mathbb{N} T ∈ N m = m ( T ) ∈ N m=m(T) \in\mathbb{N} m = m ( T ) ∈ N
m ( T ) ≥ ( ∑ s = 0 T γ s 0.6 ) ⋅ log ( T + 1 ) . \sqrt{m(T)} \ge\left(\sum_{s=0}^{T} \gamma_s^{0.6}\right)\cdot\log(T+1). m ( T ) ≥ ( s = 0 ∑ T γ s 0.6 ) ⋅ log ( T + 1 ) . (i) Well-specified case. Suppose that, for all t ∈ [ T ] t \in[T] t ∈ [ T ]
ℓ t d ℓ − d ∥ ∇ f ℓ , P ^ θ t , m P G D , Q ^ ∥ L 2 2 ( P ^ θ t , m P G D ) 2 ≥ C MMD ℓ 2 ( P θ t , m P G D , Q ) . \ell_t^{d}\ell^{-d}\left\|\nabla f_{\ell,\widehat{P}_{\theta_{t,m}}^{\mathrm{PGD}},\widehat{Q}}\right\|_{L_2^2(\widehat{P}_{\theta_{t,m}}^{\mathrm{PGD}})}^2 \ge C\,\operatorname{MMD}_{\ell}^2\left(P_{\theta_{t,m}}^{\mathrm{PGD}},Q\right). ℓ t d ℓ − d ∇ f ℓ , P θ t , m PGD , Q L 2 2 ( P θ t , m PGD ) 2 ≥ C MMD ℓ 2 ( P θ t , m PGD , Q ) . Then, there exists a remainder r T r_T r T
E m [ MMD ℓ 2 ( P θ T , m P G D , Q ) ] ≤ r T , \mathbb{E}_m\left[\operatorname{MMD}_{\ell}^2\left(P_{\theta_{T,m}}^{\mathrm{PGD}},Q\right)\right] \le r_T, E m [ MMD ℓ 2 ( P θ T , m PGD , Q ) ] ≤ r T , and r T → 0 r_T \to0 r T → 0 T → ∞ T \to\infty T → ∞
(ii) Mis-specified case. Suppose that, for all t ∈ [ T ] t \in[T] t ∈ [ T ]
ℓ t d ℓ − d ∥ ∇ f ℓ , P ^ θ t , m P G D , Q ^ ∥ L 2 2 ( P ^ θ t , m P G D ) 2 ≥ C ( MMD ℓ 2 ( P θ t , m P G D , Q ) − min θ ∈ Θ MMD ℓ 2 ( P θ , Q ) ) . \ell_t^{d}\ell^{-d}\left\|\nabla f_{\ell,\widehat{P}_{\theta_{t,m}}^{\mathrm{PGD}},\widehat{Q}}\right\|_{L_2^2(\widehat{P}_{\theta_{t,m}}^{\mathrm{PGD}})}^2 \ge C\left(\operatorname{MMD}_{\ell}^2\left(P_{\theta_{t,m}}^{\mathrm{PGD}},Q\right)-\min_{\theta\in\Theta}\operatorname{MMD}_{\ell}^2(P_\theta,Q)\right). ℓ t d ℓ − d ∇ f ℓ , P θ t , m PGD , Q L 2 2 ( P θ t , m PGD ) 2 ≥ C ( MMD ℓ 2 ( P θ t , m PGD , Q ) − θ ∈ Θ min MMD ℓ 2 ( P θ , Q ) ) . Then, there exists a remainder r T r_T r T
E m [ MMD ℓ 2 ( P θ T , m P G D , Q ) ] − min θ ∈ Θ MMD ℓ 2 ( P θ , Q ) ≤ r T , \mathbb{E}_m\left[\operatorname{MMD}_{\ell}^2\left(P_{\theta_{T,m}}^{\mathrm{PGD}},Q\right)\right]-\min_{\theta\in\Theta}\operatorname{MMD}_{\ell}^2(P_\theta,Q) \le r_T, E m [ MMD ℓ 2 ( P θ T , m PGD , Q ) ] − θ ∈ Θ min MMD ℓ 2 ( P θ , Q ) ≤ r T , and r T → 0 r_T \to0 r T → 0 T → ∞ T \to\infty T → ∞
The proof is given in Section C.2. The conditions on the step size γ t \gamma_t γ t ℓ t \ell_t ℓ t Theorem 5.2 . As the regularization parameter λ t \lambda_t λ t U \mathcal{U} U R = 1 \mathfrak{R}=1 R = 1 E m \mathbb{E}_m E m Q ^ m \widehat{Q}_m Q m m m m Q Q Q Q ^ m \widehat{Q}_m Q m γ t ≳ t − 1 \gamma_t \gtrsim t^{-1} γ t ≳ t − 1 m ( T ) ≳ T 14 / 5 ( log T ) 2 m(T) \gtrsim T^{14/5}(\log T)^2 m ( T ) ≳ T 14/5 ( log T ) 2
Theorem 5.10 gives two convergence guarantees in expectation over the empirical target sample. In the well-specified case, the model class is rich. enough to represent the target distribution in MMD, and the bound shows that our PGD drives terminal value MMD ℓ 2 ( P θ T , m P G D , Q ) \operatorname{MMD}_{\ell}^{2}(\mathbb{P}^{\mathrm{PGD}}_{\theta_{T,m}},\mathbb{Q}) MMD ℓ 2 ( P θ T , m PGD , Q )
In the misspecified case, however, exact recovery of Q \mathbb{Q} Q min θ ∈ Θ MMD ℓ 2 ( P θ , Q ) \min_{\theta\in\Theta}\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta},\mathbb{Q}) min θ ∈ Θ MMD ℓ 2 ( P θ , Q )
A related result is Proposition 5.2 of Chérief-Abdellatif and Alquier (2022), which establishes convergence under convexity of the map θ ↦ MMD 2 ( P θ , Q ) \theta\mapsto\operatorname{MMD}^{2}(\mathbb{P}_{\theta},\mathbb{Q}) θ ↦ MMD 2 ( P θ , Q ) Figure 5.1 . By contrast, our result does not rely on convexity of the MMD objective. Instead, it uses two key ingredients: 1) adaptive choice of kernel lengthscale, and 2) a close approximation of nonparametric MMD descent. Our expected gradient-dominance condition is nevertheless compatible with the classical convex setting. In particular, if θ ↦ MMD 2 ( P θ , Q ) \theta\mapsto\operatorname{MMD}^{2}(\mathbb{P}_{\theta},\mathbb{Q}) θ ↦ MMD 2 ( P θ , Q ) α \alpha α C = ( 2 p ) − 1 α C=(2p)^{-1}\alpha C = ( 2 p ) − 1 α asymptotic and relies on the stated gradient-dominance and projection-residual assumptions, it is among the first global convergence guarantees for finding minimum MMD estimators without assuming convexity of the MMD objective. Extension to a non-asymptotic convergence rate, with respect to both m m m t t t γ t \gamma_t γ t ℓ t \ell_t ℓ t R \mathfrak{R} R
Below we outline a quick sketch of the proof of Theorem 5.10. The key step is to prove the following descent lemma : for any t ∈ N t\in\mathbb{N} t ∈ N
E m [ MMD ℓ 2 ( P θ t + 1 , m P G D , Q ) − MMD ℓ 2 ( P θ t , m P G D , Q ) ] ≤ − 2 ( 1 − R ) γ t ∥ ∇ f ℓ , P θ t , m P G D , Q ∥ L d 2 ( P θ t , m P G D ) 2 + o ( γ t ) (41) \begin{aligned}
\mathbb{E}_{m}\left[\operatorname{MMD}_{\ell}^{2}\left(\mathbb{P}^{\mathrm{PGD}}_{\theta_{t+1,m}},\mathbb{Q}\right)-\operatorname{MMD}_{\ell}^{2}\left(\mathbb{P}^{\mathrm{PGD}}_{\theta_{t,m}},\mathbb{Q}\right)\right]
&\leq-2(1-\mathfrak{R})\gamma_t\left\|\nabla f_{\ell,\mathbb{P}^{\mathrm{PGD}}_{\theta_{t,m}},\mathbb{Q}}\right\|_{L_{d}^{2}(\mathbb{P}^{\mathrm{PGD}}_{\theta_{t,m}})}^{2}+o(\gamma_t) \tag*{(41)}
\end{aligned} E m [ MMD ℓ 2 ( P θ t + 1 , m PGD , Q ) − MMD ℓ 2 ( P θ t , m PGD , Q ) ] ≤ − 2 ( 1 − R ) γ t ∇ f ℓ , P θ t , m PGD , Q L d 2 ( P θ t , m PGD ) 2 + o ( γ t ) (41) To establish this, we define two auxiliary distributions: 1) P ^ t + 1 : = ( Id − \widehat{\mathbb{P}}_{t+1}:=(\operatorname{Id}- P t + 1 := ( Id − γ t ∇ f ℓ , P θ t , m P G D , Q # P θ t , m P G D \gamma_t\nabla f_{\ell,\mathbb{P}^{\mathrm{PGD}}_{\theta_{t,m}},\mathbb{Q}}\#\mathbb{P}^{\mathrm{PGD}}_{\theta_{t,m}} γ t ∇ f ℓ , P θ t , m PGD , Q # P θ t , m PGD nonparametric MMD update; and 2) P θ t + 1 P G D : = P θ t − γ t Δ θ t P G D \mathbb{P}^{\mathrm{PGD}}_{\theta_{t+1}} := \mathbb{P}_{\theta_t-\gamma_t\Delta^{\mathrm{PGD}}_{\theta_t}} P θ t + 1 PGD := P θ t − γ t Δ θ t PGD true target distribution Q \mathbb{Q} Q (40) . The proof of (41) is then a combination of the following inequalities.
MMD ℓ 2 ( P ^ t + 1 , Q ) − MMD ℓ 2 ( P θ t , m P G D , Q ) ≤ − 2 γ t ℓ t d ℓ − d ∥ ∇ f ℓ , P θ t , m P G D , Q ∥ L 2 d ( P θ t , m P G D ) 2 + o ( γ t ) \operatorname{MMD}_{\ell}^{2}(\widehat{\mathbb{P}}_{t+1},\mathbb{Q})-\operatorname{MMD}_{\ell}^{2}(\mathbb{P}^{\mathrm{PGD}}_{\theta_{t,m}},\mathbb{Q})\leq-2\gamma_t\ell_t^d\ell^{-d}\left\lVert\nabla f_{\ell,\mathbb{P}^{\mathrm{PGD}}_{\theta_{t,m}},\mathbb{Q}}\right\rVert_{L_2^d(\mathbb{P}^{\mathrm{PGD}}_{\theta_{t,m}})}^2+o(\gamma_t) MMD ℓ 2 ( P t + 1 , Q ) − MMD ℓ 2 ( P θ t , m PGD , Q ) ≤ − 2 γ t ℓ t d ℓ − d ∇ f ℓ , P θ t , m PGD , Q L 2 d ( P θ t , m PGD ) 2 + o ( γ t ) MMD ℓ 2 ( P θ t + 1 P G D , Q ) − MMD ℓ 2 ( P ^ t + 1 , Q ) ≤ 2 γ t R ∥ ∇ f ℓ , P θ t , m P G D , Q ∥ L 2 d ( P θ t , m P G D ) 2 + o ( γ t ) \operatorname{MMD}_{\ell}^{2}(\mathbb{P}^{\mathrm{PGD}}_{\theta_{t+1}},\mathbb{Q})-\operatorname{MMD}_{\ell}^{2}(\widehat{\mathbb{P}}_{t+1},\mathbb{Q})\leq2\gamma_t\mathcal{R}\left\lVert\nabla f_{\ell,\mathbb{P}^{\mathrm{PGD}}_{\theta_{t,m}},\mathbb{Q}}\right\rVert_{L_2^d(\mathbb{P}^{\mathrm{PGD}}_{\theta_{t,m}})}^2+o(\gamma_t) MMD ℓ 2 ( P θ t + 1 PGD , Q ) − MMD ℓ 2 ( P t + 1 , Q ) ≤ 2 γ t R ∇ f ℓ , P θ t , m PGD , Q L 2 d ( P θ t , m PGD ) 2 + o ( γ t ) E m [ MMD ℓ 2 ( P θ t + 1 , m P G D , Q ) ] ≤ MMD ℓ 2 ( P θ t + 1 P G D , Q ) + C k γ λ − 1 p ℓ − 1 ℓ t − 1 d ⋅ 1 m . \mathbb{E}_m\left[\operatorname{MMD}_{\ell}^{2}(\mathbb{P}^{\mathrm{PGD}}_{\theta_{t+1,m}},\mathbb{Q})\right]\leq\operatorname{MMD}_{\ell}^{2}(\mathbb{P}^{\mathrm{PGD}}_{\theta_{t+1}},\mathbb{Q})
+C_k\gamma\lambda^{-1}p\ell^{-1}\ell_t^{-1}\sqrt{d}\cdot\frac{1}{\sqrt{m}}. E m [ MMD ℓ 2 ( P θ t + 1 , m PGD , Q ) ] ≤ MMD ℓ 2 ( P θ t + 1 PGD , Q ) + C k γ λ − 1 p ℓ − 1 ℓ t − 1 d ⋅ m 1 . The first inequality is proved in Theorem C.1 and controls the decrease of squared MMD under the nonparametric MMD descent after one step. The second inequality is proved in Theorem 5.7 and controls the difference of squared MMD between two schemes after one step. The third inequality is proved in Theorem C.5, and bounds the finite-sample error from using Q ^ m \widehat{\mathbb{Q}}_m Q m Q \mathbb{Q} Q C k C_k C k (41) . The proof of Theorem 5.10 is concluded with a discrete Gronwall’s lemma.
Computational complexity We now turn to how to compute our PGD update Δ t , m P G D \Delta^{\mathrm{PGD}}_{t,m} Δ t , m PGD (40) in practice. Given n n n { z i } i = 1 n \{z_i\}_{i=1}^n { z i } i = 1 n ρ \rho ρ m m m { y i } i = 1 m \{y_i\}_{i=1}^m { y i } i = 1 m Q \mathbb{Q} Q Δ t , m P G D \Delta^{\mathrm{PGD}}_{t,m} Δ t , m PGD Δ ^ t , m P G D : = ( H ^ t + λ t I d ) − 1 g ^ t \widehat{\Delta}^{\mathrm{PGD}}_{t,m}:=(\widehat{H}_t+\lambda_t\mathrm{Id})^{-1}\widehat{g}_t Δ t , m PGD := ( H t + λ t Id ) − 1 g t
H ^ t = 1 n ∑ i = 1 n J θ G θ t ( z i ) T J θ G θ t ( z i ) , \widehat{H}_t=\frac{1}{n}\sum_{i=1}^{n}J_{\theta}G_{\theta_t}(z_i)^{\mathsf{T}}J_{\theta}G_{\theta_t}(z_i), H t = n 1 i = 1 ∑ n J θ G θ t ( z i ) T J θ G θ t ( z i ) , g ^ t = ∇ θ t { ∑ i , j = 1 n k ( G θ t ( z i ) , G θ t ( z j ) ) n 2 − 2 ∑ i , j = 1 n , m k ( G θ t ( z i ) , y j ) n m } . \widehat{g}_t=\nabla_{\theta_t}\left\{\frac{\sum_{i,j=1}^{n}k(G_{\theta_t}(z_i),G_{\theta_t}(z_j))}{n^2}-\frac{2\sum_{i,j=1}^{n,m}k(G_{\theta_t}(z_i),y_j)}{nm}\right\}. g t = ∇ θ t { n 2 ∑ i , j = 1 n k ( G θ t ( z i ) , G θ t ( z j )) − nm 2 ∑ i , j = 1 n , m k ( G θ t ( z i ) , y j ) } . Relative to (40) , this implementation additionally replaces the expec tation with respect to ρ \rho ρ { z i } i = 1 n \{z_i\}_{i=1}^{n} { z i } i = 1 n g ^ t \widehat{g}_t g t MMD ℓ 2 ( P θ , Q ) \operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta},\mathbb{Q}) MMD ℓ 2 ( P θ , Q ) g ^ t \widehat{g}_t g t H ^ t \widehat{H}_t H t O ( n min ( d , p ) + n d p 2 ) \mathcal{O}(n\min(d,p)+ndp^{2}) O ( n min ( d , p ) + n d p 2 ) J θ G θ t ( z i ) ∈ R d × p J_{\theta}G_{\theta_t}(z_i) \in\mathbb{R}^{d \times p} J θ G θ t ( z i ) ∈ R d × p p × p p \times p p × p ( H ^ t + λ t I d ) − 1 (\widehat{H}_t+\lambda_t\mathrm{Id})^{-1} ( H t + λ t Id ) − 1 O ( p 3 ) \mathcal{O}(p^{3}) O ( p 3 ) (39) of Briol et al. (2019a), which involves a double expectation. In practice, this extra cost is worthwhile as it gives a more faithful update direction that leads to global convergence, as proved in Theorem 5.10 and verified empirically next in § 7.4 . The cost can be reduced by computing both g ^ t \widehat{g}_t g t H ^ t \widehat{H}_t H t p p p H ^ t \widehat{H}_t H t § 7.4 are relatively small-scale, we leave these implementations to future work.
Our convergence result in Theorem 5.10 accounts only for the finite-sample error arising from replacing Q \mathbb{Q} Q Q ^ m \widehat{\mathbb{Q}}_m Q m { z i } i = 1 n \{z_i\}_{i=1}^{n} { z i } i = 1 n ρ \rho ρ λ t \lambda_t λ t
Q ^ m : = 1 m ∑ i = 1 m δ y i \widehat{Q}_m := \frac{1}{m}\sum_{i=1}^{m}\delta_{y_i} Q m := m 1 ∑ i = 1 m δ y i y 1 , … , y m y_1,\ldots,y_m y 1 , … , y m Q \mathbb{Q} Q Q ^ m \widehat{Q}_m Q m Q ^ m ∉ { P θ : θ ∈ Θ } \widehat{Q}_m \notin\{\mathbb{P}_{\theta} : \theta\in\Theta\} Q m ∈ / { P θ : θ ∈ Θ } θ ∗ \theta^{*} θ ∗ θ m ∗ \theta_m^{*} θ m ∗ θ m ∗ \theta_m^{*} θ m ∗ θ ∗ \theta^{*} θ ∗ Q ^ m \widehat{Q}_m Q m θ ∗ \theta^{*} θ ∗
A practical guide to choosing ℓ t \ell_t ℓ t λ t \lambda_t λ t γ t \gamma_t γ t The lengthscale schedule ( ℓ t ) t ∈ N (\ell_t)_{t\in\mathbb{N}} ( ℓ t ) t ∈ N ℓ \ell ℓ ℓ t = max { ℓ , 100 ℓ ⋅ η t } \ell_t = \max\{\ell, 100\ell\cdot\eta^t\} ℓ t = max { ℓ , 100 ℓ ⋅ η t } η ∈ [ 0.99 , 1 ) \eta\in[0.99,1) η ∈ [ 0.99 , 1 ) γ t \gamma_t γ t γ t = γ 0 ⋅ ( 1 + t ) − β \gamma_t = \gamma_0 \cdot(1+t)^{-\beta} γ t = γ 0 ⋅ ( 1 + t ) − β β ∈ ( 0 , 1 ) \beta\in(0,1) β ∈ ( 0 , 1 ) Theorems 5.2 and 5.10. For the regularization parameter λ t \lambda_t λ t ℓ t \ell_t ℓ t λ t \lambda_t λ t λ t \lambda_t λ t γ t \gamma_t γ t
Experiments
In this section, we provide empirical evidence to illustrate the effectiveness of our preconditioned gradient descent (PGD) scheme. For comparison, we also consider standard gradient descent (GD) and NGD (Briol et al., 2019a), which uses Eq. (39) as the preconditioner. Our PGD preconditioner is computationally cheaper to compute than the NGD preconditioner, but both methods are more expensive per iteration than GD. All three methods are run until observed convergence, when the difference between two consecutive iterations falling below 10 − 4 10^{-4} 1 0 − 4
Nonparametric MMD descent. We first consider a synthetic experiment where the target distribution is a mixture of 8 two-dimensional (d = 2 d = 2 d = 2 Theorem 5.2 . We use a particle implementation of nonparametric MMD descent (Arbel et al., 2019; Chen et al., 2025b). Let { y i } i = 1 m \{y_i\}_{i=1}^{m} { y i } i = 1 m m m m Q Q Q { x i ( 0 ) } i = 1 N \{x_i^{(0)}\}_{i=1}^{N} { x i ( 0 ) } i = 1 N N N N N ( 0 , 1 ) \mathcal{N}(0, 1) N ( 0 , 1 ) t ∈ N t \in\mathbb{N} t ∈ N
x i ( t + 1 ) = x i ( t ) − γ t ∇ f ℓ t , P ^ N , t , Q ^ m ( x i ( t ) ) , ∀ i ∈ [ N ] , x_i^{(t+1)} = x_i^{(t)} - \gamma_t \nabla f_{\ell_t,\widehat{P}_{N,t},\widehat{Q}_m}\left(x_i^{(t)}\right), \qquad\forall i \in[N], x i ( t + 1 ) = x i ( t ) − γ t ∇ f ℓ t , P N , t , Q m ( x i ( t ) ) , ∀ i ∈ [ N ] , where Q ^ m = 1 m ∑ j = 1 m δ y i \widehat{Q}_m = \frac{1}{m}\sum_{j=1}^{m}\delta_{y_i} Q m = m 1 ∑ j = 1 m δ y i P ^ N , t = 1 N ∑ i = 1 N δ x i ( t ) \widehat{P}_{N,t} = \frac{1}{N}\sum_{i=1}^{N}\delta_{x_i^{(t)}} P N , t = N 1 ∑ i = 1 N δ x i ( t ) ℓ t = ℓ = 0.1 \ell_t = \ell= 0.1 ℓ t = ℓ = 0.1 ℓ t = max { ℓ , 10.0 × 0.9999 t } \ell_t = \max\{\ell, 10.0 \times0.9999^t\} ℓ t = max { ℓ , 10.0 × 0.999 9 t } { 0.99 , 0.995 , 0.999 , 0.9995 , 0.9999 } \{0.99, 0.995, 0.999, 0.9995, 0.9999\} { 0.99 , 0.995 , 0.999 , 0.9995 , 0.9999 } γ t = 0.01 / ( 1 + t ) 0.1 \gamma_t = 0.01/(1 + t)^{0.1} γ t = 0.01/ ( 1 + t ) 0.1 Theorem 5.2 .
The two plots in the third panel of Figure 5.3 show that the adaptive scheme attains a smaller final MMD ℓ ( P t , Q ) \operatorname{MMD}_{\ell}(\mathit{P}_t, Q) MMD ℓ ( P t , Q ) ℓ t \ell_t ℓ t ℓ \ell ℓ ℓ t \ell_t ℓ t ℓ \ell ℓ Figure 5.3 , we also plot the left- and right-hand sides of the key condition in Theorem 5.2 , with constant C = 0.01 C = 0.01 C = 0.01
Figure 5.3. Mixture of Gaussians. From left to right: first and final locations of 100 particles; MMD versus iterations; MMD versus particle number N N N Theorem 5.2 . Results are over 10 seeds; curves show medians and bands the 25–75% percentiles.
G-and-k distribution. We next evaluate our PGD scheme on the gand-k distribution, a commonly used benchmark for minimum distance estimators ((Briol et al., 2019a; Dellaporta et al., 2022)). This distribution has widespread utility, including in the pricing of AirBnB rentals ((Rodrigues et al., 2020)), air pollution modelling ((Rayner and MacGillivray, 2002)) and non-life insurance ((Peters et al., 2016)). Despite being a univariate model, it does not admit a closed-form likelihood: specifically, the statistical model is given by ( G θ ) # ρ (G_\theta)_\#\rho ( G θ ) # ρ ρ \rho ρ [ 0 , 1 ] [0,1] [ 0 , 1 ] G θ G_\theta G θ θ = ( a , b , c , exp ( k ) ) \theta=(a,b,c,\exp(k)) θ = ( a , b , c , exp ( k )) G θ G_\theta G θ θ \theta θ Assumption 5.3 holds.
For this experiment, we draw m = 1000 m=1000 m = 1000 Q = ( G θ ∗ ) # ρ Q=(G_{\theta^*})_\#\rho Q = ( G θ ∗ ) # ρ θ ∗ = ( 3 , 1 , 1 , exp ( − log 2 ) ) \theta^*=(3,1,1,\exp(-\log2)) θ ∗ = ( 3 , 1 , 1 , exp ( − log 2 )) well-specified setting. For GD and NGD, we use fixed step sizes γ = 0.1 \gamma=0.1 γ = 0.1 γ t = 0.1 / ( 1 + t ) 0.1 \gamma_t=0.1/(1+t)^{0.1} γ t = 0.1/ ( 1 + t ) 0.1 ℓ t = ℓ = 2 \ell_t=\ell=2 ℓ t = ℓ = 2 ℓ t = max { ℓ , 10.0 × 0.99 t } \ell_t=\max\{\ell,10.0\times0.99^t\} ℓ t = max { ℓ , 10.0 × 0.9 9 t } { 0.99 , 0.995 , 0.999 , 0.9995 , 0.9999 } \{0.99,0.995,0.999,0.9995,0.9999\} { 0.99 , 0.995 , 0.999 , 0.9995 , 0.9999 } Figure 5.4 , both PGD and NGD outperform standard GD. In Figure 5.4 , both PGD and NGD perform better than standard GD over the reported optimization horizon. In the first panel, both preconditioned methods reach the optimum in fewer iterations; and in the second panel, standard GD remains far from the global minimum over the optimization horizon, especially in the more challenging third and fourth parameters. These results suggest that preconditioning not only accelerates convergence, but also helps overcome the non-convexity of the MMD objective. NGD performs slightly better than our PGD scheme, albeit at a larger cost of computing the preconditioner due to the double expectation. More empirical studies are thus needed to better understand which preconditioner is preferable in practice.
Figure 5.4. G-and-k distribution. From left to right: two optimization trajectories toward the global optimum (yellow star), MMD versus iteration, and left- and right-hand sides of the condition in Theorem 5.10. In the left two plots, two trajectories for each method are shown. Results are over 10 seeds; curves show medians and bands the 25–75% percentiles.
Inference for Lotka-Volterra model under corruptions. We next consider an example where the model P θ \mathbb{P}_{\theta} P θ ( X 1 , t , X 2 , t ) (X_{1,t}, X_{2,t}) ( X 1 , t , X 2 , t ) θ = ( X 1 , 0 , X 2 , 0 ) \theta= (X_{1,0}, X_{2,0}) θ = ( X 1 , 0 , X 2 , 0 ) Assumption 5.3 ; the square-root coefficients in the Euler–Maruyama recursion make such a verification delicate near the boundary of the positive orthant.
We study robustness to data corruption, a setting where minimum MMD estimators are particularly relevant. Specifically, following the setting of Briol et al. (2019a), we consider inferring the true initial condition θ ∗ = ( 100 , 120 ) \theta^{*} = (100, 120) θ ∗ = ( 100 , 120 ) p % p\% p % θ † = ( 50 , 50 ) \theta^{\dagger} = (50, 50) θ † = ( 50 , 50 ) γ = 100 \gamma= 100 γ = 100 γ t = 300 / ( 1 + t ) 0.1 \gamma_t = 300/(1 + t)^{0.1} γ t = 300/ ( 1 + t ) 0.1 ℓ t = ℓ = 30 \ell_t = \ell= 30 ℓ t = ℓ = 30 ℓ t = max { 30 , 3000 × 0.9995 t } \ell_t = \max\{30, 3000 \times0.9995^t\} ℓ t = max { 30 , 3000 × 0.999 5 t } { 0.95 , 0.99 , 0.995 , 0.999 , 0.9995 , 0.9999 } \{0.95, 0.99, 0.995, 0.999, 0.9995, 0.9999\} { 0.95 , 0.99 , 0.995 , 0.999 , 0.9995 , 0.9999 }
Our results are shown in Figure 5.5 . In particular, the left and middle panels show that, when initialized at θ 0 = ( 90 , 90 ) \theta_0=(90,90) θ 0 = ( 90 , 90 ) θ 0 = ( 50 , 60 ) \theta_0=(50,60) θ 0 = ( 50 , 60 ) Figure 5.5 compares the methods across different corruption levels, p ∈ { 0 , 15 , 35 } p \in\{0,15,35\} p ∈ { 0 , 15 , 35 } p p p
Figure 5.5. Lotka-Volterra model. Left: Two optimization trajectories toward the global optimum (yellow star) under two initializations. Middle: Evolution of θ 1 , θ 2 \theta_1,\theta_2 θ 1 , θ 2 Right: Evolution of θ 1 , θ 2 \theta_1,\theta_2 θ 1 , θ 2
Composite goodness-of-fit test. Finally, we consider a toggle-switch model: a stochastic dynamical model for interacting gene-expression levels over a fixed time horizon T ∈ N \mathcal{T} \in\mathbb{N} T ∈ N θ \theta θ P θ ; T \mathbb{P}_{\theta;T} P θ ; T T \mathcal{T} T P θ ; T = ( G θ ; T ) # ρ \mathbb{P}_{\theta;\mathcal{T}} = (G_{\theta;\mathcal{T}})_{\#}\rho P θ ; T = ( G θ ; T ) # ρ T \mathcal{T} T G θ ; T G_{\theta;\mathcal{T}} G θ ; T
We evaluate a composite goodness-of-fit test for the model class { P θ ; T : θ ∈ Θ } \{\mathbb{P}_{\theta;\mathcal{T}} : \theta\in\Theta\} { P θ ; T : θ ∈ Θ } H 0 : Q ∈ { P θ ; T : θ ∈ Θ } H_0 : \mathbb{Q} \in\{\mathbb{P}_{\theta;\mathcal{T}} : \theta\in\Theta\} H 0 : Q ∈ { P θ ; T : θ ∈ Θ } Δ m : = m min θ ∈ Θ MMD 2 ( P θ ; T , Q ^ m ) \Delta_m := m \min_{\theta\in\Theta} \operatorname{MMD}^2(\mathbb{P}_{\theta;\mathcal{T}}, \widehat{\mathbb{Q}}_m) Δ m := m min θ ∈ Θ MMD 2 ( P θ ; T , Q m ) Q ^ m \widehat{\mathbb{Q}}_m Q m m m m Q \mathbb{Q} Q Δ m \Delta_m Δ m Q = P θ ∗ ; 50 \mathbb{Q} = \mathbb{P}_{\theta^*;50} Q = P θ ∗ ; 50 { P θ ; 50 : θ ∈ Θ } \{\mathbb{P}_{\theta;50} : \theta\in\Theta\} { P θ ; 50 : θ ∈ Θ } Q ∈ { P θ ; T : θ ∈ Θ } \mathbb{Q} \in\{\mathbb{P}_{\theta;\mathcal{T}} : \theta\in\Theta\} Q ∈ { P θ ; T : θ ∈ Θ } Q = P θ ∗ ; 20 \mathbb{Q} = \mathbb{P}_{\theta^*;20} Q = P θ ∗ ; 20 { P θ ; 300 : θ ∈ Θ } \{\mathbb{P}_{\theta;300} : \theta\in\Theta\} { P θ ; 300 : θ ∈ Θ } Q ∉ { P θ ; T : θ ∈ Θ } \mathbb{Q} \notin\{\mathbb{P}_{\theta;\mathcal{T}} : \theta\in\Theta\} Q ∈ / { P θ ; T : θ ∈ Θ } B = 50 B = 50 B = 50 α = 0.05 \alpha= 0.05 α = 0.05 γ = 0.2 \gamma= 0.2 γ = 0.2 ℓ = 120 \ell= 120 ℓ = 120 γ t = 0.2 / ( 1 + t ) 0.1 \gamma_t = 0.2/(1+t)^{0.1} γ t = 0.2/ ( 1 + t ) 0.1 ℓ t = max { 120 , 2000 × 0.95 t } \ell_t = \max\{120, 2000 \times0.95^t\} ℓ t = max { 120 , 2000 × 0.9 5 t } { 0.95 , 0.99 , 0.995 , 0.999 , 0.9995 , 0.9999 } \{0.95, 0.99, 0.995, 0.999, 0.9995, 0.9999\} { 0.95 , 0.99 , 0.995 , 0.999 , 0.9995 , 0.9999 }
In Table 5.1 , we compare PGD and GD in terms of the Type-I errors for testing under the null, and powers for testing under the alternative. Following Appendix D.§ 5.3 of Key et al. (2025), GD is run with a multi-start scheme: I = 400 \mathcal{I} = 400 I = 400 I = 1 \mathcal{I} = 1 I = 1 Table 5.1 . By contrast, PGD is run from a single initialization, I = 1 \mathcal{I} = 1 I = 1 I = 1 I = 1 I = 1 I = 400 I = 400 I = 400
Table 5.1. Composite goodness-of-fit test. Critical values are computed by parametric bootstrap with B = 50 B=50 B = 50 α = 0.05 \alpha=0.05 α = 0.05 ± \pm ±
Ablation study. Our proposed PGD scheme introduces two main components: an adaptive lengthscale and a preconditioner. To assess their respective contributions, we conduct an ablation study in Figure 5.6 , comparing GD and PGD under both fixed and adaptive lengthscales, respectively. The results suggest that adaptive lengthscale plays a more prominent role in improving convergence for the Lotka–Volterra model, whereas the preconditioner is the more important component for the g-and-k distribution. The relative importance of the two components may be problem-dependent; in practice, we therefore recommend using both.
Figure 5.6. Ablation study: Standard GD with fixed/adaptive kernel lengthscales and our PGD with fixed/adaptive kernel lengthscales. Left: G-and-K distribution and Right: Lotka-Volterra model in well-specified setting with θ 0 = ( 90 , 90 ) \theta_0 = (90, 90) θ 0 = ( 90 , 90 )
Conclusions
We propose a new preconditioned gradient descent scheme for computing minimum MMD estimators, and establish an asymptotic convergence guarantee to a global minimizer under explicit non-convex gradient-dominance and projection-residual conditions. A key limitation of our current analysis is that it relies on an a priori condition on the adaptive choice of kernel lengthscales. Although such conditions are common in the analysis of kernel-based gradient flows (Arbel et al., 2019; Chen et al., 2025b), removing or relaxing this requirement would make our analysis more appealing. Another limitation is the projection-residual condition in Assumption 5.4 , which we posit is necessary, as it reflects the unavoidable error incurred when approximating the nonparametric update by a parametric update.
Looking ahead, it would be interesting to extend our techniques to the optimization of other popular minimum-distance estimation methods (Basu et al., 2011), such as minimum Stein discrepancy estimation (Barp et al., 2019), minimum Fisher divergence estimation (Hyvärinen and Dayan, 2005), minimum Wasserstein distance estimation (Bassetti et al., 2006; Zhang and Chen, 2022; Lang et al., 2026), and many others. A main challenge is that, unlike the MMD, these discrepancies typically do not yield a simple computable finite-sample expression for the associated nonparametric Wasserstein gradient. Minimum kernel Stein discrepancy estimation (Barp et al., 2019) is the closest to our minimum MMD estimation; however, it requires access to the model score, whereas in our pushforward setting P θ = ( G θ ) # ρ P_\theta= (G_\theta)_\#\rho P θ = ( G θ ) # ρ
Conclusions and Future Work
This thesis has studied the minimization of squared maximum mean discrepancy (MMD) as a unifying framework for comparing, approximating, and sampling from probability distributions. The central message is that MMD is not only a statistically meaningful discrepancy between probability measures, but also an optimization objective whose structure can be exploited to design practical algorithms for computational statistics and machine learning. The preceding chapters of this thesis develop this perspective in three settings: numerical integration, sampling and generative modelling, and statistical inference.
Future Work
Causal inference
One promising future direction is the intersection between minimum MMD methods and causal inference. This direction is a natural continuation of my recent work on causal inference (Chen et al., 2024a; Shen et al., 2025b; Chen et al., 2025e). Causal problems are fundamentally distributional: interventions change the data-generating distribution, counterfactual questions compare distributions across parallel worlds. This makes them a natural domain for MMD-based discrepancies, especially in settings where likelihoods are unavailable but samples from observational, simulated, or interventional distributions can be obtained. An important future direction is to combine MMD-based distribution matching with doubly robust causal estimation, building on recent doubly robust methods for proxy causal learning and counterfactual policy mean embeddings (Bozkurt et al., 2025; Zenati et al., 2025).
Scalable generative models
Another promising direction for future research is the application of MMD to large-scale generative models. MMD and the minimization of which has once been popular in the literature of generative adversarial networks (GANs), e.g., the MMD GANs (Li et al., 2017; Arbel et al., 2018; Bi’nkowski et al., 2018). Although diffusion models have largely superseded GANs since 2020 (Song et al., 2021; Ho et al., 2020), recent work on one-step, high-quality generationincluding drifting models (Deng et al., 2026; Gretton et al., 2026) and W-Flow (Han et al., 2026)suggests that Wasserstein gradient flows provide a promising framework for developing one-step generative models that achieve state-of-the-art performance on ImageNet256 dataset. Based on the striking similarity of their approaches and MMD gradient flows (Arbel et al., 2019), I believe MMD flows have the potential to achieve state-of-the-art performances as well, provided with a good latent encoder.
MMD with adaptive kernels and distributions
Beyond the relatively direct extensions to causal inference and generative modelling, a more fundamental question arises concerning the continued relevance of MMD:
Looking back from 2026, one might wonder: why should we still care about maximum mean discrepancy (MMD)?
MMD is a kernel-based discrepancy introduced around 2010, before the widespread rise of modern deep learning. At first glance, MMD may therefore appear outdated, particularly when instantiated with a Gaussian kernel. Its fixed, non-adaptive feature representation may seem poorly suited to high-dimensional structured data such as images and text.
But who says that kernels must rely on fixed features? Growing empirical evidence suggests that, when equipped with features learned by neural networks, kernel methods can achieve competitive performance in applications ranging from hypothesis testing (Liu et al., 2020), regression (Galashov et al., 2025b) to generative modelling (Galashov et al., 2025a). Indeed, a neural network can be viewed as learning an adaptive feature representation, upon which a kernel-like notion of similarity is implicitly constructed (Jacot et al., 2018; Takakura and Suzuki, 2024; Ba et al., 2022; Suzuki et al., 2023). From this broader perspective of MMD with adaptive kernels, many exciting open questions remain. How should kernel features be adapted to optimize downstream performance? Should the adaptation mechanism differ across regression, generative modelling, statistical inference, and causal inference? Although the preceding chapters have provided partial answers to some of these questions, a general theory of adaptive kernels for MMD remains largely undeveloped.
And who says that the distributions for which the MMD is being computed must themselves be also fixed? As this thesis has demonstrated, minimizing MMD provides a versatile framework for designing practical algorithms in computational statistics and machine learning (Key et al., 2025; Chen et al., 2025d, 2024c; Kang et al., 2025; Chen et al., 2025c,b; Tian et al., 2026; Chen et al., 2026; Kang et al., 2026; Chen et al., 2024a; Shen et al., 2025b; Chen et al., 2025e). The methods developed here consider settings in which one distribution remains fixed while the other evolves along a trajectory toward the fixed reference. Many other areas remain to be explored, including causal inference, optimal transport, inverse problems, where allowing one of the two distributions to evolve may offer new algorithmic possibilities.
Supplementary Material for Minimum MMD for Integration in Nested Expectations
Additional notations For two normed vector spaces A A A B B B A ≅ B A \cong B A ≅ B A A A B B B c 1 , c 2 > 0 c_1,c_2 > 0 c 1 , c 2 > 0 c 1 ∥ h ∥ A ≤ ∥ h ∥ B ≤ c 2 ∥ h ∥ A c_1\lVert h\rVert_A \le\lVert h\rVert_B \le c_2\lVert h\rVert_A c 1 ∥ h ∥ A ≤ ∥ h ∥ B ≤ c 2 ∥ h ∥ A h ∈ A h \in A h ∈ A ∥ ⋅ ∥ A ≅ ∥ ⋅ ∥ B \lVert\cdot\rVert_A \cong\lVert\cdot\rVert_B ∥ ⋅ ∥ A ≅ ∥ ⋅ ∥ B A ⊆ B A \subseteq B A ⊆ B A A A B B B A ↪ B A \hookrightarrow B A ↪ B ∥ T ∥ \lVert T\rVert ∥ T ∥ T : A → B T : A \to B T : A → B f : X ⊆ R d → R f : \mathcal{X} \subseteq\mathbb{R}^{d} \to\mathbb{R} f : X ⊆ R d → R α ∈ N d \alpha\in\mathbb{N}^{d} α ∈ N d ∂ x α f \partial_{x}^{\alpha}f ∂ x α f ∂ x 1 α 1 ⋯ ∂ x n α n f \partial_{x_1}^{\alpha_1} \cdots\partial_{x_n}^{\alpha_n} f ∂ x 1 α 1 ⋯ ∂ x n α n f D x α f D_{x}^{\alpha}f D x α f f ∈ W 2 s ( X ) f \in W_{2}^{s}(\mathcal{X}) f ∈ W 2 s ( X ) ∥ f ∥ s , 2 : = ∥ f ∥ W 2 s ( X ) \lVert f\rVert_{s,2} := \lVert f\rVert_{W_{2}^{s}(\mathcal{X})} ∥ f ∥ s , 2 := ∥ f ∥ W 2 s ( X ) ≲ \lesssim ≲ ≤ \le ≤
Existing Results on Kernel Ridge Regression
In this section, we present Theorem A.1 to A.3 which are adaptation of theorems from Fischer and Steinwart (2020) applied to Sobolev spaces. These propositions are foundations of the proof of Theorem 3.1 in Section A.3.
In the standard regression setting, we are given N N N { x i , y i } i = 1 N \{x_i,y_i\}_{i=1}^{N} { x i , y i } i = 1 N P \mathbf{P} P X × R \mathcal{X} \times\mathbb{R} X × R X ⊂ R d \mathcal{X} \subset\mathbb{R}^{d} X ⊂ R d P \mathbf{P} P X \mathcal{X} X Q \mathbf{Q} Q P ( ⋅ ∣ x ) \mathbf{P}(\cdot\mid x) P ( ⋅ ∣ x ) σ , L > 0 \sigma, L > 0 σ , L > 0 x x x
∫ R ∣ y − h ∗ ( x ) ∣ m P ( d y ∣ x ) ≤ 1 2 m ! σ 2 L m − 2 (42) \int_{\mathbb{R}} |y-h^{*}(x)|^{m} \mathbf{P}(\mathrm{d}y \mid x) \leq\frac{1}{2}m!\sigma^{2}L^{m-2} \tag*{(42)} ∫ R ∣ y − h ∗ ( x ) ∣ m P ( d y ∣ x ) ≤ 2 1 m ! σ 2 L m − 2 (42) is satisfied for Q \mathbf{Q} Q x ∈ X x \in\mathcal{X} x ∈ X m ≥ 2 m \geq2 m ≥ 2 σ = L = σ 0 \sigma= L = \sigma_{0} σ = L = σ 0 P ( ⋅ ∣ x ) \mathbf{P}(\cdot\mid x) P ( ⋅ ∣ x ) σ 0 \sigma_{0} σ 0 σ = L = 0 \sigma= L = 0 σ = L = 0
In a regression problem, the target of interest is the Bayes predictor h ∗ : X → R h^{*}\colon\mathcal{X} \to\mathbb{R} h ∗ : X → R x ↦ E [ Y ∣ X = x ] x \mapsto\mathbb{E}[Y \mid X=x] x ↦ E [ Y ∣ X = x ] h ∗ h^{*} h ∗ k : X × X → R k\colon\mathcal{X} \times\mathcal{X} \to\mathbb{R} k : X × X → R h ^ λ : X → R \hat{h}_{\lambda}\colon\mathcal{X} \to\mathbb{R} h ^ λ : X → R λ > 0 \lambda> 0 λ > 0
h ^ λ = arg min h ∈ H k { λ ∥ h ∥ H k 2 + 1 N ∑ i = 1 N ( y i − h ( x i ) ) 2 } . (43) \hat{h}_{\lambda} = \underset{h \in\mathcal{H}_{k}}{\arg\min}\left\{\lambda\lVert h\rVert_{\mathcal{H}_{k}}^{2}+\frac{1}{N}\sum_{i=1}^{N}(y_{i}-h(x_{i}))^{2}\right\}. \tag*{(43)} h ^ λ = h ∈ H k arg min { λ ∥ h ∥ H k 2 + N 1 i = 1 ∑ N ( y i − h ( x i ) ) 2 } . (43) H k \mathcal{H}_{k} H k k k k
h ^ λ = k ( ⋅ , x 1 : N ) ( k ( x 1 : N , x 1 : N ) + N λ I d N ) − 1 y 1 : N . \hat{h}_{\lambda}=k(\cdot,x_{1:N})\left(k(x_{1:N},x_{1:N})+N\lambda I_{d_{N}}\right)^{-1}y_{1:N}. h ^ λ = k ( ⋅ , x 1 : N ) ( k ( x 1 : N , x 1 : N ) + N λ I d N ) − 1 y 1 : N . We also introduce an auxiliary function h λ : X → Y h_{\lambda}\colon\mathcal{X} \to\mathcal{Y} h λ : X → Y
h λ = arg min f ∈ H k { λ ∥ f ∥ H k 2 + ∫ X × R ( y − f ( x ) ) 2 P ( d x , d y ) } . (44) h_{\lambda}=\underset{f\in\mathcal{H}_{k}}{\arg\min}\left\{\lambda\lVert f\rVert_{\mathcal{H}_{k}}^{2}+\int_{\mathcal{X}\times\mathbb{R}}(y-f(x))^{2}\mathbf{P}(\mathrm{d}x,\mathrm{d}y)\right\}. \tag*{(44)} h λ = f ∈ H k arg min { λ ∥ f ∥ H k 2 + ∫ X × R ( y − f ( x ) ) 2 P ( d x , d y ) } . (44) In regression setting, it is of interest to study the generalization error between the estimator h ^ λ \hat{h}_{\lambda} h ^ λ h ∗ h^{*} h ∗ ∥ h ^ λ − h ∗ ∥ L 2 ( Q ) \lVert\hat{h}_{\lambda}-h^{*}\rVert_{L_{2}(\mathbb{Q})} ∥ h ^ λ − h ∗ ∥ L 2 ( Q ) N N N
∥ h ^ λ − h ∗ ∥ L 2 ( Q ) ≤ ∥ h ^ λ − h λ ∥ L 2 ( Q ) + ∥ h λ − h ∗ ∥ L 2 ( Q ) , (45) \lVert\hat{h}_{\lambda}-h^{*}\rVert_{L_{2}(\mathbb{Q})} \leq\lVert\hat{h}_{\lambda}-h_{\lambda}\rVert_{L_{2}(\mathbb{Q})}+\lVert h_{\lambda}-h^{*}\rVert_{L_{2}(\mathbb{Q})}, \tag*{(45)} ∥ h ^ λ − h ∗ ∥ L 2 ( Q ) ≤ ∥ h ^ λ − h λ ∥ L 2 ( Q ) + ∥ h λ − h ∗ ∥ L 2 ( Q ) , (45) where the first term ∥ h ^ λ − h λ ∥ L 2 ( Q ) \lVert\hat{h}_{\lambda}-h_{\lambda}\rVert_{L_{2}(\mathbb{Q})} ∥ h ^ λ − h λ ∥ L 2 ( Q ) ∥ h λ − h ∗ ∥ L 2 ( Q ) \lVert h_{\lambda}-h^{*}\rVert_{L_{2}(\mathbb{Q})} ∥ h λ − h ∗ ∥ L 2 ( Q )
(S1) k k k s > d 2 s > \frac{d}{2} s > 2 d
(S2) Q \mathbb{Q} Q X \mathcal{X} X p : X → R p:\mathcal{X}\to\mathbb{R} p : X → R G 0 , G 1 G_{0},G_{1} G 0 , G 1 G 0 ≤ p ( x ) ≤ G 1 G_{0}\leq p(x)\leq G_{1} G 0 ≤ p ( x ) ≤ G 1 x ∈ X x\in\mathcal{X} x ∈ X
(S3) The Bayes predictor h ∗ ∈ W 2 s ( X ) h^{*}\in W_{2}^{s}(\mathcal{X}) h ∗ ∈ W 2 s ( X )
(S4) There exists universal constants σ , L > 0 \sigma,L>0 σ , L > 0
Proposition A.1 (Approximation error). Under Assumptions (S1)-(S4),
∥ h λ − h ∗ ∥ L 2 ( Q ) ≤ ∥ h ∗ ∥ s , 2 λ 1 2 . \lVert h_{\lambda}-h^{*}\rVert_{L_{2}(\mathbb{Q})}\leq\lVert h^{*}\rVert_{s,2}\lambda^{\frac{1}{2}}. ∥ h λ − h ∗ ∥ L 2 ( Q ) ≤ ∥ h ∗ ∥ s , 2 λ 2 1 . Proof. This is direct application of Lemma 14 of Fischer and Steinwart (2020) with β = 1 \beta=1 β = 1 γ = 0 \gamma=0 γ = 0
Proposition A.2 (Estimation error). Suppose Assumptions (S1)-(S4) hold. Let N ( λ ) \mathcal{N}(\lambda) N ( λ ) k α k_{\alpha} k α Theorem A.10 . If N > A λ , τ N>A_{\lambda,\tau} N > A λ , τ 1 − 4 e − τ 1-4e^{-\tau} 1 − 4 e − τ
∥ h λ − h ^ λ ∥ L 2 ( Q ) 2 ≤ 576 τ 2 N ( L 2 D λ − d 2 s + M 2 λ 1 − d 2 s ∥ h ∗ ∥ s , 2 2 + M 2 L λ 2 N λ − d 2 s ) . (46) \lVert h_{\lambda}-\hat{h}_{\lambda}\rVert_{L_{2}(\mathbb{Q})}^{2}\leq\frac{576\tau^{2}}{N}\left(L^{2}D\lambda^{-\frac{d}{2s}}+M^{2}\lambda^{1-\frac{d}{2s}}\lVert h^{*}\rVert_{s,2}^{2}+M^{2}\frac{L_{\lambda}^{2}}{N}\lambda^{-\frac{d}{2s}}\right). \tag*{(46)} ∥ h λ − h ^ λ ∥ L 2 ( Q ) 2 ≤ N 576 τ 2 ( L 2 D λ − 2 s d + M 2 λ 1 − 2 s d ∥ h ∗ ∥ s , 2 2 + M 2 N L λ 2 λ − 2 s d ) . (46) where D D D M M M N N N g λ g_{\lambda} g λ A λ , τ A_{\lambda,\tau} A λ , τ L λ L_{\lambda} L λ
g λ : = log ( 2 e N ( λ ) ∥ Σ Q ∥ + λ ∥ Σ Q ∥ ) , A λ , τ : = 8 k α 2 τ g λ λ − d 2 s , g_{\lambda} := \log\left(2e\mathcal{N}(\lambda)\frac{\lVert\Sigma_{Q}\rVert+\lambda}{\lVert\Sigma_{Q}\rVert}\right), \qquad A_{\lambda,\tau} := 8k_{\alpha}^{2}\tau g_{\lambda}\lambda^{-\frac{d}{2s}}, g λ := log ( 2 e N ( λ ) ∥ Σ Q ∥ ∥ Σ Q ∥ + λ ) , A λ , τ := 8 k α 2 τ g λ λ − 2 s d , L λ : = max { L , λ 1 2 − d 4 s ( ∥ h ∗ ∥ L ∞ ( Q ) + k α ∥ h ∗ ∥ s , 2 ) } . L_{\lambda} := \max\left\{L,\lambda^{\frac{1}{2}-\frac{d}{4s}}\left(\lVert h^{*}\rVert_{L_{\infty}(Q)}+k_{\alpha}\lVert h^{*}\rVert_{s,2}\right)\right\}. L λ := max { L , λ 2 1 − 4 s d ( ∥ h ∗ ∥ L ∞ ( Q ) + k α ∥ h ∗ ∥ s , 2 ) } . Proof. This proposition is a special case of Theorem 16 in Fischer and Steinwart (2020) under the following adaptations towards our Sobolev space setting: 1) Theorem A.9 proves that N ( λ ) ≤ D λ − d 2 s \mathcal{N}(\lambda) \le D\lambda^{-\frac{d}{2s}} N ( λ ) ≤ D λ − 2 s d Theorem A.10 proves that k α ≤ M k_{\alpha} \le M k α ≤ M α = d 2 s \alpha= \frac{d}{2s} α = 2 s d ∥ h ∗ − h λ ∥ L ∞ ( Q ) \lVert h^{*}-h_{\lambda}\rVert_{L_{\infty}(Q)} ∥ h ∗ − h λ ∥ L ∞ ( Q ) λ 1 2 − d 4 s ( ∥ h ∗ ∥ L ∞ ( Q ) + k α ∥ h ∗ ∥ s , 2 ) \lambda^{\frac{1}{2}-\frac{d}{4s}}\left(\lVert h^{*}\rVert_{L_{\infty}(Q)}+k_{\alpha}\lVert h^{*}\rVert_{s,2}\right) λ 2 1 − 4 s d ( ∥ h ∗ ∥ L ∞ ( Q ) + k α ∥ h ∗ ∥ s , 2 ) ∥ Σ Q ∥ \lVert\Sigma_{Q}\rVert ∥ Σ Q ∥
Proposition A.3. Suppose Assumptions (S1)-(S4) hold. For A λ , τ A_{\lambda,\tau} A λ , τ L λ L_{\lambda} L λ N > A λ , τ N>A_{\lambda,\tau} N > A λ , τ 1 − 4 e − τ 1-4e^{-\tau} 1 − 4 e − τ
∥ h ^ λ − h ∗ ∥ L 2 ( Q ) 2 ≤ 576 τ 2 N ( L 2 D λ − d 2 s + M 2 λ 1 − d 2 s ∥ h ∗ ∥ s , 2 2 + 2 M 2 L 2 N λ − d 2 s ) + ∥ h ∗ ∥ s , 2 2 λ . \lVert\hat{h}_{\lambda}-h^{*}\rVert_{L_{2}(Q)}^{2}\le\frac{576\tau^{2}}{N}\left(L^{2}D\lambda^{-\frac{d}{2s}}+M^{2}\lambda^{1-\frac{d}{2s}}\lVert h^{*}\rVert_{s,2}^{2}+2M^{2}\frac{L^{2}}{N}\lambda^{-\frac{d}{2s}}\right)+\lVert h^{*}\rVert_{s,2}^{2}\lambda. ∥ h ^ λ − h ∗ ∥ L 2 ( Q ) 2 ≤ N 576 τ 2 ( L 2 D λ − 2 s d + M 2 λ 1 − 2 s d ∥ h ∗ ∥ s , 2 2 + 2 M 2 N L 2 λ − 2 s d ) + ∥ h ∗ ∥ s , 2 2 λ . Proof. By the triangle inequality in (A.4), combining Theorem A.1 and Theorem A.2 finishes the proof.
Noiseless Kernel Ridge Regression (Kernel Quadrature)
In this section, we present the upper bound on the generalization error ∥ h ∗ − h ^ λ ∥ L 2 ( Q ) \lVert h^{*}-\hat{h}_{\lambda}\rVert_{L_{2}(Q)} ∥ h ∗ − h ^ λ ∥ L 2 ( Q ) Theorem 3.1 in the next section. Our proof follows the outline of the proof for Theorem 1 in Fischer and Steinwart (2020), modified for our choice of regularization parameter λ \lambda λ λ > 0 \lambda> 0 λ > 0
Suppose we have N N N x 1 : N x_{1:N} x 1 : N Q \mathbb{Q} Q X \mathcal{X} X N N N noiseless function evaluations h ∗ ( x 1 : N ) h^*(x_{1:N}) h ∗ ( x 1 : N ) h ∗ : X ⊂ R d → R h^*:\mathcal{X} \subset\mathbb{R}^{d} \to\mathbb{R} h ∗ : X ⊂ R d → R
Proposition A.4. Let X ⊂ R d \mathcal{X} \subset\mathbb{R}^{d} X ⊂ R d x 1 : N x_{1:N} x 1 : N N N N Q \mathbb{Q} Q h ^ λ N ( ⋅ ) : = k ( ⋅ , x 1 : N ) ( k ( x 1 : N , x 1 : N ) + N λ N I d N ) − 1 h ∗ ( x 1 : N ) \hat{h}_{\lambda_N}(\cdot) := k(\cdot, x_{1:N})\left(k(x_{1:N}, x_{1:N}) + N\lambda_N\mathrm{Id}_{N}\right)^{-1}h^*(x_{1:N}) h ^ λ N ( ⋅ ) := k ( ⋅ , x 1 : N ) ( k ( x 1 : N , x 1 : N ) + N λ N Id N ) − 1 h ∗ ( x 1 : N ) λ N ≍ N − 2 s d ( log N ) 2 s + 2 d \lambda_N \asymp N^{-\frac{2s}{d}}(\log N)^{\frac{2s+2}{d}} λ N ≍ N − d 2 s ( log N ) d 2 s + 2 N 0 > 0 N_0 > 0 N 0 > 0 N > N 0 N > N_0 N > N 0
∥ h ∗ − h ^ λ N ∥ L 2 ( Q ) ≤ C τ N − s d ( log N ) s + 1 d ∥ h ∗ ∥ s , 2 (47) \lVert h^* - \hat{h}_{\lambda_N}\rVert_{L_2(\mathbb{Q})} \le\mathcal{C}_{\tau}N^{-\frac{s}{d}}(\log N)^{\frac{s+1}{d}}\lVert h^*\rVert_{s,2} \tag*{(47)} ∥ h ∗ − h ^ λ N ∥ L 2 ( Q ) ≤ C τ N − d s ( log N ) d s + 1 ∥ h ∗ ∥ s , 2 (47) holds with probability at least 1 − 4 e − τ 1 - 4e^{-\tau} 1 − 4 e − τ C = C ( X , G 0 , G 1 ) \mathcal{C} = \mathcal{C}(\mathcal{X}, G_0, G_1) C = C ( X , G 0 , G 1 ) X , G 0 , G 1 \mathcal{X}, G_0, G_1 X , G 0 , G 1
Proof. Notice that h ^ λ N \hat{h}_{\lambda_N} h ^ λ N y i y_i y i h ∗ ( x i ) h^*(x_i) h ∗ ( x i ) h λ N h_{\lambda_N} h λ N y y y h ∗ ( x ) h^*(x) h ∗ ( x ) L = 0 L = 0 L = 0
Similar to the proof of Theorem A.3, we decompose the generalization error into an estimation error term ∥ h λ N − h ^ λ N ∥ L 2 ( Q ) \lVert h_{\lambda_N} - \hat{h}_{\lambda_N}\rVert_{L_2(\mathbb{Q})} ∥ h λ N − h ^ λ N ∥ L 2 ( Q ) ∥ h λ N − h ∗ ∥ L 2 ( Q ) \lVert h_{\lambda_N} - h^*\rVert_{L_2(\mathbb{Q})} ∥ h λ N − h ∗ ∥ L 2 ( Q ) Approximation error Take λ N ≍ N − 2 s d ( log N ) 2 s + 2 d \lambda_N \asymp N^{-\frac{2s}{d}}(\log N)^{\frac{2s+2}{d}} λ N ≍ N − d 2 s ( log N ) d 2 s + 2
∥ h λ N − h ∗ ∥ L 2 ( Q ) ≤ λ N 1 2 ∥ h ∗ ∥ s , 2 ≍ N − s d ( log N ) s + 1 d ∥ h ∗ ∥ s , 2 . \lVert h_{\lambda_N}-h^*\rVert_{L_2(Q)} \leq\lambda_N^{\frac{1}{2}}\lVert h^*\rVert_{s,2} \asymp N^{-\frac{s}{d}}(\log N)^{\frac{s+1}{d}}\lVert h^*\rVert_{s,2}. ∥ h λ N − h ∗ ∥ L 2 ( Q ) ≤ λ N 2 1 ∥ h ∗ ∥ s , 2 ≍ N − d s ( log N ) d s + 1 ∥ h ∗ ∥ s , 2 . Estimation error Recall all the constants g λ N g_{\lambda_N} g λ N A λ N , τ A_{\lambda_N,\tau} A λ N , τ L λ N L_{\lambda_N} L λ N L = 0 L=0 L = 0 L λ N = λ N 1 2 − d 4 s ( ∥ h ∗ ∥ L ∞ ( Q ) + k α ∥ h ∗ ∥ s , 2 ) L_{\lambda_N}=\lambda_N^{\frac{1}{2}-\frac{d}{4s}}(\lVert h^*\rVert_{L_\infty(Q)}+k_\alpha\lVert h^*\rVert_{s,2}) L λ N = λ N 2 1 − 4 s d (∥ h ∗ ∥ L ∞ ( Q ) + k α ∥ h ∗ ∥ s , 2 ) N 0 N_0 N 0 N ≥ A λ N , τ N\geq A_{\lambda_N,\tau} N ≥ A λ N , τ N ≥ N 0 N\geq N_0 N ≥ N 0 lim N → ∞ A λ N , τ / N → 0 \lim_{N\to\infty}A_{\lambda_N,\tau}/N\to0 lim N → ∞ A λ N , τ / N → 0
lim N → ∞ A λ N , τ N = 8 k α 2 g λ N λ N − d 2 s N = 8 ( log N ) − s + 1 s k α 2 τ log ( 2 e N ( λ N ) ∥ Σ Q ∥ + λ N ∥ Σ Q ∥ ) . \lim_{N\to\infty}\frac{A_{\lambda_N,\tau}}{N}
=\frac{8k_\alpha^2g_{\lambda_N}\lambda_N^{-\frac{d}{2s}}}{N}
=8(\log N)^{-\frac{s+1}{s}}k_\alpha^2\tau\log\left(2e\mathcal{N}(\lambda_N)\frac{\lVert\Sigma_Q\rVert+\lambda_N}{\lVert\Sigma_Q\rVert}\right). N → ∞ lim N A λ N , τ = N 8 k α 2 g λ N λ N − 2 s d = 8 ( log N ) − s s + 1 k α 2 τ log ( 2 e N ( λ N ) ∥ Σ Q ∥ ∥ Σ Q ∥ + λ N ) . where N ( λ N ) \mathcal{N}(\lambda_N) N ( λ N ) k α 2 k_\alpha^2 k α 2 Theorem A.10 . Since lim N → ∞ λ N = lim N → ∞ N − 2 s d ( log N ) 2 s + 2 d = 0 \lim_{N\to\infty}\lambda_N=\lim_{N\to\infty}N^{-\frac{2s}{d}}(\log N)^{\frac{2s+2}{d}}=0 lim N → ∞ λ N = lim N → ∞ N − d 2 s ( log N ) d 2 s + 2 = 0 N ′ N' N ′ λ N ≤ ∥ Σ Q ∥ \lambda_N\leq\lVert\Sigma_Q\rVert λ N ≤ ∥ Σ Q ∥ N ≥ N ′ N\geq N' N ≥ N ′ N N N
lim N → ∞ A λ N , τ N ≤ lim N → ∞ 8 ( log N ) − s + 1 s k α 2 τ log ( 4 e N ( λ N ) ) ≤ lim N → ∞ 8 ( log N ) − s + 1 s k α 2 τ log ( 4 e D λ N − d 2 s ) = lim N → ∞ 8 ( log N ) − s + 1 s k α 2 τ log ( 4 e D ) + lim N → ∞ 8 ( log N ) − s + 1 s k α 2 τ log ( N ( log N ) − s + 1 s ) ≤ lim N → ∞ 16 ( log N ) − s + 1 s k α 2 τ log ( N ) = 0 , (48) \begin{aligned}
\lim_{N\to\infty}\frac{A_{\lambda_N,\tau}}{N}
&\leq\lim_{N\to\infty}8(\log N)^{-\frac{s+1}{s}}k_\alpha^2\tau\log\left(4e\mathcal{N}(\lambda_N)\right)\\
&\leq\lim_{N\to\infty}8(\log N)^{-\frac{s+1}{s}}k_\alpha^2\tau\log\left(4eD\lambda_N^{-\frac{d}{2s}}\right)\\
&=\lim_{N\to\infty}8(\log N)^{-\frac{s+1}{s}}k_\alpha^2\tau\log(4eD)\\
&\quad+\lim_{N\to\infty}8(\log N)^{-\frac{s+1}{s}}k_\alpha^2\tau\log\left(N(\log N)^{-\frac{s+1}{s}}\right)\\
&\leq\lim_{N\to\infty}16(\log N)^{-\frac{s+1}{s}}k_\alpha^2\tau\log(N)\\
&=0, \tag*{(48)}
\end{aligned} N → ∞ lim N A λ N , τ ≤ N → ∞ lim 8 ( log N ) − s s + 1 k α 2 τ log ( 4 e N ( λ N ) ) ≤ N → ∞ lim 8 ( log N ) − s s + 1 k α 2 τ log ( 4 eD λ N − 2 s d ) = N → ∞ lim 8 ( log N ) − s s + 1 k α 2 τ log ( 4 eD ) + N → ∞ lim 8 ( log N ) − s s + 1 k α 2 τ log ( N ( log N ) − s s + 1 ) ≤ N → ∞ lim 16 ( log N ) − s s + 1 k α 2 τ log ( N ) = 0 , (48) where M M M D D D Theorem A.10 . So there exists N ′ ′ N'' N ′′ N ≥ A λ N , τ N\geq A_{\lambda_N,\tau} N ≥ A λ N , τ N ≥ N ′ ′ N\geq N'' N ≥ N ′′ N 0 = max { N ′ , N ′ ′ } N_0=\max\{N',N''\} N 0 = max { N ′ , N ′′ } N ≥ A λ N , τ N\geq A_{\lambda_N,\tau} N ≥ A λ N , τ N ≥ N 0 N\geq N_0 N ≥ N 0 B ∗ : = ∥ h ∗ ∥ L ∞ ( Q ) + M ∥ h ∗ ∥ s , 2 B_*:=\lVert h^*\rVert_{L_\infty(Q)}+M\lVert h^*\rVert_{s,2} B ∗ := ∥ h ∗ ∥ L ∞ ( Q ) + M ∥ h ∗ ∥ s , 2 1 − 4 e − τ 1-4e^{-\tau} 1 − 4 e − τ
∥ h λ N − h ^ λ N ∥ L 2 ( Q ) 2 ≤ 576 τ 2 N ( M 2 λ N 1 − d 2 s ∥ h ∗ ∥ s , 2 2 + M 2 λ N 1 − d 2 s B ∗ 2 1 N λ N − d s ) \lVert h_{\lambda_N}-\hat{h}_{\lambda_N}\rVert_{L_2(Q)}^2\leq\frac{576\tau^2}{N}\left(M^2\lambda_N^{1-\frac{d}{2s}}\lVert h^*\rVert_{s,2}^2+M^2\lambda_N^{1-\frac{d}{2s}}B_*^2\frac{1}{N}\lambda_N^{-\frac{d}{s}}\right) ∥ h λ N − h ^ λ N ∥ L 2 ( Q ) 2 ≤ N 576 τ 2 ( M 2 λ N 1 − 2 s d ∥ h ∗ ∥ s , 2 2 + M 2 λ N 1 − 2 s d B ∗ 2 N 1 λ N − s d ) ≲ 576 τ 2 N ( M 2 N 1 − 2 s d ( log N ) s + 1 s 2 s − d d ∥ h ∗ ∥ s , 2 2 + M 2 B ∗ 2 N 1 − 2 s d ( log N ) s + 1 2 s 4 s − d d ) \lesssim\frac{576\tau^{2}}{N}\left(M^{2}N^{1-\frac{2s}{d}}(\log N)^{\frac{s+1}{s}\frac{2s-d}{d}}\lVert h^{*}\rVert_{s,2}^{2}+M^{2}B_{*}^{2}N^{1-\frac{2s}{d}}(\log N)^{\frac{s+1}{2s}\frac{4s-d}{d}}\right) ≲ N 576 τ 2 ( M 2 N 1 − d 2 s ( log N ) s s + 1 d 2 s − d ∥ h ∗ ∥ s , 2 2 + M 2 B ∗ 2 N 1 − d 2 s ( log N ) 2 s s + 1 d 4 s − d ) ≤ 576 τ 2 N − 2 s d ( log N ) 2 s + 2 d ( M 2 ∥ h ∗ ∥ s , 2 2 + M 2 B ∗ 2 ) . \leq576\tau^{2}N^{-\frac{2s}{d}}(\log N)^{\frac{2s+2}{d}}\left(M^{2}\lVert h^{*}\rVert_{s,2}^{2}+M^{2}B_{*}^{2}\right). ≤ 576 τ 2 N − d 2 s ( log N ) d 2 s + 2 ( M 2 ∥ h ∗ ∥ s , 2 2 + M 2 B ∗ 2 ) . So we have, with probability at least 1 − 4 e − τ 1-4e^{-\tau} 1 − 4 e − τ
∥ h λ N − h ^ λ N ∥ L 2 ( Q ) ≤ 24 τ N − s d ( log N ) s + 1 d ( ( M + M 2 ) ∥ h ∗ ∥ s , 2 + M ∥ h ∗ ∥ L ∞ ( Q ) ) . \lVert h_{\lambda_N}-\hat{h}_{\lambda_N}\rVert_{L_{2}(\mathbb{Q})}\leq24\tau N^{-\frac{s}{d}}(\log N)^{\frac{s+1}{d}}\left((M+M^{2})\lVert h^{*}\rVert_{s,2}+M\lVert h^{*}\rVert_{L_{\infty}(\mathbb{Q})}\right). ∥ h λ N − h ^ λ N ∥ L 2 ( Q ) ≤ 24 τ N − d s ( log N ) d s + 1 ( ( M + M 2 ) ∥ h ∗ ∥ s , 2 + M ∥ h ∗ ∥ L ∞ ( Q ) ) . Combine approximation and estimation error Combining the above two inequalities on approximation error ∥ h λ N − h ∗ ∥ L 2 ( Q ) \lVert h_{\lambda_N}-h^{*}\rVert_{L_{2}(\mathbb{Q})} ∥ h λ N − h ∗ ∥ L 2 ( Q ) ∥ h λ N − h ^ λ N ∥ L 2 ( Q ) \lVert h_{\lambda_N}-\hat{h}_{\lambda_N}\rVert_{L_{2}(\mathbb{Q})} ∥ h λ N − h ^ λ N ∥ L 2 ( Q ) 1 − 4 e − τ 1-4e^{-\tau} 1 − 4 e − τ
∥ h ∗ − h ^ λ N ∥ L 2 ( Q ) ≤ 24 τ N − s d ( log N ) s + 1 d ( ( 1 + M + M 2 ) ∥ h ∗ ∥ s , 2 + M ∥ h ∗ ∥ L ∞ ( Q ) ) . \lVert h^{*}-\hat{h}_{\lambda_N}\rVert_{L_{2}(\mathbb{Q})}\leq24\tau N^{-\frac{s}{d}}(\log N)^{\frac{s+1}{d}}\left((1+M+M^{2})\lVert h^{*}\rVert_{s,2}+M\lVert h^{*}\rVert_{L_{\infty}(\mathbb{Q})}\right). ∥ h ∗ − h ^ λ N ∥ L 2 ( Q ) ≤ 24 τ N − d s ( log N ) d s + 1 ( ( 1 + M + M 2 ) ∥ h ∗ ∥ s , 2 + M ∥ h ∗ ∥ L ∞ ( Q ) ) . Finally, following the arguments of Theorem A.10 that the operator norm of W 2 s ( X ) ↪ L ∞ ( X ) W_{2}^{s}(\mathcal{X})\hookrightarrow L_{\infty}(\mathcal{X}) W 2 s ( X ) ↪ L ∞ ( X ) ∥ h ∗ ∥ L ∞ ( Q ) ≤ R ∥ h ∗ ∥ s , 2 \lVert h^{*}\rVert_{L_{\infty}(\mathbb{Q})}\leq R\lVert h^{*}\rVert_{s,2} ∥ h ∗ ∥ L ∞ ( Q ) ≤ R ∥ h ∗ ∥ s , 2 R R R X \mathcal{X} X G 0 G_{0} G 0 G 1 G_{1} G 1 1 − 4 e − τ 1-4e^{-\tau} 1 − 4 e − τ
∥ h ∗ − h ^ λ N ∥ L 2 ( Q ) ≤ 24 τ N − s d ( log N ) s + 1 d ( 1 + ( 1 + R ) M + M 2 ) ∥ h ∗ ∥ s , 2 = E τ N − s d ( log N ) s + 1 d ∥ h ∗ ∥ s , 2 , \begin{aligned}
\lVert h^{*}-\hat{h}_{\lambda_N}\rVert_{L_{2}(\mathbb{Q})}\leq24\tau N^{-\frac{s}{d}}(\log N)^{\frac{s+1}{d}}\left(1+(1+R)M+M^{2}\right)\lVert h^{*}\rVert_{s,2} \\
&=\mathcal{E}\tau N^{-\frac{s}{d}}(\log N)^{\frac{s+1}{d}}\lVert h^{*}\rVert_{s,2},
\end{aligned} ∥ h ∗ − h ^ λ N ∥ L 2 ( Q ) ≤ 24 τ N − d s ( log N ) d s + 1 ( 1 + ( 1 + R ) M + M 2 ) ∥ h ∗ ∥ s , 2 = E τ N − d s ( log N ) d s + 1 ∥ h ∗ ∥ s , 2 , for E : = 24 ( 1 + ( 1 + R ) M + M 2 ) \mathcal{E}:=24(1+(1+R)M+M^{2}) E := 24 ( 1 + ( 1 + R ) M + M 2 )
Corollary A.5. Let X \mathcal{X} X R d \mathbb{R}^{d} R d x 1 : N x_{1:N} x 1 : N N N N Q \mathbb{Q} Q I ^ K Q : = E X ∼ Q [ k ( X , x 1 : N ) ] ( k ( x 1 : N , x 1 : N ) + N λ N I d N ) − 1 h ∗ ( x 1 : N ) \hat{I}_{K\mathbb{Q}}:=\mathbb{E}_{X\sim\mathbb{Q}}[k(X,x_{1:N})](k(x_{1:N},x_{1:N})+N\lambda_N I_{d_N})^{-1}h^{*}(x_{1:N}) I ^ K Q := E X ∼ Q [ k ( X , x 1 : N )] ( k ( x 1 : N , x 1 : N ) + N λ N I d N ) − 1 h ∗ ( x 1 : N ) λ N ≍ N − 2 s d ( log N ) 2 s + 2 d \lambda_N\asymp N^{-\frac{2s}{d}}(\log N)^{\frac{2s+2}{d}} λ N ≍ N − d 2 s ( log N ) d 2 s + 2 N 0 > 0 N_{0}>0 N 0 > 0 N > N 0 N>N_{0} N > N 0
∣ I ^ K Q − ∫ X h ∗ ( x ) d Q ( x ) ∣ ≤ E τ N − s d ( log N ) s + 1 d (49) \left|\hat{I}_{K\mathbb{Q}}-\int_{\mathcal{X}}h^{*}(x)\,d\mathbb{Q}(x)\right|\leq\mathcal{E}\tau N^{-\frac{s}{d}}(\log N)^{\frac{s+1}{d}} \tag*{(49)} I ^ K Q − ∫ X h ∗ ( x ) d Q ( x ) ≤ E τ N − d s ( log N ) d s + 1 (49) holds with probability at least 1 − 4 e − τ 1-4e^{-\tau} 1 − 4 e − τ C \mathcal{C} C N N N
The proof of Theorem A.5 is a direct application of Theorem A.4 after observing the following.
∣ I ^ K Q − ∫ X h ∗ ( x ) d Q ( x ) ∣ ≤ ∫ X ∣ h ^ λ N ( x ) − h ∗ ( x ) ∣ d Q ( x ) = ∥ h ∗ − h ^ λ N ∥ L 1 ( Q ) ≤ ∥ h ∗ − h ^ λ N ∥ L 2 ( Q ) . \left|\widehat{I}_{K\mathbb{Q}}-\int_{\mathcal{X}} h^*(x)\,\mathrm{d}\mathbb{Q}(x)\right|
\leq\int_{\mathcal{X}}\left|\widehat{h}_{\lambda_N}(x)-h^*(x)\right|\,\mathrm{d}\mathbb{Q}(x)
= \lVert h^*-\widehat{h}_{\lambda_N}\rVert_{L_1(\mathbb{Q})}
\leq\lVert h^*-\widehat{h}_{\lambda_N}\rVert_{L_2(\mathbb{Q})}. I K Q − ∫ X h ∗ ( x ) d Q ( x ) ≤ ∫ X h λ N ( x ) − h ∗ ( x ) d Q ( x ) = ∥ h ∗ − h λ N ∥ L 1 ( Q ) ≤ ∥ h ∗ − h λ N ∥ L 2 ( Q ) . Remark A.6. We prove in Theorem A.4 that the generalization error of h ^ λ N \widehat{h}_{\lambda_N} h λ N O ~ ( N − s d ) \widetilde{\mathcal{O}}(N^{-\frac{s}{d}}) O ( N − d s ) O ( N − s 2 s + d ) \mathcal{O}(N^{-\frac{s}{2s+d}}) O ( N − 2 s + d s ) λ N ≍ N − 2 s d ( log N ) 2 s + 2 d \lambda_N \asymp N^{-\frac{2s}{d}}(\log N)^{\frac{2s+2}{d}} λ N ≍ N − d 2 s ( log N ) d 2 s + 2 λ N ≍ N − 2 s 2 s + d \lambda_N \asymp N^{-\frac{2s}{2s+d}} λ N ≍ N − 2 s + d 2 s O ~ ( N − s d ) \widetilde{\mathcal{O}}(N^{-\frac{s}{d}}) O ( N − d s )
Remark A.7 (Comparison to existing upper bound of kernel (Bayesian) quadrature). The upper bound in Theorem A.5 matches existing analysis based on scattered data approximation in the literature of both kernel quadrature and Bayesian quadrature (Sommariva and Vianello, 2006; Briol et al., 2019b; Wynne et al., 2021) and is known to be minimax optimal (Novak, 2016, 2006). Existing analysis takes λ = 0 \lambda=0 λ = 0 k ( x 1 : N , x 1 : N ) k(x_{1:N},x_{1:N}) k ( x 1 : N , x 1 : N ) λ N ≍ N − 2 s d ( log N ) 2 s + 2 d \lambda_N \asymp N^{-\frac{2s}{d}}(\log N)^{\frac{2s+2}{d}} λ N ≍ N − d 2 s ( log N ) d 2 s + 2
Proof of Theorem 3.1
For any θ ∈ Θ \theta\in\Theta θ ∈ Θ F ^ K Q : Θ → R \hat{F}_{K_Q}:\Theta\to\mathbb{R} F ^ K Q : Θ → R J ^ K Q : Θ → R \hat{J}_{K_Q}:\Theta\to\mathbb{R} J ^ K Q : Θ → R F ^ K Q ( θ t ) \hat{F}_{K_Q}(\theta_t) F ^ K Q ( θ t ) J ^ K Q ( θ t ) \hat{J}_{K_Q}(\theta_t) J ^ K Q ( θ t ) (9) to all θ ∈ Θ \theta\in\Theta θ ∈ Θ θ ∈ Θ \theta\in\Theta θ ∈ Θ x 1 : N ( θ ) : = [ x 1 ( θ ) , … , x N ( θ ) ] T x_{1:N}^{(\theta)} := [x_1^{(\theta)},\ldots,x_N^{(\theta)}]^{\mathsf{T}} x 1 : N ( θ ) := [ x 1 ( θ ) , … , x N ( θ ) ] T N N N P θ P_\theta P θ
J ^ K Q ( θ ; x 1 : N ( θ ) ) : = ( ∫ X k X ( x , x 1 : N ( θ ) ) d P θ ( x ) ) × ( k X ( x 1 : N ( θ ) , x 1 : N ( θ ) ) + N λ X I d N ) − 1 g ( x 1 : N ( θ ) , θ ) , (50) \hat{J}_{K_Q}(\theta;x_{1:N}^{(\theta)}) := \left(\int_{\mathcal{X}} k_{\mathcal{X}}(x,x_{1:N}^{(\theta)})\mathrm{d}P_\theta(x)\right) \times\left(k_{\mathcal{X}}(x_{1:N}^{(\theta)},x_{1:N}^{(\theta)})+N\lambda_{\mathcal{X}}I d_N\right)^{-1}g(x_{1:N}^{(\theta)},\theta), \tag*{(50)} J ^ K Q ( θ ; x 1 : N ( θ ) ) := ( ∫ X k X ( x , x 1 : N ( θ ) ) d P θ ( x ) ) × ( k X ( x 1 : N ( θ ) , x 1 : N ( θ ) ) + N λ X I d N ) − 1 g ( x 1 : N ( θ ) , θ ) , (50) F ^ K Q ( θ ; x 1 : N ( θ ) ) : = f ( J ^ K Q ( θ ; x 1 : N ( θ ) ) ) , (51) \hat{F}_{K_Q}(\theta;x_{1:N}^{(\theta)}) := f\left(\hat{J}_{K_Q}(\theta;x_{1:N}^{(\theta)})\right), \tag*{(51)} F ^ K Q ( θ ; x 1 : N ( θ ) ) := f ( J ^ K Q ( θ ; x 1 : N ( θ ) ) ) , (51) where we explicitly specify the dependence of samples x 1 : N ( θ ) x_{1:N}^{(\theta)} x 1 : N ( θ ) θ \theta θ
J ˉ K Q ( θ ) : = E x 1 : N ( θ ) ∼ P θ [ J ^ K Q ( θ ; x 1 : N ( θ ) ) ] = ∫ J ^ K Q ( θ ; x 1 : N ) ∏ i = 1 N p ( x i ; θ ) d x 1 : N , \bar{J}_{K_Q}(\theta) := \mathbb{E}_{x_{1:N}^{(\theta)}\sim P_\theta}\left[\hat{J}_{K_Q}(\theta;x_{1:N}^{(\theta)})\right] = \int\hat{J}_{K_Q}(\theta;x_{1:N})\prod_{i=1}^{N}p(x_i;\theta)\mathrm{d}x_{1:N}, J ˉ K Q ( θ ) := E x 1 : N ( θ ) ∼ P θ [ J ^ K Q ( θ ; x 1 : N ( θ ) ) ] = ∫ J ^ K Q ( θ ; x 1 : N ) i = 1 ∏ N p ( x i ; θ ) d x 1 : N , F ˉ K Q ( θ ) : = E x 1 : N ( θ ) ∼ P θ [ F ^ K Q ( θ ; x 1 : N ( θ ) ) ] = ∫ F ^ K Q ( θ ; x 1 : N ) ∏ i = 1 N p ( x i ; θ ) d x 1 : N , (52) \bar{F}_{K_Q}(\theta) := \mathbb{E}_{x_{1:N}^{(\theta)}\sim P_\theta}\left[\hat{F}_{K_Q}(\theta;x_{1:N}^{(\theta)})\right] = \int\hat{F}_{K_Q}(\theta;x_{1:N})\prod_{i=1}^{N}p(x_i;\theta)\mathrm{d}x_{1:N}, \tag*{(52)} F ˉ K Q ( θ ) := E x 1 : N ( θ ) ∼ P θ [ F ^ K Q ( θ ; x 1 : N ( θ ) ) ] = ∫ F ^ K Q ( θ ; x 1 : N ) i = 1 ∏ N p ( x i ; θ ) d x 1 : N , (52) which marginalize out the dependence on samples x 1 : N ( θ ) x_{1:N}^{(\theta)} x 1 : N ( θ ) J ˉ K Q ∈ L 2 ( Q ) \bar{J}_{K_Q} \in L_2(\mathbb{Q}) J ˉ K Q ∈ L 2 ( Q ) g ( x , ⋅ ) ∈ W 2 s Θ ( Θ ) ⊂ L 2 ( Θ ) ≅ L 2 ( Q ) g(x,\cdot) \in W_2^{s_\Theta}(\Theta) \subset L_2(\Theta) \cong L_2(\mathbb{Q}) g ( x , ⋅ ) ∈ W 2 s Θ ( Θ ) ⊂ L 2 ( Θ ) ≅ L 2 ( Q ) p ( x i ; ⋅ ) ∈ L 2 ( Q ) p(x_i;\cdot) \in L_2(\mathbb{Q}) p ( x i ; ⋅ ) ∈ L 2 ( Q ) F ˉ K Q ∈ L 2 ( Q ) \bar{F}_{K_Q} \in L_2(\mathbb{Q}) F ˉ K Q ∈ L 2 ( Q ) f f f ∣ I − I ^ N K Q ∣ |I-\hat{I}_{N K_Q}| ∣ I − I ^ N K Q ∣
∣ I − I ^ N K Q ∣ = ∣ ∫ Θ F ( θ ) q ( θ ) d θ − ( ∫ Θ k Θ ( θ , θ 1 : T ) q ( θ ) d θ ) × ( k Θ ( θ 1 : T , θ 1 : T ) + T λ Θ I d T ) − 1 F ^ K Q ( θ 1 : T ) ∣ \left|I-\hat{I}_{N K_Q}\right|
= \left|\int_\Theta F(\theta)q(\theta)\mathrm{d}\theta-\left(\int_\Theta k_\Theta(\theta,\theta_{1:T})q(\theta)\mathrm{d}\theta\right)\times\left(k_\Theta(\theta_{1:T},\theta_{1:T})+T\lambda_\Theta I d_T\right)^{-1}\hat{F}_{K_Q}(\theta_{1:T})\right| I − I ^ N K Q = ∫ Θ F ( θ ) q ( θ ) d θ − ( ∫ Θ k Θ ( θ , θ 1 : T ) q ( θ ) d θ ) × ( k Θ ( θ 1 : T , θ 1 : T ) + T λ Θ I d T ) − 1 F ^ K Q ( θ 1 : T ) ≤ ∣ ∫ Θ F ( θ ) q ( θ ) d θ − ∫ Θ F ˉ K Q ( θ ) q ( θ ) d θ ∣ + ∣ ∫ Θ F ˉ K Q ( θ ) q ( θ ) d θ − ( ∫ Θ k Θ ( θ , θ 1 : T ) q ( θ ) d θ ) × ( k Θ ( θ 1 : T , θ 1 : T ) + T λ Θ Id T ) − 1 F ^ K Q ( θ 1 : T ) ∣ ≤ E Θ ∼ Q [ ∣ F ( θ ) − F ˉ K Q ( θ ) ∣ ] ⏟ Stage I error + ∥ F ˉ K Q ( ⋅ ) − k ( ⋅ , θ 1 : T ) ( k Θ ( θ 1 : T , θ 1 : T ) + T λ Θ Id T ) − 1 F ^ K Q ( θ 1 : T ) ∥ L 2 ( Q ) ⏟ Stage II error . (53) \begin{aligned}
\leq{}& \left|\int_{\Theta} F(\theta)q(\theta)\,d\theta-\int_{\Theta}\bar{F}_{KQ}(\theta)q(\theta)\,d\theta\right|\\
&+\left|\int_{\Theta}\bar{F}_{KQ}(\theta)q(\theta)\,d\theta-\left(\int_{\Theta}k_{\Theta}(\theta,\theta_{1:T})q(\theta)\,d\theta\right)\right.\\
&\qquad\left.\times\left(k_{\Theta}(\theta_{1:T},\theta_{1:T})+T\lambda_{\Theta}\operatorname{Id}_{T}\right)^{-1}\hat{F}_{KQ}(\theta_{1:T})\right|\\
\leq{}& \underbrace{\mathbb{E}_{\Theta\sim\mathbb{Q}}\left[\left|F(\theta)-\bar{F}_{KQ}(\theta)\right|\right]}_{\text{Stage I error}}\\
&+\underbrace{\left\|\bar{F}_{KQ}(\cdot)-k(\cdot,\theta_{1:T})\left(k_{\Theta}(\theta_{1:T},\theta_{1:T})+T\lambda_{\Theta}\operatorname{Id}_{T}\right)^{-1}\hat{F}_{KQ}(\theta_{1:T})\right\|_{L_{2}(\mathbb{Q})}}_{\text{Stage II error}}. \tag*{(53)}
\end{aligned} ≤ ≤ ∫ Θ F ( θ ) q ( θ ) d θ − ∫ Θ F ˉ K Q ( θ ) q ( θ ) d θ + ∫ Θ F ˉ K Q ( θ ) q ( θ ) d θ − ( ∫ Θ k Θ ( θ , θ 1 : T ) q ( θ ) d θ ) × ( k Θ ( θ 1 : T , θ 1 : T ) + T λ Θ Id T ) − 1 F ^ K Q ( θ 1 : T ) Stage I error E Θ ∼ Q [ F ( θ ) − F ˉ K Q ( θ ) ] + Stage II error F ˉ K Q ( ⋅ ) − k ( ⋅ , θ 1 : T ) ( k Θ ( θ 1 : T , θ 1 : T ) + T λ Θ Id T ) − 1 F ^ K Q ( θ 1 : T ) L 2 ( Q ) . (53) The last inequality holds because ∥ ⋅ ∥ L 1 ( Q ) ≤ ∥ ⋅ ∥ L 2 ( Q ) \|\cdot\|_{L_{1}(\mathbb{Q})}\leq\|\cdot\|_{L_{2}(\mathbb{Q})} ∥ ⋅ ∥ L 1 ( Q ) ≤ ∥ ⋅ ∥ L 2 ( Q )
Stage I Error
From Assumption (4), f f f S 4 S_{4} S 4
∣ F ˉ K Q ( θ ) − F ( θ ) ∣ = ∣ E x 1 : N ( θ ) ∼ P θ [ F ^ K Q ( θ ; x 1 : N ( θ ) ) − F ( θ ) ] ∣ ≤ E x 1 : N ( θ ) ∼ P θ [ ∣ F ^ K Q ( θ ; x 1 : N ( θ ) ) − F ( θ ) ∣ ] ≤ S 4 E x 1 : N ( θ ) ∼ P θ [ ∣ J ^ K Q ( θ ; x 1 : N ( θ ) ) − J ( θ ) ∣ ] , (54) \begin{aligned}
\left|\bar{F}_{KQ}(\theta)-F(\theta)\right|=\left|\mathbb{E}_{x_{1:N}^{(\theta)}\sim\mathbb{P}_{\theta}}\left[\hat{F}_{KQ}(\theta;x_{1:N}^{(\theta)})-F(\theta)\right]\right| \\
&\leq\mathbb{E}_{x_{1:N}^{(\theta)}\sim\mathbb{P}_{\theta}}\left[\left|\hat{F}_{KQ}(\theta;x_{1:N}^{(\theta)})-F(\theta)\right|\right] \\
&\leq S_{4}\mathbb{E}_{x_{1:N}^{(\theta)}\sim\mathbb{P}_{\theta}}\left[\left|\hat{J}_{KQ}(\theta;x_{1:N}^{(\theta)})-J(\theta)\right|\right], \tag*{(54)}
\end{aligned} F ˉ K Q ( θ ) − F ( θ ) = E x 1 : N ( θ ) ∼ P θ [ F ^ K Q ( θ ; x 1 : N ( θ ) ) − F ( θ ) ] ≤ E x 1 : N ( θ ) ∼ P θ [ F ^ K Q ( θ ; x 1 : N ( θ ) ) − F ( θ ) ] ≤ S 4 E x 1 : N ( θ ) ∼ P θ [ J ^ K Q ( θ ; x 1 : N ( θ ) ) − J ( θ ) ] , (54) where the first inequality holds by Jensen inequality and the last inequality holds by Lipschitz continuity of f f f
g ^ ( x , θ ; x 1 : N ( θ ) ) = k X ( x , x 1 : N ( θ ) ) ( k X ( x 1 : N ( θ ) , x 1 : N ( θ ) ) + N λ X Id N ) − 1 g ( x 1 : N ( θ ) , θ ) . (55) \hat{g}(x,\theta;x_{1:N}^{(\theta)})=k_{\mathcal{X}}(x,x_{1:N}^{(\theta)})\left(k_{\mathcal{X}}(x_{1:N}^{(\theta)},x_{1:N}^{(\theta)})+N\lambda_{\mathcal{X}}\operatorname{Id}_{N}\right)^{-1}g(x_{1:N}^{(\theta)},\theta). \tag*{(55)} g ^ ( x , θ ; x 1 : N ( θ ) ) = k X ( x , x 1 : N ( θ ) ) ( k X ( x 1 : N ( θ ) , x 1 : N ( θ ) ) + N λ X Id N ) − 1 g ( x 1 : N ( θ ) , θ ) . (55) Here g ^ ( ⋅ , θ ; x 1 : N ( θ ) ) ∈ L 2 ( P θ ) \hat{g}(\cdot,\theta;x_{1:N}^{(\theta)})\in L_{2}(\mathbb{P}_{\theta}) g ^ ( ⋅ , θ ; x 1 : N ( θ ) ) ∈ L 2 ( P θ ) k X k_{\mathcal{X}} k X g ( ⋅ , θ ) ∈ L 2 ( P θ ) g(\cdot,\theta)\in L_{2}(\mathbb{P}_{\theta}) g ( ⋅ , θ ) ∈ L 2 ( P θ )
∣ J ^ K Q ( θ ; x 1 : N ( θ ) ) − J ( θ ) ∣ = ∣ ∫ ( g ^ ( x , θ ; x 1 : N ( θ ) ) − g ( x , θ ) ) p ( x ; θ ) d x ∣ ≤ ∥ g ^ ( ⋅ , θ ; x 1 : N ( θ ) ) − g ( ⋅ , θ ) ∥ L 2 ( P θ ) . (56) \begin{aligned}
\left|\hat{J}_{KQ}(\theta;x_{1:N}^{(\theta)})-J(\theta)\right|=\left|\int\left(\hat{g}(x,\theta;x_{1:N}^{(\theta)})-g(x,\theta)\right)p(x;\theta)\,dx\right| \\
&\leq\left\|\hat{g}(\cdot,\theta;x_{1:N}^{(\theta)})-g(\cdot,\theta)\right\|_{L_{2}(\mathbb{P}_{\theta})}. \tag*{(56)}
\end{aligned} J ^ K Q ( θ ; x 1 : N ( θ ) ) − J ( θ ) = ∫ ( g ^ ( x , θ ; x 1 : N ( θ ) ) − g ( x , θ ) ) p ( x ; θ ) d x ≤ g ^ ( ⋅ , θ ; x 1 : N ( θ ) ) − g ( ⋅ , θ ) L 2 ( P θ ) . (56) Based on Assumption (2), g ( ⋅ , θ ) ∈ W 2 s X ( X ) g(\cdot,\theta)\in W_{2}^{s_{\mathcal{X}}}(\mathcal{X}) g ( ⋅ , θ ) ∈ W 2 s X ( X ) θ ∈ Θ \theta\in\Theta θ ∈ Θ λ X , N ≍ N − 2 s X d X ( log N ) 2 s X + 2 d X \lambda_{\mathcal{X},N}\asymp N^{-2\frac{s_{\mathcal{X}}}{d_{\mathcal{X}}}}(\log N)^{\frac{2s_{\mathcal{X}}+2}{d_{\mathcal{X}}}} λ X , N ≍ N − 2 d X s X ( log N ) d X 2 s X + 2 N 0 N_{0} N 0 N > N 0 N > N_0 N > N 0
∥ g ^ ( ⋅ , θ ; x 1 : N ( θ ) ) − g ( ⋅ , θ ) ∥ L 2 ( P θ ) ≤ C τ N − s X d X ( log N ) s X + 1 d X ∥ g ( ⋅ , θ ) ∥ s X , 2 . (57) \left\|\widehat{g}(\cdot,\theta;x_{1:N}^{(\theta)})-g(\cdot,\theta)\right\|_{L_2(\mathbb{P}_\theta)}
\leq\mathcal{C}_{\tau}N^{-\frac{s_X}{d_X}}(\log N)^{\frac{s_X+1}{d_X}}\left\|g(\cdot,\theta)\right\|_{s_X,2}. \tag*{(57)} g ( ⋅ , θ ; x 1 : N ( θ ) ) − g ( ⋅ , θ ) L 2 ( P θ ) ≤ C τ N − d X s X ( log N ) d X s X + 1 ∥ g ( ⋅ , θ ) ∥ s X , 2 . (57) holds with probability at least 1 − 4 e − τ 1-4e^{-\tau} 1 − 4 e − τ x 1 : N ( θ ) x_{1:N}^{(\theta)} x 1 : N ( θ ) P θ \mathbb{P}_\theta P θ C \mathcal{C} C N N N
E x 1 : N ( θ ) ∼ P θ [ ∥ g ^ ( ⋅ , θ ; x 1 : N ( θ ) ) − g ( ⋅ , θ ) ∥ L 2 ( P θ ) ] ≤ C N − s X d X ( log N ) s X + 1 d X ∥ g ( ⋅ , θ ) ∥ s X , 2 . (58) \mathbb{E}_{x_{1:N}^{(\theta)}\sim\mathbb{P}_\theta}\left[\left\|\widehat{g}(\cdot,\theta;x_{1:N}^{(\theta)})-g(\cdot,\theta)\right\|_{L_2(\mathbb{P}_\theta)}\right]
\leq\mathcal{C}N^{-\frac{s_X}{d_X}}(\log N)^{\frac{s_X+1}{d_X}}\left\|g(\cdot,\theta)\right\|_{s_X,2}. \tag*{(58)} E x 1 : N ( θ ) ∼ P θ [ g ( ⋅ , θ ; x 1 : N ( θ ) ) − g ( ⋅ , θ ) L 2 ( P θ ) ] ≤ C N − d X s X ( log N ) d X s X + 1 ∥ g ( ⋅ , θ ) ∥ s X , 2 . (58) By plugging the above inequality back into (A.15), we obtain
E x 1 : N ( θ ) ∼ P θ [ ∣ J ^ K Q ( θ ; x 1 : N ( θ ) ) − J ( θ ) ∣ ] ≤ C N − s X d X ( log N ) s X + 1 d X ∥ g ( ⋅ , θ ) ∥ s X , 2 . \mathbb{E}_{x_{1:N}^{(\theta)}\sim\mathbb{P}_\theta}\left[\left|\widehat{J}_{KQ}(\theta;x_{1:N}^{(\theta)})-J(\theta)\right|\right]
\leq\mathcal{C}N^{-\frac{s_X}{d_X}}(\log N)^{\frac{s_X+1}{d_X}}\left\|g(\cdot,\theta)\right\|_{s_X,2}. E x 1 : N ( θ ) ∼ P θ [ J K Q ( θ ; x 1 : N ( θ ) ) − J ( θ ) ] ≤ C N − d X s X ( log N ) d X s X + 1 ∥ g ( ⋅ , θ ) ∥ s X , 2 . Therefore, the Stage I error can be upper bounded by
E θ ∼ Q [ ∣ F ( θ ) − F ‾ K Q ( θ ) ∣ ] ≤ S 4 E x 1 : N ( θ ) ∼ P θ [ ∣ J ^ K Q ( θ ; x 1 : N ( θ ) ) − J ( θ ) ∣ ] ∥ g ( ⋅ , θ ) ∥ s X , 2 ≤ S 4 S 1 C N − s X d X ( log N ) s X + 1 d X = C 3 N − s X d X ( log N ) s X + 1 d X , (59) \begin{aligned}
\mathbb{E}_{\theta\sim Q}\left[\left|F(\theta)-\overline{F}_{KQ}(\theta)\right|\right]
\leq S_4\mathbb{E}_{x_{1:N}^{(\theta)}\sim\mathbb{P}_\theta}\left[\left|\widehat{J}_{KQ}(\theta;x_{1:N}^{(\theta)})-J(\theta)\right|\right]\left\|g(\cdot,\theta)\right\|_{s_X,2} \\
&\leq S_4S_1\mathcal{C}N^{-\frac{s_X}{d_X}}(\log N)^{\frac{s_X+1}{d_X}} \\
&= C_3N^{-\frac{s_X}{d_X}}(\log N)^{\frac{s_X+1}{d_X}}, \tag*{(59)}
\end{aligned} E θ ∼ Q [ F ( θ ) − F K Q ( θ ) ] ≤ S 4 E x 1 : N ( θ ) ∼ P θ [ J K Q ( θ ; x 1 : N ( θ ) ) − J ( θ ) ] ∥ g ( ⋅ , θ ) ∥ s X , 2 ≤ S 4 S 1 C N − d X s X ( log N ) d X s X + 1 = C 3 N − d X s X ( log N ) d X s X + 1 , (59) where C 3 : = S 4 S 1 C C_3:=S_4S_1\mathcal{C} C 3 := S 4 S 1 C N N N
Stage II Error The upper bound on the stage II error is done in five steps. In step one, we prove that J ^ K Q ( ⋅ ; x 1 : N ( θ ) ) ∈ W 2 s Θ ( Θ ) \widehat{J}_{KQ}(\cdot;x_{1:N}^{(\theta)})\in W_2^{s_\Theta}(\Theta) J K Q ( ⋅ ; x 1 : N ( θ ) ) ∈ W 2 s Θ ( Θ ) x 1 : N ( θ ) x_{1:N}^{(\theta)} x 1 : N ( θ ) J ∈ W 2 s Θ ( Θ ) J\in W_2^{s_\Theta}(\Theta) J ∈ W 2 s Θ ( Θ ) ∥ J ^ K Q ( ⋅ ; x 1 : N ( θ ) ) ∥ s Θ , 2 \left\|\widehat{J}_{KQ}(\cdot;x_{1:N}^{(\theta)})\right\|_{s_\Theta,2} J K Q ( ⋅ ; x 1 : N ( θ ) ) s Θ , 2 ∥ J ^ K Q ( ⋅ ; x 1 : N ( θ ) ) ∥ s Θ , 2 ≤ ∥ J ∥ s Θ , 2 + ∥ J − J ^ K Q ( ⋅ ; x 1 : N ( θ ) ) ∥ s Θ , 2 \left\|\widehat{J}_{KQ}(\cdot;x_{1:N}^{(\theta)})\right\|_{s_\Theta,2}\leq\left\|J\right\|_{s_\Theta,2}+\left\|J-\widehat{J}_{KQ}(\cdot;x_{1:N}^{(\theta)})\right\|_{s_\Theta,2} J K Q ( ⋅ ; x 1 : N ( θ ) ) s Θ , 2 ≤ ∥ J ∥ s Θ , 2 + J − J K Q ( ⋅ ; x 1 : N ( θ ) ) s Θ , 2 F ‾ K Q ( θ ) = E x 1 : N ( θ ) [ f ( J ^ K Q ( ⋅ ; x 1 : N ( θ ) ) ) ] \overline{F}_{KQ}(\theta)=\mathbb{E}_{x_{1:N}^{(\theta)}}\left[f\left(\widehat{J}_{KQ}(\cdot;x_{1:N}^{(\theta)})\right)\right] F K Q ( θ ) = E x 1 : N ( θ ) [ f ( J K Q ( ⋅ ; x 1 : N ( θ ) ) ) ] x 1 : N ( θ ) x_{1:N}^{(\theta)} x 1 : N ( θ )
Step One. In this step, we are going to show that J ^ K Q \widehat{J}_{KQ} J K Q W 2 s Θ ( Θ ) W_{2}^{s_{\Theta}}(\Theta) W 2 s Θ ( Θ ) x 1 : N ( θ ) x_{1:N}^{(\theta)} x 1 : N ( θ ) J ^ K Q ( θ ) \hat{J}_{\mathrm{KQ}}(\theta) J ^ KQ ( θ ) θ \theta θ θ ↦ ∫ X k X ( x , x 1 : N ( θ ) ) p ( x ; θ ) d x \theta\mapsto\int_{\mathcal{X}} k_{\mathcal{X}}(x,x_{1:N}^{(\theta)})p(x;\theta)dx θ ↦ ∫ X k X ( x , x 1 : N ( θ ) ) p ( x ; θ ) d x θ ↦ g ( x 1 : N ( θ ) , θ ) \theta\mapsto g(x_{1:N}^{(\theta)},\theta) θ ↦ g ( x 1 : N ( θ ) , θ )
We are going to show that θ ↦ ∫ X k X ( x , x i ( θ ) ) p ( x ; θ ) d x \theta\mapsto\int_{\mathcal{X}} k_{\mathcal{X}}(x,x_i^{(\theta)})p(x;\theta)dx θ ↦ ∫ X k X ( x , x i ( θ ) ) p ( x ; θ ) d x W 2 s Θ ( Θ ) W_{2}^{s_{\Theta}}(\Theta) W 2 s Θ ( Θ ) i ∈ { 1 , … , N } i \in\{1,\ldots,N\} i ∈ { 1 , … , N } s Θ s_{\Theta} s Θ L 2 ( Θ ) \mathcal{L}^{2}(\Theta) L 2 ( Θ ) φ : Θ → R \varphi:\Theta\to\mathbb{R} φ : Θ → R Θ \Theta Θ φ ∈ C c ∞ ( Θ ) \varphi\in C_{c}^{\infty}(\Theta) φ ∈ C c ∞ ( Θ ) β \beta β ∂ β φ \partial^{\beta}\varphi ∂ β φ θ ↦ p ( x ; θ ) ∈ W 2 s Θ ( Θ ) \theta\mapsto p(x;\theta)\in W_{2}^{s_{\Theta}}(\Theta) θ ↦ p ( x ; θ ) ∈ W 2 s Θ ( Θ ) ∣ β ∣ ≤ s Θ |\beta|\le s_{\Theta} ∣ β ∣ ≤ s Θ θ ↦ D θ β p ( x ; θ ) ∈ L 2 ( Θ ) \theta\mapsto D_{\theta}^{\beta}p(x;\theta)\in\mathcal{L}^{2}(\Theta) θ ↦ D θ β p ( x ; θ ) ∈ L 2 ( Θ )
∫ Θ φ ( θ ) ∫ X k X ( x , x i ( θ ) ) D θ β p ( x ; θ ) d x d θ = ( i ) ∫ X k X ( x , x i ( θ ) ) ∫ Θ φ ( θ ) D θ β p ( x ; θ ) d θ d x = ( i i ) ( − 1 ) ∣ β ∣ ∫ X k X ( x , x i ( θ ) ) ∫ Θ ∂ β φ ( θ ) p ( x ; θ ) d θ d x = ( i i i ) ( − 1 ) ∣ β ∣ ∫ Θ ∂ β φ ( θ ) ∫ X k X ( x , x i ( θ ) ) p ( x ; θ ) d x d θ . (60) \begin{aligned}
\int_{\Theta}\varphi(\theta)\int_{\mathcal{X}}k_{\mathcal{X}}(x,x_i^{(\theta)})D_{\theta}^{\beta}p(x;\theta)\,dx\,d\theta
&\overset{(i)}{=}\int_{\mathcal{X}}k_{\mathcal{X}}(x,x_i^{(\theta)})\int_{\Theta}\varphi(\theta)D_{\theta}^{\beta}p(x;\theta)\,d\theta\,dx\\
&\overset{(ii)}{=}(-1)^{|\beta|}\int_{\mathcal{X}}k_{\mathcal{X}}(x,x_i^{(\theta)})\int_{\Theta}\partial^{\beta}\varphi(\theta)p(x;\theta)\,d\theta\,dx\\
&\overset{(iii)}{=}(-1)^{|\beta|}\int_{\Theta}\partial^{\beta}\varphi(\theta)\int_{\mathcal{X}}k_{\mathcal{X}}(x,x_i^{(\theta)})p(x;\theta)\,dx\,d\theta. \tag*{(60)}
\end{aligned} ∫ Θ φ ( θ ) ∫ X k X ( x , x i ( θ ) ) D θ β p ( x ; θ ) d x d θ = ( i ) ∫ X k X ( x , x i ( θ ) ) ∫ Θ φ ( θ ) D θ β p ( x ; θ ) d θ d x = ( ii ) ( − 1 ) ∣ β ∣ ∫ X k X ( x , x i ( θ ) ) ∫ Θ ∂ β φ ( θ ) p ( x ; θ ) d θ d x = ( iii ) ( − 1 ) ∣ β ∣ ∫ Θ ∂ β φ ( θ ) ∫ X k X ( x , x i ( θ ) ) p ( x ; θ ) d x d θ . (60) In the above chain of derivations, we are allowed to swap the integration order in ( i ) (i) ( i ) k X k_{\mathcal{X}} k X θ ↦ φ ( θ ) ⋅ D θ β p ( x ; θ ) ∈ L 1 ( Θ ) \theta\mapsto\varphi(\theta)\cdot D_{\theta}^{\beta}p(x;\theta)\in L_{1}(\Theta) θ ↦ φ ( θ ) ⋅ D θ β p ( x ; θ ) ∈ L 1 ( Θ ) D θ β p ( x ; ⋅ ) ∈ L 2 ( Θ ) D_{\theta}^{\beta}p(x;\cdot)\in L_{2}(\Theta) D θ β p ( x ; ⋅ ) ∈ L 2 ( Θ ) φ ∈ L 2 ( Θ ) \varphi\in L_{2}(\Theta) φ ∈ L 2 ( Θ ) ( i i ) (ii) ( ii ) D θ β p ( x ; θ ) D_{\theta}^{\beta}p(x;\theta) D θ β p ( x ; θ ) ( i i i ) (iii) ( iii ) ∫ X k X ( x , x i ( θ ) ) p ( x ; θ ) d x \int_{\mathcal{X}}k_{\mathcal{X}}(x,x_i^{(\theta)})p(x;\theta)\,dx ∫ X k X ( x , x i ( θ ) ) p ( x ; θ ) d x β \beta β
D θ β [ ∫ X k X ( x , x i ( θ ) ) p ( x ; θ ) d x ] = ∫ X k X ( x , x i ( θ ) ) D θ β p ( x ; θ ) d x D_{\theta}^{\beta}\left[\int_{\mathcal{X}}k_{\mathcal{X}}(x,x_i^{(\theta)})p(x;\theta)\,dx\right]=\int_{\mathcal{X}}k_{\mathcal{X}}(x,x_i^{(\theta)})D_{\theta}^{\beta}p(x;\theta)\,dx D θ β [ ∫ X k X ( x , x i ( θ ) ) p ( x ; θ ) d x ] = ∫ X k X ( x , x i ( θ ) ) D θ β p ( x ; θ ) d x Also, since k X k_{\mathcal{X}} k X θ ↦ D θ β p ( x ; θ ) ∈ L 2 ( Θ ) \theta\mapsto D_{\theta}^{\beta}p(x;\theta)\in L^{2}(\Theta) θ ↦ D θ β p ( x ; θ ) ∈ L 2 ( Θ ) L 2 ( Θ ) \mathcal{L}^{2}(\Theta) L 2 ( Θ ) f 1 , i ( θ ) : = ∫ X k X ( x , x i ( θ ) ) p ( x ; θ ) d x f_{1,i}(\theta):=\int_{\mathcal{X}}k_{\mathcal{X}}(x,x_i^{(\theta)})p(x;\theta)\,dx f 1 , i ( θ ) := ∫ X k X ( x , x i ( θ ) ) p ( x ; θ ) d x
∑ ∣ β ∣ ≤ s Θ ∥ D θ β f 1 , i ∥ L 2 ( Θ ) 2 = ∑ ∣ β ∣ ≤ s Θ ∥ ∫ X k X ( x , x i ( θ ) ) D θ β p ( x ; ⋅ ) d x ∥ L 2 ( Θ ) 2 ≤ ( i ) Vol ( X ) ∑ ∣ β ∣ ≤ s Θ ∫ Θ ∫ X ∣ k X ( x , x i ( θ ) ) D θ β p ( x ; θ ) ∣ 2 d x d θ ≤ Vol ( X ) κ 2 ∑ ∣ β ∣ ≤ s Θ ∫ Θ ∫ X ∣ D θ β p ( x ; θ ) ∣ 2 d x d θ ≡ Vol ( X ) κ 2 ∫ X ∥ p ( x ; ⋅ ) ∥ s Θ , 2 2 d x . \begin{aligned}
\sum_{|\beta|\le s_{\Theta}}\left\|D_{\theta}^{\beta}f_{1,i}\right\|_{L_{2}(\Theta)}^{2}
=\sum_{|\beta|\le s_{\Theta}}\left\|\int_{\mathcal{X}}k_{\mathcal{X}}(x,x_i^{(\theta)})D_{\theta}^{\beta}p(x;\cdot)\,dx\right\|_{L_{2}(\Theta)}^{2}
\overset{(i)}{\le}\operatorname{Vol}(\mathcal{X})\sum_{|\beta|\le s_{\Theta}}\int_{\Theta}\int_{\mathcal{X}}\left|k_{\mathcal{X}}(x,x_i^{(\theta)})D_{\theta}^{\beta}p(x;\theta)\right|^{2}\,dx\,d\theta\\
&\leq\operatorname{Vol}(\mathcal{X})\kappa^2 \sum_{|\beta|\leq s_{\Theta}} \int_{\Theta}\int_{\mathcal{X}}\left|D_{\theta}^{\beta}p(x;\theta)\right|^2\,dx\,d\theta\\
&\equiv\operatorname{Vol}(\mathcal{X})\kappa^2\int_{\mathcal{X}}\|p(x;\cdot)\|_{s_{\Theta},2}^2\,dx.
\end{aligned} ∣ β ∣ ≤ s Θ ∑ D θ β f 1 , i L 2 ( Θ ) 2 = ∣ β ∣ ≤ s Θ ∑ ∫ X k X ( x , x i ( θ ) ) D θ β p ( x ; ⋅ ) d x L 2 ( Θ ) 2 ≤ ( i ) Vol ( X ) ∣ β ∣ ≤ s Θ ∑ ∫ Θ ∫ X k X ( x , x i ( θ ) ) D θ β p ( x ; θ ) 2 d x d θ ≤ Vol ( X ) κ 2 ∣ β ∣ ≤ s Θ ∑ ∫ Θ ∫ X D θ β p ( x ; θ ) 2 d x d θ ≡ Vol ( X ) κ 2 ∫ X ∥ p ( x ; ⋅ ) ∥ s Θ , 2 2 d x . In the above chain of derivations, ( i ) (i) ( i ) ∣ ∫ X f ( x ) d x ∣ 2 ≤ Vol ( X ) ∫ X ∣ f ( x ) ∣ 2 d x \left|\int_{\mathcal{X}}f(x)dx\right|^2\leq\operatorname{Vol}(\mathcal{X})\int_{\mathcal{X}}|f(x)|^2dx ∫ X f ( x ) d x 2 ≤ Vol ( X ) ∫ X ∣ f ( x ) ∣ 2 d x X \mathcal{X} X ( i i ) (ii) ( ii ) k X k_{\mathcal{X}} k X κ \kappa κ ∫ Θ ∣ D θ β p ( x ; θ ) ∣ 2 d θ < ∞ \int_{\Theta}|D_{\theta}^{\beta}p(x;\theta)|^2d\theta<\infty ∫ Θ ∣ D θ β p ( x ; θ ) ∣ 2 d θ < ∞ ( i i i ) (iii) ( iii ) p ( x ; ⋅ ) ∈ W 2 s Θ ( Θ ) p(x;\cdot)\in W_2^{s_{\Theta}}(\Theta) p ( x ; ⋅ ) ∈ W 2 s Θ ( Θ ) x ∈ X x\in\mathcal{X} x ∈ X ( i i i ) (iii) ( iii )
As a result, for any i , j ∈ { 1 , … , N } i,j\in\{1,\ldots,N\} i , j ∈ { 1 , … , N } f 1 , i : θ ↦ ∫ X k X ( x , x i ( θ ) ) p ( x ; θ ) d x f_{1,i}:\theta\mapsto\int_{\mathcal{X}}k_{\mathcal{X}}(x,x_i^{(\theta)})p(x;\theta)dx f 1 , i : θ ↦ ∫ X k X ( x , x i ( θ ) ) p ( x ; θ ) d x f 2 , j : θ ↦ g ( x j ( θ ) , θ ) f_{2,j}:\theta\mapsto g(x_j^{(\theta)},\theta) f 2 , j : θ ↦ g ( x j ( θ ) , θ ) W 2 s Θ ( Θ ) W_2^{s_{\Theta}}(\Theta) W 2 s Θ ( Θ ) f 1 , i ⋅ f 2 , j ∈ W 2 s Θ ( Θ ) f_{1,i}\cdot f_{2,j}\in W_2^{s_{\Theta}}(\Theta) f 1 , i ⋅ f 2 , j ∈ W 2 s Θ ( Θ ) J K Q \mathcal{J}_{KQ} J K Q f 1 , i ⋅ f 2 , j f_{1,i}\cdot f_{2,j} f 1 , i ⋅ f 2 , j W 2 s Θ ( Θ ) W_2^{s_{\Theta}}(\Theta) W 2 s Θ ( Θ )
Step Two. In this step, we are going to show that J : θ ↦ ∫ X g ( x , θ ) p ( x ; θ ) d x \mathcal{J}:\theta\mapsto\int_{\mathcal{X}}g(x,\theta)p(x;\theta)dx J : θ ↦ ∫ X g ( x , θ ) p ( x ; θ ) d x W 2 s Θ ( Θ ) W_2^{s_{\Theta}}(\Theta) W 2 s Θ ( Θ ) g ( x , ⋅ ) ∈ W 2 s Θ ( Θ ) g(x,\cdot)\in W_2^{s_{\Theta}}(\Theta) g ( x , ⋅ ) ∈ W 2 s Θ ( Θ ) p ( x ; ⋅ ) ∈ W 2 s Θ ( Θ ) p(x;\cdot)\in W_2^{s_{\Theta}}(\Theta) p ( x ; ⋅ ) ∈ W 2 s Θ ( Θ ) θ ↦ g ( x , θ ) ⋅ p ( x , θ ) ∈ W 2 s Θ ( Θ ) \theta\mapsto g(x,\theta)\cdot p(x,\theta)\in W_2^{s_{\Theta}}(\Theta) θ ↦ g ( x , θ ) ⋅ p ( x , θ ) ∈ W 2 s Θ ( Θ ) ∣ β ∣ ≤ s Θ |\beta|\leq s_{\Theta} ∣ β ∣ ≤ s Θ
D θ β ∫ X g ( x , θ ) p ( x ; θ ) d x = ∫ X D θ β ( g ( x , θ ) p ( x ; θ ) ) d x . (61) D_{\theta}^{\beta}\int_{\mathcal{X}}g(x,\theta)p(x;\theta)dx=\int_{\mathcal{X}}D_{\theta}^{\beta}\left(g(x,\theta)p(x;\theta)\right)dx. \tag*{(61)} D θ β ∫ X g ( x , θ ) p ( x ; θ ) d x = ∫ X D θ β ( g ( x , θ ) p ( x ; θ ) ) d x . (61) We are now ready to study the Sobolev norm of J \mathcal{J} J
∥ J ∥ s Θ , 2 2 : = ∑ ∣ β ∣ ≤ s Θ ∫ Θ ∣ D θ β ∫ X p ( x ; θ ) g ( x , θ ) d x ∣ 2 d θ \|\mathcal{J}\|_{s_{\Theta},2}^2:=\sum_{|\beta|\leq s_{\Theta}}\int_{\Theta}\left|D_{\theta}^{\beta}\int_{\mathcal{X}}p(x;\theta)g(x,\theta)dx\right|^2d\theta ∥ J ∥ s Θ , 2 2 := ∣ β ∣ ≤ s Θ ∑ ∫ Θ D θ β ∫ X p ( x ; θ ) g ( x , θ ) d x 2 d θ = ( i ) ∑ ∣ β ∣ ≤ s Θ ∫ Θ ∣ ∫ X D θ β ( p ( x ; θ ) g ( x , θ ) ) d x ∣ 2 d θ ≤ Vol ( X ) ∫ X ∑ ∣ β ∣ ≤ s Θ ∫ Θ ∣ D θ β ( p ( x ; θ ) g ( x , θ ) ) ∣ 2 d θ d x \begin{aligned}
\mathrel{\overset{(i)}{=}}\sum_{|\beta|\leq s_{\Theta}}\int_{\Theta}\left|\int_{\mathcal{X}}D_{\theta}^{\beta}\left(p(x;\theta)g(x,\theta)\right)dx\right|^2d\theta\\
&\leq\operatorname{Vol}(\mathcal{X})\int_{\mathcal{X}}\sum_{|\beta|\leq s_{\Theta}}\int_{\Theta}\left|D_{\theta}^{\beta}\left(p(x;\theta)g(x,\theta)\right)\right|^2d\theta dx
\end{aligned} = ( i ) ∣ β ∣ ≤ s Θ ∑ ∫ Θ ∫ X D θ β ( p ( x ; θ ) g ( x , θ ) ) d x 2 d θ ≤ Vol ( X ) ∫ X ∣ β ∣ ≤ s Θ ∑ ∫ Θ D θ β ( p ( x ; θ ) g ( x , θ ) ) 2 d θ d x = ( i i i ) Vol ( X ) ∫ X ∥ p ( x ; ⋅ ) g ( x , ⋅ ) ∥ s Θ , 2 2 d x ≤ Vol ( X ) 2 S 2 2 S 3 2 (62) \begin{aligned}
\mathrel{\overset{(iii)}{=}}\operatorname{Vol}(\mathcal{X})\int_{\mathcal{X}}\|p(x;\cdot)g(x,\cdot)\|_{s_{\Theta},2}^2dx \\
&\leq\operatorname{Vol}(\mathcal{X})^2S_2^2S_3^2 \tag*{(62)}
\end{aligned} = ( iii ) Vol ( X ) ∫ X ∥ p ( x ; ⋅ ) g ( x , ⋅ ) ∥ s Θ , 2 2 d x ≤ Vol ( X ) 2 S 2 2 S 3 2 (62) Here, ( i ) (i) ( i ) ( i i ) (ii) ( ii ) ∣ ∫ X f ( x ) d x ∣ 2 ≤ Vol ( X ) ∫ X ∣ f ( x ) ∣ 2 d x \left|\int_{\mathcal{X}} f(x)\mathrm{d}x\right|^2 \le\operatorname{Vol}(\mathcal{X})\int_{\mathcal{X}} |f(x)|^2\mathrm{d}x ∫ X f ( x ) d x 2 ≤ Vol ( X ) ∫ X ∣ f ( x ) ∣ 2 d x X \mathcal{X} X ( i i i ) (iii) ( iii ) ( i v ) (iv) ( i v ) ∥ g ( x , ⋅ ) ∥ s Θ , 2 ≤ S 2 \lVert g(x,\cdot)\rVert_{s_{\Theta},2} \le S_2 ∥ g ( x , ⋅ ) ∥ s Θ , 2 ≤ S 2 ∥ p ( x ; ⋅ ) ∥ s Θ , 2 ≤ S 3 \lVert p(x;\cdot)\rVert_{s_{\Theta},2} \le S_3 ∥ p ( x ; ⋅ ) ∥ s Θ , 2 ≤ S 3
Step Three. In this step, we study the Sobolev norm of J ^ K Q \hat{J}_{\mathrm{KQ}} J ^ KQ x 1 : N ( θ ) x_{1:N}^{(\theta)} x 1 : N ( θ ) ∥ J − J ^ K Q ( ⋅ ; x 1 : N ( θ ) ) ∥ s Θ , 2 + ∥ J ∥ s Θ , 2 \lVert J-\hat{J}_{\mathrm{KQ}}(\cdot;x_{1:N}^{(\theta)})\rVert_{s_{\Theta},2}+\lVert J\rVert_{s_{\Theta},2} ∥ J − J ^ KQ ( ⋅ ; x 1 : N ( θ ) ) ∥ s Θ , 2 + ∥ J ∥ s Θ , 2 g ( x 1 : N ( θ ) , ⋅ ) ∈ W 2 s Θ ( Θ ) g(x_{1:N}^{(\theta)},\cdot)\in W_{2}^{s_{\Theta}}(\Theta) g ( x 1 : N ( θ ) , ⋅ ) ∈ W 2 s Θ ( Θ )
g ^ ( x , ⋅ ; x 1 : N ( θ ) ) = k X ( x , x 1 : N ( θ ) ) ( k X ( x 1 : N ( θ ) , x 1 : N ( θ ) ) + N λ X I D N ) − 1 × g ( x 1 : N ( θ ) , ⋅ ) \hat{g}(x,\cdot;x_{1:N}^{(\theta)})=k_{\mathcal{X}}(x,x_{1:N}^{(\theta)})\left(k_{\mathcal{X}}(x_{1:N}^{(\theta)},x_{1:N}^{(\theta)})+N\lambda_{\mathcal{X}}I_{D_N}\right)^{-1}
\times g(x_{1:N}^{(\theta)},\cdot) g ^ ( x , ⋅ ; x 1 : N ( θ ) ) = k X ( x , x 1 : N ( θ ) ) ( k X ( x 1 : N ( θ ) , x 1 : N ( θ ) ) + N λ X I D N ) − 1 × g ( x 1 : N ( θ ) , ⋅ ) is in W 2 s Θ ( Θ ) W_{2}^{s_{\Theta}}(\Theta) W 2 s Θ ( Θ ) x 1 : N ( θ ) x_{1:N}^{(\theta)} x 1 : N ( θ )
∥ p ( x ; ⋅ ) ( g ( x , ⋅ ) − g ^ ( x , ⋅ ; x 1 : N ( θ ) ) ) ∥ s Θ , 2 ≤ ∥ p ( x ; ⋅ ) ∥ s Θ , 2 ∥ g ( x , ⋅ ) − g ^ ( x , ⋅ ; x 1 : N ( θ ) ) ∥ s Θ , 2 ≤ S 3 ∥ g ( x , ⋅ ) − g ^ ( x , ⋅ ; x 1 : N ( θ ) ) ∥ s Θ , 2 , (63) \left\lVert p(x;\cdot)\left(g(x,\cdot)-\hat{g}(x,\cdot;x_{1:N}^{(\theta)})\right)\right\rVert_{s_{\Theta},2}
\le\lVert p(x;\cdot)\rVert_{s_{\Theta},2}\left\lVert g(x,\cdot)-\hat{g}(x,\cdot;x_{1:N}^{(\theta)})\right\rVert_{s_{\Theta},2}
\le S_3\left\lVert g(x,\cdot)-\hat{g}(x,\cdot;x_{1:N}^{(\theta)})\right\rVert_{s_{\Theta},2}, \tag*{(63)} p ( x ; ⋅ ) ( g ( x , ⋅ ) − g ^ ( x , ⋅ ; x 1 : N ( θ ) ) ) s Θ , 2 ≤ ∥ p ( x ; ⋅ ) ∥ s Θ , 2 g ( x , ⋅ ) − g ^ ( x , ⋅ ; x 1 : N ( θ ) ) s Θ , 2 ≤ S 3 g ( x , ⋅ ) − g ^ ( x , ⋅ ; x 1 : N ( θ ) ) s Θ , 2 , (63) where the first inequality holds by Theorem A.11 and the second inequality holds by Assumption (3) that ∥ p ( x ; ⋅ ) ∥ s Θ , 2 ≤ S 3 \lVert p(x;\cdot)\rVert_{s_{\Theta},2}\le S_3 ∥ p ( x ; ⋅ ) ∥ s Θ , 2 ≤ S 3 r ( x , θ ) : = g ( x , θ ) − g ^ ( x , θ ; x 1 : N ( θ ) ) r(x,\theta):=g(x,\theta)-\hat{g}(x,\theta;x_{1:N}^{(\theta)}) r ( x , θ ) := g ( x , θ ) − g ^ ( x , θ ; x 1 : N ( θ ) ) J ^ K Q − J \hat{J}_{\mathrm{KQ}}-J J ^ KQ − J
∥ J ^ K Q ( ⋅ ; x 1 : N ( θ ) ) − J ∥ s Θ , 2 2 = ∑ ∣ β ∣ ≤ s Θ ∥ ∫ X D θ β ( p ( x ; ⋅ ) r ( x , ⋅ ) ) d x ∥ L 2 ( Θ ) 2 = ( i ) ∑ ∣ β ∣ ≤ s Θ ∥ ∫ X D θ β ( p ( x ; ⋅ ) r ( x , ⋅ ) ) d x ∥ L 2 ( Θ ) 2 ≤ ( i i ) Vol ( X ) ∫ X ∑ ∣ β ∣ ≤ s Θ ∫ Θ ∣ D θ β ( p ( x ; θ ) r ( x , θ ) ) ∣ 2 d θ d x = ( i i i ) Vol ( X ) ∫ X ∥ p ( x ; ⋅ ) r ( x , ⋅ ) ∥ s Θ , 2 2 d x ≤ ( i v ) Vol ( X ) S 3 2 ∫ X ∥ r ( x , ⋅ ) ∥ s Θ , 2 2 d x , (64) \begin{aligned}
\left\lVert\hat{J}_{\mathrm{KQ}}(\cdot;x_{1:N}^{(\theta)})-J\right\rVert_{s_{\Theta},2}^{2}
&=\sum_{|\beta|\le s_{\Theta}}\left\lVert\int_{\mathcal{X}}D_{\theta}^{\beta}\bigl(p(x;\cdot)r(x,\cdot)\bigr)\mathrm{d}x\right\rVert_{L_{2}(\Theta)}^{2}\\
&\mathrel{\overset{(i)}{=}}\sum_{|\beta|\le s_{\Theta}}\left\lVert\int_{\mathcal{X}}D_{\theta}^{\beta}\bigl(p(x;\cdot)r(x,\cdot)\bigr)\mathrm{d}x\right\rVert_{L_{2}(\Theta)}^{2}\\
&\mathrel{\overset{(ii)}{\le}}\operatorname{Vol}(\mathcal{X})\int_{\mathcal{X}}\sum_{|\beta|\le s_{\Theta}}\int_{\Theta}\left|D_{\theta}^{\beta}\bigl(p(x;\theta)r(x,\theta)\bigr)\right|^{2}\mathrm{d}\theta\mathrm{d}x\\
&\mathrel{\overset{(iii)}{=}}\operatorname{Vol}(\mathcal{X})\int_{\mathcal{X}}\left\lVert p(x;\cdot)r(x,\cdot)\right\rVert_{s_{\Theta},2}^{2}\mathrm{d}x\\
&\mathrel{\overset{(iv)}{\le}}\operatorname{Vol}(\mathcal{X})S_{3}^{2}\int_{\mathcal{X}}\left\lVert r(x,\cdot)\right\rVert_{s_{\Theta},2}^{2}\mathrm{d}x, \tag*{(64)}
\end{aligned} J ^ KQ ( ⋅ ; x 1 : N ( θ ) ) − J s Θ , 2 2 = ∣ β ∣ ≤ s Θ ∑ ∫ X D θ β ( p ( x ; ⋅ ) r ( x , ⋅ ) ) d x L 2 ( Θ ) 2 = ( i ) ∣ β ∣ ≤ s Θ ∑ ∫ X D θ β ( p ( x ; ⋅ ) r ( x , ⋅ ) ) d x L 2 ( Θ ) 2 ≤ ( ii ) Vol ( X ) ∫ X ∣ β ∣ ≤ s Θ ∑ ∫ Θ D θ β ( p ( x ; θ ) r ( x , θ ) ) 2 d θ d x = ( iii ) Vol ( X ) ∫ X ∥ p ( x ; ⋅ ) r ( x , ⋅ ) ∥ s Θ , 2 2 d x ≤ ( i v ) Vol ( X ) S 3 2 ∫ X ∥ r ( x , ⋅ ) ∥ s Θ , 2 2 d x , (64) where the above chain of derivations ( i ) (i) ( i ) ( i v ) (iv) ( i v )
∫ X ∥ r ( x , ⋅ ) ∥ s Θ , 2 2 d x = ∑ ∣ β ∣ ≤ s Θ ∫ Θ ∫ X ∣ D θ β r ( x , θ ) ∣ 2 d x d θ = ∑ ∣ β ∣ ≤ s Θ ∥ D θ β r ∥ L 2 ( X × Θ ) 2 . (65) \int_{\mathcal{X}}\|r(x,\cdot)\|_{s_{\Theta},2}^{2}\,dx
= \sum_{|\beta|\leq s_{\Theta}}\int_{\Theta}\int_{\mathcal{X}}\left|D_{\theta}^{\beta}r(x,\theta)\right|^{2}\,dx\,d\theta
= \sum_{|\beta|\leq s_{\Theta}}\left\|D_{\theta}^{\beta}r\right\|_{L_{2}(\mathcal{X}\times\Theta)}^{2}. \tag*{(65)} ∫ X ∥ r ( x , ⋅ ) ∥ s Θ , 2 2 d x = ∣ β ∣ ≤ s Θ ∑ ∫ Θ ∫ X D θ β r ( x , θ ) 2 d x d θ = ∣ β ∣ ≤ s Θ ∑ D θ β r L 2 ( X × Θ ) 2 . (65) By Assumption (2), D θ β g ( ⋅ , θ ) ∈ W 2 s X ( X ) D_{\theta}^{\beta}g(\cdot,\theta)\in W_{2}^{s_{\mathcal{X}}}(\mathcal{X}) D θ β g ( ⋅ , θ ) ∈ W 2 s X ( X ) ∣ β ∣ ≤ s Θ |\beta|\leq s_{\Theta} ∣ β ∣ ≤ s Θ h ∗ ( ⋅ ) : = D θ β g ( ⋅ , θ ) h^{*}(\cdot):=D_{\theta}^{\beta}g(\cdot,\theta) h ∗ ( ⋅ ) := D θ β g ( ⋅ , θ ) h ^ λ ( ⋅ ) : = D θ β g ^ ( ⋅ , θ ; x 1 : N ( θ ) ) = k X ( ⋅ , x 1 : N ( θ ) ) ( k X ( x 1 : N ( θ ) ; x 1 : N ( θ ) ) + N λ X I d N ) − 1 D θ β g ( x 1 : N ( θ ) , θ ; x 1 : N ( θ ) ) \hat{h}_{\lambda}(\cdot):=D_{\theta}^{\beta}\hat{g}(\cdot,\theta;x_{1:N}^{(\theta)})=k_{\mathcal{X}}(\cdot,x_{1:N}^{(\theta)})(k_{\mathcal{X}}(x_{1:N}^{(\theta)};x_{1:N}^{(\theta)})+N\lambda_{\mathcal{X}}I_{d_N})^{-1}D_{\theta}^{\beta}g(x_{1:N}^{(\theta)},\theta;x_{1:N}^{(\theta)}) h ^ λ ( ⋅ ) := D θ β g ^ ( ⋅ , θ ; x 1 : N ( θ ) ) = k X ( ⋅ , x 1 : N ( θ ) ) ( k X ( x 1 : N ( θ ) ; x 1 : N ( θ ) ) + N λ X I d N ) − 1 D θ β g ( x 1 : N ( θ ) , θ ; x 1 : N ( θ ) )
∥ D θ β g ( ⋅ , θ ) − D θ β g ^ ( ⋅ , θ ; x 1 : N ( θ ) ) ∥ L 2 ( P θ ) ≤ E τ N − s X d X ( log N ) s X + 1 d X ∥ D θ β g ( ⋅ , θ ) ∥ s X , 2 (66) \left\|D_{\theta}^{\beta}g(\cdot,\theta)-D_{\theta}^{\beta}\hat{g}(\cdot,\theta;x_{1:N}^{(\theta)})\right\|_{L_{2}(\mathbb{P}_{\theta})}
\leq\mathcal{E}_{\tau}N^{-\frac{s_{\mathcal{X}}}{d_{\mathcal{X}}}}(\log N)^{\frac{s_{\mathcal{X}}+1}{d_{\mathcal{X}}}}\left\|D_{\theta}^{\beta}g(\cdot,\theta)\right\|_{s_{\mathcal{X}},2} \tag*{(66)} D θ β g ( ⋅ , θ ) − D θ β g ^ ( ⋅ , θ ; x 1 : N ( θ ) ) L 2 ( P θ ) ≤ E τ N − d X s X ( log N ) d X s X + 1 D θ β g ( ⋅ , θ ) s X , 2 (66) holds with probability at least 1 − 4 e − τ 1-4e^{-\tau} 1 − 4 e − τ C \mathcal{C} C X \mathcal{X} X G 0 , X G_{0,\mathcal{X}} G 0 , X G 1 , X G_{1,\mathcal{X}} G 1 , X L 2 ( P θ ) ≅ L 2 ( X ) L_{2}(\mathbb{P}_{\theta})\cong L_{2}(\mathcal{X}) L 2 ( P θ ) ≅ L 2 ( X ) ∥ f ∥ L 2 ( X ) ≤ Vol ( X ) − 1 G 0 , X − 1 ∥ f ∥ L 2 ( P θ ) \|f\|_{L_{2}(\mathcal{X})}\leq\operatorname{Vol}(\mathcal{X})^{-1}G_{0,\mathcal{X}}^{-1}\|f\|_{L_{2}(\mathbb{P}_{\theta})} ∥ f ∥ L 2 ( X ) ≤ Vol ( X ) − 1 G 0 , X − 1 ∥ f ∥ L 2 ( P θ ) f ∈ L 2 ( P θ ) f\in L_{2}(\mathbb{P}_{\theta}) f ∈ L 2 ( P θ ) θ ∈ Θ \theta\in\Theta θ ∈ Θ ∣ β ∣ ≤ s Θ |\beta|\leq s_{\Theta} ∣ β ∣ ≤ s Θ 1 − 4 e − τ 1-4e^{-\tau} 1 − 4 e − τ
∥ D θ β g ( ⋅ , θ ) − D θ β g ^ ( ⋅ , θ ; x 1 : N ( θ ) ) ∥ L 2 ( X ) ≤ E τ Vol ( X ) − 1 G 0 , X − 1 N − s X d X × ( log N ) s X + 1 d X ∥ D θ β g ( ⋅ , θ ) ∥ s X , 2 ≤ E τ Vol ( X ) − 1 G 0 , X − 1 N − s X d X ( log N ) s X + 1 d X S 1 . (67) \begin{aligned}
\left\|D_{\theta}^{\beta}g(\cdot,\theta)-D_{\theta}^{\beta}\hat{g}(\cdot,\theta;x_{1:N}^{(\theta)})\right\|_{L_{2}(\mathcal{X})}
&\leq\mathcal{E}_{\tau}\operatorname{Vol}(\mathcal{X})^{-1}G_{0,\mathcal{X}}^{-1}N^{-\frac{s_{\mathcal{X}}}{d_{\mathcal{X}}}} \\
&\quad\times(\log N)^{\frac{s_{\mathcal{X}}+1}{d_{\mathcal{X}}}}\left\|D_{\theta}^{\beta}g(\cdot,\theta)\right\|_{s_{\mathcal{X}},2} \\
&\leq\mathcal{E}_{\tau}\operatorname{Vol}(\mathcal{X})^{-1}G_{0,\mathcal{X}}^{-1}N^{-\frac{s_{\mathcal{X}}}{d_{\mathcal{X}}}}(\log N)^{\frac{s_{\mathcal{X}}+1}{d_{\mathcal{X}}}}S_{1}. \tag*{(67)}
\end{aligned} D θ β g ( ⋅ , θ ) − D θ β g ^ ( ⋅ , θ ; x 1 : N ( θ ) ) L 2 ( X ) ≤ E τ Vol ( X ) − 1 G 0 , X − 1 N − d X s X × ( log N ) d X s X + 1 D θ β g ( ⋅ , θ ) s X , 2 ≤ E τ Vol ( X ) − 1 G 0 , X − 1 N − d X s X ( log N ) d X s X + 1 S 1 . (67) By plugging (A.26) into (A.24), and then plugging the result into (A.23), we get that with probability at least 1 − 4 e − τ 1-4e^{-\tau} 1 − 4 e − τ
∥ J ^ K Q ( ⋅ ; x 1 : N ( θ ) ) − J ∥ s Θ , 2 ≤ ( ∑ ∣ β ∣ ≤ s Θ 1 ) E τ G 0 , X − 1 Vol ( X ) − 1 S 1 S 3 × N − s X d X ( log N ) s X + 1 d X = ( s Θ + d Θ − 1 d Θ − 1 ) E τ G 0 , X − 1 Vol ( X ) − 1 S 1 S 3 × N − s X d X ( log N ) s X + 1 d X . (68) \begin{aligned}
\left\|\hat{J}_{KQ}(\cdot;x_{1:N}^{(\theta)})-J\right\|_{s_{\Theta},2}
&\leq\left(\sum_{|\beta|\leq s_{\Theta}}1\right)\mathcal{E}_{\tau}G_{0,\mathcal{X}}^{-1}\operatorname{Vol}(\mathcal{X})^{-1}S_{1}S_{3} \\
&\quad\times N^{-\frac{s_{\mathcal{X}}}{d_{\mathcal{X}}}}(\log N)^{\frac{s_{\mathcal{X}}+1}{d_{\mathcal{X}}}} \\
&= \binom{s_{\Theta}+d_{\Theta}-1}{d_{\Theta}-1}\mathcal{E}_{\tau}G_{0,\mathcal{X}}^{-1}\operatorname{Vol}(\mathcal{X})^{-1}S_{1}S_{3} \\
&\quad\times N^{-\frac{s_{\mathcal{X}}}{d_{\mathcal{X}}}}(\log N)^{\frac{s_{\mathcal{X}}+1}{d_{\mathcal{X}}}}. \tag*{(68)}
\end{aligned} J ^ K Q ( ⋅ ; x 1 : N ( θ ) ) − J s Θ , 2 ≤ ∣ β ∣ ≤ s Θ ∑ 1 E τ G 0 , X − 1 Vol ( X ) − 1 S 1 S 3 × N − d X s X ( log N ) d X s X + 1 = ( d Θ − 1 s Θ + d Θ − 1 ) E τ G 0 , X − 1 Vol ( X ) − 1 S 1 S 3 × N − d X s X ( log N ) d X s X + 1 . (68) By combining this result with the bound ∥ J ∥ s Θ , 2 ≤ Vol ( X ) S 2 S 3 \lVert J\rVert_{s_\Theta,2} \leq\operatorname{Vol}(\mathcal{X})S_2S_3 ∥ J ∥ s Θ , 2 ≤ Vol ( X ) S 2 S 3 1 − 4 e − τ 1-4e^{-\tau} 1 − 4 e − τ N > N 0 N>N_0 N > N 0
∥ J ^ K Q ( ⋅ ; x 1 : N ( θ ) ) ∥ s Θ , 2 ≤ ∥ J ^ K Q ( ⋅ ; x 1 : N ( θ ) ) − J ∥ s Θ , 2 + ∥ J ∥ s Θ , 2 ≤ ( s Θ + d Θ − 1 d Θ − 1 ) C τ G 0 , X − 1 Vol ( X ) − 1 S 1 S 3 × N − s X d X ( log N ) s X + 1 d X + Vol ( X ) S 2 S 3 ≤ 2 Vol ( X ) S 2 S 3 . (69) \begin{aligned}
\left\lVert\widehat{J}_{\mathrm{KQ}}(\cdot;x_{1:N}^{(\theta)})\right\rVert_{s_\Theta,2}
&\leq\left\lVert\widehat{J}_{\mathrm{KQ}}(\cdot;x_{1:N}^{(\theta)})-J\right\rVert_{s_\Theta,2}+\left\lVert J\right\rVert_{s_\Theta,2}\\
&\leq\binom{s_\Theta+d_\Theta-1}{d_\Theta-1}C\tau G_{0,\mathcal{X}}^{-1}\operatorname{Vol}(\mathcal{X})^{-1}S_1S_3\\
&\quad\times N^{-\frac{s_\mathcal{X}}{d_\mathcal{X}}}(\log N)^{\frac{s_\mathcal{X}+1}{d_\mathcal{X}}}+\operatorname{Vol}(\mathcal{X})S_2S_3\\
&\leq2\operatorname{Vol}(\mathcal{X})S_2S_3. \tag*{(69)}
\end{aligned} J KQ ( ⋅ ; x 1 : N ( θ ) ) s Θ , 2 ≤ J KQ ( ⋅ ; x 1 : N ( θ ) ) − J s Θ , 2 + ∥ J ∥ s Θ , 2 ≤ ( d Θ − 1 s Θ + d Θ − 1 ) C τ G 0 , X − 1 Vol ( X ) − 1 S 1 S 3 × N − d X s X ( log N ) d X s X + 1 + Vol ( X ) S 2 S 3 ≤ 2 Vol ( X ) S 2 S 3 . (69) where N 0 N_0 N 0
Step Four. In this step, we are going to upper bound the Sobolev norm of F ˉ K Q \bar{F}_{\mathrm{KQ}} F ˉ KQ F ^ K Q = f ∘ J ^ K Q \widehat{F}_{\mathrm{KQ}}=f\circ\widehat{J}_{\mathrm{KQ}} F KQ = f ∘ J KQ W 2 s Θ ( Θ ) W_2^{s_\Theta}(\Theta) W 2 s Θ ( Θ ) f f f s Θ + 1 s_\Theta+1 s Θ + 1 ∥ J ^ ( ⋅ ; x 1 : N ( θ ) ) ∥ s Θ , 2 ≤ Vol ( X ) S 2 S 3 \left\lVert\widehat{J}(\cdot;x_{1:N}^{(\theta)})\right\rVert_{s_\Theta,2}\leq\operatorname{Vol}(\mathcal{X})S_2S_3 J ( ⋅ ; x 1 : N ( θ ) ) s Θ , 2 ≤ Vol ( X ) S 2 S 3 1 − 4 e − τ 1-4e^{-\tau} 1 − 4 e − τ ∥ F ^ K Q ( ⋅ ; x 1 : N ( θ ) ) ∥ s Θ , 2 ≤ C 6 \left\lVert\widehat{F}_{\mathrm{KQ}}(\cdot;x_{1:N}^{(\theta)})\right\rVert_{s_\Theta,2}\leq C_6 F KQ ( ⋅ ; x 1 : N ( θ ) ) s Θ , 2 ≤ C 6 1 − 4 e − τ 1-4e^{-\tau} 1 − 4 e − τ F ˉ K Q ( θ ) \bar{F}_{\mathrm{KQ}}(\theta) F ˉ KQ ( θ )
F ˉ K Q ( θ ) = E x 1 : N ( θ ) ∼ P θ [ F ^ K Q ( θ ; x 1 : N ( θ ) ) ] = ∫ X ⋯ ∫ X F ^ K Q ( θ ; x 1 : N ( θ ) ) ∏ n = 1 N p ( x n ( θ ) , θ ) d x 1 ( θ ) ⋯ d x N ( θ ) . \begin{aligned}
\bar{F}_{\mathrm{KQ}}(\theta)=\mathbb{E}_{x_{1:N}^{(\theta)}\sim P_\theta}\left[\widehat{F}_{\mathrm{KQ}}(\theta;x_{1:N}^{(\theta)})\right] \\
&=\int_{\mathcal{X}}\cdots\int_{\mathcal{X}}\widehat{F}_{\mathrm{KQ}}(\theta;x_{1:N}^{(\theta)})\prod_{n=1}^{N}p(x_n^{(\theta)},\theta)\,\mathrm{d}x_1^{(\theta)}\cdots\mathrm{d}x_N^{(\theta)}.
\end{aligned} F ˉ KQ ( θ ) = E x 1 : N ( θ ) ∼ P θ [ F KQ ( θ ; x 1 : N ( θ ) ) ] = ∫ X ⋯ ∫ X F KQ ( θ ; x 1 : N ( θ ) ) n = 1 ∏ N p ( x n ( θ ) , θ ) d x 1 ( θ ) ⋯ d x N ( θ ) . For any i = 1 , … , N i=1,\ldots,N i = 1 , … , N ∥ p ( x i ( θ ) , ⋅ ) ∥ s Θ , 2 ≤ S 3 \left\lVert p(x_i^{(\theta)},\cdot)\right\rVert_{s_\Theta,2}\leq S_3 p ( x i ( θ ) , ⋅ ) s Θ , 2 ≤ S 3 ∥ F ^ K Q ( ⋅ ; x 1 : N ( θ ) ) ∥ s Θ , 2 ≤ C 6 \left\lVert\widehat{F}_{\mathrm{KQ}}(\cdot;x_{1:N}^{(\theta)})\right\rVert_{s_\Theta,2}\leq C_6 F KQ ( ⋅ ; x 1 : N ( θ ) ) s Θ , 2 ≤ C 6 ∥ p ( x i ( θ ) , ⋅ ) F ^ K Q ( ⋅ ; x 1 : N ( θ ) ) ∥ s Θ , 2 \left\lVert p(x_i^{(\theta)},\cdot)\widehat{F}_{\mathrm{KQ}}(\cdot;x_{1:N}^{(\theta)})\right\rVert_{s_\Theta,2} p ( x i ( θ ) , ⋅ ) F KQ ( ⋅ ; x 1 : N ( θ ) ) s Θ , 2 x i ( θ ) ↦ p ( x i ( θ ) , ⋅ ) F ^ K Q ( ⋅ ; x 1 : N ( θ ) ) x_i^{(\theta)}\mapsto p(x_i^{(\theta)},\cdot)\widehat{F}_{\mathrm{KQ}}(\cdot;x_{1:N}^{(\theta)}) x i ( θ ) ↦ p ( x i ( θ ) , ⋅ ) F KQ ( ⋅ ; x 1 : N ( θ ) ) L X \mathcal{L}_{\mathcal{X}} L X
∥ F ˉ K Q ∥ s Θ , 2 ≤ ∫ X ⋯ ∫ X ∥ F ^ K Q ( ⋅ ; x 1 : N ( θ ) ) ∏ n = 1 N p ( x n ( θ ) , ⋅ ) ∥ s Θ , 2 d x 1 : N ( θ ) \left\lVert\bar{F}_{\mathrm{KQ}}\right\rVert_{s_\Theta,2}\leq\int_{\mathcal{X}}\cdots\int_{\mathcal{X}}\left\lVert\widehat{F}_{\mathrm{KQ}}(\cdot;x_{1:N}^{(\theta)})\prod_{n=1}^{N}p(x_n^{(\theta)},\cdot)\right\rVert_{s_\Theta,2}\,\mathrm{d}x_{1:N}^{(\theta)} F ˉ KQ s Θ , 2 ≤ ∫ X ⋯ ∫ X F KQ ( ⋅ ; x 1 : N ( θ ) ) n = 1 ∏ N p ( x n ( θ ) , ⋅ ) s Θ , 2 d x 1 : N ( θ ) ≤ ∫ X ⋯ ∫ X ∥ F ^ K Q ( ⋅ ; x 1 : N ( θ ) ) ∥ S θ , 2 ( ∏ n = 1 N ∥ p ( x n ( θ ) ; ⋅ ) ∥ S θ , 2 ) d x 1 : N ( θ ) ≤ C 6 S 3 N Vol ( X ) N ≤ C 6 . (70) \begin{aligned}
&\le\int_{\mathcal{X}}\cdots\int_{\mathcal{X}}\left\|\widehat{F}_{\mathrm{KQ}}(\cdot;x_{1:N}^{(\theta)})\right\|_{\mathcal{S}_\theta,2}\left(\prod_{n=1}^{N}\left\|p(x_n^{(\theta)};\cdot)\right\|_{\mathcal{S}_\theta,2}\right) d x_{1:N}^{(\theta)}\\
&\le C_6 S_3^N\operatorname{Vol}(\mathcal{X})^N\\
&\le C_6. \tag*{(70)}
\end{aligned} ≤ ∫ X ⋯ ∫ X F KQ ( ⋅ ; x 1 : N ( θ ) ) S θ , 2 ( n = 1 ∏ N p ( x n ( θ ) ; ⋅ ) S θ , 2 ) d x 1 : N ( θ ) ≤ C 6 S 3 N Vol ( X ) N ≤ C 6 . (70) The last inequality holds by S 3 ≤ 1 S_3 \le1 S 3 ≤ 1 X = [ 0 , 1 ] d x \mathcal{X}=[0,1]^{d_x} X = [ 0 , 1 ] d x Vol ( X ) = 1 \operatorname{Vol}(\mathcal{X})=1 Vol ( X ) = 1
Step Five. We are now ready to upper bound the stage II error, which was defined as
Stage II error = ∥ F ‾ K Q ( ⋅ ) − k ( ⋅ , θ 1 : T ) × ( k Θ ( θ 1 : T ; θ 1 : T ) + λ Θ I d T ) − 1 F ^ K Q ( θ 1 : T ) ∥ L 2 ( Q ) . \operatorname{Stage\ II\ error}=
\left\|
\begin{aligned}
&\overline{F}_{\mathrm{KQ}}(\cdot)-k(\cdot,\theta_{1:T})\\
&\quad\times\left(k_{\Theta}(\theta_{1:T};\theta_{1:T})+\lambda_{\Theta}\mathcal{I}_{d_T}\right)^{-1}\widehat{F}_{\mathrm{KQ}}(\theta_{1:T})
\end{aligned}
\right\|_{L_2(\mathbb{Q})}. Stage II error = F KQ ( ⋅ ) − k ( ⋅ , θ 1 : T ) × ( k Θ ( θ 1 : T ; θ 1 : T ) + λ Θ I d T ) − 1 F KQ ( θ 1 : T ) L 2 ( Q ) . The idea is to treat the stage II error as the generalization error of kernel ridge regression—which can be bounded via Theorem A.3. Given i.i.d. observations ( θ 1 , F ^ K Q ( θ 1 ; x 1 : N ( θ 1 ) ) ) , … , ( θ T , F ^ K Q ( θ T ; x 1 : N ( θ T ) ) ) (\theta_1,\widehat{F}_{\mathrm{KQ}}(\theta_1;x_{1:N}^{(\theta_1)})),\ldots,(\theta_T,\widehat{F}_{\mathrm{KQ}}(\theta_T;x_{1:N}^{(\theta_T)})) ( θ 1 , F KQ ( θ 1 ; x 1 : N ( θ 1 ) )) , … , ( θ T , F KQ ( θ T ; x 1 : N ( θ T ) )) F ‾ K Q ( θ ) = E x 1 : N ( θ ) ∼ P θ F ^ K Q ( θ ; x 1 : N ( θ ) ) \overline{F}_{\mathrm{KQ}}(\theta)=\mathbb{E}_{x_{1:N}^{(\theta)}\sim\mathbb{P}_{\theta}}\widehat{F}_{\mathrm{KQ}}(\theta;x_{1:N}^{(\theta)}) F KQ ( θ ) = E x 1 : N ( θ ) ∼ P θ F KQ ( θ ; x 1 : N ( θ ) ) F ^ K Q ( θ ; x 1 : N ( θ ) ) \widehat{F}_{\mathrm{KQ}}(\theta;x_{1:N}^{(\theta)}) F KQ ( θ ; x 1 : N ( θ ) ) F ‾ K Q ( θ ) \overline{F}_{\mathrm{KQ}}(\theta) F KQ ( θ ) r : Θ → R r:\Theta\to\mathbb{R} r : Θ → R r ( θ ; x 1 : N ( θ ) ) = F ^ K Q ( θ ; x 1 : N ( θ ) ) − F ‾ K Q ( θ ) r(\theta;x_{1:N}^{(\theta)})=\widehat{F}_{\mathrm{KQ}}(\theta;x_{1:N}^{(\theta)})-\overline{F}_{\mathrm{KQ}}(\theta) r ( θ ; x 1 : N ( θ ) ) = F KQ ( θ ; x 1 : N ( θ ) ) − F KQ ( θ ) E x 1 : N ( θ ) ∼ P θ [ r ( θ ) ] = 0 \mathbb{E}_{x_{1:N}^{(\theta)}\sim\mathbb{P}_{\theta}}[r(\theta)]=0 E x 1 : N ( θ ) ∼ P θ [ r ( θ )] = 0 m ≥ 2 m\ge2 m ≥ 2
E x 1 : N ( θ ) ∼ P θ [ ∣ r ( θ ) ∣ m ] = E x 1 : N ( θ ) ∼ P θ [ ∣ F ^ K Q ( θ ; x 1 : N ( θ ) ) − F ‾ K Q ( θ ) ∣ m ] ≤ ( i ) 2 m − 1 E x 1 : N ( θ ) ∼ P θ [ ∣ F ^ K Q ( θ ; x 1 : N ( θ ) ) − F ( θ ) ∣ m ] + 2 m − 1 ∣ F ‾ K Q ( θ ) − F ( θ ) ∣ m = 2 m − 1 E x 1 : N ( θ ) ∼ P θ [ ∣ F ^ K Q ( θ ; x 1 : N ( θ ) ) − F ( θ ) ∣ m ] + 2 m − 1 ∣ E x 1 : N ( θ ) ∼ P θ [ F ^ K Q ( θ ; x 1 : N ( θ ) ) − F ( θ ) ] ∣ m ≤ 2 m − 1 E x 1 : N ( θ ) ∼ P θ [ ∣ F ^ K Q ( θ ; x 1 : N ( θ ) ) − F ( θ ) ∣ m ] + 2 m − 1 E x 1 : N ( θ ) ∼ P θ [ ∣ F ^ K Q ( θ ; x 1 : N ( θ ) ) − F ( θ ) ∣ m ] = 2 m E x 1 : N ( θ ) ∼ P θ [ ∣ F ^ K Q ( θ ; x 1 : N ( θ ) ) − F ( θ ) ∣ m ] \begin{aligned}
\mathbb{E}_{x_{1:N}^{(\theta)}\sim\mathbb{P}_{\theta}}\left[|r(\theta)|^m\right]
&=\mathbb{E}_{x_{1:N}^{(\theta)}\sim\mathbb{P}_{\theta}}\left[\left|\widehat{F}_{\mathrm{KQ}}(\theta;x_{1:N}^{(\theta)})-\overline{F}_{\mathrm{KQ}}(\theta)\right|^m\right]\\
&\mathrel{\overset{(i)}{\le}}2^{m-1}\mathbb{E}_{x_{1:N}^{(\theta)}\sim\mathbb{P}_{\theta}}\left[\left|\widehat{F}_{\mathrm{KQ}}(\theta;x_{1:N}^{(\theta)})-F(\theta)\right|^m\right]\\
&\quad+2^{m-1}\left|\overline{F}_{\mathrm{KQ}}(\theta)-F(\theta)\right|^m\\
&=2^{m-1}\mathbb{E}_{x_{1:N}^{(\theta)}\sim\mathbb{P}_{\theta}}\left[\left|\widehat{F}_{\mathrm{KQ}}(\theta;x_{1:N}^{(\theta)})-F(\theta)\right|^m\right]\\
&\quad+2^{m-1}\left|\mathbb{E}_{x_{1:N}^{(\theta)}\sim\mathbb{P}_{\theta}}\left[\widehat{F}_{\mathrm{KQ}}(\theta;x_{1:N}^{(\theta)})-F(\theta)\right]\right|^m\\
&\le2^{m-1}\mathbb{E}_{x_{1:N}^{(\theta)}\sim\mathbb{P}_{\theta}}\left[\left|\widehat{F}_{\mathrm{KQ}}(\theta;x_{1:N}^{(\theta)})-F(\theta)\right|^m\right]\\
&\quad+2^{m-1}\mathbb{E}_{x_{1:N}^{(\theta)}\sim\mathbb{P}_{\theta}}\left[\left|\widehat{F}_{\mathrm{KQ}}(\theta;x_{1:N}^{(\theta)})-F(\theta)\right|^m\right]\\
&=2^m\mathbb{E}_{x_{1:N}^{(\theta)}\sim\mathbb{P}_{\theta}}\left[\left|\widehat{F}_{\mathrm{KQ}}(\theta;x_{1:N}^{(\theta)})-F(\theta)\right|^m\right]
\end{aligned} E x 1 : N ( θ ) ∼ P θ [ ∣ r ( θ ) ∣ m ] = E x 1 : N ( θ ) ∼ P θ [ F KQ ( θ ; x 1 : N ( θ ) ) − F KQ ( θ ) m ] ≤ ( i ) 2 m − 1 E x 1 : N ( θ ) ∼ P θ [ F KQ ( θ ; x 1 : N ( θ ) ) − F ( θ ) m ] + 2 m − 1 F KQ ( θ ) − F ( θ ) m = 2 m − 1 E x 1 : N ( θ ) ∼ P θ [ F KQ ( θ ; x 1 : N ( θ ) ) − F ( θ ) m ] + 2 m − 1 E x 1 : N ( θ ) ∼ P θ [ F KQ ( θ ; x 1 : N ( θ ) ) − F ( θ ) ] m ≤ 2 m − 1 E x 1 : N ( θ ) ∼ P θ [ F KQ ( θ ; x 1 : N ( θ ) ) − F ( θ ) m ] + 2 m − 1 E x 1 : N ( θ ) ∼ P θ [ F KQ ( θ ; x 1 : N ( θ ) ) − F ( θ ) m ] = 2 m E x 1 : N ( θ ) ∼ P θ [ F KQ ( θ ; x 1 : N ( θ ) ) − F ( θ ) m ] ≤ 2 m S 4 m E x 1 : N ( θ ) ∼ P θ [ ∣ J ^ K Q ( θ ; x 1 : N ( θ ) ) − J ( θ ) ∣ m ] \le2^{m}S_{4}^{m}\mathbb{E}_{x_{1:N}^{(\theta)}\sim P_{\theta}}\left[\left|\widehat{J}_{\mathrm{KQ}}(\theta;x_{1:N}^{(\theta)})-J(\theta)\right|^{m}\right] ≤ 2 m S 4 m E x 1 : N ( θ ) ∼ P θ [ J KQ ( θ ; x 1 : N ( θ ) ) − J ( θ ) m ] ≤ ( i i ) 2 m m ! S 4 m S 1 m E m N − m s X d X ( log N ) m s X + 1 d X . (71) \underset{(ii)}{\le} 2^{m}m!S_{4}^{m}S_{1}^{m}\mathcal{E}^{m}N^{-\frac{ms_{\mathcal{X}}}{d_{\mathcal{X}}}}(\log N)^{\frac{ms_{\mathcal{X}}+1}{d_{\mathcal{X}}}}. \tag*{(71)} ( ii ) ≤ 2 m m ! S 4 m S 1 m E m N − d X m s X ( log N ) d X m s X + 1 . (71) In the above chain of derivations, ( i ) (i) ( i ) ( a + b ) m ≤ 2 m − 1 ( a m + b m ) (a+b)^{m}\le2^{m-1}(a^{m}+b^{m}) ( a + b ) m ≤ 2 m − 1 ( a m + b m ) ( i i ) (ii) ( ii ) ∣ J ^ K Q ( θ ; x 1 : N ( θ ) ) − J ( θ ) ∣ ≤ E τ N − s X d X ( log N ) s X + 1 d X S 1 \left|\widehat{J}_{\mathrm{KQ}}(\theta;x_{1:N}^{(\theta)})-J(\theta)\right|\le\mathcal{E}\tau N^{-\frac{s_{\mathcal{X}}}{d_{\mathcal{X}}}}(\log N)^{\frac{s_{\mathcal{X}}+1}{d_{\mathcal{X}}}}S_{1} J KQ ( θ ; x 1 : N ( θ ) ) − J ( θ ) ≤ E τ N − d X s X ( log N ) d X s X + 1 S 1 1 − 4 e − τ 1-4e^{-\tau} 1 − 4 e − τ E x 1 : N ( θ ) ∼ P θ ∣ J ^ K Q ( θ ; x 1 : N ( θ ) ) − J ( θ ) ∣ m \mathbb{E}_{x_{1:N}^{(\theta)}\sim P_{\theta}}\left|\widehat{J}_{\mathrm{KQ}}(\theta;x_{1:N}^{(\theta)})-J(\theta)\right|^{m} E x 1 : N ( θ ) ∼ P θ J KQ ( θ ; x 1 : N ( θ ) ) − J ( θ ) m r r r
σ = L = 2 S 4 S 1 E N − s X d X ( log N ) s X + 1 d X = C 7 N − s X d X ( log N ) s X + 1 d X , \sigma=L=2S_{4}S_{1}\mathcal{E}N^{-\frac{s_{\mathcal{X}}}{d_{\mathcal{X}}}}(\log N)^{\frac{s_{\mathcal{X}}+1}{d_{\mathcal{X}}}}=C_{7}N^{-\frac{s_{\mathcal{X}}}{d_{\mathcal{X}}}}(\log N)^{\frac{s_{\mathcal{X}}+1}{d_{\mathcal{X}}}}, σ = L = 2 S 4 S 1 E N − d X s X ( log N ) d X s X + 1 = C 7 N − d X s X ( log N ) d X s X + 1 , for C 7 : = 2 S 4 S 1 E C_{7}:=2S_{4}S_{1}\mathcal{E} C 7 := 2 S 4 S 1 E N , T N,T N , T k Θ k_{\Theta} k Θ N ( λ Θ ) \mathcal{N}(\lambda_{\Theta}) N ( λ Θ ) D Θ λ Θ − d Θ 2 s Θ D_{\Theta}\lambda_{\Theta}^{-\frac{d_{\Theta}}{2s_{\Theta}}} D Θ λ Θ − 2 s Θ d Θ k α k_{\alpha} k α α = 2 s Θ d Θ \alpha=\frac{2s_{\Theta}}{d_{\Theta}} α = d Θ 2 s Θ M Θ M_{\Theta} M Θ ∥ Σ Q ∥ \lVert\Sigma_{Q}\rVert ∥ Σ Q ∥
g λ Θ : = log ( 2 e N ( λ Θ ) ∥ Σ Q ∥ + λ Θ ∥ Σ Q ∥ ) , A λ Θ , τ : = 8 k α 2 τ g λ Θ λ Θ − d Θ 2 s Θ , g_{\lambda_{\Theta}}:=\log\left(2e\mathcal{N}(\lambda_{\Theta})\frac{\lVert\Sigma_{Q}\rVert+\lambda_{\Theta}}{\lVert\Sigma_{Q}\rVert}\right),\qquad A_{\lambda_{\Theta},\tau}:=8k_{\alpha}^{2}\tau g_{\lambda_{\Theta}}\lambda_{\Theta}^{-\frac{d_{\Theta}}{2s_{\Theta}}}, g λ Θ := log ( 2 e N ( λ Θ ) ∥ Σ Q ∥ ∥ Σ Q ∥ + λ Θ ) , A λ Θ , τ := 8 k α 2 τ g λ Θ λ Θ − 2 s Θ d Θ , L λ Θ : = max { L , λ Θ 1 2 − d Θ 4 s Θ ( ∥ F ‾ K Q ∥ L ∞ ( Q ) + k α ∥ F ‾ K Q ∥ s Θ , 2 ) } . L_{\lambda_{\Theta}}:=\max\left\{L,\lambda_{\Theta}^{\frac{1}{2}-\frac{d_{\Theta}}{4s_{\Theta}}}\left(\lVert\overline{F}_{\mathrm{KQ}}\rVert_{L_{\infty}(Q)}+k_{\alpha}\lVert\overline{F}_{\mathrm{KQ}}\rVert_{s_{\Theta},2}\right)\right\}. L λ Θ := max { L , λ Θ 2 1 − 4 s Θ d Θ ( ∥ F KQ ∥ L ∞ ( Q ) + k α ∥ F KQ ∥ s Θ , 2 ) } . Applying Theorem A.3 shows that, for T > A λ Θ , τ T>A_{\lambda_{\Theta},\tau} T > A λ Θ , τ
∥ F ‾ K Q − k ( ⋅ , θ 1 : T ) ( k Θ ( θ 1 : T , θ 1 : T ) + T λ Θ I d T ) − 1 F ^ K Q ( θ 1 : T ) ∥ L 2 ( Q ) 2 ≤ 576 τ 2 T ( L 2 D Θ λ Θ − d Θ 2 s Θ + M Θ 2 λ Θ − d Θ 2 s Θ ∥ F ‾ K Q ∥ s Θ , 2 2 + 2 M Θ 2 λ Θ T λ Θ − d Θ 2 s Θ ) + ∥ F ‾ K Q ∥ s Θ , 2 2 λ Θ . (72) \begin{aligned}
\left\lVert\overline{F}_{\mathrm{KQ}}-k(\cdot,\theta_{1:T})\left(k_{\Theta}(\theta_{1:T},\theta_{1:T})+T\lambda_{\Theta}I_{d_{T}}\right)^{-1}\widehat{F}_{\mathrm{KQ}}(\theta_{1:T})\right\rVert_{L_{2}(Q)}^{2} \\
&\le\frac{576\tau^{2}}{T}\left(L^{2}D_{\Theta}\lambda_{\Theta}^{-\frac{d_{\Theta}}{2s_{\Theta}}}+M_{\Theta}^{2}\lambda_{\Theta}^{-\frac{d_{\Theta}}{2s_{\Theta}}}\left\lVert\overline{F}_{\mathrm{KQ}}\right\rVert_{s_{\Theta},2}^{2}+2M_{\Theta}^{2}\frac{\lambda_{\Theta}}{T}\lambda_{\Theta}^{-\frac{d_{\Theta}}{2s_{\Theta}}}\right)
+\left\lVert\overline{F}_{\mathrm{KQ}}\right\rVert_{s_{\Theta},2}^{2}\lambda_{\Theta}. \tag*{(72)}
\end{aligned} F KQ − k ( ⋅ , θ 1 : T ) ( k Θ ( θ 1 : T , θ 1 : T ) + T λ Θ I d T ) − 1 F KQ ( θ 1 : T ) L 2 ( Q ) 2 ≤ T 576 τ 2 ( L 2 D Θ λ Θ − 2 s Θ d Θ + M Θ 2 λ Θ − 2 s Θ d Θ F KQ s Θ , 2 2 + 2 M Θ 2 T λ Θ λ Θ − 2 s Θ d Θ ) + F KQ s Θ , 2 2 λ Θ . (72) holds with probability at least 1 − 4 e − τ 1-4e^{-\tau} 1 − 4 e − τ λ Θ ≍ T − 2 s Θ d Θ ( log T ) 2 s Θ + 2 d Θ \lambda_{\Theta}\asymp T^{-2\frac{s_{\Theta}}{d_{\Theta}}}(\log T)^{\frac{2s_{\Theta}+2}{d_{\Theta}}} λ Θ ≍ T − 2 d Θ s Θ ( log T ) d Θ 2 s Θ + 2
lim T → ∞ A λ Θ , τ T ≤ lim T → ∞ 16 ( log T ) − s Θ + 1 s Θ k α 2 τ log ( T ) = 0. (73) \lim_{T\to\infty}\frac{A_{\lambda_{\Theta},\tau}}{T}\leq\lim_{T\to\infty}16(\log T)^{-\frac{s_{\Theta}+1}{s_{\Theta}}}k_{\alpha}^{2}\tau\log(T)=0. \tag*{(73)} T → ∞ lim T A λ Θ , τ ≤ T → ∞ lim 16 ( log T ) − s Θ s Θ + 1 k α 2 τ log ( T ) = 0. (73) It means there exists a finite T 0 > 0 T_{0}>0 T 0 > 0 T > A λ Θ , τ T>A_{\lambda_{\Theta},\tau} T > A λ Θ , τ T > T 0 T>T_{0} T > T 0 1 − 4 e − τ 1-4e^{-\tau} 1 − 4 e − τ
∥ F ˉ K Q ∥ L ∞ ( Q ) = ∥ F ˉ K Q ∥ L ∞ ( Θ ) ≤ R Θ ∥ F ˉ K Q ∥ s Θ , 2 ≤ R Θ C 6 (74) \|\bar{F}_{KQ}\|_{L_{\infty}(Q)}=\|\bar{F}_{KQ}\|_{L_{\infty}(\Theta)}\leq R_{\Theta}\|\bar{F}_{KQ}\|_{s_{\Theta},2}\leq R_{\Theta}C_{6} \tag*{(74)} ∥ F ˉ K Q ∥ L ∞ ( Q ) = ∥ F ˉ K Q ∥ L ∞ ( Θ ) ≤ R Θ ∥ F ˉ K Q ∥ s Θ , 2 ≤ R Θ C 6 (74) based on (A.29) and the fact that W 2 s Θ ( Θ ) ↪ L ∞ ( Θ ) W_{2}^{s_{\Theta}}(\Theta)\hookrightarrow L_{\infty}(\Theta) W 2 s Θ ( Θ ) ↪ L ∞ ( Θ ) ∥ W 2 s Θ ( Θ ) ↪ L ∞ ( Θ ) ∥ ≤ R Θ \|W_{2}^{s_{\Theta}}(\Theta)\hookrightarrow L_{\infty}(\Theta)\|\leq R_{\Theta} ∥ W 2 s Θ ( Θ ) ↪ L ∞ ( Θ ) ∥ ≤ R Θ
L λ Θ ≤ max { L , T − s Θ d Θ + 1 2 ( log T ) s Θ + 1 2 s Θ + 2 s Θ − d Θ d Θ ( R Θ + M Θ ) C 6 } = max { C 7 N − s X d X ( log N ) s X + 1 d X , T − s Θ d Θ + 1 2 ( log T ) s Θ + 1 2 s Θ + 2 s Θ − d Θ d Θ ( R Θ + M Θ ) C 6 } . \begin{aligned}
L_{\lambda_{\Theta}}&\leq\max\left\{L,T^{-\frac{s_{\Theta}}{d_{\Theta}}+\frac{1}{2}}(\log T)^{\frac{s_{\Theta}+1}{2s_{\Theta}}+\frac{2s_{\Theta}-d_{\Theta}}{d_{\Theta}}}(R_{\Theta}+M_{\Theta})C_{6}\right\}\\
&=\max\left\{C_{7}N^{-\frac{s_{X}}{d_{X}}}(\log N)^{\frac{s_{X}+1}{d_{X}}},T^{-\frac{s_{\Theta}}{d_{\Theta}}+\frac{1}{2}}(\log T)^{\frac{s_{\Theta}+1}{2s_{\Theta}}+\frac{2s_{\Theta}-d_{\Theta}}{d_{\Theta}}}(R_{\Theta}+M_{\Theta})C_{6}\right\}.
\end{aligned} L λ Θ ≤ max { L , T − d Θ s Θ + 2 1 ( log T ) 2 s Θ s Θ + 1 + d Θ 2 s Θ − d Θ ( R Θ + M Θ ) C 6 } = max { C 7 N − d X s X ( log N ) d X s X + 1 , T − d Θ s Θ + 2 1 ( log T ) 2 s Θ s Θ + 1 + d Θ 2 s Θ − d Θ ( R Θ + M Θ ) C 6 } . So the above (A.31) can be further upper bounded by
≤ 576 τ 2 T ( C 7 2 N − 2 s X d X ( log N ) 2 s X + 2 d X D Θ T ( log T ) − s Θ + 1 s Θ + M Θ 2 T − 2 s Θ d Θ + 1 ( log T ) s Θ + 1 s Θ + 2 s Θ − d Θ d Θ C 6 2 ) + 576 τ 2 T ⋅ 2 M Θ 2 max { C 7 2 N − 2 s X d X ( log N ) 2 s X + 2 d X , T − 2 s Θ d Θ + 1 ( log T ) s Θ + 1 s Θ + 2 s Θ − d Θ d Θ ( R Θ + M Θ ) 2 C 6 2 } ( log T ) − s Θ + 1 s Θ + C 6 2 T − 2 s Θ d Θ ( log T ) 2 s Θ + 2 d Θ ) = 576 τ 2 ( C 7 2 N − 2 s X d X ( log N ) 2 s X + 2 d X D Θ ( log T ) − s Θ + 1 s Θ + M Θ 2 T − 2 s Θ d Θ ( log T ) s Θ + 1 s Θ + 2 s Θ − d Θ d Θ C 6 2 ) + 576 τ 2 ⋅ 2 M Θ 2 max { C 7 2 N − 2 s X d X ( log N ) 2 s X + 2 d X T − 1 , T − 2 s Θ d Θ ( log T ) s Θ + 1 s Θ + 2 s Θ − d Θ d Θ ( R Θ + M Θ ) 2 C 6 2 } ( log T ) − s Θ + 1 s Θ + C 6 2 T − 2 s Θ d Θ ( log T ) 2 s Θ + 2 d Θ ) . \begin{aligned}
\leq{}&\frac{576\tau^{2}}{T}\left(C_{7}^{2}N^{-2\frac{s_{X}}{d_{X}}}(\log N)^{\frac{2s_{X}+2}{d_{X}}}D_{\Theta}T(\log T)^{-\frac{s_{\Theta}+1}{s_{\Theta}}}\right.\\
&\left.\qquad+M_{\Theta}^{2}T^{-2\frac{s_{\Theta}}{d_{\Theta}}+1}(\log T)^{\frac{s_{\Theta}+1}{s_{\Theta}}+\frac{2s_{\Theta}-d_{\Theta}}{d_{\Theta}}}C_{6}^{2}\right)\\
&+\frac{576\tau^{2}}{T}\cdot2M_{\Theta}^{2}\max\left\{C_{7}^{2}N^{-2\frac{s_{X}}{d_{X}}}(\log N)^{\frac{2s_{X}+2}{d_{X}}},\right.\\
&\left.\qquad T^{-2\frac{s_{\Theta}}{d_{\Theta}}+1}(\log T)^{\frac{s_{\Theta}+1}{s_{\Theta}}+\frac{2s_{\Theta}-d_{\Theta}}{d_{\Theta}}}(R_{\Theta}+M_{\Theta})^{2}C_{6}^{2}\right\}(\log T)^{-\frac{s_{\Theta}+1}{s_{\Theta}}}\\
&\left.\qquad+C_{6}^{2}T^{-2\frac{s_{\Theta}}{d_{\Theta}}}(\log T)^{\frac{2s_{\Theta}+2}{d_{\Theta}}}\right)\\
={}&576\tau^{2}\left(C_{7}^{2}N^{-2\frac{s_{X}}{d_{X}}}(\log N)^{\frac{2s_{X}+2}{d_{X}}}D_{\Theta}(\log T)^{-\frac{s_{\Theta}+1}{s_{\Theta}}}\right.\\
&\left.\qquad+M_{\Theta}^{2}T^{-2\frac{s_{\Theta}}{d_{\Theta}}}(\log T)^{\frac{s_{\Theta}+1}{s_{\Theta}}+\frac{2s_{\Theta}-d_{\Theta}}{d_{\Theta}}}C_{6}^{2}\right)\\
&+576\tau^{2}\cdot2M_{\Theta}^{2}\max\left\{C_{7}^{2}N^{-2\frac{s_{X}}{d_{X}}}(\log N)^{\frac{2s_{X}+2}{d_{X}}}T^{-1},\right.\\
&\left.\qquad T^{-2\frac{s_{\Theta}}{d_{\Theta}}}(\log T)^{\frac{s_{\Theta}+1}{s_{\Theta}}+\frac{2s_{\Theta}-d_{\Theta}}{d_{\Theta}}}(R_{\Theta}+M_{\Theta})^{2}C_{6}^{2}\right\}(\log T)^{-\frac{s_{\Theta}+1}{s_{\Theta}}}\\
&\left.\qquad+C_{6}^{2}T^{-2\frac{s_{\Theta}}{d_{\Theta}}}(\log T)^{\frac{2s_{\Theta}+2}{d_{\Theta}}}\right).
\end{aligned} ≤ = T 576 τ 2 ( C 7 2 N − 2 d X s X ( log N ) d X 2 s X + 2 D Θ T ( log T ) − s Θ s Θ + 1 + M Θ 2 T − 2 d Θ s Θ + 1 ( log T ) s Θ s Θ + 1 + d Θ 2 s Θ − d Θ C 6 2 ) + T 576 τ 2 ⋅ 2 M Θ 2 max { C 7 2 N − 2 d X s X ( log N ) d X 2 s X + 2 , T − 2 d Θ s Θ + 1 ( log T ) s Θ s Θ + 1 + d Θ 2 s Θ − d Θ ( R Θ + M Θ ) 2 C 6 2 } ( log T ) − s Θ s Θ + 1 + C 6 2 T − 2 d Θ s Θ ( log T ) d Θ 2 s Θ + 2 ) 576 τ 2 ( C 7 2 N − 2 d X s X ( log N ) d X 2 s X + 2 D Θ ( log T ) − s Θ s Θ + 1 + M Θ 2 T − 2 d Θ s Θ ( log T ) s Θ s Θ + 1 + d Θ 2 s Θ − d Θ C 6 2 ) + 576 τ 2 ⋅ 2 M Θ 2 max { C 7 2 N − 2 d X s X ( log N ) d X 2 s X + 2 T − 1 , T − 2 d Θ s Θ ( log T ) s Θ s Θ + 1 + d Θ 2 s Θ − d Θ ( R Θ + M Θ ) 2 C 6 2 } ( log T ) − s Θ s Θ + 1 + C 6 2 T − 2 d Θ s Θ ( log T ) d Θ 2 s Θ + 2 ) . ≤ 576 τ 2 C 7 2 N − 2 s X d X ( log N ) 2 s X + 2 d X D Θ + 576 τ 2 M Θ 2 T − 2 s Θ d Θ ( log T ) 2 s Θ + 2 d Θ C 6 2 + 576 τ 2 ⋅ 2 M Θ 2 C 7 2 N − 2 s X d X ( log N ) 2 s X + 2 d X + 576 τ 2 ⋅ 2 M Θ 2 T − 2 s Θ d Θ ( log T ) 2 s Θ + 2 d Θ ( R Θ + M Θ ) 2 C 6 2 + C 6 2 T − 2 s Θ d Θ ( log T ) 2 s Θ + 2 d Θ = τ 2 ( C 8 2 N − 2 s X d X ( log N ) 2 s X + 2 d X + C 9 2 T − 2 s Θ d Θ ( log T ) 2 s Θ + 2 d Θ ) . \begin{aligned}
&\le576\tau^{2}C_{7}^{2}N^{-\frac{2s_X}{d_X}}(\log N)^{\frac{2s_X+2}{d_X}}D_{\Theta}
+576\tau^{2}M_{\Theta}^{2}T^{-\frac{2s_{\Theta}}{d_{\Theta}}}(\log T)^{\frac{2s_{\Theta}+2}{d_{\Theta}}}C_{6}^{2}\\
&\quad+576\tau^{2}\cdot2M_{\Theta}^{2}C_{7}^{2}N^{-\frac{2s_X}{d_X}}(\log N)^{\frac{2s_X+2}{d_X}}\\
&\quad+576\tau^{2}\cdot2M_{\Theta}^{2}T^{-\frac{2s_{\Theta}}{d_{\Theta}}}(\log T)^{\frac{2s_{\Theta}+2}{d_{\Theta}}}(R_{\Theta}+M_{\Theta})^{2}C_{6}^{2}\\
&\quad+C_{6}^{2}T^{-\frac{2s_{\Theta}}{d_{\Theta}}}(\log T)^{\frac{2s_{\Theta}+2}{d_{\Theta}}}\\
&= \tau^{2}\left(C_{8}^{2}N^{-\frac{2s_X}{d_X}}(\log N)^{\frac{2s_X+2}{d_X}}
+C_{9}^{2}T^{-\frac{2s_{\Theta}}{d_{\Theta}}}(\log T)^{\frac{2s_{\Theta}+2}{d_{\Theta}}}\right).
\end{aligned} ≤ 576 τ 2 C 7 2 N − d X 2 s X ( log N ) d X 2 s X + 2 D Θ + 576 τ 2 M Θ 2 T − d Θ 2 s Θ ( log T ) d Θ 2 s Θ + 2 C 6 2 + 576 τ 2 ⋅ 2 M Θ 2 C 7 2 N − d X 2 s X ( log N ) d X 2 s X + 2 + 576 τ 2 ⋅ 2 M Θ 2 T − d Θ 2 s Θ ( log T ) d Θ 2 s Θ + 2 ( R Θ + M Θ ) 2 C 6 2 + C 6 2 T − d Θ 2 s Θ ( log T ) d Θ 2 s Θ + 2 = τ 2 ( C 8 2 N − d X 2 s X ( log N ) d X 2 s X + 2 + C 9 2 T − d Θ 2 s Θ ( log T ) d Θ 2 s Θ + 2 ) . C 8 , C 9 C_{8}, C_{9} C 8 , C 9 N , T N, T N , T max { a 1 , a 2 } ≤ a 1 + a 2 \max\{a_{1},a_{2}\}\le a_{1}+a_{2} max { a 1 , a 2 } ≤ a 1 + a 2
( log T ) s Θ + 1 s Θ + 2 s Θ − d Θ d Θ ≤ ( log T ) 2 s Θ + 2 d Θ , (\log T)^{\frac{s_{\Theta}+1}{s_{\Theta}}+\frac{2s_{\Theta}-d_{\Theta}}{d_{\Theta}}}
\le(\log T)^{\frac{2s_{\Theta}+2}{d_{\Theta}}}, ( log T ) s Θ s Θ + 1 + d Θ 2 s Θ − d Θ ≤ ( log T ) d Θ 2 s Θ + 2 , ( log T ) − s Θ + 1 s Θ ≤ 1. (\log T)^{-\frac{s_{\Theta}+1}{s_{\Theta}}}\le1. ( log T ) − s Θ s Θ + 1 ≤ 1. Therefore, we have that,
Stage II error : = ∥ F ˉ K Q − k ( ⋅ , θ 1 : T ) ( k Θ ( θ 1 : T , θ 1 : T ) + T λ Θ Id T ) − 1 F ^ K Q ( θ 1 : T ) ∥ L 2 ( Q ) ≤ τ ( C 8 N − s X d X ( log N ) s X + 1 d X + C 9 T − s Θ d Θ ( log T ) s Θ + 1 d Θ ) , (75) \begin{aligned}
\text{Stage II error}:={}&\left\|\bar{F}_{KQ}-k(\cdot,\theta_{1:T})\left(k_{\Theta}(\theta_{1:T},\theta_{1:T})+T\lambda_{\Theta}\operatorname{Id}_{T}\right)^{-1}\widehat{F}_{KQ}(\theta_{1:T})\right\|_{L_{2}(Q)}\\
\le{}&\tau\left(C_{8}N^{-\frac{s_X}{d_X}}(\log N)^{\frac{s_X+1}{d_X}}
+C_{9}T^{-\frac{s_{\Theta}}{d_{\Theta}}}(\log T)^{\frac{s_{\Theta}+1}{d_{\Theta}}}\right), \tag*{(75)}
\end{aligned} Stage II error := ≤ F ˉ K Q − k ( ⋅ , θ 1 : T ) ( k Θ ( θ 1 : T , θ 1 : T ) + T λ Θ Id T ) − 1 F K Q ( θ 1 : T ) L 2 ( Q ) τ ( C 8 N − d X s X ( log N ) d X s X + 1 + C 9 T − d Θ s Θ ( log T ) d Θ s Θ + 1 ) , (75) holds with probability at least 1 − 8 e − τ 1-8e^{-\tau} 1 − 8 e − τ
Combine stage I and stage II error Combining the stage I error of (A.18) and the stage II error of (A.34), we obtain
∣ I − I ^ N K Q ∣ ≤ Stage I error + Stage II error ≤ C 3 N − s X d X ( log N ) s X + 1 d X + τ ( C 8 N − s X d X ( log N ) s X + 1 d X + C 9 T − s Θ d Θ ( log T ) s Θ + 1 d Θ ) ≤ τ ( ( C 8 + C 3 ) N − s X d X ( log N ) s X + 1 d X + C 9 T − s Θ d Θ ( log T ) s Θ + 1 d Θ ) = τ ( C 1 N − s X d X ( log N ) s X + 1 d X + C 2 T − s Θ d Θ ( log T ) s Θ + 1 d Θ , \begin{aligned}
|I-\widehat{I}_{NKQ}|
&\le\text{Stage I error}+\text{Stage II error}\\
&\le C_{3}N^{-\frac{s_X}{d_X}}(\log N)^{\frac{s_X+1}{d_X}}
+\tau(C_{8}N^{-\frac{s_X}{d_X}}(\log N)^{\frac{s_X+1}{d_X}}
+C_{9}T^{-\frac{s_{\Theta}}{d_{\Theta}}}(\log T)^{\frac{s_{\Theta}+1}{d_{\Theta}}})\\
&\le\tau((C_{8}+C_{3})N^{-\frac{s_X}{d_X}}(\log N)^{\frac{s_X+1}{d_X}}
+C_{9}T^{-\frac{s_{\Theta}}{d_{\Theta}}}(\log T)^{\frac{s_{\Theta}+1}{d_{\Theta}}})\\
&= \tau(C_{1}N^{-\frac{s_X}{d_X}}(\log N)^{\frac{s_X+1}{d_X}} \\
&+ C_{2}T^{-\frac{s_{\Theta}}{d_{\Theta}}}(\log T)^{\frac{s_{\Theta}+1}{d_{\Theta}}},
\end{aligned} ∣ I − I N K Q ∣ ≤ Stage I error + Stage II error ≤ C 3 N − d X s X ( log N ) d X s X + 1 + τ ( C 8 N − d X s X ( log N ) d X s X + 1 + C 9 T − d Θ s Θ ( log T ) d Θ s Θ + 1 ) ≤ τ (( C 8 + C 3 ) N − d X s X ( log N ) d X s X + 1 + C 9 T − d Θ s Θ ( log T ) d Θ s Θ + 1 ) = τ ( C 1 N − d X s X ( log N ) d X s X + 1 + C 2 T − d Θ s Θ ( log T ) d Θ s Θ + 1 , holds with probability at least 1 − 8 e − τ 1-8e^{-\tau} 1 − 8 e − τ C 1 , C 2 C_{1}, C_{2} C 1 , C 2 N , T N,T N , T
Further Background and Auxiliary Lemmas
All the results in this section are existing results in the literature. We provide them here and prove some of them in the specific context of Sobolev spaces explicitly for the convenience of the reader.
More technical notions of Sobolev spaces and the Sobolev embedding theorem In the main text, we provide in (2.1) the standard definition of Sobolev spaces W 2 s ( X ) W_{2}^{s}(\mathcal{X}) W 2 s ( X ) s ∈ N s \in\mathbb{N} s ∈ N W 2 s ( X ) W_{2}^{s}(\mathcal{X}) W 2 s ( X ) s s s W 2 s ( X ) : = [ W 2 k ( X ) , W 2 k + 1 ( X ) ] r , 2 W_{2}^{s}(\mathcal{X}) := [W_{2}^{k}(\mathcal{X}), W_{2}^{k+1}(\mathcal{X})]_{r,2} W 2 s ( X ) := [ W 2 k ( X ) , W 2 k + 1 ( X ) ] r , 2 k ∈ N k \in\mathbb{N} k ∈ N s ∈ ( k , k + 1 ) s \in(k,k+1) s ∈ ( k , k + 1 ) r = s − ⌊ s ⌋ r=s-\lfloor s \rfloor r = s − ⌊ s ⌋ 1 0 ≤ s , t 0 \leq s,t 0 ≤ s , t 0 < r < 1 0<r<1 0 < r < 1
W 2 k ( X ) = [ W 2 s ( X ) , W 2 t ( X ) ] r , 2 , k = ( 1 − r ) s + r t . (76) W_{2}^{k}(\mathcal{X}) = [W_{2}^{s}(\mathcal{X}), W_{2}^{t}(\mathcal{X})]_{r,2}, \qquad k=(1-r)s+rt. \tag*{(76)} W 2 k ( X ) = [ W 2 s ( X ) , W 2 t ( X ) ] r , 2 , k = ( 1 − r ) s + r t . (76) A special case of the above relation is W 2 s ( X ) = [ L 2 ( X ) , W 2 t ( X ) ] s / t , 2 W_{2}^{s}(\mathcal{X})=[L_{2}(\mathcal{X}),W_{2}^{t}(\mathcal{X})]_{s/t,2} W 2 s ( X ) = [ L 2 ( X ) , W 2 t ( X ) ] s / t , 2 W 2 s ( X ) W_{2}^{s}(\mathcal{X}) W 2 s ( X ) s > d 2 s>\frac{d}{2} s > 2 d d d d X \mathcal{X} X W 2 s ( X ) W_{2}^{s}(\mathcal{X}) W 2 s ( X ) C 0 ( X ) C^{0}(\mathcal{X}) C 0 ( X ) [ f ] ∈ W 2 s ( X ) [f]\in W_{2}^{s}(\mathcal{X}) [ f ] ∈ W 2 s ( X ) f ∈ C 0 ( X ) f \in C^0(\mathcal{X}) f ∈ C 0 ( X ) I : W 2 s ( X ) → C 0 ( X ) I : W_2^s(\mathcal{X}) \to C^0(\mathcal{X}) I : W 2 s ( X ) → C 0 ( X ) I ( [ f ] ) = f I([f]) = f I ([ f ]) = f I I I W 2 s ( X ) ↪ C 0 ( X ) W_2^s(\mathcal{X}) \hookrightarrow C^0(\mathcal{X}) W 2 s ( X ) ↪ C 0 ( X ) ∥ W 2 s ( X ) ↪ C 0 ( X ) ∥ \|W_2^s(\mathcal{X}) \hookrightarrow C^0(\mathcal{X})\| ∥ W 2 s ( X ) ↪ C 0 ( X ) ∥ s , X s,\mathcal{X} s , X
More technical notions of reproducing kernel Hilbert spaces (RKHSs)
For bounded kernels, sup x ∈ X k ( x , x ) ≤ κ \sup_{x\in\mathcal{X}} k(x,x) \le\kappa sup x ∈ X k ( x , x ) ≤ κ H \mathcal{H} H L 2 ( Q ) L_2(\mathbb{Q}) L 2 ( Q ) ι Q : H → L 2 ( Q ) \iota_{\mathbb{Q}} : \mathcal{H} \to L_2(\mathbb{Q}) ι Q : H → L 2 ( Q ) f ↦ f f \mapsto f f ↦ f ι Q ∗ : L 2 ( Q ) → H \iota_{\mathbb{Q}}^* : L_2(\mathbb{Q}) \to\mathcal{H} ι Q ∗ : L 2 ( Q ) → H ι Q ∗ f ( ⋅ ) = ∫ k ( x , ⋅ ) f ( x ) d Q ( x ) \iota_{\mathbb{Q}}^*f(\cdot) = \int k(x,\cdot)f(x)\mathrm{d}\mathbb{Q}(x) ι Q ∗ f ( ⋅ ) = ∫ k ( x , ⋅ ) f ( x ) d Q ( x ) ι Q \iota_{\mathbb{Q}} ι Q L 2 ( Q ) L_2(\mathbb{Q}) L 2 ( Q ) T Q : = ι Q ι Q ∗ T_{\mathbb{Q}} := \iota_{\mathbb{Q}}\iota_{\mathbb{Q}}^* T Q := ι Q ι Q ∗ integral operator , and a H \mathcal{H} H
Σ Q : = ι Q ∗ ι Q = ∫ k ( ⋅ , x ) ⊗ k ( ⋅ , x ) d Q ( x ) , (77) \Sigma_{\mathbb{Q}} := \iota_{\mathbb{Q}}^*\iota_{\mathbb{Q}} = \int k(\cdot,x) \otimes k(\cdot,x)\mathrm{d}\mathbb{Q}(x), \tag*{(77)} Σ Q := ι Q ∗ ι Q = ∫ k ( ⋅ , x ) ⊗ k ( ⋅ , x ) d Q ( x ) , (77) (where ⊗ \otimes ⊗ ( a ⊗ b ) c : = ⟨ b , c ⟩ H a (a \otimes b)c := \langle b,c\rangle_{\mathcal{H}}a ( a ⊗ b ) c := ⟨ b , c ⟩ H a a , b , c ∈ H a,b,c \in\mathcal{H} a , b , c ∈ H covariance operator . Both Σ Q \Sigma_{\mathbb{Q}} Σ Q T Q T_{\mathbb{Q}} T Q ϱ 1 ≥ ⋯ ϱ i ≥ ⋯ ≥ 0 \varrho_1 \ge\cdots\varrho_i \ge\cdots\ge0 ϱ 1 ≥ ⋯ ϱ i ≥ ⋯ ≥ 0
Lemma A.9 (Effective dimension N ( λ ) \mathcal{N}(\lambda) N ( λ ) X ⊂ R d \mathcal{X} \subset\mathbb{R}^d X ⊂ R d Q \mathbb{Q} Q X \mathcal{X} X p : X → R p : \mathcal{X} \to\mathbb{R} p : X → R k : X × X → R k : \mathcal{X} \times\mathcal{X} \to\mathbb{R} k : X × X → R s > d 2 s > \frac{d}{2} s > 2 d { ϱ m } m ≥ 0 \{\varrho_m\}_{m\ge0} { ϱ m } m ≥ 0 T Q T_{\mathbb{Q}} T Q N : ( 0 , ∞ ) → [ 0 , ∞ ) \mathcal{N} : (0,\infty) \to[0,\infty) N : ( 0 , ∞ ) → [ 0 , ∞ ) N ( λ ) : = ∑ m ≥ 1 ϱ m ϱ m + λ \mathcal{N}(\lambda) := \sum_{m\ge1} \frac{\varrho_m}{\varrho_m+\lambda} N ( λ ) := ∑ m ≥ 1 ϱ m + λ ϱ m p ( x ) ≥ G > 0 p(x) \ge G > 0 p ( x ) ≥ G > 0 x ∈ X x \in\mathcal{X} x ∈ X N ( λ ) ≤ D λ − d 2 s \mathcal{N}(\lambda) \le D\lambda^{-\frac{d}{2s}} N ( λ ) ≤ D λ − 2 s d D D D G G G X \mathcal{X} X
Proof. First, we study the asymptotic behavior of the eigenvalues ( ϱ m ) m ≥ 1 (\varrho_m)_{m\ge1} ( ϱ m ) m ≥ 1 T Q T_{\mathbb{Q}} T Q ϱ m \varrho_m ϱ m e m 2 ( I Q ) e_m^2(I_{\mathbb{Q}}) e m 2 ( I Q ) I Q : W 2 s ( X ) → L 2 ( Q ) I_{\mathbb{Q}} : W_2^s(\mathcal{X}) \to L_2(\mathbb{Q}) I Q : W 2 s ( X ) → L 2 ( Q ) L X \mathcal{L}_{\mathcal{X}} L X X \mathcal{X} X p ( x ) ≥ G p(x) \ge G p ( x ) ≥ G x ∈ X x \in\mathcal{X} x ∈ X
d L X d Q ≤ G − 1 Vol ( X ) − 1 \frac{d\mathcal{L}_{\mathcal{X}}}{d\mathbb{Q}} \le G^{-1}\operatorname{Vol}(\mathcal{X})^{-1} d Q d L X ≤ G − 1 Vol ( X ) − 1 so ∥ L 2 ( Q ) ↪ L 2 ( X ) ∥ ≤ G − 1 Vol ( X ) − 1 \lVert L_2(\mathbb{Q}) \hookrightarrow L_2(\mathcal{X})\rVert\le G^{-1}\operatorname{Vol}(\mathcal{X})^{-1} ∥ L 2 ( Q ) ↪ L 2 ( X )∥ ≤ G − 1 Vol ( X ) − 1
e m ( I Q ) ≤ e m ( I L X ) ∥ L 2 ( Q ) ↪ L 2 ( X ) ∥ ≤ G − 1 Vol ( X ) − 1 e m ( I L X ) . e_m(I_{\mathbb{Q}}) \le e_m(I_{\mathcal{L}_{\mathcal{X}}})\lVert L_2(\mathbb{Q}) \hookrightarrow L_2(\mathcal{X})\rVert\le G^{-1}\operatorname{Vol}(\mathcal{X})^{-1}e_m(I_{\mathcal{L}_{\mathcal{X}}}). e m ( I Q ) ≤ e m ( I L X ) ∥ L 2 ( Q ) ↪ L 2 ( X )∥ ≤ G − 1 Vol ( X ) − 1 e m ( I L X ) . Moreover, (Edmunds and Triebel, 1996, Equation 4 on p. 119) shows that the entropy number e m ( I L X ) ≤ c ~ m − s / d e_m(I_{\mathcal{L}_{\mathcal{X}}}) \le\tilde{c}m^{-s/d} e m ( I L X ) ≤ c ~ m − s / d c ~ \tilde{c} c ~
e m ( I Q ) ≤ G − 1 Vol ( X ) − 1 c ~ m − s / d e_m(I_{\mathbb{Q}}) \le G^{-1}\operatorname{Vol}(\mathcal{X})^{-1}\tilde{c}m^{-s/d} e m ( I Q ) ≤ G − 1 Vol ( X ) − 1 c ~ m − s / d and consequently we have
ϱ m ≳ e m 2 ( I Q ) ≤ G − 2 Vol ( X ) − 2 c ~ 2 m − 2 s / d = : c 2 m − 2 s / d . \varrho_m \gtrsim e_m^2(I_{\mathbb{Q}}) \le G^{-2}\operatorname{Vol}(\mathcal{X})^{-2}\tilde{c}^2m^{-2s/d} =: c_2m^{-2s/d}. ϱ m ≳ e m 2 ( I Q ) ≤ G − 2 Vol ( X ) − 2 c ~ 2 m − 2 s / d =: c 2 m − 2 s / d . Next, we have
∑ m ≥ 1 ϱ m ϱ m + λ ≤ ∑ m ≥ 1 1 1 + λ c 2 − 1 m 2 s / d ≤ ∫ 0 ∞ c 2 c 2 + λ t 2 s / d d t = λ − d 2 s ∫ 0 ∞ c 2 c 2 + τ 2 s / d d τ = λ − d 2 s ∫ 0 ∞ c 2 d 2 s 1 + u 2 s d d u = λ − d 2 s c 2 d 2 s π d 2 s 1 sin ( π d 2 s ) = : D λ − d 2 s , \begin{aligned}
\sum_{m\ge1}\frac{\varrho_m}{\varrho_m+\lambda}
&\le\sum_{m\ge1}\frac{1}{1+\lambda c_2^{-1}m^{2s/d}}
\le\int_0^\infty\frac{c_2}{c_2+\lambda t^{2s/d}}\,dt \\
&= \lambda^{-\frac{d}{2s}}\int_0^\infty\frac{c_2}{c_2+\tau^{2s/d}}\,d\tau
= \lambda^{-\frac{d}{2s}}\int_0^\infty\frac{c_2^{\frac{d}{2s}}}{1+u^{\frac{2s}{d}}}\,du \\
&= \lambda^{-\frac{d}{2s}}c_2^{\frac{d}{2s}}\frac{\pi d}{2s}\frac{1}{\sin\left(\frac{\pi d}{2s}\right)}
=:D\lambda^{-\frac{d}{2s}},
\end{aligned} m ≥ 1 ∑ ϱ m + λ ϱ m ≤ m ≥ 1 ∑ 1 + λ c 2 − 1 m 2 s / d 1 ≤ ∫ 0 ∞ c 2 + λ t 2 s / d c 2 d t = λ − 2 s d ∫ 0 ∞ c 2 + τ 2 s / d c 2 d τ = λ − 2 s d ∫ 0 ∞ 1 + u d 2 s c 2 2 s d d u = λ − 2 s d c 2 2 s d 2 s π d sin ( 2 s π d ) 1 =: D λ − 2 s d , where D D D X \mathcal{X} X G G G
Lemma A.10. Let X ⊂ R d \mathcal{X} \subset\mathbb{R}^d X ⊂ R d Q \mathbb{Q} Q X \mathcal{X} X p : X → R p:\mathcal{X}\to\mathbb{R} p : X → R k : X × X → R k:\mathcal{X}\times\mathcal{X}\to\mathbb{R} k : X × X → R s > d 2 s>\frac{d}{2} s > 2 d { ϱ m , e m } m ≥ 0 \{\varrho_m,e_m\}_{m\ge0} { ϱ m , e m } m ≥ 0 T Q T_{\mathbb{Q}} T Q G 0 , G 1 > 0 G_0,G_1>0 G 0 , G 1 > 0 G 0 ≤ p ( x ) ≤ G 1 G_0\le p(x)\le G_1 G 0 ≤ p ( x ) ≤ G 1 x ∈ X x\in\mathcal{X} x ∈ X
k α : = sup x ∈ X ∑ m ≥ 1 ϱ m α e m 2 ( x ) ≤ M , (78) k_\alpha:= \sup_{x\in\mathcal{X}}\sum_{m\ge1}\varrho_m^\alpha e_m^2(x)\le M, \tag*{(78)} k α := x ∈ X sup m ≥ 1 ∑ ϱ m α e m 2 ( x ) ≤ M , (78) holds for any d 2 s < α \frac{d}{2s}<\alpha 2 s d < α M M M X \mathcal{X} X G 1 , G 0 G_1,G_0 G 1 , G 0
Proof. If t > d 2 t>\frac{d}{2} t > 2 d W 2 t ( X ) W_2^t(\mathcal{X}) W 2 t ( X ) L ∞ ( X ) L_\infty(\mathcal{X}) L ∞ ( X ) W 2 s ( X ) ↪ L ∞ ( X ) W_2^s(\mathcal{X})\hookrightarrow L_\infty(\mathcal{X}) W 2 s ( X ) ↪ L ∞ ( X )
two normed vector spaces, hence a bounded operator. And L 2 ( Q ) L_{2}(\mathbb{Q}) L 2 ( Q ) L 2 ( X ) L_{2}(\mathcal{X}) L 2 ( X ) G 0 ≤ p ( x ) ≤ G 1 G_{0} \le p(x) \le G_{1} G 0 ≤ p ( x ) ≤ G 1 x ∈ X x \in\mathcal{X} x ∈ X k α k_{\alpha} k α ∥ k ν α ∥ ∞ \lVert k_{\nu}^{\alpha}\rVert_{\infty} ∥ k ν α ∥ ∞
sup x ∈ X ∑ m ≥ 1 ϱ m α e m 2 ( x ) = ∥ [ L 2 ( Q ) , W 2 s ( X ) ] α , 2 ↪ L ∞ ( X ) ∥ . \sup_{x\in\mathcal{X}}\sum_{m\geq1}\varrho_{m}^{\alpha}e_{m}^{2}(x)=\left\lVert[L_{2}(\mathbb{Q}),W_{2}^{s}(\mathcal{X})]_{\alpha,2}\hookrightarrow L_{\infty}(\mathcal{X})\right\rVert. x ∈ X sup m ≥ 1 ∑ ϱ m α e m 2 ( x ) = ∥ [ L 2 ( Q ) , W 2 s ( X ) ] α , 2 ↪ L ∞ ( X ) ∥ . Notice that [ L 2 ( Q ) , W 2 s ( X ) ] α , 2 ≅ [ L 2 ( X ) , W 2 s ( X ) ] α , 2 ≅ W 2 s α ( X ) [L_{2}(\mathbb{Q}),W_{2}^{s}(\mathcal{X})]_{\alpha,2}\cong[L_{2}(\mathcal{X}),W_{2}^{s}(\mathcal{X})]_{\alpha,2}\cong W_{2}^{s\alpha}(\mathcal{X}) [ L 2 ( Q ) , W 2 s ( X ) ] α , 2 ≅ [ L 2 ( X ) , W 2 s ( X ) ] α , 2 ≅ W 2 s α ( X ) W 2 s α ( X ) ↪ L ∞ ( X ) W_{2}^{s\alpha}(\mathcal{X})\hookrightarrow L_{\infty}(\mathcal{X}) W 2 s α ( X ) ↪ L ∞ ( X ) s α > d 2 s\alpha> \frac{d}{2} s α > 2 d d 2 s < α \frac{d}{2s}<\alpha 2 s d < α
Lemma A.11. Let X ⊂ R d \mathcal{X}\subset\mathbb{R}^{d} X ⊂ R d W 2 s ( X ) W_{2}^{s}(\mathcal{X}) W 2 s ( X ) s > d 2 s>\frac{d}{2} s > 2 d f : X → R f:\mathcal{X}\to\mathbb{R} f : X → R g : X → R g:\mathcal{X}\to\mathbb{R} g : X → R W 2 s ( X ) W_{2}^{s}(\mathcal{X}) W 2 s ( X ) f ⋅ g f\cdot g f ⋅ g W 2 s ( X ) W_{2}^{s}(\mathcal{X}) W 2 s ( X ) ∥ f ⋅ g ∥ s ≤ ∥ f ∥ s ∥ g ∥ s \lVert f\cdot g\rVert_{s}\leq\lVert f\rVert_{s}\lVert g\rVert_{s} ∥ f ⋅ g ∥ s ≤ ∥ f ∥ s ∥ g ∥ s
Proof. This is Theorem 7.4 of Behzadan and Holst (2021) with s 1 = s 2 = s s_{1}=s_{2}=s s 1 = s 2 = s p 1 = p 2 = 2 p_{1}=p_{2}=2 p 1 = p 2 = 2
Lemma A.12. For a positive valued random variable R R R c > 0 c>0 c > 0 P ( R ≤ c τ ) ≥ 1 − exp ( − τ ) \mathbb{P}(R\leq c\tau)\geq1-\exp(-\tau) P ( R ≤ c τ ) ≥ 1 − exp ( − τ ) τ \tau τ E [ R m ] ≤ c 0 m ! \mathbb{E}[R^{m}]\leq c_{0}m! E [ R m ] ≤ c 0 m ! m ≥ 1 m\geq1 m ≥ 1 c 0 c_{0} c 0 c , m c,m c , m
Proof. Notice that R R R E [ R m ] = E [ R 2 m ] ≤ 2 c 0 Γ ( m + 1 ) = 2 c 0 m ! \mathbb{E}[R^{m}]=\mathbb{E}[\sqrt{R}^{2m}]\leq2c_{0}\Gamma(m+1)=2c_{0}m! E [ R m ] = E [ R 2 m ] ≤ 2 c 0 Γ ( m + 1 ) = 2 c 0 m ! Γ \Gamma Γ c 0 c_{0} c 0 c , m c,m c , m
Lemma A.13. For a mapping F F F X ⊂ R d \mathcal{X}\subset\mathbb{R}^{d} X ⊂ R d H H H μ \mu μ X \mathcal{X} X F F F μ \mu μ ∫ F ( x ) d μ ( x ) ∈ H \int F(x)d\mu(x)\in H ∫ F ( x ) d μ ( x ) ∈ H ∥ ∫ F ( x ) d μ ( x ) ∥ H ≤ ∫ ∥ F ( x ) ∥ H d μ ( x ) \left\lVert\int F(x)d\mu(x)\right\rVert_{H}\leq\int\lVert F(x)\rVert_{H}d\mu(x) ∫ F ( x ) d μ ( x ) H ≤ ∫ ∥ F ( x ) ∥ H d μ ( x )
Proof. This is Definition A.5.20 of Steinwart and Christmann (2008).
Supplementary Material for Minimum MMD for Generative Modelling
Proofs
In this section, we present the proofs of the results presented in Chapter 4.
Proof of the regularized MMD proposition
Note that
MMD ( μ ∥ π ) 2 = ∥ m μ − m π ∥ H 2 = ∥ ∫ k ( ⋅ , x ) d μ ( x ) − ∫ k ( ⋅ , x ) d π ( x ) ∥ H 2 = ∥ ∫ k ( ⋅ , x ) ( d μ d π ( x ) − 1 ) d π ( x ) ∥ H 2 = ∥ ι ∗ π ( d μ d π − 1 ) ∥ H 2 = ⟨ T π ( d μ d π − 1 ) , d μ d π − 1 ⟩ L 2 ( π ) = ∥ T π 1 2 ( d μ d π − 1 ) ∥ L 2 ( π ) 2 . \begin{aligned}
\operatorname{MMD}(\mu\mathbin\Vert\pi)^2 = \lVert m_\mu-m_\pi\rVert_{\mathcal{H}}^2 = \left\lVert\int k(\cdot,x)\,\mathrm{d}\mu(x)-\int k(\cdot,x)\,\mathrm{d}\pi(x)\right\rVert_{\mathcal{H}}^2 \\
&= \left\lVert\int k(\cdot,x)\left(\frac{\mathrm{d}\mu}{\mathrm{d}\pi}(x)-1\right)\mathrm{d}\pi(x)\right\rVert_{\mathcal{H}}^2 = \left\lVert\iota_*^\pi\left(\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right)\right\rVert_{\mathcal{H}}^2 \\
&= \left\langle\mathcal{T}_\pi\left(\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right),\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right\rangle_{L^2(\pi)} = \left\lVert\mathcal{T}_\pi^{\frac{1}{2}}\left(\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right)\right\rVert_{L^2(\pi)}^2.
\end{aligned} MMD ( μ ∥ π ) 2 = ∥ m μ − m π ∥ H 2 = ∫ k ( ⋅ , x ) d μ ( x ) − ∫ k ( ⋅ , x ) d π ( x ) H 2 = ∫ k ( ⋅ , x ) ( d π d μ ( x ) − 1 ) d π ( x ) H 2 = ι ∗ π ( d π d μ − 1 ) H 2 = ⟨ T π ( d π d μ − 1 ) , d π d μ − 1 ⟩ L 2 ( π ) = T π 2 1 ( d π d μ − 1 ) L 2 ( π ) 2 . Also recall that the χ 2 \chi^2 χ 2 μ \mu μ π \pi π
χ 2 ( μ ∥ π ) = ∫ ( d μ d π − 1 ) 2 d π = ∥ 1 ( d μ d π − 1 ) ∥ L 2 ( π ) 2 . \chi^2(\mu\mathbin\Vert\pi)=\int\left(\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right)^2\mathrm{d}\pi=\left\lVert1\left(\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right)\right\rVert_{L^2(\pi)}^2. χ 2 ( μ ∥ π ) = ∫ ( d π d μ − 1 ) 2 d π = 1 ( d π d μ − 1 ) L 2 ( π ) 2 . Proof of the DrMMD representation proposition
Let μ ≪ π \mu\ll\pi μ ≪ π (14) , we start from (14) and show that it recovers (13) .
DrMMD ( μ ∥ π ) = ( 1 + λ ) ∥ ( Σ π + λ Id ) − 1 2 ( m μ − m π ) ∥ H 2 = ( 1 + λ ) ∥ ( ι π ∗ ι π + λ Id ) − 1 2 ι π ∗ ( d μ d π − 1 ) ∥ H 2 = ( 1 + λ ) ⟨ ( ι π ∗ ι π + λ Id ) − 1 2 ι π ∗ ( d μ d π − 1 ) , ( ι π ∗ ι π + λ Id ) − 1 2 ι π ∗ ( d μ d π − 1 ) ⟩ H = ( 1 + λ ) ⟨ ι π ( ι π ∗ ι π + λ Id ) − 1 ι π ∗ ( d μ d π − 1 ) , d μ d π − 1 ⟩ L 2 ( π ) = ( 1 + λ ) ⟨ ι π ι π ∗ ( ι π ι π ∗ + λ Id ) − 1 ( d μ d π − 1 ) , d μ d π − 1 ⟩ L 2 ( π ) , (79) \begin{aligned}
\operatorname{DrMMD}(\mu\Vert\pi) &= (1+\lambda)\left\|(\Sigma_\pi+\lambda\operatorname{Id})^{-\frac{1}{2}}(m_\mu-m_\pi)\right\|_{\mathcal{H}}^2 \\
&= (1+\lambda)\left\|(\iota_\pi^*\iota_\pi+\lambda\operatorname{Id})^{-\frac{1}{2}}\iota_\pi^*\left(\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right)\right\|_{\mathcal{H}}^2 \\
&= (1+\lambda)\left\langle(\iota_\pi^*\iota_\pi+\lambda\operatorname{Id})^{-\frac{1}{2}}\iota_\pi^*\left(\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right),(\iota_\pi^*\iota_\pi+\lambda\operatorname{Id})^{-\frac{1}{2}}\iota_\pi^*\left(\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right)\right\rangle_{\mathcal{H}} \\
&= (1+\lambda)\left\langle\iota_\pi(\iota_\pi^*\iota_\pi+\lambda\operatorname{Id})^{-1}\iota_\pi^*\left(\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right),\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right\rangle_{L^2(\pi)} \\
&= (1+\lambda)\left\langle\iota_\pi\iota_\pi^*(\iota_\pi\iota_\pi^*+\lambda\operatorname{Id})^{-1}\left(\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right),\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right\rangle_{L^2(\pi)}, \tag*{(79)}
\end{aligned} DrMMD ( μ ∥ π ) = ( 1 + λ ) ( Σ π + λ Id ) − 2 1 ( m μ − m π ) H 2 = ( 1 + λ ) ( ι π ∗ ι π + λ Id ) − 2 1 ι π ∗ ( d π d μ − 1 ) H 2 = ( 1 + λ ) ⟨ ( ι π ∗ ι π + λ Id ) − 2 1 ι π ∗ ( d π d μ − 1 ) , ( ι π ∗ ι π + λ Id ) − 2 1 ι π ∗ ( d π d μ − 1 ) ⟩ H = ( 1 + λ ) ⟨ ι π ( ι π ∗ ι π + λ Id ) − 1 ι π ∗ ( d π d μ − 1 ) , d π d μ − 1 ⟩ L 2 ( π ) = ( 1 + λ ) ⟨ ι π ι π ∗ ( ι π ι π ∗ + λ Id ) − 1 ( d π d μ − 1 ) , d π d μ − 1 ⟩ L 2 ( π ) , (79) where the last equality follows from
ι π ( ι π ∗ ι π + λ Id ) − 1 ι π ∗ = ι π ι π ∗ ( ι π ι π ∗ + λ Id ) − 1 . \iota_\pi(\iota_\pi^*\iota_\pi+\lambda\operatorname{Id})^{-1}\iota_\pi^*=\iota_\pi\iota_\pi^*(\iota_\pi\iota_\pi^*+\lambda\operatorname{Id})^{-1}. ι π ( ι π ∗ ι π + λ Id ) − 1 ι π ∗ = ι π ι π ∗ ( ι π ι π ∗ + λ Id ) − 1 . Therefore,
DrMMD ( μ ∥ π ) = ( 1 + λ ) ⟨ T π ( T π + λ Id ) − 1 ( d μ d π − 1 ) , d μ d π − 1 ⟩ L 2 ( π ) = ( 1 + λ ) ∥ ( ( T π + λ I ) − 1 T π ) 1 2 ( d μ d π − 1 ) ∥ L 2 ( π ) 2 , \begin{aligned}
\operatorname{DrMMD}(\mu\Vert\pi) &= (1+\lambda)\left\langle T_\pi(T_\pi+\lambda\operatorname{Id})^{-1}\left(\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right),\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right\rangle_{L^2(\pi)} \\
&= (1+\lambda)\left\|\left((T_\pi+\lambda I)^{-1}T_\pi\right)^{\frac{1}{2}}\left(\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right)\right\|_{L^2(\pi)}^2,
\end{aligned} DrMMD ( μ ∥ π ) = ( 1 + λ ) ⟨ T π ( T π + λ Id ) − 1 ( d π d μ − 1 ) , d π d μ − 1 ⟩ L 2 ( π ) = ( 1 + λ ) ( ( T π + λ I ) − 1 T π ) 2 1 ( d π d μ − 1 ) L 2 ( π ) 2 , which follows from the positivity and self-adjointness of T π ( T π + λ Id ) − 1 T_\pi(T_\pi+\lambda\operatorname{Id})^{-1} T π ( T π + λ Id ) − 1 (14) is proved. Next, we are going to prove the variational formulation in (15) . Similarly, we start from (15) and show it recovers (13) . Consider
( 1 + λ ) sup h ∈ H { ∫ h d μ − ∫ ( h 2 4 + h ) d π − λ 4 ∥ h ∥ H 2 } = − ( 1 + λ ) inf h ∈ H { ∫ h 2 4 d π − ∫ h ( d μ − d π ) + λ 4 ∥ h ∥ H 2 } = − ( 1 + λ ) inf h ∈ H { 1 4 ⟨ h , Σ π h ⟩ H − ⟨ h , m μ − m π ⟩ H + λ 4 ∥ h ∥ H 2 } . = − ( 1 + λ ) inf h ∈ H { ∥ ( 1 4 Σ π + λ 4 Id ) 1 / 2 h − 1 2 ( 1 4 Σ π + λ 4 Id ) − 1 / 2 ( m μ − m π ) ∥ H 2 } + ( 1 + λ ) 4 ∥ ( 1 4 Σ π + λ 4 Id ) − 1 / 2 ( m μ − m π ) ∥ H 2 . (80) \begin{aligned}
(1+\lambda)\sup_{h\in\mathcal{H}}\left\{\int h\mathrm{d}\mu-\int\left(\frac{h^2}{4}+h\right)\mathrm{d}\pi-\frac{\lambda}{4}\|h\|_{\mathcal{H}}^2\right\}
&=-(1+\lambda)\inf_{h\in\mathcal{H}}\left\{\int\frac{h^2}{4}\mathrm{d}\pi-\int h(\mathrm{d}\mu-\mathrm{d}\pi)+\frac{\lambda}{4}\|h\|_{\mathcal{H}}^2\right\}\\
&=-(1+\lambda)\inf_{h\in\mathcal{H}}\left\{\frac{1}{4}\langle h,\Sigma_\pi h\rangle_{\mathcal{H}}-\langle h,m_\mu-m_\pi\rangle_{\mathcal{H}}+\frac{\lambda}{4}\|h\|_{\mathcal{H}}^2\right\}. \\
&= -(1+\lambda)\inf_{h\in\mathcal{H}}\left\{\left\|\left(\frac{1}{4}\Sigma_\pi+\frac{\lambda}{4}\operatorname{Id}\right)^{1/2}h-\frac{1}{2}\left(\frac{1}{4}\Sigma_\pi+\frac{\lambda}{4}\operatorname{Id}\right)^{-1/2}(m_\mu-m_\pi)\right\|_{\mathcal{H}}^2\right\}
+\frac{(1+\lambda)}{4}\left\|\left(\frac{1}{4}\Sigma_\pi+\frac{\lambda}{4}\operatorname{Id}\right)^{-1/2}(m_\mu-m_\pi)\right\|_{\mathcal{H}}^2. \tag*{(80)}
\end{aligned} ( 1 + λ ) h ∈ H sup { ∫ h d μ − ∫ ( 4 h 2 + h ) d π − 4 λ ∥ h ∥ H 2 } = − ( 1 + λ ) h ∈ H inf { ∫ 4 h 2 d π − ∫ h ( d μ − d π ) + 4 λ ∥ h ∥ H 2 } = − ( 1 + λ ) h ∈ H inf { 4 1 ⟨ h , Σ π h ⟩ H − ⟨ h , m μ − m π ⟩ H + 4 λ ∥ h ∥ H 2 } . = − ( 1 + λ ) h ∈ H inf ⎩ ⎨ ⎧ ( 4 1 Σ π + 4 λ Id ) 1/2 h − 2 1 ( 4 1 Σ π + 4 λ Id ) − 1/2 ( m μ − m π ) H 2 ⎭ ⎬ ⎫ + 4 ( 1 + λ ) ( 4 1 Σ π + 4 λ Id ) − 1/2 ( m μ − m π ) H 2 . (80) The last equality follows from completing the squares, based on which it is easy to see that the infimum is achieved at h μ , π ∗ = 2 ( Σ π + λ Id ) − 1 ( m μ − m π ) h^*_{\mu,\pi}=2(\Sigma_\pi+\lambda\operatorname{Id})^{-1}(m_\mu-m_\pi) h μ , π ∗ = 2 ( Σ π + λ Id ) − 1 ( m μ − m π ) μ ≪ π \mu\ll\pi μ ≪ π h μ , π ∗ h^*_{\mu,\pi} h μ , π ∗
h μ , π ∗ = 2 ( T π + λ I ) − 1 T π ( d μ d π − 1 ) (81) h^*_{\mu,\pi}=2(\mathcal{T}_\pi+\lambda I)^{-1}\mathcal{T}_\pi\left(\frac{d\mu}{d\pi}-1\right) \tag*{(81)} h μ , π ∗ = 2 ( T π + λ I ) − 1 T π ( d π d μ − 1 ) (81) Plugging h μ , π ∗ h^*_{\mu,\pi} h μ , π ∗
DrMMD is MMD with a regularized kernel k ~ \tilde{k} k ~
Define the regularized kernel by
k ~ ( x , x ′ ) = ⟨ ( Σ π + λ Id ) − 1 / 2 k ( ⋅ , x ) , ( Σ π + λ Id ) − 1 / 2 k ( ⋅ , x ′ ) ⟩ H . \tilde{k}(x,x')=\left\langle(\Sigma_\pi+\lambda\operatorname{Id})^{-1/2}k(\cdot,x),(\Sigma_\pi+\lambda\operatorname{Id})^{-1/2}k(\cdot,x')\right\rangle_{\mathcal{H}}. k ~ ( x , x ′ ) = ⟨ ( Σ π + λ Id ) − 1/2 k ( ⋅ , x ) , ( Σ π + λ Id ) − 1/2 k ( ⋅ , x ′ ) ⟩ H . It is clear that k ~ \tilde{k} k ~ H ~ \tilde{\mathcal{H}} H ~ k ~ ( x , ⋅ ) \tilde{k}(x,\cdot) k ~ ( x , ⋅ ) A λ : = ( Σ π + λ Id ) − 1 / 2 A_\lambda:=(\Sigma_\pi+\lambda\operatorname{Id})^{-1/2} A λ := ( Σ π + λ Id ) − 1/2
DrMMD ( μ ∥ π ) = ( 1 + λ ) ∥ A λ ( m μ − m π ) ∥ H 2 = ( 1 + λ ) ⟨ A λ ∫ k ( x , ⋅ ) d ( π − μ ) ( x ) , A λ ∫ k ( x ′ , ⋅ ) d ( π − μ ) ( x ′ ) ⟩ H = ( 1 + λ ) ⟨ ∫ A λ k ( x , ⋅ ) d ( π − μ ) ( x ) , ∫ A λ k ( x ′ , ⋅ ) d ( π − μ ) ( x ′ ) ⟩ H \begin{aligned}
\operatorname{DrMMD}(\mu\Vert\pi)
&=(1+\lambda)\left\|A_\lambda(m_\mu-m_\pi)\right\|_{\mathcal{H}}^2\\
&=(1+\lambda)\left\langle A_\lambda\int k(x,\cdot)\,d(\pi-\mu)(x),A_\lambda\int k(x',\cdot)\,d(\pi-\mu)(x')\right\rangle_{\mathcal{H}}\\
&=(1+\lambda)\left\langle\int A_\lambda k(x,\cdot)\,d(\pi-\mu)(x),\int A_\lambda k(x',\cdot)\,d(\pi-\mu)(x')\right\rangle_{\mathcal{H}}
\end{aligned} DrMMD ( μ ∥ π ) = ( 1 + λ ) ∥ A λ ( m μ − m π ) ∥ H 2 = ( 1 + λ ) ⟨ A λ ∫ k ( x , ⋅ ) d ( π − μ ) ( x ) , A λ ∫ k ( x ′ , ⋅ ) d ( π − μ ) ( x ′ ) ⟩ H = ( 1 + λ ) ⟨ ∫ A λ k ( x , ⋅ ) d ( π − μ ) ( x ) , ∫ A λ k ( x ′ , ⋅ ) d ( π − μ ) ( x ′ ) ⟩ H = ( 1 + λ ) ∬ ⟨ A λ k ( x , ⋅ ) , A λ k ( x ′ , ⋅ ) ⟩ H d ( π − μ ) ( x ) d ( π − μ ) ( x ′ ) = ( 1 + λ ) ∬ k ~ ( x , x ′ ) d ( π − μ ) ( x ) d ( π − μ ) ( x ′ ) = ( 1 + λ ) ∥ ∬ k ~ ( x , ⋅ ) d ( μ − π ) ( x ) ∥ H ~ 2 . \begin{aligned}
&= (1+\lambda)\iint\left\langle A_\lambda k(x,\cdot),A_\lambda k(x',\cdot)\right\rangle_{\mathcal{H}}\,\mathrm{d}(\pi-\mu)(x)\mathrm{d}(\pi-\mu)(x')\\
&= (1+\lambda)\iint\widetilde{k}(x,x')\,\mathrm{d}(\pi-\mu)(x)\mathrm{d}(\pi-\mu)(x')\\
&= (1+\lambda)\left\|\iint\widetilde{k}(x,\cdot)\,\mathrm{d}(\mu-\pi)(x)\right\|_{\widetilde{\mathcal{H}}}^{2}.
\end{aligned} = ( 1 + λ ) ∬ ⟨ A λ k ( x , ⋅ ) , A λ k ( x ′ , ⋅ ) ⟩ H d ( π − μ ) ( x ) d ( π − μ ) ( x ′ ) = ( 1 + λ ) ∬ k ( x , x ′ ) d ( π − μ ) ( x ) d ( π − μ ) ( x ′ ) = ( 1 + λ ) ∬ k ( x , ⋅ ) d ( μ − π ) ( x ) H 2 . In the third and fourth equality above, we are using the fact that ( Σ π + λ Id ) − 1 / 2 k ( x , ⋅ ) ∈ H (\Sigma_\pi+\lambda\operatorname{Id})^{-1/2}k(x,\cdot)\in\mathcal{H} ( Σ π + λ Id ) − 1/2 k ( x , ⋅ ) ∈ H 2 ^{2} 2 k ~ \widetilde{k} k 1 + λ 1+\lambda 1 + λ
Next, we present the Mercer decomposition of k ~ \widetilde{k} k { e i } i ≥ 1 \{e_i\}_{i\ge1} { e i } i ≥ 1 T π T_\pi T π { ϱ i e i } i ≥ 1 \{\sqrt{\varrho_i}e_i\}_{i\ge1} { ϱ i e i } i ≥ 1 Σ π \Sigma_\pi Σ π x x x x ′ x' x ′ π \pi π k ~ \widetilde{k} k
k ~ ( x , x ′ ) = ⟨ ( Σ π + λ Id ) − 1 / 2 k ( ⋅ , x ) , ( Σ π + λ Id ) − 1 / 2 k ( ⋅ , x ′ ) ⟩ H = ⟨ ( Σ π + λ Id ) − 1 / 2 ( ∑ i ≥ 1 ϱ i e i ( x ) e i ) , ( Σ π + λ Id ) − 1 / 2 ( ∑ i ≥ 1 ϱ i e i ( x ′ ) e i ) ⟩ H = ⟨ ∑ i ≥ 1 ϱ i ϱ i + λ e i ( x ) e i , ∑ i ≥ 1 ϱ i ϱ i + λ e i ( x ′ ) e i ⟩ H = ∑ i ≥ 1 ϱ i ϱ i + λ e i ( x ) e i ( x ′ ) . (82) \begin{aligned}
\widetilde{k}(x,x')&=\left\langle(\Sigma_\pi+\lambda\operatorname{Id})^{-1/2}k(\cdot,x),(\Sigma_\pi+\lambda\operatorname{Id})^{-1/2}k(\cdot,x')\right\rangle_{\mathcal{H}}\\
&=\left\langle(\Sigma_\pi+\lambda\operatorname{Id})^{-1/2}\left(\sum_{i\ge1}\varrho_i e_i(x)e_i\right),(\Sigma_\pi+\lambda\operatorname{Id})^{-1/2}\left(\sum_{i\ge1}\varrho_i e_i(x')e_i\right)\right\rangle_{\mathcal{H}}\\
&=\left\langle\sum_{i\ge1}\frac{\varrho_i}{\sqrt{\varrho_i+\lambda}}e_i(x)e_i,\sum_{i\ge1}\frac{\varrho_i}{\sqrt{\varrho_i+\lambda}}e_i(x')e_i\right\rangle_{\mathcal{H}}\\
&=\sum_{i\ge1}\frac{\varrho_i}{\varrho_i+\lambda}e_i(x)e_i(x'). \tag*{(82)}
\end{aligned} k ( x , x ′ ) = ⟨ ( Σ π + λ Id ) − 1/2 k ( ⋅ , x ) , ( Σ π + λ Id ) − 1/2 k ( ⋅ , x ′ ) ⟩ H = ⟨ ( Σ π + λ Id ) − 1/2 ( i ≥ 1 ∑ ϱ i e i ( x ) e i ) , ( Σ π + λ Id ) − 1/2 ( i ≥ 1 ∑ ϱ i e i ( x ′ ) e i ) ⟩ H = ⟨ i ≥ 1 ∑ ϱ i + λ ϱ i e i ( x ) e i , i ≥ 1 ∑ ϱ i + λ ϱ i e i ( x ′ ) e i ⟩ H = i ≥ 1 ∑ ϱ i + λ ϱ i e i ( x ) e i ( x ′ ) . (82) More properties of the regularized kernel k ~ \widetilde{k} k
Proof of Theorem 4.7
Given that d μ d π − 1 ∈ L 2 ( π ) \frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\in L^{2}(\pi) d π d μ − 1 ∈ L 2 ( π )
DrMMD ( μ ∥ π ) = ( 1 + λ ) ∥ ( T π + λ I ) − 1 T π ) 1 / 2 ( d μ d π − 1 ) ∥ L 2 ( π ) 2 ≤ ( 1 + λ ) ∥ d μ d π − 1 ∥ L 2 ( π ) 2 , \begin{aligned}
\operatorname{DrMMD}(\mu\|\pi)=(1+\lambda)\|(\mathcal{T}_\pi+\lambda I)^{-1}\mathcal{T}_\pi)^{1/2}(\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1)\|_{L^{2}(\pi)}^{2} \\
&\le(1+\lambda)\left\lVert\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right\rVert_{L^2(\pi)}^2,
\end{aligned} DrMMD ( μ ∥ π ) = ( 1 + λ ) ∥ ( T π + λ I ) − 1 T π ) 1/2 ( d π d μ − 1 ) ∥ L 2 ( π ) 2 ≤ ( 1 + λ ) d π d μ − 1 L 2 ( π ) 2 , which is finite for any λ ≥ 0 \lambda\ge0 λ ≥ 0
lim λ → 0 DrMMD ( μ ∥ π ) = ∥ lim λ → 0 ( T π + λ I ) − 1 T π ) 1 / 2 ( d μ d π − 1 ) ∥ L 2 ( π ) 2 = ∥ d μ d π − 1 ∥ L 2 ( π ) 2 = χ 2 ( μ ∥ π ) . \begin{aligned}
\lim_{\lambda\to0}\operatorname{DrMMD}(\mu\Vert\pi)
= \left\lVert\lim_{\lambda\to0}\left(\mathcal{T}_{\pi}+\lambda\mathcal{I}\right)^{-1}\mathcal{T}_{\pi})^{1/2}\left(\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right)\right\rVert_{L^2(\pi)}^2 \\
&= \left\lVert\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right\rVert_{L^2(\pi)}^2
= \chi^2(\mu\Vert\pi).
\end{aligned} λ → 0 lim DrMMD ( μ ∥ π ) = λ → 0 lim ( T π + λ I ) − 1 T π ) 1/2 ( d π d μ − 1 ) L 2 ( π ) 2 = d π d μ − 1 L 2 ( π ) 2 = χ 2 ( μ ∥ π ) . From (4.2), we have that,
DrMMD ( μ ∥ π ) = 1 + λ ∥ ( Σ π + λ Id ) − 1 2 ( m μ − m π ) ∥ H ≤ 1 + λ ∥ ( Σ π + λ Id ) − 1 2 ∥ o p ∥ m μ − m π ∥ H ≤ 1 + λ λ × MMD ( μ , π ) . (83) \begin{aligned}
\sqrt{\operatorname{DrMMD}(\mu\Vert\pi)}
= \sqrt{1+\lambda}\left\lVert(\Sigma_{\pi}+\lambda\operatorname{Id})^{-\frac{1}{2}}(m_{\mu}-m_{\pi})\right\rVert_{\mathcal{H}} \\
&\le\sqrt{1+\lambda}\left\lVert(\Sigma_{\pi}+\lambda\operatorname{Id})^{-\frac{1}{2}}\right\rVert_{\mathrm{op}}\left\lVert m_{\mu}-m_{\pi}\right\rVert_{\mathcal{H}}
\le\sqrt{\frac{1+\lambda}{\lambda}}\times\operatorname{MMD}(\mu,\pi). \tag*{(83)}
\end{aligned} DrMMD ( μ ∥ π ) = 1 + λ ( Σ π + λ Id ) − 2 1 ( m μ − m π ) H ≤ 1 + λ ( Σ π + λ Id ) − 2 1 op ∥ m μ − m π ∥ H ≤ λ 1 + λ × MMD ( μ , π ) . (83) where the last inequality follows by noticing that Σ π = ι π ∗ ι π \Sigma_{\pi}=\iota_{\pi}^{*}\iota_{\pi} Σ π = ι π ∗ ι π T π = ι π ι π ∗ \mathcal{T}_{\pi}=\iota_{\pi}\iota_{\pi}^{*} T π = ι π ι π ∗ ( Σ π + λ Id ) − 1 (\Sigma_{\pi}+\lambda\operatorname{Id})^{-1} ( Σ π + λ Id ) − 1 ( ϱ i + λ ) − 1 (\varrho_i+\lambda)^{-1} ( ϱ i + λ ) − 1 1 λ \frac{1}{\lambda} λ 1 DrMMD ( μ ∥ π ) ≤ 1 + λ λ MMD 2 ( μ ∥ π ) \operatorname{DrMMD}(\mu\Vert\pi)\le\frac{1+\lambda}{\lambda}\operatorname{MMD}^2(\mu\Vert\pi) DrMMD ( μ ∥ π ) ≤ λ 1 + λ MMD 2 ( μ ∥ π )
On the other hand, using Lemma A.10 from (Hagrass et al., 2024a), we have
MMD ( μ ∥ π ) = ∥ m μ − m π ∥ H ≤ ∥ ( Σ π + λ Id ) 1 2 ∥ o p ∥ ( Σ π + λ Id ) − 1 2 ( m μ − m π ) ∥ H = 1 1 + λ ∥ ( Σ π + λ Id ) 1 2 ∥ o p DrMMD ( μ ∥ π ) ≤ K + λ 1 + λ DrMMD ( μ ∥ π ) . (84) \begin{aligned}
\operatorname{MMD}(\mu\Vert\pi)=\left\lVert m_{\mu}-m_{\pi}\right\rVert_{\mathcal{H}}
\le\left\lVert(\Sigma_{\pi}+\lambda\operatorname{Id})^{\frac{1}{2}}\right\rVert_{\mathrm{op}}\left\lVert(\Sigma_{\pi}+\lambda\operatorname{Id})^{-\frac{1}{2}}(m_{\mu}-m_{\pi})\right\rVert_{\mathcal{H}} \\
&= \sqrt{\frac{1}{1+\lambda}}\left\lVert(\Sigma_{\pi}+\lambda\operatorname{Id})^{\frac{1}{2}}\right\rVert_{\mathrm{op}}\sqrt{\operatorname{DrMMD}(\mu\Vert\pi)}
\le\sqrt{\frac{K+\lambda}{1+\lambda}}\sqrt{\operatorname{DrMMD}(\mu\Vert\pi)}. \tag*{(84)}
\end{aligned} MMD ( μ ∥ π ) = ∥ m μ − m π ∥ H ≤ ( Σ π + λ Id ) 2 1 op ( Σ π + λ Id ) − 2 1 ( m μ − m π ) H = 1 + λ 1 ( Σ π + λ Id ) 2 1 op DrMMD ( μ ∥ π ) ≤ 1 + λ K + λ DrMMD ( μ ∥ π ) . (84) where K K K k k k Σ π \Sigma_{\pi} Σ π
1 + λ K + λ MMD 2 ( μ ∥ π ) ≤ DrMMD ( μ ∥ π ) ≤ 1 + λ λ MMD 2 ( μ ∥ π ) . \frac{1+\lambda}{K+\lambda}\operatorname{MMD}^2(\mu\Vert\pi)
\le\operatorname{DrMMD}(\mu\Vert\pi)
\le\frac{1+\lambda}{\lambda}\operatorname{MMD}^2(\mu\Vert\pi). K + λ 1 + λ MMD 2 ( μ ∥ π ) ≤ DrMMD ( μ ∥ π ) ≤ λ 1 + λ MMD 2 ( μ ∥ π ) . Therefore, lim λ → ∞ DrMMD ( μ ∥ π ) = MMD 2 ( μ ∥ π ) \lim_{\lambda\to\infty} \operatorname{DrMMD}(\mu\mathbin{\|} \pi) = \operatorname{MMD}^{2}(\mu\mathbin{\|} \pi) lim λ → ∞ DrMMD ( μ ∥ π ) = MMD 2 ( μ ∥ π )
Proof of the topology proposition
To show that DrMMD is a probability divergence, we need to show that DrMMD enjoys non-negativity and definiteness. It is easy to see that DrMMD ( μ ∥ π ) \operatorname{DrMMD}(\mu\mathbin{\|} \pi) DrMMD ( μ ∥ π ) DrMMD ( μ ∥ π ) = 0 \operatorname{DrMMD}(\mu\mathbin{\|} \pi) = 0 DrMMD ( μ ∥ π ) = 0 μ = π \mu= \pi μ = π DrMMD ( μ ∥ π ) = 0 \operatorname{DrMMD}(\mu\mathbin{\|} \pi) = 0 DrMMD ( μ ∥ π ) = 0 ∥ ( Σ π + λ Id ) − 1 / 2 ( m μ − m π ) ∥ H 2 = 0 \left\|(\Sigma_{\pi} + \lambda\operatorname{Id})^{-1/2}(m_{\mu} - m_{\pi})\right\|_{\mathcal{H}}^{2} = 0 ( Σ π + λ Id ) − 1/2 ( m μ − m π ) H 2 = 0 ( Σ π + λ Id ) − 1 / 2 (\Sigma_{\pi} + \lambda\operatorname{Id})^{-1/2} ( Σ π + λ Id ) − 1/2 m μ = m π m_{\mu} = m_{\pi} m μ = m π μ = π \mu= \pi μ = π k k k c 0 c_{0} c 0 μ = π \mu= \pi μ = π DrMMD ( μ ∥ π ) = 0 \operatorname{DrMMD}(\mu\mathbin{\|} \pi) = 0 DrMMD ( μ ∥ π ) = 0
Then we prove that DrMMD metrizes weak convergence. For the first direction, we know from (B.5) that DrMMD ( μ n ∥ π ) ≤ 1 + λ λ MMD 2 ( μ n ∥ π ) \operatorname{DrMMD}(\mu_n \mathbin{\|} \pi) \leq\frac{1+\lambda}{\lambda}\operatorname{MMD}^{2}(\mu_n \mathbin{\|} \pi) DrMMD ( μ n ∥ π ) ≤ λ 1 + λ MMD 2 ( μ n ∥ π ) MMD 2 ( μ n ∥ π ) → 0 \operatorname{MMD}^{2}(\mu_n \mathbin{\|} \pi) \to0 MMD 2 ( μ n ∥ π ) → 0 μ n \mu_n μ n π \pi π DrMMD ( μ n ∥ π ) → 0 \operatorname{DrMMD}(\mu_n \mathbin{\|} \pi) \to0 DrMMD ( μ n ∥ π ) → 0 MMD 2 ( μ ∥ π ) ≤ κ + λ 1 + λ DrMMD ( μ ∥ π ) \operatorname{MMD}^{2}(\mu\mathbin{\|} \pi) \leq\frac{\kappa+\lambda}{1+\lambda}\operatorname{DrMMD}(\mu\mathbin{\|} \pi) MMD 2 ( μ ∥ π ) ≤ 1 + λ κ + λ DrMMD ( μ ∥ π ) DrMMD ( μ n ∥ π ) → 0 \operatorname{DrMMD}(\mu_n \mathbin{\|} \pi) \to0 DrMMD ( μ n ∥ π ) → 0 MMD ( μ n ∥ π ) → 0 \operatorname{MMD}(\mu_n \mathbin{\|} \pi) \to0 MMD ( μ n ∥ π ) → 0 μ n \mu_n μ n π \pi π k k k
Proof of the DrMMD gradient-flow proposition
In order to show that F DrMMD ( ⋅ ) = DrMMD ( ⋅ ∥ π ) \mathcal{F}_{\operatorname{DrMMD}}(\cdot) = \operatorname{DrMMD}(\cdot\mathbin{\|} \pi) F DrMMD ( ⋅ ) = DrMMD ( ⋅ ∥ π ) ( 1 + λ ) ∇ h μ , π ∗ (1+\lambda)\nabla h^{*}_{\mu,\pi} ( 1 + λ ) ∇ h μ , π ∗ F DrMMD \mathcal{F}_{\operatorname{DrMMD}} F DrMMD μ \mu μ 1 μ ∈ P 2 ( R d ) \mu\in\mathcal{P}_2(\mathbb{R}^d) μ ∈ P 2 ( R d ) ϕ ∈ C c ∞ ( R d ) \phi\in C_c^\infty(\mathbb{R}^d) ϕ ∈ C c ∞ ( R d )
DrMMD ( ( Id + ∇ ϕ ) # μ ∥ π ) − DrMMD ( μ ∥ π ) ≥ ( 1 + λ ) ∫ ∇ ϕ ( x ) T ∇ h μ , π ∗ ( x ) d μ ( x ) + o ( ∥ ∇ ϕ ∥ L 2 ( μ ) ) . (85) \operatorname{DrMMD}((\operatorname{Id}+\nabla\phi)_{\#}\mu\mathbin\Vert\pi)-\operatorname{DrMMD}(\mu\mathbin\Vert\pi)\geq(1+\lambda)\int\nabla\phi(x)^{\mathsf{T}}\nabla h_{\mu,\pi}^{*}(x)\,\mathrm{d}\mu(x)+o\left(\lVert\nabla\phi\rVert_{L^2(\mu)}\right). \tag*{(85)} DrMMD (( Id + ∇ ϕ ) # μ ∥ π ) − DrMMD ( μ ∥ π ) ≥ ( 1 + λ ) ∫ ∇ ϕ ( x ) T ∇ h μ , π ∗ ( x ) d μ ( x ) + o ( ∥ ∇ ϕ ∥ L 2 ( μ ) ) . (85) Define ρ t = ( Id + t ∇ ϕ ) # μ \rho_t=(\operatorname{Id}+t\nabla\phi)_{\#}\mu ρ t = ( Id + t ∇ ϕ ) # μ φ t : R d → R d \varphi_t:\mathbb{R}^d\to\mathbb{R}^d φ t : R d → R d x ↦ x + t ∇ ϕ ( x ) x\mapsto x+t\nabla\phi(x) x ↦ x + t ∇ ϕ ( x ) g ( t ) = DrMMD ( ρ t ∥ π ) g(t)=\operatorname{DrMMD}(\rho_t\mathbin\Vert\pi) g ( t ) = DrMMD ( ρ t ∥ π ) g ( t ) g(t) g ( t ) t t t
d d t ∣ t = 0 g ( t ) = ( 1 + λ ) ∫ ∇ ϕ ( x ) T ∇ h μ , π ∗ ( x ) d μ ( x ) . \left.\frac{\mathrm{d}}{\mathrm{d}t}\right|_{t=0}g(t)=(1+\lambda)\int\nabla\phi(x)^{\mathsf{T}}\nabla h_{\mu,\pi}^{*}(x)\,\mathrm{d}\mu(x). d t d t = 0 g ( t ) = ( 1 + λ ) ∫ ∇ ϕ ( x ) T ∇ h μ , π ∗ ( x ) d μ ( x ) . Since t ↦ g ( t ) t\mapsto g(t) t ↦ g ( t ) 0 < κ < 1 0<\kappa<1 0 < κ < 1
DrMMD ( ( Id + ∇ ϕ ) # μ ∥ π ) − DrMMD ( μ ∥ π ) = g ( 1 ) − g ( 0 ) = d d t ∣ t = 0 g ( t ) + d 2 d t 2 ∣ t = κ g ( t ) . \begin{aligned}
\operatorname{DrMMD}((\operatorname{Id}+\nabla\phi)_{\#}\mu\mathbin\Vert\pi)-\operatorname{DrMMD}(\mu\mathbin\Vert\pi)=g(1)-g(0) \\
&=\left.\frac{\mathrm{d}}{\mathrm{d}t}\right|_{t=0}g(t)+\left.\frac{\mathrm{d}^2}{\mathrm{d}t^2}\right|_{t=\kappa}g(t).
\end{aligned} DrMMD (( Id + ∇ ϕ ) # μ ∥ π ) − DrMMD ( μ ∥ π ) = g ( 1 ) − g ( 0 ) = d t d t = 0 g ( t ) + d t 2 d 2 t = κ g ( t ) . Therefore, to prove (B.7), the goal is to prove that d 2 d t 2 ∣ t = κ g ( t ) ≥ o ( ∥ ∇ ϕ ∥ L 2 ( μ ) ) \left.\frac{\mathrm{d}^2}{\mathrm{d}t^2}\right|_{t=\kappa}g(t)\geq o\left(\lVert\nabla\phi\rVert_{L^2(\mu)}\right) d t 2 d 2 t = κ g ( t ) ≥ o ( ∥ ∇ ϕ ∥ L 2 ( μ ) ) t ↦ d d t g ( t ) t\mapsto\frac{\mathrm{d}}{\mathrm{d}t}g(t) t ↦ d t d g ( t )
d 2 d t 2 DrMMD ( ρ t ∥ π ) = 2 ( 1 + λ ) ∬ ∇ ϕ ( x ) T ∇ 1 ∇ 2 k ~ ( φ t ( x ) , φ t ( y ) ) ∇ ϕ ( y ) d μ ( x ) d μ ( y ) + 2 ( 1 + λ ) ∫ ∇ ϕ ( x ) T ( ∫ H 1 k ~ ( φ t ( x ) , φ t ( y ) ) d μ ( y ) − ∫ H 1 k ~ ( φ t ( x ) , y ) d π ( y ) ) ∇ ϕ ( x ) d μ ( x ) , \begin{aligned}
\frac{\mathrm{d}^2}{\mathrm{d}t^2}\operatorname{DrMMD}(\rho_t\mathbin\Vert\pi)=2(1+\lambda)\iint\nabla\phi(x)^{\mathsf{T}}\nabla_1\nabla_2\widetilde{k}(\varphi_t(x),\varphi_t(y))\nabla\phi(y)\,\mathrm{d}\mu(x)\mathrm{d}\mu(y) \\
&+2(1+\lambda)\int\nabla\phi(x)^{\mathsf{T}}\left(\int\mathbf{H}_1\widetilde{k}(\varphi_t(x),\varphi_t(y))\,\mathrm{d}\mu(y)-\int\mathbf{H}_1\widetilde{k}(\varphi_t(x),y)\,\mathrm{d}\pi(y)\right)\nabla\phi(x)\,\mathrm{d}\mu(x),
\end{aligned} d t 2 d 2 DrMMD ( ρ t ∥ π ) = 2 ( 1 + λ ) ∬ ∇ ϕ ( x ) T ∇ 1 ∇ 2 k ( φ t ( x ) , φ t ( y )) ∇ ϕ ( y ) d μ ( x ) d μ ( y ) + 2 ( 1 + λ ) ∫ ∇ ϕ ( x ) T ( ∫ H 1 k ( φ t ( x ) , φ t ( y )) d μ ( y ) − ∫ H 1 k ( φ t ( x ) , y ) d π ( y ) ) ∇ ϕ ( x ) d μ ( x ) , where k ~ \widetilde{k} k H ~ \widetilde{\mathcal{H}} H
∣ 2 ( 1 + λ ) ∬ ∇ ϕ ( x ) T ∇ 1 ∇ 2 k ~ ( φ t ( x ) , φ t ( y ) ) ∇ ϕ ( y ) d μ ( x ) d μ ( y ) ∣ \left|2(1+\lambda)\iint\nabla\phi(x)^{\mathsf{T}}\nabla_1\nabla_2\widetilde{k}(\varphi_t(x),\varphi_t(y))\nabla\phi(y)\,\mathrm{d}\mu(x)\mathrm{d}\mu(y)\right| 2 ( 1 + λ ) ∬ ∇ ϕ ( x ) T ∇ 1 ∇ 2 k ( φ t ( x ) , φ t ( y )) ∇ ϕ ( y ) d μ ( x ) d μ ( y ) ≤ 2 ( 1 + λ ) ∬ ∥ ∇ ϕ ( x ) ∥ ∥ ∇ 1 ∇ 2 k ~ ( φ t ( x ) , φ t ( y ) ) ∥ F ∥ ∇ ϕ ( y ) ∥ d μ ( x ) d μ ( y ) \leq2(1+\lambda)\iint\lVert\nabla\phi(x)\rVert\left\lVert\nabla_1\nabla_2\widetilde{k}(\varphi_t(x),\varphi_t(y))\right\rVert_F\lVert\nabla\phi(y)\rVert\,\mathrm{d}\mu(x)\mathrm{d}\mu(y) ≤ 2 ( 1 + λ ) ∬ ∥ ∇ ϕ ( x )∥ ∇ 1 ∇ 2 k ( φ t ( x ) , φ t ( y )) F ∥ ∇ ϕ ( y )∥ d μ ( x ) d μ ( y ) ≤ 2 ( 1 + λ ) K 1 d λ ( ∫ ∥ ∇ φ ( y ) ∥ d μ ( y ) ) 2 ≤ 2 ( 1 + λ ) K 1 d λ ∥ ∇ φ ∥ L 2 ( μ ) 2 . \le2(1+\lambda)\frac{K_{1d}}{\lambda}\left(\int\|\nabla\varphi(y)\|\,\mathrm{d}\mu(y)\right)^2 \le2(1+\lambda)\frac{K_{1d}}{\lambda}\|\nabla\varphi\|_{L^2(\mu)}^2. ≤ 2 ( 1 + λ ) λ K 1 d ( ∫ ∥∇ φ ( y ) ∥ d μ ( y ) ) 2 ≤ 2 ( 1 + λ ) λ K 1 d ∥∇ φ ∥ L 2 ( μ ) 2 . Using Theorem B.3 again, the second term can be upper-bounded by
∣ ∫ ∇ φ ( x ) T ( ∫ H 1 k ~ ( φ t ( x ) , φ t ( y ) ) d μ ( y ) − ∫ H 1 k ~ ( φ t ( x ) , y ) d π ( y ) ) ∇ φ ( x ) d μ ( x ) ∣ ≤ ∣ ∫ ∇ φ ( x ) T ( ∫ H 1 k ~ ( φ t ( x ) , φ t ( y ) ) d μ ( y ) ) ∇ φ ( x ) d μ ( x ) ∣ + ∣ ∫ ∇ φ ( x ) T ( ∫ H 1 k ~ ( φ t ( x ) , y ) d π ( y ) ) ∇ φ ( x ) d μ ( x ) ∣ ≤ ∫ ∥ ∇ φ ( x ) ∥ ( ∫ ∥ H 1 k ~ ( φ t ( x ) , φ t ( y ) ) ∥ F d μ ( y ) ) ∥ ∇ φ ( x ) ∥ d μ ( x ) + ∫ ∥ ∇ φ ( x ) ∥ ( ∫ ∥ H 1 k ~ ( φ t ( x ) , y ) ∥ F d π ( y ) ) ∥ ∇ φ ( x ) ∥ d μ ( x ) ≤ 2 K K 2 d λ ∥ ∇ φ ∥ L 2 ( μ ) 2 . \begin{aligned}
\left|\int\nabla\varphi(x)^{\mathsf{T}}\left(\int\mathrm{H}_1\widetilde{k}\left(\varphi_t(x),\varphi_t(y)\right)\,\mathrm{d}\mu(y)-\int\mathrm{H}_1\widetilde{k}\left(\varphi_t(x),y\right)\,\mathrm{d}\pi(y)\right)\nabla\varphi(x)\,\mathrm{d}\mu(x)\right| \\
\le\left|\int\nabla\varphi(x)^{\mathsf{T}}\left(\int\mathrm{H}_1\widetilde{k}\left(\varphi_t(x),\varphi_t(y)\right)\,\mathrm{d}\mu(y)\right)\nabla\varphi(x)\,\mathrm{d}\mu(x)\right| \\
\quad+ \left|\int\nabla\varphi(x)^{\mathsf{T}}\left(\int\mathrm{H}_1\widetilde{k}\left(\varphi_t(x),y\right)\,\mathrm{d}\pi(y)\right)\nabla\varphi(x)\,\mathrm{d}\mu(x)\right| \\
\le\int\|\nabla\varphi(x)\|\left(\int\|\mathrm{H}_1\widetilde{k}\left(\varphi_t(x),\varphi_t(y)\right)\|_F\,\mathrm{d}\mu(y)\right)\|\nabla\varphi(x)\|\,\mathrm{d}\mu(x) \\
\quad+ \int\|\nabla\varphi(x)\|\left(\int\|\mathrm{H}_1\widetilde{k}\left(\varphi_t(x),y\right)\|_F\,\mathrm{d}\pi(y)\right)\|\nabla\varphi(x)\|\,\mathrm{d}\mu(x) \\
\le2\frac{\sqrt{K}K_{2d}}{\lambda}\|\nabla\varphi\|_{L^2(\mu)}^2.
\end{aligned} ∫ ∇ φ ( x ) T ( ∫ H 1 k ( φ t ( x ) , φ t ( y ) ) d μ ( y ) − ∫ H 1 k ( φ t ( x ) , y ) d π ( y ) ) ∇ φ ( x ) d μ ( x ) ≤ ∫ ∇ φ ( x ) T ( ∫ H 1 k ( φ t ( x ) , φ t ( y ) ) d μ ( y ) ) ∇ φ ( x ) d μ ( x ) + ∫ ∇ φ ( x ) T ( ∫ H 1 k ( φ t ( x ) , y ) d π ( y ) ) ∇ φ ( x ) d μ ( x ) ≤ ∫ ∥∇ φ ( x ) ∥ ( ∫ ∥ H 1 k ( φ t ( x ) , φ t ( y ) ) ∥ F d μ ( y ) ) ∥∇ φ ( x ) ∥ d μ ( x ) + ∫ ∥∇ φ ( x ) ∥ ( ∫ ∥ H 1 k ( φ t ( x ) , y ) ∥ F d π ( y ) ) ∥∇ φ ( x ) ∥ d μ ( x ) ≤ 2 λ K K 2 d ∥∇ φ ∥ L 2 ( μ ) 2 . Combining the above two inequalities to have
∣ d 2 d t 2 DrMMD ( ρ t ∥ π ) ∣ ≤ 2 ( 1 + λ ) K 1 d λ ∥ ∇ φ ∥ L 2 ( μ ) 2 + 4 ( 1 + λ ) K K 2 d λ ∥ ∇ φ ∥ L 2 ( μ ) 2 . (86) \left|\frac{d^2}{dt^2}\operatorname{DrMMD}(\rho_t\Vert\pi)\right| \le2(1+\lambda)\frac{K_{1d}}{\lambda}\|\nabla\varphi\|_{L^2(\mu)}^2 + 4(1+\lambda)\frac{\sqrt{K}K_{2d}}{\lambda}\|\nabla\varphi\|_{L^2(\mu)}^2. \tag*{(86)} d t 2 d 2 DrMMD ( ρ t ∥ π ) ≤ 2 ( 1 + λ ) λ K 1 d ∥∇ φ ∥ L 2 ( μ ) 2 + 4 ( 1 + λ ) λ K K 2 d ∥∇ φ ∥ L 2 ( μ ) 2 . (86) Therefore, we have d 2 d t 2 g ( t ) ∣ t = k = O ( ∥ ∇ φ ∥ L 2 ( μ ) 2 ) = o ( ∥ ∇ φ ∥ L 2 ( μ ) ) \left.\frac{d^2}{dt^2}g(t)\right|_{t=k}=\mathcal{O}(\|\nabla\varphi\|_{L^2(\mu)}^2)=o(\|\nabla\varphi\|_{L^2(\mu)}) d t 2 d 2 g ( t ) t = k = O ( ∥∇ φ ∥ L 2 ( μ ) 2 ) = o ( ∥∇ φ ∥ L 2 ( μ ) ) ∥ ∇ φ ∥ L 2 ( μ ) → 0 \|\nabla\varphi\|_{L^2(\mu)}\to0 ∥∇ φ ∥ L 2 ( μ ) → 0 ( 1 + λ ) ∇ h μ , π ∗ ( x ) (1+\lambda)\nabla h_{\mu,\pi}^*(x) ( 1 + λ ) ∇ h μ , π ∗ ( x ) F D r M M D \mathcal{F}_{\mathrm{DrMMD}} F DrMMD μ \mu μ ( μ t ) t ≥ 0 (\mu_t)_{t\geq0} ( μ t ) t ≥ 0
∂ t μ t − ( 1 + λ ) ∇ ⋅ ( μ t ∇ h μ t , π ∗ ) = 0 , \partial_t\mu_t-(1+\lambda)\nabla\cdot\left(\mu_t\nabla h_{\mu_t,\pi}^*\right)=0, ∂ t μ t − ( 1 + λ ) ∇ ⋅ ( μ t ∇ h μ t , π ∗ ) = 0 , and such ( μ t ) t ≥ 0 (\mu_t)_{t\geq0} ( μ t ) t ≥ 0
Next, we are going to prove uniqueness. (B.8) indicates that F D r M M D \mathcal{F}_{\mathrm{DrMMD}} F DrMMD − ( 1 + λ ) 4 K K 2 d + 2 K 1 d λ -(1+\lambda)\frac{4\sqrt{K}K_{2d}+2K_{1d}}{\lambda} − ( 1 + λ ) λ 4 K K 2 d + 2 K 1 d ( μ t ) t (\mu_t)_t ( μ t ) t (18) is proved in (B.8).
Proof of the geometric proposition
Given μ ∈ P 2 ( R d ) \mu\in\mathcal{P}_2(\mathbb{R}^d) μ ∈ P 2 ( R d ) ϕ ∈ C c ∞ ( R d ) \phi\in C^\infty_c(\mathbb{R}^d) ϕ ∈ C c ∞ ( R d ) ( ρ t ) 0 ≤ t ≤ 1 (\rho_t)_{0\leq t\leq1} ( ρ t ) 0 ≤ t ≤ 1 μ \mu μ ( Id + ∇ ϕ ) # μ (\operatorname{Id}+\nabla\phi)_{\#}\mu ( Id + ∇ ϕ ) # μ ρ t = ( Id + t ∇ ϕ ) # μ \rho_t=(\operatorname{Id}+t\nabla\phi)_{\#}\mu ρ t = ( Id + t ∇ ϕ ) # μ ( ρ t ) 0 ≤ t ≤ 1 (\rho_t)_{0\leq t\leq1} ( ρ t ) 0 ≤ t ≤ 1 φ t : R d → R d \varphi_t:\mathbb{R}^d\to\mathbb{R}^d φ t : R d → R d x ↦ x + t ∇ ϕ ( x ) x\mapsto x+t\nabla\phi(x) x ↦ x + t ∇ ϕ ( x ) m = 2 m=2 m = 2
d 2 d t 2 ∣ t = 0 χ 2 ( ρ t ∥ π ) = ∫ d μ d π ( x ) ( ∇ V ( x ) T ∇ ϕ ( x ) − ∇ ⋅ ∇ ϕ ( x ) ) 2 d μ ( x ) + ∫ d μ d π ( x ) ∇ ϕ ( x ) T H V ( x ) ∇ ϕ ( x ) d μ ( x ) + ∫ d μ d π ( x ) ∥ H ϕ ( x ) ∥ F 2 d μ ( x ) . (87) \left.\frac{d^2}{dt^2}\right|_{t=0}\chi^2(\rho_t\Vert\pi)=\int\frac{d\mu}{d\pi}(x)\left(\nabla V(x)^\mathsf{T}\nabla\phi(x)-\nabla\cdot\nabla\phi(x)\right)^2d\mu(x)
+\int\frac{d\mu}{d\pi}(x)\nabla\phi(x)^\mathsf{T}HV(x)\nabla\phi(x)d\mu(x)
+\int\frac{d\mu}{d\pi}(x)\lVert H\phi(x)\rVert_F^2d\mu(x). \tag*{(87)} d t 2 d 2 t = 0 χ 2 ( ρ t ∥ π ) = ∫ d π d μ ( x ) ( ∇ V ( x ) T ∇ ϕ ( x ) − ∇ ⋅ ∇ ϕ ( x ) ) 2 d μ ( x ) + ∫ d π d μ ( x ) ∇ ϕ ( x ) T H V ( x ) ∇ ϕ ( x ) d μ ( x ) + ∫ d π d μ ( x ) ∥ H ϕ ( x ) ∥ F 2 d μ ( x ) . (87) V V V ∇ V \nabla V ∇ V R d \mathbb{R}^d R d R d \mathbb{R}^d R d ϕ ∈ C c ∞ ( R d ) \phi\in C^\infty_c(\mathbb{R}^d) ϕ ∈ C c ∞ ( R d ) x ↦ ∇ V ( x ) T ∇ ϕ ( x ) − ∇ ⋅ ∇ ϕ ( x ) x\mapsto\nabla V(x)^\mathsf{T}\nabla\phi(x)-\nabla\cdot\nabla\phi(x) x ↦ ∇ V ( x ) T ∇ ϕ ( x ) − ∇ ⋅ ∇ ϕ ( x ) x ↦ ∇ ϕ ( x ) T H V ( x ) ∇ ϕ ( x ) x\mapsto\nabla\phi(x)^\mathsf{T}HV(x)\nabla\phi(x) x ↦ ∇ ϕ ( x ) T H V ( x ) ∇ ϕ ( x ) x ↦ ∥ H ϕ ( x ) ∥ F 2 x\mapsto\lVert H\phi(x)\rVert_F^2 x ↦ ∥ H ϕ ( x ) ∥ F 2
Since d μ d π − 1 ∈ H ⊂ L 2 ( π ) \frac{d\mu}{d\pi}-1\in\mathcal{H}\subset L^2(\pi) d π d μ − 1 ∈ H ⊂ L 2 ( π ) ∫ d μ d π ( x ) d μ ( x ) = ∥ d μ d π − 1 ∥ L 2 ( π ) 2 + 1 < ∞ \int\frac{d\mu}{d\pi}(x)d\mu(x)=\left\lVert\frac{d\mu}{d\pi}-1\right\rVert_{L^2(\pi)}^2+1<\infty ∫ d π d μ ( x ) d μ ( x ) = d π d μ − 1 L 2 ( π ) 2 + 1 < ∞
d 2 d t 2 ∣ t = 0 χ 2 ( ρ t ∥ π ) < ∞ . (88) \left.\frac{d^2}{dt^2}\right|_{t=0}\chi^2(\rho_t\Vert\pi)<\infty. \tag*{(88)} d t 2 d 2 t = 0 χ 2 ( ρ t ∥ π ) < ∞. (88) Next, from Theorem B.9 we know that d 2 d t 2 ∣ t = 0 χ 2 ( ρ t ∥ π ) \left.\frac{d^2}{dt^2}\right|_{t=0}\chi^2(\rho_t\Vert\pi) d t 2 d 2 t = 0 χ 2 ( ρ t ∥ π )
d 2 d t 2 ∣ t = 0 χ 2 ( ρ t ∥ π ) = 2 ∫ ( ∇ ⋅ ( ∇ ϕ ( x ) μ ( x ) ) 1 π ( x ) ) 2 π ( x ) d x + 2 ∫ ∇ ϕ ( x ) T H ( μ ( x ) π ( x ) ) ∇ ϕ ( x ) μ ( x ) d x . (89) \left.\frac{d^2}{dt^2}\right|_{t=0}\chi^2(\rho_t\Vert\pi)=2\int\left(\nabla\cdot(\nabla\phi(x)\mu(x))\frac{1}{\pi(x)}\right)^2\pi(x)dx+2\int\nabla\phi(x)^\mathsf{T}H\left(\frac{\mu(x)}{\pi(x)}\right)\nabla\phi(x)\mu(x)dx. \tag*{(89)} d t 2 d 2 t = 0 χ 2 ( ρ t ∥ π ) = 2 ∫ ( ∇ ⋅ ( ∇ ϕ ( x ) μ ( x )) π ( x ) 1 ) 2 π ( x ) d x + 2 ∫ ∇ ϕ ( x ) T H ( π ( x ) μ ( x ) ) ∇ ϕ ( x ) μ ( x ) d x . (89) Recall from Theorem B.6 that
d 2 d t 2 ∣ t = 0 DrMMD ( ρ t ∥ π ) = 2 ( 1 + λ ) ∬ ∇ ϕ ( x ) T ∇ 1 ∇ 2 k ~ ( x , y ) ∇ ϕ ( y ) d μ ( x ) d μ ( y ) + 2 ( 1 + λ ) ∫ ∇ ϕ ( x ) T ( ∫ H 1 k ~ ( x , y ) d μ ( y ) − ∫ H 1 k ~ ( x , y ) d π ( y ) ) ∇ ϕ ( x ) d μ ( x ) . (90) \left.\frac{d^2}{dt^2}\right|_{t=0}\operatorname{DrMMD}(\rho_t\Vert\pi)
=2(1+\lambda)\iint\nabla\phi(x)^{\mathsf T}\nabla_1\nabla_2\tilde{k}(x,y)\nabla\phi(y)\,d\mu(x)d\mu(y)
+2(1+\lambda)\int\nabla\phi(x)^{\mathsf T}\left(\int\mathbf{H}_1\tilde{k}(x,y)\,d\mu(y)-\int\mathbf{H}_1\tilde{k}(x,y)\,d\pi(y)\right)\nabla\phi(x)\,d\mu(x). \tag*{(90)} d t 2 d 2 t = 0 DrMMD ( ρ t ∥ π ) = 2 ( 1 + λ ) ∬ ∇ ϕ ( x ) T ∇ 1 ∇ 2 k ~ ( x , y ) ∇ ϕ ( y ) d μ ( x ) d μ ( y ) + 2 ( 1 + λ ) ∫ ∇ ϕ ( x ) T ( ∫ H 1 k ~ ( x , y ) d μ ( y ) − ∫ H 1 k ~ ( x , y ) d π ( y ) ) ∇ ϕ ( x ) d μ ( x ) . (90) To prove (19) , our aim then is to compare and bound the difference of d 2 d t 2 ∣ t = 0 χ 2 ( ρ t ∥ π ) \left.\frac{d^2}{dt^2}\right|_{t=0}\chi^2(\rho_t\Vert\pi) d t 2 d 2 t = 0 χ 2 ( ρ t ∥ π ) 1 1 + λ d 2 d t 2 ∣ t = 0 DrMMD ( ρ t ∥ π ) \left.\frac{1}{1+\lambda}\frac{d^2}{dt^2}\right|_{t=0}\operatorname{DrMMD}(\rho_t\Vert\pi) 1 + λ 1 d t 2 d 2 t = 0 DrMMD ( ρ t ∥ π )
The first term of (B.11) can be rewritten as
∫ ( ∇ ⋅ ( ∇ ϕ ( x ) μ ( x ) ) 1 π ( x ) ) 2 π ( x ) d x = ∑ i ≥ 1 ⟨ ∇ ⋅ ( ∇ ϕ μ ) 1 π , e i ⟩ L 2 ( π ) 2 = ∑ i ≥ 1 ( ∫ ∇ ⋅ ( ∇ ϕ ( x ) μ ( x ) ) e i ( x ) d x ) 2 = ∑ i ≥ 1 ( ∫ ∇ ϕ ( x ) T ∇ e i ( x ) μ ( x ) d x ) 2 , (91) \int\left(\nabla\cdot(\nabla\phi(x)\mu(x))\frac{1}{\pi(x)}\right)^2\pi(x)\,dx
=\sum_{i\geq1}\left\langle\nabla\cdot(\nabla\phi\mu)\frac{1}{\pi},e_i\right\rangle_{L^2(\pi)}^2
=\sum_{i\geq1}\left(\int\nabla\cdot(\nabla\phi(x)\mu(x))e_i(x)\,dx\right)^2
=\sum_{i\geq1}\left(\int\nabla\phi(x)^{\mathsf T}\nabla e_i(x)\mu(x)\,dx\right)^2, \tag*{(91)} ∫ ( ∇ ⋅ ( ∇ ϕ ( x ) μ ( x )) π ( x ) 1 ) 2 π ( x ) d x = i ≥ 1 ∑ ⟨ ∇ ⋅ ( ∇ ϕ μ ) π 1 , e i ⟩ L 2 ( π ) 2 = i ≥ 1 ∑ ( ∫ ∇ ⋅ ( ∇ ϕ ( x ) μ ( x )) e i ( x ) d x ) 2 = i ≥ 1 ∑ ( ∫ ∇ ϕ ( x ) T ∇ e i ( x ) μ ( x ) d x ) 2 , (91) where we use integration by parts in the last line since ϕ ∈ C c ∞ ( R d ) \phi\in C^\infty_c(\mathbb{R}^d) ϕ ∈ C c ∞ ( R d ) 1 1 + λ \frac{1}{1+\lambda} 1 + λ 1
2 ∬ ∇ ϕ ( x ) T ∇ 1 ∇ 2 k ~ ( x , y ) ∇ ϕ ( y ) d μ ( x ) d μ ( y ) = 2 ∬ ∇ ϕ ( x ) T ∇ x ∇ y ( ∑ i ≥ 1 ϱ i ϱ i + λ e i ( x ) e i ( y ) ) ∇ ϕ ( y ) d μ ( x ) d μ ( y ) = 2 ∬ ( ∇ ⋅ ( ∇ ϕ ( x ) μ ( x ) ) ) ( ∇ ⋅ ( ∇ ϕ ( y ) μ ( y ) ) ) ∑ i ≥ 1 ϱ i ϱ i + λ e i ( x ) e i ( y ) d x d y . (92) 2\iint\nabla\phi(x)^{\mathsf T}\nabla_1\nabla_2\tilde{k}(x,y)\nabla\phi(y)\,d\mu(x)d\mu(y)
=2\iint\nabla\phi(x)^{\mathsf T}\nabla_x\nabla_y\left(\sum_{i\geq1}\frac{\varrho_i}{\varrho_i+\lambda}e_i(x)e_i(y)\right)\nabla\phi(y)\,d\mu(x)d\mu(y)
=2\iint\left(\nabla\cdot(\nabla\phi(x)\mu(x))\right)\left(\nabla\cdot(\nabla\phi(y)\mu(y))\right)\sum_{i\geq1}\frac{\varrho_i}{\varrho_i+\lambda}e_i(x)e_i(y)\,dxdy. \tag*{(92)} 2 ∬ ∇ ϕ ( x ) T ∇ 1 ∇ 2 k ~ ( x , y ) ∇ ϕ ( y ) d μ ( x ) d μ ( y ) = 2 ∬ ∇ ϕ ( x ) T ∇ x ∇ y ( i ≥ 1 ∑ ϱ i + λ ϱ i e i ( x ) e i ( y ) ) ∇ ϕ ( y ) d μ ( x ) d μ ( y ) = 2 ∬ ( ∇ ⋅ ( ∇ ϕ ( x ) μ ( x )) ) ( ∇ ⋅ ( ∇ ϕ ( y ) μ ( y )) ) i ≥ 1 ∑ ϱ i + λ ϱ i e i ( x ) e i ( y ) d x d y . (92) Since π ≪ L d \pi\ll\mathcal{L}^d π ≪ L d x , y ∈ R d x,y\in\mathbb{R}^d x , y ∈ R d ϕ ∈ C c ∞ ( R d ) \phi\in C^\infty_c(\mathbb{R}^d) ϕ ∈ C c ∞ ( R d )
∑ i ≥ 1 ϱ i ϱ i + λ ∬ ∣ e i ( x ) e i ( y ) ( ∇ ⋅ ( ∇ ϕ ( x ) μ ( x ) ) ) ( ∇ ⋅ ( ∇ ϕ ( y ) μ ( y ) ) ) ∣ d x d y = ∑ i ≥ 1 ϱ i ϱ i + λ ( ∫ ∣ ( ∇ ⋅ ( ∇ ϕ ( x ) μ ( x ) ) ) e i ( x ) ∣ d x ) 2 ≤ 1 λ ∑ i ≥ 1 ϱ i ( ∫ ( ∇ ⋅ ( ∇ ϕ ( x ) μ ( x ) ) μ ( x ) ) 2 μ ( x ) d x ) ( ∫ e i ( x ) 2 μ ( x ) d x ) = 1 λ ( ∫ ( ∇ log μ ( x ) T ∇ ϕ ( x ) + ∇ ⋅ ∇ ϕ ( x ) ) 2 d μ ( x ) ) ( ∫ ∑ i ≥ 1 ϱ i e i ( x ) 2 d μ ( x ) ) ≤ K λ ( ∫ ( ∇ log μ ( x ) T ∇ ϕ ( x ) + ∇ ⋅ ∇ ϕ ( x ) ) 2 d μ ( x ) ) < + ∞ . \begin{aligned}
\sum_{i\geq1}\frac{\varrho_i}{\varrho_i+\lambda}\iint\left|e_i(x)e_i(y)\left(\nabla\cdot(\nabla\phi(x)\mu(x))\right)\left(\nabla\cdot(\nabla\phi(y)\mu(y))\right)\right|\,dx\,dy
&= \sum_{i\geq1}\frac{\varrho_i}{\varrho_i+\lambda}\left(\int\left|\left(\nabla\cdot(\nabla\phi(x)\mu(x))\right)e_i(x)\right|\,dx\right)^2\\
&\leq\frac{1}{\lambda}\sum_{i\geq1}\varrho_i\left(\int\left(\frac{\nabla\cdot(\nabla\phi(x)\mu(x))}{\mu(x)}\right)^2\mu(x)\,dx\right)\left(\int e_i(x)^2\mu(x)\,dx\right)\\
&= \frac{1}{\lambda}\left(\int\left(\nabla\log\mu(x)^{\mathsf T}\nabla\phi(x)+\nabla\cdot\nabla\phi(x)\right)^2\,d\mu(x)\right)\left(\int\sum_{i\geq1}\varrho_i e_i(x)^2\,d\mu(x)\right)\\
&\leq\frac{K}{\lambda}\left(\int\left(\nabla\log\mu(x)^{\mathsf T}\nabla\phi(x)+\nabla\cdot\nabla\phi(x)\right)^2\,d\mu(x)\right)<+\infty.
\end{aligned} i ≥ 1 ∑ ϱ i + λ ϱ i ∬ ∣ e i ( x ) e i ( y ) ( ∇ ⋅ ( ∇ ϕ ( x ) μ ( x )) ) ( ∇ ⋅ ( ∇ ϕ ( y ) μ ( y )) ) ∣ d x d y = i ≥ 1 ∑ ϱ i + λ ϱ i ( ∫ ∣ ( ∇ ⋅ ( ∇ ϕ ( x ) μ ( x )) ) e i ( x ) ∣ d x ) 2 ≤ λ 1 i ≥ 1 ∑ ϱ i ( ∫ ( μ ( x ) ∇ ⋅ ( ∇ ϕ ( x ) μ ( x )) ) 2 μ ( x ) d x ) ( ∫ e i ( x ) 2 μ ( x ) d x ) = λ 1 ( ∫ ( ∇ log μ ( x ) T ∇ ϕ ( x ) + ∇ ⋅ ∇ ϕ ( x ) ) 2 d μ ( x ) ) ( ∫ i ≥ 1 ∑ ϱ i e i ( x ) 2 d μ ( x ) ) ≤ λ K ( ∫ ( ∇ log μ ( x ) T ∇ ϕ ( x ) + ∇ ⋅ ∇ ϕ ( x ) ) 2 d μ ( x ) ) < + ∞. The first inequality uses Cauchy-Schwartz, the second inequality uses ∑ i ≥ 1 ϱ i ( e i ( x ) ) 2 = k ( x , x ) ≤ K \sum_{i\geq1}\varrho_i(e_i(x))^2=k(x,x)\leq K ∑ i ≥ 1 ϱ i ( e i ( x ) ) 2 = k ( x , x ) ≤ K x → ∇ log μ ( x ) x\to\nabla\log\mu(x) x → ∇ log μ ( x ) ϕ ∈ C c ∞ ( R d ) \phi\in C^\infty_c(\mathbb{R}^d) ϕ ∈ C c ∞ ( R d )
2 ∬ ( ∇ ⋅ ( ∇ ϕ ( x ) μ ( x ) ) ) ( ∇ ⋅ ( ∇ ϕ ( y ) μ ( y ) ) ) ∑ i ≥ 1 ϱ i ϱ i + λ e i ( x ) e i ( y ) d x d y = 2 ∑ i ≥ 1 ϱ i ϱ i + λ ∬ ( ∇ ⋅ ( ∇ ϕ ( x ) μ ( x ) ) ) ( ∇ ⋅ ( ∇ ϕ ( y ) μ ( y ) ) ) e i ( x ) e i ( y ) d x d y = 2 ∑ i ≥ 1 ϱ i ϱ i + λ ( ∬ ( ∇ ⋅ ( ∇ ϕ ( x ) μ ( x ) ) ) e i ( x ) d x ) 2 = 2 ∑ i ≥ 1 ϱ i ϱ i + λ ( ∫ ∇ ϕ ( x ) T ∇ e i ( x ) μ ( x ) d x ) 2 , \begin{aligned}
2\iint\left(\nabla\cdot(\nabla\phi(x)\mu(x))\right)\left(\nabla\cdot(\nabla\phi(y)\mu(y))\right)\sum_{i\geq1}\frac{\varrho_i}{\varrho_i+\lambda}e_i(x)e_i(y)\,dx\,dy
&=2\sum_{i\geq1}\frac{\varrho_i}{\varrho_i+\lambda}\iint\left(\nabla\cdot(\nabla\phi(x)\mu(x))\right)\left(\nabla\cdot(\nabla\phi(y)\mu(y))\right)e_i(x)e_i(y)\,dx\,dy\\
&=2\sum_{i\geq1}\frac{\varrho_i}{\varrho_i+\lambda}\left(\iint\left(\nabla\cdot(\nabla\phi(x)\mu(x))\right)e_i(x)\,dx\right)^2\\
&=2\sum_{i\geq1}\frac{\varrho_i}{\varrho_i+\lambda}\left(\int\nabla\phi(x)^{\mathsf T}\nabla e_i(x)\mu(x)\,dx\right)^2,
\end{aligned} 2 ∬ ( ∇ ⋅ ( ∇ ϕ ( x ) μ ( x )) ) ( ∇ ⋅ ( ∇ ϕ ( y ) μ ( y )) ) i ≥ 1 ∑ ϱ i + λ ϱ i e i ( x ) e i ( y ) d x d y = 2 i ≥ 1 ∑ ϱ i + λ ϱ i ∬ ( ∇ ⋅ ( ∇ ϕ ( x ) μ ( x )) ) ( ∇ ⋅ ( ∇ ϕ ( y ) μ ( y )) ) e i ( x ) e i ( y ) d x d y = 2 i ≥ 1 ∑ ϱ i + λ ϱ i ( ∬ ( ∇ ⋅ ( ∇ ϕ ( x ) μ ( x )) ) e i ( x ) d x ) 2 = 2 i ≥ 1 ∑ ϱ i + λ ϱ i ( ∫ ∇ ϕ ( x ) T ∇ e i ( x ) μ ( x ) d x ) 2 , where the last equality uses integration by parts.
So the difference between the first term of (B.11) and (B.12) rescaled by 1 1 + λ \frac{1}{1+\lambda} 1 + λ 1
2 ∣ ∑ i ≥ 1 ( ∫ ∇ ϕ ( x ) T ∇ e i ( x ) μ ( x ) d x ) 2 2|\sum_{i\geq1}(\int\nabla\phi(x)^{\mathsf T}\nabla e_i(x)\mu(x)\,dx)^2 2∣ i ≥ 1 ∑ ( ∫ ∇ ϕ ( x ) T ∇ e i ( x ) μ ( x ) d x ) 2 − ∑ i ≥ 1 ϱ i ϱ i + λ ∣ ∫ ∇ ϕ ( x ) T ∇ e i ( x ) μ ( x ) d x ∣ 2 = 2 ∑ i ≥ 1 λ ϱ i + λ ( ∫ ∇ ϕ ( x ) T ∇ e i ( x ) μ ( x ) d x ) 2 . (93) -\sum_{i\ge1}\frac{\varrho_i}{\varrho_i+\lambda}\left|\int\nabla\phi(x)^{\mathsf T}\nabla e_i(x)\mu(x)dx\right|^2
=
2\sum_{i\ge1}\frac{\lambda}{\varrho_i+\lambda}\left(\int\nabla\phi(x)^{\mathsf T}\nabla e_i(x)\mu(x)dx\right)^2. \tag*{(93)} − i ≥ 1 ∑ ϱ i + λ ϱ i ∫ ∇ ϕ ( x ) T ∇ e i ( x ) μ ( x ) d x 2 = 2 i ≥ 1 ∑ ϱ i + λ λ ( ∫ ∇ ϕ ( x ) T ∇ e i ( x ) μ ( x ) d x ) 2 . (93) Now we turn to the second term. The second term of (B.11) can be rewritten as
2 ∫ ∇ ϕ ( x ) T H ( d μ d π ( x ) ) ∇ ϕ ( x ) μ ( x ) d x = 2 ∫ ∇ ϕ ( x ) T H ( d μ d π ( x ) − 1 ) ∇ ϕ ( x ) μ ( x ) d x = 2 ∫ ∇ ϕ ( x ) T H ( ∑ i ≥ 1 ⟨ d μ d π − 1 , e i ⟩ L 2 ( π ) e i ( x ) ) ∇ ϕ ( x ) μ ( x ) d x . (94) \begin{aligned}
2\int\nabla\phi(x)^{\mathsf T}\mathbf H\left(\frac{d\mu}{d\pi}(x)\right)\nabla\phi(x)\mu(x)dx
=
2\int\nabla\phi(x)^{\mathsf T}\mathbf H\left(\frac{d\mu}{d\pi}(x)-1\right)\nabla\phi(x)\mu(x)dx \\
&=2\int\nabla\phi(x)^{\mathsf T}\mathbf H\left(\sum_{i\ge1}\left\langle\frac{d\mu}{d\pi}-1,e_i\right\rangle_{L^2(\pi)}e_i(x)\right)\nabla\phi(x)\mu(x)dx. \tag*{(94)}
\end{aligned} 2 ∫ ∇ ϕ ( x ) T H ( d π d μ ( x ) ) ∇ ϕ ( x ) μ ( x ) d x = 2 ∫ ∇ ϕ ( x ) T H ( d π d μ ( x ) − 1 ) ∇ ϕ ( x ) μ ( x ) d x = 2 ∫ ∇ ϕ ( x ) T H ( i ≥ 1 ∑ ⟨ d π d μ − 1 , e i ⟩ L 2 ( π ) e i ( x ) ) ∇ ϕ ( x ) μ ( x ) d x . (94) and the second term of (B.12) rescaled by 1 1 + λ \frac{1}{1+\lambda} 1 + λ 1
2 ∫ ∇ ϕ ( x ) T H ( ∫ H 1 k ~ ( x , y ) d μ ( y ) − ∫ H 1 k ~ ( x , y ) d π ( y ) ) ∇ ϕ ( x ) d μ ( x ) = 2 ∫ ∇ ϕ ( x ) T H ( ∫ k ~ ( x , y ) d μ ( y ) − ∫ k ~ ( x , y ) d π ( y ) ) ∇ ϕ ( x ) d μ ( x ) = 2 ∫ ∇ ϕ ( x ) T H ( ∑ i ≥ 1 ϱ i ϱ i + λ e i ( x ) ∫ e i ( y ) d ( μ − π ) ( y ) ) ∇ ϕ ( x ) d μ ( x ) = 2 ∫ ∇ ϕ ( x ) T H ( ∑ i ≥ 1 ϱ i ϱ i + λ ⟨ d μ d π − 1 , e i ⟩ L 2 ( π ) e i ( x ) ) ∇ ϕ ( x ) d μ ( x ) . (95) \begin{aligned}
2\int\nabla\phi(x)^{\mathsf T}\mathbf H\left(\int\mathbf H_1\tilde{k}(x,y)d\mu(y)-\int\mathbf H_1\tilde{k}(x,y)d\pi(y)\right)\nabla\phi(x)d\mu(x) \\
&=2\int\nabla\phi(x)^{\mathsf T}\mathbf H\left(\int\tilde{k}(x,y)d\mu(y)-\int\tilde{k}(x,y)d\pi(y)\right)\nabla\phi(x)d\mu(x) \\
&=2\int\nabla\phi(x)^{\mathsf T}\mathbf H\left(\sum_{i\ge1}\frac{\varrho_i}{\varrho_i+\lambda}e_i(x)\int e_i(y)d(\mu-\pi)(y)\right)\nabla\phi(x)d\mu(x) \\
&=2\int\nabla\phi(x)^{\mathsf T}\mathbf H\left(\sum_{i\ge1}\frac{\varrho_i}{\varrho_i+\lambda}\left\langle\frac{d\mu}{d\pi}-1,e_i\right\rangle_{L^2(\pi)}e_i(x)\right)\nabla\phi(x)d\mu(x). \tag*{(95)}
\end{aligned} 2 ∫ ∇ ϕ ( x ) T H ( ∫ H 1 k ~ ( x , y ) d μ ( y ) − ∫ H 1 k ~ ( x , y ) d π ( y ) ) ∇ ϕ ( x ) d μ ( x ) = 2 ∫ ∇ ϕ ( x ) T H ( ∫ k ~ ( x , y ) d μ ( y ) − ∫ k ~ ( x , y ) d π ( y ) ) ∇ ϕ ( x ) d μ ( x ) = 2 ∫ ∇ ϕ ( x ) T H ( i ≥ 1 ∑ ϱ i + λ ϱ i e i ( x ) ∫ e i ( y ) d ( μ − π ) ( y ) ) ∇ ϕ ( x ) d μ ( x ) = 2 ∫ ∇ ϕ ( x ) T H ( i ≥ 1 ∑ ϱ i + λ ϱ i ⟨ d π d μ − 1 , e i ⟩ L 2 ( π ) e i ( x ) ) ∇ ϕ ( x ) d μ ( x ) . (95) Since π ≪ L d \pi\ll\mathcal{L}^d π ≪ L d x , y ∈ R d x,y\in\mathbb{R}^d x , y ∈ R d ∫ k ~ ( x , y ) d μ ( y ) ≤ κ λ \int\tilde{k}(x,y)d\mu(y)\le\frac{\kappa}{\lambda} ∫ k ~ ( x , y ) d μ ( y ) ≤ λ κ x ↦ k ~ ( x , y ) x\mapsto\tilde{k}(x,y) x ↦ k ~ ( x , y ) sup x ∣ H 1 k ~ ( x , y ) ∣ ≤ κ K 2 d λ \sup_x\left|\mathbf H_1\tilde{k}(x,y)\right|\le\frac{\sqrt{\kappa K_2d}}{\lambda} sup x H 1 k ~ ( x , y ) ≤ λ κ K 2 d
2 ∣ ∫ ∇ ϕ ( x ) T H ( ∑ i ≥ 1 ⟨ d μ d π − 1 , e i ⟩ L 2 ( π ) e i ( x ) ) ∇ ϕ ( x ) μ ( x ) d x 2|\int\nabla\phi(x)^{\mathsf T}\mathbf H(\sum_{i\ge1}\langle\frac{d\mu}{d\pi}-1,e_i\rangle_{L^2(\pi)}e_i(x))\nabla\phi(x)\mu(x)dx 2∣ ∫ ∇ ϕ ( x ) T H ( i ≥ 1 ∑ ⟨ d π d μ − 1 , e i ⟩ L 2 ( π ) e i ( x )) ∇ ϕ ( x ) μ ( x ) d x − ∫ ∇ ϕ ( x ) T H ( ∑ i ≥ 1 ϱ i ϱ i + λ ⟨ d μ d π − 1 , e i ⟩ L 2 ( π ) e i ( x ) ) ∇ ϕ ( x ) d μ ( x ) = 2 ∣ ∫ ∇ ϕ ( x ) T H ( ∑ i ≥ 1 λ ϱ i + λ ⟨ d μ d π − 1 , e i ⟩ L 2 ( π ) e i ( x ) ) ∇ ϕ ( x ) μ ( x ) d x ∣ . (96) \begin{aligned}
-\int\nabla\phi(x)^{\mathsf{T}}\mathbf{H}\left(\sum_{i\geq1}\frac{\varrho_i}{\varrho_i+\lambda}\left\langle\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1,e_i\right\rangle_{L^2(\pi)}e_i(x)\right)\nabla\phi(x)\mathrm{d}\mu(x) \\
&=2\left|\int\nabla\phi(x)^{\mathsf{T}}\mathbf{H}\left(\sum_{i\geq1}\frac{\lambda}{\varrho_i+\lambda}\left\langle\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1,e_i\right\rangle_{L^2(\pi)}e_i(x)\right)\nabla\phi(x)\mu(x)\mathrm{d}x\right|. \tag*{(96)}
\end{aligned} − ∫ ∇ ϕ ( x ) T H ( i ≥ 1 ∑ ϱ i + λ ϱ i ⟨ d π d μ − 1 , e i ⟩ L 2 ( π ) e i ( x ) ) ∇ ϕ ( x ) d μ ( x ) = 2 ∫ ∇ ϕ ( x ) T H ( i ≥ 1 ∑ ϱ i + λ λ ⟨ d π d μ − 1 , e i ⟩ L 2 ( π ) e i ( x ) ) ∇ ϕ ( x ) μ ( x ) d x . (96) Given d μ d π − 1 ∈ H \frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\in\mathcal{H} d π d μ − 1 ∈ H q ∈ L 2 ( π ) q\in L^2(\pi) q ∈ L 2 ( π ) d μ d π − 1 = T π 1 / 2 q \frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1=\mathcal{T}_{\pi}^{1/2}q d π d μ − 1 = T π 1/2 q ⟨ d μ d π − 1 , e i ⟩ = ϱ i 1 / 2 ⟨ q , e i ⟩ \left\langle\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1,e_i\right\rangle=\varrho_i^{1/2}\langle q,e_i\rangle ⟨ d π d μ − 1 , e i ⟩ = ϱ i 1/2 ⟨ q , e i ⟩ i i i j , r ∈ { 1 , ⋯ , d } j,r\in\{1,\cdots,d\} j , r ∈ { 1 , ⋯ , d }
g M 0 ( x ) : = ∣ ∑ i ≥ M 0 λ ϱ i + λ ⟨ d μ d π − 1 , e i ⟩ L 2 ( π ) ∂ j ∂ r e i ( x ) ∣ = ∣ ∑ i ≥ M 0 λ ϱ i + λ ϱ i 1 / 2 ⟨ q , e i ⟩ L 2 ( π ) ∂ j ∂ r e i ( x ) ∣ ≤ ( ∑ i ≥ M 0 ( λ ϱ i + λ ) 2 ⟨ q , e i ⟩ L 2 ( π ) 2 ) 1 / 2 × ( ∑ i ≥ M 0 ϱ i ( ∂ j ∂ r e i ( x ) ) 2 ) 1 / 2 ≤ ( ∑ i ≥ M 0 ⟨ q , e i ⟩ L 2 ( π ) 2 ) 1 / 2 ( ∑ i ≥ M 0 ϱ i ( ∂ j ∂ r e i ( x ) ) 2 ) 1 / 2 ≤ K 2 d ( ∑ i ≥ M 0 ⟨ q , e i ⟩ L 2 ( π ) 2 ) 1 / 2 . \begin{aligned}
g_{M_0}(x)&:=\left|\sum_{i\geq M_0}\frac{\lambda}{\varrho_i+\lambda}\left\langle\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1,e_i\right\rangle_{L^2(\pi)}\partial_j\partial_r e_i(x)\right|\\
&=\left|\sum_{i\geq M_0}\frac{\lambda}{\varrho_i+\lambda}\varrho_i^{1/2}\langle q,e_i\rangle_{L^2(\pi)}\partial_j\partial_r e_i(x)\right|\\
&\leq\left(\sum_{i\geq M_0}\left(\frac{\lambda}{\varrho_i+\lambda}\right)^2\langle q,e_i\rangle_{L^2(\pi)}^2\right)^{1/2}
\times\left(\sum_{i\geq M_0}\varrho_i\left(\partial_j\partial_r e_i(x)\right)^2\right)^{1/2}\\
&\leq\left(\sum_{i\geq M_0}\langle q,e_i\rangle_{L^2(\pi)}^2\right)^{1/2}\left(\sum_{i\geq M_0}\varrho_i\left(\partial_j\partial_r e_i(x)\right)^2\right)^{1/2}\\
&\leq\sqrt{K_{2d}}\left(\sum_{i\geq M_0}\langle q,e_i\rangle_{L^2(\pi)}^2\right)^{1/2}.
\end{aligned} g M 0 ( x ) := i ≥ M 0 ∑ ϱ i + λ λ ⟨ d π d μ − 1 , e i ⟩ L 2 ( π ) ∂ j ∂ r e i ( x ) = i ≥ M 0 ∑ ϱ i + λ λ ϱ i 1/2 ⟨ q , e i ⟩ L 2 ( π ) ∂ j ∂ r e i ( x ) ≤ ( i ≥ M 0 ∑ ( ϱ i + λ λ ) 2 ⟨ q , e i ⟩ L 2 ( π ) 2 ) 1/2 × ( i ≥ M 0 ∑ ϱ i ( ∂ j ∂ r e i ( x ) ) 2 ) 1/2 ≤ ( i ≥ M 0 ∑ ⟨ q , e i ⟩ L 2 ( π ) 2 ) 1/2 ( i ≥ M 0 ∑ ϱ i ( ∂ j ∂ r e i ( x ) ) 2 ) 1/2 ≤ K 2 d ( i ≥ M 0 ∑ ⟨ q , e i ⟩ L 2 ( π ) 2 ) 1/2 . The final inequality holds because,
∑ j , r = 1 d ∑ i ≥ 1 ϱ i ( ∂ j ∂ r e i ( x ) ) 2 ≤ ∑ j , r = 1 d ∑ i ≥ 1 ⟨ ∂ j ∂ r k ( x , ⋅ ) , e i ⟩ H 2 = ∑ j , r = 1 d ∑ i ≥ 1 ⟨ ∂ j ∂ r k ( x , ⋅ ) , ϱ i e i ⟩ H 2 = ∑ j , r = 1 d ∥ ∂ j ∂ r k ( x , ⋅ ) ∥ H 2 = ∥ H 1 k ( x , ⋅ ) ∥ H d × d 2 ≤ K 2 d . (97) \begin{aligned}
\sum_{j,r=1}^{d}\sum_{i\geq1}\varrho_i\left(\partial_j\partial_r e_i(x)\right)^2
&\leq\sum_{j,r=1}^{d}\sum_{i\geq1}\left\langle\partial_j\partial_r k(x,\cdot),e_i\right\rangle_{\mathcal{H}}^2\\
&=\sum_{j,r=1}^{d}\sum_{i\geq1}\left\langle\partial_j\partial_r k(x,\cdot),\sqrt{\varrho_i}e_i\right\rangle_{\mathcal{H}}^2\\
&=\sum_{j,r=1}^{d}\left\|\partial_j\partial_r k(x,\cdot)\right\|_{\mathcal{H}}^2\\
&=\left\|\mathbf{H}_1k(x,\cdot)\right\|_{\mathcal{H}^{d\times d}}^2\leq K_{2d}. \tag*{(97)}
\end{aligned} j , r = 1 ∑ d i ≥ 1 ∑ ϱ i ( ∂ j ∂ r e i ( x ) ) 2 ≤ j , r = 1 ∑ d i ≥ 1 ∑ ⟨ ∂ j ∂ r k ( x , ⋅ ) , e i ⟩ H 2 = j , r = 1 ∑ d i ≥ 1 ∑ ⟨ ∂ j ∂ r k ( x , ⋅ ) , ϱ i e i ⟩ H 2 = j , r = 1 ∑ d ∥ ∂ j ∂ r k ( x , ⋅ ) ∥ H 2 = ∥ H 1 k ( x , ⋅ ) ∥ H d × d 2 ≤ K 2 d . (97) Since ∥ s d q ∥ L 2 ( π ) \|s\mathrm{d}q\|_{L^2(\pi)} ∥ s d q ∥ L 2 ( π ) ∑ i ≥ M 0 ⟨ q , e i ⟩ L 2 ( π ) 2 \sum_{i\geq M_0}\langle q,e_i\rangle_{L^2(\pi)}^2 ∑ i ≥ M 0 ⟨ q , e i ⟩ L 2 ( π ) 2 M 0 → ∞ M_0\to\infty M 0 → ∞ g M 0 ( x ) g_{M_0}(x) g M 0 ( x )
2 ∣ ∫ ∇ ϕ ( x ) T H ( ∑ i ≥ 1 λ ρ i + λ ⟨ d μ d π − 1 , e i ⟩ L 2 ( π ) e i ( x ) ) ∇ ϕ ( x ) μ ( x ) d x ∣ = 2 ∣ ∫ ∇ ϕ ( x ) T ( ∑ i ≥ 1 λ ρ i + λ ρ i 1 / 2 ⟨ q , e i ⟩ L 2 ( π ) H e i ( x ) ) ∇ ϕ ( x ) μ ( x ) d x ∣ ≤ 2 ( ∑ i ≥ 1 ( λ ρ i + λ ) 2 ⟨ q , e i ⟩ L 2 ( π ) 2 ) 1 / 2 × ( ∑ i ≥ 1 ρ i ( ∫ ∇ ϕ ( x ) T H e i ( x ) ∇ ϕ ( x ) μ ( x ) d x ) 2 ) 1 / 2 ≤ 2 ( ∑ i ≥ 1 ( λ ρ i + λ ) 2 ⟨ q , e i ⟩ L 2 ( π ) 2 ) 1 / 2 × ( ∑ i ≥ 1 ρ i ∥ H e i ( x ) ∥ o p 2 ) 1 / 2 ∥ ∇ ϕ ∥ L 2 ( μ ) 2 ≤ 2 ( ∑ i ≥ 1 ( λ ρ i + λ ) 2 ⟨ q , e i ⟩ L 2 ( π ) 2 ) 1 / 2 × ( ∑ i ≥ 1 ρ i ∥ H e i ( x ) ∥ F 2 ) 1 / 2 ∥ ∇ ϕ ∥ L 2 ( μ ) 2 ≤ 2 ( ∑ i ≥ 1 ( λ ρ i + λ ) 2 ⟨ q , e i ⟩ L 2 ( π ) 2 ) 1 / 2 × K 2 d ∥ ∇ ϕ ∥ L 2 ( μ ) 2 . (98) \begin{aligned}
2\left|\int\nabla\phi(x)^{\mathsf T}\mathbf{H}\left(\sum_{i\geq1}\frac{\lambda}{\rho_i+\lambda}\left\langle\frac{d\mu}{d\pi}-1,e_i\right\rangle_{L^2(\pi)}e_i(x)\right)\nabla\phi(x)\mu(x)\,dx\right| \\
=2\left|\int\nabla\phi(x)^{\mathsf T}\left(\sum_{i\geq1}\frac{\lambda}{\rho_i+\lambda}\rho_i^{1/2}\langle q,e_i\rangle_{L^2(\pi)}\mathbf{H}e_i(x)\right)\nabla\phi(x)\mu(x)\,dx\right| \\
\leq2\left(\sum_{i\geq1}\left(\frac{\lambda}{\rho_i+\lambda}\right)^2\langle q,e_i\rangle_{L^2(\pi)}^2\right)^{1/2} \\
\qquad\times\left(\sum_{i\geq1}\rho_i\left(\int\nabla\phi(x)^{\mathsf T}\mathbf{H}e_i(x)\nabla\phi(x)\mu(x)\,dx\right)^2\right)^{1/2} \\
\leq2\left(\sum_{i\geq1}\left(\frac{\lambda}{\rho_i+\lambda}\right)^2\langle q,e_i\rangle_{L^2(\pi)}^2\right)^{1/2} \\
\qquad\times\left(\sum_{i\geq1}\rho_i\left\|\mathbf{H}e_i(x)\right\|_{\mathrm{op}}^2\right)^{1/2}\|\nabla\phi\|_{L^2(\mu)}^2 \\
\leq2\left(\sum_{i\geq1}\left(\frac{\lambda}{\rho_i+\lambda}\right)^2\langle q,e_i\rangle_{L^2(\pi)}^2\right)^{1/2} \\
\qquad\times\left(\sum_{i\geq1}\rho_i\left\|\mathbf{H}e_i(x)\right\|_{\mathrm{F}}^2\right)^{1/2}\|\nabla\phi\|_{L^2(\mu)}^2 \\
\leq2\left(\sum_{i\geq1}\left(\frac{\lambda}{\rho_i+\lambda}\right)^2\langle q,e_i\rangle_{L^2(\pi)}^2\right)^{1/2} \\
\qquad\times\sqrt{K_{2d}}\|\nabla\phi\|_{L^2(\mu)}^2. \tag*{(98)}
\end{aligned} 2 ∫ ∇ ϕ ( x ) T H ( i ≥ 1 ∑ ρ i + λ λ ⟨ d π d μ − 1 , e i ⟩ L 2 ( π ) e i ( x ) ) ∇ ϕ ( x ) μ ( x ) d x = 2 ∫ ∇ ϕ ( x ) T ( i ≥ 1 ∑ ρ i + λ λ ρ i 1/2 ⟨ q , e i ⟩ L 2 ( π ) H e i ( x ) ) ∇ ϕ ( x ) μ ( x ) d x ≤ 2 ( i ≥ 1 ∑ ( ρ i + λ λ ) 2 ⟨ q , e i ⟩ L 2 ( π ) 2 ) 1/2 × ( i ≥ 1 ∑ ρ i ( ∫ ∇ ϕ ( x ) T H e i ( x ) ∇ ϕ ( x ) μ ( x ) d x ) 2 ) 1/2 ≤ 2 ( i ≥ 1 ∑ ( ρ i + λ λ ) 2 ⟨ q , e i ⟩ L 2 ( π ) 2 ) 1/2 × ( i ≥ 1 ∑ ρ i ∥ H e i ( x ) ∥ op 2 ) 1/2 ∥∇ ϕ ∥ L 2 ( μ ) 2 ≤ 2 ( i ≥ 1 ∑ ( ρ i + λ λ ) 2 ⟨ q , e i ⟩ L 2 ( π ) 2 ) 1/2 × ( i ≥ 1 ∑ ρ i ∥ H e i ( x ) ∥ F 2 ) 1/2 ∥∇ ϕ ∥ L 2 ( μ ) 2 ≤ 2 ( i ≥ 1 ∑ ( ρ i + λ λ ) 2 ⟨ q , e i ⟩ L 2 ( π ) 2 ) 1/2 × K 2 d ∥∇ ϕ ∥ L 2 ( μ ) 2 . (98) The first inequality uses Cauchy Schwartz, the second last inequality uses that matrix operator norm is smaller than matrix Frobenius norm, and the last inequality uses (B.19). Combining together, (B.15) and (B.20), we reach
∣ 1 1 + λ d 2 d t 2 ∣ t = 0 DrMMD ( ρ t ∥ π ) − d 2 d t 2 ∣ t = 0 χ 2 ( ρ t ∥ π ) ∣ ≤ 2 ∑ i ≥ 1 λ ρ i + λ ( ∫ ∇ ϕ ( x ) T ∇ e i ( x ) μ ( x ) d x ) 2 \left|\left.\frac{1}{1+\lambda}\frac{d^2}{dt^2}\right|_{t=0}\operatorname{DrMMD}(\rho_t\Vert\pi)-\left.\frac{d^2}{dt^2}\right|_{t=0}\chi^2(\rho_t\Vert\pi)\right|
\leq2\sum_{i\geq1}\frac{\lambda}{\rho_i+\lambda}\left(\int\nabla\phi(x)^{\mathsf T}\nabla e_i(x)\mu(x)\,dx\right)^2 1 + λ 1 d t 2 d 2 t = 0 DrMMD ( ρ t ∥ π ) − d t 2 d 2 t = 0 χ 2 ( ρ t ∥ π ) ≤ 2 i ≥ 1 ∑ ρ i + λ λ ( ∫ ∇ ϕ ( x ) T ∇ e i ( x ) μ ( x ) d x ) 2 + 2 ( ∑ i ≥ 1 ( λ ρ i + λ ) 2 ⟨ q , e i ⟩ L 2 ( π ) 2 ) 1 / 2 K 2 d ∥ ∇ ϕ ∥ L 2 ( μ ) 2 = : R ‾ ( λ , μ , ∇ ϕ ) . \begin{aligned}
\qquad+2\left(\sum_{i\geq1}\left(\frac{\lambda}{\rho_i+\lambda}\right)^2\langle q,e_i\rangle_{L^2(\pi)}^2\right)^{1/2}\sqrt{K_{2d}}\|\nabla\phi\|_{L^2(\mu)}^2 \\
&=: \overline{R}(\lambda,\mu,\nabla\phi).
\end{aligned} + 2 ( i ≥ 1 ∑ ( ρ i + λ λ ) 2 ⟨ q , e i ⟩ L 2 ( π ) 2 ) 1/2 K 2 d ∥∇ ϕ ∥ L 2 ( μ ) 2 =: R ( λ , μ , ∇ ϕ ) . Therefore,
d 2 d t 2 ∣ t = 0 DrMMD ( ρ t ∥ π ) ≥ ( 1 + λ ) d 2 d t 2 ∣ t = 0 χ 2 ( ρ t ∥ π ) − ( 1 + λ ) R ˉ ( λ , μ , ∇ ϕ ) ≥ ( 1 + λ ) ∫ d μ d π ( x ) ∇ ϕ ( x ) T H V ( x ) ∇ ϕ ( x ) d μ ( x ) − ( 1 + λ ) R ˉ ( λ , μ , ∇ ϕ ) ≥ ( 1 + λ ) α ∫ d μ d π ( x ) ∥ ∇ ϕ ( x ) ∥ 2 d μ ( x ) − ( 1 + λ ) R ˉ ( λ , μ , ∇ ϕ ) , \begin{aligned}
\left.\frac{d^2}{dt^2}\right|_{t=0}\operatorname{DrMMD}(\rho_t\|\pi) \ge(1+\lambda)\left.\frac{d^2}{dt^2}\right|_{t=0}\chi^2(\rho_t\|\pi) - (1+\lambda)\bar{R}(\lambda,\mu,\nabla\phi) \\
&\ge(1+\lambda)\int\frac{d\mu}{d\pi}(x)\nabla\phi(x)^{\mathsf{T}}\mathbf{H}V(x)\nabla\phi(x)d\mu(x) - (1+\lambda)\bar{R}(\lambda,\mu,\nabla\phi) \\
&\ge(1+\lambda)\alpha\int\frac{d\mu}{d\pi}(x)\|\nabla\phi(x)\|^2d\mu(x) - (1+\lambda)\bar{R}(\lambda,\mu,\nabla\phi),
\end{aligned} d t 2 d 2 t = 0 DrMMD ( ρ t ∥ π ) ≥ ( 1 + λ ) d t 2 d 2 t = 0 χ 2 ( ρ t ∥ π ) − ( 1 + λ ) R ˉ ( λ , μ , ∇ ϕ ) ≥ ( 1 + λ ) ∫ d π d μ ( x ) ∇ ϕ ( x ) T H V ( x ) ∇ ϕ ( x ) d μ ( x ) − ( 1 + λ ) R ˉ ( λ , μ , ∇ ϕ ) ≥ ( 1 + λ ) α ∫ d π d μ ( x ) ∥∇ ϕ ( x ) ∥ 2 d μ ( x ) − ( 1 + λ ) R ˉ ( λ , μ , ∇ ϕ ) , where the second inequality is using (B.9) and the last inequality is using H V ⪰ α Id HV \succeq\alpha\operatorname{Id} H V ⪰ α Id
Define R ( λ , μ , ∇ ϕ ) : = ( 1 + λ ) R ˉ ( λ , μ , ∇ ϕ ) R(\lambda,\mu,\nabla\phi):=(1+\lambda)\bar{R}(\lambda,\mu,\nabla\phi) R ( λ , μ , ∇ ϕ ) := ( 1 + λ ) R ˉ ( λ , μ , ∇ ϕ ) lim λ → 0 R ( λ , μ , ∇ ϕ ) = 0 \lim_{\lambda\to0}R(\lambda,\mu,\nabla\phi)=0 lim λ → 0 R ( λ , μ , ∇ ϕ ) = 0 lim λ → 0 R ˉ ( λ , μ , ∇ ϕ ) = 0 \lim_{\lambda\to0}\bar{R}(\lambda,\mu,\nabla\phi)=0 lim λ → 0 R ˉ ( λ , μ , ∇ ϕ ) = 0
∑ i ≥ 1 ( ∫ ∇ ϕ ( x ) T ∇ e i ( x ) μ ( x ) d x ) 2 < d 2 d t 2 ∣ t = 0 χ 2 ( ρ t ∥ π ) < ∞ , \sum_{i\geq1}\left(\int\nabla\phi(x)^{\mathsf{T}}\nabla e_i(x)\mu(x)dx\right)^2 < \left.\frac{d^2}{dt^2}\right|_{t=0}\chi^2(\rho_t\|\pi)<\infty, i ≥ 1 ∑ ( ∫ ∇ ϕ ( x ) T ∇ e i ( x ) μ ( x ) d x ) 2 < d t 2 d 2 t = 0 χ 2 ( ρ t ∥ π ) < ∞ , using the dominated convergence theorem (Rudin, 1976), we are allowed to interchange infinite sum and taking limits,
lim λ → 0 ∑ i ≥ 1 λ ϱ i + λ ( ∫ ∇ ϕ ( x ) T ∇ e i ( x ) μ ( x ) d x ) 2 = ∑ i ≥ 1 lim λ → 0 λ ϱ i + λ ( ∫ ∇ ϕ ( x ) T ∇ e i ( x ) μ ( x ) d x ) 2 = 0. \begin{aligned}
\lim_{\lambda\to0}\sum_{i\geq1}\frac{\lambda}{\varrho_i+\lambda}\left(\int\nabla\phi(x)^{\mathsf{T}}\nabla e_i(x)\mu(x)dx\right)^2 \\
&=\sum_{i\geq1}\lim_{\lambda\to0}\frac{\lambda}{\varrho_i+\lambda}\left(\int\nabla\phi(x)^{\mathsf{T}}\nabla e_i(x)\mu(x)dx\right)^2=0.
\end{aligned} λ → 0 lim i ≥ 1 ∑ ϱ i + λ λ ( ∫ ∇ ϕ ( x ) T ∇ e i ( x ) μ ( x ) d x ) 2 = i ≥ 1 ∑ λ → 0 lim ϱ i + λ λ ( ∫ ∇ ϕ ( x ) T ∇ e i ( x ) μ ( x ) d x ) 2 = 0. Similarly, because ∑ i ≥ 1 ( λ ϱ i + λ ) 2 ⟨ q , e i ⟩ L 2 ( π ) 2 < ∥ q ∥ L 2 ( π ) 2 < ∞ \sum_{i\geq1}\left(\frac{\lambda}{\varrho_i+\lambda}\right)^2\langle q,e_i\rangle_{L^2(\pi)}^2<\|q\|_{L^2(\pi)}^2<\infty ∑ i ≥ 1 ( ϱ i + λ λ ) 2 ⟨ q , e i ⟩ L 2 ( π ) 2 < ∥ q ∥ L 2 ( π ) 2 < ∞
lim λ → 0 ∑ i ≥ 1 ( λ ϱ i + λ ) 2 ⟨ q , e i ⟩ L 2 ( π ) 2 = ∑ i ≥ 1 lim λ → 0 ( λ ϱ i + λ ) 2 ⟨ q , e i ⟩ L 2 ( π ) 2 = 0. \lim_{\lambda\to0}\sum_{i\geq1}\left(\frac{\lambda}{\varrho_i+\lambda}\right)^2\langle q,e_i\rangle_{L^2(\pi)}^2
=\sum_{i\geq1}\lim_{\lambda\to0}\left(\frac{\lambda}{\varrho_i+\lambda}\right)^2\langle q,e_i\rangle_{L^2(\pi)}^2=0. λ → 0 lim i ≥ 1 ∑ ( ϱ i + λ λ ) 2 ⟨ q , e i ⟩ L 2 ( π ) 2 = i ≥ 1 ∑ λ → 0 lim ( ϱ i + λ λ ) 2 ⟨ q , e i ⟩ L 2 ( π ) 2 = 0. Therefore, we have that
lim λ → 0 R ( λ , μ , ∇ ϕ ) = lim λ → 0 R ˉ ( λ , μ , ∇ ϕ ) = 0. \lim_{\lambda\to0}R(\lambda,\mu,\nabla\phi)=\lim_{\lambda\to0}\bar{R}(\lambda,\mu,\nabla\phi)=0. λ → 0 lim R ( λ , μ , ∇ ϕ ) = λ → 0 lim R ˉ ( λ , μ , ∇ ϕ ) = 0. And the proof of the proposition is finished.
Proof of Theorem 4.14
Considering the time derivative of KL ( μ t ∥ π ) \operatorname{KL}(\mu_t\Vert\pi) KL ( μ t ∥ π )
d d t KL ( μ t ∥ π ) = − ( 1 + λ ) ∫ ∇ h μ t , π ∗ ( x ) T ∇ log d μ t d π ( x ) μ t ( x ) d x = − ( 1 + λ ) ∫ ∇ h μ t , π ∗ ( x ) T ∇ d μ t d π ( x ) π ( x ) d x = − ( 1 + λ ) ∫ ( ∇ h μ t , π ∗ ( x ) − 2 ∇ d μ t d π ( x ) ) T ∇ d μ t d π ( x ) π ( x ) d x − 2 ( 1 + λ ) ∫ π ( x ) ∥ ∇ d μ t d π ( x ) ∥ 2 d x = − ( 1 + λ ) ∫ ( ∇ h μ t , π ∗ ( x ) − 2 ∇ ( d μ t d π ( x ) − 1 ) ) T ∇ d μ t d π ( x ) π ( x ) d x − 2 ( 1 + λ ) ∫ π ( x ) ∥ ∇ d μ t d π ( x ) ∥ 2 d x . (99) \begin{aligned}
\frac{\mathrm{d}}{\mathrm{d}t}\operatorname{KL}(\mu_t\Vert\pi)
&=-(1+\lambda)\int\nabla h^*_{\mu_t,\pi}(x)^{\mathsf{T}}\nabla\log\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}(x)\mu_t(x)\,\mathrm{d}x\\
&=-(1+\lambda)\int\nabla h^*_{\mu_t,\pi}(x)^{\mathsf{T}}\nabla\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}(x)\pi(x)\,\mathrm{d}x\\
&=-(1+\lambda)\int\left(\nabla h^*_{\mu_t,\pi}(x)-2\nabla\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}(x)\right)^{\mathsf{T}}\nabla\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}(x)\pi(x)\,\mathrm{d}x\\
&\quad-2(1+\lambda)\int\pi(x)\left\|\nabla\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}(x)\right\|^2\,\mathrm{d}x\\
&=-(1+\lambda)\int\left(\nabla h^*_{\mu_t,\pi}(x)-2\nabla\left(\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}(x)-1\right)\right)^{\mathsf{T}}\nabla\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}(x)\pi(x)\,\mathrm{d}x\\
&\quad-2(1+\lambda)\int\pi(x)\left\|\nabla\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}(x)\right\|^2\,\mathrm{d}x. \tag*{(99)}
\end{aligned} d t d KL ( μ t ∥ π ) = − ( 1 + λ ) ∫ ∇ h μ t , π ∗ ( x ) T ∇ log d π d μ t ( x ) μ t ( x ) d x = − ( 1 + λ ) ∫ ∇ h μ t , π ∗ ( x ) T ∇ d π d μ t ( x ) π ( x ) d x = − ( 1 + λ ) ∫ ( ∇ h μ t , π ∗ ( x ) − 2∇ d π d μ t ( x ) ) T ∇ d π d μ t ( x ) π ( x ) d x − 2 ( 1 + λ ) ∫ π ( x ) ∇ d π d μ t ( x ) 2 d x = − ( 1 + λ ) ∫ ( ∇ h μ t , π ∗ ( x ) − 2∇ ( d π d μ t ( x ) − 1 ) ) T ∇ d π d μ t ( x ) π ( x ) d x − 2 ( 1 + λ ) ∫ π ( x ) ∇ d π d μ t ( x ) 2 d x . (99) Case one: ∫ π ( x ) ∥ ∇ d μ t d π ( x ) ∥ 2 d x < ∞ \int\pi(x)\left\|\nabla\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}(x)\right\|^2\,\mathrm{d}x<\infty ∫ π ( x ) ∇ d π d μ t ( x ) 2 d x < ∞ i = 1 , … , d i=1,\ldots,d i = 1 , … , d lim x → ∞ ( h μ t , π ∗ ( x ) − 2 d μ t d π ( x ) ) ( ∂ i d μ t d π ( x ) ) π ( x ) → 0 \lim_{x\to\infty}\left(h^*_{\mu_t,\pi}(x)-2\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}(x)\right)\left(\partial_i\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}(x)\right)\pi(x)\to0 lim x → ∞ ( h μ t , π ∗ ( x ) − 2 d π d μ t ( x ) ) ( ∂ i d π d μ t ( x ) ) π ( x ) → 0
d d t KL ( μ t ∥ π ) = ( 1 + λ ) ∫ ( h μ t , π ∗ ( x ) − 2 ( d μ t d π ( x ) − 1 ) ) × ∇ ⋅ ( π ( x ) ∇ d μ t d π ( x ) ) d x − 2 ( 1 + λ ) ∫ π ( x ) ∥ ∇ d μ t d π ( x ) ∥ 2 d x = ( 1 + λ ) ∫ ( h μ t , π ∗ ( x ) − 2 ( d μ t d π ( x ) − 1 ) ) × ∇ ⋅ ( π ( x ) ∇ d μ t d π ( x ) ) π ( x ) π ( x ) d x − 2 ( 1 + λ ) ∫ π ( x ) ∥ ∇ d μ t d π ( x ) ∥ 2 d x ≤ ( 1 + λ ) ∥ h μ t , π ∗ − 2 ( d μ t d π − 1 ) ∥ L 2 ( π ) × ∥ ∇ ⋅ ( π ∇ d μ t d π ) π ∥ L 2 ( π ) 2 − 2 ( 1 + λ ) C P KL ( μ t ∥ π ) . (100) \begin{aligned}
\frac{\mathrm{d}}{\mathrm{d}t}\operatorname{KL}(\mu_t\Vert\pi)
&=(1+\lambda)\int\left(h^*_{\mu_t,\pi}(x)-2\left(\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}(x)-1\right)\right)\\
&\quad\times\nabla\cdot\left(\pi(x)\nabla\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}(x)\right)\,\mathrm{d}x\\
&\quad-2(1+\lambda)\int\pi(x)\left\|\nabla\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}(x)\right\|^2\,\mathrm{d}x\\
&=(1+\lambda)\int\left(h^*_{\mu_t,\pi}(x)-2\left(\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}(x)-1\right)\right)\\
&\quad\times\frac{\nabla\cdot\left(\pi(x)\nabla\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}(x)\right)}{\pi(x)}\pi(x)\,\mathrm{d}x\\
&\quad-2(1+\lambda)\int\pi(x)\left\|\nabla\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}(x)\right\|^2\,\mathrm{d}x\\
&\leq(1+\lambda)\left\|h^*_{\mu_t,\pi}-2\left(\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}-1\right)\right\|_{L^2(\pi)} \\
&\times\left\| \frac{\nabla\cdot\left(\pi\nabla\frac{d\mu_t}{d\pi}\right)}{\pi} \right\|_{L^2(\pi)}^2 - \frac{2(1+\lambda)}{C_P}\operatorname{KL}(\mu_t\Vert\pi). \tag*{(100)}
\end{aligned} d t d KL ( μ t ∥ π ) = ( 1 + λ ) ∫ ( h μ t , π ∗ ( x ) − 2 ( d π d μ t ( x ) − 1 ) ) × ∇ ⋅ ( π ( x ) ∇ d π d μ t ( x ) ) d x − 2 ( 1 + λ ) ∫ π ( x ) ∇ d π d μ t ( x ) 2 d x = ( 1 + λ ) ∫ ( h μ t , π ∗ ( x ) − 2 ( d π d μ t ( x ) − 1 ) ) × π ( x ) ∇ ⋅ ( π ( x ) ∇ d π d μ t ( x ) ) π ( x ) d x − 2 ( 1 + λ ) ∫ π ( x ) ∇ d π d μ t ( x ) 2 d x ≤ ( 1 + λ ) h μ t , π ∗ − 2 ( d π d μ t − 1 ) L 2 ( π ) × π ∇ ⋅ ( π ∇ d π d μ t ) L 2 ( π ) 2 − C P 2 ( 1 + λ ) KL ( μ t ∥ π ) . (100) where the first part of the last inequality holds by using Cauchy Schwartz, and the second part holds by the fact that KL ( μ t ∥ π ) ≤ χ 2 ( μ t ∥ π ) \operatorname{KL}(\mu_t\Vert\pi) \le\chi^2(\mu_t\Vert\pi) KL ( μ t ∥ π ) ≤ χ 2 ( μ t ∥ π ) f = d μ t d π − 1 f = \frac{d\mu_t}{d\pi} - 1 f = d π d μ t − 1 ∥ ∇ f ∥ L 2 ( π ) < ∞ \|\nabla f\|_{L^2(\pi)} < \infty ∥∇ f ∥ L 2 ( π ) < ∞ Case one and ∥ f ∥ L 2 ( π ) < ∞ \|f\|_{L^2(\pi)} < \infty ∥ f ∥ L 2 ( π ) < ∞
∫ π ( x ) ∥ ∇ d μ t d π ( x ) ∥ 2 d x ≥ 1 C P χ 2 ( μ t ∥ π ) ≥ 1 C P KL ( μ t ∥ π ) . \int\pi(x)\left\|\nabla\frac{d\mu_t}{d\pi}(x)\right\|^2 dx \ge\frac{1}{C_P}\chi^2(\mu_t\Vert\pi) \ge\frac{1}{C_P}\operatorname{KL}(\mu_t\Vert\pi). ∫ π ( x ) ∇ d π d μ t ( x ) 2 d x ≥ C P 1 χ 2 ( μ t ∥ π ) ≥ C P 1 KL ( μ t ∥ π ) . Since d μ t d π − 1 ∈ Ran ( T π r ) \frac{d\mu_t}{d\pi} - 1 \in\operatorname{Ran}(T_\pi^r) d π d μ t − 1 ∈ Ran ( T π r ) r > 0 r > 0 r > 0
∥ h μ t , π ∗ − 2 ( d μ t d π − 1 ) ∥ L 2 ( π ) ≤ 2 λ r ∥ q t ∥ L 2 ( π ) . (101) \left\|h^*_{\mu_t,\pi} - 2\left(\frac{d\mu_t}{d\pi} - 1\right)\right\|_{L^2(\pi)} \le2\lambda^r\|q_t\|_{L^2(\pi)}. \tag*{(101)} h μ t , π ∗ − 2 ( d π d μ t − 1 ) L 2 ( π ) ≤ 2 λ r ∥ q t ∥ L 2 ( π ) . (101) Then, notice that
∥ ∇ ⋅ ( π ∇ d μ t d π ) π ∥ L 2 ( π ) 2 = ∫ [ ∇ ⋅ ( π ( x ) ∇ d μ t d π ( x ) ) ] 2 π ( x ) 2 d π ( x ) = ∫ ( ∇ π ( x ) T ∇ d μ t d π ( x ) + π ( x ) ∇ ⋅ ∇ d μ t d π ( x ) ) 2 π ( x ) 2 d π ( x ) = ∫ ( ∇ log π ( x ) T ∇ d μ t d π ( x ) + Δ d μ t d π ( x ) ) 2 d π ( x ) ≤ ∫ 2 ( ∇ log π ( x ) T ∇ d μ t d π ( x ) ) 2 + 2 ( Δ d μ t d π ( x ) ) 2 d π ( x ) = 2 ∥ ∇ ( log π ) T ∇ ( d μ t d π ) ∥ L 2 ( π ) 2 + 2 ∥ Δ ( d μ t d π ) ∥ L 2 ( π ) 2 ≤ 2 J t 2 + 2 I t 2 . (102) \begin{aligned}
\left\|\frac{\nabla\cdot\left(\pi\nabla\frac{d\mu_t}{d\pi}\right)}{\pi}\right\|_{L^2(\pi)}^2
&= \int\frac{\left[\nabla\cdot\left(\pi(x)\nabla\frac{d\mu_t}{d\pi}(x)\right)\right]^2}{\pi(x)^2}\,d\pi(x) \\
&= \int\frac{\left(\nabla\pi(x)^{\mathsf T}\nabla\frac{d\mu_t}{d\pi}(x) + \pi(x)\nabla\cdot\nabla\frac{d\mu_t}{d\pi}(x)\right)^2}{\pi(x)^2}\,d\pi(x) \\
&= \int\left(\nabla\log\pi(x)^{\mathsf T}\nabla\frac{d\mu_t}{d\pi}(x) + \Delta\frac{d\mu_t}{d\pi}(x)\right)^2\,d\pi(x) \\
&\le\int2\left(\nabla\log\pi(x)^{\mathsf T}\nabla\frac{d\mu_t}{d\pi}(x)\right)^2 + 2\left(\Delta\frac{d\mu_t}{d\pi}(x)\right)^2\,d\pi(x) \\
&= 2\left\|\nabla(\log\pi)^{\mathsf T}\nabla\left(\frac{d\mu_t}{d\pi}\right)\right\|_{L^2(\pi)}^2 + 2\left\|\Delta\left(\frac{d\mu_t}{d\pi}\right)\right\|_{L^2(\pi)}^2 \le2\mathcal{J}_t^2 + 2\mathcal{I}_t^2. \tag*{(102)}
\end{aligned} π ∇ ⋅ ( π ∇ d π d μ t ) L 2 ( π ) 2 = ∫ π ( x ) 2 [ ∇ ⋅ ( π ( x ) ∇ d π d μ t ( x ) ) ] 2 d π ( x ) = ∫ π ( x ) 2 ( ∇ π ( x ) T ∇ d π d μ t ( x ) + π ( x ) ∇ ⋅ ∇ d π d μ t ( x ) ) 2 d π ( x ) = ∫ ( ∇ log π ( x ) T ∇ d π d μ t ( x ) + Δ d π d μ t ( x ) ) 2 d π ( x ) ≤ ∫ 2 ( ∇ log π ( x ) T ∇ d π d μ t ( x ) ) 2 + 2 ( Δ d π d μ t ( x ) ) 2 d π ( x ) = 2 ∇ ( log π ) T ∇ ( d π d μ t ) L 2 ( π ) 2 + 2 Δ ( d π d μ t ) L 2 ( π ) 2 ≤ 2 J t 2 + 2 I t 2 . (102) Therefore, plugging (B.23) and (B.24) back to (B.22), we have
d d t KL ( μ t ∥ π ) ≤ 4 ( 1 + λ ) λ r ∥ q t ∥ L 2 ( π ) ( J t + I t ) − 2 ( 1 + λ ) C P KL ( μ t ∥ π ) . (103) \frac{d}{dt}\operatorname{KL}(\mu_t\Vert\pi) \le4(1+\lambda)\lambda^r\|q_t\|_{L^2(\pi)}(\mathcal{J}_t+\mathcal{I}_t) - \frac{2(1+\lambda)}{C_P}\operatorname{KL}(\mu_t\Vert\pi). \tag*{(103)} d t d KL ( μ t ∥ π ) ≤ 4 ( 1 + λ ) λ r ∥ q t ∥ L 2 ( π ) ( J t + I t ) − C P 2 ( 1 + λ ) KL ( μ t ∥ π ) . (103) Case two: ∫ π ( x ) ∥ ∇ d μ t d π ( x ) ∥ 2 d x = ∞ \int\pi(x)\left\|\nabla\frac{d\mu_t}{d\pi}(x)\right\|^2 dx = \infty ∫ π ( x ) ∇ d π d μ t ( x ) 2 d x = ∞ 2 ( 1 + λ ) C P KL ( μ t ∥ π ) \frac{2(1+\lambda)}{C_P}\operatorname{KL}(\mu_t\|\pi) C P 2 ( 1 + λ ) KL ( μ t ∥ π ) Case one .
Therefore, both Case one and Case two result in (B.25). Using the Gronwall lemma, we obtain that for any T > 0 T > 0 T > 0
KL ( μ T ∥ π ) ≤ exp ( − 2 ( 1 + λ ) C P T ) KL ( μ 0 ∥ π ) + 4 ( 1 + λ ) λ r ∫ 0 T exp ( − 2 ( 1 + λ ) C P ( T − t ) ) ∥ q t ∥ L 2 ( π ) ( J t + I t ) d t , \operatorname{KL}(\mu_T\|\pi) \le\exp\left(-\frac{2(1+\lambda)}{C_P}T\right)\operatorname{KL}(\mu_0\|\pi)
+4(1+\lambda)\lambda^r\int_0^T\exp\left(-\frac{2(1+\lambda)}{C_P}(T-t)\right)\|q_t\|_{L^2(\pi)}(\mathcal{J}_t+\mathcal{I}_t)dt, KL ( μ T ∥ π ) ≤ exp ( − C P 2 ( 1 + λ ) T ) KL ( μ 0 ∥ π ) + 4 ( 1 + λ ) λ r ∫ 0 T exp ( − C P 2 ( 1 + λ ) ( T − t ) ) ∥ q t ∥ L 2 ( π ) ( J t + I t ) d t , which concludes the proof of Theorem 4.14 .
Derivation of (4.12) under stronger range assumption r = 1 2 r=\frac{1}{2} r = 2 1
Notice that for any x ∈ R d x\in\mathbb{R}^d x ∈ R d k k k
∇ ( d μ t d π − 1 ) ( x ) = ⟨ ∇ k ( x , ⋅ ) , d μ t d π − 1 ⟩ H d ≤ K 1 d ∥ d μ t d π − 1 ∥ H . \nabla\left(\frac{d\mu_t}{d\pi}-1\right)(x)=\left\langle\nabla k(x,\cdot),\frac{d\mu_t}{d\pi}-1\right\rangle_{\mathcal{H}^d}\le\sqrt{K_{1d}}\left\|\frac{d\mu_t}{d\pi}-1\right\|_{\mathcal{H}}. ∇ ( d π d μ t − 1 ) ( x ) = ⟨ ∇ k ( x , ⋅ ) , d π d μ t − 1 ⟩ H d ≤ K 1 d d π d μ t − 1 H . And since d μ t d π − 1 ∈ H \frac{d\mu_t}{d\pi}-1\in\mathcal{H} d π d μ t − 1 ∈ H q t ∈ L 2 ( π ) q_t\in L^2(\pi) q t ∈ L 2 ( π ) ⟨ d μ t d π − 1 , e i ⟩ L 2 ( π ) = ϱ i 1 / 2 ⟨ q t , e i ⟩ L 2 ( π ) \left\langle\frac{d\mu_t}{d\pi}-1,e_i\right\rangle_{L^2(\pi)}=\varrho_i^{1/2}\langle q_t,e_i\rangle_{L^2(\pi)} ⟨ d π d μ t − 1 , e i ⟩ L 2 ( π ) = ϱ i 1/2 ⟨ q t , e i ⟩ L 2 ( π ) i i i
∥ d μ t d π − 1 ∥ H 2 = ∑ i ≥ 1 1 ϱ i ⟨ d μ t d π − 1 , e i ⟩ L 2 ( π ) 2 = ∑ i ≥ 1 ⟨ q t , e i ⟩ L 2 ( π ) 2 = ∥ q t ∥ L 2 ( π ) 2 . \left\|\frac{d\mu_t}{d\pi}-1\right\|_{\mathcal{H}}^2=\sum_{i\ge1}\frac{1}{\varrho_i}\left\langle\frac{d\mu_t}{d\pi}-1,e_i\right\rangle_{L^2(\pi)}^2=\sum_{i\ge1}\langle q_t,e_i\rangle_{L^2(\pi)}^2=\|q_t\|_{L^2(\pi)}^2. d π d μ t − 1 H 2 = i ≥ 1 ∑ ϱ i 1 ⟨ d π d μ t − 1 , e i ⟩ L 2 ( π ) 2 = i ≥ 1 ∑ ⟨ q t , e i ⟩ L 2 ( π ) 2 = ∥ q t ∥ L 2 ( π ) 2 . Combining the above two equations, we have
∥ ∇ ( log π ) T ∇ ( d μ t d π − 1 ) ∥ L 2 ( π ) ≤ K 1 d ∥ q t ∥ L 2 ( π ) ∥ ∇ log π ∥ L 2 ( π ) . \left\|\nabla(\log\pi)^\mathsf{T}\nabla\left(\frac{d\mu_t}{d\pi}-1\right)\right\|_{L^2(\pi)}\le\sqrt{K_{1d}}\|q_t\|_{L^2(\pi)}\|\nabla\log\pi\|_{L^2(\pi)}. ∇ ( log π ) T ∇ ( d π d μ t − 1 ) L 2 ( π ) ≤ K 1 d ∥ q t ∥ L 2 ( π ) ∥∇ log π ∥ L 2 ( π ) . Also, for the other one, notice that
Δ ( d μ t d π ) ( x ) ≤ ∥ H ( d μ t d π − 1 ) ( x ) ∥ F = ∥ ⟨ H k ( x , ⋅ ) , d μ t d π − 1 ⟩ ∥ F ≤ ∥ H k ( x , ⋅ ) ∥ H d × d ∥ d μ t d π − 1 ∥ H ≤ K 2 d ∥ q t ∥ L 2 ( π ) . \Delta\left(\frac{d\mu_t}{d\pi}\right)(x)\le\left\|\mathbf{H}\left(\frac{d\mu_t}{d\pi}-1\right)(x)\right\|_F=\left\|\left\langle\mathbf{H}k(x,\cdot),\frac{d\mu_t}{d\pi}-1\right\rangle\right\|_F
\le\|\mathbf{H}k(x,\cdot)\|_{\mathcal{H}^{d\times d}}\left\|\frac{d\mu_t}{d\pi}-1\right\|_{\mathcal{H}}\le\sqrt{K_{2d}}\|q_t\|_{L^2(\pi)}. Δ ( d π d μ t ) ( x ) ≤ H ( d π d μ t − 1 ) ( x ) F = ⟨ H k ( x , ⋅ ) , d π d μ t − 1 ⟩ F ≤ ∥ H k ( x , ⋅ ) ∥ H d × d d π d μ t − 1 H ≤ K 2 d ∥ q t ∥ L 2 ( π ) . Therefore,
∥ Δ ( d μ t d π ) ∥ L 2 ( π ) ≤ K 2 d ∥ q t ∥ L 2 ( π ) . \left\lVert\Delta\left(\frac{\mathrm{d}\mu_t}{\mathrm{d}\pi}\right)\right\rVert_{L^2(\pi)} \le\sqrt{K_{2d}}\lVert q_t\rVert_{L^2(\pi)}. Δ ( d π d μ t ) L 2 ( π ) ≤ K 2 d ∥ q t ∥ L 2 ( π ) . Proof of KL descent lemma
We know that μ n + 1 = ( Id − γ ( 1 + λ ) ∇ h ) # μ n \mu_{n+1}=(\operatorname{Id}-\gamma(1+\lambda)\nabla h)\#\mu_n μ n + 1 = ( Id − γ ( 1 + λ ) ∇ h ) # μ n h μ n , π h_{\mu_n,\pi} h μ n , π ρ t = ( Id − t ( 1 + λ ) ∇ h ) # μ n \rho_t=(\operatorname{Id}-t(1+\lambda)\nabla h)\#\mu_n ρ t = ( Id − t ( 1 + λ ) ∇ h ) # μ n ρ 0 = μ n \rho_0=\mu_n ρ 0 = μ n ρ γ = μ n + 1 \rho_\gamma=\mu_{n+1} ρ γ = μ n + 1 μ n + 1 \mu_{n+1} μ n + 1 μ n \mu_n μ n
KL ( μ n + 1 ∥ π ) − KL ( μ n ∥ π ) = KL ( ρ γ ∥ π ) − KL ( ρ 0 ∥ π ) = d d t KL ( ρ t ∥ π ) ∣ t = 0 γ + ∫ 0 γ ( γ − t ) d 2 d t 2 KL ( ρ t ∥ π ) d t . (104) \begin{aligned}
\operatorname{KL}(\mu_{n+1}\Vert\pi)-\operatorname{KL}(\mu_n\Vert\pi)
&=\operatorname{KL}(\rho_\gamma\Vert\pi)-\operatorname{KL}(\rho_0\Vert\pi)\\
&=\left.\frac{\mathrm{d}}{\mathrm{d}t}\operatorname{KL}(\rho_t\Vert\pi)\right|_{t=0}\gamma+\int_0^\gamma(\gamma-t)\frac{\mathrm{d}^2}{\mathrm{d}t^2}\operatorname{KL}(\rho_t\Vert\pi)\,\mathrm{d}t. \tag*{(104)}
\end{aligned} KL ( μ n + 1 ∥ π ) − KL ( μ n ∥ π ) = KL ( ρ γ ∥ π ) − KL ( ρ 0 ∥ π ) = d t d KL ( ρ t ∥ π ) t = 0 γ + ∫ 0 γ ( γ − t ) d t 2 d 2 KL ( ρ t ∥ π ) d t . (104) For the first term of (B.26),
d d t KL ( ρ t ∥ π ) ∣ t = 0 = − ( 1 + λ ) E μ n [ ∇ log d μ n d π ⊤ ∇ h ] = − ( 1 + λ ) E π [ ∇ d μ n d π ⊤ ∇ h ] = − 2 ( 1 + λ ) ∥ ∇ d μ n d π ∥ L 2 ( π ) 2 + ( 1 + λ ) E π [ ∇ d μ n d π ⊤ ( 2 ∇ d μ n d π − ∇ h ) ] = − 2 ( 1 + λ ) ∥ ∇ d μ n d π ∥ L 2 ( π ) 2 + ( 1 + λ ) ∫ ( ∇ d μ n d π ( x ) ) ⊤ ( 2 ∇ ( d μ n d π ( x ) − 1 ) − ∇ h ( x ) ) π ( x ) d x = − 2 ( 1 + λ ) ∥ ∇ d μ n d π ∥ L 2 ( π ) 2 + ( 1 + λ ) ∫ ( 2 ( d μ n d π ( x ) − 1 ) − h ( x ) ) ∇ ⋅ ( π ( x ) ∇ d μ n d π ( x ) ) d x ≤ − ( 1 + λ ) 2 C P 2 χ 2 ( μ n ∥ π ) + ( 1 + λ ) ∥ h − 2 ( d μ n d π − 1 ) ∥ L 2 ( π ) ∥ ∇ ⋅ ( π ∇ d μ n d π ) π ∥ L 2 ( π ) . (105) \begin{aligned}
\left.\frac{\mathrm{d}}{\mathrm{d}t}\operatorname{KL}(\rho_t\Vert\pi)\right|_{t=0}
&=-(1+\lambda)\mathbb{E}_{\mu_n}\left[\nabla\log\frac{\mathrm{d}\mu_n}{\mathrm{d}\pi}^{\top}\nabla h\right]
=-(1+\lambda)\mathbb{E}_{\pi}\left[\nabla\frac{\mathrm{d}\mu_n}{\mathrm{d}\pi}^{\top}\nabla h\right]\\
&=-2(1+\lambda)\left\lVert\nabla\frac{\mathrm{d}\mu_n}{\mathrm{d}\pi}\right\rVert_{L^2(\pi)}^2
+(1+\lambda)\mathbb{E}_{\pi}\left[\nabla\frac{\mathrm{d}\mu_n}{\mathrm{d}\pi}^{\top}\left(2\nabla\frac{\mathrm{d}\mu_n}{\mathrm{d}\pi}-\nabla h\right)\right]\\
&=-2(1+\lambda)\left\lVert\nabla\frac{\mathrm{d}\mu_n}{\mathrm{d}\pi}\right\rVert_{L^2(\pi)}^2\\
&\quad+(1+\lambda)\int\left(\nabla\frac{\mathrm{d}\mu_n}{\mathrm{d}\pi}(x)\right)^{\top}\left(2\nabla\left(\frac{\mathrm{d}\mu_n}{\mathrm{d}\pi}(x)-1\right)-\nabla h(x)\right)\pi(x)\,\mathrm{d}x\\
&=-2(1+\lambda)\left\lVert\nabla\frac{\mathrm{d}\mu_n}{\mathrm{d}\pi}\right\rVert_{L^2(\pi)}^2\\
&\quad+(1+\lambda)\int\left(2\left(\frac{\mathrm{d}\mu_n}{\mathrm{d}\pi}(x)-1\right)-h(x)\right)\nabla\cdot\left(\pi(x)\nabla\frac{\mathrm{d}\mu_n}{\mathrm{d}\pi}(x)\right)\,\mathrm{d}x\\
&\le-(1+\lambda)\frac{2}{C_P^2}\chi^2(\mu_n\Vert\pi)
+(1+\lambda)\left\lVert h-2\left(\frac{\mathrm{d}\mu_n}{\mathrm{d}\pi}-1\right)\right\rVert_{L^2(\pi)}
\left\lVert\frac{\nabla\cdot\left(\pi\nabla\frac{\mathrm{d}\mu_n}{\mathrm{d}\pi}\right)}{\pi}\right\rVert_{L^2(\pi)}. \tag*{(105)}
\end{aligned} d t d KL ( ρ t ∥ π ) t = 0 = − ( 1 + λ ) E μ n [ ∇ log d π d μ n ⊤ ∇ h ] = − ( 1 + λ ) E π [ ∇ d π d μ n ⊤ ∇ h ] = − 2 ( 1 + λ ) ∇ d π d μ n L 2 ( π ) 2 + ( 1 + λ ) E π [ ∇ d π d μ n ⊤ ( 2∇ d π d μ n − ∇ h ) ] = − 2 ( 1 + λ ) ∇ d π d μ n L 2 ( π ) 2 + ( 1 + λ ) ∫ ( ∇ d π d μ n ( x ) ) ⊤ ( 2∇ ( d π d μ n ( x ) − 1 ) − ∇ h ( x ) ) π ( x ) d x = − 2 ( 1 + λ ) ∇ d π d μ n L 2 ( π ) 2 + ( 1 + λ ) ∫ ( 2 ( d π d μ n ( x ) − 1 ) − h ( x ) ) ∇ ⋅ ( π ( x ) ∇ d π d μ n ( x ) ) d x ≤ − ( 1 + λ ) C P 2 2 χ 2 ( μ n ∥ π ) + ( 1 + λ ) h − 2 ( d π d μ n − 1 ) L 2 ( π ) π ∇ ⋅ ( π ∇ d π d μ n ) L 2 ( π ) . (105) where the fourth equality uses an integration by parts, and the last inequality uses Poincaré inequality for the first term under similar arguments in Section B.1.8 and uses Cauchy-Schwarz for the second term. Using Theorem B.5 and the derivations in (B.24), (B.27) can be further upper bounded by
d d t ∣ t = 0 KL ( ρ t ∥ π ) ≤ − ( 1 + λ ) 2 C P 2 χ 2 ( μ n ∥ π ) + 2 ( 1 + λ ) λ r Q ( J + I ) . (106) \left.\frac{d}{dt}\right|_{t=0}\operatorname{KL}(\rho_t\Vert\pi) \le-(1+\lambda)\frac{2}{C_P^2}\chi^2(\mu_n\Vert\pi)+2(1+\lambda)\lambda^r Q(\mathcal{J}+\mathcal{I}). \tag*{(106)} d t d t = 0 KL ( ρ t ∥ π ) ≤ − ( 1 + λ ) C P 2 2 χ 2 ( μ n ∥ π ) + 2 ( 1 + λ ) λ r Q ( J + I ) . (106) Then, for the second term of (B.26), we know from Example 15.9 of Villani et al. (2009) (taking m = 1 m=1 m = 1
d 2 d t 2 KL ( ρ t ∥ π ) = ( 1 + λ ) 2 ∫ ∇ h ( x ) T H V ( φ t ( x ) ) ∇ h ( x ) d μ n ( x ) + ( 1 + λ ) 2 ∫ ∥ H h ( x ) ( Id − t ( 1 + λ ) H h ( x ) ) − 1 ∥ F 2 d μ n ( x ) . \frac{d^2}{dt^2}\operatorname{KL}(\rho_t\Vert\pi)=(1+\lambda)^2\int\nabla h(x)^\mathsf{T}\mathbf{H}V(\varphi_t(x))\nabla h(x)d\mu_n(x)
+(1+\lambda)^2\int\left\Vert\mathbf{H}h(x)\left(\operatorname{Id}-t(1+\lambda)\mathbf{H}h(x)\right)^{-1}\right\Vert_F^2d\mu_n(x). d t 2 d 2 KL ( ρ t ∥ π ) = ( 1 + λ ) 2 ∫ ∇ h ( x ) T H V ( φ t ( x )) ∇ h ( x ) d μ n ( x ) + ( 1 + λ ) 2 ∫ H h ( x ) ( Id − t ( 1 + λ ) H h ( x ) ) − 1 F 2 d μ n ( x ) . Because 2 ( 1 + λ ) γ χ 2 ( μ 0 ∥ π ) K 2 d λ ≤ ζ − 1 ζ 2(1+\lambda)\gamma\sqrt{\chi^2(\mu_0\Vert\pi)}\frac{K_{2d}}{\lambda}\leq\frac{\zeta-1}{\zeta} 2 ( 1 + λ ) γ χ 2 ( μ 0 ∥ π ) λ K 2 d ≤ ζ ζ − 1 1 < ζ < 2 1<\zeta<2 1 < ζ < 2
d 2 d t 2 KL ( ρ t ∥ π ) ≤ 4 ( 1 + λ ) 2 β χ 2 ( μ n ∥ π ) K 1 d λ + 4 ( 1 + λ ) 2 ζ 2 χ 2 ( μ n ∥ π ) K 2 d λ . (107) \frac{d^2}{dt^2}\operatorname{KL}(\rho_t\Vert\pi)\leq4(1+\lambda)^2\beta\chi^2(\mu_n\Vert\pi)\frac{K_{1d}}{\lambda}+4(1+\lambda)^2\zeta^2\chi^2(\mu_n\Vert\pi)\frac{K_{2d}}{\lambda}. \tag*{(107)} d t 2 d 2 KL ( ρ t ∥ π ) ≤ 4 ( 1 + λ ) 2 β χ 2 ( μ n ∥ π ) λ K 1 d + 4 ( 1 + λ ) 2 ζ 2 χ 2 ( μ n ∥ π ) λ K 2 d . (107) Combining the above two inequalities (B.28) and (B.29) and plugging them back into (B.26), we obtain
KL ( μ n + 1 ∥ π ) − KL ( μ n ∥ π ) ≤ − 2 C P 2 ( 1 + λ ) 2 χ 2 ( μ n ∥ π ) γ + 2 ( 1 + λ ) λ r Q ( J + I ) + 2 ( 1 + λ ) 2 γ 2 ( β + ζ 2 ) χ 2 ( μ n ∥ π ) K 1 d + K 2 d λ ≤ − 2 C P 2 χ 2 ( μ n ∥ π ) γ + 4 γ λ r Q ( J + I ) + 8 γ 2 ( β + ζ 2 ) χ 2 ( μ n ∥ π ) K 1 d + K 2 d λ , \begin{aligned}
\operatorname{KL}(\mu_{n+1}\Vert\pi)-\operatorname{KL}(\mu_n\Vert\pi)
&\leq-\frac{2}{C_P^2}(1+\lambda)^2\chi^2(\mu_n\Vert\pi)\gamma\\
&\quad+2(1+\lambda)\lambda^r Q(\mathcal{J}+\mathcal{I})\\
&\quad+2(1+\lambda)^2\gamma^2(\beta+\zeta^2)\chi^2(\mu_n\Vert\pi)\frac{K_{1d}+K_{2d}}{\lambda}\\
&\leq-\frac{2}{C_P^2}\chi^2(\mu_n\Vert\pi)\gamma\\
&\quad+4\gamma\lambda^r Q(\mathcal{J}+\mathcal{I})\\
&\quad+8\gamma^2(\beta+\zeta^2)\chi^2(\mu_n\Vert\pi)\frac{K_{1d}+K_{2d}}{\lambda},
\end{aligned} KL ( μ n + 1 ∥ π ) − KL ( μ n ∥ π ) ≤ − C P 2 2 ( 1 + λ ) 2 χ 2 ( μ n ∥ π ) γ + 2 ( 1 + λ ) λ r Q ( J + I ) + 2 ( 1 + λ ) 2 γ 2 ( β + ζ 2 ) χ 2 ( μ n ∥ π ) λ K 1 d + K 2 d ≤ − C P 2 2 χ 2 ( μ n ∥ π ) γ + 4 γ λ r Q ( J + I ) + 8 γ 2 ( β + ζ 2 ) χ 2 ( μ n ∥ π ) λ K 1 d + K 2 d , where the last inequality holds by using 0 < λ ≤ 1 0<\lambda\leq1 0 < λ ≤ 1
Proof of Theorem 4.21
In order to use Theorem 4.18 in the proof, first we are going to show that Theorem 4.18 holds under the conditions of Theorem 4.21 . Notice that conditions 1-4 of Theorem 4.18 are precisely the conditions 1-4 of Theorem 4.21 . To use Theorem 4.18 in the proof of Theorem 4.21 , the only thing left is to check that the condition of step size γ \gamma γ (26) is satisfied.
In Theorem 4.21 , λ n \lambda_n λ n ( 2 γ χ 2 ( μ n ∥ π ) ( β + ζ 2 ) ( K 1 d + K 2 d ) Q ( J + I ) ) 1 r + 1 ∧ 1 \left(2\gamma\frac{\chi^2(\mu_n\|\pi)(\beta+\zeta^2)(K_{1d}+K_{2d})}{Q(\mathcal{J}+\mathcal{I})}\right)^{\frac{1}{r+1}} \wedge1 ( 2 γ Q ( J + I ) χ 2 ( μ n ∥ π ) ( β + ζ 2 ) ( K 1 d + K 2 d ) ) r + 1 1 ∧ 1 λ n \lambda_n λ n
2 γ ( 1 + λ n ) χ 2 ( μ n ∥ π ) K 2 d λ n ≤ 4 γ χ 2 ( μ n ∥ π ) K 2 d λ n = 2 ( 2 γ ) 2 r + 1 2 r + 2 χ 2 ( μ n ∥ π ) r 2 r + 2 ( Q ( J + I ) ) 1 2 r + 2 ( 1 K 1 d + K 2 d 1 β + ζ 2 ) 1 2 r + 2 K 2 d 1 2 \begin{aligned}
2\gamma(1+\lambda_n)\sqrt{\chi^2(\mu_n\|\pi)\frac{K_{2d}}{\lambda_n}} \le4\gamma\sqrt{\chi^2(\mu_n\|\pi)\frac{K_{2d}}{\lambda_n}} \\
&=2(2\gamma)^{\frac{2r+1}{2r+2}}\chi^2(\mu_n\|\pi)^{\frac{r}{2r+2}}\left(Q(\mathcal{J}+\mathcal{I})\right)^{\frac{1}{2r+2}}\left(\frac{1}{K_{1d}+K_{2d}}\frac{1}{\beta+\zeta^2}\right)^{\frac{1}{2r+2}}K_{2d}^{\frac{1}{2}}
\end{aligned} 2 γ ( 1 + λ n ) χ 2 ( μ n ∥ π ) λ n K 2 d ≤ 4 γ χ 2 ( μ n ∥ π ) λ n K 2 d = 2 ( 2 γ ) 2 r + 2 2 r + 1 χ 2 ( μ n ∥ π ) 2 r + 2 r ( Q ( J + I ) ) 2 r + 2 1 ( K 1 d + K 2 d 1 β + ζ 2 1 ) 2 r + 2 1 K 2 d 2 1 ≤ ( ∗ ) ( 8 γ ) 2 r + 1 2 r + 2 Q 2 r + 1 2 r + 2 ( J + I ) 1 2 r + 2 ( 1 β + ζ 2 ) 1 2 r + 2 K 2 d r 2 r + 2 ≤ ζ − 1 ζ ( 1 β + ζ 2 ) r 2 r + 2 ≤ ζ − 1 ζ . (108) \begin{aligned}
\mathrel{\overset{(*)}{\le}} (8\gamma)^{\frac{2r+1}{2r+2}}Q^{\frac{2r+1}{2r+2}}(\mathcal{J}+\mathcal{I})^{\frac{1}{2r+2}}\left(\frac{1}{\beta+\zeta^2}\right)^{\frac{1}{2r+2}}K_{2d}^{\frac{r}{2r+2}} \\
&\le\frac{\zeta-1}{\zeta}\left(\frac{1}{\beta+\zeta^2}\right)^{\frac{r}{2r+2}} \le\frac{\zeta-1}{\zeta}. \tag*{(108)}
\end{aligned} ≤ ( ∗ ) ( 8 γ ) 2 r + 2 2 r + 1 Q 2 r + 2 2 r + 1 ( J + I ) 2 r + 2 1 ( β + ζ 2 1 ) 2 r + 2 1 K 2 d 2 r + 2 r ≤ ζ ζ − 1 ( β + ζ 2 1 ) 2 r + 2 r ≤ ζ ζ − 1 . (108) ( ∗ ) (*) ( ∗ ) K 2 d K 1 d + K 2 d ≤ 1 \frac{K_{2d}}{K_{1d}+K_{2d}}\le1 K 1 d + K 2 d K 2 d ≤ 1
χ 2 ( μ n ∥ π ) = ∥ d μ n d π − 1 ∥ L 2 ( π ) 2 = ∥ T r π q n ∥ L 2 ( π ) 2 ≤ K 2 r Q 2 ≤ Q 2 . (109) \chi^2(\mu_n\|\pi)=\left\|\frac{d\mu_n}{d\pi}-1\right\|_{L^2(\pi)}^2=\|T^r\pi q_n\|_{L^2(\pi)}^2\le K^{2r}Q^2\le Q^2. \tag*{(109)} χ 2 ( μ n ∥ π ) = d π d μ n − 1 L 2 ( π ) 2 = ∥ T r π q n ∥ L 2 ( π ) 2 ≤ K 2 r Q 2 ≤ Q 2 . (109) The second last inequality of (B.30) holds due to the constraint on γ \gamma γ (29) , and the last inequality of (B.30) holds because β + ζ 2 ≥ 1 \beta+\zeta^2\ge1 β + ζ 2 ≥ 1
On the other hand, if λ n \lambda_n λ n γ \gamma γ (29) , we have
2 γ ( 1 + λ n ) χ 2 ( μ n ∥ π ) K 2 d λ n = 4 γ χ 2 ( μ n ∥ π ) K 2 d ≤ ζ − 1 ζ . 2\gamma(1+\lambda_n)\chi^2(\mu_n\|\pi)\frac{K_{2d}}{\lambda_n}=4\gamma\chi^2(\mu_n\|\pi)K_{2d}\le\frac{\zeta-1}{\zeta}. 2 γ ( 1 + λ n ) χ 2 ( μ n ∥ π ) λ n K 2 d = 4 γ χ 2 ( μ n ∥ π ) K 2 d ≤ ζ ζ − 1 . Therefore, all the conditions of Theorem 4.18 have been verified. So, if we select
λ n = ( 2 γ χ 2 ( μ n ∥ π ) ( β + ζ 2 ) ( K 1 d + K 2 d ) Q ( J + I ) ) 1 r + 1 ∧ 1 , \lambda_n=\left(2\gamma\frac{\chi^2(\mu_n\|\pi)(\beta+\zeta^2)(K_{1d}+K_{2d})}{Q(\mathcal{J}+\mathcal{I})}\right)^{\frac{1}{r+1}}\wedge1, λ n = ( 2 γ Q ( J + I ) χ 2 ( μ n ∥ π ) ( β + ζ 2 ) ( K 1 d + K 2 d ) ) r + 1 1 ∧ 1 , then
KL ( μ n + 1 ∥ π ) − KL ( μ n ∥ π ) ≤ − γ 2 C P χ 2 ( μ n ∥ π ) + 4 γ λ n r Q ( J + I ) + 8 γ 2 ( β + ζ 2 ) χ 2 ( μ n ∥ π ) K 1 d + K 2 d λ n . (110) \begin{aligned}
\operatorname{KL}(\mu_{n+1}\|\pi)-\operatorname{KL}(\mu_n\|\pi)
&\leq-\gamma\frac{2}{C_P}\chi^2(\mu_n\|\pi)\\
&\quad+4\gamma\lambda_n^rQ(\mathcal{J}+\mathcal{I})
+8\gamma^2(\beta+\zeta^2)\chi^2(\mu_n\|\pi)\frac{K_{1d}+K_{2d}}{\lambda_n}. \tag*{(110)}
\end{aligned} KL ( μ n + 1 ∥ π ) − KL ( μ n ∥ π ) ≤ − γ C P 2 χ 2 ( μ n ∥ π ) + 4 γ λ n r Q ( J + I ) + 8 γ 2 ( β + ζ 2 ) χ 2 ( μ n ∥ π ) λ n K 1 d + K 2 d . (110) By observing (B.32), the first term on the right-hand side − 2 C P χ 2 ( μ n ∥ π ) γ -\frac{2}{C_P}\chi^2(\mu_n\|\pi)\gamma − C P 2 χ 2 ( μ n ∥ π ) γ n n n ( Δ 1 ) : = 4 γ λ n r Q ( J + I ) (\Delta_1):=4\gamma\lambda_n^rQ(\mathcal{J}+\mathcal{I}) ( Δ 1 ) := 4 γ λ n r Q ( J + I ) ( Δ 2 ) : = 8 γ 2 ( β + ζ 2 ) χ 2 ( μ n ∥ π ) K 1 d + K 2 d λ n (\Delta_2):=8\gamma^2(\beta+\zeta^2)\chi^2(\mu_n\|\pi)\frac{K_{1d}+K_{2d}}{\lambda_n} ( Δ 2 ) := 8 γ 2 ( β + ζ 2 ) χ 2 ( μ n ∥ π ) λ n K 1 d + K 2 d G ( λ n ) = ( Δ 1 ) + ( Δ 2 ) G(\lambda_n)=(\Delta_1)+(\Delta_2) G ( λ n ) = ( Δ 1 ) + ( Δ 2 ) λ n \lambda_n λ n d d λ n G ( λ n ) = 0 \frac{d}{d\lambda_n}G(\lambda_n)=0 d λ n d G ( λ n ) = 0 λ n = ( 2 γ χ 2 ( μ n ∥ π ) ( β + ζ 2 ) ( K 1 d + K 2 d ) Q ( J + I ) ) 1 r + 1 \lambda_n=\left(2\gamma\frac{\chi^2(\mu_n\|\pi)(\beta+\zeta^2)(K_{1d}+K_{2d})}{Q(\mathcal{J}+\mathcal{I})}\right)^{\frac{1}{r+1}} λ n = ( 2 γ Q ( J + I ) χ 2 ( μ n ∥ π ) ( β + ζ 2 ) ( K 1 d + K 2 d ) ) r + 1 1 λ n \lambda_n λ n
KL ( μ n + 1 ∥ π ) − KL ( μ n ∥ π ) ≤ − 2 C P χ 2 ( μ n ∥ π ) γ + 4 γ ( 2 γ ) r r + 1 ( χ 2 ( μ n ∥ π ) ( β + ζ 2 ) ( K 1 d + K 2 d ) ) r r + 1 × ( Q ( J + I ) ) 1 r + 1 ≤ − 2 C P χ 2 ( μ n ∥ π ) γ + 8 γ 1 + r r + 1 ( χ 2 ( μ n ∥ π ) ( β + ζ 2 ) ( K 1 d + K 2 d ) ) r r + 1 × ( Q ( J + I ) ) 1 r + 1 ≤ − 2 C P χ 2 ( μ n ∥ π ) γ + 8 γ 1 + r r + 1 Q 2 r + 1 r + 1 ( ( K 1 d + K 2 d ) ( β + ζ 2 ) ) r r + 1 ( J + I ) 1 r + 1 , (111) \begin{aligned}
\operatorname{KL}(\mu_{n+1}\|\pi)-\operatorname{KL}(\mu_n\|\pi)
&\leq-\frac{2}{C_P}\chi^2(\mu_n\|\pi)\gamma
+4\gamma(2\gamma)^{\frac{r}{r+1}}
\left(\chi^2(\mu_n\|\pi)(\beta+\zeta^2)(K_{1d}+K_{2d})\right)^{\frac{r}{r+1}}\\
&\quad\times\left(Q(\mathcal{J}+\mathcal{I})\right)^{\frac{1}{r+1}}\\
&\leq-\frac{2}{C_P}\chi^2(\mu_n\|\pi)\gamma
+8\gamma^{1+\frac{r}{r+1}}
\left(\chi^2(\mu_n\|\pi)(\beta+\zeta^2)(K_{1d}+K_{2d})\right)^{\frac{r}{r+1}}\\
&\quad\times\left(Q(\mathcal{J}+\mathcal{I})\right)^{\frac{1}{r+1}}\\
&\leq-\frac{2}{C_P}\chi^2(\mu_n\|\pi)\gamma
+8\gamma^{1+\frac{r}{r+1}}Q^{\frac{2r+1}{r+1}}
\left((K_{1d}+K_{2d})(\beta+\zeta^2)\right)^{\frac{r}{r+1}}
(\mathcal{J}+\mathcal{I})^{\frac{1}{r+1}}, \tag*{(111)}
\end{aligned} KL ( μ n + 1 ∥ π ) − KL ( μ n ∥ π ) ≤ − C P 2 χ 2 ( μ n ∥ π ) γ + 4 γ ( 2 γ ) r + 1 r ( χ 2 ( μ n ∥ π ) ( β + ζ 2 ) ( K 1 d + K 2 d ) ) r + 1 r × ( Q ( J + I ) ) r + 1 1 ≤ − C P 2 χ 2 ( μ n ∥ π ) γ + 8 γ 1 + r + 1 r ( χ 2 ( μ n ∥ π ) ( β + ζ 2 ) ( K 1 d + K 2 d ) ) r + 1 r × ( Q ( J + I ) ) r + 1 1 ≤ − C P 2 χ 2 ( μ n ∥ π ) γ + 8 γ 1 + r + 1 r Q r + 1 2 r + 1 ( ( K 1 d + K 2 d ) ( β + ζ 2 ) ) r + 1 r ( J + I ) r + 1 1 , (111) where the last inequality holds because of (B.31). Since χ 2 ( μ n ∥ π ) ≥ KL ( μ n ∥ π ) \chi^2(\mu_n\|\pi)\geq\operatorname{KL}(\mu_n\|\pi) χ 2 ( μ n ∥ π ) ≥ KL ( μ n ∥ π )
KL ( μ n + 1 ∥ π ) ≤ ( 1 − γ 2 C P ) KL ( μ n ∥ π ) + 8 γ 1 + r r + 1 Q 2 r + 1 r + 1 ( ( K 1 d + K 2 d ) ( β + ζ 2 ) ) r r + 1 ( J + I ) 1 r + 1 . \begin{aligned}
\operatorname{KL}(\mu_{n+1}\|\pi)\leq\left(1-\gamma\frac{2}{C_P}\right)\operatorname{KL}(\mu_n\|\pi) \\
&+ 8\gamma^{1+\frac{r}{r+1}} Q^{\frac{2r+1}{r+1}} \left((K_{1d}+K_{2d})(\beta+\zeta^2)\right)^{\frac{r}{r+1}}(\mathcal{J}+\mathcal{I})^{\frac{1}{r+1}}.
\end{aligned} KL ( μ n + 1 ∥ π ) ≤ ( 1 − γ C P 2 ) KL ( μ n ∥ π ) + 8 γ 1 + r + 1 r Q r + 1 2 r + 1 ( ( K 1 d + K 2 d ) ( β + ζ 2 ) ) r + 1 r ( J + I ) r + 1 1 . After iterating, we obtain
KL ( μ n max ∥ π ) ≤ ( 1 − γ 2 C P ) n max KL ( μ 0 ∥ π ) + 4 γ r r + 1 C P Q 2 r + 1 r + 1 ( ( K 1 d + K 2 d ) ( β + ζ 2 ) ) r r + 1 × ( J + I ) 1 r + 1 ≤ exp ( − 2 n max γ C P ) KL ( μ 0 ∥ π ) + 4 γ r r + 1 C P Q 2 r + 1 r + 1 ( ( K 1 d + K 2 d ) ( β + ζ 2 ) ) r r + 1 × ( J + I ) 1 r + 1 \begin{aligned}
\operatorname{KL}(\mu_{n_{\max}}\mathbin\Vert\pi) \leq\left(1-\gamma\frac{2}{C_P}\right)^{n_{\max}}\operatorname{KL}(\mu_0\mathbin\Vert\pi) \\
&+4\gamma^{\frac{r}{r+1}}C_PQ^{\frac{2r+1}{r+1}}\left((K_{1d}+K_{2d})(\beta+\zeta^2)\right)^{\frac{r}{r+1}} \\
&\times(\mathcal{J}+\mathcal{I})^{\frac{1}{r+1}} \\
&\leq\exp\left(-\frac{2n_{\max}\gamma}{C_P}\right)\operatorname{KL}(\mu_0\mathbin\Vert\pi)+4\gamma^{\frac{r}{r+1}}C_PQ^{\frac{2r+1}{r+1}}\left((K_{1d}+K_{2d})(\beta+\zeta^2)\right)^{\frac{r}{r+1}} \\
&\times(\mathcal{J}+\mathcal{I})^{\frac{1}{r+1}}
\end{aligned} KL ( μ n m a x ∥ π ) ≤ ( 1 − γ C P 2 ) n m a x KL ( μ 0 ∥ π ) + 4 γ r + 1 r C P Q r + 1 2 r + 1 ( ( K 1 d + K 2 d ) ( β + ζ 2 ) ) r + 1 r × ( J + I ) r + 1 1 ≤ exp ( − C P 2 n m a x γ ) KL ( μ 0 ∥ π ) + 4 γ r + 1 r C P Q r + 1 2 r + 1 ( ( K 1 d + K 2 d ) ( β + ζ 2 ) ) r + 1 r × ( J + I ) r + 1 1 and the result follows.
Proof of the second discrete-time KL theorem
B.1.11 Proof of Theorem 4.22 The proof of Theorem 4.22 is also based on Theorem 4.18 proved in the last section. Recalling (B.33) yet with adaptive step size γ n \gamma_n γ n
KL ( μ n + 1 ∥ π ) − KL ( μ n ∥ π ) ≤ − 2 C P χ 2 ( μ n ∥ π ) γ n + 8 γ n 1 + r r + 1 × ( χ 2 ( μ n ∥ π ) ( β + ζ 2 ) ( K 1 d + K 2 d ) ) r r + 1 ( Q ( J + I ) ) 1 r + 1 . \begin{aligned}
\operatorname{KL}(\mu_{n+1}\mathbin\Vert\pi)-\operatorname{KL}(\mu_n\mathbin\Vert\pi)
\leq-\frac{2}{C_P}\chi^2(\mu_n\mathbin\Vert\pi)\gamma_n
+8\gamma_n^{1+\frac{r}{r+1}} \\
&\times\left(\chi^2(\mu_n\mathbin\Vert\pi)(\beta+\zeta^2)(K_{1d}+K_{2d})\right)^{\frac{r}{r+1}}
\left(Q(\mathcal{J}+\mathcal{I})\right)^{\frac{1}{r+1}}.
\end{aligned} KL ( μ n + 1 ∥ π ) − KL ( μ n ∥ π ) ≤ − C P 2 χ 2 ( μ n ∥ π ) γ n + 8 γ n 1 + r + 1 r × ( χ 2 ( μ n ∥ π ) ( β + ζ 2 ) ( K 1 d + K 2 d ) ) r + 1 r ( Q ( J + I ) ) r + 1 1 . From (4.19), we have
8 γ n r r + 1 ( ( K 1 d + K 2 d ) ( β + ζ 2 ) ) r r + 1 ( Q ( J + I ) ) 1 r + 1 ≤ 1 C P χ 2 ( μ n ∥ π ) 1 r + 1 , 8\gamma_n^{\frac{r}{r+1}}\left((K_{1d}+K_{2d})(\beta+\zeta^2)\right)^{\frac{r}{r+1}}
\left(Q(\mathcal{J}+\mathcal{I})\right)^{\frac{1}{r+1}}
\leq\frac{1}{C_P}\chi^2(\mu_n\mathbin\Vert\pi)^{\frac{1}{r+1}}, 8 γ n r + 1 r ( ( K 1 d + K 2 d ) ( β + ζ 2 ) ) r + 1 r ( Q ( J + I ) ) r + 1 1 ≤ C P 1 χ 2 ( μ n ∥ π ) r + 1 1 , so that we have
KL ( μ n + 1 ∥ π ) − KL ( μ n ∥ π ) ≤ − 1 C P χ 2 ( μ n ∥ π ) γ n ≤ − 1 C P KL ( μ n ∥ π ) γ n . \operatorname{KL}(\mu_{n+1}\mathbin\Vert\pi)-\operatorname{KL}(\mu_n\mathbin\Vert\pi)
\leq-\frac{1}{C_P}\chi^2(\mu_n\mathbin\Vert\pi)\gamma_n
\leq-\frac{1}{C_P}\operatorname{KL}(\mu_n\mathbin\Vert\pi)\gamma_n. KL ( μ n + 1 ∥ π ) − KL ( μ n ∥ π ) ≤ − C P 1 χ 2 ( μ n ∥ π ) γ n ≤ − C P 1 KL ( μ n ∥ π ) γ n . Hence
KL ( μ n + 1 ∥ π ) ≤ ( 1 − 1 C P γ n ) KL ( μ n ∥ π ) . (112) \operatorname{KL}(\mu_{n+1}\Vert\pi) \le\left(1-\frac{1}{C_P}\gamma_n\right)\operatorname{KL}(\mu_n\Vert\pi). \tag*{(112)} KL ( μ n + 1 ∥ π ) ≤ ( 1 − C P 1 γ n ) KL ( μ n ∥ π ) . (112) After iterating n n n 1 1 1 n max n_{\max} n m a x
Finite-sample expression for DrMMD
By defining the following operators,
S x : H → R M , f ↦ 1 M [ f ( x ( 1 ) ) , … , f ( x ( M ) ) ] T , S_x:\mathcal{H}\to\mathbb{R}^{M},\quad f\mapsto\frac{1}{\sqrt{M}}\left[f\left(x^{(1)}\right),\ldots,f\left(x^{(M)}\right)\right]^{\mathsf{T}}, S x : H → R M , f ↦ M 1 [ f ( x ( 1 ) ) , … , f ( x ( M ) ) ] T , S x ∗ : R M → H , α ↦ 1 M ∑ i = 1 M α i k ( x ( i ) , ⋅ ) , S_x^*:\mathbb{R}^{M}\to\mathcal{H},\quad\alpha\mapsto\frac{1}{\sqrt{M}}\sum_{i=1}^{M}\alpha_i k\left(x^{(i)},\cdot\right), S x ∗ : R M → H , α ↦ M 1 i = 1 ∑ M α i k ( x ( i ) , ⋅ ) , S y : H → R N , f ↦ 1 N [ f ( y ( 1 ) ) , … , f ( y ( N ) ) ] T , S_y:\mathcal{H}\to\mathbb{R}^{N},\quad f\mapsto\frac{1}{\sqrt{N}}\left[f\left(y^{(1)}\right),\ldots,f\left(y^{(N)}\right)\right]^{\mathsf{T}}, S y : H → R N , f ↦ N 1 [ f ( y ( 1 ) ) , … , f ( y ( N ) ) ] T , S y ∗ : R N → H , α ↦ 1 N ∑ i = 1 N α i k ( y ( i ) , ⋅ ) . S_y^*:\mathbb{R}^{N}\to\mathcal{H},\quad\alpha\mapsto\frac{1}{\sqrt{N}}\sum_{i=1}^{N}\alpha_i k\left(y^{(i)},\cdot\right). S y ∗ : R N → H , α ↦ N 1 i = 1 ∑ N α i k ( y ( i ) , ⋅ ) . Then we have
Σ ^ π = S x ∗ S x , K x x = M S x S x ∗ , K x y = M N S x S y ∗ . \widehat{\Sigma}_{\pi}=S_x^*S_x,\qquad K_{xx}=MS_xS_x^*,\qquad K_{xy}=\sqrt{MN}S_xS_y^*. Σ π = S x ∗ S x , K xx = M S x S x ∗ , K x y = M N S x S y ∗ . Using these, note that
h μ ^ , π ^ ∗ = 2 ( Σ ^ π + λ Id ) − 1 ( m μ ^ − m π ^ ) = 2 ( 1 M ∑ i = 1 M k ( x ( i ) , ⋅ ) ⊗ k ( x ( i ) , ⋅ ) + λ Id ) − 1 × ( 1 N ∑ i = 1 N k ( y ( i ) , ⋅ ) − 1 M ∑ i = 1 M k ( x ( i ) , ⋅ ) ) = 2 ( S x ∗ S x + λ Id ) − 1 ( 1 N S y ∗ 1 N − 1 M S x ∗ 1 M ) . (113) \begin{aligned}
h_{\widehat{\mu},\widehat{\pi}}^*
&=2\left(\widehat{\Sigma}_{\pi}+\lambda\operatorname{Id}\right)^{-1}\left(m_{\widehat{\mu}}-m_{\widehat{\pi}}\right)\\
&=2\left(\frac{1}{M}\sum_{i=1}^{M}k\left(x^{(i)},\cdot\right)\otimes k\left(x^{(i)},\cdot\right)+\lambda\operatorname{Id}\right)^{-1}\\
&\qquad\times\left(\frac{1}{N}\sum_{i=1}^{N}k\left(y^{(i)},\cdot\right)-\frac{1}{M}\sum_{i=1}^{M}k\left(x^{(i)},\cdot\right)\right)\\
&=2\left(S_x^*S_x+\lambda\operatorname{Id}\right)^{-1}\left(\frac{1}{\sqrt{N}}S_y^*\mathbf{1}_N-\frac{1}{\sqrt{M}}S_x^*\mathbf{1}_M\right). \tag*{(113)}
\end{aligned} h μ , π ∗ = 2 ( Σ π + λ Id ) − 1 ( m μ − m π ) = 2 ( M 1 i = 1 ∑ M k ( x ( i ) , ⋅ ) ⊗ k ( x ( i ) , ⋅ ) + λ Id ) − 1 × ( N 1 i = 1 ∑ N k ( y ( i ) , ⋅ ) − M 1 i = 1 ∑ M k ( x ( i ) , ⋅ ) ) = 2 ( S x ∗ S x + λ Id ) − 1 ( N 1 S y ∗ 1 N − M 1 S x ∗ 1 M ) . (113) From the Woodbury inversion lemma, we have that
( S x ∗ S x + λ Id ) − 1 = 1 λ Id − 1 λ S x ∗ ( S x S x ∗ + λ Id ) − 1 S x . \left(S_x^*S_x+\lambda\operatorname{Id}\right)^{-1}=\frac{1}{\lambda}\operatorname{Id}-\frac{1}{\lambda}S_x^*\left(S_xS_x^*+\lambda\operatorname{Id}\right)^{-1}S_x. ( S x ∗ S x + λ Id ) − 1 = λ 1 Id − λ 1 S x ∗ ( S x S x ∗ + λ Id ) − 1 S x . Plugging the above into (B.35), we obtain
h μ ^ , π ^ ∗ = 2 ( 1 λ Id − 1 λ S x ∗ ( S x S x ∗ + λ Id ) − 1 S x ) × ( 1 N S y ∗ 1 N − 1 M S x ∗ 1 M ) = 2 λ ( 1 N S y ∗ 1 N − 1 M S x ∗ 1 M ) − 2 λ S x ∗ ( 1 M K x x + λ Id ) − 1 1 N M K x y 1 N + 2 λ S x ∗ ( 1 M K x x + λ Id ) − 1 1 M M K x x 1 M = 2 N λ k ( ⋅ , y 1 : N ) 1 N − 2 M λ k ( ⋅ , x 1 : M ) 1 M − 2 N λ k ( ⋅ , x 1 : M ) ( K x x + M λ Id ) − 1 K x y 1 N + 2 M λ k ( ⋅ , x 1 : M ) ( K x x + M λ Id ) − 1 K x x 1 M . (114) \begin{aligned}
h^*_{\hat{\mu},\hat{\pi}}={}&2\left(\frac{1}{\lambda}\operatorname{Id}-\frac{1}{\lambda}S_x^*(S_xS_x^*+\lambda\operatorname{Id})^{-1}S_x\right)\\
&\times\left(\frac{1}{\sqrt{N}}S_y^*\mathbf{1}_N-\frac{1}{\sqrt{M}}S_x^*\mathbf{1}_M\right)\\
={}&\frac{2}{\lambda}\left(\frac{1}{\sqrt{N}}S_y^*\mathbf{1}_N-\frac{1}{\sqrt{M}}S_x^*\mathbf{1}_M\right)\\
&-\frac{2}{\lambda}S_x^*\left(\frac{1}{M}K_{xx}+\lambda\operatorname{Id}\right)^{-1}\frac{1}{N\sqrt{M}}K_{xy}\mathbf{1}_N\\
&+\frac{2}{\lambda}S_x^*\left(\frac{1}{M}K_{xx}+\lambda\operatorname{Id}\right)^{-1}\frac{1}{M\sqrt{M}}K_{xx}\mathbf{1}_M\\
={}&\frac{2}{N\lambda}k\left(\cdot,y^{1:N}\right)\mathbf{1}_N-\frac{2}{M\lambda}k\left(\cdot,x^{1:M}\right)\mathbf{1}_M\\
&-\frac{2}{N\lambda}k\left(\cdot,x^{1:M}\right)\left(K_{xx}+M\lambda\operatorname{Id}\right)^{-1}K_{xy}\mathbf{1}_N\\
&+\frac{2}{M\lambda}k\left(\cdot,x^{1:M}\right)\left(K_{xx}+M\lambda\operatorname{Id}\right)^{-1}K_{xx}\mathbf{1}_M. \tag*{(114)}
\end{aligned} h μ ^ , π ^ ∗ = = = 2 ( λ 1 Id − λ 1 S x ∗ ( S x S x ∗ + λ Id ) − 1 S x ) × ( N 1 S y ∗ 1 N − M 1 S x ∗ 1 M ) λ 2 ( N 1 S y ∗ 1 N − M 1 S x ∗ 1 M ) − λ 2 S x ∗ ( M 1 K xx + λ Id ) − 1 N M 1 K x y 1 N + λ 2 S x ∗ ( M 1 K xx + λ Id ) − 1 M M 1 K xx 1 M N λ 2 k ( ⋅ , y 1 : N ) 1 N − M λ 2 k ( ⋅ , x 1 : M ) 1 M − N λ 2 k ( ⋅ , x 1 : M ) ( K xx + M λ Id ) − 1 K x y 1 N + M λ 2 k ( ⋅ , x 1 : M ) ( K xx + M λ Id ) − 1 K xx 1 M . (114) Obtaining DrMMD ( μ ^ ∥ π ^ ) \operatorname{DrMMD}(\hat{\mu}\Vert\hat{\pi}) DrMMD ( μ ^ ∥ π ^ ) h μ ^ , π ^ ∗ h^*_{\hat{\mu},\hat{\pi}} h μ ^ , π ^ ∗
DrMMD ( μ ^ ∥ π ^ ) = ( 1 + λ ) ∥ ( Σ π ^ + λ Id ) − 1 2 ( m μ ^ − m π ^ ) ∥ H 2 = ( 1 + λ ) ⟨ 1 2 h μ ^ , π ^ ∗ , m μ ^ − m π ^ ⟩ H = ( 1 + λ ) ⟨ 1 2 h μ ^ , π ^ ∗ , 1 N ∑ i = 1 N k ( y ( i ) , ⋅ ) − 1 M ∑ i = 1 M k ( x ( i ) , ⋅ ) ⟩ H = 1 + λ λ ( 1 N 2 1 N ⊤ K y y 1 N + 1 M 2 1 M ⊤ K x x 1 M − 2 M N 1 M ⊤ K x y 1 N − 1 N 2 1 N ⊤ K x y ( K x x + M λ Id ) − 1 K x y 1 N + 2 N M 1 M ⊤ K x x ( K x x + M λ Id ) − 1 K x y 1 N − 1 M 2 1 M ⊤ K x x ( K x x + M λ Id ) − 1 K x x 1 M ) . (115) \begin{aligned}
\operatorname{DrMMD}(\hat{\mu}\Vert\hat{\pi})&=(1+\lambda)\left\|(\Sigma_{\hat{\pi}}+\lambda\operatorname{Id})^{-\frac{1}{2}}(m_{\hat{\mu}}-m_{\hat{\pi}})\right\|_{\mathcal{H}}^2\\
&=(1+\lambda)\left\langle\frac{1}{2}h^*_{\hat{\mu},\hat{\pi}},m_{\hat{\mu}}-m_{\hat{\pi}}\right\rangle_{\mathcal{H}}\\
&=(1+\lambda)\left\langle\frac{1}{2}h^*_{\hat{\mu},\hat{\pi}},\frac{1}{N}\sum_{i=1}^{N}k\left(y^{(i)},\cdot\right)-\frac{1}{M}\sum_{i=1}^{M}k\left(x^{(i)},\cdot\right)\right\rangle_{\mathcal{H}}\\
&=\frac{1+\lambda}{\lambda}\left(\frac{1}{N^2}\mathbf{1}_N^\top K_{yy}\mathbf{1}_N+\frac{1}{M^2}\mathbf{1}_M^\top K_{xx}\mathbf{1}_M-\frac{2}{MN}\mathbf{1}_M^\top K_{xy}\mathbf{1}_N\right.\\
&\qquad\left.-\frac{1}{N^2}\mathbf{1}_N^\top K_{xy}(K_{xx}+M\lambda\operatorname{Id})^{-1}K_{xy}\mathbf{1}_N+\frac{2}{NM}\mathbf{1}_M^\top K_{xx}(K_{xx}+M\lambda\operatorname{Id})^{-1}K_{xy}\mathbf{1}_N\right.\\
&\qquad\left.-\frac{1}{M^2}\mathbf{1}_M^\top K_{xx}(K_{xx}+M\lambda\operatorname{Id})^{-1}K_{xx}\mathbf{1}_M\right). \tag*{(115)}
\end{aligned} DrMMD ( μ ^ ∥ π ^ ) = ( 1 + λ ) ( Σ π ^ + λ Id ) − 2 1 ( m μ ^ − m π ^ ) H 2 = ( 1 + λ ) ⟨ 2 1 h μ ^ , π ^ ∗ , m μ ^ − m π ^ ⟩ H = ( 1 + λ ) ⟨ 2 1 h μ ^ , π ^ ∗ , N 1 i = 1 ∑ N k ( y ( i ) , ⋅ ) − M 1 i = 1 ∑ M k ( x ( i ) , ⋅ ) ⟩ H = λ 1 + λ ( N 2 1 1 N ⊤ K y y 1 N + M 2 1 1 M ⊤ K xx 1 M − M N 2 1 M ⊤ K x y 1 N − N 2 1 1 N ⊤ K x y ( K xx + M λ Id ) − 1 K x y 1 N + N M 2 1 M ⊤ K xx ( K xx + M λ Id ) − 1 K x y 1 N − M 2 1 1 M ⊤ K xx ( K xx + M λ Id ) − 1 K xx 1 M ) . (115) Further Background on Wasserstein Space
( P 2 ( R d ) , W 2 ) (\mathcal{P}_2(\mathbb{R}^d), W_2) ( P 2 ( R d ) , W 2 )
Let μ \mu μ π \pi π P 2 ( R d ) \mathcal{P}_2(\mathbb{R}^d) P 2 ( R d ) Π ( μ , π ) \Pi(\mu,\pi) Π ( μ , π ) μ \mu μ π \pi π Π ( μ , π ) = { Γ ∈ P ( R d × R d ) ; ( proj 1 ) # Γ = μ , ( proj 2 ) # Γ = π } \Pi(\mu,\pi) = \{\Gamma\in\mathcal{P}(\mathbb{R}^d \times\mathbb{R}^d); (\operatorname{proj}_1)_\# \Gamma= \mu, (\operatorname{proj}_2)_\# \Gamma= \pi\} Π ( μ , π ) = { Γ ∈ P ( R d × R d ) ; ( proj 1 ) # Γ = μ , ( proj 2 ) # Γ = π } proj 1 \operatorname{proj}_1 proj 1 proj 2 \operatorname{proj}_2 proj 2 ( x , y ) ↦ x (x,y) \mapsto x ( x , y ) ↦ x ( x , y ) ↦ y (x,y) \mapsto y ( x , y ) ↦ y # \# # μ \mu μ π \pi π
W 2 ( μ , π ) = ( inf Γ ∈ Π ( μ , π ) ∫ ∥ x − y ∥ 2 d Γ ( x , y ) ) 1 2 . W_2(\mu,\pi) = \left(\inf_{\Gamma\in\Pi(\mu,\pi)} \int\lVert x-y\rVert^2 \, \mathrm{d}\Gamma(x,y)\right)^{\frac{1}{2}}. W 2 ( μ , π ) = ( Γ ∈ Π ( μ , π ) inf ∫ ∥ x − y ∥ 2 d Γ ( x , y ) ) 2 1 . and ( P 2 ( R d ) , W 2 ) (\mathcal{P}_2(\mathbb{R}^d), W_2) ( P 2 ( R d ) , W 2 ) μ \mu μ Γ ∗ = ( Id , T ) # μ \Gamma^* = (\operatorname{Id},T)_\#\mu Γ ∗ = ( Id , T ) # μ T # μ = π T_\#\mu= \pi T # μ = π T T T T ( x ) = x + ∇ ϕ ( x ) T(x) = x + \nabla\phi(x) T ( x ) = x + ∇ ϕ ( x ) ϕ \phi ϕ μ \mu μ
For an absolutely continuous μ \mu μ T T T T # μ = π T_\#\mu= \pi T # μ = π ( ρ t ) 0 ≤ t ≤ 1 (\rho_t)_{0 \le t \le1} ( ρ t ) 0 ≤ t ≤ 1 μ \mu μ π \pi π
ρ t = ( ( 1 − t ) Id + t T ) # μ = ( Id + t ∇ ϕ ) # μ . \rho_t = ((1-t)\operatorname{Id} + tT)_\#\mu= (\operatorname{Id} + t\nabla\phi)_\#\mu. ρ t = (( 1 − t ) Id + tT ) # μ = ( Id + t ∇ ϕ ) # μ . Therefore, in this paper, we always use ( Id + t ∇ ϕ ) # μ (\operatorname{Id} + t\nabla\phi)_\#\mu ( Id + t ∇ ϕ ) # μ ϕ ∈ C c ∞ ( R d ) \phi\in C_c^\infty(\mathbb{R}^d) ϕ ∈ C c ∞ ( R d ) 2 μ \mu μ φ t ( x ) = x + t ∇ ϕ ( x ) \varphi_t(x) = x + t\nabla\phi(x) φ t ( x ) = x + t ∇ ϕ ( x ) ω t ( x ) = [ ∇ ϕ ∘ φ t − 1 ] ( x ) \omega_t(x) = [\nabla\phi\circ\varphi_t^{-1}](x) ω t ( x ) = [ ∇ ϕ ∘ φ t − 1 ] ( x ) ω t \omega_t ω t ρ t \rho_t ρ t π \pi π ∥ ω t ∥ L 2 ( ρ t ) = ∥ ∇ ϕ ∥ L 2 ( μ ) \lVert\omega_t\rVert_{L^2(\rho_t)} = \lVert\nabla\phi\rVert_{L^2(\mu)} ∥ ω t ∥ L 2 ( ρ t ) = ∥ ∇ ϕ ∥ L 2 ( μ ) t ∈ [ 0 , 1 ] t \in[0,1] t ∈ [ 0 , 1 ] ( ρ t ) 0 ≤ t ≤ 1 (\rho_t)_{0 \le t \le1} ( ρ t ) 0 ≤ t ≤ 1
A functional F : P 2 ( R d ) → R \mathcal{F}:\mathcal{P}_2(\mathbb{R}^d)\to\mathbb{R} F : P 2 ( R d ) → R μ \mu μ π \pi π
F ( ρ t ) ≤ ( 1 − t ) F ( μ ) + t F ( π ) , ∀ t ∈ [ 0 , 1 ] , (116) \mathcal{F}(\rho_t)\leq(1-t)\mathcal{F}(\mu)+t\mathcal{F}(\pi),\qquad\forall t\in[0,1], \tag*{(116)} F ( ρ t ) ≤ ( 1 − t ) F ( μ ) + t F ( π ) , ∀ t ∈ [ 0 , 1 ] , (116) where ( ρ t ) t ∈ [ 0 , 1 ] (\rho_t)_{t\in[0,1]} ( ρ t ) t ∈ [ 0 , 1 ] μ \mu μ π \pi π F \mathcal{F} F ρ t \rho_t ρ t ν t = ( 1 − t ) μ + t π \nu_t=(1-t)\mu+t\pi ν t = ( 1 − t ) μ + t π
Auxiliary Results
In this appendix, we collect all technical results required to prove the main results of the paper.
Lemma B.1. For μ ≪ π \mu\ll\pi μ ≪ π χ 2 \chi^2 χ 2
χ 2 ( μ ∥ π ) = sup h ∈ L 2 ( π ) ∫ h d μ − ∫ ( h + 1 4 h 2 ) d π , \chi^2(\mu\mathbin\Vert\pi)=\sup_{h\in L^2(\pi)}\int h\mathrm{d}\mu-\int\left(h+\frac{1}{4}h^2\right)\mathrm{d}\pi, χ 2 ( μ ∥ π ) = h ∈ L 2 ( π ) sup ∫ h d μ − ∫ ( h + 4 1 h 2 ) d π , where it is sufficient to restrict the variational set to L 2 ( π ) L^2(\pi) L 2 ( π ) f f f
Proof. For μ ≪ π \mu\ll\pi μ ≪ π
χ 2 ( μ ∥ π ) = sup h { ∫ h d μ − ∫ ( h 2 4 + h ) d π } = sup h { ∫ d μ d π h d π − ∫ ( h 2 4 + h ) d π } = sup h { ∫ ( d μ d π − h 2 4 − h ) d π } = − inf h { ∫ [ h 2 4 − h ( d μ d π − 1 ) ] d π } = − inf h { ∫ [ h 2 − ( d μ d π − 1 ) ] 2 d π } + ∫ ( d μ d π − 1 ) 2 d π . \begin{aligned}
\chi^2(\mu\mathbin\Vert\pi)
&=\sup_h\left\{\int h\mathrm{d}\mu-\int\left(\frac{h^2}{4}+h\right)\mathrm{d}\pi\right\}\\
&=\sup_h\left\{\int\frac{\mathrm{d}\mu}{\mathrm{d}\pi}h\mathrm{d}\pi-\int\left(\frac{h^2}{4}+h\right)\mathrm{d}\pi\right\}
=\sup_h\left\{\int\left(\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-\frac{h^2}{4}-h\right)\mathrm{d}\pi\right\}\\
&=-\inf_h\left\{\int\left[\frac{h^2}{4}-h\left(\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right)\right]\mathrm{d}\pi\right\}\\
&=-\inf_h\left\{\int\left[\frac{h}{2}-\left(\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right)\right]^2\mathrm{d}\pi\right\}
+\int\left(\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right)^2\mathrm{d}\pi.
\end{aligned} χ 2 ( μ ∥ π ) = h sup { ∫ h d μ − ∫ ( 4 h 2 + h ) d π } = h sup { ∫ d π d μ h d π − ∫ ( 4 h 2 + h ) d π } = h sup { ∫ ( d π d μ − 4 h 2 − h ) d π } = − h inf { ∫ [ 4 h 2 − h ( d π d μ − 1 ) ] d π } = − h inf { ∫ [ 2 h − ( d π d μ − 1 ) ] 2 d π } + ∫ ( d π d μ − 1 ) 2 d π . Clearly, the above equation is minimized at h ∗ = 2 ( d μ d π − 1 ) h^* = 2\left(\frac{d\mu}{d\pi} - 1\right) h ∗ = 2 ( d π d μ − 1 ) χ 2 ( μ ∥ π ) = ∫ ( d μ d π − 1 ) 2 d π \chi^2(\mu\mathbin\Vert\pi) = \int\left(\frac{d\mu}{d\pi} - 1\right)^2d\pi χ 2 ( μ ∥ π ) = ∫ ( d π d μ − 1 ) 2 d π d μ d π − 1 ∈ L 2 ( π ) \frac{d\mu}{d\pi} - 1 \in L^2(\pi) d π d μ − 1 ∈ L 2 ( π ) L 2 ( π ) L^2(\pi) L 2 ( π )
Lemma B.2. Under Assumptions 4.1 and 4.2, the mappings x ↦ k ( x , ⋅ ) x \mapsto k(x,\cdot) x ↦ k ( x , ⋅ ) x ↦ ∇ 1 k ( x , ⋅ ) x \mapsto\nabla_1 k(x,\cdot) x ↦ ∇ 1 k ( x , ⋅ )
∥ k ( x , ⋅ ) − k ( y , ⋅ ) ∥ H ≤ K 1 d ∥ x − y ∥ , \lVert k(x,\cdot)-k(y,\cdot)\rVert_{\mathcal{H}} \le\sqrt{K_1 d}\lVert x-y\rVert, ∥ k ( x , ⋅ ) − k ( y , ⋅ ) ∥ H ≤ K 1 d ∥ x − y ∥ , ∥ ∇ 1 k ( x , ⋅ ) − ∇ 1 k ( y , ⋅ ) ∥ H d ≤ K 2 d ∥ x − y ∥ . \lVert\nabla_1 k(x,\cdot)-\nabla_1 k(y,\cdot)\rVert_{\mathcal{H}^d} \le\sqrt{K_2 d}\lVert x-y\rVert. ∥ ∇ 1 k ( x , ⋅ ) − ∇ 1 k ( y , ⋅ ) ∥ H d ≤ K 2 d ∥ x − y ∥ . Proof. This is Lemma 7 from Glaser et al. (2021).
Lemma B.3. Under Assumptions 4.1 and 4.2, the regularized kernel k ~ \widetilde{k} k
∣ k ~ ( x , y ) ∣ ≤ K λ \lvert\widetilde{k}(x,y)\rvert\le\frac{K}{\lambda} ∣ k ( x , y )∣ ≤ λ K
∂ i k ~ ( x , y ) = ⟨ ( Σ π + λ Id ) − 1 ∂ i k ( x , ⋅ ) , k ( y , ⋅ ) ⟩ H ′ \partial_i\widetilde{k}(x,y) = \left\langle(\Sigma_\pi+\lambda\operatorname{Id})^{-1}\partial_i k(x,\cdot),k(y,\cdot)\right\rangle_{\mathcal{H}'} ∂ i k ( x , y ) = ⟨ ( Σ π + λ Id ) − 1 ∂ i k ( x , ⋅ ) , k ( y , ⋅ ) ⟩ H ′
∥ ∇ 1 k ~ ( x , y ) ∥ 2 = ∑ i = 1 d ∂ i k ~ ( x , y ) 2 ≤ K K 1 d λ 2 \lVert\nabla_1\widetilde{k}(x,y)\rVert^2 = \sum_{i=1}^{d}\partial_i\widetilde{k}(x,y)^2 \le\frac{KK_1d}{\lambda^2} ∥ ∇ 1 k ( x , y ) ∥ 2 = ∑ i = 1 d ∂ i k ( x , y ) 2 ≤ λ 2 K K 1 d
∂ i ∂ i + d k ~ ( x , y ) = ⟨ ( Σ π + λ Id ) − 1 ∂ i k ( x , ⋅ ) , ∂ i k ( y , ⋅ ) ⟩ H ′ \partial_i\partial_{i+d}\widetilde{k}(x,y) = \left\langle(\Sigma_\pi+\lambda\operatorname{Id})^{-1}\partial_i k(x,\cdot),\partial_i k(y,\cdot)\right\rangle_{\mathcal{H}'} ∂ i ∂ i + d k ( x , y ) = ⟨ ( Σ π + λ Id ) − 1 ∂ i k ( x , ⋅ ) , ∂ i k ( y , ⋅ ) ⟩ H ′
∥ ∇ 1 ∇ 2 k ~ ( x , y ) ∥ F 2 = ∑ i = 1 d ∂ i ∂ i + d k ~ ( x , y ) 2 ≤ K 1 d λ 2 \lVert\nabla_1\nabla_2\widetilde{k}(x,y)\rVert_F^2 = \sum_{i=1}^{d}\partial_i\partial_{i+d}\widetilde{k}(x,y)^2 \le\frac{K_1d}{\lambda^2} ∥ ∇ 1 ∇ 2 k ( x , y ) ∥ F 2 = ∑ i = 1 d ∂ i ∂ i + d k ( x , y ) 2 ≤ λ 2 K 1 d
∂ i ∂ j k ~ ( x , y ) = ⟨ ( Σ π + λ Id ) − 1 ∂ i ∂ j k ( x , ⋅ ) , k ( y , ⋅ ) ⟩ H ′ \partial_i\partial_j\widetilde{k}(x,y) = \left\langle(\Sigma_\pi+\lambda\operatorname{Id})^{-1}\partial_i\partial_j k(x,\cdot),k(y,\cdot)\right\rangle_{\mathcal{H}'} ∂ i ∂ j k ( x , y ) = ⟨ ( Σ π + λ Id ) − 1 ∂ i ∂ j k ( x , ⋅ ) , k ( y , ⋅ ) ⟩ H ′
∥ H 1 k ~ ( x , y ) ∥ F 2 = ∑ i , j = 1 d ∂ i ∂ j k ~ ( x , y ) 2 ≤ K K 2 d λ 2 \lVert\mathbf{H}_1\widetilde{k}(x,y)\rVert_F^2 = \sum_{i,j=1}^{d}\partial_i\partial_j\widetilde{k}(x,y)^2 \le\frac{KK_2d}{\lambda^2} ∥ H 1 k ( x , y ) ∥ F 2 = ∑ i , j = 1 d ∂ i ∂ j k ( x , y ) 2 ≤ λ 2 K K 2 d
∥ ∇ 1 k ~ ( x , x ′ ) − ∇ 1 k ~ ( y , y ′ ) ∥ ≤ K K 2 d λ ( ∥ x − y ∥ + ∥ x ′ − y ′ ∥ ) \lVert\nabla_1\widetilde{k}(x,x')-\nabla_1\widetilde{k}(y,y')\rVert\le\frac{\sqrt{KK_2d}}{\lambda}\left(\lVert x-y\rVert+\lVert x'-y'\rVert\right) ∥ ∇ 1 k ( x , x ′ ) − ∇ 1 k ( y , y ′ )∥ ≤ λ K K 2 d ( ∥ x − y ∥ + ∥ x ′ − y ′ ∥ )
Proof. Notice that
k ~ ( x , y ) = ⟨ ( Σ π + λ Id ) − 1 k ( x , ⋅ ) , k ( y , ⋅ ) ⟩ H \widetilde{k}(x,y)=\left\langle(\Sigma_\pi+\lambda\operatorname{Id})^{-1}k(x,\cdot),k(y,\cdot)\right\rangle_{\mathcal{H}} k ( x , y ) = ⟨ ( Σ π + λ Id ) − 1 k ( x , ⋅ ) , k ( y , ⋅ ) ⟩ H ≤ 1 λ ∥ k ( x , ⋅ ) ∥ H ∥ k ( y , ⋅ ) ∥ H ≤ K λ , \le\frac{1}{\lambda}\lVert k(x,\cdot)\rVert_{\mathcal{H}}\lVert k(y,\cdot)\rVert_{\mathcal{H}}\le\frac{K}{\lambda}, ≤ λ 1 ∥ k ( x , ⋅ ) ∥ H ∥ k ( y , ⋅ ) ∥ H ≤ λ K , so the first bullet point is proved. Before we prove the second bullet point, we first prove the differentiability of x ↦ k ~ ( x , y ) x\mapsto\widetilde{k}(x,y) x ↦ k ( x , y ) i ∈ { 1 , … , d } i\in\{1,\ldots,d\} i ∈ { 1 , … , d }
h ∈ R h \in\mathbb{R} h ∈ R Δ i ∈ R d \Delta_i \in\mathbb{R}^d Δ i ∈ R d i i i h h h y ∈ R d y \in\mathbb{R}^d y ∈ R d
lim h → 0 k ~ ( x + Δ i , y ) − k ~ ( x , y ) h = lim h → 0 1 h ⟨ ( Σ π + λ I d ) − 1 ( k ( x + Δ i , ⋅ ) − k ( x , ⋅ ) ) , k ( y , ⋅ ) ⟩ H ≤ lim h → 0 K λ ∥ k ( x , ⋅ ) − k ( x + Δ i , ⋅ ) ∥ H h ≤ lim h → 0 K λ K 1 d ∥ Δ i ∥ h = K λ K 1 d . \lim_{h \to0} \frac{\widetilde{k}(x+\Delta_i,y)-\widetilde{k}(x,y)}{h}
= \lim_{h \to0} \frac{1}{h} \left\langle(\Sigma_\pi+\lambda\mathrm{Id})^{-1}\bigl(k(x+\Delta_i,\cdot)-k(x,\cdot)\bigr),k(y,\cdot)\right\rangle_{\mathcal{H}}
\leq\lim_{h \to0} \frac{\sqrt{K}}{\lambda}\frac{\lVert k(x,\cdot)-k(x+\Delta_i,\cdot)\rVert_{\mathcal{H}}}{h}
\leq\lim_{h \to0} \frac{\sqrt{K}}{\lambda}\sqrt{K_{1d}}\frac{\lVert\Delta_i\rVert}{h}
= \frac{\sqrt{K}}{\lambda}\sqrt{K_{1d}}. h → 0 lim h k ( x + Δ i , y ) − k ( x , y ) = h → 0 lim h 1 ⟨ ( Σ π + λ Id ) − 1 ( k ( x + Δ i , ⋅ ) − k ( x , ⋅ ) ) , k ( y , ⋅ ) ⟩ H ≤ h → 0 lim λ K h ∥ k ( x , ⋅ ) − k ( x + Δ i , ⋅ ) ∥ H ≤ h → 0 lim λ K K 1 d h ∥ Δ i ∥ = λ K K 1 d . So x ↦ k ~ ( x , y ) x \mapsto\widetilde{k}(x,y) x ↦ k ( x , y ) y ∈ R d y \in\mathbb{R}^d y ∈ R d k k k f ∈ H f \in\mathcal{H} f ∈ H ∂ x i f ( x ) = ⟨ ∂ i k ( x , ⋅ ) , f ⟩ H \partial_{x_i}f(x)=\left\langle\partial_i k(x,\cdot),f\right\rangle_{\mathcal{H}} ∂ x i f ( x ) = ⟨ ∂ i k ( x , ⋅ ) , f ⟩ H
∂ i k ~ ( x , y ) = ∂ x i ⟨ k ( x , ⋅ ) , ( Σ π + λ I d ) − 1 k ( y , ⋅ ) ⟩ H = ⟨ ∂ i k ( x , ⋅ ) , ( Σ π + λ I d ) − 1 k ( y , ⋅ ) ⟩ H . \begin{aligned}
\partial_i\widetilde{k}(x,y)=\partial_{x_i}\left\langle k(x,\cdot),(\Sigma_\pi+\lambda\mathrm{Id})^{-1}k(y,\cdot)\right\rangle_{\mathcal{H}} \\
&=\left\langle\partial_i k(x,\cdot),(\Sigma_\pi+\lambda\mathrm{Id})^{-1}k(y,\cdot)\right\rangle_{\mathcal{H}}.
\end{aligned} ∂ i k ( x , y ) = ∂ x i ⟨ k ( x , ⋅ ) , ( Σ π + λ Id ) − 1 k ( y , ⋅ ) ⟩ H = ⟨ ∂ i k ( x , ⋅ ) , ( Σ π + λ Id ) − 1 k ( y , ⋅ ) ⟩ H . So the second bullet point is proved. Next, notice that
∥ ∇ 1 k ~ ( x , y ) ∥ 2 = ∑ i = 1 d ⟨ ∂ i k ( x , ⋅ ) , ( Σ π + λ I d ) − 1 k ( y , ⋅ ) ⟩ H 2 ≤ 1 λ 2 ∑ i = 1 d ∥ ∂ i k ( x , ⋅ ) ∥ H 2 ∥ k ( y , ⋅ ) ∥ H 2 ≤ 1 λ 2 ∥ ∇ 1 k ( x , ⋅ ) ∥ H d 2 ∥ k ( y , ⋅ ) ∥ H 2 ≤ K K 1 d λ 2 . \begin{aligned}
\left\lVert\nabla_1\widetilde{k}(x,y)\right\rVert^2
=\sum_{i=1}^{d}\left\langle\partial_i k(x,\cdot),(\Sigma_\pi+\lambda\mathrm{Id})^{-1}k(y,\cdot)\right\rangle_{\mathcal{H}}^2 \\
&\leq\frac{1}{\lambda^2}\sum_{i=1}^{d}\left\lVert\partial_i k(x,\cdot)\right\rVert_{\mathcal{H}}^2\left\lVert k(y,\cdot)\right\rVert_{\mathcal{H}}^2 \\
&\leq\frac{1}{\lambda^2}\left\lVert\nabla_1 k(x,\cdot)\right\rVert_{\mathcal{H}^d}^2\left\lVert k(y,\cdot)\right\rVert_{\mathcal{H}}^2
\leq\frac{K K_{1d}}{\lambda^2}.
\end{aligned} ∇ 1 k ( x , y ) 2 = i = 1 ∑ d ⟨ ∂ i k ( x , ⋅ ) , ( Σ π + λ Id ) − 1 k ( y , ⋅ ) ⟩ H 2 ≤ λ 2 1 i = 1 ∑ d ∥ ∂ i k ( x , ⋅ ) ∥ H 2 ∥ k ( y , ⋅ ) ∥ H 2 ≤ λ 2 1 ∥ ∇ 1 k ( x , ⋅ ) ∥ H d 2 ∥ k ( y , ⋅ ) ∥ H 2 ≤ λ 2 K K 1 d . So the third bullet point is proved. Similar arguments above lead to bullet points 4 to 7.
Finally, to prove bullet point 8, notice that
∥ ∇ 1 k ~ ( x , x ′ ) − ∇ 1 k ~ ( y , x ′ ) ∥ 2 = ∑ i = 1 d ⟨ ∂ i k ( x , ⋅ ) − ∂ i k ( y , ⋅ ) , ( Σ π + λ I d ) − 1 k ( x ′ , ⋅ ) ⟩ H 2 ≤ 1 λ 2 ∑ i = 1 d ∥ ∂ i k ( x , ⋅ ) − ∂ i k ( y , ⋅ ) ∥ H 2 ∥ k ( x ′ , ⋅ ) ∥ H 2 ≤ 1 λ 2 ∥ ∇ 1 k ( x , ⋅ ) − ∇ 1 k ( y , ⋅ ) ∥ H d 2 ∥ k ( x ′ , ⋅ ) ∥ H 2 ≤ K K 2 d λ 2 ∥ x − y ∥ 2 , \begin{aligned}
\left\lVert\nabla_1\widetilde{k}(x,x')-\nabla_1\widetilde{k}(y,x')\right\rVert^2
=\sum_{i=1}^{d}\left\langle\partial_i k(x,\cdot)-\partial_i k(y,\cdot),(\Sigma_\pi+\lambda\mathrm{Id})^{-1}k(x',\cdot)\right\rangle_{\mathcal{H}}^2 \\
&\leq\frac{1}{\lambda^2}\sum_{i=1}^{d}\left\lVert\partial_i k(x,\cdot)-\partial_i k(y,\cdot)\right\rVert_{\mathcal{H}}^2\left\lVert k(x',\cdot)\right\rVert_{\mathcal{H}}^2 \\
&\leq\frac{1}{\lambda^2}\left\lVert\nabla_1 k(x,\cdot)-\nabla_1 k(y,\cdot)\right\rVert_{\mathcal{H}^d}^2\left\lVert k(x',\cdot)\right\rVert_{\mathcal{H}}^2 \\
&\le\frac{K K_{2d}}{\lambda^2}\lVert x-y\rVert^2,
\end{aligned} ∇ 1 k ( x , x ′ ) − ∇ 1 k ( y , x ′ ) 2 = i = 1 ∑ d ⟨ ∂ i k ( x , ⋅ ) − ∂ i k ( y , ⋅ ) , ( Σ π + λ Id ) − 1 k ( x ′ , ⋅ ) ⟩ H 2 ≤ λ 2 1 i = 1 ∑ d ∥ ∂ i k ( x , ⋅ ) − ∂ i k ( y , ⋅ ) ∥ H 2 ∥ k ( x ′ , ⋅ ) ∥ H 2 ≤ λ 2 1 ∥ ∇ 1 k ( x , ⋅ ) − ∇ 1 k ( y , ⋅ ) ∥ H d 2 ∥ k ( x ′ , ⋅ ) ∥ H 2 ≤ λ 2 K K 2 d ∥ x − y ∥ 2 , where the last inequality uses Theorem B.2. Therefore,
∥ ∇ 1 k ~ ( x , x ′ ) − ∇ 1 k ~ ( y , y ′ ) ∥ ≤ ∥ ∇ 1 k ~ ( x , x ′ ) − ∇ 1 k ~ ( y , x ′ ) ∥ + ∥ ∇ 1 k ~ ( y , x ′ ) − ∇ 1 k ~ ( y , y ′ ) ∥ ≤ K K 2 d λ ( ∥ x − y ∥ + ∥ x ′ − y ′ ∥ ) . \begin{aligned}
\lVert\nabla_1\widetilde{k}(x,x')-\nabla_1\widetilde{k}(y,y')\rVert
&\le\lVert\nabla_1\widetilde{k}(x,x')-\nabla_1\widetilde{k}(y,x')\rVert\\
&\quad+ \lVert\nabla_1\widetilde{k}(y,x')-\nabla_1\widetilde{k}(y,y')\rVert\\
&\le\frac{\sqrt{K K_{2d}}}{\lambda}\left(\lVert x-y\rVert+\lVert x'-y'\rVert\right).
\end{aligned} ∥ ∇ 1 k ( x , x ′ ) − ∇ 1 k ( y , y ′ )∥ ≤ ∥ ∇ 1 k ( x , x ′ ) − ∇ 1 k ( y , x ′ )∥ + ∥ ∇ 1 k ( y , x ′ ) − ∇ 1 k ( y , y ′ )∥ ≤ λ K K 2 d ( ∥ x − y ∥ + ∥ x ′ − y ′ ∥ ) . So the bullet point 8 is proved.
Lemma B.4. For any two distributions μ 0 , μ 1 ∈ P 2 ( R d ) \mu_0,\mu_1 \in\mathcal{P}_2(\mathbb{R}^d) μ 0 , μ 1 ∈ P 2 ( R d ) with associated DrMMD witness functions h μ 0 , π ∗ , h μ 1 , π ∗ h^*_{\mu_0,\pi},h^*_{\mu_1,\pi} h μ 0 , π ∗ , h μ 1 , π ∗ defined in Theorem 4.6, we have
∥ h μ 1 , π ∗ − h μ 0 , π ∗ ∥ H ≤ 2 K K 1 d λ W 2 ( μ 0 , μ 1 ) , and ∥ h μ 0 , π ∗ ∥ H ≤ 2 K λ . \lVert h^*_{\mu_1,\pi}-h^*_{\mu_0,\pi}\rVert_{\mathcal{H}}\le\frac{2\sqrt{K K_{1d}}}{\lambda}W_2(\mu_0,\mu_1),\qquad\text{and}\qquad\lVert h^*_{\mu_0,\pi}\rVert_{\mathcal{H}}\le\frac{2\sqrt{K}}{\lambda}. ∥ h μ 1 , π ∗ − h μ 0 , π ∗ ∥ H ≤ λ 2 K K 1 d W 2 ( μ 0 , μ 1 ) , and ∥ h μ 0 , π ∗ ∥ H ≤ λ 2 K . Both the witness function h μ , π ∗ h^*_{\mu,\pi} h μ , π ∗ and its gradient ∇ h μ , π ∗ \nabla h^*_{\mu,\pi} ∇ h μ , π ∗ are Lipschitz continuous, i.e.,
∣ h μ 1 , π ∗ ( x ) − h μ 0 , π ∗ ( y ) ∣ ≤ L ( W 2 ( μ 0 , μ 1 ) + ∥ x − y ∥ ) ; \left|h^*_{\mu_1,\pi}(x)-h^*_{\mu_0,\pi}(y)\right|\le L\left(W_2(\mu_0,\mu_1)+\lVert x-y\rVert\right); h μ 1 , π ∗ ( x ) − h μ 0 , π ∗ ( y ) ≤ L ( W 2 ( μ 0 , μ 1 ) + ∥ x − y ∥ ) ; ∥ ∇ h μ 1 , π ∗ ( x ) − ∇ h μ 0 , π ∗ ( y ) ∥ ≤ L ( W 2 ( μ 0 , μ 1 ) + ∥ x − y ∥ ) , \lVert\nabla h^*_{\mu_1,\pi}(x)-\nabla h^*_{\mu_0,\pi}(y)\rVert\le L\left(W_2(\mu_0,\mu_1)+\lVert x-y\rVert\right), ∥ ∇ h μ 1 , π ∗ ( x ) − ∇ h μ 0 , π ∗ ( y )∥ ≤ L ( W 2 ( μ 0 , μ 1 ) + ∥ x − y ∥ ) , where the constant L = 1 λ max { 2 K K 1 d , 2 K K 2 d , 2 K 1 d } L=\frac{1}{\lambda}\max\left\{2\sqrt{K K_{1d}},2\sqrt{K K_{2d}},2K_{1d}\right\} L = λ 1 max { 2 K K 1 d , 2 K K 2 d , 2 K 1 d }
Proof. Let γ ∈ Γ ( μ 0 , μ 1 ) \gamma\in\Gamma(\mu_0,\mu_1) γ ∈ Γ ( μ 0 , μ 1 ) μ 0 \mu_0 μ 0 μ 1 \mu_1 μ 1
∥ h μ 1 , π ∗ − h μ 0 , π ∗ ∥ H 2 = 4 ∥ ( Σ π + λ I d ) − 1 ( ∫ k ( x , ⋅ ) d ( μ 1 − μ 0 ) ) ∥ H 2 ≤ 4 λ 2 ∥ ∫ ( k ( x , ⋅ ) − k ( y , ⋅ ) ) d γ ( x , y ) ∥ H 2 ≤ 4 K 1 d λ 2 ∫ ∥ x − y ∥ 2 d γ ( x , y ) = 4 K 1 d λ 2 W 2 2 ( μ 0 , μ 1 ) , \begin{aligned}
\lVert h^*_{\mu_1,\pi}-h^*_{\mu_0,\pi}\rVert_{\mathcal{H}}^2
&=4\left\lVert\left(\Sigma_\pi+\lambda I_d\right)^{-1}\left(\int k(x,\cdot)\,\mathrm{d}(\mu_1-\mu_0)\right)\right\rVert_{\mathcal{H}}^2\\
&\le\frac{4}{\lambda^2}\left\lVert\int\left(k(x,\cdot)-k(y,\cdot)\right)\,\mathrm{d}\gamma(x,y)\right\rVert_{\mathcal{H}}^2\\
&\le\frac{4K_{1d}}{\lambda^2}\int\lVert x-y\rVert^2\,\mathrm{d}\gamma(x,y)\\
&=\frac{4K_{1d}}{\lambda^2}W_2^2(\mu_0,\mu_1),
\end{aligned} ∥ h μ 1 , π ∗ − h μ 0 , π ∗ ∥ H 2 = 4 ( Σ π + λ I d ) − 1 ( ∫ k ( x , ⋅ ) d ( μ 1 − μ 0 ) ) H 2 ≤ λ 2 4 ∫ ( k ( x , ⋅ ) − k ( y , ⋅ ) ) d γ ( x , y ) H 2 ≤ λ 2 4 K 1 d ∫ ∥ x − y ∥ 2 d γ ( x , y ) = λ 2 4 K 1 d W 2 2 ( μ 0 , μ 1 ) , where the first inequality holds because Σ π \Sigma_\pi Σ π
ator, and the second inequality uses Theorem B.2. Also note that
∥ h μ 0 , π ⋆ ∥ H = ∥ 2 ( Σ π + λ I d ) − 1 ( ∫ k ( x , ⋅ ) d μ 0 − ∫ k ( x , ⋅ ) d π ) ∥ H ≤ 2 K λ . \left\|h_{\mu_0,\pi}^{\star}\right\|_{\mathcal{H}}=\left\|2\left(\Sigma_{\pi}+\lambda\mathrm{Id}\right)^{-1}\left(\int k(x,\cdot)\mathrm{d}\mu_0-\int k(x,\cdot)\mathrm{d}\pi\right)\right\|_{\mathcal{H}}\leq\frac{2\sqrt{K}}{\lambda}. h μ 0 , π ⋆ H = 2 ( Σ π + λ Id ) − 1 ( ∫ k ( x , ⋅ ) d μ 0 − ∫ k ( x , ⋅ ) d π ) H ≤ λ 2 K . So the first part has been proved. Furthermore, note that
∣ h μ 1 , π ⋆ ( x ) − h μ 0 , π ⋆ ( y ) ∣ ≤ ∣ h μ 1 , π ⋆ ( x ) − h μ 0 , π ⋆ ( x ) ∣ + ∣ h μ 0 , π ⋆ ( x ) − h μ 0 , π ⋆ ( y ) ∣ ≤ ∥ k ( x , ⋅ ) ∥ H ∥ h μ 1 , π ⋆ − h μ 0 , π ⋆ ∥ H + ∥ k ( x , ⋅ ) − k ( y , ⋅ ) ∥ H ∥ h μ 0 , π ⋆ ∥ H ≤ 2 K K 1 d λ W 2 ( μ 0 , μ 1 ) + 2 K K 1 d λ ∥ x − y ∥ ≤ 2 K K 1 d λ ( W 2 ( μ 0 , μ 1 ) + ∥ x − y ∥ ) ≤ L ( W 2 ( μ 0 , μ 1 ) + ∥ x − y ∥ ) . \begin{aligned}
\left|h_{\mu_1,\pi}^{\star}(x)-h_{\mu_0,\pi}^{\star}(y)\right|
&\leq\left|h_{\mu_1,\pi}^{\star}(x)-h_{\mu_0,\pi}^{\star}(x)\right|+\left|h_{\mu_0,\pi}^{\star}(x)-h_{\mu_0,\pi}^{\star}(y)\right| \\
&\leq\left\|k(x,\cdot)\right\|_{\mathcal{H}}\left\|h_{\mu_1,\pi}^{\star}-h_{\mu_0,\pi}^{\star}\right\|_{\mathcal{H}}+\left\|k(x,\cdot)-k(y,\cdot)\right\|_{\mathcal{H}}\left\|h_{\mu_0,\pi}^{\star}\right\|_{\mathcal{H}} \\
&\leq\frac{2\sqrt{K K_{1d}}}{\lambda}W_2(\mu_0,\mu_1)+\frac{2\sqrt{K K_{1d}}}{\lambda}\left\|x-y\right\| \\
&\leq\frac{2\sqrt{K K_{1d}}}{\lambda}\left(W_2(\mu_0,\mu_1)+\left\|x-y\right\|\right)\leq L\left(W_2(\mu_0,\mu_1)+\left\|x-y\right\|\right).
\end{aligned} h μ 1 , π ⋆ ( x ) − h μ 0 , π ⋆ ( y ) ≤ h μ 1 , π ⋆ ( x ) − h μ 0 , π ⋆ ( x ) + h μ 0 , π ⋆ ( x ) − h μ 0 , π ⋆ ( y ) ≤ ∥ k ( x , ⋅ ) ∥ H h μ 1 , π ⋆ − h μ 0 , π ⋆ H + ∥ k ( x , ⋅ ) − k ( y , ⋅ ) ∥ H h μ 0 , π ⋆ H ≤ λ 2 K K 1 d W 2 ( μ 0 , μ 1 ) + λ 2 K K 1 d ∥ x − y ∥ ≤ λ 2 K K 1 d ( W 2 ( μ 0 , μ 1 ) + ∥ x − y ∥ ) ≤ L ( W 2 ( μ 0 , μ 1 ) + ∥ x − y ∥ ) . and
∥ ∇ h μ 1 , π ⋆ ( x ) − ∇ h μ 0 , π ⋆ ( y ) ∥ ≤ ∥ ∇ h μ 1 , π ⋆ ( x ) − ∇ h μ 0 , π ⋆ ( x ) ∥ + ∥ ∇ h μ 0 , π ⋆ ( x ) − ∇ h μ 0 , π ⋆ ( y ) ∥ ≤ ∥ ∇ 1 k ( x , ⋅ ) ∥ H d ∥ h μ 1 , π ⋆ − h μ 0 , π ⋆ ∥ H + ∥ ∇ 1 k ( x , ⋅ ) − ∇ 1 k ( y , ⋅ ) ∥ H d ∥ h μ 0 , π ⋆ ∥ H ≤ 2 K 1 d λ W 2 ( μ 0 , μ 1 ) + 2 K K 2 d λ ∥ x − y ∥ ≤ L ( W 2 ( μ 0 , μ 1 ) + ∥ x − y ∥ ) \begin{aligned}
\left\|\nabla h_{\mu_1,\pi}^{\star}(x)-\nabla h_{\mu_0,\pi}^{\star}(y)\right\|
&\leq\left\|\nabla h_{\mu_1,\pi}^{\star}(x)-\nabla h_{\mu_0,\pi}^{\star}(x)\right\|+\left\|\nabla h_{\mu_0,\pi}^{\star}(x)-\nabla h_{\mu_0,\pi}^{\star}(y)\right\| \\
&\leq\left\|\nabla_1 k(x,\cdot)\right\|_{\mathcal{H}^d}\left\|h_{\mu_1,\pi}^{\star}-h_{\mu_0,\pi}^{\star}\right\|_{\mathcal{H}}+\left\|\nabla_1 k(x,\cdot)-\nabla_1 k(y,\cdot)\right\|_{\mathcal{H}^d}\left\|h_{\mu_0,\pi}^{\star}\right\|_{\mathcal{H}} \\
&\leq\frac{2K_{1d}}{\lambda}W_2(\mu_0,\mu_1)+\frac{2\sqrt{K K_{2d}}}{\lambda}\left\|x-y\right\|\leq L\left(W_2(\mu_0,\mu_1)+\left\|x-y\right\|\right)
\end{aligned} ∇ h μ 1 , π ⋆ ( x ) − ∇ h μ 0 , π ⋆ ( y ) ≤ ∇ h μ 1 , π ⋆ ( x ) − ∇ h μ 0 , π ⋆ ( x ) + ∇ h μ 0 , π ⋆ ( x ) − ∇ h μ 0 , π ⋆ ( y ) ≤ ∥ ∇ 1 k ( x , ⋅ ) ∥ H d h μ 1 , π ⋆ − h μ 0 , π ⋆ H + ∥ ∇ 1 k ( x , ⋅ ) − ∇ 1 k ( y , ⋅ ) ∥ H d h μ 0 , π ⋆ H ≤ λ 2 K 1 d W 2 ( μ 0 , μ 1 ) + λ 2 K K 2 d ∥ x − y ∥ ≤ L ( W 2 ( μ 0 , μ 1 ) + ∥ x − y ∥ ) and the result follows. □
Lemma B.5. Given two probability measures μ ≪ π \mu\ll\pi μ ≪ π d μ d π − 1 ∈ Ran ( T π r ) \frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\in\operatorname{Ran}(\mathcal{T}_{\pi}^{r}) d π d μ − 1 ∈ Ran ( T π r ) r > 0 r>0 r > 0 q ∈ L 2 ( π ) q\in L^2(\pi) q ∈ L 2 ( π ) d μ d π − 1 = T π r q \frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1=\mathcal{T}_{\pi}^{r}q d π d μ − 1 = T π r q
∥ h − 2 ( d μ d π − 1 ) ∥ L 2 ( π ) ≤ 2 λ r ∥ q ∥ L 2 ( π ) , \left\|h-2\left(\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\right)\right\|_{L^2(\pi)}\leq2\lambda^{r}\left\|q\right\|_{L^2(\pi)}, h − 2 ( d π d μ − 1 ) L 2 ( π ) ≤ 2 λ r ∥ q ∥ L 2 ( π ) , where h = 2 ( Σ π + λ I d ) − 1 ( m π − m μ ) h=2(\Sigma_{\pi}+\lambda\mathrm{Id})^{-1}(m_{\pi}-m_{\mu}) h = 2 ( Σ π + λ Id ) − 1 ( m π − m μ )
Proof. Given the assumption that d μ d π − 1 ∈ Ran ( T π r ) \frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1\in\operatorname{Ran}(\mathcal{T}_{\pi}^{r}) d π d μ − 1 ∈ Ran ( T π r ) r > 0 r>0 r > 0 q ∈ L 2 ( π ) q\in L^2(\pi) q ∈ L 2 ( π ) d μ d π − 1 = T π r q \frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1=\mathcal{T}_{\pi}^{r}q d π d μ − 1 = T π r q ⟨ d μ d π − 1 , e i ⟩ L 2 ( π ) = ρ i r ⟨ q , e i ⟩ L 2 ( π ) \left\langle\frac{\mathrm{d}\mu}{\mathrm{d}\pi}-1,e_i\right\rangle_{L^2(\pi)}=\rho_i^r\langle q,e_i\rangle_{L^2(\pi)} ⟨ d π d μ − 1 , e i ⟩ L 2 ( π ) = ρ i r ⟨ q , e i ⟩ L 2 ( π ) μ ≪ π \mu\ll\pi μ ≪ π k k k x x x μ \mu μ π \pi π
m μ − m π = ∫ k ( x , ⋅ ) d ( μ − π ) ( x ) = ∑ i ≥ 1 ϱ i ( ∫ e i ( x ) d ( μ − π ) ( x ) ) e i = ϱ i ⟨ d μ d π − 1 , e i ⟩ L 2 ( π ) e i . (117) m_{\mu}-m_{\pi}=\int k(x,\cdot)d(\mu-\pi)(x)=\sum_{i\geq1}\varrho_i\left(\int e_i(x)d(\mu-\pi)(x)\right)e_i
=\varrho_i\left\langle\frac{d\mu}{d\pi}-1,e_i\right\rangle_{L^2(\pi)}e_i. \tag*{(117)} m μ − m π = ∫ k ( x , ⋅ ) d ( μ − π ) ( x ) = i ≥ 1 ∑ ϱ i ( ∫ e i ( x ) d ( μ − π ) ( x ) ) e i = ϱ i ⟨ d π d μ − 1 , e i ⟩ L 2 ( π ) e i . (117) So we have,
∥ h − 2 ( d μ d π − 1 ) ∥ L 2 ( π ) = 2 ∥ ( Σ π + λ Id ) − 1 ( m μ − m π ) − ( d μ d π − 1 ) ∥ L 2 ( π ) = 2 ∥ ∑ i ≥ 1 ϱ i ϱ i + λ ⟨ d μ d π − 1 , e i ⟩ L 2 ( π ) e i − ∑ i ≥ 1 ⟨ d μ d π − 1 , e i ⟩ L 2 ( π ) e i ∥ L 2 ( π ) = 2 ∥ ∑ i ≥ 1 λ ϱ i + λ ⟨ d μ d π − 1 , e i ⟩ L 2 ( π ) e i ∥ L 2 ( π ) = 2 ∥ ∑ i ≥ 1 λ ϱ i r ϱ i + λ ⟨ q , e i ⟩ L 2 ( π ) e i ∥ L 2 ( π ) ≤ 2 λ r ∥ q ∥ L 2 ( π ) , \begin{aligned}
\left\|h-2\left(\frac{d\mu}{d\pi}-1\right)\right\|_{L^2(\pi)}
&=2\left\|(\Sigma_\pi+\lambda\operatorname{Id})^{-1}(m_\mu-m_\pi)-\left(\frac{d\mu}{d\pi}-1\right)\right\|_{L^2(\pi)}\\
&=2\left\|\sum_{i\geq1}\frac{\varrho_i}{\varrho_i+\lambda}\left\langle\frac{d\mu}{d\pi}-1,e_i\right\rangle_{L^2(\pi)}e_i-\sum_{i\geq1}\left\langle\frac{d\mu}{d\pi}-1,e_i\right\rangle_{L^2(\pi)}e_i\right\|_{L^2(\pi)}\\
&=2\left\|\sum_{i\geq1}\frac{\lambda}{\varrho_i+\lambda}\left\langle\frac{d\mu}{d\pi}-1,e_i\right\rangle_{L^2(\pi)}e_i\right\|_{L^2(\pi)}
=2\left\|\sum_{i\geq1}\frac{\lambda\varrho_i^r}{\varrho_i+\lambda}\langle q,e_i\rangle_{L^2(\pi)}e_i\right\|_{L^2(\pi)}\\
&\leq2\lambda^r\|q\|_{L^2(\pi)},
\end{aligned} h − 2 ( d π d μ − 1 ) L 2 ( π ) = 2 ( Σ π + λ Id ) − 1 ( m μ − m π ) − ( d π d μ − 1 ) L 2 ( π ) = 2 i ≥ 1 ∑ ϱ i + λ ϱ i ⟨ d π d μ − 1 , e i ⟩ L 2 ( π ) e i − i ≥ 1 ∑ ⟨ d π d μ − 1 , e i ⟩ L 2 ( π ) e i L 2 ( π ) = 2 i ≥ 1 ∑ ϱ i + λ λ ⟨ d π d μ − 1 , e i ⟩ L 2 ( π ) e i L 2 ( π ) = 2 i ≥ 1 ∑ ϱ i + λ λ ϱ i r ⟨ q , e i ⟩ L 2 ( π ) e i L 2 ( π ) ≤ 2 λ r ∥ q ∥ L 2 ( π ) , where the last inequality is obtained by using
λ ϱ i r ϱ i + λ = ( ϱ i ϱ i + λ ) r ( λ ϱ i + λ ) 1 − r λ r ≤ λ r . \frac{\lambda\varrho_i^r}{\varrho_i+\lambda}
=\left(\frac{\varrho_i}{\varrho_i+\lambda}\right)^r\left(\frac{\lambda}{\varrho_i+\lambda}\right)^{1-r}\lambda^r\leq\lambda^r. ϱ i + λ λ ϱ i r = ( ϱ i + λ ϱ i ) r ( ϱ i + λ λ ) 1 − r λ r ≤ λ r . Hence the proof. □
Lemma B.6. Let ρ ∈ P 2 ( R d ) \rho\in\mathcal{P}_2(\mathbb{R}^d) ρ ∈ P 2 ( R d ) ϕ ∈ C ∞ ( R d ) \phi\in C^\infty(\mathbb{R}^d) ϕ ∈ C ∞ ( R d ) ( ρ s ) 0 ≤ s ≤ 1 (\rho_s)_{0\leq s\leq1} ( ρ s ) 0 ≤ s ≤ 1 ρ \rho ρ ( Id + ∇ ϕ ) # ρ (\operatorname{Id}+\nabla\phi)_\#\rho ( Id + ∇ ϕ ) # ρ ρ s = ( Id + s ∇ ϕ ) # ρ \rho_s=(\operatorname{Id}+s\nabla\phi)_\#\rho ρ s = ( Id + s ∇ ϕ ) # ρ φ s : R d → R d \varphi_s:\mathbb{R}^d\to\mathbb{R}^d φ s : R d → R d x ↦ x + s ∇ ϕ ( x ) x\mapsto x+s\nabla\phi(x) x ↦ x + s ∇ ϕ ( x ) k ~ \widetilde{k} k H ~ \widetilde{\mathcal{H}} H s ↦ DrMMD ( ρ s ∥ π ) s\mapsto\operatorname{DrMMD}(\rho_s\|\pi) s ↦ DrMMD ( ρ s ∥ π )
d d s DrMMD ( ρ s ∥ π ) = 2 ( 1 + λ ) ∫ ∇ ϕ ( x ) T ( ∫ ∇ 1 k ~ ( φ s ( x ) , φ s ( z ) ) d ρ ( z ) − ∫ ∇ 1 k ~ ( φ s ( x ) , z ) d π ( z ) ) d ρ ( x ) . (118) \frac{d}{ds}\operatorname{DrMMD}(\rho_s\|\pi)
=2(1+\lambda)\int\nabla\phi(x)^{\mathsf{T}}\left(\int\nabla_1\widetilde{k}(\varphi_s(x),\varphi_s(z))d\rho(z)-\int\nabla_1\widetilde{k}(\varphi_s(x),z)d\pi(z)\right)d\rho(x). \tag*{(118)} d s d DrMMD ( ρ s ∥ π ) = 2 ( 1 + λ ) ∫ ∇ ϕ ( x ) T ( ∫ ∇ 1 k ( φ s ( x ) , φ s ( z )) d ρ ( z ) − ∫ ∇ 1 k ( φ s ( x ) , z ) d π ( z ) ) d ρ ( x ) . (118) Moreover,
d d s ∣ s = 0 DrMMD ( ρ s ∥ π ) = 2 ( 1 + λ ) ∫ ∇ ϕ ( x ) T ( ∫ ∇ 1 k ~ ( x , z ) d ρ ( z ) − ∫ ∇ 1 k ~ ( x , z ) d π ( z ) ) d ρ ( x ) . (119) \left.\frac{d}{ds}\right|_{s=0}\operatorname{DrMMD}(\rho_s\mathbin\Vert\pi)
=2(1+\lambda)\int\nabla\phi(x)^{\mathsf T}\left(\int\nabla_1\widetilde{k}(x,z)d\rho(z)-\int\nabla_1\widetilde{k}(x,z)d\pi(z)\right)d\rho(x). \tag*{(119)} d s d s = 0 DrMMD ( ρ s ∥ π ) = 2 ( 1 + λ ) ∫ ∇ ϕ ( x ) T ( ∫ ∇ 1 k ( x , z ) d ρ ( z ) − ∫ ∇ 1 k ( x , z ) d π ( z ) ) d ρ ( x ) . (119) Additionally, the mapping s ↦ d d s DrMMD ( ρ s ∥ π ) s\mapsto\frac{d}{ds}\operatorname{DrMMD}(\rho_s\mathbin\Vert\pi) s ↦ d s d DrMMD ( ρ s ∥ π ) DrMMD ( ρ s ∥ π ) \operatorname{DrMMD}(\rho_s\mathbin\Vert\pi) DrMMD ( ρ s ∥ π )
d 2 d s 2 DrMMD ( ρ s ∥ π ) = 2 ( 1 + λ ) ∬ ∇ ϕ ( x ) T ∇ 1 ∇ 2 k ~ ( φ s ( x ) , φ s ( z ) ) ∇ ϕ ( z ) d ρ ( x ) d ρ ( z ) (120) \frac{d^2}{ds^2}\operatorname{DrMMD}(\rho_s\mathbin\Vert\pi)
=2(1+\lambda)\iint\nabla\phi(x)^{\mathsf T}\nabla_1\nabla_2\widetilde{k}(\varphi_s(x),\varphi_s(z))\nabla\phi(z)d\rho(x)d\rho(z) \tag*{(120)} d s 2 d 2 DrMMD ( ρ s ∥ π ) = 2 ( 1 + λ ) ∬ ∇ ϕ ( x ) T ∇ 1 ∇ 2 k ( φ s ( x ) , φ s ( z )) ∇ ϕ ( z ) d ρ ( x ) d ρ ( z ) (120) + 2 ( 1 + λ ) ∫ ∇ ϕ ( x ) T ( ∫ H 1 k ~ ( φ s ( x ) , φ s ( z ) ) d ρ ( z ) − ∫ H 1 k ~ ( φ s ( x ) , z ) d π ( z ) ) ∇ ϕ ( x ) d ρ ( x ) , (121) +2(1+\lambda)\int\nabla\phi(x)^{\mathsf T}\left(\int\mathbf{H}_1\widetilde{k}(\varphi_s(x),\varphi_s(z))d\rho(z)-\int\mathbf{H}_1\widetilde{k}(\varphi_s(x),z)d\pi(z)\right)\nabla\phi(x)d\rho(x), \tag*{(121)} + 2 ( 1 + λ ) ∫ ∇ ϕ ( x ) T ( ∫ H 1 k ( φ s ( x ) , φ s ( z )) d ρ ( z ) − ∫ H 1 k ( φ s ( x ) , z ) d π ( z ) ) ∇ ϕ ( x ) d ρ ( x ) , (121) with
d 2 d s 2 ∣ s = 0 DrMMD ( ρ s ∥ π ) = 2 ( 1 + λ ) ∬ ∇ ϕ ( x ) T ∇ 1 ∇ 2 k ~ ( x , z ) ∇ ϕ ( z ) d ρ ( x ) d ρ ( z ) + 2 ( 1 + λ ) ∫ ∇ ϕ ( x ) T ( ∫ H 1 k ~ ( x , z ) d ρ ( z ) − ∫ H 1 k ~ ( x , z ) d π ( z ) ) ∇ ϕ ( x ) d ρ ( x ) . (122) \begin{aligned}
\left.\frac{d^2}{ds^2}\right|_{s=0}\operatorname{DrMMD}(\rho_s\mathbin\Vert\pi)
=2(1+\lambda)\iint\nabla\phi(x)^{\mathsf T}\nabla_1\nabla_2\widetilde{k}(x,z)\nabla\phi(z)d\rho(x)d\rho(z) \\
&+2(1+\lambda)\int\nabla\phi(x)^{\mathsf T}\left(\int\mathbf{H}_1\widetilde{k}(x,z)d\rho(z)-\int\mathbf{H}_1\widetilde{k}(x,z)d\pi(z)\right)\nabla\phi(x)d\rho(x). \tag*{(122)}
\end{aligned} d s 2 d 2 s = 0 DrMMD ( ρ s ∥ π ) = 2 ( 1 + λ ) ∬ ∇ ϕ ( x ) T ∇ 1 ∇ 2 k ( x , z ) ∇ ϕ ( z ) d ρ ( x ) d ρ ( z ) + 2 ( 1 + λ ) ∫ ∇ ϕ ( x ) T ( ∫ H 1 k ( x , z ) d ρ ( z ) − ∫ H 1 k ( x , z ) d π ( z ) ) ∇ ϕ ( x ) d ρ ( x ) . (122) Proof. Recall that DrMMD \operatorname{DrMMD} DrMMD ( 1 + λ ) (1+\lambda) ( 1 + λ ) MMD 2 \operatorname{MMD}^2 MMD 2 k ~ \widetilde{k} k (16) and the associated RKHS H ~ \widetilde{\mathcal{H}} H k ~ \widetilde{k} k s = 0 s=0 s = 0
Lemma B.7. For μ 0 ≪ π \mu_0\ll\pi μ 0 ≪ π h = 2 ( Σ π + λ Id ) − 1 ( m π − m μ 0 ) h=2(\Sigma_\pi+\lambda\operatorname{Id})^{-1}(m_\pi-m_{\mu_0}) h = 2 ( Σ π + λ Id ) − 1 ( m π − m μ 0 ) φ t : R d → R d \varphi_t:\mathbb{R}^d\to\mathbb{R}^d φ t : R d → R d x ↦ x − t ( 1 + λ ) ∇ h ( x ) x\mapsto x-t(1+\lambda)\nabla h(x) x ↦ x − t ( 1 + λ ) ∇ h ( x ) π ∝ exp ( − V ) \pi\propto\exp(-V) π ∝ exp ( − V ) H V ⪯ β Id \mathbf{H}V\preceq\beta\operatorname{Id} H V ⪯ β Id
step size γ \gamma γ
2 ( 1 + λ ) γ χ 2 ( μ 0 ∥ π ) K 2 d λ ≤ ζ − 1 ζ 2(1+\lambda)\gamma\sqrt{\chi^2(\mu_0\Vert\pi)\frac{K_{2d}}{\lambda}}\leq\frac{\zeta-1}{\zeta} 2 ( 1 + λ ) γ χ 2 ( μ 0 ∥ π ) λ K 2 d ≤ ζ ζ − 1 for some constant 1 < ζ < 2 1<\zeta<2 1 < ζ < 2 t ∈ [ 0 , γ ] t\in[0,\gamma] t ∈ [ 0 , γ ]
∫ ∇ h ( x ) T H V ( φ t ( x ) ) ∇ h ( x ) d μ 0 ( x ) ≤ 4 β χ 2 ( μ 0 ∥ π ) K 1 d λ ; (123) \int\nabla h(x)^{\mathsf T}\mathsf{H}\mathsf{V}(\varphi_t(x))\nabla h(x)\,d\mu_0(x)\leq4\beta\chi^2(\mu_0\Vert\pi)\frac{K_{1d}}{\lambda}; \tag*{(123)} ∫ ∇ h ( x ) T HV ( φ t ( x )) ∇ h ( x ) d μ 0 ( x ) ≤ 4 β χ 2 ( μ 0 ∥ π ) λ K 1 d ; (123) ∫ ∥ H h ( x ) ( I d − t ( 1 + λ ) H h ( x ) ) − 1 ∥ F 2 d μ 0 ( x ) ≤ 4 ζ 2 χ 2 ( μ 0 ∥ π ) K 2 d λ . (124) \int\left\|\mathsf{H}h(x)\left(\mathsf{Id}-t(1+\lambda)\mathsf{H}h(x)\right)^{-1}\right\|_{\mathsf F}^{2}\,d\mu_0(x)\leq4\zeta^2\chi^2(\mu_0\Vert\pi)\frac{K_{2d}}{\lambda}. \tag*{(124)} ∫ H h ( x ) ( Id − t ( 1 + λ ) H h ( x ) ) − 1 F 2 d μ 0 ( x ) ≤ 4 ζ 2 χ 2 ( μ 0 ∥ π ) λ K 2 d . (124) Proof. First, recall (B.1),
∥ ∇ h ( x ) ∥ 2 = 4 ∥ ∇ ( Σ π + λ I d ) − 1 ( m π − m μ 0 ) ( x ) ∥ 2 = 4 ∥ ∇ ( ∑ i ≥ 1 ϱ i ϱ i + λ ⟨ d μ 0 d π − 1 , e i ⟩ L 2 ( π ) e i ( x ) ) ∥ 2 . (125) \begin{aligned}
\|\nabla h(x)\|^2=4\left\|\nabla(\Sigma_\pi+\lambda\mathsf{Id})^{-1}(m_\pi-m_{\mu_0})(x)\right\|^2 \\
&=4\left\|\nabla\left(\sum_{i\geq1}\frac{\varrho_i}{\varrho_i+\lambda}\left\langle\frac{d\mu_0}{d\pi}-1,e_i\right\rangle_{L^2(\pi)}e_i(x)\right)\right\|^2. \tag*{(125)}
\end{aligned} ∥∇ h ( x ) ∥ 2 = 4 ∇ ( Σ π + λ Id ) − 1 ( m π − m μ 0 ) ( x ) 2 = 4 ∇ ( i ≥ 1 ∑ ϱ i + λ ϱ i ⟨ d π d μ 0 − 1 , e i ⟩ L 2 ( π ) e i ( x ) ) 2 . (125) For any j ∈ { 1 , ⋯ , d } j\in\{1,\cdots,d\} j ∈ { 1 , ⋯ , d }
g M 0 ( x ) : = ∑ i ≥ M 0 ∣ ϱ i 1 / 2 ⟨ d μ 0 d π − 1 , e i ⟩ L 2 ( π ) ∂ j e i ( x ) ∣ ≤ ( ∑ i ≥ M 0 ϱ i ( ∂ j e i ( x ) ) 2 ) 1 2 ( ∑ i ≥ M 0 ⟨ d μ 0 d π − 1 , e i ⟩ L 2 ( π ) 2 ) 1 2 ≤ ( ∑ i ≥ M 0 ϱ i ( ∂ j e i ( x ) ) 2 ) 1 2 ∥ d μ 0 d π − 1 ∥ L 2 ( π ) . \begin{aligned}
g_{M_0}(x):=\sum_{i\geq M_0}\left|\varrho_i^{1/2}\left\langle\frac{d\mu_0}{d\pi}-1,e_i\right\rangle_{L^2(\pi)}\partial_j e_i(x)\right| \\
&\leq\left(\sum_{i\geq M_0}\varrho_i(\partial_j e_i(x))^2\right)^{\frac{1}{2}}\left(\sum_{i\geq M_0}\left\langle\frac{d\mu_0}{d\pi}-1,e_i\right\rangle_{L^2(\pi)}^2\right)^{\frac{1}{2}} \\
&\leq\left(\sum_{i\geq M_0}\varrho_i(\partial_j e_i(x))^2\right)^{\frac{1}{2}}\left\|\frac{d\mu_0}{d\pi}-1\right\|_{L^2(\pi)}.
\end{aligned} g M 0 ( x ) := i ≥ M 0 ∑ ϱ i 1/2 ⟨ d π d μ 0 − 1 , e i ⟩ L 2 ( π ) ∂ j e i ( x ) ≤ ( i ≥ M 0 ∑ ϱ i ( ∂ j e i ( x ) ) 2 ) 2 1 ( i ≥ M 0 ∑ ⟨ d π d μ 0 − 1 , e i ⟩ L 2 ( π ) 2 ) 2 1 ≤ ( i ≥ M 0 ∑ ϱ i ( ∂ j e i ( x ) ) 2 ) 2 1 d π d μ 0 − 1 L 2 ( π ) . Since
∑ i ≥ 1 ∑ j = 1 d ϱ i ( ∂ j e i ( x ) ) 2 = ∑ i ≥ 1 ∑ j = 1 d ϱ i ⟨ ∂ j k ( x , ⋅ ) , e i ⟩ H 2 = ∑ i ≥ 1 ∑ j = 1 d ⟨ ∂ j k ( x , ⋅ ) , ϱ i e i ⟩ H 2 = ∑ j = 1 d ∥ ∂ j k ( x , ⋅ ) ∥ H 2 = ∥ ∇ 1 k ( x , ⋅ ) ∥ H 2 ≤ K 1 d , (126) \begin{aligned}
\sum_{i\geq1}\sum_{j=1}^{d}\varrho_i(\partial_j e_i(x))^2=\sum_{i\geq1}\sum_{j=1}^{d}\varrho_i\langle\partial_j k(x,\cdot),e_i\rangle_{\mathcal H}^2 \\
&=\sum_{i\geq1}\sum_{j=1}^{d}\langle\partial_j k(x,\cdot),\sqrt{\varrho_i}e_i\rangle_{\mathcal H}^2 \\
&=\sum_{j=1}^{d}\|\partial_j k(x,\cdot)\|_{\mathcal H}^2=\|\nabla_1 k(x,\cdot)\|_{\mathcal H}^2\leq K_{1d}, \tag*{(126)}
\end{aligned} i ≥ 1 ∑ j = 1 ∑ d ϱ i ( ∂ j e i ( x ) ) 2 = i ≥ 1 ∑ j = 1 ∑ d ϱ i ⟨ ∂ j k ( x , ⋅ ) , e i ⟩ H 2 = i ≥ 1 ∑ j = 1 ∑ d ⟨ ∂ j k ( x , ⋅ ) , ϱ i e i ⟩ H 2 = j = 1 ∑ d ∥ ∂ j k ( x , ⋅ ) ∥ H 2 = ∥ ∇ 1 k ( x , ⋅ ) ∥ H 2 ≤ K 1 d , (126) so ∑ i ≥ M 0 ϱ i ( ∂ j e i ( x ) ) 2 \sum_{i\ge M_0} \varrho_i(\partial_j e_i(x))^2 ∑ i ≥ M 0 ϱ i ( ∂ j e i ( x ) ) 2 0 0 0 g M 0 ( x ) g_{M_0}(x) g M 0 ( x ) 0 0 0
∥ ∇ h ( x ) ∥ 2 = 4 ∥ ∑ i ≥ 1 ϱ i ϱ i + λ ⟨ d μ 0 d π − 1 , e i ⟩ L 2 ( π ) ∇ e i ( x ) ∥ 2 ≤ 4 ( ∑ i ≥ 1 ϱ i ( ϱ i + λ ) 2 ⟨ d μ 0 d π − 1 , e i ⟩ L 2 ( π ) 2 ) ( ∑ i ≥ 1 ϱ i ∥ ∇ e i ( x ) ∥ 2 ) ≤ 4 λ ( ∑ i ≥ 1 ⟨ d μ 0 d π − 1 , e i ⟩ L 2 ( π ) 2 ) ( ∑ i ≥ 1 ϱ i ∥ ∇ e i ( x ) ∥ 2 ) ≤ 4 χ 2 ( μ 0 ∥ π ) K 1 d λ . (127) \begin{aligned}
\|\nabla h(x)\|^2
&=4\left\|\sum_{i\ge1}\frac{\varrho_i}{\varrho_i+\lambda}\left\langle\frac{\mathrm{d}\mu_0}{\mathrm{d}\pi}-1,e_i\right\rangle_{L^2(\pi)}\nabla e_i(x)\right\|^2 \\
&\le4\left(\sum_{i\ge1}\frac{\varrho_i}{(\varrho_i+\lambda)^2}\left\langle\frac{\mathrm{d}\mu_0}{\mathrm{d}\pi}-1,e_i\right\rangle_{L^2(\pi)}^2\right)\left(\sum_{i\ge1}\varrho_i\|\nabla e_i(x)\|^2\right) \\
&\le\frac{4}{\lambda}\left(\sum_{i\ge1}\left\langle\frac{\mathrm{d}\mu_0}{\mathrm{d}\pi}-1,e_i\right\rangle_{L^2(\pi)}^2\right)\left(\sum_{i\ge1}\varrho_i\|\nabla e_i(x)\|^2\right) \\
&\le4\chi^2(\mu_0\Vert\pi)\frac{K_{1d}}{\lambda}. \tag*{(127)}
\end{aligned} ∥∇ h ( x ) ∥ 2 = 4 i ≥ 1 ∑ ϱ i + λ ϱ i ⟨ d π d μ 0 − 1 , e i ⟩ L 2 ( π ) ∇ e i ( x ) 2 ≤ 4 ( i ≥ 1 ∑ ( ϱ i + λ ) 2 ϱ i ⟨ d π d μ 0 − 1 , e i ⟩ L 2 ( π ) 2 ) ( i ≥ 1 ∑ ϱ i ∥∇ e i ( x ) ∥ 2 ) ≤ λ 4 ( i ≥ 1 ∑ ⟨ d π d μ 0 − 1 , e i ⟩ L 2 ( π ) 2 ) ( i ≥ 1 ∑ ϱ i ∥∇ e i ( x ) ∥ 2 ) ≤ 4 χ 2 ( μ 0 ∥ π ) λ K 1 d . (127) The first inequality follows from Cauchy Schwartz, the penultimate inequality follows by noting that ϱ i ( ϱ i + λ ) 2 ≤ 1 λ \frac{\varrho_i}{(\varrho_i+\lambda)^2}\le\frac{1}{\lambda} ( ϱ i + λ ) 2 ϱ i ≤ λ 1 H V ⪯ β I d \mathbf{H}V\preceq\beta\mathrm{Id} H V ⪯ β Id
∫ ∇ h ( x ) T H V ( φ t ( x ) ) ∇ h ( x ) d μ 0 ( x ) ≤ β ∫ ∥ ∇ h ( x ) ∥ 2 d μ 0 ( x ) ≤ 4 β χ 2 ( μ 0 ∥ π ) K 1 d λ . \int\nabla h(x)^{\mathsf{T}}\mathbf{H}V(\varphi_t(x))\nabla h(x)\,\mathrm{d}\mu_0(x)
\le\beta\int\|\nabla h(x)\|^2\,\mathrm{d}\mu_0(x)
\le4\beta\chi^2(\mu_0\Vert\pi)\frac{K_{1d}}{\lambda}. ∫ ∇ h ( x ) T H V ( φ t ( x )) ∇ h ( x ) d μ 0 ( x ) ≤ β ∫ ∥∇ h ( x ) ∥ 2 d μ 0 ( x ) ≤ 4 β χ 2 ( μ 0 ∥ π ) λ K 1 d . We now turn to proving the second statement. Similarly to (B.11), we have
∥ H h ( x ) ∥ F 2 ≤ 4 χ 2 ( μ 0 ∥ π ) K 2 d λ . (128) \|\mathbf{H}h(x)\|_F^2\le4\chi^2(\mu_0\Vert\pi)\frac{K_{2d}}{\lambda}. \tag*{(128)} ∥ H h ( x ) ∥ F 2 ≤ 4 χ 2 ( μ 0 ∥ π ) λ K 2 d . (128) Using 2 ( 1 + λ ) γ χ 2 ( μ 0 ∥ π ) K 2 d λ ≤ ζ − 1 ζ 2(1+\lambda)\gamma\sqrt{\chi^2(\mu_0\Vert\pi)\frac{K_{2d}}{\lambda}}\le\frac{\zeta-1}{\zeta} 2 ( 1 + λ ) γ χ 2 ( μ 0 ∥ π ) λ K 2 d ≤ ζ ζ − 1 1 < ζ < 2 1<\zeta<2 1 < ζ < 2 I d − t ( 1 + λ ) H h ( x ) \mathrm{Id}-t(1+\lambda)\mathbf{H}h(x) Id − t ( 1 + λ ) H h ( x )
∥ ( I d − t ( 1 + λ ) H h ( x ) ) − 1 ∥ F ≤ ∑ m ≥ 0 ∥ t ( 1 + λ ) H h ( x ) ∥ F m ≤ ∑ m ≥ 0 ( γ ( 1 + λ ) 2 χ 2 ( μ 0 ∥ π ) K 2 d λ ) m (129) \begin{aligned}
\left\|(\mathrm{Id}-t(1+\lambda)\mathbf{H}h(x))^{-1}\right\|_F
\le\sum_{m\ge0}\left\|t(1+\lambda)\mathbf{H}h(x)\right\|_F^m \\
&\le\sum_{m\ge0}\left(\gamma(1+\lambda)2\sqrt{\chi^2(\mu_0\Vert\pi)\frac{K_{2d}}{\lambda}}\right)^m \tag*{(129)}
\end{aligned} ( Id − t ( 1 + λ ) H h ( x ) ) − 1 F ≤ m ≥ 0 ∑ ∥ t ( 1 + λ ) H h ( x ) ∥ F m ≤ m ≥ 0 ∑ ( γ ( 1 + λ ) 2 χ 2 ( μ 0 ∥ π ) λ K 2 d ) m (129) ≤ ∑ m ≥ 0 ( ζ − 1 ζ ) m = ζ . (130) \le\sum_{m\ge0}\left(\frac{\zeta-1}{\zeta}\right)^{m}=\zeta. \tag*{(130)} ≤ m ≥ 0 ∑ ( ζ ζ − 1 ) m = ζ . (130) Therefore, (B.8) is proved by combining (B.12) and (B.13),
∫ ∥ H h ( x ) ( I d − t ( 1 + λ ) H h ( x ) ) − 1 ∥ F 2 d μ 0 ( x ) ≤ ∫ ∥ H h ( x ) ∥ F 2 ∥ ( I d − t ( 1 + λ ) H h ( x ) ) − 1 ∥ F 2 d μ 0 ( x ) ≤ 4 ζ 2 χ 2 ( μ 0 ∥ π ) K 2 d λ , \begin{aligned}
\int\left\|Hh(x)\left(\mathrm{Id}-t(1+\lambda)Hh(x)\right)^{-1}\right\|_{F}^{2}\,d\mu_{0}(x) \\
&\le\int\|Hh(x)\|_{F}^{2}\left\|\left(\mathrm{Id}-t(1+\lambda)Hh(x)\right)^{-1}\right\|_{F}^{2}\,d\mu_{0}(x)\le4\zeta^{2}\chi^{2}(\mu_{0}\|\pi)\frac{K_{2d}}{\lambda},
\end{aligned} ∫ H h ( x ) ( Id − t ( 1 + λ ) H h ( x ) ) − 1 F 2 d μ 0 ( x ) ≤ ∫ ∥ H h ( x ) ∥ F 2 ( Id − t ( 1 + λ ) H h ( x ) ) − 1 F 2 d μ 0 ( x ) ≤ 4 ζ 2 χ 2 ( μ 0 ∥ π ) λ K 2 d , and the result follows. □
Lemma B.8. Let H \mathcal{H} H ξ 1 , … , ξ n : Ω → H \xi_{1},\ldots,\xi_{n}:\Omega\to\mathcal{H} ξ 1 , … , ξ n : Ω → H n n n H \mathcal{H} H ∥ ξ i ∥ H ≤ B \|\xi_{i}\|_{\mathcal{H}}\le B ∥ ξ i ∥ H ≤ B
E ∥ 1 n ∑ i = 1 n ξ i − E [ ξ 1 ] ∥ H ≤ 2 π B 2 n , and E ∥ 1 n ∑ i = 1 n ξ i − E [ ξ 1 ] ∥ H 2 ≤ B 2 n . \mathbb{E}\left\|\frac{1}{n}\sum_{i=1}^{n}\xi_{i}-\mathbb{E}[\xi_{1}]\right\|_{\mathcal{H}}\le\frac{\sqrt{2\pi}B}{2\sqrt{n}},
\qquad\text{and}\qquad
\mathbb{E}\left\|\frac{1}{n}\sum_{i=1}^{n}\xi_{i}-\mathbb{E}[\xi_{1}]\right\|_{\mathcal{H}}^{2}\le\frac{B^{2}}{n}. E n 1 i = 1 ∑ n ξ i − E [ ξ 1 ] H ≤ 2 n 2 π B , and E n 1 i = 1 ∑ n ξ i − E [ ξ 1 ] H 2 ≤ n B 2 . Proof. We know from Corollary 6.15 of Steinwart and Christmann (2008) that
P ( ∥ 1 n ∑ i = 1 n ξ i − E [ ξ 1 ] ∥ H ≥ t ) ≤ 2 exp ( − 2 n t 2 / B 2 ) . \mathbb{P}\left(\left\|\frac{1}{n}\sum_{i=1}^{n}\xi_{i}-\mathbb{E}[\xi_{1}]\right\|_{\mathcal{H}}\ge t\right)\le2\exp\left(-2nt^{2}/B^{2}\right). P ( n 1 i = 1 ∑ n ξ i − E [ ξ 1 ] H ≥ t ) ≤ 2 exp ( − 2 n t 2 / B 2 ) . Then, denote R : = ∥ 1 n ∑ i = 1 n ξ i − E [ ξ 1 ] ∥ H R:=\left\|\frac{1}{n}\sum_{i=1}^{n}\xi_{i}-\mathbb{E}[\xi_{1}]\right\|_{\mathcal{H}} R := n 1 ∑ i = 1 n ξ i − E [ ξ 1 ] H
E [ R ] = ∫ 0 ∞ P ( R ≥ t ) d t ≤ 2 ∫ 0 ∞ exp ( − 2 n t 2 / B 2 ) d t = 2 π B 2 n . \mathbb{E}[R]=\int_{0}^{\infty}\mathbb{P}(R\ge t)\,dt\le2\int_{0}^{\infty}\exp\left(-2nt^{2}/B^{2}\right)\,dt=\frac{\sqrt{2\pi}B}{2\sqrt{n}}. E [ R ] = ∫ 0 ∞ P ( R ≥ t ) d t ≤ 2 ∫ 0 ∞ exp ( − 2 n t 2 / B 2 ) d t = 2 n 2 π B . The other part is proved similarly. □
Lemma B.9 (Wasserstein Hessian of F χ 2 \mathcal{F}_{\chi^{2}} F χ 2 Let ρ ∈ P 2 ( R d ) \rho\in\mathcal{P}_{2}(\mathbb{R}^{d}) ρ ∈ P 2 ( R d ) ϕ ∈ C ∞ ( R d ) \phi\in C^{\infty}(\mathbb{R}^{d}) ϕ ∈ C ∞ ( R d ) ( ρ s ) 0 ≤ s ≤ 1 (\rho_{s})_{0\le s\le1} ( ρ s ) 0 ≤ s ≤ 1 ρ \rho ρ ( I d + ∇ ϕ ) # ρ (\mathrm{Id}+\nabla\phi)_{\#}\rho ( Id + ∇ ϕ ) # ρ ρ s = ( I d + s ∇ ϕ ) # ρ \rho_{s}=(\mathrm{Id}+s\nabla\phi)_{\#}\rho ρ s = ( Id + s ∇ ϕ ) # ρ φ s : R d → R d \varphi_{s}:\mathbb{R}^{d}\to\mathbb{R}^{d} φ s : R d → R d x ↦ x + s ∇ ϕ ( x ) x\mapsto x+s\nabla\phi(x) x ↦ x + s ∇ ϕ ( x ) ω s : R d → R d \omega_{s}:\mathbb{R}^{d}\to\mathbb{R}^{d} ω s : R d → R d x ↦ [ ∇ ϕ ∘ φ s − 1 ] ( x ) x\mapsto[\nabla\phi\circ\varphi_{s}^{-1}](x) x ↦ [ ∇ ϕ ∘ φ s − 1 ] ( x ) π ( x ) ∝ exp ( − V ( x ) ) \pi(x)\propto\exp(-V(x)) π ( x ) ∝ exp ( − V ( x )) s ↦ χ 2 ( ρ s ∥ π ) s\mapsto\chi^{2}(\rho_{s}\|\pi) s ↦ χ 2 ( ρ s ∥ π )
d 2 d s 2 χ 2 ( ρ s ∥ π ) = ∫ ρ s ( x ) π ( x ) ( ∇ V ( x ) T ω s ( x ) − ∇ ⋅ ω s ( x ) ) 2 d ρ s ( x ) \frac{d^{2}}{ds^{2}}\chi^{2}(\rho_{s}\|\pi)=\int\frac{\rho_{s}(x)}{\pi(x)}\left(\nabla V(x)^{\mathsf{T}}\omega_{s}(x)-\nabla\cdot\omega_{s}(x)\right)^{2}\,d\rho_{s}(x) d s 2 d 2 χ 2 ( ρ s ∥ π ) = ∫ π ( x ) ρ s ( x ) ( ∇ V ( x ) T ω s ( x ) − ∇ ⋅ ω s ( x ) ) 2 d ρ s ( x ) + ∫ ρ s ( x ) π ( x ) ω s ( x ) T H V ( x ) ω s ( x ) d ρ s ( x ) + ∫ ρ s ( x ) π ( x ) ∥ ∇ ω s ( x ) ∥ F 2 d ρ s ( x ) . (131) +\int\frac{\rho_{s}(x)}{\pi(x)}\omega_{s}(x)^{\mathsf{T}}HV(x)\omega_{s}(x)\,d\rho_{s}(x)+\int\frac{\rho_{s}(x)}{\pi(x)}\|\nabla\omega_{s}(x)\|_{F}^{2}\,d\rho_{s}(x). \tag*{(131)} + ∫ π ( x ) ρ s ( x ) ω s ( x ) T H V ( x ) ω s ( x ) d ρ s ( x ) + ∫ π ( x ) ρ s ( x ) ∥∇ ω s ( x ) ∥ F 2 d ρ s ( x ) . (131) Equivalently, the second order derivative of s ↦ χ 2 ( ρ s ∥ π ) s \mapsto\chi^2(\rho_s\Vert\pi) s ↦ χ 2 ( ρ s ∥ π )
d 2 d s 2 χ 2 ( ρ s ∥ π ) = 2 ∫ ( ∇ ⋅ ( ω s ( x ) ρ s ( x ) ) 1 π ( x ) ) 2 π ( x ) d x + 2 ∫ ω s ( x ) T H ( ρ s ( x ) π ( x ) ) ω s ( x ) ρ s ( x ) d x . (132) \frac{d^2}{ds^2}\chi^2(\rho_s\Vert\pi)=2\int\left(\nabla\cdot(\omega_s(x)\rho_s(x))\frac{1}{\pi(x)}\right)^2\pi(x)dx+2\int\omega_s(x)^{\mathsf{T}}\mathrm{H}\left(\frac{\rho_s(x)}{\pi(x)}\right)\omega_s(x)\rho_s(x)dx. \tag*{(132)} d s 2 d 2 χ 2 ( ρ s ∥ π ) = 2 ∫ ( ∇ ⋅ ( ω s ( x ) ρ s ( x )) π ( x ) 1 ) 2 π ( x ) d x + 2 ∫ ω s ( x ) T H ( π ( x ) ρ s ( x ) ) ω s ( x ) ρ s ( x ) d x . (132) Proof. (B.15) is provided in the Example 15.9 of Villani et al. (2009) by taking m = 2 m=2 m = 2 x x x ρ s \rho_s ρ s π \pi π ω s \omega_s ω s
In order to prove (B.16), we need to expand the terms in (B.15) accordingly. We denote the three terms in the RHS of (B.15) as ( A ) (A) ( A ) ( B ) (B) ( B ) ( C ) (C) ( C )
( A ) = ∫ ρ s 2 π ( ∇ ∇ T ω s − ∇ ⋅ ω s ) 2 d x = ∫ ρ s 2 π ( − 1 π ∇ π T ω s − ∇ ⋅ ω s ) 2 d x = ∫ ρ s 2 π 3 ( ∇ π T ω s ) 2 d x ⏟ ( A 1 ) + 2 ∫ ρ s 2 π 2 ( ∇ π T ω s ) ∇ ⋅ ω s d x ⏟ ( A 2 ) + ∫ ρ s 2 π ( ∇ ⋅ ω s ) 2 d x ⏟ ( A 3 ) . \begin{aligned}
(A)&=\int\frac{\rho_s^2}{\pi}\left(\nabla\nabla^{\mathsf{T}}\omega_s-\nabla\cdot\omega_s\right)^2dx\\
&=\int\frac{\rho_s^2}{\pi}\left(-\frac{1}{\pi}\nabla\pi^{\mathsf{T}}\omega_s-\nabla\cdot\omega_s\right)^2dx\\
&=\underbrace{\int\frac{\rho_s^2}{\pi^3}(\nabla\pi^{\mathsf{T}}\omega_s)^2dx}_{(A_1)}\\
&\quad+2\underbrace{\int\frac{\rho_s^2}{\pi^2}(\nabla\pi^{\mathsf{T}}\omega_s)\nabla\cdot\omega_sdx}_{(A_2)}\\
&\quad+\underbrace{\int\frac{\rho_s^2}{\pi}(\nabla\cdot\omega_s)^2dx}_{(A_3)}.
\end{aligned} ( A ) = ∫ π ρ s 2 ( ∇ ∇ T ω s − ∇ ⋅ ω s ) 2 d x = ∫ π ρ s 2 ( − π 1 ∇ π T ω s − ∇ ⋅ ω s ) 2 d x = ( A 1 ) ∫ π 3 ρ s 2 ( ∇ π T ω s ) 2 d x + 2 ( A 2 ) ∫ π 2 ρ s 2 ( ∇ π T ω s ) ∇ ⋅ ω s d x + ( A 3 ) ∫ π ρ s 2 ( ∇ ⋅ ω s ) 2 d x . Then we are going to use integration by parts for ( A 2 ) (A_2) ( A 2 ) ( A 3 ) (A_3) ( A 3 )
( A 2 ) = 2 ∫ ρ s 2 π 2 ( ∇ π T ω s ) ∇ ⋅ ω s d x = − 2 ∫ ω s T ∇ ( ρ s 2 π 2 ( ∇ π T ω s ) ) d x = − 4 ∫ ( ω s T ∇ ρ s ) ( ∇ π T ω s ) ρ s π 2 d x + 4 ∫ ( ω s T ∇ π ) 2 ρ s 2 π 3 d x \begin{aligned}
(A_2)&=2\int\frac{\rho_s^2}{\pi^2}(\nabla\pi^{\mathsf{T}}\omega_s)\nabla\cdot\omega_sdx\\
&=-2\int\omega_s^{\mathsf{T}}\nabla\left(\frac{\rho_s^2}{\pi^2}(\nabla\pi^{\mathsf{T}}\omega_s)\right)dx\\
&=-4\int(\omega_s^{\mathsf{T}}\nabla\rho_s)(\nabla\pi^{\mathsf{T}}\omega_s)\frac{\rho_s}{\pi^2}dx+4\int(\omega_s^{\mathsf{T}}\nabla\pi)^2\frac{\rho_s^2}{\pi^3}dx
\end{aligned} ( A 2 ) = 2 ∫ π 2 ρ s 2 ( ∇ π T ω s ) ∇ ⋅ ω s d x = − 2 ∫ ω s T ∇ ( π 2 ρ s 2 ( ∇ π T ω s ) ) d x = − 4 ∫ ( ω s T ∇ ρ s ) ( ∇ π T ω s ) π 2 ρ s d x + 4 ∫ ( ω s T ∇ π ) 2 π 3 ρ s 2 d x − 2 ∫ ( ω s T H π ω s ) ρ s 2 π 2 d x − 2 ∫ ( ω s T ∇ π ∇ π ) ρ s 2 π 2 d x -2\int(\omega_s^{\mathsf T}\mathbf{H}\pi\omega_s)\frac{\rho_s^2}{\pi^2}\,dx-2\int(\omega_s^{\mathsf T}\nabla\pi\nabla\pi)\frac{\rho_s^2}{\pi^2}\,dx − 2 ∫ ( ω s T H π ω s ) π 2 ρ s 2 d x − 2 ∫ ( ω s T ∇ π ∇ π ) π 2 ρ s 2 d x ( A 3 ) = ∫ ρ s 2 π ( ∇ ⋅ ω s ) 2 d x = − ∫ ω s T ∇ ( ∇ ⋅ ω s ρ s 2 π ) d x = − ∫ ω s T ∇ ( ∇ ⋅ ω s ) ρ s 2 π d x ⏟ ( A 31 ) + ∫ ∇ ⋅ ω s ( ω s T ∇ ρ s ) 2 ρ s π d x ⏟ ( A 32 ) + ∫ ∇ ⋅ ω s ( ω s T ∇ π ) ρ s 2 π 2 d x ⏟ ( A 33 ) . \begin{aligned}
(A_3)&=\int\frac{\rho_s^2}{\pi}(\nabla\cdot\omega_s)^2\,dx\\
&=-\int\omega_s^{\mathsf T}\nabla\left(\nabla\cdot\omega_s\frac{\rho_s^2}{\pi}\right)\,dx\\
&=\underbrace{-\int\omega_s^{\mathsf T}\nabla(\nabla\cdot\omega_s)\frac{\rho_s^2}{\pi}\,dx}_{(A_{31})}\\
&\quad+\underbrace{\int\nabla\cdot\omega_s(\omega_s^{\mathsf T}\nabla\rho_s)\frac{2\rho_s}{\pi}\,dx}_{(A_{32})}\\
&\quad+\underbrace{\int\nabla\cdot\omega_s(\omega_s^{\mathsf T}\nabla\pi)\frac{\rho_s^2}{\pi^2}\,dx}_{(A_{33})}.
\end{aligned} ( A 3 ) = ∫ π ρ s 2 ( ∇ ⋅ ω s ) 2 d x = − ∫ ω s T ∇ ( ∇ ⋅ ω s π ρ s 2 ) d x = ( A 31 ) − ∫ ω s T ∇ ( ∇ ⋅ ω s ) π ρ s 2 d x + ( A 32 ) ∫ ∇ ⋅ ω s ( ω s T ∇ ρ s ) π 2 ρ s d x + ( A 33 ) ∫ ∇ ⋅ ω s ( ω s T ∇ π ) π 2 ρ s 2 d x . Furthermore, we use integration by parts for ( A 32 ) (A_{32}) ( A 32 ) ( A 33 ) (A_{33}) ( A 33 )
( A 32 ) = − 2 ∫ ∇ ⋅ ω s ( ω s T ∇ ρ s ) ρ s π d x = 2 ∫ ω s T ∇ ( ( ω s T ∇ ρ s ) ρ s π ) d x = 2 ∫ ( ω s T ∇ ω s ∇ ρ s ) ρ s π d x + 2 ∫ ( ω s T H ρ s ω s ) ρ s π d x + 2 ∫ ( ω s T ∇ ρ s ) 2 1 π d x − 2 ∫ ( ω s T ∇ ρ s ) ( ω s T ∇ π ) ρ s π 2 d x , \begin{aligned}
(A_{32})&=-2\int\nabla\cdot\omega_s(\omega_s^{\mathsf T}\nabla\rho_s)\frac{\rho_s}{\pi}\,dx\\
&=2\int\omega_s^{\mathsf T}\nabla\left((\omega_s^{\mathsf T}\nabla\rho_s)\frac{\rho_s}{\pi}\right)\,dx\\
&=2\int(\omega_s^{\mathsf T}\nabla\omega_s\nabla\rho_s)\frac{\rho_s}{\pi}\,dx+2\int(\omega_s^{\mathsf T}\mathbf{H}_{\rho_s}\omega_s)\frac{\rho_s}{\pi}\,dx\\
&\quad+2\int(\omega_s^{\mathsf T}\nabla\rho_s)^2\frac{1}{\pi}\,dx\\
&\quad-2\int(\omega_s^{\mathsf T}\nabla\rho_s)(\omega_s^{\mathsf T}\nabla\pi)\frac{\rho_s}{\pi^2}\,dx,
\end{aligned} ( A 32 ) = − 2 ∫ ∇ ⋅ ω s ( ω s T ∇ ρ s ) π ρ s d x = 2 ∫ ω s T ∇ ( ( ω s T ∇ ρ s ) π ρ s ) d x = 2 ∫ ( ω s T ∇ ω s ∇ ρ s ) π ρ s d x + 2 ∫ ( ω s T H ρ s ω s ) π ρ s d x + 2 ∫ ( ω s T ∇ ρ s ) 2 π 1 d x − 2 ∫ ( ω s T ∇ ρ s ) ( ω s T ∇ π ) π 2 ρ s d x , ( A 33 ) = ∫ ∇ ⋅ ω s ( ω s T ∇ π ) ρ s 2 π 2 d x = − ∫ ω s T ∇ ( ( ω s T ∇ π ) ρ s 2 π 2 ) d x = − ∫ ( ω s T ∇ ω s ∇ π ) ρ s 2 π 2 d x − ∫ ( ω s T H π ω s ) ρ s 2 π 2 d x − 2 ∫ ( ω s T ∇ ρ s ) ( ω s T ∇ π ) ρ s π 2 d x + 2 ∫ ( ω s T ∇ π ) 2 ρ s 2 π 3 d x . \begin{aligned}
(A_{33})&=\int\nabla\cdot\omega_s(\omega_s^{\mathsf T}\nabla\pi)\frac{\rho_s^2}{\pi^2}\,dx\\
&=-\int\omega_s^{\mathsf T}\nabla\left((\omega_s^{\mathsf T}\nabla\pi)\frac{\rho_s^2}{\pi^2}\right)\,dx\\
&=-\int(\omega_s^{\mathsf T}\nabla\omega_s\nabla\pi)\frac{\rho_s^2}{\pi^2}\,dx-\int(\omega_s^{\mathsf T}\mathbf{H}_{\pi}\omega_s)\frac{\rho_s^2}{\pi^2}\,dx\\
&\quad-2\int(\omega_s^{\mathsf T}\nabla\rho_s)(\omega_s^{\mathsf T}\nabla\pi)\frac{\rho_s}{\pi^2}\,dx\\
&\quad+2\int(\omega_s^{\mathsf T}\nabla\pi)^2\frac{\rho_s^2}{\pi^3}\,dx.
\end{aligned} ( A 33 ) = ∫ ∇ ⋅ ω s ( ω s T ∇ π ) π 2 ρ s 2 d x = − ∫ ω s T ∇ ( ( ω s T ∇ π ) π 2 ρ s 2 ) d x = − ∫ ( ω s T ∇ ω s ∇ π ) π 2 ρ s 2 d x − ∫ ( ω s T H π ω s ) π 2 ρ s 2 d x − 2 ∫ ( ω s T ∇ ρ s ) ( ω s T ∇ π ) π 2 ρ s d x + 2 ∫ ( ω s T ∇ π ) 2 π 3 ρ s 2 d x . Having completed ( A ) (A) ( A ) ( B ) (B) ( B )
( B ) = ∫ ρ s 2 π ω s T H V ω s d x = − ∫ ω s T H π ω s ρ s 2 π 2 d x + ∫ ( ∇ π T ω s ) 2 ρ s 2 π 3 d x . (133) \begin{aligned}
(B)=\int\frac{\rho_s^2}{\pi}\omega_s^{\mathsf T}\mathbf{H}V\omega_s\,dx \\
&= - \int\omega_s^{\mathsf{T}}\mathbf{H}\pi\omega_s \frac{\rho_s^2}{\pi^2}\,dx + \int(\nabla\pi^{\mathsf{T}}\omega_s)^2 \frac{\rho_s^2}{\pi^3}\,dx. \tag*{(133)}
\end{aligned} ( B ) = ∫ π ρ s 2 ω s T H V ω s d x = − ∫ ω s T H π ω s π 2 ρ s 2 d x + ∫ ( ∇ π T ω s ) 2 π 3 ρ s 2 d x . (133) So, combining ( A ) (A) ( A ) ( B ) (B) ( B ) ( C ) (C) ( C )
d 2 d s 2 χ 2 ( ρ s ∥ π ) = ( A ) + ( B ) + ( C ) = ( A 1 ) + ( A 2 ) + ( A 31 ) + ( A 32 ) + ( A 33 ) + ( B ) + ( C ) = ∫ ρ s 2 π 3 ( ∇ π T ω s ) 2 d x − 4 ∫ ( ω s T ∇ ρ s ) ( ∇ π T ω s ) ρ s π 2 d x + 4 ∫ ( ω s T ∇ π ) 2 ρ s 2 π 3 d x − 2 ∫ ( ω s T H π ω s ) ρ s 2 π 2 d x − 2 ∫ ( ω s T ∇ ω s ∇ π ) ρ s 2 π 2 d x + ( A 31 ) + ( A 32 ) + ( A 33 ) − ∫ ω s T H π ω s ρ s 2 π 2 d x + ∫ ( ∇ π T ω s ) 2 ρ s 2 π 3 d x + ∫ ρ s 2 π ∥ ∇ ω s ∥ F 2 d x . \begin{aligned}
\frac{d^2}{ds^2}\chi^2(\rho_s\Vert\pi)
&= (A) + (B) + (C) \\
&= (A_{1}) + (A_{2}) + (A_{31}) + (A_{32}) \\
&\quad+ (A_{33}) + (B) + (C) \\
&= \int\frac{\rho_s^2}{\pi^3}(\nabla\pi^{\mathsf{T}}\omega_s)^2\,dx \\
&\quad- 4\int(\omega_s^{\mathsf{T}}\nabla\rho_s)(\nabla\pi^{\mathsf{T}}\omega_s)\frac{\rho_s}{\pi^2}\,dx \\
&\quad+ 4\int(\omega_s^{\mathsf{T}}\nabla\pi)^2\frac{\rho_s^2}{\pi^3}\,dx - 2\int(\omega_s^{\mathsf{T}}\mathbf{H}\pi\omega_s)\frac{\rho_s^2}{\pi^2}\,dx \\
&\quad- 2\int(\omega_s^{\mathsf{T}}\nabla\omega_s\nabla\pi)\frac{\rho_s^2}{\pi^2}\,dx + (A_{31}) + (A_{32}) + (A_{33}) \\
&\quad- \int\omega_s^{\mathsf{T}}\mathbf{H}\pi\omega_s\frac{\rho_s^2}{\pi^2}\,dx + \int(\nabla\pi^{\mathsf{T}}\omega_s)^2\frac{\rho_s^2}{\pi^3}\,dx \\
&\quad+ \int\frac{\rho_s^2}{\pi}\lVert\nabla\omega_s\rVert_F^2\,dx.
\end{aligned} d s 2 d 2 χ 2 ( ρ s ∥ π ) = ( A ) + ( B ) + ( C ) = ( A 1 ) + ( A 2 ) + ( A 31 ) + ( A 32 ) + ( A 33 ) + ( B ) + ( C ) = ∫ π 3 ρ s 2 ( ∇ π T ω s ) 2 d x − 4 ∫ ( ω s T ∇ ρ s ) ( ∇ π T ω s ) π 2 ρ s d x + 4 ∫ ( ω s T ∇ π ) 2 π 3 ρ s 2 d x − 2 ∫ ( ω s T H π ω s ) π 2 ρ s 2 d x − 2 ∫ ( ω s T ∇ ω s ∇ π ) π 2 ρ s 2 d x + ( A 31 ) + ( A 32 ) + ( A 33 ) − ∫ ω s T H π ω s π 2 ρ s 2 d x + ∫ ( ∇ π T ω s ) 2 π 3 ρ s 2 d x + ∫ π ρ s 2 ∥ ∇ ω s ∥ F 2 d x . Since ω s = ∇ ϕ ∘ φ s − 1 \omega_s=\nabla\phi\circ\varphi_s^{-1} ω s = ∇ ϕ ∘ φ s − 1 R d \mathbb{R}^d R d R d \mathbb{R}^d R d ω s , i : = [ ∇ ϕ ∘ φ s − 1 ] i \omega_{s,i}:=[\nabla\phi\circ\varphi_s^{-1}]_i ω s , i := [ ∇ ϕ ∘ φ s − 1 ] i R d \mathbb{R}^d R d R \mathbb{R} R ω s , i ∂ j ω s , i \omega_{s,i}\partial_j\omega_{s,i} ω s , i ∂ j ω s , i ϕ ∈ C c ∞ \phi\in C_c^\infty ϕ ∈ C c ∞
0 = ∑ i , j ∫ ∂ j ( ω s , i ∂ j ω s , i ρ s 2 π ) d x 0=\sum_{i,j}\int\partial_j\left(\omega_{s,i}\partial_j\omega_{s,i}\frac{\rho_s^2}{\pi}\right)\,dx 0 = i , j ∑ ∫ ∂ j ( ω s , i ∂ j ω s , i π ρ s 2 ) d x = ∑ i , j ∫ ∂ j ω s , i ∂ j ω s , i ρ s 2 π d x + ∑ i , j ∫ ω s , i ∂ j j ω s , i ρ s 2 π d x + ∑ i , j ∫ ω s , i ∂ j ω s , i ( 2 ρ s ∂ j ρ s π − ρ s 2 ∂ j π π 2 ) d x = ∫ ρ s 2 π ∥ ∇ ω s ∥ F 2 d x + ∫ ρ s 2 π ( ω s T ∇ ( ∇ ⋅ ω s ) ) d x + ∫ ( ω s T ∇ ω s ∇ ρ s ) 2 ρ s π d x − ∫ ( ω s T ∇ ω s ∇ π ) ρ s 2 π 2 d x . \begin{aligned}
&=\sum_{i,j}\int\partial_j\omega_{s,i}\partial_j\omega_{s,i}\frac{\rho_s^2}{\pi}\,dx+\sum_{i,j}\int\omega_{s,i}\partial_{jj}\omega_{s,i}\frac{\rho_s^2}{\pi}\,dx \\
&\quad+\sum_{i,j}\int\omega_{s,i}\partial_j\omega_{s,i}\left(\frac{2\rho_s\partial_j\rho_s}{\pi}-\frac{\rho_s^2\partial_j\pi}{\pi^2}\right)\,dx \\
&=\int\frac{\rho_s^2}{\pi}\lVert\nabla\omega_s\rVert_F^2\,dx+\int\frac{\rho_s^2}{\pi}(\omega_s^{\mathsf{T}}\nabla(\nabla\cdot\omega_s))\,dx \\
&\quad+\int(\omega_s^{\mathsf{T}}\nabla\omega_s\nabla\rho_s)\frac{2\rho_s}{\pi}\,dx \\
&\quad-\int(\omega_s^{\mathsf{T}}\nabla\omega_s\nabla\pi)\frac{\rho_s^2}{\pi^2}\,dx.
\end{aligned} = i , j ∑ ∫ ∂ j ω s , i ∂ j ω s , i π ρ s 2 d x + i , j ∑ ∫ ω s , i ∂ j j ω s , i π ρ s 2 d x + i , j ∑ ∫ ω s , i ∂ j ω s , i ( π 2 ρ s ∂ j ρ s − π 2 ρ s 2 ∂ j π ) d x = ∫ π ρ s 2 ∥ ∇ ω s ∥ F 2 d x + ∫ π ρ s 2 ( ω s T ∇ ( ∇ ⋅ ω s )) d x + ∫ ( ω s T ∇ ω s ∇ ρ s ) π 2 ρ s d x − ∫ ( ω s T ∇ ω s ∇ π ) π 2 ρ s 2 d x . Therefore, replace
∫ ρ s 2 π ∥ ∇ ω s ∥ F 2 d x , \int\frac{\rho_s^2}{\pi}\lVert\nabla\omega_s\rVert_F^2\,dx, ∫ π ρ s 2 ∥ ∇ ω s ∥ F 2 d x , and notice that
− ∫ ρ s 2 π ( ω s T ∇ ( ∇ ⋅ ω s ) ) d x = ( A 31 ) . -\int\frac{\rho_s^2}{\pi}(\omega_s^{\mathsf T}\nabla(\nabla\cdot\omega_s))\,dx=(A_{31}). − ∫ π ρ s 2 ( ω s T ∇ ( ∇ ⋅ ω s )) d x = ( A 31 ) . We then have the following:
d 2 d s 2 χ 2 ( ρ s ∥ π ) = ∫ ρ s 2 π 3 ( ∇ π T ω s ) 2 d x − 4 ∫ ( ω s T ∇ ρ s ) ( ∇ π T ω s ) ρ s π 2 d x + 4 ∫ ( ω s T ∇ π ) 2 ρ s 2 π 3 d x − 2 ∫ ( ω s T H π ω s ) ρ s 2 π 2 d x − 2 ∫ ( ω s T ∇ ω s ∇ π ) ρ s 2 π 2 d x + ( A 31 ) + ( A 32 ) + ( A 33 ) − ∫ ( ω s T H π ω s ) ρ s 2 π 2 d x + ∫ ( ∇ π T ω s ) 2 2 ρ s 2 π 3 d x − ∫ ρ s 2 π ( ω s T ∇ ( ∇ ⋅ ω s ) ) d x − ∫ ( ω s T ∇ ω s ∇ ρ s ) 2 ρ s π d x + ∫ ( ω s T ∇ ω s ∇ π ) ρ s 2 π 2 d x . \begin{aligned}
\frac{d^2}{ds^2}\chi^2(\rho_s\Vert\pi)
={}&\int\frac{\rho_s^2}{\pi^3}(\nabla\pi^{\mathsf T}\omega_s)^2\,dx
-4\int(\omega_s^{\mathsf T}\nabla\rho_s)(\nabla\pi^{\mathsf T}\omega_s)\frac{\rho_s}{\pi^2}\,dx\\
&+4\int(\omega_s^{\mathsf T}\nabla\pi)^2\frac{\rho_s^2}{\pi^3}\,dx
-2\int(\omega_s^{\mathsf T}\mathbf{H}\pi\omega_s)\frac{\rho_s^2}{\pi^2}\,dx\\
&-2\int(\omega_s^{\mathsf T}\nabla\omega_s\nabla\pi)\frac{\rho_s^2}{\pi^2}\,dx
+(A_{31})+(A_{32})+(A_{33})\\
&-\int(\omega_s^{\mathsf T}\mathbf{H}\pi\omega_s)\frac{\rho_s^2}{\pi^2}\,dx
+\int(\nabla\pi^{\mathsf T}\omega_s)^2\frac{2\rho_s^2}{\pi^3}\,dx\\
&-\int\frac{\rho_s^2}{\pi}(\omega_s^{\mathsf T}\nabla(\nabla\cdot\omega_s))\,dx
-\int(\omega_s^{\mathsf T}\nabla\omega_s\nabla\rho_s)\frac{2\rho_s}{\pi}\,dx
+\int(\omega_s^{\mathsf T}\nabla\omega_s\nabla\pi)\frac{\rho_s^2}{\pi^2}\,dx.
\end{aligned} d s 2 d 2 χ 2 ( ρ s ∥ π ) = ∫ π 3 ρ s 2 ( ∇ π T ω s ) 2 d x − 4 ∫ ( ω s T ∇ ρ s ) ( ∇ π T ω s ) π 2 ρ s d x + 4 ∫ ( ω s T ∇ π ) 2 π 3 ρ s 2 d x − 2 ∫ ( ω s T H π ω s ) π 2 ρ s 2 d x − 2 ∫ ( ω s T ∇ ω s ∇ π ) π 2 ρ s 2 d x + ( A 31 ) + ( A 32 ) + ( A 33 ) − ∫ ( ω s T H π ω s ) π 2 ρ s 2 d x + ∫ ( ∇ π T ω s ) 2 π 3 2 ρ s 2 d x − ∫ π ρ s 2 ( ω s T ∇ ( ∇ ⋅ ω s )) d x − ∫ ( ω s T ∇ ω s ∇ ρ s ) π 2 ρ s d x + ∫ ( ω s T ∇ ω s ∇ π ) π 2 ρ s 2 d x . Next, we combine the terms and obtain
= 2 ( ( A 31 ) + ( A 32 ) + ( A 33 ) ) + 6 ∫ ρ s 2 π 3 ( ∇ π T ω s ) 2 d x − 4 ∫ ( ω s T ∇ ρ s ) ( ∇ π T ω s ) ρ s π 2 d x − 3 ∫ ( ω s T H π ω s ) ρ s 2 π 2 d x − ∫ ( ω s T ∇ ω s ∇ π ) ρ s 2 π 2 d x − 2 ∫ ( ω s T ∇ ω s ∇ ρ s ) ρ s π d x − ( A 32 ) − ( A 33 ) . \begin{aligned}
={}&2\bigl((A_{31})+(A_{32})+(A_{33})\bigr)
+6\int\frac{\rho_s^2}{\pi^3}(\nabla\pi^{\mathsf T}\omega_s)^2\,dx\\
&-4\int(\omega_s^{\mathsf T}\nabla\rho_s)(\nabla\pi^{\mathsf T}\omega_s)\frac{\rho_s}{\pi^2}\,dx
-3\int(\omega_s^{\mathsf T}\mathbf{H}\pi\omega_s)\frac{\rho_s^2}{\pi^2}\,dx\\
&-\int(\omega_s^{\mathsf T}\nabla\omega_s\nabla\pi)\frac{\rho_s^2}{\pi^2}\,dx
-2\int(\omega_s^{\mathsf T}\nabla\omega_s\nabla\rho_s)\frac{\rho_s}{\pi}\,dx\\
&-(A_{32})-(A_{33}).
\end{aligned} = 2 ( ( A 31 ) + ( A 32 ) + ( A 33 ) ) + 6 ∫ π 3 ρ s 2 ( ∇ π T ω s ) 2 d x − 4 ∫ ( ω s T ∇ ρ s ) ( ∇ π T ω s ) π 2 ρ s d x − 3 ∫ ( ω s T H π ω s ) π 2 ρ s 2 d x − ∫ ( ω s T ∇ ω s ∇ π ) π 2 ρ s 2 d x − 2 ∫ ( ω s T ∇ ω s ∇ ρ s ) π ρ s d x − ( A 32 ) − ( A 33 ) . Recall that ( A 31 ) + ( A 32 ) + ( A 33 ) = ( A 3 ) (A_{31})+(A_{32})+(A_{33})=(A_3) ( A 31 ) + ( A 32 ) + ( A 33 ) = ( A 3 ) ( A 32 ) (A_{32}) ( A 32 ) ( A 33 ) (A_{33}) ( A 33 )
= 2 ∫ ρ s 2 π ( ∇ ⋅ ω s ) 2 d x + 6 ∫ ρ s 2 π 3 ( ∇ π T ω s ) 2 d x − 4 ∫ ( ω s T ∇ ρ s ) ( ∇ π T ω s ) ρ s π 2 d x − 3 ∫ ( ω s T H π ω s ) ρ s 2 π 2 d x − ∫ ( ω s T ∇ ω s ∇ π ) ρ s 2 π 2 d x − 2 ∫ ( ω s T ∇ ω s ∇ ρ s ) ρ s π d x − ( 2 ∫ ( ω s T ∇ ω s ∇ ρ s ) ρ s π d x + 2 ∫ ( ω s T H ρ s ω s ) ρ s π d x + 2 ∫ ( ω s T ∇ ρ s ) 2 1 π d x − 2 ∫ ( ω s T ∇ ρ s ) ( ω s T ∇ π ) ρ s π 2 d x ) − ( − ∫ ( ω s T ∇ ω s ∇ π ) ρ s 2 π 2 d x − ∫ ( ω s T H π ω s ) ρ s 2 π 2 d x − 2 ∫ ( ω s T ∇ ρ s ) ( ω s T ∇ π ) ρ s 2 π 2 d x + 2 ∫ ( ω s T ∇ π ) 2 ρ s 2 π 3 d x ) \begin{aligned}
={}&2\int\frac{\rho_s^2}{\pi}(\nabla\cdot\omega_s)^2\,dx
+6\int\frac{\rho_s^2}{\pi^3}(\nabla\pi^{\mathsf T}\omega_s)^2\,dx
-4\int(\omega_s^{\mathsf T}\nabla\rho_s)(\nabla\pi^{\mathsf T}\omega_s)\frac{\rho_s}{\pi^2}\,dx\\
&-3\int(\omega_s^{\mathsf T}\mathbf{H}\pi\omega_s)\frac{\rho_s^2}{\pi^2}\,dx
-\int(\omega_s^{\mathsf T}\nabla\omega_s\nabla\pi)\frac{\rho_s^2}{\pi^2}\,dx
-2\int(\omega_s^{\mathsf T}\nabla\omega_s\nabla\rho_s)\frac{\rho_s}{\pi}\,dx\\
&-\left(2\int(\omega_s^{\mathsf T}\nabla\omega_s\nabla\rho_s)\frac{\rho_s}{\pi}\,dx
+2\int(\omega_s^{\mathsf T}\mathbf{H}\rho_s\omega_s)\frac{\rho_s}{\pi}\,dx
+2\int(\omega_s^{\mathsf T}\nabla\rho_s)^2\frac{1}{\pi}\,dx
-2\int(\omega_s^{\mathsf T}\nabla\rho_s)(\omega_s^{\mathsf T}\nabla\pi)\frac{\rho_s}{\pi^2}\,dx\right)\\
&-\left(-\int(\omega_s^{\mathsf T}\nabla\omega_s\nabla\pi)\frac{\rho_s^2}{\pi^2}\,dx
-\int(\omega_s^{\mathsf T}\mathbf{H}\pi\omega_s)\frac{\rho_s^2}{\pi^2}\,dx
-2\int(\omega_s^{\mathsf T}\nabla\rho_s)(\omega_s^{\mathsf T}\nabla\pi)\frac{\rho_s^2}{\pi^2}\,dx
+2\int(\omega_s^{\mathsf T}\nabla\pi)^2\frac{\rho_s^2}{\pi^3}\,dx\right)
\end{aligned} = 2 ∫ π ρ s 2 ( ∇ ⋅ ω s ) 2 d x + 6 ∫ π 3 ρ s 2 ( ∇ π T ω s ) 2 d x − 4 ∫ ( ω s T ∇ ρ s ) ( ∇ π T ω s ) π 2 ρ s d x − 3 ∫ ( ω s T H π ω s ) π 2 ρ s 2 d x − ∫ ( ω s T ∇ ω s ∇ π ) π 2 ρ s 2 d x − 2 ∫ ( ω s T ∇ ω s ∇ ρ s ) π ρ s d x − ( 2 ∫ ( ω s T ∇ ω s ∇ ρ s ) π ρ s d x + 2 ∫ ( ω s T H ρ s ω s ) π ρ s d x + 2 ∫ ( ω s T ∇ ρ s ) 2 π 1 d x − 2 ∫ ( ω s T ∇ ρ s ) ( ω s T ∇ π ) π 2 ρ s d x ) − ( − ∫ ( ω s T ∇ ω s ∇ π ) π 2 ρ s 2 d x − ∫ ( ω s T H π ω s ) π 2 ρ s 2 d x − 2 ∫ ( ω s T ∇ ρ s ) ( ω s T ∇ π ) π 2 ρ s 2 d x + 2 ∫ ( ω s T ∇ π ) 2 π 3 ρ s 2 d x ) 2 ∫ ( ω s T H ρ s ω s ) ρ s π d x − 2 ∫ ( ω s T H π ω s ) ρ s 2 π 2 d x = − 4 ∫ ( ω s T ∇ ρ s ) ( ω s T ∇ π ) ρ s π 2 d x + 4 ∫ ρ s 2 π 3 ( ∇ π T ω s ) 2 d x ⏟ ( M ) + 2 ∫ ρ s 2 π ( ∇ ⋅ ω s ) 2 d x − 2 ∫ ( ω s T ∇ ρ s ) 2 1 π d x − 4 ∫ ( ω s T ∇ ω s ∇ ρ s ) ρ s π d x ⏟ ( N 1 ) + − 4 ∫ ( ω s T H ρ s ω s ) ρ s π d x + 4 ∫ ( ω s T ∇ ρ s ) ( ω s T ∇ π ) ρ s π 2 d x ⏟ ( N 2 ) . \begin{aligned}
&2\int\left(\omega_s^{\mathsf{T}}\mathbf{H}_{\rho_s}\omega_s\right)\frac{\rho_s}{\pi}\,dx
-2\int\left(\omega_s^{\mathsf{T}}\mathbf{H}_{\pi}\omega_s\right)\frac{\rho_s^2}{\pi^2}\,dx\\
&\qquad=\underbrace{-4\int\left(\omega_s^{\mathsf{T}}\nabla\rho_s\right)\left(\omega_s^{\mathsf{T}}\nabla\pi\right)\frac{\rho_s}{\pi^2}\,dx
+4\int\frac{\rho_s^2}{\pi^3}\left(\nabla_\pi^{\mathsf{T}}\omega_s\right)^2\,dx}_{(M)}\\
&\qquad+\underbrace{2\int\frac{\rho_s^2}{\pi}\left(\nabla\cdot\omega_s\right)^2\,dx
-2\int\left(\omega_s^{\mathsf{T}}\nabla\rho_s\right)^2\frac{1}{\pi}\,dx
-4\int\left(\omega_s^{\mathsf{T}}\nabla_{\omega_s}\nabla\rho_s\right)\frac{\rho_s}{\pi}\,dx}_{(N_1)}\\
&\qquad+\underbrace{-4\int\left(\omega_s^{\mathsf{T}}\mathbf{H}_{\rho_s}\omega_s\right)\frac{\rho_s}{\pi}\,dx
+4\int\left(\omega_s^{\mathsf{T}}\nabla\rho_s\right)\left(\omega_s^{\mathsf{T}}\nabla\pi\right)\frac{\rho_s}{\pi^2}\,dx}_{(N_2)}.
\end{aligned} 2 ∫ ( ω s T H ρ s ω s ) π ρ s d x − 2 ∫ ( ω s T H π ω s ) π 2 ρ s 2 d x = ( M ) − 4 ∫ ( ω s T ∇ ρ s ) ( ω s T ∇ π ) π 2 ρ s d x + 4 ∫ π 3 ρ s 2 ( ∇ π T ω s ) 2 d x + ( N 1 ) 2 ∫ π ρ s 2 ( ∇ ⋅ ω s ) 2 d x − 2 ∫ ( ω s T ∇ ρ s ) 2 π 1 d x − 4 ∫ ( ω s T ∇ ω s ∇ ρ s ) π ρ s d x + ( N 2 ) − 4 ∫ ( ω s T H ρ s ω s ) π ρ s d x + 4 ∫ ( ω s T ∇ ρ s ) ( ω s T ∇ π ) π 2 ρ s d x . Now we analyze ( M ) (M) ( M ) ( N 1 ) + ( N 2 ) (N_1)+(N_2) ( N 1 ) + ( N 2 )
( M ) = 2 ∫ ( ω s T H ( ρ s π ) ω s ) ρ s d x , (M)=2\int\left(\omega_s^{\mathsf{T}}\mathbf{H}\left(\frac{\rho_s}{\pi}\right)\omega_s\right)\rho_s\,dx, ( M ) = 2 ∫ ( ω s T H ( π ρ s ) ω s ) ρ s d x , and
( N 1 ) + ( N 2 ) = 2 ∫ ρ s 2 π ( ∇ ⋅ ω s ) 2 d x + 2 ∫ ( ω s T ∇ ρ s ) 2 1 π d x − 4 ∫ ( ω s T ∇ ω s ∇ ρ s ) ρ s π d x − 4 ∫ ( ω s T H ρ s ω s ) ρ s π d x − 4 ∫ ( ω s T ∇ ρ s ) 2 1 π d x + 4 ∫ ( ω s T ∇ ρ s ) ( ω s T ∇ π ) ρ s π 2 d x = 2 ∫ ρ s 2 π ( ∇ ⋅ ω s ) 2 d x + 2 ∫ ( ω s T ∇ ρ s ) 2 1 π d x − 4 ∫ ( ω s T ∇ ( ω s T ∇ ρ s ) ) ρ s π d x − 4 ∫ ( ω s T ∇ ρ s ) ( ω s T ∇ ρ s π ) d x = 2 ∫ ρ s 2 π ( ∇ ⋅ ω s ) 2 d x + 2 ∫ ( ω s T ∇ ρ s ) 2 1 π d x + 4 ∫ ∇ ⋅ ω s ( ω s T ∇ ρ s ) ρ s π d x = 2 ∫ ( ∇ ⋅ ( ω s ρ s ) 1 π ) 2 π d x . \begin{aligned}
(N_1)+(N_2)
={}&2\int\frac{\rho_s^2}{\pi}\left(\nabla\cdot\omega_s\right)^2dx
+2\int\left(\omega_s^{\mathsf{T}}\nabla\rho_s\right)^2\frac{1}{\pi}\,dx
-4\int\left(\omega_s^{\mathsf{T}}\nabla_{\omega_s}\nabla\rho_s\right)\frac{\rho_s}{\pi}\,dx\\
&-4\int\left(\omega_s^{\mathsf{T}}\mathbf{H}_{\rho_s}\omega_s\right)\frac{\rho_s}{\pi}\,dx
-4\int\left(\omega_s^{\mathsf{T}}\nabla\rho_s\right)^2\frac{1}{\pi}\,dx
+4\int\left(\omega_s^{\mathsf{T}}\nabla\rho_s\right)\left(\omega_s^{\mathsf{T}}\nabla\pi\right)\frac{\rho_s}{\pi^2}\,dx\\
={}&2\int\frac{\rho_s^2}{\pi}\left(\nabla\cdot\omega_s\right)^2dx
+2\int\left(\omega_s^{\mathsf{T}}\nabla\rho_s\right)^2\frac{1}{\pi}\,dx
-4\int\left(\omega_s^{\mathsf{T}}\nabla\left(\omega_s^{\mathsf{T}}\nabla\rho_s\right)\right)\frac{\rho_s}{\pi}\,dx\\
&-4\int\left(\omega_s^{\mathsf{T}}\nabla\rho_s\right)\left(\omega_s^{\mathsf{T}}\nabla\frac{\rho_s}{\pi}\right)\,dx\\
={}&2\int\frac{\rho_s^2}{\pi}\left(\nabla\cdot\omega_s\right)^2dx
+2\int\left(\omega_s^{\mathsf{T}}\nabla\rho_s\right)^2\frac{1}{\pi}\,dx
+4\int\nabla\cdot\omega_s\left(\omega_s^{\mathsf{T}}\nabla\rho_s\right)\frac{\rho_s}{\pi}\,dx\\
={}&2\int\left(\nabla\cdot\left(\omega_s\rho_s\right)\frac{1}{\pi}\right)^2\pi\,dx.
\end{aligned} ( N 1 ) + ( N 2 ) = = = = 2 ∫ π ρ s 2 ( ∇ ⋅ ω s ) 2 d x + 2 ∫ ( ω s T ∇ ρ s ) 2 π 1 d x − 4 ∫ ( ω s T ∇ ω s ∇ ρ s ) π ρ s d x − 4 ∫ ( ω s T H ρ s ω s ) π ρ s d x − 4 ∫ ( ω s T ∇ ρ s ) 2 π 1 d x + 4 ∫ ( ω s T ∇ ρ s ) ( ω s T ∇ π ) π 2 ρ s d x 2 ∫ π ρ s 2 ( ∇ ⋅ ω s ) 2 d x + 2 ∫ ( ω s T ∇ ρ s ) 2 π 1 d x − 4 ∫ ( ω s T ∇ ( ω s T ∇ ρ s ) ) π ρ s d x − 4 ∫ ( ω s T ∇ ρ s ) ( ω s T ∇ π ρ s ) d x 2 ∫ π ρ s 2 ( ∇ ⋅ ω s ) 2 d x + 2 ∫ ( ω s T ∇ ρ s ) 2 π 1 d x + 4 ∫ ∇ ⋅ ω s ( ω s T ∇ ρ s ) π ρ s d x 2 ∫ ( ∇ ⋅ ( ω s ρ s ) π 1 ) 2 π d x . Since d 2 d s 2 χ 2 ( ρ s ∥ π ) = ( M ) + ( N 1 ) + ( N 2 ) \frac{d^2}{ds^2}\chi^2(\rho_s\Vert\pi)=(M)+(N_1)+(N_2) d s 2 d 2 χ 2 ( ρ s ∥ π ) = ( M ) + ( N 1 ) + ( N 2 )
Appendix C
Supplementary Material for Chapter 5
Section C.1 provides proof for main convergence theorem of the nonparametric descent scheme; Section C.2 provides proof for main convergence theorem of the parametric descent scheme. Section C.3 provides a detailed discussion on the connection of our projected gradient descent (PGD) scheme with natural gradient descent. Section C.4 provides all the auxiliary results for the proof.
Proof of Theorem 5.2
From Theorem C.1 below, we have for any t ∈ N t \in\mathbb{N} t ∈ N
MMD ℓ 2 ( P t + 1 , Q ) − MMD ℓ 2 ( P t , Q ) ≤ − 2 γ t ( ℓ t ℓ − 1 ) d ∥ ∇ f ℓ , P t , Q ∥ L 2 d ( P t ) 2 + 4 γ t ( ℓ t ℓ − 1 ) d C d , s ℓ t 2 − ℓ 2 ℓ 3 + C k γ t 2 ℓ 4 + C k γ t 2 ( ℓ t 2 − ℓ 2 ) 2 ℓ 8 . \operatorname{MMD}_{\ell}^{2}(P_{t+1}, Q) - \operatorname{MMD}_{\ell}^{2}(P_t, Q) \leq-2\gamma_t \left(\ell_t \ell^{-1}\right)^d \left\lVert\nabla f_{\ell,P_t,Q} \right\rVert_{L_2^d(P_t)}^2
+ 4\gamma_t \left(\ell_t \ell^{-1}\right)^d \frac{C_{d,s}\sqrt{\ell_t^2-\ell^2}}{\ell^3} + \frac{C_k\gamma_t^2}{\ell^4} + C_k\gamma_t^2\frac{(\ell_t^2-\ell^2)^2}{\ell^8}. MMD ℓ 2 ( P t + 1 , Q ) − MMD ℓ 2 ( P t , Q ) ≤ − 2 γ t ( ℓ t ℓ − 1 ) d ∥ ∇ f ℓ , P t , Q ∥ L 2 d ( P t ) 2 + 4 γ t ( ℓ t ℓ − 1 ) d ℓ 3 C d , s ℓ t 2 − ℓ 2 + ℓ 4 C k γ t 2 + C k γ t 2 ℓ 8 ( ℓ t 2 − ℓ 2 ) 2 . Under the assumption that ( ℓ t ℓ − 1 ) d ∥ ∇ f ℓ , P t , Q ∥ L 2 d ( P t ) 2 ≥ C MMD ℓ 2 ( P t , Q ) \left(\ell_t\ell^{-1}\right)^d \left\lVert\nabla f_{\ell,P_t,Q} \right\rVert_{L_2^d(P_t)}^2 \geq C\operatorname{MMD}_{\ell}^{2}(P_t,Q) ( ℓ t ℓ − 1 ) d ∥ ∇ f ℓ , P t , Q ∥ L 2 d ( P t ) 2 ≥ C MMD ℓ 2 ( P t , Q )
MMD ℓ 2 ( P t + 1 , Q ) ≤ ( 1 − 2 C γ t ) MMD ℓ 2 ( P t , Q ) + 4 γ t ( ℓ t ℓ − 1 ) d C d , s ℓ t 2 − ℓ 2 ℓ 3 + C k γ t 2 ℓ 4 + C k γ t 2 ( ℓ t 2 − ℓ 2 ) 2 ℓ 8 . \operatorname{MMD}_{\ell}^{2}(P_{t+1}, Q) \leq(1-2C\gamma_t)\operatorname{MMD}_{\ell}^{2}(P_t,Q)
+ 4\gamma_t \left(\ell_t\ell^{-1}\right)^d \frac{C_{d,s}\sqrt{\ell_t^2-\ell^2}}{\ell^3} + \frac{C_k\gamma_t^2}{\ell^4} + C_k\gamma_t^2\frac{(\ell_t^2-\ell^2)^2}{\ell^8}. MMD ℓ 2 ( P t + 1 , Q ) ≤ ( 1 − 2 C γ t ) MMD ℓ 2 ( P t , Q ) + 4 γ t ( ℓ t ℓ − 1 ) d ℓ 3 C d , s ℓ t 2 − ℓ 2 + ℓ 4 C k γ t 2 + C k γ t 2 ℓ 8 ( ℓ t 2 − ℓ 2 ) 2 . By the discrete Gronwall’s lemma (Evans, 2022, Appendix B.2, Gronwall’s inequality), iterating the above relation from t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1
MMD ℓ 2 ( P T , Q ) ≤ exp ( − 2 C ∑ s = 0 T − 1 γ s ) MMD ℓ 2 ( P 0 , Q ) (134) \operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{T},\mathbb{Q}) \leq\exp\left(-2C\sum_{s=0}^{T-1}\gamma_s\right)\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{0},\mathbb{Q}) \tag*{(134)} MMD ℓ 2 ( P T , Q ) ≤ exp ( − 2 C s = 0 ∑ T − 1 γ s ) MMD ℓ 2 ( P 0 , Q ) (134) + ∑ t = 0 T − 1 exp ( − 2 C ∑ s = t + 1 T − 1 γ s ) × [ 4 γ t ( ℓ t ℓ − 1 ) d C d , s ℓ t 2 − ℓ 2 ℓ 3 + C k γ t 2 ℓ 4 + C k γ t 2 ( ℓ t 2 − ℓ 2 ) 2 ℓ 8 ] . (135) \begin{aligned}
&+\sum_{t=0}^{T-1}\exp\left(-2C\sum_{s=t+1}^{T-1}\gamma_s\right)\\
&\quad\times\left[4\gamma_t\left(\ell_t\ell^{-1}\right)^d\frac{C_{d,s}\sqrt{\ell_t^2-\ell^2}}{\ell^3}+\frac{C_k\gamma_t^2}{\ell^4}+C_k\gamma_t^2\frac{(\ell_t^2-\ell^2)^2}{\ell^8}\right]. \tag*{(135)}
\end{aligned} + t = 0 ∑ T − 1 exp ( − 2 C s = t + 1 ∑ T − 1 γ s ) × [ 4 γ t ( ℓ t ℓ − 1 ) d ℓ 3 C d , s ℓ t 2 − ℓ 2 + ℓ 4 C k γ t 2 + C k γ t 2 ℓ 8 ( ℓ t 2 − ℓ 2 ) 2 ] . (135) The first term in Eq. (C.1) converges to 0 as T → ∞ T\to\infty T → ∞ lim T → ∞ ∑ s = 0 T − 1 γ s → ∞ \lim_{T\to\infty}\sum_{s=0}^{T-1}\gamma_s\to\infty lim T → ∞ ∑ s = 0 T − 1 γ s → ∞ ϵ > 0 \epsilon>0 ϵ > 0 ℓ t 2 − ℓ 2 → 0 \ell_t^2-\ell^2\to0 ℓ t 2 − ℓ 2 → 0 γ t → 0 \gamma_t\to0 γ t → 0 ( ℓ t ℓ − 1 ) d → 1 (\ell_t\ell^{-1})^d\to1 ( ℓ t ℓ − 1 ) d → 1 t → ∞ t\to\infty t → ∞ t 0 ∈ N t_0\in\mathbb{N} t 0 ∈ N 4 ( ℓ t ℓ − 1 ) d C d , s ℓ t 2 − ℓ 2 ℓ 3 ≤ ϵ 3 4(\ell_t\ell^{-1})^d\frac{C_{d,s}\sqrt{\ell_t^2-\ell^2}}{\ell^3}\leq\frac{\epsilon}{3} 4 ( ℓ t ℓ − 1 ) d ℓ 3 C d , s ℓ t 2 − ℓ 2 ≤ 3 ϵ C k γ t ℓ 4 ≤ ϵ 3 \frac{C_k\gamma_t}{\ell^4}\leq\frac{\epsilon}{3} ℓ 4 C k γ t ≤ 3 ϵ C k γ t ( ℓ t 2 − ℓ 2 ) 2 ℓ 8 ≤ ϵ 3 \frac{C_k\gamma_t(\ell_t^2-\ell^2)^2}{\ell^8}\leq\frac{\epsilon}{3} ℓ 8 C k γ t ( ℓ t 2 − ℓ 2 ) 2 ≤ 3 ϵ t > t 0 t>t_0 t > t 0
( C .2 ) = ∑ t = 0 t 0 exp ( − 2 C ∑ s = t + 1 T − 1 γ s ) × [ 4 γ t ( ℓ t ℓ − 1 ) d C d , s ℓ t 2 − ℓ 2 ℓ 3 + C k γ t 2 ℓ 4 + C k γ t 2 ( ℓ t 2 − ℓ 2 ) 2 ℓ 8 ] + ∑ t = t 0 + 1 T − 1 exp ( − 2 C ∑ s = t + 1 T − 1 γ s ) × [ 4 γ t ( ℓ t ℓ − 1 ) d C d , s ℓ t 2 − ℓ 2 ℓ 3 + C k γ t 2 ℓ 4 + C k γ t 2 ( ℓ t 2 − ℓ 2 ) 2 ℓ 8 ] . \begin{aligned}
(\mathrm{C.2})={}&\sum_{t=0}^{t_0}\exp\left(-2C\sum_{s=t+1}^{T-1}\gamma_s\right)\\
&\quad\times\left[4\gamma_t\left(\ell_t\ell^{-1}\right)^d\frac{C_{d,s}\sqrt{\ell_t^2-\ell^2}}{\ell^3}+\frac{C_k\gamma_t^2}{\ell^4}+C_k\gamma_t^2\frac{(\ell_t^2-\ell^2)^2}{\ell^8}\right]\\
&+\sum_{t=t_0+1}^{T-1}\exp\left(-2C\sum_{s=t+1}^{T-1}\gamma_s\right)\\
&\quad\times\left[4\gamma_t\left(\ell_t\ell^{-1}\right)^d\frac{C_{d,s}\sqrt{\ell_t^2-\ell^2}}{\ell^3}+\frac{C_k\gamma_t^2}{\ell^4}+C_k\gamma_t^2\frac{(\ell_t^2-\ell^2)^2}{\ell^8}\right].
\end{aligned} ( C.2 ) = t = 0 ∑ t 0 exp ( − 2 C s = t + 1 ∑ T − 1 γ s ) × [ 4 γ t ( ℓ t ℓ − 1 ) d ℓ 3 C d , s ℓ t 2 − ℓ 2 + ℓ 4 C k γ t 2 + C k γ t 2 ℓ 8 ( ℓ t 2 − ℓ 2 ) 2 ] + t = t 0 + 1 ∑ T − 1 exp ( − 2 C s = t + 1 ∑ T − 1 γ s ) × [ 4 γ t ( ℓ t ℓ − 1 ) d ℓ 3 C d , s ℓ t 2 − ℓ 2 + ℓ 4 C k γ t 2 + C k γ t 2 ℓ 8 ( ℓ t 2 − ℓ 2 ) 2 ] . The first term is a finite sum which converges to 0 as T → ∞ T\to\infty T → ∞ exp ( − 2 C ∑ s = t + 1 T − 1 γ s ) → 0 \exp\left(-2C\sum_{s=t+1}^{T-1}\gamma_s\right)\to0 exp ( − 2 C ∑ s = t + 1 T − 1 γ s ) → 0 t 0 t_0 t 0 ϵ ∑ t = t 0 + 1 T − 1 exp ( − 2 C ∑ s = t + 1 T − 1 γ s ) γ t \epsilon\sum_{t=t_0+1}^{T-1}\exp\left(-2C\sum_{s=t+1}^{T-1}\gamma_s\right)\gamma_t ϵ ∑ t = t 0 + 1 T − 1 exp ( − 2 C ∑ s = t + 1 T − 1 γ s ) γ t w t , T : = exp ( − 2 C ∑ s = t + 1 T − 1 γ s ) w_{t,T}:=\exp\left(-2C\sum_{s=t+1}^{T-1}\gamma_s\right) w t , T := exp ( − 2 C ∑ s = t + 1 T − 1 γ s ) γ t ≤ 1 C ( 1 − e − 2 C γ t ) \gamma_t\leq\frac{1}{C}(1-e^{-2C\gamma_t}) γ t ≤ C 1 ( 1 − e − 2 C γ t ) 0 < γ t ≤ 1 2 C 0<\gamma_t\leq\frac{1}{2C} 0 < γ t ≤ 2 C 1
ϵ ∑ t = t 0 + 1 T − 1 w t , T γ t ≤ ϵ C ∑ t = t 0 + 1 T − 1 w t , T × ( 1 − e − 2 C γ t ) = ϵ C ∑ t = t 0 + 1 T − 1 [ exp ( − 2 C ∑ s = t + 1 T − 1 γ s ) − exp ( − 2 C ∑ s = t T − 1 γ s ) ] = ϵ C (136) \begin{aligned}
\epsilon\sum_{t=t_0+1}^{T-1}w_{t,T}\gamma_t\leq\frac{\epsilon}{C}\sum_{t=t_0+1}^{T-1}w_{t,T} \\
&\times\left(1-e^{-2C\gamma_t}\right) \\
&= \frac{\epsilon}{C}\sum_{t=t_0+1}^{T-1}\left[\exp\left(-2C\sum_{s=t+1}^{T-1}\gamma_s\right)-\exp\left(-2C\sum_{s=t}^{T-1}\gamma_s\right)\right] \\
&= \frac{\epsilon}{C} \tag*{(136)}
\end{aligned} ϵ t = t 0 + 1 ∑ T − 1 w t , T γ t ≤ C ϵ t = t 0 + 1 ∑ T − 1 w t , T × ( 1 − e − 2 C γ t ) = C ϵ t = t 0 + 1 ∑ T − 1 [ exp ( − 2 C s = t + 1 ∑ T − 1 γ s ) − exp ( − 2 C s = t ∑ T − 1 γ s ) ] = C ϵ (136) × ( 1 − exp ( − 2 C ∑ s = t 0 + 1 T − 1 γ s ) ) ≤ ϵ C . (137) \times\left(1-\exp\left(-2C\sum_{s=t_0+1}^{T-1}\gamma_s\right)\right)\leq\frac{\epsilon}{C}. \tag*{(137)} × ( 1 − exp ( − 2 C s = t 0 + 1 ∑ T − 1 γ s ) ) ≤ C ϵ . (137) The sum is bounded. Since C C C t t t ϵ \epsilon ϵ
lim T → ∞ MMD ℓ 2 ( P T , Q ) = 0. \lim_{T\to\infty}\operatorname{MMD}_{\ell}^{2}(P_T,Q)=0. T → ∞ lim MMD ℓ 2 ( P T , Q ) = 0. Proposition C.1 (Descent Lemma). Let kernel k k k t ∈ N t\in\mathbb{N} t ∈ N Q ∈ P 2 ( R d ) Q\in\mathcal{P}_2(\mathbb{R}^d) Q ∈ P 2 ( R d ) P t + 1 = ( Id − γ ∇ f ℓ t , P t , Q ) # P t P_{t+1}=(\operatorname{Id}-\gamma\nabla f_{\ell_t,P_t,Q})_{\#}P_t P t + 1 = ( Id − γ ∇ f ℓ t , P t , Q ) # P t (34) with lengthscale ℓ t > 0 \ell_t>0 ℓ t > 0 γ > 0 \gamma>0 γ > 0 0 < ℓ < ℓ t 0<\ell<\ell_t 0 < ℓ < ℓ t
MMD ℓ 2 ( P t + 1 , Q ) − MMD ℓ 2 ( P t , Q ) ≤ − 2 γ ( ℓ t ℓ − 1 ) d ∥ ∇ f ℓ , P t , Q ∥ L d 2 ( P t ) 2 + 4 γ ( ℓ t ℓ − 1 ) d C d , s ℓ t 2 − ℓ 2 ℓ 3 + C k γ 2 ℓ 4 + C k γ 2 ( ℓ t 2 − ℓ 2 ) 2 ℓ 8 . \operatorname{MMD}_{\ell}^{2}(P_{t+1},Q)-\operatorname{MMD}_{\ell}^{2}(P_t,Q)
\leq-2\gamma\left(\ell_t\ell^{-1}\right)^d\left\|\nabla f_{\ell,P_t,Q}\right\|_{L^2_d(P_t)}^2
+4\gamma\left(\ell_t\ell^{-1}\right)^d\frac{C_{d,s}\sqrt{\ell_t^2-\ell^2}}{\ell^3}
+\frac{C_k\gamma^2}{\ell^4}
+C_k\gamma^2\frac{(\ell_t^2-\ell^2)^2}{\ell^8}. MMD ℓ 2 ( P t + 1 , Q ) − MMD ℓ 2 ( P t , Q ) ≤ − 2 γ ( ℓ t ℓ − 1 ) d ∥ ∇ f ℓ , P t , Q ∥ L d 2 ( P t ) 2 + 4 γ ( ℓ t ℓ − 1 ) d ℓ 3 C d , s ℓ t 2 − ℓ 2 + ℓ 4 C k γ 2 + C k γ 2 ℓ 8 ( ℓ t 2 − ℓ 2 ) 2 . Here, C k C_k C k k k k ℓ \ell ℓ C d , s C_{d,s} C d , s d d d s s s
Proof. In the proof of this proposition, since the time index is always fixed at t t t f ℓ : = f ℓ , P t , Q f_\ell:=f_{\ell,P_t,Q} f ℓ := f ℓ , P t , Q
MMD ℓ 2 ( P t + 1 , Q ) − MMD ℓ 2 ( P t , Q ) ≤ − 2 γ ∫ ∇ f ℓ ( x ) ⊤ ∇ f ℓ t ( x ) d P t ( x ) + C k γ 2 ℓ 2 ∥ ∇ f ℓ t ∥ L d 2 ( P t ) 2 . (138) \operatorname{MMD}_{\ell}^{2}(P_{t+1},Q)-\operatorname{MMD}_{\ell}^{2}(P_t,Q)
\leq-2\gamma\int\nabla f_{\ell}(x)^\top\nabla f_{\ell_t}(x)\,\mathrm{d}P_t(x)
+\frac{C_k\gamma^2}{\ell^2}\left\|\nabla f_{\ell_t}\right\|_{L^2_d(P_t)}^2. \tag*{(138)} MMD ℓ 2 ( P t + 1 , Q ) − MMD ℓ 2 ( P t , Q ) ≤ − 2 γ ∫ ∇ f ℓ ( x ) ⊤ ∇ f ℓ t ( x ) d P t ( x ) + ℓ 2 C k γ 2 ∥ ∇ f ℓ t ∥ L d 2 ( P t ) 2 . (138) The first term on the right hand side of Eq. (C.5) can be further upper bounded by the following:
− ∫ ∇ f ℓ ( x ) T ∇ f ℓ ( x ) d P t ( x ) = − ( ℓ t ℓ − 1 ) d ∫ ∇ f ℓ ( x ) T E U ∼ ν [ ∇ f ℓ ( x + U ) ] d P t ( x ) = − ( ℓ t ℓ − 1 ) d [ ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 + ∫ ∇ f ℓ ( x ) T { E U ∼ ν [ ∇ f ℓ ( x + U ) ] − ∇ f ℓ ( x ) } d P t ( x ) ] . -\int\nabla f_{\ell}(x)^{\mathsf{T}}\nabla f_{\ell}(x)\,\mathrm{d}P_t(x)
=-(\ell_t\ell^{-1})^d\int\nabla f_{\ell}(x)^{\mathsf{T}}\mathbb{E}_{U\sim\nu}\left[\nabla f_{\ell}(x+U)\right]\,\mathrm{d}P_t(x)
=-(\ell_t\ell^{-1})^d\left[\left\|\nabla f_{\ell}\right\|_{L_2^d(P_t)}^2+\int\nabla f_{\ell}(x)^{\mathsf{T}}\left\{\mathbb{E}_{U\sim\nu}\left[\nabla f_{\ell}(x+U)\right]-\nabla f_{\ell}(x)\right\}\,\mathrm{d}P_t(x)\right]. − ∫ ∇ f ℓ ( x ) T ∇ f ℓ ( x ) d P t ( x ) = − ( ℓ t ℓ − 1 ) d ∫ ∇ f ℓ ( x ) T E U ∼ ν [ ∇ f ℓ ( x + U ) ] d P t ( x ) = − ( ℓ t ℓ − 1 ) d [ ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 + ∫ ∇ f ℓ ( x ) T { E U ∼ ν [ ∇ f ℓ ( x + U ) ] − ∇ f ℓ ( x ) } d P t ( x ) ] . The second last line holds by Theorem C.13, where ν \nu ν R d \mathbb{R}^d R d E U ∼ ν [ ∥ U ∥ ] ≤ C d , s ℓ t 2 − ℓ 2 \mathbb{E}_{U\sim\nu}[\|U\|]\le C_{d,s}\sqrt{\ell_t^2-\ell^2} E U ∼ ν [ ∥ U ∥ ] ≤ C d , s ℓ t 2 − ℓ 2 C d , s C_{d,s} C d , s d d d s s s
≤ − ( ℓ t ℓ − 1 ) d [ ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 − ∫ ∥ ∇ f ℓ ( x ) ∥ ∥ E U ∼ ν [ ∇ f ℓ ( x + U ) ] − ∇ f ℓ ( x ) ∥ d P t ( x ) ] ≤ − ( ℓ t ℓ − 1 ) d [ ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 − ∫ ∥ ∇ f ℓ ( x ) ∥ E U ∼ ν ∥ ∇ f ℓ ( x + U ) − ∇ f ℓ ( x ) ∥ d P t ( x ) ] ≤ − ( ℓ t ℓ − 1 ) d [ ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 − 2 C k ℓ 2 ∥ ∇ f ℓ ∥ L 2 d ( P t ) E U ∼ ν [ ∥ U ∥ ] ] = − ( ℓ t ℓ − 1 ) d [ ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 − ∥ ∇ f ℓ ∥ L 2 d ( P t ) C d , s C k ℓ t 2 − ℓ 2 ℓ 2 ] . \begin{aligned}
&\leq-(\ell_t\ell^{-1})^d\left[\left\|\nabla f_{\ell}\right\|_{L_2^d(P_t)}^2-\int\left\|\nabla f_{\ell}(x)\right\|\left\|\mathbb{E}_{U\sim\nu}\left[\nabla f_{\ell}(x+U)\right]-\nabla f_{\ell}(x)\right\|\,\mathrm{d}P_t(x)\right]\\
&\leq-(\ell_t\ell^{-1})^d\left[\left\|\nabla f_{\ell}\right\|_{L_2^d(P_t)}^2-\int\left\|\nabla f_{\ell}(x)\right\|\mathbb{E}_{U\sim\nu}\left\|\nabla f_{\ell}(x+U)-\nabla f_{\ell}(x)\right\|\,\mathrm{d}P_t(x)\right]\\
&\leq-(\ell_t\ell^{-1})^d\left[\left\|\nabla f_{\ell}\right\|_{L_2^d(P_t)}^2-\frac{2C_k}{\ell^2}\left\|\nabla f_{\ell}\right\|_{L_2^d(P_t)}\mathbb{E}_{U\sim\nu}\left[\|U\|\right]\right]\\
&=-(\ell_t\ell^{-1})^d\left[\left\|\nabla f_{\ell}\right\|_{L_2^d(P_t)}^2-\left\|\nabla f_{\ell}\right\|_{L_2^d(P_t)}\frac{C_{d,s}C_k\sqrt{\ell_t^2-\ell^2}}{\ell^2}\right].
\end{aligned} ≤ − ( ℓ t ℓ − 1 ) d [ ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 − ∫ ∥ ∇ f ℓ ( x ) ∥ ∥ E U ∼ ν [ ∇ f ℓ ( x + U ) ] − ∇ f ℓ ( x ) ∥ d P t ( x ) ] ≤ − ( ℓ t ℓ − 1 ) d [ ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 − ∫ ∥ ∇ f ℓ ( x ) ∥ E U ∼ ν ∥ ∇ f ℓ ( x + U ) − ∇ f ℓ ( x ) ∥ d P t ( x ) ] ≤ − ( ℓ t ℓ − 1 ) d [ ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 − ℓ 2 2 C k ∥ ∇ f ℓ ∥ L 2 d ( P t ) E U ∼ ν [ ∥ U ∥ ] ] = − ( ℓ t ℓ − 1 ) d [ ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 − ∥ ∇ f ℓ ∥ L 2 d ( P t ) ℓ 2 C d , s C k ℓ t 2 − ℓ 2 ] . The second last inequality holds by the fact that x ↦ ∇ f ℓ ( x ) x\mapsto\nabla f_{\ell}(x) x ↦ ∇ f ℓ ( x ) ( 2 C k / ℓ 2 ) (2C_k/\ell^2) ( 2 C k / ℓ 2 ) C k C_k C k U ∼ ν U\sim\nu U ∼ ν
∥ ∇ f ℓ t ∥ L 2 d ( P t ) 2 ≤ 2 ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 + 2 ∥ ∇ f ℓ t − ∇ f ℓ ∥ L 2 d ( P t ) 2 \left\|\nabla f_{\ell_t}\right\|_{L_2^d(P_t)}^2\leq2\left\|\nabla f_{\ell}\right\|_{L_2^d(P_t)}^2+2\left\|\nabla f_{\ell_t}-\nabla f_{\ell}\right\|_{L_2^d(P_t)}^2 ∥ ∇ f ℓ t ∥ L 2 d ( P t ) 2 ≤ 2 ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 + 2 ∥ ∇ f ℓ t − ∇ f ℓ ∥ L 2 d ( P t ) 2 ≤ 2 ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 + C k ( ℓ t 2 − ℓ 2 ℓ 3 ) 2 . \le2\lVert\nabla f_{\ell}\rVert_{L_{2}^{d}(P_t)}^{2}+C_k\left(\frac{\ell_t^2-\ell^2}{\ell^3}\right)^2. ≤ 2 ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 + C k ( ℓ 3 ℓ t 2 − ℓ 2 ) 2 . Here, C k C_k C k k k k ℓ \ell ℓ
MMD ℓ 2 ( P ^ t + 1 , Q ) − MMD ℓ 2 ( P t , Q ) ≤ − 2 γ ( ℓ t ℓ − 1 ) d [ ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 − ∥ ∇ f ℓ ∥ L 2 d ( P t ) C d , s C k ℓ t 2 − ℓ 2 ℓ 2 + C k γ 2 ℓ 2 [ 2 ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 + C k ( ℓ t 2 − ℓ 2 ℓ 3 ) 2 ] ] . \begin{aligned}
\operatorname{MMD}_{\ell}^{2}(\widehat{P}_{t+1},\mathbb{Q})-\operatorname{MMD}_{\ell}^{2}(P_t,\mathbb{Q})
&\le-2\gamma\left(\ell_t\ell^{-1}\right)^d\left[\lVert\nabla f_{\ell}\rVert_{L_{2}^{d}(P_t)}^2-\lVert\nabla f_{\ell}\rVert_{L_{2}^{d}(P_t)}\frac{C_{d,s}C_k\sqrt{\ell_t^2-\ell^2}}{\ell^2}\right.\\
&\qquad\left.+\frac{C_k\gamma^2}{\ell^2}\left[2\lVert\nabla f_{\ell}\rVert_{L_{2}^{d}(P_t)}^2+C_k\left(\frac{\ell_t^2-\ell^2}{\ell^3}\right)^2\right]\right].
\end{aligned} MMD ℓ 2 ( P t + 1 , Q ) − MMD ℓ 2 ( P t , Q ) ≤ − 2 γ ( ℓ t ℓ − 1 ) d [ ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 − ∥ ∇ f ℓ ∥ L 2 d ( P t ) ℓ 2 C d , s C k ℓ t 2 − ℓ 2 + ℓ 2 C k γ 2 [ 2 ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 + C k ( ℓ 3 ℓ t 2 − ℓ 2 ) 2 ] ] . Expanding, this gives,
MMD ℓ 2 ( P ^ t + 1 , Q ) − MMD ℓ 2 ( P t , Q ) ≤ − 2 γ ( ℓ t ℓ − 1 ) d ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 + 2 γ ( ℓ t ℓ − 1 ) d ∥ ∇ f ℓ ∥ L 2 d ( P t ) C d , s C k ℓ t 2 − ℓ 2 ℓ 2 + C k γ 2 ℓ 2 ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 + C k γ 2 ( ℓ t 2 − ℓ 2 ) 2 ℓ 8 ≤ − 2 γ ( ℓ t ℓ − 1 ) d ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 + 4 γ ( ℓ t ℓ − 1 ) d C d , s C k ℓ t 2 − ℓ 2 ℓ 3 + C k γ 2 ℓ 4 + C k γ 2 ( ℓ t 2 − ℓ 2 ) 2 ℓ 8 , \begin{aligned}
\operatorname{MMD}_{\ell}^{2}(\widehat{P}_{t+1},\mathbb{Q})-\operatorname{MMD}_{\ell}^{2}(P_t,\mathbb{Q})
&\le-2\gamma\left(\ell_t\ell^{-1}\right)^d\lVert\nabla f_{\ell}\rVert_{L_{2}^{d}(P_t)}^2\\
&\quad+2\gamma\left(\ell_t\ell^{-1}\right)^d\lVert\nabla f_{\ell}\rVert_{L_{2}^{d}(P_t)}\frac{C_{d,s}C_k\sqrt{\ell_t^2-\ell^2}}{\ell^2}\\
&\quad+\frac{C_k\gamma^2}{\ell^2}\lVert\nabla f_{\ell}\rVert_{L_{2}^{d}(P_t)}^2+C_k\gamma^2\frac{(\ell_t^2-\ell^2)^2}{\ell^8}\\
&\le-2\gamma\left(\ell_t\ell^{-1}\right)^d\lVert\nabla f_{\ell}\rVert_{L_{2}^{d}(P_t)}^2\\
&\quad+4\gamma\left(\ell_t\ell^{-1}\right)^d\frac{C_{d,s}C_k\sqrt{\ell_t^2-\ell^2}}{\ell^3}+\frac{C_k\gamma^2}{\ell^4}+C_k\gamma^2\frac{(\ell_t^2-\ell^2)^2}{\ell^8},
\end{aligned} MMD ℓ 2 ( P t + 1 , Q ) − MMD ℓ 2 ( P t , Q ) ≤ − 2 γ ( ℓ t ℓ − 1 ) d ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 + 2 γ ( ℓ t ℓ − 1 ) d ∥ ∇ f ℓ ∥ L 2 d ( P t ) ℓ 2 C d , s C k ℓ t 2 − ℓ 2 + ℓ 2 C k γ 2 ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 + C k γ 2 ℓ 8 ( ℓ t 2 − ℓ 2 ) 2 ≤ − 2 γ ( ℓ t ℓ − 1 ) d ∥ ∇ f ℓ ∥ L 2 d ( P t ) 2 + 4 γ ( ℓ t ℓ − 1 ) d ℓ 3 C d , s C k ℓ t 2 − ℓ 2 + ℓ 4 C k γ 2 + C k γ 2 ℓ 8 ( ℓ t 2 − ℓ 2 ) 2 , where the last step holds by ∥ ∇ f ℓ ∥ L 2 d ( P t ) ≤ C k ℓ − 1 \lVert\nabla f_{\ell}\rVert_{L_{2}^{d}(P_t)}\le C_k\ell^{-1} ∥ ∇ f ℓ ∥ L 2 d ( P t ) ≤ C k ℓ − 1 C k C_k C k
Proof of the parametric convergence theorem
Proof. In the proof, we drop the superscript PGD for notational simplicity. Let P ^ t + 1 = ( Id − γ t ∇ f ℓ t , P θ t , m , Q ) # P θ t , m \widehat{P}_{t+1}=(\operatorname{Id}-\gamma_t\nabla f_{\ell_t,P_{\theta_t,m},\mathbb{Q}})\#P_{\theta_t,m} P t + 1 = ( Id − γ t ∇ f ℓ t , P θ t , m , Q ) # P θ t , m P θ t + 1 = P θ t − γ t Δ θ P_{\theta_{t+1}}=P_{\theta_t-\gamma_t\Delta_{\theta}} P θ t + 1 = P θ t − γ t Δ θ (38) . For comparison purposes, let P θ t + 1 , m = P θ t − γ t Δ θ P G D P_{\theta_{t+1,m}}=P_{\theta_t-\gamma_t\Delta_{\theta}^{\mathrm{PGD}}} P θ t + 1 , m = P θ t − γ t Δ θ PGD (40) . P ^ t + 1 \widehat{P}_{t+1} P t + 1 P θ t + 1 P_{\theta_{t+1}} P θ t + 1 P θ t + 1 , m P_{\theta_{t+1,m}} P θ t + 1 , m
MMD ℓ 2 ( P ^ t + 1 , Q ) − MMD ℓ 2 ( P θ t , m , Q ) ≤ − 2 γ t ( ℓ t ℓ − 1 ) d ∥ ∇ f ℓ , P θ t , m , Q ∥ L 2 d ( P θ t , m ) 2 + 4 γ t ( ℓ t ℓ − 1 ) d C d , s ℓ t 2 − ℓ 2 ℓ 3 + C k γ t 2 ℓ 4 + C k γ t 2 ( ℓ t 2 − ℓ 2 ) 2 ℓ 8 . \operatorname{MMD}_{\ell}^{2}(\widehat{\mathbb{P}}_{t+1},\mathbb{Q})-\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta_{t,m}},\mathbb{Q}) \leq-2\gamma_t\left(\ell_t\ell^{-1}\right)^d \left\|\nabla f_{\ell,\mathbb{P}_{\theta_{t,m}},\mathbb{Q}}\right\|_{L_2^d(\mathbb{P}_{\theta_{t,m}})}^2
+4\gamma_t\left(\ell_t\ell^{-1}\right)^d\frac{C_{d,s}\sqrt{\ell_t^2-\ell^2}}{\ell^3}
+\frac{C_k\gamma_t^2}{\ell^4}
+C_k\gamma_t^2\frac{\left(\ell_t^2-\ell^2\right)^2}{\ell^8}. MMD ℓ 2 ( P t + 1 , Q ) − MMD ℓ 2 ( P θ t , m , Q ) ≤ − 2 γ t ( ℓ t ℓ − 1 ) d ∇ f ℓ , P θ t , m , Q L 2 d ( P θ t , m ) 2 + 4 γ t ( ℓ t ℓ − 1 ) d ℓ 3 C d , s ℓ t 2 − ℓ 2 + ℓ 4 C k γ t 2 + C k γ t 2 ℓ 8 ( ℓ t 2 − ℓ 2 ) 2 . Then, from Theorem 5.7, or more precisely from Theorem C.2, we have
MMD ℓ 2 ( P θ t + 1 , Q ) − MMD ℓ 2 ( P ^ t + 1 , Q ) ≤ 2 γ t R ∥ ∇ f ℓ , P θ t , m , Q ∥ L 2 d ( P θ t , m ) 2 + C k γ t R ℓ t 2 − ℓ 2 ℓ 4 + C k d 1 2 p 2 5 γ t 2 5 U 4 5 ℓ − 1 ( ( 1 + ℓ − 1 ) 1 5 ℓ t − 2 5 + C k γ t 2 ℓ − 2 ℓ t − 2 ) . \operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta_{t+1}},\mathbb{Q})-\operatorname{MMD}_{\ell}^{2}(\widehat{\mathbb{P}}_{t+1},\mathbb{Q})
\leq2\gamma_t\mathfrak{R}\left\|\nabla f_{\ell,\mathbb{P}_{\theta_{t,m}},\mathbb{Q}}\right\|_{L_2^d(\mathbb{P}_{\theta_{t,m}})}^2
+C_k\gamma_t\mathfrak{R}\frac{\ell_t^2-\ell^2}{\ell^4}
+C_kd^{\frac{1}{2}}p^{\frac{2}{5}}\gamma_t^{\frac{2}{5}}U^{\frac{4}{5}}\ell^{-1}\left(\left(1+\ell^{-1}\right)^{\frac{1}{5}}\ell_t^{-\frac{2}{5}}+C_k\gamma_t^2\ell^{-2}\ell_t^{-2}\right). MMD ℓ 2 ( P θ t + 1 , Q ) − MMD ℓ 2 ( P t + 1 , Q ) ≤ 2 γ t R ∇ f ℓ , P θ t , m , Q L 2 d ( P θ t , m ) 2 + C k γ t R ℓ 4 ℓ t 2 − ℓ 2 + C k d 2 1 p 5 2 γ t 5 2 U 5 4 ℓ − 1 ( ( 1 + ℓ − 1 ) 5 1 ℓ t − 5 2 + C k γ t 2 ℓ − 2 ℓ t − 2 ) . Finally, applying Theorem C.5 with evaluation lengthscale ℓ \ell ℓ ℓ t \ell_t ℓ t Q ^ m \widehat{\mathbb{Q}}_m Q m
E m [ MMD ℓ 2 ( P θ t + 1 , m , Q ) − MMD ℓ 2 ( P θ t + 1 , Q ) ] ≤ C k γ t ℓ t − 1 p ℓ − 1 ℓ t − 1 d ⋅ 1 m . \mathbb{E}_m\left[\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta_{t+1,m}},\mathbb{Q})-\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta_{t+1}},\mathbb{Q})\right]\leq C_k\gamma_t\ell_t^{-1}p\ell^{-1}\ell_t^{-1}\sqrt{d}\cdot\frac{1}{\sqrt{m}}. E m [ MMD ℓ 2 ( P θ t + 1 , m , Q ) − MMD ℓ 2 ( P θ t + 1 , Q ) ] ≤ C k γ t ℓ t − 1 p ℓ − 1 ℓ t − 1 d ⋅ m 1 . Adding the above three upper bounds and taking expectation over Q ^ m \widehat{\mathbb{Q}}_m Q m
E m [ MMD ℓ 2 ( P θ t + 1 , m , Q ) − MMD ℓ 2 ( P θ t , m , Q ) ] ≤ − 2 ( 1 − R ) γ t E m [ ( ℓ t ℓ − 1 ) d ∥ ∇ f ℓ , P θ t , m , Q ∥ L 2 d ( P θ t , m ) 2 ] + C k γ t R ℓ t 2 − ℓ 2 ℓ 4 + C k d 1 2 p 2 5 γ t 2 5 U 4 5 × ℓ − 1 ( 1 + ℓ − 1 ) 1 5 ℓ t − 2 5 + C k γ t 2 ℓ − 2 ℓ t − 2 + 4 γ t ( ℓ t ℓ − 1 ) d C d , s ℓ t 2 − ℓ 2 ℓ 3 + C k γ t 2 ℓ 4 + C k γ t 2 ( ℓ t 2 − ℓ 2 ) 2 ℓ 8 + C k γ t 3 5 p 1 5 ℓ − 1 ( 1 + ℓ − 1 ) − 2 5 ℓ t − 1 5 U 4 5 ⋅ 1 m . (139) \begin{aligned}
\mathbb{E}_m\left[\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta_{t+1,m}},\mathbb{Q})-\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta_{t,m}},\mathbb{Q})\right]
&\leq-2(1-\mathfrak{R})\gamma_t\mathbb{E}_m\left[\left(\ell_t\ell^{-1}\right)^d\left\|\nabla f_{\ell,\mathbb{P}_{\theta_{t,m}},\mathbb{Q}}\right\|_{L_2^d(\mathbb{P}_{\theta_{t,m}})}^2\right] \\
&\quad+C_k\gamma_t\mathfrak{R}\frac{\ell_t^2-\ell^2}{\ell^4}+C_kd^{\frac{1}{2}}p^{\frac{2}{5}}\gamma_t^{\frac{2}{5}}U^{\frac{4}{5}} \\
&\quad\times\ell^{-1}\left(1+\ell^{-1}\right)^{\frac{1}{5}}\ell_t^{-\frac{2}{5}}+C_k\gamma_t^2\ell^{-2}\ell_t^{-2} \\
&\quad+4\gamma_t\left(\ell_t\ell^{-1}\right)^d\frac{C_{d,s}\sqrt{\ell_t^2-\ell^2}}{\ell^3}+\frac{C_k\gamma_t^2}{\ell^4} \\
&\quad+C_k\gamma_t^2\frac{\left(\ell_t^2-\ell^2\right)^2}{\ell^8} \\
&\quad+C_k\gamma_t^{\frac{3}{5}}p^{\frac{1}{5}}\ell^{-1}\left(1+\ell^{-1}\right)^{-\frac{2}{5}}\ell_t^{-\frac{1}{5}}U^{\frac{4}{5}}\cdot\frac{1}{\sqrt{m}}. \tag*{(139)}
\end{aligned} E m [ MMD ℓ 2 ( P θ t + 1 , m , Q ) − MMD ℓ 2 ( P θ t , m , Q ) ] ≤ − 2 ( 1 − R ) γ t E m [ ( ℓ t ℓ − 1 ) d ∇ f ℓ , P θ t , m , Q L 2 d ( P θ t , m ) 2 ] + C k γ t R ℓ 4 ℓ t 2 − ℓ 2 + C k d 2 1 p 5 2 γ t 5 2 U 5 4 × ℓ − 1 ( 1 + ℓ − 1 ) 5 1 ℓ t − 5 2 + C k γ t 2 ℓ − 2 ℓ t − 2 + 4 γ t ( ℓ t ℓ − 1 ) d ℓ 3 C d , s ℓ t 2 − ℓ 2 + ℓ 4 C k γ t 2 + C k γ t 2 ℓ 8 ( ℓ t 2 − ℓ 2 ) 2 + C k γ t 5 3 p 5 1 ℓ − 1 ( 1 + ℓ − 1 ) − 5 2 ℓ t − 5 1 U 5 4 ⋅ m 1 . (139) Denote the sum of Eq. (C.7) and Eq. (C.8) as M t \mathfrak{M}_t M t t t t m m m γ t 3 5 m − 1 / 2 \gamma_t^{\frac{3}{5}}m^{-1/2} γ t 5 3 m − 1/2 C k p 1 5 ℓ − 1 ( ( 1 + ℓ − 1 ) ) − 2 5 ℓ − 1 5 U 2 5 = : N C_k p^{\frac{1}{5}}\ell^{-1}((1+\ell^{-1}))^{-\frac{2}{5}}\ell^{-\frac{1}{5}}\mathcal{U}^{\frac{2}{5}}=: \mathfrak{N} C k p 5 1 ℓ − 1 (( 1 + ℓ − 1 ) ) − 5 2 ℓ − 5 1 U 5 2 =: N t t t m m m
E m [ MMD ℓ 2 ( P θ t + 1 , m , Q ) ] ≤ ( 1 − 2 C ( 1 − R ) γ t ) E m [ MMD ℓ 2 ( P θ t , m , Q ) ] + M t + N ⋅ γ t 3 5 1 m . \mathbb{E}_m\left[\operatorname{MMD}_{\ell}^{2}(P_{\theta_{t+1,m}},Q)\right]\leq\left(1-2C(1-\mathfrak{R})\gamma_t\right)\mathbb{E}_m\left[\operatorname{MMD}_{\ell}^{2}(P_{\theta_{t,m}},Q)\right]+\mathfrak{M}_t+\mathfrak{N}\cdot\gamma_t^{\frac{3}{5}}\frac{1}{\sqrt{m}}. E m [ MMD ℓ 2 ( P θ t + 1 , m , Q ) ] ≤ ( 1 − 2 C ( 1 − R ) γ t ) E m [ MMD ℓ 2 ( P θ t , m , Q ) ] + M t + N ⋅ γ t 5 3 m 1 . The following strategy will be similar to the proof in the nonparametric case in Section C.1. Fix the optimization horizon T ∈ N T\in\mathbb{N} T ∈ N t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1
E m [ MMD ℓ 2 ( P θ T , m , Q ) ] ≤ exp ( − 2 C ( 1 − R ) ∑ s = 0 T − 1 γ s ) MMD ℓ 2 ( P θ 0 , Q ) + ∑ t = 0 T − 1 exp ( − 2 C ( 1 − R ) ∑ s = t + 1 T − 1 γ s ) M t + ∑ t = 0 T − 1 exp ( − 2 C ( 1 − R ) ∑ s = t + 1 T − 1 γ s ) × N γ t 3 5 1 m . (140) \begin{aligned}
\mathbb{E}_m\left[\operatorname{MMD}_{\ell}^{2}(P_{\theta_{T,m}},Q)\right]
&\leq\exp\left(-2C(1-\mathfrak{R})\sum_{s=0}^{T-1}\gamma_s\right)\operatorname{MMD}_{\ell}^{2}(P_{\theta_0},Q) \\
&\quad+\sum_{t=0}^{T-1}\exp\left(-2C(1-\mathfrak{R})\sum_{s=t+1}^{T-1}\gamma_s\right)\mathfrak{M}_t \\
&\quad+\sum_{t=0}^{T-1}\exp\left(-2C(1-\mathfrak{R})\sum_{s=t+1}^{T-1}\gamma_s\right) \\
&\qquad\qquad\times\mathfrak{N}\gamma_t^{\frac{3}{5}}\frac{1}{\sqrt{m}}. \tag*{(140)}
\end{aligned} E m [ MMD ℓ 2 ( P θ T , m , Q ) ] ≤ exp ( − 2 C ( 1 − R ) s = 0 ∑ T − 1 γ s ) MMD ℓ 2 ( P θ 0 , Q ) + t = 0 ∑ T − 1 exp ( − 2 C ( 1 − R ) s = t + 1 ∑ T − 1 γ s ) M t + t = 0 ∑ T − 1 exp ( − 2 C ( 1 − R ) s = t + 1 ∑ T − 1 γ s ) × N γ t 5 3 m 1 . (140) The first term in Eq. (C.10) converges to 0 as T → ∞ T\to\infty T → ∞ lim T → ∞ ∑ s = 0 T − 1 γ s → ∞ \lim_{T\to\infty}\sum_{s=0}^{T-1}\gamma_s\to\infty lim T → ∞ ∑ s = 0 T − 1 γ s → ∞ ε > 0 \varepsilon>0 ε > 0 M t \mathfrak{M}_t M t lim t → ∞ M t γ t − 1 = 0 \lim_{t\to\infty}\mathfrak{M}_t\gamma_t^{-1}=0 lim t → ∞ M t γ t − 1 = 0 t 0 ∈ N t_0\in\mathbb{N} t 0 ∈ N M t γ t − 1 < ε \mathfrak{M}_t\gamma_t^{-1}<\varepsilon M t γ t − 1 < ε t > t 0 t>t_0 t > t 0
(C.11) = ∑ t = 0 t 0 exp ( − 2 C ( 1 − R ) ∑ s = t + 1 T − 1 γ s ) M t + ∑ t = t 0 + 1 T − 1 exp ( − 2 C ( 1 − R ) ∑ s = t + 1 T − 1 γ s ) M t . \begin{aligned}
\text{(C.11)}={}&\sum_{t=0}^{t_0}\exp\left(-2C(1-\mathfrak{R})\sum_{s=t+1}^{T-1}\gamma_s\right)\mathfrak{M}_t \\
&+\sum_{t=t_0+1}^{T-1}\exp\left(-2C(1-\mathfrak{R})\sum_{s=t+1}^{T-1}\gamma_s\right)\mathfrak{M}_t.
\end{aligned} (C.11) = t = 0 ∑ t 0 exp ( − 2 C ( 1 − R ) s = t + 1 ∑ T − 1 γ s ) M t + t = t 0 + 1 ∑ T − 1 exp ( − 2 C ( 1 − R ) s = t + 1 ∑ T − 1 γ s ) M t . The first term in Eq. (C.11) is a finite sum which converges to 0 as T → ∞ T \to\infty T → ∞ exp ( − 2 C ( 1 − R ) ∑ s = t + 1 T − 1 γ s ) → 0 \exp\left(-2\mathcal{C}(1-\mathfrak{R})\sum_{s=t+1}^{T-1}\gamma_s\right) \to0 exp ( − 2 C ( 1 − R ) ∑ s = t + 1 T − 1 γ s ) → 0 t 0 t_0 t 0 ϵ ∑ t = t 0 + 1 T − 1 exp ( − 2 C ( 1 − R ) ∑ s = t + 1 T − 1 γ s ) γ t \epsilon\sum_{t=t_0+1}^{T-1}\exp\left(-2\mathcal{C}(1-\mathfrak{R})\sum_{s=t+1}^{T-1}\gamma_s\right)\gamma_t ϵ ∑ t = t 0 + 1 T − 1 exp ( − 2 C ( 1 − R ) ∑ s = t + 1 T − 1 γ s ) γ t 0 < γ t ≤ 1 2 C ( 1 − R ) 0 < \gamma_t \le\frac{1}{2\mathcal{C}(1-\mathfrak{R})} 0 < γ t ≤ 2 C ( 1 − R ) 1 ϵ \epsilon ϵ ϵ \epsilon ϵ T → ∞ T \to\infty T → ∞ m ( T ) ≥ ( ∑ s = 0 T γ s 0.6 ) ⋅ log ( T + 1 ) \sqrt{m(T)} \ge\left(\sum_{s=0}^{T}\gamma_s^{0.6}\right)\cdot\log(T+1) m ( T ) ≥ ( ∑ s = 0 T γ s 0.6 ) ⋅ log ( T + 1 )
(C.13) ≤ ∑ t = 0 T − 1 N ⋅ γ t 3 5 1 m ( T ) ≤ N log ( T + 1 ) → 0 , as T → ∞ . (141) \text{(C.13)} \le\sum_{t=0}^{T-1}\mathfrak{N}\cdot\gamma_t^{\frac{3}{5}}\frac{1}{\sqrt{m(T)}} \le\frac{\mathfrak{N}}{\log(T+1)} \to0,\quad\text{as }T\to\infty. \tag*{(141)} (C.13) ≤ t = 0 ∑ T − 1 N ⋅ γ t 5 3 m ( T ) 1 ≤ log ( T + 1 ) N → 0 , as T → ∞. (141) Therefore, we have proved that all the three terms in Eq. (C.10), Eq. (C.11) and Eq. (C.13) vanish as T → ∞ T \to\infty T → ∞ r T r_T r T
E m [ MMD ℓ 2 ( P θ T , m , Q ) ] ≤ r T , \mathbb{E}_m\left[\operatorname{MMD}_{\ell}^{2}(P_{\theta_{T,m}},\mathbb{Q})\right]\le r_T, E m [ MMD ℓ 2 ( P θ T , m , Q ) ] ≤ r T , This proves part (i).
For the mis-specified case, under the alternative gradient-dominance condition in part (ii), Eq. (C.6) gives the same recursion for the excess objective:
MMD ℓ 2 ( P θ t + 1 , m , Q ) − min θ ∈ Θ MMD ℓ 2 ( P θ , Q ) ≤ ( 1 − 2 C ( 1 − R ) γ t ) ( MMD ℓ 2 ( P θ t , m , Q ) − min θ ∈ Θ MMD ℓ 2 ( P θ , Q ) ) + M t + N ⋅ γ t 3 5 1 m . \operatorname{MMD}_{\ell}^{2}(P_{\theta_{t+1,m}},\mathbb{Q})-\min_{\theta\in\Theta}\operatorname{MMD}_{\ell}^{2}(P_\theta,\mathbb{Q})
\le\left(1-2\mathcal{C}(1-\mathfrak{R})\gamma_t\right)
\left(\operatorname{MMD}_{\ell}^{2}(P_{\theta_{t,m}},\mathbb{Q})-\min_{\theta\in\Theta}\operatorname{MMD}_{\ell}^{2}(P_\theta,\mathbb{Q})\right)+\mathfrak{M}_t+\mathfrak{N}\cdot\gamma_t^{\frac{3}{5}}\frac{1}{\sqrt{m}}. MMD ℓ 2 ( P θ t + 1 , m , Q ) − θ ∈ Θ min MMD ℓ 2 ( P θ , Q ) ≤ ( 1 − 2 C ( 1 − R ) γ t ) ( MMD ℓ 2 ( P θ t , m , Q ) − θ ∈ Θ min MMD ℓ 2 ( P θ , Q ) ) + M t + N ⋅ γ t 5 3 m 1 . The same expected-recursion and discrete Gronwall argument therefore imply that
E m [ MMD ℓ 2 ( P θ T , m , Q ) − min θ ∈ Θ MMD ℓ 2 ( P θ , Q ) ] ≤ r T , \mathbb{E}_m\left[\operatorname{MMD}_{\ell}^{2}(P_{\theta_{T,m}},\mathbb{Q})-\min_{\theta\in\Theta}\operatorname{MMD}_{\ell}^{2}(P_\theta,\mathbb{Q})\right]\le r_T, E m [ MMD ℓ 2 ( P θ T , m , Q ) − θ ∈ Θ min MMD ℓ 2 ( P θ , Q ) ] ≤ r T , where the corresponding deterministic remainder satisfies r T → 0 r_T\to0 r T → 0 T → ∞ T\to\infty T → ∞
Proof of Theorem 5.7
Proposition C.2 (Optimality of Δ θ P G D \Delta\theta^{\mathrm{PGD}} Δ θ PGD P 0 = P θ 0 \mathbb{P}_0=\mathbb{P}_{\theta_0} P 0 = P θ 0 γ 0 > 0 \gamma_0>0 γ 0 > 0 P 1 \mathbb{P}_1 P 1 (34) . Denote P θ 1 P G D \mathbb{P}_{\theta_1}^{\mathrm{PGD}} P θ 1 PGD (35) with a preconditioned update Δ θ P G D \Delta\theta^{\mathrm{PGD}} Δ θ PGD λ 0 ∝ ( γ p 2 ( 1 + ℓ − 1 ) ℓ 0 − 2 ) 2 5 \lambda_0\propto(\gamma p^2(1+\ell^{-1})\ell_0^{-2})^{\frac{2}{5}} λ 0 ∝ ( γ p 2 ( 1 + ℓ − 1 ) ℓ 0 − 2 ) 5 2
MMD ℓ 2 ( P θ 1 P G D , Q ) − MMD ℓ 2 ( P 1 , Q ) ≤ 2 γ 0 R ∥ ∇ f ℓ , P 0 , Q ∥ L 2 d ( P 0 ) 2 + C k γ 0 R ℓ 0 2 − ℓ 2 ℓ 4 + C k d 1 2 p 2 5 γ 0 6 5 γ 4 5 ℓ − 1 ( ( 1 + ℓ − 1 ) 1 5 ℓ 0 − 2 5 ) + C k γ 0 2 ℓ − 2 ℓ 0 − 2 . (142) \operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta_1}^{\mathrm{PGD}},\mathbb{Q})-\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_1,\mathbb{Q})
\leq2\gamma_0\mathcal{R}\lVert\nabla f_{\ell,\mathbb{P}_0,\mathbb{Q}}\rVert_{L_2^d(\mathbb{P}_0)}^2+C_k\gamma_0\mathcal{R}\frac{\ell_0^2-\ell^2}{\ell^4}
+C_kd^{\frac{1}{2}}p^{\frac{2}{5}}\gamma_0^{\frac{6}{5}}\gamma^{\frac{4}{5}}\ell^{-1}\left((1+\ell^{-1})^{\frac{1}{5}}\ell_0^{-\frac{2}{5}}\right)+C_k\gamma_0^2\ell^{-2}\ell_0^{-2}. \tag*{(142)} MMD ℓ 2 ( P θ 1 PGD , Q ) − MMD ℓ 2 ( P 1 , Q ) ≤ 2 γ 0 R ∥ ∇ f ℓ , P 0 , Q ∥ L 2 d ( P 0 ) 2 + C k γ 0 R ℓ 4 ℓ 0 2 − ℓ 2 + C k d 2 1 p 5 2 γ 0 5 6 γ 5 4 ℓ − 1 ( ( 1 + ℓ − 1 ) 5 1 ℓ 0 − 5 2 ) + C k γ 0 2 ℓ − 2 ℓ 0 − 2 . (142) Here, C k C_k C k
Proof. Let P 0 = P θ 0 = ( G θ 0 ) # ρ \mathbb{P}_0=\mathbb{P}_{\theta_0}=(G_{\theta_0})_{\#}\rho P 0 = P θ 0 = ( G θ 0 ) # ρ t = 0 t=0 t = 0 γ 0 , λ 0 \gamma_0,\lambda_0 γ 0 , λ 0
P θ 1 = ( G θ 0 − γ Δ θ ) # ρ , \mathbb{P}_{\theta_1}=(G_{\theta_0-\gamma\Delta\theta})_{\#}\rho, P θ 1 = ( G θ 0 − γ Δ θ ) # ρ , P 1 = ( Id − γ ∇ f ℓ 0 , P 0 , Q ) # P 0 = ( Id − γ ∇ f ℓ 0 , P 0 , Q ) # ( G θ 0 ) # ρ . \begin{aligned}
\mathbb{P}_1=(\operatorname{Id}-\gamma\nabla f_{\ell_0,\mathbb{P}_0,\mathbb{Q}})_{\#}\mathbb{P}_0 \\
&=(\operatorname{Id}-\gamma\nabla f_{\ell_0,\mathbb{P}_0,\mathbb{Q}})_{\#}(G_{\theta_0})_{\#}\rho.
\end{aligned} P 1 = ( Id − γ ∇ f ℓ 0 , P 0 , Q ) # P 0 = ( Id − γ ∇ f ℓ 0 , P 0 , Q ) # ( G θ 0 ) # ρ . Note that here Δ θ \Delta\theta Δ θ R p \mathbb{R}^p R p Q \mathbb{Q} Q Q \mathbb{Q} Q
I 1 ( γ ) : = MMD ℓ 2 ( ( Id − γ ∇ f ℓ 0 , P 0 , Q ) # P 0 , Q ) = E z ∼ ρ , z ′ ∼ ρ [ k ℓ ( G θ 0 ( z ) − γ ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z ) ) , G θ 0 ( z ′ ) − γ ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z ′ ) ) ) ] . (143) \begin{aligned}
\mathcal{I}_1(\gamma):=\operatorname{MMD}_{\ell}^{2}\left((\operatorname{Id}-\gamma\nabla f_{\ell_0,\mathbb{P}_0,\mathbb{Q}})_{\#}\mathbb{P}_0,\mathbb{Q}\right) \\
&=\mathbb{E}_{z\sim\rho,z'\sim\rho}\left[
\begin{array}{l}
k_{\ell}\left(G_{\theta_0}(z)-\gamma\nabla f_{\ell_0,\mathbb{P}_0,\mathbb{Q}}(G_{\theta_0}(z)),\right.\\
\left.\qquad G_{\theta_0}(z')-\gamma\nabla f_{\ell_0,\mathbb{P}_0,\mathbb{Q}}(G_{\theta_0}(z'))\right)
\end{array}
\right]. \tag*{(143)}
\end{aligned} I 1 ( γ ) := MMD ℓ 2 ( ( Id − γ ∇ f ℓ 0 , P 0 , Q ) # P 0 , Q ) = E z ∼ ρ , z ′ ∼ ρ [ k ℓ ( G θ 0 ( z ) − γ ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z )) , G θ 0 ( z ′ ) − γ ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z ′ )) ) ] . (143) − 2 E z ∼ ρ , y ∼ Q [ k ℓ ( G θ 0 ( z ) − γ ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z ) ) , y ) ] + E y ∼ Q , y ′ ∼ Q [ k ℓ ( y , y ′ ) ] . (144) -2\mathbb{E}_{z\sim\rho,y\sim Q}\left[k_{\ell}\left(G_{\theta_0}(z)-\gamma\nabla f_{\ell_0,P_0,Q}(G_{\theta_0}(z)),y\right)\right]
+\mathbb{E}_{y\sim Q,y'\sim Q}\left[k_{\ell}(y,y')\right]. \tag*{(144)} − 2 E z ∼ ρ , y ∼ Q [ k ℓ ( G θ 0 ( z ) − γ ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z )) , y ) ] + E y ∼ Q , y ′ ∼ Q [ k ℓ ( y , y ′ ) ] . (144) And
I 2 ( γ ) : = MMD ℓ 2 ( ( G θ 0 − γ Δ θ ) # ρ , Q ) = E z , z ′ ∼ ρ [ k ℓ ( G θ 0 − γ Δ θ ( z ) , G θ 0 − γ Δ θ ( z ′ ) ) ] (145) I_2(\gamma):=\operatorname{MMD}_{\ell}^{2}\left((G_{\theta_0-\gamma\Delta\theta})_{\#}\rho,Q\right)
=\mathbb{E}_{z,z'\sim\rho}\left[k_{\ell}\left(G_{\theta_0-\gamma\Delta\theta}(z),G_{\theta_0-\gamma\Delta\theta}(z')\right)\right] \tag*{(145)} I 2 ( γ ) := MMD ℓ 2 ( ( G θ 0 − γ Δ θ ) # ρ , Q ) = E z , z ′ ∼ ρ [ k ℓ ( G θ 0 − γ Δ θ ( z ) , G θ 0 − γ Δ θ ( z ′ ) ) ] (145) − 2 E z ∼ ρ , y ∼ Q [ k ℓ ( G θ 0 − γ Δ θ ( z ) , y ) ] + E y , y ′ ∼ Q [ k ℓ ( y , y ′ ) ] (146) -2\mathbb{E}_{z\sim\rho,y\sim Q}\left[k_{\ell}\left(G_{\theta_0-\gamma\Delta\theta}(z),y\right)\right]
+\mathbb{E}_{y,y'\sim Q}\left[k_{\ell}(y,y')\right] \tag*{(146)} − 2 E z ∼ ρ , y ∼ Q [ k ℓ ( G θ 0 − γ Δ θ ( z ) , y ) ] + E y , y ′ ∼ Q [ k ℓ ( y , y ′ ) ] (146) For the Gaussian kernel and for the Matérn kernel with smoothness s > 1 s>1 s > 1 ( x , y ) ↦ k ℓ ( x , y ) (x,y)\mapsto k_{\ell}(x,y) ( x , y ) ↦ k ℓ ( x , y ) ℓ − 1 \ell^{-1} ℓ − 1 ℓ − 2 \ell^{-2} ℓ − 2 γ \gamma γ
(C.15) = E z , z ′ [ k ℓ ( G θ 0 ( z ) , G θ 0 ( z ′ ) ) ] − 2 γ E z , z ′ [ ∇ 1 k ℓ ( G θ 0 ( z ) , G θ 0 ( z ′ ) ) T ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z ) ) ] + κ 1 2 γ 2 (147) \begin{aligned}
\text{(C.15)}={}&\mathbb{E}_{z,z'}\left[k_{\ell}\left(G_{\theta_0}(z),G_{\theta_0}(z')\right)\right]\\
&-2\gamma\mathbb{E}_{z,z'}\left[\nabla_1 k_{\ell}\left(G_{\theta_0}(z),G_{\theta_0}(z')\right)^{\mathsf T}\nabla f_{\ell_0,P_0,Q}\left(G_{\theta_0}(z)\right)\right]+\frac{\kappa_1}{2}\gamma^2 \tag*{(147)}
\end{aligned} (C.15) = E z , z ′ [ k ℓ ( G θ 0 ( z ) , G θ 0 ( z ′ ) ) ] − 2 γ E z , z ′ [ ∇ 1 k ℓ ( G θ 0 ( z ) , G θ 0 ( z ′ ) ) T ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z ) ) ] + 2 κ 1 γ 2 (147) (C.16) = − 2 E z , y [ k ℓ ( G θ 0 ( z ) , y ) ] + 2 γ E z , y [ ∇ 1 k ℓ ( G θ 0 ( z ) , y ) T ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z ) ) ] + κ 2 2 γ 2 + E y , y ′ ∼ Q [ k ℓ ( y , y ′ ) ] . (148) \begin{aligned}
\text{(C.16)}={}&-2\mathbb{E}_{z,y}\left[k_{\ell}\left(G_{\theta_0}(z),y\right)\right]+2\gamma\mathbb{E}_{z,y}\left[\nabla_1 k_{\ell}\left(G_{\theta_0}(z),y\right)^{\mathsf T}\nabla f_{\ell_0,P_0,Q}\left(G_{\theta_0}(z)\right)\right]\\
&+\frac{\kappa_2}{2}\gamma^2+\mathbb{E}_{y,y'\sim Q}\left[k_{\ell}(y,y')\right]. \tag*{(148)}
\end{aligned} (C.16) = − 2 E z , y [ k ℓ ( G θ 0 ( z ) , y ) ] + 2 γ E z , y [ ∇ 1 k ℓ ( G θ 0 ( z ) , y ) T ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z ) ) ] + 2 κ 2 γ 2 + E y , y ′ ∼ Q [ k ℓ ( y , y ′ ) ] . (148) Denote y s = G θ 0 ( z ) − s γ ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z ) ) ∈ R d y_s=G_{\theta_0}(z)-s\gamma\nabla f_{\ell_0,P_0,Q}(G_{\theta_0}(z))\in\mathbb{R}^d y s = G θ 0 ( z ) − s γ ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z )) ∈ R d s ∈ [ 0 , 1 ] s\in[0,1] s ∈ [ 0 , 1 ] y s ′ y'_s y s ′ κ 1 , κ 2 ∈ R \kappa_1,\kappa_2\in\mathbb{R} κ 1 , κ 2 ∈ R
max { ∣ κ 1 ∣ , ∣ κ 2 ∣ } ≤ sup s ∈ [ 0 , 1 ] E z , z ′ [ ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z ) ) T ∇ 11 2 k ℓ ( y s , y s ′ ) ∇ f ℓ 0 , P 0 , Q ( G θ s ( z ) ) ] + sup s ∈ [ 0 , 1 ] E z , z ′ [ ∇ f ℓ 0 , P 0 , Q ( G θ s ( z ) ) T ∇ 12 2 k ℓ ( y s , y s ′ ) ∇ f ℓ 0 , P 0 , Q ( G θ s ( z ) ) ] ≤ C k ℓ − 2 ℓ 0 − 2 . \begin{aligned}
\max\{|\kappa_1|,|\kappa_2|\}
&\leq\sup_{s\in[0,1]}\mathbb{E}_{z,z'}\left[\nabla f_{\ell_0,P_0,Q}(G_{\theta_0}(z))^{\mathsf T}\nabla^2_{11}k_{\ell}(y_s,y'_s)\nabla f_{\ell_0,P_0,Q}(G_{\theta_s}(z))\right]\\
&\quad+\sup_{s\in[0,1]}\mathbb{E}_{z,z'}\left[\nabla f_{\ell_0,P_0,Q}(G_{\theta_s}(z))^{\mathsf T}\nabla^2_{12}k_{\ell}(y_s,y'_s)\nabla f_{\ell_0,P_0,Q}(G_{\theta_s}(z))\right]\\
&\leq C_k\ell^{-2}\ell_0^{-2}.
\end{aligned} max { ∣ κ 1 ∣ , ∣ κ 2 ∣ } ≤ s ∈ [ 0 , 1 ] sup E z , z ′ [ ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z ) ) T ∇ 11 2 k ℓ ( y s , y s ′ ) ∇ f ℓ 0 , P 0 , Q ( G θ s ( z )) ] + s ∈ [ 0 , 1 ] sup E z , z ′ [ ∇ f ℓ 0 , P 0 , Q ( G θ s ( z ) ) T ∇ 12 2 k ℓ ( y s , y s ′ ) ∇ f ℓ 0 , P 0 , Q ( G θ s ( z )) ] ≤ C k ℓ − 2 ℓ 0 − 2 . The last inequality holds by the bounds
sup y , y ′ ∥ ∇ 11 2 k ℓ ( y , y ′ ) ∥ o p ≤ C k ℓ − 2 , sup y , y ′ ∥ ∇ 12 2 k ℓ ( y , y ′ ) ∥ o p ≤ C k ℓ − 2 , \sup_{y,y'} \left\lVert\nabla^2_{11} k_\ell(y,y') \right\rVert_{\mathrm{op}} \le C_k\ell^{-2}, \qquad\sup_{y,y'} \left\lVert\nabla^2_{12} k_\ell(y,y') \right\rVert_{\mathrm{op}} \le C_k\ell^{-2}, y , y ′ sup ∇ 11 2 k ℓ ( y , y ′ ) op ≤ C k ℓ − 2 , y , y ′ sup ∇ 12 2 k ℓ ( y , y ′ ) op ≤ C k ℓ − 2 , and sup x ∥ ∇ f ℓ 0 , P 0 , Q ( x ) ∥ ≤ C k ℓ 0 − 1 \sup_x \left\lVert\nabla f_{\ell_0,\mathbb{P}_0,\mathbb{Q}}(x) \right\rVert\le C_k\ell_0^{-1} sup x ∥ ∇ f ℓ 0 , P 0 , Q ( x ) ∥ ≤ C k ℓ 0 − 1
J 1 ( γ ) = MMD ℓ 2 ( P θ 0 , Q ) − 2 γ ⋅ E z [ ∇ f ℓ , P 0 , Q ( G θ 0 ( z ) ) T \mathcal{J}_1(\gamma) = \operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta_0}, \mathbb{Q}) - 2\gamma\cdot\mathbb{E}_z \left[ \nabla f_{\ell,\mathbb{P}_0,\mathbb{Q}}(G_{\theta_0}(z))^{\mathsf{T}} \right. J 1 ( γ ) = MMD ℓ 2 ( P θ 0 , Q ) − 2 γ ⋅ E z [ ∇ f ℓ , P 0 , Q ( G θ 0 ( z ) ) T × ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z ) ) ] + κ 1 + κ 2 2 γ 2 . (149) \left. \qquad\times\nabla f_{\ell_0,\mathbb{P}_0,\mathbb{Q}}(G_{\theta_0}(z)) \right] + \frac{\kappa_1+\kappa_2}{2}\gamma^2. \tag*{(149)} × ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z )) ] + 2 κ 1 + κ 2 γ 2 . (149) Similarly, we can perform the same analysis for J 2 ( γ ) \mathcal{J}_2(\gamma) J 2 ( γ ) θ ↦ G θ ( z ) \theta\mapsto G_\theta(z) θ ↦ G θ ( z )
( C .17 ) = E z , z ′ [ k ℓ ( G θ 0 ( z ) , G θ 0 ( z ′ ) ) ] (\mathrm{C.17}) = \mathbb{E}_{z,z'}\left[k_\ell\left(G_{\theta_0}(z),G_{\theta_0}(z')\right)\right] ( C.17 ) = E z , z ′ [ k ℓ ( G θ 0 ( z ) , G θ 0 ( z ′ ) ) ] − 2 γ E z , z ′ [ ∇ 1 k ℓ ( G θ 0 ( z ) , G θ 0 ( z ′ ) ) T J θ G θ 0 ( z ) Δ θ ] + κ 3 γ 2 2 , \qquad- 2\gamma\mathbb{E}_{z,z'}\left[\nabla_1 k_\ell\left(G_{\theta_0}(z),G_{\theta_0}(z')\right)^{\mathsf{T}} J_\theta G_{\theta_0}(z)\Delta\theta\right] + \kappa_3\frac{\gamma^2}{2}, − 2 γ E z , z ′ [ ∇ 1 k ℓ ( G θ 0 ( z ) , G θ 0 ( z ′ ) ) T J θ G θ 0 ( z ) Δ θ ] + κ 3 2 γ 2 , ( C .18 ) = − 2 E z , y [ k ℓ ( G θ 0 ( z ) , y ) ] + 2 γ E z , y [ ∇ 1 k ℓ ( G θ 0 ( z ) , y ) T J θ G θ 0 ( z ) Δ θ ] + κ 4 γ 2 2 . (\mathrm{C.18}) = -2\mathbb{E}_{z,y}\left[k_\ell\left(G_{\theta_0}(z),y\right)\right] + 2\gamma\mathbb{E}_{z,y}\left[\nabla_1 k_\ell\left(G_{\theta_0}(z),y\right)^{\mathsf{T}} J_\theta G_{\theta_0}(z)\Delta\theta\right] + \kappa_4\frac{\gamma^2}{2}. ( C.18 ) = − 2 E z , y [ k ℓ ( G θ 0 ( z ) , y ) ] + 2 γ E z , y [ ∇ 1 k ℓ ( G θ 0 ( z ) , y ) T J θ G θ 0 ( z ) Δ θ ] + κ 4 2 γ 2 . Denote θ s = θ 0 − s γ Δ θ ∈ Θ \theta_s=\theta_0-s\gamma\Delta\theta\in\Theta θ s = θ 0 − s γ Δ θ ∈ Θ s ∈ [ 0 , 1 ] s\in[0,1] s ∈ [ 0 , 1 ] κ 3 , κ 4 ∈ R \kappa_3,\kappa_4\in\mathbb{R} κ 3 , κ 4 ∈ R
max { ∣ κ 3 ∣ , ∣ κ 4 ∣ } ≤ sup s ∈ [ 0 , 1 ] E z , z ′ [ ( J θ G θ s ( z ) Δ θ ) T ∇ 11 2 k ℓ ( G θ s ( z ) , G θ s ( z ′ ) ) × ( J θ G θ s ( z ) Δ θ ) ] + sup s ∈ [ 0 , 1 ] E z , z ′ [ ( J θ G θ s ( z ) Δ θ ) T ∇ 12 2 k ℓ ( G θ s ( z ) , G θ s ( z ′ ) ) × ( J θ G θ s ( z ′ ) Δ θ ) ] + sup s ∈ [ 0 , 1 ] E z , z ′ [ ∇ 1 k ℓ ( G θ s ( z ) , G θ s ( z ′ ) ) T × ( Δ θ T ∇ θ 2 G θ s ( z ) Δ θ ) ] ≤ sup x , x ′ ∥ ∇ 11 2 k ℓ ( x , x ′ ) ∥ o p × sup s ∈ [ 0 , 1 ] E z [ ∥ J θ G θ s ( z ) ∥ o p 2 ] ∥ Δ θ ∥ 2 + sup x , x ′ ∥ ∇ 1 , 2 2 k ℓ ( x , x ′ ) ∥ o p × sup s ∈ [ 0 , 1 ] [ E z ∥ J θ G θ s ( z ) ∥ o p ] 2 ∥ Δ θ ∥ 2 + sup x , x ′ ∥ ∇ 1 k ℓ ( x , x ′ ) ∥ × sup s ∈ [ 0 , 1 ] [ E z ∥ ∇ θ 2 G θ s ( z ) ∥ o p ] ∥ Δ θ ∥ 2 ≤ C k p d ( ℓ − 2 + ℓ − 1 ) ∥ Δ θ ∥ 2 . \begin{aligned}
\max\{|\kappa_3|,|\kappa_4|\}
&\le\sup_{s\in[0,1]}\mathbb{E}_{z,z'}\left[
\left(J_\theta G_{\theta_s}(z)\Delta\theta\right)^{\mathsf{T}}
\nabla^2_{11}k_\ell\left(G_{\theta_s}(z),G_{\theta_s}(z')\right)\right.\\
&\qquad\qquad\left.\times\left(J_\theta G_{\theta_s}(z)\Delta\theta\right)
\right]\\
&\quad+\sup_{s\in[0,1]}\mathbb{E}_{z,z'}\left[
\left(J_\theta G_{\theta_s}(z)\Delta\theta\right)^{\mathsf{T}}
\nabla^2_{12}k_\ell\left(G_{\theta_s}(z),G_{\theta_s}(z')\right)\right.\\
&\qquad\qquad\left.\times\left(J_\theta G_{\theta_s}(z')\Delta\theta\right)
\right]\\
&\quad+\sup_{s\in[0,1]}\mathbb{E}_{z,z'}\left[
\nabla_1 k_\ell\left(G_{\theta_s}(z),G_{\theta_s}(z')\right)^{\mathsf{T}}\right.\\
&\qquad\qquad\left.\times\left(\Delta\theta^{\mathsf{T}}\nabla^2_\theta G_{\theta_s}(z)\Delta\theta\right)
\right]\\
&\le\sup_{x,x'}\left\lVert\nabla^2_{11}k_\ell(x,x')\right\rVert_{\mathrm{op}}\\
&\quad\times\sup_{s\in[0,1]}\mathbb{E}_z\left[\left\lVert J_\theta G_{\theta_s}(z)\right\rVert_{\mathrm{op}}^2\right]\left\lVert\Delta\theta\right\rVert^2 \\
&+ \sup_{x,x'} \left\lVert\nabla^2_{1,2} k_\ell(x,x') \right\rVert_{\mathrm{op}}
\times\sup_{s\in[0,1]} \left[\mathbb{E}_z \left\lVert J_\theta G_{\theta_s}(z) \right\rVert_{\mathrm{op}}\right]^2 \left\lVert\Delta\theta\right\rVert^2 \\
&+ \sup_{x,x'} \left\lVert\nabla_1 k_\ell(x,x') \right\rVert
\times\sup_{s\in[0,1]} \left[\mathbb{E}_z \left\lVert\nabla^2_\theta G_{\theta_s}(z) \right\rVert_{\mathrm{op}}\right] \left\lVert\Delta\theta\right\rVert^2 \\
&\le C_k p\sqrt{d}\left(\ell^{-2}+\ell^{-1}\right)\left\lVert\Delta\theta\right\rVert^2.
\end{aligned} max { ∣ κ 3 ∣ , ∣ κ 4 ∣ } ≤ s ∈ [ 0 , 1 ] sup E z , z ′ [ ( J θ G θ s ( z ) Δ θ ) T ∇ 11 2 k ℓ ( G θ s ( z ) , G θ s ( z ′ ) ) × ( J θ G θ s ( z ) Δ θ ) ] + s ∈ [ 0 , 1 ] sup E z , z ′ [ ( J θ G θ s ( z ) Δ θ ) T ∇ 12 2 k ℓ ( G θ s ( z ) , G θ s ( z ′ ) ) × ( J θ G θ s ( z ′ ) Δ θ ) ] + s ∈ [ 0 , 1 ] sup E z , z ′ [ ∇ 1 k ℓ ( G θ s ( z ) , G θ s ( z ′ ) ) T × ( Δ θ T ∇ θ 2 G θ s ( z ) Δ θ ) ] ≤ x , x ′ sup ∇ 11 2 k ℓ ( x , x ′ ) op × s ∈ [ 0 , 1 ] sup E z [ ∥ J θ G θ s ( z ) ∥ op 2 ] ∥ Δ θ ∥ 2 + x , x ′ sup ∇ 1 , 2 2 k ℓ ( x , x ′ ) op × s ∈ [ 0 , 1 ] sup [ E z ∥ J θ G θ s ( z ) ∥ op ] 2 ∥ Δ θ ∥ 2 + x , x ′ sup ∥ ∇ 1 k ℓ ( x , x ′ ) ∥ × s ∈ [ 0 , 1 ] sup [ E z ∇ θ 2 G θ s ( z ) op ] ∥ Δ θ ∥ 2 ≤ C k p d ( ℓ − 2 + ℓ − 1 ) ∥ Δ θ ∥ 2 . The second last step holds by Assumption 5.3 along with the bounds that
sup s E z [ ∥ J θ G θ s ( z ) ∥ o p 2 ] ≤ sup θ ∈ Θ E z ∼ ρ [ ∥ J θ G θ ( z ) ∥ o p 2 ] ≤ sup θ ∈ Θ ∑ j = 1 p E z ∼ ρ [ ∥ ∂ θ j G θ ( z ) ∥ 2 ] ≤ p , \begin{aligned}
\sup_s \mathbb{E}_z\left[\left\lVert J_\theta G_{\theta_s}(z) \right\rVert_{\mathrm{op}}^2\right]
\le\sup_{\theta\in\Theta}\mathbb{E}_{z\sim\rho}\left[\left\lVert J_\theta G_\theta(z) \right\rVert_{\mathrm{op}}^2\right] \\
&\le\sup_{\theta\in\Theta}\sum_{j=1}^{p}\mathbb{E}_{z\sim\rho}\left[\left\lVert\partial_{\theta_j}G_\theta(z) \right\rVert^2\right]\le p,
\end{aligned} s sup E z [ ∥ J θ G θ s ( z ) ∥ op 2 ] ≤ θ ∈ Θ sup E z ∼ ρ [ ∥ J θ G θ ( z ) ∥ op 2 ] ≤ θ ∈ Θ sup j = 1 ∑ p E z ∼ ρ [ ∂ θ j G θ ( z ) 2 ] ≤ p , sup s E z [ ∥ ∇ θ 2 G θ s ( z ) ∥ o p ] ≤ sup θ ∈ Θ E z [ ∥ ∇ θ 2 G θ ( z ) ∥ o p 2 ] ≤ sup θ ∈ Θ ∑ i , j = 1 p E z ∼ ρ [ ∥ ∂ θ j ∂ θ i G θ ( z ) ∥ 2 ] ≤ p . \begin{aligned}
\sup_s\mathbb{E}_z\left[\left\lVert\nabla^2_\theta G_{\theta_s}(z) \right\rVert_{\mathrm{op}}\right]
\le\sup_{\theta\in\Theta}\sqrt{\mathbb{E}_z\left[\left\lVert\nabla^2_\theta G_\theta(z) \right\rVert_{\mathrm{op}}^2\right]} \\
&\le\sup_{\theta\in\Theta}\sqrt{\sum_{i,j=1}^{p}\mathbb{E}_{z\sim\rho}\left[\left\lVert\partial_{\theta_j}\partial_{\theta_i}G_\theta(z) \right\rVert^2\right]}\le p.
\end{aligned} s sup E z [ ∇ θ 2 G θ s ( z ) op ] ≤ θ ∈ Θ sup E z [ ∥ ∇ θ 2 G θ ( z ) ∥ op 2 ] ≤ θ ∈ Θ sup i , j = 1 ∑ p E z ∼ ρ [ ∂ θ j ∂ θ i G θ ( z ) 2 ] ≤ p . and by that sup x , x ′ ∥ ∇ 1 k ℓ ( x , x ′ ) ∥ ≤ C k d ℓ − 1 \sup_{x,x'}\left\lVert\nabla_1 k_\ell(x,x') \right\rVert\le C_k\sqrt{d}\ell^{-1} sup x , x ′ ∥ ∇ 1 k ℓ ( x , x ′ ) ∥ ≤ C k d ℓ − 1 sup x , x ′ ∥ ∇ 1 , 2 2 k ℓ ( x , x ′ ) ∥ o p ≤ C k d ℓ − 2 \sup_{x,x'}\left\lVert\nabla^2_{1,2} k_\ell(x,x') \right\rVert_{\mathrm{op}}\le C_k\sqrt{d}\ell^{-2} sup x , x ′ ∇ 1 , 2 2 k ℓ ( x , x ′ ) op ≤ C k d ℓ − 2 ∇ θ 2 G θ s \nabla^2_\theta G_{\theta_s} ∇ θ 2 G θ s
∇ θ 2 G θ ( z ) [ u , v ] : = ( u ⊤ ∇ θ 2 [ G θ ( z ) ] 1 v , … , u ⊤ ∇ θ 2 [ G θ ( z ) ] d v ) \nabla^2_\theta G_\theta(z)[u,v] := \left(u^\top\nabla^2_\theta[G_\theta(z)]_{1}v,\ldots,u^\top\nabla^2_\theta[G_\theta(z)]_{d}v\right) ∇ θ 2 G θ ( z ) [ u , v ] := ( u ⊤ ∇ θ 2 [ G θ ( z ) ] 1 v , … , u ⊤ ∇ θ 2 [ G θ ( z ) ] d v ) ∈ R d , ∀ u , v ∈ R p , \in\mathbb{R}^{d},\qquad\forall u,v\in\mathbb{R}^{p}, ∈ R d , ∀ u , v ∈ R p , and its operator norm is
∥ ∇ θ 2 G θ ( z ) ∥ o p : = sup ∥ u ∥ = 1 , ∥ v ∥ = 1 ∥ ∇ θ 2 G θ ( z ) [ u , v ] ∥ . \left\lVert\nabla^2_\theta G_\theta(z) \right\rVert_{\mathrm{op}} := \sup_{\lVert u\rVert=1,\lVert v\rVert=1}\left\lVert\nabla^2_\theta G_\theta(z)[u,v] \right\rVert. ∇ θ 2 G θ ( z ) op := ∥ u ∥ = 1 , ∥ v ∥ = 1 sup ∇ θ 2 G θ ( z ) [ u , v ] . Combining the above two equalities on Eq. (C.17) and Eq. (C.18), we obtain
I 2 ( γ ) = MMD ℓ 2 ( P θ 0 , Q ) − 2 γ ⋅ E z [ ∇ f ℓ , P 0 , Q ( G θ 0 ( z ) ) ⊤ J θ G θ 0 ( z ) Δ θ ] + κ 3 + κ 4 2 γ 2 . (150) \mathcal{I}_2(\gamma)=\operatorname{MMD}_{\ell}^{2}(P_{\theta_0},\mathbb{Q})-2\gamma\cdot\mathbb{E}_z\left[\nabla f_{\ell,P_0,Q}(G_{\theta_0}(z))^\top J_\theta G_{\theta_0}(z)\Delta\theta\right]+\frac{\kappa_3+\kappa_4}{2}\gamma^2. \tag*{(150)} I 2 ( γ ) = MMD ℓ 2 ( P θ 0 , Q ) − 2 γ ⋅ E z [ ∇ f ℓ , P 0 , Q ( G θ 0 ( z ) ) ⊤ J θ G θ 0 ( z ) Δ θ ] + 2 κ 3 + κ 4 γ 2 . (150) Now, from the Taylor expansion of I 1 ( γ ) \mathcal{I}_1(\gamma) I 1 ( γ ) I 2 ( γ ) \mathcal{I}_2(\gamma) I 2 ( γ )
∣ MMD ℓ 2 ( ( Id − γ ∇ f ℓ 0 , P 0 , Q ) # P 0 , Q ) − MMD ℓ 2 ( ( G θ 0 − γ Δ θ ) # P , Q ) ∣ : = ∣ I 1 ( γ ) − I 2 ( γ ) ∣ = 2 γ ∣ E z [ ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z ) ) T ( ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z ) ) − J G G θ 0 ( z ) Δ θ ) ] + ∣ κ 1 + κ 2 2 γ 2 − κ 3 + κ 4 2 γ 2 ∣ ≤ 2 γ ∥ ∇ f ℓ 0 , P 0 , Q ∥ L 2 d ( P 0 ) × E z [ ∥ ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z ) ) − J G G θ 0 ( z ) Δ θ ∥ 2 ] ⏟ J ( Δ θ ) + C k γ 2 ( p d ( ℓ − 2 + ℓ − 1 ) ∥ Δ θ ∥ 2 + ℓ − 2 ℓ 0 − 2 ) . (151) \begin{aligned}
&\left|\operatorname{MMD}_{\ell}^{2}\left((\operatorname{Id}-\gamma\nabla f_{\ell_0,\mathbb{P}_0,\mathbb{Q}})\#\mathbb{P}_0,\mathbb{Q}\right)\right.\\
&\qquad\left.-\operatorname{MMD}_{\ell}^{2}\left((G_{\theta_0-\gamma\Delta\theta})\#\mathbb{P},\mathbb{Q}\right)\right| := \left|\mathcal{I}_1(\gamma)-\mathcal{I}_2(\gamma)\right|\\
&=2\gamma\left|\mathbb{E}_{z}\left[\nabla f_{\ell_0,\mathbb{P}_0,\mathbb{Q}}(G_{\theta_0}(z))^{\mathsf{T}}\left(\nabla f_{\ell_0,\mathbb{P}_0,\mathbb{Q}}(G_{\theta_0}(z))-\mathbf{J}_{G}G_{\theta_0}(z)\Delta\theta\right)\right]\right.\\
&\qquad\left.+\left|\frac{\kappa_1+\kappa_2}{2}\gamma^2-\frac{\kappa_3+\kappa_4}{2}\gamma^2\right|\right.\\
&\leq2\gamma\left\|\nabla f_{\ell_0,\mathbb{P}_0,\mathbb{Q}}\right\|_{L^d_2(\mathbb{P}_0)}\\
&\qquad\times\sqrt{\underbrace{\mathbb{E}_{z}\left[\left\|\nabla f_{\ell_0,\mathbb{P}_0,\mathbb{Q}}(G_{\theta_0}(z))-\mathbf{J}_{G}G_{\theta_0}(z)\Delta\theta\right\|^2\right]}_{\mathcal{J}(\Delta\theta)}}\\
&\qquad+C_k\gamma^2\left(p\sqrt{d}(\ell^{-2}+\ell^{-1})\|\Delta\theta\|^2+\ell^{-2}\ell_0^{-2}\right). \tag*{(151)}
\end{aligned} MMD ℓ 2 ( ( Id − γ ∇ f ℓ 0 , P 0 , Q ) # P 0 , Q ) − MMD ℓ 2 ( ( G θ 0 − γ Δ θ ) # P , Q ) := ∣ I 1 ( γ ) − I 2 ( γ ) ∣ = 2 γ E z [ ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z ) ) T ( ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z )) − J G G θ 0 ( z ) Δ θ ) ] + 2 κ 1 + κ 2 γ 2 − 2 κ 3 + κ 4 γ 2 ≤ 2 γ ∥ ∇ f ℓ 0 , P 0 , Q ∥ L 2 d ( P 0 ) × J ( Δ θ ) E z [ ∥ ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z )) − J G G θ 0 ( z ) Δ θ ∥ 2 ] + C k γ 2 ( p d ( ℓ − 2 + ℓ − 1 ) ∥Δ θ ∥ 2 + ℓ − 2 ℓ 0 − 2 ) . (151) The last inequality holds by applying Cauchy-Schwarz for the first term, and by plugging in the bounds on ∣ κ 1 ∣ , ∣ κ 2 ∣ , ∣ κ 3 ∣ , ∣ κ 4 ∣ \left|\kappa_1\right|,\left|\kappa_2\right|,\left|\kappa_3\right|,\left|\kappa_4\right| ∣ κ 1 ∣ , ∣ κ 2 ∣ , ∣ κ 3 ∣ , ∣ κ 4 ∣ min u ∈ R p J ( u ) ≤ R 2 ∥ ∇ f ℓ 0 , P θ 0 , Q ∥ L 2 d ( P θ 0 ) 2 = R 2 ∥ ∇ f ℓ 0 , P 0 , Q ∥ L 2 d ( P 0 ) 2 \min_{u\in\mathbb{R}^p}\mathcal{J}(u)\leq\mathfrak{R}^2\|\nabla f_{\ell_0,\mathbb{P}_{\theta_0},\mathbb{Q}}\|_{L^d_2(\mathbb{P}_{\theta_0})}^2=\mathfrak{R}^2\|\nabla f_{\ell_0,\mathbb{P}_0,\mathbb{Q}}\|_{L^d_2(\mathbb{P}_0)}^2 min u ∈ R p J ( u ) ≤ R 2 ∥∇ f ℓ 0 , P θ 0 , Q ∥ L 2 d ( P θ 0 ) 2 = R 2 ∥∇ f ℓ 0 , P 0 , Q ∥ L 2 d ( P 0 ) 2 u 0 ∗ : = arg min J ( u ) u_0^*:=\arg\min\mathcal{J}(u) u 0 ∗ := arg min J ( u ) ∥ u 0 ∗ ∥ ≤ U < ∞ \|u_0^*\|\leq\mathcal{U}<\infty ∥ u 0 ∗ ∥ ≤ U < ∞ r ∈ L 2 ( ρ ) r\in L_2(\rho) r ∈ L 2 ( ρ ) ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( ⋅ ) ) = J θ G θ 0 ( ⋅ ) u 0 ∗ + r ( ⋅ ) \nabla f_{\ell_0,\mathbb{P}_0,\mathbb{Q}}(G_{\theta_0}(\cdot))=\mathbf{J}_{\theta}G_{\theta_0}(\cdot)u_0^*+r(\cdot) ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( ⋅ )) = J θ G θ 0 ( ⋅ ) u 0 ∗ + r ( ⋅ ) ∥ r ∥ L 2 ( ρ ) 2 ≤ R 2 ∥ ∇ f ℓ 0 , P 0 , Q ∥ L 2 d ( P 0 ) 2 \|r\|_{L_2(\rho)}^2\leq\mathfrak{R}^2\|\nabla f_{\ell_0,\mathbb{P}_0,\mathbb{Q}}\|_{L^d_2(\mathbb{P}_0)}^2 ∥ r ∥ L 2 ( ρ ) 2 ≤ R 2 ∥∇ f ℓ 0 , P 0 , Q ∥ L 2 d ( P 0 ) 2 Δ θ 0 P G D \Delta\theta_0^{\mathrm{PGD}} Δ θ 0 PGD (38) ,
Δ θ 0 P G D = { E z ∼ ρ [ J θ G θ 0 ( z ) T J θ G θ 0 ( z ) ] + λ I d } − 1 × E z ∼ ρ [ J θ G θ 0 ( z ) T ∇ f ℓ 0 , P θ 0 , Q ( G θ 0 ( z ) ) ] . \Delta\theta_0^{\mathrm{PGD}}=
\left\{\mathbb{E}_{z\sim\rho}\left[\mathbf{J}_{\theta}G_{\theta_0}(z)^{\mathsf{T}}\mathbf{J}_{\theta}G_{\theta_0}(z)\right]+\lambda\mathbf{I}_d\right\}^{-1}
\times\mathbb{E}_{z\sim\rho}\left[\mathbf{J}_{\theta}G_{\theta_0}(z)^{\mathsf{T}}\nabla f_{\ell_0,\mathbb{P}_{\theta_0},\mathbb{Q}}(G_{\theta_0}(z))\right]. Δ θ 0 PGD = { E z ∼ ρ [ J θ G θ 0 ( z ) T J θ G θ 0 ( z ) ] + λ I d } − 1 × E z ∼ ρ [ J θ G θ 0 ( z ) T ∇ f ℓ 0 , P θ 0 , Q ( G θ 0 ( z )) ] . From Theorem C.3, we have that
J ( Δ θ P G D ) = E z ∼ ρ [ ∥ ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z ) ) − J θ G θ 0 ( z ) Δ θ P G D ∥ 2 ] ≤ λ ∥ u 0 ∗ ∥ 2 + R 2 ∥ ∇ f ℓ 0 , P 0 , Q ∥ L 2 d ( P 0 ) 2 ≤ λ U 2 + R 2 ∥ ∇ f ℓ 0 , P 0 , Q ∥ L 2 d ( P 0 ) 2 . \begin{aligned}
\mathcal{J}(\Delta\theta^{\mathrm{PGD}})&=\mathbb{E}_{z\sim\rho}\left[\left\|\nabla f_{\ell_0,\mathbb{P}_0,\mathbb{Q}}(G_{\theta_0}(z))-\mathbf{J}_{\theta}G_{\theta_0}(z)\Delta\theta^{\mathrm{PGD}}\right\|^2\right]\\
&\leq\lambda\|u_0^*\|^2+\mathfrak{R}^2\|\nabla f_{\ell_0,\mathbb{P}_0,\mathbb{Q}}\|_{L^d_2(\mathbb{P}_0)}^2\\
&\leq\lambda\mathcal{U}^2+\mathfrak{R}^2\|\nabla f_{\ell_0,\mathbb{P}_0,\mathbb{Q}}\|_{L^d_2(\mathbb{P}_0)}^2.
\end{aligned} J ( Δ θ PGD ) = E z ∼ ρ [ ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z )) − J θ G θ 0 ( z ) Δ θ PGD 2 ] ≤ λ ∥ u 0 ∗ ∥ 2 + R 2 ∥∇ f ℓ 0 , P 0 , Q ∥ L 2 d ( P 0 ) 2 ≤ λ U 2 + R 2 ∥∇ f ℓ 0 , P 0 , Q ∥ L 2 d ( P 0 ) 2 . We also have that,
∥ Δ θ P G D ∥ 2 ≤ 1 λ 2 ∥ E z ∼ ρ [ J θ G θ 0 ( z ) T ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z ) ) ] ∥ 2 ≤ C k p λ 2 ℓ 0 − 2 . \left\|\Delta^{\theta\mathrm{PGD}}\right\|^{2} \leq\frac{1}{\lambda^{2}} \left\|\mathbb{E}_{z\sim\rho}\left[\mathcal{J}_{\theta}G_{\theta_{0}}(z)^{\mathsf{T}}\nabla f_{\ell_{0},\mathbb{P}_{0},\mathbb{Q}}(G_{\theta_{0}}(z))\right]\right\|^{2} \leq\frac{C_{k}p}{\lambda^{2}}\ell_{0}^{-2}. Δ θ PGD 2 ≤ λ 2 1 E z ∼ ρ [ J θ G θ 0 ( z ) T ∇ f ℓ 0 , P 0 , Q ( G θ 0 ( z )) ] 2 ≤ λ 2 C k p ℓ 0 − 2 . Denote P θ 1 P G D : = ( G θ 0 − γ Δ θ P G D ) # ρ \mathbb{P}_{\theta_{1}^{\mathrm{PGD}}}:=(G_{\theta_{0}-\gamma\Delta^{\theta\mathrm{PGD}}})_{\#}\rho P θ 1 PGD := ( G θ 0 − γ Δ θ PGD ) # ρ Δ θ P G D \Delta^{\theta\mathrm{PGD}} Δ θ PGD
MMD ℓ 2 ( P θ 1 P G D , Q ) − MMD ℓ 2 ( P 1 , Q ) ≤ 2 γ ∥ ∇ f ℓ , P 0 , Q ∥ L 2 d ( P 0 ) ⋅ ( λ U 2 + R 2 ∥ ∇ f ℓ 0 , P 0 , Q ∥ L 2 d ( P 0 ) 2 ) + C k γ 2 ( p d ( ℓ − 2 + ℓ − 1 ) ∥ Δ θ P G D ∥ 2 + ℓ − 2 ℓ 0 − 2 ) (152) \begin{aligned}
\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta_{1}^{\mathrm{PGD}}},\mathbb{Q})-\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{1},\mathbb{Q})
&\leq2\gamma\|\nabla f_{\ell,\mathbb{P}_{0},\mathbb{Q}}\|_{L_{2}^{d}(\mathbb{P}_{0})}\cdot\sqrt{\left(\lambda\mathcal{U}^{2}+\mathcal{R}^{2}\|\nabla f_{\ell_{0},\mathbb{P}_{0},\mathbb{Q}}\|_{L_{2}^{d}(\mathbb{P}_{0})}^{2}\right)} \\
&\quad+ C_{k}\gamma^{2}\left(p\sqrt{d}(\ell^{-2}+\ell^{-1})\|\Delta^{\theta\mathrm{PGD}}\|^{2}+\ell^{-2}\ell_{0}^{-2}\right) \tag*{(152)}
\end{aligned} MMD ℓ 2 ( P θ 1 PGD , Q ) − MMD ℓ 2 ( P 1 , Q ) ≤ 2 γ ∥∇ f ℓ , P 0 , Q ∥ L 2 d ( P 0 ) ⋅ ( λ U 2 + R 2 ∥∇ f ℓ 0 , P 0 , Q ∥ L 2 d ( P 0 ) 2 ) + C k γ 2 ( p d ( ℓ − 2 + ℓ − 1 ) ∥ Δ θ PGD ∥ 2 + ℓ − 2 ℓ 0 − 2 ) (152) ≤ 2 γ R ∥ ∇ f ℓ , P 0 , Q ∥ L 2 d ( P 0 ) ⋅ ∥ ∇ f ℓ 0 , P 0 , Q ∥ L 2 d ( P 0 ) + C k γ λ d ℓ − 1 U (Approximation error) + C k γ 2 ( p d ( ℓ − 2 + ℓ − 1 ) p λ 2 ℓ 0 − 2 + ℓ − 2 ℓ 0 − 2 ) . (Taylor-expansion error) \begin{aligned}
&\leq2\gamma\mathcal{R}\|\nabla f_{\ell,\mathbb{P}_{0},\mathbb{Q}}\|_{L_{2}^{d}(\mathbb{P}_{0})}\cdot\|\nabla f_{\ell_{0},\mathbb{P}_{0},\mathbb{Q}}\|_{L_{2}^{d}(\mathbb{P}_{0})}+C_{k}\gamma\sqrt{\lambda}\sqrt{d}\ell^{-1}\mathcal{U} \\
&\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\text{(Approximation error)} \\
&\quad+ C_{k}\gamma^{2}\left(p\sqrt{d}(\ell^{-2}+\ell^{-1})\frac{p}{\lambda^{2}}\ell_{0}^{-2}+\ell^{-2}\ell_{0}^{-2}\right).
\qquad\text{(Taylor-expansion error)}
\end{aligned} ≤ 2 γ R ∥∇ f ℓ , P 0 , Q ∥ L 2 d ( P 0 ) ⋅ ∥∇ f ℓ 0 , P 0 , Q ∥ L 2 d ( P 0 ) + C k γ λ d ℓ − 1 U (Approximation error) + C k γ 2 ( p d ( ℓ − 2 + ℓ − 1 ) λ 2 p ℓ 0 − 2 + ℓ − 2 ℓ 0 − 2 ) . (Taylor-expansion error) The last step holds by a + b ≤ a + b \sqrt{a+b}\leq\sqrt{a}+\sqrt{b} a + b ≤ a + b sup z ∥ ∇ f ℓ , P 0 , Q ( z ) ∥ ≤ C k d ℓ − 1 \sup_{z}\|\nabla f_{\ell,\mathbb{P}_{0},\mathbb{Q}}(z)\|\leq C_{k}\sqrt{d}\ell^{-1} sup z ∥∇ f ℓ , P 0 , Q ( z ) ∥ ≤ C k d ℓ − 1 ∥ Δ θ P G D ∥ 2 \|\Delta^{\theta\mathrm{PGD}}\|^{2} ∥ Δ θ PGD ∥ 2 λ \lambda λ λ \lambda λ λ ≍ ( γ p 2 ( 1 + ℓ − 1 ) ℓ 0 − 2 U − 1 ) 2 5 \lambda\asymp\left(\gamma p^{2}(1+\ell^{-1})\ell_{0}^{-2}\mathcal{U}^{-1}\right)^{\frac{2}{5}} λ ≍ ( γ p 2 ( 1 + ℓ − 1 ) ℓ 0 − 2 U − 1 ) 5 2
MMD ℓ 2 ( P θ 1 P G D , Q ) − MMD ℓ 2 ( P 1 , Q ) ≤ 2 γ R ∥ ∇ f ℓ , P 0 , Q ∥ L 2 d ( P 0 ) ⋅ ∥ ∇ f ℓ 0 , P 0 , Q ∥ L 2 d ( P 0 ) + C k d 1 2 p 2 5 γ 6 5 U 4 5 ℓ − 1 ( ( 1 + ℓ − 1 ) 1 5 ℓ 0 − 2 5 + C k γ 2 5 ℓ − 2 ℓ 0 − 2 ) . \begin{aligned}
\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta_{1}^{\mathrm{PGD}}},\mathbb{Q})-\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{1},\mathbb{Q})
&\leq2\gamma\mathcal{R}\|\nabla f_{\ell,\mathbb{P}_{0},\mathbb{Q}}\|_{L_{2}^{d}(\mathbb{P}_{0})}\cdot\|\nabla f_{\ell_{0},\mathbb{P}_{0},\mathbb{Q}}\|_{L_{2}^{d}(\mathbb{P}_{0})} \\
&\quad+ C_{k}d^{\frac{1}{2}}p^{\frac{2}{5}}\gamma^{\frac{6}{5}}\mathcal{U}^{\frac{4}{5}}\ell^{-1}\left((1+\ell^{-1})^{\frac{1}{5}}\ell_{0}^{-\frac{2}{5}}+C_{k}\gamma^{\frac{2}{5}}\ell^{-2}\ell_{0}^{-2}\right).
\end{aligned} MMD ℓ 2 ( P θ 1 PGD , Q ) − MMD ℓ 2 ( P 1 , Q ) ≤ 2 γ R ∥∇ f ℓ , P 0 , Q ∥ L 2 d ( P 0 ) ⋅ ∥∇ f ℓ 0 , P 0 , Q ∥ L 2 d ( P 0 ) + C k d 2 1 p 5 2 γ 5 6 U 5 4 ℓ − 1 ( ( 1 + ℓ − 1 ) 5 1 ℓ 0 − 5 2 + C k γ 5 2 ℓ − 2 ℓ 0 − 2 ) . Then, we use Theorem C.11 which shows that ∥ ∇ f ℓ 0 , P 0 , Q ( x ) − ∇ f ℓ , P 0 , Q ( x ) ∥ ≤ C k ℓ 0 2 − ℓ 2 ℓ 3 \|\nabla f_{\ell_{0},\mathbb{P}_{0},\mathbb{Q}}(x)-\nabla f_{\ell,\mathbb{P}_{0},\mathbb{Q}}(x)\|\leq C_{k}\frac{\ell_{0}^{2}-\ell^{2}}{\ell^{3}} ∥∇ f ℓ 0 , P 0 , Q ( x ) − ∇ f ℓ , P 0 , Q ( x ) ∥ ≤ C k ℓ 3 ℓ 0 2 − ℓ 2
MMD ℓ 2 ( P θ 1 P G D , Q ) − MMD ℓ 2 ( P 1 , Q ) ≤ 2 γ R ∥ ∇ f ℓ , P 0 , Q ∥ L 2 d ( P 0 ) 2 + C k γ R ∥ ∇ f ℓ , P 0 , Q ∥ L 2 d ( P 0 ) ⋅ ℓ 0 2 − ℓ 2 ℓ 3 + C k d 1 2 p 2 5 γ 6 5 U 4 5 ℓ − 1 ( ( 1 + ℓ − 1 ) 1 5 ℓ 0 − 2 5 + C k γ 2 5 ℓ − 2 ℓ 0 − 2 ) ≤ 2 γ R ∥ ∇ f ℓ , P 0 , Q ∥ L 2 d ( P 0 ) 2 + C k γ R ℓ 0 2 − ℓ 2 ℓ 4 . + C k d 1 2 p 2 5 γ 6 5 U 4 5 ℓ − 1 ( ( 1 + ℓ − 1 ) 1 5 ℓ 0 − 2 5 + C k γ 2 ℓ − 2 ℓ 0 − 2 . \begin{aligned}
\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta_{1}^{\mathrm{PGD}}},\mathbb{Q})-\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{1},\mathbb{Q})
&\leq2\gamma\mathcal{R}\|\nabla f_{\ell,\mathbb{P}_{0},\mathbb{Q}}\|_{L_{2}^{d}(\mathbb{P}_{0})}^{2} \\
&\quad+ C_{k}\gamma\mathcal{R}\|\nabla f_{\ell,\mathbb{P}_{0},\mathbb{Q}}\|_{L_{2}^{d}(\mathbb{P}_{0})}\cdot\frac{\ell_{0}^{2}-\ell^{2}}{\ell^{3}} \\
&\quad+ C_{k}d^{\frac{1}{2}}p^{\frac{2}{5}}\gamma^{\frac{6}{5}}\mathcal{U}^{\frac{4}{5}}\ell^{-1}\left((1+\ell^{-1})^{\frac{1}{5}}\ell_{0}^{-\frac{2}{5}}+C_{k}\gamma^{\frac{2}{5}}\ell^{-2}\ell_{0}^{-2}\right) \\
&\leq2\gamma\mathcal{R}\|\nabla f_{\ell,\mathbb{P}_{0},\mathbb{Q}}\|_{L_{2}^{d}(\mathbb{P}_{0})}^{2}+C_{k}\gamma\mathcal{R}\frac{\ell_{0}^{2}-\ell^{2}}{\ell^{4}}. \\
&+ C_k d^{\frac{1}{2}}p^{\frac{2}{5}}\gamma^{\frac{6}{5}}\mathcal{U}^{\frac{4}{5}}\ell^{-1}((1+\ell^{-1})^{\frac{1}{5}}\ell_0^{-\frac{2}{5}}+C_k\gamma^2\ell^{-2}\ell_0^{-2}.
\end{aligned} MMD ℓ 2 ( P θ 1 PGD , Q ) − MMD ℓ 2 ( P 1 , Q ) ≤ 2 γ R ∥∇ f ℓ , P 0 , Q ∥ L 2 d ( P 0 ) 2 + C k γ R ∥∇ f ℓ , P 0 , Q ∥ L 2 d ( P 0 ) ⋅ ℓ 3 ℓ 0 2 − ℓ 2 + C k d 2 1 p 5 2 γ 5 6 U 5 4 ℓ − 1 ( ( 1 + ℓ − 1 ) 5 1 ℓ 0 − 5 2 + C k γ 5 2 ℓ − 2 ℓ 0 − 2 ) ≤ 2 γ R ∥∇ f ℓ , P 0 , Q ∥ L 2 d ( P 0 ) 2 + C k γ R ℓ 4 ℓ 0 2 − ℓ 2 . + C k d 2 1 p 5 2 γ 5 6 U 5 4 ℓ − 1 (( 1 + ℓ − 1 ) 5 1 ℓ 0 − 5 2 + C k γ 2 ℓ − 2 ℓ 0 − 2 . This concludes the proof.
Lemma C.3 (Residual error). Let θ ∈ Θ \theta\in\Theta θ ∈ Θ ℓ > 0 \ell> 0 ℓ > 0 u † ∈ R p u^\dagger\in\mathbb{R}^p u † ∈ R p r ∈ L 2 d ( ρ ) r \in L_2^d(\rho) r ∈ L 2 d ( ρ )
∇ f ℓ , P , Q ( G θ ( z ) ) = J θ G θ ( z ) u † + r ( z ) , ρ -a.e. z ∈ Z . \nabla f_{\ell,P,Q}(G_\theta(z)) = J_\theta G_\theta(z)u^\dagger+ r(z), \qquad\rho\text{-a.e. } z \in\mathcal{Z}. ∇ f ℓ , P , Q ( G θ ( z )) = J θ G θ ( z ) u † + r ( z ) , ρ -a.e. z ∈ Z . Then, for Δ θ P G D \Delta\theta^{\mathrm{PGD}} Δ θ PGD (38) ,
Δ θ P G D = { E z ∼ ρ [ J θ G θ ( z ) T J θ G θ ( z ) ] + λ I d } − 1 E z ∼ ρ [ J θ G θ ( z ) T ∇ f ℓ , P , Q ( G θ ( z ) ) ] , \Delta\theta^{\mathrm{PGD}}=\left\{\mathbb{E}_{z\sim\rho}\left[J_\theta G_\theta(z)^\mathsf{T}J_\theta G_\theta(z)\right]+\lambda I_d\right\}^{-1}\mathbb{E}_{z\sim\rho}\left[J_\theta G_\theta(z)^\mathsf{T}\nabla f_{\ell,P,Q}(G_\theta(z))\right], Δ θ PGD = { E z ∼ ρ [ J θ G θ ( z ) T J θ G θ ( z ) ] + λ I d } − 1 E z ∼ ρ [ J θ G θ ( z ) T ∇ f ℓ , P , Q ( G θ ( z )) ] , there holds
E z ∼ ρ [ ∥ ∇ f ℓ , P , Q ( G θ ( z ) ) − J θ G θ ( z ) Δ θ P G D ∥ 2 ] ≤ E z ∼ ρ [ ∥ r ( z ) ∥ 2 ] + λ ∥ u † ∥ 2 . \mathbb{E}_{z\sim\rho}\left[\left\|\nabla f_{\ell,P,Q}(G_\theta(z))-J_\theta G_\theta(z)\Delta\theta^{\mathrm{PGD}}\right\|^2\right]\leq\mathbb{E}_{z\sim\rho}\left[\|r(z)\|^2\right]+\lambda\|u^\dagger\|^2. E z ∼ ρ [ ∇ f ℓ , P , Q ( G θ ( z )) − J θ G θ ( z ) Δ θ PGD 2 ] ≤ E z ∼ ρ [ ∥ r ( z ) ∥ 2 ] + λ ∥ u † ∥ 2 . Proof. Define an operator A : R p → L 2 d ( ρ ) A:\mathbb{R}^p\to L_2^d(\rho) A : R p → L 2 d ( ρ ) ( A u ) ( z ) : = J θ G θ ( z ) u (Au)(z):=J_\theta G_\theta(z)u ( A u ) ( z ) := J θ G θ ( z ) u A ∗ : L 2 d ( ρ ) → R p A^*:L_2^d(\rho)\to\mathbb{R}^p A ∗ : L 2 d ( ρ ) → R p A ∗ h = E z ∼ ρ [ J θ G θ ( z ) T h ( z ) ] A^*h=\mathbb{E}_{z\sim\rho}[J_\theta G_\theta(z)^\mathsf{T}h(z)] A ∗ h = E z ∼ ρ [ J θ G θ ( z ) T h ( z )] A A A A ∗ A^* A ∗ A ∗ A = E z ∼ ρ [ J θ G θ ( z ) T J θ G θ ( z ) ] ∈ R p × p A^*A=\mathbb{E}_{z\sim\rho}[J_\theta G_\theta(z)^\mathsf{T}J_\theta G_\theta(z)]\in\mathbb{R}^{p\times p} A ∗ A = E z ∼ ρ [ J θ G θ ( z ) T J θ G θ ( z )] ∈ R p × p
In the following proof, write f : = f ℓ , P , Q f:=f_{\ell,P,Q} f := f ℓ , P , Q g : = ∇ f ∘ G θ ∈ L 2 d ( ρ ) g:=\nabla f\circ G_\theta\in L_2^d(\rho) g := ∇ f ∘ G θ ∈ L 2 d ( ρ ) g = A u † + r g=Au^\dagger+r g = A u † + r Δ θ P G D = ( A ∗ A + λ I ) − 1 A ∗ g \Delta\theta^{\mathrm{PGD}}=(A^*A+\lambda I)^{-1}A^*g Δ θ PGD = ( A ∗ A + λ I ) − 1 A ∗ g Δ θ P G D \Delta\theta^{\mathrm{PGD}} Δ θ PGD
u ↦ ∥ g − A u ∥ L 2 d ( ρ ) 2 + λ ∥ u ∥ 2 . u\mapsto\|g-Au\|_{L_2^d(\rho)}^2+\lambda\|u\|^2. u ↦ ∥ g − A u ∥ L 2 d ( ρ ) 2 + λ ∥ u ∥ 2 . Therefore, by optimality of Δ θ P G D \Delta\theta^{\mathrm{PGD}} Δ θ PGD u = u † u=u^\dagger u = u †
∥ g − A Δ θ P G D ∥ L 2 d ( ρ ) 2 + λ ∥ Δ θ P G D ∥ 2 ≤ ∥ g − A u † ∥ L 2 d ( ρ ) 2 + λ ∥ u † ∥ 2 . \|g-A\Delta\theta^{\mathrm{PGD}}\|_{L_2^d(\rho)}^2+\lambda\|\Delta\theta^{\mathrm{PGD}}\|^2\leq\|g-Au^\dagger\|_{L_2^d(\rho)}^2+\lambda\|u^\dagger\|^2. ∥ g − A Δ θ PGD ∥ L 2 d ( ρ ) 2 + λ ∥Δ θ PGD ∥ 2 ≤ ∥ g − A u † ∥ L 2 d ( ρ ) 2 + λ ∥ u † ∥ 2 . Since g − A u † = r g-Au^\dagger=r g − A u † = r
∥ g − A Δ θ P G D ∥ L 2 d ( ρ ) 2 + λ ∥ Δ θ P G D ∥ 2 ≤ ∥ r ∥ L 2 d ( ρ ) 2 + λ ∥ u † ∥ 2 . \left\lVert g-A\Delta\theta^{\mathrm{PGD}}\right\rVert_{L_2^d(\rho)}^2+\lambda\left\lVert\Delta\theta^{\mathrm{PGD}}\right\rVert^2\leq\left\lVert r\right\rVert_{L_2^d(\rho)}^2+\lambda\left\lVert u^\dagger\right\rVert^2. g − A Δ θ PGD L 2 d ( ρ ) 2 + λ Δ θ PGD 2 ≤ ∥ r ∥ L 2 d ( ρ ) 2 + λ u † 2 . Dropping the nonnegative term λ ∥ Δ θ P G D ∥ 2 \lambda\left\lVert\Delta\theta^{\mathrm{PGD}}\right\rVert^2 λ Δ θ PGD 2
∥ g − A Δ θ P G D ∥ L 2 d ( ρ ) 2 ≤ ∥ r ∥ L 2 d ( ρ ) 2 + λ ∥ u † ∥ 2 . \left\lVert g-A\Delta\theta^{\mathrm{PGD}}\right\rVert_{L_2^d(\rho)}^2\leq\left\lVert r\right\rVert_{L_2^d(\rho)}^2+\lambda\left\lVert u^\dagger\right\rVert^2. g − A Δ θ PGD L 2 d ( ρ ) 2 ≤ ∥ r ∥ L 2 d ( ρ ) 2 + λ u † 2 . Rewriting this in the original notation gives
E z ∼ ρ [ ∥ ∇ f ℓ , P , Q ( G θ ( z ) ) − J θ G θ ( z ) Δ θ P G D ∥ 2 ] ≤ E z ∼ ρ [ ∥ r ( z ) ∥ 2 ] + λ ∥ u † ∥ 2 . \mathbb{E}_{z\sim\rho}\left[\left\lVert\nabla f_{\ell,\mathsf{P},\mathsf{Q}}(G_\theta(z))-J_\theta G_\theta(z)\Delta\theta^{\mathrm{PGD}}\right\rVert^2\right]\leq\mathbb{E}_{z\sim\rho}\left[\left\lVert r(z)\right\rVert^2\right]+\lambda\left\lVert u^\dagger\right\rVert^2. E z ∼ ρ [ ∇ f ℓ , P , Q ( G θ ( z )) − J θ G θ ( z ) Δ θ PGD 2 ] ≤ E z ∼ ρ [ ∥ r ( z ) ∥ 2 ] + λ u † 2 . This completes the proof. □
Proposition C.5 (Closeness of P θ 1 P G D \mathsf{P}_{\theta_1}^{\mathrm{PGD}} P θ 1 PGD P θ 1 , m P G D \mathsf{P}_{\theta_1,m}^{\mathrm{PGD}} P θ 1 , m PGD γ > 0 \gamma>0 γ > 0 λ > 0 \lambda>0 λ > 0 ℓ > 0 \ell>0 ℓ > 0 ℓ 0 > 0 \ell_0>0 ℓ 0 > 0 θ 0 ∈ Θ \theta_0\in\Theta θ 0 ∈ Θ P θ 1 P G D = ( G θ 0 − γ Δ θ P G D ) # ρ \mathsf{P}_{\theta_1}^{\mathrm{PGD}}=(G_{\theta_0-\gamma\Delta\theta^{\mathrm{PGD}}})_\#\rho P θ 1 PGD = ( G θ 0 − γ Δ θ PGD ) # ρ (38) , and let P θ 1 , m P G D = ( G θ 0 − γ Δ θ m P G D ) # ρ \mathsf{P}_{\theta_1,m}^{\mathrm{PGD}}=(G_{\theta_0-\gamma\Delta\theta_m^{\mathrm{PGD}}})_\#\rho P θ 1 , m PGD = ( G θ 0 − γ Δ θ m PGD ) # ρ (40) . Then
E m [ ∣ MMD ℓ 2 ( P θ 1 , m P G D , Q ) − MMD ℓ 2 ( P θ 1 P G D , Q ) ∣ ] ≤ C k γ λ − 1 p ℓ − 1 ℓ 0 − 1 d ⋅ 1 m . \mathbb{E}_m\left[\left|\operatorname{MMD}_{\ell}^2\left(\mathsf{P}_{\theta_1,m}^{\mathrm{PGD}},\mathsf{Q}\right)-\operatorname{MMD}_{\ell}^2\left(\mathsf{P}_{\theta_1}^{\mathrm{PGD}},\mathsf{Q}\right)\right|\right]\leq C_k\gamma\lambda^{-1}p\ell^{-1}\ell_0^{-1}\sqrt{d}\cdot\frac{1}{\sqrt{m}}. E m [ MMD ℓ 2 ( P θ 1 , m PGD , Q ) − MMD ℓ 2 ( P θ 1 PGD , Q ) ] ≤ C k γ λ − 1 p ℓ − 1 ℓ 0 − 1 d ⋅ m 1 . Here, C k C_k C k
Proof. For the Gaussian kernel and for the Matérn kernel with smoothness s > 1 s > 1 s > 1 ∥ k ℓ ( x , ⋅ ) − k ℓ ( y , ⋅ ) ∥ H ℓ ≤ C k ℓ − 1 ∥ x − y ∥ \lVert k_{\ell}(x,\cdot)-k_{\ell}(y,\cdot)\rVert_{\mathcal{H}_{\ell}} \le C_k\ell^{-1}\lVert x-y\rVert ∥ k ℓ ( x , ⋅ ) − k ℓ ( y , ⋅ ) ∥ H ℓ ≤ C k ℓ − 1 ∥ x − y ∥
∣ MMD ℓ ( P θ 1 , m P G D , Q ) − MMD ℓ ( P θ 1 P G D , Q ) ∣ ≤ MMD ℓ ( P θ 1 , m P G D , P θ 1 P G D ) ≤ C k ℓ − 1 ∥ G θ 0 − γ Δ θ 0 P G D ( ⋅ ) − G θ 0 − γ Δ θ 0 , m P G D ( ⋅ ) ∥ L 2 ( ρ ) ≤ C k p ℓ − 1 γ ∥ Δ θ 0 P G D − Δ θ 0 , m P G D ∥ . \begin{aligned}
\left|\operatorname{MMD}_{\ell}(\mathbb{P}^{\mathrm{PGD}}_{\theta_{1,m}},\mathbb{Q})-\operatorname{MMD}_{\ell}(\mathbb{P}^{\mathrm{PGD}}_{\theta_{1}},\mathbb{Q})\right| \le\operatorname{MMD}_{\ell}(\mathbb{P}^{\mathrm{PGD}}_{\theta_{1,m}},\mathbb{P}^{\mathrm{PGD}}_{\theta_{1}}) \\
&\le C_k\ell^{-1}\left\lVert G_{\theta_0-\gamma\Delta\theta_0^{\mathrm{PGD}}}(\cdot)-G_{\theta_0-\gamma\Delta\theta_{0,m}^{\mathrm{PGD}}}(\cdot)\right\rVert_{L^2(\rho)} \le C_k\sqrt{p}\ell^{-1}\gamma\left\lVert\Delta\theta_0^{\mathrm{PGD}}-\Delta\theta_{0,m}^{\mathrm{PGD}}\right\rVert.
\end{aligned} MMD ℓ ( P θ 1 , m PGD , Q ) − MMD ℓ ( P θ 1 PGD , Q ) ≤ MMD ℓ ( P θ 1 , m PGD , P θ 1 PGD ) ≤ C k ℓ − 1 G θ 0 − γ Δ θ 0 PGD ( ⋅ ) − G θ 0 − γ Δ θ 0 , m PGD ( ⋅ ) L 2 ( ρ ) ≤ C k p ℓ − 1 γ Δ θ 0 PGD − Δ θ 0 , m PGD . The last step holds by ∥ G θ − G θ ′ ∥ L 2 ( ρ ) ≤ p ∥ θ − θ ′ ∥ \lVert G_{\theta}-G_{\theta'}\rVert_{L^2(\rho)} \le\sqrt{p}\lVert\theta-\theta'\rVert ∥ G θ − G θ ′ ∥ L 2 ( ρ ) ≤ p ∥ θ − θ ′ ∥
Δ θ 0 P G D = { E z ∼ ρ [ J θ G θ 0 ( z ) T J θ G θ 0 ( z ) ] + λ I d } − 1 E z ∼ ρ [ J θ G θ 0 ( z ) T ∇ f ℓ 0 , P θ 0 , Q ( G θ 0 ( z ) ) ] , \Delta\theta_0^{\mathrm{PGD}}=\left\{\mathbb{E}_{z\sim\rho}\left[\mathbf{J}_{\theta}G_{\theta_0}(z)^{\mathsf{T}}\mathbf{J}_{\theta}G_{\theta_0}(z)\right]+\lambda\mathop{\mathrm{Id}}\right\}^{-1}
\mathbb{E}_{z\sim\rho}\left[\mathbf{J}_{\theta}G_{\theta_0}(z)^{\mathsf{T}}\nabla f_{\ell_0,\mathbb{P}_{\theta_0},\mathbb{Q}}(G_{\theta_0}(z))\right], Δ θ 0 PGD = { E z ∼ ρ [ J θ G θ 0 ( z ) T J θ G θ 0 ( z ) ] + λ Id } − 1 E z ∼ ρ [ J θ G θ 0 ( z ) T ∇ f ℓ 0 , P θ 0 , Q ( G θ 0 ( z )) ] , \Delta\theta_{0,m}^{\mathrm{PGD}}=\left\{\mathbb{E}_{z\sim\rho}\left[\mathbf{J}_{\theta}G_{\theta_0}(z)^{\mathsf{T}}\mathbf{J}_{\theta}G_{\theta_0}(z)\right]+\lambda\mathop{\mathrm{Id}}\right\}^{-1}
\mathbb{E}_{z\sim\rho}\left[\mathbf{J}_{\theta}G_{\theta_0}(z)^{\mathsf{T}}\nabla f_{\ell_0,\mathbb{P}_{\theta_0},\widehat{\mathbb{Q}}_m(G_{\theta_0}(z))\right].}
So we have
∥ Δ θ 0 P G D − Δ θ 0 , m P G D ∥ ≤ λ − 1 E z ∼ ρ [ J θ G θ 0 ( z ) T ( ∇ f ℓ 0 , P θ 0 , Q ( G θ 0 ( z ) ) − ∇ f ℓ 0 , P θ 0 , Q ^ m ( G θ 0 ( z ) ) ) ] ≤ λ − 1 p E z ∼ ρ [ ∥ ∇ f ℓ 0 , P θ 0 , Q ( G θ 0 ( z ) ) − ∇ f ℓ 0 , P θ 0 , Q ^ m ( G θ 0 ( z ) ) ∥ ] ≤ C k λ − 1 p ℓ 0 − 1 d MMD ℓ 0 ( Q , Q ^ m ) . \begin{aligned}
\left\lVert\Delta\theta_0^{\mathrm{PGD}}-\Delta\theta_{0,m}^{\mathrm{PGD}}\right\rVert\le\lambda^{-1}\mathbb{E}_{z\sim\rho}\left[
\mathbf{J}_{\theta}G_{\theta_0}(z)^{\mathsf{T}}
\left(\nabla f_{\ell_0,\mathbb{P}_{\theta_0},\mathbb{Q}}(G_{\theta_0}(z))
-\nabla f_{\ell_0,\mathbb{P}_{\theta_0},\widehat{\mathbb{Q}}_m}(G_{\theta_0}(z))\right)
\right] \\
&\le\lambda^{-1}\sqrt{p}\mathbb{E}_{z\sim\rho}\left[
\left\lVert
\begin{array}{c}
\nabla f_{\ell_0,\mathbb{P}_{\theta_0},\mathbb{Q}}(G_{\theta_0}(z))\\
-\nabla f_{\ell_0,\mathbb{P}_{\theta_0},\widehat{\mathbb{Q}}_m}(G_{\theta_0}(z))
\end{array}
\right\rVert
\right] \\
&\le C_k\lambda^{-1}\sqrt{p}\ell_0^{-1}\sqrt{d}\operatorname{MMD}_{\ell_0}(\mathbb{Q},\widehat{\mathbb{Q}}_m).
\end{aligned} Δ θ 0 PGD − Δ θ 0 , m PGD ≤ λ − 1 E z ∼ ρ [ J θ G θ 0 ( z ) T ( ∇ f ℓ 0 , P θ 0 , Q ( G θ 0 ( z )) − ∇ f ℓ 0 , P θ 0 , Q m ( G θ 0 ( z )) ) ] ≤ λ − 1 p E z ∼ ρ [ ∇ f ℓ 0 , P θ 0 , Q ( G θ 0 ( z )) − ∇ f ℓ 0 , P θ 0 , Q m ( G θ 0 ( z )) ] ≤ C k λ − 1 p ℓ 0 − 1 d MMD ℓ 0 ( Q , Q m ) . The second last step holds by Assumption 5.3; and the last step holds by the following: for each coordinate j ∈ { 1 , … , d } j\in\{1,\ldots,d\} j ∈ { 1 , … , d } x ∈ R d x\in\mathbb{R}^d x ∈ R d
∣ ∫ ∂ 1 , j k ℓ 0 ( x , y ) d ( Q ^ m − Q ) ( y ) ∣ ≤ ∥ ∂ 1 , j k ℓ 0 ( x , ⋅ ) ∥ H ℓ 0 MMD ℓ 0 ( Q ^ m , Q ) ≤ C k ℓ 0 − 1 MMD ℓ 0 ( Q ^ m , Q ) . \begin{aligned}
\left|\int\partial_{1,j}k_{\ell_0}(x,y)\mathop{}\!d(\widehat{\mathbb{Q}}_m-\mathbb{Q})(y)\right| \le\lVert\partial_{1,j}k_{\ell_0}(x,\cdot)\rVert_{\mathcal{H}_{\ell_0}}\operatorname{MMD}_{\ell_0}(\widehat{\mathbb{Q}}_m,\mathbb{Q}) \\
&\le C_k\ell_0^{-1}\operatorname{MMD}_{\ell_0}(\widehat{\mathbb{Q}}_m,\mathbb{Q}).
\end{aligned} ∫ ∂ 1 , j k ℓ 0 ( x , y ) d ( Q m − Q ) ( y ) ≤ ∥ ∂ 1 , j k ℓ 0 ( x , ⋅ ) ∥ H ℓ 0 MMD ℓ 0 ( Q m , Q ) ≤ C k ℓ 0 − 1 MMD ℓ 0 ( Q m , Q ) . Since the kernel is bounded by 1, the standard concentration bound gives E m [ MMD ℓ 0 ( Q ^ m , Q ) ] ≤ C k m − 1 / 2 \mathbb{E}_m[\operatorname{MMD}_{\ell_0}(\widehat{\mathbb{Q}}_m,\mathbb{Q})]\le C_km^{-1/2} E m [ MMD ℓ 0 ( Q m , Q )] ≤ C k m − 1/2
E m [ ∣ MMD ℓ 2 ( P θ 1 , m P G D , Q ) − MMD ℓ 2 ( P θ 1 P G D , Q ) ∣ ] ≤ 4 E m [ ∣ MMD ℓ ( P θ 1 , m P G D , Q ) − MMD ℓ ( P θ 1 P G D , Q ) ∣ ] ≤ C k γ λ − 1 p ℓ 0 − 1 d ⋅ 1 m . \mathbb{E}_m\left[\left|\operatorname{MMD}_{\ell}^{2}\left(\mathbb{P}_{\theta_{1,m}}^{\mathrm{PGD}},\mathbb{Q}\right)-\operatorname{MMD}_{\ell}^{2}\left(\mathbb{P}_{\theta_{1}^{\mathrm{PGD}}},\mathbb{Q}\right)\right|\right]
\leq4\mathbb{E}_m\left[\left|\operatorname{MMD}_{\ell}\left(\mathbb{P}_{\theta_{1,m}}^{\mathrm{PGD}},\mathbb{Q}\right)-\operatorname{MMD}_{\ell}\left(\mathbb{P}_{\theta_{1}^{\mathrm{PGD}}},\mathbb{Q}\right)\right|\right]\leq C_k\gamma\lambda^{-1}p\ell_0^{-1}\sqrt{d}\cdot\frac{1}{\sqrt{m}}. E m [ MMD ℓ 2 ( P θ 1 , m PGD , Q ) − MMD ℓ 2 ( P θ 1 PGD , Q ) ] ≤ 4 E m [ MMD ℓ ( P θ 1 , m PGD , Q ) − MMD ℓ ( P θ 1 PGD , Q ) ] ≤ C k γ λ − 1 p ℓ 0 − 1 d ⋅ m 1 . □ \square □
Connection with natural gradient
Natural gradient descent is a popular optimization method in probabilistic machine learning as an alternative to standard Euclidean gradient descent (Amari, 1998; Hoffman et al., 2013; Pascanu and Bengio, 2014; Martens, 2020). Let P θ \mathbb{P}_{\theta} P θ θ ∈ R p \theta\in\mathbb{R}^{p} θ ∈ R p γ > 0 \gamma> 0 γ > 0 g ∈ R p g \in\mathbb{R}^{p} g ∈ R p
h = arg min h ∈ R p − h T g + 1 2 γ KL ( P θ ∥ P θ + h ) . (153) h=\underset{h\in\mathbb{R}^{p}}{\arg\min}\ -h^{\mathsf{T}}g+\frac{1}{2\gamma}\operatorname{KL}\left(\mathbb{P}_{\theta}\middle\|\mathbb{P}_{\theta+h}\right). \tag*{(153)} h = h ∈ R p arg min − h T g + 2 γ 1 KL ( P θ ∥ P θ + h ) . (153) Under the well-known approximation of the KL divergence and the Fisher information matrix KL ( P θ ∥ P θ + h ) ≈ h T F h \operatorname{KL}\left(\mathbb{P}_{\theta}\middle\|\mathbb{P}_{\theta+h}\right)\approx h^{\mathsf{T}}Fh KL ( P θ ∥ P θ + h ) ≈ h T F h h = γ F − 1 g ∈ R p h=\gamma F^{-1}g\in\mathbb{R}^{p} h = γ F − 1 g ∈ R p
A straight-forward MMD analogue of the natural gradient descent would be defined as the following:
h = arg min h ∈ R p − h T g + 1 2 γ MMD ℓ 2 ( P θ , P θ + h ) . (154) h=\underset{h\in\mathbb{R}^{p}}{\arg\min}\ -h^{\mathsf{T}}g+\frac{1}{2\gamma}\operatorname{MMD}_{\ell}^{2}\left(\mathbb{P}_{\theta},\mathbb{P}_{\theta+h}\right). \tag*{(154)} h = h ∈ R p arg min − h T g + 2 γ 1 MMD ℓ 2 ( P θ , P θ + h ) . (154) For the Gaussian kernel and for the Matérn kernel with smoothness s > 1 s > 1 s > 1 h ↦ MMD ℓ 2 ( P θ , P θ + h ) h\mapsto\operatorname{MMD}_{\ell}^{2}\left(\mathbb{P}_{\theta},\mathbb{P}_{\theta+h}\right) h ↦ MMD ℓ 2 ( P θ , P θ + h ) h = 0 h=0 h = 0
MMD ℓ 2 ( P θ , P θ + h ) = 0 + h T d d h MMD ℓ 2 ( P θ , P θ + h ) ∣ h = 0 + 1 2 h T d 2 d h 2 MMD ℓ 2 ( P θ , P θ + h ) ∣ h = 0 h + o ( ∥ h ∥ 2 ) . \operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta},\mathbb{P}_{\theta+h}) = 0 + h^{\mathsf{T}} \left.\frac{\mathrm{d}}{\mathrm{d}h}\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta},\mathbb{P}_{\theta+h})\right|_{h=0} + \frac{1}{2}h^{\mathsf{T}}\left.\frac{\mathrm{d}^{2}}{\mathrm{d}h^{2}}\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta},\mathbb{P}_{\theta+h})\right|_{h=0}h + o(\lVert h\rVert^{2}). MMD ℓ 2 ( P θ , P θ + h ) = 0 + h T d h d MMD ℓ 2 ( P θ , P θ + h ) h = 0 + 2 1 h T d h 2 d 2 MMD ℓ 2 ( P θ , P θ + h ) h = 0 h + o (∥ h ∥ 2 ) . In the following derivation, we focus on the case where P θ = ( G θ ) # ρ \mathbb{P}_{\theta}=(G_{\theta})_{\#}\rho P θ = ( G θ ) # ρ ρ \rho ρ
d d h ∣ h = 0 1 2 MMD ℓ 2 ( P θ , P θ + h ) = 0. \left.\frac{\mathrm{d}}{\mathrm{d}h}\right|_{h=0}\frac{1}{2}\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta},\mathbb{P}_{\theta+h})=0. d h d h = 0 2 1 MMD ℓ 2 ( P θ , P θ + h ) = 0. And,
d 2 d h 2 ∣ h = 0 1 2 MMD ℓ 2 ( P θ , P θ + h ) = ∬ J θ G θ ( z ) T ∇ 1 ∇ 2 k ( G θ ( z ) , G θ ( z ′ ) ) J θ G θ ( z ′ ) d ρ ( z ) d ρ ( z ′ ) = E z , z ′ ∼ ρ [ J θ G θ ( z ) T ∇ 1 ∇ 2 k ℓ ( G θ ( z ) , G θ ( z ′ ) ) J θ G θ ( z ′ ) ] . \begin{aligned}
\left.\frac{\mathrm{d}^{2}}{\mathrm{d}h^{2}}\right|_{h=0}\frac{1}{2}\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta},\mathbb{P}_{\theta+h}) = \iint\mathbf{J}_{\theta}G_{\theta}(z)^{\mathsf{T}}\nabla_{1}\nabla_{2}k(G_{\theta}(z),G_{\theta}(z'))\mathbf{J}_{\theta}G_{\theta}(z')\,\mathrm{d}\rho(z)\mathrm{d}\rho(z') \\
&= \mathbb{E}_{z,z'\sim\rho}\left[\mathbf{J}_{\theta}G_{\theta}(z)^{\mathsf{T}}\nabla_{1}\nabla_{2}k_{\ell}(G_{\theta}(z),G_{\theta}(z'))\mathbf{J}_{\theta}G_{\theta}(z')\right].
\end{aligned} d h 2 d 2 h = 0 2 1 MMD ℓ 2 ( P θ , P θ + h ) = ∬ J θ G θ ( z ) T ∇ 1 ∇ 2 k ( G θ ( z ) , G θ ( z ′ )) J θ G θ ( z ′ ) d ρ ( z ) d ρ ( z ′ ) = E z , z ′ ∼ ρ [ J θ G θ ( z ) T ∇ 1 ∇ 2 k ℓ ( G θ ( z ) , G θ ( z ′ )) J θ G θ ( z ′ ) ] . Therefore, we have
MMD ℓ 2 ( P θ , P θ + h ) = h T { E z , z ′ ∼ ρ [ J θ G θ ( z ) T ∇ 1 ∇ 2 k ℓ ( G θ ( z ) , G θ ( z ′ ) ) J θ G θ ( z ′ ) ] } h + o ( ∥ h ∥ 2 ) . \operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta},\mathbb{P}_{\theta+h}) = h^{\mathsf{T}}\left\{\mathbb{E}_{z,z'\sim\rho}\left[\mathbf{J}_{\theta}G_{\theta}(z)^{\mathsf{T}}\nabla_{1}\nabla_{2}k_{\ell}(G_{\theta}(z),G_{\theta}(z'))\mathbf{J}_{\theta}G_{\theta}(z')\right]\right\}h + o(\lVert h\rVert^{2}). MMD ℓ 2 ( P θ , P θ + h ) = h T { E z , z ′ ∼ ρ [ J θ G θ ( z ) T ∇ 1 ∇ 2 k ℓ ( G θ ( z ) , G θ ( z ′ )) J θ G θ ( z ′ ) ] } h + o (∥ h ∥ 2 ) . The resulting minimizer of Eq. (C.24) thus admits a closed-form (Briol et al., 2019a)
h = γ { E z , z ′ ∼ ρ [ J θ G θ ( z ) T ∇ 1 ∇ 2 k ℓ ( G θ ( z ) , G θ ( z ′ ) ) J θ G θ ( z ′ ) ] } − 1 g , (155) h = \gamma\left\{\mathbb{E}_{z,z'\sim\rho}\left[\mathbf{J}_{\theta}G_{\theta}(z)^{\mathsf{T}}\nabla_{1}\nabla_{2}k_{\ell}(G_{\theta}(z),G_{\theta}(z'))\mathbf{J}_{\theta}G_{\theta}(z')\right]\right\}^{-1}g, \tag*{(155)} h = γ { E z , z ′ ∼ ρ [ J θ G θ ( z ) T ∇ 1 ∇ 2 k ℓ ( G θ ( z ) , G θ ( z ′ )) J θ G θ ( z ′ ) ] } − 1 g , (155) where g g g (38) in the main text, which we recall below:
h = γ { E z ∼ ρ [ J θ G θ t ( z ) ⊤ J θ G θ t ( z ) ] } − 1 g . h = \gamma\left\{\mathbb{E}_{z\sim\rho}\left[J_\theta G_{\theta_t}(z)^\top J_\theta G_{\theta_t}(z)\right]\right\}^{-1}g. h = γ { E z ∼ ρ [ J θ G θ t ( z ) ⊤ J θ G θ t ( z ) ] } − 1 g . Other preconditioning methods directly on P 2 ( R d ) \mathcal{P}_2(\mathbb{R}^d) P 2 ( R d )
Auxiliary Results
Lemma C.6. Let φ : [ 0 , ∞ ) → R \varphi: [0,\infty) \to\mathbb{R} φ : [ 0 , ∞ ) → R either the Gaussian profile φ ( r ) = exp ( − r 2 / 2 ) \varphi(r) = \exp(-r^2/2) φ ( r ) = exp ( − r 2 /2 ) or the Matérn profile with smoothness parameter s > 1 s > 1 s > 1 φ s ( r ) = 2 1 − s Γ ( s ) ( 2 s r ) s K s ( 2 s r ) \varphi_s(r) = \frac{2^{1-s}}{\Gamma(s)}(\sqrt{2sr})^s K_s(\sqrt{2sr}) φ s ( r ) = Γ ( s ) 2 1 − s ( 2 sr ) s K s ( 2 sr ) K s K_s K s a > 0 a > 0 a > 0 ψ a ( t ) : = φ ( ∥ t ∥ a ) \psi_a(t) := \varphi\left(\frac{\lVert t\rVert}{\sqrt{a}}\right) ψ a ( t ) := φ ( a ∥ t ∥ ) a 1 > a 2 > 0 a_1 > a_2 > 0 a 1 > a 2 > 0
sup t ∈ R d ∣ ψ a 1 ( t ) − ψ a 2 ( t ) ∣ ≤ C 0 ∣ a 1 − a 2 ∣ a 2 , \sup_{t\in\mathbb{R}^d}\lvert\psi_{a_1}(t)-\psi_{a_2}(t)\rvert\le C_0\frac{\lvert a_1-a_2\rvert}{a_2}, t ∈ R d sup ∣ ψ a 1 ( t ) − ψ a 2 ( t )∣ ≤ C 0 a 2 ∣ a 1 − a 2 ∣ , sup t ∈ R d ∥ ∇ ψ a 1 ( t ) − ∇ ψ a 2 ( t ) ∥ ≤ C 1 ∣ a 1 − a 2 ∣ a 2 3 / 2 . \sup_{t\in\mathbb{R}^d}\lVert\nabla\psi_{a_1}(t)-\nabla\psi_{a_2}(t)\rVert\le C_1\frac{\lvert a_1-a_2\rvert}{a_2^{3/2}}. t ∈ R d sup ∥ ∇ ψ a 1 ( t ) − ∇ ψ a 2 ( t )∥ ≤ C 1 a 2 3/2 ∣ a 1 − a 2 ∣ . Here, C 0 , C 1 C_0,C_1 C 0 , C 1
Proof. Let ϱ : = ∥ t ∥ a \varrho:= \frac{\lVert t\rVert}{\sqrt{a}} ϱ := a ∥ t ∥ ψ a ( t ) \psi_a(t) ψ a ( t ) a a a ∂ a ψ a ( t ) = − 1 2 a ϱ φ ′ ( ϱ ) \partial_a\psi_a(t) = -\frac{1}{2a}\varrho\varphi'(\varrho) ∂ a ψ a ( t ) = − 2 a 1 ϱ φ ′ ( ϱ ) ∣ ∂ a ψ a ( t ) ∣ ≤ 1 a ( 1 2 sup ϱ ≥ 0 ϱ ∣ φ ′ ( ϱ ) ∣ ) \lvert\partial_a\psi_a(t)\rvert\le\frac{1}{a}\left(\frac{1}{2}\sup_{\varrho\ge0}\varrho\lvert\varphi'(\varrho)\rvert\right) ∣ ∂ a ψ a ( t )∣ ≤ a 1 ( 2 1 sup ϱ ≥ 0 ϱ ∣ φ ′ ( ϱ )∣ )
C 0 : = 1 2 sup ϱ ≥ 0 ϱ ∣ φ ′ ( ϱ ) ∣ . C_0 := \frac{1}{2}\sup_{\varrho\ge0}\varrho\lvert\varphi'(\varrho)\rvert. C 0 := 2 1 ϱ ≥ 0 sup ϱ ∣ φ ′ ( ϱ )∣ . For the Gaussian profile, C 0 < ∞ C_0 < \infty C 0 < ∞ ϱ ∣ φ ′ ( ϱ ) ∣ = ϱ 2 e − ϱ 2 / 2 \varrho\lvert\varphi'(\varrho)\rvert= \varrho^2 e^{-\varrho^2/2} ϱ ∣ φ ′ ( ϱ )∣ = ϱ 2 e − ϱ 2 /2
For the Matérn profile with s > 1 s > 1 s > 1 K s K_s K s ϱ ↓ 0 \varrho\downarrow0 ϱ ↓ 0
φ s ( ϱ ) = 1 − s 2 ( s − 1 ) ϱ 2 + o ( ϱ 2 ) , \varphi_s(\varrho)=1-\frac{s}{2(s-1)}\varrho^2+o(\varrho^2), φ s ( ϱ ) = 1 − 2 ( s − 1 ) s ϱ 2 + o ( ϱ 2 ) , and hence ϱ ∣ φ s ′ ( ϱ ) ∣ = O ( ϱ 2 ) \varrho\lvert\varphi_s'(\varrho)\rvert= O(\varrho^2) ϱ ∣ φ s ′ ( ϱ )∣ = O ( ϱ 2 )
of K s K_s K s φ s ′ \varphi_s' φ s ′ ϱ ∣ φ s ′ ( ϱ ) ∣ \varrho|\varphi_s'(\varrho)| ϱ ∣ φ s ′ ( ϱ ) ∣ C 0 < ∞ C_0 < \infty C 0 < ∞ a 1 > a 2 > 0 a_1 > a_2 > 0 a 1 > a 2 > 0 ∣ ψ a 1 ( t ) − ψ a 2 ( t ) ∣ ≤ ∣ a 1 − a 2 ∣ sup a ∈ [ a 2 , a 1 ] ∣ ∂ a ψ a ( t ) ∣ |\psi_{a_1}(t) - \psi_{a_2}(t)| \le|a_1 - a_2| \sup_{a\in[a_2,a_1]}|\partial_a\psi_a(t)| ∣ ψ a 1 ( t ) − ψ a 2 ( t ) ∣ ≤ ∣ a 1 − a 2 ∣ sup a ∈ [ a 2 , a 1 ] ∣ ∂ a ψ a ( t ) ∣
∣ ψ a 1 ( t ) − ψ a 2 ( t ) ∣ ≤ C 0 ∣ a 1 − a 2 ∣ sup a ∈ [ a 2 , a 1 ] 1 a = C 0 ∣ a 1 − a 2 ∣ a 2 . |\psi_{a_1}(t) - \psi_{a_2}(t)| \le C_0|a_1-a_2| \sup_{a\in[a_2,a_1]} \frac{1}{a} = C_0 \frac{|a_1-a_2|}{a_2}. ∣ ψ a 1 ( t ) − ψ a 2 ( t ) ∣ ≤ C 0 ∣ a 1 − a 2 ∣ a ∈ [ a 2 , a 1 ] sup a 1 = C 0 a 2 ∣ a 1 − a 2 ∣ . Taking the supremum over t ∈ R d t \in\mathbb{R}^d t ∈ R d
We next control the gradient. For t ≠ 0 t \ne0 t = 0 e t : = t ∥ t ∥ e_t := \frac{t}{\|t\|} e t := ∥ t ∥ t ∇ t ψ a ( t ) = a − 1 / 2 φ ′ ( ϱ ) e t \nabla_t\psi_a(t) = a^{-1/2}\varphi'(\varrho)e_t ∇ t ψ a ( t ) = a − 1/2 φ ′ ( ϱ ) e t s > 1 s > 1 s > 1 φ ′ ( 0 ) = 0 \varphi'(0) = 0 φ ′ ( 0 ) = 0 t = 0 t = 0 t = 0 a a a e t e_t e t a a a
∂ a ( ∇ t ψ a ( t ) ) = − 1 2 a 3 / 2 φ ′ ( ϱ ) e t + a − 1 / 2 φ ′ ′ ( ϱ ) ( − ϱ 2 a ) e t . \partial_a(\nabla_t\psi_a(t)) = -\frac{1}{2a^{3/2}}\varphi'(\varrho)e_t + a^{-1/2}\varphi''(\varrho)\left(-\frac{\varrho}{2a}\right)e_t. ∂ a ( ∇ t ψ a ( t )) = − 2 a 3/2 1 φ ′ ( ϱ ) e t + a − 1/2 φ ′′ ( ϱ ) ( − 2 a ϱ ) e t . Hence, ∂ a ( ∇ t ψ a ( t ) ) = − 1 2 a 3 / 2 [ φ ′ ( ϱ ) + ϱ φ ′ ′ ( ϱ ) ] e t \partial_a(\nabla_t\psi_a(t)) = -\frac{1}{2a^{3/2}}[\varphi'(\varrho)+\varrho\varphi''(\varrho)]e_t ∂ a ( ∇ t ψ a ( t )) = − 2 a 3/2 1 [ φ ′ ( ϱ ) + ϱ φ ′′ ( ϱ )] e t ∥ ∂ a ( ∇ t ψ a ( t ) ) ∥ ≤ 1 a 3 / 2 ( 1 2 sup ϱ ≥ 0 ∣ φ ′ ( ϱ ) + ϱ φ ′ ′ ( ϱ ) ∣ ) \|\partial_a(\nabla_t\psi_a(t))\| \le\frac{1}{a^{3/2}}\left(\frac{1}{2}\sup_{\varrho\ge0}|\varphi'(\varrho)+\varrho\varphi''(\varrho)|\right) ∥ ∂ a ( ∇ t ψ a ( t )) ∥ ≤ a 3/2 1 ( 2 1 sup ϱ ≥ 0 ∣ φ ′ ( ϱ ) + ϱ φ ′′ ( ϱ ) ∣ )
C 1 : = 1 2 sup ϱ ≥ 0 ∣ φ ′ ( ϱ ) + ϱ φ ′ ′ ( ϱ ) ∣ . C_1 := \frac{1}{2}\sup_{\varrho\ge0}|\varphi'(\varrho)+\varrho\varphi''(\varrho)|. C 1 := 2 1 ϱ ≥ 0 sup ∣ φ ′ ( ϱ ) + ϱ φ ′′ ( ϱ ) ∣. For the Gaussian profile, φ ′ ( ϱ ) = − ϱ e − ϱ 2 / 2 \varphi'(\varrho)=-\varrho e^{-\varrho^2/2} φ ′ ( ϱ ) = − ϱ e − ϱ 2 /2 φ ′ ′ ( ϱ ) = ( ϱ 2 − 1 ) e − ϱ 2 / 2 \varphi''(\varrho)=(\varrho^2-1)e^{-\varrho^2/2} φ ′′ ( ϱ ) = ( ϱ 2 − 1 ) e − ϱ 2 /2 φ ′ ( ϱ ) + ϱ φ ′ ′ ( ϱ ) = ( ϱ 3 − 2 ϱ ) e − ϱ 2 / 2 \varphi'(\varrho)+\varrho\varphi''(\varrho)=(\varrho^3-2\varrho)e^{-\varrho^2/2} φ ′ ( ϱ ) + ϱ φ ′′ ( ϱ ) = ( ϱ 3 − 2 ϱ ) e − ϱ 2 /2 [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) C 1 < ∞ C_1 < \infty C 1 < ∞ s > 1 s > 1 s > 1 φ s ′ ( ϱ ) + ϱ φ s ′ ′ ( ϱ ) = O ( ϱ ) \varphi_s'(\varrho)+\varrho\varphi_s''(\varrho)=O(\varrho) φ s ′ ( ϱ ) + ϱ φ s ′′ ( ϱ ) = O ( ϱ ) ϱ ↓ 0 \varrho\downarrow0 ϱ ↓ 0 ϱ ≈ ∞ \varrho\approx\infty ϱ ≈ ∞ K s K_s K s φ s \varphi_s φ s sup ϱ ≥ 0 ∣ φ s ′ ( ϱ ) + ϱ φ s ′ ′ ( ϱ ) ∣ < ∞ \sup_{\varrho\ge0}|\varphi_s'(\varrho)+\varrho\varphi_s''(\varrho)| < \infty sup ϱ ≥ 0 ∣ φ s ′ ( ϱ ) + ϱ φ s ′′ ( ϱ ) ∣ < ∞ C 1 < ∞ C_1 < \infty C 1 < ∞ s > 1 s > 1 s > 1
Lemma C.7 (Strong convexity implies gradient dominance). Suppose that
Figure C.1. The example of Q = Unif ( S d − 1 ) \mathbb{Q} = \operatorname{Unif}(\mathbb{S}^{d-1}) Q = Unif ( S d − 1 )
θ ↦ MMD ℓ 2 ( P θ , Q ) \theta\mapsto\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta},\mathbb{Q}) θ ↦ MMD ℓ 2 ( P θ , Q ) α \alpha α
MMD ℓ 2 ( P θ , Q ) − min θ ∈ Θ MMD ℓ 2 ( P θ , Q ) ≤ 2 p α E z ∼ ρ [ ∥ ∇ f ℓ , P θ , Q ( G θ ( z ) ) ∥ 2 ] . \operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta},\mathbb{Q})-\min_{\theta\in\Theta}\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta},\mathbb{Q})\leq\frac{2p}{\alpha}\mathbb{E}_{z\sim\rho}\left[\left\|\nabla f_{\ell,\mathbb{P}_{\theta},\mathbb{Q}}(G_{\theta}(z))\right\|^{2}\right]. MMD ℓ 2 ( P θ , Q ) − θ ∈ Θ min MMD ℓ 2 ( P θ , Q ) ≤ α 2 p E z ∼ ρ [ ∥ ∇ f ℓ , P θ , Q ( G θ ( z )) ∥ 2 ] . Proof. Since α \alpha α
MMD ℓ 2 ( P θ , Q ) − min θ ∈ Θ MMD ℓ 2 ( P θ , Q ) ≤ 1 2 α ∥ ∇ θ MMD ℓ 2 ( P θ , Q ) ∥ 2 = 2 α ∥ E z ∼ ρ [ J θ G θ ( z ) T ∇ f ℓ , P θ , Q ( G θ ( z ) ) ] ∥ 2 = 2 α ∑ j = 1 p ∣ E z ∼ ρ [ ∂ θ j G θ ( z ) T ∇ f ℓ , P θ , Q ( G θ ( z ) ) ] ∣ 2 ≤ 2 α ∑ j = 1 p E z ∼ ρ [ ∥ ∂ θ j G θ ( z ) ∥ 2 ] × E z ∼ ρ [ ∥ ∇ f ℓ , P θ , Q ( G θ ( z ) ) ∥ 2 ] ≤ 2 p α E z ∼ ρ [ ∥ ∇ f ℓ , P θ , Q ( G θ ( z ) ) ∥ 2 ] . \begin{aligned}
\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta},\mathbb{Q})-\min_{\theta\in\Theta}\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta},\mathbb{Q})
&\leq\frac{1}{2\alpha}\left\|\nabla_{\theta}\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta},\mathbb{Q})\right\|^{2}\\
&= \frac{2}{\alpha}\left\|\mathbb{E}_{z\sim\rho}\left[J_{\theta}G_{\theta}(z)^{\mathsf{T}}\nabla f_{\ell,\mathbb{P}_{\theta},\mathbb{Q}}(G_{\theta}(z))\right]\right\|^{2}\\
&= \frac{2}{\alpha}\sum_{j=1}^{p}\left|\mathbb{E}_{z\sim\rho}\left[\partial_{\theta_j}G_{\theta}(z)^{\mathsf{T}}\nabla f_{\ell,\mathbb{P}_{\theta},\mathbb{Q}}(G_{\theta}(z))\right]\right|^{2}\\
&\leq\frac{2}{\alpha}\sum_{j=1}^{p}\mathbb{E}_{z\sim\rho}\left[\left\|\partial_{\theta_j}G_{\theta}(z)\right\|^{2}\right]\times\mathbb{E}_{z\sim\rho}\left[\left\|\nabla f_{\ell,\mathbb{P}_{\theta},\mathbb{Q}}(G_{\theta}(z))\right\|^{2}\right]\\
&\leq\frac{2p}{\alpha}\mathbb{E}_{z\sim\rho}\left[\left\|\nabla f_{\ell,\mathbb{P}_{\theta},\mathbb{Q}}(G_{\theta}(z))\right\|^{2}\right].
\end{aligned} MMD ℓ 2 ( P θ , Q ) − θ ∈ Θ min MMD ℓ 2 ( P θ , Q ) ≤ 2 α 1 ∇ θ MMD ℓ 2 ( P θ , Q ) 2 = α 2 E z ∼ ρ [ J θ G θ ( z ) T ∇ f ℓ , P θ , Q ( G θ ( z )) ] 2 = α 2 j = 1 ∑ p E z ∼ ρ [ ∂ θ j G θ ( z ) T ∇ f ℓ , P θ , Q ( G θ ( z )) ] 2 ≤ α 2 j = 1 ∑ p E z ∼ ρ [ ∂ θ j G θ ( z ) 2 ] × E z ∼ ρ [ ∥ ∇ f ℓ , P θ , Q ( G θ ( z )) ∥ 2 ] ≤ α 2 p E z ∼ ρ [ ∥ ∇ f ℓ , P θ , Q ( G θ ( z )) ∥ 2 ] . The last inequality holds by Assumption 5.3. □
Lemma C.8 (Local minima of parametric and nonparametric descent). Let θ ⋆ \theta^{\star} θ ⋆ θ ↦ MMD ℓ 2 ( P θ , Q ) \theta\mapsto\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_{\theta},\mathbb{Q}) θ ↦ MMD ℓ 2 ( P θ , Q ) P θ ⋆ \mathbb{P}_{\theta^{\star}} P θ ⋆ P ↦ MMD ℓ 2 ( P , Q ) \mathbb{P}\mapsto\operatorname{MMD}_{\ell}^{2}(\mathbb{P},\mathbb{Q}) P ↦ MMD ℓ 2 ( P , Q )
Proof. Since θ ⋆ \theta^{\star} θ ⋆ u ∈ R p u\in\mathbb{R}^{p} u ∈ R p
0 = E z ∼ ρ [ ∇ f ℓ , P θ ∗ , Q ( G θ ∗ ( z ) ) ⊤ J θ G θ ∗ ( z ) u ] . 0 = \mathbb{E}_{z\sim\rho}\left[\nabla f_{\ell,\mathbb{P}_{\theta^*},\mathbb{Q}}(G_{\theta^*}(z))^\top J_{\theta}G_{\theta^*}(z)u\right]. 0 = E z ∼ ρ [ ∇ f ℓ , P θ ∗ , Q ( G θ ∗ ( z ) ) ⊤ J θ G θ ∗ ( z ) u ] . In other words, at such a stationary point, the function z ↦ ∇ f ℓ , P θ ∗ , Q ( G θ ∗ ( z ) ) z\mapsto\nabla f_{\ell,\mathbb{P}_{\theta^*},\mathbb{Q}}(G_{\theta^*}(z)) z ↦ ∇ f ℓ , P θ ∗ , Q ( G θ ∗ ( z )) L 2 ( ρ ) L_2(\rho) L 2 ( ρ ) z ↦ J θ G θ ∗ ( z ) u z\mapsto J_{\theta}G_{\theta^*}(z)u z ↦ J θ G θ ∗ ( z ) u u ∈ R p u\in\mathbb{R}^p u ∈ R p a u ( z ) : = J θ G θ ∗ ( z ) u a_u(z):=J_{\theta}G_{\theta^*}(z)u a u ( z ) := J θ G θ ∗ ( z ) u b ( z ) : = ∇ f ℓ , P θ ∗ , Q ( G θ ∗ ( z ) ) b(z):=\nabla f_{\ell,\mathbb{P}_{\theta^*},\mathbb{Q}}(G_{\theta^*}(z)) b ( z ) := ∇ f ℓ , P θ ∗ , Q ( G θ ∗ ( z )) θ ∗ \theta^* θ ∗
E z ∼ ρ [ ∥ a u ( z ) − b ( z ) ∥ 2 ] = E z ∼ ρ [ ∥ a u ( z ) ∥ 2 ] + E z ∼ ρ [ ∥ b ( z ) ∥ 2 ] − 2 E z ∼ ρ [ b ( z ) ⊤ a u ( z ) ] = E z ∼ ρ [ ∥ a u ( z ) ∥ 2 ] + E z ∼ ρ [ ∥ b ( z ) ∥ 2 ] . \begin{aligned}
\mathbb{E}_{z\sim\rho}\left[\lVert a_u(z)-b(z)\rVert^2\right]
=
\mathbb{E}_{z\sim\rho}\left[\lVert a_u(z)\rVert^2\right]
+
\mathbb{E}_{z\sim\rho}\left[\lVert b(z)\rVert^2\right]
-
2\mathbb{E}_{z\sim\rho}\left[b(z)^\top a_u(z)\right] \\
&=
\mathbb{E}_{z\sim\rho}\left[\lVert a_u(z)\rVert^2\right]
+
\mathbb{E}_{z\sim\rho}\left[\lVert b(z)\rVert^2\right].
\end{aligned} E z ∼ ρ [ ∥ a u ( z ) − b ( z ) ∥ 2 ] = E z ∼ ρ [ ∥ a u ( z ) ∥ 2 ] + E z ∼ ρ [ ∥ b ( z ) ∥ 2 ] − 2 E z ∼ ρ [ b ( z ) ⊤ a u ( z ) ] = E z ∼ ρ [ ∥ a u ( z ) ∥ 2 ] + E z ∼ ρ [ ∥ b ( z ) ∥ 2 ] . where the cross term vanishes due to the first-order condition from above. Thus the minimiser of the residual is obtained by any direction u u u
E ρ [ ∥ J θ G θ ∗ ( z ) u ∥ 2 ] = 0 , \mathbb{E}_{\rho}\left[\lVert J_{\theta}G_{\theta^*}(z)u\rVert^2\right]=0, E ρ [ ∥ J θ G θ ∗ ( z ) u ∥ 2 ] = 0 , in which case
min u ∈ R p E z ∼ ρ [ ∥ a u ( z ) − b ( z ) ∥ 2 ] = ∥ ∇ f ℓ , P θ ∗ , Q ∥ L 2 ( P θ ∗ ) 2 . \min_{u\in\mathbb{R}^p}\mathbb{E}_{z\sim\rho}\left[\lVert a_u(z)-b(z)\rVert^2\right]
=
\left\lVert\nabla f_{\ell,\mathbb{P}_{\theta^*},\mathbb{Q}}\right\rVert^2_{L_2(\mathbb{P}_{\theta^*})}. u ∈ R p min E z ∼ ρ [ ∥ a u ( z ) − b ( z ) ∥ 2 ] = ∥ ∇ f ℓ , P θ ∗ , Q ∥ L 2 ( P θ ∗ ) 2 . For Assumption 5.4 to hold, we would then require
∥ ∇ f ℓ , P θ ∗ , Q ∥ L 2 ( P θ ∗ ) 2 ≤ R 2 ∥ ∇ f ℓ , P θ ∗ , Q ∥ L 2 ( P θ ∗ ) 2 \left\lVert\nabla f_{\ell,\mathbb{P}_{\theta^*},\mathbb{Q}}\right\rVert^2_{L_2(\mathbb{P}_{\theta^*})}
\leq\mathcal{R}^2\left\lVert\nabla f_{\ell,\mathbb{P}_{\theta^*},\mathbb{Q}}\right\rVert^2_{L_2(\mathbb{P}_{\theta^*})} ∥ ∇ f ℓ , P θ ∗ , Q ∥ L 2 ( P θ ∗ ) 2 ≤ R 2 ∥ ∇ f ℓ , P θ ∗ , Q ∥ L 2 ( P θ ∗ ) 2 for some fixed 0 < R < 1 0<\mathcal{R}<1 0 < R < 1 ∥ ∇ f ℓ , P θ ∗ , Q ∥ L 2 ( P θ ∗ ) = 0 \left\lVert\nabla f_{\ell,\mathbb{P}_{\theta^*},\mathbb{Q}}\right\rVert_{L_2(\mathbb{P}_{\theta^*})}=0 ∥ ∇ f ℓ , P θ ∗ , Q ∥ L 2 ( P θ ∗ ) = 0 P θ ∗ \mathbb{P}_{\theta^*} P θ ∗ P ↦ MMD ℓ 2 ( P , Q ) \mathbb{P}\mapsto\operatorname{MMD}^2_{\ell}(\mathbb{P},\mathbb{Q}) P ↦ MMD ℓ 2 ( P , Q )
Lemma C.9 (Example for Assumption 5.4). Let Q \mathbb{Q} Q R d \mathbb{R}^d R d P θ = ( G θ ) # Unif ( S d − 1 ) \mathbb{P}_{\theta}=(G_{\theta})_{\#}\operatorname{Unif}(\mathbb{S}^{d-1}) P θ = ( G θ ) # Unif ( S d − 1 ) G θ : z ↦ θ z G_{\theta}:z\mapsto\theta z G θ : z ↦ θ z θ ∈ R \theta\in\mathbb{R} θ ∈ R k ℓ k_{\ell} k ℓ ℓ > 0 \ell>0 ℓ > 0 θ ∈ R \theta\in\mathbb{R} θ ∈ R ∇ f ℓ , P θ , Q ( G θ ( ⋅ ) ) ∈ span { J θ G θ ( ⋅ ) u : u ∈ R } \nabla f_{\ell,\mathbb{P}_{\theta},\mathbb{Q}}(G_{\theta}(\cdot))\in\operatorname{span}\{J_{\theta}G_{\theta}(\cdot)u:u\in\mathbb{R}\} ∇ f ℓ , P θ , Q ( G θ ( ⋅ )) ∈ span { J θ G θ ( ⋅ ) u : u ∈ R } θ ∈ R \theta\in\mathbb{R} θ ∈ R
Proof. Since the Gaussian kernel is radial, k ℓ ( x , y ) = exp ( − ( 2 ℓ 2 ) − 1 ∥ x − y ∥ 2 ) k_{\ell}(x,y)=\exp(-\left(2\ell^2\right)^{-1}\lVert x-y\rVert^2) k ℓ ( x , y ) = exp ( − ( 2 ℓ 2 ) − 1 ∥ x − y ∥ 2 ) ℓ > 0 \ell> 0 ℓ > 0 f ℓ , P θ , Q ( x ) = ∫ k ℓ ( x , y ) d ( P θ − Q ) ( y ) f_{\ell,\mathbb{P}_\theta,\mathbb{Q}}(x) = \int k_\ell(x,y)\,\mathrm{d}(\mathbb{P}_\theta-\mathbb{Q})(y) f ℓ , P θ , Q ( x ) = ∫ k ℓ ( x , y ) d ( P θ − Q ) ( y ) φ θ : [ 0 , ∞ ) → R \varphi_\theta: [0,\infty) \to\mathbb{R} φ θ : [ 0 , ∞ ) → R f ℓ , P θ , Q ( x ) = φ θ ( ∥ x ∥ ) f_{\ell,\mathbb{P}_\theta,\mathbb{Q}}(x) = \varphi_\theta(\lVert x\rVert) f ℓ , P θ , Q ( x ) = φ θ (∥ x ∥) θ ≠ 0 \theta\ne0 θ = 0 z ∈ S d − 1 z \in\mathbb{S}^{d-1} z ∈ S d − 1
∇ f ℓ , P θ , Q ( G θ ( z ) ) = φ θ ′ ( ∣ θ ∣ ) θ z ∣ θ ∣ = sign ( θ ) φ θ ′ ( ∣ θ ∣ ) z . \nabla f_{\ell,\mathbb{P}_\theta,\mathbb{Q}}(G_\theta(z)) = \varphi_\theta'(|\theta|)\frac{\theta z}{|\theta|} = \operatorname{sign}(\theta)\varphi_\theta'(|\theta|)z. ∇ f ℓ , P θ , Q ( G θ ( z )) = φ θ ′ ( ∣ θ ∣ ) ∣ θ ∣ θ z = sign ( θ ) φ θ ′ ( ∣ θ ∣ ) z . When θ = 0 \theta= 0 θ = 0 f ℓ , P 0 , Q f_{\ell,\mathbb{P}_0,\mathbb{Q}} f ℓ , P 0 , Q J θ G θ ( z ) = z J_\theta G_\theta(z) = z J θ G θ ( z ) = z u = sign ( θ ) φ θ ′ ( ∣ θ ∣ ) ∈ R u = \operatorname{sign}(\theta)\varphi_\theta'(|\theta|) \in\mathbb{R} u = sign ( θ ) φ θ ′ ( ∣ θ ∣ ) ∈ R θ \theta θ ∇ f ℓ , P θ , Q ( G θ ( z ) ) = J θ G θ ( z ) u \nabla f_{\ell,\mathbb{P}_\theta,\mathbb{Q}}(G_\theta(z)) = J_\theta G_\theta(z)u ∇ f ℓ , P θ , Q ( G θ ( z )) = J θ G θ ( z ) u z ∈ S d − 1 z \in\mathbb{S}^{d-1} z ∈ S d − 1 ∣ u ∣ ≤ sup x ∥ ∇ f ℓ , P θ , Q ( x ) ∥ ≤ 2 ℓ − 1 |u| \le\sup_x \lVert\nabla f_{\ell,\mathbb{P}_\theta,\mathbb{Q}}(x)\rVert\le2\ell^{-1} ∣ u ∣ ≤ sup x ∥ ∇ f ℓ , P θ , Q ( x )∥ ≤ 2 ℓ − 1 ℓ − 1 \ell^{-1} ℓ − 1
Remark C.10. In Theorem 5.5, even when in the simplest case where Q = Unif ( S d − 1 ) \mathbb{Q} = \operatorname{Unif}(\mathbb{S}^{d-1}) Q = Unif ( S d − 1 ) P θ = ( G θ ) # Unif ( S d − 1 ) \mathbb{P}_\theta= (G_\theta)_\#\operatorname{Unif}(\mathbb{S}^{d-1}) P θ = ( G θ ) # Unif ( S d − 1 ) θ \theta θ θ ↦ MMD 2 ( P θ , Q ) \theta\mapsto\operatorname{MMD}^2(\mathbb{P}_\theta,\mathbb{Q}) θ ↦ MMD 2 ( P θ , Q ) ℓ > 0 \ell> 0 ℓ > 0
Lemma C.11 (Continuity in lengthscale). Suppose Assumption 5.1 holds. Let lengthscale ℓ > 0 \ell> 0 ℓ > 0 f ℓ , P , Q ( ⋅ ) = ∫ k ℓ ( x , ⋅ ) d ( P − Q ) ( x ) f_{\ell,\mathbb{P},\mathbb{Q}}(\cdot) = \int k_\ell(x,\cdot)\,\mathrm{d}(\mathbb{P}-\mathbb{Q})(x) f ℓ , P , Q ( ⋅ ) = ∫ k ℓ ( x , ⋅ ) d ( P − Q ) ( x ) ℓ 1 > ℓ 2 > 0 \ell_1 > \ell_2 > 0 ℓ 1 > ℓ 2 > 0
∥ ∇ f ℓ 1 , P , Q ( x ) − ∇ f ℓ 2 , P , Q ( x ) ∥ ≤ 2 C k ℓ 1 2 − ℓ 2 2 ℓ 2 3 . \lVert\nabla f_{\ell_1,\mathbb{P},\mathbb{Q}}(x)-\nabla f_{\ell_2,\mathbb{P},\mathbb{Q}}(x)\rVert\le2C_k\frac{\ell_1^2-\ell_2^2}{\ell_2^3}. ∥ ∇ f ℓ 1 , P , Q ( x ) − ∇ f ℓ 2 , P , Q ( x )∥ ≤ 2 C k ℓ 2 3 ℓ 1 2 − ℓ 2 2 . Here, C k C_k C k
Proof. From Theorem C.6, we have for any x , y ∈ R d x,y \in\mathbb{R}^d x , y ∈ R d
∥ ∇ 2 k ℓ 1 ( x , y ) − ∇ 2 k ℓ 2 ( x , y ) ∥ ≤ C k ℓ 1 2 − ℓ 2 2 ℓ 2 3 , \lVert\nabla_2 k_{\ell_1}(x,y)-\nabla_2 k_{\ell_2}(x,y)\rVert\le C_k\frac{\ell_1^2-\ell_2^2}{\ell_2^3}, ∥ ∇ 2 k ℓ 1 ( x , y ) − ∇ 2 k ℓ 2 ( x , y )∥ ≤ C k ℓ 2 3 ℓ 1 2 − ℓ 2 2 , where C k C_k C k
So we have
∥ ∇ f ℓ 1 , P , Q ( y ) − ∇ f ℓ 2 , P , Q ( y ) ∥ = ∥ ∫ ( ∇ 2 k ℓ 1 ( x , y ) − ∇ 2 k ℓ 2 ( x , y ) ) d ( P − Q ) ( x ) ∥ ≤ ∫ ∥ ∇ 2 k ℓ 1 ( x , y ) − ∇ 2 k ℓ 2 ( x , y ) ∥ d ∣ P − Q ∣ ( x ) ≤ C k ℓ 1 2 − ℓ 2 2 ℓ 2 3 ∫ d ∣ P − Q ∣ ( x ) ≤ 2 C k ℓ 1 2 − ℓ 2 2 ℓ 2 3 . \begin{aligned}
\left\|\nabla f_{\ell_1,\mathbb{P},\mathbb{Q}}(y)-\nabla f_{\ell_2,\mathbb{P},\mathbb{Q}}(y)\right\|
&=\left\|\int\left(\nabla_2 k_{\ell_1}(x,y)-\nabla_2 k_{\ell_2}(x,y)\right)\,\mathrm{d}(\mathbb{P}-\mathbb{Q})(x)\right\|\\
&\leq\int\left\|\nabla_2 k_{\ell_1}(x,y)-\nabla_2 k_{\ell_2}(x,y)\right\|\,\mathrm{d}|\mathbb{P}-\mathbb{Q}|(x)\\
&\leq C_k\frac{\ell_1^2-\ell_2^2}{\ell_2^3}\int\mathrm{d}|\mathbb{P}-\mathbb{Q}|(x)
\leq2C_k\frac{\ell_1^2-\ell_2^2}{\ell_2^3}.
\end{aligned} ∥ ∇ f ℓ 1 , P , Q ( y ) − ∇ f ℓ 2 , P , Q ( y ) ∥ = ∫ ( ∇ 2 k ℓ 1 ( x , y ) − ∇ 2 k ℓ 2 ( x , y ) ) d ( P − Q ) ( x ) ≤ ∫ ∥ ∇ 2 k ℓ 1 ( x , y ) − ∇ 2 k ℓ 2 ( x , y ) ∥ d ∣ P − Q ∣ ( x ) ≤ C k ℓ 2 3 ℓ 1 2 − ℓ 2 2 ∫ d ∣ P − Q ∣ ( x ) ≤ 2 C k ℓ 2 3 ℓ 1 2 − ℓ 2 2 . In the above derivations, ∣ P − Q ∣ \lvert\mathbb{P}-\mathbb{Q}\rvert ∣ P − Q ∣ P − Q \mathbb{P}-\mathbb{Q} P − Q
Lemma C.12 (One-step descent inequality for MMD). Let kernel k ℓ k_\ell k ℓ ℓ > 0 \ell>0 ℓ > 0 G : R d → R d G:\mathbb{R}^d\to\mathbb{R}^d G : R d → R d G ∈ L 2 d ( P 0 ) G\in L^d_2(\mathbb{P}_0) G ∈ L 2 d ( P 0 ) P 0 ∈ P 2 ( R d ) \mathbb{P}_0\in\mathcal{P}_2(\mathbb{R}^d) P 0 ∈ P 2 ( R d ) P 1 = ( Id − γ G ) # P 0 \mathbb{P}_1=(\operatorname{Id}-\gamma G)_\#\mathbb{P}_0 P 1 = ( Id − γ G ) # P 0 ℓ > 0 \ell>0 ℓ > 0
MMD ℓ 2 ( P 1 , Q ) ≤ MMD ℓ 2 ( P 0 , Q ) − 2 γ ∫ ∇ f ℓ , P 0 , Q ( x ) ⊤ G ( x ) d P 0 ( x ) + C k γ 2 ℓ 2 ∥ G ∥ L 2 d ( P 0 ) 2 . \operatorname{MMD}_{\ell}^{2}(\mathbb{P}_1,\mathbb{Q})
\leq\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_0,\mathbb{Q})
-2\gamma\int\nabla f_{\ell,\mathbb{P}_0,\mathbb{Q}}(x)^\top G(x)\,\mathrm{d}\mathbb{P}_0(x)
+\frac{C_k\gamma^2}{\ell^2}\lVert G\rVert_{L^d_2(\mathbb{P}_0)}^2. MMD ℓ 2 ( P 1 , Q ) ≤ MMD ℓ 2 ( P 0 , Q ) − 2 γ ∫ ∇ f ℓ , P 0 , Q ( x ) ⊤ G ( x ) d P 0 ( x ) + ℓ 2 C k γ 2 ∥ G ∥ L 2 d ( P 0 ) 2 . Here, C k C_k C k
Proof. Define, for t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ]
G t ( x ) : = x − t γ G ( x ) , P t : = ( G t ) # P 0 , G_t(x):=x-t\gamma G(x),\qquad\mathbb{P}_t:=(G_t)_\#\mathbb{P}_0, G t ( x ) := x − t γ G ( x ) , P t := ( G t ) # P 0 , so that P 0 = T 0 # P 0 \mathbb{P}_0=T_{0\#}\mathbb{P}_0 P 0 = T 0# P 0 P 1 = ( Id − γ G ) # P 0 \mathbb{P}_1=(\operatorname{Id}-\gamma G)_\#\mathbb{P}_0 P 1 = ( Id − γ G ) # P 0 X 0 ∼ P 0 X_0\sim\mathbb{P}_0 X 0 ∼ P 0
X t : = G t ( X 0 ) = X 0 − t γ G ( X 0 ) , X_t:=G_t(X_0)=X_0-t\gamma G(X_0), X t := G t ( X 0 ) = X 0 − t γ G ( X 0 ) , which gives us X t ∼ P t X_t\sim\mathbb{P}_t X t ∼ P t X ˙ t = − γ G ( X 0 ) \dot{X}_t=-\gamma G(X_0) X ˙ t = − γ G ( X 0 )
MMD ℓ 2 ( P t , Q ) = E [ k ℓ ( X t , X t ′ ) ] − 2 E [ k ℓ ( X t , Z ) ] + E [ k ℓ ( Z , Z ′ ) ] , \operatorname{MMD}_{\ell}^{2}(\mathbb{P}_t,\mathbb{Q})
=\mathbb{E}\left[k_\ell(X_t,X'_t)\right]-2\mathbb{E}\left[k_\ell(X_t,Z)\right]+\mathbb{E}\left[k_\ell(Z,Z')\right], MMD ℓ 2 ( P t , Q ) = E [ k ℓ ( X t , X t ′ ) ] − 2 E [ k ℓ ( X t , Z ) ] + E [ k ℓ ( Z , Z ′ ) ] , X t , X t ′ ∼ P t , Z , Z ′ ∼ Q . X_t,X'_t\sim\mathbb{P}_t,\qquad Z,Z'\sim\mathbb{Q}. X t , X t ′ ∼ P t , Z , Z ′ ∼ Q . We can differentiate with respect to t t t X ˙ t = − γ G ( X 0 ) \dot{X}_t=-\gamma G(X_0) X ˙ t = − γ G ( X 0 ) X ˙ t ′ = − γ G ( X 0 ′ ) \dot{X}'_t=-\gamma G(X'_0) X ˙ t ′ = − γ G ( X 0 ′ )
d d t MMD ℓ 2 ( P t , Q ) = − 2 γ E X 0 ∼ P 0 [ ∇ f ℓ , P t , Q ( X 0 − t γ G ( X 0 ) ) T G ( X 0 ) ] . \frac{\mathrm{d}}{\mathrm{d}t}\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_t,\mathbb{Q})=-2\gamma\mathbb{E}_{X_0\sim\mathbb{P}_0}\left[\nabla f_{\ell,\mathbb{P}_t,\mathbb{Q}}(X_0-t\gamma G(X_0))^{\mathsf{T}}G(X_0)\right]. d t d MMD ℓ 2 ( P t , Q ) = − 2 γ E X 0 ∼ P 0 [ ∇ f ℓ , P t , Q ( X 0 − t γ G ( X 0 ) ) T G ( X 0 ) ] . Then, by the fundamental theorem of calculus (Rudin, 1976), we have
MMD ℓ 2 ( P 1 , Q ) − MMD ℓ 2 ( P 0 , Q ) = − 2 γ ∫ 0 1 E X 0 ∼ P 0 [ ∇ f ℓ , P t , Q ( X 0 − t γ G ( X 0 ) ) T G ( X 0 ) ] d t . \begin{aligned}
\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_1,\mathbb{Q})-\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_0,\mathbb{Q})
&=-2\gamma\int_{0}^{1}\mathbb{E}_{X_0\sim\mathbb{P}_0}\left[\nabla f_{\ell,\mathbb{P}_t,\mathbb{Q}}(X_0-t\gamma G(X_0))^{\mathsf{T}}G(X_0)\right]\mathrm{d}t.
\end{aligned} MMD ℓ 2 ( P 1 , Q ) − MMD ℓ 2 ( P 0 , Q ) = − 2 γ ∫ 0 1 E X 0 ∼ P 0 [ ∇ f ℓ , P t , Q ( X 0 − t γ G ( X 0 ) ) T G ( X 0 ) ] d t . By writing ∇ f ℓ , P t , Q ( X 0 − t γ G ( X 0 ) ) = ∇ f ℓ , P 0 , Q ( X 0 ) + ( ∇ f ℓ , P t , Q ( X 0 − t γ G ( X 0 ) ) − ∇ f ℓ , P 0 , Q ( X 0 ) ) \nabla f_{\ell,\mathbb{P}_t,\mathbb{Q}}(X_0-t\gamma G(X_0))=\nabla f_{\ell,\mathbb{P}_0,\mathbb{Q}}(X_0)+(\nabla f_{\ell,\mathbb{P}_t,\mathbb{Q}}(X_0-t\gamma G(X_0))-\nabla f_{\ell,\mathbb{P}_0,\mathbb{Q}}(X_0)) ∇ f ℓ , P t , Q ( X 0 − t γ G ( X 0 )) = ∇ f ℓ , P 0 , Q ( X 0 ) + ( ∇ f ℓ , P t , Q ( X 0 − t γ G ( X 0 )) − ∇ f ℓ , P 0 , Q ( X 0 ))
\begin{aligned}
\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_1,\mathbb{Q})
={}&\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_0,\mathbb{Q})-2\gamma\int\nabla f_{\ell,\mathbb{P}_0,\mathbb{Q}}(x)^{\mathsf{T}}G(x)\,\mathrm{d}\mathbb{P}_0(x)\\
&-2\gamma\int_{0}^{1}\mathbb{E}_{X_0\sim\mathbb{P}_0}\left[\left(\nabla f_{\ell,\mathbb{P}_t,\mathbb{Q}}(X_0-t\gamma G(X_0))-\n\nabla f_{\ell,\mathbb{P}_0,\mathbb{Q}}(X_0)\right)^{\mathsf{T}}G(X_0)\right]\mathrm{d}t. \tag*{(156)}
\end{aligned}
For the last term of the right hand side, by adding and subtracting ∇ f ℓ , P t , Q ( X 0 ) \nabla f_{\ell,\mathbb{P}_t,\mathbb{Q}}(X_0) ∇ f ℓ , P t , Q ( X 0 ) ( ∇ f ℓ , P t , Q ( X 0 − t γ G ( X 0 ) ) − ∇ f ℓ , P 0 , Q ( X 0 ) ) (\nabla f_{\ell,\mathbb{P}_t,\mathbb{Q}}(X_0-t\gamma G(X_0))-\nabla f_{\ell,\mathbb{P}_0,\mathbb{Q}}(X_0)) ( ∇ f ℓ , P t , Q ( X 0 − t γ G ( X 0 )) − ∇ f ℓ , P 0 , Q ( X 0 ))
\begin{aligned}
-2\gamma\int_{0}^{1}\mathbb{E}_{X_0\sim\mathbb{P}_0}\left[\left(\nabla f_{\ell,\mathbb{P}_t,\mathbb{Q}}(X_0-t\gamma G(X_0))-\n\nabla f_{\ell,\mathbb{P}_0,\mathbb{Q}}(X_0)\right)^{\mathsf{T}}G(X_0)\right]\mathrm{d}t\\
=-2\gamma\int_{0}^{1}\mathbb{E}_{X_0\sim\mathbb{P}_0}\left[\left(\nabla f_{\ell,\mathbb{P}_t,\mathbb{Q}}(X_0-t\gamma G(X_0))-\n\nabla f_{\ell,\mathbb{P}_t,\mathbb{Q}}(X_0)\right)^{\mathsf{T}}G(X_0)\right]\mathrm{d}t. \tag*{(157)}
\end{aligned}
− 2 γ ∫ 0 1 E X 0 ∼ P 0 [ ( ∇ f ℓ , P t , Q ( X 0 ) − ∇ f ℓ , P 0 , Q ( X 0 ) ) T G ( X 0 ) ] d t . (158) -2\gamma\int_{0}^{1}\mathbb{E}_{X_0\sim\mathbb{P}_0}\left[\left(\nabla f_{\ell,\mathbb{P}_t,\mathbb{Q}}(X_0)-\nabla f_{\ell,\mathbb{P}_0,\mathbb{Q}}(X_0)\right)^{\mathsf{T}}G(X_0)\right]\mathrm{d}t. \tag*{(158)} − 2 γ ∫ 0 1 E X 0 ∼ P 0 [ ( ∇ f ℓ , P t , Q ( X 0 ) − ∇ f ℓ , P 0 , Q ( X 0 ) ) T G ( X 0 ) ] d t . (158) Since x ↦ ∇ f ℓ , P t , Q ( x ) x\mapsto\nabla f_{\ell,\mathbb{P}_t,\mathbb{Q}}(x) x ↦ ∇ f ℓ , P t , Q ( x ) ( C k / ℓ 2 ) (C_k/\ell^2) ( C k / ℓ 2 ) s > 1 s>1 s > 1
∥ ∇ f ℓ , P t , Q ( X t ) − ∇ f ℓ , P t , Q ( X 0 ) ∥ ≤ C k ℓ 2 ∥ X t − X 0 ∥ = C k t γ ℓ 2 ∥ G ( X 0 ) ∥ . \left\|\nabla f_{\ell,\mathbb{P}_t,\mathbb{Q}}(X_t)-\nabla f_{\ell,\mathbb{P}_t,\mathbb{Q}}(X_0)\right\|
\leq\frac{C_k}{\ell^2}\left\|X_t-X_0\right\|
=\frac{C_k t\gamma}{\ell^2}\left\|G(X_0)\right\|. ∥ ∇ f ℓ , P t , Q ( X t ) − ∇ f ℓ , P t , Q ( X 0 ) ∥ ≤ ℓ 2 C k ∥ X t − X 0 ∥ = ℓ 2 C k t γ ∥ G ( X 0 ) ∥ . C k C_k C k
\begin{aligned}
\text{(C.27)} &\le2\gamma\int_0^1 \mathbb{E}_{X_0\sim\mathbb{P}_0}\left[\left\|\nabla f_{\ell,P_t,Q}(X_t)-\nabla f_{\ell,P_t,Q}(X_0)\right\| \\
&\qquad\qquad\qquad\qquad\qquad\qquad\times\left\|G(X_0)\right\|\right]\mathrm{d}t \\
&\le2\gamma\int_0^1 \frac{C_k\gamma t}{\ell^2}\mathbb{E}_{X_0\sim\mathbb{P}_0}\left[\left\|G(X_0)\right\|^2\right]\mathrm{d}t \\
&= \frac{2C_k\gamma^2}{\ell^2}\left\|G\right\|_{L^2(\mathbb{P}_0)}^2. \tag*{(159)}
\end{aligned}
Moreover, using the coupling Y t = Y 0 − γ t G ( Y 0 ) Y_t=Y_0-\gamma tG(Y_0) Y t = Y 0 − γ tG ( Y 0 ) Y 0 ∼ P 0 Y_0\sim\mathbb{P}_0 Y 0 ∼ P 0 ( C k / ℓ 2 ) (C_k/\ell^2) ( C k / ℓ 2 ) k ℓ k_\ell k ℓ x ∈ R d x\in\mathbb{R}^d x ∈ R d
∥ ∇ f ℓ , P t , Q ( x ) − ∇ f ℓ , P 0 , Q ( x ) ∥ = E Y 0 , Y t [ ∥ ∇ 1 k ℓ ( x , Y t ) − ∇ 1 k ℓ ( x , Y 0 ) ∥ ] ≤ C k t γ ℓ 2 E [ ∥ G ( Y 0 ) ∥ ] . \begin{aligned}
\left\|\nabla f_{\ell,P_t,Q}(x)-\nabla f_{\ell,P_0,Q}(x)\right\|
&=\mathbb{E}_{Y_0,Y_t}\left[\left\|\nabla_1 k_\ell(x,Y_t)-\nabla_1 k_\ell(x,Y_0)\right\|\right]\\
&\le\frac{C_k t\gamma}{\ell^2}\mathbb{E}\left[\left\|G(Y_0)\right\|\right].
\end{aligned} ∥ ∇ f ℓ , P t , Q ( x ) − ∇ f ℓ , P 0 , Q ( x ) ∥ = E Y 0 , Y t [ ∥ ∇ 1 k ℓ ( x , Y t ) − ∇ 1 k ℓ ( x , Y 0 ) ∥ ] ≤ ℓ 2 C k t γ E [ ∥ G ( Y 0 ) ∥ ] . Hence, by Cauchy-Schwarz, integrating over t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ]
(C.28) ≤ 2 γ ∫ 0 1 C k t γ ℓ 2 E [ ∥ G ( X 0 ) ∥ 2 ] d t = C k γ 2 ℓ 2 ∥ G ∥ L 2 ( P 0 ) 2 . (160) \begin{aligned}
\text{(C.28)} &\le2\gamma\int_0^1\frac{C_k t\gamma}{\ell^2}\mathbb{E}\left[\left\|G(X_0)\right\|^2\right]\mathrm{d}t\\
&=\frac{C_k\gamma^2}{\ell^2}\left\|G\right\|_{L^2(\mathbb{P}_0)}^2. \tag*{(160)}
\end{aligned} (C.28) ≤ 2 γ ∫ 0 1 ℓ 2 C k t γ E [ ∥ G ( X 0 ) ∥ 2 ] d t = ℓ 2 C k γ 2 ∥ G ∥ L 2 ( P 0 ) 2 . (160) Combining the last two upper bounds on Eq. (C.27) and Eq. (C.28), along with inequalities with Eq. (C.26), we obtain
MMD ℓ 2 ( P 1 , Q ) ≤ MMD ℓ 2 ( P 0 , Q ) − 2 γ ∫ ∇ f ℓ , P 0 , Q ( x ) T G ( x ) d P 0 ( x ) + C k γ 2 ℓ 2 ∥ G ∥ L 2 ( P 0 ) 2 . \operatorname{MMD}_{\ell}^{2}(\mathbb{P}_1,\mathbb{Q})\le\operatorname{MMD}_{\ell}^{2}(\mathbb{P}_0,\mathbb{Q})-2\gamma\int\nabla f_{\ell,\mathbb{P}_0,\mathbb{Q}}(x)^\mathsf{T}G(x)\mathrm{d}\mathbb{P}_0(x)+\frac{C_k\gamma^2}{\ell^2}\left\|G\right\|_{L^2(\mathbb{P}_0)}^2. MMD ℓ 2 ( P 1 , Q ) ≤ MMD ℓ 2 ( P 0 , Q ) − 2 γ ∫ ∇ f ℓ , P 0 , Q ( x ) T G ( x ) d P 0 ( x ) + ℓ 2 C k γ 2 ∥ G ∥ L 2 ( P 0 ) 2 . Lemma C.13 (Bandwidth adjustment via convolution with noise). Let k ℓ k_\ell k ℓ s s s f ℓ , P , Q ( ⋅ ) = ∫ k ℓ ( x , ⋅ ) d ( P − Q ) ( x ) f_{\ell,\mathbb{P},\mathbb{Q}}(\cdot)=\int k_\ell(x,\cdot)\mathrm{d}(\mathbb{P}-\mathbb{Q})(x) f ℓ , P , Q ( ⋅ ) = ∫ k ℓ ( x , ⋅ ) d ( P − Q ) ( x ) ℓ ′ > ℓ > 0 \ell' > \ell> 0 ℓ ′ > ℓ > 0 ν \nu ν R d \mathbb{R}^d R d
for any x ∈ R d x \in\mathbb{R}^{d} x ∈ R d
E U ∼ ν [ ∇ f ℓ , P , Q ( x + U ) ] = ( ℓ ℓ ′ − 1 ) d ∇ f ℓ ′ , P , Q ( x ) , E U ∼ ν [ ∥ U ∥ ] ≤ C d , s ℓ ′ 2 − ℓ 2 . (161) \mathbb{E}_{U\sim\nu}\left[\nabla f_{\ell,\mathbb{P},\mathbb{Q}}(x+U)\right] = \left(\ell\ell^{\prime-1}\right)^{d}\nabla f_{\ell^{\prime},\mathbb{P},\mathbb{Q}}(x), \qquad\mathbb{E}_{U\sim\nu}[\lVert U\rVert] \le C_{d,s}\sqrt{\ell^{\prime2}-\ell^{2}}. \tag*{(161)} E U ∼ ν [ ∇ f ℓ , P , Q ( x + U ) ] = ( ℓ ℓ ′ − 1 ) d ∇ f ℓ ′ , P , Q ( x ) , E U ∼ ν [∥ U ∥] ≤ C d , s ℓ ′2 − ℓ 2 . (161) Here, C d , s C_{d,s} C d , s d d d s s s
Proof. Notice that
E U ∼ ν [ ∇ f ℓ , P , Q ( x + U ) ] = ∫ R d ∫ R d ∇ 1 k ℓ ( x + u , y ) d ( P − Q ) ( y ) d ν ( u ) = ∫ R d ∇ x ( ∫ R d k ℓ ( x + u , y ) d ν ( u ) ) d ( P − Q ) ( y ) , \begin{aligned}
\mathbb{E}_{U\sim\nu}\left[\nabla f_{\ell,\mathbb{P},\mathbb{Q}}(x+U)\right] = \int_{\mathbb{R}^{d}}\int_{\mathbb{R}^{d}} \nabla_{1}k_{\ell}(x+u,y)\,\mathrm{d}(\mathbb{P}-\mathbb{Q})(y)\,\mathrm{d}\nu(u) \\
&= \int_{\mathbb{R}^{d}}\nabla_{x}\left(\int_{\mathbb{R}^{d}} k_{\ell}(x+u,y)\,\mathrm{d}\nu(u)\right)\,\mathrm{d}(\mathbb{P}-\mathbb{Q})(y),
\end{aligned} E U ∼ ν [ ∇ f ℓ , P , Q ( x + U ) ] = ∫ R d ∫ R d ∇ 1 k ℓ ( x + u , y ) d ( P − Q ) ( y ) d ν ( u ) = ∫ R d ∇ x ( ∫ R d k ℓ ( x + u , y ) d ν ( u ) ) d ( P − Q ) ( y ) , where the second equality is from using Fubini’s theorem (Rudin, 1976) given that k ℓ k_{\ell} k ℓ y ∈ R d y \in\mathbb{R}^{d} y ∈ R d
∫ R d k ℓ ( x + u , y ) d ν ( u ) = ( ℓ ℓ ′ − 1 ) d k ℓ ′ ( x , y ) . \int_{\mathbb{R}^{d}} k_{\ell}(x+u,y)\,\mathrm{d}\nu(u) = \left(\ell\ell^{\prime-1}\right)^{d}k_{\ell^{\prime}}(x,y). ∫ R d k ℓ ( x + u , y ) d ν ( u ) = ( ℓ ℓ ′ − 1 ) d k ℓ ′ ( x , y ) . And, we obtain
E U ∼ ν [ ∇ f ℓ , P , Q ( x + U ) ] = ∫ R d ( ℓ ℓ ′ − 1 ) d ∇ 1 k ℓ ′ ( x , y ) d ( P − Q ) ( y ) = ( ℓ ℓ ′ − 1 ) d ∇ f ℓ ′ , P , Q ( x ) . \begin{aligned}
\mathbb{E}_{U\sim\nu}\left[\nabla f_{\ell,\mathbb{P},\mathbb{Q}}(x+U)\right] = \int_{\mathbb{R}^{d}}\left(\ell\ell^{\prime-1}\right)^{d}\nabla_{1}k_{\ell^{\prime}}(x,y)\,\mathrm{d}(\mathbb{P}-\mathbb{Q})(y) \\
&= \left(\ell\ell^{\prime-1}\right)^{d}\nabla f_{\ell^{\prime},\mathbb{P},\mathbb{Q}}(x).
\end{aligned} E U ∼ ν [ ∇ f ℓ , P , Q ( x + U ) ] = ∫ R d ( ℓ ℓ ′ − 1 ) d ∇ 1 k ℓ ′ ( x , y ) d ( P − Q ) ( y ) = ( ℓ ℓ ′ − 1 ) d ∇ f ℓ ′ , P , Q ( x ) . The first moment bound on U ∼ ν U\sim\nu U ∼ ν
Notations: We denote F [ f ] : R d → R \mathcal{F}[f] : \mathbb{R}^{d} \to\mathbb{R} F [ f ] : R d → R f : R d → R f : \mathbb{R}^{d} \to\mathbb{R} f : R d → R
Lemma C.14 (Noise perturbation). Let k ℓ k_{\ell} k ℓ s s s Assumption 5.1 . Then, for any ℓ ′ > ℓ > 0 \ell^{\prime} > \ell> 0 ℓ ′ > ℓ > 0 ν \nu ν R d \mathbb{R}^{d} R d E U ∼ ν [ k ℓ ( x + U , y ) ] = ( ℓ ℓ ′ − 1 ) d k ℓ ′ ( x , y ) \mathbb{E}_{U\sim\nu}[k_{\ell}(x+U,y)] = (\ell\ell^{\prime-1})^{d}k_{\ell^{\prime}}(x,y) E U ∼ ν [ k ℓ ( x + U , y )] = ( ℓ ℓ ′ − 1 ) d k ℓ ′ ( x , y ) x , y ∈ R d x,y \in\mathbb{R}^{d} x , y ∈ R d E U ∼ ν [ ∥ U ∥ ] ≤ C d , s ℓ ′ 2 − ℓ 2 \mathbb{E}_{U\sim\nu}[\lVert U\rVert] \le C_{d,s}\sqrt{\ell^{\prime2}-\ell^{2}} E U ∼ ν [∥ U ∥] ≤ C d , s ℓ ′2 − ℓ 2 C d , s C_{d,s} C d , s d d d s s s
Proof. Write k ℓ ( x , y ) = ψ ℓ ( x − y ) k_{\ell}(x,y)=\psi_{\ell}(x-y) k ℓ ( x , y ) = ψ ℓ ( x − y ) S ℓ = F [ ψ ℓ ] : R d → R S_{\ell}=\mathcal{F}[\psi_{\ell}]:\mathbb{R}^{d}\to\mathbb{R} S ℓ = F [ ψ ℓ ] : R d → R ψ ℓ \psi_{\ell} ψ ℓ ν \nu ν
ν ^ ( ω ) : = ∫ e − i ω ⊤ u d ν ( u ) = ( ℓ ℓ ′ − 1 ) d S ℓ ′ ( ω ) S ℓ ( ω ) . (162) \hat{\nu}(\omega):=\int e^{-i\omega^{\top}u}d\nu(u)=\left(\ell\ell'^{-1}\right)^{d}\frac{S_{\ell'}(\omega)}{S_{\ell}(\omega)}. \tag*{(162)} ν ^ ( ω ) := ∫ e − i ω ⊤ u d ν ( u ) = ( ℓ ℓ ′ − 1 ) d S ℓ ( ω ) S ℓ ′ ( ω ) . (162) Indeed, based on the convolution theorem for Fourier transforms (see Katznelson [1 ] , Chapter VI, Section 2, pp. 144–145), this gives
F [ E U ∼ ν ψ ℓ ( ⋅ + U ) ] ( ω ) = F [ ∫ R d ψ ℓ ( ⋅ + u ) d ν ( u ) ] = F [ ∫ R d ψ ℓ ( ⋅ − u ) d ν ( u ) ] = S ℓ ( ω ) ⋅ ν ^ ( ω ) = ( ℓ ℓ ′ − 1 ) d S ℓ ′ ( ω ) , \begin{aligned}
\mathcal{F}\left[\mathbb{E}_{U\sim\nu}\psi_{\ell}(\cdot+U)\right](\omega)
&=\mathcal{F}\left[\int_{\mathbb{R}^{d}}\psi_{\ell}(\cdot+u)d\nu(u)\right]
=\mathcal{F}\left[\int_{\mathbb{R}^{d}}\psi_{\ell}(\cdot-u)d\nu(u)\right]\\
&=S_{\ell}(\omega)\cdot\hat{\nu}(\omega)
=\left(\ell\ell'^{-1}\right)^{d}S_{\ell'}(\omega),
\end{aligned} F [ E U ∼ ν ψ ℓ ( ⋅ + U ) ] ( ω ) = F [ ∫ R d ψ ℓ ( ⋅ + u ) d ν ( u ) ] = F [ ∫ R d ψ ℓ ( ⋅ − u ) d ν ( u ) ] = S ℓ ( ω ) ⋅ ν ^ ( ω ) = ( ℓ ℓ ′ − 1 ) d S ℓ ′ ( ω ) , and hence we obtain the desired result: E U ∼ ν [ k ℓ ( x + U , y ) ] = ( ℓ ℓ ′ − 1 ) d k ℓ ′ ( x , y ) \mathbb{E}_{U\sim\nu}[k_{\ell}(x+U,y)]=(\ell\ell'^{-1})^{d}k_{\ell'}(x,y) E U ∼ ν [ k ℓ ( x + U , y )] = ( ℓ ℓ ′ − 1 ) d k ℓ ′ ( x , y )
Next, we are going to prove Eq. (C.30) for the Gaussian and Matérn kernels.
For the Gaussian kernel, we have S ℓ ( ω ) = C d ℓ d exp ( − ℓ 2 ∥ ω ∥ 2 2 ) S_{\ell}(\omega)=C_{d}\ell^{d}\exp\left(-\frac{\ell^{2}\lVert\omega\rVert^{2}}{2}\right) S ℓ ( ω ) = C d ℓ d exp ( − 2 ℓ 2 ∥ ω ∥ 2 )
( ℓ ℓ ′ − 1 ) d S ℓ ′ ( ω ) S ℓ ( ω ) = exp ( − ( ℓ ′ 2 − ℓ 2 ) ∥ ω ∥ 2 2 ) , \left(\ell\ell'^{-1}\right)^{d}\frac{S_{\ell'}(\omega)}{S_{\ell}(\omega)}
=\exp\left(-\frac{(\ell'^{2}-\ell^{2})\lVert\omega\rVert^{2}}{2}\right), ( ℓ ℓ ′ − 1 ) d S ℓ ( ω ) S ℓ ′ ( ω ) = exp ( − 2 ( ℓ ′ 2 − ℓ 2 ) ∥ ω ∥ 2 ) , which is the characteristic function of a normal distribution ν = N ( 0 , ( ℓ ′ 2 − ℓ 2 ) I d ) \nu=\mathcal{N}(0,(\ell'^{2}-\ell^{2})I_{d}) ν = N ( 0 , ( ℓ ′ 2 − ℓ 2 ) I d ) E U ∼ ν [ ∥ U ∥ ] ≤ E U ∼ ν [ ∥ U ∥ 2 ] = d ( ℓ ′ 2 − ℓ 2 ) \mathbb{E}_{U\sim\nu}[\lVert U\rVert]\leq\sqrt{\mathbb{E}_{U\sim\nu}[\lVert U\rVert^{2}]}=\sqrt{d(\ell'^{2}-\ell^{2})} E U ∼ ν [∥ U ∥] ≤ E U ∼ ν [∥ U ∥ 2 ] = d ( ℓ ′ 2 − ℓ 2 )
Next, for the Matérn kernel with smoothness parameter s s s [2 ] ):
S ℓ ( ω ) = C s , d ℓ d ( 1 + ℓ 2 ∥ ω ∥ 2 2 s ) − ( s + d / 2 ) . S_{\ell}(\omega)=C_{s,d}\ell^{d}\left(1+\frac{\ell^{2}\lVert\omega\rVert^{2}}{2s}\right)^{-(s+d/2)}. S ℓ ( ω ) = C s , d ℓ d ( 1 + 2 s ℓ 2 ∥ ω ∥ 2 ) − ( s + d /2 ) . Hence
( ℓ ℓ ′ − 1 ) d S ℓ ′ ( ω ) S ℓ ( ω ) = ( 1 + ℓ 2 ∥ ω ∥ 2 2 s 1 + ℓ ′ 2 ∥ ω ∥ 2 2 s ) s + d / 2 . (163) \left(\ell\ell'^{-1}\right)^{d}\frac{S_{\ell'}(\omega)}{S_{\ell}(\omega)}
=\left(\frac{1+\frac{\ell^{2}\lVert\omega\rVert^{2}}{2s}}{1+\frac{\ell'^{2}\lVert\omega\rVert^{2}}{2s}}\right)^{s+d/2}. \tag*{(163)} ( ℓ ℓ ′ − 1 ) d S ℓ ( ω ) S ℓ ′ ( ω ) = ( 1 + 2 s ℓ ′ 2 ∥ ω ∥ 2 1 + 2 s ℓ 2 ∥ ω ∥ 2 ) s + d /2 . (163) Since ℓ ′ > ℓ \ell' > \ell ℓ ′ > ℓ
g : r ↦ ( 1 + ℓ 2 r 2 s 1 + ℓ ′ 2 r 2 s ) s + d / 2 , r ≥ 0 , (164) g : r \mapsto\left(\frac{1+\frac{\ell^{2}r}{2s}}{1+\frac{\ell'^{2}r}{2s}}\right)^{s+d/2}, \qquad r \ge0, \tag*{(164)} g : r ↦ ( 1 + 2 s ℓ ′ 2 r 1 + 2 s ℓ 2 r ) s + d /2 , r ≥ 0 , (164) is completely monotone1 g ( 0 ) = 1 g(0)=1 g ( 0 ) = 1 h ( r ) : = − log g ( r ) h(r):=-\log g(r) h ( r ) := − log g ( r ) h ( r ) = ( s + d 2 ) [ log ( 1 + ℓ ′ 2 r 2 s ) − log ( 1 + ℓ ′ 2 r 2 s ) ] h(r)=\left(s+\frac{d}{2}\right)\left[\log\left(1+\frac{\ell'^{2}r}{2s}\right)-\log\left(1+\frac{\ell'^{2}r}{2s}\right)\right] h ( r ) = ( s + 2 d ) [ log ( 1 + 2 s ℓ ′ 2 r ) − log ( 1 + 2 s ℓ ′ 2 r ) ]
h ′ ( r ) = ( s + d 2 ) ( ℓ ′ 2 − ℓ 2 ) / ( 2 s ) ( 1 + ℓ 2 r 2 s ) ( 1 + ℓ ′ 2 r 2 s ) . h'(r)=\left(s+\frac{d}{2}\right)\frac{\left(\ell'^{2}-\ell^{2}\right)/(2s)}{\left(1+\frac{\ell^{2}r}{2s}\right)\left(1+\frac{\ell'^{2}r}{2s}\right)}. h ′ ( r ) = ( s + 2 d ) ( 1 + 2 s ℓ 2 r ) ( 1 + 2 s ℓ ′ 2 r ) ( ℓ ′ 2 − ℓ 2 ) / ( 2 s ) . The prefactor is positive because ℓ ′ > ℓ \ell' > \ell ℓ ′ > ℓ a > 0 a>0 a > 0 r ↦ ( 1 + a r ) − 1 r\mapsto(1+ar)^{-1} r ↦ ( 1 + a r ) − 1
( − 1 ) n d n d r n ( 1 + a r ) − 1 = n ! a n ( 1 + a r ) − n − 1 ≥ 0 , ∀ n ∈ N . (-1)^n\frac{\mathrm{d}^n}{\mathrm{d}r^n}(1+ar)^{-1}=n!a^n(1+ar)^{-n-1}\ge0,\qquad\forall n\in\mathbb{N}. ( − 1 ) n d r n d n ( 1 + a r ) − 1 = n ! a n ( 1 + a r ) − n − 1 ≥ 0 , ∀ n ∈ N . Therefore both factors
r ↦ ( 1 + ℓ 2 r 2 s ) − 1 , r ↦ ( 1 + ℓ ′ 2 r 2 s ) − 1 r\mapsto\left(1+\frac{\ell^{2}r}{2s}\right)^{-1},\qquad r\mapsto\left(1+\frac{\ell'^{2}r}{2s}\right)^{-1} r ↦ ( 1 + 2 s ℓ 2 r ) − 1 , r ↦ ( 1 + 2 s ℓ ′ 2 r ) − 1 are completely monotone. Since products of completely monotone functions are completely monotone ((Schilling et al., 2012, Corollary 1.6)), it follows that h ′ h' h ′
Thus h h h 2 h ( 0 ) = 0 h(0)=0 h ( 0 ) = 0 h ′ ≥ 0 h'\ge0 h ′ ≥ 0 h ′ h' h ′ h h h exp ( − u h ) \exp(-uh) exp ( − u h ) u > 0 u>0 u > 0 u = 1 u=1 u = 1
g ( r ) = exp ( − h ( r ) ) g(r)=\exp(-h(r)) g ( r ) = exp ( − h ( r )) is completely monotone. Also g ( 0 ) = 1 g(0)=1 g ( 0 ) = 1 g g g g ( 0 ) = 1 g(0)=1 g ( 0 ) = 1
