Introduction
A growing body of empirical and theoretical work has raised the concern that machine learning generative models trained iteratively on data produced by earlier generations of themselves — rather than on real, independently sourced data — can drift away from the true underlying distribution over successive generations, a phenomenon often called model collapse [22 , 1 ] . As more of the data on which models are trained is itself generated by other models, understanding the conditions under which this recursive process is destructive to the model and how we can deal with that has become an important theoretical question [3 , 10 , 14 , 21 , 5 , 4 , 2 , 11 , 8 ] .
There are several approaches to iterative retraining, two of which are commonly referred to as Accumulate and Replace . In the Accumulate method, we start with a set of real data, and at each iteration, we augment the training set with newly generated data while retaining all data from previous iterations. In contrast, in the Replace method, the training set at each iteration consists only of the newly generated data from the previous iteration; all data from earlier iterations are discarded. The Accumulate method is generally considered a more realistic setting for iterative retraining and has been shown to exhibit degradation that remains bounded across iterations [10 , 14 , 21 ] . In contrast, the Replace method can lead to unbounded degradation, with the learned distribution progressively diverging from the original data distribution [22 , 14 ] .
Existing literature tries to theoretically explain the dramatic improvement of current top-tier generative models. This is done by considering the existence of independent real data in each iteration. However, it has only been investigated under the assumption that the real data can be perfectly separated from the generated data [1 , 11 , 8 ] . In this case, an optimal weighting scheme for the separate real and generated data used for training has been studied [11 ] . However, exact separation is not realistic [19 , 16 ] , and uncurated data is still commonly used. In this paper, to the best of our knowledge, we are the first to bridge this gap by considering a less restrictive and more realistic setup in which real and generated data are not handled differently during training.
We study Accumulate -based iterative estimation of the mean μ ∈ R d \boldsymbol{\mu} \in\mathbb{R}^{d} μ ∈ R d and covariance matrix Σ ∈ R d × d \boldsymbol{\Sigma} \in\mathbb{R}^{d \times d} Σ ∈ R d × d of a Gaussian data distribution N ( μ , Σ ) \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma}) N ( μ , Σ ) , a standard setting in theoretical works on model collapse [22 , 1 , 14 , 11 ] . At each round t t t , a batch of new samples is drawn from a mixture of (i) the current fitted Gaussian model and (ii) the true distribution, and all data accumulated so far is used to refit the mean and covariance by standard empirical estimators, μ ( t ) \boldsymbol{\mu}^{(t)} μ ( t ) and Σ ( t ) \boldsymbol{\Sigma}^{(t)} Σ ( t ) , but without distinguishing between real and generated samples. This setting strips away the complications of neural networks while retaining the essential recursive feedback loop and admits rigorous analysis.
Our guiding question is: under what conditions does this process not collapse . In other words, when do the iterated estimators ( μ ( t ) , Σ ( t ) ) (\boldsymbol{\mu}^{(t)}, \boldsymbol{\Sigma}^{(t)}) ( μ ( t ) , Σ ( t ) ) converge to the true ( μ , Σ ) (\boldsymbol{\mu}, \boldsymbol{\Sigma}) ( μ , Σ ) , and when do they instead settle on a biased, degenerate limit? We answer this question completely for both estimators. For the mean estimator, we also analyze the rate of convergence. Here, the mixing fraction α t ∈ [ 0 , 1 ] \alpha_t \in[0, 1] α t ∈ [ 0 , 1 ] , the expected fraction of data in the new batch drawn from the current model, plays a decisive role, and we provide an explicit, checkable condition on how it may vary across rounds while still guaranteeing consistency. That separates collapse from non-collapse. Our main contributions include:
We prove that the mean estimator μ ( t ) \boldsymbol{\mu}^{(t)} μ ( t ) is unbiased, while the covariance estimator Σ ( t ) \boldsymbol{\Sigma}^{(t)} Σ ( t ) is biased, with the exact bias given by − Cov ( μ ( t ) ) -\operatorname{Cov}(\boldsymbol{\mu}^{(t)}) − Cov ( μ ( t ) ) .
We provide a closed-form equivalent condition for the consistency of the estimated mean in L 2 L^2 L 2 , the consistency of the covariance estimator in expected Frobenius norm, and the convergence to zero of the expected squared 2-Wasserstein distance between the estimated and true distributions.
We use our result to show that several scenarios that naturally arise when scraping the web for data are sufficient to guarantee convergence, thereby preventing model collapse. First, convergence holds for an arbitrarily small but fixed positive fraction of real data ( 1 − α ) (1-\alpha) ( 1 − α ) at every iteration. Second, convergence still holds when the fraction of real data vanishes across iterations, provided that it does not vanish too quickly.
For a fixed synthetic fraction 0 ≤ α < 1 0 \leq\alpha< 1 0 ≤ α < 1 (i.e., when not all data is generated), we derive the exact rate of convergence (up to constant factors) of the mean estimator to the true mean. Specifically, we prove that if less than half of each new batch is generated, then the rate of convergence matches the optimal rate. However, when more than half of each new batch is generated, the rate of convergence becomes arbitrarily slow as the fraction of synthetic data increases. This result motivates the ongoing research on the detection and curation of generated samples: it shows that for practical finite t t t in the high-synthetic regime, synthetic data detection can substantially improve estimation accuracy.
Related work
Existing works have shown that under the Replace workflow, iterative retraining degrades without bound [22 , 4 , 14 ] , whereas under the Accumulate workflow the degradation remains bounded [10 , 14 , 21 ] .
Several works show that collapse can be avoided, but under restrictive conditions. Bertrand et al. [3 ] prove the stability of iterative retraining on a mixture of real and synthetic data, but only locally, when the initial model is sufficiently close to the data distribution and the proportion of real data is sufficiently large. Barzilai & Shamir [2 ] provide non-asymptotic guarantees under data accumulation; however, for a fixed sample size n n n , their bound holds only up to a finite number of iterations t t t that depends on n n n , and thus does not cover the regime of fixed n n n with t → ∞ t \to\infty t → ∞ . Moreover, beyond an initial (or fixed) real dataset, these works do not incorporate new real data at subsequent iterations.
In practice, data scraped from the web at each iteration contains new real data alongside generated data. Several works study this setting [1 , 11 , 8 ] , but either discard earlier data or assume that real and generated samples can be told apart, e.g., in order to weight them separately. The latter is unrealistic, since reliably detecting generated content remains an open problem [19 , 16 ] , although detection, when possible, can substantially mitigate collapse [7 ] . To the best of our knowledge, no prior work analyzes iterative retraining with accumulated data in which fresh real samples are not distinguished from generated ones.
Theory
In this section, we formalize the iterative retraining process, define the corresponding mean and covariance estimators, and state our main theoretical results, which characterize when the estimators converge to the true parameters and the rate at which this convergence occurs. Finally, we validate our results through experiments and provide additional insights into the rate of decay.
Problem setup: data generating process and estimators
In the theoretical model, we consider the accumulate method, where the process is initialized with a dataset of real samples. Similar to a vast body of theoretical work on model collapse [22 , 1 , 14 , 4 , 11 , 5 ] , we assume that the real data distribution is a multivariate Gaussian distribution. At each iteration, the training dataset is augmented with both generated and real data. Since the model does not know which samples are real and which are generated, it treats all samples identically and estimates the underlying distribution using the entire accumulated dataset. Formally, at each iteration, each newly added sample is drawn from the real distribution with some probability and from the current estimated distribution with the complementary probability.
Data-generating process. Let μ ∈ R d \mu\in\mathbb{R}^{d} μ ∈ R d , and let Σ ∈ R d × d \Sigma\in\mathbb{R}^{d \times d} Σ ∈ R d × d be symmetric positive-definite. The process unfolds in discrete iterations t = 1 , 2 , 3 , … t = 1, 2, 3, \ldots t = 1 , 2 , 3 , …
Iteration 1 (real data). Observe n r ≥ 1 n_r \ge1 n r ≥ 1 i.i.d. samples { x j ( 0 ) } j = 1 n r ∼ N ( μ , Σ ) \{x_j^{(0)}\}_{j=1}^{n_r} \sim\mathcal{N}(\mu, \Sigma) { x j ( 0 ) } j = 1 n r ∼ N ( μ , Σ ) and form the empirical mean and covariance
μ ( 1 ) = 1 n r ∑ j = 1 n r x j ( 0 ) , Σ ( 1 ) = 1 n r ∑ j = 1 n r ( x j ( 0 ) − μ ( 1 ) ) ( x j ( 0 ) − μ ( 1 ) ) T . (1) \mu^{(1)} = \frac{1}{n_r} \sum_{j=1}^{n_r} x_j^{(0)}, \qquad\Sigma^{(1)} = \frac{1}{n_r} \sum_{j=1}^{n_r} (x_j^{(0)} - \mu^{(1)})(x_j^{(0)} - \mu^{(1)})^{\mathsf{T}}. \tag*{(1)} μ ( 1 ) = n r 1 j = 1 ∑ n r x j ( 0 ) , Σ ( 1 ) = n r 1 j = 1 ∑ n r ( x j ( 0 ) − μ ( 1 ) ) ( x j ( 0 ) − μ ( 1 ) ) T . (1)
Iteration t ≥ 2 t \ge2 t ≥ 2 (mixed data). Given μ ( t − 1 ) \mu^{(t-1)} μ ( t − 1 ) and Σ ( t − 1 ) \Sigma^{(t-1)} Σ ( t − 1 ) , draw n b ( t − 1 ) ≥ 1 n_b(t-1) \ge1 n b ( t − 1 ) ≥ 1 i.i.d. samples from the mixture
x i ( t − 1 ) ∼ α t − 1 N ( μ ( t − 1 ) , Σ ( t − 1 ) ) + ( 1 − α t − 1 ) N ( μ , Σ ) , α t − 1 ∈ [ 0 , 1 ] , (2) x_i^{(t-1)} \sim\alpha_{t-1}\mathcal{N}\left(\mu^{(t-1)}, \Sigma^{(t-1)}\right) + (1-\alpha_{t-1})\mathcal{N}(\mu, \Sigma), \qquad\alpha_{t-1} \in[0,1], \tag*{(2)} x i ( t − 1 ) ∼ α t − 1 N ( μ ( t − 1 ) , Σ ( t − 1 ) ) + ( 1 − α t − 1 ) N ( μ , Σ ) , α t − 1 ∈ [ 0 , 1 ] , (2)
where 1 ≤ i ≤ n b ( t − 1 ) 1 \le i \le n_b(t-1) 1 ≤ i ≤ n b ( t − 1 ) . The parameter α t − 1 ∈ [ 0 , 1 ] \alpha_{t-1} \in[0,1] α t − 1 ∈ [ 0 , 1 ] controls the fraction of the new batch that is drawn from the current model. Specifically, α t − 1 = 1 \alpha_{t-1} = 1 α t − 1 = 1 means the new batch is entirely synthetic, while α t − 1 = 0 \alpha_{t-1} = 0 α t − 1 = 0 means it is entirely real. Denote the total number of samples at iteration t t t by N t N_t N t and the inverse ratio of the growth by ρ t \rho_t ρ t :
N t : = n r + ∑ s = 1 t − 1 n b ( s ) , ρ t : = N t N t + 1 ( t ≥ 1 ) . (3) N_t := n_r + \sum_{s=1}^{t-1} n_b(s), \qquad\rho_t := \frac{N_t}{N_{t+1}} \qquad(t \ge1). \tag*{(3)} N t := n r + s = 1 ∑ t − 1 n b ( s ) , ρ t := N t + 1 N t ( t ≥ 1 ) . (3)
Note that ρ t ∈ ( 0 , 1 ) \rho_t \in(0,1) ρ t ∈ ( 0 , 1 ) for every t ≥ 1 t \ge1 t ≥ 1 .
Estimators. At every round, the model is refit using the pooled empirical mean and covariance of all samples accumulated so far. For every t ≥ 1 t \ge1 t ≥ 1 , we write y 1 , … , y N t y_1,\ldots,y_{N_t} y 1 , … , y N t for the N t N_t N t accumulated points, consisting of the original real batch { x j ( 0 ) } j = 1 n r \{x_j^{(0)}\}_{j=1}^{n_r} { x j ( 0 ) } j = 1 n r and the accumulated batches { x i ( s ) } i = 1 n b ( s ) \{x_i^{(s)}\}_{i=1}^{n_b(s)} { x i ( s ) } i = 1 n b ( s ) , 1 ≤ s ≤ t − 1 1 \le s \le t-1 1 ≤ s ≤ t − 1 . Define the pooled empirical estimators:
μ ( t ) : = 1 N t ∑ k = 1 N t y k , Σ ( t ) : = 1 N t ∑ k = 1 N t ( y k − μ ( t ) ) ( y k − μ ( t ) ) T , (4) \mu^{(t)} := \frac{1}{N_t}\sum_{k=1}^{N_t} y_k, \qquad\Sigma^{(t)} := \frac{1}{N_t}\sum_{k=1}^{N_t}(y_k-\mu^{(t)})(y_k-\mu^{(t)})^{\mathsf{T}}, \tag*{(4)} μ ( t ) := N t 1 k = 1 ∑ N t y k , Σ ( t ) := N t 1 k = 1 ∑ N t ( y k − μ ( t ) ) ( y k − μ ( t ) ) T , (4)
which for t = 1 t=1 t = 1 coincides with (1). These iterative estimators are standard and widely used in the literature [22 , 14 ] . Conveniently, they admit online recursions in terms of the previous estimates and the newly drawn batch. Finally, we write
D t : = N ( μ ( t ) , Σ ( t ) ) , D r e a l : = N ( μ , Σ ) , (5) \mathcal{D}_t := \mathcal{N}(\mu^{(t)},\Sigma^{(t)}), \qquad\mathcal{D}_{\mathrm{real}} := \mathcal{N}(\mu,\Sigma), \tag*{(5)} D t := N ( μ ( t ) , Σ ( t ) ) , D real := N ( μ , Σ ) , (5)
for the estimated distribution at iteration t t t and the true data distribution, respectively. We measure the discrepancy between them by the 2-Wasserstein distance, W 2 ( ⋅ , ⋅ ) W_2(\cdot,\cdot) W 2 ( ⋅ , ⋅ ) , defined in Appendix § 7 .4.4, which has also been used by Shumailov et al. [22 ] . For μ ( t ) \mu^{(t)} μ ( t ) and Σ ( t ) \Sigma^{(t)} Σ ( t ) themselves, we measure the discrepancy from the true values by the Euclidean norm, ∥ ⋅ ∥ 2 \lVert\cdot\rVert_2 ∥ ⋅ ∥ 2 , and the Frobenius norm, ∥ ⋅ ∥ F \lVert\cdot\rVert_F ∥ ⋅ ∥ F .
Finally, we state our standing assumption on how new samples are generated.
Assumption 1 (Conditional independence). For every t ≥ 2 t \ge2 t ≥ 2 , conditionally on ( μ ( t − 1 ) , Σ ( t − 1 ) ) (\mu^{(t-1)},\Sigma^{(t-1)}) ( μ ( t − 1 ) , Σ ( t − 1 ) ) , the new samples { x i ( t − 1 ) } i = 1 n b ( t − 1 ) \{x_i^{(t-1)}\}_{i=1}^{n_b(t-1)} { x i ( t − 1 ) } i = 1 n b ( t − 1 ) are drawn i.i.d. from the mixture (2) and are independent of all previously drawn samples { x i ( s ) } \{x_i^{(s)}\} { x i ( s ) } , 0 ≤ s ≤ t − 2 0 \le s \le t-2 0 ≤ s ≤ t − 2 .
In words, the past influences a new batch only through the current fitted model. This is not a restriction on the setting, but a formalization of the data-generating process described above.
Theoretical results
In this section, we present our key theoretical results. In Section 3.2.1, we establish that the mean estimator is unbiased and derive an explicit formula for the bias of the covariance estimator. In Section 3.2.2, we provide an equivalent (if and only if) condition for the convergence of the estimated parameters to the truth, and hence for the convergence of the estimated distribution to the true distribution. We then show that the model converges when the fraction of real data is fixed and arbitrarily small, as well as when the fraction of real data tends to zero sufficiently slowly. In Section 3.2.3, we analyze the convergence rate of the mean estimator.
The proofs for all the claims are deferred to Appendix § 7 .
Bias of estimators and covariance of the mean estimator
Here, we derive the bias of each estimator and the covariance of the mean estimator. First, let us denote by C t C_t C t the covariance of the mean estimator
C t : = Cov ( μ ( t ) ) = E [ ( μ ( t ) − E [ μ ( t ) ] ) ( μ ( t ) − E [ μ ( t ) ] ) T ] , (6) C_t := \operatorname{Cov}(\mu^{(t)}) = \mathbb{E}\left[(\mu^{(t)}-\mathbb{E}[\mu^{(t)}])(\mu^{(t)}-\mathbb{E}[\mu^{(t)}])^{\mathsf{T}}\right], \tag*{(6)} C t := Cov ( μ ( t ) ) = E [ ( μ ( t ) − E [ μ ( t ) ]) ( μ ( t ) − E [ μ ( t ) ] ) T ] , (6)
and by δ ( t ) \delta^{(t)} δ ( t ) the deviation of the mean estimator from the truth:
δ ( t ) : = μ ( t ) − μ . (7) \delta^{(t)} := \mu^{(t)}-\mu. \tag*{(7)} δ ( t ) := μ ( t ) − μ . (7)
We now establish the unbiasedness of the mean estimator and the bias of the covariance estimator.
Theorem 3.1 (Bias of estimators). *Under the setup of Section 3.1, for all t ≥ 1 t \ge1 t ≥ 1
(a) The mean estimator is unbiased:
E [ μ ( t ) ] = μ . (8) \mathbb{E}[\mu^{(t)}] = \mu. \tag*{(8)} E [ μ ( t ) ] = μ . (8)
(b) The bias of the covariance estimator is:
E [ Σ ( t ) ] = Σ − C t . (9) \mathbb{E}[\Sigma^{(t)}] = \Sigma- C_t. \tag*{(9)} E [ Σ ( t ) ] = Σ − C t . (9)
Proof idea. Part (a) follows by induction using the recursion formula obtained for μ ( t ) \mu^{(t)} μ ( t ) (Eq. (29) in the appendix). For part (b), we write Σ ( t ) \Sigma^{(t)} Σ ( t ) as the empirical second moment around the true mean μ \mu μ minus δ ( t ) δ ( t ) T \delta^{(t)}\delta^{(t)\mathsf{T}} δ ( t ) δ ( t ) T (Lemma A.2 ), and then use induction.
Remark. The covariance estimator is negatively biased by exactly the covariance of the mean estimator C t ⪰ 0 C_t \succeq0 C t ⪰ 0 (‘⪰ 0 \succeq0 ⪰ 0 ’ should be understood in the positive semi-definite sense). In particular, since C 1 = Σ / n r C_1 = \Sigma/n_r C 1 = Σ/ n r , we have E [ Σ ( 1 ) ] = n r − 1 n r Σ \mathbb{E}[\Sigma^{(1)}] = \frac{n_r-1}{n_r}\Sigma E [ Σ ( 1 ) ] = n r n r − 1 Σ , recovering the familiar downward bias of the maximum likelihood covariance estimator under Gaussian distribution. We now show that this bias never exceeds twice its value at t = 1 t = 1 t = 1 , regardless of how much generated data is used.
Lemma 3.2 (Bound on C t C_t C t ). Under the setup of § 3.1 , for all t ≥ 1 t \ge1 t ≥ 1 :
0 ⪯ C t ⪯ 2 n r Σ . 0 \preceq C_t \preceq\frac{2}{n_r}\Sigma. 0 ⪯ C t ⪯ n r 2 Σ.
Remark. By Theorem 3.1 (b), the upper bound in Lemma 3.2 is equivalent to E [ Σ ( t ) ] ⪰ ( 1 − 2 n r ) Σ \mathbb{E}[\Sigma^{(t)}] \succeq\left(1-\frac{2}{n_r}\right)\Sigma E [ Σ ( t ) ] ⪰ ( 1 − n r 2 ) Σ : the downward bias of the covariance estimator is at most twice its value at t = 1 t = 1 t = 1 . Note that for n r = 1 n_r = 1 n r = 1 the bound is weaker than C t ⪯ Σ C_t \preceq\Sigma C t ⪯ Σ , which also holds for every t ≥ 1 t \ge1 t ≥ 1 : it follows from Theorem 3.1 (b) and Σ ( t ) ⪰ 0 \Sigma^{(t)} \succeq0 Σ ( t ) ⪰ 0 (by construction).
Since μ ( t ) \mu^{(t)} μ ( t ) is unbiased (Theorem 3.1 (a)), from (6) we have that
C t = E [ ( μ ( t ) − μ ) ( μ ( t ) − μ ) T ] = E [ δ ( t ) δ ( t ) T ] . (10) C_t = \mathbb{E}\left[(\mu^{(t)}-\mu)(\mu^{(t)}-\mu)^{\mathsf{T}}\right] = \mathbb{E}\left[\delta^{(t)}\delta^{(t)\mathsf{T}}\right]. \tag*{(10)} C t = E [ ( μ ( t ) − μ ) ( μ ( t ) − μ ) T ] = E [ δ ( t ) δ ( t ) T ] . (10)
Using ∥ v ∥ 2 2 = Tr ( v v T ) \lVert v\rVert_2^2 = \operatorname{Tr}(vv^{\mathsf{T}}) ∥ v ∥ 2 2 = Tr ( v v T ) , the linearity of the trace and expectation, and (10),
E [ ∥ μ ( t ) − μ ∥ 2 2 ] = E [ ∥ δ ( t ) ∥ 2 2 ] = E [ Tr ( δ ( t ) δ ( t ) T ) ] = Tr ( E [ δ ( t ) δ ( t ) T ] ) = Tr ( C t ) . (11) \mathbb{E}\left[\lVert\mu^{(t)}-\mu\rVert_2^2\right] = \mathbb{E}\left[\lVert\delta^{(t)}\rVert_2^2\right] = \mathbb{E}\left[\operatorname{Tr}\left(\delta^{(t)}\delta^{(t)\mathsf{T}}\right)\right] = \operatorname{Tr}\left(\mathbb{E}\left[\delta^{(t)}\delta^{(t)\mathsf{T}}\right]\right) = \operatorname{Tr}(C_t). \tag*{(11)} E [ ∥ μ ( t ) − μ ∥ 2 2 ] = E [ ∥ δ ( t ) ∥ 2 2 ] = E [ Tr ( δ ( t ) δ ( t ) T ) ] = Tr ( E [ δ ( t ) δ ( t ) T ] ) = Tr ( C t ) . (11)
Thus, Tr ( C t ) \operatorname{Tr}(C_t) Tr ( C t ) is the mean-squared error of μ ( t ) \mu^{(t)} μ ( t ) around μ \mu μ .
Convergence to the true distribution
Crucially, we analyze in closed form when this iterative estimation process converges to the true distribution. Below, we establish that the condition for convergence of the estimated parameters to the truth collapses to a single elementary series in { α t } t ≥ 1 \{\alpha_t\}_{t\ge1} { α t } t ≥ 1 , { n b ( t ) } t ≥ 1 \{n_b(t)\}_{t\ge1} { n b ( t ) } t ≥ 1 , and n r n_r n r . Recall from (3) that ρ t : = N t / N t + 1 \rho_t := N_t/N_{t+1} ρ t := N t / N t + 1 .
Theorem 3.3 (Sharp elementary criterion for consistency). Under the setup of § 3.1 , the following are equivalent:
(a) The following series diverges:
∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∑ t = 1 ∞ ( 1 − α t ) n b ( t ) N t + 1 = ∞ . (12) \sum_{t=1}^{\infty}(1-\alpha_t)(1-\rho_t) = \sum_{t=1}^{\infty}(1-\alpha_t)\frac{n_b(t)}{N_{t+1}} = \infty. \tag*{(12)} t = 1 ∑ ∞ ( 1 − α t ) ( 1 − ρ t ) = t = 1 ∑ ∞ ( 1 − α t ) N t + 1 n b ( t ) = ∞. (12) (b) The mean estimator μ ( t ) \mu^{(t)} μ ( t ) converges to μ \mu μ in L 2 L^2 L 2 , i.e., in mean square:
E [ ∥ μ ( t ) − μ ∥ 2 2 ] → t → ∞ 0. (13) \mathbb{E}\left[\lVert\mu^{(t)}-\mu\rVert_2^2\right] \xrightarrow{t\to\infty} 0. \tag*{(13)} E [ ∥ μ ( t ) − μ ∥ 2 2 ] t → ∞ 0. (13) (c) The covariance estimator converges to Σ \Sigma Σ in expected Frobenius norm:
E [ ∥ Σ ( t ) − Σ ∥ F ] → t → ∞ 0. (14) \mathbb{E}\left[\lVert\Sigma^{(t)}-\Sigma\rVert_F\right] \xrightarrow{t\to\infty} 0. \tag*{(14)} E [ ∥ Σ ( t ) − Σ ∥ F ] t → ∞ 0. (14) (d) The expected squared 2-Wasserstein distance between the estimated distribution and the true distribution tends to zero:
E [ W 2 2 ( D t , D r e a l ) ] → t → ∞ 0. (15) \mathbb{E}\left[W_2^2(D_t,D_{\mathrm{real}})\right] \xrightarrow{t\to\infty} 0. \tag*{(15)} E [ W 2 2 ( D t , D real ) ] t → ∞ 0. (15)
Proof idea. Since E [ ∥ μ ( t ) − μ ∥ 2 2 ] = Tr ( C t ) \mathbb{E}[\lVert\mu^{(t)}-\mu\rVert_{2}^{2}]=\operatorname{Tr}(C_t) E [∥ μ ( t ) − μ ∥ 2 2 ] = Tr ( C t ) , we derive a recursion for C t C_t C t (Theorem A.11 ) and use it to show that C t → 0 C_t\to0 C t → 0 if and only if (12) holds. For the covariance, a similar recursion for the empirical second moment around the true mean gives sufficiency, while necessity follows from the bias − C t -C_t − C t (Theorem 3.1 ). For W 2 W_2 W 2 , the Gaussian closed form splits W 2 2 ( D t , D r e a l ) W_2^2(\mathcal{D}_t,\mathcal{D}_{\mathrm{real}}) W 2 2 ( D t , D real ) into ∥ μ ( t ) − μ ∥ 2 2 \lVert\mu^{(t)}-\mu\rVert_2^2 ∥ μ ( t ) − μ ∥ 2 2 plus a nonnegative term bounded by d ∥ Σ ( t ) − Σ ∥ F \sqrt{d}\lVert\Sigma^{(t)}-\Sigma\rVert_F d ∥ Σ ( t ) − Σ ∥ F . Taking expectations yields sufficiency from the mean and covariance results, and necessity from the mean result alone.
Remark (interpretation). The factor 1 − α t 1-\alpha_t 1 − α t is the fraction of the batch added at round t t t that is drawn from the true distribution, while 1 − ρ t = n b ( t ) / N t + 1 1-\rho_t=n_b(t)/N_{t+1} 1 − ρ t = n b ( t ) / N t + 1 is the weight this batch carries in the pool. Their product is therefore the share of genuinely fresh real information injected at round t t t , and the model escapes collapse precisely when these shares are non-summable. The convergences of the mean, the covariance and the fitted distribution hold or fail together, and, by Markov’s inequality, each implies convergence in probability.
Discussion (implications). In the remainder of this section, we characterize when the series ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) \sum_{t=1}^{\infty}(1-\alpha_t)(1-\rho_t) ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) diverges, which, using Theorem 3.3 , allows us to determine when the iterative process converges to the true parameters.
We start by providing a sufficient condition for the mixing fraction for convergence.
Lemma 3.4 (Sufficient condition on the mixing fraction). If sup t ≥ 1 α t < 1 \sup_{t\ge1}\alpha_t<1 sup t ≥ 1 α t < 1 , then
∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ (16) \sum_{t=1}^{\infty}(1-\alpha_t)(1-\rho_t)=\infty \tag*{(16)} t = 1 ∑ ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ (16)
In particular, for a fixed mixing fraction α t ≡ α ∈ [ 0 , 1 ) \alpha_t\equiv\alpha\in[0,1) α t ≡ α ∈ [ 0 , 1 ) , the condition of Lemma 3.4 holds. Thus, any fixed positive fraction 1 − α 1-\alpha 1 − α of real data in each batch suffices for consistency, even when the real samples added at each iteration are vastly outnumbered by the generated ones. This is encouraging in practice: scraping the web naturally yields some new real data alongside generated data, and any fixed fraction of it suffices for convergence.
The next case we consider is an asymptotically vanishing fraction of real data. We show that the model can still be consistent, provided that the synthetic fraction tends to 1 slowly enough. Specifically, take n b ( t ) ≡ n b ≥ 1 n_b(t)\equiv n_b\ge1 n b ( t ) ≡ n b ≥ 1 , so that
N t + 1 = n r + n b t ⟹ 1 − ρ t = n b N t + 1 ≍ 1 t . N_{t+1}=n_r+n_bt\quad\Longrightarrow\quad1-\rho_t=\frac{n_b}{N_{t+1}}\asymp\frac{1}{t}. N t + 1 = n r + n b t ⟹ 1 − ρ t = N t + 1 n b ≍ t 1 .
Here, f ( t ) ≍ g ( t ) f(t)\asymp g(t) f ( t ) ≍ g ( t ) means that for all sufficiently large t t t , the ratio f ( t ) / g ( t ) f(t)/g(t) f ( t ) / g ( t ) is bounded above and below by positive constants (independent of t t t ). The criterion (12) for convergence of the estimated parameters to the true parameters then reads ∑ t = 1 ∞ 1 − α t t = ∞ \sum_{t=1}^{\infty}\frac{1-\alpha_t}{t}=\infty ∑ t = 1 ∞ t 1 − α t = ∞ . For mixing fractions { α t } t = 1 ∞ \{\alpha_t\}_{t=1}^{\infty} { α t } t = 1 ∞ tending to 1 from below, this gives
α t = 1 − 1 t , α t = 1 − 1 t ⟹ ∑ t = 1 ∞ 1 − α t t < ∞ , \alpha_t=1-\frac{1}{t},\quad\alpha_t=1-\frac{1}{\sqrt{t}}\quad\Longrightarrow\quad\sum_{t=1}^{\infty}\frac{1-\alpha_t}{t}<\infty, α t = 1 − t 1 , α t = 1 − t 1 ⟹ t = 1 ∑ ∞ t 1 − α t < ∞ ,
α t = 1 − 1 ln ( t + 2 ) , α t = 1 − 1 ln ln ( t + 16 ) ⟹ ∑ t = 1 ∞ 1 − α t t = ∞ . \alpha_t=1-\frac{1}{\ln(t+2)},\quad\alpha_t=1-\frac{1}{\ln\ln(t+16)}\quad\Longrightarrow\quad\sum_{t=1}^{\infty}\frac{1-\alpha_t}{t}=\infty. α t = 1 − ln ( t + 2 ) 1 , α t = 1 − ln ln ( t + 16 ) 1 ⟹ t = 1 ∑ ∞ t 1 − α t = ∞.
Thus, consistency holds in the last two cases, despite the fraction of real data tending to zero as t → ∞ t\to\infty t → ∞ . Note that all four sequences satisfy sup t ≥ 1 α t = 1 \sup_{t\ge1}\alpha_t=1 sup t ≥ 1 α t = 1 , so Lemma 3.4 covers none of them.
Finally, we consider the case α t ≡ 1 \alpha_t\equiv1 α t ≡ 1 , in which no real samples are added at any iteration. Then, ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = 0 \sum_{t=1}^{\infty}(1-\alpha_t)(1-\rho_t)=0 ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = 0 , so the condition from Theorem 3.3 is not satisfied. Thus, the estimated parameters do not converge to the true parameters, consistent with existing results in the literature [5 ] . In a purely replacement-based scheme, the expected squared 2-Wasserstein distance W 2 W_2 W 2 can diverge [22 ] . In contrast, for our accumulation-based scheme, we next show that the estimators neither converge to the true parameters nor diverge. This is consistent with prior work showing that accumulating data keeps the error bounded [10 , 14 ] . Here, we provide explicit two-sided bounds in terms of n r n_r n r . This highlights a fundamental difference between the two schemes.
Proposition 3.5 (Bounded error at α = 1 \alpha= 1 α = 1 ). Under the setup of § 3.1 , if α t ≡ 1 \alpha_t \equiv1 α t ≡ 1 , then
Tr ( Σ ) n r ≤ E [ ∥ μ ( t ) − μ ∥ 2 2 ] ≤ 2 Tr ( Σ ) n r \frac{\operatorname{Tr}(\Sigma)}{n_r} \leq\mathbb{E}\left[\left\|\mu^{(t)}-\mu\right\|_2^2\right] \leq\frac{2\operatorname{Tr}(\Sigma)}{n_r} n r Tr ( Σ ) ≤ E [ μ ( t ) − μ 2 2 ] ≤ n r 2 Tr ( Σ ) ∥ Σ ∥ F n r ≤ E [ ∥ Σ ( t ) − Σ ∥ F ] ≤ 2 Tr ( Σ ) n r + 1 n r A Σ + 3 n r exp ( 3 n r ) B Σ \frac{\|\Sigma\|_F}{n_r} \leq\mathbb{E}\left[\left\|\Sigma^{(t)}-\Sigma\right\|_F\right] \leq\frac{2\operatorname{Tr}(\Sigma)}{n_r}+\frac{1}{\sqrt{n_r}}\sqrt{A_\Sigma+\frac{3}{n_r}\exp\left(\frac{3}{n_r}\right)B_\Sigma} n r ∥Σ ∥ F ≤ E [ Σ ( t ) − Σ F ] ≤ n r 2 Tr ( Σ ) + n r 1 A Σ + n r 3 exp ( n r 3 ) B Σ for every t ≥ 1 t \geq1 t ≥ 1 , where A Σ = 5 Tr ( Σ ) 2 + 3 ∥ Σ ∥ F 2 A_\Sigma=5\operatorname{Tr}(\Sigma)^2+3\|\Sigma\|_F^2 A Σ = 5 Tr ( Σ ) 2 + 3∥Σ ∥ F 2 and B Σ = 5 Tr ( Σ ) 2 + 4 ∥ Σ ∥ F 2 B_\Sigma=5\operatorname{Tr}(\Sigma)^2+4\|\Sigma\|_F^2 B Σ = 5 Tr ( Σ ) 2 + 4∥Σ ∥ F 2 .
Proof idea. For the mean, we use the recursion for C t C_t C t (Theorem A.11 ) to show that Tr ( C t ) \operatorname{Tr}(C_t) Tr ( C t ) is non-decreasing in t t t , starting from Tr ( C 1 ) = Tr ( Σ ) / n r \operatorname{Tr}(C_1)=\operatorname{Tr}(\Sigma)/n_r Tr ( C 1 ) = Tr ( Σ ) / n r , and bounded. For the covariance, the lower bound follows from the bias − C t -C_t − C t (Theorem 3.1 ), and the upper bound combines the bound on Tr ( C t ) \operatorname{Tr}(C_t) Tr ( C t ) with an upper bound on the error of the empirical second moment around the true mean.
Rate of convergence
We have shown that, for every fixed α < 1 \alpha< 1 α < 1 , the estimators are consistent. This result concerns the limit of infinitely many iterations. In practical training scenarios, however, only a finite number of iterations is performed. We, therefore, ask: how quickly do the estimated parameters converge to the true parameters? Below, we establish the rate of convergence for the mean estimator. In § 3.3 , we show similar behavior for the covariance estimator.
The next result gives the exact convergence rate and reveals a sharp transition at α = 1 2 \alpha= \frac{1}{2} α = 2 1 .
Theorem 3.6 (Rate of decay). Assume the setup of § 3.1 , α t ≡ α ∈ [ 0 , 1 ] \alpha_t \equiv\alpha\in[0,1] α t ≡ α ∈ [ 0 , 1 ] and n b ( t ) ≡ n b n_b(t) \equiv n_b n b ( t ) ≡ n b . Then
E [ ∥ μ ( t ) − μ ∥ 2 2 ] ≍ { N t − 1 , α < 1 2 , N t − 1 ln N t , α = 1 2 , N t − 2 ( 1 − α ) , α > 1 2 , (17) \mathbb{E}\left[\left\|\mu^{(t)}-\mu\right\|_2^2\right]\asymp
\begin{cases}
N_t^{-1}, & \alpha< \frac{1}{2},\\
N_t^{-1}\ln N_t, & \alpha= \frac{1}{2},\\
N_t^{-2(1-\alpha)}, & \alpha> \frac{1}{2},
\end{cases} \tag*{(17)} E [ μ ( t ) − μ 2 2 ] ≍ ⎩ ⎨ ⎧ N t − 1 , N t − 1 ln N t , N t − 2 ( 1 − α ) , α < 2 1 , α = 2 1 , α > 2 1 , (17) where N t = n r + ( t − 1 ) n b ≍ t N_t=n_r+(t-1)n_b\asymp t N t = n r + ( t − 1 ) n b ≍ t . The implied constants depend only on α \alpha α , n b n_b n b , n r n_r n r and Tr ( Σ ) \operatorname{Tr}(\Sigma) Tr ( Σ ) .
Proof idea. We derive the asymptotic rate of each term in the closed form of C t C_t C t (Eq. (38) in the appendix) by comparing sums with integrals.
Remark (interpretation). For α < 1 2 \alpha< \frac{1}{2} α < 2 1 , the estimator attains the parametric rate N t − 1 N_t^{-1} N t − 1 of the mean of N t N_t N t i.i.d. real samples: synthetic data inflates the constant but not the exponent. At α = 1 2 \alpha= \frac{1}{2} α = 2 1 a phase transition occurs, and for α > 1 2 \alpha> \frac{1}{2} α > 2 1 the exponent 2 ( 1 − α ) 2(1-\alpha) 2 ( 1 − α ) decreases linearly to 0 as α → 1 \alpha\to1 α → 1 . Consistency thus survives for every α < 1 \alpha< 1 α < 1 , but becomes arbitrarily slow: halving the error requires multiplying N t N_t N t by 2 1 / [ 2 ( 1 − α ) ] 2^{1/[2(1-\alpha)]} 2 1/ [ 2 ( 1 − α )] , a factor of 32 at α = 0.9 \alpha= 0.9 α = 0.9 and of 10 15 10^{15} 1 0 15 at α = 0.99 \alpha= 0.99 α = 0.99 . Equivalently, the pooled estimator is only as accurate as the mean of order N t 2 ( 1 − α ) N_t^{2(1-\alpha)} N t 2 ( 1 − α ) i.i.d. real samples, a vanishing fraction of N t N_t N t . At α = 1 \alpha= 1 α = 1 , the rate reads N t − 2 ( 1 − α ) = 1 N_t^{-2(1-\alpha)}=1 N t − 2 ( 1 − α ) = 1 : the error does not vanish (Theorem 3.3 ), but it does not diverge either (Proposition 3.5 ).
Remark (the value of identifying real samples). The gain from perfect detection of synthetic samples, i.e., from keeping only the real samples, stays bounded for α < 1 2 \alpha< \frac{1}{2} α < 2 1 , but grows like N t 2 α − 1 N_t^{2\alpha-1} N t 2 α − 1 for α > 1 2 \alpha> \frac{1}{2} α > 2 1 , i.e., faster as the fraction of generated samples increases. For large but finite N t N_t N t , it is largest at some α \alpha α strictly between 0 and 1, i.e., when generated data dominates but real data is still being added in meaningful amounts (see Appendix § 8 for more discussion).
Empirical verification
In this section, we simulate the theoretical setup in order to further support our theoretical results and also gain more insights into the model, e.g., the rate of convergence of the covariance estimator.
Setup. We fix d = 4 d=4 d = 4 , μ = ( − 1 , − 1 3 , 1 3 , 1 ) T \mu=(-1,-\frac{1}{3},\frac{1}{3},1)^\mathsf{T} μ = ( − 1 , − 3 1 , 3 1 , 1 ) T , and [ Σ ] i j = s i s j 2 − ∣ i − j ∣ [\Sigma]_{ij}=s_i s_j 2^{-|i-j|} [ Σ ] ij = s i s j 2 − ∣ i − j ∣ with s = ( 1 , 4 3 , 5 3 , 2 ) T s=(1,\frac{4}{3},\frac{5}{3},2)^\mathsf{T} s = ( 1 , 3 4 , 3 5 , 2 ) T , so that Tr ( Σ ) = 9.56 \operatorname{Tr}(\Sigma)=9.56 Tr ( Σ ) = 9.56 , ∥ Σ ∥ F = 6.18 \|\Sigma\|_F=6.18 ∥Σ ∥ F = 6.18 , and Σ \Sigma Σ has a condition number 9.58. Additionally, we take n b ( t ) ≡ n b = 10 n_b(t)\equiv n_b=10 n b ( t ) ≡ n b = 10 and n r = 20 n_r=20 n r = 20 . Two families of mixing fractions are used: a grid of 15 constant values α ∈ { 0 , 0.1 , 0.2 , 0.3 , 0.4 , 0.45 , 0.5 , 0.55 , 0.6 , 0.7 , 0.8 , 0.9 , 0.95 , 0.99 , 1 } \alpha\in\{0,0.1,0.2,0.3,0.4,0.45,0.5,0.55,0.6,0.7,0.8,0.9,0.95,0.99,1\} α ∈ { 0 , 0.1 , 0.2 , 0.3 , 0.4 , 0.45 , 0.5 , 0.55 , 0.6 , 0.7 , 0.8 , 0.9 , 0.95 , 0.99 , 1 } , spanning the transition of Theorem 3.6 and both endpoints of Theorem 3.3, and the four sequences α t = 1 − 1 t \alpha_t=1-\frac{1}{t} α t = 1 − t 1 , 1 − 1 t 1-\frac{1}{\sqrt{t}} 1 − t 1 , 1 − 1 ln ( t + 2 ) 1-\frac{1}{\ln(t+2)} 1 − l n ( t + 2 ) 1 , 1 − 1 ln ln ( t + 16 ) 1-\frac{1}{\ln\ln(t+16)} 1 − l n l n ( t + 16 ) 1 , all with sup t ≥ 1 α t = 1 \sup_{t\geq1}\alpha_t=1 sup t ≥ 1 α t = 1 . Each new sample is drawn independently from the mixture (2), and in each iteration we update ( μ ( t ) , Σ ( t ) ) (\boldsymbol{\mu}^{(t)},\boldsymbol{\Sigma}^{(t)}) ( μ ( t ) , Σ ( t ) ) according to the iterative estimation procedure. Each regime is run for T = 10 5 T=10^{5} T = 1 0 5 iterations (N T ≈ 10 6 N_T\approx10^{6} N T ≈ 1 0 6 ). Unless stated otherwise, expectations are estimated by averages over M = 20,000 M=20{,}000 M = 20 , 000 independent runs, each drawing its own n r n_r n r initial real samples. At this M M M , the Monte-Carlo standard error of the averaged mean-square error is about 1% of its value, below the resolution of the figures (see also Appendix C).
Convergence at fixed α \alpha α . Every curve in Figure 1 is a Monte-Carlo average over the M = 20,000 M=20{,}000 M = 20 , 000 runs. The exception is W 2 2 W_2^2 W 2 2 , computed from its closed form for Gaussians ((94) and (102)): since each evaluation requires a matrix square root, we average it over 10,000 of the runs and evaluate it at 55 logarithmically spaced iterations rather than at every iteration. The top panels (a)–(c) illustrate Theorem 3.3 for constant α \alpha α : for every 0 ≤ α < 1 0\leq\alpha<1 0 ≤ α < 1 , the errors of the mean (a), the covariance (b) and the fitted distribution (c) decrease with N t N_t N t , whereas at α ≡ 1 \alpha\equiv1 α ≡ 1 , when no real data is added after the first iteration, they level off at a positive value (Proposition 3.5 ). Thus, when the real-data fraction is fixed, consistency depends only on whether real data is present at all. Its proportion governs the rate instead (Theorem 3.6 ): the decay slows as α \alpha α grows, and becomes very slow as α → 1 \alpha\to1 α → 1 . At α = 0.99 \alpha= 0.99 α = 0.99 the mean-square error decays only like N t − 0.02 N_t^{-0.02} N t − 0.02 , too slowly to approach zero within the simulated horizon. For large α \alpha α , the mean-square error also first increases, as predicted by an additional result (Proposition A.27 ) that we deferred to Appendix § 7 .9 due to space limitations: above α ∗ ( n r , n b ) ≈ 0.65 \alpha^{*}(n_r,n_b) \approx0.65 α ∗ ( n r , n b ) ≈ 0.65 it rises before it decreases. By the recursion of Theorem A.11 , it rises by 25% at α = 0.9 \alpha= 0.9 α = 0.9 and by 62% at α = 0.99 \alpha= 0.99 α = 0.99 , and by 74% at α = 1 \alpha= 1 α = 1 , where it never decreases.
Figure 1. Convergence under constant and time-varying synthetic fractions. Top: constant α \alpha α , from 0 (dark) to 1 (light, see color bar): (a) E [ ∥ μ ( t ) − μ ∥ 2 2 ] \mathbb{E}[\lVert\boldsymbol{\mu}^{(t)}-\boldsymbol{\mu}\rVert_{2}^{2}] E [∥ μ ( t ) − μ ∥ 2 2 ] , (b) E [ ∥ Σ ( t ) − Σ ∥ F ] \mathbb{E}[\lVert\boldsymbol{\Sigma}^{(t)}-\boldsymbol{\Sigma}\rVert_{F}] E [∥ Σ ( t ) − Σ ∥ F ] , and (c) E [ W 2 2 ( D t , D r e a l ) ] \mathbb{E}[W_{2}^{2}(\mathcal{D}_{t},\mathcal{D}_{\mathrm{real}})] E [ W 2 2 ( D t , D real )] . Bottom: (d)–(f) the same quantities for α t = 1 − 1 t \alpha_t=1-\frac{1}{t} α t = 1 − t 1 , 1 − 1 t 1-\frac{1}{\sqrt{t}} 1 − t 1 , 1 − 1 ln ( t + 2 ) 1-\frac{1}{\ln(t+2)} 1 − l n ( t + 2 ) 1 and 1 − 1 ln ln ( t + 16 ) 1-\frac{1}{\ln\ln(t+16)} 1 − l n l n ( t + 16 ) 1 , with α ≡ 1 \alpha\equiv1 α ≡ 1 (dashed) for reference.
The sharpness of the criterion. Panels (d)–(f) in Figure 1 show the four sequences mentioned above. All four have sup t ≥ 1 α t = 1 \sup_{t \ge1} \alpha_t = 1 sup t ≥ 1 α t = 1 and a vanishing fraction of real data, yet they split as predicted by (12) : for α t = 1 − 1 / t \alpha_t = 1 - 1/t α t = 1 − 1/ t and 1 − 1 / t 1 - 1/\sqrt{t} 1 − 1/ t , whose series converges, the errors approach strictly positive limits, as at α ≡ 1 \alpha\equiv1 α ≡ 1 , while for 1 − 1 / ln ( t + 2 ) 1 - 1/\ln(t + 2) 1 − 1/ ln ( t + 2 ) and 1 − 1 / ln ln ( t + 16 ) 1 - 1/\ln\ln(t + 16) 1 − 1/ ln ln ( t + 16 ) , whose series diverges, they keep decreasing. Thus, what matters is not whether the real-data fraction vanishes but how fast.
Rate of convergence. Figure 2 (a) divides E [ ∥ μ ( t ) − μ ∥ 2 2 ] \mathbb{E}[\lVert\mu^{(t)}-\mu\rVert_{2}^{2}] E [∥ μ ( t ) − μ ∥ 2 2 ] by the rate r α ( N t ) r_{\alpha}(N_t) r α ( N t ) stated in (17) . Here, the error is computed exactly by Tr ( C t ) \operatorname{Tr}(C_t) Tr ( C t ) (Equation (11) ) using the recursion of Theorem A.11 . Appendix § 9 (Figure 3 ) confirms that these exact values agree with the simulation. Each curve in Figure 2 (a) is normalized by its own endpoint, so that flattening at 1 confirms the predicted rate. All five values α ∈ { 0 , 0.3 , 0.5 , 0.7 , 0.9 } \alpha\in\{0, 0.3, 0.5, 0.7, 0.9\} α ∈ { 0 , 0.3 , 0.5 , 0.7 , 0.9 } level off, on both sides of the transition at α = 1 2 \alpha= \frac{1}{2} α = 2 1 . At α = 0 \alpha= 0 α = 0 the curve is flat for every N t N_t N t , since then C t = Σ / N t C_t = \Sigma/N_t C t = Σ/ N t exactly. Figure 2 (b) repeats this for the covariance, dividing E [ ∥ Σ ( t ) − Σ ∥ F ] \mathbb{E}[\lVert\Sigma^{(t)}-\Sigma\rVert_F] E [∥ Σ ( t ) − Σ ∥ F ] , estimated from the M = 20,000 M = 20{,}000 M = 20 , 000 Monte-Carlo runs, by r α ( N t ) \sqrt{r_{\alpha}(N_t)} r α ( N t ) , the square root because the Frobenius error is a distance while the mean-square error in (a) is a squared distance. The same five curves level off, so empirically the covariance converges at the rate of the mean, including the transition at α = 1 2 \alpha= \frac{1}{2} α = 2 1 , although Theorem 3.6 covers only the mean.
Figure 2. Rates of convergence and the effect of synthetic-sample detection. (a) E [ ∥ μ ( t ) − μ ∥ 2 2 ] / r α ( N t ) \mathbb{E}[\lVert\boldsymbol{\mu}^{(t)}-\boldsymbol{\mu}\rVert_{2}^{2}]/r_{\alpha}(N_t) E [∥ μ ( t ) − μ ∥ 2 2 ] / r α ( N t ) and (b) E [ ∥ Σ ( t ) − Σ ∥ F ] / r α ( N t ) \mathbb{E}[\lVert\boldsymbol{\Sigma}^{(t)}-\boldsymbol{\Sigma}\rVert_{F}]/r_{\alpha}(N_t) E [∥ Σ ( t ) − Σ ∥ F ] / r α ( N t ) , where r α r_{\alpha} r α is the rate of Theorem 3.6, for α ∈ { 0 , 0.3 , 0.5 , 0.7 , 0.9 } \alpha\in\{0,0.3,0.5,0.7,0.9\} α ∈ { 0 , 0.3 , 0.5 , 0.7 , 0.9 } . Each curve is scaled to equal 1 at the largest N t N_t N t . (c) The gain from perfect detection of synthetic samples, N o r a c l e / N e f f N_{\mathrm{oracle}}/N_{\mathrm{eff}} N oracle / N eff , for constant α ∈ [ 0 , 1 ) \alpha\in[0,1) α ∈ [ 0 , 1 ) (see color bar).
Figure 3. Exact E [ ∥ μ ( t ) − μ ∥ 2 2 ] \mathbb{E}[\lVert\mu^{(t)}-\mu\rVert_2^2] E [∥ μ ( t ) − μ ∥ 2 2 ] against simulation. (a) Constant α \alpha α (color bar) and (b) the four time-varying sequences of Figure 1 , with α ≡ 1 \alpha\equiv1 α ≡ 1 (dashed). Lines are the recursion of Theorem A.11 , and circles are Monte-Carlo averages.
The value of identifying real samples. Figure 2 (c) shows the gain G α ( N t ) = N o r a c l e ( N t ) / N e f f ( N t ) G_{\alpha}(N_t) = N_{\mathrm{oracle}}(N_t)/N_{\mathrm{eff}}(N_t) G α ( N t ) = N oracle ( N t ) / N eff ( N t ) from perfect detection of synthetic samples. Here, N e f f = Tr ( Σ ) / E [ ∥ μ ( t ) − μ ∥ 2 2 ] N_{\mathrm{eff}} = \operatorname{Tr}(\Sigma)/\mathbb{E}[\lVert\mu^{(t)}-\mu\rVert_{2}^{2}] N eff = Tr ( Σ ) / E [∥ μ ( t ) − μ ∥ 2 2 ] is the number of real i.i.d. samples that would give the same mean-square error, and N o r a c l e = ( 1 − α ) N t + α n r N_{\mathrm{oracle}} = (1-\alpha)N_t+\alpha n_r N oracle = ( 1 − α ) N t + α n r is the expected number of real samples in the pool (discussed more in Appendix § 8 ). N e f f N_{\mathrm{eff}} N eff is computed exactly using the recursion of Theorem A.11 . The gain saturates for α < 1 2 \alpha< \frac{1}{2} α < 2 1 and grows like N t 2 α − 1 N_t^{2\alpha-1} N t 2 α − 1 for α > 1 2 \alpha> \frac{1}{2} α > 2 1 , reaching about 10 3 10^3 1 0 3 at N t = 10 6 N_t = 10^6 N t = 1 0 6 for α ∈ [ 0.9 , 0.95 ] \alpha\in[0.9,0.95] α ∈ [ 0.9 , 0.95 ] . At α = 0.99 \alpha= 0.99 α = 0.99 it is smaller, since the oracle also runs out of real data as α → 1 \alpha\to1 α → 1 .
Experiments with Diffusion Models
Our paper focuses on theoretical contributions. Yet, to demonstrate that insights generalize beyond the theoretical setup, in Appendix § 10 we present results of iterative training of a DDPM-based diffusion model (Ho et al. [12 ] ), on the CIFAR-10 dataset (Krizhevsky et al. [15 ] ). We examine the effect of different fixed values of α \alpha α , with no distinction between real and generated data, on the Fréchet Inception Distance (FID) quality metric. Aligning with our theory: 1) for every α < 1 \alpha< 1 α < 1 , the FID improves upon its initial value at every iteration, although real and generated images are pooled indistinguishably; 2) for α = 1 \alpha= 1 α = 1 , the FID never improves but remains bounded; 3) the FID is ordered by α \alpha α at every iteration, and its improvement slows as α \alpha α grows; 4) for α = 0.25 < 1 2 \alpha= 0.25 < \frac{1}{2} α = 0.25 < 2 1 , the curve follows that of α = 0 \alpha= 0 α = 0 up to a roughly constant offset, i.e., synthetic data inflates the constant but not the rate; and 5) discarding the generated images would improve the final FID far more for α = 0.75 > 1 2 \alpha= 0.75 > \frac{1}{2} α = 0.75 > 2 1 than for α = 0.25 < 1 2 \alpha= 0.25 < \frac{1}{2} α = 0.25 < 2 1 . An apparent gap between the theoretical setup and the diffusion experiments is that, for a large α < 1 \alpha< 1 α < 1 , the FID stops improving at some t t t . This may result from the simple training strategy used (which could be improved) and the relatively simple diffusion model (which could be scaled up).
Conclusion
Large generative models are commonly trained on data scraped from the web, an increasing share of which is generated by earlier models. Prior work shows that iterative training degrades performance: 1) without bound when earlier data is discarded, and 2) in a bounded manner when data accumulates. The continued improvement of generative models has been explained by the availability of fresh real data, but only under the unrealistic assumption that it can be perfectly separated from generated data. In this paper, we studied the more realistic setting in which fresh real and generated samples are pooled indistinguishably. We showed, for a tractable Gaussian data model, that the estimated distribution converges to the true one if and only if the share of fresh real information (quantified exactly) is not summable. Our results reveal that consistency holds for any fixed positive fraction of real data, and even for a fraction that vanishes slowly enough. Yet, the convergence rate of the mean estimator deteriorates as generated samples become more prevalent: it matches the rate of purely real data as long as generated samples are a minority of the newly collected data, and becomes arbitrarily slow once they are a majority. This motivates ongoing research on detecting and curating generated samples, and indicates when such efforts are worthwhile: as more data is collected, the gain from perfect detection is bounded when generated samples are a minority of the newly collected data, but grows without bound with the amount of collected data otherwise. Finally, while our work focuses on iterative training of generative models from a distribution estimation perspective, a compelling direction for future work is to explore whether even biased models can still generate augmented data that is useful for downstream tasks, similar to how low-level pre-processing can benefit downstream classification despite idealized information-theoretic principles (Turgeman & Tirer, 2026).
Acknowledgments
The work is supported by the Israel Science Foundation grant No. 1940/23 and the MOST grant No. 0007091.
Proofs
Section A.1 defines three scalar sequences, which are used widely in the proofs.
Scalar sequences
For t ≥ 1 t \ge1 t ≥ 1 , we define three scalar sequences:
λ t : = N t + n b ( t ) α t N t + 1 , β t : = n b ( t ) N t + 1 2 , γ t : = λ t 2 − α t 2 β t . (18) \lambda_t := \frac{N_t+n_b(t)\alpha_t}{N_{t+1}}, \qquad\beta_t := \frac{n_b(t)}{N_{t+1}^2}, \qquad\gamma_t := \lambda_t^2-\alpha_t^2\beta_t. \tag*{(18)} λ t := N t + 1 N t + n b ( t ) α t , β t := N t + 1 2 n b ( t ) , γ t := λ t 2 − α t 2 β t . (18)
Additionally, let us define
Γ s , t : = { ∏ k = s t γ k s ≤ t 1 s > t . (19) \Gamma_{s,t} := \begin{cases} \prod_{k=s}^{t}\gamma_k & s \le t \\ 1 & s > t \end{cases}. \tag*{(19)} Γ s , t := { ∏ k = s t γ k 1 s ≤ t s > t . (19)
Section A.2 collects the technical lemmas used throughout.
Technical Lemmas
We first formalize the conditioning used throughout the proofs.
Definition 1 (Natural filtration). For every t ≥ 1 t \ge1 t ≥ 1 , let
F t : = σ ( { x j ( 0 ) } j = 1 n b , { x i ( s ) } 1 ≤ i ≤ n b ( s ) , 1 ≤ s ≤ t − 1 ) (20) \mathcal{F}_t := \sigma\left(\left\{x_j^{(0)}\right\}_{j=1}^{n_b},\left\{x_i^{(s)}\right\}_{1\le i\le n_b(s),\,1\le s\le t-1}\right) \tag*{(20)} F t := σ ( { x j ( 0 ) } j = 1 n b , { x i ( s ) } 1 ≤ i ≤ n b ( s ) , 1 ≤ s ≤ t − 1 ) (20) denote the σ \sigma σ -algebra generated by all samples drawn before the batch { x i ( t ) } i = 1 n b ( t ) \{x_i^{(t)}\}_{i=1}^{n_b(t)} { x i ( t ) } i = 1 n b ( t ) . For t = 1 t=1 t = 1 , the second family is empty, so F 1 \mathcal{F}_1 F 1 is generated by the original real batch alone.
Since each iteration only adds samples, F 1 ⊆ F 2 ⊆ ⋯ \mathcal{F}_1 \subseteq\mathcal{F}_2 \subseteq\cdots F 1 ⊆ F 2 ⊆ ⋯ is a filtration, and ( μ ( t ) , Σ ( t ) ) (\mu^{(t)},\Sigma^{(t)}) ( μ ( t ) , Σ ( t ) ) is F t \mathcal{F}_t F t -measurable by (4) . Hence, Assumption 1 is equivalent to the following: for every t ≥ 1 t \ge1 t ≥ 1 , conditionally on F t \mathcal{F}_t F t , the samples { x i ( t ) } i = 1 n b ( t ) \{x_i^{(t)}\}_{i=1}^{n_b(t)} { x i ( t ) } i = 1 n b ( t ) are i.i.d. from the mixture (2) with parameters ( μ ( t ) , Σ ( t ) ) (\mu^{(t)},\Sigma^{(t)}) ( μ ( t ) , Σ ( t ) ) . This is the form used throughout the proofs.
Next, we record the conditional first and second moments of a newly drawn sample. These are used in the proof of Theorems 3.1 and A.11 .
Lemma A.1 (Conditional moments of the mixture samples). Under Assumption 1, conditional on F t \mathcal{F}_t F t , each newly drawn sample x i ( t ) x_i^{(t)} x i ( t ) (t ≥ 1 t \ge1 t ≥ 1 ) satisfies, using (7) (δ ( t ) = μ ( t ) − μ \delta^{(t)}=\mu^{(t)}-\mu δ ( t ) = μ ( t ) − μ ):
E [ x i ( t ) ∣ F t ] = α t μ ( t ) + ( 1 − α t ) μ = μ + α t δ ( t ) , (21) \mathbb{E}\left[x_i^{(t)} \mid\mathcal{F}_t\right] = \alpha_t\mu^{(t)}+(1-\alpha_t)\mu=\mu+\alpha_t\delta^{(t)}, \tag*{(21)} E [ x i ( t ) ∣ F t ] = α t μ ( t ) + ( 1 − α t ) μ = μ + α t δ ( t ) , (21) Cov ( x i ( t ) ∣ F t ) = α t Σ ( t ) + ( 1 − α t ) Σ + α t ( 1 − α t ) δ ( t ) ( δ ( t ) ) T , (22) \operatorname{Cov}\left(x_i^{(t)} \mid\mathcal{F}_t\right) = \alpha_t\Sigma^{(t)}+(1-\alpha_t)\Sigma+\alpha_t(1-\alpha_t)\delta^{(t)}\left(\delta^{(t)}\right)^{\mathsf{T}}, \tag*{(22)} Cov ( x i ( t ) ∣ F t ) = α t Σ ( t ) + ( 1 − α t ) Σ + α t ( 1 − α t ) δ ( t ) ( δ ( t ) ) T , (22) E [ ( x i ( t ) − μ ) ( x i ( t ) − μ ) T ∣ F t ] = α t Σ ( t ) + ( 1 − α t ) Σ + α t δ ( t ) ( δ ( t ) ) T . (23) \mathbb{E}\left[\left(x_i^{(t)}-\mu\right)\left(x_i^{(t)}-\mu\right)^{\mathsf{T}}\mid\mathcal{F}_t\right] = \alpha_t\Sigma^{(t)}+(1-\alpha_t)\Sigma+\alpha_t\delta^{(t)}\left(\delta^{(t)}\right)^{\mathsf{T}}. \tag*{(23)} E [ ( x i ( t ) − μ ) ( x i ( t ) − μ ) T ∣ F t ] = α t Σ ( t ) + ( 1 − α t ) Σ + α t δ ( t ) ( δ ( t ) ) T . (23)
Proof. Given F t \mathcal{F}_t F t , the sample x i ( t ) x_i^{(t)} x i ( t ) is drawn from the two-component mixture (2) with weights α t \alpha_t α t on N ( μ ( t ) , Σ ( t ) ) \mathcal{N}(\mu^{(t)},\Sigma^{(t)}) N ( μ ( t ) , Σ ( t ) ) and 1 − α t 1-\alpha_t 1 − α t on N ( μ , Σ ) \mathcal{N}(\mu,\Sigma) N ( μ , Σ ) .
Equation (21). By linearity, the mixture mean is α t μ ( t ) + ( 1 − α t ) μ = μ + α t δ ( t ) \alpha_t\mu^{(t)}+(1-\alpha_t)\mu=\mu+\alpha_t\delta^{(t)} α t μ ( t ) + ( 1 − α t ) μ = μ + α t δ ( t ) .
Equation (22). Let Z ∈ { 1 , 2 } Z \in\{1,2\} Z ∈ { 1 , 2 } be the component indicator with P ( Z = 1 ∣ F t ) = α t \mathbb{P}(Z=1\mid\mathcal{F}_t)=\alpha_t P ( Z = 1 ∣ F t ) = α t , so that x i ( t ) ∣ { Z = 1 } ∼ N ( μ ( t ) , Σ ( t ) ) x_i^{(t)}\mid\{Z=1\}\sim\mathcal{N}(\mu^{(t)},\Sigma^{(t)}) x i ( t ) ∣ { Z = 1 } ∼ N ( μ ( t ) , Σ ( t ) ) and x i ( t ) ∣ { Z = 2 } ∼ N ( μ , Σ ) x_i^{(t)}\mid\{Z=2\}\sim\mathcal{N}(\mu,\Sigma) x i ( t ) ∣ { Z = 2 } ∼ N ( μ , Σ ) . The law of total covariance gives
Cov ( x i ( t ) ∣ F t ) = E [ Cov ( x i ( t ) ∣ Z , F t ) ∣ F t ] + Cov ( E [ x i ( t ) ∣ Z , F t ] ∣ F t ) . \operatorname{Cov}\left(x_i^{(t)}\mid\mathcal{F}_t\right)=\mathbb{E}\left[\operatorname{Cov}\left(x_i^{(t)}\mid Z,\mathcal{F}_t\right)\mid\mathcal{F}_t\right]+\operatorname{Cov}\left(\mathbb{E}\left[x_i^{(t)}\mid Z,\mathcal{F}_t\right]\mid\mathcal{F}_t\right). Cov ( x i ( t ) ∣ F t ) = E [ Cov ( x i ( t ) ∣ Z , F t ) ∣ F t ] + Cov ( E [ x i ( t ) ∣ Z , F t ] ∣ F t ) .
The first summand A A A reads:
Cov ( x i ( t ) ∣ Z , F t ) = { Σ ( t ) Z = 1 Σ Z = 2 ⇒ A = α t Σ ( t ) + ( 1 − α t ) Σ . (24) \operatorname{Cov}\left(x_i^{(t)}\mid Z,\mathcal{F}_t\right)=\begin{cases}\Sigma^{(t)} & Z=1 \\ \Sigma& Z=2\end{cases}\Rightarrow A=\alpha_t\Sigma^{(t)}+(1-\alpha_t)\Sigma. \tag*{(24)} Cov ( x i ( t ) ∣ Z , F t ) = { Σ ( t ) Σ Z = 1 Z = 2 ⇒ A = α t Σ ( t ) + ( 1 − α t ) Σ. (24)
We now compute the second summand B B B . We have
P ( Z ) : = E [ x i ( t ) ∣ Z , F t ] = { μ ( t ) Z = 1 μ Z = 2 ⇒ B = E [ ( P ( Z ) − μ P ) ( P ( Z ) − μ P ) ⊤ ∣ F t ] \mathcal{P}(Z) := \mathbb{E}[x_i^{(t)} \mid Z,\mathcal{F}_t] =
\begin{cases}
\mu^{(t)} & Z=1\\
\mu& Z=2
\end{cases}
\Rightarrow B = \mathbb{E}[(\mathcal{P}(Z)-\mu_{\mathcal{P}})(\mathcal{P}(Z)-\mu_{\mathcal{P}})^\top\mid\mathcal{F}_t] P ( Z ) := E [ x i ( t ) ∣ Z , F t ] = { μ ( t ) μ Z = 1 Z = 2 ⇒ B = E [( P ( Z ) − μ P ) ( P ( Z ) − μ P ) ⊤ ∣ F t ]
where
μ P = E [ P ( Z ) ∣ F t ] = α t μ ( t ) + ( 1 − α t ) μ . \mu_{\mathcal{P}} = \mathbb{E}[\mathcal{P}(Z)\mid\mathcal{F}_t] = \alpha_t\mu^{(t)}+(1-\alpha_t)\mu. μ P = E [ P ( Z ) ∣ F t ] = α t μ ( t ) + ( 1 − α t ) μ .
That is, B = E [ P c ( Z ) ∣ F t ] B=\mathbb{E}[\mathcal{P}_c(Z)\mid\mathcal{F}_t] B = E [ P c ( Z ) ∣ F t ] , where
( P ( Z ) − μ P ) = { ( 1 − α t ) ( μ ( t ) − μ ) Z = 1 − α t ( μ ( t ) − μ ) Z = 2 = { ( 1 − α t ) δ ( t ) Z = 1 − α t δ ( t ) Z = 2 (\mathcal{P}(Z)-\mu_{\mathcal{P}})=
\begin{cases}
(1-\alpha_t)(\mu^{(t)}-\mu) & Z=1\\
-\alpha_t(\mu^{(t)}-\mu) & Z=2
\end{cases}
=
\begin{cases}
(1-\alpha_t)\delta^{(t)} & Z=1\\
-\alpha_t\delta^{(t)} & Z=2
\end{cases} ( P ( Z ) − μ P ) = { ( 1 − α t ) ( μ ( t ) − μ ) − α t ( μ ( t ) − μ ) Z = 1 Z = 2 = { ( 1 − α t ) δ ( t ) − α t δ ( t ) Z = 1 Z = 2
P c ( Z ) = ( P ( Z ) − μ P ) ( P ( Z ) − μ P ) ⊤ = { ( 1 − α t ) 2 δ ( t ) ( δ ( t ) ) ⊤ Z = 1 α t 2 δ ( t ) ( δ ( t ) ) ⊤ Z = 2 \mathcal{P}_c(Z)=(\mathcal{P}(Z)-\mu_{\mathcal{P}})(\mathcal{P}(Z)-\mu_{\mathcal{P}})^\top
=
\begin{cases}
(1-\alpha_t)^2\delta^{(t)}(\delta^{(t)})^\top& Z=1\\
\alpha_t^2\delta^{(t)}(\delta^{(t)})^\top& Z=2
\end{cases} P c ( Z ) = ( P ( Z ) − μ P ) ( P ( Z ) − μ P ) ⊤ = { ( 1 − α t ) 2 δ ( t ) ( δ ( t ) ) ⊤ α t 2 δ ( t ) ( δ ( t ) ) ⊤ Z = 1 Z = 2
Finally, the second part reads
B = α t ( ( 1 − α t ) δ ( t ) ) ( ( 1 − α t ) δ ( t ) ) ⊤ + ( 1 − α t ) ( α t δ ( t ) ) ( α t δ ( t ) ) ⊤ = α t ( 1 − α t ) δ ( t ) ( δ ( t ) ) ⊤ , B=\alpha_t\bigl((1-\alpha_t)\delta^{(t)}\bigr)\bigl((1-\alpha_t)\delta^{(t)}\bigr)^\top+(1-\alpha_t)\bigl(\alpha_t\delta^{(t)}\bigr)\bigl(\alpha_t\delta^{(t)}\bigr)^\top=\alpha_t(1-\alpha_t)\delta^{(t)}(\delta^{(t)})^\top, B = α t ( ( 1 − α t ) δ ( t ) ) ( ( 1 − α t ) δ ( t ) ) ⊤ + ( 1 − α t ) ( α t δ ( t ) ) ( α t δ ( t ) ) ⊤ = α t ( 1 − α t ) δ ( t ) ( δ ( t ) ) ⊤ ,
using α t ( 1 − α t ) 2 + ( 1 − α t ) α t 2 = α t ( 1 − α t ) \alpha_t(1-\alpha_t)^2+(1-\alpha_t)\alpha_t^2=\alpha_t(1-\alpha_t) α t ( 1 − α t ) 2 + ( 1 − α t ) α t 2 = α t ( 1 − α t ) . Adding the two terms yields (22).
Equation (23). For any random vector x x x and fixed μ \mu μ , E [ ( x − μ ) ( x − μ ) ⊤ ] = Cov ( x ) + ( E [ x ] − μ ) ( E [ x ] − μ ) ⊤ \mathbb{E}[(x-\mu)(x-\mu)^\top]=\operatorname{Cov}(x)+(\mathbb{E}[x]-\mu)(\mathbb{E}[x]-\mu)^\top E [( x − μ ) ( x − μ ) ⊤ ] = Cov ( x ) + ( E [ x ] − μ ) ( E [ x ] − μ ) ⊤ . Applying this conditionally on F t \mathcal{F}_t F t with E [ x i ( t ) ∣ F t ] − μ = α t δ ( t ) \mathbb{E}[x_i^{(t)}\mid\mathcal{F}_t]-\mu=\alpha_t\delta^{(t)} E [ x i ( t ) ∣ F t ] − μ = α t δ ( t ) and substituting (22),
α t Σ ( t ) + ( 1 − α t ) Σ + α t ( 1 − α t ) δ ( t ) ( δ ( t ) ) ⊤ + α t 2 δ ( t ) ( δ ( t ) ) ⊤ = α t Σ ( t ) + ( 1 − α t ) Σ + α t δ ( t ) ( δ ( t ) ) ⊤ , \alpha_t\Sigma^{(t)}+(1-\alpha_t)\Sigma+\alpha_t(1-\alpha_t)\delta^{(t)}(\delta^{(t)})^\top+\alpha_t^2\delta^{(t)}(\delta^{(t)})^\top=\alpha_t\Sigma^{(t)}+(1-\alpha_t)\Sigma+\alpha_t\delta^{(t)}(\delta^{(t)})^\top, α t Σ ( t ) + ( 1 − α t ) Σ + α t ( 1 − α t ) δ ( t ) ( δ ( t ) ) ⊤ + α t 2 δ ( t ) ( δ ( t ) ) ⊤ = α t Σ ( t ) + ( 1 − α t ) Σ + α t δ ( t ) ( δ ( t ) ) ⊤ ,
since α t ( 1 − α t ) + α t 2 = α t \alpha_t(1-\alpha_t)+\alpha_t^2=\alpha_t α t ( 1 − α t ) + α t 2 = α t .
Next, we derive a more useful representation of Σ ( t ) \Sigma^{(t)} Σ ( t ) that is centered at μ \mu μ . It is used in the proof of Theorems 3.1 and A.22 .
Lemma A.2 (Pooled scatter identity). Fix t ≥ 1 t\geq1 t ≥ 1 . The estimators μ ( t ) , Σ ( t ) \mu^{(t)},\Sigma^{(t)} μ ( t ) , Σ ( t ) defined in (4) satisfy, using (7) (δ ( t ) = μ ( t ) − μ \delta^{(t)}=\mu^{(t)}-\mu δ ( t ) = μ ( t ) − μ ):
Σ ( t ) = 1 N t ( ∑ j = 1 n r ( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) ⊤ + ∑ s = 1 t − 1 ∑ i = 1 n b ( s ) ( x i ( s ) − μ ) ( x i ( s ) − μ ) ⊤ ) − δ ( t ) ( δ ( t ) ) ⊤ (25) \Sigma^{(t)}=\frac{1}{N_t}\left(\sum_{j=1}^{n_r}(x_j^{(0)}-\mu)(x_j^{(0)}-\mu)^\top+\sum_{s=1}^{t-1}\sum_{i=1}^{n_b^{(s)}}(x_i^{(s)}-\mu)(x_i^{(s)}-\mu)^\top\right)-\delta^{(t)}(\delta^{(t)})^\top \tag*{(25)} Σ ( t ) = N t 1 j = 1 ∑ n r ( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) ⊤ + s = 1 ∑ t − 1 i = 1 ∑ n b ( s ) ( x i ( s ) − μ ) ( x i ( s ) − μ ) ⊤ − δ ( t ) ( δ ( t ) ) ⊤ (25)
Proof. By (4), we have
∑ k = 1 N t y k = N t μ ( t ) . \sum_{k=1}^{N_t}y_k=N_t\mu^{(t)}. k = 1 ∑ N t y k = N t μ ( t ) .
Thus,
∑ k = 1 N t ( y k − μ ) = N t μ ( t ) − N t μ = N t δ ( t ) . \sum_{k=1}^{N_t}(y_k-\mu)=N_t\mu^{(t)}-N_t\mu=N_t\delta^{(t)}. k = 1 ∑ N t ( y k − μ ) = N t μ ( t ) − N t μ = N t δ ( t ) .
Expanding the defining sum in (4), we get
N t Σ ( t ) = ∑ k = 1 N t ( y k − μ ( t ) ) ( y k − μ ( t ) ) T = ∑ k = 1 N t ( ( y k − μ ) − δ ( t ) ) ( ( y k − μ ) − δ ( t ) ) T = ∑ k = 1 N t ( y k − μ ) ( y k − μ ) T − ∑ k = 1 N t ( y k − μ ) ( δ ( t ) ) T − δ ( t ) ∑ k = 1 N t ( y k − μ ) T + N t δ ( t ) ( δ ( t ) ) T = ∑ k = 1 N t ( y k − μ ) ( y k − μ ) T − N t ⋅ δ ( t ) ( δ ( t ) ) T − N t ⋅ δ ( t ) ( δ ( t ) ) T + N t ⋅ δ ( t ) ( δ ( t ) ) T = ∑ k = 1 N t ( y k − μ ) ( y k − μ ) T − N t ⋅ δ ( t ) ( δ ( t ) ) T . \begin{aligned}
N_t\Sigma^{(t)} = \sum_{k=1}^{N_t}\left(\boldsymbol{y}_k-\mu^{(t)}\right)\left(\boldsymbol{y}_k-\mu^{(t)}\right)^{\mathsf{T}} \\
&= \sum_{k=1}^{N_t}\left(\left(\boldsymbol{y}_k-\mu\right)-\delta^{(t)}\right)\left(\left(\boldsymbol{y}_k-\mu\right)-\delta^{(t)}\right)^{\mathsf{T}} \\
&= \sum_{k=1}^{N_t}\left(\boldsymbol{y}_k-\mu\right)\left(\boldsymbol{y}_k-\mu\right)^{\mathsf{T}}-\sum_{k=1}^{N_t}\left(\boldsymbol{y}_k-\mu\right)\left(\delta^{(t)}\right)^{\mathsf{T}}-\delta^{(t)}\sum_{k=1}^{N_t}\left(\boldsymbol{y}_k-\mu\right)^{\mathsf{T}}+N_t\delta^{(t)}\left(\delta^{(t)}\right)^{\mathsf{T}} \\
&= \sum_{k=1}^{N_t}\left(\boldsymbol{y}_k-\mu\right)\left(\boldsymbol{y}_k-\mu\right)^{\mathsf{T}}-N_t\cdot\delta^{(t)}\left(\delta^{(t)}\right)^{\mathsf{T}}-N_t\cdot\delta^{(t)}\left(\delta^{(t)}\right)^{\mathsf{T}}+N_t\cdot\delta^{(t)}\left(\delta^{(t)}\right)^{\mathsf{T}} \\
&= \sum_{k=1}^{N_t}\left(\boldsymbol{y}_k-\mu\right)\left(\boldsymbol{y}_k-\mu\right)^{\mathsf{T}}-N_t\cdot\delta^{(t)}\left(\delta^{(t)}\right)^{\mathsf{T}}.
\end{aligned} N t Σ ( t ) = k = 1 ∑ N t ( y k − μ ( t ) ) ( y k − μ ( t ) ) T = k = 1 ∑ N t ( ( y k − μ ) − δ ( t ) ) ( ( y k − μ ) − δ ( t ) ) T = k = 1 ∑ N t ( y k − μ ) ( y k − μ ) T − k = 1 ∑ N t ( y k − μ ) ( δ ( t ) ) T − δ ( t ) k = 1 ∑ N t ( y k − μ ) T + N t δ ( t ) ( δ ( t ) ) T = k = 1 ∑ N t ( y k − μ ) ( y k − μ ) T − N t ⋅ δ ( t ) ( δ ( t ) ) T − N t ⋅ δ ( t ) ( δ ( t ) ) T + N t ⋅ δ ( t ) ( δ ( t ) ) T = k = 1 ∑ N t ( y k − μ ) ( y k − μ ) T − N t ⋅ δ ( t ) ( δ ( t ) ) T .
Dividing both sides by N t N_t N t , we get the required identity:
Σ ( t ) = 1 N t ( ∑ j = 1 n r ( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) T + ∑ s = 1 t − 1 ∑ i = 1 n b ( s ) ( x i ( s ) − μ ) ( x i ( s ) − μ ) T ) − δ ( t ) ( δ ( t ) ) T . \Sigma^{(t)}=\frac{1}{N_t}\left(\sum_{j=1}^{n_r}\left(\boldsymbol{x}_j^{(0)}-\mu\right)\left(\boldsymbol{x}_j^{(0)}-\mu\right)^{\mathsf{T}}+\sum_{s=1}^{t-1}\sum_{i=1}^{n_b^{(s)}}\left(\boldsymbol{x}_i^{(s)}-\mu\right)\left(\boldsymbol{x}_i^{(s)}-\mu\right)^{\mathsf{T}}\right)-\delta^{(t)}\left(\delta^{(t)}\right)^{\mathsf{T}}. Σ ( t ) = N t 1 j = 1 ∑ n r ( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) T + s = 1 ∑ t − 1 i = 1 ∑ n b ( s ) ( x i ( s ) − μ ) ( x i ( s ) − μ ) T − δ ( t ) ( δ ( t ) ) T .
Next, we record an elementary consequence of the Cauchy–Schwarz inequality, used in the proofs of Lemmas A.4, A.23, and A.19.
Lemma A.3 (ℓ 1 \ell_1 ℓ 1 –ℓ 2 \ell_2 ℓ 2 comparison). Let n ≥ 1 n \ge1 n ≥ 1 and a 1 , … , a n ∈ R a_1,\ldots,a_n \in\mathbb{R} a 1 , … , a n ∈ R . Then
∑ i = 1 n ∣ a i ∣ ≤ n ( ∑ i = 1 n a i 2 ) 1 / 2 , and consequently ( ∑ i = 1 n a i ) 2 ≤ n ∑ i = 1 n a i 2 . (26) \sum_{i=1}^{n}|a_i|\le\sqrt{n}\left(\sum_{i=1}^{n}a_i^2\right)^{1/2},
\qquad\text{and consequently}\qquad
\left(\sum_{i=1}^{n}a_i\right)^2\le n\sum_{i=1}^{n}a_i^2. \tag*{(26)} i = 1 ∑ n ∣ a i ∣ ≤ n ( i = 1 ∑ n a i 2 ) 1/2 , and consequently ( i = 1 ∑ n a i ) 2 ≤ n i = 1 ∑ n a i 2 . (26)
Proof. Write a : = ( ∣ a 1 ∣ , … , ∣ a n ∣ ) T \boldsymbol{a}:=(|a_1|,\ldots,|a_n|)^{\mathsf{T}} a := ( ∣ a 1 ∣ , … , ∣ a n ∣ ) T and 1 n : = ( 1 , … , 1 ) T \boldsymbol{1}_n:=(1,\ldots,1)^{\mathsf{T}} 1 n := ( 1 , … , 1 ) T . The Cauchy–Schwarz inequality gives
∑ i = 1 n ∣ a i ∣ = ⟨ a , 1 n ⟩ ≤ ∥ a ∥ 2 ∥ 1 n ∥ 2 = n ( ∑ i = 1 n a i 2 ) 1 / 2 , \sum_{i=1}^{n}|a_i|=\langle\boldsymbol{a},\boldsymbol{1}_n\rangle\le\|\boldsymbol{a}\|_2\|\boldsymbol{1}_n\|_2=\sqrt{n}\left(\sum_{i=1}^{n}a_i^2\right)^{1/2}, i = 1 ∑ n ∣ a i ∣ = ⟨ a , 1 n ⟩ ≤ ∥ a ∥ 2 ∥ 1 n ∥ 2 = n ( i = 1 ∑ n a i 2 ) 1/2 ,
which is the first inequality. For the second, the triangle inequality gives ∣ ∑ i = 1 n a i ∣ ≤ ∑ i = 1 n ∣ a i ∣ \left|\sum_{i=1}^{n}a_i\right|\le\sum_{i=1}^{n}|a_i| ∣ ∑ i = 1 n a i ∣ ≤ ∑ i = 1 n ∣ a i ∣ . Both sides are nonnegative, so squaring and applying the first inequality yields
( ∑ i = 1 n a i ) 2 ≤ ( ∑ i = 1 n ∣ a i ∣ ) 2 ≤ n ∑ i = 1 n a i 2 . \left(\sum_{i=1}^{n}a_i\right)^2\le\left(\sum_{i=1}^{n}|a_i|\right)^2\le n\sum_{i=1}^{n}a_i^2. ( i = 1 ∑ n a i ) 2 ≤ ( i = 1 ∑ n ∣ a i ∣ ) 2 ≤ n i = 1 ∑ n a i 2 .
We now record an elementary fact about positive-semi-definite matrices. It is used in the proofs of Lemma A.10, Theorems A.12 and A.22, and Lemma A.16.
Lemma A.4 (Trace–Frobenius comparison). Let A ∈ R d × d A\in\mathbb{R}^{d\times d} A ∈ R d × d be symmetric with A ⪰ 0 A\succeq0 A ⪰ 0 . Then:
(a) ∥ A ∥ F ≤ Tr ( A ) ≤ d ∥ A ∥ F \|A\|_F\le\operatorname{Tr}(A)\le\sqrt{d}\,\|A\|_F ∥ A ∥ F ≤ Tr ( A ) ≤ d ∥ A ∥ F . In particular, ∥ A ∥ F 2 ≤ Tr ( A ) 2 \|A\|_F^2\le\operatorname{Tr}(A)^2 ∥ A ∥ F 2 ≤ Tr ( A ) 2 .
(b) If { A t } t ≥ 1 ⊂ R d × d \{A_t\}_{t\ge1}\subset\mathbb{R}^{d\times d} { A t } t ≥ 1 ⊂ R d × d are symmetric and positive-semi-definite, then
∥ A t ∥ F → t → ∞ 0 ⟺ Tr ( A t ) → t → ∞ 0. (27) \|A_t\|_F\xrightarrow[t\to\infty]{}0\quad\Longleftrightarrow\quad\operatorname{Tr}(A_t)\xrightarrow[t\to\infty]{}0. \tag*{(27)} ∥ A t ∥ F t → ∞ 0 ⟺ Tr ( A t ) t → ∞ 0. (27)
Proof. (a) By the spectral theorem, A = Q D Q T A = QDQ^{\mathsf{T}} A = Q D Q T with Q Q Q orthogonal and D = diag ( ν 1 , … , ν d ) D = \operatorname{diag}(\nu_1,\ldots,\nu_d) D = diag ( ν 1 , … , ν d ) , where ν 1 , … , ν d ≥ 0 \nu_1,\ldots,\nu_d \ge0 ν 1 , … , ν d ≥ 0 are the eigenvalues of A A A (real by symmetry, nonnegative since A ⪰ 0 A \succeq0 A ⪰ 0 ). As the trace and the Frobenius norm are invariant under orthogonal conjugation,
Tr ( A ) = ∑ i = 1 d ν i , ∥ A ∥ F 2 = ∑ i = 1 d ν i 2 . (28) \operatorname{Tr}(A) = \sum_{i=1}^{d}\nu_i,\qquad\lVert A\rVert_F^2 = \sum_{i=1}^{d}\nu_i^2. \tag*{(28)} Tr ( A ) = i = 1 ∑ d ν i , ∥ A ∥ F 2 = i = 1 ∑ d ν i 2 . (28)
Lower bound. Expanding the square and discarding the cross terms, which are nonnegative because every ν i ≥ 0 \nu_i \ge0 ν i ≥ 0 ,
Tr ( A ) 2 = ( ∑ i = 1 d ν i ) 2 = ∑ i = 1 d ν i 2 + ∑ i ≠ j ν i ν j ≥ ∑ i = 1 d ν i 2 = ∥ A ∥ F 2 . \operatorname{Tr}(A)^2 = \left(\sum_{i=1}^{d}\nu_i\right)^2 = \sum_{i=1}^{d}\nu_i^2 + \sum_{i\ne j}\nu_i\nu_j \ge\sum_{i=1}^{d}\nu_i^2 = \lVert A\rVert_F^2. Tr ( A ) 2 = ( i = 1 ∑ d ν i ) 2 = i = 1 ∑ d ν i 2 + i = j ∑ ν i ν j ≥ i = 1 ∑ d ν i 2 = ∥ A ∥ F 2 .
Both Tr ( A ) \operatorname{Tr}(A) Tr ( A ) and ∥ A ∥ F \lVert A\rVert_F ∥ A ∥ F are nonnegative, so taking square roots gives ∥ A ∥ F ≤ Tr ( A ) \lVert A\rVert_F \le\operatorname{Tr}(A) ∥ A ∥ F ≤ Tr ( A ) , together with the squared form ∥ A ∥ F 2 ≤ Tr ( A ) 2 \lVert A\rVert_F^2 \le\operatorname{Tr}(A)^2 ∥ A ∥ F 2 ≤ Tr ( A ) 2 .
Upper bound. Since every ν i ≥ 0 \nu_i \ge0 ν i ≥ 0 , Lemma A.3 gives
Tr ( A ) = ∑ i = 1 d ν i ≤ d ( ∑ i = 1 d ν i 2 ) 1 / 2 = d ∥ A ∥ F . \operatorname{Tr}(A) = \sum_{i=1}^{d}\nu_i \le\sqrt{d}\left(\sum_{i=1}^{d}\nu_i^2\right)^{1/2} = \sqrt{d}\,\lVert A\rVert_F. Tr ( A ) = i = 1 ∑ d ν i ≤ d ( i = 1 ∑ d ν i 2 ) 1/2 = d ∥ A ∥ F .
(b) Apply (a) to each A t A_t A t :
∥ A t ∥ F ≤ Tr ( A t ) ≤ d ∥ A t ∥ F , t ≥ 1. \lVert A_t\rVert_F \le\operatorname{Tr}(A_t) \le\sqrt{d}\,\lVert A_t\rVert_F,\qquad t\ge1. ∥ A t ∥ F ≤ Tr ( A t ) ≤ d ∥ A t ∥ F , t ≥ 1.
The constants 1 1 1 and d \sqrt{d} d do not depend on t t t , so the two sequences are squeezed to 0 0 0 together, which is (27) .
Next, we prove that the sequence β t \beta_t β t , defined in (18) , is summable. It is used in the proof of Theorems A.13 , 3.6, and A.22.
Lemma A.5 (Summability of β s \beta_s β s ). The weights β s \beta_s β s from (18) satisfy
∑ s = 1 ∞ β s ≤ 1 n r . \sum_{s=1}^{\infty}\beta_s \le\frac{1}{n_r}. s = 1 ∑ ∞ β s ≤ n r 1 .
Proof. Since N s + 1 = N s + n b ( s ) ≥ N s N_{s+1} = N_s + n_b(s) \ge N_s N s + 1 = N s + n b ( s ) ≥ N s from (3) ,
β s = n b ( s ) N s + 1 2 = N s + 1 − N s N s + 1 2 ≤ N s + 1 − N s N s N s + 1 = 1 N s − 1 N s + 1 , \beta_s = \frac{n_b(s)}{N_{s+1}^2} = \frac{N_{s+1}-N_s}{N_{s+1}^2} \le\frac{N_{s+1}-N_s}{N_sN_{s+1}} = \frac{1}{N_s} - \frac{1}{N_{s+1}}, β s = N s + 1 2 n b ( s ) = N s + 1 2 N s + 1 − N s ≤ N s N s + 1 N s + 1 − N s = N s 1 − N s + 1 1 ,
where we used the definition of β s \beta_s β s from (18) . Summing telescopes:
∑ s = 1 t − 1 β s ≤ 1 N 1 − 1 N t < 1 N 1 = 1 n r . \sum_{s=1}^{t-1}\beta_s \le\frac{1}{N_1} - \frac{1}{N_t} < \frac{1}{N_1} = \frac{1}{n_r}. s = 1 ∑ t − 1 β s ≤ N 1 1 − N t 1 < N 1 1 = n r 1 .
Taking the limit t → ∞ t\to\infty t → ∞ , we get ∑ s = 1 ∞ β s ≤ 1 / n r \sum_{s=1}^{\infty}\beta_s \le1/n_r ∑ s = 1 ∞ β s ≤ 1/ n r .
:::
Next, we prove a lemma that is used in the proof of Theorem A.13 and Lemma A.25.
Lemma A.6 (Product-decay criterion). Let ( r s ) s ≥ 1 ⊂ ( 0 , 1 ) (r_s)_{s\ge1}\subset(0,1) ( r s ) s ≥ 1 ⊂ ( 0 , 1 ) . If ∏ s = 1 t r s → 0 \prod_{s=1}^{t}r_s\to0 ∏ s = 1 t r s → 0 as t → ∞ t\to\infty t → ∞ , then ∑ s = 1 ∞ ( 1 − r s ) = ∞ \sum_{s=1}^{\infty}(1-r_s)=\infty ∑ s = 1 ∞ ( 1 − r s ) = ∞ .
Proof. We prove the contrapositive: if ∑ s = 1 ∞ ( 1 − r s ) < ∞ \sum_{s=1}^{\infty}(1-r_s)<\infty ∑ s = 1 ∞ ( 1 − r s ) < ∞ , then ∏ s = 1 t r s \prod_{s=1}^{t}r_s ∏ s = 1 t r s has a positive limit (as 1 − r s > 0 1-r_s>0 1 − r s > 0 , the series either converges or diverges to + ∞ +\infty + ∞ , so this is indeed the negation of the conclusion).
Assume ∑ s = 1 ∞ ( 1 − r s ) < ∞ \sum_{s=1}^{\infty}(1-r_s)<\infty ∑ s = 1 ∞ ( 1 − r s ) < ∞ . Then, 1 − r s → 0 1-r_s\to0 1 − r s → 0 , so there is s 0 ≥ 1 s_0\ge1 s 0 ≥ 1 with r s ≥ 1 2 r_s\ge\frac{1}{2} r s ≥ 2 1 for s ≥ s 0 s\ge s_0 s ≥ s 0 . In addition, for x ∈ [ 1 2 , 1 ) x \in[\frac{1}{2},1) x ∈ [ 2 1 , 1 ) , the function g ( x ) : = 2 ( 1 − x ) + ln x g(x) := 2(1-x)+\ln x g ( x ) := 2 ( 1 − x ) + ln x is decreasing: g ′ ( x ) = 1 / x − 2 ≤ 0 g'(x) = 1/x-2 \leq0 g ′ ( x ) = 1/ x − 2 ≤ 0 , and g ( 1 ) = 0 g(1) = 0 g ( 1 ) = 0 . Thus, g ≥ 0 g \geq0 g ≥ 0 on [ 1 2 , 1 ) [\frac{1}{2},1) [ 2 1 , 1 ) , i.e., − ln x ≤ 2 ( 1 − x ) -\ln x \leq2(1-x) − ln x ≤ 2 ( 1 − x ) .
Finally, for s ≥ s 0 s \geq s_0 s ≥ s 0 , we have r s ≥ 1 2 r_s \geq\frac{1}{2} r s ≥ 2 1 , so ∑ s ≥ s 0 ( − ln r s ) ≤ 2 ∑ s ≥ s 0 ( 1 − r s ) < ∞ \sum_{s \geq s_0}(-\ln r_s) \leq2\sum_{s \geq s_0}(1-r_s) < \infty ∑ s ≥ s 0 ( − ln r s ) ≤ 2 ∑ s ≥ s 0 ( 1 − r s ) < ∞ . That is, ∑ s ≥ 1 ( − ln r s ) < ∞ \sum_{s \geq1}(-\ln r_s) < \infty ∑ s ≥ 1 ( − ln r s ) < ∞ (the finitely many remaining terms are finite as r s > 0 r_s > 0 r s > 0 ) and ∏ s = 1 t r s = exp ( − ∑ s = 1 t ( − ln r s ) ) \prod_{s=1}^{t}r_s = \exp(-\sum_{s=1}^{t}(-\ln r_s)) ∏ s = 1 t r s = exp ( − ∑ s = 1 t ( − ln r s )) converges to a strictly positive limit, as claimed.
We now provide more useful expressions concerning the estimators μ ( t ) \mu^{(t)} μ ( t ) and Σ ( t ) \Sigma^{(t)} Σ ( t ) .
Theorem A.7 (Estimators formulas). Fix t ≥ 2 t \geq2 t ≥ 2 and consider the setup in Section 3.1. Then:
(a) The mean obeys the recursion
μ ( t ) = ρ t − 1 μ ( t − 1 ) + 1 N t ∑ i = 1 n b ( t − 1 ) x i ( t − 1 ) . (29) \mu^{(t)} = \rho_{t-1}\mu^{(t-1)} + \frac{1}{N_t}\sum_{i=1}^{n_b(t-1)}x_i^{(t-1)}. \tag*{(29)} μ ( t ) = ρ t − 1 μ ( t − 1 ) + N t 1 i = 1 ∑ n b ( t − 1 ) x i ( t − 1 ) . (29) (b) Splitting the pool into the initial real batch and the subsequent batches, the covariance reads
Σ ( t ) = 1 N t ( ∑ j = 1 n r ( x j ( 0 ) − μ ( t ) ) ( x j ( 0 ) − μ ( t ) ) ⊤ + ∑ s = 1 t − 1 ∑ i = 1 n b ( s ) ( x i ( s ) − μ ( t ) ) ( x i ( s ) − μ ( t ) ) ⊤ ) . (30) \Sigma^{(t)} = \frac{1}{N_t}\left(\sum_{j=1}^{n_r}(x_j^{(0)}-\mu^{(t)})(x_j^{(0)}-\mu^{(t)})^\top+ \sum_{s=1}^{t-1}\sum_{i=1}^{n_b(s)}(x_i^{(s)}-\mu^{(t)})(x_i^{(s)}-\mu^{(t)})^\top\right). \tag*{(30)} Σ ( t ) = N t 1 j = 1 ∑ n r ( x j ( 0 ) − μ ( t ) ) ( x j ( 0 ) − μ ( t ) ) ⊤ + s = 1 ∑ t − 1 i = 1 ∑ n b ( s ) ( x i ( s ) − μ ( t ) ) ( x i ( s ) − μ ( t ) ) ⊤ . (30)
Remark. (29) expresses the updated mean as a weighted average of the previous estimate and the new batch mean. (30) is the pooled sample covariance centered at the estimated mean μ ( t ) \mu^{(t)} μ ( t ) .
Proof. Step 1 (part (a): mean recursion (29). Split the most recent batch s = t − 1 s=t-1 s = t − 1 off from the definition (4) of μ ( t ) \mu^{(t)} μ ( t ) :
μ ( t ) = 1 N t ( ∑ j = 1 n r x j ( 0 ) + ∑ s = 1 t − 2 ∑ i = 1 n b ( s ) x i ( s ) ) + 1 N t ∑ i = 1 n b ( t − 1 ) x i ( t − 1 ) . \mu^{(t)} = \frac{1}{N_t}\left(\sum_{j=1}^{n_r}x_j^{(0)}+\sum_{s=1}^{t-2}\sum_{i=1}^{n_b(s)}x_i^{(s)}\right)+\frac{1}{N_t}\sum_{i=1}^{n_b(t-1)}x_i^{(t-1)}. μ ( t ) = N t 1 j = 1 ∑ n r x j ( 0 ) + s = 1 ∑ t − 2 i = 1 ∑ n b ( s ) x i ( s ) + N t 1 i = 1 ∑ n b ( t − 1 ) x i ( t − 1 ) .
The parenthesized sum is exactly the pool of the first N t − 1 = n r + ∑ s = 1 t − 2 n b ( s ) N_{t-1}=n_r+\sum_{s=1}^{t-2}n_b(s) N t − 1 = n r + ∑ s = 1 t − 2 n b ( s ) points, so applying the definition (4) at iteration t − 1 t-1 t − 1 —which for t = 2 t=2 t = 2 is precisely (1)—gives ∑ j = 1 n r x j ( 0 ) + ∑ s = 1 t − 2 ∑ i = 1 n b ( s ) x i ( s ) = N t − 1 μ ( t − 1 ) \sum_{j=1}^{n_r}x_j^{(0)}+\sum_{s=1}^{t-2}\sum_{i=1}^{n_b(s)}x_i^{(s)}=N_{t-1}\mu^{(t-1)} ∑ j = 1 n r x j ( 0 ) + ∑ s = 1 t − 2 ∑ i = 1 n b ( s ) x i ( s ) = N t − 1 μ ( t − 1 ) . With ρ t − 1 = N t − 1 / N t \rho_{t-1}=N_{t-1}/N_t ρ t − 1 = N t − 1 / N t from (3),
μ ( t ) = N t − 1 N t μ ( t − 1 ) + 1 N t ∑ i = 1 n b ( t − 1 ) x i ( t − 1 ) = ρ t − 1 μ ( t − 1 ) + 1 N t ∑ i = 1 n b ( t − 1 ) x i ( t − 1 ) , \mu^{(t)} = \frac{N_{t-1}}{N_t}\mu^{(t-1)}+\frac{1}{N_t}\sum_{i=1}^{n_b(t-1)}x_i^{(t-1)}=\rho_{t-1}\mu^{(t-1)}+\frac{1}{N_t}\sum_{i=1}^{n_b(t-1)}x_i^{(t-1)}, μ ( t ) = N t N t − 1 μ ( t − 1 ) + N t 1 i = 1 ∑ n b ( t − 1 ) x i ( t − 1 ) = ρ t − 1 μ ( t − 1 ) + N t 1 i = 1 ∑ n b ( t − 1 ) x i ( t − 1 ) ,
which is (29).
Step 2 (part (b)). Relabeling the N t N_t N t points in (4) by provenance splits the pooled scatter into the initial real batch and the subsequent batches:
Σ ( t ) = 1 N t ( ∑ j = 1 n r ( x j ( 0 ) − μ ( t ) ) ( x j ( 0 ) − μ ( t ) ) ⊤ + ∑ s = 1 t − 1 ∑ i = 1 n b ( s ) ( x i ( s ) − μ ( t ) ) ( x i ( s ) − μ ( t ) ) ⊤ ) , \Sigma^{(t)} = \frac{1}{N_t}\left(\sum_{j=1}^{n_r}(x_j^{(0)}-\mu^{(t)})(x_j^{(0)}-\mu^{(t)})^\top+\sum_{s=1}^{t-1}\sum_{i=1}^{n_b(s)}(x_i^{(s)}-\mu^{(t)})(x_i^{(s)}-\mu^{(t)})^\top\right), Σ ( t ) = N t 1 j = 1 ∑ n r ( x j ( 0 ) − μ ( t ) ) ( x j ( 0 ) − μ ( t ) ) ⊤ + s = 1 ∑ t − 1 i = 1 ∑ n b ( s ) ( x i ( s ) − μ ( t ) ) ( x i ( s ) − μ ( t ) ) ⊤ ,
which is exactly (30). This completes the proof.
Finally, we prove that the estimated covariance Σ ( t ) \Sigma^{(t)} Σ ( t ) is PSD.
Lemma A.8 (The covariance estimator is PSD). Fix t ≥ 1 t \geq1 t ≥ 1 . Consider the setup in Section 3.1. For every realization of the data, the sample covariance matrix Σ ( t ) \Sigma^{(t)} Σ ( t ) is positive-semi-definite.
Proof. Fix t ≥ 1 t \ge1 t ≥ 1 . By (30) — with the convention that the double sum ∑ s = 1 t − 1 \sum_{s=1}^{t-1} ∑ s = 1 t − 1 is empty when t = 1 t=1 t = 1 , so that Σ ( 1 ) \Sigma^{(1)} Σ ( 1 ) reduces to (1) — the estimator is a nonnegatively weighted sum of outer products of the centered points x i ( s ) − μ ( t ) x_i^{(s)}-\mu^{(t)} x i ( s ) − μ ( t ) . Each such outer product is symmetric, hence so is Σ ( t ) \Sigma^{(t)} Σ ( t ) . Moreover, for every v ∈ R d v \in\mathbb{R}^d v ∈ R d ,
v ⊤ Σ ( t ) v = 1 N t ( ∑ j = 1 n r ( v ⊤ ( x j ( 0 ) − μ ( t ) ) ) 2 + ∑ s = 1 t − 1 ∑ i = 1 n b ( s ) ( v ⊤ ( x i ( s ) − μ ( t ) ) ) 2 ) ≥ 0 , v^\top\Sigma^{(t)}v=\frac{1}{N_t}\left(\sum_{j=1}^{n_r}\left(v^\top(x_j^{(0)}-\mu^{(t)})\right)^2+\sum_{s=1}^{t-1}\sum_{i=1}^{n_b^{(s)}}\left(v^\top(x_i^{(s)}-\mu^{(t)})\right)^2\right)\ge0, v ⊤ Σ ( t ) v = N t 1 j = 1 ∑ n r ( v ⊤ ( x j ( 0 ) − μ ( t ) ) ) 2 + s = 1 ∑ t − 1 i = 1 ∑ n b ( s ) ( v ⊤ ( x i ( s ) − μ ( t ) ) ) 2 ≥ 0 ,
since it is a sum of squares scaled by 1 / N t > 0 1/N_t>0 1/ N t > 0 . Hence Σ ( t ) ⪰ 0 \Sigma^{(t)}\succeq0 Σ ( t ) ⪰ 0 .
Remark (singular covariance). If Σ ( t ) \Sigma^{(t)} Σ ( t ) is singular, N ( μ ( t ) , Σ ( t ) ) \mathcal{N}(\mu^{(t)},\Sigma^{(t)}) N ( μ ( t ) , Σ ( t ) ) in (2) denotes the corresponding degenerate Gaussian, i.e., the law of μ ( t ) + ( Σ ( t ) ) 1 / 2 z \mu^{(t)}+(\Sigma^{(t)})^{1/2}z μ ( t ) + ( Σ ( t ) ) 1/2 z with z ∼ N ( 0 , I d ) z\sim\mathcal{N}(0,I_d) z ∼ N ( 0 , I d ) , where ( Σ ( t ) ) 1 / 2 (\Sigma^{(t)})^{1/2} ( Σ ( t ) ) 1/2 is the unique symmetric PSD square root, which exists by Lemma A.8.
In the following subsections, we present the proofs of Theorem 3.1 , Lemma 3.2 , Theorem 3.3 , Lemma 3.4 , Proposition 3.5 , Theorem 3.6 and Proposition A.27 .
Proof of Theorem 3.1
Proof of part (a): μ ( t ) \mu^{(t)} μ ( t ) is unbiased. We prove by induction on t ≥ 1 t \ge1 t ≥ 1 .
Base case (t = 1 t=1 t = 1 ). By (1) , μ ( 1 ) = 1 n r ∑ j = 1 n r x j ( 0 ) \mu^{(1)}=\frac{1}{n_r}\sum_{j=1}^{n_r}x_j^{(0)} μ ( 1 ) = n r 1 ∑ j = 1 n r x j ( 0 ) is the sample mean of n r n_r n r i.i.d. draws from N ( μ , Σ ) \mathcal{N}(\mu,\Sigma) N ( μ , Σ ) , so
E [ μ ( 1 ) ] = 1 n r ∑ j = 1 n r E [ x j ( 0 ) ] = μ . \mathbb{E}\left[\mu^{(1)}\right]=\frac{1}{n_r}\sum_{j=1}^{n_r}\mathbb{E}\left[x_j^{(0)}\right]=\mu. E [ μ ( 1 ) ] = n r 1 j = 1 ∑ n r E [ x j ( 0 ) ] = μ .
Inductive step. Fix t ≥ 2 t \ge2 t ≥ 2 and assume E [ μ ( t − 1 ) ] = μ \mathbb{E}\left[\mu^{(t-1)}\right]=\mu E [ μ ( t − 1 ) ] = μ . Taking expectations in (29) and using the linearity of expectation, we get
E [ μ ( t ) ] = ρ t − 1 E [ μ ( t − 1 ) ] + 1 N t ∑ i = 1 n b ( t − 1 ) E [ x i ( t − 1 ) ] . (31) \mathbb{E}\left[\mu^{(t)}\right]=\rho_{t-1}\mathbb{E}\left[\mu^{(t-1)}\right]+\frac{1}{N_t}\sum_{i=1}^{n_b^{(t-1)}}\mathbb{E}\left[x_i^{(t-1)}\right]. \tag*{(31)} E [ μ ( t ) ] = ρ t − 1 E [ μ ( t − 1 ) ] + N t 1 i = 1 ∑ n b ( t − 1 ) E [ x i ( t − 1 ) ] . (31)
To evaluate E [ x i ( t − 1 ) ] \mathbb{E}\left[x_i^{(t-1)}\right] E [ x i ( t − 1 ) ] , we condition on F t − 1 \mathcal{F}_{t-1} F t − 1 (recall (20) ). By Assumption 1, given F t − 1 \mathcal{F}_{t-1} F t − 1 , the samples x i ( t − 1 ) x_i^{(t-1)} x i ( t − 1 ) are i.i.d. from the mixture (2) , whose mean is α t − 1 μ ( t − 1 ) + ( 1 − α t − 1 ) μ \alpha_{t-1}\mu^{(t-1)}+(1-\alpha_{t-1})\mu α t − 1 μ ( t − 1 ) + ( 1 − α t − 1 ) μ . Hence, by the tower property and the induction hypothesis,
E [ x i ( t − 1 ) ] = E [ E [ x i ( t − 1 ) ∣ F t − 1 ] ] = E [ α t − 1 μ ( t − 1 ) + ( 1 − α t − 1 ) μ ] = α t − 1 E [ μ ( t − 1 ) ] + ( 1 − α t − 1 ) μ = α t − 1 μ + ( 1 − α t − 1 ) μ = μ . \begin{aligned}
\mathbb{E}\left[x_i^{(t-1)}\right]=\mathbb{E}\left[\mathbb{E}\left[x_i^{(t-1)}\mid\mathcal{F}_{t-1}\right]\right]=\mathbb{E}\left[\alpha_{t-1}\mu^{(t-1)}+(1-\alpha_{t-1})\mu\right] \\
&=\alpha_{t-1}\mathbb{E}\left[\mu^{(t-1)}\right]+(1-\alpha_{t-1})\mu=\alpha_{t-1}\mu+(1-\alpha_{t-1})\mu=\mu.
\end{aligned} E [ x i ( t − 1 ) ] = E [ E [ x i ( t − 1 ) ∣ F t − 1 ] ] = E [ α t − 1 μ ( t − 1 ) + ( 1 − α t − 1 ) μ ] = α t − 1 E [ μ ( t − 1 ) ] + ( 1 − α t − 1 ) μ = α t − 1 μ + ( 1 − α t − 1 ) μ = μ .
Substituting E [ μ ( t − 1 ) ] = μ \mathbb{E}\left[\mu^{(t-1)}\right]=\mu E [ μ ( t − 1 ) ] = μ and E [ x i ( t − 1 ) ] = μ \mathbb{E}\left[x_i^{(t-1)}\right]=\mu E [ x i ( t − 1 ) ] = μ into (31) ,
E [ μ ( t ) ] = ρ t − 1 μ + n b ( t − 1 ) N t μ = N t − 1 + n b ( t − 1 ) N t μ = μ , \mathbb{E}\left[\mu^{(t)}\right]=\rho_{t-1}\mu+\frac{n_b(t-1)}{N_t}\mu=\frac{N_{t-1}+n_b(t-1)}{N_t}\mu=\mu, E [ μ ( t ) ] = ρ t − 1 μ + N t n b ( t − 1 ) μ = N t N t − 1 + n b ( t − 1 ) μ = μ ,
where we used ρ t − 1 = N t − 1 / N t \rho_{t-1}=N_{t-1}/N_t ρ t − 1 = N t − 1 / N t together with N t − 1 + n b ( t − 1 ) = N t N_{t-1}+n_b(t-1)=N_t N t − 1 + n b ( t − 1 ) = N t , which follows directly from (3) .
Proof of part (b): bias of Σ ( t ) \Sigma^{(t)} Σ ( t ) . Step 1: the pooled scatter identity. From (10) , we have
E [ δ ( s ) ( δ ( s ) ) ⊤ ] = E [ ( μ ( s ) − μ ) ( μ ( s ) − μ ) ⊤ ] = C s , s ≥ 1. (32) \mathbb{E}\left[\delta^{(s)}(\delta^{(s)})^\top\right]=\mathbb{E}\left[(\mu^{(s)}-\mu)(\mu^{(s)}-\mu)^\top\right]=C_s,\qquad s\ge1. \tag*{(32)} E [ δ ( s ) ( δ ( s ) ) ⊤ ] = E [ ( μ ( s ) − μ ) ( μ ( s ) − μ ) ⊤ ] = C s , s ≥ 1. (32)
Multiplying by N t N_t N t and taking expectations in (25) , we get
N t E [ Σ ( t ) ] = ∑ j = 1 n r E [ ( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) ⊤ ] + ∑ s = 1 t − 1 ∑ i = 1 n b ( s ) E [ ( x i ( s ) − μ ) ( x i ( s ) − μ ) ⊤ ] − N t C t = n r Σ + ∑ s = 1 t − 1 ∑ i = 1 n b ( s ) E [ ( x i ( s ) − μ ) ( x i ( s ) − μ ) ⊤ ] − N t C t . (33) \begin{aligned}
N_t\mathbb{E}\left[\Sigma^{(t)}\right]=\sum_{j=1}^{n_r}\mathbb{E}\left[(x_j^{(0)}-\mu)(x_j^{(0)}-\mu)^\top\right]+\sum_{s=1}^{t-1}\sum_{i=1}^{n_b^{(s)}}\mathbb{E}\left[(x_i^{(s)}-\mu)(x_i^{(s)}-\mu)^\top\right]-N_tC_t \\
&=n_r\Sigma+\sum_{s=1}^{t-1}\sum_{i=1}^{n_b^{(s)}}\mathbb{E}\left[(x_i^{(s)}-\mu)(x_i^{(s)}-\mu)^\top\right]-N_tC_t. \tag*{(33)}
\end{aligned} N t E [ Σ ( t ) ] = j = 1 ∑ n r E [ ( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) ⊤ ] + s = 1 ∑ t − 1 i = 1 ∑ n b ( s ) E [ ( x i ( s ) − μ ) ( x i ( s ) − μ ) ⊤ ] − N t C t = n r Σ + s = 1 ∑ t − 1 i = 1 ∑ n b ( s ) E [ ( x i ( s ) − μ ) ( x i ( s ) − μ ) ⊤ ] − N t C t . (33)
where we used that x j ( 0 ) ∼ N ( μ , Σ ) x_j^{(0)} \sim\mathcal{N}(\mu, \boldsymbol{\Sigma}) x j ( 0 ) ∼ N ( μ , Σ ) are i.i.d., so each contributes E [ ( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) T ] = Σ \mathbb{E}[(x_j^{(0)}-\mu)(x_j^{(0)}-\mu)^{\mathsf{T}}] = \boldsymbol{\Sigma} E [( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) T ] = Σ .
Step 2: induction on t t t . We show E [ Σ ( t ) ] = Σ − C t \mathbb{E}[\boldsymbol{\Sigma}^{(t)}] = \boldsymbol{\Sigma} - C_t E [ Σ ( t ) ] = Σ − C t .
Base case (t = 1 t = 1 t = 1 ). We have N 1 = n r N_1 = n_r N 1 = n r , and thus (33) gives
n r E [ Σ ( 1 ) ] = n r Σ − n r C 1 ⇒ E [ Σ ( 1 ) ] = Σ − C 1 . n_r\mathbb{E}[\boldsymbol{\Sigma}^{(1)}] = n_r\boldsymbol{\Sigma} - n_rC_1 \Rightarrow\mathbb{E}[\boldsymbol{\Sigma}^{(1)}] = \boldsymbol{\Sigma} - C_1. n r E [ Σ ( 1 ) ] = n r Σ − n r C 1 ⇒ E [ Σ ( 1 ) ] = Σ − C 1 .
Inductive step. Assume E [ Σ ( s ) ] = Σ − C s \mathbb{E}[\boldsymbol{\Sigma}^{(s)}] = \boldsymbol{\Sigma} - C_s E [ Σ ( s ) ] = Σ − C s for all 1 ≤ s ≤ t − 1 1 \le s \le t - 1 1 ≤ s ≤ t − 1 . For a newly drawn sample x i ( s ) x_i^{(s)} x i ( s ) with 1 ≤ s ≤ t − 1 1 \le s \le t - 1 1 ≤ s ≤ t − 1 , the tower property, identity (23) from Lemma A.1, the induction hypothesis, and E [ δ ( s ) ( δ ( s ) ) T ] = C s \mathbb{E}[\delta^{(s)}(\delta^{(s)})^{\mathsf{T}}] = C_s E [ δ ( s ) ( δ ( s ) ) T ] = C s (from (32)) give
E [ ( x i ( s ) − μ ) ( x i ( s ) − μ ) T ] = E [ E [ ( x i ( s ) − μ ) ( x i ( s ) − μ ) T ∣ F s ] ] = E [ α s Σ ( s ) + ( 1 − α s ) Σ + α s δ ( s ) ( δ ( s ) ) T ] = α s E [ Σ ( s ) ] + ( 1 − α s ) Σ + α s C s = α s ( Σ − C s ) + ( 1 − α s ) Σ + α s C s = Σ . \begin{aligned}
\mathbb{E}[(x_i^{(s)}-\mu)(x_i^{(s)}-\mu)^{\mathsf{T}}]
= \mathbb{E}\left[\mathbb{E}\left[(x_i^{(s)}-\mu)(x_i^{(s)}-\mu)^{\mathsf{T}} \mid\mathcal{F}_s\right]\right] \\
&= \mathbb{E}\left[\alpha_s\boldsymbol{\Sigma}^{(s)} + (1-\alpha_s)\boldsymbol{\Sigma} + \alpha_s\delta^{(s)}(\delta^{(s)})^{\mathsf{T}}\right] \\
&= \alpha_s\mathbb{E}[\boldsymbol{\Sigma}^{(s)}] + (1-\alpha_s)\boldsymbol{\Sigma} + \alpha_s C_s \\
&= \alpha_s(\boldsymbol{\Sigma} - C_s) + (1-\alpha_s)\boldsymbol{\Sigma} + \alpha_s C_s = \boldsymbol{\Sigma}.
\end{aligned} E [( x i ( s ) − μ ) ( x i ( s ) − μ ) T ] = E [ E [ ( x i ( s ) − μ ) ( x i ( s ) − μ ) T ∣ F s ] ] = E [ α s Σ ( s ) + ( 1 − α s ) Σ + α s δ ( s ) ( δ ( s ) ) T ] = α s E [ Σ ( s ) ] + ( 1 − α s ) Σ + α s C s = α s ( Σ − C s ) + ( 1 − α s ) Σ + α s C s = Σ .
Substituting into (33) and using n r + ∑ s = 1 t − 1 n b ( s ) = N t n_r + \sum_{s=1}^{t-1} n_b(s) = N_t n r + ∑ s = 1 t − 1 n b ( s ) = N t ,
N t E [ Σ ( t ) ] = n r Σ + ∑ s = 1 t − 1 n b ( s ) Σ − N t C t = N t Σ − N t C t , N_t\mathbb{E}[\boldsymbol{\Sigma}^{(t)}] = n_r\boldsymbol{\Sigma} + \sum_{s=1}^{t-1} n_b(s)\boldsymbol{\Sigma} - N_tC_t = N_t\boldsymbol{\Sigma} - N_tC_t, N t E [ Σ ( t ) ] = n r Σ + s = 1 ∑ t − 1 n b ( s ) Σ − N t C t = N t Σ − N t C t ,
so E [ Σ ( t ) ] = Σ − C t \mathbb{E}[\boldsymbol{\Sigma}^{(t)}] = \boldsymbol{\Sigma} - C_t E [ Σ ( t ) ] = Σ − C t .
Steps towards proving Theorem 3.3
In this subsection, we present the steps towards proving Theorem 3.3.
Range of scalar sequences
First, we characterize the ranges of the three sequences defined in (18) and ρ t \rho_t ρ t , defined in (3). These ranges will be used extensively in the subsequent results.
Lemma A.9 (Range of λ t \lambda_t λ t , γ t \gamma_t γ t , ρ t \rho_t ρ t ). 0 < λ t ≤ 1 0 < \lambda_t \le1 0 < λ t ≤ 1 , 0 < γ t < 1 0 < \gamma_t < 1 0 < γ t < 1 and 0 < ρ t < 1 0 < \rho_t < 1 0 < ρ t < 1 for all t ≥ 1 t \ge1 t ≥ 1 .
Remark. For 1 ≤ s ≤ t 1 \le s \le t 1 ≤ s ≤ t , (19), and Lemma A.9 imply 0 < Γ s , t < 1 0 < \Gamma_{s,t} < 1 0 < Γ s , t < 1 .
Proof. Fix t ≥ 1 t \ge1 t ≥ 1 . From (3) we have N t + 1 = N t + n b ( t ) N_{t+1} = N_t + n_b(t) N t + 1 = N t + n b ( t ) . Remember that N t ≥ n r ≥ 1 N_t \ge n_r \ge1 N t ≥ n r ≥ 1 and n b ( t ) ≥ 1 n_b(t) \ge1 n b ( t ) ≥ 1 .
Range of λ t \lambda_t λ t . By (18) and α t ∈ [ 0 , 1 ] \alpha_t \in[0,1] α t ∈ [ 0 , 1 ] , we have
λ t = N t + n b ( t ) α t N t + 1 = N t + n b ( t ) α t N t + n b ( t ) ⇒ 0 < λ t ≤ 1 \lambda_t = \frac{N_t+n_b(t)\alpha_t}{N_{t+1}} = \frac{N_t+n_b(t)\alpha_t}{N_t+n_b(t)} \Rightarrow0 < \lambda_t \le1 λ t = N t + 1 N t + n b ( t ) α t = N t + n b ( t ) N t + n b ( t ) α t ⇒ 0 < λ t ≤ 1
Range of ρ t \rho_t ρ t . By (3) and n b ( t ) ≥ 1 n_b(t) \ge1 n b ( t ) ≥ 1 , we have
ρ t = N t N t + 1 = N t N t + n b ( t ) ⇒ 0 < ρ t < 1 \rho_t = \frac{N_t}{N_{t+1}} = \frac{N_t}{N_t+n_b(t)} \Rightarrow0 < \rho_t < 1 ρ t = N t + 1 N t = N t + n b ( t ) N t ⇒ 0 < ρ t < 1
Positivity γ t > 0 \gamma_t > 0 γ t > 0 . By (18) we have
β t = n b ( t ) N t + 1 2 = n b ( t ) ( N t + n b ( t ) ) 2 (34) \beta_t = \frac{n_b(t)}{N_{t+1}^{2}} = \frac{n_b(t)}{(N_t+n_b(t))^{2}} \tag*{(34)} β t = N t + 1 2 n b ( t ) = ( N t + n b ( t ) ) 2 n b ( t ) (34)
Substituting (34) into the definition γ t = λ t 2 − α t 2 β t \gamma_t = \lambda_t^2 - \alpha_t^2\beta_t γ t = λ t 2 − α t 2 β t from (18),
γ t = ( N t + n b ( t ) α t N t + n b ( t ) ) 2 − α t 2 n b ( t ) ( N t + n b ( t ) ) 2 = ( N t + n b ( t ) α t ) 2 − α t 2 n b ( t ) ( N t + n b ( t ) ) 2 = N t 2 + 2 N t n b ( t ) α t + α t 2 n b ( t ) ( n b ( t ) − 1 ) ( N t + n b ( t ) ) 2 ≥ N t 2 ( N t + n b ( t ) ) 2 > 0 , (35) \begin{aligned}
\gamma_t
&= \left(\frac{N_t+n_b(t)\alpha_t}{N_t+n_b(t)}\right)^2-\frac{\alpha_t^2n_b(t)}{(N_t+n_b(t))^2}\\
&=\frac{(N_t+n_b(t)\alpha_t)^2-\alpha_t^2n_b(t)}{(N_t+n_b(t))^2}\\
&=\frac{N_t^2+2N_tn_b(t)\alpha_t+\alpha_t^2n_b(t)(n_b(t)-1)}{(N_t+n_b(t))^2}\\
&\geq\frac{N_t^2}{(N_t+n_b(t))^2}>0, \tag*{(35)}
\end{aligned} γ t = ( N t + n b ( t ) N t + n b ( t ) α t ) 2 − ( N t + n b ( t ) ) 2 α t 2 n b ( t ) = ( N t + n b ( t ) ) 2 ( N t + n b ( t ) α t ) 2 − α t 2 n b ( t ) = ( N t + n b ( t ) ) 2 N t 2 + 2 N t n b ( t ) α t + α t 2 n b ( t ) ( n b ( t ) − 1 ) ≥ ( N t + n b ( t ) ) 2 N t 2 > 0 , (35)
where we used 2 N t n b ( t ) α t ≥ 0 2N_tn_b(t)\alpha_t\geq0 2 N t n b ( t ) α t ≥ 0 and α t 2 n b ( t ) ( n b ( t ) − 1 ) ≥ 0 \alpha_t^2n_b(t)(n_b(t)-1)\geq0 α t 2 n b ( t ) ( n b ( t ) − 1 ) ≥ 0 (as n b ( t ) ≥ 1 n_b(t)\geq1 n b ( t ) ≥ 1 and α t ∈ [ 0 , 1 ] \alpha_t\in[0,1] α t ∈ [ 0 , 1 ] ). Finally, γ t > 0 \gamma_t>0 γ t > 0 .
Upper bound γ t < 1 \gamma_t<1 γ t < 1 . From (35) ,
1 − γ t = ( N t + n b ( t ) ) 2 − ( N t + n b ( t ) α t ) 2 + α t 2 n b ( t ) ( N t + n b ( t ) ) 2 = 2 N t n b ( t ) ( 1 − α t ) + n b 2 ( t ) ( 1 − α t 2 ) + α t 2 n b ( t ) ( N t + n b ( t ) ) 2 = ( 1 − α t ) n b ( t ) ⋅ ( 2 N t + n b ( t ) ( 1 + α t ) ) + α t 2 n b ( t ) ( N t + n b ( t ) ) 2 = n b ( t ) ( N t + n b ( t ) ) 2 ⋅ ( ( 1 − α t ) ⋅ ( 2 N t + n b ( t ) ( 1 + α t ) ) + α t 2 ) . \begin{aligned}
1-\gamma_t
&=\frac{(N_t+n_b(t))^2-(N_t+n_b(t)\alpha_t)^2+\alpha_t^2n_b(t)}{(N_t+n_b(t))^2}\\
&=\frac{2N_tn_b(t)(1-\alpha_t)+n_b^2(t)(1-\alpha_t^2)+\alpha_t^2n_b(t)}{(N_t+n_b(t))^2}\\
&=\frac{(1-\alpha_t)n_b(t)\cdot(2N_t+n_b(t)(1+\alpha_t))+\alpha_t^2n_b(t)}{(N_t+n_b(t))^2}\\
&=\frac{n_b(t)}{(N_t+n_b(t))^2}\cdot\left((1-\alpha_t)\cdot(2N_t+n_b(t)(1+\alpha_t))+\alpha_t^2\right).
\end{aligned} 1 − γ t = ( N t + n b ( t ) ) 2 ( N t + n b ( t ) ) 2 − ( N t + n b ( t ) α t ) 2 + α t 2 n b ( t ) = ( N t + n b ( t ) ) 2 2 N t n b ( t ) ( 1 − α t ) + n b 2 ( t ) ( 1 − α t 2 ) + α t 2 n b ( t ) = ( N t + n b ( t ) ) 2 ( 1 − α t ) n b ( t ) ⋅ ( 2 N t + n b ( t ) ( 1 + α t )) + α t 2 n b ( t ) = ( N t + n b ( t ) ) 2 n b ( t ) ⋅ ( ( 1 − α t ) ⋅ ( 2 N t + n b ( t ) ( 1 + α t )) + α t 2 ) .
That is,
1 − γ t = β t ⋅ ( ( 1 − α t ) ⋅ ( 2 N t + n b ( t ) ( 1 + α t ) ) + α t 2 ) (36) 1-\gamma_t=\beta_t\cdot\left((1-\alpha_t)\cdot(2N_t+n_b(t)(1+\alpha_t))+\alpha_t^2\right) \tag*{(36)} 1 − γ t = β t ⋅ ( ( 1 − α t ) ⋅ ( 2 N t + n b ( t ) ( 1 + α t )) + α t 2 ) (36)
If 0 ≤ α t < 1 0\leq\alpha_t<1 0 ≤ α t < 1 , the expression above is strictly positive. Otherwise, α t = 1 \alpha_t=1 α t = 1 , and we have
1 − γ t = n b ( t ) ( N t + n b ( t ) ) 2 > 0. 1-\gamma_t=\frac{n_b(t)}{(N_t+n_b(t))^2}>0. 1 − γ t = ( N t + n b ( t ) ) 2 n b ( t ) > 0.
In either case, 1 − γ t > 0 ⇒ γ t < 1 1-\gamma_t>0\Rightarrow\gamma_t<1 1 − γ t > 0 ⇒ γ t < 1 .
Consistency of μ ( t ) \mu^{(t)} μ ( t )
We start by characterizing when E [ ∥ μ ( t ) − μ ∥ 2 2 ] → t → ∞ 0 \mathbb{E}[\lVert\mu^{(t)}-\mu\rVert_2^2]\xrightarrow{t\to\infty}0 E [∥ μ ( t ) − μ ∥ 2 2 ] t → ∞ 0 . First, we provide an equivalent condition in terms of the covariance of μ ( t ) \mu^{(t)} μ ( t ) .
Lemma A.10 (L 2 L^2 L 2 consistency equivalence). Under Assumption 1 , we have
C t → 0 ⟺ E [ ∥ μ ( t ) − μ ∥ 2 2 ] → t → ∞ 0 C_t\to0\Longleftrightarrow\mathbb{E}[\lVert\mu^{(t)}-\mu\rVert_2^2]\xrightarrow{t\to\infty}0 C t → 0 ⟺ E [∥ μ ( t ) − μ ∥ 2 2 ] t → ∞ 0 .
Remark. We note C t → 0 C_t\to0 C t → 0 with the Frobenius norm. This is valid because C t ∈ R d × d C_t\in\mathbb{R}^{d\times d} C t ∈ R d × d is in a finite dimensional space, which means that all norms are equivalent. Additionally, from Theorem 3.1(b), Equation (37) is equivalent to the asymptotic unbiasedness of the covariance estimator Σ ( t ) \Sigma^{(t)} Σ ( t ) .
Proof. By (11), E [ ∥ μ ( t ) − μ ∥ 2 2 ] → t → ∞ 0 \mathbb{E}[\lVert\mu^{(t)}-\mu\rVert_2^2]\xrightarrow{t\to\infty}0 E [∥ μ ( t ) − μ ∥ 2 2 ] t → ∞ 0 if and only if Tr ( C t ) → 0 \operatorname{Tr}(C_t)\to0 Tr ( C t ) → 0 . By (10), the matrix C t C_t C t is symmetric, and it is positive-semi-definite as a covariance matrix. Hence, Lemma A.4(b) gives Tr ( C t ) → 0 ⟺ C t → 0 \operatorname{Tr}(C_t)\to0\Longleftrightarrow C_t\to0 Tr ( C t ) → 0 ⟺ C t → 0 . Combining the two equivalences gives the goal equivalence.
Next, let us provide a closed-form formula for the covariance of μ ( t ) \mu^{(t)} μ ( t ) .
Theorem A.11 (Covariance of μ ( t ) \mu^{(t)} μ ( t ) ). Under Assumption 1 , with λ t \lambda_t λ t , β t \beta_t β t , γ t \gamma_t γ t as in (18), and Γ s , t \Gamma_{s,t} Γ s , t as in (19):
(a) Recursion.
C t = γ t − 1 C t − 1 + β t − 1 Σ , t ≥ 2 , C 1 = Σ n r . (37) C_t = \gamma_{t-1} C_{t-1} + \beta_{t-1}\Sigma,\qquad t \ge2,\qquad C_1 = \frac{\Sigma}{n_r}. \tag*{(37)} C t = γ t − 1 C t − 1 + β t − 1 Σ , t ≥ 2 , C 1 = n r Σ . (37)
(b) Closed form.
C t = Γ 1 , t − 1 C 1 + Σ ∑ s = 1 t − 1 β s Γ s + 1 , t − 1 . (38) C_t = \Gamma_{1,t-1} C_1 + \Sigma\sum_{s=1}^{t-1} \beta_s \Gamma_{s+1,t-1}. \tag*{(38)} C t = Γ 1 , t − 1 C 1 + Σ s = 1 ∑ t − 1 β s Γ s + 1 , t − 1 . (38)
Remark. In the recursion (38), γ t − 1 ∈ ( 0 , 1 ) \gamma_{t-1}\in(0,1) γ t − 1 ∈ ( 0 , 1 ) (Lemma A.9) acts as a damping factor on the covariance from the previous iteration, while the added terms β t − 1 Σ \beta_{t-1}\Sigma β t − 1 Σ are summable (Lemma A.5). Together, these yield the bound C t ⪯ 2 n r Σ C_t \preceq\frac{2}{n_r}\Sigma C t ⪯ n r 2 Σ of Lemma 3.2.
Proof of part (a): recursion. Apply the law of total covariance to μ ( t ) \mu^{(t)} μ ( t ) , conditioning on F t − 1 \mathcal{F}_{t-1} F t − 1 :
C t = Cov ( E [ μ ( t ) ∣ F t − 1 ] ) + E [ Cov ( μ ( t ) ∣ F t − 1 ) ] . (40) C_t = \operatorname{Cov}\left(\mathbb{E}[\mu^{(t)}\mid\mathcal{F}_{t-1}]\right)+\mathbb{E}\left[\operatorname{Cov}(\mu^{(t)}\mid\mathcal{F}_{t-1})\right]. \tag*{(40)} C t = Cov ( E [ μ ( t ) ∣ F t − 1 ] ) + E [ Cov ( μ ( t ) ∣ F t − 1 ) ] . (40)
Term 1 (covariance of the conditional mean). By (29) and (21) from Lemma A.1,
E [ μ ( t ) ∣ F t − 1 ] = E [ ρ t − 1 μ ( t − 1 ) + 1 N t ∑ i = 1 n b ( t − 1 ) x i ( t − 1 ) ∣ F t − 1 ] = ρ t − 1 μ ( t − 1 ) + 1 N t ∑ i = 1 n b ( t − 1 ) E [ x i ( t − 1 ) ∣ F t − 1 ] = ρ t − 1 μ ( t − 1 ) + n b ( t − 1 ) N t ( α t − 1 μ ( t − 1 ) + ( 1 − α t − 1 ) μ ) = N t − 1 + n b ( t − 1 ) α t − 1 N t μ ( t − 1 ) + ( 1 − α t − 1 ) n b ( t − 1 ) N t μ = λ t − 1 μ ( t − 1 ) + ( 1 − λ t − 1 ) μ (41) \begin{aligned}
\mathbb{E}[\mu^{(t)}\mid\mathcal{F}_{t-1}]
&=\mathbb{E}\left[\left.\rho_{t-1}\mu^{(t-1)}+\frac{1}{N_t}\sum_{i=1}^{n_b(t-1)}x_i^{(t-1)}\right|\mathcal{F}_{t-1}\right]\\
&=\rho_{t-1}\mu^{(t-1)}+\frac{1}{N_t}\sum_{i=1}^{n_b(t-1)}\mathbb{E}[x_i^{(t-1)}\mid\mathcal{F}_{t-1}]\\
&=\rho_{t-1}\mu^{(t-1)}+\frac{n_b(t-1)}{N_t}\left(\alpha_{t-1}\mu^{(t-1)}+(1-\alpha_{t-1})\mu\right)\\
&=\frac{N_{t-1}+n_b(t-1)\alpha_{t-1}}{N_t}\mu^{(t-1)}+\frac{(1-\alpha_{t-1})n_b(t-1)}{N_t}\mu\\
&=\lambda_{t-1}\mu^{(t-1)}+(1-\lambda_{t-1})\mu \tag*{(41)}
\end{aligned} E [ μ ( t ) ∣ F t − 1 ] = E ρ t − 1 μ ( t − 1 ) + N t 1 i = 1 ∑ n b ( t − 1 ) x i ( t − 1 ) F t − 1 = ρ t − 1 μ ( t − 1 ) + N t 1 i = 1 ∑ n b ( t − 1 ) E [ x i ( t − 1 ) ∣ F t − 1 ] = ρ t − 1 μ ( t − 1 ) + N t n b ( t − 1 ) ( α t − 1 μ ( t − 1 ) + ( 1 − α t − 1 ) μ ) = N t N t − 1 + n b ( t − 1 ) α t − 1 μ ( t − 1 ) + N t ( 1 − α t − 1 ) n b ( t − 1 ) μ = λ t − 1 μ ( t − 1 ) + ( 1 − λ t − 1 ) μ (41)
where we used the definition of λ t \lambda_t λ t in (18), and
1 − λ t = 1 − N t + n b ( t ) α t N t + 1 = N t + 1 − N t − n b ( t ) α t N t + 1 = ( 1 − α t ) n b ( t ) N t + 1 . 1-\lambda_t=1-\frac{N_t+n_b(t)\alpha_t}{N_{t+1}}=\frac{N_{t+1}-N_t-n_b(t)\alpha_t}{N_{t+1}}=\frac{(1-\alpha_t)n_b(t)}{N_{t+1}}. 1 − λ t = 1 − N t + 1 N t + n b ( t ) α t = N t + 1 N t + 1 − N t − n b ( t ) α t = N t + 1 ( 1 − α t ) n b ( t ) .
As λ t − 1 \lambda_{t-1} λ t − 1 is deterministic and ( 1 − λ t − 1 ) μ (1-\lambda_{t-1})\mu ( 1 − λ t − 1 ) μ is constant, from (42) we have
Cov ( E [ μ ( t ) ∣ F t − 1 ] ) = λ t − 1 2 Cov ( μ ( t − 1 ) ) = λ t − 1 2 C t − 1 . (42) \operatorname{Cov}\left(\mathbb{E}[\mu^{(t)}\mid\mathcal{F}_{t-1}]\right)=\lambda_{t-1}^{2}\operatorname{Cov}(\mu^{(t-1)})=\lambda_{t-1}^{2}C_{t-1}. \tag*{(42)} Cov ( E [ μ ( t ) ∣ F t − 1 ] ) = λ t − 1 2 Cov ( μ ( t − 1 ) ) = λ t − 1 2 C t − 1 . (42)
Term 2 (expectation of the conditional covariance). By (29), the conditional covariance reads:
Cov ( μ ( t ) ∣ F t − 1 ) = Cov ( ρ t − 1 μ ( t − 1 ) + 1 N t ∑ i = 1 n b ( t − 1 ) x i ( t − 1 ) ∣ F t − 1 ) = n b ( t − 1 ) N t 2 Cov ( x 1 ( t − 1 ) ∣ F t − 1 ) = β t − 1 Cov ( x 1 ( t − 1 ) ∣ F t − 1 ) \begin{aligned}
\operatorname{Cov}(\mu^{(t)}\mid\mathcal{F}_{t-1})
&=\operatorname{Cov}\left(\left.\rho_{t-1}\mu^{(t-1)}+\frac{1}{N_t}\sum_{i=1}^{n_b(t-1)}x_i^{(t-1)}\right|\mathcal{F}_{t-1}\right)\\
&=\frac{n_b(t-1)}{N_t^2}\operatorname{Cov}(x_1^{(t-1)}\mid\mathcal{F}_{t-1})\\
&=\beta_{t-1}\operatorname{Cov}(x_1^{(t-1)}\mid\mathcal{F}_{t-1})
\end{aligned} Cov ( μ ( t ) ∣ F t − 1 ) = Cov ρ t − 1 μ ( t − 1 ) + N t 1 i = 1 ∑ n b ( t − 1 ) x i ( t − 1 ) F t − 1 = N t 2 n b ( t − 1 ) Cov ( x 1 ( t − 1 ) ∣ F t − 1 ) = β t − 1 Cov ( x 1 ( t − 1 ) ∣ F t − 1 )
where we used that given F t − 1 \mathcal{F}_{t-1} F t − 1 , the term ρ t − 1 μ ( t − 1 ) \rho_{t-1}\mu^{(t-1)} ρ t − 1 μ ( t − 1 ) is deterministic, and the n b ( t − 1 ) n_b(t-1) n b ( t − 1 ) new samples are i.i.d. (Assumption 1), so the cross-covariances vanish. We also used the definition of β t \beta_t β t in (18). Using (22) from Lemma A.1 with t − 1 t-1 t − 1 , the conditional covariance is:
Cov ( μ ( t ) ∣ F t − 1 ) = β t − 1 ( α t − 1 Σ ( t − 1 ) + ( 1 − α t − 1 ) Σ + α t − 1 ( 1 − α t − 1 ) δ ( t − 1 ) ( δ ( t − 1 ) ) T ) . (43) \operatorname{Cov}(\mu^{(t)} \mid\mathcal{F}_{t-1}) = \beta_{t-1}\left(\alpha_{t-1}\Sigma^{(t-1)} + (1-\alpha_{t-1})\Sigma+ \alpha_{t-1}(1-\alpha_{t-1})\delta^{(t-1)}(\delta^{(t-1)})^{\mathsf{T}}\right). \tag*{(43)} Cov ( μ ( t ) ∣ F t − 1 ) = β t − 1 ( α t − 1 Σ ( t − 1 ) + ( 1 − α t − 1 ) Σ + α t − 1 ( 1 − α t − 1 ) δ ( t − 1 ) ( δ ( t − 1 ) ) T ) . (43)
Taking the outer expectation and using E [ Σ ( t − 1 ) ] = Σ − C t − 1 \mathbb{E}[\Sigma^{(t-1)}] = \Sigma- C_{t-1} E [ Σ ( t − 1 ) ] = Σ − C t − 1 (Theorem 3.1(b)) together with E [ δ ( t − 1 ) ( δ ( t − 1 ) ) T ] = C t − 1 \mathbb{E}[\delta^{(t-1)}(\delta^{(t-1)})^{\mathsf{T}}] = C_{t-1} E [ δ ( t − 1 ) ( δ ( t − 1 ) ) T ] = C t − 1 (δ ( t − 1 ) = μ ( t − 1 ) − μ \delta^{(t-1)} = \mu^{(t-1)}-\mu δ ( t − 1 ) = μ ( t − 1 ) − μ and (10)), from (44) we have
E [ Cov ( μ ( t ) ∣ F t − 1 ) ] = β t − 1 ( α t − 1 ( Σ − C t − 1 ) + ( 1 − α t − 1 ) Σ + α t − 1 ( 1 − α t − 1 ) C t − 1 ) = β t − 1 ( α t − 1 Σ − α t − 1 C t − 1 + ( 1 − α t − 1 ) Σ + α t − 1 C t − 1 − α t − 1 2 C t − 1 ) = β t − 1 ( Σ − α t − 1 2 C t − 1 ) . (44) \begin{aligned}
\mathbb{E}[\operatorname{Cov}(\mu^{(t)} \mid\mathcal{F}_{t-1})] &= \beta_{t-1}\left(\alpha_{t-1}(\Sigma-C_{t-1})+(1-\alpha_{t-1})\Sigma+\alpha_{t-1}(1-\alpha_{t-1})C_{t-1}\right) \\
&= \beta_{t-1}\left(\alpha_{t-1}\Sigma-\alpha_{t-1}C_{t-1}+(1-\alpha_{t-1})\Sigma+\alpha_{t-1}C_{t-1}-\alpha_{t-1}^{2}C_{t-1}\right) \\
&= \beta_{t-1}\left(\Sigma-\alpha_{t-1}^{2}C_{t-1}\right). \tag*{(44)}
\end{aligned} E [ Cov ( μ ( t ) ∣ F t − 1 )] = β t − 1 ( α t − 1 ( Σ − C t − 1 ) + ( 1 − α t − 1 ) Σ + α t − 1 ( 1 − α t − 1 ) C t − 1 ) = β t − 1 ( α t − 1 Σ − α t − 1 C t − 1 + ( 1 − α t − 1 ) Σ + α t − 1 C t − 1 − α t − 1 2 C t − 1 ) = β t − 1 ( Σ − α t − 1 2 C t − 1 ) . (44)
Combining. From (41), (43) and (45), the covariance C t C_t C t reads:
C t = λ t − 1 2 C t − 1 + β t − 1 ( Σ − α t − 1 2 C t − 1 ) = ( λ t − 1 2 − α t − 1 2 β t − 1 ) C t − 1 + β t − 1 Σ = γ t − 1 C t − 1 + β t − 1 Σ \begin{aligned}
C_t &= \lambda_{t-1}^{2}C_{t-1}+\beta_{t-1}\left(\Sigma-\alpha_{t-1}^{2}C_{t-1}\right) \\
&= \left(\lambda_{t-1}^{2}-\alpha_{t-1}^{2}\beta_{t-1}\right)C_{t-1}+\beta_{t-1}\Sigma\\
&= \gamma_{t-1}C_{t-1}+\beta_{t-1}\Sigma
\end{aligned} C t = λ t − 1 2 C t − 1 + β t − 1 ( Σ − α t − 1 2 C t − 1 ) = ( λ t − 1 2 − α t − 1 2 β t − 1 ) C t − 1 + β t − 1 Σ = γ t − 1 C t − 1 + β t − 1 Σ
where we used the definition of γ t \gamma_t γ t in (18). The initial condition is
C 1 = Cov ( 1 n r ∑ j = 1 n r x j ( 0 ) ) = 1 n r 2 n r Σ = Σ n r . (45) C_1=\operatorname{Cov}\left(\frac{1}{n_r}\sum_{j=1}^{n_r}x_j^{(0)}\right)=\frac{1}{n_r^2}n_r\Sigma=\frac{\Sigma}{n_r}. \tag*{(45)} C 1 = Cov ( n r 1 j = 1 ∑ n r x j ( 0 ) ) = n r 2 1 n r Σ = n r Σ . (45)
Proof of part (b): closed form. We prove (39) via induction on t ≥ 1 t\ge1 t ≥ 1 .
For t = 1 t=1 t = 1 , the right-hand side of (39) is Γ 1 , 0 C 1 = C 1 \Gamma_{1,0}C_1=C_1 Γ 1 , 0 C 1 = C 1 (empty sum; Γ 1 , 0 = 1 \Gamma_{1,0}=1 Γ 1 , 0 = 1 by (19)).
Assuming C t − 1 = Γ 1 , t − 2 C 1 + Σ ∑ s = 1 t − 2 β s Γ s + 1 , t − 2 C_{t-1}=\Gamma_{1,t-2}C_1+\Sigma\sum_{s=1}^{t-2}\beta_s\Gamma_{s+1,t-2} C t − 1 = Γ 1 , t − 2 C 1 + Σ ∑ s = 1 t − 2 β s Γ s + 1 , t − 2 and substituting into the recursion (38),
C t = γ t − 1 ( Γ 1 , t − 2 C 1 + Σ ∑ s = 1 t − 2 β s Γ s + 1 , t − 2 ) + β t − 1 Σ C_t=\gamma_{t-1}\left(\Gamma_{1,t-2}C_1+\Sigma\sum_{s=1}^{t-2}\beta_s\Gamma_{s+1,t-2}\right)+\beta_{t-1}\Sigma C t = γ t − 1 ( Γ 1 , t − 2 C 1 + Σ s = 1 ∑ t − 2 β s Γ s + 1 , t − 2 ) + β t − 1 Σ
= γ t − 1 Γ 1 , t − 2 C 1 + Σ ∑ s = 1 t − 2 β s γ t − 1 Γ s + 1 , t − 2 + β t − 1 Σ Γ t , t − 1 = Γ 1 , t − 1 C 1 + Σ ∑ s = 1 t − 1 β s Γ s + 1 , t − 1 , \begin{aligned}
&=\gamma_{t-1}\Gamma_{1,t-2}C_1+\Sigma\sum_{s=1}^{t-2}\beta_s\gamma_{t-1}\Gamma_{s+1,t-2}+\beta_{t-1}\Sigma\Gamma_{t,t-1} \\
&=\Gamma_{1,t-1}C_1+\Sigma\sum_{s=1}^{t-1}\beta_s\Gamma_{s+1,t-1},
\end{aligned} = γ t − 1 Γ 1 , t − 2 C 1 + Σ s = 1 ∑ t − 2 β s γ t − 1 Γ s + 1 , t − 2 + β t − 1 Σ Γ t , t − 1 = Γ 1 , t − 1 C 1 + Σ s = 1 ∑ t − 1 β s Γ s + 1 , t − 1 ,
using γ t − 1 Γ s , t − 2 = Γ s , t − 1 \gamma_{t-1}\Gamma_{s,t-2}=\Gamma_{s,t-1} γ t − 1 Γ s , t − 2 = Γ s , t − 1 and the empty product Γ t , t − 1 = 1 \Gamma_{t,t-1}=1 Γ t , t − 1 = 1 , so that β t − 1 Σ \beta_{t-1}\Sigma β t − 1 Σ becomes the s = t − 1 s=t-1 s = t − 1 term of the sum. We used the definition of Γ s , t \Gamma_{s,t} Γ s , t in (19) throughout.
Proof of part (c): α t ≡ 0 \alpha_t\equiv0 α t ≡ 0 . Assume α t ≡ 0 \alpha_t\equiv0 α t ≡ 0 . From (18) we have:
γ t = λ t 2 = N t 2 N t + 1 2 , β t = n b ( t ) N t + 1 2 . (46) \gamma_t=\lambda_t^2=\frac{N_t^2}{N_{t+1}^2}, \qquad\beta_t=\frac{n_b(t)}{N_{t+1}^2}. \tag*{(46)} γ t = λ t 2 = N t + 1 2 N t 2 , β t = N t + 1 2 n b ( t ) . (46)
We prove that C t = Σ / N t C_t=\Sigma/N_t C t = Σ/ N t by induction on t ≥ 1 t\ge1 t ≥ 1 :
C 1 = Σ n r = Σ N 1 . C_1 = \frac{\Sigma}{n_r} = \frac{\Sigma}{N_1}. C 1 = n r Σ = N 1 Σ .
C t = γ t − 1 C t − 1 + β t − 1 Σ = N t − 1 2 N t 2 ⋅ Σ N t − 1 + n b ( t − 1 ) N t 2 Σ = N t − 1 + n b ( t − 1 ) N t 2 Σ = Σ N t . \begin{aligned}
C_t &= \gamma_{t-1}C_{t-1} + \beta_{t-1}\Sigma\\
&= \frac{N_{t-1}^2}{N_t^2} \cdot\frac{\Sigma}{N_{t-1}} + \frac{n_b(t-1)}{N_t^2}\Sigma\\
&= \frac{N_{t-1} + n_b(t-1)}{N_t^2}\Sigma\\
&= \frac{\Sigma}{N_t}.
\end{aligned} C t = γ t − 1 C t − 1 + β t − 1 Σ = N t 2 N t − 1 2 ⋅ N t − 1 Σ + N t 2 n b ( t − 1 ) Σ = N t 2 N t − 1 + n b ( t − 1 ) Σ = N t Σ .
where we used the formulas for γ t \gamma_t γ t , β t \beta_t β t from (46) , and N t − 1 + n b ( t − 1 ) = N t N_{t-1} + n_b(t-1) = N_t N t − 1 + n b ( t − 1 ) = N t from (3) .
Proof of part (d): α t ≡ 1 \alpha_t \equiv1 α t ≡ 1 . Assume α t ≡ 1 \alpha_t \equiv1 α t ≡ 1 . From N t + n b ( t ) = N t + 1 N_t + n_b(t) = N_{t+1} N t + n b ( t ) = N t + 1 (follows from (3) ) we have
λ t = N t + n b ( t ) α t N t + 1 = N t + n b ( t ) N t + 1 = 1. \lambda_t = \frac{N_t + n_b(t)\alpha_t}{N_{t+1}} = \frac{N_t + n_b(t)}{N_{t+1}} = 1. λ t = N t + 1 N t + n b ( t ) α t = N t + 1 N t + n b ( t ) = 1.
Hence, γ t = λ t 2 − α t 2 β t = 1 − β t \gamma_t = \lambda_t^2 - \alpha_t^2\beta_t = 1-\beta_t γ t = λ t 2 − α t 2 β t = 1 − β t . The recursion (37) then reads C t = ( 1 − β t − 1 ) C t − 1 + β t − 1 Σ C_t = (1-\beta_{t-1})C_{t-1}+\beta_{t-1}\Sigma C t = ( 1 − β t − 1 ) C t − 1 + β t − 1 Σ .
Proof of part (e): scalar form. We prove by induction on t ≥ 1 t \ge1 t ≥ 1 . For t = 1 t = 1 t = 1 , C 1 = Σ / n r = c 1 Σ C_1 = \Sigma/n_r = c_1\Sigma C 1 = Σ/ n r = c 1 Σ by (45) . Assuming C t − 1 = c t − 1 Σ C_{t-1} = c_{t-1}\Sigma C t − 1 = c t − 1 Σ , the recursion (37) gives
C t = γ t − 1 c t − 1 Σ + β t − 1 Σ = c t Σ . C_t = \gamma_{t-1}c_{t-1}\Sigma+ \beta_{t-1}\Sigma= c_t\Sigma. C t = γ t − 1 c t − 1 Σ + β t − 1 Σ = c t Σ.
We can now prove Lemma 3.2 , stated in the main text.
Proof of Lemma 3.2. The lower bound C t ⪰ 0 C_t \succeq0 C t ⪰ 0 holds since C t C_t C t is a covariance matrix (6) .
For the upper bound, by Theorem A.11 (e) we have C t = c t Σ C_t = c_t\Sigma C t = c t Σ , with c t c_t c t given by (39) . First, c t ≥ 0 c_t \ge0 c t ≥ 0 for every t ≥ 1 t \ge1 t ≥ 1 by induction, since c 1 = 1 / n r > 0 c_1 = 1/n_r > 0 c 1 = 1/ n r > 0 , and γ t > 0 \gamma_t > 0 γ t > 0 and β t > 0 \beta_t > 0 β t > 0 for every t ≥ 1 t \ge1 t ≥ 1 (Lemma A.9 and (18) ). Hence, using γ t − 1 < 1 \gamma_{t-1} < 1 γ t − 1 < 1 (Lemma A.9 ),
c t = γ t − 1 c t − 1 + β t − 1 ≤ c t − 1 + β t − 1 , t ≥ 2. c_t = \gamma_{t-1}c_{t-1} + \beta_{t-1} \le c_{t-1} + \beta_{t-1}, \qquad t \ge2. c t = γ t − 1 c t − 1 + β t − 1 ≤ c t − 1 + β t − 1 , t ≥ 2.
Unrolling this inequality and applying Lemma A.5,
c t ≤ c 1 + ∑ s = 1 t − 1 β s ≤ 1 n r + 1 n r = 2 n r , t ≥ 1 , c_t \le c_1 + \sum_{s=1}^{t-1}\beta_s \le\frac{1}{n_r} + \frac{1}{n_r} = \frac{2}{n_r}, \qquad t \ge1, c t ≤ c 1 + s = 1 ∑ t − 1 β s ≤ n r 1 + n r 1 = n r 2 , t ≥ 1 ,
where for t = 1 t = 1 t = 1 the sum is empty. Since Σ ≻ 0 \Sigma\succ0 Σ ≻ 0 , we conclude
C t = c t Σ ⪯ 2 n r Σ . C_t = c_t\Sigma\preceq\frac{2}{n_r}\Sigma. C t = c t Σ ⪯ n r 2 Σ.
We need to characterize when the estimated mean converges to the truth. By Lemma A.10 , this is equivalent to characterizing when C t → t → ∞ 0 C_t \xrightarrow{t\to\infty} 0 C t t → ∞ 0 . The following theorem addresses this.
Theorem A.12 (General necessary and sufficient condition). Under Assumption 1 , with λ t \lambda_t λ t , β t \beta_t β t , γ t \gamma_t γ t as in (18) , Γ s , t \Gamma_{s,t} Γ s , t as in (19) , C t → 0 C_t \to0 C t → 0 as t → ∞ t \to\infty t → ∞ if and only if
Γ 1 , t − 1 → 0 and S t : = ∑ s = 1 t − 1 β s Γ s + 1 , t − 1 → 0. (47) \Gamma_{1,t-1} \to0 \qquad\text{and} \qquad S_t := \sum_{s=1}^{t-1} \beta_s \Gamma_{s+1,t-1} \to0. \tag*{(47)} Γ 1 , t − 1 → 0 and S t := s = 1 ∑ t − 1 β s Γ s + 1 , t − 1 → 0. (47)
Remark. From (37), it follows that Theorem A.12 provides a necessary and sufficient condition for E [ ∥ μ ( t ) − μ ∥ 2 2 ] → t → ∞ 0 \mathbb{E}[\lVert\mu^{(t)}-\mu\rVert_2^2] \xrightarrow{t\to\infty} 0 E [∥ μ ( t ) − μ ∥ 2 2 ] t → ∞ 0 .
Proof. By the closed form of Theorem A.11 (b), (38) , we have
C t = Γ 1 , t − 1 C 1 + S t Σ , S t = ∑ s = 1 t − 1 β s Γ s + 1 , t − 1 . (48) C_t = \Gamma_{1,t-1}C_1 + S_t\Sigma, \qquad S_t = \sum_{s=1}^{t-1}\beta_s\Gamma_{s+1,t-1}. \tag*{(48)} C t = Γ 1 , t − 1 C 1 + S t Σ , S t = s = 1 ∑ t − 1 β s Γ s + 1 , t − 1 . (48)
Sufficiency. Suppose Γ 1 , t − 1 → 0 \Gamma_{1,t-1} \to0 Γ 1 , t − 1 → 0 and S t → 0 S_t \to0 S t → 0 . By the triangle inequality and ∥ c A ∥ F = ∣ c ∣ ∥ A ∥ F \lVert cA\rVert_F = |c|\lVert A\rVert_F ∥ c A ∥ F = ∣ c ∣ ∥ A ∥ F for a scalar c c c ,
0 ≤ ∥ C t ∥ F ≤ Γ 1 , t − 1 ∥ C 1 ∥ F + S t ∥ Σ ∥ F → 0 , 0 \le\lVert C_t\rVert_F \le\Gamma_{1,t-1}\lVert C_1\rVert_F + S_t\lVert\Sigma\rVert_F \to0, 0 ≤ ∥ C t ∥ F ≤ Γ 1 , t − 1 ∥ C 1 ∥ F + S t ∥ Σ ∥ F → 0 ,
since ∥ C 1 ∥ F \lVert C_1\rVert_F ∥ C 1 ∥ F and ∥ Σ ∥ F \lVert\Sigma\rVert_F ∥ Σ ∥ F are finite constants. We also used that Γ 1 , t − 1 ≥ 0 \Gamma_{1,t-1} \ge0 Γ 1 , t − 1 ≥ 0 , S t ≥ 0 S_t \ge0 S t ≥ 0 . Hence C t → 0 C_t \to0 C t → 0 .
Necessity. Suppose C t → 0 C_t \to0 C t → 0 . By (10) , the matrix C t C_t C t is symmetric, and it is positive-semi-definite as a covariance matrix. Hence, Lemma A.4 (b) gives Tr ( C t ) → 0 \operatorname{Tr}(C_t) \to0 Tr ( C t ) → 0 . From (48) , we have
Tr ( C t ) = Γ 1 , t − 1 Tr ( C 1 ) + S t Tr ( Σ ) . \operatorname{Tr}(C_t) = \Gamma_{1,t-1}\operatorname{Tr}(C_1) + S_t\operatorname{Tr}(\Sigma). Tr ( C t ) = Γ 1 , t − 1 Tr ( C 1 ) + S t Tr ( Σ ) .
Both summands are nonnegative: Γ 1 , t − 1 ≥ 0 \Gamma_{1,t-1} \ge0 Γ 1 , t − 1 ≥ 0 and S t ≥ 0 S_t \ge0 S t ≥ 0 by (19) together with γ t ∈ ( 0 , 1 ) \gamma_t \in(0,1) γ t ∈ ( 0 , 1 ) and β t > 0 \beta_t > 0 β t > 0 (Lemma A.9 ), while Tr ( C 1 ) = Tr ( Σ ) / n r > 0 \operatorname{Tr}(C_1) = \operatorname{Tr}(\Sigma)/n_r > 0 Tr ( C 1 ) = Tr ( Σ ) / n r > 0 and Tr ( Σ ) > 0 \operatorname{Tr}(\Sigma) > 0 Tr ( Σ ) > 0 since Σ ≻ 0 \Sigma\succ0 Σ ≻ 0 . Discarding one summand at a time and dividing by these two positive constants gives
0 ≤ Γ 1 , t − 1 ≤ Tr ( C t ) Tr ( C 1 ) → 0 , 0 ≤ S t ≤ Tr ( C t ) Tr ( Σ ) → 0. 0 \le\Gamma_{1,t-1} \le\frac{\operatorname{Tr}(C_t)}{\operatorname{Tr}(C_1)} \to0, \qquad0 \le S_t \le\frac{\operatorname{Tr}(C_t)}{\operatorname{Tr}(\Sigma)} \to0. 0 ≤ Γ 1 , t − 1 ≤ Tr ( C 1 ) Tr ( C t ) → 0 , 0 ≤ S t ≤ Tr ( Σ ) Tr ( C t ) → 0.
Hence, Γ 1 , t − 1 → 0 \Gamma_{1,t-1} \to0 Γ 1 , t − 1 → 0 and S t → 0 S_t \to0 S t → 0 .
Next, we show that the condition for C t → t → ∞ 0 C_t \xrightarrow{t\to\infty} 0 C t t → ∞ 0 collapses to a single elementary series in { α t } t ≥ 1 \{\alpha_t\}_{t\ge1} { α t } t ≥ 1 , { n b ( t ) } t ≥ 1 \{n_b(t)\}_{t\ge1} { n b ( t ) } t ≥ 1 , and n r n_r n r .
Theorem A.13 (Sharp elementary criterion for consistency). Under Assumption 1 , with ρ t \rho_t ρ t as in (3) ,
C t → 0 ⟺ ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∑ t = 1 ∞ ( 1 − α t ) n b ( t ) N t + 1 = ∞ . (49) C_t \to0 \quad\Longleftrightarrow\quad\sum_{t=1}^{\infty}(1-\alpha_t)(1-\rho_t) = \sum_{t=1}^{\infty}(1-\alpha_t)\frac{n_b(t)}{N_{t+1}} = \infty. \tag*{(49)} C t → 0 ⟺ t = 1 ∑ ∞ ( 1 − α t ) ( 1 − ρ t ) = t = 1 ∑ ∞ ( 1 − α t ) N t + 1 n b ( t ) = ∞. (49) and equivalently for E [ ∥ μ ( t ) − μ ∥ 2 2 ] → t → ∞ 0 \mathbb{E}[\lVert\mu^{(t)}-\mu\rVert_2^2] \xrightarrow{t\to\infty} 0 E [∥ μ ( t ) − μ ∥ 2 2 ] t → ∞ 0 . In particular, S t → 0 S_t \to0 S t → 0 whenever Γ 1 , t − 1 → 0 \Gamma_{1,t-1} \to0 Γ 1 , t − 1 → 0 , so the second condition in (47) is redundant.
Proof. It is convenient to route the argument through two intermediate conditions. Write
(a) C t → 0 C_t \to0 C t → 0 , \qquad (b) Γ 1 , t − 1 → 0 \Gamma_{1,t-1} \to0 Γ 1 , t − 1 → 0 , \qquad (c) ∑ t = 1 ∞ ( 1 − γ t ) = ∞ \sum_{t=1}^{\infty}(1-\gamma_t)=\infty ∑ t = 1 ∞ ( 1 − γ t ) = ∞ , \qquad (d) ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ \sum_{t=1}^{\infty}(1-\alpha_t)(1-\rho_t)=\infty ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ ,
so that the assertion of the theorem is (a)⇔ \Leftrightarrow ⇔ (d). Theorem A.12 already gives (a)⇒ \Rightarrow ⇒ (b); we prove (b)⇒ \Rightarrow ⇒ (a), (b)⇔ \Leftrightarrow ⇔ (c) and (c)⇔ \Leftrightarrow ⇔ (d), which together yield (a)⇔ \Leftrightarrow ⇔ (d) as well as the equivalences among (b), (c) and (d). The final restatement in terms of E [ ∥ μ ( t ) − μ ∥ 2 2 ] → t → ∞ 0 \mathbb{E}[\lVert\mu^{(t)}-\mu\rVert_2^2] \xrightarrow{t\to\infty} 0 E [∥ μ ( t ) − μ ∥ 2 2 ] t → ∞ 0 is Lemma A.10 .
Step 1: (b)⇒ \Rightarrow ⇒ (a). We first prove that Γ 1 , t − 1 → 0 \Gamma_{1,t-1} \to0 Γ 1 , t − 1 → 0 implies $S_t \to 0.` Assume that Γ 1 , t − 1 → 0 \Gamma_{1,t-1} \to0 Γ 1 , t − 1 → 0 , and fix ε > 0 \varepsilon> 0 ε > 0 . By Lemma A.5 choose s 0 s_0 s 0 with ∑ s > s 0 β s < ε \sum_{s>s_0}\beta_s < \varepsilon ∑ s > s 0 β s < ε , and for t − 1 > s 0 t-1>s_0 t − 1 > s 0 split
S t = ∑ s = 1 t − 1 β s Γ s + 1 , t − 1 = ∑ s = 1 s 0 β s Γ s + 1 , t − 1 ⏟ A t + ∑ s = s 0 + 1 t − 1 β s Γ s + 1 , t − 1 ⏟ B t . S_t=\sum_{s=1}^{t-1}\beta_s\Gamma_{s+1,t-1}=\underbrace{\sum_{s=1}^{s_0}\beta_s\Gamma_{s+1,t-1}}_{A_t}+\underbrace{\sum_{s=s_0+1}^{t-1}\beta_s\Gamma_{s+1,t-1}}_{B_t}. S t = s = 1 ∑ t − 1 β s Γ s + 1 , t − 1 = A t s = 1 ∑ s 0 β s Γ s + 1 , t − 1 + B t s = s 0 + 1 ∑ t − 1 β s Γ s + 1 , t − 1 .
By Lemma A.9 every γ k ∈ ( 0 , 1 ) \gamma_k \in(0,1) γ k ∈ ( 0 , 1 ) , so each Γ s + 1 , t − 1 = ∏ k = s + 1 t − 1 γ k ≤ 1 \Gamma_{s+1,t-1}=\prod_{k=s+1}^{t-1}\gamma_k\leq1 Γ s + 1 , t − 1 = ∏ k = s + 1 t − 1 γ k ≤ 1 and B t ≤ ∑ s > s 0 β s < ε B_t\leq\sum_{s>s_0}\beta_s<\varepsilon B t ≤ ∑ s > s 0 β s < ε .
In addition, for s ≤ s 0 s\leq s_0 s ≤ s 0 , from the definition of Γ s , t \Gamma_{s,t} Γ s , t in (19) :
Γ s + 1 , t − 1 = Γ s 0 + 1 , t − 1 ⋅ Γ s + 1 , s 0 ≤ Γ s 0 + 1 , t − 1 \Gamma_{s+1,t-1}=\Gamma_{s_0+1,t-1}\cdot\Gamma_{s+1,s_0}\leq\Gamma_{s_0+1,t-1} Γ s + 1 , t − 1 = Γ s 0 + 1 , t − 1 ⋅ Γ s + 1 , s 0 ≤ Γ s 0 + 1 , t − 1
Thus,
0 ≤ A t ≤ Γ s 0 + 1 , t − 1 ⋅ ∑ s = 1 s 0 β s = ∑ s = 1 s 0 β s Γ 1 , s 0 ⋅ Γ 1 , t − 1 → 0 0\leq A_t\leq\Gamma_{s_0+1,t-1}\cdot\sum_{s=1}^{s_0}\beta_s=\frac{\sum_{s=1}^{s_0}\beta_s}{\Gamma_{1,s_0}}\cdot\Gamma_{1,t-1}\to0 0 ≤ A t ≤ Γ s 0 + 1 , t − 1 ⋅ s = 1 ∑ s 0 β s = Γ 1 , s 0 ∑ s = 1 s 0 β s ⋅ Γ 1 , t − 1 → 0
That is, A t → 0 A_t\to0 A t → 0 . Therefore, lim sup t S t ≤ ε \limsup_t S_t\leq\varepsilon lim sup t S t ≤ ε , and as ε > 0 \varepsilon>0 ε > 0 was arbitrary and S t ≥ 0 S_t\geq0 S t ≥ 0 we get S t → 0 S_t\to0 S t → 0 . Both conditions of (47) now hold, so Theorem A.12 gives C t → 0 C_t\to0 C t → 0 . This also proves the final assertion of the theorem.
Step 2: ( b ) ⇔ ( c ) (b)\Leftrightarrow(c) ( b ) ⇔ ( c ) . Assume (c). Since γ t ∈ ( 0 , 1 ) \gamma_t\in(0,1) γ t ∈ ( 0 , 1 ) , the bound ln u ≤ u − 1 \ln u\leq u-1 ln u ≤ u − 1 gives
Γ 1 , t − 1 = ∏ s = 1 t − 1 γ s = exp ( ∑ s = 1 t − 1 ln γ s ) ≤ exp ( − ∑ s = 1 t − 1 ( 1 − γ s ) ) → t → ∞ 0 , \Gamma_{1,t-1}=\prod_{s=1}^{t-1}\gamma_s=\exp\left(\sum_{s=1}^{t-1}\ln\gamma_s\right)\leq\exp\left(-\sum_{s=1}^{t-1}(1-\gamma_s)\right)\xrightarrow{t\to\infty}0, Γ 1 , t − 1 = s = 1 ∏ t − 1 γ s = exp ( s = 1 ∑ t − 1 ln γ s ) ≤ exp ( − s = 1 ∑ t − 1 ( 1 − γ s ) ) t → ∞ 0 ,
which, together with Γ 1 , t − 1 ≥ 0 \Gamma_{1,t-1}\geq0 Γ 1 , t − 1 ≥ 0 , is (b). Conversely, (b) together with γ t ∈ ( 0 , 1 ) \gamma_t\in(0,1) γ t ∈ ( 0 , 1 ) is precisely the hypothesis of Lemma A.6, whose conclusion is (c).
Step 3: ( c ) ⇔ ( d ) (c)\Leftrightarrow(d) ( c ) ⇔ ( d ) . Remember that from (3) and (18), we have N t + 1 = N t + n b ( t ) N_{t+1}=N_t+n_b(t) N t + 1 = N t + n b ( t ) and β t = n b ( t ) N t + 1 2 \beta_t=\frac{n_b(t)}{N_{t+1}^2} β t = N t + 1 2 n b ( t ) .
Using the identity (36), we have
1 − γ t = β t [ ( 1 − α t ) K t + α t 2 ] , K t : = 2 N t + n b ( t ) ( 1 + α t ) . (50) 1-\gamma_t=\beta_t\left[(1-\alpha_t)K_t+\alpha_t^2\right],\qquad K_t:=2N_t+n_b(t)(1+\alpha_t). \tag*{(50)} 1 − γ t = β t [ ( 1 − α t ) K t + α t 2 ] , K t := 2 N t + n b ( t ) ( 1 + α t ) . (50)
Because 0 ≤ α t 2 ≤ 1 0\leq\alpha_t^2\leq1 0 ≤ α t 2 ≤ 1 , Lemma A.5 yields
∑ t = 1 ∞ β t α t 2 ≤ ∑ t = 1 ∞ β t ≤ 1 n r < ∞ . \sum_{t=1}^{\infty}\beta_t\alpha_t^2\leq\sum_{t=1}^{\infty}\beta_t\leq\frac{1}{n_r}<\infty. t = 1 ∑ ∞ β t α t 2 ≤ t = 1 ∑ ∞ β t ≤ n r 1 < ∞.
Thus, the second term in (51) is summable and can never cause divergence. Hence
∑ t = 1 ∞ ( 1 − γ t ) = ∞ ⟺ ∑ t = 1 ∞ β t ( 1 − α t ) K t = ∞ . (51) \sum_{t=1}^{\infty}(1-\gamma_t)=\infty\Longleftrightarrow\sum_{t=1}^{\infty}\beta_t(1-\alpha_t)K_t=\infty. \tag*{(51)} t = 1 ∑ ∞ ( 1 − γ t ) = ∞ ⟺ t = 1 ∑ ∞ β t ( 1 − α t ) K t = ∞. (51)
We bound K t K_t K t on both sides by a multiple of N t + 1 N_{t+1} N t + 1 . Since N t ≥ 0 N_t\geq0 N t ≥ 0 and α t ≥ 0 \alpha_t\geq0 α t ≥ 0 ,
K t = 2 N t + n b ( t ) ( 1 + α t ) ≥ N t + n b ( t ) = N t + 1 , K_t=2N_t+n_b(t)(1+\alpha_t)\geq N_t+n_b(t)=N_{t+1}, K t = 2 N t + n b ( t ) ( 1 + α t ) ≥ N t + n b ( t ) = N t + 1 ,
and since α t ≤ 1 \alpha_t\leq1 α t ≤ 1 ,
K t ≤ 2 N t + 2 n b ( t ) = 2 N t + 1 . K_t\leq2N_t+2n_b(t)=2N_{t+1}. K t ≤ 2 N t + 2 n b ( t ) = 2 N t + 1 .
That is,
( 1 − α t ) β t N t + 1 ≤ β t ( 1 − α t ) K t ≤ 2 ( 1 − α t ) β t N t + 1 . (1-\alpha_t)\beta_tN_{t+1}\leq\beta_t(1-\alpha_t)K_t\leq2(1-\alpha_t)\beta_tN_{t+1}. ( 1 − α t ) β t N t + 1 ≤ β t ( 1 − α t ) K t ≤ 2 ( 1 − α t ) β t N t + 1 .
By (3), we have β t N t + 1 = n b ( t ) N t + 1 = 1 − N t N t + 1 = 1 − ρ t \beta_tN_{t+1}=\frac{n_b(t)}{N_{t+1}}=1-\frac{N_t}{N_{t+1}}=1-\rho_t β t N t + 1 = N t + 1 n b ( t ) = 1 − N t + 1 N t = 1 − ρ t . Thus:
( 1 − α t ) ( 1 − ρ t ) ≤ β t ( 1 − α t ) K t ≤ 2 ( 1 − α t ) ( 1 − ρ t ) . (1-\alpha_t)(1-\rho_t)\leq\beta_t(1-\alpha_t)K_t\leq2(1-\alpha_t)(1-\rho_t). ( 1 − α t ) ( 1 − ρ t ) ≤ β t ( 1 − α t ) K t ≤ 2 ( 1 − α t ) ( 1 − ρ t ) .
The two series therefore diverge together, which with (52) gives ( c ) ⇔ ( d ) (c)\Leftrightarrow(d) ( c ) ⇔ ( d ) .
Consistency of Σ ( t ) \Sigma^{(t)} Σ ( t )
Next, we characterize when E [ ∥ Σ ( t ) − Σ ∥ F 2 ] → t → ∞ 0 \mathbb{E}\left[\left\|\Sigma^{(t)}-\Sigma\right\|_{F}^{2}\right]\xrightarrow{t\to\infty}0 E [ Σ ( t ) − Σ F 2 ] t → ∞ 0 . We start with eight technical lemmas. First, we denote the Gaussian fourth-moment constant
C Σ : = Tr ( Σ ) 2 + 2 ∥ Σ ∥ F 2 . (52) C_{\Sigma}:=\operatorname{Tr}(\Sigma)^{2}+2\|\Sigma\|_{F}^{2}. \tag*{(52)} C Σ := Tr ( Σ ) 2 + 2∥Σ ∥ F 2 . (52)
It equals E ∥ g ∥ 2 4 \mathbb{E}\|g\|_{2}^{4} E ∥ g ∥ 2 4 for g ∼ N ( 0 , Σ ) g\sim\mathcal{N}(0,\Sigma) g ∼ N ( 0 , Σ ) : by Isserlis’ theorem E [ g i 2 g j 2 ] = Σ i i Σ j j + 2 Σ i j 2 \mathbb{E}[g_i^{2}g_j^{2}]=\Sigma_{ii}\Sigma_{jj}+2\Sigma_{ij}^{2} E [ g i 2 g j 2 ] = Σ ii Σ j j + 2 Σ ij 2 , so E ∥ g ∥ 2 4 = ∑ i , j ( Σ i i Σ j j + 2 Σ i j 2 ) = Tr ( Σ ) 2 + 2 ∥ Σ ∥ F 2 \mathbb{E}\|g\|_{2}^{4}=\sum_{i,j}(\Sigma_{ii}\Sigma_{jj}+2\Sigma_{ij}^{2})=\operatorname{Tr}(\Sigma)^{2}+2\|\Sigma\|_{F}^{2} E ∥ g ∥ 2 4 = ∑ i , j ( Σ ii Σ j j + 2 Σ ij 2 ) = Tr ( Σ ) 2 + 2∥Σ ∥ F 2 .
First, we isolate the dependence of the covariance estimator on the mean error.
Lemma A.14 (Real-centered second-moment statistic). Define
S ^ t : = 1 N t ( ∑ j = 1 n r ( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) T + ∑ s = 1 t − 1 ∑ i = 1 n b ( s ) ( x i ( s ) − μ ) ( x i ( s ) − μ ) T ) . (53) \widehat{S}_{t}:=\frac{1}{N_{t}}\left(\sum_{j=1}^{n_{r}}(x_{j}^{(0)}-\mu)(x_{j}^{(0)}-\mu)^{\mathsf{T}}+\sum_{s=1}^{t-1}\sum_{i=1}^{n_{b}(s)}(x_{i}^{(s)}-\mu)(x_{i}^{(s)}-\mu)^{\mathsf{T}}\right). \tag*{(53)} S t := N t 1 j = 1 ∑ n r ( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) T + s = 1 ∑ t − 1 i = 1 ∑ n b ( s ) ( x i ( s ) − μ ) ( x i ( s ) − μ ) T . (53) Then, using (7) (δ ( t ) = μ ( t ) − μ \delta^{(t)}=\mu^{(t)}-\mu δ ( t ) = μ ( t ) − μ ):
(i) S ^ t ⪰ 0 \widehat{S}_{t}\succeq0 S t ⪰ 0 almost surely;
(ii) Σ ( t ) = S ^ t − δ ( t ) ( δ ( t ) ) T \Sigma^{(t)}=\widehat{S}_{t}-\delta^{(t)}(\delta^{(t)})^{\mathsf{T}} Σ ( t ) = S t − δ ( t ) ( δ ( t ) ) T . Thus, Tr ( Σ ( t ) ) = Tr ( S ^ t ) − ∥ δ ( t ) ∥ 2 2 \operatorname{Tr}(\Sigma^{(t)})=\operatorname{Tr}(\widehat{S}_{t})-\|\delta^{(t)}\|_{2}^{2} Tr ( Σ ( t ) ) = Tr ( S t ) − ∥ δ ( t ) ∥ 2 2 ;
(iii) for t ≥ 2 t\geq2 t ≥ 2 , S ^ t = ρ t − 1 S ^ t − 1 + 1 N t ∑ i = 1 n b ( t − 1 ) ( x i ( t − 1 ) − μ ) ( x i ( t − 1 ) − μ ) T \widehat{S}_{t}=\rho_{t-1}\widehat{S}_{t-1}+\frac{1}{N_{t}}\sum_{i=1}^{n_{b}(t-1)}(x_{i}^{(t-1)}-\mu)(x_{i}^{(t-1)}-\mu)^{\mathsf{T}} S t = ρ t − 1 S t − 1 + N t 1 ∑ i = 1 n b ( t − 1 ) ( x i ( t − 1 ) − μ ) ( x i ( t − 1 ) − μ ) T ;
(iv) for t ≥ 2 t\geq2 t ≥ 2 , E [ S ^ t ∣ F t − 1 ] = λ t − 1 S ^ t − 1 + ( 1 − λ t − 1 ) Σ \mathbb{E}[\widehat{S}_{t}\mid\mathcal{F}_{t-1}]=\lambda_{t-1}\widehat{S}_{t-1}+(1-\lambda_{t-1})\Sigma E [ S t ∣ F t − 1 ] = λ t − 1 S t − 1 + ( 1 − λ t − 1 ) Σ , and consequently E [ S ^ t ] = Σ \mathbb{E}[\widehat{S}_{t}]=\Sigma E [ S t ] = Σ for all t ≥ 1 t\geq1 t ≥ 1 .
Proof. (i) S ^ t \widehat{S}_{t} S t is a non-negatively weighted sum of outer products. Hence, it is symmetric PSD.
(ii) Using the pooled-scatter identity (25) and recalling (7) (δ ( t ) = μ ( t ) − μ \delta^{(t)}=\mu^{(t)}-\mu δ ( t ) = μ ( t ) − μ ) gives Σ ( t ) = S ^ t − δ ( t ) ( δ ( t ) ) T \Sigma^{(t)}=\widehat{S}_{t}-\delta^{(t)}(\delta^{(t)})^{\mathsf{T}} Σ ( t ) = S t − δ ( t ) ( δ ( t ) ) T ; the trace identity follows from Tr ( δ ( t ) ( δ ( t ) ) T ) = ∥ δ ( t ) ∥ 2 2 \operatorname{Tr}(\delta^{(t)}(\delta^{(t)})^{\mathsf{T}})=\|\delta^{(t)}\|_{2}^{2} Tr ( δ ( t ) ( δ ( t ) ) T ) = ∥ δ ( t ) ∥ 2 2 .
(iii) Fix t ≥ 2 t\geq2 t ≥ 2 . From (54), we have
S ^ t = 1 N t ( ∑ j = 1 n r ( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) T + ∑ s = 1 t − 1 ∑ i = 1 n b ( s ) ( x i ( s ) − μ ) ( x i ( s ) − μ ) T ) = 1 N t ( ∑ j = 1 n r ( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) T + ∑ s = 1 t − 2 ∑ i = 1 n b ( s ) ( x i ( s ) − μ ) ( x i ( s ) − μ ) T + ∑ i = 1 n b ( t − 1 ) ( x i ( t − 1 ) − μ ) ( x i ( t − 1 ) − μ ) T ) = N t − 1 N t S ^ t − 1 + 1 N t ∑ i = 1 n b ( t − 1 ) ( x i ( t − 1 ) − μ ) ( x i ( t − 1 ) − μ ) T = ρ t − 1 S ^ t − 1 + 1 N t ∑ i = 1 n b ( t − 1 ) ( x i ( t − 1 ) − μ ) ( x i ( t − 1 ) − μ ) T \begin{aligned}
\widehat{S}_{t}
&=\frac{1}{N_{t}}\left(\sum_{j=1}^{n_{r}}(x_{j}^{(0)}-\mu)(x_{j}^{(0)}-\mu)^{\mathsf{T}}+\sum_{s=1}^{t-1}\sum_{i=1}^{n_{b}(s)}(x_{i}^{(s)}-\mu)(x_{i}^{(s)}-\mu)^{\mathsf{T}}\right)\\
&=\frac{1}{N_{t}}\left(\sum_{j=1}^{n_{r}}(x_{j}^{(0)}-\mu)(x_{j}^{(0)}-\mu)^{\mathsf{T}}+\sum_{s=1}^{t-2}\sum_{i=1}^{n_{b}(s)}(x_{i}^{(s)}-\mu)(x_{i}^{(s)}-\mu)^{\mathsf{T}}+\sum_{i=1}^{n_{b}(t-1)}(x_{i}^{(t-1)}-\mu)(x_{i}^{(t-1)}-\mu)^{\mathsf{T}}\right)\\
&=\frac{N_{t-1}}{N_{t}}\widehat{S}_{t-1}+\frac{1}{N_{t}}\sum_{i=1}^{n_{b}(t-1)}(x_{i}^{(t-1)}-\mu)(x_{i}^{(t-1)}-\mu)^{\mathsf{T}}\\
&=\rho_{t-1}\widehat{S}_{t-1}+\frac{1}{N_{t}}\sum_{i=1}^{n_{b}(t-1)}(x_{i}^{(t-1)}-\mu)(x_{i}^{(t-1)}-\mu)^{\mathsf{T}}
\end{aligned} S t = N t 1 j = 1 ∑ n r ( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) T + s = 1 ∑ t − 1 i = 1 ∑ n b ( s ) ( x i ( s ) − μ ) ( x i ( s ) − μ ) T = N t 1 j = 1 ∑ n r ( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) T + s = 1 ∑ t − 2 i = 1 ∑ n b ( s ) ( x i ( s ) − μ ) ( x i ( s ) − μ ) T + i = 1 ∑ n b ( t − 1 ) ( x i ( t − 1 ) − μ ) ( x i ( t − 1 ) − μ ) T = N t N t − 1 S t − 1 + N t 1 i = 1 ∑ n b ( t − 1 ) ( x i ( t − 1 ) − μ ) ( x i ( t − 1 ) − μ ) T = ρ t − 1 S t − 1 + N t 1 i = 1 ∑ n b ( t − 1 ) ( x i ( t − 1 ) − μ ) ( x i ( t − 1 ) − μ ) T
We used the definition of ρ t \rho_{t} ρ t from (3). (iv) From (iii), we have
E [ S ^ t ∣ F t − 1 ] = ρ t − 1 E [ S ^ t − 1 ∣ F t − 1 ] + 1 N t ∑ i = 1 n b ( t − 1 ) E [ ( x i ( t − 1 ) − μ ) ( x i ( t − 1 ) − μ ) ⊤ ∣ F t − 1 ] = ρ t − 1 S ^ t − 1 + 1 N t ∑ i = 1 n b ( t − 1 ) ( α t − 1 Σ ( t − 1 ) + ( 1 − α t − 1 ) Σ + α t − 1 δ ( t − 1 ) ( δ ( t − 1 ) ) ⊤ ) = N t − 1 N t S ^ t − 1 + 1 N t ∑ i = 1 n b ( t − 1 ) ( α t − 1 S ^ t − 1 + ( 1 − α t − 1 ) Σ ) = N t − 1 + α t − 1 n b ( t − 1 ) N t S ^ t − 1 + ( 1 − α t − 1 ) n b ( t − 1 ) N t Σ = λ t − 1 S ^ t − 1 + ( 1 − λ t − 1 ) Σ . \begin{aligned}
\mathbb{E}[\widehat{S}_t \mid\mathcal{F}_{t-1}] = \rho_{t-1}\mathbb{E}[\widehat{S}_{t-1} \mid\mathcal{F}_{t-1}] + \frac{1}{N_t}\sum_{i=1}^{n_b(t-1)} \mathbb{E}\left[(x_i^{(t-1)}-\mu)(x_i^{(t-1)}-\mu)^\top\mid\mathcal{F}_{t-1}\right] \\
&= \rho_{t-1}\widehat{S}_{t-1}+\frac{1}{N_t}\sum_{i=1}^{n_b(t-1)}\left(\alpha_{t-1}\Sigma^{(t-1)}+(1-\alpha_{t-1})\Sigma+\alpha_{t-1}\delta^{(t-1)}(\delta^{(t-1)})^\top\right) \\
&= \frac{N_{t-1}}{N_t}\widehat{S}_{t-1}+\frac{1}{N_t}\sum_{i=1}^{n_b(t-1)}\left(\alpha_{t-1}\widehat{S}_{t-1}+(1-\alpha_{t-1})\Sigma\right) \\
&= \frac{N_{t-1}+\alpha_{t-1}n_b(t-1)}{N_t}\widehat{S}_{t-1}+\frac{(1-\alpha_{t-1})n_b(t-1)}{N_t}\Sigma\\
&= \lambda_{t-1}\widehat{S}_{t-1}+(1-\lambda_{t-1})\Sigma.
\end{aligned} E [ S t ∣ F t − 1 ] = ρ t − 1 E [ S t − 1 ∣ F t − 1 ] + N t 1 i = 1 ∑ n b ( t − 1 ) E [ ( x i ( t − 1 ) − μ ) ( x i ( t − 1 ) − μ ) ⊤ ∣ F t − 1 ] = ρ t − 1 S t − 1 + N t 1 i = 1 ∑ n b ( t − 1 ) ( α t − 1 Σ ( t − 1 ) + ( 1 − α t − 1 ) Σ + α t − 1 δ ( t − 1 ) ( δ ( t − 1 ) ) ⊤ ) = N t N t − 1 S t − 1 + N t 1 i = 1 ∑ n b ( t − 1 ) ( α t − 1 S t − 1 + ( 1 − α t − 1 ) Σ ) = N t N t − 1 + α t − 1 n b ( t − 1 ) S t − 1 + N t ( 1 − α t − 1 ) n b ( t − 1 ) Σ = λ t − 1 S t − 1 + ( 1 − λ t − 1 ) Σ.
We used
S ^ t − 1 \widehat{S}_{t-1} S t − 1 is F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable.
ρ t − 1 = N t − 1 / N t \rho_{t-1}=N_{t-1}/N_t ρ t − 1 = N t − 1 / N t from (3).
From (23) (Lemma A.1), we have
E [ ( x i ( t − 1 ) − μ ) ( x i ( t − 1 ) − μ ) ⊤ ∣ F t − 1 ] = α t − 1 Σ ( t − 1 ) + ( 1 − α t − 1 ) Σ + α t − 1 δ ( t − 1 ) ( δ ( t − 1 ) ) ⊤ \mathbb{E}\left[(x_i^{(t-1)}-\mu)(x_i^{(t-1)}-\mu)^\top\mid\mathcal{F}_{t-1}\right]=\alpha_{t-1}\Sigma^{(t-1)}+(1-\alpha_{t-1})\Sigma+\alpha_{t-1}\delta^{(t-1)}(\delta^{(t-1)})^\top E [ ( x i ( t − 1 ) − μ ) ( x i ( t − 1 ) − μ ) ⊤ ∣ F t − 1 ] = α t − 1 Σ ( t − 1 ) + ( 1 − α t − 1 ) Σ + α t − 1 δ ( t − 1 ) ( δ ( t − 1 ) ) ⊤
Σ ( t − 1 ) + δ ( t − 1 ) ( δ ( t − 1 ) ) ⊤ = S ^ t − 1 \Sigma^{(t-1)}+\delta^{(t-1)}(\delta^{(t-1)})^\top=\widehat{S}_{t-1} Σ ( t − 1 ) + δ ( t − 1 ) ( δ ( t − 1 ) ) ⊤ = S t − 1 from (ii) at index t − 1 t-1 t − 1 .
The definition of λ t \lambda_t λ t from (18), and
1 − λ t = 1 − N t + n b ( t ) α t N t + 1 = N t + 1 − N t − n b ( t ) α t N t + 1 = ( 1 − α t ) n b ( t ) N t + 1 . 1-\lambda_t=1-\frac{N_t+n_b(t)\alpha_t}{N_{t+1}}=\frac{N_{t+1}-N_t-n_b(t)\alpha_t}{N_{t+1}}=\frac{(1-\alpha_t)n_b(t)}{N_{t+1}}. 1 − λ t = 1 − N t + 1 N t + n b ( t ) α t = N t + 1 N t + 1 − N t − n b ( t ) α t = N t + 1 ( 1 − α t ) n b ( t ) .
That is, we proved that
E [ S ^ t ∣ F t − 1 ] = λ t − 1 S ^ t − 1 + ( 1 − λ t − 1 ) Σ . (54) \mathbb{E}[\widehat{S}_t \mid\mathcal{F}_{t-1}]=\lambda_{t-1}\widehat{S}_{t-1}+(1-\lambda_{t-1})\Sigma. \tag*{(54)} E [ S t ∣ F t − 1 ] = λ t − 1 S t − 1 + ( 1 − λ t − 1 ) Σ. (54)
Finally, we prove unbiasedness by induction on t ≥ 1 t\ge1 t ≥ 1 . At t = 1 t=1 t = 1 , we have
S ^ 1 = 1 n r ∑ j = 1 n r ( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) ⊤ ⇒ E [ S ^ 1 ] = 1 n r ∑ j = 1 n r E [ ( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) ⊤ ] = Σ \widehat{S}_1=\frac{1}{n_r}\sum_{j=1}^{n_r}(x_j^{(0)}-\mu)(x_j^{(0)}-\mu)^\top\Rightarrow\mathbb{E}[\widehat{S}_1]=\frac{1}{n_r}\sum_{j=1}^{n_r}\mathbb{E}\left[(x_j^{(0)}-\mu)(x_j^{(0)}-\mu)^\top\right]=\Sigma S 1 = n r 1 j = 1 ∑ n r ( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) ⊤ ⇒ E [ S 1 ] = n r 1 j = 1 ∑ n r E [ ( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) ⊤ ] = Σ
Assume that E [ S ^ t − 1 ] = Σ \mathbb{E}[\widehat{S}_{t-1}]=\Sigma E [ S t − 1 ] = Σ . From (55) and the tower property, we have
E [ S ^ t ] = E [ E [ S ^ t ∣ F t − 1 ] ] = E [ λ t − 1 S ^ t − 1 + ( 1 − λ t − 1 ) Σ ] = λ t − 1 E [ S ^ t − 1 ] + ( 1 − λ t − 1 ) Σ = Σ . \begin{aligned}
\mathbb{E}[\widehat{S}_t]=\mathbb{E}\left[\mathbb{E}[\widehat{S}_t\mid\mathcal{F}_{t-1}]\right] \\
&=\mathbb{E}\left[\lambda_{t-1}\widehat{S}_{t-1}+(1-\lambda_{t-1})\Sigma\right] \\
&=\lambda_{t-1}\mathbb{E}[\widehat{S}_{t-1}]+(1-\lambda_{t-1})\Sigma\\
&=\Sigma.
\end{aligned} E [ S t ] = E [ E [ S t ∣ F t − 1 ] ] = E [ λ t − 1 S t − 1 + ( 1 − λ t − 1 ) Σ ] = λ t − 1 E [ S t − 1 ] + ( 1 − λ t − 1 ) Σ = Σ.
We now state a technical lemma that will be helpful later on.
Lemma A.15 (Rayleigh bound). Let A ∈ R d × d A\in\mathbb{R}^{d\times d} A ∈ R d × d be symmetric with A ⪰ 0 A\succeq0 A ⪰ 0 . Then for every x ∈ R d x\in\mathbb{R}^d x ∈ R d ,
x ⊤ A x ≤ Tr ( A ) ∥ x ∥ 2 2 x^\top A x\le\operatorname{Tr}(A)\lVert x\rVert_2^2 x ⊤ A x ≤ Tr ( A ) ∥ x ∥ 2 2
Proof. By the spectral theorem, A = Q D Q T A = QDQ^{\mathsf{T}} A = Q D Q T with Q Q Q orthogonal and D = diag ( ν 1 , … , ν d ) D = \operatorname{diag}(\nu_1,\ldots,\nu_d) D = diag ( ν 1 , … , ν d ) , where ν 1 , … , ν d \nu_1,\ldots,\nu_d ν 1 , … , ν d are the eigenvalues of A A A : real because A A A is symmetric, and nonnegative because A ⪰ 0 A \succeq0 A ⪰ 0 , since ν i = q i T A q i ≥ 0 \nu_i = q_i^{\mathsf{T}}Aq_i \ge0 ν i = q i T A q i ≥ 0 for the corresponding unit eigenvector q i q_i q i .
Fix x ∈ R d x \in\mathbb{R}^d x ∈ R d and set y : = Q T x y := Q^{\mathsf{T}}x y := Q T x . Since Q Q Q is orthogonal, ∥ y ∥ 2 = ∥ x ∥ 2 \lVert y\rVert_2 = \lVert x\rVert_2 ∥ y ∥ 2 = ∥ x ∥ 2 , and
x T A x = x T Q D Q T x = y T D y = ∑ i = 1 d ν i y i 2 . x^{\mathsf{T}}Ax = x^{\mathsf{T}}QDQ^{\mathsf{T}}x = y^{\mathsf{T}}Dy = \sum_{i=1}^{d} \nu_i y_i^2. x T A x = x T Q D Q T x = y T D y = i = 1 ∑ d ν i y i 2 .
Every y i 2 ≥ 0 y_i^2 \ge0 y i 2 ≥ 0 , so bounding each ν i \nu_i ν i by ν max : = max 1 ≤ j ≤ d ν j \nu_{\max} := \max_{1 \le j \le d} \nu_j ν m a x := max 1 ≤ j ≤ d ν j gives
∑ i = 1 d ν i y i 2 ≤ ν max ∑ i = 1 d y i 2 = ν max ∥ y ∥ 2 2 = ν max ∥ x ∥ 2 2 . \sum_{i=1}^{d} \nu_i y_i^2 \le\nu_{\max}\sum_{i=1}^{d} y_i^2 = \nu_{\max}\lVert y\rVert_2^2 = \nu_{\max}\lVert x\rVert_2^2. i = 1 ∑ d ν i y i 2 ≤ ν m a x i = 1 ∑ d y i 2 = ν m a x ∥ y ∥ 2 2 = ν m a x ∥ x ∥ 2 2 .
Finally, all eigenvalues are nonnegative, so discarding the remaining d − 1 d-1 d − 1 of them from the trace yields
ν max ≤ ∑ i = 1 d ν i = Tr ( A ) , \nu_{\max} \le\sum_{i=1}^{d} \nu_i = \operatorname{Tr}(A), ν m a x ≤ i = 1 ∑ d ν i = Tr ( A ) ,
and combining the last two displays gives the claim. □
Lemma A.16 (Conditional fourth-moment bound). For every t ≥ 1 t \ge1 t ≥ 1 and each newly drawn sample x i ( t ) x_i^{(t)} x i ( t ) ,
E [ ∥ x i ( t ) − μ ∥ 2 4 ∣ F t ] ≤ 3 α t Tr ( S ^ t ) 2 + C Σ , \mathbb{E}\left[\lVert x_i^{(t)}-\mu\rVert_2^4 \mid\mathcal{F}_t\right] \le3\alpha_t \operatorname{Tr}(\widehat{S}_t)^2 + C_{\Sigma}, E [ ∥ x i ( t ) − μ ∥ 2 4 ∣ F t ] ≤ 3 α t Tr ( S t ) 2 + C Σ , while for the real batch E [ ∥ x j ( 0 ) − μ ∥ 2 4 ] = C Σ \mathbb{E}\left[\lVert x_j^{(0)}-\mu\rVert_2^4\right] = C_{\Sigma} E [ ∥ x j ( 0 ) − μ ∥ 2 4 ] = C Σ .
Proof. Given F t \mathcal{F}_t F t , the sample x i ( t ) x_i^{(t)} x i ( t ) is drawn from the mixture (2) , so from the total expectation theorem
E [ ∥ x i ( t ) − μ ∥ 2 4 ∣ F t ] = α t E [ ∥ Y ( t ) − μ ∥ 2 4 ∣ F t ] + ( 1 − α t ) C Σ , (55) \mathbb{E}\left[\lVert x_i^{(t)}-\mu\rVert_2^4 \mid\mathcal{F}_t\right] = \alpha_t \mathbb{E}\left[\lVert Y^{(t)}-\mu\rVert_2^4 \mid\mathcal{F}_t\right] + (1-\alpha_t)C_{\Sigma}, \tag*{(55)} E [ ∥ x i ( t ) − μ ∥ 2 4 ∣ F t ] = α t E [ ∥ Y ( t ) − μ ∥ 2 4 ∣ F t ] + ( 1 − α t ) C Σ , (55) where Y ( t ) ∼ N ( μ ( t ) , Σ ( t ) ) Y^{(t)} \sim\mathcal{N}(\mu^{(t)},\Sigma^{(t)}) Y ( t ) ∼ N ( μ ( t ) , Σ ( t ) ) given F t \mathcal{F}_t F t , and the real component contributes E [ ∥ W − μ ∥ 2 4 ] = C Σ \mathbb{E}\left[\lVert W-\mu\rVert_2^4\right] = C_{\Sigma} E [ ∥ W − μ ∥ 2 4 ] = C Σ for W ∼ N ( μ , Σ ) W \sim\mathcal{N}(\mu,\Sigma) W ∼ N ( μ , Σ ) (by (52) ). It remains to bound the synthetic term.
From (7) , we have δ ( t ) : = μ ( t ) − μ \delta^{(t)} := \mu^{(t)}-\mu δ ( t ) := μ ( t ) − μ is F t \mathcal{F}_t F t -measurable. Thus, write Y ( t ) − μ = y ( t ) + δ ( t ) Y^{(t)}-\mu= y^{(t)}+\delta^{(t)} Y ( t ) − μ = y ( t ) + δ ( t ) with y ( t ) : = Y ( t ) − μ ( t ) ∼ N ( 0 , Σ ( t ) ) y^{(t)} := Y^{(t)}-\mu^{(t)} \sim\mathcal{N}(0,\Sigma^{(t)}) y ( t ) := Y ( t ) − μ ( t ) ∼ N ( 0 , Σ ( t ) ) given F t \mathcal{F}_t F t . Since y ( t ) y^{(t)} y ( t ) is conditionally symmetric about 0 0 0 , the odd terms in y ( t ) y^{(t)} y ( t ) vanish, and
E [ ∥ Y ( t ) − μ ∥ 2 4 ∣ F t ] = E [ ∥ y ( t ) + δ ( t ) ∥ 2 4 ∣ F t ] = E [ ∥ y ( t ) ∥ 2 4 ∣ F t ] + 4 E [ ( ( y ( t ) ) T δ ( t ) ) 2 ∣ F t ] + 2 ∥ δ ( t ) ∥ 2 2 E [ ∥ y ( t ) ∥ 2 2 ∣ F t ] + ∥ δ ( t ) ∥ 2 4 = E [ ∥ y ( t ) ∥ 2 4 ∣ F t ] + 4 E [ ( ( y ( t ) ) T δ ( t ) ) 2 ∣ F t ] + 2 p ⋅ E [ ∥ y ( t ) ∥ 2 2 ∣ F t ] + p 2 , (56) \begin{aligned}
\mathbb{E}\left[\lVert Y^{(t)}-\mu\rVert_2^4 \mid\mathcal{F}_t\right]
&= \mathbb{E}\left[\lVert y^{(t)}+\delta^{(t)}\rVert_2^4 \mid\mathcal{F}_t\right]\\
&= \mathbb{E}\left[\lVert y^{(t)}\rVert_2^4 \mid\mathcal{F}_t\right]
+4\mathbb{E}\left[\left((y^{(t)})^{\mathsf{T}}\delta^{(t)}\right)^2 \mid\mathcal{F}_t\right]
+2\lVert\delta^{(t)}\rVert_2^2\mathbb{E}\left[\lVert y^{(t)}\rVert_2^2 \mid\mathcal{F}_t\right]
+\lVert\delta^{(t)}\rVert_2^4\\
&= \mathbb{E}\left[\lVert y^{(t)}\rVert_2^4 \mid\mathcal{F}_t\right]
+4\mathbb{E}\left[\left((y^{(t)})^{\mathsf{T}}\delta^{(t)}\right)^2 \mid\mathcal{F}_t\right]
+2p\cdot\mathbb{E}\left[\lVert y^{(t)}\rVert_2^2 \mid\mathcal{F}_t\right]+p^2, \tag*{(56)}
\end{aligned} E [ ∥ Y ( t ) − μ ∥ 2 4 ∣ F t ] = E [ ∥ y ( t ) + δ ( t ) ∥ 2 4 ∣ F t ] = E [ ∥ y ( t ) ∥ 2 4 ∣ F t ] + 4 E [ ( ( y ( t ) ) T δ ( t ) ) 2 ∣ F t ] + 2 ∥ δ ( t ) ∥ 2 2 E [ ∥ y ( t ) ∥ 2 2 ∣ F t ] + ∥ δ ( t ) ∥ 2 4 = E [ ∥ y ( t ) ∥ 2 4 ∣ F t ] + 4 E [ ( ( y ( t ) ) T δ ( t ) ) 2 ∣ F t ] + 2 p ⋅ E [ ∥ y ( t ) ∥ 2 2 ∣ F t ] + p 2 , (56) where we denote u : = Tr ( S ^ t ) u := \operatorname{Tr}(\widehat{S}_t) u := Tr ( S t ) and p : = ∥ δ ( t ) ∥ 2 2 p := \lVert\delta^{(t)}\rVert_2^2 p := ∥ δ ( t ) ∥ 2 2 . In order to further calculate and bound the above expression, we will use y ( t ) ∣ F t ∼ N ( 0 , Σ ( t ) ) y^{(t)} \mid\mathcal{F}_t \sim\mathcal{N}(0,\Sigma^{(t)}) y ( t ) ∣ F t ∼ N ( 0 , Σ ( t ) ) , and use Tr ( Σ ( t ) ) = Tr ( S ^ t ) − ∥ δ ( t ) ∥ 2 2 = u − p \operatorname{Tr}(\Sigma^{(t)}) = \operatorname{Tr}(\widehat{S}_t)-\lVert\delta^{(t)}\rVert_2^2 = u-p Tr ( Σ ( t ) ) = Tr ( S t ) − ∥ δ ( t ) ∥ 2 2 = u − p (Lemma A.14(ii)).
First, conditionally Σ ( t ) \Sigma^{(t)} Σ ( t ) is fixed, so by (52) applied to Σ ( t ) \Sigma^{(t)} Σ ( t ) ,
E [ ∥ y ( t ) ∥ 2 4 ∣ F t ] = Tr ( Σ ( t ) ) 2 + 2 ∥ Σ ( t ) ∥ F 2 ≤ 3 Tr ( Σ ( t ) ) 2 = 3 ( u − p ) 2 , \mathbb{E}\left[\lVert y^{(t)}\rVert_2^4 \mid\mathcal{F}_t\right] = \operatorname{Tr}(\Sigma^{(t)})^2 + 2\lVert\Sigma^{(t)}\rVert_F^2 \le3\operatorname{Tr}(\Sigma^{(t)})^2 = 3(u-p)^2, E [ ∥ y ( t ) ∥ 2 4 ∣ F t ] = Tr ( Σ ( t ) ) 2 + 2 ∥ Σ ( t ) ∥ F 2 ≤ 3 Tr ( Σ ( t ) ) 2 = 3 ( u − p ) 2 , the inequality using ∥ A ∥ F 2 ≤ Tr ( A ) 2 \lVert A\rVert_F^2 \le\operatorname{Tr}(A)^2 ∥ A ∥ F 2 ≤ Tr ( A ) 2 for symmetric positive-semi-definite A A A (Lemma A.4 (a), applied to Σ ( t ) ⪰ 0 \Sigma^{(t)} \succeq0 Σ ( t ) ⪰ 0 by Lemma A.8).
Second, E [ ∥ y ( t ) ∥ 2 2 ∣ F t ] = Tr ( Σ ( t ) ) = u − p \mathbb{E}\left[\lVert y^{(t)}\rVert_2^2 \mid\mathcal{F}_t\right] = \operatorname{Tr}(\Sigma^{(t)}) = u-p E [ ∥ y ( t ) ∥ 2 2 ∣ F t ] = Tr ( Σ ( t ) ) = u − p .
Finally,
E [ ( ( y ( t ) ) T δ ( t ) ) 2 | F t ] = E [ ( δ ( t ) ) T y ( t ) ( y ( t ) ) T δ ( t ) | F t ] = ( δ ( t ) ) T E [ y ( t ) ( y ( t ) ) T | F t ] δ ( t ) = ( δ ( t ) ) T Σ ( t ) δ ( t ) = ( δ ( t ) ) T S ^ t δ ( t ) − ∥ δ ( t ) ∥ 2 4 ≤ Tr ( S ^ t ) ∥ δ ( t ) ∥ 2 2 − ∥ δ ( t ) ∥ 2 4 = u p − p 2 = p ( u − p ) . \begin{aligned}
\mathbb{E}\left[\left(\left(y^{(t)}\right)^{\mathsf{T}}\delta^{(t)}\right)^2 \middle| \mathcal{F}_t\right]
&= \mathbb{E}\left[\left(\delta^{(t)}\right)^{\mathsf{T}}y^{(t)}\left(y^{(t)}\right)^{\mathsf{T}}\delta^{(t)} \middle| \mathcal{F}_t\right]\\
&= \left(\delta^{(t)}\right)^{\mathsf{T}}\mathbb{E}\left[y^{(t)}\left(y^{(t)}\right)^{\mathsf{T}} \middle| \mathcal{F}_t\right]\delta^{(t)}\\
&= \left(\delta^{(t)}\right)^{\mathsf{T}}\Sigma^{(t)}\delta^{(t)}\\
&= \left(\delta^{(t)}\right)^{\mathsf{T}}\widehat{S}_t\delta^{(t)}-\left\|\delta^{(t)}\right\|_2^4\\
&\leq\operatorname{Tr}(\widehat{S}_t)\left\|\delta^{(t)}\right\|_2^2-\left\|\delta^{(t)}\right\|_2^4\\
&= up-p^2\\
&=p(u-p).
\end{aligned} E [ ( ( y ( t ) ) T δ ( t ) ) 2 F t ] = E [ ( δ ( t ) ) T y ( t ) ( y ( t ) ) T δ ( t ) F t ] = ( δ ( t ) ) T E [ y ( t ) ( y ( t ) ) T F t ] δ ( t ) = ( δ ( t ) ) T Σ ( t ) δ ( t ) = ( δ ( t ) ) T S t δ ( t ) − δ ( t ) 2 4 ≤ Tr ( S t ) δ ( t ) 2 2 − δ ( t ) 2 4 = u p − p 2 = p ( u − p ) .
We used Σ ( t ) = S ^ t − δ ( t ) ( δ ( t ) ) T \Sigma^{(t)}=\widehat{S}_t-\delta^{(t)}\left(\delta^{(t)}\right)^{\mathsf{T}} Σ ( t ) = S t − δ ( t ) ( δ ( t ) ) T (Lemma A.14(ii)), and Lemma A.15 with S ^ t ≥ 0 \widehat{S}_t\geq0 S t ≥ 0 by Lemma A.14(i). Collecting terms in (58),
E [ ∥ Y ( t ) − μ ∥ 2 4 | F t ] = E [ ∥ y ( t ) ∥ 2 4 | F t ] + 4 E [ ( ( y ( t ) ) T δ ( t ) ) 2 | F t ] + 2 p ⋅ E [ ∥ y ( t ) ∥ 2 2 | F t ] + p 2 ≤ 3 ( u − p ) 2 + 4 p ( u − p ) + 2 p ( u − p ) + p 2 = 3 ( u − p ) 2 + 6 p ( u − p ) + p 2 = 3 u 2 − 3 p 2 + p 2 = 3 u 2 − 2 p 2 ≤ 3 u 2 = 3 Tr ( S ^ t ) 2 . \begin{aligned}
\mathbb{E}\left[\left\|Y^{(t)}-\mu\right\|_2^4 \middle| \mathcal{F}_t\right]
&=\mathbb{E}\left[\left\|y^{(t)}\right\|_2^4 \middle| \mathcal{F}_t\right]+4\mathbb{E}\left[\left(\left(y^{(t)}\right)^{\mathsf{T}}\delta^{(t)}\right)^2 \middle| \mathcal{F}_t\right]+2p\cdot\mathbb{E}\left[\left\|y^{(t)}\right\|_2^2 \middle| \mathcal{F}_t\right]+p^2\\
&\leq3(u-p)^2+4p(u-p)+2p(u-p)+p^2\\
&=3(u-p)^2+6p(u-p)+p^2\\
&=3u^2-3p^2+p^2\\
&=3u^2-2p^2\\
&\leq3u^2\\
&=3\operatorname{Tr}(\widehat{S}_t)^2.
\end{aligned} E [ Y ( t ) − μ 2 4 F t ] = E [ y ( t ) 2 4 F t ] + 4 E [ ( ( y ( t ) ) T δ ( t ) ) 2 F t ] + 2 p ⋅ E [ y ( t ) 2 2 F t ] + p 2 ≤ 3 ( u − p ) 2 + 4 p ( u − p ) + 2 p ( u − p ) + p 2 = 3 ( u − p ) 2 + 6 p ( u − p ) + p 2 = 3 u 2 − 3 p 2 + p 2 = 3 u 2 − 2 p 2 ≤ 3 u 2 = 3 Tr ( S t ) 2 .
We used − 2 p 2 ≤ 0 -2p^2\leq0 − 2 p 2 ≤ 0 , and
3 ( u − p ) 2 + 6 p ( u − p ) = 3 ( u − p ) ( u − p + 2 p ) = 3 ( u − p ) ( u + p ) = 3 u 2 − 3 p 2 . 3(u-p)^2+6p(u-p)=3(u-p)(u-p+2p)=3(u-p)(u+p)=3u^2-3p^2. 3 ( u − p ) 2 + 6 p ( u − p ) = 3 ( u − p ) ( u − p + 2 p ) = 3 ( u − p ) ( u + p ) = 3 u 2 − 3 p 2 .
Substituting into (57) and bounding 1 − α t ≤ 1 1-\alpha_t\leq1 1 − α t ≤ 1 gives (56).
Lemma A.17 (Recursion unrolling). Let { a t } t ≥ 1 \{a_t\}_{t\geq1} { a t } t ≥ 1 , { r t } t ≥ 1 \{r_t\}_{t\geq1} { r t } t ≥ 1 , { q t } t ≥ 1 \{q_t\}_{t\geq1} { q t } t ≥ 1 be nonnegative sequences with a t ≤ r t − 1 a t − 1 + q t − 1 a_t\leq r_{t-1}a_{t-1}+q_{t-1} a t ≤ r t − 1 a t − 1 + q t − 1 for all t ≥ 2 t\geq2 t ≥ 2 . Then for every t ≥ 2 t\geq2 t ≥ 2 ,
a t ≤ ( ∏ k = 1 t − 1 r k ) a 1 + ∑ s = 1 t − 1 ( ∏ k = s + 1 t − 1 r k ) q s , a_t\leq\left(\prod_{k=1}^{t-1}r_k\right)a_1+\sum_{s=1}^{t-1}\left(\prod_{k=s+1}^{t-1}r_k\right)q_s, a t ≤ ( k = 1 ∏ t − 1 r k ) a 1 + s = 1 ∑ t − 1 ( k = s + 1 ∏ t − 1 r k ) q s , with the empty product equal to 1.
Proof. We prove by induction on t t t . The base case t = 2 t=2 t = 2 is the hypothesis a 2 ≤ r 1 a 1 + q 1 a_2\leq r_1a_1+q_1 a 2 ≤ r 1 a 1 + q 1 . Assuming the bound at t − 1 t-1 t − 1 , we get
a t ≤ r t − 1 a t − 1 + q t − 1 ≤ r t − 1 [ ( ∏ k = 1 t − 2 r k ) a 1 + ∑ s = 1 t − 2 ( ∏ k = s + 1 t − 2 r k ) q s ] + q t − 1 = ( ∏ k = 1 t − 1 r k ) a 1 + ∑ s = 1 t − 2 ( ∏ k = s + 1 t − 1 r k ) q s + q t − 1 = ( ∏ k = 1 t − 1 r k ) a 1 + ∑ s = 1 t − 1 ( ∏ k = s + 1 t − 1 r k ) q s , \begin{aligned}
a_t&\leq r_{t-1}a_{t-1}+q_{t-1}\leq r_{t-1}\left[\left(\prod_{k=1}^{t-2}r_k\right)a_1+\sum_{s=1}^{t-2}\left(\prod_{k=s+1}^{t-2}r_k\right)q_s\right]+q_{t-1}\\
&=\left(\prod_{k=1}^{t-1}r_k\right)a_1+\sum_{s=1}^{t-2}\left(\prod_{k=s+1}^{t-1}r_k\right)q_s+q_{t-1}\\
&=\left(\prod_{k=1}^{t-1}r_k\right)a_1+\sum_{s=1}^{t-1}\left(\prod_{k=s+1}^{t-1}r_k\right)q_s,
\end{aligned} a t ≤ r t − 1 a t − 1 + q t − 1 ≤ r t − 1 [ ( k = 1 ∏ t − 2 r k ) a 1 + s = 1 ∑ t − 2 ( k = s + 1 ∏ t − 2 r k ) q s ] + q t − 1 = ( k = 1 ∏ t − 1 r k ) a 1 + s = 1 ∑ t − 2 ( k = s + 1 ∏ t − 1 r k ) q s + q t − 1 = ( k = 1 ∏ t − 1 r k ) a 1 + s = 1 ∑ t − 1 ( k = s + 1 ∏ t − 1 r k ) q s ,
where q t − 1 q_{t-1} q t − 1 is the s = t − 1 s=t-1 s = t − 1 term (empty product = 1).
Lemma A.18 (Uniform bound on the amplification products). Let r t : = λ t 2 + 3 α t β t r_t := \lambda_t^2 + 3\alpha_t\beta_t r t := λ t 2 + 3 α t β t for t ≥ 1 t \ge1 t ≥ 1 , with λ t , β t \lambda_t,\beta_t λ t , β t as in (18). Then r t ≥ 0 r_t \ge0 r t ≥ 0 for every t ≥ 1 t \ge1 t ≥ 1 , and for all integers 1 ≤ a ≤ b < ∞ 1 \le a \le b < \infty 1 ≤ a ≤ b < ∞ ,
∏ t = a b r t ≤ exp ( 3 n r ) < ∞ . (57) \prod_{t=a}^{b} r_t \le\exp\left(\frac{3}{n_r}\right) < \infty. \tag*{(57)} t = a ∏ b r t ≤ exp ( n r 3 ) < ∞. (57) In particular the bound is uniform in a a a and b b b , and holds for every { α t } ⊂ [ 0 , 1 ] \{\alpha_t\} \subset[0,1] { α t } ⊂ [ 0 , 1 ] and { n b ( t ) ≥ 1 } \{n_b(t) \ge1\} { n b ( t ) ≥ 1 } , with no further hypothesis.
Proof. Fix t ≥ 1 t \ge1 t ≥ 1 . By Lemma A.9 we have λ t ∈ ( 0 , 1 ] \lambda_t \in(0,1] λ t ∈ ( 0 , 1 ] , hence λ t 2 ≤ 1 \lambda_t^2 \le1 λ t 2 ≤ 1 ; and α t ∈ [ 0 , 1 ] \alpha_t \in[0,1] α t ∈ [ 0 , 1 ] gives 3 α t β t ≤ 3 β t 3\alpha_t\beta_t \le3\beta_t 3 α t β t ≤ 3 β t . Adding,
0 ≤ r t ≤ 1 + 3 β t , (58) 0 \le r_t \le1 + 3\beta_t, \tag*{(58)} 0 ≤ r t ≤ 1 + 3 β t , (58) the lower bound because both summands defining r t r_t r t are nonnegative. Fix integers 1 ≤ a ≤ b 1 \le a \le b 1 ≤ a ≤ b . From (60), we have
∏ t = a b r t ≤ ∏ t = a b ( 1 + 3 β t ) ≤ ∏ t = 1 ∞ ( 1 + 3 β t ) = exp ( ∑ t = 1 ∞ ln ( 1 + 3 β t ) ) ≤ exp ( 3 ∑ t = 1 ∞ β t ) ≤ exp ( 3 n r ) . \begin{aligned}
\prod_{t=a}^{b} r_t &\le\prod_{t=a}^{b}(1+3\beta_t) \le\prod_{t=1}^{\infty}(1+3\beta_t) \\
&= \exp\left(\sum_{t=1}^{\infty}\ln(1+3\beta_t)\right) \\
&\le\exp\left(3\sum_{t=1}^{\infty}\beta_t\right) \\
&\le\exp\left(\frac{3}{n_r}\right).
\end{aligned} t = a ∏ b r t ≤ t = a ∏ b ( 1 + 3 β t ) ≤ t = 1 ∏ ∞ ( 1 + 3 β t ) = exp ( t = 1 ∑ ∞ ln ( 1 + 3 β t ) ) ≤ exp ( 3 t = 1 ∑ ∞ β t ) ≤ exp ( n r 3 ) . Where we used 1 + 3 β t ≥ 1 1+3\beta_t \ge1 1 + 3 β t ≥ 1 , ln ( 1 + x ) ≤ x \ln(1+x) \le x ln ( 1 + x ) ≤ x for x ≥ 0 x \ge0 x ≥ 0 , and ∑ t = 1 ∞ β t ≤ 1 / n r \sum_{t=1}^{\infty}\beta_t \le1/n_r ∑ t = 1 ∞ β t ≤ 1/ n r from Lemma A.5. □
Lemma A.19 (Uniform second-moment bound on S ^ t \widehat{S}_t S t ). Under Assumption 1,
B ∗ : = sup t ≥ 1 E [ Tr ( S ^ t ) 2 ] < ∞ . B^* := \sup_{t\ge1}\mathbb{E}\left[\operatorname{Tr}(\widehat{S}_t)^2\right] < \infty. B ∗ := t ≥ 1 sup E [ Tr ( S t ) 2 ] < ∞. Proof. Write u t : = Tr ( S ^ t ) u_t := \operatorname{Tr}(\widehat{S}_t) u t := Tr ( S t ) , and denote
v t : = Var ( u t ) = E [ Tr ( S ^ t ) 2 ] − Tr ( Σ ) 2 . (59) v_t := \operatorname{Var}(u_t) = \mathbb{E}\left[\operatorname{Tr}(\widehat{S}_t)^2\right] - \operatorname{Tr}(\Sigma)^2. \tag*{(59)} v t := Var ( u t ) = E [ Tr ( S t ) 2 ] − Tr ( Σ ) 2 . (59) We used E [ u t ] = E [ Tr ( S ^ t ) ] = Tr ( E [ S ^ t ] ) = Tr ( Σ ) \mathbb{E}[u_t] = \mathbb{E}[\operatorname{Tr}(\widehat{S}_t)] = \operatorname{Tr}(\mathbb{E}[\widehat{S}_t]) = \operatorname{Tr}(\Sigma) E [ u t ] = E [ Tr ( S t )] = Tr ( E [ S t ]) = Tr ( Σ ) (Lemma A.14(iv)). We establish a recursion on u t u_t u t . From Lemma A.14(iii), we have for every $t \ge 2
u t = ρ t − 1 u t − 1 + 1 N t ∑ i = 1 n b ( t − 1 ) Tr ( ( x i ( t − 1 ) − μ ) ( x i ( t − 1 ) − μ ) T ) = ρ t − 1 u t − 1 + 1 N t ∑ i = 1 n b ( t − 1 ) ∥ x i ( t − 1 ) − μ ∥ 2 2 (60) \begin{aligned}
u_t &= \rho_{t-1}u_{t-1} + \frac{1}{N_t}\sum_{i=1}^{n_b(t-1)} \operatorname{Tr}\left((x_i^{(t-1)}-\mu)(x_i^{(t-1)}-\mu)^\mathsf{T}\right) \\
&= \rho_{t-1}u_{t-1} + \frac{1}{N_t}\sum_{i=1}^{n_b(t-1)}\left\|x_i^{(t-1)}-\mu\right\|_2^2 \tag*{(60)}
\end{aligned} u t = ρ t − 1 u t − 1 + N t 1 i = 1 ∑ n b ( t − 1 ) Tr ( ( x i ( t − 1 ) − μ ) ( x i ( t − 1 ) − μ ) T ) = ρ t − 1 u t − 1 + N t 1 i = 1 ∑ n b ( t − 1 ) x i ( t − 1 ) − μ 2 2 (60) and u 1 = 1 n r ∑ j = 1 n r ∥ x j ( 0 ) − μ ∥ 2 2 u_1 = \frac{1}{n_r}\sum_{j=1}^{n_r}\left\|x_j^{(0)}-\mu\right\|_2^2 u 1 = n r 1 ∑ j = 1 n r x j ( 0 ) − μ 2 2 . From (61), we need to prove that sup t ≥ 1 v t < ∞ \sup_{t\ge1}v_t < \infty sup t ≥ 1 v t < ∞ . We will use the law of total variance, which presupposes u t ∈ L 2 u_t \in L^2 u t ∈ L 2 . We verify that by induction on t ≥ 1 t \ge1 t ≥ 1 .
*Square-integrability at every finite t . ∗ t.* t . ∗ For t = 1 t=1 t = 1 , the samples { x i ( 0 ) } i = 1 n r \{x_i^{(0)}\}_{i=1}^{n_r} { x i ( 0 ) } i = 1 n r are i.i.d. from N ( μ , Σ ) \mathcal{N}(\mu,\Sigma) N ( μ , Σ ) , and thus
v 1 = 1 n r Var ( ∥ x 1 ( 0 ) − μ ∥ 2 2 ) ≤ 1 n r E [ ∥ x 1 ( 0 ) − μ ∥ 2 4 ] = 1 n r C Σ < ∞ , (61) v_1 = \frac{1}{n_r}\operatorname{Var}\left(\left\|x_1^{(0)}-\mu\right\|_2^2\right) \le\frac{1}{n_r}\mathbb{E}\left[\left\|x_1^{(0)}-\mu\right\|_2^4\right] = \frac{1}{n_r}C_\Sigma< \infty, \tag*{(61)} v 1 = n r 1 Var ( x 1 ( 0 ) − μ 2 2 ) ≤ n r 1 E [ x 1 ( 0 ) − μ 2 4 ] = n r 1 C Σ < ∞ , (61)
where we used Var ( X ) ≤ E [ X 2 ] \operatorname{Var}(X) \le\mathbb{E}[X^2] Var ( X ) ≤ E [ X 2 ] . From (61), we have E [ u 1 2 ] = v 1 + Tr ( Σ ) 2 < ∞ \mathbb{E}[u_1^2] = v_1 + \operatorname{Tr}(\Sigma)^2 < \infty E [ u 1 2 ] = v 1 + Tr ( Σ ) 2 < ∞ , thus u 1 ∈ L 2 u_1 \in L^2 u 1 ∈ L 2 . Assume u t − 1 ∈ L 2 u_{t-1} \in L^2 u t − 1 ∈ L 2 , then
E [ u t 2 ] ≤ 2 ( ρ t − 1 2 E [ u t − 1 2 ] + n b ( t − 1 ) N t 2 ∑ i = 1 n b ( t − 1 ) E [ ∥ x i ( t − 1 ) − μ ∥ 2 4 ] ) \mathbb{E}[u_t^2] \le2\left(\rho_{t-1}^2\mathbb{E}[u_{t-1}^2] + \frac{n_b(t-1)}{N_t^2}\sum_{i=1}^{n_b(t-1)}\mathbb{E}\left[\lVert x_i^{(t-1)}-\mu\rVert_2^4\right]\right) E [ u t 2 ] ≤ 2 ρ t − 1 2 E [ u t − 1 2 ] + N t 2 n b ( t − 1 ) i = 1 ∑ n b ( t − 1 ) E [ ∥ x i ( t − 1 ) − μ ∥ 2 4 ]
≤ 2 ( ρ t − 1 2 E [ u t − 1 2 ] + n b ( t − 1 ) N t 2 ∑ i = 1 n b ( t − 1 ) ( 3 α t − 1 E [ u t − 1 2 ] + C Σ ) ) < ∞ , \begin{aligned}
\le2\left(\rho_{t-1}^2\mathbb{E}[u_{t-1}^2] + \frac{n_b(t-1)}{N_t^2}\sum_{i=1}^{n_b(t-1)}\left(3\alpha_{t-1}\mathbb{E}[u_{t-1}^2]+C_\Sigma\right)\right) \\
&<\infty,
\end{aligned} ≤ 2 ρ t − 1 2 E [ u t − 1 2 ] + N t 2 n b ( t − 1 ) i = 1 ∑ n b ( t − 1 ) ( 3 α t − 1 E [ u t − 1 2 ] + C Σ ) < ∞ ,
where the first inequality applies Lemma A.3 twice: with n = 2 n=2 n = 2 to the two summands of (62), i.e., ( a + b ) 2 ≤ 2 ( a 2 + b 2 ) (a+b)^2\le2(a^2+b^2) ( a + b ) 2 ≤ 2 ( a 2 + b 2 ) , and with n = n b ( t − 1 ) n=n_b(t-1) n = n b ( t − 1 ) to the sum ∑ i = 1 n b ( t − 1 ) ∥ x i ( t − 1 ) − μ ∥ 2 2 \sum_{i=1}^{n_b(t-1)}\lVert x_i^{(t-1)}-\mu\rVert_2^2 ∑ i = 1 n b ( t − 1 ) ∥ x i ( t − 1 ) − μ ∥ 2 2 , which gives
( ∑ i = 1 n b ( t − 1 ) ∥ x i ( t − 1 ) − μ ∥ 2 2 ) 2 ≤ n b ( t − 1 ) ∑ i = 1 n b ( t − 1 ) ∥ x i ( t − 1 ) − μ ∥ 2 4 . \left(\sum_{i=1}^{n_b(t-1)}\lVert x_i^{(t-1)}-\mu\rVert_2^2\right)^2\le n_b(t-1)\sum_{i=1}^{n_b(t-1)}\lVert x_i^{(t-1)}-\mu\rVert_2^4. i = 1 ∑ n b ( t − 1 ) ∥ x i ( t − 1 ) − μ ∥ 2 2 2 ≤ n b ( t − 1 ) i = 1 ∑ n b ( t − 1 ) ∥ x i ( t − 1 ) − μ ∥ 2 4 .
The second inequality uses the following bound: from Lemma A.16 and the tower property, we have
E [ ∥ x i ( t − 1 ) − μ ∥ 2 4 ] = E [ E [ ∥ x i ( t − 1 ) − μ ∥ 2 4 ∣ F t − 1 ] ] ≤ E [ 3 α t − 1 Tr ( S ^ t − 1 ) 2 + C Σ ] = 3 α t − 1 E [ u t − 1 2 ] + C Σ . \mathbb{E}\left[\lVert x_i^{(t-1)}-\mu\rVert_2^4\right]=\mathbb{E}\left[\mathbb{E}\left[\lVert x_i^{(t-1)}-\mu\rVert_2^4\mid\mathcal{F}_{t-1}\right]\right]\le\mathbb{E}\left[3\alpha_{t-1}\operatorname{Tr}(\widehat{S}_{t-1})^2+C_\Sigma\right]=3\alpha_{t-1}\mathbb{E}[u_{t-1}^2]+C_\Sigma. E [ ∥ x i ( t − 1 ) − μ ∥ 2 4 ] = E [ E [ ∥ x i ( t − 1 ) − μ ∥ 2 4 ∣ F t − 1 ] ] ≤ E [ 3 α t − 1 Tr ( S t − 1 ) 2 + C Σ ] = 3 α t − 1 E [ u t − 1 2 ] + C Σ .
The law of total variance. Fix t ≥ 2 t\ge2 t ≥ 2 . The law of total variance dictates that
v t = Var ( E [ u t ∣ F t − 1 ] ) + E [ Var ( u t ∣ F t − 1 ) ] . (62) v_t=\operatorname{Var}\left(\mathbb{E}[u_t\mid\mathcal{F}_{t-1}]\right)+\mathbb{E}\left[\operatorname{Var}(u_t\mid\mathcal{F}_{t-1})\right]. \tag*{(62)} v t = Var ( E [ u t ∣ F t − 1 ] ) + E [ Var ( u t ∣ F t − 1 ) ] . (62)
Taking the trace of Lemma A.14(iv), E [ u t ∣ F t − 1 ] = λ t − 1 u t − 1 + ( 1 − λ t − 1 ) Tr ( Σ ) \mathbb{E}[u_t\mid\mathcal{F}_{t-1}]=\lambda_{t-1}u_{t-1}+(1-\lambda_{t-1})\operatorname{Tr}(\Sigma) E [ u t ∣ F t − 1 ] = λ t − 1 u t − 1 + ( 1 − λ t − 1 ) Tr ( Σ ) . Thus, the first summand reads
Var ( E [ u t ∣ F t − 1 ] ) = λ t − 1 2 v t − 1 . (63) \operatorname{Var}\left(\mathbb{E}[u_t\mid\mathcal{F}_{t-1}]\right)=\lambda_{t-1}^2v_{t-1}. \tag*{(63)} Var ( E [ u t ∣ F t − 1 ] ) = λ t − 1 2 v t − 1 . (63)
For the second summand, we use (62):
Var ( u t ∣ F t − 1 ) = n b ( t − 1 ) N t 2 Var ( ∥ x 1 ( t − 1 ) − μ ∥ 2 2 ∣ F t − 1 ) ≤ β t − 1 E [ ∥ x 1 ( t − 1 ) − μ ∥ 2 4 ∣ F t − 1 ] ≤ β t − 1 ( 3 α t − 1 u t − 1 2 + C Σ ) . (64) \begin{aligned}
\operatorname{Var}(u_t\mid\mathcal{F}_{t-1})=\frac{n_b(t-1)}{N_t^2}\operatorname{Var}\left(\lVert x_1^{(t-1)}-\mu\rVert_2^2\mid\mathcal{F}_{t-1}\right)\le\beta_{t-1}\mathbb{E}\left[\lVert x_1^{(t-1)}-\mu\rVert_2^4\mid\mathcal{F}_{t-1}\right] \\
&\le\beta_{t-1}\left(3\alpha_{t-1}u_{t-1}^2+C_\Sigma\right). \tag*{(64)}
\end{aligned} Var ( u t ∣ F t − 1 ) = N t 2 n b ( t − 1 ) Var ( ∥ x 1 ( t − 1 ) − μ ∥ 2 2 ∣ F t − 1 ) ≤ β t − 1 E [ ∥ x 1 ( t − 1 ) − μ ∥ 2 4 ∣ F t − 1 ] ≤ β t − 1 ( 3 α t − 1 u t − 1 2 + C Σ ) . (64)
We used that u t − 1 u_{t-1} u t − 1 is F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable, and conditionally on F t − 1 \mathcal{F}_{t-1} F t − 1 the samples { x i ( t − 1 ) } i = 1 n b ( t − 1 ) \{x_i^{(t-1)}\}_{i=1}^{n_b(t-1)} { x i ( t − 1 ) } i = 1 n b ( t − 1 ) are i.i.d. (Assumption 1), Var ≤ \operatorname{Var}\le Var ≤ second moment together with β t \beta_t β t definition from (18), and Lemma A.16. From (64), (65) and (66), we have
v t ≤ λ t − 1 2 v t − 1 + E [ 3 β t − 1 α t − 1 u t − 1 2 + β t − 1 C Σ ] = ( λ t − 1 2 + 3 α t − 1 β t − 1 ) v t − 1 + β t − 1 ( 3 α t − 1 Tr ( Σ ) 2 + C Σ ) . (65) \begin{aligned}
v_t\le\lambda_{t-1}^2v_{t-1}+\mathbb{E}\left[3\beta_{t-1}\alpha_{t-1}u_{t-1}^2+\beta_{t-1}C_\Sigma\right] \\
&=(\lambda_{t-1}^2+3\alpha_{t-1}\beta_{t-1})v_{t-1}+\beta_{t-1}\left(3\alpha_{t-1}\operatorname{Tr}(\Sigma)^2+C_\Sigma\right). \tag*{(65)}
\end{aligned} v t ≤ λ t − 1 2 v t − 1 + E [ 3 β t − 1 α t − 1 u t − 1 2 + β t − 1 C Σ ] = ( λ t − 1 2 + 3 α t − 1 β t − 1 ) v t − 1 + β t − 1 ( 3 α t − 1 Tr ( Σ ) 2 + C Σ ) . (65)
where we used E [ u t − 1 2 ] = v t − 1 + Tr ( Σ ) 2 \mathbb{E}[u_{t-1}^2]=v_{t-1}+\operatorname{Tr}(\Sigma)^2 E [ u t − 1 2 ] = v t − 1 + Tr ( Σ ) 2 from (61). From (67), we have:
v t ≤ r t − 1 v t − 1 + q t − 1 , { r t − 1 : = λ t − 1 2 + 3 α t − 1 β t − 1 , q t − 1 : = β t − 1 ( 3 α t − 1 Tr ( Σ ) 2 + C Σ ) . (66) v_t\le r_{t-1}v_{t-1}+q_{t-1},\qquad
\begin{cases}
r_{t-1}:=\lambda_{t-1}^2+3\alpha_{t-1}\beta_{t-1},\\
q_{t-1}:=\beta_{t-1}\left(3\alpha_{t-1}\operatorname{Tr}(\Sigma)^2+C_\Sigma\right).
\end{cases} \tag*{(66)} v t ≤ r t − 1 v t − 1 + q t − 1 , { r t − 1 := λ t − 1 2 + 3 α t − 1 β t − 1 , q t − 1 := β t − 1 ( 3 α t − 1 Tr ( Σ ) 2 + C Σ ) . (66)
Uniform bound. Applying Lemma A.17 to (68) and using Lemma A.18, we have
v t ≤ ( ∏ k = 1 t − 1 r k ) v 1 + ∑ s = 1 t − 1 ( ∏ k = s + 1 t − 1 r k ) q s ≤ exp ( 3 n r ) v 1 + exp ( 3 n r ) ∑ s = 1 t − 1 q s ≤ exp ( 3 n r ) ( v 1 + D 0 ) . (67) \begin{aligned}
v_t\le\left(\prod_{k=1}^{t-1}r_k\right)v_1+\sum_{s=1}^{t-1}\left(\prod_{k=s+1}^{t-1}r_k\right)q_s \\
&\le\exp\left(\frac{3}{n_r}\right)v_1+\exp\left(\frac{3}{n_r}\right)\sum_{s=1}^{t-1}q_s \\
&\le\exp\left(\frac{3}{n_r}\right)(v_1+D_0). \tag*{(67)}
\end{aligned} v t ≤ ( k = 1 ∏ t − 1 r k ) v 1 + s = 1 ∑ t − 1 ( k = s + 1 ∏ t − 1 r k ) q s ≤ exp ( n r 3 ) v 1 + exp ( n r 3 ) s = 1 ∑ t − 1 q s ≤ exp ( n r 3 ) ( v 1 + D 0 ) . (67)
uniformly in t t t . Using (65) , we defined
D 0 : = ∑ s = 1 ∞ q s ≤ ( 3 Tr ( Σ ) 2 + C Σ ) ∑ t = 1 ∞ β t ≤ 1 n r ( C Σ + 3 Tr ( Σ ) 2 ) . (68) D_0 := \sum_{s=1}^{\infty} q_s \leq\left(3\operatorname{Tr}(\Sigma)^2+C_{\Sigma}\right)\sum_{t=1}^{\infty}\beta_t
\leq\frac{1}{n_r}\left(C_{\Sigma}+3\operatorname{Tr}(\Sigma)^2\right). \tag*{(68)} D 0 := s = 1 ∑ ∞ q s ≤ ( 3 Tr ( Σ ) 2 + C Σ ) t = 1 ∑ ∞ β t ≤ n r 1 ( C Σ + 3 Tr ( Σ ) 2 ) . (68)
The first inequality follows from α t − 1 ≤ 1 \alpha_{t-1} \leq1 α t − 1 ≤ 1 , and the second inequality follows from Lemma A.5. Finally, from (59) , (67) and (68) , we have
sup t ≥ 1 E [ Tr ( S ^ t ) 2 ] = sup t ≥ 1 ( v t + Tr ( Σ ) 2 ) ≤ exp ( 3 n r ) ( v 1 + D 0 ) + Tr ( Σ ) 2 < ∞ . (69) \sup_{t\geq1}\mathbb{E}\left[\operatorname{Tr}(\widehat{S}_t)^2\right]=\sup_{t\geq1}\left(v_t+\operatorname{Tr}(\Sigma)^2\right)\leq\exp\left(\frac{3}{n_r}\right)(v_1+D_0)+\operatorname{Tr}(\Sigma)^2<\infty. \tag*{(69)} t ≥ 1 sup E [ Tr ( S t ) 2 ] = t ≥ 1 sup ( v t + Tr ( Σ ) 2 ) ≤ exp ( n r 3 ) ( v 1 + D 0 ) + Tr ( Σ ) 2 < ∞. (69)
Lemma A.20 (Divergence of 1 − λ t 2 1-\lambda_t^2 1 − λ t 2 ). If ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ \sum_{t=1}^{\infty}(1-\alpha_t)(1-\rho_t)=\infty ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ , then ∑ t = 1 ∞ ( 1 − λ t 2 ) = ∞ \sum_{t=1}^{\infty}(1-\lambda_t^2)=\infty ∑ t = 1 ∞ ( 1 − λ t 2 ) = ∞ .
Proof. From the definition of λ t \lambda_t λ t (18) and the definition of ρ t \rho_t ρ t in (3) , we have
1 − λ t = 1 − N t + n b ( t ) α t N t + 1 = N t + 1 − N t − n b ( t ) α t N t + 1 = n b ( t ) ( 1 − α t ) N t + 1 = ( 1 − ρ t ) ( 1 − α t ) , 1-\lambda_t=1-\frac{N_t+n_b(t)\alpha_t}{N_{t+1}}=\frac{N_{t+1}-N_t-n_b(t)\alpha_t}{N_{t+1}}=\frac{n_b(t)(1-\alpha_t)}{N_{t+1}}=(1-\rho_t)(1-\alpha_t), 1 − λ t = 1 − N t + 1 N t + n b ( t ) α t = N t + 1 N t + 1 − N t − n b ( t ) α t = N t + 1 n b ( t ) ( 1 − α t ) = ( 1 − ρ t ) ( 1 − α t ) , where we used N t + 1 = N t + n b ( t ) N_{t+1}=N_t+n_b(t) N t + 1 = N t + n b ( t ) and ρ t = N t / N t + 1 ⇒ 1 − ρ t = n b ( t ) / N t + 1 \rho_t=N_t/N_{t+1}\Rightarrow1-\rho_t=n_b(t)/N_{t+1} ρ t = N t / N t + 1 ⇒ 1 − ρ t = n b ( t ) / N t + 1 . As λ t ∈ ( 0 , 1 ] \lambda_t\in(0,1] λ t ∈ ( 0 , 1 ] (Lemma A.9 ), 1 − λ t 2 = ( 1 − λ t ) ( 1 + λ t ) ≥ 1 − λ t 1-\lambda_t^2=(1-\lambda_t)(1+\lambda_t)\geq1-\lambda_t 1 − λ t 2 = ( 1 − λ t ) ( 1 + λ t ) ≥ 1 − λ t , thus
∑ t = 1 ∞ ( 1 − λ t 2 ) ≥ ∑ t = 1 ∞ ( 1 − λ t ) = ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ \sum_{t=1}^{\infty}(1-\lambda_t^2)\geq\sum_{t=1}^{\infty}(1-\lambda_t)=\sum_{t=1}^{\infty}(1-\alpha_t)(1-\rho_t)=\infty t = 1 ∑ ∞ ( 1 − λ t 2 ) ≥ t = 1 ∑ ∞ ( 1 − λ t ) = t = 1 ∑ ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ by hypothesis.
Lemma A.21 (A useful upper bound). Let X X X be a random matrix with E [ ∥ X ∥ F 2 ] < ∞ \mathbb{E}[\lVert X\rVert_F^2]<\infty E [∥ X ∥ F 2 ] < ∞ , and G \mathcal{G} G be a sub-σ \sigma σ -algebra. Define Z : = X − E [ X ∣ G ] Z:=X-\mathbb{E}[X\mid\mathcal{G}] Z := X − E [ X ∣ G ] . Then,
E [ ∥ Z ∥ F 2 ∣ G ] ≤ E [ ∥ X ∥ F 2 ∣ G ] . \mathbb{E}[\lVert Z\rVert_F^2\mid\mathcal{G}]\leq\mathbb{E}[\lVert X\rVert_F^2\mid\mathcal{G}]. E [∥ Z ∥ F 2 ∣ G ] ≤ E [∥ X ∥ F 2 ∣ G ] . Proof. Indeed, we have
E [ ∥ Z ∥ F 2 ∣ G ] = E [ Tr ( Z T Z ) ∣ G ] = Tr ( E [ Z T Z ∣ G ] ) = Tr ( E [ ( X − E [ X ∣ G ] ) T ( X − E [ X ∣ G ] ) ∣ G ] ) = Tr ( E [ X T X ∣ G ] − E [ X T E [ X ∣ G ] ∣ G ] − E [ E [ X ∣ G ] T X ∣ G ] + E [ E [ X ∣ G ] T E [ X ∣ G ] ∣ G ] ) = E [ Tr ( X T X ) ∣ G ] − Tr ( E [ X ∣ G ] T E [ X ∣ G ] + E [ X ∣ G ] T E [ X ∣ G ] − E [ X ∣ G ] T E [ X ∣ G ] ) = E [ Tr ( X T X ) ∣ G ] − Tr ( E [ X ∣ G ] T E [ X ∣ G ] ) = E [ ∥ X ∥ F 2 ∣ G ] − ∥ E [ X ∣ G ] ∥ F 2 ≤ E [ ∥ X ∥ F 2 ∣ G ] . \begin{aligned}
\mathbb{E}[\lVert Z\rVert_F^2\mid\mathcal{G}]&=\mathbb{E}[\operatorname{Tr}(Z^{\mathsf{T}}Z)\mid\mathcal{G}]\\
&=\operatorname{Tr}\left(\mathbb{E}[Z^{\mathsf{T}}Z\mid\mathcal{G}]\right)\\
&=\operatorname{Tr}\left(\mathbb{E}[(X-\mathbb{E}[X\mid\mathcal{G}])^{\mathsf{T}}(X-\mathbb{E}[X\mid\mathcal{G}])\mid\mathcal{G}]\right)\\
&=\operatorname{Tr}\left(\mathbb{E}[X^{\mathsf{T}}X\mid\mathcal{G}]-\mathbb{E}[X^{\mathsf{T}}\mathbb{E}[X\mid\mathcal{G}]\mid\mathcal{G}]-\mathbb{E}[\mathbb{E}[X\mid\mathcal{G}]^{\mathsf{T}}X\mid\mathcal{G}]+\mathbb{E}[\mathbb{E}[X\mid\mathcal{G}]^{\mathsf{T}}\mathbb{E}[X\mid\mathcal{G}]\mid\mathcal{G}]\right)\\
&=\mathbb{E}[\operatorname{Tr}(X^{\mathsf{T}}X)\mid\mathcal{G}]-\operatorname{Tr}\left(\mathbb{E}[X\mid\mathcal{G}]^{\mathsf{T}}\mathbb{E}[X\mid\mathcal{G}]+\mathbb{E}[X\mid\mathcal{G}]^{\mathsf{T}}\mathbb{E}[X\mid\mathcal{G}]-\mathbb{E}[X\mid\mathcal{G}]^{\mathsf{T}}\mathbb{E}[X\mid\mathcal{G}]\right)\\
&=\mathbb{E}[\operatorname{Tr}(X^{\mathsf{T}}X)\mid\mathcal{G}]-\operatorname{Tr}\left(\mathbb{E}[X\mid\mathcal{G}]^{\mathsf{T}}\mathbb{E}[X\mid\mathcal{G}]\right)\\
&=\mathbb{E}[\lVert X\rVert_F^2\mid\mathcal{G}]-\lVert\mathbb{E}[X\mid\mathcal{G}]\rVert_F^2\\
&\leq\mathbb{E}[\lVert X\rVert_F^2\mid\mathcal{G}].
\end{aligned} E [∥ Z ∥ F 2 ∣ G ] = E [ Tr ( Z T Z ) ∣ G ] = Tr ( E [ Z T Z ∣ G ] ) = Tr ( E [( X − E [ X ∣ G ] ) T ( X − E [ X ∣ G ]) ∣ G ] ) = Tr ( E [ X T X ∣ G ] − E [ X T E [ X ∣ G ] ∣ G ] − E [ E [ X ∣ G ] T X ∣ G ] + E [ E [ X ∣ G ] T E [ X ∣ G ] ∣ G ] ) = E [ Tr ( X T X ) ∣ G ] − Tr ( E [ X ∣ G ] T E [ X ∣ G ] + E [ X ∣ G ] T E [ X ∣ G ] − E [ X ∣ G ] T E [ X ∣ G ] ) = E [ Tr ( X T X ) ∣ G ] − Tr ( E [ X ∣ G ] T E [ X ∣ G ] ) = E [∥ X ∥ F 2 ∣ G ] − ∥ E [ X ∣ G ] ∥ F 2 ≤ E [∥ X ∥ F 2 ∣ G ] . We used ∥ A ∥ F 2 = Tr ( A T A ) \lVert A\rVert_F^2=\operatorname{Tr}(A^{\mathsf{T}}A) ∥ A ∥ F 2 = Tr ( A T A ) , E [ Tr ( A ) ∣ G ] = Tr ( E [ A ∣ G ] ) \mathbb{E}[\operatorname{Tr}(A)\mid\mathcal{G}]=\operatorname{Tr}(\mathbb{E}[A\mid\mathcal{G}]) E [ Tr ( A ) ∣ G ] = Tr ( E [ A ∣ G ]) and that E [ X ∣ G ] \mathbb{E}[X\mid\mathcal{G}] E [ X ∣ G ] is G \mathcal{G} G -measurable.
Finally, we characterize when Σ ( t ) \Sigma^{(t)} Σ ( t ) is consistent (in expected Frobenius norm).
Theorem A.22 (Frobenius norm consistency). Under Assumption 1 ,
E [ ∥ Σ ( t ) − Σ ∥ F ] → t → ∞ 0 ⟺ ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ . \mathbb{E}\left[\lVert\Sigma^{(t)}-\Sigma\rVert_F\right]\xrightarrow[t\to\infty]{}0\quad\Longleftrightarrow\quad\sum_{t=1}^{\infty}(1-\alpha_t)(1-\rho_t)=\infty. E [ ∥ Σ ( t ) − Σ ∥ F ] t → ∞ 0 ⟺ t = 1 ∑ ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞.
Proof. Let P t : = E [ ∥ S ^ t − Σ ∥ F 2 ] P_t := \mathbb{E}[\|\widehat{\mathbf{S}}_t-\boldsymbol{\Sigma}\|_F^2] P t := E [ ∥ S t − Σ ∥ F 2 ] .
P t P_t P t is finite. Indeed,
P t ≤ 2 ( E [ ∥ S ^ t ∥ F 2 ] + ∥ Σ ∥ F 2 ) ≤ 2 ( E [ Tr ( S ^ t ) 2 ] + ∥ Σ ∥ F 2 ) < ∞ . P_t \le2\left(\mathbb{E}\left[\|\widehat{\mathbf{S}}_t\|_F^2\right]+\|\boldsymbol{\Sigma}\|_F^2\right) \le2\left(\mathbb{E}\left[\operatorname{Tr}(\widehat{\mathbf{S}}_t)^2\right]+\|\boldsymbol{\Sigma}\|_F^2\right)<\infty. P t ≤ 2 ( E [ ∥ S t ∥ F 2 ] + ∥ Σ ∥ F 2 ) ≤ 2 ( E [ Tr ( S t ) 2 ] + ∥ Σ ∥ F 2 ) < ∞.
We used ∥ A − B ∥ F 2 ≤ 2 ( ∥ A ∥ F 2 + ∥ B ∥ F 2 ) \|\mathbf{A}-\mathbf{B}\|_F^2 \le2(\|\mathbf{A}\|_F^2+\|\mathbf{B}\|_F^2) ∥ A − B ∥ F 2 ≤ 2 ( ∥ A ∥ F 2 + ∥ B ∥ F 2 ) and the monotonicity of the expectation. Then, since S ^ t \widehat{\mathbf{S}}_t S t is symmetric positive-semi-definite (Lemma A.14(ii)), we have ∥ S ^ t ∥ F 2 ≤ Tr ( S ^ t ) 2 \|\widehat{\mathbf{S}}_t\|_F^2\le\operatorname{Tr}(\widehat{\mathbf{S}}_t)^2 ∥ S t ∥ F 2 ≤ Tr ( S t ) 2 (Lemma A.4 (a)). The final bound inequality follows from Lemma A.19 , together with ∥ Σ ∥ F < ∞ \|\boldsymbol{\Sigma}\|_F<\infty ∥ Σ ∥ F < ∞ .
Steps 1–3 establish sufficiency, i.e. that ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ \sum_{t=1}^{\infty}(1-\alpha_t)(1-\rho_t)=\infty ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ implies E [ ∥ Σ ( t ) − Σ ∥ F ] → 0 \mathbb{E}[\|\boldsymbol{\Sigma}^{(t)}-\boldsymbol{\Sigma}\|_F]\to0 E [ ∥ Σ ( t ) − Σ ∥ F ] → 0 . Step 4 establishes necessity.
Step 1 (A recursion for P t P_t P t ). For t ≥ 2 t\ge2 t ≥ 2 , define
V ~ t : = S ^ t − Σ , ξ ~ t : = S ^ t − E [ S ^ t ∣ F t − 1 ] . (70) \widetilde{\mathbf{V}}_t:=\widehat{\mathbf{S}}_t-\boldsymbol{\Sigma},\qquad\widetilde{\boldsymbol{\xi}}_t:=\widehat{\mathbf{S}}_t-\mathbb{E}[\widehat{\mathbf{S}}_t\mid\mathcal{F}_{t-1}]. \tag*{(70)} V t := S t − Σ , ξ t := S t − E [ S t ∣ F t − 1 ] . (70)
A recursion for V ~ t \widetilde{\mathbf{V}}_t V t . From Lemma A.14(iv), we have
V ~ t − ξ ~ t = E [ S ^ t ∣ F t − 1 ] − Σ = λ t − 1 ( S ^ t − 1 − Σ ) = λ t − 1 V ~ t − 1 . \widetilde{\mathbf{V}}_t-\widetilde{\boldsymbol{\xi}}_t=\mathbb{E}[\widehat{\mathbf{S}}_t\mid\mathcal{F}_{t-1}]-\boldsymbol{\Sigma}=\lambda_{t-1}\left(\widehat{\mathbf{S}}_{t-1}-\boldsymbol{\Sigma}\right)=\lambda_{t-1}\widetilde{\mathbf{V}}_{t-1}. V t − ξ t = E [ S t ∣ F t − 1 ] − Σ = λ t − 1 ( S t − 1 − Σ ) = λ t − 1 V t − 1 .
Thus,
V ~ t = λ t − 1 V ~ t − 1 + ξ ~ t . (71) \widetilde{\mathbf{V}}_t=\lambda_{t-1}\widetilde{\mathbf{V}}_{t-1}+\widetilde{\boldsymbol{\xi}}_t. \tag*{(71)} V t = λ t − 1 V t − 1 + ξ t . (71)
Conditional expectation of ξ ~ t \widetilde{\boldsymbol{\xi}}_t ξ t . From (70) , the following identity holds:
E [ ξ ~ t ∣ F t − 1 ] = E [ S ^ t ∣ F t − 1 ] − E [ S ^ t ∣ F t − 1 ] = 0. (72) \mathbb{E}[\widetilde{\boldsymbol{\xi}}_t\mid\mathcal{F}_{t-1}]=\mathbb{E}[\widehat{\mathbf{S}}_t\mid\mathcal{F}_{t-1}]-\mathbb{E}[\widehat{\mathbf{S}}_t\mid\mathcal{F}_{t-1}]=0. \tag*{(72)} E [ ξ t ∣ F t − 1 ] = E [ S t ∣ F t − 1 ] − E [ S t ∣ F t − 1 ] = 0. (72)
Final computation. From (70) , we have P t = E [ ∥ V ~ t ∥ F 2 ] P_t=\mathbb{E}[\|\widetilde{\mathbf{V}}_t\|_F^2] P t = E [ ∥ V t ∥ F 2 ] . Thus,
P t = E [ ⟨ V ~ t , V ~ t ⟩ F ] = E [ ⟨ λ t − 1 V ~ t − 1 + ξ ~ t , λ t − 1 V ~ t − 1 + ξ ~ t ⟩ F ] = λ t − 1 2 E [ ⟨ V ~ t − 1 , V ~ t − 1 ⟩ F ] + 2 λ t − 1 E [ ⟨ V ~ t − 1 , ξ ~ t ⟩ F ] + E [ ⟨ ξ ~ t , ξ ~ t ⟩ F ] = λ t − 1 2 E [ ∥ V ~ t − 1 ∥ F 2 ] + E [ ∥ ξ ~ t ∥ F 2 ] = λ t − 1 2 P t − 1 + E [ ∥ ξ ~ t ∥ F 2 ] . (73) \begin{aligned}
P_t&=\mathbb{E}[\langle\widetilde{\mathbf{V}}_t,\widetilde{\mathbf{V}}_t\rangle_F]\\
&=\mathbb{E}[\langle\lambda_{t-1}\widetilde{\mathbf{V}}_{t-1}+\widetilde{\boldsymbol{\xi}}_t,\lambda_{t-1}\widetilde{\mathbf{V}}_{t-1}+\widetilde{\boldsymbol{\xi}}_t\rangle_F]\\
&=\lambda_{t-1}^2\mathbb{E}[\langle\widetilde{\mathbf{V}}_{t-1},\widetilde{\mathbf{V}}_{t-1}\rangle_F]+2\lambda_{t-1}\mathbb{E}[\langle\widetilde{\mathbf{V}}_{t-1},\widetilde{\boldsymbol{\xi}}_t\rangle_F]+\mathbb{E}[\langle\widetilde{\boldsymbol{\xi}}_t,\widetilde{\boldsymbol{\xi}}_t\rangle_F]\\
&=\lambda_{t-1}^2\mathbb{E}[\|\widetilde{\mathbf{V}}_{t-1}\|_F^2]+\mathbb{E}[\|\widetilde{\boldsymbol{\xi}}_t\|_F^2]\\
&=\lambda_{t-1}^2P_{t-1}+\mathbb{E}[\|\widetilde{\boldsymbol{\xi}}_t\|_F^2]. \tag*{(73)}
\end{aligned} P t = E [⟨ V t , V t ⟩ F ] = E [⟨ λ t − 1 V t − 1 + ξ t , λ t − 1 V t − 1 + ξ t ⟩ F ] = λ t − 1 2 E [⟨ V t − 1 , V t − 1 ⟩ F ] + 2 λ t − 1 E [⟨ V t − 1 , ξ t ⟩ F ] + E [⟨ ξ t , ξ t ⟩ F ] = λ t − 1 2 E [ ∥ V t − 1 ∥ F 2 ] + E [ ∥ ξ t ∥ F 2 ] = λ t − 1 2 P t − 1 + E [ ∥ ξ t ∥ F 2 ] . (73)
We defined the inner product ⟨ A , B ⟩ F : = Tr ( A T B ) = ∑ i = 1 d ∑ j = 1 d A i j B i j \langle\mathbf{A},\mathbf{B}\rangle_F:=\operatorname{Tr}(\mathbf{A}^{\mathsf{T}}\mathbf{B})=\sum_{i=1}^{d}\sum_{j=1}^{d}A_{ij}B_{ij} ⟨ A , B ⟩ F := Tr ( A T B ) = ∑ i = 1 d ∑ j = 1 d A ij B ij , and used E [ ⟨ V ~ t − 1 , ξ ~ t ⟩ F ] = 0 \mathbb{E}[\langle\widetilde{\mathbf{V}}_{t-1},\widetilde{\boldsymbol{\xi}}_t\rangle_F]=0 E [⟨ V t − 1 , ξ t ⟩ F ] = 0 . Indeed, V ~ t − 1 \widetilde{\mathbf{V}}_{t-1} V t − 1 is F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable, and from (72) we have:
E [ ⟨ V ~ t − 1 , ξ ~ t ⟩ F ] = E [ E [ Tr ( V ~ t − 1 T ξ ~ t ) ∣ F t − 1 ] ] = E [ Tr ( E [ V ~ t − 1 T ξ ~ t ∣ F t − 1 ] ) ] = E [ Tr ( V ~ t − 1 T E [ ξ ~ t ∣ F t − 1 ] ) ] = E [ Tr ( V ~ t − 1 T 0 ) ] = 0. \begin{aligned}
\mathbb{E}[\langle\widetilde{\mathbf{V}}_{t-1},\widetilde{\boldsymbol{\xi}}_t\rangle_F]
&=\mathbb{E}\left[\mathbb{E}\left[\operatorname{Tr}(\widetilde{\mathbf{V}}_{t-1}^{\mathsf{T}}\widetilde{\boldsymbol{\xi}}_t)\mid\mathcal{F}_{t-1}\right]\right]\\
&=\mathbb{E}\left[\operatorname{Tr}\left(\mathbb{E}[\widetilde{\mathbf{V}}_{t-1}^{\mathsf{T}}\widetilde{\boldsymbol{\xi}}_t\mid\mathcal{F}_{t-1}]\right)\right]\\
&=\mathbb{E}\left[\operatorname{Tr}\left(\widetilde{\mathbf{V}}_{t-1}^{\mathsf{T}}\mathbb{E}[\widetilde{\boldsymbol{\xi}}_t\mid\mathcal{F}_{t-1}]\right)\right]\\
&=\mathbb{E}[\operatorname{Tr}(\widetilde{\mathbf{V}}_{t-1}^{\mathsf{T}}0)]\\
&=0.
\end{aligned} E [⟨ V t − 1 , ξ t ⟩ F ] = E [ E [ Tr ( V t − 1 T ξ t ) ∣ F t − 1 ] ] = E [ Tr ( E [ V t − 1 T ξ t ∣ F t − 1 ] ) ] = E [ Tr ( V t − 1 T E [ ξ t ∣ F t − 1 ] ) ] = E [ Tr ( V t − 1 T 0 )] = 0.
Finally, from (73) , we have
P t = λ t − 1 2 P t − 1 + E [ ∥ ξ ~ t ∥ F 2 ] . (74) P_t=\lambda_{t-1}^2P_{t-1}+\mathbb{E}[\|\widetilde{\boldsymbol{\xi}}_t\|_F^2]. \tag*{(74)} P t = λ t − 1 2 P t − 1 + E [ ∥ ξ t ∥ F 2 ] . (74)
Bounding E [ ∥ ξ ~ t ∥ F 2 ] \mathbb{E}[\|\widetilde{\boldsymbol{\xi}}_t\|_F^2] E [ ∥ ξ t ∥ F 2 ] . For every s ≥ 1 s\ge1 s ≥ 1 and 1 ≤ i ≤ n b ( s ) 1\le i\le n_b(s) 1 ≤ i ≤ n b ( s ) , define
z i ( s ) : = x i ( s ) − μ . (75) z_i^{(s)}:=x_i^{(s)}-\mu. \tag*{(75)} z i ( s ) := x i ( s ) − μ . (75)
Fix t ≥ 2 t \ge2 t ≥ 2 . For every 1 ≤ s ≤ t − 2 1 \le s \le t-2 1 ≤ s ≤ t − 2 , the random vector z i ( s ) z_i^{(s)} z i ( s ) is F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable, and so is x j ( 0 ) − μ x_j^{(0)}-\mu x j ( 0 ) − μ for 1 ≤ j ≤ n r 1 \le j \le n_r 1 ≤ j ≤ n r . Hence their outer products equal their conditional expectations given F t − 1 \mathcal{F}_{t-1} F t − 1 , e.g.,
E [ z i ( s ) ( z i ( s ) ) T ∣ F t − 1 ] = z i ( s ) ( z i ( s ) ) T . \mathbb{E}\left[z_i^{(s)}\left(z_i^{(s)}\right)^{\mathsf{T}} \mid\mathcal{F}_{t-1}\right]=z_i^{(s)}\left(z_i^{(s)}\right)^{\mathsf{T}}. E [ z i ( s ) ( z i ( s ) ) T ∣ F t − 1 ] = z i ( s ) ( z i ( s ) ) T .
From (54) and (72), these terms cancel, and we get
ξ ~ t = 1 N t ∑ i = 1 n b ( t − 1 ) ( z i ( t − 1 ) ( z i ( t − 1 ) ) T − E [ z i ( t − 1 ) ( z i ( t − 1 ) ) T ∣ F t − 1 ] ) . (76) \widetilde{\xi}_t=\frac{1}{N_t}\sum_{i=1}^{n_b(t-1)}\left(z_i^{(t-1)}\left(z_i^{(t-1)}\right)^{\mathsf{T}}-\mathbb{E}\left[z_i^{(t-1)}\left(z_i^{(t-1)}\right)^{\mathsf{T}}\mid\mathcal{F}_{t-1}\right]\right). \tag*{(76)} ξ t = N t 1 i = 1 ∑ n b ( t − 1 ) ( z i ( t − 1 ) ( z i ( t − 1 ) ) T − E [ z i ( t − 1 ) ( z i ( t − 1 ) ) T ∣ F t − 1 ] ) . (76)
Let us define
Z i ( t − 1 ) : = z i ( t − 1 ) ( z i ( t − 1 ) ) T − E [ z i ( t − 1 ) ( z i ( t − 1 ) ) T ∣ F t − 1 ] . (77) Z_i^{(t-1)}:=z_i^{(t-1)}\left(z_i^{(t-1)}\right)^{\mathsf{T}}-\mathbb{E}\left[z_i^{(t-1)}\left(z_i^{(t-1)}\right)^{\mathsf{T}}\mid\mathcal{F}_{t-1}\right]. \tag*{(77)} Z i ( t − 1 ) := z i ( t − 1 ) ( z i ( t − 1 ) ) T − E [ z i ( t − 1 ) ( z i ( t − 1 ) ) T ∣ F t − 1 ] . (77)
Finally, from (78) and (79), we have
ξ ~ t = 1 N t ∑ i = 1 n b ( t − 1 ) Z i ( t − 1 ) . (78) \widetilde{\xi}_t=\frac{1}{N_t}\sum_{i=1}^{n_b(t-1)}Z_i^{(t-1)}. \tag*{(78)} ξ t = N t 1 i = 1 ∑ n b ( t − 1 ) Z i ( t − 1 ) . (78)
Conditionally on F t − 1 \mathcal{F}_{t-1} F t − 1 , the samples x i ( t − 1 ) x_i^{(t-1)} x i ( t − 1 ) are i.i.d. Thus, from (77) and (79), the random matrices Z i ( t − 1 ) Z_i^{(t-1)} Z i ( t − 1 ) are also i.i.d. as a deterministic transformation of x i ( t − 1 ) x_i^{(t-1)} x i ( t − 1 ) . That is,
E [ ∥ ξ ~ t ∥ F 2 ∣ F t − 1 ] = n b ( t − 1 ) N t 2 E [ ∥ Z i ( t − 1 ) ∥ F 2 ∣ F t − 1 ] ≤ β t − 1 E [ ∥ z i ( t − 1 ) ∥ 2 4 ∣ F t − 1 ] = β t − 1 E [ ∥ x i ( t − 1 ) − μ ∥ 2 4 ∣ F t − 1 ] ≤ β t − 1 ( 3 α t − 1 Tr ( S ^ t − 1 ) 2 + C Σ ) . \begin{aligned}
\mathbb{E}\left[\left\|\widetilde{\xi}_t\right\|_{F}^{2}\mid\mathcal{F}_{t-1}\right]=\frac{n_b(t-1)}{N_t^2}\mathbb{E}\left[\left\|Z_i^{(t-1)}\right\|_{F}^{2}\mid\mathcal{F}_{t-1}\right] \\
&\le\beta_{t-1}\mathbb{E}\left[\left\|z_i^{(t-1)}\right\|_{2}^{4}\mid\mathcal{F}_{t-1}\right] \\
&=\beta_{t-1}\mathbb{E}\left[\left\|x_i^{(t-1)}-\mu\right\|_{2}^{4}\mid\mathcal{F}_{t-1}\right] \\
&\le\beta_{t-1}\left(3\alpha_{t-1}\operatorname{Tr}(\widehat{S}_{t-1})^{2}+C_{\Sigma}\right).
\end{aligned} E [ ξ t F 2 ∣ F t − 1 ] = N t 2 n b ( t − 1 ) E [ Z i ( t − 1 ) F 2 ∣ F t − 1 ] ≤ β t − 1 E [ z i ( t − 1 ) 2 4 ∣ F t − 1 ] = β t − 1 E [ x i ( t − 1 ) − μ 2 4 ∣ F t − 1 ] ≤ β t − 1 ( 3 α t − 1 Tr ( S t − 1 ) 2 + C Σ ) .
We used the definition of β t \beta_t β t from (18) and Lemma A.21 with G = F t − 1 \mathcal{G}=\mathcal{F}_{t-1} G = F t − 1 and X = z i ( t − 1 ) ( z i ( t − 1 ) ) T X=z_i^{(t-1)}(z_i^{(t-1)})^{\mathsf{T}} X = z i ( t − 1 ) ( z i ( t − 1 ) ) T , for which
∥ X ∥ F 2 = ∥ z i ( t − 1 ) ( z i ( t − 1 ) ) T ∥ F 2 = ∥ z i ( t − 1 ) ∥ 2 4 , \|X\|_{F}^{2}=\left\|z_i^{(t-1)}(z_i^{(t-1)})^{\mathsf{T}}\right\|_{F}^{2}=\left\|z_i^{(t-1)}\right\|_{2}^{4}, ∥ X ∥ F 2 = z i ( t − 1 ) ( z i ( t − 1 ) ) T F 2 = z i ( t − 1 ) 2 4 ,
so that its hypothesis E [ ∥ X ∥ F 2 ] < ∞ \mathbb{E}[\|X\|_{F}^{2}]<\infty E [ ∥ X ∥ F 2 ] < ∞ holds: taking expectations in Lemma A.16 (using the tower property) and using Lemma A.19, E [ ∥ z i ( t − 1 ) ∥ 2 4 ] ≤ 3 α t − 1 E [ Tr ( S ^ t − 1 ) 2 ] + C Σ < ∞ \mathbb{E}[\|z_i^{(t-1)}\|_{2}^{4}]\le3\alpha_{t-1}\mathbb{E}[\operatorname{Tr}(\widehat{S}_{t-1})^{2}]+C_{\Sigma}<\infty E [ ∥ z i ( t − 1 ) ∥ 2 4 ] ≤ 3 α t − 1 E [ Tr ( S t − 1 ) 2 ] + C Σ < ∞ . We also used (77) and Lemma A.16 for the conditional fourth moment. This gives
E [ ∥ ξ ~ t ∥ F 2 ∣ F t − 1 ] ≤ β t − 1 ( 3 α t − 1 Tr ( S ^ t − 1 ) 2 + C Σ ) . (79) \mathbb{E}\left[\left\|\widetilde{\xi}_t\right\|_{F}^{2}\mid\mathcal{F}_{t-1}\right]\le\beta_{t-1}\left(3\alpha_{t-1}\operatorname{Tr}(\widehat{S}_{t-1})^{2}+C_{\Sigma}\right). \tag*{(79)} E [ ξ t F 2 ∣ F t − 1 ] ≤ β t − 1 ( 3 α t − 1 Tr ( S t − 1 ) 2 + C Σ ) . (79)
Using the tower property in (81), we have
E [ ∥ ξ ~ t ∥ F 2 ] = E [ E [ ∥ ξ ~ t ∥ F 2 ∣ F t − 1 ] ] ≤ E [ β t − 1 ( 3 α t − 1 Tr ( S ^ t − 1 ) 2 + C Σ ) ] = β t − 1 ( 3 α t − 1 E [ Tr ( S ^ t − 1 ) 2 ] + C Σ ) ≤ ( 3 B ∗ + C Σ ) β t − 1 . (80) \begin{aligned}
\mathbb{E}\left[\left\|\widetilde{\xi}_t\right\|_{F}^{2}\right]=\mathbb{E}\left[\mathbb{E}\left[\left\|\widetilde{\xi}_t\right\|_{F}^{2}\mid\mathcal{F}_{t-1}\right]\right] \\
&\le\mathbb{E}\left[\beta_{t-1}\left(3\alpha_{t-1}\operatorname{Tr}(\widehat{S}_{t-1})^{2}+C_{\Sigma}\right)\right] \\
&=\beta_{t-1}\left(3\alpha_{t-1}\mathbb{E}\left[\operatorname{Tr}(\widehat{S}_{t-1})^{2}\right]+C_{\Sigma}\right) \\
&\le(3B^{*}+C_{\Sigma})\beta_{t-1}. \tag*{(80)}
\end{aligned} E [ ξ t F 2 ] = E [ E [ ξ t F 2 ∣ F t − 1 ] ] ≤ E [ β t − 1 ( 3 α t − 1 Tr ( S t − 1 ) 2 + C Σ ) ] = β t − 1 ( 3 α t − 1 E [ Tr ( S t − 1 ) 2 ] + C Σ ) ≤ ( 3 B ∗ + C Σ ) β t − 1 . (80)
We used α t − 1 ≤ 1 \alpha_{t-1}\le1 α t − 1 ≤ 1 and the uniform bound E [ Tr ( S ^ t − 1 ) 2 ] ≤ B ∗ \mathbb{E}[\operatorname{Tr}(\widehat{S}_{t-1})^{2}]\le B^{*} E [ Tr ( S t − 1 ) 2 ] ≤ B ∗ from Lemma A.19. Finally, from (82) we have:
E [ ∥ ξ ~ t ∥ F 2 ] ≤ D β t − 1 , D : = 3 B ∗ + C Σ < ∞ . (81) \mathbb{E}\left[\left\|\widetilde{\xi}_t\right\|_{F}^{2}\right]\le D\beta_{t-1},\qquad D:=3B^{*}+C_{\Sigma}<\infty. \tag*{(81)} E [ ξ t F 2 ] ≤ D β t − 1 , D := 3 B ∗ + C Σ < ∞. (81)
Step 2 (P t → 0 P_t\to0 P t → 0 ). From (76) and (83), we have P t ≤ λ t − 1 2 P t − 1 + D β t − 1 P_t\le\lambda_{t-1}^{2}P_{t-1}+D\beta_{t-1} P t ≤ λ t − 1 2 P t − 1 + D β t − 1 . Unrolling by Lemma A.17 with r t − 1 = λ t − 1 2 ≤ 1 r_{t-1}=\lambda_{t-1}^{2}\le1 r t − 1 = λ t − 1 2 ≤ 1 (Lemma A.9) and q t − 1 = D β t − 1 q_{t-1}=D\beta_{t-1} q t − 1 = D β t − 1 ,
P t ≤ ( ∏ k = 1 t − 1 λ k 2 ) P 1 + D ∑ s = 1 t − 1 ( ∏ k = s + 1 t − 1 λ k 2 ) β s . (82) P_t\le\left(\prod_{k=1}^{t-1}\lambda_k^2\right)P_1+D\sum_{s=1}^{t-1}\left(\prod_{k=s+1}^{t-1}\lambda_k^2\right)\beta_s. \tag*{(82)} P t ≤ ( k = 1 ∏ t − 1 λ k 2 ) P 1 + D s = 1 ∑ t − 1 ( k = s + 1 ∏ t − 1 λ k 2 ) β s . (82)
Term (I). By the hypothesis ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ \sum_{t=1}^{\infty}(1-\alpha_t)(1-\rho_t)=\infty ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ and Lemma A.20,
∏ k = 1 t − 1 λ k 2 = exp ( ∑ k = 1 t − 1 ln λ k 2 ) ≤ exp ( − ∑ k = 1 t − 1 ( 1 − λ k 2 ) ) → t → ∞ 0. (83) \prod_{k=1}^{t-1}\lambda_k^2=\exp\left(\sum_{k=1}^{t-1}\ln\lambda_k^2\right)\leq\exp\left(-\sum_{k=1}^{t-1}(1-\lambda_k^2)\right)\xrightarrow{t\to\infty}0. \tag*{(83)} k = 1 ∏ t − 1 λ k 2 = exp ( k = 1 ∑ t − 1 ln λ k 2 ) ≤ exp ( − k = 1 ∑ t − 1 ( 1 − λ k 2 ) ) t → ∞ 0. (83)
We used ln ( x ) ≤ x − 1 \ln(x)\leq x-1 ln ( x ) ≤ x − 1 with x = λ k 2 ≥ 0 x=\lambda_k^2\geq0 x = λ k 2 ≥ 0 (indeed, 0 ≤ λ k ≤ 1 0\leq\lambda_k\leq1 0 ≤ λ k ≤ 1 from Lemma A.9)
Term (II). Fix ε > 0 \varepsilon>0 ε > 0 . By Lemma A.5, choose s 0 ≥ 2 s_0\geq2 s 0 ≥ 2 with
∑ s = s 0 ∞ β s < ε 2 , (84) \sum_{s=s_0}^{\infty}\beta_s<\frac{\varepsilon}{2}, \tag*{(84)} s = s 0 ∑ ∞ β s < 2 ε , (84)
and from (85), choose s 1 ≥ 1 s_1\geq1 s 1 ≥ 1 such that for every t ≥ s 1 t\geq s_1 t ≥ s 1 ,
∏ k = s 0 t − 1 λ k 2 ≤ 1 ∑ s = 1 s 0 − 1 β s ε 2 , (85) \prod_{k=s_0}^{t-1}\lambda_k^2\leq\frac{1}{\sum_{s=1}^{s_0-1}\beta_s}\frac{\varepsilon}{2}, \tag*{(85)} k = s 0 ∏ t − 1 λ k 2 ≤ ∑ s = 1 s 0 − 1 β s 1 2 ε , (85)
where we used ∏ k = s 0 t − 1 λ k 2 = ∏ k = 1 t − 1 λ k 2 ∏ k = 1 s 0 − 1 λ k 2 → t → ∞ 0 \prod_{k=s_0}^{t-1}\lambda_k^2=\frac{\prod_{k=1}^{t-1}\lambda_k^2}{\prod_{k=1}^{s_0-1}\lambda_k^2}\xrightarrow{t\to\infty}0 ∏ k = s 0 t − 1 λ k 2 = ∏ k = 1 s 0 − 1 λ k 2 ∏ k = 1 t − 1 λ k 2 t → ∞ 0 .
From (84), (86) and (87), for all t ≥ max ( s 0 + 2 , s 1 ) t\geq\max(s_0+2,s_1) t ≥ max ( s 0 + 2 , s 1 ) :
I I = ∑ s = 1 s 0 − 1 ( ∏ k = s + 1 t − 1 λ k 2 ) β s + ∑ s = s 0 t − 1 ( ∏ k = s + 1 t − 1 λ k 2 ) β s ≤ ∏ k = s 0 t − 1 λ k 2 ⋅ ∑ s = 1 s 0 − 1 β s + ∑ s = s 0 t − 1 β s ≤ ∏ k = s 0 t − 1 λ k 2 ⋅ ∑ s = 1 s 0 − 1 β s + ∑ s = s 0 ∞ β s ≤ ε 2 + ε 2 = ε , (86) \begin{aligned}
\mathrm{II}
&=\sum_{s=1}^{s_0-1}\left(\prod_{k=s+1}^{t-1}\lambda_k^2\right)\beta_s+\sum_{s=s_0}^{t-1}\left(\prod_{k=s+1}^{t-1}\lambda_k^2\right)\beta_s\\
&\leq\prod_{k=s_0}^{t-1}\lambda_k^2\cdot\sum_{s=1}^{s_0-1}\beta_s+\sum_{s=s_0}^{t-1}\beta_s\\
&\leq\prod_{k=s_0}^{t-1}\lambda_k^2\cdot\sum_{s=1}^{s_0-1}\beta_s+\sum_{s=s_0}^{\infty}\beta_s\\
&\leq\frac{\varepsilon}{2}+\frac{\varepsilon}{2}\\
&=\varepsilon, \tag*{(86)}
\end{aligned} II = s = 1 ∑ s 0 − 1 ( k = s + 1 ∏ t − 1 λ k 2 ) β s + s = s 0 ∑ t − 1 ( k = s + 1 ∏ t − 1 λ k 2 ) β s ≤ k = s 0 ∏ t − 1 λ k 2 ⋅ s = 1 ∑ s 0 − 1 β s + s = s 0 ∑ t − 1 β s ≤ k = s 0 ∏ t − 1 λ k 2 ⋅ s = 1 ∑ s 0 − 1 β s + s = s 0 ∑ ∞ β s ≤ 2 ε + 2 ε = ε , (86)
where we used λ k 2 ≤ 1 \lambda_k^2\leq1 λ k 2 ≤ 1 for all k ≥ 1 k\geq1 k ≥ 1 (Lemma A.9). Finally, (88) holds for every ε > 0 \varepsilon>0 ε > 0 , and thus (II) → 0 \to0 → 0 . Together with (I) → 0 \to0 → 0 from (85), we have P t → 0 P_t\to0 P t → 0 (indeed, from (84), P t ≤ P_t\leq P t ≤ (I) ⋅ P 1 + D ⋅ \cdot P_1+D\cdot ⋅ P 1 + D ⋅ (II) → 0 \to0 → 0 ). That is,
P t → t → ∞ 0. (87) P_t\xrightarrow{t\to\infty}0. \tag*{(87)} P t t → ∞ 0. (87)
Step 3 (from S ^ t \hat{S}_t S ^ t to Σ ( t ) \Sigma^{(t)} Σ ( t ) ). The triangle inequality gives
E [ ∥ Σ ( t ) − Σ ∥ F ] ≤ E [ ∥ S ^ t − Σ ∥ F ] + E [ ∥ Σ ( t ) − S ^ t ∥ F ] = E [ ∥ S ^ t − Σ ∥ F ] + E [ ∥ δ ( t ) ( δ ( t ) ) ⊤ ∥ F ] ≤ E [ ∥ S ^ t − Σ ∥ F 2 ] + E [ ∥ δ ( t ) ∥ 2 2 ] = P t + Tr ( C t ) . (88) \begin{aligned}
\mathbb{E}[\|\Sigma^{(t)}-\Sigma\|_F]
&\leq\mathbb{E}[\|\hat{S}_t-\Sigma\|_F]+\mathbb{E}[\|\Sigma^{(t)}-\hat{S}_t\|_F]\\
&=\mathbb{E}[\|\hat{S}_t-\Sigma\|_F]+\mathbb{E}[\|\delta^{(t)}(\delta^{(t)})^\top\|_F]\\
&\leq\sqrt{\mathbb{E}[\|\hat{S}_t-\Sigma\|_F^2]}+\mathbb{E}[\|\delta^{(t)}\|_2^2]\\
&=\sqrt{P_t}+\operatorname{Tr}(C_t). \tag*{(88)}
\end{aligned} E [ ∥ Σ ( t ) − Σ ∥ F ] ≤ E [ ∥ S ^ t − Σ ∥ F ] + E [ ∥ Σ ( t ) − S ^ t ∥ F ] = E [ ∥ S ^ t − Σ ∥ F ] + E [ ∥ δ ( t ) ( δ ( t ) ) ⊤ ∥ F ] ≤ E [ ∥ S ^ t − Σ ∥ F 2 ] + E [ ∥ δ ( t ) ∥ 2 2 ] = P t + Tr ( C t ) . (88)
We used Lemma A.14(ii) and the identity ∥ x x ⊤ ∥ F = ∥ x ∥ 2 2 \|xx^\top\|_F=\|x\|_2^2 ∥ x x ⊤ ∥ F = ∥ x ∥ 2 2 . Additionally, ( ⋅ ) 2 (\cdot)^2 ( ⋅ ) 2 is convex, and Jensen’s inequality with X = ∥ S ^ t − Σ ∥ F X=\|\hat{S}_t-\Sigma\|_F X = ∥ S ^ t − Σ ∥ F gives
E [ X ] 2 ≤ E [ X 2 ] ⇒ E [ X ] ≤ E [ X 2 ] , \mathbb{E}[X]^2\leq\mathbb{E}[X^2]\quad\Rightarrow\quad\mathbb{E}[X]\leq\sqrt{\mathbb{E}[X^2]}, E [ X ] 2 ≤ E [ X 2 ] ⇒ E [ X ] ≤ E [ X 2 ] ,
where X ≥ 0 X\geq0 X ≥ 0 a.s. Then, we used P t = E [ ∥ S ^ t − Σ ∥ F 2 ] P_t=\mathbb{E}[\|\hat{S}_t-\Sigma\|_F^2] P t = E [ ∥ S ^ t − Σ ∥ F 2 ] and E [ ∥ δ ( t ) ∥ 2 2 ] = Tr ( C t ) \mathbb{E}[\|\delta^{(t)}\|_2^2]=\operatorname{Tr}(C_t) E [ ∥ δ ( t ) ∥ 2 2 ] = Tr ( C t ) , which follows from (11). That is, from (90), we have shown that
E [ ∥ Σ ( t ) − Σ ∥ F ] ≤ P t + Tr ( C t ) . (89) \mathbb{E}[\|\Sigma^{(t)}-\Sigma\|_F]\leq\sqrt{P_t}+\operatorname{Tr}(C_t). \tag*{(89)} E [ ∥ Σ ( t ) − Σ ∥ F ] ≤ P t + Tr ( C t ) . (89)
By (89), we have P t → 0 P_t \to0 P t → 0 . Additionally, from our hypothesis ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ \sum_{t=1}^{\infty}(1-\alpha_t)(1-\rho_t)=\infty ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ and Theorem A.13 , we have C t → 0 C_t \to0 C t → 0 . Hence, Tr ( C t ) → 0 \operatorname{Tr}(C_t) \to0 Tr ( C t ) → 0 (Lemma A.10 ). Thus, both terms of (89) vanish, so E [ ∥ Σ ( t ) − Σ ∥ F ] → 0 \mathbb{E}[\lVert\Sigma^{(t)}-\Sigma\rVert_F]\to0 E [∥ Σ ( t ) − Σ ∥ F ] → 0 .
Step 4 (necessity). Conversely, suppose
∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) < ∞ . (90) \sum_{t=1}^{\infty}(1-\alpha_t)(1-\rho_t)<\infty. \tag*{(90)} t = 1 ∑ ∞ ( 1 − α t ) ( 1 − ρ t ) < ∞. (90)
Since ∥ ⋅ ∥ F \lVert\cdot\rVert_F ∥ ⋅ ∥ F is convex, Jensen’s inequality and Theorem 3.1 (b) give
E [ ∥ Σ ( t ) − Σ ∥ F ] ≥ ∥ E [ Σ ( t ) ] − Σ ∥ F = ∥ − C t ∥ F = ∥ C t ∥ F , (91) \mathbb{E}[\lVert\Sigma^{(t)}-\Sigma\rVert_F]\geq\lVert\mathbb{E}[\Sigma^{(t)}]-\Sigma\rVert_F=\lVert-C_t\rVert_F=\lVert C_t\rVert_F, \tag*{(91)} E [∥ Σ ( t ) − Σ ∥ F ] ≥ ∥ E [ Σ ( t ) ] − Σ ∥ F = ∥ − C t ∥ F = ∥ C t ∥ F , (91)
where we used Theorem 3.1 (b). By (90) and Theorem A.13 , C t ↛ 0 C_t\not\to0 C t → 0 . Hence, ∥ C t ∥ F ↛ 0 \lVert C_t\rVert_F\not\to0 ∥ C t ∥ F → 0 (Remark after Lemma A.10 ). From (91) , we have E [ ∥ Σ ( t ) − Σ ∥ F ] ↛ 0 \mathbb{E}[\lVert\Sigma^{(t)}-\Sigma\rVert_F]\not\to0 E [∥ Σ ( t ) − Σ ∥ F ] → 0 . This establishes the stated equivalence.
Convergence of the fitted distribution to the true distribution
We now characterize when the fitted distribution D t D_t D t converges to the true distribution D r e a l D_{\mathrm{real}} D real (see (5) ) in the 2-Wasserstein distance W 2 W_2 W 2 , a standard metric between probability distributions (Villani, 2009).
Definition 2 (2-Wasserstein distance). For probability distributions P , Q P,Q P , Q on R d \mathbb{R}^d R d with finite second moments,
W 2 2 ( P , Q ) : = inf π ∈ Π ( P , Q ) E ( x , y ) ∼ π [ ∥ x − y ∥ 2 ] . W_2^2(P,Q):=\inf_{\pi\in\Pi(P,Q)}\mathbb{E}_{(x,y)\sim\pi}\left[\lVert x-y\rVert^2\right]. W 2 2 ( P , Q ) := π ∈ Π ( P , Q ) inf E ( x , y ) ∼ π [ ∥ x − y ∥ 2 ] . where Π ( P , Q ) \Pi(P,Q) Π ( P , Q ) is the set of couplings of P P P and Q Q Q , i.e., joint distributions with marginals P P P and Q Q Q .
First, we prove a bound on a metric between two symmetric positive-semi-definite matrices. It is used in the proof of Theorem A.24 .
Lemma A.23 (Bures–Frobenius bound). Let A , B ∈ R d × d A,B\in\mathbb{R}^{d\times d} A , B ∈ R d × d be symmetric with A ⪰ 0 A\succeq0 A ⪰ 0 and B ⪰ 0 B\succeq0 B ⪰ 0 . Define the Bures metric B \mathcal{B} B as
B 2 ( A , B ) = Tr ( A ) + Tr ( B ) − 2 Tr ( ( A 1 / 2 B A 1 / 2 ) 1 / 2 ) . (92) \mathcal{B}^2(A,B)=\operatorname{Tr}(A)+\operatorname{Tr}(B)-2\operatorname{Tr}\left((A^{1/2}BA^{1/2})^{1/2}\right). \tag*{(92)} B 2 ( A , B ) = Tr ( A ) + Tr ( B ) − 2 Tr ( ( A 1/2 B A 1/2 ) 1/2 ) . (92) Then
B 2 ( A , B ) ≤ ∥ A 1 / 2 − B 1 / 2 ∥ F 2 ≤ ∥ A − B ∥ ∗ ≤ d ∥ A − B ∥ F , (93) \mathcal{B}^2(A,B)\leq\lVert A^{1/2}-B^{1/2}\rVert_F^2\leq\lVert A-B\rVert_*\leq\sqrt{d}\lVert A-B\rVert_F, \tag*{(93)} B 2 ( A , B ) ≤ ∥ A 1/2 − B 1/2 ∥ F 2 ≤ ∥ A − B ∥ ∗ ≤ d ∥ A − B ∥ F , (93) where ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_* ∥ ⋅ ∥ ∗ denotes the nuclear (trace) norm, i.e. the sum of the singular values.
Proof. First inequality. Expanding the Frobenius norm, we get
∥ A 1 / 2 − B 1 / 2 ∥ F 2 = Tr ( ( A 1 / 2 − B 1 / 2 ) ( A 1 / 2 − B 1 / 2 ) T ) = Tr ( A − A 1 / 2 B 1 / 2 − B 1 / 2 A 1 / 2 + B ) = Tr ( A ) + Tr ( B ) − 2 Tr ( A 1 / 2 B 1 / 2 ) . (94) \begin{aligned}
\lVert A^{1/2}-B^{1/2}\rVert_F^2=\operatorname{Tr}\left((A^{1/2}-B^{1/2})(A^{1/2}-B^{1/2})^{\mathsf{T}}\right) \\
&=\operatorname{Tr}\left(A-A^{1/2}B^{1/2}-B^{1/2}A^{1/2}+B\right) \\
&=\operatorname{Tr}(A)+\operatorname{Tr}(B)-2\operatorname{Tr}(A^{1/2}B^{1/2}). \tag*{(94)}
\end{aligned} ∥ A 1/2 − B 1/2 ∥ F 2 = Tr ( ( A 1/2 − B 1/2 ) ( A 1/2 − B 1/2 ) T ) = Tr ( A − A 1/2 B 1/2 − B 1/2 A 1/2 + B ) = Tr ( A ) + Tr ( B ) − 2 Tr ( A 1/2 B 1/2 ) . (94)
Indeed, for any C ⪰ 0 C\succeq0 C ⪰ 0 , its symmetric square root C 1 / 2 C^{1/2} C 1/2 satisfies C 1 / 2 C 1 / 2 = C C^{1/2}C^{1/2}=C C 1/2 C 1/2 = C . Finally, we have Tr ( A B ) = Tr ( B A ) \operatorname{Tr}(AB)=\operatorname{Tr}(BA) Tr ( A B ) = Tr ( B A ) . Using the definition of B 2 \mathcal{B}^2 B 2 in (92) and (94) , the claim is equivalent to
Tr ( A 1 / 2 B 1 / 2 ) ≤ Tr ( ( A 1 / 2 B A 1 / 2 ) 1 / 2 ) . (95) \operatorname{Tr}(A^{1/2}B^{1/2})\leq\operatorname{Tr}\left((A^{1/2}BA^{1/2})^{1/2}\right). \tag*{(95)} Tr ( A 1/2 B 1/2 ) ≤ Tr ( ( A 1/2 B A 1/2 ) 1/2 ) . (95)
Set M : = B 1 / 2 A 1 / 2 M:=B^{1/2}A^{1/2} M := B 1/2 A 1/2 . Then
M T M = A 1 / 2 B 1 / 2 B 1 / 2 A 1 / 2 = A 1 / 2 B A 1 / 2 . (96) M^{\mathsf{T}}M=A^{1/2}B^{1/2}B^{1/2}A^{1/2}=A^{1/2}BA^{1/2}. \tag*{(96)} M T M = A 1/2 B 1/2 B 1/2 A 1/2 = A 1/2 B A 1/2 . (96)
Since the singular values of M M M are the eigenvalues of the positive-semi-definite matrix ( M T M ) 1 / 2 (M^{\mathsf{T}}M)^{1/2} ( M T M ) 1/2 , the nuclear norm satisfies, by (96) ,
∥ M ∥ ∗ = Tr ( ( M T M ) 1 / 2 ) = Tr ( ( A 1 / 2 B A 1 / 2 ) 1 / 2 ) . (97) \lVert M\rVert_*=\operatorname{Tr}\left((M^{\mathsf{T}}M)^{1/2}\right)=\operatorname{Tr}\left((A^{1/2}BA^{1/2})^{1/2}\right). \tag*{(97)} ∥ M ∥ ∗ = Tr ( ( M T M ) 1/2 ) = Tr ( ( A 1/2 B A 1/2 ) 1/2 ) . (97)
By cyclicity,
Tr ( M ) = Tr ( A 1 / 2 B 1 / 2 ) . (98) \operatorname{Tr}(M)=\operatorname{Tr}(A^{1/2}B^{1/2}). \tag*{(98)} Tr ( M ) = Tr ( A 1/2 B 1/2 ) . (98)
Using (95) , (97) , (98) , the claim is Tr ( M ) ≤ ∥ M ∥ ∗ \operatorname{Tr}(M) \le\lVert M\rVert_\ast Tr ( M ) ≤ ∥ M ∥ ∗ , which follows immediately from von Neumann’s trace inequality [13 ] .
Second inequality. This is the Powers–Størmer inequality [18 ] ; see also Santoro & Panaretos [20 ] .
Third inequality. Let s 1 , … , s d ≥ 0 s_1,\ldots,s_d \ge0 s 1 , … , s d ≥ 0 be the singular values of A − B A-B A − B , so that ∥ A − B ∥ ∗ = ∑ i = 1 d s i \lVert A-B\rVert_\ast=\sum_{i=1}^{d}s_i ∥ A − B ∥ ∗ = ∑ i = 1 d s i and ∥ A − B ∥ F = ( ∑ i = 1 d s i 2 ) 1 / 2 \lVert A-B\rVert_F=\left(\sum_{i=1}^{d}s_i^2\right)^{1/2} ∥ A − B ∥ F = ( ∑ i = 1 d s i 2 ) 1/2 . Lemma A.3 gives
∥ A − B ∥ ∗ = ∑ i = 1 d s i ≤ d ( ∑ i = 1 d s i 2 ) 1 / 2 = d ∥ A − B ∥ F . \lVert A-B\rVert_\ast=\sum_{i=1}^{d}s_i\le\sqrt{d}\left(\sum_{i=1}^{d}s_i^2\right)^{1/2}=\sqrt{d}\lVert A-B\rVert_F. ∥ A − B ∥ ∗ = i = 1 ∑ d s i ≤ d ( i = 1 ∑ d s i 2 ) 1/2 = d ∥ A − B ∥ F .
We now characterize when the fitted distribution converges to the true distribution in the 2-Wasserstein distance.
Theorem A.24 (W 2 W_2 W 2 consistency). Under Assumption 1,
E [ W 2 2 ( D t , D r e a l ) ] → t → ∞ 0 ⟺ ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ . \mathbb{E}\left[W_2^2(\mathcal{D}_t,\mathcal{D}_{\mathrm{real}})\right]\xrightarrow{t\to\infty}0\quad\Longleftrightarrow\quad\sum_{t=1}^{\infty}(1-\alpha_t)(1-\rho_t)=\infty. E [ W 2 2 ( D t , D real ) ] t → ∞ 0 ⟺ t = 1 ∑ ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞.
Proof. Recall Definition 2. For Gaussians, it admits the closed form [6 , 9 ]
W 2 2 ( N ( μ 1 , Σ 1 ) , N ( μ 2 , Σ 2 ) ) = ∥ μ 1 − μ 2 ∥ 2 2 + B 2 ( Σ 1 , Σ 2 ) , (99) W_2^2(\mathcal{N}(\mu_1,\Sigma_1),\mathcal{N}(\mu_2,\Sigma_2))=\lVert\mu_1-\mu_2\rVert_2^2+\mathcal{B}^2(\Sigma_1,\Sigma_2), \tag*{(99)} W 2 2 ( N ( μ 1 , Σ 1 ) , N ( μ 2 , Σ 2 )) = ∥ μ 1 − μ 2 ∥ 2 2 + B 2 ( Σ 1 , Σ 2 ) , (99)
where B \mathcal{B} B is the Bures metric (92) . Applying (99) with μ 1 = μ ( t ) \mu_1=\mu^{(t)} μ 1 = μ ( t ) , Σ 1 = Σ ( t ) \Sigma_1=\Sigma^{(t)} Σ 1 = Σ ( t ) , μ 2 = μ \mu_2=\mu μ 2 = μ , Σ 2 = Σ \Sigma_2=\Sigma Σ 2 = Σ , and writing B t : = B ( Σ ( t ) , Σ ) \mathcal{B}_t:=\mathcal{B}(\Sigma^{(t)},\Sigma) B t := B ( Σ ( t ) , Σ ) , we obtain from (99) the pointwise identity
W 2 2 ( D t , D r e a l ) = ∥ μ ( t ) − μ ∥ 2 2 + B t 2 , (100) W_2^2(\mathcal{D}_t,\mathcal{D}_{\mathrm{real}})=\lVert\mu^{(t)}-\mu\rVert_2^2+\mathcal{B}_t^2, \tag*{(100)} W 2 2 ( D t , D real ) = ∥ μ ( t ) − μ ∥ 2 2 + B t 2 , (100)
in which both summands are nonnegative. Taking expectations in (100) and using E [ ∥ δ ( t ) ∥ 2 2 ] = Tr ( C t ) \mathbb{E}\left[\lVert\delta^{(t)}\rVert_2^2\right]=\operatorname{Tr}(C_t) E [ ∥ δ ( t ) ∥ 2 2 ] = Tr ( C t ) from (11) ,
E [ W 2 2 ( D t , D r e a l ) ] = E [ ∥ μ ( t ) − μ ∥ 2 2 ] + E [ B t 2 ] . (101) \mathbb{E}\left[W_2^2(\mathcal{D}_t,\mathcal{D}_{\mathrm{real}})\right]=\mathbb{E}\left[\lVert\mu^{(t)}-\mu\rVert_2^2\right]+\mathbb{E}\left[\mathcal{B}_t^2\right]. \tag*{(101)} E [ W 2 2 ( D t , D real ) ] = E [ ∥ μ ( t ) − μ ∥ 2 2 ] + E [ B t 2 ] . (101)
Sufficiency.* Suppose ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ \sum_{t=1}^{\infty}(1-\alpha_t)(1-\rho_t)=\infty ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ . By Lemma A.8 we have Σ ( t ) ⪰ 0 \Sigma^{(t)}\succeq0 Σ ( t ) ⪰ 0 , and Σ ≻ 0 \Sigma\succ0 Σ ≻ 0 by assumption, so Lemma A.23 applies and gives the pointwise bound
B t 2 ≤ d ∥ Σ ( t ) − Σ ∥ F . \mathcal{B}_t^2\le\sqrt{d}\lVert\Sigma^{(t)}-\Sigma\rVert_F. B t 2 ≤ d ∥ Σ ( t ) − Σ ∥ F .
Substituting into (101) ,
E [ W 2 2 ( D t , D r e a l ) ] ≤ E [ ∥ μ ( t ) − μ ∥ 2 2 ] + d E [ ∥ Σ ( t ) − Σ ∥ F ] . (102) \mathbb{E}\left[W_2^2(\mathcal{D}_t,\mathcal{D}_{\mathrm{real}})\right]\le\mathbb{E}\left[\lVert\mu^{(t)}-\mu\rVert_2^2\right]+\sqrt{d}\,\mathbb{E}\left[\lVert\Sigma^{(t)}-\Sigma\rVert_F\right]. \tag*{(102)} E [ W 2 2 ( D t , D real ) ] ≤ E [ ∥ μ ( t ) − μ ∥ 2 2 ] + d E [ ∥ Σ ( t ) − Σ ∥ F ] . (102)
By Theorem A.13 , the hypothesis gives E [ ∥ μ ( t ) − μ ∥ 2 2 ] → 0 \mathbb{E}\left[\lVert\mu^{(t)}-\mu\rVert_2^2\right]\to0 E [ ∥ μ ( t ) − μ ∥ 2 2 ] → 0 . By Theorem A.22 , the hypothesis also gives E [ ∥ Σ ( t ) − Σ ∥ F ] → 0 \mathbb{E}\left[\lVert\Sigma^{(t)}-\Sigma\rVert_F\right]\to0 E [ ∥ Σ ( t ) − Σ ∥ F ] → 0 . Both terms on the right-hand side of (102) therefore vanish, so E [ W 2 2 ( D t , D r e a l ) ] → 0 \mathbb{E}\left[W_2^2(\mathcal{D}_t,\mathcal{D}_{\mathrm{real}})\right]\to0 E [ W 2 2 ( D t , D real ) ] → 0 .
Necessity. Conversely, suppose E [ W 2 2 ( D t , D r e a l ) ] → 0 \mathbb{E}\left[W_2^2(\mathcal{D}_t,\mathcal{D}_{\mathrm{real}})\right]\to0 E [ W 2 2 ( D t , D real ) ] → 0 . Since E [ B t 2 ] ≥ 0 \mathbb{E}\left[\mathcal{B}_t^2\right]\ge0 E [ B t 2 ] ≥ 0 , identity (101) gives
0 ≤ E [ ∥ μ ( t ) − μ ∥ 2 2 ] ≤ E [ W 2 2 ( D t , D r e a l ) ] → t → ∞ 0. 0\le\mathbb{E}\left[\lVert\mu^{(t)}-\mu\rVert_2^2\right]\le\mathbb{E}\left[W_2^2(\mathcal{D}_t,\mathcal{D}_{\mathrm{real}})\right]\xrightarrow{t\to\infty}0. 0 ≤ E [ ∥ μ ( t ) − μ ∥ 2 2 ] ≤ E [ W 2 2 ( D t , D real ) ] t → ∞ 0.
Thus, E [ ∥ μ ( t ) − μ ∥ 2 2 ] → 0 \mathbb{E}\left[\lVert\mu^{(t)}-\mu\rVert_2^2\right]\to0 E [ ∥ μ ( t ) − μ ∥ 2 2 ] → 0 . Using Theorem A.13 and Lemma A.10 , we have ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ \sum_{t=1}^{\infty}(1-\alpha_t)(1-\rho_t)=\infty ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ . This establishes the stated equivalence.
Proof of Theorem 3.3
Theorem 3.3 characterizes in closed form when this iterative estimation process converges to the true distribution. We now assemble it from Lemma A.10 and Theorems A.13, A.22, and A.24.
Proof. It suffices to show that (a) is equivalent to each of (b), (c), and (d). First, (a)⇔(b) follows by combining Theorem A.13 with Lemma A.10. Second, (a)⇔(c) is exactly Theorem A.22. Finally, (a)⇔(d) is precisely Theorem A.24.
Proof of Lemma 3.4
Before proving Lemma 3.4, we prove an important technical lemma.
Lemma A.25 (Diverging series). Let 0 < a s → ∞ 0<a_s\to\infty 0 < a s → ∞ be strictly increasing; then
∑ s = 1 ∞ ( 1 − a s a s + 1 ) = ∞ \sum_{s=1}^{\infty}\left(1-\frac{a_s}{a_{s+1}}\right)=\infty s = 1 ∑ ∞ ( 1 − a s + 1 a s ) = ∞
Proof. Take { a s } s = 1 ∞ \{a_s\}_{s=1}^{\infty} { a s } s = 1 ∞ strictly increasing such that 0 < a s → ∞ 0<a_s\to\infty 0 < a s → ∞ . We now use Lemma A.6 with r s = a s a s + 1 ∈ ( 0 , 1 ) r_s=\frac{a_s}{a_{s+1}}\in(0,1) r s = a s + 1 a s ∈ ( 0 , 1 ) (where we used a s > 0 a_s>0 a s > 0 and a s < a s + 1 a_s<a_{s+1} a s < a s + 1 ). The condition of the lemma holds:
∏ s = 1 t r s = ∏ s = 1 t a s a s + 1 = a 1 a t + 1 → t → ∞ 0 , \prod_{s=1}^{t}r_s=\prod_{s=1}^{t}\frac{a_s}{a_{s+1}}=\frac{a_1}{a_{t+1}}\xrightarrow{t\to\infty}0, s = 1 ∏ t r s = s = 1 ∏ t a s + 1 a s = a t + 1 a 1 t → ∞ 0 ,
where we used a t → ∞ a_t\to\infty a t → ∞ . This proves
∑ s = 1 ∞ ( 1 − a s a s + 1 ) = ∑ s = 1 ∞ ( 1 − r s ) = ∞ \sum_{s=1}^{\infty}\left(1-\frac{a_s}{a_{s+1}}\right)=\sum_{s=1}^{\infty}(1-r_s)=\infty s = 1 ∑ ∞ ( 1 − a s + 1 a s ) = s = 1 ∑ ∞ ( 1 − r s ) = ∞
Proof of Lemma 3.4. Let α ˉ : = sup t ≥ 1 α t ∈ [ 0 , 1 ) \bar{\alpha}:=\sup_{t\geq1}\alpha_t\in[0,1) α ˉ := sup t ≥ 1 α t ∈ [ 0 , 1 ) . We prove that ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ \sum_{t=1}^{\infty}(1-\alpha_t)(1-\rho_t)=\infty ∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) = ∞ . Indeed, note that
∑ t = 1 ∞ ( 1 − α t ) ( 1 − ρ t ) ≥ ( 1 − α ˉ ) ∑ t = 1 ∞ ( 1 − ρ t ) = ( 1 − α ˉ ) ∑ t = 1 ∞ ( 1 − N t N t + 1 ) = ∞ . \sum_{t=1}^{\infty}(1-\alpha_t)(1-\rho_t)\geq(1-\bar{\alpha})\sum_{t=1}^{\infty}(1-\rho_t)=(1-\bar{\alpha})\sum_{t=1}^{\infty}\left(1-\frac{N_t}{N_{t+1}}\right)=\infty. t = 1 ∑ ∞ ( 1 − α t ) ( 1 − ρ t ) ≥ ( 1 − α ˉ ) t = 1 ∑ ∞ ( 1 − ρ t ) = ( 1 − α ˉ ) t = 1 ∑ ∞ ( 1 − N t + 1 N t ) = ∞.
We used 1 − α ˉ > 0 1-\bar{\alpha}>0 1 − α ˉ > 0 and
∑ t = 1 ∞ ( 1 − N t N t + 1 ) = ∞ . \sum_{t=1}^{\infty}\left(1-\frac{N_t}{N_{t+1}}\right)=\infty. t = 1 ∑ ∞ ( 1 − N t + 1 N t ) = ∞.
This directly follows from Lemma A.25 with a t = N t a_t=N_t a t = N t . Indeed, N t = N t − 1 + n b ( t − 1 ) > N t − 1 N_t=N_{t-1}+n_b(t-1)>N_{t-1} N t = N t − 1 + n b ( t − 1 ) > N t − 1 is strictly increasing, and N t = n r + ∑ s = 1 t − 1 n b ( s ) ≥ t − 1 → ∞ ⇒ N t → ∞ N_t=n_r+\sum_{s=1}^{t-1}n_b(s)\geq t-1\to\infty\Rightarrow N_t\to\infty N t = n r + ∑ s = 1 t − 1 n b ( s ) ≥ t − 1 → ∞ ⇒ N t → ∞ .
Proof of Proposition 3.5
Proof. Assume α t ≡ 1 \alpha_t\equiv1 α t ≡ 1 throughout. We start by bounding E [ ∥ μ ( t ) − μ ∥ 2 2 ] \mathbb{E}\left[\|\mu^{(t)}-\mu\|_2^2\right] E [ ∥ μ ( t ) − μ ∥ 2 2 ] .
Step 1 (lower bound). By Theorem A.11(d), the recursion becomes
C t = ( 1 − β t − 1 ) C t − 1 + β t − 1 Σ , (103) C_t=(1-\beta_{t-1})C_{t-1}+\beta_{t-1}\boldsymbol{\Sigma}, \tag*{(103)} C t = ( 1 − β t − 1 ) C t − 1 + β t − 1 Σ , (103)
and hence
C t − C t − 1 = β t − 1 ( Σ − C t − 1 ) , t ≥ 2. C_t-C_{t-1}=\beta_{t-1}\left(\boldsymbol{\Sigma}-C_{t-1}\right),\qquad t\geq2. C t − C t − 1 = β t − 1 ( Σ − C t − 1 ) , t ≥ 2.
By Theorem 3.1(b) and Lemma A.8, C t − 1 = Σ − E [ Σ ( t − 1 ) ] ⪯ Σ C_{t-1}=\boldsymbol{\Sigma}-\mathbb{E}\left[\boldsymbol{\Sigma}^{(t-1)}\right]\preceq\boldsymbol{\Sigma} C t − 1 = Σ − E [ Σ ( t − 1 ) ] ⪯ Σ , and β t − 1 > 0 \beta_{t-1}>0 β t − 1 > 0 by (18). Hence, C t − C t − 1 ⪰ 0 C_t-C_{t-1}\succeq0 C t − C t − 1 ⪰ 0 , i.e., C t ⪰ C t − 1 C_t\succeq C_{t-1} C t ⪰ C t − 1 for every t ≥ 2 t\geq2 t ≥ 2 . By induction,
C t ⪰ C 1 (104) C_t\succeq C_1 \tag*{(104)} C t ⪰ C 1 (104)
for all t ≥ 1 t \ge1 t ≥ 1 . Since the trace is monotone with respect to the positive semidefinite order,
Tr ( C t ) ≥ Tr ( C 1 ) = Tr ( Σ ) n r , \operatorname{Tr}(C_t) \ge\operatorname{Tr}(C_1) = \frac{\operatorname{Tr}(\Sigma)}{n_r}, Tr ( C t ) ≥ Tr ( C 1 ) = n r Tr ( Σ ) ,
where we used (45) . Therefore, by (11) ,
Tr ( Σ ) n r ≤ E [ ∥ μ ( t ) − μ ∥ 2 2 ] . (105) \frac{\operatorname{Tr}(\Sigma)}{n_r} \le\mathbb{E}\left[\|\mu^{(t)}-\mu\|_2^2\right]. \tag*{(105)} n r Tr ( Σ ) ≤ E [ ∥ μ ( t ) − μ ∥ 2 2 ] . (105)
Step 2 (upper bound). Applying trace in (103) , we get
Tr ( C t ) = ( 1 − β t − 1 ) Tr ( C t − 1 ) + β t − 1 Tr ( Σ ) ≤ Tr ( C t − 1 ) + β t − 1 Tr ( Σ ) , \begin{aligned}
\operatorname{Tr}(C_t) &= (1-\beta_{t-1})\operatorname{Tr}(C_{t-1})+\beta_{t-1}\operatorname{Tr}(\Sigma)\\
&\le\operatorname{Tr}(C_{t-1})+\beta_{t-1}\operatorname{Tr}(\Sigma),
\end{aligned} Tr ( C t ) = ( 1 − β t − 1 ) Tr ( C t − 1 ) + β t − 1 Tr ( Σ ) ≤ Tr ( C t − 1 ) + β t − 1 Tr ( Σ ) ,
where we used β t − 1 > 0 \beta_{t-1}>0 β t − 1 > 0 by (18) . By induction,
Tr ( C t ) ≤ Tr ( C 1 ) + ( ∑ s = 2 t β s − 1 ) Tr ( Σ ) ≤ Tr ( Σ ) n r + ( ∑ s = 1 ∞ β s ) Tr ( Σ ) ≤ 2 Tr ( Σ ) n r , (106) \begin{aligned}
\operatorname{Tr}(C_t) &\le\operatorname{Tr}(C_1)+\left(\sum_{s=2}^{t}\beta_{s-1}\right)\operatorname{Tr}(\Sigma)\\
&\le\frac{\operatorname{Tr}(\Sigma)}{n_r}+\left(\sum_{s=1}^{\infty}\beta_s\right)\operatorname{Tr}(\Sigma)\\
&\le\frac{2\operatorname{Tr}(\Sigma)}{n_r}, \tag*{(106)}
\end{aligned} Tr ( C t ) ≤ Tr ( C 1 ) + ( s = 2 ∑ t β s − 1 ) Tr ( Σ ) ≤ n r Tr ( Σ ) + ( s = 1 ∑ ∞ β s ) Tr ( Σ ) ≤ n r 2 Tr ( Σ ) , (106)
where we used (45) , β t > 0 \beta_t>0 β t > 0 from (18) and Lemma A.5. Therefore, by (11) ,
E [ ∥ μ ( t ) − μ ∥ 2 2 ] ≤ 2 Tr ( Σ ) n r . (107) \mathbb{E}\left[\|\mu^{(t)}-\mu\|_2^2\right]\le\frac{2\operatorname{Tr}(\Sigma)}{n_r}. \tag*{(107)} E [ ∥ μ ( t ) − μ ∥ 2 2 ] ≤ n r 2 Tr ( Σ ) . (107)
Combining (105) and (107) , we get
Tr ( Σ ) n r ≤ E [ ∥ μ ( t ) − μ ∥ 2 2 ] ≤ 2 Tr ( Σ ) n r . (108) \frac{\operatorname{Tr}(\Sigma)}{n_r}\le\mathbb{E}\left[\|\mu^{(t)}-\mu\|_2^2\right]\le\frac{2\operatorname{Tr}(\Sigma)}{n_r}. \tag*{(108)} n r Tr ( Σ ) ≤ E [ ∥ μ ( t ) − μ ∥ 2 2 ] ≤ n r 2 Tr ( Σ ) . (108)
Finally, we bound E [ ∥ Σ ( t ) − Σ ∥ F ] \mathbb{E}\left[\|\Sigma^{(t)}-\Sigma\|_F\right] E [ ∥ Σ ( t ) − Σ ∥ F ] .
Step 3 (lower bound). Using (91) and (104) ,
E [ ∥ Σ ( t ) − Σ ∥ F ] ≥ ∥ C t ∥ F ≥ ∥ C 1 ∥ F ≥ ∥ Σ ∥ F n r (109) \begin{aligned}
\mathbb{E}\left[\|\Sigma^{(t)}-\Sigma\|_F\right] &\ge\|C_t\|_F\\
&\ge\|C_1\|_F\\
&\ge\frac{\|\Sigma\|_F}{n_r} \tag*{(109)}
\end{aligned} E [ ∥ Σ ( t ) − Σ ∥ F ] ≥ ∥ C t ∥ F ≥ ∥ C 1 ∥ F ≥ n r ∥Σ ∥ F (109)
where we used the monotonicity of the Frobenius norm with respect to the positive semidefinite order and (45) .
Step 4 (upper bound). Note that (74) , (81) , (82) and (88) in the proof of Theorem A.22 do not use the hypothesis ∑ s = 1 ∞ ( 1 − α s ) ( 1 − ρ s ) = ∞ \sum_{s=1}^{\infty}(1-\alpha_s)(1-\rho_s)=\infty ∑ s = 1 ∞ ( 1 − α s ) ( 1 − ρ s ) = ∞ . Hence, they hold for α t ≡ 1 \alpha_t\equiv1 α t ≡ 1 . Combining (88) and (106) , we get
E [ ∥ Σ ( t ) − Σ ∥ F ] ≤ P t + Tr ( C t ) ≤ P t + 2 Tr ( Σ ) n r . (110) \begin{aligned}
\mathbb{E}\left[\|\Sigma^{(t)}-\Sigma\|_F\right] &\le\sqrt{P_t}+\operatorname{Tr}(C_t)\\
&\le\sqrt{P_t}+\frac{2\operatorname{Tr}(\Sigma)}{n_r}. \tag*{(110)}
\end{aligned} E [ ∥ Σ ( t ) − Σ ∥ F ] ≤ P t + Tr ( C t ) ≤ P t + n r 2 Tr ( Σ ) . (110)
Using (18) and α t ≡ 1 \alpha_t\equiv1 α t ≡ 1 , we have for every t ≥ 1 t\ge1 t ≥ 1 ,
λ t = N t + n b ( t ) α t N t + 1 = N t + n b ( t ) N t + 1 = N t + 1 N t + 1 = 1. \lambda_t=\frac{N_t+n_b(t)\alpha_t}{N_{t+1}}=\frac{N_t+n_b(t)}{N_{t+1}}=\frac{N_{t+1}}{N_{t+1}}=1. λ t = N t + 1 N t + n b ( t ) α t = N t + 1 N t + n b ( t ) = N t + 1 N t + 1 = 1.
Thus, λ t ≡ 1 \lambda_t \equiv1 λ t ≡ 1 . Thus, (82) reads
P t ≤ ( ∏ k = 1 t − 1 λ k 2 ) P 1 + D ∑ s = 1 t − 1 ( ∏ k = s + 1 t − 1 λ k 2 ) β s = P 1 + D ∑ s = 1 t − 1 β s ≤ P 1 + D ∑ s = 1 ∞ β s ≤ P 1 + D n r , (111) \begin{aligned}
P_t &\le\left(\prod_{k=1}^{t-1}\lambda_k^2\right)P_1+D\sum_{s=1}^{t-1}\left(\prod_{k=s+1}^{t-1}\lambda_k^2\right)\beta_s \\
&=P_1+D\sum_{s=1}^{t-1}\beta_s \\
&\le P_1+D\sum_{s=1}^{\infty}\beta_s \\
&\le P_1+\frac{D}{n_r}, \tag*{(111)}
\end{aligned} P t ≤ ( k = 1 ∏ t − 1 λ k 2 ) P 1 + D s = 1 ∑ t − 1 ( k = s + 1 ∏ t − 1 λ k 2 ) β s = P 1 + D s = 1 ∑ t − 1 β s ≤ P 1 + D s = 1 ∑ ∞ β s ≤ P 1 + n r D , (111)
where we used β t > 0 \beta_t>0 β t > 0 from (18) and Lemma A.5. It is left to bound P 1 P_1 P 1 and D D D .
Bounding D D D . From (81) , we have
D = 3 B ∗ + C Σ , (112) D=3B^*+C_{\Sigma}, \tag*{(112)} D = 3 B ∗ + C Σ , (112)
where C Σ C_{\Sigma} C Σ is defined in (52) , and B ∗ = sup t ≥ 1 E [ Tr ( S ^ t ) 2 ] B^*=\sup_{t\geq1}\mathbb{E}[\operatorname{Tr}(\widehat{S}_t)^2] B ∗ = sup t ≥ 1 E [ Tr ( S t ) 2 ] is finite from Lemma A.19 . We now bound B ∗ B^* B ∗ . Using (69) , we have
B ∗ ≤ exp ( 3 n r ) ( v 1 + D 0 ) + Tr ( Σ ) 2 . (113) B^*\leq\exp\left(\frac{3}{n_r}\right)(v_1+D_0)+\operatorname{Tr}(\Sigma)^2. \tag*{(113)} B ∗ ≤ exp ( n r 3 ) ( v 1 + D 0 ) + Tr ( Σ ) 2 . (113)
Using (61) and (68) ,
v 1 ≤ 1 n r C Σ , (114) v_1\leq\frac{1}{n_r}C_{\Sigma}, \tag*{(114)} v 1 ≤ n r 1 C Σ , (114)
and
D 0 ≤ 1 n r ( C Σ + 3 Tr ( Σ ) 2 ) . (115) D_0\leq\frac{1}{n_r}\left(C_{\Sigma}+3\operatorname{Tr}(\Sigma)^2\right). \tag*{(115)} D 0 ≤ n r 1 ( C Σ + 3 Tr ( Σ ) 2 ) . (115)
Combining (112) , (113) , (114) , and (115) , we get
D ≤ 3 ( exp ( 3 n r ) ( v 1 + D 0 ) + Tr ( Σ ) 2 ) + C Σ ≤ 3 ( exp ( 3 n r ) ( 1 n r C Σ + 1 n r ( C Σ + 3 Tr ( Σ ) 2 ) ) + Tr ( Σ ) 2 ) + C Σ = 3 n r exp ( 3 n r ) ( 2 C Σ + 3 Tr ( Σ ) 2 ) + 3 Tr ( Σ ) 2 + C Σ . (116) \begin{aligned}
D &\leq3\left(\exp\left(\frac{3}{n_r}\right)(v_1+D_0)+\operatorname{Tr}(\Sigma)^2\right)+C_{\Sigma} \\
&\leq3\left(\exp\left(\frac{3}{n_r}\right)\left(\frac{1}{n_r}C_{\Sigma}+\frac{1}{n_r}\left(C_{\Sigma}+3\operatorname{Tr}(\Sigma)^2\right)\right)+\operatorname{Tr}(\Sigma)^2\right)+C_{\Sigma} \\
&=\frac{3}{n_r}\exp\left(\frac{3}{n_r}\right)\left(2C_{\Sigma}+3\operatorname{Tr}(\Sigma)^2\right)+3\operatorname{Tr}(\Sigma)^2+C_{\Sigma}. \tag*{(116)}
\end{aligned} D ≤ 3 ( exp ( n r 3 ) ( v 1 + D 0 ) + Tr ( Σ ) 2 ) + C Σ ≤ 3 ( exp ( n r 3 ) ( n r 1 C Σ + n r 1 ( C Σ + 3 Tr ( Σ ) 2 ) ) + Tr ( Σ ) 2 ) + C Σ = n r 3 exp ( n r 3 ) ( 2 C Σ + 3 Tr ( Σ ) 2 ) + 3 Tr ( Σ ) 2 + C Σ . (116)
Bounding P 1 P_1 P 1 . For a random matrix X ∈ R d × d \mathbf{X} \in\mathbb{R}^{d \times d} X ∈ R d × d , we write Cov ( X ) : = Cov ( vec ( X ) ) ∈ R d 2 × d 2 \operatorname{Cov}(\mathbf{X}) := \operatorname{Cov}(\operatorname{vec}(\mathbf{X})) \in\mathbb{R}^{d^2 \times d^2} Cov ( X ) := Cov ( vec ( X )) ∈ R d 2 × d 2 . As for random vectors, Cov \operatorname{Cov} Cov is additive over independent summands. We have
P 1 = E [ ∥ S ^ 1 − Σ ∥ F 2 ] = E [ ∥ S ^ 1 − E [ S ^ 1 ] ∥ F 2 ] = Tr ( Cov ( S ^ 1 ) ) = Tr ( Cov ( 1 n r ∑ j = 1 n r ( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) T ) ) = Tr ( 1 n r 2 ∑ j = 1 n r Cov ( ( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) T ) ) = 1 n r Tr ( Cov ( ( x 1 ( 0 ) − μ ) ( x 1 ( 0 ) − μ ) T ) ) = 1 n r ( E [ ∥ ( x 1 ( 0 ) − μ ) ( x 1 ( 0 ) − μ ) T ∥ F 2 ] − ∥ E [ ( x 1 ( 0 ) − μ ) ( x 1 ( 0 ) − μ ) T ] ∥ F 2 ) = 1 n r ( E [ ∥ x 1 ( 0 ) − μ ∥ 2 4 ] − ∥ Σ ∥ F 2 ) = 1 n r ( C Σ − ∥ Σ ∥ F 2 ) . (117) \begin{aligned}
P_1 &= \mathbb{E}\left[\left\|\widehat{\mathbf{S}}_1-\boldsymbol{\Sigma}\right\|_F^2\right]\\
&= \mathbb{E}\left[\left\|\widehat{\mathbf{S}}_1-\mathbb{E}[\widehat{\mathbf{S}}_1]\right\|_F^2\right]\\
&= \operatorname{Tr}\left(\operatorname{Cov}(\widehat{\mathbf{S}}_1)\right)\\
&= \operatorname{Tr}\left(\operatorname{Cov}\left(\frac{1}{n_r}\sum_{j=1}^{n_r}\left(\mathbf{x}_j^{(0)}-\boldsymbol{\mu}\right)\left(\mathbf{x}_j^{(0)}-\boldsymbol{\mu}\right)^{\mathsf{T}}\right)\right)\\
&= \operatorname{Tr}\left(\frac{1}{n_r^2}\sum_{j=1}^{n_r}\operatorname{Cov}\left(\left(\mathbf{x}_j^{(0)}-\boldsymbol{\mu}\right)\left(\mathbf{x}_j^{(0)}-\boldsymbol{\mu}\right)^{\mathsf{T}}\right)\right)\\
&= \frac{1}{n_r}\operatorname{Tr}\left(\operatorname{Cov}\left(\left(\mathbf{x}_1^{(0)}-\boldsymbol{\mu}\right)\left(\mathbf{x}_1^{(0)}-\boldsymbol{\mu}\right)^{\mathsf{T}}\right)\right)\\
&= \frac{1}{n_r}\left(\mathbb{E}\left[\left\|\left(\mathbf{x}_1^{(0)}-\boldsymbol{\mu}\right)\left(\mathbf{x}_1^{(0)}-\boldsymbol{\mu}\right)^{\mathsf{T}}\right\|_F^2\right]-\left\|\mathbb{E}\left[\left(\mathbf{x}_1^{(0)}-\boldsymbol{\mu}\right)\left(\mathbf{x}_1^{(0)}-\boldsymbol{\mu}\right)^{\mathsf{T}}\right]\right\|_F^2\right)\\
&= \frac{1}{n_r}\left(\mathbb{E}\left[\left\|\mathbf{x}_1^{(0)}-\boldsymbol{\mu}\right\|_2^4\right]-\|\boldsymbol{\Sigma}\|_F^2\right)\\
&= \frac{1}{n_r}\left(C_{\boldsymbol{\Sigma}}-\|\boldsymbol{\Sigma}\|_F^2\right). \tag*{(117)}
\end{aligned} P 1 = E [ S 1 − Σ F 2 ] = E [ S 1 − E [ S 1 ] F 2 ] = Tr ( Cov ( S 1 ) ) = Tr ( Cov ( n r 1 j = 1 ∑ n r ( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) T ) ) = Tr ( n r 2 1 j = 1 ∑ n r Cov ( ( x j ( 0 ) − μ ) ( x j ( 0 ) − μ ) T ) ) = n r 1 Tr ( Cov ( ( x 1 ( 0 ) − μ ) ( x 1 ( 0 ) − μ ) T ) ) = n r 1 ( E [ ( x 1 ( 0 ) − μ ) ( x 1 ( 0 ) − μ ) T F 2 ] − E [ ( x 1 ( 0 ) − μ ) ( x 1 ( 0 ) − μ ) T ] F 2 ) = n r 1 ( E [ x 1 ( 0 ) − μ 2 4 ] − ∥ Σ ∥ F 2 ) = n r 1 ( C Σ − ∥ Σ ∥ F 2 ) . (117)
First, we used the definitions of P t P_t P t and S ^ t \widehat{\mathbf{S}}_t S t from Theorem A.22 and (54), respectively. Then, we used Lemma A.14(iv), the additivity of the covariance for the i.i.d. samples { x j ( 0 ) } j = 1 n r \left\{\mathbf{x}_j^{(0)}\right\}_{j=1}^{n_r} { x j ( 0 ) } j = 1 n r and the following property
Tr ( Cov ( X ) ) = E [ ∥ X − E [ X ] ∥ F 2 ] = E [ ∥ X ∥ F 2 ] − ∥ E [ X ] ∥ F 2 . \operatorname{Tr}(\operatorname{Cov}(\mathbf{X}))=\mathbb{E}\left[\|\mathbf{X}-\mathbb{E}[\mathbf{X}]\|_F^2\right]=\mathbb{E}\left[\|\mathbf{X}\|_F^2\right]-\|\mathbb{E}[\mathbf{X}]\|_F^2. Tr ( Cov ( X )) = E [ ∥ X − E [ X ] ∥ F 2 ] = E [ ∥ X ∥ F 2 ] − ∥ E [ X ] ∥ F 2 .
Finally, we used the identities E [ ( x 1 ( 0 ) − μ ) ( x 1 ( 0 ) − μ ) T ] = Σ \mathbb{E}\left[\left(\mathbf{x}_1^{(0)}-\boldsymbol{\mu}\right)\left(\mathbf{x}_1^{(0)}-\boldsymbol{\mu}\right)^{\mathsf{T}}\right]=\boldsymbol{\Sigma} E [ ( x 1 ( 0 ) − μ ) ( x 1 ( 0 ) − μ ) T ] = Σ , ∥ x T ∥ F 2 = ∥ x ∥ 2 2 \|\mathbf{x}^{\mathsf{T}}\|_F^2=\|\mathbf{x}\|_2^2 ∥ x T ∥ F 2 = ∥ x ∥ 2 2 , and C Σ = E [ ∥ x 1 ( 0 ) − μ ∥ 4 4 ] C_{\boldsymbol{\Sigma}}=\mathbb{E}\left[\|\mathbf{x}_1^{(0)}-\boldsymbol{\mu}\|_4^4\right] C Σ = E [ ∥ x 1 ( 0 ) − μ ∥ 4 4 ] .
Combining (113), (118) and (119), we get
P t ≤ 1 n r ( C Σ − ∥ Σ ∥ F 2 ) + 3 n r 2 exp ( 3 n r ) ( 2 C Σ + 3 Tr ( Σ ) 2 ) + 1 n r ( 3 Tr ( Σ ) 2 + C Σ ) = 3 n r 2 exp ( 3 n r ) ( 2 C Σ + 3 Tr ( Σ ) 2 ) + 1 n r ( 2 C Σ + 3 Tr ( Σ ) 2 − ∥ Σ ∥ F 2 ) = 3 n r 2 exp ( 3 n r ) ( 5 Tr ( Σ ) 2 + 4 ∥ Σ ∥ F 2 ) + 1 n r ( 5 Tr ( Σ ) 2 + 3 ∥ Σ ∥ F 2 ) . (118) \begin{aligned}
P_t &\leq\frac{1}{n_r}\left(C_{\boldsymbol{\Sigma}}-\|\boldsymbol{\Sigma}\|_F^2\right)+\frac{3}{n_r^2}\exp\left(\frac{3}{n_r}\right)\left(2C_{\boldsymbol{\Sigma}}+3\operatorname{Tr}(\boldsymbol{\Sigma})^2\right)+\frac{1}{n_r}\left(3\operatorname{Tr}(\boldsymbol{\Sigma})^2+C_{\boldsymbol{\Sigma}}\right)\\
&= \frac{3}{n_r^2}\exp\left(\frac{3}{n_r}\right)\left(2C_{\boldsymbol{\Sigma}}+3\operatorname{Tr}(\boldsymbol{\Sigma})^2\right)+\frac{1}{n_r}\left(2C_{\boldsymbol{\Sigma}}+3\operatorname{Tr}(\boldsymbol{\Sigma})^2-\|\boldsymbol{\Sigma}\|_F^2\right)\\
&= \frac{3}{n_r^2}\exp\left(\frac{3}{n_r}\right)\left(5\operatorname{Tr}(\boldsymbol{\Sigma})^2+4\|\boldsymbol{\Sigma}\|_F^2\right)+\frac{1}{n_r}\left(5\operatorname{Tr}(\boldsymbol{\Sigma})^2+3\|\boldsymbol{\Sigma}\|_F^2\right). \tag*{(118)}
\end{aligned} P t ≤ n r 1 ( C Σ − ∥ Σ ∥ F 2 ) + n r 2 3 exp ( n r 3 ) ( 2 C Σ + 3 Tr ( Σ ) 2 ) + n r 1 ( 3 Tr ( Σ ) 2 + C Σ ) = n r 2 3 exp ( n r 3 ) ( 2 C Σ + 3 Tr ( Σ ) 2 ) + n r 1 ( 2 C Σ + 3 Tr ( Σ ) 2 − ∥ Σ ∥ F 2 ) = n r 2 3 exp ( n r 3 ) ( 5 Tr ( Σ ) 2 + 4∥ Σ ∥ F 2 ) + n r 1 ( 5 Tr ( Σ ) 2 + 3∥ Σ ∥ F 2 ) . (118)
where we used (53).
Then, combining (112) and (120), we get
E [ ∥ Σ ( t ) − Σ ∥ F ] ≤ 3 n r 2 exp ( 3 n r ) ( 5 Tr ( Σ ) 2 + 4 ∥ Σ ∥ F 2 ) + 1 n r ( 5 Tr ( Σ ) 2 + 3 ∥ Σ ∥ F 2 ) + 2 Tr ( Σ ) n r = 2 Tr ( Σ ) n r + 1 n r 5 Tr ( Σ ) 2 + 3 ∥ Σ ∥ F 2 + 3 n r exp ( 3 n r ) ( 5 Tr ( Σ ) 2 + 4 ∥ Σ ∥ F 2 ) . (119) \begin{aligned}
\mathbb{E}\left[\|\boldsymbol{\Sigma}^{(t)}-\boldsymbol{\Sigma}\|_F\right]
&\leq\sqrt{\frac{3}{n_r^2}\exp\left(\frac{3}{n_r}\right)\left(5\operatorname{Tr}(\boldsymbol{\Sigma})^2+4\|\boldsymbol{\Sigma}\|_F^2\right)+\frac{1}{n_r}\left(5\operatorname{Tr}(\boldsymbol{\Sigma})^2+3\|\boldsymbol{\Sigma}\|_F^2\right)+\frac{2\operatorname{Tr}(\boldsymbol{\Sigma})}{n_r}}\\
&= \frac{2\operatorname{Tr}(\boldsymbol{\Sigma})}{n_r}+\frac{1}{\sqrt{n_r}}\sqrt{5\operatorname{Tr}(\boldsymbol{\Sigma})^2+3\|\boldsymbol{\Sigma}\|_F^2}+\frac{3}{n_r}\exp\left(\frac{3}{n_r}\right)\left(5\operatorname{Tr}(\boldsymbol{\Sigma})^2+4\|\boldsymbol{\Sigma}\|_F^2\right). \tag*{(119)}
\end{aligned} E [ ∥ Σ ( t ) − Σ ∥ F ] ≤ n r 2 3 exp ( n r 3 ) ( 5 Tr ( Σ ) 2 + 4∥ Σ ∥ F 2 ) + n r 1 ( 5 Tr ( Σ ) 2 + 3∥ Σ ∥ F 2 ) + n r 2 Tr ( Σ ) = n r 2 Tr ( Σ ) + n r 1 5 Tr ( Σ ) 2 + 3∥ Σ ∥ F 2 + n r 3 exp ( n r 3 ) ( 5 Tr ( Σ ) 2 + 4∥ Σ ∥ F 2 ) . (119)
Finally, defining A Σ = 5 Tr ( Σ ) 2 + 3 ∥ Σ ∥ F 2 A_{\boldsymbol{\Sigma}}=5\operatorname{Tr}(\boldsymbol{\Sigma})^2+3\|\boldsymbol{\Sigma}\|_F^2 A Σ = 5 Tr ( Σ ) 2 + 3∥ Σ ∥ F 2 and B Σ = 5 Tr ( Σ ) 2 + 4 ∥ Σ ∥ F 2 B_{\boldsymbol{\Sigma}}=5\operatorname{Tr}(\boldsymbol{\Sigma})^2+4\|\boldsymbol{\Sigma}\|_F^2 B Σ = 5 Tr ( Σ ) 2 + 4∥ Σ ∥ F 2 , and combining (111) and (121), we get
∥ Σ ∥ F n r ≤ E [ ∥ Σ ( t ) − Σ ∥ F ] ≤ 2 Tr ( Σ ) n r + 1 n r A Σ + 3 n r exp ( 3 n r ) B Σ . (120) \frac{\|\boldsymbol{\Sigma}\|_F}{n_r}\leq\mathbb{E}\left[\|\boldsymbol{\Sigma}^{(t)}-\boldsymbol{\Sigma}\|_F\right]\leq\frac{2\operatorname{Tr}(\boldsymbol{\Sigma})}{n_r}+\frac{1}{\sqrt{n_r}}\sqrt{A_{\boldsymbol{\Sigma}}+\frac{3}{n_r}\exp\left(\frac{3}{n_r}\right)B_{\boldsymbol{\Sigma}}.} \tag*{(120)} n r ∥ Σ ∥ F ≤ E [ ∥ Σ ( t ) − Σ ∥ F ] ≤ n r 2 Tr ( Σ ) + n r 1 A Σ + n r 3 exp ( n r 3 ) B Σ . (120)
From (110) and (122), the proposition follows.
Proof of Theorem 3.6
We first state a lemma that will be used in the following proof to compare sums and integrals.
Lemma A.26 (Sum and integral equivalence). Let a ∈ Z a \in\mathbb{Z} a ∈ Z , let f : [ a , ∞ ) → ( 0 , ∞ ) f : [a,\infty) \to(0,\infty) f : [ a , ∞ ) → ( 0 , ∞ ) be non-increasing, and fix h ≥ 0 h \ge0 h ≥ 0 . Then for every integer b > a b > a b > a :
(a)
∫ a b f ( x ) d x ≤ ∑ t = a b − 1 f ( t ) ≤ f ( a ) + ∫ a b f ( x ) d x ; in particular ∑ t = a b − 1 f ( t ) = ∫ a b f ( x ) d x + O ( 1 ) . \int_a^b f(x)\,\mathrm{d}x \le\sum_{t=a}^{b-1} f(t) \le f(a) + \int_a^b f(x)\,\mathrm{d}x; \text{ in particular } \sum_{t=a}^{b-1} f(t) = \int_a^b f(x)\,\mathrm{d}x + O(1). ∫ a b f ( x ) d x ≤ t = a ∑ b − 1 f ( t ) ≤ f ( a ) + ∫ a b f ( x ) d x ; in particular t = a ∑ b − 1 f ( t ) = ∫ a b f ( x ) d x + O ( 1 ) . (b)
∑ t = a b − 1 f ( t ) ≍ ∫ a b f ( x ) d x . \sum_{t=a}^{b-1} f(t) \asymp\int_a^b f(x)\,\mathrm{d}x. t = a ∑ b − 1 f ( t ) ≍ ∫ a b f ( x ) d x . (c)
∫ a + h b + h f ( x ) d x ≍ ∫ a b f ( x ) d x . \int_{a+h}^{b+h} f(x)\,\mathrm{d}x \asymp\int_a^b f(x)\,\mathrm{d}x. ∫ a + h b + h f ( x ) d x ≍ ∫ a b f ( x ) d x . All implied constants are independent of b b b and depend only on f f f and h h h . If moreover ∫ a ∞ f ( x ) d x = ∞ \int_a^\infty f(x)\,\mathrm{d}x = \infty ∫ a ∞ f ( x ) d x = ∞ , then the ratio of the two sides in (b) and (c) tends to 1 1 1 as b → ∞ b \to\infty b → ∞ , so ≍ \asymp ≍ may be replaced by ∼ \sim ∼ .
Proof. Since f f f is monotone on [ a , ∞ ) [a,\infty) [ a , ∞ ) , it is Riemann integrable on every compact sub-interval, so all integrals below are well defined and finite. Note also that f > 0 f > 0 f > 0 everywhere, so every quantity appearing in a denominator is nonzero.
Throughout, we use the following elementary comparison: for every real y ≥ a y \ge a y ≥ a , monotonicity gives f ( y ) ≥ f ( x ) ≥ f ( y + 1 ) f(y) \ge f(x) \ge f(y+1) f ( y ) ≥ f ( x ) ≥ f ( y + 1 ) for all x ∈ [ y , y + 1 ] x \in[y,y+1] x ∈ [ y , y + 1 ] , and hence
f ( y ) ≥ ∫ y y + 1 f ( x ) d x ≥ f ( y + 1 ) . (121) f(y) \ge\int_y^{y+1} f(x)\,\mathrm{d}x \ge f(y+1). \tag*{(121)} f ( y ) ≥ ∫ y y + 1 f ( x ) d x ≥ f ( y + 1 ) . (121)
(a) Fix an integer b > a b > a b > a . Applying the first inequality in (121) at y = t y = t y = t and summing over t = a , … , b − 1 t = a,\ldots,b-1 t = a , … , b − 1 ,
∑ t = a b − 1 f ( t ) ≥ ∑ t = a b − 1 ∫ t t + 1 f ( x ) d x = ∫ a b f ( x ) d x , \sum_{t=a}^{b-1} f(t) \ge\sum_{t=a}^{b-1} \int_t^{t+1} f(x)\,\mathrm{d}x = \int_a^b f(x)\,\mathrm{d}x, t = a ∑ b − 1 f ( t ) ≥ t = a ∑ b − 1 ∫ t t + 1 f ( x ) d x = ∫ a b f ( x ) d x ,
which is the lower bound. For the upper bound, apply the second inequality in (121) at y = t − 1 y = t - 1 y = t − 1 , i.e. f ( t ) ≤ ∫ t − 1 t f ( x ) d x f(t) \le\int_{t-1}^{t} f(x)\,\mathrm{d}x f ( t ) ≤ ∫ t − 1 t f ( x ) d x , and sum over t = a + 1 , … , b − 1 t = a+1,\ldots,b-1 t = a + 1 , … , b − 1 (an empty sum if b = a + 1 b = a+1 b = a + 1 ):
∑ t = a b − 1 f ( t ) = f ( a ) + ∑ t = a + 1 b − 1 f ( t ) ≤ f ( a ) + ∑ t = a + 1 b − 1 ∫ t − 1 t f ( x ) d x = f ( a ) + ∫ a b − 1 f ( x ) d x ≤ f ( a ) + ∫ a b f ( x ) d x , \sum_{t=a}^{b-1} f(t) = f(a) + \sum_{t=a+1}^{b-1} f(t) \le f(a) + \sum_{t=a+1}^{b-1} \int_{t-1}^{t} f(x)\,\mathrm{d}x = f(a) + \int_a^{b-1} f(x)\,\mathrm{d}x \le f(a) + \int_a^b f(x)\,\mathrm{d}x, t = a ∑ b − 1 f ( t ) = f ( a ) + t = a + 1 ∑ b − 1 f ( t ) ≤ f ( a ) + t = a + 1 ∑ b − 1 ∫ t − 1 t f ( x ) d x = f ( a ) + ∫ a b − 1 f ( x ) d x ≤ f ( a ) + ∫ a b f ( x ) d x ,
the last step because f > 0 f > 0 f > 0 . Consequently, writing D ( b ) : = ∑ t = a b − 1 f ( t ) − ∫ a b f ( x ) d x D(b) := \sum_{t=a}^{b-1} f(t) - \int_a^b f(x)\,\mathrm{d}x D ( b ) := ∑ t = a b − 1 f ( t ) − ∫ a b f ( x ) d x , we have shown
0 ≤ D ( b ) ≤ f ( a ) for every integer b > a , (122) 0 \le D(b) \le f(a) \qquad\text{for every integer } b > a, \tag*{(122)} 0 ≤ D ( b ) ≤ f ( a ) for every integer b > a , (122)
which is precisely the assertion ∑ t = a b − 1 f ( t ) = ∫ a b f ( x ) d x + O ( 1 ) \sum_{t=a}^{b-1} f(t) = \int_a^b f(x)\,\mathrm{d}x + O(1) ∑ t = a b − 1 f ( t ) = ∫ a b f ( x ) d x + O ( 1 ) , with implied constant f ( a ) f(a) f ( a ) independent of b b b .
(b) Fix an integer b > a b > a b > a ; then b ≥ a + 1 b \ge a+1 b ≥ a + 1 , so by (121) applied at y = a y = a y = a ,
∫ a b f ( x ) d x ≥ ∫ a a + 1 f ( x ) d x ≥ f ( a + 1 ) > 0 ⇒ 1 f ( a + 1 ) ∫ a b f ( x ) d x ≥ 1. (123) \int_a^b f(x)\,\mathrm{d}x \ge\int_a^{a+1} f(x)\,\mathrm{d}x \ge f(a+1) > 0 \Rightarrow\frac{1}{f(a+1)} \int_a^b f(x)\,\mathrm{d}x \ge1. \tag*{(123)} ∫ a b f ( x ) d x ≥ ∫ a a + 1 f ( x ) d x ≥ f ( a + 1 ) > 0 ⇒ f ( a + 1 ) 1 ∫ a b f ( x ) d x ≥ 1. (123)
Combining (122) and (123) ,
∑ t = a b − 1 f ( t ) ≤ f ( a ) + ∫ a b f ( x ) d x ≤ f ( a ) f ( a + 1 ) ∫ a b f ( x ) d x + ∫ a b f ( x ) d x , \sum_{t=a}^{b-1} f(t) \le f(a) + \int_a^b f(x)\,\mathrm{d}x \le\frac{f(a)}{f(a+1)} \int_a^b f(x)\,\mathrm{d}x + \int_a^b f(x)\,\mathrm{d}x, t = a ∑ b − 1 f ( t ) ≤ f ( a ) + ∫ a b f ( x ) d x ≤ f ( a + 1 ) f ( a ) ∫ a b f ( x ) d x + ∫ a b f ( x ) d x ,
so that, together with the lower bound from (a),
∫ a b f ( x ) d x ≤ ∑ t = a b − 1 f ( t ) ≤ ( 1 + f ( a ) f ( a + 1 ) ) ∫ a b f ( x ) d x . \int_a^b f(x)\,\mathrm{d}x \le\sum_{t=a}^{b-1} f(t) \le\left(1+\frac{f(a)}{f(a+1)}\right)\int_a^b f(x)\,\mathrm{d}x. ∫ a b f ( x ) d x ≤ t = a ∑ b − 1 f ( t ) ≤ ( 1 + f ( a + 1 ) f ( a ) ) ∫ a b f ( x ) d x .
Both constants are independent of b b b , proving (b).
(c) Fix h ≥ 0 h \ge0 h ≥ 0 and an integer b > a b>a b > a . The change of variables x = y + h x=y+h x = y + h gives ∫ a + h b + h f ( x ) d x = ∫ a b f ( y + h ) d y \int_{a+h}^{b+h} f(x)\,\mathrm{d}x=\int_a^b f(y+h)\,\mathrm{d}y ∫ a + h b + h f ( x ) d x = ∫ a b f ( y + h ) d y , and since f ( y + h ) ≤ f ( y ) f(y+h)\le f(y) f ( y + h ) ≤ f ( y ) for h ≥ 0 h\ge0 h ≥ 0 ,
∫ a + h b + h f ( x ) d x ≤ ∫ a b f ( x ) d x . (124) \int_{a+h}^{b+h} f(x)\,\mathrm{d}x \le\int_a^b f(x)\,\mathrm{d}x. \tag*{(124)} ∫ a + h b + h f ( x ) d x ≤ ∫ a b f ( x ) d x . (124)
For the reverse bound, additivity of the integral yields
∫ a b f ( x ) d x − ∫ a + h b + h f ( x ) d x = ∫ a a + h f ( x ) d x − ∫ b b + h f ( x ) d x ≤ ∫ a a + h f ( x ) d x ≤ h f ( a ) , (125) \int_a^b f(x)\,\mathrm{d}x-\int_{a+h}^{b+h}f(x)\,\mathrm{d}x=\int_a^{a+h}f(x)\,\mathrm{d}x-\int_b^{b+h}f(x)\,\mathrm{d}x\le\int_a^{a+h}f(x)\,\mathrm{d}x\le hf(a), \tag*{(125)} ∫ a b f ( x ) d x − ∫ a + h b + h f ( x ) d x = ∫ a a + h f ( x ) d x − ∫ b b + h f ( x ) d x ≤ ∫ a a + h f ( x ) d x ≤ h f ( a ) , (125)
using f > 0 f>0 f > 0 and then f ≤ f ( a ) f\le f(a) f ≤ f ( a ) on [ a , a + h ] [a,a+h] [ a , a + h ] , because f f f is non-increasing. Since b ≥ a + 1 b\ge a+1 b ≥ a + 1 , the interval [ a + h , b + h ] [a+h,b+h] [ a + h , b + h ] contains [ a + h , a + h + 1 ] [a+h,a+h+1] [ a + h , a + h + 1 ] , so by (121) at y = a + h y=a+h y = a + h ,
∫ a + h b + h f ( x ) d x ≥ ∫ a + h a + h + 1 f ( x ) d x ≥ f ( a + h + 1 ) > 0 ⇒ 1 f ( a + h + 1 ) ∫ a + h b + h f ( x ) d x ≥ 1. \int_{a+h}^{b+h}f(x)\,\mathrm{d}x\ge\int_{a+h}^{a+h+1}f(x)\,\mathrm{d}x\ge f(a+h+1)>0\Rightarrow\frac{1}{f(a+h+1)}\int_{a+h}^{b+h}f(x)\,\mathrm{d}x\ge1. ∫ a + h b + h f ( x ) d x ≥ ∫ a + h a + h + 1 f ( x ) d x ≥ f ( a + h + 1 ) > 0 ⇒ f ( a + h + 1 ) 1 ∫ a + h b + h f ( x ) d x ≥ 1.
Feeding this into (125) ,
∫ a b f ( x ) d x ≤ ∫ a + h b + h f ( x ) d x + h f ( a ) ≤ ( 1 + h f ( a ) f ( a + h + 1 ) ) ∫ a + h b + h f ( x ) d x , \int_a^b f(x)\,\mathrm{d}x\le\int_{a+h}^{b+h}f(x)\,\mathrm{d}x+hf(a)\le\left(1+\frac{hf(a)}{f(a+h+1)}\right)\int_{a+h}^{b+h}f(x)\,\mathrm{d}x, ∫ a b f ( x ) d x ≤ ∫ a + h b + h f ( x ) d x + h f ( a ) ≤ ( 1 + f ( a + h + 1 ) h f ( a ) ) ∫ a + h b + h f ( x ) d x ,
which together with (124) proves (c) with constants independent of b b b .
The case ∫ a ∞ f ( x ) d x = ∞ \int_a^\infty f(x)\,\mathrm{d}x=\infty ∫ a ∞ f ( x ) d x = ∞ . Here ∫ a b f ( x ) d x → ∞ \int_a^b f(x)\,\mathrm{d}x\to\infty ∫ a b f ( x ) d x → ∞ as b → ∞ b\to\infty b → ∞ , so we may divide by it for all large b b b . By (122) ,
∑ t = a b − 1 f ( t ) ∫ a b f ( x ) d x = 1 + D ( b ) ∫ a b f ( x ) d x ∈ [ 1 , 1 + f ( a ) ∫ a b f ( x ) d x ] → b → ∞ 1 , \frac{\sum_{t=a}^{b-1}f(t)}{\int_a^b f(x)\,\mathrm{d}x}=1+\frac{D(b)}{\int_a^b f(x)\,\mathrm{d}x}\in\left[1,1+\frac{f(a)}{\int_a^b f(x)\,\mathrm{d}x}\right]\xrightarrow[b\to\infty]{}1, ∫ a b f ( x ) d x ∑ t = a b − 1 f ( t ) = 1 + ∫ a b f ( x ) d x D ( b ) ∈ [ 1 , 1 + ∫ a b f ( x ) d x f ( a ) ] b → ∞ 1 ,
giving ∑ t = a b − 1 f ( t ) ∼ ∫ a b f ( x ) d x \sum_{t=a}^{b-1}f(t)\sim\int_a^b f(x)\,\mathrm{d}x ∑ t = a b − 1 f ( t ) ∼ ∫ a b f ( x ) d x . Similarly, both ∫ a a + h f \int_a^{a+h}f ∫ a a + h f and ∫ b b + h f \int_b^{b+h}f ∫ b b + h f lie in [ 0 , h f ( a ) ] [0,hf(a)] [ 0 , h f ( a )] , so by (125) the difference ∫ a b f − ∫ a + h b + h f \int_a^b f-\int_{a+h}^{b+h}f ∫ a b f − ∫ a + h b + h f lies in [ − h f ( a ) , h f ( a ) ] [-hf(a),hf(a)] [ − h f ( a ) , h f ( a )] ; dividing by ∫ a b f ( x ) d x → ∞ \int_a^b f(x)\,\mathrm{d}x\to\infty ∫ a b f ( x ) d x → ∞ gives
∣ 1 − ∫ a + h b + h f ( x ) d x ∫ a b f ( x ) d x ∣ ≤ h f ( a ) ∫ a b f ( x ) d x → b → ∞ 0 ⇒ ∫ a + h b + h f ( x ) d x ∫ a b f ( x ) d x → b → ∞ 1. \left|1-\frac{\int_{a+h}^{b+h}f(x)\,\mathrm{d}x}{\int_a^b f(x)\,\mathrm{d}x}\right|\le\frac{hf(a)}{\int_a^b f(x)\,\mathrm{d}x}\xrightarrow[b\to\infty]{}0\Rightarrow\frac{\int_{a+h}^{b+h}f(x)\,\mathrm{d}x}{\int_a^b f(x)\,\mathrm{d}x}\xrightarrow[b\to\infty]{}1. 1 − ∫ a b f ( x ) d x ∫ a + h b + h f ( x ) d x ≤ ∫ a b f ( x ) d x h f ( a ) b → ∞ 0 ⇒ ∫ a b f ( x ) d x ∫ a + h b + h f ( x ) d x b → ∞ 1.
Proof of Theorem 3.6. Throughout α t ≡ α ∈ [ 0 , 1 ] \alpha_t\equiv\alpha\in[0,1] α t ≡ α ∈ [ 0 , 1 ] and n b ( t ) ≡ n b n_b(t)\equiv n_b n b ( t ) ≡ n b are fixed, so that N t = n r + ( t − 1 ) n b N_t=n_r+(t-1)n_b N t = n r + ( t − 1 ) n b and κ = 2 ( 1 − α ) ∈ [ 0 , 2 ] \kappa=2(1-\alpha)\in[0,2] κ = 2 ( 1 − α ) ∈ [ 0 , 2 ] (note that κ = 0 \kappa=0 κ = 0 when α = 1 \alpha=1 α = 1 ). Since n b = N t + 1 − N t n_b=N_{t+1}-N_t n b = N t + 1 − N t and N t ≥ n r N_t\ge n_r N t ≥ n r , we have
N t + 1 N t = N t + n b N t = 1 + n b N t ≤ 1 + n b n r , \frac{N_{t+1}}{N_t}=\frac{N_t+n_b}{N_t}=1+\frac{n_b}{N_t}\le1+\frac{n_b}{n_r}, N t N t + 1 = N t N t + n b = 1 + N t n b ≤ 1 + n r n b ,
so N t ≍ N t + 1 ≍ t n b N_t\asymp N_{t+1}\asymp tn_b N t ≍ N t + 1 ≍ t n b ; all implied constants below depend only on α \alpha α , n b n_b n b , n r n_r n r and Tr ( Σ ) \operatorname{Tr}(\Sigma) Tr ( Σ ) .
We use that for strictly positive sequences, an ≍ \asymp ≍ relation valid for all large indices extends to all of them by enlarging the constants over the remaining finite set. Additionally, by (11), note that E [ ∥ μ ( t ) − μ ∥ 2 2 ] = Tr ( C t ) \mathbb{E}[\|\mu^{(t)}-\mu\|_2^2]=\operatorname{Tr}(C_t) E [ ∥ μ ( t ) − μ ∥ 2 2 ] = Tr ( C t ) . Thus, giving an asymptotic analysis for Tr ( C t ) \operatorname{Tr}(C_t) Tr ( C t ) suffices.
Step 1: ∑ t = 1 m ( 1 − γ t ) = κ ln N m + 1 + O ( 1 ) \sum_{t=1}^{m}(1-\gamma_t)=\kappa\ln N_{m+1}+O(1) ∑ t = 1 m ( 1 − γ t ) = κ ln N m + 1 + O ( 1 ) . By (51) with α t ≡ α \alpha_t\equiv\alpha α t ≡ α ,
1 − γ t = n b [ ( 1 − α ) ( 2 N t + n b ( 1 + α ) ) + α 2 ] N t + 1 2 = κ n b N t N t + 1 2 + n b [ ( 1 − α ) n b ( 1 + α ) + α 2 ] N t + 1 2 = κ n b N t + 1 − n b N t + 1 2 + n b [ n b ( 1 − α 2 ) + α 2 ] N t + 1 2 = κ n b N t + 1 + ( n b ( 1 − α 2 ) + α 2 − κ n b ) n b N t + 1 2 = κ n b N t + 1 + ( α 2 − n b ( 1 − α ) 2 ) β t . \begin{aligned}
1-\gamma_t
&=\frac{n_b\left[(1-\alpha)(2N_t+n_b(1+\alpha))+\alpha^2\right]}{N_{t+1}^2}\\
&=\kappa n_b\frac{N_t}{N_{t+1}^2}+\frac{n_b\left[(1-\alpha)n_b(1+\alpha)+\alpha^2\right]}{N_{t+1}^2}\\
&=\kappa n_b\frac{N_{t+1}-n_b}{N_{t+1}^2}+\frac{n_b\left[n_b(1-\alpha^2)+\alpha^2\right]}{N_{t+1}^2}\\
&=\frac{\kappa n_b}{N_{t+1}}+\left(n_b(1-\alpha^2)+\alpha^2-\kappa n_b\right)\frac{n_b}{N_{t+1}^2}\\
&=\frac{\kappa n_b}{N_{t+1}}+\left(\alpha^2-n_b(1-\alpha)^2\right)\beta_t.
\end{aligned} 1 − γ t = N t + 1 2 n b [ ( 1 − α ) ( 2 N t + n b ( 1 + α )) + α 2 ] = κ n b N t + 1 2 N t + N t + 1 2 n b [ ( 1 − α ) n b ( 1 + α ) + α 2 ] = κ n b N t + 1 2 N t + 1 − n b + N t + 1 2 n b [ n b ( 1 − α 2 ) + α 2 ] = N t + 1 κ n b + ( n b ( 1 − α 2 ) + α 2 − κ n b ) N t + 1 2 n b = N t + 1 κ n b + ( α 2 − n b ( 1 − α ) 2 ) β t .
where we used
n b ( 1 − α 2 ) + α 2 − κ n b = n b ( 1 − α 2 − 2 ( 1 − α ) ) + α 2 = n b ( − α 2 + 2 α − 1 ) + α 2 = α 2 − n b ( 1 − α ) 2 . n_b(1-\alpha^2)+\alpha^2-\kappa n_b=n_b\left(1-\alpha^2-2(1-\alpha)\right)+\alpha^2=n_b(-\alpha^2+2\alpha-1)+\alpha^2=\alpha^2-n_b(1-\alpha)^2. n b ( 1 − α 2 ) + α 2 − κ n b = n b ( 1 − α 2 − 2 ( 1 − α ) ) + α 2 = n b ( − α 2 + 2 α − 1 ) + α 2 = α 2 − n b ( 1 − α ) 2 .
Thus,
∑ t = 1 m ( 1 − γ t ) = κ n b ∑ t = 1 m 1 N t + 1 + ( α 2 − n b ( 1 − α ) 2 ) ∑ t = 1 m β t = κ ln N m + 1 + O ( 1 ) . (126) \sum_{t=1}^{m}(1-\gamma_t)=\kappa n_b\sum_{t=1}^{m}\frac{1}{N_{t+1}}+\left(\alpha^2-n_b(1-\alpha)^2\right)\sum_{t=1}^{m}\beta_t=\kappa\ln N_{m+1}+O(1). \tag*{(126)} t = 1 ∑ m ( 1 − γ t ) = κ n b t = 1 ∑ m N t + 1 1 + ( α 2 − n b ( 1 − α ) 2 ) t = 1 ∑ m β t = κ ln N m + 1 + O ( 1 ) . (126)
We used Lemma A.5 (∑ t = 1 ∞ β t < ∞ \sum_{t=1}^{\infty}\beta_t<\infty ∑ t = 1 ∞ β t < ∞ ) and
∑ t = 1 m 1 N t + 1 = 1 n b ln N m + 1 + O ( 1 ) . \sum_{t=1}^{m}\frac{1}{N_{t+1}}=\frac{1}{n_b}\ln N_{m+1}+O(1). t = 1 ∑ m N t + 1 1 = n b 1 ln N m + 1 + O ( 1 ) .
It follows from Lemma A.26(a) with a = 1 a=1 a = 1 , b = m + 1 b=m+1 b = m + 1 , and the function f ( x ) = 1 / ( n r + n b x ) f(x)=1/(n_r+n_bx) f ( x ) = 1/ ( n r + n b x ) . It is positive and non-increasing, thus:
∑ t = 1 m 1 N t + 1 = ∑ t = 1 m 1 n r + n b t = ∑ t = 1 m f ( t ) = ∫ 1 m + 1 f ( x ) d x + O ( 1 ) = ∫ 1 m + 1 1 n r + n b x d x + O ( 1 ) = ( 1 n b ln ( n r + n b x ) ) ∣ 1 m + 1 + O ( 1 ) = 1 n b ( ln ( n r + n b ( m + 1 ) ) − ln ( n r + n b ) ) + O ( 1 ) = 1 n b ⋅ ln ( N m + 2 ) + O ( 1 ) = 1 n b ln ( N m + 1 ) + O ( 1 ) . \begin{aligned}
\sum_{t=1}^{m}\frac{1}{N_{t+1}}
&=\sum_{t=1}^{m}\frac{1}{n_r+n_bt}=\sum_{t=1}^{m}f(t)\\
&=\int_{1}^{m+1}f(x)\,dx+O(1)\\
&=\int_{1}^{m+1}\frac{1}{n_r+n_bx}\,dx+O(1)\\
&=\left(\frac{1}{n_b}\ln(n_r+n_bx)\right)\bigg|_{1}^{m+1}+O(1)\\
&=\frac{1}{n_b}\left(\ln(n_r+n_b(m+1))-\ln(n_r+n_b)\right)+O(1)\\
&=\frac{1}{n_b}\cdot\ln(N_{m+2})+O(1)\\
&=\frac{1}{n_b}\ln(N_{m+1})+O(1).
\end{aligned} t = 1 ∑ m N t + 1 1 = t = 1 ∑ m n r + n b t 1 = t = 1 ∑ m f ( t ) = ∫ 1 m + 1 f ( x ) d x + O ( 1 ) = ∫ 1 m + 1 n r + n b x 1 d x + O ( 1 ) = ( n b 1 ln ( n r + n b x ) ) 1 m + 1 + O ( 1 ) = n b 1 ( ln ( n r + n b ( m + 1 )) − ln ( n r + n b ) ) + O ( 1 ) = n b 1 ⋅ ln ( N m + 2 ) + O ( 1 ) = n b 1 ln ( N m + 1 ) + O ( 1 ) .
We used N m + 2 ≍ N m + 1 ⇒ ln ( N m + 2 ) = ln ( N m + 1 ) + O ( 1 ) N_{m+2}\asymp N_{m+1}\Rightarrow\ln(N_{m+2})=\ln(N_{m+1})+O(1) N m + 2 ≍ N m + 1 ⇒ ln ( N m + 2 ) = ln ( N m + 1 ) + O ( 1 ) . This holds uniformly in m m m : 1 ≤ N m + 2 / N m + 1 ≤ 1 + n b / n r 1\le N_{m+2}/N_{m+1}\le1+n_b/n_r 1 ≤ N m + 2 / N m + 1 ≤ 1 + n b / n r .
Step 2: Γ 1 , m ≲ N m + 1 − κ \Gamma_{1,m}\lesssim N_{m+1}^{-\kappa} Γ 1 , m ≲ N m + 1 − κ , uniformly in m ≥ 1 m\ge1 m ≥ 1 . From (51) and α t ∈ [ 0 , 1 ] \alpha_t\in[0,1] α t ∈ [ 0 , 1 ] , we have
0 ≤ 1 − γ t = n b N t + 1 2 ⋅ [ ( 1 − α t ) ⋅ ( 2 N t + n b ( 1 + α t ) ) + α t 2 ] ≤ n b N t + 1 2 ⋅ ( 2 N t + 2 n b + 1 ) = n b N t + 1 2 ⋅ ( 2 N t + 1 + 1 ) ≤ 3 n b N t + 1 ⟶ t → ∞ 0. \begin{aligned}
0\le1-\gamma_t
&=\frac{n_b}{N_{t+1}^2}\cdot\left[(1-\alpha_t)\cdot(2N_t+n_b(1+\alpha_t))+\alpha_t^2\right]\\
&\le\frac{n_b}{N_{t+1}^2}\cdot(2N_t+2n_b+1)\\
&=\frac{n_b}{N_{t+1}^2}\cdot(2N_{t+1}+1)\\
&\le\frac{3n_b}{N_{t+1}}\underset{t\to\infty}{\longrightarrow}0.
\end{aligned} 0 ≤ 1 − γ t = N t + 1 2 n b ⋅ [ ( 1 − α t ) ⋅ ( 2 N t + n b ( 1 + α t )) + α t 2 ] ≤ N t + 1 2 n b ⋅ ( 2 N t + 2 n b + 1 ) = N t + 1 2 n b ⋅ ( 2 N t + 1 + 1 ) ≤ N t + 1 3 n b t → ∞ ⟶ 0.
That is, 1 − γ t → 0 1-\gamma_t \to0 1 − γ t → 0 , and there is t 1 t_1 t 1 with 1 − γ t ≤ 1 2 1-\gamma_t \le\frac{1}{2} 1 − γ t ≤ 2 1 for t ≥ t 1 t \ge t_1 t ≥ t 1 . That is,
t ≥ t 1 ⇒ γ t ≥ 1 2 . (127) t \ge t_1 \Rightarrow\gamma_t \ge\frac{1}{2}. \tag*{(127)} t ≥ t 1 ⇒ γ t ≥ 2 1 . (127)
Finally, from (19) , we have
Γ 1 , m = ∏ t = 1 m γ t = exp ( ∑ t = 1 m ln γ t ) = exp ( − ∑ t = 1 m − ln γ t ) = exp ( − B m ) , (128) \Gamma_{1,m}=\prod_{t=1}^{m}\gamma_t=\exp\left(\sum_{t=1}^{m}\ln\gamma_t\right)=\exp\left(-\sum_{t=1}^{m}-\ln\gamma_t\right)=\exp(-B_m), \tag*{(128)} Γ 1 , m = t = 1 ∏ m γ t = exp ( t = 1 ∑ m ln γ t ) = exp ( − t = 1 ∑ m − ln γ t ) = exp ( − B m ) , (128)
where we defined
B m : = ∑ t = 1 m − ln γ t . (129) B_m:=\sum_{t=1}^{m}-\ln\gamma_t. \tag*{(129)} B m := t = 1 ∑ m − ln γ t . (129)
Before we analyze B m B_m B m , we record two elementary bounds on the logarithm. First, − ln u ≥ 0 -\ln u\ge0 − ln u ≥ 0 for every u ∈ ( 0 , 1 ] u\in(0,1] u ∈ ( 0 , 1 ] , by monotonicity of ln \ln ln . Second,
( 1 − u ) ≤ − ln u ≤ ( 1 − u ) + ( 1 − u ) 2 , u ∈ [ 1 2 , 1 ] . (130) (1-u)\le-\ln u\le(1-u)+(1-u)^2,\qquad u\in\left[\frac{1}{2},1\right]. \tag*{(130)} ( 1 − u ) ≤ − ln u ≤ ( 1 − u ) + ( 1 − u ) 2 , u ∈ [ 2 1 , 1 ] . (130)
Indeed, writing x : = 1 − u ∈ [ 0 , 1 2 ] x:=1-u\in[0,\frac{1}{2}] x := 1 − u ∈ [ 0 , 2 1 ] , the series − ln u = − ln ( 1 − x ) = ∑ k = 1 ∞ x k / k -\ln u=-\ln(1-x)=\sum_{k=1}^{\infty}x^k/k − ln u = − ln ( 1 − x ) = ∑ k = 1 ∞ x k / k converges, and
0 ≤ − ln u − ( 1 − u ) = ∑ k = 2 ∞ x k k ≤ 1 2 ∑ k = 2 ∞ x k = x 2 2 ( 1 − x ) = ( 1 − u ) 2 2 u ≤ ( 1 − u ) 2 , 0\le-\ln u-(1-u)=\sum_{k=2}^{\infty}\frac{x^k}{k}\le\frac{1}{2}\sum_{k=2}^{\infty}x^k=\frac{x^2}{2(1-x)}=\frac{(1-u)^2}{2u}\le(1-u)^2, 0 ≤ − ln u − ( 1 − u ) = k = 2 ∑ ∞ k x k ≤ 2 1 k = 2 ∑ ∞ x k = 2 ( 1 − x ) x 2 = 2 u ( 1 − u ) 2 ≤ ( 1 − u ) 2 ,
the first inequality because every term is nonnegative, and the last because u ≥ 1 2 u\ge\frac{1}{2} u ≥ 2 1 . Rearranging gives (132).
Fix m ≥ t 1 m\ge t_1 m ≥ t 1 . From (131), an upper bound on B m B_m B m reads:
B m = ∑ t = 1 t 1 − 1 − ln γ t ⏟ O ( 1 ) + ∑ t = t 1 m − ln γ t ≤ O ( 1 ) + ∑ t = t 1 m ( 1 − γ t ) + ∑ t = t 1 m ( 1 − γ t ) 2 ≤ O ( 1 ) + ∑ t = 1 m ( 1 − γ t ) + ∑ t = 1 m 9 n b 2 N t + 1 2 = O ( 1 ) + κ ln N m + 1 + 9 n b ∑ t = 1 m β t = O ( 1 ) + κ ln N m + 1 . \begin{aligned}
B_m&=\underbrace{\sum_{t=1}^{t_1-1}-\ln\gamma_t}_{O(1)}+\sum_{t=t_1}^{m}-\ln\gamma_t\\
&\le O(1)+\sum_{t=t_1}^{m}(1-\gamma_t)+\sum_{t=t_1}^{m}(1-\gamma_t)^2\\
&\le O(1)+\sum_{t=1}^{m}(1-\gamma_t)+\sum_{t=1}^{m}\frac{9n_b^2}{N_{t+1}^2}\\
&=O(1)+\kappa\ln N_{m+1}+9n_b\sum_{t=1}^{m}\beta_t\\
&=O(1)+\kappa\ln N_{m+1}.
\end{aligned} B m = O ( 1 ) t = 1 ∑ t 1 − 1 − ln γ t + t = t 1 ∑ m − ln γ t ≤ O ( 1 ) + t = t 1 ∑ m ( 1 − γ t ) + t = t 1 ∑ m ( 1 − γ t ) 2 ≤ O ( 1 ) + t = 1 ∑ m ( 1 − γ t ) + t = 1 ∑ m N t + 1 2 9 n b 2 = O ( 1 ) + κ ln N m + 1 + 9 n b t = 1 ∑ m β t = O ( 1 ) + κ ln N m + 1 .
We used (127) and (130) . Then, we used (126) , ( 1 − γ t ) ≤ 3 n b / N t + 1 (1-\gamma_t)\le3n_b/N_{t+1} ( 1 − γ t ) ≤ 3 n b / N t + 1 , the definition of β t \beta_t β t from (18) , and the summability of β t \beta_t β t from Lemma A.5. That is,
B m ≤ κ ln N m + 1 + O ( 1 ) (131) B_m\le\kappa\ln N_{m+1}+O(1) \tag*{(131)} B m ≤ κ ln N m + 1 + O ( 1 ) (131)
Additionally, from (131), a lower bound on B m B_m B m is:
B m ≥ ∑ t = t 1 m − ln γ t ≥ ∑ t = t 1 m ( 1 − γ t ) = ∑ t = 1 m ( 1 − γ t ) + O ( 1 ) = κ ln N m + 1 + O ( 1 ) . (132) \begin{aligned}
B_m&\ge\sum_{t=t_1}^{m}-\ln\gamma_t\\
&\ge\sum_{t=t_1}^{m}(1-\gamma_t)\\
&=\sum_{t=1}^{m}(1-\gamma_t)+O(1)\\
&=\kappa\ln N_{m+1}+O(1). \tag*{(132)}
\end{aligned} B m ≥ t = t 1 ∑ m − ln γ t ≥ t = t 1 ∑ m ( 1 − γ t ) = t = 1 ∑ m ( 1 − γ t ) + O ( 1 ) = κ ln N m + 1 + O ( 1 ) . (132)
In the first step, we discarded the terms t < t 1 t < t_1 t < t 1 , which is legitimate since − ln γ t ≥ 0 -\ln\gamma_t \ge0 − ln γ t ≥ 0 for every t t t , by γ t ∈ ( 0 , 1 ) \gamma_t \in(0,1) γ t ∈ ( 0 , 1 ) (Lemma A.9 ). In the second, we applied the left-hand inequality of (130) , valid because γ t ∈ [ 1 2 , 1 ) \gamma_t \in[\frac{1}{2},1) γ t ∈ [ 2 1 , 1 ) for t ≥ t 1 t \ge t_1 t ≥ t 1 by (127) . In the third, we re-added the finitely many terms t < t 1 t < t_1 t < t 1 , each lying in [ 0 , 1 ] [0,1] [ 0 , 1 ] , at a cost of O ( 1 ) O(1) O ( 1 ) . The last step is (126) .
That is, from (131) and (132) ,
B m = κ ln N m + 1 + O ( 1 ) . (133) B_m = \kappa\ln N_{m+1} + O(1). \tag*{(133)} B m = κ ln N m + 1 + O ( 1 ) . (133)
Finally, from (128) and (133) , we have
Γ 1 , m = exp ( − κ ln N m + 1 + O ( 1 ) ) = exp ( ln ( N m + 1 − κ ) + O ( 1 ) ) ≍ N m + 1 − κ . (134) \begin{aligned}
\Gamma_{1,m} &= \exp(-\kappa\ln N_{m+1} + O(1))\\
&= \exp(\ln(N_{m+1}^{-\kappa}) + O(1))\\
&\asymp N_{m+1}^{-\kappa}. \tag*{(134)}
\end{aligned} Γ 1 , m = exp ( − κ ln N m + 1 + O ( 1 )) = exp ( ln ( N m + 1 − κ ) + O ( 1 )) ≍ N m + 1 − κ . (134)
This was derived for m ≥ t 1 m \ge t_1 m ≥ t 1 . For each of the finitely many m < t 1 m < t_1 m < t 1 , the ratio Γ 1 , m / N m + 1 − κ \Gamma_{1,m}/N_{m+1}^{-\kappa} Γ 1 , m / N m + 1 − κ is a strictly positive real number, so over this finite range, it is bounded between its minimum and its maximum, both positive. Enlarging the implied constants accordingly extends the equivalence to every m ≥ 1 m \ge1 m ≥ 1 .
That is, Γ 1 , m ≍ N m + 1 − κ \Gamma_{1,m} \asymp N_{m+1}^{-\kappa} Γ 1 , m ≍ N m + 1 − κ , where the implied constants are uniform in m ≥ 1 m \ge1 m ≥ 1 .
Step 3: Γ s + 1 , t − 1 ≍ ( N s + 1 / N t ) κ \Gamma_{s+1,t-1} \asymp(N_{s+1}/N_t)^\kappa Γ s + 1 , t − 1 ≍ ( N s + 1 / N t ) κ . Indeed, from Step 2 and (19) , we have
Γ s + 1 , t − 1 = Γ 1 , t − 1 Γ 1 , s ≍ N t − κ N s + 1 − κ = ( N s + 1 N t ) κ . \Gamma_{s+1,t-1}=\frac{\Gamma_{1,t-1}}{\Gamma_{1,s}}\asymp\frac{N_t^{-\kappa}}{N_{s+1}^{-\kappa}}=\left(\frac{N_{s+1}}{N_t}\right)^\kappa. Γ s + 1 , t − 1 = Γ 1 , s Γ 1 , t − 1 ≍ N s + 1 − κ N t − κ = ( N t N s + 1 ) κ .
The implied constants are uniform in s s s and t t t .
Step 4: evaluation of S t S_t S t . By Step 3 and β s = n b / N s + 1 2 \beta_s=n_b/N_{s+1}^2 β s = n b / N s + 1 2 from (18) , we have
S t = ∑ s = 1 t − 1 β s Γ s + 1 , t − 1 ≍ ∑ s = 1 t − 1 n b N s + 1 2 ⋅ N s + 1 κ N t κ = n b N t − κ ∑ s = 1 t − 1 N s + 1 κ − 2 = n b N t − κ ∑ s = 1 t − 1 ( n r + n b s ) κ − 2 , (135) \begin{aligned}
S_t&=\sum_{s=1}^{t-1}\beta_s\Gamma_{s+1,t-1}\asymp\sum_{s=1}^{t-1}\frac{n_b}{N_{s+1}^2}\cdot\frac{N_{s+1}^{\kappa}}{N_t^\kappa}\\
&=n_bN_t^{-\kappa}\sum_{s=1}^{t-1}N_{s+1}^{\kappa-2}\\
&=n_bN_t^{-\kappa}\sum_{s=1}^{t-1}(n_r+n_bs)^{\kappa-2}, \tag*{(135)}
\end{aligned} S t = s = 1 ∑ t − 1 β s Γ s + 1 , t − 1 ≍ s = 1 ∑ t − 1 N s + 1 2 n b ⋅ N t κ N s + 1 κ = n b N t − κ s = 1 ∑ t − 1 N s + 1 κ − 2 = n b N t − κ s = 1 ∑ t − 1 ( n r + n b s ) κ − 2 , (135)
where the constants of Step 3 are independent of t t t and therefore factor out of the sum term by term. Let us now use Lemma A.26 (b) with a = 1 a=1 a = 1 and f ( s ) = ( n r + n b s ) κ − 2 f(s)=(n_r+n_bs)^{\kappa-2} f ( s ) = ( n r + n b s ) κ − 2 . Indeed, κ − 2 ≤ 0 \kappa-2\le0 κ − 2 ≤ 0 , and thus f ( s ) f(s) f ( s ) is positive and non-increasing.
∑ s = 1 t − 1 ( n r + n b s ) κ − 2 ≲ ∫ 1 t ( n r + n b x ) κ − 2 d x = u = n r + n b x 1 n b ∫ n r + n b n r + n b t u p d u = 1 n b ∫ n r + n b N t + n b u p d u ≲ 1 n b ∫ n r N t u p d u , (136) \begin{aligned}
\sum_{s=1}^{t-1}(n_r+n_bs)^{\kappa-2}&\lesssim\int_1^t(n_r+n_bx)^{\kappa-2}\,dx\\
&\mathrel{\underset{u=n_r+n_bx}{=}}\frac{1}{n_b}\int_{n_r+n_b}^{n_r+n_bt}u^p\,du\\
&=\frac{1}{n_b}\int_{n_r+n_b}^{N_t+n_b}u^p\,du\\
&\lesssim\frac{1}{n_b}\int_{n_r}^{N_t}u^p\,du, \tag*{(136)}
\end{aligned} s = 1 ∑ t − 1 ( n r + n b s ) κ − 2 ≲ ∫ 1 t ( n r + n b x ) κ − 2 d x u = n r + n b x = n b 1 ∫ n r + n b n r + n b t u p d u = n b 1 ∫ n r + n b N t + n b u p d u ≲ n b 1 ∫ n r N t u p d u , (136)
where we defined p : = κ − 2 = − 2 α ∈ [ − 2 , 0 ] p:=\kappa-2=-2\alpha\in[-2,0] p := κ − 2 = − 2 α ∈ [ − 2 , 0 ] , and used Lemma A.26 (c) (using the decreasing and positive function u ↦ u p u\mapsto u^p u ↦ u p ). Let us now split into two cases. From (136) , we have
∑ s = 1 t − 1 ( n r + n b s ) κ − 2 ≲ 1 n b ln ( u ) ∣ n r N t = 1 n b ln ( N t n r ) . (137) \sum_{s=1}^{t-1}(n_r+n_bs)^{\kappa-2}\lesssim\frac{1}{n_b}\ln(u)\Big|_{n_r}^{N_t}=\frac{1}{n_b}\ln\left(\frac{N_t}{n_r}\right). \tag*{(137)} s = 1 ∑ t − 1 ( n r + n b s ) κ − 2 ≲ n b 1 ln ( u ) n r N t = n b 1 ln ( n r N t ) . (137)
If p ≠ − 1 p \ne-1 p = − 1 (equivalent to κ ≠ 1 \kappa\ne1 κ = 1 , i.e., 0 ≤ κ < 1 0 \le\kappa< 1 0 ≤ κ < 1 or 1 < κ ≤ 2 1 < \kappa\le2 1 < κ ≤ 2 ), we have
∑ s = 1 t − 1 ( n r + n b s ) κ − 2 ≲ 1 n b u p + 1 p + 1 ∣ n r N t = 1 n b N t κ − 1 − n r κ − 1 κ − 1 ≲ { n r κ − 1 n b ( 1 − κ ) 0 ≤ κ < 1 N t κ − 1 n b ( κ − 1 ) 1 < κ ≤ 2 (138) \sum_{s=1}^{t-1}(n_r+n_b s)^{\kappa-2} \lesssim\frac{1}{n_b}\frac{u^{p+1}}{p+1}\bigg|_{n_r}^{N_t}=\frac{1}{n_b}\frac{N_t^{\kappa-1}-n_r^{\kappa-1}}{\kappa-1}\lesssim\begin{cases}\dfrac{n_r^{\kappa-1}}{n_b(1-\kappa)} & 0\le\kappa<1\\[4pt]\dfrac{N_t^{\kappa-1}}{n_b(\kappa-1)} & 1<\kappa\le2\end{cases} \tag*{(138)} s = 1 ∑ t − 1 ( n r + n b s ) κ − 2 ≲ n b 1 p + 1 u p + 1 n r N t = n b 1 κ − 1 N t κ − 1 − n r κ − 1 ≲ ⎩ ⎨ ⎧ n b ( 1 − κ ) n r κ − 1 n b ( κ − 1 ) N t κ − 1 0 ≤ κ < 1 1 < κ ≤ 2 (138)
From (137) and (138) , we have
∑ s = 1 t − 1 ( n r + n b s ) κ − 2 ≲ 1 n b { 1 0 ≤ κ < 1 ln ( N t ) κ = 1 N t κ − 1 1 < κ ≤ 2 (139) \sum_{s=1}^{t-1}(n_r+n_b s)^{\kappa-2}\lesssim\frac{1}{n_b}\begin{cases}1 & 0\le\kappa<1\\ \ln(N_t) & \kappa=1\\ N_t^{\kappa-1} & 1<\kappa\le2\end{cases} \tag*{(139)} s = 1 ∑ t − 1 ( n r + n b s ) κ − 2 ≲ n b 1 ⎩ ⎨ ⎧ 1 ln ( N t ) N t κ − 1 0 ≤ κ < 1 κ = 1 1 < κ ≤ 2 (139)
Let us now substitute (139) into (135) . We get
S t ≲ { N t − κ 0 ≤ κ < 1 N t − 1 ln ( N t ) κ = 1 N t − 1 1 < κ ≤ 2 (140) S_t\lesssim\begin{cases}N_t^{-\kappa} & 0\le\kappa<1\\ N_t^{-1}\ln(N_t) & \kappa=1\\ N_t^{-1} & 1<\kappa\le2\end{cases} \tag*{(140)} S t ≲ ⎩ ⎨ ⎧ N t − κ N t − 1 ln ( N t ) N t − 1 0 ≤ κ < 1 κ = 1 1 < κ ≤ 2 (140)
Step 5: combination. Taking traces in the closed form (38) and using C 1 = Σ / n r C_1=\Sigma/n_r C 1 = Σ/ n r ,
Tr ( C t ) = Γ 1 , t − 1 Tr ( Σ ) n r + Tr ( Σ ) S t ≲ N t − κ + S t , (141) \operatorname{Tr}(C_t)=\Gamma_{1,t-1}\frac{\operatorname{Tr}(\Sigma)}{n_r}+\operatorname{Tr}(\Sigma)S_t\lesssim N_t^{-\kappa}+S_t, \tag*{(141)} Tr ( C t ) = Γ 1 , t − 1 n r Tr ( Σ ) + Tr ( Σ ) S t ≲ N t − κ + S t , (141)
where the two summands are non-negative, Tr ( Σ ) > 0 \operatorname{Tr}(\Sigma)>0 Tr ( Σ ) > 0 and Tr ( C 1 ) = Tr ( Σ ) / n r > 0 \operatorname{Tr}(C_1)=\operatorname{Tr}(\Sigma)/n_r>0 Tr ( C 1 ) = Tr ( Σ ) / n r > 0 are constants, and Γ 1 , t − 1 ≲ N t − κ \Gamma_{1,t-1}\lesssim N_t^{-\kappa} Γ 1 , t − 1 ≲ N t − κ by Step 2 (Equation (134) ); a sum of two non-negative terms is ≲ \lesssim ≲ the sum of any two quantities where each ≲ \lesssim ≲ to the corresponding term.
Let us now split into three cases, using (140) for S t S_t S t asymptotics, and (141) for the final Tr ( C t ) \operatorname{Tr}(C_t) Tr ( C t ) asymptotics. We also use κ = 2 ( 1 − α ) \kappa=2(1-\alpha) κ = 2 ( 1 − α ) .
If 0 ≤ κ < 1 0\le\kappa<1 0 ≤ κ < 1 (equivalent to 1 2 < α ≤ 1 \frac{1}{2}<\alpha\le1 2 1 < α ≤ 1 ), we have S t ≲ N t − κ S_t\lesssim N_t^{-\kappa} S t ≲ N t − κ . Both terms in (141) have the same asymptotics, and we obtain Tr ( C t ) ≲ N t − κ = N t − 2 ( 1 − α ) \operatorname{Tr}(C_t)\lesssim N_t^{-\kappa}=N_t^{-2(1-\alpha)} Tr ( C t ) ≲ N t − κ = N t − 2 ( 1 − α ) .
If κ = 1 \kappa=1 κ = 1 (equivalent to α = 1 2 \alpha=\frac{1}{2} α = 2 1 ), we have S t ≲ N t − 1 ln ( N t ) S_t\lesssim N_t^{-1}\ln(N_t) S t ≲ N t − 1 ln ( N t ) . The second term in (141) dominates, and we obtain Tr ( C t ) ≲ N t − 1 ln ( N t ) \operatorname{Tr}(C_t)\lesssim N_t^{-1}\ln(N_t) Tr ( C t ) ≲ N t − 1 ln ( N t ) .
If 1 < κ ≤ 2 1<\kappa\le2 1 < κ ≤ 2 (equivalent to 0 ≤ α < 1 2 0\le\alpha<\frac{1}{2} 0 ≤ α < 2 1 ), we have S t ≲ N t − 1 S_t\lesssim N_t^{-1} S t ≲ N t − 1 . Together with N t − κ = o ( N t − 1 ) N_t^{-\kappa}=o(N_t^{-1}) N t − κ = o ( N t − 1 ) , the second term in (141) dominates, and we get Tr ( C t ) ≲ N t − 1 \operatorname{Tr}(C_t)\lesssim N_t^{-1} Tr ( C t ) ≲ N t − 1 .
Finally, we get
Tr ( C t ) ≲ { N t − 1 0 ≤ α < 1 2 N t − 1 ln ( N t ) α = 1 2 N t − 2 ( 1 − α ) 1 2 < α ≤ 1 \operatorname{Tr}(C_t)\lesssim\begin{cases}N_t^{-1} & 0\le\alpha<\frac{1}{2}\\ N_t^{-1}\ln(N_t) & \alpha=\frac{1}{2}\\ N_t^{-2(1-\alpha)} & \frac{1}{2}<\alpha\le1\end{cases} Tr ( C t ) ≲ ⎩ ⎨ ⎧ N t − 1 N t − 1 ln ( N t ) N t − 2 ( 1 − α ) 0 ≤ α < 2 1 α = 2 1 2 1 < α ≤ 1
Which is exactly (17). Note, that for α = 1 \alpha=1 α = 1 we get Tr ( C t ) ≲ 1 \operatorname{Tr}(C_t)\lesssim1 Tr ( C t ) ≲ 1 , which means that Tr ( C t ) \operatorname{Tr}(C_t) Tr ( C t ) stays bounded as t → ∞ t\to\infty t → ∞ .
Remark. When α = 0 \alpha=0 α = 0 , (17) gives Tr ( C t ) ≲ N t − 1 \operatorname{Tr}(C_t)\lesssim N_t^{-1} Tr ( C t ) ≲ N t − 1 , in agreement with the exact identity C t = Σ / N t C_t=\Sigma/N_t C t = Σ/ N t of Theorem A.11(c).
Additional result: the error of the mean estimator increases initially for large α \alpha α
In Figure 1 (a), the mean-square error of μ ( t ) \mu^{(t)} μ ( t ) first increases before it decreases when α \alpha α is large. The following proposition characterizes exactly when this happens, shows that the error then has a single peak, and bounds how large the increase can be.
Proposition A.27 (Initial growth of the error). Under Assumption 1, let α t ≡ α ∈ [ 0 , 1 ] \alpha_t \equiv\alpha\in[0,1] α t ≡ α ∈ [ 0 , 1 ] and n b ( t ) ≡ n b n_b(t) \equiv n_b n b ( t ) ≡ n b . Let
α ∗ ( n r , n b ) : = { n r 2 + ( n b − 1 ) ( n r + n b ) − n r n b − 1 , n b ≥ 2 , n r + 1 2 n r , n b = 1 , (142) \alpha^*(n_r,n_b) := \begin{cases}
\frac{\sqrt{n_r^2+(n_b-1)(n_r+n_b)}-n_r}{n_b-1}, & n_b \ge2,\\[6pt]
\frac{n_r+1}{2n_r}, & n_b=1,
\end{cases} \tag*{(142)} α ∗ ( n r , n b ) := ⎩ ⎨ ⎧ n b − 1 n r 2 + ( n b − 1 ) ( n r + n b ) − n r , 2 n r n r + 1 , n b ≥ 2 , n b = 1 , (142) which is the unique root in [ 0 , 1 ] [0,1] [ 0 , 1 ] of g ( α ) : = ( n b − 1 ) α 2 + 2 n r α − ( n r + n b ) g(\alpha):=(n_b-1)\alpha^2+2n_r\alpha-(n_r+n_b) g ( α ) := ( n b − 1 ) α 2 + 2 n r α − ( n r + n b ) and satisfies 1 2 < α ∗ ≤ 1 \frac{1}{2}<\alpha^*\le1 2 1 < α ∗ ≤ 1 . Then:
(a) Threshold. The error increases at the first iteration, c 2 > c 1 c_2>c_1 c 2 > c 1 , if and only if α > α ∗ \alpha>\alpha^* α > α ∗ . For α ≤ α ∗ \alpha\le\alpha^* α ≤ α ∗ , c t c_t c t is non-increasing in t t t .
(b) Single peak. There is t ∗ ∈ { 1 , 2 , … } ∪ { ∞ } t^*\in\{1,2,\ldots\}\cup\{\infty\} t ∗ ∈ { 1 , 2 , … } ∪ { ∞ } such that c t c_t c t is strictly increasing for t ≤ t ∗ t\le t^* t ≤ t ∗ and non-increasing for t ≥ t ∗ t\ge t^* t ≥ t ∗ . If α < 1 \alpha<1 α < 1 , then t ∗ < ∞ t^*<\infty t ∗ < ∞ and N t ∗ ≤ max { n r , n r 2 ( 1 − α ) + n b } N_{t^*}\le\max\left\{n_r,\frac{n_r}{2(1-\alpha)}+n_b\right\} N t ∗ ≤ max { n r , 2 ( 1 − α ) n r + n b } .
(c) Bounded growth. For every t t t , c t ≤ c 1 + ∑ s ≥ 1 β s ≤ 2 c 1 c_t\le c_1+\sum_{s\ge1}\beta_s\le2c_1 c t ≤ c 1 + ∑ s ≥ 1 β s ≤ 2 c 1 .
Remark (interpretation). When α > α ∗ \alpha>\alpha^* α > α ∗ , the error first rises above its initial value Tr ( Σ ) / n r \operatorname{Tr}(\Sigma)/n_r Tr ( Σ ) / n r , then decreases for α < 1 \alpha<1 α < 1 , and never comes back down at α = 1 \alpha=1 α = 1 . Three observations follow. First, since α ∗ > 1 2 \alpha^*>\frac{1}{2} α ∗ > 2 1 , the initial rise occurs only where the rate of Theorem 3.6 is already degraded. Second, α ∗ \alpha^* α ∗ depends only on n r n_r n r and n b n_b n b , and not on Σ \Sigma Σ or d d d , as c t c_t c t is independent of both. As n b / n r → ∞ n_b/n_r\to\infty n b / n r → ∞ , i.e., when each batch is much larger than the initial pool of real samples, we have α ∗ → 1 \alpha^*\to1 α ∗ → 1 : the real samples in the batch, ( 1 − α ) n b (1-\alpha)n_b ( 1 − α ) n b of them in expectation, outnumber the initial pool unless α \alpha α is close to 1, so the error rises only if almost the entire batch is generated. As n b / n r → 0 n_b/n_r\to0 n b / n r → 0 , i.e., when each batch is much smaller than the initial pool, we have α ∗ → 1 2 \alpha^*\to\frac{1}{2} α ∗ → 2 1 : the initial estimate is already accurate, so a mostly generated batch adds more error than it removes. Third, the rise is modest: by (c), the error at most doubles. For n r = 20 n_r=20 n r = 20 and n b = 10 n_b=10 n b = 10 , α ∗ ≈ 0.654 \alpha^*\approx0.654 α ∗ ≈ 0.654 , and at α = 1 \alpha=1 α = 1 the error rises to 1.74 times its initial value.
Proof. Throughout, α t ≡ α \alpha_t\equiv\alpha α t ≡ α and n b ( t ) ≡ n b n_b(t)\equiv n_b n b ( t ) ≡ n b , so that N t = n r + ( t − 1 ) n b N_t=n_r+(t-1)n_b N t = n r + ( t − 1 ) n b is increasing in t t t , and β t , γ t \beta_t,\gamma_t β t , γ t are given by (18).
Reduction to a scalar sequence. Taking traces in Theorem A.11(e) and using (11), we have E [ ∥ μ ( t ) − μ ∥ 2 2 ] = c t Tr ( Σ ) \mathbb{E}[\lVert\mu^{(t)}-\mu\rVert_2^2]=c_t\operatorname{Tr}(\Sigma) E [∥ μ ( t ) − μ ∥ 2 2 ] = c t Tr ( Σ ) . It therefore suffices to study the scalar sequence c t c_t c t defined in (40):
c 1 = 1 n r , c t + 1 = γ t c t + β t ( t ≥ 1 ) . c_1=\frac{1}{n_r},\qquad c_{t+1}=\gamma_t c_t+\beta_t\qquad(t\ge1). c 1 = n r 1 , c t + 1 = γ t c t + β t ( t ≥ 1 ) .
Step 1: important definitions and identities. We need to analyze the difference sequences:
c t + 1 − c t = γ t c t + β t − c t = β t − ( 1 − γ t ) c t = ( 1 − γ t ) ( β t 1 − γ t − c t ) = ( 1 − γ t ) ( θ t − c t ) . (143) \begin{aligned}
c_{t+1}-c_t&=\gamma_t c_t+\beta_t-c_t\\
&=\beta_t-(1-\gamma_t)c_t\\
&=(1-\gamma_t)\left(\frac{\beta_t}{1-\gamma_t}-c_t\right)\\
&=(1-\gamma_t)(\theta_t-c_t). \tag*{(143)}
\end{aligned} c t + 1 − c t = γ t c t + β t − c t = β t − ( 1 − γ t ) c t = ( 1 − γ t ) ( 1 − γ t β t − c t ) = ( 1 − γ t ) ( θ t − c t ) . (143)
We defined
θ t : = β t 1 − γ t = 1 ( 1 − α ) K t + α 2 , K t = 2 N t + n b ( 1 + α ) , (144) \theta_t:=\frac{\beta_t}{1-\gamma_t}=\frac{1}{(1-\alpha)K_t+\alpha^2},\qquad K_t=2N_t+n_b(1+\alpha), \tag*{(144)} θ t := 1 − γ t β t = ( 1 − α ) K t + α 2 1 , K t = 2 N t + n b ( 1 + α ) , (144)
where we used (36). Using (145), we have
c t + 1 − θ t = ( c t + 1 − c t ) + ( c t − θ t ) = − ( 1 − γ t ) ( c t − θ t ) + ( c t − θ t ) = γ t ( c t − θ t ) . (145) \begin{aligned}
c_{t+1}-\theta_t&=(c_{t+1}-c_t)+(c_t-\theta_t)\\
&=-(1-\gamma_t)(c_t-\theta_t)+(c_t-\theta_t)\\
&=\gamma_t(c_t-\theta_t). \tag*{(145)}
\end{aligned} c t + 1 − θ t = ( c t + 1 − c t ) + ( c t − θ t ) = − ( 1 − γ t ) ( c t − θ t ) + ( c t − θ t ) = γ t ( c t − θ t ) . (145)
Step 2: once the error stops increasing, it never increases again. Fix t ≥ 1 t\ge1 t ≥ 1 , and suppose that c t + 1 ≤ c t c_{t+1}\le c_t c t + 1 ≤ c t . Then, from (145) and γ t < 1 \gamma_t<1 γ t < 1 (Lemma A.9), we have θ t − c t ≤ 0 ⇒ c t ≥ θ t \theta_t-c_t\le0\Rightarrow c_t\ge\theta_t θ t − c t ≤ 0 ⇒ c t ≥ θ t . Thus, from (147) and γ t > 0 \gamma_t>0 γ t > 0 (Lemma A.9), we have
c t + 1 − θ t ≥ 0 ⇒ c t + 1 ≥ θ t . (146) c_{t+1}-\theta_t\ge0\Rightarrow c_{t+1}\ge\theta_t. \tag*{(146)} c t + 1 − θ t ≥ 0 ⇒ c t + 1 ≥ θ t . (146)
From (144) , θ t \theta_t θ t is non-increasing in t t t , thus θ t ≥ θ t + 1 \theta_t \ge\theta_{t+1} θ t ≥ θ t + 1 , and from (146) , we have c t + 1 ≥ θ t + 1 c_{t+1} \ge\theta_{t+1} c t + 1 ≥ θ t + 1 . Using (143) , we get c t + 2 − c t + 1 ≤ 0 ⇒ c t + 2 ≤ c t + 1 c_{t+2}-c_{t+1} \le0 \Rightarrow c_{t+2} \le c_{t+1} c t + 2 − c t + 1 ≤ 0 ⇒ c t + 2 ≤ c t + 1 . Hence, by induction c s + 1 ≤ c s c_{s+1} \le c_s c s + 1 ≤ c s for all s ≥ t s \ge t s ≥ t .
(a) Threshold. Using (143) and γ t < 1 \gamma_t < 1 γ t < 1 (Lemma A.9 ) at t = 1 t=1 t = 1 , c 2 > c 1 c_2 > c_1 c 2 > c 1 if and only if c 1 < θ 1 c_1 < \theta_1 c 1 < θ 1 . With c 1 = 1 / n r c_1=1/n_r c 1 = 1/ n r , N 1 = n r N_1=n_r N 1 = n r and (144) , this reads
1 n r < 1 ( 1 − α ) ( 2 n r + n b ( 1 + α ) ) + α 2 ⟺ ( 1 − α ) ( 2 n r + n b ( 1 + α ) ) + α 2 < n r \frac{1}{n_r}<\frac{1}{(1-\alpha)(2n_r+n_b(1+\alpha))+\alpha^2}\Longleftrightarrow(1-\alpha)(2n_r+n_b(1+\alpha))+\alpha^2<n_r n r 1 < ( 1 − α ) ( 2 n r + n b ( 1 + α )) + α 2 1 ⟺ ( 1 − α ) ( 2 n r + n b ( 1 + α )) + α 2 < n r
⟺ α 2 + n b ( 1 − α 2 ) + 2 n r ( 1 − α ) − n r < 0 \Longleftrightarrow\alpha^2+n_b(1-\alpha^2)+2n_r(1-\alpha)-n_r<0 ⟺ α 2 + n b ( 1 − α 2 ) + 2 n r ( 1 − α ) − n r < 0
⟺ ( 1 − n b ) α 2 − 2 n r α + ( n r + n b ) < 0 \Longleftrightarrow(1-n_b)\alpha^2-2n_r\alpha+(n_r+n_b)<0 ⟺ ( 1 − n b ) α 2 − 2 n r α + ( n r + n b ) < 0
Define
g ( α ) = ( n b − 1 ) α 2 + 2 n r α − ( n r + n b ) . (147) g(\alpha)=(n_b-1)\alpha^2+2n_r\alpha-(n_r+n_b). \tag*{(147)} g ( α ) = ( n b − 1 ) α 2 + 2 n r α − ( n r + n b ) . (147)
Thus, c 2 > c 1 c_2>c_1 c 2 > c 1 if and only if g ( α ) > 0 g(\alpha)>0 g ( α ) > 0 . On [ 0 , 1 ] [0,1] [ 0 , 1 ] , g g g is strictly increasing, as
g ′ ( α ) = 2 ( n b − 1 ) α + 2 n r > 0 , g'(\alpha)=2(n_b-1)\alpha+2n_r>0, g ′ ( α ) = 2 ( n b − 1 ) α + 2 n r > 0 ,
and
g ( 0 ) = − ( n r + n b ) < 0 , g ( 1 2 ) = − 3 n b + 1 4 < 0 , g ( 1 ) = n r − 1 ≥ 0. g(0)=-(n_r+n_b)<0,\qquad g\left(\frac{1}{2}\right)=-\frac{3n_b+1}{4}<0,\qquad g(1)=n_r-1\ge0. g ( 0 ) = − ( n r + n b ) < 0 , g ( 2 1 ) = − 4 3 n b + 1 < 0 , g ( 1 ) = n r − 1 ≥ 0.
Hence g g g has a unique root α ∗ ∈ ( 1 2 , 1 ] \alpha^*\in\left(\frac{1}{2},1\right] α ∗ ∈ ( 2 1 , 1 ] , which solving the quadratic g ( α ) = 0 g(\alpha)=0 g ( α ) = 0 shows to be given by (142) :
α ∗ ( n r , n b ) : = { n r 2 + ( n b − 1 ) ( n r + n b ) − n r n b − 1 , n b ≥ 2 , n r + 1 2 n r , n b = 1 , \alpha^*(n_r,n_b):=
\begin{cases}
\frac{\sqrt{n_r^2+(n_b-1)(n_r+n_b)}-n_r}{n_b-1}, & n_b\ge2,\\[6pt]
\frac{n_r+1}{2n_r}, & n_b=1,
\end{cases} α ∗ ( n r , n b ) := ⎩ ⎨ ⎧ n b − 1 n r 2 + ( n b − 1 ) ( n r + n b ) − n r , 2 n r n r + 1 , n b ≥ 2 , n b = 1 ,
And g ( α ) > 0 g(\alpha)>0 g ( α ) > 0 if and only if α > α ∗ \alpha>\alpha^* α > α ∗ . Finally, if α ≤ α ∗ \alpha\le\alpha^* α ≤ α ∗ , then c 2 ≤ c 1 c_2\le c_1 c 2 ≤ c 1 , and by Step 2 with t = 1 t=1 t = 1 , c t c_t c t is non-increasing for all t t t .
(b) Single peak. Let t ∗ : = min { t ≥ 1 : c t + 1 ≤ c t } t^*:=\min\{t\ge1:c_{t+1}\le c_t\} t ∗ := min { t ≥ 1 : c t + 1 ≤ c t } , with t ∗ = ∞ t^*=\infty t ∗ = ∞ if no such t t t exists. By definition, c t + 1 > c t c_{t+1}>c_t c t + 1 > c t for all t < t ∗ t<t^* t < t ∗ , so c t c_t c t is strictly increasing for t ≤ t ∗ t\le t^* t ≤ t ∗ . If t ∗ < ∞ t^*<\infty t ∗ < ∞ , Step 2 with t = t ∗ t=t^* t = t ∗ shows that c t c_t c t is non-increasing for t ≥ t ∗ t\ge t^* t ≥ t ∗ .
It remains to show that t ∗ < ∞ t^*<\infty t ∗ < ∞ when α < 1 \alpha<1 α < 1 , and to bound N t ∗ N_{t^*} N t ∗ . Since K t ≥ 2 N t K_t\ge2N_t K t ≥ 2 N t and α 2 ≥ 0 \alpha^2\ge0 α 2 ≥ 0 , (144) gives
θ t ≤ 1 2 ( 1 − α ) N t . \theta_t\le\frac{1}{2(1-\alpha)N_t}. θ t ≤ 2 ( 1 − α ) N t 1 .
Since N t → ∞ N_t\to\infty N t → ∞ , there is a first index t 1 t_1 t 1 with N t 1 ≥ n r / ( 2 ( 1 − α ) ) N_{t_1}\ge n_r/(2(1-\alpha)) N t 1 ≥ n r / ( 2 ( 1 − α )) , and then θ t 1 ≤ 1 / n r = c 1 \theta_{t_1}\le1/n_r=c_1 θ t 1 ≤ 1/ n r = c 1 . We claim that t ∗ ≤ t 1 t^*\le t_1 t ∗ ≤ t 1 . Assume, by contradiction, that t ∗ > t 1 t^*>t_1 t ∗ > t 1 . Then c t c_t c t is increasing up to t 1 t_1 t 1 , so c t 1 ≥ c 1 ≥ θ t 1 c_{t_1}\ge c_1\ge\theta_{t_1} c t 1 ≥ c 1 ≥ θ t 1 . By (143) , c t 1 + 1 − c t 1 = ( 1 − γ t 1 ) ( θ t 1 − c t 1 ) ≤ 0 c_{t_1+1}-c_{t_1}=(1-\gamma_{t_1})(\theta_{t_1}-c_{t_1})\le0 c t 1 + 1 − c t 1 = ( 1 − γ t 1 ) ( θ t 1 − c t 1 ) ≤ 0 , so t 1 t_1 t 1 belongs to the set defining t ∗ t^* t ∗ . Hence t ∗ ≤ t 1 t^*\le t_1 t ∗ ≤ t 1 , which is a contradiction. Therefore, t ∗ ≤ t 1 t^*\le t_1 t ∗ ≤ t 1 .
Since N t N_t N t is increasing, N t ∗ ≤ N t 1 N_{t^*}\le N_{t_1} N t ∗ ≤ N t 1 . If t 1 = 1 t_1=1 t 1 = 1 , then N t 1 = n r N_{t_1}=n_r N t 1 = n r . Otherwise, by the minimality of t 1 t_1 t 1 , N t 1 − 1 < n r / ( 2 ( 1 − α ) ) N_{t_1-1}<n_r/(2(1-\alpha)) N t 1 − 1 < n r / ( 2 ( 1 − α )) , and hence N t 1 = N t 1 − 1 + n b < n r / ( 2 ( 1 − α ) ) + n b N_{t_1}=N_{t_1-1}+n_b<n_r/(2(1-\alpha))+n_b N t 1 = N t 1 − 1 + n b < n r / ( 2 ( 1 − α )) + n b . In both cases,
N t ∗ ≤ max { n r , n r 2 ( 1 − α ) + n b } . N_{t^*}\le\max\left\{n_r,\frac{n_r}{2(1-\alpha)}+n_b\right\}. N t ∗ ≤ max { n r , 2 ( 1 − α ) n r + n b } .
(c) Bounded growth. Since γ t < 1 \gamma_t<1 γ t < 1 (Lemma A.9 ),
c t + 1 = γ t c t + β t ≤ c t + β t . c_{t+1}=\gamma_tc_t+\beta_t\le c_t+\beta_t. c t + 1 = γ t c t + β t ≤ c t + β t .
Summing over t ≥ 1 t\ge1 t ≥ 1 and using Lemma A.5, we get
c t ≤ c 1 + ∑ s ≥ 1 β s ≤ 1 n r + 1 n r = 2 c 1 . c_t\le c_1+\sum_{s\ge1}\beta_s\le\frac{1}{n_r}+\frac{1}{n_r}=2c_1. c t ≤ c 1 + s ≥ 1 ∑ β s ≤ n r 1 + n r 1 = 2 c 1 .
The value of identifying real samples
We quantify the gain from perfect detection of synthetic samples under the setting of Theorem 3.6 , i.e., α t ≡ α \alpha_t \equiv\alpha α t ≡ α and n b ( t ) ≡ n b n_b(t) \equiv n_b n b ( t ) ≡ n b . The pool contains, in expectation, N o r a c l e : = ( 1 − α ) N t + α n r N_{\mathrm{oracle}} := (1-\alpha)N_t + \alpha n_r N oracle := ( 1 − α ) N t + α n r real samples, which is approximately the sample size available to an oracle that identifies and keeps only them. On the other hand, the sample mean of N N N i.i.d. real samples has an error of Tr ( Σ ) / N \operatorname{Tr}(\Sigma)/N Tr ( Σ ) / N , so the pooled estimator has the effective sample size N e f f : = Tr ( Σ ) / E [ ∥ μ ( t ) − μ ∥ 2 2 ] = Tr ( Σ ) / Tr ( C t ) N_{\mathrm{eff}} := \operatorname{Tr}(\Sigma)/\mathbb{E}[\|\mu^{(t)}-\mu\|_2^2] = \operatorname{Tr}(\Sigma)/\operatorname{Tr}(C_t) N eff := Tr ( Σ ) / E [ ∥ μ ( t ) − μ ∥ 2 2 ] = Tr ( Σ ) / Tr ( C t ) (by (11) ), i.e., the number of i.i.d. real samples that attain the same error. The gain from perfect detection is the ratio
G α ( N t ) : = N o r a c l e N e f f = ( ( 1 − α ) N t + α n r ) Tr ( C t ) Tr ( Σ ) . (148) G_\alpha(N_t) := \frac{N_{\mathrm{oracle}}}{N_{\mathrm{eff}}} = ((1-\alpha)N_t+\alpha n_r)\frac{\operatorname{Tr}(C_t)}{\operatorname{Tr}(\Sigma)}. \tag*{(148)} G α ( N t ) := N eff N oracle = (( 1 − α ) N t + α n r ) Tr ( Σ ) Tr ( C t ) . (148)
As N t → ∞ N_t \to\infty N t → ∞ , Theorem 3.6 gives
G α ( N t ) ≍ { 1 , 0 ≤ α < 1 2 , ln N t , α = 1 2 , N t 2 α − 1 , 1 2 < α < 1. (149) G_\alpha(N_t) \asymp\begin{cases}
1, & 0 \le\alpha< \frac{1}{2},\\
\ln N_t, & \alpha= \frac{1}{2},\\
N_t^{2\alpha-1}, & \frac{1}{2} < \alpha< 1.
\end{cases} \tag*{(149)} G α ( N t ) ≍ ⎩ ⎨ ⎧ 1 , ln N t , N t 2 α − 1 , 0 ≤ α < 2 1 , α = 2 1 , 2 1 < α < 1. (149)
In words, when generated samples are a minority of each new batch, perfect detection improves the effective sample size by at most a constant factor, no matter how much data is collected. When they are a majority, the improvement grows without bound as more data is collected. Moreover, the gain grows with α \alpha α in the long run: for any 0 ≤ α < α ′ < 1 0 \le\alpha< \alpha' < 1 0 ≤ α < α ′ < 1 with α ′ > 1 2 \alpha' > \frac{1}{2} α ′ > 2 1 , we have G α ′ ( N t ) / G α ( N t ) → ∞ G_{\alpha'}(N_t)/G_\alpha(N_t) \to\infty G α ′ ( N t ) / G α ( N t ) → ∞ as N t → ∞ N_t \to\infty N t → ∞ . Thus, asymptotically, the gain from detection is largest exactly where the pooled estimator converges most slowly.
At a fixed, finite N t N_t N t , however, the gain does not increase all the way to α = 1 \alpha=1 α = 1 . At the two extremes,
G 0 ( N t ) = 1 , 1 ≤ G 1 ( N t ) = n r Tr ( C t ) Tr ( Σ ) ≤ 2 , (150) G_0(N_t)=1,\qquad1 \le G_1(N_t)=n_r\frac{\operatorname{Tr}(C_t)}{\operatorname{Tr}(\Sigma)} \le2, \tag*{(150)} G 0 ( N t ) = 1 , 1 ≤ G 1 ( N t ) = n r Tr ( Σ ) Tr ( C t ) ≤ 2 , (150)
where the bounds follow from Proposition 3.5 and Lemma 3.2 . At α = 0 \alpha=0 α = 0 there is nothing to detect, and at α = 1 \alpha=1 α = 1 the oracle retains only the n r n_r n r initial real samples. For fixed t t t , G α ( N t ) G_\alpha(N_t) G α ( N t ) is continuous in α \alpha α , and by (151), for any fixed α ∈ ( 1 2 , 1 ) \alpha\in(\frac{1}{2},1) α ∈ ( 2 1 , 1 ) it exceeds 2 once N t N_t N t is large enough. Hence, for large but finite N t N_t N t , the gain is maximized at some α \alpha α strictly between 0 and 1. In practice, detection is therefore worth most when generated data dominates, but real data is still being added in meaningful amounts. The potential for such detection to improve performance is supported empirically by prior work on identifying and curating model-generated data [7 ] .
Additional details on empirical verification and discussion
Under the same setup as § 3.3 , we present a numerical verification of the theoretical recursion of Tr ( C t ) = E [ ∥ μ ( t ) − μ ∥ 2 2 ] \operatorname{Tr}(C_t)=\mathbb{E}[\|\mu^{(t)}-\mu\|_2^2] Tr ( C t ) = E [ ∥ μ ( t ) − μ ∥ 2 2 ] (Equation (11) and Theorem A.11 ).
Exact mean-square error. The recursion of Theorem A.11 gives the mean-square error of μ ( t ) \mu^{(t)} μ ( t ) exactly, without sampling; Figure 2 (a) and (b) use these exact values. Figure 3 compares them with the Monte-Carlo averages of § 3.3 : in all 19 regimes and across the whole range of N t N_t N t , the two differ by at most 3.4 standard errors, i.e., only by Monte-Carlo noise.
Finally, we verify numerically that μ ( t ) \mu^{(t)} μ ( t ) is unbiased and that the bias of Σ ( t ) \Sigma^{(t)} Σ ( t ) is exactly − C t -C_t − C t (Theorem 3.1 ). Both checks use T = 400 T=400 T = 400 iterations, so that N t N_t N t ranges up to about 4000.
Unbiasedness of the mean. Theorem 3.1 (a) states that E [ μ ( t ) ] = μ \mathbb{E}[\mu^{(t)}]=\mu E [ μ ( t ) ] = μ for every t ≥ 1 t \ge1 t ≥ 1 . Figure 4 illustrates this with M = 50,000 M=50{,}000 M = 50 , 000 independent runs of T = 400 T=400 T = 400 iterations. In all four regimes, the average of each coordinate lies on its true value at every N t N_t N t . The shaded bands show that this is not because single runs are accurate: they scatter widely, and at α = 0.9 \alpha=0.9 α = 0.9 and α = 1 \alpha=1 α = 1 the band barely narrows, since the estimator converges slowly or not at all. At α = 1 \alpha=1 α = 1 , in particular, μ ( t ) \mu^{(t)} μ ( t ) is unbiased without being consistent, in line with Proposition 3.5 . To quantify the agreement beyond what the figure can resolve, we compare the Monte-Carlo average with the true value in units of its standard error: over all 19 regimes, all 400 iterations and all four coordinates, the difference never exceeds 2.81 standard errors, as expected from Monte-Carlo noise alone.
Figure 4. Unbiasedness of μ ( t ) \mu^{(t)} μ ( t ) for four constant values of α \alpha α . For each coordinate i i i , the colored line is the Monte-Carlo average of μ i ( t ) \mu_i^{(t)} μ i ( t ) , the wide gray line underneath is the true value μ i \mu_i μ i , and the shaded band is ± 1 \pm1 ± 1 standard deviation of a single run’s μ i ( t ) \mu_i^{(t)} μ i ( t ) .
Bias of the covariance. Unlike the mean, Σ ( t ) \Sigma^{(t)} Σ ( t ) is biased: by Theorem 3.1 (b), E [ Σ ( t ) ] − Σ = − C t \mathbb{E}[\Sigma^{(t)}]-\Sigma=-C_t E [ Σ ( t ) ] − Σ = − C t , so the covariance is underestimated by exactly the covariance of the mean estimator. Figure 5 checks this for the trace. In all four regimes, the measured bias lies on the predicted Tr ( C t ) \operatorname{Tr}(C_t) Tr ( C t ) over the whole range: it decays like N t − 1 N_t^{-1} N t − 1 at α = 0 \alpha=0 α = 0 , more slowly at α = 0.5 \alpha=0.5 α = 0.5 and α = 0.9 \alpha=0.9 α = 0.9 , and at α = 1 \alpha=1 α = 1 it grows and then levels off at a positive value, so the covariance estimator remains biased forever (Proposition 3.5 ). Across all 19 regimes and all 400 iterations, the measured bias differs from the prediction by at most 3.19 standard errors, as expected from Monte-Carlo noise alone.
Figure 5. The bias of Σ ( t ) \Sigma^{(t)} Σ ( t ) for four constant values of α \alpha α . Lines are the predicted downward bias of the trace, Tr ( C t ) \operatorname{Tr}(C_t) Tr ( C t ) , computed from the recursion of Theorem A.11 ; circles are the measured Tr ( Σ ) − Tr ( E b [ Σ ( t ) ] ) \operatorname{Tr}(\Sigma)-\operatorname{Tr}(\mathbb{E}_b[\Sigma^{(t)}]) Tr ( Σ ) − Tr ( E b [ Σ ( t ) ]) , averaged over 2 × 10 6 2 \times10^6 2 × 1 0 6 Monte-Carlo runs.
Experiments with Diffusion Models
While our paper focuses on theoretical contributions, in this section we empirically examine the extent to which the behavior observed in a practical deep learning setting is consistent with our theoretical results.
Data and protocol. We consider CIFAR-10 (Krizhevsky et al. [15 ] ) and a diffusion model (Sohl-Dickstein et al. [23 ] ; Song et al. [25 ] ) based on the DDPM framework (Ho et al. [12 ] ), using the U-Net implementation of Zou (https://github.com/zoubohao/DenoisingDiffusionProbabilityModel-ddpm-), and train unconditional models. We split the 50,000 CIFAR-10 training images into a class-balanced validation set of 5,000 images and 45,000 images available for training; the 10,000 test images are used only for evaluation. We set n r = n b = 5,000 n_r=n_b=5{,}000 n r = n b = 5 , 000 and t max = 9 t_{\max}=9 t m a x = 9 , so that the pool at iteration t t t contains N t = n r + ( t − 1 ) n b N_t=n_r+(t-1)n_b N t = n r + ( t − 1 ) n b images and the final pool, N 9 = 45,000 N_9=45{,}000 N 9 = 45 , 000 , uses all available training images. We consider four fixed mixing fractions, α ∈ { 0 , 0.25 , 0.75 , 1 } \alpha\in\{0, 0.25, 0.75, 1\} α ∈ { 0 , 0.25 , 0.75 , 1 } . At t = 1 t=1 t = 1 we train a DDPM on n r n_r n r class-balanced real images; this model is shared by all four values of α \alpha α . At each iteration t ≥ 2 t\geq2 t ≥ 2 , we accumulate the previous pool with α n b \alpha n_b α n b samples generated by the model of iteration t − 1 t-1 t − 1 and ( 1 − α ) n b (1-\alpha)n_b ( 1 − α ) n b previously unused, class-balanced real images, and the training treats real and generated images identically: minibatches are drawn uniformly from the pool, with no labels, reweighting or filtering.
Unlike the theoretical setup, where each sample is drawn from the mixture independently, here the numbers of generated and real images in each batch are fixed. This keeps the consumption of real images exact and class-balanced, and differs negligibly from independent draws, whose count of generated images fluctuates by less than 1% of the batch (one standard deviation) for n b = 5,000 n_b=5{,}000 n b = 5 , 000 .
Training. All models share one recipe: 1,000 diffusion steps with a linear noise schedule from 10 − 4 10^{-4} 1 0 − 4 to 0.02, the ϵ \epsilon ϵ -prediction objective, Adam with learning rate 2 × 10 − 4 2 \times10^{-4} 2 × 1 0 − 4 after a 5,000-step linear warm-up, batch size 128, gradient clipping at 1, dropout 0.1, and an exponential moving average (EMA) of the weights with decay 0.9999. All sampling uses the EMA weights and DDIM Song et al., [24 ] with 200 steps. We fix the number of optimization steps rather than the number of epochs, so that pool size is not confounded with the amount of computation. In each iteration t t t the training is from scratch on the whole pool, except for α = 0.75 \alpha= 0.75 α = 0.75 and t ≥ 4 t \ge4 t ≥ 4 , where we use warm-start from the previous iteration’s model with learning rate of lower order.
Checkpoint selection. Since diffusion models overfit on datasets of this size, each model should be trained neither too little nor too long, so that pools of different sizes are compared fairly. We save a checkpoint every 50,000 steps and select the one minimizing the FID between 10,000 generated samples and the validation set; we report the FID of the same samples against the test set, which is never used for selection. The default budget is 200,000 steps; iterations in which the validation FID was still decreasing at the end of the budget were retrained with a larger budget (up to 800,000 steps), which was needed only for α ∈ { 0 , 0.25 } \alpha\in\{0, 0.25\} α ∈ { 0 , 0.25 } . Validation and test FIDs selected the same checkpoint in every iteration, and over all checkpoints they differed by a nearly constant offset of 2.44 ± \pm ± 0.07.
FID values. We compute FID with clean-fid Parmar et al., [17 ] from 10,000 DDIM samples against the held-out test set. Published DDPM results on CIFAR-10 (e.g., 3.17 in Ho et al. [12 ] ) use 50,000 samples, 1,000-step sampling and training-set statistics, which yields substantially lower values: for our best model (α = 0 \alpha= 0 α = 0 , t = 9 t = 9 t = 9 , trained on 45,000 images), 50,000 samples against the training set give 5.1, compared with 9.7 on the test set. Our FIDs are therefore comparable across the curves of Figure 6 , but not to published numbers.
Figure 6. Test FID of DDPM on CIFAR-10 as a function of the pool size N t N_t N t , for mixing fractions α ∈ { 0 , 0.25 , 0.75 , 1 } \alpha\in\{0, 0.25, 0.75, 1\} α ∈ { 0 , 0.25 , 0.75 , 1 } (single seed). At each iteration, the checkpoint with the lowest validation FID is selected and its test FID is reported. The dashed line is the α = 0 \alpha= 0 α = 0 model at t = 9 t = 9 t = 9 , trained on all 45,000 training images (FID 9.7 on the test set, and 5.1 with 50,000 samples against the training set; see text). Differences below ≈ 0.5 \approx0.5 ≈ 0.5 FID between consecutive iterations are within run-to-run variation.
Reporting. We report the test FID of the selected checkpoint as a function of N t N_t N t for each α \alpha α , for a single training seed. The horizontal line is the α = 0 \alpha= 0 α = 0 model at t = 9 t = 9 t = 9 , i.e., a DDPM trained on all 45,000 training images with the same protocol. From duplicate runs, differences below ≈ 0.5 \approx0.5 ≈ 0.5 FID between consecutive iterations are within run-to-run variation. We expect the curves for α < 1 \alpha< 1 α < 1 to approach this reference value, whereas for α = 1 \alpha= 1 α = 1 , where no new real data are introduced, we expect the FID to approach a finite limiting value.
All experiments were run on a server with eight NVIDIA RTX 6000 Ada GPUs, training one model per GPU. Training costs about 3.9 GPU-hours per 100,000 steps including checkpoint evaluation, e.g., about 8, 15 and 31 hours for budgets of 200,000, 400,000 and 800,000 steps; the runs shown in Figure 6 total roughly 400 GPU-hours.
Comparison with the theory. The central prediction of our theory holds: although real and generated images are pooled and treated identically, with no detection, labeling or reweighting, every α < 1 \alpha< 1 α < 1 improves on the initial model, whereas α = 1 \alpha= 1 α = 1 degrades. At every iteration t ≥ 2 t \ge2 t ≥ 2 , the FID for α ∈ { 0 , 0.25 , 0.75 } \alpha\in\{0, 0.25, 0.75\} α ∈ { 0 , 0.25 , 0.75 } is below its initial value of 33.8, while for α = 1 \alpha= 1 α = 1 it stays fallen below it. Thus, a fixed fraction of fresh real data suffices to avoid collapse even when it cannot be told apart from generated data. This comparison is meaningful beyond the Gaussian setting because FID is the squared 2-Wasserstein distance between Gaussians fitted to the Inception features of generated and test images, i.e., the quantity of Theorem 3.3 evaluated in a feature space. The finer predictions carry over as well. First, the curves are ordered by α \alpha α at every iteration t ≥ 2 t \ge2 t ≥ 2 : a larger fraction of generated data always yields a worse model. Second, the improvement slows as α \alpha α grows, in line with Theorem 3.6 : from N 1 = 5,000 N_1 = 5{,}000 N 1 = 5 , 000 to N 3 = 15,000 N_3 = 15{,}000 N 3 = 15 , 000 , the FID decreases by 19.9 for α = 0 \alpha= 0 α = 0 , 16.7 for α = 0.25 \alpha= 0.25 α = 0.25 and only 6.8 for α = 0.75 \alpha= 0.75 α = 0.75 , the one value above 1 2 \frac{1}{2} 2 1 , whose predicted rate N t − 1 / 2 N_t^{-1/2} N t − 1/2 is slower than the rate N t − 1 N_t^{-1} N t − 1 of α < 1 2 \alpha< \frac{1}{2} α < 2 1 . For α = 0.25 \alpha= 0.25 α = 0.25 , the curve follows that of α = 0 \alpha= 0 α = 0 with a roughly constant offset of 3–4 FID, consistent with synthetic data inflating the constant but not the exponent. Third, at α = 1 \alpha= 1 α = 1 the FID rises from 33.8 to 39.7, with increments that shrink over the last iterations, matching the bounded, non-decreasing error of Propositions 3.5 and A.27 .
The value of identifying real samples (§ 3.2.3 ) is also visible. Discarding all generated images would leave, at t = 9 t = 9 t = 9 , 35,000 real images for α = 0.25 \alpha= 0.25 α = 0.25 and 15,000 for α = 0.75 \alpha= 0.75 α = 0.75 , i.e., the α = 0 \alpha= 0 α = 0 model at N t = 35,000 N_t = 35{,}000 N t = 35 , 000 (FID 10.3) and at N t = 15,000 N_t = 15{,}000 N t = 15 , 000 (FID 13.8). Compared with FIDs of 13.5 and above 27, respectively, perfect detection would gain about 3 FID for α = 0.25 < 1 2 \alpha= 0.25 < \frac{1}{2} α = 0.25 < 2 1 , but more than 13 for α = 0.75 > 1 2 \alpha= 0.75 > \frac{1}{2} α = 0.75 > 2 1 , as the theory predicts.