betaFeedback

WorksMathematics

Model Collapse Is Avoidable Even as Real Data Vanishes: A Sharp Criterion for Iterative Gaussian Estimation

Abstract

Large generative models are commonly trained on data collected from the open web, which may contain data produced by previous generations of the same model. Previous work has shown that recursively training on model-generated data can progressively degrade model performance, a phenomenon known as model collapse. Yet, generative models continue to improve dramatically over time. Existing theory suggests that this is enabled by the availability of new real data at each iteration, kept separate from generated data. However, exact separation is not realistic, and uncurated data is still commonly used. We bridge this gap by considering a more realistic setup in which real and generated data are not handled differently during training. Specifically, we present a comprehensive theoretical study of iterative estimation of the mean and covariance of a multivariate Gaussian distribution. At each iteration, samples are drawn from a mixture of the real data distribution and the estimated distribution from the previous iteration, but are used indistinguishably. We provide a single equivalent condition for consistency of the mean and covariance estimators in Euclidean norms, and convergence to zero of the expected squared 2-Wasserstein distance between the estimated and true distributions. Using this result, we show that consistency holds, i.e., model collapse is avoided, for any fixed real data fraction above zero, and even for a fraction that vanishes across iterations, provided it does not vanish too quickly. However, more real data (or data curation) remains important at practical finite times: the convergence rate of the mean estimator deteriorates as the real data fraction approaches zero. We support our results with an empirical investigation of the theoretical setup, and with experiments on iterative training of a diffusion model on CIFAR-10 that correlate with our theory.

Introduction

A growing body of empirical and theoretical work has raised the concern that machine learning generative models trained iteratively on data produced by earlier generations of themselves — rather than on real, independently sourced data — can drift away from the true underlying distribution over successive generations, a phenomenon often called model collapse [22, 1]. As more of the data on which models are trained is itself generated by other models, understanding the conditions under which this recursive process is destructive to the model and how we can deal with that has become an important theoretical question [3, 10, 14, 21, 5, 4, 2, 11, 8].

There are several approaches to iterative retraining, two of which are commonly referred to as Accumulate and Replace. In the Accumulate method, we start with a set of real data, and at each iteration, we augment the training set with newly generated data while retaining all data from previous iterations. In contrast, in the Replace method, the training set at each iteration consists only of the newly generated data from the previous iteration; all data from earlier iterations are discarded. The Accumulate method is generally considered a more realistic setting for iterative retraining and has been shown to exhibit degradation that remains bounded across iterations [10, 14, 21]. In contrast, the Replace method can lead to unbounded degradation, with the learned distribution progressively diverging from the original data distribution [22, 14].

Existing literature tries to theoretically explain the dramatic improvement of current top-tier generative models. This is done by considering the existence of independent real data in each iteration. However, it has only been investigated under the assumption that the real data can be perfectly separated from the generated data [1, 11, 8]. In this case, an optimal weighting scheme for the separate real and generated data used for training has been studied [11]. However, exact separation is not realistic [19, 16], and uncurated data is still commonly used. In this paper, to the best of our knowledge, we are the first to bridge this gap by considering a less restrictive and more realistic setup in which real and generated data are not handled differently during training.

We study Accumulate-based iterative estimation of the mean μ∈Rd\boldsymbol{\mu} \in\mathbb{R}^{d} and covariance matrix Σ∈Rd×d\boldsymbol{\Sigma} \in\mathbb{R}^{d \times d} of a Gaussian data distribution N(μ,Σ)\mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma}), a standard setting in theoretical works on model collapse [22, 1, 14, 11]. At each round tt, a batch of new samples is drawn from a mixture of (i) the current fitted Gaussian model and (ii) the true distribution, and all data accumulated so far is used to refit the mean and covariance by standard empirical estimators, μ(t)\boldsymbol{\mu}^{(t)} and Σ(t)\boldsymbol{\Sigma}^{(t)}, but without distinguishing between real and generated samples. This setting strips away the complications of neural networks while retaining the essential recursive feedback loop and admits rigorous analysis.

Our guiding question is: under what conditions does this process not collapse. In other words, when do the iterated estimators (μ(t),Σ(t))(\boldsymbol{\mu}^{(t)}, \boldsymbol{\Sigma}^{(t)}) converge to the true (μ,Σ)(\boldsymbol{\mu}, \boldsymbol{\Sigma}), and when do they instead settle on a biased, degenerate limit? We answer this question completely for both estimators. For the mean estimator, we also analyze the rate of convergence. Here, the mixing fraction αt∈[0,1]\alpha_t \in[0, 1], the expected fraction of data in the new batch drawn from the current model, plays a decisive role, and we provide an explicit, checkable condition on how it may vary across rounds while still guaranteeing consistency. That separates collapse from non-collapse. Our main contributions include:

  • We prove that the mean estimator μ(t)\boldsymbol{\mu}^{(t)} is unbiased, while the covariance estimator Σ(t)\boldsymbol{\Sigma}^{(t)} is biased, with the exact bias given by −Cov⁡(μ(t))-\operatorname{Cov}(\boldsymbol{\mu}^{(t)}).

  • We provide a closed-form equivalent condition for the consistency of the estimated mean in L2L^2, the consistency of the covariance estimator in expected Frobenius norm, and the convergence to zero of the expected squared 2-Wasserstein distance between the estimated and true distributions.

  • We use our result to show that several scenarios that naturally arise when scraping the web for data are sufficient to guarantee convergence, thereby preventing model collapse. First, convergence holds for an arbitrarily small but fixed positive fraction of real data (1−α)(1-\alpha) at every iteration. Second, convergence still holds when the fraction of real data vanishes across iterations, provided that it does not vanish too quickly.

  • For a fixed synthetic fraction 0≤α<10 \leq\alpha< 1 (i.e., when not all data is generated), we derive the exact rate of convergence (up to constant factors) of the mean estimator to the true mean. Specifically, we prove that if less than half of each new batch is generated, then the rate of convergence matches the optimal rate. However, when more than half of each new batch is generated, the rate of convergence becomes arbitrarily slow as the fraction of synthetic data increases. This result motivates the ongoing research on the detection and curation of generated samples: it shows that for practical finite tt in the high-synthetic regime, synthetic data detection can substantially improve estimation accuracy.

  • We present an empirical investigation of the theoretical model that supports our theoretical results and provides further insight into the rate of convergence.

  • We complement our theoretical work with an empirical study of how different fixed real-data ratios affect the FID of a diffusion model iteratively trained on CIFAR-10, showing results that correlate with our theory.

Related work

Existing works have shown that under the Replace workflow, iterative retraining degrades without bound [22, 4, 14], whereas under the Accumulate workflow the degradation remains bounded [10, 14, 21].

Several works show that collapse can be avoided, but under restrictive conditions. Bertrand et al. [3] prove the stability of iterative retraining on a mixture of real and synthetic data, but only locally, when the initial model is sufficiently close to the data distribution and the proportion of real data is sufficiently large. Barzilai & Shamir [2] provide non-asymptotic guarantees under data accumulation; however, for a fixed sample size nn, their bound holds only up to a finite number of iterations tt that depends on nn, and thus does not cover the regime of fixed nn with t→∞t \to\infty. Moreover, beyond an initial (or fixed) real dataset, these works do not incorporate new real data at subsequent iterations.

In practice, data scraped from the web at each iteration contains new real data alongside generated data. Several works study this setting [1, 11, 8], but either discard earlier data or assume that real and generated samples can be told apart, e.g., in order to weight them separately. The latter is unrealistic, since reliably detecting generated content remains an open problem [19, 16], although detection, when possible, can substantially mitigate collapse [7]. To the best of our knowledge, no prior work analyzes iterative retraining with accumulated data in which fresh real samples are not distinguished from generated ones.

Theory

In this section, we formalize the iterative retraining process, define the corresponding mean and covariance estimators, and state our main theoretical results, which characterize when the estimators converge to the true parameters and the rate at which this convergence occurs. Finally, we validate our results through experiments and provide additional insights into the rate of decay.

Problem setup: data generating process and estimators

In the theoretical model, we consider the accumulate method, where the process is initialized with a dataset of real samples. Similar to a vast body of theoretical work on model collapse [22, 1, 14, 4, 11, 5], we assume that the real data distribution is a multivariate Gaussian distribution. At each iteration, the training dataset is augmented with both generated and real data. Since the model does not know which samples are real and which are generated, it treats all samples identically and estimates the underlying distribution using the entire accumulated dataset. Formally, at each iteration, each newly added sample is drawn from the real distribution with some probability and from the current estimated distribution with the complementary probability.

Data-generating process. Let μ∈Rd\mu\in\mathbb{R}^{d}, and let Σ∈Rd×d\Sigma\in\mathbb{R}^{d \times d} be symmetric positive-definite. The process unfolds in discrete iterations t=1,2,3,…t = 1, 2, 3, \ldots

Iteration 1 (real data). Observe nr≥1n_r \ge1 i.i.d. samples {xj(0)}j=1nr∼N(μ,Σ)\{x_j^{(0)}\}_{j=1}^{n_r} \sim\mathcal{N}(\mu, \Sigma) and form the empirical mean and covariance

μ(1)=1nr∑j=1nrxj(0),Σ(1)=1nr∑j=1nr(xj(0)−μ(1))(xj(0)−μ(1))T.(1)\mu^{(1)} = \frac{1}{n_r} \sum_{j=1}^{n_r} x_j^{(0)}, \qquad\Sigma^{(1)} = \frac{1}{n_r} \sum_{j=1}^{n_r} (x_j^{(0)} - \mu^{(1)})(x_j^{(0)} - \mu^{(1)})^{\mathsf{T}}. \tag*{(1)}

Iteration t≥2t \ge2 (mixed data). Given μ(t−1)\mu^{(t-1)} and Σ(t−1)\Sigma^{(t-1)}, draw nb(t−1)≥1n_b(t-1) \ge1 i.i.d. samples from the mixture

xi(t−1)∼αt−1N(μ(t−1),Σ(t−1))+(1−αt−1)N(μ,Σ),αt−1∈[0,1],(2)x_i^{(t-1)} \sim\alpha_{t-1}\mathcal{N}\left(\mu^{(t-1)}, \Sigma^{(t-1)}\right) + (1-\alpha_{t-1})\mathcal{N}(\mu, \Sigma), \qquad\alpha_{t-1} \in[0,1], \tag*{(2)}

where 1≤i≤nb(t−1)1 \le i \le n_b(t-1). The parameter αt−1∈[0,1]\alpha_{t-1} \in[0,1] controls the fraction of the new batch that is drawn from the current model. Specifically, αt−1=1\alpha_{t-1} = 1 means the new batch is entirely synthetic, while αt−1=0\alpha_{t-1} = 0 means it is entirely real. Denote the total number of samples at iteration tt by NtN_t and the inverse ratio of the growth by ρt\rho_t:

Nt:=nr+∑s=1t−1nb(s),ρt:=NtNt+1(t≥1).(3)N_t := n_r + \sum_{s=1}^{t-1} n_b(s), \qquad\rho_t := \frac{N_t}{N_{t+1}} \qquad(t \ge1). \tag*{(3)}

Note that ρt∈(0,1)\rho_t \in(0,1) for every t≥1t \ge1.

Estimators. At every round, the model is refit using the pooled empirical mean and covariance of all samples accumulated so far. For every t≥1t \ge1, we write y1,…,yNty_1,\ldots,y_{N_t} for the NtN_t accumulated points, consisting of the original real batch {xj(0)}j=1nr\{x_j^{(0)}\}_{j=1}^{n_r} and the accumulated batches {xi(s)}i=1nb(s)\{x_i^{(s)}\}_{i=1}^{n_b(s)}, 1≤s≤t−11 \le s \le t-1. Define the pooled empirical estimators:

μ(t):=1Nt∑k=1Ntyk,Σ(t):=1Nt∑k=1Nt(yk−μ(t))(yk−μ(t))T,(4)\mu^{(t)} := \frac{1}{N_t}\sum_{k=1}^{N_t} y_k, \qquad\Sigma^{(t)} := \frac{1}{N_t}\sum_{k=1}^{N_t}(y_k-\mu^{(t)})(y_k-\mu^{(t)})^{\mathsf{T}}, \tag*{(4)}

which for t=1t=1 coincides with (1). These iterative estimators are standard and widely used in the literature [22, 14]. Conveniently, they admit online recursions in terms of the previous estimates and the newly drawn batch. Finally, we write

Dt:=N(μ(t),Σ(t)),Dreal:=N(μ,Σ),(5)\mathcal{D}_t := \mathcal{N}(\mu^{(t)},\Sigma^{(t)}), \qquad\mathcal{D}_{\mathrm{real}} := \mathcal{N}(\mu,\Sigma), \tag*{(5)}

for the estimated distribution at iteration tt and the true data distribution, respectively. We measure the discrepancy between them by the 2-Wasserstein distance, W2(⋅,⋅)W_2(\cdot,\cdot), defined in Appendix § 7.4.4, which has also been used by Shumailov et al. [22]. For μ(t)\mu^{(t)} and Σ(t)\Sigma^{(t)} themselves, we measure the discrepancy from the true values by the Euclidean norm, ∥⋅∥2\lVert\cdot\rVert_2, and the Frobenius norm, ∥⋅∥F\lVert\cdot\rVert_F.

Finally, we state our standing assumption on how new samples are generated.

Assumption 1 (Conditional independence). For every t≥2t \ge2, conditionally on (μ(t−1),Σ(t−1))(\mu^{(t-1)},\Sigma^{(t-1)}), the new samples {xi(t−1)}i=1nb(t−1)\{x_i^{(t-1)}\}_{i=1}^{n_b(t-1)} are drawn i.i.d. from the mixture (2) and are independent of all previously drawn samples {xi(s)}\{x_i^{(s)}\}, 0≤s≤t−20 \le s \le t-2.

In words, the past influences a new batch only through the current fitted model. This is not a restriction on the setting, but a formalization of the data-generating process described above.

Theoretical results

In this section, we present our key theoretical results. In Section 3.2.1, we establish that the mean estimator is unbiased and derive an explicit formula for the bias of the covariance estimator. In Section 3.2.2, we provide an equivalent (if and only if) condition for the convergence of the estimated parameters to the truth, and hence for the convergence of the estimated distribution to the true distribution. We then show that the model converges when the fraction of real data is fixed and arbitrarily small, as well as when the fraction of real data tends to zero sufficiently slowly. In Section 3.2.3, we analyze the convergence rate of the mean estimator.

The proofs for all the claims are deferred to Appendix § 7.

Bias of estimators and covariance of the mean estimator

Here, we derive the bias of each estimator and the covariance of the mean estimator. First, let us denote by CtC_t the covariance of the mean estimator

Ct:=Cov⁡(μ(t))=E[(μ(t)−E[μ(t)])(μ(t)−E[μ(t)])T],(6)C_t := \operatorname{Cov}(\mu^{(t)}) = \mathbb{E}\left[(\mu^{(t)}-\mathbb{E}[\mu^{(t)}])(\mu^{(t)}-\mathbb{E}[\mu^{(t)}])^{\mathsf{T}}\right], \tag*{(6)}

and by δ(t)\delta^{(t)} the deviation of the mean estimator from the truth:

References

  1. [1]Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G. Baraniuk. Self-Consuming Generative Models Go MAD. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=ShjMHFmPS0.arxiv.org/abs/2307.01850
  2. [2]Daniel Barzilai and Ohad Shamir. When Models Don’t Collapse: On the Consistency of Iterative MLE. In Advances in Neural Information Processing Systems, volume 38, 2025.arxiv.org/abs/2505.19046
  3. [3]Quentin Bertrand, Avishek Joey Bose, Alexandre Duplessis, Marco Jiralerspong, and Gauthier Gidel. On the Stability of Iterative Retraining of Generative Models on their own Data. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=JORA fH2XFd.arxiv.org/abs/2310.00429
  4. [4]Elvis Dohmatob, Yunzhen Feng, and Julia Kempe. Model Collapse Demystified: The Case of Regression. In Advances in Neural Information Processing Systems, volume 37, 2024.arxiv.org/abs/2402.07712
  5. [5]Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian, and Julia Kempe. Strong Model Collapse. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2410.04840
  6. [6]D. C. Dowson and B. V. Landau. The Fréchet distance between multivariate normal distributions. Journal of Multivariate Analysis, 12(3):450–455, 1982.DOI
  7. [7]George Drayson, Emine Yilmaz, and Vasileios Lampos. Machine-generated text detection prevents language model collapse. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 29657–29673, Suzhou, China, 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.emnlp-main.1506. URL https://aclanthology.org/2025.emnlp-main.1506/.DOI
  8. [8]Anvit Garg, Sohom Bhattacharya, and Pragya Sur. Preventing Model Collapse Under Over-parametrization: Optimal Mixing Ratios for Interpolation Learning and Ridge Regression. In The Fourteenth International Conference on Learning Representations, 2026.arxiv.org/abs/2509.22341
  9. [9]Matthias Gelbrich. On a Formula for the L2 Wasserstein Metric between Measures on Euclidean and Hilbert Spaces. Mathematische Nachrichten, 147(1):185–203, 1990. doi: 10.1002/mana.19901470121.DOI
  10. [10]Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Tomasz Korbak, Henry Sleight, Rajashree Agrawal, John Hughes, Dhruv Bhandarkar Pai, Andrey Gromov, Dan Roberts, Diyı Yang, David L. Donoho, and Sanmi Koyejo. Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=5B2K4LRgmz.
  11. [11]Hengzhi He, Shirong Xu, and Guang Cheng. Recursive Learning Without Collapse: A Weighting-Based Stabilization Framework. Journal of the Royal Statistical Society Series B: Statistical Methodology, 2026. doi: 10.1093/jrsssb/qkag099. arXiv:2502.18049.DOI
  12. [12]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840–6851, 2020.arxiv.org/abs/2006.11239
  13. [13]Roger A. Horn and Charles R. Johnson. Matrix Analysis. Cambridge University Press, 2 edition, 2012.
  14. [14]Joshua Kazdan, Rylan Schaeffer, Apratim Dey, Matthias Gerstgrasser, Rafael Rafailov, David L. Donoho, and Sanmi Koyejo. Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 29469–29494, 2025.arxiv.org/abs/2410.16713
  15. [15]Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
  16. [16]Chunxiao Li, Xiaoxiao Wang, Meiling Li, Boming Miao, Peng Sun, Yunjian Zhang, Xiangyang Ji, and Yao Zhu. Bridging the gap between ideal and real-world evaluation: Benchmarking AI-generated image detection in challenging scenarios. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 20379–20389. IEEE, 2025.arxiv.org/abs/2509.09172
  17. [17]Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in GAN evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11410–11420, 2022.arxiv.org/abs/2104.11222
  18. [18]Robert T. Powers and Erling Størmer. Free states of the canonical anticommutation relations. Communications in Mathematical Physics, 16(1):1–33, 1970. doi: 10.1007/BF01645492.DOI
  19. [19]Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can AI-Generated Text be Reliably Detected? Stress Testing AI Text Detectors Under Various Attacks. Transactions on Machine Learning Research, 2025. ISSN 2835-8856.
  20. [20]Leonardo V. Santoro and Victor M. Panaretos. Large sample theory for Bures–Wasserstein barycentres. The Annals of Applied Probability, 35(5):3215–3241, 2025.arxiv.org/abs/2305.15592
  21. [21]Mohamed El Amine Seddik, Sueı-Wen Chen, Soufiane Hayou, Pierre Youssef, and Merouane Debbah. How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=t3z6UIV09o.
  22. [22]Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. AI models collapse when trained on recursively generated data. Nature, 631(8022):755–759, 2024.DOI
  23. [23]Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256–2265. pmlr, 2015.
  24. [24]Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. In The Ninth International Conference on Learning Representations, 2021a.arxiv.org/abs/2010.02502
  25. [25]Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In The Ninth International Conference on Learning Representations, 2021b. URL https://openreview.net/forum?id=PxTIG12RRHS.

Paper details

vlnc:W_sw48763s
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 4

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents